Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
167 KB
Raw Blame

Rust engine: a third storage engine behind BrainEngine

Status: design only, no production change. Every switch named here ships off, and nothing changes on upgrade. The maintainer approved this plan on 2026-11-16 by taking every recommendation at the final review gate; "Approval record" under the decisions lists what that covers. Each later PR that adds a setting, migration, error code, default or MCP operation still quotes the maintainer's yes for that item. Citation convention: repo-relative paths on master 9f1d54232d (v0.60.733.0). The research maps in rust-engine/research/ were written against 36bd05806a, five commits earlier, and their line numbers refer to that commit. They are cited as research/NN (for example research/01 is docs/designs/rust-engine/research/01-engine-interface.md). gbrain-evals paths are cited as gbrain-evals:<path> at 9ecc5b20.

Recommendation

  • Build a Rust storage engine as a third implementation of BrainEngine (src/core/engine.ts:830), beside PGLiteEngine and PostgresEngine, inside this repository. It loads in process as a napi-rs addon and is selected only when a user sets engine: "rust".
  • Everything above the engine stays TypeScript: the CLI, MCP, operations, hybrid fusion, chunking, embedding, the persistence protocol, the error contract. They keep working unchanged and simply talk to a different engine.
  • A Rust engine cannot run Postgres SQL, and about 5,300 raw-SQL call-site lines sit outside the engines (re-measured at 9f1d54232d; see "Size"). The plan runs two tracks. Track 1: the Rust engine implements typed methods, refuses executeRaw with a typed error, and reports capabilities that gate subsystems not yet typed. Track 2: move raw SQL behind typed engine methods one subsystem at a time, for all engines, under a shrink-only ratchet. Track 2 is required work for the Rust engine, not optional (D18).
  • Existing brains move by shadow mode first (section "Shadow mode and swap"): Rust runs beside the current Postgres or PGLite engine as a follower, answers a sample of the same reads in the background, and a generated head-to-head report says READY or NOT READY. gbrain engine swap --to rust is allowed only on READY, keeps the old engine as a reverse shadow, and gbrain engine swap --to postgres (or --to pglite) is the way back.
  • The performance bar (decision D17): the Rust engine must be equal or better than TypeScript gbrain on Postgres and on PGLite on every axis in "Performance", including concurrent reads and writes, graph traversal, vector recall and latency, and memory. A loss on any axis is listed with its fix, never hidden. The maintainer's target is 10x better than Postgres and pgvector; equal or better on every axis is the gate to ship.
  • No SQL dialect in the Rust engine (D18): typed BrainEngine methods and the relation API only, raw SQL refused with a typed error, and every raw-SQL call site moved to typed methods as required Track 2 work.
  • Single machine first; a full multi-user Rust server that supersedes Postgres and pgvector is the target end state (D19). The embedded phases run one owner process per brain directory, but the storage core, the transaction API and the access checks are built to serve many users later (section "Path to a multi-user server"). Brains with writers on more than one machine stay on Postgres until that server passes the D17 bar.
  • No PGLite-style locking (D20). Concurrency is designed fresh for gbrain's workloads: no process-wide lock, no single-connection bottleneck, no pglite_busy equivalent, and no file lock that makes a second process fail or wait for the whole brain. Each lock and queue call site becomes a typed engine operation with a stated concurrency guarantee (section "Concurrency without PGLite-style locks").
  • PGLite deprecation gets a measurable entry gate (section "PGLite deprecation entry criteria"). Postgres deprecation is a separate decision after the multi-user server phases; shadow mode lets a single-writer Postgres brain move to Rust by choice, and nothing moves it by default.

Goal

Give every PGLite user an embedded engine that does what PGLite does today (vector ANN including filtered search, full-text, trigram, JSON fields, trigger behavior as explicit writes, graph walks, locks, the job queue), with the same contract, the same answers on deterministic operations and equal or better retrieval quality on ranked ones, without the WASM heap cap, the planner statistics problem or per-statement protocol overhead. Let any existing single-writer brain, on PGLite or Postgres, prove Rust on its own workload in shadow mode, with tracked head-to-head metrics, and swap with one verified command when the numbers say so, with a way back. Then retire PGLite once the Rust engine reaches measured parity.

Matching PGLite is not enough (D17). The Rust engine must be equal or better than TypeScript gbrain on Postgres (pgvector HNSW, local, same machine, same corpus, same run) and on PGLite, on every metric in "Performance": concurrent read and write throughput, latency, graph traversal, vector recall and speed, memory, cold open, import, index build and disk size. That is the gate to ship. The target beyond the gate is the maintainer's: 10x better than Postgres and pgvector (D17). The gbrain-evals preregistration fixes the margin on each axis; this plan sets no per-axis 10x number.

The end state is a full multi-user Rust server that supersedes Postgres and pgvector (D19). The embedded engine comes first, and it is built server-ready from the start, so the server phases extend it instead of replacing it.

The Rust engine does not copy PGLite's locking (D20). PGLite's locks come from running one WASM Postgres per process: one connection, one writer, and a file lock across processes. The Rust engine serves many readers, many writers and many processes on one machine, then many machines, with typed operations instead of lock primitives.

Why

Measured pain, all recorded in this repository (collected in research/12 A.1):

Symptom Measured Source What an engine without SQL changes
A type or date filter disables the vector index walk searchVector p50 462 ms filtered vs 56 ms unfiltered (Postgres, 20k pages); 195 vs 58 ms (PGLite, 10k) TODOS.md:50 The filter is applied inside the vector scan, or an exact scan runs when the filter is selective
Source-scoped vector search loses neighbours, then falls back 4 to 8 s fallback when a source holds a few percent of a large brain CHANGELOG.md, 0.60.131.0 entry Same: per-source filter bitmaps, exact scan for small scopes
PGLite HNSW build runs out of memory OOM at 1 GB maintenance_work_mem, wedged at 1.5 GB, 663 s at 50k chunks; builds past about 300k chunks at 1,024 dims are refused with pglite_vector_index_too_large CHANGELOG.md 0.60.146.0; docs/ENGINES.md "PGLite vector index ceiling" No 2 GiB WASM heap; a build with a memory budget
Stale planner statistics a hot read took about 72 s, 21 ms after a full ANALYZE; PGLite has no autovacuum TODOS.md:556 Access paths are coded, not planned
Per-write overhead on PGLite 113 to 30 ms owner wall time per write after tuning; 8.9 to 31.7 committed writes per second docs/research/pglite-persistence-throughput-2026-09-29.md No parse, plan and protocol round trip per statement
Cold first query budget 2,172 ms on PGLite vs 1,326 ms on Postgres (MCP path, 10k) scripts/scale/budgets.json No WASM boot or schema replay
Vector phase stalls at 20k pages 36 min before the WAL guard, 173 s after, RSS about 2.5 GiB TODOS.md:209 No checkpoint inside one WASM statement

The engine seam exists but is not closed: executeRaw(sql) (src/core/engine.ts:2714) lets about 1,170 files bypass the typed methods (research/01 section 9, re-measured at 9f1d54232d). That is why the plan has a Track 2.

Non-goals

  • No change to the Postgres engine's role in this plan's embedded phases. Postgres stays the engine for shared, hosted and multi-host brains until the multi-user Rust server passes the D17 bar (D19). Deprecating it is a later decision with its own plan. Shadow mode observes an engine from outside and changes neither engine's code; the only engine-side addition is the swap's write fence, which reuses graduation custody. A swap from Postgres is allowed only for a brain with one observed writer host.
  • No Rust rewrite of the CLI, MCP server, operations, hybrid fusion, chunkers, embedding, persistence coordinator or doctor. These stay TypeScript.
  • No separate repository, binary, command name or MCP server. No gbrain-rust.
  • No ranking or result change visible to callers unless an eval gate and the maintainer's yes allow it (decision D5).
  • No networked Rust server before the opt-in swap (D19). The server is a later phase (section "Path to a multi-user server"), and each of its phases starts only with the maintainer's yes. The embedded phases keep PGLite's process model, one owner process per brain directory, and refuse multi-host writers. The storage core must not depend on that model (see "Process model").
  • No PGLite-style locking (D20): no process-wide lock, no single-connection bottleneck, no pglite_busy equivalent, no file lock that makes a second process fail or wait for the whole brain, and no one-for-one copy of gbrain's lock primitives.
  • No SQL dialect and no query language in the Rust engine (D18). It serves typed BrainEngine methods and the relation API only.
  • No native reader for a PGLite data directory. Migration goes through the TypeScript copier.
  • No use of the Rust engine as an accelerator inside the PGLite or Postgres engines (option B below, deferred).

Constraints

From the user (binding):

  • Third engine behind BrainEngine, in this repository. PGLite and Postgres stay fully supported.
  • gbrain-evals is part of the parity proof.
  • Work lands through the maintainers' merge process, which lands every change on master.

From the merge process rules (binding):

  • Small PRs against master, each labeled rust-engine. Land by commenting queue: #<n> <full head sha>; no self-merge. Bring master in with merge commits; no rebase or force-push of a pushed branch.
  • No edits to VERSION, the package.json version, plugin manifests, CHANGELOG.md or release tooling. Each PR adds changelog.d/<branch-slug>.md; a PR that changes behavior for an existing user also adds behavior-changes.d/<id>.md.
  • full-ci label once the head is final. A changed or removed assertion, golden or baseline row gets a line in the PR body saying what it was, what it is and why.
  • All Rust sources, Cargo files and Rust build scripts live under one top-level directory. A PR that adds Rust code does not change how the existing engines behave.
  • The toolchain and CI PR goes first and alone, and states the CI time it adds.
  • Install and upgrade never need a Rust toolchain and never fail because a native binary is missing. A brain with no Rust engine available behaves exactly as today. Nothing uses the Rust engine unless a user asks.
  • the maintainer's explicit yes, quoted in the PR body, before: the engine-select config key or value, any default change, a schema migration, a new error code, a new MCP operation, anything that changes what a caller can read.
  • Same contract, same answers: the Rust engine passes the engine tests PGLite and Postgres pass. CLI and JSON output, MCP names and shapes, the markdown brain format, content hashes, fingerprints and deterministic operations stay byte-exact. Ranked search may return different results only when they are equal or better on the gbrain-evals retrieval metrics (D5, D17). Privacy and visibility match exactly; an engine that filters visibility differently is a blocker.
  • Git stays the parent: where a source has a checkout, content goes file, commit, then index. Brains without a repository stay fully supported, with the database as their record.
  • A heads-up: comment before changing shared files: src/core/engine.ts, src/core/engine-sql/, schema and migration files, scripts/, .github/, test/postgres-unit-arms.txt, goldens under test/fixtures/goldens/.
  • Public artifact rules from CLAUDE.md: placeholders only (alice-example, acme-example, source-a), security described functionally.

Architecture

What changes and what stays

CLI / MCP / operations / persistence coordinator / hybrid fusion / chunkers   (TypeScript, unchanged)
                 |  BrainEngine (typed methods only for Rust)
     +-----------+------------+--------------------------+
 PGLiteEngine           PostgresEngine            RustEngine (src/core/rust-engine.ts)
 (WASM Postgres)        (pgvector server)               |  process.dlopen
                                                gbrain_engine.node (napi-rs, rust/)
                                                        |
                                    record store + text index + vector index + graph

Because everything above the engine stays, the contract pieces the earlier full-rewrite plan had to rebuild come for free: operation names and schemas, the agent error envelope (docs/protocol/AGENT_OPERATOR_v1.md), degraded-result channels (docs/protocol/MCP_META_CHANNELS.md), write receipts and exit 10, memory-verb semantics (docs/protocol/MEMORY_VERBS_v1.md), the trusted-local versus remote split (OperationContext.remote, src/core/ops/contract.ts:502). The Rust engine only has to return the same rows for deterministic reads, and equal or better ranked results as D17 defines them.

Directory layout

rust/                                   all Rust sources, Cargo files, Rust build scripts
  Cargo.toml  Cargo.lock  rust-toolchain.toml  deny.toml
  crates/gbrain-store/                  record store, format header, relation catalog, outbox, counters
  crates/gbrain-text/                   Postgres-compatible text analysis, tsquery parser, ts_rank, CJK
  crates/gbrain-trgm/                   pg_trgm-compatible trigrams and similarity
  crates/gbrain-vector/                 VectorIndex trait, exact SIMD scan, ANN, filter bitmaps
  crates/gbrain-graph/                  adjacency lists, BFS walks with the engine row caps
  crates/gbrain-domain/                 typed domain methods, trigger equivalents, write invariants
  crates/gbrain-napi/                   napi-rs binding; the only crate that knows JavaScript
  bench/                                bake-off and performance harness
  catalog/relations.json                relation catalog generated from the PGLite schema
  fixtures/                             exactness fixtures generated from PGLite (ts_rank, pg_trgm, fingerprints, triggers)
  scripts/                              build, verify, catalog export, fixture export (TypeScript run with bun)
src/core/rust-engine.ts                 RustEngine implements BrainEngine
src/core/rust-engine/                   loader, capability table, transaction wrapper, row mapping
src/core/shadow/                        shadow mode: engine wrapper, follower, reconcile, comparisons, ledger, report
test/rust-unit-arms.txt                 test files whose Rust arm runs (grows only)
.github/workflows/rust-engine.yml       path-filtered CI lane

The TypeScript wrapper sits beside pglite-engine.ts and postgres-engine.ts, following "Adding a new engine" in docs/ENGINES.md.

Binding: napi-rs addon, in process

  • Chosen: an in-process Node-API addon built with napi-rs, loaded by the TypeScript wrapper. PGLite already runs in process, so the process model, the owner lock and the owner IPC socket carry over, with the owner protocol extended for the Rust engine (PR 18, D20). Vectors cross as typed arrays without a serialization step. The repository already ships and loads a native addon (native/locks/, loaded in src/core/persistence/native-lock.ts:94).
  • Rejected: a sidecar process over a socket. It adds a second process lifecycle and supervisor, serializes every row and vector, and makes transaction(fn) a cross-process protocol. Its one advantage is crash isolation, and in process that is only partly replaced. catch_unwind on every exported function, with panic = "unwind" in the release profile, turns a Rust panic into a typed error. It does not catch segfaults, aborts, allocation failures or faults inside native dependencies; those still end the process. The plan makes them rarer (no unsafe in gbrain crates without review, fuzzing at the binding boundary, cargo deny and cargo audit) and survivable (a store that recovers at the next open after any process death, proven by the crash robot), and accepts the residual risk for the preview. A sidecar is reconsidered if preview crash reports show native faults (see "Deferred").
  • Every call that touches disk or an index runs as an async task on a Rust worker pool, so the Bun event loop never blocks. PGLite runs queries on the main thread today (TODOS.md:556).
  • transaction(fn) keeps its shape: the wrapper opens a Rust write transaction handle and passes fn a transaction-bound clone of the engine, the same pattern PostgresEngine uses. Nested calls map to savepoints, an abort rolls back, readOnly refuses writes (research/01 section 6). AbortSignal maps to a cancellation token checked between steps.
  • Writer admission is starvation free. A caller waits for a writer permit from a bounded writer pool (D17, D19) asynchronously, on the JavaScript side, before it uses any worker thread. An open write transaction runs its commands on its own transaction executor, so the worker pool size never decides whether the transaction that holds the writer can finish. The wait is bounded by the caller's AbortSignal or deadline and ends with the existing cancelled or timeout code; a cancelled waiter leaves no permit or queued task behind.
  • Re-entry is refused. A call on the outer engine from inside its own transaction(fn) would wait forever for the writer it already holds. The wrapper detects it and throws internal_error with reason transaction_reentry (added to decision D2) instead of hanging. lockPageKeys maps to the typed per-key single-flight operation (D20): it holds only the named keys, in sorted order, until the transaction ends, and never blocks other keys.
  • Lifecycle. disconnect() and the resident-consumer stop barrier stop admission, cancel queued tasks, wait a bounded time for running tasks, and then close the store. A task cancelled after it started reports what it committed through the receipt. PR 1 proves the Bun behavior this needs before any engine code: shutdown with pending tasks, cancellation after start, process exit with pending native work, and a threadsafe-function queue that is full or closing.
  • Loading (src/core/rust-engine/loader.ts), modeled on loadBindingSync: resolve the target (linux-x64-glibc, darwin-arm64, ...), then load with process.dlopen from GBRAIN_RUST_ENGINE_ADDON (absolute path; development and CI, decision D3) or from <gbrain home>/engines/rust/<engine_build>/<target>.node. A literal require is not used, so compiled release binaries do not grow for users who never select the engine. The addon exports target, engine_abi (an integer bumped when the binding surface changes), engine_build (a digest of the rust/ tree it was built from), catalog_version and the store format_versions it reads and writes. Each release carries a generated manifest pinning the exact SHA-256 of the addon for each target. The loader refuses any other file and any engine_abi, catalog_version or format mismatch. Under GBRAIN_RUST_ENGINE_ADDON only the release digest pin is skipped; the engine_abi, catalog and format checks still apply, so a stale local build fails with its reason instead of opening the store. A release that leaves rust/ unchanged carries the previous digests forward, so the installed addon stays valid.
  • The addon is loaded only by constructEngine when engine is rust (dynamic import, as PGLite's WASM is today). A missing or mismatched addon gives engine_unavailable (decision D2) with the fix for its reason ("Error contract for the Rust engine" below), and nothing else changes.

Storage replacement map

Postgres feature today Used for Rust replacement Parity rule
Tables, rows, transactions, savepoints every domain gbrain-store: one table per relation in an embedded record store (redb or bundled SQLite used as a key-value store, picked by the bake-off); concurrent writers with snapshot isolation (per-relation or sharded writers with group commit, or an MVCC store, picked by the bake-off), concurrent snapshot readers; transactions and snapshots are bound to a connection, not to a thread or a process, so the same core serves many connections and processes (D17, D19) PGLite semantics: nested is a savepoint, abort rolls back. readOnly refuses writes, as on Postgres; PGLite ignores the option today (src/core/engine.ts:158), so this is a deliberate, stronger behavior, listed in the parity matrix as a divergence
pgvector vector, halfvec, HNSW, <=> chunk, image and multimodal embeddings; facts; takes vectors stored in records (f32, or f16 for halfvec columns); ANN index derived from them; exact SIMD scan always available cosine equal to pgvector within 1e-6; ANN recall@10 against exact no lower than pgvector's on the same corpus
Index walk, scope scan, iterative scan, escalations (search/vector-statement.ts, vector-pool.ts) filtered vector search roaring bitmaps per source, type, modality, page kind, date bucket, deleted and visibility class; exact scan below a filter-size threshold, filter-aware ANN above it, bounded escalation keep the vector_candidates_incomplete reason; one row per page; ties score DESC, page_id ASC, chunk_id ASC
to_tsvector('english'), setweight, GIN, websearch_to_tsquery, phraseto_tsquery, ts_rank, ts_rank_cd keyword, title and facts arms gbrain-text: the default-parser token rules gbrain's text hits, Snowball English stemmer, Postgres English stopwords, weights A to D with positions, websearch query grammar, ts_rank and ts_rank_cd formulas; postings in tantivy with a custom scorer or in the record store (bake-off) decision D5
gbrain_fts_input (CJK spacing), CJK LIKE scan title arm, CJK path the function ported; an n-gram substring index for CJK same matches as search/cjk-keyword-sql.ts; same ordering in the ts_rank compatibility mode, and equal or better retrieval quality with a better default ranker (D5, D17)
pg_trgm (similarity, %, word_similarity, GIN) find by name, resolver fuzzy arm, saved facts, takes search gbrain-trgm reproducing pg_trgm's trigram extraction and similarity, thresholds 0.3 and 0.6 unless configured scores equal to PGLite on a fixture set
JSONB, GIN, ->, @>, jsonb_to_recordset, jsonpath frontmatter, payloads, job data, purge matching JSON values in records; secondary indexes only for keys that are filtered on (frontmatter emails, __param_hash); purge matching in Rust code stored once as a document; output key order matches jsonb
Triggers (26 in src/schema.sql, more from migrations: research/02 section 9) generations, revisions, search vectors, projection queue, withdrawal and purge guards, trust stamping, access state one Rust write hook per trigger in gbrain-domain, run inside the same write transaction identical resulting rows, proven by a trigger parity matrix
Sequences, SERIAL ids, page_generation_clock_seq, graph_generation_seq durable allocator reservations, made before a value is handed out and never rolled back by an abort (as Postgres and PGLite sequences behave; graduation-copy.ts keeps aborted values from being reissued) monotonic across restarts and aborts; tests abort a transaction after allocating, restart, and check the value is never reissued
WITH RECURSIVE traverse_graph, retirement chains, rank_entities gbrain-graph: BFS over forward and reverse adjacency per source TRAVERSE_WALK_ROW_CAP = 20000, TRAVERSE_PATH_ROW_CAP = 5000, (slug, id) order, truncated flags
Advisory locks, FOR UPDATE, FOR KEY SHARE, SKIP LOCKED fact inserts per entity, queue and journal claims, leases typed engine operations, one per call site (leased claims, compare-and-set writes, idempotent upserts, per-key single-flight), listed in "Concurrency without PGLite-style locks" (D20); never a process-wide lock same claim order; fencing tokens checked in the transaction that writes; no lost update, no double claim, no deadlock under concurrent callers
LISTEN/NOTIFY minion wake on direct Postgres only nothing: PGLite already polls unchanged
RLS, set_config GUCs (gbrain.write_sources and others) defense in depth on Postgres; managed-writer guard a typed write-scope argument on every mutating method; the invariants I1 to I15 in research/09 section 12, enforced in gbrain-domain, never in gbrain-napi or the TypeScript wrapper, so they become the per-user access model a server enforces (D19) per-operation authorization tests
NULLS NOT DISTINCT, partial unique indexes, CHECK, FK cascade and restrict link identity, timeline dedupe, idempotency keys, tokens unique keys with a NULL sentinel; validated types; cascades and restricts in the same transaction same refusals and error codes
md5(), sha256(), lower(), regexp_replace inside SQL embedded_text_hash, fact fingerprints computed in TypeScript and passed in where Track 2 can move them; otherwise Rust code fuzzed against PGLite byte-identical (see "Exactness hazards")
NUMERIC, DATEMULTIRANGE, TIMESTAMPTZ, BYTEA calibration, link validity ranges, times, segment texts and attachments exact decimal type with Postgres rounding; interval sets; microsecond UTC integers; blobs exact
pg_stat*, pg_class, EXPLAIN gates, pools, PgBouncer Postgres doctor checks, scale plan gates not applicable, reported the way PGLite reports opsPglite: 'not_applicable' today n/a

Data on disk

~/.gbrain/brain.rust/                         database_path (a directory)
  FORMAT                                      JSON: format_version, engine_abi, catalog_version,
                                              schema_version (the TS config.version ledger it matches), created_by
  records.<ext>                               record store: relations, secondary indexes, counters, outbox
  text/                                       derived keyword index; records the outbox generation it reflects
  vectors/<relation>.<column>/                derived ANN graph; records its generation
  .gbrain-persistence.sock                    owner IPC socket (existing rule, src/core/context/resolve-ipc.ts)
~/.gbrain/brain.rust.gbrain-owner.lock        kernel lock sibling (existing rule, src/core/pglite-lock.ts); it decides
                                              which process hosts the store, never a lock a caller waits on (D20)
  • Record keys are order-preserving encodings of each relation's primary key. Values carry catalog column ids plus typed values, so adding a nullable column needs no rewrite.
  • rust/catalog/relations.json is generated from the live PGLite catalog at a pinned schema version (the same way src/core/persistence/graduation-inventory.ts reads the catalog): relation names, columns, types, keys, uniques and FKs, plus the secondary indexes the Rust engine keeps. Keeping Postgres relation names and columns lets the graduation inventory, digests and verification be reused through an engine-neutral adapter ("Migration and rollback"); the current helpers issue catalog and sequence SQL and cannot run on Rust unchanged.
  • One commit point. A write commits records and outbox entries in one store transaction (allocator reservations are durable separately and survive an abort, see the sequences row), then applies the outbox to the text and vector indexes, in generation order, before the call returns. A search issued after a write returns sees it, as today. If the process dies between commit and apply, the open catches the indexes up before serving reads.
  • Snapshot contract for search. Every search pins one record snapshot generation N. Text postings and vector entries carry the generation range in which they are valid, and the search uses only entries valid at N, so a reader at N never sees content, scores or permissions from N+1. Superseded entries are reclaimed only after the oldest open reader has passed their generation. As a backstop, each candidate's indexed revision is checked against the record at N, and deleted, withdrawn and visibility state is re-checked there before ranking; a mismatch drops the candidate and the search continues with the exact path for that arm, reporting the existing vector_candidates_incomplete reason when it cannot fill the page.
  • Failure after commit. If applying the outbox fails while the process stays alive (disk full, an index error, cancellation), the write is still committed: the call reports it as committed through its receipt and never as rolled back. The relation is marked behind at that generation; searches that touch it use the exact path for the affected arm until replay succeeds; replay retries in generation order and is idempotent; gbrain engine status and doctor report the lag.
  • The store refuses a newer format_version and backs up before upgrading an older one. gbrain doctor --fix rebuilds derived indexes from records without re-embedding and never writes records.

Process model

  • In the embedded phases the Rust engine keeps one owner process per brain directory, holding the kernel lock through the existing native lock addon, and other processes reach it through the existing owner IPC socket (research/08 sections 2 and 3). This is a temporary limit of the embedded phases, not a lock design the core depends on (D19, D20). Unlike PGLite, a second process never fails or waits for the whole brain: it becomes a client of the owner for reads and writes, with no pglite_busy equivalent, and if the owner exits a client takes over hosting the store, retrying calls in flight by request id. The existing owner protocol cannot do this as it stands, so PR 18 extends it (see "Owner protocol for the Rust engine" below). The limit ends when the bake-off picks a store that several processes open directly without a process-wide writer lock, or at the server phase. The persistence consumer election already treats a one-process engine as "no other owner" (src/core/persistence/consumer-election.ts).
  • Those paths branch on engine.kind === 'pglite' today. Track 1 replaces those branches with one capability, single_owner, which PGLite and Rust report, so the Rust engine reuses the code instead of copying it. Some of these checks run on the config before any engine exists (src/core/context/resolve-ipc.ts keys the owner socket and secret on cfg.engine === 'pglite'; src/core/persistence/graduation-serve-guard.ts does the same), so a config-level helper, engineIsSingleOwner(cfg.engine), answers the same question there, and both forms are covered by the kind-branch guard.
  • The owner process is a deployment choice for the embedded phases, not an assumption of the storage core (D19). The core has no global singletons and no per-process state that assumes one caller: every store handle, transaction, snapshot, writer permit and cache belongs to an engine instance, and every transaction belongs to a connection. Transactions and snapshots work across connections and processes, not only across threads in one process: IPC clients of the owner today and server clients later get the same isolation. A Rust test opens two engine instances on two stores in one process and runs concurrent transactions from several client processes through the owner IPC socket.
  • Owner protocol for the Rust engine (PR 18, D20). Today's owner protocol carries writes and a few reads only (PERSISTENCE_IPC_OPERATIONS in src/core/persistence/ipc.ts; no search and no transactions), refuses a request over 8 connections with queue_capacity, and makes a client wait for the engine lock when the lock holder is a command that serves no socket (src/core/persistence/local-client.ts). For a Rust brain: every owner publishes its socket, a one-shot CLI command included, so no holder makes a client wait for the lock; the protocol carries every BrainEngine method a client needs, search and connection-bound transactions included; a request over the connection limit queues inside its own deadline instead of failing at once, ends with timeout or cancelled only when that deadline passes or the caller cancels, and a cancel crosses the socket to the owner; at a handoff, a transaction still open on the old owner aborts and is reported as aborted, never as partly committed, and the client reruns it by request id, while a single call retries by request id against the write receipt. Doctor and doctor --fix on a Rust brain go through the running owner as an administration request, and take the owner role themselves only when no owner runs (see "Maintenance path"). A test runs each rule across processes: queueing over the limit, cancel, handoff during a transaction, and doctor beside a running owner.
  • Inside the owner, many writers run at once on the concurrent-writer store the bake-off picks (D17, D19); readers use snapshots and never block writers. The store must keep these guarantees: claim and lease exclusivity, per-entity fact ordering, fencing tokens checked in the writing transaction, the transactional invariants the parity matrix lists, and, on swap rollback or shutdown, fencing and draining every admitted writer before the store closes. The writer permit (see "Writer admission") comes from a bounded writer pool with the same starvation-free and cancellation rules.
  • Index-backed reads inside an open write transaction read committed index state. A guard test lists any call site that searches inside its own write transaction (expected: none); if one exists, it is fixed before that subsystem is allowed on the Rust engine.

Concurrency without PGLite-style locks

The maintainer's principle (D20) is no PGLite-style locking. gbrain's TypeScript code today serializes work with lock primitives: named advisory locks, row locks, SKIP LOCKED queue claims, a lease table and PGLite's whole-brain owner lock with its pglite_busy error (research/08 sections 3 to 6, research/07). The Rust engine does not copy them one for one. Each call site gets a typed engine operation with a stated guarantee. Track 2 moves each call site to its operation for all engines; on Postgres the operation may still take the Postgres lock inside its own SQL.

Call site today Typed engine operation Concurrency guarantee
Brain open: pglite-lock.ts kernel lock, PgliteBusyError (pglite_busy), LiveServeLockError open_brain: the first process hosts the store, any other process becomes its client, and a client takes over hosting when the owner exits a second process never fails or waits for the whole brain; calls in flight at a handoff retry by request id
Schema init: pg_advisory_lock(42) (postgres-engine/init-schema-lock.ts, db.ts) upgrade_format, a compare-and-set on format_version inside open exactly one upgrade runs; every caller sees the old or the new format, never a partial one
Write journal claims: claimNextWrite and batch claims with FOR UPDATE ... SKIP LOCKED (persistence/journal.ts, persistence/consumer.ts) claim_next_writes(lane, limit, lease) each request has one claimant at a time; the claim order and eligibility stay as today (CLAIM_ORDER and claimableWriteSql in persistence/journal.ts: with foreground_priority on, the default, a claimable foreground request goes before the sync rows it passes, then sequence; per-worktree and per-source heads, deferred rows and recovery blocks still gate a claim); the lease carries a fencing token; an expired lease is reclaimed
Effect claims (persistence/effect-journal.ts, persistence/embedding-noop-settle.ts) claim_effects(kind, limit, lease) one claimant at a time, with a fenced execution token that the completion record must match; an expired or dead-owner claim is reclaimed, as claimPersistenceEffect does today, so an effect can run again after a crash between its external action and its recorded completion; each effect kind stays idempotent or keeps its own recovery step; UNIQUE(request_id, kind) stops duplicate journal rows only, not a second execution; a crash test kills the owner at that boundary on every engine
Counters and caps: persistence_counters FOR UPDATE (persistence/journal.ts) reserve_capacity(key, delta, cap), an atomic compare-and-add no over-admission and no lost update
Minion job claim, stall, timeout and watchdog: FOR UPDATE SKIP LOCKED LIMIT 1 (minions/queue.ts) claim_next_job(queue, worker, lease), renew_job_lease, reclaim_stalled_jobs no double claim; completion and failure check the lease token
Minion admission, quota and max-waiting advisory locks (minions/queue.ts) admit_job(spec): idempotency key, admission policy, quota and caps checked and the job inserted in one operation the admission contract stays as today (minions/queue.ts): an idempotency key maps to one row; parameter coalescing applies only where the job's policy turns it on, the payload carries parameters, no idempotency key is given and the submit has no parent, and it uses the same job states and window; caps are never exceeded; the cases in test/minions-admission.test.ts stay, including the rerun of a running job, separate children and a submit after the window ends
Parent row lock on child submit and roll-up (minions/queue.ts) submit_child_job(parent, spec), roll_up_child(parent, child) exact child counts and roll-ups with concurrent children
Budget meter and rate leases (minions/budget-meter.ts, minions/rate-leases.ts) reserve_budget(key, amount, cap), acquire_rate_lease(key, ttl) no over-spend; a lease ends at its TTL
Facts per entity: pg_advisory_xact_lock on source and entity (engine-sql/facts.ts) append_fact(entity, fact), with the per-entity sequence assigned at commit one total, stable fact order per entity; no lost fact
Page body chunks, writeback settings, mentions pass, chronicle reservation (engine-sql/page-body.ts, facts/writeback-settings.ts, mentions/pass.ts, chronicle/ledger.ts) compare-and-set writes on the row revision, idempotent upserts on the natural key, reserve_chronicle_range no lost update; a stale writer gets a typed conflict and re-reads
Generation fence and publish (generations/fence.ts, generations/publish.ts) publish_generation(expected), a compare-and-set generations stay monotonic; two publishers never win the same generation
Derivation scope lock (derivationScopeLockSql in page-body/schema.ts, sweeps in page-body/segment-derive.ts): writers that bring rows into a sweep's scope hold it shared, and a sweep moves to its verify walk and ends only under it exclusively enter_derivation_scope(source) in the writer's transaction, and finish_derivation_sweep(source, sweep), which checks the writers that entered the scope since the sweep was recorded a sweep ends only after every writer open when it was recorded has committed or aborted and the verify walk sees that writer's rows; a sweep never deletes a derivation row that a writer still open will bring back; the open-writer cases in test/helpers/derivation-purge-cases.ts run on every engine; writers in other sources never wait
Cycle locks and the sync drain lease: gbrain_cycle_locks, fenceLockRow FOR KEY SHARE (db-lock.ts, persistence/sync-lease.ts) acquire_lease(id, ttl), renew_lease(token), assert_lease(token) checked in the writing transaction one holder; fenced work never blocks a renewal; a stale holder's write is refused
Consumer election, heartbeat and worktree sweep (persistence/consumer-election.ts, persistence/consumer-heartbeat.ts) register_consumer, heartbeat, claim_sweep one full consumer per host; each sweep row is claimed once
Grant moves, token backfill and OAuth client admission (grants/legacy-token.ts, minions/delegated-admission.ts) move_grant(expected_state), claim_backfill_batch, admit_delegated(client, expected) no double grant; each backfill row is claimed once
Links, withdrawals and purge row locks (research/06) checks and compare-and-set on the target's state in the writing transaction a withdrawn or purged target is never linked; restrict refusals are unchanged
lockPageKeys single_flight(keys) for one transaction holds only the named keys, in sorted order; released at commit or abort
  • No operation takes a process-wide or brain-wide lock on the hot path. Waits are bounded per call and per key, end with the existing cancelled or timeout code, and never block the whole brain.
  • Filesystem resource locks stay as they are, because they lock one file-system resource outside the store, not the brain, and do not depend on the engine. Each waits a bounded time for that resource only, or tries once without waiting; a busy resource refuses only the step that needs it, usually with a typed code and a retry with the same request id, and the Rust work keeps today's result for each one (a socket bind that finds the path taken returns no lock, and a setup step reports its own error). They are: Git checkout locks (brain-commit.lock, the push lock and the shell flock hook); worktree coordination and refresh locks (persistence/ownership.ts, topology-locks.ts, topology-clone.ts, coordination-lock.ts, worktree-refresh.ts, head-blocker.ts); the writer registration lock per lane (persistence/identity.ts); the reconciliation backup capacity lock (persistence/reconcile-backup.ts); socket bind locks (cli-socket/server.ts, context/ipc-path.ts); skills-directory and setup locks (harness/native-router.ts, shared-skills/adapter.ts, shared-skills/migration-export.ts, setup/brain-first.ts, setup/hermes-turn-hook.ts); and the probes in persistence/activation.ts and persistence/administration.ts, which take their lock only for a moment to test it and then release it. The migration orchestration lock (migration-orchestration-lock.ts) runs only during apply-migrations, never on the hot path; on a Rust brain the store side of an upgrade is upgrade_format. The PR 6 recount covers these files too.
  • Proof: each typed operation has a parity test against TypeScript gbrain on PGLite and Postgres that runs concurrent callers and checks the same observable outcome (no lost update, no double claim, no deadlock), and a stress test with many writers in many processes (see "Crash, property, concurrency and fuzz tests"). The call-site list is re-counted when PR 6's ratchet lands; a lock call site with no row here fails that check.

Engine selection

  • ~/.gbrain/config.json gains the value engine: "rust" with a database_path (default ~/.gbrain/brain.rust). GBRAIN_DATABASE_URL and DATABASE_URL keep implying Postgres (src/core/config.ts:738). Decision D1.
  • constructEngine (src/core/engine-construct.ts) gains a rust case and its unsupported-engine message names all three engines. BrainEngine.kind (src/core/engine.ts:832) gains 'rust', and so does every other 'postgres' | 'pglite' engine union (the config type in src/core/config.ts, targetEngine in src/commands/migrate-engine.ts, the config shape in src/core/context/resolve-ipc.ts). The graduation inventory's engines record becomes { pglite, postgres, rust } with no default for rust, so a row that does not say how Rust stores it fails to compile.
  • 268 direct kind === / !== comparisons exist in src today. A new guard, check:engine-kind-branches, baselines them and fails on new two-way branches. Before a subsystem is allowed on the Rust engine, its branches become capability questions or an exhaustive byEngineKind({ pglite, postgres, rust }), so a Rust brain never falls silently into a Postgres branch.
  • Users reach the engine only through gbrain migrate --to rust (and back with --to pglite), after the preview opens (decision D8). openEngine accepts kind rust for harnesses. gbrain init keeps its PGLite default.

Capability gating

  • BrainEngine.capabilities is a read-only set of named subsystem capabilities (raw_sql, persistence_journal, minion_queue, access_control, belief_ledger, single_owner, postgres_admin, ...). PGLite and Postgres report everything they have today. Rust reports what it implements.
  • One table, src/core/engine-capabilities.ts, records the capabilities each operation, CLI command record (src/cli/command-table.ts), cycle phase and minion handler needs. Anything not listed needs raw_sql, which Rust never has. The check runs at dispatch, before any work: the caller gets engine_capability_missing naming the capability, and nothing is written. Cycle phases and handlers without their capability are skipped with a degraded reason on the existing channel instead of failing the cycle.
  • executeRaw, executeRawDirect, withReservedConnection and any other raw-SQL entry point on the Rust engine throw engine_capability_missing with reason raw_sql (D18). The Rust engine has no SQL dialect to fall back on. This is a backstop for a missed path, not the gate; the Rust test arm proves each allowed operation never reaches it.
  • Every BrainEngine member is classified in src/core/rust-engine/members.ts, a table typed satisfies Record<keyof BrainEngine, 'implemented' | Capability>. A new interface member fails to compile until it is classified, so the list cannot drift by hand. A member not yet implemented throws engine_capability_missing naming its capability.
  • check:rust-capability-coverage prints the share of operations, commands, phases and handlers available on Rust. It is the progress number and the first PGLite-deprecation criterion. It is derived from the member table and the dispatch table, not kept as a separate list.
  • Postgres-only administration (pool diagnostics, pg_stat checks, RLS scope binding, Postgres graduation targets, PgBouncer handling) needs postgres_admin and stays unavailable on Rust by design.

Error contract for the Rust engine

  • Every new code and reason is a row in src/core/error-registry.ts (class, summary, why, reasons, suggestion, default read-only fix, docs anchor). bun run build:error-codes regenerates docs/guides/error-codes.md, and gbrain errors <code> answers offline. Errors use the existing envelope (docs/protocol/AGENT_OPERATOR_v1.md): stderr and a non-zero exit on the CLI, one --json document, isError over MCP.
  • Each envelope's detail carries caller-safe context: the gbrain version, the target, expected and found engine_abi, engine_build, catalog_version and store format, the names (never the values) of the environment variables that mattered, and home-relative paths only. Every refusal says whether anything was written.
  • One fix per reason:
Code and reason Fix Actor, next Recovery
engine_unavailable addon_missing gbrain engine install rust; verify with gbrain engine status --json agent, run nothing was touched; retry the failed call once after the install
engine_unavailable abi_mismatch, build_mismatch gbrain engine install rust (fetches the addon this release pins) agent, run as above; under GBRAIN_RUST_ENGINE_ADDON the fix says to rebuild with bun run rust:build instead
engine_unavailable unsupported_platform none to run: no addon exists for this target; open the brain on a supported preview host, or keep this brain on PGLite user, tell_user_to_run do not retry the install on this target
engine_unavailable load_failed gbrain engine install rust once; if it fails again, report with detail.diagnostic_id agent, run, then report one retry; the sanitized loader cause is in detail
engine_unavailable fetch_failed after connectivity returns, gbrain engine install rust once agent, run the staged download was discarded and the active addon is unchanged; one retry, then report
engine_unavailable verification_failed none; report it; there is never a bypass user, report the downloaded file is deleted and the active addon is unchanged
engine_capability_missing before dispatch none to run on this brain: tell the user the operation is not in the preview, with the capability list anchor; moving back is the user's choice, previewed read-only with gbrain migrate --to pglite --plan user, ask_user nothing ran
engine_capability_missing raw_sql (backstop) read the write receipt (gbrain write-request -- <id>) before any resubmission agent, run (read only) detail.outcome is none when the enclosing transaction rolled back, else partial or unknown; reconcile before retrying
engine_store_format_unsupported newer_format gbrain upgrade to at least the release named in FORMAT created_by user, tell_user_to_run the store is never downgraded or rewritten
engine_store_format_unsupported unreadable keep the directory as it is; restore from a verified backup (gbrain backup restore) user, ask_user nothing was written
engine_store_format_unsupported upgrade_failed none to run; the store stays at its old format and detail names the backup written before the upgrade user, report the old format remains readable by the previous release
  • Caught panics: gbrain-napi installs its own panic hook, because the default hook prints the payload to stderr. The hook writes a sanitized trace (no row content, no argument values) to <gbrain home>/engines/rust/diagnostics/<id>.txt with mode 0600 and the call returns internal_error naming the operation, with detail.diagnostic_id and detail.outcome. A panic inside a write transaction aborts it; the wrapper then closes and reopens the store, which runs open-time recovery, before the next call. Over HTTP MCP only the diagnostic id is returned.
  • A cancellation that arrives after commit reports the write as committed through its receipt, never as cancelled.
  • Faults that end the process (segfaults, aborts) cannot be caught. The next open records that the previous owner ended without closing the store, and doctor rust_engine_available reports it with the engine_build and the recovery result.
  • Maintenance path: gbrain doctor and doctor --fix on a Rust brain send their work to the running owner as an administration request; when no owner runs, they take the owner role themselves. Either way they read FORMAT and the records without opening the derived indexes, so a corrupt index or a failed open-time replay can still be inspected and rebuilt. --fix rebuilds derived state only and never writes records.
  • Human output keeps the shared renderer (Error [code], Fix:, Why:, Docs:); agents read fix from the --json document.
  • The reasons added here (fetch_failed, verification_failed, newer_format, unreadable, upgrade_failed) are part of decision D2.

Privacy and visibility

  • The source-scope ladder (sourceScopeOpts, src/core/ops/context.ts:646), trust checks and redaction stay in TypeScript and are unchanged. The engine receives scope and visibility arguments exactly as the other engines do.
  • Inside the engine, source scope, private pages, readable_by, hard excludes and soft-deletes apply inside each candidate set (vector, keyword, title, CJK, facts, relational), before ranking and before any text leaves the engine (research/03, "Must preserve exactly"). Malformed readable_by fails closed as !nobody; restrictions_present is maintained in the same write transaction (research/09 I6 and I7).
  • Proof: the operations source-isolation matrix, test/helpers/two-engine.ts isolation cases, test/memory-verbs-conformance.test.ts, and gbrain-evals Cat 22 (0 leaks) all run on the Rust arm. Any difference in what a caller can read is a release blocker, not a tolerance.

Exactness hazards

Each has a fixture generated from PGLite under rust/fixtures/ and a test on the Rust arm.

  • Content hash and chunk boundaries: computed in TypeScript (src/core/utils.ts contentHash, src/core/chunkers/), so they do not change. The engine stores what it is given.
  • embedded_text_hash = md5(chunk_text): computed in SQL today; Rust computes md5 over the same UTF-8 bytes.
  • Fact fingerprint: gbrain_fact_fingerprint is SQL (src/core/facts/withdrawal-schema.ts:36) using Postgres lower() and POSIX classes; withdrawals and purge tombstones key on it, so a different value silently un-forgets a fact. Migration carries stored fingerprints verbatim. PGLite and Postgres keep computing it in SQL, unchanged, because existing Postgres fingerprints depend on that host's lower() and whitespace classes. The Rust engine computes gbrain_fact_fingerprint and gbrain_fact_fingerprint_v1 in its own code, fuzzed against PGLite's SQL on a corpus with non-ASCII text, mixed case and every whitespace class; before a Postgres swap, the same corpus runs against that host's SQL for both versions (see Fingerprints under shadow mode). Moving the computation out of SQL for all engines, or a versioned normalization, is a separate change that needs the maintainer's yes.
  • Page generation, the global generation clock, knowledge_revision and text_projection_revision rules (src/schema.sql:182-205, :255-278, :1904-1961): ported predicates plus a scenario matrix per column.
  • Belief ledger hashes (event_key, state_hash, prior_hash): carried verbatim; new events use the same byte encoding; the chain is verified after migration.
  • NUMERIC calibration bucketing: exact decimals with Postgres rounding; golden buckets at boundaries.
  • now() is the transaction start time in Postgres: the Rust write transaction carries one timestamp, at microsecond precision.
  • Text ordering and lower() follow PGLite's collation, which is the engine being replaced; the engine-graduation spike's mixed-case and accented key set (docs/designs/engine-graduation-spike.md, "Setup") is part of the fixture.
  • jsonb output: keys come back in jsonb order with duplicates collapsed; the Rust engine returns the same order.
  • Segmented page bodies (page_segment_texts, page_body_manifest) exist only in the database (research/02 section 12): stored and migrated as user data, never treated as rebuildable.
  • Keyword ranking: decision D5.

The SQL problem and the chosen strategy

Size

  • At 36bd05806a (research/01 section 9): about 4,890 executeRaw / executeRawDirect call-site lines in about 1,140 files outside the engine files, plus 186 tagged-template sql lines in 44 files, 20 .unsafe( and 29 .query( lines (research/01 section 9).
  • Re-measured at 9f1d54232d with the pattern executeRaw(Direct)?\s*[<(] over src, excluding pglite-engine.ts, postgres-engine.ts, engine.ts and src/core/engine-sql/: 5,324 lines in 1,174 files. The raw-SQL ratchet baseline (PR 6) is generated from the tree at its own commit, never copied from this plan.
  • Only 15 storage domains live in src/core/engine-sql/; 54 engine members still own their SQL (scripts/engine-sql-baseline.tsv), including both keyword search methods and config.
  • The largest raw-SQL user is src/core/persistence (353 files, 83,862 lines, 1,487 raw-SQL lines at 9f1d54232d), then src/commands (614, of which commands/doctor 294), core/repair (334), core/facts (244), core/minions (204), core/cycle (167), core/trust (135), core/shared-skills (125), core/page-state (113), core/belief (109), core/access (108) and core/search (100) (research/01, research/08, re-counted with the pattern above). Persistence, page state and facts (1,844 lines together) gate Phase 7 and so the preview: their size sets the date of first user value.
  • The ratchet (scripts/check-engine-sql-ratchet.ts) sees only engine class members. It cannot see SQL in helpers that take Pick<BrainEngine, 'executeRaw'>, such as src/core/feedback/store.ts:340.
  • Tests: 790 of the 830 files that call testBackends() (re-measured at 9f1d54232d) also call executeRaw for setup or assertions.

Options considered

A. Typed engine plus a typed-methods track (chosen). Track 1 builds the Rust engine against typed methods and gates the rest by capability. Track 2 moves raw SQL into typed engine methods subsystem by subsystem, for all engines.

  • Fixes the root causes (planner statistics, filtered ANN, WASM limits) because the Rust engine has no SQL planner at all.
  • Leaves PGLite and Postgres behavior unchanged: Track 2 moves SQL without changing it, and the same tests prove it.
  • Makes the seam real. docs/ENGINES.md already says "the interface assumes no SQL", while executeRaw says otherwise; Track 2 makes the first statement true.
  • Cost: Track 2 is most of the work. It is done as many small, behavior-preserving PRs that also give the Postgres engines typed, testable boundaries.

B. A Rust search and index accelerator first. Rust would own vector and text indexes as a derived sidecar while PGLite or Postgres stays the record store.

  • Fixes the two largest latency pains (filtered ANN and FTS) and the PGLite HNSW out-of-memory quickly.
  • But it changes how the existing engines behave, which the merge process rules keep out of Rust PRs; it needs a second commit point kept consistent with Postgres across crashes (an outbox fed by triggers); Postgres hosts would need the native addon; it does nothing for planner statistics on other tables, per-statement overhead or the WASM data cap; and it does not move toward retiring PGLite.
  • Its useful parts (gbrain-vector, gbrain-text) are built in option A anyway. Kept as a possible later offer to Postgres users (open question 8).

C. An embedded Rust SQL engine that speaks enough Postgres, through a translation shim.

  • gbrain's SQL uses plpgsql triggers, jsonpath, tsvector, WITH RECURSIVE, window functions, DISTINCT ON, LATERAL, FOR UPDATE SKIP LOCKED, advisory locks, set_config and catalog queries. Rust SQL libraries (sqlparser-rs, DataFusion, GlueSQL) cover parsing and analytics, not this dialect with OLTP semantics.
  • It would also bring back a planner, which is the source of the measured pain. Rejected.

C'. A native embedded Postgres binary in place of WASM. Not Rust, but worth naming: it keeps all SQL working and removes the 2 GiB cap. It ships a database server and extension builds per platform, keeps the planner and filtered-ANN problems, and does not match the user's direction. Rejected for this plan.

Track 2 rules

  • One subsystem (or one file family) per PR. The SQL moves into src/core/engine-sql/<domain>.ts behind typed methods, and both Postgres-dialect engines delegate, following the "Storage method" row of CONTRIBUTING.md "Where does my change go?". No behavior change; changed assertions explained in the PR body.
  • Methods are named for what they mean (journal.claimNext(...)), not for the SQL they replace.
  • New methods are grouped as domain objects on the engine (engine.journal, engine.queue, engine.access, ...), each declared in its own module and referenced from engine.ts with one line, so engine.ts grows by one line per domain, not per method. This limits contention on a shared file and its check:module-size ceiling.
  • A new shrink-only ratchet, check:engine-raw-sql-ratchet with scripts/engine-raw-sql-baseline.tsv, counts raw-SQL call sites per file outside the engines, including helpers that take the engine or an executor. It runs in verify. A new raw-SQL call site outside an engine fails it. Decision D6.
  • Each Track 2 PR also moves the touched tests' setup and assertions off executeRaw onto typed helpers or the relation API (below), so those test files can join the Rust arm.
  • Order follows what an embedded brain reaches first: search arms and keyword methods; the 24 shared baseline method families (config, raw data, ingest log, ontology, dream verdicts, eval capture); the persistence journal, claims, counters, publication, recovery records and effects; page state, projections, generations and page bodies; facts (remember, forget, withdrawal, purge) and takes; the belief ledger; the minion queue, cycle locks and budget meter; access, trust, tokens and OAuth; mentions and entity identity; cycle phases and dream; repair; doctor checks; remaining commands.
  • Track 2 is required for the Rust engine, not optional (D18). Every raw-SQL call site a Rust brain can reach moves to typed methods or the relation API: migrations (as typed store-format steps on Rust, never SQL), doctor, the tests in the Rust arm, and the seven gbrain-evals files that run raw SQL against gbrain's schema (cat18, cat19, cat22, cat24, cat27, cat28 and longmemeval.ts, research/11), which that repository's maintainers move under D11. Capability gating is a temporary state on the way, never the end state. Checkpoint C1 can still narrow or stop the Rust engine as a whole.
  • Postgres-only administration is never moved; it needs postgres_admin, which a Rust brain never reports, so the command refuses with the typed error.
  • Coordination with the merge process, per T-PR: a heads-up: comment naming the subsystem, the files and the baseline rows before the branch is pushed. If the merge process reports open PRs on the same files, the T-PR waits or is cut smaller instead of racing them. Churn on master in the 30 days before 9f1d54232d (first-parent commits): src/core/engine.ts 21, src/core/engine-sql/ 59, src/core/persistence/ 279, scripts/module-size-limits.tsv 215.
  • src/core/engine.ts is 2,890 lines, exactly its check:module-size ceiling (scripts/module-size-limits.tsv:19). Each domain object adds at most one line to it, its types live in their own module, and the ceiling row change with its reason rides in the same PR under the scripts/ heads-up.
  • Track 2 is written by the Rust engine effort and landed and sequenced by the merge process, which does not write code. Before the first T-PR, a question: to the merge process asks whether other work it lands should follow the D6 ratchet the same way, so SQL moves in one direction only.

The relation API

A small typed, non-SQL interface on every engine: readRelation(name, { equals, range, orderBy, limit, after }), relationDigest(name), and, on the Rust engine only, importRelation(name, rows). It is not a query language: equality and range filters on catalog columns only. It serves three users: tests that today inspect rows with executeRaw; the migration copier's target side; and the verification digest that both engines compute the same way.

Phases

Track 2 starts in Phase 4 with the raw-SQL ratchet (PR 6), the search arms and the 24 shared baseline method families. The rest of Track 2 starts only after Checkpoint C1. Each phase lists entry and exit criteria.

First value goes to PGLite users on darwin-arm64 or linux-x64-glibc who hit the measured pain above (filtered vector latency, the HNSW memory ceiling, stale planner statistics, write throughput). The preview (Phase 8) is built for them.

Phase 0: research map (done). Twelve maps in docs/designs/rust-engine/research/, pinned to gbrain 36bd05806a and gbrain-evals 9ecc5b20: 01 engine interface and raw-SQL bypass; 02 schema, triggers, RLS; 03 search pipeline; 04 ingest, chunking, embedding; 05 CLI and MCP surface; 06 graph and knowledge layer; 07 jobs, queue and cycles; 08 concurrency and persistence; 09 access and security invariants; 10 tests as an oracle; 11 gbrain-evals as an oracle; 12 performance and operations.

Phase 1: toolchain and CI. Entry: this plan approved, and an effort budget through Checkpoint C1 recorded in this document by the maintainer (recorded: 8 calendar weeks or 25 merged PRs from PR 1, whichever comes first). Work: PR 1 alone. Exit: the CI lane is green; a trivial addon loads in Bun 1.4.4 and in a bun build --compile executable through process.dlopen; async tasks, typed arrays and panic-to-error work in Bun; shutdown with pending tasks, cancellation after start, process exit with pending native work and a full or closing threadsafe-function queue behave as the lifecycle rules above require; the added CI time is reported to the merge process.

Phase 2: storage, ANN and text bake-off. Entry: Phase 1 landed. Work: compare record stores (redb, and bundled SQLite used only as a plain key-value record store; brain relations and their invariants are never built from SQLite tables, constraints or triggers, since the Rust engine has no SQL dialect, D18), ANN options (usearch with predicate filtering, LanceDB filtered ANN with its exact-scan fallback, an own HNSW, exact SIMD scan over roaring bitmaps) and text options (tantivy with a custom ts_rank scorer, an own postings table). Corpora at 10k, 200k and 1M chunks with real 1,024- and 1,536-dimension vectors (the scale fixture's 16 nonzero dimensions make HNSW look easy, scripts/scale/fixture.ts:70). Measure recall@10 against exact, p50 and p99 at filter selectivities 100, 50, 10, 1 and 0.1 percent, build time and peak RSS, disk size, crash recovery, addon size, and how exactly ts_rank is reproduced; PGLite and Postgres on the same machine as baselines. Per D17 it also measures concurrent read throughput at 1, 4, 16 and 64 clients, concurrent write throughput at 1, 4, 16 and 64 writers, graph traversal at depth 1 to 5 against Postgres WITH RECURSIVE, and peak and steady RSS per 100k chunks. Beating Postgres on the writer curve is a bake-off criterion. The record store must support concurrent writers (D17, D19): for example per-relation or sharded writers with group commit, or an MVCC store; a single writer with group commit is measured as a baseline only. Server-ready criteria join the bake-off (D19): transactions and snapshots work across processes and connections, not only threads in one process (measured with several client processes through the owner IPC socket); no global singletons or per-process state that assumes one caller; and a store format that can carry a commit log or snapshot stream a follower can consume, even though no follower is built yet (section "Path to a multi-user server"). Readers stay lock-free snapshots and use all cores, including parallel ANN and text arms. No PGLite-style locking joins the criteria (D20): no process-wide writer lock on the hot path; a second process opens the brain and reads and writes without busy errors; lock waits, conflict retries and p99 under contention are measured and reported for every option. The embedded phases still run one owner process per brain directory and refuse multi-host writers (D19), but no store option may depend on either. Before any Rust number exists, bakeoff.md names the target machines and the memory envelope (proposed: a 16 GB darwin-arm64 laptop and an 8 GB linux-x64-glibc VM, with a resident-memory budget at 200k chunks written there), and a memory budget per 100k chunks (f16 or quantized vectors and memory-mapped indexes allowed), measured against Postgres RSS as well as PGLite's. Exit: docs/designs/rust-engine/bakeoff.md names the choices with data, and they meet the performance targets and the D17 bar below on those machines, or bakeoff.md lists each loss with its planned fix for Checkpoint C1. bakeoff.md compares only SQL-free designs: every record-store option is judged as a key-value store behind the typed methods, on correctness burden, migration and maintenance as well as speed (D18). The idiomatic-SQLite slice the review record mentions was removed by D18. LanceDB is judged with its binary size counted. The simplest option that meets the targets and the server-ready criteria wins; an own HNSW only with a measured advantage.

Phase 3: parity oracle. Entry: Phase 1 landed; runs beside Phase 2. Work: the in-tree harnesses (src/commands/eval-longmemeval.ts and src/eval/) take any BrainEngine instead of PGLiteEngine, which makes them the primary oracle with no driver protocol. In gbrain-evals, under that repository's terms (D11), its maintainers write and land the changes: one helper over gbrain's exported engine-factory, used only in the eval-gate runners (frozen and counted files stay untouched); setup-only SQL moves to typed calls where gbrain has them, while SQL that independently checks what gbrain stored moves to typed methods or the relation API too, as D18 requires; until a file moves, its SQL checks report not_applicable on the Rust arm and no gate that depends on them passes for Rust; one neutral embedding fixture keyed (model, dims, side, sha256(text)); and a LongMemEval warm-cache rebuild (about $2) only after the current LongMemEval work there is done, with a decision line and a cost cap, never committed. research/11 maps an older gbrain-evals commit; the maintainers' count on current main is 91 engine-construction sites in 83 files and 184 raw-SQL calls in 67 files, the three runners research/11 calls broken work, and the gbrain dependency is already pinned. gbrain's in-tree harness is the main parity check; gbrain-evals confirms it. Exit: two PGLite runs at the pinned release give identical results; the PGLite and Postgres baseline numbers are recorded (D17).

Phase 4: engine seam. Entry: Phase 1 landed; D1, D2, D3 and D6 approved. Work: PRs 4 to 9. Exit: a Rust brain constructs, reports its capabilities and refuses everything else with typed errors; the Rust test arm runs; the raw-SQL and kind-branch ratchets run in verify; no change for any existing user.

Checkpoint C1: continue, narrow or stop. Entry: Phases 2, 3 and 4 done. Inputs: bakeoff.md; a real-brain read benchmark (the maintainer exports their own brain with rust/scripts/export-corpus.ts and runs the bake-off harness against PGLite and Postgres on the same brain, whatever engine the brain uses; only aggregate numbers are published and no user is switched); a replay against an engine the brain does not use is labeled snapshot replay, not live shadow evidence; the Track 2 size per subsystem re-counted at that commit; effort spent against the Phase 1 budget. Outcomes, decided by the maintainer and recorded in bakeoff.md: continue (the rest of Track 2 starts); narrow (keep the ratchets, the capability set and the typed methods already landed, and pause Rust domain work); or stop (one PR removes rust/, the Rust test arm and the rust kind, and keeps the ratchets and typed methods, which help the existing engines).

Phase 5: core storage domains. Entry: Phases 2 and 4 done; Checkpoint C1 decided continue. Work: PRs 10 to 14b implement the 15 migrated engine-sql domains, config, and the trigger equivalents. Exit: the domain tests, test/e2e/engine-parity.test.ts cases for those domains and the trigger parity matrix pass on the Rust arm; a small PGLite to Rust to PGLite roundtrip through the graduation adapter verifies exactly on the domains implemented so far, including consumed but unused sequence values, generated columns, withdrawals and database-only content.

Phase 6: search parity. Entry: Phase 5 done; Track 2 search arms landed; the Phase 3 exit met, including the gbrain-evals changes landed and pinned (D11). Work: PRs 15 to 17. Exit: search tests pass on the Rust arm; eval gate E1; performance targets at 10k and 200k met, 1M reported.

Phase 7: write path and owner model. Entry: Track 2 persistence, page-state and facts subsystems landed. Work: PRs 18 to 20 plus the Rust side of those Track 2 subsystems. Exit: put_page, remember, forget, sync and the minimum jobs (sync, embed, extract, backlinks) run through the persistence coordinator on Rust; the crash robot passes; memory-verb conformance 24/24; eval gate E2.

Phase 8: preview. Entry: capability coverage of the core loop (ingest, sync, search, query, the seven memory verbs, the minimum jobs, doctor) is complete; gbrain backup create and backup restore work on Rust brains (PR 21b) and a full-store restore drill passes on both preview targets; the addon upgrade drill passes (stage, verify, switch, keep the previous addon; an upgrade with no network leaves the brain working or refusing with engine_unavailable, with no data touched); paired runs with confidence intervals show the Rust engine equal or better than Postgres and PGLite on every metric in "Performance" (D17); D4, D8 and D12 approved. Work: PRs 21, 21b, 22 and 23: migrate --to rust and back, backup and restore, store format and doctor checks, released addons with a staged upgrade. Exit: preview users can move a brain to Rust and back with verification; eval gate E3.

Phase S: shadow mode. Entry: Phase 6 done (the follower needs importRelation and the graduation adapter from Phases 5 and 14b; read comparisons need search); D13 approved. Work: PRs S1 to S4, delivered so evidence arrives early: after S2 the supported read and search comparisons run with a provisional Markdown and JSON report, before write comparisons and full reconcile exist. Every partial report says NOT READY. Operations Rust cannot answer yet count as capability_missing, which makes the remaining Track 2 work visible on the real workload. The maintainer runs shadow mode on their own brain from S2 on; its reports feed the preview decision (D8) and PGLite deprecation criterion 9. Exit: 14 days of shadowing on the maintainer's brain with zero privacy divergences and a report that lists every remaining blocker.

Phase S-swap: swap and the way back. Entry: Phase 8 entry criteria hold (backup and restore, staged addon upgrade); D14 and D15 approved. Work: PR S5. Exit: the swap, failback and recovery-point drills pass on both preview targets for a PGLite brain and a single-writer Postgres brain; the maintainer's brain reaches READY and is swapped with a reverse shadow kept for the recommended period. Shadow mode then becomes the recommended route into the preview, with migrate --to rust kept for PGLite users who choose the direct path.

Phase 9: full parity. Work: the remaining Track 2 subsystems and their Rust side; more platforms (PR 24). Exit: every PGLite deprecation entry criterion below holds; eval gate E4.

Phase 10: PGLite deprecation. A separate PR after the maintainer's yes (D9): the gbrain init default changes, a deprecations.d/ record and notices ship, and PGLite stays loadable for migrate --to rust for a long, stated window.

Postgres deprecation: deferred. It comes after the multi-user server phases below, as its own decision with its own plan covering hosted and multi-host brains, pooled concurrency and remote clients. Nothing in this plan removes or degrades the Postgres engine.

Path to a multi-user server

The target end state (D19) is a full multi-user Rust server that supersedes Postgres and pgvector and is 10x better than them (D17). Phases 1 to 10 build the embedded engine. The phases below come after the opt-in swap (Phase S-swap). This plan approves none of them: each starts only after the maintainer's yes, recorded in this document, and each exit measures the D17 bar against Postgres in the same configuration. Until Phase M3 passes that bar, brains with writers on more than one machine stay on Postgres.

Phase M1: server mode and wire protocol. Entry: the maintainer's yes; Phase S-swap done; the server-ready criteria in "Process model" hold. Work: a server mode in which one Rust engine process serves a brain to many clients; a wire protocol that carries the typed BrainEngine methods, so the MCP and CLI contract stays the same (operation names and schemas, the error envelope, write receipts and exit 10, the memory verbs). Exit: the engine tests and memory-verb conformance pass through the wire protocol; CLI, JSON and MCP goldens are identical against an embedded brain and a served brain; the latency the wire adds is measured and listed.

Phase M2: authentication and per-user access. Entry: M1 done; the maintainer's yes. Work: per-user credentials on the server; the access checks in gbrain-domain (the typed write-scope argument, source scoping and the invariants I1 to I15) enforced per user and per connection, as the replacement for Postgres RLS. Exit: the per-operation authorization tests pass for each user; Rust equivalents of the RLS-role tests pass; gbrain-evals Cat 22 has zero leaks.

Phase M3: multi-process and multi-machine writers. Entry: M2 done; the maintainer's yes. Work: writers on several hosts write through the server; claim and lease exclusivity, fencing tokens and per-entity fact ordering hold across connections and hosts; multiple_writer_hosts no longer refuses a brain the Rust server serves. Exit: concurrent write throughput at 1, 4, 16 and 64 writers from several hosts meets the D17 bar against Postgres with the same clients; the crash robot passes with writers on more than one machine; a multi-host Postgres brain moves to a served Rust brain through shadow mode and back.

Phase M4: replication, high availability and backups. Entry: M3 done; the maintainer's yes. Work: a follower that consumes the store's commit log or snapshot stream; failover with fencing, so only one primary writes; online backup and point-in-time restore. Exit: failover and restore drills pass with no committed write lost outside the stated loss interval; backup and restore cover everything Postgres backups cover on the same brain.

Phase M5: Postgres deprecation, a separate decision. Entry: M1 to M4 done, and the server meets the D17 bar on every axis, hosted and multi-host brains included. Work: its own plan and the maintainer's separate yes. Exit: the maintainer's decision on that plan is recorded in this document; this plan authorizes no deprecation. Nothing before then removes or degrades the Postgres engine.

These server phases constrain choices in Phases 1 to 8:

  • Store bake-off (Phase 2): concurrent writers are required; transactions and snapshots must work across connections and processes; no option may depend on one owner process; the format must be able to carry a commit log or snapshot stream.
  • Transaction API (the interface in PR 8, Phase 4; its implementation in Phase 5): a transaction is bound to a connection handle, not to a thread or a process; transaction(fn) holds no state that a second connection could not hold; the writer permit comes from a bounded pool.
  • Access layer (Phases 5 and 7): the typed write-scope argument and the I1 to I15 checks live in gbrain-domain, never in gbrain-napi or the TypeScript wrapper, so a server enforces the same checks per user.
  • Store format (PR 19): the format note describes the replication and backup hooks; they are designed, not built.

PGLite deprecation entry criteria

All must hold at one pinned commit, on every platform where PGLite would be deprecated:

  1. Capability coverage is complete: check:rust-capability-coverage reports no operation, command, phase or handler that a PGLite brain can run and a Rust brain refuses (Postgres-only administration excluded by name).
  2. The raw-SQL ratchet shows zero call sites reachable from those entries.
  3. Every file in the testBackends() set has a Rust arm listed in test/rust-unit-arms.txt and passing, except rows in a reviewed test/rust-expected-divergence.tsv limited to Postgres-dialect tests (SQL-text goldens, prepared-statement and RLS-role tests), each with a reason.
  4. CLI, agent-contract, catalog, decide and trust goldens are identical on a Rust brain; doctor goldens differ only in the listed engine-specific checks.
  5. Eval gates E1 to E4 pass.
  6. The Rust engine is no slower than PGLite on any scripts/scale/budgets.json entry, meets the targets below at 200k and 1M chunks, and meets the D17 bar: equal or better than Postgres and PGLite on every metric in "Performance".
  7. migrate --to rust and back verify exactly on the engine-graduation fixtures (1k, 10k, the embedding variants and the hand-made legacy brain) and preserve every database-only category.
  8. The crash robot and the concurrency suite pass on the Rust arm.
  9. At least four weeks of daily use on real brains with no data loss, no privacy difference and no open P1 bug against the engine. Dates, brain size bands and incidents are recorded in docs/designs/rust-engine/preview-log.md. Shadow reports from at least three real brains (the maintainer's included) reach READY on the deprecation commit, with their aggregates in docs/designs/rust-engine/shadow-log.md.
  10. Prebuilt addons exist for each platform being deprecated.
  11. Every PGLite-only user surface has a Rust path or a stated replacement: backup create and backup restore (src/core/backup/snapshot.ts refuses non-PGLite brains today), agent-install setup (src/core/agent-install/setup.ts), init --pglite, pglite-repair and reinit-pglite, and the eval harnesses that construct PGLiteEngine.
  12. An upgrade with no network access leaves every Rust brain either working on its pinned addon or refusing with engine_unavailable and the fix command, with no data touched; proven by a drill on each platform.

PR sequence

Every PR targets master, carries the rust-engine label and a changelog.d/ fragment, gets full-ci when its head is final, and lands through a queue: comment. "Heads-up" means a heads-up: comment before the change, kept in its own small PR. "Yes" names the decision whose quote goes in the PR body. No PR before Phase 8 changes anything for an existing user, so none adds a behavior-changes.d/ entry until then.

# Plain title Main paths Heads-up Yes
1 Rust toolchain and CI lane for the engine (alone, first) rust/ workspace with a trivial gbrain-napi crate, rust-toolchain.toml, Cargo.lock, deny.toml; rust/README.md contributor quickstart (rustup is the only prerequisite; bun run rust:build builds the addon and prints its path and the GBRAIN_RUST_ENGINE_ADDON line; one GBRAIN_TEST_BACKEND=rust bun test <file> command; how to unset the override; the measured first-build time); a rust:build script entry in package.json; .github/workflows/rust-engine.yml (path-filtered to rust/**, src/core/rust-engine*, src/core/engine.ts, src/core/engine-*.ts, src/core/engine-sql/**, src/schema.sql, src/core/migrate.ts, src/core/schema-migrations/**, src/core/persistence/graduation-*.ts, test/helpers/test-backends.ts, test/rust-unit-arms.txt, scripts/rust-unit-arms.ts and the workflow itself; fmt, clippy, tests, cargo-deny, addon build on linux-x64 and darwin-arm64, Bun load smoke including a compiled executable) .github/ (full review; CI time stated) none
2 Bake-off harness rust/bench/, rust/scripts/export-corpus.ts none none
3 Bake-off report docs/designs/rust-engine/bakeoff.md none none
4 In-tree eval harnesses accept any engine src/commands/eval-longmemeval.ts, src/eval/** (types widened from PGLiteEngine to BrainEngine) none none
5 Engine capability set, reported in full by both engines src/core/engine.ts (capabilities), both engines, src/core/engine-capabilities.ts engine.ts none
6 Raw-SQL and engine-kind ratchets scripts/check-engine-raw-sql-ratchet.ts, scripts/engine-raw-sql-baseline.tsv, scripts/check-engine-kind-branches.ts and baseline, package.json script entries, verify scripts/ D6
7 Relation API on the Postgres-dialect engines src/core/engine-sql/relations.ts, engine.ts engine.ts, engine-sql/ none
8 Rust engine shell, refusing everything with typed errors src/core/rust-engine.ts, src/core/rust-engine/ (including members.ts), engine-construct.ts, kind union and every other two-engine union, inventory engines.rust type, config value, docs/ENGINES.md "Adding a new engine" note that a non-SQL engine implements typed methods instead of a SqlExecutor, openEngine kind, --engine on in-tree eval commands, error registry, doctor check rust_engine_available (not_applicable unless selected) engine.ts, doctor goldens D1, D2, D3
9 Rust test arm test/helpers/test-backends.ts (GBRAIN_TEST_BACKEND=rust selects the arm for files in test/rust-unit-arms.txt; never implicit), scripts/rust-unit-arms.ts, an exhaustive engine-for-backend test helper and the Rust-kind assertion, rust-engine.yml job, .github/workflows/full-ci.yml dispatching rust-engine.yml only when the same path filter matches (estimate: about 10 minutes per OS for a cold addon build and 3 minutes with the cache, on two OSes in parallel; PR 1 reports the measured time), a trigger-selection check test harness, .github/ none
10 Rust store core: sources, config, pages, tags, generation and revision rules rust/crates/gbrain-store, gbrain-domain, rust/catalog/, wrapper methods none none
11 Chunks, stored vectors, exact vector scan, files, code edges rust/crates/*, wrapper none none
12 Links, timeline and graph walks gbrain-graph, gbrain-domain none none
13 Facts and takes storage with withdrawal and purge guards gbrain-domain, rust/fixtures/ none none
14 Segmented page bodies, salience, health, titles gbrain-domain none none
14b Engine-neutral graduation adapter and an early PGLite to Rust to PGLite roundtrip on the domains so far src/core/persistence/graduation-*.ts (catalog inspection, canonical rows, counters, fencing and verification through the adapter), src/core/rust-engine/ persistence/ none
15 Keyword, title and CJK search on Rust gbrain-text none D5
16 Filtered vector search, facts arm, cosine rescore on Rust gbrain-vector none none
17 Trigram matching: find, resolver, saved facts on Rust gbrain-trgm none none
T1 to T14 "Move SQL behind engine methods" (Track 2, one subsystem each, in the order listed above), each followed by an R-PR implementing it on Rust and adding test files to the Rust arm src/core/engine-sql/<domain>.ts, the subsystem's files, its tests engine.ts, engine-sql/, baseline rows none
18 Single-owner capability replaces PGLite kind checks in lock and IPC paths; a second process becomes an owner client for reads and writes, never pglite_busy (D20); the owner protocol extension in "Process model" src/core/pglite-lock.ts callers, resolve-ipc.ts, persistence/ipc.ts, persistence/local-client.ts, consumer election none none
19 Store format version, schema migration dispatch for Rust stores, index lag and rebuild checks gbrain-store, src/core/migrate.ts and src/core/schema-migrations/types.ts (Rust disposition per migration), doctor checks doctor goldens, migrate.ts D2, D7
20 Crash robot, fuzz and scale cells for Rust scripts/persistence/, scale-tier.yml, rust-engine.yml scripts/, .github/ none
21 gbrain migrate --to rust and back, PGLite brains only the graduation adapter from PR 14b on both directions, inventory engines.rust rows, src/commands/migrate-engine.ts none D1
21b gbrain backup create and backup restore for Rust brains, with a full-store restore drill src/core/backup/ (a Rust snapshot taken under the owner lock at one committed generation, with derived indexes carried with their generation or rebuilt; the PGLite snapshot path unchanged), src/commands/backup.ts none D1
22 Released, attested engine addons with a pinned digest manifest, gbrain engine install rust (with --from <file> for offline import) and a staged upgrade release.yml, src/cli/commands/engine.ts (the engine dispatcher, which routes only status today), a new src/commands/engine-install.ts, src/core/binary-self-update.ts staging, post-upgrade refresh, engine status fields .github/ (full review, install path) D4
23 Rust engine preview documentation docs/ENGINES.md (new section with a tested user quickstart: check the target, gbrain backup create, gbrain engine install rust, gbrain migrate --to rust --plan, then --yes --expect <hash>, one search, gbrain engine status, and the way back; expected output for each step; the capability list; a RustEngine column in the capability matrix), docs/guides/troubleshooting.md symptom rows for each new code and reason, INSTALL_FOR_AGENTS.md (an agent never moves a brain to Rust unless the user asks), README.md limits, how to report a preview problem (a GitHub issue titled rust engine: with gbrain engine status --json and gbrain doctor --json attached; no telemetry) none D8
S1 Shadow mode core: supervised shadow child process, transaction-aware engine wrapper, mirror queue, metrics ledger and evidence manifest (off unless started) src/core/shadow/ (wrapper built with registerEngineView and extended to transaction-bound engines and group publication, invalidation collection with savepoint discard, bounded fail-open IPC, child supervisor, ledger writer), src/core/engine-factory.ts (wraps only when shadow is on), config keys engine-factory.ts D13
S2 Read comparisons: pipeline observer, captured caller context and provider outputs, shadow execution mode without side effects, revision- and generation-aware classification, per-item privacy check, provisional report src/core/shadow/compare*.ts, src/core/shadow/observer.ts (hooks in src/core/search/hybrid/arms.ts and the operation dispatcher), result normalizers, replay buffer, rust/crates/gbrain-napi read-only entry points src/core/search/ (heads-up) D13
S3 Seed, follow and reconcile, with the Postgres read-only source side and the live-shadow policy of the graduation adapter, allocator manifests, and rolled-back write comparisons on a copy-on-write snapshot src/core/persistence/graduation-*.ts (one transaction-bound REPEATABLE READ export handle with a snapshot-age budget, shared-bound key intervals, migration/live-shadow/reverse-shadow policies, comparison inventory), src/core/shadow/reconcile.ts, src/core/shadow/write-compare.ts persistence/ (full review: stored data) D13
S4 gbrain engine shadow commands, the generated report and charts, doctor check rust_shadow, and the shadow quickstart src/cli/commands/engine.ts, src/commands/engine-shadow.ts, src/core/shadow/report*.ts (decision brief, markdown, JSON, SVG), bench, daily rollup as an autopilot cycle step, doctor check and goldens; docs/ENGINES.md shadow quickstart with expected output, time to first report, resource and privacy notice, stop and delete; troubleshooting rows; INSTALL_FOR_AGENTS.md rule (agents read status and reports freely; starting a shadow needs the user's request) doctor goldens D13, D14
S5 gbrain engine swap with durable writer observations, the pre-swap recovery checkpoint, the write fence, route verification, the reverse shadow, failback and recovery points, and the swap docs src/commands/engine-swap.ts (persisted states, --status, --resume), writer-observation table and writers, the graduation adapter's Rust to Postgres and Rust to PGLite directions under reverse-shadow, error registry rows, behavior-changes.d/, docs/ENGINES.md swap and failback guide schema and migrations (writer observations), persistence/, engine-graduation.ts (full review: stored data and auth) D14, D15
24 More platforms: linux-arm64, darwin-x64, musl, Windows rust-engine.yml, release.yml .github/ none
25 PGLite deprecation (later) gbrain init default, deprecations.d/, notices, behavior-changes.d/ config defaults D9

S1 to S4 start after PR 17, with S2's provisional report first; S5 waits for PRs 21b and 22. PRs 2 to 4 can run in parallel after PR 1. T-PRs for the search arms and the 24 shared baseline families start after PR 6; the other T-PRs start after Checkpoint C1. Both run beside PRs 10 to 17.

Testing and proof

Engine test arm

  • testBackends() (test/helpers/test-backends.ts:15) gains a rust backend selected only by GBRAIN_TEST_BACKEND=rust and only for files listed in test/rust-unit-arms.txt. Default developer runs are unchanged and need no Rust toolchain. Selecting rust without a loadable addon fails loudly, as requirePostgresTestDatabase does for Postgres.
  • Most test files choose their engine with a two-way branch such as backend === 'postgres' ? requirePostgresTestDatabase() : undefined, which would build a PGLite brain under a [rust] label and pass. So the arm is selected by a runner, scripts/rust-unit-arms.ts, modeled on scripts/postgres-unit-arms.ts, which runs only the listed files with GBRAIN_TEST_BACKEND=rust; an exhaustive engine-for-backend helper replaces the two-way branch in a file before it is listed; and every engine opened under the Rust arm must report kind === 'rust', or the file fails. A check refuses a listed file that still has a two-way backend branch.
  • The Rust CI lane builds the addon once and runs the listed files, sharded like the unit lane.
  • Divergences live in test/rust-expected-divergence.tsv and run as test.failing, the EXPECTED_FAILURES pattern of test/helpers/two-engine.ts:308, so an unexpected pass fails.
  • Targets beyond the unit files: test/e2e/engine-parity.test.ts with a Rust half; the two-engine isolation suite; test/memory-verbs-conformance.test.ts; the 81 black-box CLI files (research/10 section 7) run against a temporary Rust brain.

Goldens

  • Engine-neutral goldens (cli, catalog, decide, trust, agent-contract fixtures) must be identical on a Rust brain.
  • Doctor goldens: identical except engine-specific checks, listed in the PR that adds them.
  • test/fixtures/goldens/sql-text/ and pglite-upgrade-replay are Postgres-dialect artifacts and do not run on Rust.
  • Rust-only exactness fixtures (trigger results, ts_rank, pg_trgm, fingerprints, jsonb order, collation) live in rust/fixtures/, generated from PGLite by rust/scripts/, so they do not churn shared golden files.

Eval gates

Rules: TypeScript on PGLite, TypeScript on Postgres and Rust run on the same machine, the same fixture and one pinned gbrain commit; thresholds are written before any Rust number exists and are not changed after; published numbers (LongMemEval from v0.28.8) are sanity checks only.

Gate Phase Benchmarks Pass
E1 6 In-tree LongMemEval _s retrieval only (500 questions, top-5, embedding fixture): keyword, vector, hybrid. gbrain-evals Cat 13, Cat 13b, Cat 10, Cat 12, Cat 7 R@5 per adapter at least both the PGLite and the Postgres value, per-question top-5 differences listed; Cat 13 nDCG@5 and Cat 13b top-1 at least both values; Cat 10 133/133; Cat 12 50/50; Cat 7 reported
E2 7 BrainBench Cats 1+2 (graph on), Cat 3, Cat 4, type-accuracy (after its call-site fix), Cat 24, memory-verb conformance P@5 and R@5 at least both the PGLite and the Postgres value; Cat 4 100%; Cat 24 5/5; conformance 24/24
E3 8 Cat 22, Cat 19, Cat 27, Cat 23 (scored against PGLite's pg_trgm behavior) Cat 22 zero leaks; others within 0.5 points or equal counts
E4 9 All of the above at the final pinned commit; minion durability and throughput (test/e2e/bench-vs-openclaw/); LLM-judged cats reported in bands set beforehand all prior thresholds hold

The E1 and E2 retrieval thresholds are "equal or better", not "within N points", because a better ranker or a higher-recall vector index returns different results (D17). With the ts_rank compatibility ranker (D5), the keyword arm must also reproduce PGLite's fixture ordering exactly; that mode stays the regression baseline. E3 is an intentional exception to the D17 change: its thresholds stay as written, and Cat 22 stays at zero leaks.

Performance

  • Scales: 10k, 200k and 1M chunks with real 1,024- and 1,536-dimension vectors; filters on source, type and date at 100, 50, 10, 1 and 0.1 percent selectivity.
  • Bar (D17): the Rust engine must be equal or better than TypeScript gbrain on Postgres (pgvector HNSW, local, same machine, same corpus, same run) and on PGLite, on every metric below. The absolute targets further down are floors, not the bar.
  • Target (D17): 10x better than Postgres and pgvector, the maintainer's goal. The gate stays equal or better; the gbrain-evals preregistration fixes the margin on each axis, and this plan sets no per-axis 10x number.
  • Metrics, each a named step in rust/bench/ and a card in the gbrain-evals engine-bench (D16):
    • concurrent read throughput at 1, 4, 16 and 64 clients;
    • concurrent write throughput at 1, 4, 16 and 64 writers: committed writes per second through transaction(fn), outbox apply included;
    • p50 and p99 for get_page, put_page, keyword, vector, hybrid and filtered search, single client and at fixed concurrency; end-to-end query with a precomputed query embedding (no expansion or rerank) and separately with them;
    • graph traversal (traverse_graph, backlinks, rank_entities) at depth 1 to 5 against Postgres WITH RECURSIVE;
    • vector search recall@10 against exact, and p50 and p99 against pgvector HNSW, at 200k and 1M chunks, 1,024 and 1,536 dimensions, filter selectivities 100 to 0.1 percent;
    • peak and steady RSS, and RSS per 100k chunks;
    • cold open to first get;
    • import and sync throughput, index build time and disk size.
  • Gate: a Rust arm is not called ready for the opt-in swap (D14 criterion 8) or the preview (D8) unless it is equal or better than Postgres and PGLite on every metric, from paired runs with confidence intervals. A loss on any axis is listed in bakeoff.md and in the shadow report with its fix, never hidden.
  • Proposed targets, confirmed or corrected by the bake-off before Phase 5: vector arm p50 at most 20 ms and p99 at most 100 ms at 200k chunks with or without filters; no filtered query more than 2x its unfiltered time; cold engine open at most 50 ms; committed writes at least 3x PGLite; a bounded-memory build at 1M chunks (budget set by the bake-off). These are floors; the D17 bar applies on top of them.
  • Each target has a named step in rust/bench/ that prints machine, engine, data size, N, p50 and p99, with PGLite and Postgres measured on the same machine in the same run. Write throughput is measured through transaction(fn) with the writer held across awaits and includes the outbox apply before return. Peak and steady resident memory at 200k and 1M chunks is reported against the memory envelope and the per-100k-chunk budget in bakeoff.md, beside Postgres RSS for the same corpus (f32 vectors at 1M chunks and 1,024 dimensions are about 3.8 GiB raw before any graph).
  • A Rust cell joins scale-tier.yml. Benchmarks report; CI gates on relative regressions.

Crash, property, concurrency and fuzz tests

  • Crash: SIGKILL between record commit and index apply, during import, during migration copy and during index rebuild; the open recovers, index lag returns to zero, and a deleted or withdrawn item is never returned.
  • Failure after commit without a crash: inject a failure after the record commit, after the text apply and before the vector apply while the owner stays alive; the receipt says committed, later reads are correct, replay brings the lag to zero, and a retried write is idempotent.
  • Snapshot isolation: pause a search across a concurrent content and permission update; the paused reader returns correct results for its snapshot and never newer restricted content.
  • Typed concurrency operations (D20): each operation in "Concurrency without PGLite-style locks" runs with concurrent callers on PGLite, Postgres and Rust and gives the same observable outcome: no lost update, no double claim, no deadlock. A stress cell runs 64 writers across 8 processes on one brain; a second process opening the brain never gets a busy error. The TLA+ properties in spec/tla/ stay as tests.
  • Writer admission: start far more concurrent writers than worker threads while the first transaction awaits and then makes another engine call; all finish or time out within the bound, a cancelled waiter leaves nothing behind, re-entry gives transaction_reentry, and disconnect() finishes cleanly.
  • Fingerprints: create withdrawals and purges before a change, then insert equivalent non-ASCII and mixed-case claims on Rust after migration; they stay suppressed.
  • The crash robot (scripts/persistence/README.md: 1,000 seeded schedules, eight SIGKILL boundaries, 10,000 writes from four producers) gets a Rust cell.
  • Property tests (proptest): record encoding round trips, outbox idempotence, filtered search equal to exact search over the same bitmap, trigger hooks equal to the fixture.
  • Concurrency with pause points: a page update against an in-flight embedding job in both completion orders; the refresh-barrier safety properties (spec/tla/refresh-barrier/) as executable tests.
  • Fuzz (cargo-fuzz): the tsquery parser, the analyzer, row decoding and every binding entry point with malformed arguments; none may crash the process.
  • cargo deny and cargo audit in the Rust lane.
  • Drills on darwin-arm64 and linux-x64-glibc before the preview: full-store backup create and backup restore into an empty home with per-relation digests equal; addon upgrade online, offline and with an interrupted fetch, each ending with the brain working or refusing with engine_unavailable and no data touched.
  • Time-to-first-query drill on both preview targets, in a clean home with the 10k-page engine-graduation fixture: from gbrain engine install rust to the first search answered on Rust, following the PR 23 quickstart verbatim. Target: 5 minutes. The measured time is written in the preview docs. The PR 1 contributor quickstart is timed the same way (first build reported; edit, build and test loop under 1 minute).
  • Failure drills: a corrupt derived index and a failed open-time replay are repaired by doctor --fix through the maintenance path; a newer FORMAT gives newer_format with no write; an injected panic inside a write transaction returns internal_error with a diagnostic id, rolls back, and the next call succeeds; each engine install failure state leaves the active addon unchanged.
  • Error rows: one test per code and reason checks the registry row, the rendered fix, the detail context, and that no environment variable value and no absolute home path appears in the envelope or the diagnostic file.

Shadow mode tests

  • Off path: with shadow mode off, the engine factory returns the bare engine (kind and identity checked), no shadow module is imported on the hot path, and the persistence throughput harness shows no change against master.
  • Isolation: inject a slow, failing, panicking, aborting and hanging shadow child and a full IPC queue; every primary call returns the same result, nothing awaits the shadow, the child restarts, and the fault is recorded.
  • Commit capture: outer rollback, nested savepoint rollback, group publication and a committed write with a failed receipt each publish exactly the committed invalidations.
  • Reconcile: deletion, key change, insertion at an interval bound, an empty source relation and a delayed older batch arriving after a newer update all end with equal digests and no newer state overwritten.
  • Allocators: rollback-consumed ids, a deleted highest id and the first insert after each swap direction.
  • Classification: an index-only change, a membership change and an injected wrong Rust revision; a real Rust bug is never classified stale, and a repair never turns a mismatch into a match.
  • Privacy: an authorized item that moves into Rust's top 10 is a ranking difference; a wrong visibility rule in the shadow gives privacy_diverged on the first event, pauses read mirroring and raises the doctor error. Remote member and folder scope are covered.
  • No side effects: a mirrored search with cache hits and several vector arms makes zero provider calls, writes no telemetry, cache or retrieval timestamp, and nothing flushes later.
  • Write compare: create, conditional update, append, delete, no-op and multi-row writes; write_uncomparable when no equal preimage exists; the shadow's logical state is unchanged afterwards.
  • Policies: active jobs and running effects through seed, reconcile and both swap directions; no job or effect runs twice and no worker starts in a shadow store.
  • Overflow and missed writes: fill the queue past its bound; raw SQL, a second process and a second host write while shadow runs; reconcile repairs them and reports counts by relation.
  • Fence and eligibility: a stopped second host, two processes on one host, an already-open transaction and an old client writing after cutover are all refused or reported; unknown writer coverage refuses the swap.
  • Route: an environment DATABASE_URL override and an old serving process block the swap before authority changes; the probe confirms the effective engine after the switch.
  • Swap faults: kill, disk full and a failed digest at each persisted state; --resume continues, authority never points at an unverified engine, and the old primary is unchanged before the fence.
  • Failback and recovery drills on both preview targets for a PGLite brain and a single-writer Postgres brain: swap, write, fail back with every acknowledged write kept; make Rust unreadable and restore a recovery point with the loss interval stated; copy corruption into the reverse shadow and check that reverse following freezes.
  • Evidence: a change of Rust build or comparator starts a new epoch and resets readiness; an errored or stale attempt never counts as coverage; missing evidence gives NOT READY.
  • Report: the ledger and report contain no slug, title, query text or page content (a canary string scan); an expired explain sample is refused, never replaced.

Migration and rollback

  • gbrain migrate --to rust reuses the engine graduation machinery (src/commands/migrate-graduation.ts, src/core/persistence/graduation-*.ts), which already copies PGLite to Postgres from a deny-by-default inventory with carry, rebind, rebuild and discard classes (docs/designs/engine-graduation-spike.md).
  • The inventory gains an engines.rust column. A migration that adds a relation already fails test/graduation-inventory.test.ts until it has a row; after the preview it must also say how Rust stores it (decision D7).
  • Schema migrations on Rust stores. Each Migration (src/core/schema-migrations/types.ts) gains a Rust disposition: catalog_only (an additive change the catalog absorbs), postgres_only (an explicit no-op with its reason), or a Rust handler with its own verify through typed methods or the relation API. The runner in src/core/migrate.ts dispatches on it, since the SQL path and SQL-issuing handlers cannot run on Rust. Per migration, the catalog change, the data handler and the version ledger row commit in one store transaction; FORMAT is then rewritten by atomic rename, and an open that finds FORMAT behind the records repairs it from the records. A check fails a migration above the Rust baseline that has no disposition. Before PR 19, a catalog mismatch is refused by the loader and development stores are rebuilt.
  • The graduation helpers issue SQL on both sides today (catalog columns in graduation-digest.ts, sequence positions in graduation-copy.ts, metadata, counts and sequences in graduation-verify.ts). An engine-neutral graduation adapter keeps the custody state machine and replaces those assumptions: catalog inspection, canonical row export, counter and sequence transfer (never below a consumed value), fencing, import and verification, implemented for PGLite and Rust (PR 14b).
  • The preview moves PGLite brains to Rust and back. The target side writes through the Rust importRelation batch API in primary-key keyset order. Derived indexes build after the copy; embeddings are carried, never re-embedded. Postgres brains move only through shadow mode and gbrain engine swap (section "Shadow mode and swap"), which requires a single writer host and measures fingerprint agreement before the swap; migrate --to rust from Postgres stays deferred and refuses with a message that the brain stays on Postgres. The swap and the preview refuse a brain with writers on more than one machine, with a clear message that it stays on Postgres until the Rust server phase passes the D17 bar (D19).
  • Same safety as graduation: writers quiesced and the journal drained first; a plan hash confirmed with --yes --expect <hash>; refused while a live gbrain serve holds the source lock; refused before touching anything when the addon is unavailable.
  • migrate --to rust, and migrate --to pglite from a Rust brain, have the graduation modes and usage line: --plan (alias --dry-run, read-only), --yes --expect <hash>, --status, --resume, --rollback-to-source, and --json with progress on stderr. The plan lists the addon status and its install step, the disk space needed, the estimated copy time, blockers with their fix (for example a live gbrain serve and how to stop it), and the capability report: each operation, command, cycle phase and minion handler the brain would lose on Rust, with its reason.
  • The command never prompts. A missing addon is installed by the run only as a step listed in the plan. A caller without --yes --expect <hash> gets the plan and exit 3 with the fix.
  • The run report ends with one filtered vector search timed on the source before the copy and on the target after verification. It uses a stored chunk embedding as the query vector, so no provider is called, and prints both times and the result count only.
  • Verification: counts plus a per-relation content digest that both engines compute through relationDigest over a canonical row encoding. Config switches only after verification passes. The source directory is tombstoned, as graduation does, so older binaries refuse to write to it.
  • Preserved without exception: withdrawals and purge tombstones (fingerprints verbatim), belief events (chain verified after copy), tokens and OAuth rows (hashes only), access and policy state, segmented page bodies, page versions, journal and receipts, spend ledgers, calibration profiles, config rows except the engine-local keys.
  • Rollback path 1: if the Rust brain took no writes since migration (its write counter equals the migration mark), the switch back restores config and lifts the tombstone. Rollback path 2: gbrain migrate --to pglite copies the Rust brain back through the same copier and verification.
  • Neither rollback path is disaster recovery: both need a readable Rust store. Recovery from a damaged store uses gbrain backup create and backup restore, extended to Rust brains in PR 21b (today src/core/backup/snapshot.ts refuses non-PGLite brains), and a full-store restore drill is a preview entry criterion. The 8 GiB archive cap (src/core/backup/archive.ts) applies to Rust brains too; above it, the quiesced physical-copy runbook (docs/guides/in-agent-setup.md, "Large or non-standard brains") gains the Rust directory layout, and the restore drill covers one store above the cap by that path.
  • Fault injection at each stage (kill, disk full, blocked relation, corrupt index) checks that the reported target state matches reality and that the source is never modified.

Shadow mode and swap

The safest way to move an existing brain, Postgres or PGLite, is to run Rust beside it first, measure both engines on the same live workload, and swap only when the numbers show Rust is clearly working. Shadow mode is that path. It is the recommended route for every existing brain; migrate --to rust stays as the direct path for PGLite brains.

What shadow mode does

  • The current engine stays the primary. It serves every read and write, and its answers are the only answers callers see. Rust is a shadow: a full local copy of the brain that follows the primary and answers the same reads in the background.
  • Shadow mode is opt-in per brain (gbrain engine shadow start --plan, then --yes --expect <hash>). When it is off, no shadow code is constructed: the engine factory returns the bare engine, and a test proves it.
  • Crash isolation. While another engine is primary, a supervised child process owns the shadow store and loads the Rust addon; the serving process never loads it. Mirror work crosses a bounded, fail-open IPC queue. A child exit, native abort, hang or resource exhaustion disables mirroring and is recorded; the primary keeps serving. The in-process binding stays the design for a Rust primary.
  • No foreground call awaits shadow work, and the shadow never fails a call. The shadow uses CPU, memory and disk on the same machine, so the claim is "no awaited work on the request path", not "zero impact"; the report shows the measured load. When the queue is full, the shadow sheds sampled reads first, then falls behind (reported as lag).
  • Shadow mode does not change the Postgres or PGLite engine code. It has two observation seams above BrainEngine: a transaction-aware engine wrapper for typed calls, and a scoped observer in the operation and search pipeline (src/core/ops/contract.ts OperationContext, the arm calls in src/core/search/hybrid/arms.ts) for the operation, the immutable caller-access context and the final output. The one exception is the swap fence (below), which reuses graduation custody.

How the shadow stays in step

The shadow is an eventual-state copy, not a commit-log replica. Three mechanisms keep it in step:

  1. Seed. A full copy from the primary through the graduation adapter (PR 14b) under its live-shadow policy, read through one transaction-bound export handle inside one REPEATABLE READ read-only transaction on Postgres (one snapshot on PGLite), with a total snapshot-age budget as well as statement timeouts. The primary is not fenced and not tombstoned: a shadow is a copy, not a move.
  2. Follow. The wrapper wraps transaction-bound callback engines too (transaction, transactionDirect, the grouped publication runner in src/core/persistence/group-publish.ts). It collects the relation and key invalidations each write makes, drops those of rolled-back savepoints and transactions, and publishes them only after the outer commit succeeds, including commits whose operation receipt reports failure (coordinator.ts:92). The follower re-reads those keys from the primary and replaces them in Rust through importRelation, so stored ids and fingerprints stay identical. It never replays the call. Queue loss, a restart or an uncertain commit outcome invalidates the follower's coverage watermark until reconcile completes.
  3. Reconcile. Writes the wrapper cannot see (raw SQL, other processes, other hosts, cron) are caught by reconcile. Intervals are defined by shared primary-key bounds (composite keys, empty intervals and the unbounded tail included), never by independently numbered batches. A differing interval is replaced atomically in the shadow, including deletion of shadow-only keys and their dependent state. Snapshot installation is serialized with follower application, and an older generation never overwrites newer state. Reconcile is rate limited, read-only on the primary, runs continuously at low priority with a full pass at least nightly, and reports the rows it repaired by relation; rows the follower should have caught count as follower misses. Comparisons over an interval being repaired are suspended.
  • Allocators. Row imports keep stored ids; sequence and allocator state is copied separately as a manifest, including values consumed by rolled-back transactions (graduation-copy.ts sequence transfer), and verified with the sequence check in graduation-verify.ts.
  • Policies. The graduation adapter gets explicit migration, live-shadow and reverse-shadow policies. The migration-time transforms (jobs reset to waiting, running effects requeued, managed persistence disabled, caches discarded; graduation-inventory.ts) apply only at cutover. A shadow store never starts job, persistence-effect, sync or maintenance workers. A versioned comparison inventory separates carried data, runtime-only state, derived indexes and excluded caches, and every exclusion is reported.

The shadow lives on one host, the shadow host: the machine that would own the brain after a swap. On Postgres brains with several hosts, the others keep writing to Postgres as today and reconcile picks their writes up.

What is compared

  • Reads. A sampled share of reads (shadow.read_sample, default 0.1 [proposed]; 1.0 for brains under 10k pages) is run again on the shadow with the captured caller-access context. Provider work is never repeated: the primary's resolved query vectors, expansion and rerank outputs and ranking settings are captured and passed to the shadow. A shadow execution mode suppresses telemetry buffers, cache writes, retrieval timestamps and feedback before they are scheduled.
  • Classification. Each comparison ends as match, stale, inconclusive, diverged, privacy_diverged, rust_error or capability_missing. stale needs evidence: a captured snapshot or generation showing an intervening primary change or unapplied follower work, for collections, missing rows and non-page relations as well as pages. Unequal revision strings alone are not enough. Without that evidence the result is inconclusive: the original mismatch is kept, and both engines are run again against a current stable generation. A later repair never turns an unexplained mismatch into a match.
  • Deterministic operations (get_page, list, backlinks, timeline, find, stats, facts reads, and the memory verbs except the ranked part of their results) must match exactly, including order. Where a memory verb ranks results (for example recall, which runs hybrid search), its deterministic fields, envelope and visibility filtering match exactly, and its ranked list is compared under "Ranked operations" below.
  • Ranked operations are compared per arm, correlated by request and role (cache hits, degraded paths and several vector arms included); fusion is TypeScript on both. The trigram arm must match exactly. Keyword and title arms must match exactly when Rust runs the ts_rank compatibility ranker (D5); with a better default ranker they report overlap@10 and rank correlation, and their quality is gated on the eval gates instead (D17). The vector arm reports overlap@10 against the primary, which is reported and not gated, and scores both engines' top 10 against the same exact scan; Rust's recall@10 must be at least the primary's (pgvector HNSW on Postgres). The fused top 10 reports overlap@10 and rank correlation.
  • Privacy. Every item in every mirrored Rust response is checked against the primary's read policy for the captured caller at a comparable policy generation (src/core/search/private-visibility.ts), at row and content level. An item the caller may not read is privacy_diverged; an authorized item that is simply outside the primary's top k is a ranking difference. One privacy_diverged event pauses read mirroring, raises a doctor error and resets the readiness window. The check is never sampled within a mirrored response; the report states that unsampled requests are unobserved.
  • Writes. A sampled share of typed writes (shadow.write_sample, default 0.05 [proposed]) runs against an isolated copy-on-write snapshot of the shadow that is validated to equal the primary's preimage, with the captured typed inputs, configuration, clock and randomness. The full mutation set is compared (deletes, dependent rows, allocator behavior, index effects) with explicit mappings for generated ids. When no equal preimage is available the result is write_uncomparable and does not count. The shadow's logical data, allocators and indexes are unchanged afterwards.
  • Fingerprints. Fact rows store no fingerprint: the primary computes it in SQL when asked (src/core/engine-sql/facts.ts:458), and the withdrawal guard matches a ledger row against both gbrain_fact_fingerprint and gbrain_fact_fingerprint_v1 (src/core/facts/withdrawal-schema.ts:73). So for every fact row the follower imports, the follower asks the primary host to compute both values in its own SQL, Rust computes both in its own code, and the two pairs are compared; fingerprint agreement on a Postgres brain is a measured count. That count does not prove that withdrawals and purges keep matching: a purged claim keeps only its text-free hash (src/core/facts/purge-schema.ts), so no stored row shows a difference for it. Two more checks cover that. First, a probe corpus (non-ASCII text, mixed case, every whitespace class) runs through the primary host's own gbrain_fact_fingerprint and gbrain_fact_fingerprint_v1 SQL and through Rust; a difference in either version refuses the swap (engine_swap_refused, reason fingerprint_semantics_differ) and is reported with the host's server version and collation settings. Second, a regression test withdraws and purges a claim on Postgres, removes its original text, swaps, then re-imports the same claim text, which must stay suppressed; the same test covers a legacy withdrawal recorded with only the _v1 fingerprint whose original text is gone.
  • Latency and resources. Observed latencies (primary on the real call, shadow in the background) are reported as observed only and never support a speed claim. Speed claims come from gbrain engine shadow bench: buffered samples replayed on both engines back to back in alternating order on the shadow host. Bench never writes to the primary or to the live shadow store. Read samples replay against the live stores. Write samples (putPage, imports, chunk upserts) replay only against disposable copy-on-write snapshots of both engines, taken at the same generation, with workers, embedding calls, file writes and job enqueues disabled; the snapshots are deleted after the run, and bench refuses write samples when no equal pair of snapshots is available. For Postgres the primary time includes the network, which is the head-to-head a user feels. Peak RSS, store size, index lag, follower lag and the extra load on the primary are sampled every minute.

Evidence, tracking and the generated report

  • Evidence epochs. Each shadow run has a versioned evidence manifest: brain id, primary engine and version, Rust build digest, comparator and adapter versions, catalog, capability set and the approved readiness-policy hash. A change to any of them starts a new epoch; an unrelated release with identical relevant artifacts does not. Readiness counts only evidence from the epoch of the candidate being swapped.
  • Metrics ledger. Every comparison appends one row under <gbrain home>/engines/rust/shadow/metrics/ (one JSONL file per day, mode 0600): an opaque sample id, time, epoch id, operation, both latencies, result counts, classification, reason code and keyed hashes of row ids (HMAC with a per-brain secret). No query text, page content or slugs, so an aggregate report can be shared.
  • Replay buffer. Separately, a bounded in-memory buffer in the owner process keeps the exact typed request, the captured embedding and the caller-access context of recent samples. It never enters the ledger or a report and is discarded on eviction or exit. explain and bench use only buffered samples; an expired sample gives a structured refusal and the command to collect fresh evidence. Durable replay storage is deferred.
  • A daily rollup (an autopilot cycle step on the owner, or report on demand) keeps 90 days of per-operation counts and percentiles.
  • gbrain engine shadow status [--json]: on or off, seed progress, time to first completed comparison, follower lag and coverage watermark, reconcile state, queue depth, sample rates, last privacy event, evidence collected so far, the estimated remaining time at the observed traffic rate, and the verdict.
  • gbrain engine shadow report [--since 14d] [--json] [--out <dir>] writes shadow-report.md, shadow-report.json (with a schema version) and two SVG charts. The report opens with a one-screen decision brief: READY or NOT READY, whether more evidence alone can fix it, the largest measured win, the largest regression, the top blockers and the next read-only command. Then a table of primary versus Rust p50 and p99 from paired bench samples (difference, N, confidence interval), and every D17 loss with its planned fix, then every readiness criterion with its value and threshold, coverage and exclusions per operation (attempted, completed, errored, stale, inconclusive, dropped, capability-missing), retrieval quality, write, fingerprint and privacy results, lag, primary load, RSS, disk and recovery evidence. --since changes the history shown, never the readiness window.
  • gbrain engine shadow explain <sample-id> reruns a buffered diverged sample locally, labels the current-state rerun separately from the original observation, and is the only command that shows content. It is local CLI only: the operation is cliOnly, because localOnly still admits stdio MCP (src/core/ops/callable.ts:30) and stdio callers are untrusted. The handler also refuses unless ctx.remote === false, and tests confirm that stdio MCP, HTTP MCP and the thin client are all refused. It reaches the owner process through the owner IPC socket, which is 0600 and accepts only the same OS user. The rerun uses the caller-access context recorded with the sample, never a wider one, and prints to the terminal only.
  • Doctor check rust_shadow (not_applicable when off) reports lag, a privacy event, a stalled follower or reconcile, a child fault and a full disk.
  • Aggregate numbers only from the maintainer's brain, and from preview users who explicitly export a sanitized report, go into docs/designs/rust-engine/shadow-log.md. Nothing is uploaded automatically.

Swap readiness

READY needs every criterion below, from evidence in the current epoch (thresholds proposed in D14 and fixed before any candidate result is collected):

  1. At least 14 days of shadowing in the epoch; follower lag p99 under 60 s; three consecutive nightly reconcile passes that repaired nothing.
  2. Every operation class used in the window meets its minimum of successful comparisons (at least 10,000 reads in total). An errored, stale, inconclusive or dropped attempt is not successful coverage and never counts as a match. Missing evidence is NOT READY, never an omitted row. None was capability_missing in the last 7 days.
  3. Zero unresolved rust_error and zero unexplained diverged results on deterministic operations in the last 7 days.
  4. The trigram arm agrees exactly on every comparable sample. Keyword and title arms agree exactly when Rust runs the ts_rank compatibility ranker; with a better default ranker (D5), the release carries E1 and E2 results at least equal to both the PGLite and the Postgres values instead. The vector arm's recall@10 against exact is at least 0.98 and at least the primary's recall@10 against the same exact scan. Overlap@10 with the primary, for the vector arm and the fused top 10, is reported and not gated (D17).
  5. Zero unexplained differences on at least 1,000 comparable write comparisons.
  6. Zero privacy_diverged events in the window.
  7. Zero fingerprint mismatches and a passing allocator verification.
  8. In paired bench samples (at least 100 pairs each, with confidence intervals), Rust p50 and p99 are no worse than the primary's on every operation class, and better on filtered vector search by the margin D14 sets. The release's bakeoff.md and engine-bench results show Rust equal or better than both Postgres and PGLite on every metric in "Performance" (D17). Zero samples or an inconclusive result never means faster, and a loss on any axis is listed in the report with its fix.
  9. Zero panics and zero child faults in the shadow.
  10. A backup and restore drill on the shadow store passed; the release carries eval gates E1 to E3.
  11. Eligibility: durable writer observations (below) show one writer host for the last 7 days with no unknown writer path (a brain with writers on more than one machine stays on Postgres until the Rust server passes the D17 bar, D19); a personal brain, not a mounted team brain; the shadow host on a preview target.

There is no readiness override in the preview. A NOT READY plan can be read but not run. Privacy, fingerprints, topology, platform, capabilities, recovery drills, caught-up following and final digest checks can never be waived; any later override for duration or sample counts needs its own approved policy.

Swap and the way back

  • Writer observations. Consumer heartbeats are diagnostics only (rows are purged after 90 seconds and deleted on clean shutdown, consumer-heartbeat.ts). S5 adds durable writer observations for the eligibility window. They are written only while a shadow is enabled for that brain, by every process that writes to it; a brain without a shadow gets the empty table from the migration and no extra write on any path. The eligibility window starts when the shadow starts. A write path with no observation makes eligibility unknown, which refuses the swap. S5 carries a behavior-changes.d row for the migration.
  • gbrain engine swap --to rust --plan is read only. It prints the verdict, eligibility, the capability report, disk space, the expected write pause, every environment variable, launcher and service definition known to select the current engine (names only, never values; src/core/config.ts:738 lets GBRAIN_DATABASE_URL and DATABASE_URL override the file), and a plan hash.
  • gbrain engine swap --to rust --yes --expect <hash> runs these persisted states, each resumable with --resume and visible with --status:
    1. Write an immutable, verified pre-swap recovery checkpoint of the primary.
    2. Establish a durable, database-enforced write fence on the primary through graduation custody (engine-graduation.ts): ordinary clients are refused, already-admitted work is drained with the graduation drain, and only the reverse follower may write. The fence stays after cutover and across restarts.
    3. Drain the follower, run a final reconcile, advance Rust allocators to at least the primary's high-water marks, and compare full relation digests and sequence verification on both engines.
    4. Switch authority atomically, restart the owner, and have a fresh process resolve the normal caller route and confirm the brain identity and the effective engine. The run refuses before step 4 while an unresolved environment or service override or an old serving process remains. Success is reported only after the probe passes.
  • Reverse shadow. The old primary becomes the reverse shadow: the follower copies Rust's committed writes back under the reverse-shadow policy, and comparisons, the ledger and the report keep running in the other direction. Reverse following freezes on a detected integrity divergence.
  • Failback. gbrain engine swap --to postgres (or --to pglite) uses the same plan, fence, drain, verify and route-probe steps in reverse, needs no readiness verdict, and keeps every acknowledged write. Failback is for a readable, correct Rust store. It is not protection against corruption copied into the reverse shadow.
  • Recovery. Verified recovery points (the pre-swap checkpoint plus periodic ones during the reverse-shadow period) are kept separately from the reverse shadow. If Rust becomes unreadable or corrupt, recovery restores a chosen recovery point, states the interval of writes that would be lost, and needs explicit approval for that loss.
  • gbrain engine shadow stop is idempotent, keeps data and authority, and says whether it stopped a reverse follower. stop --delete has its own --plan and --yes --expect <hash>, previews the exact directory and says that the fast way back is lost; for a reverse shadow on Postgres it never drops the database. The recommended reverse-shadow period is 30 days [proposed].
  • All start, swap and delete commands follow the graduation contract (src/commands/migrate-graduation.ts): --plan (alias --dry-run) writes nothing; no approval gives confirmation_required and exit 3; --json prints one document on stdout with progress on stderr; recovery instructions never need a working engine connection.
  • For Postgres brains, the swap moves the brain from a server database to a store on the shadow host. Other hosts then use the thin client (remote MCP) against the shadow host; the plan lists each observed writer and the command that moves it. A host that read Postgres directly was a trusted local caller and would become a remote caller. An owner-bound credential (a token bound to the owner member, created by gbrain auth on the shadow host and handed over by the user) relaxes private pages and private facts, but it does not restore every trusted-local read: member and folder restrictions still apply to the owner (src/core/access/caller.ts, src/core/search/private-visibility.ts), and remote takes reads default to the world holder (readHolders, src/core/ops/context.ts). So the swap plan lists, per observed host, what that host reads today and what it would read through its planned credential: source grants, member and folder restrictions, private pages and facts, and takes holders. It refuses the swap (engine_swap_refused, reason host_access_unbound) until each listed host has a credential, and (reason host_access_changed) while any host's reads would differ. Accepting a listed difference is a separate user approval with its own plan hash (--accept-access-change --expect <hash>); it is never implied by the swap approval. The swap never widens or narrows what a caller can read without that step.

Errors

engine_swap_refused (D15) has a recovery table before S4. Each reason (not_ready, multiple_writer_hosts, writer_coverage_unknown, team_brain, follower_behind, digest_mismatch, route_override, host_access_unbound, host_access_changed, fingerprint_semantics_differ) defines its registry class, exit code, the measured blocker, the authority state, side effects, a safe fix.next, a read-only fix.verify, a retry bound and a docs anchor. not_ready points to the report, never to an override. multiple_writer_hosts, writer_coverage_unknown and team_brain need a user topology decision; multiple_writer_hosts says plainly that a brain with writers on more than one machine stays on Postgres until the Rust server phase passes the D17 bar (D19, section "Path to a multi-user server"). follower_behind gives a bounded wait and a status check. digest_mismatch keeps diagnostics and leaves the old authority in place, and is not retried blindly. route_override names the variable or service, never its value. host_access_changed lists each host's differences and points to the separate access approval. fingerprint_semantics_differ keeps the old authority and is not retried until the engine or the source host changes. Interrupted runs carry the run id and a sanitized diagnostic id. CLI, doctor and --resume use the same builders.

Privacy, cost and agents

  • A shadow is a second full copy of the brain on the shadow host's disk, created only on the owner's request. The start plan states its size, the free space, the child process it will start and its resource bounds; the store directory is 0700.
  • Follower and reconcile reads use a dedicated connection with statement_timeout, a snapshot-age budget, a rate cap and read-only transactions, and pause under lock or load pressure. The report shows the extra load as queries per minute.
  • Shadow mode makes no provider call: embeddings are copied, never computed.
  • Agents may read status, report and every --plan without asking. Starting a shadow needs the user's request; a swap needs separate approval of the swap plan. READY is evidence, not permission. Start never enables general autopilot, paid maintenance or an upload.

Engine benchmark in gbrain-evals

The maintainer asked for a public benchmark of the engines, built alongside the Rust engine, that shows which one is fastest, best, lowest latency and cheapest. It lives in gbrain-evals, not here, and it starts before any Rust code exists so the PGLite and Postgres baseline is on record first.

  • Name and base: engine-bench in gbrain-evals (not engine-shootout, which collides with that repository's counted shootout-* runners), built on its existing perf.ts, scale-latency and graph-backend runners. The gbrain-evals maintainers write and land it (D11); this repository asks for it and answers questions on a gbrain PR. The published report follows that repository's takeaways-first report shape.
  • Engines: PGLite (in-process) and Postgres with pgvector (a local container) from the first PR. The Rust arm joins when PR 17 lands search parity, through the same BrainEngine factory; it is never benchmarked before it passes the parity gates, so a fast wrong answer cannot win.
  • Same inputs on every engine: one committed corpus and query set, seeded vectors, fixed seeds, the same pinned gbrain release for every engine (D11), the same machine, and warm-up runs discarded. No provider calls, so a run costs nothing beyond the machine.
  • What it measures, per engine:
    • Latency: p50, p95 and p99 per operation (putPage, getPage, keyword, vector and hybrid search, filtered search, backlinks, listPages, timeline), single client and at fixed concurrency.
    • Throughput: bulk import pages per second and chunk upserts per second at 1k and 10k pages.
    • Cold start: time from open to the first answered query.
    • Footprint: peak resident memory and on-disk size after import.
    • The D17 metrics in "Performance": concurrent read and write throughput at 1, 4, 16 and 64 clients or writers, graph traversal at depth 1 to 5, vector recall@10 against exact with p50 and p99 at 200k and 1M chunks, and RSS per 100k chunks. This repository asks for each card through the D11 channel.
    • Quality: Recall@5 and nDCG@10 on the committed qrels, plus top-k agreement with the Postgres arm, so "best" means the same or better answers, not only speed.
    • Cost: CPU seconds and memory per 1,000 queries, turned into dollars with a price model stated in the report (a list price per vCPU-hour and per GB-month, cited with its date). Hosted Postgres adds its published instance price as a separate line. The report shows the measured inputs, so a reader can apply their own prices.
  • Output: one JSON per run under eval/reports/engine-bench/ and a head-to-head card per metric. The report names a winner per metric and says plainly where an engine loses.
  • Shared with shadow mode: gbrain engine shadow bench replays the same operation set on a user's own brain, reads against the live stores and writes only against disposable snapshots (see "Latency and resources"), so it never changes the user's brain. The public benchmark answers "which engine is better on a fixed corpus"; the shadow report answers "is Rust better on my brain".
  • Landing: by gbrain-evals' own maintainers (D11). The benchmark is its own experiment there: a preregistration that names the decision the result changes (which engine to pick for a brain size; later, whether PGLite can be deprecated in favor of Rust), uses seeded vectors, makes no provider calls, and says why no plain-prompt arm applies (no agent is in the loop); paired comparisons with confidence intervals, cost per operation, and runs at gbrain's newest release. At most two PRs per experiment: the preregistration with the runner, then the results. Public numbers get a claim check and gbrain's disclosure line before they merge. Compute stays inside the shared limits, and every VM is destroyed when its run ends. Experiment 1 compares PGLite and Postgres now. Each Rust arm is a new experiment, after PR 17, and is re-run before PGLite deprecation.

Distribution

  • Installs (bun install -g github:garrytan/gbrain#stable) never compile Rust and never download the addon. No postinstall script and no npm dependency is added; napi-rs is used through Cargo only.
  • The eight lock addons are committed (native/locks/prebuilds/, 16 to 70 KB each). The engine addon will be megabytes per target, so committing eight of them per change would grow the repository quickly. Recommended (decision D4): release.yml builds per-target addons on native runners and attaches them to the release with build attestations; gbrain engine install rust downloads the addon whose digest the running release's manifest pins for this target, verifies it the way src/core/binary-self-update.ts verifies the release binary (SHA-256 plus attestation pinned to this repository's release workflow), and stores it under <gbrain home>/engines/rust/<engine_build>/. migrate --to rust offers the same step. gbrain engine install rust is idempotent (an installed, verified addon exits 0) and has a --json form. gbrain engine install rust --from <file> imports an addon copied by hand and verifies it against the same pinned digest; it is the offline path, and GBRAIN_RUST_ENGINE_ADDON stays a development override. Each install failure state (interrupted fetch, attestation unavailable, digest mismatch, no disk space or permission, the new addon failing its first open) leaves the active addon unchanged and maps to a reason in the error contract. After the new addon has opened the brain, addons older than the previous one are removed.
  • Upgrades: a release that leaves rust/ unchanged pins the same digest, so the installed addon keeps working. When the pinned digest changes and a Rust brain exists, gbrain upgrade and self-upgrade fetch and verify the new addon into its own <engine_build>/ directory before switching, and keep the previous addon until the new one has opened the brain once. If that fetch fails, the binary upgrade still completes and the Rust brain reports engine_unavailable with the install command, as post-upgrade does; an upgrade never fails for want of the addon. A raw bun install -g upgrade has no staging step: post-upgrade fetches the addon, and if that fails the brain reports engine_unavailable with the install command and a doctor finding, with no data touched. gbrain engine status (and its --json document, as additive fields under schema_version: 1) reports the target and whether it is a preview target, the addon path (home-relative), its digest and the pinned digest, the store format, the index lag and the capability report. Brains on PGLite or Postgres never trigger a fetch.
  • Platforms: darwin-arm64 and linux-x64-glibc first (the two release binary targets, .github/workflows/release.yml:128-132), then linux-arm64-glibc, darwin-x64, musl and Windows, matching the lock addon's target list before PGLite is deprecated on each.
  • The addon size is measured in the bake-off. Embedding it in the compiled release binaries is reconsidered only if it is small enough that every user can carry it.

Decisions needing the maintainer's yes

Approval record

On 2026-11-16 the maintainer answered the final review gate (revision 2) with "do it", which takes the recommended option on every gate item. Gate item ids are lowercase so they do not collide with the decision numbers below or the shadow PR names.

Gate item Question Approved option Plan decision
d1 How broad SQL rewrites start Checkpoint C1 gates broad Track 2 D6
d2 Preview platforms darwin-arm64 and linux-x64-glibc, backup and staged upgrade required D12
d3 to d11 Bake-off scope, migrate report, offline install, first-query target, Fix: placement, third-engine arrangement, search snapshots, graduation adapter, fact fingerprints The recommendation in each, as written in the plan plan text
u1 Who lands gbrain-evals changes gbrain-evals' own maintainers write and land them; requests go through one channel, a question: or heads-up: line on a gbrain PR; no maintainer step D11
u2 Effort budget through Checkpoint C1 8 calendar weeks or 25 merged PRs from PR 1, whichever comes first Phase 1 entry
u3 1M-chunk benchmark corpus and spend A permissively licensed public corpus, embedded once at 1,024 dimensions D10
s1 Shadow mode config, commands, first report D13 as written; first report after PR S2 D13
s2 Swap readiness The eleven criteria, no override in the preview D14
s3 Where the shadow runs A supervised child process D13
s4 Postgres swap In this plan, with the fence and one writer host D15
s5 Safety net after a swap Reverse shadow plus recovery points D14
s6 Sharing shadow results Local reports, manual sanitized export only D13
p1 The plan as a whole Approved all

D16 (the engine benchmark) comes from the maintainer's direction quoted in D16. D17 (the performance bar) comes from the maintainer's ruling quoted in D17. D18 (no SQL dialect) and D19 (single machine first, with a multi-user Rust server as the end state) come from the maintainer's answers quoted in each, including the 10:33 PM PST amendment quoted in D17 and D19. D20 (no PGLite-style locking) comes from the maintainer's principle quoted in D20. D1 to D5 and D7 to D10 are approved as directions here; the PR that adds each concrete key, code or default quotes the yes again, as the merge process requires.

Each yes is quoted in the body of the PR that needs it.

  • D1. Engine selection. The config value engine: "rust" (with database_path, default ~/.gbrain/brain.rust), gbrain migrate --to rust and back, kind rust in openEngine, and --engine on in-tree eval commands. Needed by PR 8 and PR 21.
  • D2. New error codes. engine_unavailable (reasons: addon_missing, abi_mismatch, build_mismatch, unsupported_platform, load_failed), engine_capability_missing (reasons: raw_sql, or the capability name), engine_store_format_unsupported (newer or unreadable store format). Needed by PRs 8 and 19. Review added the reasons fetch_failed and verification_failed (engine_unavailable, PR 22) and newer_format, unreadable and upgrade_failed (engine_store_format_unsupported). Eng review added the reason transaction_reentry on the existing internal_error code (PR 8).
  • D3. Development override. The environment variable GBRAIN_RUST_ENGINE_ADDON (absolute path to a locally built addon), for development and CI. Needed by PR 8.
  • D4. Distribution. Recommended: attested release assets fetched only on request (gbrain engine install rust, migrate --to rust) or, for an existing Rust brain, when an upgrade changes the pinned digest (staged by gbrain upgrade and self-upgrade, best effort in post-upgrade). Alternatives: commit prebuilds like the lock addon (simple, grows the repository by tens of megabytes per change); embed in compiled binaries (no fetch for binary installs, larger binary for everyone). Needed by PR 22.
  • D5. Keyword ranking. Changed by D17. The Rust keyword, title and facts arms reproduce ts_rank and ts_rank_cd with Postgres default weights and normalization as a compatibility mode, gated on fixture ordering; that mode is the regression baseline. A better ranker (BM25 or another) may be the default when it is equal or better than both Postgres and PGLite on the E1 and E2 retrieval metrics (R@5, nDCG@5, P@5, top-1). That default changes the result order callers see, so the PR that sets it quotes the yes. Needed by PR 15.
  • D6. Track 2 for all contributors. New SQL outside the engines fails a shrink-only ratchet; raw SQL moves behind typed engine methods grouped by domain; new two-way kind branches fail a guard. Needed by PR 6.
  • D7. Schema change policy after the preview. A migration that adds or changes a relation also adds its Rust catalog and inventory entries and its Rust disposition (catalog_only, postgres_only or a Rust handler with verify) in the same PR; the Rust arm in CI catches a miss. Needed before PR 23.
  • D8. Opening the preview. Users may run migrate --to rust once the Phase 8 entry criteria hold, including the D17 performance bar, with unsupported features refused by capability. Needed by PR 23.
  • D9. PGLite deprecation. Approval of the entry criteria in this plan now; the default change and notices later, in PR 25, with a separate yes.
  • D10. Eval spend and pins. About $2 to regenerate the LongMemEval embedding cache, the pinned gbrain commit for parity runs, and the E1 to E4 thresholds as written.
  • D11. gbrain-evals landing. Resolved, on that repository's final terms: its own maintainers write the gbrain-evals code and land it. This work opens no PRs there. It asks through one channel, a question: or heads-up: line on a gbrain PR, and the maintainer runs no step. The gbrain pin keeps following gbrain's newest release through bun eval/runner/gbrain-pin.ts bump, the only pin scheme; parity runs name the pinned release. A Rust build enters as a frozen gbrain-rust alias named in its preregistration, with the addon digest, the same way the existing frozen aliases are. Frozen or counted files there (preregistered experiments' counted files, sealed paths, receipts under docs/benchmarks/*/counted/) are never changed. Needed by the Phase 3 exit, and so by E1.
  • D13. Shadow mode. The commands gbrain engine shadow start|stop|status|report|explain|bench, the config keys shadow.enabled, shadow.read_sample (default 0.1, 1.0 under 10k pages), shadow.write_sample (default 0.05), shadow.queue_max, shadow.cpu_share and shadow.ledger_retention_days (default 90), the ledger and evidence manifest format, the supervised shadow child process, the doctor check rust_shadow, and shadow mode for Postgres primaries. Everything is off until a user starts it. The maintainer's request: "make it possible to run rust in shadow mode to existing postgres and make it easy to swap once it shows its clearly working (good metrics head to head, we should track it and generate it)". Needed by S1 to S4.
  • D14. Swap readiness. The eleven criteria in "Swap readiness" (criteria 4 and 8 as D17 changed them) with their numbers and the margin for "better on filtered vector search", fixed before candidate results exist; no readiness override in the preview; a 30-day recommended reverse-shadow period; recovery-point retention and the acceptable loss interval for disaster recovery. Needed by S4 and S5.
  • D15. Swap fence and Postgres swaps. A durable, database-enforced write fence on the old primary through graduation custody (the one engine-side addition), a writer-observation table (a schema migration), and Postgres swaps only with one observed writer host for 7 days; the brain is then served from the shadow host and other hosts move to the thin client. New error code engine_swap_refused with the reasons and recovery table in "Errors", including host_access_unbound, host_access_changed and fingerprint_semantics_differ. Other hosts of a swapped Postgres brain move to owner-bound credentials listed in the swap plan; the swap is refused while any host's reads would change, unless the user approves that difference separately. Needed by S5.
  • D16. Engine benchmark. A public engine benchmark in gbrain-evals (see "Engine benchmark in gbrain-evals"), PGLite and Postgres first, Rust after PR 17. The maintainer's words: "alongside this as you work on the rust engine you should probably also work on a garrytan/gbrain-evals benchmark of the different engines to show which one is fastest/best/lowest latency/cheapest". No change to this repository.
  • D12. Preview contract. The preview targets darwin-arm64 and linux-x64-glibc only. It opens only with verified backup and restore and the staged addon upgrade. Windows and the other platforms come before PGLite deprecation on each, not before the preview. Needed by PR 23.
  • D17. Performance bar: better than Postgres on every axis. The maintainer's ruling, 2026-11-16 at 10:11 PM PST: "oh i want this rust engine to be better than postgres multi-threaded and more performant and better at all the things: better graph traversal, faster. better pgvector / vector search. everything must be better. more multi-threaded and better memory management and faster." The maintainer added the target at 10:33 PM PST: "i think we should engineer it knowing we will want full multi-user rust that totally supercedes and is 10x better than using postgres and pgvector". 10x better than Postgres and pgvector is the target; equal or better on every axis stays the gate to ship. This plan sets no per-axis 10x number: the gbrain-evals preregistration fixes each margin before candidate results exist. The bar changes from "never slower than PGLite" to equal or better than TypeScript gbrain on Postgres (pgvector HNSW, local, same machine, same corpus, same run) and on PGLite, on every metric in "Performance"; the proposed absolute targets stay as floors. A Rust arm is not called ready for the swap (D14 criterion 8) or the preview (D8) without paired runs with confidence intervals on every metric, and a loss on any axis is listed in bakeoff.md and the shadow report with its fix. Better ranking or higher vector recall returns different results than Postgres, so ranked arms are gated on equal or better retrieval quality (E1 and E2: R@5, nDCG@5, P@5, top-1), not on identical results; vector overlap@10 with the primary is reported, not gated, and recall@10 against exact must be at least pgvector's. Everything else stays byte-exact: CLI and JSON output, MCP names and shapes, the markdown brain format, content hashes, fingerprints and deterministic operations, with zero privacy leaks. Multi-writer throughput, graph traversal and memory join the Phase 2 bake-off, with a concurrent-writer store required (D19), and a memory budget per 100k chunks set in bakeoff.md before any Rust number exists. Sections changed: Recommendation, Goal, Constraints, Architecture (the contract line), the storage replacement map (records, locks and CJK rows), the owner's write serialization, Phase 2, the Phase 3 exit, Checkpoint C1, the Phase 8 entry, PGLite deprecation criterion 6, Eval gates, Performance, the shadow's deterministic and ranked comparisons, the shadow report, swap readiness criteria 4 and 8, the engine benchmark, Risks, D5, D8 and D14. One owner process per brain directory and the refusal of multi-host writers hold for the embedded phases only; D19 makes a multi-user Rust server the end state.
  • D18. No SQL dialect in the Rust engine. The maintainer, "do it." (10:24 PM PST 2026-11-16), answering asks A1. The Rust engine has no SQL dialect and no query language: typed BrainEngine methods plus the relation API only. executeRaw and every other raw-SQL entry point are refused on Rust with a typed error (engine_capability_missing, reason raw_sql). The plain SQLite tables-and-triggers slice leaves the Phase 2 bake-off; SQLite may stay only as a plain key-value record store option. Every raw-SQL call site a Rust brain can reach (migrations, doctor, tests, and the seven gbrain-evals files) moves to typed methods; this is required Track 2 work, not optional. Sections changed: Recommendation, Non-goals, Capability gating, Track 2 rules, Phase 2, Phase 3 and the bake-off criteria.
  • D19. Single machine first; a full multi-user Rust server that supersedes Postgres and pgvector is the target end state. The maintainer, "do it." (10:24 PM PST 2026-11-16), answering asks B1, then amended on 2026-11-16 at 10:33 PM PST: "i think we should engineer it knowing we will want full multi-user rust that totally supercedes and is 10x better than using postgres and pgvector". The embedded phases run one owner process per brain directory, and brains with writers on more than one machine stay on Postgres until the Rust server passes the D17 bar. The engine is built for the server now: the storage core has no global singletons or per-process state that assumes one caller; transactions and snapshots work across connections and processes, not only threads in one process; the access checks (the typed write-scope argument and the invariants I1 to I15) live in the engine layer, gbrain-domain, as the RLS replacement a server enforces per user, never in the napi wrapper; and the store format considers replication and backup hooks (a commit log or snapshot stream a follower can consume), designed now and built later. The concurrent-writer store is required in the Phase 2 bake-off, and working across processes and connections is a bake-off criterion. The server phases are in "Path to a multi-user server"; each needs the maintainer's yes before it starts. The swap and the preview refuse a brain that has multi-host writers, with a clear message that it stays on Postgres until the Rust server phase passes the D17 bar (multiple_writer_hosts). Sections changed: Recommendation, Goal, Non-goals, the storage replacement map (records and access rows), Writer admission, Process model, Phase 2, Postgres deprecation, Path to a multi-user server (new), Performance, Migration and rollback, swap readiness criterion 11, Errors, the approval record, D17, open question 12, Risks and Deferred.
  • D20. No PGLite-style locking. The maintainer, 2026-11-16 at 10:39 PM PST: "the other thing about pglite - pglite sucks! there are so many locks and problems with it. the rust implementation we want it to be pristine and perfect for our cases." Concurrency is designed fresh for gbrain's workloads: many readers, many writers and many processes on one machine, then many machines (the D19 end state). There is no process-wide lock, no single-connection bottleneck, no busy error (no pglite_busy equivalent) and no cross-process file lock that makes a second process fail or wait for the whole brain. The Rust engine does not copy gbrain's lock primitives one for one (named transaction and advisory locks, row locks, SKIP LOCKED queue claims, pglite_busy handling): each lock and queue call site gets a typed engine operation with a stated guarantee, listed in "Concurrency without PGLite-style locks"; each has a parity test against TypeScript gbrain under concurrent callers and a stress test with many writers and many processes. The Phase 2 bake-off adds three criteria: no process-wide writer lock on the hot path, a second process reads and writes without busy errors, and contention measured and reported. One owner process per brain directory stays only as a temporary limit of the embedded phases, not a lock design the core depends on (D19). Sections changed: Recommendation, Goal, Non-goals, the storage replacement map (locks row), Binding (lockPageKeys), Data on disk, Process model (including the owner protocol), Errors (maintenance path), Concurrency without PGLite-style locks (new), Phase 2, Crash, property, concurrency and fuzz tests, PR 18, the approval record and Risks. Needed by PRs 2, 3, 6, 8, 18 and 20.

Open questions

  1. Who lands gbrain-evals changes? The merge process covers this repository only, and research/11 describes an older, unpinned snapshot of the evals repository. Resolved in D11: that repository's own maintainers write and land them from requests on a gbrain PR, and the pin follows gbrain's newest release through its pin tool.
  2. If the bake-off picks bundled SQLite as the record store (as a plain key-value store only, D18), constructEngine's message that SQLite is unsupported and docs/ENGINES.md need rewording, since SQLite would be an internal detail of the Rust engine. Acceptable?
  3. Which public 1M-vector corpus (license and dimensions) serves the 1M benchmark, or is a paid one-time embedding of a public corpus preferred?
  4. Does any caller search inside its own write transaction? The guard test answers it; a positive answer needs a design note before that subsystem moves.
  5. The keyword arm does not wrap the query in gbrain_fts_input (research/03 section 9). The Rust engine copies current behavior; should a fix ship separately for all engines?
  6. Which collation and lower() behavior does PGLite use in practice? The fixtures pin the Rust engine to it, and the answer may show a PGLite and Postgres difference worth its own issue.
  7. When is Windows needed? Proposed in D12: after the preview, before PGLite deprecation on Windows. Confirm with D12.
  8. Is a Rust index accelerator for Postgres brains (option B) wanted later?
  9. Resolved: Rust brains get backup create and backup restore (PR 21b) before the preview. The existing commands refuse non-PGLite brains, and migrate --to pglite needs a readable store, so it is not recovery.
  10. Is four weeks of daily use the right dogfood length for criterion 9?
  11. Are the swap thresholds in D14 strict enough, or should the vector arm require exact equality with the primary when the brain is small enough for exact search on both?
  12. Settled by D19: in the embedded phases, brains with writers on more than one machine stay on Postgres; the multi-user Rust server that would serve them is a later phase (section "Path to a multi-user server"). Is the maintainer's own brain single-writer? C1 checks it before any swap timeline is given.
  13. Should preview users be invited to export and submit a sanitized aggregate report for shadow-log.md (never uploaded automatically), or only the maintainer's brain?
  14. For quiet brains, should readiness accept provider-free replay coverage, counted separately from live comparisons, to reach the per-operation minimums sooner?

Risks

  • Track 2 dominates the schedule. About 5,300 call-site lines across many subsystems, while master moves daily. Mitigation: the shrink-only ratchet stops growth from day one; subsystems are ordered by what an embedded brain reaches; each PR is small and independently useful to the Postgres engines.
  • Contention on shared files. engine.ts and engine-sql/ change in many PRs. Mitigation: domain objects add one line to engine.ts per domain; heads-up comments; the merge process sequences the stack.
  • Exactness drift. ts_rank, pg_trgm, fingerprints, decimals, collation, jsonb order. Mitigation: PGLite-generated fixtures for each, eval gates, and verbatim carry of stored hashes.
  • Silent Postgres branches. A kind check that sends Rust down the Postgres path. Mitigation: the kind-branch guard and capability questions before a subsystem is allowed on Rust.
  • A missed trigger is a silent loss of enforcement (research/09 risk 1). Mitigation: one Rust write chokepoint, the trigger parity matrix, and per-operation authorization tests.
  • A native crash takes down the process. Mitigation: catch_unwind on every export (panics only), fuzzed boundary, crash-safe store, no unsafe in gbrain crates without review. Segfaults, aborts and dependency faults remain a residual risk, accepted for the preview; a sidecar is reconsidered if crash reports show them.
  • Memory. An in-process ANN at 1M chunks and 1,024 dimensions is several GiB as f32 (docs/ENGINES.md, "Vector index sizing": about 7.6 GiB per million as vector). Mitigation: the bake-off measures f16 storage, memory-mapped vectors and quantization against recall.
  • Better results look like different results. A better ranker or a higher-recall vector index changes what callers see (D17). Mitigation: the ts_rank compatibility mode stays as the regression baseline, ranked arms are gated on eval quality at least equal to both existing engines, and overlap with the primary is reported in every shadow report.
  • The D17 bar may not be reachable on every axis. Postgres is a mature multi-writer server, and one axis, such as many concurrent writers, may resist an embedded design. Mitigation: the bake-off measures the writer curve before Phase 5, a concurrent-writer store is required, every loss is listed with its fix, and Checkpoint C1 decides continue, narrow or stop on that data.
  • Building for a server slows the embedded preview. The server-ready criteria (connection-bound transactions, no singletons, replication hooks in the format) add work before any server exists (D19). Mitigation: the criteria constrain choices but build nothing early; the bake-off picks the simplest option that meets them; each server phase starts only with the maintainer's yes.
  • A server needs the access enforcement Postgres gets from RLS. Mitigation: the access checks live in gbrain-domain from Phase 5, with per-operation authorization tests, so Phase M2 adds per-user credentials, not a new access model.
  • A typed operation misses a guarantee a lock gave implicitly (D20). A lock can protect more than its call site names. Mitigation: each operation's parity test runs concurrent callers against TypeScript gbrain on both engines, the stress cell runs many writers in many processes, and the TLA+ properties stay as tests.
  • Bun and Node-API gaps. napi-rs features may behave differently in Bun. Mitigation: PR 1 proves loading, async tasks and typed arrays in Bun 1.4.4 and in a compiled executable before any engine code.
  • Downloaded native code. Mitigation: attestation and hash verification pinned to this repository's release workflow, the same as the self-updating binary, plus the per-release digest manifest; fetch only on explicit request or for an existing Rust brain whose pinned digest changed.
  • Schema drift after release. New TypeScript migrations land several times a week. Mitigation: D7 plus the Rust arm in CI; additive changes need only a catalog entry.
  • Split brain during migration. Mitigation: quiesce, tombstone, verify before switching, and the two rollback paths.
  • CI cost. A Rust lane plus a growing Rust test arm. Mitigation: path filters, one addon build per run reused by shards, and the added time reported in PR 1 and PR 9.
  • Time to first value. The preview waits for Track 2 persistence, page state and facts (1,844 raw-SQL lines), so no user benefits for many PRs. Mitigation: Checkpoint C1 decides continue, narrow or stop with real-brain data and effort against budget; the named first cohort and two preview targets keep the preview small.
  • A native crash in the shadow. Mitigation: the shadow runs in a supervised child process while another engine is primary; the serving process never loads the addon.
  • Corruption copied back. A Rust bug could write bad data that the reverse shadow then copies. Mitigation: an immutable pre-swap recovery checkpoint, periodic recovery points, and reverse following that freezes on an integrity divergence.
  • Shadow load on the primary. The follower and reconcile read from Postgres or PGLite. Mitigation: read-only transactions with statement_timeout, a rate cap, a CPU share, pause under lock or load pressure, and the load shown in the report.
  • A second full copy of private data. Mitigation: opt-in per brain, a disk and privacy notice in the plan step, 0700 store, shadow stop --delete, and a ledger with keyed hashes only.
  • False confidence from sampling. A rare operation may be mirrored too few times. Mitigation: criterion 2 requires every operation used in the window to be mirrored, the privacy check is never sampled, and report shows per-operation counts beside each verdict.
  • Shadow drift. A missed write makes the shadow disagree and could block a good swap or hide a bad one. Mitigation: the follower copies rows instead of replaying calls, reconcile digests every range, stale is rechecked before it counts, and the swap compares full digests after the final reconcile.
  • Addon drift on upgrade. A same-ABI addon with older behavior could load, or an upgrade could leave a Rust brain without a usable addon. Mitigation: the pinned digest manifest, the staged upgrade that keeps the previous addon, and the offline upgrade drill (criterion 12).

Deferred

  • Option B, a Rust accelerator inside the PGLite or Postgres engines: it changes existing engine behavior, which Rust PRs may not do. Revisit after the preview (open question 8).
  • Platforms beyond darwin-arm64 and linux-x64-glibc before the preview: PR 24 still lands before PGLite deprecation on each platform (D12).
  • A sidecar process binding: revisit only if preview crash reports show native faults.
  • The gbrain_fts_input keyword-arm fix (open question 5): a separate change for all engines, never inside a Rust PR.
  • A stdio driver protocol and an oracle-ts shim for gbrain-evals: not needed while the in-tree gbrain eval harnesses and engine-factory cover the gates.
  • LLM-judged gbrain-evals categories as gates: they stay reported in bands set beforehand (E4).
  • Moving the Fix: line to the last line of the shared CLI error renderer (src/core/agent-output.ts prints it second): a change for every command, never inside a Rust PR. Agents read fix from --json.
  • A Rust preview issue template under .github/ISSUE_TEMPLATE/: the preview docs name the reporting route instead.
  • A gbrain engine remove rust command: removing an addon by hand is enough for the preview.
  • gbrain migrate --to rust from a Postgres brain. Single-writer Postgres brains use shadow mode and engine swap. Postgres brains with writers on several hosts stay on Postgres until the Rust server passes the D17 bar (D19).
  • The multi-user Rust server, Phases M1 to M5: designed for now, built later, each phase with the maintainer's yes (section "Path to a multi-user server").
  • Moving fact fingerprint computation out of SQL for all engines, or a versioned normalization: it can change Postgres behavior, so it needs its own change and the maintainer's yes.

Review record

The accepted review requirements (binding on the PRs they name), the review record, the decision audit trail and the review scorecards are in rust-engine/review-record.md.