Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
50.0 KB
Raw Blame

Persistence validation

Run the complete gate from a source install (bun install --frozen-lockfile --ignore-scripts works):

bun --no-env-file scripts/persistence/validate.ts --engine=pglite
DATABASE_URL=postgres://test-user:test-password@localhost:5432/gbrain_test \
  bun --no-env-file scripts/persistence/validate.ts --engine=postgres

Postgres requires a test-shaped database URL and permission to create and drop databases. Every phase gets a fresh, randomly named gbrain_persistence_test_* database. Successful runs drop only those databases. Failed runs retain their synthetic scratch directories and databases for inspection. PGLite uses temporary disk datastores, reopened in separate processes. Child homes and writer lock paths are temporary; no operator brain or provider credentials enter children.

The default gate executes, per engine:

Workload Required result
1,000 seeded schedules (seed 5105) 100 executions of each of the ten cases below; every schedule checks exact journal counter conservation and no leftover claimable work
Eight actual SIGKILL boundaries Acknowledged requests survive reopening; unfinished file effects recover; committed DB/file/receipt state stays committed; original request replay returns the same outcome
10,000 logical writes Four independent producer processes, four principals and four source roots; every request commits once; every canonical snapshot and file matches the receipt; zero pending requests or unresolved recovery records

CI runs this full gate on pushes to master and manual dispatches. Pull requests run the same schedules and crash boundaries with a 2,500-write soak (--operations=2500), so their manifests report full_gate: false; a soak regression that needs more volume is caught on master.

The eight executed crash boundaries are admitted, prepared, before_publication, staging_flushed, after_publication, before_commit, after_commit, and after_response. The flushed boundary writes its event synchronously and blocks the child before rename. The response boundary serves a real fixture HTTP receipt; the parent fully reads and verifies it before SIGKILL. Recovery verifies that no recorded or unaccounted temporary sibling remains. Partial or unexpected staging bytes preserve quota and require explicit recovery; a committed receipt is never reversed to resolve them.

Run just these process crashes with --schedules=0 --operations=0; its manifest correctly reports full_gate: false. Input hashes include the atomic writer, staging helper, recovery models, journal, coordinator and effect sinks.

The ten schedule families are concurrent identical/changed-intent replay; competing creates and replacements using one revision; cancellation versus publication; rollback/lost response at all five coordinator hooks; obsolete claim renewal/release; FIFO within each root with unrelated-root progress; MVCC/serialized coherent reads; unexpected external file bytes blocking recovery; concurrent quota admission; and revocation after acceptance. A seeded PRNG varies principals, roots, payloads, concurrent widths and submission order. Real production coordinator hooks control transaction and filesystem boundaries. This is a bounded schedule sample, not exhaustive model checking.

The crash cases kill a process after admission, durable prepare, immediately before publication, after staging flush/close, after rename, before commit, after commit, and after a delivered response. The PGLite case exercises the datastore owner's death. The Postgres case kills the client/owner process while the database server remains running. These are process-crash RPO=0 checks; they do not simulate power loss, storage controller failure or database-server loss.

Postgres runs two resident consumers and producers with independent database connections. PGLite has one resident owner; producer processes submit through a fixture-only loopback endpoint into the real admission API. That endpoint does not test production authentication or MCP/IPC framing; the existing receipt and transport suites cover those contracts. Both engines use the real consumer, authority checks, kernel locks, coordinator, durable files and database transactions. Four writes per producer stay in flight; every seventeenth logical write is replayed under the same request ID.

Crash robot

The crash robot runs generated sequences of real operations against a managed brain and checks a reference model after every step, after every crash and recovery, and after the final drain. It is a phase of validate.ts with a time budget (--robot-seconds, default 300; 0 skips it). Run it alone with:

bun --no-env-file scripts/persistence/validate.ts --engine=pglite \
  --schedules=0 --operations=0 --no-crashes --robot-seconds=300

Every op is a JSON descriptor (ops.ts) executed through its registered operation handler with a real OperationContext: page writes, edits, deletes and restores, remember/forget, takes add and supersede, timeline entries, a user commit followed by sync_brain, a GitHub connector publish, and credential revocation. Actors are the local CLI principal and two remote agents, an OAuth client and a legacy access token, authenticated through the production token verifier. Two sources share slugs, sit in two durability- hardened Git checkouts, and a third source is the connector's.

Each run starts from a fresh datastore. The fixed sequences always run: withdrawal followed by a stale republication, overlapping slugs across two sources, competing writers on one page, caller-bound replay, an authority revoked while effects are pending, and sync plus connector publish racing direct writes. A counting pass records every crash seam a sequence reaches (src/core/persistence/fault-points.ts); the robot then crashes each seam at least once: the worker freezes at the seam, the driver SIGKILLs it, and a fresh worker checks the state left behind, drains, resubmits the interrupted requests with their original request ids and finishes the sequence. Random sequences fill the rest of the budget. Process faults run once per engine: a stale index.lock, a hung git commit that holds index.lock (it releases the lock on SIGTERM, not on SIGKILL), on Postgres every session of the run database dropped every 400 ms, and the owner killed right after a facts-absorb job's extraction commits (deterministic chat and embedding stubs); the job then runs again and every absorbed fact must be active exactly once. The robot records each session-drop round it injects. Its own reads (oracle checks, drain polls, $current revision lookups, token re-verification) retry a closed connection only after a round that began before the error terminated a session; any other close fails the run with the next step, so a database, pooler or product-side close is never retried away.

The reference model (model.ts) checks: a committed receipt is visible at once and later revisions move only through committed ops; a late receipt names a revision the page really had; after a concurrent group a page ends at a revision a committed member returned, or at the write of a committed sync or connector publish (which return no revision) whose marker it carries, and every receipted revision a later member replaced is in the page's history; a refused write never becomes visible; a withdrawn fact never returns through recall, get_page or the facts table; a write never lands in another source; another principal reusing a request id never receives the original receipt; takes and facts rows equal their page fences; content rows carry write attribution; no orphan rows; every request and effect drains (within 20 s on PGLite, whose restarted owner releases a dead owner's claims, and within the 2-minute effect lease plus margin on Postgres); a fresh write still commits. lock-order.ts checks each transaction's row locks: worktrees before sources, sources in id order, no exclusive brain-row lock inside a publication, and the brain row before any worktree, source or counter row whenever one transaction locks both (a write to persistence_requests or persistence_effects counts as the brain-row FOR SHARE read its protocol trigger takes).

Postgres runs connect through GBRAIN_PGBOUNCER_URL when it is set, with prepared statements off. A Postgres run with a budget under 300 s (the pull-request run) skips what can wait out a dead owner's claim lease: the effect:* seams, consumer:prepared, the publication seams prepared, before_publication and after_publication, and the pooler disconnect fault. Full runs crash every seam and run every fault. The manifest's robot key lists every run (schedule, seed, seam and occurrence or process fault), sequences_x_crash_points, lease_bound_seams_skipped, lease_bound_faults_skipped and violations with the observed trace; robot_full_gate is true for a full-budget run with no violations, no fault_hook_errors (a fault hook that threw instead of freezing the worker) and no faults_not_reached (a worker that finished without reaching its requested seam, outside the per-scenario UNREACHED_FAULT_ALLOWLIST in robot-driver.ts); either kind of run also fails validation with a nonzero exit and is listed in failing_runs for --replay. A run with no violations that misses its fault by exactly one slot of its count run, which a concurrent group's other legal outcome makes, runs once more at its last slot and is listed in faults_retargeted.

A failing run prints two commands. --replay=<manifest> re-runs exactly the manifest's failing runs. --shrink=<manifest> removes ops from the first failing run (an op whose producer was removed goes with it) while a violation of the same class remains, accepts the result only when it reproduces in 3 of 3 reruns, and writes <manifest>.shrunk.json. Accepted shrunk sequences are checked in under test/fixtures/crash-robot/ and replayed by test/persistence-crash-robot.slow.test.ts.

For a local Postgres and PgBouncer replay:

docker network create gbrain-robot
docker run -d --name robot-pg --network gbrain-robot -p 127.0.0.1:5432:5432 \
  -e POSTGRES_USER=gbrain_test -e POSTGRES_PASSWORD=gbrain_test -e POSTGRES_DB=gbrain_test \
  pgvector/pgvector:pg16
docker run -d --name robot-pgbouncer --network gbrain-robot -p 127.0.0.1:55433:5432 \
  -e DB_HOST=robot-pg -e DB_PORT=5432 -e DB_USER=gbrain_test -e DB_PASSWORD=gbrain_test \
  -e POOL_MODE=transaction -e AUTH_TYPE=plain -e MAX_CLIENT_CONN=200 -e DEFAULT_POOL_SIZE=10 \
  -e IGNORE_STARTUP_PARAMETERS=extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path \
  edoburu/pgbouncer:latest
PGPASSWORD=gbrain_test DATABASE_URL=postgres://[email protected]:5432/gbrain_test \
  GBRAIN_PGBOUNCER_URL=postgres://[email protected]:55433/gbrain_test \
  bun --no-env-file scripts/persistence/validate.ts --engine=postgres --replay=<manifest>

buildHistoryFixture(engine, { pages, seed, sources, worktrees }) (history-fixture.ts) builds a managed brain with real persistence history (up to 10,000 pages) through the same op protocol: committed and queued requests, a delayed Git effect, withdrawals, superseded takes, page versions, chronicle ledger rows, local writers, an OAuth client and token, and an access token with unified grant columns. The engine must be initialized and not yet activated, under an isolated GBRAIN_HOME.

would-have-caught.ts measures the gate against past fixes: for each fix frozen in would-have-caught.json it reverse-applies the fix's src/ hunks in a scratch worktree and runs the pre-robot gate and the robot phase against it, after a HEAD control that must pass.

The default manifest is .context/persistence-<engine>-manifest.json. It contains the actual completed case counts, crash outcomes, runtime/platform, latency distributions (p50/p95/p99/max), concurrent canonical-read checks, throughput, peak resident RSS, duplicate replay count, per-phase source hashes and final accounting results. A failed run writes a failed manifest. Smaller runs (--schedules=50 --operations=64) are useful for iteration and always report full_gate: false; --no-crashes does too. Use --seed=... for another reproducible sample and --manifest=... to keep multiple records. Performance numbers describe a synthetic body+timeline+tag workload with a durable file per write; they exclude provider calls, Git publication and remote network latency. Compare like-for-like runtime, storage and process counts before setting or changing latency budgets.

On failure, the manifest includes the last cached state of each producer's at-most-four active requests and a bounded owner snapshot: queue states and ages, root ownership epochs, counters, consumer activity and error codes. It excludes content, filesystem paths, authority, credentials and error messages. Owner diagnostics have a two-second budget; an unavailable owner adds a timeout marker and never changes the original failure or the 120-second receipt deadline.

The runner writes <manifest>.retained.json with mode 0600, listing the retained scratch root, worker PIDs and exact cleanup commands. After inspection, verify those workers have stopped, run the listed database commands using the original loopback test DATABASE_URL in the environment, then remove the listed scratch directory. The metadata contains generated database names, never a connection URL. Keep the retained directory private: its original config.json files contain the test connection URL. Do not upload it with the diagnostic manifest. Successful runs retain no fixtures.

persistence-validation.yml runs the full gate on Linux x64 for both engines under Bun 1.4.4 and uploads every manifest. Native OS/architecture coverage is separately required by native-locks.yml; its configured matrix must not be mistaken for locally executed runtime evidence.

All stress fixtures explicitly activate managed persistence after registering canonical roots. Workers assert that activation remains enabled and use a synthetic host identity confined to the runner's temporary home. Matrix read probes are seeded before activation; the measured writes use the coordinator. Known permanent transaction failures stay failed after conditional filesystem recovery, allowing the next request for that root to proceed.

The same workflow executes scripts/persistence/matrix.ts, requiring both DATABASE_URL (direct test connection) and GBRAIN_PGBOUNCER_URL (a real transaction-mode pooler with wildcard database routing). Both supplied database names must pass the test-safety guard. Administrative CREATE/DROP statements use the direct server's postgres maintenance database, so another E2E shard resetting the shared test database cannot terminate this connection. Every engine connection still uses a fresh generated test database. The 24 cells cover direct/pooler transport, RLS on/off under a non-superuser role, ordinary pools 1/2/3 and shared pools versus a separate direct pool of size one. Each cell proves short control progress while the production bulk reservation API holds every permitted long-running slot. Size one keeps canonical work queued with writer_pool_capacity; sizes two and three commit the same request after bulk work drains. A separate fixture checks manifest-verified transfer between distinct host identities/checkouts, stale-owner refusal, retained coordination paths across root replacement and source-incarnation fencing. The default matrix manifest is .context/persistence-runtime-matrix.json; missing mandatory URLs fail the standalone gate. The ordinary E2E entry skips outside a configured pooler lane and refuses to skip when GBRAIN_CI_REQUIRE_PGBOUNCER=1.

The heavy process worker allows 90 minutes; its CI job allows 110 minutes. This accommodates disk-PGLite durability on slower VM storage without reducing the 10,000 actual mutation requirement.

The required read-workload validity lane runs scripts/persistence/performance.ts --engine=pglite --informational (or --engine=postgres with the same guarded test URL). It keeps the existing heavy workload's 500-page text corpus, 200 hybrid searches per phase, four writers and an advisory 50% loaded-versus-idle p99 threshold. Three fresh child processes/databases each measure idle reads followed by reads with public put_page writes; the report compares the median loaded p99 to the median idle p99 on the same runner. This measures the cost of additional concurrent work, not a before/after comparison of a code change. Each run requires actual committed writes, zero failed reads/writes and at least 90% coverage of the read window by the union of in-flight public mutation intervals. An idle gap cannot be hidden by a late writer completion. Actual writes must commit during the read window. Both phases yield one event-loop turn between queries (outside individual query timing), so PGLite's immediate promise chain cannot starve resident-consumer timers and fabricate overlap using only queued requests. Corpus seeding uses the same public mutation path. This is keyless keyword search through hybridSearch, without a provider or remote embedding latency.

The manifest records all three runs, exact source hashes, storage/runtime and runner characteristics, admission/completion distributions, queue age, recovery bytes, RSS, throughput and Postgres activity samples. The harness records Bun's known Failed to get memory usage exception as a null RSS sample and counts it in rss_unavailable_samples; peak_rss_bytes is the maximum available sample, or null when none are available. These counters and peaks include every settled sample, including those completing during final validation. This known sampling limitation does not invalidate actual read/write measurements or relax the latency and overlap gates. Unexpected sampler errors still invalidate the run and retain only safe diagnostic fields. The harness measures durable admission when the public handler's top-level queued journal transaction resolves, and completion when its terminal committed receipt is observed. Nested savepoints never count as admission. The same harness proxy observes warmup and pressure writes; measurement buffers reset after warmup. Every completed write must have its own earlier admission observation, and missing or incomplete timing distributions invalidate the aggregate gate. Pool gauges are explicitly a tracked SQL subset; pg_stat_activity separately records active/idle sessions in the fresh fixture database, including the sampler. PGLite keeps the original in-memory read-latency storage model; the separate 10,000-write durability lane uses disk storage. Smaller corpus options are recorded as full_gate: false. tests/heavy/read_latency_under_sync.sh retains its optional STRICT_LATENCY=1 interface and now runs the same three-sample harness; sample validity always fails closed. CI passes --informational on every supported Bun version and both engines: a valid workload over the 50% threshold exits successfully, but the manifest keeps its verdict: "fail", status: "failed" and full_gate: false threshold result. Invalid samples, failed reads/writes and insufficient overlap still fail CI. Omit --informational to enforce the threshold locally; the threshold value and measurement workload have not changed.

Test suites

These suites own the durable-persistence contracts. test:full alone does not run the complete native, runtime, crash, soak, deployment-matrix and read-latency gates; tie each result to its tested revision and disclose skipped cells.

Canonical reconciliation and accepted writes

test/persistence-reconcile-merge.test.ts pins loss-preserving field choices. test/persistence-reconcile.test.ts runs the guarded repair and replay contracts on PGLite and, with an explicit safe DATABASE_URL, isolated PostgreSQL databases. It covers stale preconditions, current/original grants, private facts, retained backups, ordinary mutations after repair, and competing publications. test/reconcile-owner-journey.serial.test.ts (stdio) and test/reconcile-owner-journey-http.serial.test.ts (HTTP) drive real CLI requests through PGLite owners before and after activation, restart the owner, and independently read the newly remembered private fact and provenance (test/helpers/reconcile-owner-journey.ts).

test/reconcile-crash-*.slow.test.ts and test/e2e/reconcile-crash*.test.ts kill real processes at all eight publication boundaries with activation off/on. PostgreSQL uses one file per activation state to stay within the unchanged per-file cap. Optional GBRAIN_TEST_RECONCILE_CRASH_MANIFEST_DIR retains executed-case evidence. test/e2e/reconcile-pgbouncer.test.ts requires the transaction-mode pooler when GBRAIN_CI_REQUIRE_PGBOUNCER=1 and proves repair followed by a new private memory write. The durable-persistence workflow runs these contracts on every supported Bun version and uploads the crash manifests; local CI runs the slow and E2E lanes.

test/put-page-persistence.test.ts and test/e2e/put-page-persistence-postgres.test.ts pin durable page acceptance and ordinary-error publication: native contention returns an accepted pending receipt without changing the page, and replay of its original UUID commits exactly once after release. Filesystem or required source-path failure rolls back the database transaction. Embedding failure preserves the canonical receipt; a delayed result superseded by another revision cannot install vectors. The PGLite suite also covers scoped physical file paths, unchanged-content no-ops, legacy hashes, deletion/recreation, and sanitized diagnostics. Actual process-death boundaries belong to the crash suites below.

test/subagent-required-writes.test.ts and test/subagent-put-page-rejection.serial.test.ts distinguish a persisted write from prose-only completion, rejected imports, and historical rejected ledger envelopes across the Anthropic, gateway, and oneshot lanes. Unchanged saves, optional-write jobs, and saved pages with failed enrichment are positive controls. test/cycle/global-freshness-postcondition.serial.test.ts exercises the registered maintenance handler with failed phases, incomplete children, budget deferrals, abort/lock loss, and successful warning-only controls.

Native-matrix publication and managed writers

The glibc Linux, macOS and Windows matrix also runs persistence-publication-native.serial.test.ts, persistence-git-publication.test.ts, persistence-sync-origin-native.test.ts and backup-portability-native.serial.test.ts in separate Bun processes. These exercise real Git publication, historical source origins and backup restoration with fresh-process reopen. macOS and Windows set GBRAIN_TEST_REQUIRE_CASE_INSENSITIVE=1; a case-sensitive fixture must fail rather than silently skip the publication regression. Inspect executed test counts before claiming native coverage; a configured lane alone is not evidence.

The publication, sync-origin, processing-option and company-sync suites also run separately with an explicit PostgreSQL URL in the persistence deployment matrix. test/e2e/persistence-publication-parity.test.ts and test/e2e/persistence-sync-{origin,options,company}-parity.test.ts include them in the full local E2E gate; the keyless serial runner alone cannot exercise their PostgreSQL branches.

Focused safety coverage: test/apply-migrations-safety.serial.test.ts checks force dry-run previews before DB/ledger access and failed-phase partial exit; test/real-home-guard-preload.test.ts pins the test-home fingerprint backstop (detection, not prevention). Managed retry, durable diagnostics, restart and PGLite/Postgres parity are covered by test/persistence-sync-failures.serial.test.ts and test/e2e/managed-sync-failures.test.ts. Backup remote readback and fsync fault cases run in test/backup-verification.serial.test.ts and test/backup-fsync.serial.test.ts; test/e2e/backup-coverage-parity.test.ts covers PGLite/Postgres page/fact/config parity. Output redaction uses test/search/output-redaction.serial.test.ts and test/search/output-redaction.test.ts, including unchanged internal capture.

Managed writer fixtures use isolated PGLite and guarded disposable Postgres: test/e2e/fact-vector-repair-parity.test.ts, test/e2e/fact-embedding-backfill-parity.test.ts, and test/fact-backfill-resident.test.ts cover preserved vectors, bounded NULL-only fact backfill, selected-config refusal and owner-held PGLite IPC; test/ai/google-embed-batch-items.test.ts pins 100-item provider batches. test/persistence-embedding-effects.test.ts, test/persistence-effect-retry.test.ts, and test/embedding-completion-atomic.serial.test.ts cover partial vector completion, exhausted durable attempts and state-bound explicit retry. test/managed-extract-atoms.test.ts, test/managed-facts-backstop.test.ts and their test/e2e/ counterparts exercise admitted atom/fact replay, including fresh-process facts authority. test/persistence-connectors.test.ts covers managed bound/unbound Google/GitHub sources, API pagination and source-scoped deletions. test/persistence-connector-retry.test.ts covers explicit retry, compaction, checkpoint dependency identity, concurrent approval and lost acknowledgements. Each suite creates its own home, engines and lifecycle through test/helpers/connector-fixture.ts; the helper shares no live engine or mutable suite state. Their separate E2E entry points, test/e2e/managed-connector-routing.test.ts and test/e2e/managed-connector-retry.test.ts, retain the runner's default 180-second per-file cap without duplicating the base cases in the retry lane. Linux root runners execute the complete EACCES case in an isolated setpriv child and assert UID 65534 before testing permissions. This needs a readable checkout, not changes to the parent process identity or checkout permissions; the CI runner image supplies setpriv. test/managed-maintenance.test.ts and test/helpers/maintenance-restart.ts cover local synthesize/patterns/ consolidation, restart replay, retired takes and semantic snapshots; test/managed-unsupported-preflight.serial.test.ts checks that the managed facts-family bulk lanes refuse an unaccepted writer before spend, and test/managed-facts-writers.test.ts proves each of them (fence reconcile, phantom redirect, fence writes, loops extraction, bulk conversation facts) publishes through the coordinator on PGLite and Postgres. These use synthetic provider/API transports, not paid model calls or production connectors. PGLite dream/job CLI with an active owner is not proven delegated by the live fact-backfill IPC test.

test/facts-worker-config.test.ts and its PostgreSQL E2E counterpart dispose the original consumer before executing a real facts-absorb job. They verify the worker passes trusted selected configuration, ignores job-supplied configuration and settles the entity-page effect with zero fact or chunk embedding calls when disabled. Fact extraction still captures the generated fact with a NULL embedding.

test/managed-facts-embedding.test.ts and its PostgreSQL counterpart bind retained fact vectors to the selected brain's model and dimensions, including equal-width host/mount mismatches, keyless capture, policy changes and replay without new spend. test/managed-atom-regressions.test.ts and its PostgreSQL counterpart preserve later target edits through explicit retries and honor database-only storage policy without relaxing source authority. test/managed-synthesis-postprocess.test.ts and its E2E wrapper verify that completed quote/provenance work never rewrites a later user edit, while unfinished work resumes against its original revision. The synthesis suite also preserves the existing same-date summary on replay and rebuilds a complete index after partial recovery. test/managed-atom-compaction.test.ts and its PostgreSQL counterpart age and compact real receipts: permanent completion identity still prevents repeated extraction, while expired retry payloads produce an explicit refusal without changing terminal outcomes or compaction accounting. test/managed-facts-compaction.test.ts and its PostgreSQL counterpart cover the same lifetime boundary for explicit and derived fact-batch identities, including failed or partially committed batches and successful replay without new spend. Connector sweep fencing and physical-path normalization have separate parity coverage in test/persistence-connector-fencing.test.ts. Standalone crash/recovery cases live in test/persistence-connector-recovery.test.ts and their own E2E wrapper so they do not share the routing file's wall-clock budget; their original assertions, child watchdogs and per-file timeout are unchanged.

test/managed-atoms-cli.slow.test.ts exercises real disk-backed PGLite CLI recovery with a loopback provider: live-owner refusal, graceful owner stop, malformed extraction, explicit same-input retry, idempotent replay and owner restart. Fresh-process readback checks the private canonical file, searchable chunk, retained failure receipt, committed completion and released leases. test/managed-connector-routing.serial.test.ts pins actual activation and performSync routing for API sources; the maintenance suite also drives runCycle with eligible facts in two sources and proves the other source is unchanged. The E2E wrapper files ensure these optional PostgreSQL arms execute in the database lane rather than only passing their PGLite controls.

The persistence invariant jobs run the complete scripts/persistence/validate.ts gate (10,000-write soak) on pushes to master and manual dispatches. Pull requests run the same schedules and crash boundaries with a 2,500-write soak; the full PGLite soak alone takes 15-28 minutes and would otherwise set every PR's wall time. test/scripts/data-safety-native-workflow.test.ts pins the split. The crash robot (generated sequences of real operations, SIGKILL at every crash seam they reach, process faults, the reference model) runs as its own job beside the soak: 150 s on pull requests, 600 s elsewhere, Postgres through a transaction-mode PgBouncer. That job also replays the shrunk crash-robot regressions (test/persistence-crash-robot.slow.test.ts) and the history fixture test on both engines; see scripts/persistence/README.md.

For platform-only feedback, dispatch gh workflow run test.yml --ref <branch> -f native_only=true. This explicit manual option uses a separate concurrency group so it does not cancel an ongoing full persistence soak. Its native-only-validation-scope artifact records the exact commit and full_ci: false; it never emits the required test-status check for unrun full CI. Omitting the option preserves every normal PR, push and full manual gate.

Schedules and process crashes

test/persistence-consumer-scheduling.test.ts pins completion wake-ups, including a wake-up arriving during an active tick, without lowering the idle poll interval. Per-root deadlines preserve blocked/retryable backoff even while another root keeps committing; expired deadlines permit retries. Shutdown drains active preparation without starting another request. test/persistence-root-refresh.test.ts checks that unchanged root registrations do not replace their durable files while unbound, moved and original bound paths all remain fenced.

test/e2e/persistence-http-liveness.test.ts drives real authenticated HTTP MCP with legacy and source-bound OAuth credentials against an isolated PostgreSQL database. It covers pending page/fact writes, lost acknowledgments, owner interruption, exact canonical readback, 44 pages over three clients, and current receipt authorization. Set GBRAIN_TEST_OLD_BINARY to a retained compatible executable to additionally exercise old/new owner handoffs in both directions for queued, actually claimed and recovery-bearing rows. Interrupted owners are reaped and native exclusion is verified before their successors start; only the abandoned running lease is advanced by the fixture. An omitted executable skips these compatibility cases, not proves them. test/e2e/persistence-phase-liveness.test.ts holds real table/row locks and ordinary/direct pool slots to verify phase cancellation (including capacity marking), tracked queued BEGIN, renewal, expired-head FIFO and shutdown fences. A loopback TCP gate separately delays or rejects cold direct initialization, checking retained work through shutdown and same-ID completion after retry. The memory-mutation tests use a deterministic loopback embedding transport for slow and aborted preparation; they do not contact a paid provider. Receipt contract, MCP parser, IPC and CLI tests pin the same nested allowlist and advisory age policy. Diagnostic tests exercise both fresh/index-upgrade parity and a genuinely interrupted PostgreSQL concurrent index build when a PostgreSQL fixture is supplied.

test/persistence-chaos.slow.test.ts and test/e2e/persistence-chaos.test.ts execute real journal/coordinator schedules and eight SIGKILL publication boundaries, followed by a small multi-process soak. The Postgres test creates and drops fresh test databases, requiring CREATEDB on the explicit test URL. It never truncates the shared E2E database. The reusable persistence-validation.yml gate runs 1,000 schedules and 10,000 writes per engine under Bun 1.4.4 and uploads actual executed-case manifests; the sections above cover workloads, reruns, performance measurements and the process-crash scope.

test/e2e/persistence-runtime-matrix.test.ts additionally requires the real transaction-mode PgBouncer fixture. Its 24 cells exercise direct/pooler connections, enforced RLS under a non-bypass role, ordinary pool sizes 1/2/3, and shared pools or a separate one-connection direct route. It verifies reserved short control capacity while bulk connections remain held, then drains and commits the original request. Ownership cases cover mismatched successor manifests, stale owners, root replacement under a held kernel lock, and actual source deletion/recreation. The reusable persistence lane runs this matrix on every supported Bun version and uploads its manifest.

The required persistence lane also runs scripts/persistence/performance.ts on both engines and every supported Bun version. Three independent instances use the existing 500-page/200-query read-latency corpus, with public put_page mutations and actual in-flight interval coverage of at least 90%. Any read or write failure invalidates the sample. CI uses --informational: the 50% loaded-versus-idle p99 threshold is advisory, not a merge blocker. Loaded reads compete with additional write work, so this ratio alone is not evidence of a change regressing the same workload. Manifests retain the original threshold verdict, each sample, admission/commit latency, queue age, RSS, recovery bytes and pool activity. The CLI without --informational still enforces the threshold. The original heavy shell entry invokes this harness; its optional STRICT_LATENCY=1 flag affects only the latency threshold, never validity requirements.

Managed-sync convergence gate

test/sync-convergence.serial.test.ts (#6650, the #6712 L0 gate) is a real-process profile of this gate. It uses the same fault seam (src/core/persistence/fault-points.ts, preloaded into real gbrain serve and gbrain sync processes by test/fixtures/convergence-fault-kill.ts, since production never installs a hook). The preload acts once, at the nth hit of one point, through a registry of actions: kill (SIGKILL), stall (until the harness releases it), edit (an uncommitted worktree edit) and throw (an error carrying a given SQLSTATE, as the driver reports a database error). Each scenario builds its own Postgres database behind transaction-mode PgBouncer, reached through an in-test TCP proxy that adds the configured round-trip latency to the pooled and the direct URL.

The chaos modes, each named in failures and selectable with GBRAIN_TEST_CONVERGENCE_MODES:

Mode What happens Expected end state On master
(a) client kills, (b) serve kills A kill loop of GBRAIN_TEST_CONVERGENCE_KILLS change rounds, each ending in one kill -9 at a point of the curated list (publication:before_commit, publication:after_commit, consumer:preparing, lane:applied, sync:before_group_admission, sync:mid_checkpoint, sync:held_settled, sync:passed_over, sync:batch_entries, sync:tail_started), cycling a --no-delegate client, a serve running a delegated catch-up, and a serve with its delegating client; a sync:passed_over round syncs with --no-bulk, so its primed edit is admitted before any re-freeze can read it (#8838) strict convergence runs
(c) concurrent writers During the kill loop: gbrain put rewrites of 5% of the notes, remember facts on an entity page, gbrain extract timeline --source db (a managed maintenance job) strict convergence; every acknowledged write in the ledger runs
(c-db) DB-only competitor With sync.write_through=false, a DB-only put lands while the run is stalled on that page's frozen entry, and another on a page whose file did not change the raced page is held concurrent_write (needs a person), its file keeps the committed bytes and its page the DB write; both pages are left out of disk == DB runs
(d) out-of-band Git While a run is stalled mid-manifest: git mv and git rm of 1% of the files, committed, and deletion of 1% uncommitted moves and removals converge strictly; each uncommitted deletion is listed by gbrain pages ghosts with the gbrain repair ghost-pages exit and left out of disk == DB runs
(e) mid-run edit The sync:after_freeze hook writes an uncommitted edit to a seeded entry after it was frozen the run's receipt and sync status name worktree_dirty; after the editor commits, strict convergence runs (turned on by PR 1 of #6712)
(f) pooler drop Every pooled connection drops at the proxy after 20 committed pages of a bulk run (GBRAIN_TEST_CONVERGENCE_PGBOUNCER_CONTAINER=<name> restarts that container instead) no new hold; the drain resumes within the owner-kill stall bound; strict convergence runs (turned on by PR 2 of #6712: a pass whose connection died releases its unpublished claims once the database answers, instead of holding them under the publication lease)
(g) two drainers sync --all --no-delegate beside a serve draining a delegated sync of the same source strict convergence runs
(lease) drain-lease takeover A --no-delegate leader stalls right after it takes the source's drain lease (sync:lease_acquired), two --no-delegate followers queue behind it, and the leader is SIGKILLed a follower reports following, one takes over inside the lease TTL (2 min) and finishes synced, no follower ends blocked; strict convergence runs (#6712 PR 1)
(knock-on) One page's publication throws P0001 once inside a bulk group, then a trigger refuses one page with P0001 for a whole run no other page of the run is cancelled; strict convergence once the trigger is gone runs (turned on by PR 1 of #6712)
(systemic) A trigger on pages raises an unregistered P0001 (one scenario) or 55000 (another) on every page write for one run, then is dropped the refused run does not report success and its receipt names the SQLSTATE; strict convergence once the trigger is gone runs (turned on by PR 1 of #6712)

A skipped mode stays written; naming it in GBRAIN_TEST_CONVERGENCE_MODES runs it anyway, which is how the PR that fixes it turns it on.

After the chaos every scenario gets a fresh serve (what a supervisor restarts) and at most two plain gbrain sync --no-pull runs, with no operator step; only exit 0 with outcome synced counts, and a synced run that HEAD has moved past since (a queued write's Git commit landing during or after it, the commits named in the log) is followed by the second. Then the oracle, each check its own test named by this gate's violation class: disk == database both ways (except the pages the mode's end state names), no request queued, running, recovering or holding a recovery record, every non-committed terminal request superseded or carrying a registered code, the cursor at HEAD, no lost fact or timeline row on the large page (timeline duplicates are printed, not asserted), the next sync delegating to the live serve whatever heartbeat a killed client left, no sync after an owner kill stalling longer than the consumer liveness window plus one renewal and 30 s of slack, and two oracles independent of the brain's own bookkeeping:

  • The acknowledged-write ledger. Every writer operation that returned success (a harness commit, a put, a remember, a maintenance run, a DB-only write, a git mv/git rm, an uncommitted deletion) is recorded by the harness with the time it started and the time it was acknowledged, and where it landed (the page, its file, or an additive fact). For every page and location the final state must show a write that no later acknowledged write causally supersedes (none started after it was acknowledged); a remembered fact must survive unless a page write on its entity may have been ordered after it.
  • The status oracle. gbrain sync status --source <id> --json must carry every field the operator loop reads (checkpoint_blockers and cursors arrays, checkpoint_blockers empty, terminal_failures ok with no active blocker, needs_human, holds exactly the end state's, last_error and next null) and a finished cursor (done, or retired to null). checkpoint_blockers is present on every status, empty when no checkpoint is pending (test/sync-status-checkpoint-blockers-shape.test.ts pins the four cursor states); PR 2 of #6712 moves the heavy reads behind sync status --detail and the oracle then reads that. An always-on calibration test proves the oracle fails when any required field is missing.

Every random choice (victims, fault points and their nth hit, writer targets) comes from GBRAIN_TEST_CONVERGENCE_SEED, and every failure prints the seed, the scenario's modes, the fault plan, which faults fired, and a one-line repro command. [convergence <scenario>] lines print elapsed times and the longest stall each sync printed.

It skips unless GBRAIN_TEST_CONVERGENCE=1. Run it against the E2E Postgres plus a PgBouncer in front of it:

docker network create gbtest
docker run -d --name gbrain-test-pg --network gbtest -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test -p 127.0.0.1:5434:5432 pgvector/pgvector:pg16
docker run -d --name gbrain-test-pgbouncer --network gbtest -e DB_HOST=gbrain-test-pg \
  -e DB_USER=postgres -e DB_PASSWORD=postgres -e POOL_MODE=transaction -e AUTH_TYPE=plain \
  -e IGNORE_STARTUP_PARAMETERS=extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path \
  -p 127.0.0.1:6543:5432 edoburu/pgbouncer:latest
export GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@localhost:6543/gbrain_test
export GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test
GBRAIN_TEST_CONVERGENCE_LATENCY_MS=10 bun run test:convergence     # the PR cell
GBRAIN_TEST_CONVERGENCE_MODES=e bun run test:convergence            # one mode, skipped or not
docker rm -f gbrain-test-pgbouncer gbrain-test-pg && docker network rm gbtest

Knobs (GBRAIN_TEST_CONVERGENCE_*): SEED (default 6650), MODES (default: every mode not skipped), NOTES (default 200; the large page is always there), KILLS (default 6), LATENCY_MS (default 50; 0 for a fast local loop), SYNC_TIMEOUT_MS (per child process, default 900000), SCENARIO_TIMEOUT_MS (per scenario, default 3600000), LOG_DIR (keeps convergence.log and every client, serve and writer log per scenario) and PGBOUNCER_CONTAINER (mode (f)).

CI: the convergence job of .github/workflows/persistence-validation.yml, on every supported Bun version with the deployment matrix's pgvector and transaction-mode PgBouncer services. Pull requests run it when scripts/ci-convergence-scope.sh selects their diff (persistence, sync, projections, storage, the harness and its workflow; never package.json alone; bun.lock and vendor/postgres/ select it, since a driver change is a persistence change); merge-queue runs and pushes always run it. Those run the PR cell: 200 notes at 10 ms, 6 kills, seed 6650.

The nightly cell is its own scheduled workflow, .github/workflows/convergence-nightly.yml ("Convergence nightly", its own concurrency group, outside test.yml and its test-status gate): 2,000 notes at 50 ms, 20 kills, the run id as the seed (workflow_dispatch takes a seed to replay one), with a 300-minute step timeout inside a 330-minute job, so the cleanup step (it kills any harness process a timed-out step left), the failure summary (every failing test with its seed, fault plan, fired faults and repro line) and the upload of LOG_DIR still run. The nightly watcher (nightly-watch.yml) files its failures like every other scheduled workflow.

Busy-agent soak cell

test/busy-agent-soak.serial.test.ts (L5) runs the convergence gate's real-process brain (test/helpers/convergence-brain.ts, shared with test/sync-convergence.serial.test.ts: Postgres behind transaction-mode PgBouncer through latency proxies, real src/cli.ts processes, a resident serve, the fault seam, the acknowledged-write ledger and the oracle) under the workload a busy personal agent puts on one managed checkout. Its counts and cadences come from a profile: the committed generic one, test/fixtures/busy-agent-profile.json, unless GBRAIN_TEST_BUSY_AGENT_PROFILE names another (a profile derived from a real deployment stays private). A profile holds exactly the test's PROFILE_SHAPE keys, each a positive integer, plus an optional description; it never holds content.

Mode What happens Expected end state
(h) refresh ticks With persistence.refresh_barrier on, scheduled sources refresh ticks pull upstream commits while foreground puts land inside the fence, some on notes the tick changes, and an agent writes across every tick every tick's refresh completes and releases its fence; every write that reached the refreshing checkout is answered pending with a receipt naming the refresh, never refused, and commits after it, or ends revision_conflict with nothing applied when the refresh changed its file
(i) foreign writers Plain-git committers and an uncommitted-file writer beside gbrain doctor foreign_writers names every commit gbrain did not make and none it made, and the aged uncommitted files
(j) legacy rows Rows with no import record in every content-evidence direction sources git-gap classifies each by its direction evidence; both_moved and no_evidence rows end held file_drift
(k) strict conversion sources canonical --to git over several adopt groups, stalled after the first (conversion:group_committed) while another clone pushes, door writes land and a refresh runs the conversion ends converted, strict, with nothing left to plan; every door write and refresh is accepted or refused with a registered code; the other agent's commit reaches HEAD
(l) segmented staging A segmented write whose staging expires or is aborted at segmented:manifest_staged the write refuses body_staging_expired, the page keeps its body, a resend commits and the sweep leaves no live staging
(m) lost update A file-writing cron (git commit-push --base) and an MCP edit_page writer race on one page edit_page refuses revision_conflict, commit-push --base refuses commit_base_moved, and no acknowledged write is silently reverted

Every scenario then gets the convergence gate's oracle (above). It skips unless GBRAIN_TEST_BUSY_AGENT=1; with the Postgres and PgBouncer of the gate above:

GBRAIN_TEST_BUSY_AGENT_LATENCY_MS=0 bun run test:busy-agent      # every mode, fast local loop
GBRAIN_TEST_BUSY_AGENT_MODES=k bun run test:busy-agent           # one mode

Knobs (GBRAIN_TEST_BUSY_AGENT_*): SEED (default 8800), MODES, PROFILE, LATENCY_MS (default 50), SYNC_TIMEOUT_MS (default 900000), SCENARIO_TIMEOUT_MS (default 3600000) and LOG_DIR.

CI: the busy-agent job of .github/workflows/convergence-nightly.yml, on the same runner class, at 50 ms with the run id as the seed. It is informational (continue-on-error) for its first seven nights; after that a release touching persistence, sync or Git needs it green on the release's exact sha.

Engine graduation tests

Graduation (gbrain migrate --to postgres) is tested against two fixtures. test/fixtures/graduation/legacy-brain.ts builds a small brain by hand-written SQL on a fresh schema, and its expected outcomes live in expected.json beside it (test/graduation-legacy-fixture.test.ts proves the build matches and that the target checker discriminates). scripts/persistence/graduation-fixture.ts wraps buildHistoryFixture for the 1k and 10k history brains, builds keylessly in a child process, and caches each build as a tarball keyed by the fixture sources, schema version, seed and size (GBRAIN_GRADUATION_FIXTURE_CACHE, default ~/.cache/gbrain-graduation-fixtures). A restore re-homes paths and owner stamps and marks sources synced, so the source doctor stays green.

The E2E suites drive the real CLI in child processes, each split into files of about two minutes or less so CI queues spread them: graduation-crash-run-N and graduation-crash-rollback-N (SIGKILL at every run and rollback boundary), graduation-faults (ENOSPC on a tmpfs tablespace, which needs Docker; password rotation; DDL route mismatch), graduation-clients (older releases) and graduation-clients-serve (respawned and resident serve, stale CLI and MCP configs), and graduation-cli (agent flow, --force), graduation-cli-topologies (PgBouncer through GBRAIN_PGBOUNCER_URL, a NOSUPERUSER role), graduation-cli-history (the 1k round trip) and graduation-cli-zero-mutation-N (zero-mutation --plan/--status, each file probing a share of the custody boundaries). Kill and pause points come from graduationBoundary() hooks that only test/helpers/graduation-hooks-preload.ts registers. Older release binaries are built once per tag under GBRAIN_OLDER_RELEASE_DIR. A crash case takes about 16 seconds; scripts/run-e2e.sh gives every graduation file four times its per-file cap. scripts/persistence/graduation-ttv.ts records the commands and wall time from the first plan to a green doctor, the run's phase timings and query p50/p95 on both engines. The 1k-page gate is five minutes; tests/heavy/graduation_10k.sh reports the 10k run.