Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript test suites as a parity oracle
Line numbers refer to gbrain 36bd05806a.
Scope: gbrain v0.60.733.0 (commit 36bd05806a) at garrytan/gbrain, read only. Nothing was run; all counts come from find/rg over the checkout this session. "Verified" means read or counted in code or docs. "Inferred" means drawn from names, headers or docs without running the test.
1. Headline numbers (verified)
- 6,486
*.test.tsfiles undertest/(6,489 repo-wide; the 3 extra sit outsidetest/). - About 53,100
test(/it(call sites by regex. Approximate: nesteddescribeand.eachexpansion are not counted. - Disjoint tier buckets, by precedence (e2e directory, then
.serial,.slow,.live, unit remainder): e2e 923, serial 476 (outside e2e; 502 total including e2e), slow 62, live 1 (2 more live files sit intest/e2e), unit 5,024. These sum to 6,486. - Only 81 test files spawn the CLI without importing
src/directly or through a helper (see section 7). Roughly 500 test cases are that black-box set.
2. Tier model
Lanes are defined in docs/TESTING.md ("File taxonomy", l.1389; "Test command tiers", l.402) and the event matrix in docs/ci-event-parity.md.
| Tier | Files | Selector / command | Database | CI cadence (from docs) |
|---|---|---|---|---|
| unit | 5,024 | *.test.ts outside test/e2e; bun run test (scripts/run-unit-parallel.sh) |
PGLite; DATABASE_URL unset in the lane |
every PR, 10 weighted shards |
| serial | 476 (+26 in e2e) | *.serial.test.ts, listed in scripts/serial-files.tsv (476 rows); bun run test:serial |
PGLite | 4 pooled workers, one Bun process per file |
| slow | 62 | *.slow.test.ts; bun run test:slow |
PGLite | named slow jobs or unit shards |
| live | 3 | *.live.test.ts, test/live/; paid provider calls |
real providers | never in PR CI |
| e2e | 923 | test/e2e/*.test.ts; bun run test:e2e (scripts/run-e2e.sh) |
real Postgres via DATABASE_URL; skipped when unset |
PR gets a selected subset; full corpus on master, merge queue, nightly |
| heavy | 9 .sh (2 are _ helpers) + 3 _*.ts workloads in tests/heavy/ |
bun run test:heavy |
Postgres | label-gated on PR; nightly heavy-tests.yml |
| fuzz | 3 | test/fuzz/*.test.ts |
none | in unit lane (about 3 s) |
| admin | 29 files with admin in the name; 1 Playwright admin/e2e/*.pw.ts |
bun run test:admin (PGLite in-memory, Chromium) |
PGLite | required admin-browser job |
| stress | harness scripts/stress/ + 6 name matches |
bun run test:stress; PR gate stress-changed-tests runs each touched file 10x |
Postgres arms | PR (changed files), nightly race hunt |
| scale | harness scripts/scale/ + about 15 name matches |
bun run test:scale (package.json l.103) |
both | scale-tier.yml, nightly 10k-50k |
| convergence | test/sync-convergence.serial.test.ts (+ helpers) |
bun run test:convergence sets GBRAIN_TEST_CONVERGENCE=1 (l.107) |
PGLite | convergence-nightly.yml; PR cell in persistence-validation.yml |
| busy-agent | test/busy-agent-soak.serial.test.ts, profile test/fixtures/busy-agent-profile.json |
bun run test:busy-agent sets GBRAIN_TEST_BUSY_AGENT=1 (l.108) |
PGLite | dispatch-driven |
| tla | spec/tla/ (2 modules, 15 .cfg) |
bun run test:tla (l.109) |
none for TLC | tla-spec.yml nightly and on touching PRs |
Per-directory split (verified; unit / serial / slow): root test/*.test.ts 4,168 / 418 / 46; large-pages 164 / 3 / 12; scripts 152 / 0 / 3; pathological 127 / 2 / 0; search 111 / 14 / 0; ai 80 / 4 / 0; cycle 36 / 2 / 0; skillopt 32 / 5 / 0; decide 15 / 22 / 0; chunkers 17; ingestion 12; core 10 / 3 / 1; minions 9 / 1 / 0; audit 9; facts 8; conversation-parser 8; brainstorm 8 / 2; code-intel 6; then 1 to 5 files each for enrich, context, budget, bench, regressions, fuzz, extract, schema-pack, progressive-batch, fixtures, eval-contradictions, commands, eval. test/e2e holds 923 files; test/helpers holds 12 *.test.ts plus about 200 helper modules.
Note that decide/ is serial-heavy (22 of 37 files), so its tests are process-isolated by design.
3. Database requirements: PGLite versus DATABASE_URL
- Unit, serial and slow lanes unset
DATABASE_URL(docs/TESTING.md l.271). A file whose Postgres arm is gated on it runs that arm only where a Postgres lane names it: a workflow step, atest/e2e/wrapper (registerPostgresTests), atests/heavyscript, or a row inscripts/e2e-backend-matrix.txt(77 rows).test/postgres-unit-arms.txt(776 rows) lists unit files that run their Postgres arm inpersistence-validation.yml. test/helpers/test-backends.ts:requirePostgresTestDatabase(l.6) refuses to skip a Postgres arm silently.testBackends()(l.15) returns['pglite','postgres']whenDATABASE_URLis set and['pglite']otherwise;GBRAIN_TEST_BACKEND=postgres|postgres-direct|pgbouncerforces the Postgres arm.registerPostgresTests()(l.24) wraps loaders withGBRAIN_TEST_BACKEND=postgres.
- Counts (verified by
rg -l): 831 files calltestBackends(; 356 files mentionregisterPostgresTests; 284 of the*-postgres.test.tswrappers call it; 1,022 files readprocess.env.DATABASE_URL. test/e2e/files are gated: 399 mentionDATABASE_URL; 181 have neither a gate nor a registration call (inferred: these run withDATABASE_URLset by the runner, or are skipped).- PGLite has a default-on schema snapshot (
test/fixtures/pglite-snapshot.tar, generated byscripts/build-pglite-snapshot.ts; docs l.571). Every runner activates it, so the Rust port has no equivalent for the snapshot itself.
4. Two-engine parity helpers
test/helpers/two-engine.ts(366 lines) runs two brains in one process with separate homes, AI keys, embedding models and fake OpenAI-compatible providers.openTestBrain(l.57) is the seam that later waves replace.CASES(l.114): 26 isolation cases.EXPECTED_FAILURES(l.308): 13 rows, each naming the wave that removes it (W2, W5, W6). These run astest.failing, so an unexpected pass fails the suite.twoEngineIsolationSuite(kind, urls)(l.324) is run fromtest/two-engine-isolation.serial.test.ts(PGLite) andtest/e2e/two-engine-isolation.test.ts(Postgres).
test/helpers/engine-sql-capability-cases.tspins three dialect capabilities, each with a boundary-size and a concurrent-write case:maxBindParamsPerStatement(PGLite splits code-edge inserts below 30,000 binds),transactionAdvisoryLocks(Postgrespg_advisory_xact_lockon same-entity fact inserts), andprobesEmbeddingCast(::vectorversus live column type). Registered fromtest/engine-sql-capabilities.test.tsandtest/e2e/engine-sql-capabilities-parity.test.ts.- Other engine-sql parity files:
test/e2e/engine-sql-prepare-parity.test.ts(pg_prepared_statements),test/e2e/engine-sql-normalize-parity.test.ts(column decode shapes),test/e2e/engine-sql-rls-scope.test.ts(non-ownerNOBYPASSRLSrole),test/e2e/engine-sql-transaction-parity.test.ts. - Name-level: 56
*parity*.test.tsfiles exist across the tree; 36 sit undertest/e2e. - Executor-level:
test/executor-binding-matrix.test.ts(PGLite arm) andtest/e2e/executor-binding-matrix.test.ts(run twice: direct and PgBouncer, withprepare=false). - Shared scenario code across two engines is not treated as duplicate coverage (docs l.209). Engine-specific boundaries (JSONB, locks, pooler) are owned by the Postgres lane.
5. Golden and contract tests
test/helpers/golden.tsis the golden harness. Goldens live undertest/fixtures/goldens/(59 files:doctor17,sql-text14,catalog10,cli8,pglite-upgrade-replay5,decide3,trust2). Each golden uses a named normalizer proven stable by capturing twice. Regenerate withGBRAIN_TEST_UPDATE_GOLDENS=1. 32 test files import the harness.- Most relevant to a Rust engine:
test/cli-goldens.test.ts:--helptext,--version,--tools-jsonbytes (sha256 pinned), unknown-command and unknown-flag exit codes and messages. UsesrunCliBatchagainst the realsrc/cli.ts(test/fixtures/goldens/cli/help-and-errors.json,tools-json.json).test/cli-json-commands.test.tsand the agent-contract tests: under--json, stdout is exactly one document or NDJSON, and failures are envelopes withcode,suggestion,contract_version: 1.doctorgoldens (test/helpers/doctor-json-golden.ts).
test/engine-sql-sql-text.test.tspins SQL text byte-for-byte. It is a conversion guard, not product behavior; it dies with the SQL layer.- Refactor-wave goldens pin output across rewrites; the docs say a golden is "a reviewer-visible diff" when regenerated (golden.ts header).
6. TLA+ specs
spec/tla/README.mddescribes TLC checks plus mutants.bun scripts/spec/mutants.tsapplies each mutant to a model copy (TLC must report a violation) and to the code (the named test must fail).spec/tla/drain-lease/DrainLease.tla(344 lines): the managed-sync drain lease. One drainer per source. The lease is one row ingbrain_cycle_locks; the acquisition token is the fencing token. Fenced transactions take the rowFOR KEY SHARE(#7442). The model is Postgres only; PGLite is one connection. Invariants:OneDrainer,FencedTxnsMatchRow,NoSpuriousLapse,CursorMonotone. Code:src/core/persistence/sync-lease.ts,src/core/db-lock.ts. 14 mutants (mutants.json). Configs:DrainLease.cfg,DrainLeaseSafety.cfg,DrainLeaseStamp.cfg,DrainLeaseLanesNightly.cfg.spec/tla/refresh-barrier/RefreshBarrier.tla(884 lines): the refresh write barrier and deferred-write queue for one managed worktree and one page. Models Postgres's four row-lock modes, deadlock as a safety property (40P01), and crash-and-recover of the refreshing process (MaxCrashes). Code:src/core/persistence/journal.ts,worktree-refresh.ts,refresh-barrier.ts,shared-skills/restore.ts. 22 mutants. 11.cfgfiles cover adoption, import holds, liveness and nightly variants.- Test linkage:
test/spec-drain-lease-constants.test.tschecks the.cfgtiming constants against the code (per theDrainLease.tlaheader, l.13).
7. Black-box versus white-box
Method: for every *.test.ts I classified (a) direct imports from src/ (relative specifiers), (b) CLI spawn (test/helpers/cli-spawn.ts or a literal src/cli.ts), and (c) whether the helper modules the test imports reach src/ at depth 1. Depth-1 is an upper bound, because transitive helper chains were not followed.
| Class | Files | Test blocks (approx.) | Notes |
|---|---|---|---|
Direct src/ importers (white-box) |
5,545 | ~50,200 | Call functions and types from src/. Not portable. |
No direct src/ import |
941 | ~2,900 | Mixed: see below. |
| ...of which read source or doc text | 227 | Structural guards (test-reads-source-ok, structural-suites.tsv, 488 rows). Not portable. |
|
| ...of which spawn the CLI | 88 | 7 reach src/ via a helper (depth 1). |
|
| Black-box candidates (spawn only, clean at depth 1) | 81 | ~496 | 64 root, 10 e2e, 7 scripts/. |
The 81 black-box files include cli-goldens.test.ts, cli-json-commands.test.ts, cli-help-discoverability.test.ts, doctor-cli-smoke.test.ts, export-help.test.ts, e2e/serve-http-oauth.test.ts, e2e/sources-remote-mcp.test.ts and e2e/zombie-reaping.test.ts. Their assertions are on stdout bytes, JSON envelopes, exit codes and files written under a temp HOME. Two examples were read: test/cli-json-commands.test.ts (imports only bun:test, Node built-ins and ./helpers/cli-spawn.ts) and test/helpers/cli-spawn.ts.
The 941 non-importers are not uniformly portable. The 7 that reach src/ through a helper (for example test/e2e/agent-journey-postgres.test.ts, test/serve-stdio-bound-explicit-read.test.ts) can become portable only after the helper is split into a protocol layer.
Most test/e2e wrappers are white-box at the loader level: registerPostgresTests calls scenario modules that import src/. For example the 349 files using registerPostgresTests depend on test/helpers/*-scenarios.ts, which are white-box.
Estimate of reuse:
- Run unchanged with a binary seam: about 500 cases in 81 files.
- Portable after helper refactoring: about 2,400 cases in roughly 860 non-importers, but most are scenario or source-text tests that need per-file review.
- Not portable: about 50,000 cases in 5,545 direct importers, plus the source-text guards.
Cases that a Rust engine can check beyond the 500 are only the ones that have a CLI or MCP surface and a DB-free fixture.
8. Harness proposal (GBRAIN_BIN)
- The seam already exists.
test/helpers/cli-spawn.ts:63hard-codesCLI_PATH = src/cli.ts;:73exports_setSpawnTarget;:116builds[process.execPath, '--no-env-file', CLI_PATH]unless overridden. ReadGBRAIN_BINthere: if set, spawn[GBRAIN_BIN]and drop--no-env-file. Ten files acrosstest/,scripts/andtests/already useGBRAIN_BIN(for exampletest/plugin-turn-hook.test.tsandscripts/dx-explore.ts:834), so the name is in use. - The env contract the Rust engine must honor (already set by the helper):
HOMEandGBRAIN_HOMEpinned to a temp dir;DATABASE_URL,GBRAIN_DATABASE_URL,GBRAIN_REMOTE_CLIENT_SECRETstripped;GBRAIN_SKIP_STARTUP_HOOKS=1. - Run the 81 files from a TS checkout with
GBRAIN_BIN=<rust-engine CLI> bun test <files>. Bun stays as the test driver, so the Rust team needs no new runner for the first pass. - Track results in a ledger keyed by (file, test name), following the identity model in
scripts/ci-executed-counts.ts(docs l.121). Mark each as pass, fail, or expected divergence, the last with atest.failing-style row likeEXPECTED_FAILURES. - Derive new Rust-native black-box cases from the goldens (
cli-help-v1,cli-tools-json-v1) rather than from TS internals.
Rust engine implications
Must preserve exactly (verified against the black-box set):
- CLI surface:
--helpand--versiontext,--tools-jsonbytes, unknown-command and unknown-flag exit codes and messages (cli-goldens). --jsoncontract: one document per command, NDJSON where stated, error envelopes withcode,suggestion,contract_version: 1.- Environment handling:
HOME,GBRAIN_HOME,GBRAIN_SKIP_STARTUP_HOOKS, and strippedDATABASE_URL. The Rust engine must not read a.envthe tests strip. - Doctor output shape (
doctorgoldens).
Can be simplified or dropped:
- Two-engine parity (
two-engine.ts,testBackends,registerPostgresTests, the*-parityfiles, the PgBouncer backend matrix,postgres-unit-arms.txt). One engine means one arm. - PGLite schema snapshot and its CI cache machinery.
- The isolation lint R1 to R9 (
scripts/check-test-isolation.sh). These rules guard Bun's module registry andprocess.env. The lint's allowlist (63 lines) disappears with the TS code. - SQL-text goldens (
engine-sql-sql-text) and the engine-sql capability cases, which describe Postgres dialect behavior. - Structural source-text guards (227 files), which check TS source shape, not product behavior.
Postgres features each test depends on (what the Rust engine must replicate or replace):
pg_advisory_xact_lock: fact-insert serialization (transactionAdvisoryLocks).- Row locks and
FOR KEY SHARE: the drain-lease fence (DrainLease.tla) and the refresh barrier (RefreshBarrier.tla, four lock modes, deadlock as a safety property). NOBYPASSRLSandFORCE RLSroles:engine-sql-rls-scope, projection-statistics tests.- pgvector HNSW plan choice and
vector/halfveccasts:vector-candidate-safety-postgres,vector-plan-real-column-postgres. pg_prepared_statementsandprepare=falsebehavior behind PgBouncer:engine-sql-prepare-parity,backend-matrix.- Inferred, not mapped:
LISTEN/NOTIFYandSKIP LOCKEDqueues are presumably exercised by the minions and persistence suites.
Risks:
- Oracle narrowness: only about 500 cases are truly black-box. Most of the 6,486 files would need rewriting, not reuse. Plan a Rust-native suite from the goldens and the CLI contract, not a rerun of TS unit tests.
- Helpers couple to
BrainEngineinternals (for exampleengine.executeRaw,engineSqlExecutor). Any Rust port of the helpers needs a new public seam. - Timing ceilings are tuned for Bun and WASM PGLite (per-file caps, 1536 MB per slot, 820 s crash-robot ceiling). Do not carry them over unchanged.
- TLA+ models are Postgres-specific. If the Rust engine keeps a single process and in-process locks,
DrainLeaseandRefreshBarrierneed a new model, not a port. - Goldens are generated on TS master. A Rust output diff must be classified as intended or a bug, with a documented exception list (the
EXPECTED_FAILURESpattern works).
Open questions
- Is the oracle the TS binary's behavior (goldens and CLI contract), or the TS test files themselves? The answer decides whether to port tests or write new black-box cases.
- Which goldens are contract-bearing and must stay byte-identical (
cli,tools-json,doctor), and which are TS-formatting artifacts? - Does the Rust engine keep multi-process writers? If not,
DrainLeaseand the refresh barrier stop applying, and the crash-robot scheduling need not be ported. - Who owns the
GBRAIN_BINseam in the TS test tree? The seam must be added either in a branch of gbrain or in a copied test tree. - Should the 3 live provider files and the
evalssuite (21 files ingbrain-evals/test/eval/, mostly pure eval logic) run against the Rust engine? They depend on provider keys and agbrainpackage import (one file importsgbrain/engine). - Depth-1 import analysis may undercount black-box tests. Is a transitive pass worth the cost before committing to the 81-file estimate?