Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Update the key-files entries for the recovered engine surface and segmented write refusals

garrytan6796450f732d
diff --git a/docs/architecture/key-files/core-services-2-continued.md b/docs/architecture/key-files/core-services-2-continued.md
index 990618b29d..de2a931486 100644
--- a/docs/architecture/key-files/core-services-2-continued.md
+++ b/docs/architecture/key-files/core-services-2-continued.md
@@ -24,7 +24,7 @@ The page-state concurrency foundation under `src/core/page-state/` and the share
- `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path).
-- `src/core/page-body/page-shape.ts` — Wave B PR 1 (spec 1.5): segmented page bodies are unreachable as strings. `guardSegmentedBody` (called by `rowToPage` for a row with `body_storage = 'segmented'`, so by every snapshot and `getPage` read) replaces `compiled_truth` / `timeline` with enumerable getters and setters that throw `page_body_segmented`, so property reads, spread, `JSON.stringify`, `structuredClone` and assignment all fail loud instead of seeing an empty body. `isSegmentedPage`, `assertInlineBody(page, site)` (explicit refusal where a writer would otherwise act before a getter fires, e.g. SQL-only writers; every `page_body_segmented` refusal's fix reads the page with `get_page body: 'auto'`), `pageBodyBytes(page)` (`body_bytes` for segmented pages), `pageBodySegmentedError` and `pageBodyOverCeilingError` (the refusal of a fence or timeline write through the put path on a page over 24 MiB; `wholeBodySnapshot` in `persistence/segmented-apply.ts` reads a page within it back as strings and refuses above it: semantic timeline writes, `takes-prepare.ts`, `facts-prepare.ts`, `reconcile-state.ts`, `gbrain core`). Tag writes on a segmented page never read the body: `prepareSegmentedTagMutation` (`persistence/page-prepare.ts`) flips them in metadata over the live manifest (`test/large-pages/segmented-semantic-writes.test.ts`). `bodyUnread(page)` says a segmented page's body was not read back as strings (over the single-string ceiling, see `materializedPage`); such a body was written by the streamed import, which holds fenced files, so `fenceBody(page)` reads it as empty for the fence merges and `withoutBody(page)` copies the page's other fields for a render whose body comes from the incoming write (a shrunk file rewrites such a page: `test/large-pages/segmented-shrink-below-ceiling.test.ts`). `Page` stays one shape (`body_storage?`, `body_bytes?`); the static half is `scripts/check-page-body-readers.ts` with `scripts/page-body-readers-baseline.tsv` (docs/TESTING.md#page-body-readers). Pinned by `test/large-pages/segmented-body-unreachable.test.ts`.
+- `src/core/page-body/page-shape.ts` — Wave B PR 1 (spec 1.5): segmented page bodies are unreachable as strings. `guardSegmentedBody` (called by `rowToPage` for a row with `body_storage = 'segmented'`, so by every snapshot and `getPage` read) replaces `compiled_truth` / `timeline` with enumerable getters and setters that throw `page_body_segmented`, so property reads, spread, `JSON.stringify`, `structuredClone` and assignment all fail loud instead of seeing an empty body. `isSegmentedPage`, `assertInlineBody(page, site)` (explicit refusal where a writer would otherwise act before a getter fires, e.g. SQL-only writers; a read's `page_body_segmented` refusal fixes with `get_page body: 'auto'`; for a write that fix changes nothing, so `asSegmentedWriteRefusal` (called by MCP dispatch on a mutating op) swaps it for a report-only step naming `edit_page`, `append_page`, `add_tag` and `remove_tag`, and `readsPageInPieces(fix)` tells the two apart, which remember's unattributed-save fix in `persistence/verb-errors.ts` also uses), `pageBodyBytes(page)` (`body_bytes` for segmented pages), `pageBodySegmentedError` and `pageBodyOverCeilingError` (the refusal of a fence or timeline write through the put path on a page over 24 MiB; `wholeBodySnapshot` in `persistence/segmented-apply.ts` reads a page within it back as strings and refuses above it: semantic timeline writes, `takes-prepare.ts`, `facts-prepare.ts`, `reconcile-state.ts`, `gbrain core`). Tag writes on a segmented page never read the body: `prepareSegmentedTagMutation` (`persistence/page-prepare.ts`) flips them in metadata over the live manifest (`test/large-pages/segmented-semantic-writes.test.ts`). `bodyUnread(page)` says a segmented page's body was not read back as strings (over the single-string ceiling, see `materializedPage`); such a body was written by the streamed import, which holds fenced files, so `fenceBody(page)` reads it as empty for the fence merges and `withoutBody(page)` copies the page's other fields for a render whose body comes from the incoming write (a shrunk file rewrites such a page: `test/large-pages/segmented-shrink-below-ceiling.test.ts`). `Page` stays one shape (`body_storage?`, `body_bytes?`); the static half is `scripts/check-page-body-readers.ts` with `scripts/page-body-readers-baseline.tsv` (docs/TESTING.md#page-body-readers). Pinned by `test/large-pages/segmented-body-unreachable.test.ts`.
- `src/core/page-body/limits.ts` — Wave B size classes, ceilings, retention numbers and config keys in one place. `readPageBodyPolicy` is the single question every write asks: `segmented` is the effective `pages.segmented_storage` (unset means `SEGMENTED_STORAGE_DEFAULT`, `on`; `off` or an unrecognized value is off; held off while `segmentedWriterBlockers` finds a recent writer that blocks, and an unreadable gate counts as blocked), `maxBytes` the engine ceiling (256 MiB Postgres, 128 MiB PGLite, lowered by `pages.max_bytes`) while on and `PAGE_MAX_BYTES` while off. Config is read per call; only the writer gate is cached (`SEGMENTED_WRITER_GATE_TTL_MS`, `_resetSegmentedWriterGateForTests`). `chooseBodyStorage` applies the 1 MiB / 512 KiB hysteresis. The writer gate (`segmentedGateWriters` in `persistence/writer-versions.ts`) reads seven days of admitted requests in any state, committed publications, `ingest_log` rows and, on Postgres, `persistence_consumers` heartbeats; a row blocks when its version is below `SEGMENTED_WRITER_FLOOR` (`page-body/writer-floor.ts`: the `since` of the BEHAVIOR_CHANGES row with id `segmented_switch_on`, this build's VERSION while it is `TBD`, so the merge-slot stamp edits only that row) or unparsable, or when it lacks the segment-reader mark (migration v254: `admitter_segment_reader`, `consumer_segment_reader`, `persistence_consumers.segment_reader`, `ingest_log.writer_segment_reader`), because releases 0.60.252.0-0.60.262.0 have no segment readers and a version string cannot tell them apart. Not seen: an older binary's unmanaged single-page write, a run that dies before it logs, and read-only or export use. `pageRecoveryBytes` is the default `persistence.limits.worktree_recovery_bytes` (read by `persistence/limits.ts` `readJournalLimits` in its one config statement): twice the larger of the body the configured switch admits (`segmentedMaxBytes`, or `PAGE_MAX_BYTES` while off) and the largest page stored in segments (deleted pages included; one probe of `pages_segmented_body_bytes_idx`, v260, or the engine ceiling while that index is not valid, so a limit read never scans `pages`) plus `RECOVERY_FILE_HEADROOM_BYTES` per copy, never below 256 MiB, because a restore reserves its file twice; it ignores the writer gate, and turning the switch off or lowering `pages.max_bytes` never strands a stored page. `readPageBodyPolicy` treats a failed read of the switch or `pages.max_bytes` as off, never as unset. Doctor `segmented_storage` reports a hold left only by this host's pre-upgrade writes as information with its clear time when, for each, this build has since made the same kind of write (role and principal, `segmentedGateEvidence` `first_seen`) after it; otherwise (a cron or npx gbrain on the host) it warns. `PROJECTION_LOCK_HOLD_BYTES` (512 KiB of segments, an occurrence larger than that alone) and `PROJECTION_RETIRE_MAX_ROWS` (2,500 rows) bound what one projection transaction inserts or deletes while it holds the page's key lock (`page-body/ingest.ts` `stageOccurrenceChunks`, `page-body/projection.ts` `retireStaleChunks`), about a tenth of the 1 s `lock_timeout` a write to the page, or a source-exclusive write anywhere on the source (the lock's `page_write_guards` insert key-share locks the source row), waits under in its publication, so a write never waits out its bound behind a background build (`test/large-pages/projection-lock-hold.test.ts`). Tests: `test/large-pages/segmented-switch.test.ts`, `test/large-pages/recovery-budget-default.test.ts`, `test/large-pages/recovery-budget-ceiling.slow.test.ts`.
- `src/core/ops/page-upload.ts` — the five upload-session operations (`begin_page_upload`, `upload_page_part`, `get_page_upload`, `commit_page_upload`, `abort_page_upload`; write scope, client slug-fence allowlist). Begin routes the source and slug as put_page does, runs put_page's grant and visibility authorization before reading the page (a private page answers as a missing one), and reserves the declared bytes; every later call re-checks the binding (creating principal, write source inside the read scope, source incarnation, the page id or, for create, a still-absent slug) and answers any mismatch with an `upload_not_found` identical to an unknown id; abort needs only the creating principal, its source and the slug fences, so a session whose page or incarnation moved can still be freed. A refusal at commit (stale revision, sha mismatch) and a commit whose write ended without committing discard the session; only a write whose `body_ref` names the session is recorded as its commit, so a begin request_id another write already used attaches nothing. Commit claims the session with an atomic open-to-sealed `sealUpload` before hashing; a call that loses the claim, or finds a lapsed session another call already moved on (`settleLapse`), reads the session again and follows its real state (sealed or committing replays the commit, committed returns the first receipt), so `upload_expired` is answered only when this call expired it (`uploadCommitProbe` is the forced-race seam, `test/large-pages/upload-review.test.ts`). After the hash check commit submits an ordinary put_page carrying `body_ref` (`stagedBody`), so a second commit replays the first receipt; a replace re-checks `expected_revision` and the shrink guard. Append commits through `admitPageAppend` (persistence/page-append.ts) with the staged `body_ref`. Begin admits create and replace up to the page ceiling (`pages.max_bytes`, the engine ceiling by default) and an append up to PAGE_MAX_BYTES; over PAGE_MAX_BYTES a remote writer is refused `file_too_large` at begin where the write gate applies to an agent-written page.
- `src/core/persistence/page-upload.ts` — upload-session storage: `page_body_stagings` rows of kind `upload` and their raw `page_body_staging_parts`, `part_max_frame_bytes` from the transport's `ctx.inboundFrameBytes` (at most 4 MiB, minus a 4 KiB envelope), part UTF-8 and code-point checks, the staged-quota reservation (limits.ts `stagedCounterKeys`, recounted under the counter locks at begin) and its release on commit, abort or expiry (`transitionUploads` with `RELEASE_STAGED_CTE`, which deletes the parts in the same statement; the segment sweep expires lapsed sessions the same way). An open session expires past its idle or hard deadline, a sealed one only past its hard deadline, which sealing extends by one idle period; a lapsed session is never sealed, so the brain-wide sum at begin leaves out other principals' lapsed sessions; `expireIfLapsed` reports whether its update changed the row. A session takes at most `uploadPartLimit` parts (64 plus one per 64 KiB declared) unless the part completes it (`upload_too_many_parts`). `withStagedUpload` binds a put_page or append_page preparation to the sealed session and marks it committed in the publication transaction; `prepareStagedUpload` (from `preparePageMutation`) reads a body within PAGE_MAX_BYTES back as put_page content and hands a larger one to upload-streamed.ts. `stagedUploadWork` counts whole-body reads for the memory tests.
diff --git a/docs/architecture/key-files/engines-1.md b/docs/architecture/key-files/engines-1.md
index 4dcafe3bb3..0cca901d63 100644
--- a/docs/architecture/key-files/engines-1.md
+++ b/docs/architecture/key-files/engines-1.md
@@ -14,7 +14,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect(config, { onpoisoned })` passes the owner engine's poisoned-discard hook to the singleton pool and returns `Promise<boolean>` — `true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting the pool end so a concurrent connect can't join a pool that's already closing. The end routes through `endPoolBounded(pool)` — a gbrain-owned `Promise.race` of `pool.end({ timeout: POOL_END_TIMEOUT_SECONDS })` against a hard timer — so a PgBouncer transaction-mode drain that never settles can't hang teardown — the CLI teardown contract (finishCliTeardown's computed-deadline backstop + flushThenExit's fence-and-grace exit in cli-force-exit.ts) bounds it and delivers pending stdout before exit. `connection-manager.ts` ends its direct + read pools concurrently through the same helper so the per-pool bounds don't stack. `resolveMaxLifetimeSeconds(env?)` — explicit client-pool `max_lifetime` for all four postgres() call sites (matches the postgres.js implicit 30-60min jittered default; `GBRAIN_POOL_MAX_LIFETIME_S` overrides, 0 disables; warn-once on invalid). Pinned by `test/db-pool-max-lifetime.test.ts`. `resolveUrlConnectTimeout(url)` supplies `connect_timeout` to the same four call sites from each pool's own URL query (postgres.js lets the explicit option win over the URL, so the URL value is honoured only through this resolver): whole seconds, capped at the runtime's 2^31-1 ms timer limit; `0`, junk or no parameter means 10s. Pinned by `test/db-connect-timeout.test.ts`. `connectWithRetry` retries the `isRetryableConnError` set (3 attempts, 1s/2s backoff; `GBRAIN_NO_RETRY_CONNECT=1` makes it one); `retryConnectTimeout: true` adds postgres.js `CONNECT_TIMEOUT` (`isConnectTimeoutError` in `retry-matcher.ts`) and is passed only by the per-worker pools of parallel import and incremental sync, whose URL the parent engine already reached (#5946), so a first connect still fails fast on a timeout. Pinned by `test/worker-pool-connect-retry.test.ts`. #6543: `resolveReadPoolPipeline(url)` returns `{ value, reason }` for the three read-side pools (module singleton, instance pool, connection-manager read pool; `readPoolPipelineOptions` spreads it): `0` for a transaction-mode URL (port 6543 or `?prepare=false`, read from the URL because `GBRAIN_PREPARE` wins inside `resolvePrepare`) or `GBRAIN_PREPARE=0`, the key omitted (driver default 100, never `undefined`) for a session pooler or direct URL, and omitted whenever the URL names `max_pipeline` or `PGMAX_PIPELINE` is non-empty, since the driver lets the options object outrank both; pinned by `test/read-pool-pipeline-6543.test.ts` and `test/e2e/pgbouncer-pipelined-read-pool.test.ts`.
-- `src/core/degraded-engine.ts` + `src/core/degraded-marker.ts` — degraded-mode serve, STARTUP-SCOPED by design: when Postgres is unreachable at `gbrain serve` startup, serve boots on `createDegradedEngine` (a concrete object whose method set is enumerated from PostgresEngine's prototype at construction — no dynamic `get` trap; `kind` is an explicit getter, read synchronously at ~29 branch sites; `disconnect()` is a no-op while dead so shutdown never triggers a reconnect) instead of dying. Every tool call attempts ONE reconnect (single-flight across concurrent callers, min-interval gated ~5s — no connect storms against a dying pooler); calls inside the window throw the STORED original error (stale-but-honest, refreshed on every real attempt) so MCP dispatch classifies the REAL error shape into the `database_error` envelope + marker. The `reconnect` callback is the FULL deferred connectEngine (config merge, gateway, migrations — guarded in cli.ts: a vanished config.json throws instead of connectEngine's process.exit killing the live server); first success swaps in the live engine permanently and fires recovery callbacks (serve re-runs deferred boot + sends `tools/list_changed` — clients that handshook degraded hold the reduced gate-hidden catalog and must refresh; the callback is a no-op once shutdown has started) — but the TRIGGERING call gets `DegradedRecoveredRetryError`, never a result: its source scope was resolved under the degraded fallback, so one client retry buys scope correctness (no call ever executes under the fallback scope against a live engine). Callers waiting on an in-flight attempt are latency-capped (`callerWaitMs`, default 2s — they get the stored error while the attempt finishes in the background, so a timeout-class failure can't stall the MCP handshake). Prototype GETTERS delegate too (throw the stored error while dead — never a silent `undefined`); `DEGRADED_LAST_ERROR` is a read-only accessor for the stored diagnosis (drives the HTTP `/health` degraded reason without consuming a reconnect). Serve-side degraded posture: source scope honors a validated `GBRAIN_SOURCE` (tier `env`) before seed_default, and `stdioVisibleTools` fail-closes every gated op WITHOUT touching the engine. The degraded gate keys on the RESOLVED brain's engine (a PGLite mount on a postgres host keeps die-on-startup). Mid-session outages ride `PostgresEngine.reconnect()` + per-call classified envelopes — degraded mode adds no mid-session state; PGLite startup failures keep die-on-startup (single-writer lock makes a lazy proxy wrong there; that lane's repair is pglite-repair). Kill switch `GBRAIN_SERVE_DEGRADED=0` (or `false`); structured `[gbrain-serve] DEGRADED/RECOVERED` stderr lines. `degraded-marker.ts` is the dependency-free seam (`Symbol.for`-registered markers + `isEngineDegraded`/`onEngineRecovered`/`degradedLastError`) so server.ts/serve.ts/http-transport.ts can ask about degraded state without importing the heavy Postgres engine module.
+- `src/core/degraded-engine.ts` + `src/core/degraded-marker.ts` — degraded-mode serve, STARTUP-SCOPED by design: when Postgres is unreachable at `gbrain serve` startup, serve boots on `createDegradedEngine` (a concrete object whose method set is enumerated from PostgresEngine's prototype at construction — no dynamic `get` trap; `kind` is an explicit getter, read synchronously at ~29 branch sites; `disconnect()` is a no-op while dead so shutdown never triggers a reconnect) instead of dying. Every tool call attempts ONE reconnect (single-flight across concurrent callers, min-interval gated ~5s — no connect storms against a dying pooler); calls inside the window throw the STORED original error (stale-but-honest, refreshed on every real attempt) so MCP dispatch classifies the REAL error shape into the `database_error` envelope + marker. The `reconnect` callback is the FULL deferred connectEngine (config merge, gateway, migrations — guarded in cli.ts: a vanished config.json throws instead of connectEngine's process.exit killing the live server); first success swaps in the live engine permanently, aligns the stand-in's surface with it (`alignWith`, #9224: every property reachable on the live engine, instance fields and symbol keys included, delegates to it, mirrored names it lacks are dropped, and mirrored methods are not `async`, so a synchronous method such as `registerBeforeDisconnect` returns its own value) and fires recovery callbacks (serve re-runs deferred boot + sends `tools/list_changed` — clients that handshook degraded hold the reduced gate-hidden catalog and must refresh; the callback is a no-op once shutdown has started) — but the TRIGGERING call gets `DegradedRecoveredRetryError`, never a result: its source scope was resolved under the degraded fallback, so one client retry buys scope correctness (no call ever executes under the fallback scope against a live engine). Callers waiting on an in-flight attempt are latency-capped (`callerWaitMs`, default 2s — they get the stored error while the attempt finishes in the background, so a timeout-class failure can't stall the MCP handshake). Prototype GETTERS delegate too (throw the stored error while dead — never a silent `undefined`); `DEGRADED_LAST_ERROR` is a read-only accessor for the stored diagnosis (drives the HTTP `/health` degraded reason without consuming a reconnect). Serve-side degraded posture: source scope honors a validated `GBRAIN_SOURCE` (tier `env`) before seed_default, and `stdioVisibleTools` fail-closes every gated op WITHOUT touching the engine. The degraded gate keys on the RESOLVED brain's engine (a PGLite mount on a postgres host keeps die-on-startup). Mid-session outages ride `PostgresEngine.reconnect()` + per-call classified envelopes — degraded mode adds no mid-session state; PGLite startup failures keep die-on-startup (single-writer lock makes a lazy proxy wrong there; that lane's repair is pglite-repair). Kill switch `GBRAIN_SERVE_DEGRADED=0` (or `false`); structured `[gbrain-serve] DEGRADED/RECOVERED` stderr lines. `degraded-marker.ts` is the dependency-free seam (`Symbol.for`-registered markers + `isEngineDegraded`/`onEngineRecovered`/`degradedLastError`) so server.ts/serve.ts/http-transport.ts can ask about degraded state without importing the heavy Postgres engine module.
- `src/core/docker-postgres.ts` — gbrain's OWN docker Postgres container, the shared seam between the init ladder's docker rung and db-repair's `conn_refused` auto arm. `GBRAIN_PG_CONTAINER = 'gbrain-postgres'` / `GBRAIN_PG_IMAGE = 'pgvector/pgvector:pg16'` / `GBRAIN_PG_HOST_PORT = 5434` live HERE and only here (DRY — the init rung and the repair arm must never carry two string literals that can drift). Ownership contract: gbrain starts and reuses this container, NEVER stops or removes it, and never touches a container whose credentials it cannot recover via `inspectCredentials` (docker inspect → `POSTGRES_PASSWORD` env + the container's ORIGINAL port mapping — a freshly generated password can never match a surviving container; unrecoverable → refuse with a recreate-with-consent recipe). Hardening: the port binds loopback-only (`127.0.0.1:5434` — a single-machine brain store, not a network service); data lives on the named `gbrain-pgdata` volume so a container recreate never loses the brain; the password rides the child ENVIRONMENT (bare `-e POSTGRES_PASSWORD` — argv is world-readable in `ps`); `docker run` gets a 10-minute timeout (the first run synchronously pulls the image; 30s stays for cheap ps/inspect/start calls). `isGbrainDockerUrl` is the one predicate for "does this URL point at our container" (loopback + dedicated port — never error text).