Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
15.8 KB
Raw Blame

Core Persistence: write capacity

Cumulative write caps (permanent request ids, receipt bytes; opt-in since GBRA-108, DEFAULT_JOURNAL_LIMITS holds Infinity and rows of an unset cap are left out of diagnostics, the forecast and repair capacity) and the paths that keep them honest (#6696). Guide: concurrent writes, growth model and quota.

  • src/core/persistence/receipt-bytes.ts — receiptReservation({intent, authority}, extra): what an admission reserves (serialized intent and authority plus RECEIPT_COMPLETION_BYTES 12 KiB, never below RECEIPT_RESERVATION_FLOOR 16 KiB) on single, group and caller-overridden paths (journal.ts takes the larger of a caller's value and the formula); reservationForRetained is the same estimate from bytes alone (the sync preflight); retainedReceiptBytes is what a terminal or compacted row keeps, shared with intent-redaction.ts and compaction; EFFECT_BYTES_SQL and TRUE_UP_CLAMP_SQL (counts clamped decrements under persistence.true_up.clamped).
  • src/core/persistence/receipt-true-up.ts — trueUpRequests(engine, ids): after the settling transaction commits, in its own short transaction, locks the brain and principal counters in key order, then a compare-and-set on each request row (still terminal_reservation as read, no queued or running effect, same effect bytes) shrinks it to retainedReceiptBytes and releases the difference. Idempotent; never touches updated_at. Called from the effect runner (effects.ts, every claimed request), embedding-settlement.ts, embedding-noop-settle.ts (the embed --stale bulk settle), effect-retry.ts (reconciled) and backup/snapshot.ts (restore). test/persistence-true-up-sites.test.ts fails on an effect state-change site that is not wired. sweepCompactedReservations is the bounded one-shot pass over receipts compacted with their full reservation (cursor persistence.true_up.compacted_through, done at the end).
  • src/core/persistence/admission-origin.ts — withAdmissionOrigin('interactive' | 'background') (an outer background scope wins), resolveAdmissionOrigin (unset reads background, never derived from ctx.remote), persistence.capacity.background_pause_pct (default 80) and backgroundPaused, the write_capacity_backpressure refusal background producers get past the line. A background scope also carries the per-run owner cache (backgroundOwnerCheck); backgroundWorktreeInactive (a recovering worktree, read under the admission's lock in journal.ts) and backgroundOwnerUnverified refuse owner_unavailable pre-receipt with the owner-status fix; a draining worktree still admits.
  • src/core/persistence/owner-verification.ts — #6741 C: the claim's owner verification (verifyOwnerRoot: paths recorded, physical root, marker and reservation intact; a device-only change is restampable), shared by writer onboarding and background admissions; assertBackgroundOwner refuses a background admission owner_unavailable once per source per run (cache: admission-origin.ts backgroundOwnerCheck). Tests: test/background-owner-verification.test.ts.
  • src/core/persistence/capacity-forecast.ts — readCapacityForecast: per principal, one statement walks persistence_requests_principal newest first until created_at leaves the 7-day window and buckets by day; gross and net (compacted) receipt rates, days_to_exhaustion (at the larger of the 7-day mean and the last 24 hours), basis (:legacy_under_reserved while receipts admitted before the intent-sized reservation remain), recommended_limit (one more year at the 7-day mean, GST-2107, so one bulk day is not projected for a year). light skips the reclaim read and table size. Thresholds persistence.capacity.warn_days (30) and fail_days (7).
  • src/core/engine-key.ts + src/core/engine-root.ts — the key every per-engine cache uses. engineKey(engine) returns the engine itself through the ENGINE_ROOT symbol (each engine stores itself there; a page-transaction clone inherits it, forwarding proxies and a recovered degraded engine return it), and otherwise walks a page-transaction clone, a read-through view or a statement-deadline wrapper (engineCacheKey) to the engine behind it. A process with mounted brains keeps one entry per brain in the search telemetry writer (getTelemetryWriter(engine)), search.track_retrieval, minion admission policy, mcp.strict_params last-known-good, publication ceiling, write_capacity_low throttle, search.hnsw_iterative_scan and search.vector_legacy_guard, and a transaction or view shares its engine's entry. A one-brain process behaves as before. Test: test/engine-keyed-caches.test.ts.
  • src/core/persistence/capacity-snapshot.ts — cachedCapacitySnapshot: one light forecast per engine per 60 s, shared by every caller; capacityNotices builds write_capacity_low (kind info, muteable; it says it is not a refusal and its fix is the read-only doctor check, get_brain_health over MCP, GST-2107; hourly per engine and scope on write and sync receipts, scopes limits a remote caller to the brain and its own principal) and write_capacity_backpressure (once per pause start). Read by mcp/dispatch.ts, cli/main.ts, commands/sync/{run,operator}.ts, cycle.ts and the admin /metrics gauges in serve-http-metrics.ts. #6803: the cache keys on the real engine (engineCacheKey), the shared fetch is bounded at CAPACITY_FETCH_BOUND_MS (10 s) and a timed-out fetch is never cached; capacitySnapshotWithin (sync status, write and sync receipts: CAPACITY_READ_WAIT_MS, 1.5 s) returns unknown for that call alone, and capacityNotices on an unknown snapshot returns no notice and leaves throttle and pause state as it was. Test: test/capacity-bounded-read-6803.test.ts.
  • src/core/persistence/compaction-pass.ts — runCompactionPass, the consumer's idle and volume maintenance: compactWriteReceiptsDetailed, housekeepReceipts (GBRA-108: while receipts not yet compacted reserve more than persistence.housekeeping.receipt_target_bytes (limits.ts readCompactableReceiptBytes; compacted receipts never count) with the default retention, up to 100 receipts older than HOUSEKEEPING_FLOOR_DAYS that EARLY_SQL admits: settled non-job, non-connector writes that are not a subagent's own (restrictedNamespace), and refused maintenance or managed sync page writes; counted as early; an error counts as housekeeping_error and the sweep still runs), one sweep step, and the persistence.maintenance.last_compaction record (at, host_id, compacted, freed_bytes, skipped_by_reason). Each step resumes from the timer queue (journal.ts yieldToEventLoop), as do the consumer's maintenance chain, the per-table queue VACUUM and the effects drain passes (NOOP_SETTLE_BATCH 50), so on PGLite a request waits behind one step at most. The retention pass (compactWriteReceiptsDetailed) skips terminal-classify.ts UNRESOLVED_CALLER_REFUSAL_SQL (a caller refusal repair failed-writes could still replay, GBRA-108). journal.ts compactReceipt compacts one receipt in its own transaction (counters first, lock_timeout 2 s, statement_timeout 15 s) and skips open managed-sync runs (OPEN_SYNC_CURSOR_EXCLUSION) and the newest sync receipt per worktree (newestSyncReceipt).
  • src/core/repair/receipt-compaction.ts — gbrain repair receipt-compaction: explicit-only, preview-bound, destructive consent, publication: 'projection'; --older-than (the pages purge-deleted grammar) floored at the configured retention unless --force, committed only unless --include-failures; the approved set binds an absolute cutoff and the newest sequence. Test: test/repair-receipt-compaction.test.ts.
  • src/core/persistence/refusal-breaker.ts — assertRefusalBreakerClosed: submitMaintenance refuses before admission after persistence.maintenance.refusal_breaker_threshold (3; 0 off) same-code terminal refusals in 24 h for the same request digest (source, slug, intent with its revision and planner); listSuppressedWrites feeds doctor maintenance_write_drift. The ghost case (canonical_file_missing, tagged by page-prepare.ts, kept in error_detail.origin, returned by liveFileDrift) refuses before admission in prepared-maintenance.ts. Test: test/maintenance-ghost-breaker-6696.test.ts.
  • Sync: sync-run.ts preflightSyncCapacity refuses write_quota_insufficient before a run's first admission; sync-drain.ts stops a later cap refusal resumable / capacity_paused (drain.capacity_pause). Test: test/sync-capacity-preflight.test.ts.
  • Doctor: persistence_capacity (forecast of set caps; with none set, receipt storage against the housekeeping target, an operator warning that never blocks a write; test/persistence-opt-in-caps.test.ts), persistence_compaction and persistence_counter_drift (src/commands/doctor/checks/persistence-compaction.ts; one-statement comparison of counters with request and lifecycle reservations). Test: test/doctor-persistence-compaction.test.ts.

Publication cost

What a managed publish costs in statements and round trips, and how it is measured (#6712 PR 3). Budgets: test/managed-import-statement-budget.test.ts (PGLite) and test/e2e/managed-import-statement-budget.test.ts (Postgres wire trace) count one YC-shaped page's publication transaction; test/sync-status-statement-budget.test.ts holds sync status to a fixed statement count at 61,000 receipts. Helpers: test/helpers/yc-shaped-page.ts, statement-counter.ts, managed-import-budget.ts, latency-proxy.ts.

  • src/core/persistence/publication-timing.ts (#6712 L6) — timePublicationStep('apply' | 'effects', slug, run) wraps a publication member's apply and its queuePublicationEffects call (coordinator.ts, group-publish.ts): process-local totals (publicationStepTotals) for tests and benchmarks, and one [sync] publish <step> <slug>: Nms line on stderr only under GBRAIN_SYNC_TRACE. It measures and never changes the outcome (a throwing step is timed and rethrown).
  • addCodeReferenceLinks in src/core/engine-sql/links.ts (#6712 L6) — a page's doc↔impl code-reference edges (documents from the page, documented_by back) in one set-based statement: endpoints resolve in the page's source under FOR KEY SHARE, a missing endpoint drops its edge, a repeated edge keeps its last context, and a re-import updates context and origin field in place. import-file.ts and page-body/streamed-import.ts call it once per page inside a savepoint (a failure leaves the page import intact), where each reference used to cost two sequential addLink statements.
  • src/core/postgres-engine/pool-stats.ts — #6317 (C1, G1): driverPoolStats(sql) reads the vendored postgres.js driver's sql.pool getter (vendor/postgres, patch regenerated) for real {checked_out, max, waiters}; enginePoolStats(engine) labels the source (driver or tracked_subset from pool-gauge.ts). Filled into the heartbeat row, writer status running claims and getPoolDiagnostics().
  • src/core/persistence/maintenance-wait.ts — how long a maintenance publisher waits for its accepted write to commit before the receipt reports write_pending (#5854). MAINTENANCE_WRITE_WAIT_MS is 30 s; MaintenanceWriteWait is one budget per job: each wait is capped by the job's remaining deadline (zero at or past it), and once one publish is still pending after its wait (observe) every later publish in that job waits zero and is deferred like it. A pending write keeps its request identity and commits later; it is never reported as committed. A MaintenanceWriteWait with no deadline also sets the replay window of a subagent tool write whose job has none (minions/tool-write-identity.ts). __setMaintenanceWriteWaitForTests replaces the base wait, those tool writes included. Tests: test/maintenance-write-wait.test.ts, test/e2e/maintenance-write-wait.test.ts.
  • src/core/persistence/effect-settle.ts — gbrain sources writer settle-effects <source> (--request-id|--parked-code) [--unsettle] [--dry-run|--apply --expect <plan_hash>] (writer_settle_effects, an administration extension) (#7423, #7378): records data.settled (and appends to data.settlements) on a failed Git effect no retry can move, never touching its failure, parked targets, request or bytes. It settles only on an exact-target proof re-checked under row locks at apply: each target declared db_only in the current binding's gbrain.yml (sourceDbOnlyDir), or, for an effect guardEffectSource refuses source_changed because the source moved to a new worktree (same incarnation, this host owning the active binding), a later committed and pushed Git effect of the current worktree whose expected_hash equals the canonical file now and a blob at the already-fetched @{upstream} equal to it (no fetch). Withdrawal effects, claims, open recovery records, revoked writers, replaced sources and unknown targets refuse; an effect of the current binding that is not db_only refuses retry_instead. The plan hash is doctor --remediate's computePlanHash over the brain, source, selector, owner epoch and each proof (preview_changed otherwise). Readers that skip settled rows: doctor parked_effects, rearmWithheldTargets, readParkedGroups, readGitPushHolds, retryParkedPushes/--parked-code, retryIndexLockParks, sync status, failedEffectHistory; retryParkedEffects reports a settled effect as action: settled. retryParkedPushes also reaches a push that failed without parked targets (#7378, failedCodes). #7985: an effect outside its source's current binding (CURRENT_BINDING_SQL in parked-groups.ts, the guard's own test) gets its own readParkedGroups group (former_binding: binding_changed or source_replaced, recorded and current worktree, up to five request ids, the settle preview as command), doctor's per-effect command for it is settle-effects --request-id, retryParkedEffects rethrows the guard's source_changed naming both bindings with that preview as fix, and the bulk retries' former_worktree lists its worktree and requests. --retire (retiredTarget) settles such an effect as former_binding_retired only on evidence: a replaced source (authority not re-checked), or each target's page absent, deleted, replaced (page_id) or at another revision than effect.revision; else page_not_moved, and a current-binding effect not_former_binding; its plan operation is writer_retire_effects and its fix actor: user (test/orphan-effect-diagnostics-7985.test.ts). Tests: test/persistence-effect-settle.test.ts (scenarios in test/helpers/effect-settle-scenarios.ts) + the Postgres arm in test/e2e/w5-persistence-postgres.test.ts.
  • src/core/persistence/owner-activity.ts — GBRA-248: ownerActivity(engine, worktreeIds) reads each worktree owner's liveness (a persistence_consumers row renewed within the lapse window) and last activity (the greatest of the worktree heartbeat, which a consumer's clean stop stamps, its newest consumer renewal and the newest committed receipt, read through the foreground-committed and sync-watermark indexes). stale is true when the owner is not this host and not live, and either this process holds a PGLite file under another identity or the owner has been quiet past OWNER_STALE_MS (24 h; with no activity recorded, the oldest open request's age decides). startOwnerRead runs it inside a budget the caller already has (the journal's 500 ms receipt read; awaitWrite's last 500 ms, which a shorter wait is extended to, with each worktree's answer reused for 10 s so a batch pays one read) and cancels it past that; staleness uses the database clock; writeHealth maps a stale owner to owner_unavailable (unless recovery outranks it) with a diagnostic.owner block carrying only last_active, present whenever the owner is stale, and acceptPendingCovers in write-wait.ts keeps such a write at exit 10 under --accept-pending. Read-only; null activity authorizes nothing.