Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
37.2 KB
Raw Blame

Key files: Runtime

Subsystem index. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.

  • src/core/abort-check.ts — one canonical place for cooperative-abort checks across gbrain's long loops. isAborted(signal?) → boolean (for loops that break and return partial progress). throwIfAborted(signal?, label?) throws an AbortError (name === 'AbortError') at phase boundaries, preferring the signal's reason ('wall-clock'/'lock-lost'/'shutdown') so the unwind self-describes. anySignal(internal, external?) composes two signals into one that fires when EITHER does (platform AbortSignal.any with a manual-relay fallback), returning the internal unchanged when there's no external so non-aborting callers pay nothing. Threading these checks through runPhaseEmbed → runEmbedCore → embedAll(Stale)/embedPage lets the embed phase bail and release gbrain_cycle_locks immediately (an embed phase that ignored its abort signal would hold the lock and make later autopilot cycles skip with cycle_already_running). Coverage spans every long cycle-reachable phase: extract (incremental extractForSlugs + the full-walk extractLinksFromDir/extractTimelineFromDir, all via runSlidingPool's signal), extract_facts (per-page loop + the per-page embed signal + runPhantomRedirectPass's 30s lock-retry), consolidate's bucket loop, and lint (which is synchronous, so it awaits a periodic yield to let the signal land). runCycle adds a terminal abort check before stamping last_full_cycle_at so a cancelled cycle never reports a completed full run, plus a per-phase duration_ms warning that names any phase overrunning the worker's 30s force-evict deadline. Pinned by test/abort-check.test.ts + test/cycle-abort.test.ts.

  • src/core/abort-signals.ts — leaf module (no imports): anyAbortSignal(signals) fans several signals into one real AbortSignal, tolerating duck-typed { aborted } stubs (polled every 50 ms) where AbortSignal.any would throw; returns {signal, dispose}, and dispose() must run when the scope ends so long-lived caller signals do not accumulate listeners. cycle.ts re-exports it; the extract_atoms drain imports it directly without loading the cycle. Tests: test/cycle-any-abort-signal.test.ts.

  • src/core/background-work.ts — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the engine goes away," with TWO exit points. registerBackgroundWorkDrainer({name, order, drain(timeoutMs, mode), abort?}) over a Map<name, BackgroundWorkDrainer> (idempotent registration by name; __registerDrainerForTest returns an unregister handle); mode is 'exit' | 'disconnect' ('exit' = CLI teardown, engine still live, residual buffers may flush; 'disconnect' = an engine is mid-disconnect, sinks await only IN-FLIGHT work and never start new writes). drainAllBackgroundWorkForCliExit({timeoutMs}) runs mode 'exit' with abort allowed; drainBackgroundWorkBeforeDisconnect({timeoutMs}) is called by BOTH engines' disconnect() so an in-flight statement settles before the underlying handle closes — PGLite's close() deadlocks PERMANENTLY with a statement in flight — and it NEVER calls abort() (permanent process state, wrong for a long-lived gbrain serve disconnecting one engine); a partial disconnect-drain warns once per sink to stderr. The module stays a zero-import leaf on purpose (both engines import it statically). Drains in explicit (order, name) order — facts FIRST (order 0) so its abort-path DB logIngest runs against the freshest live engine — and AWAITS abort() only when drain() reports unfinished>0 in 'exit' mode. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. SIX sinks register at module import: facts/queue.ts (order 0; abort=shutdown() cancels a hung facts:absorb Haiku via internalAbort), last-retrieved.ts (order 1), search/hybrid.ts (order 2; awaitPendingSearchCacheWrites bounded via Promise.race), eval-capture.ts (order 3; captureEvalCandidate self-tracks its promise via awaitPendingEvalCaptures), context/volunteer-events.ts (order 4 — batched volunteer-event INSERTs, drained like the rest), search/telemetry.ts (order 5 — ... (line truncated to 2000 chars)

  • src/core/backoff.ts — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier.

  • src/core/cli-force-exit.ts — single owner of one-shot CLI exit + teardown, designed as a PAIR with the exit seam runCli() in src/cli/main.ts (called by the src/cli.ts entry point after its fast paths). finishCliTeardown({engine, drainTimeoutMs?}) is teardown-ONLY (never exits on the clean path): arms a REF'D backstop (unref'd would let a hung teardown exit naturally, skipping the flush and exiting with whatever PGLite scribbled into process.exitCode) whose deadline is COMPUTED from the bounds it guards (computeTeardownDeadlineMs = sinks × drainTimeoutMs + sinks × SINK_DRAIN_TIMEOUT_MS disconnect-drain bound + the RESOLVED PGLite close bound (pgliteCloseTimeoutMs() from background-work.ts — an operator-raised GBRAIN_PGLITE_CLOSE_TIMEOUT_MS widens this backstop too, never a hardcoded copy of the default) + facts-abort grace + 2 × pool-end bound + slack, floor 10s — the disconnect-drain and close terms budget engine.disconnect()'s own drain pass and bounded close so the backstop can't fire while every component honored its own bound; GBRAIN_TEARDOWN_DEADLINE_MS env override is the incident escape hatch), drains every background-work sink, disconnects the engine (a throw is warned + swallowed — the exit code reports the OPERATION, not the cleanup), then returns. The exit VERDICT lives in a gbrain-owned channel (setCliExitVerdict/currentExitCode; mirror-writes process.exitCode but NEVER reads it back) because PGLite's Emscripten runtime scribbles its own status into process.exitCode at arbitrary points mid-run — every writer that means to set the CLI exit code (op-dispatch catch, reindex, frontmatter, transcripts, brainstorm, autopilot, doctor's FAIL verdict, extract, and cli.ts's swept inner exits — friction, claw-test, smoke-test, the no-DB eval runners, status/status-thin, whoknows-thin) calls setCliExitVerdict; test/cli-exit-verdict-pin.test.ts greps src/ so the next raw process.exitCode = write fails CI instead of silently rep... (line truncated to 2000 chars)

  • src/core/exit-codes.ts — CLI exit statuses with a contract beyond 0/1: MIGRATIONS_RUNNING_EXIT_CODE (75, read by gbrain upgrade as a lock conflict; re-exported by migration-orchestration-lock.ts) and PENDING_WRITE_EXIT_CODE (10, an admitted write still pending) and BUDGET_STOP_EXIT_CODE (11, gbrain embed --stale stopped at its time budget with stale chunks left; stdout names the remaining count and resume command). Agent contract v1 table: 0 ok, 1 failed, 2 usage (USAGE_EXIT_CODE), 3 confirmation_required only (CONFIRMATION_REQUIRED_EXIT_CODE), 10 pending write, 11 budget stop, 75 migration lock, 124 timeout, 130 interrupted; test/exit-codes.test.ts enumerates every exit 3 in src/; guide docs/guides/exit-codes.md.

  • src/core/cli-options.ts — Global CLI flag parser. parseGlobalFlags(argv) returns {cliOpts, rest} with --quiet / --progress-json / --progress-interval=<ms> / --brain <id> stripped. --brain is the brain-axis (which database) selector: exact-match only (--brain-* per-command flags pass through), value validated against the mount-id regex at parse time, missing/malformed value THROWS — never a silent host fallback. connectEngine in src/cli.ts feeds it (plus the ambient GBRAIN_BRAIN_ID / .gbrain-mount / mount-path tiers) through resolveBrainId → BrainRegistry.getBrain, which throws UnknownBrainError for an unregistered id; mounts get no auto-migrations and keep the host-config AI gateway. getCliOptions() / setCliOptions() expose a module-level singleton so commands reach resolved flags without parameter threading. cliOptsToProgressOptions() maps to reporter options. childGlobalFlags() returns the flag suffix to append to execSync('gbrain ...') calls in migration orchestrators (propagates --brain=<id> so children stay on the parent's brain). OperationContext.cliOpts extends shared-op dispatch for MCP callers. CliOptions carries explain: boolean. parseGlobalFlags recognizes --explain anywhere in argv (stripped before command dispatch). --wait <seconds> (#5232 write wait, 0-600 s, no units; a malformed value is invalid_params, exit 2) is slotted like --timeout: a route in WAIT_OWNING_ROUTES (sources writer movement, whose own --wait takes a duration) gets every --wait handed back appended in the space spelling, order kept (#6625). src/cli.ts formatResult for search + query cases routes to formatResultsExplain from src/core/search/explain-formatter.ts when CliOptions.explain is set; falls through to the existing JSON / human formatters otherwise. maybeBackground(opName, fingerprintArgs, runDirect) helper. Same semantics in TTY and cron (no --no-tty-detect flag, no surprise behavior change... (line truncated to 2000 chars)

  • src/core/cli-preflight.ts — runCliPreflight(), the first statement of cli.ts:main(). (0) Re-run detection: GBRAIN_CWD_ENV_QUARANTINED (CWD_ENV_QUARANTINED_MARKER) is parsed as JSON {cwd, neutral, wrapperPid} and honoured ONLY when the process's startup cwd realpath-equals neutral and neutral holds none of CWD_DOTENV_FILES (wrapperPid is informational — accepted only as a positive safe integer, otherwise ignored); on a verified hop it deletes the marker, chdir(cwd)s back, skips step 1 (the parent already warned), and arms installWrapperWatchdog: an unref()ed 1 s interval comparing process.ppid (live in Bun) with wrapperPid — a mismatch prints one [env] line, rmdirSyncs the empty neutral dir the wrapper's finally never reached, and exits 137, so a supervisor that SIGKILLs the wrapper pid (restartCurrentChild, job-isolation children, hook pushes) takes the real re-run down within about a second instead of leaving an orphan; an unverified marker is ignored, never chdir'd on. (1) parseCwdDotenv(cwd) ONCE, shared by the cwd == config dir check and quarantineCwdDotenv(process.env, cwd, { assignments }). (1a) If the drop removed a NON-EMPTY GBRAIN_GUARDRAILS_MODULE, print GUARDRAILS_ASSIGNED_BY_CWD_DOTENV and exit 1 — fail-closed, never a re-run without the operator's firewall (the ~/.gbrain/.env home is read after the quarantine and unaffected). (1b) Otherwise, whenever anything was dropped: Bun.spawn([execPath, ...execArgv (dev runtime only; resolveRuntimePathsresolves a.//../-relative — or bare-and-existing — value of --preload/-r/--require/--import, --config/-c, --env-file, --tsconfig-override, both spellings, against the ORIGINAL cwd because the re-run starts in the empty neutral dir; everything else passes through verbatim, in order), entry/user args], { cwd: <fresh mkdtemp dir, realpath'd so neutralis absolute+canonical>, env: process.env minus undefined + marker, plus the operator's real... (line truncated to 2000 chars)

  • src/core/cli-socket/ + src/cli/cli-socket-client.ts — the opt-in warm CLI socket for Postgres brains (guide); off unless gbrain serve --cli-socket or DB-plane serve.cli_socket, which loadConfigWithEngine merges from its bulk snapshot only (mergeServeCliSocket: serve_cli_socket in the config connectEngine publishes in MERGED_CONFIG_BY_ENGINE, both in src/core/published-config.ts), so serve decides without loading this module or reading config; off, it does neither. server.ts startCliSocket (called from both serve lanes only when on; stdio after the stdout redirect) binds $GBRAIN_HOME/.gbrain/run/cli.sock only on a Postgres engine whose resolved brain (resolveBrainId, the tier chain connectEngine used) is the host (a mount serve logs why and never binds), after fs-guard.ts ensurePrivateDir (0700, this uid, never a symlink; an open directory this uid owns is tightened, anything else refused; GBRAIN_HOME and .gbrain above it must be this uid's with no group/other write bit unless sticky, checked by ancestorProblem on both sides and never repaired), a peer-credential self-test on a scratch socket, the native bind lock and a stale-socket probe; listen() binds synchronously under process.umask(0o177) (restored right after), so the kernel creates the socket 0600, and verifyBoundSocket only checks it (lstat: our private socket; nothing is chmod'ed through the path); it is unlinked on exit only while its inode is still ours. At most 8 requests are active, and a request counts until every promise it started settles (ExecutedRequest.settled), not until it answered; one still unsettled at the 30 s round-trip cap is logged once. Every accepted connection's uid comes from peer-cred.ts (bun:ffi getsockopt(SO_PEERCRED) on linux x64/arm64, getpeereid on darwin, on the socket's _handle.fd) and any other or unreadable uid is dropped before a byte is parsed; with no reader the serve logs why and never ... (line truncated to 2000 chars)

  • src/core/vector-prewarm.ts — opt-in HNSW prewarm for Postgres brains (ENGINES). readVectorPrewarmPlan reads only catalogs and file sizes: every valid hnsw index in current_schema() with pg_relation_size, pg_size_bytes(current_setting('shared_buffers')), effective_cache_size (missing_ok, reported only) and the pg_prewarm extension state. prewarmFits: shared_buffers alone decides, the total must be at most PREWARM_SHARED_BUFFERS_SHARE (0.5) of it, because 'buffer' mode fills Postgres's own cache and a bigger prewarm would evict the heap and keyword-index pages other reads need; effective_cache_size is a planner hint the server never allocates, so it never decides. runVectorPrewarm refuses PGLite (vector_prewarm_unsupported_engine), a misfit (vector_prewarm_wont_fit, host_admin fix naming the shared_buffers needed) and a missing extension (vector_prewarm_extension_missing) before any pg_prewarm call, and a dry run reads nothing; otherwise one transaction per index, largest first, with SET LOCAL statement_timeout = 0 and pg_prewarm(<index>, 'buffer'). Callers: gbrain embeddings prewarm (CLI-only, never an operation, so no MCP caller reaches it) and prewarmAtServeStart, which runServe starts in the background only when DB-plane serve.vector_prewarm is true (mergeServeVectorPrewarm in published-config.ts, snapshot only); it logs the load or the refusal and never stops the serve. Pinned by test/vector-prewarm.test.ts, test/serve-vector-prewarm-off.test.ts (off: no module load, no engine call, no log line) and test/e2e/vector-prewarm-postgres.test.ts (prewarm, refusal under a simulated 16 kB shared_buffers, missing extension).

  • src/core/console-prefix.ts — AsyncLocalStorage<string>-backed per-source line-prefix helper. Exports withSourcePrefix(id, fn) (runs fn with id as active prefix; nested wraps replace then restore), getSourcePrefix() (read-only accessor; test seam), slog(...) / serr(...) (prefix-aware console.log/console.error), withHumanLineSink(sink, fn) (#6317: every slog/serr line under fn, Bun timers included, goes to sink instead of the streams; the serve captures a delegated managed drain's lines per job for sync_status), and withHumanLogsToStderr(fn) (runs fn with every slog line, prefixed or not, routed to stderr; runSync wraps its body in it under --json so stdout carries only the ONE JSON envelope (the cost-gate status object rides inside it as cost_gate, #4684) while serr and direct console.log(JSON.stringify(..)) sites are unaffected; scoped via its own AsyncLocalStorage, never a process-wide console rebind). Embedded-newline-safe: a multi-line string under prefix [foo] emits [foo] line1\n[foo] line2. Outside a wrap, slog/serr fall through to bare console.log/console.error so single-source callers see identical output (back-compat invariant). Use src.id (slug-validated by sources add) NOT src.name (free-form) to defeat log-injection through newline/control-character names. Coverage: src/commands/sync.ts performSync + callees, src/commands/embed.ts runEmbedCore + helpers, src/core/progress.ts emitHumanLine, src/commands/import.ts runImport's human-only lines (info() + the end-of-run Import complete summary — so the first-sync full import under sync --json lands on stderr too).

  • src/core/mcp-client.ts — the thin-client transport (trust boundary). RemoteMcpError, extractResponseMeta and ignoredRemoteParams live in src/core/remote-mcp-error.ts (re-exported here) so local CLI paths can name them without loading the MCP SDK; the CLI imports this module only on a routed call. callRemoteTool(config, toolName, args, opts) with CallRemoteToolOptions {timeoutMs, signal}; buildAbortController composes an external signal with the timeout, and every request carries X-Gbrain-Client: gbrain-remote-cli/<version> so hosts serve it full search/query rows (it never sends fields); buildMcpRequestOptions forwards timeoutMs as the SDK request deadline (otherwise its own 60s default wins); an SDK RequestTimeout classifies as network/timeout, never unreachable. All transport errors normalize to RemoteMcpError via the toRemoteMcpError funnel: stable RemoteMcpErrorReason union, RemoteMcpErrorDetail.kind ('timeout' | 'aborted' | 'unreachable') sub-tag, RemoteMcpErrorDetail.code carrying server-supplied error codes (e.g. missing_scope). A /token 429 is rate_limited with RemoteMcpErrorDetail.retry_after_s parsed from Retry-After (parseRetryAfterSeconds in remote-mcp-probe.ts); any other non-auth /token failure is token, never discovery. extractToolErrorCode parses the operation {error: string, message} shape as well as legacy envelopes. Mutation arguments retain one request UUID before the first transport attempt, across OAuth refresh and caller retries. Mutation errors retain that UUID and distinguish unknown delivery from requests never submitted; the shared CLI renderer emits same-ID retry guidance without fabricating a receipt. Received pending or terminal error receipts and frozen verb metadata survive deadlines during client cleanup, including the refreshed attempt. Only the locked SDK’s typed StreamableHTTPError with numeric code 401 triggers one refresh and reconstructed client; application error ... (line truncated to 2000 chars)

  • src/core/op-checkpoint.ts — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint)). Per-op fingerprint helpers (embedFingerprint, extractFingerprint, reindexFingerprint, integrityFingerprint, purgeFingerprint) compute sha8(canonical-JSON(relevant-params)) so re-running with the same params resumes from completed_keys and re-running with different params (e.g. --limit 100 vs --limit 200) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. The 7-day TTL GC runs in the cycle's purge phase. All writes (recordCompleted, clearOpCheckpoint) route through engine.executeRawDirect + withRetry(BULK_RETRY_OPTS) so they survive Supavisor pool exhaustion, and recordCompleted returns boolean (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-completed_keys semantics. Resumable sync uses the additive appendCompleted(key, deltaKeys) / appendCompletedOnce (the latter no-retry for the SIGTERM path) which INSERT a delta into the op_checkpoint_paths child table (migration v115: (op, fingerprint, path) PK, FK to op_checkpoints ON DELETE CASCADE) via a single writable-CTE unnest($3::text[]) write — O(delta), never an O(N²) full-set rewrite. loadOpCheckpoint returns the UNION ALL of legacy completed_keys + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on jsonb_typeof(completed_keys) = 'array' so a non-array (scalar) parent row can't make jsonb_array_elements_text throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_arr... (line truncated to 2000 chars)

  • src/core/persistence/preview-approval.ts — preview-bound approval shared by jobs authorize-legacy --select, jobs cancel --select, repair stale-atoms, repair extractor-facts and the other preview-bound repair kinds such as repair ontology-facts (fix wave 5, ENG-O1). previewHash(parts) is SHA-256 over key-sorted JSON, byte-identical to authorityDigest; saveApprovedSet / loadApprovedSet / clearApprovedSet keep the approved items in op_checkpoints (op='repair-approval', fingerprint <command>:<hash>); retireAfterLastItem applies one item and clears the set only after the last item returns, so an item that throws keeps it and the printed apply resumes with the same --expect (test/repair-timeline-order.test.ts). A missing, foreign, malformed or older-than-APPROVAL_MAX_AGE_DAYS (7, the purge TTL) set refuses whole with preview_changed. Tests: test/preview-approval.test.ts (PGLite; Postgres via test/e2e/preview-approval-postgres.test.ts). #6351: loadApprovedSet takes an optional lazy emptyHash (the hash a preview of the same selection prints when it finds nothing, from current state including source incarnations); with no saved row and a matching hash it returns an empty set, so the apply command an empty preview prints succeeds while a changed incarnation still refuses preview_changed. The eight preview-bound repair kinds without a local emptyPlanHash (stale-atoms, captured-facts, conversation-labels, extractor-facts, failed-writes, frontmatter, loop-facts, ontology-facts) pass it. Pinned by test/repair-empty-preview-apply.test.ts.

  • src/core/process-cleanup.ts — cleanup registry + abnormal-termination handlers (SIGTERM/SIGHUP/SIGPIPE, uncaughtException/unhandledRejection, EPIPE-on-stdout) that release locks before exit. registerCleanup(name, fn) returns a deregister handle; tryAcquireDbLock auto-registers. abortOnCleanupPass(name) wraps that for child work: it returns { signal, release }, the signal aborts when the cleanup pass runs (sync's git pull uses it), and release deregisters it. installSignalHandlers() is idempotent and called from INSIDE cli.ts's import.meta.main seam (first statement before main()), NOT at module load — installing at import time would leak a process-wide SIGTERM→process.exit(143) handler into any process that merely imports cli.ts (a bun test runner would die mid-suite when a test emits a synthetic SIGTERM, misreading rc=143 as an external kill). Spawned/compiled CLI processes are entrypoints, so they still install. Every attached listener ref is recorded so _resetForTests() DETACHES them (clearing flags alone would leave the exit(143) listener live on the shared runner). Tests that must emit synthetic signals strip foreign listeners around the emit (see test/run-child-entry.test.ts). #5079: serve --http (detected by isHttpServeInvocation in src/core/serve-invocation.ts) installs it with keepServingOnLogEpipe, so a closed stdout/stderr pipe (SIGPIPE under Bun, or an EPIPE stream error) drops the log write instead of exiting; stdio serve and every other command still exit. Pinned by test/serve-http-log-epipe.test.ts. #5062: registerSignalOwner(name, onSignal, signals?) lets a long-lived command (jobs supervisor, jobs work) claim SIGTERM/SIGHUP/SIGPIPE (GST-2869: signals narrows the claim, so gbrain sync takes only SIGTERM/SIGHUP and a broken pipe keeps the generic path; hasSignalOwner() lets sync stand aside under a worker's owner) and a broken stdout/stderr pipe: while an owner is registered the generic handler only forwards the signal (no cleanup pass, no exit), so the owner can drain while still holding its DB lock and then exit through triggerCleanupAndExit; crashes keep the generic path. Pinned by `test/pro... (line truncated to 2000 chars)

  • src/core/read-heartbeat.ts + src/cli/slow-read.ts — slow whole-brain reads say what they are doing. A CliCommandRecord.slowRead(args) (src/cli/command-table.ts) names the read for doctor, sync status, sources status, orphans, anomalies, status and diag; beginCommandRead (skipped for --help and --quiet) wraps the CLI_ONLY dispatch in src/cli/main.ts and calls beginSlowRead, which prints one stderr start line and an unref'd still reading line every READ_HEARTBEAT_EVERY_MS (10 s); stdout is untouched. withReadMeta(doc) adds elapsed_ms and complete to a --json document and lists incomplete ({field, reason}) from the markers the documents already carry (incompleteFields: a status: not_read object, complete: false or a *_complete: false key, budget_hit: true, a doctor check unknown for its budget); an error envelope gets elapsed_ms only and an array stays bare. writeStdoutFinal (cli-force-exit.ts) applies it to a guarded document during a slow read, and orphans, sources status and status call it directly. noteInterrupted prints the interrupt line and, under --json, writes {error: 'interrupted', signal, read, elapsed_ms, complete: false} at once through the writer cli-force-exit.ts registers (setInterruptedDocumentWriter); a SIGINT handler is installed only when nothing else listens. The fallback document for exits 129, 130 and 143 is code: 'interrupted'. Pinned by test/read-heartbeat-cli.test.ts (real spawned CLI fixture test/fixtures/slow-read-cli.ts: SIGTERM, SIGINT, SIGHUP, SIGKILL after SIGTERM, no --json, a kill outside a read) and test/read-meta-diag.test.ts.

  • src/commands/diag.ts (+ src/cli/commands/diag.ts, src/cli/help/diag.ts) — gbrain diag [--json] [--budget-ms N] [--query TEXT], read-only and refused on a thin client: one bounded bundle an agent pastes whole. collectDiag runs six sections in turn, each through diagSection under min(its cap in DIAG_SECTION_CAP_MS, what is left of the budget, default 30 s): gbrain (version, engine, runtime), skew (doctor runtime_build_skew and writer_version), sources (per source last_sync_at, last commit, behind_head from readGitBehind, the managed pass row's state from readLivePassRows), holds (counts per source and code), pool (Postgres only: backends by state, lock waits, idle transactions over 60 s, pg_stat_database session endings on 14+) and search (one search's stage timings and result count, never its results). Each section carries elapsed_ms and status ok, not_read (reason budget), not_applicable or error (with its code); the bundle carries elapsed_ms, complete and incomplete, error sections included. It never reads a source's file level (sync status --detail). Pinned by test/read-meta-diag.test.ts.

  • src/core/process-watchdog.ts — out-of-band hard-deadline killer for gbrain sync. A spinning sync (synchronous catastrophic-regex / ReDoS in pack link-inference) STARVES the main event loop, so the existing SIGTERM handler (process-cleanup.ts), --timeout setTimeout, and abort-flag checks can't fire — the process becomes unkillable-by-SIGTERM and, under cron, orphans pile up for 24h+. installProcessWatchdog({deadlineMs, graceMs?, label?, heartbeatMs?, onWarn?}): WatchdogHandle spawns a Bun worker_threads Worker via new Worker(code, {eval: true, workerData}) — its own OS thread + event loop fires even while main is in an unyielding sync loop. At deadlineMs it process.kill(process.pid, 'SIGTERM') (clean-shutdown chance if responsive); at deadlineMs+graceMs process.kill(process.pid, 'SIGKILL') (uncatchable — guaranteed death under starvation). Signaling SELF has NO PID-reuse footgun (current PID never reused while alive — the reason the detached-child-watches-parent design was rejected). eval: true bakes the worker body into the bun build --compile binary with no separate-file embedding. Empirically validated on Bun 1.3.13 (worker timer + SIGKILL killed a while(true){}-starved process). handle.dispose() (clean-exit finally) worker.terminate()s it; unref()'d so it never keeps the process alive. Pure watchdogDecision(elapsedMs, deadlineMs, graceMs) → 'wait'|'sigterm'|'sigkill' extracted for unit tests, and pure exported clampWatchdogTimers(deadlineMs, graceMs) + MAX_WATCHDOG_TIMER_MS clamp BOTH worker timers so the deadline AND the deadline+grace SUM stay ≤ 2^31−1 — setTimeout overflow-fires above that at ~1ms, which for the sum timer would be an instant SIGKILL of a healthy process (unit-tested as pure arithmetic only; never arm a real max-deadline worker in-suite — its firing SIGTERMs the test runner itself). Optional heartbeatMs emits periodic [<label>] parent alive Ns, hard-kill in ~Ms lines (visible in cron logs ev... (line truncated to 2000 chars)

  • src/core/forward-progress.ts — process-wide forward-progress signal, zero imports. noteForwardProgress() records that a unit of work completed (called by every progress.ts tick); onForwardProgress(listener) subscribes and returns the unsubscribe; lastForwardProgressAt() reads the last note. Consumers are deadlines that must never stop a run that is still progressing: the sync watchdog (process-watchdog.ts, progressWindowMs) and the gbrain serve boot deadline. Heartbeats are not progress. GBRA-133 PR 3a: noteCommittedProgress() records committed work (a page written, deleted or renamed, a cursor advanced; the sync import, delete and rename loops, the drain's page commits and checkpoint, and a followed drain holder whose committed count rose) and is also a forward note; onCommittedProgress / lastCommittedProgressAt read it. The sync watchdog listens to committed notes only.

  • src/core/spawn.ts — the one module in src/ that launches subprocesses with windowsHide defaulted (#4992, contributed by @rokas-tarasevicius in PR #6105). A detached: true child on Windows has no console, so every console program it launches without windowsHide opens a visible console window that takes focus; the Stop-hook push, the backup check, the job supervisor, its worker and every job child run that way. Wraps spawn, spawnSync, exec, execSync, execFile, execFileSync (Node's overloads and util.promisify.custom kept) plus bunSpawn / bunSpawnSync, each defaulting windowsHide: true and keeping an explicit false (only openBrowser in creds/redirect.ts). Calls go through child_process and Bun at call time, so spyOn in tests still intercepts them. Trade-off: with every stdio stream piped or ignored, libuv turns the flag into CREATE_NO_WINDOW and the child gets a windowless console of its own, so a program that prompts on the console directly prompts invisibly; gbrain's networked git calls set GIT_TERMINAL_PROMPT=0 (git-remote.ts also SSH_ASKPASS_REQUIRE=never), and inherited-stdio launches still attach to the terminal. scripts/check-windows-hide.ts (bun run check:windows-hide, in verify) is a RATCHET: a new runtime child_process import or Bun.spawn* in src/ fails unless the file is in its NOT_YET_MIGRATED list (the minions supervisor/worker/child/readiness/isolation/registry/shell handler, workspace-push.ts, cli/main.ts, context/hook-heartbeat.ts, the backup check, cli-force-exit.ts, cli-preflight.ts, autopilot-lock.ts, the autopilot commands and bounded-child-exec.ts are migrated; the rest, wave-13-claimed files included, are listed and lose the entry when they move), and a listed file that stops launching directly fails as stale. Pinned by test/spawn-windows-hide.test.ts (argument shapes; the detached supervisor, worker and push-git sites), test/scripts/windows-hide-guard.test.ts (forms, ... (line truncated to 2000 chars)

  • src/core/embed-stale-progress.ts — StaleChunkProgress (#5968, wave 15 P2.6 [R28]): the exact tally behind embed --stale progress. Unit: stale chunks. total starts at the census and only grows: listed(rows) raises it to the rows actually listed (a writer adding stale rows behind the keyset cursor), reentered(remaining) to what a paced re-entry will relist, and pageDone(census, handled) by the heal delta when one oversized stored chunk became several, in the same step that adds the page's rows to done (failed, filtered and quarantined rows included), so done <= total at every callback and a finished run reads 100%. Pinned by test/embed-stale-progress-5968.serial.test.ts.

  • src/core/progress.ts — Shared bulk-action progress reporter. Writes to stderr. Modes: auto (TTY \r-rewriting; non-TTY plain lines), human, json (JSONL), quiet. Rate-gated by minIntervalMs and minItems. startHeartbeat(reporter, note) for single long queries. child() composes phase paths. setTotal(total) (optional on the interface; #5968) replaces a running phase's total once the exact amount of work is known, so a caller whose work grows mid-phase (a heal split, a re-entry) never reports past 100%. Singleton SIGINT/SIGTERM coordinator emits abort events for every live phase. EPIPE defense on both sync throws and stream 'error' events. Every tick calls noteForwardProgress(), so bulk work feeds the progress-aware deadlines; its only import is that zero-import leaf. emitHumanLine is prefix-aware — inside a withSourcePrefix(id, ...) scope from src/core/console-prefix.ts it prepends [id] (and TTY-rewrite mode \r\x1b[2K carries the prefix inside the clear-to-EOL escape); emitJson is intentionally NOT prefixed so NDJSON consumers don't choke on a [id] {...} shape. JSON tick events carry cpu_ms, the main thread's CPU time since the phase started (process.threadCpuUsage()); the scale tier's PGLite import-rate gate (scripts/scale/gates.ts) judges on it.

  • src/core/retry.ts — canonical retry primitive for transient connection errors. Exports withRetry<T>(fn, opts) execution wrapper + BULK_RETRY_OPTS constant ({maxRetries:3, delayMs:1000, delayMaxMs:10000, jitter:'decorrelated'}, tuned for Supabase Supavisor's 5-10s circuit-breaker recovery) + BATCH_AUDIT_SITES typed const (closed enum of every audit-emission site) + resolveBulkRetryOpts(env) (reads GBRAIN_BULK_MAX_RETRIES/GBRAIN_BULK_RETRY_BASE_MS/GBRAIN_BULK_RETRY_MAX_MS with >=0 validation, throws on bad input with a paste-ready hint) + abortableSleep(ms, signal?) + RetryAbortError (tagged error for clean shutdown) + computeNextDelay() (pure-fn for 3 jitter modes: 'none', 'full', 'decorrelated'). The execution wrapper is consumed by postgres-engine.ts + pglite-engine.ts batch primitives (addLinksBatch / addTimelineEntriesBatch / upsertChunks) so every caller inherits retry as part of the data-primitive's contract. CI guard scripts/check-no-double-retry.sh fails the build on withRetry(...engine.batch...) patterns (prevents 3×3=9 retry amplification); scripts/check-batch-audit-site.sh validates every string-literal auditSite: '...' against the closed BATCH_AUDIT_SITES enum. Decorrelated jitter (AWS-style: uniform(base, prevDelay*3) capped at delayMaxMs) — 'full' jitter would allow near-zero retries that re-hit the recovering breaker. WithRetryOpts has an optional reconnect?: () => Promise<void> callback awaited in the catch branch AFTER isRetryableConnError classification but BEFORE the inter-attempt sleep — lets engine-level callers rebuild a dead pool/singleton between attempts. PostgresEngine.batchRetry injects () => this.reconnect() (the race-safe _reconnecting guard kicks in). Fail-loud: a reconnect throw PROPAGATES as the new error, replacing the symptomatic "No database connection". onRetry callbacks are awaited (sync arrows work identically; async callbacks correctly delay the sleep). Pi... (line truncated to 2000 chars)

  • src/core/stall-env.ts — shared env resolver for the stall-watchdog knobs (resolveStallAbortSecondsFromEnv(envVar, defaultSec)): one parameterized implementation behind BOTH GBRAIN_SYNC_STALL_ABORT_SECONDS and GBRAIN_EMBED_STALL_ABORT_SECONDS so the two surfaces' semantics cannot drift — unset/empty/garbage → the caller's default; any finite number returned as-is (<= 0 disables the watchdog). Deliberately a leaf module (neither the sync nor the embed cluster imports the other's machinery just to parse an env var); env-only incident knobs by design, no config-dashboard surface.

  • src/core/zombie-reap.ts — idempotent installSigchldHandler() so JS-spawned children get reaped via Bun's internal waitpid(). Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from src/cli.ts (Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via _uninstallSigchldHandlerForTests(). Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via src/core/minions/spawn-helpers.ts); Layer 3 is the container's own tini for hard Bun crashes.