Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
17.1 KB
Raw Blame

Key files: Core Cycle primitive

Subsystem index. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.

  • src/core/cycle.ts — brain maintenance cycle primitive (ALL_PHASES is the ordered source of truth; fence_repair follows sync and runs the fences repair kind through cycle/fence-repair.ts). runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport> composes phases in semantic order along the core spine lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans, with the extraction/graph/consolidation phases (extract_facts, extract_atoms, resolve_symbol_edges, …) slotted between per the ordering comments on ALL_PHASES. synthesize runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); patterns runs after extract so it reads a fresh graph (subagent put_page sets ctx.remote=true and skips auto-link/timeline by default, so extract is the canonical materialization); recompute_emotional_weight sees the union of syncPagesAffected + synthesizeWrittenSlugs + the source window's slugs (cycle/source-window.ts: a per-source cycle that runs no sync works on at most 2,000 pages changed since its previous cycle, #6803, #8573; its cursor is stamped as UTC ISO with microseconds via to_char, never ::text, so a non-ISO session DateStyle cannot reset the walk, #8771) incrementally, or all pages when none of those anchors is set (full backfill via gbrain dream --phase recompute_emotional_weight). CycleReport.schema_version: "1" is stable; totals is additive (pages_emotional_weight_recomputed, transcripts_processed, synth_pages_written, patterns_written). Three callers: gbrain dream CLI, gbrain autopilot daemon inline path, the Minions autopilot-cycle handler. runPhaseLint skips with status: 'skipped' and details.reason: 'disabled' before any scan when cycle.lint.enabled is explicitly false (#6369, cycleLintPhaseEnabled in cycle/lint-fix-setting.ts; onceForPhase === 'lint' threads through as opts.once and bypasses it for one run; gbrain lint never reads it), repairs with fix: true unless DB config cycle.lint_fix is explicitly false (report-only; gbrain lint --fix unaffected); cycle.lint_exclude (#6134, cycle/lint-fix-setting.ts cycleLintExcludes/parseCycleLintExclude) lists basenames the lint phase and the lint/lint-fix minion handlers skip (details.excluded; config set refuses a path); managed repairs left pending keep the phase at warn and surface as details.pending (test/managed-lint.test.ts). Coordination via gbrain_cycle_locks DB table + ~/.gbrain/cycle.lock file lock with PID-liveness for PGLite; the two handles compose into ONE lock whose refresh() treats the fenced DB row as the authoritative multi-writer identity and the file half as best-effort freshness — the file lock is refreshed ONLY while the fenced DB refresh reports ownership, so a losing holder can never clobber the successor's file lock on the very tick it detects the steal. A dedicated serialized refresher — startCycleLockRefresher(lock, controller, lockId) (exported; unref'd setInterval at max(15s, TTL/6), env-only override GBRAIN_CYCLE_LOCK_REFRESH_MS, in-flight guard so a slow refresh never overlaps the next tick) — heartbeats the lock through long phases (synthesis/patterns/consolidation waits routinely outlive the TTL with no other heartbeat). A fenced miss aborts the controller with LockStolenError; that steal signal combines with the worker's external signal via the exported anyAbortSignal(signals) (src/core/abort-signals.ts, re-exported; duck-type-tolerant — stubs without addEventListener are observed by poll; returns {signal, dispose} and dispose detaches the caller-signal listener + clears the poll timer so daemons don't leak), and the run stops at the next phase boundary with a structured partial report carrying reason: 'lock_stolen' (for the 5 long phases — synthesize / extract_atoms / patterns / synthesize_concepts / consolidate — the steal races the phase promise and stops the WAIT immediately; extract_atoms also receives the cycle signal as a real AbortSignal and stops at its next item, the others run to their own bounded timeout). A busy DB lock's cycle_already_running skip carries lock_holder (id, holder_pid, holder_host, age_ms). A source-scoped cycle that selects a brain-wide phase (in NEEDS_LOCK and not source scope) also tries the shared gbrain-cycle lease through cycle/lock-set.ts; when another cycle holds it, the source phases run and each brain-wide phase is reported skipped with maintenance_lock_busy (an all-skipped selection reports skipped). Transient refresh errors log and retry next tick (the TTL stays the backstop). yieldBetweenPhases runs between phases; yieldDuringPhase is in-phase keepalive; onPhaseStart(phase, index, count) is awaited inside timePhase as each selected phase starts its work (index and count within the resolved selection; a throwing hook is logged, never fatal), which the autopilot-cycle handler turns into job progress. Engine nullable; lock-skip on read-only phase selections. CycleOpts.signal?: AbortSignal propagates the worker's abort signal with checkAborted() between every phase. CycleOpts.deadlineAtMs (the enclosing minion job's ABSOLUTE wall-clock deadline, threaded from MinionJobContext.deadlineAtMs by the autopilot-cycle handler; null for direct gbrain dream callers) flows into patterns, propose_takes, synthesize AND fence_repair so time-spending phases derive their deadlines from the REAL remaining job budget instead of duplicated literals that collide with the job's own kill timer. Pinned by test/cycle-lock-steal.serial.test.ts (mid-run steal → partial report, no further phases, successor row intact) + test/cycle-any-abort-signal.test.ts + test/db-lock-fencing.test.ts. runPhaseSync returns pagesAffected via SyncPhaseResult (threaded to runPhaseExtract as the 4th arg) and takes willRunExtractPhase: boolean setting noExtract: phases.includes('extract') so gbrain dream --phase sync doesn't silently lose extraction. The extract phase calls runExtractCore and extractStaleFromDB with quiet: true (and jsonMode: false), so the helpers' human summaries never land on stdout ahead of the dream --json CycleReport while batch-loss diagnostics stay human-readable on stderr (quiet is its own knob on ExtractOpts — using jsonMode: true as a stand-in flipped the stderr channel to JSON events in a plain gbrain dream); totals.pages_extracted counts pages (pages_processed from the targeted pass + stale_pages_drained), not links created. resolveSourceForDir(engine, brainDir) threads sourceId to performSync() so sync reads the per-source sources.last_commit anchor (not the drift-prone global config.sync.last_commit). It matches only an absolute local_path; with no match the sync phase checks default and fails sync_anchor_relative (with its fix) while that path is relative. CycleOpts.brainDir is string | null; when null (checkout-less postgres/Supabase brain, or a connector source) the 5 filesystem phases (lint/backlinks/sync/synthesize/patterns) skip with details.reason: 'no_brain_dir', extract runs the database-only stale drain (cycle/connector-extract.ts), and the DB-only phases run; resolveSourceForDir is null-tolerant. cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir) is the canonical per-source scope for extract_facts/extract_atoms/calibration — and for synthesize (threaded as SynthesizePhaseOpts.sourceId so synthesized pages land in the cycle's resolved source, not 'default') — so gbrain dream --source repo-a reconciles repo-a's facts even with no checkout (instead of scoping to 'default' while stamping repo-a fresh). deriveStatus counts edges_resolved/edges_ambiguous as work so an edges-only cycle reports ok not clean, and scores ONLY attempted phases — the implicit source-cycle exclusion skip-records are bookkeeping and never dilute failure aggregation; the jobs.ts autopilot-cycle + phase-wrapper handlers pass null (not '.') when no repo is configured. The cycle is SPLIT for autopilot fan-out along the PHASE_SCOPE taxonomy in src/core/cycle/phase-scope.ts (see its entry): cycle.ts derives SOURCE_PHASES / MIXED_PHASES / GLOBAL_PHASES / MAINTENANCE_PHASES (mixed ∪ global, original cycle order) plus SOURCE_FRESHNESS_PHASES (deterministic, non-LLM: lint/backlinks/sync/extract/extract_facts/recompute_emotional_weight — defined in phase-scope.ts) and SOURCE_BACKGROUND_PHASES (LLM-backed/unbounded source work = SOURCE minus FRESHNESS). resolveCyclePhases(requested, sourceId) at the shared runCycle boundary: default/no-source → requested ?? ALL_PHASES (the canonical default cycle remains full); a named non-default source with NO explicit phases → SOURCE_FRESHNESS_PHASES only (the freshness keeper's implicit dream --source X path must stamp freshness without first draining LLM-backed maintenance); explicit phase lists are honored VERBATIM — dream --source X --phase synthesize (and --input <file>, which implies synthesize) is deliberate operator intent, and --phase orphans --source X still narrows orphans via forceGlobalOrphans. The N-way duplication guard lives at the QUEUE boundary instead: the autopilot-cycle handler intersects queued per-source payloads with SOURCE_FRESHNESS_PHASES (queued payloads may carry mixed+background phases; an all-rejected or empty list is an explicit no-op skip with reason all_phases_rejected_by_normalization, never an implicit run, and rejected phases surface on the job result as phases_rejected_by_normalization). On the implicit path, excluded phases surface as skipped with details.reason: 'excluded_from_implicit_source_cycle' + phase_scope. Per-source autopilot-cycle jobs enqueue phases: SOURCE_FRESHNESS_PHASES and stamp last_source_cycle_at; the single autopilot-global-maintenance job runs MAINTENANCE_PHASES (no sourceId) and stamps the brain-level autopilot.last_global_at config key (LAST_GLOBAL_AT_KEY). SOURCE_BACKGROUND phases have NO automatic lane on multi-source brains (the legacy full-cycle job fires only when the brain has no sources rows) — run them explicitly (gbrain dream --source X --phase extract_atoms) until the background lane lands (see TODOS). The freshness stamp gate is opts.sourceId && phases.length > 0 && engine && !dryRun && !aborted && status ∈ {ok, clean, partial}; last_full_cycle_at is still written alongside last_source_cycle_at on a per-source success for doctor/legacy readers (not a gate for the brain-wide phases). A lost filesystem-walk or incremental extract write (ExtractBatchWriteError, code extract_write_lost, src/core/extract-watermark.ts) leaves only its pages unstamped and makes the extract phase warn, like a GBRA-130 A-1 stale drain failure; the same cycle's stale drain retries those pages. Pinned by test/dream-postgres.serial.test.ts + test/jobs-autopilot-cycle-braindir.test.ts + test/autopilot-global-maintenance.test.ts + test/cycle-enabled-phase-completeness.test.ts. runPhaseLint + runPhaseBacklinks carry the export keyword so behavioral tests can drive them directly (internal helpers exposed for test-only consumption; downstream code should NOT depend on them). Pinned by test/cycle-legacy-phases.test.ts (11 cases across both phases: clean run → status='ok', partial fix → status='warn' with dryRun in details, dry-run path doesn't write, throw-from-lib → status='fail' with the wrapper's try/catch envelope populated). with src/core/cycle/extract-atoms.ts + src/core/cycle/synthesize-concepts.ts + src/commands/extract.ts + src/commands/doctor.ts + src/core/op-checkpoint.ts: six daily-driver ops behaviors. (1) Batch idempotency: atomsExistingForHashes(engine, sourceId, hashes[]) (exported from src/core/cycle/extract-atoms.ts) answers in one batched SQL roundtrip (never a per-hash loop) returning already-extracted content_hash16 values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 pages_atom_source_hash_idx (partial expression index on frontmatter->>'source_hash' for atom rows where deleted_at IS NULL; Postgres CREATE INDEX CONCURRENTLY with invalid-remnant pre-drop, PGLite plain). (2) Cycle lock TTL + heartbeat: LOCK_TTL_MINUTES = 5; buildYieldDuringPhase(lock, outer) (exported, with LockHandle) calls lock.refresh() + any external hook on every fire, throttled to 30s via maybeYield, firing both in the main loop AND immediately after every await chat(...); synthesize_concepts uses the same throttled hook. A crashed cycle releases its lock within one short TTL, while the dedicated startCycleLockRefresher interval (see the coordination sentence above) keeps a healthy long-running cycle alive even across a single multi-minute await chat(...) — the timer fires during awaits, so no single slow call can silently expire the lock. (3) Progress wiring: progress?: ProgressReporter opt on ExtractAtomsOpts and SynthesizeConceptsOpts; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on cycle.extract_atoms.extract_atoms.work); phases only call tick()/heartbeat(), cycle.ts owns start()/finish(). (4) by-mention resume: mentionsFingerprint({source, type, since, gazetteerHash}) in src/core/op-checkpoint.ts — the gazetteer hash (hashGazetteer in src/core/by-mention.ts — every entry's source_id/slug/title/tokens, not just the first-token bucket keys, so a new "Acme Labs" beside an existing "Acme Corp" moves it) is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); findMentionedEntities keys its self-link guard and first-mention dedup on (source_id, slug), so under link_resolution.cross_source a foreign namesake of the scanning page is a real target and same-slug pages in two sources both link; the gazetteer itself is entity-page titles PLUS live-verified page_aliases entries joined to entity-typed pages (ambiguous aliases and alias-vs-title collisions within a source are skipped), so body mentions of a documented alias link too; gbrain extract links --by-mention resumes via op_checkpoints with flushAndCheckpoint ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; --dry-run skips both load and write. (5) sync_consolidation doctor check (multi-source brains see a paste-ready gbrain sync --all --parallel 4 --workers 4 --skip-failed; a managed brain (managedPersistenceEnabled, false when persistence_brain is missing) sees gbrain sync --all --no-pull --hard-deadline 13m every 15 minutes on the owner host with gbrain sources refresh <id>; chained first for remote-tracking checkouts; single-source "not applicable"; SQL errors return warn via the check's own try/catch). (6) Test-isolation: test/cycle-last-full-cycle-at.test.ts + test/schema-cli.test.ts use per-test GBRAIN_HOME=tempdir. Pinned by test/cycle/extract-atoms-batch.test.ts, test/core/cycle.serial.test.ts (5-minute DB-lock and file-lock TTL behavior), test/op-checkpoint-mentions-fingerprint.test.ts, test/cycle/extract-atoms-progress.test.ts, test/cycle/synthesize-concepts-progress.test.ts, test/cycle/yield-during-phase-refresh.test.ts, test/cycle/yield-during-phase-throttle.test.ts, test/extract-by-mention-resume.test.ts, test/doctor-sync-consolidation.test.ts. Companion sync --all recipe block in skills/cron-scheduler/SKILL.md. synthesize_concepts writes concept pages through importFromContent (the same parse→chunk→embed pipeline put_page uses, with put_page's isAvailable('embedding') → noEmbed gate) so concepts/ pages carry content_chunks + embeddings and are reachable by retrieval (where source-boost.ts weights them 1.3×). purge phase (soft-delete TTLs) also GCs stale op_checkpoints rows older than 7 days and, when cycle.purge.mcp_request_log_retention_days is set, prunes the request log (runMcpLogRetention; purged_mcp_request_log_count in details only when above 0). Non-fatal on pre-v67 brains (DROP-target-table check before DELETE). The cycle threads its abort signal into the embed phase (runPhaseEmbed(engine, dryRun, signal)) so a timed-out cycle's long embed phase honors cancellation and releases gbrain_cycle_locks right away instead of after a full backlog run. The extract_facts phase runs the S9 conflict sweep as its tail (ai/decide/sweep.ts conflictSweepTail) only when decide.slots.conflict.mode is not off, adding details.decide_conflict; with the slot off the phase output is unchanged. The extract_atoms / synthesize_concepts pack gate (#6393) is src/core/cycle/pack-phase-gate.ts (Core Cycle (continued)); packDeclaresPhase(engine, phase, sourceId?) delegates to it.