Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
10.0 KB
Raw Blame

Progress events

Canonical reference for the JSONL progress stream that gbrain writes to stderr when a bulk command runs with --progress-json. Additive changes only; no renames or removals without a major version bump.

Most humans won't read this page. Agents parsing progress will.

When do I get these events?

Any of these commands stream events when --progress-json is set:

  • gbrain doctor (DB checks, JSONB integrity, markdown body completeness, integrity sample)
  • gbrain orphans
  • gbrain embed
  • gbrain files sync
  • gbrain export
  • gbrain extract [links|timeline|all] (fs or db source)
  • gbrain import
  • gbrain sync
  • gbrain migrate --to … (graduation reports one phase per step and ticks per table and batch; the legacy copier reports the migrate.copy_* phases below)
  • gbrain repair-jsonb
  • gbrain check-backlinks
  • gbrain lint
  • gbrain integrity auto
  • gbrain eval
  • gbrain eval brainbench
  • gbrain apply-migrations (the orchestrator + every child command)
  • gbrain transcripts ingest (per-file ticks + a per-session heartbeat over the import set)

Non-bulk commands (stats, graph-query, get, put, etc.) don't emit events — they return in under a second.

Channel

  • Progress events: stderr, one JSON object per line, \n-terminated.
  • Data results (--json payloads from each command): stdout.
  • Final human summaries: stdout.

Agents can safely capture stdout for their result parsing and read stderr separately for progress.

Flags

Flag Behavior
(none) Auto. TTY: \r-rewriting single line. Non-TTY: plain line-per-event on stderr.
--progress-json Force JSON-lines mode on stderr (this doc).
--quiet Suppress progress entirely. Warnings and final output still print.
--progress-interval=<ms> Override the minimum interval between tick emits (default 1000).

Global flags: parsed by src/core/cli-options.ts before command dispatch, so gbrain --progress-json doctor works the same as gbrain doctor --progress-json (the latter also works — per-command parsers see the flag via the shared CliOptions singleton).

Event types

Every event is a single-line JSON object with these common fields:

Field Type Notes
event string One of: start, tick, heartbeat, finish, abort.
phase string Machine-stable snake_case, dot-separated. See "Phase names" below.
ts ISO 8601 UTC string Event emission time.
elapsed_ms number Ms since the phase started. Present on tick/heartbeat/finish/abort.

start

Emitted when a phase begins.

{"event":"start","phase":"doctor.db_checks","ts":"2026-04-20T12:34:56.789Z"}
{"event":"start","phase":"import.files","total":52000,"ts":"2026-04-20T12:34:56.789Z"}

Optional fields:

  • total — the total item count if known at start.

tick

Emitted periodically during iteration. Time- and item-gated: the reporter won't emit more often than minIntervalMs (default 1000) and minItems (default max(10, ceil(total/100))).

{"event":"tick","phase":"orphans.scan","done":15000,"total":52000,"pct":28.8,"elapsed_ms":4200,"cpu_ms":3900,"eta_ms":10300,"ts":"..."}

Fields:

  • done — items completed in this phase.
  • cpu_ms — CPU time (user + system) the emitting process's main thread has used since the phase started. Unlike elapsed_ms it excludes time the host gave the CPU to other work, and it excludes work on runtime helper threads (garbage collection, JIT) and in other processes (a Postgres server, a provider). The scale tier's import-rate gate times PGLite imports with it. Under a hypervisor whose guest kernel does not subtract steal time (Ubuntu's generic kernels), it also includes any vCPU stall that landed while this thread was current, whether the host reports it as steal (the vCPU waiting for a CPU) or not (the vCPU asleep in the host, as on a page the host had to back or reclaim first), so on an oversubscribed VM it can exceed the thread's real work.
  • total — total items, if known. Omitted when the scan doesn't have a total up front (e.g. a streaming iterator).
  • pct — done/total * 100, one decimal. Omitted when total is unknown.
  • eta_ms — projected ms until done === total, from the observed rate. Omitted when total is unknown.
  • note — optional string with the current item (e.g. a slug or filename).

heartbeat

Emitted for long-running single operations that don't iterate (e.g. SELECT against a 50K-row table). No done, no total — just a signal that work is still happening.

{"event":"heartbeat","phase":"doctor.markdown_body_completeness","note":"scanning pages for truncation…","elapsed_ms":1000,"ts":"..."}

finish

Emitted when a phase completes normally.

{"event":"finish","phase":"import.files","done":52000,"total":52000,"elapsed_ms":187000,"ts":"..."}

abort

Emitted by a single process-level SIGINT/SIGTERM handler that tracks every live phase. After abort, no further events emit for that phase.

{"event":"abort","phase":"doctor.markdown_body_completeness","reason":"SIGINT","elapsed_ms":5300,"ts":"..."}

Phase names

Phases use snake_case.dot.path naming. A fresh reporter starts at the root; child() composition appends to the parent's current phase, so a sync that calls import emits sync.import.<file>, not import.<file>.

Stable phase names:

  • doctor.db_checks (umbrella for all DB-side doctor checks)
  • doctor.pglite_probe (the scratch-store probe; only when PGLite init failed with an unexplained/damage-class disk state or --probe-pglite was passed — a cold start can take 5–20s, so the heartbeat is the only sign of life)
  • orphans.scan
  • embed.pages (for embed --stale the unit is stale chunks since #5968: total is the census, grown by heal splits and re-entries so done <= total always; other embed paths tick per page)
  • extract.links_fs, extract.timeline_fs, extract.links_db, extract.timeline_db
  • import.files
  • sync.deletes, sync.renames, sync.imports
  • migrate.copy_pages, migrate.copy_links
  • migrate.copy_facts (heartbeat-only, one tick per fact row; the facts table re-copies fully on every run, so the tick count is the whole table)
  • migrate.reembed (the re-embed pass of gbrain migrate embeddings; total is the stale-chunk backlog at the start of the pass, so it can grow slightly if a writer adds chunks mid-run)
  • repair_jsonb.run, repair_jsonb.<table>.<column>
  • backlinks.scan
  • backlinks.fix — heartbeat-only (no total): the fix loop runs per-file locking + parse-validation + atomic writes, so agents see forward progress while it works through the gap list
  • lint.pages
  • integrity.auto
  • eval.single, eval.ab
  • eval.brainbench — ticks carry a note but no total: continuity pairs replay once per (writer, reader) ordering, so the tick count exceeds the fixture count and a percentage would lie
  • export.pages
  • files.sync
  • transcripts.ingest (one tick per session-log file; sessions inside a multi-session file — the hermes store, consumer exports — don't get their own ticks, so total = file count; each session emits a heartbeat instead)
  • sync.github_materialize (github-kind source sweep: the phase restarts once per repo with that repo's item count as total — counts are only known after each repo's enumeration — and every item ticks exactly once; scope resolution, per-repo listing and detail fetches emit heartbeats)
  • sync.google_materialize (google-kind source sweep: no total — the newest-first Gmail backfill drains a window whose size isn't known up front — with one tick per Gmail thread materialized, note carrying the thread id; the contacts, calendar, and gmail service sweeps each emit heartbeats while they run)

Sub-phases exposed via child():

  • sync.import.files — nested inside a sync
  • apply_migrations.v0_12_2.jsonb_repair — nested inside the orchestrator

Subprocess inheritance

When a parent CLI spawns gbrain … child processes (mostly in src/commands/migrations/*), global flags (--quiet, --progress-json, --progress-interval) are propagated to the child's argv via the childGlobalFlags() helper in src/core/cli-options.ts. Child stderr passes straight through stdio: 'inherit' so the event stream is one merged JSONL feed on the parent's stderr.

One exception: the orchestrator phase in migrations/v0_12_2.ts that captures child stdout (repair-jsonb --dry-run --json for verification) does not pass --progress-json to avoid any risk of stdout pollution breaking the orchestrator's JSON.parse. Its stdio is explicit: ['ignore', 'pipe', 'inherit'] so stderr still flows through.

Minion jobs

gbrain jobs work (the Minion worker daemon) keeps progress in the DB, not on stderr. Each Minion handler that runs a bulk core (embed, sync, extract, import, backlinks) calls job.updateProgress({done, total, …}) per iteration. Agents read per-job progress via the get_job_progress MCP operation or gbrain jobs get <id>. A sync job writes the phase it is in (resolve_repo, git_pull, full_import, import with banked_files, or the managed drain's managed_sync.* fields), and autopilot-cycle / autopilot-global-maintenance write {phase, phase_index, phase_count, started_at} as each cycle phase starts (the phase names are gbrain dream --phase names). These writes are best-effort: a failed progress write never fails the job.

The jobs work daemon itself emits coarse one-line-per-job stderr output for liveness only. Per-page detail lives in the DB.

Compatibility

  • Added: only. A new event type, a new field, a new phase name — all safe. Agents must ignore unknown fields and unknown event types.
  • Removed/renamed: never without a major version bump.
  • Schema changes: announced in CHANGELOG.md and in skills/migrations/v<next>.md.

If your agent depends on this schema and something surprises you, open an issue with the event you received and what you expected.