Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
concept synthesis: recheck the abort signal before each publish and keep accounting on abort
garrytana40afb6956af
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c8c05e5af4..499829109c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -20,7 +20,7 @@ identifiers and attribution are available in the pre-removal Git revision**`gbrain dream --help --json` prints its help as one JSON document.** Under `--json`, stdout carries only a JSON document, and dream printed its help as plain text, which went to stderr. So stdout was empty, the run exited 0 and logged a missing-document event that `gbrain doctor` counts under its agent-contract check. It now prints `{"command": "dream", "help": "<the help text>", "contract_version": 1}`, and `dream retriage --help --json` and `dream reset-key --help --json` do the same. `gbrain dream --help` without `--json` prints exactly the same text as before. Reported by @Kkkkkuro (#8510).-**Concept synthesis stops when its maintenance job times out or is cancelled.** The `synthesize_concepts` dream phase never looked at the cycle's abort signal, so with a slow model it kept calling the model and publishing concept pages after its job had timed out, and an aborted model call could be written as a template fallback page. The phase now passes the signal to the model call and checks it before each concept and before each publication, so the run stops before the next concept page. No setting changed. Reported by @Nazim22 (#7986).+**Concept synthesis stops when its maintenance job times out or is cancelled.** The `synthesize_concepts` dream phase never looked at the cycle's abort signal, so with a slow model it kept calling the model and publishing concept pages after its job had timed out, and an aborted model call could be written as a template fallback page. The phase now passes the signal to the model call and checks it before each concept and again right before each page is written, so nothing is published once the job is cancelled. The spend of the model calls that finished and the failed-attempt counts are still saved, so the recorded spend and the retry limit keep counting them. On a managed brain, concepts synthesized but not yet published when the job stops are synthesized again on the next run. No setting changed. Reported by @Nazim22 (#7986).### Itemized changes@@ -28,7 +28,7 @@ identifiers and attribution are available in the pre-removal Git revision- `src/core/cycle/source-window.ts`: the listing instant and the last page's `updated_at` are formatted with `to_char(... AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`. Test: `test/sync-disabled-cycle-window.test.ts` (the real queued handler under `German, DMY` advances window by window; `German`, `SQL` and `Postgres` styles under a +05:45 time zone give ISO UTC cursors with microseconds and the same windows), now also run on Postgres by `test/e2e/sync-disabled-cycle-window-postgres.test.ts`.- `src/core/cycle/extract-atoms-drain.ts`: the no-progress recount runs after every batch with `extracted === 0`, not only when `skipped` is also 0. Test: `test/extract-atoms-drain.test.ts` (a skip-only batch over a flat backlog stops `no_progress` after one batch; skip-only batches that shrink the backlog drain).- `src/cli/help/render.ts`: new `printHelpText(command, text, args)`, the `--help --json` document for a handler that prints its own help text; `src/commands/dream.ts`, `dream-retriage.ts` and `dream-reset-key.ts` print their help through it. Test: `test/dream-help-json-8510.test.ts` (the real CLI: one document per help, no `json_document_missing` event, plain `--help` stdout equal to the document text).-- `src/core/cycle/synthesize-concepts.ts`, `src/core/cycle.ts`: `signal` option, passed from the cycle like extract_atoms', handed to the narrative `chat` call and checked before each concept, before its publication and before the final managed flush; an abort throws `AbortError`. Test: `test/cycle/synthesize-concepts-abort-7986.test.ts` (`runCycle` under the creator pack: a job aborted during the second model call publishes only the first concept, makes no third call, and an aborted call is not published).+- `src/core/cycle/synthesize-concepts.ts`, `src/core/cycle.ts`: `signal` option, passed from the cycle like extract_atoms', handed to the narrative `chat` call and checked before each concept, right before each publication and before the final managed flush; an abort saves the retry counters and the finished spend to the rollup, then throws `AbortError`. Test: `test/cycle/synthesize-concepts-abort-7986.test.ts` (`runCycle` under the creator pack: a job aborted during the second model call publishes only the first concept and makes no third call; an aborted call is not published; an abort during the page read before the first publication publishes nothing; an aborted run still writes its spend and failed attempts).## [0.60.620.0] - 2026-11-09diff --git a/docs/architecture/key-files/agent-bootstrap-continued.md b/docs/architecture/key-files/agent-bootstrap-continued.mdindex fadeb2eb3e..76cefacc2b 100644--- a/docs/architecture/key-files/agent-bootstrap-continued.md+++ b/docs/architecture/key-files/agent-bootstrap-continued.md@@ -37,5 +37,5 @@- `src/core/bootstrap/verify.ts` + `status.ts` — verify is the definition of done: fail-soft check suite over the REAL write path (put_page op, waiting as long as a CLI write: `resolveCliWriteWaitMs`, so `GBRAIN_WRITE_WAIT_MS`/`persistence.write_wait_ms`, default 30 s; a write still pending after that names its request and `gbrain write-request` poll command → write-through file under `brain/` → in-process sweep → graph floor via link tables → recall), the keyless magic-moment check (`## Facts` fence → zero-LLM reconciliation → world-visibility read-back), source_id collision resolution (as the one bootstrap subcommand holding an engine: a manifest `source_id` already registered to a DIFFERENT checkout → derives a stable `workspace-<8char-path-hash>`, persists it to agent.json, names the re-register steps — every consumer reads `manifest.source_id`), the `embedding_plane` check (keyed installs live-embed ONE probe string and compare the RETURNED width to the actual `content_chunks.embedding` width — a plane split is a named FAIL with the recovery command, keyless passes, a dead probe warns; runs BEFORE the roundtrip so a keyless-passing roundtrip can never certify a brain whose keyed writes all fail), token sweep, byte floors, secret scan, deny globs, repo privacy, hooks smoke (in-process IPC), capability report, first-run tour; snapshots kept last-5 under `<home>/bootstrap/`. The roundtrip's probe section runs under one try/finally, so both fixed-slug probes are removed on every exit, early failure returns included (#6001; pinned by the early-exit case in `test/bootstrap-verify.serial.test.ts`). On a managed brain each probe is purged through the coordinator and its purge tombstones (page, plus the fact/take purges of that request) are then cleared with the host-only `unpurgePage`, also on every sweep, so the constant probe bodies are never refused `purged_content` on the next run and `page_purges` does not grow (#6634). status owns the ordered PHASES list (the runbook defers to it), artifact-first detection, install.jsonl, the runbook version-stamp skew check, and the support blob doctor/agents relay verbatim.- `src/core/bootstrap/template-repo.ts` + `scripts/generate-template-repo.ts` — deterministic public-template generation (render `--minimal` + placeholder manifest + stamped README); published only by the release workflow after diffing against the vendored tree.- `scripts/check-bootstrap-tag.sh` / `scripts/check-bootstrap-templates.sh` — CI guards: sanctioned distribution ref only (`stable`; promote-stable moves it to a published release whose exact tree passed the soak; `latest-stable`, the alpha line, is refused) + runbook stamp == VERSION; template↔question-bank token bijection + placeholder-only assertion + offline generator↔vendored byte-diff + runbook-phase↔status.ts consistency. Both skip gracefully when their subjects are absent.-- `src/core/cycle/synthesize-concepts.ts` concept-quality addendum — a `concepts/<x>` page whose frontmatter lacks `synthesized_by: synthesize_concepts-*` belongs to a human or another writer: the phase checks it before any LLM spend, never rewrites it, and reports it in `details.skipped_human_owned` (pinned by `test/cycle/synthesize-concepts-human-owned.test.ts`). Concept refs are normalized before grouping (`conceptStemFor`: last path segment through `slugifySegment`, validated by `validatePageSlug`; one atom counts once per concept), so case and spacing variants share one page and no slug carries a space; punctuation variants (`network-effects` / `networkeffects`) fold per run through `cycle/concept-redirects.ts` `addSpellingRedirects` (#5965; documented with `concept-publication.ts` in `core-cycle-continued.md`). Each page records `member_hash` (member atom slugs, titles and bodies without their `## Facts` / `## Takes` sections, which the prompt also omits, plus the synthesis model; a page still carrying the full-body hash whose inputs are unchanged is rehashed without a model call, `details.rehashed`; a narrative a `chat_fallback_chain` model wrote is hashed under that model, so the next cycle re-synthesizes it); an unchanged hash on an `llm` or `deterministic_tier` page is skipped without spend (`details.skipped_unchanged`), and a budget or error fallback never replaces an existing `llm` narrative (`details.kept_existing_narrative`); fallback pages retry next cycle. Pinned by `test/cycle/synthesize-concepts-identity.test.ts`. Eligible groups are processed deterministically by tier, descending atom count, then concept slug so the fixed LLM budget reaches the strongest evidence first regardless of database row order. Every page and phase receipt distinguishes `llm`, intended `deterministic_tier`, `budget_fallback`, and `error_fallback` synthesis modes. The narrative call's output cap is sized from the resolved `models.dream.synthesize` model via `resolveSynthMaxOutputTokens`: 500 for non-thinking models, the gateway's `THINKING_MODEL_MAX_OUTPUT_TOKENS` (no phase-private number — DeepSeek v4 truncates at 8192-class caps) when the gateway's shared `isThinkingModel` matches (name-matched Claude 5 or recipe-declared `thinking_by_default`; unknown providers count as non-thinking), so a tier-`reasoning` thinking model no longer spends the whole budget on reasoning and persists a template stub or truncated chain-of-thought as the concept narrative. Pinned by `test/cycle/synthesize-concepts-token-cap.test.ts`. Right after each concept page write the phase banks concept<->member-atom provenance edges through `engine.addLinksBatch` (`link_source: 'concept-provenance'`; `synthesized_from` concept->atom, `synthesizes` atom->concept; audit site `cycle.synthesize_concepts.provenance`), both endpoints scoped to the cycle's source, so the concept pages are graph-reachable (backlinks, relational recall, doctor `graph_signals_coverage`/`orphans`) even though the prompt forbids enumerating atoms in the body. The dedicated `link_source` keeps reconcile passes from pruning them; `ON CONFLICT DO NOTHING` makes re-runs the backfill; a failed or zero-row edge write lands in `link_warnings[]` (phase `warn`) — NOT in `failures[]`, which means "LLM-failed → template fallback" downstream and drives the rollup's `halt_delta`/`round_completed_delta` — and never aborts the page write. Atom discovery (the `type = 'atom'` scan) is scoped to the cycle source (`opts.sourceId ?? 'default'`) so same-slug atoms from other sources are never grouped into this source's concepts. Pinned by the `#4589` describe in `test/cycle/extract-atoms-synthesize-concepts.test.ts` and `test/cycle/synthesize-concepts-source-scope.test.ts`. The phase budget is `cycle.synthesize_concepts.budget_usd` (finite USD above 0, default 1.5; `config set` refuses other values, a stored invalid value keeps the default with a named-key warning, specs in `src/core/cycle/phase-config-values.ts`); groups refused at the cap are counted in the summary and `details.warnings`. Pinned by `test/cycle-synthesize-concepts-budget-config.test.ts`. A narrative whose stop reason is `length`, `refusal` or `content_filter` is a failure (template fallback for a new concept; an existing `llm` page and its `member_hash` stay), and failed narratives are bounded per concept and member hash by `cycle/concept-retry-bound.ts` (`ConceptRetryBound`, DB-only config key `dream.concepts.failed_attempts`, `MAX_CONCEPT_ATTEMPTS` = 3): a bounded concept is skipped before any spend into `details.skipped_retry_bound` until its members change, a narrative succeeds, or the key is unset (`test/cycle/synthesize-concepts-stop-reason.test.ts`). #7986: `opts.signal` (cycle.ts passes the cycle signal, as for extract_atoms) goes to the narrative model call and is checked before each concept and before its publication; an abort throws `AbortError` (uncontained), so a timed-out job publishes no later concept, never writes an aborted call as `error_fallback`, and skips the final managed flush, receipt and rollup (`test/cycle/synthesize-concepts-abort-7986.test.ts`).+- `src/core/cycle/synthesize-concepts.ts` concept-quality addendum — a `concepts/<x>` page whose frontmatter lacks `synthesized_by: synthesize_concepts-*` belongs to a human or another writer: the phase checks it before any LLM spend, never rewrites it, and reports it in `details.skipped_human_owned` (pinned by `test/cycle/synthesize-concepts-human-owned.test.ts`). Concept refs are normalized before grouping (`conceptStemFor`: last path segment through `slugifySegment`, validated by `validatePageSlug`; one atom counts once per concept), so case and spacing variants share one page and no slug carries a space; punctuation variants (`network-effects` / `networkeffects`) fold per run through `cycle/concept-redirects.ts` `addSpellingRedirects` (#5965; documented with `concept-publication.ts` in `core-cycle-continued.md`). Each page records `member_hash` (member atom slugs, titles and bodies without their `## Facts` / `## Takes` sections, which the prompt also omits, plus the synthesis model; a page still carrying the full-body hash whose inputs are unchanged is rehashed without a model call, `details.rehashed`; a narrative a `chat_fallback_chain` model wrote is hashed under that model, so the next cycle re-synthesizes it); an unchanged hash on an `llm` or `deterministic_tier` page is skipped without spend (`details.skipped_unchanged`), and a budget or error fallback never replaces an existing `llm` narrative (`details.kept_existing_narrative`); fallback pages retry next cycle. Pinned by `test/cycle/synthesize-concepts-identity.test.ts`. Eligible groups are processed deterministically by tier, descending atom count, then concept slug so the fixed LLM budget reaches the strongest evidence first regardless of database row order. Every page and phase receipt distinguishes `llm`, intended `deterministic_tier`, `budget_fallback`, and `error_fallback` synthesis modes. The narrative call's output cap is sized from the resolved `models.dream.synthesize` model via `resolveSynthMaxOutputTokens`: 500 for non-thinking models, the gateway's `THINKING_MODEL_MAX_OUTPUT_TOKENS` (no phase-private number — DeepSeek v4 truncates at 8192-class caps) when the gateway's shared `isThinkingModel` matches (name-matched Claude 5 or recipe-declared `thinking_by_default`; unknown providers count as non-thinking), so a tier-`reasoning` thinking model no longer spends the whole budget on reasoning and persists a template stub or truncated chain-of-thought as the concept narrative. Pinned by `test/cycle/synthesize-concepts-token-cap.test.ts`. Right after each concept page write the phase banks concept<->member-atom provenance edges through `engine.addLinksBatch` (`link_source: 'concept-provenance'`; `synthesized_from` concept->atom, `synthesizes` atom->concept; audit site `cycle.synthesize_concepts.provenance`), both endpoints scoped to the cycle's source, so the concept pages are graph-reachable (backlinks, relational recall, doctor `graph_signals_coverage`/`orphans`) even though the prompt forbids enumerating atoms in the body. The dedicated `link_source` keeps reconcile passes from pruning them; `ON CONFLICT DO NOTHING` makes re-runs the backfill; a failed or zero-row edge write lands in `link_warnings[]` (phase `warn`) — NOT in `failures[]`, which means "LLM-failed → template fallback" downstream and drives the rollup's `halt_delta`/`round_completed_delta` — and never aborts the page write. Atom discovery (the `type = 'atom'` scan) is scoped to the cycle source (`opts.sourceId ?? 'default'`) so same-slug atoms from other sources are never grouped into this source's concepts. Pinned by the `#4589` describe in `test/cycle/extract-atoms-synthesize-concepts.test.ts` and `test/cycle/synthesize-concepts-source-scope.test.ts`. The phase budget is `cycle.synthesize_concepts.budget_usd` (finite USD above 0, default 1.5; `config set` refuses other values, a stored invalid value keeps the default with a named-key warning, specs in `src/core/cycle/phase-config-values.ts`); groups refused at the cap are counted in the summary and `details.warnings`. Pinned by `test/cycle-synthesize-concepts-budget-config.test.ts`. A narrative whose stop reason is `length`, `refusal` or `content_filter` is a failure (template fallback for a new concept; an existing `llm` page and its `member_hash` stay), and failed narratives are bounded per concept and member hash by `cycle/concept-retry-bound.ts` (`ConceptRetryBound`, DB-only config key `dream.concepts.failed_attempts`, `MAX_CONCEPT_ATTEMPTS` = 3): a bounded concept is skipped before any spend into `details.skipped_retry_bound` until its members change, a narrative succeeds, or the key is unset (`test/cycle/synthesize-concepts-stop-reason.test.ts`). #7986: `opts.signal` (cycle.ts passes the cycle signal, as for extract_atoms) goes to the narrative model call and is checked before each concept, again right before each `writer.queue`/`publish` (after the reads that prepare it) and before the final managed flush. On abort the phase saves the retry counters and a rollup row with the finished calls' spend (no completed round), then throws `AbortError` (uncontained): nothing more is published, an aborted call is never written as `error_fallback`, and queued but unflushed managed concepts are dropped and redone next run (`test/cycle/synthesize-concepts-abort-7986.test.ts`).- `src/core/bootstrap/harness-cleanup.ts` — #6743 shared-skills cleanup convergence for the harness apply (step 10), `--status` and doctor's `bootstrap_harness_health`. `convergeHarnessCleanup` leaves stale enrollments (a working prior enrollment stays until the current one is ready; one whose credential is already revoked always leaves) and then revokes each carried `previous_ids` entry that no not-yet-left enrollment holds, one id at a time. `leaveRevokedEnrollment` runs the adapter's local leave with a caller that never contacts the host and records the entry left with reason `credential_revoked`: a revoked principal cannot authenticate `leave_brain`, so its membership is inert. `harnessCleanupItems` reports what stays pending per id (`revoke_pending`, `leave_pending`, `enrollment_retained`, `credential_unreadable`, with the token-row state from the read-only `tokenStatesWith` lookup); the apply and `--status` exit `bootstrap_cleanup_pending` (14) with a `fix` that re-runs the same command when every id converges and otherwise names the owner's revoke or restore step. Pinned by `test/bootstrap-harness-cleanup-6743.test.ts`.diff --git a/src/core/cycle/synthesize-concepts.ts b/src/core/cycle/synthesize-concepts.tsindex 65a97677a2..38d1e44dab 100644--- a/src/core/cycle/synthesize-concepts.ts+++ b/src/core/cycle/synthesize-concepts.ts@@ -351,8 +351,17 @@ export async function runPhaseSynthesizeConcepts(};const writer = conceptWriter({ engine, maintenance, sourceId: src, importSourceId: opts.sourceId, brainDir: opts.brainDir, writeThrough: conceptFiles !== null,retries, publicationDeferred, publicationHeld, linkWarnings, written: countWritten });- for (const group of atomGroups) {+ // #7986: an aborted run publishes nothing more, but keeps the retry counters and the spend of the calls it finished.+ const stopIfAborted = async (): Promise<void> => {+ if (!opts.signal?.aborted) return;+ if (!opts.dryRun) {+ await retries.save(engine);+ await upsertExtractRollup(engine, { kind: 'concepts', source_id: src, cost_delta: estimatedSpendUsd });+ }throwIfAborted(opts.signal, 'synthesize_concepts');+ };+ for (const group of atomGroups) {+ await stopIfAborted();const conceptSlug = `concepts/${group.conceptSlug}`;// A concept page this phase did not write belongs to a human (or another// writer). Check before any spend; never replace its body.@@ -459,7 +468,7 @@ export async function runPhaseSynthesizeConcepts(retries.fail(`${opts.sourceId ?? 'default'}:${conceptSlug}`, memberHash);}} catch (err) {- if (opts.signal?.aborted) throw err;+ await stopIfAborted();const msg = err instanceof Error ? err.message : String(err);// #3044 adoption: a whole-run LLM outage must not overwrite// existing concept pages with error_fallback stub narratives.@@ -494,7 +503,7 @@ export async function runPhaseSynthesizeConcepts(}if (!grandfathered) synthesisModeCounts[synthesisMode]++;- throwIfAborted(opts.signal, 'synthesize_concepts');+ await stopIfAborted();if (!opts.dryRun) {// #5525: tighten-only — a concept already private stays private.const prior = await engine.getPage(conceptSlug, { sourceId: src });@@ -521,6 +530,7 @@ export async function runPhaseSynthesizeConcepts(const write: ConceptWrite = { concept: group.conceptSlug, conceptSlug, atomSlugs: group.atomSlugs, visibility, covered, synthesisMode, synthesized, narrative,revision: existingSnapshot?.revision ?? null, baseline: existing?.compiled_truth ?? '' };// GBRA-60: a managed run's concepts publish in batches (concept-writes.ts), each one publication group on Postgres.+ await stopIfAborted();await (maintenance ? writer.queue(write) : writer.publish(write));} else countWritten();@@ -529,7 +539,7 @@ export async function runPhaseSynthesizeConcepts(// right cadence (30s instead of every-group).await maybeYield();}- throwIfAborted(opts.signal, 'synthesize_concepts');+ await stopIfAborted();if (maintenance) await writer.flush();// v0.42 Wave B3: receipt + rollup for synthesize_concepts. Receipt/rollupdiff --git a/test/cycle/synthesize-concepts-abort-7986.test.ts b/test/cycle/synthesize-concepts-abort-7986.test.tsindex fb6a70b5c5..bf782cc11a 100644--- a/test/cycle/synthesize-concepts-abort-7986.test.ts+++ b/test/cycle/synthesize-concepts-abort-7986.test.ts@@ -43,12 +43,12 @@ async function conceptPages(): Promise<string[]> {return (await engine.executeRaw<{ slug: string }>("SELECT slug FROM pages WHERE slug LIKE 'concepts/%' AND deleted_at IS NULL ORDER BY slug")).map(r => r.slug);}-async function cycleAbortedOnSecondCall(transport: (opts: ChatOpts, job: AbortController) => Promise<ChatResult>) {- const job = new AbortController();+async function cycleAbortedOnSecondCall(transport: (opts: ChatOpts, job: AbortController) => Promise<ChatResult>,+ first: ChatResult = answer('A narrative about the first concept.'), job = new AbortController()) {let calls = 0;__setChatTransportForTests(async (opts) => {calls++;- if (calls === 1) return answer('A narrative about the first concept.');+ if (calls === 1) return first;return transport(opts, job);});try {@@ -85,4 +85,36 @@ describe('#7986 synthesize_concepts stops on the cycle abort signal', () => {expect(seen[0]?.aborted).toBe(true);expect(run.calls).toBe(2);}, 60_000);++ test('a job aborted while the first concept is being prepared for publication publishes nothing', async () => {+ const job = new AbortController();+ const getPage = PGLiteEngine.prototype.getPage;+ // A this-preserving wrapper: a transaction's engine inherits it and must still read on its own connection.+ engine.getPage = async function (this: PGLiteEngine, slug: string, opts?: Parameters<typeof getPage>[1]) {+ if (slug.startsWith('concepts/')) job.abort(new Error('job timed out'));+ return getPage.call(this, slug, opts);+ } as typeof engine.getPage;+ try {+ const run = await cycleAbortedOnSecondCall(async () => answer('never reached'), undefined, job);+ expect(run.pages).toEqual([]);+ expect(run.outcome).toMatchObject({ error: { name: 'AbortError' } });+ expect(run.calls).toBe(1);+ } finally {+ delete (engine as { getPage?: unknown }).getPage;+ }+ }, 60_000);++ test('an aborted run still records the spend and the failed attempts of the calls it finished', async () => {+ const clipped = { ...answer('A clipped narr'), stopReason: 'length' as const };+ const run = await cycleAbortedOnSecondCall(async (_opts, job) => {+ job.abort(new Error('job timed out'));+ return answer('A narrative the slow model finished after the job timed out.');+ }, clipped);+ expect(run.outcome).toMatchObject({ error: { name: 'AbortError' } });+ const [rollup] = await engine.executeRaw<{ cost_usd: number; round_completed_count: number }>(+ "SELECT cost_usd::float8 AS cost_usd, round_completed_count FROM extract_rollup_7d WHERE kind = 'concepts' AND source_id = 'default'");+ expect(rollup?.cost_usd).toBeGreaterThan(0);+ expect(rollup?.round_completed_count).toBe(0);+ expect(await engine.getConfig('dream.concepts.failed_attempts')).toContain('concepts/alpha');+ }, 60_000);});