Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Dream runs no longer write a fallback file into a read-only native checkout (#8935)
garrytan78e3741064c1
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f994846c0a..6e664c35b2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -24,6 +24,7 @@ identifiers and attribution are available in the pre-removal Git revision- **Failed repairs change nothing (#7960 item 1).** When `gbrain repair provenance --apply` fails in the database after writing the page file, it puts the file back and reverts its commit.- **Dream write-through (#8092).** After a refused commit, rewrites of existing atom and concept files and removals of retired atoms' files are retried on the next run, and a busy commit lock no longer freezes the worker's event loop.- **Read-only mirrors stay read-only (#7098).** On a brain without managed persistence, a source marked `gbrain sources mirror-readonly` is left alone by page write-through and delete, timeline entries, fact fences, `forget`, the extract cycle, `sync --full` and `brainstorm --save`; owner trust repairs list its pages `needs_review`.+- **Dream runs leave read-only mirrors alone too (#8935).** On a brain without managed persistence, the dream cycle no longer writes, rewrites or removes page files in a source marked `gbrain sources mirror-readonly` (or a clone gbrain owns), and no longer puts a fallback file for another source under `.sources/` inside such a checkout. The pages stay in the database, and the mirror's `git pull --ff-only` keeps working. Reported and first fixed by @javieraldape in #9212.- **An agent's sessions are not saved as you (#8080).** Dream synthesis and OpenClaw sub-agent compaction honor the agent speaker marker. When an agent continues a session already saved as yours, gbrain saves nothing from it and reports `speaker_mismatch`. `gbrain repair agent-session-facts --apply` reads the session corpus once per run and keeps in-page rows attributed correctly after a slow write.### The commit door@@ -53,7 +54,7 @@ Brainstorm message idea from @javieraldape (#7113, rewritten on master). Every c**Git is the parent, so the database now follows a commit that already landed.** On a source configured `canonical: git`, a write is committed to Git first and its database row second. When something failed between the two (an unexpected error after the Git commit, before the row committed), the write ended `failed` with `storage_error` and the reply said "nothing was saved", although Git held the change. For a `forget` that meant the fact stayed active until the next sync, no withdrawal was recorded, and the same claim could be remembered again as a new fact (GST-2517).-- The owner now replays such a write. The replay finds its change already committed and records it without a second commit: a `forget` turns the fact inactive and records its withdrawal, and a `remember` or `put_page` gets its one row. A shared fact forgotten by the owner and a member's own-page fact behave the same. A killed process in the same window was already recovered this way.+- The owner now replays such a write. The replay finds its change already committed and records it without a second commit. A `forget` turns the fact inactive and records its withdrawal, and a `remember` or `put_page` gets its one row. A shared fact forgotten by the owner and a member's own-page fact behave the same. A killed process in the same window was already recovered this way.- If the replay fails too, the request ends failed with the landed commit on its receipt (`landed_before_row`). A `remember` or `forget` reply then names that commit and says the database has not recorded it, instead of "nothing was saved". A `forget` submitted again with a new request_id records the withdrawal without a second commit.- `gbrain repair failed-writes` no longer says a refused `forget` "will be produced again by the next dream cycle": only callers make forgets, so it lists one as a write to resubmit.diff --git a/docs/architecture/key-files/core-cycle-patterns.md b/docs/architecture/key-files/core-cycle-patterns.mdindex 532fd495b8..ba45e36159 100644--- a/docs/architecture/key-files/core-cycle-patterns.md+++ b/docs/architecture/key-files/core-cycle-patterns.md@@ -21,7 +21,7 @@ Patterns, drift reports and the cycle modules listed after them.- `src/core/cycle/triage-rescue.ts` — THE dream triage gate (`passesTriageGate`) + the verified-segment rescue (`applyTriageRescue`). Rescue fires only for scores in `[floor, threshold)` (floor inclusive; at/above threshold is the plain gate's job) with content_type in the buried-signal allowlist (default mixed/reflection/idea/strategy/people — never routine/technical) and ≥ minSegments of the judge's own segments verifying as normalized transcript substrings (`normForGrounding`, ≥40 chars, deduped by normalized quote — repetition is not more evidence). $0 — no LLM calls; fabricated segments cannot fire it; works on cached verdicts (gate-time only). Fail-closed on every malformed shape (null score, missing/short/non-string segments), never throws. `minSegments: 0` is the kill switch (gate degenerates to the plain threshold). ONE-GATE RULE: `runTriagePass` (report construction — `worth`/`rescued`/telemetry/dry-run), the synthesize fan-out, and `dream retriage` (reconcile-queue cancels + `--audit-rejects` sampling) all read this predicate — a second hand-rolled `score >= threshold` check is how an operator sweep cancels exactly the jobs the rescue admitted. Config: `dream.triage.rescue_floor` / `rescue_min_segments` / `rescue_content_types`. Pinned by `test/cycle-triage-rescue.test.ts` + the rescue suites in `test/cycle-synthesize-triage.test.ts` / `test/dream-retriage.test.ts`. System One S7: a decide verdict (`model` starting with `DECIDE_VERDICT_PREFIX`) is gated against the S7 slot threshold (the optional fifth argument) and never enters the rescue band.-- `src/core/cycle/derived-write-through.ts` — opt-in markdown write-through for generated pages on an unmanaged brain (#5041). `cycle.extract_atoms.write_through` / `cycle.synthesize_concepts.write_through` (off by default; `true`/`1`/`yes`/`on`) make `derivedWriteThrough` return a hook (null when off, managed or dry run): `extract_atoms` writes each item's atoms after the completion flip and removes the files of atoms it retired; `synthesize_concepts` writes through `publishClassicConcept`, which also rewrites a concept that already has a file whatever the key says. Creating the hook writes up to 200 generated pages per run that have no file yet (`source_path` NULL; pending atoms excluded), so enabling the key backfills. `writeDerivedPageThrough` keeps a page database-only when its target is ignored by git or sits in a declared `storage.db_only` directory (`derivedFileTarget`), replaces an existing file only when `replaceableFile` (`facts/file-body-guard.ts`) finds it equal to the page or one of its recent page_versions rows, then calls `writePageThrough` bound to the bytes compared with `inlineOnly` (a generated page stored in segments is skipped `segmented_page`, since the batch rollback compares file strings) and returns its result (`written`, `skipped`, `error`) so `publishClassicConcept` can surface a failed write. #8073: every file write and retired-file unlink is uncommitted (`deferCommit`) and the batch (an item's atoms plus its retired atoms' files, the backfill, or one concept) commits once through the shared-checkout door `commitPathsLocked` (`git/commit-push.ts`) under the source filesystem lock, naming only the batch's paths that exist or are tracked; a directory that is not a Git checkout, or whose checkout lookup fails (`classifyGitCheckout` `unknown`), has nothing to commit. A batch waits for a busy commit lock without blocking the event loop (`acquireCommitLockAsync`, up to 30 s), then takes it synchronously with a 2 s cap. A refused commit is rolled back under the commit lock (2 s wait; still busy, nothing is restored and every file stays in the pending ledger for that writer's commit): it leaves a file whose change HEAD already holds (another writer's locked commit swept it from the ledger) as committed, restores each other file still holding the bytes the batch wrote (`WriteThroughResult.content`; prior bytes, or removed), clears the `source_path` the write bound, settles the restored paths out of the ledger (`settlePending`), and returns a typed `DerivedCommitFailure` (`{code, message, paths, kept}`); the hook then writes nothing more that run and the phases report `details.write_through_failed` with `warn` (`reportDerivedWriteThrough`), while `writeDerivedPageThrough` returns it as `error` so the concept is held. Each page whose change was rolled back or left unwritten is marked in `op_checkpoints` (op `derived-write-retry`, fingerprint `<source>:<slug>`, pruned after the usual 7 days), and the next hook's backfill selects marked pages, live (rewritten, an update of an existing file included) and retired (file removed), clearing each mark once its change commits. Each file written or removed is also recorded in the door's pending ledger (`notePending`) right after the write, so a crash before the commit leaves it for the next `commitPathsLocked` in that checkout to sweep (ledger paths inside the source root that still differ from HEAD; only paths gbrain wrote are recorded). The dream reverse-write (`writeDreamPageFile`, `removeDreamPageFile`, `reverseWriteDreamRefs`) commits the same way: one batch per `reverseWriteDreamRefs` call (committed in a `finally`, so an abort still commits what it wrote) and one per summary write or retired-file removal; a refused commit rolls the files back and returns or pushes a `write_through_failed` hold, which the phases report through their existing `file_holds`. A checkout carrying a canonical owner marker commits through `commitAndPush`'s named-paths-only path (`commitManagedPaths`), and paths git ignores are left uncommitted. Tests: `test/cycle/derived-write-through-commit.test.ts` (with the `derivedWriteSeams.beforeCommit` crash fault point) and `test/cycle/dream-reverse-write-commit.test.ts` (PGLite and Postgres). A retired atom's file is removed under the same filter and comparison, the unlink rechecking the bytes under the source filesystem lock. A file holding an edit the database lacks (or an unindexed file at a backfilled page's path) is left as it is and listed in the hook's `held`, which both phases report as `details.file_holds` (`file_database_drift`) with a `warn` status. Tests: `test/cycle/derived-write-through.test.ts`, `test/pathological/d10-derived-atom-file-overwrite.test.ts`, `test/pathological/d11-derived-retired-atom-unlink.test.ts`. #6982: `writeDreamPageFile` is the dream reverse-write (synthesize, patterns, the cycle summary): nothing with `sync.write_through` off, for a read-only mirror or an owned url clone (#8935: `sourceMirrorReadOnly`, before any path, so no brainDir fallback either; `removeDreamPageFile` keeps the file) or a `storage.db_only` slug; the path is `resolvePageWriteTarget`'s (recorded source_path), falling back to the caller's brainDir layout when the source has no root or its root is missing; the file is replaced only when `replaceableFile` (facts/file-body-guard.ts) finds nothing in it the database lacks and, once the replacement is staged, still holds the compared bytes right before the rename, else the result carries the `file_database_drift` hold. It writes directly (not `writePageThrough`/`atomicWriteFileSync`) because a checkout a subagent tool write claimed through the coordinator refuses those with `writer_coordinator_required`. `reportFileHolds` lists every phase's holds in `details.file_holds` and turns ok into warn.+- `src/core/cycle/derived-write-through.ts` — opt-in markdown write-through for generated pages on an unmanaged brain (#5041). `cycle.extract_atoms.write_through` / `cycle.synthesize_concepts.write_through` (off by default; `true`/`1`/`yes`/`on`) make `derivedWriteThrough` return a hook (null when off, managed or dry run): `extract_atoms` writes each item's atoms after the completion flip and removes the files of atoms it retired; `synthesize_concepts` writes through `publishClassicConcept`, which also rewrites a concept that already has a file whatever the key says. Creating the hook writes up to 200 generated pages per run that have no file yet (`source_path` NULL; pending atoms excluded), so enabling the key backfills. `writeDerivedPageThrough` keeps a page database-only when its target is ignored by git or sits in a declared `storage.db_only` directory (`derivedFileTarget`), replaces an existing file only when `replaceableFile` (`facts/file-body-guard.ts`) finds it equal to the page or one of its recent page_versions rows, then calls `writePageThrough` bound to the bytes compared with `inlineOnly` (a generated page stored in segments is skipped `segmented_page`, since the batch rollback compares file strings) and returns its result (`written`, `skipped`, `error`) so `publishClassicConcept` can surface a failed write. #8073: every file write and retired-file unlink is uncommitted (`deferCommit`) and the batch (an item's atoms plus its retired atoms' files, the backfill, or one concept) commits once through the shared-checkout door `commitPathsLocked` (`git/commit-push.ts`) under the source filesystem lock, naming only the batch's paths that exist or are tracked; a directory that is not a Git checkout, or whose checkout lookup fails (`classifyGitCheckout` `unknown`), has nothing to commit. A batch waits for a busy commit lock without blocking the event loop (`acquireCommitLockAsync`, up to 30 s), then takes it synchronously with a 2 s cap. A refused commit is rolled back under the commit lock (2 s wait; still busy, nothing is restored and every file stays in the pending ledger for that writer's commit): it leaves a file whose change HEAD already holds (another writer's locked commit swept it from the ledger) as committed, restores each other file still holding the bytes the batch wrote (`WriteThroughResult.content`; prior bytes, or removed), clears the `source_path` the write bound, settles the restored paths out of the ledger (`settlePending`), and returns a typed `DerivedCommitFailure` (`{code, message, paths, kept}`); the hook then writes nothing more that run and the phases report `details.write_through_failed` with `warn` (`reportDerivedWriteThrough`), while `writeDerivedPageThrough` returns it as `error` so the concept is held. Each page whose change was rolled back or left unwritten is marked in `op_checkpoints` (op `derived-write-retry`, fingerprint `<source>:<slug>`, pruned after the usual 7 days), and the next hook's backfill selects marked pages, live (rewritten, an update of an existing file included) and retired (file removed), clearing each mark once its change commits. Each file written or removed is also recorded in the door's pending ledger (`notePending`) right after the write, so a crash before the commit leaves it for the next `commitPathsLocked` in that checkout to sweep (ledger paths inside the source root that still differ from HEAD; only paths gbrain wrote are recorded). The dream reverse-write (`writeDreamPageFile`, `removeDreamPageFile`, `reverseWriteDreamRefs`) commits the same way: one batch per `reverseWriteDreamRefs` call (committed in a `finally`, so an abort still commits what it wrote) and one per summary write or retired-file removal; a refused commit rolls the files back and returns or pushes a `write_through_failed` hold, which the phases report through their existing `file_holds`. A checkout carrying a canonical owner marker commits through `commitAndPush`'s named-paths-only path (`commitManagedPaths`), and paths git ignores are left uncommitted. Tests: `test/cycle/derived-write-through-commit.test.ts` (with the `derivedWriteSeams.beforeCommit` crash fault point) and `test/cycle/dream-reverse-write-commit.test.ts` (PGLite and Postgres). A retired atom's file is removed under the same filter and comparison, the unlink rechecking the bytes under the source filesystem lock. A file holding an edit the database lacks (or an unindexed file at a backfilled page's path) is left as it is and listed in the hook's `held`, which both phases report as `details.file_holds` (`file_database_drift`) with a `warn` status. Tests: `test/cycle/derived-write-through.test.ts`, `test/pathological/d10-derived-atom-file-overwrite.test.ts`, `test/pathological/d11-derived-retired-atom-unlink.test.ts`. #6982: `writeDreamPageFile` is the dream reverse-write (synthesize, patterns, the cycle summary): nothing with `sync.write_through` off, for a read-only mirror or an owned url clone (#8935: `sourceMirrorReadOnly`, before any path, so no brainDir fallback either, and no `.sources/<id>` fallback inside a read-only native checkout; `removeDreamPageFile` keeps the file) or a `storage.db_only` slug; the path is `resolvePageWriteTarget`'s (recorded source_path), falling back to the caller's brainDir layout when the source has no root or its root is missing; the file is replaced only when `replaceableFile` (facts/file-body-guard.ts) finds nothing in it the database lacks and, once the replacement is staged, still holds the compared bytes right before the rename, else the result carries the `file_database_drift` hold. It writes directly (not `writePageThrough`/`atomicWriteFileSync`) because a checkout a subagent tool write claimed through the coordinator refuses those with `writer_coordinator_required`. `reportFileHolds` lists every phase's holds in `details.file_holds` and turns ok into warn.- `src/core/cycle/concept-publication.ts` — managed `synthesize_concepts` publication: `publishManagedConcept` publishes through `publishMaintenancePage` (a concept with a recorded `source_path` republishes to its file unless a `db_only` tier covers it; new concepts are database-only), and `composeConceptRepublication` keeps an existing concept's facts/takes fences, timeline, tags and other frontmatter, holding the concept (`concept_preservation_hold`) when a fence is duplicated, unbalanced, unparseable or below the timeline sentinel. `replaceConceptProvenanceLinks` banks the concept-provenance edges and retires the edges of atoms that left the group in one transaction (coordinated on a managed brain), only while the concept is at the revision its publication produced (GBRA-161 PR 5, C-4). On an unmanaged brain `publishClassicConcept` writes under the page lock the fence writers take: it re-reads the page, defers (`revision_conflict`) when the narrative changed since synthesis started, another writer owns it, or the page's canonical file holds anything the database lacks (`replaceableFile`: body, fences, timeline, frontmatter, tags, type or title equal to neither the page nor a recent page_versions row; formatting is ignored; this also covers an unindexed file at a new concept's path, and the message names the page with `file_database_drift`), and otherwise composes the new narrative with the database's latest fences, timeline, tags and frontmatter, so a take added during synthesis survives (`test/pathological/d12-concept-stale-fences.test.ts`, `d13-concept-occupied-path.test.ts`). The file write after `importPage` passes the bytes read under the lock as `expectedFileBytes`, so an edit landing during the import is kept and the concept deferred; a failed file write holds the concept (`concept_write_through_failed`) instead of reporting success (W9F item 3, `test/cycle/concept-publication-file-recheck.test.ts`). `stripFenceSections` is the shared narrative view. Tests: `test/cycle/synthesize-concepts-fences.test.ts`. The phase keeps the private-first, provenance, then promotion order; revision races are deferred and holds rep... (line truncated to 2000 chars)- `src/core/cycle/connector-atoms.ts` — the connector atom setting (#5856). `CONNECTOR_ATOM_PAGES_KEY` (`cycle.extract_atoms.connector_pages`, on when unset; `connectorAtomPagesEnabled` treats only false/0/off/no as opted out) gates `CONNECTOR_ATOM_PAGE_TYPES` (`email`, `meeting`) of sources whose `config.kind` is a connector kind (`CONNECTOR_SOURCE_KINDS`). `connectorAtomExclusionSql` returns the `pages p` predicate atom discovery and the backlog count append (empty unless opted out), so the routine cycle, `dream --drain` and autopilot auto-drain extract connector mail and calendar pages under the auto-drain budget until the user opts out. Other page types of a connector source and every checkout-backed page are unaffected. Tests: `test/connector-atom-pages.test.ts`, `test/autopilot-auto-drain-dispatch.test.ts`, and the `atom_drain_default_on` / `atom_drain_opted_out` cases of `test/helpers/managed-connector-job-contract.ts`, run by `test/managed-connector-job-contract.test.ts` and its Postgres twin `test/e2e/managed-connector-job-contract.test.ts`.diff --git a/src/core/cycle/derived-write-through.ts b/src/core/cycle/derived-write-through.tsindex cc14af064c..6735d1d483 100644--- a/src/core/cycle/derived-write-through.ts+++ b/src/core/cycle/derived-write-through.ts@@ -397,12 +397,13 @@ export async function derivedWriteThrough(/** What a dream reverse-write did with one page's file. `held` is its `file_database_drift` entry. */export type DreamPageFileWrite = { written: true; path: string } | { written: false; held?: string; skipped?: string; error?: string };-/** The file of a dream page: its record target (`resolvePageWriteTarget`), else the caller's brainDir layout; null when it has none. */+/** The file of a dream page: its record target (`resolvePageWriteTarget`), else the caller's brainDir layout, unless brainDir is the native source's read-only checkout (#8935). */async function dreamFileTarget(engine: BrainEngine, brainDir: string, slug: string, sourceId: string, nativeSourceId: string):Promise<{ filePath: string; root: string } | { skipped: string }> {const target = await resolvePageWriteTarget(engine, slug, sourceId);if (!target.ok && (!brainDir || (target.skipped !== 'no_repo_configured' && target.skipped !== 'repo_not_found'))) return { skipped: target.skipped };if (target.ok) return { filePath: target.filePath, root: target.writeRoot };+ if (sourceId !== nativeSourceId && await sourceMirrorReadOnly(engine, nativeSourceId)) return { skipped: 'read_only_checkout' };return { filePath: sourceId === nativeSourceId ? join(brainDir, `${slug}.md`) : join(brainDir, '.sources', sourceId, `${slug}.md`), root: brainDir };}diff --git a/test/cycle/dream-reverse-write-commit.test.ts b/test/cycle/dream-reverse-write-commit.test.tsindex bb546381e1..58c1b61513 100644--- a/test/cycle/dream-reverse-write-commit.test.ts+++ b/test/cycle/dream-reverse-write-commit.test.ts@@ -250,5 +250,21 @@ for (const { backend, databaseUrl } of pathologicalArms()) {expect((await b.engine.getPage(IDEA, { sourceId: MIRROR }))?.title).toBe('Alice Example mirror idea');});}, 180_000);++ test('#8935: a writable source with no checkout gets no brain directory fallback file when that directory is a read-only mirror', async () => {+ await withBrain(databaseUrl, 'git', async b => {+ await setMirror(b.engine, SOURCE, true);+ await b.engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1)', [MIRROR]);+ await importFromContent(b.engine, IDEA, page('Alice Example other idea', 'Kept in the database only.'), { noEmbed: true, sourceId: MIRROR });+ const head = b.git('rev-parse', 'HEAD');+ const held: string[] = [];+ const written = await reverseWriteDreamRefs(b.engine, b.brainDir, [{ slug: IDEA, source_id: MIRROR }],+ { phase: 'synthesize', nativeSourceId: SOURCE, render: (p, tags) => serializePageToMarkdown(p, tags), held });+ const removal = await removeDreamPageFile(b.engine, b.brainDir, IDEA, MIRROR, SOURCE);+ expect({ written, held, removed: removal.removed, fallback: existsSync(join(b.root, '.sources', MIRROR)), head: b.git('rev-parse', 'HEAD'), dirty: dirty(b) })+ .toEqual({ written: 0, held: [], removed: false, fallback: false, head, dirty: '' });+ expect((await b.engine.getPage(IDEA, { sourceId: MIRROR }))?.title).toBe('Alice Example other idea');+ });+ }, 180_000);});}