Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(cycle): #8073 dream reverse-write commits through the door per batch; note each derived write so a crash-left batch is swept
garrytan0cab1f4c5fdc
diff --git a/docs/architecture/key-files/core-cycle-patterns.md b/docs/architecture/key-files/core-cycle-patterns.mdindex b3f08133fb..2ffbe5913c 100644--- a/docs/architecture/key-files/core-cycle-patterns.md+++ b/docs/architecture/key-files/core-cycle-patterns.md@@ -21,7 +21,7 @@ Patterns, drift reports and the cycle modules listed after them.- `src/core/cycle/triage-rescue.ts` — THE dream triage gate (`passesTriageGate`) + the verified-segment rescue (`applyTriageRescue`). Rescue fires only for scores in `[floor, threshold)` (floor inclusive; at/above threshold is the plain gate's job) with content_type in the buried-signal allowlist (default mixed/reflection/idea/strategy/people — never routine/technical) and ≥ minSegments of the judge's own segments verifying as normalized transcript substrings (`normForGrounding`, ≥40 chars, deduped by normalized quote — repetition is not more evidence). $0 — no LLM calls; fabricated segments cannot fire it; works on cached verdicts (gate-time only). Fail-closed on every malformed shape (null score, missing/short/non-string segments), never throws. `minSegments: 0` is the kill switch (gate degenerates to the plain threshold). ONE-GATE RULE: `runTriagePass` (report construction — `worth`/`rescued`/telemetry/dry-run), the synthesize fan-out, and `dream retriage` (reconcile-queue cancels + `--audit-rejects` sampling) all read this predicate — a second hand-rolled `score >= threshold` check is how an operator sweep cancels exactly the jobs the rescue admitted. Config: `dream.triage.rescue_floor` / `rescue_min_segments` / `rescue_content_types`. Pinned by `test/cycle-triage-rescue.test.ts` + the rescue suites in `test/cycle-synthesize-triage.test.ts` / `test/dream-retriage.test.ts`. System One S7: a decide verdict (`model` starting with `DECIDE_VERDICT_PREFIX`) is gated against the S7 slot threshold (the optional fifth argument) and never enters the rescue band.-- `src/core/cycle/derived-write-through.ts` — opt-in markdown write-through for generated pages on an unmanaged brain (#5041). `cycle.extract_atoms.write_through` / `cycle.synthesize_concepts.write_through` (off by default; `true`/`1`/`yes`/`on`) make `derivedWriteThrough` return a hook (null when off, managed or dry run): `extract_atoms` writes each item's atoms after the completion flip and removes the files of atoms it retired; `synthesize_concepts` writes through `publishClassicConcept`, which also rewrites a concept that already has a file whatever the key says. Creating the hook writes up to 200 generated pages per run that have no file yet (`source_path` NULL; pending atoms excluded), so enabling the key backfills. `writeDerivedPageThrough` keeps a page database-only when its target is ignored by git or sits in a declared `storage.db_only` directory (`derivedFileTarget`), replaces an existing file only when `replaceableFile` (`facts/file-body-guard.ts`) finds it equal to the page or one of its recent page_versions rows, then calls `writePageThrough` bound to the bytes compared and returns its result (`written`, `skipped`, `error`) so `publishClassicConcept` can surface a failed write. #8073: every file write and retired-file unlink is uncommitted (`deferCommit`) and the batch (an item's atoms plus its retired atoms' files, the backfill, or one concept) commits once through the shared-checkout door `commitPathsLocked` (`git/commit-push.ts`) under the source filesystem lock, naming only the batch's paths that exist or are tracked; a directory that is not a Git checkout has nothing to commit. A refused commit restores each file still holding the batch's bytes (prior bytes, or removed), clears the `source_path` the write bound so the next backfill rewrites it, and returns a typed `DerivedCommitFailure` (`{code, message, paths, kept}`); the hook then writes nothing more that run and the phases report `details.write_through_failed` with `warn` (`reportDerivedWriteThrough`), while `writeDerivedPageThrough` returns it as `error` so the concept is held. Tests: `test/cycle/derived-write-through-commit.test.ts` (PGLite and Postgres). A retired atom's file is removed under the same filter and comparison, the unlink rechecking the bytes under the source filesystem lock. A file holding an edit the database lacks (or an unindexed file at a backfilled page's path) is left as it is and listed in the hook's `held`, which both phases report as `details.file_holds` (`file_database_drift`) with a `warn` status. Tests: `test/cycle/derived-write-through.test.ts`, `test/pathological/d10-derived-atom-file-overwrite.test.ts`, `test/pathological/d11-derived-retired-atom-unlink.test.ts`. #6982: `writeDreamPageFile` is the dream reverse-write (synthesize, patterns, the cycle summary): nothing with `sync.write_through` off, for an owned url clone or a `storage.db_only` slug; the path is `resolvePageWriteTarget`'s (recorded source_path), falling back to the caller's brainDir layout when the source has no root or its root is missing; the file is replaced only when `replaceableFile` (facts/file-body-guard.ts) finds nothing in it the database lacks and, once the replacement is staged, still holds the compared bytes right before the rename, else the result carries the `file_database_drift` hold. It writes directly (not `writePageThrough`/`atomicWriteFileSync`) because a checkout a subagent tool write claimed through the coordinator refuses those with `writer_coordinator_required`. `reportFileHolds` lists every phase's holds in `details.file_holds` and turns ok into warn.+- `src/core/cycle/derived-write-through.ts` — opt-in markdown write-through for generated pages on an unmanaged brain (#5041). `cycle.extract_atoms.write_through` / `cycle.synthesize_concepts.write_through` (off by default; `true`/`1`/`yes`/`on`) make `derivedWriteThrough` return a hook (null when off, managed or dry run): `extract_atoms` writes each item's atoms after the completion flip and removes the files of atoms it retired; `synthesize_concepts` writes through `publishClassicConcept`, which also rewrites a concept that already has a file whatever the key says. Creating the hook writes up to 200 generated pages per run that have no file yet (`source_path` NULL; pending atoms excluded), so enabling the key backfills. `writeDerivedPageThrough` keeps a page database-only when its target is ignored by git or sits in a declared `storage.db_only` directory (`derivedFileTarget`), replaces an existing file only when `replaceableFile` (`facts/file-body-guard.ts`) finds it equal to the page or one of its recent page_versions rows, then calls `writePageThrough` bound to the bytes compared and returns its result (`written`, `skipped`, `error`) so `publishClassicConcept` can surface a failed write. #8073: every file write and retired-file unlink is uncommitted (`deferCommit`) and the batch (an item's atoms plus its retired atoms' files, the backfill, or one concept) commits once through the shared-checkout door `commitPathsLocked` (`git/commit-push.ts`) under the source filesystem lock, naming only the batch's paths that exist or are tracked; a directory that is not a Git checkout has nothing to commit. A refused commit restores each file still holding the batch's bytes (prior bytes, or removed), clears the `source_path` the write bound so the next backfill rewrites it, and returns a typed `DerivedCommitFailure` (`{code, message, paths, kept}`); the hook then writes nothing more that run and the phases report `details.write_through_failed` with `warn` (`reportDerivedWriteThrough`), while `writeDerivedPageThrough` returns it as `error` so the concept is held. Each file written or removed is also recorded in the door's pending ledger (`notePending`) right after the write, so a crash before the commit leaves it for the next `commitPathsLocked` in that checkout to sweep (ledger paths inside the source root that still differ from HEAD; only paths gbrain wrote are recorded). The dream reverse-write (`writeDreamPageFile`, `removeDreamPageFile`, `reverseWriteDreamRefs`) commits the same way: one batch per `reverseWriteDreamRefs` call (committed in a `finally`, so an abort still commits what it wrote) and one per summary write or retired-file removal; a refused commit rolls the files back and returns or pushes a `write_through_failed` hold, which the phases report through their existing `file_holds`. A checkout carrying a canonical owner marker commits through `commitAndPush`'s named-paths-only path (`commitManagedPaths`), and paths git ignores are left uncommitted. Tests: `test/cycle/derived-write-through-commit.test.ts` (with the `derivedWriteSeams.beforeCommit` crash fault point) and `test/cycle/dream-reverse-write-commit.test.ts` (PGLite and Postgres). A retired atom's file is removed under the same filter and comparison, the unlink rechecking the bytes under the source filesystem lock. A file holding an edit the database lacks (or an unindexed file at a backfilled page's path) is left as it is and listed in the hook's `held`, which both phases report as `details.file_holds` (`file_database_drift`) with a `warn` status. Tests: `test/cycle/derived-write-through.test.ts`, `test/pathological/d10-derived-atom-file-overwrite.test.ts`, `test/pathological/d11-derived-retired-atom-unlink.test.ts`. #6982: `writeDreamPageFile` is the dream reverse-write (synthesize, patterns, the cycle summary): nothing with `sync.write_through` off, for an owned url clone or a `storage.db_only` slug; the path is `resolvePageWriteTarget`'s (recorded source_path), falling back to the caller's brainDir layout when the source has no root or its root is missing; the file is replaced only when `replaceableFile` (facts/file-body-guard.ts) finds nothing in it the database lacks and, once the replacement is staged, still holds the compared bytes right before the rename, else the result carries the `file_database_drift` hold. It writes directly (not `writePageThrough`/`atomicWriteFileSync`) because a checkout a subagent tool write claimed through the coordinator refuses those with `writer_coordinator_required`. `reportFileHolds` lists every phase's holds in `details.file_holds` and turns ok into warn.- `src/core/cycle/concept-publication.ts` — managed `synthesize_concepts` publication: `publishManagedConcept` publishes through `publishMaintenancePage` (a concept with a recorded `source_path` republishes to its file unless a `db_only` tier covers it; new concepts are database-only), and `composeConceptRepublication` keeps an existing concept's facts/takes fences, timeline, tags and other frontmatter, holding the concept (`concept_preservation_hold`) when a fence is duplicated, unbalanced, unparseable or below the timeline sentinel. `replaceConceptProvenanceLinks` banks the concept-provenance edges and retires the edges of atoms that left the group in one transaction (coordinated on a managed brain), only while the concept is at the revision its publication produced (GBRA-161 PR 5, C-4). On an unmanaged brain `publishClassicConcept` writes under the page lock the fence writers take: it re-reads the page, defers (`revision_conflict`) when the narrative changed since synthesis started, another writer owns it, or the page's canonical file holds anything the database lacks (`replaceableFile`: body, fences, timeline, frontmatter, tags, type or title equal to neither the page nor a recent page_versions row; formatting is ignored; this also covers an unindexed file at a new concept's path, and the message names the page with `file_database_drift`), and otherwise composes the new narrative with the database's latest fences, timeline, tags and frontmatter, so a take added during synthesis survives (`test/pathological/d12-concept-stale-fences.test.ts`, `d13-concept-occupied-path.test.ts`). The file write after `importPage` passes the bytes read under the lock as `expectedFileBytes`, so an edit landing during the import is kept and the concept deferred; a failed file write holds the concept (`concept_write_through_failed`) instead of reporting success (W9F item 3, `test/cycle/concept-publication-file-recheck.test.ts`). `stripFenceSections` is the shared narrative view. Tests: `test/cycle/synthesize-concepts-fences.test.ts`. The phase keeps the private-first, provenance, then promotion order; revision races are deferred and holds rep... (line truncated to 2000 chars)- `src/core/cycle/connector-atoms.ts` — the connector atom setting (#5856). `CONNECTOR_ATOM_PAGES_KEY` (`cycle.extract_atoms.connector_pages`, on when unset; `connectorAtomPagesEnabled` treats only false/0/off/no as opted out) gates `CONNECTOR_ATOM_PAGE_TYPES` (`email`, `meeting`) of sources whose `config.kind` is a connector kind (`CONNECTOR_SOURCE_KINDS`). `connectorAtomExclusionSql` returns the `pages p` predicate atom discovery and the backlog count append (empty unless opted out), so the routine cycle, `dream --drain` and autopilot auto-drain extract connector mail and calendar pages under the auto-drain budget until the user opts out. Other page types of a connector source and every checkout-backed page are unaffected. Tests: `test/connector-atom-pages.test.ts`, `test/autopilot-auto-drain-dispatch.test.ts`, and the `atom_drain_default_on` / `atom_drain_opted_out` cases of `test/helpers/managed-connector-job-contract.ts`, run by `test/managed-connector-job-contract.test.ts` and its Postgres twin `test/e2e/managed-connector-job-contract.test.ts`.diff --git a/docs/architecture/key-files/workspace-push-and-backup.md b/docs/architecture/key-files/workspace-push-and-backup.mdindex d0c6b208c9..b612acc80a 100644--- a/docs/architecture/key-files/workspace-push-and-backup.md+++ b/docs/architecture/key-files/workspace-push-and-backup.md@@ -12,5 +12,5 @@ Key files:## Shared-checkout git writer-- `src/core/git/commit-push.ts` + `src/core/git/commit-guard.ts` — writer safety in a shared checkout (#7381). `integrate` replaces the tip 3-way merge when any local commit is patch-equivalent to an upstream one (`rev-list --cherry-mark`): it replays the local commits onto the remote tip in memory (`merge-tree --merge-base` on git 2.40+, a 3-way `read-tree` otherwise), leaving out the equivalent ones and clean merges, so a copy the remote edited never conflicts and one it reverted is never re-applied; the merge message lists what it left out. `supersededPaths` resolves a content conflict to ours when the remote's blob is one the same path held earlier in ours' history (last 200 commits touching it) and a different blob replaced it there; the push reports them as `superseded_remote_paths`, and `integrate_conflict` lists only the rest (`detail.superseded` names them). `commitPaths` refuses a path outside the source roots (`configuredSourceRoots`, git config `gbrain.sourceRoot`, recorded by `sources harden`'s `recordSourceRoot`) and a file another writer staged under a named directory (`commit_prestaged_paths`), and restores the exact index entries (`update-index --index-info`) when git refuses the commit. Every commit gbrain makes carries `GBRAIN_COMMIT_VIA=<lock path>`. Every pathspec it passes goes through `literalPathspecs` (`git-env.ts`), never `git --literal-pathspecs`, so the commit's hooks keep Git's normal pathspec rules: a pre-commit guard that selects files by glob still sees and can refuse them (`git_unavailable`, paths unstaged). `commitPathsLocked` (write-through) never commits without the lock: a busy lock or refused commit notes the file in `brain-writethrough-pending.log` (git dir, O_APPEND lines), which the next locked write-through and `commit-push --pending` commit and `settlePending` prunes. A publish refuses local non-merge commits outside the roots (`commit_outside_source_root`; `--allow-outside-source-root` lifts both refusals, and `sources harden` passes it for its root scaffolding). With git config `gbrain.quarantine` (opt-in) a content conflict publishes the remote's version of each conflicted file plus every other local change, names the local commit under `refs/gbrain/quarantine/<time>-<sha>` and appends it to `brain-quarantine.log` (`readQuarantine` lists entries whose ref still exists); structural conflicts still refuse. `commit-guard.ts` installs the opt-in `prepare-commit-msg` guard (`gbrain git guard`), which admits only commits naming a held commit lock, passes merges and in-progress operations, and honors `GBRAIN_COMMIT_GUARD=off`. The helper's no-gbrain fallback (`renderCommitPushHelper`) marks children with the absolute lock path and releases the commit lock before pushing under `brain-publish.lock`. Tests: `test/git-writer-safety-7381.serial.test.ts`, `test/git-literal-pathspec-hooks-7759.test.ts`.+- `src/core/git/commit-push.ts` + `src/core/git/commit-guard.ts` — writer safety in a shared checkout (#7381). `integrate` replaces the tip 3-way merge when any local commit is patch-equivalent to an upstream one (`rev-list --cherry-mark`): it replays the local commits onto the remote tip in memory (`merge-tree --merge-base` on git 2.40+, a 3-way `read-tree` otherwise), leaving out the equivalent ones and clean merges, so a copy the remote edited never conflicts and one it reverted is never re-applied; the merge message lists what it left out. `supersededPaths` resolves a content conflict to ours when the remote's blob is one the same path held earlier in ours' history (last 200 commits touching it) and a different blob replaced it there; the push reports them as `superseded_remote_paths`, and `integrate_conflict` lists only the rest (`detail.superseded` names them). `commitPaths` refuses a path outside the source roots (`configuredSourceRoots`, git config `gbrain.sourceRoot`, recorded by `sources harden`'s `recordSourceRoot`) and a file another writer staged under a named directory (`commit_prestaged_paths`), and restores the exact index entries (`update-index --index-info`) when git refuses the commit. Every commit gbrain makes carries `GBRAIN_COMMIT_VIA=<lock path>`. Every pathspec it passes goes through `literalPathspecs` (`git-env.ts`), never `git --literal-pathspecs`, so the commit's hooks keep Git's normal pathspec rules: a pre-commit guard that selects files by glob still sees and can refuse them (`git_unavailable`, paths unstaged). `commitPathsLocked` (write-through) never commits without the lock: a busy lock or refused commit notes the file in `brain-writethrough-pending.log` (git dir, O_APPEND lines), which the next locked write-through and `commit-push --pending` commit and `settlePending` prunes. `notePending` is exported: the derived and dream write-through (`cycle/derived-write-through.ts`, #8073) records each file it writes before its batch commit, so a crash in between leaves the file for that sweep. A publish refuses local non-merge commits outside the roots (`commit_outside_source_root`; `--allow-outside-source-root` lifts both refusals, and `sources harden` passes it for its root scaffolding). With git config `gbrain.quarantine` (opt-in) a content conflict publishes the remote's version of each conflicted file plus every other local change, names the local commit under `refs/gbrain/quarantine/<time>-<sha>` and appends it to `brain-quarantine.log` (`readQuarantine` lists entries whose ref still exists); structural conflicts still refuse. `commit-guard.ts` installs the opt-in `prepare-commit-msg` guard (`gbrain git guard`), which admits only commits naming a held commit lock, passes merges and in-progress operations, and honors `GBRAIN_COMMIT_GUARD=off`. The helper's no-gbrain fallback (`renderCommitPushHelper`) marks children with the absolute lock path and releases the commit lock before pushing under `brain-publish.lock`. Tests: `test/git-writer-safety-7381.serial.test.ts`, `test/git-literal-pathspec-hooks-7759.test.ts`.- `src/commands/doctor/checks/git-writer.ts` — `checkout_lagging`, `dirty_age`, `commits_outside_source_root`, `quarantined_changes` (#7381, brain category, after `git_convergence`): over the unmanaged source checkouts with an upstream (`sourceRoots` from git-convergence, grouped by toplevel with each source's toplevel-relative root), never fetching, every git call bounded at 5 s. Lagging warns when `HEAD..@{u}` holds a commit over an hour old (fix: `gbrain git commit-push --pull --path <top>`, agent). Dirty age warns on pending write-through files older than 30 minutes (`pendingWriteThroughStillDirty`; fix: `commit-push --pending`, agent) and on files staged that long (ask_user: unstage or commit by name). Outside-root warns on unpushed non-merge commits that change files outside the DB source roots plus `gbrain.sourceRoot` (`localCommitsOutsideRoots`; ask_user fix with `--allow-outside-source-root`), and says when the checkout has no root recorded. Test: `test/doctor-git-writer-7381.test.ts`.diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.tsindex a32c69f07d..cef8624599 100644--- a/src/core/behavior-change-notice.ts+++ b/src/core/behavior-change-notice.ts@@ -1372,7 +1372,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;{ since: '0.60.514.0', id: 'storage_ignored_dirs_8015', text: 'When gbrain.yml lists a deprecated `supabase_only` next to canonical keys (which ignores it), a managed sync now holds only the missing-file pages under the ignored directories `db_only` does not already cover, and deletes other missing-file pages as usual; it used to hold every missing-file page of the source. `gbrain storage status` now names those directories, and the deprecation warning no longer points to a `gbrain doctor --fix` rename that does not exist. A `db_only` or `supabase_only` line whose inline value the parser refuses (a missing comma, a nested list, a bare scalar) now counts as unresolved even when another key resolves, so missing-file pages are held rather than deleted.' },{ since: '0.60.514.0', id: 'company_db_only_keep_8014', text: 'A company-brain source now keeps a live page under a declared `storage.db_only` directory when its file is not in the approved manifest, and keeps every such page while its gbrain.yml storage section is invalid, as other managed sources do; it used to delete them. The decision reads gbrain.yml at the approved commit, never the working tree.' },{ since: '0.60.514.0', id: 'redaction_marker_not_credential_7997', text: 'The transcript `labeled_credential` detector no longer reads an importer redaction marker (`login **<REDACTED:user-pattern>**`) as a typed user/password pair, so a correctly redacted page clears `gbrain transcripts audit-secrets`; a password written beside a marker is still found. The audit detector version moves to 3, so `gbrain doctor` asks for a fresh audit once.' },- { since: 'TBD', id: 'derived_write_through_commits_8073', text: 'With `cycle.extract_atoms.write_through` or `cycle.synthesize_concepts.write_through` on, the atom and concept files the dream cycle writes into a source that is a Git checkout are now committed: one local commit per batch (an item\'s atoms plus the files of the atoms it retired, the backfill, or one concept), naming only those files, through the checkout\'s commit lock (a hardened checkout\'s post-commit hook pushes it). They used to stay untracked or modified, which made the scheduled sync skip every run. A commit the checkout refuses rolls back the files it wrote, reports `write_through_failed` with a warn status, and the next run writes them again. Atom files left uncommitted by earlier versions are not touched; commit them with `gbrain git commit-push -m <msg> -- <atom directory>`.' },+ { since: 'TBD', id: 'derived_write_through_commits_8073', text: 'Files the dream cycle writes into a source that is a Git checkout are now committed: atom and concept files (with `cycle.extract_atoms.write_through` or `cycle.synthesize_concepts.write_through` on) and the reverse-written synthesis, patterns and cycle-summary pages and their removals. Each batch (an item\'s atoms plus the files of the atoms it retired, the backfill, one concept, or one phase\'s reverse-written pages) is one local commit naming only those files, made through the checkout\'s commit lock (a hardened checkout\'s post-commit hook pushes it; a checkout with a canonical owner marker gets a named-paths-only commit). They used to stay untracked or modified, which made the scheduled sync skip every run. A commit the checkout refuses rolls back the files it wrote and reports `write_through_failed` with a warn status; atoms and concepts are written again on the next run. Each written file is also recorded in the pending write-through ledger, so files a crash left uncommitted go into the next commit gbrain makes in that checkout. Files left uncommitted by earlier versions are not touched; commit them with `gbrain git commit-push -m <msg> -- <paths>`.' },];/** The newest disclosed change's release: the notice id moves only when a release adds rows. */diff --git a/src/core/cycle/derived-write-through.ts b/src/core/cycle/derived-write-through.tsindex aa55d2ad69..de1e42bafb 100644--- a/src/core/cycle/derived-write-through.ts+++ b/src/core/cycle/derived-write-through.ts@@ -37,9 +37,9 @@ import { isDbOnly, loadStorageConfig } from '../storage-config.ts';import { deletePageThrough, isWriteThroughDisabled, resolvePageWriteTarget, writePageThrough, type PageWriteTarget } from '../write-through.ts';import { urlCloneReadOnly } from '../url-clone.ts';import { gitChildEnv, literalPathspecs } from '../git-env.ts';-import { atomicWriteFileSync } from '../atomic-write.ts';import { classifyGitCheckout } from '../git-checkout.ts';-import { commitPathsLocked } from '../git/commit-push.ts';+import { checkoutRoot, commitAndPush, commitPathsLocked, notePending, scopePaths } from '../git/commit-push.ts';+import { managedFilesystemRootFor, refreshManagedFilesystemRoots } from '../persistence/filesystem-guard.ts';import { withSourceFilesystemLock } from '../minions/source-filesystem.ts';import { fileDatabaseDriftToken, replaceableFile } from '../facts/file-body-guard.ts';@@ -113,10 +113,24 @@ async function writeDerivedFile(engine: BrainEngine, slug: string, sourceId: strconst result = await writePageThrough(engine, slug, { sourceId, expectedFileBytes, deferCommit: true });if (result.error) console.error(`[write-through] ${slug}: ${result.error}`);if (!result.written || !result.path) return result;+ noteWritten(resolved.target.writeRoot, result.path);return { ...result, change: { slug, sourceId, filePath: result.path, writeRoot: resolved.target.writeRoot, prior: expectedFileBytes,after: readFileSync(result.path, 'utf-8'), bound: row?.source_path === null ? resolved.target.sourcePathToBind : null } };}+/**+ * Record a file this module just wrote or removed in the checkout's pending write-through ledger (`notePending`), so a+ * crash before its batch commits leaves it for the next locked commit there (`commitPathsLocked` sweeps the ledger's+ * paths that are inside the source root and still differ from HEAD). Only paths gbrain wrote are ever recorded.+ */+function noteWritten(root: string, filePath: string): void {+ try {+ if (classifyGitCheckout(root) !== 'git') return;+ const top = checkoutRoot(root);+ notePending(top, scopePaths(top, [filePath], top));+ } catch { /* best-effort: the batch commit right after is the primary path */ }+}+/*** Write one generated page's markdown file and commit it (#8073). Never* throws; the result says whether a file was written, why not, or the write@@ -149,6 +163,7 @@ async function deleteDerivedFile(engine: BrainEngine, slug: string, sourceId: stif (file.expectedFileBytes === null) return { removed: false, skipped: 'file_not_present' };const result = await deletePageThrough(engine, slug, { sourceId, target: resolved.target, expectedFileBytes: file.expectedFileBytes, deferCommit: true });if (!result.removed) return result;+ noteWritten(resolved.target.writeRoot, resolved.target.filePath);return { ...result, change: { slug, sourceId, filePath: resolved.target.filePath, writeRoot: resolved.target.writeRoot, prior: file.expectedFileBytes, after: null, bound: null } };}@@ -159,20 +174,27 @@ async function deleteDerivedFile(engine: BrainEngine, slug: string, sourceId: st* a Git checkout has nothing to commit: its files are the record. A refused commit rolls the batch back* (`rollBackDerived`) and comes back as a typed failure.*/-async function commitDerivedChanges(engine: BrainEngine, phase: keyof typeof DERIVED_WRITE_THROUGH_KEYS,- changes: readonly DerivedFileChange[]): Promise<DerivedCommitFailure[]> {+async function commitDerivedChanges(engine: BrainEngine, label: string, changes: readonly DerivedFileChange[]): Promise<DerivedCommitFailure[]> {const failures: DerivedCommitFailure[] = [];for (const root of new Set(changes.map(c => c.writeRoot))) {const batch = changes.filter(c => c.writeRoot === root);if (classifyGitCheckout(root) === 'not_git') continue;derivedWriteSeams.beforeCommit?.(root);try {- await withSourceFilesystemLock(engine, root, async () => {+ await refreshManagedFilesystemRoots(engine);+ const managed = managedFilesystemRootFor(root) !== null;+ const commit = async () => {const rels = batch.map(c => relative(root, c.filePath));- const tracked = new Set(execFileSync('git', ['-C', root, 'ls-files', '-z', '--', ...literalPathspecs(rels)], { env: gitChildEnv(), encoding: 'utf8' }).split('\0'));- const paths = batch.filter((c, i) => existsSync(c.filePath) || tracked.has(rels[i]!)).map(c => c.filePath);- if (paths.length) commitPathsLocked(root, paths, `gbrain: ${phase} write-through (${paths.length} generated file(s))`, { lockWaitMs: DERIVED_COMMIT_LOCK_WAIT_MS });- });+ const tracked = new Set(gitPaths(root, ['ls-files', '-z', '--', ...literalPathspecs(rels)]));+ const ignored = new Set(gitPaths(root, ['check-ignore', '-z', '--stdin', '--no-index'], rels.join('\0')));+ const paths = batch.filter((c, i) => !ignored.has(rels[i]!) && (existsSync(c.filePath) || tracked.has(rels[i]!))).map(c => c.filePath);+ if (!paths.length) return;+ const message = `gbrain: ${label} write-through (${paths.length} generated file(s))`;+ if (managed) commitAndPush({ repo: root, cwd: root, paths, message, lockWaitMs: DERIVED_COMMIT_LOCK_WAIT_MS });+ else commitPathsLocked(root, paths, message, { lockWaitMs: DERIVED_COMMIT_LOCK_WAIT_MS });+ };+ if (managed) await commit();+ else await withSourceFilesystemLock(engine, root, commit);} catch (error) {failures.push(await rollBackDerived(engine, root, batch, error));}@@ -180,6 +202,16 @@ async function commitDerivedChanges(engine: BrainEngine, phase: keyof typeof DERreturn failures;}+/** NUL-separated paths a git listing command prints from `root` (exit 1, as `check-ignore` gives for none, is an empty list). */+function gitPaths(root: string, args: string[], input?: string): string[] {+ try {+ return execFileSync('git', ['-C', root, ...args], { env: gitChildEnv(), encoding: 'utf8', ...(input === undefined ? {} : { input }) }).split('\0').filter(Boolean);+ } catch (error) {+ if ((error as { status?: number }).status === 1) return [];+ throw error;+ }+}+/*** Undo a batch whose commit was refused: each file still holding the bytes the batch wrote gets its prior bytes back (or is* removed when it had none), and the source_path the write bound is cleared so the next run's backfill writes the page@@ -193,7 +225,11 @@ async function rollBackDerived(engine: BrainEngine, root: string, batch: readonltry {if ((existsSync(change.filePath) ? readFileSync(change.filePath, 'utf-8') : null) !== change.after) throw new Error('file_changed');if (change.prior === null) rmSync(change.filePath, { force: true });- else atomicWriteFileSync(change.filePath, change.prior);+ else {+ mkdirSync(dirname(change.filePath), { recursive: true });+ writeFileSync(`${change.filePath}.${process.pid}.tmp`, change.prior, 'utf-8');+ renameSync(`${change.filePath}.${process.pid}.tmp`, change.filePath);+ }if (change.bound) {await engine.executeRaw('UPDATE pages SET source_path = NULL WHERE source_id = $1 AND slug = $2 AND source_path = $3 AND deleted_at IS NULL',[change.sourceId, change.slug, change.bound]);@@ -211,7 +247,7 @@ export function reportFileHolds(result: PhaseResult, held: readonly string[] | uif (!held?.length) return result;return { ...result, status: result.status === 'fail' ? 'fail' : 'warn',details: { ...result.details, file_holds: [...new Set([...((result.details.file_holds as string[] | undefined) ?? []), ...held])] },- summary: `${result.summary} (${held.length} ${noun} file(s) left as they are: file_database_drift)` };+ summary: `${result.summary} (${held.length} ${noun} file(s) left as they are: ${[...new Set(held.map(h => /^\S+ \((\w+)/.exec(h)?.[1] ?? 'file_database_drift'))].join(', ')})` };}/** `result` with the hook's held files (`reportFileHolds`) and refused commits reported: `details.write_through_failed` and a `warn` status unless it failed. */@@ -281,12 +317,22 @@ export async function derivedWriteThrough(/** What a dream reverse-write did with one page's file. `held` is its `file_database_drift` entry. */export type DreamPageFileWrite = { written: true; path: string } | { written: false; held?: string; skipped?: string; error?: string };+/** The file of a dream page: its record target (`resolvePageWriteTarget`), else the caller's brainDir layout; null when it has none. */+async function dreamFileTarget(engine: BrainEngine, brainDir: string, slug: string, sourceId: string, nativeSourceId: string):+ Promise<{ filePath: string; root: string } | { skipped: string }> {+ const target = await resolvePageWriteTarget(engine, slug, sourceId);+ if (!target.ok && (!brainDir || (target.skipped !== 'no_repo_configured' && target.skipped !== 'repo_not_found'))) return { skipped: target.skipped };+ if (target.ok) return { filePath: target.filePath, root: target.writeRoot };+ return { filePath: sourceId === nativeSourceId ? join(brainDir, `${slug}.md`) : join(brainDir, '.sources', sourceId, `${slug}.md`), root: brainDir };+}+/*** #6982: write the file of a page the dream cycle wrote in the database (a synthesis or patterns output, the cycle- * summary) without replacing an edit only the file holds. Nothing is written with `sync.write_through` off, for a clone- * gbrain owns, or for a declared `storage.db_only` slug: the database is the copy of record. The file is the page's- * file of record (`resolvePageWriteTarget`: recorded source_path, own-local_path roots); a source with no usable- * checkout (no root configured, or the root is missing) keeps the caller's brainDir layout (`<brainDir>/<slug>.md`,+ * summary) without replacing an edit only the file holds, then commit it (#8073: `commitDerivedChanges`; a refused+ * commit rolls the file back and returns its `write_through_failed` hold). Nothing is written with `sync.write_through`+ * off, for a clone gbrain owns, or for a declared `storage.db_only` slug: the database is the copy of record. The file is+ * the page's file of record (`resolvePageWriteTarget`: recorded source_path, own-local_path roots); a source with no+ * usable checkout (no root configured, or the root is missing) keeps the caller's brainDir layout (`<brainDir>/<slug>.md`,* foreign sources under `.sources/<id>/`). The file is replaced only when `replaceableFile` finds nothing in it the* database lacks, and is re-read after the replacement is staged, right before the rename, so an edit that lands in* between is kept as well.@@ -294,52 +340,66 @@ export type DreamPageFileWrite = { written: true; path: string } | { written: fa* This is the reverse-write's own direct write (a staged sibling renamed over the file), as before, not* `writePageThrough` or `atomicWriteFileSync`: once a subagent's tool write has gone through the coordinator, the* checkout carries an owner marker and both refuse `writer_coordinator_required`, which would leave the file behind the- * database for the next coordinated write to refuse as drift.+ * database for the next coordinated write to refuse as drift. Such a checkout commits through commit-push's named-paths+ * path instead of the classic door.*/export async function writeDreamPageFile(engine: BrainEngine, brainDir: string, slug: string, sourceId: string,- opts: { nativeSourceId: string; markdown: string }): Promise<DreamPageFileWrite> {- if (await isWriteThroughDisabled(engine)) return { written: false, skipped: 'disabled_by_config' };- if (await urlCloneReadOnly(engine, sourceId)) return { written: false, skipped: 'url_clone' };- const target = await resolvePageWriteTarget(engine, slug, sourceId);- if (!target.ok && (!brainDir || (target.skipped !== 'no_repo_configured' && target.skipped !== 'repo_not_found'))) return { written: false, skipped: target.skipped };+ opts: { nativeSourceId: string; markdown: string; label?: string }): Promise<DreamPageFileWrite> {+ const file = await writeDreamFile(engine, brainDir, slug, sourceId, opts);+ if (!file.change) return file.result;+ const [failure] = await commitDerivedChanges(engine, opts.label ?? 'dream', [file.change]);+ return failure ? { written: false, held: dreamCommitFailedToken(slug, failure) } : file.result;+}++/** The file write of `writeDreamPageFile`, uncommitted, with the change it made. */+async function writeDreamFile(engine: BrainEngine, brainDir: string, slug: string, sourceId: string,+ opts: { nativeSourceId: string; markdown: string }): Promise<{ result: DreamPageFileWrite; change?: DerivedFileChange }> {+ if (await isWriteThroughDisabled(engine)) return { result: { written: false, skipped: 'disabled_by_config' } };+ if (await urlCloneReadOnly(engine, sourceId)) return { result: { written: false, skipped: 'url_clone' } };+ const target = await dreamFileTarget(engine, brainDir, slug, sourceId, opts.nativeSourceId);+ if ('skipped' in target) return { result: { written: false, skipped: target.skipped } };try {- const storage = loadStorageConfig(target.ok ? target.writeRoot : brainDir);- if (storage && isDbOnly(slug, storage)) return { written: false, skipped: 'db_only' };+ const storage = loadStorageConfig(target.root);+ if (storage && isDbOnly(slug, storage)) return { result: { written: false, skipped: 'db_only' } };} catch { /* unreadable gbrain.yml: sync reports it */ }- const filePath = target.ok ? target.filePath- : sourceId === opts.nativeSourceId ? join(brainDir, `${slug}.md`) : join(brainDir, '.sources', sourceId, `${slug}.md`);+ const { filePath } = target;const check = await replaceableFile(engine, sourceId, slug, filePath);- if (!check.ok) return { written: false, held: dreamFileDriftToken(slug, sourceId, 'before') };+ if (!check.ok) return { result: { written: false, held: dreamFileDriftToken(slug, sourceId, 'before') } };mkdirSync(dirname(filePath), { recursive: true });const staged = `${filePath}.${process.pid}.tmp`;writeFileSync(staged, opts.markdown, 'utf-8');if ((existsSync(filePath) ? readFileSync(filePath, 'utf-8') : null) !== check.expectedFileBytes) {rmSync(staged, { force: true });- return { written: false, held: dreamFileDriftToken(slug, sourceId, 'during') };+ return { result: { written: false, held: dreamFileDriftToken(slug, sourceId, 'during') } };}renameSync(staged, filePath);- return { written: true, path: filePath };+ noteWritten(target.root, filePath);+ return { result: { written: true, path: filePath },+ change: { slug, sourceId, filePath, writeRoot: target.root, prior: check.expectedFileBytes, after: opts.markdown, bound: null } };}/*** GBRA-161 PR 4: remove the file of a dream page a generation retired, resolved the way `writeDreamPageFile` wrote it, only* when the retired page (or one of its versions) holds everything in it; re-read right before the unlink. A file holding- * an edit the database lacks stays, with its `file_database_drift` entry returned as `held`.+ * an edit the database lacks stays, with its `file_database_drift` entry returned as `held`. The removal commits+ * (#8073); a refused commit puts the file back and returns its `write_through_failed` hold.*/export async function removeDreamPageFile(engine: BrainEngine, brainDir: string, slug: string, sourceId: string,nativeSourceId: string): Promise<{ removed: boolean; held?: string }> {if (await isWriteThroughDisabled(engine) || await urlCloneReadOnly(engine, sourceId)) return { removed: false };- const target = await resolvePageWriteTarget(engine, slug, sourceId);- if (!target.ok && (!brainDir || (target.skipped !== 'no_repo_configured' && target.skipped !== 'repo_not_found'))) return { removed: false };- const filePath = target.ok ? target.filePath- : sourceId === nativeSourceId ? join(brainDir, `${slug}.md`) : join(brainDir, '.sources', sourceId, `${slug}.md`);+ const target = await dreamFileTarget(engine, brainDir, slug, sourceId, nativeSourceId);+ if ('skipped' in target) return { removed: false };+ const { filePath } = target;if (!existsSync(filePath)) return { removed: false };const check = await replaceableFile(engine, sourceId, slug, filePath);if (!check.ok || readFileSync(filePath, 'utf-8') !== check.expectedFileBytes) {return { removed: false, held: fileDatabaseDriftToken(slug, sourceId, 'before', "the retired dream page's file was not removed") };}rmSync(filePath, { force: true });- return { removed: true };+ noteWritten(target.root, filePath);+ const [failure] = await commitDerivedChanges(engine, 'dream retirement',+ [{ slug, sourceId, filePath, writeRoot: target.root, prior: check.expectedFileBytes, after: null, bound: null }]);+ return failure ? { removed: false, held: dreamCommitFailedToken(slug, failure) } : { removed: true };}/** #6982: the hold entry for a dream page whose file holds an edit the database lacks. */@@ -347,35 +407,53 @@ function dreamFileDriftToken(slug: string, sourceId: string, when: 'before' | 'dreturn fileDatabaseDriftToken(slug, sourceId, when, 'the dream output stays in the database only', 'the dream reverse-write');}+/** #8073: the hold entry for a dream page whose file commit the checkout refused (the file was rolled back). */+function dreamCommitFailedToken(slug: string, failure: DerivedCommitFailure): string {+ return `${slug} (write_through_failed: ${failure.code}: the checkout refused the commit of this page's file, so the file was rolled back `+ + `and the dream output stays in the database only; ${failure.kept.length ? `${failure.kept.length} file(s) changed meanwhile were left as they are; ` : ''}`+ + 'fix the checkout (git status there), then the next write of the page commits its file)';+}+/*** The dream reverse-write loop (synthesize, patterns, #6982): each ref's page rendered by `render` and written through- * `writeDreamPageFile`. A page whose file holds an edit the database lacks keeps that file and its `file_database_drift`- * entry is pushed to `held`; an abort stops before the next file; other per-page failures are logged and skipped.- * Returns the number of files written.+ * `writeDreamFile`, then the files written commit as one batch (#8073), also when an abort or an error stops the loop. A+ * page whose file holds an edit the database lacks keeps that file and its `file_database_drift` entry is pushed to+ * `held`; a refused commit rolls the batch back and pushes each page's `write_through_failed` entry; an abort stops+ * before the next file; other per-page failures are logged and skipped. Returns the number of files written and committed.*/export async function reverseWriteDreamRefs<R extends { slug: string; source_id: string }>(engine: BrainEngine, brainDir: string, refs: R[],opts: { phase: string; nativeSourceId: string; render: (page: Page, tags: string[], ref: R) => string; signal?: AbortSignal; held?: string[] }): Promise<number> {- let count = 0;- for (const ref of refs) {- const { slug, source_id } = ref;- throwIfAborted(opts.signal, `[dream] ${opts.phase} reverse-write`);- // v0.32.8 F6: validate source_id is filesystem-safe before any join().- validateSourceId(source_id);- const page = await engine.getPage(slug, { sourceId: source_id });- if (!page) continue;- const tags = await engine.getTags(slug, { sourceId: source_id });- // #4077: re-check after the row reads — an abort that lands during- // getPage/getTags must not reach this ref's file write.- throwIfAborted(opts.signal, `[dream] ${opts.phase} reverse-write`);- try {- const file = await writeDreamPageFile(engine, brainDir, slug, source_id, { nativeSourceId: opts.nativeSourceId, markdown: opts.render(page, tags, ref) });- if (file.written) count++;- else if (file.held) opts.held?.push(file.held);- else if (file.error) process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${file.error}\n`);- } catch (e) {- // Per-slug failures are non-fatal — phase continues.- process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${e instanceof Error ? e.message : String(e)}\n`);+ const changes: DerivedFileChange[] = [];+ let rolledBack = 0;+ try {+ for (const ref of refs) {+ const { slug, source_id } = ref;+ throwIfAborted(opts.signal, `[dream] ${opts.phase} reverse-write`);+ // v0.32.8 F6: validate source_id is filesystem-safe before any join().+ validateSourceId(source_id);+ const page = await engine.getPage(slug, { sourceId: source_id });+ if (!page) continue;+ const tags = await engine.getTags(slug, { sourceId: source_id });+ // #4077: re-check after the row reads — an abort that lands during+ // getPage/getTags must not reach this ref's file write.+ throwIfAborted(opts.signal, `[dream] ${opts.phase} reverse-write`);+ try {+ const file = await writeDreamFile(engine, brainDir, slug, source_id, { nativeSourceId: opts.nativeSourceId, markdown: opts.render(page, tags, ref) });+ if (file.change) changes.push(file.change);+ else if (!file.result.written && file.result.held) opts.held?.push(file.result.held);+ else if (!file.result.written && file.result.error) process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${file.result.error}\n`);+ } catch (e) {+ // Per-slug failures are non-fatal — phase continues.+ process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${e instanceof Error ? e.message : String(e)}\n`);+ }+ }+ } finally {+ for (const failure of await commitDerivedChanges(engine, opts.phase, changes)) {+ for (const change of changes.filter(c => failure.paths.includes(relative(c.writeRoot, c.filePath)))) {+ opts.held?.push(dreamCommitFailedToken(change.slug, failure));+ rolledBack++;+ }}}- return count;+ return changes.length - rolledBack;}diff --git a/src/core/git/commit-push.ts b/src/core/git/commit-push.tsindex 3db5da516e..1f14441991 100644--- a/src/core/git/commit-push.ts+++ b/src/core/git/commit-push.ts@@ -500,8 +500,12 @@ export function pendingWriteThroughStillDirty(repo: string): PendingWriteThroughreturn pending.filter(e => dirty.has(e.path));}-/** Appends without the lock (O_APPEND), so a writer that could not get it still records its file. */-function notePending(root: string, paths: string[]): void {+/**+ * Appends without the lock (O_APPEND), so a writer that could not get it still records its file. `root` is the checkout+ * root and `paths` are relative to it (`scopePaths`). The derived write-through also records each file right after writing+ * it (#8073), so a crash before its commit leaves the file for the next locked commit to sweep.+ */+export function notePending(root: string, paths: string[]): void {try { appendFileSync(absGitPath(root, PENDING_NAME), paths.map(p => `${new Date().toISOString()}\t${JSON.stringify(p)}\n`).join('')); } catch { /* best-effort */ }}diff --git a/test/cycle/dream-reverse-write-commit.test.ts b/test/cycle/dream-reverse-write-commit.test.tsindex ca5b7c7cf0..7a45bb50df 100644--- a/test/cycle/dream-reverse-write-commit.test.ts+++ b/test/cycle/dream-reverse-write-commit.test.ts@@ -11,7 +11,7 @@* (no local_path, no brain directory) gets no file; a directory that is not a Git checkout keeps its files.* Fails when: a reverse-write leaves its file uncommitted (the pre-fix path) or leaves a file behind after a refused* commit.- * Seams: none; the dream outputs are imported pages. PGLite always, Postgres when DATABASE_URL is set.+ * Seams: none; the dream outputs are imported pages, rendered as the database holds them (serializePageToMarkdown). PGLite always, Postgres when DATABASE_URL is set.*/import { describe, expect, test } from 'bun:test';import { execFileSync } from 'node:child_process';@@ -19,9 +19,9 @@ import { appendFileSync, chmodSync, existsSync, mkdirSync, mkdtempSync, readFileimport { tmpdir } from 'node:os';import { join } from 'node:path';import { removeDreamPageFile, reverseWriteDreamRefs, writeDreamPageFile } from '../../src/core/cycle/derived-write-through.ts';-import { renderPageToMarkdown } from '../../src/core/cycle/synthesize.ts';import type { BrainEngine } from '../../src/core/engine.ts';import { importFromContent } from '../../src/core/import-file.ts';+import { serializePageToMarkdown } from '../../src/core/markdown.ts';import { managedPersistenceEnabled } from '../../src/core/persistence/ownership.ts';import { _resetWriteThroughCacheForTest } from '../../src/core/write-through.ts';import { isolatedSharedSkillsEngine } from '../helpers/shared-skills-engine.ts';@@ -75,7 +75,7 @@ async function withBrain(databaseUrl: string | undefined, layout: Layout, run: (async reverseWrite(slugs) {const held: string[] = [];const written = await reverseWriteDreamRefs(engine, brainDir, slugs.map(slug => ({ slug, source_id: SOURCE })),- { phase: 'synthesize', nativeSourceId: SOURCE, render: renderPageToMarkdown, held });+ { phase: 'synthesize', nativeSourceId: SOURCE, render: (p, tags) => serializePageToMarkdown(p, tags), held });return { written, held };},});@@ -110,7 +110,7 @@ for (const { backend, databaseUrl } of pathologicalArms()) {test('the cycle summary file write commits, and a retired dream page\'s file removal commits', async () => {await withBrain(databaseUrl, 'git', async b => {- const summary = await writeDreamPageFile(b.engine, b.brainDir, SUMMARY, SOURCE, { nativeSourceId: SOURCE, markdown: renderPageToMarkdown((await b.engine.getPage(SUMMARY, { sourceId: SOURCE }))!, []) });+ const summary = await writeDreamPageFile(b.engine, b.brainDir, SUMMARY, SOURCE, { nativeSourceId: SOURCE, markdown: serializePageToMarkdown((await b.engine.getPage(SUMMARY, { sourceId: SOURCE }))!, []) });expect({ written: summary.written, paths: lastCommitPaths(b), dirty: dirty(b) }).toEqual({ written: true, paths: [`${SUMMARY}.md`], dirty: '' });await b.reverseWrite([IDEA]);const before = commitCount(b);diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txtindex 332b87964d..2ab7bb2a8b 100644--- a/test/postgres-unit-arms.txt+++ b/test/postgres-unit-arms.txt@@ -48,6 +48,7 @@ test/conversation-facts-outcome-marker-purge-7778.test.tstest/conversation-facts-oversized-sidecar.test.tstest/conversation-facts-purged-claim-7438.test.tstest/cycle/derived-write-through-commit.test.ts+test/cycle/dream-reverse-write-commit.test.tstest/cycle/synthesize-concepts-merge.test.tstest/db-lock-reconnect.test.tstest/decide/conflict-proposals-managed.test.ts