Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Make the Git effect commit by plumbing and key the scan exemption on the sha gbrain created

garrytan7aa391c52c7c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a22720e676..52a3c59b23 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,11 +18,11 @@ identifiers and attribution are available in the pre-removal Git revision
- `src/core/persistence/effect-git.ts`: `pushGitRoot` reads the remote branch tip with `git ls-remote`, scans `<tip>..<HEAD sha>` and pushes that sha only as a fast-forward of the tip, leased on it; a tip that is not an ancestor, or a remote that moves before the push lands (`stale info`), goes to the keep-checkout publish, which now runs with `--scan-secrets`. The in-process scan gets 10 s; a range it cannot finish goes to the keep-checkout publish. A remote without the branch gets HEAD's whole history scanned and the branch created with a lease that it does not exist; one too large to scan in process holds with `publish_scan_unreadable` naming `git ls-remote --heads`. A refusal from either path holds the effects as `publish_secret_found` with the user's fix.
- `src/core/git/commit-push.ts`: a `scanSecrets` publish scans `<base>..<candidate>` before every push and leases the push on that base, so each merge with a newly fetched remote tip is scanned again.
-- `src/core/git/range-secret-scan.ts`: `rangeSecretRefusal`, the one refusal both callers throw (`publish_scan_unreadable` for a scan that could not read the range, `publish_secret_found` otherwise; a gate-file change says the user pushes it). `scanRange` marks a scan that ran past its budget (`timed_out`) and scans a whole history when the base is empty. It skips commits the commit ledger records as page writes (`effect`, `door`, `hosted_repo`); `gbrain git commit-push` commits, adopted outside edits and commits the ledger no longer holds are still scanned, and an unreadable ledger means everything is scanned. A Git effect's commit is skipped only when its record is `verified`: the sha read under the writer lock right after the commit, its only parent the HEAD the commit started from, its paths exactly the effect's, each committed blob the bytes whose sha256 the effect recorded, and HEAD unchanged after those checks. A script commit or amend landing in between, or a page file changed after the effect checked it, fails one of them and stays scanned, and so do records written before this check existed. Its time budget is also checked while the diff is parsed.
+- `src/core/git/range-secret-scan.ts`: `rangeSecretRefusal`, the one refusal both callers throw (`publish_scan_unreadable` for a scan that could not read the range, `publish_secret_found` otherwise; a gate-file change says the user pushes it). `scanRange` marks a scan that ran past its budget (`timed_out`) and scans a whole history when the base is empty. It skips commits the commit ledger records as page writes (`effect`, `door`, `hosted_repo`); `gbrain git commit-push` commits, adopted outside edits and commits the ledger no longer holds are still scanned, and an unreadable ledger means everything is scanned. A Git effect now makes its commit by plumbing (`commit-tree` over its staged paths, then `update-ref HEAD <new> <old>`, which refuses when HEAD moved), so the ledger records a sha gbrain created before any ref names it. The scan skips an effect commit only by that sha, recorded `verified` when every staged file holds the bytes whose sha256 the effect recorded (the page walk passes the hash it checked too). An amend, reset or reword of it is a different commit and is scanned, a page file changed before it was staged leaves it unverified, and records written before this check existed are scanned. Its time budget is also checked while the diff is parsed.
- `src/commands/doctor/checks/git-convergence.ts`: the agent push fixes for a stranded managed checkout and after committing its modified pages run `gbrain git commit-push --push-only --keep-checkout --scan-secrets`, so a script's commit is scanned there too.
- `src/core/persistence/git-push-holds.ts`: `publish_secret_found` and `publish_scan_unreadable` are held push codes, so doctor, `gbrain sync status` and `retry-effects --parked-pushes` cover them, with an ask-the-user fix (the refused commits, or `git fsck`).
- Docs: `docs/guides/shared-checkout-git.md` (`publish_secret_found` and `publish_scan_unreadable` rows), `docs/guides/live-sync.md`.
-- Tests: `test/effect-git-secret-gate.test.ts` (an ahead checkout, a diverged one, a remote rewound past a pushed secret, a `.gitattributes` change and a remote without the branch whose history holds a secret each refuse with nothing pushed; a clean ahead checkout pushes, a clean history creates a missing branch, and a scan past the in-process budget still scans in the child; through the effect worker the page write's effect holds, sync status names the fix, and an allowlisted false positive releases it, parked or not; a page write whose body holds a credential-shaped value still backs up; a script commit or amend racing the effect commit, a page file changed after the effect checked it, and an unverified ledger record each hold with nothing pushed; a repaired unreadable commit releases its hold); `test/persistence-git-removal-push-7089.test.ts` and `test/persistence-git-subdir-root-7091.test.ts` (doctor's push fixes scan); `test/guarded-publish.test.ts` (a remote rewound past a pushed secret is not published again); `test/git-literal-pathspec-hooks-7759.test.ts` publishes its probe `.gitattributes` commit first, since the effect's push now refuses one; `test/cli-flag-equals.test.ts` lists `range-secret-scan.ts` as a git child argv reader.
+- Tests: `test/effect-git-secret-gate.test.ts` (an ahead checkout, a diverged one, a remote rewound past a pushed secret, a `.gitattributes` change and a remote without the branch whose history holds a secret each refuse with nothing pushed; a clean ahead checkout pushes, a clean history creates a missing branch, and a scan past the in-process budget still scans in the child; through the effect worker the page write's effect holds, sync status names the fix, and an allowlisted false positive releases it, parked or not; a page write whose body holds a credential-shaped value still backs up; a script commit, amend or reword racing the effect commit, a page file changed after the effect checked it, and an unverified ledger record each hold with nothing pushed; a source-scan walk's page with a credential-shaped body still backs up; a repaired unreadable commit releases its hold); `test/persistence-git-removal-push-7089.test.ts` and `test/persistence-git-subdir-root-7091.test.ts` (doctor's push fixes scan); `test/guarded-publish.test.ts` (a remote rewound past a pushed secret is not published again); `test/git-literal-pathspec-hooks-7759.test.ts` publishes its probe `.gitattributes` commit first, since the effect's push now refuses one; `test/cli-flag-equals.test.ts` lists `range-secret-scan.ts` as a git child argv reader.
## [0.60.620.0] - 2026-11-09
diff --git a/docs/architecture/key-files/core-persistence.md b/docs/architecture/key-files/core-persistence.md
index 89100dcca2..d20acf7333 100644
--- a/docs/architecture/key-files/core-persistence.md
+++ b/docs/architecture/key-files/core-persistence.md
@@ -36,7 +36,7 @@ Managed facts extraction (withdrawn-candidate skip and recovery, conversation fa
- `src/core/persistence/params.ts` — runtime-leaf mutation parameter and receipt schemas shared with the frozen verb registry. Its operation-contract import is type-only; runtime validators live in separate modules to avoid a registry initialization cycle. `preconditions.ts` validates UUID/revision/force syntax. `purge-params.ts` validates local-only purge shape/trust before admission and publication, keeping purge diagnostics out of generic precondition imports and unrelated command flag scans. `test/persistence-cold-import.test.ts` loads each validator/contract/registry entry in a fresh process and preserves schema and typed-error identity. `CAPTURE_EVENT_PARAMS` are full-surface-only.
- `src/core/persistence/write-wait.ts` — the CLI write-wait and exit contract (#5232). `resolveCliWriteWaitMs` (`--wait` seconds > `GBRAIN_WRITE_WAIT_MS` > file-plane `persistence.write_wait_ms` > 30 s; agents keep `AGENT_WRITE_WAIT_MS`); `writeExchangeBudget` (transport deadline = wait + 15 s admission headroom; an explicit `--timeout` shortens the requested owner wait instead); `replayWhilePending` (replays the identical request ID while an older owner, a remote server or a subagent tool's own bounded wait reports pending; the caller-supplied `sleep` decides how each pause ends, which is how subagent tool writes stop on abort); `writeErrorExitCode` keys on an admitted non-terminal receipt (`admittedPendingReceipt` in `types.ts`): pending → `PENDING_WRITE_EXIT_CODE` (0 with `--accept-pending`/`GBRAIN_ACCEPT_PENDING=1` when `acceptPendingCovers` holds: no `diagnostic.owner` stale-owner block, GBRA-248), anything else → 1. Pinned by `test/pending-write-exit.serial.test.ts` and `test/pending-write-exit-owner.test.ts`. `pollCommand` (#6255) renders the envelope's receipt `fix.argv` (`gbrain write-request --brain <id> -- <request_id>`), else pins that argv to the CLI routing; each argument is shell-quoted (`src/core/shell-quote.ts`, fix wave 13 P1.15).
- `src/core/persistence/page-edit.ts` — `edit_page` (#5616): splits the canonical serialization into editable text and protected takes/facts segments whose remote view matches get_page's sanitized `content` byte for byte; `applyPageEdits` applies ordered exact-once replacements inside editable text only (typed `edit_no_match`/`edit_ambiguous_match`/`edit_protected_span`/`edit_invalid`), so canonical fences never move; `editDiff` caps the caller-view diff at 8 KB; `boundedUnifiedDiff` keeps its LCS table under `EDIT_PAGE_DIFF_CELL_BUDGET` (4M cells) by diffing only the window between common leading/trailing lines, and a window still over budget becomes one coarse hunk flagged `diff_coarse` (pinned by `test/large-pages/edit-diff-bounded.test.ts`). Called at submission (`page-mutations.ts`, before admission; skipped for a segmented page) and again on the locked snapshot in `page-prepare.ts`, which gives edit_page put_page's revision-bound `editing` timeline semantics. Pinned by `test/edit-page.test.ts`. `within` and the segmented path: [large page writes](core-services-2-continued.md#large-page-writes).
-- `src/core/persistence/{effect-schema,effect-model,effect-journal,effect-recovery,effect-git,effects}.ts` — bounded postcommit outbox (migration 155). Canonical publication atomically queues Git and embedding work; withdrawals queue three versioned, bounded exact-page manifests with durable slug cursors. `effect-targets.ts` converts legacy source scans with complete bounded discovery keyed on the request's own forgotten fact (falling back to the whole source ledger only when that fact row is gone) while retaining each cursor; ambiguous or over-capacity legacy intent stays pending. Older workers must be quiesced before rollout. Independent mirror recovery reserves capacity before file effects, finishes forward without changing revision or receipt, and blocks only its worktree until reconciled. Git holds the native root lock, disables hooks, uses literal pathspecs and target-only commits, and pushes without pull/rebase. A claimed single-file Git effect brings up to 99 more ready single-file Git effects of its worktree (`claimCoalescedGitEffects`, same readiness, recovery and withdrawal-mirror ordering predicates, `SKIP LOCKED`); the runner validates each path under the worktree lock, makes one `commit --only` per group (`commitGitTargets`: one `ls-files`, `status`, `add` and `diff` for the group; untracked absent or ignored targets take the per-target refusal path) and pushes each root once at the end of the pass (`pushGitRoot`). A page file target's optional `commit` note (`GitCommitNote`, set only by trusted local preparers: a fence repair's `fenceRepairCommit`) is queued as `commit_subject`/`commit_line` in the Git effect's data and passed by the runner to `commitGitTargets`/`publishGitEffect`: a noted path committing alone takes its subject; a group keeps `gbrain: persist N canonical memory updates` and lists the noted paths' lines in the body, in path order; control characters are stripped, and targets without a note commit exactly as before (`test/fence-repair-receipts.test.ts`). A group yields to publications still queued or running for its worktree (one update requeues it with `publication_pending`, 250 ms apart) for at most 20 claims, so a serial producer publishes ahead and its Git work lands as one group, and a write admitted behind a backlog of full groups takes the worktree lock after the group already committing (wave 9; on master it waited out the backlog and its 5 s write wait). `claimCoalescedGitEffects` checks the worktree-wide conditions (owner, refresh fence, recovery blocks) once as an InitPlan, so the claim walks `persistence_effects_pending` in order and stops at its limit (351 → 25 ms per claim with 22k ready effects). A path failure fails only its effect; a push failure leaves the group queued with `git_push_unavailable`, and the next pass finds nothing to commit and pushes once. Every path listing compares relative to the source root, which may sit below the Git toplevel (#7091: status prefix stripped, `diff --relative`). That holds for a removal too (#7089): its retry finds the path untracked and absent (`target_absent`), and it joins the push while the root is ahead of its upstream (`gitRootAhead`, a local `rev-list @{upstream}..HEAD`, no upstream is not ahead), as does `publishGitEffect`, recording `target_absent` with the push result; on a root that is not ahead it completes without a push, so a down remote never parks it. A Git page walk that ends while its durable root is ahead pushes before completing (`test/persistence-git-removal-push-7089.test.ts` + Postgres wrapper). In a root without the durability hook a group runs no git command: it takes each root's lock once, guards each source incarnation once, validates every path as before and records the `durability_not_enabled` outcomes in one update (a 50k import's ~30k queued effects drain inside a minute of serve) (#5530; tests: `persistence-git-coalescing-5530.slow.test.ts` + Postgres wrapper). A Git effect for a declared db_only page completes as skipped (`reason: db_only`), keeping its gitignored cache file out of Git. A Git or withdrawal target parks after five consecutive execution failures (`data.target_failures` for `data.failing_target`, reset when the target advances or a different target fails; shutdown cancellation, contention, dependency waits (including recovery-capacity waits), transient database failures and raw errors `classifyItemFailure` calls run-scoped (a lost connection, a failed transaction, an abort, a full disk) do not count). A `links` or `facts-backstop` effect counts its failures the same way (`data.target_failures`, no target; run-scoped failures, including run-scoped refusal codes, never count, and a 57014 counts as `statement_timeout`): at five a links effect completes as skipped with the code (`outcome: { links: 'skipped', reason }`; the edges are derived and `gbrain extract --stale` recovers them) and a facts-backstop effect fails with it; a pass's counted failures settle when the pass ends under an `ItemRun` (`effects.derived_retry_budget`), and a pass where most of them failed counts none (each retries as `item_failures_systemic`) (GBRA-130 A-11, `test/pathological/a11-effect-retry-budget.test.ts`). An embedding scan (a withdrawal's exact targets or a source scan) parks the page whose provider call keeps failing once its per-page budget (`MAX_RATE_LIMIT_RETRIES`) is spent, with the failure's code, and moves on with a fresh budget; provider throttling (`isEmbedRetriableError`), configuration, budget and disabled errors and run-scoped failures stay effect-wide (A-13, `a13-embedding-target-park.test.ts`): a scan records it in `data.parked` and moves on, a single-target effect or a failure outside any target parks the whole effect, and an effect with parked targets finishes as `failed`/`targets_parked`, never committed. A mirror or Git scan never parks on a sync-skip metafile page (`RESOLVER.md` carries the managed durability block by design) or on a file whose bytes hold an uncoordinated local edit (`file_database_drift`): the mirror materializes the withdrawal in the database without touching the file, both scans record the page in `data.skipped` (at most 100) and advance, so the request's Git and embedding effects are no longer gated behind a parked mirror (#5396). A page stored in segments is mirrored and committed too, streamed (see the segmented withdrawal mirror entry in `core-persistence-request-indexes.md`). Both claim queries hold a request's Git and embedding effects only while its withdrawal mirror is queued or running; each Git or embedding attempt rereads its mirror (`mirrorGates`, every mirror not committed plus a committed one that recorded its publications): a mirror queued or running again (retry-effects after the claim) makes the attempt wait (`withdrawal_mirror_pending`); otherwise the mirror's recovery records each page whose file it rewrote (`data.published`) and its cursor is never proof: a Git page walk commits a page the mirror passed only when the file's raw bytes hash-equal the mirror rendering of the current snapshot (the serialization `mirrorPage` writes; this also covers a mirror from before publications were recorded and a published file later reverted by hand) and otherwise skips it (`withdrawal_not_mirrored`), a single-file Git effect needs the recorded publication, embedding behind a failed mirror needs the recorded publication, and behind a failed mirror every other page, including a single-file Git effect's page in a coalesced group, is parked with `withdrawal_mirror_failed`, so Git never commits a file that may still hold withdrawn text. When the mirror commits, the same transaction re-arms its request's Git and embedding targets parked `withdrawal_mirror_failed` (`rearmWithheldTargets`: a scan revisits them through `data.retry_slugs`; other parks stay), and each effect pass repeats it for the requests it touched, so no second retry-effects is needed. Lifecycle settlement (`topology-locks.ts`) waits on a queued mirror, not a failed one (A-12, `a12-withdrawal-mirror-gate.test.ts`). Migration 170 indexes parked effects for the `parked_effects` doctor check. `retryParkedEffects` (the `retry-effects` admin path) previews parked targets or authorizes one more attempt per target (`data.retry_slugs`, visited before the cursor); it also covers parked embedding targets, failed Git, withdrawal-mirror, links and facts-backstop effects without parked targets (one more attempt; a mirror that failed `source_changed` reruns from its cursor once the source is back) and an embedding effect that failed `source_changed`, so one call reruns a failed mirror and re-arms the pages it withheld; `parkedOnly` keeps the credential-change retry (`retryParkedPushes`) to parked Git and withdrawal effects; tests: `persistence-effect-parking.test.ts`. `repair embedding-effects` skips failed embedding effects with parked targets and zero-norm refusals. Old targets resolve under the same hash/confinement checks without changing frozen data; ambiguous absent spellings refuse after scoped staged/unstaged deletion checks. Embedding runs outside transactions/OS locks and installs with projection CAS. Receipt helpers return only public effect state/reason/push. Consumer shutdown aborts providers and awaits workers; receipt compaction is bounded. Tests: `persistence-effects.test.ts`, `persistence-git-publication.test.ts`, `persistence-publication-native.serial.test.ts`. Embedding completion checks active column/vector, embedded stamp, text hash, model and page signature; partial work sends only stale chunks and preserves valid siblings. A partial synopsis rebuild completes before title demotion. Provider timeouts use the gateway budget, five durable page claims/backoff, execution-token CAS after provider work and atomic vector/stamp/effect completion. Keyless work skips cleanly. `retry-effects` is an explicit one-shot failed-effect allowance; current vectors reconcile provider-free without changing canonical receipts. A git refusal on an index lock (git's own message names the lock, so linked worktrees resolve) is `git_index_locked` contention (retried, not counted toward parking) while the lock is younger than 10 minutes, and `git_index_stale` (counted, naming the lock) when older or future-dated. Only the writer, holding its writer lock, removes a lock `index-lock-reclaim.ts` proves abandoned (its header lists the proof), with a receipt doctor `parked_effects` reports, then retries the step and the effects that lock parked; otherwise `git_index_stale` carries an `ask_user` fix. Steps run with `GIT_OPTIONAL_LOCKS=0`, an index-scaled deadline (`gitStepTimeoutMs`) and a 10 s SIGTERM grace (#6932).
+- `src/core/persistence/{effect-schema,effect-model,effect-journal,effect-recovery,effect-git,effects}.ts` — bounded postcommit outbox (migration 155). Canonical publication atomically queues Git and embedding work; withdrawals queue three versioned, bounded exact-page manifests with durable slug cursors. `effect-targets.ts` converts legacy source scans with complete bounded discovery keyed on the request's own forgotten fact (falling back to the whole source ledger only when that fact row is gone) while retaining each cursor; ambiguous or over-capacity legacy intent stays pending. Older workers must be quiesced before rollout. Independent mirror recovery reserves capacity before file effects, finishes forward without changing revision or receipt, and blocks only its worktree until reconciled. Git holds the native root lock, disables hooks, uses literal pathspecs and target-only commits, and pushes without pull/rebase. A claimed single-file Git effect brings up to 99 more ready single-file Git effects of its worktree (`claimCoalescedGitEffects`, same readiness, recovery and withdrawal-mirror ordering predicates, `SKIP LOCKED`); the runner validates each path under the worktree lock, makes one `commit-tree` per group (`commitGitTargets`: one `ls-files`, `status`, `add` and `diff` for the group; untracked absent or ignored targets take the per-target refusal path) and pushes each root once at the end of the pass (`pushGitRoot`). A page file target's optional `commit` note (`GitCommitNote`, set only by trusted local preparers: a fence repair's `fenceRepairCommit`) is queued as `commit_subject`/`commit_line` in the Git effect's data and passed by the runner to `commitGitTargets`/`publishGitEffect`: a noted path committing alone takes its subject; a group keeps `gbrain: persist N canonical memory updates` and lists the noted paths' lines in the body, in path order; control characters are stripped, and targets without a note commit exactly as before (`test/fence-repair-receipts.test.ts`). A group yields to publications still queued or running for its worktree (one update requeues it with `publication_pending`, 250 ms apart) for at most 20 claims, so a serial producer publishes ahead and its Git work lands as one group, and a write admitted behind a backlog of full groups takes the worktree lock after the group already committing (wave 9; on master it waited out the backlog and its 5 s write wait). `claimCoalescedGitEffects` checks the worktree-wide conditions (owner, refresh fence, recovery blocks) once as an InitPlan, so the claim walks `persistence_effects_pending` in order and stops at its limit (351 → 25 ms per claim with 22k ready effects). A path failure fails only its effect; a push failure leaves the group queued with `git_push_unavailable`, and the next pass finds nothing to commit and pushes once. Every path listing compares relative to the source root, which may sit below the Git toplevel (#7091: status prefix stripped, `diff --relative`). That holds for a removal too (#7089): its retry finds the path untracked and absent (`target_absent`), and it joins the push while the root is ahead of its upstream (`gitRootAhead`, a local `rev-list @{upstream}..HEAD`, no upstream is not ahead), as does `publishGitEffect`, recording `target_absent` with the push result; on a root that is not ahead it completes without a push, so a down remote never parks it. A Git page walk that ends while its durable root is ahead pushes before completing (`test/persistence-git-removal-push-7089.test.ts` + Postgres wrapper). In a root without the durability hook a group runs no git command: it takes each root's lock once, guards each source incarnation once, validates every path as before and records the `durability_not_enabled` outcomes in one update (a 50k import's ~30k queued effects drain inside a minute of serve) (#5530; tests: `persistence-git-coalescing-5530.slow.test.ts` + Postgres wrapper). A Git effect for a declared db_only page completes as skipped (`reason: db_only`), keeping its gitignored cache file out of Git. A Git or withdrawal target parks after five consecutive execution failures (`data.target_failures` for `data.failing_target`, reset when the target advances or a different target fails; shutdown cancellation, contention, dependency waits (including recovery-capacity waits), transient database failures and raw errors `classifyItemFailure` calls run-scoped (a lost connection, a failed transaction, an abort, a full disk) do not count). A `links` or `facts-backstop` effect counts its failures the same way (`data.target_failures`, no target; run-scoped failures, including run-scoped refusal codes, never count, and a 57014 counts as `statement_timeout`): at five a links effect completes as skipped with the code (`outcome: { links: 'skipped', reason }`; the edges are derived and `gbrain extract --stale` recovers them) and a facts-backstop effect fails with it; a pass's counted failures settle when the pass ends under an `ItemRun` (`effects.derived_retry_budget`), and a pass where most of them failed counts none (each retries as `item_failures_systemic`) (GBRA-130 A-11, `test/pathological/a11-effect-retry-budget.test.ts`). An embedding scan (a withdrawal's exact targets or a source scan) parks the page whose provider call keeps failing once its per-page budget (`MAX_RATE_LIMIT_RETRIES`) is spent, with the failure's code, and moves on with a fresh budget; provider throttling (`isEmbedRetriableError`), configuration, budget and disabled errors and run-scoped failures stay effect-wide (A-13, `a13-embedding-target-park.test.ts`): a scan records it in `data.parked` and moves on, a single-target effect or a failure outside any target parks the whole effect, and an effect with parked targets finishes as `failed`/`targets_parked`, never committed. A mirror or Git scan never parks on a sync-skip metafile page (`RESOLVER.md` carries the managed durability block by design) or on a file whose bytes hold an uncoordinated local edit (`file_database_drift`): the mirror materializes the withdrawal in the database without touching the file, both scans record the page in `data.skipped` (at most 100) and advance, so the request's Git and embedding effects are no longer gated behind a parked mirror (#5396). A page stored in segments is mirrored and committed too, streamed (see the segmented withdrawal mirror entry in `core-persistence-request-indexes.md`). Both claim queries hold a request's Git and embedding effects only while its withdrawal mirror is queued or running; each Git or embedding attempt rereads its mirror (`mirrorGates`, every mirror not committed plus a committed one that recorded its publications): a mirror queued or running again (retry-effects after the claim) makes the attempt wait (`withdrawal_mirror_pending`); otherwise the mirror's recovery records each page whose file it rewrote (`data.published`) and its cursor is never proof: a Git page walk commits a page the mirror passed only when the file's raw bytes hash-equal the mirror rendering of the current snapshot (the serialization `mirrorPage` writes; this also covers a mirror from before publications were recorded and a published file later reverted by hand) and otherwise skips it (`withdrawal_not_mirrored`), a single-file Git effect needs the recorded publication, embedding behind a failed mirror needs the recorded publication, and behind a failed mirror every other page, including a single-file Git effect's page in a coalesced group, is parked with `withdrawal_mirror_failed`, so Git never commits a file that may still hold withdrawn text. When the mirror commits, the same transaction re-arms its request's Git and embedding targets parked `withdrawal_mirror_failed` (`rearmWithheldTargets`: a scan revisits them through `data.retry_slugs`; other parks stay), and each effect pass repeats it for the requests it touched, so no second retry-effects is needed. Lifecycle settlement (`topology-locks.ts`) waits on a queued mirror, not a failed one (A-12, `a12-withdrawal-mirror-gate.test.ts`). Migration 170 indexes parked effects for the `parked_effects` doctor check. `retryParkedEffects` (the `retry-effects` admin path) previews parked targets or authorizes one more attempt per target (`data.retry_slugs`, visited before the cursor); it also covers parked embedding targets, failed Git, withdrawal-mirror, links and facts-backstop effects without parked targets (one more attempt; a mirror that failed `source_changed` reruns from its cursor once the source is back) and an embedding effect that failed `source_changed`, so one call reruns a failed mirror and re-arms the pages it withheld; `parkedOnly` keeps the credential-change retry (`retryParkedPushes`) to parked Git and withdrawal effects; tests: `persistence-effect-parking.test.ts`. `repair embedding-effects` skips failed embedding effects with parked targets and zero-norm refusals. Old targets resolve under the same hash/confinement checks without changing frozen data; ambiguous absent spellings refuse after scoped staged/unstaged deletion checks. Embedding runs outside transactions/OS locks and installs with projection CAS. Receipt helpers return only public effect state/reason/push. Consumer shutdown aborts providers and awaits workers; receipt compaction is bounded. Tests: `persistence-effects.test.ts`, `persistence-git-publication.test.ts`, `persistence-publication-native.serial.test.ts`. Embedding completion checks active column/vector, embedded stamp, text hash, model and page signature; partial work sends only stale chunks and preserves valid siblings. A partial synopsis rebuild completes before title demotion. Provider timeouts use the gateway budget, five durable page claims/backoff, execution-token CAS after provider work and atomic vector/stamp/effect completion. Keyless work skips cleanly. `retry-effects` is an explicit one-shot failed-effect allowance; current vectors reconcile provider-free without changing canonical receipts. A git refusal on an index lock (git's own message names the lock, so linked worktrees resolve) is `git_index_locked` contention (retried, not counted toward parking) while the lock is younger than 10 minutes, and `git_index_stale` (counted, naming the lock) when older or future-dated. Only the writer, holding its writer lock, removes a lock `index-lock-reclaim.ts` proves abandoned (its header lists the proof), with a receipt doctor `parked_effects` reports, then retries the step and the effects that lock parked; otherwise `git_index_stale` carries an `ask_user` fix. Steps run with `GIT_OPTIONAL_LOCKS=0`, an index-scaled deadline (`gitStepTimeoutMs`) and a 10 s SIGTERM grace (#6932).
- `src/core/persistence/{ipc,local-client}.ts` — dedicated local persistence socket and engine-free CLI delegation. The narrow protocol carries a durable brain UUID, existing local writer registration, client source/cwd, and one request UUID; the production provider verifies current authority and dispatches through operations. Max frame 32 MiB (five-million-byte content even with JSON escaping), eight connections/in-flight dispatches, incomplete-frame timeout 15s. Lost responses preserve the original request ID with unknown submission state and never automatically replay or open another engine. Brain/mount routing precedes lock inspection. A lock holder that is not a serve and answers no owner socket (a concurrent CLI call) is not a resident owner: `maybeDelegateLocalOperation` returns unhandled and the engine path waits for the PGLite lock instead of answering `owner_unavailable`; a serve holder or an answering socket keeps every failure final. Owner-side `resolveSourceId(..., { skipLocalSignals: true })` prevents serve env/dotfiles from overriding client routing. For an op that owns a `source` param (`opOwnsSource`, `src/core/ops/contract.ts`), `params.source` stays on the wire as the op's own value (provenance) and routing comes from the ambient tiers, as on the direct and thin-client CLI routes (`test/cli-op-owned-source.test.ts`). The production provider dispatches the trusted `local_cli` principal's allow-listed skill-administration ops with the `'stdio'` locality marker, so the four `localOnly` ones (`get_skill_retention`, `prune_skill_revisions`, `retain_skill_revision`, `import_skill_proposal`) work while a serve owns the brain, as do `takes_remove` and `takes_rebuild` (`LOCAL_CLI_LOCAL_ONLY_OPS`) for the `local_cli` principal (write scope still checked; `gbrain takes remove` and `gbrain takes rebuild` defer their engine like the other takes verbs); other local calls keep no marker (`test/persistence-local-only-ops.test.ts`). `bindResolveIpcForServe` accepts the provider as its third argument and closes the dedicated listener with hook IPC. `residentPersistenceConfig` resolves a resident serve's brain the same way (`--brain`, `GBRAIN_BRAIN_ID`, `.gbrain-mount`, mount path), so the owner socket and the provider's consumer config follow a mounted brain instead of the host config (#5237); hook IPC stays keyed off the host config. Tests: `test/persistence-ipc.test.ts`, `test/persistence-local-client.test.ts`, `test/resolve-ipc-binding.test.ts`, `test/serve-mounted-owner.test.ts`. A credential-free, read-only `projection_status` request (like `capabilities`) returns the resident's projection backlog counts (`PersistenceProjectionStatus`, no identifiers) so doctor's `text_projection_readiness` can report it while the resident holds a PGLite datastore (#5401, `test/projection-resident-doctor.test.ts`).
diff --git a/docs/architecture/key-files/workspace-push-and-backup.md b/docs/architecture/key-files/workspace-push-and-backup.md
index 1f028aafe6..5e38e59e01 100644
--- a/docs/architecture/key-files/workspace-push-and-backup.md
+++ b/docs/architecture/key-files/workspace-push-and-backup.md
@@ -13,7 +13,7 @@ Key files:
## Shared-checkout git writer
- `src/core/git/commit-push.ts` + `src/core/git/commit-guard.ts` — writer safety in a shared checkout (#7381). `integrate` replaces the tip 3-way merge when any local commit is patch-equivalent to an upstream one (`rev-list --cherry-mark`): it replays the local commits onto the remote tip in memory (`merge-tree --merge-base` on git 2.40+, a 3-way `read-tree` otherwise), leaving out the equivalent ones and clean merges, so a copy the remote edited never conflicts and one it reverted is never re-applied; the merge message lists what it left out. `supersededPaths` resolves a content conflict to ours when the remote's blob is one the same path held earlier in ours' history (last 200 commits touching it) and a different blob replaced it there; the push reports them as `superseded_remote_paths`, and `integrate_conflict` lists only the rest (`detail.superseded` names them). `commitPaths` refuses a path outside the source roots (`configuredSourceRoots`, git config `gbrain.sourceRoot`, recorded by `sources harden`'s `recordSourceRoot`) and a file another writer staged under a named directory (`commit_prestaged_paths`), and restores the exact index entries (`update-index --index-info`) when git refuses the commit. Every commit gbrain makes carries `GBRAIN_COMMIT_VIA=<lock path>`. Every pathspec it passes goes through `literalPathspecs` (`git-env.ts`), never `git --literal-pathspecs`, so the commit's hooks keep Git's normal pathspec rules: a pre-commit guard that selects files by glob still sees and can refuse them (`git_unavailable`, paths unstaged). `commitPathsLocked` (write-through) never commits without the lock: a busy lock or refused commit notes the file in `brain-writethrough-pending.log` (git dir, O_APPEND lines), which the next locked write-through and `commit-push --pending` commit and `settlePending` prunes. `notePending` is exported: the derived and dream write-through (`cycle/derived-write-through.ts`, #8073) records each file it writes before its batch commit, so a crash in between leaves the file for that sweep. A publish refuses local non-merge commits outside the roots (`commit_outside_source_root`; `--allow-outside-source-root` lifts both refusals, and `sources harden` passes it for its root scaffolding). With git config `gbrain.quarantine` (opt-in) a content conflict publishes the remote's version of each conflicted file plus every other local change, names the local commit under `refs/gbrain/quarantine/<time>-<sha>` and appends it to `brain-quarantine.log` (`readQuarantine` lists entries whose ref still exists); structural conflicts still refuse. `commit-guard.ts` installs the opt-in `prepare-commit-msg` guard (`gbrain git guard`), which admits only commits naming a held commit lock, passes merges and in-progress operations, and honors `GBRAIN_COMMIT_GUARD=off`. The helper's no-gbrain fallback (`renderCommitPushHelper`) marks children with the absolute lock path and releases the commit lock before pushing under `brain-publish.lock`. The CLI (`src/commands/git.ts`) refuses `checkout_not_a_source` before staging when this brain home's managed-root registry records source checkouts and the resolved checkout is none of them and has no `gbrain.sourceRoot` (#8662); a registered checkout must contain the resolved one, or be a subdirectory its HEAD tracks, so a repository that only encloses one is refused, and a registry that cannot be listed refuses `writer_coordinator_required` (GST-1623). Tests: `test/git-writer-safety-7381.serial.test.ts`, `test/git-literal-pathspec-hooks-7759.test.ts`, `test/git-commit-push-not-a-source-8662.test.ts`.
-- `src/core/git/range-secret-scan.ts` — `scanRange(root, from, to)`, the secret gate for automatic publishes: `gbrain git commit-push --push-only --scan-secrets` (the child `gbrain sources refresh` runs to publish a diverged managed checkout) scans `trackingRef..target` under the publish lock, right before the push. Plumbing only (`rev-list`, `cat-file commit`, one streamed `diff-tree --stdin -p --root --cc`), so log, diff and attribute settings never change what is scanned. Commit messages, added lines and merge resolutions go through `scanText` with the `.gbrain-scan-allow` allowlist; a finding, added text over the scan cap or a change to `.gbrain-scan-allow`/`.gitattributes` refuses `publish_secret_found` (file, line, rule, commit, never the value), and a range it cannot read in full (git failure, unrecognised output, 300 s budget) refuses `publish_scan_unreadable`. Commits the commit ledger records as page writes are not scanned: door and hosted-repository commits, and Git effect commits recorded `verified` (`ownCommit` in `effect-git.ts`: the sha read under the writer lock right after the commit, its only parent the start HEAD, its paths exactly the effect's, each blob the bytes whose sha256 the effect recorded, HEAD unchanged after the checks); adopted records, records without `verified` and an unreadable ledger leave the commits scanned. Doctor `git_convergence`'s agent push fixes pass `--scan-secrets` too.
+- `src/core/git/range-secret-scan.ts` — `scanRange(root, from, to)`, the secret gate for automatic publishes: `gbrain git commit-push --push-only --scan-secrets` (the child `gbrain sources refresh` runs to publish a diverged managed checkout) scans `trackingRef..target` under the publish lock, right before the push. Plumbing only (`rev-list`, `cat-file commit`, one streamed `diff-tree --stdin -p --root --cc`), so log, diff and attribute settings never change what is scanned. Commit messages, added lines and merge resolutions go through `scanText` with the `.gbrain-scan-allow` allowlist; a finding, added text over the scan cap or a change to `.gbrain-scan-allow`/`.gitattributes` refuses `publish_secret_found` (file, line, rule, commit, never the value), and a range it cannot read in full (git failure, unrecognised output, 300 s budget) refuses `publish_scan_unreadable`. Commits the commit ledger records as page writes are not scanned: door and hosted-repository commits, and Git effect commits recorded `verified` (`effectCommit` in `effect-git.ts` builds the commit with `commit-tree` and moves HEAD with an `update-ref` compare-and-swap, so the recorded sha is gbrain's own; `verified` when each staged blob is the bytes whose sha256 the effect recorded); adopted records, records without `verified` and an unreadable ledger leave the commits scanned. Doctor `git_convergence`'s agent push fixes pass `--scan-secrets` too.
- `src/commands/doctor/checks/git-writer.ts` — `checkout_lagging`, `dirty_age`, `commits_outside_source_root`, `quarantined_changes` (#7381, brain category, after `git_convergence`): over the unmanaged source checkouts with an upstream (`sourceRoots` from git-convergence, grouped by toplevel with each source's toplevel-relative root), never fetching, every git call bounded at 5 s. Lagging warns when `HEAD..@{u}` holds a commit over an hour old (fix: `gbrain git commit-push --pull --path <top>`, agent). Dirty age warns on pending write-through files older than 30 minutes (`pendingWriteThroughStillDirty`; fix: `commit-push --pending`, agent) and on files staged that long (ask_user: unstage or commit by name). Outside-root warns on unpushed non-merge commits that change files outside the DB source roots plus `gbrain.sourceRoot` (`localCommitsOutsideRoots`; ask_user fix with `--allow-outside-source-root`), and says when the checkout has no root recorded. Test: `test/doctor-git-writer-7381.test.ts`.
- `src/core/git/commit-ledger.ts` — the commit ledger (PR B, docs/designs/REFRESH_WRITE_BARRIER.md part 3): `<git common dir>/gbrain/commit-ledger.jsonl`, one O_APPEND JSON line under 1 KiB per record, under its own flock `commit-ledger.lock` (start line, append, 4 MiB rotation to `commit-ledger.1.jsonl`). `recordCommit`/`recordCommitAsync` take the sha and writer kind (`commit_push`, `door`, `merge`, `replay`, `effect`, `sync_init`, `workspace_push`, `bootstrap`, `hosted_repo`); porcelain sites call `recordIntent` (head_before, `messageDigest` of the message) before `git commit`, plumbing sites record the `commit-tree` sha before the ref moves. A failed append never fails the commit (logged once); a directory outside git records nothing. `readLedger` reads both files; only a missing file reads as empty, any other read error throws (GST-1623). Tests: `test/commit-ledger.test.ts`.
- `src/commands/doctor/checks/foreign-writers.ts` — `foreign_writers` (PR B part 4): every registered source checkout on this host, managed worktrees included, deduplicated by git common dir. Reflogs (`logs/HEAD`, `logs/refs/heads/*`, `worktrees/*/logs/HEAD`, newest 5,000 entries each) are classified by `classifyReflogMessage`: porcelain creations by new sha, other ref moves by the `old..new` range within 10 minutes of the entry, merges/fast-forwards/resets/checkouts/branch/clone excluded. Unledgered commits fail (`plain`, `amend`, `rewrite`, `plumbing`; `guard_bypassed` when the enforce guard predates the commit); intent matches are `unconfirmed` (warn, never written back); entries before the ledger start are uncovered. Dirty page files come from `status --porcelain=v2` under the source roots minus queued/running git effects and the write-through pending ledger. GST-1623: a foreign commit carries `gbrain_trailer` when its trailer paragraph has a `gbrain-*:` line (named as possibly an older gbrain without the ledger; still fails); `partial_reasons` names what the scan skipped (a root inside a checkout whose `rev-parse` fails, roots past `MAX_ROOTS` 50, an unreadable ledger, a failed or timed-out status, given up to 30 s within doctor's budget, the budget itself), and a scan that read no repository but skipped one warns. Repo-less: `not_applicable`. Tests: `test/doctor-foreign-writers.test.ts`.
diff --git a/docs/architecture/topologies.md b/docs/architecture/topologies.md
index 3bbf18fbad..e1ab988424 100644
--- a/docs/architecture/topologies.md
+++ b/docs/architecture/topologies.md
@@ -830,7 +830,7 @@ elsewhere), and never during a pending recovery or transfer
or rebases the checkout.
<a id="files-that-are-not-pages"></a>**Files that are not pages.** The owner's
-Git effect commits only the page paths it wrote (`git commit --only`) and then
+Git effect commits only the page paths it wrote (a `commit-tree` of them) and then
pushes HEAD. Everything else in the checkout (`MEMORY.md`, daily notes,
`AGENTS.md`, scripts) is the operator's: commit it with
`gbrain git commit-push -m "<msg>" -- <paths>`, which commits only those paths
diff --git a/docs/designs/DATABASE_CANONICAL_MEMORY.md b/docs/designs/DATABASE_CANONICAL_MEMORY.md
index c9676e0560..f593743a95 100644
--- a/docs/designs/DATABASE_CANONICAL_MEMORY.md
+++ b/docs/designs/DATABASE_CANONICAL_MEMORY.md
@@ -97,7 +97,7 @@ import semantics have to change.
| `put_page`, `put_pages`, `capture`, `remember`/`forget`, `edit_page`, `append_page`, `delete_page`, takes ops | yes | yes (managed: before the row; classic: after) | managed: Git effect | managed: journal, revision preconditions, recovery record, worktree lock (`src/core/ops/pages.ts:358-370`); classic: per-source native lock, no check against an editor's bytes |
| Managed sync | yes | managed: frontmatter write-back in some paths (#7084) | no | frozen bytes plus page revision per entry; holds on disagreement |
| Classic sync | yes | full sync re-exports never-committed pages (`src/commands/sync/full.ts:518-535`) | pulls unless `--no-pull` | `gbrain-sync:<source>` database lock |
-| Managed Git effect | no | no | `git commit --only` of the paths it wrote, push HEAD, never pulls, parks after 5 failures (`src/core/persistence/effect-git.ts:298`, `effect-model.ts:43`) | owner's native worktree lock |
+| Managed Git effect | no | no | a `commit-tree` of the paths it wrote, push HEAD, never pulls, parks after 5 failures (`src/core/persistence/effect-git.ts:298`, `effect-model.ts:43`) | owner's native worktree lock |
| Connectors (Google, GitHub, chat transcripts) | yes | yes unless database-only (`src/core/google/sweep-shared.ts:150-195`, `src/core/github-source.ts:970-1035`) | no | managed: coordinator; classic: compare-before-replace guard, item held on mismatch |
| Facts fence and takes writers | yes | yes (classic: file first, then rows; `src/core/facts/fence-write.ts:378-635`, `src/core/takes-write.ts`) | no | page lock or filesystem lock |
| Dream/cycle phases (atoms, concepts, synthesize, patterns, lint fix, extract_facts, fence and content repair) | yes | yes (managed through maintenance requests; classic through derived write-through) | classic lint fix commits | 20 overwrite sites needed the compare-before-replace guard added by PR #7000 (#6982 class) |
diff --git a/docs/guides/shared-checkout-git.md b/docs/guides/shared-checkout-git.md
index 242919eb93..985ac01a71 100644
--- a/docs/guides/shared-checkout-git.md
+++ b/docs/guides/shared-checkout-git.md
@@ -185,7 +185,7 @@ Follow the error's `fix.next` as the
| `git_index_stale` | 1 | An `index.lock` older than 10 minutes blocks the checkout. Check that no git command is running there, tell the user, and remove the named lock file only with their agreement. |
| `git_operation_in_progress` | 1 | A rebase, merge, cherry-pick or revert is unfinished in the checkout. gbrain never aborts it for you. Follow [stuck rebase](#stuck-rebase). |
| `git_detached_head` | 1 | The checkout is on a detached HEAD or an unborn branch. Check out the branch the remote tracks (`git switch <branch>`), or make the first commit, then rerun. |
-| <a id="publish_secret_found"></a>`publish_secret_found` | 1 | Only with `--scan-secrets`, which every automatic publish uses: the one `gbrain sources refresh` runs on its own, the push of a managed worktree's Git effects and doctor `git_convergence`'s push fixes. A commit about to be pushed that gbrain did not make for a page write (its commit ledger records its own page-write commits, whose bodies are stored raw by design and are not scanned; a Git effect's commit counts only when it was checked, right after it was made, to have the effect's start HEAD as parent, exactly the effect's files and the bytes the effect wrote; one that adopted a file edited outside gbrain, or one older than the retained ledger, is scanned) adds a value that looks like a secret, in its message, an added line (even one a later commit removed) or a merge resolution, adds text too large to scan, or changes `.gbrain-scan-allow` or a `.gitattributes` (files that decide what the scan sees); nothing was pushed. The message names each file, line, rule and commit, never the value. Ask the user: remove the value from those unpushed commits, or add a false positive's fingerprint to `.gbrain-scan-allow`, then retry. A commit that changes `.gbrain-scan-allow` or a `.gitattributes` is never published automatically: the user reviews it and pushes it. Never publish them with a command that skips the scan. On a managed worktree its Git effects hold meanwhile (`gbrain sync status` and doctor list them; the page writes are safe in the database and in local commits); after the fix, `gbrain sources writer retry-effects <source> --parked-pushes` runs the ones that parked. |
+| <a id="publish_secret_found"></a>`publish_secret_found` | 1 | Only with `--scan-secrets`, which every automatic publish uses: the one `gbrain sources refresh` runs on its own, the push of a managed worktree's Git effects and doctor `git_convergence`'s push fixes. A commit about to be pushed that gbrain did not make for a page write (its commit ledger records its own page-write commits, whose bodies are stored raw by design and are not scanned; a Git effect's commit counts only by the sha gbrain created for it, when its files held the bytes the effect wrote, so an amended or reworded copy is scanned; one that adopted a file edited outside gbrain, or one older than the retained ledger, is scanned) adds a value that looks like a secret, in its message, an added line (even one a later commit removed) or a merge resolution, adds text too large to scan, or changes `.gbrain-scan-allow` or a `.gitattributes` (files that decide what the scan sees); nothing was pushed. The message names each file, line, rule and commit, never the value. Ask the user: remove the value from those unpushed commits, or add a false positive's fingerprint to `.gbrain-scan-allow`, then retry. A commit that changes `.gbrain-scan-allow` or a `.gitattributes` is never published automatically: the user reviews it and pushes it. Never publish them with a command that skips the scan. On a managed worktree its Git effects hold meanwhile (`gbrain sync status` and doctor list them; the page writes are safe in the database and in local commits); after the fix, `gbrain sources writer retry-effects <source> --parked-pushes` runs the ones that parked. |
| <a id="publish_scan_unreadable"></a>`publish_scan_unreadable` | 1 | Only with `--scan-secrets`: the scan could not read the commits about to be pushed in full (a missing or corrupt object, git output it does not recognise, or its 300 s budget), so it reported no secret and nothing was pushed. Run `git fsck` in the checkout, fix what it names, then retry. When the remote has no such branch, a managed worktree's Git effect scans the branch's whole history and creates it only when that scan passes; a history too large to scan in the owner process holds the effects with this code and names `git ls-remote --heads <remote>`: check the upstream, then point it at the right branch or review the history and push the branch yourself. Never publish with a command that skips the scan. On a managed worktree its Git effects hold meanwhile, as for `publish_secret_found`. |
| `integrate_conflict` | 1 | Local and remote commits changed the same file; nothing was merged and the checkout is unchanged. Stop, relay the question to the user, then [resolve](#conflicts). |
| `push_unconfirmed` | 4 | The commit is durable locally but no push destination confirmed it (network, credentials, branch protection, a remote that kept moving, or another push that held the push lock for the whole wait). Fix what the message names, then run `gbrain git commit-push --push-only`. Never force-push. |
diff --git a/src/core/persistence/effect-git.ts b/src/core/persistence/effect-git.ts
index 6615ab8efe..6ddb6536a4 100644
--- a/src/core/persistence/effect-git.ts
+++ b/src/core/persistence/effect-git.ts
@@ -276,7 +276,7 @@ export function commitMessageArgs(paragraphs: string[], dir: string | (() => str
}
/**
- * #5530: commit a group of single-file targets with one `commit --only`. The
+ * #5530: commit a group of single-file targets with one commit (`effectCommit`). The
* caller owns the native worktree lock and has probed durability. Each
* target's outcome (or its own error) is keyed by its requested relative
* path; a failing target never enters the commit. Nothing is pushed.
@@ -361,76 +361,86 @@ export async function commitGitTargets(root: string, relativePaths: string[], si
}
}
if (!changed.length) return;
- // --only keeps unrelated staged paths out of this commit. After a lost
+ // Only these paths are committed; unrelated staged paths stay staged. After a lost
// database acknowledgment the same HEAD/file state is an exact no-op.
const commitPaths = [...new Set(changed.map(c => c.path))];
const noted = new Map<string, GitCommitNote>();
for (const c of changed) { const note = notes?.get(c.requested); if (note && !noted.has(c.path)) noted.set(c.path, note); }
- let messageDir: string | undefined;
- let result, headBefore: string | null = null;
- try {
- const paragraphs = commitMessage(commitPaths, noted);
- const message = commitMessageArgs(paragraphs, () => {
- mkdirSync(persistenceHome(), { recursive: true, mode: 0o700 });
- return messageDir = mkdtempSync(join(persistenceHome(), 'commit-message-'));
- });
- const before = await git(session, ['rev-parse', '--verify', '--quiet', 'HEAD']);
- headBefore = before.code === 0 ? before.stdout.trim() : null;
- await recordIntentAsync(root, { writer: 'effect', head_before: headBefore, message: paragraphs.join('\n\n') });
- result = await git(session, [...(commitPaths.some(path => removed.has(path)) ? NO_FSCACHE : []), 'commit', '--only', ...message, '--', ...literalPathspecs(commitPaths)]);
- } finally { if (messageDir) rmSync(messageDir, { recursive: true, force: true }); }
- if (result.code === 0) {
- const wanted = new Map<string, string | null>();
- for (const c of changed) {
- const hash = expected?.get(c.requested);
- if (hash !== undefined && (!wanted.has(c.path) || wanted.get(c.path) === hash)) wanted.set(c.path, hash);
- else wanted.set(c.path, '');
- }
- const own = await ownCommit(session, root, prefix, headBefore, commitPaths, wanted);
- const trailers = [...noted.values()].flatMap(note => note.trailers ?? []);
- const supersedes = supersededBy(trailers), adopted = adoptedBy(trailers);
- if (own) await recordCommitAsync(root, { sha: own.sha, writer: 'effect', ...(own.verified ? { verified: true } : {}),
- ...(supersedes.length ? { supersedes } : {}), ...(adopted.length ? { adopted } : {}) });
+ const wanted = new Map<string, string | null>();
+ for (const c of changed) {
+ const hash = expected?.get(c.requested);
+ wanted.set(c.path, hash !== undefined && (!wanted.has(c.path) || wanted.get(c.path) === hash) ? hash : '');
}
- const outcome = result.code === 0 ? { git: 'committed' } : gitFailure('Cannot commit the canonical Git target.',
+ mkdirSync(persistenceHome(), { recursive: true, mode: 0o700 });
+ const scratch = mkdtempSync(join(persistenceHome(), 'effect-commit-'));
+ let created: string | null;
+ try { created = await effectCommit(session, root, scratch, commitPaths, wanted, commitMessage(commitPaths, noted).map(stripspace).filter(Boolean), noted); }
+ finally { rmSync(scratch, { recursive: true, force: true }); }
+ const outcome = created ? { git: 'committed' } : gitFailure('Cannot commit the canonical Git target.',
`git commit failed for ${commitPaths.length} file(s); a missing Git identity (user.name, user.email) in that checkout is one cause to check.`);
for (const c of changed) results.set(c.requested, outcome);
});
return results;
}
+/** Git's whitespace cleanup of a `-m` paragraph, as `git commit -m` applies it: trailing blanks, repeated and edge blank lines go. */
+export function stripspace(paragraph: string): string {
+ const lines: string[] = [];
+ for (const line of paragraph.split('\n').map(text => text.replace(/[ \t\r]+$/, ''))) if (line || lines.length && lines[lines.length - 1]) lines.push(line);
+ while (lines.length && !lines[lines.length - 1]) lines.pop();
+ return lines.join('\n');
+}
+
/**
- * #8896: the commit `commit --only` just made, read with rev-parse under the writer lock, and whether the secret scan may
- * trust it as a page write. Null (left out of the ledger) unless its only parent is `headBefore` and it changes exactly
- * `paths`: a script's commit or amend that landed in between differs in one of them. `verified` also needs, for every
- * path, the sha256 of the bytes the effect wrote (`wanted`; null for a removal, '' for none known) to match the file read
- * once now, the committed blob to have that read's object id, and HEAD to be unchanged after the checks. Any mismatch,
- * unknown expectation or git failure leaves the commit scanned.
+ * #8896: a Git effect's commit, made by plumbing so its sha is gbrain's own before any ref names it. The index entries
+ * `git add` just staged for `paths` are laid over HEAD's tree in a scratch index, `commit-tree` writes the commit, the
+ * ledger records its sha, and `update-ref HEAD <new> <old>` moves the branch only if HEAD is still the commit it was
+ * built on; a moved HEAD is read again and the commit rebuilt once. Null when git fails or HEAD moved twice. The
+ * secret scan skips an effect commit only by this recorded sha, so an amend, reset or reword is a different commit and
+ * is scanned. The record is `verified` only when every path's staged blob is the bytes the effect wrote (`wanted`: the
+ * sha256 the effect recorded, null for a removal, '' for none known): the file, read once, hashes to it and has the
+ * staged blob's id; an outside edit that reached the index first fails that, and the commit is scanned.
*/
-async function ownCommit(session: GitSession, root: string, prefix: string, headBefore: string | null, paths: string[],
- wanted: ReadonlyMap<string, string | null>): Promise<{ sha: string; verified: boolean } | null> {
- const head = async () => { const r = await git(session, ['rev-parse', '--verify', '--quiet', 'HEAD']); return r.code === 0 ? r.stdout.trim() : ''; };
- const sha = await head();
- if (!sha) return null;
- const parents = await git(session, ['rev-list', '--parents', '-n', '1', sha]);
- if (parents.code !== 0 || parents.stdout.trim().split(' ').slice(1).join(' ') !== (headBefore ?? '')) return null;
- const full = new Set(paths.map(path => prefix + path));
- const diff = await git(session, ['diff-tree', '--no-commit-id', '--name-only', '-r', '-z', '--no-renames', '--root', sha]);
- const touched = diff.code === 0 ? diff.stdout.split('\0').filter(Boolean) : null;
- if (!touched || touched.length !== full.size || !touched.every(path => full.has(path))) return null;
- if (paths.some(path => wanted.get(path) === '' || !wanted.has(path))) return { sha, verified: false };
- const format = await git(session, ['rev-parse', '--show-object-format']);
- const listed = await git(session, ['ls-tree', '-r', '-z', '--full-tree', sha, '--', ...literalPathspecs([...full])]);
- if (format.code !== 0 || listed.code !== 0) return { sha, verified: false };
- const blobs = new Map(listed.stdout.split('\0').filter(Boolean).map(entry => {
- const [mode, type, oid] = entry.slice(0, entry.indexOf('\t')).split(' ');
- return [entry.slice(entry.indexOf('\t') + 1), mode?.startsWith('100') && type === 'blob' ? oid : ''] as const;
+async function effectCommit(session: GitSession, root: string, scratch: string, paths: string[],
+ wanted: ReadonlyMap<string, string | null>, paragraphs: string[], noted: ReadonlyMap<string, GitCommitNote>): Promise<string | null> {
+ const layout = await git(session, ['rev-parse', '--show-toplevel', '--show-prefix', '--show-object-format']);
+ const listed = await git(session, ['ls-files', '-s', '-z', '--', ...literalPathspecs(paths)]);
+ if (layout.code !== 0 || listed.code !== 0) return null;
+ const [top, prefix, format] = layout.stdout.split('\n');
+ const staged = new Map(listed.stdout.split('\0').filter(Boolean).map(entry => {
+ const [mode, oid, stage] = entry.slice(0, entry.indexOf('\t')).split(' ');
+ return [entry.slice(entry.indexOf('\t') + 1), { mode: mode!, oid: oid!, stage: stage! }] as const;
}));
- for (const path of paths) {
- const hash = wanted.get(path)!, oid = blobs.get(prefix + path);
- if (hash === null ? oid !== undefined : !oid || fileDigests(join(root, path), format.stdout.trim()) !== `${hash} ${oid}`) return { sha, verified: false };
+ if ([...staged.values()].some(entry => entry.stage !== '0')) return null;
+ const verified = paths.every(path => {
+ const hash = wanted.get(path), entry = staged.get(path);
+ if (hash === undefined || hash === '') return false;
+ return hash === null ? !entry : !!entry && entry.mode.startsWith('100') && fileDigests(join(root, path), format!) === `${hash} ${entry.oid}`;
+ });
+ const trailers = [...noted.values()].flatMap(note => note.trailers ?? []);
+ const supersedes = supersededBy(trailers), adopted = adoptedBy(trailers);
+ const message = commitMessageArgs(paragraphs, scratch);
+ for (let attempt = 0; ; attempt++) {
+ const before = await git(session, ['rev-parse', '--verify', '--quiet', 'HEAD']);
+ const head = before.code === 0 ? before.stdout.trim() : null;
+ await recordIntentAsync(root, { writer: 'effect', head_before: head, message: paragraphs.join('\n\n') });
+ const index: GitSession = { ...session, root: top!, indexFile: join(scratch, `index-${attempt}`) };
+ const removals = paths.filter(path => !staged.has(path)).map(path => prefix + path);
+ const entries = paths.flatMap(path => { const entry = staged.get(path); return entry ? ['--cacheinfo', `${entry.mode},${entry.oid},${prefix}${path}`] : []; });
+ if ((await git(index, head ? ['read-tree', head] : ['read-tree', '--empty'])).code !== 0
+ || entries.length && (await git(index, ['update-index', '--add', ...entries])).code !== 0
+ || removals.length && (await git(index, ['update-index', '--force-remove', '--', ...removals])).code !== 0) return null;
+ const tree = await git(index, ['write-tree']);
+ if (tree.code !== 0) return null;
+ const commit = await git(session, ['commit-tree', tree.stdout.trim(), ...(head ? ['-p', head] : []), ...message]);
+ if (commit.code !== 0) return null;
+ const created = commit.stdout.trim();
+ await recordCommitAsync(root, { sha: created, writer: 'effect', ...(verified ? { verified: true } : {}),
+ ...(supersedes.length ? { supersedes } : {}), ...(adopted.length ? { adopted } : {}) });
+ const moved = await git(session, ['update-ref', '-m', `commit: ${paragraphs[0]!.split('\n')[0]}`, 'HEAD', created, head ?? '']);
+ if (moved.code === 0) return created;
+ if (attempt) return null;
}
- return { sha, verified: await head() === sha };
}
/** `<sha256> <git blob id>` of the file's bytes, read once; null when it cannot be read or changes size while read. */
diff --git a/src/core/persistence/effects.ts b/src/core/persistence/effects.ts
index e424f847c2..f36ef33585 100644
--- a/src/core/persistence/effects.ts
+++ b/src/core/persistence/effects.ts
@@ -396,12 +396,16 @@ async function publishGitPage(engine: BrainEngine, effect: PersistenceEffect, bi
// hash the mirror recorded writing at this revision, else the rendering measured now (a segmented page's from its
// segments, unreadPageFile, never held as one string).
const mirrored = attempt.gate?.files.get(snapshot.page.slug);
- if (verdict === 'evidence' && (snapshot.page.deleted_at || persistenceFileHash(path) !== (mirrored?.revision === snapshot.revision ? mirrored.hash
- : content === null ? (await unreadPageFile(engine, snapshot, keep)).hash : fileContentHash(content)))) {
+ const rendered = verdict !== 'evidence' || snapshot.page.deleted_at ? undefined : mirrored?.revision === snapshot.revision ? mirrored.hash
+ : content === null ? (await unreadPageFile(engine, snapshot, keep)).hash : fileContentHash(content);
+ if (verdict === 'evidence' && (snapshot.page.deleted_at || persistenceFileHash(path) !== rendered)) {
await skipTarget(engine, effect, snapshot, 'withdrawal_not_mirrored');
return;
}
- const result = await publishGitEffect(root, relative(root, path).split(sep).join('/'), opts.signal, hardened, await gitCommitNote(engine, effect), undefined, owner);
+ // #8896: the bytes this walk expects, so its commit is a page write the secret scan skips when the file holds them.
+ const expected = effect.data.approved_file?.slug === snapshot.page.slug ? effect.data.approved_file.expected_sha256
+ : rendered ?? (content === null ? undefined : fileContentHash(content));
+ const result = await publishGitEffect(root, relative(root, path).split(sep).join('/'), opts.signal, hardened, await gitCommitNote(engine, effect), undefined, owner, expected);
if (result.reason === 'durability_not_enabled') await completeEffect(engine, effect, result);
else await finishPage(engine, effect, snapshot, result);
}
@@ -909,7 +913,7 @@ async function runEffectPass(engine: BrainEngine, config: GBrainConfig, opts: Ef
//
// #5530: a claimed single-file git effect brings up to 99 more ready
// single-file git effects of its worktree. Under the worktree lock each path
- // is validated as before, the group makes one `commit --only`, and every
+ // is validated as before, the group makes one commit, and every
// root pushes once at the end of the pass. A path failure fails only its
// effect; a push failure leaves the whole group retryable (the next pass
// finds nothing to commit and pushes once). Nothing holds a lock or a
diff --git a/src/core/persistence/git-door.ts b/src/core/persistence/git-door.ts
index eaaf69ff09..8fdd19370f 100644
--- a/src/core/persistence/git-door.ts
+++ b/src/core/persistence/git-door.ts
@@ -44,7 +44,7 @@ import { join, relative, resolve, sep } from 'node:path';
import type { Notice } from '../agent-output.ts';
import { literalPathspecs } from '../git-env.ts';
import { OperationError, opError } from '../ops/contract.ts';
-import { commitMessage, commitMessageArgs, git, gitBytes, gitFailure, gitStepTimeoutMs, ignoredTargetError, reclaimAbandonedIndexLock, withSession, type GitSession } from './effect-git.ts';
+import { commitMessage, commitMessageArgs, git, gitBytes, gitFailure, gitStepTimeoutMs, ignoredTargetError, reclaimAbandonedIndexLock, stripspace, withSession, type GitSession } from './effect-git.ts';
import { adoptedBy, type GitCommitNote } from './effect-model.ts';
import { persistenceHome } from './identity.ts';
import { ignoreCandidates } from './ignore-rules.ts';
@@ -184,14 +184,6 @@ function rewriteTree(dir: Dir, oidBytes: number): Buffer {
return Buffer.concat(chunks);
}
-/** Git's whitespace cleanup of a `-m` paragraph, as `git commit -m` applies it: trailing blanks, repeated and edge blank lines go. */
-function stripspace(paragraph: string): string {
- const lines: string[] = [];
- for (const line of paragraph.split('\n').map(text => text.replace(/[ \t\r]+$/, ''))) if (line || lines.length && lines[lines.length - 1]) lines.push(line);
- while (lines.length && !lines[lines.length - 1]) lines.pop();
- return lines.join('\n');
-}
-
async function doorCommit(session: GitSession, root: string, files: DoorFile[], onIndexRestored?: (paths: string[]) => void): Promise<string> {
const symbolic = await git(session, ['symbolic-ref', '-q', 'HEAD']);
if (symbolic.code !== 0) throw opError('git_detached_head', `${root} is on a detached HEAD, so the strict source's write has no branch to commit to; nothing was published.`,
diff --git a/test/effect-git-secret-gate.test.ts b/test/effect-git-secret-gate.test.ts
index 91cda30781..3150654610 100644
--- a/test/effect-git-secret-gate.test.ts
+++ b/test/effect-git-secret-gate.test.ts
@@ -250,22 +250,26 @@ describe('#8896 pushGitRoot scans the commits it publishes', () => {
/**
* A PATH whose `git` runs `step` once (`{git}` in it is the real git), before or after the first call whose arguments
- * contain `match`. The real git runs with the original PATH, so a git that is itself a PATH-resolving shim never finds
+ * contain one of `match`. The real git runs with the original PATH, so a git that is itself a PATH-resolving shim never finds
* this wrapper again.
*/
-function racingGit(c: Checkout, match: string, when: 'before' | 'after', step: string): string {
+function racingGit(c: Checkout, match: string[], when: 'before' | 'after', step: string): string {
const path = process.env.PATH ?? '', real = `PATH='${path}' '${Bun.which('git')!}'`;
const bin = join(c.home, 'race-bin'), mark = join(c.home, 'raced');
const hook = `if [ ! -f '${mark}' ]; then touch '${mark}'; ${step.replaceAll('{git}', real)}; fi`;
+ const cases = match.map(m => `*" ${m} "*`).join('|');
mkdirSync(bin);
writeFileSync(join(bin, 'git'), ['#!/bin/sh',
- when === 'before' ? `case " $* " in *" ${match} "*) ${hook};; esac` : '',
+ when === 'before' ? `case " $* " in ${cases}) ${hook};; esac` : '',
`${real} "$@"`, 'code=$?',
- when === 'after' ? `case " $* " in *" ${match} "*) if [ $code -eq 0 ]; then ${hook}; fi;; esac` : '',
+ when === 'after' ? `case " $* " in ${cases}) if [ $code -eq 0 ]; then ${hook}; fi;; esac` : '',
'exit $code', ''].join('\n'), { mode: 0o755 });
return `${bin}:${path}`;
}
+/** The git call after which the Git effect's commit exists: the porcelain commit, or the plumbing commit's ref move. */
+const EFFECT_COMMIT = ['commit --only', 'update-ref -m commit:'];
+
interface ManagedWrite { engine: BrainEngine; sourceId: string; worktreeId: string; runEffects: () => Promise<void>; effect: () => Promise<{ state: string; error_code: string | null }> }
/** A managed source on the checkout with git durability on; `script` commits into the checkout before one page write whose Git effect is queued. */
@@ -364,7 +368,7 @@ describe('#8896 through the effect worker', () => {
test('a script commit that lands right after the effect commit is not ledgered as a page write, so its secret holds the push', async () => {
const c = checkout();
const before = remoteTip(c);
- const PATH = racingGit(c, 'commit --only', 'after', `mkdir -p '${c.root}/scripts-out'; printf 'Deploy token %s\\n' '${token}' > '${c.root}/scripts-out/race.md';`
+ const PATH = racingGit(c, EFFECT_COMMIT, 'after', `mkdir -p '${c.root}/scripts-out'; printf 'Deploy token %s\\n' '${token}' > '${c.root}/scripts-out/race.md';`
+ ` {git} -C '${c.root}' add scripts-out/race.md && {git} -C '${c.root}' -c core.hooksPath=/dev/null commit -q -m 'script races the effect'`);
await withEnv({ PATH }, () => managedWrite(c, () => {}, async w => {
await w.runEffects();
@@ -377,7 +381,7 @@ describe('#8896 through the effect worker', () => {
test('a script amending the effect commit (same parent) is not ledgered as a page write, so its secret holds the push', async () => {
const c = checkout();
const before = remoteTip(c);
- const PATH = racingGit(c, 'commit --only', 'after', `mkdir -p '${c.root}/scripts-out'; printf 'Deploy token %s\\n' '${token}' > '${c.root}/scripts-out/race.md';`
+ const PATH = racingGit(c, EFFECT_COMMIT, 'after', `mkdir -p '${c.root}/scripts-out'; printf 'Deploy token %s\\n' '${token}' > '${c.root}/scripts-out/race.md';`
+ ` {git} -C '${c.root}' add scripts-out/race.md && {git} -C '${c.root}' -c core.hooksPath=/dev/null commit -q --amend --no-edit`);
await withEnv({ PATH }, () => managedWrite(c, () => {}, async w => {
await w.runEffects();
@@ -388,10 +392,35 @@ describe('#8896 through the effect worker', () => {
}));
}, 180_000);
+ test('a script rewording the effect commit (same parent, paths and bytes) is not trusted, so its message is scanned', async () => {
+ const c = checkout();
+ const before = remoteTip(c);
+ const PATH = racingGit(c, EFFECT_COMMIT, 'after', `{git} -C '${c.root}' -c core.hooksPath=/dev/null commit -q --amend -m "Deploy token ${token}"`);
+ await withEnv({ PATH }, () => managedWrite(c, () => {}, async w => {
+ await w.runEffects();
+ expect(git(c.root, 'log', '-1', '--format=%B')).toContain(token);
+ expect(await w.effect()).toMatchObject({ error_code: 'publish_secret_found' });
+ expect(remoteTip(c)).toBe(before);
+ }));
+ }, 180_000);
+
+ test('a source-scan walk commits a page whose body holds a credential-shaped value as gbrain\'s own, so it still backs up', async () => {
+ const c = checkout();
+ await managedWrite(c, () => {}, async w => {
+ const snapshot = (await w.engine.readPageSnapshot('notes/acme-example', { sourceId: w.sourceId }))!;
+ writeFileSync(join(c.root, 'notes/acme-example.md'), serializePageToMarkdown(snapshot.page, snapshot.tags));
+ await w.engine.executeRaw(`UPDATE persistence_effects SET data = data || '{"source_scan": true}'::jsonb WHERE kind='git'`);
+ for (let i = 0; i < 4 && (await w.effect()).state !== 'committed'; i++) await w.runEffects();
+ expect(git(c.root, 'log', '-1', '--format=%s').trim()).not.toBe('Initial');
+ expect(await w.effect()).toMatchObject({ state: 'committed', error_code: null });
+ expect(git(c.remote, 'show', 'main:notes/acme-example.md')).toContain(token);
+ }, `Connection notes: ${token}`);
+ }, 180_000);
+
test('a page file changed after the effect checked its bytes is not ledgered as verified, so the changed bytes are scanned', async () => {
const c = checkout();
const before = remoteTip(c);
- const PATH = racingGit(c, 'add -A', 'before', `printf 'Deploy token %s\\n' '${token}' >> '${c.root}/notes/acme-example.md'`);
+ const PATH = racingGit(c, ['add -A'], 'before', `printf 'Deploy token %s\\n' '${token}' >> '${c.root}/notes/acme-example.md'`);
await withEnv({ PATH }, () => managedWrite(c, () => {}, async w => {
await w.runEffects();
expect(git(c.root, 'show', 'HEAD:notes/acme-example.md')).toContain(token);