Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Scan adopted outside edits and commits a script slips in after a page write
garrytanaa278e5d87a0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 69a47ddd38..972078d50d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,7 +18,7 @@ identifiers and attribution are available in the pre-removal Git revision- `src/core/persistence/effect-git.ts`: `pushGitRoot` reads the remote branch tip with `git ls-remote`, scans `<tip>..<HEAD sha>` and pushes that sha only as a fast-forward of the tip, leased on it; a tip that is not an ancestor, or a remote that moves before the push lands (`stale info`), goes to the keep-checkout publish, which now runs with `--scan-secrets`. The in-process scan gets 10 s; a range it cannot finish goes to the keep-checkout publish. A remote without the branch gets HEAD's whole history scanned and the branch created with a lease that it does not exist; one too large to scan in process holds with `publish_scan_unreadable` naming `git ls-remote --heads`. A refusal from either path holds the effects as `publish_secret_found` with the user's fix.- `src/core/git/commit-push.ts`: a `scanSecrets` publish scans `<base>..<candidate>` before every push and leases the push on that base, so each merge with a newly fetched remote tip is scanned again.-- `src/core/git/range-secret-scan.ts`: `rangeSecretRefusal`, the one refusal both callers throw (`publish_scan_unreadable` for a scan that could not read the range, `publish_secret_found` otherwise; a gate-file change says the user pushes it). `scanRange` marks a scan that ran past its budget (`timed_out`) and scans a whole history when the base is empty. It skips commits the commit ledger records as page writes (`effect`, `door`, `hosted_repo`); `gbrain git commit-push` commits are still scanned. Its time budget is also checked while the diff is parsed.+- `src/core/git/range-secret-scan.ts`: `rangeSecretRefusal`, the one refusal both callers throw (`publish_scan_unreadable` for a scan that could not read the range, `publish_secret_found` otherwise; a gate-file change says the user pushes it). `scanRange` marks a scan that ran past its budget (`timed_out`) and scans a whole history when the base is empty. It skips commits the commit ledger records as page writes (`effect`, `door`, `hosted_repo`); `gbrain git commit-push` commits, adopted outside edits and commits the ledger no longer holds are still scanned, and an unreadable ledger means everything is scanned. A Git effect records its commit in the ledger only when that commit's parent is the HEAD it started from, so a script commit landing right after it is never recorded as gbrain's. Its time budget is also checked while the diff is parsed.- `src/core/persistence/git-push-holds.ts`: `publish_secret_found` and `publish_scan_unreadable` are held push codes, so doctor, `gbrain sync status` and `retry-effects --parked-pushes` cover them, with an ask-the-user fix (the refused commits, or `git fsck`).- Docs: `docs/guides/shared-checkout-git.md` (`publish_secret_found` and `publish_scan_unreadable` rows), `docs/guides/live-sync.md`.- Tests: `test/effect-git-secret-gate.test.ts` (an ahead checkout, a diverged one, a remote rewound past a pushed secret, a `.gitattributes` change and a remote without the branch whose history holds a secret each refuse with nothing pushed; a clean ahead checkout pushes, a clean history creates a missing branch, and a scan past the in-process budget still scans in the child; through the effect worker the page write's effect holds, sync status names the fix, and an allowlisted false positive releases it, parked or not; a page write whose body holds a credential-shaped value still backs up; a repaired unreadable commit releases its hold); `test/guarded-publish.test.ts` (a remote rewound past a pushed secret is not published again); `test/git-literal-pathspec-hooks-7759.test.ts` publishes its probe `.gitattributes` commit first, since the effect's push now refuses one; `test/cli-flag-equals.test.ts` lists `range-secret-scan.ts` as a git child argv reader.diff --git a/docs/guides/shared-checkout-git.md b/docs/guides/shared-checkout-git.mdindex 3e5525b7b9..4f44c96383 100644--- a/docs/guides/shared-checkout-git.md+++ b/docs/guides/shared-checkout-git.md@@ -185,7 +185,7 @@ Follow the error's `fix.next` as the| `git_index_stale` | 1 | An `index.lock` older than 10 minutes blocks the checkout. Check that no git command is running there, tell the user, and remove the named lock file only with their agreement. || `git_operation_in_progress` | 1 | A rebase, merge, cherry-pick or revert is unfinished in the checkout. gbrain never aborts it for you. Follow [stuck rebase](#stuck-rebase). || `git_detached_head` | 1 | The checkout is on a detached HEAD or an unborn branch. Check out the branch the remote tracks (`git switch <branch>`), or make the first commit, then rerun. |-| <a id="publish_secret_found"></a>`publish_secret_found` | 1 | Only with `--scan-secrets`, which every automatic publish uses: the one `gbrain sources refresh` runs on its own and the push of a managed worktree's Git effects. A commit about to be pushed that gbrain did not make for a page write (its commit ledger records its own page-write commits, whose bodies are stored raw by design and are not scanned) adds a value that looks like a secret, in its message, an added line (even one a later commit removed) or a merge resolution, adds text too large to scan, or changes `.gbrain-scan-allow` or a `.gitattributes` (files that decide what the scan sees); nothing was pushed. The message names each file, line, rule and commit, never the value. Ask the user: remove the value from those unpushed commits, or add a false positive's fingerprint to `.gbrain-scan-allow`, then retry. A commit that changes `.gbrain-scan-allow` or a `.gitattributes` is never published automatically: the user reviews it and pushes it. Never publish them with a command that skips the scan. On a managed worktree its Git effects hold meanwhile (`gbrain sync status` and doctor list them; the page writes are safe in the database and in local commits); after the fix, `gbrain sources writer retry-effects <source> --parked-pushes` runs the ones that parked. |+| <a id="publish_secret_found"></a>`publish_secret_found` | 1 | Only with `--scan-secrets`, which every automatic publish uses: the one `gbrain sources refresh` runs on its own and the push of a managed worktree's Git effects. A commit about to be pushed that gbrain did not make for a page write (its commit ledger records its own page-write commits, whose bodies are stored raw by design and are not scanned; one that adopted a file edited outside gbrain, or one older than the retained ledger, is scanned) adds a value that looks like a secret, in its message, an added line (even one a later commit removed) or a merge resolution, adds text too large to scan, or changes `.gbrain-scan-allow` or a `.gitattributes` (files that decide what the scan sees); nothing was pushed. The message names each file, line, rule and commit, never the value. Ask the user: remove the value from those unpushed commits, or add a false positive's fingerprint to `.gbrain-scan-allow`, then retry. A commit that changes `.gbrain-scan-allow` or a `.gitattributes` is never published automatically: the user reviews it and pushes it. Never publish them with a command that skips the scan. On a managed worktree its Git effects hold meanwhile (`gbrain sync status` and doctor list them; the page writes are safe in the database and in local commits); after the fix, `gbrain sources writer retry-effects <source> --parked-pushes` runs the ones that parked. || <a id="publish_scan_unreadable"></a>`publish_scan_unreadable` | 1 | Only with `--scan-secrets`: the scan could not read the commits about to be pushed in full (a missing or corrupt object, git output it does not recognise, or its 300 s budget), so it reported no secret and nothing was pushed. Run `git fsck` in the checkout, fix what it names, then retry. When the remote has no such branch, a managed worktree's Git effect scans the branch's whole history and creates it only when that scan passes; a history too large to scan in the owner process holds the effects with this code and names `git ls-remote --heads <remote>`: check the upstream, then point it at the right branch or review the history and push the branch yourself. Never publish with a command that skips the scan. On a managed worktree its Git effects hold meanwhile, as for `publish_secret_found`. || `integrate_conflict` | 1 | Local and remote commits changed the same file; nothing was merged and the checkout is unchanged. Stop, relay the question to the user, then [resolve](#conflicts). || `push_unconfirmed` | 4 | The commit is durable locally but no push destination confirmed it (network, credentials, branch protection, a remote that kept moving, or another push that held the push lock for the whole wait). Fix what the message names, then run `gbrain git commit-push --push-only`. Never force-push. |diff --git a/src/core/git/range-secret-scan.ts b/src/core/git/range-secret-scan.tsindex bf74da7641..5ccd82eac6 100644--- a/src/core/git/range-secret-scan.ts+++ b/src/core/git/range-secret-scan.ts@@ -49,7 +49,11 @@ const CONFIG = ['-c', 'core.quotePath=false', '-c', 'core.attributesFile=/dev/nuconst DIFF = ['diff-tree', '--stdin', '-r', '-p', '--root', '--cc', '--no-renames', '--no-ext-diff', '--no-textconv', '--no-color','--full-index', '--src-prefix=a/', '--dst-prefix=b/', '--unified=0'];-/** Ledger writers whose commits hold only page writes: a Git effect's commit and a hosted repository's door commits. */+/**+ * Ledger writers whose commits hold only page writes: a Git effect's commit and a hosted repository's door commits. A+ * record that adopted a file edited outside gbrain (`adopted`) carries bytes gbrain did not write, so it is scanned, and+ * so is a page-write commit older than the retained ledger or made before it existed.+ */const PAGE_WRITERS: ReadonlySet<string> = new Set(['effect', 'door', 'hosted_repo']);class Unreadable extends Error { constructor(message: string, readonly timedOut = false) { super(message); } }@@ -64,7 +68,7 @@ function run(root: string, args: string[], deadline: number, opts: { input?: str/** The commits the ledger records as page writes; an unreadable ledger reads as none, so everything is scanned. */function pageWriteCommits(root: string, deadline: number): Set<string> {const common = must(run(root, ['rev-parse', '--path-format=absolute', '--git-common-dir'], deadline), 'rev-parse').toString().trim();- try { return new Set(readLedger(common).flatMap(record => record.e === 'commit' && PAGE_WRITERS.has(record.writer) ? [record.sha] : [])); }+ try { return new Set(readLedger(common).flatMap(record => record.e === 'commit' && PAGE_WRITERS.has(record.writer) && !record.adopted?.length ? [record.sha] : [])); }catch { return new Set(); }}function must(r: Run, what: string): Buffer {diff --git a/src/core/persistence/effect-git.ts b/src/core/persistence/effect-git.tsindex e0dd8ed69b..25e96d74e7 100644--- a/src/core/persistence/effect-git.ts+++ b/src/core/persistence/effect-git.ts@@ -366,7 +366,7 @@ export async function commitGitTargets(root: string, relativePaths: string[], siconst noted = new Map<string, GitCommitNote>();for (const c of changed) { const note = notes?.get(c.requested); if (note && !noted.has(c.path)) noted.set(c.path, note); }let messageDir: string | undefined;- let result;+ let result, headBefore: string | null = null;try {const paragraphs = commitMessage(commitPaths, noted);const message = commitMessageArgs(paragraphs, () => {@@ -374,14 +374,18 @@ export async function commitGitTargets(root: string, relativePaths: string[], sireturn messageDir = mkdtempSync(join(persistenceHome(), 'commit-message-'));});const before = await git(session, ['rev-parse', '--verify', '--quiet', 'HEAD']);- await recordIntentAsync(root, { writer: 'effect', head_before: before.code === 0 ? before.stdout.trim() : null, message: paragraphs.join('\n\n') });+ headBefore = before.code === 0 ? before.stdout.trim() : null;+ await recordIntentAsync(root, { writer: 'effect', head_before: headBefore, message: paragraphs.join('\n\n') });result = await git(session, [...(commitPaths.some(path => removed.has(path)) ? NO_FSCACHE : []), 'commit', '--only', ...message, '--', ...literalPathspecs(commitPaths)]);} finally { if (messageDir) rmSync(messageDir, { recursive: true, force: true }); }if (result.code === 0) {- const head = await git(session, ['rev-parse', 'HEAD']);+ // #8896: the secret scan trusts ledgered page writes, so HEAD is recorded only when its parent is the HEAD this commit+ // started from: a commit that landed on top in between (a script's plain `git commit`) stays unrecorded and scanned.+ const head = await git(session, ['rev-list', '--parents', '-n', '1', 'HEAD']);+ const [sha, ...parents] = head.code === 0 ? head.stdout.trim().split(' ') : [];const trailers = [...noted.values()].flatMap(note => note.trailers ?? []);const supersedes = supersededBy(trailers), adopted = adoptedBy(trailers);- if (head.code === 0) await recordCommitAsync(root, { sha: head.stdout.trim(), writer: 'effect', ...(supersedes.length ? { supersedes } : {}), ...(adopted.length ? { adopted } : {}) });+ if (sha && parents.join(' ') === (headBefore ?? '')) await recordCommitAsync(root, { sha, writer: 'effect', ...(supersedes.length ? { supersedes } : {}), ...(adopted.length ? { adopted } : {}) });}const outcome = result.code === 0 ? { git: 'committed' } : gitFailure('Cannot commit the canonical Git target.',`git commit failed for ${commitPaths.length} file(s); a missing Git identity (user.name, user.email) in that checkout is one cause to check.`);diff --git a/test/effect-git-secret-gate.test.ts b/test/effect-git-secret-gate.test.tsindex 0491e06f71..c47982642a 100644--- a/test/effect-git-secret-gate.test.ts+++ b/test/effect-git-secret-gate.test.ts@@ -220,6 +220,21 @@ describe('#8896 pushGitRoot scans the commits it publishes', () => {expect(remoteTip(c)).toBe(page);}, 60_000);+ test('a page-write record that adopted an outside edit, or a ledger that cannot be read, leaves the commit scanned', async () => {+ const c = checkout();+ const adopted = scriptCommit(c.root, 'notes/page.md', `Adopted edit with ${token}\n`);+ recordCommit(c.root, { sha: adopted, writer: 'effect', ref: null, adopted: ['notes/page.md'] });+ expect((await withEnv(childEnv(c), () => refusal(pushGitRoot(c.root)))).code).toBe('publish_secret_found');++ const d = checkout();+ const page = scriptCommit(d.root, 'notes/page.md', `Saved note with ${token}\n`);+ recordCommit(d.root, { sha: page, writer: 'effect', ref: null });+ const ledger = join(d.root, '.git', 'gbrain', 'commit-ledger.jsonl');+ rmSync(ledger); mkdirSync(ledger);+ expect((await withEnv(childEnv(d), () => refusal(pushGitRoot(d.root)))).code).toBe('publish_secret_found');+ expect(remoteTip(d)).not.toBe(page);+ }, 60_000);+test('a clean ahead checkout still pushes', async () => {const c = checkout();scriptCommit(c.root, 'notes/page.md', 'A clean page\n');@@ -323,6 +338,26 @@ describe('#8896 through the effect worker', () => {}, `Connection notes: ${token}`);}, 180_000);+ test('a script commit that lands right after the effect commit is not ledgered as a page write, so its secret holds the push', async () => {+ const c = checkout();+ const before = remoteTip(c);+ const real = Bun.which('git')!;+ const bin = join(c.home, 'race-bin'), mark = join(c.home, 'raced');+ mkdirSync(bin);+ writeFileSync(join(bin, 'git'), [+ '#!/bin/sh', `'${real}' "$@"`, 'code=$?',+ `case " $* " in *" commit --only "*) if [ $code -eq 0 ] && [ ! -f '${mark}' ]; then touch '${mark}';`,+ ` mkdir -p '${c.root}/scripts-out'; printf 'Deploy token %s\\n' '${token}' > '${c.root}/scripts-out/race.md';`,+ ` '${real}' -C '${c.root}' add scripts-out/race.md && '${real}' -C '${c.root}' -c core.hooksPath=/dev/null commit -q -m 'script races the effect'; fi;; esac`,+ 'exit $code', ''].join('\n'), { mode: 0o755 });+ await withEnv({ PATH: `${bin}:${process.env.PATH}` }, () => managedWrite(c, () => {}, async w => {+ await w.runEffects();+ expect(git(c.root, 'log', '-1', '--format=%s').trim()).toBe('script races the effect');+ expect(await w.effect()).toMatchObject({ error_code: 'publish_secret_found' });+ expect(remoteTip(c)).toBe(before);+ }));+ }, 180_000);+test('a parked publish_secret_found effect is released by retry-effects --parked-pushes once the value is allowlisted', async () => {const c = checkout();const before = remoteTip(c);