Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: git-reconcile's fetch gets the transport hardening every other remote git call has

garrytanf48d8312297c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 40412abae5..35271096de 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,7 +18,7 @@ identifiers and attribution are available in the pre-removal Git revision
- `src/core/git/reconcile.ts` (new): one checkout under its own reconcile lock (outermost; no other path takes it, so it cannot deadlock a publish); fetch with the tracking ref's move logged, then a `gbrain git commit-push --pull` child, a `--push-only --scan-secrets --expect-remote <checked upstream>` child (a remote that moved in between is fetched and checked for a rewrite again), or `upstream_rewritten`; a fetch that loses the tracking-ref race to a concurrent fetch retries.
- `src/core/git/upstream-history.ts` (new): the rewritten-upstream check, moved out of `src/core/persistence/worktree-refresh.ts` so refresh and reconcile share it.
-- `src/commands/sources-git-reconcile.ts` (new), `src/commands/sources.ts`: the command, grouping sources by checkout; a managed checkout waits out a running refresh up to `--wait`. A checkout that fails with an unexpected error (an unwritable `.git`, say) is reported for that checkout alone and the others still reconcile; only a failure of the whole run (the database, the disk) stops it.
+- `src/commands/sources-git-reconcile.ts` (new), `src/commands/sources.ts`: the command, grouping sources by checkout; a managed checkout waits out a running refresh up to `--wait`. A checkout that fails with an unexpected error (an unwritable `.git`, say) is reported for that checkout alone and the others still reconcile; only a failure of the whole run (the database, the disk) stops it. Its fetch carries the transport hardening every other remote git call has (no redirects, no external helpers, no submodules, never a credential prompt).
- `src/commands/autopilot-git-reconcile.ts` (new), `src/commands/autopilot-daemon.ts`: the per-tick reconcile in the daemon, logged.
- Docs and skills: `docs/guides/live-sync.md`, `docs/guides/shared-checkout-git.md`, `docs/guides/cron-schedule.md` (allow minutes for fetch and push calls), `skills/cron-scheduler/SKILL.md` (the managed cron line runs `sources git-reconcile`).
- Tests: `test/sources-git-reconcile.test.ts` (level, stranded, behind, diverged, conflict, secret, rewritten upstream, a force-push between the check and the push, eight concurrent callers while another clone pushes, managed with durability on and off and publishing off, no checkout, publishing off, a checkout gbrain did not clone, sync off, a missing folder), `test/autopilot-git-reconcile.test.ts` (the daemon's event loop keeps ticking during a publish child; a refusal is reported with no value printed; a hung child is killed).
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 38316417dc..2df0c5bf27 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1494,9 +1494,9 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.622.0', id: 'purge_cancels_deferred_write_gst1602', text: 'With `persistence.refresh_barrier` on, `gbrain forget <id> --purge` no longer refuses `purge_blocked_pending_recovery` while a write deferred behind a refresh holds the claim in its staged body: the deferred write was never claimed, so the purge cancels it with `purged_content`, as it does a queued one, and removes the content at once instead of up to 15 minutes later.' },
{ since: '0.60.623.0', id: 'owner_reads_private_gst2059', text: 'A brain can now name its owner: `gbrain access owner set <member>` (config `access.owner_member`, an enabled member). Every remote credential or stdio server bound to that member reads the whole brain, `visibility: private` pages, privacy-alias pages (`access: private`, ...), derived and pinned-question pages included, through `get_page`, `search`, `query`, `recall` and the other page reads in the sources its grant reads, and `recall` returns its private facts. `readable_by` and folder rules still bind it, and it is not relaxed when it speaks for a shared-room audience. A page the owner\'s credential creates in `default` through `put_page` is not kept to the owner (no `readable_by: [member:<id>]`). Other members, unbound and public callers are unchanged. `_meta.access.relaxation` reads `owner` for such a call, and `gbrain access check --token <name>` reports it. `gbrain repair private-labels` (named runs only) previews the `access: private` labels stamped on ordinary pages with counts per category, keeps real restrictions, and clears the rest in reviewed batches (`--apply --expect <hash>`), recording each change so `--class restore` can put it back. The bulk-ingestion page template no longer writes an `access:` line; what each designation means is in docs/guides/access-designations.md.' },
{ since: '0.60.626.0', id: 'record_origin_checkout_lockdown_8888', text: '`gbrain sources record-origin <id>... --remote --apply` now also turns git-lfs downloads and hooks off in each recorded source\'s existing checkout (its `.git/config`: `lfs.fetchexclude=*`, `core.hooksPath=/dev/null`, `lfs.locksverify=false`), not only on later reclones, in the repository\'s shared config, and refuses `write_failed` naming the key when a worktree-scoped or included setting still overrides one; a source already recorded as remote gets the lock-down on the next `--remote --apply` when its checkout lacks it, even when its files now use git-lfs, and a key that holds several values in that config is replaced by the one clone-time value. On a managed brain it takes the same worktree locks a reclone takes, refusing the id `source_busy` while a reclone, refresh or write holds them, and a reclone reads the record under its own locks. A write that fails is reported for its own id (`write_failed`) instead of ending the run. The git-lfs check reads a committed `.gitattributes` under a directory whose name holds a tab, ignores commented-out `filter=lfs` lines, says when a committed rule only declares an LFS filter, records a clone of an empty repository instead of refusing it, and names the cause of an `lfs_unknown` refusal.' },
- { since: '0.60.616.0', id: 'autopilot_git_reconcile_gbra386', text: 'Autopilot now runs the new `gbrain sources git-reconcile` each tick: every source checkout on the host with an upstream fast-forwards when it is only behind, and one gbrain cloned for the source publishes its local commits (secret scan, rewrite check bound to the push, merged with the upstream in memory, never rebased or forced); a managed checkout goes through `gbrain sources refresh`, with its Git durability rule. A checkout gbrain did not clone keeps its local commits unless `sources.refresh_publish_diverged` is `true`, and a source with sync off is skipped. A refusal is logged and retried next tick. `autopilot.git_pull false` skips it; `sources.refresh_publish_diverged false` leaves local commits unpublished.' },
- { since: '0.60.616.0', id: 'unknown_drift_history_rescreen', text: 'A `file_drift` hold on a page with no import record that an earlier release held after comparing the page only with its file at the last synced commit is now screened once more on the next sync: when the page still holds one of its file\'s earlier committed versions (only Git changed since), the Git edit imports and the hold clears; otherwise the hold stays, unchanged, and is not screened again.' },
- { since: '0.60.616.0', id: 'refresh_abandon_members_gst1465', text: '`gbrain sources refresh --abandon` of a `recovery_required` refresh now also refuses when a member\'s managed sync imported or deleted pages after the refresh began, even with its `last_commit` still at the start commit (a pass its budget stopped), and refuses `sync_drain_lease_held` while a member\'s sync pass still holds its drain lease, so the database is never left ahead of the checkout. At a HEAD past the upstream commit it names `--resume` instead of a reset. Members synced to diverged lines, or to a commit git cannot place, now get their own refusal (merge those lines, or fetch the commit) instead of a fast-forward that `--resume` would refuse again, and the refusals name the revert route for a deliberate rollback. Writer status read off the owner host names no step for such a refresh.' },
+ { since: 'TBD', id: 'autopilot_git_reconcile_gbra386', text: 'Autopilot now runs the new `gbrain sources git-reconcile` each tick: every source checkout on the host with an upstream fast-forwards when it is only behind, and one gbrain cloned for the source publishes its local commits (secret scan, rewrite check bound to the push, merged with the upstream in memory, never rebased or forced); a managed checkout goes through `gbrain sources refresh`, with its Git durability rule. A checkout gbrain did not clone keeps its local commits unless `sources.refresh_publish_diverged` is `true`, and a source with sync off is skipped. A refusal is logged and retried next tick. `autopilot.git_pull false` skips it; `sources.refresh_publish_diverged false` leaves local commits unpublished.' },
+ { since: 'TBD', id: 'unknown_drift_history_rescreen', text: 'A `file_drift` hold on a page with no import record that an earlier release held after comparing the page only with its file at the last synced commit is now screened once more on the next sync: when the page still holds one of its file\'s earlier committed versions (only Git changed since), the Git edit imports and the hold clears; otherwise the hold stays, unchanged, and is not screened again.' },
+ { since: 'TBD', id: 'refresh_abandon_members_gst1465', text: '`gbrain sources refresh --abandon` of a `recovery_required` refresh now also refuses when a member\'s managed sync imported or deleted pages after the refresh began, even with its `last_commit` still at the start commit (a pass its budget stopped), and refuses `sync_drain_lease_held` while a member\'s sync pass still holds its drain lease, so the database is never left ahead of the checkout. At a HEAD past the upstream commit it names `--resume` instead of a reset. Members synced to diverged lines, or to a commit git cannot place, now get their own refusal (merge those lines, or fetch the commit) instead of a fast-forward that `--resume` would refuse again, and the refusals name the revert route for a deliberate rollback. Writer status read off the owner host names no step for such a refresh.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/git-transport.ts b/src/core/git-transport.ts
index 312f02a6a4..c23ad978cc 100644
--- a/src/core/git-transport.ts
+++ b/src/core/git-transport.ts
@@ -105,3 +105,17 @@ export function durableSsrfFlags(): string[] {
'-c', 'protocol.ext.allow=never',
];
}
+
+/**
+ * Transport flags for a checkout's OWN configured remotes: no redirects, no
+ * external helpers, no submodule fetch surface. The file protocol keeps git's
+ * default (`user`): a bare repo on the same host is a supported brain topology
+ * (the pre-#6530 helper pushed to it with plain git), and the URL is the
+ * operator's own config, not remote-supplied input. GBRAIN_GIT_ALLOW_FILE_TRANSPORT
+ * still forces `always` for tests.
+ */
+export function ownRemoteTransportFlags(): string[] {
+ return durableSsrfFlags().flatMap((flag, i, all) =>
+ all[i - 1] === '-c' && flag.startsWith('protocol.file.allow=') && process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT !== '1'
+ ? ['protocol.file.allow=user'] : [flag]);
+}
diff --git a/src/core/git/commit-push.ts b/src/core/git/commit-push.ts
index 66ccfa80df..215fa575d8 100644
--- a/src/core/git/commit-push.ts
+++ b/src/core/git/commit-push.ts
@@ -31,7 +31,7 @@ import { scanRange } from './range-secret-scan.ts';
import { isAbsolute, join, relative, resolve } from 'node:path';
import { execFileSync } from '../spawn.ts';
import { gitChildEnv, literalPathspecs } from '../git-env.ts';
-import { GIT_ENV, GIT_ENV_AUTH, GIT_SSRF_SUBCOMMAND_FLAGS, durableSsrfFlags } from '../git-transport.ts';
+import { GIT_ENV, GIT_ENV_AUTH, GIT_SSRF_SUBCOMMAND_FLAGS, ownRemoteTransportFlags as transportFlags } from '../git-transport.ts';
import { assertManagedFilesystemWrite, hasFilesystemPublication, managedFilesystemRootFor, type ManagedFilesystemRoot } from '../persistence/filesystem-guard.ts';
import { acquireNativeLock, acquireNativeLockSync, type NativeLockSyncHandle } from '../persistence/native-lock.ts';
import { recordCommit, recordIntent } from './commit-ledger.ts';
@@ -175,20 +175,6 @@ function gitOk(root: string, args: string[], what: string, opts: Parameters<type
return r.stdout.trim();
}
-/**
- * Transport flags for this checkout's OWN configured remotes: no redirects, no
- * external helpers, no submodule fetch surface. The file protocol keeps git's
- * default (`user`): a bare repo on the same host is a supported brain topology
- * (the pre-#6530 helper pushed to it with plain git), and the URL is the
- * operator's own config, not remote-supplied input. GBRAIN_GIT_ALLOW_FILE_TRANSPORT
- * still forces `always` for tests.
- */
-function transportFlags(): string[] {
- return durableSsrfFlags().flatMap((flag, i, all) =>
- all[i - 1] === '-c' && flag.startsWith('protocol.file.allow=') && process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT !== '1'
- ? ['protocol.file.allow=user'] : [flag]);
-}
-
/** Remote operations: transport hardening on every call, credentials allowed, no prompt. */
function remoteGit(root: string, sub: 'fetch' | 'push', args: string[], timeoutMs = 120_000): GitRun {
return git(root, [...transportFlags(), sub, ...GIT_SSRF_SUBCOMMAND_FLAGS, ...args], { auth: true, timeoutMs });
diff --git a/src/core/git/reconcile.ts b/src/core/git/reconcile.ts
index a14e4884e7..434d8928ee 100644
--- a/src/core/git/reconcile.ts
+++ b/src/core/git/reconcile.ts
@@ -10,6 +10,7 @@
import { execFileBounded } from '../bounded-child-exec.ts';
import { classifyGitCheckout } from '../git-checkout.ts';
import { gitChildEnv } from '../git-env.ts';
+import { GIT_ENV_AUTH, GIT_SSRF_SUBCOMMAND_FLAGS, ownRemoteTransportFlags } from '../git-transport.ts';
import { OperationError, opError } from '../ops/contract.ts';
import { acquireNativeLock } from '../persistence/native-lock.ts';
import { CHILD_ENV, resolveChildCliInvocation } from '../minions/job-isolation.ts';
@@ -35,6 +36,12 @@ export function gitReader(root: string, timeoutMs = 120_000): GitRead {
};
}
+/** The upstream fetch: the checkout's own remote, with transport hardening (no redirects, no external helpers, no submodules) and credentials allowed but never a prompt. */
+export function upstreamFetch(root: string, remote: string, ref: string): { argv: string[]; env: NodeJS.ProcessEnv } {
+ return { argv: ['-C', root, ...ownRemoteTransportFlags(), '-c', 'core.logAllRefUpdates=true', 'fetch', ...GIT_SSRF_SUBCOMMAND_FLAGS, '--no-tags', '--quiet', remote, ref],
+ env: gitChildEnv({ ...GIT_ENV_AUTH }) };
+}
+
/** Fetch the branch's upstream (logging the tracking ref's move, so a later rewrite check has a record) and count both sides; null without an upstream. */
export async function fetchLevel(root: string, read: GitRead, fetchTimeoutMs = 120_000): Promise<CheckoutLevel | null> {
const branch = (await read(['symbolic-ref', '--quiet', '--short', 'HEAD'], 1)).stdout.trim();
@@ -44,12 +51,11 @@ export async function fetchLevel(root: string, read: GitRead, fetchTimeoutMs = 1
const remote = (await read(['config', '--get', `branch.${branch}.remote`], 1)).stdout.trim();
const merge = (await read(['config', '--get', `branch.${branch}.merge`], 1)).stdout.trim();
if (remote && remote !== '.') {
- let fetched = await execFileBounded('git', ['-C', root, '-c', 'core.logAllRefUpdates=true', 'fetch', '--no-tags', '--quiet', remote, merge || branch],
- { timeout: fetchTimeoutMs, env: gitChildEnv({ GIT_TERMINAL_PROMPT: '0' }) });
+ const fetch = upstreamFetch(root, remote, merge || branch);
+ let fetched = await execFileBounded('git', fetch.argv, { timeout: fetchTimeoutMs, env: fetch.env });
for (let attempt = 1; fetched.error && attempt < 4 && /cannot lock ref|unable to update local ref/.test(String(fetched.stderr)); attempt++) {
await new Promise(resolve => setTimeout(resolve, 250 * attempt));
- fetched = await execFileBounded('git', ['-C', root, '-c', 'core.logAllRefUpdates=true', 'fetch', '--no-tags', '--quiet', remote, merge || branch],
- { timeout: fetchTimeoutMs, env: gitChildEnv({ GIT_TERMINAL_PROMPT: '0' }) });
+ fetched = await execFileBounded('git', fetch.argv, { timeout: fetchTimeoutMs, env: fetch.env });
}
if (fetched.error) throw opError('fetch_failed', `git fetch ${remote} failed in ${root} (${String(fetched.stderr).trim().split('\n')[0] || fetched.error.message}); nothing changed.`,
'Check the remote and its credentials, then retry.', { retryable: true });
diff --git a/test/sources-git-reconcile.test.ts b/test/sources-git-reconcile.test.ts
index 5ff844b95b..3c5b1dcdcc 100644
--- a/test/sources-git-reconcile.test.ts
+++ b/test/sources-git-reconcile.test.ts
@@ -16,7 +16,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import type { BrainEngine } from '../src/core/engine.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
-import { reconcileCheckout } from '../src/core/git/reconcile.ts';
+import { reconcileCheckout, upstreamFetch } from '../src/core/git/reconcile.ts';
import { reconcileSources } from '../src/commands/sources-git-reconcile.ts';
import { runSources } from '../src/commands/sources.ts';
import { REFRESH_PUBLISH_DIVERGED_KEY } from '../src/core/persistence/worktree-refresh.ts';
@@ -72,6 +72,13 @@ const childEnv = { ...identity, GBRAIN_JOB_CHILD_CLI: childCli };
const reconcile = (root: string) => withEnv(childEnv, () => reconcileCheckout(root));
const reconcileAll = (engine: BrainEngine, ids?: string[]) => withEnv(childEnv, () => reconcileSources(engine, ids ? { sourceIds: ids } : {}));
+test('the upstream fetch carries the transport hardening of every other remote git call and never prompts', () => {
+ const { argv, env } = upstreamFetch('/checkout', 'origin', 'main');
+ expect(argv).toEqual(expect.arrayContaining(['http.followRedirects=false', 'protocol.ext.allow=never']));
+ expect(argv.indexOf('--no-recurse-submodules')).toBeGreaterThan(argv.indexOf('fetch'));
+ expect(env).toMatchObject({ GIT_TERMINAL_PROMPT: '0', GCM_INTERACTIVE: 'never' });
+});
+
test('no Git checkout and no upstream report themselves and change nothing', async () => {
const plain = join(dir, 'plain'); mkdirSync(plain);
expect(await reconcileCheckout(plain)).toMatchObject({ status: 'no_repo' });