Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: git-reconcile publishes only checkouts gbrain actually cloned, and leaves another host's managed sources alone
garrytan1df07bec672f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 35271096de..9b9f7cb59c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,11 +14,11 @@ identifiers and attribution are available in the pre-removal Git revision**Source checkouts now stay level with their remotes without help. A held page that changed only in Git imports on the next sync, and every way out of an interrupted refresh works.**-**Every source checkout now converges on its remote by itself: `gbrain sources git-reconcile`, run by autopilot each tick.** One idempotent call brings every source checkout on the host level with its upstream. A checkout that is only behind fast-forwards. A checkout gbrain cloned for the source (a recorded remote URL, the source folder at the repository root) publishes its local commits merged with the upstream in memory: the same secret scan and rewritten-upstream refusal as refresh, bound to the upstream it checked so a force-push in between refuses, never a rebase or force-push. Any other checkout, such as a source folder inside a repository gbrain did not clone, reports its local commits `unpublished` unless `sources.refresh_publish_diverged` is `true`. A managed checkout goes through `gbrain sources refresh` and its Git durability rule. Sources with sync turned off are skipped unless named. A level checkout costs one fetch and reports `in_sync`, and callers on the same checkout take turns, so any agent, cron or hook can call it as often and as concurrently as it likes. Autopilot runs it each tick on every engine, its pulls and publishes in bounded child processes, so a slow push never stalls the daemon; `autopilot.git_pull false` skips it, and `sources.refresh_publish_diverged false` leaves local commits `unpublished` while checkouts behind still fast-forward. Exit 0 means level, 1 that a checkout needs the user, 4 that a retry can fix it.+**Every source checkout now converges on its remote by itself: `gbrain sources git-reconcile`, run by autopilot each tick.** One idempotent call brings every source checkout on the host level with its upstream. A checkout that is only behind fast-forwards. A checkout gbrain cloned for the source (`gbrain sources add --url`, with the source folder at the repository root) publishes its local commits merged with the upstream in memory: the same secret scan and rewritten-upstream refusal as refresh, bound to the upstream it checked so a force-push in between refuses, never a rebase or force-push. Any other checkout, such as a source folder inside a repository gbrain did not clone, reports its local commits `unpublished` unless `sources.refresh_publish_diverged` is `true`. A managed checkout goes through `gbrain sources refresh` and its Git durability rule. Sources with sync turned off are skipped unless named. A level checkout costs one fetch and reports `in_sync`, and callers on the same checkout take turns, so any agent, cron or hook can call it as often and as concurrently as it likes. Autopilot runs it each tick on every engine, its pulls and publishes in bounded child processes, so a slow push never stalls the daemon; `autopilot.git_pull false` skips it, and `sources.refresh_publish_diverged false` leaves local commits `unpublished` while checkouts behind still fast-forward. Exit 0 means level, 1 that a checkout needs the user, 4 that a retry can fix it.- `src/core/git/reconcile.ts` (new): one checkout under its own reconcile lock (outermost; no other path takes it, so it cannot deadlock a publish); fetch with the tracking ref's move logged, then a `gbrain git commit-push --pull` child, a `--push-only --scan-secrets --expect-remote <checked upstream>` child (a remote that moved in between is fetched and checked for a rewrite again), or `upstream_rewritten`; a fetch that loses the tracking-ref race to a concurrent fetch retries.- `src/core/git/upstream-history.ts` (new): the rewritten-upstream check, moved out of `src/core/persistence/worktree-refresh.ts` so refresh and reconcile share it.-- `src/commands/sources-git-reconcile.ts` (new), `src/commands/sources.ts`: the command, grouping sources by checkout; a managed checkout waits out a running refresh up to `--wait`. A checkout that fails with an unexpected error (an unwritable `.git`, say) is reported for that checkout alone and the others still reconcile; only a failure of the whole run (the database, the disk) stops it. Its fetch carries the transport hardening every other remote git call has (no redirects, no external helpers, no submodules, never a credential prompt).+- `src/commands/sources-git-reconcile.ts` (new), `src/commands/sources.ts`: the command, grouping sources by checkout; a managed checkout waits out a running refresh up to `--wait`. A checkout that fails with an unexpected error (an unwritable `.git`, say) is reported for that checkout alone and the others still reconcile; only a failure of the whole run (the database, the disk) stops it. Its fetch carries the transport hardening every other remote git call has (no redirects, no external helpers, no submodules, never a credential prompt). On a managed brain a source with no checkout bound to this host is left alone, and a checkout with a recorded remote that gbrain did not clone keeps its local commits.- `src/commands/autopilot-git-reconcile.ts` (new), `src/commands/autopilot-daemon.ts`: the per-tick reconcile in the daemon, logged.- Docs and skills: `docs/guides/live-sync.md`, `docs/guides/shared-checkout-git.md`, `docs/guides/cron-schedule.md` (allow minutes for fetch and push calls), `skills/cron-scheduler/SKILL.md` (the managed cron line runs `sources git-reconcile`).- Tests: `test/sources-git-reconcile.test.ts` (level, stranded, behind, diverged, conflict, secret, rewritten upstream, a force-push between the check and the push, eight concurrent callers while another clone pushes, managed with durability on and off and publishing off, no checkout, publishing off, a checkout gbrain did not clone, sync off, a missing folder), `test/autopilot-git-reconcile.test.ts` (the daemon's event loop keeps ticking during a publish child; a refusal is reported with no value printed; a hung child is killed).diff --git a/docs/guides/live-sync.md b/docs/guides/live-sync.mdindex ff6a4ce497..5921f1667d 100644--- a/docs/guides/live-sync.md+++ b/docs/guides/live-sync.md@@ -700,8 +700,8 @@ vars — incident-time escape hatches, not everyday knobs.`gbrain sources git-reconcile` does this for every source checkout on thehost at once: a managed checkout goes through the refresh above; a checkout- gbrain cloned for the source (a recorded remote URL, the source folder at the- repository root) publishes its local commits the same way (secret scan,+ gbrain cloned for the source (`gbrain sources add --url`, the source folder+ at the repository root) publishes its local commits the same way (secret scan,rewrite check, merge in memory, bound to the upstream it checked so aforce-push in between refuses) and fast-forwards when it is only behind. Anyother checkout, such as a source folder inside a repository gbrain did notdiff --git a/scripts/core-world-reads-baseline.tsv b/scripts/core-world-reads-baseline.tsvindex cf757e1d89..637ac1987a 100644--- a/scripts/core-world-reads-baseline.tsv+++ b/scripts/core-world-reads-baseline.tsv@@ -431,7 +431,7 @@ src/core/git-head.ts spawn_import 1 W7: process-wide effects move to src/clisrc/core/git-remote.ts fs_import 1 W7: filesystem boundary moves to the local appsrc/core/git-remote.ts process_env 1 W6: settings plane out of the coresrc/core/git-remote.ts spawn_import 1 W7: process-wide effects move to src/cli-src/core/git-transport.ts process_env 1 W6: settings plane out of the core+src/core/git-transport.ts process_env 2 W6: settings plane out of the coresrc/core/git-visible-files.ts als_store 1 W2: engine scope (decision E) replaces ad hoc async contextsrc/core/git-visible-files.ts fs_import 1 W7: filesystem boundary moves to the local appsrc/core/git-visible-files.ts spawn_import 1 W7: process-wide effects move to src/cli@@ -442,7 +442,7 @@ src/core/git/commit-ledger.ts module_collection 1 W6: per-engine state; PR B ledsrc/core/git/commit-ledger.ts spawn_import 1 W7: process-wide effects move to src/cli; PR B ledger resolves the git common dirsrc/core/git/commit-push.ts fs_import 1 W7: filesystem boundary moves to the local appsrc/core/git/commit-push.ts module_collection 1 W6: per-engine state, or mark as a constant table-src/core/git/commit-push.ts process_env 7 W6: settings plane out of the core+src/core/git/commit-push.ts process_env 6 W6: settings plane out of the coresrc/core/git/commit-push.ts spawn_import 1 W7: process-wide effects move to src/clisrc/core/git/range-secret-scan.ts fs_import 1 W7: filesystem boundary moves to the local app (the publish scan streams git output through a private temp file)src/core/git/range-secret-scan.ts spawn_import 1 W7: process-wide effects move to src/cli (the publish scan runs git plumbing)diff --git a/src/commands/sources-git-reconcile.ts b/src/commands/sources-git-reconcile.tsindex 708d719ec9..09cd46dec2 100644--- a/src/commands/sources-git-reconcile.ts+++ b/src/commands/sources-git-reconcile.ts@@ -17,7 +17,7 @@ import { isTruthy } from '../core/connectors/config-keys.ts';import { existsSync, realpathSync } from 'node:fs';import { classifyGitCheckout } from '../core/git-checkout.ts';import { gitTopLevel } from '../core/persistence/git-gap.ts';-import { sourceConfigHasRemoteUrl } from '../core/sources-load.ts';+import { isOwnedClone } from '../core/url-clone.ts';import { isSyncDisabledConfig } from '../core/sync-policy.ts';import { OperationError } from '../core/ops/contract.ts';import { classifyItemFailure, type ClassifiedItemFailure } from '../core/item-failure.ts';@@ -82,10 +82,11 @@ export async function reconcileSources(engine: BrainEngine, opts: ReconcileOptiofor (const source of listed) {if (!explicit && isSyncDisabledConfig(configs.get(source.id))) continue;const binding = managed ? await getWorktreeBinding(engine, source.id, existingHostIdFor(engine)) : null;+ if (managed && !binding) continue;const root = binding?.local_path ?? source.local_path!;if (!existsSync(root) || classifyGitCheckout(root) === 'not_git') continue;const key = binding ? `worktree:${binding.worktree_id}` : `path:${root}`;- const cloned = sourceConfigHasRemoteUrl(configs.get(source.id)) && gitTopLevel(root) === realpathSync(root);+ const cloned = isOwnedClone({ id: source.id, local_path: source.local_path, config: configs.get(source.id) }) && gitTopLevel(root) === realpathSync(root);const group = groups.get(key) ?? { sources: [], root, managed: !!binding, cloned };group.sources.push(source.id);group.cloned &&= cloned;@@ -100,7 +101,7 @@ export async function reconcileSources(engine: BrainEngine, opts: ReconcileOptioconst result = group.managed? await withReconcileLock(group.root, opts.waitMs ?? 300_000, () => reconcileManaged(engine, group.sources[0], group.root, publish, opts)): await reconcileCheckout(group.root, { dryRun: opts.dryRun, publish, ...(opts.waitMs !== undefined ? { waitMs: opts.waitMs } : {}), ...(opts.childTimeoutMs ? { childTimeoutMs: opts.childTimeoutMs } : {}),- ...(publishSetting === null && !cloned ? { unpublishedReason: `gbrain did not clone this checkout for the source (no recorded remote, or the source folder is not the repository root); publish it with gbrain git commit-push --push-only --scan-secrets --path ${group.root}, or set ${REFRESH_PUBLISH_DIVERGED_KEY} true to publish such checkouts automatically` } : {}) });+ ...(publishSetting === null && !cloned ? { unpublishedReason: `gbrain did not clone this checkout for the source (it was not added with gbrain sources add --url, or the source folder is not the repository root); publish it with gbrain git commit-push --push-only --scan-secrets --path ${group.root}, or set ${REFRESH_PUBLISH_DIVERGED_KEY} true to publish such checkouts automatically` } : {}) });out.push({ ...entry, ...result });} catch (error) {const failure = classifyItemFailure(error);diff --git a/src/core/persistence/sync-status.ts b/src/core/persistence/sync-status.tsindex d7cf2c61c6..5c14652d89 100644--- a/src/core/persistence/sync-status.ts+++ b/src/core/persistence/sync-status.ts@@ -237,7 +237,7 @@ export interface SyncStatus {landing_read: SyncStatusRead;}export type SyncStatusState = 'synced' | 'resumable' | 'blocked';-type RefreshProbeRow = { id: string; worktree_id: string; state: import('./worktree-refresh-schema.ts').WorktreeRefreshState; source_ids: string[]; outcome: Record<string, unknown>; old_head: string; target_head: string; updated_at: string; owner_host_id: string | null };+type RefreshProbeRow = { id: string; worktree_id: string; state: import('./worktree-refresh-schema.ts').WorktreeRefreshState; source_ids: string[]; outcome: Record<string, unknown>; old_head: string; target_head: string; barrier_sequence: string | number | null; updated_at: string; owner_host_id: string | null };/** Why the source is in its state; one closed list. */export const SYNC_STATUS_REASONS = ['nothing_pending', 'held_by_design', 'held_needs_human', 'cursor_unfinished', 'live_job', 'last_error', 'holds_actionable', 'recovery_pending', 'request_pending','status_unknown', 'needs_human', 'head_blocked', 'recovery_blocked',@@ -356,7 +356,7 @@ export async function readSyncStatus(engine: BrainEngine, sourceId: string, opts(SELECT r.request_id::text FROM persistence_effects e JOIN persistence_requests r ON r.id=e.request_id WHERE e.kind='git' AND e.state='failed' AND NOT (e.data ? 'settled')AND e.request_id IN (SELECT id::uuid FROM jsonb_array_elements_text(CASE WHEN jsonb_typeof(p.completed_keys->0->'obligations'->'converged'->'writeback')='array'THEN p.completed_keys->0->'obligations'->'converged'->'writeback' ELSE '[]'::jsonb END) AS w(id)) ORDER BY e.id LIMIT 1) AS writeback_failed,- (SELECT row_to_json(f) FROM (SELECT r.id,r.worktree_id,r.state,r.source_ids,r.outcome,r.old_head,r.target_head,r.updated_at,w.owner_host_id FROM persistence_worktree_refreshes r+ (SELECT row_to_json(f) FROM (SELECT r.id,r.worktree_id,r.state,r.source_ids,r.outcome,r.old_head,r.target_head,r.barrier_sequence,r.updated_at,w.owner_host_id FROM persistence_worktree_refreshes rLEFT JOIN persistence_worktrees w ON w.id=r.worktree_id WHERE s.id=ANY(r.source_ids) AND r.state IN ${ACTIVE_REFRESH_STATES_SQL} LIMIT 1) f) AS refresh FROM sources sLEFT JOIN op_checkpoints p ON p.op=$2 AND p.fingerprint=s.id||':'||s.incarnation::textLEFT JOIN op_checkpoints w ON w.op=$3 AND w.fingerprint=s.id||':'||s.incarnation::textdiff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.tsindex de34121b99..e6b6d9cb78 100644--- a/src/core/persistence/worktree-refresh.ts+++ b/src/core/persistence/worktree-refresh.ts@@ -118,7 +118,7 @@ const retryInTick = (error: OperationError) => Object.assign(error, { [RETRY_IN_*/async function membersPastStart(engine: BrainEngine, row: Pick<WorktreeRefreshRow, 'source_ids' | 'old_head' | 'barrier_sequence'>, root: string): Promise<string[]> {const members = await engine.executeRaw<{ id: string; last_commit: string | null; imported: boolean }>(`SELECT s.id,s.last_commit,- EXISTS (SELECT 1 FROM persistence_requests r WHERE r.source_id=s.id AND r.sequence > COALESCE($2::bigint,0)+ EXISTS (SELECT 1 FROM persistence_requests r WHERE r.source_id=s.id AND $2::bigint IS NOT NULL AND r.sequence > $2::bigintAND r.intent->>'kind' IN ('managed_sync_import','managed_sync_delete') AND r.state NOT IN ('failed','conflict','cancelled')) AS importedFROM sources s WHERE s.id = ANY($1::text[]) ORDER BY s.id`, [row.source_ids, row.barrier_sequence == null ? null : String(row.barrier_sequence)]);const moved = members.filter(member => member.imported || (member.last_commit !== null && member.last_commit !== row.old_head));@@ -322,7 +322,7 @@ function wayOutFix(row: Pick<WorktreeRefreshRow, 'id' | 'source_ids' | 'old_headconst resume = (why: string): Action => ({ argv: ['gbrain', 'sources', 'refresh', source, '--resume'], consent: [], actor: 'agent', requires_exclusive: false, why });if (way.kind === 'unplaced') return { argv: ['git', '-C', root, 'fetch', '--all', '--no-tags', '--quiet'], consent: [], actor: 'agent', requires_exclusive: false,why: `Git cannot place ${unplacedText(way)} in ${root}, so recovery cannot name the commit that keeps those pages. A fetch moves only remote-tracking refs and brings back a commit a remote still holds; then --resume names the way out. If no remote holds it, ask the user.`,- verify: { argv: ['git', '-C', root, 'cat-file', '-e', `${way.members[0]!.commit}^{commit}`] },+ ...(way.members[0] ? { verify: { argv: ['git', '-C', root, 'cat-file', '-e', `${way.members[0].commit}^{commit}`] } } : {}),then: resume(`With git able to place every member commit, names the way out that keeps their pages, or finishes refresh ${row.id}.`) };const head = way.kind === 'forward' ? way.head : null;return { ask_user: true, argv: head ? ['git', '-C', root, 'merge', '--ff-only', head] : ['git', '-C', root, 'merge', '--no-edit', ...(way as { tips: string[] }).tips],diff --git a/test/autopilot-git-reconcile.test.ts b/test/autopilot-git-reconcile.test.tsindex d2b59b2724..5bebc37bee 100644--- a/test/autopilot-git-reconcile.test.ts+++ b/test/autopilot-git-reconcile.test.ts@@ -38,7 +38,7 @@ function source(name: string): { root: string; remote: string } {}async function only(name: string, root: string, remote: string) {await engine.executeRaw('DELETE FROM sources WHERE id <> $1', ['default']);- await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::jsonb)', [name, root, JSON.stringify({ remote_url: remote })]);+ await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::jsonb)', [name, root, JSON.stringify({ remote_url: remote, managed_clone: true })]);}function wrapper(name: string, onPush: string): string {const path = join(dir, name);diff --git a/test/sources-git-reconcile.test.ts b/test/sources-git-reconcile.test.tsindex 3c5b1dcdcc..8a8694c8cd 100644--- a/test/sources-git-reconcile.test.ts+++ b/test/sources-git-reconcile.test.ts@@ -238,7 +238,7 @@ test('with automatic publishing turned off, local commits stay unpublished whileconst engine = plain;try {const r = repo('optout');- await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::jsonb)', ['optout-source', r.root, JSON.stringify({ remote_url: r.remote })]);+ await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::jsonb)', ['optout-source', r.root, JSON.stringify({ remote_url: r.remote, managed_clone: true })]);await engine.setConfig(REFRESH_PUBLISH_DIVERGED_KEY, 'false');const theirs = siblingPush(r, 'notes/theirs.md', 'from the sibling\n');expect(await reconcileAll(engine)).toMatchObject([{ sources: ['optout-source'], status: 'fast_forwarded', behind: 1 }]);@@ -279,7 +279,7 @@ test('a checkout that fails with a plain error is reported for that checkout andconst broken = repo('plain-error'), healthy = repo('plain-healthy');try {await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$4::jsonb),($3,$3,$5,$6::jsonb)',- ['broken-source', broken.root, 'healthy-source', JSON.stringify({ remote_url: broken.remote }), healthy.root, JSON.stringify({ remote_url: healthy.remote })]);+ ['broken-source', broken.root, 'healthy-source', JSON.stringify({ remote_url: broken.remote, managed_clone: true }), healthy.root, JSON.stringify({ remote_url: healthy.remote, managed_clone: true })]);commit(broken.root, 'notes/stuck.md', 'stuck\n');const local = commit(healthy.root, 'notes/mine.md', 'mine\n');chmodSync(join(broken.root, '.git'), 0o555);@@ -292,3 +292,16 @@ test('a checkout that fails with a plain error is reported for that checkout andawait resetPlain();}}, 120_000);++test('a checkout with a recorded remote that gbrain did not clone keeps its local commits by default', async () => {+ const engine = plain;+ const r = repo('recorded-remote');+ try {+ await engine.executeRaw('INSERT INTO sources(id,name,local_path,config) VALUES($1,$1,$2,$3::jsonb)', ['recorded-remote', r.root, JSON.stringify({ remote_url: r.remote })]);+ const before = tip(r);+ const local = commit(r.root, 'notes/private.md', 'not for the remote\n');+ expect(await reconcileAll(engine)).toMatchObject([{ sources: ['recorded-remote'], status: 'unpublished', ahead: 1 }]);+ expect(tip(r)).toBe(before);+ expect(onRemote(r, local)).toBe(false);+ } finally { await resetPlain(); }+}, 120_000);