Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
26.0 KB
Raw Blame
/**
 * `gbrain sync` CLI flag parsing and usage text (refactor wave 1, W4 sync).
 * Invalid values print the same messages and exit with the same codes as
 * before; parsing is split in two because `--break-lock` runs (and exits)
 * between the two halves.
 */
import { loadConfig } from '../../core/config.ts';
import { parseDurationSeconds, parseWorkers } from '../../core/sync-concurrency.ts';
import { resolveNoEmbed } from '../../core/sync-git.ts';
import type { SyncOpts } from '../sync.ts';
import { parseMissingPathMode } from './missing-path.ts';
import { intFlagValue } from '../../cli/flag-values.ts';
import { usageError } from '../../cli/cli-error.ts';
import type { MissingPathMode } from './missing-path.ts';

export function printSyncHelp(): void {
  console.log(`Usage: gbrain sync [options]

Sync the brain repo's text content into the engine, then embed.

A file whose content refuses deterministically (frontmatter gbrain cannot
read without guessing, a conflicting frontmatter slug, over-size, a
content_sanity reject, or on a managed brain a facts or takes fence that
cannot be imported) is held: the rest of the source imports, the
checkpoint advances, and each hold prints its code, location and next
command. Inspect holds with 'gbrain sources status <id>'. Preview a
frontmatter fix with 'gbrain repair frontmatter --source <id>'. A fence hold
(invalid_fence) is repaired by the next maintenance run when one is active;
preview it with 'gbrain repair fences --source <id>' (read-only, no model
call), which prints the apply command, or the exact edit for a fence gbrain
will not guess. A source a file blocked before this release recovers on its
next sync ('--no-pull' on a managed brain). On a managed brain a write whose
preparation never finishes within its attempts is held as
preparation_stalled: the file is fine, so no repair applies; inspect the
writer with 'gbrain sources writer status --source <id> --json', fix what it
names, then 'gbrain sources retry-held <id>' and the same sync. Many such
stalls in one run stop it with outcome blocked / preparation_systemic.
'gbrain config set sync.holds fail' restores fail-closed blocking.

Managed brains (the flags an operator or cron needs; listed first):
  --source <id>        Scope sync to a single source. Defaults to the
                       brain's default source.
  --all                Sync every registered source instead of just the
                       default (multi-source brains).
  --no-pull            Skip 'git pull' before the sync. Implied on managed
                       brains, which never pull during sync: 'gbrain sources
                       refresh <id>' moves the checkout, and sync catches the
                       index up to it.
  --timeout <dur>      Stop the sync after <dur> (per source with --all).
                       A managed sync drains its whole backlog in one run
                       until it is done, blocked, or this budget ends;
                       progress never extends it. A stopped managed drain
                       is 'resumable' (exit 0): rerun the same command.
                       When another process is draining the source (a
                       serve, a sync job, another terminal), the run follows
                       it within this budget and its own deadline, printing
                       the holder and the pages it commits; it takes over
                       when the holder finishes. Out of budget it is
                       'resumable' with code sync_drain_lease_held; a holder
                       that commits nothing for 5 minutes stops it blocked
                       (drain_stalled). Ctrl-C stops following and leaves
                       the holder alone.
                       See docs/guides/live-sync.md (catching up a large
                       backlog on managed Postgres).
  --hard-deadline <dur>
                       Stop the whole process after <dur> (for example 13m),
                       even when a step hangs. Doctor's managed cron line is
                       'gbrain sync --all --no-pull --hard-deadline 13m'
                       every 15 minutes. A non-TTY run gets a default
                       deadline that extends while pages keep committing;
                       --no-hard-deadline waives it.
  --json               Emit a structured JSON envelope on stdout
                       ({schema_version: 1, sources, parallel,
                       ok_count, error_count, skipped_count}). Sources
                       skipped by --missing-path skip appear with
                       status 'skipped_missing_path' and their
                       local_path. All human output routes to stderr
                       (single-source runs too) so '--json | jq'
                       parses cleanly.
                       Exit codes: 0 = all sources ok or skipped,
                       1 = any error, 2 = cost-prompt-not-confirmed.
                       Managed syncs add outcome (synced | resumable |
                       blocked), drain and next {command, safe_to_loop,
                       eta_seconds, why}.
  --dry-run            Show what would be synced without writing; lists
                       every file the content screen would hold (would_hold)
                       and screen errors separately (screen_skipped).
                       Managed sources add 'convergence' to --json: every
                       path classified the way the level-triggered pass
                       will (counts, predicted decisions, deletes and the
                       mass-delete verdict, samples). Read-only.
  --no-bulk            Managed Postgres sync: publish one page per transaction
                       instead of bulk groups (each page keeps its own write
                       request either way). GBRAIN_SYNC_BULK=0 does the
                       same for every run of a process; tune the group size
                       with 'gbrain config set sync.bulk_size N'.
  --no-lanes           Same as --lanes 1: one bulk group at a time.
  --break-lock         Release the source's sync lock (the managed drain
                       lease) left by a holder that is gone: a dead pid on
                       this host, or a heartbeat past its TTL. A live holder
                       is never broken; --max-age <s> narrows it.

Classic brains / advanced (classic sync, processing opt-outs, one-off repair):
  --no-embed           Skip the embed step. Use this when the embed
                       provider is misconfigured or you want to defer
                       embedding (run 'gbrain embed --stale' later).
  --no-extract         Skip the link/timeline extraction step. Pages will
                       show as stale in 'gbrain doctor'; run
                       'gbrain extract --stale' later to catch up.
  --workers N          Run the import phase with N parallel workers
                       (alias: --concurrency). Default: 4 when the
                       diff is >100 files, else serial.
  --repo <path>        Path to the brain repo. Defaults to the path
                       saved by 'gbrain init'.
  --full               Force a full re-sync (rare; usually incremental).
  --redact-paths       With --dry-run --json: sample paths become sha256
                       prefixes and slugs are dropped, so the counts can
                       be shared.
  --src-subpath <dir>  Sync only this subdirectory of the git repo (monorepo
                       pattern: N logical sources in one repo). Git pull/diff
                       run at the repo root; imports are scoped to the subdir
                       and slugs stay root-relative (wiki/page1). Passing the
                       subdirectory directly as --repo also works.
  --exclude <glob>     Exclude files matching the glob from sync (repeatable;
                       matched against the scope-relative path).
  --include-hidden <glob>
                       Waive the leading-dot prune (.git, .obsidian, and any
                       other dot-prefixed directory — but NOT node_modules/
                       vendor/dist/build/venv/*.raw, which are never
                       waivable) for paths matching this glob (repeatable).
                       Does not reach a non-git directory's FS-walk import
                       fallback; every git-tracked source (the normal case)
                       is covered. Cannot combine with --all; persist it as
                       the sync.include_hidden config key (gbrain config set
                       sync.include_hidden '<globs>') so --all, autopilot and
                       the dream cycle honor it.
  --include-gitignored Include otherwise-syncable files matched by .gitignore.
                       Forces a full filesystem walk so periodic syncs see
                       ignored untracked content.
  --working-tree       Also import uncommitted working-tree state (untracked
                       files + uncommitted edits/deletes). Default: committed
                       changes only — uncommitted drift is counted and warned,
                       never silently ignored. Persist with
                       'gbrain config set sync.include_working_tree true'.
                       Caution: imports untracked files as-is — unignored
                       scratch files and secrets included; review 'git status'
                       before enabling, especially as persisted config.
  --skip-failed        Classic sync only: acknowledge previously-recorded
                       sync failures so the bookmark can advance. Held files
                       never need it; a managed source refuses it
                       (managed_sync_flag_refused), as it does
                       --include-gitignored: fix the file, then run the
                       plain sync again.
  --acknowledge-managed <request_id|path>
                       Managed sync, owner host only: remove a recorded
                       failure (for example a --full run's) that the imported
                       commit already settles; refuses anything still unsynced.
  --retry-failed       Deprecated: accepted for two releases and prints a
                       coaching notice. A plain sync now retries a failure
                       the previous run recorded by itself, so drop the
                       flag. Held files re-screen on the next sync when they
                       change (or with 'gbrain sources retry-held <id>'). On
                       a google or github connector source it still
                       re-approves the failed item receipts it meets.
  --reset-checkpoint   Connector source only: re-walk its window once from an empty
                       checkpoint; unchanged pages are not admitted again.
  --watch              Re-sync continuously on an interval.
  --interval N         Watch-mode interval in seconds (default 60).
  --lanes N            Managed Postgres sync: publish at most N bulk groups at
                       once (1-16, default 16, capped by the connection pool:
                       GBRAIN_POOL_SIZE minus 4). Pages still commit in file
                       order. Persist with 'gbrain config set sync.lanes N' or
                       GBRAIN_SYNC_LANES; the drain reports what limited it.
  --no-delegate        Keep this process's own write consumer even when a
                       live 'gbrain serve' could run the sync. Normally sync
                       hands the run to that serve over its IPC socket and
                       prints its progress: on PGLite because the serve holds
                       the single-writer lock, on a managed Postgres brain so
                       the host keeps one write consumer. This flag (or
                       GBRAIN_SYNC_NO_DELEGATE=1) opts out: on Postgres the
                       sync runs here beside the serve; on PGLite it fails
                       fast if a live serve holds the brain. It never skips
                       the drain lease: one drainer per managed source.
  --force-break-lock   Release it without the liveness check (destructive:
                       only when you have stopped the holder yourself).
  --no-schema-pack     Skip loading the active schema pack (no per-file pack
                       regex runs; pages use legacy prefix typing). Escape
                       hatch if a suspect pack regex is wedging sync.
                       GBRAIN_SYNC_TRACE=1 names the file being imported (hang triage).
  --parallel N         (with --all) Run up to N sources concurrently.
                       Default: min(sourceCount, --workers, 4). Each
                       source takes its own per-source DB lock
                       (gbrain-sync:<source_id>) so independent sources
                       sync without contending. Total live Postgres
                       connections per wave ≈ parallel × workers × 2
                       (per-file pool) + parent pool. Pass --parallel 1
                       to force serial.
  --missing-path M     (with --all) What to do when a source's local_path
                       does not exist on this machine: 'fail' (default —
                       loud, current behavior) or 'skip' (classify as
                       skipped_missing_path: ⊘ in the aggregate, excluded
                       from error_count and the rc=1 gate). Use skip on
                       brains whose sources were registered from more
                       than one machine.
  --yes                Accept any interactive prompts (CI / non-TTY).

Subcommands:
  gbrain sync status --source <id> [--json]
                       Where a managed catch-up stands: pass or cursor, pages
                       committed in the last 10 minutes, each hold and the
                       last error with class / safe_actions / needs_human,
                       and the next action (for an operator agent's loop).
                       'gbrain sync --status' is the same read (never a sync);
                       sync refuses --preview, which it never reads.
  gbrain sync unblock --source <id> [--apply] [--no-llm] [--no-repair] [--json]
                       Performs the safe action for every hold (re-screens a
                       held file whose edit is now committed, retries a
                       stalled preparation, repairs a content hold through
                       the content-repair lane with a hash-bound apply and
                       a receipt per file) and refuses the rest by name.
                       Runbook: docs/guides/sync-unblock-runbook.md

See also:
  gbrain embed --stale    Re-embed all stale chunks (post --no-embed).
  gbrain doctor           Diagnose dim mismatches and other sync issues.
  gbrain sources status <id>              Held files with their next command.
  gbrain repair frontmatter --source <id> Preview the fix for frontmatter holds.
  gbrain repair fences --source <id>      Preview the repair of fence holds.
  gbrain sources writer status --source <id> --json
                                          What a stalled write was doing (preparation_stalled holds).
  docs/guides/repair.md#held-files        Walkthrough.
  docs/guides/troubleshooting.md#catch-up-stuck
                                          Catch-up stuck / held N files runbook.
`);
}

/** #6712 PR 3 (L7): the coaching notice `--retry-failed` prints while it stays accepted (two releases). */
export const RETRY_FAILED_DEPRECATED = '--retry-failed is deprecated: a plain gbrain sync now retries a failure the previous run recorded by itself, '
  + 'so drop the flag (it stays accepted for two releases and changes nothing on a Git source). On a google or github connector source it still '
  + 're-approves the failed item receipts it meets.';
let retryFailedNoted = false;
/** Prints the `--retry-failed` coaching notice on stderr, once per process (the CLI pipeline and the sync command both call it). */
export function noteRetryFailedDeprecated(args: string[]): void {
  if (retryFailedNoted || !args.includes('--retry-failed')) return;
  retryFailedNoted = true;
  process.stderr.write(`[gbrain notice sync_retry_failed_deprecated] ${RETRY_FAILED_DEPRECATED}\n`);
}

/** Tests share one process across files: forget that an earlier file already printed the notice. */
export function resetRetryFailedNoticeForTests(): void {
  retryFailedNoted = false;
}

/** Flags read before the `--break-lock` branch. */
/** setTimeout treats delays above 2^31-1 ms as ~1 ms; --interval stays within that bound. */
const MAX_WATCH_INTERVAL_SECONDS = Math.floor(2_147_483_647 / 1000);

/**
 * #5988: there is no `sync --retry-held`. Held files re-screen on the next
 * sync by themselves; `sources retry-held` schedules a re-screen without
 * running anything, so the refusal names it instead of acting as an alias.
 */
function retryHeldRefusal(args: string[]) {
  const source = args.find((a, i) => args[i - 1] === '--source');
  return usageError('gbrain sync has no --retry-held flag; held files are re-screened on the next sync, and sources retry-held schedules a re-screen for a source.',
    `Run gbrain sources retry-held ${source ?? '<source-id>'}, then sync that source again without --retry-held.`, {
      fix: { argv: ['gbrain', 'sources', 'retry-held', source ?? '<source-id>'], consent: [], actor: 'agent', requires_exclusive: false,
        ...(source ? {} : { inputs: [{ name: 'source-id', how: 'gbrain sources status lists each source with its held files' }] }),
        why: 'Most held files re-screen on the next sync by themselves (changed, deleted, or readable by a newer gbrain); retry-held schedules a re-screen of every held file of the source on its next sync and runs nothing now.',
        verify: { argv: ['gbrain', 'sources', 'status', ...(source ? [source] : []), '--json'] } } });
}

export function parseSyncFlags(args: string[]) {
  if (args.includes('--retry-held')) throw retryHeldRefusal(args);
  const repoPath = args.find((a, i) => args[i - 1] === '--repo') || undefined;
  const watch = args.includes('--watch');
  // #5931 (D4): timers coerce delays above 2^31-1 ms (and NaN/0) to ~1 ms, so a bad
  // --interval used to spin the watch loop; it is a usage error (exit 2) instead.
  const intervalAt = args.indexOf('--interval');
  const interval = intervalAt === -1 ? 60 : intFlagValue(args[intervalAt + 1], '--interval', { min: 1, max: MAX_WATCH_INTERVAL_SECONDS, example: 60 });
  const dryRun = args.includes('--dry-run');
  const full = args.includes('--full');
  const noPull = args.includes('--no-pull');
  const noBulk = args.includes('--no-bulk');
  const lanesAt = args.indexOf('--lanes');
  const lanes = args.includes('--no-lanes') ? 1 : lanesAt === -1 ? undefined : intFlagValue(args[lanesAt + 1], '--lanes', { min: 1, max: 16, example: 8 });
  let noEmbed = resolveNoEmbed(args, loadConfig());
  const noExtract = args.includes('--no-extract'); // v0.42.7 #1696
  const skipFailed = args.includes('--skip-failed');
  const retryFailed = args.includes('--retry-failed'), resetCheckpoint = args.includes('--reset-checkpoint');
  const noSchemaPack = args.includes('--no-schema-pack'); // v0.41.37.0 #1569
  const includeGitignored = args.includes('--include-gitignored');
  // Untracked-gap fix: --working-tree imports uncommitted working-tree state.
  // The config fallback (sync.include_working_tree) resolves inside
  // performSync so every caller honors it; the CLI passes undefined when the
  // flag is absent.
  const workingTree = args.includes('--working-tree') ? true : undefined;
  const syncAll = args.includes('--all');
  let missingPathMode: MissingPathMode = 'fail';
  try {
    missingPathMode = parseMissingPathMode(args);
  } catch (e) {
    console.error(e instanceof Error ? e.message : String(e));
    process.exit(2);
  }
  if (missingPathMode !== 'fail' && !syncAll) {
    // Single-source sync on a missing path should stay loud — an explicit
    // `--source X` naming an absent checkout is an operator error, not a
    // multi-machine artifact. Warn instead of silently ignoring the flag.
    console.error('[gbrain] WARN: --missing-path only applies to `sync --all`; ignored here.');
  }
  const jsonOut = args.includes('--json');
  const redactPaths = args.includes('--redact-paths');
  const yesFlag = args.includes('--yes');
  // v0.41.6.0 D3: lock-recovery flags. --break-lock (safe) verifies the
  // holder is local-host + (TTL-expired OR PID-dead+60s-old) before
  // deleting the row. --force-break-lock skips the liveness check. Both
  // are refused when combined with --all (per-source invocation required;
  // v0.40 lock keys are gbrain-sync:<sourceId>).
  const breakLock = args.includes('--break-lock');
  const forceBreakLock = args.includes('--force-break-lock');

  // v0.41.13.0 (T4 + T16) — --max-age <s>: age-gated lock break via
  // last_refreshed_at semantic (NOT acquired_at — D-V3-4). Only valid with
  // --break-lock; mutually exclusive with --force-break-lock (--force skips
  // every guard; --max-age is one specific extra guard so the two policies
  // can't coexist).
  const maxAgeStr = args.find((a, i) => args[i - 1] === '--max-age');
  let maxAgeSeconds: number | undefined;
  try {
    maxAgeSeconds = parseDurationSeconds(maxAgeStr, '--max-age');
  } catch (e) {
    console.error(e instanceof Error ? e.message : String(e));
    process.exit(1);
  }
  if (maxAgeSeconds !== undefined && !breakLock) {
    console.error(`--max-age is only valid with --break-lock.`);
    process.exit(1);
  }
  if (maxAgeSeconds !== undefined && forceBreakLock) {
    console.error(`--max-age cannot be combined with --force-break-lock (force skips all guards).`);
    process.exit(1);
  }
  return { repoPath, watch, interval, dryRun, full, noPull, noBulk, lanes, noEmbed, noExtract, skipFailed, retryFailed, resetCheckpoint, noSchemaPack, includeGitignored, workingTree, syncAll, missingPathMode, jsonOut, redactPaths, yesFlag, breakLock, forceBreakLock, maxAgeSeconds };
}

export type SyncFlags = ReturnType<typeof parseSyncFlags>;

/** Flags read after the `--break-lock` branch: fan-out, scope and timeout. */
export function parseSyncFanoutFlags(args: string[], syncAll: boolean) {
  // v0.40 D4+D18: parallel `sync --all` by default; --serial opts back to v1.
  // --no-auto-embed skips the per-source embed-backfill auto-enqueue.
  // --max-sources N caps fan-out (default min(sources.length, 8)).
  const serialFlag = args.includes('--serial');
  const noAutoEmbed = args.includes('--no-auto-embed');
  const maxSourcesStr = args.find((a, i) => args[i - 1] === '--max-sources');
  const maxSources = maxSourcesStr ? parseInt(maxSourcesStr, 10) : undefined;
  if (maxSourcesStr && (!Number.isFinite(maxSources!) || maxSources! < 1)) {
    console.error(`Invalid --max-sources value: "${maxSourcesStr}". Must be a positive integer.`);
    process.exit(1);
  }
  const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
  // #753/#774: monorepo subdir-source flags. --exclude is repeatable.
  const srcSubpath = args.find((a, i) => args[i - 1] === '--src-subpath') || undefined;
  const excludePatterns: string[] = [];
  for (let i = 0; i < args.length; i++) {
    if (args[i] === '--exclude' && i + 1 < args.length) excludePatterns.push(args[i + 1]);
  }
  // --include-hidden is repeatable, same parsing shape as --exclude.
  const includeHiddenPatterns: string[] = [];
  for (let i = 0; i < args.length; i++) {
    if (args[i] === '--include-hidden' && i + 1 < args.length) includeHiddenPatterns.push(args[i + 1]);
  }
  if (syncAll && (srcSubpath || excludePatterns.length > 0 || includeHiddenPatterns.length > 0)) {
    console.error(
      `--src-subpath/--exclude/--include-hidden scope a single sync invocation; they cannot be combined with --all. ` +
      `For --all runs, register the subdirectory as the source's local_path instead ` +
      `(gbrain sources add <id> --path <repo>/<subdir>), persist exclusions with ` +
      `\`gbrain config set sync.exclude <globs>\`, and persist the dot-directory waiver with ` +
      `\`gbrain config set sync.include_hidden <globs>\` so --all, autopilot and the dream cycle honor it.`,
    );
    process.exit(1);
  }
  const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
  const parallelStr = args.find((a, i) => args[i - 1] === '--parallel');
  // v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
  // of silently falling through to auto-concurrency or NaN. Loud failure beats
  // a 4-worker spawn from a typo. v0.40.3.0: same validation applies to --parallel.
  let concurrency: number | undefined;
  let parallelOverride: number | undefined;
  try {
    concurrency = parseWorkers(concurrencyStr);
  } catch (e) {
    console.error(e instanceof Error ? e.message : String(e));
    process.exit(1);
  }
  try {
    parallelOverride = parseWorkers(parallelStr);
  } catch (e) {
    console.error(e instanceof Error ? e.message : String(e));
    process.exit(1);
  }

  // v0.41.13.0 (T16 + T6) — --timeout <s>: graceful self-termination signal
  // threaded into performSync via SyncOpts.signal. D-V3-3 invariant: when
  // combined with --all, each source gets its OWN AbortController + countdown
  // inside runOne so the budget is per-source, not shared across the fan-out.
  //
  // Validation: --timeout requires --source OR --all. Bare `gbrain sync
  // --timeout 60` (no source scope) is rejected at parse time — the natural
  // single-source case requires the user to either name the source or opt
  // into the global fan-out, so the error message tells them which to add.
  const timeoutStr = args.find((a, i) => args[i - 1] === '--timeout');
  let timeoutSeconds: number | undefined;
  try {
    timeoutSeconds = parseDurationSeconds(timeoutStr, '--timeout');
  } catch (e) {
    console.error(e instanceof Error ? e.message : String(e));
    process.exit(1);
  }
  const explicitSourceArg = args.find((a, i) => args[i - 1] === '--source');
  if (timeoutSeconds !== undefined && !syncAll && !explicitSourceArg) {
    console.error(`--timeout requires either --source <id> or --all to scope the per-source budget.`);
    process.exit(1);
  }
  return { serialFlag, noAutoEmbed, maxSources: maxSources ?? parallelOverride, strategyArg, srcSubpath, excludePatterns, includeHiddenPatterns, concurrency, timeoutSeconds, explicitSourceArg };
}

export type SyncFanoutFlags = ReturnType<typeof parseSyncFanoutFlags>;