Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
sources: drop a removed source's managed-sync cursors
garrytana0828835d2cb
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1b542abd35..1cb5655dd9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -22,6 +22,7 @@ identifiers and attribution are available in the pre-removal Git revision- `stale_mentions`: each stale example now carries its own `gbrain unlink` command with the link type the row is stored under. The hint used to say `--link-type mentions` for every row, which removes nothing for a typed link stored under a verb such as `founded`. Reported by @Masashi-Ono0611 (#8817).- `schema_pack_consistency`: pages of the types gbrain itself writes (`extract_receipt` receipts under `extracts/`, and `code` pages from the code importer) no longer count as undeclared on a custom schema pack copied before the base pack declared them. Your own undeclared types still warn. Reported by @Masashi-Ono0611 (#8818).- `embedding_state` and `gbrain embed --stale`: a page whose body holds only a protected takes fence (which never reaches search chunks) is now counted `empty` instead of `chunkless_healable`. Doctor no longer warns about it, and `embed --stale` no longer plans a heal that can never produce a chunk. A page is treated this way only when the writer already chunked its current revision to nothing; a chunkless page that was never chunked, or an `embed_skip` page an older release stored without chunks, is still healed. Reported by @akinduroifedayo (#8502).+- `managed_sync_backlog`: removing a source in the middle of a managed sync now removes its sync cursor too. The cursor used to stay behind, and doctor warned about a stalled backlog forever with a resume command that failed `unknown_source`. A cursor an earlier removal already left behind is no longer reported. Reported by @omni-nordaware (#8741).## [0.60.620.0] - 2026-11-09diff --git a/docs/architecture/key-files/commands-2.md b/docs/architecture/key-files/commands-2.mdindex d0ac8cc66e..324c257aaa 100644--- a/docs/architecture/key-files/commands-2.md+++ b/docs/architecture/key-files/commands-2.md@@ -35,7 +35,7 @@ Doctor's embedding-dimension warning treats existing chunks as existing data eve- `src/commands/doctor/checks/{safe-index,self-capture,stale-embedding-effects}.ts` — `safe_index_pending` (BRAIN: re-sealable pages below the safe-chunk fence, names `gbrain repair safe-chunks`; kept code/image pages are informational), `self_capture` (BRAIN, #5413: corpus `.txt` files whose session id matches a gbrain claude-cli scratch transcript are classified, ones with no harness transcript left are unclassifiable; classification is `isClaudeCliSelfSessionId`; details carry `newest_capture_at`, a capped sample and copy-paste quarantine commands into `<corpus>.quarantine`, and the message says new captures stopped (#5820) so only the existing files need one quarantine; never moves or deletes), `stale_embedding_effects` (OPS, #5629: committed writes with a queued embedding effect untouched for an hour; names each effect with its inspect command; reported as unsupported). Unreadable state is warn with `details.health: 'unknown'`. `writer_version` (`checks/writer-version.ts`, owned by the persistence entry) is registered in the same table as operator-required. Tests: `test/doctor-self-capture.test.ts`, `test/recovery-layer.test.ts`.-- `src/commands/doctor/checks/managed-sync-backlog.ts` — #5984: lists unfinished managed sync cursors with remaining entries, the last drain's pages/min and indexing ETA (`persistence/sync-drain.ts` `readManagedSyncBacklog`); warns when a cursor has not advanced for an hour, with its resume command. Tests: `test/sync-drain.test.ts`.+- `src/commands/doctor/checks/managed-sync-backlog.ts` — #5984: lists unfinished managed sync cursors of registered sources with remaining entries, the last drain's pages/min and indexing ETA (`persistence/sync-drain.ts` `readManagedSyncBacklog`); warns when a cursor has not advanced for an hour, with its resume command. Tests: `test/sync-drain.test.ts`.- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit] [--no-migrate]`: health checks. `checkPostgresCancellationDriver` (`postgres_cancellation_driver`) inspects a real reserved runtime owner through the shared cancellation helper: unsupported `discard()` is a typed local failure with a repair link, connection/deadline failure is a warning, and PGLite is exempt. The bounded reserve releases late arrivals and does not make general engine connect require cancellation, preserving broken-install diagnostics. `--no-migrate` keeps doctor observational: the CLI connects with `probeOnly` so a clean-or-behind schema is reported as-is instead of being auto-migrated before the checks run. The file is a façade carrying `buildChecks` (flag parse into a `DoctorContext`, then `runDoctorRegistry`), `runDoctor` and output rendering; the checks run as the ordered registry in `src/commands/doctor/registry.ts` (see that entry), the check-function library lives in bundles under `src/commands/doctor/checks/` plus four tail-cluster modules under `src/commands/doctor/` (see that entry), all re-exported here so the full surface is unchanged — structural guards pin its source text via `test/helpers/doctor-source.ts`, never by reading this file alone. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`; a version AHEAD of this client's `LATEST_VERSION` warns "upgrade this client"; and when the ledger reads current, a read-only `detectMissingColumns` diff from `src/core/schema-verify.ts` runs INSIDE the ledger-current branch — a PgBouncer-swallowed ALTER TABLE can advance `config.version` over a physically narrower table, so missing live columns downgrade the ok to a warn naming them with the `gbrain init --migrate-only` hint; diff failure is best-effort and the ledger ok stands; positional wiring pinned by `test/doctor-schema-column-diff.test.ts`), `upgrade_errors` (async `checkUpgradeErrors(engine)` over `~/.gbrain/upgrade-errors.jsonl`: the warn downgrades to an EXPLICIT status-ok line — never silence — only when the past failure is provably superseded, i.e. the running binary is at/past the failed target version AND the schema verifies current via `engine.getConfig('version')`; a missing engine or unverifiable schema keeps the warn, fail-closed, because `self-upgrade` swaps the binary before `post-upgrade` runs migrations and the binary alone can lie; pinned by `test/doctor-upgrade-errors.test.ts` + `test/doctor-upgrade-errors-stale.test.ts`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE <col> IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](../path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check (`doctor/checks/supervisor-health.ts`) consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings on the doctor surface and in `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source <id>` matches. A source holding a LIVE, non-expired per-source sync lock (`inspectLock(engine, syncLockId(source.id))` from `src/core/db-lock.ts`) is reported as actively syncing (the message names the holder pid + host) and counted in `synced_recently_count`, NOT flagged stale — the live lock is the only honest in-progress signal (checkpoint banking can't distinguish in-progress from wedged: a blocked sync banks its files but writes no anchor). A blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either falls through to the stale path and is never masked; the dynamic `db-lock` import is swallowed to a no-op on a stub engine or pre-lock-table brain, so this can only ADD an in-progress verdict, never suppress a real stale one. The in-progress note is appended to whatever verdict the buckets produce and is empty when nothing is syncing, so steady-state messages stay byte-for-byte unchanged. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. the IRON-RULE guard banning stale verb names, the sync_freshness boundary matrix, the D4 guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). `pglite_data_dir` check: fs-only check that runs when a PGLite brain FAILS to connect (`!fastMode && !engine && config.engine === 'pglite'`, placed after `orphan_clones`, before the DB-checks gate; `pgliteDataDirEntry` in `doctor/checks/db-connection.ts`): `computePgliteDataDirCheck(dataDir, diagnosis)` (exported pure fn, `computeWorkerOomLoopCheck` convention) maps the `inspectPgliteDataDir` verdict to a Check — corruption-likely/looks-healthy-but-unopenable/unsupported-layout → `fail` naming `gbrain pglite-repair --dry-run`/`--yes` or the rebuild path, live-lock/missing-dir → `warn`; all `remediation_status: 'human_only'` (Minion remediation needs the DB that is down). Escalates when ≥2 repair attempts failed inside 7 days (unclean-shutdown genesis still active → engine-switch pointer) and reports retained backup-dir inventory (orphan_clones disk-visibility class). Registered in `doctor-categories.ts` OPS_CHECK_NAMES. Pinned by `test/doctor-pglite-datadir.test.ts`. silent-failure checks: `content_hash_duplicates` (single GROUP BY over `(source_id, content_hash)` with FILTER aggregates — never N² — flagging hash groups that hold BOTH a bare and a path-prefixed slug, the wrong-import-root pattern; warn carries sample pairs + the `pages delete` → `purge-deleted --older-than 0` remediation); `undeclared_db_only_pages` (per source with a local repo: markdown pages with no backing file outside every declared + derive-phase-default db_only prefix — the one check deliberately allowed to stat the repo; it finds each source's Git root once with `sourceGitScope` and passes it to `resolveSourceLocalFilePath` as `gitScope`; `connector_database` sources are skipped and counted in the message); `db_only_collector_collision` (configured recipe `output_paths` inside a declared db_only dir — auto-gitignore means sync AND import silently skip the collector's files; same warning fires in sync's `manageGitignore` at config-write time). All warn-level, engine-parity pinned by `test/e2e/doctor-silent-death-parity.test.ts`; units in `test/doctor-silent-death-checks.test.ts`. `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`. `subagent_capability` check (layer 3 of 3). Resolves subagent model config in runtime order (`models.subagent` > `models.default` > `models.tier.subagent` > built-in default) and warns when the selected model lacks native tool-loop capability (message names the bad value + paste-ready fix `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK when subagent tier resolves to Anthropic. Tests in `test/doctor.test.ts`. `computeWorkerOomLoopCheck(engine)` is the single authoritative OOM-loop signal, unioning supervised `summarizeCrashes(readRecentSupervisorEvents(24)).by_cause.rss_watchdog` (cross-week read via `readRecentSupervisorEvents` so a Monday window can't lose Sunday) + bare-worker `minion_jobs error_text='aborted: watchdog'` count (Postgres-only; the same source `queue_health` subcheck 3 reads). Cap comes from the latest `rss_watchdog_loop` breaker alert's `max_rss_mb`, else `resolveDefaultMaxRssMb()` fallback. fail at breaker-tripped or oomKills≥5, warn at ≥1, null otherwise. `computePoolReapHealthCheck(engine)` is the Postgres-only `pool_reap_health` check reading `readRecentPoolRecoveries(1)` — fail when reconnect failures>0 (reconnect throwing is the actionable signal), warn at ≥10 reaps/hr (pooler thrash), null otherwise. Both registered by the `supervisor` registry entry after the `supervisor` block. The `supervisor` causeStr carries `rss=N (see worker_oom_loop)` and `queue_health`'s watchdog message cross-references `worker_oom_loop`. `DoctorReport.top_issues` + the cause-ranked render header. `worker_oom_loop` + `pool_reap_health` registered under ops in `doctor-categories.ts`. Pinned by `test/doctor-worker-oom-loop.test.ts`, `test/doctor-pool-reap-health.test.ts`. `supervisor_singleton` check, a SEPARATE check from `supervisor` (same split precedent as the niceness check) so a singleton-divergence warn can't clobber the crash/liveness precedence. Runs only when a `started` supervisor event was seen in the last 24h and a live engine is available. Reads the queue-scoped DB lock row (`gbrain_cycle_locks WHERE id = supervisorLockId(queue)`) and compares the lock holder (`holder_host:holder_pid`) against the local pidfile holder via the pure `classifySupervisorSingleton`. `mismatch` → warn (a second supervisor may be running with a different `--max-rss`; message names both holders, the effective cap from the `started` event's `max_rss_mb`, and the fix `gbrain jobs supervisor stop`); `single` → ok (names holder + cap); `no_lock` → no check. PR #6463: `pidfileHost` (PID-file line 3) makes the same pid under a drifted hostname `single` → `ok` + `host_label_drift`; a two-line legacy file warns `same_pid_host_label_differs`; another pid keeps the warning emitted. Best-effort try/catch (silent skip on brains without the lock table). Registered under ops in `doctor-categories.ts` as `supervisor_singleton`. Pinned by `test/supervisor-db-lock.test.ts` + `test/doctor.test.ts`. `checkBatchRetryHealth`: `batch_retry_health` check surfacing Supavisor circuit-breaker incidents. Wired into both `runDoctor` (local) and `doctorReportRemote` (thin-client). Reads last 24h. States: `ok` (zero exhausted in 24h OR <3 from a single site), `warn` (>=3 same-site OR >=5 cross-site), `fail` (>=20 sustained breaker). Surfaces bad `GBRAIN_BULK_*` env at doctor startup. Corrupt-JSONL tolerant. Paste-ready fix hints in every warn/fail message. Also reads `readRecentDbDisconnects(24)` and appends `Disconnect-call audit: N call(s) in 24h (most recent caller: <frame>).` to ALL three message paths so connection-incident signal is greppable from one `gbrain doctor --json` call (module-import wrapped in try/catch so older brains without the audit file degrade silently). Pinned by `test/doctor-batch-retry.test.ts` (10 cases). three checks wired into `runDoctor()` and the JSON envelope, all warn-only with paste-ready fix hints. (1) `checkSourceRoutingHealth(engine)` scans up to 200 pages on federated brains and flags pages whose `source_id` doesn't match what `resolveSourceWithTier()` would have picked for their `source_path`; single-source brains short-circuit to `ok`; the 200-page cap is total across the brain so doctor stays under 5s. (2) `checkOauthConfidentialHealth(engine)` probes registered confidential clients for `/token` reachability. (3) `checkAutopilotLockScope()` (pure, no engine) compares the resolved lock path to `$GBRAIN_HOME`; warns when set but the lock lives elsewhere, with a PID-safe inspection hint (`kill -0 <pid>` before deletion). Pinned by `test/doctor-v0_37_7_checks.test.ts`. `buildChecks(engine, args, dbSource, connectError?): Promise<Check[]>` exported as a test seam; the optional `connectError` is the connect failure captured by the CLI's dead-DB fallback, which the null-engine path turns into a SYNTHESIZED classified `connection` fail entry (so `checks[name=="connection"]` exists in every failure shape — smoke-test branches on it). The `connection` check's failure path classifies via `src/core/pg-access-classify.ts` into a redacted message + `details: {reason, transient, fix_hint}` naming `gbrain db-repair` (deliberately NOT `remediation[]`/`makeRemediationStep` — that lane feeds `--remediate`, whose Minion jobs need the very DB that's down; applicator boundary: `--remediate` = brain-DATA quality with a live engine, `db-repair` = DB ACCESS engine-free). The URL-only `pgbouncerPrepareCheck` helper and the engine-free `db_repair_recurrence` check run BOTH before the connection check and in the dead-DB filesystem lane; `pglite_scale` (engine-fit, warn at ≥1000 pages on pglite) runs in the DB phase. `computeDoctorReport(checks, extras?)` accepts optional `{engine, db_url_source}` which land as additive `DoctorReport.engine` / `DoctorReport.db_url_source` JSON fields (schema_version stays 2). `runDoctor` is a thin wrapper: `buildChecks → computeDoctorReport → render + process.exit`. All 10 `process.exit` sites stay in the wrapper; the two early-return paths (no engine, connection failure) return partial check lists instead of inline exits (observable output identical). Pinned by `test/doctor-behavioral.test.ts` (13 cases: pure aggregation math over `computeDoctorReport`, orchestrator cases for `--fast` skip set + `--json` flag + no-engine partial path + snapshot of load-bearing check names) and `test/doctor-cli-smoke.test.ts` (1 subprocess case spawning `bun run src/cli.ts doctor --json` against a fresh PGLite tempdir, asserting schema_version=2 envelope, status enum, non-empty checks array — the render-path coverage buildChecks-only tests miss; quarantined `.serial` because PGLite write-locks don't play with parallel runners). three checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on live DB pages matching any junk pattern that escaped ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; `--content-audit` opts into a full scan. All three warn-only with paste-ready fix hints (junk → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept). two checks wired into `runDoctor()` + the JSON envelope: `quarantined_pages` (counts pages carrying the `quarantine` marker via `engine.executeRaw` JSONB `?` existence, works on PGLite + Postgres; warn-only with a `gbrain quarantine list` hint) and `flagged_pages` (counts `content_flag` pages — searchable but odd; warn-only). Both skip gracefully (status ok, "Skipped") on engines/brains where the probe errors. Pinned by `test/doctor.test.ts`. `home_dir_in_worktree`: filesystem check walking up from `gbrainPath()` toward `$HOME` looking for a `.git` directory (main repo) or `.git` file (linked worktree pointer; Conductor + git-worktrees topology). Walk terminates at `$HOME` so a `.git` above the user's home doesn't false-positive. Honors `GBRAIN_HOME` (appends `.gbrain` to the override). Warn (not fail) with worktree-root path + paste-ready fix pointing at `GBRAIN_HOME` override or moving the brain. `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`. 4 checks: `abandoned_threads`, `calibration_freshness`, `grade_confidence_drift` (mitigation surface; math ships later), `voice_gate_health`. Schema health is a two-check pair: `schema_version` resolves through `schemaVersionHealth` (`src/core/schema-version-health.ts`, shared with the remote doctor report; a DB AHEAD of the client warns with an upgrade-this-client hint rather than suggesting migrations), and on ledger-current brains a read-only `schema_columns` check diffs live columns against the expected schema (`detectMissingColumns`, dynamically imported inside the ledger-current branch — positional guard in `test/doctor-schema-column-diff.test.ts`) and warns naming the missing columns with the `gbrain init --migrate-only` hint.Size ceiling: `scripts/module-size-limits.tsv` (per-change raise history: `git log -p` on that file). The ceiling covers the façade: re-exports, `Check`/`DoctorReport` and report scoring, the `buildChecks` flag parse, `runDoctor`, output rendering and `--locks`; every check block lives in a registry entry under `doctor/checks/*` (refactor wave 1 W4 doctor), remediation in `doctor/remediate.ts`, wave checks in `doctor/wave-checks.ts`.diff --git a/docs/architecture/key-files/files-and-sync-1.md b/docs/architecture/key-files/files-and-sync-1.mdindex 5f22b705a0..c9abba374a 100644--- a/docs/architecture/key-files/files-and-sync-1.md+++ b/docs/architecture/key-files/files-and-sync-1.md@@ -51,7 +51,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/source-resolver.ts` — 6-tier source resolution. `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside `resolveSourceId()` (unchanged). `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'sole_non_default', 'brain_default', 'seed_default']` (7 entries; order matches priority). Tier `sole_non_default` slots between `local_path` and `brain_default`: when NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'` AND the `'default'` source holds no active pages (the emptiness guard — the tier's charter is rescuing brains whose `'default'` is empty; when `'default'` carries an established corpus, auto-routing would hijack every bare write into the side-source, so the resolver falls through to `seed_default` and prints a one-line stderr notice naming both sides so the suppressed flip is diagnosable, same `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1` suppression knob as the routing nudge; a failed pages probe on an exotic/legacy schema keeps the auto-route), auto-route to it; archived sources excluded (try/catch for pre-v34 brains); private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points so they cannot drift. Exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `resolveCliSyncSource` in `src/commands/sync/run.ts` calls `resolveSourceWithTier` unconditionally so the tier fires; `src/commands/import.ts:96-128` mirrors with the tier-gated nudge. Consumed by `gbrain sources current`, `import --source-id`, `extract --source-id`, and the `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (`withEnv()` per test-isolation lint), `test/source-resolver-sole-non-default.test.ts` (20 cases incl. both-ways emptiness-guard + probe-failure fallback), `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync`), `test/pages-source-scoping-4329.test.ts` (real-engine both-ways guard coverage). Also exports `noGrantFederatedScope(engine, hasSourceGrant, sourceId)`, the widening decision transports share: returns `localFederatedSourceIds(..., 'seed_default')` ONLY when `hasSourceGrant === false` (the legacy-bearer no-grant floor), `undefined` for a granted token, for an OAuth client (flag `undefined` — a falsy gate here would widen every OAuth client), and when the resolver throws (best-effort; the scalar scope stands rather than failing the request). Called by `src/commands/serve-http.ts`; pinned by `test/no-grant-federated-scope.test.ts` (6 cases). `explicitReadBinding(engine, sourceId, tier)` (#5081) is the explicit-tier twin used by the stdio transport for `GBRAIN_SOURCE` (`env`) and `.gbrain-source` (`dotfile`) bindings: `{ sourceId, via, sourceIds, optedOut }`, where `sourceIds` is the same set `localFederatedSourceIds` computes (shared `federatedSetFor`, so an isolated anchor admits only itself) and `optedOut` lists `federated === false` sources for the denial hint; `undefined` for every other tier. Pinned by `test/explicit-read-admission-5081.test.ts`. Unscoped default-write guard family (the `seed_default` tier is the only one that actually lands a write in `'default'`): `assessDefaultWriteGuard(engine)` → `DefaultWriteAssessment { shouldGuard, defaultPages, nonDefaultPages, nonDefaultSources, failed? }` — an unindexed full-`pages` aggregate; `shouldGuard` when ≥1 non-default source holds pages AND non-default pages outnumber default's; a query failure returns the fail-open no-guard verdict stamped `failed: true`, which cachers/latchers must not treat as settled. `assessDefaultWriteGuardOnce(engine)` is the per-engine WeakMap memo (arms on the first SUCCESSFUL assessment whatever its verdict, forgets a failed attempt so the next caller retries, coalesces concurrent callers; `__resetDefaultWriteGuardMemo` is the test seam). `assessUnscopedDefaultWrite(engine, tier, mutating)` → `{ warning, assessed }` (`assessed: false` only when the aggregate failed); `formatDefaultWriteWarning(a, sourceFlag?)` / `formatDefaultWriteRefusal(cmd, a)` render the two surfaces; `defaultWriteAllowedByEnv()` is the `GBRAIN_ALLOW_DEFAULT_WRITE=1` escape hatch. Consumers: `sync` refuses (dry-run warns), `import` warns via the memo, MCP stdio's `createDefaultWriteAdvisory` prints once. Also exports `isResolverUserError(e)` — the ONE predicate for this module's user-facing throws (`Source "…" not found.` / `not found or is archived.`, `Invalid --source value`, `Invalid GBRAIN_SOURCE value`), consumed by dream.ts, agent.ts and code-scope.ts so a CLI command turns an unknown/archived source into a clean stderr line + exit 1 while anything else keeps propagating. A request snapshot serves tiers 4, 5.5 (its rows carry the 'default' emptiness probe, `default_has_pages`), `assertSourceExists` and the federated set from one sources read, sent with the config table (`resolveSourceWithTier` reads tier 5's key beside the tier-4 listing) (GBRA-156).-- `src/core/source-delete.ts` — `deleteSourceRow`: every unmanaged source delete (`sources remove`, `sources purge`, the `sources_remove` op, expired-archive purge) deletes the `sources` row and that incarnation's `persistence_source_bindings` row in one statement; bindings have no FK to `sources`. Bindings of other incarnations are left to `gbrain repair orphan-bindings` (`src/core/persistence/orphan-bindings.ts`, doctor `orphan_persistence_bindings`). The unmanaged claim check (`sync-authority.ts`) and doctor `unbound_source` count only a binding of the source's current incarnation. `EXPIRED_ARCHIVE_SQL` is the one purge-eligibility predicate (unmanaged sweep, its per-row DELETE and the managed `expiredOnly` purge): both archive timestamps must be later than the epoch and the expiry past; `UNVERIFIED_ARCHIVE_SQL` lists archived rows with a missing or epoch timestamp, which the sweep reports blocked with restore-then-archive guidance instead of purging.+- `src/core/source-delete.ts` — `deleteSourceRow`: every unmanaged source delete (`sources remove`, `sources purge`, the `sources_remove` op, expired-archive purge) deletes the `sources` row, that incarnation's `persistence_source_bindings` row and the source's `managed-sync` cursors in `op_checkpoints` (#8741) in one statement; neither has an FK to `sources`. Bindings of other incarnations are left to `gbrain repair orphan-bindings` (`src/core/persistence/orphan-bindings.ts`, doctor `orphan_persistence_bindings`). The unmanaged claim check (`sync-authority.ts`) and doctor `unbound_source` count only a binding of the source's current incarnation. `EXPIRED_ARCHIVE_SQL` is the one purge-eligibility predicate (unmanaged sweep, its per-row DELETE and the managed `expiredOnly` purge): both archive timestamps must be later than the epoch and the expiry past; `UNVERIFIED_ARCHIVE_SQL` lists archived rows with a missing or epoch timestamp, which the sweep reports blocked with restore-then-archive guidance instead of purging.- `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. `assertNoOverlappingPath(engine, id, path)` is the ONE overlapping-path guard shared by every surface that binds a `local_path` (`addSource`, `gbrain sources set-path`): a path equal to, nested inside, or enclosing another source's `local_path` throws `SourceOpError('overlapping_path')` — overlapping trees make sync/write-through attribute files to the wrong source, so no add or repair path may skip it. The guard compares by spelling AND by realpath (both the candidate and each sibling `local_path` are `realpathSync`'d when they exist, fail-open to the spelling otherwise), so a symlink resolving into or over another source's tree cannot slip past it; the stored pointer and the error message keep the operator's spelling. #6946: `isOwnedClone`/`defaultCloneDir` live in `src/core/url-clone.ts`, re-exported here.- `src/core/sync-concurrency.ts` — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the sites can't drift. `DEFAULT_PARALLEL_SOURCES = 4` is a SEPARATE constant for the per-source fan-out under `gbrain sync --all` — kept distinct from `DEFAULT_PARALLEL_WORKERS` because total live Postgres connections per wave ≈ `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-file pool)` = 32 at both defaults (each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`); `sync.ts` warns when `parallel × workers × 2 > 16`. `resolveWorkersWithClamp(engine, override, commandName, fileCount)` wraps `autoConcurrency` with a per-command stderr clamp warning on PGLite (per-(command, requested) dedup via module-scoped warned-once set with `_resetWorkersClampWarningsForTest()` seam) and is the canonical surface for every bulk-command `--workers N` flag (extract-conversation-facts, extract, edges-backfill, reindex-multimodal, reindex, reindex-code); embed.ts deliberately bypasses it and keeps `GBRAIN_EMBED_CONCURRENCY || 20`. `resolveMaxConnections()` (reads `GBRAIN_MAX_CONNECTIONS`, undefined when unset) + `clampWorkersForConnectionBudget(workers, perWorkerPool, maxConnections, parentPool)` back the opt-in single-sync connection-footprint clamp so a big sync stays under a low pooler cap (`parent_pool + workers×perWorkerPool ≤ budget`); `gbrain doctor`'s `pool_budget` check (`computePoolBudgetCheck` / `checkPoolBudget` in `src/commands/doctor/checks/consolidation-cycle.ts`, re-exported by `src/commands/doctor.ts`) warns when the budget leaves no room for a worker, pointing at `GBRAIN_POOL_SIZE=2`. Pinned by `test/pglite-workers-clamp.test.ts`.diff --git a/src/core/persistence/sync-drain.ts b/src/core/persistence/sync-drain.tsindex a4bbee73d9..62852f24e8 100644--- a/src/core/persistence/sync-drain.ts+++ b/src/core/persistence/sync-drain.ts@@ -1234,8 +1234,9 @@ export interface ManagedSyncBacklog {}/**- * Unfinished managed cursors with their remaining entries and the rate of- * their latest drain window (CEO-A18), readable from any process.+ * Unfinished managed cursors of registered sources with their remaining+ * entries and the rate of their latest drain window (CEO-A18), readable from+ * any process. A cursor left by a removed source (#8741) is not a backlog.* #6712 PR 4b (design v3.2 §3.7): plus every source whose pass row is not* `nothing_pending` (`passHealth`: converged at the checkout's HEAD and the* newest pass ended `synced` or `up_to_date`); a source's pass entry replaces@@ -1244,7 +1245,8 @@ export interface ManagedSyncBacklog {export async function readManagedSyncBacklog(engine: BrainEngine, sourceIds?: string[]): Promise<ManagedSyncBacklog[]> {const rows = await engine.executeRaw<{ header: { sourceId: string; index: number; total: number; done?: boolean; discoveredAt?: string; progress?: { startedAt: number; startIndex: number; lastAt: number; lastIndex: number };counts?: { held?: number }; processingOptions?: { noEmbed?: boolean; noExtract?: boolean; noSchemaPack?: boolean }; syncOptions?: Parameters<typeof managedSyncResumeArgs>[0]['syncOptions'] } }>(- `SELECT completed_keys->0 AS header FROM op_checkpoints WHERE op='managed-sync' AND COALESCE(completed_keys->0->>'done','false')<>'true'`);+ `SELECT completed_keys->0 AS header FROM op_checkpoints WHERE op='managed-sync' AND COALESCE(completed_keys->0->>'done','false')<>'true'+ AND EXISTS (SELECT 1 FROM sources s WHERE s.id = completed_keys->0->>'sourceId')`);const { readPassReadings, passResumeArgs, passStuck, passStuckSince } = await import('./sync-pass-readers.ts');const passes: ManagedSyncBacklog[] = (await readPassReadings(engine, { sourceIds })).filter(r => !r.health.nothing_pending).map(({ source_id, row, head, health }) => {const t = row.value.telemetry, args = passResumeArgs(source_id, row), stuck = passStuck(health) ? health.reason as 'blocked' | 'not_converged' | 'settle_mismatch' : null;diff --git a/src/core/source-delete.ts b/src/core/source-delete.tsindex 288a7d00ab..b4b6806964 100644--- a/src/core/source-delete.ts+++ b/src/core/source-delete.ts@@ -17,7 +17,9 @@ export const unverifiedArchiveReason = (id: string) =>* binding (#5732). `persistence_source_bindings` has no FK to `sources`, so a* delete that leaves the binding would make a same-id replacement read as* claimed. A binding of another incarnation is never touched here; `gbrain- * repair orphan-bindings` removes those. Returns whether a row was deleted.+ * repair orphan-bindings` removes those. The source's managed-sync cursors+ * (`op_checkpoints`, also no FK) go too (#8741): nothing can resume them, and doctor `managed_sync_backlog` would warn with a resume command+ * that fails `unknown_source`. Returns whether a row was deleted.*/export async function deleteSourceRow(engine: Pick<BrainEngine, 'executeRaw'>,@@ -30,7 +32,9 @@ export async function deleteSourceRow(const rows = await engine.executeRaw<{ id: string }>(`WITH gone AS (DELETE FROM sources WHERE id = $1 ${expired} RETURNING id, incarnation),unbound AS (DELETE FROM persistence_source_bindings b USING gone- WHERE b.source_id = gone.id AND b.source_incarnation = gone.incarnation)+ WHERE b.source_id = gone.id AND b.source_incarnation = gone.incarnation),+ cursors AS (DELETE FROM op_checkpoints c USING gone+ WHERE c.op = 'managed-sync' AND c.completed_keys->0->>'sourceId' = gone.id)SELECT id FROM gone`,[id],);diff --git a/test/doctor-managed-sync-backlog-6763.test.ts b/test/doctor-managed-sync-backlog-6763.test.tsindex eabb479864..50c09b2507 100644--- a/test/doctor-managed-sync-backlog-6763.test.ts+++ b/test/doctor-managed-sync-backlog-6763.test.ts@@ -36,8 +36,11 @@ beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); aafterAll(async () => { await engine.disconnect(); });test('#6763 a stored cursor: a fresh row write never hides an old discovery, and a done cursor is not a backlog', async () => {- const put = (fingerprint: string, header: Record<string, unknown>) => engine.executeRaw(- "INSERT INTO op_checkpoints(op, fingerprint, completed_keys, updated_at) VALUES('managed-sync', $1, jsonb_build_array($2::jsonb), now())", [fingerprint, JSON.stringify(header)]);+ const put = async (fingerprint: string, header: Record<string, unknown>) => {+ await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING', [header.sourceId]);+ await engine.executeRaw(+ "INSERT INTO op_checkpoints(op, fingerprint, completed_keys, updated_at) VALUES('managed-sync', $1, jsonb_build_array($2::jsonb), now())", [fingerprint, JSON.stringify(header)]);+ };await put('fresh', { sourceId: 'fresh-example', index: 0, total: 0, done: false, discoveredAt: minutesAgo(0) });expect(await checkManagedSyncBacklog(engine, ['fresh-example'])).toMatchObject({ status: 'ok' });await put('old', { sourceId: 'old-example', index: 0, total: 4, done: false, discoveredAt: minutesAgo(120) });diff --git a/test/source-delete-managed-cursor-8741.test.ts b/test/source-delete-managed-cursor-8741.test.tsnew file mode 100644index 0000000000..de6b1412da--- /dev/null+++ b/test/source-delete-managed-cursor-8741.test.ts@@ -0,0 +1,38 @@+/**+ * #8741: op_checkpoints has no FK to sources, so removing a source mid-sync+ * left its unfinished managed-sync cursor behind. Doctor managed_sync_backlog+ * then warned forever with a resume command that fails `unknown_source`, and+ * the managed-sync-orphans repair (which joins sources) never saw the row.+ * The cursor now goes with the source row, and the backlog reader skips a+ * cursor whose source no longer exists.+ */+import { afterAll, beforeAll, expect, test } from 'bun:test';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { checkManagedSyncBacklog } from '../src/commands/doctor/checks/managed-sync-backlog.ts';+import { deleteSourceRow } from '../src/core/source-delete.ts';++let engine: PGLiteEngine;+const twoHoursAgo = () => new Date(Date.now() - 120 * 60_000).toISOString();+const putCursor = (fingerprint: string, sourceId: string) => engine.executeRaw(+ "INSERT INTO op_checkpoints(op, fingerprint, completed_keys, updated_at) VALUES('managed-sync', $1, jsonb_build_array($2::jsonb), now())",+ [fingerprint, JSON.stringify({ sourceId, index: 1, total: 4, done: false, discoveredAt: twoHoursAgo() })]);++beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 60_000);+afterAll(async () => { await engine.disconnect(); });++test('removing a source drops its unfinished managed-sync cursor', async () => {+ await engine.executeRaw("INSERT INTO sources (id, name) VALUES ('source-a', 'source-a'), ('source-b', 'source-b')");+ await putCursor('cursor-a', 'source-a');+ await putCursor('cursor-b', 'source-b');+ expect(await checkManagedSyncBacklog(engine, ['source-a'])).toMatchObject({ status: 'warn' });+ expect(await deleteSourceRow(engine, 'source-a')).toBe(true);+ expect(await engine.executeRaw("SELECT 1 FROM op_checkpoints WHERE op = 'managed-sync' AND fingerprint = 'cursor-a'")).toHaveLength(0);+ expect(await engine.executeRaw("SELECT 1 FROM op_checkpoints WHERE op = 'managed-sync' AND fingerprint = 'cursor-b'")).toHaveLength(1);+ expect(await checkManagedSyncBacklog(engine, ['source-a'])).toBeNull();+});++test('a cursor already left behind by a removed source is not a backlog', async () => {+ await putCursor('cursor-gone', 'source-gone');+ expect(await checkManagedSyncBacklog(engine, ['source-gone'])).toBeNull();+ expect((await checkManagedSyncBacklog(engine))?.message ?? '').not.toContain('source-gone');+});