Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Rebuild pages out of search first, 4 at once on Postgres, after an upgrade from before v0.51 (#9192) (#9487)
Garry Tan044364a1ce14
diff --git a/changelog.d/capy-parallel-projection-drain-9192.md b/changelog.d/capy-parallel-projection-drain-9192.mdnew file mode 100644index 0000000000..5bcca5b043--- /dev/null+++ b/changelog.d/capy-parallel-projection-drain-9192.md@@ -0,0 +1,10 @@+---+important: regression+---+**After an upgrade from before v0.51, `gbrain projections drain` rebuilds the pages that are out of search first, and on Postgres it rebuilds 4 pages at once by default, so every page answers search much sooner.**++### Fixed+- The search-index rebuild that an upgrade from before v0.51 starts used to work through pages in queue order, so pages with a facts fence could wait hours behind pages that were still answering from their old index. `gbrain projections drain` now rebuilds the pages that are out of search first, then seals images, then rebuilds the pages still answering from their pre-upgrade index. `--dry-run` lists rows in the same order. A follow-up to #9192.+- On Postgres the drain rebuilds several pages at once, each on its own pooled connection: 4 by default, or fewer when the pool is small, and `--concurrency N` goes up to the pool size less 2. On a 50,000-page test brain with 30,000 fenced pages behind an 18 ms round trip, every page was searchable after about 86 minutes at the default. PGLite, `gbrain serve` on PGLite and the drain inside `gbrain migrate --to` still rebuild one page at a time.+- A refused or dropped extra connection lowers the width by one and the drain carries on; stderr names the new width. It never fails the drain.+- The upgrade's `[153]` line gives the rebuild time at the default width, names the faster `--concurrency` command with its time, and counts only the Markdown and code pages the drain rebuilds, not images.diff --git a/docs/architecture/key-files/commands-4-continued.md b/docs/architecture/key-files/commands-4-continued.mdindex d62f34f09c..01ad522703 100644--- a/docs/architecture/key-files/commands-4-continued.md+++ b/docs/architecture/key-files/commands-4-continued.md@@ -20,7 +20,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/commands/reindex-aliases.ts` — `gbrain reindex --aliases [--limit N] [--dry-run] [--json] [--source <id>]`: backfills the free-text alias layer for EXISTING pages whose frontmatter `aliases:` predate the alias table (the import-time projection covers new + changed pages). Reads each page's frontmatter `aliases:` (written via `engine.setPageAliases`) and, for linkable entity pages, rewrites the derived title-subject and declared rows (`writeDerivedAliases`, coordinated on managed brains). Idempotent + convergent (setPageAliases replaces a page's alias set) so no op-checkpoint needed; walks `listAllPageRefs` (cheap cross-source enumeration), `--source` narrows. A segmented linkable page derives from its body read back as strings (`materializedPage`); one over the single-string ceiling gets frontmatter and subject aliases only, as its streamed import did, and is listed in `body_not_scanned`; a per-page `page_body_segmented` refusal is listed in `skipped`. Pinned by `test/search/reindex-aliases.test.ts`.-- `src/commands/projections.ts` + `src/cli/commands/projections.ts` — `gbrain projections drain [--dry-run] [--limit <n>] [--json]` (#5401). Trusted-local CLI, not an operation. Rebuilds queued Markdown and code projections in-process in batches of up to 100 until every row queued at the run start has been tried once (it waits for the database clock to pass the run start, so a page that fails in the run is not retried), then (#6803, `media: true`) seals the `sealable` image pages through `page-state/image-seal.ts`; `--limit` caps pages tried. `--dry-run` prints every class count and the rows a drain takes (`formatDrainPreview`); the summary (`formatDrainResult`) lists the classes left with their next step and never says the backlog is empty while any remain; `--json` is the full `ProjectionDrainResult`. Under a resident it asks the resident for its counts (`projection_status`) and `residentDrainAction` builds doctor's two-step stop/drain/restart Action (`requires_exclusive` on the drain). Exit 0 (nothing tried failed), 1 (a page failed; each failure prints source, slug, reason and next action), 2 (did not run: bad usage, or `projection_owner_resident` when a process holds the selected PGLite brain, checked with `persistenceConfigForBrain` before any engine opens, so a mounted PGLite brain on a Postgres host is refused too). On Postgres it runs beside a resident owner (per-page locks plus the revision recheck). `residentDrainSteps` gives the stop-drain-restart commands for the serve service, autopilot and a manual serve, shared with doctor. Tests: `projection-drain.test.ts` (both engines via `test/e2e/projection-drain-postgres.test.ts`), `projections-drain-cli.serial.test.ts`, `projection-image-drain.test.ts` (both engines via `test/e2e/projection-image-drain-postgres.test.ts`).+- `src/commands/projections.ts` + `src/cli/commands/projections.ts` — `gbrain projections drain [--dry-run] [--limit <n>] [--concurrency <n>] [--json]` (#5401). `--concurrency` (#9192, Postgres only) sets how many pages rebuild at once on their own pooled connections: default from `drainConcurrency` in `page-state/rebuild-estimate.ts` (4, fewer on a small pool), at most the pool size less 2, refused above that; a refused or dropped connection narrows the width by one (stderr names it) without failing the drain, and PGLite rebuilds one page at a time. The drain order is pending rows search does not answer for, then image seals, then pending rows still answering from their pre-upgrade index. Trusted-local CLI, not an operation. Rebuilds queued Markdown and code projections in-process in batches of up to 100 until every row queued at the run start has been tried once (it waits for the database clock to pass the run start, so a page that fails in the run is not retried), then (#6803, `media: true`) seals the `sealable` image pages through `page-state/image-seal.ts`; `--limit` caps pages tried. `--dry-run` prints every class count and the rows a drain takes (`formatDrainPreview`); the summary (`formatDrainResult`) lists the classes left with their next step and never says the backlog is empty while any remain; `--json` is the full `ProjectionDrainResult`. Under a resident it asks the resident for its counts (`projection_status`) and `residentDrainAction` builds doctor's two-step stop/drain/restart Action (`requires_exclusive` on the drain). Exit 0 (nothing tried failed), 1 (a page failed; each failure prints source, slug, reason and next action), 2 (did not run: bad usage, or `projection_owner_resident` when a process holds the selected PGLite brain, checked with `persistenceConfigForBrain` before any engine opens, so a mounted PGLite brain on a Postgres host is refused too). On Postgres it runs beside a resident owner (per-page locks plus the revision recheck). `residentDrainSteps` gives the stop-drain-restart commands for the serve service, autopilot and a manual serve, shared with doctor. Tests: `projection-drain.test.ts` (both engines via `test/e2e/projection-drain-postgres.test.ts`), `projections-drain-cli.serial.test.ts`, `projection-image-drain.test.ts` (both engines via `test/e2e/projection-image-drain-postgres.test.ts`).- `src/commands/reindex-code.ts` — `gbrain reindex-code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks live `page_kind = 'code'` pages in source/slug order and rebuilds derived projections without rewriting canonical files, revisions or hashes. `--force` rebuilds already-current projections. Managed brains use a guarded journal receipt; resident PGLite delegates through the authenticated local administration route in `reindex-code-delegate.ts`. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). Consent (C2): the cost preview feeds `consentGate` (effect `paid`); `--yes`, `--max-cost <usd>`, a preapproval or tokenmax authorize it, otherwise exit 3 with the consent payload. Without a user cap the run gets the derived cap, except under tokenmax (`tokenmaxUncappedEnv`: stays uncapped); `--max-cost off` is the explicit uncapped choice. The owner-delegated path asks the same way (default cap). Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE guard for the cost-preview model source). --no-embed is the keyless repair path. Missing vectors remain available to the normal embed-stale workflow. Without managed persistence, `reindexCodeProjection` (`src/core/persistence/projection-reindex.ts`) re-reads and re-prepares a page whose install lost to another installer (skipped when it is now sealed and `--force` is off); a conflict that outlasts the retries is a per-page failure carrying `[page_projection_conflict]` and the re-run step (`test/page-projection-code-recovery.test.ts`).- `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. `reindexFrontmatterCli(engine, args)` takes the ALREADY-CONNECTED engine from cli.ts's dispatch; it must never build/connect its own engine — a second connect on the same PGLite data dir self-deadlocks on the data-dir lock (this process already holds it). Same rule applies to `runBackfillCommand(engine, args)` in `src/commands/backfill.ts` and any future command dispatched from cli.ts's engine-connected switch. Pinned by `test/reindex-frontmatter-connect.test.ts` (library path) and `test/reindex-frontmatter-pglite-spawn.test.ts` (CLI dispatch seam, both commands). Consent (agent operator wave C3): every run first previews one selection (the rows whose stored `effective_date`/`effective_date_source` would change, scoped by `--source` and `--slug-prefix` through `backfillEffectiveDate`'s `sourceId` filter) and hashes it; the dry run reports that `plan_hash`. An apply asks through `requireConsent` (effect `destructive`, bound to the plan: `--yes --expect <plan_hash>`; a changed row is `preview_changed`) and writes only the approved ids (`onlyIds`). `--json` never implies consent: non-interactive without approval changes nothing, returns `status: 'confirmation_required'` and exits 3 with the consent payload. Scoped walks never read or write the orchestrator's brain-wide checkpoint. Pinned by `test/reindex-frontmatter-consent.test.ts` (two-source scope, binding).diff --git a/docs/guides/repair.md b/docs/guides/repair.mdindex 8ab18d6a9d..a894af037e 100644--- a/docs/guides/repair.md+++ b/docs/guides/repair.md@@ -695,8 +695,13 @@ gbrain projections drain --dry-run # change nothing: every class count and theand the rows a drain with the same `--limit` (default 50) would take, oldestfirst, each with its queue reason (`rebuild_failed` for a row whose lastrebuild failed); `--json` gives the class counts and `rows`. The drain-rebuilds the `pending` rows first, then seals the `sealable` images in place:-it rechecks each page's class under its page lock, writes no chunk and leaves+rebuilds the `pending` rows search does not answer for first, then seals the+`sealable` images, then rebuilds the `pending` rows search still answers for from+the index they had before an upgrade (#9192). On Postgres it rebuilds 4 pages at+once, each on its own pooled connection; `--concurrency` raises that to the pool+size less 2, and a refused or dropped connection lowers it without stopping the+drain. PGLite and the resident owner rebuild one page at a time. Sealing an+image in place rechecks each page's class under its page lock, writes no chunk and leavesthe visual vector as it is, so image search ranks the page again at once. A`sealed_text_only` page (imported with `--no-embed`) gets its visual vectorfrom `gbrain embed --stale --images`. `reimport`, `protected` and `archived`diff --git a/src/cli/commands/projections.ts b/src/cli/commands/projections.tsindex d6657721fb..7c038c0d01 100644--- a/src/cli/commands/projections.ts+++ b/src/cli/commands/projections.ts@@ -22,11 +22,14 @@ export async function run(args: string[], ctx: CliDispatchContext): Promise<void};const limitAt = args.indexOf('--limit');const limitValue = limitAt === -1 ? undefined : args[limitAt + 1];- const known = new Set(['drain', '--json', '--dry-run', '--limit', ...(limitValue === undefined ? [] : [limitValue])]);+ const concurrencyAt = args.indexOf('--concurrency');+ const concurrencyValue = concurrencyAt === -1 ? undefined : args[concurrencyAt + 1];+ const known = new Set(['drain', '--json', '--dry-run', '--limit', '--concurrency', ...(limitValue === undefined ? [] : [limitValue]),+ ...(concurrencyValue === undefined ? [] : [concurrencyValue])]);const unknown = args.find(arg => !known.has(arg));if (args[0] !== 'drain' || unknown !== undefined) {await refuse(new OperationError('invalid_params', `Unknown projections argument '${unknown ?? args[0]}'.`,- 'Run `gbrain projections drain [--limit <n>] [--json]`.', projections.PROJECTION_DRAIN_DOCS));+ 'Run `gbrain projections drain [--limit <n>] [--concurrency <n>] [--json]`.', projections.PROJECTION_DRAIN_DOCS));return;}const limit = limitValue === undefined ? undefined : Number(limitValue);@@ -35,6 +38,12 @@ export async function run(args: string[], ctx: CliDispatchContext): Promise<void'Run `gbrain projections drain --limit 1000`, or omit --limit to drain everything queued.', projections.PROJECTION_DRAIN_DOCS));return;}+ const concurrency = concurrencyValue === undefined ? undefined : Number(concurrencyValue);+ if (concurrencyAt !== -1 && (!Number.isSafeInteger(concurrency) || concurrency! < 1)) {+ await refuse(new OperationError('invalid_params', `--concurrency needs a positive whole number of pages, not '${concurrencyValue ?? ''}'.`,+ 'Run `gbrain projections drain --concurrency 4`, or omit it for the default.', projections.PROJECTION_DRAIN_DOCS));+ return;+ }const { loadConfig } = await import('../../core/config.ts');const { getCliOptions } = await import('../../core/cli-options.ts');@@ -70,7 +79,16 @@ export async function run(args: string[], ctx: CliDispatchContext): Promise<voidreturn;}try {- const result = await projections.drainProjections(engine, { limit, media: true, progress: createProgress(cliOptsToProgressOptions(getCliOptions())) });+ const { drainConcurrency } = await import('../../core/page-state/rebuild-estimate.ts');+ const width = drainConcurrency(engine);+ if (concurrency !== undefined && concurrency > width.max) {+ await refuse(new OperationError('invalid_params', `--concurrency ${concurrency} is more than this database connection pool allows (${width.max}: the pool size less 2 kept free).`,+ width.max > 1 ? `Run \`gbrain projections drain --concurrency ${width.max}\`, or raise the pool size.` : 'PGLite rebuilds one page at a time: omit --concurrency.', projections.PROJECTION_DRAIN_DOCS));+ return;+ }+ const result = await projections.drainProjections(engine, { limit, media: true, progress: createProgress(cliOptsToProgressOptions(getCliOptions())),+ concurrency: concurrency ?? width.default,+ onNarrow: (n, error) => console.error(`[gbrain] A database connection was refused or dropped (${(error instanceof Error ? error.message : String(error)).split('\n')[0]!.slice(0, 200)}); rebuilding ${n} page(s) at once from here.`) });if (json) {await writeStdoutFinal(JSON.stringify(result) + '\n');} else {diff --git a/src/commands/projections.ts b/src/commands/projections.tsindex bd8dfcd17a..f748717dac 100644--- a/src/commands/projections.ts+++ b/src/commands/projections.ts@@ -18,6 +18,7 @@ import { catalogueError } from '../core/error-catalogue.ts';import type { LockHolderInfo } from '../core/pglite-lock.ts';import type { ProjectionBacklog, ProjectionDrainResult, ProjectionRebuildFailure } from '../core/page-state/projections.ts';import { IMAGE_SEAL_PAGES_PER_SECOND, TEXT_REBUILD_PAGES_PER_SECOND, projectionDrainLimit } from '../core/page-state/image-seal.ts';+import { DEFAULT_DRAIN_CONCURRENCY } from '../core/page-state/rebuild-estimate.ts';// #6286: the drain lives in core so `migrate` can run it on its target before the config flip.export { drainProjections, previewProjectionDrain, type ProjectionDrainResult } from '../core/page-state/projections.ts';import { SERVE_LAUNCHD_LABEL, SERVE_SYSTEMD_UNIT } from '../core/serve-service.ts';@@ -26,12 +27,14 @@ import { detectInstalledJob } from './autopilot/jobs.ts';export const PROJECTION_DRAIN_DOCS = 'docs/guides/repair.md#projection-drain';-export const PROJECTIONS_HELP = `Usage: gbrain projections drain [--dry-run] [--limit <n>] [--json]+export const PROJECTIONS_HELP = `Usage: gbrain projections drain [--dry-run] [--limit <n>] [--concurrency <n>] [--json]Rebuild queued Markdown and code text projections now, in batches of up to 100,until every row queued before the run started has been tried once. Pages stored-in segments are built first, except one another gbrain process is building.-Then seal the image pages the importer already attested safe (sealable). Keyless:+in segments are built first, except one another gbrain process is building, then+pages search does not answer for yet. Then seal the image pages the importer+already attested safe (sealable), and last rebuild the pages search still answers+for from the index they had before an upgrade. Keyless:no provider call, no file read, no re-embedding. Image pages it cannot seal(reimport) need their importer: preview with gbrain embed --stale --images --dry-run.Protected-OCR images and archived-source pages stay unsealed by design.@@ -42,6 +45,9 @@ Protected-OCR images and archived-source pages stay unsealed by design.--limit <n> Try at most n pages in this run (default: no cap); doctor's fixsizes it to about a minute (${projectionDrainLimit()} images,${projectionDrainLimit(1)} while text pages wait)+ --concurrency <n> Postgres: rebuild n pages at once, each on its own pooled+ connection (default ${DEFAULT_DRAIN_CONCURRENCY}; at most the pool size less 2). A refused+ or dropped connection lowers it and the drain goes on. PGLite: 1--json Print {rebuilt, superseded, failed, remaining, limited, sealed,sealed_with_visual, sealed_text_only, statistics_refreshed,remaining_by_class} to stdout (the class counts and rows with --dry-run)diff --git a/src/core/cli-flag-registry.generated.ts b/src/core/cli-flag-registry.generated.tsindex 772d01f9fc..d33235b8ed 100644--- a/src/core/cli-flag-registry.generated.ts+++ b/src/core/cli-flag-registry.generated.ts@@ -19,7 +19,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'agent': ['--allow-old-serve', '--brain', '--detach', '--fanout-manifest', '--federated-read', '--flags', '--follow', '--harness', '--help', '--http', '--json', '--max-turns', '--mcp-only', '--model', '--no-follow', '--port', '--preset', '--reissue', '--scopes', '--show-token', '--since', '--source', '--subagent-def', '--surface', '--timeout-ms', '--token-ttl', '--tools', '--url', '--word', '--yes'],'anomalies': ['--all', '--brain', '--help', '--json', '--lookback-days', '--mcp-only', '--request-id', '--sigma', '--since', '--source'],'append': ['--brain', '--expect-tail', '--expected-revision', '--file', '--force', '--help', '--json', '--region', '--request-id', '--separator', '--source'],- 'apply-migrations': ['--accept-reviewed-inventory', '--acknowledge-no-backup', '--apply', '--apply-rewrites', '--backup-confirmed', '--brain', '--break-lock', '--confirm-quiesced', '--content-root', '--dry-run', '--export-db-only', '--export-source', '--force', '--force-all', '--force-orchestrator', '--force-retry', '--force-schema', '--help', '--host-dir', '--http', '--install', '--json', '--legacy-lock', '--list', '--locks', '--migrate-only', '--migration', '--mode', '--no-autopilot-install', '--non-interactive', '--only', '--path', '--prefer-postgres', '--probe', '--repo', '--require-db', '--skip-verify', '--source', '--statement-timeout', '--uninstall', '--url', '--yes'],+ 'apply-migrations': ['--accept-reviewed-inventory', '--acknowledge-no-backup', '--apply', '--apply-rewrites', '--backup-confirmed', '--brain', '--break-lock', '--concurrency', '--confirm-quiesced', '--content-root', '--dry-run', '--export-db-only', '--export-source', '--force', '--force-all', '--force-orchestrator', '--force-retry', '--force-schema', '--help', '--host-dir', '--http', '--install', '--json', '--legacy-lock', '--list', '--locks', '--migrate-only', '--migration', '--mode', '--no-autopilot-install', '--non-interactive', '--only', '--path', '--prefer-postgres', '--probe', '--repo', '--require-db', '--skip-verify', '--source', '--statement-timeout', '--uninstall', '--url', '--yes'],'auth': ['--add', '--admin-intent', '--adopt-columns', '--adopt-permissions', '--all', '--all-new', '--allowed-operations', '--apply', '--attest-actor', '--attest-ops', '--attest-read', '--attest-sources', '--before', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--budget-usd-per-day', '--category', '--cleanup-dead-local-locks', '--client', '--code', '--confirm-quiesced', '--confirm-relocated-root', '--days', '--delegated-namespace', '--delegated-slug-prefixes', '--disable', '--dry-run', '--email', '--enable', '--enable-dcr-insecure', '--expect', '--expected-epoch', '--expected-state', '--federated-read', '--fix', '--force', '--grant-types', '--group-by', '--help', '--id', '--if-version', '--json', '--limit', '--manifest', '--max-concurrent', '--may-act-for', '--member', '--members', '--migrate-legacy', '--min-trust', '--no-pull', '--only', '--operation', '--operations', '--origin', '--parked-code', '--parked-pushes', '--pat-file', '--path', '--private-pages', '--probe', '--profile', '--qm', '--read-sources', '--reads-unscoped', '--redirect-uri', '--refresh-operations', '--repair', '--replace', '--request-id', '--reset-default', '--retire', '--scopes', '--self-transfer', '--shared-skills', '--since', '--slug-prefixes', '--source', '--source-ids', '--sources', '--state', '--surface', '--takes-holders', '--token', '--token-endpoint-auth-method', '--token-ttl', '--unsettle', '--usage', '--wait', '--warn-only', '--yes'],'autopilot': ['--apply', '--backend', '--batch', '--brain', '--channel', '--check', '--concurrent', '--dimensions', '--dir', '--drain', '--dry-run', '--embedding-dimensions', '--embedding-model', '--exit-code', '--expect', '--explain', '--ff-only', '--force', '--force-retry', '--from-pages', '--help', '--inject-bootstrap', '--inline', '--install', '--interval', '--is-ancestor', '--json', '--markdown', '--max-cost', '--max-rss', '--max-runtime', '--max-usd', '--migrate-only', '--model', '--no-autopilot-install', '--no-bun-floor-check', '--no-embed', '--no-embed-cache', '--no-inject', '--no-restart', '--no-worker', '--non-interactive', '--now', '--once', '--only', '--output', '--params', '--pause', '--phase', '--quiet', '--reader-mode', '--reason', '--refresh-cache', '--refs', '--remediation-plan', '--repo', '--restart-only', '--resume', '--retry-global', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--source', '--stale', '--status', '--stop-mcp', '--swap-only', '--tags', '--target', '--trust', '--unattended', '--uninstall', '--user', '--verify', '--version', '--yes'],'backfill': ['--batch-size', '--brain', '--concurrency', '--dry-run', '--fresh', '--help', '--json', '--keep-index', '--list', '--max-errors', '--max-rows', '--resume', '--source'],@@ -37,7 +37,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'check-backlinks': ['--accept-pending', '--as', '--background', '--brain', '--dir', '--dry-run', '--explain', '--follow', '--help', '--json', '--no-accept-pending', '--progress-interval', '--progress-json', '--quiet', '--room', '--source', '--timeout', '--timings', '--wait'],'check-resolvable': ['--brain', '--dry-run', '--fix', '--help', '--json', '--skills-dir', '--source', '--strict', '--verbose'],'check-update': ['--backend', '--brain', '--channel', '--check-only', '--exit-code', '--ff-only', '--force-retry', '--help', '--is-ancestor', '--json', '--markdown', '--migrate-only', '--no-autopilot-install', '--no-bun-floor-check', '--no-restart', '--non-interactive', '--quiet', '--refresh-cache', '--refs', '--remediation-plan', '--restart-only', '--source', '--stop-mcp', '--swap-only', '--tags', '--version', '--yes'],- 'claw-test': ['--agent', '--apply', '--brain', '--dir', '--force-retry', '--format', '--help', '--json', '--keep-tempdir', '--list-agents', '--live', '--local', '--locks', '--message', '--no-embed', '--no-embedding', '--only', '--output-format', '--pglite', '--progress-json', '--run-id', '--scenario', '--skip-verify', '--source', '--statement-timeout', '--version', '--yes'],+ 'claw-test': ['--agent', '--apply', '--brain', '--concurrency', '--dir', '--force-retry', '--format', '--help', '--json', '--keep-tempdir', '--list-agents', '--live', '--local', '--locks', '--message', '--no-embed', '--no-embedding', '--only', '--output-format', '--pglite', '--progress-json', '--run-id', '--scenario', '--skip-verify', '--source', '--statement-timeout', '--version', '--yes'],'code-callees': ['--all-sources', '--brain', '--force', '--help', '--json', '--lang', '--limit', '--no-embed', '--no-json', '--probe', '--source', '--yes'],'code-callers': ['--all-sources', '--brain', '--force', '--help', '--json', '--lang', '--limit', '--no-embed', '--no-json', '--probe', '--source', '--yes'],'code-def': ['--all-sources', '--brain', '--force', '--help', '--json', '--lang', '--limit', '--no-embed', '--no-json', '--probe', '--source', '--yes'],@@ -49,7 +49,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'conversation-parser': ['--brain', '--help', '--json', '--source'],'core': ['--apply', '--brain', '--fail-fast', '--help', '--include-core', '--json', '--priority', '--revision', '--slug', '--source', '--tools-json'],'creds': ['--allow-create-db', '--allow-docker', '--brain', '--chat-model', '--content-root', '--db-only', '--embedding-dimensions', '--embedding-model', '--expansion-model', '--force', '--git', '--grant-types', '--help', '--http', '--ids', '--issuer-url', '--json', '--key', '--local-postgres', '--mcp-only', '--mcp-url', '--migrate-only', '--model', '--no-embed', '--no-embedding', '--no-git', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--out', '--passphrase-env', '--path', '--pglite', '--plan', '--prefer-postgres', '--provenance', '--provider', '--query', '--schema-pack', '--scopes', '--search-mode', '--skip-embed-check', '--source', '--status', '--supabase', '--surface', '--to', '--url', '--version'],- 'db-repair': ['--accept-pending', '--apply', '--apply-rewrites', '--as', '--background', '--brain', '--break-lock', '--db-url', '--dry-run', '--explain', '--filter', '--follow', '--force', '--force-retry', '--format', '--help', '--http', '--json', '--legacy-lock', '--locks', '--name', '--no-accept-pending', '--only', '--path', '--prefer-postgres', '--probe', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--restart', '--room', '--skip-verify', '--source', '--statement-timeout', '--supabase', '--timeout', '--timings', '--undo-last-rewrite', '--url', '--version', '--wait', '--yes'],+ 'db-repair': ['--accept-pending', '--apply', '--apply-rewrites', '--as', '--background', '--brain', '--break-lock', '--concurrency', '--db-url', '--dry-run', '--explain', '--filter', '--follow', '--force', '--force-retry', '--format', '--help', '--http', '--json', '--legacy-lock', '--locks', '--name', '--no-accept-pending', '--only', '--path', '--prefer-postgres', '--probe', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--restart', '--room', '--skip-verify', '--source', '--statement-timeout', '--supabase', '--timeout', '--timings', '--undo-last-rewrite', '--url', '--version', '--wait', '--yes'],'decide': ['--accept-all', '--all', '--all-from', '--apply', '--brain', '--call-site', '--claims', '--dataset', '--dry-run', '--egress', '--embedding-dimensions', '--embedding-model', '--expect', '--from', '--help', '--include-withdrawals', '--input', '--items-out', '--json', '--kind', '--limit', '--max-cost', '--max-usd', '--min', '--non-interactive', '--out', '--provenance', '--provider', '--query', '--rate', '--reason', '--reason-family', '--recommended', '--remove', '--shadow', '--since', '--slot', '--source', '--status', '--suite', '--target', '--threshold', '--trust', '--undo', '--uri-prefix', '--version', '--what-if-threshold', '--yes'],'diag': ['--access', '--add', '--all', '--allowed-operations', '--apply', '--as', '--asof', '--body', '--bound-slug-prefixes', '--brain', '--budget-ms', '--card', '--client', '--count', '--cursor', '--date', '--dated-since', '--days', '--detach', '--entities', '--expect', '--expected-revision', '--federated-read', '--filter', '--follow', '--format', '--from-index', '--full', '--fuzzy', '--group-by', '--held', '--help', '--http', '--id', '--include', '--include-ambiguous', '--include-deleted', '--json', '--kind', '--limit', '--max-usd', '--model', '--multimodal', '--no-autopilot-install', '--no-pull', '--no-walk', '--of', '--only', '--page', '--params', '--patch', '--path', '--phase', '--prefer', '--preview', '--privacy-url', '--purge', '--query', '--recent', '--refresh-operations', '--replace', '--restart-only', '--reverse', '--row', '--scopes', '--session', '--side', '--since', '--slug', '--slugs', '--source', '--source-id', '--sources', '--stale', '--state', '--surface', '--terms-url', '--to', '--updated-after', '--url', '--yes'],'doctor': ['--abandon', '--abbrev-ref', '--abort', '--absolute-git-dir', '--accept-all', '--accept-capabilities', '--accept-pending', '--access', '--access-policy', '--account', '--acknowledge-managed', '--add', '--admin-intent', '--adopt', '--adopt-columns', '--adopt-permissions', '--advertise', '--after', '--agent', '--aggressive', '--alert', '--aliases', '--all', '--all-drifted', '--all-from', '--all-sources', '--allow-old-serve', '--allow-outside-source-root', '--allow-shell-jobs', '--allow-unverified-remote', '--allow-without-owner', '--allowed-operations', '--anchor', '--applied', '--apply', '--apply-rewrites', '--approval-out', '--approved', '--as', '--assign-corpus', '--assume-unchanged', '--attest-actor', '--attest-ops', '--attest-read', '--attest-sources', '--audit', '--auto', '--auto-additive', '--auto-fix', '--autostash', '--background', '--base', '--batch', '--batch-check', '--body', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--branch', '--break-lock', '--budget-ms', '--budget-policy', '--budget-tokens', '--budget-usd', '--bundled', '--by-mention', '--cached', '--cacheinfo', '--call-site', '--calls', '--catalog', '--catch-up', '--channel', '--check', '--check-downgrade', '--check-only', '--cherry-mark', '--claims', '--class', '--claude-plugins', '--clear', '--clear-never-installed', '--cli-socket', '--client', '--close', '--code', '--concurrency', '--concurrent', '--confidence', '--config', '--confirm', '--confirm-destructive', '--confirm-quiesced', '--confirm-relocated-root', '--conflicts', '--contacts-dir', '--content-audit', '--content-root', '--continue', '--conversation', '--count', '--credentials-file', '--cursor', '--cwd', '--dataset', '--day', '--days', '--db-url', '--deadline', '--decisions', '--delegated-namespace', '--delegated-slug-prefixes', '--delete-brain', '--dest', '--detach', '--detail', '--diff-filter', '--dim', '--dimensions', '--dir', '--disable', '--drain', '--drain-timeout', '--drop-field', '--dry-run', '--egress', '--email', '--embedding-dimensions', '--embedding-model', '--enable', '--enable-dcr', '--end-of-options', '--entity', '--env', '--env-file', '--error-unmatch', '--exclude', '--exclude-standard', '--exit-code', '--expand', '--expect', '--expect-local', '--expect-remote', '--expect-upstream', '--expected-epoch', '--expected-revision', '--expected-state', '--explain', '--facts', '--fast', '--federated', '--federated-read', '--fetch-timeout-ms', '--ff-only', '--file', '--filter', '--finalize', '--first-parent', '--fix', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-remove', '--force-retry', '--force-schema', '--force-with-lease', '--format', '--fresh', '--fresh-token', '--from', '--from-json', '--from-meetings', '--from-pages', '--from-prefix', '--full', '--full-tree', '--fuzzy', '--gbrain-bin', '--gbrain-skill-resolver', '--generation', '--get', '--get-all', '--get-regexp', '--get-url', '--git-common-dir', '--git-dir', '--git-path', '--grammar', '--group', '--group-by', '--hard', '--hard-deadline', '--harness', '--heads', '--held', '--help', '--history', '--hold-state', '--holds', '--home', '--http', '--id', '--if-version', '--ignore-env-override', '--ignore-missing', '--ignore-unmatch', '--images', '--import', '--include-ambiguous', '--include-covered', '--include-cross-entity', '--include-deleted', '--include-failures', '--include-flagged', '--include-frontmatter', '--include-gitignored', '--include-hidden', '--include-pseudo', '--include-repairs', '--include-withdrawals', '--index-audit', '--index-info', '--infer-dates', '--init', '--input', '--input-type', '--install', '--instructions', '--instructions-only', '--is-ancestor', '--is-bare-repository', '--is-inside-work-tree', '--is-shallow-repository', '--items-out', '--job-isolation', '--jq', '--json', '--keep', '--keep-checkout', '--key', '--kind', '--label-lowerings', '--labelled-only', '--lanes', '--lang', '--left-right', '--legacy-lock', '--limit', '--link', '--link-source', '--link-type', '--list', '--list-out', '--local', '--local-env-vars', '--locks', '--log-noop', '--manifest', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-count', '--max-facts', '--max-jobs', '--max-requests', '--max-rss', '--max-runtime', '--max-usd', '--max-waiting', '--may-act-for', '--mcp-only', '--member', '--members', '--members-file', '--merge-base', '--migrate-legacy', '--migrate-only', '--migration', '--min', '--min-n', '--mode', '--model', '--multimodal', '--name', '--name-only', '--name-status', '--nice', '--no-accept-pending', '--no-assume-unchanged', '--no-autopilot-install', '--no-autostash', '--no-capture', '--no-commit-id', '--no-confirm', '--no-cron', '--no-delegate', '--no-edit', '--no-embed', '--no-embed-cache', '--no-embedding', '--no-expand', '--no-extensions', '--no-extract', '--no-federated', '--no-filters', '--no-follow-tags', '--no-gpg-sign', '--no-hooks', '--no-index', '--no-instructions', '--no-lanes', '--no-llm', '--no-merges', '--no-migrate', '--no-open-pages', '--no-probe', '--no-pull', '--no-renames', '--no-repair', '--no-restart', '--no-schema-pack', '--no-seat', '--no-skip-worktree', '--no-stdio-member', '--no-tags', '--no-verify', '--no-walk', '--no-write-fetch-head', '--non-interactive', '--not', '--now', '--of', '--older-than', '--once', '--only', '--operation', '--operations', '--others', '--out', '--output', '--oversized', '--overwrite', '--page', '--parallel', '--params', '--parents', '--parked-code', '--parked-pushes', '--pat-file', '--path', '--path-format', '--pending', '--pglite', '--phase', '--pid-file', '--plan', '--porcelain', '--port', '--prefer', '--prefer-postgres', '--prefix', '--preload', '--preset', '--preview', '--principal', '--priority', '--privacy-url', '--private-pages', '--probe', '--probe-pglite', '--profile', '--progress-interval', '--progress-json', '--project', '--provenance', '--provider', '--providers', '--prune-orphans', '--public-url', '--pull', '--pure', '--purge', '--push', '--push-only', '--push-remote', '--qm', '--query', '--question', '--queue', '--quiet', '--rate', '--raw-only', '--read-sources', '--reader-mode', '--readers', '--reads-unscoped', '--reason', '--reauth', '--rebase', '--rebuild', '--recommended', '--recurse-submodules', '--redact-paths', '--reference-date', '--refresh', '--refresh-operations', '--refresh-skills', '--regenerate', '--reissue', '--relative', '--relower', '--remediate', '--remediation-plan', '--remote', '--remove', '--remove-core', '--repair', '--repair-unknown-markers', '--replace', '--repo', '--request', '--request-id', '--require', '--require-clean', '--reranker', '--reset', '--reset-checkpoint', '--reset-default', '--reset-session', '--resolve', '--restart-only', '--restore-only', '--resume', '--retire', '--retry-failed', '--retry-global', '--retry-held', '--retry-parked', '--reverse', '--review-lower', '--revision', '--right-only', '--rollback-to-source', '--room', '--root', '--routing', '--rules', '--run', '--runtime', '--safe', '--sample', '--save', '--scan-secrets', '--scan-timeout', '--schedule', '--scope', '--scopes', '--search-mode', '--seat', '--section', '--segmented', '--select', '--self-transfer', '--semantic', '--serves', '--session', '--session-id', '--set', '--set-upstream-to', '--shadow', '--shared-skills', '--short', '--show-current', '--show-object-format', '--show-prefix', '--show-token', '--show-toplevel', '--side', '--since', '--since-last-run', '--skill-loading', '--skills-dir', '--skills-index-only', '--skip', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--skip-worktree', '--slot', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slugs', '--slugs-file', '--source', '--source-id', '--source-ids', '--sources', '--squash', '--src-subpath', '--staged', '--stale', '--stat', '--state', '--statement-timeout', '--status', '--stdin', '--stdio-member', '--stop-mcp', '--strategy', '--strict', '--strict-json', '--suite', '--supabase', '--surface', '--swap-only', '--symbol-kind', '--symbolic-full-name', '--synopsis', '--tags', '--take', '--takes-holders', '--target', '--target-score', '--terms-url', '--threshold', '--tier', '--timeout', '--timings', '--to', '--token', '--token-ttl', '--top-k', '--topo-order', '--trust', '--tsconfig-override', '--type', '--types', '--unattended', '--unindexed', '--uninstall', '--unsafe-bypass-dream-guard', '--unschedule', '--unset', '--unset-all', '--unsettle', '--unshallow', '--untrack', '--untracked-files', '--url', '--url-env', '--user', '--vacuum', '--vectors', '--verbose', '--verify', '--version', '--wait', '--wait-drain', '--wakeshift', '--warn-only', '--watch', '--what-if-threshold', '--where', '--window', '--workers', '--working-tree', '--workspace', '--worktree', '--write-tree', '--yes'],@@ -89,7 +89,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'maintain': ['--absolute-git-dir', '--accept-dns', '--accept-pending', '--account', '--admin-intent', '--adopt-columns', '--adopt-permissions', '--advertise', '--after', '--alert', '--all', '--allow-outside-source-root', '--apply', '--as', '--assign-corpus', '--attest-read', '--audit', '--background', '--bg', '--brain', '--break-lock', '--budget-ms', '--by-mention', '--cached', '--cask', '--catch-up', '--check-only', '--class', '--clear', '--client', '--concurrency', '--conflicts', '--contacts-dir', '--content-audit', '--continue', '--conversation', '--count', '--cwd', '--db-url', '--delete-brain', '--detach', '--detail', '--diff-filter', '--dim', '--dir', '--drain', '--dry-run', '--enable', '--enable-dcr', '--entity', '--exclude', '--expect', '--expected-revision', '--expected-state', '--explain', '--facts', '--fast', '--fix', '--follow', '--force', '--force-retry', '--force-schema', '--format', '--from', '--from-meetings', '--full', '--get', '--grammar', '--group-by', '--hard-deadline', '--harness', '--held', '--help', '--home', '--http', '--https', '--id', '--ignore-missing', '--images', '--include-ambiguous', '--include-flagged', '--include-frontmatter', '--include-gitignored', '--include-hidden', '--include-repairs', '--index-audit', '--infer-dates', '--input', '--install', '--instructions-only', '--json', '--keep-checkout', '--kind', '--lang', '--left-right', '--limit', '--link-source', '--link-type', '--list', '--local', '--locks', '--markdown', '--max-cost', '--max-cost-usd', '--max-jobs', '--max-rss', '--max-usd', '--may-act-for', '--migrate-legacy', '--migrate-only', '--multimodal', '--name', '--name-only', '--ner', '--no-accept-pending', '--no-autopilot-install', '--no-capture', '--no-delegate', '--no-embed', '--no-extensions', '--no-extract', '--no-hard-deadline', '--no-instructions', '--no-pull', '--no-renames', '--no-schema-pack', '--no-walk', '--non-interactive', '--now', '--older-than', '--once', '--only', '--operations', '--operator', '--output', '--oversized', '--pack', '--page', '--parallel', '--params', '--path', '--pending', '--pglite', '--phase', '--pid-file', '--plan', '--porcelain', '--port', '--prefer', '--prefix', '--preview', '--privacy-url', '--probe', '--probe-pglite', '--progress-interval', '--progress-json', '--project', '--providers', '--prune-orphans', '--public-url', '--pull', '--purge', '--push', '--push-only', '--query', '--question', '--queue', '--quiet', '--raw-only', '--reader-mode', '--readers', '--reads-unscoped', '--reauth', '--rebase', '--rebuild', '--regenerate', '--remediate', '--remediation-plan', '--remote', '--remove', '--remove-core', '--repo', '--request-id', '--reset', '--reset-default', '--restart-only', '--resume', '--retry-failed', '--retry-held', '--retry-parked', '--reverse', '--room', '--runner', '--safe', '--scan-secrets', '--schedule', '--scope', '--scopes', '--serves', '--session', '--set-path', '--show-current', '--show-toplevel', '--since', '--skill-loading', '--skills-dir', '--skills-index-only', '--skip-failed', '--slot', '--slug', '--slugs', '--source', '--source-id', '--sources', '--src-subpath', '--staged', '--stale', '--stat', '--state', '--status', '--stdin', '--strategy', '--strict', '--supabase', '--surface', '--symbol-kind', '--takes-holders', '--target', '--target-score', '--terms-url', '--tier', '--timeout', '--timings', '--to', '--token', '--top-k', '--trust', '--type', '--unattended', '--unindexed', '--unschedule', '--untracked-files', '--url-env', '--user', '--verbose', '--verify', '--version', '--wait', '--wakeshift', '--where', '--window', '--workers', '--working-tree', '--workspace', '--yes'],'mcp': ['--admin-token-file', '--advertise', '--agent', '--all', '--arg', '--attest-actor', '--attest-ops', '--attest-read', '--bearer-token-env-var', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--budget-usd-per-day', '--client', '--command', '--credentials-file', '--credentials-out', '--cursor', '--delegate', '--delegated-namespace', '--delegated-slug-prefixes', '--dry-run', '--email', '--enable-dcr', '--env', '--federated-read', '--flow', '--force', '--funnel', '--groups', '--harness', '--header', '--help', '--if-version', '--install', '--issuer-url', '--json', '--limit', '--mcp-only', '--mcp-url', '--member', '--name', '--no-install', '--no-service', '--no-tailscale', '--oauth-client-id', '--oauth-client-secret', '--oauth-request', '--port', '--profile', '--qm', '--queue', '--reason', '--redirect-uri', '--remove', '--resume', '--root', '--scope', '--scopes', '--skills', '--source', '--status', '--surface', '--timeout-ms', '--token', '--token-endpoint-auth-method', '--token-ttl', '--url', '--yes'],'memory-proposals': ['--all', '--brain', '--dry-run', '--help', '--json', '--limit', '--reason', '--replace', '--source'],- 'migrate': ['--accept-pending', '--admin-intent', '--all', '--apply', '--as', '--background', '--batch-size', '--brain', '--catch-up', '--dim', '--drain-timeout', '--dry-run', '--embedding-dimensions', '--embedding-model', '--expect', '--expected-epoch', '--expected-state', '--explain', '--facts', '--follow', '--force', '--force-retry', '--harness', '--held', '--help', '--ignore-env-override', '--images', '--include-null-signature', '--json', '--locks', '--manifest', '--max-cost', '--max-cost-usd', '--max-usd', '--no-accept-pending', '--no-embed', '--no-embedding', '--no-migrate', '--non-interactive', '--only', '--oversized', '--pace', '--pace-max-concurrency', '--path', '--pglite', '--plan', '--priority', '--probe', '--progress-interval', '--progress-json', '--quiet', '--replace', '--reranker', '--resume', '--retarget', '--rollback-to-source', '--room', '--scope', '--skip-verify', '--slugs', '--source', '--stale', '--statement-timeout', '--status', '--timeout', '--timings', '--to', '--trigger-bypass', '--trust', '--url', '--url-env', '--wait', '--yes'],+ 'migrate': ['--accept-pending', '--admin-intent', '--all', '--apply', '--as', '--background', '--batch-size', '--brain', '--catch-up', '--concurrency', '--dim', '--drain-timeout', '--dry-run', '--embedding-dimensions', '--embedding-model', '--expect', '--expected-epoch', '--expected-state', '--explain', '--facts', '--follow', '--force', '--force-retry', '--harness', '--held', '--help', '--ignore-env-override', '--images', '--include-null-signature', '--json', '--locks', '--manifest', '--max-cost', '--max-cost-usd', '--max-usd', '--no-accept-pending', '--no-embed', '--no-embedding', '--no-migrate', '--non-interactive', '--only', '--oversized', '--pace', '--pace-max-concurrency', '--path', '--pglite', '--plan', '--priority', '--probe', '--progress-interval', '--progress-json', '--quiet', '--replace', '--reranker', '--resume', '--retarget', '--rollback-to-source', '--room', '--scope', '--skip-verify', '--slugs', '--source', '--stale', '--statement-timeout', '--status', '--timeout', '--timings', '--to', '--trigger-bypass', '--trust', '--url', '--url-env', '--wait', '--yes'],'models': ['--batch', '--brain', '--concurrent', '--cycles', '--dim', '--dimensions', '--embedding-dimensions', '--embedding-model', '--help', '--json', '--limit', '--max-tokens', '--max-usd', '--output', '--receipt-dir', '--skip', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--source', '--status', '--task', '--to', '--yes'],'mounts': ['--alias', '--brain', '--database-path', '--database-url', '--db-path', '--db-url', '--engine', '--help', '--id', '--json', '--mcp-url', '--path', '--skills-dir', '--source'],'notability-eval': ['--brain', '--embedding-dimensions', '--embedding-model', '--help', '--in', '--json', '--out', '--repo', '--skip-llm', '--source', '--status', '--target-high', '--target-low', '--target-medium'],@@ -100,7 +100,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {'pglite-repair': ['--brain', '--break-lock', '--dry-run', '--expect', '--help', '--http', '--json', '--legacy-lock', '--path', '--probe', '--source', '--vectors', '--yes'],'post-upgrade': ['--accept-reviewed-inventory', '--acknowledge-no-backup', '--admin-intent', '--all', '--all-applied', '--apply', '--apply-clean-hunks', '--backend', '--backup-confirmed', '--brain', '--catch-up', '--changed', '--channel', '--check', '--check-only', '--client', '--concurrency', '--confirm-quiesced', '--confirm-relocated-root', '--content-root', '--cost-estimate', '--embedding-dimensions', '--embedding-model', '--exit-code', '--expand', '--expect', '--expected-epoch', '--expected-state', '--explain', '--export-db-only', '--export-source', '--ff-only', '--force', '--force-all', '--force-orchestrator', '--force-retry', '--force-schema', '--format', '--held', '--help', '--host-dir', '--id', '--images', '--inject-bootstrap', '--inline', '--install', '--instructions-only', '--interval', '--is-ancestor', '--json', '--limit', '--list', '--manifest', '--markdown', '--max-rss', '--max-usd', '--may-act-for', '--member', '--migrate-only', '--migration', '--mode', '--multimodal', '--name', '--name-only', '--no-autopilot-install', '--no-bun-floor-check', '--no-embed', '--no-expand', '--no-inject', '--no-restart', '--no-worker', '--non-interactive', '--now', '--once', '--only', '--oversized', '--path', '--pause', '--phase', '--prefix', '--quiet', '--readers', '--reads-unscoped', '--reason', '--refs', '--remediation-plan', '--repo', '--require-db', '--restart-only', '--resume', '--retry-global', '--scope', '--segmented', '--self-transfer', '--skills-dir', '--skip-verify', '--slug', '--slugs-file', '--source', '--stale', '--statement-timeout', '--status', '--stop-mcp', '--swap-only', '--synopsis', '--tags', '--target', '--token', '--uninstall', '--user', '--vectors', '--verify', '--version', '--workers', '--yes'],'pricing': ['--brain', '--help', '--input', '--json', '--max-cost', '--max-runtime', '--output', '--rate', '--source'],- 'projections': ['--advertise', '--brain', '--dry-run', '--enable-dcr', '--help', '--http', '--images', '--install', '--json', '--limit', '--now', '--only', '--port', '--public-url', '--remove', '--source', '--stale', '--status', '--surface', '--user'],+ 'projections': ['--advertise', '--brain', '--concurrency', '--dry-run', '--enable-dcr', '--help', '--http', '--images', '--install', '--json', '--limit', '--now', '--only', '--port', '--public-url', '--remove', '--source', '--stale', '--status', '--surface', '--user'],'protocol': ['--access', '--add', '--all', '--allowed-operations', '--as', '--asof', '--body', '--bound-slug-prefixes', '--brain', '--card', '--client', '--cursor', '--date', '--dated-since', '--days', '--entities', '--expected-revision', '--federated-read', '--follow', '--from-index', '--full', '--fuzzy', '--group-by', '--help', '--http', '--id', '--include', '--include-deleted', '--json', '--kind', '--limit', '--max-usd', '--model', '--multimodal', '--no-autopilot-install', '--of', '--only', '--page', '--params', '--patch', '--path', '--phase', '--prefer', '--privacy-url', '--provenance', '--purge', '--recent', '--refresh-operations', '--row', '--scopes', '--session', '--since', '--slugs', '--source', '--sources', '--stale', '--surface', '--synthesize', '--target', '--terms-url', '--to', '--token', '--updated-after', '--url', '--yes'],'providers': ['--brain', '--embedding-dimensions', '--embedding-model', '--help', '--json', '--model', '--source', '--status', '--touchpoint'],'publish': ['--accent', '--bg', '--border', '--brain', '--card-bg', '--code-bg', '--error', '--fg', '--help', '--json', '--link', '--muted', '--out', '--password', '--source', '--title'],diff --git a/src/core/page-state/projections.ts b/src/core/page-state/projections.tsindex efebbe28e8..88eedfa10e 100644--- a/src/core/page-state/projections.ts+++ b/src/core/page-state/projections.ts@@ -27,6 +27,8 @@ import { advanceProjectionCursor, ineligibleKey, rememberIneligible, residentWinimport { acceptedEmbeddingInputHashes, embeddingInputHash, plainEmbeddingTier, synopsisBodyHash,type EmbeddingInputContext, type EmbeddingTier } from '../embedding-input-hash.ts';import { chunkModelMatchesSql } from '../embedding-invalidation.ts';+import { isRetryableConnError } from '../retry-matcher.ts';+import { ACTIVATION_SERVED_REVISION } from '../search/safe-chunks.ts';/*** Complete the searchable snapshot only after its sanitized chunks are installed.@@ -645,6 +647,19 @@ export interface ProjectionRebuildOptions {* per run instead of once per batch. Ignored with `window`.*/walk?: { after?: QueueKey };+ /**+ * #9192: pages rebuilt at once, each in its own transactions on its own pooled connection. Defaults to 1. When an+ * extra connection is refused or dropped while more than one page is in flight, that page goes back to the others+ * and one fewer page runs at once from then on (onNarrow); the last one keeps today's behaviour.+ */+ concurrency?: number;+ /** Receives the lowered page count, and the connection error that lowered it. */+ onNarrow?: (concurrency: number, error: unknown) => void;+ /**+ * #9192: only pages search does not answer for yet ('waiting'), or only pages it answers for from their+ * pre-upgrade index (ACTIVATION_SERVED_REVISION, 'answering').+ */+ served?: 'waiting' | 'answering';}/** A queue row's position in the walk order (updated_at as text, so microseconds survive). */@@ -754,7 +769,9 @@ export async function previewProjectionDrain(engine: Pick<BrainEngine, 'executeRconst backlog = await projectionBacklog(engine);const rows = await engine.executeRaw<{ source_id: string; slug: string; page_kind: string; reason: string; updated_at: string }>(`SELECT s.id AS source_id,j.slug,p.page_kind,j.reason,j.updated_at::text AS updated_atFROM page_projection_jobs j ${projectionJobProbes({ media: true })}- ORDER BY j.updated_at,j.source_incarnation,j.slug LIMIT $1`, [Math.max(1, limit)]);+ ORDER BY CASE WHEN p.page_kind NOT IN ('markdown','code') THEN 1+ WHEN EXISTS (SELECT 1 FROM pages q WHERE q.source_id=s.id AND q.slug=j.slug AND q.text_projection_revision='${ACTIVATION_SERVED_REVISION}'::uuid) THEN 2+ ELSE 0 END,j.updated_at,j.source_incarnation,j.slug LIMIT $1`, [Math.max(1, limit)]);return { ...backlog, rows };}@@ -786,7 +803,13 @@ export async function rebuildPendingPageProjections(engine: BrainEngine, limit =params.push(after.updated_at, after.source_incarnation, after.slug);filters.push(`(j.updated_at,j.source_incarnation,j.slug)>($${params.length - 2}::text::timestamptz,$${params.length - 1}::uuid,$${params.length})`);}- const sourceFilter = opts.pages ? `WHERE s.id=$${params.push(opts.pages.sourceId)}` : '';+ const outer: string[] = [];+ if (opts.pages) outer.push(`s.id=$${params.push(opts.pages.sourceId)}`);+ if (opts.served) {+ outer.push(`${opts.served === 'waiting' ? 'NOT ' : ''}EXISTS (SELECT 1 FROM pages q WHERE q.source_id=s.id AND q.slug=j.slug+ AND q.text_projection_revision='${ACTIVATION_SERVED_REVISION}'::uuid)`);+ }+ const sourceFilter = outer.length ? `WHERE ${outer.join(' AND ')}` : '';const selected = opts.window ? await selectWindowJobs(engine, opts.window, opts.retryCooldown === true) : null;const jobs = selected ? selected.jobs.slice(0, Number(params[0])) : await engine.executeRaw<QueuedJob>(`SELECT s.id AS source_id,j.source_incarnation,j.slug,j.revision,p.page_kind,j.updated_at::text AS updated_atFROM (SELECT source_incarnation,slug,revision,updated_at FROM page_projection_jobs j@@ -797,9 +820,14 @@ export async function rebuildPendingPageProjections(engine: BrainEngine, limit =let rebuilt = 0;let superseded = 0;let attempted = 0;- for (const [index, job] of jobs.entries()) {- if (opts.deadlineMs !== undefined && index > 0 && now() - startedAt >= opts.deadlineMs) break;- attempted = index + 1;+ const width = engine.kind === 'postgres' ? Math.max(1, Math.min(Math.floor(opts.concurrency ?? 1), jobs.length)) : 1;+ let active = width;+ let lost = 0;+ let next = 0;+ const retry: number[] = [];+ // false: a refused or dropped connection handed this page back and this worker stops.+ const runJob = async (index: number): Promise<boolean> => {+ const job = jobs[index]!;const observed = [job.source_incarnation, job.slug, job.revision];try {const prepared = await engine.transaction(async tx => {@@ -824,8 +852,8 @@ export async function rebuildPendingPageProjections(engine: BrainEngine, limit =WHERE source_incarnation=$1::uuid AND slug=$2 AND revision=$3::uuid`, [...observed, snapshot.revision]);return read;});- if (!prepared) { superseded++; continue; }- if (prepared === 'removed') { superseded++; opts.onHeal?.({ kind: 'removed', source_id: job.source_id, slug: job.slug }); continue; }+ if (!prepared) { superseded++; return true; }+ if (prepared === 'removed') { superseded++; opts.onHeal?.({ kind: 'removed', source_id: job.source_id, slug: job.slug }); return true; }if (prepared === 'deferred') {// #7172: a publication left this revision's chunk replacement; it finishes in bounded steps, within the deadline.const step = await continueDeferredProjection(engine, job.source_id, job.slug,@@ -838,7 +866,7 @@ export async function rebuildPendingPageProjections(engine: BrainEngine, limit =await engine.executeRaw(`UPDATE page_projection_jobs SET updated_at=now()${step === 'skipped' ? ",reason='canonical_change'" : ''}WHERE source_incarnation=$1::uuid AND slug=$2 AND revision=$3::uuid`, observed);}- continue;+ return true;}if (prepared === 'segmented') {const built = await buildSegmentedProjection(engine, { sourceId: job.source_id, slug: job.slug });@@ -848,28 +876,45 @@ export async function rebuildPendingPageProjections(engine: BrainEngine, limit =AND NOT ${SEGMENTED_PROJECTION_CLAIMED_SQL}`, observed);if (built.sealed) rebuilt++;else superseded++;- continue;+ return true;}const projection = await preparePageProjection(prepared);await installPageProjection(engine, prepared, projection.chunks, { seal: true, preserveEmbeddings: true, code: projection.code });rebuilt++;} catch (error) {+ if (active > 1 && isRetryableConnError(error)) {+ active--;+ lost++;+ retry.push(index);+ opts.onNarrow?.(width - lost, error);+ return false;+ }if (!(error instanceof PageRevisionConflictError)) {await engine.executeRaw(`UPDATE page_projection_jobs j SET reason='rebuild_failed',updated_at=now()WHERE j.source_incarnation=$1::uuid AND j.slug=$2 AND j.revision=$3::uuid AND NOT ${SEGMENTED_PROJECTION_CLAIMED_SQL}`, observed);if (opts.onFailure) opts.onFailure({ source_id: job.source_id, slug: job.slug, reason: (error instanceof Error ? error.message : String(error)).split('\n')[0].slice(0, 300) });else process.stderr.write('[gbrain] Projection rebuild failed; work remains queued. Inspect projection readiness and the recorded source path.\n');- continue;+ return true;}// The page moved under its install: defer the exact row this call listed// (a newer trigger-queued row already carries its own stamp) and report it.const deferred = await engine.executeRaw(`UPDATE page_projection_jobs SET updated_at=now()WHERE source_incarnation=$1::uuid AND slug=$2 AND revision=$3::uuid RETURNING slug`, observed);- if (!deferred.length) { superseded++; continue; }+ if (!deferred.length) { superseded++; return true; }opts.onHeal?.({ kind: 'deferred', source_id: job.source_id, slug: job.slug });opts.onFailure?.({ source_id: job.source_id, slug: job.slug, reason: `revision_conflict: ${error.message.split('\n')[0].slice(0, 280)}` });}- }+ return true;+ };+ const worker = async (): Promise<void> => {+ for (;;) {+ const index = retry.length ? retry.shift()! : next < jobs.length ? next++ : -1;+ if (index < 0 || (opts.deadlineMs !== undefined && index > 0 && now() - startedAt >= opts.deadlineMs)) { active--; return; }+ attempted = Math.max(attempted, index + 1);+ if (!(await runJob(index))) return;+ }+ };+ await Promise.all(Array.from({ length: width }, worker));if (selected && opts.window) {// The window is finished up to its first eligible row this call did not reach.const unreached = selected.jobs.slice(attempted).find(job => job.ord > 0);@@ -923,7 +968,8 @@ export interface ProjectionDrainResult {* after v153's activation) each of those statements scans pages, so 2,000* notes beside 93K queued images rebuilt at 14 pages/s instead of 150.*/-export async function drainProjections(engine: BrainEngine, opts: { limit?: number; progress?: ProgressReporter; reasons?: readonly string[]; media?: boolean } = {}): Promise<ProjectionDrainResult> {+export async function drainProjections(engine: BrainEngine, opts: { limit?: number; progress?: ProgressReporter; reasons?: readonly string[]; media?: boolean;+ concurrency?: number; onNarrow?: (concurrency: number, error: unknown) => void } = {}): Promise<ProjectionDrainResult> {const [{ runStart }] = await engine.executeRaw<{ runStart: string }>('SELECT now()::text AS "runStart"');// PGLite's clock can be coarse: wait until it passes the run start, so a page// that fails in this run is stamped after it and is not tried again.@@ -941,7 +987,29 @@ export async function drainProjections(engine: BrainEngine, opts: { limit?: numblet superseded = 0;let tried = 0;let seal = { sealed: 0, sealed_with_visual: 0, sealed_text_only: 0 };- const walk: { after?: QueueKey } = {};+ let concurrency = Math.max(1, Math.floor(opts.concurrency ?? 1));+ const onNarrow = (width: number, error: unknown) => { concurrency = width; opts.onNarrow?.(width, error); };+ // Continue while a batch rebuilt, removed, failed or durably deferred a+ // row; stop on a no-op batch. A bare `superseded` row (another worker+ // consumed it) is not progress, so one row can never spin the drain.+ const rebuildInline = async (served: 'waiting' | 'answering') => {+ const walk: { after?: QueueKey } = {};+ while (opts.limit === undefined || tried < opts.limit) {+ let healed = 0;+ let failedNow = 0;+ const batch = await rebuildPendingPageProjections(engine, Math.min(DRAIN_BATCH, (opts.limit ?? Infinity) - tried),+ { notAfter: runStart, reasons: opts.reasons, media: opts.media, walk, served, concurrency, onNarrow, onHeal: () => { healed++; },+ onFailure: failure => { failed.push(failure); if (!failure.reason.startsWith('revision_conflict:')) failedNow++; } });+ rebuilt += batch.rebuilt;+ superseded += batch.superseded;+ const progressed = batch.rebuilt + healed + failedNow;+ projectionDrainWork.batches++;+ projectionDrainWork.inline_pages += progressed;+ if (progressed === 0) break;+ tried += progressed;+ opts.progress?.tick(progressed, `${rebuilt} rebuilt, ${failed.length} failed`);+ }+ };try {// Wave B: pages stored in segments first, through their own route, each queued row listed once per run. A row// another process's live claim holds is not listed; one that row's build leaves unsealed stays queued.@@ -958,24 +1026,9 @@ export async function drainProjections(engine: BrainEngine, opts: { limit?: numbtried += batch.listed.length;opts.progress?.tick(batch.listed.length, `${rebuilt} rebuilt, ${failed.length} failed`);}- // Continue while a batch rebuilt, removed, failed or durably deferred a- // row; stop on a no-op batch. A bare `superseded` row (another worker- // consumed it) is not progress, so one row can never spin the drain.- while (opts.limit === undefined || tried < opts.limit) {- let healed = 0;- let failedNow = 0;- const batch = await rebuildPendingPageProjections(engine, Math.min(DRAIN_BATCH, (opts.limit ?? Infinity) - tried),- { notAfter: runStart, reasons: opts.reasons, media: opts.media, walk, onHeal: () => { healed++; },- onFailure: failure => { failed.push(failure); if (!failure.reason.startsWith('revision_conflict:')) failedNow++; } });- rebuilt += batch.rebuilt;- superseded += batch.superseded;- const progressed = batch.rebuilt + healed + failedNow;- projectionDrainWork.batches++;- projectionDrainWork.inline_pages += progressed;- if (progressed === 0) break;- tried += progressed;- opts.progress?.tick(progressed, `${rebuilt} rebuilt, ${failed.length} failed`);- }+ // #9192: pages search does not answer for yet come first, then sealable images, then pages it answers for+ // from their pre-upgrade index.+ await rebuildInline('waiting');if (opts.media && (opts.limit === undefined || tried < opts.limit)) {const { sealImageProjections } = await import('./image-seal.ts');seal = await sealImageProjections(engine, { limit: opts.limit === undefined ? undefined : opts.limit - tried, notAfter: runStart });@@ -984,6 +1037,7 @@ export async function drainProjections(engine: BrainEngine, opts: { limit?: numbawait noteProjectionRebuilds(engine, seal.sealed, async () => !(await engine.executeRaw(`SELECT 1 FROM page_projection_jobs j ${probes}WHERE j.reason<>'rebuild_failed' LIMIT 1`)).length);}+ await rebuildInline('answering');} finally {opts.progress?.finish();}diff --git a/src/core/page-state/rebuild-estimate.ts b/src/core/page-state/rebuild-estimate.tsindex c576246876..30b6c30429 100644--- a/src/core/page-state/rebuild-estimate.ts+++ b/src/core/page-state/rebuild-estimate.ts@@ -13,12 +13,25 @@ export const TEXT_REBUILD_PAGES_PER_SECOND = 80;*/export const TEXT_REBUILD_ROUND_TRIPS_PER_PAGE = 36;-/** How long a text rebuild of `pages` takes at a measured round trip of `rttMs`, in words. */-export function textRebuildDuration(pages: number, rttMs: number): string {- const minutes = Math.max(1, Math.ceil(pages * (1 / TEXT_REBUILD_PAGES_PER_SECOND + TEXT_REBUILD_ROUND_TRIPS_PER_PAGE * rttMs / 1000) / 60));+/** How long a text rebuild of `pages` takes at a measured round trip of `rttMs`, `concurrency` pages at once, in words. */+export function textRebuildDuration(pages: number, rttMs: number, concurrency = 1): string {+ const minutes = Math.max(1, Math.ceil(pages * (1 / TEXT_REBUILD_PAGES_PER_SECOND + TEXT_REBUILD_ROUND_TRIPS_PER_PAGE * rttMs / 1000) / concurrency / 60));return minutes >= 120 ? `${Math.round(minutes / 60)} hour(s)` : `${minutes} minute(s)`;}+/**+ * #9192: pages `gbrain projections drain` rebuilds at once. PGLite has one connection, so 1. On Postgres the ceiling+ * leaves two of the pool's connections free, and the default is at most 4 so a drain cannot take a small hosted+ * database's connections from a running server; `--concurrency` raises it to the ceiling.+ */+export const DEFAULT_DRAIN_CONCURRENCY = 4;+export function drainConcurrency(engine: Pick<BrainEngine, 'kind'>): { default: number; max: number } {+ if (engine.kind !== 'postgres') return { default: 1, max: 1 };+ const poolMax = (engine as { sql?: { options?: { max?: unknown } } }).sql?.options?.max;+ const max = Math.max(1, (typeof poolMax === 'number' && Number.isSafeInteger(poolMax) ? poolMax : 10) - 2);+ return { default: Math.min(DEFAULT_DRAIN_CONCURRENCY, max), max };+}+/** The median of five `SELECT 1` round trips to the engine, in milliseconds. */export async function measuredRoundTripMs(engine: Pick<BrainEngine, 'executeRaw'>): Promise<number> {const samples: number[] = [];diff --git a/src/core/schema-migrations/v153-verified-text-projection-activation.ts b/src/core/schema-migrations/v153-verified-text-projection-activation.tsindex 0d772639f3..bef07b1720 100644--- a/src/core/schema-migrations/v153-verified-text-projection-activation.ts+++ b/src/core/schema-migrations/v153-verified-text-projection-activation.ts@@ -1,22 +1,26 @@import type { Migration } from './types.ts';import { PAGE_PROJECTION_ACTIVATION_SQL, PAGE_PROJECTION_SCHEMA_SQL } from '../page-state/projection-schema.ts';import { ACTIVATION_SERVED_REVISION } from '../search/safe-chunks.ts';-import { measuredRoundTripMs, textRebuildDuration } from '../page-state/rebuild-estimate.ts';+import { drainConcurrency, measuredRoundTripMs, textRebuildDuration } from '../page-state/rebuild-estimate.ts';import { migrationNotice } from './helpers.ts';export const v153: Migration = {version: 153, name: 'verified_text_projection_activation', idempotent: true, sql: PAGE_PROJECTION_SCHEMA_SQL + PAGE_PROJECTION_ACTIVATION_SQL,// #9192: say once what search keeps serving, what waits for the rebuild, and how long it takes on this database.handler: async engine => {- const [row] = await engine.executeRaw<{ n: number; served: number }>(`SELECT count(*)::int AS n,- count(*) FILTER (WHERE p.text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid)::int AS served+ const [row] = await engine.executeRaw<{ n: number; served: number; text: number }>(`SELECT count(*)::int AS n,+ count(*) FILTER (WHERE p.text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid)::int AS served,+ count(*) FILTER (WHERE p.page_kind IS DISTINCT FROM 'image')::int AS textFROM page_projection_jobs j JOIN sources s ON s.incarnation = j.source_incarnation JOIN pages p ON p.source_id = s.id AND p.slug = j.slugWHERE j.reason = 'protocol_activation'`);- const n = Number(row?.n ?? 0), served = Number(row?.served ?? 0);+ const n = Number(row?.n ?? 0), served = Number(row?.served ?? 0), text = Number(row?.text ?? 0);if (n === 0) return;const rtt = await measuredRoundTripMs(engine);+ const width = drainConcurrency(engine);+ const atOnce = width.default > 1 ? `, ${width.default} pages at once` : '';+ const faster = width.max > width.default ? ` (about ${textRebuildDuration(text, rtt, width.max)} with gbrain projections drain --concurrency ${width.max})` : '';const waiting = n - served ? `; ${n - served} (images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker) stay out of search until rebuilt` : '';migrationNotice(` [153] ${n} page(s) queued for a keyless search index rebuild (no embedding calls). Search keeps answering from the index ${served} of them already had${waiting}. ` +- `At the ${rtt} ms round trip measured to this database the rebuild takes about ${textRebuildDuration(n, rtt)}: gbrain projections drain. Progress: gbrain doctor --only text_projection_readiness\n`);+ `At the ${rtt} ms round trip measured to this database, rebuilding its ${text} Markdown and code page(s) takes about ${textRebuildDuration(text, rtt, width.default)}${atOnce}, pages out of search first: gbrain projections drain${faster}. Progress: gbrain doctor --only text_projection_readiness\n`);},};diff --git a/test/fixtures/goldens/migrations/records.json b/test/fixtures/goldens/migrations/records.jsonindex 7b364b7b99..63475f7c1e 100644--- a/test/fixtures/goldens/migrations/records.json+++ b/test/fixtures/goldens/migrations/records.json@@ -66,6 +66,7 @@"checkpointKey": "337ef776b18f428542093e27c4d570249d6a5856e5a17e4499d26a22877af34b","chunkSearchVectorTriggerFnSql": "9ac61adf64b64fdef7c0b786d093bc7aeb51c561fd680823d2fc4fc720b2bc33","clampOAuthTokenTtls": "b1e94ea8f5b3c3870a56b0f6d00d0a274c2e9dcad912b828848aec250732645f",+ "drainConcurrency": "cc9ad7c1c52f503c765eff106f9ac06ab3cb926d2dbced37ebc9a8682cdbada7","dropInvalidConcurrentIndex": "6864d844b7dfb2fbecdd0fb9c73e06248473376feb794b3fe58f036c70d2d44c","enableRlsSql": "b7901b3d5231fcfacd3260ce0f59d4701d8c0bee902808ff48b1d8affbd444b6","ensureFactsFtsIndex": "95e723314ac30cea2db48e0e167c554ef1fa06df80cae765faa2a080a3a655c2",@@ -98,7 +99,7 @@"sqlQueryForEngine": "71fb1ccec1b22cdc1662876dbd8c17ed81e9ab83485c6965cdf5caa4fc4a4ab5","swapIdentityIndex": "84ac62b697a16d244b2e0eecc962fb0558a24c2a26123bc7e77f1c7bec7c1066","tableRows": "906d2413b13d76a06d5910885e47ed5c544be4ff3a2266e83146715721b24270",- "textRebuildDuration": "cffe5e296f8dbdd2552a4993d0e6dbf6ddef5604fef75dc1d327d4172b66211e",+ "textRebuildDuration": "72e388f0ffa80b1892a314ec7690ed196259ec396ddda3c729662cc17313ed45","verifyProjectionStatistics": "0d231b04ecfbc5a4109d6fca9567bf4e8c9e9db6c3fbbb35212e026579ed6f67","widenIdentityIndex": "86a6176bd7637d964dbc1d3cd3b1bdfa3ca0161e1ee248c37d7380a1471282e6","withSchemaLockTimeout": "95605bd3004955e11e936c7a4aa6889319b401098abf9db18ef3e48da2c3fc59"@@ -2323,12 +2324,13 @@"form": "arrow","helpers": ["ACTIVATION_SERVED_REVISION",+ "drainConcurrency","measuredRoundTripMs","migrationNotice","quietMigrationNotices","textRebuildDuration"],- "sha256": "4db3224e2165b8d654bb19cdc8653840ac363f2d9ac1ea6f910304784ceb4cf2"+ "sha256": "71f3c55940d2df4e75ea88bc106f433c4973855cc92a0e8f6ff26bccc76c6c27"},"idempotent": true,"name": "verified_text_projection_activation",diff --git a/test/parallel-projection-drain-9192.test.ts b/test/parallel-projection-drain-9192.test.tsnew file mode 100644index 0000000000..2ec3b1d58f--- /dev/null+++ b/test/parallel-projection-drain-9192.test.ts@@ -0,0 +1,154 @@+/**+ * #9192: 'gbrain projections drain' rebuilds the pages search does not answer for first, then the pages it still answers+ * for from their pre-upgrade index, and on Postgres rebuilds several pages at once. A refused extra connection lowers+ * the count and the drain finishes; PGLite stays at one page at a time.+ */+import { describe, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import postgres from '#postgres';+import type { BrainEngine } from '../src/core/engine.ts';+import { PostgresEngine } from '../src/core/postgres-engine.ts';+import { PAGE_PROJECTION_ACTIVATION_SQL } from '../src/core/page-state/projection-schema.ts';+import { drainProjections, rebuildPendingPageProjections } from '../src/core/page-state/projections.ts';+import { drainConcurrency, textRebuildDuration } from '../src/core/page-state/rebuild-estimate.ts';+import { ACTIVATION_SERVED_REVISION } from '../src/core/search/safe-chunks.ts';+import { installFixtureChunks } from './helpers/page-projection.ts';+import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';+import { testBackends } from './helpers/test-backends.ts';++const ANSWERING = Array.from({ length: 8 }, (_, i) => `notes/answering-${i}`);+const WAITING = Array.from({ length: 5 }, (_, i) => `notes/waiting-${i}`);+const body = (slug: string) => `# ${slug}\n\nThe okapi theremin note for ${slug}, with a second sentence so the chunk has some length.`;+const statements = (sql: string) => sql.split(';').map(s => s.trim()).filter(Boolean);++/** Pages activated by v153: ANSWERING keep serving their pre-upgrade index, WAITING (indexed before the safe-chunk fence) are out of search. */+async function seed(engine: BrainEngine) {+ for (const slug of [...ANSWERING, ...WAITING]) {+ await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: body(slug) });+ await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: body(slug) }]);+ }+ await engine.executeRaw('UPDATE pages SET chunker_version = 3 WHERE slug = ANY($1::text[])', [WAITING]);+ for (const statement of statements(PAGE_PROJECTION_ACTIVATION_SQL)) await engine.executeRaw(statement);+}+const states = async (engine: BrainEngine) => Object.fromEntries((await engine.executeRaw<{ slug: string; state: string }>(`SELECT slug,+ CASE WHEN text_projection_revision = knowledge_revision THEN 'rebuilt'+ WHEN text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid THEN 'answering' ELSE 'waiting' END AS state+ FROM pages ORDER BY slug`)).map(r => [r.slug, r.state]));+const chunks = async (engine: BrainEngine) => engine.executeRaw<{ slug: string; chunk_index: number; text: string }>(`SELECT p.slug, c.chunk_index, md5(c.chunk_text) AS text+ FROM pages p JOIN content_chunks c ON c.page_id = p.id ORDER BY p.slug, c.chunk_index`);++for (const backend of testBackends()) {+ const url = backend === 'postgres' ? process.env.DATABASE_URL : undefined;+ describe(`#9192 parallel projection drain (${backend})`, () => {+ test('every page out of search is rebuilt before any page that still answers', async () => {+ const { engine, close } = await isolatedSharedSkillsEngine(url);+ try {+ await seed(engine);+ const before = await states(engine);+ expect(WAITING.map(s => before[s])).toEqual(WAITING.map(() => 'waiting'));+ expect(ANSWERING.map(s => before[s])).toEqual(ANSWERING.map(() => 'answering'));+ const first = await drainProjections(engine, { limit: WAITING.length, concurrency: drainConcurrency(engine).default });+ expect(first.rebuilt).toBe(WAITING.length);+ const mid = await states(engine);+ expect(WAITING.map(s => mid[s])).toEqual(WAITING.map(() => 'rebuilt'));+ expect(ANSWERING.map(s => mid[s])).toEqual(ANSWERING.map(() => 'answering'));+ const rest = await drainProjections(engine, { concurrency: drainConcurrency(engine).default });+ expect(rest).toMatchObject({ rebuilt: ANSWERING.length, failed: [], remaining: 0 });+ expect(Object.values(await states(engine)).every(s => s === 'rebuilt')).toBe(true);+ } finally { await close(); }+ }, 120_000);++ test('several pages at once build the same chunks as one at a time', async () => {+ const one = await isolatedSharedSkillsEngine(url);+ const many = await isolatedSharedSkillsEngine(url);+ try {+ await seed(one.engine);+ await seed(many.engine);+ expect(await drainProjections(one.engine, { concurrency: 1 })).toMatchObject({ rebuilt: 13, failed: [], remaining: 0 });+ // PGLite has one connection: a requested width of 4 still rebuilds one page at a time.+ expect(await drainProjections(many.engine, { concurrency: 4 })).toMatchObject({ rebuilt: 13, failed: [], remaining: 0 });+ expect(await chunks(many.engine)).toEqual(await chunks(one.engine));+ } finally { await one.close(); await many.close(); }+ }, 120_000);+ });+}++describe('drain width', () => {+ test('PGLite drains one page at a time; Postgres defaults to at most 4 and keeps two pool connections free', () => {+ expect(drainConcurrency({ kind: 'pglite' } as BrainEngine)).toEqual({ default: 1, max: 1 });+ const pg = (max: number) => ({ kind: 'postgres', sql: { options: { max } } }) as unknown as BrainEngine;+ expect(drainConcurrency(pg(10))).toEqual({ default: 4, max: 8 });+ expect(drainConcurrency(pg(4))).toEqual({ default: 2, max: 2 });+ expect(drainConcurrency(pg(2))).toEqual({ default: 1, max: 1 });+ expect(drainConcurrency(pg(30))).toEqual({ default: 4, max: 28 });+ });+ test('the estimate divides by the pages rebuilt at once', () => {+ expect(textRebuildDuration(50_000, 18, 1)).toBe('9 hour(s)');+ expect(textRebuildDuration(50_000, 18, 4)).toBe('2 hour(s)');+ });+});++const pgUrl = testBackends().includes('postgres') ? process.env.DATABASE_URL : undefined;+describe.skipIf(!pgUrl)('a refused connection (Postgres)', () => {+ test('a database that admits only 2 connections lowers the width and the drain finishes', async () => {+ const { engine, close } = await isolatedSharedSkillsEngine(pgUrl);+ const role = `gbrain_test_cap_${randomUUID().replace(/-/g, '').slice(0, 12)}`;+ const database = new URL((engine as PostgresEngine).connectedDatabaseUrl!).pathname.slice(1);+ const admin = postgres(pgUrl!, { max: 1, prepare: false });+ const capped = new PostgresEngine();+ try {+ await seed(engine);+ await admin.unsafe(`CREATE ROLE ${role} LOGIN BYPASSRLS PASSWORD 'cap' CONNECTION LIMIT 2`);+ await engine.executeRaw(`GRANT USAGE ON SCHEMA public TO ${role}; GRANT ALL ON ALL TABLES IN SCHEMA public TO ${role};+ GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO ${role}; GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public TO ${role}`);+ const url = new URL(pgUrl!);+ url.username = role;+ url.password = 'cap';+ url.pathname = `/${database}`;+ await capped.connect({ database_url: url.toString(), poolSize: 6 });+ expect(drainConcurrency(capped)).toEqual({ default: 4, max: 4 });+ const narrowed: number[] = [];+ const result = await drainProjections(capped, { concurrency: 4, onNarrow: n => { narrowed.push(n); } });+ expect(result).toMatchObject({ rebuilt: 13, failed: [], remaining: 0 });+ expect(narrowed.length).toBeGreaterThan(0);+ expect(Math.min(...narrowed)).toBeLessThanOrEqual(2);+ expect(Object.values(await states(engine)).every(s => s === 'rebuilt')).toBe(true);+ } finally {+ await capped.disconnect();+ await close();+ await admin.unsafe(`DROP ROLE IF EXISTS ${role}`);+ await admin.end();+ }+ }, 120_000);+});++describe.skipIf(!pgUrl)('a dropped connection after other workers finished (Postgres)', () => {+ test('the lowered width counts lost connections, not workers that ran out of pages', async () => {+ const { engine, close } = await isolatedSharedSkillsEngine(pgUrl);+ try {+ await seed(engine);+ const rebuiltWaiting = async () => (await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM pages+ WHERE slug = ANY($1::text[]) AND text_projection_revision = knowledge_revision`, [WAITING]))[0]!.n;+ const original = engine.transaction.bind(engine);+ let calls = 0;+ let dropped!: () => void;+ const afterDrop = new Promise<void>(resolve => { dropped = resolve; });+ engine.transaction = (async (fn: Parameters<BrainEngine['transaction']>[0]) => {+ const call = ++calls;+ if (call === 3) {+ const deadline = Date.now() + 60_000;+ while (await rebuiltWaiting() < 2 && Date.now() < deadline) await new Promise(r => setTimeout(r, 20));+ await new Promise(r => setTimeout(r, 0));+ dropped();+ throw Object.assign(new Error('write CONNECTION_ENDED 127.0.0.1:5432'), { code: 'CONNECTION_ENDED' });+ }+ if (call === 4) await afterDrop;+ return original(fn);+ }) as BrainEngine['transaction'];+ const narrowed: number[] = [];+ const batch = await rebuildPendingPageProjections(engine, 4, { served: 'waiting', concurrency: 4, onNarrow: n => { narrowed.push(n); } });+ expect(batch.rebuilt).toBe(4);+ expect(narrowed).toEqual([3]);+ } finally { await close(); }+ }, 120_000);+});diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txtindex f839b0c171..4fada9c6e7 100644--- a/test/postgres-unit-arms.txt+++ b/test/postgres-unit-arms.txt@@ -408,6 +408,7 @@ test/page-spend-reservation.test.tstest/page-state-engine.test.tstest/pages-source-created-idx-7379.test.tstest/pages-status-7179.test.ts+test/parallel-projection-drain-9192.test.tstest/pathological/a-36-facts-batch-partial-refusal.test.tstest/pathological/a-42-atom-dot-led-slug.test.tstest/pathological/a-51-conversation-batch-booking.test.tsdiff --git a/test/projection-rebuild-progress-9192.test.ts b/test/projection-rebuild-progress-9192.test.tsindex 6d31b81aa6..cfdca1ada1 100644--- a/test/projection-rebuild-progress-9192.test.ts+++ b/test/projection-rebuild-progress-9192.test.ts@@ -7,6 +7,7 @@ import { describe, expect, spyOn, test } from 'bun:test';import type { BrainEngine } from '../src/core/engine.ts';import { PAGE_PROJECTION_ACTIVATION_SQL } from '../src/core/page-state/projection-schema.ts';import { rebuildPendingPageProjections } from '../src/core/page-state/projections.ts';+import { drainConcurrency } from '../src/core/page-state/rebuild-estimate.ts';import { v153 } from '../src/core/schema-migrations/v153-verified-text-projection-activation.ts';import { dispatchToolCall } from '../src/mcp/dispatch.ts';import { installFixtureChunks } from './helpers/page-projection.ts';@@ -56,7 +57,12 @@ for (const backend of testBackends()) {try { await v153.handler!(engine); } finally { spy.mockRestore(); }const lines = writes.filter(w => w.includes('[153]'));expect(lines).toHaveLength(1);- expect(lines[0]).toMatch(/^ {2}\[153\] 2 page\(s\) queued for a keyless search index rebuild \(no embedding calls\)\. Search keeps answering from the index 0 of them already had; 2 \(images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker\) stay out of search until rebuilt\. At the [\d.]+ ms round trip measured to this database the rebuild takes about 1 minute\(s\): gbrain projections drain\. Progress: gbrain doctor --only text_projection_readiness\n$/);+ expect(lines[0]).toMatch(/^ {2}\[153\] 2 page\(s\) queued for a keyless search index rebuild \(no embedding calls\)\. Search keeps answering from the index 0 of them already had; 2 \(images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker\) stay out of search until rebuilt\. At the [\d.]+ ms round trip measured to this database, rebuilding its 2 Markdown and code page\(s\) takes about 1 minute\(s\)(?<atOnce>, \d+ pages at once)?, pages out of search first: gbrain projections drain(?<faster> \(about 1 minute\(s\) with gbrain projections drain --concurrency \d+\))?\. Progress: gbrain doctor --only text_projection_readiness\n$/);+ const width = drainConcurrency(engine);+ const groups = lines[0]!.match(/(?<atOnce>, (?<n>\d+) pages at once)?, pages out of search first: gbrain projections drain(?<faster> \(about 1 minute\(s\) with gbrain projections drain --concurrency (?<max>\d+)\))?\./)!.groups!;+ expect(groups.n === undefined ? 1 : Number(groups.n)).toBe(width.default);+ expect(groups.max === undefined ? width.default : Number(groups.max)).toBe(width.max);+ if (engine.kind === 'pglite') expect(groups.atOnce ?? groups.faster).toBeUndefined();}, url), 120_000);});}diff --git a/test/search-during-upgrade-rebuild-9192.test.ts b/test/search-during-upgrade-rebuild-9192.test.tsindex dc5eedfc2a..f4fa7b44a1 100644--- a/test/search-during-upgrade-rebuild-9192.test.ts+++ b/test/search-during-upgrade-rebuild-9192.test.ts@@ -31,7 +31,7 @@ const drain = async (engine: BrainEngine) => { while ((await rebuildPendingPagePconst count = async (engine: BrainEngine, where: string) =>Number((await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM pages p WHERE ${where}`))[0]?.n);-async function activated(run: (engine: BrainEngine) => Promise<void>, databaseUrl?: string, opts: { belowFence?: string; preRevision?: boolean; code?: string; extra?: Record<string, { body: string; frontmatter?: Record<string, unknown> }> } = {}) {+async function activated(run: (engine: BrainEngine) => Promise<void>, databaseUrl?: string, opts: { belowFence?: string; preRevision?: boolean; code?: string; image?: string; extra?: Record<string, { body: string; timeline?: string; frontmatter?: Record<string, unknown> }> } = {}) {const { engine, close } = await isolatedSharedSkillsEngine(databaseUrl);try {for (const [i, slug] of SLUGS.entries()) {@@ -39,11 +39,12 @@ async function activated(run: (engine: BrainEngine) => Promise<void>, databaseUrawait installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT, embedding: vector(i) }]);}for (const [slug, page] of Object.entries(opts.extra ?? {})) {- await engine.putPage(slug, { type: 'note', title: 'Okapi theremin', compiled_truth: page.body, ...(page.frontmatter ? { frontmatter: page.frontmatter } : {}) });+ await engine.putPage(slug, { type: 'note', title: 'Okapi theremin', compiled_truth: page.body, ...(page.timeline ? { timeline: page.timeline } : {}), ...(page.frontmatter ? { frontmatter: page.frontmatter } : {}) });await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT }]);}if (opts.belowFence) await engine.executeRaw('UPDATE pages SET chunker_version = 3 WHERE slug = $1', [opts.belowFence]);if (opts.code) await engine.executeRaw("UPDATE pages SET page_kind = 'code', source_path = NULL WHERE slug = $1", [opts.code]);+ if (opts.image) await engine.executeRaw("UPDATE pages SET page_kind = 'image' WHERE slug = $1", [opts.image]);if (opts.preRevision) {// Pages written before v150 reach v153 with no knowledge_revision (the upgrade #9192 reported, from v145).await engine.executeRaw('ALTER TABLE pages ALTER COLUMN knowledge_revision DROP NOT NULL');@@ -139,12 +140,14 @@ for (const backend of testBackends()) {expect(served.map(r => r.slug)).toEqual(SLUGS);expect(await keywordSlugs(engine)).toEqual(SLUGS);expect((await probeProjectionReadiness(engine)).status).toBe('projection_pending');- expect((await projectionBacklog(engine)).pending).toBe(6);+ expect((await projectionBacklog(engine)).pending).toBe(8);}, url, { extra: {'notes/facts': { body: `${TEXT}\n\n<!--- gbrain:facts:begin -->\n| # | claim |\n|---|---|\n| 1 | okapi |\n<!--- gbrain:facts:end -->` },'notes/takes': { body: `${TEXT}\n\n<!--- gbrain:takes:begin -->\nokapi take\n<!--- gbrain:takes:end -->` },'notes/key': { body: [TEXT, ['-----BEGIN', ' RSA PRIVATE KEY', '-----'].join(''), 'MIIEexample', ['-----END', ' RSA PRIVATE KEY', '-----'].join('')].join('\n') },'notes/quarantined': { body: TEXT, frontmatter: { quarantine: 'junk' } },+ 'notes/timeline-key': { body: TEXT, timeline: ['- 2020-01-01: pasted', ['-----BEGIN', ' RSA PRIVATE KEY', '-----'].join(''), 'MIIEexample'].join('\n') },+ 'notes/timeline-takes': { body: TEXT, timeline: '- 2020-01-01: noted\n\n<!--- gbrain:takes:begin -->\nokapi take\n<!--- gbrain:takes:end -->' },} }), 120_000);test('a restricted caller gets no marker page it cannot read, and a soft-deleted marker page is not returned', () => accessActivated(async (engine, seen) => {@@ -215,5 +218,14 @@ for (const backend of testBackends()) {expect(line).toContain('Search keeps answering from the index 1 of them already had; 1 (images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker) stay out of search until rebuilt.');expect(line).toContain('gbrain projections drain');}, url, { belowFence: SLUGS[0] }), 120_000);++ test('the v153 estimate times the Markdown and code pages the drain rebuilds, not images', () => activated(async engine => {+ const writes: string[] = [];+ const spy = spyOn(process.stderr, 'write').mockImplementation(((chunk: string) => { writes.push(String(chunk)); return true; }) as typeof process.stderr.write);+ try { await v153.handler!(engine); } finally { spy.mockRestore(); }+ const line = writes.find(w => w.includes('[153]')) ?? '';+ expect(line).toContain('3 page(s) queued');+ expect(line).toContain('rebuilding its 2 Markdown and code page(s) takes about');+ }, url, { image: 'notes/okapi-image', extra: { 'notes/okapi-image': { body: TEXT } } }), 120_000);});}