Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(search): report synopsis-tier coverage and add gbrain reindex --synopsis (#8744)
garrytan62eeb1720a1b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ca390dd578..09d2b157c4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,21 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**A brain set to `search.mode tokenmax` now says when its pages are not on the synopsis tier, and `gbrain reindex --synopsis` puts them there.** Import and sync embed tokenmax pages at the free title tier and leave per-chunk synopses to the `contextual_reindex_per_chunk` job, but nothing ever queued that job, so a brain could search on title-tier vectors for weeks while reporting tokenmax.++### What changed++| Situation | Before | Now |+|---|---|---|+| `gbrain doctor` on a brain whose configured tier is `per_chunk_synopsis` | nothing about it | `synopsis_tier_coverage` gives chunk and page coverage, the large pages left on the title tier and the last sweep job; it warns below 50% of chunks and names `gbrain reindex --synopsis`. The remote doctor runs it too, confined to the caller's sources |+| `gbrain search modes` | the configured tier only | one line with the synopsis-tier coverage (local only; the `search_modes` op leaves it out) |+| `gbrain reindex --synopsis` | did not exist | previews pages, chunks, synopsis tokens and dollars on the configured synopsis model plus the re-embed, and spends nothing |+| `gbrain reindex --synopsis --yes [--max-usd <usd>]` | did not exist | queues one `contextual_reindex_per_chunk` job per page under one spend cap the worker enforces across them, runs them in-process and reports the new coverage. Running it again resumes: finished pages are skipped and queued jobs are picked up |++Nothing is scheduled automatically and no default changes. Inline pages over 1 MiB keep the title tier, as before. Repo-less brains work the same way: the sweep reads and writes the database only.+## [0.60.595.0] - 2026-11-06**A `memory_recall` that names sources reads as that scope, never as the attested person, so a shared turn can no longer recall one member's private facts into a shared channel.** `memory_recall` gains `source_id` and `source_ids`. Named, the read drops the person's own recent facts and every member-page fact, keeps restricted pages hidden and never relaxes `visibility: private`. The contract for it is the new `docs/contracts/qm-memory-provider/v2.json`; `v1.json` stays byte-identical and a test holds it there. Three smaller provider fixes ride along.diff --git a/scripts/serial-files.tsv b/scripts/serial-files.tsvindex 51d08ce86c..893bcd2e40 100644--- a/scripts/serial-files.tsv+++ b/scripts/serial-files.tsv@@ -460,3 +460,4 @@ test/search/identifier-probe.serial.test.ts R1 R1: mutates process.env (GBRAIN_Htest/search/slug-handles.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME)test/launcher-lanes-bunfig.serial.test.ts global-state global-state: spawns gbrain serve subprocesses against one shared PGLite brain and its data-dir lock; withEnv scopes HOME, GBRAIN_HOME and CODEX_HOME around the bootstrap and setup writerstest/extraction-review-door.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME) around real git subprocesses against tmp repos for the page-door review decisions+test/synopsis-sweep-8744.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME, GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL)diff --git a/skills/migrations/v0.40.3.0.md b/skills/migrations/v0.40.3.0.mdindex de01a2fad7..363b0e741e 100644--- a/skills/migrations/v0.40.3.0.md+++ b/skills/migrations/v0.40.3.0.md@@ -74,22 +74,18 @@ worker) offers to submit `gbrain reindex --markdown` as a Minion job.Non-TTY callers get a paste-ready hint to stderr. Suppress entirelywith `GBRAIN_NO_MODE_SWITCH_UX=1`.-If the user skipped or declined that prompt, do not start the reindex-yourself. Re-embedding every markdown page calls the embedding provider (and,-under `tokenmax`, a chat model for each chunk), so it costs money. Preview it,-tell the user how many pages it covers, and run it only after they agree:+Import, sync and `gbrain reindex --markdown` embed tokenmax pages at the title+tier; only the synopsis sweep writes per-chunk synopses, and `gbrain doctor`+(`synopsis_tier_coverage`) and `gbrain search modes` report its coverage. Do not+start it yourself: each chunk costs a chat call plus its re-embed. Preview it,+tell the user the pages and cost, and run it only after they agree:```bash-gbrain reindex --markdown --dry-run # page count; changes nothing-# only after the user agrees:-gbrain reindex --markdown --yes-# or as a background job (needs a running `gbrain jobs work` worker; on PGLite add --follow):-gbrain jobs submit reindex --params '{"markdown":true}' --yes+gbrain reindex --synopsis # pages, chunks, tokens, dollars; changes nothing+# only after the user agrees (resumable; --max-usd caps the whole run):+gbrain reindex --synopsis --yes```-Without `--yes` (or `--max-usd <usd>`) a non-interactive run exits 3 with an-`ask_user` payload instead of spending; relay its `user_message`.-## Optional: per-source CR mode overridesIf you have multiple sources and want different CR tiers per source:diff --git a/src/cli/commands/reindex.ts b/src/cli/commands/reindex.tsindex 7e9a99275d..868ac43f16 100644--- a/src/cli/commands/reindex.ts+++ b/src/cli/commands/reindex.ts@@ -48,6 +48,17 @@ export async function run(engine: BrainEngine, args: string[]): Promise<void> {}return;}+ if (args.includes('--synopsis')) {+ const { runReindexSynopsis } = await import('../../commands/reindex-synopsis.ts');+ try {+ await runReindexSynopsis(engine, args);+ } catch (e) {+ const { OperationError } = await import('../../core/ops/contract.ts');+ if (e instanceof OperationError) { setCliExitVerdict(writeCliError(e, 'reindex')); return; }+ throw e;+ }+ return;+ }if (args.includes('--segmented')) {await reindex.runReindexSegmented(engine, args);return;diff --git a/src/commands/doctor/checks/sync-search.ts b/src/commands/doctor/checks/sync-search.tsindex c81a29fb81..0795fca1ed 100644--- a/src/commands/doctor/checks/sync-search.ts+++ b/src/commands/doctor/checks/sync-search.ts@@ -13,6 +13,7 @@ import {checkLinkResolutionOpportunity,} from './calibration.ts';import { checkSyncConsolidation, checkCycleFreshness } from './consolidation-cycle.ts';+import { checkSynopsisTierCoverage } from './synopsis-coverage.ts';import {checkSyncFreshness,checkLinksExtractionLag,@@ -144,6 +145,9 @@ async function runSearchMode(ctx: DoctorContext): Promise<Check[]> {if (engine !== null) {progress.heartbeat('search_mode');checks.push(await checkSearchMode(engine));+ // #8744 synopsis_tier_coverage — configured per_chunk_synopsis tier vs the chunks that carry it.+ progress.heartbeat('synopsis_tier_coverage');+ checks.push(await checkSynopsisTierCoverage(engine));// issue #1777 — hidden_by_search_policy: chunked pages withheld from default// search by the hard-exclude prefix policy (audit the surviving excludes).progress.heartbeat('hidden_by_search_policy');@@ -236,6 +240,7 @@ export const searchModeEntry: DoctorEntry = {name: 'search_mode',emits: ['search_mode',+ 'synopsis_tier_coverage','hidden_by_search_policy','eval_drift','reranker_health',diff --git a/src/commands/doctor/checks/synopsis-coverage.ts b/src/commands/doctor/checks/synopsis-coverage.tsnew file mode 100644index 0000000000..82029ed2a6--- /dev/null+++ b/src/commands/doctor/checks/synopsis-coverage.ts@@ -0,0 +1,61 @@+/**+ * synopsis_tier_coverage (#8744): the configured contextual-retrieval tier is+ * per_chunk_synopsis (search.mode tokenmax, or a source override) but import+ * and sync embed pages at the title tier and nothing schedules the synopsis+ * sweep, so a brain can search on title-tier vectors for weeks while it+ * reports tokenmax. Warns when the synopsis tier is missing from most eligible+ * chunks, with the coverage and the preview of the explicit sweep+ * (`gbrain reindex --synopsis`). Database reads only: repo-less brains are+ * covered the same way.+ */+import type { BrainEngine } from '../../../core/engine.ts';+import type { Check } from '../../doctor.ts';+import { agentFix, checkError, infoCheck } from '../check-fix.ts';+++export async function checkSynopsisTierCoverage(engine: BrainEngine, opts: { sourceIds?: string[] } = {}): Promise<Check> {+ try {+ const sweep = await import('../../../core/synopsis-sweep.ts');+ const config = await sweep.readSynopsisTierConfig(engine);+ if (!config.configured) {+ return infoCheck('synopsis_tier_coverage', `Not applicable: the configured contextual-retrieval tier is ${config.kill_switch ? 'off (kill switch)' : config.global_tier}, not per_chunk_synopsis.`,+ 'not_applicable', undefined, { configured: false, global_tier: config.global_tier });+ }+ const { embeddingsDisabled } = await import('../../../core/embedding-disabled.ts');+ if (await embeddingsDisabled(engine)) {+ return infoCheck('synopsis_tier_coverage', 'Not applicable: embeddings are disabled on this brain, so no tier applies (keyword search keeps working).', 'disabled_by_choice', undefined, { configured: true });+ }+ const coverage = await sweep.readSynopsisCoverage(engine, { sourceIds: opts.sourceIds, config });+ const last = opts.sourceIds ? null : await sweep.readLastSynopsisSweep(engine);+ const sweepNote = last === null ? '' : last.jobs === 0 ? ' No synopsis sweep job has ever run.' : ` Last sweep job finished ${last.last_finished_at ?? 'never'} (${last.jobs} job(s) recorded).`;+ const details = {+ configured: true,+ global_tier: config.global_tier,+ synopsis_sources: opts.sourceIds ? config.synopsis_sources.filter(id => opts.sourceIds!.includes(id)) : config.synopsis_sources,+ eligible_pages: coverage.eligible_pages,+ eligible_chunks: coverage.eligible_chunks,+ synopsis_pages: coverage.synopsis_pages,+ synopsis_chunks: coverage.synopsis_chunks,+ coverage_pct: coverage.coverage_pct,+ page_coverage_pct: coverage.eligible_pages ? Math.floor((coverage.synopsis_pages / coverage.eligible_pages) * 1000) / 10 : null,+ large_title_pages: coverage.large_title_pages,+ large_title_chunks: coverage.large_title_chunks,+ ...(last ? { sweep_jobs: last.jobs, last_sweep_finished_at: last.last_finished_at } : {}),+ fix: sweep.SYNOPSIS_SWEEP_ARGV.join(' '),+ };+ const message = sweep.formatSynopsisCoverageLine(coverage) + sweepNote;+ const missingMost = coverage.coverage_pct !== null && coverage.coverage_pct < sweep.SYNOPSIS_COVERAGE_WARN_PCT;+ const pending = coverage.eligible_chunks > coverage.synopsis_chunks;+ return {+ name: 'synopsis_tier_coverage',+ status: missingMost ? 'warn' : 'ok',+ message,+ details,+ ...(pending ? { fix: agentFix([...sweep.SYNOPSIS_SWEEP_ARGV],+ 'Previews the per-chunk synopsis sweep (pages, chunks, tokens and dollars on the configured synopsis model) and changes nothing; it runs only with --yes, which is paid work the user approves.',+ 'synopsis_tier_coverage') } : {}),+ };+ } catch (e) {+ return checkError('synopsis_tier_coverage', 'check synopsis tier coverage', e);+ }+}diff --git a/src/commands/doctor/report-remote.ts b/src/commands/doctor/report-remote.tsindex a868a4cd91..4b244398a8 100644--- a/src/commands/doctor/report-remote.ts+++ b/src/commands/doctor/report-remote.ts@@ -66,6 +66,7 @@ import {multiSourceDriftNotVerified,} from './schema-pack-checks.ts';import { managedPersistenceEnabled } from '../../core/persistence/ownership.ts';+import { checkSynopsisTierCoverage } from './checks/synopsis-coverage.ts';import { brainScorePlanFix, checkError } from './check-fix.ts';import type { RenderContext } from '../../core/agent-output.ts';@@ -436,6 +437,7 @@ export async function doctorReportRemote(// - contextual_retrieval_mode IS NULL (mode never evaluated)// - synopsis-failures audit JSONL entries from the last 7 dayschecks.push(await checkContextualRetrievalCoverage(engine, { sourceIds: opts.sourceIds }));+ checks.push(await checkSynopsisTierCoverage(engine, { sourceIds: opts.sourceIds }));checks.push(await checkProjectionReadiness(engine, {sourceIds: opts.sourceIds,excludePrivate: await resolveExcludePrivatePages(engine, opts.remote),diff --git a/src/commands/reindex-synopsis.ts b/src/commands/reindex-synopsis.tsnew file mode 100644index 0000000000..9f58ec3be4--- /dev/null+++ b/src/commands/reindex-synopsis.ts@@ -0,0 +1,227 @@+/**+ * `gbrain reindex --synopsis` (#8744): the explicit, operator-run per-chunk+ * synopsis sweep for pages whose configured contextual-retrieval tier is+ * per_chunk_synopsis (search.mode tokenmax, or a source or page override).+ * Import and sync embed those pages at the free title tier and nothing else+ * schedules the synopsis tier, so this command is how it gets applied.+ *+ * Without `--yes` it previews and spends nothing: the pages and chunks still+ * on the title tier, and the token and dollar ceiling on the configured+ * synopsis model plus the re-embed. With `--yes` it queues one+ * `contextual_reindex_per_chunk` job per page (the existing job and handler),+ * all under one spend authorization whose cap (`--max-usd`, else the derived+ * cap) the worker enforces across every job, then drains them in this process.+ * It is resumable: pages already stamped per_chunk_synopsis are never+ * selected again, and a job left queued or running by an interrupted run is+ * picked up again through its idempotency key (page, source and text revision).+ * Database reads and writes only, so a repo-less brain works the same way.+ */+import type { BrainEngine } from '../core/engine.ts';+import { setCliExitVerdict } from '../core/cli-force-exit.ts';+import type { MinionJob } from '../core/minions/types.ts';+import {+ SYNOPSIS_SWEEP_ARGV,+ SYNOPSIS_SWEEP_JOB,+ formatSynopsisCoverageLine,+ planSynopsisSweep,+ readSynopsisCoverage,+ type SynopsisCoverage,+ type SynopsisSweepPlan,+} from '../core/synopsis-sweep.ts';++const TERMINAL = new Set(['completed', 'failed', 'dead', 'cancelled']);++export interface SynopsisSweepRun {+ group_id: string;+ cap_usd: number | null;+ job_ids: number[];+ /** Pages whose job at their current text revision already finished in an earlier run (not queued again). */+ already_attempted: number;+ synopsis: number;+ title: number;+ skipped: number;+ failed: number;+}++export interface ReindexSynopsisResult {+ mode: 'not_applicable' | 'preview' | 'run';+ reason?: string;+ plan?: SynopsisSweepPlan;+ coverage?: SynopsisCoverage;+ run?: SynopsisSweepRun;+ coverage_after?: SynopsisCoverage;+}++function flagValue(args: string[], flag: string): string | undefined {+ const at = args.indexOf(flag);+ return at >= 0 ? args[at + 1] : undefined;+}++function usd(value: number | null): string {+ return value === null ? 'unknown (no price for this model)' : `$${value.toFixed(2)}`;+}++function planJson(plan: SynopsisSweepPlan) {+ const { pages, config, ...rest } = plan;+ return { ...rest, global_tier: config.global_tier, synopsis_sources: config.synopsis_sources, pages: pages.length };+}++export async function runReindexSynopsis(engine: BrainEngine, args: string[], opts: { pollMs?: number } = {}): Promise<ReindexSynopsisResult> {+ const json = args.includes('--json');+ const say = (line: string) => { if (!json) console.log(line); };+ const emit = (result: ReindexSynopsisResult) => {+ if (json) {+ process.stdout.write(JSON.stringify({+ mode: result.mode,+ ...(result.reason ? { reason: result.reason } : {}),+ ...(result.plan ? { plan: planJson(result.plan) } : {}),+ ...(result.coverage ? { coverage: { ...result.coverage, config: undefined } } : {}),+ ...(result.run ? { run: result.run } : {}),+ ...(result.coverage_after ? { coverage_after: { ...result.coverage_after, config: undefined } } : {}),+ run_argv: [...SYNOPSIS_SWEEP_ARGV, '--yes'],+ }, null, 2) + '\n');+ }+ return result;+ };++ const limitRaw = flagValue(args, '--limit');+ const sourceId = flagValue(args, '--source');+ if ((args.includes('--limit') && !(limitRaw && /^\d+$/.test(limitRaw) && Number(limitRaw) > 0))+ || (args.includes('--source') && (!sourceId || sourceId.startsWith('--')))) {+ const error = args.includes('--limit') && !(limitRaw && /^\d+$/.test(limitRaw)) ? 'invalid --limit: expected a positive integer' : 'invalid --source: expected a value';+ if (json) process.stdout.write(JSON.stringify({ error }) + '\n');+ else process.stderr.write(`[reindex --synopsis] ${error}\n`);+ setCliExitVerdict(2);+ return { mode: 'not_applicable', reason: error };+ }++ const { embeddingsDisabled } = await import('../core/embedding-disabled.ts');+ if (await embeddingsDisabled(engine)) {+ const reason = 'Embeddings are disabled on this brain (keyless by choice), so there are no vectors to put on the synopsis tier; keyword search keeps working.';+ say(`[reindex --synopsis] Nothing to do. ${reason}`);+ return emit({ mode: 'not_applicable', reason });+ }++ const { resolveContextualSynopsisModel } = await import('../core/minions/handlers/contextual-reindex-per-chunk.ts');+ const { resolveSynopsisMaxTokens } = await import('../core/contextual-retrieval-service.ts');+ const synopsisModel = await resolveContextualSynopsisModel(engine);+ let embeddingPricePerMTok: number | null = null;+ let embeddingModel = 'unknown';+ try {+ const { getEmbeddingModel } = await import('../core/ai/gateway.ts');+ const { lookupEmbeddingPrice } = await import('../core/embedding-pricing.ts');+ embeddingModel = getEmbeddingModel();+ const price = lookupEmbeddingPrice(embeddingModel);+ if (price.kind === 'known') embeddingPricePerMTok = price.pricePerMTok;+ } catch { /* no configured embedding model: the embedding share stays unknown */ }++ const plan = await planSynopsisSweep(engine, {+ synopsisModel,+ synopsisMaxTokens: await resolveSynopsisMaxTokens(engine),+ ...(sourceId ? { sourceId } : {}),+ ...(limitRaw ? { limit: Number(limitRaw) } : {}),+ embeddingPricePerMTok,+ });+ if (!plan.config.configured) {+ const reason = `The configured contextual-retrieval tier is ${plan.config.kill_switch ? 'off (kill switch)' : plan.config.global_tier}, not per_chunk_synopsis, so no page needs the synopsis sweep. `+ + 'search.mode tokenmax (or search.contextual_retrieval per_chunk_synopsis, or gbrain sources set-cr-mode <id> per_chunk_synopsis) selects it.';+ say(`[reindex --synopsis] Nothing to do. ${reason}`);+ return emit({ mode: 'not_applicable', reason, plan });+ }+ const coverage = await readSynopsisCoverage(engine, { config: plan.config, ...(sourceId ? { sourceIds: [sourceId] } : {}) });+ const tierSource = plan.config.global_tier === 'per_chunk_synopsis' ? 'search mode' : `source override (${plan.config.synopsis_sources.join(', ')})`;+ say(`[reindex --synopsis] Configured tier: per_chunk_synopsis (${tierSource}).`);+ say(` ${formatSynopsisCoverageLine(coverage)}`);+ if (plan.pages.length === 0) {+ say(' Nothing to sweep: every eligible page is already on the synopsis tier.');+ return emit({ mode: 'preview', plan, coverage });+ }+ say(` The sweep would synopsize ${plan.pages.length} page(s), ${plan.chunks} chunk(s), on ${synopsisModel}:`);+ say(` synopsis input tokens up to ${plan.est_input_tokens} (every prompt uncached; the prompt cache usually serves most)`);+ say(` synopsis output tokens up to ${plan.est_output_tokens} (the output cap per chunk)`);+ say(` synopsis cost up to ${usd(plan.est_synopsis_usd)}`);+ say(` re-embed cost about ${usd(plan.est_embedding_usd)} (${embeddingModel})`);+ say(` total up to ${usd(plan.est_usd)}`);+ if (plan.remaining_pages > 0) say(` ${plan.remaining_pages} more pending page(s) are beyond --limit; a later run picks them up.`);++ if (!args.includes('--yes')) {+ say(` Preview only; nothing was queued or spent. Run it (paid, resumable): ${[...SYNOPSIS_SWEEP_ARGV, ...(sourceId ? ['--source', sourceId] : []), ...(limitRaw ? ['--limit', limitRaw] : []), '--yes'].join(' ')} [--max-usd <usd>]`);+ return emit({ mode: 'preview', plan, coverage });+ }++ const { requireConsent } = await import('../core/consent.ts');+ const { engineConsentEnv } = await import('../core/consent-cli.ts');+ const { jobSpendAuthorization, spendSubmitSummary } = await import('../core/minions/spend-authorization.ts');+ const argv = ['gbrain', 'reindex', ...args.filter(a => a !== '--yes' && a !== '--json')];+ const previewArgv = argv.filter((a, i) => a !== '--max-usd' && argv[i - 1] !== '--max-usd' && !a.startsWith('--max-usd='));+ const auth = await requireConsent({+ command: 'reindex --synopsis', effects: ['paid'], actor: 'agent',+ what: `Write per-chunk synopses for ${plan.pages.length} page(s) (${plan.chunks} chunk(s)) and re-embed them`,+ why: `The configured contextual-retrieval tier is per_chunk_synopsis, but these pages search on title-tier vectors until the sweep runs. Each chunk costs one ${synopsisModel} call plus its re-embed (up to ${usd(plan.est_usd)}).`,+ risk: 'Spends money with the chat and embedding providers. Search keeps working on the current vectors while it runs; a page whose synopsis is refused stays on the title tier.',+ user_message: `Put ${plan.pages.length} page(s) on the per-chunk synopsis tier for up to ${usd(plan.est_usd)}?`,+ argv, preview_argv: previewArgv,+ est_usd: plan.est_usd, args,+ }, engineConsentEnv(engine));+ const spend = jobSpendAuthorization(auth, { command: 'reindex --synopsis', of: plan.pages.length, argv, ...(plan.est_usd !== null ? { est_usd: plan.est_usd } : {}) });+ if (spend.cap_usd !== null && plan.est_usd !== null && plan.est_usd > spend.cap_usd) {+ say(` The $${spend.cap_usd.toFixed(2)} cap is below the ${usd(plan.est_usd)} ceiling, so the run may stop part-way; run the same command again (with a higher --max-usd) to continue.`);+ }++ const { MinionQueue } = await import('../core/minions/queue.ts');+ const queue = new MinionQueue(engine);+ await queue.ensureSchema();+ const jobs: MinionJob[] = [];+ for (const page of plan.pages) {+ jobs.push(await queue.add(SYNOPSIS_SWEEP_JOB, { page_slug: page.slug, expected_source_id: page.source_id },+ { idempotency_key: `${SYNOPSIS_SWEEP_JOB}:${page.source_id}:${page.slug}:${page.revision ?? 'unrevised'}` },+ { allowProtectedSubmit: true, spendAuthorization: spend }));+ }+ for (const line of spendSubmitSummary(spend, jobs, argv).lines) say(` ${line}`);+ const alreadyAttempted = jobs.filter(j => TERMINAL.has(j.status)).length;+ const pendingIds = jobs.filter(j => !TERMINAL.has(j.status)).map(j => j.id);+ if (alreadyAttempted) say(` ${alreadyAttempted} page(s) already had a finished sweep job at their current revision; they are not queued again until the page changes.`);++ if (pendingIds.length) {+ say(` Running ${pendingIds.length} job(s) in this process...`);+ const { MinionWorker } = await import('../core/minions/worker.ts');+ const { makeContextualReindexHandler } = await import('../core/minions/handlers/contextual-reindex-per-chunk.ts');+ const pollMs = opts.pollMs ?? 500;+ const worker = new MinionWorker(engine, { pollInterval: pollMs, healthCheckInterval: 0 });+ worker.register(SYNOPSIS_SWEEP_JOB, makeContextualReindexHandler({ engine }));+ const running = worker.start();+ let settled = false;+ const watch = (async () => {+ while (!settled) {+ await new Promise(resolve => setTimeout(resolve, pollMs));+ const rows = await engine.executeRaw<{ status: string }>('SELECT status FROM minion_jobs WHERE id = ANY($1::int[])', [pendingIds]);+ if (rows.every(r => TERMINAL.has(r.status))) { settled = true; worker.stop(); }+ }+ })();+ await running;+ settled = true;+ await watch.catch(() => undefined);+ if (worker.configurationError) {+ const { reportInlineWorkerConfiguration } = await import('./jobs-readiness.ts');+ reportInlineWorkerConfiguration(worker.configurationError);+ }+ }++ const finished = await engine.executeRaw<{ id: number | string; status: string; result: unknown; error_text: string | null }>(+ 'SELECT id, status, result, error_text FROM minion_jobs WHERE id = ANY($1::int[])', [jobs.map(j => j.id)]);+ const run: SynopsisSweepRun = { group_id: spend.group_id, cap_usd: spend.cap_usd, job_ids: jobs.map(j => j.id), already_attempted: alreadyAttempted,+ synopsis: 0, title: 0, skipped: 0, failed: 0 };+ for (const row of finished) {+ const result = (typeof row.result === 'string' ? JSON.parse(row.result) : row.result) as { mode_applied?: string } | null;+ if (row.status !== 'completed') { run.failed++; continue; }+ if (result?.mode_applied === 'per_chunk_synopsis') run.synopsis++;+ else if (result?.mode_applied === 'skipped') run.skipped++;+ else run.title++;+ }+ const coverageAfter = await readSynopsisCoverage(engine, { config: plan.config, ...(sourceId ? { sourceIds: [sourceId] } : {}) });+ say(` Done: ${run.synopsis} page(s) on the synopsis tier, ${run.title} on the title tier (synopsis refused), ${run.skipped} skipped, ${run.failed} failed.`);+ if (run.failed) say(` Failed jobs keep their reason: gbrain jobs list --status dead. Run the same command again to resume.`);+ say(` ${formatSynopsisCoverageLine(coverageAfter)}`);+ if (run.failed) setCliExitVerdict(1);+ return emit({ mode: 'run', plan, coverage, run, coverage_after: coverageAfter });+}diff --git a/src/commands/reindex.ts b/src/commands/reindex.tsindex 52bdb16ec2..8fdca8ea8f 100644--- a/src/commands/reindex.ts+++ b/src/commands/reindex.ts@@ -92,6 +92,7 @@ USAGEgbrain reindex --vectors [--dry-run] [--json]gbrain reindex --oversized [--limit N] [--dry-run] [--json] [--source <id>]gbrain reindex --segmented [--source <id>] [--slug <slug>] [--apply] [--json]+ gbrain reindex --synopsis [--source <id>] [--limit N] [--yes] [--max-usd USD] [--json]TARGETS (exactly one required)--markdown Re-chunk markdown pages whose chunker_version lags the@@ -121,10 +122,20 @@ TARGETS (exactly one required)only the segments that change, keeps the vectors ofunchanged chunk texts and leaves new texts to the embedsweep (gbrain embed --stale). Never runs on upgrade.+ --synopsis Put pages whose configured contextual-retrieval tier is+ per_chunk_synopsis (search.mode tokenmax) on that tier:+ import and sync embed them at the title tier and nothing+ else schedules the synopsis sweep. Previews by default+ (pages, chunks, tokens and dollars on the configured+ synopsis model; spends nothing). --yes queues one+ contextual_reindex_per_chunk job per page under one spend+ cap (--max-usd, else derived from the estimate) and runs+ them in this process. Resumable: run it again to continue.+ Inline pages over 1 MiB stay on the title tier.OPTIONS--type <t> --markdown only: restrict to one page type- --limit N Cap pages/chunks processed this run+ --limit N Cap pages/chunks processed this run (--synopsis: pages)--workers N --markdown and --multimodal: parallel writers per batch(--concurrency is an alias). --markdown on Postgresdefaults to 4 when more than 100 pages are pending;@@ -136,8 +147,10 @@ OPTIONSfirst (exit 3 without approval when not interactive;--dry-run, --no-embed and a keyless brain never ask);--multimodal skips its cost confirm- --max-usd USD --markdown: approve the paid run with this cost cap- --source <id> --aliases, --oversized and --segmented: restrict to one source+ --max-usd USD --markdown: approve the paid run with this cost cap;+ --synopsis: the cap shared by every queued job (with --yes)+ --source <id> --aliases, --oversized, --segmented and --synopsis: restrict+ to one source--slug <slug> --segmented only: one page--apply --segmented only: apply the previewed re-chunk--repo PATH --markdown only: brain repo override@@ -185,6 +198,7 @@ export function validateReindexModeScope(args: string[]): string | null {if (args.includes('--aliases')) return '--type is only supported with reindex --markdown, not --aliases';if (args.includes('--oversized')) return '--type is only supported with reindex --markdown, not --oversized';if (args.includes('--segmented')) return '--type is only supported with reindex --markdown, not --segmented';+ if (args.includes('--synopsis')) return '--type is only supported with reindex --markdown, not --synopsis';return null;}diff --git a/src/commands/search.ts b/src/commands/search.tsindex 8e410b7c60..26127abb97 100644--- a/src/commands/search.ts+++ b/src/commands/search.ts@@ -84,6 +84,10 @@ function formatModesText(report: SearchModesReport): string {lines.push(`Reranker: ${rr.model} (enabled but NOT running) — ${rr.fix ?? 'see gbrain doctor'}`);}}+ if (report.synopsis_coverage) {+ lines.push('');+ lines.push(report.synopsis_coverage.line);+ }lines.push('');lines.push('Mode bundles (frozen — set via `gbrain config set search.mode <mode>`):');for (const mode of SEARCH_MODES) {diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.tsindex f6a8145e33..1b1675ee66 100644--- a/src/core/doctor-categories.ts+++ b/src/core/doctor-categories.ts@@ -122,6 +122,8 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set(['content_hash_duplicates','content_sanity_audit_recent','contextual_retrieval_coverage',+ // #8744: configured per_chunk_synopsis tier vs the chunks that carry it.+ 'synopsis_tier_coverage','contradictions','conversation_facts_backlog','conversation_format_coverage',diff --git a/src/core/page-summary.ts b/src/core/page-summary.tsindex e58131fb9e..a1f3fd8ec1 100644--- a/src/core/page-summary.ts+++ b/src/core/page-summary.ts@@ -320,7 +320,7 @@ export function segmentSynopsisWindows<C extends { chunk_text: string; chunk_sou}/** Input chars of a synopsis call outside its document and chunk: the system prompt and the prompt's tags. */-const SYNOPSIS_PROMPT_OVERHEAD_CHARS = SYNOPSIS_SYSTEM_PROMPT.length + 256;+export const SYNOPSIS_PROMPT_OVERHEAD_CHARS = SYNOPSIS_SYSTEM_PROMPT.length + 256;/** The rate (USD per million tokens) an unpriced synopsis model reserves at. */const UNPRICED_SYNOPSIS_RATE = { input: 15, output: 75 };diff --git a/src/core/search/modes-report.ts b/src/core/search/modes-report.tsindex 008bfbb6a7..7babe168b7 100644--- a/src/core/search/modes-report.ts+++ b/src/core/search/modes-report.ts@@ -130,6 +130,13 @@ export interface SearchModesReport {* file-plane + process env. Absent only when readiness itself threw.*/reranker_readiness?: RerankerReadinessReport;+ /**+ * #8744: when the configured contextual-retrieval tier is per_chunk_synopsis,+ * how many chunks actually carry synopsis vectors (import and sync embed at+ * the title tier; the sweep is `gbrain reindex --synopsis`). Brain-wide+ * counts, so ABSENT on the remote surface.+ */+ synopsis_coverage?: SynopsisCoverageReport;/** #4604: what this report does NOT include (per-call plane). */per_call_note: string;_meta?: {@@ -137,6 +144,17 @@ export interface SearchModesReport {};}+export interface SynopsisCoverageReport {+ eligible_pages: number;+ eligible_chunks: number;+ synopsis_pages: number;+ synopsis_chunks: number;+ coverage_pct: number | null;+ large_title_pages: number;+ line: string;+ fix: string;+}+export interface RerankerReadinessReport {model: string;enabled: boolean;@@ -163,7 +181,8 @@ export function redactReadinessForRemote(report: SearchModesReport): SearchModes// self_hosted is deployment topology (a private base-URL override exists) —// not needed for the verdict, so it stays local too.const { model, enabled, ready } = rr;- return { ...report, reranker_readiness: { model, enabled, ready, self_hosted: false } };+ const { synopsis_coverage: _brainWide, ...rest } = report;+ return { ...rest, reranker_readiness: { model, enabled, ready, self_hosted: false } };}export async function buildModesReport(engine: BrainEngine): Promise<SearchModesReport> {@@ -227,12 +246,24 @@ export async function buildModesReport(engine: BrainEngine): Promise<SearchModes};}+ let synopsis_coverage: SynopsisCoverageReport | undefined;+ try {+ const sweep = await import('../synopsis-sweep.ts');+ const c = await sweep.readSynopsisCoverage(engine);+ if (c.config.configured) {+ synopsis_coverage = { eligible_pages: c.eligible_pages, eligible_chunks: c.eligible_chunks, synopsis_pages: c.synopsis_pages,+ synopsis_chunks: c.synopsis_chunks, coverage_pct: c.coverage_pct, large_title_pages: c.large_title_pages,+ line: sweep.formatSynopsisCoverageLine(c), fix: sweep.SYNOPSIS_SWEEP_ARGV.join(' ') };+ }+ } catch { /* coverage is advisory; the doctor check synopsis_tier_coverage reports read failures */ }+return {schema_version: 2,active_mode: resolved.resolved_mode,active_mode_valid: resolved.mode_valid,resolved: attributions,...(reranker_readiness ? { reranker_readiness } : {}),+ ...(synopsis_coverage ? { synopsis_coverage } : {}),bundles: {conservative: { ...MODE_BUNDLES.conservative },balanced: { ...MODE_BUNDLES.balanced },diff --git a/src/core/synopsis-sweep.ts b/src/core/synopsis-sweep.tsnew file mode 100644index 0000000000..b50f66a2d2--- /dev/null+++ b/src/core/synopsis-sweep.ts@@ -0,0 +1,248 @@+/**+ * #8744: the per-chunk synopsis sweep for brains whose configured contextual+ * retrieval tier is `per_chunk_synopsis` (search.mode tokenmax, or a source or+ * page override). Import and sync embed those pages at the free title tier and+ * leave the synopsis tier to the `contextual_reindex_per_chunk` job, which no+ * automatic path submits; `gbrain reindex --synopsis` is the explicit,+ * operator-run sweep. This module holds the reads shared by that command, the+ * doctor check `synopsis_tier_coverage` and the `gbrain search modes` line:+ * which pages resolve to the synopsis tier (the SQL twin of+ * resolveContextualRetrievalMode), how many of their chunks carry synopsis+ * vectors, and what the sweep would cost. Database reads only, so a repo-less+ * brain behaves exactly like a repo brain.+ *+ * Coverage is counted per chunk by the page's stamped mode: a page is stamped+ * `per_chunk_synopsis` only once every chunk is settled. Inline pages over the+ * large-page threshold stay on the title tier by design (contextual-retrieval+ * service, plan B8), so they leave the denominator and are reported apart.+ */+import type { BrainEngine } from './engine.ts';+import { loadSearchModeConfig, resolveSearchMode } from './search/mode.ts';+import { largePageBodyFilter } from './page-size-policy.ts';+import {+ SEGMENT_SYNOPSIS_LEAD_BYTES,+ SEGMENT_SYNOPSIS_WINDOW_BYTES,+ SYNOPSIS_DOC_MAX_CHARS,+ SYNOPSIS_MAX_TOKENS,+ SYNOPSIS_PROMPT_OVERHEAD_CHARS,+} from './page-summary.ts';+import { reservationCostUsd } from './budget/reservation-cost.ts';+import type { CRMode } from './types.ts';++export const SYNOPSIS_SWEEP_JOB = 'contextual_reindex_per_chunk';+export const SYNOPSIS_SWEEP_ARGV = ['gbrain', 'reindex', '--synopsis'] as const;+/** Below this share of eligible chunks on the synopsis tier, the configured tier is "missing from most chunks". */+export const SYNOPSIS_COVERAGE_WARN_PCT = 50;+/** Embedding-input chars a synopsis adds to a chunk at most (contextual-retrieval-service SYNOPSIS_WRAPPER_CHARS). */+const SYNOPSIS_WRAPPER_CHARS = 320;++export interface SynopsisTierConfig {+ /** The brain-level tier (search.mode bundle or search.contextual_retrieval override). */+ global_tier: CRMode;+ kill_switch: boolean;+ /** Sources whose own contextual_retrieval_mode is per_chunk_synopsis. */+ synopsis_sources: string[];+ /** True when any page can resolve to the synopsis tier (page frontmatter overrides aside). */+ configured: boolean;+}++export async function readSynopsisTierConfig(engine: BrainEngine): Promise<SynopsisTierConfig> {+ const knobs = resolveSearchMode(await loadSearchModeConfig(engine));+ const rows = await engine.executeRaw<{ id: string }>(+ `SELECT id FROM sources WHERE contextual_retrieval_mode = 'per_chunk_synopsis' AND NOT archived ORDER BY id`);+ const synopsisSources = rows.map(r => r.id);+ const killSwitch = knobs.contextual_retrieval_disabled === true;+ return {+ global_tier: knobs.contextual_retrieval,+ kill_switch: killSwitch,+ synopsis_sources: synopsisSources,+ configured: !killSwitch && (knobs.contextual_retrieval === 'per_chunk_synopsis' || synopsisSources.length > 0),+ };+}++/**+ * Markdown pages whose resolved tier is per_chunk_synopsis, with their stamped+ * mode, large-inline flag and chunk counts. Mirrors resolveContextualRetrievalMode:+ * a valid frontmatter value wins on the host source or a trusted mount, then a+ * valid source-row mode, then the global tier. `$1` is the global tier.+ */+function eligiblePagesSql(opts: { sourceParam?: string; withChars: boolean }): string {+ return `+ WITH chunk_stats AS (+ SELECT c.page_id, count(*)::bigint AS chunks${opts.withChars ? ', COALESCE(sum(length(c.chunk_text)), 0)::bigint AS chunk_chars' : ''}+ FROM content_chunks c+ WHERE c.chunk_source <> 'image_asset'+ GROUP BY c.page_id+ ), resolved AS (+ SELECT p.id, p.slug, p.source_id, p.contextual_retrieval_mode AS stamped,+ p.text_projection_revision AS revision,+ p.body_storage = 'segmented' AS segmented,+ (COALESCE(p.body_storage, 'inline') <> 'segmented' AND ${largePageBodyFilter('p')}) AS large_inline,+ CASE WHEN p.body_storage = 'segmented' THEN COALESCE(p.body_bytes, 0)+ ELSE length(COALESCE(p.compiled_truth, '')) + length(COALESCE(p.timeline, '')) END::bigint AS body_chars,+ CASE+ WHEN p.frontmatter ? 'contextual_retrieval'+ AND (p.source_id = 'default' OR COALESCE(s.trust_frontmatter_overrides, false))+ AND p.frontmatter->>'contextual_retrieval' IN ('none', 'title', 'per_chunk_synopsis')+ THEN p.frontmatter->>'contextual_retrieval'+ WHEN s.contextual_retrieval_mode IN ('none', 'title', 'per_chunk_synopsis') THEN s.contextual_retrieval_mode+ ELSE $1::text+ END AS effective+ FROM pages p+ LEFT JOIN sources s ON s.id = p.source_id+ WHERE p.deleted_at IS NULL AND p.page_kind = 'markdown' AND p.type <> 'extract_receipt'+ AND NOT (p.frontmatter ? 'embed_skip')+ AND NOT COALESCE(s.archived, false)+ ${opts.sourceParam ? `AND p.source_id = ANY(${opts.sourceParam}::text[])` : ''}+ )+ SELECT r.*, cs.chunks${opts.withChars ? ', cs.chunk_chars' : ''}+ FROM resolved r+ JOIN chunk_stats cs ON cs.page_id = r.id+ WHERE r.effective = 'per_chunk_synopsis'`;+}++export interface SynopsisCoverage {+ config: SynopsisTierConfig;+ eligible_pages: number;+ eligible_chunks: number;+ synopsis_pages: number;+ synopsis_chunks: number;+ /** Chunks of eligible pages on the synopsis tier, as a percentage (0..100, one decimal); null with no eligible chunks. */+ coverage_pct: number | null;+ /** Inline pages over the large-page threshold: the title tier by design, outside the denominator. */+ large_title_pages: number;+ large_title_chunks: number;+}++/** Synopsis-tier coverage. `sourceIds` confines every count to those sources (the remote doctor's grant). */+export async function readSynopsisCoverage(engine: BrainEngine, opts: { sourceIds?: string[]; config?: SynopsisTierConfig } = {}): Promise<SynopsisCoverage> {+ const config = opts.config ?? await readSynopsisTierConfig(engine);+ const empty = { eligible_pages: 0, eligible_chunks: 0, synopsis_pages: 0, synopsis_chunks: 0, coverage_pct: null, large_title_pages: 0, large_title_chunks: 0 };+ if (!config.configured) return { config, ...empty };+ const rows = await engine.executeRaw<Record<string, string | number>>(+ `SELECT+ count(*) FILTER (WHERE NOT large_inline)::bigint AS eligible_pages,+ COALESCE(sum(chunks) FILTER (WHERE NOT large_inline), 0)::bigint AS eligible_chunks,+ count(*) FILTER (WHERE NOT large_inline AND stamped = 'per_chunk_synopsis')::bigint AS synopsis_pages,+ COALESCE(sum(chunks) FILTER (WHERE NOT large_inline AND stamped = 'per_chunk_synopsis'), 0)::bigint AS synopsis_chunks,+ count(*) FILTER (WHERE large_inline)::bigint AS large_title_pages,+ COALESCE(sum(chunks) FILTER (WHERE large_inline), 0)::bigint AS large_title_chunks+ FROM (${eligiblePagesSql({ sourceParam: opts.sourceIds ? '$2' : undefined, withChars: false })}) eligible`,+ opts.sourceIds ? [config.global_tier, opts.sourceIds] : [config.global_tier]);+ const n = (key: string) => Number(rows[0]?.[key] ?? 0);+ const eligibleChunks = n('eligible_chunks');+ return {+ config,+ eligible_pages: n('eligible_pages'),+ eligible_chunks: eligibleChunks,+ synopsis_pages: n('synopsis_pages'),+ synopsis_chunks: n('synopsis_chunks'),+ coverage_pct: eligibleChunks > 0 ? Math.floor((n('synopsis_chunks') / eligibleChunks) * 1000) / 10 : null,+ large_title_pages: n('large_title_pages'),+ large_title_chunks: n('large_title_chunks'),+ };+}++export interface SynopsisSweepPage {+ id: number;+ slug: string;+ source_id: string;+ revision: string | null;+ chunks: number;+}++export interface SynopsisSweepPlan {+ config: SynopsisTierConfig;+ synopsis_model: string;+ pages: SynopsisSweepPage[];+ chunks: number;+ /** Ceiling: every synopsis prompt uncached (the prompt cache usually serves all but a page's first call). */+ est_input_tokens: number;+ /** Ceiling: every synopsis at the output-token cap. */+ est_output_tokens: number;+ /** Null when the synopsis model has no known price. */+ est_synopsis_usd: number | null;+ /** Null when the embedding model has no known price (or embedding is not configured). */+ est_embedding_usd: number | null;+ est_usd: number | null;+ /** Pending pages beyond `limit` that this run leaves for the next one. */+ remaining_pages: number;+}++/**+ * The pages a sweep would queue (eligible, not large-inline, not yet stamped+ * per_chunk_synopsis), oldest id first, with the token and dollar estimate on+ * the configured synopsis model. A page's synopsis document is its body capped+ * at SYNOPSIS_DOC_MAX_CHARS (a segmented page: its lead plus one segment window),+ * sent once per chunk with the chunk and the prompt overhead, at 4 chars per+ * token (estimateSynopsisCallUsd's rate).+ */+export async function planSynopsisSweep(engine: BrainEngine, opts: {+ synopsisModel: string;+ synopsisMaxTokens?: number;+ sourceId?: string;+ limit?: number;+ embeddingPricePerMTok?: number | null;+}): Promise<SynopsisSweepPlan> {+ const config = await readSynopsisTierConfig(engine);+ const base = { config, synopsis_model: opts.synopsisModel, pages: [], chunks: 0, est_input_tokens: 0, est_output_tokens: 0,+ est_synopsis_usd: 0, est_embedding_usd: 0, est_usd: 0, remaining_pages: 0 };+ if (!config.configured) return base;+ const rows = await engine.executeRaw<{ id: number | string; slug: string; source_id: string; revision: string | null; segmented: boolean | null;+ body_chars: number | string; chunks: number | string; chunk_chars: number | string }>(+ `SELECT id, slug, source_id, revision, segmented, body_chars, chunks, chunk_chars+ FROM (${eligiblePagesSql({ sourceParam: opts.sourceId ? '$2' : undefined, withChars: true })}) eligible+ WHERE NOT large_inline AND stamped IS DISTINCT FROM 'per_chunk_synopsis'+ ORDER BY id`,+ opts.sourceId ? [config.global_tier, [opts.sourceId]] : [config.global_tier]);+ const selected = opts.limit ? rows.slice(0, opts.limit) : rows;+ const maxTokens = opts.synopsisMaxTokens ?? SYNOPSIS_MAX_TOKENS;+ const segmentedDocChars = Math.min(SEGMENT_SYNOPSIS_LEAD_BYTES + SEGMENT_SYNOPSIS_WINDOW_BYTES, SYNOPSIS_DOC_MAX_CHARS);+ let chunks = 0;+ let inputTokens = 0;+ let embeddedChars = 0;+ for (const row of selected) {+ const pageChunks = Number(row.chunks);+ const chunkChars = Number(row.chunk_chars);+ const docChars = row.segmented ? Math.min(Number(row.body_chars), segmentedDocChars) : Math.min(Number(row.body_chars), SYNOPSIS_DOC_MAX_CHARS);+ chunks += pageChunks;+ inputTokens += Math.ceil((pageChunks * (docChars + SYNOPSIS_PROMPT_OVERHEAD_CHARS) + chunkChars) / 4);+ embeddedChars += chunkChars + pageChunks * SYNOPSIS_WRAPPER_CHARS;+ }+ const outputTokens = chunks * maxTokens;+ const synopsisUsd = chunks === 0 ? 0 : reservationCostUsd(opts.synopsisModel, 'chat', inputTokens, outputTokens);+ const embeddingUsd = opts.embeddingPricePerMTok == null ? null : (Math.ceil(embeddedChars / 3.5) / 1_000_000) * opts.embeddingPricePerMTok;+ return {+ ...base,+ pages: selected.map(r => ({ id: Number(r.id), slug: r.slug, source_id: r.source_id, revision: r.revision ?? null, chunks: Number(r.chunks) })),+ chunks,+ est_input_tokens: inputTokens,+ est_output_tokens: outputTokens,+ est_synopsis_usd: synopsisUsd,+ est_embedding_usd: embeddingUsd,+ est_usd: synopsisUsd === null || embeddingUsd === null ? null : synopsisUsd + embeddingUsd,+ remaining_pages: rows.length - selected.length,+ };+}++/** The newest finished sweep job and how many ever ran (null when the queue table is unreadable). */+export async function readLastSynopsisSweep(engine: BrainEngine): Promise<{ jobs: number; last_finished_at: string | null } | null> {+ try {+ const rows = await engine.executeRaw<{ jobs: number | string; last: string | Date | null }>(+ `SELECT count(*)::bigint AS jobs, max(finished_at) AS last FROM minion_jobs WHERE name = $1`, [SYNOPSIS_SWEEP_JOB]);+ const last = rows[0]?.last ?? null;+ return { jobs: Number(rows[0]?.jobs ?? 0), last_finished_at: last === null ? null : new Date(last).toISOString() };+ } catch {+ return null;+ }+}++/** One line for `gbrain search modes` and the doctor message: the configured tier's chunk coverage. */+export function formatSynopsisCoverageLine(c: SynopsisCoverage): string {+ if (c.eligible_chunks === 0) return 'Synopsis tier: configured (per_chunk_synopsis); no chunked pages resolve to it yet.';+ const large = c.large_title_pages ? ` ${c.large_title_pages} large page(s) (${c.large_title_chunks} chunk(s)) stay on the title tier by design.` : '';+ const pending = c.eligible_chunks - c.synopsis_chunks;+ return `Synopsis tier: ${c.coverage_pct}% of chunks (${c.synopsis_chunks}/${c.eligible_chunks}) and ${c.synopsis_pages}/${c.eligible_pages} page(s) carry per-chunk synopsis vectors`+ + (pending > 0 ? `; ${pending} chunk(s) search on title-tier vectors. Preview the sweep: ${SYNOPSIS_SWEEP_ARGV.join(' ')}` : '')+ + `.${large}`;+}diff --git a/test/e2e/synopsis-sweep-8744-postgres.test.ts b/test/e2e/synopsis-sweep-8744-postgres.test.tsnew file mode 100644index 0000000000..58538eaffe--- /dev/null+++ b/test/e2e/synopsis-sweep-8744-postgres.test.ts@@ -0,0 +1,3 @@+import { registerPostgresTests } from '../helpers/test-backends.ts';++await registerPostgresTests(() => import('../synopsis-sweep-8744.serial.test.ts'));diff --git a/test/fixtures/goldens/doctor/registry-build-checks.json b/test/fixtures/goldens/doctor/registry-build-checks.jsonindex 78d149cec1..775f640611 100644--- a/test/fixtures/goldens/doctor/registry-build-checks.json+++ b/test/fixtures/goldens/doctor/registry-build-checks.json@@ -1089,6 +1089,10 @@"category": "ops","name": "search_mode"},+ {+ "category": "brain",+ "name": "synopsis_tier_coverage"+ },{"category": "brain","name": "hidden_by_search_policy"diff --git a/test/fixtures/goldens/doctor/registry-report-remote.json b/test/fixtures/goldens/doctor/registry-report-remote.jsonindex 697343e2c4..d2ba5b7c1d 100644--- a/test/fixtures/goldens/doctor/registry-report-remote.json+++ b/test/fixtures/goldens/doctor/registry-report-remote.json@@ -165,6 +165,10 @@"category": "brain","name": "contextual_retrieval_coverage"},+ {+ "category": "brain",+ "name": "synopsis_tier_coverage"+ },{"category": "brain","name": "text_projection_readiness"diff --git a/test/synopsis-sweep-8744.serial.test.ts b/test/synopsis-sweep-8744.serial.test.tsnew file mode 100644index 0000000000..1ba91d9907--- /dev/null+++ b/test/synopsis-sweep-8744.serial.test.ts@@ -0,0 +1,195 @@+/**+ * #8744: with search.mode tokenmax (contextual retrieval per_chunk_synopsis),+ * import embeds pages at the title tier and nothing scheduled the synopsis+ * sweep, so a brain searched on title-tier vectors with no warning. Pins:+ * - doctor synopsis_tier_coverage warns on a tokenmax brain with no+ * synopsis vectors (coverage and the gbrain reindex --synopsis fix), and+ * is not applicable on a balanced brain;+ * - the preview's pages, chunks, tokens and dollars on the configured+ * synopsis model, and that it queues and spends nothing;+ * - a --yes run under a cap below one synopsis call is refused by the+ * shared spend group and stamps nothing; a later run resumes;+ * - --yes queues one contextual_reindex_per_chunk job per page under one+ * spend group and completes them in-process; the pages land on the+ * synopsis tier, the check turns ok, and a rerun has nothing to sweep;+ * - inline pages over the large-page threshold stay out of the sweep.+ * The brains here have no git repository (repo-less): the sweep reads and+ * writes the database only.+ *+ * Fake chat + embed transports (no provider is called). Serial: mutates the+ * process-global gateway, its transports and env vars. Runs on PGLite, and on+ * Postgres through test/e2e/synopsis-sweep-8744-postgres.test.ts.+ */+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';+import { mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { importFromContent } from '../src/core/import-file.ts';+import { __setChatTransportForTests, __setEmbedTransportForTests, configureGateway, resetGateway } from '../src/core/ai/gateway.ts';+import { checkSynopsisTierCoverage } from '../src/commands/doctor/checks/synopsis-coverage.ts';+import { runReindexSynopsis } from '../src/commands/reindex-synopsis.ts';+import { estimateSynopsisCallUsd, SYNOPSIS_MAX_TOKENS } from '../src/core/page-summary.ts';+import { reservationCostUsd } from '../src/core/budget/reservation-cost.ts';+import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';+import { buildModesReport, redactReadinessForRemote } from '../src/core/search/modes-report.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';+import { testBackends } from './helpers/test-backends.ts';++const DIMS = 1536;+const EMBED_MODEL = 'openai:text-embedding-3-large';+const SYNOPSIS_MODEL = 'anthropic:claude-haiku-4-5-20251001';+let chatCalls = 0;+let home: string;+const saved = { home: process.env.GBRAIN_HOME, model: process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL };++beforeAll(() => {+ home = mkdtempSync(join(tmpdir(), 'gbrain-synopsis-sweep-'));+ process.env.GBRAIN_HOME = home;+ process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL = SYNOPSIS_MODEL;+ configureGateway({ embedding_model: EMBED_MODEL, embedding_dimensions: DIMS, env: { OPENAI_API_KEY: 'sk-test-fake', ANTHROPIC_API_KEY: 'sk-ant-test-fake' } });+ __setEmbedTransportForTests((async ({ values }: { values: string[] }) =>+ ({ embeddings: values.map(() => new Array(DIMS).fill(0.001)), usage: { tokens: 0 } })) as never);+ __setChatTransportForTests(async () => {+ chatCalls++;+ return { text: 'Synopsis: a note about the example topic.', blocks: [], stopReason: 'end',+ usage: { input_tokens: 500, output_tokens: 12, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: SYNOPSIS_MODEL, providerId: 'anthropic' } as never;+ });+});++afterAll(() => {+ __setChatTransportForTests(null);+ __setEmbedTransportForTests(null);+ resetGateway();+ if (saved.home === undefined) delete process.env.GBRAIN_HOME; else process.env.GBRAIN_HOME = saved.home;+ if (saved.model === undefined) delete process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL; else process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL = saved.model;+ rmSync(home, { recursive: true, force: true });+});++beforeEach(() => { chatCalls = 0; });++const body = (title: string, paragraphs: number) => `---\ntype: note\ntitle: ${title}\n---\n\n${Array.from({ length: paragraphs }, (_, i) => `Paragraph ${i} about ${title}. `.repeat(30)).join('\n\n')}\n`;+const quiet = async <T>(fn: () => Promise<T>): Promise<T> => {+ const log = console.log;+ const write = process.stderr.write.bind(process.stderr);+ console.log = () => {};+ process.stderr.write = (() => true) as typeof process.stderr.write;+ try { return await fn(); } finally { console.log = log; process.stderr.write = write; }+};++for (const kind of testBackends()) describe(`#8744 synopsis sweep (${kind})`, () => {+ let engine: BrainEngine, close: (() => Promise<void>) | undefined;+ beforeAll(async () => {+ if (kind === 'postgres') ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!));+ else { engine = new PGLiteEngine(); await engine.connect({ embedding_dimensions: DIMS } as never); await engine.initSchema(); }+ await engine.setConfig('search.mode', 'balanced');+ await importFromContent(engine, 'notes/alpha-example', body('Alpha example', 2), { sourceId: 'default' });+ await importFromContent(engine, 'notes/beta-example', body('Beta example', 6), { sourceId: 'default' });+ await importFromContent(engine, 'notes/large-example', body('Large example', 1), { sourceId: 'default' });+ // An inline page over the 1 MiB large-page threshold (legacy shape): the title tier by design.+ await engine.executeRaw(`UPDATE pages SET compiled_truth = repeat('x', 1100000) WHERE slug = 'notes/large-example'`);+ }, 120_000);+ afterAll(async () => { if (close) await close(); else await engine.disconnect(); });++ const chunkStats = async (slug: string) => {+ const rows = await engine.executeRaw<{ chunk_text: string; body: string }>(+ `SELECT c.chunk_text, COALESCE(p.compiled_truth, '') || COALESCE(p.timeline, '') AS body+ FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE p.slug = $1 AND c.chunk_source <> 'image_asset' ORDER BY c.chunk_index`, [slug]);+ return rows;+ };++ test('balanced brain: the check is not applicable and the command has nothing to do', async () => {+ const check = await checkSynopsisTierCoverage(engine);+ expect(check.status).toBe('ok');+ expect(check.message).toContain('Not applicable');+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis']));+ expect(result.mode).toBe('not_applicable');+ });++ test('tokenmax brain with no synopsis vectors: doctor warns with coverage and the preview command', async () => {+ await engine.setConfig('search.mode', 'tokenmax');+ const modes = await engine.executeRaw<{ mode: string | null }>(`SELECT contextual_retrieval_mode AS mode FROM pages ORDER BY slug`);+ expect(modes.every(m => m.mode !== 'per_chunk_synopsis')).toBe(true);+ const check = await checkSynopsisTierCoverage(engine);+ expect(check.status).toBe('warn');+ expect(check.message).toContain('0% of chunks');+ expect(check.message).toContain('gbrain reindex --synopsis');+ expect(check.message).toContain('No synopsis sweep job has ever run');+ expect(check.message).toContain('1 large page(s)');+ expect(check.details).toMatchObject({ coverage_pct: 0, synopsis_chunks: 0, eligible_pages: 2, large_title_pages: 1, sweep_jobs: 0 });+ expect(check.fix?.argv).toEqual(['gbrain', 'reindex', '--synopsis']);+ const report = await buildModesReport(engine);+ expect(report.synopsis_coverage).toMatchObject({ coverage_pct: 0, synopsis_chunks: 0, fix: 'gbrain reindex --synopsis' });+ expect(report.synopsis_coverage!.line).toContain('0% of chunks');+ expect(redactReadinessForRemote(report).synopsis_coverage).toBeUndefined();+ });++ test('preview: pages, chunks, tokens and dollars on the configured synopsis model; nothing queued', async () => {+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis']));+ expect(result.mode).toBe('preview');+ const plan = result.plan!;+ expect(plan.synopsis_model).toBe(SYNOPSIS_MODEL);+ expect(plan.pages.map(p => p.slug)).toEqual(['notes/alpha-example', 'notes/beta-example']);+ const alpha = await chunkStats('notes/alpha-example');+ const beta = await chunkStats('notes/beta-example');+ expect(alpha.length).toBeGreaterThan(0);+ expect(beta.length).toBeGreaterThan(alpha.length);+ const all = [...alpha, ...beta];+ expect(plan.chunks).toBe(all.length);+ expect(plan.est_output_tokens).toBe(all.length * SYNOPSIS_MAX_TOKENS);+ // The ceiling matches the per-call reservation estimator summed over every chunk (one call per chunk).+ const perCall = all.reduce((sum, c) => sum + estimateSynopsisCallUsd(SYNOPSIS_MODEL, c.body, c.chunk_text, SYNOPSIS_MAX_TOKENS), 0);+ expect(plan.est_synopsis_usd).not.toBeNull();+ expect(plan.est_synopsis_usd!).toBeGreaterThan(0);+ expect(plan.est_synopsis_usd!).toBeCloseTo(perCall, 4);+ expect(plan.est_synopsis_usd).toBe(reservationCostUsd(SYNOPSIS_MODEL, 'chat', plan.est_input_tokens, plan.est_output_tokens));+ const price = lookupEmbeddingPrice(EMBED_MODEL);+ expect(price.kind).toBe('known');+ expect(plan.est_embedding_usd!).toBeGreaterThan(0);+ expect(plan.est_usd!).toBeCloseTo(plan.est_synopsis_usd! + plan.est_embedding_usd!, 10);+ const jobs = await engine.executeRaw<{ n: number | string }>(`SELECT count(*)::int AS n FROM minion_jobs WHERE name = 'contextual_reindex_per_chunk'`);+ expect(Number(jobs[0]!.n)).toBe(0);+ expect(chatCalls).toBe(0);+ });++ test('--yes under a cap below one synopsis call: the shared spend cap refuses the work, nothing is stamped, and the run says so', async () => {+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis', '--yes', '--max-usd', '0.0001'], { pollMs: 100 }));+ expect(result.mode).toBe('run');+ expect(result.run!.cap_usd).toBe(0.0001);+ expect(result.run!.failed).toBe(2);+ expect(result.run!.synopsis).toBe(0);+ expect(chatCalls).toBe(0);+ const stamped = await engine.executeRaw<{ n: number | string }>(`SELECT count(*)::int AS n FROM pages WHERE contextual_retrieval_mode = 'per_chunk_synopsis'`);+ expect(Number(stamped[0]!.n)).toBe(0);+ });++ test('--yes queues and completes one job per page; pages land on the synopsis tier and a rerun has nothing to sweep', async () => {+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis', '--yes', '--max-usd', '5'], { pollMs: 100 }));+ expect(result.mode).toBe('run');+ expect(result.run).toMatchObject({ synopsis: 2, title: 0, skipped: 0, failed: 0, already_attempted: 0, cap_usd: 5 });+ expect(chatCalls).toBe(result.plan!.chunks);+ const jobs = await engine.executeRaw<{ status: string; group_id: string | null; slug: string }>(+ `SELECT status, spend_authorization->>'group_id' AS group_id, data->>'page_slug' AS slug+ FROM minion_jobs WHERE name = 'contextual_reindex_per_chunk' AND id = ANY($1::int[]) ORDER BY id`, [result.run!.job_ids]);+ expect(jobs.map(j => j.slug)).toEqual(['notes/alpha-example', 'notes/beta-example']);+ expect(jobs.every(j => j.status === 'completed' && j.group_id === result.run!.group_id)).toBe(true);+ const modes = await engine.executeRaw<{ slug: string; mode: string | null }>(`SELECT slug, contextual_retrieval_mode AS mode FROM pages ORDER BY slug`);+ expect(modes).toEqual([+ { slug: 'notes/alpha-example', mode: 'per_chunk_synopsis' },+ { slug: 'notes/beta-example', mode: 'per_chunk_synopsis' },+ { slug: 'notes/large-example', mode: 'title' },+ ]);+ expect(result.coverage_after!.coverage_pct).toBe(100);+ const check = await checkSynopsisTierCoverage(engine);+ expect(check.status).toBe('ok');+ expect(check.message).toContain('100% of chunks');+ expect(check.fix).toBeUndefined();++ chatCalls = 0;+ const rerun = await quiet(() => runReindexSynopsis(engine, ['--synopsis', '--yes'], { pollMs: 100 }));+ expect(rerun.mode).toBe('preview');+ expect(rerun.plan!.pages).toEqual([]);+ expect(chatCalls).toBe(0);+ });+});