Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(search): report synopsis-tier coverage and add gbrain reindex --synopsis (#8744)

garrytan62eeb1720a1b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ca390dd578..09d2b157c4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,21 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A brain set to `search.mode tokenmax` now says when its pages are not on the synopsis tier, and `gbrain reindex --synopsis` puts them there.** Import and sync embed tokenmax pages at the free title tier and leave per-chunk synopses to the `contextual_reindex_per_chunk` job, but nothing ever queued that job, so a brain could search on title-tier vectors for weeks while reporting tokenmax.
+
+### What changed
+
+| Situation | Before | Now |
+|---|---|---|
+| `gbrain doctor` on a brain whose configured tier is `per_chunk_synopsis` | nothing about it | `synopsis_tier_coverage` gives chunk and page coverage, the large pages left on the title tier and the last sweep job; it warns below 50% of chunks and names `gbrain reindex --synopsis`. The remote doctor runs it too, confined to the caller's sources |
+| `gbrain search modes` | the configured tier only | one line with the synopsis-tier coverage (local only; the `search_modes` op leaves it out) |
+| `gbrain reindex --synopsis` | did not exist | previews pages, chunks, synopsis tokens and dollars on the configured synopsis model plus the re-embed, and spends nothing |
+| `gbrain reindex --synopsis --yes [--max-usd <usd>]` | did not exist | queues one `contextual_reindex_per_chunk` job per page under one spend cap the worker enforces across them, runs them in-process and reports the new coverage. Running it again resumes: finished pages are skipped and queued jobs are picked up |
+
+Nothing is scheduled automatically and no default changes. Inline pages over 1 MiB keep the title tier, as before. Repo-less brains work the same way: the sweep reads and writes the database only.
+
## [0.60.595.0] - 2026-11-06
**A `memory_recall` that names sources reads as that scope, never as the attested person, so a shared turn can no longer recall one member's private facts into a shared channel.** `memory_recall` gains `source_id` and `source_ids`. Named, the read drops the person's own recent facts and every member-page fact, keeps restricted pages hidden and never relaxes `visibility: private`. The contract for it is the new `docs/contracts/qm-memory-provider/v2.json`; `v1.json` stays byte-identical and a test holds it there. Three smaller provider fixes ride along.
diff --git a/scripts/serial-files.tsv b/scripts/serial-files.tsv
index 51d08ce86c..893bcd2e40 100644
--- a/scripts/serial-files.tsv
+++ b/scripts/serial-files.tsv
@@ -460,3 +460,4 @@ test/search/identifier-probe.serial.test.ts R1 R1: mutates process.env (GBRAIN_H
test/search/slug-handles.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME)
test/launcher-lanes-bunfig.serial.test.ts global-state global-state: spawns gbrain serve subprocesses against one shared PGLite brain and its data-dir lock; withEnv scopes HOME, GBRAIN_HOME and CODEX_HOME around the bootstrap and setup writers
test/extraction-review-door.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME) around real git subprocesses against tmp repos for the page-door review decisions
+test/synopsis-sweep-8744.serial.test.ts R1 R1: mutates process.env (GBRAIN_HOME, GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL)
diff --git a/skills/migrations/v0.40.3.0.md b/skills/migrations/v0.40.3.0.md
index de01a2fad7..363b0e741e 100644
--- a/skills/migrations/v0.40.3.0.md
+++ b/skills/migrations/v0.40.3.0.md
@@ -74,22 +74,18 @@ worker) offers to submit `gbrain reindex --markdown` as a Minion job.
Non-TTY callers get a paste-ready hint to stderr. Suppress entirely
with `GBRAIN_NO_MODE_SWITCH_UX=1`.
-If the user skipped or declined that prompt, do not start the reindex
-yourself. Re-embedding every markdown page calls the embedding provider (and,
-under `tokenmax`, a chat model for each chunk), so it costs money. Preview it,
-tell the user how many pages it covers, and run it only after they agree:
+Import, sync and `gbrain reindex --markdown` embed tokenmax pages at the title
+tier; only the synopsis sweep writes per-chunk synopses, and `gbrain doctor`
+(`synopsis_tier_coverage`) and `gbrain search modes` report its coverage. Do not
+start it yourself: each chunk costs a chat call plus its re-embed. Preview it,
+tell the user the pages and cost, and run it only after they agree:
```bash
-gbrain reindex --markdown --dry-run # page count; changes nothing
-# only after the user agrees:
-gbrain reindex --markdown --yes
-# or as a background job (needs a running `gbrain jobs work` worker; on PGLite add --follow):
-gbrain jobs submit reindex --params '{"markdown":true}' --yes
+gbrain reindex --synopsis # pages, chunks, tokens, dollars; changes nothing
+# only after the user agrees (resumable; --max-usd caps the whole run):
+gbrain reindex --synopsis --yes
```
-Without `--yes` (or `--max-usd <usd>`) a non-interactive run exits 3 with an
-`ask_user` payload instead of spending; relay its `user_message`.
-
## Optional: per-source CR mode overrides
If you have multiple sources and want different CR tiers per source:
diff --git a/src/cli/commands/reindex.ts b/src/cli/commands/reindex.ts
index 7e9a99275d..868ac43f16 100644
--- a/src/cli/commands/reindex.ts
+++ b/src/cli/commands/reindex.ts
@@ -48,6 +48,17 @@ export async function run(engine: BrainEngine, args: string[]): Promise<void> {
}
return;
}
+ if (args.includes('--synopsis')) {
+ const { runReindexSynopsis } = await import('../../commands/reindex-synopsis.ts');
+ try {
+ await runReindexSynopsis(engine, args);
+ } catch (e) {
+ const { OperationError } = await import('../../core/ops/contract.ts');
+ if (e instanceof OperationError) { setCliExitVerdict(writeCliError(e, 'reindex')); return; }
+ throw e;
+ }
+ return;
+ }
if (args.includes('--segmented')) {
await reindex.runReindexSegmented(engine, args);
return;
diff --git a/src/commands/doctor/checks/sync-search.ts b/src/commands/doctor/checks/sync-search.ts
index c81a29fb81..0795fca1ed 100644
--- a/src/commands/doctor/checks/sync-search.ts
+++ b/src/commands/doctor/checks/sync-search.ts
@@ -13,6 +13,7 @@ import {
checkLinkResolutionOpportunity,
} from './calibration.ts';
import { checkSyncConsolidation, checkCycleFreshness } from './consolidation-cycle.ts';
+import { checkSynopsisTierCoverage } from './synopsis-coverage.ts';
import {
checkSyncFreshness,
checkLinksExtractionLag,
@@ -144,6 +145,9 @@ async function runSearchMode(ctx: DoctorContext): Promise<Check[]> {
if (engine !== null) {
progress.heartbeat('search_mode');
checks.push(await checkSearchMode(engine));
+ // #8744 synopsis_tier_coverage — configured per_chunk_synopsis tier vs the chunks that carry it.
+ progress.heartbeat('synopsis_tier_coverage');
+ checks.push(await checkSynopsisTierCoverage(engine));
// issue #1777 — hidden_by_search_policy: chunked pages withheld from default
// search by the hard-exclude prefix policy (audit the surviving excludes).
progress.heartbeat('hidden_by_search_policy');
@@ -236,6 +240,7 @@ export const searchModeEntry: DoctorEntry = {
name: 'search_mode',
emits: [
'search_mode',
+ 'synopsis_tier_coverage',
'hidden_by_search_policy',
'eval_drift',
'reranker_health',
diff --git a/src/commands/doctor/checks/synopsis-coverage.ts b/src/commands/doctor/checks/synopsis-coverage.ts
new file mode 100644
index 0000000000..82029ed2a6
--- /dev/null
+++ b/src/commands/doctor/checks/synopsis-coverage.ts
@@ -0,0 +1,61 @@
+/**
+ * synopsis_tier_coverage (#8744): the configured contextual-retrieval tier is
+ * per_chunk_synopsis (search.mode tokenmax, or a source override) but import
+ * and sync embed pages at the title tier and nothing schedules the synopsis
+ * sweep, so a brain can search on title-tier vectors for weeks while it
+ * reports tokenmax. Warns when the synopsis tier is missing from most eligible
+ * chunks, with the coverage and the preview of the explicit sweep
+ * (`gbrain reindex --synopsis`). Database reads only: repo-less brains are
+ * covered the same way.
+ */
+import type { BrainEngine } from '../../../core/engine.ts';
+import type { Check } from '../../doctor.ts';
+import { agentFix, checkError, infoCheck } from '../check-fix.ts';
+
+
+export async function checkSynopsisTierCoverage(engine: BrainEngine, opts: { sourceIds?: string[] } = {}): Promise<Check> {
+ try {
+ const sweep = await import('../../../core/synopsis-sweep.ts');
+ const config = await sweep.readSynopsisTierConfig(engine);
+ if (!config.configured) {
+ return infoCheck('synopsis_tier_coverage', `Not applicable: the configured contextual-retrieval tier is ${config.kill_switch ? 'off (kill switch)' : config.global_tier}, not per_chunk_synopsis.`,
+ 'not_applicable', undefined, { configured: false, global_tier: config.global_tier });
+ }
+ const { embeddingsDisabled } = await import('../../../core/embedding-disabled.ts');
+ if (await embeddingsDisabled(engine)) {
+ return infoCheck('synopsis_tier_coverage', 'Not applicable: embeddings are disabled on this brain, so no tier applies (keyword search keeps working).', 'disabled_by_choice', undefined, { configured: true });
+ }
+ const coverage = await sweep.readSynopsisCoverage(engine, { sourceIds: opts.sourceIds, config });
+ const last = opts.sourceIds ? null : await sweep.readLastSynopsisSweep(engine);
+ const sweepNote = last === null ? '' : last.jobs === 0 ? ' No synopsis sweep job has ever run.' : ` Last sweep job finished ${last.last_finished_at ?? 'never'} (${last.jobs} job(s) recorded).`;
+ const details = {
+ configured: true,
+ global_tier: config.global_tier,
+ synopsis_sources: opts.sourceIds ? config.synopsis_sources.filter(id => opts.sourceIds!.includes(id)) : config.synopsis_sources,
+ eligible_pages: coverage.eligible_pages,
+ eligible_chunks: coverage.eligible_chunks,
+ synopsis_pages: coverage.synopsis_pages,
+ synopsis_chunks: coverage.synopsis_chunks,
+ coverage_pct: coverage.coverage_pct,
+ page_coverage_pct: coverage.eligible_pages ? Math.floor((coverage.synopsis_pages / coverage.eligible_pages) * 1000) / 10 : null,
+ large_title_pages: coverage.large_title_pages,
+ large_title_chunks: coverage.large_title_chunks,
+ ...(last ? { sweep_jobs: last.jobs, last_sweep_finished_at: last.last_finished_at } : {}),
+ fix: sweep.SYNOPSIS_SWEEP_ARGV.join(' '),
+ };
+ const message = sweep.formatSynopsisCoverageLine(coverage) + sweepNote;
+ const missingMost = coverage.coverage_pct !== null && coverage.coverage_pct < sweep.SYNOPSIS_COVERAGE_WARN_PCT;
+ const pending = coverage.eligible_chunks > coverage.synopsis_chunks;
+ return {
+ name: 'synopsis_tier_coverage',
+ status: missingMost ? 'warn' : 'ok',
+ message,
+ details,
+ ...(pending ? { fix: agentFix([...sweep.SYNOPSIS_SWEEP_ARGV],
+ 'Previews the per-chunk synopsis sweep (pages, chunks, tokens and dollars on the configured synopsis model) and changes nothing; it runs only with --yes, which is paid work the user approves.',
+ 'synopsis_tier_coverage') } : {}),
+ };
+ } catch (e) {
+ return checkError('synopsis_tier_coverage', 'check synopsis tier coverage', e);
+ }
+}
diff --git a/src/commands/doctor/report-remote.ts b/src/commands/doctor/report-remote.ts
index a868a4cd91..4b244398a8 100644
--- a/src/commands/doctor/report-remote.ts
+++ b/src/commands/doctor/report-remote.ts
@@ -66,6 +66,7 @@ import {
multiSourceDriftNotVerified,
} from './schema-pack-checks.ts';
import { managedPersistenceEnabled } from '../../core/persistence/ownership.ts';
+import { checkSynopsisTierCoverage } from './checks/synopsis-coverage.ts';
import { brainScorePlanFix, checkError } from './check-fix.ts';
import type { RenderContext } from '../../core/agent-output.ts';
@@ -436,6 +437,7 @@ export async function doctorReportRemote(
// - contextual_retrieval_mode IS NULL (mode never evaluated)
// - synopsis-failures audit JSONL entries from the last 7 days
checks.push(await checkContextualRetrievalCoverage(engine, { sourceIds: opts.sourceIds }));
+ checks.push(await checkSynopsisTierCoverage(engine, { sourceIds: opts.sourceIds }));
checks.push(await checkProjectionReadiness(engine, {
sourceIds: opts.sourceIds,
excludePrivate: await resolveExcludePrivatePages(engine, opts.remote),
diff --git a/src/commands/reindex-synopsis.ts b/src/commands/reindex-synopsis.ts
new file mode 100644
index 0000000000..9f58ec3be4
--- /dev/null
+++ b/src/commands/reindex-synopsis.ts
@@ -0,0 +1,227 @@
+/**
+ * `gbrain reindex --synopsis` (#8744): the explicit, operator-run per-chunk
+ * synopsis sweep for pages whose configured contextual-retrieval tier is
+ * per_chunk_synopsis (search.mode tokenmax, or a source or page override).
+ * Import and sync embed those pages at the free title tier and nothing else
+ * schedules the synopsis tier, so this command is how it gets applied.
+ *
+ * Without `--yes` it previews and spends nothing: the pages and chunks still
+ * on the title tier, and the token and dollar ceiling on the configured
+ * synopsis model plus the re-embed. With `--yes` it queues one
+ * `contextual_reindex_per_chunk` job per page (the existing job and handler),
+ * all under one spend authorization whose cap (`--max-usd`, else the derived
+ * cap) the worker enforces across every job, then drains them in this process.
+ * It is resumable: pages already stamped per_chunk_synopsis are never
+ * selected again, and a job left queued or running by an interrupted run is
+ * picked up again through its idempotency key (page, source and text revision).
+ * Database reads and writes only, so a repo-less brain works the same way.
+ */
+import type { BrainEngine } from '../core/engine.ts';
+import { setCliExitVerdict } from '../core/cli-force-exit.ts';
+import type { MinionJob } from '../core/minions/types.ts';
+import {
+ SYNOPSIS_SWEEP_ARGV,
+ SYNOPSIS_SWEEP_JOB,
+ formatSynopsisCoverageLine,
+ planSynopsisSweep,
+ readSynopsisCoverage,
+ type SynopsisCoverage,
+ type SynopsisSweepPlan,
+} from '../core/synopsis-sweep.ts';
+
+const TERMINAL = new Set(['completed', 'failed', 'dead', 'cancelled']);
+
+export interface SynopsisSweepRun {
+ group_id: string;
+ cap_usd: number | null;
+ job_ids: number[];
+ /** Pages whose job at their current text revision already finished in an earlier run (not queued again). */
+ already_attempted: number;
+ synopsis: number;
+ title: number;
+ skipped: number;
+ failed: number;
+}
+
+export interface ReindexSynopsisResult {
+ mode: 'not_applicable' | 'preview' | 'run';
+ reason?: string;
+ plan?: SynopsisSweepPlan;
+ coverage?: SynopsisCoverage;
+ run?: SynopsisSweepRun;
+ coverage_after?: SynopsisCoverage;
+}
+
+function flagValue(args: string[], flag: string): string | undefined {
+ const at = args.indexOf(flag);
+ return at >= 0 ? args[at + 1] : undefined;
+}
+
+function usd(value: number | null): string {
+ return value === null ? 'unknown (no price for this model)' : `$${value.toFixed(2)}`;
+}
+
+function planJson(plan: SynopsisSweepPlan) {
+ const { pages, config, ...rest } = plan;
+ return { ...rest, global_tier: config.global_tier, synopsis_sources: config.synopsis_sources, pages: pages.length };
+}
+
+export async function runReindexSynopsis(engine: BrainEngine, args: string[], opts: { pollMs?: number } = {}): Promise<ReindexSynopsisResult> {
+ const json = args.includes('--json');
+ const say = (line: string) => { if (!json) console.log(line); };
+ const emit = (result: ReindexSynopsisResult) => {
+ if (json) {
+ process.stdout.write(JSON.stringify({
+ mode: result.mode,
+ ...(result.reason ? { reason: result.reason } : {}),
+ ...(result.plan ? { plan: planJson(result.plan) } : {}),
+ ...(result.coverage ? { coverage: { ...result.coverage, config: undefined } } : {}),
+ ...(result.run ? { run: result.run } : {}),
+ ...(result.coverage_after ? { coverage_after: { ...result.coverage_after, config: undefined } } : {}),
+ run_argv: [...SYNOPSIS_SWEEP_ARGV, '--yes'],
+ }, null, 2) + '\n');
+ }
+ return result;
+ };
+
+ const limitRaw = flagValue(args, '--limit');
+ const sourceId = flagValue(args, '--source');
+ if ((args.includes('--limit') && !(limitRaw && /^\d+$/.test(limitRaw) && Number(limitRaw) > 0))
+ || (args.includes('--source') && (!sourceId || sourceId.startsWith('--')))) {
+ const error = args.includes('--limit') && !(limitRaw && /^\d+$/.test(limitRaw)) ? 'invalid --limit: expected a positive integer' : 'invalid --source: expected a value';
+ if (json) process.stdout.write(JSON.stringify({ error }) + '\n');
+ else process.stderr.write(`[reindex --synopsis] ${error}\n`);
+ setCliExitVerdict(2);
+ return { mode: 'not_applicable', reason: error };
+ }
+
+ const { embeddingsDisabled } = await import('../core/embedding-disabled.ts');
+ if (await embeddingsDisabled(engine)) {
+ const reason = 'Embeddings are disabled on this brain (keyless by choice), so there are no vectors to put on the synopsis tier; keyword search keeps working.';
+ say(`[reindex --synopsis] Nothing to do. ${reason}`);
+ return emit({ mode: 'not_applicable', reason });
+ }
+
+ const { resolveContextualSynopsisModel } = await import('../core/minions/handlers/contextual-reindex-per-chunk.ts');
+ const { resolveSynopsisMaxTokens } = await import('../core/contextual-retrieval-service.ts');
+ const synopsisModel = await resolveContextualSynopsisModel(engine);
+ let embeddingPricePerMTok: number | null = null;
+ let embeddingModel = 'unknown';
+ try {
+ const { getEmbeddingModel } = await import('../core/ai/gateway.ts');
+ const { lookupEmbeddingPrice } = await import('../core/embedding-pricing.ts');
+ embeddingModel = getEmbeddingModel();
+ const price = lookupEmbeddingPrice(embeddingModel);
+ if (price.kind === 'known') embeddingPricePerMTok = price.pricePerMTok;
+ } catch { /* no configured embedding model: the embedding share stays unknown */ }
+
+ const plan = await planSynopsisSweep(engine, {
+ synopsisModel,
+ synopsisMaxTokens: await resolveSynopsisMaxTokens(engine),
+ ...(sourceId ? { sourceId } : {}),
+ ...(limitRaw ? { limit: Number(limitRaw) } : {}),
+ embeddingPricePerMTok,
+ });
+ if (!plan.config.configured) {
+ const reason = `The configured contextual-retrieval tier is ${plan.config.kill_switch ? 'off (kill switch)' : plan.config.global_tier}, not per_chunk_synopsis, so no page needs the synopsis sweep. `
+ + 'search.mode tokenmax (or search.contextual_retrieval per_chunk_synopsis, or gbrain sources set-cr-mode <id> per_chunk_synopsis) selects it.';
+ say(`[reindex --synopsis] Nothing to do. ${reason}`);
+ return emit({ mode: 'not_applicable', reason, plan });
+ }
+ const coverage = await readSynopsisCoverage(engine, { config: plan.config, ...(sourceId ? { sourceIds: [sourceId] } : {}) });
+ const tierSource = plan.config.global_tier === 'per_chunk_synopsis' ? 'search mode' : `source override (${plan.config.synopsis_sources.join(', ')})`;
+ say(`[reindex --synopsis] Configured tier: per_chunk_synopsis (${tierSource}).`);
+ say(` ${formatSynopsisCoverageLine(coverage)}`);
+ if (plan.pages.length === 0) {
+ say(' Nothing to sweep: every eligible page is already on the synopsis tier.');
+ return emit({ mode: 'preview', plan, coverage });
+ }
+ say(` The sweep would synopsize ${plan.pages.length} page(s), ${plan.chunks} chunk(s), on ${synopsisModel}:`);
+ say(` synopsis input tokens up to ${plan.est_input_tokens} (every prompt uncached; the prompt cache usually serves most)`);
+ say(` synopsis output tokens up to ${plan.est_output_tokens} (the output cap per chunk)`);
+ say(` synopsis cost up to ${usd(plan.est_synopsis_usd)}`);
+ say(` re-embed cost about ${usd(plan.est_embedding_usd)} (${embeddingModel})`);
+ say(` total up to ${usd(plan.est_usd)}`);
+ if (plan.remaining_pages > 0) say(` ${plan.remaining_pages} more pending page(s) are beyond --limit; a later run picks them up.`);
+
+ if (!args.includes('--yes')) {
+ say(` Preview only; nothing was queued or spent. Run it (paid, resumable): ${[...SYNOPSIS_SWEEP_ARGV, ...(sourceId ? ['--source', sourceId] : []), ...(limitRaw ? ['--limit', limitRaw] : []), '--yes'].join(' ')} [--max-usd <usd>]`);
+ return emit({ mode: 'preview', plan, coverage });
+ }
+
+ const { requireConsent } = await import('../core/consent.ts');
+ const { engineConsentEnv } = await import('../core/consent-cli.ts');
+ const { jobSpendAuthorization, spendSubmitSummary } = await import('../core/minions/spend-authorization.ts');
+ const argv = ['gbrain', 'reindex', ...args.filter(a => a !== '--yes' && a !== '--json')];
+ const previewArgv = argv.filter((a, i) => a !== '--max-usd' && argv[i - 1] !== '--max-usd' && !a.startsWith('--max-usd='));
+ const auth = await requireConsent({
+ command: 'reindex --synopsis', effects: ['paid'], actor: 'agent',
+ what: `Write per-chunk synopses for ${plan.pages.length} page(s) (${plan.chunks} chunk(s)) and re-embed them`,
+ why: `The configured contextual-retrieval tier is per_chunk_synopsis, but these pages search on title-tier vectors until the sweep runs. Each chunk costs one ${synopsisModel} call plus its re-embed (up to ${usd(plan.est_usd)}).`,
+ risk: 'Spends money with the chat and embedding providers. Search keeps working on the current vectors while it runs; a page whose synopsis is refused stays on the title tier.',
+ user_message: `Put ${plan.pages.length} page(s) on the per-chunk synopsis tier for up to ${usd(plan.est_usd)}?`,
+ argv, preview_argv: previewArgv,
+ est_usd: plan.est_usd, args,
+ }, engineConsentEnv(engine));
+ const spend = jobSpendAuthorization(auth, { command: 'reindex --synopsis', of: plan.pages.length, argv, ...(plan.est_usd !== null ? { est_usd: plan.est_usd } : {}) });
+ if (spend.cap_usd !== null && plan.est_usd !== null && plan.est_usd > spend.cap_usd) {
+ say(` The $${spend.cap_usd.toFixed(2)} cap is below the ${usd(plan.est_usd)} ceiling, so the run may stop part-way; run the same command again (with a higher --max-usd) to continue.`);
+ }
+
+ const { MinionQueue } = await import('../core/minions/queue.ts');
+ const queue = new MinionQueue(engine);
+ await queue.ensureSchema();
+ const jobs: MinionJob[] = [];
+ for (const page of plan.pages) {
+ jobs.push(await queue.add(SYNOPSIS_SWEEP_JOB, { page_slug: page.slug, expected_source_id: page.source_id },
+ { idempotency_key: `${SYNOPSIS_SWEEP_JOB}:${page.source_id}:${page.slug}:${page.revision ?? 'unrevised'}` },
+ { allowProtectedSubmit: true, spendAuthorization: spend }));
+ }
+ for (const line of spendSubmitSummary(spend, jobs, argv).lines) say(` ${line}`);
+ const alreadyAttempted = jobs.filter(j => TERMINAL.has(j.status)).length;
+ const pendingIds = jobs.filter(j => !TERMINAL.has(j.status)).map(j => j.id);
+ if (alreadyAttempted) say(` ${alreadyAttempted} page(s) already had a finished sweep job at their current revision; they are not queued again until the page changes.`);
+
+ if (pendingIds.length) {
+ say(` Running ${pendingIds.length} job(s) in this process...`);
+ const { MinionWorker } = await import('../core/minions/worker.ts');
+ const { makeContextualReindexHandler } = await import('../core/minions/handlers/contextual-reindex-per-chunk.ts');
+ const pollMs = opts.pollMs ?? 500;
+ const worker = new MinionWorker(engine, { pollInterval: pollMs, healthCheckInterval: 0 });
+ worker.register(SYNOPSIS_SWEEP_JOB, makeContextualReindexHandler({ engine }));
+ const running = worker.start();
+ let settled = false;
+ const watch = (async () => {
+ while (!settled) {
+ await new Promise(resolve => setTimeout(resolve, pollMs));
+ const rows = await engine.executeRaw<{ status: string }>('SELECT status FROM minion_jobs WHERE id = ANY($1::int[])', [pendingIds]);
+ if (rows.every(r => TERMINAL.has(r.status))) { settled = true; worker.stop(); }
+ }
+ })();
+ await running;
+ settled = true;
+ await watch.catch(() => undefined);
+ if (worker.configurationError) {
+ const { reportInlineWorkerConfiguration } = await import('./jobs-readiness.ts');
+ reportInlineWorkerConfiguration(worker.configurationError);
+ }
+ }
+
+ const finished = await engine.executeRaw<{ id: number | string; status: string; result: unknown; error_text: string | null }>(
+ 'SELECT id, status, result, error_text FROM minion_jobs WHERE id = ANY($1::int[])', [jobs.map(j => j.id)]);
+ const run: SynopsisSweepRun = { group_id: spend.group_id, cap_usd: spend.cap_usd, job_ids: jobs.map(j => j.id), already_attempted: alreadyAttempted,
+ synopsis: 0, title: 0, skipped: 0, failed: 0 };
+ for (const row of finished) {
+ const result = (typeof row.result === 'string' ? JSON.parse(row.result) : row.result) as { mode_applied?: string } | null;
+ if (row.status !== 'completed') { run.failed++; continue; }
+ if (result?.mode_applied === 'per_chunk_synopsis') run.synopsis++;
+ else if (result?.mode_applied === 'skipped') run.skipped++;
+ else run.title++;
+ }
+ const coverageAfter = await readSynopsisCoverage(engine, { config: plan.config, ...(sourceId ? { sourceIds: [sourceId] } : {}) });
+ say(` Done: ${run.synopsis} page(s) on the synopsis tier, ${run.title} on the title tier (synopsis refused), ${run.skipped} skipped, ${run.failed} failed.`);
+ if (run.failed) say(` Failed jobs keep their reason: gbrain jobs list --status dead. Run the same command again to resume.`);
+ say(` ${formatSynopsisCoverageLine(coverageAfter)}`);
+ if (run.failed) setCliExitVerdict(1);
+ return emit({ mode: 'run', plan, coverage, run, coverage_after: coverageAfter });
+}
diff --git a/src/commands/reindex.ts b/src/commands/reindex.ts
index 52bdb16ec2..8fdca8ea8f 100644
--- a/src/commands/reindex.ts
+++ b/src/commands/reindex.ts
@@ -92,6 +92,7 @@ USAGE
gbrain reindex --vectors [--dry-run] [--json]
gbrain reindex --oversized [--limit N] [--dry-run] [--json] [--source <id>]
gbrain reindex --segmented [--source <id>] [--slug <slug>] [--apply] [--json]
+ gbrain reindex --synopsis [--source <id>] [--limit N] [--yes] [--max-usd USD] [--json]
TARGETS (exactly one required)
--markdown Re-chunk markdown pages whose chunker_version lags the
@@ -121,10 +122,20 @@ TARGETS (exactly one required)
only the segments that change, keeps the vectors of
unchanged chunk texts and leaves new texts to the embed
sweep (gbrain embed --stale). Never runs on upgrade.
+ --synopsis Put pages whose configured contextual-retrieval tier is
+ per_chunk_synopsis (search.mode tokenmax) on that tier:
+ import and sync embed them at the title tier and nothing
+ else schedules the synopsis sweep. Previews by default
+ (pages, chunks, tokens and dollars on the configured
+ synopsis model; spends nothing). --yes queues one
+ contextual_reindex_per_chunk job per page under one spend
+ cap (--max-usd, else derived from the estimate) and runs
+ them in this process. Resumable: run it again to continue.
+ Inline pages over 1 MiB stay on the title tier.
OPTIONS
--type <t> --markdown only: restrict to one page type
- --limit N Cap pages/chunks processed this run
+ --limit N Cap pages/chunks processed this run (--synopsis: pages)
--workers N --markdown and --multimodal: parallel writers per batch
(--concurrency is an alias). --markdown on Postgres
defaults to 4 when more than 100 pages are pending;
@@ -136,8 +147,10 @@ OPTIONS
first (exit 3 without approval when not interactive;
--dry-run, --no-embed and a keyless brain never ask);
--multimodal skips its cost confirm
- --max-usd USD --markdown: approve the paid run with this cost cap
- --source <id> --aliases, --oversized and --segmented: restrict to one source
+ --max-usd USD --markdown: approve the paid run with this cost cap;
+ --synopsis: the cap shared by every queued job (with --yes)
+ --source <id> --aliases, --oversized, --segmented and --synopsis: restrict
+ to one source
--slug <slug> --segmented only: one page
--apply --segmented only: apply the previewed re-chunk
--repo PATH --markdown only: brain repo override
@@ -185,6 +198,7 @@ export function validateReindexModeScope(args: string[]): string | null {
if (args.includes('--aliases')) return '--type is only supported with reindex --markdown, not --aliases';
if (args.includes('--oversized')) return '--type is only supported with reindex --markdown, not --oversized';
if (args.includes('--segmented')) return '--type is only supported with reindex --markdown, not --segmented';
+ if (args.includes('--synopsis')) return '--type is only supported with reindex --markdown, not --synopsis';
return null;
}
diff --git a/src/commands/search.ts b/src/commands/search.ts
index 8e410b7c60..26127abb97 100644
--- a/src/commands/search.ts
+++ b/src/commands/search.ts
@@ -84,6 +84,10 @@ function formatModesText(report: SearchModesReport): string {
lines.push(`Reranker: ${rr.model} (enabled but NOT running) — ${rr.fix ?? 'see gbrain doctor'}`);
}
}
+ if (report.synopsis_coverage) {
+ lines.push('');
+ lines.push(report.synopsis_coverage.line);
+ }
lines.push('');
lines.push('Mode bundles (frozen — set via `gbrain config set search.mode <mode>`):');
for (const mode of SEARCH_MODES) {
diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.ts
index f6a8145e33..1b1675ee66 100644
--- a/src/core/doctor-categories.ts
+++ b/src/core/doctor-categories.ts
@@ -122,6 +122,8 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
'content_hash_duplicates',
'content_sanity_audit_recent',
'contextual_retrieval_coverage',
+ // #8744: configured per_chunk_synopsis tier vs the chunks that carry it.
+ 'synopsis_tier_coverage',
'contradictions',
'conversation_facts_backlog',
'conversation_format_coverage',
diff --git a/src/core/page-summary.ts b/src/core/page-summary.ts
index e58131fb9e..a1f3fd8ec1 100644
--- a/src/core/page-summary.ts
+++ b/src/core/page-summary.ts
@@ -320,7 +320,7 @@ export function segmentSynopsisWindows<C extends { chunk_text: string; chunk_sou
}
/** Input chars of a synopsis call outside its document and chunk: the system prompt and the prompt's tags. */
-const SYNOPSIS_PROMPT_OVERHEAD_CHARS = SYNOPSIS_SYSTEM_PROMPT.length + 256;
+export const SYNOPSIS_PROMPT_OVERHEAD_CHARS = SYNOPSIS_SYSTEM_PROMPT.length + 256;
/** The rate (USD per million tokens) an unpriced synopsis model reserves at. */
const UNPRICED_SYNOPSIS_RATE = { input: 15, output: 75 };
diff --git a/src/core/search/modes-report.ts b/src/core/search/modes-report.ts
index 008bfbb6a7..7babe168b7 100644
--- a/src/core/search/modes-report.ts
+++ b/src/core/search/modes-report.ts
@@ -130,6 +130,13 @@ export interface SearchModesReport {
* file-plane + process env. Absent only when readiness itself threw.
*/
reranker_readiness?: RerankerReadinessReport;
+ /**
+ * #8744: when the configured contextual-retrieval tier is per_chunk_synopsis,
+ * how many chunks actually carry synopsis vectors (import and sync embed at
+ * the title tier; the sweep is `gbrain reindex --synopsis`). Brain-wide
+ * counts, so ABSENT on the remote surface.
+ */
+ synopsis_coverage?: SynopsisCoverageReport;
/** #4604: what this report does NOT include (per-call plane). */
per_call_note: string;
_meta?: {
@@ -137,6 +144,17 @@ export interface SearchModesReport {
};
}
+export interface SynopsisCoverageReport {
+ eligible_pages: number;
+ eligible_chunks: number;
+ synopsis_pages: number;
+ synopsis_chunks: number;
+ coverage_pct: number | null;
+ large_title_pages: number;
+ line: string;
+ fix: string;
+}
+
export interface RerankerReadinessReport {
model: string;
enabled: boolean;
@@ -163,7 +181,8 @@ export function redactReadinessForRemote(report: SearchModesReport): SearchModes
// self_hosted is deployment topology (a private base-URL override exists) —
// not needed for the verdict, so it stays local too.
const { model, enabled, ready } = rr;
- return { ...report, reranker_readiness: { model, enabled, ready, self_hosted: false } };
+ const { synopsis_coverage: _brainWide, ...rest } = report;
+ return { ...rest, reranker_readiness: { model, enabled, ready, self_hosted: false } };
}
export async function buildModesReport(engine: BrainEngine): Promise<SearchModesReport> {
@@ -227,12 +246,24 @@ export async function buildModesReport(engine: BrainEngine): Promise<SearchModes
};
}
+ let synopsis_coverage: SynopsisCoverageReport | undefined;
+ try {
+ const sweep = await import('../synopsis-sweep.ts');
+ const c = await sweep.readSynopsisCoverage(engine);
+ if (c.config.configured) {
+ synopsis_coverage = { eligible_pages: c.eligible_pages, eligible_chunks: c.eligible_chunks, synopsis_pages: c.synopsis_pages,
+ synopsis_chunks: c.synopsis_chunks, coverage_pct: c.coverage_pct, large_title_pages: c.large_title_pages,
+ line: sweep.formatSynopsisCoverageLine(c), fix: sweep.SYNOPSIS_SWEEP_ARGV.join(' ') };
+ }
+ } catch { /* coverage is advisory; the doctor check synopsis_tier_coverage reports read failures */ }
+
return {
schema_version: 2,
active_mode: resolved.resolved_mode,
active_mode_valid: resolved.mode_valid,
resolved: attributions,
...(reranker_readiness ? { reranker_readiness } : {}),
+ ...(synopsis_coverage ? { synopsis_coverage } : {}),
bundles: {
conservative: { ...MODE_BUNDLES.conservative },
balanced: { ...MODE_BUNDLES.balanced },
diff --git a/src/core/synopsis-sweep.ts b/src/core/synopsis-sweep.ts
new file mode 100644
index 0000000000..b50f66a2d2
--- /dev/null
+++ b/src/core/synopsis-sweep.ts
@@ -0,0 +1,248 @@
+/**
+ * #8744: the per-chunk synopsis sweep for brains whose configured contextual
+ * retrieval tier is `per_chunk_synopsis` (search.mode tokenmax, or a source or
+ * page override). Import and sync embed those pages at the free title tier and
+ * leave the synopsis tier to the `contextual_reindex_per_chunk` job, which no
+ * automatic path submits; `gbrain reindex --synopsis` is the explicit,
+ * operator-run sweep. This module holds the reads shared by that command, the
+ * doctor check `synopsis_tier_coverage` and the `gbrain search modes` line:
+ * which pages resolve to the synopsis tier (the SQL twin of
+ * resolveContextualRetrievalMode), how many of their chunks carry synopsis
+ * vectors, and what the sweep would cost. Database reads only, so a repo-less
+ * brain behaves exactly like a repo brain.
+ *
+ * Coverage is counted per chunk by the page's stamped mode: a page is stamped
+ * `per_chunk_synopsis` only once every chunk is settled. Inline pages over the
+ * large-page threshold stay on the title tier by design (contextual-retrieval
+ * service, plan B8), so they leave the denominator and are reported apart.
+ */
+import type { BrainEngine } from './engine.ts';
+import { loadSearchModeConfig, resolveSearchMode } from './search/mode.ts';
+import { largePageBodyFilter } from './page-size-policy.ts';
+import {
+ SEGMENT_SYNOPSIS_LEAD_BYTES,
+ SEGMENT_SYNOPSIS_WINDOW_BYTES,
+ SYNOPSIS_DOC_MAX_CHARS,
+ SYNOPSIS_MAX_TOKENS,
+ SYNOPSIS_PROMPT_OVERHEAD_CHARS,
+} from './page-summary.ts';
+import { reservationCostUsd } from './budget/reservation-cost.ts';
+import type { CRMode } from './types.ts';
+
+export const SYNOPSIS_SWEEP_JOB = 'contextual_reindex_per_chunk';
+export const SYNOPSIS_SWEEP_ARGV = ['gbrain', 'reindex', '--synopsis'] as const;
+/** Below this share of eligible chunks on the synopsis tier, the configured tier is "missing from most chunks". */
+export const SYNOPSIS_COVERAGE_WARN_PCT = 50;
+/** Embedding-input chars a synopsis adds to a chunk at most (contextual-retrieval-service SYNOPSIS_WRAPPER_CHARS). */
+const SYNOPSIS_WRAPPER_CHARS = 320;
+
+export interface SynopsisTierConfig {
+ /** The brain-level tier (search.mode bundle or search.contextual_retrieval override). */
+ global_tier: CRMode;
+ kill_switch: boolean;
+ /** Sources whose own contextual_retrieval_mode is per_chunk_synopsis. */
+ synopsis_sources: string[];
+ /** True when any page can resolve to the synopsis tier (page frontmatter overrides aside). */
+ configured: boolean;
+}
+
+export async function readSynopsisTierConfig(engine: BrainEngine): Promise<SynopsisTierConfig> {
+ const knobs = resolveSearchMode(await loadSearchModeConfig(engine));
+ const rows = await engine.executeRaw<{ id: string }>(
+ `SELECT id FROM sources WHERE contextual_retrieval_mode = 'per_chunk_synopsis' AND NOT archived ORDER BY id`);
+ const synopsisSources = rows.map(r => r.id);
+ const killSwitch = knobs.contextual_retrieval_disabled === true;
+ return {
+ global_tier: knobs.contextual_retrieval,
+ kill_switch: killSwitch,
+ synopsis_sources: synopsisSources,
+ configured: !killSwitch && (knobs.contextual_retrieval === 'per_chunk_synopsis' || synopsisSources.length > 0),
+ };
+}
+
+/**
+ * Markdown pages whose resolved tier is per_chunk_synopsis, with their stamped
+ * mode, large-inline flag and chunk counts. Mirrors resolveContextualRetrievalMode:
+ * a valid frontmatter value wins on the host source or a trusted mount, then a
+ * valid source-row mode, then the global tier. `$1` is the global tier.
+ */
+function eligiblePagesSql(opts: { sourceParam?: string; withChars: boolean }): string {
+ return `
+ WITH chunk_stats AS (
+ SELECT c.page_id, count(*)::bigint AS chunks${opts.withChars ? ', COALESCE(sum(length(c.chunk_text)), 0)::bigint AS chunk_chars' : ''}
+ FROM content_chunks c
+ WHERE c.chunk_source <> 'image_asset'
+ GROUP BY c.page_id
+ ), resolved AS (
+ SELECT p.id, p.slug, p.source_id, p.contextual_retrieval_mode AS stamped,
+ p.text_projection_revision AS revision,
+ p.body_storage = 'segmented' AS segmented,
+ (COALESCE(p.body_storage, 'inline') <> 'segmented' AND ${largePageBodyFilter('p')}) AS large_inline,
+ CASE WHEN p.body_storage = 'segmented' THEN COALESCE(p.body_bytes, 0)
+ ELSE length(COALESCE(p.compiled_truth, '')) + length(COALESCE(p.timeline, '')) END::bigint AS body_chars,
+ CASE
+ WHEN p.frontmatter ? 'contextual_retrieval'
+ AND (p.source_id = 'default' OR COALESCE(s.trust_frontmatter_overrides, false))
+ AND p.frontmatter->>'contextual_retrieval' IN ('none', 'title', 'per_chunk_synopsis')
+ THEN p.frontmatter->>'contextual_retrieval'
+ WHEN s.contextual_retrieval_mode IN ('none', 'title', 'per_chunk_synopsis') THEN s.contextual_retrieval_mode
+ ELSE $1::text
+ END AS effective
+ FROM pages p
+ LEFT JOIN sources s ON s.id = p.source_id
+ WHERE p.deleted_at IS NULL AND p.page_kind = 'markdown' AND p.type <> 'extract_receipt'
+ AND NOT (p.frontmatter ? 'embed_skip')
+ AND NOT COALESCE(s.archived, false)
+ ${opts.sourceParam ? `AND p.source_id = ANY(${opts.sourceParam}::text[])` : ''}
+ )
+ SELECT r.*, cs.chunks${opts.withChars ? ', cs.chunk_chars' : ''}
+ FROM resolved r
+ JOIN chunk_stats cs ON cs.page_id = r.id
+ WHERE r.effective = 'per_chunk_synopsis'`;
+}
+
+export interface SynopsisCoverage {
+ config: SynopsisTierConfig;
+ eligible_pages: number;
+ eligible_chunks: number;
+ synopsis_pages: number;
+ synopsis_chunks: number;
+ /** Chunks of eligible pages on the synopsis tier, as a percentage (0..100, one decimal); null with no eligible chunks. */
+ coverage_pct: number | null;
+ /** Inline pages over the large-page threshold: the title tier by design, outside the denominator. */
+ large_title_pages: number;
+ large_title_chunks: number;
+}
+
+/** Synopsis-tier coverage. `sourceIds` confines every count to those sources (the remote doctor's grant). */
+export async function readSynopsisCoverage(engine: BrainEngine, opts: { sourceIds?: string[]; config?: SynopsisTierConfig } = {}): Promise<SynopsisCoverage> {
+ const config = opts.config ?? await readSynopsisTierConfig(engine);
+ const empty = { eligible_pages: 0, eligible_chunks: 0, synopsis_pages: 0, synopsis_chunks: 0, coverage_pct: null, large_title_pages: 0, large_title_chunks: 0 };
+ if (!config.configured) return { config, ...empty };
+ const rows = await engine.executeRaw<Record<string, string | number>>(
+ `SELECT
+ count(*) FILTER (WHERE NOT large_inline)::bigint AS eligible_pages,
+ COALESCE(sum(chunks) FILTER (WHERE NOT large_inline), 0)::bigint AS eligible_chunks,
+ count(*) FILTER (WHERE NOT large_inline AND stamped = 'per_chunk_synopsis')::bigint AS synopsis_pages,
+ COALESCE(sum(chunks) FILTER (WHERE NOT large_inline AND stamped = 'per_chunk_synopsis'), 0)::bigint AS synopsis_chunks,
+ count(*) FILTER (WHERE large_inline)::bigint AS large_title_pages,
+ COALESCE(sum(chunks) FILTER (WHERE large_inline), 0)::bigint AS large_title_chunks
+ FROM (${eligiblePagesSql({ sourceParam: opts.sourceIds ? '$2' : undefined, withChars: false })}) eligible`,
+ opts.sourceIds ? [config.global_tier, opts.sourceIds] : [config.global_tier]);
+ const n = (key: string) => Number(rows[0]?.[key] ?? 0);
+ const eligibleChunks = n('eligible_chunks');
+ return {
+ config,
+ eligible_pages: n('eligible_pages'),
+ eligible_chunks: eligibleChunks,
+ synopsis_pages: n('synopsis_pages'),
+ synopsis_chunks: n('synopsis_chunks'),
+ coverage_pct: eligibleChunks > 0 ? Math.floor((n('synopsis_chunks') / eligibleChunks) * 1000) / 10 : null,
+ large_title_pages: n('large_title_pages'),
+ large_title_chunks: n('large_title_chunks'),
+ };
+}
+
+export interface SynopsisSweepPage {
+ id: number;
+ slug: string;
+ source_id: string;
+ revision: string | null;
+ chunks: number;
+}
+
+export interface SynopsisSweepPlan {
+ config: SynopsisTierConfig;
+ synopsis_model: string;
+ pages: SynopsisSweepPage[];
+ chunks: number;
+ /** Ceiling: every synopsis prompt uncached (the prompt cache usually serves all but a page's first call). */
+ est_input_tokens: number;
+ /** Ceiling: every synopsis at the output-token cap. */
+ est_output_tokens: number;
+ /** Null when the synopsis model has no known price. */
+ est_synopsis_usd: number | null;
+ /** Null when the embedding model has no known price (or embedding is not configured). */
+ est_embedding_usd: number | null;
+ est_usd: number | null;
+ /** Pending pages beyond `limit` that this run leaves for the next one. */
+ remaining_pages: number;
+}
+
+/**
+ * The pages a sweep would queue (eligible, not large-inline, not yet stamped
+ * per_chunk_synopsis), oldest id first, with the token and dollar estimate on
+ * the configured synopsis model. A page's synopsis document is its body capped
+ * at SYNOPSIS_DOC_MAX_CHARS (a segmented page: its lead plus one segment window),
+ * sent once per chunk with the chunk and the prompt overhead, at 4 chars per
+ * token (estimateSynopsisCallUsd's rate).
+ */
+export async function planSynopsisSweep(engine: BrainEngine, opts: {
+ synopsisModel: string;
+ synopsisMaxTokens?: number;
+ sourceId?: string;
+ limit?: number;
+ embeddingPricePerMTok?: number | null;
+}): Promise<SynopsisSweepPlan> {
+ const config = await readSynopsisTierConfig(engine);
+ const base = { config, synopsis_model: opts.synopsisModel, pages: [], chunks: 0, est_input_tokens: 0, est_output_tokens: 0,
+ est_synopsis_usd: 0, est_embedding_usd: 0, est_usd: 0, remaining_pages: 0 };
+ if (!config.configured) return base;
+ const rows = await engine.executeRaw<{ id: number | string; slug: string; source_id: string; revision: string | null; segmented: boolean | null;
+ body_chars: number | string; chunks: number | string; chunk_chars: number | string }>(
+ `SELECT id, slug, source_id, revision, segmented, body_chars, chunks, chunk_chars
+ FROM (${eligiblePagesSql({ sourceParam: opts.sourceId ? '$2' : undefined, withChars: true })}) eligible
+ WHERE NOT large_inline AND stamped IS DISTINCT FROM 'per_chunk_synopsis'
+ ORDER BY id`,
+ opts.sourceId ? [config.global_tier, [opts.sourceId]] : [config.global_tier]);
+ const selected = opts.limit ? rows.slice(0, opts.limit) : rows;
+ const maxTokens = opts.synopsisMaxTokens ?? SYNOPSIS_MAX_TOKENS;
+ const segmentedDocChars = Math.min(SEGMENT_SYNOPSIS_LEAD_BYTES + SEGMENT_SYNOPSIS_WINDOW_BYTES, SYNOPSIS_DOC_MAX_CHARS);
+ let chunks = 0;
+ let inputTokens = 0;
+ let embeddedChars = 0;
+ for (const row of selected) {
+ const pageChunks = Number(row.chunks);
+ const chunkChars = Number(row.chunk_chars);
+ const docChars = row.segmented ? Math.min(Number(row.body_chars), segmentedDocChars) : Math.min(Number(row.body_chars), SYNOPSIS_DOC_MAX_CHARS);
+ chunks += pageChunks;
+ inputTokens += Math.ceil((pageChunks * (docChars + SYNOPSIS_PROMPT_OVERHEAD_CHARS) + chunkChars) / 4);
+ embeddedChars += chunkChars + pageChunks * SYNOPSIS_WRAPPER_CHARS;
+ }
+ const outputTokens = chunks * maxTokens;
+ const synopsisUsd = chunks === 0 ? 0 : reservationCostUsd(opts.synopsisModel, 'chat', inputTokens, outputTokens);
+ const embeddingUsd = opts.embeddingPricePerMTok == null ? null : (Math.ceil(embeddedChars / 3.5) / 1_000_000) * opts.embeddingPricePerMTok;
+ return {
+ ...base,
+ pages: selected.map(r => ({ id: Number(r.id), slug: r.slug, source_id: r.source_id, revision: r.revision ?? null, chunks: Number(r.chunks) })),
+ chunks,
+ est_input_tokens: inputTokens,
+ est_output_tokens: outputTokens,
+ est_synopsis_usd: synopsisUsd,
+ est_embedding_usd: embeddingUsd,
+ est_usd: synopsisUsd === null || embeddingUsd === null ? null : synopsisUsd + embeddingUsd,
+ remaining_pages: rows.length - selected.length,
+ };
+}
+
+/** The newest finished sweep job and how many ever ran (null when the queue table is unreadable). */
+export async function readLastSynopsisSweep(engine: BrainEngine): Promise<{ jobs: number; last_finished_at: string | null } | null> {
+ try {
+ const rows = await engine.executeRaw<{ jobs: number | string; last: string | Date | null }>(
+ `SELECT count(*)::bigint AS jobs, max(finished_at) AS last FROM minion_jobs WHERE name = $1`, [SYNOPSIS_SWEEP_JOB]);
+ const last = rows[0]?.last ?? null;
+ return { jobs: Number(rows[0]?.jobs ?? 0), last_finished_at: last === null ? null : new Date(last).toISOString() };
+ } catch {
+ return null;
+ }
+}
+
+/** One line for `gbrain search modes` and the doctor message: the configured tier's chunk coverage. */
+export function formatSynopsisCoverageLine(c: SynopsisCoverage): string {
+ if (c.eligible_chunks === 0) return 'Synopsis tier: configured (per_chunk_synopsis); no chunked pages resolve to it yet.';
+ const large = c.large_title_pages ? ` ${c.large_title_pages} large page(s) (${c.large_title_chunks} chunk(s)) stay on the title tier by design.` : '';
+ const pending = c.eligible_chunks - c.synopsis_chunks;
+ return `Synopsis tier: ${c.coverage_pct}% of chunks (${c.synopsis_chunks}/${c.eligible_chunks}) and ${c.synopsis_pages}/${c.eligible_pages} page(s) carry per-chunk synopsis vectors`
+ + (pending > 0 ? `; ${pending} chunk(s) search on title-tier vectors. Preview the sweep: ${SYNOPSIS_SWEEP_ARGV.join(' ')}` : '')
+ + `.${large}`;
+}
diff --git a/test/e2e/synopsis-sweep-8744-postgres.test.ts b/test/e2e/synopsis-sweep-8744-postgres.test.ts
new file mode 100644
index 0000000000..58538eaffe
--- /dev/null
+++ b/test/e2e/synopsis-sweep-8744-postgres.test.ts
@@ -0,0 +1,3 @@
+import { registerPostgresTests } from '../helpers/test-backends.ts';
+
+await registerPostgresTests(() => import('../synopsis-sweep-8744.serial.test.ts'));
diff --git a/test/fixtures/goldens/doctor/registry-build-checks.json b/test/fixtures/goldens/doctor/registry-build-checks.json
index 78d149cec1..775f640611 100644
--- a/test/fixtures/goldens/doctor/registry-build-checks.json
+++ b/test/fixtures/goldens/doctor/registry-build-checks.json
@@ -1089,6 +1089,10 @@
"category": "ops",
"name": "search_mode"
},
+ {
+ "category": "brain",
+ "name": "synopsis_tier_coverage"
+ },
{
"category": "brain",
"name": "hidden_by_search_policy"
diff --git a/test/fixtures/goldens/doctor/registry-report-remote.json b/test/fixtures/goldens/doctor/registry-report-remote.json
index 697343e2c4..d2ba5b7c1d 100644
--- a/test/fixtures/goldens/doctor/registry-report-remote.json
+++ b/test/fixtures/goldens/doctor/registry-report-remote.json
@@ -165,6 +165,10 @@
"category": "brain",
"name": "contextual_retrieval_coverage"
},
+ {
+ "category": "brain",
+ "name": "synopsis_tier_coverage"
+ },
{
"category": "brain",
"name": "text_projection_readiness"
diff --git a/test/synopsis-sweep-8744.serial.test.ts b/test/synopsis-sweep-8744.serial.test.ts
new file mode 100644
index 0000000000..1ba91d9907
--- /dev/null
+++ b/test/synopsis-sweep-8744.serial.test.ts
@@ -0,0 +1,195 @@
+/**
+ * #8744: with search.mode tokenmax (contextual retrieval per_chunk_synopsis),
+ * import embeds pages at the title tier and nothing scheduled the synopsis
+ * sweep, so a brain searched on title-tier vectors with no warning. Pins:
+ * - doctor synopsis_tier_coverage warns on a tokenmax brain with no
+ * synopsis vectors (coverage and the gbrain reindex --synopsis fix), and
+ * is not applicable on a balanced brain;
+ * - the preview's pages, chunks, tokens and dollars on the configured
+ * synopsis model, and that it queues and spends nothing;
+ * - a --yes run under a cap below one synopsis call is refused by the
+ * shared spend group and stamps nothing; a later run resumes;
+ * - --yes queues one contextual_reindex_per_chunk job per page under one
+ * spend group and completes them in-process; the pages land on the
+ * synopsis tier, the check turns ok, and a rerun has nothing to sweep;
+ * - inline pages over the large-page threshold stay out of the sweep.
+ * The brains here have no git repository (repo-less): the sweep reads and
+ * writes the database only.
+ *
+ * Fake chat + embed transports (no provider is called). Serial: mutates the
+ * process-global gateway, its transports and env vars. Runs on PGLite, and on
+ * Postgres through test/e2e/synopsis-sweep-8744-postgres.test.ts.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { __setChatTransportForTests, __setEmbedTransportForTests, configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
+import { checkSynopsisTierCoverage } from '../src/commands/doctor/checks/synopsis-coverage.ts';
+import { runReindexSynopsis } from '../src/commands/reindex-synopsis.ts';
+import { estimateSynopsisCallUsd, SYNOPSIS_MAX_TOKENS } from '../src/core/page-summary.ts';
+import { reservationCostUsd } from '../src/core/budget/reservation-cost.ts';
+import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';
+import { buildModesReport, redactReadinessForRemote } from '../src/core/search/modes-report.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const DIMS = 1536;
+const EMBED_MODEL = 'openai:text-embedding-3-large';
+const SYNOPSIS_MODEL = 'anthropic:claude-haiku-4-5-20251001';
+let chatCalls = 0;
+let home: string;
+const saved = { home: process.env.GBRAIN_HOME, model: process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL };
+
+beforeAll(() => {
+ home = mkdtempSync(join(tmpdir(), 'gbrain-synopsis-sweep-'));
+ process.env.GBRAIN_HOME = home;
+ process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL = SYNOPSIS_MODEL;
+ configureGateway({ embedding_model: EMBED_MODEL, embedding_dimensions: DIMS, env: { OPENAI_API_KEY: 'sk-test-fake', ANTHROPIC_API_KEY: 'sk-ant-test-fake' } });
+ __setEmbedTransportForTests((async ({ values }: { values: string[] }) =>
+ ({ embeddings: values.map(() => new Array(DIMS).fill(0.001)), usage: { tokens: 0 } })) as never);
+ __setChatTransportForTests(async () => {
+ chatCalls++;
+ return { text: 'Synopsis: a note about the example topic.', blocks: [], stopReason: 'end',
+ usage: { input_tokens: 500, output_tokens: 12, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: SYNOPSIS_MODEL, providerId: 'anthropic' } as never;
+ });
+});
+
+afterAll(() => {
+ __setChatTransportForTests(null);
+ __setEmbedTransportForTests(null);
+ resetGateway();
+ if (saved.home === undefined) delete process.env.GBRAIN_HOME; else process.env.GBRAIN_HOME = saved.home;
+ if (saved.model === undefined) delete process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL; else process.env.GBRAIN_CONTEXTUAL_SYNOPSIS_MODEL = saved.model;
+ rmSync(home, { recursive: true, force: true });
+});
+
+beforeEach(() => { chatCalls = 0; });
+
+const body = (title: string, paragraphs: number) => `---\ntype: note\ntitle: ${title}\n---\n\n${Array.from({ length: paragraphs }, (_, i) => `Paragraph ${i} about ${title}. `.repeat(30)).join('\n\n')}\n`;
+const quiet = async <T>(fn: () => Promise<T>): Promise<T> => {
+ const log = console.log;
+ const write = process.stderr.write.bind(process.stderr);
+ console.log = () => {};
+ process.stderr.write = (() => true) as typeof process.stderr.write;
+ try { return await fn(); } finally { console.log = log; process.stderr.write = write; }
+};
+
+for (const kind of testBackends()) describe(`#8744 synopsis sweep (${kind})`, () => {
+ let engine: BrainEngine, close: (() => Promise<void>) | undefined;
+ beforeAll(async () => {
+ if (kind === 'postgres') ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!));
+ else { engine = new PGLiteEngine(); await engine.connect({ embedding_dimensions: DIMS } as never); await engine.initSchema(); }
+ await engine.setConfig('search.mode', 'balanced');
+ await importFromContent(engine, 'notes/alpha-example', body('Alpha example', 2), { sourceId: 'default' });
+ await importFromContent(engine, 'notes/beta-example', body('Beta example', 6), { sourceId: 'default' });
+ await importFromContent(engine, 'notes/large-example', body('Large example', 1), { sourceId: 'default' });
+ // An inline page over the 1 MiB large-page threshold (legacy shape): the title tier by design.
+ await engine.executeRaw(`UPDATE pages SET compiled_truth = repeat('x', 1100000) WHERE slug = 'notes/large-example'`);
+ }, 120_000);
+ afterAll(async () => { if (close) await close(); else await engine.disconnect(); });
+
+ const chunkStats = async (slug: string) => {
+ const rows = await engine.executeRaw<{ chunk_text: string; body: string }>(
+ `SELECT c.chunk_text, COALESCE(p.compiled_truth, '') || COALESCE(p.timeline, '') AS body
+ FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE p.slug = $1 AND c.chunk_source <> 'image_asset' ORDER BY c.chunk_index`, [slug]);
+ return rows;
+ };
+
+ test('balanced brain: the check is not applicable and the command has nothing to do', async () => {
+ const check = await checkSynopsisTierCoverage(engine);
+ expect(check.status).toBe('ok');
+ expect(check.message).toContain('Not applicable');
+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis']));
+ expect(result.mode).toBe('not_applicable');
+ });
+
+ test('tokenmax brain with no synopsis vectors: doctor warns with coverage and the preview command', async () => {
+ await engine.setConfig('search.mode', 'tokenmax');
+ const modes = await engine.executeRaw<{ mode: string | null }>(`SELECT contextual_retrieval_mode AS mode FROM pages ORDER BY slug`);
+ expect(modes.every(m => m.mode !== 'per_chunk_synopsis')).toBe(true);
+ const check = await checkSynopsisTierCoverage(engine);
+ expect(check.status).toBe('warn');
+ expect(check.message).toContain('0% of chunks');
+ expect(check.message).toContain('gbrain reindex --synopsis');
+ expect(check.message).toContain('No synopsis sweep job has ever run');
+ expect(check.message).toContain('1 large page(s)');
+ expect(check.details).toMatchObject({ coverage_pct: 0, synopsis_chunks: 0, eligible_pages: 2, large_title_pages: 1, sweep_jobs: 0 });
+ expect(check.fix?.argv).toEqual(['gbrain', 'reindex', '--synopsis']);
+ const report = await buildModesReport(engine);
+ expect(report.synopsis_coverage).toMatchObject({ coverage_pct: 0, synopsis_chunks: 0, fix: 'gbrain reindex --synopsis' });
+ expect(report.synopsis_coverage!.line).toContain('0% of chunks');
+ expect(redactReadinessForRemote(report).synopsis_coverage).toBeUndefined();
+ });
+
+ test('preview: pages, chunks, tokens and dollars on the configured synopsis model; nothing queued', async () => {
+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis']));
+ expect(result.mode).toBe('preview');
+ const plan = result.plan!;
+ expect(plan.synopsis_model).toBe(SYNOPSIS_MODEL);
+ expect(plan.pages.map(p => p.slug)).toEqual(['notes/alpha-example', 'notes/beta-example']);
+ const alpha = await chunkStats('notes/alpha-example');
+ const beta = await chunkStats('notes/beta-example');
+ expect(alpha.length).toBeGreaterThan(0);
+ expect(beta.length).toBeGreaterThan(alpha.length);
+ const all = [...alpha, ...beta];
+ expect(plan.chunks).toBe(all.length);
+ expect(plan.est_output_tokens).toBe(all.length * SYNOPSIS_MAX_TOKENS);
+ // The ceiling matches the per-call reservation estimator summed over every chunk (one call per chunk).
+ const perCall = all.reduce((sum, c) => sum + estimateSynopsisCallUsd(SYNOPSIS_MODEL, c.body, c.chunk_text, SYNOPSIS_MAX_TOKENS), 0);
+ expect(plan.est_synopsis_usd).not.toBeNull();
+ expect(plan.est_synopsis_usd!).toBeGreaterThan(0);
+ expect(plan.est_synopsis_usd!).toBeCloseTo(perCall, 4);
+ expect(plan.est_synopsis_usd).toBe(reservationCostUsd(SYNOPSIS_MODEL, 'chat', plan.est_input_tokens, plan.est_output_tokens));
+ const price = lookupEmbeddingPrice(EMBED_MODEL);
+ expect(price.kind).toBe('known');
+ expect(plan.est_embedding_usd!).toBeGreaterThan(0);
+ expect(plan.est_usd!).toBeCloseTo(plan.est_synopsis_usd! + plan.est_embedding_usd!, 10);
+ const jobs = await engine.executeRaw<{ n: number | string }>(`SELECT count(*)::int AS n FROM minion_jobs WHERE name = 'contextual_reindex_per_chunk'`);
+ expect(Number(jobs[0]!.n)).toBe(0);
+ expect(chatCalls).toBe(0);
+ });
+
+ test('--yes under a cap below one synopsis call: the shared spend cap refuses the work, nothing is stamped, and the run says so', async () => {
+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis', '--yes', '--max-usd', '0.0001'], { pollMs: 100 }));
+ expect(result.mode).toBe('run');
+ expect(result.run!.cap_usd).toBe(0.0001);
+ expect(result.run!.failed).toBe(2);
+ expect(result.run!.synopsis).toBe(0);
+ expect(chatCalls).toBe(0);
+ const stamped = await engine.executeRaw<{ n: number | string }>(`SELECT count(*)::int AS n FROM pages WHERE contextual_retrieval_mode = 'per_chunk_synopsis'`);
+ expect(Number(stamped[0]!.n)).toBe(0);
+ });
+
+ test('--yes queues and completes one job per page; pages land on the synopsis tier and a rerun has nothing to sweep', async () => {
+ const result = await quiet(() => runReindexSynopsis(engine, ['--synopsis', '--yes', '--max-usd', '5'], { pollMs: 100 }));
+ expect(result.mode).toBe('run');
+ expect(result.run).toMatchObject({ synopsis: 2, title: 0, skipped: 0, failed: 0, already_attempted: 0, cap_usd: 5 });
+ expect(chatCalls).toBe(result.plan!.chunks);
+ const jobs = await engine.executeRaw<{ status: string; group_id: string | null; slug: string }>(
+ `SELECT status, spend_authorization->>'group_id' AS group_id, data->>'page_slug' AS slug
+ FROM minion_jobs WHERE name = 'contextual_reindex_per_chunk' AND id = ANY($1::int[]) ORDER BY id`, [result.run!.job_ids]);
+ expect(jobs.map(j => j.slug)).toEqual(['notes/alpha-example', 'notes/beta-example']);
+ expect(jobs.every(j => j.status === 'completed' && j.group_id === result.run!.group_id)).toBe(true);
+ const modes = await engine.executeRaw<{ slug: string; mode: string | null }>(`SELECT slug, contextual_retrieval_mode AS mode FROM pages ORDER BY slug`);
+ expect(modes).toEqual([
+ { slug: 'notes/alpha-example', mode: 'per_chunk_synopsis' },
+ { slug: 'notes/beta-example', mode: 'per_chunk_synopsis' },
+ { slug: 'notes/large-example', mode: 'title' },
+ ]);
+ expect(result.coverage_after!.coverage_pct).toBe(100);
+ const check = await checkSynopsisTierCoverage(engine);
+ expect(check.status).toBe('ok');
+ expect(check.message).toContain('100% of chunks');
+ expect(check.fix).toBeUndefined();
+
+ chatCalls = 0;
+ const rerun = await quiet(() => runReindexSynopsis(engine, ['--synopsis', '--yes'], { pollMs: 100 }));
+ expect(rerun.mode).toBe('preview');
+ expect(rerun.plan!.pages).toEqual([]);
+ expect(chatCalls).toBe(0);
+ });
+});