Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Purge retires extracted facts, embedding migrations stop refusing, re-synthesis and re-embeds stop repeating (#9395)

Garry Tan4244fd774f8a
diff --git a/behavior-changes.d/resynthesis_hold_6753.md b/behavior-changes.d/resynthesis_hold_6753.md
new file mode 100644
index 0000000000..f236bc478c
--- /dev/null
+++ b/behavior-changes.d/resynthesis_hold_6753.md
@@ -0,0 +1 @@
+gbrain dream no longer re-synthesizes a transcript that keeps growing on every pass. A transcript synthesized within dream.synthesize.cooldown_hours (12 by default) is held until that window passes, and the run reports how many grown transcripts it held (resynthesis_cooldown). A busy transcript is now re-synthesized at most twice a day instead of up to 24 times; the daily spend ceiling is unchanged. An explicit gbrain dream --input or --date run is never held. To re-synthesize sooner, lower dream.synthesize.cooldown_hours or pass --input.
diff --git a/changelog.d/capy-severe-five.md b/changelog.d/capy-severe-five.md
new file mode 100644
index 0000000000..b01c7c93a5
--- /dev/null
+++ b/changelog.d/capy-severe-five.md
@@ -0,0 +1,23 @@
+---
+important: data-loss
+---
+**Purging a page retires the facts extracted from it, `gbrain migrate embeddings` stops refusing on some databases, a growing transcript is no longer re-synthesized on every dream pass, legacy embedding labels no longer wipe valid vectors, and search says how far an upgrade's index rebuild has got.**
+
+<!-- behavior-change: resynthesis_hold_6753 -->
+
+### Fixed
+
+- **Facts extracted from a purged page are retired with it** (#8859, reported by @ocscdsa). Facts the extractor filed on no page and that name the page only in their context stayed active after the page was retired or purged. Retire and purge now match them too. Facts left behind by earlier purges are not changed here.
+- **`gbrain migrate embeddings` no longer always refuses with "Source census changed while acquiring locks"** on databases whose collation sorts source ids differently from code-unit order (#9307, reported by @ozabramov; first fixed by @Masashi-Ono0611 in #9310).
+- **A growing transcript is re-synthesized at most twice a day** (#6753, reported by @andreineacsu). A transcript synthesized within `dream.synthesize.cooldown_hours` is held, so repeats drop from up to 24 a day to 2. The daily spend ceiling is unchanged ($120 worst case). Explicit `--input` and `--date` runs are never held.
+- **Chunks stamped with a bare model label keep their vectors** (#8533, reported by @babyspectrum). The false-stamp guard, the projection preserve check and the stale-signature invalidation now share one rule: a bare label counts when its model name and width match the configured model; a different explicit provider prefix is still invalidated. Before, the last two nulled valid vectors and forced a paid re-embed.
+- **Search says how far an upgrade's index rebuild has got** (#9192, reported by @jonahberg). While the rebuild runs, the partial-results notice gives pages done out of total, the measured rate and the command to watch progress, and the upgrade prints it once.
+
+### Itemized changes
+
+- `src/core/facts/gone-page-facts.ts`, `src/core/facts/subject-infer.ts`, `src/core/persistence/page-purge.ts`: extracted-from-page facts are expired and purged with their page.
+- `src/commands/migrate-embeddings.ts`: the locked census is sorted the same way as the requested source list.
+- `src/core/cycle/synthesize-completion.ts`, `src/core/cycle/synthesize.ts`: per-transcript re-synthesis hold.
+- `src/core/embedding-invalidation.ts`, `src/core/engine-sql/chunks.ts`, `src/core/page-state/projections.ts`: one `chunkModelMatchesSql` rule.
+- `src/core/search/retrieval-coverage.ts`, `src/core/ops/search.ts`, `src/core/interop-notices.ts`, v153's notice handler: rebuild progress.
+- Tests: one new test per fix, on PGLite and Postgres where the bug is engine-specific; the #5051 test changes one assertion and its count.
diff --git a/docs/architecture/canonical-writers.tsv b/docs/architecture/canonical-writers.tsv
index 46a213977c..d12c32f127 100644
--- a/docs/architecture/canonical-writers.tsv
+++ b/docs/architecture/canonical-writers.tsv
@@ -91,7 +91,7 @@ src/core/extract-timeline-from-meetings.ts 2 coordinator #6273: on a managed bra
src/core/facts/backstop.ts 3 engine_guard Canonical table SQL trigger requires coordinator capability; legacy calls fail before canonical DB publication.
src/core/facts/fence-fact-append.ts 1 coordinator prepareFenceFactAppend (G4) inserts one facts row only inside apply(tx), the coordinator's publication transaction, after the file holding its fence row was published and after the page row; index-existing mode first reads the cell back from the published file and refuses fence_row_changed, so no facts row exists ahead of its fence row.
src/core/facts/fence-write.ts 2 coordinator Managed brains publish fence rows through the managed_facts_entity intent before any file or row write; unmanaged writes keep the source filesystem ownership check before editing canonical fact fences.
-src/core/facts/gone-page-facts.ts 2 engine_guard A retention purge expires the purged pages' facts (filed on the page, or about it and held by no page row) inside the transaction that deletes the page rows; both unmanaged purges run in maintenanceTransaction (purge-deleted.ts), and on a managed brain the canonical table SQL trigger refuses the write outside the coordinator. Database only: the page file is already gone, and rows are expired, never deleted.
+src/core/facts/gone-page-facts.ts 3 engine_guard A retention purge expires the purged pages' facts (filed on the page, extracted from it and filed on none, or about it and held by no page row) inside the transaction that deletes the page rows; both unmanaged purges run in maintenanceTransaction (purge-deleted.ts), and on a managed brain the canonical table SQL trigger refuses the write outside the coordinator. Database only: the page file is already gone, and rows are expired, never deleted.
src/core/facts/forget.ts 4 filesystem_guard Managed callers journal withdrawal; legacy filesystem helper and the supersession-only expiry require coordinated write.
src/core/facts/ontology-rekey.ts 2 engine_guard B-10 (#6948 sweep): dropRekeyDuplicates deletes the ontology observations a re-key would duplicate on idx_facts_ontology_stint_dedup and repoints supersession links at the survivor, inside its caller's transaction immediately before the caller's UPDATE (the rename's maintenanceTransaction, the relink_facts publication, the phantom redirect's fact move, repair facts-rehome's move). The canonical table SQL trigger requires coordinator capability, so a legacy call fails before publication.
src/core/facts/proposal-supersede.ts 4 filesystem_guard System One S9 accept/undo (local CLI only): unmanaged brains update the two facts rows and strike or restore the fence row in one transaction under the source filesystem lock and page lock, publishing the file last; managed brains run the same change as the coordinator mutation decide_proposal.
diff --git a/docs/architecture/system-of-record.md b/docs/architecture/system-of-record.md
index c8592810dd..6c887a442a 100644
--- a/docs/architecture/system-of-record.md
+++ b/docs/architecture/system-of-record.md
@@ -260,7 +260,7 @@ or a physical projection that leaves attribution untouched):
- `src/core/facts/fence-fact-append.ts` (1): `prepareFenceFactAppend` inserts the facts row of a fence row the request's page file holds, in its caller's coordinated publication after the page row (remember and its callers).
- `src/core/facts/fence-write.ts` (1): the markdown-first fence reconcile insert runs in `maintenanceTransaction` (its page body mirror goes through `facts/file-body-guard.ts`).
- `src/core/facts/file-body-guard.ts` (2): the #6982 checked body mirror (`replaceStoredBody`: the page_versions row and the body refresh, under the page row lock) runs in its caller's `maintenanceTransaction` (fence write, unfenced-facts fence, phantom redirect, forget, proposal supersede).
-- `src/core/facts/gone-page-facts.ts` (2): a retention purge expires the purged pages' facts in its delete transaction, in `maintenanceTransaction` (the bulk purge and the per-tombstone purge of a source the belief ledger records; `purgeDeletedPagesCoordinated`).
+- `src/core/facts/gone-page-facts.ts` (3): a retention purge expires the purged pages' facts in its delete transaction, in `maintenanceTransaction` (the bulk purge and the per-tombstone purge of a source the belief ledger records; `purgeDeletedPagesCoordinated`).
- `src/core/facts/forget.ts` (4): the legacy fence expiry, withdrawal and strike-through each run in `maintenanceTransaction` (nested inside a caller's coordinated write, they keep its actor).
- `src/core/facts/ontology-rekey.ts` (1): B-10's duplicate-observation drop runs in its caller's transaction immediately before the caller's re-key: the rename's `maintenanceTransaction`, the `relink_facts` publication, the phantom redirect's fact move and `repair facts-rehome`'s move; every caller is attributed.
- `src/core/facts/proposal-supersede.ts` (4): unmanaged `decide` proposal accept and undo run in one `maintenanceTransaction` each; managed brains publish a `decide_proposal` request.
diff --git a/plugin-variants/gbrain-coding/skills/maintain/SKILL.md b/plugin-variants/gbrain-coding/skills/maintain/SKILL.md
index 8c21b0b3a1..9cd64687b1 100644
--- a/plugin-variants/gbrain-coding/skills/maintain/SKILL.md
+++ b/plugin-variants/gbrain-coding/skills/maintain/SKILL.md
@@ -267,9 +267,13 @@ in `dream.synthesize.last_completion_ts` and is written only when every child
completed and nothing was deferred (synthesis budget, triage time budget or
publish pending), never on skipped or failed runs. So under a standing
backlog the cooldown never applies and synthesize runs every cycle until the
-backlog drains (#6753); it is not a cap of ~2 runs a day. To cap spend, use
+backlog drains (#6753); it is not a cap of ~2 runs a day. The same setting
+also holds a growing transcript: one whose earlier version was synthesized
+within `dream.synthesize.cooldown_hours` is skipped as `resynthesis_cooldown`
+until that window passes, so it is re-synthesized at most about twice a day.
+To cap spend, use
`dream.synthesize.max_submissions_per_source_per_day`. Explicit `--input` /
-`--date` / `--from` / `--to` invocations bypass cooldown.
+`--date` / `--from` / `--to` invocations bypass cooldown and the hold.
**`--dry-run` semantics:** runs the scored triage pass (judges + caches
verdicts for new files) but skips the synthesis subagents. NOT zero LLM
diff --git a/plugin/skills/maintain/SKILL.md b/plugin/skills/maintain/SKILL.md
index 8c21b0b3a1..9cd64687b1 100644
--- a/plugin/skills/maintain/SKILL.md
+++ b/plugin/skills/maintain/SKILL.md
@@ -267,9 +267,13 @@ in `dream.synthesize.last_completion_ts` and is written only when every child
completed and nothing was deferred (synthesis budget, triage time budget or
publish pending), never on skipped or failed runs. So under a standing
backlog the cooldown never applies and synthesize runs every cycle until the
-backlog drains (#6753); it is not a cap of ~2 runs a day. To cap spend, use
+backlog drains (#6753); it is not a cap of ~2 runs a day. The same setting
+also holds a growing transcript: one whose earlier version was synthesized
+within `dream.synthesize.cooldown_hours` is skipped as `resynthesis_cooldown`
+until that window passes, so it is re-synthesized at most about twice a day.
+To cap spend, use
`dream.synthesize.max_submissions_per_source_per_day`. Explicit `--input` /
-`--date` / `--from` / `--to` invocations bypass cooldown.
+`--date` / `--from` / `--to` invocations bypass cooldown and the hold.
**`--dry-run` semantics:** runs the scored triage pass (judges + caches
verdicts for new files) but skips the synthesis subagents. NOT zero LLM
diff --git a/skills/maintain/SKILL.md b/skills/maintain/SKILL.md
index e0698e238d..637b1c4da2 100644
--- a/skills/maintain/SKILL.md
+++ b/skills/maintain/SKILL.md
@@ -266,9 +266,13 @@ in `dream.synthesize.last_completion_ts` and is written only when every child
completed and nothing was deferred (synthesis budget, triage time budget or
publish pending), never on skipped or failed runs. So under a standing
backlog the cooldown never applies and synthesize runs every cycle until the
-backlog drains (#6753); it is not a cap of ~2 runs a day. To cap spend, use
+backlog drains (#6753); it is not a cap of ~2 runs a day. The same setting
+also holds a growing transcript: one whose earlier version was synthesized
+within `dream.synthesize.cooldown_hours` is skipped as `resynthesis_cooldown`
+until that window passes, so it is re-synthesized at most about twice a day.
+To cap spend, use
`dream.synthesize.max_submissions_per_source_per_day`. Explicit `--input` /
-`--date` / `--from` / `--to` invocations bypass cooldown.
+`--date` / `--from` / `--to` invocations bypass cooldown and the hold.
**`--dry-run` semantics:** runs the scored triage pass (judges + caches
verdicts for new files) but skips the synthesis subagents. NOT zero LLM
diff --git a/skills/skills.lock.history.json b/skills/skills.lock.history.json
index c21663ecd8..11446137e4 100644
--- a/skills/skills.lock.history.json
+++ b/skills/skills.lock.history.json
@@ -579,6 +579,7 @@
"92111d9203e940a6f068f9b3abe7516e7831de9a7e77d234bb6da6125c3d663a",
"964165253b5f7919ba0807c43c092318faad1539605316c49137c5ac8726fc34",
"987b470e34a6cad2c3c12f9b4a77183fdeee0dcb918f792b7545e6f552fd4aa1",
+ "c23b9a91edb5f11306a31a3b00d5262f9f7250eee266a608f2f1bdafecbdde74",
"cb45aba2ea4075195ed244f28571457cec8cb42f70fe209b570c29528d857b9e",
"de3d9a6ce414470ce4bd406b28d9c8ab80ee90a56d51ff1bc5136cae9fead00d",
"e0bfe8769b2fe19a28c22a36f3e24f5df83201fba0d08050824ff8b913cdf74d",
diff --git a/skills/skills.lock.json b/skills/skills.lock.json
index 264cbc85d4..cc4e0a9759 100644
--- a/skills/skills.lock.json
+++ b/skills/skills.lock.json
@@ -95,7 +95,7 @@
"idea-lineage/SKILL.md": "8cb36179dbf94abc17065613b8e773b1bb2b22670742aa77347b91ac885a05c5",
"idea-lineage/routing-eval.jsonl": "ee2e00704b9accb7dd58bb8f126a3bc04a2c40be499180fa505dbf6d5061cd41",
"ingest/SKILL.md": "1acc220ce6e9e65876fa2ccede6ce11f9c75457feb6695ef904955f8debecaae",
- "maintain/SKILL.md": "5b5d28e3f4e6b2bcd8fa4644c70a157998493a1215fba9828e511ca9314676f6",
+ "maintain/SKILL.md": "c23b9a91edb5f11306a31a3b00d5262f9f7250eee266a608f2f1bdafecbdde74",
"manifest.json": "f9d16966d26fe9b37ef6ce9609eaa29d06142a409f3b531f1289b1a8c1bc929e",
"mcp-access/SKILL.md": "4373e0164981eb7f8cd3794a2eae895d77426cef478c8d9d959de585256d4b50",
"measure-before-you-fix/SKILL.md": "11906da29a1a31976b13d4796b686d9a9981825737846a85f32d88bf431b9b36",
@@ -226,7 +226,7 @@
"skillpack-check/SKILL.md": "0d2d2d201ed72e19b47d7e3743b5bbdfb5c2655d1f34d29447c601fca3908f9e",
"skillpack-harvest/SKILL.md": "0f86314b58d25b5511809f567c04aedc3fba6fe35ed642ac52d8216ae365b6eb",
"skillpack-harvest/routing-eval.jsonl": "cb4783288e95af3132b32ecb40a54cffc095f57b36240f96a25c2d2adf5e68c6",
- "skills.lock.history.json": "9743603c04e940a73e4757197c7d878adb736f0d42a53c0be0e25eb95de6bca9",
+ "skills.lock.history.json": "ec17d5a33fe97b37cc436cabe5284082c1ea12ce93bd485da1eaf7c3b4109f12",
"smoke-test/SKILL.md": "f5099e1d2ab9dc6827096249e68414f7ae357c29b39fb069a54bb7fb74f414a5",
"soul-audit/SKILL.md": "7a48e180849f7502d332878c443397f90a3756f45caa94f76bcfcc7c997469c8",
"storage-manifest/SKILL.md": "1debbefba61007a17cda1525496875d1edfde90642adf3ec031f25a71724807e",
diff --git a/src/commands/migrate-embeddings.ts b/src/commands/migrate-embeddings.ts
index f1be5849bd..500f70449e 100644
--- a/src/commands/migrate-embeddings.ts
+++ b/src/commands/migrate-embeddings.ts
@@ -674,7 +674,7 @@ export async function executeMigrationFlow(
}
const lockedCensus = await engine.executeRaw<{ id: string; incarnation: string; archived: boolean }>('SELECT id,incarnation::text,archived FROM sources ORDER BY id');
- if (JSON.stringify(lockedCensus.map(s => s.id)) !== JSON.stringify(sourceIds)) {
+ if (JSON.stringify(lockedCensus.map(s => s.id).sort()) !== JSON.stringify(sourceIds)) {
return { status: 'apply_failed', reason: 'Source census changed while acquiring locks; no embedding work performed. Retry the scoped preview.' };
}
expectedSources = JSON.stringify(lockedCensus);
diff --git a/src/core/cli-flag-registry.generated.ts b/src/core/cli-flag-registry.generated.ts
index dda84e0bdb..a0c04c75fb 100644
--- a/src/core/cli-flag-registry.generated.ts
+++ b/src/core/cli-flag-registry.generated.ts
@@ -37,7 +37,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'check-backlinks': ['--accept-pending', '--as', '--background', '--brain', '--dir', '--dry-run', '--explain', '--follow', '--help', '--json', '--no-accept-pending', '--progress-interval', '--progress-json', '--quiet', '--room', '--source', '--timeout', '--timings', '--wait'],
'check-resolvable': ['--brain', '--dry-run', '--fix', '--help', '--json', '--skills-dir', '--source', '--strict', '--verbose'],
'check-update': ['--backend', '--brain', '--channel', '--check-only', '--exit-code', '--ff-only', '--force-retry', '--help', '--is-ancestor', '--json', '--markdown', '--migrate-only', '--no-autopilot-install', '--no-bun-floor-check', '--no-restart', '--non-interactive', '--quiet', '--refresh-cache', '--refs', '--remediation-plan', '--restart-only', '--source', '--stop-mcp', '--swap-only', '--tags', '--version', '--yes'],
- 'claw-test': ['--agent', '--apply', '--brain', '--dir', '--force-retry', '--format', '--help', '--json', '--keep-tempdir', '--list-agents', '--live', '--local', '--locks', '--message', '--no-embed', '--no-embedding', '--output-format', '--pglite', '--progress-json', '--run-id', '--scenario', '--skip-verify', '--source', '--statement-timeout', '--version', '--yes'],
+ 'claw-test': ['--agent', '--apply', '--brain', '--dir', '--force-retry', '--format', '--help', '--json', '--keep-tempdir', '--list-agents', '--live', '--local', '--locks', '--message', '--no-embed', '--no-embedding', '--only', '--output-format', '--pglite', '--progress-json', '--run-id', '--scenario', '--skip-verify', '--source', '--statement-timeout', '--version', '--yes'],
'code-callees': ['--all-sources', '--brain', '--force', '--help', '--json', '--lang', '--limit', '--no-embed', '--no-json', '--probe', '--source', '--yes'],
'code-callers': ['--all-sources', '--brain', '--force', '--help', '--json', '--lang', '--limit', '--no-embed', '--no-json', '--probe', '--source', '--yes'],
'code-def': ['--all-sources', '--brain', '--force', '--help', '--json', '--lang', '--limit', '--no-embed', '--no-json', '--probe', '--source', '--yes'],
@@ -49,7 +49,7 @@ export const CLI_FLAG_REGISTRY: Record<string, readonly string[]> = {
'conversation-parser': ['--brain', '--help', '--json', '--source'],
'core': ['--apply', '--brain', '--fail-fast', '--help', '--include-core', '--json', '--priority', '--revision', '--slug', '--source', '--tools-json'],
'creds': ['--allow-create-db', '--allow-docker', '--brain', '--chat-model', '--content-root', '--db-only', '--embedding-dimensions', '--embedding-model', '--expansion-model', '--force', '--git', '--grant-types', '--help', '--http', '--ids', '--issuer-url', '--json', '--key', '--local-postgres', '--mcp-only', '--mcp-url', '--migrate-only', '--model', '--no-embed', '--no-embedding', '--no-git', '--non-interactive', '--oauth-client-id', '--oauth-client-secret', '--out', '--passphrase-env', '--path', '--pglite', '--plan', '--prefer-postgres', '--provenance', '--provider', '--query', '--schema-pack', '--scopes', '--search-mode', '--skip-embed-check', '--source', '--status', '--supabase', '--surface', '--to', '--url', '--version'],
- 'db-repair': ['--accept-pending', '--apply', '--apply-rewrites', '--as', '--background', '--brain', '--break-lock', '--db-url', '--dry-run', '--explain', '--filter', '--follow', '--force', '--force-retry', '--format', '--help', '--http', '--json', '--legacy-lock', '--locks', '--name', '--no-accept-pending', '--path', '--prefer-postgres', '--probe', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--restart', '--room', '--skip-verify', '--source', '--statement-timeout', '--supabase', '--timeout', '--timings', '--undo-last-rewrite', '--url', '--version', '--wait', '--yes'],
+ 'db-repair': ['--accept-pending', '--apply', '--apply-rewrites', '--as', '--background', '--brain', '--break-lock', '--db-url', '--dry-run', '--explain', '--filter', '--follow', '--force', '--force-retry', '--format', '--help', '--http', '--json', '--legacy-lock', '--locks', '--name', '--no-accept-pending', '--only', '--path', '--prefer-postgres', '--probe', '--progress-interval', '--progress-json', '--quiet', '--remediate', '--restart', '--room', '--skip-verify', '--source', '--statement-timeout', '--supabase', '--timeout', '--timings', '--undo-last-rewrite', '--url', '--version', '--wait', '--yes'],
'decide': ['--accept-all', '--all', '--all-from', '--apply', '--brain', '--call-site', '--claims', '--dataset', '--dry-run', '--egress', '--embedding-dimensions', '--embedding-model', '--expect', '--from', '--help', '--include-withdrawals', '--input', '--items-out', '--json', '--kind', '--limit', '--max-cost', '--max-usd', '--min', '--non-interactive', '--out', '--provenance', '--provider', '--query', '--rate', '--reason', '--reason-family', '--recommended', '--remove', '--shadow', '--since', '--slot', '--source', '--status', '--suite', '--target', '--threshold', '--trust', '--undo', '--uri-prefix', '--version', '--what-if-threshold', '--yes'],
'diag': ['--access', '--add', '--all', '--allowed-operations', '--apply', '--as', '--asof', '--body', '--bound-slug-prefixes', '--brain', '--budget-ms', '--card', '--client', '--count', '--cursor', '--date', '--dated-since', '--days', '--detach', '--entities', '--expect', '--expected-revision', '--federated-read', '--filter', '--follow', '--format', '--from-index', '--full', '--fuzzy', '--group-by', '--held', '--help', '--http', '--id', '--include', '--include-ambiguous', '--include-deleted', '--json', '--kind', '--limit', '--max-usd', '--model', '--multimodal', '--no-autopilot-install', '--no-pull', '--no-walk', '--of', '--only', '--page', '--params', '--patch', '--path', '--phase', '--prefer', '--preview', '--privacy-url', '--purge', '--query', '--recent', '--refresh-operations', '--replace', '--restart-only', '--reverse', '--row', '--scopes', '--session', '--side', '--since', '--slug', '--slugs', '--source', '--source-id', '--sources', '--stale', '--state', '--surface', '--terms-url', '--to', '--updated-after', '--url', '--yes'],
'doctor': ['--abandon', '--abbrev-ref', '--abort', '--absolute-git-dir', '--accept-all', '--accept-capabilities', '--accept-pending', '--access', '--access-policy', '--account', '--acknowledge-managed', '--add', '--admin-intent', '--adopt', '--adopt-columns', '--adopt-permissions', '--advertise', '--after', '--agent', '--aggressive', '--alert', '--aliases', '--all', '--all-drifted', '--all-from', '--all-sources', '--allow-old-serve', '--allow-outside-source-root', '--allow-shell-jobs', '--allow-unverified-remote', '--allow-without-owner', '--allowed-operations', '--anchor', '--applied', '--apply', '--apply-rewrites', '--approval-out', '--approved', '--as', '--assign-corpus', '--assume-unchanged', '--attest-actor', '--attest-ops', '--attest-read', '--attest-sources', '--audit', '--auto', '--auto-additive', '--auto-fix', '--autostash', '--background', '--base', '--batch', '--batch-check', '--body', '--bound-brain', '--bound-max-concurrent', '--bound-slug-prefixes', '--bound-source', '--bound-tools', '--brain', '--branch', '--break-lock', '--budget-ms', '--budget-policy', '--budget-tokens', '--budget-usd', '--bundled', '--by-mention', '--cached', '--cacheinfo', '--call-site', '--catalog', '--catch-up', '--channel', '--check', '--check-downgrade', '--check-only', '--cherry-mark', '--claims', '--class', '--claude-plugins', '--clear', '--clear-never-installed', '--cli-socket', '--client', '--close', '--code', '--concurrency', '--concurrent', '--confidence', '--config', '--confirm', '--confirm-destructive', '--confirm-quiesced', '--confirm-relocated-root', '--conflicts', '--contacts-dir', '--content-audit', '--content-root', '--continue', '--conversation', '--count', '--credentials-file', '--cursor', '--cwd', '--dataset', '--day', '--days', '--db-url', '--deadline', '--decisions', '--delegated-namespace', '--delegated-slug-prefixes', '--delete-brain', '--dest', '--detach', '--detail', '--diff-filter', '--dim', '--dimensions', '--dir', '--disable', '--drain', '--drain-timeout', '--drop-field', '--dry-run', '--egress', '--email', '--embedding-dimensions', '--embedding-model', '--enable', '--enable-dcr', '--end-of-options', '--entity', '--env', '--env-file', '--error-unmatch', '--exclude', '--exclude-standard', '--exit-code', '--expand', '--expect', '--expect-local', '--expect-remote', '--expect-upstream', '--expected-epoch', '--expected-revision', '--expected-state', '--explain', '--facts', '--fast', '--federated', '--federated-read', '--fetch-timeout-ms', '--ff-only', '--file', '--filter', '--finalize', '--first-parent', '--fix', '--follow', '--force', '--force-break-lock', '--force-rechunk', '--force-remove', '--force-retry', '--force-schema', '--force-with-lease', '--format', '--fresh', '--fresh-token', '--from', '--from-json', '--from-meetings', '--from-pages', '--from-prefix', '--full', '--full-tree', '--fuzzy', '--gbrain-bin', '--gbrain-skill-resolver', '--generation', '--get', '--get-all', '--get-regexp', '--get-url', '--git-common-dir', '--git-dir', '--git-path', '--grammar', '--group', '--group-by', '--hard', '--hard-deadline', '--harness', '--heads', '--held', '--help', '--history', '--hold-state', '--holds', '--home', '--http', '--id', '--if-version', '--ignore-env-override', '--ignore-missing', '--ignore-unmatch', '--images', '--import', '--include-ambiguous', '--include-covered', '--include-cross-entity', '--include-deleted', '--include-failures', '--include-flagged', '--include-frontmatter', '--include-gitignored', '--include-hidden', '--include-pseudo', '--include-repairs', '--include-withdrawals', '--index-audit', '--index-info', '--infer-dates', '--init', '--input', '--input-type', '--install', '--instructions', '--instructions-only', '--is-ancestor', '--is-bare-repository', '--is-inside-work-tree', '--is-shallow-repository', '--items-out', '--job-isolation', '--jq', '--json', '--keep', '--keep-checkout', '--key', '--kind', '--label-lowerings', '--labelled-only', '--lanes', '--lang', '--left-right', '--legacy-lock', '--limit', '--link', '--link-source', '--link-type', '--list', '--list-out', '--local', '--local-env-vars', '--locks', '--log-noop', '--manifest', '--markdown', '--max-age', '--max-cost', '--max-cost-usd', '--max-count', '--max-facts', '--max-jobs', '--max-requests', '--max-rss', '--max-runtime', '--max-usd', '--max-waiting', '--may-act-for', '--mcp-only', '--member', '--members', '--members-file', '--merge-base', '--migrate-legacy', '--migrate-only', '--migration', '--min', '--min-n', '--mode', '--model', '--multimodal', '--name', '--name-only', '--name-status', '--nice', '--no-accept-pending', '--no-assume-unchanged', '--no-autopilot-install', '--no-autostash', '--no-capture', '--no-commit-id', '--no-confirm', '--no-cron', '--no-delegate', '--no-edit', '--no-embed', '--no-embed-cache', '--no-embedding', '--no-expand', '--no-extensions', '--no-extract', '--no-federated', '--no-filters', '--no-follow-tags', '--no-gpg-sign', '--no-hooks', '--no-index', '--no-instructions', '--no-lanes', '--no-llm', '--no-merges', '--no-migrate', '--no-open-pages', '--no-optional-locks', '--no-probe', '--no-pull', '--no-renames', '--no-repair', '--no-restart', '--no-schema-pack', '--no-seat', '--no-skip-worktree', '--no-stdio-member', '--no-tags', '--no-verify', '--no-walk', '--no-write-fetch-head', '--non-interactive', '--not', '--now', '--of', '--older-than', '--once', '--only', '--operation', '--operations', '--others', '--out', '--output', '--oversized', '--overwrite', '--page', '--parallel', '--params', '--parents', '--parked-code', '--parked-pushes', '--pat-file', '--path', '--path-format', '--pending', '--pglite', '--phase', '--pid-file', '--plan', '--porcelain', '--port', '--prefer', '--prefer-postgres', '--prefix', '--preload', '--preset', '--preview', '--principal', '--priority', '--privacy-url', '--private-pages', '--probe', '--probe-pglite', '--profile', '--progress-interval', '--progress-json', '--project', '--provenance', '--provider', '--providers', '--prune-orphans', '--public-url', '--pull', '--pure', '--purge', '--push', '--push-only', '--push-remote', '--qm', '--query', '--question', '--queue', '--quiet', '--rate', '--raw-only', '--read-sources', '--reader-mode', '--readers', '--reads-unscoped', '--reason', '--reauth', '--rebase', '--rebuild', '--recommended', '--recurse-submodules', '--redact-paths', '--reference-date', '--refresh', '--refresh-operations', '--refresh-skills', '--regenerate', '--reissue', '--relative', '--relower', '--remediate', '--remediation-plan', '--remote', '--remove', '--remove-core', '--repair', '--repair-unknown-markers', '--replace', '--repo', '--request', '--request-id', '--require', '--require-clean', '--reranker', '--reset', '--reset-checkpoint', '--reset-default', '--reset-session', '--resolve', '--restart-only', '--restore-only', '--resume', '--retire', '--retry-failed', '--retry-global', '--retry-held', '--retry-parked', '--reverse', '--review-lower', '--revision', '--right-only', '--rollback-to-source', '--room', '--root', '--routing', '--rules', '--run', '--runtime', '--safe', '--sample', '--save', '--scan-secrets', '--scan-timeout', '--schedule', '--scope', '--scopes', '--search-mode', '--seat', '--section', '--segmented', '--select', '--self-transfer', '--semantic', '--serves', '--session', '--session-id', '--set', '--set-upstream-to', '--shadow', '--shared-skills', '--short', '--show-current', '--show-object-format', '--show-prefix', '--show-token', '--show-toplevel', '--side', '--since', '--since-last-run', '--skill-loading', '--skills-dir', '--skills-index-only', '--skip', '--skip-bare-tweet', '--skip-failed', '--skip-urls', '--skip-verify', '--skip-worktree', '--slot', '--slot-a-model', '--slot-b-model', '--slot-c-model', '--slug', '--slugs', '--slugs-file', '--source', '--source-id', '--source-ids', '--sources', '--squash', '--src-subpath', '--staged', '--stale', '--stat', '--state', '--statement-timeout', '--status', '--stdin', '--stdio-member', '--stop-mcp', '--strategy', '--strict', '--strict-json', '--suite', '--supabase', '--surface', '--swap-only', '--symbol-kind', '--symbolic-full-name', '--synopsis', '--tags', '--take', '--takes-holders', '--target', '--target-score', '--terms-url', '--threshold', '--tier', '--timeout', '--timings', '--to', '--token', '--token-ttl', '--top-k', '--topo-order', '--trust', '--tsconfig-override', '--type', '--types', '--unattended', '--unindexed', '--uninstall', '--unsafe-bypass-dream-guard', '--unschedule', '--unset', '--unset-all', '--unsettle', '--unshallow', '--untrack', '--untracked-files', '--url', '--url-env', '--user', '--vacuum', '--vectors', '--verbose', '--verify', '--version', '--wait', '--wait-drain', '--wakeshift', '--warn-only', '--watch', '--what-if-threshold', '--where', '--window', '--workers', '--working-tree', '--workspace', '--worktree', '--write-tree', '--yes'],
diff --git a/src/core/cycle/synthesize-completion.ts b/src/core/cycle/synthesize-completion.ts
index fbae2437aa..8953675e7c 100644
--- a/src/core/cycle/synthesize-completion.ts
+++ b/src/core/cycle/synthesize-completion.ts
@@ -144,6 +144,42 @@ async function loadSuccessfulSynthesisKeys(
return rows.map(row => row.idempotency_key);
}
+/**
+ * #6753: the synth-v2 completions of the source recorded within the last `hours`, with when each finished: completed
+ * subagent jobs and the durable completion archive. Failed, cancelled and still-running jobs are not completions.
+ */
+export async function recentSynthesisCompletions(engine: BrainEngine, sourceId: string, hours: number): Promise<Array<{ key: string; at: number }>> {
+ if (hours <= 0) return [];
+ const rows = await engine.executeRaw<{ idempotency_key: string; at: string | Date }>(
+ `SELECT idempotency_key, finished_at AS at FROM minion_jobs
+ WHERE name = 'subagent' AND status = 'completed' AND finished_at > now() - ($3 || ' hours')::interval
+ AND COALESCE(NULLIF(data->>'source_id', ''), 'default') = $1 AND idempotency_key LIKE $2
+ UNION ALL
+ SELECT idempotency_key, completed_at FROM dream_synthesis_completions
+ WHERE source_id = $1 AND idempotency_key LIKE $2 AND completed_at > now() - ($3 || ' hours')::interval`,
+ [sourceId, 'dream:synth-v2:%', String(hours)],
+ );
+ return rows.map(row => ({ key: row.idempotency_key, at: new Date(row.at).getTime() }));
+}
+
+/**
+ * #6753: a growing transcript (a live session) has a new content hash on every pass, so its completion key never
+ * matches and it was synthesized again in full each time. When another version of the same file completed in full
+ * (single or a whole chunk set, as findSynthV2Completion counts it) within `hours`, the current version waits: the
+ * ISO time the hold ends, else null. The hold only delays; once it ends the current version synthesizes once, whole.
+ */
+export function resynthesisHoldUntil(recent: Array<{ key: string; at: number }>, filePath: string, hash16: string,
+ sourceId: string, hours: number): string | null {
+ const prefix = `dream:synth-v2:${encodeURIComponent(sourceId)}:filename:${encodeURIComponent(basename(filePath))}:`;
+ const mine = recent.filter(r => r.key.startsWith(prefix));
+ let latest = 0;
+ for (const other of new Set(mine.map(r => r.key.slice(prefix.length, prefix.length + 16)))) {
+ if (other === hash16 || !findSynthV2Completion(mine.map(r => r.key), filePath, other, sourceId)) continue;
+ for (const r of mine) if (r.key.startsWith(prefix + other)) latest = Math.max(latest, r.at);
+ }
+ return latest > 0 ? new Date(latest + hours * 3_600_000).toISOString() : null;
+}
+
/**
* Mirror of findLegacyCompletion for the synth-v2 key family (grammar as
* produced by the submit loop / parsed by `parseSynthV2Key`): `'single'` when
@@ -236,3 +272,22 @@ export async function noNewTranscripts(engine: BrainEngine, transcripts: Discove
return noCandidates(`no new transcripts to synthesize (${transcripts.length} already synthesized)`, { transcripts_discovered: transcripts.length,
synthesis_state: { candidates: 0, already_synthesized: transcripts.length }, children_submitted: 0, skips });
}
+
+/**
+ * #6753: the per-transcript re-synthesis hold for one run, read once: a function giving the `resynthesis_cooldown`
+ * skip reason for a transcript an earlier version of which completed within `hours`, else null. Zero hours (an explicit
+ * --input/--date run, or a cooldown of 0) holds nothing.
+ */
+export async function resynthesisHold(engine: BrainEngine, sourceId: string, hours: number): Promise<(filePath: string, hash16: string, identity: string) => string | null> {
+ const recent = await recentSynthesisCompletions(engine, sourceId, hours);
+ return (filePath, hash16, identity) => {
+ const until = resynthesisHoldUntil(recent, filePath, hash16, identity, hours);
+ return until && `resynthesis_cooldown: an earlier version of this transcript was synthesized within ${hours}h; the current version synthesizes after ${until}`;
+ };
+}
+
+/** #6753: the summary's note of transcripts held by the per-transcript re-synthesis hold, empty when none. */
+export function resynthesisHoldSuffix(skips: Array<{ reason: string }>): string {
+ const held = skips.filter(s => s.reason.startsWith('resynthesis_cooldown')).length;
+ return held ? ` (${held} grown transcript(s) held by resynthesis_cooldown)` : '';
+}
diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts
index 3aa6c051f1..c56bb040dd 100644
--- a/src/core/cycle/synthesize.ts
+++ b/src/core/cycle/synthesize.ts
@@ -80,7 +80,7 @@ import { stampDreamProvenance, summaryProvenanceLines } from './dream-provenance
import { activateSynthesisGenerations } from './synthesize-generations.ts';
import { summaryDerivation, withTranscriptTaint } from './dream-taint.ts';
import { DREAM_SUMMARY_KIND, publishSinglePage } from '../generations/adapters/single-page.ts';
-import { findSynthV2Completion, noCandidates, noNewTranscripts, partitionCompletedSynthesis } from './synthesize-completion.ts';
+import { findSynthV2Completion, noCandidates, noNewTranscripts, partitionCompletedSynthesis, resynthesisHold, resynthesisHoldSuffix } from './synthesize-completion.ts';
// Re-exports: the drain was peeled to inline-drain.ts (dream-wave C7), the
// allow-list loader to filing-rules.ts (#2397); patterns.ts and the
@@ -723,7 +723,7 @@ async function runPhaseSynthesizeInner(
const synthMeter = new BudgetMeter({ budgetUsd: config.budgetUsd, allowUnpriced: config.allowUnpriced, pricingOverrides: await loadPricingOverrides(engine), phase: 'synthesize' });
const childMaxOutputTokens = resolveMaxOutputTokens(undefined, await engine.getConfig('agent.max_output_tokens').catch(() => null), config.model);
- const breaker = await loadDreamBreaker(engine);
+ const [breaker, resynthesisHeld] = await Promise.all([loadDreamBreaker(engine), resynthesisHold(engine, cycleSourceId, explicitTarget ? 0 : config.cooldownHours)]);
// Admission-quota latch: once a submit is rejected, every later transcript
// this run would be rejected too — record one skip per remaining file
// without hammering the queue.
@@ -795,7 +795,7 @@ async function runPhaseSynthesizeInner(
continue;
}
- const refusal = breaker && synthBreakerRefusal(breaker, { synthesisIdentity, sourceId: cycleSourceId, filePath: t.filePath, hash16 });
+ const refusal = resynthesisHeld(t.filePath, hash16, synthesisIdentity) || breaker && synthBreakerRefusal(breaker, { synthesisIdentity, sourceId: cycleSourceId, filePath: t.filePath, hash16 });
if (refusal) { process.stderr.write(`[dream] ${t.basename}: ${refusal}\n`); skipReports.push({ filePath: t.filePath, reason: refusal }); continue; }
const chunks = splitTranscriptByBudget(t.content, t.contentHash, maxCharsPerChunk);
@@ -1037,7 +1037,7 @@ async function runPhaseSynthesizeInner(
// a run that did no synthesis must not suppress the retry window that
// would pick these files up.
if (childIds.length === 0) {
- return ok(`no synthesis submitted (${skipReports.length} passing file(s) skipped)${deferralSuffix}`, {
+ return ok(`no synthesis submitted (${skipReports.length} passing file(s) skipped)${resynthesisHoldSuffix(skipReports)}${deferralSuffix}`, {
transcripts_discovered: transcripts.length,
transcripts_processed: 0,
pages_written: 0,
@@ -1372,7 +1372,7 @@ async function runPhaseSynthesizeInner(
const turnsSamples = childOutcomes.filter(
(o): o is { jobId: number; status: string; turns: number } => typeof o.turns === 'number',
);
- return reportFileHolds(withPostprocessConflicts(postprocessConflicts, withPublishPending(publishPending, ok(`${submittedTranscripts} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s${deferralSuffix}`, {
+ return reportFileHolds(withPostprocessConflicts(postprocessConflicts, withPublishPending(publishPending, ok(`${submittedTranscripts} transcript(s) synthesized in ${(ms / 1000).toFixed(1)}s${resynthesisHoldSuffix(skipReports)}${deferralSuffix}`, {
transcripts_discovered: transcripts.length,
transcripts_processed: submittedTranscripts,
pages_written: writtenSlugs.length,
diff --git a/src/core/embedding-invalidation.ts b/src/core/embedding-invalidation.ts
index c174ef5d38..5094b32687 100644
--- a/src/core/embedding-invalidation.ts
+++ b/src/core/embedding-invalidation.ts
@@ -44,9 +44,20 @@ export function splitEmbeddingSignature(signature: string): { model: string; dim
return { model: signature.slice(0, separator), dims };
}
+/**
+ * #8533: whether a chunk's model label names the target model: the label itself, or its bare tail (`text-embedding-3-large`
+ * for `openai:text-embedding-3-large`), which chunks embedded before provider-prefixed labels carry. Every predicate that
+ * decides whether a stored vector is still in the target space compares labels through this one rule. A bare tail counts
+ * only where `bareWidthOk` (the stored vector's width equals the target dimensions) holds; an explicit different prefix
+ * never matches. `model` is an SQL expression referenced once (a bound parameter in the fragment twin).
+ */
+export function chunkModelMatchesSql(label: string, model: string, bareWidthOk: string): string {
+ return `EXISTS (SELECT 1 FROM (SELECT ${model}::text AS m) target WHERE ${label} IS NOT DISTINCT FROM target.m OR (${label} = substr(target.m, strpos(target.m, ':') + 1) AND ${bareWidthOk}))`;
+}
+
export function currentSpaceChunkPredicate(colId: string, modelParam: number, dimsParam: number, alias = 'cc'): string {
return `COALESCE(${alias}.${colId} IS NOT NULL
- AND ${alias}.model = $${modelParam}
+ AND ${chunkModelMatchesSql(`${alias}.model`, `$${modelParam}`, 'TRUE')}
AND ${alias}.embedded_text_hash = md5(${alias}.chunk_text)
AND vector_dims(${alias}.${colId}) = $${dimsParam}::int, false)`;
}
@@ -96,8 +107,7 @@ export function falseStampPageWhere(colId: string, signatureParam = 1, modelPara
AND EXISTS (
SELECT 1 FROM content_chunks c
WHERE c.page_id = p.id AND c.${colId} IS NOT NULL
- AND c.model IS NOT NULL AND c.model <> $${modelParam}
- AND c.model <> substr($${modelParam}, strpos($${modelParam}, ':') + 1)
+ AND c.model IS NOT NULL AND NOT ${chunkModelMatchesSql('c.model', `$${modelParam}`, `vector_dims(c.${colId})::text = regexp_replace($${signatureParam}::text, '^.*:', '')`)}
)`;
}
diff --git a/src/core/engine-sql/chunks.ts b/src/core/engine-sql/chunks.ts
index eabdb7c8f9..1ce93ebb97 100644
--- a/src/core/engine-sql/chunks.ts
+++ b/src/core/engine-sql/chunks.ts
@@ -35,7 +35,7 @@ import {
} from '../search/embedding-column.ts';
import { chunkDocumentOrder, chunkGenerationFilter, chunkManifestJoin, chunkVisibleFilter, chunkWriteInvalidations, currentTextProjectionFilter, movableChunkSql, requiresSafeChunks, safeChunksFilter } from '../search/safe-chunks.ts';
import { privatePagesFilterForCaller } from '../search/private-visibility.ts';
-import { splitEmbeddingSignature, lockEmbeddingSources } from '../embedding-invalidation.ts';
+import { currentSpaceChunkPredicate, splitEmbeddingSignature, lockEmbeddingSources } from '../embedding-invalidation.ts';
import { EMBED_SKIP_FILTER_FRAGMENT } from '../embed-skip.ts';
import { QUARANTINE_FILTER_FRAGMENT } from '../quarantine.ts';
import { pageBudgetLimitedFilter } from '../page-size-policy.ts';
@@ -571,16 +571,12 @@ export async function setPageEmbeddingSignature(exec: SqlExecutor, slug: string,
}
/**
- * `currentSpaceChunkPredicate` (embedding-invalidation.ts) as a fragment:
- * identical text, values bound in place of its hand-numbered `$model` /
- * `$dims`. `test/engine-sql-chunks.test.ts` pins the two against each other.
+ * `currentSpaceChunkPredicate` (embedding-invalidation.ts) as a fragment: built from its text, with values bound in
+ * place of its `$model` / `$dims` parameters, so the two cannot drift. `test/engine-sql-chunks.test.ts` pins them.
*/
export function currentSpaceChunkFragment(column: string, model: string, dims: number | null): SqlFragment {
- const colId = trustedSql(quoteIdentifier(column));
- return sqlFragment`COALESCE(cc.${colId} IS NOT NULL
- AND cc.model = ${model}
- AND cc.embedded_text_hash = md5(cc.chunk_text)
- AND vector_dims(cc.${colId}) = ${dims}::int, false)`;
+ const strings = currentSpaceChunkPredicate(quoteIdentifier(column), 9001, 9002).split(/\$900[12]/);
+ return sqlFragment(Object.assign([...strings], { raw: [...strings] }), model, dims);
}
export async function invalidateStaleSignatureEmbeddings(
diff --git a/src/core/facts/gone-page-facts.ts b/src/core/facts/gone-page-facts.ts
index 9ab99c82b8..993cef3d66 100644
--- a/src/core/facts/gone-page-facts.ts
+++ b/src/core/facts/gone-page-facts.ts
@@ -6,16 +6,32 @@
* - the facts filed on the page (`source_markdown_slug` is the page). Ontology observations (`dimension` set) are not
* filed there: their `source_markdown_slug` is provenance, so an observation about a live entity sourced from the
* purged page stays;
+ * - the facts extracted from the page and filed on none (#8859: the backstop's database-only rows, which name the page
+ * only as their context head; extractedFromPageSql);
* - the facts about the page (`entity_slug` is the page) that no page row holds. A fact held by another page, live or
* soft-deleted, stays that page's: a restore brings it back, and that page's own purge expires it.
*
* Each case is its own update, so each uses its slug index.
*/
+/** facts.source values written by page-extraction paths, whose context head is the source page slug. */
+export const PAGE_EXTRACTION_SOURCE_LIST = ['mcp:put_page', 'sync:import', 'file_upload', 'code_import'] as const;
+
+/**
+ * #8859: true when the fact `alias` was extracted from the page whose slug is the SQL expression `slug` and is filed on
+ * no page: recordedPageSlug (subject-infer.ts) in SQL. Before #5831 atom pages produced such rows by the thousand.
+ */
+export function extractedFromPageSql(alias: string, slug: string): string {
+ return `(${alias}.source_markdown_slug IS NULL AND ${alias}.source IN (${PAGE_EXTRACTION_SOURCE_LIST.map(s => `'${s}'`).join(', ')})
+ AND (${alias}.context = ${slug} OR starts_with(${alias}.context, ${slug} || ' — ')))`;
+}
+
/** `$1` source ids and `$2` slugs, paired by position. */
const GONE = 'unnest($1::text[], $2::text[]) AS g(source_id, slug)';
const EXPIRE_FILED_SQL = `UPDATE facts f SET expired_at = now(), row_num = NULL FROM ${GONE}
WHERE f.source_id = g.source_id AND f.source_markdown_slug = g.slug AND f.expired_at IS NULL AND f.dimension IS NULL`;
+const EXPIRE_EXTRACTED_SQL = `UPDATE facts f SET expired_at = now(), row_num = NULL FROM ${GONE}
+ WHERE f.source_id = g.source_id AND f.expired_at IS NULL AND ${extractedFromPageSql('f', 'g.slug')}`;
const EXPIRE_ABOUT_SQL = `UPDATE facts f SET expired_at = now(), row_num = NULL FROM ${GONE}
WHERE f.source_id = g.source_id AND f.entity_slug = g.slug AND f.expired_at IS NULL
AND NOT EXISTS (SELECT 1 FROM pages hp WHERE hp.source_id = f.source_id AND hp.slug = f.source_markdown_slug AND hp.slug <> g.slug)`;
@@ -28,5 +44,6 @@ export async function expireGonePageFacts(
if (!pages.length) return;
const params = [pages.map(p => p.source_id), pages.map(p => p.slug)];
await exec.executeRaw(EXPIRE_FILED_SQL, params);
+ await exec.executeRaw(EXPIRE_EXTRACTED_SQL, params);
await exec.executeRaw(EXPIRE_ABOUT_SQL, params);
}
diff --git a/src/core/facts/subject-infer.ts b/src/core/facts/subject-infer.ts
index bd2946e159..c2a27712b3 100644
--- a/src/core/facts/subject-infer.ts
+++ b/src/core/facts/subject-infer.ts
@@ -25,6 +25,7 @@ import type { BrainEngine } from '../engine.ts';
import { extractCandidates, isCommonCapitalizedWord } from '../context/entity-salience.ts';
import { isFactEntityPage, resolveStrictEntityReference, type StrictResolution } from '../entities/resolve.ts';
import { remoteReadFloorSql } from '../search/private-visibility.ts';
+import { PAGE_EXTRACTION_SOURCE_LIST } from './gone-page-facts.ts';
export type InferredVia = 'page' | 'mention';
export type InferMissReason = 'ambiguous' | 'unverified_match' | 'no_mention' | 'no_page';
@@ -56,8 +57,7 @@ export function contextHead(context: string | null | undefined): string | null {
return head || null;
}
-/** facts.source values written by page-extraction paths, whose context head is the source page slug. */
-const PAGE_EXTRACTION_SOURCES = new Set(['mcp:put_page', 'sync:import', 'file_upload', 'code_import']);
+const PAGE_EXTRACTION_SOURCES = new Set<string>(PAGE_EXTRACTION_SOURCE_LIST);
/**
* The page a stored fact was extracted from, when its row records one: a
diff --git a/src/core/interop-notices.ts b/src/core/interop-notices.ts
index 885332a31a..c5714c460f 100644
--- a/src/core/interop-notices.ts
+++ b/src/core/interop-notices.ts
@@ -149,9 +149,12 @@ function projectionReadinessFix(): Action {
function projectionPendingWhy(coverage: RetrievalCoverage, could: number, total: number): string {
const counted = `(${could} of ${total}${coverage.capped ? '+' : ''} pending page(s) could match this query)`;
const images = coverage.could_affect.pending_images ?? 0;
- if (images === 0) return `recently written pages are being indexed and are not in the search index yet ${counted}`;
+ const rebuild = coverage.rebuild;
+ const rate = rebuild?.pages_per_second ? `, at about ${rebuild.pages_per_second < 10 ? rebuild.pages_per_second.toFixed(1) : Math.round(rebuild.pages_per_second)} page(s) per second measured since it started` : '';
+ const progress = rebuild ? `; the search index rebuild an upgrade started is ${rebuild.done} of ${rebuild.total} page(s) done${rate}, and \`gbrain doctor --only text_projection_readiness\` shows its progress` : '';
+ if (images === 0) return `${rebuild ? 'pages wait for that rebuild' : 'recently written pages are being indexed'} and are not in the search index yet ${counted}${progress}`;
const text = could - images;
- return `${images} image page(s) that could match are not in the search index and will not be indexed on their own: sealable images wait for \`gbrain projections drain\` and the rest for a paid re-import (\`gbrain embed --stale --images\`), and doctor's text_projection_readiness check counts them by class${text > 0 ? `; ${text} recently written page(s) are being indexed` : ''} ${counted}`;
+ return `${images} image page(s) that could match are not in the search index and will not be indexed on their own: sealable images wait for \`gbrain projections drain\` and the rest for a paid re-import (\`gbrain embed --stale --images\`), and doctor's text_projection_readiness check counts them by class${text > 0 ? `; ${text} recently written page(s) are being indexed` : ''} ${counted}${progress}`;
}
/** Doctor on the brain host; `run_doctor` where callable (renderAction drops it otherwise). */
@@ -208,7 +211,7 @@ export function degradedRecallNotice(
const byChoice = opts.optedOut || entries.some(s => s.reason === 'embedding_disabled');
const emb = fixKind === 'embeddings' ? (byChoice ? { byChoice: true } as { fix?: Action; byChoice: boolean } : embeddingsFix(opts.config, opts.transport)) : undefined;
const onlyKeywordNeedsDoctor = keywordSkipped && names.every(n => n === 'keyword_arm_failed' || DEGRADED_STAGE_GUIDANCE[n].fix !== 'doctor');
- const onlyImagesNeedDoctor = (opts.coverage?.could_affect?.pending_images ?? 0) > 0 && names.every(n => n === 'projection_pending' || DEGRADED_STAGE_GUIDANCE[n].fix !== 'doctor');
+ const onlyImagesNeedDoctor = ((opts.coverage?.could_affect?.pending_images ?? 0) > 0 || opts.coverage?.rebuild !== undefined) && names.every(n => n === 'projection_pending' || DEGRADED_STAGE_GUIDANCE[n].fix !== 'doctor');
const fix = emb ? emb.fix
: fixKind === 'doctor' ? (onlyKeywordNeedsDoctor ? keywordPlanFix() : onlyImagesNeedDoctor ? projectionReadinessFix() : doctorFix('Doctor names the failing retrieval dependency and its fix.'))
: rerank ? rerankFix(rerank, opts.retry) : undefined;
diff --git a/src/core/ops/search.ts b/src/core/ops/search.ts
index 47d6f885f4..aa6e91b3fb 100644
--- a/src/core/ops/search.ts
+++ b/src/core/ops/search.ts
@@ -50,7 +50,7 @@ import { SAFE_FENCE_CHUNKER_VERSION, chunkManifestReads } from '../search/safe-c
import { expandEngineTypeFilters } from '../schema-pack/query-types.ts';
import { reconcileTypeFilter } from './search-type-filter.ts';
import { probeProjectionReadiness } from '../search/projection-readiness.ts';
-import { countEmbeddingHeld, countPendingProjections, coverageRelevance, type RetrievalCoverage } from '../search/retrieval-coverage.ts';
+import { countEmbeddingHeld, countPendingProjections, countProjectionRebuild, coverageRelevance, type RetrievalCoverage } from '../search/retrieval-coverage.ts';
import { resolveBoostMap, resolveHardExcludes } from '../search/source-boost.ts';
import { pageReadFilter } from '../search/read-policy-sql.ts';
import { QUERY_DESCRIPTION, SEARCH_DESCRIPTION, SEARCH_MATCH_DESCRIPTION } from '../operations-descriptions.ts';
@@ -420,10 +420,11 @@ async function buildRetrievalResponseMeta(
const readinessScope = { ...scope, excludePrivate, types: opts.types, excludeSlugPrefixes };
// #6721: these reads depend on nothing but the inputs above, so they run at once: on a pooled Postgres
// the wall time is the slowest read's round trips, not their sum. The answer-feedback write still runs last.
- const [safeIndexPending, [readiness, pendingCount], entityFacts, matched, held, sourceFreshness, embeddingHeld, filterNotice] = await Promise.all([
+ const [safeIndexPending, [readiness, pendingCount, rebuild], entityFacts, matched, held, sourceFreshness, embeddingHeld, filterNotice] = await Promise.all([
timed('safe_index', ctx.remote !== false && hasUnsealedPagesInScope(ctx, scope, excludePrivate, { types: opts.types, excludeSlugPrefixes })),
- timed('readiness', probeProjectionReadiness(ctx.engine, readinessScope).then(async r =>
- [r, r.status === 'projection_pending' ? await countPendingProjections(ctx.engine, readinessScope, relevance) : null] as const)),
+ timed('readiness', probeProjectionReadiness(ctx.engine, readinessScope).then(async r => r.status === 'projection_pending'
+ ? [r, ...await Promise.all([countPendingProjections(ctx.engine, readinessScope, relevance), countProjectionRebuild(ctx.engine, readinessScope)])] as const
+ : [r, null, null] as const)),
// Facts saved about the entity the query names come first (newest first, one hop through entities they name).
timed('entity_facts', opts.entity && ctx.emitResponseMeta ? entitySavedFacts(ctx.engine, opts.entity, { remote: ctx.remote !== false, excludePrivate, minTrust: scope.minTrust }) : []),
timed('saved_facts', matchingSavedFacts(ctx, scope, queryText, aliases.flatMap(a => [a.name, a.alias]))),
@@ -486,6 +487,7 @@ async function buildRetrievalResponseMeta(
...(embedding ? { embedding_held: embedding.could_affect } : {}) },
...(pending?.capped || embedding?.capped ? { capped: true as const } : {}),
...(topHitExact ? { top_hit_exact: true as const } : {}),
+ ...(pending && rebuild ? { rebuild } : {}),
} : undefined;
const heldNotice = heldFilesNotice(held.couldAffect === 0 ? [] : heldFiles, ctx.remote !== false, held.couldAffect ?? undefined,
embedding?.could_affect ? embedding : undefined);
diff --git a/src/core/page-state/projections.ts b/src/core/page-state/projections.ts
index fe84665774..efebbe28e8 100644
--- a/src/core/page-state/projections.ts
+++ b/src/core/page-state/projections.ts
@@ -26,6 +26,7 @@ import { projectionClassSql, sealableImageSql } from './projection-class.ts';
import { advanceProjectionCursor, ineligibleKey, rememberIneligible, residentWindowCtes, type ProjectionScanCursor } from './projection-window.ts';
import { acceptedEmbeddingInputHashes, embeddingInputHash, plainEmbeddingTier, synopsisBodyHash,
type EmbeddingInputContext, type EmbeddingTier } from '../embedding-input-hash.ts';
+import { chunkModelMatchesSql } from '../embedding-invalidation.ts';
/**
* Complete the searchable snapshot only after its sanitized chunks are installed.
@@ -194,7 +195,7 @@ async function planProjectionKeep(tx: BrainEngine, current: PageSnapshot, contex
const provenance = embeddingInputContext(context, current.page.title, context.corpusGeneration, chunks);
const col = quoteIdentifier(context.column.name);
const recorded = new Map((await tx.executeRaw<{ id: number; embedding_input_hash: string | null; movable: boolean }>(
- `SELECT id,embedding_input_hash,${movableChunkSql(col)} AND model IS NOT DISTINCT FROM $2 AS movable FROM content_chunks WHERE page_id=$1`,
+ `SELECT id,embedding_input_hash,${movableChunkSql(col)} AND ${chunkModelMatchesSql('model', '$2', `vector_dims(${col}) = ${context.column.dimensions}`)} AS movable FROM content_chunks WHERE page_id=$1`,
[current.page.id, context.provenanceModel])).map(row => [Number(row.id), row]));
const currentInput = (chunk: Chunk) => {
const hash = recorded.get(Number(chunk.id))?.embedding_input_hash ?? null;
@@ -253,7 +254,7 @@ async function planDeferredReplacement(tx: BrainEngine, sourceId: string, slug:
await tx.executeRaw('DELETE FROM content_chunks WHERE page_id=$1 AND NOT(id=ANY($2::int[]))', [pageId, plan.kept.map(chunk => Number(chunk.id))]);
// chunk-visibility-ok: the kept rows of a deferred replacement whose vector the current page no longer accepts
const stale = (await tx.executeRaw<{ id: number }>(`SELECT id FROM content_chunks WHERE page_id=$1 AND ${quoteIdentifier(context.column.name)} IS NOT NULL
- AND (model IS DISTINCT FROM $2 OR embedded_text_hash <> md5(chunk_text) OR NOT(id=ANY($3::int[]))) ORDER BY chunk_index`,
+ AND (NOT ${chunkModelMatchesSql('model', '$2', `vector_dims(${quoteIdentifier(context.column.name)}) = ${context.column.dimensions}`)} OR embedded_text_hash <> md5(chunk_text) OR NOT(id=ANY($3::int[]))) ORDER BY chunk_index`,
[pageId, plan.provenanceModel, plan.current])).map(row => Number(row.id));
const moved = new Set(plan.moves.map(move => move.id));
const occupied = new Set([...plan.kept.filter(row => !moved.has(Number(row.id))).map(row => row.chunk_index), ...plan.moves.map(move => move.to)]);
@@ -358,7 +359,7 @@ export async function installPageProjection(engine: BrainEngine, prepared: Proje
await tx.executeRaw(`UPDATE content_chunks SET ${col}=NULL,
embedded_at=NULL,embedded_text_hash=NULL,embedding_input_hash=NULL,
embedding_pending_since=COALESCE(embedding_pending_since,now()) WHERE page_id=$1 AND
- (model IS DISTINCT FROM $2 OR embedded_text_hash <> md5(chunk_text) OR NOT(id=ANY($3::int[])))`,
+ (NOT ${chunkModelMatchesSql('model', '$2', `vector_dims(${col}) = ${column.dimensions}`)} OR embedded_text_hash <> md5(chunk_text) OR NOT(id=ANY($3::int[])))`,
[snapshot.page.id, provenanceModel, currentIds]);
} else if (opts.seal) await tx.deleteChunks(slug, { sourceId });
await tx.upsertChunks(slug, chunks, { sourceId, expectedRevision: snapshot.revision, embeddingColumn: context.column });
diff --git a/src/core/persistence/page-purge.ts b/src/core/persistence/page-purge.ts
index d0b6a1b367..4cf2e7d0fd 100644
--- a/src/core/persistence/page-purge.ts
+++ b/src/core/persistence/page-purge.ts
@@ -51,6 +51,7 @@ import { carryingNeedles, findRequestsForSlugs, JOURNAL_SWEEP_MATCH_CAP, recheck
import { dropPurgedFenceRows } from '../facts/purge-overlay.ts';
import { isOutcomeMarker, outcomeMarkerSql } from '../facts/audit-sources.ts';
import { derivedRowsFrom, hideDerivedRows } from '../facts/derivation-inputs.ts';
+import { extractedFromPageSql } from '../facts/gone-page-facts.ts';
import { contentHash } from '../utils.ts';
import { isSegmentedPage } from '../page-body/page-shape.ts';
import { rewrittenContentHash } from '../page-body/segment-rewrite.ts';
@@ -86,9 +87,12 @@ const purgeContext = new Map<string, PagePurgeContext>();
const PURGE_CONTEXT_LIMIT = 64;
const purgeReceipts = new Map<string, { receipt: Record<string, unknown>; purge: Record<string, unknown> }>();
+/** #8859: the page's facts, $1 its source and $2 its slug: those filed on it and those extracted from it and filed on none. */
+const PAGE_FACTS = `(source_markdown_slug=$2 OR ${extractedFromPageSql('facts', '$2')})`;
+
/** The page's claims (facts filed on it, its takes) that the journal sweep searches for, bounded, trimmed, deduplicated. */
async function readPurgeClaims(engine: Pick<BrainEngine, 'executeRaw'>, sourceId: string, slug: string, pageId: number) {
- const factRows = await engine.executeRaw<{ id: number; fact: string; source: string | null }>('SELECT id,fact,source FROM facts WHERE source_id=$1 AND source_markdown_slug=$2 ORDER BY id LIMIT 1000', [sourceId, slug]);
+ const factRows = await engine.executeRaw<{ id: number; fact: string; source: string | null }>(`SELECT id,fact,source FROM facts WHERE source_id=$1 AND ${PAGE_FACTS} ORDER BY id LIMIT 1000`, [sourceId, slug]);
const takeRows = await engine.executeRaw<{ id: number; claim: string }>('SELECT id,claim FROM takes WHERE page_id=$1 ORDER BY id LIMIT 1000', [pageId]);
const needles = [...new Set([...factRows.filter(f => !isOutcomeMarker(f)).map(f => f.fact), ...takeRows.map(t => t.claim)].map(t => t.trim()).filter(t => t.length >= 8))].slice(0, 32);
return { factRows, takeRows, needles };
@@ -96,7 +100,7 @@ async function readPurgeClaims(engine: Pick<BrainEngine, 'executeRaw'>, sourceId
/** Every claim of the page (its facts, withdrawn ones included, and its takes), trimmed and deduplicated, unbounded: the segment-text sweep's needles. */
async function readSegmentClaims(engine: Pick<BrainEngine, 'executeRaw'>, sourceId: string, slug: string, pageId: number): Promise<string[]> {
- const rows = await engine.executeRaw<{ c: string }>(`SELECT fact AS c FROM facts WHERE source_id=$1 AND source_markdown_slug=$2 AND NOT ${outcomeMarkerSql('facts')} UNION SELECT claim FROM takes WHERE page_id=$3`,
+ const rows = await engine.executeRaw<{ c: string }>(`SELECT fact AS c FROM facts WHERE source_id=$1 AND ${PAGE_FACTS} AND NOT ${outcomeMarkerSql('facts')} UNION SELECT claim FROM takes WHERE page_id=$3`,
[sourceId, slug, pageId]);
return [...new Set(rows.map(r => r.c.trim()).filter(t => t.length >= 8))];
}
@@ -241,7 +245,7 @@ async function tombstonePage(tx: BrainEngine, sourceId: string, slug: string, pa
const page = snapshot.page;
await tx.executeRaw(`INSERT INTO fact_purges(source_id,visibility,subject,fact_hash,request_id,actor,reason)
SELECT DISTINCT source_id,visibility,COALESCE(entity_slug,'*'),gbrain_fact_fingerprint(fact),$3::uuid,$4,'page purge'
- FROM facts WHERE source_id=$1 AND source_markdown_slug=$2 AND NOT ${outcomeMarkerSql('facts')} ON CONFLICT DO NOTHING`, [sourceId, slug, requestId, actor]);
+ FROM facts WHERE source_id=$1 AND ${PAGE_FACTS} AND NOT ${outcomeMarkerSql('facts')} ON CONFLICT DO NOTHING`, [sourceId, slug, requestId, actor]);
await tx.executeRaw(`INSERT INTO take_purges(source_id,subject,claim_hash,request_id)
SELECT DISTINCT $1,$2,gbrain_fact_fingerprint(claim),$4::uuid FROM takes WHERE page_id=$3 ON CONFLICT DO NOTHING`, [sourceId, slug, pageId, requestId]);
// Tombstone the content hash of the page and of every retained version, each as an import of that body would
@@ -297,7 +301,7 @@ export async function purgePageInTransaction(tx: BrainEngine, row: WriteRequest,
const lifetimes = ledgerPresent ? await pageLifetimesOf(tx, sourceId, slug, page.id) : [];
const inventory = ledgerPresent ? await lookup('ledger_redaction', () => pagePurgeClaims(tx, { sourceId, slug, pageId: page.id, lifetimes })) : { claims: [], fps: [] };
const mirrorFacts = retention ? [] : await tx.executeRaw<{ fact_hash: string; visibility: string; subject: string }>(`SELECT DISTINCT gbrain_fact_fingerprint(fact) AS fact_hash,
- visibility, COALESCE(entity_slug,'*') AS subject FROM facts WHERE source_id=$1 AND source_markdown_slug=$2 AND NOT ${outcomeMarkerSql('facts')}`, [sourceId, slug]);
+ visibility, COALESCE(entity_slug,'*') AS subject FROM facts WHERE source_id=$1 AND ${PAGE_FACTS} AND NOT ${outcomeMarkerSql('facts')}`, [sourceId, slug]);
const mirrorTakes = retention ? [] : (await tx.executeRaw<{ claim_hash: string }>('SELECT DISTINCT gbrain_fact_fingerprint(claim) AS claim_hash FROM takes WHERE page_id=$1', [page.id])).map(t => t.claim_hash);
const hashes = new Set<string>();
if (!retention) await tombstonePage(tx, sourceId, slug, page.id, snapshot, row.id, actor, { hash: storedHash, generation: Number(stored?.body_generation) }, hashes);
@@ -332,7 +336,7 @@ export async function purgePageInTransaction(tx: BrainEngine, row: WriteRequest,
const cleared = await clearRebuildableCache(tx, 'query_cache');
if (cleared === null) missingCaches.push('query_cache');
removed.query_cache = cleared ?? 0;
- removed.facts = await count('DELETE FROM facts WHERE source_id=$1 AND source_markdown_slug=$2 RETURNING 1', [sourceId, slug]);
+ removed.facts = await count(`DELETE FROM facts WHERE source_id=$1 AND ${PAGE_FACTS} RETURNING 1`, [sourceId, slug]);
Object.assign(removed, await scrubPurgedSlugStores(tx, sourceId, [slug]));
// #6511 [R2]: the preparation sweep's targets, plus the tail past its ceiling and the rows pending at scan time, re-tested here.
const recheck = await lookup('tail_recheck', () => recheckRequestsCarrying(tx, sourceId, needles, prepared.sweep,
diff --git a/src/core/schema-migrations/v153-verified-text-projection-activation.ts b/src/core/schema-migrations/v153-verified-text-projection-activation.ts
index edf1ffee8f..6f2eaf150e 100644
--- a/src/core/schema-migrations/v153-verified-text-projection-activation.ts
+++ b/src/core/schema-migrations/v153-verified-text-projection-activation.ts
@@ -1,4 +1,13 @@
import type { Migration } from './types.ts';
import { PAGE_PROJECTION_ACTIVATION_SQL, PAGE_PROJECTION_SCHEMA_SQL } from '../page-state/projection-schema.ts';
+import { migrationNotice } from './helpers.ts';
-export const v153: Migration = { version: 153, name: 'verified_text_projection_activation', idempotent: true, sql: PAGE_PROJECTION_SCHEMA_SQL + PAGE_PROJECTION_ACTIVATION_SQL };
+export const v153: Migration = {
+ version: 153, name: 'verified_text_projection_activation', idempotent: true, sql: PAGE_PROJECTION_SCHEMA_SQL + PAGE_PROJECTION_ACTIVATION_SQL,
+ // #9192: the activation takes every page out of the search index until the drain rebuilds it, so say so once.
+ handler: async engine => {
+ const [row] = await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM page_projection_jobs WHERE reason = 'protocol_activation'");
+ const n = Number(row?.n ?? 0);
+ if (n > 0) migrationNotice(` [153] ${n} page(s) queued for a search index rebuild; search and query results stay partial and say so until it finishes. Progress: gbrain doctor --only text_projection_readiness\n`);
+ },
+};
diff --git a/src/core/search/retrieval-coverage.ts b/src/core/search/retrieval-coverage.ts
index 2e7988a22e..eaf912f088 100644
--- a/src/core/search/retrieval-coverage.ts
+++ b/src/core/search/retrieval-coverage.ts
@@ -55,8 +55,12 @@ export interface RetrievalCoverage {
capped?: true;
/** The top result is the page the query named (exact slug, title or alias) and its projection is current: pending pages are not counted as able to affect this answer. */
top_hit_exact?: true;
+ /** #9192: while the upgrade's index rebuild (v153 activation) drains, pages in scope done of total and the rate measured since it started. */
+ rebuild?: ProjectionRebuildProgress;
}
+export interface ProjectionRebuildProgress { done: number; total: number; pages_per_second: number | null }
+
export interface ScopedCount {
total: number; could_affect: number; capped: boolean;
/** With an exact top hit: whether that page is itself counted. */
@@ -150,3 +154,27 @@ export function countEmbeddingHeld(engine: Pick<BrainEngine, 'executeRaw'>, scop
return countInScope(engine, scope, relevance, 'JOIN page_embedding_budget b ON b.page_id = p.id',
`${budgetRowAppliesFilter('b', 'p')} AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
}
+
+/**
+ * #9192: progress of the index rebuild an upgrade's activation queued (rows still queued with reason
+ * `protocol_activation`), over the pages in scope: how many are done of the total, and the rate measured from the
+ * start of the rebuild. Null when no such rebuild is running or the count fails.
+ */
+export async function countProjectionRebuild(engine: Pick<BrainEngine, 'executeRaw'>, scope: ProjectionReadinessScope): Promise<ProjectionRebuildProgress | null> {
+ const params: unknown[] = [];
+ const filters = coverageScopeFilters(scope, params);
+ try {
+ const [row] = await engine.executeRaw<{ seconds: number | null; total: number; done: number }>(
+ `WITH started AS (SELECT EXTRACT(EPOCH FROM now() - min(updated_at))::float8 AS seconds FROM page_projection_jobs WHERE reason = 'protocol_activation')
+ SELECT started.seconds, count(p.id)::int AS total,
+ count(p.id) FILTER (WHERE p.text_projection_revision IS NOT DISTINCT FROM p.knowledge_revision)::int AS done
+ FROM started LEFT JOIN (pages p JOIN sources s ON s.id = p.source_id) ON started.seconds IS NOT NULL AND p.deleted_at IS NULL
+ AND ${projectionClassSql('p', 's')} NOT IN ('protected','archived') AND ${filters.join(' AND ')}
+ GROUP BY started.seconds`, params);
+ if (row?.seconds === null || row?.seconds === undefined) return null;
+ const done = Number(row.done), total = Number(row.total), seconds = Number(row.seconds);
+ return { done, total, pages_per_second: seconds > 0 && done > 0 ? done / seconds : null };
+ } catch {
+ return null;
+ }
+}
diff --git a/test/cycle-synthesize-resynthesis-hold-6753.test.ts b/test/cycle-synthesize-resynthesis-hold-6753.test.ts
new file mode 100644
index 0000000000..7328879f54
--- /dev/null
+++ b/test/cycle-synthesize-resynthesis-hold-6753.test.ts
@@ -0,0 +1,232 @@
+/**
+ * #6753: a growing transcript (a live session) has a new content hash on every pass, so it was synthesized again in
+ * full each time, and under a standing backlog the phase cooldown is never stamped, so nothing bounded it. Now a
+ * transcript whose earlier version completed within dream.synthesize.cooldown_hours waits (`resynthesis_cooldown` in
+ * skips and in the summary) and synthesizes once, whole, when the hold ends. A failed or partial earlier attempt
+ * starts no hold.
+ */
+import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
+import { mkdtempSync, rmSync, writeFileSync } from 'fs';
+import { tmpdir } from 'os';
+import { join } from 'path';
+import { createHash } from 'node:crypto';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { withEnv } from './helpers/with-env.ts';
+import { runPhaseSynthesize, TRIAGE_VERSION } from '../src/core/cycle/synthesize.ts';
+import { TIER_DEFAULTS } from '../src/core/model-config.ts';
+
+// Canonical shared-engine block (check-test-isolation R3/R4).
+let engine: PGLiteEngine;
+let schemaVersion: string;
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({ engine: 'pglite' } as never);
+ await engine.initSchema();
+ // resetPgliteState truncates `config`, wiping the `version` row that
+ // MinionQueue.ensureSchema checks. Capture it so beforeEach can restore.
+ schemaVersion = (await engine.getConfig('version')) ?? '7';
+}, 60_000);
+
+afterAll(async () => {
+ await engine.disconnect();
+});
+
+beforeEach(async () => {
+ await resetPgliteState(engine);
+ await engine.setConfig('version', schemaVersion);
+});
+
+interface Rig {
+ engine: PGLiteEngine;
+ brainDir: string;
+ corpusDir: string;
+ cleanup: () => Promise<void>;
+}
+
+/** Per-test dirs + dream config on the SHARED engine (reset by beforeEach). */
+async function setupRig(): Promise<Rig> {
+ const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-cap-brain-'));
+ const corpusDir = mkdtempSync(join(tmpdir(), 'gbrain-cap-corpus-'));
+ await engine.setConfig('dream.synthesize.enabled', 'true');
+ await engine.setConfig('dream.synthesize.session_corpus_dir', corpusDir);
+ return {
+ engine,
+ brainDir,
+ corpusDir,
+ cleanup: async () => {
+ try { rmSync(brainDir, { recursive: true, force: true }); } catch { /* */ }
+ try { rmSync(corpusDir, { recursive: true, force: true }); } catch { /* */ }
+ },
+ };
+}
+
+async function withSubagentAutoCancel<T>(
+ engine: PGLiteEngine,
+ body: () => Promise<T>,
+ opts: { excludeQueue?: string } = {},
+): Promise<T> {
+ let stopped = false;
+ const loop = (async () => {
+ while (!stopped) {
+ await new Promise(r => setTimeout(r, 50));
+ try {
+ // excludeQueue: deliberately-seeded fixture rows must be handled by
+ // the code under test, not this poller (a poller cancel changes the
+ // row's coalescibility and races the assertion).
+ await engine.executeRaw(
+ `UPDATE minion_jobs
+ SET status = 'cancelled', finished_at = now()
+ WHERE name = 'subagent' AND status IN ('waiting', 'active')
+ AND ($1::text IS NULL OR queue <> $1)`,
+ [opts.excludeQueue ?? null],
+ );
+ } catch { /* race against shutdown is fine */ }
+ }
+ })();
+ try {
+ return await body();
+ } finally {
+ stopped = true;
+ await loop;
+ }
+}
+
+/** Write a small transcript + seed a passing triage-v1 verdict for it. */
+async function seedPassingFile(rig: Rig, name: string): Promise<string> {
+ const content = `conversation in ${name}\n`.repeat(200);
+ const filePath = join(rig.corpusDir, name);
+ writeFileSync(filePath, content);
+ const hash = createHash('sha256').update(content, 'utf8').digest('hex');
+ await rig.engine.putDreamVerdict(filePath, hash, {
+ worth_processing: true,
+ reasons: ['seed'],
+ score: 0.9,
+ content_type: null,
+ segments: [],
+ entities: [],
+ model: TIER_DEFAULTS.utility,
+ triage_version: TRIAGE_VERSION,
+ });
+ return filePath;
+}
+
+/** Seed a recent (or old) synth-v2 submission row for the cap counter. */
+async function seedSubmissionRow(rig: Rig, opts: { ageHours?: number; status?: string; sourceId?: string; tag: string }): Promise<void> {
+ await rig.engine.executeRaw(
+ `INSERT INTO minion_jobs (submission_authority, name, queue, status, data, idempotency_key, created_at)
+ VALUES ('{"version":1,"kind":"application"}'::jsonb, 'subagent', 'dream-inline-old-run', $1,
+ jsonb_build_object('source_id', $2::text),
+ $3,
+ now() - ($4 || ' hours')::interval)`,
+ [opts.status ?? 'waiting', opts.sourceId ?? 'default',
+ `dream:synth-v2:default:filename:${opts.tag}:0123456789abcdef`,
+ String(opts.ageHours ?? 1)],
+ );
+}
+
+interface CapDetails {
+ children_submitted: number;
+ skips: Array<{ filePath: string; reason: string }>;
+}
+
+async function runPhase(
+ rig: Rig,
+ opts: { date?: string; excludeQueue?: string; now?: () => Date } = {},
+): Promise<CapDetails & { summary: string }> {
+ // No key + isolated GBRAIN_HOME: every seeded verdict is a cache hit, so no
+ // judge call happens; the env isolation is belt-and-suspenders against a
+ // dev machine whose config file carries a real key.
+ const { excludeQueue, ...phaseOpts } = opts;
+ const tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-cap-isol-'));
+ try {
+ const result = await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: tmpHome }, () =>
+ withSubagentAutoCancel(rig.engine, () =>
+ runPhaseSynthesize(rig.engine, { brainDir: rig.brainDir, dryRun: false, ...phaseOpts }),
+ { excludeQueue }));
+ // CDX-4 (#4217 family): in this keyless harness every submitted child
+ // dies, and an all-children-dead run is now an honest phase failure
+ // (fan-out details preserved). Zero-submission runs still report 'ok'.
+ // This suite's subject is the CAP accounting in details, not the phase
+ // verdict.
+ if (result.status !== 'ok') {
+ expect(result.error?.code).toBe('SYNTH_ALL_CHILDREN_DEAD');
+ }
+ return { ...(result.details as unknown as CapDetails), summary: String(result.summary ?? '') };
+ } finally {
+ try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* */ }
+ }
+}
+
+
+const NAME = '2026-08-01-live-session.txt';
+const EARLIER = 'conversation turn\n'.repeat(200);
+const earlierHash = createHash('sha256').update(EARLIER, 'utf8').digest('hex').slice(0, 16);
+
+async function earlierAttempt(rig: Rig, opts: { status: string; finishedHoursAgo: number; suffix?: string }) {
+ await rig.engine.executeRaw(`INSERT INTO minion_jobs (submission_authority, name, queue, status, data, idempotency_key, created_at, finished_at)
+ VALUES ('{"version":1,"kind":"application"}'::jsonb, 'subagent', 'dream-inline-old-run', $1, jsonb_build_object('source_id','default'), $2,
+ now() - (($3::numeric + 0.1) || ' hours')::interval, now() - ($3 || ' hours')::interval)`,
+ [opts.status, `dream:synth-v2:default:filename:${NAME}:${earlierHash}${opts.suffix ?? ''}`, String(opts.finishedHoursAgo)]);
+}
+const held = (d: CapDetails) => d.skips.filter(s => s.reason.startsWith('resynthesis_cooldown'));
+
+describe('#6753 per-transcript re-synthesis hold', () => {
+ test('a grown transcript whose earlier version completed minutes ago waits, and the summary says so', async () => {
+ const rig = await setupRig();
+ try {
+ await earlierAttempt(rig, { status: 'completed', finishedHoursAgo: 0.1 });
+ await seedPassingFile(rig, NAME);
+ const details = await runPhase(rig);
+ expect(details.children_submitted).toBe(0);
+ expect(held(details)).toHaveLength(1);
+ expect(details.summary).toContain('held by resynthesis_cooldown');
+ } finally { await rig.cleanup(); }
+ }, 60_000);
+
+ test('once the hold ends the current version is synthesized', async () => {
+ const rig = await setupRig();
+ try {
+ await earlierAttempt(rig, { status: 'completed', finishedHoursAgo: 13 });
+ await seedPassingFile(rig, NAME);
+ const details = await runPhase(rig);
+ expect(held(details)).toHaveLength(0);
+ expect(details.children_submitted).toBe(1);
+ } finally { await rig.cleanup(); }
+ }, 60_000);
+
+ test('a failed earlier attempt starts no hold', async () => {
+ const rig = await setupRig();
+ try {
+ await earlierAttempt(rig, { status: 'failed', finishedHoursAgo: 0.1 });
+ await seedPassingFile(rig, NAME);
+ const details = await runPhase(rig);
+ expect(held(details)).toHaveLength(0);
+ expect(details.children_submitted).toBe(1);
+ } finally { await rig.cleanup(); }
+ }, 60_000);
+
+ test('a partial chunk set starts no hold', async () => {
+ const rig = await setupRig();
+ try {
+ await earlierAttempt(rig, { status: 'completed', finishedHoursAgo: 0.1, suffix: ':c0of2' });
+ await seedPassingFile(rig, NAME);
+ const details = await runPhase(rig);
+ expect(held(details)).toHaveLength(0);
+ expect(details.children_submitted).toBe(1);
+ } finally { await rig.cleanup(); }
+ }, 60_000);
+
+ test('a cooldown of 0 turns the hold off', async () => {
+ const rig = await setupRig();
+ try {
+ await rig.engine.setConfig('dream.synthesize.cooldown_hours', '0');
+ await earlierAttempt(rig, { status: 'completed', finishedHoursAgo: 0.1 });
+ await seedPassingFile(rig, NAME);
+ const details = await runPhase(rig);
+ expect(held(details)).toHaveLength(0);
+ expect(details.children_submitted).toBe(1);
+ } finally { await rig.cleanup(); }
+ }, 60_000);
+});
diff --git a/test/e2e/migrate-embeddings-census-collation-9307.test.ts b/test/e2e/migrate-embeddings-census-collation-9307.test.ts
new file mode 100644
index 0000000000..49cbb8bb4e
--- /dev/null
+++ b/test/e2e/migrate-embeddings-census-collation-9307.test.ts
@@ -0,0 +1,47 @@
+/**
+ * #9307: `gbrain migrate embeddings` built its lock set from a JavaScript-sorted source list and compared it with
+ * `SELECT ... FROM sources ORDER BY id`, which Postgres orders by the database collation. Under a linguistic
+ * collation (en_US.utf8, the pgvector image's default) hyphens are ignored at the first level, so ids such as
+ * `auto-memory` and `automax-notes` come back in the other order and every apply refused with "Source census
+ * changed while acquiring locks". The census is now compared as a set, so the migration runs.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import type { BrainEngine } from '../../src/core/engine.ts';
+import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../../src/core/ai/gateway.ts';
+import { runSchemaTransition } from '../../src/core/embedding-migration.ts';
+import { executeMigrationFlow, planMigrationFlow } from '../../src/commands/migrate-embeddings.ts';
+import { migrationWaveFixture } from '../helpers/migration-wave-fixture.ts';
+import { withEnv } from '../helpers/with-env.ts';
+
+const model = 'openai:text-embedding-3-small';
+for (const kind of ['pglite', 'postgres'] as const) {
+ const suite = kind === 'postgres' && !process.env.DATABASE_URL ? describe.skip : describe;
+ suite(`migrate embeddings source census under the database collation (${kind})`, () => {
+ let fixture: Awaited<ReturnType<typeof migrationWaveFixture>>;
+ let engine: BrainEngine;
+ let home: string;
+ beforeAll(async () => {
+ fixture = await migrationWaveFixture(kind); engine = fixture.engine;
+ home = mkdtempSync(join(tmpdir(), 'migration-census-collation-')); mkdirSync(join(home, '.gbrain'));
+ await runSchemaTransition(engine, 8);
+ await engine.setConfig('embedding_model', model); await engine.setConfig('embedding_dimensions', '8');
+ writeFileSync(join(home, '.gbrain', 'config.json'), JSON.stringify({ engine: kind, embedding_model: model, embedding_dimensions: 8, openai_api_key: 'synthetic-only' }));
+ }, 60_000);
+ afterAll(async () => { __setEmbedTransportForTests(null); resetGateway(); await fixture?.close(); rmSync(home, { recursive: true, force: true }); });
+
+ test('ids a linguistic collation orders differently from code-unit order do not refuse the apply', async () => {
+ for (const id of ['auto-memory', 'automax-notes']) await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT DO NOTHING', [id]);
+ configureGateway({ embedding_model: model, embedding_dimensions: 16, env: { OPENAI_API_KEY: 'synthetic-only' } });
+ __setEmbedTransportForTests(async ({ values }) => ({ values, warnings: [], embeddings: values.map(() => Array(16).fill(0.25)), usage: { tokens: values.length * 8 } }));
+ const result = await withEnv({ GBRAIN_HOME: home, GBRAIN_EMBEDDING_MODEL: undefined, GBRAIN_EMBEDDING_DIMENSIONS: undefined }, async () => {
+ const opts = { to: model, dim: 16, reranker: 'off', maxCostUsd: 1, quiet: true };
+ return executeMigrationFlow(engine, await planMigrationFlow(engine, opts), opts);
+ });
+ expect(JSON.stringify(result)).not.toContain('Source census changed');
+ expect(result.status).toBe('completed');
+ }, 120_000);
+ });
+}
diff --git a/test/embedding-bare-model-label-8533.test.ts b/test/embedding-bare-model-label-8533.test.ts
new file mode 100644
index 0000000000..f7601249ab
--- /dev/null
+++ b/test/embedding-bare-model-label-8533.test.ts
@@ -0,0 +1,80 @@
+/**
+ * #8533: chunks embedded before provider-prefixed model labels store the bare model name (`text-embedding-3-large`)
+ * while the brain's model is prefixed (`openai:text-embedding-3-large`). The false-stamp check already treated that
+ * label as the same model, but a projection rebuild that preserves embeddings and the stale-signature invalidation
+ * compared labels exactly, so they nulled vectors whose text hash and width were correct and queued a paid re-embed.
+ * Every one of those predicates now compares labels through one rule (chunkModelMatchesSql): a bare tail counts only
+ * when the stored vector's width equals the target dimensions, and an explicit different prefix never counts.
+ */
+import { describe, expect, test } from 'bun:test';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { rebuildPendingPageProjections } from '../src/core/page-state/projections.ts';
+import { countFalseStampedChunks, invalidateStaleSignatureEmbeddingsGuarded } from '../src/core/embedding-invalidation.ts';
+import { installFixtureChunks } from './helpers/page-projection.ts';
+import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const MODEL = 'openai:text-embedding-3-large';
+const TEXT = 'A synthetic page body about example widgets.';
+
+async function brainWith(label: string, run: (engine: BrainEngine, pageId: number, dims: number) => Promise<void>, databaseUrl?: string, model = MODEL) {
+ const { engine, close } = await isolatedSharedSkillsEngine(databaseUrl);
+ try {
+ await engine.setConfig('embedding_model', model);
+ const [{ dims }] = await engine.executeRaw<{ dims: number }>("SELECT atttypmod AS dims FROM pg_attribute WHERE attrelid='content_chunks'::regclass AND attname='embedding'");
+ await engine.putPage('notes/bare-label-example', { type: 'note', title: 'Bare label example', compiled_truth: TEXT });
+ await installFixtureChunks(engine, 'notes/bare-label-example', [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT }]);
+ const [page] = await engine.executeRaw<{ id: number }>("SELECT id FROM pages WHERE slug='notes/bare-label-example'");
+ await engine.executeRaw(`UPDATE content_chunks SET embedding=array_fill(0.1::real, ARRAY[${Number(dims)}])::vector, model=$2, embedded_at=now(),
+ embedded_text_hash=md5(chunk_text) WHERE page_id=$1`, [page!.id, label]);
+ await run(engine, Number(page!.id), Number(dims));
+ } finally { await close(); }
+}
+const vectors = async (engine: BrainEngine, pageId: number) => (await engine.executeRaw<{ kept: boolean }>(
+ 'SELECT embedding IS NOT NULL AS kept FROM content_chunks WHERE page_id=$1 ORDER BY chunk_index', [pageId])).map(r => r.kept);
+
+for (const backend of testBackends()) {
+ const url = backend === 'postgres' ? process.env.DATABASE_URL : undefined;
+ describe(`#8533 a bare model label keeps its vector (${backend})`, () => {
+ for (const label of [MODEL, 'text-embedding-3-large']) {
+ test(`projection rebuild preserving embeddings keeps a chunk labelled ${label}`, () => brainWith(label, async (engine, pageId) => {
+ await engine.executeRaw('UPDATE pages SET text_projection_revision=NULL WHERE id=$1', [pageId]);
+ await engine.executeRaw(`INSERT INTO page_projection_jobs(source_incarnation,slug,revision,reason) SELECT s.incarnation,p.slug,p.knowledge_revision,'protocol_activation'
+ FROM pages p JOIN sources s ON s.id=p.source_id WHERE p.id=$1 ON CONFLICT DO NOTHING`, [pageId]);
+ expect((await rebuildPendingPageProjections(engine, 20)).rebuilt).toBeGreaterThan(0);
+ expect(await vectors(engine, pageId)).toEqual([true]);
+ }, url), 120_000);
+
+ test(`stale-signature invalidation keeps a chunk labelled ${label}`, () => brainWith(label, async (engine, pageId, dims) => {
+ await engine.executeRaw('UPDATE pages SET embedding_signature=NULL WHERE id=$1', [pageId]);
+ await invalidateStaleSignatureEmbeddingsGuarded(engine, { signature: `${MODEL}:${dims}`, includeNullSignature: true });
+ expect(await vectors(engine, pageId)).toEqual([true]);
+ }, url), 120_000);
+ }
+
+ test('a bare label whose vector width differs from the target dimensions is still invalidated', () => brainWith('text-embedding-3-large', async (engine, pageId, dims) => {
+ await engine.executeRaw('UPDATE pages SET embedding_signature=$2 WHERE id=$1', [pageId, `${MODEL}:${dims + 1}`]);
+ expect((await countFalseStampedChunks(engine, MODEL, dims + 1)).chunks).toBe(1);
+ await engine.executeRaw('UPDATE pages SET embedding_signature=NULL WHERE id=$1', [pageId]);
+ await invalidateStaleSignatureEmbeddingsGuarded(engine, { signature: `${MODEL}:${dims + 1}`, includeNullSignature: true });
+ expect(await vectors(engine, pageId)).toEqual([false]);
+ }, url), 120_000);
+
+ const OLLAMA = 'ollama:example-model:latest';
+ test('an explicit different provider prefix is still invalidated by the stale-signature check', () => brainWith('openai:example-model:latest', async (engine, pageId, dims) => {
+ await engine.executeRaw('UPDATE pages SET embedding_signature=$2 WHERE id=$1', [pageId, `${OLLAMA}:${dims}`]);
+ expect((await countFalseStampedChunks(engine, OLLAMA, dims)).chunks).toBe(1);
+ await engine.executeRaw('UPDATE pages SET embedding_signature=NULL WHERE id=$1', [pageId]);
+ await invalidateStaleSignatureEmbeddingsGuarded(engine, { signature: `${OLLAMA}:${dims}`, includeNullSignature: true });
+ expect(await vectors(engine, pageId)).toEqual([false]);
+ }, url, OLLAMA), 120_000);
+
+ test('an explicit different provider prefix is still invalidated by a projection rebuild', () => brainWith('openai:example-model:latest', async (engine, pageId) => {
+ await engine.executeRaw('UPDATE pages SET text_projection_revision=NULL WHERE id=$1', [pageId]);
+ await engine.executeRaw(`INSERT INTO page_projection_jobs(source_incarnation,slug,revision,reason) SELECT s.incarnation,p.slug,p.knowledge_revision,'protocol_activation'
+ FROM pages p JOIN sources s ON s.id=p.source_id WHERE p.id=$1 ON CONFLICT DO NOTHING`, [pageId]);
+ expect((await rebuildPendingPageProjections(engine, 20)).rebuilt).toBeGreaterThan(0);
+ expect(await vectors(engine, pageId)).toEqual([false]);
+ }, url, OLLAMA), 120_000);
+ });
+}
diff --git a/test/embedding-signature-stale.test.ts b/test/embedding-signature-stale.test.ts
index 5e3413bb39..7b7bddd817 100644
--- a/test/embedding-signature-stale.test.ts
+++ b/test/embedding-signature-stale.test.ts
@@ -89,11 +89,11 @@ describe('embedding_signature stale semantics', () => {
const invalidated = guarded
? await invalidateStaleSignatureEmbeddingsGuarded(engine, opts)
: await engine.invalidateStaleSignatureEmbeddings(opts);
- expect(invalidated).toBe(4);
+ expect(invalidated).toBe(3);
const rows = await engine.executeRaw<{ chunk_index: number }>(
`SELECT chunk_index FROM content_chunks WHERE embedding IS NOT NULL ORDER BY chunk_index`,
);
- expect(rows.map((row) => row.chunk_index)).toEqual([0]);
+ expect(rows.map((row) => row.chunk_index)).toEqual([0, 2]);
});
}
}
diff --git a/test/fixtures/goldens/migrations/records.json b/test/fixtures/goldens/migrations/records.json
index a7a4fb18f6..8290e64c5e 100644
--- a/test/fixtures/goldens/migrations/records.json
+++ b/test/fixtures/goldens/migrations/records.json
@@ -2316,7 +2316,14 @@
"version": 152
},
{
- "handler": "absent",
+ "handler": {
+ "form": "arrow",
+ "helpers": [
+ "migrationNotice",
+ "quietMigrationNotices"
+ ],
+ "sha256": "4fd47425063a5a78da6ded62f86e56cb1d0587604a99289e4dce3d1834105df6"
+ },
"idempotent": true,
"name": "verified_text_projection_activation",
"sql": {
diff --git a/test/fixtures/goldens/sql-text/chunks.json b/test/fixtures/goldens/sql-text/chunks.json
index c5e357bb93..637110d3c9 100644
--- a/test/fixtures/goldens/sql-text/chunks.json
+++ b/test/fixtures/goldens/sql-text/chunks.json
@@ -486,8 +486,8 @@
"number",
"string"
],
- "sql": "UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedding_pending_since = now() FROM pages p WHERE cc.page_id = p.id AND p.deleted_at IS NULL AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND cc.\"embedding\" IS NOT NULL AND NOT COALESCE(cc.\"embedding\" IS NOT NULL AND cc.model = $2 AND cc.embedded_text_hash = md5(cc.chunk_text) AND vector_dims(cc.\"embedding\") = $3::int, false) AND p.embedding_signature IS NOT NULL AND p.embedding_signature <> $4 RETURNING cc.page_id",
- "textSha256": "3ce3ae1c80591bc67dd42aa8fd96381113d78112b8d9639f94c66c2fd990defb"
+ "sql": "UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedding_pending_since = now() FROM pages p WHERE cc.page_id = p.id AND p.deleted_at IS NULL AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND cc.\"embedding\" IS NOT NULL AND NOT COALESCE(cc.\"embedding\" IS NOT NULL AND EXISTS (SELECT 1 FROM (SELECT $2::text AS m) target WHERE cc.model IS NOT DISTINCT FROM target.m OR (cc.model = substr(target.m, strpos(target.m, ':') + 1) AND TRUE)) AND cc.embedded_text_hash = md5(cc.chunk_text) AND vector_dims(cc.\"embedding\") = $3::int, false) AND p.embedding_signature IS NOT NULL AND p.embedding_signature <> $4 RETURNING cc.page_id",
+ "textSha256": "a3c62939bd323a1c8cdf3839be27f1991e375ce8d5254ae9dfac8d7027252e0e"
},
{
"event": "commit",
@@ -523,8 +523,8 @@
"string",
"string"
],
- "sql": "UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedding_pending_since = now() FROM pages p WHERE cc.page_id = p.id AND p.deleted_at IS NULL AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND cc.\"embedding\" IS NOT NULL AND NOT COALESCE(cc.\"embedding\" IS NOT NULL AND cc.model = $2 AND cc.embedded_text_hash = md5(cc.chunk_text) AND vector_dims(cc.\"embedding\") = $3::int, false) AND (p.embedding_signature IS NULL OR p.embedding_signature <> $4) AND p.source_id = $5 RETURNING cc.page_id",
- "textSha256": "1e9cf1d1f6783e8c9bed7fe1cb42cdc67435878649e9e0527182294f3002aeaa"
+ "sql": "UPDATE content_chunks cc SET \"embedding\" = NULL, embedded_at = NULL, embedding_pending_since = now() FROM pages p WHERE cc.page_id = p.id AND p.deleted_at IS NULL AND p.source_id=ANY($1::text[]) AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived) AND cc.\"embedding\" IS NOT NULL AND NOT COALESCE(cc.\"embedding\" IS NOT NULL AND EXISTS (SELECT 1 FROM (SELECT $2::text AS m) target WHERE cc.model IS NOT DISTINCT FROM target.m OR (cc.model = substr(target.m, strpos(target.m, ':') + 1) AND TRUE)) AND cc.embedded_text_hash = md5(cc.chunk_text) AND vector_dims(cc.\"embedding\") = $3::int, false) AND (p.embedding_signature IS NULL OR p.embedding_signature <> $4) AND p.source_id = $5 RETURNING cc.page_id",
+ "textSha256": "c2e93d531619e7de430580fd2e365183fed163d7ba8e92f5af6fae373a864cf0"
},
{
"event": "commit",
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index 2a47a65941..c28b10da95 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -108,6 +108,7 @@ test/edge-whitespace-twins.test.ts
test/effect-commit-moved-head.test.ts
test/embed-once-per-content-version.serial.test.ts
test/embed-stale-images-managed-6529.test.ts
+test/embedding-bare-model-label-8533.test.ts
test/embedding-completion-atomic.serial.test.ts
test/enforced-member-writes-setting.serial.test.ts
test/enforced-member-writes.serial.test.ts
@@ -542,8 +543,10 @@ test/physical-root-birth-tolerance-6501.test.ts
test/physical-root-copied-brain-fix-7671.test.ts
test/physical-root-moved-8625.test.ts
test/privacy-label-ignored.test.ts
+test/projection-rebuild-progress-9192.test.ts
test/provenance-drift-7891.test.ts
test/provenance-stamps-7873.test.ts
+test/purge-context-facts-8859.test.ts
test/purge-expires-page-facts.test.ts
test/purge-fact.test.ts
test/purge-global-marker.test.ts
diff --git a/test/projection-rebuild-progress-9192.test.ts b/test/projection-rebuild-progress-9192.test.ts
new file mode 100644
index 0000000000..a16add213f
--- /dev/null
+++ b/test/projection-rebuild-progress-9192.test.ts
@@ -0,0 +1,59 @@
+/**
+ * #9192: v153's activation takes every page out of the search index until the projection drain rebuilds it. While that
+ * rebuild runs, search and query say how far it is (pages done of total, the rate measured since it started) and which
+ * command shows progress, and the upgrade says so once when the activation runs.
+ */
+import { describe, expect, spyOn, test } from 'bun:test';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PAGE_PROJECTION_ACTIVATION_SQL } from '../src/core/page-state/projection-schema.ts';
+import { rebuildPendingPageProjections } from '../src/core/page-state/projections.ts';
+import { v153 } from '../src/core/schema-migrations/v153-verified-text-projection-activation.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { installFixtureChunks } from './helpers/page-projection.ts';
+import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const TEXT = 'The quokka harmonica appears in this synthetic note.';
+
+async function activated(run: (engine: BrainEngine) => Promise<void>, databaseUrl?: string) {
+ const { engine, close } = await isolatedSharedSkillsEngine(databaseUrl);
+ try {
+ for (const slug of ['notes/quokka-one', 'notes/quokka-two']) {
+ await engine.putPage(slug, { type: 'note', title: 'Quokka harmonica', compiled_truth: TEXT });
+ await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT }]);
+ }
+ for (const statement of PAGE_PROJECTION_ACTIVATION_SQL.split(';').map(s => s.trim()).filter(Boolean)) await engine.executeRaw(statement);
+ await run(engine);
+ } finally { await close(); }
+}
+const answer = async (engine: BrainEngine, tool: string) => JSON.stringify(await dispatchToolCall(engine, tool, { query: 'harmonica synthetic' }, { remote: false }));
+
+for (const backend of testBackends()) {
+ const url = backend === 'postgres' ? process.env.DATABASE_URL : undefined;
+ describe(`#9192 the index rebuild after activation is reported (${backend})`, () => {
+ test('search and query name pages done of total, the measured rate and the progress command', () => activated(async engine => {
+ await new Promise(resolve => setTimeout(resolve, 20));
+ expect((await rebuildPendingPageProjections(engine, 1)).rebuilt).toBe(1);
+ for (const tool of ['search', 'query']) {
+ const text = await answer(engine, tool);
+ expect(text).toContain('the search index rebuild an upgrade started is 1 of 2 page(s) done, at about ');
+ expect(text).toContain('page(s) per second measured since it started');
+ expect(text).toContain('gbrain doctor --only text_projection_readiness');
+ }
+ }, url), 120_000);
+
+ test('the rebuild clause ends with the rebuild', () => activated(async engine => {
+ while ((await rebuildPendingPageProjections(engine, 20)).rebuilt > 0) { /* drain */ }
+ expect(await answer(engine, 'search')).not.toContain('the search index rebuild an upgrade started');
+ }, url), 120_000);
+
+ test('the v153 handler prints the queued count and the progress command once', () => activated(async engine => {
+ const writes: string[] = [];
+ const spy = spyOn(process.stderr, 'write').mockImplementation(((chunk: string) => { writes.push(String(chunk)); return true; }) as typeof process.stderr.write);
+ try { await v153.handler!(engine); } finally { spy.mockRestore(); }
+ expect(writes.filter(w => w.includes('[153]'))).toEqual([
+ ' [153] 2 page(s) queued for a search index rebuild; search and query results stay partial and say so until it finishes. Progress: gbrain doctor --only text_projection_readiness\n',
+ ]);
+ }, url), 120_000);
+ });
+}
diff --git a/test/purge-context-facts-8859.test.ts b/test/purge-context-facts-8859.test.ts
new file mode 100644
index 0000000000..fd531ca63e
--- /dev/null
+++ b/test/purge-context-facts-8859.test.ts
@@ -0,0 +1,73 @@
+/**
+ * #8859: facts the backstop extracted from a page and filed on none (no `source_markdown_slug`, the page slug as the
+ * head of `context`; atom pages wrote thousands before #5831) outlived that page's purge, by both purge paths (a direct
+ * `delete_page purge: true` and the retention expiry of a soft-deleted page), and stayed active and recallable. Both
+ * paths now take them; a fact whose context only names the page as free text (not a page-extraction source) stays.
+ */
+import { describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import type { BrainEngine } from '../src/core/engine.ts';
+import type { OperationContext } from '../src/core/ops/contract.ts';
+import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
+import { purgeDeletedPagesCoordinated } from '../src/core/persistence/purge-deleted.ts';
+import { managedBrain } from './helpers/managed-brain.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const ATOM = 'atoms/2026-09-30/example-claim';
+const MEETING = 'meetings/example-meeting';
+
+async function put(ctx: OperationContext, slug: string, type: string) {
+ await submitPageMutation(ctx, { operation: 'put_page', params: { slug, source_id: 'default', request_id: randomUUID(),
+ content: `---\ntype: ${type}\ntitle: Example\n---\nA synthetic page.\n` } });
+}
+async function pageId(engine: BrainEngine, slug: string) {
+ const [row] = await engine.executeRaw<{ id: number }>("SELECT id FROM pages WHERE source_id='default' AND slug=$1", [slug]);
+ return Number(row!.id);
+}
+/** Rows as an earlier version left them: written under replica role, so no current writer guard applies. */
+const legacyRow = <T>(engine: BrainEngine, sql: string, params: unknown[]) => engine.transaction(async tx => {
+ await tx.executeRaw('SET LOCAL session_replication_role = replica');
+ return tx.executeRaw<T>(sql, params);
+});
+async function fact(engine: BrainEngine, text: string, entity: string | null, context: string, opts: { input?: number; source?: string } = {}) {
+ const [row] = await legacyRow<{ id: number }>(engine, `INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability, source, context, valid_from)
+ VALUES ('default', $1, $2, 'fact', 'private', 'high', $4, $3, now()) RETURNING id`, [entity, text, context, opts.source ?? 'mcp:put_page']);
+ if (opts.input) await legacyRow(engine, "INSERT INTO derivation_inputs (derived_table, derived_id, input_table, input_id, source_id) VALUES ('facts',$1,'pages',$2,'default')", [String(row!.id), String(opts.input)]);
+ return Number(row!.id);
+}
+const active = async (engine: BrainEngine, ids: number[]) => (await engine.executeRaw<{ id: number }>(
+ 'SELECT id FROM facts WHERE id = ANY($1::bigint[]) AND expired_at IS NULL ORDER BY id', [ids])).map(r => Number(r.id));
+
+async function scenario(engine: BrainEngine, ctx: OperationContext, mode: 'direct' | 'expiry') {
+ for (const [s, t] of [[ATOM, 'note'], [MEETING, 'meeting'], ['people/alice-example', 'person'], ['notes/kept-example', 'note']] as const) await put(ctx, s, t);
+ const meetingId = await pageId(engine, MEETING);
+ const legacy = await fact(engine, 'Example claim one from the atom', null, ATOM);
+ const noted = await fact(engine, 'Example claim two from the atom', null, `${ATOM} — entity inferred from text`);
+ const derived = await fact(engine, 'Alice Example joined a new team', 'people/alice-example', MEETING, { input: meetingId });
+ const kept = await fact(engine, 'Kept example claim', null, 'notes/kept-example');
+ const remembered = await fact(engine, 'A remembered example claim', null, ATOM, { source: 'mcp:remember' });
+ for (const slug of [ATOM, MEETING]) {
+ const live = await engine.readPageSnapshot(slug, { sourceId: 'default' });
+ await submitPageMutation(ctx, { operation: 'delete_page', params: { slug, source_id: 'default', request_id: randomUUID(), expected_revision: live!.revision } });
+ if (mode === 'direct') {
+ const snap = await engine.readPageSnapshot(slug, { sourceId: 'default', includeDeleted: true });
+ await submitPageMutation(ctx, { operation: 'delete_page', params: { slug, source_id: 'default', purge: true, request_id: randomUUID(), expected_revision: snap!.revision } });
+ }
+ }
+ if (mode === 'expiry') await purgeDeletedPagesCoordinated(engine, 0);
+ expect(await engine.readPageSnapshot(ATOM, { sourceId: 'default', includeDeleted: true })).toBeNull();
+ expect(await engine.readPageSnapshot(MEETING, { sourceId: 'default', includeDeleted: true })).toBeNull();
+ return { activeFacts: await active(engine, [legacy, noted, derived, kept, remembered]), ids: { legacy, noted, derived, kept, remembered } };
+}
+
+for (const backend of testBackends()) {
+ describe(`#8859 purge takes context-linked facts (${backend})`, () => {
+ const opts = backend === 'postgres' ? { databaseUrl: process.env.DATABASE_URL } : {};
+ for (const mode of ['direct', 'expiry'] as const) {
+ test(`${mode} purge`, () => managedBrain(async ({ engine, ctx }) => {
+ const r = await scenario(engine, ctx, mode);
+ expect(r.activeFacts).toEqual([r.ids.kept, r.ids.remembered]);
+ }, opts), 180_000);
+ }
+ });
+}