Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(#6961): reuse proven vectors on fenced pages instead of re-embedding the whole page

garrytan11e13abfa45e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 440a30bacd..19c53ca464 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,16 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**A write to a page with a facts or takes fence no longer re-embeds the whole page: `remember`, timeline and fence writes now embed only the chunks whose text changed (#6961).**
+
+Vector reuse was turned off for every page with a protected fence, so each write to an entity page deleted and re-embedded all of its chunks. One reporter measured 13 to 15 million embedding tokens a day from that alone. Entity pages carry the fences and take the most writes.
+
+### Itemized changes
+
+- **Fenced page reuse (#6961).** A fenced page now reuses a stored vector when its recorded embedding input equals the one the import would embed, the same check every other page gets. The stored index must still be sealed by the fence-safe chunker, so a reused vector's text holds no private fence row. A chunk whose text changed, such as the fence chunk a new fact lands in, is embedded again. A legacy vector with no recorded input is never reused on a fenced page, because nothing proves what text it was computed from. A one-row fact append on a five-chunk entity page now makes one embedding call instead of five. Contributed by @javieraldape (#7055, rewritten on master).
+
## [0.60.538.0] - 2026-11-01
**Agents connected over MCP now see the chronicle reads, takes reads and a quoted page's title and frontmatter quoted as source material, and the context gbrain's hooks inject quotes facts below "your notes" too.**
diff --git a/docs/architecture/key-files/files-and-sync-1.md b/docs/architecture/key-files/files-and-sync-1.md
index 0918898eb9..249a0ff4f5 100644
--- a/docs/architecture/key-files/files-and-sync-1.md
+++ b/docs/architecture/key-files/files-and-sync-1.md
@@ -31,7 +31,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/import-screen.ts` — `screenImportContent`: the one content screen (published exemption, size, frontmatter hold, slug conflict, Tier 1: a fixable fence is admitted normalized, a residual refuses typed (`coordinated`) or warns (`lenient`); optional sanity reject; `holdNoncanonicalPrivate`: `visibility_noncanonical`) shared by `importFromContent`, managed import and managed sync publication. `contentRefusalError` keeps each site's wire `error`; `isContentRefusal(code, message)` matches new codes and exact pre-upgrade strings, never capacity or transient refusals; `contentRefusalFromReceipt` restores code, reason, key and line on receipt replies. `PAGE_OVER_LIMIT_SUGGESTION` is the one next step of a page write over the page limit (put_page receipts and `page-prepare.ts`): it names the limit and says only content over it becomes several pages. Tests: `test/import-screen.test.ts`, `test/sync-frontmatter-recovery.test.ts`, `test/put-page-recoverable-frontmatter.test.ts`. #6259 marker strip: [quarantine cluster](quarantine.md).
-- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). Every markdown import resolves and stamps `contextual_retrieval_mode` and `corpus_generation` with the canonical write, `--no-embed` included (#5621), so later embed passes reproduce the page's wrapping; a failed source-policy read fails the import unless the row or table is missing (`SourceRowNotFoundError`, `isUndefinedTableError`). Unchanged markdown or code content whose page is below the safe-chunk fence is re-sealed projection-only (direct imports install it; coordinated `prepare` imports stay no-ops and queue a `safe_chunk_reseal` projection job), with no page write or admission. Freshly embedded chunks carry `embedding_input_hash` for the tier they were built under. The `beforeCommit` hook runs inside the guarded canonical transaction so a rejected file write rolls back its projections; optional legacy source-path repair uses a savepoint. Deferred importer embedding callbacks capture the committed text projection and indexing context, then install with an UPDATE-only revision/chunk/hash check. Tests: `test/import-publication-guards.test.ts`. Coordinated `prepare` imports resolve and atomically stamp the page/source/global contextual wrapping mode even with `noEmbed:true`; the durable outbox later embeds that revision. The existing paid-synopsis title fallback and mount trust gate remain in force, while embedding signatures only certify completed vectors. `test/persistence-contextual-embedding.test.ts` covers both engines, policy precedence, deferred provider execution, and a competing title replacement. `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent` reconciles tags by provenance (`tags.tag_source`, migration v175): frontmatter tags go through `addTag(..., { tagSource: 'frontmatter' })`, and a row still marked 'frontmatter' whose tag left the frontmatter is deleted in the import transaction. Every other `addTag` stamps 'added' (add_tag, enrichment, the code importer) and is never import-deleted, so `gbrain reindex --markdown` keeps enrichment tags; legacy NULL rows are adopted only while the frontmatter still lists them; prepared (managed) imports keep add-only tags, matching the union their canonical file renders. Pinned by `test/reindex-preserve-tags.test.ts`, `test/import-frontmatter-tag-removal.test.ts` + `test/import-file.test.ts`. identity decisions live in `src/core/import-identity.ts` (`decideImportIdentity`, `collidingSlugOwner`; pure reads, the importer applies them). The dedup pre-check calls `engine.findDuplicatePage?.(sourceId, {hash, frontmatterId, excludeSlug})` (optional `?` so test doubles compile; the caller's own slug never matches, and a `frontmatter.id` match ranks ahead of a bare hash match). Posture: MOVE when the `frontmatter.id` matches and the other page's OWN origin proves it moved (#5675): that origin is the `file://` path `importFromFile` records in `pages.source_uri` (never replacing non-file provenance), else the source's configured root (`local_path`, or the default source's legacy `sync.repo_path`). An origin that still exists moves only when it is this very file under another relative path (overlapping roots converge); a gone origin needs this file under that same root, and differing content also needs the same file name or title, so a missing path under another root, or a new note reusing a deleted note's template id, never renames a live page. `updateSlug` carries the row (page id, links, facts incl. `entity_slug`/`source_markdown_slug`, `page_aliases`, timeline, plus a rename alias) and the new content imports onto it, so a full sync's delete-reconcile has nothing live to remove; SKIP only when the id AND the content (`content_hash`, or title and body for a row written outside the importer) both match while the recorded file still exists (overlapping ingest roots); IMPORT with a warning when the id matches but content differs (templates, copy-paste; an edit is never frozen by the other page); WARN-ALWAYS on a bare content_hash collision; FAIL CLOSED on lookup error; bypass via `--force-rechunk`. Content-only callers (`put_page`, transcripts) never move a page. Soft-deleted pages excluded at the engine layer so tombstones don't block legitimate re-imports under new slugs. Slug collisions: when another live file (`existing.source_path`, realpath-distinct) already owns the slug, the incoming file is skipped with `skip_reason: 'slug_collision'` and a stderr warning, unless the incoming file is named exactly `<slug>.md` (the file the link extractor reads for that slug), so two spellings never overwrite each other. An embedding failure never blocks the text write: chunks commit with NULL vectors, no embedding signature is stamped, the result carries `embedding_deferred: true`, and the stale sweep embeds them later (a code file whose embed failed is not stamped either). An inline markdown re-import embeds only chunks whose embedding input changed: `planEmbeddingReuse` (keyed on chunk source + exact text) reuses a stored vector when the old index is sealed (`requireSafeChunks`), the page's `embedding_signature`, contextual mode, `corpus_generation` and (under the title wrapper) sanitized title are unchanged, and neither the old nor the new body holds protected fences; `--force-rechunk` and post-commit-embedding imports reuse nothing (`test/import-markdown-embedding-reuse.test.ts`); identical rows stay put (`import-chunk-reuse.ts`). Offset-less frontmatter datetimes are read in the brain's `brain.timezone` (IANA, validated by `gbrain config set`; unset or invalid = UTC; the YAML parser tags naive timestamps with `NAIVE_DATETIME` so the wall-clock reading survives), while date-only values stay UTC calendar dates (`test/effective-date-brain-timezone.test.ts`). `effective_date` for an undated page falls back to `fallbackCreatedAt` (stored fallback date, else the earliest of the file's birth/modification time and — when `sync.git_first_commit_dates` is on — its git first-commit date via `fileTimes`, else now; `src/core/git-first-commit.ts` maps paths in one `git log --diff-filter=A` pass per `runImport`, skipping shallow clones; `test/effective-date-git-first-commit.test.ts`), never the latest write. Pinned by `test/import-dedup-frontmatter-id.test.ts`, `test/import-identity-move.test.ts`, `test/import-slug-collision.test.ts`, `test/import-embed-outage.test.ts` and `test/effective-date-calendar.test.ts`. `importFromContent` is the narrow waist every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It runs a three-tier content-quality disposition via `assessContentSanity` from `src/core/content-sanity.ts` BEFORE chunking: (1) high-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE (stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search) OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`; (2) fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker (page stays fully searchable, marker rides search results + get_page to warn the agent); (3) oversize → `embed_skip` soft-block via `buildEmbedSkipMarker()` PLUS a `content_flag:oversized` marker, AND deletes any pre-existing chunks in the same transaction so search can't surface stale chunks. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync, but an `embed_skip` transition (present vs absent) defeats both unchanged fast paths so a threshold change rebuilds or removes chunks without `--force`. A trusted import whose content the gate no longer blocks drops an incoming `embed_skip` (and a `content_flag`) whose reason is `oversized`; other skip reasons, quarantined pages and the sanity bypass keep them. `gbrain import` honors `errors > 0` for non-zero exit. `classifyErrorCode` in `src/core/sync.ts` recognizes the `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these. `extractEntityRefs` (canonical; matches both `[Name](../people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks), `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled` config helper. Link candidates match any dir-shaped path (existence-checked at persist). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. Pinned by `test/import-file-content-sanity.test.ts`. `importImageFile` records `source_path` on every image page and skips an unchanged image only when its index is complete for the run: a visual vector when embedding is on, and OCR (`ocr_status: 'done'` in frontmatter, or legacy stored OCR text) when OCR is enabled, so an image first imported under `--no-embed` or an OCR skip is rebuilt on its next import (`test/import-image-retry.test.ts`). `importCodeFile` writes `source_path: relativePath` on every code page (the same repo-relative path markdown imports record) so the full-sync reconcile — which only considers `source_path IS NOT NULL` rows — can retire a deleted code file's page; `putPage` COALESCEs the column, so rows imported before this keep NULL until the file changes, `sync --force`, or `gbrain reindex-code --force`. A frontmatter `slug:` that would become the page identity and ends in `.md`/`.mdx` is held as `frontmatter_slug_conflict` unless that exact (source_id, slug) page is live (`persistence/suffixed-slug.ts`).
+- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). Every markdown import resolves and stamps `contextual_retrieval_mode` and `corpus_generation` with the canonical write, `--no-embed` included (#5621), so later embed passes reproduce the page's wrapping; a failed source-policy read fails the import unless the row or table is missing (`SourceRowNotFoundError`, `isUndefinedTableError`). Unchanged markdown or code content whose page is below the safe-chunk fence is re-sealed projection-only (direct imports install it; coordinated `prepare` imports stay no-ops and queue a `safe_chunk_reseal` projection job), with no page write or admission. Freshly embedded chunks carry `embedding_input_hash` for the tier they were built under. The `beforeCommit` hook runs inside the guarded canonical transaction so a rejected file write rolls back its projections; optional legacy source-path repair uses a savepoint. Deferred importer embedding callbacks capture the committed text projection and indexing context, then install with an UPDATE-only revision/chunk/hash check. Tests: `test/import-publication-guards.test.ts`. Coordinated `prepare` imports resolve and atomically stamp the page/source/global contextual wrapping mode even with `noEmbed:true`; the durable outbox later embeds that revision. The existing paid-synopsis title fallback and mount trust gate remain in force, while embedding signatures only certify completed vectors. `test/persistence-contextual-embedding.test.ts` covers both engines, policy precedence, deferred provider execution, and a competing title replacement. `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent` reconciles tags by provenance (`tags.tag_source`, migration v175): frontmatter tags go through `addTag(..., { tagSource: 'frontmatter' })`, and a row still marked 'frontmatter' whose tag left the frontmatter is deleted in the import transaction. Every other `addTag` stamps 'added' (add_tag, enrichment, the code importer) and is never import-deleted, so `gbrain reindex --markdown` keeps enrichment tags; legacy NULL rows are adopted only while the frontmatter still lists them; prepared (managed) imports keep add-only tags, matching the union their canonical file renders. Pinned by `test/reindex-preserve-tags.test.ts`, `test/import-frontmatter-tag-removal.test.ts` + `test/import-file.test.ts`. identity decisions live in `src/core/import-identity.ts` (`decideImportIdentity`, `collidingSlugOwner`; pure reads, the importer applies them). The dedup pre-check calls `engine.findDuplicatePage?.(sourceId, {hash, frontmatterId, excludeSlug})` (optional `?` so test doubles compile; the caller's own slug never matches, and a `frontmatter.id` match ranks ahead of a bare hash match). Posture: MOVE when the `frontmatter.id` matches and the other page's OWN origin proves it moved (#5675): that origin is the `file://` path `importFromFile` records in `pages.source_uri` (never replacing non-file provenance), else the source's configured root (`local_path`, or the default source's legacy `sync.repo_path`). An origin that still exists moves only when it is this very file under another relative path (overlapping roots converge); a gone origin needs this file under that same root, and differing content also needs the same file name or title, so a missing path under another root, or a new note reusing a deleted note's template id, never renames a live page. `updateSlug` carries the row (page id, links, facts incl. `entity_slug`/`source_markdown_slug`, `page_aliases`, timeline, plus a rename alias) and the new content imports onto it, so a full sync's delete-reconcile has nothing live to remove; SKIP only when the id AND the content (`content_hash`, or title and body for a row written outside the importer) both match while the recorded file still exists (overlapping ingest roots); IMPORT with a warning when the id matches but content differs (templates, copy-paste; an edit is never frozen by the other page); WARN-ALWAYS on a bare content_hash collision; FAIL CLOSED on lookup error; bypass via `--force-rechunk`. Content-only callers (`put_page`, transcripts) never move a page. Soft-deleted pages excluded at the engine layer so tombstones don't block legitimate re-imports under new slugs. Slug collisions: when another live file (`existing.source_path`, realpath-distinct) already owns the slug, the incoming file is skipped with `skip_reason: 'slug_collision'` and a stderr warning, unless the incoming file is named exactly `<slug>.md` (the file the link extractor reads for that slug), so two spellings never overwrite each other. An embedding failure never blocks the text write: chunks commit with NULL vectors, no embedding signature is stamped, the result carries `embedding_deferred: true`, and the stale sweep embeds them later (a code file whose embed failed is not stamped either). An inline markdown re-import embeds only chunks whose embedding input changed: `planEmbeddingReuse` (keyed on chunk source + exact text) reuses a stored vector when the old index is sealed (`requireSafeChunks`), the page's `embedding_signature`, contextual mode, `corpus_generation` and (under the title wrapper) sanitized title are unchanged, and a fenced page has a recorded input (#6961); `--force-rechunk` and post-commit-embedding imports reuse nothing (`test/import-markdown-embedding-reuse.test.ts`); identical rows stay put (`import-chunk-reuse.ts`). Offset-less frontmatter datetimes are read in the brain's `brain.timezone` (IANA, validated by `gbrain config set`; unset or invalid = UTC; the YAML parser tags naive timestamps with `NAIVE_DATETIME` so the wall-clock reading survives), while date-only values stay UTC calendar dates (`test/effective-date-brain-timezone.test.ts`). `effective_date` for an undated page falls back to `fallbackCreatedAt` (stored fallback date, else the earliest of the file's birth/modification time and — when `sync.git_first_commit_dates` is on — its git first-commit date via `fileTimes`, else now; `src/core/git-first-commit.ts` maps paths in one `git log --diff-filter=A` pass per `runImport`, skipping shallow clones; `test/effective-date-git-first-commit.test.ts`), never the latest write. Pinned by `test/import-dedup-frontmatter-id.test.ts`, `test/import-identity-move.test.ts`, `test/import-slug-collision.test.ts`, `test/import-embed-outage.test.ts` and `test/effective-date-calendar.test.ts`. `importFromContent` is the narrow waist every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It runs a three-tier content-quality disposition via `assessContentSanity` from `src/core/content-sanity.ts` BEFORE chunking: (1) high-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE (stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search) OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`; (2) fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker (page stays fully searchable, marker rides search results + get_page to warn the agent); (3) oversize → `embed_skip` soft-block via `buildEmbedSkipMarker()` PLUS a `content_flag:oversized` marker, AND deletes any pre-existing chunks in the same transaction so search can't surface stale chunks. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync, but an `embed_skip` transition (present vs absent) defeats both unchanged fast paths so a threshold change rebuilds or removes chunks without `--force`. A trusted import whose content the gate no longer blocks drops an incoming `embed_skip` (and a `content_flag`) whose reason is `oversized`; other skip reasons, quarantined pages and the sanity bypass keep them. `gbrain import` honors `errors > 0` for non-zero exit. `classifyErrorCode` in `src/core/sync.ts` recognizes the `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these. `extractEntityRefs` (canonical; matches both `[Name](../people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks), `extractPageLinks`, `inferLinkType` heuristics (attended/works_at/invested_in/founded/advises/source/mentions), `parseTimelineEntries`, `isAutoLinkEnabled` config helper. Link candidates match any dir-shaped path (existence-checked at persist). Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. Pinned by `test/import-file-content-sanity.test.ts`. `importImageFile` records `source_path` on every image page and skips an unchanged image only when its index is complete for the run: a visual vector when embedding is on, and OCR (`ocr_status: 'done'` in frontmatter, or legacy stored OCR text) when OCR is enabled, so an image first imported under `--no-embed` or an OCR skip is rebuilt on its next import (`test/import-image-retry.test.ts`). `importCodeFile` writes `source_path: relativePath` on every code page (the same repo-relative path markdown imports record) so the full-sync reconcile — which only considers `source_path IS NOT NULL` rows — can retire a deleted code file's page; `putPage` COALESCEs the column, so rows imported before this keep NULL until the file changes, `sync --force`, or `gbrain reindex-code --force`. A frontmatter `slug:` that would become the page identity and ends in `.md`/`.mdx` is held as `frontmatter_slug_conflict` unless that exact (source_id, slug) page is live (`persistence/suffixed-slug.ts`).
- `src/core/link-source-check-repair.ts` — schema-drift self-heal for the `links_link_source_check` CHECK constraint (third instance of the `timeline-dedup-repair.ts` / `pages-upsert-arbiter.ts` class). Migration v114 opened `link_source` from a closed allowlist to the kebab-case gate; the migrator trusts the version ledger, so a brain stamped >= 114 whose live constraint reverted to the pre-v114 allowlist (drifted restore, manual DDL) rejects every kebab provenance write (`atom-provenance`, `concept-provenance`) while the ledger reads current. `LINK_SOURCE_KEBAB_RE` is the ONE exported copy of the regex (a drift-guard test pins it to the literal in migrate.ts v114, `schema.sql`, `pglite-schema.ts`). `checkLinkSourceCheck(engine)` probes `pg_constraint` (`pg_get_constraintdef` + `convalidated`) and classifies the drift as `absent` / `wrong_def` / `not_validated` (read-only; powers the `links_link_source_check` doctor check in `doctor/checks/core-health.ts`, whose severity follows what writes do: only `wrong_def` rejects kebab writes → fail; `absent` / `not_validated` still accept them → warn). `repairLinkSourceCheck(engine)` first COUNTs rows that fail the gate and returns `reason: 'violations'` WITHOUT any DDL when there are any (the prior constraint survives; rows are never touched; a brain with a bad row never pays for a failing ALTER at every engine open), then restores v114's definition mirroring the migration's engine split: Postgres runs DROP + `ADD ... NOT VALID` in one `engine.transaction` (atomic, no scan) and `VALIDATE CONSTRAINT` as a separate `engine.runMigration` outside it (SHARE UPDATE EXCLUSIVE, never a plain ADD's ACCESS EXCLUSIVE + full-table scan), with a `not_validated` drift getting the VALIDATE only; PGLite keeps the one-shot DROP+ADD transaction. A DDL failure re-counts violators (a row written between probe and ALTER) and rethrows when there are none. `LINK_SOURCE_GATE_MIGRATION_VERSION` (114) is the ledger gate `runMigrations` applies before calling it. Pinned by `test/link-source-check-repair.test.ts` (PGLite, incl. a recording-engine assertion that violators trigger no DDL and that a sub-114 ledger never probes) and the DATABASE_URL-gated `test/e2e/link-source-check-repair-postgres.test.ts` (two-phase DDL sequence on real Postgres).
diff --git a/scripts/postgres-arm-weights-postgres-only.json b/scripts/postgres-arm-weights-postgres-only.json
index 012090627d..c9054f71bc 100644
--- a/scripts/postgres-arm-weights-postgres-only.json
+++ b/scripts/postgres-arm-weights-postgres-only.json
@@ -52,6 +52,7 @@
"test/fact-purge-lookup-timeout-6511.test.ts": 8734,
"test/facts-publication-large-fence-6898.test.ts": 37407,
"test/fence-repair-receipts.test.ts": 3695,
+ "test/fenced-embed-reuse-6961.test.ts": 2072,
"test/forecast-journal.test.ts": 21056,
"test/frontmatter-reserved-keys-6454.test.ts": 6227,
"test/get-chunks-paging.test.ts": 8467,
diff --git a/scripts/postgres-arm-weights.json b/scripts/postgres-arm-weights.json
index a7aca2295a..cfc6fcf172 100644
--- a/scripts/postgres-arm-weights.json
+++ b/scripts/postgres-arm-weights.json
@@ -67,6 +67,7 @@
"test/facts-subject-infer-7405.test.ts": 4890,
"test/fence-fact-append.test.ts": 32856,
"test/fence-repair-receipts.test.ts": 8636,
+ "test/fenced-embed-reuse-6961.test.ts": 4602,
"test/file-drift-soft-delete-hold-release-gbra133.test.ts": 12766,
"test/forecast-journal.test.ts": 22619,
"test/frontmatter-reserved-keys-6454.test.ts": 4771,
diff --git a/src/core/import-chunk-reuse.ts b/src/core/import-chunk-reuse.ts
index 58e5d8c448..c2e16fefed 100644
--- a/src/core/import-chunk-reuse.ts
+++ b/src/core/import-chunk-reuse.ts
@@ -34,7 +34,9 @@ export interface KeptChunkRow { id: number; chunk_index: number; vector: boolean
export async function reuseStoredChunkVectors(
exec: BrainEngine,
into: ChunkInput[],
- { slug, sourceId, title, corpusGeneration, tier, prepared, keepUnembedded = false }: { slug: string; sourceId: string; title: string; corpusGeneration: string | null; tier: 'title' | 'none'; prepared: boolean; keepUnembedded?: boolean },
+ { slug, sourceId, title, corpusGeneration, tier, prepared, keepUnembedded = false, requireRecordedInput = false }: { slug: string; sourceId: string; title: string; corpusGeneration: string | null; tier: 'title' | 'none'; prepared: boolean; keepUnembedded?: boolean;
+ /** #6961: a fenced page admits only vectors whose embedding input was recorded, never the legacy raw-text match. */
+ requireRecordedInput?: boolean },
): Promise<Map<number, KeptChunkRow | null>> {
const reused = new Map<number, KeptChunkRow | null>();
const target = await embeddingWriteTarget(exec);
@@ -55,7 +57,7 @@ export async function reuseStoredChunkVectors(
// Reuse is keyed on chunk source + text, so a stored chunk's current-input
// hash is the hash its matching new chunk would record.
const all = await exec.getChunks(slug, { sourceId, includeEmbedding: true, requireSafeChunks: true });
- const stored = all.filter(chunk => chunk.embedding && canReuseMarkdownVector(recorded.get(chunk.chunk_index), rows[0]!.contextual_retrieval_mode, tier, provenance, chunk));
+ const stored = all.filter(chunk => chunk.embedding && (!requireRecordedInput || recorded.get(chunk.chunk_index) != null) && canReuseMarkdownVector(recorded.get(chunk.chunk_index), rows[0]!.contextual_retrieval_mode, tier, provenance, chunk));
const identity = (chunk: ChunkInput | Chunk) => JSON.stringify([chunk.chunk_text, chunk.chunk_source,
chunk.language ?? null, chunk.symbol_name ?? null, chunk.symbol_type ?? null, chunk.start_line ?? null, chunk.end_line ?? null,
chunk.parent_symbol_path ?? null, chunk.doc_comment ?? null, chunk.symbol_name_qualified ?? null, chunk.modality ?? 'text']);
diff --git a/src/core/import-file.ts b/src/core/import-file.ts
index 6628f6255c..479fe4e3da 100644
--- a/src/core/import-file.ts
+++ b/src/core/import-file.ts
@@ -665,8 +665,8 @@ export async function importFromContent(
// recorded embedding input (#5553: column, model, dimensions, wrapper tier
// and exact input) equals the one this import would embed, or, for a vector
// recorded before provenance existed, when the input is the raw chunk text
- // under the same model. The old index must be sealed and neither body may
- // hold protected fences, so no reused vector can carry a private sibling.
+ // under the same model. The old index must be sealed (no private fence row in a chunk), and on a fenced page
+ // only a vector with a recorded input is reused (#6961); a changed fence chunk has new text and is embedded again.
// A prepared (persistence) import defers embedding to its effect worker, so
// it reuses inside its publication transaction, under the page guard, from
// the chunks it is about to replace; only the chunks left without a vector
@@ -677,8 +677,8 @@ export async function importFromContent(
const largePage = isLargePageBytes(bodyBytes) || segmented !== null;
const postCommitEmbedding = largePage ? undefined : opts.onPostCommitEmbedding;
const reusable = !!existing && !existing.deleted_at && !opts.forceRechunk && !postCommitEmbedding && chunks.length > 0 && bodyStorageOf(existing) !== 'segmented'
- && !(opts.prepare && modeRequiresSynopsis(effectiveCRMode)) && !hasProtectedBody(`${existing.compiled_truth}\n${existing.timeline ?? ''}`) && !hasProtectedBody(`${parsed.compiled_truth}\n${parsed.timeline ?? ''}`);
- const reuseCtx = { slug, sourceId: sourceId ?? 'default', title: parsed.title, corpusGeneration, tier: effectiveCRMode === 'title' ? 'title' as const : 'none' as const };
+ && !(opts.prepare && modeRequiresSynopsis(effectiveCRMode));
+ const reuseCtx = { slug, sourceId: sourceId ?? 'default', title: parsed.title, corpusGeneration, tier: effectiveCRMode === 'title' ? 'title' as const : 'none' as const, requireRecordedInput: reusable && (hasProtectedBody(`${existing.compiled_truth}\n${existing.timeline ?? ''}`) || hasProtectedBody(`${parsed.compiled_truth}\n${parsed.timeline ?? ''}`)) };
const reuseStoredVectors = (exec: BrainEngine, into: ChunkInput[], keepUnembedded = false) => reuseStoredChunkVectors(exec, into, { ...reuseCtx, prepared: !!opts.prepare, keepUnembedded });
const reused = reusable && !noEmbed && !opts.prepare ? await reuseStoredVectors(engine, chunks) : new Map<number, number | null>();
diff --git a/test/fenced-embed-reuse-6961.test.ts b/test/fenced-embed-reuse-6961.test.ts
new file mode 100644
index 0000000000..d9e79435a6
--- /dev/null
+++ b/test/fenced-embed-reuse-6961.test.ts
@@ -0,0 +1,83 @@
+/**
+ * #6961: a write to a page with a facts or takes fence re-embeds only the chunks whose text changed. Before, the
+ * A13 reuse gate turned off for any fenced page, so a one-row fact append re-embedded every chunk of the entity page.
+ * A fenced page reuses only vectors whose recorded embedding input equals the one this import would embed; a legacy
+ * vector with no recorded input is re-embedded there, since nothing proves it was computed from the sanitized text.
+ *
+ * PGLite always, Postgres when DATABASE_URL is set; the embedding transport is stubbed ($0).
+ */
+
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const DIMS = 1536;
+let embedded: string[] = [];
+let counter = 0;
+
+function gateway() {
+ configureGateway({ embedding_model: 'openai:text-embedding-3-large', embedding_dimensions: DIMS, env: { OPENAI_API_KEY: 'sk-test' } });
+ __setEmbedTransportForTests((async ({ values }: { values: string[] }) => {
+ embedded.push(...values);
+ return { embeddings: values.map(() => { const v = new Array(DIMS).fill(0); v[counter++ % DIMS] = 1; return v; }), usage: { tokens: values.length } };
+ }) as never);
+}
+
+const paragraph = (word: string) => Array.from({ length: 400 }, (_, i) => `${word}${i}`).join(' ') + '.';
+const FENCE = (rows: string[]) => ['## Facts', '<!--- gbrain:facts:begin -->',
+ '| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |',
+ '|---|-------|------|------------|------------|------------|------------|-------------|--------|---------|',
+ ...rows.map((claim, i) => `| ${i + 1} | ${claim} | fact | 1.0 | world | medium | 2017-01-01 | | chat | |`), '<!--- gbrain:facts:end -->', ''].join('\n');
+const PERSON = (facts: string[]) => `---\ntitle: Alice Example\ntype: person\n---\n# Alice Example\n\n${
+ ['alpha', 'bravo', 'charlie', 'delta'].map(w => `## Section ${w}\n\n${paragraph(w)}`).join('\n\n')}\n\n${FENCE(facts)}`;
+const FACTS = ['Alice Example works at Acme Example', 'Alice Example lives in Springfield'];
+const APPENDED = 'Alice Example joined the board of Fund A';
+
+for (const backend of testBackends()) describe(`#6961 fenced page vector reuse (${backend})`, () => {
+ let engine: BrainEngine;
+ let close: () => Promise<void>;
+ const chunkCount = async (slug: string) => (await engine.getChunks(slug, { includeUnsealed: true })).length;
+
+ beforeAll(async () => { resetGateway(); gateway(); ({ engine, close } = await isolatedSharedSkillsEngine(backend === 'postgres' ? process.env.DATABASE_URL : undefined)); }, 120_000);
+ afterAll(async () => { __setEmbedTransportForTests(null); resetGateway(); await close(); }, 60_000);
+ beforeEach(() => { embedded = []; gateway(); });
+
+ // Value: protects=a one-row fact append on a fenced entity page embeds only the chunk the row changed (#6961: about
+ // 13-15M embedding tokens a day on one reference brain); fails_when=the reuse gate is off for fenced pages and every
+ // chunk is re-embedded; why_new=the only fenced-page reuse test pinned the full re-embed; seam=embed transport stub
+ test('a one-row fact append re-embeds only the changed fence chunk', async () => {
+ const slug = 'people/alice-example';
+ await importFromContent(engine, slug, PERSON(FACTS));
+ const total = await chunkCount(slug);
+ expect(total).toBeGreaterThanOrEqual(4);
+ embedded = [];
+ await importFromContent(engine, slug, PERSON([...FACTS, APPENDED]));
+ expect(embedded.length).toBeGreaterThan(0);
+ expect(embedded.length).toBeLessThan(total);
+ expect(embedded.some(text => text.includes(APPENDED))).toBe(true);
+ for (const text of embedded) expect(text).not.toMatch(/alpha0 |bravo0 |charlie0 /);
+ expect((await engine.getChunks(slug, { includeEmbedding: true, includeUnsealed: true })).every(chunk => chunk.embedding)).toBe(true);
+ }, 120_000);
+
+ // Value: protects=a fenced page never reuses a vector whose input is unproven (no recorded embedding input), so no
+ // reused vector can carry text the sanitized chunk lacks; fails_when=the legacy raw-text branch (tier none, no stored mode)
+ // admits it on a fenced page; why_new=fenced pages reused nothing before #6961; seam=embedding_input_hash and the page mode cleared in SQL (a pre-provenance page)
+ test('a legacy vector with no recorded input is re-embedded on a fenced page', async () => {
+ const slug = 'people/alice-example-legacy';
+ await engine.setConfig('search.contextual_retrieval', 'none');
+ try {
+ await importFromContent(engine, slug, PERSON(FACTS));
+ const total = await chunkCount(slug);
+ await engine.executeRaw(`UPDATE content_chunks SET embedding_input_hash = NULL WHERE page_id = (SELECT id FROM pages WHERE slug = $1)`, [slug]);
+ await engine.executeRaw(`UPDATE pages SET contextual_retrieval_mode = NULL WHERE slug = $1`, [slug]);
+ embedded = [];
+ await importFromContent(engine, slug, PERSON([...FACTS, APPENDED]));
+ expect(embedded.length).toBe(total);
+ } finally {
+ await engine.executeRaw(`DELETE FROM config WHERE key = 'search.contextual_retrieval'`);
+ }
+ }, 120_000);
+});
diff --git a/test/import-inline-chunk-keep.test.ts b/test/import-inline-chunk-keep.test.ts
index 1703e754cc..590dd25a4f 100644
--- a/test/import-inline-chunk-keep.test.ts
+++ b/test/import-inline-chunk-keep.test.ts
@@ -136,14 +136,17 @@ for (const kind of ['pglite', 'postgres'] as const) {
expect(embeddedRows.every(row => row.embedding !== null)).toBe(true);
}, 120_000);
- test('an unchanged re-import with a protected fence keeps nothing', async () => {
+ test('an edit with a protected fence keeps its unchanged rows, and the fence text reaches no row (#6961)', async () => {
const fence = '\n<!--- gbrain:takes:begin -->\nPRIVATE_CANARY\n<!--- gbrain:takes:end -->\n';
await write('notes/protected', page('Protected', ['mike', 'november', 'oscar'], fence));
const before = await rows('notes/protected');
await write('notes/protected', page('Protected', ['mike', 'november', 'papa'], fence), true);
const after = await rows('notes/protected');
- expect(after.some(row => before.some(old => old.id === row.id))).toBe(false);
- expect(after.every(row => row.embedding === null)).toBe(true);
+ const kept = after.filter(row => before.some(old => old.id === row.id));
+ expect(kept.length).toBeGreaterThan(0);
+ for (const row of kept) expect(row).toEqual(before.find(old => old.id === row.id)!);
+ expect(after.filter(row => row.embedding === null).every(row => String(row.chunk_text).includes('papa'))).toBe(true);
+ expect(after.some(row => String(row.chunk_text).includes('PRIVATE_CANARY'))).toBe(false);
}, 120_000);
});
}
diff --git a/test/import-markdown-embedding-reuse.test.ts b/test/import-markdown-embedding-reuse.test.ts
index ecf8637c88..cfe28638b2 100644
--- a/test/import-markdown-embedding-reuse.test.ts
+++ b/test/import-markdown-embedding-reuse.test.ts
@@ -1,8 +1,9 @@
/**
* A13: a markdown edit re-embeds only the chunks whose embedding input
* changed. Unchanged chunks keep their stored vectors when the chunk text,
- * chunk source, contextual wrapper and embedding signature all match and
- * neither the old nor the new body holds protected (facts/takes) fences.
+ * chunk source, contextual wrapper and embedding signature all match. On a page
+ * with protected (facts/takes) fences only a vector with a recorded embedding
+ * input is reused (#6961).
*
* PGLite in-memory; the embedding transport is stubbed ($0).
*/
@@ -95,14 +96,17 @@ describe('markdown import reuses unchanged chunk vectors (A13)', () => {
expect(embedded.length).toBe((await vectors(slug)).size);
});
- test('a page with protected fences never reuses vectors', async () => {
+ test('a page with protected fences reuses only vectors with a recorded input, and the fence text reaches no chunk (#6961)', async () => {
const slug = 'notes/reuse-protected';
const fence = '\n<!--- gbrain:takes:begin -->\nPRIVATE_CANARY\n<!--- gbrain:takes:end -->\n';
await importFromContent(engine, slug, page('Protected', ['alpha', 'bravo', 'charlie'], fence));
embedded = [];
await importFromContent(engine, slug, page('Protected', ['alpha', 'bravo', 'delta'], fence));
- const count = (await vectors(slug)).size;
- expect(embedded.length).toBe(count);
+ const after = await vectors(slug);
+ expect(embedded.length).toBeGreaterThan(0);
+ expect(embedded.length).toBeLessThan(after.size);
+ for (const text of embedded) expect(text).toContain('delta');
+ for (const text of [...embedded, ...after.keys()]) expect(text).not.toContain('PRIVATE_CANARY');
});
test('--force-rechunk re-embeds every chunk', async () => {
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index 9688eda501..a5b9661dba 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -81,6 +81,7 @@ test/facts-publication-large-fence-6898.test.ts
test/facts-subject-infer-7405.test.ts
test/fence-fact-append.test.ts
test/fence-repair-receipts.test.ts
+test/fenced-embed-reuse-6961.test.ts
test/file-drift-soft-delete-hold-release-gbra133.test.ts
test/forecast-journal.test.ts
test/frontmatter-reserved-keys-6454.test.ts