Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(#6961): a chunk row kept at its own index takes its chunk's new byte span

garrytan33692fa7c92f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 722a00d2a5..f68020e999 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,7 +18,8 @@ Vector reuse was turned off for every page with a protected fence, so each write
### Itemized changes
-- **Fenced page reuse (#6961).** A fenced page now reuses a stored vector when its recorded embedding input equals the one the import would embed, the same check every other page gets. The stored index must still be sealed by the fence-safe chunker, so a reused vector's text holds no private fence row. A chunk whose text changed, such as the fence chunk a new fact lands in, is embedded again. The importer never reuses a legacy vector with no recorded input on a fenced page, because nothing proves what text it was computed from. A one-row fact append on a five-chunk entity page now makes one embedding call instead of five. Contributed by @javieraldape (#7055, rewritten on master).
+- **Fenced page reuse (#6961).** A fenced page now reuses a stored vector when its recorded embedding input equals the one the import would embed, the same check every other page gets. The stored index must still be sealed by the fence-safe chunker, so a reused vector's text holds no private fence row. A chunk whose text changed, such as the fence chunk a new fact lands in, is embedded again. On a fenced page the importer reuses only vectors with a recorded embedding input, so a legacy vector with none is embedded again. A one-row fact append on a five-chunk entity page now makes one embedding call instead of five. Contributed by @javieraldape (#7055, rewritten on master).
+- **Kept chunk spans (#6961).** A chunk row kept at its own index now takes its chunk's new byte span when earlier text on the page grew, so `get_page` windows around a chunk and search chunk windows point into the current body. Before, only rows that moved to another index got the new span.
## [0.60.539.0] - 2026-11-01
diff --git a/docs/architecture/key-files/core-services-2-continued.md b/docs/architecture/key-files/core-services-2-continued.md
index a8da98793c..11e0bb5f96 100644
--- a/docs/architecture/key-files/core-services-2-continued.md
+++ b/docs/architecture/key-files/core-services-2-continued.md
@@ -9,7 +9,7 @@ The page-state concurrency foundation under `src/core/page-state/` and the share
- `src/core/page-state/` — Canonical page concurrency foundation shared by both engines. `schema.ts` (migration 150) adds source incarnation UUIDs, logical page revisions, text-projection revisions, and persistent guards for absent pages. `guards.ts` locks sorted sources then page identities in an existing transaction; a transaction or savepoint never re-acquires a key it holds, which covers absent pages because the only pages INSERT runs after `putPage` takes the same guard (`test/e2e/page-guards-postgres.test.ts`); callers lock sources before authority/request/fact rows. `snapshot.ts` computes the normalized body/timeline fingerprints only for a page with fact withdrawals, page-subject purges or a source '*' purge marker (they feed only the withdrawal overlay), as `page-snapshot-batch.ts` does. `snapshot.ts` reads page/tags/withdrawals in one MVCC statement and applies withdrawals using database-normalized text. `transactions.ts` makes helper transaction APIs compose through savepoints. `versions.ts` records the complete canonical snapshot; `rename-alias.ts` records `slug_aliases(old -> new)` inside both engines' `updateSlug` transaction, repoints aliases that named the old slug and drops one spelled like the new slug; `moveSlugBindings` moves the page's facts (`entity_slug`, `source_markdown_slug`) and `page_aliases` in the same transaction, letting a purged page's leftovers at the new slug yield (`test/rename-slug-alias.test.ts`); `movePageIdReferences` (#5431) moves the page-id-keyed rows of a stale rename row onto the surviving row (`links` from/to/origin, `timeline_entries` page and event, `page_versions`), dropping stale-side duplicates on the `links` composite key and the timeline dedup indexes (`test/rename-slug-alias.test.ts`); `materialize.ts` persists only verified canonical overlay bytes without another logical revision. `types.ts` defines revision preconditions: no revision is create-only (an existing page refuses `revision_required`, #5385, its own code since the omission is the caller's and not a race), a stale revision or a page another writer created where an internal writer asserted none (`expectedRevision: null`) is `revision_conflict`, explicit force overwrites. Plain legacy `putPage` calls keep compatibility until their owning operation adopts the managed write protocol. `projection-schema.ts` (migration 153) invalidates unverified prior projections once and queues durable work; `projections.ts` rebuilds Markdown without a worktree/provider and installs only matching canonical revisions. Embedding installs also compare exact chunk identities/text. Source importers seal code/image projections; all read arms hide unsealed projections.
- `src/core/page-state/projections.ts` — `sealPageTextProjection` and `sealImportedPage` write the page keyword vector through `pageSearchVectorSql` (`fts-language.ts`: title weight A, sanitized timeline weight C, both through `gbrain_fts_input`), the same expression `search-vector-backfill.ts` uses. Captures canonical revision, source incarnation, page identity, the complete raw chunk inventory and resolved indexing inputs before preparing derived work. Embedding reads require a current text seal; healing/rebuild capture can explicitly include unsealed pages. Projection installation compares its originating seal, chunk identities/text/source/completion state, chunker version and indexing context (including stored corpus generation) under the page guard before replacing any row. A same-revision re-chunk may replace the snapshot it actually captured; a delayed older installation cannot discard newer chunks or vectors. Rebuild also rechecks its queued job before capture so a completed job from an old listing stays completed. Chunkless healing sanitizes complete body/timeline fences before splitting, preserving withdrawn history in direct page reads while excluding it from retrieval. Tests: `page-projection-origin.test.ts`, `page-projection-vector-preservation.test.ts` (PGLite and isolated Postgres), `embed-stale-chunkless-pages.serial.test.ts` (public healing entry). Shared `markdown-chunks.ts` and `code-chunks.ts` preserve fenced code and source-language metadata. A preserving rebuild keeps a vector only when its `content_chunks.embedding_input_hash` (`src/core/embedding-input-hash.ts`) equals one recomputed from the current page, or, with no recorded hash, when the page is non-contextual and the legacy model/text-hash checks pass; a contextual chunk without provenance is nulled once. `installPageEmbeddings` records the hash in the vector's UPDATE (the plain re-embed tier by default, or the caller's `built` tier and corpus generation), and returns `{ installed, complete }`: it matches each offered vector to the page's current chunk by source and exact text (not index) and writes it only when its input hash for the captured context is one the current context accepts, in batched `jsonb_to_recordset` UPDATEs, so a page that changed under the provider call keeps the vectors of its unchanged chunks; the embedding signature is stamped only when `complete` (same chunk sequence, nothing dropped). A preserving rebuild likewise keys kept vectors by chunk content, so a top insert shifts vectors with their chunks (tests: `page-embedding-content-validity.test.ts`). Also `stampEmbeddingInputs` does the same for full imports. `resealSafeChunks` re-seals a page below the safe-chunk fence from its unchanged body with no canonical write; `projectionBelowSafeFence` is its predicate. Tests add `projection-embedding-input-hash.test.ts` and `safe-chunk-reseal.test.ts`. It replaces outgoing code edges without deleting incoming edges to retained chunks and clears backfill watermarks. Failed jobs rotate with a retained diagnostic; successful drain refreshes planner statistics. `rebuildPendingPageProjections(engine, limit, { deadlineMs, now, notAfter, retryCooldown, onFailure })` (#5401) checks an injectable clock between pages, bounds a run to rows queued at or before its start (`notAfter`, cast from text in SQL so microseconds survive the driver), applies the 30 s `rebuild_failed` cooldown for the resident, and treats a lock or snapshot error as that page's failure instead of aborting the batch; `projectionBacklog` returns counts only (pending, failed, oldest age) for the drain command and the resident's `projection_status` IPC reply. A same-revision conflict throws `PageProjectionConflictError` naming the fields that changed (`text_projection_revision`, `chunk_digest`, `indexing_context`), the slug and source; `retryProjectionConflict(attempt)` re-runs an attempt that captures a fresh snapshot at most three more times with jitter, then rethrows (`page_projection_conflict` on the wire). Tests: `page-projection-origin.test.ts` (resident rebuild race, two-writer progress, exhausted-retry CLI/JSON output).
-- `src/core/page-state/chunk-moves.ts` — #7172: `CHUNK_VECTOR_WRITE_BUDGET` (5), the most new `content_chunks` row versions carrying a vector (inserts, and non-HOT updates of `chunk_index`, text or vector; each adds an HNSW entry) one transaction of a managed chunk replacement makes, with its derivation from the publication's 5 s statement bound (a version without a vector is not counted); `planMoveWaves` plans the in-place renumbering of kept rows within a budget (one UPDATE per wave, a cycle parks one row at `-1 - target`), `applyMoveWaves` runs it. `import-file.ts` (with `boundedChunkWrites`, set by page-prepare, import-prepare and sync-prepare) commits a page whose replacement exceeds the budget with its projection pending (queue reason `chunk_replacement_deferred`), and `projections.ts` `continueDeferredProjection` finishes it after commit in steps of at most the budget; `rebuildPendingPageProjections` resumes an interrupted one.
+- `src/core/page-state/chunk-moves.ts` — #7172: `CHUNK_VECTOR_WRITE_BUDGET` (5), the most new `content_chunks` row versions carrying a vector (inserts, and non-HOT updates of `chunk_index`, text or vector; each adds an HNSW entry) one transaction of a managed chunk replacement makes, with its derivation from the publication's 5 s statement bound (a version without a vector is not counted); `planMoveWaves` plans the in-place renumbering of kept rows within a budget (one UPDATE per wave, a cycle parks one row at `-1 - target`), `applyMoveWaves` runs it; `planImportChunkWrites` adds a last uncounted wave that gives a row kept at its own index its chunk's new span (#6961). `import-file.ts` (with `boundedChunkWrites`, set by page-prepare, import-prepare and sync-prepare) commits a page whose replacement exceeds the budget with its projection pending (queue reason `chunk_replacement_deferred`), and `projections.ts` `continueDeferredProjection` finishes it after commit in steps of at most the budget; `rebuildPendingPageProjections` resumes an interrupted one.
- `src/core/page-state/projection-class.ts` — #6803: `projectionClassSql(p, s)`, the one CASE that puts each queued page in exactly one class (`archived`, `text` for Markdown/code and segmented pages, `protected` for a protected fence, `sealable` per `sealableImageSql`, else `reimport`), shared by `projectionBacklog`, the drain probe, search readiness and `embed --stale --images`. `sealableImageSql`: an image page at or after the safe-chunk fence whose one chunk (index 0, `image_asset`) carries the page's own text.
- `src/core/page-state/image-seal.ts` — #6803: `sealImageProjections` restores the importer's seal on `sealable` images in groups of 500: a primary-key window of queue rows probed by each page's unique key (a statistics-less PGLite planner otherwise probes through `pages_dedup_idx`), then under the page guard a revision compare-and-set with the class rechecked; no chunk write, the visual vector untouched, a text vector whose recorded input no longer matches nulled for `embed --stale`. `IMAGE_SEAL_PAGES_PER_SECOND` is the measured rate (its comment names the run); `projectionDrainLimit(textPending)` sizes doctor's `--limit` to half `AGENT_TOOL_LIMIT_MS` at that rate, or at the measured `TEXT_REBUILD_PAGES_PER_SECOND` while queued text waits (the drain takes text first).
- `src/core/page-state/projection-window.ts` — #6803: the resident's bounded queue read: per pass at most `RESIDENT_PROBE_BUDGET` (500) rows, the 100 newest by `updated_at` plus a primary-key window from a per-engine cursor that starts at a random key and wraps; a negative cache (100k keys) skips image rows the resident never builds. The consumer's idle probe walks the same window and advances the same cursor. Tests: `projection-resident-window.test.ts`.
diff --git a/src/core/import-chunk-reuse.ts b/src/core/import-chunk-reuse.ts
index c2e16fefed..72310b1e8d 100644
--- a/src/core/import-chunk-reuse.ts
+++ b/src/core/import-chunk-reuse.ts
@@ -18,7 +18,7 @@ import { segmentSynopsisInputHashes, segmentSynopsisLead } from './page-body/seg
* A stored row an import keeps: its id and the index it holds now (a kept row whose new chunk sits elsewhere moves
* there). `vector`: the row holds one, so moving it is a vector-index write (page-state/chunk-moves.ts).
*/
-export interface KeptChunkRow { id: number; chunk_index: number; vector: boolean }
+export interface KeptChunkRow { id: number; chunk_index: number; vector: boolean; projected_byte_start: number | null; projected_byte_end: number | null }
/**
* A13 reuse for one page import: copies each stored vector the reuse gate
@@ -62,6 +62,8 @@ export async function reuseStoredChunkVectors(
chunk.language ?? null, chunk.symbol_name ?? null, chunk.symbol_type ?? null, chunk.start_line ?? null, chunk.end_line ?? null,
chunk.parent_symbol_path ?? null, chunk.doc_comment ?? null, chunk.symbol_name_qualified ?? null, chunk.modality ?? 'text']);
const key = (chunk: { chunk_source?: string | null; chunk_text: string }) => `${chunk.chunk_source}\0${chunk.chunk_text}`;
+ const kept = (row: Chunk, vector: boolean): KeptChunkRow => ({ id: row.id, chunk_index: row.chunk_index, vector,
+ projected_byte_start: row.projected_byte_start ?? null, projected_byte_end: row.projected_byte_end ?? null });
// A stored row at the chunk's own index first, then FIFO by source + text: one page can hold two identical chunks.
const atIndex = new Map(stored.map(chunk => [chunk.chunk_index, chunk]));
const matches = new Map<number, Chunk>();
@@ -76,7 +78,7 @@ export async function reuseStoredChunkVectors(
if (row.model) into[i].model = row.model;
// The stored row already holds exactly what this chunk would insert (identity, vector, recorded input and text
// hash, no image or multimodal vector the insert would drop), so it can stay, moved to this chunk's index.
- reused.set(i, recorded.get(row.chunk_index) != null && textOnly.has(row.chunk_index) && identity(row) === identity(into[i]) ? { id: row.id, chunk_index: row.chunk_index, vector: true } : null);
+ reused.set(i, recorded.get(row.chunk_index) != null && textOnly.has(row.chunk_index) && identity(row) === identity(into[i]) ? kept(row, true) : null);
}
// A stored row that never got a vector stays in place under a new chunk that
// will not get one either (a --no-embed or deferred write): the same upsert
@@ -87,7 +89,7 @@ export async function reuseStoredChunkVectors(
for (const row of all) {
const i = byIndex.get(row.chunk_index);
if (i === undefined || reused.has(i) || into[i]!.embedding || !unembedded.has(row.chunk_index)) continue;
- if (identity(row) === identity(into[i]!)) reused.set(i, { id: row.id, chunk_index: row.chunk_index, vector: false });
+ if (identity(row) === identity(into[i]!)) reused.set(i, kept(row, false));
}
}
return reused;
diff --git a/src/core/page-state/chunk-moves.ts b/src/core/page-state/chunk-moves.ts
index d4ae5a348d..ed50132bee 100644
--- a/src/core/page-state/chunk-moves.ts
+++ b/src/core/page-state/chunk-moves.ts
@@ -66,15 +66,22 @@ export function planMoveWaves(moves: readonly ChunkMove[], occupied: ReadonlyMap
* publication), `defer` says the vector-index writes (moves of rows with a vector, inserts of chunks carrying one) pass
* CHUNK_VECTOR_WRITE_BUDGET, so the publication leaves the replacement to `continueDeferredProjection`.
*/
-export function planImportChunkWrites(fresh: readonly ChunkInput[], kept: ReadonlyMap<number, { id: number; chunk_index: number; vector: boolean }>, bounded: boolean):
+export function planImportChunkWrites(fresh: readonly ChunkInput[],
+ kept: ReadonlyMap<number, { id: number; chunk_index: number; vector: boolean; projected_byte_start?: number | null; projected_byte_end?: number | null }>, bounded: boolean):
{ waves: ChunkMove[][]; defer: boolean } {
- const moves = [...kept].filter(([i, row]) => row.chunk_index !== fresh[i]!.chunk_index).map(([i, row]) => ({ id: row.id, from: row.chunk_index,
+ const move = (i: number, row: { id: number; chunk_index: number; vector: boolean }): ChunkMove => ({ id: row.id, from: row.chunk_index,
to: fresh[i]!.chunk_index, projected_byte_start: fresh[i]!.projected_byte_start ?? null, projected_byte_end: fresh[i]!.projected_byte_end ?? null,
- ...(row.vector ? {} : { vectorless: true }) }));
+ ...(row.vector ? {} : { vectorless: true }) });
+ const moves = [...kept].filter(([i, row]) => row.chunk_index !== fresh[i]!.chunk_index).map(([i, row]) => move(i, row));
+ // #6961: a row kept at its own index whose chunk's span moved (earlier text grew) takes the new span in a last wave;
+ // a span-only update changes no indexed column, so it adds no HNSW entry and is not counted against the budget.
+ const respans = [...kept].filter(([i, row]) => row.chunk_index === fresh[i]!.chunk_index && row.projected_byte_start !== undefined
+ && (row.projected_byte_start !== (fresh[i]!.projected_byte_start ?? null) || row.projected_byte_end !== (fresh[i]!.projected_byte_end ?? null))).map(([i, row]) => move(i, row));
const vectorInserts = fresh.filter((chunk, i) => !kept.has(i) && (chunk.embedding || chunk.embedding_image)).length;
const plan = planMoveWaves(moves, new Map([...kept.values()].map(row => [row.chunk_index, row.id])),
bounded ? Math.max(0, CHUNK_VECTOR_WRITE_BUDGET - vectorInserts) : Number.POSITIVE_INFINITY);
- return { waves: plan.waves, defer: bounded && fresh.length > 0 && (!plan.done || vectorInserts > CHUNK_VECTOR_WRITE_BUDGET) };
+ return { waves: respans.length ? [...plan.waves, respans] : plan.waves,
+ defer: bounded && fresh.length > 0 && (!plan.done || vectorInserts > CHUNK_VECTOR_WRITE_BUDGET) };
}
/** One UPDATE per wave; a landed row takes its new chunk's span, a parked row keeps its own until it lands. */
diff --git a/test/chunk-moves.test.ts b/test/chunk-moves.test.ts
index fcdb557ddc..e33b98283e 100644
--- a/test/chunk-moves.test.ts
+++ b/test/chunk-moves.test.ts
@@ -11,7 +11,8 @@
* 3. Pure; no engine.
*/
import { describe, expect, test } from 'bun:test';
-import { planMoveWaves, type ChunkMove } from '../src/core/page-state/chunk-moves.ts';
+import { CHUNK_VECTOR_WRITE_BUDGET, planImportChunkWrites, planMoveWaves, type ChunkMove } from '../src/core/page-state/chunk-moves.ts';
+import type { ChunkInput } from '../src/core/types.ts';
/** Applies the waves to `at` (index -> id), checking each wave lands only on free indexes. */
function replay(at: Map<number, number>, waves: ChunkMove[][]): Map<number, number> {
@@ -77,3 +78,19 @@ describe('planMoveWaves', () => {
expect([...replay(swap, parked.waves)].sort((a, b) => a[0] - b[0])).toEqual([[0, 11], [1, 10]]);
});
});
+
+describe('planImportChunkWrites', () => {
+ // #6961: a row kept at its own index whose chunk shifted (earlier text grew) takes the new span; the update changes no
+ // indexed column, so more of them than the budget still leave a bounded publication undeferred.
+ test('a row kept at its own index takes its chunk span in a last wave, uncounted against the budget', () => {
+ const count = CHUNK_VECTOR_WRITE_BUDGET + 2;
+ const fresh = Array.from({ length: count }, (_, i) => ({ chunk_index: i, chunk_text: `c${i}`, chunk_source: 'compiled_truth',
+ projected_byte_start: 100 + i * 10, projected_byte_end: 109 + i * 10 })) as ChunkInput[];
+ const kept = new Map(fresh.map((_, i) => [i, { id: 50 + i, chunk_index: i, vector: true,
+ projected_byte_start: i === 0 ? 100 : 90 + i * 10, projected_byte_end: i === 0 ? 109 : 99 + i * 10 }]));
+ const plan = planImportChunkWrites(fresh, kept, true);
+ expect(plan.defer).toBe(false);
+ expect(plan.waves).toEqual([fresh.slice(1).map((chunk, i) => ({ id: 51 + i, from: i + 1, to: i + 1,
+ projected_byte_start: chunk.projected_byte_start, projected_byte_end: chunk.projected_byte_end }))]);
+ });
+});
diff --git a/test/fenced-embed-reuse-6961.test.ts b/test/fenced-embed-reuse-6961.test.ts
index 061cbfa5f2..83ce88be96 100644
--- a/test/fenced-embed-reuse-6961.test.ts
+++ b/test/fenced-embed-reuse-6961.test.ts
@@ -33,6 +33,8 @@ const FENCE = (rows: string[]) => ['## Facts', '<!--- gbrain:facts:begin -->',
...rows.map((claim, i) => `| ${i + 1} | ${claim} | fact | 1.0 | world | medium | 2017-01-01 | | chat | |`), '<!--- gbrain:facts:end -->', ''].join('\n');
const PERSON = (facts: string[]) => `---\ntitle: Alice Example\ntype: person\n---\n# Alice Example\n\n${
['alpha', 'bravo', 'charlie', 'delta'].map(w => `## Section ${w}\n\n${paragraph(w)}`).join('\n\n')}\n\n${FENCE(facts)}`;
+const MIDDLE = (facts: string[]) => `---\ntitle: Alice Example\ntype: person\n---\n# Alice Example\n\n## Section alpha\n\n${paragraph('alpha')}\n\n${
+ FENCE(facts)}\n${['bravo', 'charlie', 'delta'].map(w => `## Section ${w}\n\n${paragraph(w)}`).join('\n\n')}\n`;
const FACTS = ['Alice Example works at Acme Example', 'Alice Example lives in Springfield'];
const APPENDED = 'Alice Example joined the board of Fund A';
@@ -64,6 +66,22 @@ for (const backend of testBackends()) describe(`#6961 fenced page vector reuse (
expect((await engine.getChunks(slug, { includeEmbedding: true, includeUnsealed: true })).every(chunk => chunk.embedding)).toBe(true);
}, 120_000);
+ // Value: protects=a kept row after a mid-page fact append carries its chunk's new byte span, so chunk-centered reads
+ // still locate it; fails_when=a kept row at an unchanged index keeps the span of the previous body; why_new=fenced
+ // pages rewrote every row before #6961, so their spans were always fresh; seam=a fresh import of the same body
+ test('a mid-page fact append keeps every chunk span current', async () => {
+ const slug = 'people/alice-example-middle';
+ const spans = async (s: string) => (await engine.getChunks(s, { includeUnsealed: true }))
+ .map(chunk => [chunk.chunk_index, chunk.projected_byte_start ?? null, chunk.projected_byte_end ?? null]);
+ await importFromContent(engine, slug, MIDDLE(FACTS));
+ const total = await chunkCount(slug);
+ embedded = [];
+ await importFromContent(engine, slug, MIDDLE([...FACTS, APPENDED]));
+ expect(embedded.length).toBeLessThan(total);
+ await importFromContent(engine, 'people/alice-example-middle-fresh', MIDDLE([...FACTS, APPENDED]));
+ expect(await spans(slug)).toEqual(await spans('people/alice-example-middle-fresh'));
+ }, 120_000);
+
// Value: protects=a fenced page never reuses a vector whose input is unproven (no recorded embedding input), so no
// reused vector can carry text the sanitized chunk lacks; fails_when=the legacy raw-text branch (tier none, no stored mode)
// admits it on a fenced page; why_new=fenced pages reused nothing before #6961; seam=embedding_input_hash and the page mode cleared in SQL (a pre-provenance page)