Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(import): importFromContent keeps its body and awaits it in the engine scope, so callers' async frames stay on the stack (#8523 r3)

garrytan380a74048138
diff --git a/scripts/function-size-baseline.tsv b/scripts/function-size-baseline.tsv
index 188a3bf0ac..5b7bba68c9 100644
--- a/scripts/function-size-baseline.tsv
+++ b/scripts/function-size-baseline.tsv
@@ -38,7 +38,8 @@ src/core/cycle/synthesize-concepts.ts runPhaseSynthesizeConcepts 498 seed: W5 ba
src/core/cycle/synthesize.ts runPhaseSynthesizeInner 1057 +1 #7805: the session-source routing parameter the discovery call site fills (logic in cycle/synthesize-source-routing.ts); +1 #5575 I2: output refs take their transcript taint before the provenance stamp (the stamp line stays the signal-guarded literal cycle-abort pins); seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +3 #5854: publish_pending from postprocess, a pending summary publish deferred instead of failing the phase; +1 fix wave 11 #6160: oneshot_task_prefixes on the child job data; +6 GBRA-161 PR 4 (#7065): the synthesis generation activation after postprocess (logic in cycle/synthesize-generations.ts), its file holds and details.synthesis_generations; +1 GBRA-161 PR 5 (S-2): the summary revision read before the children are submitted; +1 GBRA-60: a managed child carries dream_output, so it writes each page verified and stamped (logic in cycle/synthesize-child-verify.ts)
src/core/embed-stale.ts embedStaleForSource 302 PR 4 E round 2 (gbrain-evals#180): the page embedding lease gives up at the drain deadline with a busy lease (page skipped) and is renewed inside the install transaction; lease logic lives in core/page-embedding-lease.ts
src/core/facts/backstop.ts runPipelineBodyInner 358 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1)
-src/core/import-file.ts importScreenedContent 583 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +2 wave 8 lane D1 #5158: key-order fallback equality and legacy-hash ordering; lowered to 807 (#5988: screen and sanity config peeled to core/import-screen.ts); +1 #6007: the prepared apply reports its written page and seal (PreparedImportApplied) so put_page skips a re-read and a second seal; +1 #5333: embed-skip transition defeats the unchanged fast paths (the stale-marker clear stays inside the sanity block); +3 #6188: Tier 1 fence step in place before the hidden-row merge (logic in fence-repair/import-step.ts); -146 #5575: content-sanity/write-gate disposition moved to import-screen.ts settleContentDisposition; +4 TODO: large-pages A6/T33: a page over PAGE_INLINE_MAX_BYTES commits with NULL vectors and embeds through the metered stale sweep; +1 #6722: an embed_skip page is chunked and never embedded (noEmbed folds the marker); +16 Wave B lane 1d (TODO wave-b PR 1): a body above the inline threshold takes the segmented stage and generation flip at the storage step (logic in persistence/segmented-apply.ts); +3 Wave B review: reuseSegments option (a rename re-preparation reuses the staged segmented body) and the remote fence merge over fenceBody; +2 Wave B gem A: a coordinated demotion versions the segmented page before demoting it; renamed from importFromContent and lowered to 583 (#8523 r2: the slug check, content screen and engine scope entry stay in importFromContent, which calls this body)
+src/core/import-file.ts importFromContent 694 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +2 wave 8 lane D1 #5158: key-order fallback equality and legacy-hash ordering; lowered to 807 (#5988: screen and sanity config peeled to core/import-screen.ts); +1 #6007: the prepared apply reports its written page and seal (PreparedImportApplied) so put_page skips a re-read and a second seal; +1 #5333: embed-skip transition defeats the unchanged fast paths (the stale-marker clear stays inside the sanity block); +3 #6188: Tier 1 fence step in place before the hidden-row merge (logic in fence-repair/import-step.ts); -146 #5575: content-sanity/write-gate disposition moved to import-screen.ts settleContentDisposition; +4 TODO: large-pages A6/T33: a page over PAGE_INLINE_MAX_BYTES commits with NULL vectors and embeds through the metered stale sweep; +1 #6722: an embed_skip page is chunked and never embedded (noEmbed folds the marker); +16 Wave B lane 1d (TODO wave-b PR 1): a body above the inline threshold takes the segmented stage and generation flip at the storage step (logic in persistence/segmented-apply.ts); +3 Wave B review: reuseSegments option (a rename re-preparation reuses the staged segmented body) and the remote fence merge over fenceBody; +2 Wave B gem A: a coordinated demotion versions the segmented page before demoting it; +2 #8523 B3a: the screened body runs as the importScreened closure, entered in an embedded engine's scope after the engine-free screen and the source check
+src/core/import-file.ts importFromContent>importScreened 569 #8523 B3a: importFromContent's body after the engine-free screen and the source check, run in an embedded engine's scope (seed: W5 baseline, refactor wave 1; decomposition deferred to wave 2, plan D1)
src/core/minions/handlers/subagent-oneshot.ts runSubagentOneshot 378 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +1 fix wave 9 lane B #4921: the oneshot sub-budget is passed as the call timeoutMs so the 300 s gateway backstop cannot cut it
src/core/minions/handlers/subagent.ts makeSubagentHandler 890 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +3 fix wave 9 lane B #4921: per-turn timeout read once, passed to oneshot and gateway loop; +2 GBRA-60: a trusted dream child gets the put_pages batch tool (buildBrainPutPagesTool) from the oneshot tool options; +1 #7850: a dream synthesis child re-checks its session source before any model call (one call line; the check is refuseChangedSessionSource in cycle/synthesize-source-routing.ts)
src/core/minions/handlers/subagent.ts makeSubagentHandler>subagentHandlerInner 820 seed: W5 baseline, refactor wave 1 (decomposition deferred to wave 2, plan D1); +3 fix wave 9 lane B #4921: per-turn timeout read once, passed to oneshot and gateway loop; +2 GBRA-60: a trusted dream child gets the put_pages batch tool (buildBrainPutPagesTool) from the oneshot tool options; +1 #7850: a dream synthesis child re-checks its session source before any model call (one call line; the check is refuseChangedSessionSource in cycle/synthesize-source-routing.ts)
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index f1dd37db09..8df57cbb79 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -17,7 +17,7 @@ src/core/cycle.ts 3345 ratchet +2 GBRA-130 C-2 (#6946): the sync phase reads an
src/core/cycle/synthesize.ts 3244 ratchet +3 #7805: each captured session synthesizes only into its own frozen source (routing import, the outer wrapper, the routing parameter; logic in cycle/synthesize-source-routing.ts); +26 System One v1 S7/S8: triage and grounding decide delegation (triage-decide.ts, grounding-decide.ts); merged waves; grown v0.46.26.0 megawave + master rolling lease via shared throttled renewer; grown wave-g (#4506): cycle summaries can stay out of the source repo; +18: expired-verdict sweep + DeepSeek judge thinking pin (#4069); +47: #4077 cooperative-abort threading; +110: eval fix wave F6 spend surfacing (withChatPhase wrap, triage/child token telemetry, zero-page counter); +32: F1b quote-verify call site + kill switch; +2: F1a/F3/F4a OUTPUT POLICY rules; +59: F2 verified-segment rescue (gate at report construction, rescue knobs, telemetry); +19: rubric v2 peak-scoring + docblock rescue branch; +50: #4337 summary link-sample cap + dream_created_cycle_date preserve-on-rerun; +5: #4348 resolveCycleDate wiring (local-day summary bucketing); +3 round-3 rebase true-up; +55: #4805 synth-v2 completion skip (findSynthV2Completion + prefix-parameterized loadSuccessfulSynthesisKeys); +10: #4886 mode-aware buildSynthesisPrompt (oneshot children drop tool-use/final-prose guidance); #5105: small legacy writer/refusal and shared revision CAS delegates; +4: merge of #5666 durable synthesis completions with the v0.59.18.0 verify-epoch wiring; +7 #5413/#5471 discovery exclusions (self-capture session ids, dream output directories); +6 fix wave 2 (v0.60.5.0) integration: W7 paid-loop breaker check before submission and released-key daily-cap count on top of the W1b discovery exclusions; +6 v0.60.5.0 master-merge true-up: the wave-2 growth (+13 W1b, +6 W7) re-summed over master v0.59.18.0 at 3222 lines with no slack; +76 fix wave 3: per-run USD gate + fail-closed daily cap (C-13), child first-write ownership for stamp/reverse-write (C-8, #5685), cycle-date prompt hint (C-15), opt-in attribution rule (#5425); fix wave 3 x fix wave 2 (v0.60.5.0) composition: +19 at the merge; +2 #5806 triageNow clock seam forwarded to triage (interface field + call-site option); +4 #5854: a postprocess or summary publish still pending after its wait is deferred (publish_pending warn, no cooldown stamp; helper in synthesize-postprocess.ts); +14 #5490 review: a triage verdict a chat_fallback_chain model gave is used for the run but not cached (TriageResult.answeredBy, judge-client marker, guarded putDreamVerdict); Foundations 2 Lane C: maintenanceTransaction import (attributed summary page); +14 #5490 review: a triage verdict a chat_fallback_chain model gave is used for the run but not cached (TriageResult.answeredBy, judge-client marker, guarded putDreamVerdict); Foundations 2 Lane C: maintenanceTransaction import (attributed summary page); +14 #5490 review: a triage verdict a chat_fallback_chain model gave is used for the run but not cached (TriageResult.answeredBy, judge-client marker, guarded putDreamVerdict); lowered to 3200 by wave 9 lane W #5145: completion-key helpers (findLegacyCompletion, findSynthV2Completion, loadSuccessfulSynthesisKeys) peeled to synthesize-completion.ts with the new pre-triage partition; +13 P2 extraction.date_grounding: transcript observation date line instead of a run-date "Today's date" (config field + prompt arg); +1 fix wave 11 #6160: oneshot_task_prefixes on the child job data; +18 fix wave 12 #6069: unreliable-verdict backoff in runTriagePass (report code, pass backoff count, triage details entry; marker logic in triage-backoff.ts); +1 fix wave 12 #6242: SYNTH_PHASE_FAIL keeps an OperationError's suggestion and code (OperationError import); +3 wave 17 #6616: slug discovery and postprocessing share the canonical committed child outputs (loadCommittedSynthesisOutputs in synthesize-postprocess.ts; one call-site line, one parameter, one lookup line); lowered to 3224 by wave 17 #6642: the summary provenance section moved to summaryProvenanceLines in dream-provenance.ts; +6 GBRA-130 B-6 (#6860): surrogate-safe triage cuts (truncateUtf8) and the per-transcript verdict-cache write call site (logic in triage-backoff.ts VerdictCacheWrites); +2 GBRA-161 PR 4 (#7065): the synthesize phase activates each synthesized transcript's outputs as its generation (activateSynthesisGenerations import and call site; logic in cycle/synthesize-generations.ts); +8 GBRA-161 PR 5 (S-2): the dream summary publishes as a single-item dream_summary generation bound to its pre-run revision (publishSinglePage with a generation_fenced skip); +1 GBRA-60: a managed synthesis child carries dream_output (verify and stamp before put_page; cycle/synthesize-child-verify.ts)
src/core/cycle/extract-atoms.ts 1658 ratchet crossed the 1500 unlisted cap in the atoms/extraction fix wave: +21 embed-aware cost gate (#4878), +65 atoms-array anchor scan (#4913); peel candidate: the parse block (parseAtomsOutcome / parseArrayAtOffset / atomsFromParsedArray) into extract-atoms-parse.ts; +35: #4853 transcript-origin atoms link to their conversation page; +130: #4916 transcript failure-count/tombstone (extract_atoms_transcript_state, TODOS P2(b)); wave composition: #4916 transcript-state skip + #4948 zero-atom prompt + #4943 meetings source-id summed; +3: wave review — a later `[]` is an honest zero-yield, a non-atom-shaped array is a counted failure; +13: wave review — explicit budget keeps the cap over an unpriced embed route ($0-priced); +14: wave review — transcript-state write errors surfaced, missing table warns once; merged reliability/shared-skills source integration; +37 fix wave 3: stable undated atom dates + stale-atom retirement (C-14); +3 #5525: effectiveVisibility stamped on non-managed atoms; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +3 at the merge; +6 fix wave 3 lane A (#5601): accepted-pending atom batches count as progress (helper in persistence/accepted-pending.ts); +4 fix wave 4: #5705 transcript-as-data system prompt lines and its message helper, #5627 response schema import (the atom-type list moved to extract-atoms-schema.ts); +30 #5809/#5832: drain hard/soft stop signals (ExtractAtomsOpts.signal/stopSignal, pre-commit abort checks, no strike or post-stop writes after a hard stop) and the cancellable backlog count; +1 #5812: transcript items are paste-stripped (corpusTextForExtraction import) before the input cut; +1 #5820: extract_atoms discovery passes selfCaptureSessionIds like synthesize; +10 #5856/#5854: drain attempt state (one BudgetTracker + one publish wait across batches) and the opt-in connector email/meeting exclusion (predicate in connector-atoms.ts); +1 Foundations 2 Lane C: maintenanceTransaction import (attributed stale-atom retirement); +5 fix wave 9 lane P (#5041): opt-in atom write-through hook and retired-atom slugs returned for file removal (logic in derived-write-through.ts); the parse block (parseAtomsOutcome / parseArrayAtOffset / atomsFromParsedArray) is peeled into extract-atoms-parse.ts (-193, fix wave 10 lane B); +35: #4853 transcript-origin atoms link to their conversation page; +130: #4916 transcript failure-count/tombstone (extract_atoms_transcript_state, TODOS P2(b)); wave composition: #4916 transcript-state skip + #4948 zero-atom prompt + #4943 meetings source-id summed; +3: wave review — a later `[]` is an honest zero-yield, a non-atom-shaped array is a counted failure; +13: wave review — explicit budget keeps the cap over an unpriced embed route ($0-priced); +14: wave review — transcript-state write errors surfaced, missing table warns once; merged reliability/shared-skills source integration; +37 fix wave 3: stable undated atom dates + stale-atom retirement (C-14); +3 #5525: effectiveVisibility stamped on non-managed atoms; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +3 at the merge; +6 fix wave 3 lane A (#5601): accepted-pending atom batches count as progress (helper in persistence/accepted-pending.ts); +4 fix wave 4: #5705 transcript-as-data system prompt lines and its message helper, #5627 response schema import (the atom-type list moved to extract-atoms-schema.ts); +30 #5809/#5832: drain hard/soft stop signals (ExtractAtomsOpts.signal/stopSignal, pre-commit abort checks, no strike or post-stop writes after a hard stop) and the cancellable backlog count; +1 #5812: transcript items are paste-stripped (corpusTextForExtraction import) before the input cut; +1 #5820: extract_atoms discovery passes selfCaptureSessionIds like synthesize; +10 #5856/#5854: drain attempt state (one BudgetTracker + one publish wait across batches) and the opt-in connector email/meeting exclusion (predicate in connector-atoms.ts); +1 Foundations 2 Lane C: maintenanceTransaction import (attributed stale-atom retirement); +5 fix wave 9 lane P (#5041): opt-in atom write-through hook and retired-atom slugs returned for file removal (logic in derived-write-through.ts); +17 #5031: same-page case-insensitive atom adoption fallback (one page-atom scan per source page per run); +12 P2 atomsPrompt helper: extraction.date_grounding source observation date + rule in the atoms prompt; +4 GBRA-75: octet_length-bounded body-length predicate (minCompiledTruthChars) for discovery and the backlog count; GBRA-130 A-42 (#6804): dot-led atom stems and the legacy-slug adoption guard
src/core/engine.ts 2866 ratchet grown v0.46.26.0 megawave: relational/visibility/usage contract surface (#4352 #4218); grown wave-g (#4524 #4527): orphans-mode contract + removeLink deleted count; wave-k: +4 fail-closed TrajectoryOpts.remote contract doc (#3855), +2 FactListOpts.unconsolidatedOnly (#4070), +14 getAllConfig bulk-read contract (#3980); +1: getBacklinkCounts page-id contract doc (#4380); +9: FactListOpts.grep SQL pre-limit contract (#3851); +13: dream-verdict TTL contract (#4069); +33: softDeletePages batch soft-delete contract (#4587); +8: #4592 scoped getStats/getHealth contract doc; +5 read-time TTL validity contract docs (active vs history read semantics); +12: #4280 findOrphanPages type/quarantined metadata contract; +14: traversePathsDetailed row-cap contract (ship-review); +12: resolveSlugWithAliasDetailed owning-source contract (ship-review) | merged v0.48.1.0: master #4788 + wave traversePathsDetailed/resolveSlugWithAliasDetailed; master: | merged v0.48.1.0 true-up; corrective release: read-policy signatures and Chronicle scope; +7: #4882 listFactsSince entitySlug/sessionId composition doc; #5105: canonical page state shared module and transaction/snapshot delegates; #5105: guarded derived projection preparation and read seals; #5105: resident-stop registration and direct transaction contract; #5105: +1 raw preservation contract for legacy engine migration; merged company ingestion/schema routing changes; safety wave: fact embedding model and text identity contract; +5 write-path wave: findDuplicatePage excludeSlug + id-first ranking contract; +2 fw3 A14: addTag tagSource provenance contract; evidence delivery: getChunkWindows interface member + type import (+3); #5761: +4 markPagesAttendanceBlocked member + countStalePagesForExtraction attendance option; +4 #5888: FactListOpts.fingerprint + FactRow.fact_fingerprint contract (hot-memory duplicate collapse); +1 GBRA-49: LinkReadScope temporal opts on the link read contract; +4 #5984: upsertChunks sealChunkerVersion and createVersion preimage contract docs (round-trip diet); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2683 measured); +10 P7 relationalChainHop member + ChainHopOpts/ChainHopEdge imports (SQL in search/read-enrichment.ts); +14 contributor audit wave P0: listFactsKeyset member (delta facts keyset read contract) + FactRow.created_at_iso; +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); 3 P2 speaker attribution: FactAttribution type + FactRow/NewFact.attributed_to fields + findCandidateDuplicates attributedTo option contract; +4 #5984 lane apply diet: setPageAliases inline option and upsertChunks pageId contract docs; +10 #6278: executeRaw timeoutMs contract (transaction-local statement_timeout for bounded preparation reads); +2 GBRA-69: replaceDerivedLinksBatch member; master v0.60.136.0 merge into #5575 memory trust (both sides kept); +6 GBRA-67: readPageSnapshotsBatch interface member (BrainEngine signatures cannot move); +1 GBRA-75 wave 10: optional findDuplicatePages member (batched findDuplicatePage, contract in engine-sql/pages.ts); +4 FactListOpts/TrajectoryOpts.excludePrivate fact-provenance visibility contract (interface fields cannot move); +18 GBRA-76: TransactionOptions (abort signal) interface and the transaction/transactionDirect signatures (abort logic in postgres-engine/transaction-abort.ts); +2 W14 P3.11 (#4381): LinkBatchInput.resolution_type on top of W13; +1 C2: findCandidateDuplicates arm:'keyword' contract doc (facts.candidate_fusion); re-summed at the master 6ccf30c3e merge (2708 integration + 2707 master - 2707 base); +5 Cat 40 Hard fix wave F2 (#6271): countKeywordPages/searchKeywordPages BrainEngine signatures (SQL in engine-sql/keyword-pages.ts); +6 W15 P1.2 (#6454): countStaleFrontmatterKeyPages/listStaleFrontmatterKeyPages/stripStaleFrontmatterKeys contract (SQL in engine-sql/pages.ts); +2 large pages Wave A sub-a: getChunkPage signature (get_chunks paging; SQL in engine-sql/chunks.ts); +5 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows signatures + type import (SQL in engine-sql/chunks.ts); +2 Cat 40 next wave part 1: non-ranking link types on the relationalFanout/relationalChainHop/getBacklinkCounts/getAdjacencyBoosts contract docs (policy in link-ranking-policy.ts; re-summed at the 0.60.242.0 merge: master 2794 + 2); +26 Wave B PR 1: segmented page storage members (allocateBodyOccurrenceIds..sweepSegments) + page-body type import (SQL in engine-sql/page-body.ts); merge v0.60.248.0: Wave B PR 1 and master growth compose at 2822; +1 LHF wave 2 item 1: executeRaw opts gain jitOff/prepare (the JIT-off typed-search type check; logic in ops/search-type-filter.ts); +9 PR2 fix lane X: BrainEngine.keywordBoundedRead, one unscoped read under the keyword statement bound for the segmented-region read (interface member and its doc cannot move): 2823 + 9; +9 GBRA-177: TakesListOpts.oldestFirst/sinceOnOrBefore/ungradedFirst for the grade_takes window (interface fields cannot move; SQL in engine-sql/takes.ts): 2832 + 9; +6 GBRA-156: TransactionOptions.pipelineFirst (the declaration goes out with BEGIN behind a transaction-mode pooler; predicate in persistence/protocol.ts); interface fields cannot move; +1 GBRA-161 PR 2: deprecation notes on insertFacts({ deleteForPageFirst }), deleteFactsForPage and expireFact (interface docs cannot move); +7 GBRA-156: TransactionOptions.pipelineEarly (search read bodies go out behind BEGIN; predicate in postgres-engine/search-settings.ts); interface fields cannot move; +2 GBRA-156: TransactionOptions.readOnly (search read transactions open BEGIN READ ONLY; 2858 measured); grown v0.46.26.0 megawave: relational/visibility/usage contract surface (#4352 #4218); re-summed at the Wave B PR 4 merge with v0.60.452.0 (2866 measured)
-src/core/import-file.ts 2094 ratchet +2 #7880: importFromContent's holdNoncanonicalPrivate option (interface field; the hold is screenImportContent's); +2 #7848: importFromFile holds a non-canonical private request (predicate and hold in persistence/visibility-request.ts); +2 W1 belief ledger: the legacy import is an unmanaged projection point (logic in belief/projection-point.ts); +2: pre-publication withdrawal validation; grown v0.46.26.0 megawave: OCR budget + image import + embed-retry rewire (#3973 #3374); grown wave-g (#4530): per-model chunk-token cap wiring; +24 #4548: row-level visibility-aware fence merge on remote write-back (replaces the #2044 whole-block swap + #4553/#4555 warn-only paths); +7 #4546: merge extended to the timeline-embedded fence; +5 #4107: maybeOcr gates on the OCR model's provider; +3 #3893; +7: #4702 disabled_patterns pass-through to the sanity gate; composition true-up (fix-wave multi-train); +8 #4903: importCodeFile writes source_path so the full-sync reconcile can see code pages; -11: wave review — local fence append replaced by facts-fence.replaceOrInsertFactsFence (#4756 rule); security: durable queued authority and confined filesystem dispatch; master integration: grown v0.46.26.0 megawave: OCR budget + image import + embed-retry rewire (#3973 #3374); grown wave-g (#4530): per-model chunk-token cap wiring; +24 #4548: row-level visibility-aware fence merge on remote write-back (replaces the #2044 whole-block swap + #4553/#4555 warn-only paths); +7 #4546: merge extended to the timeline-embedded fence; +5 #4107: maybeOcr gates on the OCR model's provider; +3 #3893; +7: #4702 disabled_patterns pass-through to the sanity gate; composition true-up (fix-wave multi-train); +8 #4903: importCodeFile writes source_path so the full-sync reconcile can see code pages; -11: wave review — local fence append replaced by facts-fence.replaceOrInsertFactsFence (#4756 rule); harness support: boundary wiring to shared grants, isolated setup, withdrawal and delegated guards; merged v0.49: withdrawal preservation after data-only frontmatter parsing; #5105: small legacy writer/refusal and shared revision CAS delegates; #5105: +8 prepared contextual retrieval policy and retry boundary wiring; upstream integration: preserve independently reviewed master additions with branch facade extraction; #5252 wave: shared provider-free Markdown and code preparation; merged company ingestion/schema routing changes; master retrieval/projection integration with company ingestion; shared content: prepare managed image imports without bypassing canonical publication; +29 write-path wave: identity move/collision wiring (logic in import-identity.ts), deferred embedding on provider failure, image index completeness, file-time date anchor; +22 fw3 A13: markdown re-import reuses unchanged chunk vectors under the seal/wrapper/signature/protected-fence gate; +2 fw3 A12: importFromFile firstCommitAt option; +3 fw3 A14: managed imports keep add-only tags; +4 #5675: record the file origin move inference binds to (logic in import-identity.ts); fw3 integration: A12/A13/A14 (+27) and #5675 (+4) compose; +17 wave-2 W4: safe-chunk re-seal of unchanged legacy markdown/code pages (#5050 #5247) and embedding-input provenance on inline imports (#5553); -1 fix wave 2 (v0.60.5.0) integration: W4 delta re-summed over the merged master line count; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +16 at the merge; +11 #4616: markdown and code imports keep the usable vectors of a partially refused batch and leave refused chunks for embed --stale; lowered to 2230: #5621 contextual-mode resolution peeled to core/import-contextual-mode.ts (shared with gbrain repair contextual-mode); +2 wave 8 lane D1 #5158: key-order fallback equality call (comparison lives in page-state/import-guard.ts); +1 F1c: direct imports run in maintenanceTransaction (write attribution; helper in persistence/attribution.ts); lowered to 2178: #5988 content screen and content-sanity config peeled to core/import-screen.ts; +1 #6007: the prepared apply returns its page id and seal (PreparedImportApplied); +1 #5333: trusted imports drop a stale oversized embed_skip/content_flag the gate no longer sets, and an embed_skip transition defeats both unchanged fast paths; wave 10 collector: lane E (#5333) and the other lanes compose at 2183; grown #6188 (+17): fence options interface, result fields and the Tier 1 step call (logic in fence-repair/import-step.ts); +2 #5984 single-write fast path: the coordinated existingSnapshot option (interface field) composes with the lane apply diet at 2202; +1 GBRA-69: a coordinated apply defers its chunker seal to the caller (sealImportedPage in page-state/projections.ts); lowered to 2023: #5001 the edge-mapping loop moved to code-chunks.ts mapCodeEdges (shared with the prepared apply); W14 P3.13 (#5001) lowered 2049→2023 by moving the edge loop to mapCodeEdges; +2 W13 PR1 on top (2025 measured); +2 #6529: the unchanged-image skip path records source_path (refreshSourcePath, as #4588 does for Markdown); +4 large pages A6/T33: code files keep CODE_FILE_MAX_BYTES, a page over PAGE_INLINE_MAX_BYTES commits with NULL vectors for the metered stale sweep; +1 #6722: an embed_skip page is chunked and never embedded (noEmbed folds the marker); merge v0.60.240.0: large pages and master growth compose at 2032; +24 Wave B lane 1d: the segmented storage step (stage before publication, flip in the apply, seal and post-commit projection; logic in persistence/segmented-apply.ts and page-body/ingest.ts) and the streamed-import dispatch over the single-string ceiling (logic in page-body/streamed-import.ts); merge v0.60.248.0: Wave B PR 1 and master growth compose at 2056; +4 Wave B review: a rename re-preparation reuses the staged segmented body (reuseSegments option) and the remote fence merge reads an unread body as fence-free (fenceBody in page-body/page-shape.ts); +2 Wave B gem A: a coordinated demotion versions the segmented page before demoting it (createVersion order); lanes compose at 2062; +2 Wave B lane 2q: ImportResult.searchable (interface field; the value comes from SegmentedImport.afterCommit);+2: pre-publication withdrawal validation; +9 #7172: kept chunk rows move to their new index and a bounded managed publication defers its chunk replacement (logic in page-state/chunk-moves.ts and page-state/projections.ts continueDeferredProjection); merge v0.60.332.0: #7172 and master growth compose at 2075; +5 GBRA-283 B3a: the four import entry points re-enter an embedded engine scope (needsEngineScope in core/engine-scope.ts) and its import; +10 #8523 r2: importFromContent screens before it enters the engine scope and re-enters through importScreenedContent (no self-call)
+src/core/import-file.ts 2086 ratchet +2 #7880: importFromContent's holdNoncanonicalPrivate option (interface field; the hold is screenImportContent's); +2 #7848: importFromFile holds a non-canonical private request (predicate and hold in persistence/visibility-request.ts); +2 W1 belief ledger: the legacy import is an unmanaged projection point (logic in belief/projection-point.ts); +2: pre-publication withdrawal validation; grown v0.46.26.0 megawave: OCR budget + image import + embed-retry rewire (#3973 #3374); grown wave-g (#4530): per-model chunk-token cap wiring; +24 #4548: row-level visibility-aware fence merge on remote write-back (replaces the #2044 whole-block swap + #4553/#4555 warn-only paths); +7 #4546: merge extended to the timeline-embedded fence; +5 #4107: maybeOcr gates on the OCR model's provider; +3 #3893; +7: #4702 disabled_patterns pass-through to the sanity gate; composition true-up (fix-wave multi-train); +8 #4903: importCodeFile writes source_path so the full-sync reconcile can see code pages; -11: wave review — local fence append replaced by facts-fence.replaceOrInsertFactsFence (#4756 rule); security: durable queued authority and confined filesystem dispatch; master integration: grown v0.46.26.0 megawave: OCR budget + image import + embed-retry rewire (#3973 #3374); grown wave-g (#4530): per-model chunk-token cap wiring; +24 #4548: row-level visibility-aware fence merge on remote write-back (replaces the #2044 whole-block swap + #4553/#4555 warn-only paths); +7 #4546: merge extended to the timeline-embedded fence; +5 #4107: maybeOcr gates on the OCR model's provider; +3 #3893; +7: #4702 disabled_patterns pass-through to the sanity gate; composition true-up (fix-wave multi-train); +8 #4903: importCodeFile writes source_path so the full-sync reconcile can see code pages; -11: wave review — local fence append replaced by facts-fence.replaceOrInsertFactsFence (#4756 rule); harness support: boundary wiring to shared grants, isolated setup, withdrawal and delegated guards; merged v0.49: withdrawal preservation after data-only frontmatter parsing; #5105: small legacy writer/refusal and shared revision CAS delegates; #5105: +8 prepared contextual retrieval policy and retry boundary wiring; upstream integration: preserve independently reviewed master additions with branch facade extraction; #5252 wave: shared provider-free Markdown and code preparation; merged company ingestion/schema routing changes; master retrieval/projection integration with company ingestion; shared content: prepare managed image imports without bypassing canonical publication; +29 write-path wave: identity move/collision wiring (logic in import-identity.ts), deferred embedding on provider failure, image index completeness, file-time date anchor; +22 fw3 A13: markdown re-import reuses unchanged chunk vectors under the seal/wrapper/signature/protected-fence gate; +2 fw3 A12: importFromFile firstCommitAt option; +3 fw3 A14: managed imports keep add-only tags; +4 #5675: record the file origin move inference binds to (logic in import-identity.ts); fw3 integration: A12/A13/A14 (+27) and #5675 (+4) compose; +17 wave-2 W4: safe-chunk re-seal of unchanged legacy markdown/code pages (#5050 #5247) and embedding-input provenance on inline imports (#5553); -1 fix wave 2 (v0.60.5.0) integration: W4 delta re-summed over the merged master line count; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +16 at the merge; +11 #4616: markdown and code imports keep the usable vectors of a partially refused batch and leave refused chunks for embed --stale; lowered to 2230: #5621 contextual-mode resolution peeled to core/import-contextual-mode.ts (shared with gbrain repair contextual-mode); +2 wave 8 lane D1 #5158: key-order fallback equality call (comparison lives in page-state/import-guard.ts); +1 F1c: direct imports run in maintenanceTransaction (write attribution; helper in persistence/attribution.ts); lowered to 2178: #5988 content screen and content-sanity config peeled to core/import-screen.ts; +1 #6007: the prepared apply returns its page id and seal (PreparedImportApplied); +1 #5333: trusted imports drop a stale oversized embed_skip/content_flag the gate no longer sets, and an embed_skip transition defeats both unchanged fast paths; wave 10 collector: lane E (#5333) and the other lanes compose at 2183; grown #6188 (+17): fence options interface, result fields and the Tier 1 step call (logic in fence-repair/import-step.ts); +2 #5984 single-write fast path: the coordinated existingSnapshot option (interface field) composes with the lane apply diet at 2202; +1 GBRA-69: a coordinated apply defers its chunker seal to the caller (sealImportedPage in page-state/projections.ts); lowered to 2023: #5001 the edge-mapping loop moved to code-chunks.ts mapCodeEdges (shared with the prepared apply); W14 P3.13 (#5001) lowered 2049→2023 by moving the edge loop to mapCodeEdges; +2 W13 PR1 on top (2025 measured); +2 #6529: the unchanged-image skip path records source_path (refreshSourcePath, as #4588 does for Markdown); +4 large pages A6/T33: code files keep CODE_FILE_MAX_BYTES, a page over PAGE_INLINE_MAX_BYTES commits with NULL vectors for the metered stale sweep; +1 #6722: an embed_skip page is chunked and never embedded (noEmbed folds the marker); merge v0.60.240.0: large pages and master growth compose at 2032; +24 Wave B lane 1d: the segmented storage step (stage before publication, flip in the apply, seal and post-commit projection; logic in persistence/segmented-apply.ts and page-body/ingest.ts) and the streamed-import dispatch over the single-string ceiling (logic in page-body/streamed-import.ts); merge v0.60.248.0: Wave B PR 1 and master growth compose at 2056; +4 Wave B review: a rename re-preparation reuses the staged segmented body (reuseSegments option) and the remote fence merge reads an unread body as fence-free (fenceBody in page-body/page-shape.ts); +2 Wave B gem A: a coordinated demotion versions the segmented page before demoting it (createVersion order); lanes compose at 2062; +2 Wave B lane 2q: ImportResult.searchable (interface field; the value comes from SegmentedImport.afterCommit);+2: pre-publication withdrawal validation; +9 #7172: kept chunk rows move to their new index and a bounded managed publication defers its chunk replacement (logic in page-state/chunk-moves.ts and page-state/projections.ts continueDeferredProjection); merge v0.60.332.0: #7172 and master growth compose at 2075; +7 GBRA-283 B3a (#8523): the three file import entry points re-enter an embedded engine's scope (needsEngineScope in core/engine-scope.ts), importFromContent runs its screened body in that scope as a closure, and the deferred embed resumes the import's scope
src/core/migrate.ts 602 ratchet schema-migration runner only; migrations live in src/core/schema-migrations/ (W3); +3 #5216: resumable page revision backfill runs after every migration pass while pages.knowledge_revision is nullable
src/core/operations.ts 246 ratchet +1 #7168: graphRankOperations import (ops/graph-rank.ts; spread shares the orphans line); peel target: containment sprint C4-C7; grown v0.46.26.0 megawave op wiring; +chat-connectors wiring (master merge); +5 v0.47 gmail-loops: loopsOperations spread + OP_AREAS; #5105: receipt operation registry wiring; shared-skills integration on company-ingestion base; +2 security wave: withOutputRedaction applied at op registration; +1 #5616: pageEditOperations import (spread shares the pages line); +1 Foundations 1 F1b: attributionOperations import (spread shares the admin line); +1 agent operator wave A6: ops/notices.ts import (mute_notice); +2 agent operator closeout A1: registerOpRoutes import + call (routing logic in fix-routing.ts); +2 retrieval feedback: feedbackOperations import and spread (ops/feedback.ts); +1 #6007: pageBatchOperations import (spread shares the pages line); +1 P5: wanted_pages area row; lowered GBRA-75 W11 cold start: OP_AREAS + finalize step moved to operation-load.ts (shared with the CLI per-op loader); +1 C4 pinned questions: questionsOperations import and spread (ops/questions.ts; its areas live in operation-load.ts OP_AREAS); +1 Wave B append: pageAppendOperations import (spread shares the pages line); +1 Wave B upload sessions: pageUploadOperations import (spread shares the pages line); +1 #7191: search-exact import; +1 GBRA-156: whois import (ops/whois.ts; spread shares the chunks line); +1 GST-359 D: qmMemoryOperations import (ops/qm-memory.ts; spread shares the memory line)
src/core/pglite-engine.ts 3103 ratchet +4 GBRA-283 B1 (#8455): initSchema seeds an engine opened from settings with settings.embedding and refuses a mismatch, and its open skips the snapshot fast path (logic in embedded-engine.ts; engine methods cannot move); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2983 measured); 4 Foundations 2 Lane F: autocommit write statements take the WAL checkpoint guard (classifier writesWal in pglite-engine/checkpoint-guard.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); E1 bootstrap moved to engine-sql/bootstrap.ts; ceiling rationale: docs/architecture/key-files/engines-1.md entry for src/core/pglite-engine.ts; evidence delivery: getChunkWindows delegate + type import (+3; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +12 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3213 #5824: searchVector SQL moved to search/vector-statement.ts (-166); lowered to 3117 #5889: searchTitles SQL moved to engine-sql/titles.ts (-108); lowered to 2915 F4a: getHealth SQL moved to engine-sql/health.ts (-202); +6 F4b: planner-sensitive reads call beforePlannerRead/plannerRead (logic in planner-stats.ts); +52 engine graduation (GBRA-50 G2a): marker/tombstone checks before and after the kernel lock (logic in persistence/graduation-custody.ts), lock-retaining close and held-lock open (engine lifecycle methods cannot move); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2983 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +5 GBRA-67: readPageSnapshotsBatch delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages delegate (SQL in engine-sql/pages.ts); +2 GBRA-76: transaction/transactionDirect take TransactionOptions and refuse an already-aborted signal before BEGIN; +2 #4772 (W14 PR3): entityTypes HealthDeps delegate + import (policy in schema-pack/entity-types.ts, SQL in engine-sql/health.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3023 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates (SQL in engine-sql/pages.ts); +6 large pages Wave A sub-a: getChunkPage delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +7 LHF wave 2 item 5: request config snapshot delegate lines in getConfig/getAllConfig/setConfig/unsetConfig (logic in config-snapshot.ts); +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3072 + 3; +5 PR2 fix lane X: keywordBoundedRead runs the read on the engine itself (engine methods cannot move): 3075 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3080 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3085); +2 #7159: the frontmatter where clause in searchKeyword and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); +2 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity with an orFallback rank parameter like Postgres (logic in search/keyword-or-rarity.ts; engine methods cannot move); +5 agent operator gate fixes: repair-failed marker gate before the lock and the marker record after a failed auto-repair (logic in pglite-repair.ts, refusal in pglite-repair-consent.ts); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3097 measured); +2 GBRA-283 Wave A (#8460): the ENGINE_ROOT import and field (engine identity for per-engine caches; class fields cannot move)
diff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsv
index 6265359fc0..ab78dfb516 100644
--- a/scripts/page-body-readers-baseline.tsv
+++ b/scripts/page-body-readers-baseline.tsv
@@ -312,10 +312,10 @@ src/core/google/loops-extract.ts runLoopsExtract field STREAM tail slice(-12000)
src/core/google/sweep-shared.ts withStoredFactsFence field INLINE asserts inline (assertInlineBody); importRendered refuses a segmented page first
src/core/import-file.ts applyImportTransaction call:putPage SEGW lane 1d file (streaming ingest); code/image pages inline
src/core/import-file.ts importCodeFile call:putPage SEGW lane 1d file (streaming ingest); code/image pages inline
-src/core/import-file.ts importScreenedContent call:contentHash META lane 1d file
-src/core/import-file.ts importScreenedContent call:putPage SEGW lane 1d file (streaming ingest); code/image pages inline
-src/core/import-file.ts importScreenedContent call:refreshPageBody SEGW lane 1d file (streaming ingest); code/image pages inline
-src/core/import-file.ts importScreenedContent field SEGW lane 1d file (streaming ingest); code/image pages inline
+src/core/import-file.ts importFromContent call:contentHash META lane 1d file
+src/core/import-file.ts importFromContent call:putPage SEGW lane 1d file (streaming ingest); code/image pages inline
+src/core/import-file.ts importFromContent call:refreshPageBody SEGW lane 1d file (streaming ingest); code/image pages inline
+src/core/import-file.ts importFromContent field SEGW lane 1d file (streaming ingest); code/image pages inline
src/core/import-file.ts importImageFile field SEGW lane 1d file (streaming ingest); code/image pages inline
src/core/import-identity.ts decideImportIdentity field META
src/core/import-screen.ts assessImportSanity field STREAM
diff --git a/src/core/import-file.ts b/src/core/import-file.ts
index e3b89475ad..1bb213f5c1 100644
--- a/src/core/import-file.ts
+++ b/src/core/import-file.ts
@@ -359,6 +359,12 @@ export async function importFromContent(
// rolling back the whole import (#430).
slug = validateSlug(slug);
+ // v0.18.0+ multi-source: when caller is syncing under a non-default source,
+ // every per-page tx call must carry `sourceId` so writes target the right
+ // (source_id, slug) row. Pre-fix, putPage relied on the schema DEFAULT and
+ // silently fabricated a duplicate at (default, slug) — causing later
+ // bare-slug subqueries (getTags, deleteChunks, etc.) to crash with 21000.
+ const sourceId = opts.sourceId;
// Reject oversized payloads before any parsing, chunking, or embedding happens.
// Uses Buffer.byteLength to count UTF-8 bytes the same way disk size would,
// so the network path behaves identically to the file path.
@@ -366,592 +372,578 @@ export async function importFromContent(
if (screen.status === 'refused') {
return { slug, status: screen.refusal.code === 'file_too_large' ? 'skipped' : 'error', chunks: 0, error: screen.refusal.message, refusal: screen.refusal, skip_reason: screen.refusal.code };
}
- if (needsEngineScope(engine)) return withEngineScope(engine, () => importScreenedContent(engine, slug, content, opts, screen));
- return importScreenedContent(engine, slug, content, opts, screen);
-}
-
-async function importScreenedContent(
- engine: BrainEngine,
- slug: string,
- content: string,
- opts: NonNullable<Parameters<typeof importFromContent>[3]>,
- screen: ReturnType<typeof screenImportContent>,
-): Promise<ImportResult> {
- // v0.18.0+ multi-source: when caller is syncing under a non-default source,
- // every per-page tx call must carry `sourceId` so writes target the right
- // (source_id, slug) row. Pre-fix, putPage relied on the schema DEFAULT and
- // silently fabricated a duplicate at (default, slug) — causing later
- // bare-slug subqueries (getTags, deleteChunks, etc.) to crash with 21000.
- const sourceId = opts.sourceId;
const parsed = (screen as { parsed: ParsedMarkdown }).parsed;
if (!opts.prepare) await assertUnmanagedCanonicalWriter(engine, 'direct content import', { sourceId: sourceId ?? 'default' });
- // Canonicalize only free-prose fields before protected-fence parsing, hidden
- // row merging, hashing and indexing. Frontmatter identities stay untouched.
- parsed.title = sanitizeText(parsed.title);
- parsed.compiled_truth = sanitizeText(parsed.compiled_truth);
- parsed.timeline = sanitizeText(parsed.timeline);
-
- // #1699/#6259 trust boundary: only the gate and owner-tier paths keep gate-owned markers (import-screen.ts).
- stripGateOwnedMarkers(parsed, opts);
-
- // Vendor-neutral guardrail seam (observe-only, fail-open). Runs AFTER
- // parseMarkdown and the size guard, BEFORE content-sanity, hash compute,
- // chunking, embedding, and DB write — so a registered guardrail sees the
- // full markdown payload at the exact pre-persist moment. The returned
- // verdict is intentionally ignored: this seam cannot block or mutate the
- // ingest. No-op when zero guardrails are registered (OSS default).
- await runGuardrails({
- hook: 'file_storage.markdown',
- content,
- metadata: {
- slug,
- source_id: sourceId ?? 'default',
- source_path: opts.sourcePath ?? null,
- source_kind: opts.source_kind ?? null,
- source_uri: opts.source_uri ?? null,
- ingested_via: opts.ingested_via ?? null,
- content_type: 'markdown',
- },
- });
-
- // Content-sanity (#1699) and write-gate (#5575) disposition before the hash (import-screen.ts).
- const disposition = await settleContentDisposition(engine, parsed, { slug, sourceId, stripped: opts.preserveGateMarkers !== true, ...(opts.writeGate ? { writeGate: opts.writeGate } : {}) });
- const { quarantined: pageQuarantined, flagged: pageFlagged, flagReason: pageFlagReason } = disposition;
-
- // v0.39.3.0 CV8 — DB content_hash excludes timestamp-bearing frontmatter
- // keys so identical body content from `gbrain capture` (which stamps
- // `captured_at` and `ingested_at` per call) produces a stable hash.
- // Pre-fix, every capture-cli invocation produced a fresh hash because
- // the timestamp changed, defeating:
- // - the existing.content_hash === hash short-circuit below (every
- // capture re-chunked + re-embedded unchanged content — wasted
- // embedding spend)
- // - the daemon's 24h LRU dedup (separate consumer keyed on same hash)
- //
- // We strip ONLY the timestamp keys, not the whole frontmatter object.
- // Stripping all frontmatter would regress sync: a user adding a tag
- // would update the frontmatter without changing the body, the hash
- // would not change, and tag reconciliation would silently no-op
- // (this function returns early on hash-match).
- //
- // v0.42 (#1699): gate-derived quarantine/content_flag/embed_skip markers
- // carry assessed_at timestamps. They describe the assessment, not source
- // content, so they must be EXCLUDED from the hash — otherwise re-sync
- // re-chunks + re-embeds forever (a markup-heavy page keeps chunks, so this
- // is real, unbounded embedding spend). Same bug class as the captured_at /
- // ingested_at fix above; the gate re-derives the markers deterministically
- // on the next import, so dropping them from the hash is safe.
- // #1035: fetch the existing page BEFORE the hash compute so (a) the type
- // preservation below participates in the hash (a no-op re-put stays a
- // hash-match skip) and (b) the hash short-circuit below reuses this row.
- // Scoped to the exact (source_id, slug) row the writes below target —
- // engine.putPage defaults to 'default' when sourceId is unset, so the read
- // mirrors that default instead of matching the slug in ANY source (the
- // unscoped-check/scoped-write bug class).
- const existingSnapshot = !opts.prepare ? null : await materializedSnapshot(engine, opts.coordinated && opts.existingSnapshot !== undefined ? opts.existingSnapshot : await engine.readPageSnapshot(slug, { sourceId: sourceId ?? 'default', includeDeleted: true }));
- const existing = opts.prepare ? existingSnapshot?.page ?? null : await materializedPage(engine, await engine.getPage(slug, { sourceId: sourceId ?? 'default', includeDeleted: true }));
- if (existing) stabilizeSafetyAssessments(parsed.frontmatter, existing.frontmatter);
-
- const collisionOwner = collidingSlugOwner(existing, slug, opts.sourceRoot, opts.sourcePath);
- if (collisionOwner) {
- process.stderr.write(
- `[import] slug collision: ${opts.sourcePath} and ${collisionOwner} both map to ${slug} ` +
- `in source ${sourceId ?? 'default'}; keeping ${collisionOwner}. Rename one file to import both.\n`
- );
- return { slug, status: 'skipped', chunks: 0, skip_reason: 'slug_collision' };
- }
+ const importScreened = async (): Promise<ImportResult> => {
+ // Canonicalize only free-prose fields before protected-fence parsing, hidden
+ // row merging, hashing and indexing. Frontmatter identities stay untouched.
+ parsed.title = sanitizeText(parsed.title);
+ parsed.compiled_truth = sanitizeText(parsed.compiled_truth);
+ parsed.timeline = sanitizeText(parsed.timeline);
+
+ // #1699/#6259 trust boundary: only the gate and owner-tier paths keep gate-owned markers (import-screen.ts).
+ stripGateOwnedMarkers(parsed, opts);
+
+ // Vendor-neutral guardrail seam (observe-only, fail-open). Runs AFTER
+ // parseMarkdown and the size guard, BEFORE content-sanity, hash compute,
+ // chunking, embedding, and DB write — so a registered guardrail sees the
+ // full markdown payload at the exact pre-persist moment. The returned
+ // verdict is intentionally ignored: this seam cannot block or mutate the
+ // ingest. No-op when zero guardrails are registered (OSS default).
+ await runGuardrails({
+ hook: 'file_storage.markdown',
+ content,
+ metadata: {
+ slug,
+ source_id: sourceId ?? 'default',
+ source_path: opts.sourcePath ?? null,
+ source_kind: opts.source_kind ?? null,
+ source_uri: opts.source_uri ?? null,
+ ingested_via: opts.ingested_via ?? null,
+ content_type: 'markdown',
+ },
+ });
- // #2044 / #4548: remote get_page/fetch intentionally strip non-'world'
- // facts rows before an untrusted caller ever sees them. A documented
- // get_page -> edit -> put_page round-trip therefore arrives MISSING rows
- // the caller structurally could never have seen — their absence is not an
- // intentional delete. Restore exactly those rows via the row-level,
- // visibility-aware merge (restoreHiddenFactRows): world-visible rows the
- // caller saw in full are never restored, so a legitimate edit/deletion of
- // a visible row — including deleting a pure-world fence outright — is
- // honored as written (#4554). Trusted local writers see everything, so
- // the merge never fires for them.
- // The same hazard covers the `timeline` column (#4546): a Facts fence
- // below the `<!-- timeline -->` sentinel is a normal placement, and #4547
- // strips its non-'world' rows for remote readers too — so a round-trip
- // write-back arrives missing those rows as well. Each column merges
- // against its own existing counterpart (fences don't migrate between
- // columns here; splitBody keeps them where the caller wrote them).
- // #6188 (E7): Tier 1 in place before the hidden-row merge and withdrawal preservation; the hash covers the normalized body.
- const fences = await applyImportFences({ engine, sourceId: sourceId ?? 'default', slug, mode: opts.fences ?? 'lenient', screened: opts.fenceScreen, screen, parsed, activePack: opts.activePack, remote: opts.remote === true, existing });
- if (fences.refused) return fences.refused;
- if (opts.remote === true && existing) {
- const stored = fenceBody(existing);
- parsed.compiled_truth = mergeHiddenFactRowsIntoBody(slug, parsed.compiled_truth, stored.compiled_truth);
- parsed.timeline = mergeHiddenFactRowsIntoBody(slug, parsed.timeline, stored.timeline);
- }
- const { assertPreparedFactWithdrawals, preserveWithdrawnFenceRows } = await import('./facts/withdrawal.ts');
- parsed.compiled_truth = (await preserveWithdrawnFenceRows(engine, sourceId ?? 'default', parsed.compiled_truth, slug)).trim();
- parsed.timeline = (await preserveWithdrawnFenceRows(engine, sourceId ?? 'default', parsed.timeline, slug)).trim();
-
- // #1035: absence of an explicit frontmatter `type:` on an EXISTING page
- // means "preserve the stored type", not "re-infer". Pre-fix, a round-trip
- // put (get_page → edit body → put_page without `type:`) silently regressed
- // a curated type to the path-inferred default ('concept' for bare slugs).
- // Explicit frontmatter type stays an override; new pages still infer. The stored subtype and grandfather stamp are kept too.
- if (existing) alignWithStored(parsed, existing, false);
- else resolveParsedSubtype(parsed, existing);
- // Alias-footgun visibility: an explicit frontmatter `type:` that is an
- // ALIAS of a canonical pack type (or entirely undeclared) is stored
- // literally and never re-normalized — different agents can silently file
- // the same concept under different types/directories. Classify it here
- // (once per file, aggregated once per type per run by sync/import) so the
- // misroute class is loud. Purely advisory: the type is still stored as-is.
- let typeWarning: ImportResult['type_warning'];
- if (parsed.typeExplicit === true && opts.activePack) {
- const cls = classifyStoredType(parsed.type, opts.activePack);
- if (cls.kind === 'alias_of') {
- typeWarning = { kind: 'alias_of', type: parsed.type, canonical: cls.canonical, directory: cls.directory };
- } else if (cls.kind === 'undeclared') {
- typeWarning = { kind: 'undeclared', type: parsed.type };
+ // Content-sanity (#1699) and write-gate (#5575) disposition before the hash (import-screen.ts).
+ const disposition = await settleContentDisposition(engine, parsed, { slug, sourceId, stripped: opts.preserveGateMarkers !== true, ...(opts.writeGate ? { writeGate: opts.writeGate } : {}) });
+ const { quarantined: pageQuarantined, flagged: pageFlagged, flagReason: pageFlagReason } = disposition;
+
+ // v0.39.3.0 CV8 — DB content_hash excludes timestamp-bearing frontmatter
+ // keys so identical body content from `gbrain capture` (which stamps
+ // `captured_at` and `ingested_at` per call) produces a stable hash.
+ // Pre-fix, every capture-cli invocation produced a fresh hash because
+ // the timestamp changed, defeating:
+ // - the existing.content_hash === hash short-circuit below (every
+ // capture re-chunked + re-embedded unchanged content — wasted
+ // embedding spend)
+ // - the daemon's 24h LRU dedup (separate consumer keyed on same hash)
+ //
+ // We strip ONLY the timestamp keys, not the whole frontmatter object.
+ // Stripping all frontmatter would regress sync: a user adding a tag
+ // would update the frontmatter without changing the body, the hash
+ // would not change, and tag reconciliation would silently no-op
+ // (this function returns early on hash-match).
+ //
+ // v0.42 (#1699): gate-derived quarantine/content_flag/embed_skip markers
+ // carry assessed_at timestamps. They describe the assessment, not source
+ // content, so they must be EXCLUDED from the hash — otherwise re-sync
+ // re-chunks + re-embeds forever (a markup-heavy page keeps chunks, so this
+ // is real, unbounded embedding spend). Same bug class as the captured_at /
+ // ingested_at fix above; the gate re-derives the markers deterministically
+ // on the next import, so dropping them from the hash is safe.
+ // #1035: fetch the existing page BEFORE the hash compute so (a) the type
+ // preservation below participates in the hash (a no-op re-put stays a
+ // hash-match skip) and (b) the hash short-circuit below reuses this row.
+ // Scoped to the exact (source_id, slug) row the writes below target —
+ // engine.putPage defaults to 'default' when sourceId is unset, so the read
+ // mirrors that default instead of matching the slug in ANY source (the
+ // unscoped-check/scoped-write bug class).
+ const existingSnapshot = !opts.prepare ? null : await materializedSnapshot(engine, opts.coordinated && opts.existingSnapshot !== undefined ? opts.existingSnapshot : await engine.readPageSnapshot(slug, { sourceId: sourceId ?? 'default', includeDeleted: true }));
+ const existing = opts.prepare ? existingSnapshot?.page ?? null : await materializedPage(engine, await engine.getPage(slug, { sourceId: sourceId ?? 'default', includeDeleted: true }));
+ if (existing) stabilizeSafetyAssessments(parsed.frontmatter, existing.frontmatter);
+
+ const collisionOwner = collidingSlugOwner(existing, slug, opts.sourceRoot, opts.sourcePath);
+ if (collisionOwner) {
+ process.stderr.write(
+ `[import] slug collision: ${opts.sourcePath} and ${collisionOwner} both map to ${slug} ` +
+ `in source ${sourceId ?? 'default'}; keeping ${collisionOwner}. Rename one file to import both.\n`
+ );
+ return { slug, status: 'skipped', chunks: 0, skip_reason: 'slug_collision' };
}
- }
-
- // #3694: the hash formula lives in ONE place — utils.contentHash — shared
- // with both engines' putPage fallback, so a page written via putPage and
- // the same page re-imported by sync produce the SAME hash (pre-fix they
- // diverged and every putPage→sync roundtrip re-chunked + re-embedded).
- // The helper strips HASH_EPHEMERAL_FRONTMATTER_KEYS + the tags key from a
- // frontmatter copy and folds sorted tags in — the exact former inline
- // formula (byte-parity pinned by test/content-hash-parity-3694.test.ts).
- // Sort tags in place first to preserve the pre-#3694 downstream behavior
- // (parsedPage.tags was sorted by the old inline `.sort()` mutation).
- const suppliedTrustMarker = parsed.frontmatter.trust_tier; // CEO-21: the no-op check also reads the caller's own marker
- if (opts.prepare) opts.prepareFrontmatter?.(parsed);
- parsed.tags.sort();
- const hash = contentHash({
- title: parsed.title,
- type: parsed.type,
- compiled_truth: parsed.compiled_truth,
- timeline: parsed.timeline,
- frontmatter: parsed.frontmatter,
- tags: parsed.tags,
- });
- const parsedPage: ParsedPage = {
- type: parsed.type,
- title: parsed.title,
- compiled_truth: parsed.compiled_truth,
- timeline: parsed.timeline || '',
- frontmatter: parsed.frontmatter,
- tags: parsed.tags,
- };
-
- const originUri = fileOriginUri(existing?.source_uri, opts.sourceRoot, opts.sourcePath);
- const persistUnchanged = async (refreshBody = false) => {
- return maintenanceTransaction(engine, async tx => {
- await assertImportBase(tx, slug, sourceId ?? 'default', existing);
- if (refreshBody) {
- const stored = await readStoredBody(tx, sourceId ?? 'default', slug, true);
- if (!stored || !fileBodyMatchesStored(content, slug, stored)) return false;
- await tx.refreshPageBody(slug, sourceId ?? 'default', parsed.compiled_truth, parsed.timeline || '', hash);
+ // #2044 / #4548: remote get_page/fetch intentionally strip non-'world'
+ // facts rows before an untrusted caller ever sees them. A documented
+ // get_page -> edit -> put_page round-trip therefore arrives MISSING rows
+ // the caller structurally could never have seen — their absence is not an
+ // intentional delete. Restore exactly those rows via the row-level,
+ // visibility-aware merge (restoreHiddenFactRows): world-visible rows the
+ // caller saw in full are never restored, so a legitimate edit/deletion of
+ // a visible row — including deleting a pure-world fence outright — is
+ // honored as written (#4554). Trusted local writers see everything, so
+ // the merge never fires for them.
+ // The same hazard covers the `timeline` column (#4546): a Facts fence
+ // below the `<!-- timeline -->` sentinel is a normal placement, and #4547
+ // strips its non-'world' rows for remote readers too — so a round-trip
+ // write-back arrives missing those rows as well. Each column merges
+ // against its own existing counterpart (fences don't migrate between
+ // columns here; splitBody keeps them where the caller wrote them).
+ // #6188 (E7): Tier 1 in place before the hidden-row merge and withdrawal preservation; the hash covers the normalized body.
+ const fences = await applyImportFences({ engine, sourceId: sourceId ?? 'default', slug, mode: opts.fences ?? 'lenient', screened: opts.fenceScreen, screen, parsed, activePack: opts.activePack, remote: opts.remote === true, existing });
+ if (fences.refused) return fences.refused;
+ if (opts.remote === true && existing) {
+ const stored = fenceBody(existing);
+ parsed.compiled_truth = mergeHiddenFactRowsIntoBody(slug, parsed.compiled_truth, stored.compiled_truth);
+ parsed.timeline = mergeHiddenFactRowsIntoBody(slug, parsed.timeline, stored.timeline);
+ }
+ const { assertPreparedFactWithdrawals, preserveWithdrawnFenceRows } = await import('./facts/withdrawal.ts');
+ parsed.compiled_truth = (await preserveWithdrawnFenceRows(engine, sourceId ?? 'default', parsed.compiled_truth, slug)).trim();
+ parsed.timeline = (await preserveWithdrawnFenceRows(engine, sourceId ?? 'default', parsed.timeline, slug)).trim();
+
+ // #1035: absence of an explicit frontmatter `type:` on an EXISTING page
+ // means "preserve the stored type", not "re-infer". Pre-fix, a round-trip
+ // put (get_page → edit body → put_page without `type:`) silently regressed
+ // a curated type to the path-inferred default ('concept' for bare slugs).
+ // Explicit frontmatter type stays an override; new pages still infer. The stored subtype and grandfather stamp are kept too.
+ if (existing) alignWithStored(parsed, existing, false);
+ else resolveParsedSubtype(parsed, existing);
+ // Alias-footgun visibility: an explicit frontmatter `type:` that is an
+ // ALIAS of a canonical pack type (or entirely undeclared) is stored
+ // literally and never re-normalized — different agents can silently file
+ // the same concept under different types/directories. Classify it here
+ // (once per file, aggregated once per type per run by sync/import) so the
+ // misroute class is loud. Purely advisory: the type is still stored as-is.
+ let typeWarning: ImportResult['type_warning'];
+ if (parsed.typeExplicit === true && opts.activePack) {
+ const cls = classifyStoredType(parsed.type, opts.activePack);
+ if (cls.kind === 'alias_of') {
+ typeWarning = { kind: 'alias_of', type: parsed.type, canonical: cls.canonical, directory: cls.directory };
+ } else if (cls.kind === 'undeclared') {
+ typeWarning = { kind: 'undeclared', type: parsed.type };
}
- await refreshSourcePath(tx, slug, sourceId, opts.sourcePath, existing, originUri);
- if (opts.beforeCommit) await verifyPageReadable(tx, slug, refreshBody ? hash : existing?.content_hash ?? hash, sourceId, 'importFromContent');
- await opts.beforeCommit?.(tx, slug);
- return true;
- });
- };
+ }
- // Rebuild stale projections without granting --force-rechunk's external-ID dedup override.
- const needsProjectionRebuild = !opts.prepare && existing && existing.text_projection_revision !== existing.knowledge_revision;
- const embedSkipChanged = !!existing && isEmbedSkipped(existing.frontmatter) !== isEmbedSkipped(parsed.frontmatter);
- // #3694 one-time reconcile: a row written by the PRE-fix putPage formula
- // carries the legacy hash. When the parsed file matches that legacy hash,
- // the content is unchanged — stamp the canonical hash via the narrow
- // refreshPageBody UPDATE (no chunk churn, no re-embed, no version snapshot)
- // and skip. The next import then hits the fast path below.
- const legacyHashMatch = !!existing && !existing.deleted_at && !opts.prepare && !opts.forceRechunk && !needsProjectionRebuild && !embedSkipChanged
- && typeof engine.refreshPageBody === 'function' && existing.content_hash === contentHashLegacy({
+ // #3694: the hash formula lives in ONE place — utils.contentHash — shared
+ // with both engines' putPage fallback, so a page written via putPage and
+ // the same page re-imported by sync produce the SAME hash (pre-fix they
+ // diverged and every putPage→sync roundtrip re-chunked + re-embedded).
+ // The helper strips HASH_EPHEMERAL_FRONTMATTER_KEYS + the tags key from a
+ // frontmatter copy and folds sorted tags in — the exact former inline
+ // formula (byte-parity pinned by test/content-hash-parity-3694.test.ts).
+ // Sort tags in place first to preserve the pre-#3694 downstream behavior
+ // (parsedPage.tags was sorted by the old inline `.sort()` mutation).
+ const suppliedTrustMarker = parsed.frontmatter.trust_tier; // CEO-21: the no-op check also reads the caller's own marker
+ if (opts.prepare) opts.prepareFrontmatter?.(parsed);
+ parsed.tags.sort();
+ const hash = contentHash({
title: parsed.title,
type: parsed.type,
compiled_truth: parsed.compiled_truth,
timeline: parsed.timeline,
frontmatter: parsed.frontmatter,
+ tags: parsed.tags,
});
- const unchangedAs = async (page: ParsedPage) => !!existing && !existing.deleted_at && !opts.forceRechunk && !needsProjectionRebuild && !embedSkipChanged
- && !trustMarkerChanged(existing.frontmatter, page.frontmatter) && (existing.content_hash === hash
- ? !opts.prepare || sameCanonicalImport(existingSnapshot, page)
- : !legacyHashMatch && await sameContentAnyKeyOrder(engine, existing, existingSnapshot?.tags ?? null, page, sourceId ?? 'default'));
- const unchanged = await unchangedAs(parsedPage) || (suppliedTrustMarker !== parsed.frontmatter.trust_tier
- && await unchangedAs({ ...parsedPage, frontmatter: suppliedTrustMarkerView(parsed.frontmatter, suppliedTrustMarker) }));
- if (existing && unchanged) {
- // #5050: unchanged content chunked before the safe-chunk fence is re-sealed
- // projection-only; the canonical write stays a no-op.
- const reseal = await projectionBelowSafeFence(engine, existing.id);
- if (opts.prepare) {
- const result: ImportResult = { slug, status: 'skipped', chunks: 0, parsedPage, ...(typeWarning ? { type_warning: typeWarning } : {}), ...fences.fields };
- return opts.prepare({ slug, parsedPage, observedRevision: (existing as typeof existing & { knowledge_revision?: string }).knowledge_revision ?? null,
- noop: true, result, validate: async () => {},
- apply: async tx => { if (reseal) await queuePageProjection(tx, sourceId ?? 'default', slug, 'safe_chunk_reseal'); } });
- }
- await persistUnchanged();
- const resealed = reseal ? await resealSafeChunks(engine, slug, sourceId ?? 'default') : null;
- return { slug, status: 'skipped', chunks: 0, parsedPage, ...(typeWarning ? { type_warning: typeWarning } : {}), ...(resealed ? { resealed } : {}), ...fences.fields };
- }
- if (existing && legacyHashMatch && bodyStorageOf(existing) === 'segmented') throw pageBodySegmentedError(existing, 'legacy hash reconciliation');
- if (existing && legacyHashMatch && fileBodyMatchesStored(content, slug, existing) && await persistUnchanged(true)) {
- const resealed = await projectionBelowSafeFence(engine, existing.id) ? await resealSafeChunks(engine, slug, sourceId ?? 'default') : null;
- return { slug, status: 'skipped', chunks: 0, parsedPage, ...(typeWarning ? { type_warning: typeWarning } : {}), ...(resealed ? { resealed } : {}), ...fences.fields };
- }
+ const parsedPage: ParsedPage = {
+ type: parsed.type,
+ title: parsed.title,
+ compiled_truth: parsed.compiled_truth,
+ timeline: parsed.timeline || '',
+ frontmatter: parsed.frontmatter,
+ tags: parsed.tags,
+ };
- // Identity dedup (#1309): move, skip a true duplicate, or index both.
- // See decideImportIdentity for the posture.
- const fmId = (parsed.frontmatter as Record<string, unknown> | undefined)?.id;
- const fmIdStr = typeof fmId === 'string' && fmId.length > 0 ? fmId : null;
- const identity = opts.forceRechunk ? { kind: 'none' as const } : await decideImportIdentity(engine, {
- sourceId: sourceId ?? 'default', slug, hash, frontmatterId: fmIdStr, sourcePath: opts.sourcePath, sourceRoot: opts.sourceRoot,
- body: { title: parsed.title, compiled_truth: parsed.compiled_truth, timeline: parsed.timeline || '' },
- });
- if (identity.kind === 'move' && !existing && !opts.prepare
- && await maintenanceBeliefTransaction(engine, { sourceId: sourceId ?? 'default', slugs: [identity.dupSlug, slug], op: 'import_move' }, tx => tx.updateSlug(identity.dupSlug, slug, { sourceId: sourceId ?? 'default' })) > 0) {
- process.stderr.write(
- `[import] ${opts.sourcePath} carries frontmatter.id=${fmIdStr} from moved file ${identity.dupSourcePath}; ` +
- `renamed ${identity.dupSlug} -> ${slug} in source ${sourceId ?? 'default'}.\n`
- );
- return importFromContent(engine, slug, content, opts);
- }
- if (identity.kind === 'duplicate') {
- if (opts.prepare) {
- const result: ImportResult = { slug: identity.dupSlug, status: 'skipped', chunks: 0, parsedPage, ...fences.fields };
- return opts.prepare({ slug: identity.dupSlug, parsedPage, observedRevision: (existing as (typeof existing & { knowledge_revision?: string }) | null)?.knowledge_revision ?? null,
- noop: true, result, validate: async () => {}, apply: async () => {} });
+ const originUri = fileOriginUri(existing?.source_uri, opts.sourceRoot, opts.sourcePath);
+ const persistUnchanged = async (refreshBody = false) => {
+ return maintenanceTransaction(engine, async tx => {
+ await assertImportBase(tx, slug, sourceId ?? 'default', existing);
+ if (refreshBody) {
+ const stored = await readStoredBody(tx, sourceId ?? 'default', slug, true);
+ if (!stored || !fileBodyMatchesStored(content, slug, stored)) return false;
+ await tx.refreshPageBody(slug, sourceId ?? 'default', parsed.compiled_truth, parsed.timeline || '', hash);
+ }
+ await refreshSourcePath(tx, slug, sourceId, opts.sourcePath, existing, originUri);
+ if (opts.beforeCommit) await verifyPageReadable(tx, slug, refreshBody ? hash : existing?.content_hash ?? hash, sourceId, 'importFromContent');
+ await opts.beforeCommit?.(tx, slug);
+ return true;
+ });
+ };
+
+ // Rebuild stale projections without granting --force-rechunk's external-ID dedup override.
+ const needsProjectionRebuild = !opts.prepare && existing && existing.text_projection_revision !== existing.knowledge_revision;
+ const embedSkipChanged = !!existing && isEmbedSkipped(existing.frontmatter) !== isEmbedSkipped(parsed.frontmatter);
+ // #3694 one-time reconcile: a row written by the PRE-fix putPage formula
+ // carries the legacy hash. When the parsed file matches that legacy hash,
+ // the content is unchanged — stamp the canonical hash via the narrow
+ // refreshPageBody UPDATE (no chunk churn, no re-embed, no version snapshot)
+ // and skip. The next import then hits the fast path below.
+ const legacyHashMatch = !!existing && !existing.deleted_at && !opts.prepare && !opts.forceRechunk && !needsProjectionRebuild && !embedSkipChanged
+ && typeof engine.refreshPageBody === 'function' && existing.content_hash === contentHashLegacy({
+ title: parsed.title,
+ type: parsed.type,
+ compiled_truth: parsed.compiled_truth,
+ timeline: parsed.timeline,
+ frontmatter: parsed.frontmatter,
+ });
+ const unchangedAs = async (page: ParsedPage) => !!existing && !existing.deleted_at && !opts.forceRechunk && !needsProjectionRebuild && !embedSkipChanged
+ && !trustMarkerChanged(existing.frontmatter, page.frontmatter) && (existing.content_hash === hash
+ ? !opts.prepare || sameCanonicalImport(existingSnapshot, page)
+ : !legacyHashMatch && await sameContentAnyKeyOrder(engine, existing, existingSnapshot?.tags ?? null, page, sourceId ?? 'default'));
+ const unchanged = await unchangedAs(parsedPage) || (suppliedTrustMarker !== parsed.frontmatter.trust_tier
+ && await unchangedAs({ ...parsedPage, frontmatter: suppliedTrustMarkerView(parsed.frontmatter, suppliedTrustMarker) }));
+ if (existing && unchanged) {
+ // #5050: unchanged content chunked before the safe-chunk fence is re-sealed
+ // projection-only; the canonical write stays a no-op.
+ const reseal = await projectionBelowSafeFence(engine, existing.id);
+ if (opts.prepare) {
+ const result: ImportResult = { slug, status: 'skipped', chunks: 0, parsedPage, ...(typeWarning ? { type_warning: typeWarning } : {}), ...fences.fields };
+ return opts.prepare({ slug, parsedPage, observedRevision: (existing as typeof existing & { knowledge_revision?: string }).knowledge_revision ?? null,
+ noop: true, result, validate: async () => {},
+ apply: async tx => { if (reseal) await queuePageProjection(tx, sourceId ?? 'default', slug, 'safe_chunk_reseal'); } });
+ }
+ await persistUnchanged();
+ const resealed = reseal ? await resealSafeChunks(engine, slug, sourceId ?? 'default') : null;
+ return { slug, status: 'skipped', chunks: 0, parsedPage, ...(typeWarning ? { type_warning: typeWarning } : {}), ...(resealed ? { resealed } : {}), ...fences.fields };
}
- process.stderr.write(
- `[import] skipping ${opts.sourcePath ?? slug}: identical to ${identity.dupSlug} ` +
- `(frontmatter.id=${fmIdStr}) in source ${sourceId ?? 'default'}. ` +
- `Pass --force-rechunk to override.\n`
- );
- return { slug: identity.dupSlug, status: 'skipped', chunks: 0, parsedPage, ...fences.fields };
- }
- if (identity.kind === 'shared_id') {
- process.stderr.write(
- `[import] WARNING: ${opts.sourcePath ?? slug} shares frontmatter.id=${fmIdStr} with ${identity.dupSlug} ` +
- `but its content differs. Indexing both.\n`
- );
- } else if (identity.kind === 'shared_hash') {
- process.stderr.write(
- `[import] WARNING: ${opts.sourcePath ?? slug} shares content_hash with ${identity.dupSlug} ` +
- `(${hash.slice(0, 8)}) but not its frontmatter.id. Indexing both.\n`
- );
- }
- // Preserve the importer projection (including fenced code and zero-chunk
- // dispositions) in the provider-free path used by background rebuilds too.
- const bodyBytes = Buffer.byteLength(parsed.compiled_truth, 'utf-8') + Buffer.byteLength(parsed.timeline ?? '', 'utf-8');
- const storage = await stringWriteStorage(engine, bodyBytes, bodyStorageOf(existing));
- const chunks = storage.storage === 'segmented' ? [] : await prepareFenceAwareMarkdownChunks(engine, sourceId ?? 'default', slug, parsed);
+ if (existing && legacyHashMatch && bodyStorageOf(existing) === 'segmented') throw pageBodySegmentedError(existing, 'legacy hash reconciliation');
+ if (existing && legacyHashMatch && fileBodyMatchesStored(content, slug, existing) && await persistUnchanged(true)) {
+ const resealed = await projectionBelowSafeFence(engine, existing.id) ? await resealSafeChunks(engine, slug, sourceId ?? 'default') : null;
+ return { slug, status: 'skipped', chunks: 0, parsedPage, ...(typeWarning ? { type_warning: typeWarning } : {}), ...(resealed ? { resealed } : {}), ...fences.fields };
+ }
- // Embedding failures propagate unless onPostCommitEmbedding lets the caller
- // report enrichment separately from the already-persisted content.
- //
- // v0.40.3.0 contextual retrieval wrapper (D20-T1 chunk_text separation):
- // - Resolve effective CR mode via the page/source/global override chain.
- // - For title tier (free): build the title-only prefix and wrap chunks
- // inline at embed time. Per-chunk generated synopsis tier is NOT supported
- // on the import path — that's an async backfill via the Minion handler
- // (the cost prompt + 10s grace UX from D3 gates spending; inline import
- // path takes the cheaper title-only treatment for tokenmax pages here
- // and defers per-chunk synopsis to the Minion-driven sweep).
- // - Stored chunk_text stays canonical; only the embedding input is wrapped.
- // - Code chunks (chunk_source='fenced_code') bypass wrapping per D20-T4.
- // Every markdown import records its wrapping convention with the canonical
- // write, --no-embed included (#5621): later embed passes reproduce the stored
- // mode, so an unstamped page would stay unwrapped. No provider runs here.
- // #3885: the real source row applies (a stored `sources set-cr-mode`); the
- // inline path demotes per_chunk_synopsis to the free title tier.
- const { mode: effectiveCRMode, corpusGeneration } = await resolveImportContextualMode(engine, sourceId ?? 'default', parsed.frontmatter, opts.prepare !== undefined);
- // Wave B: a body staged as segments publishes with the generation flip instead of putPage and the chunk replacement.
- const segmented = storage.policy && storage.storage === 'segmented' ? await stageSegmentedImport(engine, { slug, sourceId: sourceId ?? 'default', existing,
- page: parsedPage, policy: storage.policy, owner: opts.segmentedOwner, reuse: opts.reuseSegments, baseRevision: (existing as { knowledge_revision?: string } | null)?.knowledge_revision ?? null,
- chunking: { quarantined: pageQuarantined, corpusGeneration, tier: effectiveCRMode === 'title' ? 'title' : 'none' } }) : null;
-
- // A13: an edit re-embeds only chunks whose embedding input changed. A stored
- // vector is reused for a chunk with the same source and text when its
- // recorded embedding input (#5553: column, model, dimensions, wrapper tier
- // and exact input) equals the one this import would embed, or, for a vector
- // recorded before provenance existed, when the input is the raw chunk text
- // under the same model. The old index must be sealed (no private fence row in a chunk), and on a fenced page
- // only a vector with a recorded input is reused (#6961); a changed fence chunk has new text and is embedded again.
- // A prepared (persistence) import defers embedding to its effect worker, so
- // it reuses inside its publication transaction, under the page guard, from
- // the chunks it is about to replace; only the chunks left without a vector
- // reach the provider.
- // A large or segmented page (the hysteresis band too) commits with NULL vectors; the stale
- // sweep embeds it under the per-page cap and the sampled junk check, so no provider call holds the write.
- const noEmbed = opts.noEmbed || isEmbedSkipped(parsed.frontmatter);
- const largePage = isLargePageBytes(bodyBytes) || segmented !== null;
- const postCommitEmbedding = largePage ? undefined : opts.onPostCommitEmbedding;
- const reusable = !!existing && !existing.deleted_at && !opts.forceRechunk && !postCommitEmbedding && chunks.length > 0 && bodyStorageOf(existing) !== 'segmented'
- && !(opts.prepare && modeRequiresSynopsis(effectiveCRMode));
- const reuseCtx = { slug, sourceId: sourceId ?? 'default', title: parsed.title, corpusGeneration, tier: effectiveCRMode === 'title' ? 'title' as const : 'none' as const, requireRecordedInput: reusable && (hasProtectedBody(`${existing.compiled_truth}\n${existing.timeline ?? ''}`) || hasProtectedBody(`${parsed.compiled_truth}\n${parsed.timeline ?? ''}`)) };
- const reuseStoredVectors = (exec: BrainEngine, into: ChunkInput[], keepUnembedded = false) => reuseStoredChunkVectors(exec, into, { ...reuseCtx, prepared: !!opts.prepare, keepUnembedded });
- const reused = reusable && !noEmbed && !opts.prepare ? await reuseStoredVectors(engine, chunks) : new Map<number, number | null>();
-
- let embeddingPartial: EmbeddingZeroNormError | undefined;
- const embedChunks = async () => {
- const pending = chunks.map((_, i) => i).filter(i => !reused.has(i));
- if (noEmbed || pending.length === 0) return;
- const prefix =
- modeRequiresWrapper(effectiveCRMode) && !modeRequiresSynopsis(effectiveCRMode)
- ? buildContextualPrefix(parsed.title, null)
- : null;
- const wrappedTexts = pending.map(i => prefix ? wrapChunkForEmbedding(chunks[i].chunk_text, prefix, chunks[i].chunk_source) : chunks[i].chunk_text);
- // #4616: store the usable vectors; refused chunks stay NULL for embed --stale.
- const { vectors: embeddings, refused } = await embedBatchKeepingUsable(wrappedTexts);
- embeddingPartial = refused;
- pending.forEach((i, j) => {
- if (!embeddings[j]) return;
- chunks[i].embedding = embeddings[j]!;
- // token_count tracks the wrapped string length so cost reporting
- // reflects what we actually sent to the embedder.
- chunks[i].token_count = Math.ceil(wrappedTexts[j].length / 4);
+ // Identity dedup (#1309): move, skip a true duplicate, or index both.
+ // See decideImportIdentity for the posture.
+ const fmId = (parsed.frontmatter as Record<string, unknown> | undefined)?.id;
+ const fmIdStr = typeof fmId === 'string' && fmId.length > 0 ? fmId : null;
+ const identity = opts.forceRechunk ? { kind: 'none' as const } : await decideImportIdentity(engine, {
+ sourceId: sourceId ?? 'default', slug, hash, frontmatterId: fmIdStr, sourcePath: opts.sourcePath, sourceRoot: opts.sourceRoot,
+ body: { title: parsed.title, compiled_truth: parsed.compiled_truth, timeline: parsed.timeline || '' },
});
- };
- // An embedding outage never blocks the text write: the page and its chunks
- // commit with NULL vectors and no embedding signature, and the stale sweep
- // (`gbrain embed --stale`, sync's embed pass, the cycle) embeds them once
- // the provider recovers.
- let embeddingDeferred = largePage && !noEmbed && (chunks.length > 0 || (segmented !== null && !pageQuarantined));
- if (!opts.onPostCommitEmbedding && !largePage) {
- try {
- await embedChunks();
- if (embeddingPartial) {
- embeddingDeferred = true;
- process.stderr.write(`[import] ${slug}: ${embeddingPartial.message} ${embeddingPartial.suggestionFor(slug, sourceId)}\n`);
+ if (identity.kind === 'move' && !existing && !opts.prepare
+ && await maintenanceBeliefTransaction(engine, { sourceId: sourceId ?? 'default', slugs: [identity.dupSlug, slug], op: 'import_move' }, tx => tx.updateSlug(identity.dupSlug, slug, { sourceId: sourceId ?? 'default' })) > 0) {
+ process.stderr.write(
+ `[import] ${opts.sourcePath} carries frontmatter.id=${fmIdStr} from moved file ${identity.dupSourcePath}; ` +
+ `renamed ${identity.dupSlug} -> ${slug} in source ${sourceId ?? 'default'}.\n`
+ );
+ return importFromContent(engine, slug, content, opts);
+ }
+ if (identity.kind === 'duplicate') {
+ if (opts.prepare) {
+ const result: ImportResult = { slug: identity.dupSlug, status: 'skipped', chunks: 0, parsedPage, ...fences.fields };
+ return opts.prepare({ slug: identity.dupSlug, parsedPage, observedRevision: (existing as (typeof existing & { knowledge_revision?: string }) | null)?.knowledge_revision ?? null,
+ noop: true, result, validate: async () => {}, apply: async () => {} });
}
- } catch (err) {
- embeddingDeferred = true;
- chunks.forEach((c, i) => { if (!reused.has(i)) { c.embedding = undefined; c.token_count = undefined; } });
process.stderr.write(
- `[import] ${slug}: embedding failed (${err instanceof Error ? err.message : String(err)}); ` +
- `text saved, chunks queued for \`gbrain embed --stale\`.\n`
+ `[import] skipping ${opts.sourcePath ?? slug}: identical to ${identity.dupSlug} ` +
+ `(frontmatter.id=${fmIdStr}) in source ${sourceId ?? 'default'}. ` +
+ `Pass --force-rechunk to override.\n`
+ );
+ return { slug: identity.dupSlug, status: 'skipped', chunks: 0, parsedPage, ...fences.fields };
+ }
+ if (identity.kind === 'shared_id') {
+ process.stderr.write(
+ `[import] WARNING: ${opts.sourcePath ?? slug} shares frontmatter.id=${fmIdStr} with ${identity.dupSlug} ` +
+ `but its content differs. Indexing both.\n`
+ );
+ } else if (identity.kind === 'shared_hash') {
+ process.stderr.write(
+ `[import] WARNING: ${opts.sourcePath ?? slug} shares content_hash with ${identity.dupSlug} ` +
+ `(${hash.slice(0, 8)}) but not its frontmatter.id. Indexing both.\n`
);
}
- }
- // Transaction wraps the canonical DB writes. Every per-page tx call carries the
- // caller's sourceId so writes target (sourceId, slug) rather than the
- // schema DEFAULT — required for multi-source brains; harmless ('default')
- // for single-source callers.
- const txOpts = { sourceId: sourceId ?? 'default' };
- let persistedProjection: ProjectionSnapshot | null = null;
- const [timeZone, mentionPolicy] = await Promise.all([loadBrainTimeZone(engine), readMentionPolicy(engine).catch(() => null)]);
- let chunkReplacementDeferred = false;
- const applyPrepared = async (tx: BrainEngine, preimage?: PageSnapshot | null) => {
- if (!opts.coordinated) await assertImportBase(tx, slug, txOpts.sourceId, existing, reused.size > 0);
- // #5984: the canonical writes go out in two pipelined waves of mutually independent statements; the
- // second depends on the first (the page row and its id, the old chunks gone). A failure rolls back the caller's transaction.
- // v0.29.1 — compute effective_date from frontmatter precedence chain.
- // Filename comes from importFromFile path (basename) or the slug tail
- // (put_page MCP op fallback). An undated page falls back to a stable
- // anchor (fallbackCreatedAt): its stored fallback date, else its file's
- // timestamp, else now — never the latest write. The result drives the recency boost and since/until
- // filters when callers opt in; nothing in the default search path
- // consults it.
- const filenameForChain = opts.filename ?? slug.split('/').pop() ?? slug;
- const nowDate = new Date();
- const { date: effectiveDate, source: effectiveDateSource } = computeEffectiveDate({
- slug,
- frontmatter: parsed.frontmatter,
- filename: filenameForChain,
- timeZone,
- updatedAt: existing?.updated_at ?? nowDate,
- createdAt: fallbackCreatedAt({ existing, fileTimes: opts.fileTimes, now: nowDate }),
- });
+ // Preserve the importer projection (including fenced code and zero-chunk
+ // dispositions) in the provider-free path used by background rebuilds too.
+ const bodyBytes = Buffer.byteLength(parsed.compiled_truth, 'utf-8') + Buffer.byteLength(parsed.timeline ?? '', 'utf-8');
+ const storage = await stringWriteStorage(engine, bodyBytes, bodyStorageOf(existing));
+ const chunks = storage.storage === 'segmented' ? [] : await prepareFenceAwareMarkdownChunks(engine, sourceId ?? 'default', slug, parsed);
+
+ // Embedding failures propagate unless onPostCommitEmbedding lets the caller
+ // report enrichment separately from the already-persisted content.
+ //
+ // v0.40.3.0 contextual retrieval wrapper (D20-T1 chunk_text separation):
+ // - Resolve effective CR mode via the page/source/global override chain.
+ // - For title tier (free): build the title-only prefix and wrap chunks
+ // inline at embed time. Per-chunk generated synopsis tier is NOT supported
+ // on the import path — that's an async backfill via the Minion handler
+ // (the cost prompt + 10s grace UX from D3 gates spending; inline import
+ // path takes the cheaper title-only treatment for tokenmax pages here
+ // and defers per-chunk synopsis to the Minion-driven sweep).
+ // - Stored chunk_text stays canonical; only the embedding input is wrapped.
+ // - Code chunks (chunk_source='fenced_code') bypass wrapping per D20-T4.
+ // Every markdown import records its wrapping convention with the canonical
+ // write, --no-embed included (#5621): later embed passes reproduce the stored
+ // mode, so an unstamped page would stay unwrapped. No provider runs here.
+ // #3885: the real source row applies (a stored `sources set-cr-mode`); the
+ // inline path demotes per_chunk_synopsis to the free title tier.
+ const { mode: effectiveCRMode, corpusGeneration } = await resolveImportContextualMode(engine, sourceId ?? 'default', parsed.frontmatter, opts.prepare !== undefined);
+ // Wave B: a body staged as segments publishes with the generation flip instead of putPage and the chunk replacement.
+ const segmented = storage.policy && storage.storage === 'segmented' ? await stageSegmentedImport(engine, { slug, sourceId: sourceId ?? 'default', existing,
+ page: parsedPage, policy: storage.policy, owner: opts.segmentedOwner, reuse: opts.reuseSegments, baseRevision: (existing as { knowledge_revision?: string } | null)?.knowledge_revision ?? null,
+ chunking: { quarantined: pageQuarantined, corpusGeneration, tier: effectiveCRMode === 'title' ? 'title' : 'none' } }) : null;
+
+ // A13: an edit re-embeds only chunks whose embedding input changed. A stored
+ // vector is reused for a chunk with the same source and text when its
+ // recorded embedding input (#5553: column, model, dimensions, wrapper tier
+ // and exact input) equals the one this import would embed, or, for a vector
+ // recorded before provenance existed, when the input is the raw chunk text
+ // under the same model. The old index must be sealed (no private fence row in a chunk), and on a fenced page
+ // only a vector with a recorded input is reused (#6961); a changed fence chunk has new text and is embedded again.
+ // A prepared (persistence) import defers embedding to its effect worker, so
+ // it reuses inside its publication transaction, under the page guard, from
+ // the chunks it is about to replace; only the chunks left without a vector
+ // reach the provider.
+ // A large or segmented page (the hysteresis band too) commits with NULL vectors; the stale
+ // sweep embeds it under the per-page cap and the sampled junk check, so no provider call holds the write.
+ const noEmbed = opts.noEmbed || isEmbedSkipped(parsed.frontmatter);
+ const largePage = isLargePageBytes(bodyBytes) || segmented !== null;
+ const postCommitEmbedding = largePage ? undefined : opts.onPostCommitEmbedding;
+ const reusable = !!existing && !existing.deleted_at && !opts.forceRechunk && !postCommitEmbedding && chunks.length > 0 && bodyStorageOf(existing) !== 'segmented'
+ && !(opts.prepare && modeRequiresSynopsis(effectiveCRMode));
+ const reuseCtx = { slug, sourceId: sourceId ?? 'default', title: parsed.title, corpusGeneration, tier: effectiveCRMode === 'title' ? 'title' as const : 'none' as const, requireRecordedInput: reusable && (hasProtectedBody(`${existing.compiled_truth}\n${existing.timeline ?? ''}`) || hasProtectedBody(`${parsed.compiled_truth}\n${parsed.timeline ?? ''}`)) };
+ const reuseStoredVectors = (exec: BrainEngine, into: ChunkInput[], keepUnembedded = false) => reuseStoredChunkVectors(exec, into, { ...reuseCtx, prepared: !!opts.prepare, keepUnembedded });
+ const reused = reusable && !noEmbed && !opts.prepare ? await reuseStoredVectors(engine, chunks) : new Map<number, number | null>();
+
+ let embeddingPartial: EmbeddingZeroNormError | undefined;
+ const embedChunks = async () => {
+ const pending = chunks.map((_, i) => i).filter(i => !reused.has(i));
+ if (noEmbed || pending.length === 0) return;
+ const prefix =
+ modeRequiresWrapper(effectiveCRMode) && !modeRequiresSynopsis(effectiveCRMode)
+ ? buildContextualPrefix(parsed.title, null)
+ : null;
+ const wrappedTexts = pending.map(i => prefix ? wrapChunkForEmbedding(chunks[i].chunk_text, prefix, chunks[i].chunk_source) : chunks[i].chunk_text);
+ // #4616: store the usable vectors; refused chunks stay NULL for embed --stale.
+ const { vectors: embeddings, refused } = await embedBatchKeepingUsable(wrappedTexts);
+ embeddingPartial = refused;
+ pending.forEach((i, j) => {
+ if (!embeddings[j]) return;
+ chunks[i].embedding = embeddings[j]!;
+ // token_count tracks the wrapped string length so cost reporting
+ // reflects what we actually sent to the embedder.
+ chunks[i].token_count = Math.ceil(wrappedTexts[j].length / 4);
+ });
+ };
+ // An embedding outage never blocks the text write: the page and its chunks
+ // commit with NULL vectors and no embedding signature, and the stale sweep
+ // (`gbrain embed --stale`, sync's embed pass, the cycle) embeds them once
+ // the provider recovers.
+ let embeddingDeferred = largePage && !noEmbed && (chunks.length > 0 || (segmented !== null && !pageQuarantined));
+ if (!opts.onPostCommitEmbedding && !largePage) {
+ try {
+ await embedChunks();
+ if (embeddingPartial) {
+ embeddingDeferred = true;
+ process.stderr.write(`[import] ${slug}: ${embeddingPartial.message} ${embeddingPartial.suggestionFor(slug, sourceId)}\n`);
+ }
+ } catch (err) {
+ embeddingDeferred = true;
+ chunks.forEach((c, i) => { if (!reused.has(i)) { c.embedding = undefined; c.token_count = undefined; } });
+ process.stderr.write(
+ `[import] ${slug}: embedding failed (${err instanceof Error ? err.message : String(err)}); ` +
+ `text saved, chunks queued for \`gbrain embed --stale\`.\n`
+ );
+ }
+ }
- const demote = !segmented && bodyStorageOf(existing) === 'segmented';
- // Before the page write: an unpinned version reads the page itself, a demoted one references the generation it retires.
- if (existing && (!preimage || demote)) await tx.createVersion(slug, preimage ? { ...txOpts, preimage } : txOpts);
- if (demote) await demoteSegmentedPage(tx, Number(existing!.id));
- // Every import keeps each stored row that is identical to its new chunk and
- // writes only the rest, so an edit rewrites only the chunks it changed. The
- // reuse is re-read here, under the page lock, so a kept row is one this
- // transaction sees; --no-embed and deferred-embed imports keep vectors too.
- const fresh = reusable ? chunks.map(chunk => ({ ...chunk })) : chunks;
- const kept = new Map([...(fresh !== chunks ? await reuseStoredVectors(tx, fresh, true) : [])]
- .filter((entry): entry is [number, KeptChunkRow] => entry[1] !== null));
- const keptIds = [...kept.values()].map(row => row.id);
- // #7172: a kept row whose chunk moved changes index in place; a managed publication over CHUNK_VECTOR_WRITE_BUDGET
- // vector-index writes deletes the rows it does not keep, commits the page pending and finishes after commit (chunk-moves.ts).
- const chunkWrites = planImportChunkWrites(fresh, kept, !segmented && !!opts.prepare && opts.boundedChunkWrites === true);
- const deferChunks = chunkReplacementDeferred = chunkWrites.defer;
- let written: Page | undefined;
- await pipelined(tx, [
- () => assertPreparedFactWithdrawals(tx, txOpts.sourceId, parsed.compiled_truth, parsed.timeline || '', slug),
- async () => { if (existing && preimage && !demote) await tx.createVersion(slug, { ...txOpts, preimage }); },
- async () => {
- if (segmented) return segmented.apply(tx, { type: parsed.type, title: parsed.title, frontmatter: parsed.frontmatter, page_kind: 'markdown',
- effective_date: effectiveDate, effective_date_source: effectiveDateSource, import_filename: filenameForChain, source_path: opts.sourcePath ?? null,
- source_kind: opts.source_kind ?? null, source_uri: opts.source_uri ?? originUri, ingested_via: opts.ingested_via ?? null }, { mode: effectiveCRMode, corpusGeneration });
- written = await tx.putPage(slug, {
- type: parsed.type,
- title: parsed.title,
- compiled_truth: parsed.compiled_truth,
- timeline: parsed.timeline || '',
- frontmatter: parsed.frontmatter,
- content_hash: hash,
- effective_date: effectiveDate,
- effective_date_source: effectiveDateSource,
- import_filename: filenameForChain,
- // Preserve the repo-relative path for sync's delete/rename handling.
- source_path: opts.sourcePath ?? null,
- // v0.39.3.0 provenance write-through (WARN-8). Engine layer applies
- // COALESCE-preserve UPDATE so omitting these on a later put_page
- // doesn't erase the original ingestion's audit trail.
- source_kind: opts.source_kind ?? null,
- source_uri: opts.source_uri ?? originUri,
- ingested_via: opts.ingested_via ?? null,
- // ingested_at is server-stamped at the engine layer when any
- // provenance write fires; never client-controlled.
- // Empty-overwrite escape hatch only when the caller vouched (file
- // import / explicit allow_empty); otherwise the engine guard stays on.
- }, { ...txOpts, ...(opts.allowEmptyOverwrite === true ? { allowEmptyOverwrite: true } : {}), inline: true,
- ...(effectiveCRMode !== 'none' ? { contextualRetrieval: { mode: effectiveCRMode, corpusGeneration } } : {}) });
- // v0.40.3.0: the contextual retrieval state is stamped with the page write (the `none` tier's
- // stamp, which also clears tier vectors, follows it). Deferred embedders reproduce it; every
- // chunk row is replaced below. This stamp certifies no vector: embedding_signature stays deferred.
- if (effectiveCRMode === 'none') await tx.updatePageContextualRetrievalState(slug, sourceId ?? 'default', effectiveCRMode, corpusGeneration);
- },
- // Replace every derived row atomically. Only vectors the A13 reuse gate above admitted carry over; a new
- // seal otherwise inherits nothing from an older index, whose contextual vector may have included a private sibling.
- async () => segmented ? undefined : keptIds.length ? tx.executeRaw(`DELETE FROM content_chunks WHERE page_id = (SELECT id FROM pages WHERE slug = $1 AND source_id = $2)
+ // Transaction wraps the canonical DB writes. Every per-page tx call carries the
+ // caller's sourceId so writes target (sourceId, slug) rather than the
+ // schema DEFAULT — required for multi-source brains; harmless ('default')
+ // for single-source callers.
+ const txOpts = { sourceId: sourceId ?? 'default' };
+ let persistedProjection: ProjectionSnapshot | null = null;
+ const [timeZone, mentionPolicy] = await Promise.all([loadBrainTimeZone(engine), readMentionPolicy(engine).catch(() => null)]);
+ let chunkReplacementDeferred = false;
+ const applyPrepared = async (tx: BrainEngine, preimage?: PageSnapshot | null) => {
+ if (!opts.coordinated) await assertImportBase(tx, slug, txOpts.sourceId, existing, reused.size > 0);
+ // #5984: the canonical writes go out in two pipelined waves of mutually independent statements; the
+ // second depends on the first (the page row and its id, the old chunks gone). A failure rolls back the caller's transaction.
+ // v0.29.1 — compute effective_date from frontmatter precedence chain.
+ // Filename comes from importFromFile path (basename) or the slug tail
+ // (put_page MCP op fallback). An undated page falls back to a stable
+ // anchor (fallbackCreatedAt): its stored fallback date, else its file's
+ // timestamp, else now — never the latest write. The result drives the recency boost and since/until
+ // filters when callers opt in; nothing in the default search path
+ // consults it.
+ const filenameForChain = opts.filename ?? slug.split('/').pop() ?? slug;
+ const nowDate = new Date();
+ const { date: effectiveDate, source: effectiveDateSource } = computeEffectiveDate({
+ slug,
+ frontmatter: parsed.frontmatter,
+ filename: filenameForChain,
+ timeZone,
+ updatedAt: existing?.updated_at ?? nowDate,
+ createdAt: fallbackCreatedAt({ existing, fileTimes: opts.fileTimes, now: nowDate }),
+ });
+
+ const demote = !segmented && bodyStorageOf(existing) === 'segmented';
+ // Before the page write: an unpinned version reads the page itself, a demoted one references the generation it retires.
+ if (existing && (!preimage || demote)) await tx.createVersion(slug, preimage ? { ...txOpts, preimage } : txOpts);
+ if (demote) await demoteSegmentedPage(tx, Number(existing!.id));
+ // Every import keeps each stored row that is identical to its new chunk and
+ // writes only the rest, so an edit rewrites only the chunks it changed. The
+ // reuse is re-read here, under the page lock, so a kept row is one this
+ // transaction sees; --no-embed and deferred-embed imports keep vectors too.
+ const fresh = reusable ? chunks.map(chunk => ({ ...chunk })) : chunks;
+ const kept = new Map([...(fresh !== chunks ? await reuseStoredVectors(tx, fresh, true) : [])]
+ .filter((entry): entry is [number, KeptChunkRow] => entry[1] !== null));
+ const keptIds = [...kept.values()].map(row => row.id);
+ // #7172: a kept row whose chunk moved changes index in place; a managed publication over CHUNK_VECTOR_WRITE_BUDGET
+ // vector-index writes deletes the rows it does not keep, commits the page pending and finishes after commit (chunk-moves.ts).
+ const chunkWrites = planImportChunkWrites(fresh, kept, !segmented && !!opts.prepare && opts.boundedChunkWrites === true);
+ const deferChunks = chunkReplacementDeferred = chunkWrites.defer;
+ let written: Page | undefined;
+ await pipelined(tx, [
+ () => assertPreparedFactWithdrawals(tx, txOpts.sourceId, parsed.compiled_truth, parsed.timeline || '', slug),
+ async () => { if (existing && preimage && !demote) await tx.createVersion(slug, { ...txOpts, preimage }); },
+ async () => {
+ if (segmented) return segmented.apply(tx, { type: parsed.type, title: parsed.title, frontmatter: parsed.frontmatter, page_kind: 'markdown',
+ effective_date: effectiveDate, effective_date_source: effectiveDateSource, import_filename: filenameForChain, source_path: opts.sourcePath ?? null,
+ source_kind: opts.source_kind ?? null, source_uri: opts.source_uri ?? originUri, ingested_via: opts.ingested_via ?? null }, { mode: effectiveCRMode, corpusGeneration });
+ written = await tx.putPage(slug, {
+ type: parsed.type,
+ title: parsed.title,
+ compiled_truth: parsed.compiled_truth,
+ timeline: parsed.timeline || '',
+ frontmatter: parsed.frontmatter,
+ content_hash: hash,
+ effective_date: effectiveDate,
+ effective_date_source: effectiveDateSource,
+ import_filename: filenameForChain,
+ // Preserve the repo-relative path for sync's delete/rename handling.
+ source_path: opts.sourcePath ?? null,
+ // v0.39.3.0 provenance write-through (WARN-8). Engine layer applies
+ // COALESCE-preserve UPDATE so omitting these on a later put_page
+ // doesn't erase the original ingestion's audit trail.
+ source_kind: opts.source_kind ?? null,
+ source_uri: opts.source_uri ?? originUri,
+ ingested_via: opts.ingested_via ?? null,
+ // ingested_at is server-stamped at the engine layer when any
+ // provenance write fires; never client-controlled.
+ // Empty-overwrite escape hatch only when the caller vouched (file
+ // import / explicit allow_empty); otherwise the engine guard stays on.
+ }, { ...txOpts, ...(opts.allowEmptyOverwrite === true ? { allowEmptyOverwrite: true } : {}), inline: true,
+ ...(effectiveCRMode !== 'none' ? { contextualRetrieval: { mode: effectiveCRMode, corpusGeneration } } : {}) });
+ // v0.40.3.0: the contextual retrieval state is stamped with the page write (the `none` tier's
+ // stamp, which also clears tier vectors, follows it). Deferred embedders reproduce it; every
+ // chunk row is replaced below. This stamp certifies no vector: embedding_signature stays deferred.
+ if (effectiveCRMode === 'none') await tx.updatePageContextualRetrievalState(slug, sourceId ?? 'default', effectiveCRMode, corpusGeneration);
+ },
+ // Replace every derived row atomically. Only vectors the A13 reuse gate above admitted carry over; a new
+ // seal otherwise inherits nothing from an older index, whose contextual vector may have included a private sibling.
+ async () => segmented ? undefined : keptIds.length ? tx.executeRaw(`DELETE FROM content_chunks WHERE page_id = (SELECT id FROM pages WHERE slug = $1 AND source_id = $2)
AND NOT (id = ANY($3::int[]))`, [slug, txOpts.sourceId, keptIds]) : tx.deleteChunks(slug, txOpts),
- // Alias projection and readback share the page commit. A later writer can
- // no longer turn a successful import into a postcommit verification error.
- () => writePageAliases(tx, slug, sourceId ?? 'default', parsed, opts.activePack, mentionPolicy),
- ]);
- const pageId = segmented?.applied ? segmented.applied.pageId : written && written.deleted_at == null ? written.id : undefined;
- const deferSeal = opts.coordinated === true && pageId !== undefined && !segmented;
- await pipelined(tx, [
- async () => {
- // Tag reconciliation (A14): frontmatter tags carry tag_source='frontmatter'.
- // A frontmatter-owned row whose tag left the frontmatter is deleted; rows
- // explicit adds own ('added': add_tag, enrichment, the code importer) and
- // legacy NULL rows are never deleted here (#1621: reindex must not wipe
- // enrichment). A legacy row still in the frontmatter is adopted. Prepared
- // (managed) imports keep the add-only union their canonical file renders.
- if (!opts.prepare) await tx.executeRaw(`DELETE FROM tags t USING pages p WHERE p.id = t.page_id AND p.source_id = $1 AND p.slug = $2
+ // Alias projection and readback share the page commit. A later writer can
+ // no longer turn a successful import into a postcommit verification error.
+ () => writePageAliases(tx, slug, sourceId ?? 'default', parsed, opts.activePack, mentionPolicy),
+ ]);
+ const pageId = segmented?.applied ? segmented.applied.pageId : written && written.deleted_at == null ? written.id : undefined;
+ const deferSeal = opts.coordinated === true && pageId !== undefined && !segmented;
+ await pipelined(tx, [
+ async () => {
+ // Tag reconciliation (A14): frontmatter tags carry tag_source='frontmatter'.
+ // A frontmatter-owned row whose tag left the frontmatter is deleted; rows
+ // explicit adds own ('added': add_tag, enrichment, the code importer) and
+ // legacy NULL rows are never deleted here (#1621: reindex must not wipe
+ // enrichment). A legacy row still in the frontmatter is adopted. Prepared
+ // (managed) imports keep the add-only union their canonical file renders.
+ if (!opts.prepare) await tx.executeRaw(`DELETE FROM tags t USING pages p WHERE p.id = t.page_id AND p.source_id = $1 AND p.slug = $2
AND t.tag_source = 'frontmatter' AND NOT (t.tag = ANY($3::text[]))`, [txOpts.sourceId, slug, parsed.tags]);
- if (pageId !== undefined) return addPageTags(tx, Number(pageId), parsed.tags, opts.prepare ? undefined : 'frontmatter');
- for (const tag of parsed.tags) await tx.addTag(slug, tag, opts.prepare ? txOpts : { ...txOpts, tagSource: 'frontmatter' });
- },
- async () => {
- if (deferChunks) return;
- if (chunks.length > 0) {
- await applyMoveWaves(tx, Number(pageId), chunkWrites.waves);
- const embeddingColumn = await stampEmbeddingInputs(tx, fresh, null,
- { title: parsed.title, tier: effectiveCRMode === 'title' ? 'title' : 'none', corpusGeneration });
- await tx.upsertChunks(slug, kept.size ? fresh.filter((_, i) => !kept.has(i)) : fresh, { ...txOpts, ...(embeddingColumn ? { embeddingColumn } : {}), sealChunkerVersion: MARKDOWN_CHUNKER_VERSION,
- ...(pageId !== undefined ? { pageId: Number(pageId) } : {}), ...(deferSeal ? { deferSeal: true as const } : {}) });
- // v0.41.31: stamp embedding provenance when this import actually
- // embedded (not --no-embed), so a later model/dims swap is detectable
- // as stale via embed --stale. The deferred/backfill + per-slug embed
- // paths stamp too; this covers the inline import/sync path.
- if (!noEmbed && !postCommitEmbedding && !embeddingDeferred) {
- // D9: signature is null when the gateway is unconfigured — skip the
- // stamp (a wrong signature is worse than none).
- const importSig = currentEmbeddingSignature();
- if (importSig) await tx.setPageEmbeddingSignature(slug, { sourceId, signature: importSig });
+ if (pageId !== undefined) return addPageTags(tx, Number(pageId), parsed.tags, opts.prepare ? undefined : 'frontmatter');
+ for (const tag of parsed.tags) await tx.addTag(slug, tag, opts.prepare ? txOpts : { ...txOpts, tagSource: 'frontmatter' });
+ },
+ async () => {
+ if (deferChunks) return;
+ if (chunks.length > 0) {
+ await applyMoveWaves(tx, Number(pageId), chunkWrites.waves);
+ const embeddingColumn = await stampEmbeddingInputs(tx, fresh, null,
+ { title: parsed.title, tier: effectiveCRMode === 'title' ? 'title' : 'none', corpusGeneration });
+ await tx.upsertChunks(slug, kept.size ? fresh.filter((_, i) => !kept.has(i)) : fresh, { ...txOpts, ...(embeddingColumn ? { embeddingColumn } : {}), sealChunkerVersion: MARKDOWN_CHUNKER_VERSION,
+ ...(pageId !== undefined ? { pageId: Number(pageId) } : {}), ...(deferSeal ? { deferSeal: true as const } : {}) });
+ // v0.41.31: stamp embedding provenance when this import actually
+ // embedded (not --no-embed), so a later model/dims swap is detectable
+ // as stale via embed --stale. The deferred/backfill + per-slug embed
+ // paths stamp too; this covers the inline import/sync path.
+ if (!noEmbed && !postCommitEmbedding && !embeddingDeferred) {
+ // D9: signature is null when the gateway is unconfigured — skip the
+ // stamp (a wrong signature is worse than none).
+ const importSig = currentEmbeddingSignature();
+ if (importSig) await tx.setPageEmbeddingSignature(slug, { sourceId, signature: importSig });
+ }
+ } else {
+ // Seal only the completed, full-body sanitized replacement (the upsert above seals its own); a
+ // body-only write or a failed transaction must never certify old stored fragments.
+ if (!deferSeal && !segmented) await tx.executeRaw('UPDATE pages SET chunker_version = $1 WHERE source_id = $2 AND slug = $3', [MARKDOWN_CHUNKER_VERSION, txOpts.sourceId, slug]);
}
- } else {
- // Seal only the completed, full-body sanitized replacement (the upsert above seals its own); a
- // body-only write or a failed transaction must never certify old stored fragments.
- if (!deferSeal && !segmented) await tx.executeRaw('UPDATE pages SET chunker_version = $1 WHERE source_id = $2 AND slug = $3', [MARKDOWN_CHUNKER_VERSION, txOpts.sourceId, slug]);
- }
- },
- ]);
- if (!opts.coordinated && !segmented && !deferChunks) await sealPageTextProjection(tx, slug, txOpts.sourceId);
- if (!opts.coordinated && segmented?.applied?.complete) await sealSegmentedPage(tx, segmented.applied.pageId);
-
- // v0.19.0 E1 — doc↔impl linking: if this markdown page cites code paths
- // (e.g. 'src/core/sync.ts:42'), create bidirectional edges to the code
- // page, all in one statement (addCodeReferenceLinks). An edge whose code
- // page is missing is skipped — guides imported before their code repo
- // syncs are common, and the missing edges land later via
- // `gbrain reconcile-links` (Layer 8 D3, v0.21.0).
- const codeRefs = extractCodeRefs(sanitizeRemoteBody(parsed.compiled_truth) + '\n' + sanitizeRemoteBody(parsed.timeline || ''));
- // For doc↔impl edges, both endpoints are within the same source as the
- // markdown page being imported. Cross-source edges (markdown in one
- // source, code in another) are skipped as missing — never the pre-fix
- // cross-product fan-out, which silently wired edges to whichever
- // same-slug page Postgres returned first across sources.
- if (codeRefs.length) {
- await tx.transaction(savepoint => addCodeReferenceLinks(savepoint, slug, sourceId || 'default',
- codeRefs.map(ref => ({ codeSlug: slugifyCodePath(ref.path), path: ref.path, line: ref.line })))).catch(() => undefined);
- }
- if (!opts.coordinated) await verifyPageReadable(tx, slug, hash, sourceId, 'importFromContent');
- await opts.beforeCommit?.(tx, slug);
- await disposition.persistReceipt(tx, { existed: !!existing });
- if (postCommitEmbedding && !noEmbed && chunks.length > 0) {
- // Capture the complete installed projection before releasing the page
- // guard. Deferred provider results cannot replace newer text or chunks.
- persistedProjection = await readProjectionSnapshot(tx, slug, txOpts.sourceId);
- }
- if (segmented?.applied) return { pageId, sealed: !opts.coordinated && segmented.applied.complete, segmented: { complete: segmented.applied.complete } };
- // The reason routes a resident or drain rebuild of this revision to the bounded continuation.
- if (deferChunks) await queuePageProjection(tx, txOpts.sourceId, slug, DEFERRED_CHUNK_REPLACEMENT);
- if (deferChunks) return { pageId, sealed: false, projectionPending: true as const };
- return { pageId, sealed: !opts.coordinated && !opts.beforeCommit, ...(deferSeal ? { chunkerSeal: MARKDOWN_CHUNKER_VERSION } : {}) };
- };
- if (opts.prepare) return opts.prepare({
- slug, parsedPage, observedRevision: (existing as (typeof existing & { knowledge_revision?: string }) | null)?.knowledge_revision ?? null,
- noop: false, contentHash: hash, result: { slug, status: 'imported', chunks: segmented ? segmented.prepared.rows.chunks : chunks.length, parsedPage,
- ...(pageQuarantined ? { quarantined: true } : {}), ...(pageFlagged ? { flagged: true, flag_reason: pageFlagReason } : {}), ...(disposition.gate ? { gate: disposition.gate } : {}), ...fences.fields },
- validate: tx => assertPreparedFactWithdrawals(tx, txOpts.sourceId, parsed.compiled_truth, parsed.timeline || '', slug),
- apply: applyPrepared, ...(segmented ? { afterCommit: async (target: BrainEngine) => { await segmented.afterCommit(target); }, segmented }
- : opts.boundedChunkWrites ? { afterCommit: async (target: BrainEngine) => { if (chunkReplacementDeferred) await continueDeferredProjection(target, txOpts.sourceId, slug); } } : {}),
- });
- await withUnmanagedBeliefs(engine, txOpts.sourceId, [slug], 'import', run => maintenanceTransaction(engine, tx => run(tx, () => applyPrepared(tx)), writeTrustOfGate(opts.writeGate)).catch(async (err: unknown) => {
- await abortSegmentedStaging(engine, segmented?.prepared);
- // #4287: name the dimension-mismatch rollback instead of letting the bare
- // pgvector message ("expected N dimensions, not M") surface with no code,
- // no consequence and no fix. S2: name the registry-ACTIVE column the
- // write actually targeted (best-effort — plane-agnostic wording when the
- // registry itself is unreadable).
- let activeColName: string | undefined;
- const msg = err instanceof Error ? err.message : String(err);
- if (/expected \d+ dimensions, not \d+/.test(msg)) {
- try {
- const { resolveActiveEmbeddingColumnFromEngine } = await import('./search/embedding-column.ts');
- activeColName = (await resolveActiveEmbeddingColumnFromEngine(engine, { fallbackToLegacy: true })).name;
- } catch { /* keep the plane-agnostic wording */ }
- }
- throw decorateEmbeddingDimError(err, slug, activeColName);
- }));
- const searchable = await segmented?.afterCommit(engine);
-
- if (postCommitEmbedding && !noEmbed && chunks.length > 0) {
- const importScope = currentEngineScope();
- postCommitEmbedding(() => runInEngineScope(importScope, async (): Promise<ImportEmbeddingResult> => {
- try {
- if (!persistedProjection) return { status: 'superseded' };
- const signature = currentEmbeddingSignature();
- await embedChunks();
- const installed = await installPageEmbeddings(engine, persistedProjection, chunks, embeddingPartial ? undefined : signature ?? undefined);
- return { status: installed.complete ? 'embedded' : 'superseded' };
- } catch {
- return { status: 'failed', error: 'Page content was saved, but embedding failed. Check the embedding provider and database on the brain host, then run gbrain embed --stale --source <source-id>.' };
+ },
+ ]);
+ if (!opts.coordinated && !segmented && !deferChunks) await sealPageTextProjection(tx, slug, txOpts.sourceId);
+ if (!opts.coordinated && segmented?.applied?.complete) await sealSegmentedPage(tx, segmented.applied.pageId);
+
+ // v0.19.0 E1 — doc↔impl linking: if this markdown page cites code paths
+ // (e.g. 'src/core/sync.ts:42'), create bidirectional edges to the code
+ // page, all in one statement (addCodeReferenceLinks). An edge whose code
+ // page is missing is skipped — guides imported before their code repo
+ // syncs are common, and the missing edges land later via
+ // `gbrain reconcile-links` (Layer 8 D3, v0.21.0).
+ const codeRefs = extractCodeRefs(sanitizeRemoteBody(parsed.compiled_truth) + '\n' + sanitizeRemoteBody(parsed.timeline || ''));
+ // For doc↔impl edges, both endpoints are within the same source as the
+ // markdown page being imported. Cross-source edges (markdown in one
+ // source, code in another) are skipped as missing — never the pre-fix
+ // cross-product fan-out, which silently wired edges to whichever
+ // same-slug page Postgres returned first across sources.
+ if (codeRefs.length) {
+ await tx.transaction(savepoint => addCodeReferenceLinks(savepoint, slug, sourceId || 'default',
+ codeRefs.map(ref => ({ codeSlug: slugifyCodePath(ref.path), path: ref.path, line: ref.line })))).catch(() => undefined);
+ }
+ if (!opts.coordinated) await verifyPageReadable(tx, slug, hash, sourceId, 'importFromContent');
+ await opts.beforeCommit?.(tx, slug);
+ await disposition.persistReceipt(tx, { existed: !!existing });
+ if (postCommitEmbedding && !noEmbed && chunks.length > 0) {
+ // Capture the complete installed projection before releasing the page
+ // guard. Deferred provider results cannot replace newer text or chunks.
+ persistedProjection = await readProjectionSnapshot(tx, slug, txOpts.sourceId);
+ }
+ if (segmented?.applied) return { pageId, sealed: !opts.coordinated && segmented.applied.complete, segmented: { complete: segmented.applied.complete } };
+ // The reason routes a resident or drain rebuild of this revision to the bounded continuation.
+ if (deferChunks) await queuePageProjection(tx, txOpts.sourceId, slug, DEFERRED_CHUNK_REPLACEMENT);
+ if (deferChunks) return { pageId, sealed: false, projectionPending: true as const };
+ return { pageId, sealed: !opts.coordinated && !opts.beforeCommit, ...(deferSeal ? { chunkerSeal: MARKDOWN_CHUNKER_VERSION } : {}) };
+ };
+ if (opts.prepare) return opts.prepare({
+ slug, parsedPage, observedRevision: (existing as (typeof existing & { knowledge_revision?: string }) | null)?.knowledge_revision ?? null,
+ noop: false, contentHash: hash, result: { slug, status: 'imported', chunks: segmented ? segmented.prepared.rows.chunks : chunks.length, parsedPage,
+ ...(pageQuarantined ? { quarantined: true } : {}), ...(pageFlagged ? { flagged: true, flag_reason: pageFlagReason } : {}), ...(disposition.gate ? { gate: disposition.gate } : {}), ...fences.fields },
+ validate: tx => assertPreparedFactWithdrawals(tx, txOpts.sourceId, parsed.compiled_truth, parsed.timeline || '', slug),
+ apply: applyPrepared, ...(segmented ? { afterCommit: async (target: BrainEngine) => { await segmented.afterCommit(target); }, segmented }
+ : opts.boundedChunkWrites ? { afterCommit: async (target: BrainEngine) => { if (chunkReplacementDeferred) await continueDeferredProjection(target, txOpts.sourceId, slug); } } : {}),
+ });
+ await withUnmanagedBeliefs(engine, txOpts.sourceId, [slug], 'import', run => maintenanceTransaction(engine, tx => run(tx, () => applyPrepared(tx)), writeTrustOfGate(opts.writeGate)).catch(async (err: unknown) => {
+ await abortSegmentedStaging(engine, segmented?.prepared);
+ // #4287: name the dimension-mismatch rollback instead of letting the bare
+ // pgvector message ("expected N dimensions, not M") surface with no code,
+ // no consequence and no fix. S2: name the registry-ACTIVE column the
+ // write actually targeted (best-effort — plane-agnostic wording when the
+ // registry itself is unreadable).
+ let activeColName: string | undefined;
+ const msg = err instanceof Error ? err.message : String(err);
+ if (/expected \d+ dimensions, not \d+/.test(msg)) {
+ try {
+ const { resolveActiveEmbeddingColumnFromEngine } = await import('./search/embedding-column.ts');
+ activeColName = (await resolveActiveEmbeddingColumnFromEngine(engine, { fallbackToLegacy: true })).name;
+ } catch { /* keep the plane-agnostic wording */ }
}
+ throw decorateEmbeddingDimError(err, slug, activeColName);
}));
- }
+ const searchable = await segmented?.afterCommit(engine);
+
+ if (postCommitEmbedding && !noEmbed && chunks.length > 0) {
+ const importScope = currentEngineScope();
+ postCommitEmbedding(() => runInEngineScope(importScope, async (): Promise<ImportEmbeddingResult> => {
+ try {
+ if (!persistedProjection) return { status: 'superseded' };
+ const signature = currentEmbeddingSignature();
+ await embedChunks();
+ const installed = await installPageEmbeddings(engine, persistedProjection, chunks, embeddingPartial ? undefined : signature ?? undefined);
+ return { status: installed.complete ? 'embedded' : 'superseded' };
+ } catch {
+ return { status: 'failed', error: 'Page content was saved, but embedding failed. Check the embedding provider and database on the brain host, then run gbrain embed --stale --source <source-id>.' };
+ }
+ }));
+ }
- return {
- slug,
- status: 'imported',
- chunks: segmented ? segmented.prepared.rows.chunks : chunks.length,
- parsedPage,
- ...(pageQuarantined ? { quarantined: true } : {}),
- ...(pageFlagged ? { flagged: true, flag_reason: pageFlagReason } : {}),
- ...(typeWarning ? { type_warning: typeWarning } : {}),
- ...(embeddingDeferred ? { embedding_deferred: true } : {}), ...fences.fields, ...(searchable !== undefined ? { searchable } : {}),
+ return {
+ slug,
+ status: 'imported',
+ chunks: segmented ? segmented.prepared.rows.chunks : chunks.length,
+ parsedPage,
+ ...(pageQuarantined ? { quarantined: true } : {}),
+ ...(pageFlagged ? { flagged: true, flag_reason: pageFlagReason } : {}),
+ ...(typeWarning ? { type_warning: typeWarning } : {}),
+ ...(embeddingDeferred ? { embedding_deferred: true } : {}), ...fences.fields, ...(searchable !== undefined ? { searchable } : {}),
+ };
};
+ return await (needsEngineScope(engine) ? withEngineScope(engine, importScreened) : importScreened());
}
/**