Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

docs: key-files entry for repair segmented-tiling names its import provenance record

garrytan3054fa3e155a
diff --git a/docs/architecture/key-files/core-services-2-continued.md b/docs/architecture/key-files/core-services-2-continued.md
index 0eb34fcee7..e5606c4b85 100644
--- a/docs/architecture/key-files/core-services-2-continued.md
+++ b/docs/architecture/key-files/core-services-2-continued.md
@@ -35,7 +35,7 @@ The page-state concurrency foundation under `src/core/page-state/` and the share
- `src/core/persistence/large-file.ts` — Wave B PR 2 fix lane D: canonical files over 24 MiB (`LARGE_FILE_MIN_BYTES`) on the publication path, never one string or Buffer. `hashFile`/`hashDescriptor` (the persistence file hash, read in one-segment pieces), `FileBytes` (a file read on demand through one descriptor, the `ByteSource` the drift check's `fileBodyDigests` takes; `close` refuses `source_changed` when the file changed while read), `StreamedFileContent` (restore_page's file: export's serializer over the segments, measured once in preparation, produced again by `stageStreamedContent` into the reserved publication stage after the recovery record is durable and before the publication transaction: streamed into an in-progress sibling (`streamingStagePath`, `<stage>.streaming`), verified, fsynced and only then renamed to the stage's name, so a stream a crash cut short is never at that name and recovery removes it unread; a mismatch removes the stream and refuses `revision_conflict`; seam `large_file:stage_streaming` after each piece), and `stageBeforeCopy` (the prior file copied by the kernel into its `before` stage, hashed and fsynced; the recovery record names it by path and SHA-256 instead of embedding base64, and reserves the file once). `largeFileWork` (@internal) counts bytes read, written and copied and the largest piece. Coordinator: `pageRecoveryRecord` → `largeRecoveryRecord`, `stageLargeFiles`, `publishStagedContent`; a group publication never takes such a member (`groupable`). Flip lane A: the withdrawal mirror of a segmented page (`effects.ts` `mirrorPage`, any size) renders its file with `page-prepare.ts` `unreadPageFile` (a `StreamedFileContent`) and writes it with `stageStreamedContent` before its recovery transaction (`effect-recovery.ts` `stageStreamedMirror`); `existsAt` is the lstat-based leaf check it uses. `prepareFileTarget` reads the prior file through `FileBytes` at any size for a page stored in segments or a streamed target (never `readFileSync` whole), so its drift check reads in pieces of at most a segment. Test: `test/large-pages/segmented-withdrawal-mirror.test.ts`. Tests: `test/large-pages/segmented-delete-restore-file.slow.test.ts` (27 and 90 MiB round trips byte for byte, both engines), `test/large-pages/segmented-delete-crash.slow.test.ts` (SIGKILL between file removal and commit, and after the restoration rename).
- `src/core/page-body/body-digests.ts` — Wave B PR 2: the digests the drift check of a segmented page over the single-string ceiling compares (`page-prepare.ts` `segmentedFileMatches`). `fileBodyDigests(bytes, reading, filePath, opts)` runs the streamed importer's pass 1 (`BodyScanner`) over one reading of the file's bytes (an inferred-frontmatter prefix, then the bytes past a BOM), then SHA-256 over each region's byte ranges, and reports a facts fence marker; `storedBodyDigests(engine, pageId, ledger)` hashes each region of the trusted stored body (`readBodyRange`, one segment per read, `readGarbageCollector`). `bodyDigestWork` counts file and stored bytes and the largest piece handed to a scan or hash (at most a segment). `overlaidFileBodyDigests` overlays the file one complete fence block at a time; inside an open block a read stops at FENCE_MAX_BYTES plus `FENCE_READ_LOOKAHEAD` (4 bytes), so an over-bound fence refuses `page_body_segmented` without reading a further segment (`test/pathological/lp-g-file-digest-fence-bound.test.ts`). In `page-prepare.ts`, `prepareFileTarget` compares a page whose body was not read back through `segmentedFileMatches` (the raw reading, then the inferred one from `streamed-import.ts` `inferredHeadPrefix`; the same normalizations as `fileMatchesSnapshot` via `importedFileType`); a file body with a facts fence under a withdrawal ledger that applies refuses `page_body_segmented` instead of reading as drift. `restore_page` of such a page (`prepareUnreadRestore`) versions it, clears `deleted_at` and reseals (or leaves the projection queued when chunks are missing) without re-importing, and publishes `serializePageToMarkdown`'s bytes streamed from the segments by `segmentedExportMarkdown` into one buffer (under the 25 MiB pre-push scan cap); a core page and an external_untrusted writer refuse `page_body_segmented`, and an agent-tier writer's advisory write-gate scan is not re-run over the unchanged body. Tests: `test/large-pages/segmented-file-drift.test.ts`.
- `src/core/page-body/body-tiling.ts` (`untiledSegmentedPages`) + `schema.ts` (`liveTilingSql`, `untiledSql`) + `src/core/engine-sql/page-body.ts` — GST-1209: a live generation tiles its body when its manifest rows, as every reader selects them, cover `body_bytes` from byte 0 with no gap. `flipBodyGeneration` checks the generation it made live inside its transaction and refuses one that does not tile (`body_staging_expired`, detail reason `generation_incomplete`, nothing published). `setBodyStagingState` moves only a live staging (open, sealed, prepared): a move on to prepared or committed of a staging a fact purge aborted or a sweep expired (its staged rows deleted) refuses `body_staging_expired` instead of reviving it. `untiledSegmentedPages` lists live segmented pages of unarchived sources that do not tile (read-only, one aggregate per page), for doctor `segmented_tiling` (`src/commands/doctor/checks/segmented-tiling.ts`, BRAIN, host only, its own registry entry `segmentedTilingEntry` in `checks/graph-health.ts` so the wave checks' budget never hides it; the fix always names `--source`) and the repair (`survey` lists at most `PREVIEW_PAGE_LIMIT` pages and reports `truncated`). A sync hold naming `page_body_rows_unreadable` reads needs_human / repair and its `gitHoldFix` previews `gbrain repair segmented-tiling --source <id> --slug <slug>`.
-- `src/core/repair/segmented-tiling.ts` — `gbrain repair segmented-tiling --source <id> [--slug <slug>]` (GST-1209), explicit-only, preview-bound, trusted local CLI on the owner host, `checks: ['segmented_tiling']`. The preview lists each untiled page with its file, HEAD's blob (oid, SHA-256, size) and sync hold under a hash; a page with no file, a file HEAD lacks or one over `pages.max_bytes`, a file another sync hold withholds (any but the unreadable-body `page_write_failed` one) or one `unreadPageImportScreen` (sync-screen.ts, the next sync's screen against the unread snapshot) refuses is listed held; a source with no Git checkout refuses `not_a_git_repo`. `--apply --expect` re-checks revision, generation and blob (`preview_changed`), then per page in one transaction under the key lock (coordinated maintenance on a managed brain) versions the damaged generation, stages the screened body after `preserveWithdrawnFenceRows` (purged claims stay out) through `prepareSegmentedBody` without reading the damaged body, flips with `applySegmentedBody` (title, type, frontmatter, tags and trust tier kept), clears the previewed hold, calls `requestGitHoldRetry` and writes an `op_checkpoints` receipt (`segmented-tiling-receipt`) and a `segmented-tiling` audit line. Tests: `test/large-pages/segmented-tiling-repair.test.ts`.
+- `src/core/repair/segmented-tiling.ts` — `gbrain repair segmented-tiling --source <id> [--slug <slug>]` (GST-1209), explicit-only, preview-bound, trusted local CLI on the owner host, `checks: ['segmented_tiling']`. The preview lists each untiled page with its file, HEAD's blob (oid, SHA-256, size) and sync hold under a hash; a page with no file, a file HEAD lacks or one over `pages.max_bytes`, a file another sync hold withholds (any but the unreadable-body `page_write_failed` one) or one `unreadPageImportScreen` (sync-screen.ts, the next sync's screen against the unread snapshot) refuses is listed held; a source with no Git checkout refuses `not_a_git_repo`. `--apply --expect` re-checks revision, generation and blob (`preview_changed`), then per page in one transaction under the key lock (coordinated maintenance on a managed brain) versions the damaged generation, stages the screened body after `preserveWithdrawnFenceRows` (purged claims stay out) through `prepareSegmentedBody` without reading the damaged body, flips with `applySegmentedBody` (title, type, frontmatter, tags and trust tier kept), clears the previewed hold, records the new revision as imported from the HEAD blob (`recordSyncImportProvenance`) when the file imports to the page's stored title, type, frontmatter and tags (so the next sync holds no `file_drift`), calls `requestGitHoldRetry` and writes an `op_checkpoints` receipt (`segmented-tiling-receipt`) and a `segmented-tiling` audit line. Tests: `test/large-pages/segmented-tiling-repair.test.ts`.
- `src/core/page-body/segment-facts.ts` + `segment-conversation.ts` — Wave B PR 2 lane 2e2 (spec 8, full-pass mode): the facts fence reconcile and conversation facts over a segmented body. `segmentedFactsFenceView` is `factsFenceView` (facts-fence.ts: each section's parsed fence, which sections hold one, and the #3625 timeline marker) read from the segments: one SQL query selects the live manifest rows that carry facts slots, a forced cut, or a begin/end marker (bytea `position`, nothing transferred), and `bodyBlocks`' `want` option reads only the runs holding them; the segmenter never cuts inside a fence, so a fence lies in one block (a longer one is a forced run, refused above `BLOCK_MAX_BYTES` with `page_body_segmented`), and joining one across blocks is a bounded fallback. `segmentedBodyMatchesFile` is `canonicalCacheState`'s stale-cache check for a segmented page: the canonical file streamed through `BodyScanner` against the stored regions, by length and SHA-256. `readSegmentedConversationInput` gives `readConversationInput` (conversation-parser/body.ts) the summary body through ranged reads: whole within `CONVERSATION_WINDOW_BYTES` (`PAGE_MAX_BYTES`), else the newest whole segments that fit (never starting inside a forced run) with the window's `fallbackDate` from its first segment's entry state; extract-conversation-facts parses that window, passes the date to the parse and the extractor, and writes a `tail_window` receipt (`window_rule: no_prior_facts`), but only on a page with no conversation facts: one that has any keeps them and is skipped (`skipped_too_large`, `window_rule: prior_facts_kept`), since the run clears every prior fact. Each loop counts its reads in `segmentedPassWork`. Consumers: `runExtractFacts` (`planFenceReconcile`, the stale-cache guard, the managed fence-facts request), the sweep's fence candidates (segmented pages with facts slots), `extract-conversation-facts`. Tests: `test/large-pages/segmented-facts-reconcile.test.ts`, `segmented-conversation-facts.test.ts`, `segmented-derived-linear.test.ts`.
- `src/core/page-window/bounded-lead.ts` — leads, snippets and synopses read a bounded prefix, never the whole body: fan-out SQL selects `boundedBodyPrefixSql(alias)` (`left(coalesce(alias.compiled_truth, alias.truth_lead), 65536)`: a segmented page's NULL body falls back to its stored lead; the one COALESCE of body and lead the page body reader ratchet allows by name), `boundedRawPrefix` cuts a body at 64 KiB on a code point, and `overlaidBoundedRows` / `overlaidBoundedPrefix` replace a prefix holding a `gbrain:facts:begin` marker with the prefix of the withdrawal-overlaid body (one batched `readPageSnapshotsBatch` read; a page it no longer returns, or a failed read, gets an empty lead). A segmented page's row carries its stored `truth_lead`; when that lead holds a facts fence it is overlaid alone with the withdrawal ledger the snapshot read returns (`overlayCanonicalBodies`, then the same 64 KiB cut), never by reading segments. `pageLeadSource` gives getPage callers (exact-slug and alias snippets) a page's compiled truth, or a segmented page's `truth_lead` without touching its throwing body getters. `projectedLead` is that prefix through `boundedProjectedPrefix` (`document.ts`). Consumers: the exact-slug tier (`exact-lookup.ts`) and alias injection (`alias-hop.ts`) snippets, relational hydrate, the reflex pointer synopsis (`retrieval-reflex.ts` `safeSynopsis`: world-only reads take the projected prefix, the trusted entity-card widening the raw prefix through its own fence strip), the entity card's hit row and `entity names` leads, and think's anchor and window-floor rows (`think/gather.ts`: the raw overlaid prefix for the trusted local CLI, the projected prefix otherwise). Small pages render as before. Pinned by `test/bounded-lead.test.ts` and `test/large-pages/segmented-leads.test.ts`.
- `src/core/page-window/document.ts` — the one read projection every remote window, outline and lead slices: `projectPageText` (fence overlay split, `sanitizeRemoteBody` + `credentialSafeProjection`, timeline after `TIMELINE_SEPARATOR`), `pageDocument` / `readPageDocument` (UTF-8 bytes + timeline byte offset, with the page's fence overlay), `chunkDocumentSpan` + `spanHoldsChunk` (a chunk's stored `projected_byte_*` span mapped into the document and verified), `boundedProjectedPrefix`, `locateChunksByStoredSpans` (byte spans to string offsets in one pass), `DEFAULT_INLINE_BODY_BYTES` (256 KB).