Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript
44.0 KB
Raw Blame

Key files: Core Services (2/3, continued)

Subsystem index · Previous entries. Read only the entries relevant to your change. Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG. The page-state concurrency foundation under src/core/page-state/ and the shared progressive-batch primitive. Dated edition reads (get_page as_of) are in Valid time.

  • src/core/ops/get-page-slice.ts — get_page section / grep / context (#7180), params GET_PAGE_SLICE_PARAMS (full surface only; the get_page description names them on starter, and dispatch accepts them there). ops/get-page-body.ts routes a slice here: parseSliceRequest makes it a window-mode request carrying slice, and readPartialBody, after authorization and the quarantine body_omitted check, passes its read document's stream() (the inline bytes, or SegmentedDocument.stream() one segment at a time) with the window's metadata and size. Both scan the read projection line by line, so a segmented page is never held whole (memory: the returned text, one segment, at most context paragraphs; a line over 1 MiB is scanned in pieces). readSection: the first ATX heading (HEADING, outside FENCE blocks, as the outline sees them) whose whitespace-folded text equals the request case-insensitively (a leading ## is ignored), down to the next heading of the same or higher level or the timeline separator; returns section {heading, level, heading_path, line_start, line_end, text} and stops reading at the section's end; a miss is invalid_params naming up to 20 headings with detail.total_headings and an outline fix. readGrep: blank-line paragraphs or unfenced table rows with a line containing the literal term (case-insensitive, no regex), context paragraphs either side (default 1, clamped to 5), windows that overlap or touch merged; grep {term, context, total_matches, returned_matches, omitted_matches, blocks[{line_start, line_end, match_starts, text}]}. Caps: 50 matching paragraphs and SLICE_MAX_CHARS (100,000), fewer when the egress cap is smaller (sliceCharBudget); truncated says a cap was hit, and a cut section adds omitted_chars and a next window read. Line numbers are the read view's, the ones body: 'window' lines reads back. A slice with a selector, include_content, content_only, view: markdown, a body other than auto/window, or known_as_of is invalid_params. Every get_page shape (whole, content_only, partial, slice) carries chars, the characters of page text it holds (projectGetPage, readPartialBody). Test: test/get-page-slices.test.ts (both engines).

  • src/core/page-state/ — Canonical page concurrency foundation shared by both engines. schema.ts (migration 150) adds source incarnation UUIDs, logical page revisions, text-projection revisions, and persistent guards for absent pages. guards.ts locks sorted sources then page identities in an existing transaction; a transaction or savepoint never re-acquires a key it holds, which covers absent pages because the only pages INSERT runs after putPage takes the same guard (test/e2e/page-guards-postgres.test.ts); callers lock sources before authority/request/fact rows. snapshot.ts computes the normalized body/timeline fingerprints only for a page with fact withdrawals, page-subject purges or a source '*' purge marker (they feed only the withdrawal overlay), as page-snapshot-batch.ts does. snapshot.ts reads page/tags/withdrawals in one MVCC statement and applies withdrawals using database-normalized text. transactions.ts makes helper transaction APIs compose through savepoints. versions.ts records the complete canonical snapshot; rename-alias.ts records slug_aliases(old -> new) inside both engines' updateSlug transaction, repoints aliases that named the old slug and drops one spelled like the new slug, and keeps every move (and any alias it drops) in slug_rename_history (v285), which only the folder-rule lineage test reads; moveSlugBindings moves the page's facts (entity_slug, source_markdown_slug) and page_aliases in the same transaction, letting a purged page's leftovers at the new slug yield (test/rename-slug-alias.test.ts); movePageIdReferences (#5431) moves the page-id-keyed rows of a stale rename row onto the surviving row (links from/to/origin, timeline_entries page and event, page_versions), dropping stale-side duplicates on the links composite key and the timeline dedup indexes (test/rename-slug-alias.test.ts); materialize.ts persists only verified canonical overlay bytes without another logical revision. types.ts defines revision preconditions: no revision is create-only (an existing page refuses revision_required, #5385, its own code since the omission is the caller's and not a race), a stale revision or a page another writer created where an internal writer asserted none (expectedRevision: null) is revision_conflict, explicit force overwrites. Plain legacy putPage calls keep compatibility until their owning operation adopts the managed write protocol. projection-schema.ts (migration 153) invalidates unverified prior projections once and queues durable work, but a Markdown or code page at or after the safe-chunk fence with no facts or takes fence, private-key text or quarantine marker keeps its old index in search until rebuilt (servedTextProjectionFilter); projections.ts rebuilds Markdown without a worktree/provider and installs only matching canonical revisions. Embedding installs also compare exact chunk identities/text. Source importers seal code/image projections; all read arms hide unsealed projections.

  • src/core/page-state/projections.ts — sealPageTextProjection (defined in the light page-state/text-seal.ts, re-exported here, so engine-sql/chunks.ts sealChunks can run it without loading the projection pipeline) and sealImportedPage write the page keyword vector through pageSearchVectorSql (fts-language.ts: title weight A, sanitized timeline weight C, both through gbrain_fts_input), the same expression search-vector-backfill.ts uses. Captures canonical revision, source incarnation, page identity, the complete raw chunk inventory and resolved indexing inputs before preparing derived work. Embedding reads require a current text seal; healing/rebuild capture can explicitly include unsealed pages. Projection installation compares its originating seal, chunk identities/text/source/completion state, chunker version and indexing context (including stored corpus generation) under the page guard before replacing any row. A same-revision re-chunk may replace the snapshot it actually captured; a delayed older installation cannot discard newer chunks or vectors. Rebuild also rechecks its queued job before capture so a completed job from an old listing stays completed. Chunkless healing sanitizes complete body/timeline fences before splitting, preserving withdrawn history in direct page reads while excluding it from retrieval. Tests: page-projection-origin.test.ts, page-projection-vector-preservation.test.ts (PGLite and isolated Postgres), embed-stale-chunkless-pages.serial.test.ts (public healing entry). Shared markdown-chunks.ts and code-chunks.ts preserve fenced code and source-language metadata. A preserving rebuild keeps a vector only when its content_chunks.embedding_input_hash (src/core/embedding-input-hash.ts) equals one recomputed from the current page, or, with no recorded hash, when the page is non-contextual and the legacy model/text-hash checks pass; a contextual chunk without provenance is nulled once. installPageEmbeddings records the hash in the vector's UPDATE (the plain re-embed tier by default, or the caller's built tier and corpus generation), and returns { installed, complete }: it matches each offered vector to the page's current chunk by source and exact text (not index) and writes it only when its input hash for the captured context is one the current context accepts, in batched jsonb_to_recordset UPDATEs, so a page that changed under the provider call keeps the vectors of its unchanged chunks; the embedding signature is stamped only when complete (same chunk sequence, nothing dropped). A preserving rebuild likewise keys kept vectors by chunk content, so a top insert shifts vectors with their chunks (tests: page-embedding-content-validity.test.ts). Also stampEmbeddingInputs does the same for full imports. resealSafeChunks re-seals a page below the safe-chunk fence from its unchanged body with no canonical write; projectionBelowSafeFence is its predicate. Tests add projection-embedding-input-hash.test.ts and safe-chunk-reseal.test.ts. It replaces outgoing code edges without deleting incoming edges to retained chunks and clears backfill watermarks. Failed jobs rotate with a retained diagnostic; successful drain refreshes planner statistics. rebuildPendingPageProjections(engine, limit, { deadlineMs, now, notAfter, retryCooldown, onFailure }) (#5401) checks an injectable clock between pages, bounds a run to rows queued at or before its start (notAfter, cast from text in SQL so microseconds survive the driver), applies the 30 s rebuild_failed cooldown for the resident, and treats a lock or snapshot error as that page's failure instead of aborting the batch; projectionBacklog returns counts only (pending, failed, oldest age) for the drain command and the resident's projection_status IPC reply. A same-revision conflict throws PageProjectionConflictError naming the fields that changed (text_projection_revision, chunk_digest, indexing_context), the slug and source; retryProjectionConflict(attempt) re-runs an attempt that captures a fresh snapshot at most three more times with jitter, then rethrows (page_projection_conflict on the wire). Tests: page-projection-origin.test.ts (resident rebuild race, two-writer progress, exhausted-retry CLI/JSON output).

  • src/core/page-state/chunk-moves.ts — #7172: CHUNK_VECTOR_WRITE_BUDGET (5), the most new content_chunks row versions carrying a vector (inserts, and non-HOT updates of chunk_index, text or vector; each adds an HNSW entry) one transaction of a managed chunk replacement makes, with its derivation from the publication's 5 s statement bound (a version without a vector is not counted); planMoveWaves plans the in-place renumbering of kept rows within a budget (one UPDATE per wave, a cycle parks one row at -1 - target), applyMoveWaves runs it. import-file.ts (with boundedChunkWrites, set by page-prepare, import-prepare and sync-prepare) commits a page whose replacement exceeds the budget with its projection pending (queue reason chunk_replacement_deferred), and projections.ts continueDeferredProjection finishes it after commit in steps of at most the budget; rebuildPendingPageProjections resumes an interrupted one.

  • src/core/page-state/projection-class.ts — #6803: projectionClassSql(p, s), the one CASE that puts each queued page in exactly one class (archived, text for Markdown/code and segmented pages, protected for a protected fence, sealable per sealableImageSql, else reimport), shared by projectionBacklog, the drain probe, search readiness and embed --stale --images. sealableImageSql: an image page at or after the safe-chunk fence whose one chunk (index 0, image_asset) carries the page's own text.

  • src/core/page-state/image-seal.ts — #6803: sealImageProjections restores the importer's seal on sealable images in groups of 500: a primary-key window of queue rows probed by each page's unique key (a statistics-less PGLite planner otherwise probes through pages_dedup_idx), then under the page guard a revision compare-and-set with the class rechecked; no chunk write, the visual vector untouched, a text vector whose recorded input no longer matches nulled for embed --stale. IMAGE_SEAL_PAGES_PER_SECOND is the measured rate (its comment names the run); projectionDrainLimit(textPending) sizes doctor's --limit to half AGENT_TOOL_LIMIT_MS at that rate, or at the measured TEXT_REBUILD_PAGES_PER_SECOND (rebuild-estimate.ts; it also times v153's notice) while queued text waits (the drain takes text first).

  • src/core/page-state/projection-window.ts — #6803: the resident's bounded queue read: per pass at most RESIDENT_PROBE_BUDGET (500) rows, the 100 newest by updated_at plus a primary-key window from a per-engine cursor that starts at a random key and wraps; a negative cache (100k keys) skips image rows the resident never builds. The consumer's idle probe walks the same window and advances the same cursor. Tests: projection-resident-window.test.ts.

  • src/core/page-state/projection-status.ts — #6803: the resident's cached projectionBacklog for the projection_status IPC op, refreshed when a cheap signal (queue count, max updated_at, source archive flags) moves, checked every 60 s and at most 10 minutes old.

  • src/core/page-state/projection-upgrade-notice.ts — #6803: gbrain post-upgrade's read-only [AGENT] block while sealable or reimport image pages wait, naming gbrain projections drain --dry-run with an estimate at the measured seal rate.

  • src/core/page-state/credential-reseal.ts + src/core/schema-migrations/v189-pages-credential-projection-pending.ts + src/commands/doctor/checks/credential-projection.ts — the targeted re-chunk of pages indexed before the credential-safe projection. Schema migration v187 (on the first connect after upgrade) sets text_projection_revision = NULL on every live markdown/code page whose compiled_truth or timeline matches -----(BEGIN|END) [A-Z ]*PRIVATE KEY----- and upserts a page_projection_jobs row (reason credential_projection), so the current-text-projection seal withholds their old chunks from every retrieval path, local and remote (code-walk hydrateChunks in search/two-pass.ts applies the same filter). The marker literal is frozen as shipped. credentialProjectionPending(engine, sourceIds?) counts pending pages in active sources as rebuildable and kept (code pages with no frontmatter.file or source_path, which wait for their importer). resealCredentialProjections re-seals each rebuildable page from its unchanged body through resealSafeChunks (the repair safe-chunks installer), one page per transaction, with no page write, journal admission or provider call; vectors of unchanged chunk text are kept and re-chunked key chunks wait for gbrain embed --stale; a revision conflict counts as superseded; a page that fails item-scoped (ItemRun site migration.credential_reseal) is held in op_checkpoints credential-reseal-hold keyed by page id (revision, chunker generation, code, redacted message, consecutive attempts) and stays withheld, a run-scoped failure or most pages failing is thrown and records nothing. readCredentialResealHolds returns the holds that still apply (page live, same revision and chunker generation, still unsealed) with each one's fix (gbrain repair safe-chunks --source <id> --apply, or gbrain projections drain for a segmented page); a hold never blocks a retry. CREDENTIAL_RESEAL_HELD lets repair safe-chunks select held pages, and doctor text_projection_readiness warns with credential_reseal_held, held_pages (local callers only) and the safe-chunks fix. The doctor check credential_projection_pending (a WAVE_CHECKS entry, resolution: 'operator', brain category) reports pages_pending, kept_pages and held_pages (with the retry command) and warns while pages_pending or kept_pages is non-zero, naming gbrain apply-migrations --yes --no-autopilot-install (no provider calls), then gbrain embed --stale, and the re-import for kept code pages; a read error is warn with unknown health. A segmented page is pending when a live manifest row stores a pem region; it is re-sealed through buildSegmentedProjection.

    Embedding-only publication also requires a live, unarchived source and nondeleted page under the same source/page row locks. A prepared vector cannot bypass a later archive, model change or withdrawal. test/e2e/migration-wave-prepared-withdrawal.test.ts exercises both publication orders and exact unchanged-vector refusal on both engines.

  • src/core/page-state/version-types.ts — complete canonical version contract with nullable legacy fields, re-exported by types.ts for existing callers. page-state/versions.ts records deletion, timeline, tags and originating revision in the page publication transaction. Wave B: a segmented page's version references its body generation (body_storage: 'segmented', body_generation, body_bytes, content_hash, NULL bodies; O(1)); pruneSegmentedVersions keeps the newest versions.segmented_max_per_page and prunes oldest-first while bytes pinned beyond the live generation exceed max(versions.segmented_max_pinned_bytes, body_bytes), then sweepSegments; get_versions returns their size and, with include_body, a body_ref; revert_version refuses them with page_body_segmented until PR 3. A holding page (isHoldingSlug) keeps its newest LARGE_PAGE_VERSIONS_KEPT inline versions whatever their size, at most HOLDING_PAGE_VERSIONS_PRUNED_PER_WRITE (100) removed per save (pruneHoldingPageVersions, run only by the fence append after a save that adds a fact; a held move and purge, which edit saved versions themselves, never prune) (GST-2497). LARGE_PAGE_VERSIONS_KEPT lives in page-body/limits.ts.

  • src/core/progressive-batch/ — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: types.ts (Stage, StageVerdict, AbortReason, discriminated Verifier union OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier, Policy, StageReport), orchestrator.ts (runProgressiveBatch(items, verifier, policy, runner) — reads getCurrentBudgetTracker() ahead of Policy.maxCostUsd fail-closed; null both ways triggers abort_cost_cap reason='no_budget_safety_net'), audit.ts (ISO-week JSONL at ~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl via the shared audit-writer primitive), stage-report.ts (ASCII formatter for the default Policy.onStageReport). Env knobs: GBRAIN_PROGRESSIVE_BATCH_DISABLED=1, GBRAIN_PROGRESSIVE_BATCH_AUTO=1 (skip Ctrl-C grace), GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via Policy.interactiveAbortMs > 0. Pinned by test/progressive-batch/orchestrator.test.ts (35 cases, every verdict path).

  • src/core/page-body/page-shape.ts — Wave B PR 1 (spec 1.5): segmented page bodies are unreachable as strings. guardSegmentedBody (called by rowToPage for a row with body_storage = 'segmented', so by every snapshot and getPage read) replaces compiled_truth / timeline with enumerable getters and setters that throw page_body_segmented, so property reads, spread, JSON.stringify, structuredClone and assignment all fail loud instead of seeing an empty body. isSegmentedPage, assertInlineBody(page, site) (explicit refusal where a writer would otherwise act before a getter fires, e.g. SQL-only writers; a read's page_body_segmented refusal fixes with get_page body: 'auto'; for a write that fix changes nothing, so asSegmentedWriteRefusal (called by MCP dispatch on a mutating op) swaps it for a report-only step naming edit_page, append_page, add_tag and remove_tag, and readsPageInPieces(fix) tells the two apart, which remember's unattributed-save fix in persistence/verb-errors.ts also uses), pageBodyBytes(page) (body_bytes for segmented pages), pageBodySegmentedError and pageBodyOverCeilingError (the refusal of a fence or timeline write through the put path on a page over 24 MiB; wholeBodySnapshot in persistence/segmented-apply.ts reads a page within it back as strings and refuses above it: semantic timeline writes, takes-prepare.ts, facts-prepare.ts, reconcile-state.ts, gbrain core). Tag writes on a segmented page never read the body: prepareSegmentedTagMutation (persistence/page-prepare.ts) flips them in metadata over the live manifest (test/large-pages/segmented-semantic-writes.test.ts). bodyUnread(page) says a segmented page's body was not read back as strings (over the single-string ceiling, see materializedPage); such a body was written by the streamed import, which holds fenced files, so fenceBody(page) reads it as empty for the fence merges and withoutBody(page) copies the page's other fields for a render whose body comes from the incoming write (a shrunk file rewrites such a page: test/large-pages/segmented-shrink-below-ceiling.test.ts). Page stays one shape (body_storage?, body_bytes?); the static half is scripts/check-page-body-readers.ts with scripts/page-body-readers-baseline.tsv (docs/ci-guards.md#page-body-readers). Pinned by test/large-pages/segmented-body-unreachable.test.ts.

  • src/core/page-body/limits.ts — Wave B size classes, ceilings, retention numbers and config keys in one place. readPageBodyPolicy is the single question every write asks: segmented is the effective pages.segmented_storage (unset means SEGMENTED_STORAGE_DEFAULT, on; off or an unrecognized value is off; held off while segmentedWriterBlockers finds a recent writer that blocks, and an unreadable gate counts as blocked), maxBytes the engine ceiling (256 MiB Postgres, 128 MiB PGLite, lowered by pages.max_bytes) while on and PAGE_MAX_BYTES while off. Config is read per call; only the writer gate is cached (SEGMENTED_WRITER_GATE_TTL_MS, _resetSegmentedWriterGateForTests). chooseBodyStorage applies the 1 MiB / 512 KiB hysteresis. The writer gate (segmentedGateWriters in persistence/writer-versions.ts) reads seven days of admitted requests in any state, committed publications, ingest_log rows and, on Postgres, persistence_consumers heartbeats; a row blocks when its version is below SEGMENTED_WRITER_FLOOR (page-body/writer-floor.ts: the since of the BEHAVIOR_CHANGES row with id segmented_switch_on, this build's VERSION while it is TBD, so the merge-slot stamp edits only that row) or unparsable, or when it lacks the segment-reader mark (migration v254: admitter_segment_reader, consumer_segment_reader, persistence_consumers.segment_reader, ingest_log.writer_segment_reader), because releases 0.60.252.0-0.60.262.0 have no segment readers and a version string cannot tell them apart. Not seen: an older binary's unmanaged single-page write, a run that dies before it logs, and read-only or export use. pageRecoveryBytes is the default persistence.limits.worktree_recovery_bytes (read by persistence/limits.ts readJournalLimits in its one config statement): twice the larger of the body the configured switch admits (segmentedMaxBytes, or PAGE_MAX_BYTES while off) and the largest page stored in segments (deleted pages included; one probe of pages_segmented_body_bytes_idx, v260, or the engine ceiling while that index is not valid, so a limit read never scans pages) plus RECOVERY_FILE_HEADROOM_BYTES per copy, never below 256 MiB, because a restore reserves its file twice; it ignores the writer gate, and turning the switch off or lowering pages.max_bytes never strands a stored page. readPageBodyPolicy treats a failed read of the switch or pages.max_bytes as off, never as unset. Doctor segmented_storage reports a hold left only by this host's pre-upgrade writes as information with its clear time when, for each, this build has since made the same kind of write (role and principal, segmentedGateEvidence first_seen) after it; otherwise (a cron or npx gbrain on the host) it warns. PROJECTION_LOCK_HOLD_BYTES (512 KiB of segments, an occurrence larger than that alone) and PROJECTION_RETIRE_MAX_ROWS (2,500 rows) bound what one projection transaction inserts or deletes while it holds the page's key lock (page-body/ingest.ts stageOccurrenceChunks, page-body/projection.ts retireStaleChunks), about a tenth of the 1 s lock_timeout a write to the page, or a source-exclusive write anywhere on the source (the lock's page_write_guards insert key-share locks the source row), waits under in its publication, so a write never waits out its bound behind a background build (test/large-pages/projection-lock-hold.test.ts). Tests: test/large-pages/segmented-switch.test.ts, test/large-pages/recovery-budget-default.test.ts, test/large-pages/recovery-budget-ceiling.slow.test.ts.

  • src/core/ops/page-upload.ts — the five upload-session operations (begin_page_upload, upload_page_part, get_page_upload, commit_page_upload, abort_page_upload; write scope, client slug-fence allowlist). Begin routes the source and slug as put_page does, runs put_page's grant and visibility authorization before reading the page (a private page answers as a missing one), and reserves the declared bytes; every later call re-checks the binding (creating principal, write source inside the read scope, source incarnation, the page id or, for create, a still-absent slug) and answers any mismatch with an upload_not_found identical to an unknown id; abort needs only the creating principal, its source and the slug fences, so a session whose page or incarnation moved can still be freed. A refusal at commit (stale revision, sha mismatch) and a commit whose write ended without committing discard the session; only a write whose body_ref names the session is recorded as its commit, so a begin request_id another write already used attaches nothing. Commit claims the session with an atomic open-to-sealed sealUpload before hashing; a call that loses the claim, or finds a lapsed session another call already moved on (settleLapse), reads the session again and follows its real state (sealed or committing replays the commit, committed returns the first receipt), so upload_expired is answered only when this call expired it (uploadCommitProbe is the forced-race seam, test/large-pages/upload-review.test.ts). After the hash check commit submits an ordinary put_page carrying body_ref (stagedBody), so a second commit replays the first receipt; a replace re-checks expected_revision and the shrink guard. Append commits through admitPageAppend (persistence/page-append.ts) with the staged body_ref. Begin admits create and replace up to the page ceiling (pages.max_bytes, the engine ceiling by default) and an append up to PAGE_MAX_BYTES; over PAGE_MAX_BYTES a remote writer is refused file_too_large at begin where the write gate applies to an agent-written page.

  • src/core/persistence/page-upload.ts — upload-session storage: page_body_stagings rows of kind upload and their raw page_body_staging_parts, part_max_frame_bytes from the transport's ctx.inboundFrameBytes (at most 4 MiB, minus a 4 KiB envelope), part UTF-8 and code-point checks, the staged-quota reservation (limits.ts stagedCounterKeys, recounted under the counter locks at begin) and its release on commit, abort or expiry (transitionUploads with RELEASE_STAGED_CTE, which deletes the parts in the same statement; the segment sweep expires lapsed sessions the same way). An open session expires past its idle or hard deadline, a sealed one only past its hard deadline, which sealing extends by one idle period; a lapsed session is never sealed, so the brain-wide sum at begin leaves out other principals' lapsed sessions; expireIfLapsed reports whether its update changed the row. A session takes at most uploadPartLimit parts (64 plus one per 64 KiB declared) unless the part completes it (upload_too_many_parts). withStagedUpload binds a put_page or append_page preparation to the sealed session and marks it committed in the publication transaction; prepareStagedUpload (from preparePageMutation) reads a body within PAGE_MAX_BYTES back as put_page content and hands a larger one to upload-streamed.ts. stagedUploadWork counts whole-body reads for the memory tests.

  • src/core/persistence/upload-streamed.ts — a committed upload over PAGE_MAX_BYTES: the sealed parts are spooled one part at a time into a private temporary file that the segmented streamed import (page-body/streamed-import.ts) reads, verifying the admitted sha256; the spool is removed before publication. A remote replace of a page holding a takes fence, a facts fence with a non-world row (or one that does not parse; segmentedFactsFenceView reads only the fence runs of a segmented page), or timeline rows (without drop_timeline) refuses upload_protected_content, since put_page's whole-body takes, hidden-facts and timeline checks cannot run streamed. A remote upload keeps the stored page's provenance stamps (source_kind, ingested_via, ingested_at) and drops the caller's (keepStoredProvenance), and its tier is the lower of the caller's frontmatter and the frontmatter it publishes (provenanceWriteTrust, #7873). A remote writer gets put_page's gate and page tier (content.writer), so where the write gate applies the commit refuses file_too_large; a trusted local writer takes the source owner's tier, as a streamed local put --file does. A worktree-bound page publishes database-only (db_only).

  • src/core/page-body/read-range.ts + segment-projection.ts + segment-ledger.ts — Wave B PR 2 (spec 7.1, 7.3): the segment reader. openSegmentedBody reads a page's live manifest rows and its withdrawal ledger (readBodyLedger), the rows (and, given expect, the caller's slug and source, the ledger) alongside the page row: rows of another generation than the page row names, or a ledger read under another name, are read again; loadSegments reads a batch's metadata alongside its first text; readBodyRange(engine, { pageId, space, start, end, audience, region?, body? }) streams pieces one segment text at a time, never the whole body. Remote reads and every projected-space read serve the read projection: projectSegment projects whole segments from their stored protected regions (takes, truncated tail and markers out, keys as the projection token, facts regions rendered world-only and credential-projected), after the withdrawal overlay (overlaySegmentText; a fence split by a forced cut is omitted for reads; a trusted read (trustedSegments) joins the segments it spans, and a segment ending inside a facts marker with the next, and overlays them whole, refusing page_body_segmented above FENCE_MAX_BYTES (limits.ts: counted from the begin marker through the end marker, the bound withdrawal discovery's readStoredFence, the purge rewrite and the file's overlaid digest share), or page_projection_stale when the read starts inside the block) and the fence overlay (held and purged rows out, rows below the page tier in the low-tier block after the truth). A transform or stale metadata recomputes the regions from the segment's entry state; with none usable the read refuses page_projection_stale, never raw text. Trusted stored-space reads give the overlaid stored stream. projectedLayout (cached on the handle), projectedUnits (rows and fixed text in projected order, for page-window/segmented.ts), projectedTotals, projectedLineStart and projectedOutline (outline from stored metadata, no segment text) locate projected bytes; the concatenation equals projectPageText of the joined body byte for byte except where a facts fence is longer than a segment (shows less). The chunker (segment-chunks.ts) chunks the same projection, so sealed segmented pages carry MARKDOWN_CHUNKER_VERSION; sealSegmentedPage indexes the overlaid, sanitized timeline lead. Pinned by test/large-pages/projection-parity.test.ts, remote-window-privacy.test.ts and sealed-segmented-search.test.ts (fixture test/helpers/segmented-privacy-fixture.ts).

  • src/core/page-body/segment-rewrite.ts — fence rewrites of a segmented body without holding it as one string (forget strikes, purge drops). Only stored facts/takes protected regions are decoded and passed, in body order with their region, to a FenceTransform (a region crossing forced cuts is joined with its neighbours, refused above PAGE_MAX_BYTES). rewriteLiveSegmentedFences re-stages the live body through prepareSegmentedBody from the segment stream with the changed units replaced and flips it in the caller's transaction (trust tier and soft delete kept; null when nothing changed); rewriteSegmentedGeneration rebuilds one retained generation under its own number and updates its versions' hash and size; rewrittenContentHash streams the hash of a rewritten generation; segmentedBodyContains searches a body segment by segment with the previous segment's tail carried over. Tests: test/large-pages/purge-segments.test.ts, forget-scope.test.ts, page-purge-segments.test.ts.

  • src/core/page-body/segment-blocks.ts + segment-links.ts + segment-takes.ts — Wave B PR 2 (spec 8, full-pass mode): the derived passes over a segmented body, O(page) work and O(segment) memory, not yet incremental. bodyBlocks(engine, page, { regions?, separator?, want? }) yields the trusted, withdrawal-overlaid joined body one block at a time (a segment, or a run joined across forced cuts, refused above BLOCK_MAX_BYTES (read-range.ts) with page_body_segmented; want skips a run unread), so each block starts where the parsers start fresh; truthLead reads a bounded lead; readGarbageCollector forces a collection every 4 MiB read; detachedValue copies a result kept past its block so it pins no segment text; segmentedPassWork counts the passes' work for the linearity test. extractSegmentedPageLinks runs extractBodyLinks over each block in a window (bodyWindows: the whole blocks before it, 4096 chars after, the headings still open), with the role phrases a first pass found (segmentedRoleText), and merges candidates in extractPageLinks order, plus the temporal inputs (TemporalBody). parseTakesFenceBlocks gives parseTakesFence of the joined body. Consumers: link preparation and the links effect, stale-links maintenance, source reconciliation, the sweep, extract links|timeline|takes --source db, extract --stale (the stale list includes segmented pages since PR 2 fix lane J1; test test/large-pages/segmented-extraction-stale.test.ts), the mention pass (pageMentions), the canonical prior view (segmentedPriorView), segmentedTimelineText, the chronicle judge and the facts planner (planSegmentedExtraction). Tests: test/large-pages/segmented-derived-parity.test.ts, segmented-derived-passes.test.ts, segmented-derived-linear.test.ts, segmented-derived-memory.slow.test.ts, segmented-derived-work.test.ts. segment-lines.ts serves the line-local maintenance scans: bodyLines yields the joined text's complete lines a block at a time (a line cut by a block boundary is carried, counted in segmentedPassWork.carry), consumed by gbrain integrity and reconcile-links; segmentedRegionContains searches one region for literal markers across block boundaries (repair connector-fences). Test: test/large-pages/segmented-maintenance-commands.test.ts.

  • src/core/persistence/large-file.ts — Wave B PR 2 fix lane D: canonical files over 24 MiB (LARGE_FILE_MIN_BYTES) on the publication path, never one string or Buffer. hashFile/hashDescriptor (the persistence file hash, read in one-segment pieces), FileBytes (a file read on demand through one descriptor, the ByteSource the drift check's fileBodyDigests takes; close refuses source_changed when the file changed while read), StreamedFileContent (restore_page's file: export's serializer over the segments, measured once in preparation, produced again by stageStreamedContent into the reserved publication stage after the recovery record is durable and before the publication transaction: streamed into an in-progress sibling (streamingStagePath, <stage>.streaming), verified, fsynced and only then renamed to the stage's name, so a stream a crash cut short is never at that name and recovery removes it unread; a mismatch removes the stream and refuses revision_conflict; seam large_file:stage_streaming after each piece), and stageBeforeCopy (the prior file copied by the kernel into its before stage, hashed and fsynced; the recovery record names it by path and SHA-256 instead of embedding base64, and reserves the file once). largeFileWork (@internal) counts bytes read, written and copied and the largest piece. Coordinator: pageRecoveryRecord → largeRecoveryRecord, stageLargeFiles, publishStagedContent; a group publication never takes such a member (groupable). Flip lane A: the withdrawal mirror of a segmented page (effects.ts mirrorPage, any size) renders its file with page-prepare.ts unreadPageFile (a StreamedFileContent) and writes it with stageStreamedContent before its recovery transaction (effect-recovery.ts stageStreamedMirror); existsAt is the lstat-based leaf check it uses. prepareFileTarget reads the prior file through FileBytes at any size for a page stored in segments or a streamed target (never readFileSync whole), so its drift check reads in pieces of at most a segment. Test: test/large-pages/segmented-withdrawal-mirror.test.ts. Tests: test/large-pages/segmented-delete-restore-file.slow.test.ts (27 and 90 MiB round trips byte for byte, both engines), test/large-pages/segmented-delete-crash.slow.test.ts (SIGKILL between file removal and commit, and after the restoration rename).

  • src/core/page-body/body-digests.ts — Wave B PR 2: the digests the drift check of a segmented page over the single-string ceiling compares (page-prepare.ts segmentedFileMatches). fileBodyDigests(bytes, reading, filePath, opts) runs the streamed importer's pass 1 (BodyScanner) over one reading of the file's bytes (an inferred-frontmatter prefix, then the bytes past a BOM), then SHA-256 over each region's byte ranges, and reports a facts fence marker; storedBodyDigests(engine, pageId, ledger) hashes each region of the trusted stored body (readBodyRange, one segment per read, readGarbageCollector). bodyDigestWork counts file and stored bytes and the largest piece handed to a scan or hash (at most a segment). overlaidFileBodyDigests overlays the file one complete fence block at a time; inside an open block a read stops at FENCE_MAX_BYTES plus FENCE_READ_LOOKAHEAD (4 bytes), so an over-bound fence refuses page_body_segmented without reading a further segment (test/pathological/lp-g-file-digest-fence-bound.test.ts). In page-prepare.ts, prepareFileTarget compares a page whose body was not read back through segmentedFileMatches (the raw reading, then the inferred one from streamed-import.ts inferredHeadPrefix; the same normalizations as fileMatchesSnapshot via importedFileType); a file body with a facts fence under a withdrawal ledger that applies refuses page_body_segmented instead of reading as drift. restore_page of such a page (prepareUnreadRestore) versions it, clears deleted_at and reseals (or leaves the projection queued when chunks are missing) without re-importing, and publishes serializePageToMarkdown's bytes streamed from the segments by segmentedExportMarkdown into one buffer (under the 25 MiB pre-push scan cap); a core page and an external_untrusted writer refuse page_body_segmented, and an agent-tier writer's advisory write-gate scan is not re-run over the unchanged body. Tests: test/large-pages/segmented-file-drift.test.ts.

  • src/core/page-body/body-tiling.ts (untiledSegmentedPages) + schema.ts (liveTilingSql, untiledSql) + src/core/engine-sql/page-body.ts — GST-1209: a live generation tiles its body when its manifest rows, as every reader selects them, cover body_bytes from byte 0 with no gap. flipBodyGeneration checks the generation it made live inside its transaction and refuses one that does not tile (body_staging_expired, detail reason generation_incomplete, nothing published). setBodyStagingState moves only a live staging (open, sealed, prepared): a move on to prepared or committed of a staging a fact purge aborted or a sweep expired (its staged rows deleted) refuses body_staging_expired instead of reviving it. untiledSegmentedPages lists live segmented pages of unarchived sources that do not tile (read-only, one aggregate per page), for doctor segmented_tiling (src/commands/doctor/checks/segmented-tiling.ts, BRAIN, host only, its own registry entry segmentedTilingEntry in checks/graph-health.ts so the wave checks' budget never hides it; the fix always names --source) and the repair (survey lists at most PREVIEW_PAGE_LIMIT pages and reports truncated). A sync hold naming page_body_rows_unreadable reads needs_human / repair and its gitHoldFix previews gbrain repair segmented-tiling --source <id> --slug <slug>.

  • src/core/repair/segmented-tiling.ts — gbrain repair segmented-tiling --source <id> [--slug <slug>] (GST-1209), explicit-only, preview-bound, trusted local CLI on the owner host, checks: ['segmented_tiling']. The preview lists each untiled page with its file, HEAD's blob (oid, SHA-256, size) and sync hold under a hash; a page with no file, a file HEAD lacks or one over pages.max_bytes, a file another sync hold withholds (any but the unreadable-body page_write_failed one) or one unreadPageImportScreen (sync-screen.ts, the next sync's screen against the unread snapshot) refuses is listed held; a source with no Git checkout refuses not_a_git_repo. --apply --expect re-checks revision, generation and blob (preview_changed), then per page in one transaction under the key lock (coordinated maintenance on a managed brain) versions the damaged generation, stages the screened body after preserveWithdrawnFenceRows (purged claims stay out) through prepareSegmentedBody without reading the damaged body, flips with applySegmentedBody (title, type, frontmatter, tags and trust tier kept), clears the previewed hold, records the new revision as imported from the HEAD blob (recordSyncImportProvenance) when the file imports to the page's stored title, type, frontmatter and tags (so the next sync holds no file_drift), calls requestGitHoldRetry and writes an op_checkpoints receipt (segmented-tiling-receipt) and a segmented-tiling audit line. Tests: test/large-pages/segmented-tiling-repair.test.ts.

  • src/core/page-body/segment-facts.ts + segment-conversation.ts — Wave B PR 2 lane 2e2 (spec 8, full-pass mode): the facts fence reconcile and conversation facts over a segmented body. segmentedFactsFenceView is factsFenceView (facts-fence.ts: each section's parsed fence, which sections hold one, and the #3625 timeline marker) read from the segments: one SQL query selects the live manifest rows that carry facts slots, a forced cut, or a begin/end marker (bytea position, nothing transferred), and bodyBlocks' want option reads only the runs holding them; the segmenter never cuts inside a fence, so a fence lies in one block (a longer one is a forced run, refused above BLOCK_MAX_BYTES with page_body_segmented), and joining one across blocks is a bounded fallback. segmentedBodyMatchesFile is canonicalCacheState's stale-cache check for a segmented page: the canonical file streamed through BodyScanner against the stored regions, by length and SHA-256. readSegmentedConversationInput gives readConversationInput (conversation-parser/body.ts) the summary body through ranged reads: whole within CONVERSATION_WINDOW_BYTES (PAGE_MAX_BYTES), else the newest whole segments that fit (never starting inside a forced run) with the window's fallbackDate from its first segment's entry state; extract-conversation-facts parses that window, passes the date to the parse and the extractor, and writes a tail_window receipt (window_rule: no_prior_facts), but only on a page with no conversation facts: one that has any keeps them and is skipped (skipped_too_large, window_rule: prior_facts_kept), since the run clears every prior fact. Each loop counts its reads in segmentedPassWork. Consumers: runExtractFacts (planFenceReconcile, the stale-cache guard, the managed fence-facts request), the sweep's fence candidates (segmented pages with facts slots), extract-conversation-facts. Tests: test/large-pages/segmented-facts-reconcile.test.ts, segmented-conversation-facts.test.ts, segmented-derived-linear.test.ts.

Continued in Core Services (2/3, page windows and large writes).