Garry's Opinionated OpenClaw/Hermes Agent Brain
Key files: Core Services (2/3, continued)
Subsystem index · Previous entries. Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
The page-state concurrency foundation under src/core/page-state/ and the shared progressive-batch primitive. Dated edition reads (get_page as_of) are in Valid time.
src/core/ops/get-page-slice.ts— get_pagesection/grep/context(#7180), paramsGET_PAGE_SLICE_PARAMS(full surface only; the get_page description names them on starter, and dispatch accepts them there).ops/get-page-body.tsroutes a slice here:parseSliceRequestmakes it a window-mode request carryingslice, andreadPartialBody, after authorization and the quarantinebody_omittedcheck, passes its read document'sstream()(the inline bytes, orSegmentedDocument.stream()one segment at a time) with the window's metadata and size. Both scan the read projection line by line, so a segmented page is never held whole (memory: the returned text, one segment, at mostcontextparagraphs; a line over 1 MiB is scanned in pieces).readSection: the first ATX heading (HEADING, outsideFENCEblocks, as the outline sees them) whose whitespace-folded text equals the request case-insensitively (a leading##is ignored), down to the next heading of the same or higher level or the timeline separator; returnssection {heading, level, heading_path, line_start, line_end, text}and stops reading at the section's end; a miss isinvalid_paramsnaming up to 20 headings withdetail.total_headingsand an outline fix.readGrep: blank-line paragraphs or unfenced table rows with a line containing the literal term (case-insensitive, no regex),contextparagraphs either side (default 1, clamped to 5), windows that overlap or touch merged;grep {term, context, total_matches, returned_matches, omitted_matches, blocks[{line_start, line_end, match_starts, text}]}. Caps: 50 matching paragraphs andSLICE_MAX_CHARS(100,000), fewer when the egress cap is smaller (sliceCharBudget);truncatedsays a cap was hit, and a cut section addsomitted_charsand anextwindow read. Line numbers are the read view's, the onesbody: 'window'linesreads back. A slice with a selector, include_content, content_only,view: markdown, a body other than auto/window, or known_as_of isinvalid_params. Every get_page shape (whole, content_only, partial, slice) carrieschars, the characters of page text it holds (projectGetPage,readPartialBody). Test:test/get-page-slices.test.ts(both engines).src/core/page-state/— Canonical page concurrency foundation shared by both engines.schema.ts(migration 150) adds source incarnation UUIDs, logical page revisions, text-projection revisions, and persistent guards for absent pages.guards.tslocks sorted sources then page identities in an existing transaction; a transaction or savepoint never re-acquires a key it holds, which covers absent pages because the only pages INSERT runs afterputPagetakes the same guard (test/e2e/page-guards-postgres.test.ts); callers lock sources before authority/request/fact rows.snapshot.tscomputes the normalized body/timeline fingerprints only for a page with fact withdrawals, page-subject purges or a source '*' purge marker (they feed only the withdrawal overlay), aspage-snapshot-batch.tsdoes.snapshot.tsreads page/tags/withdrawals in one MVCC statement and applies withdrawals using database-normalized text.transactions.tsmakes helper transaction APIs compose through savepoints.versions.tsrecords the complete canonical snapshot;rename-alias.tsrecordsslug_aliases(old -> new)inside both engines'updateSlugtransaction, repoints aliases that named the old slug and drops one spelled like the new slug, and keeps every move (and any alias it drops) inslug_rename_history(v285), which only the folder-rule lineage test reads;moveSlugBindingsmoves the page's facts (entity_slug,source_markdown_slug) andpage_aliasesin the same transaction, letting a purged page's leftovers at the new slug yield (test/rename-slug-alias.test.ts);movePageIdReferences(#5431) moves the page-id-keyed rows of a stale rename row onto the surviving row (linksfrom/to/origin,timeline_entriespage and event,page_versions), dropping stale-side duplicates on thelinkscomposite key and the timeline dedup indexes (test/rename-slug-alias.test.ts);materialize.tspersists only verified canonical overlay bytes without another logical revision.types.tsdefines revision preconditions: no revision is create-only (an existing page refusesrevision_required, #5385, its own code since the omission is the caller's and not a race), a stale revision or a page another writer created where an internal writer asserted none (expectedRevision: null) isrevision_conflict, explicit force overwrites. Plain legacyputPagecalls keep compatibility until their owning operation adopts the managed write protocol.projection-schema.ts(migration 153) invalidates unverified prior projections once and queues durable work, but a Markdown or code page at or after the safe-chunk fence with no facts or takes fence, private-key text or quarantine marker keeps its old index in search until rebuilt (servedTextProjectionFilter);projections.tsrebuilds Markdown without a worktree/provider and installs only matching canonical revisions. Embedding installs also compare exact chunk identities/text. Source importers seal code/image projections; all read arms hide unsealed projections.src/core/page-state/projections.ts—sealPageTextProjection(defined in the lightpage-state/text-seal.ts, re-exported here, soengine-sql/chunks.tssealChunkscan run it without loading the projection pipeline) andsealImportedPagewrite the page keyword vector throughpageSearchVectorSql(fts-language.ts: title weight A, sanitized timeline weight C, both throughgbrain_fts_input), the same expressionsearch-vector-backfill.tsuses. Captures canonical revision, source incarnation, page identity, the complete raw chunk inventory and resolved indexing inputs before preparing derived work. Embedding reads require a current text seal; healing/rebuild capture can explicitly include unsealed pages. Projection installation compares its originating seal, chunk identities/text/source/completion state, chunker version and indexing context (including stored corpus generation) under the page guard before replacing any row. A same-revision re-chunk may replace the snapshot it actually captured; a delayed older installation cannot discard newer chunks or vectors. Rebuild also rechecks its queued job before capture so a completed job from an old listing stays completed. Chunkless healing sanitizes complete body/timeline fences before splitting, preserving withdrawn history in direct page reads while excluding it from retrieval. Tests:page-projection-origin.test.ts,page-projection-vector-preservation.test.ts(PGLite and isolated Postgres),embed-stale-chunkless-pages.serial.test.ts(public healing entry). Sharedmarkdown-chunks.tsandcode-chunks.tspreserve fenced code and source-language metadata. A preserving rebuild keeps a vector only when itscontent_chunks.embedding_input_hash(src/core/embedding-input-hash.ts) equals one recomputed from the current page, or, with no recorded hash, when the page is non-contextual and the legacy model/text-hash checks pass; a contextual chunk without provenance is nulled once.installPageEmbeddingsrecords the hash in the vector's UPDATE (the plain re-embed tier by default, or the caller'sbuilttier and corpus generation), and returns{ installed, complete }: it matches each offered vector to the page's current chunk by source and exact text (not index) and writes it only when its input hash for the captured context is one the current context accepts, in batchedjsonb_to_recordsetUPDATEs, so a page that changed under the provider call keeps the vectors of its unchanged chunks; the embedding signature is stamped only whencomplete(same chunk sequence, nothing dropped). A preserving rebuild likewise keys kept vectors by chunk content, so a top insert shifts vectors with their chunks (tests:page-embedding-content-validity.test.ts). AlsostampEmbeddingInputsdoes the same for full imports.resealSafeChunksre-seals a page below the safe-chunk fence from its unchanged body with no canonical write;projectionBelowSafeFenceis its predicate. Tests addprojection-embedding-input-hash.test.tsandsafe-chunk-reseal.test.ts. It replaces outgoing code edges without deleting incoming edges to retained chunks and clears backfill watermarks. Failed jobs rotate with a retained diagnostic; successful drain refreshes planner statistics.rebuildPendingPageProjections(engine, limit, { deadlineMs, now, notAfter, retryCooldown, onFailure })(#5401) checks an injectable clock between pages, bounds a run to rows queued at or before its start (notAfter, cast from text in SQL so microseconds survive the driver), applies the 30 srebuild_failedcooldown for the resident, and treats a lock or snapshot error as that page's failure instead of aborting the batch;projectionBacklogreturns counts only (pending, failed, oldest age) for the drain command and the resident'sprojection_statusIPC reply. A same-revision conflict throwsPageProjectionConflictErrornaming the fields that changed (text_projection_revision,chunk_digest,indexing_context), the slug and source;retryProjectionConflict(attempt)re-runs an attempt that captures a fresh snapshot at most three more times with jitter, then rethrows (page_projection_conflicton the wire). Tests:page-projection-origin.test.ts(resident rebuild race, two-writer progress, exhausted-retry CLI/JSON output).src/core/page-state/chunk-moves.ts— #7172:CHUNK_VECTOR_WRITE_BUDGET(5), the most newcontent_chunksrow versions carrying a vector (inserts, and non-HOT updates ofchunk_index, text or vector; each adds an HNSW entry) one transaction of a managed chunk replacement makes, with its derivation from the publication's 5 s statement bound (a version without a vector is not counted);planMoveWavesplans the in-place renumbering of kept rows within a budget (one UPDATE per wave, a cycle parks one row at-1 - target),applyMoveWavesruns it.import-file.ts(withboundedChunkWrites, set by page-prepare, import-prepare and sync-prepare) commits a page whose replacement exceeds the budget with its projection pending (queue reasonchunk_replacement_deferred), andprojections.tscontinueDeferredProjectionfinishes it after commit in steps of at most the budget;rebuildPendingPageProjectionsresumes an interrupted one.src/core/page-state/projection-class.ts— #6803:projectionClassSql(p, s), the one CASE that puts each queued page in exactly one class (archived,textfor Markdown/code and segmented pages,protectedfor a protected fence,sealablepersealableImageSql, elsereimport), shared byprojectionBacklog, the drain probe, search readiness andembed --stale --images.sealableImageSql: an image page at or after the safe-chunk fence whose one chunk (index 0,image_asset) carries the page's own text.src/core/page-state/image-seal.ts— #6803:sealImageProjectionsrestores the importer's seal onsealableimages in groups of 500: a primary-key window of queue rows probed by each page's unique key (a statistics-less PGLite planner otherwise probes throughpages_dedup_idx), then under the page guard a revision compare-and-set with the class rechecked; no chunk write, the visual vector untouched, a text vector whose recorded input no longer matches nulled forembed --stale.IMAGE_SEAL_PAGES_PER_SECONDis the measured rate (its comment names the run);projectionDrainLimit(textPending)sizes doctor's--limitto halfAGENT_TOOL_LIMIT_MSat that rate, or at the measuredTEXT_REBUILD_PAGES_PER_SECOND(rebuild-estimate.ts; it also times v153's notice) while queued text waits (the drain takes text first).src/core/page-state/projection-window.ts— #6803: the resident's bounded queue read: per pass at mostRESIDENT_PROBE_BUDGET(500) rows, the 100 newest byupdated_atplus a primary-key window from a per-engine cursor that starts at a random key and wraps; a negative cache (100k keys) skips image rows the resident never builds. The consumer's idle probe walks the same window and advances the same cursor. Tests:projection-resident-window.test.ts.src/core/page-state/projection-status.ts— #6803: the resident's cachedprojectionBacklogfor theprojection_statusIPC op, refreshed when a cheap signal (queue count, maxupdated_at, source archive flags) moves, checked every 60 s and at most 10 minutes old.src/core/page-state/projection-upgrade-notice.ts— #6803:gbrain post-upgrade's read-only[AGENT]block whilesealableorreimportimage pages wait, naminggbrain projections drain --dry-runwith an estimate at the measured seal rate.src/core/page-state/credential-reseal.ts+src/core/schema-migrations/v189-pages-credential-projection-pending.ts+src/commands/doctor/checks/credential-projection.ts— the targeted re-chunk of pages indexed before the credential-safe projection. Schema migration v187 (on the first connect after upgrade) setstext_projection_revision = NULLon every live markdown/code page whosecompiled_truthortimelinematches-----(BEGIN|END) [A-Z ]*PRIVATE KEY-----and upserts apage_projection_jobsrow (reasoncredential_projection), so the current-text-projection seal withholds their old chunks from every retrieval path, local and remote (code-walkhydrateChunksinsearch/two-pass.tsapplies the same filter). The marker literal is frozen as shipped.credentialProjectionPending(engine, sourceIds?)counts pending pages in active sources asrebuildableandkept(code pages with nofrontmatter.fileorsource_path, which wait for their importer).resealCredentialProjectionsre-seals each rebuildable page from its unchanged body throughresealSafeChunks(therepair safe-chunksinstaller), one page per transaction, with no page write, journal admission or provider call; vectors of unchanged chunk text are kept and re-chunked key chunks wait forgbrain embed --stale; a revision conflict counts as superseded; a page that fails item-scoped (ItemRun sitemigration.credential_reseal) is held in op_checkpointscredential-reseal-holdkeyed by page id (revision, chunker generation, code, redacted message, consecutive attempts) and stays withheld, a run-scoped failure or most pages failing is thrown and records nothing.readCredentialResealHoldsreturns the holds that still apply (page live, same revision and chunker generation, still unsealed) with each one's fix (gbrain repair safe-chunks --source <id> --apply, orgbrain projections drainfor a segmented page); a hold never blocks a retry.CREDENTIAL_RESEAL_HELDletsrepair safe-chunksselect held pages, and doctortext_projection_readinesswarns withcredential_reseal_held,held_pages(local callers only) and the safe-chunks fix. The doctor checkcredential_projection_pending(aWAVE_CHECKSentry,resolution: 'operator', brain category) reportspages_pending,kept_pagesandheld_pages(with the retry command) and warns whilepages_pendingorkept_pagesis non-zero, naminggbrain apply-migrations --yes --no-autopilot-install(no provider calls), thengbrain embed --stale, and the re-import for kept code pages; a read error iswarnwith unknown health. A segmented page is pending when a live manifest row stores apemregion; it is re-sealed throughbuildSegmentedProjection.Embedding-only publication also requires a live, unarchived source and nondeleted page under the same source/page row locks. A prepared vector cannot bypass a later archive, model change or withdrawal.
test/e2e/migration-wave-prepared-withdrawal.test.tsexercises both publication orders and exact unchanged-vector refusal on both engines.src/core/page-state/version-types.ts— complete canonical version contract with nullable legacy fields, re-exported bytypes.tsfor existing callers.page-state/versions.tsrecords deletion, timeline, tags and originating revision in the page publication transaction. Wave B: a segmented page's version references its body generation (body_storage: 'segmented',body_generation,body_bytes,content_hash, NULL bodies; O(1));pruneSegmentedVersionskeeps the newestversions.segmented_max_per_pageand prunes oldest-first while bytes pinned beyond the live generation exceed max(versions.segmented_max_pinned_bytes, body_bytes), thensweepSegments;get_versionsreturns their size and, withinclude_body, abody_ref;revert_versionrefuses them withpage_body_segmenteduntil PR 3. A holding page (isHoldingSlug) keeps its newestLARGE_PAGE_VERSIONS_KEPTinline versions whatever their size, at mostHOLDING_PAGE_VERSIONS_PRUNED_PER_WRITE(100) removed per save (pruneHoldingPageVersions, run only by the fence append after a save that adds a fact; a held move and purge, which edit saved versions themselves, never prune) (GST-2497).LARGE_PAGE_VERSIONS_KEPTlives inpage-body/limits.ts.src/core/progressive-batch/— shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules:types.ts(Stage, StageVerdict, AbortReason, discriminatedVerifierunionOutputCountVerifier | IdempotentMutationVerifier | NoopVerifier, Policy, StageReport),orchestrator.ts(runProgressiveBatch(items, verifier, policy, runner)— readsgetCurrentBudgetTracker()ahead ofPolicy.maxCostUsdfail-closed; null both ways triggersabort_cost_cap reason='no_budget_safety_net'),audit.ts(ISO-week JSONL at~/.gbrain/audit/progressive-batch-YYYY-Www.jsonlvia the sharedaudit-writerprimitive),stage-report.ts(ASCII formatter for the defaultPolicy.onStageReport). Env knobs:GBRAIN_PROGRESSIVE_BATCH_DISABLED=1,GBRAIN_PROGRESSIVE_BATCH_AUTO=1(skip Ctrl-C grace),GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site viaPolicy.interactiveAbortMs > 0. Pinned bytest/progressive-batch/orchestrator.test.ts(35 cases, every verdict path).src/core/page-body/page-shape.ts— Wave B PR 1 (spec 1.5): segmented page bodies are unreachable as strings.guardSegmentedBody(called byrowToPagefor a row withbody_storage = 'segmented', so by every snapshot andgetPageread) replacescompiled_truth/timelinewith enumerable getters and setters that throwpage_body_segmented, so property reads, spread,JSON.stringify,structuredCloneand assignment all fail loud instead of seeing an empty body.isSegmentedPage,assertInlineBody(page, site)(explicit refusal where a writer would otherwise act before a getter fires, e.g. SQL-only writers; a read'spage_body_segmentedrefusal fixes withget_page body: 'auto'; for a write that fix changes nothing, soasSegmentedWriteRefusal(called by MCP dispatch on a mutating op) swaps it for a report-only step namingedit_page,append_page,add_tagandremove_tag, andreadsPageInPieces(fix)tells the two apart, which remember's unattributed-save fix inpersistence/verb-errors.tsalso uses),pageBodyBytes(page)(body_bytesfor segmented pages),pageBodySegmentedErrorandpageBodyOverCeilingError(the refusal of a fence or timeline write through the put path on a page over 24 MiB;wholeBodySnapshotinpersistence/segmented-apply.tsreads a page within it back as strings and refuses above it: semantic timeline writes,takes-prepare.ts,facts-prepare.ts,reconcile-state.ts,gbrain core). Tag writes on a segmented page never read the body:prepareSegmentedTagMutation(persistence/page-prepare.ts) flips them in metadata over the live manifest (test/large-pages/segmented-semantic-writes.test.ts).bodyUnread(page)says a segmented page's body was not read back as strings (over the single-string ceiling, seematerializedPage); such a body was written by the streamed import, which holds fenced files, sofenceBody(page)reads it as empty for the fence merges andwithoutBody(page)copies the page's other fields for a render whose body comes from the incoming write (a shrunk file rewrites such a page:test/large-pages/segmented-shrink-below-ceiling.test.ts).Pagestays one shape (body_storage?,body_bytes?); the static half isscripts/check-page-body-readers.tswithscripts/page-body-readers-baseline.tsv(docs/ci-guards.md#page-body-readers). Pinned bytest/large-pages/segmented-body-unreachable.test.ts.src/core/page-body/limits.ts— Wave B size classes, ceilings, retention numbers and config keys in one place.readPageBodyPolicyis the single question every write asks:segmentedis the effectivepages.segmented_storage(unset meansSEGMENTED_STORAGE_DEFAULT,on;offor an unrecognized value is off; held off whilesegmentedWriterBlockersfinds a recent writer that blocks, and an unreadable gate counts as blocked),maxBytesthe engine ceiling (256 MiB Postgres, 128 MiB PGLite, lowered bypages.max_bytes) while on andPAGE_MAX_BYTESwhile off. Config is read per call; only the writer gate is cached (SEGMENTED_WRITER_GATE_TTL_MS,_resetSegmentedWriterGateForTests).chooseBodyStorageapplies the 1 MiB / 512 KiB hysteresis. The writer gate (segmentedGateWritersinpersistence/writer-versions.ts) reads seven days of admitted requests in any state, committed publications,ingest_logrows and, on Postgres,persistence_consumersheartbeats; a row blocks when its version is belowSEGMENTED_WRITER_FLOOR(page-body/writer-floor.ts: thesinceof the BEHAVIOR_CHANGES row with idsegmented_switch_on, this build's VERSION while it isTBD, so the merge-slot stamp edits only that row) or unparsable, or when it lacks the segment-reader mark (migration v254:admitter_segment_reader,consumer_segment_reader,persistence_consumers.segment_reader,ingest_log.writer_segment_reader), because releases 0.60.252.0-0.60.262.0 have no segment readers and a version string cannot tell them apart. Not seen: an older binary's unmanaged single-page write, a run that dies before it logs, and read-only or export use.pageRecoveryBytesis the defaultpersistence.limits.worktree_recovery_bytes(read bypersistence/limits.tsreadJournalLimitsin its one config statement): twice the larger of the body the configured switch admits (segmentedMaxBytes, orPAGE_MAX_BYTESwhile off) and the largest page stored in segments (deleted pages included; one probe ofpages_segmented_body_bytes_idx, v260, or the engine ceiling while that index is not valid, so a limit read never scanspages) plusRECOVERY_FILE_HEADROOM_BYTESper copy, never below 256 MiB, because a restore reserves its file twice; it ignores the writer gate, and turning the switch off or loweringpages.max_bytesnever strands a stored page.readPageBodyPolicytreats a failed read of the switch orpages.max_bytesas off, never as unset. Doctorsegmented_storagereports a hold left only by this host's pre-upgrade writes as information with its clear time when, for each, this build has since made the same kind of write (role and principal,segmentedGateEvidencefirst_seen) after it; otherwise (a cron or npx gbrain on the host) it warns.PROJECTION_LOCK_HOLD_BYTES(512 KiB of segments, an occurrence larger than that alone) andPROJECTION_RETIRE_MAX_ROWS(2,500 rows) bound what one projection transaction inserts or deletes while it holds the page's key lock (page-body/ingest.tsstageOccurrenceChunks,page-body/projection.tsretireStaleChunks), about a tenth of the 1 slock_timeouta write to the page, or a source-exclusive write anywhere on the source (the lock'spage_write_guardsinsert key-share locks the source row), waits under in its publication, so a write never waits out its bound behind a background build (test/large-pages/projection-lock-hold.test.ts). Tests:test/large-pages/segmented-switch.test.ts,test/large-pages/recovery-budget-default.test.ts,test/large-pages/recovery-budget-ceiling.slow.test.ts.src/core/ops/page-upload.ts— the five upload-session operations (begin_page_upload,upload_page_part,get_page_upload,commit_page_upload,abort_page_upload; write scope, client slug-fence allowlist). Begin routes the source and slug as put_page does, runs put_page's grant and visibility authorization before reading the page (a private page answers as a missing one), and reserves the declared bytes; every later call re-checks the binding (creating principal, write source inside the read scope, source incarnation, the page id or, for create, a still-absent slug) and answers any mismatch with anupload_not_foundidentical to an unknown id; abort needs only the creating principal, its source and the slug fences, so a session whose page or incarnation moved can still be freed. A refusal at commit (stale revision, sha mismatch) and a commit whose write ended without committing discard the session; only a write whosebody_refnames the session is recorded as its commit, so a begin request_id another write already used attaches nothing. Commit claims the session with an atomic open-to-sealedsealUploadbefore hashing; a call that loses the claim, or finds a lapsed session another call already moved on (settleLapse), reads the session again and follows its real state (sealed or committing replays the commit, committed returns the first receipt), soupload_expiredis answered only when this call expired it (uploadCommitProbeis the forced-race seam,test/large-pages/upload-review.test.ts). After the hash check commit submits an ordinary put_page carryingbody_ref(stagedBody), so a second commit replays the first receipt; a replace re-checksexpected_revisionand the shrink guard. Append commits throughadmitPageAppend(persistence/page-append.ts) with the stagedbody_ref. Begin admits create and replace up to the page ceiling (pages.max_bytes, the engine ceiling by default) and an append up to PAGE_MAX_BYTES; over PAGE_MAX_BYTES a remote writer is refusedfile_too_largeat begin where the write gate applies to an agent-written page.src/core/persistence/page-upload.ts— upload-session storage:page_body_stagingsrows of kinduploadand their rawpage_body_staging_parts,part_max_frame_bytesfrom the transport'sctx.inboundFrameBytes(at most 4 MiB, minus a 4 KiB envelope), part UTF-8 and code-point checks, the staged-quota reservation (limits.tsstagedCounterKeys, recounted under the counter locks at begin) and its release on commit, abort or expiry (transitionUploadswithRELEASE_STAGED_CTE, which deletes the parts in the same statement; the segment sweep expires lapsed sessions the same way). An open session expires past its idle or hard deadline, a sealed one only past its hard deadline, which sealing extends by one idle period; a lapsed session is never sealed, so the brain-wide sum at begin leaves out other principals' lapsed sessions;expireIfLapsedreports whether its update changed the row. A session takes at mostuploadPartLimitparts (64 plus one per 64 KiB declared) unless the part completes it (upload_too_many_parts).withStagedUploadbinds a put_page or append_page preparation to the sealed session and marks it committed in the publication transaction;prepareStagedUpload(frompreparePageMutation) reads a body within PAGE_MAX_BYTES back as put_page content and hands a larger one to upload-streamed.ts.stagedUploadWorkcounts whole-body reads for the memory tests.src/core/persistence/upload-streamed.ts— a committed upload over PAGE_MAX_BYTES: the sealed parts are spooled one part at a time into a private temporary file that the segmented streamed import (page-body/streamed-import.ts) reads, verifying the admitted sha256; the spool is removed before publication. A remote replace of a page holding a takes fence, a facts fence with a non-world row (or one that does not parse;segmentedFactsFenceViewreads only the fence runs of a segmented page), or timeline rows (withoutdrop_timeline) refusesupload_protected_content, since put_page's whole-body takes, hidden-facts and timeline checks cannot run streamed. A remote upload keeps the stored page's provenance stamps (source_kind,ingested_via,ingested_at) and drops the caller's (keepStoredProvenance), and its tier is the lower of the caller's frontmatter and the frontmatter it publishes (provenanceWriteTrust, #7873). A remote writer gets put_page's gate and page tier (content.writer), so where the write gate applies the commit refusesfile_too_large; a trusted local writer takes the source owner's tier, as a streamed localput --filedoes. A worktree-bound page publishes database-only (db_only).src/core/page-body/read-range.ts+segment-projection.ts+segment-ledger.ts— Wave B PR 2 (spec 7.1, 7.3): the segment reader.openSegmentedBodyreads a page's live manifest rows and its withdrawal ledger (readBodyLedger), the rows (and, givenexpect, the caller's slug and source, the ledger) alongside the page row: rows of another generation than the page row names, or a ledger read under another name, are read again;loadSegmentsreads a batch's metadata alongside its first text;readBodyRange(engine, { pageId, space, start, end, audience, region?, body? })streams pieces one segment text at a time, never the whole body. Remote reads and every projected-space read serve the read projection:projectSegmentprojects whole segments from their stored protected regions (takes, truncated tail and markers out, keys as the projection token, facts regions rendered world-only and credential-projected), after the withdrawal overlay (overlaySegmentText; a fence split by a forced cut is omitted for reads; a trusted read (trustedSegments) joins the segments it spans, and a segment ending inside a facts marker with the next, and overlays them whole, refusingpage_body_segmentedaboveFENCE_MAX_BYTES(limits.ts: counted from the begin marker through the end marker, the bound withdrawal discovery'sreadStoredFence, the purge rewrite and the file's overlaid digest share), orpage_projection_stalewhen the read starts inside the block) and the fence overlay (held and purged rows out, rows below the page tier in the low-tier block after the truth). A transform or stale metadata recomputes the regions from the segment's entry state; with none usable the read refusespage_projection_stale, never raw text. Trusted stored-space reads give the overlaid stored stream.projectedLayout(cached on the handle),projectedUnits(rows and fixed text in projected order, forpage-window/segmented.ts),projectedTotals,projectedLineStartandprojectedOutline(outline from stored metadata, no segment text) locate projected bytes; the concatenation equalsprojectPageTextof the joined body byte for byte except where a facts fence is longer than a segment (shows less). The chunker (segment-chunks.ts) chunks the same projection, so sealed segmented pages carryMARKDOWN_CHUNKER_VERSION;sealSegmentedPageindexes the overlaid, sanitized timeline lead. Pinned bytest/large-pages/projection-parity.test.ts,remote-window-privacy.test.tsandsealed-segmented-search.test.ts(fixturetest/helpers/segmented-privacy-fixture.ts).src/core/page-body/segment-rewrite.ts— fence rewrites of a segmented body without holding it as one string (forget strikes, purge drops). Only storedfacts/takesprotected regions are decoded and passed, in body order with their region, to aFenceTransform(a region crossing forced cuts is joined with its neighbours, refused abovePAGE_MAX_BYTES).rewriteLiveSegmentedFencesre-stages the live body throughprepareSegmentedBodyfrom the segment stream with the changed units replaced and flips it in the caller's transaction (trust tier and soft delete kept; null when nothing changed);rewriteSegmentedGenerationrebuilds one retained generation under its own number and updates its versions' hash and size;rewrittenContentHashstreams the hash of a rewritten generation;segmentedBodyContainssearches a body segment by segment with the previous segment's tail carried over. Tests:test/large-pages/purge-segments.test.ts,forget-scope.test.ts,page-purge-segments.test.ts.src/core/page-body/segment-blocks.ts+segment-links.ts+segment-takes.ts— Wave B PR 2 (spec 8, full-pass mode): the derived passes over a segmented body, O(page) work and O(segment) memory, not yet incremental.bodyBlocks(engine, page, { regions?, separator?, want? })yields the trusted, withdrawal-overlaid joined body one block at a time (a segment, or a run joined across forced cuts, refused aboveBLOCK_MAX_BYTES(read-range.ts) withpage_body_segmented;wantskips a run unread), so each block starts where the parsers start fresh;truthLeadreads a bounded lead;readGarbageCollectorforces a collection every 4 MiB read;detachedValuecopies a result kept past its block so it pins no segment text;segmentedPassWorkcounts the passes' work for the linearity test.extractSegmentedPageLinksrunsextractBodyLinksover each block in a window (bodyWindows: the whole blocks before it, 4096 chars after, the headings still open), with the role phrases a first pass found (segmentedRoleText), and merges candidates inextractPageLinksorder, plus the temporal inputs (TemporalBody).parseTakesFenceBlocksgivesparseTakesFenceof the joined body. Consumers: link preparation and the links effect, stale-links maintenance, source reconciliation, the sweep,extract links|timeline|takes --source db,extract --stale(the stale list includes segmented pages since PR 2 fix lane J1; testtest/large-pages/segmented-extraction-stale.test.ts), the mention pass (pageMentions), the canonical prior view (segmentedPriorView),segmentedTimelineText, the chronicle judge and the facts planner (planSegmentedExtraction). Tests:test/large-pages/segmented-derived-parity.test.ts,segmented-derived-passes.test.ts,segmented-derived-linear.test.ts,segmented-derived-memory.slow.test.ts,segmented-derived-work.test.ts.segment-lines.tsserves the line-local maintenance scans:bodyLinesyields the joined text's complete lines a block at a time (a line cut by a block boundary is carried, counted insegmentedPassWork.carry), consumed bygbrain integrityandreconcile-links;segmentedRegionContainssearches one region for literal markers across block boundaries (repairconnector-fences). Test:test/large-pages/segmented-maintenance-commands.test.ts.src/core/persistence/large-file.ts— Wave B PR 2 fix lane D: canonical files over 24 MiB (LARGE_FILE_MIN_BYTES) on the publication path, never one string or Buffer.hashFile/hashDescriptor(the persistence file hash, read in one-segment pieces),FileBytes(a file read on demand through one descriptor, theByteSourcethe drift check'sfileBodyDigeststakes;closerefusessource_changedwhen the file changed while read),StreamedFileContent(restore_page's file: export's serializer over the segments, measured once in preparation, produced again bystageStreamedContentinto the reserved publication stage after the recovery record is durable and before the publication transaction: streamed into an in-progress sibling (streamingStagePath,<stage>.streaming), verified, fsynced and only then renamed to the stage's name, so a stream a crash cut short is never at that name and recovery removes it unread; a mismatch removes the stream and refusesrevision_conflict; seamlarge_file:stage_streamingafter each piece), andstageBeforeCopy(the prior file copied by the kernel into itsbeforestage, hashed and fsynced; the recovery record names it by path and SHA-256 instead of embedding base64, and reserves the file once).largeFileWork(@internal) counts bytes read, written and copied and the largest piece. Coordinator:pageRecoveryRecord→largeRecoveryRecord,stageLargeFiles,publishStagedContent; a group publication never takes such a member (groupable). Flip lane A: the withdrawal mirror of a segmented page (effects.tsmirrorPage, any size) renders its file withpage-prepare.tsunreadPageFile(aStreamedFileContent) and writes it withstageStreamedContentbefore its recovery transaction (effect-recovery.tsstageStreamedMirror);existsAtis the lstat-based leaf check it uses.prepareFileTargetreads the prior file throughFileBytesat any size for a page stored in segments or a streamed target (neverreadFileSyncwhole), so its drift check reads in pieces of at most a segment. Test:test/large-pages/segmented-withdrawal-mirror.test.ts. Tests:test/large-pages/segmented-delete-restore-file.slow.test.ts(27 and 90 MiB round trips byte for byte, both engines),test/large-pages/segmented-delete-crash.slow.test.ts(SIGKILL between file removal and commit, and after the restoration rename).src/core/page-body/body-digests.ts— Wave B PR 2: the digests the drift check of a segmented page over the single-string ceiling compares (page-prepare.tssegmentedFileMatches).fileBodyDigests(bytes, reading, filePath, opts)runs the streamed importer's pass 1 (BodyScanner) over one reading of the file's bytes (an inferred-frontmatter prefix, then the bytes past a BOM), then SHA-256 over each region's byte ranges, and reports a facts fence marker;storedBodyDigests(engine, pageId, ledger)hashes each region of the trusted stored body (readBodyRange, one segment per read,readGarbageCollector).bodyDigestWorkcounts file and stored bytes and the largest piece handed to a scan or hash (at most a segment).overlaidFileBodyDigestsoverlays the file one complete fence block at a time; inside an open block a read stops at FENCE_MAX_BYTES plusFENCE_READ_LOOKAHEAD(4 bytes), so an over-bound fence refusespage_body_segmentedwithout reading a further segment (test/pathological/lp-g-file-digest-fence-bound.test.ts). Inpage-prepare.ts,prepareFileTargetcompares a page whose body was not read back throughsegmentedFileMatches(the raw reading, then the inferred one fromstreamed-import.tsinferredHeadPrefix; the same normalizations asfileMatchesSnapshotviaimportedFileType); a file body with a facts fence under a withdrawal ledger that applies refusespage_body_segmentedinstead of reading as drift.restore_pageof such a page (prepareUnreadRestore) versions it, clearsdeleted_atand reseals (or leaves the projection queued when chunks are missing) without re-importing, and publishesserializePageToMarkdown's bytes streamed from the segments bysegmentedExportMarkdowninto one buffer (under the 25 MiB pre-push scan cap); a core page and an external_untrusted writer refusepage_body_segmented, and an agent-tier writer's advisory write-gate scan is not re-run over the unchanged body. Tests:test/large-pages/segmented-file-drift.test.ts.src/core/page-body/body-tiling.ts(untiledSegmentedPages) +schema.ts(liveTilingSql,untiledSql) +src/core/engine-sql/page-body.ts— GST-1209: a live generation tiles its body when its manifest rows, as every reader selects them, coverbody_bytesfrom byte 0 with no gap.flipBodyGenerationchecks the generation it made live inside its transaction and refuses one that does not tile (body_staging_expired, detail reasongeneration_incomplete, nothing published).setBodyStagingStatemoves only a live staging (open, sealed, prepared): a move on to prepared or committed of a staging a fact purge aborted or a sweep expired (its staged rows deleted) refusesbody_staging_expiredinstead of reviving it.untiledSegmentedPageslists live segmented pages of unarchived sources that do not tile (read-only, one aggregate per page), for doctorsegmented_tiling(src/commands/doctor/checks/segmented-tiling.ts, BRAIN, host only, its own registry entrysegmentedTilingEntryinchecks/graph-health.tsso the wave checks' budget never hides it; the fix always names--source) and the repair (surveylists at mostPREVIEW_PAGE_LIMITpages and reportstruncated). A sync hold namingpage_body_rows_unreadablereads needs_human / repair and itsgitHoldFixpreviewsgbrain repair segmented-tiling --source <id> --slug <slug>.src/core/repair/segmented-tiling.ts—gbrain repair segmented-tiling --source <id> [--slug <slug>](GST-1209), explicit-only, preview-bound, trusted local CLI on the owner host,checks: ['segmented_tiling']. The preview lists each untiled page with its file, HEAD's blob (oid, SHA-256, size) and sync hold under a hash; a page with no file, a file HEAD lacks or one overpages.max_bytes, a file another sync hold withholds (any but the unreadable-bodypage_write_failedone) or oneunreadPageImportScreen(sync-screen.ts, the next sync's screen against the unread snapshot) refuses is listed held; a source with no Git checkout refusesnot_a_git_repo.--apply --expectre-checks revision, generation and blob (preview_changed), then per page in one transaction under the key lock (coordinated maintenance on a managed brain) versions the damaged generation, stages the screened body afterpreserveWithdrawnFenceRows(purged claims stay out) throughprepareSegmentedBodywithout reading the damaged body, flips withapplySegmentedBody(title, type, frontmatter, tags and trust tier kept), clears the previewed hold, records the new revision as imported from the HEAD blob (recordSyncImportProvenance) when the file imports to the page's stored title, type, frontmatter and tags (so the next sync holds nofile_drift), callsrequestGitHoldRetryand writes anop_checkpointsreceipt (segmented-tiling-receipt) and asegmented-tilingaudit line. Tests:test/large-pages/segmented-tiling-repair.test.ts.src/core/page-body/segment-facts.ts+segment-conversation.ts— Wave B PR 2 lane 2e2 (spec 8, full-pass mode): the facts fence reconcile and conversation facts over a segmented body.segmentedFactsFenceViewisfactsFenceView(facts-fence.ts: each section's parsed fence, which sections hold one, and the #3625 timeline marker) read from the segments: one SQL query selects the live manifest rows that carry facts slots, a forced cut, or a begin/end marker (byteaposition, nothing transferred), andbodyBlocks'wantoption reads only the runs holding them; the segmenter never cuts inside a fence, so a fence lies in one block (a longer one is a forced run, refused aboveBLOCK_MAX_BYTESwithpage_body_segmented), and joining one across blocks is a bounded fallback.segmentedBodyMatchesFileiscanonicalCacheState's stale-cache check for a segmented page: the canonical file streamed throughBodyScanneragainst the stored regions, by length and SHA-256.readSegmentedConversationInputgivesreadConversationInput(conversation-parser/body.ts) the summary body through ranged reads: whole withinCONVERSATION_WINDOW_BYTES(PAGE_MAX_BYTES), else the newest whole segments that fit (never starting inside a forced run) with the window'sfallbackDatefrom its first segment's entry state; extract-conversation-facts parses that window, passes the date to the parse and the extractor, and writes atail_windowreceipt (window_rule: no_prior_facts), but only on a page with no conversation facts: one that has any keeps them and is skipped (skipped_too_large,window_rule: prior_facts_kept), since the run clears every prior fact. Each loop counts its reads insegmentedPassWork. Consumers:runExtractFacts(planFenceReconcile, the stale-cache guard, the managed fence-facts request), the sweep's fence candidates (segmented pages with facts slots),extract-conversation-facts. Tests:test/large-pages/segmented-facts-reconcile.test.ts,segmented-conversation-facts.test.ts,segmented-derived-linear.test.ts.
Continued in Core Services (2/3, page windows and large writes).