Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
test+docs(grants): pin private-pages operator output; state which reads honor the grant (#8104, #7821)
garrytan98d24e9ec389
diff --git a/docs/architecture/key-files/agent-bootstrap.md b/docs/architecture/key-files/agent-bootstrap.mdindex 36b939ce31..1d4853a780 100644--- a/docs/architecture/key-files/agent-bootstrap.md+++ b/docs/architecture/key-files/agent-bootstrap.md@@ -30,7 +30,7 @@ User-facing contract: `docs/guides/bootstrap.md`. Runbook the paste block fetche- `src/core/bootstrap/opencode-json.ts` — the ONE direct opencode-config writer (workspace stdio lane, harness remote lane, connect `--install`). ALL edits ride jsonc-parser `modify`/`applyEdits` (comments/formatting/EOLs survive byte-for-byte outside the edited range — opencode's own `mcp add` preserves comments, and JSONC is its effective grammar for BOTH `.json` and `.jsonc` filenames, which MERGE when both exist). Ownership is a 4-state STRUCTURAL FINGERPRINT (`opencodeEntryKind` → ours-same-source | ours-other-source | foreign | absent; GBRAIN_SOURCE EQUALITY for local entries, receipt-url match or the `{env:GBRAIN_REMOTE_TOKEN}` interpolation for remote), never a marker key (a future strict-schema flip must not brick the host). Distinct read-failure classes (ENOENT create / empty-as-{} / unreadable refuse); foreign refusal on write AND remove; post-render validation (our entry round-trips + every other key survives) keeps the original on failure; 0600 for inline-bearer targets; backups are UNIQUE per operation (`<config>.bak-<hex>`, returned in the result with `writtenText`, the exact rendered bytes) so overlapping runs can never clobber each other's snapshot, and a backup is chmod'd 0600 whenever the COPIED content carries an inline bearer (write AND remove paths); the ours-other-source refusal is caller-appropriate (url + `--force` wording on the remote/expect-url path, GBRAIN_SOURCE wording on the local path); `removeOpencodeMcpEntry` takes a caller expectation + optional `skipOtherSource` — the uninstall lane passes THIS workspace's source id so an ours-other-source match becomes a calm skip-with-note (another workspace's registration is never deleted; foreign still refuses); `reconcileOpencodeSiblingGlobal` clears a same-name gbrain entry from the OTHER global filename before a global write (foreign → refuse naming both files) — opencode merges both, so a leftover sibling entry is a shadow registration; the bun-run ownership lane is ANCHORED (an exact `gbrain`/`gbrain-*` path segment in some arg — a gbrainy-fork cli path is NOT ours, fail-closed); `parseOpencodeEntryBearer` recovers the harness `--status` token url-matched only; `opencodeRemoteEntryExists` is the stdio-lane ownership arbiter (codexBlockOwnsName analog). Callers hold `acquireBootstrapLock` (config-dir → opencode-dir ordering). Pinned by `test/opencode-json.test.ts` (incl. the fingerprint truth-table suite).- `src/core/bootstrap/atomic-write.ts` — the ONE atomic config-file writer for bootstrap host surfaces: symlink-target-resolving, mode-inheriting (`freshMode` for new files, `forceMode` for secret-bearing targets), random-suffix tmp + rename, tmp unlinked best-effort when write/chmod/rename throws (ENOSPC/EACCES never strand `.tmp-` litter), and the existsSync→realpathSync race falls back to fresh-path resolution instead of throwing raw ENOENT. hooks.ts (JSON), codex-toml.ts (TOML+EOL), and opencode-json.ts (JSONC) all swap through it; serialization and EOL policy stay caller-side.- `src/core/bootstrap/harness.ts` — `gbrain bootstrap harness`: machine-level wiring of framework-spawned Claude Code/Codex/opencode sessions to a RUNNING `gbrain serve --http`, no `agent.json`. The opencode target mirrors the codex posture: forced-wire on explicit `--harness opencode` (the JSONC writer needs no CLI), one managed `mcp.<name>` remote entry with the inline bearer header (0600) via `opencode-json.ts`, rotation across a url change recognized through the PRIOR receipt's url, failed-smoke rollback restoring the run's UNIQUE backup or removing a fresh entry — guarded by a content compare against the exact text this run wrote, so a NEWER registration that landed after the lock released is never clobbered (rollback skips with a note; the fresh mint is revoked either way), with the backup unlinked once consumed or once the smoke verifies — `--remove` classifying against the receipt url (not-ours skips with a note), and `--status` bearer recovery via `parseOpencodeEntryBearer` (url-matched). An explicit private posture (file-plane `memory.visibility_posture`) prints a pre-consent WARNING outside registrar mode: the wired HTTP sessions read world facts only, so they cannot read back what they save as private. Consent block in the honesty register (reach as fact, transcript capture its own numbered item + `--no-capture`, off-ramps in the same breath), then `requireConsent` via `consentGate` (effects `persistent_install`, plus `credentials` when it mints the token; `--yes` approves; the TTY prompt runs only when `isInteractive()`, and EOF/decline or no human at the terminal writes nothing and exits 3 with the consent payload, whose argv never repeats `--token`); /health probe with a loopback guard on `--url` (remote brains are `gbrain connect`'s charter; `--token` makes it a pure registrar); mint-first rotation (previous token revoked BY ID only after every target confirms + the bearer smoke passes — clients are never dead mid-swap; revoke-by-name never happens); the smoke is canary-gated: a random same-format bearer must FAIL auth before the real token is sent, so a loopback impostor is caught whichever way it answers, and ANY failed smoke rolls back symmetrically — fresh registrations removed, replaced ones restored (an unrestorable replacement fails the target honestly), the freshly-added pre-approval stripped, and the fresh mint revoked immediately so nothing live stays pointed at an unverified endpoint; the `permissions.allow` pre-approval only lands after the MCP registration itself confirms; stale prior-target cleanup runs AFTER the smoke passes, so a run that failed to establish its replacement never unwires working prior wiring; write-ahead harness receipt (targets persist as `pending` at mint time and flip per-target, so a crash leaves consumable state); registration ownership checks (`--force` to replace a foreign-url server; `--remove` skips what it no longer owns); user-XOR-project hook scopes; `GBRAIN_HOOK_LANE=harness` on hook commands so `gbrain hook` yields to a workspace bootstrap install in the cwd (Claude Code merges settings scopes — same event must not fire twice); the hook `GBRAIN_SOURCE`, the receipt `source_id`, and the token grant bind to a validated `--source` (scalar write floor), else to the source `HarnessDeps.resolveHookSource` resolves through the SAME chain the serve's resolve-IPC binding runs (`resolveSourceWithTier`: env → dotfile → local_path → `sources.default` → sole populated non-default → seed default) with that source's federated read set as the grant, so the hooks' `turn_context` claim is never `source_mismatch`; a resolution landing on `default`/`__all__` is the federated floor. Lookup failures are NOT papered over with `default`: a typo'd `--source` or stale env/dotfile throws `SOURCE_UNRESOLVED` before any mint or receipt; an explicit `--source` with an unopenable engine binds unverified with a warning; a live PGLite serve (engine cannot open) refuses with `LIVE_SERVE` when no token was supplied, printing (and in `details.steps`) stop / `auth create` + `auth rescope --operations` with the harness grant / restart / `--token` (#6661) and, on the `--token` lane without `--source`, wires the hooks UNPINNED with a warning (an unpinned hook resolves through the live serve's own binding; the receipt records the federated floor `default` plus `source_pinned: false`); any other lookup failure throws `SOURCE_UNRESOLVED` naming `--source`; Postgres degradation + serve-version-skew honesty lines (`isServeOlderThanScopes` pinned to `SCOPES_MIN_SERVE_VERSION`, the first scope-aware release, so later CLI bumps never re-trigger the warning); `--status` probes the live truth with host-config token recovery (redacted; the Claude Code lane recovers a bearer ONLY from a registration whose URL matches the receipt — never another install's credential; the codex fallback reads OUR managed block at the receipt-recorded path, its `url` key not yet compared — TODOS.md; the opencode fallback IS url-compared inside `parseOpencodeEntryBearer`) and honest `verify: unavailable` degrades, with a cron-honest exit contract: 0 only when serve + token + every target verify and the rotation has converged (honest degrades count), 1 on an unreachable serve, a failed token verify, failed/pending targets, unconverged rotations, or a half-removed receipt (zero targets, minted token awaiting deferred revoke — also a doctor FAIL); no receipt prints honestly and exits 0 in plain mode, 2 under `--json`; `--json` on apply emits ONLY the final JSON document on stdout (prose → stderr); `--remove` is engine-free-first and defers the token revoke with exact instructions under a live PGLite serve; it removes hook entries that lost their marker when their command is this install's (`harness-hooks.ts`), keeps the hooks target failed on an unowned lookalike, and `--remove --dry-run` previews without a lock or any write. Hooks targets record `launcher` and `seat` on the receipt (the ownership identity for later runs); the permission target carries `mechanism: 'added'` forward from a prior confirmed receipt target so a re-run never relabels its own `mcp__<name>` as pre-existing, and the failed-smoke rollback removes only an entry added this run. Locks on the gbrain HOME, plus the host config dir around codex/opencode writes, removals, rollbacks, and stale cleanup (stale prior-target cleanup nests the codex/opencode config-dir lock inside the held claude config-dir lock — same claude-first ordering as apply/remove, same-dir skip — so its read-modify-write can't interleave with a concurrent config-dir-locked writer); `bootstrap uninstall` holds the HOME lock across the whole teardown and runs harness removal FIRST (revoke needs the DB alive; `--delete-brain` would destroy harness.json) and treats NO_RECEIPT/HOME_GUARD/RECEIPT_MISMATCH as "no workspace install" once harness wiring is cleared. Pinned by `test/bootstrap-harness.serial.test.ts` + `test/bootstrap-harness-lifecycle.serial.test.ts`. The production mint (`mintHarnessToken`, exported) rotates each host's token from the token it replaces via `carryLegacyGrant`. `--refresh-skills` (`src/core/bootstrap/harness-skills.ts`, `refreshHarnessSkills`) re-joins each live shared-skills entry under its recorded credential so the receipt and native router adopt the server's current enrollment epoch, with no token rotation; `localEnrollment` reads the adapter receipt epoch for `--status`, and `supersededEnrollments` is doctor's read-only comparison against `shared_skill_members` (`bootstrap_harness_health` warn, code `shared_skills_epoch_superseded`). Pinned by `test/harness-skills-epoch.test.ts`. The Codex SessionEnd capture hook obeys the same recorded consent as the Claude lane (`wireCodexCapture`: hooks on, capture accepted, never registrar mode) and is a numbered consent item when wired. #6661 option B: `harness-pending-verify.ts` (a `--token` run on PGLite with no serve answering ends `pending_verify`, exit 13, re-run as `fix`; `--status` partial until it passes).-- `src/core/token-mint.ts` — programmatic legacy-token mint/revoke for the harness lane: `mintLegacyToken` (scopes → the TEXT[] column; required `takesHolders` per-token allow-list, harness default `['world']`; optional `permissions.source_id` federation array mirroring the stdio lane's `localFederatedSourceIds` grant, element 0 = write floor, `[]` stored as the explicit no-source grant; `RETURNING id`) and `revokeLegacyTokenById` (never touches same-name siblings). `carryLegacyGrant` + `readActiveTokenPermissions` are the harness rotation carry-over: takes holders and the source grant carry as stored (explicit empty lists included; an explicit `--source` wins), a stored operation snapshot narrows to the run's snapshot plus only skills-policy-added operations, and newer operations are reported as withheld (pinned by `test/harness-rotation-grants.test.ts`). Exports `TOKEN_ID_RE`, the canonical token-id shape shared with the `auth revoke --id` CLI gate. Canonical `hashToken`/`generateToken` from `src/core/utils.ts`. Pinned by `test/token-mint.test.ts`. `insertUnifiedToken` (shared with `auth create`) births every token on the unified grant columns at revision 1 plus the `permissions` mirror; a brain whose schema predates the columns refuses with `migrations_pending` (fix: `gbrain apply-migrations --yes`) and nothing is written; `carryLegacyGrant` parses the prior grant from its JSONB through `tokenGrantFromPermissions`, so a drifted prior carries its JSON edit (the `--adopt-permissions` resolution).+- `src/core/token-mint.ts` — legacy-token mint/revoke for the harness lane: `mintLegacyToken` (scopes → the TEXT[] column; required `takesHolders` per-token allow-list, harness default `['world']`; optional `permissions.source_id` federation array mirroring the stdio lane's `localFederatedSourceIds` grant, element 0 = write floor, `[]` stored as the explicit no-source grant; `RETURNING id`) and `revokeLegacyTokenById` (never touches same-name siblings). `carryLegacyGrant` + `readActiveTokenPermissions` are the harness rotation carry-over: takes holders and the source grant carry as stored (explicit empty lists included; an explicit `--source` wins), a stored operation snapshot narrows to the run's snapshot plus only skills-policy-added operations, newer operations are reported as withheld, and `private_pages` is never carried (`readActiveTokenPrivatePages`; the harness prints the rescope; pinned by `test/harness-rotation-grants.test.ts`). Exports `TOKEN_ID_RE` (shared with the `auth revoke --id` gate). Pinned by `test/token-mint.test.ts`. `insertUnifiedToken` (shared with `auth create`) births every token on the unified grant columns at revision 1 plus the `permissions` mirror; a brain whose schema predates the columns refuses with `migrations_pending` (fix: `gbrain apply-migrations --yes`) and nothing is written; `carryLegacyGrant` parses the prior grant from its JSONB through `tokenGrantFromPermissions`, so a drifted prior carries its JSON edit (the `--adopt-permissions` resolution).- `src/commands/hook.ts` — engine-free `gbrain hook {session-start,user-prompt,stop,session-end,compact}` (zero engine modules in the import graph; a hook must NEVER contend for the PGLite writer lock). session-start's context pack goes through `src/core/context/session-start-pack.ts`: when `sessionStartAwaitsServe` holds (source `startup`/`resume`, harness claude-code, `claudeCodeSpawnsServe(cwd)`), the hook's deadline widens from `SESSION_START_DEADLINE_MS` (1.5s) to `SESSION_START_SERVE_WAIT_DEADLINE_MS` (4s, under the 5s harness timeout) and `requestSessionStartPack` retries every 100ms, re-resolving socket and secret, until the serve Claude Code spawned alongside SessionStart answers (Cat 41 first push: Claude Code 2.1.285 starts MCP servers in parallel with SessionStart and a fresh serve binds ~2s later); a wait that runs out records heartbeat `degraded`/`serve_not_ready`. Pinned by `test/hook-session-start-serve-wait.serial.test.ts`. When `GBRAIN_HOOK_LANE=harness` (set on harness-mode hook commands), each event PARSES the cwd's workspace settings carriers — `.claude/settings.local.json` AND the committed `.claude/settings.json` — and yields silently only when a live `bootstrap-v1` hook entry wires THAT event: the workspace install wins over user-scope harness wiring so the merged settings scopes can't fire the same event twice, unwired events still run, and a repo committing marker-lookalike strings in unrelated fields can't disable the machine-wide capture lane (fail-open — a read/parse hiccup means the event runs normally). user-prompt: stdin hook JSON → transcript-path confinement → last-4-turns window + cross-turn dedupe (the transcript's `hook_additional_context` attachments — the blocks WE previously injected — ride `priorContextText`, deduplicated and capped at `PRIOR_CONTEXT_MAX_BYTES` (32KB, so the advisory payload can never blow the IPC message cap; one oversized block is skipped without evicting smaller ones), so a page is volunteered once per session, not once per mention; structured extraction only, never raw-turn substring matching) → IPC turn_context (with a feedback-loop `channel`, `--harness <claude-code|codex>`, default claude-code) → `hookSpecificOutput.additionalContext` under an 800ms self-deadline; every path fails open (exit 0, empty stdout) with a typed reason in the heartbeat. Listed in cli.ts's `STARTUP_HOOK_SKIP_COMMANDS` (per-prompt invocations must never spawn a detached check-update child; membership is pinned by a source grep — the runtime path no-ops under NODE_ENV=test). session-start: file-plane digest (allowlisted MEMORY.md sections, push staleness, prior failures) + crashed-session recovery push gated on an initialized manifest. session-end: a transcript path or payload cwd carrying the claude-cli scratch fingerprint (`isClaudeCliSelfTranscriptPath`) is gbrain's own subprocess session — heartbeat `segment: self_transcript`, no parse, no corpus write; otherwise confined full-transcript parse → redacted corpus write (session-id filename dedup, retention prune via `gcCorpusTurnFiles`: an extracted `.txt` goes at `corpus_retention_days`, one with no `.ingested` sidecar only at `CORPUS_UNINGESTED_RETENTION_FACTOR` (3) times that, E-N1, pinned by `test/corpus-retention-uningested.serial.test.ts`; an assistant-only whole-file parse heartbeats `degraded`/`no_user_turns`, E-N4; the corpus scan runs highEntropy whenever the relay is on — the relay child derives its egress task line from the corpus text — and stays vendor-prefix-only otherwise) → parser-drift detection (`bytes>0 && turns==0` is loud) → best-effort workspace push. session-start recovery + session-end pushes run in a DETACHED child so the hook returns immediately (a synchronous inline push would block harness startup on a dirty tree); the corpus write is atomic and clears the stale ingested/in-progress sidecars so a resumed session re-ingests its appended transcript. Heartbeat JSONL is counters/reasons only by construction; `readHeartbeatTail` feeds doctor. `GBRAIN_HOOKS=0` kills all events. Session-end capture dispatches per harness through `captureSpecFor(io.harness)` (`src/core/transcripts/capture-spec.ts`): claude-code and codex each pin their OWN confinement root + parser; unknown/undefined/opencode resolve to the claude spec (golden-pinned). The codex lane confines to `codexSessionsDir()` + `codexArchivedSessionsDir()` (codex moves rollouts into the flat archived store), parses rollouts via `parseCodexHookTranscript`, and falls back to bounded id-matched discovery across both stores when the SessionEnd payload carries `transcript_path: null` or a path that no longer exists (`missing_path`; `unreadable` is reserved for EACCES/IO faults) — the fallback and prior-run relay failures are DEFERRED heartbeat reasons, applied after current-session reasons so they never mask one (first-degrade-wins). ONE gate covers the whole optional Memorable seam — `memorableGateAllowed(cfg)` in hook-heartbeat.ts: the config flag (`integrations.memorable.enabled === true`, file plane), the `GBRAIN_MEMORABLE` kill switch (any common negative spelling, trimmed; env can only ever disable), AND the gbrain-authored consent stamp (`memorable-consent.json` — see hook-heartbeat.ts). Off is the default: no tool-call collection, no receipt, no spawn. On, session-end runs the shared `redactedToolCallsJson` (span-filtered to the corpus window, highEntropy ALWAYS on — those args are the one artifact that leaves the machine) and `recordAndRelayReceipt` (receipt dedup by post-redaction content hash → CLI-side consent evidence → `resolveMemorableBin` → detached fire-and-forget `memorable record` spawn; both capture lanes trim memorable-relay.jsonl). Fail-closed egress guards: a scanner-import failure skips the receipt+relay entirely (`memorable_relay_skipped_unscanned` — parity with the openclaw lane), and a newest-mtime discovery GUESS (no session-id match) writes the local corpus but never relays (`memorable_relay_skipped_newest_guess` — the newest rollout can be a different, still-running session). A payload without session_id adopts the transcript's own id post-parse (never a shared unknown.txt corpus), and the gate answer is hoisted BEFORE the parse and collection is opt-in, so the default gate-off population never collects tool calls (`collectToolCalls: memorableAllowed`). Operator-facing doc: `docs/memorable-agents.md` (install is `npm i -g memorable-cli`; closed source, no source install, no embedding model to configure). `spawnDetachedPush` passes `env: process.env` explicitly so the detached push child inherits the parent's quarantined env view. `tryExecAsync` (the git wrapper behind `resolveBootstrapWorkspaceRoot` / `repoPhaseComplete`) also passes `env: process.env` so git sees the quarantined view, not Bun's startup snapshot; pinned by `test/hook-push-spawn-env.test.ts`. Seat identity (#4618): session-end (before any corpus write of the session), compact (before `bankCompactSegment`) and the stop hook's writeback bank (before `bankWritebackTurn`) record `<sessionId>.seat.json` through `context/seat.ts` — `GBRAIN_SEAT` when valid, else the hashed harness-home seat; `seat_label_invalid`, `seat_conflict` (first seat kept) and `seat_write_failed` are heartbeat reasons with a fixed `hint` (session-end applies them last, like the deferred reasons). `gcCorpusArtifacts` reaps a seat sidecar or session source record once no `.txt` of its session remains, after a 10-minute grace; session-end counts live sessions across the spool and the legacy top level (#6611). Pinned by `test/session-seat-capture.serial.test.ts`. `HOOK_EVENTS` is the router inventory from `src/cli/subcommands.ts`; `hook <event> --help` prints usage and never runs the event. `user-prompt` confines and parses `transcript_path` through `captureSpecFor(io.harness)`, so `--harness codex` reads the rollout under `CODEX_HOME/sessions` (the Claude projects root otherwise), and skips the Claude-only pressure notice for codex. #5941: the pack and user-prompt arms ask for `contextSourceId` (session-start-pack.ts): `GBRAIN_SOURCE`, else the session's frozen source record, else the cwd's `.gbrain-source` walk; a serve bound elsewhere answers `source_mismatch` (the heartbeat reason).- `src/core/transcripts/claude-code-jsonl.ts` — the Claude Code transcript parser as a dated spec-target (tool_use/tool_result/thinking/image/sidechain/summary/compact-boundary shapes; placeholders for non-text content). `recognizedOnly`: whole file read, every line parsed, no conversation message, some session scaffolding (session-end: `empty_session`). Both the hook-tail and full-session parsers skip root `isSidechain`, `isMeta`, or `isCompactSummary` exactly `true`; nested markers and truthy non-booleans do not exclude a turn. User text with a non-empty string root `origin.kind` other than `'human'` is excluded; missing, empty, or unstructured origins stay compatible, without guessing from message text. Assistant text and non-text placeholders remain eligible, while skipped metadata entries also contribute no collected tool calls. `test/claude-code-jsonl-origin.test.ts` uses synthetic fixtures, not new live-harness validation. `genuineUserTurnIndexes` separately records accepted user-role turns containing real text blocks (harness tags and paste blocks removed first), because Claude stores tool-result-only entries with `role:'user'`; archival consumers keep every placeholder while prompt-only consumers select by structure. Also extracts `injectedContextBlocks` — the `hook_additional_context` attachment lines a gbrain hook previously injected (verified live against claude CLI 2.1.224), plus a SessionStart `hook_success` attachment's `content` from the gbrain envelope on (session-start stdout, #7026; marker-filtered, so a foreign hook's blocks are excluded and another tool's output can't suppress volunteering — a same-user mislabeling guard, not an authenticity check), the user-prompt hook's cross-turn dedupe input; `confineTranscriptPath` (contained under `~/.claude/projects`, `.jsonl`, lstat-rejects symlinks, byte cap; under WSL, a Windows drive-literal `transcript_path` — Claude Code on the Windows host invoking hooks via `wsl.exe` — is translated through `src/core/wsl-paths.ts` and then held to the SAME confinement, rooted Windows-side, so translation never widens what a hook may read). Fixtures: `test/fixtures/conversation-formats/claude-code.jsonl` (synthetic, privacy-guarded) + `test/fixtures/hook-transcript.jsonl` (real captured hook round-trip). `parseTranscript` additionally returns `toolCalls`, but collection is OPT-IN (`collectToolCalls: true`; the bare parse — every per-prompt lane — collects nothing, so tool INPUTS are never retained for users who never opted in): the tool name + input args for every `tool_use` block, oldest → newest, each joined to its `tool_result` outcome by `tool_use_id` and then stripped of that id so no transcript-internal identifier reaches a consumer; every STRING in a collected input is bounded to `TOOL_CALL_VALUE_MAX_CHARS` (32k) with an explicit `…[N chars omitted]` marker (`capToolCallInput`, shared with the codex lane — one receipt-size ceiling for every harness). It is a parallel extraction, never a change to the placeholder-only archival rendering, which is load-bearing for the token-budget-constrained ambient-recall path; existing callers that ignore the field see no behavior change.- `src/core/transcripts/pasted-content.ts` — #5812, the ONE Claude Code paste detector (`<pasted_content id="…">…</pasted_content id="…">`, tag shape from a real 2.1.287 transcript; depends on that harness format). `stripPastedContent` removes closed blocks and an unclosed tail only when the opening tag has the exact harness shape, scanning with `indexOf` (no lazy regex); `stripPastedContentFromCorpus` strips inside each `[user]` block of the `toCorpusText` rendering, never across the flattened corpus, so an unclosed paste ends with its turn. Consumers: the writeback gate (banked text), `isGenuineUserText` in the parser (classification only; turn, corpus, segment and ingest text keep the paste), and `corpusTextForExtraction` (`corpus-segments.ts`: a `.wb-` file is one turn, other corpus files per block) used by the sweep, the checkpoint harvest and dream `extract_atoms`. `synthesize` keeps pastes. Tests: `test/transcripts-pasted-content.test.ts`, `test/facts/writeback-gate.test.ts`, `test/extract-atoms-pasted-content.test.ts`.diff --git a/docs/architecture/key-files/core-persistence-facts.md b/docs/architecture/key-files/core-persistence-facts.mdindex 28f34718f7..e49db9103d 100644--- a/docs/architecture/key-files/core-persistence-facts.md+++ b/docs/architecture/key-files/core-persistence-facts.md@@ -21,4 +21,4 @@ Managed facts extraction: the entity-intent preparer and publisher, conversation- `src/commands/doctor/checks/fact-bulk-dates.ts` — `fact_bulk_dates` (wave check, #7247): `findBulkFactDays` over the scoped sources (remote callers: their permitted sources); warns per day with fact and entity counts, top source tags and `dated_claims`, says the dates record loading, not occurrence, and names the `gbrain repair fact-claim-dates --source <id>` preview. Read-only.- `src/core/repair/fact-claim-dates.ts` — `gbrain repair fact-claim-dates --source <id> [--day YYYY-MM-DD]` (#7247), explicit-only, preview-bound, destructive (`--yes`). Candidates: active facts on a bulk day (`findBulkFactDays`, or `--day`) whose claim has exactly one earlier ISO date (`claimDate`). A fence row's `valid_from` cell is rewritten in the page's one fence section and published by revision-bound `put_page` (the facts row is re-projected); a database-only fact is updated in one `maintenanceTransaction` on an unmanaged brain and counted `database_only_managed` on a managed one. Pages with an unparsed, split or segmented fence, or a rewrite that does not read back as the planned rows, are listed and kept. The hash binds revisions, row numbers and new dates; a fresh apply recomputes the plan and refuses `preview_changed` on any difference. Test: `test/repair-fact-claim-dates-7247.test.ts`.-- `src/core/facts/fence-fact-append.ts` — `prepareFenceFactAppend(engine, row, config, snapshot, { fact, pendingTier, mode?, supersede? | rowNum }, signal)`: the one way a coordinator preparer puts a single fact into the facts fence of the request's page (`row.slug`); remember uses it, and so does a managed owner release of a held fact (`trust/release-publish.ts`, the `trust_release` operation: append for a held fact with an entity slug, index-existing for a hold naming `payload.row_num`, whose cell fields the owner previewed and the request binds: a moved cell refuses `preview_changed`). It returns `{ prepared, rowNum, fencesNormalized, apply(tx) }`: `prepared` is the page write to compose (its `file`, `exclusiveSources`, `validate`; null in index-existing), and `apply` runs in the publication transaction after the coordinator published the file, writing the page row (tier kept, `trust/fence-append.ts`) and then the facts row with `row_num` and `source_markdown_slug`, so a facts row never precedes its fence row in Git. `append` (default) normalizes the stored fence (#6188), renders the row with `fenceRowOf` (`unfenced-facts.ts`, the one renderer) at one past the highest row number in the fence and among the page's facts rows (#6696 C9), strikes `supersede.rowNum` when given, and prepares the page write `unprojected` (`preparePageMutation` option: the helper writes the facts row, so the page publishes without canonical projection whatever the operation) under `withPendingFenceRows` at `pendingTier`. `index-existing` takes a row the file already holds (a fence-guard hold's `payload.row_num`): the cell must pass `fenceCellReadBackProblem` against the fact, else it refuses `fence_row_changed` (reason `row_missing` / `struck` / `empty` / `visibility` / `text`) and writes nothing; otherwise the facts row derives from the cell (`extractFactsFromFenceText`, what a re-import projects) with the caller's embedding and session, and no file changes; the caller's observedRevision is `snapshot.revision`, which the coordinator re-checks under the page lock. An occupied row number fails the insert with `storage_error` naming the occupant. Tests: `test/fence-fact-append.test.ts` (PGLite and Postgres: append, supersede, index-existing from a real fence-guard hold, each refusal, and remember's fence row byte for byte against the earlier construction).+- `src/core/facts/fence-fact-append.ts` — `prepareFenceFactAppend(engine, row, config, snapshot, { fact, pendingTier, mode?, supersede? | rowNum }, signal)`: the one way a coordinator preparer puts a single fact into the facts fence of the request's page (`row.slug`); remember uses it, and so does a managed owner release of a held fact (`trust/release-publish.ts`, the `trust_release` operation: append for a held fact with an entity slug, index-existing for a hold naming `payload.row_num`, whose cell fields the owner previewed and the request binds: a moved cell refuses `preview_changed`; journaled as mode `index-previewed`, which a preparer built before the field preview refuses, and an `index-existing` request with no fields digest refuses `preview_changed` asking for a rerun). It returns `{ prepared, rowNum, fencesNormalized, apply(tx) }`: `prepared` is the page write to compose (its `file`, `exclusiveSources`, `validate`; null in index-existing), and `apply` runs in the publication transaction after the coordinator published the file, writing the page row (tier kept, `trust/fence-append.ts`) and then the facts row with `row_num` and `source_markdown_slug`, so a facts row never precedes its fence row in Git. `append` (default) normalizes the stored fence (#6188), renders the row with `fenceRowOf` (`unfenced-facts.ts`, the one renderer) at one past the highest row number in the fence and among the page's facts rows (#6696 C9), strikes `supersede.rowNum` when given, and prepares the page write `unprojected` (`preparePageMutation` option: the helper writes the facts row, so the page publishes without canonical projection whatever the operation) under `withPendingFenceRows` at `pendingTier`. `index-existing` takes a row the file already holds (a fence-guard hold's `payload.row_num`): the cell must pass `fenceCellReadBackProblem` against the fact, else it refuses `fence_row_changed` (reason `row_missing` / `struck` / `empty` / `visibility` / `text`) and writes nothing; otherwise the facts row derives from the cell (`extractFactsFromFenceText`, what a re-import projects) with the caller's embedding and session, and no file changes; the caller's observedRevision is `snapshot.revision`, which the coordinator re-checks under the page lock. An occupied row number fails the insert with `storage_error` naming the occupant. Tests: `test/fence-fact-append.test.ts` (PGLite and Postgres: append, supersede, index-existing from a real fence-guard hold, each refusal, and remember's fence row byte for byte against the earlier construction).diff --git a/docs/guides/multi-source-brains.md b/docs/guides/multi-source-brains.mdindex 4e0a19a4e9..a3fa5f252f 100644--- a/docs/guides/multi-source-brains.md+++ b/docs/guides/multi-source-brains.md@@ -256,13 +256,18 @@ who may read it, before any member's bot connects:```A token with no source grant, a federated source the grant does not name,- and a request's `source_ids` never add a source; `context_pack`, `delta`- and writes keep hiding private pages. In a named source the grant also- opens pages with a privacy alias (`access`, `confidentiality`), atoms,- concepts and events derived from private or unknown origins, and pages- whose declared lineage is private; pinned questions stay owner-only.- `--private-pages none` hides them again on the next request, and- `gbrain auth list` shows each token's setting.+ and a request's `source_ids` never add a source. The grant applies to+ search, query, recall, entity and the page, chunk, version, timeline, link+ and graph reads; `context_pack`, `delta`, `think`, recorded-time reads+ (`known_as_of`) and writes keep hiding private pages even in a granted+ source, so a page the agent can open may read as missing there. In a+ named source the grant also opens pages with a privacy alias (`access`,+ `confidentiality`), atoms, concepts and events derived from private or+ unknown origins, and pages whose declared lineage is private; pinned+ questions stay owner-only. `--private-pages none` hides them again on the+ next request, and `gbrain auth list` shows each token's setting. A harness+ token rotation (`gbrain setup`, `gbrain bootstrap harness`) does not carry+ the grant; it prints the rescope that grants it again.> **Warning: one shared source has no boundary inside it.** A bot that answers> everyone with one unscoped token over a single source exposes everything indiff --git a/docs/mcp/ADMIN.md b/docs/mcp/ADMIN.mdindex 664f96c23f..ce070ec3a9 100644--- a/docs/mcp/ADMIN.md+++ b/docs/mcp/ADMIN.md@@ -460,7 +460,7 @@ gbrain auth rescope --client CLIENT_ID --private-pages none # hide them again| `--takes-holders` | `a,b` or `none` | Takes-holder allow-list for a token or client (default `world`); `none` hides every take. || `--operations` | `op,...`, `none` or `all` | Operation snapshot; `none` refuses every operation. Client only: `all` stores no snapshot and clears the profile, so the scopes and the surface alone decide, including operations later upgrades add (tokens: `--reset-default operations`). || `--scopes` | `read,write,...` | Replaces the scopes. |-| `--private-pages` | `granted-sources` or `none` | Whether remote reads see `visibility: private` pages. `none` (the default for every token and client) hides them, as before. `granted-sources` shows them only in the sources this grant names: a token's `--sources` list, or a client's write source plus its read sources. The no-grant floor, federated sources the grant does not name, a client's bound source and a request's `source_id` / `source_ids` never add a source. `context_pack`, `delta` and every write keep hiding private pages. In a named source it also opens privacy-alias pages (`access`, `confidentiality`), atoms, concepts and events derived from private or unknown origins, and pages with private declared lineage; pinned questions stay owner-only. Stored in its own column, not in the legacy permissions JSON. |+| `--private-pages` | `granted-sources` or `none` | Whether remote reads see `visibility: private` pages. `none` (the default for every token and client) hides them, as before. `granted-sources` shows them only in the sources this grant names: a token's `--sources` list, or a client's write source plus its read sources. The no-grant floor, federated sources the grant does not name, a client's bound source and a request's `source_id` / `source_ids` never add a source. The grant applies to `search`, `query`, `recall`, `entity`, `get_page`, `list_pages`, `get_chunks`, `get_versions`, `get_timeline`, `get_links`, `get_backlinks`, `traverse_graph` and `resolve_slugs`. `context_pack`, `delta`, `think`, recorded-time reads (`get_page` and `get_versions` with `known_as_of`) and every write keep hiding private pages even in a granted source, so a page `get_page` returns can read as missing there. In a named source it also opens privacy-alias pages (`access`, `confidentiality`), atoms, concepts and events derived from private or unknown origins, and pages with private declared lineage; pinned questions stay owner-only. Stored in its own column, not in the legacy permissions JSON, so a harness token rotation does not carry it; the rotation prints the rescope that grants it again. || `--reset-default` | `sources,takes-holders,operations` | Token only: restores the `auth create` default for those axes: no source grant (the historical `default` floor), holders `world`, no operation snapshot. || `--refresh-operations` | with optional `--add op,...` or `--all-new` | Token only: previews operations added since the snapshot. Without `--add` or `--all-new` nothing is widened. || `--if-version` | `N` | Refuses unless the stored grant revision is `N`; the refusal names the current revision. |diff --git a/test/auth-private-pages-output.test.ts b/test/auth-private-pages-output.test.tsnew file mode 100644index 0000000000..89df687a8c--- /dev/null+++ b/test/auth-private-pages-output.test.ts@@ -0,0 +1,119 @@+/**+ * #8104 item 3: the operator output of the private-pages grant axis (GBRA-156). Authoring gate:+ * (1) protects `gbrain auth list --json` (its exact field set: no token_hash, a status and a private_pages field per token),+ * the "Private pages" column of `auth list`, `auth clients` (text and JSON) and `auth rescope` (token and client), the+ * `renderPrivatePages` labels, and the top rung of the `listClientRows` fallback ladder (a brain without the column+ * reads as none); (2) fails when token_hash or another raw column reaches the JSON, a granted principal prints as none,+ * or a brain without the column errors instead of listing; (3) existing coverage pinned none of this output;+ * (4) seam: the real CLI handlers against a PGLite brain under a sandboxed GBRAIN_HOME (console.log captured).+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { createEngine } from '../src/core/engine-factory.ts';+import type { BrainEngine } from '../src/core/engine.ts';+import { listClientRows, renderPrivatePages, runAuth } from '../src/commands/auth.ts';+import { privatePagesFromColumn } from '../src/core/grants/model.ts';+import { mintLegacyToken } from '../src/core/token-mint.ts';+import { withEnv } from './helpers/with-env.ts';++let root: string, home: string, db: string;+const GRANTED = 'granted-sources (private pages in the sources this grant names)';++async function withBrain<T>(run: (engine: BrainEngine) => Promise<T>): Promise<T> {+ const config = { engine: 'pglite' as const, database_path: db };+ const engine = await createEngine(config);+ await engine.connect(config);+ try { return await run(engine); } finally { await engine.disconnect(); }+}++const env = () => ({ GBRAIN_HOME: home, HOME: home, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined, GBRAIN_SOURCE: undefined });++/** Runs one `gbrain auth` command and returns what it printed to stdout. */+async function auth(args: string[]): Promise<string> {+ const lines: string[] = [];+ const original = console.log;+ console.log = (...parts: unknown[]) => { lines.push(parts.join(' ')); };+ try { await withEnv(env(), () => runAuth(args)); } finally { console.log = original; }+ return lines.join('\n');+}++beforeAll(async () => {+ root = mkdtempSync(join(tmpdir(), 'gb-auth-private-pages-'));+ home = join(root, 'home');+ db = join(root, 'db');+ mkdirSync(join(home, '.gbrain'), { recursive: true });+ writeFileSync(join(home, '.gbrain', 'config.json'), JSON.stringify({ engine: 'pglite', database_path: db }));+ await withBrain(async engine => {+ await engine.initSchema();+ const granted = await mintLegacyToken(engine, { name: 'alice-example', scopes: ['read'], takesHolders: ['world'], sourceGrant: ['default'] });+ await engine.executeRaw("UPDATE access_tokens SET private_pages = 'granted_sources' WHERE id = $1::uuid", [granted.id]);+ await mintLegacyToken(engine, { name: 'bob-example', scopes: ['read'], takesHolders: ['world'] });+ const revoked = await mintLegacyToken(engine, { name: 'old-example', scopes: ['read'], takesHolders: ['world'] });+ await engine.executeRaw('UPDATE access_tokens SET revoked_at = now() WHERE id = $1::uuid', [revoked.id]);+ for (const [id, pages] of [['client-granted', 'granted_sources'], ['client-plain', 'none']]) {+ await engine.executeRaw(`INSERT INTO oauth_clients (client_id, client_name, scope, source_id, federated_read, private_pages)+ VALUES ($1, $1, 'read', 'default', ARRAY['default'], $2)`, [id, pages]);+ }+ });+}, 120_000);++afterAll(() => { rmSync(root, { recursive: true, force: true }); });++describe('private-pages grant output (#8104)', () => {+ test('renderPrivatePages labels the two modes; anything else reads as none', () => {+ expect(renderPrivatePages('granted_sources')).toBe(GRANTED);+ expect(renderPrivatePages('none')).toBe('none');+ expect(renderPrivatePages(undefined)).toBe('none');+ expect(privatePagesFromColumn('granted-sources')).toBe('none');+ expect(privatePagesFromColumn(null)).toBe('none');+ });++ test('auth list --json prints exactly id, name, scopes, created_at, last_used_at, status and private_pages: never token_hash', async () => {+ const tokens = JSON.parse(await auth(['list', '--json'])) as Array<Record<string, unknown>>;+ for (const t of tokens) {+ expect(Object.keys(t).sort()).toEqual(['created_at', 'id', 'last_used_at', 'name', 'private_pages', 'scopes', 'status']);+ }+ expect(JSON.stringify(tokens)).not.toContain('token_hash');+ const byName = Object.fromEntries(tokens.map(t => [t.name as string, [t.status, t.private_pages]]));+ expect(byName).toEqual({ 'alice-example': ['active', 'granted_sources'], 'bob-example': ['active', 'none'], 'old-example': ['revoked', 'none'] });+ });++ test('auth list prints a Private pages column', async () => {+ const text = await auth(['list']);+ expect(text.split('\n')[0]).toContain('Private pages');+ expect(text.split('\n').find(l => l.includes('alice-example'))).toEndWith(GRANTED);+ expect(text.split('\n').find(l => l.includes('bob-example'))).toEndWith(' none');+ });++ test('auth clients prints the private pages line and the private_pages JSON field', async () => {+ const text = await auth(['clients']);+ expect(text).toContain(`client-granted (client-granted)`);+ const block = (id: string) => text.slice(text.indexOf(`${id} (${id})`)).split('\n').slice(0, 5).join('\n');+ expect(block('client-granted')).toContain(` private pages: ${GRANTED}`);+ expect(block('client-plain')).toContain(' private pages: none');+ const json = JSON.parse(await auth(['clients', '--json'])) as { clients: Array<{ client_id: string; private_pages: string }> };+ expect(Object.fromEntries(json.clients.map(c => [c.client_id, c.private_pages]))).toEqual({ 'client-granted': 'granted_sources', 'client-plain': 'none' });+ });++ test('auth rescope prints the private pages change for a token and the axis for a client', async () => {+ const token = await auth(['rescope', '--token', 'bob-example', '--private-pages', 'granted-sources']);+ expect(token).toContain(` Private pages: none -> ${GRANTED}`);+ const client = await auth(['rescope', '--client', 'client-plain', '--private-pages', 'granted-sources']);+ expect(client).toContain(` Private pages: ${GRANTED}`);+ });++ test('a brain without the private_pages column lists its clients with private pages none (the top rung of the fallback ladder)', async () => {+ // Last test of the file: the column is dropped from this throwaway brain.+ const rows = await withBrain(async engine => {+ await engine.executeRaw('ALTER TABLE oauth_clients DROP COLUMN private_pages');+ return listClientRows(engine);+ });+ const granted = rows.find(r => r.client_id === 'client-granted')!;+ expect(granted.private_pages).toBeUndefined();+ expect(granted.allowed_operations).not.toBeUndefined();+ expect(privatePagesFromColumn(granted.private_pages)).toBe('none');+ expect(await auth(['clients'])).toContain(' private pages: none');+ });+});diff --git a/test/private-pages-grant.test.ts b/test/private-pages-grant.test.tsindex 56faed48dc..cd82595d8d 100644--- a/test/private-pages-grant.test.ts+++ b/test/private-pages-grant.test.ts@@ -264,6 +264,7 @@ for (const kind of testBackends()) {for (const who of ['alice-example', 'client-alice', 'bot-all', 'floor']) {const out = await call(tokens[who], 'delta', { since, include_private: true });expect({ who, isError: out.isError, listed: out.text.includes(PRIVATE_SLUG) }).toEqual({ who, isError: false, listed: false });+ if (who === 'alice-example') expect({ who, world: out.text.includes(TARGET_SLUG) }).toEqual({ who, world: true });for (const secret of Object.values(SECRET)) expect({ who, leaked: out.text.includes(secret) }).toEqual({ who, leaked: false });}});@@ -282,7 +283,7 @@ for (const kind of testBackends()) {expect({ owner, slug, seen: out.text.includes(canary) }).toEqual({ owner, slug, seen: !slug.startsWith('questions/') });}const listed = await call(tokens[owner], 'list_pages', { slug_prefix: 'questions/' });- expect({ owner, listed: listed.text.includes('questions/harbor-pin') }).toEqual({ owner, listed: false });+ expect({ owner, isError: listed.isError, listed: listed.text.includes('questions/harbor-pin') }).toEqual({ owner, isError: false, listed: false });}for (const other of ['bob-example', 'floor', 'alice-none']) {for (const [slug, canary] of extra) {@@ -292,6 +293,26 @@ for (const kind of testBackends()) {}});+ // Value: protects=#8104: get_versions in a granted source hides an older version that carried the pinned_question marker even when the+ // live page no longer carries it; fails_when=the widened arm checks the marker on the live page only; why_new=the pinned-question case+ // above keeps the marker on the live page; seam=none (owner edits through importFromContent); no_claim=how a marker leaves a question page+ test('get_versions in a granted source hides a version that carried the pinned_question marker after the live page dropped it', async () => {+ const slug = 'questions/harbor-unpinned';+ const page = (marker: boolean, canary: string) => ['---', 'type: question', 'title: Harbor unpinned', 'visibility: private',+ ...(marker ? ['pinned_question: pin-example'] : []), '---', '# Harbor unpinned', '', `Owner note ${canary}.`].join('\n');+ await importFromContent(engine, slug, page(true, 'canarypinnedversion'), { sourceId: ALICE });+ await engine.createVersion(slug, { sourceId: ALICE });+ await importFromContent(engine, slug, page(false, 'canaryunpinnedversion'), { sourceId: ALICE });+ await engine.createVersion(slug, { sourceId: ALICE });+ const all = await local('get_versions', { slug, include_body: true });+ expect({ pinned: all.includes('canarypinnedversion'), unpinned: all.includes('canaryunpinnedversion') }).toEqual({ pinned: true, unpinned: true });+ for (const owner of ['alice-example', 'client-alice']) {+ const out = await call(tokens[owner], 'get_versions', { slug, include_body: true });+ expect({ owner, isError: out.isError, pinned: out.text.includes('canarypinnedversion'), unpinned: out.text.includes('canaryunpinnedversion') })+ .toEqual({ owner, isError: false, pinned: false, unpinned: true });+ }+ });+test('a rescope back to none hides the pages on the next call (token and client)', async () => {expect((await call(tokens['alice-example'], 'get_page', { slug: PRIVATE_SLUG })).text).toContain(SECRET[ALICE]);await rescopeToken('alice-example', '--private-pages', 'none');