Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: keep storage cards as restricted as the pages they came from
garrytand7a54effbc2e
diff --git a/changelog.d/capy-storage-manifest-adopt.md b/changelog.d/capy-storage-manifest-adopt.mdindex 133307ae8b..c3057e0738 100644--- a/changelog.d/capy-storage-manifest-adopt.md+++ b/changelog.d/capy-storage-manifest-adopt.md@@ -1,16 +1,16 @@**A storage manifest: agents can register what they archived to object storage and find it again later.** A new `storage` source holds one validated card per archived item: what it is, where every copy lives (store, bucket, path, restic repository and snapshot, size, file count, hash and its kind, when it was verified), where it came from and which skill restores it. `gbrain storage-register` adds a card (from stdin or `--card`) and never overwrites one; `gbrain storage-update <id> --expected-revision <rev>` patches a card bound to the revision it read, keeps the page's access labels and records restores newest first; `gbrain storage-find "<words>"` ranks the exact id, then an exact alias, then the cards holding most of the words, and a miss lists the closest cards; `gbrain storage-get` and `gbrain storage-list` read. On a brain with a Git checkout, `gbrain storage init` hosts the shelf in its own repository first (preview, then `--expect <plan_hash>`; it needs managed persistence), so each card is a commit before it is indexed. A repo-less brain gets a database-only shelf on the first register.-Cards never carry credentials or commands. A card is refused when any field looks like a secret, an environment assignment, a credential variable, a URL with user info or a presigned URL, or when any key is named `command`, `cmd`, `script`, `exec`, `shell` or `instructions`; object keys are checked too, a value under a key named for a credential (`password`, `application_key`, `secret_access_key`) refuses while a key naming where the credential lives (`token_env`, `password_file`) passes, and the refusal never echoes the value. A first shelf creation that stopped part way is finished by the next register or `storage init`, and an archived shelf refuses until it is restored. A restore is `{ skill, args }` with flat scalar arguments. gbrain stamps `registered_by` and `origin` itself, and holds no store credentials.+Cards never carry credentials or commands. A card is refused when any field looks like a secret, an environment assignment, a credential variable, a URL with user info or a presigned URL, or when any key is named `command`, `cmd`, `script`, `exec`, `shell` or `instructions`; object keys are checked too, a value under a key whose last word names a credential (`password`, `restic_password`, `application_key`, `access_token`) refuses while any other key passes (`token_env`, `password_file`, `token_count`, `api_key_last4`), and the refusal never echoes the value. A first shelf creation that stopped part way is finished by the next register or `storage init`, and an archived shelf refuses until it is restored. A restore is `{ skill, args }` with flat scalar arguments. gbrain stamps `registered_by` and `origin` itself, and holds no store credentials.The shelf is written only through the storage verbs. `put_page`, `remember` and every other page writer are refused `storage_source_managed` there, a request parameter cannot open the door, register and update run only from the trusted local CLI, and a dream cycle or enrichment run scoped to the shelf skips it. The source is not federated, so a remote connection reads cards only when its grant names `storage`.-**Agents find the shelf, and older archive records move onto it.** The brain-first lookup table every harness reads gains one row, "Find something archived to B2, Supabase or S3" (`gbrain storage-find` / `storage_find`), and a new `storage-manifest` skill says when to register, to find before restoring, to restore through the agent's own restore skill with the card's `restore.skill` and `restore.args`, and to record the restore with `gbrain storage-update`. `gbrain storage adopt` moves interim records (`stored-object` pages under `storage-manifest/` holding their record as JSON under `## Record`) onto the shelf: the preview shows each mapped card and every field a card refuses with its reason; `--apply --expect <plan_hash>` registers each card with `origin: adopt` and tags the original `storage-manifest-adopted`, deleting nothing, and refuses until each refused field is named with `--drop-field` (a credential-shaped value always refuses); `--finalize` previews, then soft-deletes the adopted originals only when every card matches its original (a restore recorded since is not a difference); the preview names any other section an original holds and flags one longer than the 64 KiB it reads, and an original edited after the preview, even just before its delete, refuses. `gbrain storage export --jsonl` and `gbrain storage import --jsonl <file>` (preview per line, then `--apply --expect`) rebuild a shelf from a file, registering missing cards with `origin: import` and never overwriting a differing card; export leaves out and names a stored card that fails the card rules. Applies are idempotent and report `side_effects` and, when a run stops part way, the command that finishes it. The new doctor check `storage_manifest` warns about cards that fail the card schema, interim records not adopted yet and a hosted shelf whose directory is gone. On a brain with a Git checkout, adopt and import wait for `gbrain storage init`.+**Agents find the shelf, and older archive records move onto it.** The brain-first lookup table every harness reads gains one row, "Find something archived to B2, Supabase or S3" (`gbrain storage-find` / `storage_find`), and a new `storage-manifest` skill says when to register, to find before restoring, to restore through the agent's own restore skill with the card's `restore.skill` and `restore.args`, and to record the restore with `gbrain storage-update`. `gbrain storage adopt` moves interim records (`stored-object` pages under `storage-manifest/` holding their record as JSON under `## Record`) onto the shelf: the preview shows each mapped card and every field a card refuses with its reason; `--apply --expect <plan_hash>` registers each card with `origin: adopt` and tags the original `storage-manifest-adopted`, deleting nothing; each card takes its original's access labels (`visibility`, `access`, `confidentiality`, `readable_by`, with `readable_by` narrowed to the readers of every folder rule over the original), so a card is never read wider than its original, and an original whose readers and folder rules share no reader, or whose rules are still being applied, is blocked; and refuses until each refused field is named with `--drop-field` (a credential-shaped value always refuses); `--finalize` previews, then soft-deletes the adopted originals only when every card matches its original (a restore recorded since is not a difference); the preview names any other section an original holds and flags one longer than the 64 KiB it reads, and an original edited after the preview, even just before its delete, refuses. `gbrain storage export --jsonl` and `gbrain storage import --jsonl <file>` (preview per line, then `--apply --expect`) rebuild a shelf from a file, registering missing cards with `origin: import` and never overwriting a differing card; each line carries its card's access labels under `labels` and import restores them (a card on the shelf with other labels is a conflict); export leaves out and names a stored card that fails the card rules. Applies are idempotent and report `side_effects` and, when a run stops part way, the command that finishes it; a write gbrain accepted that has not committed yet is listed under `pending` with its receipt and the `gbrain write-request <id>` poll (exit 10), never reported as no change. The new doctor check `storage_manifest` warns about cards that fail the card schema, interim records not adopted yet and a hosted shelf whose directory is gone. On a brain with a Git checkout, adopt and import wait for `gbrain storage init`.### Itemized changes- `src/core/storage-manifest/adopt.ts`, `src/core/storage-manifest/jsonl.ts`, `src/commands/storage.ts`: `gbrain storage adopt` (preview, apply, finalize) and `gbrain storage export|import --jsonl`, each preview-first and bound to its plan_hash (a stale one refuses `preview_changed`). `schema.ts` exports the credential-shape rules, the command-key test and the card keys adopt maps onto, plus `cardContent`/`cardDigest` for comparing copies; `source.ts` gains `shelfWritable` (the refusal a storage write would get, without creating anything); `find.ts` gains `readShelfCards`.- `src/commands/doctor/checks/storage-manifest.ts`, `src/commands/doctor/registry.ts`, `src/core/doctor-categories.ts`: the read-only `storage_manifest` check (brain category), `not_applicable` with no shelf and no interim record. Doctor goldens regenerated for the new check.-- `src/core/error-registry.ts`, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`: `storage_adopt_field_refused` (consent: the user agrees to leave refused fields out) and `storage_adopt_unverified`; `storage_record_invalid` gains reasons `no_record` and `duplicate_id`.+- `src/core/error-registry.ts`, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`: `storage_adopt_field_refused` (consent: the user agrees to leave refused fields out) and `storage_adopt_unverified`; `storage_record_invalid` gains reasons `no_record`, `duplicate_id`, `access_unpreservable`, `access_pending` and `labels`.- `skills/conventions/brain-first.md` (and its generated copies in `AGENTS.md`, `INSTALL_FOR_AGENTS.md`, `skills/RESOLVER.md`, the bootstrap template and the plugin trees), `skills/storage-manifest/SKILL.md`, `skills/RESOLVER.md`, `skills/manifest.json`, `openclaw.plugin.json`: the lookup row and the skill.- `docs/guides/storage-manifest.md`: the guide (verbs, repo-less and Git-checkout brains, adopt, export and import, doctor, privacy).- Tests: `test/storage-manifest-adopt.test.ts`, `test/storage-manifest-jsonl.test.ts` and `test/doctor-storage-manifest.test.ts` (PGLite, repo-less) and `test/storage-manifest-adopt-hosted.test.ts` (managed PGLite with a Git checkout).@@ -23,6 +23,6 @@ The shelf is written only through the storage verbs. `put_page`, `remember` and- Goldens: `test/fixtures/goldens/cli/tools-json.json` gains the three read tools; `test/fixtures/goldens/exports/types.json` gains the new error codes in the `canonical` union and lists `ThinkSynthesisStatus` in a new order (the new schema module declares the `not_json` literal first; the type is unchanged).- Tests: `test/storage-manifest.test.ts` (PGLite, repo-less) and `test/storage-manifest-hosted.test.ts` (managed PGLite with a Git checkout).- `src/core/schema-pack/base/gbrain-base-v2.yaml`: version 1.6.0 declares `stored-object`, so cards never read as an undeclared type and `unify-types` leaves them alone (`test/schema-pack-stored-object-type.test.ts`; doctor goldens regenerated for the new pack identity).-- `src/core/engine-sql/sources.ts`: the local-path source list (autopilot's fan-out, `cycle_freshness`) leaves out the hosted shelf, whose cycle is always skipped. A register that loses a same-id race is refused `storage_card_exists`.+- `src/core/engine-sql/sources.ts`: the local-path source list (autopilot's fan-out, `cycle_freshness`) leaves out the hosted shelf, whose cycle is always skipped. A register that loses a same-id race is refused `storage_card_exists`. `storage-update` rewrites a card whose stamped fields were removed by hand instead of failing.<!-- behavior-change: storage_shelf_stored_object_type_8894 -->diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex 8d14a36ac3..56755ed734 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -5464,9 +5464,9 @@ More: [docs/guides/write-refusals.md#storage-manifest-refusals](../../docs/guide| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| A storage card failed validation (a missing or malformed field, a command field, or a field gbrain stamps itself); nothing was written. | Every card must say where the item is, how big it is and how to restore it, and carries no commands, so a card is complete and safe to act on when it is found months later. | Fix the field the message names and send the card again; gbrain storage-register --help lists the card fields. | agent | `repeat the read that failed` | 1 | no |+| A storage card failed validation (a missing or malformed field, a command field, or a field gbrain stamps itself); nothing was written. | Every card must say where the item is, how big it is and how to restore it, and carries no commands, so a card is complete and safe to act on when it is found months later. | Fix the field the message names and send the card again; gbrain storage-register --help lists the card fields. For an adopt blocked on access_pending or access_unpreservable, settle the access rules over the interim page (gbrain access status) or give it readers its folder rules also list, then preview again. | agent | `repeat the read that failed` | 1 | no |-Reasons: `schema`, `command_field`, `stamped_field`, `not_json`, `not_object`, `too_large`, `id_change`, `no_record`, `duplicate_id`.+Reasons: `schema`, `command_field`, `stamped_field`, `not_json`, `not_object`, `too_large`, `id_change`, `no_record`, `duplicate_id`, `access_unpreservable`, `access_pending`, `labels`.More: [docs/guides/write-refusals.md#storage-manifest-refusals](../../docs/guides/write-refusals.md#storage-manifest-refusals)diff --git a/docs/guides/storage-manifest.md b/docs/guides/storage-manifest.mdindex 78747370c3..d92626607e 100644--- a/docs/guides/storage-manifest.md+++ b/docs/guides/storage-manifest.md@@ -57,25 +57,35 @@ Agents that archived before the shelf existed often kept interim pages: `type: safter the user agrees to leave those fields out. It registers each card with `origin: adopt` through the samepath `storage-register` uses, then tags the original `storage-manifest-adopted`. It deletes nothing. Acredential-shaped value (a secret, an environment assignment, a credential variable, a URL password, a signed URL,- a value under a credential-named key such as `restore.args.password`) always refuses, whatever is dropped: fix the source page first.+ a value under a key whose last word names a credential, such as `restore.args.password` or `restic_password`) always+ refuses, whatever is dropped: fix the source page first. Each card takes its original's access labels (`visibility`,+ `access`, `confidentiality`, `readable_by`), with `readable_by` narrowed to the readers that every folder rule over+ the original also lists (readers match literally, so a member is not matched to its groups), so no card is read+ wider than its original. An original whose readers and folder rules share no reader (`access_unpreservable`), or+ whose rules are still being applied (`access_pending`), is blocked until you restrict the card some other way.3. `gbrain storage adopt --finalize` previews soft-deleting the tagged originals; its `--expect <plan_hash>` run deletesthem (restorable for 72 hours with `gbrain restore`), only when every adopted card is on the shelf and matches its- original's mapped card. A restore recorded with `storage-update` since adoption is not a difference. The preview+ original's mapped card and carries the access labels the original needs. A restore recorded with `storage-update` since adoption is not a difference. The previewnames any `##` section in an original's body other than `## Record` (notes the card does not carry, which go withthe page; a timeline section gbrain stores apart from the body is not listed) and flags an original longer than the 64 KiB it reads, so you read the rest first. The plan_hash binds eachoriginal's whole content, checked again just before each delete, so an original edited after the preview refuses`preview_changed`.A rerun skips cards already on the shelf that match and originals already tagged. A card on the shelf that differs is-a conflict: it is reported and never overwritten, and its original stays untagged. Every apply reports-`side_effects: none | partial | complete`; a run that stopped part way prints the command that finishes it.+a conflict (a card without the original's access labels too): it is reported and never overwritten, and its original+stays untagged. Every apply reports `side_effects: none | partial | complete`; a run that stopped part way prints the+command that finishes it. A write gbrain accepted that has not committed yet is listed under `pending` with its+receipt: poll it with `gbrain write-request <id>` (the command exits 10) and rerun once it has committed, never submit+it again.## Export and import (JSONL)`gbrain storage export --jsonl [--out <new file>]` writes every card as one JSON line, sorted by id, with the fields-gbrain stamps (`registered_by`, `origin`). `gbrain storage import --jsonl <file>` previews each line as+gbrain stamps (`registered_by`, `origin`) and, under `labels`, the card page's access labels. `gbrain storage import+--jsonl <file>` previews each line as`create`, `identical`, `conflict` or `invalid` with a `plan_hash`; `--apply --expect <plan_hash>` registers the-missing cards with `origin: import` (gbrain stamps both fields again), never overwrites a differing card, and skips+missing cards with `origin: import` (gbrain stamps both fields again) and the line's labels, so a restricted card stays+restricted, never overwrites a differing card (other labels count as a difference), and skipsinvalid lines without showing their values. Export leaves out a stored card that fails the card rules (a hand-edited card file, say) and names it. Exportfrom one brain and import into an empty one reproduces every valid card.diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 4551892f5d..4510f041f7 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -1122,7 +1122,7 @@ An adopt or import apply whose plan changed since its preview refuses `preview_c| Code | What happened | What to do ||---|---|---|| `storage_source_managed` | `put_page`, `remember` or another page writer targeted the shelf. | Register with `gbrain storage-register`, change with `gbrain storage-update`. |-| `storage_record_invalid` | A card field is missing or malformed, names a command (`command`, `script`, `exec` and similar), or is one gbrain sets itself (`registered_by`, `origin`). `storage_get` refuses it too for a stored card that breaks these rules, such as a hand-edited card file. | Fix the field the message names. Restore runs through `restore.skill` and `restore.args`. For a stored card, `gbrain doctor --only storage_manifest` lists the invalid cards, and the refusal carries the card's revision for `gbrain storage-update <id> --expected-revision <rev> --patch '<json>'`. |+| `storage_record_invalid` | A card field is missing or malformed, names a command (`command`, `script`, `exec` and similar), or is one gbrain sets itself (`registered_by`, `origin`). On adopt, reason `access_unpreservable` means an interim page's `readable_by` and the folder rules over it share no reader, and `access_pending` that a rule over it is still being applied, so no card could carry the same restriction; on import, `labels` means a line's `labels` holds something other than access labels. `storage_get` refuses it too for a stored card that breaks these rules, such as a hand-edited card file. | Fix the field the message names. Restore runs through `restore.skill` and `restore.args`. For `access_pending`, finish the rule change (`gbrain access status`) and preview again; for `access_unpreservable`, give the page readers its folder rules also list. For a stored card, `gbrain doctor --only storage_manifest` lists the invalid cards, and the refusal carries the card's revision for `gbrain storage-update <id> --expected-revision <rev> --patch '<json>'`. || `storage_record_secret` | A field looks like a secret or has a credential shape: an environment assignment, a credential-named variable, a URL password or a signed-URL parameter. The value is not echoed. | Remove it; the restore skill reads credentials from its own environment. || `storage_card_exists` | A card with that id is already registered. | Update it with the revision `storage_get` returns, or use another id. || `storage_revision_conflict` | The card changed since its revision was read; `detail` holds the current revision. | Read it again, apply the change, update once. |diff --git a/src/commands/storage.ts b/src/commands/storage.tsindex 29880c24be..719d32ae91 100644--- a/src/commands/storage.ts+++ b/src/commands/storage.ts@@ -13,6 +13,7 @@ import { nativeFileTarget } from '../core/persistence/native-file-target.ts';import { opError, OperationError, type OperationContext } from '../core/ops/contract.ts';import { readFix } from '../core/ops/op-fix.ts';import { setCliExitVerdict } from '../core/cli-force-exit.ts';+import { PENDING_WRITE_EXIT_CODE } from '../core/exit-codes.ts';/*** Distinct nominal types for the two tier-keyed numeric maps. Both shapes@@ -511,11 +512,19 @@ async function shelfRefusal(error: unknown, json: boolean): Promise<void> {const argvText = (argv: string[]) => argv.map(shellQuote).join(' ');-function printOutcome(result: { side_effects: string; failure?: { item: string; code: string; message: string }; rerun?: string[] }): void {+type ShelfOutcome = { side_effects: string; failure?: { item: string; code: string; message: string }; pending?: Array<{ item: string; request_id: string; poll: string[] }>; rerun?: string[] };++/** Exit 1 on a failure or a conflict, else exit 10 while an accepted write is still pending (poll it; never resubmit). */+function outcomeExit(result: ShelfOutcome, refused: boolean): void {+ if (result.failure || refused) setCliExitVerdict(1);+ else if (result.pending?.length) setCliExitVerdict(PENDING_WRITE_EXIT_CODE);+}++function printOutcome(result: ShelfOutcome): void {console.log(`Side effects: ${result.side_effects}.`);- if (!result.failure) return;- console.log(`Stopped at ${result.failure.item} (${result.failure.code}): ${result.failure.message}`);- if (result.rerun) console.log(`Rerun after fixing it: ${argvText(result.rerun)}`);+ for (const held of result.pending ?? []) console.log(`Accepted, still being written: ${held.item} (poll it with ${argvText(held.poll)}; do not submit it again).`);+ if (result.failure) console.log(`Stopped at ${result.failure.item} (${result.failure.code}): ${result.failure.message}`);+ if (result.rerun) console.log(`${result.failure ? 'Rerun after fixing it' : 'Rerun once it has committed, to finish'}: ${argvText(result.rerun)}`);}async function runStorageAdopt(engine: BrainEngine, args: string[]): Promise<void> {@@ -539,7 +548,7 @@ async function runStorageAdopt(engine: BrainEngine, args: string[]): Promise<voiconst result = finalize? expect ? await adopt.applyFinalize(ctx, { ...opts, expect }) : await adopt.previewFinalize(ctx, opts): expect ? await adopt.applyAdopt(ctx, { ...opts, expect, dropFields }) : await adopt.previewAdopt(ctx, opts);- if (!result.dry_run && (result.failure || (result.action === 'adopt' && result.conflicts.length))) setCliExitVerdict(1);+ if (!result.dry_run) outcomeExit(result, result.action === 'adopt' && result.conflicts.length > 0);if (json) { console.log(JSON.stringify(result)); return; }if (result.dry_run) {for (const line of result.summary) console.log(line);@@ -593,7 +602,7 @@ async function runStorageJsonl(engine: BrainEngine, subcommand: 'export' | 'impo{ fix: readFix('Prints the import plan and its plan_hash; writes nothing.', { argv: ['gbrain', 'storage', 'import', '--jsonl', file, '--json'] }) });if (expect !== undefined) {const result = await jsonl.applyImport(await shelfCliContext(engine, 'default'), file, text, expect);- if (result.failure || result.conflicts.length || result.invalid.length) setCliExitVerdict(1);+ outcomeExit(result, result.conflicts.length > 0 || result.invalid.length > 0);if (json) { console.log(JSON.stringify(result)); return; }console.log(`Created ${result.created.length} card(s); ${result.identical.length} already on the shelf.`);if (result.conflicts.length) console.log(`Conflicts (a different card with that id is on the shelf; nothing was overwritten): ${result.conflicts.join(', ')}`);diff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex b18ae934bf..0bb54e49e0 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -843,7 +843,7 @@ export const CODES = {storage_adopt_unverified: { class: 'caller', summary: "An adopted interim original has no card on the storage shelf that matches its record, so finalize deleted nothing.", why: 'Finalize soft-deletes an original only when its card holds everything the record mapped to, so no archive record is lost.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Compare the original with its card (gbrain storage-get), fix the card or the record, then preview gbrain storage adopt --finalize again.' },storage_card_exists: { class: 'caller', summary: "A storage card with this id is already on the storage shelf; nothing was written.", why: 'Registering never overwrites a card, so two archives cannot silently share one id; a change to an existing card names the revision it read.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Change the card with gbrain storage-update <id> --expected-revision <revision from storage_get>, or register the new item under another id.' },storage_manifest_missing: { class: 'consent', actor: 'user', summary: "This brain has no storage shelf to write to (or the storage source ID belongs to another source); nothing was written.", why: 'On a brain with a Git checkout the shelf is a hosted git-canonical repository, created only after the user previews it, and hosting needs managed persistence. A repo-less brain gets its database-only shelf on the first register.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Preview gbrain storage init --dir <absolute path> --dry-run and apply the command it prints after the user agrees.' },- storage_record_invalid: { class: 'caller', summary: "A storage card failed validation (a missing or malformed field, a command field, or a field gbrain stamps itself); nothing was written.", why: 'Every card must say where the item is, how big it is and how to restore it, and carries no commands, so a card is complete and safe to act on when it is found months later.', reasons: ['schema', 'command_field', 'stamped_field', 'not_json', 'not_object', 'too_large', 'id_change', 'no_record', 'duplicate_id'], docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Fix the field the message names and send the card again; gbrain storage-register --help lists the card fields.' },+ storage_record_invalid: { class: 'caller', summary: "A storage card failed validation (a missing or malformed field, a command field, or a field gbrain stamps itself); nothing was written.", why: 'Every card must say where the item is, how big it is and how to restore it, and carries no commands, so a card is complete and safe to act on when it is found months later.', reasons: ['schema', 'command_field', 'stamped_field', 'not_json', 'not_object', 'too_large', 'id_change', 'no_record', 'duplicate_id', 'access_unpreservable', 'access_pending', 'labels'], docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Fix the field the message names and send the card again; gbrain storage-register --help lists the card fields. For an adopt blocked on access_pending or access_unpreservable, settle the access rules over the interim page (gbrain access status) or give it readers its folder rules also list, then preview again.' },storage_record_secret: { class: 'caller', summary: "A storage card field looks like a secret or has a credential shape; nothing was written and the value is not echoed.", why: 'Cards are read by agents and copied into the stores; credentials stay in the environment of the skill that restores an item.', reasons: ['secret_scanner', 'env_assignment', 'credential_variable', 'url_userinfo', 'presigned_url', 'credential_key'], docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Remove the secret, environment assignment, credential variable, URL password, signed-URL parameter or credential-named field from the card.' },storage_revision_conflict: { class: 'caller', summary: "A storage card changed since the revision the update named was read; nothing was written.", why: 'Updates bind to the revision they read, so two writers never silently overwrite each other; the refusal carries the current revision in detail.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Read the card again with storage_get, apply the change to what it says now, and update once with its revision.' },storage_source_managed: { class: 'caller', summary: "A page write targeted the storage shelf outside the storage verbs; nothing was written.", why: 'The storage source holds only validated cards, so put_page, remember and the other page writers are refused there.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Register cards with gbrain storage-register and change them with gbrain storage-update.' },diff --git a/src/core/ops/storage-manifest.ts b/src/core/ops/storage-manifest.tsindex c57a7cbad4..21cb872db9 100644--- a/src/core/ops/storage-manifest.ts+++ b/src/core/ops/storage-manifest.ts@@ -111,13 +111,16 @@ function cardExists(id: string, revision: string): OperationError {{ fix: getFix(id), detail: revision });}-/** Register a new card. Exported for `storage adopt` and `storage import` (origin adopt / import). */-export async function registerCard(ctx: OperationContext, input: CardInput, origin: (typeof ORIGINS)[number], requestId?: string): Promise<Record<string, unknown>> {+/**+ * Register a new card. Exported for `storage adopt` and `storage import` (origin adopt / import), which pass the access+ * labels (visibility, readable_by) the card page must carry so it is read no wider than its original.+ */+export async function registerCard(ctx: OperationContext, input: CardInput, origin: (typeof ORIGINS)[number], requestId?: string, labels?: Record<string, unknown>): Promise<Record<string, unknown>> {await ensureStorageSource(ctx.engine);const existing = await ctx.engine.readPageSnapshot(input.id, { sourceId: STORAGE_SOURCE_ID });if (existing) throw cardExists(input.id, existing.revision);const card: Card = { ...input, registered_by: await principalLabel(ctx), origin };- const response = await writeCard(ctx, card, { requestId });+ const response = await writeCard(ctx, card, { requestId, stored: labels });return { id: card.id, source_id: STORAGE_SOURCE_ID, origin, write: response };}@@ -265,7 +268,7 @@ const storage_update: Operation = {const patch = parsePatch(p.patch);const { registered_by, origin, ...current } = found.card;const input = parseCardInput({ ...current, ...patch, id });- let card: Card = { ...input, registered_by, origin };+ let card = { ...input, ...(registered_by !== undefined ? { registered_by } : {}), ...(origin !== undefined ? { origin } : {}) } as Card;if (typeof p.restored_to === 'string' && p.restored_to.trim()) {card = withRestore(card, { at: new Date().toISOString(), to: p.restored_to.trim(), by: await principalLabel(ctx), verified: p.restored_verified === true });const { registered_by: _stampedBy, origin: _stampedOrigin, ...unstamped } = card;diff --git a/src/core/storage-manifest/adopt.ts b/src/core/storage-manifest/adopt.tsindex 7360190d95..b8a6f48848 100644--- a/src/core/storage-manifest/adopt.ts+++ b/src/core/storage-manifest/adopt.ts@@ -29,8 +29,12 @@ import { registerCard } from '../ops/storage-manifest.ts';import { boundedBodyPrefixSql } from '../page-window/bounded-lead.ts';import { BOUNDED_PREFIX_BYTES } from '../page-window/document.ts';import { submitPageMutation } from '../persistence/page-mutations.ts';+import { acceptedPendingReceipt } from '../persistence/accepted-pending.ts';+import { ruleCoversSql } from '../access/folder-rules.ts';+import { hasReaders, normalizeReaders } from '../access/principals.ts';+import { canonicalJson } from '../remediation-step.ts';import { readShelfCards } from './find.ts';-import { CARD_KEYS, cardContent, cardDigest, credentialShape, holdsCredential, isCommandKey, parseCardInput, STAMPED_KEYS, STORED_OBJECT_TYPE, type CardInput } from './schema.ts';+import { accessLabels, CARD_KEYS, cardContent, cardDigest, credentialShape, holdsCredential, isCommandKey, parseCardInput, STAMPED_KEYS, STORED_OBJECT_TYPE, type AccessLabels, type CardInput } from './schema.ts';import { shelfWritable, STORAGE_SOURCE_ID } from './source.ts';export const ADOPT_DEFAULT_PREFIX = 'storage-manifest/';@@ -41,7 +45,7 @@ export type RefusedReason = 'command_field' | 'stamped_field' | 'unknown_field';export interface RefusedField { field: string; reason: RefusedReason }export interface EntryProblem { code: 'storage_record_secret' | 'storage_record_invalid'; reason: string; message: string }export type AdoptState = 'register' | 'retag' | 'adopted' | 'conflict' | 'blocked';-export interface AdoptEntry { slug: string; card_id: string | null; state: AdoptState; card: CardInput | null; refused: RefusedField[]; problem: EntryProblem | null }+export interface AdoptEntry { slug: string; card_id: string | null; state: AdoptState; card: CardInput | null; labels: AccessLabels | null; refused: RefusedField[]; problem: EntryProblem | null }export interface AdoptOptions { sourceId: string; prefix: string }export interface AdoptPlan {action: 'adopt'; dry_run: true; source_id: string; prefix: string; shelf: 'exists' | 'creates';@@ -49,22 +53,38 @@ export interface AdoptPlan {}export type SideEffects = 'none' | 'partial' | 'complete';export interface StepFailure { item: string; code: string; message: string }+/** A write gbrain accepted that has not committed yet: poll its receipt; submitting it again is never needed. */+export interface PendingWrite { item: string; request_id: string; state: string; poll: string[] }++/** The pending write an error reports (an accepted, non-terminal receipt), or null for any other error. */+export function pendingWrite(item: string, error: unknown): PendingWrite | null {+ const receipt = acceptedPendingReceipt(error);+ return receipt ? { item, request_id: receipt.request_id, state: receipt.state, poll: ['gbrain', 'write-request', receipt.request_id] } : null;+}++/** What a run changed: an accepted pending write counts as a change, and as unfinished. */+export function sideEffects(wrote: boolean, unfinished: boolean): SideEffects {+ return !wrote ? 'none' : unfinished ? 'partial' : 'complete';+}export interface AdoptApplyResult {action: 'adopt'; dry_run: false; source_id: string; prefix: string; plan_hash: string; side_effects: SideEffects;results: Array<{ slug: string; card_id: string | null; outcome: 'registered' | 'retagged' | 'skipped' | 'conflict' }>;- conflicts: string[]; failure?: StepFailure; rerun?: string[]; finalize?: string[];+ conflicts: string[]; failure?: StepFailure; pending?: PendingWrite[]; rerun?: string[]; finalize?: string[];}export interface FinalizeEntry { slug: string; card_id: string | null; state: 'delete' | 'deleted' | 'unverified' | 'not_adopted'; other_sections?: string[]; unread_tail?: boolean; content?: string }export interface FinalizePlan { action: 'finalize'; dry_run: true; source_id: string; prefix: string; entries: FinalizeEntry[]; plan_hash: string; summary: string[]; apply?: string[] }export interface FinalizeApplyResult {action: 'finalize'; dry_run: false; source_id: string; prefix: string; plan_hash: string; side_effects: SideEffects;- deleted: string[]; skipped: string[]; failure?: StepFailure; rerun?: string[];+ deleted: string[]; skipped: string[]; failure?: StepFailure; pending?: PendingWrite[]; rerun?: string[];}-interface InterimRow { slug: string; title: string; frontmatter: unknown; body: string | null; content_hash: string | null; long_body: boolean; tagged: boolean; deleted: boolean }+interface InterimAccess { folder_rule_ids?: string[] | null; folder_readers?: unknown; access_pending?: boolean }+interface InterimRow extends InterimAccess { slug: string; title: string; frontmatter: unknown; body: string | null; content_hash: string | null; long_body: boolean; tagged: boolean; deleted: boolean }const INTERIM_COLUMNS = `p.slug, p.title, p.frontmatter, ${boundedBodyPrefixSql('p')} AS body, p.content_hash,- coalesce(p.body_bytes, 0) > ${BOUNDED_PREFIX_BYTES} AS long_body`;+ coalesce(p.body_bytes, 0) > ${BOUNDED_PREFIX_BYTES} AS long_body, p.folder_rule_ids,+ (SELECT jsonb_agg(r.readable_by) FROM access_rules r WHERE r.source_id || ':' || r.rule_id = ANY(p.folder_rule_ids)) AS folder_readers,+ EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND ${ruleCoversSql('p.slug', 'pc.prefix')}) AS access_pending`;/** What a finalize plan binds for one original: its title, frontmatter and whole content (content_hash), so any edit after the preview changes it. */const originalDigest = (row: Pick<InterimRow, 'title' | 'frontmatter' | 'body' | 'content_hash'>) => sha256(JSON.stringify([row.title, row.frontmatter, row.content_hash ?? row.body]));@@ -147,27 +167,52 @@ function invalidRecord(reason: string, message: string): EntryProblem {return { code: 'storage_record_invalid', reason, message };}+const sameLabels = (a: AccessLabels | null | undefined, b: AccessLabels | null | undefined) => !!a && !!b && canonicalJson(a) === canonicalJson(b);++/**+ * The access labels the card takes so it is read no wider than its original: the original's own labels, with+ * readable_by narrowed to the readers that it and every folder rule over the original all list. Readers match literally+ * (a member is not matched to its groups), which can only narrow. A problem instead when they share no reader, or a rule+ * over the original is still being applied.+ */+function originalLabels(row: InterimAccess, frontmatter: Record<string, unknown>): { labels: AccessLabels; problem: null } | { labels: null; problem: EntryProblem } {+ const labels = accessLabels(frontmatter);+ const raw = typeof row.folder_readers === 'string' ? JSON.parse(row.folder_readers) as unknown : row.folder_readers;+ const rules = Array.isArray(raw) ? raw as string[][] : [];+ if (row.access_pending || rules.length !== (row.folder_rule_ids?.length ?? 0)) {+ return { labels: null, problem: invalidRecord('access_pending', 'an access rule over the page is still being applied, so who may read it is not settled (gbrain access status names the rule)') };+ }+ if (!rules.length) return { labels, problem: null };+ const lists = [...(hasReaders(frontmatter) ? [normalizeReaders(frontmatter)!] : []), ...rules];+ const readers = lists.reduce((common, list) => common.filter((reader) => list.includes(reader)));+ if (!readers.length) return { labels: null, problem: invalidRecord('access_unpreservable', 'its readable_by and the folder rules over it name no reader in common, so a card cannot carry the same restriction') };+ return { labels: { ...labels, readable_by: [...new Set(readers)].sort() }, problem: null };+}+/** Map one interim page onto a card. Pure: reads nothing beyond the row. */-export function mapInterimPage(row: Pick<InterimRow, 'slug' | 'title' | 'frontmatter' | 'body'>, prefix: string): Pick<AdoptEntry, 'card' | 'refused' | 'problem'> {+export function mapInterimPage(row: Pick<InterimRow, 'slug' | 'title' | 'frontmatter' | 'body'> & InterimAccess, prefix: string): Pick<AdoptEntry, 'card' | 'labels' | 'refused' | 'problem'> {const text = recordJson(row.body ?? '');- if (text === null) return { card: null, refused: [], problem: invalidRecord('no_record', 'the page has no json block under a ## Record heading in its first 64 KiB') };+ if (text === null) return { card: null, labels: null, refused: [], problem: invalidRecord('no_record', 'the page has no json block under a ## Record heading in its first 64 KiB') };let record: unknown;- try { record = JSON.parse(text); } catch { return { card: null, refused: [], problem: invalidRecord('not_json', 'the ## Record block is not valid JSON') }; }- if (!record || typeof record !== 'object' || Array.isArray(record)) return { card: null, refused: [], problem: invalidRecord('not_object', 'the ## Record block is not a JSON object') };+ try { record = JSON.parse(text); } catch { return { card: null, labels: null, refused: [], problem: invalidRecord('not_json', 'the ## Record block is not valid JSON') }; }+ if (!record || typeof record !== 'object' || Array.isArray(record)) return { card: null, labels: null, refused: [], problem: invalidRecord('not_object', 'the ## Record block is not a JSON object') };const refused: RefusedField[] = [];const stripped = strip(record, '', refused) as Record<string, unknown>;const secret = credentialAt(record, '');- if (secret) return { card: null, refused: [], problem: { code: 'storage_record_secret', reason: secret.shape, message: `field ${secret.field} has a credential shape (${secret.shape}); the value is not shown` } };+ if (secret) return { card: null, labels: null, refused: [], problem: { code: 'storage_record_secret', reason: secret.shape, message: `field ${secret.field} has a credential shape (${secret.shape}); the value is not shown` } };const frontmatter = (typeof row.frontmatter === 'string' ? JSON.parse(row.frontmatter) : row.frontmatter ?? {}) as Record<string, unknown>;const aliases = Array.isArray(frontmatter.aliases) ? frontmatter.aliases.filter((a): a is string => typeof a === 'string') : [];const mapped = { ...stripped, id: stripped.id ?? row.slug.slice(prefix.length).replace(/\//g, '-'), title: stripped.title ?? row.title, aliases: stripped.aliases ?? aliases };+ let card: CardInput;try {- return { card: parseCardInput(mapped), refused, problem: null };+ card = parseCardInput(mapped);} catch (error) {if (!(error instanceof OperationError)) throw error;- return { card: null, refused, problem: { code: error.code === 'storage_record_secret' ? 'storage_record_secret' : 'storage_record_invalid',+ return { card: null, labels: null, refused, problem: { code: error.code === 'storage_record_secret' ? 'storage_record_secret' : 'storage_record_invalid',reason: error.reason ?? 'schema', message: error.message.replace(/; nothing was written\.$/, '') } };}+ const access = originalLabels(row, frontmatter);+ return access.problem ? { card: null, labels: null, refused, problem: access.problem } : { card, labels: access.labels, refused, problem: null };}async function listInterim(ctx: OperationContext, opts: AdoptOptions, includeDeleted: boolean): Promise<InterimRow[]> {@@ -191,31 +236,33 @@ async function assertInterimSource(ctx: OperationContext, opts: AdoptOptions): Pexport async function previewAdopt(ctx: OperationContext, opts: AdoptOptions): Promise<AdoptPlan> {await assertInterimSource(ctx, opts);const { creates, exists } = await shelfWritable(ctx.engine);- const shelf = new Map((await readShelfCards(ctx.engine)).map((row) => [row.id, row.card]));+ const shelf = new Map((await readShelfCards(ctx.engine)).map((row) => [row.id, row]));const seen = new Set<string>();const entries: AdoptEntry[] = (await listInterim(ctx, opts, false)).map((row): AdoptEntry => {const mapped = mapInterimPage(row, opts.prefix);const card_id = mapped.card?.id ?? null;- if (card_id && seen.has(card_id)) return { slug: row.slug, card_id, state: 'blocked', card: null, refused: mapped.refused, problem: invalidRecord('duplicate_id', `another interim page maps to card id ${card_id}`) };+ if (card_id && seen.has(card_id)) return { slug: row.slug, card_id, state: 'blocked', card: null, labels: null, refused: mapped.refused, problem: invalidRecord('duplicate_id', `another interim page maps to card id ${card_id}`) };if (card_id) seen.add(card_id);if (!mapped.card) return { slug: row.slug, card_id, state: 'blocked', ...mapped };const onShelf = shelf.get(mapped.card.id);- const existing = onShelf ? cardContent(onShelf) : null;- const state: AdoptState = !shelf.has(mapped.card.id) ? 'register'- : !existing || cardDigest(existing) !== cardDigest(mapped.card) ? 'conflict' : row.tagged ? 'adopted' : 'retag';+ const existing = onShelf?.card ? cardContent(onShelf.card) : null;+ const state: AdoptState = !onShelf ? 'register'+ : !existing || cardDigest(existing) !== cardDigest(mapped.card) || !sameLabels(onShelf.labels, mapped.labels) ? 'conflict' : row.tagged ? 'adopted' : 'retag';return { slug: row.slug, card_id, state, ...mapped };});const counts: Record<AdoptState, number> = { register: 0, retag: 0, adopted: 0, conflict: 0, blocked: 0 };for (const entry of entries) counts[entry.state]++;const drop_fields = [...new Set(entries.filter((e) => e.state !== 'blocked').flatMap((e) => e.refused.map((f) => f.field)))].sort();const plan_hash = sha256(JSON.stringify({ v: 1, kind: 'adopt', source_id: opts.sourceId, prefix: opts.prefix, entries: entries.map((e) => ({- slug: e.slug, card: e.card ? cardDigest(e.card) : null, refused: e.refused.map((f) => `${f.field}:${f.reason}`),+ slug: e.slug, card: e.card ? cardDigest(e.card) : null, labels: e.labels ? canonicalJson(e.labels) : null, refused: e.refused.map((f) => `${f.field}:${f.reason}`),problem: e.problem ? `${e.problem.code}:${e.problem.reason}` : null, conflict: e.state === 'conflict' })) }));const summary = [`${entries.length} interim record(s) under ${opts.prefix} in source ${opts.sourceId}: ${counts.register} to register, ${counts.retag} already on the shelf to tag, ${counts.adopted} already adopted, ${counts.conflict} conflict(s), ${counts.blocked} blocked.`];if (creates && counts.register) summary.push(exists ? `The first card finishes the storage shelf (source ${STORAGE_SOURCE_ID}) that an earlier creation left without its role.`: `The first card creates the storage shelf (source ${STORAGE_SOURCE_ID}, database-only: this brain has no Git checkout, so the database is the record).`);+ const restricted = entries.filter((e) => e.state === 'register' && e.labels && Object.keys(e.labels).length).length;+ if (restricted) summary.push(`${restricted} card(s) take their original's access labels (visibility, readable_by, and the readers of the folder rules over it), so each is read no wider than its original.`);if (drop_fields.length) summary.push(`Fields a card refuses, left out only when each is named with --drop-field: ${drop_fields.join(', ')}. The original pages keep them until finalize.`);- if (counts.conflict) summary.push('A conflict is a card already on the shelf that differs from the record: it is never overwritten, and its original is not tagged.');+ if (counts.conflict) summary.push('A conflict is a card already on the shelf that differs from the record or from the access labels its original needs: it is never overwritten, and its original is not tagged.');if (counts.blocked) summary.push('Blocked pages need their record fixed on the page first; apply refuses until then.');const apply = counts.blocked || !(counts.register + counts.retag) ? undefined: [...adoptArgv(opts), ...drop_fields.flatMap((f) => ['--drop-field', f]), '--apply', '--expect', plan_hash];@@ -256,48 +303,52 @@ export async function applyAdopt(ctx: OperationContext, opts: AdoptOptions & { euser_message: `These fields of the interim storage records cannot go on a card and would be left out: ${missing.join(', ')}. The original pages keep them until you finalize. Leave them out?` } });}const results: AdoptApplyResult['results'] = [];+ const pending: PendingWrite[] = [];let failure: StepFailure | undefined;for (const entry of plan.entries) {if (entry.state === 'conflict' || entry.state === 'adopted') { results.push({ slug: entry.slug, card_id: entry.card_id, outcome: entry.state === 'conflict' ? 'conflict' : 'skipped' }); continue; }try {if (entry.state === 'register') {- await registerCard(ctx, entry.card!, 'adopt');+ await registerCard(ctx, entry.card!, 'adopt', undefined, entry.labels!);results.push({ slug: entry.slug, card_id: entry.card_id, outcome: 'registered' });}await submitPageMutation(ctx, { operation: 'add_tag', params: { slug: entry.slug, tag: ADOPTED_TAG, source_id: opts.sourceId } });results.push({ slug: entry.slug, card_id: entry.card_id, outcome: 'retagged' });} catch (error) {+ const held = pendingWrite(entry.slug, error);+ if (held) { pending.push(held); continue; }if (!(error instanceof OperationError)) throw error;failure = { item: entry.slug, code: error.canonical ?? error.code, message: error.message };break;}}- const wrote = results.some((r) => r.outcome === 'registered' || r.outcome === 'retagged');+ const wrote = pending.length > 0 || results.some((r) => r.outcome === 'registered' || r.outcome === 'retagged');return { action: 'adopt', dry_run: false, source_id: opts.sourceId, prefix: opts.prefix, plan_hash: plan.plan_hash,- side_effects: failure ? (wrote ? 'partial' : 'none') : wrote ? 'complete' : 'none', results,+ side_effects: sideEffects(wrote, !!failure || pending.length > 0), results,conflicts: results.filter((r) => r.outcome === 'conflict').map((r) => r.slug),- ...(failure ? { failure, rerun } : { finalize: [...adoptArgv(opts), '--finalize'] }) };+ ...(failure ? { failure } : {}), ...(pending.length ? { pending } : {}),+ ...(failure || pending.length ? { rerun } : { finalize: [...adoptArgv(opts), '--finalize'] }) };}/** Preview finalize: the tagged originals to soft-delete. Writes nothing. */export async function previewFinalize(ctx: OperationContext, opts: AdoptOptions): Promise<FinalizePlan> {await assertInterimSource(ctx, opts);- const shelf = new Map((await readShelfCards(ctx.engine)).map((row) => [row.id, row.card]));+ const shelf = new Map((await readShelfCards(ctx.engine)).map((row) => [row.id, row]));const entries: FinalizeEntry[] = (await listInterim(ctx, opts, true)).flatMap((row): FinalizeEntry[] => {if (!row.tagged) return row.deleted ? [] : [{ slug: row.slug, card_id: null, state: 'not_adopted' }];- const { card } = mapInterimPage(row, opts.prefix);+ const { card, labels } = mapInterimPage(row, opts.prefix);const content = originalDigest(row);if (row.deleted) return [{ slug: row.slug, card_id: card?.id ?? null, state: 'deleted', content }];const onShelf = card ? shelf.get(card.id) : undefined;- const existing = onShelf ? cardContent(onShelf) : null;- const state = card && existing && matchesAdopted(existing, card) ? 'delete' : 'unverified';+ const existing = onShelf?.card ? cardContent(onShelf.card) : null;+ const state = card && existing && matchesAdopted(existing, card) && sameLabels(onShelf!.labels, labels) ? 'delete' : 'unverified';const other = otherSections(row.body ?? '');return [{ slug: row.slug, card_id: card?.id ?? null, state, ...(other.length ? { other_sections: other } : {}), ...(row.long_body || (row.body?.length ?? 0) >= BOUNDED_PREFIX_BYTES ? { unread_tail: true } : {}), content }];});const count = (state: FinalizeEntry['state']) => entries.filter((e) => e.state === state).length;const plan_hash = sha256(JSON.stringify({ v: 1, kind: 'finalize', source_id: opts.sourceId, prefix: opts.prefix,entries: entries.filter((e) => e.state !== 'not_adopted').map((e) => ({ slug: e.slug, card_id: e.card_id, unverified: e.state === 'unverified', content: e.content })) }));- const summary = [`${count('delete')} adopted original(s) to soft-delete (restorable for 72 hours), ${count('deleted')} already deleted, ${count('unverified')} whose card is missing or differs, ${count('not_adopted')} not adopted yet (kept).`];+ const summary = [`${count('delete')} adopted original(s) to soft-delete (restorable for 72 hours), ${count('deleted')} already deleted, ${count('unverified')} whose card is missing or differs (in content or access labels), ${count('not_adopted')} not adopted yet (kept).`];const dropping = entries.filter((e) => e.state === 'delete' && e.other_sections);if (dropping.length) summary.push(`${dropping.length} original(s) to delete hold sections the card does not carry (${[...new Set(dropping.flatMap((e) => e.other_sections!))].join(', ')}): show the user, and move what matters into the card with gbrain storage-update before finalizing.`);const long = entries.filter((e) => e.state === 'delete' && e.unread_tail).map((e) => e.slug);@@ -317,6 +368,7 @@ export async function applyFinalize(ctx: OperationContext, opts: AdoptOptions &'Compare each original with its card (gbrain storage-get with the card id the finalize preview lists), fix the card or the record, then preview finalize again.', { fix: previewFix(opts, true) });}const deleted: string[] = [];+ const pending: PendingWrite[] = [];let failure: StepFailure | undefined;for (const entry of plan.entries.filter((e) => e.state === 'delete')) {try {@@ -328,13 +380,16 @@ export async function applyFinalize(ctx: OperationContext, opts: AdoptOptions &await submitPageMutation(ctx, { operation: 'delete_page', params: { slug: entry.slug, source_id: opts.sourceId, ...(expected ? { expected_revision: expected } : {}) } });deleted.push(entry.slug);} catch (error) {+ const held = pendingWrite(entry.slug, error);+ if (held) { pending.push(held); continue; }if (!(error instanceof OperationError)) throw error;failure = { item: entry.slug, code: error.canonical ?? error.code, message: error.message };break;}}return { action: 'finalize', dry_run: false, source_id: opts.sourceId, prefix: opts.prefix, plan_hash: plan.plan_hash,- side_effects: failure ? (deleted.length ? 'partial' : 'none') : deleted.length ? 'complete' : 'none', deleted,+ side_effects: sideEffects(deleted.length > 0 || pending.length > 0, !!failure || pending.length > 0), deleted,skipped: plan.entries.filter((e) => e.state !== 'delete').map((e) => e.slug),+ ...(pending.length ? { pending } : {}),...(failure ? { failure, rerun: failure.code === 'preview_changed' ? previewFix(opts, true).argv! : [...adoptArgv(opts), '--finalize', '--expect', plan.plan_hash] } : {}) };}diff --git a/src/core/storage-manifest/find.ts b/src/core/storage-manifest/find.tsindex 0d2369a361..1ad0dfd237 100644--- a/src/core/storage-manifest/find.ts+++ b/src/core/storage-manifest/find.ts@@ -9,7 +9,7 @@ import { pageReadFilter } from '../search/read-policy-sql.ts';import { normalizeAlias } from '../search/alias-normalize.ts';import { boundedBodyPrefixSql } from '../page-window/bounded-lead.ts';import { STORAGE_SOURCE_ID } from './door.ts';-import { cardFromPage, STORED_OBJECT_TYPE, type Card } from './schema.ts';+import { accessLabels, cardFromPage, STORED_OBJECT_TYPE, type AccessLabels, type Card } from './schema.ts';const STOPWORDS = new Set(['a', 'an', 'and', 'the', 'of', 'to', 'in', 'on', 'for', 'from', 'my', 'our', 'it', 'that', 'this', 'back', 'bring', 'restore', 'find', 'where', 'is', 'was', 'what', 'did', 'go', 'with', 'please']);const CLOSEST_FLOOR = 0.1;@@ -80,12 +80,12 @@ export async function listCards(engine: BrainEngine, opts: { limit: number; offsreturn { cards: cards.map((c) => ({ ...c, stores: c.stores ?? [] })), total: Number(n) };}-/** Every live page on the shelf, sorted by id, with the card it stores (null when its page holds none). Trusted local readers only. */-export async function readShelfCards(engine: BrainEngine): Promise<Array<{ id: string; card: Card | null }>> {+/** Every live page on the shelf, sorted by id, with the card it stores (null when its page holds none) and the page's access labels. Trusted local readers only. */+export async function readShelfCards(engine: BrainEngine): Promise<Array<{ id: string; card: Card | null; labels: AccessLabels }>> {const rows = await engine.executeRaw<{ slug: string; title: string; type: string | null; frontmatter: unknown }>('SELECT p.slug, p.title, p.type, p.frontmatter FROM pages p WHERE p.source_id = $1 AND p.deleted_at IS NULL ORDER BY p.slug', [STORAGE_SOURCE_ID]);return rows.map((row) => {const frontmatter = typeof row.frontmatter === 'string' ? JSON.parse(row.frontmatter) as Record<string, unknown> : (row.frontmatter ?? {}) as Record<string, unknown>;- return { id: row.slug, card: row.type === STORED_OBJECT_TYPE ? cardFromPage({ slug: row.slug, title: row.title, frontmatter }) : null };+ return { id: row.slug, card: row.type === STORED_OBJECT_TYPE ? cardFromPage({ slug: row.slug, title: row.title, frontmatter }) : null, labels: accessLabels(frontmatter) };});}diff --git a/src/core/storage-manifest/jsonl.ts b/src/core/storage-manifest/jsonl.tsindex 0e06c023c5..ebbe367c83 100644--- a/src/core/storage-manifest/jsonl.ts+++ b/src/core/storage-manifest/jsonl.ts@@ -1,6 +1,7 @@/*** Storage shelf recovery as JSONL (GST-1943). `gbrain storage export --jsonl` writes one card per line, sorted by id,- * stamped fields (`registered_by`, `origin`) included. `gbrain storage import --jsonl <file>` previews each line+ * stamped fields (`registered_by`, `origin`) included, and the card page's access labels (visibility, readable_by) under+ * `labels`, so an import restricts each card as it was. `gbrain storage import --jsonl <file>` previews each line* (create / identical / conflict / invalid) with a plan_hash, and the apply with that plan_hash registers the missing* cards through registerCard with origin `import`: gbrain stamps registered_by and origin again, so a file cannot set* them. A card on the shelf that differs from its line is a conflict, reported and never overwritten. Idempotent: a@@ -14,18 +15,21 @@ import { readFix } from '../ops/op-fix.ts';import { registerCard } from '../ops/storage-manifest.ts';import type { BrainEngine } from '../engine.ts';import { readShelfCards } from './find.ts';-import { cardContent, cardDigest, parseCardInput, type CardInput } from './schema.ts';+import { cardContent, cardDigest, parseAccessLabels, parseCardInput, type AccessLabels, type CardInput } from './schema.ts';import { shelfWritable } from './source.ts';-import type { SideEffects, StepFailure } from './adopt.ts';+import { pendingWrite, sideEffects, type PendingWrite, type SideEffects, type StepFailure } from './adopt.ts';export type ImportAction = 'create' | 'identical' | 'conflict' | 'invalid';export interface ImportLine { line: number; id: string | null; action: ImportAction; code?: string; reason?: string; message?: string }export interface ImportPlan { action: 'import'; dry_run: true; file: string; lines: ImportLine[]; counts: Record<ImportAction, number>; plan_hash: string; summary: string[]; apply?: string[] }export interface ImportApplyResult {action: 'import'; dry_run: false; file: string; plan_hash: string; side_effects: SideEffects;- created: string[]; identical: string[]; conflicts: string[]; invalid: number[]; failure?: StepFailure; rerun?: string[];+ created: string[]; identical: string[]; conflicts: string[]; invalid: number[]; failure?: StepFailure; pending?: PendingWrite[]; rerun?: string[];}-interface ParsedLine { line: number; id: string | null; card: CardInput | null; code?: string; reason?: string; message?: string }+interface ParsedLine { line: number; id: string | null; card: CardInput | null; labels?: AccessLabels; code?: string; reason?: string; message?: string }+interface LineCard { card: CardInput; labels: AccessLabels }++const labelsDigest = (labels: AccessLabels) => canonicalJson(labels);const sha256 = (text: string) => createHash('sha256').update(text).digest('hex');@@ -36,7 +40,7 @@ const sha256 = (text: string) => createHash('sha256').update(text).digest('hex')export async function exportShelfJsonl(engine: BrainEngine): Promise<{ lines: string[]; skipped: string[]; invalid: string[] }> {const rows = await readShelfCards(engine);const valid = rows.filter((row) => row.card && cardContent(row.card));- return { lines: valid.map((row) => canonicalJson(row.card)), skipped: rows.filter((row) => !row.card).map((row) => row.id),+ return { lines: valid.map((row) => canonicalJson(Object.keys(row.labels).length ? { ...row.card, labels: row.labels } : row.card)), skipped: rows.filter((row) => !row.card).map((row) => row.id),invalid: rows.filter((row) => row.card && !valid.includes(row)).map((row) => row.id) };}@@ -52,38 +56,40 @@ function parseLines(text: string): ParsedLine[] {let value: unknown;try { value = JSON.parse(raw); } catch { return [invalidLine(line, null, 'not_json', 'the line is not valid JSON')]; }if (!value || typeof value !== 'object' || Array.isArray(value)) return [invalidLine(line, null, 'not_object', 'the line is not a JSON object')];- const { registered_by: _by, origin: _origin, ...content } = value as Record<string, unknown>;+ const { registered_by: _by, origin: _origin, labels: rawLabels, ...content } = value as Record<string, unknown>;const id = typeof content.id === 'string' ? content.id : null;let card: CardInput;- try { card = parseCardInput(content); } catch (error) {+ let labels: AccessLabels;+ try { card = parseCardInput(content); labels = rawLabels === undefined ? {} : parseAccessLabels(rawLabels); } catch (error) {if (!(error instanceof OperationError)) throw error;return [invalidLine(line, id, error.reason ?? 'schema', error.message.replace(/; nothing was written\.$/, ''), error.code)];}if (seen.has(card.id)) return [invalidLine(line, card.id, 'duplicate_id', `an earlier line holds card id ${card.id}`)];seen.add(card.id);- return [{ line, id: card.id, card }];+ return [{ line, id: card.id, card, labels }];});}/** Preview an import: per line what the apply would do. Writes nothing; a brain with a Git checkout refuses until storage init has run. */-export async function previewImport(engine: BrainEngine, file: string, text: string): Promise<ImportPlan & { cards: Map<number, CardInput> }> {+export async function previewImport(engine: BrainEngine, file: string, text: string): Promise<ImportPlan & { cards: Map<number, LineCard> }> {await shelfWritable(engine);- const shelf = new Map((await readShelfCards(engine)).map((row) => [row.id, row.card]));- const cards = new Map<number, CardInput>();- const lines: ImportLine[] = parseLines(text).map(({ card, ...rest }) => {+ const shelf = new Map((await readShelfCards(engine)).map((row) => [row.id, row]));+ const cards = new Map<number, LineCard>();+ const lines: ImportLine[] = parseLines(text).map(({ card, labels, ...rest }) => {if (!card) return { ...rest, action: 'invalid' };- cards.set(rest.line, card);- if (!shelf.has(card.id)) return { line: rest.line, id: card.id, action: 'create' };+ cards.set(rest.line, { card, labels: labels! });const onShelf = shelf.get(card.id);- const existing = onShelf ? cardContent(onShelf) : null;- return { line: rest.line, id: card.id, action: existing && cardDigest(existing) === cardDigest(card) ? 'identical' : 'conflict' };+ if (!onShelf) return { line: rest.line, id: card.id, action: 'create' };+ const existing = onShelf.card ? cardContent(onShelf.card) : null;+ const same = existing && cardDigest(existing) === cardDigest(card) && labelsDigest(onShelf.labels) === labelsDigest(labels!);+ return { line: rest.line, id: card.id, action: same ? 'identical' : 'conflict' };});const counts: Record<ImportAction, number> = { create: 0, identical: 0, conflict: 0, invalid: 0 };for (const line of lines) counts[line.action]++;const plan_hash = sha256(JSON.stringify({ v: 1, kind: 'import', lines: lines.map((l) => ({ line: l.line, id: l.id,- card: cards.has(l.line) ? cardDigest(cards.get(l.line)!) : null, invalid: l.action === 'invalid' ? l.reason : null, conflict: l.action === 'conflict' })) }));+ card: cards.has(l.line) ? cardDigest(cards.get(l.line)!.card) : null, labels: cards.has(l.line) ? labelsDigest(cards.get(l.line)!.labels) : null, invalid: l.action === 'invalid' ? l.reason : null, conflict: l.action === 'conflict' })) }));const summary = [`${lines.length} card line(s) in ${file}: ${counts.create} to create, ${counts.identical} already on the shelf, ${counts.conflict} conflict(s), ${counts.invalid} invalid.`];- if (counts.conflict) summary.push('A conflict is a card already on the shelf that differs from its line; import never overwrites it.');+ if (counts.conflict) summary.push('A conflict is a card already on the shelf that differs from its line, its access labels included; import never overwrites it.');if (counts.invalid) summary.push('Invalid lines are skipped; the line numbers and reasons are listed (values are not shown).');const apply = counts.create ? ['gbrain', 'storage', 'import', '--jsonl', file, '--apply', '--expect', plan_hash] : undefined;return { action: 'import', dry_run: true, file, lines, counts, plan_hash, summary, ...(apply ? { apply } : {}), cards };@@ -99,12 +105,16 @@ export async function applyImport(ctx: OperationContext, file: string, text: str{ fix: readFix('Prints the current import plan and its plan_hash; writes nothing.', { argv: previewArgv }) });}const created: string[] = [];+ const pending: PendingWrite[] = [];let failure: StepFailure | undefined;for (const line of plan.lines.filter((l) => l.action === 'create')) {+ const { card, labels } = cards.get(line.line)!;try {- await registerCard(ctx, cards.get(line.line)!, 'import');+ await registerCard(ctx, card, 'import', undefined, labels);created.push(line.id!);} catch (error) {+ const held = pendingWrite(line.id!, error);+ if (held) { pending.push(held); continue; }if (!(error instanceof OperationError)) throw error;failure = { item: line.id!, code: error.canonical ?? error.code, message: error.message };break;@@ -112,7 +122,9 @@ export async function applyImport(ctx: OperationContext, file: string, text: str}const ids = (action: ImportAction) => plan.lines.filter((l) => l.action === action).map((l) => l.id!);return { action: 'import', dry_run: false, file, plan_hash: plan.plan_hash,- side_effects: failure ? (created.length ? 'partial' : 'none') : created.length ? 'complete' : 'none',+ side_effects: sideEffects(created.length > 0 || pending.length > 0, !!failure || pending.length > 0),created, identical: ids('identical'), conflicts: ids('conflict'), invalid: plan.lines.filter((l) => l.action === 'invalid').map((l) => l.line),- ...(failure ? { failure, rerun: plan.apply ?? ['gbrain', 'storage', 'import', '--jsonl', file, '--apply', '--expect', plan.plan_hash] } : {}) };+ ...(pending.length ? { pending } : {}),+ ...(failure ? { failure } : {}),+ ...(failure || pending.length ? { rerun: plan.apply ?? ['gbrain', 'storage', 'import', '--jsonl', file, '--apply', '--expect', plan.plan_hash] } : {}) };}diff --git a/src/core/storage-manifest/schema.ts b/src/core/storage-manifest/schema.tsindex a39eea539f..011997e3bc 100644--- a/src/core/storage-manifest/schema.ts+++ b/src/core/storage-manifest/schema.ts@@ -12,6 +12,7 @@ import { z } from 'zod';import { opError, OperationError } from '../ops/contract.ts';import { scanTextSpans } from '../secret-scan.ts';import { canonicalJson } from '../remediation-step.ts';+import { LABEL_KEYS } from '../persistence/label-edit.ts';export const STORED_OBJECT_TYPE = 'stored-object';export const CARD_ID_RE = /^[a-z0-9][a-z0-9-]{1,79}$/;@@ -88,8 +89,7 @@ export type Card = CardInput & { registered_by: string; origin: (typeof ORIGINS)const COMMAND_KEYS = new Set(['command', 'cmd', 'script', 'exec', 'shell', 'instructions']);export const STAMPED_KEYS = ['registered_by', 'origin'] as const;const CREDENTIAL_NAME = /(?:KEY|SECRET|TOKEN|PASSWORD|PASSWD|PASS|CRED|AUTH)/i;-const CREDENTIAL_KEY = /(?:^|_)(?:password|passwd|passphrase|secret|token|credentials?|(?:api|access|secret|application|account|private|auth|master|encryption)_?key)(?:_|$)/;-const REFERENCE_KEY = /_(?:env|var|variable|file|path|name|ref|id)$/;+const CREDENTIAL_KEY = /(?:^|_)(?:password|passwd|passphrase|secret|token|credentials?|(?:api|access|secret|application|account|private|auth|master|encryption)_?key)$/;const ENV_ASSIGNMENT = /(?:^|[\s;&|(])[A-Za-z_][A-Za-z0-9_]{1,}=\S/;const VAR_REFERENCE = /\$\{?([A-Za-z_][A-Za-z0-9_]*)/g;const URL_USERINFO = /[a-z][a-z0-9+.-]*:\/\/[^/\s@]*:[^/\s@]*@/i;@@ -131,11 +131,14 @@ export function isCommandKey(key: string): boolean {return COMMAND_KEYS.has(key.toLowerCase());}-/** A non-empty string or number under a key named for a credential (password, application_key), unless the key names where it lives (password_file, token_env). */+/**+ * A non-empty string or number under a key whose last word names a credential (password, restic_password, application_key).+ * A key that only starts with one (token_env, password_file, token_count, api_key_last4) names something else.+ */export function holdsCredential(key: string, value: unknown): boolean {if (!(typeof value === 'number' || (typeof value === 'string' && value.trim() !== ''))) return false;const name = key.replace(/([a-z0-9])([A-Z])/g, '$1_$2').replace(/-/g, '_').toLowerCase();- return CREDENTIAL_KEY.test(name) && !REFERENCE_KEY.test(name);+ return CREDENTIAL_KEY.test(name);}function credentialKeyRefusal(path: string): OperationError {@@ -223,6 +226,26 @@ function withoutEmpty(value: unknown): unknown {}/** Stable text of a card's content (null and absent fields are the same), for comparing and hashing. */+/** The page labels that decide who reads a page (visibility, its access and confidentiality aliases, readable_by). */+export const ACCESS_LABEL_KEYS: readonly string[] = [...LABEL_KEYS, 'readable_by'];+export type AccessLabels = Record<string, unknown>;++/** The access labels a page's frontmatter carries. */+export function accessLabels(frontmatter: Record<string, unknown>): AccessLabels {+ return Object.fromEntries(ACCESS_LABEL_KEYS.filter((key) => Object.hasOwn(frontmatter, key)).map((key) => [key, frontmatter[key]]));+}++/** An export line's `labels`: access label keys only, each a string (readable_by a reader or a list of readers). */+export function parseAccessLabels(value: unknown): AccessLabels {+ if (!value || typeof value !== 'object' || Array.isArray(value)) throw invalid('labels', 'must be an object of access labels', 'labels');+ for (const [key, item] of Object.entries(value)) {+ if (!ACCESS_LABEL_KEYS.includes(key)) throw invalid(`labels.${key}`, `is not an access label (${ACCESS_LABEL_KEYS.join(', ')})`, 'labels');+ const ok = typeof item === 'string' || (key === 'readable_by' && Array.isArray(item) && item.length > 0 && item.every((r) => typeof r === 'string'));+ if (!ok) throw invalid(`labels.${key}`, key === 'readable_by' ? 'must be a reader or a list of readers' : 'must be a string', 'labels');+ }+ return value as AccessLabels;+}+export function cardDigest(card: CardInput): string {return canonicalJson(withoutEmpty(card));}diff --git a/test/storage-manifest-adopt.test.ts b/test/storage-manifest-adopt.test.tsindex 89a275f26c..78b5d9fe0e 100644--- a/test/storage-manifest-adopt.test.ts+++ b/test/storage-manifest-adopt.test.ts@@ -10,7 +10,11 @@* half-finished run printed still applies; a differing card is a conflict, never overwritten, its original untagged;* - finalize previews, then soft-deletes only the tagged originals whose card matches (storage_adopt_unverified when* one differs), restorable afterwards;- * - a brain with a Git checkout refuses storage_manifest_missing until storage init has run.+ * - a brain with a Git checkout refuses storage_manifest_missing until storage init has run;+ * - each card takes its original's access labels, readable_by narrowed to the readers of the folder rules over it, so a+ * storage-only remote caller reads no card its original was kept from; no common reader blocks the page, and a card+ * that lost the labels is a conflict whose original finalize never deletes;+ * - a write accepted but still pending is reported with its receipt (side_effects partial, never none).*/import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';import { randomUUID } from 'node:crypto';@@ -19,6 +23,7 @@ import { operationsByName } from '../src/core/operations.ts';import type { OperationContext } from '../src/core/ops/contract.ts';import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';import { applyAdopt, applyFinalize, previewAdopt, previewFinalize, ADOPTED_TAG } from '../src/core/storage-manifest/adopt.ts';+import { proposeRevision } from '../src/core/access/policy.ts';import { resetPgliteState } from './helpers/reset-pglite.ts';let engine: PGLiteEngine;@@ -38,11 +43,22 @@ beforeEach(async () => {await engine.unsetConfig('sync.repo_path');});-const ctx = (): OperationContext => ({+const ctx = (over: Partial<OperationContext> = {}): OperationContext => ({engine, config: { engine: engine.kind, embedding_disabled: true } as OperationContext['config'],- logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote: false, sourceId: 'default',+ logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote: false, sourceId: 'default', ...over,}) as OperationContext;-const call = (name: string, params: Record<string, unknown>) => operationsByName[name]!.handler(ctx(), params) as Promise<Record<string, any>>;+const call = (name: string, params: Record<string, unknown>, c: OperationContext = ctx()) => operationsByName[name]!.handler(c, params) as Promise<Record<string, any>>;+const storageOnly = () => ctx({ remote: true, transport: 'http', sourceId: 'storage',+ auth: { token: 'fixture', clientId: 'client-s', principal: { kind: 'oauth_client', id: 'client-s' }, scopes: ['read'], sourceId: 'storage', allowedSources: ['storage'] } } as Partial<OperationContext>);+const restrict = (prefix: string, readers: string[]) => proposeRevision(engine, 'default', [{ id: 'restricted', prefix, readable_by: readers }], { origin: 'cli', store: 'database' });+async function committed(slug: string, sourceId: string, gone = false): Promise<void> {+ for (let i = 0; i < 100; i++) {+ const rows = await engine.executeRaw('SELECT 1 FROM pages WHERE source_id = $1 AND slug = $2 AND deleted_at IS NULL', [sourceId, slug]);+ if (rows.length === (gone ? 0 : 1)) return;+ await new Promise((resolve) => setTimeout(resolve, 100));+ }+ throw new Error(`${slug} did not commit`);+}const refusal = (promise: Promise<unknown>) => promise.then(() => null, (e: { code?: string; canonical?: string; reason?: string; message?: string; fix?: Record<string, unknown> }) =>({ code: e.canonical ?? e.code, reason: e.reason ?? null, message: e.message ?? '', fix: e.fix ?? null }));const opts = { sourceId: 'default', prefix: 'storage-manifest/' };@@ -61,9 +77,9 @@ function record(id: string, over: Record<string, unknown> = {}): Record<string,const FENCE = '```';-async function interim(slug: string, rec: Record<string, unknown> | string, type = 'stored-object'): Promise<void> {+async function interim(slug: string, rec: Record<string, unknown> | string, type = 'stored-object', labels = ''): Promise<void> {const json = typeof rec === 'string' ? rec : JSON.stringify(rec, null, 2);- const content = `---\ntype: ${type}\ntitle: Interim ${slug}\n---\n# Interim ${slug}\n\n## Record\n\n${FENCE}json\n${json}\n${FENCE}\n\n## Restore\n\nAsk the archive agent.\n`;+ const content = `---\ntype: ${type}\ntitle: Interim ${slug}\n${labels ? `${labels}\n` : ''}---\n# Interim ${slug}\n\n## Record\n\n${FENCE}json\n${json}\n${FENCE}\n\n## Restore\n\nAsk the archive agent.\n`;await call('put_page', { slug, content, request_id: randomUUID() });}@@ -246,4 +262,60 @@ describe('storage adopt on a repo-less brain', () => {expect((await refusal(previewAdopt(ctx(), opts)))?.code).toBe('storage_manifest_missing');expect(await engine.executeRaw("SELECT 1 FROM sources WHERE id='storage'")).toEqual([]);}, 60_000);++ test('adopt carries each original\'s access labels and folder-rule readers, so a storage-only remote caller reads no card its original was kept from', async () => {+ await restrict('storage-manifest/restricted/', ['member:alice-example', 'member:bob-example']);+ await interim('storage-manifest/dataset-m', record('dataset-m'), 'stored-object', 'readable_by: [alice-example]');+ await interim('storage-manifest/dataset-p', record('dataset-p'), 'stored-object', 'visibility: private');+ await interim('storage-manifest/restricted/dataset-r', record('dataset-r'), 'stored-object', 'readable_by: [bob-example, carol-example]');+ await interim('storage-manifest/dataset-o', record('dataset-o'));+ const plan = await previewAdopt(ctx(), opts);+ expect(Object.fromEntries(plan.entries.map((e) => [e.card_id, e.labels]))).toEqual({+ 'dataset-m': { readable_by: ['alice-example'] }, 'dataset-o': {}, 'dataset-p': { visibility: 'private' }, 'dataset-r': { readable_by: ['member:bob-example'] } });+ expect(plan.summary.join(' ')).toContain("3 card(s) take their original's access labels");+ expect(await applyAdopt(ctx(), { ...opts, expect: plan.plan_hash, dropFields: [] })).toMatchObject({ side_effects: 'complete' });+ const stored = await engine.executeRaw<{ slug: string; visibility: string | null; readers: unknown }>(+ "SELECT slug, frontmatter->>'visibility' AS visibility, frontmatter->'readable_by' AS readers FROM pages WHERE source_id = 'storage' ORDER BY slug");+ expect(stored).toEqual([{ slug: 'dataset-m', visibility: null, readers: ['alice-example'] }, { slug: 'dataset-o', visibility: null, readers: null },+ { slug: 'dataset-p', visibility: 'private', readers: null }, { slug: 'dataset-r', visibility: null, readers: ['member:bob-example'] }]);+ for (const id of ['dataset-m', 'dataset-p', 'dataset-r']) expect((await refusal(call('storage_get', { id }, storageOnly())))?.code).toBe('not_found');+ expect((await call('storage_get', { id: 'dataset-o' }, storageOnly())).card.id).toBe('dataset-o');+ expect((await call('storage_list', {}, storageOnly())).cards.map((c: { id: string }) => c.id)).toEqual(['dataset-o']);+ expect((await call('storage_list', {})).total).toBe(4);+ }, 60_000);++ test('an original whose readers and folder rules share no reader is blocked; a card that lost its original\'s labels is a conflict and its original is never finalized', async () => {+ await restrict('storage-manifest/restricted/', ['member:alice-example']);+ await interim('storage-manifest/restricted/dataset-x', record('dataset-x'), 'stored-object', 'readable_by: [carol-example]');+ const blocked = await previewAdopt(ctx(), opts);+ expect(blocked.entries.map((e) => [e.state, e.problem?.reason])).toEqual([['blocked', 'access_unpreservable']]);+ expect((await refusal(applyAdopt(ctx(), { ...opts, expect: blocked.plan_hash, dropFields: [] })))?.reason).toBe('access_unpreservable');+ expect(await engine.executeRaw("SELECT 1 FROM pages WHERE source_id = 'storage'")).toEqual([]);++ const x = await engine.readPageSnapshot('storage-manifest/restricted/dataset-x', { sourceId: 'default' });+ await call('delete_page', { slug: 'storage-manifest/restricted/dataset-x', expected_revision: x!.revision });+ await interim('storage-manifest/dataset-m', record('dataset-m'), 'stored-object', 'readable_by: [alice-example]');+ const plan = await previewAdopt(ctx(), opts);+ await applyAdopt(ctx(), { ...opts, expect: plan.plan_hash, dropFields: [] });+ await engine.executeRaw("UPDATE pages SET frontmatter = frontmatter - 'readable_by' WHERE source_id = 'storage' AND slug = 'dataset-m'");+ expect((await previewFinalize(ctx(), opts)).entries.map((e) => [e.slug, e.state])).toEqual([['storage-manifest/dataset-m', 'unverified']]);+ await engine.executeRaw('DELETE FROM tags WHERE tag = $1', [ADOPTED_TAG]);+ expect((await previewAdopt(ctx(), opts)).entries.map((e) => e.state)).toEqual(['conflict']);+ }, 60_000);++ test('a write accepted but still pending is reported with its receipt, never as no change, and the rerun finishes it', async () => {+ await interim('storage-manifest/dataset-a', record('dataset-a'));+ const plan = await previewAdopt(ctx(), opts);+ const held = await applyAdopt(ctx({ writeWaitMs: 0 }), { ...opts, expect: plan.plan_hash, dropFields: [] });+ expect(held).toMatchObject({ side_effects: 'partial', results: [], pending: [{ item: 'storage-manifest/dataset-a', state: expect.any(String), poll: ['gbrain', 'write-request', expect.any(String)] }],+ rerun: ['gbrain', 'storage', 'adopt', '--apply', '--expect', plan.plan_hash] });+ expect(held.failure).toBeUndefined();+ expect(held.finalize).toBeUndefined();+ await committed('dataset-a', 'storage');+ expect(await applyAdopt(ctx(), { ...opts, expect: plan.plan_hash, dropFields: [] })).toMatchObject({ side_effects: 'complete', results: [{ outcome: 'retagged' }] });+ const finalize = await previewFinalize(ctx(), opts);+ const deleting = await applyFinalize(ctx({ writeWaitMs: 0 }), { ...opts, expect: finalize.plan_hash });+ expect(deleting).toMatchObject({ side_effects: 'partial', deleted: [], pending: [{ item: 'storage-manifest/dataset-a' }] });+ await committed('storage-manifest/dataset-a', 'default', true);+ }, 60_000);});diff --git a/test/storage-manifest-jsonl.test.ts b/test/storage-manifest-jsonl.test.tsindex 02a51c9694..64bf15ea55 100644--- a/test/storage-manifest-jsonl.test.ts+++ b/test/storage-manifest-jsonl.test.ts@@ -4,7 +4,10 @@* - importing that export into an empty brain previews every line as create, and the apply (bound to its plan_hash)* reproduces every card with origin import, so a second export matches the first apart from the stamps;* - a rerun is idempotent (identical, side_effects none); a card on the shelf that differs is a conflict, never- * overwritten; invalid lines (not JSON, a credential shape, a duplicate id) are listed without their values.+ * overwritten; invalid lines (not JSON, a credential shape, a duplicate id) are listed without their values;+ * - export carries each card page's access labels and import restores them (a label difference is a conflict), so a+ * storage-only remote caller reads no restricted card after a recovery;+ * - a write accepted but still pending is reported with its receipt (side_effects partial, never none).*/import { afterAll, beforeAll, describe, expect, test } from 'bun:test';import { PGLiteEngine } from '../src/core/pglite-engine.ts';@@ -30,11 +33,15 @@ afterAll(async () => {await target.disconnect();}, 60_000);-const ctx = (engine: PGLiteEngine): OperationContext => ({+const ctx = (engine: PGLiteEngine, over: Partial<OperationContext> = {}): OperationContext => ({engine, config: { engine: engine.kind, embedding_disabled: true } as OperationContext['config'],- logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote: false, sourceId: 'default',+ logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote: false, sourceId: 'default', ...over,}) as OperationContext;-const call = (engine: PGLiteEngine, name: string, params: Record<string, unknown>) => operationsByName[name]!.handler(ctx(engine), params) as Promise<Record<string, any>>;+const call = (engine: PGLiteEngine, name: string, params: Record<string, unknown>, c: OperationContext = ctx(engine)) => operationsByName[name]!.handler(c, params) as Promise<Record<string, any>>;+const storageOnly = (engine: PGLiteEngine) => ctx(engine, { remote: true, transport: 'http', sourceId: 'storage',+ auth: { token: 'fixture', clientId: 'client-s', principal: { kind: 'oauth_client', id: 'client-s' }, scopes: ['read'], sourceId: 'storage', allowedSources: ['storage'] } } as Partial<OperationContext>);+const labelsOf = (engine: PGLiteEngine, id: string) => engine.executeRaw<{ visibility: string | null; readers: unknown }>(+ "SELECT frontmatter->>'visibility' AS visibility, frontmatter->'readable_by' AS readers FROM pages WHERE source_id = 'storage' AND slug = $1", [id]);function card(id: string, over: Record<string, unknown> = {}): Record<string, unknown> {return {@@ -105,4 +112,38 @@ describe('storage shelf JSONL export and import', () => {expect(exported.invalid).toEqual(['dataset-e']);expect(exported.skipped).toEqual([]);}, 60_000);++ test('export carries each card page\'s access labels and import restores them, so a storage-only remote caller reads neither restricted card', async () => {+ await call(source, 'storage_register', { card: card('dataset-m') });+ await call(source, 'storage_register', { card: card('dataset-p') });+ await source.executeRaw(`UPDATE pages SET frontmatter = frontmatter || '{"readable_by":["alice-example"]}'::jsonb WHERE source_id = 'storage' AND slug = 'dataset-m'`);+ await source.executeRaw(`UPDATE pages SET frontmatter = frontmatter || '{"visibility":"private"}'::jsonb WHERE source_id = 'storage' AND slug = 'dataset-p'`);+ const lines = (await exportShelfJsonl(source)).lines.filter((l) => ['dataset-m', 'dataset-p'].includes(JSON.parse(l).id));+ expect(lines.map((l) => JSON.parse(l).labels)).toEqual([{ readable_by: ['alice-example'] }, { visibility: 'private' }]);+ const text = `${lines.join('\n')}\n`;+ const { cards: _c, ...plan } = await previewImport(target, 'labels.jsonl', text);+ expect(plan.lines.map((l) => l.action)).toEqual(['create', 'create']);+ expect(await applyImport(ctx(target), 'labels.jsonl', text, plan.plan_hash)).toMatchObject({ side_effects: 'complete', created: ['dataset-m', 'dataset-p'] });+ expect(await labelsOf(target, 'dataset-m')).toEqual([{ visibility: null, readers: ['alice-example'] }]);+ expect(await labelsOf(target, 'dataset-p')).toEqual([{ visibility: 'private', readers: null }]);+ for (const id of ['dataset-m', 'dataset-p']) expect(await call(target, 'storage_get', { id }, storageOnly(target)).then(() => null, (e) => e.code)).toBe('not_found');+ expect((await call(target, 'storage_get', { id: 'dataset-d' }, storageOnly(target))).card.id).toBe('dataset-d');++ const relabelled = lines[0]!.replace('"readable_by":["alice-example"]', '"readable_by":["bob-example"]');+ const unlabelled = JSON.stringify({ ...JSON.parse(lines[1]!), labels: undefined });+ const badLabels = JSON.stringify({ ...card('dataset-q'), labels: { owner: 'alice-example' } });+ const { cards: _again, ...second } = await previewImport(target, 'relabel.jsonl', [relabelled, unlabelled, badLabels].join('\n'));+ expect(second.lines.map((l) => [l.id, l.action, l.reason ?? null])).toEqual([['dataset-m', 'conflict', null], ['dataset-p', 'conflict', null], ['dataset-q', 'invalid', 'labels']]);+ }, 60_000);++ test('an import write accepted but still pending is reported with its receipt, never as no change', async () => {+ const text = `${JSON.stringify(card('dataset-w'))}\n`;+ const { cards: _c, ...plan } = await previewImport(target, 'pending.jsonl', text);+ const held = await applyImport(ctx(target, { writeWaitMs: 0 }), 'pending.jsonl', text, plan.plan_hash);+ expect(held).toMatchObject({ side_effects: 'partial', created: [], pending: [{ item: 'dataset-w', poll: ['gbrain', 'write-request', expect.any(String)] }],+ rerun: ['gbrain', 'storage', 'import', '--jsonl', 'pending.jsonl', '--apply', '--expect', plan.plan_hash] });+ expect(held.failure).toBeUndefined();+ for (let i = 0; i < 100 && !(await target.executeRaw("SELECT 1 FROM pages WHERE source_id = 'storage' AND slug = 'dataset-w'")).length; i++) await new Promise((r) => setTimeout(r, 100));+ expect(await applyImport(ctx(target), 'pending.jsonl', text, plan.plan_hash)).toMatchObject({ side_effects: 'none', identical: ['dataset-w'] });+ }, 60_000);});diff --git a/test/storage-manifest.test.ts b/test/storage-manifest.test.tsindex 00d3721386..15b811a18d 100644--- a/test/storage-manifest.test.ts+++ b/test/storage-manifest.test.ts@@ -6,7 +6,7 @@* the closest cards; storage_list filters by store and status;* - validation refuses a missing field, a command field, a stamped field, a scanner secret (in a value or an object* key) and every credential shape (env assignment, credential variable, URL userinfo, presigned URL, a value under- * a credential-named key), without echoing the value; a key naming where a credential lives (token_env) passes;+ * a credential-named key), without echoing the value; a key whose last word names no credential (token_env, token_count, api_key_last4) passes;* - storage_update keeps the page's access frontmatter (visibility, readable_by) when it rewrites the card;* - the write guard: put_page, remember and forget_fact into the shelf are refused storage_source_managed, a forged* parameter does not open it, and the storage verbs still write;@@ -135,6 +135,8 @@ describe('storage shelf on a repo-less brain', () => {[card('a-10', { details: { AKIAIOSFODNN7EXAMPLE: 'b2 key' } }), 'storage_record_secret', 'secret_scanner'],[card('a-11', { restore: { skill: 'restore-data', args: { id: 'a-11', applicationKey: 'K005abcdef123ghijkl456mnopqr789' } } }), 'storage_record_secret', 'credential_key'],[card('a-12', { details: { restic_password: 'abcdef123-correct-horse' } }), 'storage_record_secret', 'credential_key'],+ [card('a-14', { restore: { skill: 'restore-data', args: { id: 'a-14', access_token: 'abcdef123ghijklmno' } } }), 'storage_record_secret', 'credential_key'],+ [card('a-15', { details: { secretAccessKey: 'abcdef123/pqrstuvw' } }), 'storage_record_secret', 'credential_key'],];for (const [input, code, reason] of cases) {const r = await refusal(call('storage_register', { card: input }));@@ -144,7 +146,8 @@ describe('storage shelf on a repo-less brain', () => {expect(r!.message).not.toContain('K005abcdef123');}expect(await engine.executeRaw("SELECT 1 FROM pages WHERE source_id='storage'")).toEqual([]);- await call('storage_register', { card: card('a-13', { restore: { skill: 'restore-data', args: { id: 'a-13', key: 'backups/a-13.tar', password_file: '/etc/restic/pass', token_env: 'B2_APPLICATION_KEY' } } }) });+ await call('storage_register', { card: card('a-13', { restore: { skill: 'restore-data', args: { id: 'a-13', key: 'backups/a-13.tar', password_file: '/etc/restic/pass', token_env: 'B2_APPLICATION_KEY' } },+ details: { token_count: 1200, secret_version: 'v2', password_changed_at: '2026-11-01T00:00:00Z', api_key_last4: '7c2f' } }) });}, 30_000);test('the write guard refuses put_page and remember into the shelf, forged parameters included', async () => {@@ -238,6 +241,16 @@ describe('storage shelf on a repo-less brain', () => {expect((await call('storage_get', { id: 'item-b' })).card.status).toBe('retired');}, 30_000);+ test('storage_update rewrites a card whose stamped fields were removed by hand', async () => {+ await call('storage_register', { card: card('unstamped-item') });+ await engine.executeRaw("UPDATE pages SET frontmatter = frontmatter #- '{storage,registered_by}' #- '{storage,origin}' WHERE source_id = 'storage' AND slug = 'unstamped-item'");+ const { revision } = await call('storage_get', { id: 'unstamped-item' });+ await call('storage_update', { id: 'unstamped-item', expected_revision: revision, patch: { status: 'retired' } });+ await call('storage_update', { id: 'unstamped-item', expected_revision: (await call('storage_get', { id: 'unstamped-item' })).revision, restored_to: '/scratch/unstamped-item' });+ const after = await call('storage_get', { id: 'unstamped-item' });+ expect({ status: after.card.status, restores: after.card.restores.length, by: after.card.registered_by ?? null }).toEqual({ status: 'restored', restores: 1, by: null });+ }, 30_000);+test('remote callers cannot write; a bound client without the storage grant reads nothing; only a grant naming the shelf reads it', async () => {await call('storage_register', { card: card('remote-item') });const remote = ctx({ remote: true, transport: 'http' } as Partial<OperationContext>);