Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pre-landing review fixes

garrytanba2de3bd8def
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index 22a33253b0..8d14a36ac3 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -5476,9 +5476,9 @@ More: [docs/guides/write-refusals.md#storage-manifest-refusals](../../docs/guide
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| A storage card field looks like a secret or has a credential shape; nothing was written and the value is not echoed. | Cards are read by agents and copied into the stores; credentials stay in the environment of the skill that restores an item. | Remove the secret, environment assignment, credential variable, URL password or signed-URL parameter from the named field. | agent | `repeat the read that failed` | 1 | no |
+| A storage card field looks like a secret or has a credential shape; nothing was written and the value is not echoed. | Cards are read by agents and copied into the stores; credentials stay in the environment of the skill that restores an item. | Remove the secret, environment assignment, credential variable, URL password, signed-URL parameter or credential-named field from the card. | agent | `repeat the read that failed` | 1 | no |
-Reasons: `secret_scanner`, `env_assignment`, `credential_variable`, `url_userinfo`, `presigned_url`.
+Reasons: `secret_scanner`, `env_assignment`, `credential_variable`, `url_userinfo`, `presigned_url`, `credential_key`.
More: [docs/guides/write-refusals.md#storage-manifest-refusals](../../docs/guides/write-refusals.md#storage-manifest-refusals)
diff --git a/docs/guides/storage-manifest.md b/docs/guides/storage-manifest.md
index 9c73ffbdad..78747370c3 100644
--- a/docs/guides/storage-manifest.md
+++ b/docs/guides/storage-manifest.md
@@ -56,8 +56,8 @@ Agents that archived before the shelf existed often kept interim pages: `type: s
2. The printed apply, `--apply --expect <plan_hash>` with one `--drop-field <dotted.name>` per refused field, runs
after the user agrees to leave those fields out. It registers each card with `origin: adopt` through the same
path `storage-register` uses, then tags the original `storage-manifest-adopted`. It deletes nothing. A
- credential-shaped value (a secret, an environment assignment, a credential variable, a URL password, a signed URL)
- always refuses, whatever is dropped: fix the source page first.
+ credential-shaped value (a secret, an environment assignment, a credential variable, a URL password, a signed URL,
+ a value under a credential-named key such as `restore.args.password`) always refuses, whatever is dropped: fix the source page first.
3. `gbrain storage adopt --finalize` previews soft-deleting the tagged originals; its `--expect <plan_hash>` run deletes
them (restorable for 72 hours with `gbrain restore`), only when every adopted card is on the shelf and matches its
original's mapped card. A restore recorded with `storage-update` since adoption is not a difference. The preview
diff --git a/src/core/engine-sql/sources.ts b/src/core/engine-sql/sources.ts
index d81d904675..fe03be345b 100644
--- a/src/core/engine-sql/sources.ts
+++ b/src/core/engine-sql/sources.ts
@@ -20,14 +20,15 @@ export async function listAllSources(exec: LegacyUnscopedRead, opts?: {
// how many archived rows exist. ORDER BY (id='default') DESC, id
// matches sources-ops.listSources for operator-output stability.
// localPathOnly skips pure-DB sources so autopilot fan-out doesn't
- // dispatch jobs that would fall back to the global sync.repo_path.
+ // dispatch jobs that would fall back to the global sync.repo_path, and
+ // the storage shelf, whose cycle is always skipped (GST-1943).
const includeArchived = opts?.includeArchived === true;
const localPathOnly = opts?.localPathOnly === true;
const { rows } = await exec.run(sqlFragment`
SELECT id, name, local_path, last_sync_at, config
FROM sources
WHERE (${includeArchived} OR archived IS NOT TRUE)
- AND (${!localPathOnly} OR local_path IS NOT NULL)
+ AND (${!localPathOnly} OR (local_path IS NOT NULL AND config->>'role' IS DISTINCT FROM 'storage_manifest'))
ORDER BY (id = 'default') DESC, id
`);
return rows.map((r) => ({
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 9d0f6f440f..b18ae934bf 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -844,7 +844,7 @@ export const CODES = {
storage_card_exists: { class: 'caller', summary: "A storage card with this id is already on the storage shelf; nothing was written.", why: 'Registering never overwrites a card, so two archives cannot silently share one id; a change to an existing card names the revision it read.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Change the card with gbrain storage-update <id> --expected-revision <revision from storage_get>, or register the new item under another id.' },
storage_manifest_missing: { class: 'consent', actor: 'user', summary: "This brain has no storage shelf to write to (or the storage source ID belongs to another source); nothing was written.", why: 'On a brain with a Git checkout the shelf is a hosted git-canonical repository, created only after the user previews it, and hosting needs managed persistence. A repo-less brain gets its database-only shelf on the first register.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Preview gbrain storage init --dir <absolute path> --dry-run and apply the command it prints after the user agrees.' },
storage_record_invalid: { class: 'caller', summary: "A storage card failed validation (a missing or malformed field, a command field, or a field gbrain stamps itself); nothing was written.", why: 'Every card must say where the item is, how big it is and how to restore it, and carries no commands, so a card is complete and safe to act on when it is found months later.', reasons: ['schema', 'command_field', 'stamped_field', 'not_json', 'not_object', 'too_large', 'id_change', 'no_record', 'duplicate_id'], docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Fix the field the message names and send the card again; gbrain storage-register --help lists the card fields.' },
- storage_record_secret: { class: 'caller', summary: "A storage card field looks like a secret or has a credential shape; nothing was written and the value is not echoed.", why: 'Cards are read by agents and copied into the stores; credentials stay in the environment of the skill that restores an item.', reasons: ['secret_scanner', 'env_assignment', 'credential_variable', 'url_userinfo', 'presigned_url'], docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Remove the secret, environment assignment, credential variable, URL password or signed-URL parameter from the named field.' },
+ storage_record_secret: { class: 'caller', summary: "A storage card field looks like a secret or has a credential shape; nothing was written and the value is not echoed.", why: 'Cards are read by agents and copied into the stores; credentials stay in the environment of the skill that restores an item.', reasons: ['secret_scanner', 'env_assignment', 'credential_variable', 'url_userinfo', 'presigned_url', 'credential_key'], docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Remove the secret, environment assignment, credential variable, URL password, signed-URL parameter or credential-named field from the card.' },
storage_revision_conflict: { class: 'caller', summary: "A storage card changed since the revision the update named was read; nothing was written.", why: 'Updates bind to the revision they read, so two writers never silently overwrite each other; the refusal carries the current revision in detail.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Read the card again with storage_get, apply the change to what it says now, and update once with its revision.' },
storage_source_managed: { class: 'caller', summary: "A page write targeted the storage shelf outside the storage verbs; nothing was written.", why: 'The storage source holds only validated cards, so put_page, remember and the other page writers are refused there.', docs: 'docs/guides/write-refusals.md#storage-manifest-refusals', suggestion: 'Register cards with gbrain storage-register and change them with gbrain storage-update.' },
storage_error: { class: 'server', summary: "Reading or writing durable storage failed.",
diff --git a/src/core/ops/storage-manifest.ts b/src/core/ops/storage-manifest.ts
index ff83b6cdbb..c57a7cbad4 100644
--- a/src/core/ops/storage-manifest.ts
+++ b/src/core/ops/storage-manifest.ts
@@ -91,6 +91,10 @@ async function writeCard(ctx: OperationContext, card: Card, opts: { expectedRevi
...(opts.requestId ? { request_id: opts.requestId } : {}) } });
return response;
} catch (error) {
+ if (error instanceof OperationError && error.code === 'revision_required' && !opts.expectedRevision) {
+ const current = await ctx.engine.readPageSnapshot(card.id, { sourceId: STORAGE_SOURCE_ID });
+ throw cardExists(card.id, current?.revision ?? 'unknown');
+ }
if (error instanceof OperationError && error.code === 'revision_conflict') {
const current = await ctx.engine.readPageSnapshot(card.id, { sourceId: STORAGE_SOURCE_ID });
throw opError('storage_revision_conflict', `Card ${card.id} changed since revision ${opts.expectedRevision} was read; nothing was written.`,
@@ -101,13 +105,17 @@ async function writeCard(ctx: OperationContext, card: Card, opts: { expectedRevi
}
}
+function cardExists(id: string, revision: string): OperationError {
+ return opError('storage_card_exists', `A card with id ${id} is already on the storage shelf; nothing was written.`,
+ `Change it with gbrain storage-update ${id} --expected-revision ${revision}, or register the new item under another id.`,
+ { fix: getFix(id), detail: revision });
+}
+
/** Register a new card. Exported for `storage adopt` and `storage import` (origin adopt / import). */
export async function registerCard(ctx: OperationContext, input: CardInput, origin: (typeof ORIGINS)[number], requestId?: string): Promise<Record<string, unknown>> {
await ensureStorageSource(ctx.engine);
const existing = await ctx.engine.readPageSnapshot(input.id, { sourceId: STORAGE_SOURCE_ID });
- if (existing) throw opError('storage_card_exists', `A card with id ${input.id} is already on the storage shelf; nothing was written.`,
- `Change it with gbrain storage-update ${input.id} --expected-revision ${existing.revision}, or register the new item under another id.`,
- { fix: getFix(input.id), detail: existing.revision });
+ if (existing) throw cardExists(input.id, existing.revision);
const card: Card = { ...input, registered_by: await principalLabel(ctx), origin };
const response = await writeCard(ctx, card, { requestId });
return { id: card.id, source_id: STORAGE_SOURCE_ID, origin, write: response };
diff --git a/src/core/storage-manifest/adopt.ts b/src/core/storage-manifest/adopt.ts
index 37477230de..7360190d95 100644
--- a/src/core/storage-manifest/adopt.ts
+++ b/src/core/storage-manifest/adopt.ts
@@ -30,7 +30,7 @@ import { boundedBodyPrefixSql } from '../page-window/bounded-lead.ts';
import { BOUNDED_PREFIX_BYTES } from '../page-window/document.ts';
import { submitPageMutation } from '../persistence/page-mutations.ts';
import { readShelfCards } from './find.ts';
-import { CARD_KEYS, cardContent, cardDigest, credentialShape, isCommandKey, parseCardInput, STAMPED_KEYS, STORED_OBJECT_TYPE, type CardInput } from './schema.ts';
+import { CARD_KEYS, cardContent, cardDigest, credentialShape, holdsCredential, isCommandKey, parseCardInput, STAMPED_KEYS, STORED_OBJECT_TYPE, type CardInput } from './schema.ts';
import { shelfWritable, STORAGE_SOURCE_ID } from './source.ts';
export const ADOPT_DEFAULT_PREFIX = 'storage-manifest/';
@@ -136,6 +136,7 @@ function credentialAt(value: unknown, path: string): { field: string; shape: str
for (const [key, item] of items) {
const keyShape = Array.isArray(value) ? null : credentialShape(key);
if (keyShape) return { field: `${path || '(record)'} (a key)`, shape: keyShape };
+ if (!Array.isArray(value) && holdsCredential(key, item)) return { field: path ? `${path}.${key}` : key, shape: 'credential_key' };
const found = credentialAt(item, path ? `${path}.${key}` : key);
if (found) return found;
}
diff --git a/src/core/storage-manifest/schema.ts b/src/core/storage-manifest/schema.ts
index 66a4940a7e..a39eea539f 100644
--- a/src/core/storage-manifest/schema.ts
+++ b/src/core/storage-manifest/schema.ts
@@ -88,6 +88,8 @@ export type Card = CardInput & { registered_by: string; origin: (typeof ORIGINS)
const COMMAND_KEYS = new Set(['command', 'cmd', 'script', 'exec', 'shell', 'instructions']);
export const STAMPED_KEYS = ['registered_by', 'origin'] as const;
const CREDENTIAL_NAME = /(?:KEY|SECRET|TOKEN|PASSWORD|PASSWD|PASS|CRED|AUTH)/i;
+const CREDENTIAL_KEY = /(?:^|_)(?:password|passwd|passphrase|secret|token|credentials?|(?:api|access|secret|application|account|private|auth|master|encryption)_?key)(?:_|$)/;
+const REFERENCE_KEY = /_(?:env|var|variable|file|path|name|ref|id)$/;
const ENV_ASSIGNMENT = /(?:^|[\s;&|(])[A-Za-z_][A-Za-z0-9_]{1,}=\S/;
const VAR_REFERENCE = /\$\{?([A-Za-z_][A-Za-z0-9_]*)/g;
const URL_USERINFO = /[a-z][a-z0-9+.-]*:\/\/[^/\s@]*:[^/\s@]*@/i;
@@ -106,6 +108,7 @@ function walk(value: unknown, path: string, visit: (path: string, text: string)
for (const [key, item] of Object.entries(value)) {
const at = path ? `${path}.${key}` : key;
if (isCommandKey(key)) throw invalid(at, 'a card carries no commands; restore runs through restore.skill and restore.args', 'command_field');
+ if (holdsCredential(key, item)) throw credentialKeyRefusal(at);
visit(`${path || '(card)'} (a key)`, key);
walk(item, at, visit);
}
@@ -128,6 +131,19 @@ export function isCommandKey(key: string): boolean {
return COMMAND_KEYS.has(key.toLowerCase());
}
+/** A non-empty string or number under a key named for a credential (password, application_key), unless the key names where it lives (password_file, token_env). */
+export function holdsCredential(key: string, value: unknown): boolean {
+ if (!(typeof value === 'number' || (typeof value === 'string' && value.trim() !== ''))) return false;
+ const name = key.replace(/([a-z0-9])([A-Z])/g, '$1_$2').replace(/-/g, '_').toLowerCase();
+ return CREDENTIAL_KEY.test(name) && !REFERENCE_KEY.test(name);
+}
+
+function credentialKeyRefusal(path: string): OperationError {
+ return opError('storage_record_secret', `Storage card field ${path} is named for a credential and holds a value; nothing was written.`,
+ 'Remove that field. Cards name where an item is and how to restore it; the restore skill reads credentials from its own environment (name the variable, e.g. token_env, instead).',
+ { reason: 'credential_key' });
+}
+
/** Throws storage_record_secret without echoing the value. */
function assertNoCredential(path: string, value: string): void {
const shape = credentialShape(value);
diff --git a/test/storage-manifest-adopt.test.ts b/test/storage-manifest-adopt.test.ts
index e10f39fcf3..89a275f26c 100644
--- a/test/storage-manifest-adopt.test.ts
+++ b/test/storage-manifest-adopt.test.ts
@@ -139,6 +139,11 @@ describe('storage adopt on a repo-less brain', () => {
const keyed = (await previewAdopt(ctx(), opts)).entries.find((e) => e.slug === 'storage-manifest/dataset-k')!;
expect(keyed).toMatchObject({ state: 'blocked', refused: [], problem: { code: 'storage_record_secret', reason: 'secret_scanner' } });
expect(JSON.stringify(await previewAdopt(ctx(), opts))).not.toContain('AKIAIOSFODNN7EXAMPLE');
+
+ await interim('storage-manifest/dataset-p', record('dataset-p', { details: { password: 'fake-pass-value' } }));
+ const named = (await previewAdopt(ctx(), opts)).entries.find((e) => e.slug === 'storage-manifest/dataset-p')!;
+ expect(named).toMatchObject({ state: 'blocked', card: null, problem: { code: 'storage_record_secret', reason: 'credential_key' } });
+ expect(JSON.stringify(named)).not.toContain('fake-pass-value');
}, 60_000);
test('an unreadable record and a duplicate card id block the apply', async () => {
diff --git a/test/storage-manifest.test.ts b/test/storage-manifest.test.ts
index b0a85a6a79..00d3721386 100644
--- a/test/storage-manifest.test.ts
+++ b/test/storage-manifest.test.ts
@@ -5,8 +5,8 @@
* - storage_find ranks exact id, then exact alias, then most of the phrase's words; a miss says not_found and lists
* the closest cards; storage_list filters by store and status;
* - validation refuses a missing field, a command field, a stamped field, a scanner secret (in a value or an object
- * key) and every credential shape (env assignment, credential variable, URL userinfo, presigned URL), without
- * echoing the value;
+ * key) and every credential shape (env assignment, credential variable, URL userinfo, presigned URL, a value under
+ * a credential-named key), without echoing the value; a key naming where a credential lives (token_env) passes;
* - storage_update keeps the page's access frontmatter (visibility, readable_by) when it rewrites the card;
* - the write guard: put_page, remember and forget_fact into the shelf are refused storage_source_managed, a forged
* parameter does not open it, and the storage verbs still write;
@@ -20,7 +20,9 @@
* when its grant names the storage source (an ungranted bound client and an unbound stdio connection are refused);
* - derived writers skip the shelf: a dream cycle scoped to the storage source runs no phase, and enrichment
* considers no card there (no model call, no page write);
- * - a hosted shelf never becomes the sole non-default source that bare writes route to;
+ * - a hosted shelf never becomes the sole non-default source that bare writes route to, and stays out of the
+ * local-path source list autopilot fans out over and cycle_freshness reads (its cycle is always skipped);
+ * - two concurrent registers of one id land once; the other is refused storage_card_exists;
* - a fresh keyless brain, whose default source sits in the gbrain-owned content directory init creates (no Git),
* counts as repo-less: the first register creates the shelf; a Git checkout there needs storage init.
*/
@@ -131,14 +133,18 @@ describe('storage shelf on a repo-less brain', () => {
[card('a-8', { locations: [{ store: 's3', path: 'https://user:@example.invalid/x', tool: 'object', size_bytes: 1 }] }), 'storage_record_secret', 'url_userinfo'],
[card('a-9', { locations: [{ store: 's3', path: 'https://example.invalid/x?X-Amz-Signature=abc', tool: 'object', size_bytes: 1 }] }), 'storage_record_secret', 'presigned_url'],
[card('a-10', { details: { AKIAIOSFODNN7EXAMPLE: 'b2 key' } }), 'storage_record_secret', 'secret_scanner'],
+ [card('a-11', { restore: { skill: 'restore-data', args: { id: 'a-11', applicationKey: 'K005abcdef123ghijkl456mnopqr789' } } }), 'storage_record_secret', 'credential_key'],
+ [card('a-12', { details: { restic_password: 'abcdef123-correct-horse' } }), 'storage_record_secret', 'credential_key'],
];
for (const [input, code, reason] of cases) {
const r = await refusal(call('storage_register', { card: input }));
expect({ id: input.id, code: r?.code, reason: r?.reason }).toEqual({ id: input.id, code, reason });
expect(r!.message).not.toContain('abcdef123');
expect(r!.message).not.toContain('AKIAIOSFODNN7EXAMPLE');
+ expect(r!.message).not.toContain('K005abcdef123');
}
expect(await engine.executeRaw("SELECT 1 FROM pages WHERE source_id='storage'")).toEqual([]);
+ await call('storage_register', { card: card('a-13', { restore: { skill: 'restore-data', args: { id: 'a-13', key: 'backups/a-13.tar', password_file: '/etc/restic/pass', token_env: 'B2_APPLICATION_KEY' } } }) });
}, 30_000);
test('the write guard refuses put_page and remember into the shelf, forged parameters included', async () => {
@@ -209,6 +215,9 @@ describe('storage shelf on a repo-less brain', () => {
test('register never overwrites; update binds to its revision and records restores newest first', async () => {
await call('storage_register', { card: card('item-b') });
expect((await refusal(call('storage_register', { card: card('item-b') })))?.code).toBe('storage_card_exists');
+ const raced = await Promise.allSettled([call('storage_register', { card: card('item-r', { title: 'First' }) }), call('storage_register', { card: card('item-r', { title: 'Second' }) })]);
+ const lost = raced.find((r) => r.status === 'rejected') as PromiseRejectedResult | undefined;
+ expect({ landed: raced.filter((r) => r.status === 'fulfilled').length, code: lost?.reason?.code }).toEqual({ landed: 1, code: 'storage_card_exists' });
const first = await call('storage_get', { id: 'item-b' });
await call('storage_update', { id: 'item-b', expected_revision: first.revision, restored_to: '/scratch/item-b', restored_verified: true });
const stale = await refusal(call('storage_update', { id: 'item-b', expected_revision: first.revision, patch: { status: 'retired' } }));
@@ -270,6 +279,8 @@ describe('storage shelf on a repo-less brain', () => {
expect(await resolveSourceWithTier(engine, null, tmpdir())).toMatchObject({ source_id: 'default', tier: 'seed_default' });
await engine.executeRaw("INSERT INTO sources (id, name, local_path, config) VALUES ('vault-a', 'vault-a', $1, '{}'::jsonb)", [join(home, 'vault-a')]);
expect(await resolveSourceWithTier(engine, null, tmpdir())).toMatchObject({ source_id: 'vault-a', tier: 'sole_non_default' });
+ expect((await engine.listAllSources({ localPathOnly: true })).map((s) => s.id)).toEqual(['vault-a']);
+ expect((await engine.listAllSources()).map((s) => s.id)).toContain('storage');
}, 30_000);
test('a brain with a Git checkout needs storage init first, and init needs managed persistence', async () => {