Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: pre-landing review fixes
garrytan8399d2aa3553
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d38c0829a3..f9a3ccce0c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,13 +12,13 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A host that stores a file's bytes itself can now record and check them.** On an engine opened from settings, `file_register` records the files row for an object the host already wrote at the path `file_upload` would use, after reading it back and matching its size and SHA-256, and `file_verify` lists which rows still have their object or re-hashes one. Both use the engine's own `settings.storage` (local, s3 or supabase, with an optional key `prefix` so brains sharing a bucket never see each other's objects), never the process storage config. A brain the CLI or a server opens is unchanged: there both refuse `embedded_engine_required`, and neither is ever an MCP tool. Contributed by @bradgessler (#8457).+**A host that stores a file's bytes itself can now record and check them.** On an engine opened from settings, `file_register` records the files row for an object the host already wrote at the path `file_upload` would use, after reading it back and matching its size and SHA-256, and `file_verify` lists which rows still have their object or re-hashes one. Every file operation on such an engine, `file_upload` and `file_url` included, uses the engine's own `settings.storage` (a local path, or s3 with its endpoint and keys, and an optional one-segment key `prefix` so brains sharing a bucket each keep their own keys), never the process storage config. A brain the CLI or a server opens is unchanged: there both refuse `embedded_engine_required`, and neither is ever an MCP tool. Contributed by @bradgessler (#8457).### Itemized changes-- `src/core/ops/files.ts`: `file_register` accepts only the upload layout path (`<page_slug>/<filename>`, or `unsorted/<first 8 hash chars>-<filename>`), refuses `page_not_found` for a page the caller's read policy hides (member access and privacy, the private-pages hatch included), and records nothing unless the stored object matches. Registering the same path again updates its row.+- `src/core/ops/files.ts`: `file_register` accepts only the upload layout path (`<page_slug>/<filename>`, or `unsorted/<first 8 hash chars>-<filename>`), refuses `page_not_found` for a page that does not exist or that the caller's read policy hides (member access and privacy, the private-pages hatch included), and records nothing unless the stored object matches. Rows record their store's backend and prefix. Registering the same path again updates its row.- A store that does not answer is the new `store_unavailable` code, never a file reported missing and never a row recorded.-- `openEngine` refuses `engine_settings_invalid` for a `storage` setting missing a value its backend needs (so no backend falls back to a default path or environment credentials), an unknown field or a malformed prefix; `embeddedSettings()` reads back `hasSecret` instead of the secret.+- `openEngine` refuses `engine_settings_invalid` for a `storage` setting missing a value its backend needs (so no backend falls back to a default path or environment credentials), an unknown field, a backend other than local or s3, or a prefix that is not one segment; an s3 store must name its endpoint, so no AWS_* variable or ~/.aws file picks it; `embeddedSettings()` reads back `hasSecret` instead of the secret.- `src/core/storage/local.ts` and `s3.ts` stream an object's size and SHA-256 (`hashObject`); other backends hash one download.- `test/file-register.test.ts`: real PGLite engines and a real local store; the unreachable store is a closed local port.diff --git a/docs/ENGINES.md b/docs/ENGINES.mdindex 4a4429abe9..641b93071e 100644--- a/docs/ENGINES.md+++ b/docs/ENGINES.md@@ -777,7 +777,8 @@ working directory:- **Close.** `disconnect()` closes the engine for good and drops its settings, even while a reconnect is in flight(any pool that reconnect builds is ended). A later `connect()` or `reconnect()` refuses `engine_closed`; call`openEngine` again instead.-- **Files.** `settings.storage` (local, s3 or supabase; optional key `prefix`) backs `file_register` and `file_verify`.+- **Files.** Every file op uses `settings.storage` (local, or s3 with its endpoint), keys under a one-segment `prefix`;+ brains sharing a bucket each set their own.Still process-wide: file_upload's storage, transcription, the AI call log and timeouts, `GBRAIN_*` tuning variables (pool lifetime,connect and session timeouts, `GBRAIN_PREPARE`, `GBRAIN_DIRECT_POOL_SIZE`, `GBRAIN_DISABLE_DIRECT_POOL`,diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex 814abe55a0..82dee039be 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -1349,7 +1349,7 @@ More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing. | These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged. | Upload with gbrain files upload and check with gbrain files verify, or call it on an engine opened from settings. | agent | `repeat the read that failed` | 1 | no |+| An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing. | These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged. | Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload). | agent | `repeat the read that failed` | 1 | no |More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/ENGINES.md#opening-an-engine-from-settings-experimental)@@ -4507,7 +4507,7 @@ Override: reason `database_contention`: retryable, exit 1, retryable.| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| The object store an engine opened from settings names did not answer a file check (file_register, file_verify), so nothing was recorded and no file was reported missing. | A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there. | Check the store named in settings.storage (its endpoint, bucket and keys), then call again. | agent | `repeat the read that failed` | 1 | yes |+| The object store an engine opened from settings names did not answer a file check (file_register, file_verify, file_url), so nothing was recorded and no file was reported missing. | A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there. | Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again. | agent | `repeat the read that failed` | 1 | yes |More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/ENGINES.md#opening-an-engine-from-settings-experimental)diff --git a/llms-full.txt b/llms-full.txtindex 1c33fa95d6..4cc86831fb 100644--- a/llms-full.txt+++ b/llms-full.txt@@ -6224,7 +6224,8 @@ working directory:- **Close.** `disconnect()` closes the engine for good and drops its settings, even while a reconnect is in flight(any pool that reconnect builds is ended). A later `connect()` or `reconnect()` refuses `engine_closed`; call`openEngine` again instead.-- **Files.** `settings.storage` (local, s3 or supabase; optional key `prefix`) backs `file_register` and `file_verify`.+- **Files.** Every file op uses `settings.storage` (local, or s3 with its endpoint), keys under a one-segment `prefix`;+ brains sharing a bucket each set their own.Still process-wide: file_upload's storage, transcription, the AI call log and timeouts, `GBRAIN_*` tuning variables (pool lifetime,connect and session timeouts, `GBRAIN_PREPARE`, `GBRAIN_DIRECT_POOL_SIZE`, `GBRAIN_DISABLE_DIRECT_POOL`,diff --git a/src/core/embedded-engine.ts b/src/core/embedded-engine.tsindex 490aef2a1d..17feb101e5 100644--- a/src/core/embedded-engine.ts+++ b/src/core/embedded-engine.ts@@ -31,23 +31,21 @@ export interface EngineSettings {*/ai?: EngineAISettings;/**- * The object store file_register and file_verify check, as values: `local` needs `localPath`, `s3` its bucket and- * keys, `supabase` its bucket, `projectUrl` and `serviceRoleKey`. Every object key is `<prefix>/<path>` when- * `prefix` is set, so brains sharing a bucket never see each other's objects; no prefix keeps today's keys.+ * The object store every file operation on this engine uses, as values: `local` needs `localPath`, `s3` its bucket,+ * endpoint and keys (nothing is read from AWS_* variables or ~/.aws). Every object key is `<prefix>/<path>` when+ * `prefix` (one segment) is set; brains sharing a bucket each set a different prefix. No prefix keeps today's keys.*/storage?: EngineStorageSettings;}export interface EngineStorageSettings {- backend: 'local' | 's3' | 'supabase';+ backend: 'local' | 's3';bucket?: string;localPath?: string;region?: string;endpoint?: string;accessKeyId?: string;secretAccessKey?: string;- projectUrl?: string;- serviceRoleKey?: string;prefix?: string;}@@ -61,7 +59,7 @@ type DeepReadonly<T> = { readonly [K in keyof T]: DeepReadonly<T[K]> };export type EmbeddedSettingsView = DeepReadonly<Omit<EngineSettings, 'writer' | 'ai' | 'storage'> & {writer?: { id: string; lane: 'cli' | 'stdio' };ai?: { providers: Record<string, { baseUrl?: string; hasApiKey: boolean }>; models?: EngineAISettings['models'] };- storage?: Omit<EngineStorageSettings, 'secretAccessKey' | 'serviceRoleKey'> & { hasSecret: boolean };+ storage?: Omit<EngineStorageSettings, 'secretAccessKey'> & { hasSecret: boolean };}>;interface EmbeddedRecord {@@ -150,7 +148,7 @@ export function embeddedSettings(engine: object): EmbeddedSettingsView | undefin.map(([id, p]) => [id, { ...(p.baseUrl ? { baseUrl: p.baseUrl } : {}), hasApiKey: !!p.apiKey }]));const view = structuredClone({ ...rest, ...(writer ? { writer: { id: writer.id, lane: writer.lane } } : {}),...(ai ? { ai: { providers: providers!, ...(ai.models ? { models: { ...ai.models } } : {}) } } : {}),- ...(storage ? { storage: (({ secretAccessKey, serviceRoleKey, ...open }) => ({ ...open, hasSecret: !!(secretAccessKey || serviceRoleKey) }))(storage) } : {}) });+ ...(storage ? { storage: (({ secretAccessKey, ...open }) => ({ ...open, hasSecret: !!secretAccessKey }))(storage) } : {}) });const freeze = (value: unknown): void => {if (!value || typeof value !== 'object' || Object.isFrozen(value)) return;Object.freeze(value);diff --git a/src/core/engine-open.ts b/src/core/engine-open.tsindex db3fb88522..6900bd7e78 100644--- a/src/core/engine-open.ts+++ b/src/core/engine-open.ts@@ -28,20 +28,20 @@ const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;const AI_MODELS = ['chat', 'expansion', 'reranker'];const STORAGE_NEEDS: Record<string, readonly (keyof EngineStorageSettings)[]> = {- local: ['localPath'], s3: ['bucket', 'accessKeyId', 'secretAccessKey'], supabase: ['bucket', 'projectUrl', 'serviceRoleKey'],+ local: ['localPath'], s3: ['bucket', 'endpoint', 'accessKeyId', 'secretAccessKey'],};-const STORAGE_FIELDS = new Set(['backend', 'bucket', 'localPath', 'region', 'endpoint', 'accessKeyId', 'secretAccessKey', 'projectUrl', 'serviceRoleKey', 'prefix']);-const PREFIX = /^[a-z0-9][a-z0-9_-]*(\/[a-z0-9][a-z0-9_-]*)*$/i;+const STORAGE_FIELDS = new Set(['backend', 'bucket', 'localPath', 'region', 'endpoint', 'accessKeyId', 'secretAccessKey', 'prefix']);+const PREFIX = /^[a-z0-9][a-z0-9_-]*$/i;-/** Every value the backend needs is given, so no backend falls back to a default path or reads credentials from the environment. */+/** Every value the backend needs is given, so no backend falls back to a default path or reads an endpoint or credentials from the environment. */function checkStorage(storage: EngineStorageSettings): void {const needs = storage && typeof storage === 'object' ? STORAGE_NEEDS[storage.backend] : undefined;- if (!needs) throw invalid('storage.backend', "storage is { backend: 'local' | 's3' | 'supabase', ... }.");+ if (!needs) throw invalid('storage.backend', "storage is { backend: 'local' | 's3', ... }.");const extra = Object.keys(storage).find(field => !STORAGE_FIELDS.has(field));if (extra) throw invalid(`storage.${extra}`, `storage takes ${[...STORAGE_FIELDS].join(', ')}.`);const missing = needs.find(field => !text(storage[field]));if (missing) throw invalid(`storage.${missing}`, `A ${storage.backend} store needs ${needs.join(', ')}.`);- if (storage.prefix !== undefined && !PREFIX.test(storage.prefix)) throw invalid('storage.prefix', 'prefix is one or more slash-separated segments of letters, digits, - and _.');+ if (storage.prefix !== undefined && !PREFIX.test(storage.prefix)) throw invalid('storage.prefix', 'prefix is one segment of letters, digits, - and _.');}/** Names the setting that is wrong, never a key: a key's value is never part of a refusal. */diff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex 3da633c901..ad68659be1 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -302,7 +302,7 @@ export const CODES = {embedding_width_mismatch: { class: 'caller', summary: "The embedding model cannot produce the width of the brain's vector column; nothing was changed." },embedded_core_required: { class: 'caller', summary: "openEngine (gbrain/engine-open) was called without embedded_core: true, so no engine was opened.", why: 'Opening an engine from settings alone is an experimental embedding surface that is off by default; a host opts in explicitly so no existing caller changes behavior.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass embedded_core: true in the settings; the CLI, gbrain serve and mounts open brains through their own connect path and need nothing.' },engine_embedding_mismatch: { class: 'host_only', actor: 'user', summary: "An engine opened from settings names an embedding model or width that differs from the one its database already records, so initSchema changed nothing.", why: 'The vector column has one width and the brain one embedding model; seeding another over it would make every stored and future embedding unreadable. Changing it is an explicit re-embed the user decides on.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Pass the database's recorded model and width in settings.embedding, or read the recorded identity with gbrain migrate embeddings --status on the brain host and preview a migration from there.", fix: { argv: ['gbrain', 'migrate', 'embeddings', '--status', '--json'], consent: [], actor: 'agent', requires_exclusive: false, why: "Read-only: reports the brain's recorded embedding model and width and what a migration would change." } },- embedded_engine_required: { class: 'caller', summary: "An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing.", why: 'These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Upload with gbrain files upload and check with gbrain files verify, or call it on an engine opened from settings.' },+ embedded_engine_required: { class: 'caller', summary: "An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing.", why: 'These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload).' },engine_closed: { class: 'caller', summary: 'An engine opened from settings was disconnected, which closes it for good, so connect() and reconnect() refuse and nothing was reopened.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call openEngine again with the settings to open a new engine.' },engine_host_missing: { class: 'caller', summary: "An engine opened from settings has no host identity, so nothing that needs one ran and no host.json was created.", why: 'An embedded engine publishes and claims work under the host id its host gave at open; minting a host.json under the home directory instead would tie its writes to whatever process happens to share that directory.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass host: { id: <uuid> } to openEngine; keep the same id for the same host across restarts.' },engine_scope_missing: { class: 'caller', summary: "An engine opened from settings made an AI call (embed, chat, expansion or rerank) through a provider its settings.ai does not name, or a call ran outside any engine scope in a process holding such engines, so nothing was sent.", why: "An embedded engine's AI calls reach only the providers and keys its host passed as values. Falling back to a process.env key, ~/.gbrain/config.json or the process-wide gateway would spend another brain's key and send this brain's text where its host never pointed it.", docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass the provider in settings.ai.providers (apiKey, and baseUrl when it is not the default) to openEngine, or point settings.ai.models (or settings.embedding) at a provider the engine has.' },@@ -733,7 +733,7 @@ export const CODES = {stable_channel_unresolved: { class: 'unavailable', reasons: ['unreadable', 'no_stable', 'no_release_tag'], summary: "`gbrain upgrade --channel stable` (or `self_upgrade.channel stable`) could not name the stable release: the repository's tags could not be read, no release has been promoted to stable yet, or the `stable` tag does not share a commit with a `v<version>` release tag. Nothing was changed.", why: "The stable channel installs only a release a maintainer promoted after the production-shaped soak passed on its exact tree; without that answer it installs nothing rather than guess.", docs: 'docs/guides/upgrades-auto-update.md#stable-channel', suggestion: 'Retry later, or upgrade on the latest channel with `gbrain upgrade --channel latest`.' },stale_unavailable: { class: 'unavailable', summary: "No complete, current authorized shared-skills view is available; cached shared skills stay unused.", docs: 'docs/guides/shared-brain-skills.md#troubleshoot-leave-and-recover' },state_mismatch: { class: 'caller', summary: "Google connect credential error: state mismatch.", docs: 'docs/guides/google-connect.md#troubleshooting' },- store_unavailable: { class: 'retryable', summary: "The object store an engine opened from settings names did not answer a file check (file_register, file_verify), so nothing was recorded and no file was reported missing.", why: 'A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Check the store named in settings.storage (its endpoint, bucket and keys), then call again." },+ store_unavailable: { class: 'retryable', summary: "The object store an engine opened from settings names did not answer a file check (file_register, file_verify, file_url), so nothing was recorded and no file was reported missing.", why: 'A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again." },storage_error: { class: 'server', summary: "Reading or writing durable storage failed.",reasons: ['database_contention'], reason_overrides: { database_contention: { class: 'retryable' } },why: "Usually a dropped connection or a failed statement; the same error after a rerun is not transient. `reason: database_contention`: a write admission that deadlocks, serialization failures, lock timeouts or statement timeouts kept refusing for its whole retry budget; no receipt was confirmed, so the same request_id is safe to retry once the contention clears. `detail: persistence_mode_unknown` (#6482): the persistence_brain singleton is missing, duplicated or carries a non-boolean `enabled`, so gbrain cannot tell whether the brain is managed and refuses every write, sync, connector run and activation step that depends on it rather than route a managed brain through the unmanaged writer; only a well-formed `enabled: false` row selects the unmanaged path.",diff --git a/src/core/ops/files.ts b/src/core/ops/files.tsindex 8854328f83..3f39755500 100644--- a/src/core/ops/files.ts+++ b/src/core/ops/files.ts@@ -105,7 +105,8 @@ const file_upload: Operation = {// read (file_url, files verify, restore) would trust a phantom object.// Typed error BEFORE any insert; git-tracked small files have their own// lane (`gbrain files upload-raw --page <slug>`).- if (!ctx.config.storage) {+ const embedded = isEmbeddedEngine(ctx.engine) ? await embeddedFileStore(ctx, 'file_upload') : null;+ if (!embedded && !ctx.config.storage) {const slugOk = pageSlug !== null && /^[a-z0-9][a-z0-9._/-]*$/i.test(pageSlug);// #5963: the git-tracked lane is refused on a managed worktree, so it is not offered there.if (slugOk && await (await import('../raw-file-upload.ts')).rawUploadRefusedFor(ctx.engine, pageSlug!, ctx.sourceId ?? 'default')) throw opError('storage_error',@@ -120,7 +121,7 @@ const file_upload: Operation = {);}const { createStorage } = await import('../storage.ts');- const storage = await createStorage(ctx.config.storage as any);+ const storage = embedded?.storage ?? await createStorage(ctx.config.storage as any);const { sqlQueryForEngine, executeRawJsonb, FILES_METADATA_MERGE_SQL } = await import('../sql-query.ts');const sql = sqlQueryForEngine(ctx.engine);@@ -159,7 +160,7 @@ const file_upload: Operation = {mime_type = EXCLUDED.mime_type,${FILES_METADATA_MERGE_SQL}`,[pageSlug, filename, storagePath, mimeType, stat.size, hash],- [{ storage: (ctx.config.storage as { backend: string }).backend }],+ [embedded?.metadata ?? { storage: (ctx.config.storage as { backend: string }).backend }],);} catch (dbErr) {// Rollback: clean up storage if DB write failed@@ -194,7 +195,8 @@ const file_url: Operation = {// #4302: resolve a REAL URL from the backend, after confirming the object// is actually there — the old `gbrain:files/<path>` placeholder pointed// at nothing and hid rows whose bytes had vanished.- if (!ctx.config.storage) {+ const embedded = isEmbeddedEngine(ctx.engine);+ if (!embedded && !ctx.config.storage) {throw opError('storage_error',`No storage backend configured — cannot produce a URL for ${p.storage_path}.`,@@ -202,8 +204,11 @@ const file_url: Operation = {);}const { createStorage } = await import('../storage.ts');- const storage = await createStorage(ctx.config.storage as any);- const present = await storage.exists(rows[0].storage_path as string).catch(() => false);+ const storage = embedded ? await embeddedStorage(ctx, 'file_url') : await createStorage(ctx.config.storage as any);+ const present = await storage.exists(rows[0].storage_path as string).catch(error => {+ if (embedded) throw storeUnavailable('file_url', error);+ return false;+ });if (!present) {throw opError('storage_error',@@ -215,18 +220,24 @@ const file_url: Operation = {},};-/** file_register and file_verify serve a host that stores bytes itself: they run only on an engine opened from settings, against its settings.storage. */+/** file_register and file_verify serve a host that stores bytes itself: they run only on an engine opened from settings, against its settings.storage, which every file op on such an engine uses. */async function embeddedStorage(ctx: OperationContext, op: string) {if (!isEmbeddedEngine(ctx.engine)) throw opError('embedded_engine_required', `${op} runs only on an engine an embedding host opened from settings (embedded_core: true).`,- 'On a brain the CLI opens, upload the file with gbrain files upload and check it with gbrain files verify.');+ 'A brain the CLI or a server opens stores files only through file_upload (gbrain files upload), so there is nothing to register or check this way.');const settings = embeddedSettingsRecord(ctx.engine)?.storage;if (!settings) throw opError('storage_error', `This engine was opened with no settings.storage, so ${op} cannot check any stored object.`,- 'Open the engine with storage (local | s3 | supabase, with its values) in its settings; the process storage config is never used for an engine opened from settings. Nothing was recorded.');+ 'Open the engine with storage (local or s3, with its values) in its settings; the process storage config is never used for an engine opened from settings. Nothing was recorded.');const { createStorage, prefixedStorage } = await import('../storage.ts');const { prefix, ...config } = settings;try { return prefixedStorage(await createStorage({ bucket: '', ...config }), prefix); } catch (error) { throw storeUnavailable(op, error); }}+async function embeddedFileStore(ctx: OperationContext, op: string) {+ const storage = await embeddedStorage(ctx, op);+ const { backend, prefix } = embeddedSettingsRecord(ctx.engine)!.storage!;+ return { storage, metadata: { storage: backend, ...(prefix ? { prefix } : {}) } };+}+function storeUnavailable(op: string, error: unknown) {return opError('store_unavailable', `${op} could not ask the storage backend: ${error instanceof Error ? error.message : String(error)}`,'The backend did not answer, so nothing is known about the object and nothing was recorded or reported missing. Check the backend with gbrain doctor --json, then call again.');@@ -239,8 +250,8 @@ function uploadLayoutPath(pageSlug: string | null, filename: string, hash: strinasync function pageReadable(ctx: OperationContext, slug: string): Promise<boolean> {const rows = await ctx.engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM pages fp WHERE fp.slug = $1- AND ${await resolveExcludePrivatePages(ctx.engine, ctx.remote) ? `NOT ${privatePagesFilterForCaller('fp')}` : 'FALSE'}`, [slug]);- return rows[0]!.n === 0;+ AND ${await resolveExcludePrivatePages(ctx.engine, ctx.remote) ? privatePagesFilterForCaller('fp') : 'TRUE'}`, [slug]);+ return rows[0]!.n > 0;}const file_register: Operation = {@@ -261,7 +272,7 @@ const file_register: Operation = {localOnly: true,cliOnly: { argv: ['gbrain', 'files', 'upload', '<file>', '--page', '<page_slug>'] },handler: async (ctx, p) => {- const storage = await embeddedStorage(ctx, 'file_register');+ const { storage, metadata } = await embeddedFileStore(ctx, 'file_register');const filename = p.filename as string, hash = p.content_hash as string, size = p.size_bytes as number;const pageSlug = (p.page_slug as string | undefined) || null;validateFilename(filename);@@ -281,7 +292,7 @@ const file_register: Operation = {catch (error) { throw storeUnavailable('file_register', error); }if (!stored) throw opError('storage_error', `The storage backend has no object at ${storagePath}, so no row was recorded.`, 'Store the bytes at that path first, then register it.');if (stored.size !== size || stored.sha256 !== hash) throw opError('storage_error', `The object at ${storagePath} does not match size_bytes and content_hash, so no row was recorded.`,- `The stored object is ${stored.size} bytes with SHA-256 ${stored.sha256}. Register the values of the bytes actually stored.`);+ 'Register the size and SHA-256 of the bytes actually stored at that path.');const { executeRawJsonb, FILES_METADATA_MERGE_SQL } = await import('../sql-query.ts');await executeRawJsonb(ctx.engine,`INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)@@ -292,7 +303,7 @@ const file_register: Operation = {mime_type = EXCLUDED.mime_type,${FILES_METADATA_MERGE_SQL}`,[pageSlug, filename, storagePath, (p.mime_type as string | undefined) ?? null, size, hash],- [{ storage: embeddedSettingsRecord(ctx.engine)!.storage!.backend }]);+ [metadata]);return { status: 'registered', storage_path: storagePath, size_bytes: size };},};diff --git a/test/file-register.test.ts b/test/file-register.test.tsindex 40162d41d5..9bff26b217 100644--- a/test/file-register.test.ts+++ b/test/file-register.test.ts@@ -4,7 +4,7 @@*/import { afterAll, afterEach, beforeAll, describe, expect, test } from 'bun:test';import { createHash, randomUUID } from 'node:crypto';-import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs';+import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';import { dirname, join } from 'node:path';import { embeddedSettings, openEngine, type EngineSettings, type EngineStorageSettings } from '../src/core/engine-open.ts';@@ -34,6 +34,7 @@ function store(path: string, bytes: Buffer): void {mkdirSync(dirname(join(storageRoot, path)), { recursive: true });writeFileSync(join(storageRoot, path), bytes);}+const addPage = (engine: BrainEngine, slug: string) => importFromContent(engine, slug, ['---', 'type: note', `title: ${slug}`, '---', `# ${slug}`, '', 'Plan body.'].join('\n'));const fileRows = async (engine: BrainEngine) => Number((await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM files'))[0]!.n);const engines: BrainEngine[] = [];@@ -53,6 +54,7 @@ afterAll(() => rmSync(root, { recursive: true, force: true }));describe('file_register', () => {test('records one row for an object stored at the file_upload path, and registering again updates it', async () => {const engine = await embedded();+ await addPage(engine, 'notes/plan');const bytes = Buffer.from('report bytes');store('notes/plan/report.pdf', bytes);const params = { storage_path: 'notes/plan/report.pdf', filename: 'report.pdf', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length, mime_type: 'application/pdf' };@@ -62,6 +64,20 @@ describe('file_register', () => {await run('file_register', ctxFor(engine), { ...params, content_hash: sha(next), size_bytes: next.length });const rows = await engine.executeRaw<{ page_slug: string; content_hash: string; size_bytes: unknown }>('SELECT page_slug, content_hash, size_bytes FROM files');expect(rows.map(r => ({ ...r, size_bytes: Number(r.size_bytes) }))).toEqual([{ page_slug: 'notes/plan', content_hash: sha(next), size_bytes: next.length }]);+ expect(await run('file_register', ctxFor(engine, { dryRun: true }), { ...params, filename: 'other.pdf', storage_path: 'notes/plan/other.pdf' }))+ .toEqual({ dry_run: true, action: 'file_register', storage_path: 'notes/plan/other.pdf' });+ expect(await fileRows(engine)).toBe(1);+ });++ test('refuses a page that does not exist and an object it cannot read back, recording nothing', async () => {+ const engine = await embedded();+ const bytes = Buffer.from('orphan');+ store('notes/none/o.txt', bytes);+ expect((await failure(() => run('file_register', ctxFor(engine), { storage_path: 'notes/none/o.txt', filename: 'o.txt', page_slug: 'notes/none', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('page_not_found');+ await addPage(engine, 'notes/plan');+ mkdirSync(join(storageRoot, 'notes/plan/dir.txt'), { recursive: true });+ expect((await failure(() => run('file_register', ctxFor(engine), { storage_path: 'notes/plan/dir.txt', filename: 'dir.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('store_unavailable');+ expect(await fileRows(engine)).toBe(0);});test('refuses a path other than the file_upload layout, a missing or different object and bad inputs, recording nothing', async () => {@@ -100,6 +116,7 @@ describe('file_register', () => {test('two brains sharing one store under different prefixes see only their own objects', async () => {const a = await embedded({ storage: { ...storage, prefix: 'brain-a' } });const b = await embedded({ storage: { ...storage, prefix: 'brain-b' } });+ await Promise.all([addPage(a, 'notes/plan'), addPage(b, 'notes/plan')]);const bytes = Buffer.from('brain a bytes');store('brain-a/notes/plan/a.txt', bytes);const params = { storage_path: 'notes/plan/a.txt', filename: 'a.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length };@@ -107,6 +124,22 @@ describe('file_register', () => {expect((await failure(() => run('file_register', ctxFor(b), params)))?.code).toBe('storage_error');expect(await fileRows(b)).toBe(0);expect(await run('file_verify', ctxFor(a), { storage_path: 'notes/plan/a.txt' })).toEqual({ storage_path: 'notes/plan/a.txt', present: true, size_matches: true, hash_matches: true });+ const [{ metadata }] = await a.executeRaw<{ metadata: unknown }>('SELECT metadata FROM files');+ expect(metadata).toEqual({ storage: 'local', prefix: 'brain-a' });+ });++ test('file_upload and file_url on an engine opened from settings use its store and prefix, never the process store', async () => {+ const engine = await embedded({ storage: { ...storage, prefix: 'brain-a' } });+ await addPage(engine, 'notes/plan');+ const source = join(root, 'upload.txt');+ writeFileSync(source, 'uploaded bytes');+ const processStore = { backend: 'local', bucket: 'test', localPath: join(root, 'process-store') };+ const ctx = ctxFor(engine, { config: { storage: processStore } } as never);+ expect(await run('file_upload', ctx, { path: source, page_slug: 'notes/plan' })).toMatchObject({ status: 'uploaded', storage_path: 'notes/plan/upload.txt' });+ expect(readFileSync(join(storageRoot, 'brain-a/notes/plan/upload.txt'), 'utf8')).toBe('uploaded bytes');+ expect(existsSync(join(root, 'process-store'))).toBe(false);+ expect(await run('file_url', ctx, { storage_path: 'notes/plan/upload.txt' })).toEqual({ storage_path: 'notes/plan/upload.txt', url: `file://${join(storageRoot, 'brain-a/notes/plan/upload.txt')}` });+ expect(await run('file_verify', ctx, { storage_path: 'notes/plan/upload.txt' })).toMatchObject({ present: true, hash_matches: true });});test('a store that cannot be reached is store_unavailable and records nothing', async () => {@@ -124,8 +157,8 @@ describe('settings.storage', () => {test('openEngine refuses a store missing a value it needs, an unknown backend, field or prefix, and the view hides its secret', async () => {const cases: Array<[unknown, string]> = [[{ backend: 'nope' }, 'storage.backend'], [{ backend: 'local' }, 'storage.localPath'],- [{ backend: 's3', bucket: 'b', accessKeyId: 'k' }, 'storage.secretAccessKey'], [{ backend: 'supabase', bucket: 'b', projectUrl: 'https://x.example' }, 'storage.serviceRoleKey'],- [{ ...storage, region2: 'x' }, 'storage.region2'], [{ ...storage, prefix: '../a' }, 'storage.prefix'], [{ ...storage, prefix: 'a/' }, 'storage.prefix'],+ [{ backend: 'supabase', bucket: 'b' }, 'storage.backend'], [{ ...unreachable, endpoint: undefined }, 'storage.endpoint'], [{ ...unreachable, secretAccessKey: '' }, 'storage.secretAccessKey'],+ [{ ...storage, region2: 'x' }, 'storage.region2'], [{ ...storage, prefix: '../a' }, 'storage.prefix'], [{ ...storage, prefix: 'a/b' }, 'storage.prefix'],];for (const [given, setting] of cases) {const error = await failure(() => openEngine(settings({ storage: given as EngineStorageSettings })));@@ -145,11 +178,35 @@ describe('hashing a stored object', () => {const s3 = new S3Storage({ backend: 's3', bucket: 'test' }, client as never);expect(await s3.hashObject('a/b.txt')).toEqual({ size: 23, sha256: sha(Buffer.concat(parts)) });});++ test('a backend without hashObject is hashed from one download', async () => {+ const { hashStoredObject } = await import('../src/core/storage.ts');+ const bytes = Buffer.from('downloaded');+ expect(await hashStoredObject({ download: async () => bytes } as never, 'a.txt')).toEqual({ size: bytes.length, sha256: sha(bytes) });+ });++ test('an s3 store sends every request to its settings endpoint, never one named by AWS_* variables', async () => {+ const hits = { settings: 0, env: 0 };+ const settingsServer = Bun.serve({ port: 0, fetch: () => { hits.settings++; return new Response(null, { status: 404 }); } });+ const envServer = Bun.serve({ port: 0, fetch: () => { hits.env++; return new Response(null, { status: 200 }); } });+ try {+ await withEnv({ AWS_ENDPOINT_URL_S3: `http://127.0.0.1:${envServer.port}`, AWS_ENDPOINT_URL: `http://127.0.0.1:${envServer.port}` }, async () => {+ const engine = await embedded({ storage: { ...unreachable, endpoint: `http://127.0.0.1:${settingsServer.port}` } });+ const bytes = Buffer.from('s3');+ const error = await failure(() => run('file_register', ctxFor(engine), { storage_path: `unsorted/${sha(bytes).slice(0, 8)}-s.txt`, filename: 's.txt', content_hash: sha(bytes), size_bytes: 2 }));+ expect([error?.code, hits.env, hits.settings > 0]).toEqual(['storage_error', 0, true]);+ });+ } finally {+ settingsServer.stop(true);+ envServer.stop(true);+ }+ });});describe('file_verify', () => {test('lists presence per row, reads one object back, and notices bytes changed underneath', async () => {const engine = await embedded();+ await addPage(engine, 'notes/plan');const a = Buffer.from('alpha'), b = Buffer.from('bravo');store('notes/plan/a.txt', a);store('notes/plan/b.txt', b);diff --git a/test/fixtures/goldens/exports/types.json b/test/fixtures/goldens/exports/types.jsonindex e321e36a54..9c5b5ce061 100644--- a/test/fixtures/goldens/exports/types.json+++ b/test/fixtures/goldens/exports/types.json@@ -4433,7 +4433,7 @@"kind": "type","name": "EmbeddedSettingsView","signature": [- "{ readonly pglite?: DeepReadonly<{ shared_buffers?: string; } | undefined>; readonly host: DeepReadonly<{ id: string; }>; readonly database: DeepReadonly<{ url?: string; directUrl?: string; path?: string; }>; readonly embedding?: DeepReadonly<{ model: string; dimensions: number; } | undefined>; readonly embedded_core: DeepReadonly<boolean>; readonly engine: DeepReadonly<\"postgres\" | \"pglite\">; readonly pool?: DeepReadonly<{ max?: number; } | undefined>; readonly keyword_statement_timeout_ms?: number | undefined; readonly writer?: DeepReadonly<{ id: string; lane: \"cli\" | \"stdio\"; } | undefined>; readonly ai?: DeepReadonly<{ providers: Record<string, { baseUrl?: string; hasApiKey: boolean; }>; models?: EngineAISettings[\"models\"]; } | undefined>; readonly storage?: DeepReadonly<(Omit<EngineStorageSettings, \"secretAccessKey\" | \"serviceRoleKey\"> & { hasSecret: boolean; }) | undefined>; }"+ "{ readonly pglite?: DeepReadonly<{ shared_buffers?: string; } | undefined>; readonly host: DeepReadonly<{ id: string; }>; readonly database: DeepReadonly<{ url?: string; directUrl?: string; path?: string; }>; readonly embedding?: DeepReadonly<{ model: string; dimensions: number; } | undefined>; readonly embedded_core: DeepReadonly<boolean>; readonly engine: DeepReadonly<\"postgres\" | \"pglite\">; readonly pool?: DeepReadonly<{ max?: number; } | undefined>; readonly keyword_statement_timeout_ms?: number | undefined; readonly writer?: DeepReadonly<{ id: string; lane: \"cli\" | \"stdio\"; } | undefined>; readonly ai?: DeepReadonly<{ providers: Record<string, { baseUrl?: string; hasApiKey: boolean; }>; models?: EngineAISettings[\"models\"]; } | undefined>; readonly storage?: DeepReadonly<(Omit<EngineStorageSettings, \"secretAccessKey\"> & { hasSecret: boolean; }) | undefined>; }"]},{@@ -4465,15 +4465,13 @@"kind": "interface","members": ["accessKeyId?: string | undefined",- "backend: \"local\" | \"s3\" | \"supabase\"",+ "backend: \"local\" | \"s3\"","bucket?: string | undefined","endpoint?: string | undefined","localPath?: string | undefined","prefix?: string | undefined",- "projectUrl?: string | undefined","region?: string | undefined",- "secretAccessKey?: string | undefined",- "serviceRoleKey?: string | undefined"+ "secretAccessKey?: string | undefined"],"name": "EngineStorageSettings"},