Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: pre-landing review fixes

garrytan8399d2aa3553
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d38c0829a3..f9a3ccce0c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,13 +12,13 @@ identifiers and attribution are available in the pre-removal Git revision
## [Unreleased]
-**A host that stores a file's bytes itself can now record and check them.** On an engine opened from settings, `file_register` records the files row for an object the host already wrote at the path `file_upload` would use, after reading it back and matching its size and SHA-256, and `file_verify` lists which rows still have their object or re-hashes one. Both use the engine's own `settings.storage` (local, s3 or supabase, with an optional key `prefix` so brains sharing a bucket never see each other's objects), never the process storage config. A brain the CLI or a server opens is unchanged: there both refuse `embedded_engine_required`, and neither is ever an MCP tool. Contributed by @bradgessler (#8457).
+**A host that stores a file's bytes itself can now record and check them.** On an engine opened from settings, `file_register` records the files row for an object the host already wrote at the path `file_upload` would use, after reading it back and matching its size and SHA-256, and `file_verify` lists which rows still have their object or re-hashes one. Every file operation on such an engine, `file_upload` and `file_url` included, uses the engine's own `settings.storage` (a local path, or s3 with its endpoint and keys, and an optional one-segment key `prefix` so brains sharing a bucket each keep their own keys), never the process storage config. A brain the CLI or a server opens is unchanged: there both refuse `embedded_engine_required`, and neither is ever an MCP tool. Contributed by @bradgessler (#8457).
### Itemized changes
-- `src/core/ops/files.ts`: `file_register` accepts only the upload layout path (`<page_slug>/<filename>`, or `unsorted/<first 8 hash chars>-<filename>`), refuses `page_not_found` for a page the caller's read policy hides (member access and privacy, the private-pages hatch included), and records nothing unless the stored object matches. Registering the same path again updates its row.
+- `src/core/ops/files.ts`: `file_register` accepts only the upload layout path (`<page_slug>/<filename>`, or `unsorted/<first 8 hash chars>-<filename>`), refuses `page_not_found` for a page that does not exist or that the caller's read policy hides (member access and privacy, the private-pages hatch included), and records nothing unless the stored object matches. Rows record their store's backend and prefix. Registering the same path again updates its row.
- A store that does not answer is the new `store_unavailable` code, never a file reported missing and never a row recorded.
-- `openEngine` refuses `engine_settings_invalid` for a `storage` setting missing a value its backend needs (so no backend falls back to a default path or environment credentials), an unknown field or a malformed prefix; `embeddedSettings()` reads back `hasSecret` instead of the secret.
+- `openEngine` refuses `engine_settings_invalid` for a `storage` setting missing a value its backend needs (so no backend falls back to a default path or environment credentials), an unknown field, a backend other than local or s3, or a prefix that is not one segment; an s3 store must name its endpoint, so no AWS_* variable or ~/.aws file picks it; `embeddedSettings()` reads back `hasSecret` instead of the secret.
- `src/core/storage/local.ts` and `s3.ts` stream an object's size and SHA-256 (`hashObject`); other backends hash one download.
- `test/file-register.test.ts`: real PGLite engines and a real local store; the unreachable store is a closed local port.
diff --git a/docs/ENGINES.md b/docs/ENGINES.md
index 4a4429abe9..641b93071e 100644
--- a/docs/ENGINES.md
+++ b/docs/ENGINES.md
@@ -777,7 +777,8 @@ working directory:
- **Close.** `disconnect()` closes the engine for good and drops its settings, even while a reconnect is in flight
(any pool that reconnect builds is ended). A later `connect()` or `reconnect()` refuses `engine_closed`; call
`openEngine` again instead.
-- **Files.** `settings.storage` (local, s3 or supabase; optional key `prefix`) backs `file_register` and `file_verify`.
+- **Files.** Every file op uses `settings.storage` (local, or s3 with its endpoint), keys under a one-segment `prefix`;
+ brains sharing a bucket each set their own.
Still process-wide: file_upload's storage, transcription, the AI call log and timeouts, `GBRAIN_*` tuning variables (pool lifetime,
connect and session timeouts, `GBRAIN_PREPARE`, `GBRAIN_DIRECT_POOL_SIZE`, `GBRAIN_DISABLE_DIRECT_POOL`,
diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.md
index 814abe55a0..82dee039be 100644
--- a/docs/guides/error-codes.md
+++ b/docs/guides/error-codes.md
@@ -1349,7 +1349,7 @@ More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing. | These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged. | Upload with gbrain files upload and check with gbrain files verify, or call it on an engine opened from settings. | agent | `repeat the read that failed` | 1 | no |
+| An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing. | These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged. | Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload). | agent | `repeat the read that failed` | 1 | no |
More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/ENGINES.md#opening-an-engine-from-settings-experimental)
@@ -4507,7 +4507,7 @@ Override: reason `database_contention`: retryable, exit 1, retryable.
| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable |
|---|---|---|---|---|---|---|
-| The object store an engine opened from settings names did not answer a file check (file_register, file_verify), so nothing was recorded and no file was reported missing. | A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there. | Check the store named in settings.storage (its endpoint, bucket and keys), then call again. | agent | `repeat the read that failed` | 1 | yes |
+| The object store an engine opened from settings names did not answer a file check (file_register, file_verify, file_url), so nothing was recorded and no file was reported missing. | A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there. | Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again. | agent | `repeat the read that failed` | 1 | yes |
More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/ENGINES.md#opening-an-engine-from-settings-experimental)
diff --git a/llms-full.txt b/llms-full.txt
index 1c33fa95d6..4cc86831fb 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -6224,7 +6224,8 @@ working directory:
- **Close.** `disconnect()` closes the engine for good and drops its settings, even while a reconnect is in flight
(any pool that reconnect builds is ended). A later `connect()` or `reconnect()` refuses `engine_closed`; call
`openEngine` again instead.
-- **Files.** `settings.storage` (local, s3 or supabase; optional key `prefix`) backs `file_register` and `file_verify`.
+- **Files.** Every file op uses `settings.storage` (local, or s3 with its endpoint), keys under a one-segment `prefix`;
+ brains sharing a bucket each set their own.
Still process-wide: file_upload's storage, transcription, the AI call log and timeouts, `GBRAIN_*` tuning variables (pool lifetime,
connect and session timeouts, `GBRAIN_PREPARE`, `GBRAIN_DIRECT_POOL_SIZE`, `GBRAIN_DISABLE_DIRECT_POOL`,
diff --git a/src/core/embedded-engine.ts b/src/core/embedded-engine.ts
index 490aef2a1d..17feb101e5 100644
--- a/src/core/embedded-engine.ts
+++ b/src/core/embedded-engine.ts
@@ -31,23 +31,21 @@ export interface EngineSettings {
*/
ai?: EngineAISettings;
/**
- * The object store file_register and file_verify check, as values: `local` needs `localPath`, `s3` its bucket and
- * keys, `supabase` its bucket, `projectUrl` and `serviceRoleKey`. Every object key is `<prefix>/<path>` when
- * `prefix` is set, so brains sharing a bucket never see each other's objects; no prefix keeps today's keys.
+ * The object store every file operation on this engine uses, as values: `local` needs `localPath`, `s3` its bucket,
+ * endpoint and keys (nothing is read from AWS_* variables or ~/.aws). Every object key is `<prefix>/<path>` when
+ * `prefix` (one segment) is set; brains sharing a bucket each set a different prefix. No prefix keeps today's keys.
*/
storage?: EngineStorageSettings;
}
export interface EngineStorageSettings {
- backend: 'local' | 's3' | 'supabase';
+ backend: 'local' | 's3';
bucket?: string;
localPath?: string;
region?: string;
endpoint?: string;
accessKeyId?: string;
secretAccessKey?: string;
- projectUrl?: string;
- serviceRoleKey?: string;
prefix?: string;
}
@@ -61,7 +59,7 @@ type DeepReadonly<T> = { readonly [K in keyof T]: DeepReadonly<T[K]> };
export type EmbeddedSettingsView = DeepReadonly<Omit<EngineSettings, 'writer' | 'ai' | 'storage'> & {
writer?: { id: string; lane: 'cli' | 'stdio' };
ai?: { providers: Record<string, { baseUrl?: string; hasApiKey: boolean }>; models?: EngineAISettings['models'] };
- storage?: Omit<EngineStorageSettings, 'secretAccessKey' | 'serviceRoleKey'> & { hasSecret: boolean };
+ storage?: Omit<EngineStorageSettings, 'secretAccessKey'> & { hasSecret: boolean };
}>;
interface EmbeddedRecord {
@@ -150,7 +148,7 @@ export function embeddedSettings(engine: object): EmbeddedSettingsView | undefin
.map(([id, p]) => [id, { ...(p.baseUrl ? { baseUrl: p.baseUrl } : {}), hasApiKey: !!p.apiKey }]));
const view = structuredClone({ ...rest, ...(writer ? { writer: { id: writer.id, lane: writer.lane } } : {}),
...(ai ? { ai: { providers: providers!, ...(ai.models ? { models: { ...ai.models } } : {}) } } : {}),
- ...(storage ? { storage: (({ secretAccessKey, serviceRoleKey, ...open }) => ({ ...open, hasSecret: !!(secretAccessKey || serviceRoleKey) }))(storage) } : {}) });
+ ...(storage ? { storage: (({ secretAccessKey, ...open }) => ({ ...open, hasSecret: !!secretAccessKey }))(storage) } : {}) });
const freeze = (value: unknown): void => {
if (!value || typeof value !== 'object' || Object.isFrozen(value)) return;
Object.freeze(value);
diff --git a/src/core/engine-open.ts b/src/core/engine-open.ts
index db3fb88522..6900bd7e78 100644
--- a/src/core/engine-open.ts
+++ b/src/core/engine-open.ts
@@ -28,20 +28,20 @@ const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const AI_MODELS = ['chat', 'expansion', 'reranker'];
const STORAGE_NEEDS: Record<string, readonly (keyof EngineStorageSettings)[]> = {
- local: ['localPath'], s3: ['bucket', 'accessKeyId', 'secretAccessKey'], supabase: ['bucket', 'projectUrl', 'serviceRoleKey'],
+ local: ['localPath'], s3: ['bucket', 'endpoint', 'accessKeyId', 'secretAccessKey'],
};
-const STORAGE_FIELDS = new Set(['backend', 'bucket', 'localPath', 'region', 'endpoint', 'accessKeyId', 'secretAccessKey', 'projectUrl', 'serviceRoleKey', 'prefix']);
-const PREFIX = /^[a-z0-9][a-z0-9_-]*(\/[a-z0-9][a-z0-9_-]*)*$/i;
+const STORAGE_FIELDS = new Set(['backend', 'bucket', 'localPath', 'region', 'endpoint', 'accessKeyId', 'secretAccessKey', 'prefix']);
+const PREFIX = /^[a-z0-9][a-z0-9_-]*$/i;
-/** Every value the backend needs is given, so no backend falls back to a default path or reads credentials from the environment. */
+/** Every value the backend needs is given, so no backend falls back to a default path or reads an endpoint or credentials from the environment. */
function checkStorage(storage: EngineStorageSettings): void {
const needs = storage && typeof storage === 'object' ? STORAGE_NEEDS[storage.backend] : undefined;
- if (!needs) throw invalid('storage.backend', "storage is { backend: 'local' | 's3' | 'supabase', ... }.");
+ if (!needs) throw invalid('storage.backend', "storage is { backend: 'local' | 's3', ... }.");
const extra = Object.keys(storage).find(field => !STORAGE_FIELDS.has(field));
if (extra) throw invalid(`storage.${extra}`, `storage takes ${[...STORAGE_FIELDS].join(', ')}.`);
const missing = needs.find(field => !text(storage[field]));
if (missing) throw invalid(`storage.${missing}`, `A ${storage.backend} store needs ${needs.join(', ')}.`);
- if (storage.prefix !== undefined && !PREFIX.test(storage.prefix)) throw invalid('storage.prefix', 'prefix is one or more slash-separated segments of letters, digits, - and _.');
+ if (storage.prefix !== undefined && !PREFIX.test(storage.prefix)) throw invalid('storage.prefix', 'prefix is one segment of letters, digits, - and _.');
}
/** Names the setting that is wrong, never a key: a key's value is never part of a refusal. */
diff --git a/src/core/error-registry.ts b/src/core/error-registry.ts
index 3da633c901..ad68659be1 100644
--- a/src/core/error-registry.ts
+++ b/src/core/error-registry.ts
@@ -302,7 +302,7 @@ export const CODES = {
embedding_width_mismatch: { class: 'caller', summary: "The embedding model cannot produce the width of the brain's vector column; nothing was changed." },
embedded_core_required: { class: 'caller', summary: "openEngine (gbrain/engine-open) was called without embedded_core: true, so no engine was opened.", why: 'Opening an engine from settings alone is an experimental embedding surface that is off by default; a host opts in explicitly so no existing caller changes behavior.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass embedded_core: true in the settings; the CLI, gbrain serve and mounts open brains through their own connect path and need nothing.' },
engine_embedding_mismatch: { class: 'host_only', actor: 'user', summary: "An engine opened from settings names an embedding model or width that differs from the one its database already records, so initSchema changed nothing.", why: 'The vector column has one width and the brain one embedding model; seeding another over it would make every stored and future embedding unreadable. Changing it is an explicit re-embed the user decides on.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Pass the database's recorded model and width in settings.embedding, or read the recorded identity with gbrain migrate embeddings --status on the brain host and preview a migration from there.", fix: { argv: ['gbrain', 'migrate', 'embeddings', '--status', '--json'], consent: [], actor: 'agent', requires_exclusive: false, why: "Read-only: reports the brain's recorded embedding model and width and what a migration would change." } },
- embedded_engine_required: { class: 'caller', summary: "An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing.", why: 'These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Upload with gbrain files upload and check with gbrain files verify, or call it on an engine opened from settings.' },
+ embedded_engine_required: { class: 'caller', summary: "An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing.", why: 'These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload).' },
engine_closed: { class: 'caller', summary: 'An engine opened from settings was disconnected, which closes it for good, so connect() and reconnect() refuse and nothing was reopened.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call openEngine again with the settings to open a new engine.' },
engine_host_missing: { class: 'caller', summary: "An engine opened from settings has no host identity, so nothing that needs one ran and no host.json was created.", why: 'An embedded engine publishes and claims work under the host id its host gave at open; minting a host.json under the home directory instead would tie its writes to whatever process happens to share that directory.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass host: { id: <uuid> } to openEngine; keep the same id for the same host across restarts.' },
engine_scope_missing: { class: 'caller', summary: "An engine opened from settings made an AI call (embed, chat, expansion or rerank) through a provider its settings.ai does not name, or a call ran outside any engine scope in a process holding such engines, so nothing was sent.", why: "An embedded engine's AI calls reach only the providers and keys its host passed as values. Falling back to a process.env key, ~/.gbrain/config.json or the process-wide gateway would spend another brain's key and send this brain's text where its host never pointed it.", docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass the provider in settings.ai.providers (apiKey, and baseUrl when it is not the default) to openEngine, or point settings.ai.models (or settings.embedding) at a provider the engine has.' },
@@ -733,7 +733,7 @@ export const CODES = {
stable_channel_unresolved: { class: 'unavailable', reasons: ['unreadable', 'no_stable', 'no_release_tag'], summary: "`gbrain upgrade --channel stable` (or `self_upgrade.channel stable`) could not name the stable release: the repository's tags could not be read, no release has been promoted to stable yet, or the `stable` tag does not share a commit with a `v<version>` release tag. Nothing was changed.", why: "The stable channel installs only a release a maintainer promoted after the production-shaped soak passed on its exact tree; without that answer it installs nothing rather than guess.", docs: 'docs/guides/upgrades-auto-update.md#stable-channel', suggestion: 'Retry later, or upgrade on the latest channel with `gbrain upgrade --channel latest`.' },
stale_unavailable: { class: 'unavailable', summary: "No complete, current authorized shared-skills view is available; cached shared skills stay unused.", docs: 'docs/guides/shared-brain-skills.md#troubleshoot-leave-and-recover' },
state_mismatch: { class: 'caller', summary: "Google connect credential error: state mismatch.", docs: 'docs/guides/google-connect.md#troubleshooting' },
- store_unavailable: { class: 'retryable', summary: "The object store an engine opened from settings names did not answer a file check (file_register, file_verify), so nothing was recorded and no file was reported missing.", why: 'A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Check the store named in settings.storage (its endpoint, bucket and keys), then call again." },
+ store_unavailable: { class: 'retryable', summary: "The object store an engine opened from settings names did not answer a file check (file_register, file_verify, file_url), so nothing was recorded and no file was reported missing.", why: 'A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again." },
storage_error: { class: 'server', summary: "Reading or writing durable storage failed.",
reasons: ['database_contention'], reason_overrides: { database_contention: { class: 'retryable' } },
why: "Usually a dropped connection or a failed statement; the same error after a rerun is not transient. `reason: database_contention`: a write admission that deadlocks, serialization failures, lock timeouts or statement timeouts kept refusing for its whole retry budget; no receipt was confirmed, so the same request_id is safe to retry once the contention clears. `detail: persistence_mode_unknown` (#6482): the persistence_brain singleton is missing, duplicated or carries a non-boolean `enabled`, so gbrain cannot tell whether the brain is managed and refuses every write, sync, connector run and activation step that depends on it rather than route a managed brain through the unmanaged writer; only a well-formed `enabled: false` row selects the unmanaged path.",
diff --git a/src/core/ops/files.ts b/src/core/ops/files.ts
index 8854328f83..3f39755500 100644
--- a/src/core/ops/files.ts
+++ b/src/core/ops/files.ts
@@ -105,7 +105,8 @@ const file_upload: Operation = {
// read (file_url, files verify, restore) would trust a phantom object.
// Typed error BEFORE any insert; git-tracked small files have their own
// lane (`gbrain files upload-raw --page <slug>`).
- if (!ctx.config.storage) {
+ const embedded = isEmbeddedEngine(ctx.engine) ? await embeddedFileStore(ctx, 'file_upload') : null;
+ if (!embedded && !ctx.config.storage) {
const slugOk = pageSlug !== null && /^[a-z0-9][a-z0-9._/-]*$/i.test(pageSlug);
// #5963: the git-tracked lane is refused on a managed worktree, so it is not offered there.
if (slugOk && await (await import('../raw-file-upload.ts')).rawUploadRefusedFor(ctx.engine, pageSlug!, ctx.sourceId ?? 'default')) throw opError('storage_error',
@@ -120,7 +121,7 @@ const file_upload: Operation = {
);
}
const { createStorage } = await import('../storage.ts');
- const storage = await createStorage(ctx.config.storage as any);
+ const storage = embedded?.storage ?? await createStorage(ctx.config.storage as any);
const { sqlQueryForEngine, executeRawJsonb, FILES_METADATA_MERGE_SQL } = await import('../sql-query.ts');
const sql = sqlQueryForEngine(ctx.engine);
@@ -159,7 +160,7 @@ const file_upload: Operation = {
mime_type = EXCLUDED.mime_type,
${FILES_METADATA_MERGE_SQL}`,
[pageSlug, filename, storagePath, mimeType, stat.size, hash],
- [{ storage: (ctx.config.storage as { backend: string }).backend }],
+ [embedded?.metadata ?? { storage: (ctx.config.storage as { backend: string }).backend }],
);
} catch (dbErr) {
// Rollback: clean up storage if DB write failed
@@ -194,7 +195,8 @@ const file_url: Operation = {
// #4302: resolve a REAL URL from the backend, after confirming the object
// is actually there — the old `gbrain:files/<path>` placeholder pointed
// at nothing and hid rows whose bytes had vanished.
- if (!ctx.config.storage) {
+ const embedded = isEmbeddedEngine(ctx.engine);
+ if (!embedded && !ctx.config.storage) {
throw opError(
'storage_error',
`No storage backend configured — cannot produce a URL for ${p.storage_path}.`,
@@ -202,8 +204,11 @@ const file_url: Operation = {
);
}
const { createStorage } = await import('../storage.ts');
- const storage = await createStorage(ctx.config.storage as any);
- const present = await storage.exists(rows[0].storage_path as string).catch(() => false);
+ const storage = embedded ? await embeddedStorage(ctx, 'file_url') : await createStorage(ctx.config.storage as any);
+ const present = await storage.exists(rows[0].storage_path as string).catch(error => {
+ if (embedded) throw storeUnavailable('file_url', error);
+ return false;
+ });
if (!present) {
throw opError(
'storage_error',
@@ -215,18 +220,24 @@ const file_url: Operation = {
},
};
-/** file_register and file_verify serve a host that stores bytes itself: they run only on an engine opened from settings, against its settings.storage. */
+/** file_register and file_verify serve a host that stores bytes itself: they run only on an engine opened from settings, against its settings.storage, which every file op on such an engine uses. */
async function embeddedStorage(ctx: OperationContext, op: string) {
if (!isEmbeddedEngine(ctx.engine)) throw opError('embedded_engine_required', `${op} runs only on an engine an embedding host opened from settings (embedded_core: true).`,
- 'On a brain the CLI opens, upload the file with gbrain files upload and check it with gbrain files verify.');
+ 'A brain the CLI or a server opens stores files only through file_upload (gbrain files upload), so there is nothing to register or check this way.');
const settings = embeddedSettingsRecord(ctx.engine)?.storage;
if (!settings) throw opError('storage_error', `This engine was opened with no settings.storage, so ${op} cannot check any stored object.`,
- 'Open the engine with storage (local | s3 | supabase, with its values) in its settings; the process storage config is never used for an engine opened from settings. Nothing was recorded.');
+ 'Open the engine with storage (local or s3, with its values) in its settings; the process storage config is never used for an engine opened from settings. Nothing was recorded.');
const { createStorage, prefixedStorage } = await import('../storage.ts');
const { prefix, ...config } = settings;
try { return prefixedStorage(await createStorage({ bucket: '', ...config }), prefix); } catch (error) { throw storeUnavailable(op, error); }
}
+async function embeddedFileStore(ctx: OperationContext, op: string) {
+ const storage = await embeddedStorage(ctx, op);
+ const { backend, prefix } = embeddedSettingsRecord(ctx.engine)!.storage!;
+ return { storage, metadata: { storage: backend, ...(prefix ? { prefix } : {}) } };
+}
+
function storeUnavailable(op: string, error: unknown) {
return opError('store_unavailable', `${op} could not ask the storage backend: ${error instanceof Error ? error.message : String(error)}`,
'The backend did not answer, so nothing is known about the object and nothing was recorded or reported missing. Check the backend with gbrain doctor --json, then call again.');
@@ -239,8 +250,8 @@ function uploadLayoutPath(pageSlug: string | null, filename: string, hash: strin
async function pageReadable(ctx: OperationContext, slug: string): Promise<boolean> {
const rows = await ctx.engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM pages fp WHERE fp.slug = $1
- AND ${await resolveExcludePrivatePages(ctx.engine, ctx.remote) ? `NOT ${privatePagesFilterForCaller('fp')}` : 'FALSE'}`, [slug]);
- return rows[0]!.n === 0;
+ AND ${await resolveExcludePrivatePages(ctx.engine, ctx.remote) ? privatePagesFilterForCaller('fp') : 'TRUE'}`, [slug]);
+ return rows[0]!.n > 0;
}
const file_register: Operation = {
@@ -261,7 +272,7 @@ const file_register: Operation = {
localOnly: true,
cliOnly: { argv: ['gbrain', 'files', 'upload', '<file>', '--page', '<page_slug>'] },
handler: async (ctx, p) => {
- const storage = await embeddedStorage(ctx, 'file_register');
+ const { storage, metadata } = await embeddedFileStore(ctx, 'file_register');
const filename = p.filename as string, hash = p.content_hash as string, size = p.size_bytes as number;
const pageSlug = (p.page_slug as string | undefined) || null;
validateFilename(filename);
@@ -281,7 +292,7 @@ const file_register: Operation = {
catch (error) { throw storeUnavailable('file_register', error); }
if (!stored) throw opError('storage_error', `The storage backend has no object at ${storagePath}, so no row was recorded.`, 'Store the bytes at that path first, then register it.');
if (stored.size !== size || stored.sha256 !== hash) throw opError('storage_error', `The object at ${storagePath} does not match size_bytes and content_hash, so no row was recorded.`,
- `The stored object is ${stored.size} bytes with SHA-256 ${stored.sha256}. Register the values of the bytes actually stored.`);
+ 'Register the size and SHA-256 of the bytes actually stored at that path.');
const { executeRawJsonb, FILES_METADATA_MERGE_SQL } = await import('../sql-query.ts');
await executeRawJsonb(ctx.engine,
`INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
@@ -292,7 +303,7 @@ const file_register: Operation = {
mime_type = EXCLUDED.mime_type,
${FILES_METADATA_MERGE_SQL}`,
[pageSlug, filename, storagePath, (p.mime_type as string | undefined) ?? null, size, hash],
- [{ storage: embeddedSettingsRecord(ctx.engine)!.storage!.backend }]);
+ [metadata]);
return { status: 'registered', storage_path: storagePath, size_bytes: size };
},
};
diff --git a/test/file-register.test.ts b/test/file-register.test.ts
index 40162d41d5..9bff26b217 100644
--- a/test/file-register.test.ts
+++ b/test/file-register.test.ts
@@ -4,7 +4,7 @@
*/
import { afterAll, afterEach, beforeAll, describe, expect, test } from 'bun:test';
import { createHash, randomUUID } from 'node:crypto';
-import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs';
+import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { embeddedSettings, openEngine, type EngineSettings, type EngineStorageSettings } from '../src/core/engine-open.ts';
@@ -34,6 +34,7 @@ function store(path: string, bytes: Buffer): void {
mkdirSync(dirname(join(storageRoot, path)), { recursive: true });
writeFileSync(join(storageRoot, path), bytes);
}
+const addPage = (engine: BrainEngine, slug: string) => importFromContent(engine, slug, ['---', 'type: note', `title: ${slug}`, '---', `# ${slug}`, '', 'Plan body.'].join('\n'));
const fileRows = async (engine: BrainEngine) => Number((await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM files'))[0]!.n);
const engines: BrainEngine[] = [];
@@ -53,6 +54,7 @@ afterAll(() => rmSync(root, { recursive: true, force: true }));
describe('file_register', () => {
test('records one row for an object stored at the file_upload path, and registering again updates it', async () => {
const engine = await embedded();
+ await addPage(engine, 'notes/plan');
const bytes = Buffer.from('report bytes');
store('notes/plan/report.pdf', bytes);
const params = { storage_path: 'notes/plan/report.pdf', filename: 'report.pdf', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length, mime_type: 'application/pdf' };
@@ -62,6 +64,20 @@ describe('file_register', () => {
await run('file_register', ctxFor(engine), { ...params, content_hash: sha(next), size_bytes: next.length });
const rows = await engine.executeRaw<{ page_slug: string; content_hash: string; size_bytes: unknown }>('SELECT page_slug, content_hash, size_bytes FROM files');
expect(rows.map(r => ({ ...r, size_bytes: Number(r.size_bytes) }))).toEqual([{ page_slug: 'notes/plan', content_hash: sha(next), size_bytes: next.length }]);
+ expect(await run('file_register', ctxFor(engine, { dryRun: true }), { ...params, filename: 'other.pdf', storage_path: 'notes/plan/other.pdf' }))
+ .toEqual({ dry_run: true, action: 'file_register', storage_path: 'notes/plan/other.pdf' });
+ expect(await fileRows(engine)).toBe(1);
+ });
+
+ test('refuses a page that does not exist and an object it cannot read back, recording nothing', async () => {
+ const engine = await embedded();
+ const bytes = Buffer.from('orphan');
+ store('notes/none/o.txt', bytes);
+ expect((await failure(() => run('file_register', ctxFor(engine), { storage_path: 'notes/none/o.txt', filename: 'o.txt', page_slug: 'notes/none', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('page_not_found');
+ await addPage(engine, 'notes/plan');
+ mkdirSync(join(storageRoot, 'notes/plan/dir.txt'), { recursive: true });
+ expect((await failure(() => run('file_register', ctxFor(engine), { storage_path: 'notes/plan/dir.txt', filename: 'dir.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('store_unavailable');
+ expect(await fileRows(engine)).toBe(0);
});
test('refuses a path other than the file_upload layout, a missing or different object and bad inputs, recording nothing', async () => {
@@ -100,6 +116,7 @@ describe('file_register', () => {
test('two brains sharing one store under different prefixes see only their own objects', async () => {
const a = await embedded({ storage: { ...storage, prefix: 'brain-a' } });
const b = await embedded({ storage: { ...storage, prefix: 'brain-b' } });
+ await Promise.all([addPage(a, 'notes/plan'), addPage(b, 'notes/plan')]);
const bytes = Buffer.from('brain a bytes');
store('brain-a/notes/plan/a.txt', bytes);
const params = { storage_path: 'notes/plan/a.txt', filename: 'a.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length };
@@ -107,6 +124,22 @@ describe('file_register', () => {
expect((await failure(() => run('file_register', ctxFor(b), params)))?.code).toBe('storage_error');
expect(await fileRows(b)).toBe(0);
expect(await run('file_verify', ctxFor(a), { storage_path: 'notes/plan/a.txt' })).toEqual({ storage_path: 'notes/plan/a.txt', present: true, size_matches: true, hash_matches: true });
+ const [{ metadata }] = await a.executeRaw<{ metadata: unknown }>('SELECT metadata FROM files');
+ expect(metadata).toEqual({ storage: 'local', prefix: 'brain-a' });
+ });
+
+ test('file_upload and file_url on an engine opened from settings use its store and prefix, never the process store', async () => {
+ const engine = await embedded({ storage: { ...storage, prefix: 'brain-a' } });
+ await addPage(engine, 'notes/plan');
+ const source = join(root, 'upload.txt');
+ writeFileSync(source, 'uploaded bytes');
+ const processStore = { backend: 'local', bucket: 'test', localPath: join(root, 'process-store') };
+ const ctx = ctxFor(engine, { config: { storage: processStore } } as never);
+ expect(await run('file_upload', ctx, { path: source, page_slug: 'notes/plan' })).toMatchObject({ status: 'uploaded', storage_path: 'notes/plan/upload.txt' });
+ expect(readFileSync(join(storageRoot, 'brain-a/notes/plan/upload.txt'), 'utf8')).toBe('uploaded bytes');
+ expect(existsSync(join(root, 'process-store'))).toBe(false);
+ expect(await run('file_url', ctx, { storage_path: 'notes/plan/upload.txt' })).toEqual({ storage_path: 'notes/plan/upload.txt', url: `file://${join(storageRoot, 'brain-a/notes/plan/upload.txt')}` });
+ expect(await run('file_verify', ctx, { storage_path: 'notes/plan/upload.txt' })).toMatchObject({ present: true, hash_matches: true });
});
test('a store that cannot be reached is store_unavailable and records nothing', async () => {
@@ -124,8 +157,8 @@ describe('settings.storage', () => {
test('openEngine refuses a store missing a value it needs, an unknown backend, field or prefix, and the view hides its secret', async () => {
const cases: Array<[unknown, string]> = [
[{ backend: 'nope' }, 'storage.backend'], [{ backend: 'local' }, 'storage.localPath'],
- [{ backend: 's3', bucket: 'b', accessKeyId: 'k' }, 'storage.secretAccessKey'], [{ backend: 'supabase', bucket: 'b', projectUrl: 'https://x.example' }, 'storage.serviceRoleKey'],
- [{ ...storage, region2: 'x' }, 'storage.region2'], [{ ...storage, prefix: '../a' }, 'storage.prefix'], [{ ...storage, prefix: 'a/' }, 'storage.prefix'],
+ [{ backend: 'supabase', bucket: 'b' }, 'storage.backend'], [{ ...unreachable, endpoint: undefined }, 'storage.endpoint'], [{ ...unreachable, secretAccessKey: '' }, 'storage.secretAccessKey'],
+ [{ ...storage, region2: 'x' }, 'storage.region2'], [{ ...storage, prefix: '../a' }, 'storage.prefix'], [{ ...storage, prefix: 'a/b' }, 'storage.prefix'],
];
for (const [given, setting] of cases) {
const error = await failure(() => openEngine(settings({ storage: given as EngineStorageSettings })));
@@ -145,11 +178,35 @@ describe('hashing a stored object', () => {
const s3 = new S3Storage({ backend: 's3', bucket: 'test' }, client as never);
expect(await s3.hashObject('a/b.txt')).toEqual({ size: 23, sha256: sha(Buffer.concat(parts)) });
});
+
+ test('a backend without hashObject is hashed from one download', async () => {
+ const { hashStoredObject } = await import('../src/core/storage.ts');
+ const bytes = Buffer.from('downloaded');
+ expect(await hashStoredObject({ download: async () => bytes } as never, 'a.txt')).toEqual({ size: bytes.length, sha256: sha(bytes) });
+ });
+
+ test('an s3 store sends every request to its settings endpoint, never one named by AWS_* variables', async () => {
+ const hits = { settings: 0, env: 0 };
+ const settingsServer = Bun.serve({ port: 0, fetch: () => { hits.settings++; return new Response(null, { status: 404 }); } });
+ const envServer = Bun.serve({ port: 0, fetch: () => { hits.env++; return new Response(null, { status: 200 }); } });
+ try {
+ await withEnv({ AWS_ENDPOINT_URL_S3: `http://127.0.0.1:${envServer.port}`, AWS_ENDPOINT_URL: `http://127.0.0.1:${envServer.port}` }, async () => {
+ const engine = await embedded({ storage: { ...unreachable, endpoint: `http://127.0.0.1:${settingsServer.port}` } });
+ const bytes = Buffer.from('s3');
+ const error = await failure(() => run('file_register', ctxFor(engine), { storage_path: `unsorted/${sha(bytes).slice(0, 8)}-s.txt`, filename: 's.txt', content_hash: sha(bytes), size_bytes: 2 }));
+ expect([error?.code, hits.env, hits.settings > 0]).toEqual(['storage_error', 0, true]);
+ });
+ } finally {
+ settingsServer.stop(true);
+ envServer.stop(true);
+ }
+ });
});
describe('file_verify', () => {
test('lists presence per row, reads one object back, and notices bytes changed underneath', async () => {
const engine = await embedded();
+ await addPage(engine, 'notes/plan');
const a = Buffer.from('alpha'), b = Buffer.from('bravo');
store('notes/plan/a.txt', a);
store('notes/plan/b.txt', b);
diff --git a/test/fixtures/goldens/exports/types.json b/test/fixtures/goldens/exports/types.json
index e321e36a54..9c5b5ce061 100644
--- a/test/fixtures/goldens/exports/types.json
+++ b/test/fixtures/goldens/exports/types.json
@@ -4433,7 +4433,7 @@
"kind": "type",
"name": "EmbeddedSettingsView",
"signature": [
- "{ readonly pglite?: DeepReadonly<{ shared_buffers?: string; } | undefined>; readonly host: DeepReadonly<{ id: string; }>; readonly database: DeepReadonly<{ url?: string; directUrl?: string; path?: string; }>; readonly embedding?: DeepReadonly<{ model: string; dimensions: number; } | undefined>; readonly embedded_core: DeepReadonly<boolean>; readonly engine: DeepReadonly<\"postgres\" | \"pglite\">; readonly pool?: DeepReadonly<{ max?: number; } | undefined>; readonly keyword_statement_timeout_ms?: number | undefined; readonly writer?: DeepReadonly<{ id: string; lane: \"cli\" | \"stdio\"; } | undefined>; readonly ai?: DeepReadonly<{ providers: Record<string, { baseUrl?: string; hasApiKey: boolean; }>; models?: EngineAISettings[\"models\"]; } | undefined>; readonly storage?: DeepReadonly<(Omit<EngineStorageSettings, \"secretAccessKey\" | \"serviceRoleKey\"> & { hasSecret: boolean; }) | undefined>; }"
+ "{ readonly pglite?: DeepReadonly<{ shared_buffers?: string; } | undefined>; readonly host: DeepReadonly<{ id: string; }>; readonly database: DeepReadonly<{ url?: string; directUrl?: string; path?: string; }>; readonly embedding?: DeepReadonly<{ model: string; dimensions: number; } | undefined>; readonly embedded_core: DeepReadonly<boolean>; readonly engine: DeepReadonly<\"postgres\" | \"pglite\">; readonly pool?: DeepReadonly<{ max?: number; } | undefined>; readonly keyword_statement_timeout_ms?: number | undefined; readonly writer?: DeepReadonly<{ id: string; lane: \"cli\" | \"stdio\"; } | undefined>; readonly ai?: DeepReadonly<{ providers: Record<string, { baseUrl?: string; hasApiKey: boolean; }>; models?: EngineAISettings[\"models\"]; } | undefined>; readonly storage?: DeepReadonly<(Omit<EngineStorageSettings, \"secretAccessKey\"> & { hasSecret: boolean; }) | undefined>; }"
]
},
{
@@ -4465,15 +4465,13 @@
"kind": "interface",
"members": [
"accessKeyId?: string | undefined",
- "backend: \"local\" | \"s3\" | \"supabase\"",
+ "backend: \"local\" | \"s3\"",
"bucket?: string | undefined",
"endpoint?: string | undefined",
"localPath?: string | undefined",
"prefix?: string | undefined",
- "projectUrl?: string | undefined",
"region?: string | undefined",
- "secretAccessKey?: string | undefined",
- "serviceRoleKey?: string | undefined"
+ "secretAccessKey?: string | undefined"
],
"name": "EngineStorageSettings"
},