Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
feat(files): bring back file_register on engines opened from settings
garrytan0056aa5cba20
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6bba964b72..660a7dc71e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -12,16 +12,17 @@ identifiers and attribution are available in the pre-removal Git revision## [Unreleased]-**A brain opened from settings keeps its files in its own store.** On an engine opened from settings, every file operation (`file_upload`, `file_url`, page purge and the new `file_verify`) uses the engine's own `settings.storage`: a local path, or s3 with its endpoint and keys, and an optional one-segment key `prefix` so brains sharing a bucket each keep their own keys. The process storage config is never used there. `file_verify` lists which rows still have their object or re-hashes one. A brain the CLI or a server opens is unchanged, and there `file_verify` refuses `embedded_engine_required`; it is never an MCP tool. `file_register`, which lets a host record bytes it stored itself, ships separately. Contributed by @bradgessler (#8457).+**A brain opened from settings keeps its files in its own store.** On an engine opened from settings, every file operation (`file_upload`, `file_url`, page purge and the new `file_register` and `file_verify`) uses the engine's own `settings.storage`: a local path, or s3 with its endpoint and keys, and an optional one-segment key `prefix` so brains sharing a bucket each keep their own keys. The process storage config is never used there. `file_register` records the files row for an object the host already wrote at the path `file_upload` would use, after reading it back and matching its size and SHA-256, and `file_verify` lists which rows still have their object or re-hashes one. A brain the CLI or a server opens is unchanged, and there both refuse `embedded_engine_required`; neither is ever an MCP tool. Contributed by @bradgessler (#8457).### Itemized changes+- `src/core/ops/files.ts`: `file_register` accepts only the upload layout path (`<page_slug>/<filename>`, or `unsorted/<first 8 hash chars>-<filename>`), refuses `page_not_found` for a page that does not exist or that the caller's read policy hides (member access and privacy, the private-pages hatch included), and records nothing unless the stored object matches. Rows record their store's backend and prefix, and the page and source they belong to, so purging that page finds them. Registering the same path again updates its row, and a registration that would move a row to another page, filename or source is refused `invalid_params`.- `src/core/storage/s3.ts`: `file_url` on an s3 store with an endpoint builds the URL from the endpoint the SDK resolves, instead of throwing.-- A store that does not answer `file_verify` or `file_url` is the new `store_unavailable` code, never a file reported missing.+- A store that does not answer is the new `store_unavailable` code, never a file reported missing and never a row recorded.- On an engine opened from settings, `delete_page` purge deletes a page's files from `settings.storage` too. A file it can't find there, or a store that doesn't answer, is reported in `blobs_remaining`, never as deleted; files an embedded engine's `file_upload` wrote before this release live in the process store, so delete those remaining paths by hand. Rows an embedded engine's `file_upload` wrote before this release point at the process store, so `file_url` and `file_verify` look for them in `settings.storage`.- `openEngine` refuses `engine_settings_invalid` for a `storage` setting missing a value its backend needs (so no backend falls back to a default path or environment credentials), an unknown field, a backend other than local or s3, or a prefix that is not one segment; an s3 store must name its endpoint, so no AWS_* variable or ~/.aws file picks it; `embeddedSettings()` reads back `hasSecret` instead of the secret.- `src/core/storage/local.ts` and `s3.ts` stream an object's size and SHA-256 (`hashObject`); other backends hash one download.-- `test/embedded-file-store.test.ts`: real PGLite engines and a real local store; the unreachable store is a closed local port.+- `test/embedded-file-store.test.ts` and `test/file-register.test.ts`: real PGLite engines and a real local store; the unreachable store is a closed local port.## [0.60.565.0] - 2026-11-03diff --git a/docs/architecture/managed-mutating-writers.tsv b/docs/architecture/managed-mutating-writers.tsvindex 8f31522fd8..c401fc3b9c 100644--- a/docs/architecture/managed-mutating-writers.tsv+++ b/docs/architecture/managed-mutating-writers.tsv@@ -38,6 +38,7 @@ op sync_brain coordinator Managed sync admits each changed file through the coorop put_raw_data no_canonical_write raw_data side table; not guarded canonical content.op log_ingest no_canonical_write ingest_log audit rows only.op file_upload no_canonical_write File storage objects and their index rows, confined to the upload root.+op file_register no_canonical_write A files index row for an object an embedded host already stored at the upload layout path.op submit_job no_canonical_write Queues a minion job; its handler owns any write and its own boundary.op cancel_job no_canonical_write Minion queue state only.op retry_job no_canonical_write Minion queue state only.diff --git a/docs/guides/error-codes.md b/docs/guides/error-codes.mdindex 79c1a8e167..82dee039be 100644--- a/docs/guides/error-codes.md+++ b/docs/guides/error-codes.md@@ -1349,7 +1349,7 @@ More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| An operation that serves an embedding host (file_verify) was called on an engine the CLI or a server opened, so it did nothing. | These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged. | Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload). | agent | `repeat the read that failed` | 1 | no |+| An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing. | These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged. | Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload). | agent | `repeat the read that failed` | 1 | no |More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/ENGINES.md#opening-an-engine-from-settings-experimental)@@ -4507,7 +4507,7 @@ Override: reason `database_contention`: retryable, exit 1, retryable.| Meaning | Why | Next step | Who acts · consent | Verify | Exit | Retryable ||---|---|---|---|---|---|---|-| The object store an engine opened from settings names did not answer a file check (file_verify, file_url), so nothing was recorded and no file was reported missing. | A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there. | Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again. | agent | `repeat the read that failed` | 1 | yes |+| The object store an engine opened from settings names did not answer a file check (file_register, file_verify, file_url), so nothing was recorded and no file was reported missing. | A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there. | Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again. | agent | `repeat the read that failed` | 1 | yes |More: [docs/ENGINES.md#opening-an-engine-from-settings-experimental](../../docs/ENGINES.md#opening-an-engine-from-settings-experimental)diff --git a/plugin/README.md b/plugin/README.mdindex acc3919694..6c65438bda 100644--- a/plugin/README.md+++ b/plugin/README.md@@ -8,7 +8,7 @@ addition/exclusion).## MCP surface note (read once)-The plugin's MCP server runs `gbrain serve --surface full` — 186+The plugin's MCP server runs `gbrain serve --surface full` — 187operations, the same surface every stdio registration gbrain writes pins.Every gbrain operation the bundled skills name is on it. A harness that capsits tool count can narrow this machine's plugin surface withdiff --git a/src/core/error-registry.ts b/src/core/error-registry.tsindex 6602cda613..ad68659be1 100644--- a/src/core/error-registry.ts+++ b/src/core/error-registry.ts@@ -302,7 +302,7 @@ export const CODES = {embedding_width_mismatch: { class: 'caller', summary: "The embedding model cannot produce the width of the brain's vector column; nothing was changed." },embedded_core_required: { class: 'caller', summary: "openEngine (gbrain/engine-open) was called without embedded_core: true, so no engine was opened.", why: 'Opening an engine from settings alone is an experimental embedding surface that is off by default; a host opts in explicitly so no existing caller changes behavior.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass embedded_core: true in the settings; the CLI, gbrain serve and mounts open brains through their own connect path and need nothing.' },engine_embedding_mismatch: { class: 'host_only', actor: 'user', summary: "An engine opened from settings names an embedding model or width that differs from the one its database already records, so initSchema changed nothing.", why: 'The vector column has one width and the brain one embedding model; seeding another over it would make every stored and future embedding unreadable. Changing it is an explicit re-embed the user decides on.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Pass the database's recorded model and width in settings.embedding, or read the recorded identity with gbrain migrate embeddings --status on the brain host and preview a migration from there.", fix: { argv: ['gbrain', 'migrate', 'embeddings', '--status', '--json'], consent: [], actor: 'agent', requires_exclusive: false, why: "Read-only: reports the brain's recorded embedding model and width and what a migration would change." } },- embedded_engine_required: { class: 'caller', summary: "An operation that serves an embedding host (file_verify) was called on an engine the CLI or a server opened, so it did nothing.", why: 'These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload).' },+ embedded_engine_required: { class: 'caller', summary: "An operation that serves an embedding host (file_register, file_verify) was called on an engine the CLI or a server opened, so it did nothing.", why: 'These operations trust a host that stores bytes itself; a brain opened the usual way stores files through file_upload, so the operations stay off there and its behavior is unchanged.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call it on an engine opened from settings with settings.storage; a brain the CLI or a server opens stores files only through file_upload (gbrain files upload).' },engine_closed: { class: 'caller', summary: 'An engine opened from settings was disconnected, which closes it for good, so connect() and reconnect() refuse and nothing was reopened.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Call openEngine again with the settings to open a new engine.' },engine_host_missing: { class: 'caller', summary: "An engine opened from settings has no host identity, so nothing that needs one ran and no host.json was created.", why: 'An embedded engine publishes and claims work under the host id its host gave at open; minting a host.json under the home directory instead would tie its writes to whatever process happens to share that directory.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass host: { id: <uuid> } to openEngine; keep the same id for the same host across restarts.' },engine_scope_missing: { class: 'caller', summary: "An engine opened from settings made an AI call (embed, chat, expansion or rerank) through a provider its settings.ai does not name, or a call ran outside any engine scope in a process holding such engines, so nothing was sent.", why: "An embedded engine's AI calls reach only the providers and keys its host passed as values. Falling back to a process.env key, ~/.gbrain/config.json or the process-wide gateway would spend another brain's key and send this brain's text where its host never pointed it.", docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: 'Pass the provider in settings.ai.providers (apiKey, and baseUrl when it is not the default) to openEngine, or point settings.ai.models (or settings.embedding) at a provider the engine has.' },@@ -733,7 +733,7 @@ export const CODES = {stable_channel_unresolved: { class: 'unavailable', reasons: ['unreadable', 'no_stable', 'no_release_tag'], summary: "`gbrain upgrade --channel stable` (or `self_upgrade.channel stable`) could not name the stable release: the repository's tags could not be read, no release has been promoted to stable yet, or the `stable` tag does not share a commit with a `v<version>` release tag. Nothing was changed.", why: "The stable channel installs only a release a maintainer promoted after the production-shaped soak passed on its exact tree; without that answer it installs nothing rather than guess.", docs: 'docs/guides/upgrades-auto-update.md#stable-channel', suggestion: 'Retry later, or upgrade on the latest channel with `gbrain upgrade --channel latest`.' },stale_unavailable: { class: 'unavailable', summary: "No complete, current authorized shared-skills view is available; cached shared skills stay unused.", docs: 'docs/guides/shared-brain-skills.md#troubleshoot-leave-and-recover' },state_mismatch: { class: 'caller', summary: "Google connect credential error: state mismatch.", docs: 'docs/guides/google-connect.md#troubleshooting' },- store_unavailable: { class: 'retryable', summary: "The object store an engine opened from settings names did not answer a file check (file_verify, file_url), so nothing was recorded and no file was reported missing.", why: 'A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again." },+ store_unavailable: { class: 'retryable', summary: "The object store an engine opened from settings names did not answer a file check (file_register, file_verify, file_url), so nothing was recorded and no file was reported missing.", why: 'A failed read says nothing about whether the object exists; reporting it missing could lead a host to re-upload or delete a file that is still there.', docs: 'docs/ENGINES.md#opening-an-engine-from-settings-experimental', suggestion: "Check the store named in settings.storage (its path, or its endpoint, bucket and keys), then call again." },storage_error: { class: 'server', summary: "Reading or writing durable storage failed.",reasons: ['database_contention'], reason_overrides: { database_contention: { class: 'retryable' } },why: "Usually a dropped connection or a failed statement; the same error after a rerun is not transient. `reason: database_contention`: a write admission that deadlocks, serialization failures, lock timeouts or statement timeouts kept refusing for its whole retry budget; no receipt was confirmed, so the same request_id is safe to retry once the contention clears. `detail: persistence_mode_unknown` (#6482): the persistence_brain singleton is missing, duplicated or carries a non-boolean `enabled`, so gbrain cannot tell whether the brain is managed and refuses every write, sync, connector run and activation step that depends on it rather than route a managed brain through the unmanaged writer; only a well-formed `enabled: false` row selects the unmanaged path.",diff --git a/src/core/operation-load.ts b/src/core/operation-load.tsindex 9535517d72..99f83e35d1 100644--- a/src/core/operation-load.ts+++ b/src/core/operation-load.ts@@ -79,7 +79,7 @@ const OP_AREAS: Record<string, string> = {log_ingest: 'ingest', get_ingest_log: 'ingest',get_usage: 'admin',// files (localOnly)- file_list: 'files', file_upload: 'files', file_url: 'files', file_verify: 'files',+ file_list: 'files', file_upload: 'files', file_url: 'files', file_register: 'files', file_verify: 'files',// jobs (Minions + agent lane)submit_job: 'jobs', get_job: 'jobs', wait_job: 'jobs', list_jobs: 'jobs', cancel_job: 'jobs',retry_job: 'jobs', get_job_progress: 'jobs', pause_job: 'jobs',diff --git a/src/core/operation-loaders.generated.ts b/src/core/operation-loaders.generated.tsindex d4ace27b8d..20c53aac3f 100644--- a/src/core/operation-loaders.generated.ts+++ b/src/core/operation-loaders.generated.ts@@ -144,6 +144,7 @@ export const OPERATION_LOADERS: Record<string, readonly [() => Promise<Record<st"file_list": [load_files, "filesOperations"],"file_upload": [load_files, "filesOperations"],"file_url": [load_files, "filesOperations"],+ "file_register": [load_files, "filesOperations"],"file_verify": [load_files, "filesOperations"],"submit_job": [load_jobs, "jobsOperations"],"get_job": [load_jobs, "jobsOperations"],diff --git a/src/core/operation-manifest.generated.ts b/src/core/operation-manifest.generated.tsindex b1c671a453..8db565957b 100644--- a/src/core/operation-manifest.generated.ts+++ b/src/core/operation-manifest.generated.ts@@ -3683,6 +3683,54 @@ export const OPERATION_MANIFEST: OperationMeta[] = ["localOnly": true,"area": "files"},+ {+ "name": "file_register",+ "idempotent": true,+ "outputRedaction": "no_stored_text",+ "description": "Record the files row for an object the host already stored at the path file_upload would use (<page_slug>/<filename>, or unsorted/<first 8 hash chars>-<filename>), after reading it back and matching its size and SHA-256. Use when an embedded host writes bytes to storage itself. Runs only on an engine opened from settings; registering the same path again updates its row.",+ "params": {+ "storage_path": {+ "type": "string",+ "required": true,+ "description": "Where the host stored the object; must be the file_upload layout for page_slug, filename and content_hash"+ },+ "filename": {+ "type": "string",+ "required": true+ },+ "content_hash": {+ "type": "string",+ "required": true,+ "description": "SHA-256 of the bytes, 64 lowercase hex characters"+ },+ "size_bytes": {+ "type": "integer",+ "required": true,+ "minimum": 0+ },+ "page_slug": {+ "type": "string",+ "description": "Page the file belongs to"+ },+ "mime_type": {+ "type": "string"+ }+ },+ "mutating": true,+ "scope": "admin",+ "localOnly": true,+ "cliOnly": {+ "argv": [+ "gbrain",+ "files",+ "upload",+ "<file>",+ "--page",+ "<page_slug>"+ ]+ },+ "area": "files"+ },{"name": "file_verify","mutating": false,diff --git a/src/core/ops/files.ts b/src/core/ops/files.tsindex 9be61d6543..c495136093 100644--- a/src/core/ops/files.ts+++ b/src/core/ops/files.ts@@ -220,10 +220,10 @@ const file_url: Operation = {},};-/** file_verify serves a host that stores bytes itself: it runs only on an engine opened from settings, against its settings.storage, which every file op on such an engine uses. */+/** file_register and file_verify serve a host that stores bytes itself: they run only on an engine opened from settings, against its settings.storage, which every file op on such an engine uses. */async function embeddedStorage(ctx: OperationContext, op: string) {if (!isEmbeddedEngine(ctx.engine)) throw opError('embedded_engine_required', `${op} runs only on an engine an embedding host opened from settings (embedded_core: true).`,- 'A brain the CLI or a server opens stores files only through file_upload (gbrain files upload), so there is nothing to check this way.');+ 'A brain the CLI or a server opens stores files only through file_upload (gbrain files upload), so there is nothing to register or check this way.');const settings = embeddedSettingsRecord(ctx.engine)?.storage;if (!settings) throw opError('storage_error', `This engine was opened with no settings.storage, so ${op} cannot check any stored object.`,'Open the engine with storage (local or s3, with its values) in its settings; the process storage config is never used for an engine opened from settings. Nothing was recorded.');@@ -242,6 +242,81 @@ function storeUnavailable(op: string, error: unknown) {'The backend did not answer, so nothing is known about the object and nothing was recorded or reported missing. Check the backend with gbrain doctor --json, then call again.');}+/** The path file_upload stores a file under; file_register accepts no other, so a registered row only names what an upload could have written. */+function uploadLayoutPath(pageSlug: string | null, filename: string, hash: string): string {+ return pageSlug ? `${pageSlug}/${filename}` : `unsorted/${hash.slice(0, 8)}-${filename}`;+}++async function readablePageId(ctx: OperationContext, slug: string, sourceId: string): Promise<number | undefined> {+ const rows = await ctx.engine.executeRaw<{ id: number }>(`SELECT fp.id FROM pages fp WHERE fp.slug = $1 AND fp.source_id = $2+ AND ${await resolveExcludePrivatePages(ctx.engine, ctx.remote) ? privatePagesFilterForCaller('fp') : 'TRUE'}`, [slug, sourceId]);+ return rows[0]?.id;+}++const file_register: Operation = {+ name: 'file_register',+ idempotent: true,+ outputRedaction: 'no_stored_text',+ description: 'Record the files row for an object the host already stored at the path file_upload would use (<page_slug>/<filename>, or unsorted/<first 8 hash chars>-<filename>), after reading it back and matching its size and SHA-256. Use when an embedded host writes bytes to storage itself. Runs only on an engine opened from settings; registering the same path again updates its row.',+ params: {+ storage_path: { type: 'string', required: true, description: 'Where the host stored the object; must be the file_upload layout for page_slug, filename and content_hash' },+ filename: { type: 'string', required: true },+ content_hash: { type: 'string', required: true, description: 'SHA-256 of the bytes, 64 lowercase hex characters' },+ size_bytes: { type: 'integer', required: true, minimum: 0 },+ page_slug: { type: 'string', description: 'Page the file belongs to' },+ mime_type: { type: 'string' },+ },+ mutating: true,+ scope: 'admin',+ localOnly: true,+ cliOnly: { argv: ['gbrain', 'files', 'upload', '<file>', '--page', '<page_slug>'] },+ handler: async (ctx, p) => {+ const { storage, metadata } = await embeddedFileStore(ctx, 'file_register');+ const filename = p.filename as string, hash = p.content_hash as string, size = p.size_bytes as number;+ const pageSlug = (p.page_slug as string | undefined) || null;+ validateFilename(filename);+ if (pageSlug) validatePageSlug(pageSlug);+ if (!/^[0-9a-f]{64}$/.test(hash)) throw opError('invalid_params', 'content_hash must be the SHA-256 of the bytes as 64 lowercase hex characters.', 'Hash the stored bytes with SHA-256 and pass the hex digest.');+ if (!Number.isSafeInteger(size) || size < 0) throw opError('invalid_params', 'size_bytes must be a whole number of bytes.', 'Pass the stored object\'s size in bytes.');+ const storagePath = uploadLayoutPath(pageSlug, filename, hash);+ if (p.storage_path !== storagePath) throw opError('invalid_params', `storage_path must be ${storagePath} for this page, filename and hash.`,+ `Store the object at ${storagePath} (the path file_upload uses) and register that path. Nothing was recorded.`);+ const sourceId = ctx.sourceId ?? 'default';+ const pageId = pageSlug ? await readablePageId(ctx, pageSlug, sourceId) : null;+ if (pageId === undefined) throw opError('page_not_found', `No page '${pageSlug}' in source '${sourceId}' is visible to this caller.`, 'Attach the file to a page this caller can read. Nothing was recorded.');+ const existing = await ctx.engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM files f WHERE f.storage_path = $1 AND NOT (${await fileReadableSql(ctx)})`, [storagePath]);+ if (existing[0]!.n > 0) throw opError('page_not_found', `The file at ${storagePath} belongs to a page this caller cannot read.`, 'Nothing was recorded.');+ const owned = await ctx.engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM files WHERE storage_path = $1+ AND (page_slug IS DISTINCT FROM $2 OR filename <> $3 OR source_id IS DISTINCT FROM $4)`, [storagePath, pageSlug, filename, sourceId]);+ if (owned[0]!.n > 0) throw opError('invalid_params', `${storagePath} is already registered to a different page, filename or source.`,+ 'A registration never moves an existing row. Register under a path no other row uses. Nothing was recorded.');+ if (ctx.dryRun) return { dry_run: true, action: 'file_register', storage_path: storagePath };+ const { hashStoredObject } = await import('../storage.ts');+ let stored: { size: number; sha256: string } | null;+ try { stored = await storage.exists(storagePath) ? await hashStoredObject(storage, storagePath) : null; }+ catch (error) { throw storeUnavailable('file_register', error); }+ if (!stored) throw opError('storage_error', `The storage backend has no object at ${storagePath}, so no row was recorded.`, 'Store the bytes at that path first, then register it.');+ if (stored.size !== size || stored.sha256 !== hash) throw opError('storage_error', `The object at ${storagePath} does not match size_bytes and content_hash, so no row was recorded.`,+ 'Register the size and SHA-256 of the bytes actually stored at that path.');+ const { executeRawJsonb, FILES_METADATA_MERGE_SQL } = await import('../sql-query.ts');+ const written = await executeRawJsonb(ctx.engine,+ `INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, source_id, page_id, metadata)+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb)+ ON CONFLICT (storage_path) DO UPDATE SET+ content_hash = EXCLUDED.content_hash,+ size_bytes = EXCLUDED.size_bytes,+ mime_type = EXCLUDED.mime_type,+ ${FILES_METADATA_MERGE_SQL}+ WHERE files.page_slug IS NOT DISTINCT FROM EXCLUDED.page_slug AND files.filename = EXCLUDED.filename+ AND files.source_id IS NOT DISTINCT FROM EXCLUDED.source_id+ RETURNING 1`,+ [pageSlug, filename, storagePath, (p.mime_type as string | undefined) ?? null, size, hash, sourceId, pageId],+ [metadata]);+ if (!written.length) throw opError('invalid_params', `${storagePath} was registered to a different page, filename or source meanwhile.`, 'Nothing was recorded.');+ return { status: 'registered', storage_path: storagePath, size_bytes: size };+ },+};+const file_verify: Operation = {name: 'file_verify',mutating: false,@@ -277,4 +352,4 @@ const file_verify: Operation = {};// Ops in EXACTLY the canonical `operations` array order.-export const filesOperations: Operation[] = [file_list, file_upload, file_url, file_verify];+export const filesOperations: Operation[] = [file_list, file_upload, file_url, file_register, file_verify];diff --git a/test/file-register.test.ts b/test/file-register.test.tsnew file mode 100644index 0000000000..4f9b962726--- /dev/null+++ b/test/file-register.test.ts@@ -0,0 +1,231 @@+/**+ * GBRA-283 Wave C: file_register, for an embedding host that stores a file's bytes itself.+ * Contributed by @bradgessler (#8457); rewritten on master. Real PGLite engines and a real local storage backend.+ */+import { afterAll, afterEach, beforeAll, describe, expect, test } from 'bun:test';+import { createHash, randomUUID } from 'node:crypto';+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { dirname, join } from 'node:path';+import { openEngine, type EngineSettings, type EngineStorageSettings } from '../src/core/engine-open.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { importFromContent } from '../src/core/import-file.ts';+import { operationsByName, type OperationContext } from '../src/core/operations.ts';+import { dispatchToolCall } from '../src/mcp/dispatch.ts';+import { isCallable } from '../src/core/ops/callable.ts';+import { __resetPrivateVisibilityCacheForTests } from '../src/core/search/private-visibility.ts';+import type { BrainEngine } from '../src/core/engine.ts';+import { withEnv } from './helpers/with-env.ts';++const root = mkdtempSync(join(tmpdir(), 'gbrain-file-register-'));+const storageRoot = join(root, 'store');+const storage: EngineStorageSettings = { backend: 'local', localPath: storageRoot };+const unreachable: EngineStorageSettings = { backend: 's3', bucket: 'test', endpoint: 'http://127.0.0.1:1', region: 'us-east-1', accessKeyId: 'test-key', secretAccessKey: 'test-secret' };+const settings = (over: Partial<EngineSettings> = {}): EngineSettings => ({ embedded_core: true, engine: 'pglite', database: {}, host: { id: randomUUID() }, embedding: { model: 'openai:text-embedding-3-small', dimensions: 1536 }, storage, ...over });+const ctxFor = (engine: BrainEngine, over: Partial<OperationContext> = {}): OperationContext => ({+ engine, config: {}, logger: { info: () => {}, warn: () => {}, error: () => {} }, dryRun: false, remote: false, ...over,+} as unknown as OperationContext);+const run = (op: string, ctx: OperationContext, params: Record<string, unknown>) => operationsByName[op]!.handler(ctx, params);+const failure = async (work: () => Promise<unknown>) => {+ try { await work(); return null; } catch (error) { return error as Error & { code?: string }; }+};+const sha = (bytes: Buffer) => createHash('sha256').update(bytes).digest('hex');+function store(path: string, bytes: Buffer): void {+ mkdirSync(dirname(join(storageRoot, path)), { recursive: true });+ writeFileSync(join(storageRoot, path), bytes);+}+const addPage = (engine: BrainEngine, slug: string) => importFromContent(engine, slug, ['---', 'type: note', `title: ${slug}`, '---', `# ${slug}`, '', 'Plan body.'].join('\n'));+const fileRows = async (engine: BrainEngine) => Number((await engine.executeRaw<{ n: number }>('SELECT count(*)::int AS n FROM files'))[0]!.n);++const engines: BrainEngine[] = [];+async function embedded(over: Partial<EngineSettings> = {}): Promise<BrainEngine> {+ const engine = await openEngine(settings(over));+ await engine.initSchema();+ engines.push(engine);+ return engine;+}+afterEach(async () => {+ await Promise.all(engines.splice(0).map(engine => engine.disconnect()));+ rmSync(storageRoot, { recursive: true, force: true });+ __resetPrivateVisibilityCacheForTests();+});+afterAll(() => rmSync(root, { recursive: true, force: true }));++describe('file_register', () => {+ test('records one row for an object stored at the file_upload path, and registering again updates it', async () => {+ const engine = await embedded();+ await addPage(engine, 'notes/plan');+ const bytes = Buffer.from('report bytes');+ store('notes/plan/report.pdf', bytes);+ const params = { storage_path: 'notes/plan/report.pdf', filename: 'report.pdf', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length, mime_type: 'application/pdf' };+ expect(await run('file_register', ctxFor(engine), params)).toEqual({ status: 'registered', storage_path: 'notes/plan/report.pdf', size_bytes: bytes.length });+ const next = Buffer.from('report bytes, revised');+ store('notes/plan/report.pdf', next);+ await run('file_register', ctxFor(engine), { ...params, content_hash: sha(next), size_bytes: next.length });+ const rows = await engine.executeRaw<{ page_slug: string; content_hash: string; size_bytes: unknown }>('SELECT page_slug, content_hash, size_bytes FROM files');+ expect(rows.map(r => ({ ...r, size_bytes: Number(r.size_bytes) }))).toEqual([{ page_slug: 'notes/plan', content_hash: sha(next), size_bytes: next.length }]);+ expect(await run('file_register', ctxFor(engine, { dryRun: true }), { ...params, filename: 'other.pdf', storage_path: 'notes/plan/other.pdf' }))+ .toEqual({ dry_run: true, action: 'file_register', storage_path: 'notes/plan/other.pdf' });+ expect(await fileRows(engine)).toBe(1);+ });++ test('records the page and source a registration belongs to, so purging that page in a non-default source finds it', async () => {+ const engine = await embedded();+ await engine.executeRaw("INSERT INTO sources (id, name) VALUES ('source-a', 'source-a') ON CONFLICT DO NOTHING");+ await importFromContent(engine, 'notes/plan', ['---', 'type: note', 'title: plan', '---', '# plan', '', 'Body.'].join('\n'), { sourceId: 'source-a' });+ const bytes = Buffer.from('source-a bytes');+ store('notes/plan/a.txt', bytes);+ await run('file_register', ctxFor(engine, { sourceId: 'source-a' }), { storage_path: 'notes/plan/a.txt', filename: 'a.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length });+ const [page] = await engine.executeRaw<{ id: number }>("SELECT id FROM pages WHERE slug='notes/plan' AND source_id='source-a'");+ expect(await engine.executeRaw('SELECT source_id, page_id FROM files')).toEqual([{ source_id: 'source-a', page_id: page!.id }]);+ expect((await failure(() => run('file_register', ctxFor(engine, { sourceId: 'default' }), { storage_path: 'notes/plan/a.txt', filename: 'a.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('page_not_found');+ });++ test('refuses a registration that would move an existing row to another page, keeping the row as it was', async () => {+ const engine = await embedded();+ await addPage(engine, 'unsorted');+ const old = Buffer.from('loose file');+ const path = `unsorted/${sha(old).slice(0, 8)}-x.txt`;+ store(path, old);+ await run('file_register', ctxFor(engine), { storage_path: path, filename: 'x.txt', content_hash: sha(old), size_bytes: old.length });+ const next = Buffer.from('bytes for the page');+ store(path, next);+ const err = await failure(() => run('file_register', ctxFor(engine), { storage_path: path, filename: `${sha(old).slice(0, 8)}-x.txt`, page_slug: 'unsorted', content_hash: sha(next), size_bytes: next.length }));+ expect(err?.code).toBe('invalid_params');+ expect(await engine.executeRaw('SELECT page_slug, filename, content_hash FROM files')).toEqual([{ page_slug: null, filename: 'x.txt', content_hash: sha(old) }]);+ });++ test('refuses a page that does not exist and an object it cannot read back, recording nothing', async () => {+ const engine = await embedded();+ const bytes = Buffer.from('orphan');+ store('notes/none/o.txt', bytes);+ expect((await failure(() => run('file_register', ctxFor(engine), { storage_path: 'notes/none/o.txt', filename: 'o.txt', page_slug: 'notes/none', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('page_not_found');+ await addPage(engine, 'notes/plan');+ mkdirSync(join(storageRoot, 'notes/plan/dir.txt'), { recursive: true });+ expect((await failure(() => run('file_register', ctxFor(engine), { storage_path: 'notes/plan/dir.txt', filename: 'dir.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length })))?.code).toBe('store_unavailable');+ expect(await fileRows(engine)).toBe(0);+ });++ test('refuses a path other than the file_upload layout, a missing or different object and bad inputs, recording nothing', async () => {+ const engine = await embedded();+ const bytes = Buffer.from('loose bytes');+ const hash = sha(bytes);+ store(`unsorted/${hash.slice(0, 8)}-scan.png`, bytes);+ store('elsewhere/scan.png', bytes);+ const base = { storage_path: `unsorted/${hash.slice(0, 8)}-scan.png`, filename: 'scan.png', content_hash: hash, size_bytes: bytes.length };+ const cases: Array<[Record<string, unknown>, string, OperationContext?]> = [+ [{ ...base, storage_path: 'elsewhere/scan.png' }, 'invalid_params'],+ [{ ...base, storage_path: '../outside/scan.png' }, 'invalid_params'],+ [{ ...base, content_hash: 'A'.repeat(64) }, 'invalid_params'],+ [{ ...base, size_bytes: 1.5 }, 'invalid_params'],+ [{ ...base, filename: '../scan.png' }, 'invalid_params'],+ [{ ...base, size_bytes: bytes.length + 1 }, 'storage_error'],+ [{ ...base, filename: 'other.png', storage_path: `unsorted/${hash.slice(0, 8)}-other.png` }, 'storage_error'],+ ];+ for (const [params, code, ctx] of cases) expect([params, (await failure(() => run('file_register', ctx ?? ctxFor(engine), params)))?.code]).toEqual([params, code]);+ expect(await fileRows(engine)).toBe(0);+ await run('file_register', ctxFor(engine), base);+ expect(await fileRows(engine)).toBe(1);+ });++ test('an engine opened without settings.storage refuses even when the process has a storage config', async () => {+ const engine = await embedded({ storage: undefined });+ const bytes = Buffer.from('loose bytes');+ store(`unsorted/${sha(bytes).slice(0, 8)}-scan.png`, bytes);+ const params = { storage_path: `unsorted/${sha(bytes).slice(0, 8)}-scan.png`, filename: 'scan.png', content_hash: sha(bytes), size_bytes: bytes.length };+ expect((await failure(() => run('file_register', ctxFor(engine, { config: { storage } } as never), params)))?.code).toBe('storage_error');+ expect(await fileRows(engine)).toBe(0);+ });++ test('two brains sharing one store under different prefixes register only their own objects', async () => {+ const a = await embedded({ storage: { ...storage, prefix: 'brain-a' } });+ const b = await embedded({ storage: { ...storage, prefix: 'brain-b' } });+ await Promise.all([addPage(a, 'notes/plan'), addPage(b, 'notes/plan')]);+ const bytes = Buffer.from('brain a bytes');+ store('brain-a/notes/plan/a.txt', bytes);+ const params = { storage_path: 'notes/plan/a.txt', filename: 'a.txt', page_slug: 'notes/plan', content_hash: sha(bytes), size_bytes: bytes.length };+ await run('file_register', ctxFor(a), params);+ expect((await failure(() => run('file_register', ctxFor(b), params)))?.code).toBe('storage_error');+ expect(await fileRows(b)).toBe(0);+ const [{ metadata }] = await a.executeRaw<{ metadata: unknown }>('SELECT metadata FROM files');+ expect(metadata).toEqual({ storage: 'local', prefix: 'brain-a' });+ });++ test('a store that cannot be reached is store_unavailable and records nothing', async () => {+ const engine = await embedded({ storage: unreachable });+ const bytes = Buffer.from('x');+ const error = await failure(() => run('file_register', ctxFor(engine), { storage_path: `unsorted/${sha(bytes).slice(0, 8)}-a.txt`, filename: 'a.txt', content_hash: sha(bytes), size_bytes: 1 }));+ expect(error?.code).toBe('store_unavailable');+ expect(await fileRows(engine)).toBe(0);+ });++ test('an s3 store sends the read-back to its settings endpoint, never one named by AWS_* variables', async () => {+ const hits = { settings: 0, env: 0 };+ const settingsServer = Bun.serve({ port: 0, fetch: () => { hits.settings++; return new Response(null, { status: 404 }); } });+ const envServer = Bun.serve({ port: 0, fetch: () => { hits.env++; return new Response(null, { status: 200 }); } });+ try {+ await withEnv({ AWS_ENDPOINT_URL_S3: `http://127.0.0.1:${envServer.port}`, AWS_ENDPOINT_URL: `http://127.0.0.1:${envServer.port}` }, async () => {+ const engine = await embedded({ storage: { ...unreachable, endpoint: `http://127.0.0.1:${settingsServer.port}` } });+ const bytes = Buffer.from('s3');+ const error = await failure(() => run('file_register', ctxFor(engine), { storage_path: `unsorted/${sha(bytes).slice(0, 8)}-s.txt`, filename: 's.txt', content_hash: sha(bytes), size_bytes: 2 }));+ expect([error?.code, hits.env, hits.settings > 0]).toEqual(['storage_error', 0, true]);+ });+ } finally {+ settingsServer.stop(true);+ envServer.stop(true);+ }+ });+});++describe('who may call file_register', () => {+ let cliEngine: PGLiteEngine;+ beforeAll(async () => {+ cliEngine = new PGLiteEngine();+ await cliEngine.connect({});+ await cliEngine.initSchema();+ });+ afterAll(() => cliEngine.disconnect());++ test('an engine the CLI opens refuses with embedded_engine_required and records nothing', async () => {+ const bytes = Buffer.from('cli');+ store(`unsorted/${sha(bytes).slice(0, 8)}-c.txt`, bytes);+ const params = { storage_path: `unsorted/${sha(bytes).slice(0, 8)}-c.txt`, filename: 'c.txt', content_hash: sha(bytes), size_bytes: 3 };+ expect((await failure(() => run('file_register', ctxFor(cliEngine), params)))?.code).toBe('embedded_engine_required');+ expect(await fileRows(cliEngine)).toBe(0);+ });++ test('it is not listed or callable over MCP, stdio included', async () => {+ const engine = await embedded();+ for (const transport of ['stdio', 'http'] as const) expect(isCallable(operationsByName.file_register!, { transport, surface: 'full', scopes: ['admin'], publishGates: {} })).toBe(false);+ const result = await dispatchToolCall(engine, 'file_register', {}, { remote: true, transport: 'stdio' } as never);+ expect(result.isError).toBe(true);+ expect(await fileRows(engine)).toBe(0);+ });++ test('a caller under the read policy cannot attach to or replace a file on a page it cannot read; a local caller can', async () => {+ const engine = await embedded();+ await importFromContent(engine, 'notes/secret', ['---', 'type: note', 'title: Secret', 'visibility: private', '---', '# Secret', '', 'Private body.'].join('\n'));+ const bytes = Buffer.from('secret attachment');+ store('notes/secret/s.txt', bytes);+ const params = { storage_path: 'notes/secret/s.txt', filename: 's.txt', page_slug: 'notes/secret', content_hash: sha(bytes), size_bytes: bytes.length };+ const remote = ctxFor(engine, { remote: true });+ expect((await failure(() => run('file_register', remote, params)))?.code).toBe('page_not_found');+ expect(await fileRows(engine)).toBe(0);+ await run('file_register', ctxFor(engine), params);+ expect((await failure(() => run('file_register', remote, params)))?.code).toBe('page_not_found');+ expect(await fileRows(engine)).toBe(1);+ });++ test('a page naming its readers stays closed to an unlisted remote caller even with the private-pages hatch open', async () => {+ const engine = await embedded();+ await importFromContent(engine, 'notes/members', ['---', 'type: note', 'title: Members', 'readable_by: [alice-example]', '---', '# Members', '', 'Member body.'].join('\n'));+ const bytes = Buffer.from('member attachment');+ store('notes/members/m.txt', bytes);+ const params = { storage_path: 'notes/members/m.txt', filename: 'm.txt', page_slug: 'notes/members', content_hash: sha(bytes), size_bytes: bytes.length };+ await withEnv({ GBRAIN_REMOTE_PRIVATE_PAGES: '1' }, async () => {+ __resetPrivateVisibilityCacheForTests();+ expect((await failure(() => run('file_register', ctxFor(engine, { remote: true }), params)))?.code).toBe('page_not_found');+ expect(await fileRows(engine)).toBe(0);+ });+ });+});diff --git a/test/fixtures/goldens/cli/tools-json.json b/test/fixtures/goldens/cli/tools-json.jsonindex 25383b2eac..cddfacd14f 100644--- a/test/fixtures/goldens/cli/tools-json.json+++ b/test/fixtures/goldens/cli/tools-json.json@@ -2,8 +2,8 @@"golden": {"exitCode": 0,"stderr": "",- "stdoutBytes": 269760,- "stdoutSha256": "dd3dc10529f90a617d4ddfde2eaffd90fe3892a5d23ffe0f6457aa9557e5babb",+ "stdoutBytes": 271352,+ "stdoutSha256": "3b86f387e3b0e1804aa1a99b97d49d939c69fee5be7861b6ed4fcb7084b3d4a6","toolNames": ["remember","entity",@@ -90,6 +90,7 @@"file_list","file_upload","file_url",+ "file_register","file_verify","submit_job","get_job",@@ -4293,6 +4294,55 @@},"scope": "admin"},+ {+ "description": "Record the files row for an object the host already stored at the path file_upload would use (<page_slug>/<filename>, or unsorted/<first 8 hash chars>-<filename>), after reading it back and matching its size and SHA-256. Use when an embedded host writes bytes to storage itself. Runs only on an engine opened from settings; registering the same path again updates its row.",+ "idempotent": true,+ "local_only": true,+ "mutating": true,+ "name": "file_register",+ "parameters": {+ "content_hash": "string",+ "filename": "string",+ "mime_type": "string?",+ "page_slug": "string?",+ "size_bytes": "integer",+ "storage_path": "string"+ },+ "required_scopes": [],+ "schema": {+ "properties": {+ "content_hash": {+ "description": "SHA-256 of the bytes, 64 lowercase hex characters",+ "type": "string"+ },+ "filename": {+ "type": "string"+ },+ "mime_type": {+ "type": "string"+ },+ "page_slug": {+ "description": "Page the file belongs to",+ "type": "string"+ },+ "size_bytes": {+ "type": "integer"+ },+ "storage_path": {+ "description": "Where the host stored the object; must be the file_upload layout for page_slug, filename and content_hash",+ "type": "string"+ }+ },+ "required": [+ "storage_path",+ "filename",+ "content_hash",+ "size_bytes"+ ],+ "type": "object"+ },+ "scope": "admin"+ },{"description": "Check files rows against the storage backend. Without storage_path, lists up to 100 rows (optionally one page's) with whether each object is present; with it, reads that object back and says whether its size and SHA-256 match the row. Runs only on an engine opened from settings. A backend that fails the check is store_unavailable, never a file reported missing.","idempotent": true,diff --git a/test/operations-coverage-ledger.test.ts b/test/operations-coverage-ledger.test.tsindex bb44f2f0af..45a7b18344 100644--- a/test/operations-coverage-ledger.test.ts+++ b/test/operations-coverage-ledger.test.ts@@ -116,6 +116,7 @@ const LEDGER: Record<string, string> = {file_list: 'test/file-upload-engine-context.test.ts',file_upload: 'test/file-upload-engine-context.test.ts',file_url: 'test/file-upload-engine-context.test.ts',+ file_register: 'test/file-register.test.ts',file_verify: 'test/embedded-file-store.test.ts',submit_job: 'test/submit-queue-state.test.ts',get_job: 'test/jobs-ops-token-redaction.test.ts',diff --git a/test/operations-trust-boundary.test.ts b/test/operations-trust-boundary.test.tsindex 92d795f849..bb649ab230 100644--- a/test/operations-trust-boundary.test.ts+++ b/test/operations-trust-boundary.test.ts@@ -235,6 +235,7 @@ describe('mcpOperations filter — localOnly ops are excluded from the HTTP-expo'entity_identity_unlink','extraction_review','file_list',+ 'file_register','file_upload','file_url','file_verify',@@ -575,7 +576,7 @@ describe('ops-module system-access containment — top-level fs/child_process im// one of the memory verbs from src/core/verbs.ts — nothing else.expect(opsFiles).toContain('files.ts');expect(opsFiles).toContain('jobs.ts');- expect(await contributedOps('files.ts')).toEqual(['file_list', 'file_upload', 'file_url', 'file_verify']);+ expect(await contributedOps('files.ts')).toEqual(['file_list', 'file_register', 'file_upload', 'file_url', 'file_verify']);expect(await contributedOps('jobs.ts')).toContain('submit_job');const attributed = new Set<string>(verbOperations.map(op => op.name));