Skip to content

3D engine that AI agents drive

Rescan asset folders so mid-session downloads can be placed (#2209)

Nguyen Xuan An346564713c75
diff --git a/changes/2209.md b/changes/2209.md
new file mode 100644
index 00000000..4fa3434b
--- /dev/null
+++ b/changes/2209.md
@@ -0,0 +1 @@
+- `scene_asset_catalog` and name placements look at the `--assets` folder again, so a GLB, image or HDRI downloaded mid-session is listed by the next catalog call and placed by name without restarting the server; a removed file leaves the list (#2209)
diff --git a/docs/mcp.md b/docs/mcp.md
index 7ab11baf..2b9441d1 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -64,7 +64,7 @@ The memory guard of a render job or review follows `--memory`: the job fails whe
`--assets` selects an absolute directory containing GLB files.
The asset catalog reads regular files directly inside that directory, with limits of 64 files and 64 MiB per file. One call that places several assets, motion sources and sequence frames sends them to the preview page in one message, so their total is capped at 64 MiB too (`maxCombinedBytes`); grid files (`.67vg`) and look files count too. A call over it fails with an error that names the limit and the total, and the agent places the rest in later calls. The cap also holds for the whole scene: a render job loads every unique asset at once, so `scene_apply` refuses an add-node that would take the scene's unique assets (one asset placed many times counts once) past it, before the preview page loads anything. The scene's grid files count in that total, and a render job or review of a scene already over it fails with the same message before the preview page is handed over, so the page keeps its scene. The server adds up the file sizes first, so a refused call reads no file. Decoding takes 1,000,000 vertices, 4,096 parts and 1,024 joints at most; `scene_capabilities` reports every number under `assets`. They are lower than the `loadGltf` defaults (256 MiB, 8,000,000 vertices) because the server hands the whole file to the preview page as one base64 string, which crashed the page from 80 MiB in a local Chrome test. A file over a limit fails with an error that names the limit and the value it hit; a larger scene loads through `loadGltf` in code (see [gltf.md](gltf.md)). All limits live in `src/agent-asset-limits.js`.
-Asset IDs bind file contents through SHA-256. Changed files require a runtime restart before they receive refreshed IDs.
+Asset IDs bind file contents through SHA-256. The `--assets` folder is looked at again (names, sizes and modification times only, no file is read) whenever `scene_asset_catalog` is called and whenever a recipe, edit or icon item names a GLB or image sequence by name, so a GLB, image or HDRI copied in during the session is listed by the next catalog call and can be placed by name without a restart. A removed file leaves the list, and a rewritten file is listed under its new ID. An ID already pinned in a scene keeps its bytes check: if that file changed on disk, the read fails and the scene needs the new ID. The same count and size limits apply on every look. Installed packs are still checked once per runtime.
Catalog imports retain supported textures, skins, morphs and animation clips.
A Draco-compressed GLB (`KHR_draco_mesh_compression`, common in Sketchfab and NASA downloads) loads as it is: the server decodes its meshes when it reads the catalog, with the optional `draco3d` package, and the preview page receives a plain GLB. Install it where the MCP server runs (`npm install draco3d`). Without it, `scene_asset_catalog` lists the file under `skipped` with that advice, or re-export the file without Draco (in Blender, untick Compression in the glTF export). The 64 MiB limit applies to the decoded file, which is several times larger than the compressed one. A file whose Draco data holds a different point or face count than its accessors declare is skipped with both numbers.
A GLB with KTX2 textures (`KHR_texture_basisu`, what gltf-transform `optimize` and Sketchfab's optimised exports write) loads as it is too: the server transcodes each KTX2 image to a PNG (base mip level only, at most 4096 x 4096 pixels per image and 64 million pixels per file) with the Basis transcoder that the optional `three` package ships, and the preview page receives a plain GLB. Install it where the MCP server runs (`npm install three`). Without it, `scene_asset_catalog` lists the file under `skipped` with that advice, or re-export the textures as PNG or JPEG. The 64 MiB limit applies to the transcoded file.
diff --git a/scripts/agent-asset-catalog.mjs b/scripts/agent-asset-catalog.mjs
index a503c99e..6c073add 100644
--- a/scripts/agent-asset-catalog.mjs
+++ b/scripts/agent-asset-catalog.mjs
@@ -357,6 +357,30 @@ async function hashFrames(sequence, frameEntries) {
return frameIds;
}
+/**
+ * A cheap summary of the catalog files in `root` (name, size and modification time, no bytes read), so the runtime can tell whether a
+ * file was added, removed or rewritten since the catalog was built (#2209). Only the files the catalog lists are looked at, and at
+ * most as many as the catalog accepts plus one, so a huge folder costs a bounded number of stats.
+ */
+export async function fingerprintAssetDirectory(root) {
+ let directory;
+ let listing;
+ try {
+ directory = await realpath(root);
+ listing = await readdir(directory, { withFileTypes: true });
+ } catch {
+ // An unreadable folder has no fingerprint; building the catalog reports the real error with its own advice.
+ return 'unreadable';
+ }
+ const names = listing.filter(entry => /\.(glb|png|jpe?g|webp|hdr|cube)$/i.test(entry.name)).map(entry => entry.name).sort();
+ const parts = [`${names.length}`];
+ for (const name of names.slice(0, MAXIMUM_ASSETS + TEXTURE_SEQUENCE_LIMITS.maxFrames + 1)) {
+ const status = await lstat(join(directory, name)).catch(() => null);
+ parts.push(`${name}:${status ? `${status.size}:${status.mtimeMs}` : 'gone'}`);
+ }
+ return parts.join('|');
+}
+
/**
* Explicit local directory catalog. IDs bind bytes, rather than agent-supplied file paths.
* `packFiles` adds the validated GLB, image, .hdr and .cube files of installed packs (scripts/agent-packs.mjs),
diff --git a/scripts/agent-runtime.mjs b/scripts/agent-runtime.mjs
index 98d3f737..f87a7601 100644
--- a/scripts/agent-runtime.mjs
+++ b/scripts/agent-runtime.mjs
@@ -24,7 +24,7 @@ import { chooseNativeTemporal, createTemporalAccumulator, temporalFrameSeed } fr
import { readNativeCapacity } from './native-capacity.mjs';
import { chooseNativeInstancing, chooseNativeKernel, chooseTraceBackend, createNativeTracerLocator, fetchNativeExport, isNativeTimeLimitError, traceFrameNative, traceTexelsNative } from './agent-native-trace.mjs';
import { addonCapabilities, addonFileFor, addonImportMap } from './mcp-addons.mjs';
-import { createAgentAssetCatalog } from './agent-asset-catalog.mjs';
+import { createAgentAssetCatalog, fingerprintAssetDirectory } from './agent-asset-catalog.mjs';
import { launchWithShaderCache, shaderCacheDirectory } from './agent-shader-cache.mjs';
import { createPackRegistry } from './agent-packs.mjs';
import { MAXIMUM_INSTALLED_PACKS, MAXIMUM_PACK_BYTES, MAXIMUM_PACK_FILES, MAXIMUM_PACK_FILE_BYTES, MAXIMUM_PACK_JSON_BYTES, MAXIMUM_PACK_ITEMS_PER_KIND, PACK_CONTENT_KINDS } from '../src/pack-manifest.js';
@@ -197,6 +197,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
let queue = Promise.resolve();
let closing;
let catalogPromise;
+ let catalogFingerprint;
let packsPromise;
let jobsPromise;
// Set by submitJob for the one job that may run; runJob takes it. Kept in the job folder, the history never leaves the machine.
@@ -370,6 +371,25 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
catalogPromise ??= packs().then(registry => createAgentAssetCatalog(assetRoot ?? null, registry?.files));
return catalogPromise;
}
+ /**
+ * The catalog after a stat-only look at the --assets folder: a file that appeared, vanished or was rewritten since the last build
+ * makes a new catalog (#2209), so a download made mid-session is listed and placeable by name. Used where an agent asks for the
+ * list or names a file; reads by asset ID keep the catalog they were pinned against.
+ */
+ async function rescannedCatalog() {
+ if (assetRoot) {
+ const fingerprint = await fingerprintAssetDirectory(assetRoot);
+ if (fingerprint !== catalogFingerprint) {
+ catalogFingerprint = fingerprint;
+ catalogPromise = undefined;
+ }
+ }
+ const ready = catalog();
+ ready.catch(() => {
+ if (catalogPromise === ready) catalogPromise = undefined;
+ });
+ return ready;
+ }
// Installed data-only packs (docs/packs.md): checked once per runtime; their GLB, image, look and HDRI files join the catalog.
function packs() {
if (!packsRoot) return Promise.resolve(null);
@@ -379,15 +399,15 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
/** Replaces each `asset: { name }` placement of a recipe or edit list with its catalog id, so everything after this holds the pinned hash. */
async function pinNamedAssets(value) {
- if (namesCatalogFiles(value)) resolveAssetPlacementNames(value, (await catalog()).resolveName);
- if (namesCatalogSequences(value)) await resolveCatalogSequenceNames(value, pattern => catalog().then(assets => assets.resolveSequence(pattern)));
+ if (namesCatalogFiles(value)) resolveAssetPlacementNames(value, (await rescannedCatalog()).resolveName);
+ if (namesCatalogSequences(value)) await resolveCatalogSequenceNames(value, pattern => rescannedCatalog().then(assets => assets.resolveSequence(pattern)));
}
/** The same for the asset recipe of scene_create_asset: `assetName` (a catalog file name) becomes `assetId`. */
async function pinNamedAssetRecipe(recipe) {
if (recipe?.assetName === undefined) return;
if (recipe.assetId !== undefined) throw new TypeError('Pass exactly one of recipe.assetId (the pinned SHA-256) or recipe.assetName (the catalog file name), not both.');
- recipe.assetId = (await catalog()).resolveName(recipe.assetName);
+ recipe.assetId = (await rescannedCatalog()).resolveName(recipe.assetName);
delete recipe.assetName;
}
@@ -441,7 +461,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
*/
async function pinIconItems(settings) {
if (!Array.isArray(settings.items)) return settings;
- const readyCatalog = await catalog();
+ const readyCatalog = await rescannedCatalog();
const items = await Promise.all(settings.items.map(async item => {
if (typeof item !== 'string') return item;
if (/^[0-9a-f]{64}$/.test(item)) {
@@ -1290,7 +1310,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
return { pack, [kind === 'recipes' ? 'recipe' : 'material']: recipe ?? material, json: (await packs()).read(kind, pack, recipe ?? material) };
}
if (!assetRoot && !packsRoot) return { configured: false, assets: [] };
- return { configured: true, ...(await catalog()).list(), ...(packsRoot && { packs: (await packs()).list() }) };
+ return { configured: true, ...(await rescannedCatalog()).list(), ...(packsRoot && { packs: (await packs()).list() }) };
});
},
createAsset({ recipe }) {
diff --git a/tests/agent-runtime.test.js b/tests/agent-runtime.test.js
index 7763615c..8f39595d 100644
--- a/tests/agent-runtime.test.js
+++ b/tests/agent-runtime.test.js
@@ -1078,6 +1078,56 @@ runTest('scene_create_asset opens a catalog GLB by assetName and refuses assetNa
await assert.rejects(runtime.createAsset({ recipe: { ...assetRecipe, assetId: sheepId, assetName: 'sheep_body.glb' } }), /exactly one of recipe\.assetId/);
});
+// Files that arrive or leave the --assets folder after the server started (#2209).
+async function rescanHarness(context) {
+ const directory = await mkdtemp(join(tmpdir(), 'crucible-rescan-assets-'));
+ context.after(() => rm(directory, { recursive: true, force: true }));
+ await cp(new URL('./fixtures/kenney-boat/', import.meta.url), directory, { recursive: true });
+ const boatBytes = await readFile(join(directory, 'boat-fishing-small.glb'));
+ await rm(join(directory, 'boat-fishing-small.glb'));
+ await writeFile(join(directory, 'first.glb'), boatBytes);
+ const harness = await reviewRuntimeHarness({ runtimeOptions: { assets: directory } });
+ const addGlb = (name, marker) => writeFile(join(directory, name), withDistinctDocument(boatBytes, marker));
+ return { ...harness, directory, addGlb };
+}
+
+function radianceFile() {
+ return Buffer.concat([Buffer.from('#?RADIANCE\nFORMAT=32-bit_rle_rgbe\n\n-Y 2 +X 4\n'), Buffer.alloc(2 * 4 * 4, 128)]);
+}
+
+const listedNames = async runtime => (await runtime.assetCatalog()).assets.map(asset => asset.name);
+
+runTest('a GLB and an HDRI copied into --assets after start are listed by the next catalog call and placed by name (#2209)', async context => {
+ const { runtime, sessionId, sourceCalls, directory, addGlb } = await rescanHarness(context);
+ assert.deepEqual(await listedNames(runtime), ['first.glb']);
+ await addGlb('downloaded.glb', 'downloaded');
+ await writeFile(join(directory, 'sky.hdr'), radianceFile());
+ const listing = await runtime.assetCatalog();
+ assert.deepEqual(listing.assets.map(asset => asset.name).sort(), ['downloaded.glb', 'first.glb', 'sky.hdr']);
+ const downloadedId = listing.assets.find(asset => asset.name === 'downloaded.glb').assetId;
+ await runtime.apply({ sessionId, operations: [{ type: 'add-node', node: { id: 'new-boat', asset: { name: 'downloaded.glb' } } }] });
+ assert.deepEqual(sourceCalls.filter(call => call.method === 'apply').at(-1).input.operations[0].node.asset, { id: downloadedId });
+});
+
+runTest('an asset name that appeared since the last catalog call is found without listing the catalog first (#2209)', async context => {
+ const { runtime, sessionId, sourceCalls, addGlb } = await rescanHarness(context);
+ await listedNames(runtime);
+ await addGlb('late.glb', 'late');
+ await runtime.apply({ sessionId, operations: [{ type: 'add-node', node: { id: 'late-boat', asset: { name: 'late.glb' } } }] });
+ assert.equal(sourceCalls.filter(call => call.method === 'apply').at(-1).input.operations[0].node.asset.id.length, 64);
+});
+
+runTest('a file removed from --assets leaves the catalog and a rewritten file gets its new id (#2209)', async context => {
+ const { runtime, directory, addGlb } = await rescanHarness(context);
+ await addGlb('temporary.glb', 'one');
+ const before = (await runtime.assetCatalog()).assets.find(asset => asset.name === 'temporary.glb').assetId;
+ await addGlb('temporary.glb', 'two');
+ const after = (await runtime.assetCatalog()).assets.find(asset => asset.name === 'temporary.glb').assetId;
+ assert.notEqual(after, before);
+ await rm(join(directory, 'temporary.glb'));
+ assert.deepEqual(await listedNames(runtime), ['first.glb']);
+});
+
// The same GLB with one extra field in its JSON chunk, so the catalog gives it another asset id.
function withDistinctDocument(bytes, marker) {
const jsonLength = bytes.readUInt32LE(12);