3D engine that AI agents drive
Add image sequence colour maps to recipe materials (part of #339)
Nguyen Xuan Ancc2e2127c2db
diff --git a/changes/339-sequence.md b/changes/339-sequence.mdnew file mode 100644index 00000000..79341f82--- /dev/null+++ b/changes/339-sequence.md@@ -0,0 +1 @@+- Add `map: { sequence: { frames, fps, loop } }` to recipe materials: a colour map that plays catalog PNG, JPEG or WebP images, with the frame `floor(motionTime * fps)` shared by `scene_look`, `scene_review` and render jobs, `updateTexture` only on a frame change, and `textureSequences` in `scene_inspect` (closes #339). The asset catalog now lists images with `kind` and `mimeType`.diff --git a/changes/339-sequence.size.md b/changes/339-sequence.size.mdnew file mode 100644index 00000000..4ba82c47--- /dev/null+++ b/changes/339-sequence.size.md@@ -0,0 +1,8 @@+- tool-total: one new material field, map.sequence, in the shared material schema that scene_create, scene_apply (set-material and add-material) and scene_create_asset repeat; the prose lives in scene_capabilities and docs.+- tool:scene_create: material.map.sequence { frames, fps, loop }, about 480 bytes of schema.+- tool:scene_apply: the same field in set-material and add-material.+- tool:scene_create_asset: the same field in the shared material schema.+- result:scene_capabilities: textureSequence documents the field, its frame rule, covered maps, image formats and limits for agents.+- scene:agent-scene: the recipe field map.sequence adds validation hooks, a per-material sequence runtime in the material handle and a textureSequences report to the scene entry (about 300 bytes gzip); the frame logic itself is in the lazy sequences module.+- scene:agent-scene+lazy: the new sequences module (frame selection, validation, descriptor runtime), the catalog image kinds and the page decoder for frames.+- goal:agent-scene: needs An to accept, it is about 300 bytes gzip over the goal for an image sequence map that cannot live outside the scene entry. If not accepted, the field can move behind a lazy module hook in a follow-up.diff --git a/docs/agent-scene.md b/docs/agent-scene.mdindex b8813ed4..32447bc2 100644--- a/docs/agent-scene.md+++ b/docs/agent-scene.md@@ -100,7 +100,7 @@ Modeling point inputs have a maximum of 4096 points.An extrusion has at most 128 holes and 4096 total contour points.Shared resource IDs generate geometry once, regardless of the number of mesh nodes.-Materials contain `id`, `tag`, `color`, `metallic`, `roughness`, `emission`, optional `emissiveColor`, `opacity`, `alphaMode`, `doubleSided` and `pattern`.+Materials contain `id`, `tag`, `color`, `metallic`, `roughness`, `emission`, optional `emissiveColor`, `opacity`, `alphaMode`, `doubleSided`, `pattern` and `map`.The defaults are `surface`, `[0.5, 0.5, 0.5]`, `0`, `0.5`, `0`, opacity `1`, `opaque` and `false`.The tag can be `surface`, `unlit`, `liquid` or `foam` and remains fixed during editing.`liquid` refracts and tints what lies behind it: its color is the light one metre lets through, so `[0.56, 0.84, 0.88]` is clear water. `foam` draws soft blended droplets.@@ -191,6 +191,21 @@ texture repeats outside 0 to 1 and tiles. `size` and `axis` only apply to `box`.fails at render and at `scene_export_glb` with a message that names `uv_unwrap`. GLB export writes the baked graph texturesand the geometry UVs.+### Image sequence maps++`map: { sequence: { frames, fps, loop } }` gives a `surface` material a colour map that plays a list of images, such as the output of a video or a simulation. It covers the colour map only (`map`; normal, roughness and emissive maps stay with `pattern`), needs geometry with UVs (`grid`, `box`, a `uv_unwrap`ped mesh), and does not combine with `pattern`. `color` defaults to white and tints the frames.++```json+{ "id": "screen", "map": { "sequence": { "frames": ["<catalog image id>", "<catalog image id>"], "fps": 12 } } }+```++- `frames` lists 2 to 64 IDs from `scene_asset_catalog`. The `--assets` folder takes PNG, JPEG and WebP files next to GLBs, with the same 16 MiB file budget, content-bound SHA-256 IDs and no links. An extension that does not match the file signature rejects the catalog. Entries report `kind: "image"` and `mimeType`; GLBs report `kind: "glb"`. No remote URLs are read.+- All frames share one size, at most 1024 pixels a side, and a scene decodes at most 64 MiB of frames. Frames are read without colour conversion and should be opaque.+- `fps` is above 0 and at most 60. The frame at scene time `t` is `floor(t * fps)`: wrapped around the list when `loop` is true (the default), held on the last frame when `loop` is false. Frame 0 shows before any motion time is set.+- The clock is `motionTime`, the one `scene_look`, `scene_review` and `motion` use. A seek is deterministic: the same time gives the same frame and the same PNG bytes. `scene_render_submit` reads it from `settings.motion.start` plus the elapsed time (or a timeline shot), so a still or video frame shows the frame that `scene_look` shows at that time. A recipe with a sequence and no authored `motion` accepts `settings.motion`.+- The renderer uploads a frame with `updateTexture` only when the frame index changes. `scene_inspect` reports `textureSequences`: material, frame count, `fps`, `loop`, current `index` and its asset ID.+- `set-material` and `add-material` take `map`; the frames must be catalog images, which the host decodes before the edit.+Nodes contain `id`, optional `label`, `parent`, `translation`, `rotation`, and `scale`.Mesh nodes also require both `geometry` and `material` resource IDs.Group nodes have neither resource reference.@@ -302,7 +317,7 @@ Final resource references and hierarchy must be valid before publication.| Operation | Fields || --- | --- || `set-node` | `id`, optional `translation`, `rotation`, `scale`, `parent`, `label`, `geometry`, `material` |-| `set-material` | `id`, optional `color`, `metallic`, `roughness`, `emission`, `emissiveColor`, `pattern` |+| `set-material` | `id`, optional `color`, `metallic`, `roughness`, `emission`, `emissiveColor`, `pattern`, `map` || `set-camera` | Optional `eye`, `target`, `up`, `fov`, `near`, `far` directly on the command || `add-geometry` | `geometry`, containing a complete geometry record || `remove-geometry` | `id` |diff --git a/package.json b/package.jsonindex ae8641cb..e28363e4 100644--- a/package.json+++ b/package.json@@ -129,6 +129,7 @@"smoke:view-transform": "node scripts/smoke-view-transform.mjs","smoke:agent-assets": "node scripts/smoke-agent-assets.mjs","smoke:agent-render-jobs": "node scripts/smoke-agent-render-jobs.mjs",+ "smoke:agent-texture-sequence": "node scripts/smoke-agent-texture-sequence.mjs","smoke:render-passes": "node scripts/smoke-render-passes.mjs","smoke:path-tracer": "node scripts/smoke-path-tracer.mjs","smoke:film-timeline": "node scripts/smoke-film-timeline.mjs",diff --git a/scripts/agent-asset-catalog.mjs b/scripts/agent-asset-catalog.mjsindex 2fc87a20..f8efd683 100644--- a/scripts/agent-asset-catalog.mjs+++ b/scripts/agent-asset-catalog.mjs@@ -5,6 +5,17 @@ import { basename, isAbsolute, join, relative, sep } from 'node:path';const MAXIMUM_ASSETS = 64;const MAXIMUM_BYTES = 16 * 1024 * 1024;+const IMAGE_EXTENSIONS = { png: /\.png$/i, jpeg: /\.jpe?g$/i, webp: /\.webp$/i };+const IMAGE_MIME = { png: 'image/png', jpeg: 'image/jpeg', webp: 'image/webp' };++/** 'png', 'jpeg' or 'webp' from the file signature, else null. Catalog images are decoded by the preview, never by the server. */+export function imageKind(bytes) {+ if (bytes.length >= 8 && bytes.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) return 'png';+ if (bytes.length >= 3 && bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) return 'jpeg';+ if (bytes.length >= 12 && bytes.toString('latin1', 0, 4) === 'RIFF' && bytes.toString('latin1', 8, 12) === 'WEBP' && bytes.readUInt32LE(4) + 8 === bytes.length) return 'webp';+ return null;+}+function selfContainedGlb(bytes) {if (bytes.length < 20 || bytes.readUInt32LE(0) !== 0x46546c67 || bytes.readUInt32LE(4) !== 2 || bytes.readUInt32LE(8) !== bytes.length) {throw new Error('Asset must be a complete glTF 2.0 binary file.');@@ -69,26 +80,33 @@ export async function createAgentAssetCatalog(root) {if (after.size !== current.size || after.mtimeMs !== current.mtimeMs || after.ctimeMs !== current.ctimeMs) {throw new Error('Asset changed during reading.');}- const document = selfContainedGlb(bytes);+ const imageFormat = imageKind(bytes);+ if (imageFormat && !IMAGE_EXTENSIONS[imageFormat].test(filename)) throw new Error('Image extension must match its PNG, JPEG or WebP content.');+ if (!imageFormat && !/\.glb$/i.test(filename)) throw new Error('Catalog files must be GLB, PNG, JPEG or WebP, with a matching extension.');+ const document = imageFormat ? null : selfContainedGlb(bytes);const assetId = createHash('sha256').update(bytes).digest('hex');- return { bytes, document, assetId };+ return { bytes, document, assetId, imageFormat };} finally {await handle.close();}}const files = (await readdir(directory, { withFileTypes: true }))- .filter(entry => /\.glb$/i.test(entry.name)).sort((left, right) => left.name.localeCompare(right.name));+ .filter(entry => /\.(glb|png|jpe?g|webp)$/i.test(entry.name)).sort((left, right) => left.name.localeCompare(right.name));if (files.length > MAXIMUM_ASSETS) throw new Error('Asset catalog exceeds the 64-file budget.');for (const file of files) {- const { bytes, document, assetId } = await readFile(file.name);+ const { bytes, document, assetId, imageFormat } = await readFile(file.name);if (entries.has(assetId)) continue;+ if (imageFormat) {+ entries.set(assetId, { filename: file.name, metadata: Object.freeze({ assetId, name: file.name, bytes: bytes.length, kind: 'image', mimeType: IMAGE_MIME[imageFormat] }) });+ continue;+ }const animations = document.animations ?? [];if (!Array.isArray(animations)) throw new Error('GLB animations must be an array.');entries.set(assetId, {filename: file.name,metadata: Object.freeze({- assetId, name: file.name, bytes: bytes.length,+ assetId, name: file.name, bytes: bytes.length, kind: 'glb',meshes: document.meshes?.length ?? 0,skins: document.skins?.length ?? 0,images: document.images?.length ?? 0,@@ -107,7 +125,7 @@ export async function createAgentAssetCatalog(root) {if (!entry) throw new Error('Unknown asset ID. Read the configured asset catalog first.');const result = await readFile(entry.filename);if (result.assetId !== assetId) throw new Error('Asset bytes changed since catalog creation. Restart the runtime to refresh the catalog.');- if ((result.document.animations?.length ?? 0) > 256) throw new Error('Asset exceeds the 256-clip decoding budget.');+ if ((result.document?.animations?.length ?? 0) > 256) throw new Error('Asset exceeds the 256-clip decoding budget.');return result.bytes;},});diff --git a/scripts/agent-preview-browser.js b/scripts/agent-preview-browser.jsindex 93225bc6..4952c3ba 100644--- a/scripts/agent-preview-browser.js+++ b/scripts/agent-preview-browser.js@@ -16,6 +16,8 @@ import { createAgentAssetScene } from '../src/agent-asset.js';import { parseGltf } from '../src/gltf.js';import { assetCopyKey, recipeAssetNeeds } from '../src/agent-scene-assets.js';import { exportGlb } from '../src/export.js';+import '../src/texture-update.js';+import { TEXTURE_SEQUENCE_LIMITS } from '../src/texture-sequence-limits.js';import { weightTracksFromGltfAnimation } from '../src/export-skeleton.js';import { clipFromGltfAnimation } from '../src/skeleton-pose.js';import { sessionProblem } from '../src/agent-errors.js';@@ -97,6 +99,59 @@ function renderInfo(renderer) {};}+// Sequence frames arrive as catalog image bytes keyed by asset id. They are decoded here, without colour conversion or+// premultiplication, so the same file gives the same pixels in a preview and in a render job.+async function decodeSequenceFrame(id, base64) {+ const bitmap = await createImageBitmap(new Blob([bytesFromBase64(base64)]), { premultiplyAlpha: 'none', colorSpaceConversion: 'none' });+ try {+ if (bitmap.width > TEXTURE_SEQUENCE_LIMITS.maxSide || bitmap.height > TEXTURE_SEQUENCE_LIMITS.maxSide) {+ throw new RangeError(`Sequence frame ${id.slice(0, 12)} is ${bitmap.width}x${bitmap.height}; frames are at most ${TEXTURE_SEQUENCE_LIMITS.maxSide} pixels on a side.`);+ }+ const surface = new OffscreenCanvas(bitmap.width, bitmap.height);+ const context = surface.getContext('2d', { willReadFrequently: true });+ context.drawImage(bitmap, 0, 0);+ const { data } = context.getImageData(0, 0, bitmap.width, bitmap.height);+ return { width: bitmap.width, height: bitmap.height, data };+ } finally {+ bitmap.close();+ }+}++async function loadSequenceFrames(pool, sequenceData) {+ for (const [id, base64] of Object.entries(sequenceData ?? {})) {+ if (pool.has(id)) continue;+ try {+ pool.set(id, await decodeSequenceFrame(id, base64));+ } catch (error) {+ if (error instanceof RangeError) throw error;+ throw new Error(`Sequence frame ${id.slice(0, 12)} is not a PNG, JPEG or WebP image the browser can decode.`);+ }+ const decodedBytes = [...pool.values()].reduce((total, frame) => total + frame.data.length, 0);+ if (decodedBytes > TEXTURE_SEQUENCE_LIMITS.maxDecodedBytes) {+ pool.delete(id);+ throw new RangeError(`Sequence frames decode to more than ${TEXTURE_SEQUENCE_LIMITS.maxDecodedBytes / 1048576} MiB. Use fewer or smaller frames.`);+ }+ }+ return pool;+}++// Each frame change is one updateTexture per renderer. A renderer starts on frame 0, the image its descriptor holds.+const uploadedFrames = new WeakMap();+function syncSequenceTextures(record) {+ const uploaded = uploadedFrames.get(record.renderer) ?? new Map();+ uploadedFrames.set(record.renderer, uploaded);+ for (const { descriptor, index, image } of record.scene.sequenceTextures?.() ?? []) {+ if ((uploaded.get(descriptor) ?? 0) === index) continue;+ if (!record.renderer.textures.has(descriptor)) continue;+ record.renderer.updateTexture(descriptor, { image });+ uploaded.set(descriptor, index);+ }+}++function sceneIsTextured(scene) {+ return hasPatternTextures(scene.parts) || (scene.sequenceTextures?.().length ?? 0) > 0;+}+function hasPatternTextures(parts) {return parts.some(part => part.material.boxMapScale || part.material.emissiveMap);}@@ -148,7 +203,7 @@ function draw(record, time = 0, camera = record.scene.camera, frameOptions = {})// Shrink the existing target first so a smaller look can recover a staging-budget failure.record.renderer.resize();if (record.rendererTopology !== record.scene.topologyRevision) {- const textured = hasPatternTextures(record.scene.parts);+ const textured = sceneIsTextured(record.scene);// replaceParts refuses skinned and morphed parts, so an edited imported part takes a new renderer as well.if (textured || record.rendererTextured || record.asset) {rebuildRenderer(record);@@ -163,6 +218,7 @@ function draw(record, time = 0, camera = record.scene.camera, frameOptions = {})}syncAutomaticGround(record);syncShadowCascades(record);+ syncSequenceTextures(record);const environment = record.scene.environment;const effects = record.scene.effects;const shadow = record.scene.shadow;@@ -276,7 +332,7 @@ function previousCameraFor(record, motionTime) {return previousCamera;}-function createPreview(scene, { sessionId, memoryBudgetMiB, backend }, asset, placedAssets) {+function createPreview(scene, { sessionId, memoryBudgetMiB, backend }, asset, placedAssets, sequenceFrames) {if (active) {throw new Error('Dispose the current scene before creating another.');}@@ -285,7 +341,7 @@ function createPreview(scene, { sessionId, memoryBudgetMiB, backend }, asset, pltry {renderer = initializeRenderer(scene, { memoryBudgetMiB, backend });const record = { scene, renderer, asset, placedAssets, sessionId, options: { memoryBudgetMiB, backend },- checkpoints: new Map(), branches: createBranchStore(), rendererTopology: scene.topologyRevision, rendererTextured: hasPatternTextures(scene.parts) };+ checkpoints: new Map(), branches: createBranchStore(), rendererTopology: scene.topologyRevision, rendererTextured: sceneIsTextured(scene), sequenceFrames };draw(record);const result = status(record);if (backend === 'gpu' && /swiftshader|llvmpipe|software/i.test(result.render.graphics)) {@@ -388,9 +444,9 @@ function useKernelWorkers() {return kernelWorkersAvailable() && globalThis.__67jsKernelWorkers !== 'off';}-function buildSceneOffMainThread(recipe) {+function buildSceneOffMainThread(recipe, options) {return buildWithKernels(() => {- const scene = createAgentScene(recipe);+ const scene = createAgentScene(recipe, {}, options);for (const part of scene.parts) {void part.material.map;}@@ -596,7 +652,7 @@ window.__agentPreview = {try {record.renderer = initializeRenderer(record.scene, record.options);record.rendererTopology = record.scene.topologyRevision;- record.rendererTextured = hasPatternTextures(record.scene.parts);+ record.rendererTextured = sceneIsTextured(record.scene);record.shadowKey = null;draw(record);return status(record);@@ -681,13 +737,14 @@ window.__agentPreview = {try {await loadAllAgentSceneModules();await loadPlacedAssets(pool, input.recipe, input.assetData);+ const sequenceFrames = await loadSequenceFrames(new Map(), input.sequenceData);// Placed assets are decoded on this thread, so the scene is built here and not on the kernel worker.return await afterMeshBooleans(input.recipe, () => {startBuildReport();const startedAt = performance.now();- const scene = createAgentScene(input.recipe, {}, { assets: pool });+ const scene = createAgentScene(input.recipe, {}, { assets: pool, sequenceFrames });finishBuildReport(startedAt);- return createPreview(scene, input, undefined, pool);+ return createPreview(scene, input, undefined, pool, sequenceFrames);});} catch (error) {disposePlacedAssets(pool);@@ -697,17 +754,18 @@ window.__agentPreview = {create(input) {if (active) throw new Error('Dispose the current scene before creating another.');if (recipeAssetNeeds(input.recipe).length) return this.createWithAssets(input);- return afterMeshBooleans(input.recipe, () => {+ return afterMeshBooleans(input.recipe, async () => {+ const sequenceFrames = await loadSequenceFrames(new Map(), input.sequenceData);startBuildReport();const startedAt = performance.now();if (!useKernelWorkers()) {- const scene = createAgentScene(input.recipe);+ const scene = createAgentScene(input.recipe, {}, { sequenceFrames });finishBuildReport(startedAt);- return createPreview(scene, input);+ return createPreview(scene, input, undefined, undefined, sequenceFrames);}- return buildSceneOffMainThread(input.recipe).then(scene => {+ return buildSceneOffMainThread(input.recipe, { sequenceFrames }).then(scene => {finishBuildReport(startedAt);- return createPreview(scene, input);+ return createPreview(scene, input, undefined, undefined, sequenceFrames);});});},@@ -737,6 +795,11 @@ window.__agentPreview = {return { sessionId, ...scene.inspectMeshes(options) };},apply(input) {+ if (input.sequenceData) {+ const { sequenceFrames } = requireSession(input.sessionId);+ if (!sequenceFrames) throw new Error('This scene was created from an imported asset and has no image sequences.');+ return loadSequenceFrames(sequenceFrames, input.sequenceData).then(() => this.apply({ ...input, sequenceData: undefined }));+ }if (input.sources) {const { scene } = requireSession(input.sessionId);return decodeMotionSources(input.sources).then(decoded => {diff --git a/scripts/agent-render-jobs.mjs b/scripts/agent-render-jobs.mjsindex 42e963f4..7f7eb07a 100644--- a/scripts/agent-render-jobs.mjs+++ b/scripts/agent-render-jobs.mjs@@ -108,16 +108,18 @@ function settingsFrom(input, clips, recipe) {if (input.audio !== undefined) result.audio = audioFrom(input.audio, input.kind);if (input.motion !== undefined) {objectKeys(input.motion, ['start'], 'motion');- if (!['67js.scene', '67js.asset'].includes(recipe?.format) || !recipe.motion) {- throw new RangeError('Motion settings require a recipe with authored motion.');+ // A map sequence reads the same clock, so a recipe with one takes motion settings without authored motion.+ const sequenceOnly = !recipe?.motion && JSON.stringify(recipe?.materials ?? []).includes('"sequence":{');+ if (!['67js.scene', '67js.asset'].includes(recipe?.format) || (!recipe.motion && !sequenceOnly)) {+ throw new RangeError('Motion settings require a recipe with authored motion or a map sequence.');}- const motionDuration = boundedNumber(recipe.motion.duration, 0.1, 30, 'motion duration');- if (recipe.motion.loop !== undefined && typeof recipe.motion.loop !== 'boolean') {+ const motionDuration = sequenceOnly ? 3600 : boundedNumber(recipe.motion.duration, 0.1, 30, 'motion duration');+ if (recipe.motion?.loop !== undefined && typeof recipe.motion.loop !== 'boolean') {throw new TypeError('Motion loop must be boolean.');}const start = boundedNumber(input.motion.start === undefined ? 0 : input.motion.start, 0, 3600, 'motion start');boundedNumber(start + lastSampleTime, 0, 3600, 'motion end');- if (!recipe.motion.loop && start + lastSampleTime > motionDuration) {+ if (!sequenceOnly && !recipe.motion.loop && start + lastSampleTime > motionDuration) {throw new RangeError('Motion range exceeds the authored duration.');}result.motion = { start };diff --git a/scripts/agent-runtime.mjs b/scripts/agent-runtime.mjsindex a6af7f77..f7339f95 100644--- a/scripts/agent-runtime.mjs+++ b/scripts/agent-runtime.mjs@@ -11,6 +11,8 @@ import { PATH_TRACER_CAPABILITIES } from '../src/path-tracer-capabilities.js';import { chooseDenoiser, finishWithOidn } from './agent-path-trace-denoise.mjs';import { createAgentAssetCatalog } from './agent-asset-catalog.mjs';import { recipeAssetNeeds } from '../src/agent-scene-assets.js';+import { sequenceFrameIds } from '../src/agent-texture-sequence.js';+import { TEXTURE_SEQUENCE_CAPABILITIES } from '../src/texture-sequence-limits.js';import { createAgentRenderJobs, EXPORT_DIRECTORY_NAME } from './agent-render-jobs.mjs';import { createFrameEncoder } from './frame-encoder.mjs';import { DEFAULT_PAGE_ENCODER_QUANTIZER, createFrameReceiver, startVideoEncoderInPage } from './gpu-frame-reader.mjs';@@ -137,9 +139,16 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow/** Bytes of the catalog assets a recipe or edit list places, base64 by asset id, for the page that decodes them. */async function placedAssetData(value) {const needs = recipeAssetNeeds(value);- if (!needs.length) return {};const entries = await Promise.all(needs.map(async ({ id }) => [id, (await (await catalog()).read(id)).toString('base64')]));- return { assetData: Object.fromEntries(entries) };+ return { ...(needs.length ? { assetData: Object.fromEntries(entries) } : {}), ...(await sequenceFrameData(value)) };+ }++ /** Bytes of the catalog images a recipe or edit list uses as map sequence frames, base64 by asset id. The page decodes them. */+ async function sequenceFrameData(value) {+ const ids = sequenceFrameIds(value);+ if (!ids.length) return {};+ const entries = await Promise.all(ids.map(async id => [id, (await (await catalog()).read(id)).toString('base64')]));+ return { sequenceData: Object.fromEntries(entries) };}// Motion layers can play a clip of another catalog asset (`from.asset`). The page gets those assets' bytes by catalog ID,@@ -372,7 +381,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', browjobPage.setDefaultTimeout(15000);await jobPage.goto(origin);await jobPage.waitForFunction(() => Boolean(window.__agentPreview));- const input = { sessionId: jobId, recipe: snapshot.recipe, memoryBudgetMiB: memory, backend,+ const input = { sessionId: jobId, recipe: snapshot.recipe, memoryBudgetMiB: memory, backend, ...(await sequenceFrameData(snapshot.recipe)),...(snapshot.assetBytes ? { data: snapshot.assetBytes.toString('base64'), ...(await motionSources([snapshot.recipe.motion])) } : {}) };const created = await jobPage.evaluate(({ method, input }) => window.__agentPreview[method](input),{ method: snapshot.assetBytes ? 'createAsset' : 'create', input });@@ -562,7 +571,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', browreturn Object.freeze({async capabilities() {return {- scene: AGENT_SCENE_CAPABILITIES,+ scene: { ...AGENT_SCENE_CAPABILITIES, textureSequence: TEXTURE_SEQUENCE_CAPABILITIES },sceneLimits: SCENE_LIMITS,preview: { backend, memoryBudgetMiB: memory, maximumWidth: 1024, maximumHeight: 1024, maximumImageBytes: MAXIMUM_IMAGE_BYTES, maximumActiveScenes: 1, maximumQueuedCalls: MAXIMUM_QUEUED_CALLS, evaluationTimeoutMs: EVALUATION_TIMEOUT_MS },review: { ...REVIEW_LIMITS, backend, memoryBudgetMiB: memory, evaluationTimeoutMs: EVALUATION_TIMEOUT_MS,@@ -571,7 +580,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', browphase: closing && phase !== 'closed' ? 'closing' : phase,availableIn: packageMetadata.version,assets: {- configured: Boolean(assetRoot), formats: ['self-contained GLB'],+ configured: Boolean(assetRoot), formats: ['self-contained GLB', 'PNG, JPEG or WebP image (map sequence frame)'],maximumBytes: 16 * 1024 * 1024, maximumAssets: 64,maximumVertices: 250000, maximumParts: 4096, maximumTexturePixels: 4194304, maximumJoints: 1024, maximumClips: 256,commands: ['set-root', 'set-asset-node', 'set-motion', 'set-animation', 'set-material', 'set-camera', 'set-environment', 'set-effects', 'set-shadow'],diff --git a/scripts/mcp-server.mjs b/scripts/mcp-server.mjsindex 3ba6d675..3622ff10 100644--- a/scripts/mcp-server.mjs+++ b/scripts/mcp-server.mjs@@ -114,6 +114,11 @@ const materialFields = {...materialLayerFields,subsurface: subsurface.optional(),pattern: materialPattern.optional(),+ map: z.strictObject({ sequence: z.strictObject({+ frames: z.array(z.string().regex(/^[a-f0-9]{64}$/)).min(2).max(64),+ fps: z.number().positive().max(60),+ loop: z.boolean().optional(),+ }) }).optional().describe('Image sequence colour map of catalog images; scene_capabilities textureSequence.'),color: color.optional().describe('Linear RGB base color, each component between zero and one.'),metallic: unit.optional(),roughness: unit.optional(),@@ -719,7 +724,7 @@ export function createAgentMcpServer(runtime, { version = packageMetadata.versio}register('scene_capabilities', 'Read supported recipe features and runtime budgets before authoring. Does not launch a browser.', z.strictObject({}), readOnly, async () => jsonResult(relations ? await runtime.capabilities() : withoutRelationsAdvice(await runtime.capabilities())));register('scene_create', 'Create the single active render scene from a JSON recipe. Dispose an existing scene first.', z.strictObject({ recipe: relations ? recipe : recipeWithoutRelations }), { ...edit, destructiveHint: false }, async argumentsValue => jsonResult(await runtime.create(argumentsValue)));- register('scene_asset_catalog', 'List content-bound GLB IDs from the configured local asset directory. No arbitrary paths or downloads.', z.strictObject({}), readOnly, async () => jsonResult(await runtime.assetCatalog()));+ register('scene_asset_catalog', 'List content-bound GLB and PNG, JPEG or WebP image IDs from the configured local asset directory. No arbitrary paths or downloads.', z.strictObject({}), readOnly, async () => jsonResult(await runtime.assetCatalog()));register('scene_create_asset', 'Create the active scene from a catalog GLB and editable asset recipe. Preserves supported maps, skins, morphs and clips. Dispose the existing scene first.', z.strictObject({ recipe: assetRecipe }), { ...edit, destructiveHint: false }, async argumentsValue => jsonResult(await runtime.createAsset(argumentsValue)));register('scene_inspect', 'Read revision, counts and selected nodes. For a recipe with physics, the result lists body final positions, resting times, contacts and sensor enter/leave events. Bounds describe authored geometry rather than shader displacement.', z.strictObject({ ...sessionFields, ids: z.array(identifier).max(4096).optional().describe('Selected node IDs. Omit to inspect scene order.'), kind: z.enum(['meshes', 'hierarchy']).optional().describe('Imported assets only: hierarchy exposes stable source-node IDs and joints.'), offset: z.number().int().min(0).max(4096).optional().describe('Imported assets only: pagination offset.'), limit: z.number().int().min(1).max(4096).optional().describe('Maximum returned node records. Defaults to 64.'), includeBounds: z.boolean().optional().describe('Include world geometry bounds. Defaults to false.'), path: z.record(z.string(), z.unknown()).optional().describe('Walking route over a ground node, around holes: fields in scene_capabilities navigation.') }), readOnly, async argumentsValue => jsonResult(await runtime.inspect(argumentsValue)));register('mesh_inspect', 'Check recipe geometries as polygon meshes: vertex, edge and face counts, triangles/quads/n-gons, connected and closed parts, face tags (top, bottom, side, left, right, front, back), bounds, and lint problems (non-manifold, inconsistent winding, inverted, degenerate, open, loose) with a suggested fix each. Also imported asset parts (mesh-0). Read-only and cached per geometry.', z.strictObject({ ...sessionFields, geometry: identifier.optional().describe('One geometry ID to check, as in mesh_edit.'), geometries: z.array(identifier).min(1).max(16).optional().describe('Geometry IDs to check. Omit both to check the first 16 geometries.') }), readOnly, async ({ geometry: geometryId, geometries, ...rest }) => jsonResult(await runtime.inspectMeshes({ ...rest, ...(geometries || geometryId ? { geometries: [...(geometries ?? []), ...(geometryId ? [geometryId] : [])] } : {}) })));diff --git a/scripts/measure-tool-cost.mjs b/scripts/measure-tool-cost.mjsindex 4a92a370..d008a4ee 100644--- a/scripts/measure-tool-cost.mjs+++ b/scripts/measure-tool-cost.mjs@@ -4,6 +4,7 @@ import { Client } from '@modelcontextprotocol/client';import { InMemoryTransport } from '@modelcontextprotocol/server';import { createAgentMcpServer } from './mcp-server.mjs';import { createAgentScene, AGENT_SCENE_CAPABILITIES } from '../src/agent-scene.js';+import { TEXTURE_SEQUENCE_CAPABILITIES } from '../src/texture-sequence-limits.js';import { loadAgentSceneModules } from '../src/agent-scene-modules.js';// JSON is dense in tokens: about 3.5 characters each for schemas and recipes (an estimate, not a tokenizer count).@@ -68,7 +69,7 @@ async function typicalResults(recipe) {const meshReport = scene.inspectMeshes({ geometries: ['seat'] });const { inspection, ...statusWithoutInspection } = status;return {- scene_capabilities: AGENT_SCENE_CAPABILITIES,+ scene_capabilities: { ...AGENT_SCENE_CAPABILITIES, textureSequence: TEXTURE_SEQUENCE_CAPABILITIES },scene_create: status,scene_inspect: { sessionId: SAMPLE_SESSION, ...scene.inspect() },mesh_inspect: { sessionId: SAMPLE_SESSION, ...meshReport },diff --git a/scripts/smoke-agent-texture-sequence.mjs b/scripts/smoke-agent-texture-sequence.mjsnew file mode 100644index 00000000..6e4e5c70--- /dev/null+++ b/scripts/smoke-agent-texture-sequence.mjs@@ -0,0 +1,156 @@+// Through MCP on the CPU backend: a quad whose colour map is a sequence of four generated PNG frames (red, green, blue,+// yellow). Checks that scene_look shows the frame of its motion time, that a repeated seek gives identical PNG bytes, and+// that a render job (a still and a short video) shows the same frame as scene_look at the same time.+import assert from 'node:assert/strict';+import { mkdtemp, mkdir, writeFile, readFile, rm } from 'node:fs/promises';+import { tmpdir, constants, setPriority } from 'node:os';+import { join, resolve } from 'node:path';+import { spawnSync } from 'node:child_process';+import { setTimeout as delay } from 'node:timers/promises';+import { deflateSync, crc32 } from 'node:zlib';+import { Client } from '@modelcontextprotocol/client';+import { StdioClientTransport } from '@modelcontextprotocol/client/stdio';++setPriority(0, constants.priority.PRIORITY_BELOW_NORMAL);++const FRAME_COLORS = [[230, 30, 30], [30, 230, 30], [30, 30, 230], [230, 230, 30]];+const FRAMES_PER_SECOND = 10;+const SIZE = 192;++function pngChunk(type, data) {+ const body = Buffer.concat([Buffer.from(type, 'latin1'), data]);+ const length = Buffer.alloc(4);+ length.writeUInt32BE(data.length);+ const checksum = Buffer.alloc(4);+ checksum.writeUInt32BE(crc32(body));+ return Buffer.concat([length, body, checksum]);+}++function solidPng(color, side = 16) {+ const header = Buffer.alloc(13);+ header.writeUInt32BE(side, 0);+ header.writeUInt32BE(side, 4);+ header.set([8, 2, 0, 0, 0], 8);+ const rows = Buffer.alloc((side * 3 + 1) * side);+ for (let row = 0; row < side; row++) {+ for (let column = 0; column < side; column++) rows.set(color, row * (side * 3 + 1) + 1 + column * 3);+ }+ return Buffer.concat([Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),+ pngChunk('IHDR', header), pngChunk('IDAT', deflateSync(rows)), pngChunk('IEND', Buffer.alloc(0))]);+}++/** RGB of the middle pixel of a PNG or of one raw frame, decoded by ffmpeg so the smoke needs no image library. */+function centerPixel(input, width, height) {+ const decoded = spawnSync('ffmpeg', ['-v', 'error', '-i', 'pipe:0', '-frames:v', '1', '-f', 'rawvideo', '-pix_fmt', 'rgb24', 'pipe:1'],+ { input, windowsHide: true, maxBuffer: 16 * 1024 ** 2 });+ assert.equal(decoded.status, 0, decoded.stderr.toString());+ const offset = ((Math.floor(height / 2) * width) + Math.floor(width / 2)) * 3;+ return [...decoded.stdout.subarray(offset, offset + 3)];+}++function dominantFrame(pixel) {+ const [red, green, blue] = pixel;+ if (red > blue + 60 && green > blue + 60) return 3;+ if (red > green + 60 && red > blue + 60) return 0;+ if (green > red + 60 && green > blue + 60) return 1;+ if (blue > red + 60 && blue > green + 60) return 2;+ return -1;+}++const directory = await mkdtemp(join(tmpdir(), '67js-sequence-smoke-'));+const assets = join(directory, 'assets');+const jobs = join(directory, 'jobs');+await mkdir(assets);+for (const [index, color] of FRAME_COLORS.entries()) await writeFile(join(assets, `frame-${index}.png`), solidPng(color));+const cli = resolve('scripts/mcp-server.mjs');+const transport = new StdioClientTransport({ command: process.execPath,+ args: [cli, '--backend', 'cpu', '--memory', '192', '--assets', assets, '--jobs', jobs], cwd: directory, stderr: 'pipe' });+let diagnostics = '';+transport.stderr.on('data', bytes => { diagnostics = (diagnostics + bytes).slice(-8192); });+const client = new Client({ name: '67js-sequence-proof', version: '1' });++async function call(name, argumentsValue = {}) {+ const result = await client.callTool({ name, arguments: argumentsValue }, { timeout: 120000 });+ assert.equal(result.isError, undefined, result.content?.find(item => item.type === 'text')?.text);+ return result;+}+async function look(sessionId, motionTime) {+ const result = await call('scene_look', { sessionId, width: SIZE, height: SIZE, ...(motionTime === undefined ? {} : { motionTime }) });+ return Buffer.from(result.content.find(item => item.type === 'image').data, 'base64');+}+async function completed(jobId) {+ const deadline = Date.now() + 90000;+ while (Date.now() < deadline) {+ const record = (await call('scene_render_status', { jobId })).structuredContent;+ if (['completed', 'cancelled', 'failed'].includes(record.state)) {+ assert.equal(record.state, 'completed', JSON.stringify(record));+ return;+ }+ await delay(100);+ }+ throw new Error('Job did not complete within 90 seconds.');+}++try {+ await client.connect(transport);+ assert.equal((await client.listTools()).tools.length, 22);+ const catalog = (await call('scene_asset_catalog')).structuredContent;+ const frames = catalog.assets.filter(asset => asset.kind === 'image').sort((left, right) => left.name.localeCompare(right.name)).map(asset => asset.assetId);+ assert.equal(frames.length, 4);+ const recipe = {+ format: '67js.scene', schemaVersion: 1,+ geometries: [{ id: 'quad', type: 'grid', segments: 1 }],+ materials: [{ id: 'screen', roughness: 1, map: { sequence: { frames, fps: FRAMES_PER_SECOND } } }],+ nodes: [{ id: 'screen', geometry: 'quad', material: 'screen', scale: [6, 1, 6] }],+ camera: { eye: [0, 5, 0.5], target: [0, 0, 0], fov: 40 },+ };+ const bad = await client.callTool({ name: 'scene_create', arguments: { recipe: { ...recipe, materials: [{ id: 'screen', map: { sequence: { frames: [frames[0], 'ab'.repeat(32)], fps: 10 } } }] } } });+ assert.equal(bad.isError, true, 'an unknown catalog image must be refused');+ const created = (await call('scene_create', { recipe })).structuredContent;+ const sessionId = created.sessionId;++ const shots = new Map();+ for (const index of [0, 1, 2, 3]) {+ const bytes = await look(sessionId, index / FRAMES_PER_SECOND + 0.01);+ shots.set(index, bytes);+ assert.equal(dominantFrame(centerPixel(bytes, SIZE, SIZE)), index, `scene_look at ${index / FRAMES_PER_SECOND + 0.01} s shows frame ${index}`);+ }+ assert.notDeepEqual(shots.get(0), shots.get(2));+ const wrapped = await look(sessionId, 0.43);+ assert.equal(dominantFrame(centerPixel(wrapped, SIZE, SIZE)), 0, 'the sequence loops after its last frame');++ await look(sessionId, 0.05);+ const again = await look(sessionId, 0.21);+ assert.deepEqual(again, shots.get(2), 'a repeated seek gives identical PNG bytes');+ const inspected = (await call('scene_inspect', { sessionId })).structuredContent;+ assert.deepEqual(inspected.textureSequences.map(({ material, frames: count, fps, loop, index }) => ({ material, count, fps, loop, index })),+ [{ material: 'screen', count: 4, fps: FRAMES_PER_SECOND, loop: true, index: 2 }]);++ const still = (await call('scene_render_submit', { sessionId, expectedRevision: created.revision ?? 0,+ settings: { kind: 'image', width: SIZE, height: SIZE, motion: { start: 0.21 } } })).structuredContent;+ await completed(still.jobId);+ const stillResult = (await call('scene_render_result', { jobId: still.jobId })).structuredContent;+ const stillBytes = await readFile(stillResult.artifacts[0].path);+ assert.deepEqual(centerPixel(stillBytes, SIZE, SIZE), centerPixel(shots.get(2), SIZE, SIZE), 'the job still matches scene_look at the same time');++ const video = (await call('scene_render_submit', { sessionId, expectedRevision: created.revision ?? 0,+ settings: { kind: 'video', width: 128, height: 128, fps: FRAMES_PER_SECOND, duration: 0.4, motion: { start: 0.01 } } })).structuredContent;+ await completed(video.jobId);+ const videoResult = (await call('scene_render_result', { jobId: video.jobId })).structuredContent;+ const decoded = spawnSync('ffmpeg', ['-v', 'error', '-i', videoResult.artifacts[0].path, '-f', 'rawvideo', '-pix_fmt', 'rgb24', '-threads', '1', 'pipe:1'],+ { windowsHide: true, maxBuffer: 16 * 1024 ** 2 });+ assert.equal(decoded.status, 0, decoded.stderr.toString());+ const frameBytes = 128 * 128 * 3;+ assert.equal(decoded.stdout.length, frameBytes * 4);+ for (let index = 0; index < 4; index++) {+ const offset = index * frameBytes + (64 * 128 + 64) * 3;+ assert.equal(dominantFrame([...decoded.stdout.subarray(offset, offset + 3)]), index, `video frame ${index} shows sequence frame ${index}`);+ }+ console.log(JSON.stringify({ ok: true, frames: 4, looked: [0, 1, 2, 3], stillMatchesLook: true, videoFrames: 4 }));+} catch (error) {+ console.error(diagnostics);+ throw error;+} finally {+ await client.close().catch(() => {});+ await rm(directory, { recursive: true, force: true });+}diff --git a/src/agent-scene-modules.js b/src/agent-scene-modules.jsindex 3bb480fa..2564ddfa 100644--- a/src/agent-scene-modules.js+++ b/src/agent-scene-modules.js@@ -26,6 +26,7 @@ const MODULE_LOADERS = {lightLinking: () => import('./light-linking.js'),cascades: () => import('./cascaded-shadows.js'),assets: () => import('./agent-scene-assets.js'),+ sequences: () => import('./agent-texture-sequence.js'),lineArt: () => import('./line-art.js'),viewTransform: () => import('./view-transform-shader.js'),coplanar: () => import('./coplanar-overlap.js'),@@ -62,6 +63,7 @@ const MODULE_MARKERS = {lightLinking: ['"affects"', '"keyAffects"'],cascades: ['"shadowCascades"'],assets: ['"asset":{'],+ sequences: ['"sequence":{'],lineArt: ['"lines":{', '"toon":'],viewTransform: ['"viewTransform":"agx"', '"viewTransform":"neutral"', '"look":"'],// The coplanar-overlap warning in scene_inspect appears only once this is loaded, which loadAllAgentSceneModules does; a recipe with resolveCoplanar needs it.diff --git a/src/agent-scene.js b/src/agent-scene.jsindex b9c54a4a..dac3da35 100644--- a/src/agent-scene.js+++ b/src/agent-scene.js@@ -25,6 +25,8 @@ const TERRAIN_SKY_VISIBILITY_BYTES = 4;const CAMERA_DEFAULTS = Object.freeze({ eye: [4, 3, 5], target: [0, 0, 0], up: [0, 1, 0], fov: 45, near: 0.1, far: 1000 });const NODE_FIELDS = ['translation', 'rotation', 'scale', 'parent', 'label', 'geometry', 'material', 'skin', 'relations', 'asset', 'decal', 'depthLayer'];const MATERIAL_FIELDS = ['color', 'metallic', 'roughness', 'emission', 'emissiveColor', 'opacity', 'alphaMode', 'doubleSided', 'pattern', 'subsurface', ...MATERIAL_LAYER_FIELDS];+// `map` is an image sequence (src/agent-texture-sequence.js). It is a recipe field, and the handle builds its descriptor.+const RECIPE_MATERIAL_FIELDS = [...MATERIAL_FIELDS, 'map'];const PATTERN_TEXTURE_FIELDS = ['map', 'normalMap', 'metallicRoughnessMap', 'emissiveMap', 'boxMapScale', 'boxMapAxis'];const CAMERA_FIELDS = Object.keys(CAMERA_DEFAULTS);const LOOK_OPERATION_FIELDS = Object.freeze({@@ -199,9 +201,9 @@ function copyMaterialLayers(value, result) {}function materialFrom(value) {- knownKeys(value, ['id', 'tag', ...MATERIAL_FIELDS], 'material');+ knownKeys(value, ['id', 'tag', ...RECIPE_MATERIAL_FIELDS], 'material');// A pattern brings its own colours, so the color field tints it and leaves it unchanged by default.- const defaultColor = value.pattern === undefined ? [0.5, 0.5, 0.5] : [1, 1, 1];+ const defaultColor = value.pattern === undefined && value.map === undefined ? [0.5, 0.5, 0.5] : [1, 1, 1];const graph = value.pattern?.type === 'graph' ? value.pattern.graph : {};// A graph that makes roughness, metallic or emission passes it through as written, so the factors default to 1.const graphFactor = (output, fallback) => (Object.hasOwn(graph ?? {}, output) ? 1 : fallback);@@ -214,6 +216,11 @@ function materialFrom(value) {if (result.tag !== 'surface') throw new RangeError('Material pattern needs the surface tag.');result.pattern = agentSceneModule('patterns').normalizeMaterialPattern(value.pattern);}+ if (value.map !== undefined) {+ if (result.tag !== 'surface') throw new RangeError('Material map needs the surface tag.');+ if (value.pattern !== undefined) throw new TypeError('A material takes a pattern or a map sequence, not both.');+ result.map = agentSceneModule('sequences').normalizeMapSequence(value.map);+ }result.opacity = defaultValue(value.opacity, 1);finite(result.opacity, 'opacity');if (result.opacity < 0 || result.opacity > 1) {@@ -752,11 +759,11 @@ function geometryStats(records) {function batchSignature(material) {// Textures are fixed per renderer, so a changed pattern rebuilds it like a changed tag does.// Transmission moves a material to the pass that sees the opaque scene, which is fixed per renderer.- return `${material.tag}|${material.alphaMode}|${material.opacity}|${material.transmission > 0}|${material.doubleSided}|${JSON.stringify(material.pattern ?? null)}`;+ return `${material.tag}|${material.alphaMode}|${material.opacity}|${material.transmission > 0}|${material.doubleSided}|${JSON.stringify(material.pattern ?? null)}|${JSON.stringify(material.map ?? null)}`;}-function materialHandle(snapshot) {- const holder = { snapshot };+function materialHandle(snapshot, sequence = null) {+ const holder = { snapshot, sequence };const output = {};for (const field of ['tag', ...MATERIAL_FIELDS]) {Object.defineProperty(output, field, {@@ -771,6 +778,7 @@ function materialHandle(snapshot) {enumerable: true,// Baked on first read, so validating or inspecting a recipe never pays for it.get() {+ if (field === 'map' && holder.sequence) return holder.sequence.descriptor;return holder.snapshot.pattern ? agentSceneModule('patterns').materialPatternTextures(holder.snapshot.pattern)[field] : undefined;},});@@ -907,6 +915,7 @@ function publishPose(stagedState, pose) {if (matrix) Float32Array.prototype.set.call(part.matrix, matrix);}}+ for (const handle of stagedState.materialHandles.values()) handle.holder.sequence?.seek(pose.time);stagedState.renderCamera = pose.camera;stagedState.motionTime = pose.time;const nextParts = partsWithCloth(stagedState.restParts, pose.clothGeometries);@@ -915,6 +924,10 @@ function publishPose(stagedState, pose) {return partsChanged;}+function hasSequences(sceneState) {+ return [...sceneState.materialHandles.values()].some(handle => handle.holder.sequence);+}+// The recipe an agent wrote: without the geometries, materials and part nodes that placed assets generate.function authoredRecipe(recipe) {const authored = structuredClone(recipe);@@ -1098,7 +1111,7 @@ export function createAgentScene(input, inputLimits = {}, options = {}) {// The renderer keeps tag, blending, opacity and sidedness per batch, so a handle is reused only when all match.const existing = [preferredHandle, activeHandle].find(handle => handle && batchSignature(handle.holder.snapshot) === batchSignature(material));const assetOutput = assetRuntime.materialOutputs.get(material.id);- const handle = assetOutput ? assetMaterialHandle(snapshot, assetOutput, preferredHandle, activeHandle) : existing ?? materialHandle(snapshot);+ const handle = assetOutput ? assetMaterialHandle(snapshot, assetOutput, preferredHandle, activeHandle) : existing ?? materialHandle(snapshot, material.map ? agentSceneModule('sequences').createTextureSequence(material.id, material.map, options.sequenceFrames) : null);materialHandles.set(material.id, handle);}const activeParts = new Map(state?.sourceRestParts.map(part => [part.name, part]) ?? []);@@ -1338,12 +1351,17 @@ export function createAgentScene(input, inputLimits = {}, options = {}) {get motionTime() {return state.motionTime;},+ /** The colour map of each sequence material: its descriptor, the frame to show and the pixels of that frame. */+ sequenceTextures() {+ return [...state.materialHandles.values()].filter(handle => handle.holder.sequence)+ .map(({ holder }) => ({ descriptor: holder.sequence.descriptor, index: holder.sequence.index, image: holder.sequence.image }));+ },sampleMotion(time) {if (time !== null && (!Number.isFinite(time) || time < 0 || time > 3600)) {throw new RangeError('Motion time must be null or within zero and 3600 seconds.');}- if (time !== null && !state.motionSampler && !state.physicsSampler && !state.clothSampler && !state.particleSampler && !state.assetRuntime.animators.length) {- throw new RangeError('This procedural recipe has no motion, physics, cloth or particles, and no placed asset clip.');+ if (time !== null && !state.motionSampler && !state.physicsSampler && !state.clothSampler && !state.particleSampler && !state.assetRuntime.animators.length && !hasSequences(state)) {+ throw new RangeError('This procedural recipe has no motion, physics, cloth or particles, no placed asset clip and no map sequence.');}if (publishPose(state, stagePose(state, time))) {topologyRevision++;@@ -1544,7 +1562,7 @@ export function createAgentScene(input, inputLimits = {}, options = {}) {fields = NODE_FIELDS;target = candidate.nodes.find(node => node.id === operation.id);} else if (operation.type === 'set-material') {- fields = MATERIAL_FIELDS;+ fields = RECIPE_MATERIAL_FIELDS;target = candidate.materials.find(material => material.id === operation.id);} else if (operation.type === 'set-camera') {fields = CAMERA_FIELDS;@@ -1604,6 +1622,7 @@ export function createAgentScene(input, inputLimits = {}, options = {}) {...(state.clothSampler ? { cloth: state.clothSampler.report() } : {}),...(state.particleSampler ? { particles: state.particleSampler.report() } : {}),...(state.assetRuntime.placements.length ? { assets: agentSceneModule('assets').describeAssetPlacements(state.assetRuntime, state.graph, geometry => state.localBounds.get(geometry)) } : {}),+ ...(hasSequences(state) ? { textureSequences: [...state.materialHandles.values()].filter(handle => handle.holder.sequence).map(handle => handle.holder.sequence.describe()) } : {}),...(state.bakeReport ? { bake: state.bakeReport } : {}),...(state.reflectionReport ? { reflections: state.reflectionReport } : {}),...(state.recipe.armatures ? { rig: rigReport(state.recipe, state.skinBindings) } : {}),diff --git a/src/agent-texture-sequence.js b/src/agent-texture-sequence.jsnew file mode 100644index 00000000..641824f6--- /dev/null+++ b/src/agent-texture-sequence.js@@ -0,0 +1,105 @@+import { createImage } from './material-texture-kit.js';+import { TEXTURE_SEQUENCE_LIMITS } from './texture-sequence-limits.js';++// An image sequence as the colour map of a recipe material: `map: { sequence: { frames, fps, loop } }`. The frames are+// catalog image assets that the host decodes before createAgentScene. The scene picks a frame from its motion time, so a+// preview, a seek and a render job show the same frame for the same time.++const ASSET_ID = /^[a-f0-9]{64}$/;+const MAP_FIELDS = ['sequence'];+const SEQUENCE_FIELDS = ['frames', 'fps', 'loop'];++function knownKeys(value, allowed, label) {+ if (!value || typeof value !== 'object' || Array.isArray(value)) throw new TypeError(`${label} must be an object.`);+ for (const key of Object.keys(value)) {+ if (!allowed.includes(key)) throw new TypeError(`Unknown ${label} field: ${key}. Allowed fields: ${allowed.join(', ')}.`);+ }+}++/** Validate a material `map` field. Returns the canonical form kept in the recipe. */+export function normalizeMapSequence(value) {+ knownKeys(value, MAP_FIELDS, 'material map');+ if (value.sequence === undefined) throw new TypeError('material map needs a sequence: { frames, fps }.');+ const sequence = value.sequence;+ knownKeys(sequence, SEQUENCE_FIELDS, 'map sequence');+ const { frames, fps, loop = true } = sequence;+ if (!Array.isArray(frames) || frames.length < 2 || frames.length > TEXTURE_SEQUENCE_LIMITS.maxFrames) {+ throw new RangeError(`map sequence frames must list 2 to ${TEXTURE_SEQUENCE_LIMITS.maxFrames} catalog image asset IDs.`);+ }+ for (const frame of frames) {+ if (typeof frame !== 'string' || !ASSET_ID.test(frame.toLowerCase())) {+ throw new TypeError('Each map sequence frame must be the 64 hexadecimal characters of a catalog image asset id; scene_asset_catalog lists them.');+ }+ }+ if (!Number.isFinite(fps) || fps <= 0 || fps > TEXTURE_SEQUENCE_LIMITS.maxFps) {+ throw new RangeError(`map sequence fps must be above 0 and at most ${TEXTURE_SEQUENCE_LIMITS.maxFps}.`);+ }+ if (typeof loop !== 'boolean') throw new TypeError('map sequence loop must be boolean.');+ return { sequence: { frames: frames.map(frame => frame.toLowerCase()), fps, loop } };+}++/** The frame shown at a scene time: floor(time * fps), wrapped when the sequence loops and held on the last frame otherwise. */+export function sequenceFrameIndex(sequence, time) {+ const count = sequence.frames.length;+ // The small offset keeps a time such as 0.29 s at 100 fps on frame 29 despite binary rounding.+ const raw = Math.floor(Math.max(0, time ?? 0) * sequence.fps + 1e-9);+ return sequence.loop ? raw % count : Math.min(raw, count - 1);+}++/** The catalog image IDs a recipe, or a list of edit operations, uses as sequence frames. */+export function sequenceFrameIds(value) {+ const ids = new Set();+ const visit = (node, depth) => {+ if (!node || typeof node !== 'object' || depth > 8) return;+ if (Array.isArray(node)) {+ for (const child of node) visit(child, depth + 1);+ return;+ }+ const frames = node.map?.sequence?.frames;+ if (Array.isArray(frames)) {+ for (const frame of frames) {+ if (typeof frame === 'string' && ASSET_ID.test(frame.toLowerCase())) ids.add(frame.toLowerCase());+ }+ }+ for (const child of Object.values(node)) visit(child, depth + 1);+ };+ visit(value, 0);+ return [...ids];+}++/**+ * The runtime of one material: a stable colour map descriptor holding the first frame, and the frame picked by `seek`.+ * `pool` maps catalog image IDs to decoded `{ width, height, data }` RGBA frames.+ */+export function createTextureSequence(materialId, mapField, pool) {+ const { frames, fps, loop } = mapField.sequence;+ const images = frames.map(id => {+ const frame = pool?.get(id);+ if (!frame) throw new Error(`Material ${materialId}: image ${id.slice(0, 12)} was not loaded. The host decodes catalog images before createAgentScene or apply.`);+ return frame;+ });+ const { width, height } = images[0];+ if (images.some(frame => frame.width !== width || frame.height !== height)) {+ throw new RangeError(`Material ${materialId}: every sequence frame must have the same size, ${width}x${height} here. Resize the images in the catalog.`);+ }+ const wrapped = images.map(frame => createImage(frame.data, frame.width, frame.height));+ const descriptor = { image: wrapped[0], colorSpace: 'srgb' };+ const definition = { frames, fps, loop };+ let index = 0;+ return {+ descriptor,+ seek(time) {+ index = sequenceFrameIndex(definition, time);+ return index;+ },+ get index() {+ return index;+ },+ get image() {+ return wrapped[index];+ },+ describe() {+ return { material: materialId, frames: frames.length, fps, loop, index, frame: frames[index], size: [width, height] };+ },+ };+}diff --git a/src/texture-sequence-limits.js b/src/texture-sequence-limits.jsnew file mode 100644index 00000000..03f66b74--- /dev/null+++ b/src/texture-sequence-limits.js@@ -0,0 +1,16 @@+// Colour map as an image sequence: frames are catalog images, the frame follows the scene's motion time. Kept outside+// agent-scene-capabilities.js so the scene entry does not carry text only an agent reads; the MCP runtime adds it to+// scene_capabilities.++export const TEXTURE_SEQUENCE_LIMITS = Object.freeze({+ maxFrames: 64, maxFps: 60, maxSide: 1024, maxDecodedBytes: 64 * 1024 * 1024,+});++export const TEXTURE_SEQUENCE_CAPABILITIES = Object.freeze({+ field: 'material.map = { sequence: { frames: [catalog image asset IDs], fps, loop } }',+ covers: 'color map (map) of surface-tag materials; needs UVs (grid, box and similar); not combined with pattern',+ frame: 'floor(motionTime * fps), wrapped when loop is true (default) and held on the last frame when false; frame 0 before any motion time',+ clock: 'motionTime, also in render jobs (settings.motion.start + elapsed)',+ images: 'PNG, JPEG or WebP files in the --assets folder, listed by scene_asset_catalog with kind image; opaque pixels, one size for all frames',+ ...TEXTURE_SEQUENCE_LIMITS,+});diff --git a/tests/agent-texture-sequence.test.js b/tests/agent-texture-sequence.test.jsnew file mode 100644index 00000000..b3f0cb84--- /dev/null+++ b/tests/agent-texture-sequence.test.js@@ -0,0 +1,115 @@+import test from 'node:test';+import assert from 'node:assert/strict';+import { mkdtemp, writeFile, rm } from 'node:fs/promises';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { createAgentScene, loadAllAgentSceneModules } from '../src/agent-scene.js';+import { normalizeMapSequence, sequenceFrameIds, sequenceFrameIndex } from '../src/agent-texture-sequence.js';+import { createAgentAssetCatalog } from '../scripts/agent-asset-catalog.mjs';++await loadAllAgentSceneModules();++const FRAME_IDS = ['a', 'b', 'c', 'd'].map(letter => letter.repeat(64));++function solidFrame(red) {+ const data = new Uint8ClampedArray(2 * 2 * 4);+ for (let offset = 0; offset < data.length; offset += 4) data.set([red, 0, 0, 255], offset);+ return { width: 2, height: 2, data };+}++function framePool() {+ return new Map(FRAME_IDS.map((id, index) => [id, solidFrame(index * 60)]));+}++function quadRecipe(sequence) {+ return {+ format: '67js.scene', schemaVersion: 1,+ geometries: [{ id: 'quad', type: 'grid', segments: 1 }],+ materials: [{ id: 'screen', map: { sequence } }],+ nodes: [{ id: 'screen', geometry: 'quad', material: 'screen' }],+ camera: { eye: [0, 3, 3], target: [0, 0, 0], fov: 45 },+ };+}++test('frame index is floor(time * fps), looped by default and held on the last frame without loop', () => {+ const looped = { frames: FRAME_IDS, fps: 10, loop: true };+ const held = { ...looped, loop: false };+ assert.equal(sequenceFrameIndex(looped, 0), 0);+ assert.equal(sequenceFrameIndex(looped, 0.099), 0);+ assert.equal(sequenceFrameIndex(looped, 0.1), 1);+ assert.equal(sequenceFrameIndex(looped, 0.3), 3);+ assert.equal(sequenceFrameIndex(looped, 0.4), 0);+ assert.equal(sequenceFrameIndex(looped, null), 0);+ assert.equal(sequenceFrameIndex(held, 0.4), 3);+ assert.equal(sequenceFrameIndex(held, 100), 3);+ assert.equal(sequenceFrameIndex({ frames: FRAME_IDS, fps: 100, loop: false }, 0.03), 3);+});++test('a bad frame ID, frame count, fps or loop is rejected with a message that names the field', () => {+ const good = { frames: FRAME_IDS, fps: 12 };+ assert.deepEqual(normalizeMapSequence({ sequence: { ...good, frames: FRAME_IDS.map(id => id.toUpperCase()) } }).sequence.frames, FRAME_IDS);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, frames: [FRAME_IDS[0], 'not-an-id'] } }), /64 hexadecimal characters/);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, frames: [FRAME_IDS[0]] } }), /2 to 64/);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, fps: 0 } }), /fps must be above 0/);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, fps: 61 } }), /fps must be above 0 and at most 60/);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, fps: Number.NaN } }), /fps/);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, loop: 'yes' } }), /loop must be boolean/);+ assert.throws(() => normalizeMapSequence({ sequence: { ...good, url: 'https://example.com/a.png' } }), /Unknown map sequence field: url/);+ assert.throws(() => normalizeMapSequence({ url: 'https://example.com/a.png' }), /Unknown material map field: url/);+});++test('the scene shows the frame of its motion time, reports it and seeks deterministically', () => {+ const scene = createAgentScene(quadRecipe({ frames: FRAME_IDS, fps: 10 }), {}, { sequenceFrames: framePool() });+ assert.deepEqual(scene.sequenceTextures().map(texture => texture.index), [0]);+ const descriptor = scene.parts[0].material.map;+ assert.equal(scene.sequenceTextures()[0].descriptor, descriptor);+ scene.sampleMotion(0.25);+ assert.equal(scene.sequenceTextures()[0].index, 2);+ assert.equal(scene.sequenceTextures()[0].image.data[0], 120);+ assert.equal(scene.parts[0].material.map, descriptor, 'the descriptor stays one object for the renderer');+ const reported = scene.inspect().textureSequences;+ assert.deepEqual(reported.map(({ material, frames, fps, loop, index }) => ({ material, frames, fps, loop, index })),+ [{ material: 'screen', frames: 4, fps: 10, loop: true, index: 2 }]);+ scene.sampleMotion(0.05);+ scene.sampleMotion(0.25);+ assert.equal(scene.sequenceTextures()[0].index, 2);+ scene.sampleMotion(null);+ assert.equal(scene.sequenceTextures()[0].index, 0);+});++test('the scene rejects frames that are missing or differ in size, and a map with a pattern', () => {+ assert.throws(() => createAgentScene(quadRecipe({ frames: FRAME_IDS, fps: 10 }), {}, { sequenceFrames: new Map() }), /was not loaded/);+ const mixed = framePool();+ mixed.set(FRAME_IDS[1], { width: 4, height: 4, data: new Uint8ClampedArray(64) });+ assert.throws(() => createAgentScene(quadRecipe({ frames: FRAME_IDS, fps: 10 }), {}, { sequenceFrames: mixed }), /same size/);+ const recipe = quadRecipe({ frames: FRAME_IDS, fps: 10 });+ recipe.materials[0].pattern = { type: 'bricks' };+ assert.throws(() => createAgentScene(recipe, {}, { sequenceFrames: framePool() }), /pattern or a map sequence/);+});++test('an edit can set a sequence and the frame IDs of a recipe are collected for the host', () => {+ const recipe = quadRecipe({ frames: FRAME_IDS, fps: 10 });+ assert.deepEqual(sequenceFrameIds(recipe).sort(), [...FRAME_IDS].sort());+ assert.deepEqual(sequenceFrameIds([{ type: 'set-material', id: 'screen', map: { sequence: { frames: [FRAME_IDS[0], 'zz'], fps: 1 } } }]), [FRAME_IDS[0]]);+});++test('the catalog lists PNG, JPEG and WebP images by content and rejects a mismatched extension', async context => {+ const directory = await mkdtemp(join(tmpdir(), '67js-sequence-'));+ context.after(() => rm(directory, { recursive: true, force: true }));+ const png = Buffer.concat([Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), Buffer.alloc(32, 1)]);+ const jpeg = Buffer.concat([Buffer.from([0xff, 0xd8, 0xff, 0xe0]), Buffer.alloc(32, 2)]);+ const webp = Buffer.alloc(32, 3);+ webp.write('RIFF', 0, 'latin1');+ webp.writeUInt32LE(webp.length - 8, 4);+ webp.write('WEBP', 8, 'latin1');+ await writeFile(join(directory, 'frame-0.png'), png);+ await writeFile(join(directory, 'frame-1.jpg'), jpeg);+ await writeFile(join(directory, 'frame-2.webp'), webp);+ const catalog = await createAgentAssetCatalog(directory);+ const images = catalog.list().assets;+ assert.deepEqual(images.map(asset => [asset.name, asset.kind, asset.mimeType]),+ [['frame-0.png', 'image', 'image/png'], ['frame-1.jpg', 'image', 'image/jpeg'], ['frame-2.webp', 'image', 'image/webp']]);+ assert.deepEqual(await catalog.read(images[0].assetId), png);+ await writeFile(join(directory, 'wrong.png'), jpeg);+ await assert.rejects(createAgentAssetCatalog(directory), /extension must match/);+});