3D engine that AI agents drive
Let agents reach mesh blend and decals through MCP and document the unreleased features (closes #490)
Nguyen Xuan An9bd62e51feea
diff --git a/changes/490.md b/changes/490.mdnew file mode 100644index 00000000..a2445920--- /dev/null+++ b/changes/490.md@@ -0,0 +1 @@+- Make the unreleased features reachable by agents (closes #490). The MCP recipe schema now accepts the node fields `blend` (mesh blend), `decal` and `depthLayer`, which `add-node`, `set-node` and `scene_create` rejected before. `scene_create_asset` on a file with no mesh says that a skeleton-only file is a clip source for `motion.layers[].from`. `docs/agent-scene.md` (seam fields, the refracted water-bed shadow), `docs/agent-motion.md`, `docs/gltf.md` (MCP catalog limits), `docs/guide.md` (the `world-material-shader` and `liquid-shader` imports), `public/llms.txt` (GLB support, the changelog pointer) and the build-67js-scene skill (moving cloth colliders, seams, sun bounce, skeleton-only clips) cover them.diff --git a/changes/490.size.md b/changes/490.size.mdnew file mode 100644index 00000000..0232f768--- /dev/null+++ b/changes/490.size.md@@ -0,0 +1,3 @@+- tool-total: +924 bytes so the node fields `blend`, `decal` and `depthLayer` pass the schema; before, `scene_create`, `add-node` and `set-node` rejected them, so mesh blend and the coplanar fix could not be reached through MCP.+- tool:scene_create: +308 bytes for the three node fields in the recipe node schema, with no field descriptions beyond one line on `blend`.+- tool:scene_apply: +616 bytes for the same three fields in `add-node` and `set-node`.diff --git a/docs/agent-motion.md b/docs/agent-motion.mdindex ad62af20..c9caa2a7 100644--- a/docs/agent-motion.md+++ b/docs/agent-motion.md@@ -169,6 +169,7 @@ onto this asset with `retargetClip` and the humanoid bone map. Both skeletons muarms and legs) or the call fails and lists what is missing. `scale` (0.1 to 10) and `rootMotion` (boolean) are the`retargetClip` options of the same names. `from` replaces `clip` and `bones`, and morph tracks of the source are not carried.The runtime reads the other asset by its catalog ID (at most 4 at once); a session made without a catalog cannot use it.+The source can be a skeleton-only file (`meshes: 0` in `scene_asset_catalog`, such as the 100STYLE clips): only its clips and joints are read. `scene_create_asset` on such a file fails, since it has nothing to draw.`scene_inspect` `kind: "hierarchy"` returns `retargets`: per `from` layer, `source` and `target` with `mapped` (bonesmatched), `unmapped` (joints no bone claimed, up to 24, with `unmappedCount`) and `missing`.diff --git a/docs/agent-scene.md b/docs/agent-scene.mdindex 678a28ec..e888f278 100644--- a/docs/agent-scene.md+++ b/docs/agent-scene.md@@ -106,7 +106,7 @@ Materials contain `id`, `tag`, `color`, `metallic`, `roughness`, `emission`, optThe defaults are `surface`, `[0.5, 0.5, 0.5]`, `0`, `0.5`, `0`, opacity `1`, `opaque` and `false`.The tag can be `surface`, `unlit`, `liquid`, `foam`, `water`, `foliage` or `grass` (the last two sway in `environment.wind`) and remains fixed during editing.`liquid` refracts and tints what lies behind it: its color is the light one metre lets through, so `[0.56, 0.84, 0.88]` is clear water. `foam` draws soft blended droplets.-`water` is a flat mirror sheet (a plane or thin box). The largest one sets the water level: everything below it fades by `exp(-absorption * depth)` per colour channel (`absorption` is RGB per metre), and its shadows are lifted and softened with depth times `turbidity` (0 clear to 1 murky). `waterPreset` fills both: `lake` (default), `sea` or `river`. A shadow on the surface is faint and never darkens the reflection; give the sheet an `opacity` below one when the bed should show.+`water` is a flat mirror sheet (a plane or thin box). The largest one sets the water level: everything below it fades by `exp(-absorption * depth)` per colour channel (`absorption` is RGB per metre), and its shadows are lifted and softened with depth times `turbidity` (0 clear to 1 murky). `waterPreset` fills both: `lake` (default), `sea` or `river`. A shadow on the surface is faint and never darkens the reflection; give the sheet an `opacity` below one when the bed should show. The sun's leg to a bed is bent at n = 1.33 and absorbed over `depth / cos(theta_t)`, so a clear lake keeps little shadow contrast on a bed 4 m down, and a murky river keeps about a quarter.Any `opacity` below one draws the material blended after opaque geometry, which suits glass, windows and visors. `alphaMode` accepts `opaque` or `blend`; cutout masking needs a texture and is not available in procedural recipes.`doubleSided` draws and lights both faces of thin sheets.Changing the tag, opacity, blending, sidedness or pattern replaces the affected parts in the renderer; color, metallic, roughness and emission edits keep them.@@ -303,6 +303,10 @@ An asset placement with a `crowd` field fills its group node with `count` (1 to- `clip` is a clip index, `layout` is `grid` or `scatter`, and the pose is a pure function of scene time, `seed` and the recipe camera, so a render at a time equals the frame reached by playing. People farther than `farDistance` from the recipe camera hold each pose for `1 / farRate` seconds.- The crowd takes one decoded copy of the asset, the generated part nodes (count times materials) count against `maxNodes`, and `scene_inspect` lists it under `assets` with `kind: "crowd"`, `instances`, `batches`, `frames`, `paletteBytes` and `boundsRadius`.+### Seams between meshes++Three node fields fix what looks wrong where meshes meet; `add-node` and `set-node` take all of them. `blend: { "radius": 0.15 }` (metres, 0.005 to 2) softens the seam where a rock or trunk cuts into the ground and needs a preview with depth; see [mesh-blend.md](mesh-blend.md). `decal: true` and `depthLayer` (0 to 4) stop z-fighting on flush faces; see `depthBias` in `scene_capabilities`.+### Relations between partsA mesh node can list `relations` that place it against other mesh nodes, so a resized body keeps its handle where itdiff --git a/docs/gltf.md b/docs/gltf.mdindex 2425e856..09fa8eb8 100644--- a/docs/gltf.md+++ b/docs/gltf.md@@ -48,6 +48,7 @@ Every numeric limit must be a positive safe integer.The defaults take one large exported scene, such as a 240 MB GLB with 8 million vertices and 8192 x 8192 textures, and `GLTF_CAPABILITIES.limits` holds them.A file over its limits fails with one `RangeError` that names each limit it goes over, the measured value and the value that fits, for example `glTF over its limits: vertices 9120000, over maxVertices 8000000; image 6 (6144x3072) pixels 18874368, over maxTextureImagePixels 16777216. Fix: pass { maxVertices: 9120000, maxTextureImagePixels: 18874368 }.`The file size and the decoded geometry and parts are checked as they are read, so they fail on their own.+The MCP server decodes catalog files with tighter limits: 16 MiB a file, 250,000 vertices (see [mcp.md](mcp.md)). A larger scene loads through `loadGltf` in code or split into parts; meshopt compression shrinks the file.The importer rejects cyclic hierarchies, multiple parents, and hierarchies deeper than 256 levels.It checks PNG, JPEG and WebP dimensions before browser image decoding.Decoded accessor data has a separate ceiling of four times `maxBytes`.diff --git a/docs/guide.md b/docs/guide.mdindex 328549b1..4aa94c06 100644--- a/docs/guide.md+++ b/docs/guide.md@@ -315,6 +315,7 @@ An attachment positions one rigid frame. It does not conform the accessory footpThe optional GLB addon creates these descriptors. HDR requires `EXT_color_buffer_float`. Inspect `renderer.capabilities.hdr` and `hdrFallbackReason`. The renderer falls back when floating-point targets are unavailable.Tags select specialized behavior: `armor` allows studio finish overrides. `floor` is a studio floor. `water` adds animated normals and ripples. `foliage` and `grass` add wind and accept a base-colour `map` with alpha mask, a `normalMap` and `doubleSided` for leaf cards.+In code, a renderer that draws a `water`, `foliage`, `grass`, `terrain`, `moss`, `fire` or `smoke` part needs `import '@codepawlhq/67js/addons/world-material-shader'` first, and one that draws a transmissive, `liquid` or `foam` part needs `import '@codepawlhq/67js/addons/liquid-shader'`. The terrain, vegetation, grass, clutter and film addons and the MCP server import them for you.`material.alphaMode` is `'opaque'` (default), `'mask'` (cutout at `alphaCutoff`, default 0.5, shadows follow it) or `'blend'` (drawn last, back to front, no depth write). `material.opacity` (default 1) multiplies the base map alpha. `material.doubleSided` (default false) disables back-face culling and lights both faces. These are batch state and cannot change after upload. See `docs/material-maps.md`.diff --git a/public/llms.txt b/public/llms.txtindex d2113856..40fe4558 100644--- a/public/llms.txt+++ b/public/llms.txt@@ -3,6 +3,7 @@> A WebGL2 toolkit for interactive browser scenes and deterministic PNG/MP4 output. Reuse the scene across preview and export.This file is a compact map for LLMs and coding agents. The linked guide contains the API contract and runnable examples.+Unreleased changes, with issue numbers and breaking flags, are in changelog.json next to release.json on the site. Check it before you rely on a feature from the docs.The installed package determines available APIs. Release metadata describes the website's release, which can differ from a local checkout.## Read only what your task needs@@ -156,7 +157,7 @@ Nearest projection can collapse vertices on coarse meshes. Preserve the error anVertex clearance does not certify whole-triangle clearance. Compatible static output can become a GPU morph target without repeating queries per frame.-GLB support excludes transparent and double-sided materials and compressed geometry. Unsupported features require visible errors.+GLB support loads transparent, double-sided and meshopt-compressed files. Unsupported features require visible errors. The MCP asset catalog takes 16 MiB and 250,000 vertices a file. A file with a skin or clips but no mesh loads with `parts: []` and works as a clip source (`motion.layers[].from`).Version `0.4.1` adds `addons/surface-attachment`. Read `docs/surface-attachment.md` before binding.Bind a triangle and barycentric coordinates once. Assign the retained attachment.matrix to the accessory.diff --git a/scripts/mcp-server.mjs b/scripts/mcp-server.mjsindex 374e3599..483c8eb3 100644--- a/scripts/mcp-server.mjs+++ b/scripts/mcp-server.mjs@@ -37,6 +37,9 @@ const nodeFields = {asset: placedAsset.optional(),vegetation: placedVegetation.optional(),skin: skinSpec.optional(),+ decal: z.boolean().optional(),+ depthLayer: z.number().int().min(0).max(4).optional(),+ blend: z.strictObject({ radius: z.number().min(0.005).max(2) }).optional().describe('Soft seam into another mesh, radius in m; docs/mesh-blend.md.'),label: z.string().max(256).optional().describe('Human-readable label. Identity remains the node ID.'),parent: identifier.nullable().optional().describe('Parent node ID, or null for a root. Local transforms stay unchanged.'),translation: vector3.optional().describe('Local XYZ translation.'),diff --git a/skills/build-67js-scene/SKILL.md b/skills/build-67js-scene/SKILL.mdindex eabfdca8..7b19a394 100644--- a/skills/build-67js-scene/SKILL.md+++ b/skills/build-67js-scene/SKILL.md@@ -45,11 +45,13 @@ Use `simplify-code` and `engineering-workflows` for implementation.- Glass, water, gems, car paint and cloth in a recipe use the material layer fields `transmission` (with `thickness` and `ior`), `clearcoat`, `sheenColor`, `specular`, `anisotropy` (brushed metal) and `iridescence` (soap, oil). Give glass the default `surface` tag, not `liquid`, which absorbs by thickness and reads murky; anisotropy and iridescence are ignored on a transmissive material. A transmissive material needs a nonzero `thickness` to bend the view behind it, and does not show blended surfaces. Ranges and limits are in "Material layers" in the checkout's `docs/material-maps.md`.- Layered skeletal animation in code (a clip on some joints only, additive clips, foot planting, reaching, looking at a point) uses `createAssetPose` from `src/skeleton-pose.js` and the checkout's `docs/skeleton-pose.md`. Clip layers are not available through MCP.- Jointed motion (an arm reaching, a lamp or head tracking a point) on a recipe or an imported character uses `motion.ik`, which needs no quaternions: on a recipe parent the joint nodes and put sizes on child mesh nodes, on an asset find `node-N` IDs with `scene_inspect` kind hierarchy and aim with `worldForward`. Read `motionIk` from `scene_inspect`: besides `reached`, it lists `warnings` (`wrong-bend` when a knee or elbow bends the unnatural way, `hyperextended`) and `limited` (poses a joint limit changed). Give a leg or arm `bendToward` (a world direction, such as `[0, 0, 1]` for a knee on a character facing +z) and `limits` per joint in degrees, and fix any warning before you finish. `motion.constraints` (copy, limit, `track_to`, `child_of`, `floor`, Blender's meanings, Euler XYZ degrees) and `motion.drivers` (a node or material property from a bounded JSON expression) run after IK and give the same pose at any time. On an imported asset, `motion.layers` plays its clips together (a walk layer plus a wave layer masked to `{ "joints": ["node-N"] }` for the arm): list clip ids with `scene_inspect` (`animations`), find joints with kind hierarchy, set `morph: false` on a layer that should not touch the face, and use `bones` to send a clip's track to another node. See "Inverse kinematics", "Clip layers on an asset", "Constraints" and "Drivers" in the checkout's `docs/agent-motion.md`.+- Where meshes meet: a rock or trunk sunk into ground gets node `blend: { "radius": 0.15 }` (soft seam, needs depth, `docs/mesh-blend.md`); flush faces get `decal` or `depthLayer`. A sun-lit closed room bounces sun light on its own; `environment.sunBounce` scales it (0 off). Water over a bed needs `opacity` below one to show the bed, and the bed's shadow fades with depth (`docs/agent-scene.md`).+- A skeleton-only GLB (`meshes: 0` in `scene_asset_catalog`) cannot be placed, but its clips drive a character through `motion.layers[].from`.- Agent tool calls use the installed `67js-mcp` stdio server from release `0.5.0` onward. Read `docs/mcp.md` for launch and PNG feedback.- Agent render jobs use `docs/agent-render-jobs.md`. Check source versus published support, configure a private store and poll until terminal.- A character or creature that must move (a walker, a quadruped) gets a recipe `armatures` entry and a `skin` on each mesh node: `{ "id": "rig", "preset": "humanoid", "fit": { "bounds": { "min": [...], "max": [...] } } }` (or `fit.node`), then `skin: { "armature": "rig" }` on every body part. Bones are nodes named `hips`, `leftUpperLeg`, `leftLowerLeg` and so on, so rotation tracks and `motion.ik` address them directly (rotation is local, about the bone head; positive X swings a hanging limb backward). Build the mesh at rest in the preset's pose (facing +Z, arms down), and read `rig` in `scene_inspect`: unweighted vertices, bones with no vertices, joint axis problems. Weights are by distance to the bone segments, so give limbs the same loops you want to bend. See "Authored rigs" in `docs/agent-motion.md`.- Things that fall, pile up or rest on an authored mesh (dropped objects, a ball in a bowl, a trigger zone) use the recipe `physics` section: bodies on root node IDs, `shape: 'trimesh'` on a fixed body so objects follow the mesh, `sensor: true` for triggers, `layer` and `collidesWith` for masks. Seek with `motionTime` like motion, and read `physics` from `scene_inspect` (resting times, contacts, sensor events) instead of guessing from a picture. Needs the optional peer `@dimforge/rapier3d-compat`; a limp character uses `physics.ragdoll` (`{ armature }` for a recipe armature; an imported asset takes `physics: { duration, floor, ragdoll: {} }` and falls from its current pose onto a floor at world height `floor`), needs a floor body in a scene recipe, and reports `physics.ragdoll` (`lowestY`, `highestY`, `restsAtTime`); joints have no swing limits. See "Recipe physics" in the checkout's `docs/physics.md`.-- Flags, curtains, capes and cloths draped over props use the recipe `cloth` section: a `grid` geometry node (flat sheet in XY, `segments` up to 64) listed in `sheets` with `pin` edges (`top`, `left`, a corner...), `wind` in m/s, and fixed `colliders` (`ball` or `box` fitted to their node). Seek with `motionTime`; a flag needs wind along its length (+x for a left pin), not across it. No peer needed, no self collision, no soft bodies. Units, defaults and limits are in `scene_capabilities` under `cloth`; see "Recipe cloth" in `docs/physics.md`.+- Flags, curtains, capes and cloths draped over props use the recipe `cloth` section: a `grid` geometry node (flat sheet in XY, `segments` up to 64) listed in `sheets` with `pin` edges (`top`, `left`, a corner...), `wind` in m/s, and fixed `colliders` (`ball` or `box` fitted to their node). A cape or skirt on a moving character uses moving colliders (`attach` + `shape: "capsule"` + `to`, or `armature` for one capsule per bone) and a sheet `pinTo: { node }`. Seek with `motionTime`; a flag needs wind along its length (+x for a left pin), not across it. No peer needed, no self collision, no soft bodies. Units, defaults and limits are in `scene_capabilities` under `cloth`; see "Recipe cloth" in `docs/physics.md`.- Sparks, dust, embers, confetti and fur use the recipe `particles` section: a root node whose small low-poly geometry (a box) is the particle, in an emitter system with `seed`, `count` (at most 2000), `rate` (omit for one burst), `lifetime`, `speed`, `spread`, `drag`, `floor` and an `emitter` shape; a system with `type: "hair"` grows strands from a `surface` node. Seek with `motionTime`; the same seed and time give the same particles. The node transform does not move particles, and there is no collision beyond the floor, bounce or per-particle colour. Fields and limits are in `scene_capabilities` under `particles`; see "Recipe particles and hair" in `docs/physics.md`. Smoke, fire and browser liquid are not supported.- A closed room, or any scene lit by point lights, is dark where the light does not reach. Add the recipe `bake` section (`{ "preset": "standard" }`, or `draft` to try fast; `fine` is the same as `standard`) or the `set-bake` operation: it writes bounced light into lightmaps when the scene builds and reruns only when parts, lights, materials or the bake change. `scene_inspect` shows `bake.seconds` and `bake.skipped` (alpha, skins, morphs, procedural tags and hair or particle hosts are left as they were; patterned, textured and emissive parts are baked and keep their pattern). It bakes the rest pose and no spot or area lights. Read `docs/light-bake.md`.- Agent procedural animation uses `docs/agent-motion.md` from release `0.7.0`. Author recipe tracks, then seek through `sampleMotion` or MCP `motionTime`.diff --git a/src/agent-asset.js b/src/agent-asset.jsindex 8b2c3177..54622e7e 100644--- a/src/agent-asset.js+++ b/src/agent-asset.js@@ -262,10 +262,13 @@ function framedCamera(bounds) {/** Own authoring state around one exclusively owned, decoded parseGltf asset. */export function createAgentAssetScene(asset, inputRecipe, { sources } = {}) {- if (!asset || !Array.isArray(asset.parts) || !asset.parts.length || !Array.isArray(asset.animations) ||+ if (!asset || !Array.isArray(asset.parts) || !Array.isArray(asset.animations) ||typeof asset.stagePose !== 'function' || !Array.isArray(asset.nodes)) {throw new TypeError('createAgentAssetScene requires a decoded parseGltf asset.');}+ if (!asset.parts.length) {+ throw new TypeError('This file has no meshes, so there is nothing to draw. A skeleton-only file works as a clip source: set motion.layers[].from on a character that has a mesh.');+ }if (asset.parts.length > 4096 || asset.animations.length > 256 ||new Set(asset.parts.map(part => part.material)).size > 4096) {throw new RangeError('Asset budgets allow 4096 parts, 4096 materials, and 256 animation clips.');diff --git a/tests/mcp-node-fields.test.js b/tests/mcp-node-fields.test.jsnew file mode 100644index 00000000..59c0c81d--- /dev/null+++ b/tests/mcp-node-fields.test.js@@ -0,0 +1,42 @@+import './support/load-agent-scene-modules.mjs';+import test from 'node:test';+import assert from 'node:assert/strict';+import { readFile } from 'node:fs/promises';+import { Client } from '@modelcontextprotocol/client';+import { InMemoryTransport } from '@modelcontextprotocol/server';+import { createAgentMcpServer } from '../scripts/mcp-server.mjs';++async function backendNodeFields() {+ const source = await readFile(new URL('../src/agent-scene.js', import.meta.url), 'utf8');+ const list = source.match(/const NODE_FIELDS = \[([^\]]+)\]/);+ assert.ok(list, 'NODE_FIELDS moved out of src/agent-scene.js');+ return [...list[1].matchAll(/'(\w+)'/g)].map(match => match[1]);+}++function findOperationSchema(node, type) {+ if (!node || typeof node !== 'object') return null;+ if (node.properties?.type?.const === type) return node;+ for (const value of Object.values(node)) {+ const found = findOperationSchema(value, type);+ if (found) return found;+ }+ return null;+}++test('the scene_apply set-node schema accepts every node field the backend does', async () => {+ const server = createAgentMcpServer({ async close() {} });+ const client = new Client({ name: '67js-node-fields-test', version: '1' });+ const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();+ await server.connect(serverTransport);+ await client.connect(clientTransport);+ try {+ const { tools } = await client.listTools();+ const setNode = findOperationSchema(tools.find(tool => tool.name === 'scene_apply').inputSchema, 'set-node');+ assert.ok(setNode, 'scene_apply no longer lists set-node');+ const rejected = (await backendNodeFields()).filter(field => !(field in setNode.properties));+ assert.deepEqual(rejected, [], 'These node fields pass the backend but the MCP schema rejects them');+ } finally {+ await client.close();+ await server.close();+ }+});