Skip to content

3D engine that AI agents drive

Merge remote-tracking branch 'origin/main' into 922-ask-user-time-mark

Nguyen Xuan An2091745f88ea
diff --cc docs/mcp.md
index 7c038528,2afa13ca..7b266c6d
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@@ -221,7 -221,9 +221,9 @@@ Set operations use `set-node`, `set-mat
An explicit `null` restores the authored base pose. Release `0.8.0` also supports imported source-node motion.
Imported clip sampling remains available through `set-animation`.
- `scene_ask_user` is for a change the person cannot put in words. It renders the current view, opens it in their default browser on a loopback address under a random path, and waits up to `timeoutSeconds` (default 300, at most 900) for them to pin up to 16 points with a note of at most 280 characters each and press Send. Each mark comes back with its pixel, the `partId` and world `point` and `normal` the pin lands on (all `null` on empty sky), the `distance` from the camera and the part's `material`, plus the `view` camera and a PNG with the numbered pins. A cancelled or timed-out page returns `outcome` `cancelled` or `timeout` with no marks. Nothing leaves the machine and the tool takes no code or remote URL. `motionTime` (0 to 3600 seconds) first seeks the scene's authored motion to that time, like `scene_look`, so a film scene can be marked at one moment: the picture shows that moment, the marks are resolved with the scene put back at it, and `view.motionTime` in the answer names it (left out for a scene with no motion). Not yet marks: lasso regions, dragged transforms, colours and saved views (issue #922 lists them).
+ `scene_ask_user` is for a change the person cannot put in words. It renders the current view, opens it in their default browser on a loopback address under a random path, and waits up to `timeoutSeconds` (default 300, at most 900) for them to pin up to 16 points with a note of at most 280 characters each and press Send. Each mark comes back with its pixel, the `partId` and world `point` and `normal` the pin lands on (all `null` on empty sky), the `distance` from the camera and the part's `material`, plus the `view` camera and a PNG with the numbered pins. A cancelled or timed-out page returns `outcome` `cancelled` or `timeout` with no marks. Nothing leaves the machine and the tool takes no code or remote URL.
+
-The page also has a lasso tool: the person draws a closed outline around a region (3 to 64 points, thinned from the pen stroke) and adds a note, so one mark can mean "these parts". Such a mark counts toward the 16 and comes back with the pin fields plus `kind` `"lasso"`, `points` and `parts`. `parts` lists the parts under the outline, most covered first, each as `{ partId, samples, coverage, material }`: the server picks a fixed 24 by 24 grid of pixels over the outline's bounding box, keeps the ones inside it (even-odd rule) and runs the same pick as a pin on each, so `coverage` is the share of those pixels a part holds (pixels on empty sky count toward no part, so the shares may add up to less than 1). `pixel`, `partId`, `point`, `normal`, `distance` and `material` are those of the most covered part, taken at the picked pixel nearest the outline's centre; `parts` is empty and `partId` `null` when the outline holds only sky. A part under 0.15 coverage carries `edge: true` and stays listed: it only grazes the outline. A part whose material is also used by parts outside the mark carries `materialSharedWith` (those part IDs, sorted, at most 8, plus `materialSharedWithMore` with the count left over): recolouring that material would change them too, so give the part its own material first. Pins are unchanged. The inline chat view and the review page take pins only. Not yet marks: dragged transforms, colours, saved views and film times (issue #922 lists them).
++The page also has a lasso tool: the person draws a closed outline around a region (3 to 64 points, thinned from the pen stroke) and adds a note, so one mark can mean "these parts". Such a mark counts toward the 16 and comes back with the pin fields plus `kind` `"lasso"`, `points` and `parts`. `parts` lists the parts under the outline, most covered first, each as `{ partId, samples, coverage, material }`: the server picks a fixed 24 by 24 grid of pixels over the outline's bounding box, keeps the ones inside it (even-odd rule) and runs the same pick as a pin on each, so `coverage` is the share of those pixels a part holds (pixels on empty sky count toward no part, so the shares may add up to less than 1). `pixel`, `partId`, `point`, `normal`, `distance` and `material` are those of the most covered part, taken at the picked pixel nearest the outline's centre; `parts` is empty and `partId` `null` when the outline holds only sky. A part under 0.15 coverage carries `edge: true` and stays listed: it only grazes the outline. A part whose material is also used by parts outside the mark carries `materialSharedWith` (those part IDs, sorted, at most 8, plus `materialSharedWithMore` with the count left over): recolouring that material would change them too, so give the part its own material first. Pins are unchanged. The inline chat view and the review page take pins only. `motionTime` (0 to 3600 seconds) first seeks the scene's authored motion to that time, like `scene_look`, so a film scene can be marked at one moment: the picture shows that moment, the marks are resolved with the scene put back at it, and `view.motionTime` in the answer names it (left out for a scene with no motion). Not yet marks: dragged transforms, colours and saved views (issue #922 lists them).
#### Inline in chat apps (MCP Apps)
diff --cc scripts/agent-runtime.mjs
index ce8f7a8f,7b14458b..a6a70d34
--- a/scripts/agent-runtime.mjs
+++ b/scripts/agent-runtime.mjs
@@@ -1350,8 -1344,13 +1350,13 @@@ export function createAgentRuntime({ ke
const height = input.height ?? 512;
const maxMarks = input.maxMarks ?? ANNOTATE_LIMITS.maxMarks;
const timeoutSeconds = input.timeoutSeconds ?? ANNOTATE_LIMITS.defaultTimeoutSeconds;
- const frame = await captureAnnotateFrame(input.sessionId, width, height);
+ const frame = await captureAnnotateFrame(input.sessionId, width, height, input.motionTime);
if (closing) throw new Error('The agent runtime is closing.');
+ if (askUserFixture) {
+ // Test-only (#1956): the fixture stands in for the person, and its marks go through the same pick as a browser answer.
+ const marks = validateMarks(askUserFixture.marks, { width: frame.width, height: frame.height, maxMarks });
+ return resolveSentMarks({ sessionId: input.sessionId, frame, marks, image: frame.data });
+ }
if (input.inline) {
// The nonce goes to the view in the result's _meta only, never in the content the model reads: the model cannot call the
// app-only submit tool with a person's marks because it never sees the nonce.
diff --cc scripts/mcp-server.mjs
index bae3a3fc,e3a7913a..d1e9bbe2
--- a/scripts/mcp-server.mjs
+++ b/scripts/mcp-server.mjs
@@@ -1036,7 -1037,7 +1037,7 @@@ export function createAgentMcpServer(ru
const metadata = findings ? lookMetadata : metadataWithoutFindings;
return { content: [{ type: 'image', data, mimeType }, { type: 'text', text: JSON.stringify(metadata) }], structuredContent: metadata };
});
- const askUserTool = register('scene_ask_user', 'Open the view in the browser so the person pins points with notes; waits for Send. Use for a change hard to say in words. Returns marks with partId, point, normal, note, and the marked PNG. reviewFile: the text a review page saved, read instead of opening the browser.', z.strictObject({ ...sessionFields, reviewFile: z.string().max(16384).optional().describe('Text of the file a scene_export_web review page saved.'), prompt: z.string().max(200).optional().describe('Line shown above the picture.'), motionTime: z.number().min(0).max(3600).optional(), maxMarks: z.number().int().min(1).max(16).default(16), timeoutSeconds: z.number().int().min(10).max(900).default(300) }), askUser, async argumentsValue => {
- const askUserTool = register('scene_ask_user', 'Open the view in the browser so the person pins points with notes; waits for Send. Use for a change hard to say in words. Returns marks with partId, point, normal, note, and the marked PNG. materialSharedWith: own material first. reviewFile: the text a review page saved, read instead of opening the browser.', z.strictObject({ ...sessionFields, reviewFile: z.string().max(16384).optional().describe('Text of the file a scene_export_web review page saved.'), prompt: z.string().max(200).optional().describe('Line shown above the picture.'), maxMarks: z.number().int().min(1).max(16).default(16), timeoutSeconds: z.number().int().min(10).max(900).default(300) }), askUser, async argumentsValue => {
++ const askUserTool = register('scene_ask_user', 'Open the view in the browser so the person pins points with notes; waits for Send. Use for a change hard to say in words. Returns marks with partId, point, normal, note, and the marked PNG. materialSharedWith: own material first. reviewFile: the text a review page saved, read instead of opening the browser.', z.strictObject({ ...sessionFields, reviewFile: z.string().max(16384).optional().describe('Text of the file a scene_export_web review page saved.'), prompt: z.string().max(200).optional().describe('Line shown above the picture.'), motionTime: z.number().min(0).max(3600).optional(), maxMarks: z.number().int().min(1).max(16).default(16), timeoutSeconds: z.number().int().min(10).max(900).default(300) }), askUser, async argumentsValue => {
const { content, image, viewMeta } = await runtime.askUser(clientShowsApps ? { ...argumentsValue, inline: true } : argumentsValue);
return { content: [...(image ? [{ type: 'image', data: image, mimeType: 'image/png' }] : []), { type: 'text', text: JSON.stringify(content) }], ...(viewMeta ? { _meta: { 'crucible/annotate': viewMeta } } : {}) };
});
diff --cc src/agent-scene-capabilities.js
index decdea9e,83c520e5..cb685e41
--- a/src/agent-scene-capabilities.js
+++ b/src/agent-scene-capabilities.js
@@@ -236,7 -236,7 +236,7 @@@ export const AGENT_SCENE_CAPABILITIES
terrain: 'One mesh, x and z from -size/2 to size/2, hills about y = 0. size 40 m (1-2000), resolution 64 cells (2-256), seed 1, amplitude 1 m (0-200), holes up to 16 of {center:[x,z],radius}, no surface inside; radius a few cells or more.',
vehicleDamage: 'Geometry dented {source, impacts:[{point, direction, impulse N s}] 1-8, stiffness 2000, maxDepth 0.25 m, radius 0.4}: the source (not a fragment or dent) pushed in around each point by min(maxDepth, impulse/stiffness); a part that comes off is a body on a fixed joint with breakImpulse. docs/vehicle.md.',
placedAssets: 'Node field asset:{id or name,materials?,flatten?,clip?,drive?} turns a group node into a catalog GLB (id from asset_catalog, or name: its exact file name, stored as the id; give one). Its transform places it. scene_apply add-node with an asset field places the first or any later instance, so a scene need not start with its assets. materials:[{name,id?,color,metallic,roughness,opacity,emission,emissiveColor}] override source materials by name; an id (such as "stool-heat") gives that material a handle that a set-motion material track can drive: emission and emissiveColor fade a heat glow, one id per placement. emissionMask:{axis:x|y|z,from,to} (metres in the own space of the part, smooth ramp, from above to flips it) multiplies emission so the glow sits only at one end while texture and shading stay; rasterizer only, path tracers ignore it. flatten:{color?,unlit?} flattens every material of the placement in one call: unlit true draws them without lighting, color [r,g,b] paints them one colour (textures dropped), both give a flat silhouette; a named override still wins. clip:{index,start=0,speed=1,loop=true} plays that clip from scene time start; set-motion tracks move the node and scene time seeks the clip. drive:{path | legs, cruiseSpeed | speed, wheels, lights} drives a car along a path in the placement frame (docs/vehicle.md). Max 512 placements, 16 assets, 16 animated.',
- askUser: 'scene_ask_user opens the current view in their browser; they pin up to maxMarks (16) points, each with a note, and press Send. Ask when a change is hard to say in words ("warmer here", "this bit"), not for facts you can inspect. Each mark returns partId, point, normal, note, distance and material (null on sky), plus view.camera and a PNG with numbered pins. Apply the change the note names to that partId, then to similar parts (same material or name prefix) when the note says so. outcome cancelled or timeout returns no marks: ask in text. For a film scene pass motionTime to mark one moment; view.motionTime names it. Edits made while they mark change revision; the result warns.',
- askUser: 'scene_ask_user opens the current view in their browser; they pin up to maxMarks (16) points, each with a note, and press Send. Ask when a change is hard to say in words ("warmer here", "this bit"), not for facts you can inspect. Each mark returns partId, point, normal, note, distance and material (null on sky), plus view.camera and a PNG with numbered pins. Apply the change the note names to that partId, then to similar parts (same material or name prefix) when the note says so. A mark with kind "lasso" is an outline around a region: apply the note to every part in its parts list (partId, coverage, material, most covered first), and skip a part with a tiny coverage that the note does not fit. A part with materialSharedWith has that material on parts outside the mark: give it its own material before you change it. A part with edge true only grazes the outline. outcome cancelled or timeout returns no marks: ask in text. Edits made while they mark change revision; the result warns.',
++ askUser: 'scene_ask_user opens the current view in their browser; they pin up to maxMarks (16) points, each with a note, and press Send. Ask when a change is hard to say in words ("warmer here", "this bit"), not for facts you can inspect. Each mark returns partId, point, normal, note, distance and material (null on sky), plus view.camera and a PNG with numbered pins. Apply the change the note names to that partId, then to similar parts (same material or name prefix) when the note says so. A mark with kind "lasso" is an outline around a region: apply the note to every part in its parts list (partId, coverage, material, most covered first), and skip a part with a tiny coverage that the note does not fit. A part with materialSharedWith has that material on parts outside the mark: give it its own material before you change it. A part with edge true only grazes the outline. outcome cancelled or timeout returns no marks: ask in text. For a film scene pass motionTime to mark one moment; view.motionTime names it. Edits made while they mark change revision; the result warns.',
navigation: 'scene_inspect path:{ground,from:[x,z],to:[x,z]} returns a route over the ground node around its holes, steep slopes and the nodes in avoid:[ids] (box footprints), as keyframes [{time,value:[x,y,z]}] for a translation track with interpolation LINEAR. radius 0.3 m; maxSlope 40 degrees; cellSize defaults to radius; seconds or speed (1.4 m/s); lift adds y.',
meshEdits: Object.freeze({ operations: Object.freeze(Object.keys(MESH_EDIT_OPERATIONS)), maxEdits: MAX_MESH_EDITS }),
meshStack: Object.freeze({ operations: STACK_OPERATIONS, maxNodes: MAX_STACK_NODES }),