Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Add scene_ask_user so people can pin the scene for the agent (#922)

Nguyen Xuan An56f27f12745e
diff --git a/changes/922-ask-user.md b/changes/922-ask-user.md
new file mode 100644
index 00000000..daacc0f2
--- /dev/null
+++ b/changes/922-ask-user.md
@@ -0,0 +1 @@
+- `scene_ask_user` opens the current view in the browser so a person can pin points with notes, and returns each pin as a part ID, world point and normal (#922)
diff --git a/changes/922-ask-user.size.md b/changes/922-ask-user.size.md
new file mode 100644
index 00000000..f2889907
--- /dev/null
+++ b/changes/922-ask-user.size.md
@@ -0,0 +1,4 @@
+- tool-total: scene_ask_user is a new tool (871 bytes) that lets a person pin points on the scene, so an agent gets a part ID and a world point instead of guessing from words; it keeps to three optional fields and its detail sits in scene_capabilities askUser.
+- scene:agent-scene: the askUser topic text in scene_capabilities (about 290 bytes gzip) tells an agent when to ask and how to apply a marked change, which keeps the tool schema small; the mark code itself loads only in the MCP preview page.
+- scene:agent-scene+lazy: the same topic text, counted again with the lazy chunks.
+- goal:agent-scene: the same topic text; the agent-facing guidance is worth the bytes.
diff --git a/docs/learn/connect-mcp.md b/docs/learn/connect-mcp.md
index b0175f38..bb421639 100644
--- a/docs/learn/connect-mcp.md
+++ b/docs/learn/connect-mcp.md
@@ -248,7 +248,7 @@ The Windsurf documentation pages now redirect to Devin Desktop, which names `.de
## Check that it works
-First, ask the harness to list its MCP tools. You must see 24 tools whose names start with `scene_`, `mesh_` or `scene_render_`. Some harnesses add the server name in front, for example `mcp__67js-scene__scene_look` in Claude Code.
+First, ask the harness to list its MCP tools. You must see 25 tools whose names start with `scene_`, `mesh_` or `scene_render_`. Some harnesses add the server name in front, for example `mcp__67js-scene__scene_look` in Claude Code.
Then give the agent this prompt:
diff --git a/docs/learn/models-and-ai-agents.md b/docs/learn/models-and-ai-agents.md
index 37b92c51..987ee975 100644
--- a/docs/learn/models-and-ai-agents.md
+++ b/docs/learn/models-and-ai-agents.md
@@ -10,7 +10,7 @@ Three pages cover the details:
## What the agent does
-The server has 24 tools for authoring, previewing and exporting scenes. A normal session uses them in this order:
+The server has 25 tools for authoring, previewing and exporting scenes. A normal session uses them in this order:
1. `scene_capabilities` reads the supported features and limits.
2. `scene_create` makes a scene from a [recipe](/agent-scene.md). A recipe is JSON with object IDs, geometry, materials, a hierarchy and a camera.
diff --git a/docs/mcp.md b/docs/mcp.md
index 0aa0762e..1fdd5e7e 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -77,7 +77,7 @@ Every recorded agent run used Claude Code. This table says what was tried elsewh
| Client or check | Result |
|---|---|
| Official MCP SDK client over stdio (`npm run smoke:mcp-package`) | Passes. |
-| MCP Inspector CLI 2.13.0 (`npm run smoke:mcp-clients`) | Passes, 2026-11-03, Windows, `--backend cpu`. Lists all 24 tools through `tools/list` in 7 s. `tools/call` of `scene_create` with the lamp recipe succeeds in 10 s. The CLI starts a new server for every command and the scene lives in the server, so it cannot follow with `scene_look`. That call fails with "No scene is active" and returns no image. The preview image is not testable through this CLI. |
+| MCP Inspector CLI 2.13.0 (`npm run smoke:mcp-clients`) | Passes, 2026-11-03, Windows, `--backend cpu`. Lists all 25 tools through `tools/list` in 7 s. `tools/call` of `scene_create` with the lamp recipe succeeds in 10 s. The CLI starts a new server for every command and the scene lives in the server, so it cannot follow with `scene_look`. That call fails with "No scene is active" and returns no image. The preview image is not testable through this CLI. |
| Official MCP Python SDK 2.3.0 (`npm run smoke:mcp-clients`) | Passes, 2026-11-03, Windows, `--backend cpu`. Initialize, `tools/list` (22 tools), `scene_create` and `scene_look` in one session. `scene_look` returns a 20480 byte PNG. 11 s including the first `uv` download. |
| JSON Schema 2020-12 validator with strict mode (Ajv 8), run in `npm test` | Every tool input schema compiles. |
| Tool names and descriptions | Names are 9 to 24 characters from `[a-z_]`, within the 64-character limit OpenAI, Gemini and Anthropic share. No description is over 1024 characters. |
@@ -133,6 +133,7 @@ Inspect or export the recipe before retrying. Context loss still permits recipe
| `mesh_edit` | `sessionId`, `expectedRevision`, `geometry`, `edits`, optional `replace` | Updated revision, render status and the mesh report with per-edit results |
| `mesh_stack` | `sessionId`, `expectedRevision`, `geometry`, optional `stack`, `apply` | Updated revision, render status and the mesh report with per-node results |
| `scene_look` | `sessionId`, optional `width`, `height`, `motionTime`, `labels` | PNG image and metadata |
+| `scene_ask_user` | `sessionId`, optional `prompt`, `maxMarks`, `timeoutSeconds` | Opens the view in the browser, waits for Send, returns numbered marks with `partId`, `point`, `normal`, `note`, the camera and the marked PNG |
| `scene_review` | `sessionId`, `expectedRevision`, `shots` (each with an optional `branch`), optional `width`, `height`, `labels`, `response_format` | Labeled PNG grid and metadata from an isolated snapshot |
| `scene_checkpoint` | `sessionId` | Checkpoint ID, session ID, and revision |
| `scene_restore` | `sessionId`, `expectedRevision`, and a `checkpointId`, a `revision` or a `branch` | Updated revision and render status |
@@ -174,6 +175,8 @@ Set operations use `set-node`, `set-material`, or `set-camera`, with fields dire
An explicit `null` restores the authored base pose. Release `0.8.0` also supports imported source-node motion.
Imported clip sampling remains available through `set-animation`.
+`scene_ask_user` is for a change the person cannot put in words. It renders the current view, opens it in their default browser on a loopback address under a random path, and waits up to `timeoutSeconds` (default 300, at most 900) for them to pin up to 16 points with a note of at most 280 characters each and press Send. Each mark comes back with its pixel, the `partId` and world `point` and `normal` the pin lands on (all `null` on empty sky), the `distance` from the camera and the part's `material`, plus the `view` camera and a PNG with the numbered pins. A cancelled or timed-out page returns `outcome` `cancelled` or `timeout` with no marks. Nothing leaves the machine and the tool takes no code or remote URL. Not yet marks: lasso regions, dragged transforms, colours, saved views and film times (issue #922 lists them).
+
### Response format
`scene_review`, `scene_render_status`, `scene_render_result` and `scene_capabilities` take `response_format`: `concise` or `detailed`.
diff --git a/index.html b/index.html
index b1781989..f04303e1 100644
--- a/index.html
+++ b/index.html
@@ -85,7 +85,7 @@
</div>
<div class="copy">
<h1>3D engine that<br />AI agents drive</h1>
- <p class="lede">Glaze draws real-time frames. Cook path traces stills. 24 MCP tools drive them.</p>
+ <p class="lede">Glaze draws real-time frames. Cook path traces stills. 25 MCP tools drive them.</p>
<div class="hero-actions">
<a href="/docs.html#getting-started" data-icon="book">Get started</a>
<a href="https://www.npmjs.com/package/@codepawlhq/67js"><span class="package-logo package-logo-npm brand-npm" aria-hidden="true"></span>npm</a>
@@ -168,7 +168,7 @@ renderer.render({ eye: [58, 31, 84], target: [-8, 3, -18] });</code></pre>
</div>
</div>
<div class="agent-cards">
- <a href="/mcp.md" data-icon="cube"><h3>24 MCP tools</h3><p>Create, edit, explode, blend and render over stdio.</p></a>
+ <a href="/mcp.md" data-icon="cube"><h3>25 MCP tools</h3><p>Create, edit, explode, blend and render over stdio.</p></a>
<a href="/path-tracer.md" data-icon="code"><h3>Glaze + Cook</h3><p>Real-time frames, or path-traced stills.</p></a>
<a href="/agent-review.md" data-icon="image"><h3>Visual review</h3><p>Compare labeled views and normals from one snapshot.</p></a>
</div>
diff --git a/public/llms.txt b/public/llms.txt
index 50b5c8b7..40660e73 100644
--- a/public/llms.txt
+++ b/public/llms.txt
@@ -2,7 +2,7 @@
> A 3D engine that AI agents drive. Glaze draws real-time scenes on WebGL2, and Cook path traces stills on the CPU or WebGL2, and on Vulkan with the optional render-native package. PNG, MP4 and GLB output comes from the same scene.
-67js has an MCP server (`67js-mcp`, 24 tools) that builds, edits, looks at and renders scenes from JSON recipes.
+67js has an MCP server (`67js-mcp`, 25 tools) that builds, edits, looks at and renders scenes from JSON recipes.
The same engine is a JavaScript library for pages and Node. The library API is in the guide below.
Scenes are plain data with stable IDs, so an agent can edit by ID and check each change with a rendered image.
@@ -58,6 +58,7 @@ Scenes are plain data with stable IDs, so an agent can edit by ID and check each
- `mesh_stack`: Set the non-destructive modifier stack of a recipe mesh geometry (ops and fields: scene_capabilities meshModifiers), evaluated after its...
- `scene_apply`: Atomically edit transforms, look, camera or materials.
- `scene_look`: Render the active scene as a bounded PNG, optionally with part labels.
+- `scene_ask_user`: Open the view in the browser so the person pins points with notes; waits for Send.
- `scene_review`: Compare one to four independent camera or motion samples in one labeled PNG grid.
- `scene_checkpoint`: Save a private recipe snapshot.
- `scene_restore`: Restore a checkpoint atomically using the current revision.
diff --git a/scripts/agent-preview-browser.js b/scripts/agent-preview-browser.js
index 5eff4482..3e68bef3 100644
--- a/scripts/agent-preview-browser.js
+++ b/scripts/agent-preview-browser.js
@@ -33,6 +33,7 @@ import { describeApplication } from '../src/agent-apply-report.js';
import { framingProblems, exposureProblems, lumaShares } from '../src/frame-checks.js';
import { BRANCH_LIMITS, createBranchStore, printableLabel } from '../src/agent-branches.js';
import { agentAssetDecodeOptions, withCatalogLimitAdvice } from '../src/agent-asset-limits.js';
+import { describeMarks, validateMarks } from '../src/agent-annotate.js';
import { sceneQueryFor, summarizeWalkMap, walkOptionsFor, createQueryOnlyCache } from './agent-walkmap.js';
// Open scenes by sessionId, least recently used first. One canvas serves them all, so at most one record owns a renderer;
@@ -1176,6 +1177,20 @@ window.__agentPreview = {
motionTime: record.scene.motionTime ?? null, ...render, ...(labels ? { labels } : {}) },
};
},
+ // One frame for scene_ask_user: the look, plus the camera it was drawn with so the marks can be turned into rays later.
+ annotateFrame({ sessionId }) {
+ const frame = this.look({ sessionId });
+ const camera = Object.fromEntries(Object.entries(requireSession(sessionId).scene.camera)
+ .map(([field, value]) => [field, ArrayBuffer.isView(value) ? Array.from(value) : value]));
+ return { ...frame, width: canvas.width, height: canvas.height, camera };
+ },
+ async pickMarks({ sessionId, camera, marks, width, height }) {
+ const record = requireSession(sessionId);
+ if (!allAgentSceneModulesLoaded()) await loadAllAgentSceneModules();
+ const clean = validateMarks(marks, { width, height });
+ const query = sceneQueryFor(record);
+ return { revision: record.scene.revision, marks: describeMarks(clean, camera, { width, height, raycast: ray => query.raycast(ray) }) };
+ },
checkpoint({ sessionId, checkpointId }) {
const record = requireSession(sessionId);
// The oldest checkpoint goes first when the scene is at its limit; the answer names it so the agent can drop it from its notes.
diff --git a/scripts/agent-runtime.mjs b/scripts/agent-runtime.mjs
index fba9daa4..ccbfb5a8 100644
--- a/scripts/agent-runtime.mjs
+++ b/scripts/agent-runtime.mjs
@@ -32,6 +32,9 @@ import { DEFAULT_PAGE_ENCODER_QUANTIZER, createFrameReceiver, startVideoEncoderI
import { audioInputArguments, createAudioLibrary, measureStreamSeconds } from './job-audio.mjs';
import { filmTimelineDuration, filmTimelineShotStarts, sampleFilmTimeline } from '../src/film-timeline.js';
import { isPreviewHost } from './preview-host.mjs';
+import { runAnnotateSession } from './annotate-session.mjs';
+import { openInDefaultBrowser } from './annotate-open.mjs';
+import { ANNOTATE_LIMITS } from '../src/agent-annotate.js';
import { normalizeReview, REVIEW_LIMITS } from './agent-review.mjs';
import { mergeFrameDiagnostics } from '../src/frame-checks.js';
import { BRANCH_LIMITS, validateBranchName } from '../src/agent-branches.js';
@@ -80,7 +83,7 @@ const VENDOR_FILES = Object.freeze({
});
export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', browser: browserPath, memory = 192, assets: assetRoot,
- jobs: jobRoot, ffmpeg = 'ffmpeg', audio: audioRoot,
+ jobs: jobRoot, ffmpeg = 'ffmpeg', audio: audioRoot, openPage = openInDefaultBrowser,
previewUrl = new URL(/* @vite-ignore */ '../build/mcp/preview.js', import.meta.url) } = {}) {
if (!['gpu', 'cpu'].includes(backend)) {
throw new RangeError('backend must be gpu or cpu.');
@@ -879,6 +882,38 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
return result;
});
},
+ // Shows the current view in the person's own browser, waits for their pins and notes, and resolves each pin to a part and a
+ // world point. The wait holds no preview queue slot, so other tools keep working while the person marks.
+ async askUser(input) {
+ const width = input.width ?? 768;
+ const height = input.height ?? 512;
+ if (![width, height].every(value => Number.isInteger(value) && value >= 64 && value <= 1024)) {
+ throw new RangeError('Preview dimensions must be integers from 64 to 1024.');
+ }
+ const maxMarks = input.maxMarks ?? ANNOTATE_LIMITS.maxMarks;
+ const timeoutSeconds = input.timeoutSeconds ?? ANNOTATE_LIMITS.defaultTimeoutSeconds;
+ const frame = await enqueue(async () => {
+ await requirePreviewRenderer();
+ requireSession(input.sessionId);
+ await page.setViewportSize({ width, height });
+ return invoke('annotateFrame', { sessionId: input.sessionId });
+ });
+ const session = await runAnnotateSession({ png: frame.data, width: frame.width, height: frame.height, prompt: input.prompt ?? '',
+ maxMarks, waitMs: timeoutSeconds * 1000, open: async url => {
+ console.error(`67js annotate page: ${url}`);
+ await openPage(url);
+ } });
+ const base = { sessionId: input.sessionId, revision: frame.metadata.revision, view: { width: frame.width, height: frame.height, camera: frame.camera } };
+ if (session.outcome !== 'sent') {
+ return { content: { ...base, outcome: session.outcome, marks: [], message: session.outcome === 'timeout'
+ ? `The person did not press Send within ${timeoutSeconds} seconds. Ask in text, or call scene_ask_user again with a longer timeoutSeconds.`
+ : 'The person cancelled without sending marks. Ask what they meant in text.' } };
+ }
+ const picked = await enqueue(() => invoke('pickMarks', { sessionId: input.sessionId, camera: frame.camera, marks: session.marks, width: frame.width, height: frame.height }));
+ const sceneChanged = picked.revision !== frame.metadata.revision;
+ return { image: session.image, content: { ...base, outcome: 'sent', marks: picked.marks,
+ ...(sceneChanged ? { warning: `The scene changed from revision ${frame.metadata.revision} to ${picked.revision} while the person marked; part IDs and points describe revision ${picked.revision}, the picture shows ${frame.metadata.revision}.` } : {}) } };
+ },
checkpoint(input) {
return enqueue(async () => {
await requirePreviewRenderer();
diff --git a/scripts/annotate-open.mjs b/scripts/annotate-open.mjs
new file mode 100644
index 00000000..10cceb08
--- /dev/null
+++ b/scripts/annotate-open.mjs
@@ -0,0 +1,18 @@
+import { spawn } from 'node:child_process';
+
+/** Opens a loopback address in the person's default browser. Only an address the annotate server built is passed in. */
+export function openInDefaultBrowser(url) {
+ if (!/^http:\/\/127\.0\.0\.1:\d+\/[a-f0-9]+\/$/.test(url)) {
+ return Promise.reject(new Error('Only the local annotate address can be opened.'));
+ }
+ const command = process.platform === 'win32' ? ['cmd', ['/c', 'start', '""', url]]
+ : process.platform === 'darwin' ? ['open', [url]] : ['xdg-open', [url]];
+ return new Promise((resolve, reject) => {
+ const child = spawn(command[0], command[1], { stdio: 'ignore', detached: true, windowsVerbatimArguments: true });
+ child.once('error', reject);
+ child.once('spawn', () => {
+ child.unref();
+ resolve();
+ });
+ });
+}
diff --git a/scripts/annotate-session.mjs b/scripts/annotate-session.mjs
new file mode 100644
index 00000000..3bb77aac
--- /dev/null
+++ b/scripts/annotate-session.mjs
@@ -0,0 +1,245 @@
+// The local page behind scene_ask_user (issue #922): it shows one rendered view, lets the person pin points on it with a short
+// note each, and returns the marks when they press Send. It listens on loopback only, under a random path, with no remote assets.
+import { createServer } from 'node:http';
+import { randomBytes } from 'node:crypto';
+import { isPreviewHost } from './preview-host.mjs';
+import { ANNOTATE_LIMITS, validateMarks, isBoundedPng } from '../src/agent-annotate.js';
+
+const MAXIMUM_BODY_BYTES = 2 * 1024 * 1024;
+
+function pageHtml({ prompt, width, height, maxMarks, maxNoteLength }) {
+ const escaped = prompt.replace(/[&<>"]/g, character => ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;' })[character]);
+ return `<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Mark the scene</title><style>
+*{box-sizing:border-box}
+body{margin:0;min-height:100vh;background:#182126;color:#e8eef0;font:15px/1.4 system-ui,sans-serif;display:flex;flex-direction:column;gap:16px;padding:20px}
+h1{margin:0;font-size:18px;font-weight:600}
+main{display:flex;gap:20px;align-items:flex-start;flex-wrap:wrap}
+.stage{position:relative;line-height:0;flex:0 1 ${width}px;max-width:100%;cursor:crosshair;user-select:none}
+.stage img{width:100%;height:auto;border-radius:8px;display:block}
+.pin{position:absolute;width:26px;height:26px;margin:-13px 0 0 -13px;border-radius:50%;background:#ff5a4f;color:#fff;font:600 13px/26px system-ui,sans-serif;text-align:center;box-shadow:0 0 0 2px #fff;pointer-events:none}
+aside{flex:1 1 280px;max-width:380px;display:flex;flex-direction:column;gap:12px;user-select:none}
+.row{display:flex;gap:10px;align-items:center}
+.badge{flex:none;width:26px;height:26px;border-radius:50%;background:#ff5a4f;color:#fff;font:600 13px/26px system-ui,sans-serif;text-align:center}
+input{flex:1;min-width:0;height:36px;border:0;border-radius:6px;background:#26343b;color:inherit;padding:0 10px;font:inherit;user-select:text}
+input:focus{outline:2px solid #6fb7ff}
+button{height:36px;border:0;border-radius:6px;padding:0 14px;font:inherit;color:inherit;background:transparent;cursor:pointer;display:inline-flex;gap:8px;align-items:center}
+button:hover{background:#26343b}
+button.primary{background:#2f7dd1;color:#fff}
+button.primary:hover{background:#3a8ae0}
+button.icon{width:36px;padding:0;justify-content:center;flex:none}
+button:disabled{opacity:.5;cursor:default}
+svg{width:18px;height:18px;flex:none}
+.actions{display:flex;gap:8px;margin-top:4px}
+.hint,.count{color:#9db0b8}
+.done{display:flex;gap:10px;align-items:center;font-size:17px}
+.done svg{width:26px;height:26px}
+.ok{color:#58d68d}.stop{color:#f0b429}
+</style></head><body>
+<h1>${escaped || 'Click the scene to mark what you mean'}</h1>
+<main>
+<div class="stage" id="stage"><img id="frame" src="frame.png" width="${width}" height="${height}" alt="Rendered scene"></div>
+<aside id="side">
+<div class="hint" id="hint">Click the picture to drop a pin, then say what to change there.</div>
+<div id="rows" style="display:flex;flex-direction:column;gap:8px"></div>
+<div class="count" id="count"></div>
+<div class="actions"><button class="primary" id="send">Send</button><button id="cancel">Cancel</button></div>
+</aside>
+</main>
+<script type="application/json" id="limits">${JSON.stringify({ width, height, maxMarks, maxNoteLength })}</script>
+<script src="app.js"></script>
+</body></html>`;
+}
+
+// Served as its own file because the page's content security policy allows scripts from this origin only.
+const PAGE_SCRIPT = `(() => {
+const limits = JSON.parse(document.getElementById('limits').textContent);
+const marks = [];
+const stage = document.getElementById('stage');
+const frame = document.getElementById('frame');
+const rows = document.getElementById('rows');
+const count = document.getElementById('count');
+const sendButton = document.getElementById('send');
+const trash = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M3 6h18M8 6V4h8v2M6 6l1 14h10l1-14"/></svg>';
+function render() {
+ stage.querySelectorAll('.pin').forEach(pin => pin.remove());
+ rows.textContent = '';
+ marks.forEach((mark, index) => {
+ const pin = document.createElement('div');
+ pin.className = 'pin';
+ pin.textContent = index + 1;
+ pin.style.left = (mark.x / limits.width * 100) + '%';
+ pin.style.top = (mark.y / limits.height * 100) + '%';
+ stage.appendChild(pin);
+ const row = document.createElement('div');
+ row.className = 'row';
+ const badge = document.createElement('span');
+ badge.className = 'badge';
+ badge.textContent = index + 1;
+ const input = document.createElement('input');
+ input.maxLength = limits.maxNoteLength;
+ input.value = mark.note;
+ input.placeholder = 'What should change here?';
+ input.addEventListener('input', () => { mark.note = input.value; });
+ const remove = document.createElement('button');
+ remove.className = 'icon';
+ remove.title = 'Remove pin';
+ remove.setAttribute('aria-label', 'Remove pin ' + (index + 1));
+ remove.innerHTML = trash;
+ remove.addEventListener('click', () => { marks.splice(index, 1); render(); });
+ row.append(badge, input, remove);
+ rows.appendChild(row);
+ mark.input = input;
+ });
+ count.textContent = marks.length + ' of ' + limits.maxMarks + ' pins';
+ sendButton.disabled = marks.length === 0;
+}
+stage.addEventListener('click', event => {
+ if (marks.length >= limits.maxMarks) return;
+ const box = frame.getBoundingClientRect();
+ const x = (event.clientX - box.left) / box.width * limits.width;
+ const y = (event.clientY - box.top) / box.height * limits.height;
+ marks.push({ x, y, note: '' });
+ render();
+ marks[marks.length - 1].input.focus();
+});
+function markedImage() {
+ const canvas = document.createElement('canvas');
+ canvas.width = limits.width;
+ canvas.height = limits.height;
+ const context = canvas.getContext('2d');
+ context.drawImage(frame, 0, 0, limits.width, limits.height);
+ context.font = '600 15px system-ui, sans-serif';
+ context.textAlign = 'center';
+ context.textBaseline = 'middle';
+ marks.forEach((mark, index) => {
+ context.beginPath();
+ context.arc(mark.x, mark.y, 14, 0, Math.PI * 2);
+ context.fillStyle = '#ff5a4f';
+ context.fill();
+ context.lineWidth = 2;
+ context.strokeStyle = '#fff';
+ context.stroke();
+ context.fillStyle = '#fff';
+ context.fillText(String(index + 1), mark.x, mark.y + 1);
+ });
+ return canvas.toDataURL('image/png').split(',')[1];
+}
+function finish(kind, text) {
+ const icon = kind === 'ok'
+ ? '<svg class="ok" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><path d="M7.5 12.5l3 3 6-6.5"/></svg>'
+ : '<svg class="stop" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><path d="M8 12h8"/></svg>';
+ document.getElementById('side').innerHTML = '<div class="done">' + icon + '<span>' + text + '</span></div>';
+}
+async function post(path, body) {
+ const response = await fetch(path, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
+ if (!response.ok) throw new Error((await response.text()) || 'The request failed.');
+}
+sendButton.addEventListener('click', async () => {
+ sendButton.disabled = true;
+ try {
+ await post('send', { marks: marks.map(({ x, y, note }) => ({ x, y, note })), image: markedImage() });
+ finish('ok', 'Sent to the agent. You can close this tab.');
+ } catch (error) {
+ sendButton.disabled = false;
+ document.getElementById('hint').textContent = error.message;
+ }
+});
+document.getElementById('cancel').addEventListener('click', async () => {
+ await post('cancel', {}).catch(() => {});
+ finish('stop', 'Cancelled. You can close this tab.');
+});
+render();
+})();`;
+
+function readBody(request) {
+ return new Promise((resolve, reject) => {
+ const chunks = [];
+ let total = 0;
+ request.on('data', chunk => {
+ total += chunk.length;
+ if (total > MAXIMUM_BODY_BYTES) {
+ reject(new RangeError('The request body is too large.'));
+ request.destroy();
+ return;
+ }
+ chunks.push(chunk);
+ });
+ request.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
+ request.on('error', reject);
+ });
+}
+
+/**
+ * Serves one annotate page and waits for the person. `open(url)` shows the page (the runtime opens the browser; a test fetches it).
+ * Resolves { outcome: 'sent' | 'cancelled' | 'timeout', url, marks?, image? }; a rejected `open` rejects with the page address.
+ */
+export async function runAnnotateSession({ png, width, height, prompt = '', maxMarks = ANNOTATE_LIMITS.maxMarks, waitMs, open }) {
+ const token = randomBytes(16).toString('hex');
+ const base = `/${token}/`;
+ const frameBytes = Buffer.from(png, 'base64');
+ const html = pageHtml({ prompt, width, height, maxMarks, maxNoteLength: ANNOTATE_LIMITS.maxNoteLength });
+ let finish;
+ const finished = new Promise(resolve => { finish = resolve; });
+ const server = createServer(async (request, response) => {
+ response.setHeader('Cache-Control', 'no-store');
+ response.setHeader('Content-Security-Policy', "default-src 'none'; script-src 'self'; style-src 'unsafe-inline'; img-src 'self' data: blob:; connect-src 'self'");
+ if (!isPreviewHost(request.headers.host, server.address().port) || !request.url.startsWith(base)) {
+ response.writeHead(404);
+ response.end();
+ return;
+ }
+ const route = request.url.slice(base.length);
+ try {
+ if (request.method === 'GET' && route === '') {
+ response.setHeader('Content-Type', 'text/html; charset=utf-8');
+ response.end(html);
+ } else if (request.method === 'GET' && route === 'app.js') {
+ response.setHeader('Content-Type', 'application/javascript; charset=utf-8');
+ response.end(PAGE_SCRIPT);
+ } else if (request.method === 'GET' && route === 'frame.png') {
+ response.setHeader('Content-Type', 'image/png');
+ response.end(frameBytes);
+ } else if (request.method === 'POST' && route === 'cancel') {
+ response.end('ok');
+ finish({ outcome: 'cancelled' });
+ } else if (request.method === 'POST' && route === 'send') {
+ const body = JSON.parse(await readBody(request));
+ const marks = validateMarks(body.marks, { width, height, maxMarks });
+ if (!marks.length) throw new TypeError('Add at least one pin before sending.');
+ if (!isBoundedPng(body.image)) throw new TypeError('The marked picture is not a PNG within 1 MiB.');
+ response.end('ok');
+ finish({ outcome: 'sent', marks, image: body.image });
+ } else {
+ response.writeHead(404);
+ response.end();
+ }
+ } catch (error) {
+ response.writeHead(400, { 'Content-Type': 'text/plain; charset=utf-8' });
+ response.end(error.message);
+ }
+ });
+ await new Promise((resolve, reject) => {
+ server.once('error', reject);
+ server.listen(0, '127.0.0.1', resolve);
+ });
+ const url = `http://127.0.0.1:${server.address().port}${base}`;
+ let timer;
+ try {
+ try {
+ await open(url);
+ } catch (error) {
+ throw new Error(`Could not open the browser (${error.message}). The annotate page was ${url}`);
+ }
+ const timeout = new Promise(resolve => { timer = setTimeout(() => resolve({ outcome: 'timeout' }), waitMs); });
+ return { url, ...await Promise.race([finished, timeout]) };
+ } finally {
+ clearTimeout(timer);
+ // Let the answer to the page finish: close idle connections now and any straggler after a second.
+ const straggler = setTimeout(() => server.closeAllConnections?.(), 1000);
+ await new Promise(resolve => {
+ server.close(resolve);
+ server.closeIdleConnections?.();
+ });
+ clearTimeout(straggler);
+ }
+}
diff --git a/scripts/mcp-server.mjs b/scripts/mcp-server.mjs
index 8d3fc2f3..3743b14c 100644
--- a/scripts/mcp-server.mjs
+++ b/scripts/mcp-server.mjs
@@ -789,6 +789,8 @@ const renderSettings = z.strictObject({
region: z.strictObject({ x: z.number().int().min(0), y: z.number().int().min(0), width: z.number().int().min(16), height: z.number().int().min(16) }).optional().describe('Pixels to keep from the top left. Not with renderer.'),
}).describe('Snapshot render: one PNG, eight evenly spaced PNGs, or H.264 MP4 (even dimensions). Defaults: 24 fps, one second. motion.start plays recipe tracks, animation an imported clip, both layer. Omitted motion holds time zero.');
const readOnly = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
+// Changes nothing in the scene, but opens a page for the person and waits, so a repeat call asks again.
+const askUser = { readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: false };
const edit = { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: false };
// Only these three remove something that cannot be restored: the scene, a checkpoint slot, a job's files.
const destroy = { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: false };
@@ -796,7 +798,7 @@ const destroy = { readOnlyHint: false, destructiveHint: true, idempotentHint: fa
const TOOL_TITLES = {
scene_capabilities: 'Scene capabilities', scene_create: 'Create scene', scene_asset_catalog: 'Asset catalog', scene_create_asset: 'Create scene from asset',
scene_inspect: 'Inspect scene', scene_walkmap: 'Walkable map', mesh_inspect: 'Inspect meshes', mesh_edit: 'Edit mesh', mesh_stack: 'Set mesh modifiers',
- scene_apply: 'Apply scene edits', scene_look: 'Look at scene', scene_review: 'Review camera samples', scene_checkpoint: 'Save checkpoint',
+ scene_apply: 'Apply scene edits', scene_look: 'Look at scene', scene_ask_user: 'Ask the user to mark the scene', scene_review: 'Review camera samples', scene_checkpoint: 'Save checkpoint',
scene_restore: 'Restore checkpoint', scene_release_checkpoint: 'Release checkpoint', scene_export_recipe: 'Export recipe', scene_export_glb: 'Export GLB', scene_export_web: 'Export web game',
scene_dispose: 'Dispose scene', scene_render_submit: 'Submit render job', scene_render_status: 'Render job status', scene_render_cancel: 'Cancel render job',
scene_render_result: 'Render job result', scene_render_release: 'Release render job',
@@ -925,6 +927,10 @@ export function createAgentMcpServer(runtime, { version = packageMetadata.versio
const { data, mimeType, metadata } = await runtime.look(argumentsValue);
return { content: [{ type: 'image', data, mimeType }, { type: 'text', text: JSON.stringify(metadata) }], structuredContent: metadata };
});
+ register('scene_ask_user', 'Open the view in the browser so the person pins points with notes; waits for Send. Use for a change hard to say in words. Returns marks with partId, point, normal, note, and the marked PNG.', z.strictObject({ ...sessionFields, prompt: z.string().max(200).optional().describe('Line shown above the picture.'), maxMarks: z.number().int().min(1).max(16).default(16), timeoutSeconds: z.number().int().min(10).max(900).default(300) }), askUser, async argumentsValue => {
+ const { content, image } = await runtime.askUser(argumentsValue);
+ return { content: [...(image ? [{ type: 'image', data: image, mimeType: 'image/png' }] : []), { type: 'text', text: JSON.stringify(content) }] };
+ });
register('scene_review', 'Compare one to four independent camera or motion samples in one labeled PNG grid. Preserves source revision, transient pose, viewport and checkpoints. Omitted motionTime restores authored base pose; omitted animation uses the authored asset clip pose; rendererTime defaults to zero. No render job store is required.', z.strictObject({
...sessionFields, expectedRevision,
width: z.number().int().min(64).max(480).default(384),
diff --git a/scripts/smoke-agent-post-pass.mjs b/scripts/smoke-agent-post-pass.mjs
index 2180e26a..27ae68ba 100644
--- a/scripts/smoke-agent-post-pass.mjs
+++ b/scripts/smoke-agent-post-pass.mjs
@@ -93,7 +93,7 @@ async function waitForJob(jobId) {
try {
await client.connect(transport);
- assert.equal((await client.listTools()).tools.length, 24);
+ assert.equal((await client.listTools()).tools.length, 25);
const capabilities = (await call('scene_capabilities', { topic: 'postPasses' })).structuredContent;
assert.equal(capabilities.value.limits.maxPasses, 4);
assert.ok((await call('scene_capabilities', { topic: 'operations' })).structuredContent.value.includes('set-post-passes'));
diff --git a/scripts/smoke-agent-render-jobs.mjs b/scripts/smoke-agent-render-jobs.mjs
index aee18c3e..33965408 100644
--- a/scripts/smoke-agent-render-jobs.mjs
+++ b/scripts/smoke-agent-render-jobs.mjs
@@ -57,7 +57,7 @@ async function close() {
}
try {
await connect();
- assert.equal((await client.listTools()).tools.length, 24);
+ assert.equal((await client.listTools()).tools.length, 25);
const catalog = (await call('scene_asset_catalog')).structuredContent;
const created = (await call('scene_create_asset', { recipe: {
format: '67js.asset', schemaVersion: 1, assetId: catalog.assets[0].assetId,
diff --git a/scripts/smoke-mcp-package.mjs b/scripts/smoke-mcp-package.mjs
index 480f1276..c69b015c 100644
--- a/scripts/smoke-mcp-package.mjs
+++ b/scripts/smoke-mcp-package.mjs
@@ -120,7 +120,7 @@ for (const era of ['modern', 'legacy']) {
const connectMs = performance.now() - start;
processId = transport.pid;
assert.equal(client.getServerVersion().version, metadata.version);
- assert.equal((await client.listTools()).tools.length, 24);
+ assert.equal((await client.listTools()).tools.length, 25);
const capabilities = (await call('scene_capabilities')).structuredContent;
assert.equal(capabilities.phase, 'idle');
assert.equal((await call('scene_capabilities', { topic: 'preview' })).structuredContent.value.maximumActiveScenes, 4);
diff --git a/scripts/smoke-mcp.mjs b/scripts/smoke-mcp.mjs
index ebcf1ab7..917d2a4e 100644
--- a/scripts/smoke-mcp.mjs
+++ b/scripts/smoke-mcp.mjs
@@ -53,7 +53,7 @@ try {
await client.connect(transport);
processId = transport.pid;
const listing = await client.listTools();
- assert.equal(listing.tools.length, 24);
+ assert.equal(listing.tools.length, 25);
const capabilities = (await call('scene_capabilities')).structuredContent;
assert.equal(capabilities.phase, 'idle');
const fixture = createAgentPreviewFixture();
diff --git a/src/agent-annotate.js b/src/agent-annotate.js
new file mode 100644
index 00000000..fb431e12
--- /dev/null
+++ b/src/agent-annotate.js
@@ -0,0 +1,86 @@
+// Marks a person leaves on a rendered view for scene_ask_user (issue #922): pure helpers shared by the preview page and the
+// local annotate server. A mark is a pixel and a short note; the page turns each pixel into a ray and the scene query turns
+// the ray into a part, a world point and a surface normal.
+import { mat4, vec4 } from 'gl-matrix';
+import { applyProjection } from './camera.js';
+
+export const ANNOTATE_LIMITS = Object.freeze({
+ maxMarks: 16,
+ maxNoteLength: 280,
+ maxPromptLength: 200,
+ defaultTimeoutSeconds: 300,
+ maxTimeoutSeconds: 900,
+ maxImageBytes: 1024 * 1024,
+});
+
+const PNG_SIGNATURE = [0x89, 0x50, 0x4e, 0x47];
+
+function invalidMarks(detail) {
+ return new TypeError(`The annotate marks are invalid: ${detail}`);
+}
+
+/**
+ * Checks marks sent by the annotate page. Returns clean { x, y, note } records with pixel positions inside the view.
+ * Throws a TypeError that says which mark failed, so a broken page cannot slip unbounded data to the agent.
+ */
+export function validateMarks(marks, { width, height, maxMarks = ANNOTATE_LIMITS.maxMarks }) {
+ if (!Array.isArray(marks)) throw invalidMarks('marks must be an array.');
+ if (marks.length > maxMarks) throw invalidMarks(`at most ${maxMarks} marks, got ${marks.length}.`);
+ return marks.map((mark, index) => {
+ if (!mark || typeof mark !== 'object' || Array.isArray(mark)) throw invalidMarks(`mark ${index + 1} must be an object.`);
+ const { x, y, note = '' } = mark;
+ if (!Number.isFinite(x) || x < 0 || x > width) throw invalidMarks(`mark ${index + 1} x must be a pixel from 0 to ${width}.`);
+ if (!Number.isFinite(y) || y < 0 || y > height) throw invalidMarks(`mark ${index + 1} y must be a pixel from 0 to ${height}.`);
+ if (typeof note !== 'string' || note.length > ANNOTATE_LIMITS.maxNoteLength) {
+ throw invalidMarks(`mark ${index + 1} note must be text of at most ${ANNOTATE_LIMITS.maxNoteLength} characters.`);
+ }
+ return { x: Math.round(x * 10) / 10, y: Math.round(y * 10) / 10, note: note.trim() };
+ });
+}
+
+/** True when the base64 text is a PNG within the image budget. */
+export function isBoundedPng(base64) {
+ if (typeof base64 !== 'string' || !/^[A-Za-z0-9+/]+={0,2}$/.test(base64)) return false;
+ const bytes = Buffer.from(base64, 'base64');
+ return bytes.length <= ANNOTATE_LIMITS.maxImageBytes && PNG_SIGNATURE.every((value, index) => bytes[index] === value);
+}
+
+/** The world ray through a pixel of a camera: { origin, direction } with a unit direction. */
+export function pixelRay(camera, pixelX, pixelY, width, height) {
+ const view = mat4.lookAt(mat4.create(), camera.eye, camera.target, camera.up);
+ const combined = mat4.multiply(mat4.create(), applyProjection(mat4.create(), camera, width / height), view);
+ const inverse = mat4.invert(mat4.create(), combined);
+ if (!inverse) throw new RangeError('The camera matrix cannot be inverted.');
+ const ndcX = pixelX / width * 2 - 1;
+ const ndcY = 1 - pixelY / height * 2;
+ const unproject = ndcZ => {
+ const point = vec4.transformMat4(vec4.create(), [ndcX, ndcY, ndcZ, 1], inverse);
+ return [point[0] / point[3], point[1] / point[3], point[2] / point[3]];
+ };
+ const near = unproject(-1);
+ const far = unproject(1);
+ const direction = [0, 1, 2].map(axis => far[axis] - near[axis]);
+ const length = Math.hypot(...direction);
+ return { origin: near, direction: direction.map(value => value / length) };
+}
+
+function roundedVector(vector) {
+ return Array.from(vector, value => Math.round(value * 1000) / 1000);
+}
+
+/** Numbers the marks and attaches what each one points at: part ID, world point, normal and distance, or null on empty sky. */
+export function describeMarks(marks, camera, { width, height, raycast }) {
+ return marks.map((mark, index) => {
+ const hit = raycast(pixelRay(camera, mark.x, mark.y, width, height));
+ return {
+ number: index + 1,
+ pixel: [mark.x, mark.y],
+ note: mark.note,
+ partId: hit ? hit.id : null,
+ point: hit ? roundedVector(hit.point) : null,
+ normal: hit ? roundedVector(hit.normal) : null,
+ distance: hit ? Math.round(hit.distance * 1000) / 1000 : null,
+ material: hit?.material ?? null,
+ };
+ });
+}
diff --git a/src/agent-scene-capabilities.js b/src/agent-scene-capabilities.js
index 0fdfd7c9..bc3d2758 100644
--- a/src/agent-scene-capabilities.js
+++ b/src/agent-scene-capabilities.js
@@ -202,6 +202,7 @@ export const AGENT_SCENE_CAPABILITIES = Object.freeze({
camera: 'eye, target, up, near, far, then fov (vertical degrees) or focalLength (mm, with sensorWidth mm, 36; not both). projection orthographic with orthoScale (view height in m, default 4). shift [x, y] moves the frame by a fraction of its size. fStop (0.7-64) turns on depth of field focused on target; 1.4 or lower is full blur; effects.depthOfField.blur must stay 0. Orthographic/shifted cameras use the rasterizer.',
terrain: 'One mesh, x and z from -size/2 to size/2, hills about y = 0. size 40 m (1-2000), resolution 64 cells (2-256), seed 1, amplitude 1 m (0-200), holes up to 16 of {center:[x,z],radius}, no surface inside; radius a few cells or more.',
placedAssets: 'Node field asset:{id,materials?,clip?,drive?} turns a group node into a catalog GLB (id from asset_catalog). Its transform places it. materials:[{name,color,metallic,roughness,opacity,emission,emissiveColor}] override source materials by name. clip:{index,start=0,speed=1,loop=true} plays that clip from scene time start; set-motion tracks move the node and scene time seeks the clip. drive:{path | legs, cruiseSpeed | speed, wheels, lights} drives a car along a path in the placement frame (docs/vehicle.md). Max 512 placements, 16 assets, 16 animated.',
+ askUser: 'scene_ask_user opens the current view in their browser; they pin up to maxMarks (16) points, each with a note, and press Send. Ask when a change is hard to say in words ("warmer here", "this bit"), not for facts you can inspect. Each mark returns partId, point, normal, note, distance and material (null on sky), plus view.camera and a PNG with numbered pins. Apply the change the note names to that partId, then to similar parts (same material or name prefix) when the note says so. outcome cancelled or timeout returns no marks: ask in text. Edits made while they mark change revision; the result warns.',
navigation: 'scene_inspect path:{ground,from:[x,z],to:[x,z]} returns a route over the ground node around its holes, steep slopes and the nodes in avoid:[ids] (box footprints), as keyframes [{time,value:[x,y,z]}] for a translation track with interpolation LINEAR. radius 0.3 m; maxSlope 40 degrees; cellSize defaults to radius; seconds or speed (1.4 m/s); lift adds y.',
meshEdits: Object.freeze({ operations: Object.freeze(Object.keys(MESH_EDIT_OPERATIONS)), maxEdits: MAX_MESH_EDITS }),
meshStack: Object.freeze({ operations: STACK_OPERATIONS, maxNodes: MAX_STACK_NODES }),
diff --git a/tests/agent-annotate.test.js b/tests/agent-annotate.test.js
new file mode 100644
index 00000000..bb9723c6
--- /dev/null
+++ b/tests/agent-annotate.test.js
@@ -0,0 +1,116 @@
+import test from 'node:test';
+import { request } from 'node:http';
+import assert from 'node:assert/strict';
+import { ANNOTATE_LIMITS, validateMarks, isBoundedPng, pixelRay, describeMarks } from '../src/agent-annotate.js';
+import { runAnnotateSession } from '../scripts/annotate-session.mjs';
+
+const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jRZkAAAAASUVORK5CYII=';
+const view = { width: 400, height: 200 };
+const camera = { eye: [0, 0, 10], target: [0, 0, 0], up: [0, 1, 0], fov: 60, near: 0.1, far: 100 };
+
+test('validateMarks keeps pixel positions and trims notes', () => {
+ const marks = validateMarks([{ x: 10.04, y: 20, note: ' warmer ' }, { x: 400, y: 0 }], view);
+ assert.deepEqual(marks, [{ x: 10, y: 20, note: 'warmer' }, { x: 400, y: 0, note: '' }]);
+});
+
+test('validateMarks rejects too many marks, off-view pixels, long notes and non-arrays', () => {
+ const mark = { x: 1, y: 1, note: 'a' };
+ assert.throws(() => validateMarks(Array(ANNOTATE_LIMITS.maxMarks + 1).fill(mark), view), /at most 16 marks/);
+ assert.throws(() => validateMarks([{ x: 401, y: 0 }], view), /mark 1 x/);
+ assert.throws(() => validateMarks([{ x: 0, y: -1 }], view), /mark 1 y/);
+ assert.throws(() => validateMarks([{ x: 0, y: 0, note: 'x'.repeat(281) }], view), /280 characters/);
+ assert.throws(() => validateMarks({}, view), /array/);
+ assert.throws(() => validateMarks([null], view), /object/);
+});
+
+test('isBoundedPng accepts a PNG and refuses other bytes or an oversize picture', () => {
+ assert.equal(isBoundedPng(PNG), true);
+ assert.equal(isBoundedPng(Buffer.from('not a picture').toString('base64')), false);
+ assert.equal(isBoundedPng('***'), false);
+ const oversize = Buffer.concat([Buffer.from(PNG, 'base64'), Buffer.alloc(ANNOTATE_LIMITS.maxImageBytes)]).toString('base64');
+ assert.equal(isBoundedPng(oversize), false);
+});
+
+test('pixelRay sends the centre pixel straight along the view and the corner pixels outward', () => {
+ const centre = pixelRay(camera, 200, 100, view.width, view.height);
+ assert.ok(Math.abs(centre.direction[0]) < 1e-6 && Math.abs(centre.direction[1]) < 1e-6);
+ assert.ok(centre.direction[2] < -0.999);
+ const right = pixelRay(camera, 400, 100, view.width, view.height);
+ const top = pixelRay(camera, 200, 0, view.width, view.height);
+ assert.ok(right.direction[0] > 0.3);
+ assert.ok(top.direction[1] > 0.3);
+ // A 60 degree vertical field: the top edge ray leaves at 30 degrees from the axis.
+ assert.ok(Math.abs(Math.atan(top.direction[1] / -top.direction[2]) - Math.PI / 6) < 1e-4);
+});
+
+test('describeMarks numbers marks and fills hit data, or nulls on sky', () => {
+ const rays = [];
+ const raycast = ray => {
+ rays.push(ray);
+ return rays.length === 1 ? { id: 'lamp', point: [0.12345, 1, 2], normal: [0, 1, 0], distance: 9.87654, material: 'brass' } : null;
+ };
+ const marks = describeMarks([{ x: 200, y: 100, note: 'dull' }, { x: 5, y: 5, note: '' }], camera, { ...view, raycast });
+ assert.deepEqual(marks[0], { number: 1, pixel: [200, 100], note: 'dull', partId: 'lamp', point: [0.123, 1, 2], normal: [0, 1, 0], distance: 9.877, material: 'brass' });
+ assert.deepEqual(marks[1], { number: 2, pixel: [5, 5], note: '', partId: null, point: null, normal: null, distance: null, material: null });
+});
+
+async function startSession(options, whenOpen) {
+ const opened = Promise.withResolvers();
+ const session = runAnnotateSession({ png: PNG, width: view.width, height: view.height, waitMs: 5000, ...options,
+ open: async url => { opened.resolve(url); await whenOpen?.(url); } });
+ return { url: await opened.promise, session };
+}
+
+function statusWithHost(url, host) {
+ return new Promise((resolve, reject) => {
+ const call = request(url, { headers: { Host: host } }, response => {
+ response.resume();
+ resolve(response.statusCode);
+ });
+ call.on('error', reject);
+ call.end();
+ });
+}
+
+function post(url, route, body) {
+ return fetch(url + route, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
+}
+
+test('the annotate page serves the picture and returns validated marks on Send', async () => {
+ const { url, session } = await startSession({ prompt: 'Pin <b>it</b>' });
+ const page = await (await fetch(url)).text();
+ assert.match(page, /Pin &lt;b&gt;it&lt;\/b&gt;/);
+ assert.equal((await fetch(`${url}frame.png`)).headers.get('content-type'), 'image/png');
+ assert.match(await (await fetch(`${url}app.js`)).text(), /limits/);
+ const sent = await post(url, 'send', { marks: [{ x: 10, y: 20, note: 'more red' }], image: PNG });
+ assert.equal(sent.status, 200);
+ const result = await session;
+ assert.equal(result.outcome, 'sent');
+ assert.deepEqual(result.marks, [{ x: 10, y: 20, note: 'more red' }]);
+ assert.equal(result.image, PNG);
+});
+
+test('the annotate page refuses bad marks, an empty list, a bad picture and a wrong path', async () => {
+ const { url, session } = await startSession({ maxMarks: 2 });
+ assert.equal((await post(url, 'send', { marks: [], image: PNG })).status, 400);
+ assert.equal((await post(url, 'send', { marks: [{ x: 1, y: 1 }, { x: 2, y: 2 }, { x: 3, y: 3 }], image: PNG })).status, 400);
+ assert.equal((await post(url, 'send', { marks: [{ x: 1, y: 1 }], image: 'AAAA' })).status, 400);
+ assert.equal((await fetch(url.replace(/[a-f0-9]+\/$/, 'wrong/'))).status, 404);
+ assert.equal(await statusWithHost(url, 'example.test'), 404);
+ await post(url, 'cancel', {});
+ assert.equal((await session).outcome, 'cancelled');
+});
+
+test('the annotate session returns timeout when nobody presses Send', async () => {
+ const { session } = await startSession({ waitMs: 50 });
+ const result = await session;
+ assert.equal(result.outcome, 'timeout');
+ assert.equal(result.marks, undefined);
+});
+
+test('a browser that cannot open rejects with the page address', async () => {
+ await assert.rejects(
+ runAnnotateSession({ png: PNG, width: 400, height: 200, waitMs: 50, open: async () => { throw new Error('no browser'); } }),
+ /no browser\). The annotate page was http:\/\/127\.0\.0\.1:\d+\//,
+ );
+});
diff --git a/tests/agent-preview.test.js b/tests/agent-preview.test.js
index abb4bf11..2414843d 100644
--- a/tests/agent-preview.test.js
+++ b/tests/agent-preview.test.js
@@ -197,3 +197,25 @@ test('scene_look with labels names the parts of an imported asset instead of thr
assert.equal(label.tag, null);
}
});
+
+test('annotateFrame returns the camera and pickMarks turns pixels into part hits', async () => {
+ const { preview } = previewHarness();
+ const input = recipe();
+ input.camera = { eye: [0, 0, 5], target: [0, 0, 0], up: [0, 1, 0], fov: 50 };
+ preview.create({ recipe: input, sessionId: 'marks', backend: 'cpu', memoryBudgetMiB: 64 });
+ const frame = preview.annotateFrame({ sessionId: 'marks' });
+ assert.equal(frame.width, 768);
+ assert.equal(frame.height, 512);
+ assert.deepEqual(frame.camera.eye, [0, 0, 5]);
+ const picked = await preview.pickMarks({ sessionId: 'marks', camera: frame.camera, width: frame.width, height: frame.height,
+ marks: [{ x: 384, y: 256, note: 'scuffed' }, { x: 2, y: 2, note: 'empty' }] });
+ assert.equal(picked.revision, 0);
+ const [onCube, onSky] = picked.marks;
+ assert.equal(onCube.partId, 'cube');
+ assert.equal(onCube.note, 'scuffed');
+ assert.ok(Math.abs(onCube.point[2] - 0.5) < 0.05, `front face near z 0.5, got ${onCube.point}`);
+ assert.deepEqual(onCube.normal.map(Math.round), [0, 0, 1]);
+ assert.equal(onCube.material, 'paint');
+ assert.equal(onSky.partId, null);
+ await assert.rejects(preview.pickMarks({ sessionId: 'marks', camera: frame.camera, width: 768, height: 512, marks: [{ x: 900, y: 1 }] }), /mark 1 x/);
+});
diff --git a/tests/agent-runtime.test.js b/tests/agent-runtime.test.js
index 9093810c..d1703fa3 100644
--- a/tests/agent-runtime.test.js
+++ b/tests/agent-runtime.test.js
@@ -30,6 +30,7 @@ import { filmTimelineDuration, filmTimelineShotStarts, sampleFilmTimeline } from
import { normalizeReview, REVIEW_LIMITS } from '../scripts/agent-review.mjs';
import { BRANCH_LIMITS, validateBranchName } from '../src/agent-branches.js';
import { sessionProblem, SCENE_STORE_LIMITS } from '../src/agent-errors.js';
+import { ANNOTATE_LIMITS } from '../src/agent-annotate.js';
test('runtime reports real scene capabilities without launching a browser and closes idempotently', async () => {
const runtime = createAgentRuntime({ backend: 'cpu', memory: 64 });
@@ -123,6 +124,7 @@ async function reviewRuntimeHarness({ failShot, blockShot = false, oversize = fa
VEGETATION_CAPABILITY_NOTE, CROWD_CAPABILITY_NOTE, GAME_CAPABILITIES, USER_PASS_CAPABILITIES, MESH_BLEND_CAPABILITIES, MATERIAL_SNIPPET_CAPABILITIES, AGENT_SCENE_CAPABILITIES, AGENT_SCENE_DEFAULT_LIMITS, RENDER_PASS_NAMES, RENDER_PASS_DEPTH_RANGE, VIDEO_ENCODERS, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS, MAXIMUM_MEMORY_MIB, BAKE_WORKER_MAXIMUM_SECONDS, PATH_TRACER_CAPABILITIES,
recipeAssetNeeds, gameAssetIds, sequenceFrameIds, catalogLookReferences, BUNDLED_ENVIRONMENT_NAMES, BUNDLED_CUBE_LOOK_NAMES, TEXTURE_SEQUENCE_CAPABILITIES, AGENT_ASSET_LIMITS, assertCombinedAssetBytes,
audioInputArguments, createAudioLibrary, measureStreamSeconds, filmTimelineDuration, filmTimelineShotStarts, sampleFilmTimeline, normalizeReview, REVIEW_LIMITS, isPreviewHost, BRANCH_LIMITS, validateBranchName, sessionProblem, SCENE_STORE_LIMITS,
+ runAnnotateSession: async () => ({ outcome: 'timeout' }), openInDefaultBrowser: async () => {}, ANNOTATE_LIMITS,
chromium: { launch: async () => browser }, URL, Buffer, process, performance, clearTimeout,
runtimeOptions, mkdir, rm, join, isAbsolute, fileURLToPath, writeWebBuild, EXPORT_DIRECTORY_NAME, createAgentRenderJobs: async () => ({ list: () => [], close: async () => {} }), ...contextOverrides,
setTimeout: (callback, delay) => { deadlines.push(delay); return setTimeout(callback, delay); },
diff --git a/tests/mcp-annotations.test.js b/tests/mcp-annotations.test.js
index 10e58e43..6fc3191b 100644
--- a/tests/mcp-annotations.test.js
+++ b/tests/mcp-annotations.test.js
@@ -10,7 +10,7 @@ import { Client } from '@modelcontextprotocol/client';
import { InMemoryTransport } from '@modelcontextprotocol/server';
import { createAgentMcpServer } from '../scripts/mcp-server.mjs';
-const READ_ONLY_TOOLS = ['scene_capabilities', 'scene_inspect', 'scene_look', 'scene_review', 'mesh_inspect', 'scene_walkmap',
+const READ_ONLY_TOOLS = ['scene_capabilities', 'scene_inspect', 'scene_look', 'scene_ask_user', 'scene_review', 'mesh_inspect', 'scene_walkmap',
'scene_asset_catalog', 'scene_render_status', 'scene_render_result', 'scene_export_recipe'];
const DESTRUCTIVE_TOOLS = ['scene_dispose', 'scene_render_release', 'scene_release_checkpoint'];
const TOOLS_WITH_OUTPUT_SCHEMA = ['scene_capabilities', 'scene_inspect', 'scene_checkpoint', 'scene_release_checkpoint', 'scene_render_status', 'scene_render_result'];
@@ -32,7 +32,7 @@ async function connect(context, runtime) {
test('every tool has a title and complete annotations, with the read-only and destructive sets fixed', async context => {
const client = await connect(context, { activeSessionId: () => null, async close() {} });
const { tools } = await client.listTools();
- assert.equal(tools.length, 24);
+ assert.equal(tools.length, 25);
for (const tool of tools) {
assert.ok(tool.title && tool.annotations.title === tool.title, `${tool.name} has a title`);
for (const hint of ['readOnlyHint', 'destructiveHint', 'idempotentHint']) {
diff --git a/tests/mcp-ask-user.test.js b/tests/mcp-ask-user.test.js
new file mode 100644
index 00000000..f233a08c
--- /dev/null
+++ b/tests/mcp-ask-user.test.js
@@ -0,0 +1,55 @@
+import './support/load-agent-scene-modules.mjs';
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { Client } from '@modelcontextprotocol/client';
+import { InMemoryTransport } from '@modelcontextprotocol/server';
+import { createAgentMcpServer } from '../scripts/mcp-server.mjs';
+
+const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jRZkAAAAASUVORK5CYII=';
+
+async function connect(context, runtime) {
+ const server = createAgentMcpServer(runtime);
+ const client = new Client({ name: 'ask-user-test', version: '1' });
+ const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
+ await server.connect(serverTransport);
+ await client.connect(clientTransport);
+ context.after(async () => {
+ await client.close();
+ await server.close();
+ });
+ return client;
+}
+
+test('scene_ask_user passes bounded arguments and returns the marked picture with the marks', async context => {
+ const calls = [];
+ const marks = [{ number: 1, pixel: [10, 20], note: 'dull', partId: 'lamp', point: [0, 1, 2], normal: [0, 1, 0], distance: 3, material: 'brass' }];
+ const client = await connect(context, {
+ activeSessionId: () => 'scene-1',
+ async close() {},
+ async askUser(input) {
+ calls.push(input);
+ return { image: PNG, content: { sessionId: input.sessionId, outcome: 'sent', marks } };
+ },
+ });
+ const result = await client.callTool({ name: 'scene_ask_user', arguments: { prompt: 'Pin the dull part' } });
+ assert.equal(result.isError, undefined);
+ assert.equal(calls[0].sessionId, 'scene-1');
+ assert.equal(calls[0].maxMarks, 16);
+ assert.equal(calls[0].timeoutSeconds, 300);
+ assert.equal(result.content[0].type, 'image');
+ assert.equal(JSON.parse(result.content[1].text).marks[0].partId, 'lamp');
+});
+
+test('scene_ask_user returns text only for a timeout and rejects out-of-range arguments', async context => {
+ const client = await connect(context, {
+ activeSessionId: () => 'scene-1',
+ async close() {},
+ async askUser() { return { content: { outcome: 'timeout', marks: [] } }; },
+ });
+ const timedOut = await client.callTool({ name: 'scene_ask_user', arguments: { timeoutSeconds: 10 } });
+ assert.deepEqual(timedOut.content.map(part => part.type), ['text']);
+ for (const argumentsValue of [{ maxMarks: 17 }, { maxMarks: 0 }, { timeoutSeconds: 901 }, { prompt: 'x'.repeat(201) }, { code: 'alert(1)' }, { url: 'https://example.test' }]) {
+ const rejected = await client.callTool({ name: 'scene_ask_user', arguments: argumentsValue }).catch(error => ({ isError: true, message: error.message }));
+ assert.equal(rejected.isError, true, JSON.stringify(argumentsValue));
+ }
+});
diff --git a/tests/mcp-tools.test.js b/tests/mcp-tools.test.js
index 15e80299..6f4b5c62 100644
--- a/tests/mcp-tools.test.js
+++ b/tests/mcp-tools.test.js
@@ -135,7 +135,7 @@ test('SDK advertises the complete strict JSON contract and truthful annotations'
const { client } = await connect(context);
const { tools } = await client.listTools();
assert.deepEqual(tools.map(tool => tool.name).sort(), [
- 'mesh_edit', 'mesh_inspect', 'mesh_stack', 'scene_apply', 'scene_asset_catalog', 'scene_capabilities', 'scene_checkpoint', 'scene_create', 'scene_create_asset', 'scene_dispose',
+ 'mesh_edit', 'mesh_inspect', 'mesh_stack', 'scene_apply', 'scene_ask_user', 'scene_asset_catalog', 'scene_capabilities', 'scene_checkpoint', 'scene_create', 'scene_create_asset', 'scene_dispose',
'scene_export_glb', 'scene_export_recipe', 'scene_export_web', 'scene_inspect', 'scene_look', 'scene_review', 'scene_release_checkpoint', 'scene_restore',
'scene_render_cancel', 'scene_render_release', 'scene_render_result', 'scene_render_status', 'scene_render_submit', 'scene_walkmap',
].sort());
@@ -195,7 +195,7 @@ test('review SDK exposes one image and rejects unknown or unbounded shot fields
await client.callTool({ name: 'scene_create', arguments: { recipe: recipe() } });
const { tools } = await client.listTools();
const tool = tools.find(tool => tool.name === 'scene_review');
- assert.equal(tools.length, 24);
+ assert.equal(tools.length, 25);
assert.equal(tool.annotations.readOnlyHint, true);
assert.equal(tool.inputSchema.properties.shots.maxItems, 4);
assert.equal(tool.inputSchema.properties.shots.items.additionalProperties, false);
@@ -222,7 +222,7 @@ test('branch names and labels are bounded in every tool schema and rejected befo
await client.callTool({ name: 'scene_create', arguments: { recipe: recipe() } });
const { tools } = await client.listTools();
const schema = name => tools.find(tool => tool.name === name).inputSchema;
- assert.equal(tools.length, 24, 'branches add fields, not tools');
+ assert.equal(tools.length, 25, 'branches add fields, not tools');
const pattern = '^[a-z][a-z0-9_-]{0,23}$';
assert.equal(schema('scene_apply').properties.branch.pattern, pattern);
assert.equal(schema('scene_restore').properties.branch.pattern, pattern);
diff --git a/tests/support/preview-harness.mjs b/tests/support/preview-harness.mjs
index 1102a499..bc03c137 100644
--- a/tests/support/preview-harness.mjs
+++ b/tests/support/preview-harness.mjs
@@ -18,6 +18,7 @@ import { describeApplication } from '../../src/agent-apply-report.js';
import { framingProblems, exposureProblems, lumaShares } from '../../src/frame-checks.js';
import { agentAssetDecodeOptions, withCatalogLimitAdvice } from '../../src/agent-asset-limits.js';
import { sceneQueryFor, summarizeWalkMap, walkOptionsFor, createQueryOnlyCache } from '../../scripts/agent-walkmap.js';
+import { describeMarks, validateMarks } from '../../src/agent-annotate.js';
// Runs the preview page's adapter code in a VM with a fault-injected renderer, the way the MCP preview runs it in a browser.
const source = (await readFile(new URL('../../scripts/agent-preview-browser.js', import.meta.url), 'utf8')).replace(/^import .+;\r?$/gm, '');
@@ -83,7 +84,7 @@ export function previewHarness() {
// The harness loads Manifold from node_modules directly, so the preview's served-file loader is ignored.
setManifoldModuleLoader: () => {},
needsPhysics, loadPhysicsEngine, physicsEngineLoaded, setPhysicsModuleLoader: () => {},
- sceneQueryFor, summarizeWalkMap, walkOptionsFor, createQueryOnlyCache, parseGltf, hasPatternTextures, Uint8Array, atob, sessionProblem, SCENE_STORE_LIMITS, AGENT_SCENE_DEFAULT_LIMITS, describeApplication, framingProblems, exposureProblems, lumaShares, assetCopyKey, recipeAssetNeeds, agentAssetDecodeOptions, withCatalogLimitAdvice,
+ describeMarks, validateMarks, sceneQueryFor, summarizeWalkMap, walkOptionsFor, createQueryOnlyCache, parseGltf, hasPatternTextures, Uint8Array, atob, sessionProblem, SCENE_STORE_LIMITS, AGENT_SCENE_DEFAULT_LIMITS, describeApplication, framingProblems, exposureProblems, lumaShares, assetCopyKey, recipeAssetNeeds, agentAssetDecodeOptions, withCatalogLimitAdvice,
applyProjection, applyCameraFields, mat4, resolveAgentShadow, agentEnvironmentMap, BRANCH_LIMITS,
// Line art needs a real renderer with render passes, so the fault-injected one gets a no-op install.
agentSceneModule: name => (name === 'lineArt' ? { installLineArt: renderer => renderer } : name === 'userPass' ? { ...agentSceneModule(name), installUserPasses: renderer => renderer } : agentSceneModule(name)), agentSceneModuleLoaded, createBranchStore, printableLabel,