3D engine that AI agents drive
Add the plain overshoot task, a --backend option and live overshoot runs (issue #286)
Nguyen Xuan An85ae4182c1a5
diff --git a/bench/agent-modeling/README.md b/bench/agent-modeling/README.mdindex 78b339bc..5df71fb6 100644--- a/bench/agent-modeling/README.md+++ b/bench/agent-modeling/README.md@@ -337,4 +337,17 @@ Result (measured, 3 runs per cell): branches did not save calls where the task w## An overshoot with one easing (2026-11-03, issue #286)-`overshoot-lift` asks for a 0.4 m block that rises from 0.2 m to 1.2 m in 1 s, peaks between 1.25 m and 1.45 m and settles on 1.2 m. The `overshoot` check samples the exported recipe 41 times: the start and end heights within 0.02 m, a peak in the range, and a settle after the peak. The reference uses `easeOutBack` on a `LINEAR` track: 2 keyframes. `EASING_PLAIN_SOLUTION` does the same with plain `SMOOTH` keyframes and needs 3, because the peak and the settle have to be placed by hand (its peak at t = 0.55 and 1.32 m came from trying values until the check passed, which an agent would also need a retry to do). Both pass the check and a recipe without the easing fails it (`tests/agent-modeling-suite.test.js`). Not measured: live agent runs, so the tool call counts of an agent doing the task with and without the easing are still open. Only the keyframe counts are compared here.+`overshoot-lift` asks for a 0.4 m block that rises from 0.2 m to 1.2 m in 1 s, peaks between 1.25 m and 1.45 m and settles on 1.2 m. The `overshoot` check samples the exported recipe 41 times: the start and end heights within 0.02 m, a peak in the range, and a settle after the peak. The reference uses `easeOutBack` on a `LINEAR` track: 2 keyframes. `EASING_PLAIN_SOLUTION` does the same with plain `SMOOTH` keyframes and needs 3, because the peak and the settle have to be placed by hand (its peak at t = 0.55 and 1.32 m came from trying values until the check passed, which an agent would also need a retry to do). Both pass the check and a recipe without the easing fails it (`tests/agent-modeling-suite.test.js`). Live runs are in the next section.++### Live runs of the overshoot pair (2026-11-03, issue #286)++`overshoot-lift-plain` is the same task with one added clause: use only plain keyframes and no `easing` field. The prompt of `overshoot-lift` does not mention easings, so the agent finds them or not in the tool schema. Three runs each, Sonnet through Claude Code on a subscription (no API key), CPU backend, budget $1 per run: `node bench/agent-modeling/run.mjs --claude <claude executable> --model sonnet --tasks overshoot-lift,overshoot-lift-plain --runs 3 --backend cpu --label overshoot`. The harness already drove Claude Code headless, so no new driver was needed. `--backend cpu|gpu` is new and defaults to `gpu`. Results: `results/2026-11-03-17-23-sonnet-overshoot.json`.++| Task | Pass | Tool calls (runs) | Turns | Time | Keyframes (runs) | Easing used |+| --- | --- | --- | --- | --- | --- | --- |+| overshoot-lift | 3/3 | 2, 2, 2 | 4, 4, 4 | 22 to 23 s | 6, 6, 6 | 0 of 3 |+| overshoot-lift-plain | 3/3 | 2, 2, 2 | 4, 4, 4 | 22 to 24 s | 4, 5, 5 | not allowed |++Tool calls exclude `ToolSearch`. Each run was `scene_create` with the motion inside it and one `scene_export_recipe`, with no errors and no retry.++Result (measured, 3 runs per task): the call count is the same with and without the easing, because Sonnet never used one. In the free arm all 3 runs wrote 6 plain keyframes on a `LINEAR` track with the peak at 1.35 m and passed on the first try. The 2-keyframe `easeOutBack` version in the reference was not found or not chosen. So the issue's condition is met as a comparison, and the answer is no difference in calls, 2 against 2. It does not show that the easing saves calls when an agent is told to use it, and the plain arm needed no retry, unlike the hand search that placed the reference peak. Not measured: Haiku, a prompt that names easings, or a harder curve.diff --git a/bench/agent-modeling/easing.mjs b/bench/agent-modeling/easing.mjsindex 8c3e40db..78278bf0 100644--- a/bench/agent-modeling/easing.mjs+++ b/bench/agent-modeling/easing.mjs@@ -7,17 +7,23 @@ const REST_HEIGHT = 0.2;const TARGET_HEIGHT = 1.2;const DURATION = 1;+const OVERSHOOT_PROMPT = `You are working with the 67js MCP tools. Units are metres and Y is up. Build a node named block, a box ${BLOCK_SIZE} m on every side, standing on the ground at X = 0, Z = 0 (its centre ${REST_HEIGHT} m high), with a material.+Add a ${DURATION} second motion that lifts the block straight up until its centre is ${TARGET_HEIGHT} m high, with an overshoot: on the way up it passes ${TARGET_HEIGHT} m, peaks between 1.25 m and 1.45 m, and settles back to+exactly ${TARGET_HEIGHT} m by the end of the motion, where it stays. When it is finished, call scene_export_recipe once: the exported recipe is what gets graded.`;++const OVERSHOOT_CHECKS = [+ { kind: 'lint' },+ { kind: 'overshoot', node: 'block', axis: 'y', from: REST_HEIGHT, to: TARGET_HEIGHT, duration: DURATION, peak: [1.25, 1.45] },+];+export const EASING_TASKS = [+ { id: 'overshoot-lift', prompt: OVERSHOOT_PROMPT, ownPreamble: true, checks: OVERSHOOT_CHECKS },{- id: 'overshoot-lift',- prompt: `You are working with the 67js MCP tools. Units are metres and Y is up. Build a node named block, a box ${BLOCK_SIZE} m on every side, standing on the ground at X = 0, Z = 0 (its centre ${REST_HEIGHT} m high), with a material.-Add a ${DURATION} second motion that lifts the block straight up until its centre is ${TARGET_HEIGHT} m high, with an overshoot: on the way up it passes ${TARGET_HEIGHT} m, peaks between 1.25 m and 1.45 m, and settles back to-exactly ${TARGET_HEIGHT} m by the end of the motion, where it stays. When it is finished, call scene_export_recipe once: the exported recipe is what gets graded.`,+ // The same task with the named easings ruled out, so the agent places the peak and the settle by hand.+ id: 'overshoot-lift-plain',+ prompt: OVERSHOOT_PROMPT.replace('with an overshoot:', 'with an overshoot, using only plain keyframes (do not use the easing field on any keyframe):'),ownPreamble: true,- checks: [- { kind: 'lint' },- { kind: 'overshoot', node: 'block', axis: 'y', from: REST_HEIGHT, to: TARGET_HEIGHT, duration: DURATION, peak: [1.25, 1.45] },- ],+ checks: OVERSHOOT_CHECKS,},];diff --git a/bench/agent-modeling/references.mjs b/bench/agent-modeling/references.mjsindex 8f35ee2e..71c7ae79 100644--- a/bench/agent-modeling/references.mjs+++ b/bench/agent-modeling/references.mjs@@ -13,7 +13,7 @@ import { DUSK_SOLUTION } from './dusk.mjs';import { COLONNADE_SOLUTION } from './colonnade.mjs';import { DESK_FINAL, DESK_FINAL_PLAIN } from './desk.mjs';import { CABINET_FINAL, CABINET_FINAL_PLAIN } from './cabinet.mjs';-import { EASING_SOLUTION } from './easing.mjs';+import { EASING_PLAIN_SOLUTION, EASING_SOLUTION } from './easing.mjs';const material = [{ id: 'paint', color: [0.6, 0.55, 0.5] }];@@ -390,6 +390,7 @@ export const REFERENCES = {),'rigged-walker': walkerRecipe(),'overshoot-lift': EASING_SOLUTION,+ 'overshoot-lift-plain': EASING_PLAIN_SOLUTION,'terrain-hole-walk': {format: '67js.scene', schemaVersion: 1,geometries: [diff --git a/bench/agent-modeling/results/2026-11-03-17-23-sonnet-overshoot.json b/bench/agent-modeling/results/2026-11-03-17-23-sonnet-overshoot.jsonnew file mode 100644index 00000000..22d9e493--- /dev/null+++ b/bench/agent-modeling/results/2026-11-03-17-23-sonnet-overshoot.json@@ -0,0 +1,343 @@+{+ "run": "2026-11-03-17-23-sonnet-overshoot",+ "model": "claude-sonnet-5-5",+ "date": "2026-11-03T17:26:14.898Z",+ "skill": false,+ "branches": "on",+ "relations": "on",+ "runsPerTask": 3,+ "passed": 6,+ "tasks": 6,+ "toolCalls": 12,+ "costUsd": 1.83,+ "byModel": {+ "claude-sonnet-5-5": {+ "runs": 6,+ "passed": 6,+ "successRate": 1,+ "toolCalls": 12,+ "toolErrors": 0,+ "inputTokens": 1588399,+ "outputTokens": 7632,+ "costUsd": 1.83,+ "tasks": {+ "overshoot-lift": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 6,+ "toolErrors": 0,+ "inputTokens": 795528,+ "outputTokens": 4002,+ "costUsd": 0.92+ },+ "overshoot-lift-plain": {+ "runs": 3,+ "passed": 3,+ "successRate": 1,+ "toolCalls": 6,+ "toolErrors": 0,+ "inputTokens": 792871,+ "outputTokens": 3630,+ "costUsd": 0.91+ }+ }+ }+ },+ "records": [+ {+ "task": "overshoot-lift",+ "attempt": 1,+ "pass": true,+ "passed": 2,+ "total": 2,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/1/overshoot-lift-front.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/1/overshoot-lift-side.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/1/overshoot-lift-three-quarter.png"+ ]+ },+ "turns": 4,+ "costUsd": 0.30452719999999994,+ "inputTokens": 263740,+ "outputTokens": 1409,+ "wallSeconds": 22,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "overshoot-lift-plain",+ "attempt": 1,+ "pass": true,+ "passed": 2,+ "total": 2,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/1/overshoot-lift-plain-front.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/1/overshoot-lift-plain-side.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/1/overshoot-lift-plain-three-quarter.png"+ ]+ },+ "turns": 4,+ "costUsd": 0.3000212,+ "inputTokens": 263273,+ "outputTokens": 1073,+ "wallSeconds": 22,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "overshoot-lift",+ "attempt": 2,+ "pass": true,+ "passed": 2,+ "total": 2,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/2/overshoot-lift-front.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/2/overshoot-lift-side.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/2/overshoot-lift-three-quarter.png"+ ]+ },+ "turns": 4,+ "costUsd": 0.3070874,+ "inputTokens": 265976,+ "outputTokens": 1341,+ "wallSeconds": 23,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "overshoot-lift-plain",+ "attempt": 2,+ "pass": true,+ "passed": 2,+ "total": 2,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/2/overshoot-lift-plain-front.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/2/overshoot-lift-plain-side.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/2/overshoot-lift-plain-three-quarter.png"+ ]+ },+ "turns": 4,+ "costUsd": 0.3028204,+ "inputTokens": 263672,+ "outputTokens": 1264,+ "wallSeconds": 22,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "overshoot-lift",+ "attempt": 3,+ "pass": true,+ "passed": 2,+ "total": 2,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/3/overshoot-lift-front.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/3/overshoot-lift-side.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/3/overshoot-lift-three-quarter.png"+ ]+ },+ "turns": 4,+ "costUsd": 0.30587960000000003,+ "inputTokens": 265812,+ "outputTokens": 1252,+ "wallSeconds": 22,+ "exitCode": 0,+ "stopped": "success"+ },+ {+ "task": "overshoot-lift-plain",+ "attempt": 3,+ "pass": true,+ "passed": 2,+ "total": 2,+ "failed": [],+ "model": "claude-sonnet-5-5",+ "toolCalls": 2,+ "toolErrors": 0,+ "toolsByName": {+ "ToolSearch": 1,+ "scene_create": 1,+ "scene_export_recipe": 1+ },+ "branchCalls": 0,+ "relationCalls": 0,+ "pathQueries": 0,+ "features": {+ "geometryTypes": [+ "box"+ ],+ "meshOperations": [],+ "motionIk": [],+ "motionTracks": 1+ },+ "look": {+ "silhouette": {+ "views": {+ "front": 1,+ "side": 1,+ "three-quarter": 1+ },+ "mean": 1+ },+ "images": [+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/3/overshoot-lift-plain-front.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/3/overshoot-lift-plain-side.png",+ "artifacts/agent-modeling/2026-11-03-17-23-sonnet-overshoot/look/3/overshoot-lift-plain-three-quarter.png"+ ]+ },+ "turns": 4,+ "costUsd": 0.3063998,+ "inputTokens": 265926,+ "outputTokens": 1293,+ "wallSeconds": 24,+ "exitCode": 0,+ "stopped": "success"+ }+ ]+}diff --git a/bench/agent-modeling/run.mjs b/bench/agent-modeling/run.mjsindex 76bae138..3b4b9c1d 100644--- a/bench/agent-modeling/run.mjs+++ b/bench/agent-modeling/run.mjs@@ -2,7 +2,7 @@// server, then grades the recipe the agent exported.//// node bench/agent-modeling/run.mjs --claude <claude executable> [--model sonnet] [--tasks crate,bolt]-// [--runs 3] [--budget 3] [--timeout 1200] [--label baseline] [--skill] [--branches off] [--relations off] [--resume <run name>]+// [--runs 3] [--budget 3] [--timeout 1200] [--label baseline] [--skill] [--branches off] [--relations off] [--backend gpu|cpu] [--resume <run name>]//// --runs repeats every task that many times, each in a fresh session; the summary reports success rate, tool calls,// tool errors, tokens and cost for the model and for each task.@@ -44,6 +44,7 @@ const { values: options } = parseArgs({branches: { type: 'string', default: 'on' },relations: { type: 'string', default: 'on' },resume: { type: 'string' },+ backend: { type: 'string', default: 'gpu' },},});@@ -61,7 +62,7 @@ async function skillPrompt() {function mcpConfig(assetDirectory) {return { mcpServers: { [MCP_SERVER_NAME]: {command: process.execPath,- args: [join(repository, 'scripts', 'mcp-server.mjs'), '--backend', 'gpu', '--memory', '192', '--branches', options.branches, '--relations', options.relations, ...(assetDirectory ? ['--assets', assetDirectory] : [])],+ args: [join(repository, 'scripts', 'mcp-server.mjs'), '--backend', options.backend, '--memory', '192', '--branches', options.branches, '--relations', options.relations, ...(assetDirectory ? ['--assets', assetDirectory] : [])],} } };}diff --git a/changes/286-suite.md b/changes/286-suite.mdnew file mode 100644index 00000000..06f1f898--- /dev/null+++ b/changes/286-suite.md@@ -0,0 +1 @@+- Add the `overshoot-lift-plain` task and a `--backend cpu|gpu` option to the agent modeling suite (issue #286). Three live Sonnet runs of each overshoot task passed with 2 tool calls each, because the agent never chose a named easing.diff --git a/tests/agent-modeling-suite.test.js b/tests/agent-modeling-suite.test.jsindex 7b34af6d..6de781a3 100644--- a/tests/agent-modeling-suite.test.js+++ b/tests/agent-modeling-suite.test.js@@ -292,3 +292,9 @@ test('the overshoot lift passes with one named easing or with hand-placed keyfradelete noOvershoot.motion.tracks[0].keyframes[0].easing;assert.deepEqual(failedKinds(await gradeRecipe(task, noOvershoot)), ['overshoot']);});++test('the plain overshoot task forbids easings in its prompt and grades the hand-placed keyframes', async () => {+ const task = taskById('overshoot-lift-plain');+ assert.match(task.prompt, /do not use the easing field/);+ assert.deepEqual(failedKinds(await gradeRecipe(task, structuredClone(REFERENCES['overshoot-lift-plain']))), []);+});