Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(editor): a running game can be handed a click

yhha18349856570
diff --git a/desktop/electron/agent/prompt.ts b/desktop/electron/agent/prompt.ts
index f684b1ee6..a6425cc08 100644
--- a/desktop/electron/agent/prompt.ts
+++ b/desktop/electron/agent/prompt.ts
@@ -34,6 +34,7 @@ Working style:
- Prefer one apply_scene_ops over many small calls when building something: it is one undo step for the user and one round trip for you, and it is the ONE write the user sees BEFORE it lands — they get a line-by-line preview and can strike parts of it out. A dozen set_field calls give them no such chance, so build with a batch and save the single writes for genuinely single changes.
- Units are the editor's: 1 world unit = 1 design pixel. Y is up in the world, and UI is laid out top-down.
- Game content lives in the WORLD: a board, its pieces, a character are Sprite / ShapeRenderer entities you place with Transform.position. A Canvas and its UINodes are for HUD and menus, and their placement is the layout's OUTPUT — a Transform.position written on a UI node is overwritten at the next relayout, which looks like nothing happened. Move those with layout inputs instead (UINode.position = Absolute, then left/top), or let the flow place them.
+- Behaviour is a system and data is a component, both declared through the front door: \`defineSystem([Query(Mut(Transform)), Res(Time)], (q, t) => {...})\` and \`defineComponent('Name', { speed: 100 })\`, registered with \`addSystemToSchedule\`. create_script writes either one already wired into the entry that loads it. The underscore-prefixed fields you can see on those objects (\`_params\`, \`_fn\`, \`_id\`) are the scheduler's, not an authoring surface: assembling one by hand happens to run and leaves a project nobody else's tools understand.
- A game reads input from the ENGINE, never from the DOM. In a system, ask for \`Res(Input)\` and \`Res(CameraView)\`: \`input.isMouseButtonPressed(MouseButton.Left)\`, \`input.mouseX/mouseY\` (SCREEN pixels, canvas-relative, y down), and \`camera.getWorldMousePosition()\` / \`camera.screenToWorld(x, y)\` for where that is in the world. \`document.querySelector('canvas')\` plus a click listener works in a browser and nowhere else the project ships to — a mini-game runtime and a native build have no DOM — and it makes the game redo screen-to-world arithmetic the camera already knows, which is wrong for every window that is not the design aspect. Read the shape of an API before calling it (read_project_file takes \`offset\` and \`limit\` to page a big .d.ts) rather than guessing method names into a try/catch that hides the mistake.
- Say what you did in a sentence or two. The user watched it happen — a summary of every call is noise. Speak the user's language: answer in the language they wrote to you in.`;
diff --git a/desktop/electron/mcpEndpoint.ts b/desktop/electron/mcpEndpoint.ts
index 74f860da2..4ddbc49e9 100644
--- a/desktop/electron/mcpEndpoint.ts
+++ b/desktop/electron/mcpEndpoint.ts
@@ -103,11 +103,16 @@ export async function startMcpEndpoint(
try {
server = await createExecEndpoint({
token: TOKEN,
- run: async ({ method, args, root, js, op, code, frame }: {
+ // An op's payload is forwarded WHOLE. Naming its fields here (it used to
+ // pass `{ code, frame }`, the two play_probe wanted) means every field a
+ // later op adds is dropped in transit — the tool validates it, the
+ // catalog documents it, and the op sees undefined. That is the same
+ // silent-argument-loss that made `offset` on read_project_file a lie.
+ run: async (payload: {
method?: string; args?: unknown[]; root?: string; js?: string; op?: string;
- code?: string; frame?: number;
- }) => {
- if (op) return driver.op(op, { code, frame });
+ } & Record<string, unknown>) => {
+ const { method, args, root, js, op, ...rest } = payload;
+ if (op) return driver.op(op, rest);
if (js) return driver.js(js);
return driver(method as string, args ?? [], root);
},
diff --git a/desktop/electron/surfaceDriver.ts b/desktop/electron/surfaceDriver.ts
index 63615dfc0..2af52c55b 100644
--- a/desktop/electron/surfaceDriver.ts
+++ b/desktop/electron/surfaceDriver.ts
@@ -1,126 +1,163 @@
-// SPDX-License-Identifier: Apache-2.0
-// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
-/**
- * @file surfaceDriver.ts — main-side calls INTO the live editor window.
- *
- * The tool catalog (shared/toolCatalog.mjs) dispatches through an injected
- * `driver`, and this is the one that targets the real editor app: surface
- * calls resolve on `window.__estellaEditor.surface` (the same
- * EditorControlSurface the UI uses), `root: 'editor'` calls on
- * `window.__estellaEditor` (project / asset / play doors), renderer-code
- * snippets run verbatim, and the two main-process ops reach what the renderer
- * cannot — the composited window, and the play realm's out-of-process frame.
- *
- * Deliberately transport-free. The loopback /exec endpoint wraps this
- * (mcpEndpoint.ts owns the token, discovery file and ready line), and anything
- * else in main that wants to drive the editor calls the same object directly —
- * so an in-process caller and a remote MCP client cannot drift apart.
- */
-import type { BrowserWindow } from 'electron';
-
-/** What the tool registry calls: a surface/editor method, plus the two escape
- * hatches it also knows about (`js` renderer code, `op` main-process routines). */
-export interface SurfaceDriver {
- (method: string, args?: readonly unknown[], root?: string): Promise<unknown>;
- js(code: string): Promise<unknown>;
- op(op: string, input?: { code?: string; frame?: number }): Promise<unknown>;
-}
-
-/**
- * executeJavaScript rejects a thrown error with an opaque "Script failed to
- * execute". Wrap the expression so the real message crosses the boundary — a
- * surface error like `component "X" is not on entity 5` IS the caller's
- * feedback, not a debugging detail.
- */
-/**
- * Wrap a snippet so its value comes back and its throw becomes ours.
- *
- * A probe body is a PROGRAM, not necessarily an expression — `a(); b(); "done"`
- * is what anyone writes, and inside parentheses that is a syntax error. Which
- * form to use is decided HERE, in the main process, by trying to parse it: the
- * realm has a CSP and must not be asked to compile anything. An expression keeps
- * its implicit value; a body returns what it returns.
- */
-const carryError = (code: string): string => {
- let isExpression = true;
- try {
- new Function(`return (${code})`);
- } catch {
- isExpression = false;
- }
- const value = isExpression ? `await (${code})` : `await (async () => { ${code} })()`;
- return `(async () => { try { return { v: ${value} }; } catch (e) { return { __estellaExecError: (e && e.message) || String(e) }; } })()`;
-};
-
-function unwrap(res: unknown): unknown {
- const r = res as { v?: unknown; __estellaExecError?: string } | null;
- if (r && typeof r === 'object' && r.__estellaExecError !== undefined) throw new Error(r.__estellaExecError);
- return r?.v;
-}
-
-/** JSON-encode a call's arguments, keeping `undefined` as the literal so the
- * target method's own default parameters apply. */
-const encodeArgs = (args: readonly unknown[]): string =>
- args.map((a) => (a === undefined ? 'undefined' : JSON.stringify(a))).join(',');
-
-/**
- * A driver bound to the editor window. `getWin` re-resolves per call, so a
- * recreated window keeps working and a driver built before the window exists
- * is still valid.
- */
-export function createSurfaceDriver(getWin: () => BrowserWindow | null): SurfaceDriver {
- const requireWin = (): BrowserWindow => {
- const win = getWin();
- if (!win) throw new Error('no editor window');
- return win;
- };
-
- const exec = async (code: string): Promise<unknown> => {
- const win = requireWin();
- // A call can arrive before the renderer finished booting (the endpoint comes
- // up with the window) — wait out the initial load instead of failing it.
- // Polled, never event-awaited: isLoading() covers SUBFRAMES (the play realm
- // prewarms in an iframe seconds after a project opens), but did-finish-load
- // fires only for the main frame — a call that awaited the event during a
- // subframe load parked forever, timing out whichever automation call drew
- // the short straw. Bounded: a page that never settles still gets the call,
- // and executeJavaScript's own error is a better answer than silence.
- for (let waited = 0; win.webContents.isLoading() && waited < 10_000; waited += 50) {
- await new Promise((r) => setTimeout(r, 50));
- }
- return unwrap(await win.webContents.executeJavaScript(carryError(code), true));
- };
-
- const driver = ((method: string, args: readonly unknown[] = [], root?: string) => {
- const target = root === 'editor' ? 'window.__estellaEditor' : 'window.__estellaEditor.surface';
- return exec(`${target}.${method}(${encodeArgs(args)})`);
- }) as SurfaceDriver;
-
- driver.js = exec;
-
- driver.op = async (op, input = {}) => {
- switch (op) {
- case 'screenshot': {
- // The composited window, not the viewport canvas: this is the only way to
- // see the play realm, which renders in its own frame.
- const image = await requireWin().webContents.capturePage();
- return image.toPNG().toString('base64');
- }
- case 'play_probe': {
- // The play realm is an estella:// OOPIF — only a main-process frame eval
- // reaches it (window.__estellaPlay is the probe the realm publishes).
- const frames = requireWin().webContents.mainFrame.frames.filter((f) => f.url.startsWith('estella://'));
- const at = input.frame ?? 0;
- const frame = frames[at];
- if (!frame) {
- throw new Error(`no play realm at index ${at} (${frames.length} running — enter play first)`);
- }
- return unwrap(await frame.executeJavaScript(carryError(input.code ?? 'true')));
- }
- default:
- throw new Error(`unknown main-process op: ${op}`);
- }
- };
-
- return driver;
-}
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file surfaceDriver.ts — main-side calls INTO the live editor window.
+ *
+ * The tool catalog (shared/toolCatalog.mjs) dispatches through an injected
+ * `driver`, and this is the one that targets the real editor app: surface
+ * calls resolve on `window.__estellaEditor.surface` (the same
+ * EditorControlSurface the UI uses), `root: 'editor'` calls on
+ * `window.__estellaEditor` (project / asset / play doors), renderer-code
+ * snippets run verbatim, and the two main-process ops reach what the renderer
+ * cannot — the composited window, and the play realm's out-of-process frame.
+ *
+ * Deliberately transport-free. The loopback /exec endpoint wraps this
+ * (mcpEndpoint.ts owns the token, discovery file and ready line), and anything
+ * else in main that wants to drive the editor calls the same object directly —
+ * so an in-process caller and a remote MCP client cannot drift apart.
+ */
+import type { BrowserWindow } from 'electron';
+
+/** What the tool registry calls: a surface/editor method, plus the two escape
+ * hatches it also knows about (`js` renderer code, `op` main-process routines). */
+export interface SurfaceDriver {
+ (method: string, args?: readonly unknown[], root?: string): Promise<unknown>;
+ js(code: string): Promise<unknown>;
+ op(op: string, input?: Record<string, unknown>): Promise<unknown>;
+}
+
+/**
+ * executeJavaScript rejects a thrown error with an opaque "Script failed to
+ * execute". Wrap the expression so the real message crosses the boundary — a
+ * surface error like `component "X" is not on entity 5` IS the caller's
+ * feedback, not a debugging detail.
+ */
+/**
+ * Wrap a snippet so its value comes back and its throw becomes ours.
+ *
+ * A probe body is a PROGRAM, not necessarily an expression — `a(); b(); "done"`
+ * is what anyone writes, and inside parentheses that is a syntax error. Which
+ * form to use is decided HERE, in the main process, by trying to parse it: the
+ * realm has a CSP and must not be asked to compile anything. An expression keeps
+ * its implicit value; a body returns what it returns.
+ */
+const carryError = (code: string): string => {
+ let isExpression = true;
+ try {
+ new Function(`return (${code})`);
+ } catch {
+ isExpression = false;
+ }
+ const value = isExpression ? `await (${code})` : `await (async () => { ${code} })()`;
+ return `(async () => { try { return { v: ${value} }; } catch (e) { return { __estellaExecError: (e && e.message) || String(e) }; } })()`;
+};
+
+function unwrap(res: unknown): unknown {
+ const r = res as { v?: unknown; __estellaExecError?: string } | null;
+ if (r && typeof r === 'object' && r.__estellaExecError !== undefined) throw new Error(r.__estellaExecError);
+ return r?.v;
+}
+
+/** JSON-encode a call's arguments, keeping `undefined` as the literal so the
+ * target method's own default parameters apply. */
+const encodeArgs = (args: readonly unknown[]): string =>
+ args.map((a) => (a === undefined ? 'undefined' : JSON.stringify(a))).join(',');
+
+/**
+ * A driver bound to the editor window. `getWin` re-resolves per call, so a
+ * recreated window keeps working and a driver built before the window exists
+ * is still valid.
+ */
+export function createSurfaceDriver(getWin: () => BrowserWindow | null): SurfaceDriver {
+ const requireWin = (): BrowserWindow => {
+ const win = getWin();
+ if (!win) throw new Error('no editor window');
+ return win;
+ };
+
+ const exec = async (code: string): Promise<unknown> => {
+ const win = requireWin();
+ // A call can arrive before the renderer finished booting (the endpoint comes
+ // up with the window) — wait out the initial load instead of failing it.
+ // Polled, never event-awaited: isLoading() covers SUBFRAMES (the play realm
+ // prewarms in an iframe seconds after a project opens), but did-finish-load
+ // fires only for the main frame — a call that awaited the event during a
+ // subframe load parked forever, timing out whichever automation call drew
+ // the short straw. Bounded: a page that never settles still gets the call,
+ // and executeJavaScript's own error is a better answer than silence.
+ for (let waited = 0; win.webContents.isLoading() && waited < 10_000; waited += 50) {
+ await new Promise((r) => setTimeout(r, 50));
+ }
+ return unwrap(await win.webContents.executeJavaScript(carryError(code), true));
+ };
+
+ const driver = ((method: string, args: readonly unknown[] = [], root?: string) => {
+ const target = root === 'editor' ? 'window.__estellaEditor' : 'window.__estellaEditor.surface';
+ return exec(`${target}.${method}(${encodeArgs(args)})`);
+ }) as SurfaceDriver;
+
+ driver.js = exec;
+
+ driver.op = async (op, input = {}) => {
+ switch (op) {
+ case 'screenshot': {
+ // The composited window, not the viewport canvas: this is the only way to
+ // see the play realm, which renders in its own frame.
+ const image = await requireWin().webContents.capturePage();
+ return image.toPNG().toString('base64');
+ }
+ case 'play_probe': {
+ // The play realm is an estella:// OOPIF — only a main-process frame eval
+ // reaches it (window.__estellaPlay is the probe the realm publishes).
+ const frames = requireWin().webContents.mainFrame.frames.filter((f) => f.url.startsWith('estella://'));
+ const at = Number(input.frame ?? 0);
+ const frame = frames[at];
+ if (!frame) {
+ throw new Error(`no play realm at index ${at} (${frames.length} running — enter play first)`);
+ }
+ return unwrap(await frame.executeJavaScript(carryError(String(input.code ?? 'true'))));
+ }
+ case 'play_input': {
+ // Same frame lookup as play_probe — the realm is the only thing that has
+ // an InputState — but the code is OURS, so a caller cannot mistype the
+ // facade and get silence. Everything routes through the realm's
+ // `__estellaPlay.input`, which is the platform binding's own callbacks.
+ const frames = requireWin().webContents.mainFrame.frames.filter((f) => f.url.startsWith('estella://'));
+ const at = Number(input.frame ?? 0);
+ const frame = frames[at];
+ if (!frame) {
+ throw new Error(`no play realm at index ${at} (${frames.length} running — enter play first)`);
+ }
+ const { kind } = input as { kind?: string };
+ const x = Number(input.x ?? 0), y = Number(input.y ?? 0);
+ const button = Number(input.button ?? 0);
+ const code = JSON.stringify(String(input.code ?? ''));
+ const id = 1;
+ const call = {
+ click: `i.down(${x},${y},${button}); i.up(${button});`,
+ move: `i.move(${x},${y});`,
+ down: `i.down(${x},${y},${button});`,
+ up: `i.up(${button});`,
+ wheel: `i.wheel(${x},${y});`,
+ key_down: `i.keyDown(${code});`,
+ key_up: `i.keyUp(${code});`,
+ tap: `i.touchStart(${id},${x},${y}); i.touchEnd(${id});`,
+ }[kind ?? ''];
+ if (!call) {
+ throw new Error(
+ `unknown play_input kind "${kind}" — one of click, move, down, up, wheel, key_down, key_up, tap`,
+ );
+ }
+ return unwrap(await frame.executeJavaScript(carryError(
+ `(() => { const p = window.__estellaPlay; if (!p || !p.input) `
+ + `throw new Error('this play realm publishes no input door — it predates play_input'); `
+ + `const i = p.input; ${call} return 'ok'; })()`,
+ )));
+ }
+ default:
+ throw new Error(`unknown main-process op: ${op}`);
+ }
+ };
+
+ return driver;
+}
diff --git a/desktop/shared/toolCatalog.mjs b/desktop/shared/toolCatalog.mjs
index 4d41e360c..1b296956d 100644
--- a/desktop/shared/toolCatalog.mjs
+++ b/desktop/shared/toolCatalog.mjs
@@ -495,6 +495,19 @@ export const TOOLS = [
description: "Evaluate JS inside the RUNNING play realm and return the result — the gameplay probe. One expression gives its value; several statements need an explicit `return`. window.__estellaPlay = { app, getComponent }: `getComponent(NAME)` gives the component DEFINITION, `app.world.get(entity, def)` a COPY of its data, and a write only lands through `app.world.insert(entity, def, data)` — assigning to the copy changes nothing; to drive gameplay input, dispatch KeyboardEvents on DOCUMENT (the engine listens there, not on window): document.dispatchEvent(new KeyboardEvent('keydown', {code:'ArrowRight'})). frame picks the realm in multiplayer previews (0 = host).",
schema: obj({ code: { type: 'string' }, frame: { type: 'number' } }, ['code']),
op: 'play_probe' },
+ { name: 'play_input', effect: 'irreversible',
+ description: 'Deliver a pointer or key event to the RUNNING game — the only way to exercise the input a game actually ships with. '
+ + "`kind`: 'click' (down+up at x,y), 'move', 'down', 'up', 'wheel', 'key_down', 'key_up', 'tap' (touch down+up). "
+ + 'x/y are SCREEN pixels, canvas-relative, y DOWN — the same numbers a real pointer event carries and what `Input.mouseX/mouseY` then read; '
+ + 'ask the camera (CameraView.screenToWorld / UICameraInfo.worldMouseX) for where that is in the world rather than converting by hand. '
+ + '`code` is a KeyboardEvent code for the key kinds. It goes through the platform binding\'s own callbacks, so UI gets first refusal exactly as it would for a real event. '
+ + 'A pressed EDGE (isMouseButtonPressed / isKeyPressed) lasts one frame: call step() right after, then read the result. '
+ + 'Calling the game\'s own handler instead proves only that the handler works — not that a click reaches it.',
+ schema: obj({
+ kind: { type: 'string' }, x: { type: 'number' }, y: { type: 'number' },
+ button: { type: 'number' }, code: { type: 'string' }, frame: { type: 'number' },
+ }, ['kind']),
+ op: 'play_input' },
{ name: 'world_component',
description: "A LIVE World component's data for a source entity, resolved by component name — verifies an edit actually reached the engine.",
schema: obj({ id: { type: 'number' }, component: { type: 'string' } }, ['id', 'component']),
diff --git a/desktop/src/playHost.ts b/desktop/src/playHost.ts
index ecbd213f9..bf5b75ae3 100644
--- a/desktop/src/playHost.ts
+++ b/desktop/src/playHost.ts
@@ -14,7 +14,7 @@
* Everything is same-origin estella:// (host, sdk, bundle, wasm, assets),
* sidestepping the custom-scheme cross-fetch ban.
*/
-import { createWebApp, setEditorMode, setPlayMode, initPlayRealmRuntime, getComponent, clearUserComponents, getUserComponentFingerprint, probeRegistrations, Net, MessagePortTransport, Assets, Ads, createMockAdProvider, Leaderboard, createLocalLeaderboard, registerPackagedSideModules } from 'esengine';
+import { createWebApp, setEditorMode, setPlayMode, initPlayRealmRuntime, getComponent, clearUserComponents, getUserComponentFingerprint, probeRegistrations, Net, MessagePortTransport, Assets, Ads, createMockAdProvider, Leaderboard, createLocalLeaderboard, registerPackagedSideModules, Input, inputEventCallbacks } from 'esengine';
import type { App, ESEngineModule, SceneData } from 'esengine';
import { PLAY_PROTOCOL_VERSION } from './engine/playProtocol';
import type { PlayOutbound, PlayInbound } from './engine/playProtocol';
@@ -277,7 +277,31 @@ async function buildAppAndRun(msg: InitMessage): Promise<void> {
// host's __estellaHeadless): the editor can't reach into this OOPIF except by
// main-process frame eval, and the eval needs SOMETHING to query. getComponent
// lets a probe read component data by name (world.get needs the def).
- (window as unknown as { __estellaPlay?: unknown }).__estellaPlay = { app, getComponent };
+ // `input` delivers a pointer/key event through the SAME callbacks the platform
+ // binding hands raw events to, so an injected click is routed past the UI and
+ // recorded exactly like a real one. Without it a driver can build input
+ // handling and never test it: the realm is an out-of-process frame that
+ // sendInputEvent does not reach, a synthetic MouseEvent carries no usable
+ // offsetX, and poking mouseX / mouseButtonsPressed by hand skips the router.
+ // What that costs is not hypothetical — a chess game was "verified" by calling
+ // its own click handler directly, which is how a screen-to-world bug reached
+ // the player: the one path nobody could exercise was the one that was wrong.
+ const injected = inputEventCallbacks(app.getResource(Input));
+ (window as unknown as { __estellaPlay?: unknown }).__estellaPlay = {
+ app,
+ getComponent,
+ input: {
+ move: (x: number, y: number) => injected.onPointerMove?.(x, y),
+ down: (x: number, y: number, button = 0) => injected.onPointerDown?.(button, x, y),
+ up: (button = 0) => injected.onPointerUp?.(button),
+ wheel: (dx: number, dy: number) => injected.onWheel?.(dx, dy),
+ keyDown: (code: string) => injected.onKeyDown?.(code),
+ keyUp: (code: string) => injected.onKeyUp?.(code),
+ touchStart: (id: number, x: number, y: number) => injected.onTouchStart?.(id, x, y),
+ touchMove: (id: number, x: number, y: number) => injected.onTouchMove?.(id, x, y),
+ touchEnd: (id: number) => injected.onTouchEnd?.(id),
+ },
+ };
startFrameHeartbeat();
// Surface a failed handshake as a boot error rather than a silent
diff --git a/desktop/tests/agent-prompt.test.ts b/desktop/tests/agent-prompt.test.ts
index 38232feb1..4164053a5 100644
--- a/desktop/tests/agent-prompt.test.ts
+++ b/desktop/tests/agent-prompt.test.ts
@@ -37,6 +37,16 @@ describe('the agent is told where game input comes from', () => {
expect(SYSTEM_PROMPT).toMatch(/offset.*limit|limit.*offset/);
});
+ it('names the two front doors, and that the underscore fields are not one', () => {
+ // A hand-assembled SystemDef happens to RUN, so nothing complains — it just
+ // leaves a project whose systems no tool of ours recognises.
+ expect(SYSTEM_PROMPT).toContain('defineSystem');
+ expect(SYSTEM_PROMPT).toContain('defineComponent');
+ expect(SYSTEM_PROMPT).toContain('addSystemToSchedule');
+ expect(SYSTEM_PROMPT).toMatch(/_params.*_fn|_fn.*_params/);
+ expect(SYSTEM_PROMPT).toMatch(/not an authoring surface/);
+ });
+
it('stays one text — it sits in the cached prefix and must not vary per turn', () => {
expect(typeof SYSTEM_PROMPT).toBe('string');
expect(SYSTEM_PROMPT).toBe(SYSTEM_PROMPT);