A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(editor): a running game can be handed a click
yhha18349856570
diff --git a/desktop/electron/agent/prompt.ts b/desktop/electron/agent/prompt.tsindex f684b1ee6..a6425cc08 100644--- a/desktop/electron/agent/prompt.ts+++ b/desktop/electron/agent/prompt.ts@@ -34,6 +34,7 @@ Working style:- Prefer one apply_scene_ops over many small calls when building something: it is one undo step for the user and one round trip for you, and it is the ONE write the user sees BEFORE it lands — they get a line-by-line preview and can strike parts of it out. A dozen set_field calls give them no such chance, so build with a batch and save the single writes for genuinely single changes.- Units are the editor's: 1 world unit = 1 design pixel. Y is up in the world, and UI is laid out top-down.- Game content lives in the WORLD: a board, its pieces, a character are Sprite / ShapeRenderer entities you place with Transform.position. A Canvas and its UINodes are for HUD and menus, and their placement is the layout's OUTPUT — a Transform.position written on a UI node is overwritten at the next relayout, which looks like nothing happened. Move those with layout inputs instead (UINode.position = Absolute, then left/top), or let the flow place them.+- Behaviour is a system and data is a component, both declared through the front door: \`defineSystem([Query(Mut(Transform)), Res(Time)], (q, t) => {...})\` and \`defineComponent('Name', { speed: 100 })\`, registered with \`addSystemToSchedule\`. create_script writes either one already wired into the entry that loads it. The underscore-prefixed fields you can see on those objects (\`_params\`, \`_fn\`, \`_id\`) are the scheduler's, not an authoring surface: assembling one by hand happens to run and leaves a project nobody else's tools understand.- A game reads input from the ENGINE, never from the DOM. In a system, ask for \`Res(Input)\` and \`Res(CameraView)\`: \`input.isMouseButtonPressed(MouseButton.Left)\`, \`input.mouseX/mouseY\` (SCREEN pixels, canvas-relative, y down), and \`camera.getWorldMousePosition()\` / \`camera.screenToWorld(x, y)\` for where that is in the world. \`document.querySelector('canvas')\` plus a click listener works in a browser and nowhere else the project ships to — a mini-game runtime and a native build have no DOM — and it makes the game redo screen-to-world arithmetic the camera already knows, which is wrong for every window that is not the design aspect. Read the shape of an API before calling it (read_project_file takes \`offset\` and \`limit\` to page a big .d.ts) rather than guessing method names into a try/catch that hides the mistake.- Say what you did in a sentence or two. The user watched it happen — a summary of every call is noise. Speak the user's language: answer in the language they wrote to you in.`;diff --git a/desktop/electron/mcpEndpoint.ts b/desktop/electron/mcpEndpoint.tsindex 74f860da2..4ddbc49e9 100644--- a/desktop/electron/mcpEndpoint.ts+++ b/desktop/electron/mcpEndpoint.ts@@ -103,11 +103,16 @@ export async function startMcpEndpoint(try {server = await createExecEndpoint({token: TOKEN,- run: async ({ method, args, root, js, op, code, frame }: {+ // An op's payload is forwarded WHOLE. Naming its fields here (it used to+ // pass `{ code, frame }`, the two play_probe wanted) means every field a+ // later op adds is dropped in transit — the tool validates it, the+ // catalog documents it, and the op sees undefined. That is the same+ // silent-argument-loss that made `offset` on read_project_file a lie.+ run: async (payload: {method?: string; args?: unknown[]; root?: string; js?: string; op?: string;- code?: string; frame?: number;- }) => {- if (op) return driver.op(op, { code, frame });+ } & Record<string, unknown>) => {+ const { method, args, root, js, op, ...rest } = payload;+ if (op) return driver.op(op, rest);if (js) return driver.js(js);return driver(method as string, args ?? [], root);},diff --git a/desktop/electron/surfaceDriver.ts b/desktop/electron/surfaceDriver.tsindex 63615dfc0..2af52c55b 100644--- a/desktop/electron/surfaceDriver.ts+++ b/desktop/electron/surfaceDriver.ts@@ -1,126 +1,163 @@-// SPDX-License-Identifier: Apache-2.0-// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team-/**- * @file surfaceDriver.ts — main-side calls INTO the live editor window.- *- * The tool catalog (shared/toolCatalog.mjs) dispatches through an injected- * `driver`, and this is the one that targets the real editor app: surface- * calls resolve on `window.__estellaEditor.surface` (the same- * EditorControlSurface the UI uses), `root: 'editor'` calls on- * `window.__estellaEditor` (project / asset / play doors), renderer-code- * snippets run verbatim, and the two main-process ops reach what the renderer- * cannot — the composited window, and the play realm's out-of-process frame.- *- * Deliberately transport-free. The loopback /exec endpoint wraps this- * (mcpEndpoint.ts owns the token, discovery file and ready line), and anything- * else in main that wants to drive the editor calls the same object directly —- * so an in-process caller and a remote MCP client cannot drift apart.- */-import type { BrowserWindow } from 'electron';--/** What the tool registry calls: a surface/editor method, plus the two escape- * hatches it also knows about (`js` renderer code, `op` main-process routines). */-export interface SurfaceDriver {- (method: string, args?: readonly unknown[], root?: string): Promise<unknown>;- js(code: string): Promise<unknown>;- op(op: string, input?: { code?: string; frame?: number }): Promise<unknown>;-}--/**- * executeJavaScript rejects a thrown error with an opaque "Script failed to- * execute". Wrap the expression so the real message crosses the boundary — a- * surface error like `component "X" is not on entity 5` IS the caller's- * feedback, not a debugging detail.- */-/**- * Wrap a snippet so its value comes back and its throw becomes ours.- *- * A probe body is a PROGRAM, not necessarily an expression — `a(); b(); "done"`- * is what anyone writes, and inside parentheses that is a syntax error. Which- * form to use is decided HERE, in the main process, by trying to parse it: the- * realm has a CSP and must not be asked to compile anything. An expression keeps- * its implicit value; a body returns what it returns.- */-const carryError = (code: string): string => {- let isExpression = true;- try {- new Function(`return (${code})`);- } catch {- isExpression = false;- }- const value = isExpression ? `await (${code})` : `await (async () => { ${code} })()`;- return `(async () => { try { return { v: ${value} }; } catch (e) { return { __estellaExecError: (e && e.message) || String(e) }; } })()`;-};--function unwrap(res: unknown): unknown {- const r = res as { v?: unknown; __estellaExecError?: string } | null;- if (r && typeof r === 'object' && r.__estellaExecError !== undefined) throw new Error(r.__estellaExecError);- return r?.v;-}--/** JSON-encode a call's arguments, keeping `undefined` as the literal so the- * target method's own default parameters apply. */-const encodeArgs = (args: readonly unknown[]): string =>- args.map((a) => (a === undefined ? 'undefined' : JSON.stringify(a))).join(',');--/**- * A driver bound to the editor window. `getWin` re-resolves per call, so a- * recreated window keeps working and a driver built before the window exists- * is still valid.- */-export function createSurfaceDriver(getWin: () => BrowserWindow | null): SurfaceDriver {- const requireWin = (): BrowserWindow => {- const win = getWin();- if (!win) throw new Error('no editor window');- return win;- };-- const exec = async (code: string): Promise<unknown> => {- const win = requireWin();- // A call can arrive before the renderer finished booting (the endpoint comes- // up with the window) — wait out the initial load instead of failing it.- // Polled, never event-awaited: isLoading() covers SUBFRAMES (the play realm- // prewarms in an iframe seconds after a project opens), but did-finish-load- // fires only for the main frame — a call that awaited the event during a- // subframe load parked forever, timing out whichever automation call drew- // the short straw. Bounded: a page that never settles still gets the call,- // and executeJavaScript's own error is a better answer than silence.- for (let waited = 0; win.webContents.isLoading() && waited < 10_000; waited += 50) {- await new Promise((r) => setTimeout(r, 50));- }- return unwrap(await win.webContents.executeJavaScript(carryError(code), true));- };-- const driver = ((method: string, args: readonly unknown[] = [], root?: string) => {- const target = root === 'editor' ? 'window.__estellaEditor' : 'window.__estellaEditor.surface';- return exec(`${target}.${method}(${encodeArgs(args)})`);- }) as SurfaceDriver;-- driver.js = exec;-- driver.op = async (op, input = {}) => {- switch (op) {- case 'screenshot': {- // The composited window, not the viewport canvas: this is the only way to- // see the play realm, which renders in its own frame.- const image = await requireWin().webContents.capturePage();- return image.toPNG().toString('base64');- }- case 'play_probe': {- // The play realm is an estella:// OOPIF — only a main-process frame eval- // reaches it (window.__estellaPlay is the probe the realm publishes).- const frames = requireWin().webContents.mainFrame.frames.filter((f) => f.url.startsWith('estella://'));- const at = input.frame ?? 0;- const frame = frames[at];- if (!frame) {- throw new Error(`no play realm at index ${at} (${frames.length} running — enter play first)`);- }- return unwrap(await frame.executeJavaScript(carryError(input.code ?? 'true')));- }- default:- throw new Error(`unknown main-process op: ${op}`);- }- };-- return driver;-}+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file surfaceDriver.ts — main-side calls INTO the live editor window.+ *+ * The tool catalog (shared/toolCatalog.mjs) dispatches through an injected+ * `driver`, and this is the one that targets the real editor app: surface+ * calls resolve on `window.__estellaEditor.surface` (the same+ * EditorControlSurface the UI uses), `root: 'editor'` calls on+ * `window.__estellaEditor` (project / asset / play doors), renderer-code+ * snippets run verbatim, and the two main-process ops reach what the renderer+ * cannot — the composited window, and the play realm's out-of-process frame.+ *+ * Deliberately transport-free. The loopback /exec endpoint wraps this+ * (mcpEndpoint.ts owns the token, discovery file and ready line), and anything+ * else in main that wants to drive the editor calls the same object directly —+ * so an in-process caller and a remote MCP client cannot drift apart.+ */+import type { BrowserWindow } from 'electron';++/** What the tool registry calls: a surface/editor method, plus the two escape+ * hatches it also knows about (`js` renderer code, `op` main-process routines). */+export interface SurfaceDriver {+ (method: string, args?: readonly unknown[], root?: string): Promise<unknown>;+ js(code: string): Promise<unknown>;+ op(op: string, input?: Record<string, unknown>): Promise<unknown>;+}++/**+ * executeJavaScript rejects a thrown error with an opaque "Script failed to+ * execute". Wrap the expression so the real message crosses the boundary — a+ * surface error like `component "X" is not on entity 5` IS the caller's+ * feedback, not a debugging detail.+ */+/**+ * Wrap a snippet so its value comes back and its throw becomes ours.+ *+ * A probe body is a PROGRAM, not necessarily an expression — `a(); b(); "done"`+ * is what anyone writes, and inside parentheses that is a syntax error. Which+ * form to use is decided HERE, in the main process, by trying to parse it: the+ * realm has a CSP and must not be asked to compile anything. An expression keeps+ * its implicit value; a body returns what it returns.+ */+const carryError = (code: string): string => {+ let isExpression = true;+ try {+ new Function(`return (${code})`);+ } catch {+ isExpression = false;+ }+ const value = isExpression ? `await (${code})` : `await (async () => { ${code} })()`;+ return `(async () => { try { return { v: ${value} }; } catch (e) { return { __estellaExecError: (e && e.message) || String(e) }; } })()`;+};++function unwrap(res: unknown): unknown {+ const r = res as { v?: unknown; __estellaExecError?: string } | null;+ if (r && typeof r === 'object' && r.__estellaExecError !== undefined) throw new Error(r.__estellaExecError);+ return r?.v;+}++/** JSON-encode a call's arguments, keeping `undefined` as the literal so the+ * target method's own default parameters apply. */+const encodeArgs = (args: readonly unknown[]): string =>+ args.map((a) => (a === undefined ? 'undefined' : JSON.stringify(a))).join(',');++/**+ * A driver bound to the editor window. `getWin` re-resolves per call, so a+ * recreated window keeps working and a driver built before the window exists+ * is still valid.+ */+export function createSurfaceDriver(getWin: () => BrowserWindow | null): SurfaceDriver {+ const requireWin = (): BrowserWindow => {+ const win = getWin();+ if (!win) throw new Error('no editor window');+ return win;+ };++ const exec = async (code: string): Promise<unknown> => {+ const win = requireWin();+ // A call can arrive before the renderer finished booting (the endpoint comes+ // up with the window) — wait out the initial load instead of failing it.+ // Polled, never event-awaited: isLoading() covers SUBFRAMES (the play realm+ // prewarms in an iframe seconds after a project opens), but did-finish-load+ // fires only for the main frame — a call that awaited the event during a+ // subframe load parked forever, timing out whichever automation call drew+ // the short straw. Bounded: a page that never settles still gets the call,+ // and executeJavaScript's own error is a better answer than silence.+ for (let waited = 0; win.webContents.isLoading() && waited < 10_000; waited += 50) {+ await new Promise((r) => setTimeout(r, 50));+ }+ return unwrap(await win.webContents.executeJavaScript(carryError(code), true));+ };++ const driver = ((method: string, args: readonly unknown[] = [], root?: string) => {+ const target = root === 'editor' ? 'window.__estellaEditor' : 'window.__estellaEditor.surface';+ return exec(`${target}.${method}(${encodeArgs(args)})`);+ }) as SurfaceDriver;++ driver.js = exec;++ driver.op = async (op, input = {}) => {+ switch (op) {+ case 'screenshot': {+ // The composited window, not the viewport canvas: this is the only way to+ // see the play realm, which renders in its own frame.+ const image = await requireWin().webContents.capturePage();+ return image.toPNG().toString('base64');+ }+ case 'play_probe': {+ // The play realm is an estella:// OOPIF — only a main-process frame eval+ // reaches it (window.__estellaPlay is the probe the realm publishes).+ const frames = requireWin().webContents.mainFrame.frames.filter((f) => f.url.startsWith('estella://'));+ const at = Number(input.frame ?? 0);+ const frame = frames[at];+ if (!frame) {+ throw new Error(`no play realm at index ${at} (${frames.length} running — enter play first)`);+ }+ return unwrap(await frame.executeJavaScript(carryError(String(input.code ?? 'true'))));+ }+ case 'play_input': {+ // Same frame lookup as play_probe — the realm is the only thing that has+ // an InputState — but the code is OURS, so a caller cannot mistype the+ // facade and get silence. Everything routes through the realm's+ // `__estellaPlay.input`, which is the platform binding's own callbacks.+ const frames = requireWin().webContents.mainFrame.frames.filter((f) => f.url.startsWith('estella://'));+ const at = Number(input.frame ?? 0);+ const frame = frames[at];+ if (!frame) {+ throw new Error(`no play realm at index ${at} (${frames.length} running — enter play first)`);+ }+ const { kind } = input as { kind?: string };+ const x = Number(input.x ?? 0), y = Number(input.y ?? 0);+ const button = Number(input.button ?? 0);+ const code = JSON.stringify(String(input.code ?? ''));+ const id = 1;+ const call = {+ click: `i.down(${x},${y},${button}); i.up(${button});`,+ move: `i.move(${x},${y});`,+ down: `i.down(${x},${y},${button});`,+ up: `i.up(${button});`,+ wheel: `i.wheel(${x},${y});`,+ key_down: `i.keyDown(${code});`,+ key_up: `i.keyUp(${code});`,+ tap: `i.touchStart(${id},${x},${y}); i.touchEnd(${id});`,+ }[kind ?? ''];+ if (!call) {+ throw new Error(+ `unknown play_input kind "${kind}" — one of click, move, down, up, wheel, key_down, key_up, tap`,+ );+ }+ return unwrap(await frame.executeJavaScript(carryError(+ `(() => { const p = window.__estellaPlay; if (!p || !p.input) `+ + `throw new Error('this play realm publishes no input door — it predates play_input'); `+ + `const i = p.input; ${call} return 'ok'; })()`,+ )));+ }+ default:+ throw new Error(`unknown main-process op: ${op}`);+ }+ };++ return driver;+}diff --git a/desktop/shared/toolCatalog.mjs b/desktop/shared/toolCatalog.mjsindex 4d41e360c..1b296956d 100644--- a/desktop/shared/toolCatalog.mjs+++ b/desktop/shared/toolCatalog.mjs@@ -495,6 +495,19 @@ export const TOOLS = [description: "Evaluate JS inside the RUNNING play realm and return the result — the gameplay probe. One expression gives its value; several statements need an explicit `return`. window.__estellaPlay = { app, getComponent }: `getComponent(NAME)` gives the component DEFINITION, `app.world.get(entity, def)` a COPY of its data, and a write only lands through `app.world.insert(entity, def, data)` — assigning to the copy changes nothing; to drive gameplay input, dispatch KeyboardEvents on DOCUMENT (the engine listens there, not on window): document.dispatchEvent(new KeyboardEvent('keydown', {code:'ArrowRight'})). frame picks the realm in multiplayer previews (0 = host).",schema: obj({ code: { type: 'string' }, frame: { type: 'number' } }, ['code']),op: 'play_probe' },+ { name: 'play_input', effect: 'irreversible',+ description: 'Deliver a pointer or key event to the RUNNING game — the only way to exercise the input a game actually ships with. '+ + "`kind`: 'click' (down+up at x,y), 'move', 'down', 'up', 'wheel', 'key_down', 'key_up', 'tap' (touch down+up). "+ + 'x/y are SCREEN pixels, canvas-relative, y DOWN — the same numbers a real pointer event carries and what `Input.mouseX/mouseY` then read; '+ + 'ask the camera (CameraView.screenToWorld / UICameraInfo.worldMouseX) for where that is in the world rather than converting by hand. '+ + '`code` is a KeyboardEvent code for the key kinds. It goes through the platform binding\'s own callbacks, so UI gets first refusal exactly as it would for a real event. '+ + 'A pressed EDGE (isMouseButtonPressed / isKeyPressed) lasts one frame: call step() right after, then read the result. '+ + 'Calling the game\'s own handler instead proves only that the handler works — not that a click reaches it.',+ schema: obj({+ kind: { type: 'string' }, x: { type: 'number' }, y: { type: 'number' },+ button: { type: 'number' }, code: { type: 'string' }, frame: { type: 'number' },+ }, ['kind']),+ op: 'play_input' },{ name: 'world_component',description: "A LIVE World component's data for a source entity, resolved by component name — verifies an edit actually reached the engine.",schema: obj({ id: { type: 'number' }, component: { type: 'string' } }, ['id', 'component']),diff --git a/desktop/src/playHost.ts b/desktop/src/playHost.tsindex ecbd213f9..bf5b75ae3 100644--- a/desktop/src/playHost.ts+++ b/desktop/src/playHost.ts@@ -14,7 +14,7 @@* Everything is same-origin estella:// (host, sdk, bundle, wasm, assets),* sidestepping the custom-scheme cross-fetch ban.*/-import { createWebApp, setEditorMode, setPlayMode, initPlayRealmRuntime, getComponent, clearUserComponents, getUserComponentFingerprint, probeRegistrations, Net, MessagePortTransport, Assets, Ads, createMockAdProvider, Leaderboard, createLocalLeaderboard, registerPackagedSideModules } from 'esengine';+import { createWebApp, setEditorMode, setPlayMode, initPlayRealmRuntime, getComponent, clearUserComponents, getUserComponentFingerprint, probeRegistrations, Net, MessagePortTransport, Assets, Ads, createMockAdProvider, Leaderboard, createLocalLeaderboard, registerPackagedSideModules, Input, inputEventCallbacks } from 'esengine';import type { App, ESEngineModule, SceneData } from 'esengine';import { PLAY_PROTOCOL_VERSION } from './engine/playProtocol';import type { PlayOutbound, PlayInbound } from './engine/playProtocol';@@ -277,7 +277,31 @@ async function buildAppAndRun(msg: InitMessage): Promise<void> {// host's __estellaHeadless): the editor can't reach into this OOPIF except by// main-process frame eval, and the eval needs SOMETHING to query. getComponent// lets a probe read component data by name (world.get needs the def).- (window as unknown as { __estellaPlay?: unknown }).__estellaPlay = { app, getComponent };+ // `input` delivers a pointer/key event through the SAME callbacks the platform+ // binding hands raw events to, so an injected click is routed past the UI and+ // recorded exactly like a real one. Without it a driver can build input+ // handling and never test it: the realm is an out-of-process frame that+ // sendInputEvent does not reach, a synthetic MouseEvent carries no usable+ // offsetX, and poking mouseX / mouseButtonsPressed by hand skips the router.+ // What that costs is not hypothetical — a chess game was "verified" by calling+ // its own click handler directly, which is how a screen-to-world bug reached+ // the player: the one path nobody could exercise was the one that was wrong.+ const injected = inputEventCallbacks(app.getResource(Input));+ (window as unknown as { __estellaPlay?: unknown }).__estellaPlay = {+ app,+ getComponent,+ input: {+ move: (x: number, y: number) => injected.onPointerMove?.(x, y),+ down: (x: number, y: number, button = 0) => injected.onPointerDown?.(button, x, y),+ up: (button = 0) => injected.onPointerUp?.(button),+ wheel: (dx: number, dy: number) => injected.onWheel?.(dx, dy),+ keyDown: (code: string) => injected.onKeyDown?.(code),+ keyUp: (code: string) => injected.onKeyUp?.(code),+ touchStart: (id: number, x: number, y: number) => injected.onTouchStart?.(id, x, y),+ touchMove: (id: number, x: number, y: number) => injected.onTouchMove?.(id, x, y),+ touchEnd: (id: number) => injected.onTouchEnd?.(id),+ },+ };startFrameHeartbeat();// Surface a failed handshake as a boot error rather than a silentdiff --git a/desktop/tests/agent-prompt.test.ts b/desktop/tests/agent-prompt.test.tsindex 38232feb1..4164053a5 100644--- a/desktop/tests/agent-prompt.test.ts+++ b/desktop/tests/agent-prompt.test.ts@@ -37,6 +37,16 @@ describe('the agent is told where game input comes from', () => {expect(SYSTEM_PROMPT).toMatch(/offset.*limit|limit.*offset/);});+ it('names the two front doors, and that the underscore fields are not one', () => {+ // A hand-assembled SystemDef happens to RUN, so nothing complains — it just+ // leaves a project whose systems no tool of ours recognises.+ expect(SYSTEM_PROMPT).toContain('defineSystem');+ expect(SYSTEM_PROMPT).toContain('defineComponent');+ expect(SYSTEM_PROMPT).toContain('addSystemToSchedule');+ expect(SYSTEM_PROMPT).toMatch(/_params.*_fn|_fn.*_params/);+ expect(SYSTEM_PROMPT).toMatch(/not an authoring surface/);+ });+it('stays one text — it sits in the cached prefix and must not vary per turn', () => {expect(typeof SYSTEM_PROMPT).toBe('string');expect(SYSTEM_PROMPT).toBe(SYSTEM_PROMPT);