Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

fix(editor): the agent is told where game input comes from

yhhda84559f3a3f
diff --git a/desktop/electron/agent/prompt.ts b/desktop/electron/agent/prompt.ts
index 0fe020df6..f684b1ee6 100644
--- a/desktop/electron/agent/prompt.ts
+++ b/desktop/electron/agent/prompt.ts
@@ -34,6 +34,7 @@ Working style:
- Prefer one apply_scene_ops over many small calls when building something: it is one undo step for the user and one round trip for you, and it is the ONE write the user sees BEFORE it lands — they get a line-by-line preview and can strike parts of it out. A dozen set_field calls give them no such chance, so build with a batch and save the single writes for genuinely single changes.
- Units are the editor's: 1 world unit = 1 design pixel. Y is up in the world, and UI is laid out top-down.
- Game content lives in the WORLD: a board, its pieces, a character are Sprite / ShapeRenderer entities you place with Transform.position. A Canvas and its UINodes are for HUD and menus, and their placement is the layout's OUTPUT — a Transform.position written on a UI node is overwritten at the next relayout, which looks like nothing happened. Move those with layout inputs instead (UINode.position = Absolute, then left/top), or let the flow place them.
+- A game reads input from the ENGINE, never from the DOM. In a system, ask for \`Res(Input)\` and \`Res(CameraView)\`: \`input.isMouseButtonPressed(MouseButton.Left)\`, \`input.mouseX/mouseY\` (SCREEN pixels, canvas-relative, y down), and \`camera.getWorldMousePosition()\` / \`camera.screenToWorld(x, y)\` for where that is in the world. \`document.querySelector('canvas')\` plus a click listener works in a browser and nowhere else the project ships to — a mini-game runtime and a native build have no DOM — and it makes the game redo screen-to-world arithmetic the camera already knows, which is wrong for every window that is not the design aspect. Read the shape of an API before calling it (read_project_file takes \`offset\` and \`limit\` to page a big .d.ts) rather than guessing method names into a try/catch that hides the mistake.
- Say what you did in a sentence or two. The user watched it happen — a summary of every call is noise. Speak the user's language: answer in the language they wrote to you in.`;
/**
diff --git a/desktop/tests/agent-prompt.test.ts b/desktop/tests/agent-prompt.test.ts
new file mode 100644
index 000000000..38232feb1
--- /dev/null
+++ b/desktop/tests/agent-prompt.test.ts
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file What the built-in agent is told before it has asked anything.
+ *
+ * The prompt is the ONLY guidance that reaches a turn regardless of which
+ * doors the agent happens to use. Guidance that lives solely in the New
+ * Script template reaches a project that scaffolds its systems and nobody
+ * else: three dogfood runs in a row wrote game input against the DOM —
+ * two of them in projects that already had source, one that hand-wrote
+ * its system rather than scaffolding it — and the DOM is exactly what a
+ * mini-game runtime and a native build do not have.
+ */
+import { describe, it, expect } from 'vitest';
+import { SYSTEM_PROMPT } from '../electron/agent/prompt';
+
+describe('the agent is told where game input comes from', () => {
+ it('names the engine resources rather than describing them', () => {
+ expect(SYSTEM_PROMPT).toContain('Res(Input)');
+ expect(SYSTEM_PROMPT).toContain('CameraView');
+ expect(SYSTEM_PROMPT).toMatch(/isMouseButtonPressed/);
+ expect(SYSTEM_PROMPT).toMatch(/getWorldMousePosition|screenToWorld/);
+ });
+
+ it('says what the DOM route costs, so it reads as a reason and not a rule', () => {
+ expect(SYSTEM_PROMPT).toMatch(/document\.querySelector/);
+ expect(SYSTEM_PROMPT).toMatch(/no DOM|have no DOM/i);
+ });
+
+ it('says the units, because that is the half a caller gets wrong silently', () => {
+ // mouseX/mouseY are screen pixels; treating them as world coordinates is a
+ // board that responds to clicks nowhere near where they landed.
+ expect(SYSTEM_PROMPT).toMatch(/SCREEN pixels/);
+ });
+
+ it('points at the way to read an API instead of guessing at it', () => {
+ expect(SYSTEM_PROMPT).toMatch(/offset.*limit|limit.*offset/);
+ });
+
+ it('stays one text — it sits in the cached prefix and must not vary per turn', () => {
+ expect(typeof SYSTEM_PROMPT).toBe('string');
+ expect(SYSTEM_PROMPT).toBe(SYSTEM_PROMPT);
+ });
+});
diff --git a/sdk/tests/uiMath.test.ts b/sdk/tests/uiMath.test.ts
index 17d321381..fab07b5b2 100644
--- a/sdk/tests/uiMath.test.ts
+++ b/sdk/tests/uiMath.test.ts
@@ -612,3 +612,60 @@ describe('worldToScreen carries the same third dimension', () => {
}
});
});
+
+/**
+ * The property a game reads a click through, spelled out.
+ *
+ * A 2D camera fits by HEIGHT: orthoSize is the half-height, and the half-WIDTH
+ * follows from the viewport's aspect. So the visible world is `centre ± orthoSize`
+ * vertically but `centre ± orthoSize * aspect` horizontally, and only at the
+ * design aspect do those coincide with the design box.
+ *
+ * A game that assumes the design box instead — `worldX = px * (designW / rectW)`
+ * — is exact at the centre and wrong by more the further out the click is. On a
+ * chessboard in a maximised browser window that is a whole file: click the a-file,
+ * select the b-file. Hence CameraView.screenToWorld rather than arithmetic.
+ */
+describe('an orthographic 2D camera fits by height', () => {
+ const ORTHO = 360, CX = 640, CY = 360;
+ // What the camera builds: translate to the centre, then the ortho box.
+ const camera = (vpW: number, vpH: number) => {
+ const halfW = ORTHO * (vpW / vpH);
+ const vp = ortho4(halfW, ORTHO, 1000);
+ // Fold the centre in: world → view is a translation by -centre.
+ const m = new Float32Array(vp);
+ m[12] = -CX / halfW;
+ m[13] = -CY / ORTHO;
+ return { vp: m, inv: invertMatrix4(m)!, halfW };
+ };
+
+ it('shows centre ± orthoSize*aspect across, whatever the viewport shape', () => {
+ for (const [w, h] of [[1280, 720], [1600, 800], [1920, 950]]) {
+ const { inv, halfW } = camera(w, h);
+ const left = screenToWorld(0, h / 2, inv, 0, 0, w, h);
+ const right = screenToWorld(w, h / 2, inv, 0, 0, w, h);
+ expect(left.x).toBeCloseTo(CX - halfW, 3);
+ expect(right.x).toBeCloseTo(CX + halfW, 3);
+ // Vertically it is always the design height — that is what "fit by
+ // height" means, and why only the horizontal answer moves.
+ // screenY here is GL's (up from the BOTTOM), not the DOM's: a caller
+ // holding a MouseEvent has to flip it first, and one that forgets
+ // gets a board mirrored about its middle rank.
+ const atBottom = screenToWorld(w / 2, 0, inv, 0, 0, w, h);
+ const atTop = screenToWorld(w / 2, h, inv, 0, 0, w, h);
+ expect(atBottom.y).toBeCloseTo(CY - ORTHO, 3);
+ expect(atTop.y).toBeCloseTo(CY + ORTHO, 3);
+ }
+ });
+
+ it('disagrees with design-box arithmetic away from the centre', () => {
+ const [w, h] = [1600, 800];
+ const { inv } = camera(w, h);
+ const designX = (px: number) => px * (1280 / w); // what a hand-rolled game does
+ // Dead centre: the two agree, which is why the bug looks intermittent.
+ expect(screenToWorld(w / 2, h / 2, inv, 0, 0, w, h).x).toBeCloseTo(designX(w / 2), 3);
+ // A tenth of the way in: off by more than half a 80-unit board square.
+ const near = screenToWorld(w * 0.1, h / 2, inv, 0, 0, w, h).x;
+ expect(Math.abs(near - designX(w * 0.1))).toBeGreaterThan(40);
+ });
+});