A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(debug): a connected build answers with the whole frame it composited
esengine8512f8b2f172
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 18a8b35ea..f4fee3f1f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,6 +19,7 @@ published separately; it ships inside the editor.- **Android release signing.** A Shipping APK (and App Bundle) is signed with the release key the project names under Project Settings → Android → Signing — two PEM paths, the passphrase kept in this machine's keychain (the command line reads `ESTELLA_ANDROID_KEY_PASSPHRASE`); Development builds keep the development key. A Shipping build without one says it will be refused by Google Play.- **EmscriptenGLX on WeChat.** A WeChat project can ship the engine linked with WeChat's EmscriptenGLX (Project Settings → WeChat → Render through EmscriptenGLX, off by default), which issues the engine's GL calls natively instead of through JavaScript: a draw-heavy scene spent about 12% less time rendering on an Android phone. It adds about 310KB of wasm and needs base library 3.8.12 on Android, or iOS in 高性能+ mode; any other host renders through WebGL2 as before, and the log says which. The engine is built once with `node build-tools/cli.js build -t wechat-glx`, which fetches WeChat's library at a pinned version and hash, and emsdk 4.0.10 beside the main one; an export that asks for it without it says so.- A connected development build says which graphics API it draws through — WebGL2, EmscriptenGLX or WebGPU — once its game has started: the Profiler shows it beside the device's controls, the Output Log notes it, and `device_targets` returns it as `graphics`. A WeChat build set to EmscriptenGLX reports WebGL2 on a host that does not offer it.+- A connected development build can be asked for the whole frame it composited — every camera and the UI over them — as the player sees it: `device_screenshot` returns it as a PNG, read back by the device's own engine. The frame debugger's replay shows one pass of a captured frame.- **Export profiles.** A project can name several exports of one target in `packaging.profiles` — a test and a production WeChat build, each with its own AppID, CDN root and development or shipping configuration, writing only what differs from the project's settings. The Build dialog creates, edits and deletes them per target, marking each setting a profile overrides with a reset that hands it back to the project; `estella export --profile <name>` and the agent's `export_game` build the same one, and the size history records which profile a build used.### Changeddiff --git a/desktop b/desktopindex 547804c8b..a6895d52e 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit 547804c8bde01f20d252edcb65e6b8be8b714543+Subproject commit a6895d52ed96972ab14427a699b0f335ff4b666ediff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.mdindex 80e6bf946..d29fd57f7 100644--- a/sdk/etc/index.api.md+++ b/sdk/etc/index.api.md@@ -2740,6 +2740,8 @@ url: string```| { t: 'query'; reqId: number; kind: 'frameCapture' }| { t: 'query'; reqId: number; kind: 'frameReplay'; drawIndex: number; maxSide?: number }+ /** The whole composited frame, as the engine drew it. */+ | { t: 'query'; reqId: number; kind: 'screen'; maxSide?: number }| { t: 'query'; reqId: number; kind: 'stats' }| { t: 'query'; reqId: number; kind: 'updateStatus' }| { t: 'query'; reqId: number; kind: 'snapshot'; selectedId: number | null; withTree: boolean }diff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.mdindex 13dbab512..372aa6f7e 100644--- a/sdk/etc/index.native.api.md+++ b/sdk/etc/index.native.api.md@@ -2748,6 +2748,8 @@ url: string```| { t: 'query'; reqId: number; kind: 'frameCapture' }| { t: 'query'; reqId: number; kind: 'frameReplay'; drawIndex: number; maxSide?: number }+ /** The whole composited frame, as the engine drew it. */+ | { t: 'query'; reqId: number; kind: 'screen'; maxSide?: number }| { t: 'query'; reqId: number; kind: 'stats' }| { t: 'query'; reqId: number; kind: 'updateStatus' }| { t: 'query'; reqId: number; kind: 'snapshot'; selectedId: number | null; withTree: boolean }diff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.mdindex 981944b2d..3826d3c2c 100644--- a/sdk/etc/index.node.api.md+++ b/sdk/etc/index.node.api.md@@ -2735,6 +2735,8 @@ url: string```| { t: 'query'; reqId: number; kind: 'frameCapture' }| { t: 'query'; reqId: number; kind: 'frameReplay'; drawIndex: number; maxSide?: number }+ /** The whole composited frame, as the engine drew it. */+ | { t: 'query'; reqId: number; kind: 'screen'; maxSide?: number }| { t: 'query'; reqId: number; kind: 'stats' }| { t: 'query'; reqId: number; kind: 'updateStatus' }| { t: 'query'; reqId: number; kind: 'snapshot'; selectedId: number | null; withTree: boolean }diff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.mdindex e678f7a68..db4e3383d 100644--- a/sdk/etc/minigame.api.md+++ b/sdk/etc/minigame.api.md@@ -2735,6 +2735,8 @@ url: string```| { t: 'query'; reqId: number; kind: 'frameCapture' }| { t: 'query'; reqId: number; kind: 'frameReplay'; drawIndex: number; maxSide?: number }+ /** The whole composited frame, as the engine drew it. */+ | { t: 'query'; reqId: number; kind: 'screen'; maxSide?: number }| { t: 'query'; reqId: number; kind: 'stats' }| { t: 'query'; reqId: number; kind: 'updateStatus' }| { t: 'query'; reqId: number; kind: 'snapshot'; selectedId: number | null; withTree: boolean }diff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.mdindex 65fe66ed9..7b345862b 100644--- a/sdk/etc/wechat.api.md+++ b/sdk/etc/wechat.api.md@@ -2735,6 +2735,8 @@ url: string```| { t: 'query'; reqId: number; kind: 'frameCapture' }| { t: 'query'; reqId: number; kind: 'frameReplay'; drawIndex: number; maxSide?: number }+ /** The whole composited frame, as the engine drew it. */+ | { t: 'query'; reqId: number; kind: 'screen'; maxSide?: number }| { t: 'query'; reqId: number; kind: 'stats' }| { t: 'query'; reqId: number; kind: 'updateStatus' }| { t: 'query'; reqId: number; kind: 'snapshot'; selectedId: number | null; withTree: boolean }diff --git a/sdk/src/app/app.ts b/sdk/src/app/app.tsindex 2c918ecd3..08d37d0a5 100644--- a/sdk/src/app/app.ts+++ b/sdk/src/app/app.ts@@ -1694,11 +1694,20 @@ export interface WebAppOptions {/** The graphics API a web app's renderer draws through, as a connected device reports it. */export type GraphicsPath = 'WebGPU' | 'WebGL2' | 'EmscriptenGLX';-const graphicsPaths = new WeakMap<App, GraphicsPath>();+interface AppSurface {+ graphics: GraphicsPath | null;+ size: (() => { width: number; height: number }) | null;+}+const surfaces = new WeakMap<App, AppSurface>();/** Null for an app whose module resolved its own surface. */export function graphicsPathOf(app: App): GraphicsPath | null {- return graphicsPaths.get(app) ?? null;+ return surfaces.get(app)?.graphics ?? null;+}++/** The size the engine draws at, as its host reports it; null when the host did not say. */+export function surfaceSizeOf(app: App): { width: number; height: number } | null {+ return surfaces.get(app)?.size?.() ?? null;}export function graphicsPathFor(surface: RenderSurfaceSource): GraphicsPath | null {@@ -1752,8 +1761,7 @@ export function createWebApp(module: ESEngineModule, options?: WebAppOptions): Amodule.initRenderer();break;}- const graphics = graphicsPathFor(surface);- if (graphics) graphicsPaths.set(app, graphics);+ surfaces.set(app, { graphics: graphicsPathFor(surface), size: options?.getViewportSize ?? null });// Always applied (not only when set): the renderer outlives the App on realm// reloads, so a fresh App must reset a prior session's y-sort state too.module.renderer_setYSortLayers?.((options?.ySortLayers ?? 0) >>> 0);diff --git a/sdk/src/runtime/debugChannel.ts b/sdk/src/runtime/debugChannel.tsindex 3067695e8..ee4db70e2 100644--- a/sdk/src/runtime/debugChannel.ts+++ b/sdk/src/runtime/debugChannel.ts@@ -11,11 +11,12 @@* mini-game host cannot be reached, and the editor authenticates the dial by the* token in its URL before a single frame crosses.*/-import { graphicsPathOf, type App } from '../app/app';+import { graphicsPathOf, surfaceSizeOf, type App } from '../app/app';import { getPlatform, platformNow } from '../platform/base';import type { PlatformSocket } from '../platform/types';import type { NextFrame } from '../render/frameCapture';import { captureFrameReport, captureEngineOf, replayFrameDraw, type FrameReplayImage } from '../render/frameDebugReport';+import { captureFramePixels } from '../render/framePixels';import { log } from '../util/logger';import { Assets } from '../asset/AssetPlugin';import { Lifecycle } from '../ecs/lifecycle';@@ -61,6 +62,14 @@ export function fitWithin(image: FrameReplayImage, maxSide?: number): FrameReplareturn { ...image, width: sw, height: sh, pixels: new Uint8ClampedArray(out.buffer), fullWidth: w, fullHeight: h };}+/** The engine reads its frame back bottom row first; an image is sent top row first. */+function topDown(rgba: Uint8Array, width: number, height: number): Uint8ClampedArray {+ const out = new Uint8ClampedArray(rgba.length);+ const row = width * 4;+ for (let y = 0; y < height; y++) out.set(rgba.subarray((height - 1 - y) * row, (height - y) * row), y * row);+ return out;+}+/** Resolves once @p app has finished a frame and the task that drew it has run. */function appFrame(app: App): NextFrame {return () => new Promise((resolve) => {@@ -137,6 +146,18 @@ export function startDebugChannel(config: DebugChannelConfig): void {const answer = async (q: DebugChannelQuery): Promise<void> => {const reply = (m: DebugChannelMessage): void => send(m);+ // A mini-game's socket moved a full-screen image at ~180KB/s (55 s), and+ // drops a frame that size outright: scaled to what the editor will show,+ // then sent in chunks behind the reply.+ const replyImage = (reqId: number, full: FrameReplayImage, maxSide?: number): void => {+ const { pixels, ...rest } = timed(() => fitWithin(full, maxSide));+ const chunks = Math.max(1, Math.ceil(pixels.byteLength / PIXEL_CHUNK));+ reply({ t: 'reply', reqId, data: rest, pixels: chunks });+ for (let i = 0; i < chunks; i++) {+ const part = pixels.subarray(i * PIXEL_CHUNK, Math.min(pixels.byteLength, (i + 1) * PIXEL_CHUNK));+ socket?.send(part.slice().buffer as ArrayBuffer);+ }+ };try {if (!app) {const late = new Promise<never>((_, reject) => setTimeout(@@ -147,7 +168,7 @@ export function startDebugChannel(config: DebugChannelConfig): void {const frame = nextFrame!;// A background tab gets no frames, so it would sit on this until the editor gave up.const drawsFrames = !game.hasResource(Lifecycle) || game.getResource(Lifecycle).visible;- if ((q.kind === 'frameCapture' || q.kind === 'frameReplay') && !drawsFrames) {+ if ((q.kind === 'frameCapture' || q.kind === 'frameReplay' || q.kind === 'screen') && !drawsFrames) {throw new Error('the game is in the background, where it draws no frames: bring it to the front');}if ((q.kind === 'frameCapture' || q.kind === 'frameReplay') && !captureEngineOf(game)) {@@ -183,6 +204,22 @@ export function startDebugChannel(config: DebugChannelConfig): void {reply({ t: 'reply', reqId: q.reqId, data: timed(() => worldSnapshot(game, q.selectedId, q.withTree)) });return;}+ if (q.kind === 'screen') {+ const size = surfaceSizeOf(game);+ if (!size || !game.wasmModule) throw new Error('this build cannot read its frame back: its host did not say what size it draws at');+ // Asked for inside a frame, before it is handed to the page: GL reads+ // the default framebuffer on the spot, and once presented it is cleared.+ const module = game.wasmModule;+ const shot = await new Promise<Awaited<ReturnType<typeof captureFramePixels>>>((resolve, reject) => {+ const off = game.onFrameEnd(() => {+ off();+ captureFramePixels(module, size.width, size.height, () => frame()).then(resolve, reject);+ });+ });+ if (!shot) throw new Error('the engine did not return its frame — this build cannot read its own pixels');+ replyImage(q.reqId, { width: shot.width, height: shot.height, pixels: topDown(shot.rgba, shot.width, shot.height), matchesCapture: true }, q.maxSide);+ return;+ }if (q.kind === 'control') {if (q.fps !== undefined) game.setTargetFrameRate(q.fps);if (q.paused !== undefined) game.setPaused(q.paused);@@ -195,16 +232,7 @@ export function startDebugChannel(config: DebugChannelConfig): void {reply({ t: 'reply', reqId: q.reqId, data: null });return;}- // A mini-game's socket moved a full-screen replay at ~180KB/s (55 s);- // the editor asks for what it will show.- const image = timed(() => fitWithin(full, q.maxSide));- const { pixels, ...rest } = image;- const chunks = Math.max(1, Math.ceil(pixels.byteLength / PIXEL_CHUNK));- reply({ t: 'reply', reqId: q.reqId, data: rest, pixels: chunks });- for (let i = 0; i < chunks; i++) {- const part = pixels.subarray(i * PIXEL_CHUNK, Math.min(pixels.byteLength, (i + 1) * PIXEL_CHUNK));- socket?.send(part.slice().buffer as ArrayBuffer);- }+ replyImage(q.reqId, full, q.maxSide);} catch (e) {reply({ t: 'reply', reqId: q.reqId, error: e instanceof Error ? e.message : String(e) });}diff --git a/sdk/src/runtime/debugChannelProtocol.ts b/sdk/src/runtime/debugChannelProtocol.tsindex c49e2e84f..7d3154fa4 100644--- a/sdk/src/runtime/debugChannelProtocol.ts+++ b/sdk/src/runtime/debugChannelProtocol.ts@@ -30,6 +30,8 @@ export interface DebugControl {export type DebugChannelQuery =| { t: 'query'; reqId: number; kind: 'frameCapture' }| { t: 'query'; reqId: number; kind: 'frameReplay'; drawIndex: number; maxSide?: number }+ /** The whole composited frame, as the engine drew it. */+ | { t: 'query'; reqId: number; kind: 'screen'; maxSide?: number }| { t: 'query'; reqId: number; kind: 'stats' }| { t: 'query'; reqId: number; kind: 'updateStatus' }| { t: 'query'; reqId: number; kind: 'snapshot'; selectedId: number | null; withTree: boolean }diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 710c55b22..b4fa089f1 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -1194,4 +1194,8 @@ export const NOTED = {{ note: 'Sprite masks work in WeChat and other mini-game packages: the game\'s WebGL context was created without a stencil buffer, so every mask drew as if it were not there (space-shooter\'s hull bar read as empty). The web, playable and mini-game hosts now create the context with the same attributes.' },'test(golden): space-shooter is certified on WeChat by the pixels it claims':{ internal: 'A repository golden check; no creator runs it.' },+ 'feat(debug): a connected build answers with the whole frame it composited':+ { note: 'A connected development build can be asked for the whole frame it composited — every camera and the UI over them — as the player sees it: `device_screenshot` returns it as a PNG, read back by the device\'s own engine. The frame debugger\'s replay shows one pass of a captured frame.' },+ 'feat(agent): device_screenshot shows what a connected build shows its player':+ { note: 'A connected development build can be asked for the whole frame it composited — every camera and the UI over them — as the player sees it: `device_screenshot` returns it as a PNG, read back by the device\'s own engine. The frame debugger\'s replay shows one pass of a captured frame.' },};diff --git a/tools/verify-remote-capture.mjs b/tools/verify-remote-capture.mjsindex 74156fc7f..82f208d11 100644--- a/tools/verify-remote-capture.mjs+++ b/tools/verify-remote-capture.mjs@@ -119,6 +119,19 @@ try {}check(lit > 0, `the replayed pass has drawn pixels (${lit})`);+ // A replay is one pass on a transparent background; what the player sees is+ // every pass composited, opaque everywhere.+ const screen = await server.query(target.id, 'screen', { maxSide: 400 });+ let clear = 0;+ const colours = new Set();+ for (let i = 0; screen && i < screen.pixels.length; i += 4) {+ if (screen.pixels[i + 3] < 255) clear++;+ colours.add((screen.pixels[i] << 16) | (screen.pixels[i + 1] << 8) | screen.pixels[i + 2]);+ }+ check(!!screen && screen.width > 0 && screen.pixels.byteLength === screen.width * screen.height * 4 && clear === 0,+ `the screen came back whole and opaque: ${screen?.width}x${screen?.height}, ${clear} see-through pixel(s)`);+ check(colours.size > 1, `the screen shows more than one colour (${colours.size}: ${[...colours].slice(0, 3).map((c) => c.toString(16)).join(",")})`);+const now = server.targets().find((t) => t.id === target.id);check(now?.project === 'Input Actions' && typeof now?.revision === 'string',`the build said which project and content it is (${now?.project}, ${now?.revision})`);