A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat: publish screen UI projection helpers for debugger picking
yhhddcd9c3a609f
diff --git a/desktop b/desktopindex f068cea73..348f995f4 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit f068cea73cb0144fb73047320b7cf90bf4c2e70f+Subproject commit 348f995f49e933f6985d71a5c8fec902f9692b2cdiff --git a/docs/astro/src/data/apiStability.generated.json b/docs/astro/src/data/apiStability.generated.jsonindex 364b1c156..0058db530 100644--- a/docs/astro/src/data/apiStability.generated.json+++ b/docs/astro/src/data/apiStability.generated.json@@ -3,10 +3,10 @@"counts": {"public": 115,"beta": 103,- "experimental": 2150,+ "experimental": 2168,"deprecated": 3},- "total": 2368,+ "total": 2386,"subsystems": [{"id": "ecs",diff --git a/examples/ui-localization-debug/README.md b/examples/ui-localization-debug/README.mdindex 1e77af2eb..e0e64713a 100644--- a/examples/ui-localization-debug/README.md+++ b/examples/ui-localization-debug/README.md@@ -1,6 +1,6 @@# UI localization debug-Open this project and use Window → UI Inspector → Text. Wait for layout and+Open this project and use Window → UI Debugger → Text. Wait for layout andcapture again. Original / Accents + 35% / Accents + 100% inspect a snapshotwithout editing or saving scene content. Select jumps to the authored entity.Box and text sizes are local display pixels, before Clip / Ellipsis truncation.@@ -24,23 +24,18 @@ control, capture again, then Select to find it in Details. The host must exposeresolved visibility, otherwise a warning is shown. This does not verify actualkeyboard delivery, screen readers or IME on a target device.-Layout and input shows resolved local dimensions and world axis-aligned bounds-for all UINodes. It reports disabled/hidden controls, ancestor pointer-events-passthrough, raycast-target configuration and parent masks. Expand a row for-bounds, then Select to locate its existing viewport handles. Scene edits make-the snapshot stale and disable selection until capture again.--An input candidate is a configuration result, not a successful pointer hit.-World bounds do not account for clipping or occlusion; parent masks need review.-Inspection does not raycast, change hover/focus, or add persistent gizmos.--Masked rows also show their ancestor mask chain (nearest first), mask mode and-bounds. Unrotated 2D scissor rectangles report estimated containment, partial-clipping or full clipping, plus the remaining world bounds. Stencil, rotated,-unresolved and invalid-hierarchy cases require review. This estimate excludes-camera projection, screen pixel rounding, stencil alpha and actual pointer-delivery. The browser verification fixture finishes with a dedicated nested-mask scene containing Inside / Partial / Outside controls.--This is a partial RM-009 delivery. Actual pointer hits, platform shaping,-real IME/device acceptance and layout cache performance remain separate work.+UI Debugger opens on Elements. Pick element arms the viewport or running-game+view for one selection; the selection click is consumed, then normal input+returns. Escape cancels picking. The hierarchy preserves real UI ancestry and+selection is shared with Details. The property view reads current component+values; editor computed layout refreshes while the panel is open. Edit through+Details. Runtime-only changes revert when play stops.++Screen UI picking and outlines use the renderer's ScreenOverlay projection;+world UI uses the UI camera. Picking is the engine editor selection query,+not proof of actual game event delivery. Only one hovered editor element gets+an extra outline. No clipping estimates or automatic input-success verdicts+are presented. Text and Focus order remain auxiliary snapshot views.++This is a partial RM-009 delivery. Actual event tracing, platform shaping,+IME/device acceptance and layout cache performance remain separate work.diff --git a/examples/ui-localization-debug/VALIDATION.md b/examples/ui-localization-debug/VALIDATION.mdindex 6001b0f46..e58e0990f 100644--- a/examples/ui-localization-debug/VALIDATION.md+++ b/examples/ui-localization-debug/VALIDATION.md@@ -89,28 +89,23 @@ snapshot; live-game-only dialogs and scripts need runtime acceptance separately.platform shaping, IME/accessibility/devices and performance remain open.No local Electron/native test runner was launched.-## Ancestor mask inspection increment — 2026-10-08--The inspector shows the ancestor mask chain, mode and bounds. Unrotated 2D-scissor rectangles estimate contained/partial/fully-clipped geometry; stencil,-rotation, unavailable bounds and malformed hierarchy explicitly require review.-These are world-rectangle estimates, not actual hit or visibility verdicts.-Projection, pixel rounding, stencil alpha, occlusion and pointer delivery remain-runtime acceptance work. The public SDK now exports MaskMode at runtime as well-as in declarations, avoiding an undefined enum for inspector consumers.--- SDK build and all 16 source/declaration API checks pass.-- Editor: 26 layout/i18n/menu checks pass with a single thread worker. An initial- fork-worker run exited unexpectedly; the bounded thread run completed cleanly.- Tests cover nested intersection, contained/partial/full clipping, disabled- masks, rotated/stencil/unresolved masks and unchanged inspection data.-- Browser verification includes a dedicated nested-mask scene. The fixture- removes its first disposable scene through SceneCommands before loading the- second; inspection itself does not alter either authored snapshot.-- Built-in WebGL2: 27/27 checks pass. Manual widening of InnerMask from 200 to- 280 makes the snapshot stale; refresh changes Partial to contained and Outside- to partial. Undo restores the original geometry; no manual edit was saved.-- Editor typecheck/production build and CSS-variable checks pass. The theme- gate retains the existing 18 findings; browser startup still reports missing- desktop MCP bridge and optional DragonBones, as in the prior increment.-- No local desktop/native runner was launched. RM-009 remains In Progress.+## UI Debugger element inspection — 2026-10-09++- Default Elements view provides engine picking, actual hierarchy, shared+ selection and live component values. Text and Focus order are auxiliary views.+ No clipping estimates or automatic input-success verdicts are displayed.+- Screen UI editor picking and outlines use ScreenOverlay; world UI retains its+ camera projection. SDK publishes the shared projection/domain helpers and the+ runtime MaskMode enum. SDK build and all 16 built declaration API checks pass.+- Editor: 29 hierarchy/property/picking/layout/i18n/menu tests pass. Picking tests+ cover cancellation of delayed replies, rejection of non-UI replies and no+ authoring drag. Typecheck and production build pass.+- Built-in WebGL2 fixture: 27/27 checks pass, including real nested ancestry,+ resolved layout, raw Interactable values, native editor picking and unchanged+ authored scene data. Manual pick selects Inside in both hierarchy and Details;+ editing Width 120 to 160 updates computed size without refresh. Undo restores+ 120. Verification edits were not saved.+- The standalone browser lacks the desktop project/play bridge. Running-game+ picking is covered by async host integration tests; actual play acceptance and+ event tracing remain open. Startup MCP/optional DragonBones messages remain.+- No local Electron/native test runner was launched. RM-009 remains In Progress.diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.mdindex 931ee1c1f..3bfde7b92 100644--- a/sdk/etc/index.api.md+++ b/sdk/etc/index.api.md@@ -1,7 +1,7 @@# API surface — esengine/index<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->-Symbols: 115 public · 91 beta · 1547 experimental · 3 deprecated+Symbols: 115 public · 91 beta · 1552 experimental · 3 deprecated## ABI_LAYOUT_HASH — const @experimental```@@ -7475,6 +7475,26 @@ Landscape = "landscape"Portrait = "portrait"```+## ScreenOverlay — const @experimental+```+ResourceDef<ScreenOverlayData>+```++## ScreenOverlayData — interface @experimental+```+active: boolean+layerMask: number+pointerX: number+pointerY: number+projection: Float32Array<ArrayBufferLike>+surfaceH: number+surfaceW: number+vpH: number+vpW: number+vpX: number+vpY: number+```+## ScreenRect — interface @experimental```h: number@@ -11546,6 +11566,11 @@ InputRouter(): boolean```+## isScreenEntity — function @experimental+```+(world: PickableWorld, entity: Entity): boolean+```+## isTextureRef — function @experimental```(v: UniformValue): v is TextureRef@@ -12421,6 +12446,11 @@ Plugin(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay```+## screenToUiLayout — function @experimental+```+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }+```+## screenToUiWorld — function @experimental```(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }@@ -12866,6 +12896,11 @@ UILayoutPlugin(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect```+## uiLayoutToScreen — function @experimental+```+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }+```+## uiMaskPlugin — const @experimental```UIMaskPlugindiff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.mdindex a5b27f766..c6423114c 100644--- a/sdk/etc/index.native.api.md+++ b/sdk/etc/index.native.api.md@@ -1,7 +1,7 @@# API surface — esengine/index.native<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->-Symbols: 115 public · 96 beta · 1566 experimental · 3 deprecated+Symbols: 115 public · 96 beta · 1571 experimental · 3 deprecated## ABI_LAYOUT_HASH — const @experimental```@@ -7676,6 +7676,26 @@ Landscape = "landscape"Portrait = "portrait"```+## ScreenOverlay — const @experimental+```+ResourceDef<ScreenOverlayData>+```++## ScreenOverlayData — interface @experimental+```+active: boolean+layerMask: number+pointerX: number+pointerY: number+projection: Float32Array<ArrayBufferLike>+surfaceH: number+surfaceW: number+vpH: number+vpW: number+vpX: number+vpY: number+```+## ScreenRect — interface @experimental```h: number@@ -11797,6 +11817,11 @@ InputRouter(): boolean```+## isScreenEntity — function @experimental+```+(world: PickableWorld, entity: Entity): boolean+```+## isTextureRef — function @experimental```(v: UniformValue): v is TextureRef@@ -12672,6 +12697,11 @@ Plugin(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay```+## screenToUiLayout — function @experimental+```+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }+```+## screenToUiWorld — function @experimental```(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }@@ -13117,6 +13147,11 @@ UILayoutPlugin(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect```+## uiLayoutToScreen — function @experimental+```+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }+```+## uiMaskPlugin — const @experimental```UIMaskPlugindiff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.mdindex 385982e3d..13b64c056 100644--- a/sdk/etc/index.node.api.md+++ b/sdk/etc/index.node.api.md@@ -1,7 +1,7 @@# API surface — esengine/index.node<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->-Symbols: 115 public · 93 beta · 1546 experimental · 3 deprecated+Symbols: 115 public · 93 beta · 1551 experimental · 3 deprecated## ANCHOR_AXES — const @experimental```@@ -7470,6 +7470,26 @@ Landscape = "landscape"Portrait = "portrait"```+## ScreenOverlay — const @experimental+```+ResourceDef<ScreenOverlayData>+```++## ScreenOverlayData — interface @experimental+```+active: boolean+layerMask: number+pointerX: number+pointerY: number+projection: Float32Array<ArrayBufferLike>+surfaceH: number+surfaceW: number+vpH: number+vpW: number+vpX: number+vpY: number+```+## ScreenRect — interface @experimental```h: number@@ -11541,6 +11561,11 @@ InputRouter(): boolean```+## isScreenEntity — function @experimental+```+(world: PickableWorld, entity: Entity): boolean+```+## isTextureRef — function @experimental```(v: UniformValue): v is TextureRef@@ -12426,6 +12451,11 @@ Plugin(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay```+## screenToUiLayout — function @experimental+```+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }+```+## screenToUiWorld — function @experimental```(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }@@ -12871,6 +12901,11 @@ UILayoutPlugin(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect```+## uiLayoutToScreen — function @experimental+```+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }+```+## uiMaskPlugin — const @experimental```UIMaskPlugindiff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.mdindex c943b9f71..b96d44f1e 100644--- a/sdk/etc/minigame.api.md+++ b/sdk/etc/minigame.api.md@@ -1,7 +1,7 @@# API surface — esengine/minigame<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->-Symbols: 115 public · 91 beta · 1568 experimental · 3 deprecated+Symbols: 115 public · 91 beta · 1573 experimental · 3 deprecated## ANCHOR_AXES — const @experimental```@@ -7743,6 +7743,26 @@ Landscape = "landscape"Portrait = "portrait"```+## ScreenOverlay — const @experimental+```+ResourceDef<ScreenOverlayData>+```++## ScreenOverlayData — interface @experimental+```+active: boolean+layerMask: number+pointerX: number+pointerY: number+projection: Float32Array<ArrayBufferLike>+surfaceH: number+surfaceW: number+vpH: number+vpW: number+vpX: number+vpY: number+```+## ScreenRect — interface @experimental```h: number@@ -11835,6 +11855,11 @@ InputRouter(): boolean```+## isScreenEntity — function @experimental+```+(world: PickableWorld, entity: Entity): boolean+```+## isTextureRef — function @experimental```(v: UniformValue): v is TextureRef@@ -12710,6 +12735,11 @@ Plugin(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay```+## screenToUiLayout — function @experimental+```+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }+```+## screenToUiWorld — function @experimental```(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }@@ -13155,6 +13185,11 @@ UILayoutPlugin(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect```+## uiLayoutToScreen — function @experimental+```+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }+```+## uiMaskPlugin — const @experimental```UIMaskPlugindiff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.mdindex 020da6d94..881887eea 100644--- a/sdk/etc/wechat.api.md+++ b/sdk/etc/wechat.api.md@@ -1,7 +1,7 @@# API surface — esengine/wechat<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->-Symbols: 115 public · 91 beta · 1559 experimental · 3 deprecated+Symbols: 115 public · 91 beta · 1564 experimental · 3 deprecated## ANCHOR_AXES — const @experimental```@@ -7477,6 +7477,26 @@ Landscape = "landscape"Portrait = "portrait"```+## ScreenOverlay — const @experimental+```+ResourceDef<ScreenOverlayData>+```++## ScreenOverlayData — interface @experimental+```+active: boolean+layerMask: number+pointerX: number+pointerY: number+projection: Float32Array<ArrayBufferLike>+surfaceH: number+surfaceW: number+vpH: number+vpW: number+vpX: number+vpY: number+```+## ScreenRect — interface @experimental```h: number@@ -11582,6 +11602,11 @@ InputRouter(): boolean```+## isScreenEntity — function @experimental+```+(world: PickableWorld, entity: Entity): boolean+```+## isTextureRef — function @experimental```(v: UniformValue): v is TextureRef@@ -12457,6 +12482,11 @@ Plugin(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay```+## screenToUiLayout — function @experimental+```+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }+```+## screenToUiWorld — function @experimental```(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }@@ -12902,6 +12932,11 @@ UILayoutPlugin(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect```+## uiLayoutToScreen — function @experimental+```+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }+```+## uiMaskPlugin — const @experimental```UIMaskPlugindiff --git a/sdk/src/core-ui.ts b/sdk/src/core-ui.tsindex 9b172edd8..45e5dc8e3 100644--- a/sdk/src/core-ui.ts+++ b/sdk/src/core-ui.ts@@ -54,6 +54,8 @@ export {// authored data both need the canonical type strings.UIEventType,UICameraInfo,+ ScreenOverlay, type ScreenOverlayData,+ screenToUiLayout, uiLayoutToScreen, isScreenEntity,screenToUiWorld,uiWorldToScreen,uiHitTestWorld,diff --git a/sdk/src/ui/index.ts b/sdk/src/ui/index.tsindex 38590f541..dc5ac592c 100644--- a/sdk/src/ui/index.ts+++ b/sdk/src/ui/index.ts@@ -363,7 +363,7 @@ export {type UICameraData,} from './core/ui-camera-info';-export { screenToUiWorld, uiWorldToScreen, uiHitTestWorld, uiPickWorld, uiPickAllWorld } from './util/ui-pick';+export { screenToUiWorld, uiWorldToScreen, screenToUiLayout, uiLayoutToScreen, isScreenEntity, uiHitTestWorld, uiPickWorld, uiPickAllWorld } from './util/ui-pick';export type { PickableWorld } from './util/ui-pick';export {@@ -520,3 +520,5 @@ export {export { ShelfPacker, type Packer, type PackPos } from '../util/shelfPack';export { sdfFromAlpha } from './text/sdf';export { submitTextBatch, TEXT_VERTEX_FLOATS } from './text/submit';++export { ScreenOverlay, type ScreenOverlayData } from './core/screen-overlay';diff --git a/tools/ui-localization-browser.tsx b/tools/ui-localization-browser.tsxindex 6eb2697b4..fdd97f1c1 100644--- a/tools/ui-localization-browser.tsx+++ b/tools/ui-localization-browser.tsx@@ -4,6 +4,8 @@ import { captureUITextRows, captureUIFocusRows, captureUILayoutRows } from '../dimport { EngineHost } from '../desktop/src/engine/EngineHost';import { SceneModel } from '../desktop/src/engine/SceneModel';import { SceneCommands } from '../desktop/src/engine/SceneCommands';+import { ViewportController } from '../desktop/src/engine/ViewportController';+import { uiDebuggerTree, readUIElement } from '../desktop/src/panels/uiDebuggerData';import { useEditorStore } from '../desktop/src/store/editorStore';import { inspectTextLayout, pseudoLocalize, MaskMode } from 'esengine';@@ -58,13 +60,18 @@ for (let i = 0; i < 12; i++) await new Promise(requestAnimationFrame);const maskBefore = JSON.stringify(SceneModel.serialize());const maskedRows = captureUILayoutRows();check(maskedRows.length === 6, 'Mask scene resolves Canvas, two masks and three buttons');-check(maskedRows.find(row => row.name === 'Inside')?.clip === 'inside', 'Contained button is inside rectangular masks');-check(maskedRows.find(row => row.name === 'Partial')?.clip === 'partial', 'Overflowing button is partially clipped');-check(maskedRows.find(row => row.name === 'Outside')?.clip === 'outside', 'Outside button is fully clipped');-check(maskedRows.find(row => row.name === 'Partial')?.masks.map(mask => mask.name).join(',') === 'InnerMask,OuterMask', 'Nested ancestor mask chain is resolved');-const partial = maskedRows.find(row => row.name === 'Partial');-check(!!partial?.clippedBounds && Math.abs(partial.clippedBounds.right - partial.clippedBounds.left - 60) < .01, 'Partial button retains 60 world units inside the mask');-check(JSON.stringify(SceneModel.serialize()) === maskBefore, 'Mask inspection preserves scene data');+const tree = uiDebuggerTree(SceneModel.serialize());+check(tree.length === 6, 'Debugger hierarchy contains UI nodes and their ancestors');+check(tree.find(row => row.entity.name === 'Inside')?.path === 'Canvas / OuterMask / InnerMask / Inside', 'Debugger preserves the real nested hierarchy');+const inside = maskedRows.find(row => row.name === 'Inside')!;+const detail = readUIElement(EngineHost.world!, inside.entity);+check(detail?.width === 120 && detail.height === 32, 'Selected UI detail reads actual computed layout');+check(detail?.components.some(component => component.type === 'Interactable' && component.data.raycastTarget === true), 'Debugger reads actual input component values without verdicts');+const screenRect = ViewportController.getEntityScreenRect(inside.entity)!;+const canvasRect = EngineHost.canvas!.getBoundingClientRect();+const point = { x: canvasRect.left + screenRect.x + screenRect.w / 2, y: canvasRect.top + screenRect.y + screenRect.h / 2 };+check(ViewportController.pickUIEntity(point.x, point.y) === inside.entity, 'Debugger selection uses the real engine editor picker');+check(JSON.stringify(SceneModel.serialize()) === maskBefore, 'Debugger reads and editor picking preserve scene data');status.textContent += `RESULT ${passed} passed, ${failed} failed · ${EngineHost.activeBackend}`;const dismiss = document.createElement('button');dismiss.textContent = 'Dismiss verification report';