Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat: publish screen UI projection helpers for debugger picking

yhhddcd9c3a609f
diff --git a/desktop b/desktop
index f068cea73..348f995f4 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit f068cea73cb0144fb73047320b7cf90bf4c2e70f
+Subproject commit 348f995f49e933f6985d71a5c8fec902f9692b2c
diff --git a/docs/astro/src/data/apiStability.generated.json b/docs/astro/src/data/apiStability.generated.json
index 364b1c156..0058db530 100644
--- a/docs/astro/src/data/apiStability.generated.json
+++ b/docs/astro/src/data/apiStability.generated.json
@@ -3,10 +3,10 @@
"counts": {
"public": 115,
"beta": 103,
- "experimental": 2150,
+ "experimental": 2168,
"deprecated": 3
},
- "total": 2368,
+ "total": 2386,
"subsystems": [
{
"id": "ecs",
diff --git a/examples/ui-localization-debug/README.md b/examples/ui-localization-debug/README.md
index 1e77af2eb..e0e64713a 100644
--- a/examples/ui-localization-debug/README.md
+++ b/examples/ui-localization-debug/README.md
@@ -1,6 +1,6 @@
# UI localization debug
-Open this project and use Window → UI Inspector → Text. Wait for layout and
+Open this project and use Window → UI Debugger → Text. Wait for layout and
capture again. Original / Accents + 35% / Accents + 100% inspect a snapshot
without editing or saving scene content. Select jumps to the authored entity.
Box and text sizes are local display pixels, before Clip / Ellipsis truncation.
@@ -24,23 +24,18 @@ control, capture again, then Select to find it in Details. The host must expose
resolved visibility, otherwise a warning is shown. This does not verify actual
keyboard delivery, screen readers or IME on a target device.
-Layout and input shows resolved local dimensions and world axis-aligned bounds
-for all UINodes. It reports disabled/hidden controls, ancestor pointer-events
-passthrough, raycast-target configuration and parent masks. Expand a row for
-bounds, then Select to locate its existing viewport handles. Scene edits make
-the snapshot stale and disable selection until capture again.
-
-An input candidate is a configuration result, not a successful pointer hit.
-World bounds do not account for clipping or occlusion; parent masks need review.
-Inspection does not raycast, change hover/focus, or add persistent gizmos.
-
-Masked rows also show their ancestor mask chain (nearest first), mask mode and
-bounds. Unrotated 2D scissor rectangles report estimated containment, partial
-clipping or full clipping, plus the remaining world bounds. Stencil, rotated,
-unresolved and invalid-hierarchy cases require review. This estimate excludes
-camera projection, screen pixel rounding, stencil alpha and actual pointer
-delivery. The browser verification fixture finishes with a dedicated nested
-mask scene containing Inside / Partial / Outside controls.
-
-This is a partial RM-009 delivery. Actual pointer hits, platform shaping,
-real IME/device acceptance and layout cache performance remain separate work.
+UI Debugger opens on Elements. Pick element arms the viewport or running-game
+view for one selection; the selection click is consumed, then normal input
+returns. Escape cancels picking. The hierarchy preserves real UI ancestry and
+selection is shared with Details. The property view reads current component
+values; editor computed layout refreshes while the panel is open. Edit through
+Details. Runtime-only changes revert when play stops.
+
+Screen UI picking and outlines use the renderer's ScreenOverlay projection;
+world UI uses the UI camera. Picking is the engine editor selection query,
+not proof of actual game event delivery. Only one hovered editor element gets
+an extra outline. No clipping estimates or automatic input-success verdicts
+are presented. Text and Focus order remain auxiliary snapshot views.
+
+This is a partial RM-009 delivery. Actual event tracing, platform shaping,
+IME/device acceptance and layout cache performance remain separate work.
diff --git a/examples/ui-localization-debug/VALIDATION.md b/examples/ui-localization-debug/VALIDATION.md
index 6001b0f46..e58e0990f 100644
--- a/examples/ui-localization-debug/VALIDATION.md
+++ b/examples/ui-localization-debug/VALIDATION.md
@@ -89,28 +89,23 @@ snapshot; live-game-only dialogs and scripts need runtime acceptance separately.
platform shaping, IME/accessibility/devices and performance remain open.
No local Electron/native test runner was launched.
-## Ancestor mask inspection increment — 2026-10-08
-
-The inspector shows the ancestor mask chain, mode and bounds. Unrotated 2D
-scissor rectangles estimate contained/partial/fully-clipped geometry; stencil,
-rotation, unavailable bounds and malformed hierarchy explicitly require review.
-These are world-rectangle estimates, not actual hit or visibility verdicts.
-Projection, pixel rounding, stencil alpha, occlusion and pointer delivery remain
-runtime acceptance work. The public SDK now exports MaskMode at runtime as well
-as in declarations, avoiding an undefined enum for inspector consumers.
-
-- SDK build and all 16 source/declaration API checks pass.
-- Editor: 26 layout/i18n/menu checks pass with a single thread worker. An initial
- fork-worker run exited unexpectedly; the bounded thread run completed cleanly.
- Tests cover nested intersection, contained/partial/full clipping, disabled
- masks, rotated/stencil/unresolved masks and unchanged inspection data.
-- Browser verification includes a dedicated nested-mask scene. The fixture
- removes its first disposable scene through SceneCommands before loading the
- second; inspection itself does not alter either authored snapshot.
-- Built-in WebGL2: 27/27 checks pass. Manual widening of InnerMask from 200 to
- 280 makes the snapshot stale; refresh changes Partial to contained and Outside
- to partial. Undo restores the original geometry; no manual edit was saved.
-- Editor typecheck/production build and CSS-variable checks pass. The theme
- gate retains the existing 18 findings; browser startup still reports missing
- desktop MCP bridge and optional DragonBones, as in the prior increment.
-- No local desktop/native runner was launched. RM-009 remains In Progress.
+## UI Debugger element inspection — 2026-10-09
+
+- Default Elements view provides engine picking, actual hierarchy, shared
+ selection and live component values. Text and Focus order are auxiliary views.
+ No clipping estimates or automatic input-success verdicts are displayed.
+- Screen UI editor picking and outlines use ScreenOverlay; world UI retains its
+ camera projection. SDK publishes the shared projection/domain helpers and the
+ runtime MaskMode enum. SDK build and all 16 built declaration API checks pass.
+- Editor: 29 hierarchy/property/picking/layout/i18n/menu tests pass. Picking tests
+ cover cancellation of delayed replies, rejection of non-UI replies and no
+ authoring drag. Typecheck and production build pass.
+- Built-in WebGL2 fixture: 27/27 checks pass, including real nested ancestry,
+ resolved layout, raw Interactable values, native editor picking and unchanged
+ authored scene data. Manual pick selects Inside in both hierarchy and Details;
+ editing Width 120 to 160 updates computed size without refresh. Undo restores
+ 120. Verification edits were not saved.
+- The standalone browser lacks the desktop project/play bridge. Running-game
+ picking is covered by async host integration tests; actual play acceptance and
+ event tracing remain open. Startup MCP/optional DragonBones messages remain.
+- No local Electron/native test runner was launched. RM-009 remains In Progress.
diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.md
index 931ee1c1f..3bfde7b92 100644
--- a/sdk/etc/index.api.md
+++ b/sdk/etc/index.api.md
@@ -1,7 +1,7 @@
# API surface — esengine/index
<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->
-Symbols: 115 public · 91 beta · 1547 experimental · 3 deprecated
+Symbols: 115 public · 91 beta · 1552 experimental · 3 deprecated
## ABI_LAYOUT_HASH — const @experimental
```
@@ -7475,6 +7475,26 @@ Landscape = "landscape"
Portrait = "portrait"
```
+## ScreenOverlay — const @experimental
+```
+ResourceDef<ScreenOverlayData>
+```
+
+## ScreenOverlayData — interface @experimental
+```
+active: boolean
+layerMask: number
+pointerX: number
+pointerY: number
+projection: Float32Array<ArrayBufferLike>
+surfaceH: number
+surfaceW: number
+vpH: number
+vpW: number
+vpX: number
+vpY: number
+```
+
## ScreenRect — interface @experimental
```
h: number
@@ -11546,6 +11566,11 @@ InputRouter
(): boolean
```
+## isScreenEntity — function @experimental
+```
+(world: PickableWorld, entity: Entity): boolean
+```
+
## isTextureRef — function @experimental
```
(v: UniformValue): v is TextureRef
@@ -12421,6 +12446,11 @@ Plugin
(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay
```
+## screenToUiLayout — function @experimental
+```
+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }
+```
+
## screenToUiWorld — function @experimental
```
(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }
@@ -12866,6 +12896,11 @@ UILayoutPlugin
(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect
```
+## uiLayoutToScreen — function @experimental
+```
+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }
+```
+
## uiMaskPlugin — const @experimental
```
UIMaskPlugin
diff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.md
index a5b27f766..c6423114c 100644
--- a/sdk/etc/index.native.api.md
+++ b/sdk/etc/index.native.api.md
@@ -1,7 +1,7 @@
# API surface — esengine/index.native
<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->
-Symbols: 115 public · 96 beta · 1566 experimental · 3 deprecated
+Symbols: 115 public · 96 beta · 1571 experimental · 3 deprecated
## ABI_LAYOUT_HASH — const @experimental
```
@@ -7676,6 +7676,26 @@ Landscape = "landscape"
Portrait = "portrait"
```
+## ScreenOverlay — const @experimental
+```
+ResourceDef<ScreenOverlayData>
+```
+
+## ScreenOverlayData — interface @experimental
+```
+active: boolean
+layerMask: number
+pointerX: number
+pointerY: number
+projection: Float32Array<ArrayBufferLike>
+surfaceH: number
+surfaceW: number
+vpH: number
+vpW: number
+vpX: number
+vpY: number
+```
+
## ScreenRect — interface @experimental
```
h: number
@@ -11797,6 +11817,11 @@ InputRouter
(): boolean
```
+## isScreenEntity — function @experimental
+```
+(world: PickableWorld, entity: Entity): boolean
+```
+
## isTextureRef — function @experimental
```
(v: UniformValue): v is TextureRef
@@ -12672,6 +12697,11 @@ Plugin
(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay
```
+## screenToUiLayout — function @experimental
+```
+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }
+```
+
## screenToUiWorld — function @experimental
```
(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }
@@ -13117,6 +13147,11 @@ UILayoutPlugin
(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect
```
+## uiLayoutToScreen — function @experimental
+```
+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }
+```
+
## uiMaskPlugin — const @experimental
```
UIMaskPlugin
diff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.md
index 385982e3d..13b64c056 100644
--- a/sdk/etc/index.node.api.md
+++ b/sdk/etc/index.node.api.md
@@ -1,7 +1,7 @@
# API surface — esengine/index.node
<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->
-Symbols: 115 public · 93 beta · 1546 experimental · 3 deprecated
+Symbols: 115 public · 93 beta · 1551 experimental · 3 deprecated
## ANCHOR_AXES — const @experimental
```
@@ -7470,6 +7470,26 @@ Landscape = "landscape"
Portrait = "portrait"
```
+## ScreenOverlay — const @experimental
+```
+ResourceDef<ScreenOverlayData>
+```
+
+## ScreenOverlayData — interface @experimental
+```
+active: boolean
+layerMask: number
+pointerX: number
+pointerY: number
+projection: Float32Array<ArrayBufferLike>
+surfaceH: number
+surfaceW: number
+vpH: number
+vpW: number
+vpX: number
+vpY: number
+```
+
## ScreenRect — interface @experimental
```
h: number
@@ -11541,6 +11561,11 @@ InputRouter
(): boolean
```
+## isScreenEntity — function @experimental
+```
+(world: PickableWorld, entity: Entity): boolean
+```
+
## isTextureRef — function @experimental
```
(v: UniformValue): v is TextureRef
@@ -12426,6 +12451,11 @@ Plugin
(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay
```
+## screenToUiLayout — function @experimental
+```
+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }
+```
+
## screenToUiWorld — function @experimental
```
(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }
@@ -12871,6 +12901,11 @@ UILayoutPlugin
(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect
```
+## uiLayoutToScreen — function @experimental
+```
+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }
+```
+
## uiMaskPlugin — const @experimental
```
UIMaskPlugin
diff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.md
index c943b9f71..b96d44f1e 100644
--- a/sdk/etc/minigame.api.md
+++ b/sdk/etc/minigame.api.md
@@ -1,7 +1,7 @@
# API surface — esengine/minigame
<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->
-Symbols: 115 public · 91 beta · 1568 experimental · 3 deprecated
+Symbols: 115 public · 91 beta · 1573 experimental · 3 deprecated
## ANCHOR_AXES — const @experimental
```
@@ -7743,6 +7743,26 @@ Landscape = "landscape"
Portrait = "portrait"
```
+## ScreenOverlay — const @experimental
+```
+ResourceDef<ScreenOverlayData>
+```
+
+## ScreenOverlayData — interface @experimental
+```
+active: boolean
+layerMask: number
+pointerX: number
+pointerY: number
+projection: Float32Array<ArrayBufferLike>
+surfaceH: number
+surfaceW: number
+vpH: number
+vpW: number
+vpX: number
+vpY: number
+```
+
## ScreenRect — interface @experimental
```
h: number
@@ -11835,6 +11855,11 @@ InputRouter
(): boolean
```
+## isScreenEntity — function @experimental
+```
+(world: PickableWorld, entity: Entity): boolean
+```
+
## isTextureRef — function @experimental
```
(v: UniformValue): v is TextureRef
@@ -12710,6 +12735,11 @@ Plugin
(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay
```
+## screenToUiLayout — function @experimental
+```
+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }
+```
+
## screenToUiWorld — function @experimental
```
(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }
@@ -13155,6 +13185,11 @@ UILayoutPlugin
(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect
```
+## uiLayoutToScreen — function @experimental
+```
+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }
+```
+
## uiMaskPlugin — const @experimental
```
UIMaskPlugin
diff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.md
index 020da6d94..881887eea 100644
--- a/sdk/etc/wechat.api.md
+++ b/sdk/etc/wechat.api.md
@@ -1,7 +1,7 @@
# API surface — esengine/wechat
<!-- Generated by tools/api-surface.mjs — do not edit. Run --update to accept changes. -->
-Symbols: 115 public · 91 beta · 1559 experimental · 3 deprecated
+Symbols: 115 public · 91 beta · 1564 experimental · 3 deprecated
## ANCHOR_AXES — const @experimental
```
@@ -7477,6 +7477,26 @@ Landscape = "landscape"
Portrait = "portrait"
```
+## ScreenOverlay — const @experimental
+```
+ResourceDef<ScreenOverlayData>
+```
+
+## ScreenOverlayData — interface @experimental
+```
+active: boolean
+layerMask: number
+pointerX: number
+pointerY: number
+projection: Float32Array<ArrayBufferLike>
+surfaceH: number
+surfaceW: number
+vpH: number
+vpW: number
+vpX: number
+vpY: number
+```
+
## ScreenRect — interface @experimental
```
h: number
@@ -11582,6 +11602,11 @@ InputRouter
(): boolean
```
+## isScreenEntity — function @experimental
+```
+(world: PickableWorld, entity: Entity): boolean
+```
+
## isTextureRef — function @experimental
```
(v: UniformValue): v is TextureRef
@@ -12457,6 +12482,11 @@ Plugin
(screenX: number, screenY: number, inverseVP: Float32Array, vpX: number, vpY: number, vpW: number, vpH: number): WorldRay
```
+## screenToUiLayout — function @experimental
+```
+(overlay: ScreenOverlayData, screenGLX: number, screenGLY: number): { x: number; y: number; }
+```
+
## screenToUiWorld — function @experimental
```
(camera: UICameraData, screenGLX: number, screenGLY: number): { x: number; y: number; }
@@ -12902,6 +12932,11 @@ UILayoutPlugin
(cam: CameraExtents, fit: CanvasScale | null, width: number, height: number, previewAspect?: number): WorldRect
```
+## uiLayoutToScreen — function @experimental
+```
+(overlay: ScreenOverlayData, x: number, y: number): { x: number; y: number; }
+```
+
## uiMaskPlugin — const @experimental
```
UIMaskPlugin
diff --git a/sdk/src/core-ui.ts b/sdk/src/core-ui.ts
index 9b172edd8..45e5dc8e3 100644
--- a/sdk/src/core-ui.ts
+++ b/sdk/src/core-ui.ts
@@ -54,6 +54,8 @@ export {
// authored data both need the canonical type strings.
UIEventType,
UICameraInfo,
+ ScreenOverlay, type ScreenOverlayData,
+ screenToUiLayout, uiLayoutToScreen, isScreenEntity,
screenToUiWorld,
uiWorldToScreen,
uiHitTestWorld,
diff --git a/sdk/src/ui/index.ts b/sdk/src/ui/index.ts
index 38590f541..dc5ac592c 100644
--- a/sdk/src/ui/index.ts
+++ b/sdk/src/ui/index.ts
@@ -363,7 +363,7 @@ export {
type UICameraData,
} from './core/ui-camera-info';
-export { screenToUiWorld, uiWorldToScreen, uiHitTestWorld, uiPickWorld, uiPickAllWorld } from './util/ui-pick';
+export { screenToUiWorld, uiWorldToScreen, screenToUiLayout, uiLayoutToScreen, isScreenEntity, uiHitTestWorld, uiPickWorld, uiPickAllWorld } from './util/ui-pick';
export type { PickableWorld } from './util/ui-pick';
export {
@@ -520,3 +520,5 @@ export {
export { ShelfPacker, type Packer, type PackPos } from '../util/shelfPack';
export { sdfFromAlpha } from './text/sdf';
export { submitTextBatch, TEXT_VERTEX_FLOATS } from './text/submit';
+
+export { ScreenOverlay, type ScreenOverlayData } from './core/screen-overlay';
diff --git a/tools/ui-localization-browser.tsx b/tools/ui-localization-browser.tsx
index 6eb2697b4..fdd97f1c1 100644
--- a/tools/ui-localization-browser.tsx
+++ b/tools/ui-localization-browser.tsx
@@ -4,6 +4,8 @@ import { captureUITextRows, captureUIFocusRows, captureUILayoutRows } from '../d
import { EngineHost } from '../desktop/src/engine/EngineHost';
import { SceneModel } from '../desktop/src/engine/SceneModel';
import { SceneCommands } from '../desktop/src/engine/SceneCommands';
+import { ViewportController } from '../desktop/src/engine/ViewportController';
+import { uiDebuggerTree, readUIElement } from '../desktop/src/panels/uiDebuggerData';
import { useEditorStore } from '../desktop/src/store/editorStore';
import { inspectTextLayout, pseudoLocalize, MaskMode } from 'esengine';
@@ -58,13 +60,18 @@ for (let i = 0; i < 12; i++) await new Promise(requestAnimationFrame);
const maskBefore = JSON.stringify(SceneModel.serialize());
const maskedRows = captureUILayoutRows();
check(maskedRows.length === 6, 'Mask scene resolves Canvas, two masks and three buttons');
-check(maskedRows.find(row => row.name === 'Inside')?.clip === 'inside', 'Contained button is inside rectangular masks');
-check(maskedRows.find(row => row.name === 'Partial')?.clip === 'partial', 'Overflowing button is partially clipped');
-check(maskedRows.find(row => row.name === 'Outside')?.clip === 'outside', 'Outside button is fully clipped');
-check(maskedRows.find(row => row.name === 'Partial')?.masks.map(mask => mask.name).join(',') === 'InnerMask,OuterMask', 'Nested ancestor mask chain is resolved');
-const partial = maskedRows.find(row => row.name === 'Partial');
-check(!!partial?.clippedBounds && Math.abs(partial.clippedBounds.right - partial.clippedBounds.left - 60) < .01, 'Partial button retains 60 world units inside the mask');
-check(JSON.stringify(SceneModel.serialize()) === maskBefore, 'Mask inspection preserves scene data');
+const tree = uiDebuggerTree(SceneModel.serialize());
+check(tree.length === 6, 'Debugger hierarchy contains UI nodes and their ancestors');
+check(tree.find(row => row.entity.name === 'Inside')?.path === 'Canvas / OuterMask / InnerMask / Inside', 'Debugger preserves the real nested hierarchy');
+const inside = maskedRows.find(row => row.name === 'Inside')!;
+const detail = readUIElement(EngineHost.world!, inside.entity);
+check(detail?.width === 120 && detail.height === 32, 'Selected UI detail reads actual computed layout');
+check(detail?.components.some(component => component.type === 'Interactable' && component.data.raycastTarget === true), 'Debugger reads actual input component values without verdicts');
+const screenRect = ViewportController.getEntityScreenRect(inside.entity)!;
+const canvasRect = EngineHost.canvas!.getBoundingClientRect();
+const point = { x: canvasRect.left + screenRect.x + screenRect.w / 2, y: canvasRect.top + screenRect.y + screenRect.h / 2 };
+check(ViewportController.pickUIEntity(point.x, point.y) === inside.entity, 'Debugger selection uses the real engine editor picker');
+check(JSON.stringify(SceneModel.serialize()) === maskBefore, 'Debugger reads and editor picking preserve scene data');
status.textContent += `RESULT ${passed} passed, ${failed} failed · ${EngineHost.activeBackend}`;
const dismiss = document.createElement('button');
dismiss.textContent = 'Dismiss verification report';