A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(ui): a held pointer reports its moves (pointer_move) and a given-up hold (press_cancel), every pointer event says where the pointer is, and a ContentRect-only entity can be pressed; the Cocos touches translate the engine's pointer events instead of dispatching on their own
yhhce5e97744520
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d9e650c61..290938989 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -20,6 +20,7 @@ published separately; it ships inside the editor.- **Scroll areas can be pulled past an end and spring back.** A scroll area whose movement is Elastic now does what the setting always said: a drag past an end is stiffer the further out it goes, letting go returns the content, and a flick that reaches an end bounces off it. It is opt-in per area (`ScrollMovement.Elastic`; Clamped, the default, behaves as before), and the gesture model (`ScrollMotion`, with `ScrollContainer.dragTo` and `getOverscroll`) is exported for scrolling anything else the same way.- **More easing curves.** Sine, quart, quint, expo and circ (each in, out and in-out), bounce in and in-out, and smoothstep and smootherstep join the quad, cubic, back, elastic and out-bounce curves. They are new `EasingType` members after the existing ones, so a saved tween keeps its curve, and they ease the same in a C++ tween and a script value tween.- **World content has a rectangle: `ContentRect`.** A size and the pivot the entity's transform sits on, which the `Sprite` and `Text` on the entity follow and which a `UIMask` on it clips to — no `UINode` needed, so a clipping region or a scroll viewport can stand in the world. `contentRectContains` tests a world point against it.+- **A held pointer reports where it goes, and a hold can be given up.** `pointer_move` goes to the entity a pointer is holding for as long as it is held, `press_cancel` to one whose hold was taken away (`UIControlPointers.cancelHold`, optionally handing the gesture to another entity, so a scroll area can take a drag from the button under it), and every pointer event now says which pointer it was and where, in the world and on the surface. An entity that is only a `ContentRect` with an `Interactable` can be the one pressed.- **The engine's runtime tuning is a project setting.** Project Settings → Game → Runtime sets how a scene switch fades (duration and colour), how far one frame may advance the game and how many fixed steps it may catch up, how long an asset may take to load and how soon a failed one is tried again, how much memory the texture, audio and device-loss caches keep, the text canvas size and the default font family. They were reachable only from code (`RuntimeConfig`) and no project file could carry them. The project holds `features.runtime` with only what differs from the engine's own values (`RUNTIME_BUILD_DEFAULTS`), a build ships the same block, and `initRuntime` applies it before the first asset loads on the web, desktop, native, mini-game and playable targets and in Play. The edit viewport keeps the engine's values.- **Rows for the choices that could only be written into the project file.** Under Rendering: compile shaders in the background, warm them up behind the loading screen and how long to wait, draw only changing frames and how many frames to keep after a change, each level's anti-aliasing (MSAA or temporal; the MSAA count is hidden where temporal ignores it) and its depth pre-pass. Under Game → Display: the render resolution (screen, design resolution or integer multiple). Under the web build: the servers the page may connect to, each checked by the parser the export uses. `check-project-settings` now reads the keys of `QualityConfig`, `QualityProfile` and `RuntimeBuildConfig` and fails a key that has no row and no stated reason, which is how these went unnoticed behind the one row the block had.diff --git a/docs/astro/src/content/docs/scripting/events.mdx b/docs/astro/src/content/docs/scripting/events.mdxindex 5559d88c1..8d021cd08 100644--- a/docs/astro/src/content/docs/scripting/events.mdx+++ b/docs/astro/src/content/docs/scripting/events.mdx@@ -134,7 +134,14 @@ hears its buttons:`click` · `press` · `release` · `hover_enter` · `hover_exit` · `focus` · `blur` ·`change` · `submit` · `drag_start` · `drag_move` · `drag_end` · `scroll` · `select` · `deselect` ·-`shown` · `hidden`+`pointer_move` · `press_cancel` · `shown` · `hidden`++Every pointer event (`press`, `release`, `click`, `pointer_move`, `press_cancel`) carries a `UiPointerEventData`:+which pointer (`pointer`, -1 for the mouse), where it is in the world (`worldX`, `worldY`) and on the surface+(`screenX`, `screenY`, counted up from the bottom-left). The entity that was pressed is the one that hears+`pointer_move` for as long as the pointer is held, wherever it goes. A hold can be given up with+`UIControlPointers.cancelHold(pointer, to?)`: the holder hears `press_cancel`, nothing clicks, and with `to` another entity+takes the rest of the gesture. An entity that is only a `ContentRect` plus an `Interactable` can be pressed.`shown` and `hidden` are the exception to "a widget emits them": nobody clicked anything, thenode simply arrived on screen or left it — usually because an **ancestor** was shown or hidden,diff --git a/docs/astro/src/content/docs/zh-cn/scripting/events.mdx b/docs/astro/src/content/docs/zh-cn/scripting/events.mdxindex 6922e43ac..d6e6122ad 100644--- a/docs/astro/src/content/docs/zh-cn/scripting/events.mdx+++ b/docs/astro/src/content/docs/zh-cn/scripting/events.mdx@@ -128,7 +128,13 @@ registerAction('game.award', {`click` · `press` · `release` · `hover_enter` · `hover_exit` · `focus` · `blur` ·`change` · `submit` · `drag_start` · `drag_move` · `drag_end` · `scroll` · `select` · `deselect` ·-`shown` · `hidden`+`pointer_move` · `press_cancel` · `shown` · `hidden`++每个指针事件(`press`、`release`、`click`、`pointer_move`、`press_cancel`)都带一个 `UiPointerEventData`:+是哪个指针(`pointer`,鼠标是 -1)、它在世界里的位置(`worldX`、`worldY`)和在表面上的位置(`screenX`、+`screenY`,从左下角向上数)。被按下的那个实体,在指针按住期间无论移到哪里,都由它收到 `pointer_move`。+一次按住可以放弃:`UIControlPointers.cancelHold(pointer, to?)`,持有者收到 `press_cancel`,不会触发点击;+给了 `to`,这次手势的其余部分就交给那个实体。只有 `ContentRect` 加 `Interactable` 的实体也可以被按下。`shown` 和 `hidden` 是「控件发出的」里的例外:没人点任何东西,是这个节点出现在了屏幕上、或者离开了屏幕——通常因为**某个祖先**被显示或隐藏,也就是导航、页签、弹窗在做的事。子树里的每个节点都会被告知,diff --git a/sdk/src/cocos/index.ts b/sdk/src/cocos/index.tsindex 56e920554..40809cf93 100644--- a/sdk/src/cocos/index.ts+++ b/sdk/src/cocos/index.ts@@ -27,7 +27,7 @@ export {} from './assets';export { registerBuiltinComponent, type BuiltinAdapter } from './builtins';export { UITransform, Label, LabelOutline, LabelShadow, Sprite, UIOpacity, Button, EventHandler, Widget, Layout, Camera, RichText, Mask, Toggle, ToggleContainer, ProgressBar, Slider, BlockInputEvents, ScrollView, getVisibleSize, setVisibleSize } from './ui';-export { Touch, EventTouch, pumpTouches } from './input';+export { Touch, EventTouch } from './input';export { tween, Tween, tweenUtil, easing, updateTweens } from './tween';export { instantiate, loadPrefab, providePrefab } from './prefabs';export { sys } from './sys';diff --git a/sdk/src/cocos/input.ts b/sdk/src/cocos/input.tsindex 4488912f9..4a033c7b2 100644--- a/sdk/src/cocos/input.ts+++ b/sdk/src/cocos/input.ts@@ -2,30 +2,34 @@// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team/*** @file cocos/input.ts- * @brief Touch events on nodes: `Node.on(TOUCH_START, ...)`, `EventTouch`, and the system that raises them.+ * @brief Touch events on nodes: `Node.on(TOUCH_START, ...)`, `EventTouch`, and the translation from the engine's pointer events.*- * The engine already turns the mouse and fingers into one list of pointers each frame (`uiPointersOf`) and knows the camera that- * maps a screen point to the world. What Cocos adds is who gets told: a touch is claimed at its start by the TOPMOST node that- * listens for touches and whose rectangle holds the point, the rest of the gesture (move, end, cancel) goes to that same node- * wherever the finger has gone, and the event then bubbles up through the node's ancestors, each of which hears it without being- * hit-tested, until one stops it.+ * The engine already decides who a pointer is on (the topmost interactable under it), lets that entity hold the gesture, and says+ * what happens to it: `press`, `pointer_move`, `release`, `press_cancel`, each with where the pointer is. A node that listens for+ * touches is made interactable, and those events are handed to it here as Cocos's: the touch starts on the topmost listening node,+ * the rest of the gesture goes to that same node wherever the finger goes, and the event travels the node's ancestors — the+ * capture listeners from the top down, then the target's own, then the bubbling ones back up — until one stops it.*- * A touch's location is a world point. In an imported scene the engine's world and Cocos's are the same space (the canvas sits where- * it sat, the camera is where it was), so a script's `getUILocation()` and `convertToNodeSpaceAR` mean what they meant.+ * Because it is the engine's pointer, a touch on a Cocos node also counts as "over the UI" for gameplay input, and anything the+ * engine's own UI does with pointers (hover, focus) sees these nodes too.+ *+ * A touch has two places, as in Cocos: the SCREEN point (`getLocation`, the surface in pixels from its bottom-left corner — what a+ * camera's `screenToWorld` takes) and the CANVAS point (`getUILocation`, where the UI is laid out; in an imported scene, the world).*/import { Schedule, defineSystem, addSystemToSchedule, GetWorld } from '../ecs/system';import { Res } from '../ecs/resource';import { Input, type InputState } from '../input/input';-import { uiPointersOf } from '../ui/input/pointerBook';-import { UICameraInfo, type UICameraData } from '../ui/core/ui-camera-info';-import { screenToUiWorld } from '../ui/util/ui-pick';-import { platformDevicePixelRatio } from '../platform';+import { EntityEvents, type EntityEvent, type EntityEventQueue } from '../ecs/entityEvents';+import { UIEventType } from '../ui/core/events';+import { UIControlPointers, type ControlPointer, type UIControlPointersData, type UiPointerEventData } from '../ui/input/control-pointers';+import { Interactable } from '../ui/input/interactable';+import { MOUSE_POINTER, uiPointersOf } from '../ui/input/pointerBook';+import { SystemLabel } from '../ecs/systemLabels';import type { World } from '../ecs/world';import type { Entity } from '../types';-import { Children, Parent } from '../ecs/component';import { Vec2 } from './math';import { Node, touchListeningEntities } from './node';-import { UITransform } from './ui/uiTransform';+import { ensureUITransform } from './ui/uiTransform';import { log } from '../util/logger';import { EventKeyboard, EventMouse, input as globalInput, keyCodeOf, Input as InputApi } from './inputApi';@@ -106,61 +110,47 @@ export class EventTouch {}// ----------------------------------------------------------------------------------------------------------------------// Which node is on top.+// Making listening nodes interactable.// ----------------------------------------------------------------------------------------------------------------------/** Where a node stands in draw order, as the sibling indices from the root: a later sibling, or a descendant, is above. */-function orderKey(world: World, entity: Entity): number[] {- const path: number[] = [];- let current: Entity = entity;- for (;;) {- const parent: Entity | undefined = world.tryGet(current, Parent)?.entity;- if (!parent) { path.push(Number(current)); break; }- path.push(world.tryGet(parent, Children)?.entities.indexOf(current) ?? 0);- current = parent;- }- return path.reverse();-}--/** Higher draw order first. */-function compareTopmost(a: number[], b: number[]): number {- const n = Math.min(a.length, b.length);- for (let i = 0; i < n; i++) if (a[i] !== b[i]) return b[i]! - a[i]!;- return b.length - a.length; // a descendant is above its ancestor-}--const TOUCH_TYPES = [Node.EventType.TOUCH_START, Node.EventType.TOUCH_MOVE, Node.EventType.TOUCH_END, Node.EventType.TOUCH_CANCEL];--function listensForTouch(node: Node): boolean {- for (const t of TOUCH_TYPES) if (node.hasEventListener(t)) return true;- return false;-}+/** The entities this layer has made interactable, per world, so it can take the mark away again from ones that stopped listening. */+const equipped = new WeakMap<World, Set<Entity>>();-/** The topmost active node that listens for touches and holds @p point, or null. */-function hitNode(world: World, point: Vec2): Node | null {- const hits: Array<{ node: Node; key: number[] }> = [];- for (const entity of touchListeningEntities(world)) {- if (!world.valid(entity)) continue;- const node = Node.for(world, entity);- if (!listensForTouch(node) || !node.activeInHierarchy) continue;- const transform = node.getComponent(UITransform);- if (!transform || !transform.isHit(point)) continue;- hits.push({ node, key: orderKey(world, entity) });+/**+ * A node that listens for touches is a thing a pointer can land on: it gets a rectangle (the node's `UITransform`) and the engine's+ * `Interactable`, which lets events pass through to its ancestors (the bubbling here is the node tree's, not the engine's). A node+ * that stops listening gives the mark back, so it does not keep catching touches that belong to what is under it.+ */+function syncInteractables(world: World): void {+ const have = equipped.get(world) ?? new Set<Entity>();+ equipped.set(world, have);+ const want = new Set<Entity>();+ for (const entity of touchListeningEntities(world)) if (world.valid(entity)) want.add(entity);+ for (const entity of want) {+ if (have.has(entity)) continue;+ ensureUITransform(world, entity);+ if (!world.has(entity, Interactable)) world.insert(entity, Interactable, { enabled: true, blockRaycast: false, raycastTarget: true });+ have.add(entity);+ }+ for (const entity of [...have]) {+ if (want.has(entity)) continue;+ have.delete(entity);+ if (world.valid(entity) && world.has(entity, Interactable)) world.remove(entity, Interactable);}- if (hits.length === 0) return null;- hits.sort((a, b) => compareTopmost(a.key, b.key));- return hits[0]!.node;}+// ---------------------------------------------------------------------------------------------------------------------+// Delivering a touch: capture down, target, bubble up.+// ---------------------------------------------------------------------------------------------------------------------+/**- * Hand an event down and up the tree: capture listeners of the ancestors from the top, then the target's own, then the normal- * listeners from the target up through its ancestors, until one stops it. @p chainFrom is the node the capture path is read from,- * which differs from @p from once a capturing ancestor has cancelled the target.+ * Hand an event down and up the node tree: the capture listeners of the ancestors from the top, then the target's own capture and+ * normal listeners, then the normal listeners from the target's parent up, until one stops it.*/-function dispatch(event: EventTouch, from: Node | null, chainFrom: Node | null = from): void {+function dispatch(event: EventTouch, target: Node): void {const path: Node[] = [];- for (let n: Node | null = chainFrom; n; n = n.parent) path.unshift(n);- event.target = from ?? chainFrom;+ for (let n: Node | null = target; n; n = n.parent) path.unshift(n);+ event.target = target;const guard = (fn: () => void, node: Node): void => {try { fn(); } catch (err) { log.error('cocos', `${node.name} ${event.type} listener threw`, err); }};@@ -169,75 +159,87 @@ function dispatch(event: EventTouch, from: Node | null, chainFrom: Node | null =event.currentTarget = node;guard(() => node._emitCapture(event.type, event), node);}- for (let node: Node | null = from; node && !event.propagationStopped; node = node.parent) {+ for (let node: Node | null = target; node && !event.propagationStopped; node = node.parent) {event.currentTarget = node;if (!node.hasEventListener(event.type, false)) continue;guard(() => node!.emit(event.type, event), node);}}-// ----------------------------------------------------------------------------------------------------------------------// The per-frame system.-// -----------------------------------------------------------------------------------------------------------------------interface Gesture { touch: Touch; target: Node | null; chainFrom: Node | null }-const gestures = new WeakMap<World, Map<number, Gesture>>();--function gesturesOf(world: World): Map<number, Gesture> {- let g = gestures.get(world);- if (!g) gestures.set(world, (g = new Map()));- return g;-}--/** Give an event the means to cancel the node that claimed its gesture, without bubbling the cancellation back to the canceller. */-function withCancel(event: EventTouch, gesture: Gesture, live: Touch[]): EventTouch {+const TO_COCOS: Record<string, string> = {+ [UIEventType.Press]: Node.EventType.TOUCH_START,+ [UIEventType.PointerMove]: Node.EventType.TOUCH_MOVE,+ [UIEventType.Release]: Node.EventType.TOUCH_END,+ [UIEventType.PressCancel]: Node.EventType.TOUCH_CANCEL,+};++interface TouchState { nodeTouches: Map<number, Touch>; globalTouches: Map<number, Touch>; subscribed: WeakSet<object> }+const states = new WeakMap<World, TouchState>();+const stateOf = (world: World): TouchState => {+ let s = states.get(world);+ if (!s) states.set(world, (s = { nodeTouches: new Map(), globalTouches: new Map(), subscribed: new WeakSet() }));+ return s;+};++/** Turn one of the engine's pointer events on an entity into the Cocos touch event for its node. */+function onPointerEvent(world: World, controls: UIControlPointersData, e: EntityEvent): void {+ // an engine event bubbles to interactable ancestors as copies; the node tree does its own bubbling, from the original only+ if (e.currentTarget !== e.target) return;+ const type = TO_COCOS[e.type];+ const data = e.data as UiPointerEventData | undefined;+ if (!type || !data || !world.valid(e.target)) return;+ const { nodeTouches } = stateOf(world);+ let touch = nodeTouches.get(data.pointer);+ if (!touch || type === Node.EventType.TOUCH_START) {+ touch = new Touch(data.pointer, data.worldX, data.worldY, data.screenX, data.screenY);+ nodeTouches.set(data.pointer, touch);+ } else {+ touch._move(data.worldX, data.worldY, data.screenX, data.screenY);+ }+ const event = new EventTouch(type, touch, [...nodeTouches.values()]);+ // giving the gesture to the node that is cancelling it (the one whose capture listener is running)event._cancelTarget = () => {- const target = gesture.target;- if (!target) return;- gesture.target = null;- const cancel = new EventTouch(Node.EventType.TOUCH_CANCEL, gesture.touch, live);- cancel.target = target;- cancel.currentTarget = target;- try { target.emit(Node.EventType.TOUCH_CANCEL, cancel); } catch (err) { log.error('cocos', `${target.name} cancel listener threw`, err); }+ const taker = event.currentTarget;+ if (taker && taker !== event.target) controls.cancelHold(data.pointer, taker._entity);};- return event;+ dispatch(event, Node.for(world, e.target));+ if (event.propagationStopped) e.stopPropagation();+ if (type === Node.EventType.TOUCH_END || type === Node.EventType.TOUCH_CANCEL) {+ // a cancel that hands the gesture to another node does not end the touch+ if (!controls.pointers.get(data.pointer)?.down || type === Node.EventType.TOUCH_END) nodeTouches.delete(data.pointer);+ }}-/** Turn this frame's pointers into TOUCH_START / MOVE / END events. Exported so a test (or a host with its own loop) can drive it. */-export function pumpTouches(world: World, input: InputState, camera: UICameraData): void {- const book = gesturesOf(world);- const dpr = platformDevicePixelRatio();- const surfaceH = camera.screenH;- const pointers = uiPointersOf(input);- const live = [...book.values()].map((g) => g.touch);-- for (const p of pointers) {- const sx = p.x * dpr, sy = surfaceH - p.y * dpr;- const at = camera.valid ? screenToUiWorld(camera, sx, sy) : { x: 0, y: 0 };- let gesture = book.get(p.id);+function subscribe(world: World, events: EntityEventQueue, controls: UIControlPointersData): void {+ const state = stateOf(world);+ if (state.subscribed.has(events)) return;+ state.subscribed.add(events);+ for (const type of Object.keys(TO_COCOS)) events.on(type, (e) => onPointerEvent(world, controls, e));+}- if (p.pressed && !gesture) {- const touch = new Touch(p.id, at.x, at.y, sx, sy);- const hit = hitNode(world, new Vec2(at.x, at.y));- gesture = { touch, target: hit, chainFrom: hit };- book.set(p.id, gesture);- live.push(touch);- globalInput.emit(Node.EventType.TOUCH_START, new EventTouch(Node.EventType.TOUCH_START, touch, live));- if (gesture.chainFrom) dispatch(withCancel(new EventTouch(Node.EventType.TOUCH_START, touch, live), gesture, live), gesture.target, gesture.chainFrom);+/** The global `input` hears every pointer, whether or not a node is under it. */+function pumpGlobalTouches(world: World, input: InputState, controls: UIControlPointersData): void {+ const { globalTouches } = stateOf(world);+ for (const p of uiPointersOf(input)) {+ const c = controls.pointers.get(p.id);+ if (!c) continue;+ let touch = globalTouches.get(p.id);+ const live = (): Touch[] => [...globalTouches.values()];+ if (p.pressed && !touch) {+ touch = new Touch(p.id, c.worldX, c.worldY, c.screenX, c.screenY);+ globalTouches.set(p.id, touch);+ globalInput.emit(Node.EventType.TOUCH_START, new EventTouch(Node.EventType.TOUCH_START, touch, live()));}- if (!gesture) continue;-- if (p.down && (at.x !== gesture.touch.getLocationX() || at.y !== gesture.touch.getLocationY())) {- gesture.touch._move(at.x, at.y, sx, sy);- globalInput.emit(Node.EventType.TOUCH_MOVE, new EventTouch(Node.EventType.TOUCH_MOVE, gesture.touch, live));- if (gesture.chainFrom) dispatch(withCancel(new EventTouch(Node.EventType.TOUCH_MOVE, gesture.touch, live), gesture, live), gesture.target, gesture.chainFrom);+ if (!touch) continue;+ if (p.down && (c.worldX !== touch.getUILocationX() || c.worldY !== touch.getUILocationY())) {+ touch._move(c.worldX, c.worldY, c.screenX, c.screenY);+ globalInput.emit(Node.EventType.TOUCH_MOVE, new EventTouch(Node.EventType.TOUCH_MOVE, touch, live()));}if (p.released) {- gesture.touch._move(at.x, at.y, sx, sy);+ touch._move(c.worldX, c.worldY, c.screenX, c.screenY);const type = p.cancelled ? Node.EventType.TOUCH_CANCEL : Node.EventType.TOUCH_END;- globalInput.emit(type, new EventTouch(type, gesture.touch, live));- if (gesture.chainFrom) dispatch(withCancel(new EventTouch(type, gesture.touch, live), gesture, live), gesture.target, gesture.chainFrom);- book.delete(p.id);+ globalInput.emit(type, new EventTouch(type, touch, live()));+ globalTouches.delete(p.id);}}}@@ -249,22 +251,23 @@ export function pumpTouches(world: World, input: InputState, camera: UICameraDat/** Where each input state's mouse was last frame, so a move is a change and the first frame is not one. */const lastMouseOf = new WeakMap<InputState, { x: number; y: number }>();-/** Turn this frame's key, button and wheel state into events on the global `input`. Touches are `pumpTouches`'s. */-export function pumpKeysAndMouse(input: InputState, camera: UICameraData): void {+/**+ * Turn this frame's key, button and wheel state into events on the global `input`. @p mouse is the engine's resolved mouse pointer+ * (null when there is none, as on a touch device), which is where the position comes from.+ */+export function pumpKeysAndMouse(input: InputState, mouse: ControlPointer | null): void {for (const code of input.keysPressed) globalInput.emit(InputApi.EventType.KEY_DOWN, new EventKeyboard(keyCodeOf(code), InputApi.EventType.KEY_DOWN));for (const code of input.keysDown) {if (!input.keysPressed.has(code)) globalInput.emit(InputApi.EventType.KEY_PRESSING, new EventKeyboard(keyCodeOf(code), InputApi.EventType.KEY_PRESSING));}for (const code of input.keysReleased) globalInput.emit(InputApi.EventType.KEY_UP, new EventKeyboard(keyCodeOf(code), InputApi.EventType.KEY_UP));- const dpr = platformDevicePixelRatio();- const sx = input.mouseX * dpr, sy = camera.screenH - input.mouseY * dpr;- const at = camera.valid ? screenToUiWorld(camera, sx, sy) : { x: 0, y: 0 };+ const at = mouse ? { x: mouse.worldX, y: mouse.worldY, sx: mouse.screenX, sy: mouse.screenY } : { x: 0, y: 0, sx: 0, sy: 0 };const lastMouse = lastMouseOf.get(input);const previous = lastMouse ?? at;const moved = lastMouse !== undefined && (at.x !== lastMouse.x || at.y !== lastMouse.y);const prev = new Vec2(previous.x, previous.y);- const event = (type: string): EventMouse => new EventMouse(type, at.x, at.y, prev, sx, sy);+ const event = (type: string): EventMouse => new EventMouse(type, at.x, at.y, prev, at.sx, at.sy);for (const b of input.mouseButtonsPressed) globalInput.emit(InputApi.EventType.MOUSE_DOWN, event(InputApi.EventType.MOUSE_DOWN)._button(b));if (moved) globalInput.emit(InputApi.EventType.MOUSE_MOVE, event(InputApi.EventType.MOUSE_MOVE));for (const b of input.mouseButtonsReleased) globalInput.emit(InputApi.EventType.MOUSE_UP, event(InputApi.EventType.MOUSE_UP)._button(b));@@ -275,12 +278,27 @@ export function pumpKeysAndMouse(input: InputState, camera: UICameraData): voidlastMouseOf.set(input, { x: at.x, y: at.y });}-/** Register the touch system with the engine's frame loop. Idempotent per context is the caller's business: see `declareBuiltinScripts`. */+/**+ * Register the systems that wire Cocos touches to the engine's pointer: one before the engine's UI interaction pass that makes+ * listening nodes interactable and subscribes to its events, one after it that feeds the global `input`.+ */export function declareInputSystem(): void {- const system = defineSystem(- [Res(Input), Res(UICameraInfo), GetWorld()],- (input, camera, world) => { pumpTouches(world, input, camera); pumpKeysAndMouse(input, camera); },- { name: 'Cocos:Input', touches: { opaque: true } },- );- addSystemToSchedule(Schedule.PreUpdate, system);+ addSystemToSchedule(Schedule.PreUpdate, defineSystem(+ [Res(EntityEvents), Res(UIControlPointers), GetWorld()],+ (events, controls, world) => {+ syncInteractables(world);+ // subscribed before the interaction pass, so the very first press already reaches a node+ if (events) subscribe(world, events as EntityEventQueue, controls);+ },+ { name: 'Cocos:Interactables', touches: { opaque: true }, runBefore: [SystemLabel.UIInteraction] },+ ));+ addSystemToSchedule(Schedule.PreUpdate, defineSystem(+ [Res(Input), Res(EntityEvents), Res(UIControlPointers), GetWorld()],+ (input, events, controls, world) => {+ if (!events) return; // no UI pipeline: nothing resolves a pointer+ pumpGlobalTouches(world, input, controls);+ pumpKeysAndMouse(input, controls.pointers.get(MOUSE_POINTER) ?? null);+ },+ { name: 'Cocos:Input', touches: { opaque: true }, runAfter: [SystemLabel.UIInteraction] },+ ));}diff --git a/sdk/src/cocos/node.ts b/sdk/src/cocos/node.tsindex e4c203040..3dfe1f63c 100644--- a/sdk/src/cocos/node.ts+++ b/sdk/src/cocos/node.ts@@ -345,6 +345,7 @@ export class Node {const list = map?.get(type);if (!list) return;map!.set(type, list.filter((l) => (callback && l.callback !== callback) || (target !== undefined && l.target !== target)));+ this.forgetTouchIfSilent_();}/** Remove every listener that was registered with @p target. */targetOff(target: unknown): void {@@ -352,6 +353,18 @@ export class Node {if (!map) continue;for (const [type, list] of map) map.set(type, list.filter((l) => l.target !== target));}+ this.forgetTouchIfSilent_();+ }++ /** An entity with no touch listener left is not one a pointer can land on any more. */+ private forgetTouchIfSilent_(): void {+ const set = touchListeners.get(this._world);+ if (!set?.has(this._entity)) return;+ for (const map of [this.listeners_, this.captureListeners_]) {+ if (!map) continue;+ for (const [type, list] of map) if (type.startsWith('touch-') && list.length > 0) return;+ }+ set.delete(this._entity);}hasEventListener(type: string, useCapture?: boolean): boolean {if (useCapture === true) return (this.captureListeners_?.get(type)?.length ?? 0) > 0;diff --git a/sdk/src/ui/core/events.ts b/sdk/src/ui/core/events.tsindex 91e785f75..822e36a5f 100644--- a/sdk/src/ui/core/events.ts+++ b/sdk/src/ui/core/events.ts@@ -28,6 +28,10 @@ export const UIEventType = {Click: 'click',Press: 'press',Release: 'release',+ /** A pointer holding this entity moved. Carries a {@link UiPointerEventData}; bubbles like a press. */+ PointerMove: 'pointer_move',+ /** A pointer's hold on this entity ended without a release: the system cancelled it, or something took the gesture over. */+ PressCancel: 'press_cancel',HoverEnter: 'hover_enter',HoverExit: 'hover_exit',Focus: 'focus',diff --git a/sdk/src/ui/input/control-pointers.ts b/sdk/src/ui/input/control-pointers.tsindex a44399a74..22ce060bd 100644--- a/sdk/src/ui/input/control-pointers.ts+++ b/sdk/src/ui/input/control-pointers.ts@@ -14,6 +14,29 @@ export interface ControlPointer {layoutY: number;worldX: number;worldY: number;+ /** The pointer on the drawing surface: pixels, origin bottom-left — what a camera's `screenToWorld` takes. */+ screenX: number;+ screenY: number;+}++/** What a pointer event says about the pointer that made it. */+export interface UiPointerEventData {+ pointer: number;+ worldX: number;+ worldY: number;+ layoutX: number;+ layoutY: number;+ screenX: number;+ screenY: number;}/** Per-app projection of the UI hit tester's captures; never a second raycast. */-export const UIControlPointers = defineResource<{ pointers: Map<number, ControlPointer> }>({ pointers: new Map() }, 'UIControlPointers');+export interface UIControlPointersData {+ pointers: Map<number, ControlPointer>;+ /**+ * Take what @p pointer is holding away from it: the held entity gets `press_cancel` and is clicked by nothing when the pointer+ * lifts. With @p to, that entity holds the pointer from now on and hears its moves and its release — a scroll area that has+ * decided a drag is a drag takes it from the button under the finger this way.+ */+ cancelHold(pointer: number, to?: Entity): void;+}+export const UIControlPointers = defineResource<UIControlPointersData>({ pointers: new Map(), cancelHold: () => {} }, 'UIControlPointers');diff --git a/sdk/src/ui/input/interaction.ts b/sdk/src/ui/input/interaction.tsindex 5ff883e22..1b28aa2a8 100644--- a/sdk/src/ui/input/interaction.ts+++ b/sdk/src/ui/input/interaction.ts@@ -29,15 +29,16 @@ import { fireGestureClick, expireGestureFired } from './gestureClick';import { inputRouter } from '../../input/inputRouter';import { SystemLabel, PluginName } from '../../ecs/systemLabels';import type { UILayoutGenerationData } from '../layout/ui-layout-generation';-import { UIControlPointers, type ControlPointer } from './control-pointers';+import { UIControlPointers, type ControlPointer, type UiPointerEventData } from './control-pointers';function emitWithBubbling(world: World,events: UIEventQueue,entity: Entity,type: string,+ data?: UiPointerEventData,): void {- const event = events.emit(entity, type);+ const event = events.emit(entity, type, data);if (!world.valid(entity)) return;walkParentChain(world, entity, (ancestor) => {@@ -81,7 +82,26 @@ export class UIInteractionPlugin implements Plugin {// imply. Gestures (drag, scroll, slider) stay on the primary pointer.const book = new UiPointerBook();const controlPointers = new Map<number, ControlPointer>();- app.insertResource(UIControlPointers, { pointers: controlPointers });+ const lastMoveSent = new Map<number, { x: number; y: number }>();+ /** What a pointer event tells about its pointer, from the position this frame's raycast used. */+ const dataFor = (pointer: number): UiPointerEventData | undefined => {+ const c = controlPointers.get(pointer);+ return c ? { pointer, worldX: c.worldX, worldY: c.worldY, layoutX: c.layoutX, layoutY: c.layoutY, screenX: c.screenX, screenY: c.screenY } : undefined;+ };+ app.insertResource(UIControlPointers, {+ pointers: controlPointers,+ cancelHold: (pointer: number, to?: Entity): void => {+ const held = book.cancelHold(pointer, to);+ if (held === null || !world.valid(held)) return;+ if (world.has(held, UIInteraction)) {+ world.set(held, UIInteraction, { ...world.get(held, UIInteraction), pressed: false, justReleased: false });+ }+ emitWithBubbling(world, events, held, UIEventType.PressCancel, dataFor(pointer));+ // the new holder hears the moves from here, not from wherever the last holder was told about+ const here = controlPointers.get(pointer);+ if (to !== undefined && here) lastMoveSent.set(pointer, { x: here.worldX, y: here.worldY });+ },+ });/** Where each pointer last hit, so a settled pointer skips the raycast. */const lastHit = new UiPointerHitCache();const offExecution = app.onExecutionChange(held => {@@ -168,7 +188,7 @@ export class UIInteractionPlugin implements Plugin {const layoutPointer = overlay.active ? screenToUiLayout(overlay, glX, glY) : worldPointer;controlPointers.set(pointer.id, { id: pointer.id, down: pointer.down, pressed: pointer.pressed,entity: null, pressedEntity: null, layoutX: layoutPointer.x, layoutY: layoutPointer.y,- worldX: worldPointer.x, worldY: worldPointer.y });+ worldX: worldPointer.x, worldY: worldPointer.y, screenX: glX, screenY: glY });if (!primaryWritten && (pointer.down || !hasHeldPointer)) {primaryWritten = true;camera.worldMouseX = worldPointer.x;@@ -218,18 +238,32 @@ export class UIInteractionPlugin implements Plugin {// A complete click may arrive between frames. Preserve its// edge even when the final held-state is already false.world.set(event.entity, UIInteraction, { ...world.get(event.entity, UIInteraction), justPressed: true });- emitWithBubbling(world, events, event.entity, UIEventType.Press);+ emitWithBubbling(world, events, event.entity, UIEventType.Press, dataFor(event.pointer));break;case 'release':if (world.has(event.entity, UIInteraction))world.set(event.entity, UIInteraction, { ...world.get(event.entity, UIInteraction), justReleased: true });- emitWithBubbling(world, events, event.entity, UIEventType.Release);+ emitWithBubbling(world, events, event.entity, UIEventType.Release, dataFor(event.pointer));break;case 'click':- emitWithBubbling(world, events, event.entity, UIEventType.Click);+ emitWithBubbling(world, events, event.entity, UIEventType.Click, dataFor(event.pointer));break;}}+ // A pointer holding an entity that moved: the holder hears it (and its ancestors), with where.+ for (const [id, entity] of book.cancelledHolds) {+ lastMoveSent.delete(id);+ if (active(entity)) emitWithBubbling(world, events, entity, UIEventType.PressCancel, dataFor(id));+ }+ for (const pointer of pointers) {+ const held = pointer.down ? book.heldBy(pointer.id) : null;+ const c = controlPointers.get(pointer.id);+ if (held === null || !c) { lastMoveSent.delete(pointer.id); continue; }+ const before = lastMoveSent.get(pointer.id);+ lastMoveSent.set(pointer.id, { x: c.worldX, y: c.worldY });+ if (pointer.pressed || !before || (before.x === c.worldX && before.y === c.worldY)) continue;+ if (active(held)) emitWithBubbling(world, events, held, UIEventType.PointerMove, dataFor(pointer.id));+ }book.prune(active);for (const pointer of controlPointers.values()) pointer.entity = book.heldBy(pointer.id);for (const event of pointerEvents) if (event.kind === 'press') {diff --git a/sdk/src/ui/input/pointerBook.ts b/sdk/src/ui/input/pointerBook.tsindex a8b4b01ed..12034c2a9 100644--- a/sdk/src/ui/input/pointerBook.ts+++ b/sdk/src/ui/input/pointerBook.ts@@ -104,6 +104,24 @@ export class UiPointerBook {get cancelled(): ReadonlySet<Entity> { return this.cancelled_; }+ /** The holds the pointer system itself ended this step (a cancelled pointer), by pointer. */+ private cancelledBy_ = new Map<number, Entity>();+ get cancelledHolds(): ReadonlyMap<number, Entity> { return this.cancelledBy_; }++ /**+ * End what @p pointer is holding without a release or a click, optionally handing the pointer to @p to. Returns the entity it+ * held, or null when it held nothing.+ */+ cancelHold(pointer: number, to?: Entity): Entity | null {+ const held = this.pressedBy_.get(pointer);+ // handing a hold to the one that already has it changes nothing+ if (held === undefined || held === to) return null;+ this.pressedBy_.delete(pointer);+ this.cancelled_.add(held);+ if (to !== undefined) this.pressedBy_.set(pointer, to);+ return held;+ }+/** Entities at least one pointer is over. */get hovered(): ReadonlySet<Entity> {return new Set(this.hoveredBy_.values());@@ -139,13 +157,14 @@ export class UiPointerBook {): UiPointerEvent[] {const out: UiPointerEvent[] = [];this.cancelled_.clear();+ this.cancelledBy_.clear();const live = new Set<number>();for (const p of pointers) {live.add(p.id);if (p.cancelled) {const held = this.pressedBy_.get(p.id);- if (held !== undefined) this.cancelled_.add(held);+ if (held !== undefined) { this.cancelled_.add(held); this.cancelledBy_.set(p.id, held); }this.pressedBy_.delete(p.id);}const under = hit(p);diff --git a/sdk/tests/cocos/camera-richtext-wasm.test.ts b/sdk/tests/cocos/camera-richtext-wasm.test.tsindex 86a4f927f..8353e76d6 100644--- a/sdk/tests/cocos/camera-richtext-wasm.test.ts+++ b/sdk/tests/cocos/camera-richtext-wasm.test.ts@@ -12,14 +12,13 @@ import type { ESEngineModule } from '../../src/wasm';import { App, flushPendingRegistrations } from '../../src/app/app';import { Camera as EngineCamera, Transform, ensureBuiltinComponentsRegistered } from '../../src/ecs/component';import { AppContext, setDefaultContext } from '../../src/ecs/context';-import { InputState } from '../../src/input/input';import { Text } from '../../src/ui/core/text';import { UIMask, MaskMode } from '../../src/ui/core/ui-mask';import { ContentRect } from '../../src/ui/core/content-rect';import { parseRichText } from '../../src/ui/text/rich-text-parser';import { resetClassRegistry } from '../../src/cocos/registry';import { declareBuiltinScripts } from '../../src/cocos/builtinScripts';-import { Camera, Color, EventTouch, Mask, Node, RichText, UITransform, Vec3, pumpTouches, setCurrentWorld } from '../../src/cocos';+import { Camera, Color, Mask, Node, RichText, UITransform, Vec3, setCurrentWorld } from '../../src/cocos';import { setUiCameraData } from '../../src/cocos/ui/screen';import { HAS_WASM, loadWasmModule } from '../helpers/loadWasm';@@ -127,45 +126,6 @@ describe.skipIf(!HAS_WASM)('Camera, RichText and touch locations', () => {});});- describe('touch locations', () => {- it('gives the screen point from getLocation and the canvas point from getUILocation', () => {- boot();- const input = new InputState();- const seen: Array<{ screen: number[]; ui: number[] }> = [];- const n = node('target');- n.addComponent(UITransform)!.setContentSize(2000, 2000);- n.on(Node.EventType.TOUCH_START, (e: EventTouch) => {- seen.push({ screen: [e.getLocationX(), e.getLocationY()], ui: [e.getUILocationX(), e.getUILocationY()] });- });- // a pointer at CSS (500, 250) with the surface 600 high: GL y = 600 - 250 = 350- const p = { id: 1, x: 500, y: 250 };- input.touches.set(1, p);- input.touchesStarted.set(1, p);- pumpTouches(app!.world, input, { valid: true, screenH: 600 } as never);- expect(seen).toEqual([{ screen: [500, 350], ui: [100, 50] }]);- });-- it('hands a camera the touch\'s screen point and gets the canvas point back', () => {- boot();- const cam = node('cam').addComponent(Camera)!;- const input = new InputState();- let converted: Vec3 | null = null;- const n = node('target');- n.addComponent(UITransform)!.setContentSize(2000, 2000);- n.on(Node.EventType.TOUCH_START, (e: EventTouch) => {- const l = e.getLocation();- converted = cam.screenToWorld(new Vec3(l.x, l.y, 0));- expect(converted.x).toBe(e.getUILocationX());- expect(converted.y).toBe(e.getUILocationY());- });- const p = { id: 1, x: 120, y: 80 };- input.touches.set(1, p);- input.touchesStarted.set(1, p);- pumpTouches(app!.world, input, { valid: true, screenH: 600 } as never);- expect(converted).not.toBeNull();- });- });-describe('RichText', () => {it('is a Text with markup on, found as a RichText and writes its string through', () => {boot();diff --git a/sdk/tests/cocos/controls-wasm.test.ts b/sdk/tests/cocos/controls-wasm.test.tsindex 0e4dd54bc..473c9f80a 100644--- a/sdk/tests/cocos/controls-wasm.test.ts+++ b/sdk/tests/cocos/controls-wasm.test.ts@@ -5,54 +5,25 @@** The camera is faked as the identity, as in the touch tests, so a screen pixel is a world unit with y flipped.*/-import { afterEach, beforeAll, describe, expect, it, vi } from 'vitest';+import { afterEach, beforeAll, describe, expect, it } from 'vitest';import type { ESEngineModule } from '../../src/wasm';-import { App, flushPendingRegistrations } from '../../src/app/app';import { Transform, ensureBuiltinComponentsRegistered } from '../../src/ecs/component';-import { AppContext, setDefaultContext } from '../../src/ecs/context';-import { InputState } from '../../src/input/input';-import { resetClassRegistry } from '../../src/cocos/registry';-import { declareBuiltinScripts } from '../../src/cocos/builtinScripts';import {BlockInputEvents, Button, Node, ProgressBar, ScrollView, Slider, Sprite, Toggle, ToggleContainer, UITransform, Vec2,- pumpTouches, setCurrentWorld,} from '../../src/cocos';import { HAS_WASM, loadWasmModule } from '../helpers/loadWasm';--vi.mock('../../src/ui/util/ui-pick', () => ({ screenToUiWorld: (_cam: unknown, glX: number, glY: number) => ({ x: glX, y: glY }) }));--const SURFACE_H = 800;-const camera = { valid: true, screenH: SURFACE_H } as never;+import { pointerApp, type PointerApp } from './helpers/pointerApp';describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {let module: ESEngineModule;- let registry: InstanceType<ESEngineModule['Registry']> | undefined;- let app: App | undefined;- let input: InputState;+ let P: PointerApp | undefined;beforeAll(async () => { ensureBuiltinComponentsRegistered(); module = await loadWasmModule(); });- afterEach(() => {- setCurrentWorld(null);- app?.world.disconnectCpp();- registry?.delete();- app = undefined;- registry = undefined;- });+ afterEach(() => { P?.dispose(); P = undefined; });- function boot(): void {- setDefaultContext(new AppContext());- ensureBuiltinComponentsRegistered();- resetClassRegistry();- declareBuiltinScripts();- registry = new module.Registry();- app = App.new();- app.connectCpp(registry as never, module);- flushPendingRegistrations(app);- setCurrentWorld(app.world);- input = new InputState();- }- const W = () => app!.world;- const frame = async (n = 2): Promise<void> => { for (let i = 0; i < n; i++) await app!.tick(1 / 60); };+ function boot(): void { P = pointerApp(module); }+ const W = () => P!.app.world;+ const frame = (n = 2): Promise<void> => P!.frame(n);function box(name: string, x: number, y: number, w = 100, h = 100, parent?: Node): Node {const n = new Node(name);@@ -63,12 +34,10 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {return n;}- const screenOf = (wx: number, wy: number) => ({ x: wx, y: SURFACE_H - wy });- function pump(): void { pumpTouches(W(), input, camera); input.touchesStarted.clear(); input.touchesEnded.clear(); }- function press(wx: number, wy: number, id = 1): void { const p = { id, ...screenOf(wx, wy) }; input.touches.set(id, p); input.touchesStarted.set(id, p); pump(); }- function move(wx: number, wy: number, id = 1): void { input.touches.set(id, { id, ...screenOf(wx, wy) }); pump(); }- function release(wx: number, wy: number, id = 1): void { const p = { id, ...screenOf(wx, wy) }; input.touches.delete(id); input.touchesEnded.set(id, p); pump(); }- const tap = (x: number, y: number): void => { press(x, y); release(x, y); };+ const press = (x: number, y: number, id = 1): Promise<void> => P!.press(x, y, id);+ const move = (x: number, y: number, id = 1): Promise<void> => P!.move(x, y, id);+ const release = (x: number, y: number, id = 1): Promise<void> => P!.release(x, y, id);+ const tap = (x: number, y: number): Promise<void> => P!.tap(x, y);// ---- Toggle -----------------------------------------------------------------------------------------------------@@ -88,10 +57,10 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {const seen: boolean[] = [];node.on(Toggle.EventType.TOGGLE, (x: Toggle) => seen.push(x.isChecked));expect(mark.active).toBe(false);- tap(200, 200);+ await tap(200, 200);expect(t.isChecked).toBe(true);expect(mark.active).toBe(true);- tap(200, 200);+ await tap(200, 200);expect(t.isChecked).toBe(false);expect(mark.active).toBe(false);expect(seen).toEqual([true, false]);@@ -101,7 +70,7 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {boot();const { t, mark } = await toggle('t', 200, 200);t.interactable = false;- tap(200, 200);+ await tap(200, 200);expect(t.isChecked).toBe(false);t.interactable = true;t.setIsCheckedWithoutNotify(true);@@ -120,15 +89,15 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {expect(a.t.isChecked || b.t.isChecked).toBe(true); // the container made one on when it enabledconst firstOn = a.t.isChecked ? a : b;const other = a.t.isChecked ? b : a;- tap(other.node.getWorldPosition().x, other.node.getWorldPosition().y);+ await tap(other.node.getWorldPosition().x, other.node.getWorldPosition().y);expect(other.t.isChecked).toBe(true);expect(firstOn.t.isChecked).toBe(false);// clicking the one that is on does nothing- tap(other.node.getWorldPosition().x, other.node.getWorldPosition().y);+ await tap(other.node.getWorldPosition().x, other.node.getWorldPosition().y);expect(other.t.isChecked).toBe(true);// …unless the group allows nonegroup.allowSwitchOff = true;- tap(other.node.getWorldPosition().x, other.node.getWorldPosition().y);+ await tap(other.node.getWorldPosition().x, other.node.getWorldPosition().y);expect(other.t.isChecked).toBe(false);expect(group.anyTogglesChecked()).toBe(false);});@@ -167,11 +136,11 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {await frame(1);let slid = 0;track.on(Slider.EventType.SLIDE, () => slid++);- press(250, 300); // a quarter of the way: the track spans 200..400+ await press(250, 300); // a quarter of the way: the track spans 200..400expect(s.progress).toBeCloseTo(0.25, 5);- move(350, 300);+ await move(350, 300);expect(s.progress).toBeCloseTo(0.75, 5);- release(350, 300);+ await release(350, 300);expect(knobNode.position.x).toBeCloseTo(50, 3); // 0.75 * 200 - 100expect(slid).toBe(2);});@@ -187,9 +156,9 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {await frame(1);let heard = 0;behind.on(Node.EventType.TOUCH_START, () => heard++);- tap(200, 200);+ await tap(200, 200);expect(heard).toBe(0);- tap(320, 200); // beside the blocker (it spans 150..250), still on the node behind (50..350)+ await tap(320, 200); // beside the blocker (it spans 150..250), still on the node behind (50..350)expect(heard).toBe(1);});@@ -224,10 +193,10 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {const { content, sv } = await scroller();sv.inertia = false;const top = content.position.y;- press(300, 350);- move(300, 340); // 10 px: past the drag threshold- move(300, 300); // 50 more, upward on screen = toward the content's end- release(300, 300);+ await press(300, 350);+ await move(300, 340); // 10 px: past the drag threshold+ await move(300, 300); // 50 more, upward on screen = toward the content's end+ await release(300, 300);// the finger went DOWN the screen (y decreases) by 50 after the threshold, which pulls the content down: offset shrinks, but// it was at the start, so it is an overscroll that springs back; check the direction through the offset sign mid-drag insteadexpect(sv.getScrollOffset().y).toBeLessThanOrEqual(0);@@ -239,11 +208,11 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {it('scrolls toward the end when the finger moves up the screen, and glides on after a flick', async () => {boot();const { sv } = await scroller();- press(300, 250);- for (let i = 1; i <= 8; i++) { move(300, 250 + i * 10); await frame(1); } // up the screen: world y grows+ await press(300, 250);+ for (let i = 1; i <= 8; i++) { await move(300, 250 + i * 10); } // up the screen: world y growsconst held = sv.getScrollOffset().y;expect(held).toBeGreaterThan(40);- release(300, 330);+ await release(300, 330);await frame(1);await frame(60);expect(sv.getScrollOffset().y).toBeGreaterThan(held);@@ -256,16 +225,16 @@ describe.skipIf(!HAS_WASM)('toggles, bars, sliders and scroll views', () => {// the button is 100 below the content's top edge; the content's top is at the view's top (y 400)const bx = 300, by = 300; // view centre: content top 400, button centre at 400 - 100 = 300void button;- press(bx, by);- move(bx, by + 30);- move(bx, by + 60);- release(bx, by + 60);+ await press(bx, by);+ await move(bx, by + 30);+ await move(bx, by + 60);+ await release(bx, by + 60);expect(clicks()).toBe(0);expect(sv.getScrollOffset().y).toBeGreaterThan(0);sv.scrollToTop(0);await frame(2);- tap(bx, by);+ await tap(bx, by);expect(clicks()).toBe(1);});diff --git a/sdk/tests/cocos/globals-wasm.test.ts b/sdk/tests/cocos/globals-wasm.test.tsindex 14d776d5f..60dd9bc7d 100644--- a/sdk/tests/cocos/globals-wasm.test.ts+++ b/sdk/tests/cocos/globals-wasm.test.ts@@ -7,7 +7,7 @@* key codes; the wheel is reported with Cocos's sign; the game clock advances with the engine's; and the visible size scripts read* is what the UI camera shows.*/-import { afterEach, beforeAll, describe, expect, it, vi } from 'vitest';+import { afterEach, beforeAll, describe, expect, it } from 'vitest';import type { ESEngineModule } from '../../src/wasm';import { App, flushPendingRegistrations } from '../../src/app/app';import { ensureBuiltinComponentsRegistered } from '../../src/ecs/component';@@ -16,25 +16,29 @@ import { InputState } from '../../src/input/input';import { ScreenLayout } from '../../src/ui/core/screen-layout';import { resetClassRegistry } from '../../src/cocos/registry';import { declareBuiltinScripts } from '../../src/cocos/builtinScripts';-import { pumpKeysAndMouse, pumpTouches } from '../../src/cocos/input';+import { pumpKeysAndMouse } from '../../src/cocos/input';+import type { ControlPointer } from '../../src/ui/input/control-pointers';+import { pointerApp, type PointerApp } from './helpers/pointerApp';import {EventKeyboard, EventMouse, EventTouch, Input, KeyCode, game, input, screen, setCurrentWorld, view,} from '../../src/cocos';import { HAS_WASM, loadWasmModule } from '../helpers/loadWasm';-vi.mock('../../src/ui/util/ui-pick', () => ({ screenToUiWorld: (_cam: unknown, glX: number, glY: number) => ({ x: glX, y: glY }) }));--const camera = { valid: true, screenH: 800 } as never;+/** The mouse as the engine reports it: the world point and the surface point are the same here. */+const mouseAt = (x: number, y: number): ControlPointer => ({ worldX: x, worldY: y, screenX: x, screenY: y }) as never;describe.skipIf(!HAS_WASM)('input, game and view from the frame loop', () => {let module: ESEngineModule;let registry: InstanceType<ESEngineModule['Registry']> | undefined;let app: App | undefined;+ let P: PointerApp | undefined;beforeAll(async () => { ensureBuiltinComponentsRegistered(); module = await loadWasmModule(); });afterEach(() => {input.targetOff(undefined);for (const t of Object.values(Input.EventType)) input.off(t);+ P?.dispose();+ P = undefined;setCurrentWorld(null);app?.world.disconnectCpp();registry?.delete();@@ -54,21 +58,15 @@ describe.skipIf(!HAS_WASM)('input, game and view from the frame loop', () => {setCurrentWorld(app.world);}- it('tells a listener on input about every touch, wherever it lands', () => {- boot();- const state = new InputState();+ it('tells a listener on input about every touch, wherever it lands', async () => {+ P = pointerApp(module);const log: string[] = [];for (const t of [Input.EventType.TOUCH_START, Input.EventType.TOUCH_MOVE, Input.EventType.TOUCH_END]) {input.on(t, (e: EventTouch) => log.push(`${t}@${Math.round(e.getUILocationX())},${Math.round(e.getUILocationY())}`));}- const at = (x: number, y: number) => ({ id: 1, x, y: 800 - y });- state.touches.set(1, at(300, 200)); state.touchesStarted.set(1, at(300, 200));- pumpTouches(app!.world, state, camera);- state.touchesStarted.clear();- state.touches.set(1, at(320, 200));- pumpTouches(app!.world, state, camera);- state.touches.delete(1); state.touchesEnded.set(1, at(320, 200));- pumpTouches(app!.world, state, camera);+ await P.press(300, 200);+ await P.move(320, 200);+ await P.release(320, 200);expect(log).toEqual(['touch-start@300,200', 'touch-move@320,200', 'touch-end@320,200']);});@@ -80,11 +78,11 @@ describe.skipIf(!HAS_WASM)('input, game and view from the frame loop', () => {input.on(t, (e: EventKeyboard) => log.push(`${t}:${e.keyCode}:${e.isPressed}`));}state.noteKeyDown('KeyA');- pumpKeysAndMouse(state, camera);+ pumpKeysAndMouse(state, null);state.keysPressed.clear();- pumpKeysAndMouse(state, camera); // held+ pumpKeysAndMouse(state, null); // heldstate.noteKeyUp('KeyA');- pumpKeysAndMouse(state, camera);+ pumpKeysAndMouse(state, null);expect(log).toEqual([`keydown:${KeyCode.KEY_A}:true`,`key-pressing:${KeyCode.KEY_A}:true`,@@ -100,16 +98,14 @@ describe.skipIf(!HAS_WASM)('input, game and view from the frame loop', () => {input.on(Input.EventType.MOUSE_MOVE, (e: EventMouse) => log.push(`move:${e.getDeltaX()}`));input.on(Input.EventType.MOUSE_UP, (e: EventMouse) => log.push(`up:${e.getButton()}`));input.on(Input.EventType.MOUSE_WHEEL, (e: EventMouse) => log.push(`wheel:${e.getScrollY()}`));- state.mouseX = 100; state.mouseY = 700;- pumpKeysAndMouse(state, camera); // the first frame only learns where the mouse is+ pumpKeysAndMouse(state, mouseAt(100, 100)); // the first frame only learns where the mouse isstate.noteMouseDown(0);- pumpKeysAndMouse(state, camera);+ pumpKeysAndMouse(state, mouseAt(100, 100));state.mouseButtonsPressed.clear();- state.mouseX = 130;- pumpKeysAndMouse(state, camera);+ pumpKeysAndMouse(state, mouseAt(130, 100));state.noteMouseUp(0);state.scrollDeltaY = 120; // the DOM's "down"- pumpKeysAndMouse(state, camera);+ pumpKeysAndMouse(state, mouseAt(130, 100));expect(log).toEqual(['down:0@100,100', 'move:30', 'up:0', 'wheel:-120']);});diff --git a/sdk/tests/cocos/helpers/pointerApp.ts b/sdk/tests/cocos/helpers/pointerApp.tsnew file mode 100644index 000000000..ecfe91db9--- /dev/null+++ b/sdk/tests/cocos/helpers/pointerApp.ts@@ -0,0 +1,100 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file An app with the engine's real pointer pipeline, for tests that touch Cocos nodes.+ *+ * The camera maps the world square 0..800 x 0..800 onto an 800 x 800 surface, so a world point (x, y) is the surface pixel (x, y)+ * counted from the bottom-left, and the pointer for it is at CSS (x, 800 - y). Every press, move and release is a real frame:+ * the input state is set, the app ticks, and the engine's interaction system picks, holds and reports as it does in a game.+ */+import { App, flushPendingRegistrations } from '../../../src/app/app';+import { registerComponent } from '../../../src/ecs/component';+import { AppContext, setDefaultContext } from '../../../src/ecs/context';+import { Input, InputState } from '../../../src/input/input';+import { UICameraInfo } from '../../../src/ui/core/ui-camera-info';+import { UILayoutGeneration } from '../../../src/ui/layout/ui-layout-generation';+import { uiInteractionPlugin } from '../../../src/ui/input/interaction';+import { ensureBuiltinComponentsRegistered } from '../../../src/ecs/component';+import { resetClassRegistry } from '../../../src/cocos/registry';+import { declareBuiltinScripts } from '../../../src/cocos/builtinScripts';+import { setCurrentWorld } from '../../../src/cocos';+import type { ESEngineModule } from '../../../src/wasm';++export const SURFACE = 800;++export interface PointerApp {+ app: App;+ input: InputState;+ /** One frame. */+ frame(n?: number): Promise<void>;+ /** A finger lands on the world point (x, y). */+ press(x: number, y: number, id?: number): Promise<void>;+ move(x: number, y: number, id?: number): Promise<void>;+ release(x: number, y: number, id?: number): Promise<void>;+ tap(x: number, y: number): Promise<void>;+ dispose(): void;+}++/** @p origin: how far the world's origin sits from the surface's bottom-left corner, in pixels: a surface point is the world point plus it. */+export function pointerApp(module: ESEngineModule, origin: { x: number; y: number } = { x: 0, y: 0 }): PointerApp {+ setDefaultContext(new AppContext());+ ensureBuiltinComponentsRegistered();+ resetClassRegistry();+ declareBuiltinScripts();+ const registry = new module.Registry();+ const app = App.new();+ app.connectCpp(registry as never, module);+ const input = new InputState();+ app.insertResource(Input, input);+ const s = 2 / SURFACE;+ const projection = new Float32Array([s, 0, 0, 0, 0, s, 0, 0, 0, 0, -0.001, 0, -1 + s * origin.x, -1 + s * origin.y, 0, 1]);+ app.insertResource(UICameraInfo, {+ ...UICameraInfo._default, viewProjection: projection, vpX: 0, vpY: 0, vpW: SURFACE, vpH: SURFACE, screenW: SURFACE, screenH: SURFACE, valid: true,+ });+ app.insertResource(UILayoutGeneration, { generation: 0 });+ uiInteractionPlugin.build(app);+ flushPendingRegistrations(app);+ setCurrentWorld(app.world);++ // a world point is the surface point minus the origin; CSS y counts from the top+ const css = (x: number, y: number, id: number) => ({ id, x: x + origin.x, y: SURFACE - (y + origin.y) });+ const frame = async (n = 1): Promise<void> => {+ for (let i = 0; i < n; i++) {+ app.world.ensureTransformsComposed();+ await app.tick(1 / 60);+ }+ };+ return {+ app, input, frame,+ async press(x, y, id = 1) {+ const p = css(x, y, id);+ input.touches.set(id, p);+ input.touchesStarted.set(id, p);+ await frame();+ input.touchesStarted.clear();+ },+ async move(x, y, id = 1) {+ input.touches.set(id, css(x, y, id));+ await frame();+ },+ async release(x, y, id = 1) {+ const p = css(x, y, id);+ input.touches.delete(id);+ input.touchesEnded.set(id, p);+ await frame();+ input.touchesEnded.clear();+ },+ async tap(x, y) {+ await this.press(x, y);+ await this.release(x, y);+ },+ dispose() {+ uiInteractionPlugin.cleanup(app);+ setCurrentWorld(null);+ app.world.disconnectCpp();+ registry.delete();+ },+ };+}++export { registerComponent };diff --git a/sdk/tests/cocos/touch-locations-wasm.test.ts b/sdk/tests/cocos/touch-locations-wasm.test.tsnew file mode 100644index 000000000..b612a45b0--- /dev/null+++ b/sdk/tests/cocos/touch-locations-wasm.test.ts@@ -0,0 +1,64 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file The two places a touch has: the screen point (`getLocation`) and the canvas point (`getUILocation`).+ *+ * Through the engine's real pointer pipeline, with a camera that puts the world origin 400 pixels right of and 300 above the surface's+ * bottom-left corner, so the two are different numbers and a mix-up shows.+ */+import { afterEach, beforeAll, describe, expect, it } from 'vitest';+import type { ESEngineModule } from '../../src/wasm';+import { Transform, ensureBuiltinComponentsRegistered } from '../../src/ecs/component';+import { Camera, EventTouch, Node, UITransform, Vec3 } from '../../src/cocos';+import { setUiCameraData } from '../../src/cocos/ui/screen';+import { UICameraInfo } from '../../src/ui/core/ui-camera-info';+import { HAS_WASM, loadWasmModule } from '../helpers/loadWasm';+import { pointerApp, type PointerApp } from './helpers/pointerApp';++describe.skipIf(!HAS_WASM)('touch locations', () => {+ let module: ESEngineModule;+ let P: PointerApp | undefined;++ beforeAll(async () => { ensureBuiltinComponentsRegistered(); module = await loadWasmModule(); });+ afterEach(() => { P?.dispose(); P = undefined; setUiCameraData(null); });++ function boot(): void {+ P = pointerApp(module, { x: 400, y: 300 });+ // the frame loop hands a camera the live UI camera data; this app has no pass that does, so do it directly+ setUiCameraData(P.app.getResource(UICameraInfo));+ }+ const node = (name: string): Node => { const n = new Node(name); P!.app.world.insert(n._entity, Transform); return n; };++ it('gives the screen point from getLocation and the canvas point from getUILocation', async () => {+ boot();+ const seen: Array<{ screen: number[]; ui: number[] }> = [];+ const n = node('target');+ n.addComponent(UITransform)!.setContentSize(2000, 2000);+ n.on(Node.EventType.TOUCH_START, (e: EventTouch) => {+ seen.push({ screen: [e.getLocationX(), e.getLocationY()], ui: [e.getUILocationX(), e.getUILocationY()] });+ });+ await P!.press(100, 50); // world (100, 50): the surface point (500, 350)+ expect(seen).toHaveLength(1);+ expect(seen[0]!.ui[0]).toBeCloseTo(100, 2);+ expect(seen[0]!.ui[1]).toBeCloseTo(50, 2);+ expect(seen[0]!.screen[0]).toBeCloseTo(500, 2);+ expect(seen[0]!.screen[1]).toBeCloseTo(350, 2);+ });++ it('hands a camera the screen point of the touch and gets the canvas point back', async () => {+ boot();+ const cam = node('cam').addComponent(Camera)!;+ const n = node('target');+ n.addComponent(UITransform)!.setContentSize(2000, 2000);+ let checked = 0;+ n.on(Node.EventType.TOUCH_START, (e: EventTouch) => {+ const l = e.getLocation();+ const world = cam.screenToWorld(new Vec3(l.x, l.y, 0));+ expect(world.x).toBeCloseTo(e.getUILocationX(), 2);+ expect(world.y).toBeCloseTo(e.getUILocationY(), 2);+ checked++;+ });+ await P!.press(120, 80);+ expect(checked).toBe(1);+ });+});diff --git a/sdk/tests/cocos/touch-wasm.test.ts b/sdk/tests/cocos/touch-wasm.test.tsindex 2971a47a2..d1020ff97 100644--- a/sdk/tests/cocos/touch-wasm.test.ts+++ b/sdk/tests/cocos/touch-wasm.test.ts@@ -9,54 +9,23 @@** The camera is faked as the identity (a screen pixel is a world unit, y flipped), so the geometry in the tests is the geometry on screen.*/-import { afterEach, beforeAll, describe, expect, it, vi } from 'vitest';+import { afterEach, beforeAll, describe, expect, it } from 'vitest';import type { ESEngineModule } from '../../src/wasm';-import { App, flushPendingRegistrations } from '../../src/app/app';import { Transform, ensureBuiltinComponentsRegistered } from '../../src/ecs/component';-import { AppContext, setDefaultContext } from '../../src/ecs/context';-import { InputState } from '../../src/input/input';-import { resetClassRegistry } from '../../src/cocos/registry';-import {- Button, Component, EventHandler, EventTouch, Node, NodeEventType, UITransform, Vec2, _decorator, pumpTouches, setCurrentWorld,-} from '../../src/cocos';-import { declareButton } from '../../src/cocos/ui/button';+import { Button, Component, EventHandler, EventTouch, Node, NodeEventType, UITransform, _decorator } from '../../src/cocos';import { HAS_WASM, loadWasmModule } from '../helpers/loadWasm';--// A screen pixel is a world unit: the GL y is the surface height minus the screen y.-vi.mock('../../src/ui/util/ui-pick', () => ({ screenToUiWorld: (_cam: unknown, glX: number, glY: number) => ({ x: glX, y: glY }) }));--const SURFACE_H = 800;-const camera = { valid: true, screenH: SURFACE_H } as never;+import { pointerApp, type PointerApp } from './helpers/pointerApp';describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {let module: ESEngineModule;- let registry: InstanceType<ESEngineModule['Registry']> | undefined;- let app: App | undefined;- let input: InputState;+ let P: PointerApp | undefined;beforeAll(async () => { ensureBuiltinComponentsRegistered(); module = await loadWasmModule(); });+ afterEach(() => { P?.dispose(); P = undefined; });- function connect(): void {- setDefaultContext(new AppContext());- ensureBuiltinComponentsRegistered();- resetClassRegistry();- declareButton();- registry = new module.Registry();- app = App.new();- app.connectCpp(registry as never, module);- flushPendingRegistrations(app);- setCurrentWorld(app.world);- input = new InputState();- }- afterEach(() => {- setCurrentWorld(null);- app?.world.disconnectCpp();- registry?.delete();- app = undefined;- registry = undefined;- });-- const W = () => app!.world;+ function connect(): void { P = pointerApp(module); }+ const app = () => P!.app;+ const W = () => P!.app.world;/** A node of @p w by @p h whose centre is at (x, y). */function box(name: string, x: number, y: number, w = 100, h = 100, parent?: Node): Node {@@ -68,22 +37,10 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {return n;}- // ---- driving the pointers by hand ----------------------------------------------------------------------------------- const screenOf = (wx: number, wy: number): { x: number; y: number } => ({ x: wx, y: SURFACE_H - wy });- function pump(): void { pumpTouches(W(), input, camera); input.touchesStarted.clear(); input.touchesEnded.clear(); }- function press(id: number, wx: number, wy: number): void {- const p = { id, ...screenOf(wx, wy) };- input.touches.set(id, p);- input.touchesStarted.set(id, p);- pump();- }- function move(id: number, wx: number, wy: number): void { input.touches.set(id, { id, ...screenOf(wx, wy) }); pump(); }- function release(id: number, wx: number, wy: number): void {- const p = { id, ...screenOf(wx, wy) };- input.touches.delete(id);- input.touchesEnded.set(id, p);- pump();- }+ // ---- driving the pointers: real frames through the engine's own interaction system ----------------------------------------+ const press = (id: number, x: number, y: number): Promise<void> => P!.press(x, y, id);+ const move = (id: number, x: number, y: number): Promise<void> => P!.move(x, y, id);+ const release = (id: number, x: number, y: number): Promise<void> => P!.release(x, y, id);function record(node: Node, log: string[], tag = node.name): void {for (const type of [NodeEventType.TOUCH_START, NodeEventType.TOUCH_MOVE, NodeEventType.TOUCH_END]) {@@ -91,28 +48,28 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {}}- it('tells a node about a touch that starts, moves and ends on it, with where each happened', () => {+ it('tells a node about a touch that starts, moves and ends on it, with where each happened', async () => {connect();const a = box('a', 200, 200);const log: string[] = [];record(a, log);- press(1, 210, 190);- move(1, 230, 190);- release(1, 230, 190);+ await press(1, 210, 190);+ await move(1, 230, 190);+ await release(1, 230, 190);expect(log).toEqual(['a:start@210,190', 'a:move@230,190', 'a:end@230,190']);});- it('says nothing to a node the touch missed', () => {+ it('says nothing to a node the touch missed', async () => {connect();const a = box('a', 200, 200);const log: string[] = [];record(a, log);- press(1, 400, 400);- release(1, 400, 400);+ await press(1, 400, 400);+ await release(1, 400, 400);expect(log).toEqual([]);});- it('hands the touch to the topmost of two overlapping nodes: the later sibling', () => {+ it('hands the touch to the topmost of two overlapping nodes: the later sibling', async () => {connect();const root = box('root', 0, 0, 1000, 1000);const under = box('under', 200, 200, 100, 100, root);@@ -120,26 +77,26 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {const log: string[] = [];record(under, log);record(over, log);- press(1, 210, 200); // inside both- release(1, 210, 200);+ await press(1, 210, 200); // inside both+ await release(1, 210, 200);expect(log.every((l) => l.startsWith('over:'))).toBe(true);expect(log).toHaveLength(2);});- it('keeps sending the gesture to the node that claimed it, wherever the finger goes', () => {+ it('keeps sending the gesture to the node that claimed it, wherever the finger goes', async () => {connect();const a = box('a', 200, 200);const b = box('b', 600, 200);const log: string[] = [];record(a, log);record(b, log);- press(1, 200, 200);- move(1, 600, 200); // over b, but a has the touch- release(1, 600, 200);+ await press(1, 200, 200);+ await move(1, 600, 200); // over b, but a has the touch+ await release(1, 600, 200);expect(log.map((l) => l.split('@')[0])).toEqual(['a:start', 'a:move', 'a:end']);});- it('bubbles to the ancestors nearest first, and stops where a listener says so', () => {+ it('bubbles to the ancestors nearest first, and stops where a listener says so', async () => {connect();const parent = box('parent', 300, 300, 400, 400);const child = box('child', 0, 0, 100, 100, parent);@@ -147,33 +104,33 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {const order: string[] = [];child.on(NodeEventType.TOUCH_START, () => order.push('child'));parent.on(NodeEventType.TOUCH_START, () => order.push('parent'));- press(1, 300, 300);- release(1, 300, 300);+ await press(1, 300, 300);+ await release(1, 300, 300);expect(order).toEqual(['child', 'parent']);order.length = 0;child.targetOff(undefined);child.on(NodeEventType.TOUCH_START, (e: EventTouch) => { order.push('child2'); e.stopPropagation(); });- press(2, 300, 300);- release(2, 300, 300);+ await press(2, 300, 300);+ await release(2, 300, 300);expect(order).toContain('child2');expect(order).not.toContain('parent');void grand;});- it('does not tell a switched-off node, or one under a switched-off parent', () => {+ it('does not tell a switched-off node, or one under a switched-off parent', async () => {connect();const parent = box('parent', 200, 200, 300, 300);const child = box('child', 0, 0, 100, 100, parent);const log: string[] = [];record(child, log);parent.active = false;- press(1, 200, 200);- release(1, 200, 200);+ await press(1, 200, 200);+ await release(1, 200, 200);expect(log).toEqual([]);});- it('gives the event its target and current target, and reads back a delta', () => {+ it('gives the event its target and current target, and reads back a delta', async () => {connect();const parent = box('parent', 200, 200, 300, 300);const child = box('child', 0, 0, 100, 100, parent);@@ -181,8 +138,8 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {child.on(NodeEventType.TOUCH_START, () => {});parent.on(NodeEventType.TOUCH_START, (e: EventTouch) => seen.push({ target: e.target!.name, current: e.currentTarget!.name }));child.on(NodeEventType.TOUCH_MOVE, (e: EventTouch) => seen.push({ target: 'm', current: 'm', dx: e.getDelta().x }));- press(1, 200, 200);- move(1, 215, 200);+ await press(1, 200, 200);+ await move(1, 215, 200);expect(seen[0]).toEqual({ target: 'child', current: 'parent' });expect(seen[1]!.dx).toBe(15);});@@ -202,7 +159,7 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {async function button(x = 200, y = 200): Promise<{ node: Node; button: Button }> {const node = box('btn', x, y);const b = node.addComponent<Button>('Button')!;- await app!.tick(1 / 60); // onEnable registers the listeners+ await P!.frame(1); // onEnable registers the listenersreturn { node, button: b };}@@ -217,8 +174,8 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {b.clickEvents = [handler];let clicked = 0;node.on(Button.EventType.CLICK, () => clicked++);- press(1, 205, 205);- release(1, 195, 195);+ await press(1, 205, 205);+ await release(1, 195, 195);expect(clicked).toBe(1);expect(t.calls).toEqual([['buy']]);});@@ -228,12 +185,12 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {const { node, button: b } = await button();let clicked = 0;node.on(Button.EventType.CLICK, () => clicked++);- press(1, 200, 200);- release(1, 500, 500); // dragged off and let go+ await press(1, 200, 200);+ await release(1, 500, 500); // dragged off and let goexpect(clicked).toBe(0);b.interactable = false;- press(2, 200, 200);- release(2, 200, 200);+ await press(2, 200, 200);+ await release(2, 200, 200);expect(clicked).toBe(0);});@@ -243,8 +200,8 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {const log: string[] = [];record(behind, log);await button();- press(1, 200, 200);- release(1, 200, 200);+ await press(1, 200, 200);+ await release(1, 200, 200);expect(log).toEqual([]);});@@ -254,9 +211,9 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {b.transition = Button.Transition.SCALE;b.zoomScale = 1.5;node.setScale(2, 2, 1);- press(1, 200, 200);+ await press(1, 200, 200);expect(node.scale.x).toBeCloseTo(3, 5);- release(1, 200, 200);+ await release(1, 200, 200);expect(node.scale.x).toBeCloseTo(2, 5);});@@ -266,10 +223,9 @@ describe.skipIf(!HAS_WASM)('touches and buttons on the engine', () => {let clicked = 0;node.on(Button.EventType.CLICK, () => clicked++);b.enabled = false;- await app!.tick(1 / 60); // the host sees the change and calls onDisable- press(1, 200, 200);- release(1, 200, 200);+ await P!.frame(1); // the host sees the change and calls onDisable+ await press(1, 200, 200);+ await release(1, 200, 200);expect(clicked).toBe(0);- void Vec2;});});diff --git a/sdk/tests/ui-pointer-events-wasm.test.ts b/sdk/tests/ui-pointer-events-wasm.test.tsnew file mode 100644index 000000000..597e050ad--- /dev/null+++ b/sdk/tests/ui-pointer-events-wasm.test.ts@@ -0,0 +1,194 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file What a pointer tells the entity it presses: where, which pointer, and what it does next.+ *+ * The UI interaction system already picks the topmost interactable under a pointer, lets that entity hold the gesture, and bubbles+ * what happens to it. These pin the parts a touch-style consumer needs on top: a rectangle-only entity can be the one picked, every+ * pointer event says where the pointer is, a held entity hears the pointer move, and a hold can be given up (nothing clicks).+ */+import { afterEach, beforeAll, describe, expect, it } from 'vitest';+import { App, flushPendingRegistrations } from '../src/app/app';+import { AppContext, setDefaultContext } from '../src/ecs/context';+import { EntityEvents } from '../src/ecs/entityEvents';+import { Input, InputState } from '../src/input/input';+import { Transform } from '../src/ecs/component';+import { ContentRect } from '../src/ui/core/content-rect';+import { UICameraInfo } from '../src/ui/core/ui-camera-info';+import { UIEventType } from '../src/ui/core/events';+import { UILayoutGeneration } from '../src/ui/layout/ui-layout-generation';+import { Interactable } from '../src/ui/input/interactable';+import { UIControlPointers, type UiPointerEventData } from '../src/ui/input/control-pointers';+import { uiInteractionPlugin } from '../src/ui/input/interaction';+import { registerComponent } from '../src/ecs/component';+import type { Entity } from '../src/types';+import type { ESEngineModule } from '../src/wasm';+import { HAS_WASM, loadWasmModule } from './helpers/loadWasm';++describe.skipIf(!HAS_WASM)('pointer events (WASM integration)', () => {+ let module: ESEngineModule;+ let app: App | undefined;+ let registry: InstanceType<ESEngineModule['Registry']> | undefined;++ beforeAll(async () => { module = await loadWasmModule(); });+ afterEach(() => {+ if (app) uiInteractionPlugin.cleanup(app);+ app?.world.disconnectCpp();+ registry?.delete();+ app = undefined;+ registry = undefined;+ });++ /** A 100 x 100 view: world x and y run from -50 to 50, and a pointer at CSS (50, 50) is the origin. */+ function boot(): { app: App; input: InputState; events: InstanceType<typeof import('../src/ecs/entityEvents').EntityEventQueue> } {+ setDefaultContext(new AppContext());+ registry = new module.Registry();+ app = App.new();+ app.connectCpp(registry, module);+ const input = new InputState();+ app.insertResource(Input, input);+ const projection = new Float32Array([.02, 0, 0, 0, 0, .02, 0, 0, 0, 0, -.001, 0, 0, 0, 0, 1]);+ app.insertResource(UICameraInfo, { ...UICameraInfo._default, viewProjection: projection, vpW: 100, vpH: 100, screenW: 100, screenH: 100, valid: true });+ app.insertResource(UILayoutGeneration, { generation: 0 });+ registerComponent('ContentRect', ContentRect);+ uiInteractionPlugin.build(app);+ flushPendingRegistrations(app);+ return { app, input, events: app.getResource(EntityEvents) as never };+ }++ /** An entity that is only a rectangle: nothing to draw, so no sprite to take a box from. */+ function area(a: App, x: number, y: number, w: number, h: number, blockRaycast = true): Entity {+ const e = a.world.spawn('area');+ a.world.insert(e, Transform, { position: { x, y, z: 0 } });+ a.world.insert(e, ContentRect, { size: { x: w, y: h }, pivot: { x: 0.5, y: 0.5 } });+ a.world.insert(e, Interactable, { blockRaycast });+ return e;+ }++ /** One frame with the mouse at CSS (x, y) and the button as given. */+ async function frame(a: App, input: InputState, x: number, y: number, press?: 'down' | 'up'): Promise<void> {+ input.mouseX = x;+ input.mouseY = y;+ input.mouseButtonsPressed.clear();+ input.mouseButtonsReleased.clear();+ if (press === 'down') input.noteMouseDown(0);+ if (press === 'up') input.noteMouseUp(0);+ a.world.ensureTransformsComposed();+ await a.tick(1 / 60);+ }++ it('lets an entity that is only a rectangle be the one pressed', async () => {+ const { app: a, input, events } = boot();+ const hit = area(a, 0, 0, 40, 20);+ const seen: UiPointerEventData[] = [];+ events.on(hit, UIEventType.Press, (e) => seen.push(e.data as UiPointerEventData));+ await frame(a, input, 50, 50, 'down');+ expect(seen).toHaveLength(1);+ // and outside its rectangle it is not+ await frame(a, input, 50, 50, 'up');+ const miss: unknown[] = [];+ events.on(hit, UIEventType.Press, (e) => miss.push(e));+ await frame(a, input, 5, 5, 'down');+ expect(miss).toHaveLength(0);+ });++ it('says where the pointer is in every event: which pointer, and its world and surface position', async () => {+ const { app: a, input, events } = boot();+ const hit = area(a, 0, 0, 100, 100);+ const log: Array<{ type: string; data: UiPointerEventData }> = [];+ for (const type of [UIEventType.Press, UIEventType.Release, UIEventType.Click]) {+ events.on(hit, type, (e) => log.push({ type: e.type, data: e.data as UiPointerEventData }));+ }+ await frame(a, input, 75, 25, 'down'); // right of centre by 25, above it by 25+ await frame(a, input, 75, 25, 'up');+ expect(log.map((l) => l.type)).toEqual([UIEventType.Press, UIEventType.Release, UIEventType.Click]);+ const press = log[0]!.data;+ expect(press.pointer).toBe(-1);+ expect(press.worldX).toBeCloseTo(25, 3);+ expect(press.worldY).toBeCloseTo(25, 3);+ // the surface's y is up from the bottom: CSS y 25 of 100 is GL y 75+ expect(press.screenX).toBeCloseTo(75, 3);+ expect(press.screenY).toBeCloseTo(75, 3);+ });++ it('tells the holder when the pointer moves while it is held, and not otherwise', async () => {+ const { app: a, input, events } = boot();+ const hit = area(a, 0, 0, 100, 100);+ const moves: UiPointerEventData[] = [];+ events.on(hit, UIEventType.PointerMove, (e) => moves.push(e.data as UiPointerEventData));+ await frame(a, input, 50, 50); // hovering: not a held pointer+ await frame(a, input, 60, 50);+ expect(moves).toHaveLength(0);+ await frame(a, input, 60, 50, 'down'); // the press frame is not a move+ expect(moves).toHaveLength(0);+ await frame(a, input, 60, 50); // held, still+ expect(moves).toHaveLength(0);+ await frame(a, input, 70, 50);+ await frame(a, input, 80, 40);+ expect(moves.map((m) => [Math.round(m.worldX), Math.round(m.worldY)])).toEqual([[20, 0], [30, 10]]);+ });++ it('goes on telling the holder when the pointer leaves it: the gesture is the holder\'s', async () => {+ const { app: a, input, events } = boot();+ const hit = area(a, 0, 0, 20, 20);+ const moves: number[] = [];+ events.on(hit, UIEventType.PointerMove, (e) => moves.push((e.data as UiPointerEventData).worldX));+ await frame(a, input, 50, 50, 'down');+ await frame(a, input, 90, 50); // far outside the 20 x 20 area+ expect(moves).toHaveLength(1);+ expect(moves[0]).toBeCloseTo(40, 3);+ });++ it('bubbles a move to an ancestor that lets events through', async () => {+ const { app: a, input, events } = boot();+ const parent = area(a, 0, 0, 100, 100, false);+ const child = area(a, 0, 0, 20, 20, false);+ a.world.setParent(child, parent);+ const heard: Array<[Entity, Entity]> = [];+ events.on(parent, UIEventType.PointerMove, (e) => heard.push([e.target, e.currentTarget]));+ await frame(a, input, 50, 50, 'down');+ await frame(a, input, 55, 50);+ expect(heard).toEqual([[child, parent]]);+ });++ it('lets a hold be given up: the holder hears press_cancel, and the lift clicks nothing', async () => {+ const { app: a, input, events } = boot();+ const hit = area(a, 0, 0, 100, 100);+ const log: string[] = [];+ for (const type of [UIEventType.Press, UIEventType.PressCancel, UIEventType.Release, UIEventType.Click, UIEventType.PointerMove]) {+ events.on(hit, type, (e) => log.push(e.type));+ }+ await frame(a, input, 50, 50, 'down');+ a.getResource(UIControlPointers).cancelHold(-1);+ await frame(a, input, 55, 50); // moving after the cancel is nobody's move+ await frame(a, input, 55, 50, 'up');+ expect(log).toEqual([UIEventType.Press, UIEventType.PressCancel]);+ });++ it('can hand the hold to another entity, which then hears the moves and the release', async () => {+ const { app: a, input, events } = boot();+ const scroller = area(a, 0, 0, 100, 100); // made first, so the button — made after it — is the topmost+ const button = area(a, 0, 0, 20, 20);+ a.world.setParent(button, scroller);+ const log: Array<[string, Entity]> = [];+ for (const e of [button, scroller]) {+ for (const type of [UIEventType.PressCancel, UIEventType.PointerMove, UIEventType.Release]) {+ events.on(e, type, (ev) => { if (ev.currentTarget === ev.target) log.push([ev.type, ev.target]); });+ }+ }+ await frame(a, input, 50, 50, 'down'); // the button, the topmost, holds it+ a.getResource(UIControlPointers).cancelHold(-1, scroller);+ await frame(a, input, 60, 50);+ await frame(a, input, 60, 50, 'up');+ expect(log.map(([t, e]) => `${t}:${e === button ? 'button' : 'scroller'}`)).toEqual([+ 'press_cancel:button', 'pointer_move:scroller', 'release:scroller',+ ]); // no click: the pointer lifted over the button, which is not what it held+ });++ it('cancelling a hold that is not there does nothing', async () => {+ const { app: a, input } = boot();+ area(a, 0, 0, 100, 100);+ await frame(a, input, 50, 50);+ expect(() => a.getResource(UIControlPointers).cancelHold(-1)).not.toThrow();+ });+});diff --git a/src/esengine/ui/UISystem.cpp b/src/esengine/ui/UISystem.cppindex c14d9cb19..aa776d30a 100644--- a/src/esengine/ui/UISystem.cpp+++ b/src/esengine/ui/UISystem.cpp@@ -15,6 +15,7 @@#include "../ecs/components/UINode.hpp"#include "../ecs/components/UIVisual.hpp"#include "../ecs/components/Sprite.hpp"+#include "../ecs/components/ContentRect.hpp"#include "../ecs/components/MeshRenderer.hpp"#include "../resource/ResourceManager.hpp"@@ -105,6 +106,9 @@ void UISystem::hitWorldContent(Registry& registry, const PickRay& ray,const bool sameDepth = std::abs(t - bestT) <= 1e-4f;if (!sameDepth && t > bestT) return;if (sameDepth && layer < bestLayer) return;+ // A full tie goes to the later entity, by its number rather than by where a pool happened to list it: the order content is+ // created in is the order it is drawn in, and which component pool is walked first says nothing about either.+ if (sameDepth && layer == bestLayer && entity.id() < best.id()) return;}haveBest = true;bestT = t;@@ -134,6 +138,30 @@ void UISystem::hitWorldContent(Registry& registry, const PickRay& ray,consider(entity, hitT, static_cast<i32>(sprite.layer));});+ // An entity that is only a rectangle — a touch area, a scroll viewport — has no sprite to take a box from. A sprite on the same+ // entity is handled above (it follows the rectangle), so this is for the ones that draw nothing.+ registry.eachLive<ContentRect, Interactable>(+ [&](Entity entity, ContentRect& rect, Interactable& interactable) {+ if (!registry.isEntityActive(entity)) return;+ if (!interactable.enabled || !interactable.raycastTarget) return;+ if (registry.has<UINode>(entity) || registry.has<Sprite>(entity)) return;+ if (!registry.has<Transform>(entity)) return;++ auto& t = registry.get<Transform>(entity);+ t.ensureDecomposed();++ f32 px = 0.0f, py = 0.0f, hitT = 0.0f;+ if (!rayHitsPlaneZ(ray, t.worldPosition.z, px, py, hitT)) return;++ if (!pointInTransformedBox(+ registry, entity, t, px, py, rect.size.x, rect.size.y,+ rect.pivot.x, rect.pivot.y+ )) return;++ // No sorting layer: it ties with a sprite at the same depth and loses to a higher layer, and the later entity wins a tie.+ consider(entity, hitT, 0);+ });+registry.eachLive<MeshRenderer, Interactable>([&](Entity entity, MeshRenderer& mesh, Interactable& interactable) {if (!registry.isEntityActive(entity)) return;diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex ed6e9828b..4cce7bbb3 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -28,6 +28,8 @@ export const NOTES_FLOOR = 'v0.65.0';* the reader recognises anyway.*/export const NOTED = {+ 'feat(ui): a held pointer reports its moves (pointer_move) and a given-up hold (press_cancel), every pointer event says where the pointer is, and a ContentRect-only entity can be pressed; the Cocos touches translate the engine\'s pointer events instead of dispatching on their own':+ {"note":"A held pointer reports where it goes, and a hold can be given up."},'feat(ui): ContentRect, the rectangle world content occupies — Sprite and Text follow it, a UIMask on the entity clips to it, a world point can be tested against it; the Cocos UITransform and Mask are views on it, and the importer writes it':{"note":"World content has a rectangle: `ContentRect`."},'feat(text): the engine\'s rich-text parser reads Cocos\'s tags (size, color names, br; underline/outline/click kept as plain text); Cocos Camera and RichText are views on the engine\'s Camera and Text; a touch has a screen point (getLocation) and a canvas point (getUILocation)':