Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(ui): UIWorldAnchor — a UI node over a point in the 3D world

yhha0f5d7ab2aed
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ccee5142e..fa7721b3a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,7 @@ published separately; it ships inside the editor.
- **The sky lights the scene.** Sky Atmosphere's **Light the Scene** makes the sky the scene's environment: every Ambient light that names no environment of its own lights with it, its diffuse light from the dome and the clouds and its reflection on metal and wet ground, scaled by the light's colour and intensity as an imported one is. The sky is worked out again as it changes (the sun moving, a weather coming in, clouds drifting), at most once a second; with clouds that takes about 20 ms on a desktop. The sun's disc is left out, since the directional light the sky follows is its direct light. Off by default, so a scene already lit by hand looks as it did.
- **Rooms stay dark under a sky that changes (sky occlusion).** With **Sky occlusion** on in the Lighting settings (`BakedLighting.skyOcclusion`), a probe bake keeps how much of the sky each probe sees instead of the sky itself: the probes hold the lights and their bounces, and the sky the frame has now (a live Sky Atmosphere, a weather, an environment) is added through what each probe sees of it, to the diffuse light and to the sky's reflection. A ball inside a roofed room measured 119 where the baked sky gave it 150, and a ball outside kept its light. A `.esprobes` baked this way carries `skyVisibility`; one without it lights as before. Probes are read per object, as Unity's light probes are: one large mesh spanning indoors and out takes one value.
+- **A UI node can stand over a point in the 3D world.** `UIWorldAnchor` puts its UINode at the screen position of an entity (plus an offset) or a world point every frame: an objective marker, a name over a head, a health bar. `clampToEdges` keeps an off-screen or behind-the-camera marker on the screen's edge in its true direction; `onScreen`, `behind` and `distance` read back. It is placed from the camera's pose at the end of Update, so a marker does not trail a first-person view that turned this frame. Saltline's relay and boat markers are four of them.
- **Perceivers hear.** `Perceiver.hearingRange` and a `Noise` event (where, how loud, who made it): a perceiver hears a noise within `hearingRange * loudness`, through walls, as Unreal's hearing sense does, and `Perception` keeps `heard`, where the last noise came from and `sinceHeard`. Saltline's machines come to a gunshot within 45 m.
- **A ray can meet characters.** `Physics3D.raycast(origin, direction, layers, { characters: true })` stops on a `CharacterController3D`'s capsule where it is now, on Jolt and on PhysX alike; a plain ray still passes them, which line of sight and a camera's boom rely on. Before, nothing could shoot a character that walks.
- **A 3D collider with no RigidBody3D is a static body, as in Unity.** An imported level now collides with a `MeshCollider3D` on each mesh and nothing else; before, a collider without a body built nothing and a character fell through the floor. A collider under an entity with a RigidBody3D is still left to that body, and the collider wireframe marks the same shapes the world builds. Removing an entity's RigidBody3D while its collider stays now leaves a static body rather than none.
diff --git a/sdk/src/core-ui.ts b/sdk/src/core-ui.ts
index 559b57637..f8cb7814d 100644
--- a/sdk/src/core-ui.ts
+++ b/sdk/src/core-ui.ts
@@ -28,6 +28,9 @@ export {
UIVisualFit,
UILayoutGeneration,
UIMask,
+ UIWorldAnchor,
+ placeOnScreen,
+ type UIWorldAnchorData,
measureText,
preloadGlyphs,
type GlyphPreload,
diff --git a/sdk/src/ui/core/world-anchor.ts b/sdk/src/ui/core/world-anchor.ts
new file mode 100644
index 000000000..011956934
--- /dev/null
+++ b/sdk/src/ui/core/world-anchor.ts
@@ -0,0 +1,216 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file world-anchor.ts
+ * @brief A UI node that stands over a point in the 3D world: an objective marker, a name over a head, a health bar.
+ *
+ * @details Placed from the camera's pose as it stands at the end of Update, not from the matrix the camera plugin
+ * resolved at the start of the frame: a first-person view turns in Update, and a marker read off the early
+ * matrix trails the view by a frame and swims as the player looks around.
+ */
+import type { App, Plugin } from '../../app/app';
+import { defineComponent, type ComponentDef, Transform, Camera } from '../../ecs/component';
+import type { TransformData, CameraData } from '../../ecs/component';
+import type { World } from '../../ecs/world';
+import { defineSystem, GetWorld, Schedule } from '../../ecs/system';
+import { Res } from '../../ecs/resource';
+import { SystemLabel } from '../../ecs/systemLabels';
+import type { Entity, Quat, Vec3 } from '../../types';
+import { ProjectionType, ClearFlags } from '../../wasm/wasm.generated';
+import { UINode, UIDisplay, UIPositionType, type UINodeData } from './ui-node';
+import { px } from './dimension';
+import { ScreenLayout, type ScreenLayoutData } from './screen-layout';
+
+export interface UIWorldAnchorData {
+ /** The entity to stand over; 0 = the world point `offset`. */
+ target: Entity;
+ /** Added to the target's world position (world units); the world point itself when there is no target. */
+ offset: Vec3;
+ /** The camera that looks; 0 = the active camera that clears colour with the highest priority (the view on screen). */
+ camera: Entity;
+ /** Off screen or behind the camera, stand at the screen's edge in the point's direction instead (an objective marker). */
+ clampToEdges: boolean;
+ /** How far inside the edge a clamped marker stands, in layout pixels. */
+ edgeMargin: number;
+ /** Behind the camera and not clamped: hide (display None) rather than stand where the projection mirrors it. */
+ hideWhenBehind: boolean;
+ /** Read back: the point is in front of the camera and inside the screen. */
+ onScreen: boolean;
+ /** Read back: the point is behind the camera. */
+ behind: boolean;
+ /** Read back: world distance from the camera. */
+ distance: number;
+}
+
+/**
+ * Puts the UINode it is on at the screen position of a world point, every frame. The node is placed Absolute, by its
+ * left/top insets, in its parent — which is expected to cover the screen (a screen Canvas root or a full-screen
+ * panel). Centre it on the point with negative margins of half its size.
+ *
+ * @experimental
+ */
+export const UIWorldAnchor: ComponentDef<UIWorldAnchorData> = defineComponent<UIWorldAnchorData>('UIWorldAnchor', {
+ target: 0 as Entity,
+ offset: { x: 0, y: 0, z: 0 },
+ camera: 0 as Entity,
+ clampToEdges: false,
+ edgeMargin: 24,
+ hideWhenBehind: true,
+ onScreen: false,
+ behind: false,
+ distance: 0,
+}, {
+ entityFields: ['target', 'camera'],
+ readonlyFields: ['onScreen', 'behind', 'distance'],
+ fields: {
+ target: { tooltip: 'The entity to stand over. Empty: stand over the world point Offset.' },
+ offset: { tooltip: 'Added to the target\'s position (world units): above a head, at a door. The point itself with no target.' },
+ camera: { advanced: true, tooltip: 'The camera that looks. Empty: the camera on screen.' },
+ clampToEdges: { tooltip: 'Off screen, stand at the edge in its direction instead of leaving it.' },
+ edgeMargin: { min: 0, advanced: true, shownWhen: { field: 'clampToEdges', values: [1] } },
+ hideWhenBehind: { advanced: true },
+ onScreen: { advanced: true },
+ behind: { advanced: true },
+ distance: { advanced: true },
+ },
+});
+
+/** The eye a projection is made from: where it is, which way it turns, and its lens. */
+export interface AnchorLens {
+ position: Vec3;
+ rotation: Quat;
+ perspective: boolean;
+ /** Vertical field of view, degrees (perspective). */
+ fov: number;
+ /** Half the visible height, world units (orthographic). */
+ orthoSize: number;
+ near: number;
+}
+
+export interface AnchorPlacement {
+ /** Layout pixels from the left and from the top of the screen box. */
+ left: number;
+ top: number;
+ onScreen: boolean;
+ behind: boolean;
+ distance: number;
+}
+
+function rotateInverse(q: Quat, v: Vec3): Vec3 {
+ // conjugate(q) * v * q, for a unit q
+ const x = -q.x, y = -q.y, z = -q.z, w = q.w;
+ const tx = 2 * (y * v.z - z * v.y), ty = 2 * (z * v.x - x * v.z), tz = 2 * (x * v.y - y * v.x);
+ return { x: v.x + w * tx + (y * tz - z * ty), y: v.y + w * ty + (z * tx - x * tz), z: v.z + w * tz + (x * ty - y * tx) };
+}
+
+/**
+ * Where a world point stands on the screen box, in layout pixels from its top-left corner. Pure: the system below
+ * and the tests share it.
+ *
+ * Behind the eye a perspective projection mirrors the point to the opposite side; a clamped marker is put on the edge
+ * in the point's true direction instead.
+ */
+export function placeOnScreen(point: Vec3, lens: AnchorLens, box: { width: number; height: number },
+ clamp: boolean, margin: number): AnchorPlacement {
+ const rel = { x: point.x - lens.position.x, y: point.y - lens.position.y, z: point.z - lens.position.z };
+ const distance = Math.hypot(rel.x, rel.y, rel.z);
+ const v = rotateInverse(lens.rotation, rel); // camera space: -z ahead
+ const aspect = box.width / Math.max(box.height, 1e-6);
+ let nx: number, ny: number;
+ let behind = false;
+ if (lens.perspective) {
+ const depth = -v.z;
+ behind = depth <= Math.max(lens.near, 1e-4);
+ const t = Math.tan((lens.fov * Math.PI) / 360);
+ const d = behind ? Math.max(Math.abs(depth), 1e-4) : depth;
+ nx = v.x / (d * t * aspect);
+ ny = v.y / (d * t);
+ } else {
+ nx = v.x / (lens.orthoSize * aspect);
+ ny = v.y / lens.orthoSize;
+ }
+ const onScreen = !behind && Math.abs(nx) <= 1 && Math.abs(ny) <= 1;
+ if (clamp && !onScreen) {
+ // Out to the edge along the point's direction from the centre; behind, a point straight ahead of the camera
+ // would read as the centre, so a direction that has nearly vanished is pushed to the bottom edge.
+ let dx = nx, dy = ny;
+ if (behind && Math.hypot(dx, dy) < 1e-3) { dx = 0; dy = -1; }
+ const mx = 1 - (2 * margin) / Math.max(box.width, 1e-6), my = 1 - (2 * margin) / Math.max(box.height, 1e-6);
+ const k = Math.max(Math.abs(dx) / mx, Math.abs(dy) / my, 1e-6);
+ nx = dx / k; ny = dy / k;
+ }
+ return { left: (nx * 0.5 + 0.5) * box.width, top: (0.5 - ny * 0.5) * box.height, onScreen, behind, distance };
+}
+
+/** The camera on screen: the active one that clears colour, highest priority first. */
+function screenCamera(world: World): Entity | null {
+ let best: Entity | null = null, bestPriority = -Infinity;
+ for (const e of world.getEntitiesWithComponents([Camera, Transform])) {
+ const c = world.get(e, Camera) as CameraData;
+ if (!c.isActive || (c.clearFlags & ClearFlags.Color) === 0) continue;
+ if (c.priority > bestPriority) { best = e; bestPriority = c.priority; }
+ }
+ return best;
+}
+
+/** The nodes an anchor hid for standing behind the camera — the only ones it shows again. */
+const hiddenByAnchor = new Set<Entity>();
+
+export function stepWorldAnchors(world: World, screen: ScreenLayoutData): void {
+ if (!screen.valid) return;
+ const anchors = world.getEntitiesWithComponents([UIWorldAnchor, UINode]);
+ if (anchors.length === 0) return;
+ world.ensureTransformsComposed();
+ const box = { width: screen.right - screen.left, height: screen.top - screen.bottom };
+ const auto = screenCamera(world);
+ for (const e of anchors) {
+ const a = world.get(e, UIWorldAnchor) as UIWorldAnchorData;
+ const camEntity = a.camera !== 0 && world.valid(a.camera) && world.has(a.camera, Camera) ? a.camera : auto;
+ if (camEntity === null) continue;
+ const cam = world.get(camEntity, Camera) as CameraData;
+ const ct = world.get(camEntity, Transform) as TransformData;
+ const lens: AnchorLens = {
+ position: ct.worldPosition as Vec3, rotation: ct.worldRotation as Quat,
+ perspective: cam.projectionType === ProjectionType.Perspective,
+ fov: cam.fov, orthoSize: cam.orthoSize, near: cam.nearPlane,
+ };
+ let point: Vec3 = a.offset;
+ if (a.target !== 0 && world.valid(a.target) && world.has(a.target, Transform)) {
+ const tp = (world.get(a.target, Transform) as TransformData).worldPosition as Vec3;
+ point = { x: tp.x + a.offset.x, y: tp.y + a.offset.y, z: tp.z + a.offset.z };
+ }
+ const at = placeOnScreen(point, lens, box, a.clampToEdges, a.edgeMargin);
+ const hide = at.behind && !a.clampToEdges && a.hideWhenBehind;
+ const node = world.get(e, UINode) as UINodeData;
+ const left = Math.round(at.left * 10) / 10, top = Math.round(at.top * 10) / 10;
+ // The anchor hides a node only while the point is behind, and shows again only a node IT hid: a game that hid a
+ // marker itself (an objective done) keeps it hidden.
+ let display = node.display;
+ if (hide && node.display !== UIDisplay.None) { display = UIDisplay.None; hiddenByAnchor.add(e); }
+ else if (!hide && hiddenByAnchor.has(e)) { display = UIDisplay.Flex; hiddenByAnchor.delete(e); }
+ // Written only when it changed: a write dirties the layout tree.
+ if (node.position !== UIPositionType.Absolute || node.display !== display
+ || node.insetLeft.value !== left || node.insetTop.value !== top) {
+ world.update(e, UINode, (n: UINodeData) => {
+ n.position = UIPositionType.Absolute; n.display = display;
+ n.insetLeft = px(left); n.insetTop = px(top);
+ });
+ }
+ if (a.onScreen !== at.onScreen || a.behind !== at.behind || Math.abs(a.distance - at.distance) > 1e-3) {
+ world.update(e, UIWorldAnchor, (w: UIWorldAnchorData) => { w.onScreen = at.onScreen; w.behind = at.behind; w.distance = at.distance; });
+ }
+ }
+}
+
+export class UIWorldAnchorPlugin implements Plugin {
+ name = 'uiWorldAnchor';
+ build(app: App): void {
+ app.addSystemToSchedule(Schedule.PostUpdate, defineSystem(
+ [GetWorld(), Res(ScreenLayout)],
+ (world: World, screen: ScreenLayoutData) => stepWorldAnchors(world, screen),
+ { name: 'UIWorldAnchorSystem' },
+ ), { runBefore: [SystemLabel.UILayoutLate] });
+ }
+}
+
+export const uiWorldAnchorPlugin = new UIWorldAnchorPlugin();
diff --git a/sdk/src/ui/index.ts b/sdk/src/ui/index.ts
index 7a0d488f4..254258514 100644
--- a/sdk/src/ui/index.ts
+++ b/sdk/src/ui/index.ts
@@ -60,6 +60,7 @@ export {
} from './core/ui-visual';
export { UIMask, MaskMode, type UIMaskData } from './core/ui-mask';
+export { UIWorldAnchor, placeOnScreen, type UIWorldAnchorData, type AnchorLens, type AnchorPlacement } from './core/world-anchor';
export { UIScroll, ScrollMovement, type UIScrollData } from './core/ui-scroll';
export { ContentRect, applyContentRect, contentRectContains, localPointInRect, type ContentRectData } from './core/content-rect';
export { contentRectPlugin, ContentRectPlugin } from './render/content-rect';
diff --git a/sdk/src/ui/ui-plugin.ts b/sdk/src/ui/ui-plugin.ts
index 68882dcba..acfe9ded7 100644
--- a/sdk/src/ui/ui-plugin.ts
+++ b/sdk/src/ui/ui-plugin.ts
@@ -31,6 +31,7 @@ import { focusPlugin } from './input/focus';
import { textInputPlugin } from './text/text-input-plugin';
import { uiRenderOrderPlugin } from './render/render-order';
import { uiVisibilityPlugin } from './core/visibility';
+import { uiWorldAnchorPlugin } from './core/world-anchor';
import { UICollections } from './collection/collection-host';
import { installTreeViews } from './widgets/tree-view';
import { installColorPickers } from './behavior/color-picker-system';
@@ -52,6 +53,7 @@ const SUB_PLUGINS: Plugin[] = [
inlineImagePlugin,
uiInteractionPlugin,
uiVisibilityPlugin,
+ uiWorldAnchorPlugin,
uiBehaviorPlugin,
uiControllerPlugin,
dragPlugin,
diff --git a/sdk/tests/ui-world-anchor.test.ts b/sdk/tests/ui-world-anchor.test.ts
new file mode 100644
index 000000000..e620d311d
--- /dev/null
+++ b/sdk/tests/ui-world-anchor.test.ts
@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file A UI node over a point in the world: where the projection puts it, and what an off-screen marker does.
+ */
+import { describe, it, expect } from 'vitest';
+import { placeOnScreen, type AnchorLens } from '../src/ui/core/world-anchor';
+
+const BOX = { width: 1600, height: 900 };
+// An eye at the origin looking down -z, 90 degrees tall: at depth d the screen reaches d up and d * 16/9 across.
+const eye = (over: Partial<AnchorLens> = {}): AnchorLens => ({
+ position: { x: 0, y: 0, z: 0 }, rotation: { x: 0, y: 0, z: 0, w: 1 }, perspective: true, fov: 90, orthoSize: 5, near: 0.1, ...over,
+});
+
+describe('placeOnScreen', () => {
+ it('puts a point straight ahead at the centre, at its distance', () => {
+ const p = placeOnScreen({ x: 0, y: 0, z: -10 }, eye(), BOX, false, 0);
+ expect([p.left, p.top]).toEqual([800, 450]);
+ expect(p.onScreen).toBe(true);
+ expect(p.distance).toBeCloseTo(10);
+ });
+
+ it('puts a point where the lens says: half way to the right edge, a quarter of the way up', () => {
+ const p = placeOnScreen({ x: (10 * 16) / 9 / 2, y: 2.5, z: -10 }, eye(), BOX, false, 0);
+ expect(p.left).toBeCloseTo(1200);
+ expect(p.top).toBeCloseTo(337.5);
+ });
+
+ it('turns with the eye: a quarter turn left brings a point on the -x axis to the centre', () => {
+ const left = { x: 0, y: Math.SQRT1_2, z: 0, w: Math.SQRT1_2 };
+ const p = placeOnScreen({ x: -10, y: 0, z: 0 }, eye({ rotation: left, position: { x: 0, y: 0, z: 0 } }), BOX, false, 0);
+ expect(p.left).toBeCloseTo(800);
+ expect(p.onScreen).toBe(true);
+ });
+
+ it('says a point behind is behind, and a clamped marker stands on the edge on the side it really is', () => {
+ const free = placeOnScreen({ x: 5, y: 0, z: 10 }, eye(), BOX, false, 0);
+ expect(free.behind).toBe(true);
+ expect(free.onScreen).toBe(false);
+ const pinned = placeOnScreen({ x: 5, y: 0, z: 10 }, eye(), BOX, true, 20);
+ expect(pinned.left).toBeCloseTo(1580); // right edge less the margin: the point is to the right
+ const ahead = placeOnScreen({ x: 0, y: 0, z: 10 }, eye(), BOX, true, 20);
+ expect(ahead.top).toBeCloseTo(880); // straight behind: the bottom edge, not the centre
+ });
+
+ it('pins an off-screen point in front to the edge, keeping its direction', () => {
+ const p = placeOnScreen({ x: 100, y: 10, z: -10 }, eye(), BOX, true, 0);
+ expect(p.left).toBeCloseTo(1600);
+ expect(p.top).toBeGreaterThan(0);
+ expect(p.top).toBeLessThan(450);
+ });
+
+ it('projects orthographically: half the size up is the top edge', () => {
+ const p = placeOnScreen({ x: 0, y: 5, z: -3 }, eye({ perspective: false }), BOX, false, 0);
+ expect([p.left, p.top]).toEqual([800, 0]);
+ });
+});