A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(look): a part of a picture traced as points, its edges as lines, and the colour of a patch
yhh07fd9d449074
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7a2e54c08..df3a92d16 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -22,6 +22,8 @@ published separately; it ships inside the editor.- **Glyphs can be put on the atlas before they are shown.** The first page of a novel is hundreds of new characters, all landing in one frame. `preloadGlyphs(app, label, text, { budgetMs })` makes them ahead of time, a slice a frame, drawn as that `Text` will draw them, and returns `progress`, `done` and `release()`. On a Snapdragon 8 Elite a book's 2992 characters at 28 px took 145 ms over 25 frames with the worst frame at 9.5 ms, after which the first page's frame went from 49 to 11 ms and no page made a new glyph. Preloaded glyphs hold their atlas pages (4 MB each, a few hundred characters at that size) until `release()`.- **Voxel GI lights what the camera does not see, and bounces light more than once.** The cache now reads the sun's shadow map back into the world, so a surface the sun lights behind or beside the camera still sends light into the view. Each surface also gives off again the bounce light it received, so a corner or a floor in shade gets light that took two or more bounces to reach it. A ray now brings what it meets at full strength however far it went (it used to fade with distance, 40% at 4 m with 12 cm voxels). Each voxel is weighed by how much of it a surface covers, so a bright patch no longer lights like the whole voxel around it. Together these keep the result close to a path tracer's whatever the voxel size. In a sunlit test room the whole frame carries 0.95 of Cycles' light at the default size and 0.74 at 12 cm voxels; before, it was 0.54 at the default and darker the finer the grid. With the lit patch behind the camera, the walls went from no bounce light to 25–57%. A closed room the sun only lights from outside stays dark. Two inspector controls tune it: **Off-screen Sun Albedo** (the colour the shadow map cannot give, assumed) and **Multi-bounce**. A fine grid no longer drops every voxel: a count truncated by the cache's ageing had kept them below the two samples a ray asks for. The WebGL2 fallback does the same in its own passes: each voxel asks the sun's map whether a sunlit surface stands in it, and traces two rays through last frame's grid for the light it is given. In the test room it carries 1.00 of Cycles' light at the default size (it was 0.56) and 0.80 at 12 cm; with the lit patch behind the camera 0.20 (it was 0). Its voxel pass costs about 1.3 ms more at 720p on a desktop.+- **An AI can read shapes and colours off a picture as numbers.** `reference_trace` grows a part of a reference from places inside it and answers its outline as points, ready for an `outline`, or its strongest edges as lines, ready for a `sculpt` press. `reference_probe` answers the colour at the same places in the reference and the game. `reference_compare` with `overlay: "parts"` draws only the outline of the entities named on the reference. Each shows what it found on the picture, and judges nothing.+- **An AI can take a frame apart to see why it looks wrong.** `reference_compare` draws a `region` close through the camera, from the same place and at full size, where it used to cut one out of a smaller picture and hide what a surface was doing. `camera_layers` shows one frame as the shape under a plain light, the shape under the scene's lights, the paint, the triangles and the scene before its effects. `light_sheet` shows it with each light alone. Each puts everything back.- **A fixed shadow map stays on what casts, and ambient occlusion leaves hair alone.** A sun's `shadowExtent` was centred on every mesh, so a far backdrop with `castShadows: false` took the map away from the character in front of it and her shadows disappeared; it is centred on what casts now, and the docs, which said the camera, say so. `gtao` took a strand in front of a surface for a wall and turned what was behind it black: a horizon now falls back when what lies beyond is lower, and a pixel-wide strand no longer has a normal guessed from depth. On a test head the black went from 1.17% of lit pixels to 0.07%, and shade round a solid block stays.diff --git a/desktop b/desktopindex cc024ba69..acc2c856b 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit cc024ba693bcca134b044683cf7123017475d433+Subproject commit acc2c856baf0ae660a98bcdbc5647e5f74cb038cdiff --git a/pipeline/src/look/pictures.ts b/pipeline/src/look/pictures.tsindex edf7c72f1..92e0a46d8 100644--- a/pipeline/src/look/pictures.ts+++ b/pipeline/src/look/pictures.ts@@ -228,7 +228,7 @@ const GLYPHS: Record<string, readonly number[]> = {const GLYPH_SCALE = 2;const RULER_WIDTH = 1400;-function label(image: RgbaImage, text: string, x: number, y: number): void {+export function label(image: RgbaImage, text: string, x: number, y: number): void {const { width, height, rgba } = image;const w = text.length * 4 * GLYPH_SCALE + GLYPH_SCALE, h = 7 * GLYPH_SCALE;const set = (px: number, py: number, v: number): void => {diff --git a/pipeline/src/look/trace.ts b/pipeline/src/look/trace.tsnew file mode 100644index 000000000..e21c662d0--- /dev/null+++ b/pipeline/src/look/trace.ts@@ -0,0 +1,410 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file trace.ts — shapes read out of a picture as lines of points.+ *+ * An eye reading coordinates off a grid is out by a centimetre, which a face shows. So a part of a picture is+ * grown from a place inside it, by colour, and its outline comes back as points; or the strongest edges come back as+ * lines, for folds and strands. Nothing here says whether what it found is right: the caller looks at it on the picture.+ * Pure: pixels in, points out.+ */+import type { RgbaImage } from './pictures';++export type PixelPoint = readonly [number, number];++const toLinear = (c: number): number => (c <= 0.04045 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4);+const labF = (t: number): number => (t > 0.008856 ? Math.cbrt(t) : 7.787 * t + 16 / 116);+const LINEAR = Float64Array.from({ length: 256 }, (_, i) => toLinear(i / 255));++/** The picture in CIE Lab, softened by a pixel: a video frame's compression would otherwise break every region into specks. */+function labOf(image: RgbaImage): Float32Array {+ const { width, height, rgba } = image;+ const out = new Float32Array(width * height * 3);+ for (let y = 0; y < height; y++) {+ for (let x = 0; x < width; x++) {+ let [r, g, b, n] = [0, 0, 0, 0];+ for (let dy = -1; dy <= 1; dy++) {+ for (let dx = -1; dx <= 1; dx++) {+ const [qx, qy] = [x + dx, y + dy];+ if (qx < 0 || qy < 0 || qx >= width || qy >= height) continue;+ const at = (qy * width + qx) * 4;+ r += LINEAR[rgba[at]!]!; g += LINEAR[rgba[at + 1]!]!; b += LINEAR[rgba[at + 2]!]!; n++;+ }+ }+ [r, g, b] = [r / n, g / n, b / n];+ const fx = labF((0.4124 * r + 0.3576 * g + 0.1805 * b) / 0.95047);+ const fy = labF(0.2126 * r + 0.7152 * g + 0.0722 * b);+ const fz = labF((0.0193 * r + 0.1192 * g + 0.9505 * b) / 1.08883);+ const to = (y * width + x) * 3;+ out[to] = 116 * fy - 16; out[to + 1] = 500 * (fx - fy); out[to + 2] = 200 * (fy - fz);+ }+ }+ return out;+}++/** Morphology by a square: true grows what is set, false shrinks it. */+function widen(mask: Uint8Array, width: number, height: number, radius: number, grow: boolean): Uint8Array {+ if (radius <= 0) return mask;+ const pass = (from: Uint8Array, horizontal: boolean): Uint8Array => {+ const out = new Uint8Array(from.length);+ for (let y = 0; y < height; y++) {+ for (let x = 0; x < width; x++) {+ let hit = grow ? 0 : 1;+ for (let d = -radius; d <= radius; d++) {+ const [qx, qy] = horizontal ? [x + d, y] : [x, y + d];+ // Off the picture counts as more of what is there: a part cut by the edge of a crop is not eaten from it.+ const v = qx < 0 || qy < 0 || qx >= width || qy >= height ? (grow ? 0 : 1) : from[qy * width + qx]!;+ if (grow ? v : !v) { hit = grow ? 1 : 0; break; }+ }+ out[y * width + x] = hit;+ }+ }+ return out;+ };+ return pass(pass(mask, true), false);+}++export interface GrowOptions {+ /** How far a colour may be from the seed's and still be the same thing, as a difference in Lab. Default 14: about what shading across one cloth comes to. */+ tolerance?: number;+ /** Pixels of fuzz taken off the outline, closing small gaps and dropping thin spurs. Default 2. */+ smooth?: number;+ /** Places known to be outside, which stop it where a colour alone would run on. */+ not?: readonly PixelPoint[];+}++/** What is connected to the seeds and looks like one of them, with its holes filled: a face is one thing with an eye in it. */+export function growRegion(image: RgbaImage, seeds: readonly PixelPoint[], options: GrowOptions = {}): Uint8Array {+ const { width, height } = image;+ if (seeds.length === 0) throw new Error('give at least one place inside what to trace');+ const inPicture = ([x, y]: PixelPoint): boolean => x >= 0 && y >= 0 && x < width && y < height;+ for (const seed of seeds) if (!inPicture(seed)) throw new Error(`a seed at ${seed[0].toFixed(1)}, ${seed[1].toFixed(1)} is off the picture, ${width} by ${height}`);+ const tolerance = options.tolerance ?? 14;+ if (!(tolerance > 0)) throw new Error(`tolerance ${tolerance} must be positive`);+ const smooth = options.smooth ?? 2;+ const lab = labOf(image);+ const colourAt = ([sx, sy]: PixelPoint): [number, number, number] => {+ const sum = [0, 0, 0];+ let n = 0;+ for (let y = Math.max(0, Math.round(sy) - 2); y <= Math.min(height - 1, Math.round(sy) + 2); y++) {+ for (let x = Math.max(0, Math.round(sx) - 2); x <= Math.min(width - 1, Math.round(sx) + 2); x++) {+ const at = (y * width + x) * 3;+ sum[0]! += lab[at]!; sum[1]! += lab[at + 1]!; sum[2]! += lab[at + 2]!; n++;+ }+ }+ return [sum[0]! / n, sum[1]! / n, sum[2]! / n];+ };+ const targets = seeds.map(colourAt);+ const avoid = (options.not ?? []).filter(inPicture).map(colourAt);+ const limit = tolerance * tolerance;+ const distance = (at: number, c: readonly number[]): number => (lab[at]! - c[0]!) ** 2 + (lab[at + 1]! - c[1]!) ** 2 + (lab[at + 2]! - c[2]!) ** 2;+ const fits = (i: number): boolean => {+ const at = i * 3;+ let best = Infinity;+ for (const t of targets) best = Math.min(best, distance(at, t));+ if (best > limit) return false;+ // Nearer something known to be outside than anything known to be in: not part of it.+ for (const a of avoid) if (distance(at, a) < best) return false;+ return true;+ };+ const flood = (allowed: (i: number) => boolean, starts: readonly number[]): Uint8Array => {+ const seen = new Uint8Array(width * height);+ const stack: number[] = [];+ for (const s of starts) if (!seen[s]) { seen[s] = 1; stack.push(s); }+ while (stack.length) {+ const i = stack.pop()!;+ const [x, y] = [i % width, Math.floor(i / width)];+ for (const [dx, dy] of [[1, 0], [-1, 0], [0, 1], [0, -1]] as const) {+ const [qx, qy] = [x + dx, y + dy];+ if (qx < 0 || qy < 0 || qx >= width || qy >= height) continue;+ const q = qy * width + qx;+ if (seen[q] || !allowed(q)) continue;+ seen[q] = 1;+ stack.push(q);+ }+ }+ return seen;+ };+ const starts = seeds.map(([x, y]) => Math.round(y) * width + Math.round(x)).map((i) => Math.min(i, width * height - 1));+ const grown = flood(fits, starts);+ // Closed, then opened: gaps go first, so a thin place is not cut through by the second.+ const tidy = widen(widen(widen(grown, width, height, smooth, true), width, height, smooth * 2, false), width, height, smooth, true);+ // Only what is still joined to a seed; a seed the tidying ate holds its own place.+ for (const s of starts) tidy[s] = 1;+ const kept = flood((i) => tidy[i] === 1, starts);+ // The holes: whatever the outside cannot reach.+ const border: number[] = [];+ for (let x = 0; x < width; x++) { if (!kept[x]) border.push(x); if (!kept[(height - 1) * width + x]) border.push((height - 1) * width + x); }+ for (let y = 0; y < height; y++) { if (!kept[y * width]) border.push(y * width); if (!kept[y * width + width - 1]) border.push(y * width + width - 1); }+ const outside = flood((i) => !kept[i], border);+ const filled = new Uint8Array(width * height);+ for (let i = 0; i < filled.length; i++) filled[i] = outside[i] ? 0 : 1;+ return filled;+}++const CLOCKWISE: readonly PixelPoint[] = [[1, 0], [1, 1], [0, 1], [-1, 1], [-1, 0], [-1, -1], [0, -1], [1, -1]];++/** The outer edge of the largest piece of a mask, as a walk round its border pixels, clockwise on screen. */+export function outlineOfMask(mask: Uint8Array, width: number, height: number): PixelPoint[] {+ // The largest piece, then its top-left pixel: the first of it a scan meets, whose neighbours above and to the left are not in it.+ const label = new Int32Array(width * height).fill(-1);+ let best = -1, bestSize = 0, bestStart = -1, next = 0;+ for (let i = 0; i < mask.length; i++) {+ if (!mask[i] || label[i] !== -1) continue;+ const stack = [i];+ label[i] = next;+ let size = 0;+ while (stack.length) {+ const at = stack.pop()!;+ size++;+ const [x, y] = [at % width, Math.floor(at / width)];+ for (const [dx, dy] of [[1, 0], [-1, 0], [0, 1], [0, -1]] as const) {+ const [qx, qy] = [x + dx, y + dy];+ if (qx < 0 || qy < 0 || qx >= width || qy >= height) continue;+ const q = qy * width + qx;+ if (mask[q] && label[q] === -1) { label[q] = next; stack.push(q); }+ }+ }+ if (size > bestSize) { best = next; bestSize = size; bestStart = i; }+ next++;+ }+ if (best < 0) return [];+ const isIn = (x: number, y: number): boolean => x >= 0 && y >= 0 && x < width && y < height && label[y * width + x] === best;+ const start: PixelPoint = [bestStart % width, Math.floor(bestStart / width)];+ const out: PixelPoint[] = [start];+ let [x, y] = start;+ // Came from the left: nothing is there, as this is the first of it in its row.+ let from = 4;+ const firstFrom = from;+ for (let guard = 0; guard < mask.length * 4; guard++) {+ let moved = false;+ for (let turn = 1; turn <= 8; turn++) {+ const dir = (from + turn) % 8;+ const [dx, dy] = CLOCKWISE[dir]!;+ if (!isIn(x + dx, y + dy)) continue;+ x += dx; y += dy;+ // Looking back the way it came, less one, so as to hug the outside.+ from = (dir + 4) % 8;+ moved = true;+ break;+ }+ if (!moved) break;+ if (x === start[0] && y === start[1]) {+ // Home, and about to leave the way it left the first time: the walk is closed. A thin neck is passed twice, so home alone is not enough.+ let first = -1;+ for (let turn = 1; turn <= 8; turn++) {+ const dir = (from + turn) % 8;+ const [dx, dy] = CLOCKWISE[dir]!;+ if (isIn(x + dx, y + dy)) { first = dir; break; }+ }+ let original = -1;+ for (let turn = 1; turn <= 8; turn++) {+ const dir = (firstFrom + turn) % 8;+ const [dx, dy] = CLOCKWISE[dir]!;+ if (isIn(start[0] + dx, start[1] + dy)) { original = dir; break; }+ }+ if (first === original) break;+ }+ out.push([x, y]);+ }+ return out;+}++/** Fewer points along much the same line: none of those dropped was further than @p tolerance from it. */+export function simplifyLine(points: readonly PixelPoint[], tolerance: number, closed = false): PixelPoint[] {+ if (points.length < 3) return [...points];+ const keep = new Uint8Array(points.length);+ const distanceFrom = (p: PixelPoint, a: PixelPoint, b: PixelPoint): number => {+ const [ex, ey] = [b[0] - a[0], b[1] - a[1]];+ const len = ex * ex + ey * ey;+ const t = len > 0 ? Math.max(0, Math.min(1, ((p[0] - a[0]) * ex + (p[1] - a[1]) * ey) / len)) : 0;+ return Math.hypot(p[0] - a[0] - ex * t, p[1] - a[1] - ey * t);+ };+ const run = (lo: number, hi: number): void => {+ const stack: [number, number][] = [[lo, hi]];+ while (stack.length) {+ const [a, b] = stack.pop()!;+ let far = -1, worst = tolerance;+ for (let i = a + 1; i < b; i++) {+ const d = distanceFrom(points[i]!, points[a]!, points[b]!);+ if (d > worst) { worst = d; far = i; }+ }+ if (far >= 0) { keep[far] = 1; stack.push([a, far], [far, b]); }+ }+ };+ if (closed) {+ // A closed line has no ends: the two points furthest apart serve.+ let split = 0, widest = -1;+ for (let i = 0; i < points.length; i++) {+ const d = Math.hypot(points[i]![0] - points[0]![0], points[i]![1] - points[0]![1]);+ if (d > widest) { widest = d; split = i; }+ }+ keep[0] = 1; keep[split] = 1;+ if (split > 0) run(0, split);+ const wrapped = [...points.slice(split), points[0]!];+ const inner = simplifyLine(wrapped, tolerance, false);+ const firstHalf = points.filter((_, i) => i <= split && keep[i]);+ return [...firstHalf, ...inner.slice(1, -1)];+ }+ keep[0] = 1; keep[points.length - 1] = 1;+ run(0, points.length - 1);+ return points.filter((_, i) => keep[i]);+}++export interface EdgeLineOptions {+ /** The share of the picture's pixels that may be edge, strongest first. Default 0.04. */+ share?: number;+ /** Lines shorter than this many pixels are left out. Default 24. */+ minLength?: number;+ /** How far a line may be simplified from what was found, in pixels. Default 1.5. */+ simplify?: number;+ /** At most this many lines come back, longest first. Default 24. */+ most?: number;+}++/** The strongest edges of a picture as lines, one pixel wide and joined up, longest first: folds, strands, the edge of a sleeve. */+export function traceEdgeLines(image: RgbaImage, options: EdgeLineOptions = {}): PixelPoint[][] {+ const { width, height } = image;+ const lab = labOf(image);+ const light = new Float32Array(width * height);+ // Softened further: what a fold is, is broader than what a video's blocks are.+ for (let y = 0; y < height; y++) {+ for (let x = 0; x < width; x++) {+ let [sum, n] = [0, 0];+ for (let dy = -2; dy <= 2; dy++) {+ for (let dx = -2; dx <= 2; dx++) {+ const [qx, qy] = [x + dx, y + dy];+ if (qx < 0 || qy < 0 || qx >= width || qy >= height) continue;+ sum += lab[(qy * width + qx) * 3]!; n++;+ }+ }+ light[y * width + x] = sum / n;+ }+ }+ const strength = new Float32Array(width * height);+ const gx = new Float32Array(width * height), gy = new Float32Array(width * height);+ for (let y = 1; y < height - 1; y++) {+ for (let x = 1; x < width - 1; x++) {+ const at = y * width + x;+ gx[at] = light[at + 1]! - light[at - 1]!;+ gy[at] = light[at + width]! - light[at - width]!;+ strength[at] = Math.hypot(gx[at]!, gy[at]!);+ }+ }+ // Only the crest of each edge: one pixel wide, so it can be followed.+ const crest = new Float32Array(width * height);+ for (let y = 1; y < height - 1; y++) {+ for (let x = 1; x < width - 1; x++) {+ const at = y * width + x;+ const s = strength[at]!;+ if (s <= 0) continue;+ const [ux, uy] = [Math.round(gx[at]! / s), Math.round(gy[at]! / s)];+ if (s >= strength[at + uy * width + ux]! && s >= strength[at - uy * width - ux]!) crest[at] = s;+ }+ }+ const sorted = Float32Array.from(crest.filter((v) => v > 0)).sort();+ if (sorted.length === 0) return [];+ const share = options.share ?? 0.04;+ const strong = sorted[Math.max(0, sorted.length - Math.max(1, Math.floor(width * height * share)))]!;+ const weak = strong * 0.5;+ const used = new Uint8Array(width * height);+ const isEdge = (x: number, y: number): boolean => x > 0 && y > 0 && x < width - 1 && y < height - 1 && crest[y * width + x]! >= weak && !used[y * width + x];+ const follow = (x0: number, y0: number, dx0: number, dy0: number): PixelPoint[] => {+ const line: PixelPoint[] = [];+ let [x, y, dx, dy] = [x0, y0, dx0, dy0];+ for (;;) {+ let bestScore = -Infinity, bx = 0, by = 0;+ for (const [ax, ay] of CLOCKWISE) {+ if (!isEdge(x + ax, y + ay)) continue;+ // Straight on, by preference: a crossing should not turn a line round a corner.+ const score = (dx * ax + dy * ay) / Math.hypot(ax, ay);+ if (score > bestScore) { bestScore = score; bx = ax; by = ay; }+ }+ if (bestScore === -Infinity || bestScore < -0.2) return line;+ x += bx; y += by; dx = dx * 0.7 + bx * 0.3; dy = dy * 0.7 + by * 0.3;+ used[y * width + x] = 1;+ line.push([x, y]);+ }+ };+ const lines: PixelPoint[][] = [];+ const order: number[] = [];+ for (let i = 0; i < crest.length; i++) if (crest[i]! >= strong) order.push(i);+ order.sort((a, b) => crest[b]! - crest[a]!);+ for (const i of order) {+ if (used[i]) continue;+ const [x, y] = [i % width, Math.floor(i / width)];+ used[i] = 1;+ // Along the edge: square to which way the light changes.+ const s = strength[i]! || 1;+ const [ex, ey] = [-gy[i]! / s, gx[i]! / s];+ const ahead = follow(x, y, ex, ey), behind = follow(x, y, -ex, -ey);+ lines.push([...behind.reverse(), [x, y], ...ahead]);+ }+ const minLength = options.minLength ?? 24;+ const simplify = options.simplify ?? 1.5;+ return lines.filter((l) => l.length >= minLength)+ .sort((a, b) => b.length - a.length)+ .slice(0, options.most ?? 24)+ .map((l) => simplifyLine(l, simplify, false));+}++export interface PatchColor { hex: string; linear: [number, number, number] }++/** The colour of a small round patch of a picture, averaged in light rather than in the numbers that encode it. */+export function colorOfPatch(image: RgbaImage, x: number, y: number, radius: number): PatchColor {+ const { width, height, rgba } = image;+ let [r, g, b, n] = [0, 0, 0, 0];+ for (let qy = Math.max(0, Math.floor(y - radius)); qy <= Math.min(height - 1, Math.ceil(y + radius)); qy++) {+ for (let qx = Math.max(0, Math.floor(x - radius)); qx <= Math.min(width - 1, Math.ceil(x + radius)); qx++) {+ if (Math.hypot(qx - x, qy - y) > radius + 0.5) continue;+ const at = (qy * width + qx) * 4;+ r += LINEAR[rgba[at]!]!; g += LINEAR[rgba[at + 1]!]!; b += LINEAR[rgba[at + 2]!]!; n++;+ }+ }+ if (n === 0) throw new Error(`${x.toFixed(1)}, ${y.toFixed(1)} is off the picture, ${width} by ${height}`);+ const linear: [number, number, number] = [r / n, g / n, b / n];+ const encode = (c: number): number => Math.round(255 * (c <= 0.0031308 ? c * 12.92 : 1.055 * c ** (1 / 2.4) - 0.055));+ return { hex: `#${linear.map((c) => encode(c).toString(16).padStart(2, '0')).join('')}`, linear: [Number(linear[0].toFixed(4)), Number(linear[1].toFixed(4)), Number(linear[2].toFixed(4))] };+}++/** A line drawn on a picture in place. */+export function drawLine(image: RgbaImage, points: readonly PixelPoint[], color: readonly [number, number, number], closed = false): void {+ const { width, height, rgba } = image;+ const put = (x: number, y: number): void => {+ if (x < 0 || y < 0 || x >= width || y >= height) return;+ rgba.set(color, (y * width + x) * 4);+ };+ const count = closed ? points.length : points.length - 1;+ for (let i = 0; i < count; i++) {+ const [a, b] = [points[i]!, points[(i + 1) % points.length]!];+ const steps = Math.max(1, Math.ceil(Math.hypot(b[0] - a[0], b[1] - a[1])));+ for (let s = 0; s <= steps; s++) {+ const [x, y] = [Math.round(a[0] + ((b[0] - a[0]) * s) / steps), Math.round(a[1] + ((b[1] - a[1]) * s) / steps)];+ put(x, y); put(x + 1, y); put(x, y + 1);+ }+ }+}++/** The mask of what differs between two pictures of one size. */+export function differenceMask(a: RgbaImage, b: RgbaImage, least = 6): Uint8Array {+ const out = new Uint8Array(a.width * a.height);+ for (let i = 0; i < out.length; i++) {+ const at = i * 4;+ out[i] = Math.abs(a.rgba[at]! - b.rgba[at]!) + Math.abs(a.rgba[at + 1]! - b.rgba[at + 1]!) + Math.abs(a.rgba[at + 2]! - b.rgba[at + 2]!) > least ? 1 : 0;+ }+ return out;+}++/** A picture with the border of a mask drawn on it, whatever the two sizes. */+export function maskEdgeOver(under: RgbaImage, mask: Uint8Array, maskWidth: number, maskHeight: number, color: readonly [number, number, number]): RgbaImage {+ const { width, height } = under;+ const rgba = Uint8Array.from(under.rgba);+ const at = (x: number, y: number): number => (mask[Math.min(maskHeight - 1, Math.floor((y * maskHeight) / height)) * maskWidth + Math.min(maskWidth - 1, Math.floor((x * maskWidth) / width))]!);+ for (let y = 0; y < height; y++) {+ for (let x = 0; x < width; x++) {+ const here = at(x, y);+ if (here && (!at(x - (x > 0 ? 1 : 0), y) || !at(x + (x < width - 1 ? 1 : 0), y) || !at(x, y - (y > 0 ? 1 : 0)) || !at(x, y + (y < height - 1 ? 1 : 0)))) rgba.set(color, (y * width + x) * 4);+ }+ }+ return { width, height, rgba };+}diff --git a/pipeline/tests/look-trace.test.ts b/pipeline/tests/look-trace.test.tsnew file mode 100644index 000000000..1347f39c6--- /dev/null+++ b/pipeline/tests/look-trace.test.ts@@ -0,0 +1,243 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/** Shapes read out of a picture: the outline of what was pointed at, in the places it is, and the strongest edges as lines. */+import { describe, expect, it } from 'vitest';+import { colorOfPatch, differenceMask, drawLine, growRegion, maskEdgeOver, outlineOfMask, simplifyLine, traceEdgeLines, type PixelPoint } from '../src/look/trace';+import type { RgbaImage } from '../src/look/pictures';++type Shade = (x: number, y: number) => readonly [number, number, number];+function image(width: number, height: number, shade: Shade): RgbaImage {+ const rgba = new Uint8Array(width * height * 4);+ for (let y = 0; y < height; y++) {+ for (let x = 0; x < width; x++) {+ const c = shade(x, y);+ rgba.set([c[0], c[1], c[2], 255], (y * width + x) * 4);+ }+ }+ return { width, height, rgba };+}+const BG = [20, 40, 90] as const, FG = [200, 150, 130] as const;+const disc = (cx: number, cy: number, r: number): Shade => (x, y) => (Math.hypot(x - cx, y - cy) <= r ? FG : BG);+/** The area and the middle of a closed line. */+const measure = (line: readonly PixelPoint[]): { area: number; cx: number; cy: number } => {+ let [area, cx, cy] = [0, 0, 0];+ line.forEach((p, i) => {+ const q = line[(i + 1) % line.length]!;+ const cross = p[0] * q[1] - q[0] * p[1];+ area += cross; cx += (p[0] + q[0]) * cross; cy += (p[1] + q[1]) * cross;+ });+ return { area: Math.abs(area / 2), cx: cx / (3 * area), cy: cy / (3 * area) };+};+const fromMask = (mask: Uint8Array, w: number, h: number): PixelPoint[] => outlineOfMask(mask, w, h);++describe('growRegion and outlineOfMask', () => {+ it('finds what the seed is inside, in the place it is', () => {+ const pic = image(200, 160, disc(90, 70, 40));+ const line = fromMask(growRegion(pic, [[90, 70]]), 200, 160);+ const m = measure(line);+ expect(m.cx).toBeCloseTo(90, 0);+ expect(m.cy).toBeCloseTo(70, 0);+ // Walked along border pixels, so a little inside the true edge.+ expect(m.area).toBeGreaterThan(Math.PI * 38 * 38);+ expect(m.area).toBeLessThan(Math.PI * 41 * 41);+ });++ it('goes round a concave shape and into its notch', () => {+ const shape: Shade = (x, y) => (x >= 20 && x < 120 && y >= 20 && y < 100 && !(x >= 60 && x < 80 && y < 70) ? FG : BG);+ const line = fromMask(growRegion(image(140, 120, shape), [[40, 50]], { smooth: 0 }), 140, 120);+ expect(measure(line).area).toBeCloseTo(99 * 79 - 18 * 50, -3);+ // The floor of the notch is on the outline.+ expect(line.some(([x, y]) => x > 62 && x < 78 && Math.abs(y - 70) <= 1)).toBe(true);+ });++ it('stops at what is not joined to it: one seed, one of two discs', () => {+ const two: Shade = (x, y) => (Math.hypot(x - 50, y - 50) <= 30 || Math.hypot(x - 150, y - 50) <= 30 ? FG : BG);+ const mask = growRegion(image(200, 100, two), [[50, 50]]);+ expect(mask[50 * 200 + 150]).toBe(0);+ expect(mask[50 * 200 + 50]).toBe(1);+ // Two seeds, both.+ const both = growRegion(image(200, 100, two), [[50, 50], [150, 50]]);+ expect(both[50 * 200 + 150]).toBe(1);+ });++ it('takes shading across one thing, and a tolerance says how much', () => {+ const shaded: Shade = (x, y) => { const v = 130 + (x - 30) * 1.2; return Math.hypot(x - 80, y - 60) <= 50 ? [v, v * 0.8, v * 0.7] : BG; };+ const wide = growRegion(image(160, 120, shaded), [[80, 60]], { tolerance: 40 });+ const narrow = growRegion(image(160, 120, shaded), [[80, 60]], { tolerance: 5 });+ const count = (m: Uint8Array): number => m.reduce((sum, v) => sum + v, 0);+ expect(count(wide)).toBeGreaterThan(count(narrow) * 1.5);+ });++ it('fills what is inside: a face is one thing with an eye in it', () => {+ const eye: Shade = (x, y) => (Math.hypot(x - 60, y - 60) <= 5 ? [10, 10, 10] : disc(60, 60, 40)(x, y));+ const mask = growRegion(image(120, 120, eye), [[60, 35]]);+ expect(mask[60 * 120 + 60]).toBe(1);+ });++ it('is held back by a place known to be outside, where a colour alone would run on', () => {+ // Two colours that shade into each other, and no edge between: only `not` can part them.+ const ramp: Shade = (x) => [Math.round(100 + x), 100, 100];+ const free = growRegion(image(120, 40, ramp), [[10, 20]], { tolerance: 60 });+ const held = growRegion(image(120, 40, ramp), [[10, 20]], { tolerance: 60, not: [[110, 20]] });+ const right = (m: Uint8Array): number => m[20 * 120 + 100]!;+ expect(right(free)).toBe(1);+ expect(right(held)).toBe(0);+ expect(held[20 * 120 + 10]).toBe(1);+ });++ it('keeps a part the edge of the crop cuts: it is not eaten from there', () => {+ const half: Shade = (x) => (x < 50 ? FG : BG);+ const mask = growRegion(image(100, 60, half), [[20, 30]]);+ expect(mask[30 * 100]).toBe(1);+ expect(mask[0]).toBe(1);+ });++ it('survives a speck of noise inside and a thin spur outside', () => {+ const rough: Shade = (x, y) => {+ if (x === 60 && y === 60) return BG;+ if (y === 60 && x > 80 && x < 110) return FG;+ return disc(60, 60, 25)(x, y);+ };+ const mask = growRegion(image(120, 120, rough), [[50, 50]]);+ expect(mask[60 * 120 + 60]).toBe(1);+ expect(mask[60 * 120 + 100]).toBe(0);+ });++ it('says what is wrong', () => {+ const pic = image(20, 20, () => BG);+ expect(() => growRegion(pic, [])).toThrow(/at least one/);+ expect(() => growRegion(pic, [[30, 5]])).toThrow(/off the picture/);+ expect(() => growRegion(pic, [[5, 5]], { tolerance: 0 })).toThrow(/positive/);+ });++ it('walks a mask of one piece closed, and takes the largest of several', () => {+ const mask = new Uint8Array(100);+ for (const i of [11, 12, 13, 21, 22, 23, 31, 32, 33, 77]) mask[i] = 1;+ const line = outlineOfMask(mask, 10, 10);+ expect(new Set(line.map((p) => p.join())).size).toBe(8);+ expect(line.some(([x, y]) => x === 7 && y === 7)).toBe(false);+ const [a, b] = [line[0]!, line[line.length - 1]!];+ expect(Math.max(Math.abs(a[0] - b[0]), Math.abs(a[1] - b[1]))).toBe(1);+ expect(outlineOfMask(new Uint8Array(4), 2, 2)).toEqual([]);+ });+});++describe('simplifyLine', () => {+ it('leaves the corners of what was straight', () => {+ const run: PixelPoint[] = [];+ for (let i = 0; i <= 50; i++) run.push([i, 0]);+ for (let i = 1; i <= 50; i++) run.push([50, i]);+ expect(simplifyLine(run, 0.5)).toEqual([[0, 0], [50, 0], [50, 50]]);+ });++ it('never strays from the line by more than asked', () => {+ const curve: PixelPoint[] = Array.from({ length: 200 }, (_, i) => [i, 40 * Math.sin(i / 30)]);+ const kept = simplifyLine(curve, 1);+ expect(kept.length).toBeLessThan(40);+ for (const p of curve) {+ let best = Infinity;+ for (let i = 1; i < kept.length; i++) {+ const [a, b] = [kept[i - 1]!, kept[i]!];+ const [ex, ey] = [b[0] - a[0], b[1] - a[1]];+ const t = Math.max(0, Math.min(1, ((p[0] - a[0]) * ex + (p[1] - a[1]) * ey) / (ex * ex + ey * ey)));+ best = Math.min(best, Math.hypot(p[0] - a[0] - ex * t, p[1] - a[1] - ey * t));+ }+ expect(best).toBeLessThanOrEqual(1.001);+ }+ });++ it('closes a closed line with no point given twice', () => {+ const ring: PixelPoint[] = Array.from({ length: 100 }, (_, i) => [50 + 40 * Math.cos((i / 100) * Math.PI * 2), 50 + 40 * Math.sin((i / 100) * Math.PI * 2)]);+ const kept = simplifyLine(ring, 1, true);+ expect(kept.length).toBeLessThan(30);+ expect(kept.length).toBeGreaterThan(6);+ expect(new Set(kept.map((p) => p.join())).size).toBe(kept.length);+ expect(measure(kept).area).toBeGreaterThan(Math.PI * 40 * 40 * 0.92);+ });+});++describe('traceEdgeLines', () => {+ it('finds a long edge as one line, where it is', () => {+ const step: Shade = (x, y) => (y < 0.4 * x + 30 ? [40, 60, 110] : [190, 190, 200]);+ const lines = traceEdgeLines(image(200, 160, step), { minLength: 20 });+ expect(lines.length).toBeGreaterThan(0);+ const longest = lines[0]!;+ const span = Math.hypot(longest[longest.length - 1]![0] - longest[0]![0], longest[longest.length - 1]![1] - longest[0]![1]);+ expect(span).toBeGreaterThan(150);+ for (const [x, y] of longest) expect(Math.abs(y - (0.4 * x + 30))).toBeLessThan(3);+ // A straight edge simplified is a few points and not every pixel.+ expect(longest.length).toBeLessThan(12);+ });++ it('finds nothing in a flat picture, and leaves out what is short', () => {+ expect(traceEdgeLines(image(60, 60, () => BG))).toEqual([]);+ expect(traceEdgeLines(image(100, 100, disc(50, 50, 4)), { minLength: 200 })).toEqual([]);+ });++ it('longest first, and no more than asked', () => {+ const two: Shade = (x, y) => ((y > 30 && y < 40 && x > 10 && x < 190) || (y > 100 && y < 110 && x > 60 && x < 130) ? [220, 220, 220] : [20, 20, 20]);+ const lines = traceEdgeLines(image(200, 140, two), { minLength: 10 });+ const reach = (l: readonly PixelPoint[]): number => Math.max(...l.map((p) => p[0])) - Math.min(...l.map((p) => p[0]));+ expect(reach(lines[0]!)).toBeGreaterThan(reach(lines[lines.length - 1]!));+ expect(traceEdgeLines(image(200, 140, two), { minLength: 10, most: 1 })).toHaveLength(1);+ });++ it('does the same every time', () => {+ const pic = image(120, 120, disc(60, 60, 40));+ expect(traceEdgeLines(pic)).toEqual(traceEdgeLines(pic));+ });+});++describe('colorOfPatch', () => {+ it('reads the colour there, in hex and in linear', () => {+ const pic = image(40, 40, (x) => (x < 20 ? [255, 0, 0] : [0, 0, 255]));+ expect(colorOfPatch(pic, 5, 20, 2)).toEqual({ hex: '#ff0000', linear: [1, 0, 0] });+ expect(colorOfPatch(pic, 35, 20, 2).hex).toBe('#0000ff');+ });++ it('averages in light: half black and half white is not 128', () => {+ const pic = image(40, 40, (x) => (x < 20 ? [0, 0, 0] : [255, 255, 255]));+ const c = colorOfPatch(pic, 20, 20, 6);+ expect(c.linear[0]).toBeGreaterThan(0.4);+ expect(c.linear[0]).toBeLessThan(0.65);+ expect(parseInt(c.hex.slice(1, 3), 16)).toBeGreaterThan(170);+ });++ it('says when a place is off the picture', () => {+ expect(() => colorOfPatch(image(10, 10, () => BG), 50, 5, 1)).toThrow(/off the picture/);+ });+});++describe('drawing on a picture', () => {+ it('draws a line in place, leaving the rest', () => {+ const pic = image(20, 20, () => BG);+ drawLine(pic, [[2, 10], [17, 10]], [255, 0, 255]);+ expect(Array.from(pic.rgba.subarray((10 * 20 + 10) * 4, (10 * 20 + 10) * 4 + 3))).toEqual([255, 0, 255]);+ expect(Array.from(pic.rgba.subarray((2 * 20 + 2) * 4, (2 * 20 + 2) * 4 + 3))).toEqual([...BG]);+ });++ it('a closed line returns to its start', () => {+ const pic = image(20, 20, () => BG);+ drawLine(pic, [[2, 2], [15, 2], [15, 15]], [255, 0, 255], true);+ expect(pic.rgba[(8 * 20 + 8) * 4]).toBe(255);+ });++ it('finds what two pictures differ by, ignoring what is barely different', () => {+ const a = image(10, 10, () => [100, 100, 100]);+ const b = image(10, 10, (x) => (x < 5 ? [100, 100, 100] : x < 8 ? [101, 101, 101] : [200, 100, 100]));+ const mask = differenceMask(a, b);+ expect(mask[0]).toBe(0);+ expect(mask[6]).toBe(0);+ expect(mask[9]).toBe(1);+ });++ it('draws the border of a mask at another size, and only the border', () => {+ const mask = new Uint8Array(100);+ for (let y = 3; y < 7; y++) for (let x = 3; x < 7; x++) mask[y * 10 + x] = 1;+ const over = maskEdgeOver(image(100, 100, () => BG), mask, 10, 10, [0, 255, 255]);+ const px = (x: number, y: number): number[] => Array.from(over.rgba.subarray((y * 100 + x) * 4, (y * 100 + x) * 4 + 3));+ expect(px(50, 30)).toEqual([0, 255, 255]);+ expect(px(50, 50)).toEqual([...BG]);+ expect(px(10, 10)).toEqual([...BG]);+ });+});diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex d862e30d5..05178086d 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -3884,6 +3884,10 @@ export const NOTED = {{"note":"A material's normal and metal-rough maps are read as data in the editor's Play."},"fix(minigame): no mini-game module needs BigInt64Array, so WeChat DevTools boots the game":{"note":"A WeChat mini-game opens in WeChat DevTools."},+ "feat(look): a part of a picture traced as points, its edges as lines, and the colour of a patch":+ {"note":"An AI can read shapes and colours off a picture as numbers."},+ "feat(mcp): reference_trace, reference_probe, and the outline of one part on the reference":+ {"note":"An AI can read shapes and colours off a picture as numbers."},"feat(mcp): a region is drawn close through the camera, camera_layers takes a frame apart, and light_sheet shows each light alone":{"note":"An AI can take a frame apart to see why it looks wrong."},"fix(modeling): an outline is a dome with no flat on it, a stroke fades behind what is in front, and a line can bend":