A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(look): repaint the picture so far in strokes, and an edit draws only what changed
yhh0264da8af836
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 95cb01dfd..424fd299a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -22,7 +22,7 @@ published separately; it ships inside the editor.- **Glyphs can be put on the atlas before they are shown.** The first page of a novel is hundreds of new characters, all landing in one frame. `preloadGlyphs(app, label, text, { budgetMs })` makes them ahead of time, a slice a frame, drawn as that `Text` will draw them, and returns `progress`, `done` and `release()`. On a Snapdragon 8 Elite a book's 2992 characters at 28 px took 145 ms over 25 frames with the worst frame at 9.5 ms, after which the first page's frame went from 49 to 11 ms and no page made a new glyph. Preloaded glyphs hold their atlas pages (4 MB each, a few hundred characters at that size) until `release()`.- **Voxel GI lights what the camera does not see, and bounces light more than once.** The cache now reads the sun's shadow map back into the world, so a surface the sun lights behind or beside the camera still sends light into the view. Each surface also gives off again the bounce light it received, so a corner or a floor in shade gets light that took two or more bounces to reach it. A ray now brings what it meets at full strength however far it went (it used to fade with distance, 40% at 4 m with 12 cm voxels). Each voxel is weighed by how much of it a surface covers, so a bright patch no longer lights like the whole voxel around it. Together these keep the result close to a path tracer's whatever the voxel size. In a sunlit test room the whole frame carries 0.95 of Cycles' light at the default size and 0.74 at 12 cm voxels; before, it was 0.54 at the default and darker the finer the grid. With the lit patch behind the camera, the walls went from no bounce light to 25–57%. A closed room the sun only lights from outside stays dark. Two inspector controls tune it: **Off-screen Sun Albedo** (the colour the shadow map cannot give, assumed) and **Multi-bounce**. A fine grid no longer drops every voxel: a count truncated by the cache's ageing had kept them below the two samples a ray asks for. The WebGL2 fallback does the same in its own passes: each voxel asks the sun's map whether a sunlit surface stands in it, and traces two rays through last frame's grid for the light it is given. In the test room it carries 1.00 of Cycles' light at the default size (it was 0.56) and 0.80 at 12 cm; with the lit patch behind the camera 0.20 (it was 0). Its voxel pass costs about 1.3 ms more at 720p on a desktop.-- **An AI can draw 2D pictures.** `image_write` turns layers into a `.png` beside an `.esimage`: shapes and SVG paths, gradients, strokes, shadows and glows, clipping and cutting, or pixel art as rows of characters. It answers with the picture on a mid ground and a light check and at 96, 64, 48 and 32 pixels, to look at and correct; `image_edit` changes one layer, a row or a block of pixels without restating the rest, and moves layers or wraps them in a group. Strokes can keep sharp corners, and the reply measures where each layer ended up. A layer or group can be blurred, for depth of field, and a fill can be noise for cloud, texture and film grain, the same picture from the same seed. A `paint` layer is a canvas of brush strokes with pressure, softness, tapering, bristles and smudging, added to round by round, for what shapes cannot do; one `sweep` instruction lays a mass of hair gathered into locks, one `field` covers a region in strokes along a flow, with named brushes from ink and pencil to oil and watercolor, and one `along` repeats strokes along a line. A layer can work across everything under it, to fuse separate parts into one painting, and `image_read` shows lightness alone and reads colours. Text in any script, windows in rows on a face, buildings in true perspective, reflections on wet ground and bloom over what is bright are one instruction each, and words and strokes can be laid on a building's face in its own coordinates. A `figure` layer is a posed skeleton in plain blocks from any angle to draw over, and an `image` layer brings in the team's own art: both stay out of the `.png`, and the figure reports where each joint landed and lets broad light follow the form. `image_read` enlarges a part, and `image_set` shows several together with the stroke widths and size of each. Drawn on the CPU, so a source makes the same picture on any machine.+- **An AI can draw 2D pictures.** `image_write` turns layers into a `.png` beside an `.esimage`: shapes and SVG paths, gradients, strokes, shadows and glows, clipping and cutting, or pixel art as rows of characters. It answers with the picture on a mid ground and a light check and at 96, 64, 48 and 32 pixels, to look at and correct; `image_edit` changes one layer, a row or a block of pixels without restating the rest, and moves layers or wraps them in a group. Strokes can keep sharp corners, and the reply measures where each layer ended up. A layer or group can be blurred, for depth of field, and a fill can be noise for cloud, texture and film grain, the same picture from the same seed. A `paint` layer is a canvas of brush strokes with pressure, softness, tapering, bristles and smudging, added to round by round, for what shapes cannot do; one `sweep` instruction lays a mass of hair gathered into locks, one `field` covers a region in strokes along a flow, with named brushes from ink and pencil to oil and watercolor, and one `along` repeats strokes along a line. A layer can work across everything under it, to fuse separate parts into one painting, and `image_read` shows lightness alone and reads colours. Text in any script, windows in rows on a face, buildings in true perspective, reflections on wet ground and bloom over what is bright are one instruction each, and words and strokes can be laid on a building's face in its own coordinates. `repaint` paints the finished picture again in tens of thousands of strokes, and an edit draws only the layers from the change up. A `figure` layer is a posed skeleton in plain blocks from any angle to draw over, and an `image` layer brings in the team's own art: both stay out of the `.png`, and the figure reports where each joint landed and lets broad light follow the form. `image_read` enlarges a part, and `image_set` shows several together with the stroke widths and size of each. Drawn on the CPU, so a source makes the same picture on any machine.- **An AI can build a person, and not a picture of one.** The `humanoid` step makes a whole body from a few numbers, with fingers, a face and ears, seen right from every side, and `rig: { humanoid: true }` gives it bones that fit. `garment` makes clothes from the body's own surface: loose or tight, a bell sleeve that hangs, a clean hem, a thickness; what a body wears now takes its weights from the skin under it, so an arm stays in its sleeve. `pose` holds a body in a pose by angles, or with `reach` by where a hand should be. `organic` gains `limb`, cross-sections along a path, and `fine`, regions split finer on one skin. `displace` gains `folds`, running the way cloth hangs. `scene_views` no longer takes on a shot's lens blur and grade.diff --git a/desktop b/desktopindex 6d80bbc69..e0c778259 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit 6d80bbc69aadfae39847e3cbdfbc8257c4df7178+Subproject commit e0c778259c5311355c7ed5ef9dc244304ef38bb6diff --git a/pipeline/src/image/imageRaster.ts b/pipeline/src/image/imageRaster.tsindex 0e1777248..232ac9ece 100644--- a/pipeline/src/image/imageRaster.ts+++ b/pipeline/src/image/imageRaster.ts@@ -822,6 +822,11 @@ function drawLayer(c: Canvas, layer: ImageLayer, parent: Matrix, drawn: Drawn):return;}+ if (layer.shape === 'repaint') {+ drawRepaint(c, layer, drawn, opacity, clip);+ return;+ }+if (layer.face && layer.shape === 'rect') {const corners = faceCorners(layer, drawn);if (!corners) return;@@ -1476,6 +1481,100 @@ function textOnQuad(layer: ImageLayer, m: Matrix, w: number, h: number, draw: Imreturn out;}+/** [brush size at 960 pixels across, how wrong a cell must be, 0..255, before it gets a stroke]. */+const REPAINT_LEVELS: readonly (readonly (readonly [number, number])[])[] = [+ [[26, 0], [13, 24], [6, 30]],+ [[22, 0], [11, 20], [6, 22], [3.5, 20]],+ [[20, 0], [10, 18], [5, 18], [3, 16], [2, 14]],+];+const REPAINT_BOOST: readonly (readonly [number, number])[] = [[3, 14], [1.8, 10]];++/**+ * The picture so far painted again, coarse to fine, each stroke taking its colour from under where it starts and running along the edges there.+ * Shapes become brushwork without a stroke being written: what makes a built picture read as a painted one. It costs seconds, so it comes last.+ */+function drawRepaint(c: Canvas, layer: ImageLayer, drawn: Drawn, opacity: number, clip: Float32Array | undefined): void {+ const { w, h } = c;+ const k = Math.max(w, h) / 960;+ const brush = layer.brush ?? 'oil';+ const seed = layer.seed ?? 1;+ const inner = new Canvas(w, h);+ const buffer: StrokeBuffer = { mask: new Float32Array(w * h), rgb: new Float32Array(w * h * 3), touched: [] };+ const boosted = (layer.boost ?? []).map((id) => {+ const mask = drawn.masks.get(id.replace(/^@/, ''));+ if (!mask) drawn.notes.push(`layer ${layer.id ?? 'repaint'}: boost "${id}" names no layer drawn before it`);+ return mask;+ }).filter((m): m is Float32Array => !!m);+ const passes = [+ ...REPAINT_LEVELS[layer.detail ?? 2]!.map(([size, tolerance]) => ({ size, tolerance, only: null as Float32Array[] | null })),+ ...(boosted.length ? REPAINT_BOOST.map(([size, tolerance]) => ({ size, tolerance, only: boosted })) : []),+ ];+ let count = 0;+ for (const pass of passes) {+ const size = pass.size * k;+ const sigma = Math.max(0.6, size / 3.2);+ const target = [0, 1, 2, 3].map((n) => {+ const channel = new Float32Array(w * h);+ for (let i = 0; i < channel.length; i++) channel[i] = c.px[i * 4 + n]!;+ return blurMask(channel, w, h, sigma);+ }) as [Float32Array, Float32Array, Float32Array, Float32Array];+ const light = new Float32Array(w * h);+ for (let i = 0; i < light.length; i++) light[i] = target[0][i]! * 0.3 + target[1][i]! * 0.59 + target[2][i]! * 0.11;+ const cell = Math.max(2, Math.round(size * 0.8));+ const [columns, rows] = [Math.ceil(w / cell), Math.ceil(h / cell)];+ // In no order: swept in rows, each stroke would lie the same way over the last.+ const order = Array.from({ length: columns * rows }, (_, i) => i).sort((a, b) => hash01(seed * 61 + a, count & 255) - hash01(seed * 61 + b, count & 255));+ const starts: { x: number; y: number }[] = [];+ for (const index of order) {+ const [gx, gy] = [(index % columns) * cell, Math.floor(index / columns) * cell];+ let [worst, bx, by, total, n] = [-1, gx, gy, 0, 0];+ for (let y = gy; y < Math.min(h, gy + cell); y++) {+ for (let x = gx; x < Math.min(w, gx + cell); x++) {+ const i = y * w + x;+ const e = Math.hypot(inner.px[i * 4]! - target[0][i]!, inner.px[i * 4 + 1]! - target[1][i]!, inner.px[i * 4 + 2]! - target[2][i]!) * 255;+ total += e; n++;+ if (e > worst) { worst = e; bx = x; by = y; }+ }+ }+ if (n === 0 || total / n < pass.tolerance) continue;+ if (pass.only && !pass.only.some((m) => m[by * w + bx]! >= 0.5)) continue;+ starts.push({ x: bx, y: by });+ }+ for (const { x: bx, y: by } of starts) {+ const at = by * w + bx;+ const alpha = target[3][at]!;+ if (alpha < 0.02) continue;+ const color: Rgba = [target[0][at]! / alpha, target[1][at]! / alpha, target[2][at]! / alpha, alpha];+ const points: number[][] = [[bx, by]];+ let [x, y, dx, dy] = [bx, by, 0, 0];+ for (let step = 0; step < 5; step++) {+ const [ix, iy] = [Math.max(1, Math.min(w - 2, Math.round(x))), Math.max(1, Math.min(h - 2, Math.round(y)))];+ const [gx, gy] = [light[iy * w + ix + 1]! - light[iy * w + ix - 1]!, light[(iy + 1) * w + ix]! - light[(iy - 1) * w + ix]!];+ const g = Math.hypot(gx, gy);+ // Along an edge where there is one, and a way of its own where there is not.+ const free = hash01(seed * 67 + bx, by) * Math.PI * 2;+ let [nx, ny] = g < 0.006 ? [Math.cos(free), Math.sin(free)] : [-gy / g, gx / g];+ if (nx * dx + ny * dy < 0) [nx, ny] = [-nx, -ny];+ [dx, dy] = dx === 0 && dy === 0 ? [nx, ny] : [dx * 0.5 + nx * 0.5, dy * 0.5 + ny * 0.5];+ const len = Math.hypot(dx, dy) || 1;+ x += (dx / len) * size * 0.9; y += (dy / len) * size * 0.9;+ if (x < 0 || y < 0 || x >= w || y >= h) break;+ const here = Math.round(y) * w + Math.round(x);+ if (Math.hypot(target[0][here]! - target[0][at]!, target[1][here]! - target[1][at]!, target[2][here]! - target[2][at]!) * 255 > 45) break;+ points.push([x, y]);+ }+ paintStroke(inner, { brush, points, color: toHex(color), size, seed: seed + count }, IDENTITY, 1, count, buffer);+ count++;+ }+ }+ if (layer.id) drawn.painted.set(layer.id, { entries: passes.length, strokes: count });+ for (let i = 0; i < w * h; i++) {+ const a = inner.px[i * 4 + 3]!;+ if (a <= 0) continue;+ put(c, i, inner.px[i * 4]! / a, inner.px[i * 4 + 1]! / a, inner.px[i * 4 + 2]! / a, a * opacity * (clip ? clip[i]! : 1), layer.blend ?? 'normal');+ }+}+/** What is bright in the picture so far, spilling over what is round it: a tight glow and a wide one, as a lens and the air both give. */function drawBloom(c: Canvas, layer: ImageLayer, parent: Matrix, opacity: number, clip: Float32Array | undefined): void {const { w, h } = c;@@ -1766,6 +1865,64 @@ function drawPixels(c: Canvas, layer: ImageLayer, parent: Matrix, drawn: Drawn,effectLayers(layer.effects, false, c, whole);}+/** A picture part-way through: what it costs to draw is what a small edit should not pay again. */+interface Snapshot { px: Float32Array; drawn: Omit<Drawn, 'showGuides' | 'pictures' | 'text'> }+const snapshots = new Map<string, Snapshot>();+const SNAPSHOT_AFTER_MS = 150;+const SNAPSHOT_BYTES = 512 * 1024 * 1024;+const cacheCounters = { drawn: 0, reused: 0 };++/** How many layers were drawn and how many were picked up already drawn; `clear` forgets what is kept. */+export function imageCacheStats(clear = false): { drawn: number; reused: number; kept: number } {+ const out = { ...cacheCounters, kept: snapshots.size };+ if (clear) { snapshots.clear(); cacheCounters.drawn = 0; cacheCounters.reused = 0; }+ return out;+}++function textHash(text: string): string {+ let [a, b] = [0xdeadbeef, 0x41c6ce57];+ for (let i = 0; i < text.length; i++) {+ const ch = text.charCodeAt(i);+ a = Math.imul(a ^ ch, 2654435761);+ b = Math.imul(b ^ ch, 1597334677);+ }+ return (((a ^ (a >>> 16)) >>> 0).toString(36)) + ((b ^ (b >>> 13)) >>> 0).toString(36) + text.length.toString(36);+}++/** Pictures come from files that change under the same name, so what is in them is part of the key. */+function picturesDigest(pictures: Readonly<Record<string, RgbaBitmap>> | undefined): string {+ if (!pictures) return '';+ return Object.entries(pictures).map(([name, p]) => {+ let sum = 0;+ for (let i = 0; i < p.rgba.length; i += 97) sum = (Math.imul(sum, 31) + p.rgba[i]!) | 0;+ return `${name}:${p.width}x${p.height}:${sum}`;+ }).join('|');+}++/** Masks and the rest are never written once set, so a snapshot can hold them as they are. */+function keepSnapshot(key: string, canvas: Canvas, drawn: Drawn): void {+ snapshots.set(key, {+ px: canvas.px.slice(),+ drawn: {+ masks: new Map(drawn.masks), notes: [...drawn.notes], bounds: new Map(drawn.bounds), painted: new Map(drawn.painted), figures: new Map(drawn.figures),+ figureJoints: new Map(drawn.figureJoints), images: new Map(drawn.images), blocks: new Map(drawn.blocks),+ },+ });+ let bytes = 0;+ for (const s of snapshots.values()) bytes += s.px.byteLength;+ for (const [k, s] of snapshots) {+ if (bytes <= SNAPSHOT_BYTES) break;+ snapshots.delete(k);+ bytes -= s.px.byteLength;+ }+}++function restoreSnapshot(kept: Snapshot, canvas: Canvas, drawn: Drawn): void {+ canvas.px.set(kept.px);+ drawn.masks = new Map(kept.drawn.masks); drawn.notes = [...kept.drawn.notes]; drawn.bounds = new Map(kept.drawn.bounds); drawn.painted = new Map(kept.drawn.painted);+ drawn.figures = new Map(kept.drawn.figures); drawn.figureJoints = new Map(kept.drawn.figureJoints); drawn.images = new Map(kept.drawn.images); drawn.blocks = new Map(kept.drawn.blocks);+}+export interface RenderedImage extends RgbaBitmap {notes: string[];/** Where the picture is solid, null if nowhere, and how much of its area that is, 0..1: how large it looks beside others. */@@ -1785,12 +1942,35 @@ export interface RenderedImage extends RgbaBitmap {export function renderImage(source: ImageSource, options: { guides?: boolean; pictures?: Readonly<Record<string, RgbaBitmap>>; text?: ImageTextDrawer } = {}): RenderedImage {const [w, h] = typeof source.size === 'number' ? [source.size, source.size] : source.size;const canvas = new Canvas(w, h);- if (source.background) {+ const drawn: Drawn = { masks: new Map(), notes: [], bounds: new Map(), painted: new Map(), figures: new Map(), figureJoints: new Map(), showGuides: options.guides === true, pictures: options.pictures ?? {}, images: new Map(), text: options.text, blocks: new Map() };+ // What every layer so far amounts to, chained: a layer changed leaves the keys before it as they were.+ let key = textHash(JSON.stringify([w, h, source.background ?? null, options.guides === true, picturesDigest(options.pictures)]));+ const keys = source.layers.map((layer) => (key = textHash(key + JSON.stringify(layer))));+ let from = 0;+ for (let i = keys.length - 1; i >= 0; i--) {+ const kept = snapshots.get(keys[i]!);+ if (!kept) continue;+ restoreSnapshot(kept, canvas, drawn);+ // Used last, kept longest.+ snapshots.delete(keys[i]!);+ snapshots.set(keys[i]!, kept);+ from = i + 1;+ cacheCounters.reused += from;+ break;+ }+ if (from === 0 && source.background) {const [r, g, b, a] = parseColor(source.background);for (let i = 0; i < w * h; i++) put(canvas, i, r, g, b, a, 'normal');}- const drawn: Drawn = { masks: new Map(), notes: [], bounds: new Map(), painted: new Map(), figures: new Map(), figureJoints: new Map(), showGuides: options.guides === true, pictures: options.pictures ?? {}, images: new Map(), text: options.text, blocks: new Map() };- for (const layer of source.layers) drawLayer(canvas, layer, IDENTITY, drawn);+ let since = Date.now();+ for (let i = from; i < source.layers.length; i++) {+ drawLayer(canvas, source.layers[i]!, IDENTITY, drawn);+ cacheCounters.drawn++;+ if (i === source.layers.length - 1 || Date.now() - since >= SNAPSHOT_AFTER_MS) {+ keepSnapshot(keys[i]!, canvas, drawn);+ since = Date.now();+ }+ }const rgba = new Uint8Array(w * h * 4);for (let i = 0; i < w * h; i++) {const a = canvas.px[i * 4 + 3]!;diff --git a/pipeline/src/image/imageSource.ts b/pipeline/src/image/imageSource.tsindex 9ea21c395..61eae0eaf 100644--- a/pipeline/src/image/imageSource.ts+++ b/pipeline/src/image/imageSource.ts@@ -55,7 +55,7 @@ export interface ImageTransform {origin?: readonly [number, number];}-export type ImageShape = 'rect' | 'circle' | 'ellipse' | 'ring' | 'polygon' | 'line' | 'star' | 'gear' | 'path' | 'pixels' | 'paint' | 'figure' | 'image' | 'text' | 'block' | 'perspective' | 'reflect' | 'bloom' | 'group';+export type ImageShape = 'rect' | 'circle' | 'ellipse' | 'ring' | 'polygon' | 'line' | 'star' | 'gear' | 'path' | 'pixels' | 'paint' | 'figure' | 'image' | 'text' | 'block' | 'perspective' | 'reflect' | 'bloom' | 'repaint' | 'group';/** How a brush lays colour. Any of these on a stroke, a sweep or a field, over what its `brush` starts from. */export interface ImageBrushSettings {@@ -264,6 +264,12 @@ export interface ImageLayer {within?: readonly number[];/** Text: four corners in the picture it is laid on, in order round from the top left, so it lies on a wall in perspective. */quad?: readonly (readonly number[])[];+ /** A repaint: the brush the whole picture so far is painted again with. */+ brush?: string;+ /** A repaint: how far down its strokes go, 0 broad to 2 fine, and what gets finer strokes still. */+ detail?: number;+ boost?: readonly string[];+ seed?: number;/** A bloom: how bright a pixel must be to shine, 0..1, and how much it does. */threshold?: number;strength?: number;@@ -349,11 +355,11 @@ function checkFill(v: unknown): string | null {return null;}-const FIELDS = ['id', 'shape', 'at', 'size', 'radius', 'radii', 'inner', 'count', 'tooth', 'turn', 'points', 'd', 'rows', 'palette', 'scale', 'layers', 'fill', 'stroke', 'opacity', 'blend', 'transform', 'mirror', 'effects', 'clip', 'erase', 'visible', 'blur', 'strokes', 'from', 'guide', 'height', 'yaw', 'pitch', 'build', 'pose', 'light', 'src', 'except', 'text', 'fontSize', 'font', 'weight', 'align', 'vertical', 'tracking', 'vanish', 'depth', 'faces', 'roof', 'y', 'edge', 'reach', 'ripple', 'smear', 'quad', 'threshold', 'strength', 'thin', 'grain', 'grainScale', 'dry', 'face', 'within'];+const FIELDS = ['id', 'shape', 'at', 'size', 'radius', 'radii', 'inner', 'count', 'tooth', 'turn', 'points', 'd', 'rows', 'palette', 'scale', 'layers', 'fill', 'stroke', 'opacity', 'blend', 'transform', 'mirror', 'effects', 'clip', 'erase', 'visible', 'blur', 'strokes', 'from', 'guide', 'height', 'yaw', 'pitch', 'build', 'pose', 'light', 'src', 'except', 'text', 'fontSize', 'font', 'weight', 'align', 'vertical', 'tracking', 'vanish', 'depth', 'faces', 'roof', 'y', 'edge', 'reach', 'ripple', 'smear', 'quad', 'threshold', 'strength', 'thin', 'grain', 'grainScale', 'dry', 'face', 'within', 'brush', 'detail', 'boost', 'seed'];/** Layers that are aids to drawing unless told otherwise: seen when looking, and not in the .png. */export const isGuideLayer = (l: ImageLayer): boolean => l.guide ?? (l.shape === 'figure' || l.shape === 'image' || l.shape === 'perspective');-const SHAPES: readonly ImageShape[] = ['rect', 'circle', 'ellipse', 'ring', 'polygon', 'line', 'star', 'gear', 'path', 'pixels', 'paint', 'figure', 'image', 'text', 'block', 'perspective', 'reflect', 'bloom', 'group'];+const SHAPES: readonly ImageShape[] = ['rect', 'circle', 'ellipse', 'ring', 'polygon', 'line', 'star', 'gear', 'path', 'pixels', 'paint', 'figure', 'image', 'text', 'block', 'perspective', 'reflect', 'bloom', 'repaint', 'group'];const STROKE_FIELDS = ['points', 'color', 'to', 'size', 'hardness', 'opacity', 'taper', 'bristles', 'strand', 'spread', 'vary', 'wobble', 'seed', 'smudge', 'erase', 'blend', ...BRUSH_FIELDS];/** What a brush's own settings must be, on whatever carries them. */@@ -528,6 +534,12 @@ function checkLayer(l: Record<string, unknown>, where: string, ids: Set<string>)if (l.grainScale !== undefined && !(isNum(l.grainScale) && (l.grainScale as number) > 0)) fail('grainScale must be positive');if (l.fill === undefined && l.stroke === undefined) fail('it needs a fill or a stroke, or it draws nothing');break;+ case 'repaint':+ if (l.brush !== undefined && !(typeof l.brush === 'string' && Object.hasOwn(IMAGE_BRUSHES, l.brush))) fail(`brush ${JSON.stringify(l.brush)} must be one of ${Object.keys(IMAGE_BRUSHES).join(', ')}`);+ if (l.detail !== undefined && ![0, 1, 2].includes(l.detail as number)) fail('detail must be 0, 1 or 2');+ if (l.boost !== undefined && !(Array.isArray(l.boost) && l.boost.every((b) => typeof b === 'string'))) fail('boost must be ids of layers drawn before it');+ if (l.seed !== undefined && !Number.isInteger(l.seed)) fail('seed must be a whole number');+ break;case 'bloom':for (const key of ['threshold', 'strength']) if (l[key] !== undefined && !(isNum(l[key]) && (l[key] as number) >= 0)) fail(`${key} must be 0 or more`);if (l.threshold !== undefined && (l.threshold as number) > 1) fail('threshold must be 0..1');@@ -600,7 +612,7 @@ function checkLayer(l: Record<string, unknown>, where: string, ids: Set<string>)if (s.join !== undefined && !['round', 'miter', 'bevel'].includes(s.join as string)) fail('stroke join must be round, miter or bevel');if (s.cap !== undefined && !['round', 'butt', 'square'].includes(s.cap as string)) fail('stroke cap must be round, butt or square');}- if (!['group', 'pixels', 'paint', 'figure', 'image', 'text', 'block', 'perspective', 'reflect', 'bloom', 'line'].includes(l.shape as string) && l.fill === undefined && l.stroke === undefined && !l.erase) fail('it needs a fill or a stroke, or it draws nothing');+ if (!['group', 'pixels', 'paint', 'figure', 'image', 'text', 'block', 'perspective', 'reflect', 'bloom', 'repaint', 'line'].includes(l.shape as string) && l.fill === undefined && l.stroke === undefined && !l.erase) fail('it needs a fill or a stroke, or it draws nothing');if (l.from !== undefined && l.shape !== 'paint') fail('from is for a paint layer');if (l.guide !== undefined && typeof l.guide !== 'boolean') fail('guide must be true or false');if (l.blur !== undefined && !(isNum(l.blur) && (l.blur as number) >= 0)) fail('blur must be 0 or more, in pixels');diff --git a/pipeline/tests/image-raster.test.ts b/pipeline/tests/image-raster.test.tsindex 4a6a579aa..ec8e81aae 100644--- a/pipeline/tests/image-raster.test.ts+++ b/pipeline/tests/image-raster.test.ts@@ -2,7 +2,7 @@// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team/** A picture from layers: each draws what it says and where, edges are antialiased, and what is wrong with a source is said before anything is drawn. */import { describe, expect, it } from 'vitest';-import { imageSheet, parseColor, parsePath, pickColors, renderImage, setSheet, valuesImage, zoomImage } from '../src/image/imageRaster';+import { imageCacheStats, imageSheet, parseColor, parsePath, pickColors, renderImage, setSheet, valuesImage, zoomImage } from '../src/image/imageRaster';import { applyImageEdits, IMAGE_BRUSHES, strokeWidths, validateImageSource, type ImageLayer } from '../src/image/imageSource';const draw = (layers: unknown[], size: number | [number, number] = 64, background?: string): ReturnType<typeof renderImage> => renderImage(validateImageSource({ size, layers, ...(background ? { background } : {}) }));@@ -1205,6 +1205,123 @@ describe('words on a wall, and light spilling', () => {});});+describe('drawing only what changed', () => {+ const layers: ImageLayer[] = [+ { id: 'a', shape: 'rect', at: [0, 0], size: [64, 64], fill: { type: 'noise', scale: 8, stops: [[0, '#000000'], [1, '#ffffff']] } },+ { id: 'b', shape: 'paint', clip: 'a', strokes: [{ type: 'field', tone: RED, size: 6, length: 20, angle: 20, seed: 2 }] } as never,+ { id: 'c', shape: 'circle', at: [32, 32], radius: 12, fill: BLUE },+ ];+ const source = (top: ImageLayer[]): ReturnType<typeof validateImageSource> => validateImageSource({ size: 64, layers: [...layers, ...top] });+ const fresh = (top: ImageLayer[]): ReturnType<typeof renderImage> => { imageCacheStats(true); return renderImage(source(top)); };+ const dot: ImageLayer = { id: 'd', shape: 'circle', at: [10, 10], radius: 4, fill: '#00ff00' };++ it('draws the same picture, measures and all, as drawing it all again', () => {+ imageCacheStats(true);+ renderImage(source([]));+ const continued = renderImage(source([dot]));+ const whole = fresh([dot]);+ expect(continued.rgba).toEqual(whole.rgba);+ expect(continued.layerBounds).toEqual(whole.layerBounds);+ expect(continued.painted).toEqual(whole.painted);+ expect(continued.notes).toEqual(whole.notes);+ });++ it('picks up where nothing has changed, and draws nothing for the same source', () => {+ imageCacheStats(true);+ renderImage(source([]));+ const before = imageCacheStats();+ renderImage(source([]));+ expect(imageCacheStats().drawn).toBe(before.drawn);+ renderImage(source([dot]));+ expect(imageCacheStats().drawn).toBe(before.drawn + 1);+ });++ it('is not fooled by a change lower down, a seed, a canvas size or showing guides', () => {+ imageCacheStats(true);+ const base = renderImage(source([]));+ const changed = validateImageSource({ size: 64, layers: [{ ...layers[0]!, fill: RED }, layers[1]!, layers[2]!] });+ expect(renderImage(changed).rgba).not.toEqual(base.rgba);+ const reseeded = validateImageSource({ size: 64, layers: [layers[0]!, { ...layers[1]!, strokes: [{ type: 'field', tone: RED, size: 6, length: 20, angle: 20, seed: 3 }] } as never, layers[2]!] });+ expect(renderImage(reseeded).rgba).not.toEqual(base.rgba);+ const guide = validateImageSource({ size: 64, layers: [{ id: 'g', shape: 'rect', at: [0, 0], size: [8, 8], fill: RED, guide: true }] });+ expect(renderImage(guide).rgba[3]).toBe(0);+ expect(renderImage(guide, { guides: true }).rgba[3]).toBe(255);+ });++ it('sees a picture changed under its own name', () => {+ imageCacheStats(true);+ const img = validateImageSource({ size: 8, layers: [{ shape: 'image', src: 'a.png', guide: false, size: [8, 8] }] });+ const red = { width: 1, height: 1, rgba: new Uint8Array([255, 0, 0, 255]) };+ const blue = { width: 1, height: 1, rgba: new Uint8Array([0, 0, 255, 255]) };+ expect(renderImage(img, { pictures: { 'a.png': red } }).rgba[0]).toBe(255);+ expect(renderImage(img, { pictures: { 'a.png': blue } }).rgba[0]).toBe(0);+ });++ it('does not let what comes after change what was kept', () => {+ imageCacheStats(true);+ const once = renderImage(source([]));+ renderImage(source([{ shape: 'rect', at: [0, 0], size: [64, 64], fill: '#ffffff' }]));+ expect(renderImage(source([])).rgba).toEqual(once.rgba);+ });+});++describe('repainting the picture so far', () => {+ const halves: ImageLayer[] = [+ { id: 'l', shape: 'rect', at: [0, 0], size: [64, 128], fill: RED },+ { id: 'r', shape: 'rect', at: [64, 0], size: [64, 128], fill: BLUE },+ { id: 'mark', shape: 'rect', at: [20, 20], size: [24, 24], fill: '#ffffff' },+ ];+ const paintOver = (fields: object = {}): ReturnType<typeof draw> => draw([...halves, { id: 'over', shape: 'repaint', ...fields } as never], [128, 128]);++ it('keeps the picture, its colours where they were, and is no longer the same pixels', () => {+ const before = draw(halves, [128, 128]);+ const after = paintOver();+ expect(px(after, 20, 90)[0]).toBeGreaterThan(200);+ expect(px(after, 110, 90)[2]).toBeGreaterThan(200);+ expect(px(after, 32, 32)[1]).toBeGreaterThan(160);+ expect(after.rgba).not.toEqual(before.rgba);+ expect(after.painted.over!.strokes).toBeGreaterThan(40);+ });++ it('breaks a hard edge into strokes', () => {+ const edgeOf = (img: ReturnType<typeof draw>): Set<number> => { const s = new Set<number>(); for (let y = 8; y < 120; y += 2) { let x = 0; while (x < 127 && px(img, x, y)[0]! > 128) x++; s.add(x); } return s; };+ expect(edgeOf(draw(halves.slice(0, 2), [128, 128])).size).toBe(1);+ expect(edgeOf(draw([...halves.slice(0, 2), { shape: 'repaint' } as never], [128, 128])).size).toBeGreaterThan(1);+ });++ it('takes more strokes as it goes finer, and more again where it is boosted', () => {+ const strokes = (fields: object): number => paintOver(fields).painted.over!.strokes;+ const scene: ImageLayer[] = [...halves, { id: 'noisy', shape: 'circle', at: [90, 90], radius: 24, fill: { type: 'noise', scale: 4, stops: [[0, '#000000'], [1, '#ffffff']] } }];+ const at = (fields: object): number => draw([...scene, { id: 'over', shape: 'repaint', ...fields } as never], [128, 128]).painted.over!.strokes;+ expect(at({ detail: 2 })).toBeGreaterThan(at({ detail: 0 }));+ expect(at({ detail: 0, boost: ['noisy'] })).toBeGreaterThan(at({ detail: 0 }));+ expect(strokes({})).toBeGreaterThan(0);+ });++ it('leaves what it is told to, exactly', () => {+ const kept = paintOver({ except: 'mark' });+ expect(px(kept, 22, 22)).toEqual([255, 255, 255, 255]);+ expect(px(kept, 42, 42)).toEqual([255, 255, 255, 255]);+ });++ it('takes the brush it is told, and is the same every time', () => {+ expect(paintOver({ brush: 'charcoal' }).rgba).not.toEqual(paintOver({ brush: 'oil' }).rgba);+ imageCacheStats(true);+ const a = paintOver({ seed: 4 }).rgba;+ imageCacheStats(true);+ expect(paintOver({ seed: 4 }).rgba).toEqual(a);+ expect(paintOver({ seed: 5 }).rgba).not.toEqual(a);+ });++ it('says what is wrong with one', () => {+ const bad = (l: object): unknown => validateImageSource({ size: 8, layers: [{ shape: 'repaint', ...l }] });+ expect(() => bad({ brush: 'crayon' })).toThrow(/brush "crayon" must be one of/);+ expect(() => bad({ detail: 3 })).toThrow(/detail must be 0, 1 or 2/);+ expect(() => bad({ boost: 'x' })).toThrow(/boost must be/);+ expect(paintOver({ boost: ['none'] }).notes.join()).toMatch(/boost "none" names no layer/);+ });+});+describe('working on a face', () => {const bar = (layer: ImageLayer, m: readonly number[], w: number, h: number): { fill: Float32Array; line: Float32Array | null } => {const fill = new Float32Array(w * h);diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 8edf2c3e3..850386d21 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -3892,6 +3892,10 @@ export const NOTED = {{"note":"An AI can draw 2D pictures."},"feat(mcp): image_write draws text in the editor's fonts, and documents the built world":{"note":"An AI can draw 2D pictures."},+ "feat(look): repaint the picture so far in strokes, and an edit draws only what changed":+ {"note":"An AI can draw 2D pictures."},+ "feat(mcp): image_write teaches repaint":+ {"note":"An AI can draw 2D pictures."},"feat(look): text and strokes on a face, a text's core and wear, bloom":{"note":"An AI can draw 2D pictures."},"feat(mcp): image_write teaches the order of a picture, neon and paint on a wall":