Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(look): brushes: flat tips, paper grain, dry streaks, wet mixing, and named media

yhh313c09897719
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d3da58580..27431576b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,7 +22,7 @@ published separately; it ships inside the editor.
- **Glyphs can be put on the atlas before they are shown.** The first page of a novel is hundreds of new characters, all landing in one frame. `preloadGlyphs(app, label, text, { budgetMs })` makes them ahead of time, a slice a frame, drawn as that `Text` will draw them, and returns `progress`, `done` and `release()`. On a Snapdragon 8 Elite a book's 2992 characters at 28 px took 145 ms over 25 frames with the worst frame at 9.5 ms, after which the first page's frame went from 49 to 11 ms and no page made a new glyph. Preloaded glyphs hold their atlas pages (4 MB each, a few hundred characters at that size) until `release()`.
- **Voxel GI lights what the camera does not see, and bounces light more than once.** The cache now reads the sun's shadow map back into the world, so a surface the sun lights behind or beside the camera still sends light into the view. Each surface also gives off again the bounce light it received, so a corner or a floor in shade gets light that took two or more bounces to reach it. A ray now brings what it meets at full strength however far it went (it used to fade with distance, 40% at 4 m with 12 cm voxels). Each voxel is weighed by how much of it a surface covers, so a bright patch no longer lights like the whole voxel around it. Together these keep the result close to a path tracer's whatever the voxel size. In a sunlit test room the whole frame carries 0.95 of Cycles' light at the default size and 0.74 at 12 cm voxels; before, it was 0.54 at the default and darker the finer the grid. With the lit patch behind the camera, the walls went from no bounce light to 25–57%. A closed room the sun only lights from outside stays dark. Two inspector controls tune it: **Off-screen Sun Albedo** (the colour the shadow map cannot give, assumed) and **Multi-bounce**. A fine grid no longer drops every voxel: a count truncated by the cache's ageing had kept them below the two samples a ray asks for. The WebGL2 fallback does the same in its own passes: each voxel asks the sun's map whether a sunlit surface stands in it, and traces two rays through last frame's grid for the light it is given. In the test room it carries 1.00 of Cycles' light at the default size (it was 0.56) and 0.80 at 12 cm; with the lit patch behind the camera 0.20 (it was 0). Its voxel pass costs about 1.3 ms more at 720p on a desktop.
-- **An AI can draw 2D pictures.** `image_write` turns layers into a `.png` beside an `.esimage`: shapes and SVG paths, gradients, strokes, shadows and glows, clipping and cutting, or pixel art as rows of characters. It answers with the picture on a mid ground and a light check and at 96, 64, 48 and 32 pixels, to look at and correct; `image_edit` changes one layer, a row or a block of pixels without restating the rest, and moves layers or wraps them in a group. Strokes can keep sharp corners, and the reply measures where each layer ended up. A layer or group can be blurred, for depth of field, and a fill can be noise for cloud, texture and film grain, the same picture from the same seed. A `paint` layer is a canvas of brush strokes with pressure, softness, tapering, bristles and smudging, added to round by round, for what shapes cannot do; one `sweep` instruction lays a mass of hair gathered into locks, one `field` covers a region in strokes along a flow, and one `along` repeats strokes along a line. A layer can work across everything under it, to fuse separate parts into one painting, and `image_read` shows lightness alone and reads colours. A `figure` layer is a posed skeleton in plain blocks from any angle to draw over, and an `image` layer brings in the team's own art: both stay out of the `.png`, and the figure reports where each joint landed and lets broad light follow the form. `image_read` enlarges a part, and `image_set` shows several together with the stroke widths and size of each. Drawn on the CPU, so a source makes the same picture on any machine.
+- **An AI can draw 2D pictures.** `image_write` turns layers into a `.png` beside an `.esimage`: shapes and SVG paths, gradients, strokes, shadows and glows, clipping and cutting, or pixel art as rows of characters. It answers with the picture on a mid ground and a light check and at 96, 64, 48 and 32 pixels, to look at and correct; `image_edit` changes one layer, a row or a block of pixels without restating the rest, and moves layers or wraps them in a group. Strokes can keep sharp corners, and the reply measures where each layer ended up. A layer or group can be blurred, for depth of field, and a fill can be noise for cloud, texture and film grain, the same picture from the same seed. A `paint` layer is a canvas of brush strokes with pressure, softness, tapering, bristles and smudging, added to round by round, for what shapes cannot do; one `sweep` instruction lays a mass of hair gathered into locks, one `field` covers a region in strokes along a flow, with named brushes from ink and pencil to oil and watercolor, and one `along` repeats strokes along a line. A layer can work across everything under it, to fuse separate parts into one painting, and `image_read` shows lightness alone and reads colours. A `figure` layer is a posed skeleton in plain blocks from any angle to draw over, and an `image` layer brings in the team's own art: both stay out of the `.png`, and the figure reports where each joint landed and lets broad light follow the form. `image_read` enlarges a part, and `image_set` shows several together with the stroke widths and size of each. Drawn on the CPU, so a source makes the same picture on any machine.
- **An AI can build a person, and not a picture of one.** The `humanoid` step makes a whole body from a few numbers, with fingers, a face and ears, seen right from every side, and `rig: { humanoid: true }` gives it bones that fit. `garment` makes clothes from the body's own surface: loose or tight, a bell sleeve that hangs, a clean hem, a thickness; what a body wears now takes its weights from the skin under it, so an arm stays in its sleeve. `pose` holds a body in a pose by angles, or with `reach` by where a hand should be. `organic` gains `limb`, cross-sections along a path, and `fine`, regions split finer on one skin. `displace` gains `folds`, running the way cloth hangs. `scene_views` no longer takes on a shot's lens blur and grade.
diff --git a/desktop b/desktop
index 0e7c6081f..0468ce72d 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 0e7c6081f88be99a355ca03bb6991e50a628aac7
+Subproject commit 0468ce72dc21b6901344e0ce7da6e6d1d932033f
diff --git a/pipeline/src/image/imageRaster.ts b/pipeline/src/image/imageRaster.ts
index 7bfd8c233..56ad49ff8 100644
--- a/pipeline/src/image/imageRaster.ts
+++ b/pipeline/src/image/imageRaster.ts
@@ -8,8 +8,8 @@
* On the CPU rather than through a canvas, so the same source makes the same bytes on any machine and a test can read them.
*/
import { figureLight, flattenFigure, type FlatFigure } from './imageFigure';
-import { MAX_MARK_STROKES } from './imageSource';
-import type { ImageFill, ImageLayer, ImageSource, ImageAlong, ImageEffect, ImageField, ImageMark, ImageStroke, ImageSweep, ImageTransform } from './imageSource';
+import { IMAGE_BRUSHES, MAX_MARK_STROKES } from './imageSource';
+import type { ImageFill, ImageLayer, ImageSource, ImageAlong, ImageBrushSettings, ImageEffect, ImageField, ImageMark, ImageStroke, ImageSweep, ImageTransform } from './imageSource';
export interface RgbaBitmap { width: number; height: number; rgba: Uint8Array }
@@ -807,6 +807,13 @@ function paintStrand(c: Canvas, s: ImageStroke, path: readonly PathPoint[], stra
const total = path[path.length - 1]!.along;
const [startColor, endColor] = [parseColor(s.color ?? '#000000'), parseColor(s.to ?? s.color ?? '#000000')];
const [taperIn, taperOut] = s.taper ?? [0, 0];
+ const [fadeIn, fadeOut] = s.fade ?? [0, 0];
+ const roundness = s.roundness ?? 1;
+ const held = s.angle !== undefined ? [Math.cos((s.angle * Math.PI) / 180), Math.sin((s.angle * Math.PI) / 180)] as const : null;
+ const [grain, grainScale, dry, wet, edge, flow] = [s.grain ?? 0, (s.grainScale ?? 2) * scale, s.dry ?? 0, s.wet ?? 0, s.edge ?? 0, s.flow ?? 0];
+ const [scatter, dab, spacing] = [s.scatter ?? 0, s.dab ?? 1, s.spacing];
+ let loaded: [number, number, number] | null = null;
+ let stamp = 0;
const hardness = s.hardness ?? 0.5;
const size = s.size * scale;
const smudge = s.smudge ?? 0;
@@ -832,11 +839,17 @@ function paintStrand(c: Canvas, s: ImageStroke, path: readonly PathPoint[], stra
if (total > 0 && taperIn > 0 && position < taperIn) narrowing *= smoothed(position / taperIn);
if (total > 0 && taperOut > 0 && position > 1 - taperOut) narrowing *= smoothed((1 - position) / taperOut);
const width = size * (a.pressure + (b.pressure - a.pressure) * t) * narrowing;
- const radius = strand.radius === null ? width / 2 : Math.min(strand.radius, Math.max(width, strand.radius * 0.4));
+ const reach = strand.radius === null ? width / 2 : Math.min(strand.radius, Math.max(width, strand.radius * 0.4));
+ const radius = reach * dab;
const [nx, ny] = span > 0 ? [-(b.y - a.y) / span, (b.x - a.x) / span] : [0, 0];
const drift = wobble > 0 ? gradientNoise(distance / (size * 3 + 8), strand.seed * 0.37 + 0.5, strand.seed) * 2 * wobble : 0;
const away = strand.offset * width + drift;
- const [cx, cy] = [a.x + (b.x - a.x) * t + nx * away, a.y + (b.y - a.y) * t + ny * away];
+ stamp++;
+ const [off1, off2] = scatter > 0 ? [(hash01(strand.seed * 3 + 1, stamp) - 0.5) * 2, (hash01(strand.seed * 3 + 2, stamp) - 0.5) * 2] : [0, 0];
+ const [cx, cy] = [a.x + (b.x - a.x) * t + nx * away + off1 * scatter * reach, a.y + (b.y - a.y) * t + ny * away + off2 * scatter * reach];
+ let fading = 1;
+ if (total > 0 && fadeIn > 0 && position < fadeIn) fading *= position / fadeIn;
+ if (total > 0 && fadeOut > 0 && position > 1 - fadeOut) fading *= (1 - position) / fadeOut;
if (radius >= 0.12) {
const soft = Math.max(1, radius * (1 - hardness));
@@ -846,13 +859,32 @@ function paintStrand(c: Canvas, s: ImageStroke, path: readonly PathPoint[], stra
const v = Math.max(0, Math.min(1, (radius + 0.5 - d) / soft));
return soft > 1 ? smoothed(v) : v;
};
+ // Across is the wide way of the brush, as it travels or as it is held.
+ const [ax2, ay2] = held ?? [nx, ny];
+ const shaped = (dx: number, dy: number): number => (roundness >= 1 ? Math.hypot(dx, dy) : Math.hypot(dx * ax2 + dy * ay2, (dx * ay2 - dy * ax2) / roundness));
+ const toothOf = (x: number, y: number): number => {
+ if (grain <= 0 && dry <= 0) return 1;
+ let kept = 1;
+ // A wide band and not a threshold: paper takes more or less pigment at each place, and specks of all or nothing read as static.
+ if (grain > 0) kept *= Math.max(0, Math.min(1, (fractalNoise(x / grainScale, y / grainScale, 3, 77) - (0.12 + grain * 0.62)) / 0.34 + 0.5));
+ if (dry > 0) {
+ // Hairs of the brush: fine across its width and long along its travel.
+ const across = (x - cx) * nx + (y - cy) * ny + away;
+ // Hairs gather in clumps, some wide and some fine, so the gaps are not ruled lines.
+ const hairs = gradientNoise(across / Math.max(0.8, size * 0.045), distance / (size * 3 + 8), strand.seed + 5);
+ const clumps = gradientNoise(across / Math.max(1.5, size * 0.17), distance / (size * 5 + 12), strand.seed + 9);
+ const streak = 0.5 + 0.8 * hairs + 1.1 * clumps;
+ kept *= Math.max(0, Math.min(1, (streak - (0.12 + dry * 0.68)) / 0.22 + 0.5));
+ }
+ return kept;
+ };
const [x0, x1] = [Math.max(0, Math.floor(cx - radius - 1)), Math.min(w - 1, Math.ceil(cx + radius + 1))];
const [y0, y1] = [Math.max(0, Math.floor(cy - radius - 1)), Math.min(h - 1, Math.ceil(cy + radius + 1))];
if (smudge > 0) {
carried ??= sample(cx, cy);
for (let y = y0; y <= y1; y++) {
for (let x = x0; x <= x1; x++) {
- const v = cover(Math.hypot(x + 0.5 - cx, y + 0.5 - cy)) * thin * smudge * (s.opacity ?? 1);
+ const v = cover(shaped(x + 0.5 - cx, y + 0.5 - cy)) * thin * smudge * (s.opacity ?? 1) * toothOf(x + 0.5, y + 0.5);
if (v <= 0) continue;
const at = (y * w + x) * 4;
for (let k = 0; k < 4; k++) c.px[at + k] = c.px[at + k]! + (carried[k]! - c.px[at + k]!) * v * 0.5;
@@ -863,27 +895,45 @@ function paintStrand(c: Canvas, s: ImageStroke, path: readonly PathPoint[], stra
} else {
const along = total > 0 ? distance / total : 0;
const mix = (k: number): number => Math.max(0, Math.min(1, (startColor[k]! + (endColor[k]! - startColor[k]!) * along) * strand.shade));
- const [r, g, bl] = [mix(0), mix(1), mix(2)];
- const alpha = (startColor[3] + (endColor[3] - startColor[3]) * along) * (s.opacity ?? 1) * strand.strength;
+ let [r, g, bl] = [mix(0), mix(1), mix(2)];
+ if (wet > 0) {
+ const under = sample(cx, cy);
+ const here: [number, number, number] | null = under[3] > 0.02 ? [under[0] / under[3], under[1] / under[3], under[2] / under[3]] : null;
+ // It takes up a good share where it first touches, then a little with every dab, and its own paint slowly clears what it carries.
+ const pick = here ? (loaded ? wet * 0.09 : wet * 0.45) : 0;
+ const base: [number, number, number] = loaded ?? [r, g, bl];
+ const own = [r, g, bl];
+ loaded = [0, 1, 2].map((k) => {
+ const fed = base[k]! + (own[k]! - base[k]!) * 0.012;
+ return fed + ((here ? here[k]! : fed) - fed) * pick;
+ }) as [number, number, number];
+ [r, g, bl] = loaded;
+ }
+ const alpha = (startColor[3] + (endColor[3] - startColor[3]) * along) * (s.opacity ?? 1) * strand.strength * fading;
for (let y = y0; y <= y1; y++) {
for (let x = x0; x <= x1; x++) {
- const v = cover(Math.hypot(x + 0.5 - cx, y + 0.5 - cy)) * thin * alpha;
+ const v = cover(shaped(x + 0.5 - cx, y + 0.5 - cy)) * thin * alpha * toothOf(x + 0.5, y + 0.5);
const i = y * w + x;
- if (v <= buffer.mask[i]!) continue;
- if (buffer.mask[i] === 0) buffer.touched.push(i);
- buffer.mask[i] = v;
+ const was = buffer.mask[i]!;
+ const next = flow > 0 ? was + v * flow * (1 - was) : v;
+ if (next <= was) continue;
+ if (was === 0) buffer.touched.push(i);
+ buffer.mask[i] = next;
buffer.rgb[i * 3] = r; buffer.rgb[i * 3 + 1] = g; buffer.rgb[i * 3 + 2] = bl;
}
}
}
}
if (distance >= end || total === 0) break;
- distance = Math.min(end, distance + Math.max(0.4, radius * 0.1));
+ // Narrow along its travel, a flat brush needs its dabs closer or they show.
+ distance = Math.min(end, distance + (spacing !== undefined ? Math.max(0.3, radius * 2 * spacing) : Math.max(0.4, radius * 0.1 * roundness)));
}
// Laid down all at once, so opacity is the stroke's and not the sum of its stamps.
const mode: Blend = s.blend ?? 'normal';
for (const i of buffer.touched) {
- const v = buffer.mask[i]!;
+ const covered = buffer.mask[i]!;
+ // Where a stroke thins out at its edge, the pigment that gathered there: darker at the rim and paler inside.
+ const v = edge > 0 ? Math.min(1, covered * (1 - edge * 0.45 * smoothed(Math.min(1, covered / (s.opacity ?? 1))) + edge * 1.1 * Math.max(0, 1 - Math.abs(covered / (s.opacity ?? 1) - 0.4) * 3))) : covered;
if (s.erase) for (let k = 0; k < 4; k++) c.px[i * 4 + k]! *= 1 - v;
else put(c, i, buffer.rgb[i * 3]!, buffer.rgb[i * 3 + 1]!, buffer.rgb[i * 3 + 2]!, v, mode);
buffer.mask[i] = 0;
@@ -891,7 +941,8 @@ function paintStrand(c: Canvas, s: ImageStroke, path: readonly PathPoint[], stra
buffer.touched.length = 0;
}
-function paintStroke(c: Canvas, s: ImageStroke, m: Matrix, scale: number, n: number, buffer: StrokeBuffer): void {
+function paintStroke(c: Canvas, given: ImageStroke, m: Matrix, scale: number, n: number, buffer: StrokeBuffer): void {
+ const s: ImageStroke = given.brush ? { ...IMAGE_BRUSHES[given.brush], ...given } : given;
const path = brushPath(s.points, m);
const seed = s.seed ?? n + 1;
const count = s.bristles ?? 1;
@@ -937,6 +988,12 @@ function resampled(guide: readonly (readonly number[])[], samples: number): { x:
const SWEEP_SAMPLES = 14;
/** Hairs between the guides. Each belongs to a lock and draws into it toward the tip, and a lock shares a shade, which is what makes a mass read as locks and not as combed lines. */
+const brushSettings = (s: ImageBrushSettings & { brush?: string }): Partial<ImageStroke> => {
+ const out: Record<string, unknown> = {};
+ for (const key of ['brush', 'roundness', 'angle', 'grain', 'grainScale', 'dry', 'wet', 'edge', 'scatter', 'dab', 'spacing', 'flow', 'fade'] as const) if (s[key as keyof typeof s] !== undefined) out[key] = s[key as keyof typeof s];
+ return out as Partial<ImageStroke>;
+};
+
function sweepStrokes(s: ImageSweep): ImageStroke[] {
const guides = s.guides.map((g) => resampled(g, SWEEP_SAMPLES));
const seed = s.seed ?? 1;
@@ -984,9 +1041,10 @@ function sweepStrokes(s: ImageSweep): ImageStroke[] {
const root = colorAt(u + (pick(7) - 0.5) * 0.08);
const size = (s.size ?? 1.2) * (0.7 + pick(8) * 0.6);
out.push({
- points, size, hardness: 0.7, color: toHex(root, shade),
+ ...brushSettings(s),
+ points, size, ...(s.brush && s.hardness === undefined ? {} : { hardness: s.hardness ?? 0.7 }), color: toHex(root, shade),
...(tip ? { to: toHex([root[0] + (tip[0] - root[0]) * 0.8, root[1] + (tip[1] - root[1]) * 0.8, root[2] + (tip[2] - root[2]) * 0.8, root[3]], shade) } : {}),
- opacity: (s.opacity ?? 1) * (strays ? 0.55 : 0.75 + pick(9) * 0.25),
+ opacity: (s.opacity ?? (s.brush ? IMAGE_BRUSHES[s.brush]!.opacity ?? 1 : 1)) * (strays ? 0.55 : 0.75 + pick(9) * 0.25),
taper: s.taper ?? [0.04, 0.45],
});
}
@@ -1079,8 +1137,13 @@ function fieldStrokes(s: ImageField, w: number, h: number, m: Matrix, drawn: Dra
x += rx * forward; y += ry * forward;
}
out.push({
- points, size: s.size * (0.75 + pick(4) * 0.5), hardness: s.hardness ?? 0.25, opacity: s.opacity ?? 0.6,
- ...(s.smudge ? { smudge: s.smudge } : { color: toHex(paint!(sx, sy), 1 + (pick(5) - 0.5) * 2 * vary) }), taper: [0.12, 0.3],
+ ...brushSettings(s),
+ points, size: s.size * (0.75 + pick(4) * 0.5),
+ // A named brush keeps what it is, and only a field without one takes the soft defaults.
+ ...(s.brush ? {} : { hardness: s.hardness ?? 0.25, opacity: s.opacity ?? 0.6 }),
+ ...(s.brush && s.hardness !== undefined ? { hardness: s.hardness } : {}),
+ ...(s.brush && s.opacity !== undefined ? { opacity: s.opacity } : {}),
+ ...(s.smudge ? { smudge: s.smudge } : { color: toHex(paint!(sx, sy), 1 + (pick(5) - 0.5) * 2 * vary) }), taper: s.taper ?? [0.12, 0.3],
});
}
return out;
diff --git a/pipeline/src/image/imageSource.ts b/pipeline/src/image/imageSource.ts
index b0b1ac46a..90d87d3c4 100644
--- a/pipeline/src/image/imageSource.ts
+++ b/pipeline/src/image/imageSource.ts
@@ -39,8 +39,58 @@ export interface ImageTransform {
export type ImageShape = 'rect' | 'circle' | 'ellipse' | 'ring' | 'polygon' | 'line' | 'star' | 'gear' | 'path' | 'pixels' | 'paint' | 'figure' | 'image' | 'group';
+/** How a brush lays colour. Any of these on a stroke, a sweep or a field, over what its `brush` starts from. */
+export interface ImageBrushSettings {
+ hardness?: number;
+ opacity?: number;
+ taper?: readonly [number, number];
+ /** 1 round; lower is a flat brush, thin along its travel and wide across it. */
+ roundness?: number;
+ /** Degrees the flat of the brush is held at, which gives thick and thin as a pen does. Without it the brush turns with the stroke. */
+ angle?: number;
+ /** 0..1: how much the tooth of the paper breaks it up, in a texture fixed to the picture. */
+ grain?: number;
+ grainScale?: number;
+ /** 0..1: how much the brush runs dry, in streaks along its travel. */
+ dry?: number;
+ /** 0..1: how much it picks up what is under it as it goes. */
+ wet?: number;
+ /** 0..1: pigment gathered at the edge, as watercolour dries. */
+ edge?: number;
+ /** Each dab this many brush radii off the line. */
+ scatter?: number;
+ /** The dabs as a share of the brush's size: small, and scattered, they are a spray. */
+ dab?: number;
+ /** Between dabs, as a share of a dab's width. */
+ spacing?: number;
+ /** 0..1: each dab adds this much, so paint builds where it passes, as an airbrush does, and opacity no longer caps it. */
+ flow?: number;
+ /** How much of the length at each end fades out, each 0..1. */
+ fade?: readonly [number, number];
+}
+
+/** Named starting points, in the words a painter uses. */
+export const IMAGE_BRUSHES: Readonly<Record<string, ImageBrushSettings>> = {
+ ink: { hardness: 1, taper: [0.12, 0.3] },
+ pen: { hardness: 1, roundness: 0.25, angle: -45 },
+ pencil: { hardness: 0.6, grain: 0.35, grainScale: 1.2, opacity: 0.75 },
+ charcoal: { hardness: 0.35, grain: 0.5, grainScale: 2.2, dry: 0.12, opacity: 0.95 },
+ chalk: { hardness: 0.7, grain: 0.55, grainScale: 6, opacity: 0.95 },
+ flat: { hardness: 0.85, roundness: 0.3, dry: 0.3 },
+ oil: { hardness: 0.65, roundness: 0.55, dry: 0.22, wet: 0.55, opacity: 0.95 },
+ gouache: { hardness: 0.8, roundness: 0.5, wet: 0.2, dry: 0.08, opacity: 0.95 },
+ watercolor: { hardness: 0.2, wet: 0.45, edge: 0.5, opacity: 0.45, grain: 0.05, grainScale: 5 },
+ marker: { hardness: 0.9, roundness: 0.45, angle: -30, opacity: 0.72 },
+ airbrush: { hardness: 0, flow: 0.05 },
+ spray: { hardness: 0.7, dab: 0.07, scatter: 1, spacing: 0.5, opacity: 0.9 },
+ dry: { hardness: 0.75, roundness: 0.35, dry: 0.65, opacity: 0.95 },
+};
+const BRUSH_FIELDS = ['brush', 'roundness', 'angle', 'grain', 'grainScale', 'dry', 'wet', 'edge', 'scatter', 'dab', 'spacing', 'flow', 'fade'];
+
/** One movement of a brush. Replayed in order on a `paint` layer, so the same list is the same picture. */
-export interface ImageStroke {
+export interface ImageStroke extends ImageBrushSettings {
+ /** A named brush from IMAGE_BRUSHES, which what else is given here overrides. */
+ brush?: string;
/** Through these, smoothly; a third number is pressure, 1 by default. One point is a dab. */
points: readonly (readonly number[])[];
color?: string;
@@ -69,8 +119,9 @@ export interface ImageStroke {
}
/** Many hairs between guide lines, gathered into locks. A few words of intent, and hundreds of strokes when drawn. */
-export interface ImageSweep {
+export interface ImageSweep extends ImageBrushSettings {
type: 'sweep';
+ brush?: string;
/** Two or more lines, side by side and in order, each root to tip. */
guides: readonly (readonly (readonly number[])[])[];
/** Across the guides, first to last. */
@@ -93,8 +144,9 @@ export interface ImageSweep {
}
/** A region covered in strokes that follow a flow, each its colour from `tone` where it starts. */
-export interface ImageField {
+export interface ImageField extends ImageBrushSettings {
type: 'field';
+ brush?: string;
/** An SVG path to start strokes in, or "@id" for where an earlier layer drew; the whole picture without it. Strokes may run over its edge. */
region?: string;
/** Lines the strokes follow, nearest the most, or a direction in degrees clockwise from right. */
@@ -229,11 +281,25 @@ function checkFill(v: unknown): string | null {
const FIELDS = ['id', 'shape', 'at', 'size', 'radius', 'radii', 'inner', 'count', 'tooth', 'turn', 'points', 'd', 'rows', 'palette', 'scale', 'layers', 'fill', 'stroke', 'opacity', 'blend', 'transform', 'mirror', 'effects', 'clip', 'erase', 'visible', 'blur', 'strokes', 'from', 'guide', 'height', 'yaw', 'pitch', 'build', 'pose', 'light', 'src', 'except'];
const SHAPES: readonly ImageShape[] = ['rect', 'circle', 'ellipse', 'ring', 'polygon', 'line', 'star', 'gear', 'path', 'pixels', 'paint', 'figure', 'image', 'group'];
-const STROKE_FIELDS = ['points', 'color', 'to', 'size', 'hardness', 'opacity', 'taper', 'bristles', 'strand', 'spread', 'vary', 'wobble', 'seed', 'smudge', 'erase', 'blend'];
+const STROKE_FIELDS = ['points', 'color', 'to', 'size', 'hardness', 'opacity', 'taper', 'bristles', 'strand', 'spread', 'vary', 'wobble', 'seed', 'smudge', 'erase', 'blend', ...BRUSH_FIELDS];
-const SWEEP_FIELDS = ['type', 'guides', 'colors', 'count', 'tip', 'size', 'clumps', 'gather', 'flyaway', 'vary', 'wobble', 'opacity', 'taper', 'seed'];
+/** What a brush's own settings must be, on whatever carries them. */
+function brushProblem(s: Record<string, unknown>): string | null {
+ if (s.brush !== undefined && !(typeof s.brush === 'string' && Object.hasOwn(IMAGE_BRUSHES, s.brush))) return `brush ${JSON.stringify(s.brush)} must be one of ${Object.keys(IMAGE_BRUSHES).join(', ')}`;
+ for (const key of ['roundness', 'grain', 'dry', 'wet', 'edge', 'dab', 'flow']) {
+ if (s[key] !== undefined && !(isNum(s[key]) && (s[key] as number) >= 0 && (s[key] as number) <= 1)) return `${key} must be 0..1`;
+ }
+ if (s.roundness === 0 || s.dab === 0) return `${s.roundness === 0 ? 'roundness' : 'dab'} must be above 0`;
+ for (const key of ['grainScale', 'spacing']) if (s[key] !== undefined && !(isNum(s[key]) && (s[key] as number) > 0)) return `${key} must be positive`;
+ if (s.scatter !== undefined && !(isNum(s.scatter) && (s.scatter as number) >= 0)) return 'scatter must be 0 or more';
+ if (s.angle !== undefined && !isNum(s.angle)) return 'angle must be degrees';
+ if (s.fade !== undefined && !(Array.isArray(s.fade) && s.fade.length === 2 && s.fade.every((t) => isNum(t) && t >= 0 && t <= 1))) return 'fade must be [start, end], each 0..1';
+ return null;
+}
+
+const SWEEP_FIELDS = ['type', 'guides', 'colors', 'count', 'tip', 'size', 'clumps', 'gather', 'flyaway', 'vary', 'wobble', 'opacity', 'taper', 'seed', 'hardness', ...BRUSH_FIELDS];
const ALONG_FIELDS = ['type', 'line', 'count', 'strokes', 'scale', 'turn', 'both', 'jitter', 'seed'];
-const FIELD_FIELDS = ['type', 'region', 'guides', 'angle', 'tone', 'smudge', 'size', 'length', 'coverage', 'spread', 'vary', 'opacity', 'hardness', 'seed'];
+const FIELD_FIELDS = ['type', 'region', 'guides', 'angle', 'tone', 'smudge', 'size', 'length', 'coverage', 'spread', 'vary', 'opacity', 'hardness', 'seed', 'taper', ...BRUSH_FIELDS];
const isGuide = (g: unknown): boolean => Array.isArray(g) && g.length >= 2 && g.every((p) => Array.isArray(p) && (p.length === 2 || p.length === 3) && p.every(isNum));
export const MAX_MARK_STROKES = 30000;
@@ -245,6 +311,10 @@ function markProblem(s: Record<string, unknown>): string | null {
const fields = s.type === 'sweep' ? SWEEP_FIELDS : s.type === 'along' ? ALONG_FIELDS : FIELD_FIELDS;
for (const key of Object.keys(s)) if (!fields.includes(key)) return `has unknown field "${key}"; a ${String(s.type)} takes ${fields.join(', ')}`;
if (s.seed !== undefined && !Number.isInteger(s.seed)) return 'seed must be a whole number';
+ if (s.type !== 'along') {
+ const bad = brushProblem(s);
+ if (bad) return bad;
+ }
if (s.type === 'sweep') {
if (!Array.isArray(s.guides) || s.guides.length < 2 || !s.guides.every(isGuide)) return 'needs guides: two or more lines, each two or more [x, y]';
if (!Array.isArray(s.colors) || s.colors.length === 0 || !s.colors.every(isColor)) return 'needs colors: one or more, across the guides';
@@ -292,6 +362,10 @@ function markProblem(s: Record<string, unknown>): string | null {
function strokeProblem(s: Record<string, unknown>): string | null {
if (typeof s !== 'object' || s === null) return 'must be an object';
+ if (s.type === undefined) {
+ const bad = brushProblem(s);
+ if (bad) return bad;
+ }
if (s.type === 'sweep' || s.type === 'field' || s.type === 'along') return markProblem(s);
if (s.type !== undefined) return `has type "${String(s.type)}"; there are sweep, field and along, or none for one stroke`;
for (const key of Object.keys(s)) if (!STROKE_FIELDS.includes(key)) return `has unknown field "${key}"; a stroke takes ${STROKE_FIELDS.join(', ')}`;
diff --git a/pipeline/tests/image-raster.test.ts b/pipeline/tests/image-raster.test.ts
index 15d897d17..78ffe3d16 100644
--- a/pipeline/tests/image-raster.test.ts
+++ b/pipeline/tests/image-raster.test.ts
@@ -3,7 +3,7 @@
/** A picture from layers: each draws what it says and where, edges are antialiased, and what is wrong with a source is said before anything is drawn. */
import { describe, expect, it } from 'vitest';
import { imageSheet, parseColor, parsePath, pickColors, renderImage, setSheet, valuesImage, zoomImage } from '../src/image/imageRaster';
-import { applyImageEdits, strokeWidths, validateImageSource, type ImageLayer } from '../src/image/imageSource';
+import { applyImageEdits, IMAGE_BRUSHES, strokeWidths, validateImageSource, type ImageLayer } from '../src/image/imageSource';
const draw = (layers: unknown[], size: number | [number, number] = 64, background?: string): ReturnType<typeof renderImage> => renderImage(validateImageSource({ size, layers, ...(background ? { background } : {}) }));
const px = (img: { width: number; rgba: Uint8Array }, x: number, y: number): number[] => Array.from(img.rgba.subarray((y * img.width + x) * 4, (y * img.width + x) * 4 + 4));
@@ -1153,6 +1153,123 @@ describe('what the second trial tripped on', () => {
});
});
+describe('what a brush is made of', () => {
+ const paint = (strokes: object[], extra: object = {}): ImageLayer => ({ id: 'p', shape: 'paint', strokes, ...extra } as never);
+ const line = (fields: object = {}): object => ({ points: [[8, 64], [120, 64]], color: RED, size: 30, hardness: 0.9, ...fields });
+ const pic = (strokes: object[]): ReturnType<typeof draw> => draw([paint(strokes)], [128, 128]);
+ /** How many pixels along a row inside the stroke are not fully painted. */
+ const gaps = (img: ReturnType<typeof draw>, y = 64): number => { let n = 0; for (let x = 24; x < 104; x++) if (px(img, x, y)[3]! < 128) n++; return n; };
+
+ it('a flat brush held at an angle goes thick and thin with its direction', () => {
+ const span = (points: number[][], column: boolean): number => {
+ const img = pic([{ points, color: RED, size: 30, hardness: 1, roundness: 0.15, angle: 0 }]);
+ let n = 0;
+ for (let i = 0; i < 128; i++) if ((column ? px(img, 64, i) : px(img, i, 64))[3]! > 128) n++;
+ return n;
+ };
+ // Held flat along x: a stroke going down is as wide as the brush, one going along x as thin as its edge.
+ expect(span([[64, 10], [64, 118]], false)).toBeGreaterThan(24);
+ expect(span([[10, 64], [118, 64]], true)).toBeLessThan(10);
+ });
+
+ it('a flat brush with no angle turns with the stroke, and stays its full width', () => {
+ const down = pic([{ points: [[64, 10], [64, 118]], color: RED, size: 30, hardness: 1, roundness: 0.15 }]);
+ const along = pic([{ points: [[10, 64], [118, 64]], color: RED, size: 30, hardness: 1, roundness: 0.15 }]);
+ let [a, b] = [0, 0];
+ for (let i = 0; i < 128; i++) { if (px(down, i, 64)[3]! > 128) a++; if (px(along, 64, i)[3]! > 128) b++; }
+ expect(Math.abs(a - b)).toBeLessThanOrEqual(2);
+ expect(a).toBeGreaterThan(24);
+ });
+
+ it('grain breaks a stroke up in a texture fixed to the picture, the same under every stroke', () => {
+ expect(gaps(pic([line()]))).toBe(0);
+ const rough = pic([line({ grain: 0.8 })]);
+ expect(gaps(rough)).toBeGreaterThan(10);
+ expect(gaps(pic([line({ grain: 0.8 })]))).toBe(gaps(rough));
+ const moved = pic([line({ grain: 0.8, points: [[8, 64], [100, 70], [120, 64]] })]);
+ let same = 0, both = 0;
+ for (let x = 24; x < 90; x++) { const [p, q] = [px(rough, x, 64)[3]! > 128, px(moved, x, 64)[3]! > 128]; if (p || q) both++; if (p && q) same++; }
+ expect(same / both).toBeGreaterThan(0.6);
+ });
+
+ it('dry leaves streaks that run along the stroke and not across it', () => {
+ const img = pic([line({ dry: 0.6 })]);
+ let [along, across] = [0, 0];
+ for (let i = 34; i < 94; i++) along += Math.abs(px(img, i, 64)[3]! - px(img, i + 1, 64)[3]!);
+ for (let i = 54; i < 74; i++) across += Math.abs(px(img, 64, i)[3]! - px(img, 64, i + 1)[3]!);
+ expect(across / 20).toBeGreaterThan((along / 60) * 2);
+ expect(across).toBeGreaterThan(100);
+ });
+
+ it('wet picks up what is under it and carries it along, and dry paint does not', () => {
+ const blue = { points: [[8, 64], [50, 64]], color: BLUE, size: 40, hardness: 1 };
+ const over = (wet: number): ReturnType<typeof draw> => pic([blue, { points: [[10, 64], [118, 64]], color: RED, size: 16, hardness: 1, wet }]);
+ expect(px(over(0), 30, 64)[2]).toBe(0);
+ expect(px(over(0.8), 30, 64)[2]).toBeGreaterThan(60);
+ // Out past the blue, still carrying some, and less the further it goes.
+ expect(px(over(0.8), 80, 64)[2]).toBeGreaterThan(20);
+ expect(px(over(0.8), 110, 64)[2]).toBeLessThan(px(over(0.8), 80, 64)[2]!);
+ expect(px(over(0), 80, 64)[2]).toBe(0);
+ });
+
+ it('edge gathers pigment at the rim, darker than inside', () => {
+ const img = pic([{ points: [[8, 64], [120, 64]], color: '#000000', size: 40, hardness: 0.3, opacity: 0.5, edge: 0.9 }]);
+ let best = 0;
+ for (let y = 40; y < 64; y++) best = Math.max(best, px(img, 64, y)[3]!);
+ expect(best).toBeGreaterThan(px(img, 64, 64)[3]! + 10);
+ });
+
+ it('scatter and dab make a spray, spaced and off the line', () => {
+ const img = pic([{ points: [[20, 64], [108, 64]], color: RED, size: 40, dab: 0.08, scatter: 1, spacing: 0.5, hardness: 1 }]);
+ let dots = 0, far = 0;
+ for (let y = 40; y < 90; y++) for (let x = 20; x < 108; x++) if (px(img, x, y)[3]! > 60) { dots++; if (Math.abs(y - 64) > 8) far++; }
+ expect(far).toBeGreaterThan(20);
+ expect(dots).toBeLessThan(88 * 50 * 0.9);
+ });
+
+ it('flow builds up over its own passes, as opacity does not', () => {
+ const once = pic([{ points: [[8, 64], [120, 64]], color: RED, size: 30, hardness: 0.5, flow: 0.01 }]);
+ const twice = pic([{ points: [[8, 64], [120, 64], [8, 64]], color: RED, size: 30, hardness: 0.5, flow: 0.01 }]);
+ expect(px(twice, 64, 64)[3]).toBeGreaterThan(px(once, 64, 64)[3]! + 10);
+ });
+
+ it('fade thins it out at its ends, in strength and not in width', () => {
+ const img = pic([line({ fade: [0, 0.5], hardness: 1 })]);
+ expect(px(img, 30, 64)[3]).toBeGreaterThan(px(img, 100, 64)[3]! + 80);
+ expect(px(img, 100, 70)[3]).toBeGreaterThan(0);
+ });
+
+ it('a named brush starts from its settings, and what is given overrides them', () => {
+ const base = pic([line()]);
+ const named = pic([line({ brush: 'charcoal', hardness: undefined })]);
+ expect(named.rgba).not.toEqual(base.rgba);
+ expect(gaps(named)).toBeGreaterThan(5);
+ expect(gaps(pic([line({ brush: 'charcoal', grain: 0, dry: 0 })]))).toBe(0);
+ for (const brush of Object.keys(IMAGE_BRUSHES)) expect(pic([line({ brush })]).notes).toEqual([]);
+ expect(() => validateImageSource({ size: 8, layers: [paint([line({ brush: 'crayon' })])] })).toThrow(/brush "crayon" must be one of ink,/);
+ });
+
+ it('is carried by a field and by a sweep', () => {
+ const field = (extra: object): ReturnType<typeof draw> => pic([{ type: 'field', tone: RED, size: 12, length: 40, angle: 0, seed: 1, coverage: 1, ...extra }]);
+ expect(field({ brush: 'dry' }).rgba).not.toEqual(field({}).rgba);
+ const sweep = (extra: object): ReturnType<typeof draw> => pic([{ type: 'sweep', guides: [[[10, 10], [10, 110]], [[60, 10], [60, 110]]], colors: [RED], count: 60, size: 3, seed: 1, ...extra }]);
+ expect(sweep({ brush: 'pencil' }).rgba).not.toEqual(sweep({}).rgba);
+ });
+
+ it('is the same every time', () => {
+ for (const brush of ['oil', 'watercolor', 'spray']) expect(pic([line({ brush })]).rgba).toEqual(pic([line({ brush })]).rgba);
+ });
+
+ it('says what is wrong with a setting', () => {
+ const bad = (fields: object): unknown => validateImageSource({ size: 8, layers: [paint([line(fields)])] });
+ expect(() => bad({ roundness: 0 })).toThrow(/roundness must be above 0/);
+ expect(() => bad({ grain: 2 })).toThrow(/grain must be 0..1/);
+ expect(() => bad({ fade: [0] })).toThrow(/fade must be/);
+ expect(() => bad({ spacing: 0 })).toThrow(/spacing must be positive/);
+ expect(() => bad({ angle: 'up' })).toThrow(/angle must be degrees/);
+ });
+});
+
describe('what it measures', () => {
it('finds where the picture is solid and how much of it that is', () => {
const img = draw([{ shape: 'rect', at: [10, 20], size: [16, 8], fill: RED }]);
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index f37e0e67b..17bb445de 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -3888,6 +3888,10 @@ export const NOTED = {
{"note":"An AI can draw 2D pictures."},
"feat(look): layers moved and wrapped, one pixel changed, a set and a close look drawn":
{"note":"An AI can draw 2D pictures."},
+ "feat(look): brushes: flat tips, paper grain, dry streaks, wet mixing, and named media":
+ {"note":"An AI can draw 2D pictures."},
+ "feat(mcp): image_write documents the brushes":
+ {"note":"An AI can draw 2D pictures."},
"feat(look): a figure is bones and blocks, and a picture from the project can be drawn over":
{"note":"An AI can draw 2D pictures."},
"feat(mcp): a picture as a reference, and the body no longer offered to build a character on":