Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(sound): a sound effect is the layers that make it - an .essound of tones, noise and the ringing of struck things, each shaped by an envelope, filters, a bend, a drive and repeats, with an echo and a room over the whole, makes the .wav a game plays; and a picture of the sound, its wave over its pitches, for whoever cannot hear it

yhhfcc02b0e6457
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b60c5264e..652106118 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,6 +19,7 @@ published separately; it ships inside the editor.
- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.
- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.
- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.
+- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. Nothing has been listened to: the recipes in the tool text were judged by their pictures.
- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.
- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.
- **An AI can lay leaflets along a stem in one step.** `scatter` along a `path` pointed every copy the way the path runs, turned at random, so the leaflets of a frond, the barbs of a feather or the legs of a centipede had to be written out one step each: a palm was 386 steps and a 749 KB model file. `rows` (1 or 2) stands the copies off the path instead, in a row or in pairs, `sweep` degrees from the way it runs and `lift` out of the plane, each as one number or as a start and an end. A copy's +z faces `up`, so a flat blade keeps its face to the sky whichever way it points. `sizes` shapes them along the path, widest in the middle if you say so. The same palm is 40 steps and 38 KB, and bakes to the same triangle count. A scatter without `rows` is as it was.
diff --git a/desktop b/desktop
index 48b713cac..abfd7414a 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 48b713cace41f8f56bd56cb0f4c44d30fdebe682
+Subproject commit abfd7414a992984703f7520009bf4c705a598096
diff --git a/pipeline/src/sound/soundPicture.ts b/pipeline/src/sound/soundPicture.ts
new file mode 100644
index 000000000..2679566f7
--- /dev/null
+++ b/pipeline/src/sound/soundPicture.ts
@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file soundPicture.ts — a sound as something to look at.
+ *
+ * Whoever writes a sound source here cannot hear it, and a list of numbers does not say whether a gunshot has a
+ * crack on the front of it. A picture does. The top band is the wave: how loud, when. Under it is the sound laid
+ * out by pitch against time, low at the bottom and high at the top, brighter where there is more: a thump is a
+ * blot low on the left, a hiss is a haze across the top, a tone is a line, a tone that falls is a line that falls.
+ * Faint lines mark 100 Hz, 1 kHz and 10 kHz, and tenths of a second, or whole seconds on a long sound.
+ */
+import type { RenderedSound } from './soundRender';
+
+export interface SoundPicture { width: number; height: number; rgba: Uint8Array }
+
+/** In place: the transform of @p re + i @p im, their length a power of two. */
+function fft(re: Float64Array, im: Float64Array): void {
+ const n = re.length;
+ for (let i = 1, j = 0; i < n; i++) {
+ let bit = n >> 1;
+ for (; j & bit; bit >>= 1) j ^= bit;
+ j ^= bit;
+ if (i < j) { [re[i], re[j]] = [re[j]!, re[i]!]; [im[i], im[j]] = [im[j]!, im[i]!]; }
+ }
+ for (let size = 2; size <= n; size <<= 1) {
+ const angle = (-2 * Math.PI) / size;
+ const [wr, wi] = [Math.cos(angle), Math.sin(angle)];
+ for (let start = 0; start < n; start += size) {
+ let [cr, ci] = [1, 0];
+ for (let k = 0; k < size / 2; k++) {
+ const [a, b] = [start + k, start + k + size / 2];
+ const [tr, ti] = [re[b]! * cr - im[b]! * ci, re[b]! * ci + im[b]! * cr];
+ re[b] = re[a]! - tr; im[b] = im[a]! - ti;
+ re[a] = re[a]! + tr; im[a] = im[a]! + ti;
+ [cr, ci] = [cr * wr - ci * wi, cr * wi + ci * wr];
+ }
+ }
+ }
+}
+
+const LOW = 40;
+const HIGH = 20000;
+/** From nothing through violet and orange to white, so that a little shows and a lot is not all one colour. */
+const HEAT: readonly (readonly [number, number, number])[] = [[8, 8, 14], [48, 18, 92], [140, 32, 110], [226, 84, 58], [250, 182, 60], [255, 250, 228]];
+
+function heat(t: number): [number, number, number] {
+ const f = Math.max(0, Math.min(1, t)) * (HEAT.length - 1);
+ const i = Math.min(HEAT.length - 2, Math.floor(f));
+ return [0, 1, 2].map((k) => Math.round(HEAT[i]![k]! + (HEAT[i + 1]![k]! - HEAT[i]![k]!) * (f - i))) as [number, number, number];
+}
+
+/** How much of the sound's energy lies under 200 Hz, from there to 2 kHz, and above, each 0..1. */
+export function soundBands(sound: RenderedSound): { low: number; mid: number; high: number } {
+ const size = 2048;
+ const mono = sound.channels[0]!.map((v, i) => (sound.channels.length === 2 ? (v + sound.channels[1]![i]!) / 2 : v));
+ const sums = [0, 0, 0];
+ for (let start = 0; start < mono.length; start += size) {
+ const [re, im] = [new Float64Array(size), new Float64Array(size)];
+ for (let i = 0; i < size && start + i < mono.length; i++) re[i] = mono[start + i]! * (0.5 - 0.5 * Math.cos((2 * Math.PI * i) / size));
+ fft(re, im);
+ for (let k = 1; k < size / 2; k++) {
+ const hz = (k * sound.rate) / size;
+ sums[hz < 200 ? 0 : hz < 2000 ? 1 : 2]! += re[k]! * re[k]! + im[k]! * im[k]!;
+ }
+ }
+ const all = sums[0]! + sums[1]! + sums[2]! || 1;
+ const share = (v: number): number => Math.round((v / all) * 100) / 100;
+ return { low: share(sums[0]!), mid: share(sums[1]!), high: share(sums[2]!) };
+}
+
+/** The wave over the pitches, @p width pixels across. */
+export function soundPicture(sound: RenderedSound, width = 768): SoundPicture {
+ const [waveH, specH] = [120, 264];
+ const height = waveH + specH;
+ const rgba = new Uint8Array(width * height * 4);
+ const put = (x: number, y: number, r: number, g: number, b: number, a = 1): void => {
+ if (x < 0 || x >= width || y < 0 || y >= height) return;
+ const o = (y * width + x) * 4;
+ rgba[o] = Math.round(rgba[o]! * (1 - a) + r * a);
+ rgba[o + 1] = Math.round(rgba[o + 1]! * (1 - a) + g * a);
+ rgba[o + 2] = Math.round(rgba[o + 2]! * (1 - a) + b * a);
+ rgba[o + 3] = 255;
+ };
+ for (let y = 0; y < height; y++) for (let x = 0; x < width; x++) put(x, y, y < waveH ? 22 : 8, y < waveH ? 24 : 8, y < waveH ? 30 : 14);
+
+ const frames = sound.channels[0]!.length;
+ const mono = sound.channels[0]!.map((v, i) => (sound.channels.length === 2 ? (v + sound.channels[1]![i]!) / 2 : v));
+
+ // The wave: for each column, from its lowest sample to its highest.
+ const mid = waveH / 2;
+ for (let x = 0; x < width; x++) put(x, mid, 60, 66, 80);
+ for (let x = 0; x < width; x++) {
+ const [from, to] = [Math.floor((x / width) * frames), Math.max(Math.floor((x / width) * frames) + 1, Math.floor(((x + 1) / width) * frames))];
+ let [lo, hi] = [0, 0];
+ for (let n = from; n < to && n < frames; n++) { if (mono[n]! < lo) lo = mono[n]!; if (mono[n]! > hi) hi = mono[n]!; }
+ for (let y = Math.round(mid - hi * (mid - 4)); y <= Math.round(mid - lo * (mid - 4)); y++) put(x, y, 120, 214, 255);
+ }
+
+ // The pitches: a window of the sound at each column, its strength at each height's frequency.
+ const size = 2048;
+ const [re, im] = [new Float64Array(size), new Float64Array(size)];
+ const column = new Float32Array(size / 2);
+ for (let x = 0; x < width; x++) {
+ const centre = Math.floor(((x + 0.5) / width) * frames);
+ re.fill(0); im.fill(0);
+ for (let i = 0; i < size; i++) {
+ const n = centre - size / 2 + i;
+ if (n >= 0 && n < frames) re[i] = mono[n]! * (0.5 - 0.5 * Math.cos((2 * Math.PI * i) / size));
+ }
+ fft(re, im);
+ for (let k = 0; k < size / 2; k++) column[k] = Math.hypot(re[k]!, im[k]!) / (size / 4);
+ for (let y = 0; y < specH; y++) {
+ const hz = LOW * Math.pow(HIGH / LOW, 1 - y / (specH - 1));
+ const bin = (hz * size) / sound.rate;
+ // Between two bins, and across all the bins one row covers where they are packed close.
+ const next = (LOW * Math.pow(HIGH / LOW, 1 - (y - 1) / (specH - 1)) * size) / sound.rate;
+ let v = 0;
+ for (let k = Math.max(1, Math.floor(bin)); k <= Math.min(size / 2 - 1, Math.max(Math.ceil(bin), Math.floor(next))); k++) v = Math.max(v, column[k]!);
+ const db = 20 * Math.log10(v + 1e-9);
+ const [r, g, b] = heat((db + 78) / 72);
+ put(x, waveH + y, r, g, b);
+ }
+ }
+
+ // What to read it against: 100 Hz, 1 kHz and 10 kHz across, and time down.
+ for (const hz of [100, 1000, 10000]) {
+ const y = waveH + Math.round((1 - Math.log(hz / LOW) / Math.log(HIGH / LOW)) * (specH - 1));
+ for (let x = 0; x < width; x += 2) put(x, y, 255, 255, 255, 0.28);
+ }
+ const tick = sound.seconds > 3 ? 1 : 0.1;
+ for (let t = tick; t < sound.seconds - 1e-6; t += tick) {
+ const x = Math.round((t / sound.seconds) * width);
+ for (let y = 0; y < height; y += y < waveH ? 1 : 3) put(x, y, 255, 255, 255, y < waveH ? 0.1 : 0.22);
+ }
+ for (let x = 0; x < width; x++) put(x, waveH, 90, 96, 110);
+ return { width, height, rgba };
+}
diff --git a/pipeline/src/sound/soundRender.ts b/pipeline/src/sound/soundRender.ts
new file mode 100644
index 000000000..1c61d6a14
--- /dev/null
+++ b/pipeline/src/sound/soundRender.ts
@@ -0,0 +1,323 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file soundRender.ts — a sound source worked out into samples, and the samples as a `.wav`.
+ *
+ * Every layer is rendered on its own, placed in time, and summed; then what the source says of the whole (a
+ * filter, an echo, a room, a level) is applied to the sum. Nothing depends on the machine: no random number is
+ * drawn that is not seeded, and the arithmetic is plain doubles.
+ */
+import type { SoundCurve, SoundFilter, SoundLayer, SoundSource } from './soundSource';
+
+export interface RenderedSound {
+ rate: number;
+ /** One for a sound with nothing off-centre, two otherwise: left, right. */
+ channels: Float32Array[];
+ seconds: number;
+ /** The loudest moment, 0..1, after everything. */
+ peak: number;
+ /** The level over the whole, as the square root of the mean square. */
+ rms: number;
+ /** Where each layer sounds and how loud it is on its own, before the sum. */
+ layers: { id?: string; source: string; from: number; to: number; peak: number }[];
+ notes: string[];
+}
+
+const TAU = Math.PI * 2;
+
+/** A curve read at @p t seconds: straight between its points, or along the ratio between them for a pitch. */
+function curveAt(curve: SoundCurve | undefined, t: number, fallback: number, ratio: boolean): number {
+ if (curve === undefined) return fallback;
+ if (typeof curve === 'number') return curve;
+ if (t <= curve[0]![0]) return curve[0]![1];
+ for (let i = 1; i < curve.length; i++) {
+ const [t1, v1] = curve[i]!;
+ if (t > t1) continue;
+ const [t0, v0] = curve[i - 1]!;
+ const w = (t - t0) / (t1 - t0);
+ // A pitch glides by ratio: halfway from 200 to 800 Hz is 400, as the ear has it.
+ return ratio && v0 > 0 && v1 > 0 ? v0 * Math.pow(v1 / v0, w) : v0 + (v1 - v0) * w;
+ }
+ return curve.at(-1)![1];
+}
+
+/** The same numbers for the same seed, -1..1. */
+function noiseOf(seed: number): () => number {
+ let s = (seed * 2654435761 + 0x9e3779b9) >>> 0;
+ return () => {
+ s = (s + 0x6d2b79f5) >>> 0;
+ let t = s;
+ t = Math.imul(t ^ (t >>> 15), t | 1);
+ t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
+ return ((t ^ (t >>> 14)) >>> 0) / 2147483648 - 1;
+ };
+}
+
+/** What a sharp step would alias as, taken out of the two samples beside it. */
+function polyBlep(phase: number, step: number): number {
+ if (phase < step) { const t = phase / step; return t + t - t * t - 1; }
+ if (phase > 1 - step) { const t = (phase - 1) / step; return t * t + t + t + 1; }
+ return 0;
+}
+
+/** A two-pole filter whose corner may move: the coefficients are worked out again as it does. */
+class Biquad {
+ private x1 = 0; private x2 = 0; private y1 = 0; private y2 = 0;
+ private b0 = 1; private b1 = 0; private b2 = 0; private a1 = 0; private a2 = 0;
+ private last = -1;
+ constructor(private readonly type: SoundFilter['type'], private readonly q: number, private readonly rate: number) {}
+
+ tune(freq: number): void {
+ const f = Math.max(20, Math.min(this.rate * 0.45, freq));
+ if (Math.abs(f - this.last) < 0.01) return;
+ this.last = f;
+ const w = (TAU * f) / this.rate;
+ const [cos, alpha] = [Math.cos(w), Math.sin(w) / (2 * this.q)];
+ const a0 = 1 + alpha;
+ let b0: number, b1: number, b2: number;
+ switch (this.type) {
+ case 'lowpass': [b0, b1, b2] = [(1 - cos) / 2, 1 - cos, (1 - cos) / 2]; break;
+ case 'highpass': [b0, b1, b2] = [(1 + cos) / 2, -(1 + cos), (1 + cos) / 2]; break;
+ case 'bandpass': [b0, b1, b2] = [alpha, 0, -alpha]; break;
+ default: [b0, b1, b2] = [1, -2 * cos, 1];
+ }
+ [this.b0, this.b1, this.b2, this.a1, this.a2] = [b0 / a0, b1 / a0, b2 / a0, (-2 * cos) / a0, (1 - alpha) / a0];
+ }
+
+ run(x: number): number {
+ const y = this.b0 * x + this.b1 * this.x1 + this.b2 * this.x2 - this.a1 * this.y1 - this.a2 * this.y2;
+ [this.x2, this.x1, this.y2, this.y1] = [this.x1, x, this.y1, y];
+ return y;
+ }
+}
+
+/** One striking of a layer: its samples from its own start, at @p pitch times its written pitch. */
+function strike(layer: SoundLayer, seconds: number, rate: number, pitch: number, seed: number): Float32Array {
+ const count = Math.max(1, Math.round(seconds * rate));
+ const out = new Float32Array(count);
+ const random = noiseOf(seed);
+ const filters = (layer.filters ?? []).map((f) => ({ run: new Biquad(f.type, f.q ?? 0.7, rate), freq: f.freq }));
+ const [attack, release] = [layer.attack ?? 0.002, layer.release ?? 0.01];
+ const drive = layer.drive ?? 0;
+ const push = 1 + drive * 14;
+ const pushed = Math.tanh(push);
+ let [phase, bend] = [0, 0];
+ // Pink and brown are white put through a few poles; these are their memories.
+ let [p0, p1, p2, brown] = [0, 0, 0, 0];
+
+ for (let n = 0; n < count; n++) {
+ const t = n / rate;
+ let v: number;
+ if (layer.source === 'noise') {
+ const white = random();
+ if (layer.color === 'pink') {
+ p0 = 0.99765 * p0 + white * 0.099046;
+ p1 = 0.963 * p1 + white * 0.2965164;
+ p2 = 0.57 * p2 + white * 1.0526913;
+ v = (p0 + p1 + p2 + white * 0.1848) * 0.25;
+ } else if (layer.color === 'brown') {
+ brown = (brown + white * 0.02) / 1.02;
+ v = brown * 3.5;
+ } else v = white;
+ } else if (layer.source === 'modes') {
+ v = 0;
+ for (const [hz, loud, dies] of layer.modes!) {
+ const f = hz * pitch;
+ if (f < rate * 0.48) v += loud * Math.exp(-t / dies) * Math.sin(TAU * f * t);
+ }
+ } else {
+ let hz = curveAt(layer.freq, t, 440, true) * pitch;
+ if (layer.vibrato) hz *= Math.pow(2, (layer.vibrato.depth * Math.sin(TAU * layer.vibrato.rate * t)) / 12);
+ const step = Math.min(0.49, hz / rate);
+ phase += step;
+ if (phase >= 1) phase -= Math.floor(phase);
+ let at = phase;
+ if (layer.fm) {
+ bend += step * layer.fm.ratio;
+ if (bend >= 1) bend -= Math.floor(bend);
+ at = phase + (curveAt(layer.fm.index, t, 0, false) * Math.sin(TAU * bend)) / TAU;
+ at -= Math.floor(at);
+ }
+ switch (layer.source) {
+ case 'sine': v = Math.sin(TAU * at); break;
+ case 'triangle': v = 1 - 4 * Math.abs(at - 0.5); break;
+ case 'saw': v = 2 * at - 1 - (layer.fm ? 0 : polyBlep(at, step)); break;
+ default: {
+ const width = layer.width ?? 0.5;
+ v = at < width ? 1 : -1;
+ if (!layer.fm) v += polyBlep(at, step) - polyBlep((at - width + 1) % 1, step);
+ }
+ }
+ }
+ for (const f of filters) {
+ // The corner is read every 16 samples: often enough for a sweep, seldom enough to cost little.
+ if ((n & 15) === 0) f.run.tune(curveAt(f.freq, t, 1000, true));
+ v = f.run.run(v);
+ }
+ if (drive > 0) v = Math.tanh(v * push) / pushed;
+ let level = curveAt(layer.gain, t, 1, false);
+ if (layer.decay) level *= Math.exp(-t / layer.decay);
+ if (layer.tremolo) level *= 1 - layer.tremolo.depth * (0.5 + 0.5 * Math.sin(TAU * layer.tremolo.rate * t));
+ if (attack > 0 && t < attack) level *= t / attack;
+ const left = seconds - t;
+ if (release > 0 && left < release) level *= Math.max(0, left / release);
+ out[n] = v * level;
+ }
+ return out;
+}
+
+/** Eight delays and four all-pass loops, as rooms have been faked since the seventies: dense enough to pass for one, cheap enough to run on a sum. */
+function room(input: Float32Array, rate: number, size: number, spread: number): Float32Array {
+ const scale = rate / 44100;
+ const combs = [1116, 1188, 1277, 1356, 1422, 1491, 1557, 1617].map((d) => ({ buf: new Float32Array(Math.round((d + spread) * scale)), at: 0, damp: 0 }));
+ const passes = [556, 441, 341, 225].map((d) => ({ buf: new Float32Array(Math.round((d + spread) * scale)), at: 0 }));
+ const feedback = 0.7 + size * 0.28;
+ const out = new Float32Array(input.length);
+ for (let n = 0; n < input.length; n++) {
+ const x = input[n]! * 0.015;
+ let sum = 0;
+ for (const c of combs) {
+ const y = c.buf[c.at]!;
+ c.damp = y * 0.8 + c.damp * 0.2;
+ c.buf[c.at] = x + c.damp * feedback;
+ c.at = (c.at + 1) % c.buf.length;
+ sum += y;
+ }
+ for (const p of passes) {
+ const y = p.buf[p.at]!;
+ p.buf[p.at] = sum + y * 0.5;
+ p.at = (p.at + 1) % p.buf.length;
+ sum = y - sum;
+ }
+ out[n] = sum;
+ }
+ return out;
+}
+
+/** The samples a validated source makes. */
+export function renderSound(source: SoundSource): RenderedSound {
+ const rate = source.rate ?? 44100;
+ const count = Math.max(1, Math.round(source.duration * rate));
+ // A loop is rendered a little long and its extra folded back over its start, so the end runs into the beginning.
+ const fold = source.loop ? Math.min(Math.round(0.08 * rate), Math.floor(count / 2)) : 0;
+ const total = count + fold;
+ const [left, right] = [new Float32Array(total), new Float32Array(total)];
+ const notes: string[] = [];
+ const reports: RenderedSound['layers'] = [];
+ let stereo = source.reverb !== undefined;
+
+ source.layers.forEach((layer, index) => {
+ if (layer.mute) return;
+ const start = layer.start ?? 0;
+ const room = total / rate - start;
+ const seconds = Math.min(layer.length ?? room, room);
+ const where = `layer ${index}${layer.id ? ` "${layer.id}"` : ''}`;
+ if (!(seconds > 0)) { notes.push(`${where} starts at the end of the sound and is not heard`); return; }
+ if (layer.length !== undefined && layer.length > room + 1e-6 && !layer.repeat) notes.push(`${where} is cut short: it runs ${layer.length} s from ${start} s in a sound of ${source.duration} s`);
+ const pan = layer.pan ?? 0;
+ if (pan !== 0) stereo = true;
+ // Equal power: a sound sent to one side is as loud as one in the middle.
+ const [gl, gr] = [Math.cos(((pan + 1) * Math.PI) / 4), Math.sin(((pan + 1) * Math.PI) / 4)];
+ const seed = layer.seed ?? index + 1;
+ const jitter = noiseOf(seed + 7919);
+ const times = layer.repeat?.count ?? 1;
+ let [peak, last] = [0, start];
+ for (let k = 0; k < times; k++) {
+ const at = start + k * (layer.repeat?.every ?? 0) + (k > 0 && layer.repeat?.jitter ? jitter() * layer.repeat.jitter : 0);
+ const from = Math.round(Math.max(0, at) * rate);
+ if (from >= total) break;
+ // Each striking has its own noise, as no two footsteps are the same grit.
+ const hit = strike(layer, Math.min(seconds, (total - from) / rate), rate, Math.pow(layer.repeat?.pitch ?? 1, k), seed + k * 131);
+ const loud = Math.pow(layer.repeat?.gain ?? 1, k);
+ for (let n = 0; n < hit.length && from + n < total; n++) {
+ const v = hit[n]! * loud;
+ left[from + n]! += v * gl;
+ right[from + n]! += v * gr;
+ if (Math.abs(v) > peak) peak = Math.abs(v);
+ }
+ last = Math.min(total / rate, at + hit.length / rate);
+ }
+ reports.push({ ...(layer.id ? { id: layer.id } : {}), source: layer.source, from: Math.round(start * 1000) / 1000, to: Math.round(last * 1000) / 1000, peak: Math.round(peak * 1000) / 1000 });
+ if (peak < 1e-4) notes.push(`${where} makes no sound: its level is nothing throughout`);
+ });
+
+ const channels = stereo ? [left, right] : [left.map((v, i) => (v + right[i]!) * Math.SQRT1_2)];
+ for (const [c, channel] of channels.entries()) {
+ for (const [key, type] of [['highpass', 'highpass'], ['lowpass', 'lowpass']] as const) {
+ if (source[key] === undefined) continue;
+ const filter = new Biquad(type, 0.7, rate);
+ filter.tune(source[key]!);
+ for (let n = 0; n < total; n++) channel[n] = filter.run(channel[n]!);
+ }
+ if (source.echo) {
+ const delay = Math.max(1, Math.round(source.echo.time * rate));
+ const [feedback, mix] = [source.echo.feedback ?? 0.35, source.echo.mix ?? 0.3];
+ const line = new Float32Array(delay);
+ for (let n = 0; n < total; n++) {
+ const heard = line[n % delay]!;
+ line[n % delay] = channel[n]! + heard * feedback;
+ channel[n] = channel[n]! + heard * mix;
+ }
+ }
+ if (source.reverb) {
+ const mix = source.reverb.mix ?? 0.25;
+ // The two sides are rooms a little different, which is what makes it wide.
+ const wet = room(channel, rate, source.reverb.size ?? 0.5, c * 23);
+ for (let n = 0; n < total; n++) channel[n] = channel[n]! * (1 - mix * 0.5) + wet[n]! * mix;
+ }
+ }
+
+ const out = channels.map((channel) => {
+ const kept = channel.subarray(0, count).slice();
+ if (fold > 0) {
+ for (let n = 0; n < fold; n++) {
+ const w = n / fold;
+ kept[n] = kept[n]! * w + channel[count + n]! * (1 - w);
+ }
+ } else {
+ // A sound that stops on a sample that is not nothing clicks: the last three milliseconds are let down.
+ const tail = Math.min(count, Math.round(0.003 * rate));
+ for (let n = 0; n < tail; n++) kept[count - 1 - n]! *= n / tail;
+ }
+ return kept;
+ });
+
+ let peak = 0;
+ for (const channel of out) for (const v of channel) if (Math.abs(v) > peak) peak = Math.abs(v);
+ const target = source.gain ?? 0.9;
+ if (source.normalize !== false) {
+ if (peak > 1e-6) for (const channel of out) for (let n = 0; n < count; n++) channel[n]! *= target / peak;
+ else notes.push('the sound is silent: nothing in it reaches a level that can be heard');
+ peak = peak > 1e-6 ? target : 0;
+ } else {
+ for (const channel of out) for (let n = 0; n < count; n++) channel[n]! *= target / 0.9;
+ peak *= target / 0.9;
+ if (peak > 1) {
+ notes.push(`the layers sum to ${peak.toFixed(2)} times full, and what is past full is cut off flat: lower their gain, or leave normalize on`);
+ for (const channel of out) for (let n = 0; n < count; n++) channel[n] = Math.max(-1, Math.min(1, channel[n]!));
+ peak = 1;
+ }
+ }
+ let squares = 0;
+ for (const channel of out) for (const v of channel) squares += v * v;
+ return { rate, channels: out, seconds: count / rate, peak, rms: Math.sqrt(squares / (count * out.length)), layers: reports, notes };
+}
+
+/** The samples as a 16-bit PCM `.wav`, which every platform the engine runs on plays. */
+export function encodeWav(sound: Pick<RenderedSound, 'rate' | 'channels'>): Uint8Array {
+ const [channels, frames] = [sound.channels.length, sound.channels[0]!.length];
+ const bytes = new Uint8Array(44 + frames * channels * 2);
+ const view = new DataView(bytes.buffer);
+ const text = (at: number, s: string): void => { for (let i = 0; i < s.length; i++) bytes[at + i] = s.charCodeAt(i); };
+ text(0, 'RIFF'); view.setUint32(4, 36 + frames * channels * 2, true); text(8, 'WAVE');
+ text(12, 'fmt '); view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, channels, true);
+ view.setUint32(24, sound.rate, true); view.setUint32(28, sound.rate * channels * 2, true); view.setUint16(32, channels * 2, true); view.setUint16(34, 16, true);
+ text(36, 'data'); view.setUint32(40, frames * channels * 2, true);
+ for (let n = 0; n < frames; n++) {
+ for (let c = 0; c < channels; c++) {
+ const v = Math.max(-1, Math.min(1, sound.channels[c]![n]!));
+ view.setInt16(44 + (n * channels + c) * 2, Math.round(v * 32767), true);
+ }
+ }
+ return bytes;
+}
diff --git a/pipeline/src/sound/soundSource.ts b/pipeline/src/sound/soundSource.ts
new file mode 100644
index 000000000..72b364803
--- /dev/null
+++ b/pipeline/src/sound/soundSource.ts
@@ -0,0 +1,269 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file soundSource.ts — `.essound`: a sound as the layers that make it, what a model writes and edits, and what is wrong with one.
+ *
+ * A picture is layers of shapes and a model is steps; a sound effect is layers too. A gunshot is a crack of
+ * noise, a thump under it and a tail; a footstep is a thud and a scrape; a glass is a handful of ringing tones.
+ * Each layer is one source (a tone, a noise, or a struck body's ringing), shaped in time by an envelope and in
+ * colour by filters, and the layers are summed. What is saved is this description, and the `.wav` a game plays is
+ * made from it, so a sound is changed by changing a number and never by recording again.
+ *
+ * Nothing here is random that is not seeded: the same source is the same samples on every machine.
+ */
+
+export const SOUND_SOURCE_EXT = '.essound';
+export const SOUND_SOURCE_VERSION = 1;
+export const MAX_SOUND_SECONDS = 30;
+export const SOUND_RATES = [22050, 44100, 48000] as const;
+
+/** One value for the whole of a layer, or [[seconds, value], ...] from the layer's start: a pitch that falls, a filter that opens. */
+export type SoundCurve = number | readonly (readonly [number, number])[];
+
+export const SOUND_SOURCES = ['sine', 'triangle', 'saw', 'square', 'noise', 'modes'] as const;
+export type SoundSourceKind = (typeof SOUND_SOURCES)[number];
+export const SOUND_FILTERS = ['lowpass', 'highpass', 'bandpass', 'notch'] as const;
+export const NOISE_COLORS = ['white', 'pink', 'brown'] as const;
+
+export interface SoundFilter {
+ type: (typeof SOUND_FILTERS)[number];
+ /** Hz, 20..20000. */
+ freq: SoundCurve;
+ /** How narrow: 0.7 is plain, 5 rings, 20 whistles. Default 0.7. */
+ q?: number;
+}
+
+export interface SoundLayer {
+ id?: string;
+ source: SoundSourceKind;
+ /** A tone's pitch in Hz; a curve glides between its points as the ear hears pitch. */
+ freq?: SoundCurve;
+ /** square: the share of each cycle it is high, 0.05..0.95. Default 0.5. */
+ width?: number;
+ /** noise: white is hiss, pink is rain, brown is a rumble. Default white. */
+ color?: (typeof NOISE_COLORS)[number];
+ /** modes: each [Hz, loudness, seconds to die away], the tones a struck thing rings with. */
+ modes?: readonly (readonly [number, number, number])[];
+ seed?: number;
+ /** Seconds into the sound that it begins. Default 0. */
+ start?: number;
+ /** Seconds it lasts. Default: to the end of the sound. */
+ length?: number;
+ /** Loudness, 1 being full; a curve shapes it over the layer. Default 1. */
+ gain?: SoundCurve;
+ /** Seconds to rise from nothing, and to fall to nothing at its end. Defaults 0.002 and 0.01, which only stop a click. */
+ attack?: number;
+ release?: number;
+ /** Dies away of itself: seconds to fall to about a third. A hit, a pluck, a drop. */
+ decay?: number;
+ /** Wavers in pitch: `rate` times a second, `depth` semitones each way. */
+ vibrato?: { rate: number; depth: number };
+ /** Wavers in loudness: `rate` times a second, by `depth` 0..1. */
+ tremolo?: { rate: number; depth: number };
+ /** A second tone bending this one: at `ratio` times its pitch, by `index` (0 is none, 2 is brassy, 8 is a clang). Metal, bells, lasers. */
+ fm?: { ratio: number; index: SoundCurve };
+ /** In order, each working on what the one before left. */
+ filters?: readonly SoundFilter[];
+ /** Pushed into distortion, 0..1: grit, fuzz, a speaker torn. */
+ drive?: number;
+ /** -1 is left, 1 is right. Default 0. */
+ pan?: number;
+ /** The layer struck again: `count` times in all, `every` seconds apart, each `gain` times as loud and `pitch` times as high as the last, off the beat by up to `jitter` seconds. */
+ repeat?: { count: number; every: number; gain?: number; pitch?: number; jitter?: number };
+ /** Left out of the sound, kept in the source. */
+ mute?: boolean;
+}
+
+export interface SoundSource {
+ version?: number;
+ /** Seconds. */
+ duration: number;
+ /** Samples a second. Default 44100. */
+ rate?: number;
+ /** Its end runs into its start with no click, for an engine, a wind, a hum. */
+ loop?: boolean;
+ layers: readonly SoundLayer[];
+ /** Over the whole sound, after the layers are summed: */
+ highpass?: number;
+ lowpass?: number;
+ /** A repeat of the whole: `time` seconds later, `feedback` 0..0.95 of each into the next, `mix` 0..1 of it in the sound. */
+ echo?: { time: number; feedback?: number; mix?: number };
+ /** A room round it: `size` 0..1 from a closet to a hall, `mix` 0..1 of it in the sound. */
+ reverb?: { size?: number; mix?: number };
+ /** Loudness of the whole. With `normalize` it is how loud the loudest moment is, 1 being full. Default 0.9. */
+ gain?: number;
+ /** false leaves the levels as the layers sum to. Default true: the loudest moment is brought to `gain`. */
+ normalize?: boolean;
+}
+
+const isNum = (v: unknown): v is number => typeof v === 'number' && Number.isFinite(v);
+const between = (v: unknown, lo: number, hi: number): boolean => isNum(v) && v >= lo && v <= hi;
+
+function curveProblem(v: unknown, lo: number, hi: number, what: string): string | null {
+ if (isNum(v)) return v >= lo && v <= hi ? null : `${what} must be ${lo}..${hi}`;
+ if (!Array.isArray(v) || v.length < 2) return `${what} must be a number, or two or more [seconds, value]`;
+ let last = -Infinity;
+ for (const p of v) {
+ if (!Array.isArray(p) || p.length !== 2 || !isNum(p[0]) || !isNum(p[1])) return `${what} must be [seconds, value] pairs`;
+ if (p[0] < 0) return `${what}: a time is seconds from the layer's start, none below 0`;
+ if (p[0] <= last) return `${what}: its times must rise`;
+ if (p[1] < lo || p[1] > hi) return `${what}: each value must be ${lo}..${hi}`;
+ last = p[0];
+ }
+ return null;
+}
+
+const LAYER_FIELDS = ['id', 'source', 'freq', 'width', 'color', 'modes', 'seed', 'start', 'length', 'gain', 'attack', 'release', 'decay', 'vibrato', 'tremolo', 'fm', 'filters', 'drive', 'pan', 'repeat', 'mute'];
+
+function checkLayer(l: Record<string, unknown>, where: string, ids: Set<string>, duration: number): void {
+ const fail = (what: string): never => { throw new Error(`${where}: ${what}`); };
+ if (typeof l !== 'object' || l === null || Array.isArray(l)) fail('must be an object');
+ for (const key of Object.keys(l)) if (!LAYER_FIELDS.includes(key)) fail(`unknown field "${key}"; a layer takes ${LAYER_FIELDS.join(', ')}`);
+ if (l.id !== undefined) {
+ if (typeof l.id !== 'string' || !l.id) fail('id must be a name');
+ if (ids.has(l.id as string)) fail(`the id "${String(l.id)}" is already used`);
+ ids.add(l.id as string);
+ }
+ if (!SOUND_SOURCES.includes(l.source as SoundSourceKind)) fail(`source ${JSON.stringify(l.source)} must be one of ${SOUND_SOURCES.join(', ')}`);
+ const tone = l.source !== 'noise' && l.source !== 'modes';
+ if (tone) {
+ if (l.freq === undefined) fail(`a ${String(l.source)} needs freq, its pitch in Hz`);
+ const bad = curveProblem(l.freq, 10, 20000, 'freq');
+ if (bad) fail(bad);
+ } else if (l.freq !== undefined) fail(`freq is a tone's pitch; a ${String(l.source)} has none (give noise its colour with filters, and modes their own Hz)`);
+ if (l.width !== undefined && !(l.source === 'square' && between(l.width, 0.05, 0.95))) fail('width is a square\'s, 0.05..0.95');
+ if (l.color !== undefined && !(l.source === 'noise' && NOISE_COLORS.includes(l.color as never))) fail(`color is a noise's: ${NOISE_COLORS.join(', ')}`);
+ if (l.source === 'modes') {
+ const modes = l.modes;
+ if (!Array.isArray(modes) || modes.length === 0 || modes.length > 64) fail('modes needs `modes`: 1 to 64 of [Hz, loudness, seconds to die away]');
+ for (const m of modes as unknown[]) {
+ if (!Array.isArray(m) || m.length !== 3 || !between(m[0], 10, 20000) || !between(m[1], 0, 4) || !between(m[2], 0.001, MAX_SOUND_SECONDS)) fail('each mode must be [Hz 10..20000, loudness 0..4, seconds above 0]');
+ }
+ } else if (l.modes !== undefined) fail('`modes` goes with source "modes"');
+ if (l.seed !== undefined && !Number.isInteger(l.seed)) fail('seed must be a whole number');
+ if (l.start !== undefined && !between(l.start, 0, duration)) fail(`start must be 0..${duration}, inside the sound`);
+ if (l.length !== undefined && !(isNum(l.length) && l.length > 0)) fail('length must be above 0');
+ if (l.gain !== undefined) { const bad = curveProblem(l.gain, 0, 4, 'gain'); if (bad) fail(bad); }
+ for (const key of ['attack', 'release']) if (l[key] !== undefined && !between(l[key], 0, MAX_SOUND_SECONDS)) fail(`${key} must be seconds, 0 or more`);
+ if (l.decay !== undefined && !(isNum(l.decay) && l.decay > 0)) fail('decay must be seconds above 0');
+ const wobble = (key: string, deepest: number): void => {
+ const w = l[key] as Record<string, unknown> | undefined;
+ if (w === undefined) return;
+ if (typeof w !== 'object' || w === null || !between(w.rate, 0.01, 2000) || !between(w.depth, 0, deepest) || Object.keys(w).length !== 2) fail(`${key} must be { rate: times a second, depth: 0..${deepest} }`);
+ };
+ wobble('vibrato', 24);
+ wobble('tremolo', 1);
+ if (l.fm !== undefined) {
+ const fm = l.fm as Record<string, unknown> | null;
+ if (!tone) fail('fm bends a tone; noise and modes have no pitch to bend');
+ if (typeof fm !== 'object' || fm === null || !between(fm.ratio, 0.01, 64) || Object.keys(fm).some((k) => k !== 'ratio' && k !== 'index')) fail('fm must be { ratio: 0.01..64, index }');
+ const bad = curveProblem(fm!.index, 0, 64, 'fm.index');
+ if (bad) fail(bad);
+ }
+ if (l.filters !== undefined) {
+ if (!Array.isArray(l.filters) || l.filters.length > 8) fail('filters must be a list of up to 8');
+ (l.filters as Record<string, unknown>[]).forEach((f, i) => {
+ if (typeof f !== 'object' || f === null) fail(`filter ${i} must be { type, freq, q? }`);
+ for (const key of Object.keys(f)) if (!['type', 'freq', 'q'].includes(key)) fail(`filter ${i} has unknown "${key}"; it takes type, freq, q`);
+ if (!SOUND_FILTERS.includes(f.type as never)) fail(`filter ${i}: type must be one of ${SOUND_FILTERS.join(', ')}`);
+ const bad = curveProblem(f.freq, 20, 20000, `filter ${i}: freq`);
+ if (bad) fail(bad);
+ if (f.q !== undefined && !between(f.q, 0.1, 40)) fail(`filter ${i}: q must be 0.1..40`);
+ });
+ }
+ if (l.drive !== undefined && !between(l.drive, 0, 1)) fail('drive must be 0..1');
+ if (l.pan !== undefined && !between(l.pan, -1, 1)) fail('pan must be -1..1');
+ if (l.repeat !== undefined) {
+ const r = l.repeat as Record<string, unknown> | null;
+ if (typeof r !== 'object' || r === null) fail('repeat must be { count, every, gain?, pitch?, jitter? }');
+ for (const key of Object.keys(r!)) if (!['count', 'every', 'gain', 'pitch', 'jitter'].includes(key)) fail(`repeat has unknown "${key}"; it takes count, every, gain, pitch, jitter`);
+ if (!Number.isInteger(r!.count) || (r!.count as number) < 2 || (r!.count as number) > 200) fail('repeat.count must be a whole number, 2..200');
+ if (!(isNum(r!.every) && (r!.every as number) > 0)) fail('repeat.every must be seconds above 0');
+ if (r!.gain !== undefined && !between(r!.gain, 0, 2)) fail('repeat.gain must be 0..2');
+ if (r!.pitch !== undefined && !between(r!.pitch, 0.25, 4)) fail('repeat.pitch must be 0.25..4');
+ if (r!.jitter !== undefined && !between(r!.jitter, 0, 1)) fail('repeat.jitter must be 0..1 seconds');
+ }
+ if (l.mute !== undefined && typeof l.mute !== 'boolean') fail('mute must be true or false');
+}
+
+const SOURCE_FIELDS = ['version', 'duration', 'rate', 'loop', 'layers', 'highpass', 'lowpass', 'echo', 'reverb', 'gain', 'normalize'];
+
+/** The source if it is well formed; otherwise throws, naming the layer and the field. */
+export function validateSoundSource(raw: unknown): SoundSource {
+ if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) throw new Error('a sound source must be { duration, layers }');
+ const s = raw as Record<string, unknown>;
+ for (const key of Object.keys(s)) if (!SOURCE_FIELDS.includes(key)) throw new Error(`unknown field "${key}"; a sound has ${SOURCE_FIELDS.join(', ')}`);
+ if (s.version !== undefined && s.version !== SOUND_SOURCE_VERSION) throw new Error(`version ${String(s.version)} is not one this editor reads (${SOUND_SOURCE_VERSION})`);
+ if (!(isNum(s.duration) && s.duration > 0 && s.duration <= MAX_SOUND_SECONDS)) throw new Error(`duration must be seconds, above 0 and up to ${MAX_SOUND_SECONDS}`);
+ if (s.rate !== undefined && !SOUND_RATES.includes(s.rate as never)) throw new Error(`rate must be one of ${SOUND_RATES.join(', ')}`);
+ if (s.loop !== undefined && typeof s.loop !== 'boolean') throw new Error('loop must be true or false');
+ if (s.normalize !== undefined && typeof s.normalize !== 'boolean') throw new Error('normalize must be true or false');
+ if (s.gain !== undefined && !between(s.gain, 0, 1)) throw new Error('gain must be 0..1');
+ for (const key of ['highpass', 'lowpass']) if (s[key] !== undefined && !between(s[key], 20, 20000)) throw new Error(`${key} must be Hz, 20..20000`);
+ if (s.echo !== undefined) {
+ const e = s.echo as Record<string, unknown> | null;
+ if (typeof e !== 'object' || e === null || !between(e.time, 0.01, 2)) throw new Error('echo must be { time: 0.01..2 seconds, feedback?, mix? }');
+ for (const key of Object.keys(e)) if (!['time', 'feedback', 'mix'].includes(key)) throw new Error(`echo has unknown "${key}"; it takes time, feedback, mix`);
+ if (e.feedback !== undefined && !between(e.feedback, 0, 0.95)) throw new Error('echo.feedback must be 0..0.95');
+ if (e.mix !== undefined && !between(e.mix, 0, 1)) throw new Error('echo.mix must be 0..1');
+ }
+ if (s.reverb !== undefined) {
+ const r = s.reverb as Record<string, unknown> | null;
+ if (typeof r !== 'object' || r === null) throw new Error('reverb must be { size?, mix? }');
+ for (const key of Object.keys(r)) if (!['size', 'mix'].includes(key)) throw new Error(`reverb has unknown "${key}"; it takes size, mix`);
+ for (const key of ['size', 'mix']) if (r[key] !== undefined && !between(r[key], 0, 1)) throw new Error(`reverb.${key} must be 0..1`);
+ }
+ if (!Array.isArray(s.layers) || s.layers.length === 0) throw new Error('layers must be a list of one or more');
+ if (s.layers.length > 64) throw new Error('a sound has up to 64 layers');
+ const ids = new Set<string>();
+ (s.layers as Record<string, unknown>[]).forEach((l, i) => checkLayer(l, `layer ${i}${l?.id ? ` "${String(l.id)}"` : ''}`, ids, s.duration as number));
+ return { version: SOUND_SOURCE_VERSION, ...(s as unknown as SoundSource) };
+}
+
+export type SoundEdit =
+ | { op: 'set'; layer: string | number; fields: Record<string, unknown> }
+ | { op: 'insert'; layer: SoundLayer; before?: string | number; after?: string | number }
+ | { op: 'remove'; layer: string | number }
+ | { op: 'settings'; fields: Partial<Omit<SoundSource, 'layers' | 'version'>> };
+
+/** The source with @p edits applied, in order. A `null` field removes it. Refuses without changing anything. */
+export function applySoundEdits(source: SoundSource, edits: readonly SoundEdit[]): SoundSource {
+ const next = JSON.parse(JSON.stringify(source)) as SoundSource & { layers: (SoundLayer & Record<string, unknown>)[] };
+ const find = (ref: string | number): number => {
+ const i = typeof ref === 'number' ? ref : next.layers.findIndex((l) => l.id === ref);
+ if (!Number.isInteger(i) || i < 0 || i >= next.layers.length) {
+ throw new Error(`no layer ${JSON.stringify(ref)}; ${next.layers.length} layers, ids: ${next.layers.map((l) => l.id).filter(Boolean).join(', ') || '(none)'}`);
+ }
+ return i;
+ };
+ for (const edit of edits) {
+ switch (edit?.op) {
+ case 'set': {
+ const layer = next.layers[find(edit.layer)]!;
+ for (const [key, value] of Object.entries(edit.fields ?? {})) {
+ if (value === null) delete layer[key];
+ else layer[key] = value;
+ }
+ break;
+ }
+ case 'insert': {
+ const at = edit.before !== undefined ? find(edit.before) : edit.after !== undefined ? find(edit.after) + 1 : next.layers.length;
+ next.layers.splice(at, 0, edit.layer as never);
+ break;
+ }
+ case 'remove':
+ next.layers.splice(find(edit.layer), 1);
+ break;
+ case 'settings':
+ for (const [key, value] of Object.entries(edit.fields ?? {})) {
+ if (key === 'layers' || key === 'version') throw new Error(`settings does not take ${key}; use set, insert and remove for layers`);
+ if (value === null) delete (next as unknown as Record<string, unknown>)[key];
+ else (next as unknown as Record<string, unknown>)[key] = value;
+ }
+ break;
+ default:
+ throw new Error(`an edit has op ${JSON.stringify((edit as { op?: unknown })?.op)}; there are set, insert, remove, settings`);
+ }
+ }
+ return validateSoundSource(next);
+}
diff --git a/pipeline/tests/sound.test.ts b/pipeline/tests/sound.test.ts
new file mode 100644
index 000000000..617da763a
--- /dev/null
+++ b/pipeline/tests/sound.test.ts
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * A sound from layers, held to what can be counted: how often a tone crosses nothing, where its energy lies, when it
+ * is heard, and that the same source is the same samples.
+ */
+import { describe, expect, it } from 'vitest';
+import { applySoundEdits, validateSoundSource, type SoundLayer, type SoundSource } from '../src/sound/soundSource';
+import { encodeWav, renderSound } from '../src/sound/soundRender';
+import { soundBands, soundPicture } from '../src/sound/soundPicture';
+
+const make = (layers: SoundLayer[], rest: Partial<SoundSource> = {}) => renderSound(validateSoundSource({ duration: 0.5, layers, ...rest }));
+/** Times a second the wave crosses nothing going up, between two shares of the sound: its pitch, for a plain tone. */
+function pitchOf(samples: Float32Array, rate: number, from = 0, to = 1): number {
+ const [a, b] = [Math.floor(samples.length * from), Math.floor(samples.length * to)];
+ let crossings = 0;
+ for (let n = a + 1; n < b; n++) if (samples[n - 1]! <= 0 && samples[n]! > 0) crossings++;
+ return crossings / ((b - a) / rate);
+}
+const level = (samples: Float32Array, from = 0, to = 1): number => {
+ const [a, b] = [Math.floor(samples.length * from), Math.floor(samples.length * to)];
+ let sum = 0;
+ for (let n = a; n < b; n++) sum += samples[n]! * samples[n]!;
+ return Math.sqrt(sum / Math.max(1, b - a));
+};
+
+describe('a sound from layers', () => {
+ it('makes a tone of the pitch asked, in every shape of wave', () => {
+ for (const source of ['sine', 'triangle', 'saw', 'square'] as const) {
+ const s = make([{ source, freq: 440 }]);
+ expect(pitchOf(s.channels[0]!, s.rate, 0.1, 0.9), source).toBeCloseTo(440, -1);
+ }
+ });
+
+ it('glides between the points of a pitch curve by ratio', () => {
+ const s = make([{ source: 'sine', freq: [[0, 200], [0.5, 800]] }]);
+ expect(pitchOf(s.channels[0]!, s.rate, 0, 0.1)).toBeLessThan(260);
+ expect(pitchOf(s.channels[0]!, s.rate, 0.9, 1)).toBeGreaterThan(680);
+ // Halfway it is at 400, the pitch an octave up, not at the 500 a straight line would give.
+ expect(pitchOf(s.channels[0]!, s.rate, 0.45, 0.55)).toBeGreaterThan(360);
+ expect(pitchOf(s.channels[0]!, s.rate, 0.45, 0.55)).toBeLessThan(440);
+ });
+
+ it('is the same samples for the same source, and other noise for another seed', () => {
+ const noise = (seed: number) => make([{ source: 'noise', seed }]).channels[0]!;
+ expect(Buffer.from(noise(3).buffer).equals(Buffer.from(noise(3).buffer))).toBe(true);
+ expect(Buffer.from(noise(3).buffer).equals(Buffer.from(noise(4).buffer))).toBe(false);
+ });
+
+ it('colours noise: white lies high, brown lies low', () => {
+ const share = (color: 'white' | 'pink' | 'brown') => soundBands(make([{ source: 'noise', color }]));
+ expect(share('white').high).toBeGreaterThan(0.8);
+ expect(share('brown').low).toBeGreaterThan(0.5);
+ expect(share('brown').high).toBeLessThan(0.1);
+ expect(share('pink').high).toBeLessThan(share('white').high);
+ });
+
+ it('filters: a low-pass takes the top off, a high-pass the bottom, and a corner can move', () => {
+ const through = (filters: SoundLayer['filters']) => soundBands(make([{ source: 'noise', filters }]));
+ expect(through([{ type: 'lowpass', freq: 300 }]).high).toBeLessThan(0.1);
+ expect(through([{ type: 'highpass', freq: 4000 }]).low).toBeLessThan(0.01);
+ const swept = make([{ source: 'noise', filters: [{ type: 'lowpass', freq: [[0, 8000], [0.5, 100]] }] }], { normalize: false }).channels[0]!;
+ expect(level(swept, 0.8, 1)).toBeLessThan(level(swept, 0, 0.2) * 0.4);
+ });
+
+ it('shapes loudness: a decay dies away, an attack rises, a curve is followed', () => {
+ const dying = make([{ source: 'sine', freq: 300, decay: 0.05 }]).channels[0]!;
+ expect(level(dying, 0.6, 0.8)).toBeLessThan(level(dying, 0, 0.1) * 0.05);
+ const rising = make([{ source: 'sine', freq: 300, attack: 0.4 }]).channels[0]!;
+ expect(level(rising, 0, 0.1)).toBeLessThan(level(rising, 0.7, 0.9) * 0.3);
+ const shaped = make([{ source: 'sine', freq: 300, gain: [[0, 1], [0.25, 0], [0.5, 1]] }]).channels[0]!;
+ expect(level(shaped, 0.45, 0.55)).toBeLessThan(level(shaped, 0, 0.1) * 0.3);
+ });
+
+ it('places a layer in time, and says so in the report', () => {
+ const s = make([{ id: 'late', source: 'sine', freq: 500, start: 0.3, length: 0.1 }]);
+ expect(level(s.channels[0]!, 0, 0.55)).toBe(0);
+ expect(level(s.channels[0]!, 0.62, 0.78)).toBeGreaterThan(0.3);
+ expect(level(s.channels[0]!, 0.85, 1)).toBe(0);
+ expect(s.layers).toEqual([{ id: 'late', source: 'sine', from: 0.3, to: 0.4, peak: 1 }]);
+ });
+
+ it('strikes a layer again, each time quieter and higher if told', () => {
+ const s = make([{ source: 'sine', freq: 400, length: 0.04, repeat: { count: 4, every: 0.1, gain: 0.5, pitch: 1.5 } }], { normalize: false }).channels[0]!;
+ const hit = (k: number): number => level(s, k * 0.2 + 0.01, k * 0.2 + 0.07);
+ expect(hit(1)).toBeCloseTo(hit(0) * 0.5, 1);
+ expect(hit(3)).toBeCloseTo(hit(0) * 0.125, 1);
+ expect(level(s, 0.12, 0.18)).toBe(0);
+ expect(pitchOf(s, 44100, 0.21, 0.27)).toBeGreaterThan(pitchOf(s, 44100, 0.01, 0.07) * 1.3);
+ });
+
+ it('rings the tones of a struck thing, each dying at its own rate', () => {
+ const s = make([{ source: 'modes', modes: [[600, 1, 0.3], [5000, 1, 0.06]] }], { normalize: false });
+ const early = soundBands({ ...s, channels: [s.channels[0]!.subarray(0, 2048)] });
+ const late = soundBands({ ...s, channels: [s.channels[0]!.subarray(18000, 20048)] });
+ expect(early.high).toBeGreaterThan(0.3);
+ expect(late.high).toBeLessThan(0.01);
+ expect(late.mid).toBeGreaterThan(0.95);
+ });
+
+ it('bends a tone with another: more index, more of it above', () => {
+ const top = (index: number): number => soundBands(make([{ source: 'sine', freq: 300, fm: { ratio: 3.5, index } }])).high;
+ expect(top(0)).toBeLessThan(0.01);
+ expect(top(8)).toBeGreaterThan(top(1));
+ expect(top(8)).toBeGreaterThan(0.2);
+ });
+
+ it('is one channel until something is off-centre, and then the side it was sent to is the louder', () => {
+ expect(make([{ source: 'sine', freq: 300 }]).channels).toHaveLength(1);
+ const s = make([{ source: 'sine', freq: 300, pan: -1 }]);
+ expect(s.channels).toHaveLength(2);
+ expect(level(s.channels[0]!)).toBeGreaterThan(0.5);
+ expect(level(s.channels[1]!)).toBeLessThan(1e-6);
+ });
+
+ it('brings its loudest moment to the level asked, or leaves the sum and says when it is too much', () => {
+ expect(make([{ source: 'sine', freq: 300, gain: 0.1 }]).peak).toBeCloseTo(0.9, 5);
+ expect(make([{ source: 'sine', freq: 300, gain: 0.1 }], { gain: 0.5 }).peak).toBeCloseTo(0.5, 5);
+ const raw = make([{ source: 'sine', freq: 300, gain: 0.1 }], { normalize: false });
+ expect(raw.peak).toBeCloseTo(0.1, 2);
+ const over = make([{ source: 'sine', freq: 300 }, { source: 'sine', freq: 300 }], { normalize: false });
+ expect(over.peak).toBe(1);
+ expect(over.notes.join(' ')).toMatch(/sum to 2\.00 times full/);
+ });
+
+ it('adds an echo after the sound, and a room round it', () => {
+ const dry = make([{ source: 'noise', length: 0.05 }], { normalize: false }).channels[0]!;
+ const echoed = make([{ source: 'noise', length: 0.05 }], { normalize: false, echo: { time: 0.2, feedback: 0, mix: 0.5 } }).channels[0]!;
+ expect(level(dry, 0.4, 0.5)).toBe(0);
+ expect(level(echoed, 0.4, 0.5)).toBeGreaterThan(0.1);
+ const roomy = make([{ source: 'noise', length: 0.05 }], { normalize: false, reverb: { size: 0.8, mix: 0.5 } });
+ expect(roomy.channels).toHaveLength(2);
+ expect(level(roomy.channels[0]!, 0.5, 0.9)).toBeGreaterThan(0.001);
+ });
+
+ it('runs the end of a loop into its start', () => {
+ const jump = (loop: boolean): number => {
+ const s = make([{ source: 'noise', color: 'brown', attack: 0, release: 0 }], { loop, normalize: false }).channels[0]!;
+ return Math.abs(s[0]! - s[s.length - 1]!);
+ };
+ // Brown noise wanders, so where it ends is far from where it began; folded over, the last sample and the first
+ // are no further apart than any two neighbours in it, which is 0.07 at the most.
+ expect(jump(true)).toBeLessThan(0.08);
+ const steady = make([{ source: 'sine', freq: 100, attack: 0, release: 0 }], { loop: true, normalize: false }).channels[0]!;
+ expect(level(steady, 0, 0.1)).toBeGreaterThan(0.5);
+ expect(level(steady, 0.9, 1)).toBeGreaterThan(0.5);
+ });
+
+ it('says what is silent, and what is cut short', () => {
+ expect(make([{ source: 'sine', freq: 300, gain: 0 }]).notes.join(' ')).toMatch(/layer 0 makes no sound.*the sound is silent/);
+ expect(make([{ id: 'tail', source: 'sine', freq: 300, start: 0.4, length: 0.5 }]).notes.join(' ')).toMatch(/layer 0 "tail" is cut short/);
+ });
+});
+
+describe('a sound as a file and as a picture', () => {
+ it('writes 16-bit PCM that says what it is', () => {
+ const s = make([{ source: 'sine', freq: 300, pan: 0.5 }], { rate: 22050 });
+ const wav = encodeWav(s);
+ const view = new DataView(wav.buffer);
+ expect(String.fromCharCode(...wav.subarray(0, 4), ...wav.subarray(8, 12))).toBe('RIFFWAVE');
+ expect(view.getUint16(22, true)).toBe(2);
+ expect(view.getUint32(24, true)).toBe(22050);
+ expect(view.getUint16(34, true)).toBe(16);
+ expect(view.getUint32(40, true)).toBe(11025 * 2 * 2);
+ expect(wav.length).toBe(44 + 11025 * 4);
+ });
+
+ it('draws a tone as a line at its pitch, and a higher tone higher', () => {
+ const brightestRow = (freq: number): number => {
+ const p = soundPicture(make([{ source: 'sine', freq }]), 128);
+ let [best, at] = [0, 0];
+ for (let y = 120; y < p.height; y++) {
+ const v = p.rgba[(y * p.width + 64) * 4]! + p.rgba[(y * p.width + 64) * 4 + 1]!;
+ if (v > best) [best, at] = [v, y];
+ }
+ return at;
+ };
+ expect(soundPicture(make([{ source: 'sine', freq: 300 }])).width).toBe(768);
+ expect(brightestRow(2000)).toBeLessThan(brightestRow(200) - 60);
+ });
+});
+
+describe('a sound source', () => {
+ const tone = { source: 'sine', freq: 300 };
+ const bad = (source: unknown) => () => validateSoundSource(source);
+
+ it('says what is wrong with one', () => {
+ expect(bad({ layers: [tone] })).toThrow(/duration must be seconds/);
+ expect(bad({ duration: 1, layers: [] })).toThrow(/layers must be a list of one or more/);
+ expect(bad({ duration: 1, speed: 2, layers: [tone] })).toThrow(/unknown field "speed"; a sound has/);
+ expect(bad({ duration: 1, rate: 8000, layers: [tone] })).toThrow(/rate must be one of 22050, 44100, 48000/);
+ expect(bad({ duration: 1, layers: [{ source: 'sine' }] })).toThrow(/layer 0: a sine needs freq/);
+ expect(bad({ duration: 1, layers: [{ source: 'noise', freq: 300 }] })).toThrow(/freq is a tone's pitch; a noise has none/);
+ expect(bad({ duration: 1, layers: [{ source: 'bell', freq: 300 }] })).toThrow(/source "bell" must be one of sine, triangle, saw, square, noise, modes/);
+ expect(bad({ duration: 1, layers: [{ ...tone, freq: [[0, 300], [0, 400]] }] })).toThrow(/freq: its times must rise/);
+ expect(bad({ duration: 1, layers: [{ ...tone, filters: [{ type: 'lowpass', freq: 5 }] }] })).toThrow(/filter 0: freq must be 20\.\.20000/);
+ expect(bad({ duration: 1, layers: [{ source: 'modes' }] })).toThrow(/modes needs `modes`/);
+ expect(bad({ duration: 1, layers: [{ ...tone, start: 2 }] })).toThrow(/start must be 0\.\.1, inside the sound/);
+ expect(bad({ duration: 1, layers: [{ ...tone, id: 'a' }, { ...tone, id: 'a' }] })).toThrow(/layer 1 "a": the id "a" is already used/);
+ expect(bad({ duration: 1, layers: [{ ...tone, repeat: { count: 1, every: 0.1 } }] })).toThrow(/repeat\.count must be a whole number, 2\.\.200/);
+ expect(bad({ duration: 1, layers: [tone], echo: { time: 5 } })).toThrow(/echo must be/);
+ });
+
+ it('is changed by edits, by id or by place, and refuses a bad one whole', () => {
+ const source = validateSoundSource({ duration: 1, layers: [{ id: 'body', ...tone }, { source: 'noise' }] });
+ const edited = applySoundEdits(source, [
+ { op: 'set', layer: 'body', fields: { freq: 150, decay: 0.2 } },
+ { op: 'set', layer: 1, fields: { color: 'pink' } },
+ { op: 'insert', layer: { id: 'click', source: 'noise', length: 0.01 }, before: 'body' },
+ { op: 'settings', fields: { duration: 2, reverb: { size: 0.3 } } },
+ ]);
+ expect(edited.layers.map((l) => l.id ?? l.color)).toEqual(['click', 'body', 'pink']);
+ expect(edited.layers[1]).toMatchObject({ freq: 150, decay: 0.2 });
+ expect(edited.duration).toBe(2);
+ expect(applySoundEdits(edited, [{ op: 'remove', layer: 'click' }, { op: 'set', layer: 'body', fields: { decay: null } }]).layers[0]).toEqual({ id: 'body', source: 'sine', freq: 150 });
+ expect(() => applySoundEdits(source, [{ op: 'set', layer: 'nobody', fields: {} }])).toThrow(/no layer "nobody"; 2 layers, ids: body/);
+ expect(() => applySoundEdits(source, [{ op: 'set', layer: 'body', fields: { freq: -1 } }])).toThrow(/freq must be 10\.\.20000/);
+ expect(source.layers[0]!.freq).toBe(300);
+ });
+});
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index 8b74d4f66..30e9a3752 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4586,6 +4586,10 @@ export const NOTED = {
{"note": "An AI can draw a surface: a picture that repeats, with what the light does on it."},
"feat(image): image_write makes a surface - a repeating picture is shown laid beside itself and under a light, and its normal map, surface map and material are written with it":
{"note": "An AI can draw a surface: a picture that repeats, with what the light does on it."},
+ "feat(sound): a sound effect is the layers that make it - an .essound of tones, noise and the ringing of struck things, each shaped by an envelope, filters, a bend, a drive and repeats, with an echo and a room over the whole, makes the .wav a game plays; and a picture of the sound, its wave over its pitches, for whoever cannot hear it":
+ {"note": "An AI can make a sound effect."},
+ "feat(sound): sound_write, sound_edit and sound_read - an AI makes a sound effect from layers and is answered with the picture of it, its level and where its energy lies":
+ {"note": "An AI can make a sound effect."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":