A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(music): a piece of music is its instruments, the patterns they play and the order they come in - an .esmusic of steps written as degrees of the key, with chords a bass or a pad follows, makes a .wav that runs into itself and a file for each instrument; and a picture of its notes for whoever cannot hear it
yhhd4f7b2d4516b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5ea28e856..bba307151 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -21,6 +21,7 @@ published separately; it ships inside the editor.- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.+- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. The instruments are made sounds, which suit electronic, retro and ambient music; a piano or strings that pass for real ones are not in reach of them.- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.diff --git a/desktop b/desktopindex b77866777..eb1aeb7a8 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit b77866777ef4e727e20c25eeda86f595cafbd85f+Subproject commit eb1aeb7a82a6ea7221ebd4f6d8a595082fe56103diff --git a/pipeline/src/music/musicPicture.ts b/pipeline/src/music/musicPicture.tsnew file mode 100644index 000000000..c06d3b190--- /dev/null+++ b/pipeline/src/music/musicPicture.ts@@ -0,0 +1,95 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file musicPicture.ts — a piece of music drawn for whoever cannot hear it: every note as a bar, when it sounds+ * across and how high it is up, in its instrument's colour.+ *+ * What a spectrum shows of music is a smear; what is wrong with a piece is nearly always in its notes, and those+ * can be drawn exactly. Two lines in one octave lie on top of each other, a bar with nothing in it is a gap, a+ * bass that never moves is a flat line, and a part that is too full is a block of colour.+ */+import type { RenderedMusic } from './musicRender';++export interface MusicPicture {+ width: number;+ height: number;+ rgba: Uint8Array;+ /** Which colour each instrument is drawn in. */+ colours: Record<string, string>;+ /** The lowest and highest note of the part of the picture where pitch is up, 60 being middle C. */+ low: number;+ high: number;+}++const PALETTE: [string, number, number, number][] = [+ ['blue', 90, 170, 255], ['orange', 255, 160, 70], ['green', 110, 220, 130], ['pink', 255, 120, 190],+ ['yellow', 240, 220, 90], ['cyan', 90, 225, 225], ['red', 255, 95, 95], ['violet', 180, 140, 255],+];++/** The loudness over time on top, the notes by pitch under it, and a row for each instrument that has no pitch. */+export function musicPicture(music: RenderedMusic): MusicPicture {+ const names = Object.keys(music.instruments);+ const bars = Math.max(1, Math.round(music.seconds / music.barSeconds));+ const width = Math.max(768, Math.min(1536, bars * 64));+ const pitched = music.events.filter((e) => e.midi !== null);+ const [low, high] = pitched.length ? [Math.min(...pitched.map((e) => e.midi!)) - 2, Math.max(...pitched.map((e) => e.midi!)) + 2] : [58, 62];+ const row = Math.max(3, Math.min(8, Math.floor(300 / (high - low + 1))));+ const [waveH, rollH] = [48, (high - low + 1) * row];+ const drums = names.filter((n) => music.events.some((e) => e.instrument === n && e.midi === null));+ const [drumRow, gap] = [12, 6];+ const height = waveH + rollH + (drums.length ? gap + drums.length * drumRow : 0);+ const rgba = new Uint8Array(width * height * 4);+ const put = (x: number, y: number, r: number, g: number, b: number, a = 1): void => {+ if (x < 0 || x >= width || y < 0 || y >= height) return;+ const o = (y * width + x) * 4;+ rgba[o] = Math.round(rgba[o]! * (1 - a) + r * a);+ rgba[o + 1] = Math.round(rgba[o + 1]! * (1 - a) + g * a);+ rgba[o + 2] = Math.round(rgba[o + 2]! * (1 - a) + b * a);+ rgba[o + 3] = 255;+ };+ for (let y = 0; y < height; y++) for (let x = 0; x < width; x++) put(x, y, y < waveH ? 22 : 10, y < waveH ? 24 : 10, y < waveH ? 30 : 16);++ // Loudness: each column from the quietest to the loudest sample it covers.+ const frames = music.channels[0]!.length;+ for (let x = 0; x < width; x++) {+ const [from, to] = [Math.floor((x / width) * frames), Math.floor(((x + 1) / width) * frames)];+ let top = 0;+ for (let n = from; n < to; n++) top = Math.max(top, Math.abs(music.channels[0]![n]!), Math.abs(music.channels[1]![n]!));+ for (let y = Math.round((waveH - 2) * (1 - top)); y < waveH - 1; y++) put(x, y, 120, 214, 255);+ }++ // The rows of the white keys' C, and the black keys darker, as on a keyboard laid on its side.+ for (let midi = low; midi <= high; midi++) {+ const y = waveH + (high - midi) * row;+ const black = [1, 3, 6, 8, 10].includes(((midi % 12) + 12) % 12);+ if (black) for (let dy = 0; dy < row; dy++) for (let x = 0; x < width; x++) put(x, y + dy, 0, 0, 0, 0.35);+ if (midi % 12 === 0) for (let x = 0; x < width; x++) put(x, y + row - 1, 255, 255, 255, 0.22);+ }+ // Every bar, and brighter where a part of the song begins.+ const starts = new Set(music.parts.map((p) => Math.round(p.at / music.barSeconds)));+ for (let b = 0; b <= bars; b++) {+ const x = Math.min(width - 1, Math.round((b * music.barSeconds / music.seconds) * width));+ for (let y = 0; y < height; y++) put(x, y, 255, 255, 255, starts.has(b) ? 0.5 : 0.14);+ }++ const colours: Record<string, string> = {};+ names.forEach((name, i) => { colours[name] = PALETTE[i % PALETTE.length]![0]; });+ for (const e of music.events) {+ const [, r, g, b] = PALETTE[names.indexOf(e.instrument) % PALETTE.length]!;+ const x0 = Math.round((e.at / music.seconds) * width);+ const x1 = Math.max(x0 + 2, Math.round(((e.at + e.seconds) / music.seconds) * width) - 1);+ const strength = Math.max(0.45, Math.min(1, 0.75 * e.loud));+ const [top, tall] = e.midi !== null+ ? [waveH + (high - e.midi) * row, row - (row > 3 ? 1 : 0)]+ : [waveH + rollH + gap + drums.indexOf(e.instrument) * drumRow + 2, drumRow - 4];+ // A drum is drawn as short as it is, not as long as its step.+ const end = e.midi !== null ? x1 : x0 + 3;+ for (let y = top; y < top + tall; y++) for (let x = x0; x < end && x < width; x++) put(x, y, r, g, b, strength);+ // The front of a note is bright, so two of one pitch in a row are seen as two.+ for (let y = top; y < top + tall; y++) put(x0, y, 255, 255, 255, 0.55);+ // A note outside the key has a white line over it.+ if (e.outside) for (let x = x0; x < end && x < width; x++) put(x, top, 255, 255, 255, 0.9);+ }+ for (let x = 0; x < width; x++) put(x, waveH, 90, 96, 110);+ return { width, height, rgba, colours, low, high };+}diff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.tsnew file mode 100644index 000000000..87955b683--- /dev/null+++ b/pipeline/src/music/musicRender.ts@@ -0,0 +1,310 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file musicRender.ts — a music source worked out into the notes it plays, and the notes into samples.+ *+ * Two steps, and the first is worth having on its own: `compileMusic` turns patterns of steps into notes with a+ * time, a length and a pitch, which is what a picture of the piece is drawn from and what can be checked without+ * hearing anything. `renderMusic` then strikes each instrument's layers once for every note (soundRender.ts) and+ * sums them, each instrument on its own bus so the buses can be written out apart.+ *+ * A piece that loops is not faded into itself: what rings past its end is added to its start, which is what+ * would be heard had it been playing round already.+ */+import { fft } from '../sound/soundPicture';+import { placeResponse, renderSound } from '../sound/soundRender';+import type { SoundLayer } from '../sound/soundSource';+import { musicSeconds, parseTrack, SCALES, tonicOf, type MusicInstrument, type MusicSource } from './musicSource';++export interface MusicEvent {+ instrument: string;+ /** Seconds from the start of the piece, and how long it is held. */+ at: number;+ seconds: number;+ /** The note, 60 being middle C; null for an instrument struck at its own pitch. */+ midi: number | null;+ loud: number;+ /** Which part of the song it is in. */+ part: number;+ /** Written with # or b: outside the key. */+ outside: boolean;+}++export interface CompiledMusic {+ seconds: number;+ barSeconds: number;+ events: MusicEvent[];+ parts: { pattern: string; at: number; bars: number }[];+ notes: string[];+}++export interface RenderedMusic extends CompiledMusic {+ rate: number;+ /** Left, right. */+ channels: Float32Array[];+ peak: number;+ rms: number;+ /** Each instrument alone, when the source asks for stems: they add up to `channels`. */+ stems: Record<string, Float32Array[]> | null;+ /** What each instrument played, and how loud it is beside the whole, in decibels. */+ instruments: Record<string, { notes: number; low?: string; high?: string; levelDb: number }>;+}++const NOTE_NAMES = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B'];+export const noteName = (midi: number): string => `${NOTE_NAMES[((midi % 12) + 12) % 12]}${Math.floor(midi / 12) - 1}`;++function randomOf(seed: number): () => number {+ let s = (seed * 2654435761 + 0x9e3779b9) >>> 0;+ return () => {+ s = (s + 0x6d2b79f5) >>> 0;+ let t = s;+ t = Math.imul(t ^ (t >>> 15), t | 1);+ t ^= t + Math.imul(t ^ (t >>> 7), t | 61);+ return ((t ^ (t >>> 14)) >>> 0) / 2147483648 - 1;+ };+}++/** The notes a validated source plays, in the order they begin. */+export function compileMusic(source: MusicSource): CompiledMusic {+ const beats = source.beats ?? 4;+ const barSeconds = (beats * 60) / source.tempo;+ const scale = SCALES[source.scale ?? 'minor'];+ const tonic = tonicOf(source.key);+ const human = source.humanize ?? 0.15;+ const random = randomOf(source.seed ?? 1);+ const events: MusicEvent[] = [];+ const parts: CompiledMusic['parts'] = [];+ const notes: string[] = [];+ const said = new Set<string>();+ const say = (text: string): void => { if (!said.has(text)) { said.add(text); notes.push(text); } };+ const played = new Set<string>();+ const used = new Set<string>();+ let at = 0;++ source.song.forEach((part, p) => {+ const name = typeof part === 'string' ? part : part.pattern;+ const pattern = source.patterns[name]!;+ used.add(name);+ parts.push({ pattern: name, at, bars: pattern.bars });+ const div = pattern.div ?? 4;+ const stepsToBar = beats * div;+ const step = barSeconds / stepsToBar;+ const before = events.length;+ for (const [track, line] of Object.entries(pattern.tracks)) {+ if (typeof part !== 'string' && (part.without?.includes(track) || (part.only && !part.only.includes(track)))) continue;+ const instrument = source.instruments[track]!;+ if (instrument.follow && !pattern.chords) say(`"${track}" follows the chords, and pattern "${name}" has none: it plays every bar on chord 1`);+ const steps = parseTrack(line, pattern.bars, stepsToBar, `pattern "${name}", track "${track}"`);+ for (let i = 0; i < steps.length; i++) {+ const s = steps[i]!;+ if (s.kind === 'hold' && (i === 0 || steps[i - 1]!.kind === 'rest')) say(`pattern "${name}", track "${track}": a "-" at step ${(i % stepsToBar) + 1} of bar ${Math.floor(i / stepsToBar) + 1} has no note before it to hold, and is silent`);+ if (s.kind !== 'hit' && s.kind !== 'notes') continue;+ let held = 1;+ while (i + held < steps.length && steps[i + held]!.kind === 'hold') held++;+ // Every second step late, which is what makes a line swing.+ const late = div % 2 === 0 && i % 2 === 1 ? (source.swing ?? 0) * step : 0;+ const when = Math.max(0, at + i * step + late + human * 0.012 * random());+ const loud = s.loud * (1 + human * 0.25 * random());+ played.add(track);+ if (s.kind === 'hit') {+ events.push({ instrument: track, at: when, seconds: held * step, midi: null, loud, part: p, outside: false });+ continue;+ }+ const chord = instrument.follow && pattern.chords ? pattern.chords[Math.floor(i / stepsToBar) % pattern.chords.length]! : 1;+ // A root on 5, 6 or 7 is taken below the tonic, so the chords lie round it and a bass does not climb away.+ const drop = chord >= 5 ? -7 : 0;+ for (const n of s.notes) {+ const index = n.degree - 1 + chord - 1 + drop + 7;+ const midi = 60 + tonic + scale[index % 7]! + 12 * (Math.floor(index / 7) - 1 + n.octave + (instrument.octave ?? 0)) + n.alter;+ events.push({ instrument: track, at: when, seconds: held * step, midi, loud, part: p, outside: n.alter !== 0 });+ }+ }+ }+ if (events.length === before) notes.push(`part ${p + 1} of the song ("${name}") plays nothing`);+ at += pattern.bars * barSeconds;+ });++ for (const name of Object.keys(source.instruments)) if (!played.has(name)) notes.push(`instrument "${name}" is never played`);+ for (const name of Object.keys(source.patterns)) if (!used.has(name)) notes.push(`pattern "${name}" is not in the song`);++ // What can be said of the notes without hearing them.+ const pitched = new Map<string, MusicEvent[]>();+ for (const e of events) if (e.midi !== null) pitched.set(e.instrument, [...(pitched.get(e.instrument) ?? []), e]);+ const middle = new Map<string, number>();+ for (const [name, list] of pitched) {+ const sorted = list.map((e) => e.midi!).sort((a, b) => a - b);+ middle.set(name, sorted[Math.floor(sorted.length / 2)]!);+ if (sorted[0]! < 24) notes.push(`"${name}" goes down to ${noteName(sorted[0]!)}, below where a pitch can be told: raise its octave`);+ if (sorted.at(-1)! > 100) notes.push(`"${name}" goes up to ${noteName(sorted.at(-1)!)}, which is a whistle: lower its octave`);+ const outside = list.filter((e) => e.outside).length;+ if (outside > 0) notes.push(`"${name}" plays ${outside} note${outside === 1 ? '' : 's'} outside the key`);+ }+ const names = [...pitched.keys()];+ for (let a = 0; a < names.length; a++) for (let b = a + 1; b < names.length; b++) {+ const [x, y] = [pitched.get(names[a]!)!, pitched.get(names[b]!)!];+ if (Math.abs(middle.get(names[a]!)! - middle.get(names[b]!)!) > 4) continue;+ // How much of the time the less busy of the two sounds, the other is sounding too.+ const length = (list: MusicEvent[]): number => list.reduce((sum, e) => sum + e.seconds, 0);+ let both = 0;+ for (const e of x) for (const f of y) both += Math.max(0, Math.min(e.at + e.seconds, f.at + f.seconds) - Math.max(e.at, f.at));+ if (both > 0.5 * Math.min(length(x), length(y))) notes.push(`"${names[a]}" and "${names[b]}" sound together in the same octave, round ${noteName(middle.get(names[a]!)!)}, and will blur into one: move one of them an octave`);+ }++ events.sort((a, b) => a.at - b.at);+ return { seconds: musicSeconds(source), barSeconds, events, parts, notes };+}++/** How far past its own start a layer can still be sounding, when a held note lasts @p held seconds. */+function reach(layer: SoundLayer, held: number): number {+ const again = layer.scatter ? layer.scatter.over : layer.repeat ? (layer.repeat.count - 1) * layer.repeat.every : 0;+ const own = layer.source === 'group' ? Math.max(0, ...layer.layers!.map((l) => reach(l, held))) : layer.length ?? held;+ return (layer.start ?? 0) + again + own;+}++/** A layer with no length of its own lasts as long as the note, and dies over the instrument's release. */+function forNote(layer: SoundLayer, held: number, release: number): SoundLayer {+ if (layer.source === 'group') return { ...layer, layers: layer.layers!.map((l) => forNote(l, held, release)) };+ if (layer.length !== undefined) return layer;+ return { ...layer, length: Math.max(0.01, held + release - (layer.start ?? 0)), release: layer.release ?? release };+}++const TAIL_SECONDS = 4;++/**+ * A long sound in a place: the same sum a single transform of the whole would give, taken a stretch at a time, each+ * stretch answered by the place and laid where it belongs. A piece of music is a hundred times as long as the room's+ * answer, and one transform of all of it costs several times what these do.+ */+function inPlace(response: Float32Array): (signal: Float32Array) => Float32Array {+ let size = 1;+ while (size < response.length * 2) size <<= 1;+ const block = size - response.length;+ const [hr, hi] = [new Float64Array(size), new Float64Array(size)];+ hr.set(response);+ fft(hr, hi);+ const [xr, xi] = [new Float64Array(size), new Float64Array(size)];+ return (signal) => {+ const out = new Float32Array(signal.length);+ for (let from = 0; from < signal.length; from += block) {+ xr.fill(0); xi.fill(0);+ let any = false;+ for (let n = 0; n < block && from + n < signal.length; n++) { const v = signal[from + n]!; xr[n] = v; if (v !== 0) any = true; }+ if (!any) continue;+ fft(xr, xi);+ for (let k = 0; k < size; k++) {+ const re = xr[k]! * hr[k]! - xi[k]! * hi[k]!, im = xr[k]! * hi[k]! + xi[k]! * hr[k]!;+ // Back the same way: the transform of the conjugate, shared out.+ xr[k] = re; xi[k] = -im;+ }+ fft(xr, xi);+ for (let n = 0; n < size && from + n < signal.length; n++) out[from + n]! += xr[n]! / size;+ }+ return out;+ };+}++/** The samples a validated source makes. */+export function renderMusic(source: MusicSource): RenderedMusic {+ const compiled = compileMusic(source);+ const rate = source.rate ?? 44100;+ const count = Math.max(1, Math.round(compiled.seconds * rate));+ const work = count + Math.round(TAIL_SECONDS * rate);+ const names = Object.keys(source.instruments);+ const buses = new Map(names.map((name) => [name, [new Float32Array(work), new Float32Array(work)]] as const));++ const cache = new Map<string, Float32Array[]>();+ const noteOf = (name: string, instrument: MusicInstrument, e: MusicEvent, turn: number): Float32Array[] => {+ // The same note is made a few times over and no more: enough that a hat is not one sample struck again.+ const key = `${name}|${e.midi}|${Math.round(e.seconds * 1000)}|${turn % 3}`;+ let made = cache.get(key);+ if (!made) {+ const release = instrument.release ?? 0.08;+ const layers = instrument.layers.map((l) => forNote(l, e.seconds, release));+ const duration = Math.min(8, Math.max(0.02, ...layers.map((l) => reach(l, e.seconds + release))));+ made = renderSound({ duration, rate, layers, normalize: false }, 0, { pitch: e.midi === null ? 1 : Math.pow(2, (e.midi - 69) / 12), seed: (turn % 3) * 7919, raw: true }).channels;+ cache.set(key, made);+ }+ return made;+ };++ const turns = new Map<string, number>();+ for (const e of compiled.events) {+ const instrument = source.instruments[e.instrument]!;+ const turn = turns.get(e.instrument) ?? 0;+ turns.set(e.instrument, turn + 1);+ const note = noteOf(e.instrument, instrument, e, turn);+ const [l, r] = [note[0]!, note[1] ?? note[0]!];+ const pan = instrument.pan ?? 0;+ // Equal power, and full in the middle.+ const loud = e.loud * (instrument.gain ?? 1) * Math.SQRT2;+ const [gl, gr] = [Math.cos(((pan + 1) * Math.PI) / 4) * loud, Math.sin(((pan + 1) * Math.PI) / 4) * loud];+ const [left, right] = buses.get(e.instrument)!;+ const from = Math.round(e.at * rate);+ for (let n = 0; n < l.length && from + n < work; n++) {+ left[from + n]! += l[n]! * gl;+ right[from + n]! += r[n]! * gr;+ }+ }++ // The room. One for everything; or, when the instruments are to be written apart, one each, so they still add up.+ const room = [new Float32Array(work), new Float32Array(work)];+ if (source.reverb) {+ const [size, mix] = [source.reverb.size ?? 0.4, (source.reverb.mix ?? 0.2) * 3];+ const answers = [0, 1].map((side) => inPlace(placeResponse(rate, size, 'room', side)));+ const sendOf = (name: string): number => source.instruments[name]!.reverb ?? 0.3;+ if (source.stems) {+ for (const name of names) {+ if (sendOf(name) === 0) continue;+ buses.get(name)!.forEach((channel, c) => {+ const wet = answers[c]!(channel);+ for (let n = 0; n < work; n++) channel[n]! += wet[n]! * mix * sendOf(name);+ });+ }+ } else {+ for (const c of [0, 1]) {+ const send = new Float32Array(work);+ for (const name of names) { const [bus, share] = [buses.get(name)![c]!, sendOf(name)]; if (share > 0) for (let n = 0; n < work; n++) send[n]! += bus[n]! * share; }+ const wet = answers[c]!(send);+ for (let n = 0; n < work; n++) room[c]![n] = wet[n]! * mix;+ }+ }+ }++ const loop = source.loop !== false;+ const finish = (channel: Float32Array): Float32Array => {+ const out = channel.slice(0, count);+ if (loop) for (let n = count; n < work; n++) out[(n - count) % count]! += channel[n]!;+ else { const fade = Math.min(count, Math.round(0.02 * rate)); for (let n = 0; n < fade; n++) out[count - 1 - n]! *= n / fade; }+ return out;+ };+ const stems: Record<string, Float32Array[]> = {};+ for (const name of names) stems[name] = buses.get(name)!.map(finish);+ const channels = [0, 1].map((c) => {+ const sum = finish(room[c]!);+ for (const name of names) { const stem = stems[name]![c]!; for (let n = 0; n < count; n++) sum[n]! += stem[n]!; }+ return sum;+ });++ let peak = 0;+ for (const channel of channels) for (const v of channel) if (Math.abs(v) > peak) peak = Math.abs(v);+ const notes = [...compiled.notes];+ const target = source.gain ?? 0.9;+ const scale = peak > 1e-6 ? target / peak : 1;+ if (peak <= 1e-6) notes.push('the piece is silent: nothing in it reaches a level that can be heard');+ for (const list of [channels, ...Object.values(stems)]) for (const channel of list) for (let n = 0; n < count; n++) channel[n]! *= scale;+ const rmsOf = (list: Float32Array[]): number => { let s = 0; for (const channel of list) for (const v of channel) s += v * v; return Math.sqrt(s / (count * list.length)); };+ const rms = rmsOf(channels);++ const instruments: RenderedMusic['instruments'] = {};+ for (const name of names) {+ const mine = compiled.events.filter((e) => e.instrument === name);+ const pitches = mine.filter((e) => e.midi !== null).map((e) => e.midi!);+ const level = rmsOf(stems[name]!);+ instruments[name] = {+ notes: mine.length,+ ...(pitches.length ? { low: noteName(Math.min(...pitches)), high: noteName(Math.max(...pitches)) } : {}),+ levelDb: level > 0 && rms > 0 ? Math.round(20 * Math.log10(level / rms) * 10) / 10 : -120,+ };+ }+ return { ...compiled, notes, rate, channels, peak: peak > 1e-6 ? target : 0, rms, stems: source.stems ? stems : null, instruments };+}diff --git a/pipeline/src/music/musicSource.ts b/pipeline/src/music/musicSource.tsnew file mode 100644index 000000000..0f278093b--- /dev/null+++ b/pipeline/src/music/musicSource.ts@@ -0,0 +1,281 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file musicSource.ts — a piece of music as what it is made of: instruments, the patterns they play, and the order+ * the patterns come in.+ *+ * An instrument is the layers of a sound (soundSource.ts), written as it sounds at A above middle C; a note+ * strikes it at another pitch for as long as the note lasts. A pattern is a few bars, each track a line of+ * steps. A step is a degree of the key and not a pitch in Hz, so a line written without accidentals cannot+ * leave the key, and a track that follows the bar's chord cannot leave the chord's scale either. What is saved+ * is this description; the `.wav` a game plays is made from it.+ */+import { validateSoundSource, type SoundLayer } from '../sound/soundSource';++export const MUSIC_SOURCE_EXT = '.esmusic';+export const MUSIC_SOURCE_VERSION = 1;+export const MAX_MUSIC_SECONDS = 240;++/** Semitones above the tonic of each of a scale's seven degrees. */+export const SCALES = {+ major: [0, 2, 4, 5, 7, 9, 11],+ minor: [0, 2, 3, 5, 7, 8, 10],+ dorian: [0, 2, 3, 5, 7, 9, 10],+ phrygian: [0, 1, 3, 5, 7, 8, 10],+ lydian: [0, 2, 4, 6, 7, 9, 11],+ mixolydian: [0, 2, 4, 5, 7, 9, 10],+ harmonic: [0, 2, 3, 5, 7, 8, 11],+} as const;+export type ScaleName = keyof typeof SCALES;++const TONICS: Record<string, number> = { C: 0, 'C#': 1, Db: 1, D: 2, 'D#': 3, Eb: 3, E: 4, F: 5, 'F#': 6, Gb: 6, G: 7, 'G#': 8, Ab: 8, A: 9, 'A#': 10, Bb: 10, B: 11 };++export interface MusicInstrument {+ /** The layers of its sound at A above middle C (440 Hz): a sine at 440 is its fundamental, one at 880 the octave over it. A layer with no `length` lasts as long as the note; one with a `length` (the knock at the front of a pluck, a drum) keeps its own. */+ layers: readonly SoundLayer[];+ /** Loudness against the other instruments. Default 1. */+ gain?: number;+ /** -1 is left, 1 is right. Default 0. */+ pan?: number;+ /** Octaves up or down from where its notes are written: a bass is -2. Default 0. */+ octave?: number;+ /** Its degrees count from the root of the bar's chord: 1 is the root, 3 the third, 5 the fifth, whatever the chord. A bass or a pad that follows is written once and moves with the chords. */+ follow?: boolean;+ /** Seconds a held note takes to die after it is let go. Default 0.08. */+ release?: number;+ /** How much of it goes to the room, 0..1. Default 0.3; a bass or a kick wants 0. */+ reverb?: number;+}++export interface MusicPattern {+ bars: number;+ /** Steps to a beat. Default 4, which is sixteenths; 3 is triplets; 2 is eighths. */+ div?: number;+ /** The chord of each bar, as the degree its root is on: [1, 6, 3, 7]. Fewer than the bars come round again. Only instruments that `follow` hear them. */+ chords?: readonly number[];+ /**+ * A line of steps for each instrument that plays, separated by spaces, bars by `|`. A step is: `.` silence; `-` the+ * note before, still held; a degree 1..7 with `'` for each octave up and `,` for each down, `#` or `b` before it to+ * leave the key by a semitone; `[1 3 5]` several at once; `x` the instrument at its own pitch, for a drum. `!` after+ * a step makes it louder and `?` softer.+ */+ tracks: Readonly<Record<string, string>>;+}++export type SongPart = string | { pattern: string; without?: readonly string[]; only?: readonly string[] };++export interface MusicSource {+ version?: number;+ /** Beats a minute. */+ tempo: number;+ /** Beats to a bar. Default 4. */+ beats?: number;+ /** The tonic: C, F#, Bb. */+ key: string;+ /** Default minor. */+ scale?: ScaleName;+ /** Every second step comes late by this share of a step, 0..0.6. 0.33 is a shuffle. */+ swing?: number;+ /** How unlike a machine: timing and loudness off by a little at random, 0..1. Default 0.15. */+ humanize?: number;+ seed?: number;+ instruments: Readonly<Record<string, MusicInstrument>>;+ patterns: Readonly<Record<string, MusicPattern>>;+ /** The patterns in the order they come, each whole, or with some instruments left out or only some kept. */+ song: readonly SongPart[];+ /** What rings past the end comes in at the start, so the piece runs into itself. Default true. */+ loop?: boolean;+ /** Also a file for each instrument, which add up to the whole: a game brings them in and out. */+ stems?: boolean;+ /** The room it is all in: `size` 0..1, `mix` 0..1. */+ reverb?: { size?: number; mix?: number };+ /** How loud the loudest moment is, 1 being full. Default 0.9. */+ gain?: number;+ rate?: number;+}++/** One sounding in a step: a degree, or the instrument's own pitch. */+export interface StepNote { degree: number; alter: number; octave: number }+export type Step =+ | { kind: 'rest' }+ | { kind: 'hold' }+ | { kind: 'hit'; loud: number }+ | { kind: 'notes'; loud: number; notes: StepNote[] };++const isNum = (v: unknown): v is number => typeof v === 'number' && Number.isFinite(v);+const between = (v: unknown, lo: number, hi: number): boolean => isNum(v) && v >= lo && v <= hi;+const isObj = (v: unknown): v is Record<string, unknown> => typeof v === 'object' && v !== null && !Array.isArray(v);+const NAME = /^[A-Za-z][A-Za-z0-9_-]{0,31}$/;++export const tonicOf = (key: string): number => TONICS[key]!;++const loudOf = (mark: string): number => (mark === '!' ? 1.3 : mark === '?' ? 0.6 : 1);++function noteOf(text: string): StepNote | null {+ const m = /^([#b]?)([1-7])([',]*)$/.exec(text);+ if (!m) return null;+ let octave = 0;+ for (const c of m[3]!) octave += c === "'" ? 1 : -1;+ return { degree: Number(m[2]), alter: m[1] === '#' ? 1 : m[1] === 'b' ? -1 : 0, octave };+}++/** A track's line as steps, bar by bar. Throws saying which bar is wrong and why. */+export function parseTrack(text: string, bars: number, stepsToBar: number, where: string): Step[] {+ const out: Step[] = [];+ const written = text.split('|').map((b) => b.trim());+ if (written.at(-1) === '') written.pop();+ if (written.length !== bars) throw new Error(`${where} has ${written.length} bar${written.length === 1 ? '' : 's'} between its | marks; the pattern is ${bars}`);+ written.forEach((bar, b) => {+ // A chord's notes are kept together through the split on spaces.+ const tokens = bar.replace(/\[([^\]]*)\]/g, (_, inner: string) => `[${inner.trim().split(/\s+/).join('+')}]`).split(/\s+/).filter((t) => t !== '');+ if (tokens.length !== stepsToBar) throw new Error(`${where}, bar ${b + 1} has ${tokens.length} steps; a bar here is ${stepsToBar}. Write "." for a silent step and "-" for one that holds the note before`);+ for (const token of tokens) {+ const mark = token.endsWith('!') || token.endsWith('?') ? token.slice(-1) : '';+ const body = mark ? token.slice(0, -1) : token;+ if (body === '.') out.push({ kind: 'rest' });+ else if (body === '-') out.push({ kind: 'hold' });+ else if (body === 'x') out.push({ kind: 'hit', loud: loudOf(mark) });+ else {+ const parts = body.startsWith('[') && body.endsWith(']') ? body.slice(1, -1).split('+') : [body];+ const notes = parts.map(noteOf);+ if (notes.length === 0 || notes.length > 6 || notes.some((n) => n === null)) {+ throw new Error(`${where}, bar ${b + 1}: "${token.replace(/\+/g, ' ')}" is not a step. A step is . (silence), - (hold), x (a drum), a degree 1..7 with ' or , for octaves and # or b before it, or [1 3 5] for up to six at once`);+ }+ out.push({ kind: 'notes', loud: loudOf(mark), notes: notes as StepNote[] });+ }+ }+ });+ return out;+}++const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'stems', 'reverb', 'gain', 'rate'];+const INSTRUMENT_FIELDS = ['layers', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb'];+const PATTERN_FIELDS = ['bars', 'div', 'chords', 'tracks'];++/** Seconds the song lasts, before anything that rings past its end. */+export function musicSeconds(source: MusicSource): number {+ const bar = ((source.beats ?? 4) * 60) / source.tempo;+ return source.song.reduce((sum, part) => sum + source.patterns[typeof part === 'string' ? part : part.pattern]!.bars * bar, 0);+}++/** The source if it is well formed; otherwise throws, naming the instrument, the pattern, the track and the bar. */+export function validateMusicSource(raw: unknown): MusicSource {+ if (!isObj(raw)) throw new Error('a music source must be { tempo, key, instruments, patterns, song }');+ const s = raw;+ for (const key of Object.keys(s)) if (!SOURCE_FIELDS.includes(key)) throw new Error(`unknown field "${key}"; music has ${SOURCE_FIELDS.join(', ')}`);+ if (s.version !== undefined && s.version !== MUSIC_SOURCE_VERSION) throw new Error(`version ${String(s.version)} is not one this editor reads (${MUSIC_SOURCE_VERSION})`);+ if (!between(s.tempo, 30, 300)) throw new Error('tempo must be beats a minute, 30..300');+ if (s.beats !== undefined && !(Number.isInteger(s.beats) && between(s.beats, 2, 12))) throw new Error('beats must be a whole number to the bar, 2..12');+ if (typeof s.key !== 'string' || TONICS[s.key] === undefined) throw new Error(`key must be a tonic: ${Object.keys(TONICS).join(', ')}`);+ if (s.scale !== undefined && !(typeof s.scale === 'string' && s.scale in SCALES)) throw new Error(`scale must be one of ${Object.keys(SCALES).join(', ')}`);+ if (s.swing !== undefined && !between(s.swing, 0, 0.6)) throw new Error('swing must be 0..0.6');+ if (s.humanize !== undefined && !between(s.humanize, 0, 1)) throw new Error('humanize must be 0..1');+ if (s.seed !== undefined && !Number.isInteger(s.seed)) throw new Error('seed must be a whole number');+ if (s.gain !== undefined && !between(s.gain, 0, 1)) throw new Error('gain must be 0..1');+ if (s.rate !== undefined && ![22050, 44100, 48000].includes(s.rate as number)) throw new Error('rate must be one of 22050, 44100, 48000');+ for (const key of ['loop', 'stems']) if (s[key] !== undefined && typeof s[key] !== 'boolean') throw new Error(`${key} must be true or false`);+ if (s.reverb !== undefined) {+ if (!isObj(s.reverb)) throw new Error('reverb must be { size?, mix? }');+ for (const key of Object.keys(s.reverb)) if (!['size', 'mix'].includes(key)) throw new Error(`reverb has unknown "${key}"; it takes size, mix`);+ if (s.reverb.size !== undefined && !between(s.reverb.size, 0, 1)) throw new Error('reverb.size must be 0..1');+ if (s.reverb.mix !== undefined && !between(s.reverb.mix, 0, 1)) throw new Error('reverb.mix must be 0..1');+ }++ if (!isObj(s.instruments) || Object.keys(s.instruments).length === 0) throw new Error('instruments must be { name: { layers: [...] } }, at least one');+ if (Object.keys(s.instruments).length > 16) throw new Error('at most 16 instruments');+ for (const [name, value] of Object.entries(s.instruments)) {+ const where = `instrument "${name}"`;+ if (!NAME.test(name)) throw new Error(`${where}: a name is a letter, then letters, digits, _ or -`);+ if (!isObj(value)) throw new Error(`${where} must be { layers: [...] }`);+ for (const key of Object.keys(value)) if (!INSTRUMENT_FIELDS.includes(key)) throw new Error(`${where} has unknown "${key}"; an instrument has ${INSTRUMENT_FIELDS.join(', ')}`);+ try {+ // Its layers are a sound's layers, and are wrong in the same ways.+ validateSoundSource({ duration: 8, layers: value.layers });+ } catch (e) {+ throw new Error(`${where}: ${(e as Error).message}`);+ }+ if (value.gain !== undefined && !between(value.gain, 0, 4)) throw new Error(`${where}: gain must be 0..4`);+ if (value.pan !== undefined && !between(value.pan, -1, 1)) throw new Error(`${where}: pan must be -1..1`);+ if (value.octave !== undefined && !(Number.isInteger(value.octave) && between(value.octave, -4, 4))) throw new Error(`${where}: octave must be a whole number, -4..4`);+ if (value.follow !== undefined && typeof value.follow !== 'boolean') throw new Error(`${where}: follow must be true or false`);+ if (value.release !== undefined && !between(value.release, 0, 4)) throw new Error(`${where}: release must be seconds, 0..4`);+ if (value.reverb !== undefined && !between(value.reverb, 0, 1)) throw new Error(`${where}: reverb must be 0..1`);+ }++ const beats = (s.beats as number | undefined) ?? 4;+ if (!isObj(s.patterns) || Object.keys(s.patterns).length === 0) throw new Error('patterns must be { name: { bars, tracks: { instrument: "steps" } } }, at least one');+ for (const [name, value] of Object.entries(s.patterns)) {+ const where = `pattern "${name}"`;+ if (!NAME.test(name)) throw new Error(`${where}: a name is a letter, then letters, digits, _ or -`);+ if (!isObj(value)) throw new Error(`${where} must be { bars, tracks }`);+ for (const key of Object.keys(value)) if (!PATTERN_FIELDS.includes(key)) throw new Error(`${where} has unknown "${key}"; a pattern has ${PATTERN_FIELDS.join(', ')}`);+ if (!(Number.isInteger(value.bars) && between(value.bars, 1, 32))) throw new Error(`${where}: bars must be a whole number, 1..32`);+ if (value.div !== undefined && !(Number.isInteger(value.div) && between(value.div, 1, 8))) throw new Error(`${where}: div must be whole steps to a beat, 1..8`);+ if (value.chords !== undefined) {+ if (!Array.isArray(value.chords) || value.chords.length === 0 || value.chords.length > (value.bars as number) || value.chords.some((c) => !(Number.isInteger(c) && c >= 1 && c <= 7))) {+ throw new Error(`${where}: chords must be the degree of each bar's root, 1..7, no more of them than bars`);+ }+ }+ if (!isObj(value.tracks) || Object.keys(value.tracks).length === 0) throw new Error(`${where}: tracks must be { instrument: "steps" }, at least one`);+ for (const [track, line] of Object.entries(value.tracks)) {+ if (!(track in s.instruments)) throw new Error(`${where} has a track for "${track}", which is not an instrument; the instruments are ${Object.keys(s.instruments).join(', ')}`);+ if (typeof line !== 'string') throw new Error(`${where}, track "${track}" must be a line of steps`);+ parseTrack(line, value.bars as number, beats * ((value.div as number | undefined) ?? 4), `${where}, track "${track}"`);+ }+ }++ if (!Array.isArray(s.song) || s.song.length === 0 || s.song.length > 64) throw new Error('song must be the patterns in order, 1..64 of them');+ s.song.forEach((part, i) => {+ const name = typeof part === 'string' ? part : isObj(part) ? part.pattern : undefined;+ if (typeof name !== 'string' || !(name in (s.patterns as object))) throw new Error(`song part ${i + 1}: "${String(name)}" is not a pattern; the patterns are ${Object.keys(s.patterns as object).join(', ')}`);+ if (isObj(part)) {+ for (const key of Object.keys(part)) if (!['pattern', 'without', 'only'].includes(key)) throw new Error(`song part ${i + 1} has unknown "${key}"; it takes pattern, without, only`);+ if (part.without !== undefined && part.only !== undefined) throw new Error(`song part ${i + 1}: give without or only, not both`);+ for (const key of ['without', 'only']) {+ const list = part[key];+ if (list === undefined) continue;+ if (!Array.isArray(list) || list.some((n) => typeof n !== 'string' || !(n in (s.instruments as object)))) throw new Error(`song part ${i + 1}: ${key} must be a list of instruments (${Object.keys(s.instruments as object).join(', ')})`);+ }+ }+ });+ const source = s as unknown as MusicSource;+ const seconds = musicSeconds(source);+ if (seconds > MAX_MUSIC_SECONDS) throw new Error(`the song lasts ${Math.round(seconds)} s; at most ${MAX_MUSIC_SECONDS}`);+ return source;+}++export type MusicEdit =+ | { op: 'instrument'; name: string; value: MusicInstrument | null }+ | { op: 'pattern'; name: string; value: MusicPattern | null }+ | { op: 'track'; pattern: string; track: string; steps: string | null }+ | { op: 'settings'; fields: Record<string, unknown> };++/** The source with @p edits made in order, checked as a whole. A bad edit changes nothing. */+export function applyMusicEdits(source: MusicSource, edits: readonly MusicEdit[]): MusicSource {+ const next = JSON.parse(JSON.stringify(source)) as Record<string, unknown> & { instruments: Record<string, unknown>; patterns: Record<string, { tracks: Record<string, string> }> };+ edits.forEach((edit, i) => {+ const where = `edit ${i + 1}`;+ if (!isObj(edit)) throw new Error(`${where} must be { op, ... }`);+ if (edit.op === 'instrument' || edit.op === 'pattern') {+ const into = edit.op === 'instrument' ? next.instruments : next.patterns;+ if (typeof edit.name !== 'string') throw new Error(`${where}: name the ${edit.op}`);+ if (edit.value === null) {+ if (!(edit.name in into)) throw new Error(`${where}: there is no ${edit.op} "${edit.name}"; there are ${Object.keys(into).join(', ')}`);+ delete into[edit.name];+ } else (into as Record<string, unknown>)[edit.name] = edit.value;+ } else if (edit.op === 'track') {+ const pattern = next.patterns[edit.pattern];+ if (!pattern) throw new Error(`${where}: there is no pattern "${edit.pattern}"; there are ${Object.keys(next.patterns).join(', ')}`);+ if (edit.steps === null) delete pattern.tracks[edit.track];+ else pattern.tracks[edit.track] = edit.steps;+ } else if (edit.op === 'settings') {+ if (!isObj(edit.fields)) throw new Error(`${where}: settings needs fields`);+ for (const [key, value] of Object.entries(edit.fields)) {+ if (['instruments', 'patterns'].includes(key)) throw new Error(`${where}: change ${key} with the ${key.slice(0, -1)} op`);+ if (value === null) delete next[key];+ else next[key] = value;+ }+ } else throw new Error(`${where}: op must be instrument, pattern, track or settings`);+ });+ return validateMusicSource(next);+}diff --git a/pipeline/src/sound/soundPicture.ts b/pipeline/src/sound/soundPicture.tsindex eccd67d9a..9fc5ac5ec 100644--- a/pipeline/src/sound/soundPicture.ts+++ b/pipeline/src/sound/soundPicture.ts@@ -20,19 +20,21 @@ export function fft(re: Float64Array, im: Float64Array): void {let bit = n >> 1;for (; j & bit; bit >>= 1) j ^= bit;j ^= bit;- if (i < j) { [re[i], re[j]] = [re[j]!, re[i]!]; [im[i], im[j]] = [im[j]!, im[i]!]; }+ if (i < j) { const r = re[i]!; re[i] = re[j]!; re[j] = r; const m = im[i]!; im[i] = im[j]!; im[j] = m; }}for (let size = 2; size <= n; size <<= 1) {const angle = (-2 * Math.PI) / size;- const [wr, wi] = [Math.cos(angle), Math.sin(angle)];+ const wr = Math.cos(angle), wi = Math.sin(angle), half = size >> 1;+ // Plain locals throughout: this loop is all the time a long convolution takes.for (let start = 0; start < n; start += size) {- let [cr, ci] = [1, 0];- for (let k = 0; k < size / 2; k++) {- const [a, b] = [start + k, start + k + size / 2];- const [tr, ti] = [re[b]! * cr - im[b]! * ci, re[b]! * ci + im[b]! * cr];+ let cr = 1, ci = 0;+ for (let k = 0; k < half; k++) {+ const a = start + k, b = a + half;+ const tr = re[b]! * cr - im[b]! * ci, ti = re[b]! * ci + im[b]! * cr;re[b] = re[a]! - tr; im[b] = im[a]! - ti;re[a] = re[a]! + tr; im[a] = im[a]! + ti;- [cr, ci] = [cr * wr - ci * wi, cr * wi + ci * wr];+ const next = cr * wr - ci * wi;+ ci = cr * wi + ci * wr; cr = next;}}}diff --git a/pipeline/src/sound/soundRender.ts b/pipeline/src/sound/soundRender.tsindex 90391b9e2..b84133c09 100644--- a/pipeline/src/sound/soundRender.ts+++ b/pipeline/src/sound/soundRender.ts@@ -215,7 +215,7 @@ function strike(layer: SoundLayer, seconds: number, rate: number, pitch: number,}/** @p a and @p b laid over one another at every offset: what a sound becomes in a place whose answer to a click is @p b. */-function convolve(a: Float32Array, b: Float32Array, keep: number): Float32Array {+export function convolve(a: Float32Array, b: Float32Array, keep: number): Float32Array {let size = 1;while (size < a.length + b.length) size <<= 1;const [ar, ai, br, bi] = [new Float64Array(size), new Float64Array(size), new Float64Array(size), new Float64Array(size)];@@ -237,7 +237,7 @@ function convolve(a: Float32Array, b: Float32Array, keep: number): Float32Array* top in half the time of the bottom, coming up over a hundredth of a second, with thirty-odd distinct first echoes* ahead of it. Open ground: a few slaps off far walls, each duller than the last, and a long faint tail.*/-function placeResponse(rate: number, size: number, kind: 'room' | 'open', side: number): Float32Array {+export function placeResponse(rate: number, size: number, kind: 'room' | 'open', side: number): Float32Array {const random = noiseOf(kind === 'room' ? 6923 + side * 977 : 32324 + side * 977);const unit = (): number => random() * 0.5 + 0.5;const seconds = kind === 'room' ? 0.35 + size * 2.4 : 0.6 + size * 2.6;@@ -353,8 +353,11 @@ interface Stray { pitch: number; loud: number; decay: number; late: number; seed* The samples a validated source makes. @p variant 0 is the sound as written; another is the same sound a little* unlike it, as a second shot from the same gun is: other noise, and each layer's pitch, loudness, decay and timing* off by up to the source's `vary`.+ *+ * @p at is for a sound played as a note: `pitch` times its written pitch, with `seed` for other noise, and `raw`+ * leaving its level as the layers sum to, so that many notes keep their balance against one another.*/-export function renderSound(source: SoundSource, variant = 0): RenderedSound {+export function renderSound(source: SoundSource, variant = 0, at: { pitch?: number; seed?: number; raw?: boolean } = {}): RenderedSound {const rate = source.rate ?? 44100;const count = Math.max(1, Math.round(source.duration * rate));// A loop is rendered a little long and its extra folded back over its start, so the end runs into the beginning.@@ -445,7 +448,7 @@ export function renderSound(source: SoundSource, variant = 0): RenderedSound {source.layers.forEach((layer, index) => {if (layer.mute) return;const where = `layer ${index}${layer.id ? ` "${layer.id}"` : ''}`;- const made = play(layer, index, where, { offset: 0, pitch: 1, loud: 1, pan: 0, seed: 0 }, true);+ const made = play(layer, index, where, { offset: 0, pitch: at.pitch ?? 1, loud: 1, pan: 0, seed: at.seed ?? 0 }, true);const start = layer.start ?? 0;reports.push({...(layer.id ? { id: layer.id } : {}), source: layer.source, from: Math.round(start * 1000) / 1000, to: Math.round(made.last * 1000) / 1000,@@ -499,7 +502,9 @@ export function renderSound(source: SoundSource, variant = 0): RenderedSound {let peak = 0;for (const channel of out) for (const v of channel) if (Math.abs(v) > peak) peak = Math.abs(v);const target = source.gain ?? 0.9;- if (source.normalize !== false) {+ if (at.raw) {+ // Left as it sums: whoever plays it as a note sets the level of the whole.+ } else if (source.normalize !== false) {if (peak > 1e-6) for (const channel of out) for (let n = 0; n < count; n++) channel[n]! *= target / peak;else notes.push('the sound is silent: nothing in it reaches a level that can be heard');peak = peak > 1e-6 ? target : 0;diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.tsnew file mode 100644index 000000000..b5ecde34d--- /dev/null+++ b/pipeline/tests/music.test.ts@@ -0,0 +1,204 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * Music from instruments and patterns, held to what can be counted: which notes a line of steps is, when each sounds,+ * what pitch comes out, that a loop runs into itself and that the instruments written apart add up to the whole.+ */+import { describe, expect, it } from 'vitest';+import { applyMusicEdits, validateMusicSource, type MusicSource } from '../src/music/musicSource';+import { compileMusic, noteName, renderMusic } from '../src/music/musicRender';+import { musicPicture } from '../src/music/musicPicture';++const tone = { layers: [{ source: 'sine', freq: 440 }] } as const;+const click = { layers: [{ source: 'noise', length: 0.02, decay: 0.005 }] } as const;+/** One bar of four beats at 120: two seconds, each of its sixteen steps an eighth of a second. */+const piece = (tracks: Record<string, string>, rest: Record<string, unknown> = {}, pattern: Record<string, unknown> = {}): MusicSource => validateMusicSource({+ tempo: 120, key: 'A', scale: 'minor', humanize: 0, loop: false,+ instruments: { tone, click, low: { ...tone, octave: -1, follow: true } },+ patterns: { a: { bars: 1, tracks, ...pattern } },+ song: ['a'],+ ...rest,+});+const REST = '. . . . . . . . . . . . . . . .';+const steps = (...first: string[]): string => [...first, ...REST.split(' ')].slice(0, 16).join(' ');+function pitchOf(samples: Float32Array, rate: number, from: number, to: number): number {+ const [a, b] = [Math.floor(from * rate), Math.floor(to * rate)];+ let crossings = 0;+ for (let n = a + 1; n < b; n++) if (samples[n - 1]! <= 0 && samples[n]! > 0) crossings++;+ return crossings / ((b - a) / rate);+}+const level = (samples: Float32Array, rate: number, from: number, to: number): number => {+ let sum = 0;+ for (let n = Math.floor(from * rate); n < Math.floor(to * rate); n++) sum += samples[n]! * samples[n]!;+ return Math.sqrt(sum / Math.max(1, Math.floor((to - from) * rate)));+};++describe('music from instruments and patterns', () => {+ it('reads a degree as a note of the key, with octaves and accidentals', () => {+ const notes = (line: string, rest = {}) => compileMusic(piece({ tone: line }, rest)).events.map((e) => noteName(e.midi!));+ expect(notes(steps('1', '2', '3', '4', '5', '6', '7', "1'"))).toEqual(['A4', 'B4', 'C5', 'D5', 'E5', 'F5', 'G5', 'A5']);+ expect(notes(steps('1', '2', '3'), { key: 'C', scale: 'major' })).toEqual(['C4', 'D4', 'E4']);+ expect(notes(steps("5,", "3''", '#7', 'b2'))).toEqual(['E4', 'C7', 'G#5', 'A#4']);+ expect(notes(steps('[1 3 5]'))).toEqual(['A4', 'C5', 'E5']);+ });++ it('holds a note for as many steps as are dashes after it, and starts each step on time', () => {+ const e = compileMusic(piece({ tone: steps('1', '-', '-', '.', '3', '5', '-') })).events;+ expect(e.map((x) => [x.at, x.seconds])).toEqual([[0, 0.375], [0.5, 0.125], [0.625, 0.25]]);+ // Triplets: three steps to a beat.+ const t = compileMusic(piece({ tone: '1 . 1 . . . . . . . . .' }, {}, { div: 3 })).events;+ expect(t[1]!.at).toBeCloseTo(1 / 3, 6);+ });++ it('moves a line that follows with the chords, and keeps their roots near the tonic', () => {+ const source = validateMusicSource({+ tempo: 120, key: 'A', humanize: 0, instruments: { low: { ...tone, follow: true }, tone },+ patterns: { a: { bars: 4, chords: [1, 6, 3, 7], tracks: { low: Array(4).fill(steps('1', '3', '5')).join(' | '), tone: Array(4).fill(steps('1')).join('|') } } },+ song: ['a'],+ });+ const e = compileMusic(source).events;+ const of = (name: string) => e.filter((x) => x.instrument === name).map((x) => noteName(x.midi!));+ // A minor, F below it, C above, G below: each bar the same three steps, on another chord.+ expect(of('low')).toEqual(['A4', 'C5', 'E5', 'F4', 'A4', 'C5', 'C5', 'E5', 'G5', 'G4', 'B4', 'D5']);+ // What does not follow stays where it is written.+ expect(of('tone')).toEqual(['A4', 'A4', 'A4', 'A4']);+ });++ it('plays the pitch it says', () => {+ const m = renderMusic(piece({ tone: steps('1', '-', '-', '-', '-', '-', '-', '-', "1'", '-', '-', '-', '-', '-', '-', '-'), low: REST }));+ expect(pitchOf(m.channels[0]!, m.rate, 0.1, 0.9)).toBeCloseTo(440, -1);+ expect(pitchOf(m.channels[0]!, m.rate, 1.1, 1.9)).toBeCloseTo(880, -1.3);+ expect(m.instruments.tone).toMatchObject({ notes: 2, low: 'A4', high: 'A5' });+ });++ it('strikes a drum at its own pitch, and a layer with a length keeps it however long the step', () => {+ const m = renderMusic(piece({ click: steps('x', '-', '-', '-', '.', '.', '.', '.', 'x!') }));+ expect(m.events.map((e) => e.midi)).toEqual([null, null]);+ expect(level(m.channels[0]!, m.rate, 0, 0.02)).toBeGreaterThan(0.02);+ expect(level(m.channels[0]!, m.rate, 0.1, 0.9)).toBeLessThan(1e-4);+ // The second is marked louder.+ expect(level(m.channels[0]!, m.rate, 1, 1.02)).toBeGreaterThan(level(m.channels[0]!, m.rate, 0, 0.02) * 1.15);+ });++ it('lets a held note go over the release, and no longer', () => {+ const m = renderMusic(piece({ tone: steps('1', '-', '-', '-') }, { instruments: { tone: { ...tone, release: 0.2 }, click, low: tone } }));+ expect(level(m.channels[0]!, m.rate, 0.1, 0.45)).toBeGreaterThan(0.3);+ expect(level(m.channels[0]!, m.rate, 0.75, 1.5)).toBeLessThan(1e-4);+ });++ it('brings what rings past the end of a loop in at its start', () => {+ // One long note on the last step: nearly all of it is past the end.+ const last = `${REST.slice(0, -1)}1`;+ const instruments = { tone: { layers: [{ source: 'sine', freq: 440, length: 0.8, decay: 0.3 }] }, click, low: tone };+ const open = renderMusic(piece({ tone: last }, { loop: false, instruments }));+ const round = renderMusic(piece({ tone: last }, { loop: true, instruments }));+ expect(round.channels[0]!.length).toBe(open.channels[0]!.length);+ expect(level(open.channels[0]!, open.rate, 0.05, 0.4)).toBeLessThan(1e-4);+ expect(level(round.channels[0]!, round.rate, 0.05, 0.4)).toBeGreaterThan(0.05);+ expect(pitchOf(round.channels[0]!, round.rate, 0.05, 0.4)).toBeCloseTo(440, -1);+ // And where the end meets the start there is no step in the wave.+ const c = round.channels[0]!;+ expect(Math.abs(c[0]! - c[c.length - 1]!)).toBeLessThan(0.08);+ });++ it('writes instruments apart that add up to the whole, room and all', () => {+ const m = renderMusic(piece({ tone: steps('1', '-', '.', '.', '5'), click: steps('x', '.', 'x') }, { stems: true, reverb: { size: 0.3, mix: 0.3 } }));+ expect(Object.keys(m.stems!)).toEqual(['tone', 'click', 'low']);+ let worst = 0;+ for (const c of [0, 1]) for (let n = 0; n < m.channels[c]!.length; n += 7) {+ const sum = Object.values(m.stems!).reduce((s, stem) => s + stem[c]![n]!, 0);+ worst = Math.max(worst, Math.abs(sum - m.channels[c]![n]!));+ }+ expect(worst).toBeLessThan(1e-4);+ // The room goes on after the note is over.+ expect(level(m.stems!.tone![0]!, m.rate, 0.9, 1.2)).toBeGreaterThan(1e-4);+ expect(renderMusic(piece({ tone: steps('1') })).stems).toBeNull();+ });++ it('is the same samples from the same source, humanized or not', () => {+ const source = piece({ tone: steps('1', '3', '5'), click: steps('x', 'x') }, { humanize: 0.5, seed: 7 });+ const [a, b] = [renderMusic(source), renderMusic(source)];+ expect(Buffer.from(a.channels[0]!.buffer).equals(Buffer.from(b.channels[0]!.buffer))).toBe(true);+ const second = compileMusic(source).events.filter((e) => e.instrument === 'tone')[1]!;+ expect(second.at).not.toBe(0.125);+ expect(Math.abs(second.at - 0.125)).toBeLessThan(0.007);+ });++ it('makes every second step late when it swings', () => {+ const e = compileMusic(piece({ tone: steps('1', '1', '1', '1') }, { swing: 0.4 })).events;+ expect(e.map((x) => Math.round(x.at * 1000))).toEqual([0, 175, 250, 425]);+ });++ it('leaves instruments out of a part of the song, or keeps only some', () => {+ const source = piece({ tone: steps('1'), click: steps('x') }, { song: [{ pattern: 'a', without: ['click'] }, 'a', { pattern: 'a', only: ['click'] }] });+ const m = compileMusic(source);+ expect(m.seconds).toBe(6);+ expect(m.parts.map((p) => p.at)).toEqual([0, 2, 4]);+ expect(m.events.map((e) => `${e.part}${e.instrument}`)).toEqual(['0tone', '1tone', '1click', '2click']);+ });++ it('says what is wrong with the notes without hearing them', () => {+ const notes = (tracks: Record<string, string>, rest = {}) => compileMusic(piece(tracks, rest)).notes;+ expect(notes({ tone: steps('1'), click: steps('x'), low: steps('1') })).toEqual(['"low" follows the chords, and pattern "a" has none: it plays every bar on chord 1']);+ expect(notes({ tone: steps('1') })).toEqual(expect.arrayContaining(['instrument "click" is never played', 'instrument "low" is never played']));+ expect(notes({ tone: steps('.', '-') }).join()).toMatch(/a "-" at step 2 of bar 1 has no note before it/);+ expect(notes({ tone: steps('#1', 'b3', '5') }).join()).toMatch(/"tone" plays 2 notes outside the key/);+ expect(notes({ tone: steps("1,,,,") }).join()).toMatch(/"tone" goes down to A0, below where a pitch can be told/);+ // Two lines in one octave at one time.+ const crowded = { instruments: { tone, click, low: tone, other: tone } };+ expect(notes({ tone: steps('1', '-', '-', '-'), other: steps('3', '-', '-', '-') }, crowded).join()).toMatch(/"tone" and "other" sound together in the same octave/);+ expect(notes({ tone: steps('1', '-', '-', '-'), other: steps("3'", '-', '-', '-') }, crowded).join()).not.toMatch(/same octave/);+ expect(compileMusic(piece({ tone: steps('1') }, { patterns: { a: { bars: 1, tracks: { tone: steps('1') } }, b: { bars: 1, tracks: { tone: REST } } }, song: ['a', 'b'] })).notes)+ .toEqual(expect.arrayContaining(['part 2 of the song ("b") plays nothing']));+ });++ it('refuses a source that is not well formed, naming where', () => {+ const bad = (tracks: Record<string, string>, rest = {}, pattern = {}) => () => piece(tracks, rest, pattern);+ expect(bad({ tone: '1 . . .' })).toThrow(/pattern "a", track "tone", bar 1 has 4 steps; a bar here is 16/);+ expect(bad({ tone: `${REST} | ${REST}` })).toThrow(/has 2 bars between its \| marks; the pattern is 1/);+ expect(bad({ tone: steps('8') })).toThrow(/"8" is not a step/);+ expect(bad({ tone: steps('[1 9]') })).toThrow(/"\[1 9\]" is not a step/);+ expect(bad({ drums: REST })).toThrow(/a track for "drums", which is not an instrument; the instruments are tone, click, low/);+ expect(bad({ tone: REST }, { key: 'H' })).toThrow(/key must be a tonic/);+ expect(bad({ tone: REST }, { song: ['b'] })).toThrow(/song part 1: "b" is not a pattern; the patterns are a/);+ expect(bad({ tone: REST }, { instruments: { tone: { layers: [{ source: 'sine' }] } } })).toThrow(/instrument "tone": layer 0: a sine needs freq/);+ expect(bad({ tone: REST }, {}, { chords: [1, 8] })).toThrow(/chords must be the degree of each bar's root/);+ expect(bad({ tone: REST }, { tempo: 40, song: Array(64).fill('a') })).toThrow(/the song lasts 384 s; at most 240/);+ });++ it('changes a track, an instrument or a setting without the rest restated', () => {+ const source = piece({ tone: steps('1') });+ const edited = applyMusicEdits(source, [+ { op: 'track', pattern: 'a', track: 'click', steps: steps('x') },+ { op: 'instrument', name: 'bell', value: { layers: [{ source: 'modes', modes: [[440, 1, 0.4]] }] } },+ { op: 'settings', fields: { tempo: 90, humanize: null } },+ ]);+ expect(Object.keys(edited.patterns.a!.tracks)).toEqual(['tone', 'click']);+ expect(edited.tempo).toBe(90);+ expect(edited.humanize).toBeUndefined();+ expect(Object.keys(edited.instruments)).toContain('bell');+ expect(source.tempo).toBe(120);+ expect(() => applyMusicEdits(source, [{ op: 'track', pattern: 'b', track: 'tone', steps: REST }])).toThrow(/no pattern "b"; there are a/);+ expect(() => applyMusicEdits(source, [{ op: 'instrument', name: 'tone', value: null }])).toThrow(/a track for "tone", which is not an instrument/);+ expect(() => applyMusicEdits(source, [{ op: 'track', pattern: 'a', track: 'tone', steps: '1 2' }])).toThrow(/has 2 steps/);+ });++ it('draws every note where it sounds and as high as it is', () => {+ const m = renderMusic(piece({ tone: steps('1', '-', '-', '-', '.', '.', '.', '.', "1'", '-', '-', '-'), click: steps('x') }));+ const p = musicPicture(m);+ expect(p.colours).toEqual({ tone: 'blue', click: 'orange', low: 'green' });+ expect([p.low, p.high]).toEqual([67, 83]);+ const at = (x: number, y: number): number[] => [...p.rgba.subarray((y * p.width + x) * 4, (y * p.width + x) * 4 + 3)];+ const row = (p.height - 48 - 6 - 12) / (p.high - p.low + 1);+ const yOf = (midi: number): number => 48 + (p.high - midi) * row + 1;+ const blue = (c: number[]): boolean => c[2]! > 150 && c[2]! > c[0]! + 40;+ // The low A in the first quarter of the bar, the high one in the third, and neither anywhere else.+ expect(blue(at(Math.round(p.width * 0.1), yOf(69)))).toBe(true);+ expect(blue(at(Math.round(p.width * 0.6), yOf(69)))).toBe(false);+ expect(blue(at(Math.round(p.width * 0.6), yOf(81)))).toBe(true);+ expect(blue(at(Math.round(p.width * 0.1), yOf(81)))).toBe(false);+ // The drum on its own row at the bottom, in its own colour.+ const drum = at(1, p.height - 6);+ expect(drum[0]! > 150 && drum[0]! > drum[2]! + 40).toBe(true);+ });+});diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 471478950..b443f8c60 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -4602,6 +4602,10 @@ export const NOTED = {{"note": "An AI can make a sound effect."},"chore(desktop): the sound tool text makes a footstep as a shoe on a floor":{"note": "An AI can make a sound effect."},+ "feat(music): a piece of music is its instruments, the patterns they play and the order they come in - an .esmusic of steps written as degrees of the key, with chords a bass or a pad follows, makes a .wav that runs into itself and a file for each instrument; and a picture of its notes for whoever cannot hear it":+ {"note": "An AI can make background music."},+ "feat(music): music_write, music_edit and music_read make background music from instruments and patterns - the reply is a picture of the notes and what can be said of them unheard":+ {"note": "An AI can make background music."},"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":{"note": "A sky of your own shades the ground under its clouds."},"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":