A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(music): a piece is mixed, breathes, and takes all a SoundFont gives - master compress and shelves and an instrument's lowcut and highcut, rubato that gives and takes back time across a pattern, and from a SoundFont both sides of a stereo recording with each zone's level and filter as the sample's gain and cutoff
yhhdde0025ddb01
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex dbcb84e93..73e7f5d72 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -26,7 +26,7 @@ published separately; it ships inside the editor.- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.- **Music with an introduction, and a source that fades.** Background music looped from its first sample to its last, so a piece could not open with something heard once; and an `AudioSource` in a scene started and stopped at full volume, the fades being `playBGM`'s alone. A looping voice now takes `loopStart`, seconds into the clip that it comes back to at its end: `playBGM(url, { loopStart })` and `AudioSource.loopStart` play the file from its top once and round from there after. It is exact to the sample where the clip is decoded, on the web and on the native host; where a clip is played from its compressed file (a long clip streamed on the web, a mini-game) the player is sent back when it ends, which can be heard as a breath. `AudioSource.fadeIn` and `fadeOut` are seconds: raising `playing` brings it up from nothing, lowering it lets it down before the voice is stopped, and raising it again on the way out turns it round where it is. A piece made with `music_write` marks its first parts as an introduction with `intro`, and the reply gives the `loopStart` to play it with.-- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Right notes at one loudness, each begun from nothing, are a machine reading a score, so a line can be played: `dynamics` on a pattern give each bar the loudness it begins on and move evenly between them, a crescendo and a diminuendo; `legato` on an instrument brings a note that runs straight on from the last in under it, from past the attack of its recording, as a bow that does not leave the string; and `pedal` holds every note of a piano to the end of its bar. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.+- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Right notes at one loudness, each begun from nothing, are a machine reading a score, so a line can be played: `dynamics` on a pattern give each bar the loudness it begins on and move evenly between them, a crescendo and a diminuendo; `legato` on an instrument brings a note that runs straight on from the last in under it, from past the attack of its recording, as a bow that does not leave the string; and `pedal` holds every note of a piano to the end of its bar. A piece is mixed: an instrument's `lowcut` and `highcut` take off what it has no need of, and `master` holds the loud moments down with `compress` and tilts the whole with `low` and `high`, done to the sum and to each stem alike so they still add up. And `rubato` lets each pattern begin held back, move on and ease off, every instrument together. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. A recording made in stereo comes out of a SoundFont as one file with both its sides, and each zone's level and filter come with it as the sample's `gain` and `cutoff`, so a piano's hundred and eighty zones stay its seventeen recordings. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. A layer can also be a recording in the project (`source: "sample"`), played at a speed and shaped as any other layer is, for the one sound that will not come out of waves and noise. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.diff --git a/desktop b/desktopindex 42d21fa21..fc324ff15 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit 42d21fa21812ceb4f93157146acf429697cf58a3+Subproject commit fc324ff15e356047a882cb0b97eeb9ecb0e849afdiff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.tsindex 0d2f4b197..7724a376a 100644--- a/pipeline/src/music/musicRender.ts+++ b/pipeline/src/music/musicRender.ts@@ -114,6 +114,11 @@ export function compileMusic(source: MusicSource): CompiledMusic {const instrument = source.instruments[track]!;if (instrument.follow && !pattern.chords) say(`"${track}" follows the chords, and pattern "${name}" has none: it plays every bar on chord 1`);const steps = parseTrack(line, pattern.bars, stepsToBar, `pattern "${name}", track "${track}"`);+ // Time that gives: late at the start of the phrase, made up through its middle, late again toward its end,+ // and back where it was on the last bar line, so the pattern is as long as it was written.+ const phrase = pattern.bars * barSeconds;+ const give = ((source.rubato ?? 0) * 0.08 * phrase) / (2 * Math.PI);+ const timed = (t: number): number => t + give * Math.sin((2 * Math.PI * t) / phrase);// How loud the line is at each step: between the number its bar begins on and the next bar's.const levels = pattern.dynamics?.[track] ?? pattern.dynamics?.all;const levelAt = (i: number): number => {@@ -132,16 +137,17 @@ export function compileMusic(source: MusicSource): CompiledMusic {while (i + held < steps.length && steps[i + held]!.kind === 'hold') held++;// Every second step late, which is what makes a line swing.const late = div % 2 === 0 && i % 2 === 1 ? (source.swing ?? 0) * step : 0;- const when = Math.max(0, at + i * step + late + human * 0.012 * random());+ const when = Math.max(0, at + timed(i * step + late) + human * 0.012 * random());+ const written = timed((i + held) * step) - timed(i * step);const hard = s.loud * levelAt(i);const loud = hard * (1 + human * 0.25 * random());const mark = hard > 1.15 ? 1 : hard < 0.8 ? -1 : 0;// The pedal is down to the end of the bar: the note rings on to there.- const toBarEnd = (Math.floor(i / stepsToBar) + 1) * stepsToBar * step - i * step;- const sounds = instrument.pedal && toBarEnd > held * step ? { sounds: toBarEnd } : {};+ const toBarEnd = timed((Math.floor(i / stepsToBar) + 1) * stepsToBar * step) - timed(i * step);+ const sounds = instrument.pedal && toBarEnd > written + 1e-9 ? { sounds: toBarEnd } : {};played.add(track);if (s.kind === 'hit') {- events.push({ instrument: track, at: when, seconds: held * step, midi: null, loud, part: p, outside: false, mark, hard, ...sounds });+ events.push({ instrument: track, at: when, seconds: written, midi: null, loud, part: p, outside: false, mark, hard, ...sounds });before = null;continue;}@@ -154,7 +160,7 @@ export function compileMusic(source: MusicSource): CompiledMusic {for (const n of s.notes) {const index = n.degree - 1 + chord - 1 + drop + 7;const midi = 60 + tonic + scale[index % 7]! + 12 * (Math.floor(index / 7) - 1 + n.octave + (instrument.octave ?? 0)) + n.alter;- events.push({ instrument: track, at: when, seconds: held * step, midi, loud, part: p, outside: n.alter !== 0, mark, hard, ...sounds, ...(slurred ? { slur: true } : {}) });+ events.push({ instrument: track, at: when, seconds: written, midi, loud, part: p, outside: n.alter !== 0, mark, hard, ...sounds, ...(slurred ? { slur: true } : {}) });}before = s.notes.length === 1 ? { end: i + held, event: events.at(-1)! } : null;}@@ -278,9 +284,11 @@ function sampleNote(name: string, instrument: MusicInstrument, index: number, miif (count - n < ending) level *= (count - n) / ending;for (let c = 0; c < out.length; c++) {const channel = data.channels[c]!;- out[c]![n] = (blend > 0 ? at(channel, p) * (1 - blend) + at(channel, p - round) * blend : at(channel, p)) * level;+ out[c]![n] = (blend > 0 ? at(channel, p) * (1 - blend) + at(channel, p - round) * blend : at(channel, p)) * level * (zone.gain ?? 1);}}+ // Duller than it was recorded, where the sample says so: at a fixed place, whatever the note.+ if (zone.cutoff && zone.cutoff < rate * 0.45) for (const channel of out) filtered(channel, 'lowpass', zone.cutoff, rate);return out;}@@ -332,6 +340,72 @@ export function samplePitch(data: SampleData, near: number): number | null {return data.rate / lag;}+type Shape = 'lowpass' | 'highpass' | 'lowshelf' | 'highshelf';++/** A two-pole filter run over @p channel where it lies: a cut above or below @p hz, or a shelf of @p db there. */+function filtered(channel: Float32Array, shape: Shape, hz: number, rate: number, db = 0): void {+ const w = (2 * Math.PI * Math.min(hz, rate * 0.45)) / rate;+ const [cos, sin] = [Math.cos(w), Math.sin(w)];+ let b0: number, b1: number, b2: number, a0: number, a1: number, a2: number;+ if (shape === 'lowpass' || shape === 'highpass') {+ const alpha = sin / (2 * Math.SQRT1_2);+ [b0, b1, b2] = shape === 'lowpass' ? [(1 - cos) / 2, 1 - cos, (1 - cos) / 2] : [(1 + cos) / 2, -(1 + cos), (1 + cos) / 2];+ [a0, a1, a2] = [1 + alpha, -2 * cos, 1 - alpha];+ } else {+ const A = Math.pow(10, db / 40);+ const beta = 2 * Math.sqrt(A) * (sin / 2) * Math.SQRT2;+ const sign = shape === 'lowshelf' ? 1 : -1;+ b0 = A * (A + 1 - sign * (A - 1) * cos + beta);+ b1 = sign * 2 * A * (A - 1 - sign * (A + 1) * cos);+ b2 = A * (A + 1 - sign * (A - 1) * cos - beta);+ a0 = A + 1 + sign * (A - 1) * cos + beta;+ a1 = -sign * 2 * (A - 1 + sign * (A + 1) * cos);+ a2 = A + 1 + sign * (A - 1) * cos - beta;+ }+ let x1 = 0, x2 = 0, y1 = 0, y2 = 0;+ for (let n = 0; n < channel.length; n++) {+ const x = channel[n]!;+ const y = (b0 * x + b1 * x1 + b2 * x2 - a1 * y1 - a2 * y2) / a0;+ x2 = x1; x1 = x; y2 = y1; y1 = y;+ channel[n] = y;+ }+}++/**+ * How much each moment of a mix is to be held down: 1 where it is quiet, less where it is loud. The loudness is+ * followed quickly up and slowly down, as an ear follows it, and what is over the line is let through at a share of+ * what it is over by. Worked out from the whole and laid on every instrument alike, so they still add up to it.+ */+function compression(channels: readonly Float32Array[], rate: number, amount: number, round: boolean): Float32Array {+ const frames = channels[0]!.length;+ let peak = 0;+ for (const c of channels) for (const v of c) if (Math.abs(v) > peak) peak = Math.abs(v);+ const gains = new Float32Array(frames).fill(1);+ if (!(peak > 1e-6) || amount <= 0) return gains;+ const line = peak * Math.pow(10, -(6 + 14 * amount) / 20);+ const ratio = 1.5 + 2.5 * amount;+ const [up, down] = [Math.exp(-1 / (0.005 * rate)), Math.exp(-1 / (0.14 * rate))];+ let follow = 0;+ const step = (n: number): void => {+ let loud = 0;+ for (const c of channels) loud = Math.max(loud, Math.abs(c[n]!));+ follow = loud > follow ? loud + (follow - loud) * up : loud + (follow - loud) * down;+ };+ // A piece that goes round arrives at its start from its end, so the following starts from there.+ if (round) for (let n = Math.max(0, frames - rate); n < frames; n++) step(n);+ for (let n = 0; n < frames; n++) {+ step(n);+ if (follow > line) gains[n] = (line * Math.pow(follow / line, 1 / ratio)) / follow;+ }+ // The following is always a little behind the sound, so the front of every loud note would get through whole and+ // the peak of the piece would be where it was. So the holding down begins before the note: going backward, the+ // gain may rise only so fast, which is a hand already on the fader as the note arrives.+ const rise = 1 / (0.006 * rate);+ for (let n = frames - 2; n >= 0; n--) gains[n] = Math.min(gains[n]!, gains[n + 1]! + rise);+ if (round) gains[frames - 1] = Math.min(gains[frames - 1]!, gains[0]! + rise);+ return gains;+}+const TAIL_SECONDS = 4;/**@@ -428,6 +502,15 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,}}+ // What each instrument has no need of, taken off before the room hears it.+ for (const name of names) {+ const instrument = source.instruments[name]!;+ for (const channel of buses.get(name)!) {+ if (instrument.lowcut) filtered(channel, 'highpass', instrument.lowcut, rate);+ if (instrument.highcut) filtered(channel, 'lowpass', instrument.highcut, rate);+ }+ }+// The room. One for everything; or, when the instruments are to be written apart, one each, so they still add up.const room = [new Float32Array(work), new Float32Array(work)];if (source.reverb) {@@ -484,6 +567,19 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,return sum;});+ // The whole, once summed: held together, and tilted. Done to the sum and to each instrument alike.+ if (source.master) {+ const every = [channels, ...Object.values(stems)];+ if (source.master.compress) {+ const gains = compression(channels, rate, source.master.compress, loop);+ for (const list of every) for (const channel of list) for (let n = 0; n < total; n++) channel[n]! *= gains[n]!;+ }+ for (const list of every) for (const channel of list) {+ if (source.master.low) filtered(channel, 'lowshelf', 110, rate, source.master.low);+ if (source.master.high) filtered(channel, 'highshelf', 7000, rate, source.master.high);+ }+ }+let peak = 0;for (const channel of channels) for (const v of channel) if (Math.abs(v) > peak) peak = Math.abs(v);const notes = [...compiled.notes];diff --git a/pipeline/src/music/musicSource.ts b/pipeline/src/music/musicSource.tsindex 9681e8171..a05656b84 100644--- a/pipeline/src/music/musicSource.ts+++ b/pipeline/src/music/musicSource.ts@@ -52,6 +52,10 @@ export interface MusicSample {dynamic?: (typeof DYNAMICS)[number];/** Hundredths of a semitone the recording is sharp (above 0) or flat of its note, -100..100, which is taken out as it is played. */tune?: number;+ /** How loud this recording is played beside the instrument's others, 1 being as it is: what evens one recording against the next. */+ gain?: number;+ /** Nothing of it above this many Hz: a bright recording played mellow, which is how one recording serves soft notes and loud. */+ cutoff?: number;}export const DYNAMICS = ['soft', 'normal', 'loud'] as const;@@ -73,6 +77,14 @@ export interface MusicInstrument {follow?: boolean;/** Seconds a held note takes to die after it is let go. Default 0.08. */release?: number;+ /**+ * Nothing of it below this many Hz. Every instrument but the bass and the kick has a bottom it does not need,+ * the thump of a piano's frame, the rumble under a string section, and all of them together are the mud a mix+ * drowns in: 120 to 200 for what plays in the middle, 300 and up for what plays high.+ */+ lowcut?: number;+ /** Nothing of it above this many Hz: what should sit behind the rest is duller than what is in front. */+ highcut?: number;/*** A line played in one breath. Where one note of its line runs straight into the next, with no silence between* and the next not marked `!`, the next is not struck afresh: it comes in under the one before over this many@@ -154,6 +166,18 @@ export interface MusicSource {stems?: boolean;/** The room it is all in: `size` 0..1, `mix` 0..1. */reverb?: { size?: number; mix?: number };+ /**+ * Time that gives: each pattern is a phrase that begins a little held back, moves on through its middle and+ * eases off to its end, by up to this share of 8% of the tempo, 0..1. Every instrument moves together and a+ * pattern is as long as it was. 0.3 is a player breathing; 1 is a romantic one. Not for what is danced to.+ */+ rubato?: number;+ /**+ * What is done to the whole once it is summed. `compress` 0..1 holds the loud moments down so the quiet ones+ * come up and the instruments are heard as one thing and not several (0.3 to 0.5 for most pieces); `low` and+ * `high` are decibels more or less below 110 Hz and above 7 kHz, -12..12.+ */+ master?: { compress?: number; low?: number; high?: number };/** How loud the loudest moment is, 1 being full. Default 0.9. */gain?: number;rate?: number;@@ -227,8 +251,8 @@ export function parseTrack(text: string, bars: number, stepsToBar: number, wherereturn out;}-const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'intro', 'fadeIn', 'fadeOut', 'stems', 'reverb', 'gain', 'rate'];-const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb', 'legato', 'pedal'];+const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'intro', 'fadeIn', 'fadeOut', 'stems', 'reverb', 'rubato', 'master', 'gain', 'rate'];+const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb', 'legato', 'pedal', 'lowcut', 'highcut'];const RECORDING = /\.(wav|ogg|mp3|flac|m4a|aac|webm)$/i;function checkSamples(samples: unknown, where: string): void {@@ -236,7 +260,9 @@ function checkSamples(samples: unknown, where: string): void {samples.forEach((sample: unknown, i) => {const at = `${where}, sample ${i + 1}`;if (!isObj(sample)) throw new Error(`${at} must be { file, note }`);- for (const key of Object.keys(sample)) if (!['file', 'note', 'low', 'high', 'loop', 'dynamic', 'tune'].includes(key)) throw new Error(`${at} has unknown "${key}"; a sample has file, note, low, high, loop, dynamic, tune`);+ for (const key of Object.keys(sample)) if (!['file', 'note', 'low', 'high', 'loop', 'dynamic', 'tune', 'gain', 'cutoff'].includes(key)) throw new Error(`${at} has unknown "${key}"; a sample has file, note, low, high, loop, dynamic, tune, gain, cutoff`);+ if (sample.gain !== undefined && !between(sample.gain, 0.01, 4)) throw new Error(`${at}: gain must be 0.01..4, 1 being the recording as it is`);+ if (sample.cutoff !== undefined && !between(sample.cutoff, 200, 20000)) throw new Error(`${at}: cutoff must be Hz, 200..20000`);if (sample.dynamic !== undefined && !DYNAMICS.includes(sample.dynamic as never)) throw new Error(`${at}: dynamic must be one of ${DYNAMICS.join(', ')}`);if (sample.tune !== undefined && !between(sample.tune, -100, 100)) throw new Error(`${at}: tune must be hundredths of a semitone, -100..100`);if (typeof sample.file !== 'string' || !RECORDING.test(sample.file)) throw new Error(`${at}: file must be a project path to a recording (.wav, .ogg, .mp3, .flac, .m4a)`);@@ -280,6 +306,13 @@ export function validateMusicSource(raw: unknown): MusicSource {if (s.reverb.mix !== undefined && !between(s.reverb.mix, 0, 1)) throw new Error('reverb.mix must be 0..1');}+ if (s.rubato !== undefined && !between(s.rubato, 0, 1)) throw new Error('rubato must be 0..1');+ if (s.master !== undefined) {+ if (!isObj(s.master)) throw new Error('master must be { compress?, low?, high? }');+ for (const key of Object.keys(s.master)) if (!['compress', 'low', 'high'].includes(key)) throw new Error(`master has unknown "${key}"; it takes compress, low, high`);+ if (s.master.compress !== undefined && !between(s.master.compress, 0, 1)) throw new Error('master.compress must be 0..1');+ for (const key of ['low', 'high']) if (s.master[key] !== undefined && !between(s.master[key], -12, 12)) throw new Error(`master.${key} must be decibels, -12..12`);+ }if (!isObj(s.instruments) || Object.keys(s.instruments).length === 0) throw new Error('instruments must be { name: { layers: [...] } }, at least one');if (Object.keys(s.instruments).length > 16) throw new Error('at most 16 instruments');for (const [name, value] of Object.entries(s.instruments)) {@@ -306,6 +339,8 @@ export function validateMusicSource(raw: unknown): MusicSource {if (value.reverb !== undefined && !between(value.reverb, 0, 1)) throw new Error(`${where}: reverb must be 0..1`);if (value.legato !== undefined && typeof value.legato !== 'boolean' && !between(value.legato, 0.01, 0.5)) throw new Error(`${where}: legato must be true, or the seconds one note takes to come in under the last, 0.01..0.5`);if (value.pedal !== undefined && typeof value.pedal !== 'boolean') throw new Error(`${where}: pedal must be true or false`);+ if (value.lowcut !== undefined && !between(value.lowcut, 20, 2000)) throw new Error(`${where}: lowcut must be Hz, 20..2000`);+ if (value.highcut !== undefined && !between(value.highcut, 500, 20000)) throw new Error(`${where}: highcut must be Hz, 500..20000`);}const beats = (s.beats as number | undefined) ?? 4;diff --git a/pipeline/src/music/soundFont.ts b/pipeline/src/music/soundFont.tsindex b1ecdd85c..5ba8bedef 100644--- a/pipeline/src/music/soundFont.ts+++ b/pipeline/src/music/soundFont.ts@@ -9,9 +9,12 @@* out as the `samples` a music source takes (musicSource.ts) and the recordings as samples to be written as* `.wav` files, after which the SoundFont is not needed: the project holds what it uses and no more.*- * What a SoundFont can say and this does not carry over: its filters and envelopes beyond attack and release,- * its modulators, and the second side of a recording made in stereo. Two things it does are undone on the way- * out, and said. Where it plays several recordings at once for one note (a second rank of pipes, a string+ * What comes over with a recording: both sides of one made in stereo, as one file of two; the filter its+ * zone sets over it, as the sample's `cutoff`, since a SoundFont makes a mellow instrument by dulling a+ * bright recording; and how far the zone turns it down, as its `gain`, which is what balances one recording+ * against the next. A piano's hundred and eighty zones are seventeen recordings each used many ways, and+ * they stay seventeen files. What does not come over: envelopes beyond attack and+ * release, filters that move, and modulators. Two things a SoundFont does are undone on the way out, and said. Where it plays several recordings at once for one note (a second rank of pipes, a string* under a hammer) one is kept, since a sample here is one recording to a note. And where it sets an* instrument whole octaves from its recordings, the recordings come out at the pitch they are: an instrument* here goes down an octave by its `octave`, and one whose middle C sounded the C below would be out of tune@@ -33,8 +36,8 @@ export interface SoundFontPreset {high: number;}-interface Zone { sample: number; keyLow: number; keyHigh: number; velLow: number; velHigh: number; root: number; cents: number; octaves: number; loop: boolean; offsets: [number, number, number, number]; attack: number | null; release: number | null }-interface SampleHeader { name: string; start: number; end: number; loopStart: number; loopEnd: number; rate: number; pitch: number; correction: number; type: number }+interface Zone { cutoff: number; quieter: number; sample: number; keyLow: number; keyHigh: number; velLow: number; velHigh: number; root: number; cents: number; octaves: number; loop: boolean; offsets: [number, number, number, number]; attack: number | null; release: number | null }+interface SampleHeader { name: string; start: number; end: number; loopStart: number; loopEnd: number; rate: number; pitch: number; correction: number; type: number; link: number }export interface SoundFont {name: string;@@ -45,7 +48,7 @@ export interface SoundFont {}// The generators read here, by their numbers in the format.-const GEN = { startOffset: 0, endOffset: 1, loopStartOffset: 2, loopEndOffset: 3, startCoarse: 4, endCoarse: 12, attack: 34, release: 38, instrument: 41, keyRange: 43, velRange: 44, loopStartCoarse: 45, loopEndCoarse: 50, coarseTune: 51, fineTune: 52, sampleId: 53, sampleModes: 54, rootKey: 58 } as const;+const GEN = { filterCutoff: 8, filterOpens: 11, attenuation: 48, startOffset: 0, endOffset: 1, loopStartOffset: 2, loopEndOffset: 3, startCoarse: 4, endCoarse: 12, attack: 34, release: 38, instrument: 41, keyRange: 43, velRange: 44, loopStartCoarse: 45, loopEndCoarse: 50, coarseTune: 51, fineTune: 52, sampleId: 53, sampleModes: 54, rootKey: 58 } as const;type Generators = Map<number, number>;const signed = (v: number): number => (v > 32767 ? v - 65536 : v);@@ -83,7 +86,7 @@ export function readSoundFont(bytes: Uint8Array): SoundFont {for (let at = of('shdr').at; at + 46 <= of('shdr').at + of('shdr').size - 46; at += 46) {headers.push({name: text(at, 20), start: view.getUint32(at + 20, true), end: view.getUint32(at + 24, true), loopStart: view.getUint32(at + 28, true), loopEnd: view.getUint32(at + 32, true),- rate: view.getUint32(at + 36, true), pitch: bytes[at + 40]!, correction: view.getInt8(at + 41), type: view.getUint16(at + 44, true),+ rate: view.getUint32(at + 36, true), pitch: bytes[at + 40]!, correction: view.getInt8(at + 41), link: view.getUint16(at + 42, true), type: view.getUint16(at + 44, true),});}@@ -128,7 +131,14 @@ export function readSoundFont(bytes: Uint8Array): SoundFont {if (keyLow > keyHigh || velLow > velHigh) continue;const rootKey = get(GEN.rootKey);const coarse = signed(get(GEN.coarseTune) ?? 0) + add(GEN.coarseTune);+ // The filter over it, in Hz: written as hundredths of a semitone above 8.176 Hz, and open where it is not written.+ // An envelope that opens the filter as the note sounds is taken at half its way: a filter that stood where it+ // starts would leave a trumpet as dull as its first hundredth of a second.+ const cents = signed(get(GEN.filterCutoff) ?? 13500) + add(GEN.filterCutoff) + 0.5 * Math.max(0, signed(get(GEN.filterOpens) ?? 0) + add(GEN.filterOpens));mine.push({+ cutoff: 8.176 * Math.pow(2, Math.max(1500, Math.min(13500, cents)) / 1200),+ // Tenths of a decibel it is turned down, of which players by custom take four parts in ten.+ quieter: Math.max(0, signed(get(GEN.attenuation) ?? 0) + add(GEN.attenuation)),sample: iz.get(GEN.sampleId)!, keyLow, keyHigh, velLow, velHigh,root: rootKey !== undefined && signed(rootKey) >= 0 ? rootKey : header.pitch <= 127 ? header.pitch : 60,// Whole octaves are kept apart from the rest: they are where the instrument is set, not how the recording is tuned.@@ -193,6 +203,7 @@ export function liftSoundFontPreset(font: SoundFont, index: number, dir: string)const samples = zones.map((zone): MusicSample => {const header = font.headers[zone.sample]!;const [from, to] = [Math.max(0, header.start + zone.offsets[0]), Math.min(font.data.length, header.end + zone.offsets[1])];+ const dull = zone.cutoff < Math.min(16000, (header.rate || 44100) * 0.45);const key = `${zone.sample}|${from}|${to}`;let file = named.get(key);if (!file) {@@ -202,9 +213,15 @@ export function liftSoundFontPreset(font: SoundFont, index: number, dir: string)taken.add(leaf.toLowerCase());file = `${folder}/${leaf}.wav`;named.set(key, file);- const channel = new Float32Array(Math.max(1, to - from));- for (let n = 0; n < channel.length; n++) channel[n] = font.data[from + n]! / 32768;- files[file] = { rate: header.rate || 44100, channels: [channel] };+ const side = (start: number): Float32Array => {+ const channel = new Float32Array(Math.max(1, to - from));+ for (let n = 0; n < channel.length; n++) channel[n] = (font.data[start + n] ?? 0) / 32768;+ return channel;+ };+ // Made in stereo, its other side is the recording it is linked to, as long as it is: the two are one file.+ const other = header.type === 4 ? font.headers[header.link] : undefined;+ const paired = other && other.type === 2 && other.end - other.start === header.end - header.start;+ files[file] = { rate: header.rate || 44100, channels: paired ? [side(from), side(other.start + zone.offsets[0])] : [side(from)] };}// Played `cents` higher than its root says: the same as a recording of a note that much lower.const exact = zone.root - zone.cents / 100;@@ -212,6 +229,8 @@ export function liftSoundFontPreset(font: SoundFont, index: number, dir: string)const tune = Math.round((exact - note) * 100);const [loopFrom, loopTo] = [(header.loopStart + zone.offsets[2] - from) / (header.rate || 44100), (header.loopEnd + zone.offsets[3] - from) / (header.rate || 44100)];const mid = (zone.velLow + zone.velHigh) / 2;+ // Tenths of a decibel it is turned down, of which players by custom take four parts in ten.+ const gain = Math.round(Math.pow(10, (-0.4 * zone.quieter) / 200) * 100) / 100;return {file, note: noteName(note),...(ranged ? { low: noteName(zone.keyLow), high: noteName(zone.keyHigh) } : {}),@@ -219,6 +238,8 @@ export function liftSoundFontPreset(font: SoundFont, index: number, dir: string)// Often a single turn of the wave, a few thousandths of a second: that is how a SoundFont holds a tone....(zone.loop && loopFrom >= 0 && loopTo > loopFrom + 0.0005 ? { loop: [Math.round(loopFrom * 1e5) / 1e5, Math.round(loopTo * 1e5) / 1e5] as const } : {}),...(layered && mid < 56 ? { dynamic: 'soft' as const } : layered && mid > 104 ? { dynamic: 'loud' as const } : {}),+ ...(gain < 0.995 ? { gain: Math.max(0.01, gain) } : {}),+ ...(dull ? { cutoff: Math.max(200, Math.round(zone.cutoff / 10) * 10) } : {}),};});diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.tsindex b3041963b..ed427ee1d 100644--- a/pipeline/tests/music.test.ts+++ b/pipeline/tests/music.test.ts@@ -329,7 +329,7 @@ describe('an instrument made of recordings', () => {expect(bad({ samples: [{ file: 'a.wav', note: 'C' }] })).toThrow(/note must be the note recorded, as "C4"/);expect(bad({ samples: [{ file: 'a.wav', note: 'C4', low: 'C5', high: 'C3' }] })).toThrow(/low is above high/);expect(bad({ samples: [{ file: 'a.wav', note: 'C4', loop: [0.5, 0.5] }] })).toThrow(/loop must be true, or \[from, to\]/);- expect(bad({ samples: [{ file: 'a.wav', note: 'C4', root: 'C4' }] })).toThrow(/unknown "root"; a sample has file, note, low, high, loop, dynamic, tune/);+ expect(bad({ samples: [{ file: 'a.wav', note: 'C4', root: 'C4' }] })).toThrow(/unknown "root"; a sample has file, note, low, high, loop, dynamic, tune, gain, cutoff/);expect(bad({ layers: [{ source: 'noise' }], attack: 0.1 })).toThrow(/attack is for samples/);});});@@ -598,3 +598,117 @@ describe('a line that is played and not only sounded', () => {expect(bad({ all: [] })).toThrow(/how loud each bar begins/);});});++describe('the whole, mixed and breathing', () => {+ const EIGHT = '1 - - - 1 - - - 1 - - - 1 - - -';+ /** How much of it lies at these pitches, from a quarter of a second in: the overtones of the note played, named one by one. */+ const energy = (c: Float32Array, rate: number, ...pitches: number[]): number => {+ let sum = 0;+ const [from, size] = [Math.round(rate * 0.25), 16384];+ for (const hz of pitches) {+ let [re, im] = [0, 0];+ for (let n = 0; n < size; n++) {+ const v = c[from + n]! * (0.5 - 0.5 * Math.cos((2 * Math.PI * n) / size));+ re += v * Math.cos((2 * Math.PI * hz * n) / rate); im += v * Math.sin((2 * Math.PI * hz * n) / rate);+ }+ sum += re * re + im * im;+ }+ return sum;+ };++ it('holds the loud moments down, so a quiet line comes up beside a loud one', () => {+ // One loud note, then quiet ones: what a compressor is for.+ const source = (compress?: number) => validateMusicSource({+ tempo: 120, key: 'A', humanize: 0, loop: false, stems: true, instruments: { tone, other: tone },+ patterns: { a: { bars: 2, dynamics: { tone: [1.6, 0.2, 0.2] }, tracks: { tone: `${EIGHT} | ${EIGHT}`, other: `1' - - - . . . . . . . . . . . . | ${REST}` } } },+ song: ['a'], ...(compress ? { master: { compress } } : {}),+ });+ const [plain, held] = [renderMusic(source()), renderMusic(source(0.8))];+ const quiet = (m: ReturnType<typeof renderMusic>): number => level(m.channels[0]!, m.rate, 3.1, 3.4) / m.peak;+ // By a third and more: 0.108 of full without it, 0.157 with.+ expect(quiet(held)).toBeGreaterThan(quiet(plain) * 1.3);+ // And the instruments written apart still add up to it.+ let worst = 0;+ for (let n = 0; n < held.channels[0]!.length; n += 5) worst = Math.max(worst, Math.abs(Object.values(held.stems!).reduce((s, stem) => s + stem[0]![n]!, 0) - held.channels[0]![n]!));+ expect(worst).toBeLessThan(1e-4);+ });++ it('takes the bottom off an instrument that has no need of it, and the top off one that sits behind', () => {+ const rich = { layers: [{ source: 'saw', freq: 110 }] };+ const play = (more: Record<string, unknown>) => renderMusic(validateMusicSource({+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5, instruments: { tone: { ...rich, ...more } }, patterns: { a: { bars: 1, tracks: { tone: EIGHT } } }, song: ['a'],+ })).channels[0]!;+ const [plain, thin, dull] = [play({}), play({ lowcut: 800 }), play({ highcut: 800 })];+ // After the peak of each is brought to the same place, where the energy lies is what differs.+ // The note is the A of 110 Hz: its first three overtones against four of those round 3 kHz.+ const bottom = (c: Float32Array): number => energy(c, 44100, 110, 220, 330) / energy(c, 44100, 2200, 2750, 3300, 3850);+ expect(bottom(thin)).toBeLessThan(bottom(plain) * 0.02);+ expect(bottom(dull)).toBeGreaterThan(bottom(plain) * 20);+ });++ it('tilts the whole with the two shelves', () => {+ const rich = { layers: [{ source: 'saw', freq: 55 }] };+ const play = (master?: unknown) => renderMusic(validateMusicSource({+ tempo: 120, key: 'A', humanize: 0, loop: false, instruments: { tone: rich }, patterns: { a: { bars: 1, tracks: { tone: EIGHT } } }, song: ['a'], ...(master ? { master } : {}),+ })).channels[0]!;+ // The note is the A of 55 Hz: its fundamental against four of its overtones round 10 kHz. Nine decibels is eight times the energy.+ const tilt = (c: Float32Array): number => energy(c, 44100, 9900, 10450, 11000, 11550) / energy(c, 44100, 55);+ const plain = tilt(play());+ expect(tilt(play({ high: 9 })) / plain).toBeGreaterThan(5);+ expect(tilt(play({ high: 9 })) / plain).toBeLessThan(10);+ expect(plain / tilt(play({ low: 9 }))).toBeGreaterThan(4);+ // Both down together leave the top where it was beside the bottom, near enough.+ expect(tilt(play({ low: -9, high: -9 })) / plain).toBeGreaterThan(0.5);+ expect(tilt(play({ low: -9, high: -9 })) / plain).toBeLessThan(2.5);+ // And the middle is not touched by either.+ const middle = (c: Float32Array): number => energy(c, 44100, 880, 990) / energy(c, 44100, 1100, 1210);+ expect(middle(play({ low: 9, high: 9 })) / middle(play())).toBeCloseTo(1, 1);+ });++ it('gives and takes back time across a pattern, every instrument together, and ends where it was', () => {+ const source = (rubato: number) => validateMusicSource({+ tempo: 120, key: 'A', humanize: 0, rubato, instruments: { tone, other: tone },+ patterns: { a: { bars: 4, tracks: { tone: Array(4).fill(EIGHT).join(' | '), other: Array(4).fill(EIGHT).join(' | ') } } }, song: ['a', 'a'],+ });+ const [strict, free] = [compileMusic(source(0)), compileMusic(source(1))];+ const of = (m: ReturnType<typeof compileMusic>, name: string): number[] => m.events.filter((e) => e.instrument === name).map((e) => e.at);+ const [a, b] = [of(strict, 'tone'), of(free, 'tone')];+ // Held back as the phrase begins, made up by its middle, held back again toward its end.+ expect(b[1]! - a[1]!).toBeGreaterThan(0.02);+ expect(Math.abs(b[8]! - a[8]!)).toBeLessThan(1e-9);+ expect(b[13]! - a[13]!).toBeLessThan(-0.02);+ // Never so far that the notes change order, and by no more than a twelfth of the tempo.+ for (let n = 1; n < b.length; n++) expect(b[n]! - b[n - 1]!).toBeGreaterThan(0.5 * 0.9);+ // The second time through begins on the bar line, where it would have without it.+ expect(b[16]).toBeCloseTo(8, 9);+ expect(free.seconds).toBe(strict.seconds);+ // The two instruments are together at every note.+ expect(of(free, 'other')).toEqual(b);+ // And the notes still fill the time between them.+ const first = free.events.filter((e) => e.instrument === 'tone')[0]!;+ expect(first.seconds).toBeCloseTo(b[1]! - b[0]!, 9);+ });++ it('plays a sample as loud and as dull as it says', () => {+ const bright: SampleData = { rate: 44100, channels: [Float32Array.from({ length: 22050 }, (_, n) => { const t = (2 * Math.PI * 440 * n) / 44100; return 0.3 * Math.sin(t) + 0.3 * Math.sin(8 * t); })] };+ const play = (more: Record<string, unknown>) => renderMusic(validateMusicSource({+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5, instruments: { keys: { samples: [{ file: 'a.wav', note: 'A4', ...more }] }, tone },+ patterns: { a: { bars: 1, tracks: { keys: '1 - - - . . . . . . . . . . . .', tone: '. . . . . . . . 1 - - - . . . .' } } }, song: ['a'],+ }), { 'a.wav': bright }).channels[0]!;+ // Beside a tone that is the same in both, half the gain is half as loud.+ const beside = (c: Float32Array): number => level(c, 44100, 0.1, 0.4) / level(c, 44100, 1.1, 1.4);+ expect(beside(play({ gain: 0.5 })) / beside(play({}))).toBeCloseTo(0.5, 1);+ const top = (c: Float32Array): number => energy(c, 44100, 3000, 4000) / energy(c, 44100, 400, 480);+ expect(top(play({ cutoff: 900 }))).toBeLessThan(top(play({})) * 0.05);+ });++ it('refuses a mix that is not one', () => {+ const bad = (more: Record<string, unknown>, instrument: Record<string, unknown> = {}) => () => validateMusicSource({ tempo: 120, key: 'A', instruments: { tone: { ...tone, ...instrument } }, patterns: { a: { bars: 1, tracks: { tone: EIGHT } } }, song: ['a'], ...more });+ expect(bad({ master: { compress: 2 } })).toThrow(/master\.compress must be 0\.\.1/);+ expect(bad({ master: { low: 20 } })).toThrow(/master\.low must be decibels, -12\.\.12/);+ expect(bad({ master: { limit: true } })).toThrow(/master has unknown "limit"; it takes compress, low, high/);+ expect(bad({ rubato: 3 })).toThrow(/rubato must be 0\.\.1/);+ expect(bad({}, { lowcut: 5 })).toThrow(/lowcut must be Hz, 20\.\.2000/);+ expect(bad({}, { highcut: 100 })).toThrow(/highcut must be Hz, 500\.\.20000/);+ });+});diff --git a/pipeline/tests/soundfont.test.ts b/pipeline/tests/soundfont.test.tsindex 28d26e557..8346cbb21 100644--- a/pipeline/tests/soundfont.test.ts+++ b/pipeline/tests/soundfont.test.ts@@ -24,7 +24,7 @@ const chunk = (id: string, data: Uint8Array | number[]): Uint8Array => bytesOf([const list = (type: string, ...chunks: Uint8Array[]): Uint8Array => chunk('LIST', bytesOf([...type].map((c) => c.charCodeAt(0)), ...chunks));const range = (low: number, high: number): number => low | (high << 8);-interface Recording { name: string; hz: number; rate: number; frames: number; pitch: number; correction?: number; type?: number; loop?: [number, number] }+interface Recording { name: string; hz: number; rate: number; frames: number; pitch: number; correction?: number; type?: number; link?: number; loop?: [number, number] }type Zone = [number, number][];/** A SoundFont of @p recordings, one instrument of @p zones (the first is what the others start from), and @p presets each a zone on that instrument. */@@ -36,7 +36,7 @@ function soundFont(recordings: Recording[], zones: Zone[], presets: { name: strifor (let n = 0; n < r.frames; n++) samples.push(...u16(Math.round(16000 * Math.sin((2 * Math.PI * r.hz * n) / r.rate)) & 0xffff));// The format asks for nothing after each recording.for (let n = 0; n < 46; n++) samples.push(0, 0);- headers.push(...name20(r.name), ...u32(start), ...u32(start + r.frames), ...u32(start + (r.loop?.[0] ?? 0)), ...u32(start + (r.loop?.[1] ?? 0)), ...u32(r.rate), r.pitch, (r.correction ?? 0) & 255, ...u16(0), ...u16(r.type ?? 1));+ headers.push(...name20(r.name), ...u32(start), ...u32(start + r.frames), ...u32(start + (r.loop?.[0] ?? 0)), ...u32(start + (r.loop?.[1] ?? 0)), ...u32(r.rate), r.pitch, (r.correction ?? 0) & 255, ...u16(r.link ?? 0), ...u16(r.type ?? 1));}headers.push(...name20('EOS'), ...Array(26).fill(0));const bags = (all: Zone[]): [number[], number[]] => {@@ -170,3 +170,41 @@ describe('what a SoundFont does that an instrument of samples does not', () => {expect(liftSoundFontPreset(readSoundFont(fifth), 0, 'o').instrument.samples).toEqual([{ file: 'o/Pipe.wav', note: 'D3' }]);});});++describe('what comes over with a recording', () => {+ it('takes both sides of a recording made in stereo as one file of two', () => {+ // The left side links to the right, which is as long; the right is a zone of its own in the SoundFont and is not taken twice.+ const pair: Recording[] = [{ name: 'Wide L', hz: 220, rate: 44100, frames: 4410, pitch: 57, type: 4, link: 1 }, { name: 'Wide R', hz: 330, rate: 44100, frames: 4410, pitch: 57, type: 2, link: 0 }];+ const font = soundFont(pair, [[[GEN.sampleId, 0]], [[GEN.sampleId, 1]]], [{ name: 'Wide', bank: 0, program: 0, zone: [[GEN.instrument, 0]] }]);+ const lifted = liftSoundFontPreset(readSoundFont(font), 0, 'w');+ expect(lifted.instrument.samples).toEqual([{ file: 'w/Wide_L.wav', note: 'A3' }]);+ const data = lifted.files['w/Wide_L.wav']!;+ expect(data.channels).toHaveLength(2);+ // Each side is its own recording: 220 on the left, 330 on the right.+ expect(data.channels[0]![50]).toBeCloseTo((16000 / 32768) * Math.sin((2 * Math.PI * 220 * 50) / 44100), 3);+ expect(data.channels[1]![50]).toBeCloseTo((16000 / 32768) * Math.sin((2 * Math.PI * 330 * 50) / 44100), 3);+ // A left side whose other side is not as long is taken alone.+ const odd = soundFont([pair[0]!, { ...pair[1]!, frames: 100 }], [[[GEN.sampleId, 0]]], [{ name: 'Odd', bank: 0, program: 0, zone: [[GEN.instrument, 0]] }]);+ expect(liftSoundFontPreset(readSoundFont(odd), 0, 'w').files['w/Wide_L.wav']!.channels).toHaveLength(1);+ });++ it('carries how far a zone turns its recording down and how dull it leaves it, without a file for each', () => {+ const one: Recording[] = [{ name: 'Key', hz: 220, rate: 44100, frames: 4410, pitch: 57 }];+ // 8 is the filter, in hundredths of a semitone above 8.176 Hz: 9600 is 2093 Hz. 48 is tenths of a decibel down. 11 opens the filter as the note sounds.+ const font = soundFont(one, [+ [[GEN.keyRange, range(0, 59)], [GEN.velRange, range(0, 63)], [8, 9600], [48, 100], [GEN.sampleId, 0]],+ [[GEN.keyRange, range(0, 59)], [GEN.velRange, range(64, 127)], [GEN.sampleId, 0]],+ [[GEN.keyRange, range(60, 127)], [8, 9600], [11, 2400], [GEN.sampleId, 0]],+ ], [{ name: 'Keys', bank: 0, program: 0, zone: [[GEN.instrument, 0]] }]);+ const lifted = liftSoundFontPreset(readSoundFont(font), 0, 'k');+ expect(lifted.instrument.samples).toEqual([+ // Ten decibels down, of which four are taken: 0.63. And dull, at 2090 Hz.+ { file: 'k/Key.wav', note: 'A3', low: 'C-1', high: 'B3', dynamic: 'soft', gain: 0.63, cutoff: 2090 },+ { file: 'k/Key.wav', note: 'A3', low: 'C-1', high: 'B3' },+ // A filter that opens two octaves is taken an octave open.+ { file: 'k/Key.wav', note: 'A3', low: 'C4', high: 'G9', cutoff: 4190 },+ ]);+ expect(Object.keys(lifted.files)).toEqual(['k/Key.wav']);+ expect(() => validateMusicSource({ tempo: 120, key: 'A', instruments: { keys: lifted.instrument }, patterns: { a: { bars: 1, tracks: { keys: '1 . . . . . . . . . . . . . . .' } } }, song: ['a'] })).not.toThrow();+ });+});diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex d26e6936d..18f1d79fb 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -4638,6 +4638,10 @@ export const NOTED = {{"note": "An AI can make background music."},"feat(music): the tool text says dynamics, legato and the pedal, and that a piece without them is notes and not playing":{"note": "An AI can make background music."},+ "feat(music): a piece is mixed, breathes, and takes all a SoundFont gives - master compress and shelves and an instrument's lowcut and highcut, rubato that gives and takes back time across a pattern, and from a SoundFont both sides of a stereo recording with each zone's level and filter as the sample's gain and cutoff":+ {"note": "An AI can make background music."},+ "feat(music): the tool text says how to mix a piece, when time should give, and how a part is written against another - a bass that moves, a second line, chords laid out by hand":+ {"note": "An AI can make background music."},"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":{"note": "A sky of your own shades the ground under its clouds."},"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":