Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(music): recordings played soft and loud that take turns, a recording as a layer of a sound, and an instrument out of a SoundFont - samples carry dynamic and tune and share a note in turn, a sound has a sample layer shaped as any other, and an .sf2 is read for its instruments and one lifted out as recordings with their notes, ranges, tuning and loops

yhh316d0296dd0b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 12481482f..b7789aead 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,8 +22,8 @@ published separately; it ships inside the editor.
- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.
- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.
- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.
-- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. The engine ships no recordings of its own: they are the project's.
-- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.
+- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.
+- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. A layer can also be a recording in the project (`source: "sample"`), played at a speed and shaped as any other layer is, for the one sound that will not come out of waves and noise. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.
- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.
- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.
- **An AI can lay leaflets along a stem in one step.** `scatter` along a `path` pointed every copy the way the path runs, turned at random, so the leaflets of a frond, the barbs of a feather or the legs of a centipede had to be written out one step each: a palm was 386 steps and a 749 KB model file. `rows` (1 or 2) stands the copies off the path instead, in a row or in pairs, `sweep` degrees from the way it runs and `lift` out of the plane, each as one number or as a start and an end. A copy's +z faces `up`, so a flat blade keeps its face to the sky whichever way it points. `sizes` shapes them along the path, widest in the middle if you say so. The same palm is 40 steps and 38 KB, and bakes to the same triangle count. A scatter without `rows` is as it was.
diff --git a/desktop b/desktop
index 4bdedda2d..5984d99ba 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 4bdedda2d622286c20ce44ce687b5a567d100818
+Subproject commit 5984d99ba081186a0b717e48b11a06dfca798acf
diff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.ts
index 312467c36..5de9d1f99 100644
--- a/pipeline/src/music/musicRender.ts
+++ b/pipeline/src/music/musicRender.ts
@@ -31,6 +31,8 @@ export interface MusicEvent {
part: number;
/** Written with # or b: outside the key. */
outside: boolean;
+ /** Marked softer (-1) or louder (1) than the rest; 0 for neither. */
+ mark: -1 | 0 | 1;
}
export interface CompiledMusic {
@@ -112,9 +114,10 @@ export function compileMusic(source: MusicSource): CompiledMusic {
const late = div % 2 === 0 && i % 2 === 1 ? (source.swing ?? 0) * step : 0;
const when = Math.max(0, at + i * step + late + human * 0.012 * random());
const loud = s.loud * (1 + human * 0.25 * random());
+ const mark = s.loud > 1 ? 1 : s.loud < 1 ? -1 : 0;
played.add(track);
if (s.kind === 'hit') {
- events.push({ instrument: track, at: when, seconds: held * step, midi: null, loud, part: p, outside: false });
+ events.push({ instrument: track, at: when, seconds: held * step, midi: null, loud, part: p, outside: false, mark });
continue;
}
const chord = instrument.follow && pattern.chords ? pattern.chords[Math.floor(i / stepsToBar) % pattern.chords.length]! : 1;
@@ -123,7 +126,7 @@ export function compileMusic(source: MusicSource): CompiledMusic {
for (const n of s.notes) {
const index = n.degree - 1 + chord - 1 + drop + 7;
const midi = 60 + tonic + scale[index % 7]! + 12 * (Math.floor(index / 7) - 1 + n.octave + (instrument.octave ?? 0)) + n.alter;
- events.push({ instrument: track, at: when, seconds: held * step, midi, loud, part: p, outside: n.alter !== 0 });
+ events.push({ instrument: track, at: when, seconds: held * step, midi, loud, part: p, outside: n.alter !== 0, mark });
}
}
}
@@ -180,17 +183,38 @@ function forNote(layer: SoundLayer, held: number, release: number): SoundLayer {
* to be lower, let go over the release. A recording an octave from its note is another instrument, which is why there
* are several; within three semitones the ear takes it for the same one.
*/
-function sampleNote(name: string, instrument: MusicInstrument, midi: number | null, held: number, rate: number, given: Readonly<Record<string, SampleData>>): Float32Array[] {
+const HARDNESS = { soft: -1, normal: 0, loud: 1 } as const;
+
+/**
+ * Which of an instrument's recordings plays a note. First by how hard: the recordings made as hard as the note is
+ * marked, or failing those the nearest in hardness. Then by pitch: the ones whose range holds the note, or failing
+ * those the nearest. And where several are left that are the same note, they take turns, so the same key struck
+ * four times is not one recording four times.
+ */
+export function sampleZone(instrument: MusicInstrument, midi: number | null, mark: number, turn: number): { index: number; matched: boolean } {
const zones = instrument.samples!;
- let zone = zones[0]!;
+ const hard = (i: number): number => HARDNESS[zones[i]!.dynamic ?? 'normal'];
+ let left = zones.map((_, i) => i);
+ const nearest = Math.min(...left.map((i) => Math.abs(hard(i) - mark)));
+ left = left.filter((i) => Math.abs(hard(i) - mark) === nearest);
+ const matched = nearest === 0 && mark !== 0;
if (midi !== null) {
- const ranged = zones.find((z) => (z.low !== undefined || z.high !== undefined) && midi >= (midiOf(z.low) ?? 0) && midi <= (midiOf(z.high) ?? 127));
- zone = ranged ?? zones.reduce((best, z) => (Math.abs(midiOf(z.note)! - midi) < Math.abs(midiOf(best.note)! - midi) ? z : best));
- }
+ const ranged = left.filter((i) => (zones[i]!.low !== undefined || zones[i]!.high !== undefined) && midi >= (midiOf(zones[i]!.low) ?? 0) && midi <= (midiOf(zones[i]!.high) ?? 127));
+ if (ranged.length) left = ranged;
+ const far = Math.min(...left.map((i) => Math.abs(midiOf(zones[i]!.note)! - midi)));
+ left = left.filter((i) => Math.abs(midiOf(zones[i]!.note)! - midi) === far);
+ // Of two as near, one above and one below, the turn must not swap them: only recordings of one note take turns.
+ left = left.filter((i) => midiOf(zones[i]!.note) === midiOf(zones[left[0]!]!.note));
+ } else left = [left[0]!];
+ return { index: left[turn % left.length]!, matched };
+}
+
+function sampleNote(name: string, instrument: MusicInstrument, index: number, midi: number | null, held: number, rate: number, given: Readonly<Record<string, SampleData>>): Float32Array[] {
+ const zone = instrument.samples![index]!;
const data = given[zone.file];
if (!data) throw new Error(`instrument "${name}": the recording ${zone.file} was not found`);
const frames = data.channels[0]!.length;
- const step = (midi === null ? 1 : Math.pow(2, (midi - midiOf(zone.note)!) / 12)) * (data.rate / rate);
+ const step = (midi === null ? 1 : Math.pow(2, (midi - midiOf(zone.note)! - (zone.tune ?? 0) / 100) / 12)) * (data.rate / rate);
const [attack, release] = [instrument.attack ?? 0.003, instrument.release ?? 0.08];
// Round its steady middle, or the seconds given, with the end of each turn faded into the start of the next.
let [from, to] = [0, 0];
@@ -297,12 +321,12 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,
const buses = new Map(names.map((name) => [name, [new Float32Array(work), new Float32Array(work)]] as const));
const cache = new Map<string, Float32Array[]>();
- const noteOf = (name: string, instrument: MusicInstrument, e: MusicEvent, turn: number): Float32Array[] => {
+ const noteOf = (name: string, instrument: MusicInstrument, e: MusicEvent, turn: number, zone: number): Float32Array[] => {
// The same note is made a few times over and no more: enough that a hat is not one sample struck again.
- const key = `${name}|${e.midi}|${Math.round(e.seconds * 1000)}|${turn % 3}`;
+ const key = `${name}|${e.midi}|${Math.round(e.seconds * 1000)}|${instrument.samples ? `z${zone}` : turn % 3}`;
let made = cache.get(key);
if (!made && instrument.samples) {
- made = sampleNote(name, instrument, e.midi, e.seconds, rate, given);
+ made = sampleNote(name, instrument, zone, e.midi, e.seconds, rate, given);
cache.set(key, made);
}
if (!made) {
@@ -320,11 +344,20 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,
const instrument = source.instruments[e.instrument]!;
const turn = turns.get(e.instrument) ?? 0;
turns.set(e.instrument, turn + 1);
- const note = noteOf(e.instrument, instrument, e, turn);
+ // Which recording, for an instrument of them: recordings of one note take turns, counted for each note apart.
+ let zone = { index: 0, matched: false };
+ if (instrument.samples) {
+ const of = `${e.instrument}|${e.midi}|${e.mark}`;
+ const again = turns.get(of) ?? 0;
+ turns.set(of, again + 1);
+ zone = sampleZone(instrument, e.midi, e.mark, again);
+ }
+ const note = noteOf(e.instrument, instrument, e, turn, zone.index);
const [l, r] = [note[0]!, note[1] ?? note[0]!];
const pan = instrument.pan ?? 0;
- // Equal power, and full in the middle.
- const loud = e.loud * (instrument.gain ?? 1) * Math.SQRT2;
+ // Equal power, and full in the middle. A recording made as hard as the note is marked is already that loud:
+ // the mark is not counted twice.
+ const loud = (zone.matched ? e.loud / (e.mark > 0 ? 1.3 : 0.6) : e.loud) * (instrument.gain ?? 1) * Math.SQRT2;
const [gl, gr] = [Math.cos(((pan + 1) * Math.PI) / 4) * loud, Math.sin(((pan + 1) * Math.PI) / 4) * loud];
const [left, right] = buses.get(e.instrument)!;
const from = Math.round(e.at * rate);
diff --git a/pipeline/src/music/musicSource.ts b/pipeline/src/music/musicSource.ts
index ab1d218c5..7a2330675 100644
--- a/pipeline/src/music/musicSource.ts
+++ b/pipeline/src/music/musicSource.ts
@@ -44,8 +44,18 @@ export interface MusicSample {
high?: string | number;
/** For a note held longer than the recording: true goes round its steady middle, [from, to] round those seconds of it. A piano dies of itself and has none. */
loop?: boolean | readonly [number, number];
+ /**
+ * How hard it was played. A step marked `?` is played from a "soft" recording and one marked `!` from a "loud"
+ * one, where the instrument has them: a piano struck hard is brighter, not only louder, and turning a soft
+ * recording up does not make it so. Left out, it is "normal".
+ */
+ dynamic?: (typeof DYNAMICS)[number];
+ /** Hundredths of a semitone the recording is sharp (above 0) or flat of its note, -100..100, which is taken out as it is played. */
+ tune?: number;
}
+export const DYNAMICS = ['soft', 'normal', 'loud'] as const;
+
export interface MusicInstrument {
/** The layers of its sound at A above middle C (440 Hz): a sine at 440 is its fundamental, one at 880 the octave over it. A layer with no `length` lasts as long as the note; one with a `length` (the knock at the front of a pluck, a drum) keeps its own. */
layers?: readonly SoundLayer[];
@@ -194,17 +204,19 @@ const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave
const RECORDING = /\.(wav|ogg|mp3|flac|m4a|aac|webm)$/i;
function checkSamples(samples: unknown, where: string): void {
- if (!Array.isArray(samples) || samples.length === 0 || samples.length > 64) throw new Error(`${where}: samples must be 1..64 of { file, note }`);
+ if (!Array.isArray(samples) || samples.length === 0 || samples.length > 256) throw new Error(`${where}: samples must be 1..256 of { file, note }`);
samples.forEach((sample: unknown, i) => {
const at = `${where}, sample ${i + 1}`;
if (!isObj(sample)) throw new Error(`${at} must be { file, note }`);
- for (const key of Object.keys(sample)) if (!['file', 'note', 'low', 'high', 'loop'].includes(key)) throw new Error(`${at} has unknown "${key}"; a sample has file, note, low, high, loop`);
+ for (const key of Object.keys(sample)) if (!['file', 'note', 'low', 'high', 'loop', 'dynamic', 'tune'].includes(key)) throw new Error(`${at} has unknown "${key}"; a sample has file, note, low, high, loop, dynamic, tune`);
+ if (sample.dynamic !== undefined && !DYNAMICS.includes(sample.dynamic as never)) throw new Error(`${at}: dynamic must be one of ${DYNAMICS.join(', ')}`);
+ if (sample.tune !== undefined && !between(sample.tune, -100, 100)) throw new Error(`${at}: tune must be hundredths of a semitone, -100..100`);
if (typeof sample.file !== 'string' || !RECORDING.test(sample.file)) throw new Error(`${at}: file must be a project path to a recording (.wav, .ogg, .mp3, .flac, .m4a)`);
if (midiOf(sample.note) === null) throw new Error(`${at}: note must be the note recorded, as "C4", "F#3" or "Bb2" (C4 is middle C), or its number 0..127`);
for (const key of ['low', 'high']) if (sample[key] !== undefined && midiOf(sample[key]) === null) throw new Error(`${at}: ${key} must be a note, as "C4"`);
if (sample.low !== undefined && sample.high !== undefined && midiOf(sample.low)! > midiOf(sample.high)!) throw new Error(`${at}: low is above high`);
const loop = sample.loop;
- if (loop !== undefined && typeof loop !== 'boolean' && !(Array.isArray(loop) && loop.length === 2 && isNum(loop[0]) && isNum(loop[1]) && loop[0] >= 0 && loop[1] > loop[0] + 0.02)) {
+ if (loop !== undefined && typeof loop !== 'boolean' && !(Array.isArray(loop) && loop.length === 2 && isNum(loop[0]) && isNum(loop[1]) && loop[0] >= 0 && loop[1] > loop[0] + 0.0005)) {
throw new Error(`${at}: loop must be true, or [from, to] in seconds with to past from`);
}
});
diff --git a/pipeline/src/music/soundFont.ts b/pipeline/src/music/soundFont.ts
new file mode 100644
index 000000000..76cf47822
--- /dev/null
+++ b/pipeline/src/music/soundFont.ts
@@ -0,0 +1,219 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file soundFont.ts — a SoundFont (`.sf2`) read for what a piece of music needs of it: which instruments it holds,
+ * and one of them taken out as recordings with the note, range, tuning and loop of each.
+ *
+ * A SoundFont is a bank of instruments in one file: every recording end to end in one block, and tables that
+ * say which stretch of it is which note of which instrument. Nothing here plays one. An instrument is lifted
+ * out as the `samples` a music source takes (musicSource.ts) and the recordings as samples to be written as
+ * `.wav` files, after which the SoundFont is not needed: the project holds what it uses and no more.
+ *
+ * What a SoundFont can say and this does not carry over: its filters and envelopes beyond attack and release,
+ * its modulators, and the second side of a recording made in stereo.
+ */
+import type { MusicSample } from './musicSource';
+import type { SampleData } from './musicRender';
+
+export interface SoundFontPreset {
+ /** Its place in the file, which is how it is asked for. */
+ index: number;
+ name: string;
+ /** The General MIDI numbers it answers to: bank 0 program 0 is a grand piano, bank 128 is drums. */
+ bank: number;
+ program: number;
+ /** How many recordings it plays, and the notes they cover between them. */
+ recordings: number;
+ low: number;
+ high: number;
+}
+
+interface Zone { sample: number; keyLow: number; keyHigh: number; velLow: number; velHigh: number; root: number; cents: number; loop: boolean; offsets: [number, number, number, number]; attack: number | null; release: number | null }
+interface SampleHeader { name: string; start: number; end: number; loopStart: number; loopEnd: number; rate: number; pitch: number; correction: number; type: number }
+
+export interface SoundFont {
+ name: string;
+ presets: SoundFontPreset[];
+ /** @internal */ zones: Zone[][];
+ /** @internal */ headers: SampleHeader[];
+ /** @internal */ data: Int16Array;
+}
+
+// The generators read here, by their numbers in the format.
+const GEN = { startOffset: 0, endOffset: 1, loopStartOffset: 2, loopEndOffset: 3, startCoarse: 4, endCoarse: 12, attack: 34, release: 38, instrument: 41, keyRange: 43, velRange: 44, loopStartCoarse: 45, loopEndCoarse: 50, coarseTune: 51, fineTune: 52, sampleId: 53, sampleModes: 54, rootKey: 58 } as const;
+
+type Generators = Map<number, number>;
+const signed = (v: number): number => (v > 32767 ? v - 65536 : v);
+
+/** The file read. Throws on what is not a SoundFont, saying what was found. */
+export function readSoundFont(bytes: Uint8Array): SoundFont {
+ const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+ const tag = (at: number): string => String.fromCharCode(bytes[at]!, bytes[at + 1]!, bytes[at + 2]!, bytes[at + 3]!);
+ const text = (at: number, length: number): string => { let s = ''; for (let i = 0; i < length && bytes[at + i]; i++) s += String.fromCharCode(bytes[at + i]!); return s.trim(); };
+ if (bytes.length < 12 || tag(0) !== 'RIFF' || tag(8) !== 'sfbk') throw new Error('not a SoundFont: it does not begin RIFF....sfbk');
+
+ const chunks = new Map<string, { at: number; size: number }>();
+ let name = '';
+ const walk = (from: number, to: number): void => {
+ for (let at = from; at + 8 <= to;) {
+ const size = view.getUint32(at + 4, true);
+ if (tag(at) === 'LIST') walk(at + 12, Math.min(to, at + 8 + size));
+ else {
+ chunks.set(tag(at), { at: at + 8, size: Math.min(size, bytes.length - at - 8) });
+ if (tag(at) === 'INAM') name = text(at + 8, size);
+ }
+ at += 8 + size + (size & 1);
+ }
+ };
+ walk(12, bytes.length);
+ for (const need of ['smpl', 'phdr', 'pbag', 'pgen', 'inst', 'ibag', 'igen', 'shdr']) if (!chunks.has(need)) throw new Error(`a SoundFont with no "${need}" table in it`);
+ const of = (id: string): { at: number; size: number } => chunks.get(id)!;
+
+ const smpl = of('smpl');
+ // The recordings: every one of them end to end, 16 bits a sample.
+ const data = new Int16Array(Math.floor(smpl.size / 2));
+ for (let i = 0; i < data.length; i++) data[i] = view.getInt16(smpl.at + i * 2, true);
+
+ const headers: SampleHeader[] = [];
+ for (let at = of('shdr').at; at + 46 <= of('shdr').at + of('shdr').size - 46; at += 46) {
+ headers.push({
+ name: text(at, 20), start: view.getUint32(at + 20, true), end: view.getUint32(at + 24, true), loopStart: view.getUint32(at + 28, true), loopEnd: view.getUint32(at + 32, true),
+ rate: view.getUint32(at + 36, true), pitch: bytes[at + 40]!, correction: view.getInt8(at + 41), type: view.getUint16(at + 44, true),
+ });
+ }
+
+ /** The generators of each zone of the thing whose bags run from @p first to @p last. */
+ const zonesOf = (bag: string, gen: string, first: number, last: number): Generators[] => {
+ const out: Generators[] = [];
+ for (let b = first; b < last; b++) {
+ const [from, to] = [view.getUint16(of(bag).at + b * 4, true), view.getUint16(of(bag).at + (b + 1) * 4, true)];
+ const zone: Generators = new Map();
+ for (let g = from; g < to; g++) zone.set(view.getUint16(of(gen).at + g * 4, true), view.getUint16(of(gen).at + g * 4 + 2, true));
+ out.push(zone);
+ }
+ return out;
+ };
+ const instruments = Math.floor(of('inst').size / 22) - 1;
+ const instrumentZones = (index: number): Generators[] => (index < 0 || index >= instruments ? []
+ : zonesOf('ibag', 'igen', view.getUint16(of('inst').at + index * 22 + 20, true), view.getUint16(of('inst').at + (index + 1) * 22 + 20, true)));
+ const range = (zone: Generators, id: number): [number, number] => { const v = zone.get(id); return v === undefined ? [0, 127] : [v & 255, v >> 8]; };
+ const seconds = (timecents: number | undefined): number | null => (timecents === undefined ? null : Math.pow(2, signed(timecents) / 1200));
+
+ const presets: SoundFontPreset[] = [];
+ const zones: Zone[][] = [];
+ const count = Math.floor(of('phdr').size / 38) - 1;
+ for (let p = 0; p < count; p++) {
+ const at = of('phdr').at + p * 38;
+ const mine: Zone[] = [];
+ let presetGlobal: Generators = new Map();
+ for (const pz of zonesOf('pbag', 'pgen', view.getUint16(at + 24, true), view.getUint16(at + 38 + 24, true))) {
+ // A zone that names no instrument is what the others start from.
+ if (!pz.has(GEN.instrument)) { presetGlobal = pz; continue; }
+ const [pk, pv] = [range(pz, GEN.keyRange), range(pz, GEN.velRange)];
+ const add = (id: number): number => signed(pz.get(id) ?? presetGlobal.get(id) ?? 0);
+ let global: Generators = new Map();
+ for (const iz of instrumentZones(pz.get(GEN.instrument)!)) {
+ if (!iz.has(GEN.sampleId)) { global = iz; continue; }
+ const get = (id: number): number | undefined => iz.get(id) ?? global.get(id);
+ const header = headers[iz.get(GEN.sampleId)!];
+ // The right-hand half of a stereo pair, and what lives in a sound card's ROM: left out.
+ if (!header || header.type === 2 || header.type & 0x8000) continue;
+ const [ik, iv] = [range(iz.has(GEN.keyRange) ? iz : global, GEN.keyRange), range(iz.has(GEN.velRange) ? iz : global, GEN.velRange)];
+ const [keyLow, keyHigh, velLow, velHigh] = [Math.max(ik[0], pk[0]), Math.min(ik[1], pk[1]), Math.max(iv[0], pv[0]), Math.min(iv[1], pv[1])];
+ if (keyLow > keyHigh || velLow > velHigh) continue;
+ const rootKey = get(GEN.rootKey);
+ mine.push({
+ sample: iz.get(GEN.sampleId)!, keyLow, keyHigh, velLow, velHigh,
+ root: rootKey !== undefined && signed(rootKey) >= 0 ? rootKey : header.pitch <= 127 ? header.pitch : 60,
+ cents: (signed(get(GEN.coarseTune) ?? 0) + add(GEN.coarseTune)) * 100 + signed(get(GEN.fineTune) ?? 0) + add(GEN.fineTune) + header.correction,
+ loop: ((get(GEN.sampleModes) ?? 0) & 1) === 1,
+ offsets: [
+ signed(get(GEN.startOffset) ?? 0) + signed(get(GEN.startCoarse) ?? 0) * 32768, signed(get(GEN.endOffset) ?? 0) + signed(get(GEN.endCoarse) ?? 0) * 32768,
+ signed(get(GEN.loopStartOffset) ?? 0) + signed(get(GEN.loopStartCoarse) ?? 0) * 32768, signed(get(GEN.loopEndOffset) ?? 0) + signed(get(GEN.loopEndCoarse) ?? 0) * 32768,
+ ],
+ attack: seconds(get(GEN.attack)), release: seconds(get(GEN.release)),
+ });
+ }
+ }
+ zones.push(mine);
+ presets.push({
+ index: p, name: text(at, 20), program: view.getUint16(at + 20, true), bank: view.getUint16(at + 22, true), recordings: new Set(mine.map((z) => z.sample)).size,
+ low: mine.length ? Math.min(...mine.map((z) => z.keyLow)) : 0, high: mine.length ? Math.max(...mine.map((z) => z.keyHigh)) : 0,
+ });
+ }
+ return { name, presets, zones, headers, data };
+}
+
+const NOTE_NAMES = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B'];
+const noteName = (midi: number): string => `${NOTE_NAMES[((midi % 12) + 12) % 12]}${Math.floor(midi / 12) - 1}`;
+
+export interface LiftedInstrument {
+ /** What a music source's instrument takes: `samples`, and how its notes come in and are let go where the SoundFont says. */
+ instrument: { samples: MusicSample[]; attack?: number; release?: number };
+ /** The recordings, by the path each sample names: to be written as `.wav` files there. */
+ files: Record<string, SampleData>;
+ notes: string[];
+}
+
+/**
+ * One instrument of a SoundFont as the `samples` of a music source, its recordings named under @p dir. Each keeps the
+ * note it is a recording of, the notes it is for, how far out of tune it is, where it goes round, and whether it was
+ * made soft or loud.
+ */
+export function liftSoundFontPreset(font: SoundFont, index: number, dir: string): LiftedInstrument {
+ const preset = font.presets[index];
+ if (!preset) throw new Error(`there is no instrument ${index} in this SoundFont; it has ${font.presets.length}, numbered from 0`);
+ const zones = font.zones[index]!;
+ if (zones.length === 0) throw new Error(`"${preset.name}" has no recordings that can be read`);
+ if (zones.length > 256) throw new Error(`"${preset.name}" is ${zones.length} recordings, and an instrument takes 256`);
+ const notes: string[] = [];
+ const files: Record<string, SampleData> = {};
+ const named = new Map<string, string>();
+ const taken = new Set<string>();
+ const folder = dir.replace(/\\/g, '/').replace(/\/+$/, '');
+ // Hard and soft only mean something where the instrument has more than one of them for a note.
+ const layered = zones.some((z) => z.velLow > 0 || z.velHigh < 127);
+ const ranged = zones.length > 1;
+
+ const samples = zones.map((zone): MusicSample => {
+ const header = font.headers[zone.sample]!;
+ const [from, to] = [Math.max(0, header.start + zone.offsets[0]), Math.min(font.data.length, header.end + zone.offsets[1])];
+ const key = `${zone.sample}|${from}|${to}`;
+ let file = named.get(key);
+ if (!file) {
+ const base = (header.name.replace(/[^A-Za-z0-9_-]+/g, '_').replace(/^_+|_+$/g, '') || `sample${zone.sample}`).slice(0, 40);
+ let leaf = base;
+ for (let n = 2; taken.has(leaf.toLowerCase()); n++) leaf = `${base}_${n}`;
+ taken.add(leaf.toLowerCase());
+ file = `${folder}/${leaf}.wav`;
+ named.set(key, file);
+ const channel = new Float32Array(Math.max(1, to - from));
+ for (let n = 0; n < channel.length; n++) channel[n] = font.data[from + n]! / 32768;
+ files[file] = { rate: header.rate || 44100, channels: [channel] };
+ }
+ // Played `cents` higher than its root says: the same as a recording of a note that much lower.
+ const exact = zone.root - zone.cents / 100;
+ const note = Math.max(0, Math.min(127, Math.round(exact)));
+ const tune = Math.round((exact - note) * 100);
+ const [loopFrom, loopTo] = [(header.loopStart + zone.offsets[2] - from) / (header.rate || 44100), (header.loopEnd + zone.offsets[3] - from) / (header.rate || 44100)];
+ const mid = (zone.velLow + zone.velHigh) / 2;
+ return {
+ file, note: noteName(note),
+ ...(ranged ? { low: noteName(zone.keyLow), high: noteName(zone.keyHigh) } : {}),
+ ...(tune !== 0 ? { tune: Math.max(-100, Math.min(100, tune)) } : {}),
+ // Often a single turn of the wave, a few thousandths of a second: that is how a SoundFont holds a tone.
+ ...(zone.loop && loopFrom >= 0 && loopTo > loopFrom + 0.0005 ? { loop: [Math.round(loopFrom * 1e5) / 1e5, Math.round(loopTo * 1e5) / 1e5] as const } : {}),
+ ...(layered && mid < 56 ? { dynamic: 'soft' as const } : layered && mid > 104 ? { dynamic: 'loud' as const } : {}),
+ };
+ });
+
+ const [attacks, releases] = [zones.map((z) => z.attack).filter((v): v is number => v !== null), zones.map((z) => z.release).filter((v): v is number => v !== null)];
+ const middle = (list: number[]): number => [...list].sort((a, b) => a - b)[Math.floor(list.length / 2)]!;
+ const attack = attacks.length ? Math.min(2, middle(attacks)) : 0;
+ const release = releases.length ? Math.max(0.05, Math.min(4, middle(releases))) : 0;
+ if (preset.bank === 128) notes.push(`"${preset.name}" is a drum kit: each key is another drum, so a step plays the drum on that note and not a pitch`);
+ return {
+ instrument: { samples, ...(attack > 0.01 ? { attack: Math.round(attack * 1000) / 1000 } : {}), ...(release > 0 ? { release: Math.round(release * 1000) / 1000 } : {}) },
+ files, notes,
+ };
+}
diff --git a/pipeline/src/sound/soundRender.ts b/pipeline/src/sound/soundRender.ts
index b84133c09..faad9f1b8 100644
--- a/pipeline/src/sound/soundRender.ts
+++ b/pipeline/src/sound/soundRender.ts
@@ -141,8 +141,27 @@ function noiseBody(layer: SoundLayer, count: number, rate: number, seed: number,
return out;
}
+/** A recording a layer plays: its samples and the rate they were made at. */
+export interface SoundRecording { rate: number; channels: readonly Float32Array[] }
+
+/** @p count samples of a recording, its two sides as one, played at @p speed from @p from seconds in. Past its end is silence. */
+function recordingBody(recording: SoundRecording, count: number, rate: number, speed: number, from: number): Float32Array {
+ const out = new Float32Array(count);
+ const [l, r] = [recording.channels[0]!, recording.channels[1] ?? recording.channels[0]!];
+ const step = (speed * recording.rate) / rate;
+ const start = from * recording.rate;
+ for (let n = 0; n < count; n++) {
+ const p = start + n * step;
+ const i = Math.floor(p);
+ if (i + 1 >= l.length) break;
+ const t = p - i;
+ out[n] = ((l[i]! + r[i]!) * (1 - t) + (l[i + 1]! + r[i + 1]!) * t) / 2;
+ }
+ return out;
+}
+
/** One striking of a layer: its samples from its own start, at @p pitch times its written pitch. */
-function strike(layer: SoundLayer, seconds: number, rate: number, pitch: number, seed: number): Float32Array {
+function strike(layer: SoundLayer, seconds: number, rate: number, pitch: number, seed: number, recording: SoundRecording | null = null): Float32Array {
const count = Math.max(1, Math.round(seconds * rate));
const out = new Float32Array(count);
const filters = (layer.filters ?? []).map((f) => ({ run: new Biquad(f.type, f.q ?? 0.7, rate, f.gain ?? 0), freq: f.freq }));
@@ -151,7 +170,9 @@ function strike(layer: SoundLayer, seconds: number, rate: number, pitch: number,
const drive = layer.drive ?? 0;
const push = 1 + drive * 14;
const pushed = Math.tanh(push);
- const noise = layer.source === 'noise' ? noiseBody(layer, count, rate, seed, pitch) : null;
+ // What is made ahead of the loop: a noise, or a recording played at its speed.
+ const noise = layer.source === 'noise' ? noiseBody(layer, count, rate, seed, pitch)
+ : layer.source === 'sample' ? recordingBody(recording!, count, rate, (layer.speed ?? 1) * pitch, layer.from ?? 0) : null;
// A spike: up in one sample and down over its width, as the air does in front of a bang.
const spike = layer.source === 'impulse' ? (layer.width ?? 0.0002) * rate : 0;
let [phase, bend] = [0, 0];
@@ -357,7 +378,7 @@ interface Stray { pitch: number; loud: number; decay: number; late: number; seed
* @p at is for a sound played as a note: `pitch` times its written pitch, with `seed` for other noise, and `raw`
* leaving its level as the layers sum to, so that many notes keep their balance against one another.
*/
-export function renderSound(source: SoundSource, variant = 0, at: { pitch?: number; seed?: number; raw?: boolean } = {}): RenderedSound {
+export function renderSound(source: SoundSource, variant = 0, at: { pitch?: number; seed?: number; raw?: boolean; given?: Readonly<Record<string, SoundRecording>> } = {}): RenderedSound {
const rate = source.rate ?? 44100;
const count = Math.max(1, Math.round(source.duration * rate));
// A loop is rendered a little long and its extra folded back over its start, so the end runs into the beginning.
@@ -406,7 +427,11 @@ export function renderSound(source: SoundSource, variant = 0, at: { pitch?: numb
return { peak, last, heard, strikes: strikes.length };
}
const space = total / rate - start;
- const seconds = Math.min(layer.length ?? space, space);
+ const recording = layer.source === 'sample' ? at.given?.[layer.file!] ?? null : null;
+ if (layer.source === 'sample' && !recording) throw new Error(`${where}: the recording ${layer.file} was not found`);
+ // A recording lasts as long as it lasts, at the speed it is played.
+ const whole = recording ? Math.max(0.001, (recording.channels[0]!.length / recording.rate - (layer.from ?? 0)) / ((layer.speed ?? 1) * place.pitch * stray.pitch)) : space;
+ const seconds = Math.min(layer.length ?? whole, space);
if (!(seconds > 0)) {
if (top) notes.push(`${where} starts at the end of the sound and is not heard`);
return { peak: 0, last: start, heard: 0, strikes: 0 };
@@ -428,7 +453,7 @@ export function renderSound(source: SoundSource, variant = 0, at: { pitch?: numb
const pitch = s.pitch * place.pitch * stray.pitch;
const loud = s.loud * place.loud * stray.loud;
// Each striking has its own noise, as no two footsteps are the same grit.
- const hit = strike(made, length, rate, pitch, s.seed);
+ const hit = strike(made, length, rate, pitch, s.seed, recording);
// The other ear's noise is another noise, by as much as the layer is wide.
const other = wide > 0 ? strike(made, length, rate, pitch, s.seed + 50021) : null;
// Equal power: a sound sent to one side is as loud as one in the middle.
@@ -542,8 +567,8 @@ export function encodeWav(sound: Pick<RenderedSound, 'rate' | 'channels'>): Uint
}
/** Every variant a source asks for, the first being the sound as written. */
-export function renderSoundVariants(source: SoundSource): RenderedSound[] {
- return Array.from({ length: source.variants ?? 1 }, (_, k) => renderSound(source, k));
+export function renderSoundVariants(source: SoundSource, given: Readonly<Record<string, SoundRecording>> = {}): RenderedSound[] {
+ return Array.from({ length: source.variants ?? 1 }, (_, k) => renderSound(source, k, { given }));
}
/** A recording as this file's sounds are held, so it can be drawn and measured as they are. */
diff --git a/pipeline/src/sound/soundSource.ts b/pipeline/src/sound/soundSource.ts
index 17a2da3c4..bc01c453f 100644
--- a/pipeline/src/sound/soundSource.ts
+++ b/pipeline/src/sound/soundSource.ts
@@ -20,7 +20,7 @@ export const SOUND_RATES = [22050, 44100, 48000] as const;
/** One value for the whole of a layer, or [[seconds, value], ...] from the layer's start: a pitch that falls, a filter that opens. */
export type SoundCurve = number | readonly (readonly [number, number])[];
-export const SOUND_SOURCES = ['sine', 'triangle', 'saw', 'square', 'noise', 'modes', 'impulse', 'group'] as const;
+export const SOUND_SOURCES = ['sine', 'triangle', 'saw', 'square', 'noise', 'modes', 'impulse', 'sample', 'group'] as const;
export const MAX_SOUND_VARIANTS = 8;
export type SoundSourceKind = (typeof SOUND_SOURCES)[number];
export const SOUND_FILTERS = ['lowpass', 'highpass', 'bandpass', 'notch', 'peak'] as const;
@@ -60,6 +60,16 @@ export interface SoundLayer {
wide?: number;
/** noise: white is hiss, pink is rain, brown is a rumble. Default white. */
color?: (typeof NOISE_COLORS)[number];
+ /**
+ * sample: a recording in the project, by its path, played as a layer among the made ones: the real crack of a
+ * branch under made leaves, a voice, the one sound that will not come out of waves and noise. Everything a layer
+ * has shapes it as it shapes the rest: an envelope, filters, a drive, repeats, a scatter.
+ */
+ file?: string;
+ /** sample: how fast it is played, 0.25..4; 2 is an octave up and half as long. Default 1. */
+ speed?: number;
+ /** sample: seconds into the recording that it starts from. Default 0. */
+ from?: number;
/** modes: each [Hz, loudness, seconds to die away], the tones a struck thing rings with. */
modes?: readonly (readonly [number, number, number])[];
seed?: number;
@@ -154,7 +164,7 @@ function curveProblem(v: unknown, lo: number, hi: number, what: string): string
return null;
}
-const LAYER_FIELDS = ['id', 'source', 'freq', 'width', 'band', 'wide', 'color', 'modes', 'seed', 'start', 'length', 'gain', 'attack', 'release', 'decay', 'tail', 'vibrato', 'tremolo', 'fm', 'filters', 'drive', 'pan', 'repeat', 'scatter', 'mute'];
+const LAYER_FIELDS = ['id', 'source', 'freq', 'width', 'band', 'wide', 'color', 'modes', 'file', 'speed', 'from', 'seed', 'start', 'length', 'gain', 'attack', 'release', 'decay', 'tail', 'vibrato', 'tremolo', 'fm', 'filters', 'drive', 'pan', 'repeat', 'scatter', 'mute'];
const GROUP_FIELDS = ['id', 'source', 'layers', 'start', 'gain', 'pan', 'repeat', 'scatter', 'mute'];
@@ -174,7 +184,12 @@ function checkLayer(l: Record<string, unknown>, where: string, ids: Set<string>,
ids.add(l.id as string);
}
if (!SOUND_SOURCES.includes(l.source as SoundSourceKind)) fail(`source ${JSON.stringify(l.source)} must be one of ${SOUND_SOURCES.join(', ')}`);
- const tone = l.source !== 'noise' && l.source !== 'modes' && l.source !== 'impulse' && l.source !== 'group';
+ const tone = l.source !== 'noise' && l.source !== 'modes' && l.source !== 'impulse' && l.source !== 'group' && l.source !== 'sample';
+ if (l.source === 'sample') {
+ if (typeof l.file !== 'string' || !/\.(wav|ogg|mp3|flac|m4a|aac|webm)$/i.test(l.file)) fail('a sample needs file: a project path to a recording (.wav, .ogg, .mp3, .flac, .m4a)');
+ if (l.speed !== undefined && !between(l.speed, 0.25, 4)) fail('speed must be 0.25..4');
+ if (l.from !== undefined && !between(l.from, 0, 600)) fail('from must be seconds into the recording, 0 or more');
+ } else for (const key of ['file', 'speed', 'from']) if (l[key] !== undefined) fail(`${key} goes with source "sample"`);
if (tone) {
if (l.freq === undefined) fail(`a ${String(l.source)} needs freq, its pitch in Hz`);
const bad = curveProblem(l.freq, 10, 20000, 'freq');
@@ -272,6 +287,14 @@ function checkLayer(l: Record<string, unknown>, where: string, ids: Set<string>,
const SOURCE_FIELDS = ['version', 'duration', 'rate', 'loop', 'variants', 'vary', 'reference', 'layers', 'highpass', 'lowpass', 'echo', 'reverb', 'gain', 'normalize'];
+/** The recordings a sound's layers play, each once. */
+export function soundSampleFiles(source: Pick<SoundSource, 'layers'>): string[] {
+ const files = new Set<string>();
+ const walk = (layers: readonly SoundLayer[]): void => { for (const l of layers) { if (l.source === 'sample' && l.file) files.add(l.file); if (l.layers) walk(l.layers); } };
+ walk(source.layers);
+ return [...files];
+}
+
/** The source if it is well formed; otherwise throws, naming the layer and the field. */
export function validateSoundSource(raw: unknown): SoundSource {
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) throw new Error('a sound source must be { duration, layers }');
diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.ts
index a631f7998..ac58f7c3d 100644
--- a/pipeline/tests/music.test.ts
+++ b/pipeline/tests/music.test.ts
@@ -6,7 +6,7 @@
*/
import { describe, expect, it } from 'vitest';
import { applyMusicEdits, midiOf, musicSampleFiles, validateMusicSource, type MusicSource } from '../src/music/musicSource';
-import { compileMusic, noteName, renderMusic, samplePitch, type SampleData } from '../src/music/musicRender';
+import { compileMusic, noteName, renderMusic, samplePitch, sampleZone, type SampleData } from '../src/music/musicRender';
import { musicPicture } from '../src/music/musicPicture';
const tone = { layers: [{ source: 'sine', freq: 440 }] } as const;
@@ -321,12 +321,12 @@ describe('an instrument made of recordings', () => {
const bad = (instrument: unknown) => () => validateMusicSource({ tempo: 120, key: 'A', instruments: { keys: instrument }, patterns: { a: { bars: 1, tracks: { keys: one('1') } } }, song: ['a'] });
expect(bad({})).toThrow(/instrument "keys" needs layers \(a made sound\) or samples \(recordings\), one of the two/);
expect(bad({ layers: [{ source: 'noise' }], samples: [{ file: 'a.wav', note: 'C4' }] })).toThrow(/one of the two/);
- expect(bad({ samples: [] })).toThrow(/samples must be 1\.\.64/);
+ expect(bad({ samples: [] })).toThrow(/samples must be 1\.\.256/);
expect(bad({ samples: [{ file: 'a.txt', note: 'C4' }] })).toThrow(/sample 1: file must be a project path to a recording/);
expect(bad({ samples: [{ file: 'a.wav', note: 'C' }] })).toThrow(/note must be the note recorded, as "C4"/);
expect(bad({ samples: [{ file: 'a.wav', note: 'C4', low: 'C5', high: 'C3' }] })).toThrow(/low is above high/);
expect(bad({ samples: [{ file: 'a.wav', note: 'C4', loop: [0.5, 0.5] }] })).toThrow(/loop must be true, or \[from, to\]/);
- expect(bad({ samples: [{ file: 'a.wav', note: 'C4', root: 'C4' }] })).toThrow(/unknown "root"; a sample has file, note, low, high, loop/);
+ expect(bad({ samples: [{ file: 'a.wav', note: 'C4', root: 'C4' }] })).toThrow(/unknown "root"; a sample has file, note, low, high, loop, dynamic, tune/);
expect(bad({ layers: [{ source: 'noise' }], attack: 0.1 })).toThrow(/attack is for samples/);
});
});
@@ -388,3 +388,58 @@ describe('how a piece begins, ends and meets itself', () => {
expect(renderMusic(piece(tracks, { instruments, song: [thin, 'a'] })).join).toBeNull();
});
});
+
+describe('recordings played soft and loud, and taking turns', () => {
+ const recorded = (hz: number, loud = 0.5): SampleData => ({ rate: 44100, channels: [Float32Array.from({ length: 22050 }, (_, n) => loud * Math.sin((2 * Math.PI * hz * n) / 44100))] });
+ // Each recording is of another pitch, so which one played is heard in what comes out.
+ const given = { 'n.wav': recorded(440), 's.wav': recorded(330, 0.2), 'l.wav': recorded(660), 'n2.wav': recorded(550), 'n3.wav': recorded(770), 'sharp.wav': recorded(440 * Math.pow(2, 50 / 1200)) };
+ const play = (samples: unknown[], line: string) => renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5,
+ instruments: { keys: { samples } }, patterns: { a: { bars: 1, tracks: { keys: line } } }, song: ['a'],
+ }), given);
+ const FOUR = (a: string, b: string, c: string, d: string): string => `${a} - - . ${b} - - . ${c} - - . ${d} - - .`;
+ const heard = (m: ReturnType<typeof play>, beat: number): number => Math.round(pitchOf(m.channels[0]!, 44100, beat * 0.5 + 0.05, beat * 0.5 + 0.3) / 10) * 10;
+
+ it('plays a step marked soft or loud from a recording made that way', () => {
+ const zones = [{ file: 'n.wav', note: 'A4' }, { file: 's.wav', note: 'A4', dynamic: 'soft' }, { file: 'l.wav', note: 'A4', dynamic: 'loud' }];
+ const m = play(zones, FOUR('1', '1?', '1!', '1'));
+ expect([0, 1, 2, 3].map((b) => heard(m, b))).toEqual([440, 330, 660, 440]);
+ expect(sampleZone({ samples: zones } as never, 69, -1, 0)).toEqual({ index: 1, matched: true });
+ expect(sampleZone({ samples: zones } as never, 69, 0, 0)).toEqual({ index: 0, matched: false });
+ });
+
+ it('does not count the mark twice when the recording is already that hard', () => {
+ // The soft recording is 0.2 where the normal one is 0.5: played from it, a soft step is 0.2, not 0.2 times 0.6.
+ const both = play([{ file: 'n.wav', note: 'A4' }, { file: 's.wav', note: 'A4', dynamic: 'soft' }], FOUR('1', '1?', '.', '.')).channels[0]!;
+ expect(level(both, 44100, 0.55, 0.8) / level(both, 44100, 0.05, 0.3)).toBeCloseTo(0.4, 1);
+ // With no soft recording the normal one is turned down.
+ const one = play([{ file: 'n.wav', note: 'A4' }], FOUR('1', '1?', '.', '.')).channels[0]!;
+ expect(level(one, 44100, 0.55, 0.8) / level(one, 44100, 0.05, 0.3)).toBeCloseTo(0.6, 1);
+ });
+
+ it('falls back on the nearest in hardness where there is none as hard', () => {
+ // No loud recording: a loud step takes the normal one, never the soft.
+ const m = play([{ file: 's.wav', note: 'A4', dynamic: 'soft' }, { file: 'n.wav', note: 'A4' }], FOUR('1!', '1?', '1', '.'));
+ expect([0, 1, 2].map((b) => heard(m, b))).toEqual([440, 330, 440]);
+ });
+
+ it('takes turns between recordings of one note, counted for each note apart', () => {
+ const zones = [{ file: 'n.wav', note: 'A4' }, { file: 'n2.wav', note: 'A4' }, { file: 'n3.wav', note: 'A4' }];
+ const m = play(zones, FOUR('1', '1', '1', '1'));
+ expect([0, 1, 2, 3].map((b) => heard(m, b))).toEqual([440, 550, 770, 440]);
+ // Another note between does not use up a turn of this one.
+ const mixed = play(zones, FOUR('1', "1'", '1', '1'));
+ expect([0, 2, 3].map((b) => heard(mixed, b))).toEqual([440, 550, 770]);
+ // Two recordings as near, one below and one above, are not turns of each other.
+ const beside = [{ file: 'n.wav', note: 'G4' }, { file: 'n2.wav', note: 'B4' }];
+ expect([0, 1, 2].map((t) => sampleZone({ samples: beside } as never, 69, 0, t).index)).toEqual([0, 0, 0]);
+ });
+
+ it('takes out how sharp a recording is said to be', () => {
+ // Half a semitone sharp, and said so: it plays in tune. Unsaid, it is heard sharp.
+ const tuned = play([{ file: 'sharp.wav', note: 'A4', tune: 50 }], '1 - - - - - - - . . . . . . . .').channels[0]!;
+ const raw = play([{ file: 'sharp.wav', note: 'A4' }], '1 - - - - - - - . . . . . . . .').channels[0]!;
+ expect(pitchOf(tuned, 44100, 0.05, 0.45)).toBeCloseTo(440, -0.5);
+ expect(pitchOf(raw, 44100, 0.05, 0.45)).toBeGreaterThan(449);
+ });
+});
diff --git a/pipeline/tests/sound.test.ts b/pipeline/tests/sound.test.ts
index 812130b00..a381a0365 100644
--- a/pipeline/tests/sound.test.ts
+++ b/pipeline/tests/sound.test.ts
@@ -5,7 +5,7 @@
* is heard, and that the same source is the same samples.
*/
import { describe, expect, it } from 'vitest';
-import { applySoundEdits, validateSoundSource, type SoundLayer, type SoundSource } from '../src/sound/soundSource';
+import { applySoundEdits, soundSampleFiles, validateSoundSource, type SoundLayer, type SoundSource } from '../src/sound/soundSource';
import { decodeWav, encodeWav, renderSound, renderSoundVariants } from '../src/sound/soundRender';
import { comparePicture, soundBands, soundPicture } from '../src/sound/soundPicture';
@@ -449,3 +449,58 @@ describe('layers struck together, sounds that differ, and a recording to be like
expect(lit(300, 384 + 6 + 10 + 60)).toBe(false);
});
});
+
+describe('a recording as a layer among the made ones', () => {
+ const tone = (hz: number, seconds: number, rate = 44100) => ({ rate, channels: [Float32Array.from({ length: Math.round(seconds * rate) }, (_, n) => 0.5 * Math.sin((2 * Math.PI * hz * n) / rate))] });
+ const given = { 'a.wav': tone(440, 0.3), 'slow.wav': tone(440, 0.3, 22050) };
+ const play = (layers: SoundLayer[], rest: Partial<SoundSource> = {}) => renderSound(validateSoundSource({ duration: 1, normalize: false, layers, ...rest }), 0, { given });
+
+ it('plays the recording where the layer starts, for as long as it lasts', () => {
+ const s = play([{ source: 'sample', file: 'a.wav', start: 0.2 }]).channels[0]!;
+ expect(level(s, 0, 0.19)).toBeLessThan(1e-6);
+ expect(pitchOf(s, 44100, 0.22, 0.48)).toBeCloseTo(440, -1);
+ expect(level(s, 0.22, 0.48)).toBeCloseTo(0.5 / Math.SQRT2, 1);
+ expect(level(s, 0.52, 1)).toBeLessThan(1e-6);
+ // Whatever rate it was recorded at.
+ expect(pitchOf(play([{ source: 'sample', file: 'slow.wav' }]).channels[0]!, 44100, 0.02, 0.28)).toBeCloseTo(440, -1);
+ });
+
+ it('is higher and shorter played faster, and starts from where it is told', () => {
+ const fast = play([{ source: 'sample', file: 'a.wav', speed: 2 }]).channels[0]!;
+ expect(pitchOf(fast, 44100, 0.02, 0.13)).toBeCloseTo(880, -1.3);
+ expect(level(fast, 0.17, 1)).toBeLessThan(1e-6);
+ const late = play([{ source: 'sample', file: 'a.wav', from: 0.2 }]).channels[0]!;
+ expect(level(late, 0.01, 0.09)).toBeGreaterThan(0.2);
+ expect(level(late, 0.12, 1)).toBeLessThan(1e-6);
+ });
+
+ it('is shaped as any layer is: an envelope, a filter, repeats that climb', () => {
+ const s = play([{ source: 'sample', file: 'a.wav', decay: 0.03, filters: [{ type: 'lowpass', freq: 200 }], repeat: { count: 3, every: 0.3, pitch: 2 } }]);
+ const c = s.channels[0]!;
+ // Dying away, and quieter for the filter below its pitch.
+ expect(level(c, 0.15, 0.25)).toBeLessThan(level(c, 0, 0.05) * 0.2);
+ expect(level(c, 0, 0.05)).toBeLessThan(0.2);
+ expect(s.layers[0]).toMatchObject({ source: 'sample', strikes: 3 });
+ const plain = play([{ source: 'sample', file: 'a.wav', repeat: { count: 2, every: 0.4, pitch: 2 } }]).channels[0]!;
+ expect(pitchOf(plain, 44100, 0.42, 0.53)).toBeCloseTo(880, -1.3);
+ });
+
+ it('lies among made layers, in a group, and in variants that differ', () => {
+ const source = validateSoundSource({ duration: 0.6, variants: 3, layers: [{ source: 'group', layers: [{ source: 'sample', file: 'a.wav' }, { source: 'noise', band: [2000, 8000], length: 0.05, decay: 0.01, gain: 0.3 }] }] });
+ expect(soundSampleFiles(source)).toEqual(['a.wav']);
+ const all = renderSoundVariants(source, given);
+ expect(all).toHaveLength(3);
+ expect(Buffer.from(all[1]!.channels[0]!.buffer).equals(Buffer.from(all[0]!.channels[0]!.buffer))).toBe(false);
+ expect(Math.abs(pitchOf(all[1]!.channels[0]!, 44100, 0.1, 0.4) - 440)).toBeLessThan(40);
+ });
+
+ it('refuses what a sample cannot be, and says when its recording is not there', () => {
+ const bad = (layer: unknown) => () => validateSoundSource({ duration: 1, layers: [layer] });
+ expect(bad({ source: 'sample' })).toThrow(/a sample needs file: a project path to a recording/);
+ expect(bad({ source: 'sample', file: 'a.txt' })).toThrow(/a sample needs file/);
+ expect(bad({ source: 'sample', file: 'a.wav', speed: 9 })).toThrow(/speed must be 0\.25\.\.4/);
+ expect(bad({ source: 'sample', file: 'a.wav', freq: 440 })).toThrow(/freq is a tone's pitch; a sample has none/);
+ expect(bad({ source: 'noise', file: 'a.wav' })).toThrow(/file goes with source "sample"/);
+ expect(() => renderSound(validateSoundSource({ duration: 1, layers: [{ id: 'crack', source: 'sample', file: 'b.wav' }] }))).toThrow(/layer 0 "crack": the recording b\.wav was not found/);
+ });
+});
diff --git a/pipeline/tests/soundfont.test.ts b/pipeline/tests/soundfont.test.ts
new file mode 100644
index 000000000..6803c97f5
--- /dev/null
+++ b/pipeline/tests/soundfont.test.ts
@@ -0,0 +1,141 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * A SoundFont read for the instruments in it, held to a file built here byte by byte: what comes out has to be the
+ * recordings that went in, on the notes, ranges, tuning and loops its tables say, and has to play at those pitches.
+ */
+import { describe, expect, it } from 'vitest';
+import { liftSoundFontPreset, readSoundFont } from '../src/music/soundFont';
+import { validateMusicSource } from '../src/music/musicSource';
+import { renderMusic } from '../src/music/musicRender';
+
+const GEN = { attack: 34, release: 38, instrument: 41, keyRange: 43, velRange: 44, fineTune: 52, sampleId: 53, sampleModes: 54, rootKey: 58 };
+
+function bytesOf(...parts: (Uint8Array | number[])[]): Uint8Array {
+ const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0));
+ let at = 0;
+ for (const p of parts) { out.set(p, at); at += p.length; }
+ return out;
+}
+const u16 = (v: number): number[] => [v & 255, (v >> 8) & 255];
+const u32 = (v: number): number[] => [v & 255, (v >> 8) & 255, (v >> 16) & 255, (v >>> 24) & 255];
+const name20 = (text: string): number[] => Array.from({ length: 20 }, (_, i) => text.charCodeAt(i) || 0);
+const chunk = (id: string, data: Uint8Array | number[]): Uint8Array => bytesOf([...id].map((c) => c.charCodeAt(0)), u32(data.length), data, data.length & 1 ? [0] : []);
+const list = (type: string, ...chunks: Uint8Array[]): Uint8Array => chunk('LIST', bytesOf([...type].map((c) => c.charCodeAt(0)), ...chunks));
+const range = (low: number, high: number): number => low | (high << 8);
+
+interface Recording { name: string; hz: number; rate: number; frames: number; pitch: number; correction?: number; type?: number; loop?: [number, number] }
+type Zone = [number, number][];
+
+/** A SoundFont of @p recordings, one instrument of @p zones (the first is what the others start from), and @p presets each a zone on that instrument. */
+function soundFont(recordings: Recording[], zones: Zone[], presets: { name: string; bank: number; program: number; zone: Zone }[]): Uint8Array {
+ const samples: number[] = [];
+ const headers: number[] = [];
+ for (const r of recordings) {
+ const start = samples.length / 2;
+ for (let n = 0; n < r.frames; n++) samples.push(...u16(Math.round(16000 * Math.sin((2 * Math.PI * r.hz * n) / r.rate)) & 0xffff));
+ // The format asks for nothing after each recording.
+ for (let n = 0; n < 46; n++) samples.push(0, 0);
+ headers.push(...name20(r.name), ...u32(start), ...u32(start + r.frames), ...u32(start + (r.loop?.[0] ?? 0)), ...u32(start + (r.loop?.[1] ?? 0)), ...u32(r.rate), r.pitch, (r.correction ?? 0) & 255, ...u16(0), ...u16(r.type ?? 1));
+ }
+ headers.push(...name20('EOS'), ...Array(26).fill(0));
+ const bags = (all: Zone[]): [number[], number[]] => {
+ const [bag, gen]: [number[], number[]] = [[], []];
+ let at = 0;
+ for (const zone of all) { bag.push(...u16(at), ...u16(0)); for (const [op, amount] of zone) { gen.push(...u16(op), ...u16(amount & 0xffff)); at++; } }
+ bag.push(...u16(at), ...u16(0));
+ gen.push(0, 0, 0, 0);
+ return [bag, gen];
+ };
+ const [ibag, igen] = bags(zones);
+ const [pbag, pgen] = bags(presets.map((p) => p.zone));
+ const phdr = [...presets.flatMap((p, i) => [...name20(p.name), ...u16(p.program), ...u16(p.bank), ...u16(i), ...Array(12).fill(0)]), ...name20('EOP'), ...u16(0), ...u16(0), ...u16(presets.length), ...Array(12).fill(0)];
+ const inst = [...name20('Keys'), ...u16(0), ...name20('EOI'), ...u16(zones.length)];
+ return chunk('RIFF', bytesOf([...'sfbk'].map((c) => c.charCodeAt(0)),
+ list('INFO', chunk('INAM', [...'Test Bank'].map((c) => c.charCodeAt(0)))),
+ list('sdta', chunk('smpl', samples)),
+ list('pdta', chunk('phdr', phdr), chunk('pbag', pbag), chunk('pmod', Array(10).fill(0)), chunk('pgen', pgen), chunk('inst', inst), chunk('ibag', ibag), chunk('imod', Array(10).fill(0)), chunk('igen', igen), chunk('shdr', headers))));
+}
+
+const RECORDINGS: Recording[] = [
+ { name: 'Low A', hz: 220, rate: 22050, frames: 5512, pitch: 57 },
+ { name: 'High A!', hz: 440, rate: 44100, frames: 13230, pitch: 60, loop: [4410, 8820] },
+ { name: 'High A R', hz: 440, rate: 44100, frames: 100, pitch: 69, type: 2 },
+];
+const ZONES: Zone[] = [
+ // What the others start from: let go over half a second.
+ [[GEN.release, -1200]],
+ [[GEN.keyRange, range(0, 63)], [GEN.sampleId, 0]],
+ // Its header says C4; the zone says A4, a quarter of a semitone flat, and that it goes round.
+ [[GEN.keyRange, range(64, 127)], [GEN.rootKey, 69], [GEN.fineTune, 25], [GEN.sampleModes, 1], [GEN.sampleId, 1]],
+ [[GEN.keyRange, range(64, 127)], [GEN.sampleId, 2]],
+];
+const FONT = soundFont(RECORDINGS, ZONES, [
+ { name: 'Test Keys', bank: 0, program: 4, zone: [[GEN.instrument, 0]] },
+ { name: 'Top Only', bank: 0, program: 5, zone: [[GEN.keyRange, range(72, 96)], [GEN.instrument, 0]] },
+]);
+
+describe('a SoundFont read for its instruments', () => {
+ it('lists the instruments in it, with their numbers and the notes they cover', () => {
+ const font = readSoundFont(FONT);
+ expect(font.name).toBe('Test Bank');
+ expect(font.presets).toEqual([
+ { index: 0, name: 'Test Keys', bank: 0, program: 4, recordings: 2, low: 0, high: 127 },
+ { index: 1, name: 'Top Only', bank: 0, program: 5, recordings: 1, low: 72, high: 96 },
+ ]);
+ });
+
+ it('lifts one out as samples: each recording with its note, range, tuning and loop', () => {
+ const lifted = liftSoundFontPreset(readSoundFont(FONT), 0, 'assets/samples/keys/');
+ expect(lifted.instrument).toEqual({
+ release: 0.5,
+ samples: [
+ { file: 'assets/samples/keys/Low_A.wav', note: 'A3', low: 'C-1', high: 'D#4' },
+ // A quarter of a semitone is added as it plays, which is a recording that much flat of A4.
+ { file: 'assets/samples/keys/High_A.wav', note: 'A4', low: 'E4', high: 'G9', tune: -25, loop: [0.1, 0.2] },
+ ],
+ });
+ // The recordings themselves, as long and at the rate they were, and the right-hand half left out.
+ expect(Object.keys(lifted.files)).toEqual(['assets/samples/keys/Low_A.wav', 'assets/samples/keys/High_A.wav']);
+ const low = lifted.files['assets/samples/keys/Low_A.wav']!;
+ expect([low.rate, low.channels.length, low.channels[0]!.length]).toEqual([22050, 1, 5512]);
+ expect(low.channels[0]![25]).toBeCloseTo((16000 / 32768) * Math.sin((2 * Math.PI * 220 * 25) / 22050), 3);
+ expect(lifted.files['assets/samples/keys/High_A.wav']!.channels[0]!.length).toBe(13230);
+ // And a music source takes it as it comes.
+ expect(() => validateMusicSource({ tempo: 120, key: 'A', instruments: { keys: lifted.instrument }, patterns: { a: { bars: 1, tracks: { keys: '1 . . . . . . . . . . . . . . .' } } }, song: ['a'] })).not.toThrow();
+ });
+
+ it('plays at the pitches the SoundFont would', () => {
+ const lifted = liftSoundFontPreset(readSoundFont(FONT), 0, 'k');
+ const music = renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, instruments: { keys: lifted.instrument },
+ patterns: { a: { bars: 2, tracks: { keys: "1, - - . . . . . 1 - - - - - - - | 1 - - - - - - - - - - - - - - -" } } }, song: ['a'],
+ }), lifted.files);
+ const pitch = (from: number, to: number): number => {
+ const c = music.channels[0]!;
+ let crossings = 0;
+ for (let n = Math.floor(from * 44100) + 1; n < to * 44100; n++) if (c[n - 1]! <= 0 && c[n]! > 0) crossings++;
+ return crossings / (to - from);
+ };
+ // The A below, from the low recording at half the rate.
+ expect(pitch(0.05, 0.2)).toBeCloseTo(220, -1);
+ // The A above, a quarter of a semitone up as the zone says: 446.4 Hz.
+ expect(pitch(1.1, 1.9)).toBeGreaterThan(443);
+ expect(pitch(1.1, 1.9)).toBeLessThan(450);
+ // Held two seconds from a recording of three tenths, it goes round and is still sounding at the end.
+ expect(pitch(3.2, 3.9)).toBeGreaterThan(443);
+ // No recording is said to be a note it is not.
+ expect(music.notes.join()).not.toMatch(/sounds as/);
+ });
+
+ it('keeps to the notes a preset gives an instrument', () => {
+ const lifted = liftSoundFontPreset(readSoundFont(FONT), 1, 'k');
+ expect(lifted.instrument.samples).toEqual([{ file: 'k/High_A.wav', note: 'A4', tune: -25, loop: [0.1, 0.2] }]);
+ });
+
+ it('says what is not a SoundFont, and what is not in one', () => {
+ expect(() => readSoundFont(new Uint8Array(64))).toThrow(/not a SoundFont: it does not begin RIFF\.\.\.\.sfbk/);
+ expect(() => readSoundFont(chunk('RIFF', bytesOf([...'sfbk'].map((c) => c.charCodeAt(0)))))).toThrow(/a SoundFont with no "smpl" table/);
+ expect(() => liftSoundFontPreset(readSoundFont(FONT), 7, 'k')).toThrow(/there is no instrument 7 in this SoundFont; it has 2, numbered from 0/);
+ });
+});
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index 70f4e76b2..bfdf6a885 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4622,6 +4622,10 @@ export const NOTED = {
{"note": "An AI can make background music."},
"feat(music): the reply gives the join of a loop or how long an ending rings, and the tool text says how a loop is written and who fades one":
{"note": "An AI can make background music."},
+ "feat(music): recordings played soft and loud that take turns, a recording as a layer of a sound, and an instrument out of a SoundFont - samples carry dynamic and tune and share a note in turn, a sound has a sample layer shaped as any other, and an .sf2 is read for its instruments and one lifted out as recordings with their notes, ranges, tuning and loops":
+ {"note": "An AI can make background music."},
+ "feat(music): soundfont_read lists a SoundFont and takes an instrument out of it, and a sound's sample layers read their recordings - the tool text says dynamics, turns and where recordings come from":
+ {"note": "An AI can make background music."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":