Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(music): an instrument can be recordings of a real one - samples name a recording for each of a few notes, every note of the piece is played from the one nearest it, faster to be higher and slower to be lower, and a held note goes round the steady middle of its recording

yhhedb7956cff30
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bba307151..f54c2391e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,7 +21,7 @@ published separately; it ships inside the editor.
- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.
- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.
- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.
-- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. The instruments are made sounds, which suit electronic, retro and ambient music; a piano or strings that pass for real ones are not in reach of them.
+- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. The engine ships no recordings of its own: they are the project's.
- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.
- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.
- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.
diff --git a/desktop b/desktop
index 3af478e00..aa28b943f 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 3af478e00a7bb3cb0aadac2177db316b0c3fbf2b
+Subproject commit aa28b943f9467948770de2bae061e548305a5b95
diff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.ts
index 87955b683..22ffd963d 100644
--- a/pipeline/src/music/musicRender.ts
+++ b/pipeline/src/music/musicRender.ts
@@ -14,7 +14,10 @@
import { fft } from '../sound/soundPicture';
import { placeResponse, renderSound } from '../sound/soundRender';
import type { SoundLayer } from '../sound/soundSource';
-import { musicSeconds, parseTrack, SCALES, tonicOf, type MusicInstrument, type MusicSource } from './musicSource';
+import { midiOf, musicSeconds, parseTrack, SCALES, tonicOf, type MusicInstrument, type MusicSource } from './musicSource';
+
+/** A recording as samples: what an instrument made of recordings plays from. */
+export interface SampleData { rate: number; channels: readonly Float32Array[] }
export interface MusicEvent {
instrument: string;
@@ -168,6 +171,58 @@ function forNote(layer: SoundLayer, held: number, release: number): SoundLayer {
return { ...layer, length: Math.max(0.01, held + release - (layer.start ?? 0)), release: layer.release ?? release };
}
+/**
+ * One note of an instrument made of recordings: the recording nearest the note, played faster to be higher or slower
+ * to be lower, let go over the release. A recording an octave from its note is another instrument, which is why there
+ * are several; within three semitones the ear takes it for the same one.
+ */
+function sampleNote(name: string, instrument: MusicInstrument, midi: number | null, held: number, rate: number, given: Readonly<Record<string, SampleData>>): Float32Array[] {
+ const zones = instrument.samples!;
+ let zone = zones[0]!;
+ if (midi !== null) {
+ const ranged = zones.find((z) => (z.low !== undefined || z.high !== undefined) && midi >= (midiOf(z.low) ?? 0) && midi <= (midiOf(z.high) ?? 127));
+ zone = ranged ?? zones.reduce((best, z) => (Math.abs(midiOf(z.note)! - midi) < Math.abs(midiOf(best.note)! - midi) ? z : best));
+ }
+ const data = given[zone.file];
+ if (!data) throw new Error(`instrument "${name}": the recording ${zone.file} was not found`);
+ const frames = data.channels[0]!.length;
+ const step = (midi === null ? 1 : Math.pow(2, (midi - midiOf(zone.note)!) / 12)) * (data.rate / rate);
+ const [attack, release] = [instrument.attack ?? 0.003, instrument.release ?? 0.08];
+ // Round its steady middle, or the seconds given, with the end of each turn faded into the start of the next.
+ let [from, to] = [0, 0];
+ if (zone.loop) {
+ [from, to] = zone.loop === true ? [frames * 0.4, frames * 0.85] : [zone.loop[0] * data.rate, Math.min(frames - 2, zone.loop[1] * data.rate)];
+ }
+ const round = to - from;
+ const fade = round > 0 ? Math.min(0.12 * data.rate, round / 3, from) : 0;
+ const count = Math.max(1, Math.round(Math.min(held + release, round > 0 ? 30 : (frames - 2) / step / rate) * rate));
+ const at = (channel: Float32Array, p: number): number => {
+ // Through four points, so a note played slower is not a staircase.
+ const i = Math.floor(p), t = p - i;
+ const [a, b, c, d] = [channel[i - 1] ?? channel[i]!, channel[i]!, channel[i + 1] ?? 0, channel[i + 2] ?? 0];
+ return b + 0.5 * t * (c - a + t * (2 * a - 5 * b + 4 * c - d + t * (3 * (b - c) + d - a)));
+ };
+ const out = data.channels.slice(0, 2).map(() => new Float32Array(count));
+ const letGo = Math.round(held * rate);
+ const ending = Math.min(count, Math.round(0.005 * rate));
+ for (let n = 0; n < count; n++) {
+ let p = n * step;
+ let blend = 0;
+ if (round > 0 && p >= to - fade) {
+ p = from + ((p - from) % round);
+ if (fade > 0 && p >= to - fade) blend = (p - (to - fade)) / fade;
+ }
+ let level = n < attack * rate ? n / (attack * rate) : 1;
+ if (n > letGo) level *= Math.max(0, 1 - (n - letGo) / Math.max(1, release * rate));
+ if (count - n < ending) level *= (count - n) / ending;
+ for (let c = 0; c < out.length; c++) {
+ const channel = data.channels[c]!;
+ out[c]![n] = (blend > 0 ? at(channel, p) * (1 - blend) + at(channel, p - round) * blend : at(channel, p)) * level;
+ }
+ }
+ return out;
+}
+
const TAIL_SECONDS = 4;
/**
@@ -203,8 +258,8 @@ function inPlace(response: Float32Array): (signal: Float32Array) => Float32Array
};
}
-/** The samples a validated source makes. */
-export function renderMusic(source: MusicSource): RenderedMusic {
+/** The samples a validated source makes. @p given holds the recordings its instruments play, by the path each is named by. */
+export function renderMusic(source: MusicSource, given: Readonly<Record<string, SampleData>> = {}): RenderedMusic {
const compiled = compileMusic(source);
const rate = source.rate ?? 44100;
const count = Math.max(1, Math.round(compiled.seconds * rate));
@@ -217,9 +272,13 @@ export function renderMusic(source: MusicSource): RenderedMusic {
// The same note is made a few times over and no more: enough that a hat is not one sample struck again.
const key = `${name}|${e.midi}|${Math.round(e.seconds * 1000)}|${turn % 3}`;
let made = cache.get(key);
+ if (!made && instrument.samples) {
+ made = sampleNote(name, instrument, e.midi, e.seconds, rate, given);
+ cache.set(key, made);
+ }
if (!made) {
const release = instrument.release ?? 0.08;
- const layers = instrument.layers.map((l) => forNote(l, e.seconds, release));
+ const layers = instrument.layers!.map((l) => forNote(l, e.seconds, release));
const duration = Math.min(8, Math.max(0.02, ...layers.map((l) => reach(l, e.seconds + release))));
made = renderSound({ duration, rate, layers, normalize: false }, 0, { pitch: e.midi === null ? 1 : Math.pow(2, (e.midi - 69) / 12), seed: (turn % 3) * 7919, raw: true }).channels;
cache.set(key, made);
diff --git a/pipeline/src/music/musicSource.ts b/pipeline/src/music/musicSource.ts
index 0f278093b..ab7086668 100644
--- a/pipeline/src/music/musicSource.ts
+++ b/pipeline/src/music/musicSource.ts
@@ -9,6 +9,9 @@
* steps. A step is a degree of the key and not a pitch in Hz, so a line written without accidentals cannot
* leave the key, and a track that follows the bar's chord cannot leave the chord's scale either. What is saved
* is this description; the `.wav` a game plays is made from it.
+ *
+ * Layers make sounds that are plainly made. An instrument that has to pass for a real one is recordings of it:
+ * a few notes of a piano or a section of strings, each played back faster or slower for the notes round it.
*/
import { validateSoundSource, type SoundLayer } from '../sound/soundSource';
@@ -30,9 +33,26 @@ export type ScaleName = keyof typeof SCALES;
const TONICS: Record<string, number> = { C: 0, 'C#': 1, Db: 1, D: 2, 'D#': 3, Eb: 3, E: 4, F: 5, 'F#': 6, Gb: 6, G: 7, 'G#': 8, Ab: 8, A: 9, 'A#': 10, Bb: 10, B: 11 };
+/** One recording of an instrument playing one note. */
+export interface MusicSample {
+ /** A project path to the recording. */
+ file: string;
+ /** The note it is a recording of: "C4" is middle C, "F#3", "Bb2"; or its number, 60 being middle C. */
+ note: string | number;
+ /** The notes it plays, lowest to highest. Left out, each note is played by the recording nearest it. */
+ low?: string | number;
+ high?: string | number;
+ /** For a note held longer than the recording: true goes round its steady middle, [from, to] round those seconds of it. A piano dies of itself and has none. */
+ loop?: boolean | readonly [number, number];
+}
+
export interface MusicInstrument {
/** The layers of its sound at A above middle C (440 Hz): a sine at 440 is its fundamental, one at 880 the octave over it. A layer with no `length` lasts as long as the note; one with a `length` (the knock at the front of a pluck, a drum) keeps its own. */
- layers: readonly SoundLayer[];
+ layers?: readonly SoundLayer[];
+ /** In place of layers: recordings of the instrument, each played faster or slower for the notes round it. The further a note is from its recording the less it is the instrument, so a recording every three or four semitones. */
+ samples?: readonly MusicSample[];
+ /** Samples: seconds a note takes to come in. Default 0.003. */
+ attack?: number;
/** Loudness against the other instruments. Default 1. */
gain?: number;
/** -1 is left, 1 is right. Default 0. */
@@ -109,6 +129,20 @@ const NAME = /^[A-Za-z][A-Za-z0-9_-]{0,31}$/;
export const tonicOf = (key: string): number => TONICS[key]!;
+/** "C4" as its number, 60; a number as itself. Null for what is neither. */
+export function midiOf(note: unknown): number | null {
+ if (typeof note === 'number') return Number.isInteger(note) && note >= 0 && note <= 127 ? note : null;
+ const m = typeof note === 'string' ? /^([A-G](?:#|b)?)(-?\d)$/.exec(note) : null;
+ if (!m || TONICS[m[1]!] === undefined) return null;
+ const midi = TONICS[m[1]!]! + 12 * (Number(m[2]) + 1);
+ return midi >= 0 && midi <= 127 ? midi : null;
+}
+
+/** The recordings a source plays, each once. */
+export function musicSampleFiles(source: MusicSource): string[] {
+ return [...new Set(Object.values(source.instruments).flatMap((i) => (i.samples ?? []).map((s) => s.file)))];
+}
+
const loudOf = (mark: string): number => (mark === '!' ? 1.3 : mark === '?' ? 0.6 : 1);
function noteOf(text: string): StepNote | null {
@@ -149,7 +183,25 @@ export function parseTrack(text: string, bars: number, stepsToBar: number, where
}
const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'stems', 'reverb', 'gain', 'rate'];
-const INSTRUMENT_FIELDS = ['layers', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb'];
+const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb'];
+const RECORDING = /\.(wav|ogg|mp3|flac|m4a|aac|webm)$/i;
+
+function checkSamples(samples: unknown, where: string): void {
+ if (!Array.isArray(samples) || samples.length === 0 || samples.length > 64) throw new Error(`${where}: samples must be 1..64 of { file, note }`);
+ samples.forEach((sample: unknown, i) => {
+ const at = `${where}, sample ${i + 1}`;
+ if (!isObj(sample)) throw new Error(`${at} must be { file, note }`);
+ for (const key of Object.keys(sample)) if (!['file', 'note', 'low', 'high', 'loop'].includes(key)) throw new Error(`${at} has unknown "${key}"; a sample has file, note, low, high, loop`);
+ if (typeof sample.file !== 'string' || !RECORDING.test(sample.file)) throw new Error(`${at}: file must be a project path to a recording (.wav, .ogg, .mp3, .flac, .m4a)`);
+ if (midiOf(sample.note) === null) throw new Error(`${at}: note must be the note recorded, as "C4", "F#3" or "Bb2" (C4 is middle C), or its number 0..127`);
+ for (const key of ['low', 'high']) if (sample[key] !== undefined && midiOf(sample[key]) === null) throw new Error(`${at}: ${key} must be a note, as "C4"`);
+ if (sample.low !== undefined && sample.high !== undefined && midiOf(sample.low)! > midiOf(sample.high)!) throw new Error(`${at}: low is above high`);
+ const loop = sample.loop;
+ if (loop !== undefined && typeof loop !== 'boolean' && !(Array.isArray(loop) && loop.length === 2 && isNum(loop[0]) && isNum(loop[1]) && loop[0] >= 0 && loop[1] > loop[0] + 0.02)) {
+ throw new Error(`${at}: loop must be true, or [from, to] in seconds with to past from`);
+ }
+ });
+}
const PATTERN_FIELDS = ['bars', 'div', 'chords', 'tracks'];
/** Seconds the song lasts, before anything that rings past its end. */
@@ -188,12 +240,17 @@ export function validateMusicSource(raw: unknown): MusicSource {
if (!NAME.test(name)) throw new Error(`${where}: a name is a letter, then letters, digits, _ or -`);
if (!isObj(value)) throw new Error(`${where} must be { layers: [...] }`);
for (const key of Object.keys(value)) if (!INSTRUMENT_FIELDS.includes(key)) throw new Error(`${where} has unknown "${key}"; an instrument has ${INSTRUMENT_FIELDS.join(', ')}`);
- try {
- // Its layers are a sound's layers, and are wrong in the same ways.
- validateSoundSource({ duration: 8, layers: value.layers });
- } catch (e) {
- throw new Error(`${where}: ${(e as Error).message}`);
+ if ((value.layers === undefined) === (value.samples === undefined)) throw new Error(`${where} needs layers (a made sound) or samples (recordings), one of the two`);
+ if (value.samples !== undefined) checkSamples(value.samples, where);
+ else {
+ try {
+ // Its layers are a sound's layers, and are wrong in the same ways.
+ validateSoundSource({ duration: 8, layers: value.layers });
+ } catch (e) {
+ throw new Error(`${where}: ${(e as Error).message}`);
+ }
}
+ if (value.attack !== undefined && !(value.samples !== undefined && between(value.attack, 0, 2))) throw new Error(`${where}: attack is for samples, seconds 0..2; layers have their own`);
if (value.gain !== undefined && !between(value.gain, 0, 4)) throw new Error(`${where}: gain must be 0..4`);
if (value.pan !== undefined && !between(value.pan, -1, 1)) throw new Error(`${where}: pan must be -1..1`);
if (value.octave !== undefined && !(Number.isInteger(value.octave) && between(value.octave, -4, 4))) throw new Error(`${where}: octave must be a whole number, -4..4`);
diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.ts
index b5ecde34d..cf0223bb8 100644
--- a/pipeline/tests/music.test.ts
+++ b/pipeline/tests/music.test.ts
@@ -5,8 +5,8 @@
* what pitch comes out, that a loop runs into itself and that the instruments written apart add up to the whole.
*/
import { describe, expect, it } from 'vitest';
-import { applyMusicEdits, validateMusicSource, type MusicSource } from '../src/music/musicSource';
-import { compileMusic, noteName, renderMusic } from '../src/music/musicRender';
+import { applyMusicEdits, midiOf, musicSampleFiles, validateMusicSource, type MusicSource } from '../src/music/musicSource';
+import { compileMusic, noteName, renderMusic, type SampleData } from '../src/music/musicRender';
import { musicPicture } from '../src/music/musicPicture';
const tone = { layers: [{ source: 'sine', freq: 440 }] } as const;
@@ -202,3 +202,106 @@ describe('music from instruments and patterns', () => {
expect(drum[0]! > 150 && drum[0]! > drum[2]! + 40).toBe(true);
});
});
+
+describe('an instrument made of recordings', () => {
+ /** A recording of a plain tone: @p hz for @p seconds, as loud at its end as at its start. */
+ const recorded = (hz: number, seconds: number, rate = 44100): SampleData => ({ rate, channels: [Float32Array.from({ length: Math.round(seconds * rate) }, (_, n) => 0.5 * Math.sin((2 * Math.PI * hz * n) / rate))] });
+ const given = { 'a4.wav': recorded(440, 1), 'a2.wav': recorded(110, 1), 'half.wav': recorded(440, 1, 22050) };
+ const played = (samples: unknown[], line: string, rest: Record<string, unknown> = {}, more: Record<string, unknown> = {}) => renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5,
+ instruments: { keys: { samples, ...more } }, patterns: { a: { bars: 1, tracks: { keys: line } } }, song: ['a'], ...rest,
+ }), given);
+ const HELD = '- - - - - - -';
+ const one = (note: string): string => `${note} ${HELD} . . . . . . . .`;
+
+ it('plays a recording faster to be higher and slower to be lower', () => {
+ const zone = [{ file: 'a4.wav', note: 'A4' }];
+ expect(pitchOf(played(zone, one('1')).channels[0]!, 44100, 0.1, 0.9)).toBeCloseTo(440, -1);
+ expect(pitchOf(played(zone, one("1'")).channels[0]!, 44100, 0.1, 0.4)).toBeCloseTo(880, -1.3);
+ expect(pitchOf(played(zone, one('1,')).channels[0]!, 44100, 0.1, 0.9)).toBeCloseTo(220, -1);
+ // A fifth up is half as much again.
+ expect(pitchOf(played(zone, one('5')).channels[0]!, 44100, 0.1, 0.6)).toBeCloseTo(659.3, -1);
+ });
+
+ it('is the same pitch whatever rate the recording was made at', () => {
+ expect(pitchOf(played([{ file: 'half.wav', note: 'A4' }], one('1')).channels[0]!, 44100, 0.1, 0.9)).toBeCloseTo(440, -1);
+ expect(pitchOf(played([{ file: 'a4.wav', note: 'A4' }], one('1'), { rate: 22050 }).channels[0]!, 22050, 0.1, 0.9)).toBeCloseTo(440, -1);
+ });
+
+ it('takes each note from the recording nearest it, or from the one whose range holds it', () => {
+ // The recordings are two octaves apart and each plays the notes round it.
+ const near = [{ file: 'a4.wav', note: 'A4' }, { file: 'a2.wav', note: 'A2' }];
+ expect(pitchOf(played(near, one('1')).channels[0]!, 44100, 0.1, 0.9)).toBeCloseTo(440, -1);
+ expect(pitchOf(played(near, one('1,,')).channels[0]!, 44100, 0.1, 0.9)).toBeCloseTo(110, -1);
+ expect(pitchOf(played(near, one('4,')).channels[0]!, 44100, 0.1, 0.6)).toBeCloseTo(293.7, -1);
+ expect(pitchOf(played(near, one('4,,')).channels[0]!, 44100, 0.1, 0.6)).toBeCloseTo(146.8, -1);
+ // Which one played is heard when one of them is called a note it is not: told it is A3 and given every note
+ // up to C5, the recording of A4 plays an A4 an octave high.
+ const moved = [{ file: 'a2.wav', note: 'A2' }, { file: 'a4.wav', note: 'A3', low: 'C1', high: 'C5' }];
+ expect(pitchOf(played(moved, one('1')).channels[0]!, 44100, 0.1, 0.4)).toBeCloseTo(880, -1.3);
+ // And without the range the nearer recording has it: A2, three octaves up from where it is said to be.
+ const unranged = [{ file: 'a2.wav', note: 'A4' }, { file: 'a4.wav', note: 'A1' }];
+ expect(pitchOf(played(unranged, one('1')).channels[0]!, 44100, 0.1, 0.9)).toBeCloseTo(110, -1);
+ });
+
+ it('stops where the recording ends, unless it goes round', () => {
+ const short = { 'a4.wav': recorded(440, 0.3) };
+ const line = `1 ${HELD} ${HELD} -`;
+ const render = (loop?: unknown) => renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5,
+ instruments: { keys: { samples: [{ file: 'a4.wav', note: 'A4', ...(loop !== undefined ? { loop } : {}) }] } }, patterns: { a: { bars: 1, tracks: { keys: line } } }, song: ['a'],
+ }), short);
+ const once = render().channels[0]!;
+ expect(level(once, 44100, 0.05, 0.25)).toBeGreaterThan(0.1);
+ expect(level(once, 44100, 0.4, 1.9)).toBeLessThan(1e-4);
+ for (const loop of [true, [0.1, 0.25]]) {
+ const round = render(loop).channels[0]!;
+ expect(level(round, 44100, 1.5, 1.9)).toBeGreaterThan(0.1);
+ expect(pitchOf(round, 44100, 0.5, 1.9)).toBeCloseTo(440, -1);
+ // Going round does not step: no sample leaps from the one before by more than the tone itself does.
+ let leap = 0;
+ for (let n = 1; n < round.length; n++) leap = Math.max(leap, Math.abs(round[n]! - round[n - 1]!));
+ expect(leap).toBeLessThan(0.06);
+ }
+ });
+
+ it('comes in over the attack and is let go over the release', () => {
+ const m = played([{ file: 'a4.wav', note: 'A4' }], '1 - - - . . . . . . . . . . . .', {}, { attack: 0.2, release: 0.25 }).channels[0]!;
+ expect(level(m, 44100, 0, 0.05)).toBeLessThan(level(m, 44100, 0.3, 0.45) * 0.3);
+ expect(level(m, 44100, 0.55, 0.7)).toBeGreaterThan(1e-3);
+ expect(level(m, 44100, 0.8, 1.9)).toBeLessThan(1e-4);
+ });
+
+ it('keeps both sides of a recording made in stereo, and plays a drum as it was recorded', () => {
+ const stereo = { 'two.wav': { rate: 44100, channels: [recorded(440, 0.5).channels[0]!, new Float32Array(22050)] } };
+ const m = renderMusic(validateMusicSource({
+ tempo: 120, key: 'C', scale: 'major', humanize: 0, loop: false,
+ instruments: { hit: { samples: [{ file: 'two.wav', note: 'C4' }] } }, patterns: { a: { bars: 1, tracks: { hit: 'x . . . . . . . . . . . . . . .' } } }, song: ['a'],
+ }), stereo);
+ expect(pitchOf(m.channels[0]!, 44100, 0.02, 0.1)).toBeCloseTo(440, -1);
+ expect(level(m.channels[1]!, 44100, 0, 0.5)).toBeLessThan(1e-6);
+ });
+
+ it('names its recordings, and says which is missing', () => {
+ const source = validateMusicSource({
+ tempo: 120, key: 'A', instruments: { keys: { samples: [{ file: 'a.wav', note: 60 }, { file: 'b.ogg', note: 'F#3' }] }, more: { samples: [{ file: 'a.wav', note: 'Bb2' }] }, tone },
+ patterns: { a: { bars: 1, tracks: { keys: one('1') } } }, song: ['a'],
+ });
+ expect(musicSampleFiles(source)).toEqual(['a.wav', 'b.ogg']);
+ expect(() => renderMusic(source, {})).toThrow(/instrument "keys": the recording (a\.wav|b\.ogg) was not found/);
+ expect([midiOf('C4'), midiOf('A4'), midiOf('F#3'), midiOf('Bb2'), midiOf('C-1'), midiOf('H2'), midiOf(128)]).toEqual([60, 69, 54, 46, 0, null, null]);
+ });
+
+ it('refuses samples that are not well formed', () => {
+ const bad = (instrument: unknown) => () => validateMusicSource({ tempo: 120, key: 'A', instruments: { keys: instrument }, patterns: { a: { bars: 1, tracks: { keys: one('1') } } }, song: ['a'] });
+ expect(bad({})).toThrow(/instrument "keys" needs layers \(a made sound\) or samples \(recordings\), one of the two/);
+ expect(bad({ layers: [{ source: 'noise' }], samples: [{ file: 'a.wav', note: 'C4' }] })).toThrow(/one of the two/);
+ expect(bad({ samples: [] })).toThrow(/samples must be 1\.\.64/);
+ expect(bad({ samples: [{ file: 'a.txt', note: 'C4' }] })).toThrow(/sample 1: file must be a project path to a recording/);
+ expect(bad({ samples: [{ file: 'a.wav', note: 'C' }] })).toThrow(/note must be the note recorded, as "C4"/);
+ expect(bad({ samples: [{ file: 'a.wav', note: 'C4', low: 'C5', high: 'C3' }] })).toThrow(/low is above high/);
+ expect(bad({ samples: [{ file: 'a.wav', note: 'C4', loop: [0.5, 0.5] }] })).toThrow(/loop must be true, or \[from, to\]/);
+ expect(bad({ samples: [{ file: 'a.wav', note: 'C4', root: 'C4' }] })).toThrow(/unknown "root"; a sample has file, note, low, high, loop/);
+ expect(bad({ layers: [{ source: 'noise' }], attack: 0.1 })).toThrow(/attack is for samples/);
+ });
+});
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index 3ba5e6d85..fb8c6e8e3 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4610,6 +4610,10 @@ export const NOTED = {
{"internal": "A check of tools already noted; nothing a user sees changes."},
"chore(desktop): an editor check runs the sound and music tools":
{"internal": "A check of tools already noted; nothing a user sees changes."},
+ "feat(music): an instrument can be recordings of a real one - samples name a recording for each of a few notes, every note of the piece is played from the one nearest it, faster to be higher and slower to be lower, and a held note goes round the steady middle of its recording":
+ {"note": "An AI can make background music."},
+ "feat(music): music_write plays instruments made of recordings in the project - the tool text says how many are needed and which loop, and the editor check plays one":
+ {"note": "An AI can make background music."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":