Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(music): a line is played and not only sounded - dynamics give each bar the loudness it begins on and move evenly between, legato brings a note in under the one before from past its attack, and pedal holds every note of a piano to the end of its bar

yhhbb74e3173000
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 924af0f54..8273aa057 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,7 +25,7 @@ published separately; it ships inside the editor.
- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.
- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.
- **Music with an introduction, and a source that fades.** Background music looped from its first sample to its last, so a piece could not open with something heard once; and an `AudioSource` in a scene started and stopped at full volume, the fades being `playBGM`'s alone. A looping voice now takes `loopStart`, seconds into the clip that it comes back to at its end: `playBGM(url, { loopStart })` and `AudioSource.loopStart` play the file from its top once and round from there after. It is exact to the sample where the clip is decoded, on the web and on the native host; where a clip is played from its compressed file (a long clip streamed on the web, a mini-game) the player is sent back when it ends, which can be heard as a breath. `AudioSource.fadeIn` and `fadeOut` are seconds: raising `playing` brings it up from nothing, lowering it lets it down before the voice is stopped, and raising it again on the way out turns it round where it is. A piece made with `music_write` marks its first parts as an introduction with `intro`, and the reply gives the `loopStart` to play it with.
-- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.
+- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Right notes at one loudness, each begun from nothing, are a machine reading a score, so a line can be played: `dynamics` on a pattern give each bar the loudness it begins on and move evenly between them, a crescendo and a diminuendo; `legato` on an instrument brings a note that runs straight on from the last in under it, from past the attack of its recording, as a bow that does not leave the string; and `pedal` holds every note of a piano to the end of its bar. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.
- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. A layer can also be a recording in the project (`source: "sample"`), played at a speed and shaped as any other layer is, for the one sound that will not come out of waves and noise. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.
- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.
- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.
diff --git a/desktop b/desktop
index b554a9fb4..42d21fa21 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit b554a9fb4765945dc38aa7593520638602cd076a
+Subproject commit 42d21fa21812ceb4f93157146acf429697cf58a3
diff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.ts
index b301b992a..0d2f4b197 100644
--- a/pipeline/src/music/musicRender.ts
+++ b/pipeline/src/music/musicRender.ts
@@ -31,8 +31,16 @@ export interface MusicEvent {
part: number;
/** Written with # or b: outside the key. */
outside: boolean;
- /** Marked softer (-1) or louder (1) than the rest; 0 for neither. */
+ /** Softer (-1) or louder (1) than the rest by its mark and by where it lies in the dynamics; 0 for neither. */
mark: -1 | 0 | 1;
+ /** How loud it is written, by its mark and the dynamics, before the hand's unevenness. */
+ hard: number;
+ /** How long it sounds, where that is longer than it is written: held on by the pedal to the end of its bar. */
+ sounds?: number;
+ /** It comes in under the note before it and is not struck afresh. */
+ slur?: boolean;
+ /** The note after it comes in under it, and it stays a moment longer for that. */
+ tied?: boolean;
}
export interface CompiledMusic {
@@ -106,6 +114,16 @@ export function compileMusic(source: MusicSource): CompiledMusic {
const instrument = source.instruments[track]!;
if (instrument.follow && !pattern.chords) say(`"${track}" follows the chords, and pattern "${name}" has none: it plays every bar on chord 1`);
const steps = parseTrack(line, pattern.bars, stepsToBar, `pattern "${name}", track "${track}"`);
+ // How loud the line is at each step: between the number its bar begins on and the next bar's.
+ const levels = pattern.dynamics?.[track] ?? pattern.dynamics?.all;
+ const levelAt = (i: number): number => {
+ if (!levels) return 1;
+ const bar = i / stepsToBar;
+ const [a, b] = [levels[Math.min(Math.floor(bar), levels.length - 1)]!, levels[Math.min(Math.floor(bar) + 1, levels.length - 1)]!];
+ return a + (b - a) * (bar - Math.floor(bar));
+ };
+ // The last single note of this line, and the step it ends on: what a note that follows straight on is slurred from.
+ let before: { end: number; event: MusicEvent } | null = null;
for (let i = 0; i < steps.length; i++) {
const s = steps[i]!;
if (s.kind === 'hold' && (i === 0 || steps[i - 1]!.kind === 'rest')) say(`pattern "${name}", track "${track}": a "-" at step ${(i % stepsToBar) + 1} of bar ${Math.floor(i / stepsToBar) + 1} has no note before it to hold, and is silent`);
@@ -115,21 +133,30 @@ export function compileMusic(source: MusicSource): CompiledMusic {
// Every second step late, which is what makes a line swing.
const late = div % 2 === 0 && i % 2 === 1 ? (source.swing ?? 0) * step : 0;
const when = Math.max(0, at + i * step + late + human * 0.012 * random());
- const loud = s.loud * (1 + human * 0.25 * random());
- const mark = s.loud > 1 ? 1 : s.loud < 1 ? -1 : 0;
+ const hard = s.loud * levelAt(i);
+ const loud = hard * (1 + human * 0.25 * random());
+ const mark = hard > 1.15 ? 1 : hard < 0.8 ? -1 : 0;
+ // The pedal is down to the end of the bar: the note rings on to there.
+ const toBarEnd = (Math.floor(i / stepsToBar) + 1) * stepsToBar * step - i * step;
+ const sounds = instrument.pedal && toBarEnd > held * step ? { sounds: toBarEnd } : {};
played.add(track);
if (s.kind === 'hit') {
- events.push({ instrument: track, at: when, seconds: held * step, midi: null, loud, part: p, outside: false, mark });
+ events.push({ instrument: track, at: when, seconds: held * step, midi: null, loud, part: p, outside: false, mark, hard, ...sounds });
+ before = null;
continue;
}
+ // One note running straight on from one note, and not struck harder: in the same breath.
+ const slurred = !!instrument.legato && s.notes.length === 1 && before !== null && before.end === i && s.loud <= 1;
+ if (slurred) before!.event.tied = true;
const chord = instrument.follow && pattern.chords ? pattern.chords[Math.floor(i / stepsToBar) % pattern.chords.length]! : 1;
// A root on 5, 6 or 7 is taken below the tonic, so the chords lie round it and a bass does not climb away.
const drop = chord >= 5 ? -7 : 0;
for (const n of s.notes) {
const index = n.degree - 1 + chord - 1 + drop + 7;
const midi = 60 + tonic + scale[index % 7]! + 12 * (Math.floor(index / 7) - 1 + n.octave + (instrument.octave ?? 0)) + n.alter;
- events.push({ instrument: track, at: when, seconds: held * step, midi, loud, part: p, outside: n.alter !== 0, mark });
+ events.push({ instrument: track, at: when, seconds: held * step, midi, loud, part: p, outside: n.alter !== 0, mark, hard, ...sounds, ...(slurred ? { slur: true } : {}) });
}
+ before = s.notes.length === 1 ? { end: i + held, event: events.at(-1)! } : null;
}
}
if (events.length === before) notes.push(`part ${p + 1} of the song ("${name}") plays nothing`);
@@ -174,10 +201,11 @@ function reach(layer: SoundLayer, held: number): number {
}
/** A layer with no length of its own lasts as long as the note, and dies over the instrument's release. */
-function forNote(layer: SoundLayer, held: number, release: number): SoundLayer {
- if (layer.source === 'group') return { ...layer, layers: layer.layers!.map((l) => forNote(l, held, release)) };
- if (layer.length !== undefined) return layer;
- return { ...layer, length: Math.max(0.01, held + release - (layer.start ?? 0)), release: layer.release ?? release };
+function forNote(layer: SoundLayer, held: number, release: number, slur = 0): SoundLayer {
+ if (layer.source === 'group') return { ...layer, layers: layer.layers!.map((l) => forNote(l, held, release, slur)) };
+ // A layer with a length of its own is the knock at the front of a note, which a slurred note has none of.
+ if (layer.length !== undefined) return slur > 0 ? { ...layer, mute: true } : layer;
+ return { ...layer, length: Math.max(0.01, held + release - (layer.start ?? 0)), release: layer.release ?? release, ...(slur > 0 ? { attack: Math.max(layer.attack ?? 0, slur) } : {}) };
}
/**
@@ -211,13 +239,16 @@ export function sampleZone(instrument: MusicInstrument, midi: number | null, mar
return { index: left[turn % left.length]!, matched };
}
-function sampleNote(name: string, instrument: MusicInstrument, index: number, midi: number | null, held: number, rate: number, given: Readonly<Record<string, SampleData>>): Float32Array[] {
+function sampleNote(name: string, instrument: MusicInstrument, index: number, midi: number | null, held: number, rate: number, given: Readonly<Record<string, SampleData>>, slur = 0): Float32Array[] {
const zone = instrument.samples![index]!;
const data = given[zone.file];
if (!data) throw new Error(`instrument "${name}": the recording ${zone.file} was not found`);
const frames = data.channels[0]!.length;
const step = (midi === null ? 1 : Math.pow(2, (midi - midiOf(zone.note)! - (zone.tune ?? 0) / 100) / 12)) * (data.rate / rate);
- const [attack, release] = [instrument.attack ?? 0.003, instrument.release ?? 0.08];
+ // Slurred, it is not the start of the recording that is wanted, which is the bow biting or the breath beginning,
+ // but what comes after; and it comes in over the time the note before takes to go.
+ const skip = slur > 0 ? Math.min(0.12 * data.rate, frames * 0.25) : 0;
+ const [attack, release] = [slur > 0 ? slur : instrument.attack ?? 0.003, instrument.release ?? 0.08];
// Round its steady middle, or the seconds given, with the end of each turn faded into the start of the next.
let [from, to] = [0, 0];
if (zone.loop) {
@@ -225,7 +256,7 @@ function sampleNote(name: string, instrument: MusicInstrument, index: number, mi
}
const round = to - from;
const fade = round > 0 ? Math.min(0.12 * data.rate, round / 3, from) : 0;
- const count = Math.max(1, Math.round(Math.min(held + release, round > 0 ? 30 : (frames - 2) / step / rate) * rate));
+ const count = Math.max(1, Math.round(Math.min(held + release, round > 0 ? 30 : (frames - 2 - skip) / step / rate) * rate));
const at = (channel: Float32Array, p: number): number => {
// Through four points, so a note played slower is not a staircase.
const i = Math.floor(p), t = p - i;
@@ -236,7 +267,7 @@ function sampleNote(name: string, instrument: MusicInstrument, index: number, mi
const letGo = Math.round(held * rate);
const ending = Math.min(count, Math.round(0.005 * rate));
for (let n = 0; n < count; n++) {
- let p = n * step;
+ let p = skip + n * step;
let blend = 0;
if (round > 0 && p >= to - fade) {
p = from + ((p - from) % round);
@@ -348,16 +379,21 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,
const cache = new Map<string, Float32Array[]>();
const noteOf = (name: string, instrument: MusicInstrument, e: MusicEvent, turn: number, zone: number): Float32Array[] => {
// The same note is made a few times over and no more: enough that a hat is not one sample struck again.
- const key = `${name}|${e.midi}|${Math.round(e.seconds * 1000)}|${instrument.samples ? `z${zone}` : turn % 3}`;
+ const legato = instrument.legato === true ? 0.07 : instrument.legato || 0;
+ const slur = e.slur ? legato : 0;
+ // As long as it sounds: as written, or on to the bar line under the pedal, and a moment more where the next
+ // note comes in under it.
+ const held = (e.sounds ?? e.seconds) + (e.tied ? legato : 0);
+ const key = `${name}|${e.midi}|${Math.round(held * 1000)}|${slur}|${instrument.samples ? `z${zone}` : turn % 3}`;
let made = cache.get(key);
if (!made && instrument.samples) {
- made = sampleNote(name, instrument, zone, e.midi, e.seconds, rate, given);
+ made = sampleNote(name, instrument, zone, e.midi, held, rate, given, slur);
cache.set(key, made);
}
if (!made) {
const release = instrument.release ?? 0.08;
- const layers = instrument.layers!.map((l) => forNote(l, e.seconds, release));
- const duration = Math.min(8, Math.max(0.02, ...layers.map((l) => reach(l, e.seconds + release))));
+ const layers = instrument.layers!.map((l) => forNote(l, held, release, slur));
+ const duration = Math.min(8, Math.max(0.02, ...layers.map((l) => reach(l, held + release))));
made = renderSound({ duration, rate, layers, normalize: false }, 0, { pitch: e.midi === null ? 1 : Math.pow(2, (e.midi - 69) / 12), seed: (turn % 3) * 7919, raw: true }).channels;
cache.set(key, made);
}
@@ -382,7 +418,7 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,
const pan = instrument.pan ?? 0;
// Equal power, and full in the middle. A recording made as hard as the note is marked is already that loud:
// the mark is not counted twice.
- const loud = (zone.matched ? e.loud / (e.mark > 0 ? 1.3 : 0.6) : e.loud) * (instrument.gain ?? 1) * Math.SQRT2;
+ const loud = (zone.matched ? e.loud / e.hard : e.loud) * (instrument.gain ?? 1) * Math.SQRT2;
const [gl, gr] = [Math.cos(((pan + 1) * Math.PI) / 4) * loud, Math.sin(((pan + 1) * Math.PI) / 4) * loud];
const [left, right] = buses.get(e.instrument)!;
const from = Math.round(e.at * rate);
diff --git a/pipeline/src/music/musicSource.ts b/pipeline/src/music/musicSource.ts
index eb45fdbd2..9681e8171 100644
--- a/pipeline/src/music/musicSource.ts
+++ b/pipeline/src/music/musicSource.ts
@@ -73,6 +73,20 @@ export interface MusicInstrument {
follow?: boolean;
/** Seconds a held note takes to die after it is let go. Default 0.08. */
release?: number;
+ /**
+ * A line played in one breath. Where one note of its line runs straight into the next, with no silence between
+ * and the next not marked `!`, the next is not struck afresh: it comes in under the one before over this many
+ * seconds (true is 0.07) while that one goes out, and a recording is taken from past its attack. This is a bow
+ * that does not leave the string and a breath that is not taken again; without it every note of a violin or a
+ * flute begins from nothing and a tune is a row of separate notes. Chords are struck as they were.
+ */
+ legato?: boolean | number;
+ /**
+ * The sustaining pedal, held down through each bar and changed on the bar line: every note rings on to the end
+ * of its bar however short it is written, so the notes of a broken chord pile up into the chord and one bar
+ * runs into the next. A piano without it is dry and its chords are apart.
+ */
+ pedal?: boolean;
/** How much of it goes to the room, 0..1. Default 0.3; a bass or a kick wants 0. */
reverb?: number;
}
@@ -83,6 +97,13 @@ export interface MusicPattern {
div?: number;
/** The chord of each bar, as the degree its root is on: [1, 6, 3, 7]. Fewer than the bars come round again. Only instruments that `follow` hear them. */
chords?: readonly number[];
+ /**
+ * How loud each bar begins, for an instrument or for `all` of them, 1 being as written: `{ lead: [0.6, 0.8, 1,
+ * 0.7] }`. Between one bar's number and the next the loudness moves evenly, which is a crescendo or a
+ * diminuendo; one number more than there are bars is where the last bar ends. A phrase that is all one loudness
+ * is not heard as a phrase.
+ */
+ dynamics?: Readonly<Record<string, readonly number[]>>;
/**
* A line of steps for each instrument that plays, separated by spaces, bars by `|`. A step is: `.` silence; `-` the
* note before, still held; a degree 1..7 with `'` for each octave up and `,` for each down, `#` or `b` before it to
@@ -207,7 +228,7 @@ export function parseTrack(text: string, bars: number, stepsToBar: number, where
}
const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'intro', 'fadeIn', 'fadeOut', 'stems', 'reverb', 'gain', 'rate'];
-const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb'];
+const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb', 'legato', 'pedal'];
const RECORDING = /\.(wav|ogg|mp3|flac|m4a|aac|webm)$/i;
function checkSamples(samples: unknown, where: string): void {
@@ -228,7 +249,7 @@ function checkSamples(samples: unknown, where: string): void {
}
});
}
-const PATTERN_FIELDS = ['bars', 'div', 'chords', 'tracks'];
+const PATTERN_FIELDS = ['bars', 'div', 'chords', 'dynamics', 'tracks'];
/** Seconds the song lasts, before anything that rings past its end. */
export function musicSeconds(source: MusicSource): number {
@@ -283,6 +304,8 @@ export function validateMusicSource(raw: unknown): MusicSource {
if (value.follow !== undefined && typeof value.follow !== 'boolean') throw new Error(`${where}: follow must be true or false`);
if (value.release !== undefined && !between(value.release, 0, 4)) throw new Error(`${where}: release must be seconds, 0..4`);
if (value.reverb !== undefined && !between(value.reverb, 0, 1)) throw new Error(`${where}: reverb must be 0..1`);
+ if (value.legato !== undefined && typeof value.legato !== 'boolean' && !between(value.legato, 0.01, 0.5)) throw new Error(`${where}: legato must be true, or the seconds one note takes to come in under the last, 0.01..0.5`);
+ if (value.pedal !== undefined && typeof value.pedal !== 'boolean') throw new Error(`${where}: pedal must be true or false`);
}
const beats = (s.beats as number | undefined) ?? 4;
@@ -299,6 +322,15 @@ export function validateMusicSource(raw: unknown): MusicSource {
throw new Error(`${where}: chords must be the degree of each bar's root, 1..7, no more of them than bars`);
}
}
+ if (value.dynamics !== undefined) {
+ if (!isObj(value.dynamics)) throw new Error(`${where}: dynamics must be { instrument or "all": [how loud each bar begins] }`);
+ for (const [who, levels] of Object.entries(value.dynamics)) {
+ if (who !== 'all' && !(who in s.instruments)) throw new Error(`${where}: dynamics for "${who}", which is not an instrument; the instruments are ${Object.keys(s.instruments).join(', ')}, or "all"`);
+ if (!Array.isArray(levels) || levels.length === 0 || levels.length > (value.bars as number) + 1 || levels.some((v) => !between(v, 0.1, 2))) {
+ throw new Error(`${where}: dynamics for "${who}" must be how loud each bar begins, 0.1..2 with 1 as written, one for each of its ${String(value.bars)} bars and at most one more for where the last ends`);
+ }
+ }
+ }
if (!isObj(value.tracks) || Object.keys(value.tracks).length === 0) throw new Error(`${where}: tracks must be { instrument: "steps" }, at least one`);
for (const [track, line] of Object.entries(value.tracks)) {
if (!(track in s.instruments)) throw new Error(`${where} has a track for "${track}", which is not an instrument; the instruments are ${Object.keys(s.instruments).join(', ')}`);
diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.ts
index 2090973ad..b3041963b 100644
--- a/pipeline/tests/music.test.ts
+++ b/pipeline/tests/music.test.ts
@@ -487,3 +487,114 @@ describe('an introduction heard once before the round', () => {
expect(() => piece(tracks, { instruments, song: ['a', 'a'], intro: 1 })).toThrow(/intro is the part of a loop that is heard once/);
});
});
+
+describe('a line that is played and not only sounded', () => {
+ const EIGHT = '1 - - - 1 - - - 1 - - - 1 - - -';
+
+ it('moves evenly from the loudness one bar begins on to the next', () => {
+ const source = validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, instruments: { tone, other: tone },
+ patterns: { a: { bars: 2, dynamics: { tone: [0.5, 1, 0.6] }, tracks: { tone: `${EIGHT} | ${EIGHT}`, other: `${EIGHT} | ${EIGHT}` } } }, song: ['a'],
+ });
+ const e = compileMusic(source).events;
+ const of = (name: string): number[] => e.filter((x) => x.instrument === name).map((x) => Math.round(x.loud * 1000) / 1000);
+ // Up through the first bar and down through the second, a quarter of the way at each beat.
+ expect(of('tone')).toEqual([0.5, 0.625, 0.75, 0.875, 1, 0.9, 0.8, 0.7]);
+ // An instrument with no line of its own is as written.
+ expect(of('other')).toEqual([1, 1, 1, 1, 1, 1, 1, 1]);
+ // And "all" is for every instrument that has none of its own.
+ const all = compileMusic(validateMusicSource({ ...source, patterns: { a: { ...source.patterns.a!, dynamics: { all: [0.5], tone: [1] } } } })).events;
+ expect([...new Set(all.filter((x) => x.instrument === 'other').map((x) => x.loud))]).toEqual([0.5]);
+ expect([...new Set(all.filter((x) => x.instrument === 'tone').map((x) => x.loud))]).toEqual([1]);
+ });
+
+ it('is heard to rise and fall', () => {
+ const m = renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5, instruments: { tone },
+ patterns: { a: { bars: 2, dynamics: { tone: [0.4, 1, 0.4] }, tracks: { tone: `${EIGHT} | ${EIGHT}` } } }, song: ['a'],
+ })).channels[0]!;
+ expect(level(m, 44100, 0.05, 0.4)).toBeLessThan(level(m, 44100, 2.05, 2.4) * 0.5);
+ expect(level(m, 44100, 3.55, 3.9)).toBeLessThan(level(m, 44100, 2.05, 2.4) * 0.65);
+ });
+
+ it('takes a soft or a loud recording where the dynamics make a step soft or loud', () => {
+ const recorded = (hz: number): SampleData => ({ rate: 44100, channels: [Float32Array.from({ length: 22050 }, (_, n) => 0.5 * Math.sin((2 * Math.PI * hz * n) / 44100))] });
+ const m = renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false,
+ instruments: { keys: { samples: [{ file: 'n.wav', note: 'A4' }, { file: 's.wav', note: 'A4', dynamic: 'soft' }] } },
+ patterns: { a: { bars: 2, dynamics: { keys: [0.5, 1] }, tracks: { keys: `1 - - . . . . . . . . . . . . . | 1 - - . . . . . . . . . . . . .` } } }, song: ['a'],
+ }), { 'n.wav': recorded(440), 's.wav': recorded(330) }).channels[0]!;
+ expect(pitchOf(m, 44100, 0.05, 0.3)).toBeCloseTo(330, -1);
+ expect(pitchOf(m, 44100, 2.05, 2.3)).toBeCloseTo(440, -1);
+ });
+
+ it('slurs a note that runs straight on from the last, and strikes one that does not', () => {
+ const source = (line: string, legato: unknown = true) => validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, instruments: { tone: { ...tone, legato } }, patterns: { a: { bars: 1, tracks: { tone: line } } }, song: ['a'],
+ });
+ const slurs = (line: string): string => compileMusic(source(line)).events.map((e) => (e.slur ? 's' : '-') + (e.tied ? 't' : '-')).join(' ');
+ // Four notes end to end: the first is struck, the rest come in under the one before.
+ expect(slurs('1 - - - 2 - - - 3 - - - 4 - - -')).toBe('-t st st s-');
+ // A silence between is a breath taken; an accent is a note struck; a chord is struck.
+ expect(slurs('1 - - . 2 - - - 3! - - - [1 3] - - -')).toBe('-- -- -- -- --');
+ // And an instrument without it strikes every note.
+ expect(compileMusic(source('1 - - - 2 - - - 3 - - - 4 - - -', false)).events.some((e) => e.slur || e.tied)).toBe(false);
+ expect(() => source(EIGHT, 2)).toThrow(/legato must be true, or the seconds one note takes to come in under the last/);
+ });
+
+ it('does not let the sound drop between slurred notes, where struck ones leave a gap', () => {
+ // A slow attack, so a note struck afresh begins from nothing.
+ const bowed = { layers: [{ source: 'sine', freq: 440, attack: 0.08 }], release: 0.02 };
+ const play = (legato: boolean) => renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5, instruments: { tone: { ...bowed, ...(legato ? { legato: 0.06 } : {}) } },
+ patterns: { a: { bars: 1, tracks: { tone: '1 - - - 1 - - - 1 - - - 1 - - -' } } }, song: ['a'],
+ })).channels[0]!;
+ // The quietest fiftieth of a second round each join, against the level in the middle of a note.
+ const dip = (c: Float32Array): number => {
+ let low = Infinity;
+ for (let t = 0.46; t < 0.56; t += 0.01) low = Math.min(low, level(c, 44100, t, t + 0.02));
+ return low / level(c, 44100, 0.2, 0.4);
+ };
+ expect(dip(play(false))).toBeLessThan(0.45);
+ expect(dip(play(true))).toBeGreaterThan(0.7);
+ });
+
+ it('takes a slurred recording from past its attack', () => {
+ // A recording that begins with a second of 880 Hz... of a tenth, and is 440 after: the attack is the high part.
+ const data: SampleData = { rate: 44100, channels: [Float32Array.from({ length: 44100 }, (_, n) => 0.5 * Math.sin((2 * Math.PI * (n < 3528 ? 880 : 440) * n) / 44100))] };
+ const m = renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, instruments: { keys: { samples: [{ file: 'a.wav', note: 'A4' }], legato: 0.05 } },
+ patterns: { a: { bars: 1, tracks: { keys: '1 - - - 1 - - - . . . . . . . .' } } }, song: ['a'],
+ }), { 'a.wav': data }).channels[0]!;
+ // The first note has the attack in it; the second, slurred, has none of it.
+ expect(pitchOf(m, 44100, 0.005, 0.07)).toBeGreaterThan(800);
+ expect(pitchOf(m, 44100, 0.6, 0.9)).toBeCloseTo(440, -1);
+ expect(pitchOf(m, 44100, 0.56, 0.6)).toBeLessThan(500);
+ });
+
+ it('holds every note to the end of its bar under the pedal', () => {
+ const pluck = { layers: [{ source: 'sine', freq: 440, decay: 3 }], release: 0.05 };
+ const source = (pedal: boolean) => validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, loop: false, gain: 0.5, instruments: { keys: { ...pluck, ...(pedal ? { pedal: true } : {}) } },
+ patterns: { a: { bars: 2, tracks: { keys: `1 . . . 3 . . . . . . . . . . . | ${REST}` } } }, song: ['a'],
+ });
+ const e = compileMusic(source(true)).events;
+ // Written a sixteenth each; the first rings two seconds to the bar line, the second a second and a half.
+ expect(e.map((x) => [x.seconds, x.sounds])).toEqual([[0.125, 2], [0.125, 1.5]]);
+ expect(compileMusic(source(false)).events.every((x) => x.sounds === undefined)).toBe(true);
+ const [dry, wet] = [renderMusic(source(false)).channels[0]!, renderMusic(source(true)).channels[0]!];
+ // Half a second after it is let go, the dry note is gone and the pedalled one is still there.
+ expect(level(dry, 44100, 1, 1.4)).toBeLessThan(1e-4);
+ expect(level(wet, 44100, 1, 1.4)).toBeGreaterThan(0.05);
+ // And the pedal is lifted at the bar line: nothing is left in the second bar.
+ expect(level(wet, 44100, 2.3, 3.9)).toBeLessThan(1e-3);
+ });
+
+ it('refuses dynamics that are not a loudness for each bar', () => {
+ const bad = (dynamics: unknown) => () => validateMusicSource({ tempo: 120, key: 'A', instruments: { tone }, patterns: { a: { bars: 2, dynamics, tracks: { tone: `${EIGHT} | ${EIGHT}` } } }, song: ['a'] });
+ expect(bad({ drums: [1] })).toThrow(/dynamics for "drums", which is not an instrument; the instruments are tone, or "all"/);
+ expect(bad({ tone: [1, 1, 1, 1] })).toThrow(/one for each of its 2 bars and at most one more/);
+ expect(bad({ tone: [1, 5] })).toThrow(/0\.1\.\.2 with 1 as written/);
+ expect(bad({ all: [] })).toThrow(/how loud each bar begins/);
+ });
+});
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index 036a518fd..c0b59ffc4 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4634,6 +4634,10 @@ export const NOTED = {
{"note": "Music with an introduction, and a source that fades."},
"feat(music): the reply gives where the round begins for a piece with an introduction, and the tool text says how a game plays one and how a source fades":
{"note": "An AI can make background music."},
+ "feat(music): a line is played and not only sounded - dynamics give each bar the loudness it begins on and move evenly between, legato brings a note in under the one before from past its attack, and pedal holds every note of a piano to the end of its bar":
+ {"note": "An AI can make background music."},
+ "feat(music): the tool text says dynamics, legato and the pedal, and that a piece without them is notes and not playing":
+ {"note": "An AI can make background music."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":