Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(music): a recording called a note it is not is heard and said - the pitch of each of an instrument's recordings is found, and one that is not the note it is named, or a whole library an octave out, is a warning with the note to set

yhh67bef61d44df
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c78d5140c..9bb1ee0ab 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,7 +22,7 @@ published separately; it ships inside the editor.
- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.
- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.
- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.
-- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. The engine ships no recordings of its own: they are the project's.
+- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. The engine ships no recordings of its own: they are the project's.
- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.
- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.
- **A model is written once for all its like parts, and a leaf can be a picture.** A model's source was a flat list of steps with plain numbers: a plant of ten fronds wrote out a frond ten times, and every leaf was a thin solid in one colour. Three things change that. `params` names numbers and points, and a value written `"=length * 0.4"` is a sum over them, with points that add and scale, angles in degrees, and makers for what shapes keep asking for: `polar` for a point round the upright axis, `arc` for a line that bends from one lean to another, `part` for a share of a line. `{ op: "repeat", over: [rows], steps: [...] }` runs its steps once for each row, the row's fields being names inside it, and `let` names a value for the steps after it; what is saved and edited is the source as written, so one edit to a name changes every part. The `strip` step carries a band with no thickness along a path, shaped by `widths`, folded by `cup`, with a picture laid along it from root to tip. `material.texture` names that picture, and with `alpha: "cutout"` its alpha is the outline: a leaf with veins and a torn edge, drawn once. `loft` takes an `up` for the way its section stands. The palm that was 386 steps is six steps written in two models of 2.9 and 2.4 KB, and 9,150 triangles where it was 26,814. Drawn in the engine it cost a little more GPU than the solid one all the same (about 1.1 ms against 0.8 for the frame at 1024 by 1024 on an RTX 3060): cut-out leaves draw both sides and test every pixel.
diff --git a/desktop b/desktop
index aa28b943f..fd5c7d7dd 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit aa28b943f9467948770de2bae061e548305a5b95
+Subproject commit fd5c7d7dd7ce297e9caede9e76768d552f58cf64
diff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.ts
index 22ffd963d..e2b1f3421 100644
--- a/pipeline/src/music/musicRender.ts
+++ b/pipeline/src/music/musicRender.ts
@@ -223,6 +223,31 @@ function sampleNote(name: string, instrument: MusicInstrument, midi: number | nu
return out;
}
+/**
+ * The pitch heard in a recording, in Hz, looked for within two octaves of @p near; null where it has none that can be
+ * told (a drum, a noise). It is found by how far the wave has to be slid to lie on itself again. A wave lies on itself
+ * at twice its period too, so the shortest slide that fits nearly as well as the best is taken: that is what tells a
+ * recording named an octave from what it is, which libraries that count octaves from another C are throughout.
+ */
+export function samplePitch(data: SampleData, near: number): number | null {
+ const samples = data.channels[0]!;
+ const longest = Math.ceil(data.rate / (near / 2.1));
+ const start = Math.floor(samples.length * 0.25);
+ const size = Math.min(16384, samples.length - start - longest);
+ if (size < longest * 2) return null;
+ const scores: [number, number][] = [];
+ for (let lag = Math.max(8, Math.floor(data.rate / (near * 4.2))); lag <= longest; lag++) {
+ let sum = 0, a = 0, b = 0;
+ for (let n = 0; n < size; n += 2) { const x = samples[start + n]!, y = samples[start + n + lag]!; sum += x * y; a += x * x; b += y * y; }
+ scores.push([lag, sum / Math.sqrt(a * b + 1e-12)]);
+ }
+ const top = Math.max(...scores.map((score) => score[1]));
+ if (!(top > 0.6)) return null;
+ let at = scores.findIndex((score) => score[1] >= top * 0.92);
+ while (at + 1 < scores.length && scores[at + 1]![1] > scores[at]![1]) at++;
+ return data.rate / scores[at]![0];
+}
+
const TAIL_SECONDS = 4;
/**
@@ -347,6 +372,29 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,
let peak = 0;
for (const channel of channels) for (const v of channel) if (Math.abs(v) > peak) peak = Math.abs(v);
const notes = [...compiled.notes];
+ // A recording called a note it is not puts every note played from it out of tune, and nothing in the notes shows it.
+ for (const name of names) {
+ const wrong: { file: string; said: number; is: number }[] = [];
+ const zones = source.instruments[name]!.samples ?? [];
+ for (const zone of zones) {
+ const data = given[zone.file];
+ if (!data) continue;
+ const said = midiOf(zone.note)!;
+ const hz = 440 * Math.pow(2, (said - 69) / 12);
+ const found = samplePitch(data, hz);
+ const off = found === null ? 0 : 12 * Math.log2(found / hz);
+ if (Math.abs(off) >= 0.7) wrong.push({ file: zone.file, said, is: Math.round(said + off) });
+ }
+ if (wrong.length === 0) continue;
+ const by = (w: { said: number; is: number }): number => w.is - w.said;
+ const semitones = (n: number): string => `${Math.abs(n)} semitone${Math.abs(n) === 1 ? '' : 's'} ${n > 0 ? 'higher' : 'lower'}`;
+ // A whole library out by the same amount is one thing to say, and nearly always an octave: it counts its octaves from another C.
+ if (wrong.length === zones.length && wrong.length > 1 && wrong.every((w) => by(w) === by(wrong[0]!))) {
+ notes.push(`"${name}": every one of its ${zones.length} recordings sounds ${semitones(by(wrong[0]!))} than the note it is said to be (${wrong[0]!.file} is said to be ${noteName(wrong[0]!.said)} and sounds as ${noteName(wrong[0]!.is)})${Math.abs(by(wrong[0]!)) === 12 ? ', as a library does that counts its octaves from another C' : ''}: the instrument plays that much out. Move each \`note\` by that much`);
+ } else {
+ for (const w of wrong) notes.push(`"${name}": ${w.file} is said to be ${noteName(w.said)} and sounds as ${noteName(w.is)}: every note played from it is ${semitones(by(w))} than written. Set its note to "${noteName(w.is)}"`);
+ }
+ }
const target = source.gain ?? 0.9;
const scale = peak > 1e-6 ? target / peak : 1;
if (peak <= 1e-6) notes.push('the piece is silent: nothing in it reaches a level that can be heard');
diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.ts
index cf0223bb8..88b2eb6db 100644
--- a/pipeline/tests/music.test.ts
+++ b/pipeline/tests/music.test.ts
@@ -6,7 +6,7 @@
*/
import { describe, expect, it } from 'vitest';
import { applyMusicEdits, midiOf, musicSampleFiles, validateMusicSource, type MusicSource } from '../src/music/musicSource';
-import { compileMusic, noteName, renderMusic, type SampleData } from '../src/music/musicRender';
+import { compileMusic, noteName, renderMusic, samplePitch, type SampleData } from '../src/music/musicRender';
import { musicPicture } from '../src/music/musicPicture';
const tone = { layers: [{ source: 'sine', freq: 440 }] } as const;
@@ -292,6 +292,29 @@ describe('an instrument made of recordings', () => {
expect([midiOf('C4'), midiOf('A4'), midiOf('F#3'), midiOf('Bb2'), midiOf('C-1'), midiOf('H2'), midiOf(128)]).toEqual([60, 69, 54, 46, 0, null, null]);
});
+ it('hears the pitch of a recording, and says when it is not the note it is called', () => {
+ // A tone with overtones, as an instrument has: the pitch is the lowest of them, not the loudest.
+ const rich = (hz: number): SampleData => ({ rate: 44100, channels: [Float32Array.from({ length: 44100 }, (_, n) => { const t = (2 * Math.PI * hz * n) / 44100; return 0.2 * Math.sin(t) + 0.3 * Math.sin(2 * t) + 0.15 * Math.sin(3 * t); })] });
+ expect(samplePitch(rich(220), 220)).toBeCloseTo(220, -0.5);
+ // Told it is an octave lower or higher than it is, it still finds what it is.
+ expect(samplePitch(rich(220), 110)).toBeCloseTo(220, -0.5);
+ expect(samplePitch(rich(220), 440)).toBeCloseTo(220, -0.5);
+ let seed = 12345;
+ const noise = { rate: 44100, channels: [Float32Array.from({ length: 44100 }, () => { seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0; return seed / 2147483648 - 1; })] };
+ expect(samplePitch(noise, 220)).toBeNull();
+
+ const notes = (samples: unknown[], files: Record<string, SampleData>): string[] => renderMusic(validateMusicSource({
+ tempo: 120, key: 'A', humanize: 0, instruments: { keys: { samples } }, patterns: { a: { bars: 1, tracks: { keys: one('1') } } }, song: ['a'],
+ }), files).notes;
+ const files = { 'a.wav': rich(220), 'e.wav': rich(329.63) };
+ expect(notes([{ file: 'a.wav', note: 'A3' }, { file: 'e.wav', note: 'E4' }], files)).toEqual([]);
+ expect(notes([{ file: 'a.wav', note: 'A3' }, { file: 'e.wav', note: 'D4' }], files)).toEqual(['"keys": e.wav is said to be D4 and sounds as E4: every note played from it is 2 semitones higher than written. Set its note to "E4"']);
+ // A library that counts its octaves from another C is out everywhere, and that is said once.
+ const octave = notes([{ file: 'a.wav', note: 'A2' }, { file: 'e.wav', note: 'E3' }], files);
+ expect(octave).toHaveLength(1);
+ expect(octave[0]).toMatch(/every one of its 2 recordings sounds 12 semitones higher than the note it is said to be \(a\.wav is said to be A2 and sounds as A3\), as a library does that counts its octaves from another C/);
+ });
+
it('refuses samples that are not well formed', () => {
const bad = (instrument: unknown) => () => validateMusicSource({ tempo: 120, key: 'A', instruments: { keys: instrument }, patterns: { a: { bars: 1, tracks: { keys: one('1') } } }, song: ['a'] });
expect(bad({})).toThrow(/instrument "keys" needs layers \(a made sound\) or samples \(recordings\), one of the two/);
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index eacc0b202..a905e40aa 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4614,6 +4614,10 @@ export const NOTED = {
{"note": "An AI can make background music."},
"feat(music): music_write plays instruments made of recordings in the project - the tool text says how many are needed and which loop, and the editor check plays one":
{"note": "An AI can make background music."},
+ "feat(music): a recording called a note it is not is heard and said - the pitch of each of an instrument's recordings is found, and one that is not the note it is named, or a whole library an octave out, is a warning with the note to set":
+ {"note": "An AI can make background music."},
+ "docs(music): the tool text says the reply listens to each recording and that recordings differ in level":
+ {"note": "An AI can make background music."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":