Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(audio): music plays as its layers - playLayers begins every stem of a piece on the same sample, each with a level of its own that is brought up and down while it plays, and plays the whole piece in their place where layers cannot be kept in step

yhh054fe93d9320
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b224fc2c4..59f673fd9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -32,6 +32,7 @@ published separately; it ships inside the editor.
- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.
- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.
- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.
+- **Music plays as its layers.** A piece could be exported as stems, one clip for each instrument, and nothing could play them as one piece: clips started one after another are a few milliseconds apart and stay so. `audio.playLayers({ drums, bass, lead }, { mix, levels, fadeIn, loopStart })` begins every stem on the same sample and loops them together, and what it returns brings a layer up or down while it plays, `music.setLayer('lead', 1, 2)`. It is the background music, replaced by `playBGM` and ended by `stopBGM`. In a browser's audio engine the seven stems of a piece begun this way sum to the whole piece within a ten-thousandth, through the place it goes round and on its third time round; begun a hundredth of a second apart they do not resemble it. Layers that drift are worse than none, so where clips cannot be kept in step (a mini-game, a clip played from its compressed file) the `mix` plays in their place and `music.layered` says so.
- **Music with an introduction, and a source that fades.** Background music looped from its first sample to its last, so a piece could not open with something heard once; and an `AudioSource` in a scene started and stopped at full volume, the fades being `playBGM`'s alone. A looping voice now takes `loopStart`, seconds into the clip that it comes back to at its end: `playBGM(url, { loopStart })` and `AudioSource.loopStart` play the file from its top once and round from there after. It is exact to the sample where the clip is decoded, on the web and on the native host; where a clip is played from its compressed file (a long clip streamed on the web, a mini-game) the player is sent back when it ends, which can be heard as a breath. `AudioSource.fadeIn` and `fadeOut` are seconds: raising `playing` brings it up from nothing, lowering it lets it down before the voice is stopped, and raising it again on the way out turns it round where it is. A piece made with `music_write` marks its first parts as an introduction with `intro`, and the reply gives the `loopStart` to play it with.
- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. There are scales of five notes, `pentatonic`, `pentatonicMinor` and the Chinese modes `gong`, `shang`, `jue`, `zhi` and `yu`, whose degrees count to five so that a line cannot touch the two notes they leave out; and an instrument may have a `scale` of its own, which is how such a tune is made: five notes in the flute over chords that have all seven. Right notes at one loudness, each begun from nothing, are a machine reading a score, so a line can be played: `dynamics` on a pattern give each bar the loudness it begins on and move evenly between them, a crescendo and a diminuendo; `legato` on an instrument brings a note that runs straight on from the last in under it, from past the attack of its recording, as a bow that does not leave the string; and `pedal` holds every note of a piano to the end of its bar. A piece is mixed: an instrument's `lowcut` and `highcut` take off what it has no need of, and `master` holds the loud moments down with `compress` and tilts the whole with `low` and `high`, done to the sum and to each stem alike so they still add up. And `rubato` lets each pattern begin held back, move on and ease off, every instrument together. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. A recording made in stereo comes out of a SoundFont as one file with both its sides, and each zone's level and filter come with it as the sample's `gain` and `cutoff`, so a piano's hundred and eighty zones stay its seventeen recordings. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.
- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. A layer can also be a recording in the project (`source: "sample"`), played at a speed and shaped as any other layer is, for the one sound that will not come out of waves and noise. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.
diff --git a/desktop b/desktop
index 410a7e885..643ab1244 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 410a7e88572af9f094148c8b85a14918c8d8534d
+Subproject commit 643ab1244ad3cb00ef48be2501dd1e397d44d49c
diff --git a/docs/astro/src/content/docs/assets/audio.mdx b/docs/astro/src/content/docs/assets/audio.mdx
index 184ef6f0d..f6cb26e8d 100644
--- a/docs/astro/src/content/docs/assets/audio.mdx
+++ b/docs/astro/src/content/docs/assets/audio.mdx
@@ -46,6 +46,28 @@ const play = defineSystem([Res(Audio)], (audio) => {
current track, optionally cross-fading. Both auto-preload an uncached clip and play
it once ready.
+### Music in layers
+
+A piece exported as stems (one clip for each instrument, all the same length)
+plays as layers: every stem begins on the same sample and they loop together,
+and each has a level of its own.
+
+```typescript
+const music = audio.playLayers(
+ { drums: 'assets/music/theme_drums.wav', bass: 'assets/music/theme_bass.wav', lead: 'assets/music/theme_lead.wav' },
+ { mix: 'assets/music/theme.wav', levels: { lead: 0 }, fadeIn: 1 },
+);
+music.setLayer('lead', 1, 2); // up over two seconds
+music.setLayer('drums', 0, 4); // and the drums away
+```
+
+It is the background music: it takes the place of what `playBGM` was playing,
+and `stopBGM` ends it. Layers are only music while they are in step, which
+needs decoded clips (`loadType: 'decompress'`) on the web or the native host.
+Where each clip is a player of its own (a mini-game, a clip played from its
+compressed file) the `mix` plays instead and `music.layered` is false; with
+no `mix` the layers are started one after another and may drift.
+
### What packaging does to a clip
A `.wav` is re-encoded when a game is exported, and a decoder gives an encoded
diff --git a/docs/astro/src/content/docs/zh-cn/assets/audio.mdx b/docs/astro/src/content/docs/zh-cn/assets/audio.mdx
index 25e04e655..0e50d73d3 100644
--- a/docs/astro/src/content/docs/zh-cn/assets/audio.mdx
+++ b/docs/astro/src/content/docs/zh-cn/assets/audio.mdx
@@ -40,6 +40,26 @@ const play = defineSystem([Res(Audio)], (audio) => {
`playSFX` 触发一次性播放并返回一个句柄;`playBGM` 循环并替换当前曲目,可选交叉淡入。两者都会
自动预加载未缓存的片段并在就绪后播放。
+### 分层音乐
+
+按分轨导出的曲子(每件乐器一个等长的片段)可以分层播放:所有分轨在同一个采样上
+起音并一起循环,每条有自己的音量。
+
+```typescript
+const music = audio.playLayers(
+ { drums: 'assets/music/theme_drums.wav', bass: 'assets/music/theme_bass.wav', lead: 'assets/music/theme_lead.wav' },
+ { mix: 'assets/music/theme.wav', levels: { lead: 0 }, fadeIn: 1 },
+);
+music.setLayer('lead', 1, 2); // 两秒内升起
+music.setLayer('drums', 0, 4); // 鼓慢慢退掉
+```
+
+它就是背景音乐:会替换 `playBGM` 正在播放的曲子,`stopBGM` 会结束它。分层只有在
+各层同步时才是音乐,这需要解码后播放的片段(`loadType: 'decompress'`),在 Web 或
+原生宿主上。每个片段各用一个播放器的平台(小游戏、从压缩文件直接播放的片段)会改播
+`mix` 指定的整首,此时 `music.layered` 为 false;没有给 `mix` 时各层只能先后启动,
+可能逐渐错开。
+
### 打包对音频做了什么
`.wav` 在导出游戏时会被重新编码,而解码器还原出来的片段开头会多出约四十分之一秒
diff --git a/native/host/bindings/AudioBindings.cpp b/native/host/bindings/AudioBindings.cpp
index 4d5516a09..100942436 100644
--- a/native/host/bindings/AudioBindings.cpp
+++ b/native/host/bindings/AudioBindings.cpp
@@ -55,7 +55,13 @@ esjs::Val js_audioUnload(const esjs::Args& a) {
return esjs::undefined();
}
-// es_audioPlay(bufferId, volume, pan, loop, rate, loopStart?, start?, end?) -> voiceId (-1 if unknown).
+// es_audioTime() -> the audio engine's clock, in seconds.
+esjs::Val js_audioTime(const esjs::Args&) {
+ return esjs::number(host().audio.time());
+}
+
+// es_audioPlay(bufferId, volume, pan, loop, rate, loopStart?, start?, end?, at?) -> voiceId (-1 if unknown).
+// at is a moment of es_audioTime that the voice begins on (0 is now): voices given the same one begin together.
// loopStart is seconds a looping voice comes back to in place of its start; start and end are where
// the clip begins and is over in what was decoded (end 0 is its natural end).
esjs::Val js_audioPlay(const esjs::Args& a) {
@@ -65,11 +71,12 @@ esjs::Val js_audioPlay(const esjs::Args& a) {
if (a.size() > 2) pan = esjs::toNumber(a[2]);
const bool loop = a.size() > 3 && esjs::toBool(a[3]);
if (a.size() > 4) rate = esjs::toNumber(a[4]);
- double loopStart = 0, start = 0, end = 0;
+ double loopStart = 0, start = 0, end = 0, at = 0;
if (a.size() > 5) loopStart = esjs::toNumber(a[5]);
if (a.size() > 6) start = esjs::toNumber(a[6]);
if (a.size() > 7) end = esjs::toNumber(a[7]);
- return esjs::int32(host().audio.play(buf, (f32)vol, (f32)pan, loop, (f32)rate, (f32)loopStart, (f32)start, (f32)end));
+ if (a.size() > 8) at = esjs::toNumber(a[8]);
+ return esjs::int32(host().audio.play(buf, (f32)vol, (f32)pan, loop, (f32)rate, (f32)loopStart, (f32)start, (f32)end, at));
}
// The voice-scoped commands all take the voice id as a[0].
@@ -138,7 +145,8 @@ void registerAudioBindings(HostState& h, esjs::Val global) {
if (!h.audio.ready()) return;
bindGlobal(global, "es_audioLoad", js_audioLoad, 2);
bindGlobal(global, "es_audioUnload", js_audioUnload, 1);
- bindGlobal(global, "es_audioPlay", js_audioPlay, 8);
+ bindGlobal(global, "es_audioPlay", js_audioPlay, 9);
+ bindGlobal(global, "es_audioTime", js_audioTime, 0);
bindGlobal(global, "es_audioStop", js_audioStop, 1);
bindGlobal(global, "es_audioPause", js_audioPause, 1);
bindGlobal(global, "es_audioResume", js_audioResume, 1);
diff --git a/native/host/media/native_audio.cpp b/native/host/media/native_audio.cpp
index d7cc45d1c..90d316a59 100644
--- a/native/host/media/native_audio.cpp
+++ b/native/host/media/native_audio.cpp
@@ -310,7 +310,13 @@ void AudioEngine::unload(int bufferId) {
}
}
-int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart, float start, float end) {
+double AudioEngine::time() const {
+ if (!ready()) return 0.0;
+ const ma_uint32 rate = ma_engine_get_sample_rate(&impl_->engine);
+ return rate > 0 ? static_cast<double>(ma_engine_get_time_in_pcm_frames(&impl_->engine)) / rate : 0.0;
+}
+
+int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart, float start, float end, double at) {
if (!ready()) return -1;
auto bit = impl_->buffers.find(bufferId);
if (bit == impl_->buffers.end() || bit->second.unloaded) return -1;
@@ -352,6 +358,11 @@ int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float ra
if (start > 0.0f) ma_sound_seek_to_pcm_frame(&voice.sound, frame(start));
}
}
+ // Held until a moment of the engine's clock, where one is given: started now, it is silent until then.
+ if (at > 0.0) {
+ ma_sound_set_start_time_in_pcm_frames(
+ &voice.sound, static_cast<ma_uint64>(at * ma_engine_get_sample_rate(&impl_->engine)));
+ }
ma_sound_start(&voice.sound);
bit->second.refs++;
return id;
diff --git a/native/host/media/native_audio.hpp b/native/host/media/native_audio.hpp
index 215bd6292..136244012 100644
--- a/native/host/media/native_audio.hpp
+++ b/native/host/media/native_audio.hpp
@@ -62,7 +62,11 @@ public:
/// @p loopStart: seconds a looping voice comes back to at its end, so what is before it is heard once. 0 is its start.
/// @p start, @p end: where the clip begins and is over in what was decoded, in seconds (end 0 is its natural end):
/// a clip the exporter re-encoded has silence before it and after it, and is played without either.
- int play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart = 0.0f, float start = 0.0f, float end = 0.0f);
+ /// @p at: a moment of time() that the voice begins on, 0 for now. Voices given the same moment begin on the same
+ /// frame, which calls arriving one after another cannot otherwise promise.
+ int play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart = 0.0f, float start = 0.0f, float end = 0.0f, double at = 0.0);
+ /// The engine's clock, in seconds since it started.
+ double time() const;
void stop(int voiceId);
void pause(int voiceId);
void resume(int voiceId);
diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.md
index 1197e1aa1..eb9e2c370 100644
--- a/sdk/etc/index.api.md
+++ b/sdk/etc/index.api.md
@@ -1184,6 +1184,7 @@ maxVoices: number
muteBus: (busName: string, muted: boolean) => void
playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => void
playBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandle
+playLayers: (layers: Readonly<Record<string, string>>, config?: MusicLayersConfig) => MusicLayers
playSFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandle
playTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>
preload: (url: string) => Promise<void>
@@ -7985,6 +7986,7 @@ pan: number | undefined
playbackRate: number | undefined
priority: number | undefined
startOffset: number | undefined
+together: object | undefined
volume: number | undefined
```
diff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.md
index 59e4d4aec..323c346cf 100644
--- a/sdk/etc/index.native.api.md
+++ b/sdk/etc/index.native.api.md
@@ -1184,6 +1184,7 @@ maxVoices: number
muteBus: (busName: string, muted: boolean) => void
playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => void
playBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandle
+playLayers: (layers: Readonly<Record<string, string>>, config?: MusicLayersConfig) => MusicLayers
playSFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandle
playTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>
preload: (url: string) => Promise<void>
@@ -6762,7 +6763,7 @@ width: number
```
es_audioLoad: ((bytes: ArrayBuffer, stream?: boolean) => { id: number; duration: number; bytes: number; streamed?: boolean; } | null) | undefined
es_audioPause: ((voiceId: number) => void) | undefined
-es_audioPlay: ((bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number) => number) | undefined
+es_audioPlay: ((bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number, at?: number) => number) | undefined
es_audioResume: ((voiceId: number) => void) | undefined
es_audioResumeAll: (() => void) | undefined
es_audioSetLoop: ((voiceId: number, loop: boolean) => void) | undefined
@@ -6772,6 +6773,7 @@ es_audioSetVolume: ((voiceId: number, volume: number) => void) | undefined
es_audioSpectrum: ((bins: number) => ArrayBuffer | null) | undefined
es_audioStop: ((voiceId: number) => void) | undefined
es_audioSuspendAll: (() => void) | undefined
+es_audioTime: (() => number) | undefined
es_audioUnload: ((bufferId: number) => void) | undefined
es_audioVoiceState: ((voiceId: number) => { playing: boolean; currentTime: number; } | null) | undefined
es_deviceCapabilities: (() => number) | undefined
@@ -8198,6 +8200,7 @@ pan: number | undefined
playbackRate: number | undefined
priority: number | undefined
startOffset: number | undefined
+together: object | undefined
volume: number | undefined
```
diff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.md
index 48c7cfcfe..13da622b3 100644
--- a/sdk/etc/index.node.api.md
+++ b/sdk/etc/index.node.api.md
@@ -1179,6 +1179,7 @@ maxVoices: number
muteBus: (busName: string, muted: boolean) => void
playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => void
playBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandle
+playLayers: (layers: Readonly<Record<string, string>>, config?: MusicLayersConfig) => MusicLayers
playSFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandle
playTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>
preload: (url: string) => Promise<void>
@@ -7980,6 +7981,7 @@ pan: number | undefined
playbackRate: number | undefined
priority: number | undefined
startOffset: number | undefined
+together: object | undefined
volume: number | undefined
```
diff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.md
index b5845ec9d..269ba888a 100644
--- a/sdk/etc/minigame.api.md
+++ b/sdk/etc/minigame.api.md
@@ -1179,6 +1179,7 @@ maxVoices: number
muteBus: (busName: string, muted: boolean) => void
playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => void
playBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandle
+playLayers: (layers: Readonly<Record<string, string>>, config?: MusicLayersConfig) => MusicLayers
playSFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandle
playTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>
preload: (url: string) => Promise<void>
@@ -8277,6 +8278,7 @@ pan: number | undefined
playbackRate: number | undefined
priority: number | undefined
startOffset: number | undefined
+together: object | undefined
volume: number | undefined
```
diff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.md
index 6414313b8..d3dc660e6 100644
--- a/sdk/etc/wechat.api.md
+++ b/sdk/etc/wechat.api.md
@@ -1179,6 +1179,7 @@ maxVoices: number
muteBus: (busName: string, muted: boolean) => void
playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => void
playBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandle
+playLayers: (layers: Readonly<Record<string, string>>, config?: MusicLayersConfig) => MusicLayers
playSFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandle
playTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>
preload: (url: string) => Promise<void>
@@ -7987,6 +7988,7 @@ pan: number | undefined
playbackRate: number | undefined
priority: number | undefined
startOffset: number | undefined
+together: object | undefined
volume: number | undefined
```
diff --git a/sdk/src/audio/Audio.ts b/sdk/src/audio/Audio.ts
index 108a3385f..3c2f29dfc 100644
--- a/sdk/src/audio/Audio.ts
+++ b/sdk/src/audio/Audio.ts
@@ -121,6 +121,58 @@ interface Fade {
stopAtEnd: boolean;
}
+/**
+ * A piece of music playing as its layers: each a clip of the same length, begun on the same sample and going round
+ * together, each with a level of its own that can be brought up or down while it plays. The drums come in when the
+ * fight starts and the melody goes when the player is hiding, and it is one piece of music throughout.
+ */
+export interface MusicLayers {
+ readonly names: readonly string[];
+ /**
+ * False where the layers could not be kept in step and the whole piece is playing in their place (a mini-game, a
+ * clip played from its compressed file): {@link setLayer} then changes nothing that is heard.
+ */
+ readonly layered: boolean;
+ /** The layers began on one sample and stay together. False where they were started one after another for want of a way to do better. */
+ readonly inStep: boolean;
+ /** Bring one layer to @p level, 0..1, over @p seconds (default 1; 0 is at once). */
+ setLayer(name: string, level: number, seconds?: number): void;
+ /** Where a layer is now, on its way or arrived. */
+ layerLevel(name: string): number;
+ stop(fadeOut?: number): void;
+ readonly isPlaying: boolean;
+}
+
+export interface MusicLayersConfig {
+ /**
+ * The whole piece as one clip, played where the layers cannot be kept in step. Layers that drift are worse than no
+ * layers: a mini-game plays each clip through a player of its own, and so does a clip played from its compressed
+ * file. Without it the layers are started one after another there, and a warning says so.
+ */
+ mix?: string;
+ volume?: number;
+ /** Where each layer starts, 0..1. A layer not named starts at 1. */
+ levels?: Readonly<Record<string, number>>;
+ fadeIn?: number;
+ crossFade?: number;
+ /** As for playBGM: seconds every layer comes back to at its end, for a piece with an introduction. */
+ loopStart?: number;
+}
+
+/** The layers of the music that is playing. */
+interface LayerGroup {
+ names: string[];
+ voices: Map<string, AudioHandle>;
+ levels: Map<string, number>;
+ targets: Map<string, number>;
+ /** Level a second each layer moves at toward its target. */
+ rates: Map<string, number>;
+ volume: number;
+ stopped: boolean;
+ layered: boolean;
+ inStep: boolean;
+}
+
export class AudioAPI {
private readonly voiceStates_ = new WeakMap<AudioHandle, SoftVoice>();
private readonly playbackStops_ = new Set<(generation: number) => void>();
@@ -141,6 +193,9 @@ export class AudioAPI {
private loadTypes_ = new Map<string, AudioLoadType>();
private encoded_ = new Map<string, AudioClipTrim>();
private bgmHandle_: AudioHandle | null = null;
+ /** The music that is playing, where it is playing as layers. One or the other, never both. */
+ private bgmGroup_: LayerGroup | null = null;
+ private layersWarned_ = false;
private bgmRequest_ = 0;
private playbackEpoch_ = 0;
private bgmVolume_ = 1.0;
@@ -593,11 +648,7 @@ export class AudioAPI {
const oldVolume = this.bgmVolume_;
this.bgmVolume_ = targetVolume;
- if (this.bgmHandle_ && config?.crossFade) {
- this.fadeOut_(this.bgmHandle_, config.crossFade, oldVolume);
- } else if (this.bgmHandle_) {
- this.bgmHandle_.stop();
- }
+ this.retireMusic_(config?.crossFade, oldVolume);
const fadeInDuration = config?.fadeIn ?? config?.crossFade;
this.bgmHandle_ = this.playVoice_(buffer, {
@@ -620,12 +671,135 @@ export class AudioAPI {
}
}
+ /** What was playing as music goes: at once, or down over @p fade seconds. Its layers go with it. */
+ private retireMusic_(fade: number | undefined, volume: number): void {
+ const out = (handle: AudioHandle, from: number): void => {
+ if (fade && fade > 0) this.fadeOut_(handle, fade, from);
+ else handle.stop();
+ };
+ if (this.bgmHandle_) out(this.bgmHandle_, volume);
+ this.bgmHandle_ = null;
+ const group = this.bgmGroup_;
+ this.bgmGroup_ = null;
+ if (group) {
+ group.stopped = true;
+ for (const [name, handle] of group.voices) out(handle, group.volume * (group.levels.get(name) ?? 1));
+ }
+ }
+
+ /**
+ * Background music as layers: every clip begun on the same sample and looping together, each with a level of its
+ * own. `layers` names them, `{ drums: 'a_drums.wav', melody: 'a_melody.wav' }`, and they must be as long as one
+ * another, which the stems of one piece are. It takes the place of whatever music was playing, as playBGM does,
+ * and stopBGM ends it.
+ *
+ * Layers are only music while they are in step. Where the backend decodes its clips they are, to the sample; where
+ * it cannot promise it and `mix` names the whole piece as one clip, that plays instead.
+ */
+ playLayers(layers: Readonly<Record<string, string>>, config: MusicLayersConfig = {}): MusicLayers {
+ const request = ++this.bgmRequest_;
+ const names = Object.keys(layers);
+ const level = (v: number | undefined): number => Math.max(0, Math.min(1, v ?? 1));
+ const group: LayerGroup = {
+ names, voices: new Map(), rates: new Map(), volume: config.volume ?? 1, stopped: false, layered: true, inStep: true,
+ levels: new Map(names.map((n) => [n, level(config.levels?.[n])])),
+ targets: new Map(names.map((n) => [n, level(config.levels?.[n])])),
+ };
+ const api = this;
+
+ const start = (): void => {
+ if (this.disposed_ || request !== this.bgmRequest_ || group.stopped) return;
+ const buffers = names.map((n) => this.lookupBuffer_(layers[n]!));
+ if (buffers.some((b) => !b)) {
+ log.warn('audio', `music layers did not play: ${names.filter((_, i) => !buffers[i]).join(', ')} loaded, but gone from the audio cache before they could start`);
+ return;
+ }
+ // In step where each clip is decoded and the backend starts voices on a clock; not where each is a player of its own.
+ const inStep = this.backend_.delivery === 'bytes' && buffers.every((b) => !b!.streamed);
+ if (!inStep && config.mix) {
+ group.layered = false;
+ group.inStep = false;
+ this.playBGM(config.mix, { volume: group.volume, fadeIn: config.fadeIn, crossFade: config.crossFade, loopStart: config.loopStart });
+ return;
+ }
+ if (!inStep && !this.layersWarned_) {
+ this.layersWarned_ = true;
+ log.warn('audio', 'music layers cannot be kept in step here (clips are played from their files); they are started one after another and may drift. Give playLayers a `mix` to play the whole piece instead');
+ }
+ this.cancelFades_();
+ const oldVolume = this.bgmVolume_;
+ this.bgmVolume_ = group.volume;
+ this.retireMusic_(config.crossFade, oldVolume);
+ // What was asked of a layer before it began is where it begins.
+ for (const n of names) group.levels.set(n, group.targets.get(n)!);
+ const together = {};
+ const fadeIn = config.fadeIn ?? config.crossFade;
+ names.forEach((name, i) => {
+ const volume = group.volume * group.levels.get(name)!;
+ const handle = this.playVoice_(buffers[i]!, {
+ bus: 'music', volume, loop: true, together,
+ ...(config.loopStart && config.loopStart > 0 ? { loopStart: config.loopStart } : {}),
+ });
+ group.voices.set(name, handle);
+ if (fadeIn) this.fadeIn_(handle, fadeIn, volume);
+ });
+ group.inStep = inStep;
+ this.bgmGroup_ = group;
+ };
+
+ const missing = names.filter((n) => !this.lookupBuffer_(layers[n]!));
+ if (missing.length === 0) start();
+ else {
+ void Promise.all(missing.map((n) => this.preload(layers[n]!)))
+ .then(start, (err) => log.warn('audio', `Failed to preload music layers: ${missing.map((n) => layers[n]).join(', ')}`, err));
+ }
+
+ return {
+ names,
+ get layered() { return group.layered; },
+ get inStep() { return group.inStep; },
+ setLayer(name, to, seconds = 1) {
+ if (!group.targets.has(name)) return;
+ const target = level(to);
+ group.targets.set(name, target);
+ const from = group.levels.get(name)!;
+ if (!(seconds > 0) || group.voices.size === 0) {
+ group.levels.set(name, target);
+ group.rates.delete(name);
+ api.applyLayer_(group, name);
+ } else group.rates.set(name, Math.abs(target - from) / seconds);
+ },
+ layerLevel(name) { return group.levels.get(name) ?? 0; },
+ stop(fadeOut) {
+ group.stopped = true;
+ // Only if it is still the music: a piece that has since been replaced is not stopped a second time.
+ if (api.bgmGroup_ === group || (!group.layered && request + 1 === api.bgmRequest_)) api.stopBGM(fadeOut);
+ },
+ get isPlaying() {
+ if (!group.layered) return api.bgmHandle_?.isPlaying ?? false;
+ for (const handle of group.voices.values()) if (handle.isPlaying) return true;
+ return false;
+ },
+ };
+ }
+
+ /** A layer's level written to its voice: the piece's volume times the layer's own. */
+ private applyLayer_(group: LayerGroup, name: string): void {
+ const handle = group.voices.get(name);
+ const voice = handle && this.voiceStates_.get(handle);
+ if (!voice) return;
+ voice.base = group.volume * group.levels.get(name)!;
+ if (voice.paused || voice.handle.isPlaying) this.applyVoice_(voice);
+ }
+
stopAll(): void {
this.bgmRequest_++;
const generation = ++this.playbackEpoch_;
const voices = this.softVoices_;
this.softVoices_ = [];
this.bgmHandle_ = null;
+ if (this.bgmGroup_) this.bgmGroup_.stopped = true;
+ this.bgmGroup_ = null;
this.fades_.length = 0;
this.fadingOut_.clear();
const stopped = new Set<AudioHandle>();
@@ -660,19 +834,12 @@ export class AudioAPI {
stopBGM(fadeOut?: number): void {
this.bgmRequest_++;
- if (!this.bgmHandle_) {
+ if (!this.bgmHandle_ && !this.bgmGroup_) {
if (!(fadeOut && fadeOut > 0)) this.cancelFades_();
return;
}
this.cancelFades_();
- if (fadeOut && fadeOut > 0) {
- const handle = this.bgmHandle_;
- this.bgmHandle_ = null;
- this.fadeOut_(handle, fadeOut, this.bgmVolume_);
- } else {
- this.bgmHandle_.stop();
- this.bgmHandle_ = null;
- }
+ this.retireMusic_(fadeOut, this.bgmVolume_);
}
/**
@@ -990,7 +1157,19 @@ export class AudioAPI {
* browser's: it pauses when the game does, and it exists on a device.
*/
updateFades(dt: number): void {
- if (this.suspended_ || this.fades_.length === 0 || !(dt > 0)) return;
+ if (this.suspended_ || !(dt > 0)) return;
+ // The layers of the music, each on its way to where it was last told to be.
+ const group = this.bgmGroup_;
+ if (group && group.rates.size > 0) {
+ for (const [name, rate] of group.rates) {
+ const [now, target] = [group.levels.get(name)!, group.targets.get(name)!];
+ const next = now < target ? Math.min(target, now + rate * dt) : Math.max(target, now - rate * dt);
+ group.levels.set(name, next);
+ if (next === target) group.rates.delete(name);
+ this.applyLayer_(group, name);
+ }
+ }
+ if (this.fades_.length === 0) return;
for (let i = this.fades_.length - 1; i >= 0; i--) {
const fade = this.fades_[i];
if (fade.voice.paused) continue;
diff --git a/sdk/src/audio/NativeAudioBackend.ts b/sdk/src/audio/NativeAudioBackend.ts
index 5c89d0f5c..6985667aa 100644
--- a/sdk/src/audio/NativeAudioBackend.ts
+++ b/sdk/src/audio/NativeAudioBackend.ts
@@ -74,6 +74,7 @@ export class NativeAudioBackend implements PlatformAudioBackend {
private readonly voices_ = new Map<number, NativeAudioHandle>();
/** Where a clip the exporter re-encoded really lies in what the host decoded, by buffer. */
private readonly spans_ = new Map<number, { lead: number; end: number }>();
+ private readonly together_ = new WeakMap<object, number>();
/** Reused across frames: a visualizer asks every frame, and a fresh view per
* ask would be garbage the interpreter has to collect at 60 Hz. */
private spectrumView_: Uint8Array | null = null;
@@ -123,6 +124,13 @@ export class NativeAudioBackend implements PlatformAudioBackend {
play(buffer: AudioBufferHandle, config: PlayConfig): AudioHandle {
const span = this.spans_.get(buffer.id);
+ // Voices of one token begin at one moment of the host's own clock, a twentieth of a second off: the calls that
+ // start them arrive one after another, and the device may run between two of them.
+ let at = 0;
+ if (config.together && this.audio_.time) {
+ at = this.together_.get(config.together) ?? this.audio_.time() + 0.05;
+ this.together_.set(config.together, at);
+ }
const voiceId = this.audio_.play(
buffer.id,
config.volume ?? 1,
@@ -133,6 +141,7 @@ export class NativeAudioBackend implements PlatformAudioBackend {
config.loop && config.loopStart && config.loopStart > 0 ? (span?.lead ?? 0) + config.loopStart : span?.lead ?? 0,
span?.lead ?? 0,
span?.end ?? 0,
+ at,
);
const handle = new NativeAudioHandle(
voiceId, this.audio_, buffer.duration, () => this.voices_.delete(voiceId),
diff --git a/sdk/src/audio/PlatformAudioBackend.ts b/sdk/src/audio/PlatformAudioBackend.ts
index 3621f99fa..8e67c94af 100644
--- a/sdk/src/audio/PlatformAudioBackend.ts
+++ b/sdk/src/audio/PlatformAudioBackend.ts
@@ -83,6 +83,13 @@ export interface PlayConfig {
* mini-game) the player is sent back when it ends, which can be heard as a breath.
*/
loopStart?: number;
+ /**
+ * Voices played with the same token begin on the same sample: the layers of one piece of music, which are only
+ * that piece while they are in step. The token is any object, held for as long as voices are still being added
+ * under it. A backend that plays each clip through a player of its own (a mini-game, a streamed clip) cannot
+ * promise it and starts them one after another.
+ */
+ together?: object;
}
export interface AudioBackendInitOptions {
diff --git a/sdk/src/audio/WebAudioBackend.ts b/sdk/src/audio/WebAudioBackend.ts
index 4f136a9b2..1bf2c6b43 100644
--- a/sdk/src/audio/WebAudioBackend.ts
+++ b/sdk/src/audio/WebAudioBackend.ts
@@ -43,6 +43,7 @@ class WebAudioHandle implements AudioHandle {
loopStart = 0,
lead = 0,
end = buffer.duration,
+ startedAt = context.currentTime,
) {
this.id = id;
this.source_ = source;
@@ -54,7 +55,7 @@ class WebAudioHandle implements AudioHandle {
this.pool_ = pool;
this.context_ = context;
this.startOffset_ = startOffset;
- this.startedAt_ = context.currentTime;
+ this.startedAt_ = startedAt;
this.playbackRate_ = source.playbackRate.value;
this.loop_ = source.loop;
this.loopStart_ = loopStart;
@@ -604,6 +605,9 @@ export class WebAudioBackend implements PlatformAudioBackend {
return this.handleOf_(id, this.buffers_.get(id)!);
}
+ /** The moment each token's voices begin at, for as long as the token is held. */
+ private readonly together_ = new WeakMap<object, number>();
+
private releaseStream_(handle: WebStreamHandle): void {
for (const [id, voice] of this.streams_) if (voice === handle) this.streams_.delete(id);
this.heldStreams_ = this.heldStreams_.filter((h) => h !== handle);
@@ -688,6 +692,13 @@ export class WebAudioBackend implements PlatformAudioBackend {
const loopStart = lead + (config.loop && config.loopStart && config.loopStart > 0 && lead + config.loopStart < end ? config.loopStart : 0);
// Round a part of the buffer and not the whole of it: from a loop start, or within a clip that has silence round it.
const region = !!config.loop && (loopStart > 0 || end < audioBuffer.duration);
+ // Voices of one token are told the same moment a little way off, so each begins on the same sample however
+ // long it takes to set the next one up; a voice on its own begins as soon as it can.
+ let when = 0;
+ if (config.together) {
+ when = this.together_.get(config.together) ?? this.context_.currentTime + 0.03;
+ this.together_.set(config.together, when);
+ }
const poolNode = this.pool_.acquire();
let source: AudioBufferSourceNode;
try {
@@ -707,8 +718,8 @@ export class WebAudioBackend implements PlatformAudioBackend {
const bus = this.mixer_.getBus(busName) ?? this.mixer_.sfx;
poolNode.panner.connect(bus.input);
// A clip with silence after it is stopped where it ends, so its end is told on time.
- if (record!.span && !config.loop) source.start(0, startOffset, Math.max(0, end - startOffset));
- else source.start(0, startOffset);
+ if (record!.span && !config.loop) source.start(when, startOffset, Math.max(0, end - startOffset));
+ else source.start(when, startOffset);
} catch (error) {
this.pool_.release(poolNode);
throw error;
@@ -716,7 +727,8 @@ export class WebAudioBackend implements PlatformAudioBackend {
const handleId = ++this.nextHandleId_;
const handle = new WebAudioHandle(
- handleId, source, audioBuffer, poolNode, this.pool_, this.context_, startOffset, loopStart, lead, end
+ handleId, source, audioBuffer, poolNode, this.pool_, this.context_, startOffset, loopStart, lead, end,
+ when || this.context_.currentTime,
);
return handle;
diff --git a/sdk/src/audio/index.ts b/sdk/src/audio/index.ts
index fd95e3f33..760de2905 100644
--- a/sdk/src/audio/index.ts
+++ b/sdk/src/audio/index.ts
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
-export { Audio, AudioAPI, type AudioBufferStats } from './Audio';
+export { Audio, AudioAPI, type AudioBufferStats, type MusicLayers, type MusicLayersConfig } from './Audio';
export { SoundEvents, SoundEventPlayback, type SoundEventEmitter, type SoundEventHandle,
type SoundEventStatus, type SoundEventRejection, type SoundEventStats } from './SoundEventPlayback';
export { parseSoundEvent, SoundEventSelector, type SoundEventAsset, type SoundEventClip,
diff --git a/sdk/src/platform/native/bridge.ts b/sdk/src/platform/native/bridge.ts
index 597d44fcc..513735c4b 100644
--- a/sdk/src/platform/native/bridge.ts
+++ b/sdk/src/platform/native/bridge.ts
@@ -149,7 +149,10 @@ export interface NativeAudioBridge {
* is unknown. */
/** @param loopStart seconds a looping voice comes back to; 0 is its start. A host older than this takes five arguments and loops from the start. */
/** @param start, end seconds in the decoded clip that it begins from and is over at (0 for end is its natural end): a clip the exporter re-encoded has silence before and after it. */
- play(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number): number;
+ /** @param at a moment of the host's clock, as {@link time} gives it, that the voice begins on; 0 is now. Voices given the same one begin on the same sample. */
+ play(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number, at?: number): number;
+ /** The host's audio clock, in seconds. Absent on a host older than this, where voices begin as they are asked for. */
+ time?(): number;
stop(voiceId: number): void;
pause(voiceId: number): void;
resume(voiceId: number): void;
diff --git a/sdk/src/platform/native/hostBridge.ts b/sdk/src/platform/native/hostBridge.ts
index dabdd26b2..1edaebe0e 100644
--- a/sdk/src/platform/native/hostBridge.ts
+++ b/sdk/src/platform/native/hostBridge.ts
@@ -144,7 +144,8 @@ export interface NativeHostBindings extends NativeDeviceHost {
es_audioSpectrum?(bins: number): ArrayBuffer | null;
es_audioLoad?(bytes: ArrayBuffer, stream?: boolean): { id: number; duration: number; bytes: number; streamed?: boolean } | null;
es_audioUnload?(bufferId: number): void;
- es_audioPlay?(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number): number;
+ es_audioPlay?(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number, at?: number): number;
+ es_audioTime?(): number;
es_audioStop?(voiceId: number): void;
es_audioPause?(voiceId: number): void;
es_audioResume?(voiceId: number): void;
@@ -388,7 +389,8 @@ function hostAudio(
return {
load: (bytes, stream) => bindings.es_audioLoad!(bytes, stream === true),
unload: (id) => bindings.es_audioUnload!(id),
- play: (buf, vol, pan, loop, rate, loopStart, start, end) => bindings.es_audioPlay!(buf, vol, pan, loop, rate, loopStart ?? 0, start ?? 0, end ?? 0),
+ play: (buf, vol, pan, loop, rate, loopStart, start, end, at) => bindings.es_audioPlay!(buf, vol, pan, loop, rate, loopStart ?? 0, start ?? 0, end ?? 0, at ?? 0),
+ ...(bindings.es_audioTime ? { time: () => bindings.es_audioTime!() } : {}),
stop: (v) => bindings.es_audioStop!(v),
pause: (v) => bindings.es_audioPause!(v),
resume: (v) => bindings.es_audioResume!(v),
diff --git a/sdk/tests/audio-music-layers.test.ts b/sdk/tests/audio-music-layers.test.ts
new file mode 100644
index 000000000..ddd184398
--- /dev/null
+++ b/sdk/tests/audio-music-layers.test.ts
@@ -0,0 +1,161 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file audio-music-layers.test.ts
+ * @brief A piece of music played as its layers: begun together, each with a level of its own.
+ *
+ * Held here: that every layer is asked of the backend under one token (which is what a backend starts on one
+ * sample), that a layer's level moves with the frames and reaches the voice, that the layers are the music
+ * for everything that starts, stops and replaces music, and that where they cannot be kept in step the whole
+ * piece plays in their place rather than layers that drift.
+ */
+import { describe, it, expect, vi } from 'vitest';
+import { AudioAPI } from '../src/audio/Audio';
+import type { AudioHandle, PlatformAudioBackend, PlayConfig } from '../src/audio/PlatformAudioBackend';
+
+type Voice = AudioHandle & { isPlaying: boolean; volume: number; url: string; config: PlayConfig; stop: ReturnType<typeof vi.fn> };
+
+function audioOf(delivery: 'bytes' | 'url' = 'bytes', streamed: string[] = []) {
+ const urls = new Map<number, string>();
+ const voices: Voice[] = [];
+ const backend = {
+ name: 'Mock', mixer: null, isReady: true, delivery,
+ initialize: vi.fn().mockResolvedValue(undefined), ensureResumed: vi.fn().mockResolvedValue(undefined),
+ loadBuffer: vi.fn(async (src: string) => { const id = urls.size + 1; urls.set(id, src); return { id, duration: 8, ...(streamed.includes(src) ? { streamed: true } : {}) }; }),
+ loadBufferFromData: vi.fn(), unloadBuffer: vi.fn(), suspend: vi.fn(), resume: vi.fn(), dispose: vi.fn(),
+ play: vi.fn((buffer: { id: number }, config: PlayConfig) => {
+ const v = {
+ id: voices.length + 1, pause: vi.fn(), resume: vi.fn(), setPan: vi.fn(), setLoop: vi.fn(), setPlaybackRate: vi.fn(),
+ isPlaying: true, currentTime: 0, duration: 8, volume: config.volume ?? 1, url: urls.get(buffer.id)!, config,
+ stop: vi.fn(() => { v.isPlaying = false; }), setVolume: vi.fn((x: number) => { v.volume = x; }),
+ } as unknown as Voice;
+ voices.push(v);
+ return v;
+ }),
+ } as unknown as PlatformAudioBackend;
+ const audio = new AudioAPI(backend);
+ const of = (url: string): Voice => voices.filter((v) => v.url === url).at(-1)!;
+ return { audio, voices, of };
+}
+const STEMS = { drums: 'a_drums.wav', bass: 'a_bass.wav', lead: 'a_lead.wav' };
+const loaded = async (audio: AudioAPI, ...more: string[]): Promise<void> => { await audio.preloadAll([...Object.values(STEMS), ...more]); };
+const frames = (audio: AudioAPI, n: number, dt = 0.1): void => { for (let i = 0; i < n; i++) audio.updateFades(dt); };
+
+describe('music played as its layers', () => {
+ it('asks for every layer under one token, round and round on the music bus, each at its own level', async () => {
+ const { audio, voices, of } = audioOf();
+ await loaded(audio);
+ const music = audio.playLayers(STEMS, { volume: 0.8, levels: { lead: 0 }, loopStart: 4 });
+ expect(voices.map((v) => v.url)).toEqual(['a_drums.wav', 'a_bass.wav', 'a_lead.wav']);
+ // One token for all three: that is what a backend begins on one sample.
+ expect(voices[0]!.config.together).toBeTypeOf('object');
+ expect(new Set(voices.map((v) => v.config.together)).size).toBe(1);
+ for (const v of voices) expect(v.config).toMatchObject({ bus: 'music', loop: true, loopStart: 4 });
+ expect([of('a_drums.wav').volume, of('a_lead.wav').volume]).toEqual([0.8, 0]);
+ expect([music.layered, music.inStep, music.isPlaying, music.names]).toEqual([true, true, true, ['drums', 'bass', 'lead']]);
+ // Another piece is another token.
+ audio.playLayers(STEMS);
+ expect(voices[3]!.config.together).not.toBe(voices[0]!.config.together);
+ });
+
+ it('brings a layer up and down over the seconds asked, with the frames', async () => {
+ const { audio, of } = audioOf();
+ await loaded(audio);
+ const music = audio.playLayers(STEMS, { levels: { lead: 0 } });
+ music.setLayer('lead', 1, 2);
+ frames(audio, 5);
+ expect(music.layerLevel('lead')).toBeCloseTo(0.25, 9);
+ expect(of('a_lead.wav').volume).toBeCloseTo(0.25, 9);
+ frames(audio, 20);
+ expect(of('a_lead.wav').volume).toBe(1);
+ // Arrived, it is left alone.
+ const writes = (of('a_lead.wav').setVolume as ReturnType<typeof vi.fn>).mock.calls.length;
+ frames(audio, 5);
+ expect((of('a_lead.wav').setVolume as ReturnType<typeof vi.fn>).mock.calls.length).toBe(writes);
+ // Turned round on its way, it goes from where it is; at once when no time is given it.
+ music.setLayer('lead', 0, 1);
+ frames(audio, 5);
+ expect(of('a_lead.wav').volume).toBeCloseTo(0.5, 9);
+ music.setLayer('lead', 0.3, 0);
+ expect(of('a_lead.wav').volume).toBeCloseTo(0.3, 9);
+ // The others were never touched, and a layer that is not one changes nothing.
+ expect(of('a_drums.wav').volume).toBe(1);
+ music.setLayer('strings', 1);
+ expect(music.layerLevel('strings')).toBe(0);
+ });
+
+ it('starts where it was last told to be, when it was told before its clips were loaded', async () => {
+ const { audio, voices, of } = audioOf();
+ const music = audio.playLayers(STEMS);
+ expect(voices).toHaveLength(0);
+ music.setLayer('drums', 0.2, 3);
+ await vi.waitFor(() => expect(voices).toHaveLength(3));
+ expect(of('a_drums.wav').volume).toBeCloseTo(0.2, 9);
+ expect(new Set(voices.map((v) => v.config.together)).size).toBe(1);
+ });
+
+ it('is the music: stopBGM ends it, playBGM takes its place, and it takes the place of what was playing', async () => {
+ const { audio, voices, of } = audioOf();
+ await loaded(audio, 'other.wav');
+ audio.playBGM('other.wav');
+ const music = audio.playLayers(STEMS);
+ expect(of('other.wav').stop).toHaveBeenCalledTimes(1);
+ // Down over a second, every layer, and stopped at nothing.
+ audio.stopBGM(1);
+ frames(audio, 5);
+ expect(of('a_bass.wav').volume).toBeCloseTo(0.5, 9);
+ expect(of('a_bass.wav').stop).not.toHaveBeenCalled();
+ frames(audio, 6);
+ for (const url of Object.values(STEMS)) expect(of(url).stop).toHaveBeenCalledTimes(1);
+ expect(music.isPlaying).toBe(false);
+ // A single piece after layers stops every layer.
+ const second = audio.playLayers(STEMS);
+ audio.playBGM('other.wav');
+ expect(voices.filter((v) => v.url !== 'other.wav').every((v) => !v.isPlaying)).toBe(true);
+ expect(second.isPlaying).toBe(false);
+ // And its own stop does not reach the piece that has replaced it.
+ second.stop();
+ expect(of('other.wav').isPlaying).toBe(true);
+ });
+
+ it('comes in over a fade as one piece, each layer to its own level', async () => {
+ const { audio, of } = audioOf();
+ await loaded(audio);
+ audio.playLayers(STEMS, { fadeIn: 1, levels: { bass: 0.5 } });
+ expect([of('a_drums.wav').volume, of('a_bass.wav').volume]).toEqual([0, 0]);
+ frames(audio, 5);
+ expect(of('a_drums.wav').volume).toBeCloseTo(0.5, 9);
+ expect(of('a_bass.wav').volume).toBeCloseTo(0.25, 9);
+ });
+
+ it('plays the whole piece instead where layers cannot be kept in step', async () => {
+ // Every clip a player of its own, as on a mini-game.
+ const { audio, voices, of } = audioOf('url');
+ await loaded(audio, 'a.wav');
+ const music = audio.playLayers(STEMS, { mix: 'a.wav', volume: 0.7, loopStart: 4 });
+ expect(voices.map((v) => v.url)).toEqual(['a.wav']);
+ expect(of('a.wav').config).toMatchObject({ bus: 'music', loop: true, loopStart: 4, volume: 0.7 });
+ expect([music.layered, music.inStep, music.isPlaying]).toEqual([false, false, true]);
+ // There are no layers to move.
+ music.setLayer('lead', 0, 0);
+ expect(of('a.wav').volume).toBe(0.7);
+ music.stop();
+ expect(of('a.wav').stop).toHaveBeenCalledTimes(1);
+ });
+
+ it('does the same where one of the layers is played from its compressed file', async () => {
+ const { audio, voices } = audioOf('bytes', ['a_bass.wav']);
+ await loaded(audio, 'a.wav');
+ const music = audio.playLayers(STEMS, { mix: 'a.wav' });
+ expect(voices.map((v) => v.url)).toEqual(['a.wav']);
+ expect(music.layered).toBe(false);
+ });
+
+ it('starts them one after another, and says they are not in step, where there is no whole piece to play', async () => {
+ const { audio, voices } = audioOf('url');
+ await loaded(audio);
+ const music = audio.playLayers(STEMS);
+ expect(voices).toHaveLength(3);
+ expect([music.layered, music.inStep]).toEqual([true, false]);
+ });
+});
diff --git a/sdk/tests/native-audio.test.ts b/sdk/tests/native-audio.test.ts
index 75f46449d..55cedb39f 100644
--- a/sdk/tests/native-audio.test.ts
+++ b/sdk/tests/native-audio.test.ts
@@ -28,16 +28,18 @@ function hostGlobals(): Record<string, unknown> {
/** A fake native audio engine, installed as the host's es_audio* primitives. */
function installAudioHost(scope: Record<string, unknown>) {
- const calls = { play: [] as unknown[][], stop: [] as number[], setVolume: [] as unknown[][], suspend: 0, resume: 0 };
+ const calls = { play: [] as unknown[][], at: [] as number[], stop: [] as number[], setVolume: [] as unknown[][], suspend: 0, resume: 0 };
let nextBuffer = 1;
let nextVoice = 100;
const voices = new Map<number, { playing: boolean; time: number; loop: boolean }>();
scope.es_audioLoad = (bytes: ArrayBuffer) => ({ id: nextBuffer++, duration: 2, bytes: bytes.byteLength * 4 });
scope.es_audioUnload = () => {};
- scope.es_audioPlay = (buf: number, vol: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number) => {
+ scope.es_audioPlay = (buf: number, vol: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number, at?: number) => {
const id = nextVoice++;
voices.set(id, { playing: true, time: 0, loop });
+ // The ninth is the moment of the host's clock a voice begins on, kept apart so the others read as they did.
+ if (at) calls.at.push(at);
// The sixth is where a loop comes back to, the seventh and eighth where the clip begins and ends; recorded only when they are somewhere.
calls.play.push(start || end ? [buf, vol, pan, loop, rate, loopStart, start, end] : loopStart ? [buf, vol, pan, loop, rate, loopStart] : [buf, vol, pan, loop, rate]);
return id;
@@ -171,6 +173,28 @@ describe('NativeAudioBackend', () => {
expect(host.calls.play.at(-1)).toEqual([exact.id, 1, 0, true, 1, 0, 0, close(2)]);
});
+ it('begins voices of one token at one moment of the host clock, and asks nothing of a host that has no clock', async () => {
+ const scope = audioScope();
+ const host = installAudioHost(scope);
+ let now = 10;
+ scope.es_audioTime = () => now;
+ const backend = new NativeAudioBackend(createHostBridge(scope).audio!);
+ const buffer = await backend.loadBufferFromData('stem', new ArrayBuffer(16));
+ const together = {};
+ backend.play(buffer, { loop: true, together });
+ now = 10.02;
+ backend.play(buffer, { loop: true, together });
+ backend.play(buffer, { loop: true });
+ // Both at a twentieth of a second past when the first was asked for; the third, with no token, now.
+ expect(host.calls.at).toEqual([expect.closeTo(10.05, 9), expect.closeTo(10.05, 9)]);
+
+ const old = audioScope();
+ const oldHost = installAudioHost(old);
+ const plain = new NativeAudioBackend(createHostBridge(old).audio!);
+ plain.play(await plain.loadBufferFromData('stem', new ArrayBuffer(16)), { loop: true, together: {} });
+ expect(oldHost.calls.at).toEqual([]);
+ });
+
it('loads, plays and reports state through the bridge', async () => {
const scope = audioScope();
const host = installAudioHost(scope); // capture the same closure the bridge will use
diff --git a/sdk/tests/web-audio-backend.test.ts b/sdk/tests/web-audio-backend.test.ts
index 0c38a496e..1b4b4913b 100644
--- a/sdk/tests/web-audio-backend.test.ts
+++ b/sdk/tests/web-audio-backend.test.ts
@@ -490,6 +490,27 @@ describe('WebAudioBackend', () => {
expect(handle.currentTime).toBeCloseTo(.875);
});
+ it('begins voices of one token at one moment, however long apart they were asked for', async () => {
+ const { bufferHandle, mockCtx } = await initAndLoadBuffer(backend);
+ const together = {};
+ mockCtx.currentTime = 5;
+ const first = backend.play(bufferHandle, { loop: true, together });
+ // The clock has moved on by the time the second is set up: it is told the same moment all the same.
+ mockCtx.currentTime = 5.02;
+ backend.play(bufferHandle, { loop: true, together });
+ const sources = mockCtx.createBufferSource.mock.results.slice(-2).map((r: any) => r.value);
+ expect(sources[0].start).toHaveBeenCalledWith(expect.closeTo(5.03, 9), 0);
+ expect(sources[1].start).toHaveBeenCalledWith(expect.closeTo(5.03, 9), 0);
+ // Time is counted from that moment, not from when it was asked for.
+ mockCtx.currentTime = 5.53;
+ expect(first.currentTime).toBeCloseTo(0.5, 9);
+ // Another token is another moment, and a voice with none begins as soon as it can.
+ backend.play(bufferHandle, { together: {} });
+ expect(mockCtx.createBufferSource.mock.results.at(-1).value.start).toHaveBeenCalledWith(expect.closeTo(5.56, 9), 0);
+ backend.play(bufferHandle, {});
+ expect(mockCtx.createBufferSource.mock.results.at(-1).value.start).toHaveBeenCalledWith(0, 0);
+ });
+
describe('a clip the exporter re-encoded', () => {
// Two seconds of sound that came back from the decoder with a fortieth of a second of silence before it and some after.
const trim = { seconds: 2, lead: 0.025 };
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index 4c58051cc..b1f4c0863 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4656,6 +4656,10 @@ export const NOTED = {
{"note": "A loop is seamless in a packaged game."},
"feat(music): a piece that goes round is set to be decoded when a game loads it - played from its compressed file it could not loop between two samples, and the reply says what it costs in memory":
{"note": "An AI can make background music."},
+ "feat(audio): music plays as its layers - playLayers begins every stem of a piece on the same sample, each with a level of its own that is brought up and down while it plays, and plays the whole piece in their place where layers cannot be kept in step":
+ {"note": "Music plays as its layers."},
+ "feat(music): the reply and the tool text say how a game plays a piece's stems as layers":
+ {"note": "An AI can make background music."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":