A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(audio): music with an introduction, and a source that fades - a looping voice comes back to loopStart and not to its start, on the web, a mini-game and the native host, from playBGM or an AudioSource; an AudioSource comes up over fadeIn and goes down over fadeOut before it stops; and a piece of music marks its first parts as an intro and says where the round begins
yhh9af7fb4fdec6
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 95a6fef19..d3cf902b8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -23,6 +23,7 @@ published separately; it ships inside the editor.- **A flower turns to face the eye.** A flat flower faces the sky, so a field of them is seen edge on from a few metres up and has no colour a field away; tipping each far plant's head at random (`farTilt`) showed half of them from behind. `Foliage.faceEyeNear` and `Foliage.faceEye` turn the head of every mesh plant toward the eye, by the first close to and by the second from half the draw distance away, as a clump of flowers shows its colour from the side. It is done the way a pivot painted into a uv is elsewhere: the copies carry in a second uv where each vertex of a head sits from its plant's axis, and a material's vertex hook now reads a mesh's second uv as `es_vertexUv1` (zero on a mesh with none), beside `es_vertexColor`. At 0 and 0, the default, no channel is added and nothing turns.- **A far field of flowers shows its colour.** A mesh plant kept in a far version grows across the ground to stand for those left out. A flat flower grown that way is a disc facing the sky: from an eye a few metres up it is seen edge on, and a meadow that is all flowers at your feet was plain green a field away. `Foliage.farTilt` tips the top of each far plant over by so many degrees (the last tenth of its height: the flower, not its stem), each plant its own way round, so half the field shows its face. 0, the default, leaves a far plant as it was.- **A layer with a grain is read without repeat and without hexagons.** Near the eye a terrain layer is read in hexagonal cells, each shifted and turned at random, so no tile shows. A layer with a direction to it (sand in ripples, rock in beds) turned cell by cell has its normal map lit cell by cell, and the ground showed a mosaic of light and dark hexagons a metre wide. `u_antiTileTurn` on the Terrain and Terrain Layers materials is how far round a cell may be turned: 1, the default, is any way as before; 0 shifts the cell and does not turn it, which is the setting for a layer with a grain.+- **Music with an introduction, and a source that fades.** Background music looped from its first sample to its last, so a piece could not open with something heard once; and an `AudioSource` in a scene started and stopped at full volume, the fades being `playBGM`'s alone. A looping voice now takes `loopStart`, seconds into the clip that it comes back to at its end: `playBGM(url, { loopStart })` and `AudioSource.loopStart` play the file from its top once and round from there after. It is exact to the sample where the clip is decoded, on the web and on the native host; where a clip is played from its compressed file (a long clip streamed on the web, a mini-game) the player is sent back when it ends, which can be heard as a breath. `AudioSource.fadeIn` and `fadeOut` are seconds: raising `playing` brings it up from nothing, lowering it lets it down before the voice is stopped, and raising it again on the way out turns it round where it is. A piece made with `music_write` marks its first parts as an introduction with `intro`, and the reply gives the `loopStart` to play it with.- **An AI can make background music.** A sound effect is layers; a piece of music is notes, and writing it as layers would be writing every note as a wave. A piece is now an `.esmusic`: instruments, each the layers of a sound written at the A of 440 Hz and struck at the pitch of every note; patterns of a few bars, each track a line of steps (`1 - - . 5, - 3'`, `[1 3 5]`, `x` for a drum) written as degrees of the key, so a line cannot leave it; chords that a bass or a pad follows, written once and moving with them; and the order the patterns come in, with instruments left out of a part so a piece builds. It makes a `.wav` that runs into itself, what rings past its end added to its start, and with `stems` a file for each instrument that add up to the whole, for music that grows with the game. A loop that joins without a click can still be heard to join, where a full last bar runs into a thin first one: the reply says what the last bar and the first hold and warns when they differ. A piece with `loop: false` has an end: it rings out past its last bar, and `fadeIn` and `fadeOut` bring it up and down; a loop has no fade written into it, since the game fades one with `playBGM` and `stopBGM`. Whoever writes it cannot hear it, so `music_write`, `music_edit` and `music_read` answer with a picture of the notes, every one a bar in its instrument's colour, and with what can be said of them unheard: two lines crowding one octave, notes outside the key, how loud each instrument is beside the whole. Instruments of layers are made sounds, which suit electronic, retro and ambient music. A piano or strings that pass for real ones are recordings: an instrument may instead have `samples`, a recording in the project for each of a few notes, and every note of the piece is played from the recording nearest it, faster to be higher and slower to be lower; what dies of itself ends with its recording, and what is held goes round the steady middle of it with `loop`. Each recording is listened to: one whose pitch is not the note it is called, or a whole library an octave out because it counts its octaves from another C, is a warning that names the note to set, since nothing in the notes would show an instrument out of tune. A sample marked `dynamic: "soft"` or `"loud"` plays the steps marked `?` or `!`, since a piano struck hard is brighter and not only louder; several samples of one note take turns, so a note repeated is not one recording repeated; and `tune` takes out how sharp a recording is. A SoundFont (`.sf2`) put in a project is a bank of such instruments: `soundfont_read` lists what is in it and takes one out as `.wav` files and the samples that play them, each with its note, range, tuning and loop, after which the SoundFont is not needed. The engine ships no recordings of its own: they are the project's.- **An AI can make a sound effect.** The engine played recordings and nothing here made one: a game built by an agent had pictures and models of its own and no sounds. A sound is now layers, as a picture is: `sound_write` takes a source of tones (sine, triangle, saw, square), noise (white, pink, brown) and `modes`, the tones a struck thing rings with, each placed in time and shaped by a level, a decay, filters whose corner can move, a second tone bending it, drive, pan and repeats, with an echo and a room over the whole, and writes an `.essound` and the `.wav` a scene plays. Whoever writes one cannot hear it, so the reply is a picture: the wave on top and pitch against time under it, a thump a blot low on the left, a hiss a haze across the top, a falling tone a falling line, with the level and the share of the energy that is low, middle and high. `sound_edit` changes a number and makes it again; `loop` runs the end into the start for an engine or a wind. A gunshot, footsteps, breaking glass, a pickup, an explosion, an engine loop and a laser are 150 to 500 bytes of source each and are made in under a tenth of a second. The same source is the same file on every machine. The first sounds made this way were listened to and did not pass: one noise through one filter with one decay is heard at once as a machine. So a layer can now be what a real sound is made of: noise cut to a `band` and brought back to full level, several of which die at different rates behind an `impulse`; a `tail` that lets part of a decay linger; `scatter`, which strikes a layer hundreds of times at random in time, pitch, loudness and side, for glass, gravel and rain; `repeat` with `faster`, for a thing bouncing to rest; a `peak` filter for the note a body adds; and `reverb` as a place, `room` or `open`, made by laying the sound over what that place answers to a click, with distinct first echoes and a wash whose top dies before its bottom. A layer can also be a recording in the project (`source: "sample"`), played at a speed and shaped as any other layer is, for the one sound that will not come out of waves and noise. The tool text gives the method and the numbers for a shot, glass, a step and an explosion; the step is a shoe on a floor, measured against a recording: a blow on something that rings briefly, heel then ball of the foot, a scrape, nothing under 110 Hz, and a room round it. A `group` is several layers struck together as one event, so four footsteps are one step with `repeat` (787 bytes of source where it was 24 layers), each striking with its own grit. `variants` makes up to eight files that are the same sound and not the same samples, their pitch, loudness, decay and timing off by `vary`, with a sound event beside them that plays one after another without repeating: a game that plays one file for every shot is heard as a machine however good the file. And `reference` names a recording to be like, which is drawn above the made sound in every reply, the same way and to one scale of time, and measured the same way; `sound_read` reads a recording on its own.- **An AI can draw a surface: a picture that repeats, with what the light does on it.** `image_write` drew one picture, and a picture laid beside itself shows its edges; nor is a wall only its colour. `tile: true` on an image source makes it repeat: a shape, a line or a brush stroke that leaves one edge comes in at the opposite one, a shadow and a blur go round with it, and a noise fill meets itself. `material: { relief, height, roughness, metal }` makes it a surface: each of height, roughness and metal is layers drawn as a grey picture of their own (light is high, rough, metal), or the colour's own lightness stands in for the relief, and a normal map, a map of roughness and metal and an `.esmaterial` are written beside the picture. The reply shows the picture laid beside itself two by two, where a join would run through the middle, and the same under a low light, and gives the fields to put on a mesh. A brick wall, a plank floor and a riveted plate, 256 pixels each, drew in under a third of a second apiece and tiled three across in the engine with no join. A `filter` still stops at the edges, and a noise turned or stretched by `transform` does not come round.diff --git a/desktop b/desktopindex f60bc1230..b554a9fb4 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit f60bc12300a662c52016b603db2d4936bfd0dae3+Subproject commit b554a9fb4765945dc38aa7593520638602cd076adiff --git a/docs/astro/src/content/docs/assets/audio.mdx b/docs/astro/src/content/docs/assets/audio.mdxindex 312f57b7d..3ebcceb38 100644--- a/docs/astro/src/content/docs/assets/audio.mdx+++ b/docs/astro/src/content/docs/assets/audio.mdx@@ -46,6 +46,21 @@ const play = defineSystem([Res(Audio)], (audio) => {current track, optionally cross-fading. Both auto-preload an uncached clip and playit once ready.+### An introduction, then the loop++A piece with an introduction is one file: it plays from the top once, and each+time it reaches its end it comes back to `loopStart` seconds in.++```typescript+audio.playBGM('assets/theme.ogg', { loopStart: 12.5, fadeIn: 1 });+```++The join is exact to the sample where the clip is decoded (the web's decoded+clips, and the native host). Where a clip is played from its compressed file (a+long clip streamed on the web, a mini-game) the player is sent back when it+ends, which can be heard as a short breath: decode a piece whose join matters+(`loadType: 'decompress'`).+## Sound event definitions (beta)A `.esaudioevent` is a versioned JSON asset containing `clips` with audio references@@ -215,6 +230,9 @@ the audio system starts it automatically.| `volume` | number | `1` | Volume, 0..1. || `pitch` | number | `1` | Playback speed / pitch. || `loop` | boolean | `false` | Loop the clip. |+| `loopStart` | number | `0` | With `loop`: seconds into the clip that it comes back to at its end, so an introduction plays once. |+| `fadeIn` | number | `0` | Seconds to come up to `volume` when `playing` is raised. |+| `fadeOut` | number | `0` | Seconds to go down to nothing when `playing` is lowered; the voice stops after. Raising `playing` again on the way out turns it round. || `playing` | boolean | `false` | The play flag: raise to play, lower to stop. The engine lowers it when a non-looping clip ends. || `finished` | boolean | `false` | Runtime, read-only: the clip completed; raising `playing` plays it again. || `playOnAwake` | boolean | `false` | Start automatically when the entity is enabled. |diff --git a/docs/astro/src/content/docs/zh-cn/assets/audio.mdx b/docs/astro/src/content/docs/zh-cn/assets/audio.mdxindex d9de785ea..25a645a2c 100644--- a/docs/astro/src/content/docs/zh-cn/assets/audio.mdx+++ b/docs/astro/src/content/docs/zh-cn/assets/audio.mdx@@ -40,6 +40,19 @@ const play = defineSystem([Res(Audio)], (audio) => {`playSFX` 触发一次性播放并返回一个句柄;`playBGM` 循环并替换当前曲目,可选交叉淡入。两者都会自动预加载未缓存的片段并在就绪后播放。+### 前奏放一次,然后循环++带前奏的曲子是一个文件:从头放一遍,之后每次放到结尾都回到 `loopStart` 秒处,+前奏只听到一次。++```typescript+audio.playBGM('assets/theme.ogg', { loopStart: 12.5, fadeIn: 1 });+```++解码后播放的片段(Web 上解码的片段、原生宿主)衔接精确到采样。从压缩文件直接+播放的片段(Web 上流式播放的长片段、小游戏)是在放完时把播放器送回去,可能听到+一个很短的停顿:衔接要紧的曲子请解码播放(`loadType: 'decompress'`)。+## 音效事件定义(beta)`.esaudioevent` 是带有 `version: 1` 的 JSON 资源,`clips` 保存音频引用和相对 `weight`。@@ -188,6 +201,9 @@ shot.onEnd = () => { /* the sound finished */ };| `volume` | number | `1` | 音量,0..1。 || `pitch` | number | `1` | 播放速度 / 音调。 || `loop` | boolean | `false` | 循环片段。 |+| `loopStart` | number | `0` | 配合 `loop`:放到结尾时回到片段的第几秒,前奏因此只放一次。 |+| `fadeIn` | number | `0` | `playing` 置为真时,音量升到 `volume` 所用的秒数。 |+| `fadeOut` | number | `0` | `playing` 置为假时,音量降到零所用的秒数,之后才停止;淡出途中再次置真会原地回升。 || `playing` | boolean | `false` | 播放标志:升起即播放,放下即停止。非循环片段播完时引擎会把它放下。 || `finished` | boolean | `false` | 运行时只读:片段已播完;再次升起 `playing` 会重新播放。 || `playOnAwake` | boolean | `false` | 实体启用时自动开始。 |diff --git a/docs/astro/src/data/components.generated.json b/docs/astro/src/data/components.generated.jsonindex 94ad61689..768d64831 100644--- a/docs/astro/src/data/components.generated.json+++ b/docs/astro/src/data/components.generated.json@@ -122,6 +122,27 @@"type": "boolean","default": "false"},+ {+ "key": "loopStart",+ "type": "number",+ "default": "0",+ "doc": "With Loop: seconds into the clip that it comes back to at its end, so an introduction plays once. 0 is the start.",+ "range": "≥ 0, step 0.1"+ },+ {+ "key": "fadeIn",+ "type": "number",+ "default": "0",+ "doc": "Seconds to come up to Volume when Playing is raised. 0 is at once.",+ "range": "≥ 0, step 0.1"+ },+ {+ "key": "fadeOut",+ "type": "number",+ "default": "0",+ "doc": "Seconds to go down to nothing when Playing is lowered; it stops after. 0 is at once.",+ "range": "≥ 0, step 0.1"+ },{"key": "playing","type": "boolean",@@ -179,6 +200,9 @@"animatable": ["volume","pitch",+ "loopStart",+ "fadeIn",+ "fadeOut","minDistance","maxDistance","attenuationModel",diff --git a/native/host/bindings/AudioBindings.cpp b/native/host/bindings/AudioBindings.cppindex c3fb714c8..bb2557d60 100644--- a/native/host/bindings/AudioBindings.cpp+++ b/native/host/bindings/AudioBindings.cpp@@ -55,7 +55,8 @@ esjs::Val js_audioUnload(const esjs::Args& a) {return esjs::undefined();}-// es_audioPlay(bufferId, volume, pan, loop, rate) -> voiceId (-1 if unknown).+// es_audioPlay(bufferId, volume, pan, loop, rate, loopStart?) -> voiceId (-1 if unknown).+// loopStart is seconds a looping voice comes back to in place of its start.esjs::Val js_audioPlay(const esjs::Args& a) {int32_t buf = esjs::toInt32(a[0]);double vol = 1, pan = 0, rate = 1;@@ -63,7 +64,9 @@ esjs::Val js_audioPlay(const esjs::Args& a) {if (a.size() > 2) pan = esjs::toNumber(a[2]);const bool loop = a.size() > 3 && esjs::toBool(a[3]);if (a.size() > 4) rate = esjs::toNumber(a[4]);- return esjs::int32(host().audio.play(buf, (f32)vol, (f32)pan, loop, (f32)rate));+ double loopStart = 0;+ if (a.size() > 5) loopStart = esjs::toNumber(a[5]);+ return esjs::int32(host().audio.play(buf, (f32)vol, (f32)pan, loop, (f32)rate, (f32)loopStart));}// The voice-scoped commands all take the voice id as a[0].@@ -132,7 +135,7 @@ void registerAudioBindings(HostState& h, esjs::Val global) {if (!h.audio.ready()) return;bindGlobal(global, "es_audioLoad", js_audioLoad, 2);bindGlobal(global, "es_audioUnload", js_audioUnload, 1);- bindGlobal(global, "es_audioPlay", js_audioPlay, 5);+ bindGlobal(global, "es_audioPlay", js_audioPlay, 6);bindGlobal(global, "es_audioStop", js_audioStop, 1);bindGlobal(global, "es_audioPause", js_audioPause, 1);bindGlobal(global, "es_audioResume", js_audioResume, 1);diff --git a/native/host/media/native_audio.cpp b/native/host/media/native_audio.cppindex 03a8faa1a..6f0a46929 100644--- a/native/host/media/native_audio.cpp+++ b/native/host/media/native_audio.cpp@@ -310,7 +310,7 @@ void AudioEngine::unload(int bufferId) {}}-int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rate) {+int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart) {if (!ready()) return -1;auto bit = impl_->buffers.find(bufferId);if (bit == impl_->buffers.end() || bit->second.unloaded) return -1;@@ -338,6 +338,16 @@ int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rama_sound_set_pan(&voice.sound, pan);ma_sound_set_pitch(&voice.sound, rate > 0 ? rate : 1.0f);ma_sound_set_looping(&voice.sound, loop ? MA_TRUE : MA_FALSE);+ if (loop && loopStart > 0.0f) {+ // Round from here and not from the top. The point is in the clip's own frames, which is what the data+ // source counts in; its end is left where the clip ends.+ ma_uint32 sampleRate = 0;+ ma_data_source* source = ma_sound_get_data_source(&voice.sound);+ if (source && ma_sound_get_data_format(&voice.sound, nullptr, nullptr, &sampleRate, nullptr, 0) == MA_SUCCESS && sampleRate > 0) {+ ma_data_source_set_loop_point_in_pcm_frames(+ source, static_cast<ma_uint64>(static_cast<double>(loopStart) * sampleRate), ~static_cast<ma_uint64>(0));+ }+ }ma_sound_start(&voice.sound);bit->second.refs++;return id;diff --git a/native/host/media/native_audio.hpp b/native/host/media/native_audio.hppindex 78ed24a01..a553f5b35 100644--- a/native/host/media/native_audio.hpp+++ b/native/host/media/native_audio.hpp@@ -59,7 +59,8 @@ public:bool spectrum(uint8_t* out, size_t bins);/** Start a voice on a loaded buffer. Returns its id, or -1 if unknown. */- int play(int bufferId, float volume, float pan, bool loop, float rate);+ /// @p loopStart: seconds a looping voice comes back to at its end, so what is before it is heard once. 0 is its start.+ int play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart = 0.0f);void stop(int voiceId);void pause(int voiceId);void resume(int voiceId);diff --git a/pipeline/src/music/musicRender.ts b/pipeline/src/music/musicRender.tsindex 43d977ae8..b301b992a 100644--- a/pipeline/src/music/musicRender.ts+++ b/pipeline/src/music/musicRender.ts@@ -49,6 +49,8 @@ export interface RenderedMusic extends CompiledMusic {channels: Float32Array[];peak: number;rms: number;+ /** For a loop with an introduction: seconds into the file that it comes back to. 0 where the whole of it goes round. */+ loopStart: number;/** For a piece that ends: seconds it rings on after its last bar, which the file is longer by. */tailSeconds: number;/** For a loop: what sounds in its last bar and in its first, and how loud each is, in decibels beside the whole. Null for a piece that ends. */@@ -415,6 +417,9 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,}const loop = source.loop !== false;+ // Where the round begins: the start, or the end of an introduction that is heard once.+ const loopStart = loop && source.intro ? compiled.parts[source.intro]!.at : 0;+ const roundFrom = Math.round(loopStart * rate);// A piece that ends rings out: the file runs on past its last bar until what is left cannot be heard. One that fades// out is at nothing by the end of the bar, and has nothing to ring.let total = count;@@ -425,7 +430,8 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,}const finish = (channel: Float32Array): Float32Array => {const out = channel.slice(0, total);- if (loop) for (let n = count; n < work; n++) out[(n - count) % count]! += channel[n]!;+ // What rings past the end is heard as the round begins again, which is where it is put.+ if (loop) for (let n = count; n < work; n++) out[roundFrom + ((n - count) % (count - roundFrom))]! += channel[n]!;else {if (source.fadeIn) { const rise = Math.round(source.fadeIn * rate); for (let n = 0; n < rise && n < total; n++) out[n]! *= (n / rise) ** 2; }// Let down over the seconds asked, or over the last fiftieth of one so that it does not stop on a step.@@ -491,12 +497,13 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,};const db = (level: number): number => (level > 0 && rms > 0 ? Math.round(20 * Math.log10(level / rms) * 10) / 10 : -120);join = {- ends: sounding(compiled.seconds - compiled.barSeconds, compiled.seconds), begins: sounding(0, compiled.barSeconds),- endDb: db(rmsOf(channels, count - bar, count)), startDb: db(rmsOf(channels, 0, bar)),+ // The first bar of the round, which with an introduction is not the first bar of the file.+ ends: sounding(compiled.seconds - compiled.barSeconds, compiled.seconds), begins: sounding(loopStart, loopStart + compiled.barSeconds),+ endDb: db(rmsOf(channels, count - bar, count)), startDb: db(rmsOf(channels, roundFrom, Math.min(count, roundFrom + bar))),};const [gone, come] = [join.ends.filter((n) => !join!.begins.includes(n)), join.begins.filter((n) => !join!.ends.includes(n))];- if (count > bar * 2 && (Math.abs(join.endDb - join.startDb) > 6 || gone.length + come.length > Math.max(1, names.length / 3))) {- notes.push(`where the loop meets itself the music changes at once: its last bar has ${join.ends.join(', ') || 'nothing'} at ${join.endDb} dB and its first has ${join.begins.join(', ') || 'nothing'} at ${join.startDb} dB${gone.length ? `, so ${gone.join(', ')} stop${gone.length === 1 ? 's' : ''} dead` : ''}. A loop is heard as one piece when its end is as full as its start: end on a part like the first (thin the last part with \`without\`, or begin as full as it ends), and put a build that happens once in a piece of its own with loop false`);+ if (count - roundFrom > bar * 2 && (Math.abs(join.endDb - join.startDb) > 6 || gone.length + come.length > Math.max(1, names.length / 3))) {+ notes.push(`where the loop meets itself the music changes at once: its last bar has ${join.ends.join(', ') || 'nothing'} at ${join.endDb} dB and its first has ${join.begins.join(', ') || 'nothing'} at ${join.startDb} dB${gone.length ? `, so ${gone.join(', ')} stop${gone.length === 1 ? 's' : ''} dead` : ''}. A loop is heard as one piece when its end is as full as its start: end on a part like the one it comes back to (thin the last part with \`without\`, or begin as full as it ends), and put a build that happens once in an introduction with \`intro\``);}}@@ -511,5 +518,5 @@ export function renderMusic(source: MusicSource, given: Readonly<Record<string,levelDb: level > 0 && rms > 0 ? Math.round(20 * Math.log10(level / rms) * 10) / 10 : -120,};}- return { ...compiled, notes, rate, channels, peak: peak > 1e-6 ? target : 0, rms, tailSeconds: (total - count) / rate, join, stems: source.stems ? stems : null, instruments };+ return { ...compiled, notes, rate, channels, peak: peak > 1e-6 ? target : 0, rms, loopStart, tailSeconds: (total - count) / rate, join, stems: source.stems ? stems : null, instruments };}diff --git a/pipeline/src/music/musicSource.ts b/pipeline/src/music/musicSource.tsindex 7a2330675..eb45fdbd2 100644--- a/pipeline/src/music/musicSource.ts+++ b/pipeline/src/music/musicSource.ts@@ -118,6 +118,13 @@ export interface MusicSource {* with an end: its last notes and the room ring out after the last bar, and the file is that much longer.*/loop?: boolean;+ /**+ * How many of the song's first parts are an introduction: heard once, after which the rest goes round. The file+ * is still one file; what rings past its end comes in where the introduction ends, and the reply gives that place+ * as `loopStart`, which the game hands to playBGM or sets on the AudioSource. This is where a build belongs+ * that a loop cannot hold: two instruments growing to all of them, once.+ */+ intro?: number;/** Seconds over which it comes up from nothing at its start. A loop comes round to its start again and again, so only a piece that ends has one; a game fades a loop in as it starts it. */fadeIn?: number;/** Seconds before the end of the last bar over which it goes down to nothing, for a piece that ends and has no ending written. */@@ -199,7 +206,7 @@ export function parseTrack(text: string, bars: number, stepsToBar: number, wherereturn out;}-const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'fadeIn', 'fadeOut', 'stems', 'reverb', 'gain', 'rate'];+const SOURCE_FIELDS = ['version', 'tempo', 'beats', 'key', 'scale', 'swing', 'humanize', 'seed', 'instruments', 'patterns', 'song', 'loop', 'intro', 'fadeIn', 'fadeOut', 'stems', 'reverb', 'gain', 'rate'];const INSTRUMENT_FIELDS = ['layers', 'samples', 'attack', 'gain', 'pan', 'octave', 'follow', 'release', 'reverb'];const RECORDING = /\.(wav|ogg|mp3|flac|m4a|aac|webm)$/i;@@ -316,6 +323,10 @@ export function validateMusicSource(raw: unknown): MusicSource {});const source = s as unknown as MusicSource;const seconds = musicSeconds(source);+ if (s.intro !== undefined) {+ if (s.loop === false) throw new Error('intro is the part of a loop that is heard once; a piece with loop false is heard once whole and has none');+ if (!(Number.isInteger(s.intro) && (s.intro as number) >= 1 && (s.intro as number) < s.song.length)) throw new Error(`intro must be how many of the song's first parts are heard once: a whole number from 1, leaving at least one part to go round (the song has ${s.song.length})`);+ }for (const key of ['fadeIn', 'fadeOut'] as const) {if (s[key] === undefined) continue;if (s.loop !== false) throw new Error(`${key} is for a piece that ends (loop: false). A loop comes round to its start with no gap, and a fade written into it would be heard every time round: the game fades it, playBGM(url, { fadeIn }) to start it and stopBGM(seconds) to end it`);diff --git a/pipeline/tests/music.test.ts b/pipeline/tests/music.test.tsindex dda4649e9..2090973ad 100644--- a/pipeline/tests/music.test.ts+++ b/pipeline/tests/music.test.ts@@ -446,3 +446,44 @@ describe('recordings played soft and loud, and taking turns', () => {expect(pitchOf(raw, 44100, 0.05, 0.45)).toBeGreaterThan(449);});});++describe('an introduction heard once before the round', () => {+ const FULL = '1 - - - 1 - - - 1 - - - 1 - - -';+ const instruments = { tone, click, low: tone, pad: tone, bass: tone };+ const tracks = { tone: FULL, pad: FULL.replace(/1/g, '3'), bass: FULL.replace(/1/g, '5,'), click: steps('x', '.', 'x') };+ const thin = { pattern: 'a', only: ['tone'] };++ it('says where the round begins, and meets itself there and not at the top of the file', () => {+ const m = renderMusic(piece(tracks, { loop: true, instruments, song: [thin, 'a', 'a', 'a'], intro: 1 }));+ // One bar of two seconds is the introduction.+ expect(m.loopStart).toBe(2);+ expect(m.seconds).toBe(8);+ // The round is full at both ends, and nothing is said of the thin bar before it.+ expect(m.join).toMatchObject({ begins: ['tone', 'click', 'pad', 'bass'], ends: ['tone', 'click', 'pad', 'bass'] });+ expect(m.notes.join()).not.toMatch(/meets itself/);+ // Without the introduction marked, the same song drops at the join and is told so.+ expect(renderMusic(piece(tracks, { loop: true, instruments, song: [thin, 'a', 'a', 'a'] })).notes.join()).toMatch(/meets itself/);+ });++ it('brings what rings past the end in where the round begins, and leaves the introduction as it was', () => {+ const long = { layers: [{ source: 'sine', freq: 440, length: 0.8, decay: 0.3 }] };+ const last = `${REST.slice(0, -1)}1`;+ // Three bars: silence, silence, and one long note on the last step of the last.+ const source = (intro?: number) => piece({ tone: REST }, {+ loop: true, instruments: { tone: long, click, low: tone },+ patterns: { a: { bars: 1, tracks: { click: REST } }, b: { bars: 1, tracks: { tone: last } } }, song: ['a', 'a', 'b'], ...(intro ? { intro } : {}),+ });+ const whole = renderMusic(source()).channels[0]!;+ const after = renderMusic(source(1)).channels[0]!;+ // With no introduction the tail is at the top of the file; with one it is at two seconds, and the top is silent.+ expect(level(whole, 44100, 0.05, 0.4)).toBeGreaterThan(0.05);+ expect(level(after, 44100, 0.05, 0.4)).toBeLessThan(1e-4);+ expect(level(after, 44100, 2.05, 2.4)).toBeGreaterThan(0.05);+ expect(after.length).toBe(whole.length);+ });++ it('refuses an introduction that leaves nothing to go round, or a piece that ends', () => {+ expect(() => piece(tracks, { loop: true, instruments, song: ['a', 'a'], intro: 2 })).toThrow(/intro must be how many of the song's first parts are heard once.*\(the song has 2\)/);+ expect(() => piece(tracks, { instruments, song: ['a', 'a'], intro: 1 })).toThrow(/intro is the part of a loop that is heard once/);+ });+});diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.mdindex 97669f6e1..cbf3abf57 100644--- a/sdk/etc/index.api.md+++ b/sdk/etc/index.api.md@@ -1182,7 +1182,7 @@ invalidateBuffer: (url: string) => booleanisBusMuted: (busName: string) => booleanmaxVoices: numbermuteBus: (busName: string, muted: boolean) => void-playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; }) => void+playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => voidplayBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandleplaySFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandleplayTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>@@ -1381,8 +1381,11 @@ attenuationModel: numberbus: stringclip: stringenabled: boolean+fadeIn: number+fadeOut: numberfinished: booleanloop: boolean+loopStart: numbermaxDistance: numberminDistance: numberpitch: number@@ -7975,6 +7978,7 @@ readKeyboard: boolean```bus: string | undefinedloop: boolean | undefined+loopStart: number | undefinedpan: number | undefinedplaybackRate: number | undefinedpriority: number | undefineddiff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.mdindex 507966664..2aaf94cec 100644--- a/sdk/etc/index.native.api.md+++ b/sdk/etc/index.native.api.md@@ -1182,7 +1182,7 @@ invalidateBuffer: (url: string) => booleanisBusMuted: (busName: string) => booleanmaxVoices: numbermuteBus: (busName: string, muted: boolean) => void-playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; }) => void+playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => voidplayBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandleplaySFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandleplayTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>@@ -1381,8 +1381,11 @@ attenuationModel: numberbus: stringclip: stringenabled: boolean+fadeIn: number+fadeOut: numberfinished: booleanloop: boolean+loopStart: numbermaxDistance: numberminDistance: numberpitch: number@@ -6757,7 +6760,7 @@ width: number```es_audioLoad: ((bytes: ArrayBuffer, stream?: boolean) => { id: number; duration: number; bytes: number; streamed?: boolean; } | null) | undefinedes_audioPause: ((voiceId: number) => void) | undefined-es_audioPlay: ((bufferId: number, volume: number, pan: number, loop: boolean, rate: number) => number) | undefined+es_audioPlay: ((bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number) => number) | undefinedes_audioResume: ((voiceId: number) => void) | undefinedes_audioResumeAll: (() => void) | undefinedes_audioSetLoop: ((voiceId: number, loop: boolean) => void) | undefined@@ -8188,6 +8191,7 @@ readKeyboard: boolean```bus: string | undefinedloop: boolean | undefined+loopStart: number | undefinedpan: number | undefinedplaybackRate: number | undefinedpriority: number | undefineddiff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.mdindex 6cda4e003..0608dadd9 100644--- a/sdk/etc/index.node.api.md+++ b/sdk/etc/index.node.api.md@@ -1177,7 +1177,7 @@ invalidateBuffer: (url: string) => booleanisBusMuted: (busName: string) => booleanmaxVoices: numbermuteBus: (busName: string, muted: boolean) => void-playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; }) => void+playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => voidplayBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandleplaySFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandleplayTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>@@ -1376,8 +1376,11 @@ attenuationModel: numberbus: stringclip: stringenabled: boolean+fadeIn: number+fadeOut: numberfinished: booleanloop: boolean+loopStart: numbermaxDistance: numberminDistance: numberpitch: number@@ -7970,6 +7973,7 @@ readKeyboard: boolean```bus: string | undefinedloop: boolean | undefined+loopStart: number | undefinedpan: number | undefinedplaybackRate: number | undefinedpriority: number | undefineddiff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.mdindex bac00b37b..879121b83 100644--- a/sdk/etc/minigame.api.md+++ b/sdk/etc/minigame.api.md@@ -1177,7 +1177,7 @@ invalidateBuffer: (url: string) => booleanisBusMuted: (busName: string) => booleanmaxVoices: numbermuteBus: (busName: string, muted: boolean) => void-playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; }) => void+playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => voidplayBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandleplaySFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandleplayTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>@@ -1376,8 +1376,11 @@ attenuationModel: numberbus: stringclip: stringenabled: boolean+fadeIn: number+fadeOut: numberfinished: booleanloop: boolean+loopStart: numbermaxDistance: numberminDistance: numberpitch: number@@ -8267,6 +8270,7 @@ readKeyboard: boolean```bus: string | undefinedloop: boolean | undefined+loopStart: number | undefinedpan: number | undefinedplaybackRate: number | undefinedpriority: number | undefineddiff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.mdindex 11de2a53e..b4aaeb44a 100644--- a/sdk/etc/wechat.api.md+++ b/sdk/etc/wechat.api.md@@ -1177,7 +1177,7 @@ invalidateBuffer: (url: string) => booleanisBusMuted: (busName: string) => booleanmaxVoices: numbermuteBus: (busName: string, muted: boolean) => void-playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; }) => void+playBGM: (url: string, config?: { volume?: number; fadeIn?: number; crossFade?: number; loopStart?: number; }) => voidplayBuffer: (buffer: AudioBufferHandle, config?: PlayConfig) => AudioHandleplaySFX: (url: string, config?: { volume?: number; pitch?: number; pan?: number; priority?: number; }) => AudioHandleplayTrack: (url: string, config?: PlayConfig) => Promise<AudioHandle | null>@@ -1376,8 +1376,11 @@ attenuationModel: numberbus: stringclip: stringenabled: boolean+fadeIn: number+fadeOut: numberfinished: booleanloop: boolean+loopStart: numbermaxDistance: numberminDistance: numberpitch: number@@ -7977,6 +7980,7 @@ readKeyboard: boolean```bus: string | undefinedloop: boolean | undefined+loopStart: number | undefinedpan: number | undefinedplaybackRate: number | undefinedpriority: number | undefineddiff --git a/sdk/src/audio/Audio.ts b/sdk/src/audio/Audio.tsindex be919c87b..1f9ffc4ea 100644--- a/sdk/src/audio/Audio.ts+++ b/sdk/src/audio/Audio.ts@@ -561,10 +561,17 @@ export class AudioAPI {return this.playVoice_(buffer, playConfig);}+ /**+ * Background music: one piece at a time, round and round. `fadeIn` brings it up over that many seconds and+ * `crossFade` brings it up as the piece before goes down. `loopStart` is for a piece with an introduction:+ * the file plays from its top once, and each time it reaches its end it comes back to that many seconds in,+ * so the introduction is heard once and what follows it goes round.+ */playBGM(url: string, config?: {volume?: number;fadeIn?: number;crossFade?: number;+ loopStart?: number;}): void {const request = ++this.bgmRequest_;const play = (buffer: AudioBufferHandle) => {@@ -586,6 +593,7 @@ export class AudioAPI {bus: 'music',volume: targetVolume,loop: true,+ ...(config?.loopStart && config.loopStart > 0 ? { loopStart: config.loopStart } : {}),});if (fadeInDuration) {diff --git a/sdk/src/audio/AudioComponents.ts b/sdk/src/audio/AudioComponents.tsindex c116ef872..93b1b1e2a 100644--- a/sdk/src/audio/AudioComponents.ts+++ b/sdk/src/audio/AudioComponents.ts@@ -9,6 +9,12 @@ export interface AudioSourceData {volume: number;pitch: number;loop: boolean;+ /** With `loop`: seconds into the clip that it comes back to at its end, so an introduction is heard once. 0 is the start. */+ loopStart: number;+ /** Seconds it takes to come up to its volume when `playing` is raised. 0 is at once. */+ fadeIn: number;+ /** Seconds it takes to go down to nothing when `playing` is lowered, after which it stops. 0 is at once. */+ fadeOut: number;/*** Raise it and the clip sounds; lower it and the voice stops. The engine* lowers it again when a non-looping clip ends, so it reads as "is this@@ -36,6 +42,9 @@ export const AudioSource = defineComponent<AudioSourceData>('AudioSource', {volume: 1.0,pitch: 1.0,loop: false,+ loopStart: 0,+ fadeIn: 0,+ fadeOut: 0,playing: false,finished: false,playOnAwake: false,@@ -49,6 +58,9 @@ export const AudioSource = defineComponent<AudioSourceData>('AudioSource', {}, {assetFields: [{ field: 'clip', type: 'audio' }],fields: {+ loopStart: { min: 0, step: 0.1, tooltip: 'With Loop: seconds into the clip that it comes back to at its end, so an introduction plays once. 0 is the start.' },+ fadeIn: { min: 0, step: 0.1, tooltip: 'Seconds to come up to Volume when Playing is raised. 0 is at once.' },+ fadeOut: { min: 0, step: 0.1, tooltip: 'Seconds to go down to nothing when Playing is lowered; it stops after. 0 is at once.' },playing: { tooltip: 'Raise to play, lower to stop. The engine lowers it when a non-looping clip ends.' },finished: { advanced: true, tooltip: 'Clip completed (runtime, read-only). Raise Playing to play it again.' },},diff --git a/sdk/src/audio/AudioPlugin.ts b/sdk/src/audio/AudioPlugin.tsindex 7de54f1af..f388f9ac7 100644--- a/sdk/src/audio/AudioPlugin.ts+++ b/sdk/src/audio/AudioPlugin.ts@@ -41,6 +41,10 @@ interface ActiveVoice {pitch: number;loop: boolean;spatial: boolean;+ /** How far up it is, 0..1: below 1 while it comes in or goes out. What the voice is given is the source's volume times this. */+ fade: number;+ /** Its source lowered `playing` and it is going down over the source's fadeOut; it stops at nothing. */+ leaving: boolean;}interface AudioPluginState {@@ -234,10 +238,14 @@ export class AudioPlugin implements Plugin {loop: source.loop,spatial: source.spatial,completedNaturally: false,+ // Up from nothing where the source asks to come in over a time.+ fade: source.fadeIn > 0 ? 0 : 1,+ leaving: false,handle: audioAPI.playBuffer(buffer, {bus: source.bus,- volume: source.volume,+ volume: source.fadeIn > 0 ? 0 : source.volume,loop: source.loop,+ ...(source.loop && source.loopStart > 0 ? { loopStart: source.loopStart } : {}),playbackRate: source.pitch,priority: source.priority,}),@@ -269,7 +277,9 @@ export class AudioPlugin implements Plugin {}}} else if (!playing && voice) {- stopVoice(id);+ // Lowered: it goes at once, or down over the time the source gives and then goes.+ if (source.fadeOut > 0 && voice.handle.isPlaying && voice.fade > 0) voice.leaving = true;+ else stopVoice(id);} else if (playing && voice && !voice.handle.isPlaying) {// The voice is no longer active. Lowering the flag here is// what makes it readable as "is this sounding" rather@@ -281,12 +291,25 @@ export class AudioPlugin implements Plugin {});}- const sounding = activeSourceHandles.get(id);+ let sounding = activeSourceHandles.get(id);+ let faded = false;+ if (sounding) {+ // Raised again while it was going out: it turns round from where it is.+ if (playing && sounding.leaving) sounding.leaving = false;+ const before = sounding.fade;+ if (sounding.leaving) sounding.fade = Math.max(0, sounding.fade - time.delta / source.fadeOut);+ else if (sounding.fade < 1) sounding.fade = source.fadeIn > 0 ? Math.min(1, sounding.fade + time.delta / source.fadeIn) : 1;+ faded = sounding.fade !== before;+ if (sounding.leaving && sounding.fade <= 0) {+ stopVoice(id);+ sounding = undefined;+ }+ }if (sounding) {if (sounding.pitch !== source.pitch) sounding.handle.setPlaybackRate(source.pitch);if (sounding.loop !== source.loop) sounding.handle.setLoop(source.loop);if (!source.spatial) {- if (sounding.volume !== source.volume || sounding.spatial) sounding.handle.setVolume(source.volume);+ if (sounding.volume !== source.volume || sounding.spatial || faded) sounding.handle.setVolume(source.volume * sounding.fade);if (sounding.spatial) sounding.handle.setPan(0);}sounding.volume = source.volume;@@ -314,7 +337,7 @@ export class AudioPlugin implements Plugin {const attenuation = calculateAttenuation(distance, spatialConfig);const pan = calculatePanning(sourceAt, listenerAt, listenerRight, source.maxDistance);- sounding.handle.setVolume(source.volume * attenuation);+ sounding.handle.setVolume(source.volume * attenuation * sounding.fade);sounding.handle.setPan(pan);}}diff --git a/sdk/src/audio/MiniGameAudioBackend.ts b/sdk/src/audio/MiniGameAudioBackend.tsindex 5a225d54e..4c66ec34d 100644--- a/sdk/src/audio/MiniGameAudioBackend.ts+++ b/sdk/src/audio/MiniGameAudioBackend.ts@@ -59,9 +59,26 @@ class MiniGameAudioHandle implements AudioHandle {this.dispose_(true);}+ /** Where a loop comes back to when it is not the start; the host's own loop only knows the start. */+ private loopFrom_ = 0;+ private looping_ = false;++ /** @internal A loop that comes back to @p seconds: the context is not left to loop, and is sent back when it ends. */+ loopFrom(seconds: number): void {+ this.loopFrom_ = seconds;+ this.looping_ = true;+ this.ctx_.loop = false;+ }+/** @internal The context ended on its own — dispose without a redundant stop(). */onNaturalEnd(): void {if (!this.isLive_() || this.ctx_.loop) return;+ if (this.looping_) {+ // A context that has ended starts from its startTime when it is played again.+ this.ctx_.startTime = this.loopFrom_;+ this.resume();+ return;+ }this.dispose_(false);this.onEnd?.();}@@ -97,7 +114,8 @@ class MiniGameAudioHandle implements AudioHandle {setLoop(loop: boolean): void {if (!this.isLive_()) return;- this.ctx_.loop = loop;+ if (this.loopFrom_ > 0) this.looping_ = loop;+ else this.ctx_.loop = loop;}setPlaybackRate(rate: number): void {@@ -184,6 +202,7 @@ export class MiniGameAudioBackend implements PlatformAudioBackend {this.contexts_.set(handleId, ctx);const handle = new MiniGameAudioHandle(handleId, ctx, this.contexts_, this.device_);+ if (config.loop && config.loopStart && config.loopStart > 0) handle.loopFrom(config.loopStart);ctx.onEnded(() => handle.onNaturalEnd());ctx.onError((res) => {log.error('audio', `Playback error for "${url}"`, res.errMsg);diff --git a/sdk/src/audio/NativeAudioBackend.ts b/sdk/src/audio/NativeAudioBackend.tsindex dd3193dc0..fb17b5afd 100644--- a/sdk/src/audio/NativeAudioBackend.ts+++ b/sdk/src/audio/NativeAudioBackend.ts@@ -122,6 +122,7 @@ export class NativeAudioBackend implements PlatformAudioBackend {scalar.clamp(config.pan ?? 0, -1, 1),config.loop ?? false,config.playbackRate ?? 1,+ config.loop && config.loopStart && config.loopStart > 0 ? config.loopStart : 0,);const handle = new NativeAudioHandle(voiceId, this.audio_, buffer.duration, () => this.voices_.delete(voiceId),diff --git a/sdk/src/audio/PlatformAudioBackend.ts b/sdk/src/audio/PlatformAudioBackend.tsindex dbb15c52a..8af488a34 100644--- a/sdk/src/audio/PlatformAudioBackend.ts+++ b/sdk/src/audio/PlatformAudioBackend.ts@@ -58,6 +58,13 @@ export interface PlayConfig {bus?: string;priority?: number;startOffset?: number;+ /**+ * With `loop`: seconds into the clip that it comes back to when it reaches its end, in place of its start. A+ * piece with an introduction is one file, played from the top once and round from here ever after. Exact to the+ * sample where the clip is decoded; where it is played from its compressed file (a long clip on the web, a+ * mini-game) the player is sent back when it ends, which can be heard as a breath.+ */+ loopStart?: number;}export interface AudioBackendInitOptions {diff --git a/sdk/src/audio/WebAudioBackend.ts b/sdk/src/audio/WebAudioBackend.tsindex 19aad3454..96d58ba6d 100644--- a/sdk/src/audio/WebAudioBackend.ts+++ b/sdk/src/audio/WebAudioBackend.ts@@ -24,6 +24,8 @@ class WebAudioHandle implements AudioHandle {private startOffset_: number;private playbackRate_: number;private loop_: boolean;+ /** Where a loop comes back to, in seconds; 0 is the start. */+ private readonly loopStart_: number;private readonly duration_: number;constructor(@@ -34,6 +36,7 @@ class WebAudioHandle implements AudioHandle {pool: AudioPool,context: AudioContext,startOffset: number,+ loopStart = 0,) {this.id = id;this.source_ = source;@@ -46,6 +49,7 @@ class WebAudioHandle implements AudioHandle {this.startedAt_ = context.currentTime;this.playbackRate_ = source.playbackRate.value;this.loop_ = source.loop;+ this.loopStart_ = loopStart;this.bindOnEnded_(source);}@@ -81,14 +85,20 @@ class WebAudioHandle implements AudioHandle {this.playing_ = false;const elapsed = (this.context_!.currentTime - this.startedAt_) * this.playbackRate_;this.pausedAt_ = this.startOffset_ + elapsed;- if (this.loop_ && this.buffer_!.duration > 0) {- this.pausedAt_ = this.pausedAt_ % this.buffer_!.duration;- }+ if (this.loop_ && this.buffer_!.duration > 0) this.pausedAt_ = this.wrapped_(this.pausedAt_);this.source_!.onended = null;try { this.source_!.stop(); } catch (_) { /* already stopped */ }this.source_!.disconnect();}+ /** A place in the clip for a time since it began: past the end, it has been round from the loop's start. */+ private wrapped_(position: number): number {+ const end = this.buffer_!.duration;+ if (position < end) return position;+ const round = end - this.loopStart_;+ return round > 0 ? this.loopStart_ + ((position - this.loopStart_) % round) : position % end;+ }+resume(): void {if (this.playing_ || this.stopped_) return;@@ -97,6 +107,7 @@ class WebAudioHandle implements AudioHandle {try {source.buffer = this.buffer_;source.loop = this.loop_;+ if (this.loopStart_ > 0) { source.loopStart = this.loopStart_; source.loopEnd = this.buffer_!.duration; }source.playbackRate.value = this.playbackRate_;source.connect(this.poolNode_!.gain);this.bindOnEnded_(source);@@ -179,10 +190,14 @@ class WebStreamHandle implements AudioHandle {private readonly duration_: number;private readonly onRelease_: (handle: WebStreamHandle) => void;private readonly ensureResumed_: () => Promise<void>;+ /** A loop that comes back to somewhere other than the start: the element cannot, so it is sent back when it ends. */+ private loopFrom_ = 0;+ private looping_ = false;constructor(id: number, element: HTMLAudioElement, node: MediaElementAudioSourceNode, poolNode: PooledAudioNode,pool: AudioPool, duration: number, onRelease: (handle: WebStreamHandle) => void, ensureResumed: () => Promise<void>,+ loopStart = 0,) {this.id = id;this.element_ = element;@@ -193,8 +208,14 @@ class WebStreamHandle implements AudioHandle {this.onRelease_ = onRelease;this.ensureResumed_ = ensureResumed;poolNode.source = node;+ if (loopStart > 0 && element.loop) {+ this.loopFrom_ = loopStart;+ this.looping_ = true;+ element.loop = false;+ }element.onended = () => {if (this.stopped_ || element.loop) return;+ if (this.looping_ && this.playing_) { this.begin(this.loopFrom_); return; }this.release_();this.onEnd?.();};@@ -275,7 +296,8 @@ class WebStreamHandle implements AudioHandle {setLoop(loop: boolean): void {if (this.stopped_ || !this.element_) return;- this.element_.loop = loop;+ if (this.loopFrom_ > 0) this.looping_ = loop;+ else this.element_.loop = loop;}setPlaybackRate(rate: number): void {@@ -606,6 +628,7 @@ export class WebAudioBackend implements PlatformAudioBackend {const handle = new WebStreamHandle(++this.nextHandleId_, element, node, poolNode, pool, stream.duration,(h) => this.releaseStream_(h), () => this.ensureResumed(),+ config.loop && config.loopStart && config.loopStart < stream.duration ? config.loopStart : 0,);this.streams_.set(bufferId, handle);handle.begin(config.startOffset ?? 0);@@ -646,6 +669,7 @@ export class WebAudioBackend implements PlatformAudioBackend {}const startOffset = config.startOffset ?? 0;+ const loopStart = config.loop && config.loopStart && config.loopStart > 0 && config.loopStart < audioBuffer.duration ? config.loopStart : 0;const poolNode = this.pool_.acquire();let source: AudioBufferSourceNode;try {@@ -653,6 +677,8 @@ export class WebAudioBackend implements PlatformAudioBackend {poolNode.source = source;source.buffer = audioBuffer;source.loop = config.loop ?? false;+ // Round from here, not from the top: the node does it between two samples.+ if (loopStart > 0) { source.loopStart = loopStart; source.loopEnd = audioBuffer.duration; }source.playbackRate.value = config.playbackRate ?? 1.0;source.connect(poolNode.gain);@@ -670,7 +696,7 @@ export class WebAudioBackend implements PlatformAudioBackend {const handleId = ++this.nextHandleId_;const handle = new WebAudioHandle(- handleId, source, audioBuffer, poolNode, this.pool_, this.context_, startOffset+ handleId, source, audioBuffer, poolNode, this.pool_, this.context_, startOffset, loopStart);return handle;diff --git a/sdk/src/platform/native/bridge.ts b/sdk/src/platform/native/bridge.tsindex a169d504b..5851fe8ea 100644--- a/sdk/src/platform/native/bridge.ts+++ b/sdk/src/platform/native/bridge.ts@@ -147,7 +147,8 @@ export interface NativeAudioBridge {unload(bufferId: number): void;/** Start a voice on a loaded buffer. Returns its voice id, or -1 if the buffer* is unknown. */- play(bufferId: number, volume: number, pan: number, loop: boolean, rate: number): number;+ /** @param loopStart seconds a looping voice comes back to; 0 is its start. A host older than this takes five arguments and loops from the start. */+ play(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number): number;stop(voiceId: number): void;pause(voiceId: number): void;resume(voiceId: number): void;diff --git a/sdk/src/platform/native/hostBridge.ts b/sdk/src/platform/native/hostBridge.tsindex 6580d09bc..9b6debae2 100644--- a/sdk/src/platform/native/hostBridge.ts+++ b/sdk/src/platform/native/hostBridge.ts@@ -144,7 +144,7 @@ export interface NativeHostBindings extends NativeDeviceHost {es_audioSpectrum?(bins: number): ArrayBuffer | null;es_audioLoad?(bytes: ArrayBuffer, stream?: boolean): { id: number; duration: number; bytes: number; streamed?: boolean } | null;es_audioUnload?(bufferId: number): void;- es_audioPlay?(bufferId: number, volume: number, pan: number, loop: boolean, rate: number): number;+ es_audioPlay?(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number): number;es_audioStop?(voiceId: number): void;es_audioPause?(voiceId: number): void;es_audioResume?(voiceId: number): void;@@ -388,7 +388,7 @@ function hostAudio(return {load: (bytes, stream) => bindings.es_audioLoad!(bytes, stream === true),unload: (id) => bindings.es_audioUnload!(id),- play: (buf, vol, pan, loop, rate) => bindings.es_audioPlay!(buf, vol, pan, loop, rate),+ play: (buf, vol, pan, loop, rate, loopStart) => bindings.es_audioPlay!(buf, vol, pan, loop, rate, loopStart ?? 0),stop: (v) => bindings.es_audioStop!(v),pause: (v) => bindings.es_audioPause!(v),resume: (v) => bindings.es_audioResume!(v),diff --git a/sdk/tests/audio-source-fade.test.ts b/sdk/tests/audio-source-fade.test.tsnew file mode 100644index 000000000..e3dae05be--- /dev/null+++ b/sdk/tests/audio-source-fade.test.ts@@ -0,0 +1,164 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/**+ * @file audio-source-fade.test.ts+ * @brief A source in a scene that comes in and goes out over a time, and a loop that comes back to somewhere+ * other than its start.+ *+ * Held here: that a voice with a fade starts at nothing and rises with the frames, that lowering the flag+ * lets it down before it is stopped rather than cutting it, that raising it again turns the fade round,+ * and that a source or `playBGM` with a loop start hands it to the backend.+ */+import { describe, it, expect, vi, beforeEach } from 'vitest';+import type { AudioHandle, PlatformAudioBackend } from '../src/audio/PlatformAudioBackend';++const backend = {+ name: 'Mock',+ mixer: null,+ isReady: true,+ initialize: vi.fn().mockResolvedValue(undefined),+ ensureResumed: vi.fn().mockResolvedValue(undefined),+ loadBuffer: vi.fn().mockResolvedValue({ id: 1, duration: 10 }),+ unloadBuffer: vi.fn(),+ play: vi.fn(),+ suspend: vi.fn(), resume: vi.fn(), dispose: vi.fn(),+} as unknown as PlatformAudioBackend;++vi.mock('../src/platform/base', async (importOriginal) => ({+ ...(await importOriginal<typeof import('../src/platform/base')>()),+ platformCreateAudioBackend: () => backend,+ platformOnMemoryWarning: () => () => {},+}));++const { App } = await import('../src/app/app');+const { AudioPlugin } = await import('../src/audio/AudioPlugin');+const { Audio } = await import('../src/audio/Audio');+const { AudioSource } = await import('../src/audio/AudioComponents');++type Voice = AudioHandle & { isPlaying: boolean; volume: number; setVolume: ReturnType<typeof vi.fn>; stop: ReturnType<typeof vi.fn> };+let voices: Voice[] = [];+let configs: Record<string, unknown>[] = [];++beforeEach(() => {+ vi.clearAllMocks();+ voices = [];+ configs = [];+ (backend.play as ReturnType<typeof vi.fn>).mockImplementation((_buffer: unknown, config: Record<string, unknown>) => {+ const v = {+ id: voices.length + 1, pause: vi.fn(), resume: vi.fn(), setPan: vi.fn(), setLoop: vi.fn(), setPlaybackRate: vi.fn(),+ isPlaying: true, currentTime: 0, duration: 10, volume: config.volume as number,+ stop: vi.fn(() => { v.isPlaying = false; }),+ setVolume: vi.fn((x: number) => { v.volume = x; }),+ } as unknown as Voice;+ voices.push(v);+ configs.push(config);+ return v;+ });+});++async function appWith(source: Record<string, unknown>) {+ const app = App.new();+ app.addPlugin(new AudioPlugin());+ await app.getResource(Audio).preload('theme.wav');+ const e = app.world.spawn();+ app.world.insert(e, AudioSource, { clip: 'theme.wav', bus: 'music', ...source });+ return { app, e };+}+const frames = async (app: InstanceType<typeof App>, n: number, dt = 0.1): Promise<void> => { for (let i = 0; i < n; i++) await app.tick(dt); };++describe('an AudioSource that fades', () => {+ it('comes up from nothing over fadeIn, and no further than its volume', async () => {+ const { app } = await appWith({ playing: true, volume: 0.8, fadeIn: 1 });+ await app.tick(0.1);+ expect(configs[0]!.volume).toBe(0);+ // The frame that starts it is the first step of the way up.+ expect(voices[0]!.volume).toBeCloseTo(0.08, 5);+ await frames(app, 4);+ expect(voices[0]!.volume).toBeCloseTo(0.4, 5);+ await frames(app, 10);+ expect(voices[0]!.volume).toBeCloseTo(0.8, 5);+ // Up, it is left alone: no write every frame.+ const writes = voices[0]!.setVolume.mock.calls.length;+ await frames(app, 5);+ expect(voices[0]!.setVolume.mock.calls.length).toBe(writes);+ });++ it('starts at its volume when it has no fade, as it always did', async () => {+ const { app } = await appWith({ playing: true, volume: 0.8 });+ await app.tick(0.1);+ expect(configs[0]!.volume).toBe(0.8);+ expect(voices[0]!.setVolume).not.toHaveBeenCalled();+ });++ it('goes down over fadeOut when the flag is lowered, and is stopped only at nothing', async () => {+ const { app, e } = await appWith({ playing: true, volume: 1, fadeOut: 0.5 });+ await app.tick(0.1);+ app.world.update(e, AudioSource, (d) => { d.playing = false; });+ await frames(app, 2);+ expect(voices[0]!.stop).not.toHaveBeenCalled();+ expect(voices[0]!.volume).toBeCloseTo(0.6, 5);+ await frames(app, 2);+ expect(voices[0]!.volume).toBeCloseTo(0.2, 5);+ await frames(app, 2);+ expect(voices[0]!.stop).toHaveBeenCalledTimes(1);+ // And it is gone: nothing more is played or written.+ await frames(app, 3);+ expect(backend.play).toHaveBeenCalledTimes(1);+ });++ it('is cut at once when the flag is lowered and it has no fadeOut', async () => {+ const { app, e } = await appWith({ playing: true, fadeIn: 1 });+ await app.tick(0.1);+ app.world.update(e, AudioSource, (d) => { d.playing = false; });+ await app.tick(0.1);+ expect(voices[0]!.stop).toHaveBeenCalledTimes(1);+ });++ it('turns round when the flag is raised again on its way out, without starting the clip over', async () => {+ const { app, e } = await appWith({ playing: true, volume: 1, fadeIn: 1, fadeOut: 1 });+ await frames(app, 12);+ expect(voices[0]!.volume).toBeCloseTo(1, 5);+ app.world.update(e, AudioSource, (d) => { d.playing = false; });+ await frames(app, 5);+ expect(voices[0]!.volume).toBeCloseTo(0.5, 5);+ app.world.update(e, AudioSource, (d) => { d.playing = true; });+ await frames(app, 3);+ expect(voices[0]!.volume).toBeCloseTo(0.8, 5);+ expect(voices[0]!.stop).not.toHaveBeenCalled();+ expect(backend.play).toHaveBeenCalledTimes(1);+ });++ it('is still stopped at once when its source is switched off or loses its clip', async () => {+ const { app, e } = await appWith({ playing: true, fadeOut: 2 });+ await app.tick(0.1);+ app.world.update(e, AudioSource, (d) => { d.enabled = false; });+ await app.tick(0.1);+ expect(voices[0]!.stop).toHaveBeenCalledTimes(1);+ });+});++describe('a loop that comes back to somewhere other than its start', () => {+ it('is handed to the backend by a source that loops', async () => {+ const { app } = await appWith({ playing: true, loop: true, loopStart: 12.5 });+ await app.tick(0.1);+ expect(configs[0]).toMatchObject({ loop: true, loopStart: 12.5 });+ });++ it('is not handed on by a source that does not loop, or whose loop start is the start', async () => {+ const { app } = await appWith({ playing: true, loop: false, loopStart: 12.5 });+ await app.tick(0.1);+ expect('loopStart' in configs[0]!).toBe(false);+ const again = await appWith({ playing: true, loop: true });+ await again.app.tick(0.1);+ expect('loopStart' in configs[1]!).toBe(false);+ });++ it('is handed to the backend by playBGM, with the fade in beside it', async () => {+ const { app } = await appWith({});+ const audio = app.getResource(Audio);+ audio.playBGM('theme.wav', { loopStart: 8, fadeIn: 1 });+ expect(configs[0]).toMatchObject({ bus: 'music', loop: true, loopStart: 8 });+ audio.playBGM('theme.wav');+ expect('loopStart' in configs[1]!).toBe(false);+ });+});diff --git a/sdk/tests/minigame-audio-backend.test.ts b/sdk/tests/minigame-audio-backend.test.tsindex 1e5798df1..cc1bad98e 100644--- a/sdk/tests/minigame-audio-backend.test.ts+++ b/sdk/tests/minigame-audio-backend.test.ts@@ -86,6 +86,21 @@ describe('MiniGameAudioBackend', () => {expect(handle.isPlaying).toBe(false); expect(handle.currentTime).toBe(0);expect(ctx.destroy).toHaveBeenCalledOnce();});+ it('a loop with a loop start is sent back there when it ends, and is not left to the host to loop', async () => {+ const buffer = await backend.loadBuffer('a.mp3'), handle = backend.play(buffer, { loop: true, loopStart: 12 });+ const ctx = (globalThis as any).wx.createInnerAudioContext.mock.results.at(-1).value;+ // The host's own loop only knows the start of the file.+ expect(ctx.loop).toBe(false);+ handle.onEnd = vi.fn();+ const plays = ctx.play.mock.calls.length;+ ctx.onEnded.mock.calls[0][0]();+ expect(ctx.startTime).toBe(12);+ expect(ctx.play.mock.calls.length).toBe(plays + 1);+ expect(handle.onEnd).not.toHaveBeenCalled(); expect(ctx.destroy).not.toHaveBeenCalled();+ // Told to stop looping, its next end is its last.+ handle.setLoop(false); ctx.onEnded.mock.calls[0][0]();+ expect(handle.onEnd).toHaveBeenCalledOnce(); expect(ctx.destroy).toHaveBeenCalledOnce();+ });it('natural completion is terminal before callbacks run and is delivered only once', async () => {const buffer = await backend.loadBuffer('a.mp3'), handle = backend.play(buffer, {});const ctx = (globalThis as any).wx.createInnerAudioContext.mock.results.at(-1).value;diff --git a/sdk/tests/native-audio.test.ts b/sdk/tests/native-audio.test.tsindex f1d057dd0..564ade175 100644--- a/sdk/tests/native-audio.test.ts+++ b/sdk/tests/native-audio.test.ts@@ -35,10 +35,11 @@ function installAudioHost(scope: Record<string, unknown>) {scope.es_audioLoad = (bytes: ArrayBuffer) => ({ id: nextBuffer++, duration: 2, bytes: bytes.byteLength * 4 });scope.es_audioUnload = () => {};- scope.es_audioPlay = (buf: number, vol: number, pan: number, loop: boolean, rate: number) => {+ scope.es_audioPlay = (buf: number, vol: number, pan: number, loop: boolean, rate: number, loopStart?: number) => {const id = nextVoice++;voices.set(id, { playing: true, time: 0, loop });- calls.play.push([buf, vol, pan, loop, rate]);+ // The sixth is where a loop comes back to; it is recorded only when it is somewhere.+ calls.play.push(loopStart ? [buf, vol, pan, loop, rate, loopStart] : [buf, vol, pan, loop, rate]);return id;};scope.es_audioStop = (v: number) => { voices.delete(v); calls.stop.push(v); };@@ -137,6 +138,17 @@ describe('NativeAudioBackend', () => {(scope.es_onNativeAudioEnded as (id: number) => void)(handle.id);expect(handle.onEnd).toHaveBeenCalledOnce(); main.dispose();});+ it('hands the host where a loop comes back to, and nothing for a voice that does not loop', async () => {+ const scope = audioScope();+ const host = installAudioHost(scope);+ const backend = new NativeAudioBackend(createHostBridge(scope).audio!);+ const buffer = await backend.loadBufferFromData('music', new ArrayBuffer(16));+ backend.play(buffer, { loop: true, loopStart: 8 });+ backend.play(buffer, { loopStart: 8 });+ backend.play(buffer, { loop: true });+ expect(host.calls.play).toEqual([[buffer.id, 1, 0, true, 1, 8], [buffer.id, 1, 0, false, 1], [buffer.id, 1, 0, true, 1]]);+ });+it('loads, plays and reports state through the bridge', async () => {const scope = audioScope();const host = installAudioHost(scope); // capture the same closure the bridge will usediff --git a/sdk/tests/web-audio-backend.test.ts b/sdk/tests/web-audio-backend.test.tsindex c8fa4d7ae..132d703bc 100644--- a/sdk/tests/web-audio-backend.test.ts+++ b/sdk/tests/web-audio-backend.test.ts@@ -490,6 +490,28 @@ describe('WebAudioBackend', () => {expect(handle.currentTime).toBeCloseTo(.875);});+ it('a loop with a loop start goes round from there, and a pause knows where in the round it is', async () => {+ const { bufferHandle, mockCtx } = await initAndLoadBuffer(backend);+ const handle = backend.play(bufferHandle, { loop: true, loopStart: .4 });+ const source = mockCtx.createBufferSource.mock.results.at(-1).value;+ expect([source.loop, source.loopStart, source.loopEnd]).toEqual([true, .4, 1]);+ // 2.3 s in: once through the second it lasts, then round its last 0.6 s twice and a tenth more.+ mockCtx.currentTime = 2.3; handle.pause();+ expect(handle.currentTime).toBeCloseTo(.5);+ handle.resume();+ const next = mockCtx.createBufferSource.mock.results.at(-1).value;+ expect(next.start).toHaveBeenCalledWith(0, expect.closeTo(.5, 5));+ expect([next.loop, next.loopStart, next.loopEnd]).toEqual([true, .4, 1]);+ });++ it('a loop start means nothing without a loop, or past the end of the clip', async () => {+ const { bufferHandle, mockCtx } = await initAndLoadBuffer(backend);+ for (const config of [{ loopStart: .4 }, { loop: true, loopStart: 3 }, { loop: true }]) {+ backend.play(bufferHandle, config);+ expect(mockCtx.createBufferSource.mock.results.at(-1).value.loopStart).toBeUndefined();+ }+ });+it('looping pause wraps the accumulated position after a rate change', async () => {const { bufferHandle, mockCtx } = await initAndLoadBuffer(backend);const handle = backend.play(bufferHandle, { loop: true });diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 561afb12d..74a148e2e 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -4630,6 +4630,10 @@ export const NOTED = {{"note": "An AI can make background music."},"docs(music): soundfont_read says what a SoundFont does that is undone on the way out":{"note": "An AI can make background music."},+ "feat(audio): music with an introduction, and a source that fades - a looping voice comes back to loopStart and not to its start, on the web, a mini-game and the native host, from playBGM or an AudioSource; an AudioSource comes up over fadeIn and goes down over fadeOut before it stops; and a piece of music marks its first parts as an intro and says where the round begins":+ {"note": "Music with an introduction, and a source that fades."},+ "feat(music): the reply gives where the round begins for a piece with an introduction, and the tool text says how a game plays one and how a source fades":+ {"note": "An AI can make background music."},"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":{"note": "A sky of your own shades the ground under its clouds."},"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":