Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

fix(audio): a loop is seamless in a packaged game, and an effect is on time - an exported .wav became an MP3 that decodes with 1105 samples of silence before it and more after, so every loop had a gap each time round and every sound began a fortieth of a second late; the exporter now records how long each clip really is and the web, mini-game and native backends play it from past the silence to its real end

yhh06d8441f4b0c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9ac6aec0e..b224fc2c4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,6 +16,7 @@ published separately; it ships inside the editor.
### Fixed
+- **A loop is seamless in a packaged game.** A `.wav` is re-encoded to MP3 when a game is exported, and a decoder gives an MP3 back with 1105 samples of silence before its sound and several hundred after it: measured in a browser, a one-second loop that was seamless as a `.wav` had 21 ms of silence every time round once packaged, and every sound, looping or not, began a fortieth of a second late. No tag in the file tells a player to take the silence off, and none does. The exporter now records, for each clip it re-encodes, how long it really is and what leads it, and the runtime plays the clip from past the lead to its real end and loops it between the two: exact to the sample where the clip is decoded (the web's decoded clips and the native host), and started past the lead where it is played from its file (a streamed clip on the web, a mini-game). A decoder that takes the silence off itself is seen by the length of what it returns and is not asked to twice. A project played from the editor reads its own `.wav` files and was never affected.
- **A large ground no longer throws every frame once its pages run out.** With every page slot in use (a `detail` that wants more than the 900 there are, or a long flight), `TerrainSystem` threw `Cannot read properties of undefined (reading 'slot')` each frame: the ground stopped following the eye, and the log filled. The ground chose what to draw, made new pages, then placed its nodes, and a page was marked as in use only when placed; a page made in between took the slot of the least recently drawn one, which was a page just chosen. What is drawn is now marked before anything is made, and when every slot is drawn that frame the new page waits.
## [0.104.0] - 2026-11-12
diff --git a/desktop b/desktop
index 904ca1604..410a7e885 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit 904ca16044a76b0f73963f81ff6af4f14c6edb37
+Subproject commit 410a7e88572af9f094148c8b85a14918c8d8534d
diff --git a/docs/astro/src/content/docs/assets/audio.mdx b/docs/astro/src/content/docs/assets/audio.mdx
index 3ebcceb38..184ef6f0d 100644
--- a/docs/astro/src/content/docs/assets/audio.mdx
+++ b/docs/astro/src/content/docs/assets/audio.mdx
@@ -46,6 +46,21 @@ const play = defineSystem([Res(Audio)], (audio) => {
current track, optionally cross-fading. Both auto-preload an uncached clip and play
it once ready.
+### What packaging does to a clip
+
+A `.wav` is re-encoded when a game is exported, and a decoder gives an encoded
+clip back with a fortieth of a second of silence before its sound and some after
+it. The exporter records how long each clip really is and the runtime plays it
+from past that silence to its real end, so a loop that was seamless as a `.wav`
+is seamless in the build and an effect is not late. Nothing is asked of you.
+
+Only a clip that is **decoded** can go round between two samples. A long clip
+(over 30 seconds under `auto`) is played from its compressed file instead, by a
+player that is sent back to the start when it ends: a looping piece of music
+wants `loadType: 'decompress'` in its import settings, which costs its decoded
+size in memory while it is loaded. The editor's music tool sets it for pieces up
+to two minutes.
+
### An introduction, then the loop
A piece with an introduction is one file: it plays from the top once, and each
diff --git a/docs/astro/src/content/docs/zh-cn/assets/audio.mdx b/docs/astro/src/content/docs/zh-cn/assets/audio.mdx
index 25a645a2c..25e04e655 100644
--- a/docs/astro/src/content/docs/zh-cn/assets/audio.mdx
+++ b/docs/astro/src/content/docs/zh-cn/assets/audio.mdx
@@ -40,6 +40,18 @@ const play = defineSystem([Res(Audio)], (audio) => {
`playSFX` 触发一次性播放并返回一个句柄;`playBGM` 循环并替换当前曲目,可选交叉淡入。两者都会
自动预加载未缓存的片段并在就绪后播放。
+### 打包对音频做了什么
+
+`.wav` 在导出游戏时会被重新编码,而解码器还原出来的片段开头会多出约四十分之一秒
+的静音,结尾也会多出一截。导出器会记下每个片段的真实长度,运行时从静音之后播到
+真实结尾,所以原本无缝的循环在成品里仍然无缝,音效也不会晚出声。这一步不需要你做
+任何事。
+
+只有**解码后播放**的片段才能精确到采样地循环。长片段(`auto` 下超过 30 秒)会改为
+从压缩文件直接播放,放完时播放器被送回开头:循环的音乐应在导入设置里设
+`loadType: 'decompress'`,代价是加载期间占用它解码后的内存。编辑器的音乐工具会给
+两分钟以内的曲子自动设好。
+
### 前奏放一次,然后循环
带前奏的曲子是一个文件:从头放一遍,之后每次放到结尾都回到 `loopStart` 秒处,
diff --git a/native/host/bindings/AudioBindings.cpp b/native/host/bindings/AudioBindings.cpp
index bb2557d60..4d5516a09 100644
--- a/native/host/bindings/AudioBindings.cpp
+++ b/native/host/bindings/AudioBindings.cpp
@@ -55,8 +55,9 @@ esjs::Val js_audioUnload(const esjs::Args& a) {
return esjs::undefined();
}
-// es_audioPlay(bufferId, volume, pan, loop, rate, loopStart?) -> voiceId (-1 if unknown).
-// loopStart is seconds a looping voice comes back to in place of its start.
+// es_audioPlay(bufferId, volume, pan, loop, rate, loopStart?, start?, end?) -> voiceId (-1 if unknown).
+// loopStart is seconds a looping voice comes back to in place of its start; start and end are where
+// the clip begins and is over in what was decoded (end 0 is its natural end).
esjs::Val js_audioPlay(const esjs::Args& a) {
int32_t buf = esjs::toInt32(a[0]);
double vol = 1, pan = 0, rate = 1;
@@ -64,9 +65,11 @@ esjs::Val js_audioPlay(const esjs::Args& a) {
if (a.size() > 2) pan = esjs::toNumber(a[2]);
const bool loop = a.size() > 3 && esjs::toBool(a[3]);
if (a.size() > 4) rate = esjs::toNumber(a[4]);
- double loopStart = 0;
+ double loopStart = 0, start = 0, end = 0;
if (a.size() > 5) loopStart = esjs::toNumber(a[5]);
- return esjs::int32(host().audio.play(buf, (f32)vol, (f32)pan, loop, (f32)rate, (f32)loopStart));
+ if (a.size() > 6) start = esjs::toNumber(a[6]);
+ if (a.size() > 7) end = esjs::toNumber(a[7]);
+ return esjs::int32(host().audio.play(buf, (f32)vol, (f32)pan, loop, (f32)rate, (f32)loopStart, (f32)start, (f32)end));
}
// The voice-scoped commands all take the voice id as a[0].
@@ -135,7 +138,7 @@ void registerAudioBindings(HostState& h, esjs::Val global) {
if (!h.audio.ready()) return;
bindGlobal(global, "es_audioLoad", js_audioLoad, 2);
bindGlobal(global, "es_audioUnload", js_audioUnload, 1);
- bindGlobal(global, "es_audioPlay", js_audioPlay, 6);
+ bindGlobal(global, "es_audioPlay", js_audioPlay, 8);
bindGlobal(global, "es_audioStop", js_audioStop, 1);
bindGlobal(global, "es_audioPause", js_audioPause, 1);
bindGlobal(global, "es_audioResume", js_audioResume, 1);
diff --git a/native/host/media/native_audio.cpp b/native/host/media/native_audio.cpp
index 6f0a46929..d7cc45d1c 100644
--- a/native/host/media/native_audio.cpp
+++ b/native/host/media/native_audio.cpp
@@ -310,7 +310,7 @@ void AudioEngine::unload(int bufferId) {
}
}
-int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart) {
+int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart, float start, float end) {
if (!ready()) return -1;
auto bit = impl_->buffers.find(bufferId);
if (bit == impl_->buffers.end() || bit->second.unloaded) return -1;
@@ -338,14 +338,18 @@ int AudioEngine::play(int bufferId, float volume, float pan, bool loop, float ra
ma_sound_set_pan(&voice.sound, pan);
ma_sound_set_pitch(&voice.sound, rate > 0 ? rate : 1.0f);
ma_sound_set_looping(&voice.sound, loop ? MA_TRUE : MA_FALSE);
- if (loop && loopStart > 0.0f) {
- // Round from here and not from the top. The point is in the clip's own frames, which is what the data
- // source counts in; its end is left where the clip ends.
+ if (start > 0.0f || end > 0.0f || (loop && loopStart > 0.0f)) {
+ // Places in the clip's own frames, which is what the data source counts in.
ma_uint32 sampleRate = 0;
ma_data_source* source = ma_sound_get_data_source(&voice.sound);
if (source && ma_sound_get_data_format(&voice.sound, nullptr, nullptr, &sampleRate, nullptr, 0) == MA_SUCCESS && sampleRate > 0) {
- ma_data_source_set_loop_point_in_pcm_frames(
- source, static_cast<ma_uint64>(static_cast<double>(loopStart) * sampleRate), ~static_cast<ma_uint64>(0));
+ const auto frame = [sampleRate](float seconds) { return static_cast<ma_uint64>(static_cast<double>(seconds) * sampleRate); };
+ // Round from the loop start and not from the top, and over at the clip's end and not the file's.
+ if (loop && (loopStart > 0.0f || end > 0.0f)) {
+ ma_data_source_set_loop_point_in_pcm_frames(source, frame(loopStart), end > 0.0f ? frame(end) : ~static_cast<ma_uint64>(0));
+ }
+ // Begun past the silence encoding put before it.
+ if (start > 0.0f) ma_sound_seek_to_pcm_frame(&voice.sound, frame(start));
}
}
ma_sound_start(&voice.sound);
diff --git a/native/host/media/native_audio.hpp b/native/host/media/native_audio.hpp
index a553f5b35..215bd6292 100644
--- a/native/host/media/native_audio.hpp
+++ b/native/host/media/native_audio.hpp
@@ -60,7 +60,9 @@ public:
/** Start a voice on a loaded buffer. Returns its id, or -1 if unknown. */
/// @p loopStart: seconds a looping voice comes back to at its end, so what is before it is heard once. 0 is its start.
- int play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart = 0.0f);
+ /// @p start, @p end: where the clip begins and is over in what was decoded, in seconds (end 0 is its natural end):
+ /// a clip the exporter re-encoded has silence before it and after it, and is played without either.
+ int play(int bufferId, float volume, float pan, bool loop, float rate, float loopStart = 0.0f, float start = 0.0f, float end = 0.0f);
void stop(int voiceId);
void pause(int voiceId);
void resume(int voiceId);
diff --git a/pipeline/src/assets/audioCook.ts b/pipeline/src/assets/audioCook.ts
index 434b755d2..71ea9e209 100644
--- a/pipeline/src/assets/audioCook.ts
+++ b/pipeline/src/assets/audioCook.ts
@@ -85,6 +85,20 @@ export function parseWav(bytes: Uint8Array): ParsedWav | null {
return { channels, sampleRate };
}
+/**
+ * Samples of silence an MP3 has before its sound once it is decoded: what the encoder holds back and what a decoder
+ * holds back, 576 and 529. It is the same at every rate and bitrate, and no player takes it off a file with no tag
+ * to say so, which these have not: a clip encoded here begins a fortieth of a second late and is longer than it was.
+ */
+export const MP3_LEAD_SAMPLES = 1105;
+
+/** How long a clip was before it was encoded, and the silence the encoding puts in front of it, in seconds; null if it is not a WAV. */
+export function encodedClipTrim(wav: Uint8Array): { seconds: number; lead: number } | null {
+ const parsed = parseWav(wav);
+ if (!parsed || !parsed.channels[0]?.length) return null;
+ return { seconds: parsed.channels[0].length / parsed.sampleRate, lead: MP3_LEAD_SAMPLES / parsed.sampleRate };
+}
+
type Mp3Encoder = {
configure(opts: { sampleRate: number; channels: 1 | 2; bitrate: number }): void;
encode(channels: Float32Array[]): Uint8Array;
diff --git a/pipeline/src/assets/cookAssets.ts b/pipeline/src/assets/cookAssets.ts
index c8943678d..cd7528d9a 100644
--- a/pipeline/src/assets/cookAssets.ts
+++ b/pipeline/src/assets/cookAssets.ts
@@ -443,6 +443,12 @@ export interface CookResult {
includedPaths: string[];
/** The shipped audio clips whose import settings choose how they load (not `auto`), by logical path. */
audioLoadTypes: Record<string, 'streaming' | 'decompress'>;
+ /**
+ * The shipped clips that were re-encoded, by logical path: how long each really is and the silence the encoding
+ * put before it, in seconds. The runtime plays a clip from past that silence and loops it at its real end;
+ * without this a loop that was seamless as a WAV has a gap every time round once it is packaged.
+ */
+ audioEncoded: Record<string, { seconds: number; lead: number }>;
/** uuids present in the project but unreachable — culled from the build. */
unused: string[];
/**
@@ -661,6 +667,7 @@ export async function cookAssets(
}
// WAV → MP3 (LAME wasm) rides the same lazy pattern; per-asset importer
// settings can opt a clip out (seamless loops) or pick a bitrate.
+ const encodedByUuid = new Map<string, { seconds: number; lead: number }>();
let audioEnc: typeof import('./audioCook') | null = null;
if (compressAudio) {
audioEnc = await import('./audioCook');
@@ -872,6 +879,8 @@ export async function cookAssets(
if (settings.compress) {
const mp3 = await audioEnc.encodeWavToMp3(data, settings.bitrateKbps);
if (mp3 && mp3.byteLength < data.byteLength) {
+ const trim = audioEnc.encodedClipTrim(data);
+ if (trim) encodedByUuid.set(entry.uuid, trim);
data = mp3;
ext = '.mp3';
} else if (!mp3) {
@@ -1107,9 +1116,15 @@ export async function cookAssets(
const type = e?.type === 'audio' ? audioLoadTypeOf(e.importer) : null;
if (e && type) audioLoadTypes[e.path] = type;
}
+ const audioEncoded: Record<string, { seconds: number; lead: number }> = {};
+ for (const [uuid, trim] of encodedByUuid) {
+ const e = byUuid.get(uuid);
+ // To a hundred-thousandth of a second, which is finer than a sample and keeps the config short.
+ if (e && reachable.has(uuid)) audioEncoded[e.path] = { seconds: Math.round(trim.seconds * 1e5) / 1e5, lead: Math.round(trim.lead * 1e5) / 1e5 };
+ }
return {
ok: failed.length === 0,
- outDir: absOut, manifestPath, included: [...reachable], includedPaths, audioLoadTypes, unused, warnings, failed,
+ outDir: absOut, manifestPath, included: [...reachable], includedPaths, audioLoadTypes, audioEncoded, unused, warnings, failed,
inclusion,
};
}
diff --git a/pipeline/src/export/exportGame.ts b/pipeline/src/export/exportGame.ts
index efd7df16c..5b7ad26d8 100644
--- a/pipeline/src/export/exportGame.ts
+++ b/pipeline/src/export/exportGame.ts
@@ -1318,7 +1318,7 @@ async function produceExport(opts: ExportGameOptions): Promise<ExportGameResult>
...(platform === 'web' && opts.webChannel === 'crazygames' && opts.leaderboardEncryptionKey
? { leaderboardEncryptionKey: opts.leaderboardEncryptionKey } : {}),
...(scriptsFile ? { scripts: scriptsFile } : {}),
- ...packagedRuntimeFields(withAudioLoadTypes(runtime, cook.audioLoadTypes)),
+ ...packagedRuntimeFields(withAudioLoadTypes(runtime, cook.audioLoadTypes, cook.audioEncoded)),
...(hotUpdate ? { hotUpdate } : {}),
...(sideModuleDeclarations(projectModules, platform).length > 0
? { sideModules: sideModuleDeclarations(projectModules, platform) } : {}),
diff --git a/pipeline/src/export/exportMiniGame.ts b/pipeline/src/export/exportMiniGame.ts
index d0b0a7845..216d44f55 100644
--- a/pipeline/src/export/exportMiniGame.ts
+++ b/pipeline/src/export/exportMiniGame.ts
@@ -506,7 +506,7 @@ export async function exportMiniGame(profile: MiniGameExportProfile, opts: {
// a WebGPU request cannot be honoured here — carrying it would be a config
// field the runtime is guaranteed to ignore.
const { renderBackend: _webgpuNotHere, ...packaged } =
- packagedRuntimeFields(withAudioLoadTypes(opts.runtime ?? DEFAULT_RUNTIME_CONFIG, cook.audioLoadTypes));
+ packagedRuntimeFields(withAudioLoadTypes(opts.runtime ?? DEFAULT_RUNTIME_CONFIG, cook.audioLoadTypes, cook.audioEncoded));
const themeColors = packaged.uiThemeColors;
const runtimeArgs = Object.entries(packaged)
.map(([k, v]) => (k === 'uiThemeColors'
diff --git a/pipeline/src/export/exportPlayable.ts b/pipeline/src/export/exportPlayable.ts
index c28846c04..a816bffe2 100644
--- a/pipeline/src/export/exportPlayable.ts
+++ b/pipeline/src/export/exportPlayable.ts
@@ -397,7 +397,7 @@ export async function exportPlayable(opts: {
`window.__GAME_PATHMAP__=${JSON.stringify(pathMap)};` +
span(`scenes/${sceneName}.json`, `window.__GAME_SCENES__=${JSON.stringify(scenes)};`) +
`window.__GAME_FIRST__=${JSON.stringify(sceneName)};` +
- `window.__GAME_RUNTIME__=${JSON.stringify(playableRuntimeFields(withAudioLoadTypes(opts.runtime ?? DEFAULT_RUNTIME_CONFIG, cook.audioLoadTypes)))};` +
+ `window.__GAME_RUNTIME__=${JSON.stringify(playableRuntimeFields(withAudioLoadTypes(opts.runtime ?? DEFAULT_RUNTIME_CONFIG, cook.audioLoadTypes, cook.audioEncoded)))};` +
span('game-bundle.js', `window.__GAME_BUNDLE__=${JSON.stringify(pack(Buffer.from(bundle, 'utf8')))};`);
const adProfile = opts.adProfile ?? genericPlayableProfile;
const network = playableAdInjection(adProfile, { title, orientation });
diff --git a/pipeline/src/project/runtimeConfig.ts b/pipeline/src/project/runtimeConfig.ts
index c3460bd2b..de1901c47 100644
--- a/pipeline/src/project/runtimeConfig.ts
+++ b/pipeline/src/project/runtimeConfig.ts
@@ -185,9 +185,12 @@ export function runtimeConfigOf(
*/
export function withAudioLoadTypes(
rc: RuntimeProjectConfig, loadTypes: Readonly<Record<string, 'streaming' | 'decompress'>> | undefined,
+ encoded?: Readonly<Record<string, { seconds: number; lead: number }>>,
): RuntimeProjectConfig {
- if (!loadTypes || Object.keys(loadTypes).length === 0) return rc;
- return { ...rc, audioConfig: { ...rc.audioConfig, loadTypes: { ...loadTypes } } };
+ const [types, trims] = [Object.keys(loadTypes ?? {}).length > 0, Object.keys(encoded ?? {}).length > 0];
+ if (!types && !trims) return rc;
+ // What the cook did to each clip it re-encoded rides beside how each is loaded: both are the cook's to know.
+ return { ...rc, audioConfig: { ...rc.audioConfig, ...(types ? { loadTypes: { ...loadTypes } } : {}), ...(trims ? { encoded: { ...encoded } } : {}) } };
}
/** The settings a shipped build carries, as `game.config.json` spells them. */
@@ -235,7 +238,7 @@ export function packagedRuntimeFields(rc: RuntimeProjectConfig): PackagedRuntime
...(rc.physics3dBackend === 'physx' ? { physics3dBackend: rc.physics3dBackend } : {}),
...(isDefaultPhysics3dWorld(rc.physics3dWorld) ? {} : { physics3dWorld: rc.physics3dWorld }),
...(rc.audioConfig.buses?.length || rc.audioConfig.maxVoices !== undefined
- || Object.keys(rc.audioConfig.loadTypes ?? {}).length > 0
+ || Object.keys(rc.audioConfig.loadTypes ?? {}).length > 0 || Object.keys(rc.audioConfig.encoded ?? {}).length > 0
? { audioConfig: rc.audioConfig } : {}),
...(Object.keys(rc.runtime).length > 0 ? { runtime: rc.runtime } : {}),
...(rc.achievements.length ? { achievements: rc.achievements } : {}),
diff --git a/pipeline/tests/audio-cook.test.ts b/pipeline/tests/audio-cook.test.ts
index 929725296..fd9bd120f 100644
--- a/pipeline/tests/audio-cook.test.ts
+++ b/pipeline/tests/audio-cook.test.ts
@@ -5,7 +5,7 @@
* chunks) and the WAV→MP3 transcode through the real LAME wasm encoder.
*/
import { describe, it, expect } from 'vitest';
-import { parseWav, encodeWavToMp3, audioImportSettings } from '../src/assets/audioCook';
+import { parseWav, encodeWavToMp3, audioImportSettings, encodedClipTrim, MP3_LEAD_SAMPLES } from '../src/assets/audioCook';
/** Build a minimal RIFF/WAVE file. */
function makeWav(opts: {
@@ -107,6 +107,35 @@ describe('encodeWavToMp3', () => {
});
});
+describe('what encoding does to a clip', () => {
+ it('says how long the clip really is, and the silence an MP3 has before its sound', () => {
+ expect(encodedClipTrim(makeWav({ sampleRate: 44100, frames: 88200 }))).toEqual({ seconds: 2, lead: 1105 / 44100 });
+ expect(encodedClipTrim(makeWav({ sampleRate: 22050, channels: 2, frames: 11025 }))).toEqual({ seconds: 0.5, lead: 1105 / 22050 });
+ expect(encodedClipTrim(new Uint8Array(64))).toBeNull();
+ expect(MP3_LEAD_SAMPLES).toBe(1105);
+ });
+
+ it('is what the encoder here does: the file it makes is longer than the clip by more than the lead', async () => {
+ // The number is the encoder's and a decoder's together, and is only true of this encoder: were it swapped for one
+ // that writes its own tag or holds back another amount, what the runtime is told would be wrong and this says so.
+ const wav = makeWav({ sampleRate: 44100, frames: 44100, sample: (n) => 0.3 * Math.sin((2 * Math.PI * 440 * n) / 44100) });
+ const mp3 = (await encodeWavToMp3(wav, 128))!;
+ // Frames of 1152 samples: enough of them to hold the clip and the lead, and no tag frame in front that a decoder would read the lead from.
+ let frames = 0;
+ for (let at = 0; at + 4 <= mp3.length;) {
+ if (mp3[at] !== 0xff || (mp3[at + 1]! & 0xe0) !== 0xe0) { at++; continue; }
+ const [bitrate, rate, padding] = [[0, 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320][mp3[at + 2]! >> 4]!, [44100, 48000, 32000][(mp3[at + 2]! >> 2) & 3]!, (mp3[at + 2]! >> 1) & 1];
+ if (!bitrate || !rate) { at++; continue; }
+ frames++;
+ at += Math.floor((144000 * bitrate) / rate) + padding;
+ }
+ expect(frames * 1152).toBeGreaterThanOrEqual(44100 + MP3_LEAD_SAMPLES - 529);
+ expect(frames * 1152).toBeLessThan(44100 + MP3_LEAD_SAMPLES + 1152 * 2);
+ const head = Buffer.from(mp3.subarray(0, 600)).toString('latin1');
+ expect(head.includes('Xing') || head.includes('Info')).toBe(false);
+ });
+});
+
describe('audioImportSettings', () => {
it('defaults and validates', () => {
expect(audioImportSettings(undefined)).toEqual({ compress: true, bitrateKbps: 128 });
diff --git a/pipeline/tests/cook-audio-encoded.test.ts b/pipeline/tests/cook-audio-encoded.test.ts
new file mode 100644
index 000000000..d84edbe7e
--- /dev/null
+++ b/pipeline/tests/cook-audio-encoded.test.ts
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file A cook that re-encodes a clip says what the encoding did to it.
+ *
+ * A WAV becomes an MP3 on the way into a build, and an MP3 decodes with silence before its sound and after it: a loop
+ * that was seamless has a gap every time round, and every effect is late. The cook cannot stop a decoder doing that,
+ * so it says how long each clip really is and how much silence leads it, for the runtime to play round. Held here:
+ * that it says so for the clips it re-encoded, by the path they are played by, and for no others.
+ */
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync, readdirSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import path from 'node:path';
+import { cookAssets } from '../src/assets/cookAssets';
+
+let root: string;
+const SCENE = 'aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa';
+const LOOP = '11111111-1111-4111-8111-111111111111';
+const RAW = '22222222-2222-4222-8222-222222222222';
+const UNUSED = '33333333-3333-4333-8333-333333333333';
+
+/** One second of a tone as a 16-bit mono WAV at @p rate. */
+function tone(rate: number, frames: number): Uint8Array {
+ const out = new Uint8Array(44 + frames * 2);
+ const view = new DataView(out.buffer);
+ const text = (at: number, t: string): void => { for (let i = 0; i < t.length; i++) out[at + i] = t.charCodeAt(i); };
+ text(0, 'RIFF'); view.setUint32(4, out.length - 8, true); text(8, 'WAVE');
+ text(12, 'fmt '); view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true);
+ view.setUint32(24, rate, true); view.setUint32(28, rate * 2, true); view.setUint16(32, 2, true); view.setUint16(34, 16, true);
+ text(36, 'data'); view.setUint32(40, frames * 2, true);
+ for (let n = 0; n < frames; n++) view.setInt16(44 + n * 2, Math.round(9000 * Math.sin((2 * Math.PI * 440 * n) / rate)), true);
+ return out;
+}
+
+function writeAsset(rel: string, type: string, uuid: string, body: string | Uint8Array, importer: Record<string, unknown> = {}): void {
+ const abs = path.join(root, rel);
+ mkdirSync(path.dirname(abs), { recursive: true });
+ writeFileSync(abs, body);
+ writeFileSync(`${abs}.meta`, JSON.stringify({ uuid, version: '2.0', type, importer }));
+}
+
+beforeAll(() => {
+ root = mkdtempSync(path.join(tmpdir(), 'estella-cook-audio-'));
+ writeAsset('assets/music/theme.wav', 'audio', LOOP, tone(44100, 88200));
+ // Kept as it is by its import settings: nothing is done to it, so there is nothing to say of it.
+ writeAsset('assets/music/raw.wav', 'audio', RAW, tone(22050, 22050), { compress: false });
+ writeAsset('assets/music/unused.wav', 'audio', UNUSED, tone(44100, 44100));
+ writeAsset('assets/scenes/main.esscene', 'scene', SCENE, JSON.stringify({ version: '1.0', name: 's', entities: [
+ { id: 1, name: 'Music', parent: null, children: [], components: [{ type: 'AudioSource', data: { clip: `@uuid:${LOOP}`, loop: true } }] },
+ { id: 2, name: 'Raw', parent: null, children: [], components: [{ type: 'AudioSource', data: { clip: `@uuid:${RAW}` } }] },
+ ] }));
+});
+
+afterAll(() => {
+ if (root) rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });
+});
+
+describe('a cook that re-encodes a clip', () => {
+ it('says how long it really is and what silence leads it, for that clip and no other', async () => {
+ const res = await cookAssets(root, { entryScenes: ['assets/scenes/main.esscene'], outDir: 'build', compressAudio: true });
+ expect(res.ok).toBe(true);
+ // Two seconds, and the 1105 samples an MP3 decodes with before its sound.
+ expect(res.audioEncoded).toEqual({ 'assets/music/theme.wav': { seconds: 2, lead: 0.02506 } });
+ const shipped = readdirSync(path.join(res.outDir, 'assets/music')).sort();
+ expect(shipped).toEqual(['raw.wav', 'theme.mp3']);
+ });
+
+ it('says nothing where it encodes nothing', async () => {
+ const res = await cookAssets(root, { entryScenes: ['assets/scenes/main.esscene'], outDir: 'build-plain' });
+ expect(res.audioEncoded).toEqual({});
+ expect(readdirSync(path.join(res.outDir, 'assets/music')).sort()).toEqual(['raw.wav', 'theme.wav']);
+ });
+});
diff --git a/pipeline/tests/runtime-config.test.ts b/pipeline/tests/runtime-config.test.ts
index e2e463d35..f0aae8bdc 100644
--- a/pipeline/tests/runtime-config.test.ts
+++ b/pipeline/tests/runtime-config.test.ts
@@ -183,6 +183,12 @@ describe('how an audio clip loads', () => {
it('rides the packaged audio config only when a clip asked for something', () => {
expect(packagedRuntimeFields(DEFAULT_RUNTIME_CONFIG).audioConfig).toBeUndefined();
expect(packagedRuntimeFields(withAudioLoadTypes(DEFAULT_RUNTIME_CONFIG, {})).audioConfig).toBeUndefined();
+ // What the cook's encoding did to a clip rides in the same place, with or without a load type beside it.
+ const encoded = { 'assets/theme.wav': { seconds: 53.3, lead: 0.02506 } };
+ expect(packagedRuntimeFields(withAudioLoadTypes(DEFAULT_RUNTIME_CONFIG, {}, encoded)).audioConfig).toEqual({ encoded });
+ expect(packagedRuntimeFields(withAudioLoadTypes(DEFAULT_RUNTIME_CONFIG, { 'assets/bgm.mp3': 'streaming' }, encoded)).audioConfig)
+ .toEqual({ loadTypes: { 'assets/bgm.mp3': 'streaming' }, encoded });
+ expect(packagedRuntimeFields(withAudioLoadTypes(DEFAULT_RUNTIME_CONFIG, {}, {})).audioConfig).toBeUndefined();
const rc = withAudioLoadTypes(DEFAULT_RUNTIME_CONFIG, { 'assets/bgm.mp3': 'streaming' });
expect(packagedRuntimeFields(rc).audioConfig).toEqual({ loadTypes: { 'assets/bgm.mp3': 'streaming' } });
expect(DEFAULT_RUNTIME_CONFIG.audioConfig.loadTypes).toBeUndefined();
diff --git a/sdk/etc/index.api.md b/sdk/etc/index.api.md
index a2aac8f3e..1197e1aa1 100644
--- a/sdk/etc/index.api.md
+++ b/sdk/etc/index.api.md
@@ -1197,6 +1197,7 @@ setBufferBudget: (bytes: number | null) => void
setBusDucking: (target: string, rule: BusDuckRule | null) => boolean
setBusEffects: (busName: string, effects: BusEffectDef[]) => boolean
setBusVolume: (busName: string, volume: number) => void
+setEncodedClips: (clips: Readonly<Record<string, AudioClipTrim>> | null) => void
setLoadTypes: (types: Readonly<Record<string, AudioLoadType>> | null) => void
setMasterVolume: (volume: number) => void
setMaxVoices: (max: number) => void
@@ -1366,6 +1367,7 @@ static prototype: AudioPool
## AudioProjectConfig — interface @experimental
```
buses: AudioBusDecl[] | undefined
+encoded: Record<string, { seconds: number; lead: number; }> | undefined
loadTypes: Record<string, "decompress" | "streaming"> | undefined
maxVoices: number | undefined
```
diff --git a/sdk/etc/index.native.api.md b/sdk/etc/index.native.api.md
index f3ea56060..59e4d4aec 100644
--- a/sdk/etc/index.native.api.md
+++ b/sdk/etc/index.native.api.md
@@ -1197,6 +1197,7 @@ setBufferBudget: (bytes: number | null) => void
setBusDucking: (target: string, rule: BusDuckRule | null) => boolean
setBusEffects: (busName: string, effects: BusEffectDef[]) => boolean
setBusVolume: (busName: string, volume: number) => void
+setEncodedClips: (clips: Readonly<Record<string, AudioClipTrim>> | null) => void
setLoadTypes: (types: Readonly<Record<string, AudioLoadType>> | null) => void
setMasterVolume: (volume: number) => void
setMaxVoices: (max: number) => void
@@ -1366,6 +1367,7 @@ static prototype: AudioPool
## AudioProjectConfig — interface @experimental
```
buses: AudioBusDecl[] | undefined
+encoded: Record<string, { seconds: number; lead: number; }> | undefined
loadTypes: Record<string, "decompress" | "streaming"> | undefined
maxVoices: number | undefined
```
@@ -6760,7 +6762,7 @@ width: number
```
es_audioLoad: ((bytes: ArrayBuffer, stream?: boolean) => { id: number; duration: number; bytes: number; streamed?: boolean; } | null) | undefined
es_audioPause: ((voiceId: number) => void) | undefined
-es_audioPlay: ((bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number) => number) | undefined
+es_audioPlay: ((bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number) => number) | undefined
es_audioResume: ((voiceId: number) => void) | undefined
es_audioResumeAll: (() => void) | undefined
es_audioSetLoop: ((voiceId: number, loop: boolean) => void) | undefined
diff --git a/sdk/etc/index.node.api.md b/sdk/etc/index.node.api.md
index 7f9ef5e89..48c7cfcfe 100644
--- a/sdk/etc/index.node.api.md
+++ b/sdk/etc/index.node.api.md
@@ -1192,6 +1192,7 @@ setBufferBudget: (bytes: number | null) => void
setBusDucking: (target: string, rule: BusDuckRule | null) => boolean
setBusEffects: (busName: string, effects: BusEffectDef[]) => boolean
setBusVolume: (busName: string, volume: number) => void
+setEncodedClips: (clips: Readonly<Record<string, AudioClipTrim>> | null) => void
setLoadTypes: (types: Readonly<Record<string, AudioLoadType>> | null) => void
setMasterVolume: (volume: number) => void
setMaxVoices: (max: number) => void
@@ -1361,6 +1362,7 @@ static prototype: AudioPool
## AudioProjectConfig — interface @experimental
```
buses: AudioBusDecl[] | undefined
+encoded: Record<string, { seconds: number; lead: number; }> | undefined
loadTypes: Record<string, "decompress" | "streaming"> | undefined
maxVoices: number | undefined
```
diff --git a/sdk/etc/minigame.api.md b/sdk/etc/minigame.api.md
index f20e8db20..b5845ec9d 100644
--- a/sdk/etc/minigame.api.md
+++ b/sdk/etc/minigame.api.md
@@ -1192,6 +1192,7 @@ setBufferBudget: (bytes: number | null) => void
setBusDucking: (target: string, rule: BusDuckRule | null) => boolean
setBusEffects: (busName: string, effects: BusEffectDef[]) => boolean
setBusVolume: (busName: string, volume: number) => void
+setEncodedClips: (clips: Readonly<Record<string, AudioClipTrim>> | null) => void
setLoadTypes: (types: Readonly<Record<string, AudioLoadType>> | null) => void
setMasterVolume: (volume: number) => void
setMaxVoices: (max: number) => void
@@ -1361,6 +1362,7 @@ static prototype: AudioPool
## AudioProjectConfig — interface @experimental
```
buses: AudioBusDecl[] | undefined
+encoded: Record<string, { seconds: number; lead: number; }> | undefined
loadTypes: Record<string, "decompress" | "streaming"> | undefined
maxVoices: number | undefined
```
@@ -6490,8 +6492,8 @@ dispose: () => void
ensureResumed: () => Promise<void>
initialize: (_options?: AudioBackendInitOptions) => Promise<void>
isReady: boolean
-loadBuffer: (src: string) => Promise<AudioBufferHandle>
-loadBufferFromData: (src: string, _data: ArrayBuffer) => Promise<AudioBufferHandle>
+loadBuffer: (src: string, options?: AudioLoadOptions) => Promise<AudioBufferHandle>
+loadBufferFromData: (src: string, _data: ArrayBuffer, options?: AudioLoadOptions) => Promise<AudioBufferHandle>
mixer: AudioMixer | null
name: string
play: (buffer: AudioBufferHandle, config: PlayConfig) => AudioHandle
diff --git a/sdk/etc/wechat.api.md b/sdk/etc/wechat.api.md
index 30aa6f9b0..6414313b8 100644
--- a/sdk/etc/wechat.api.md
+++ b/sdk/etc/wechat.api.md
@@ -1192,6 +1192,7 @@ setBufferBudget: (bytes: number | null) => void
setBusDucking: (target: string, rule: BusDuckRule | null) => boolean
setBusEffects: (busName: string, effects: BusEffectDef[]) => boolean
setBusVolume: (busName: string, volume: number) => void
+setEncodedClips: (clips: Readonly<Record<string, AudioClipTrim>> | null) => void
setLoadTypes: (types: Readonly<Record<string, AudioLoadType>> | null) => void
setMasterVolume: (volume: number) => void
setMaxVoices: (max: number) => void
@@ -1361,6 +1362,7 @@ static prototype: AudioPool
## AudioProjectConfig — interface @experimental
```
buses: AudioBusDecl[] | undefined
+encoded: Record<string, { seconds: number; lead: number; }> | undefined
loadTypes: Record<string, "decompress" | "streaming"> | undefined
maxVoices: number | undefined
```
diff --git a/sdk/src/audio/Audio.ts b/sdk/src/audio/Audio.ts
index 1f9ffc4ea..108a3385f 100644
--- a/sdk/src/audio/Audio.ts
+++ b/sdk/src/audio/Audio.ts
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
-import type { PlatformAudioBackend, AudioBufferHandle, AudioHandle, PlayConfig, AudioDelivery, AudioLoadOptions, AudioLoadType } from './PlatformAudioBackend';
+import type { PlatformAudioBackend, AudioBufferHandle, AudioHandle, PlayConfig, AudioDelivery, AudioLoadOptions, AudioLoadType, AudioClipTrim } from './PlatformAudioBackend';
import type { AudioMixer, BusDuckRule } from './AudioMixer';
import type { BusEffectDef } from './BusEffects';
import { defineResource } from '../ecs/resource';
@@ -139,6 +139,7 @@ export class AudioAPI {
private bufferBudgetOverride_: number | null = null;
/** Importer choices by authored path; a clip with none is `auto`. */
private loadTypes_ = new Map<string, AudioLoadType>();
+ private encoded_ = new Map<string, AudioClipTrim>();
private bgmHandle_: AudioHandle | null = null;
private bgmRequest_ = 0;
private playbackEpoch_ = 0;
@@ -284,9 +285,19 @@ export class AudioAPI {
this.loadTypes_ = new Map(Object.entries(types ?? {}).map(([path, type]) => [normalizeClipPath(path), type]));
}
+ /**
+ * What the exporter's encoding did to each clip, keyed as {@link setLoadTypes} is. A clip named here is played
+ * from past the silence encoding put before it and ended or looped at its real length. A clip already loaded
+ * keeps how it was loaded.
+ */
+ setEncodedClips(clips: Readonly<Record<string, AudioClipTrim>> | null): void {
+ this.encoded_ = new Map(Object.entries(clips ?? {}).map(([path, trim]) => [normalizeClipPath(path), trim]));
+ }
+
private loadOptions_(url: string): AudioLoadOptions | undefined {
- const loadType = this.loadTypes_.get(normalizeClipPath(url));
- return loadType ? { loadType } : undefined;
+ const key = normalizeClipPath(url);
+ const [loadType, trim] = [this.loadTypes_.get(key), this.encoded_.get(key)];
+ return loadType || trim ? { ...(loadType ? { loadType } : {}), ...(trim ? { trim } : {}) } : undefined;
}
/** Buffer residency counters — the observability side of the budget. */
diff --git a/sdk/src/audio/AudioProjectConfig.ts b/sdk/src/audio/AudioProjectConfig.ts
index c800587a7..8ed4b6c05 100644
--- a/sdk/src/audio/AudioProjectConfig.ts
+++ b/sdk/src/audio/AudioProjectConfig.ts
@@ -35,6 +35,11 @@ export interface AudioProjectConfig {
* editor and the exporter from the assets' import settings, never by hand in the project file.
*/
loadTypes?: Record<string, Exclude<AudioLoadType, 'auto'>>;
+ /**
+ * Clips the exporter re-encoded, by project-relative path: how long each really is and the silence the encoding
+ * put before it, in seconds. Written by the exporter only; a project played from its own files has none.
+ */
+ encoded?: Record<string, { seconds: number; lead: number }>;
}
const clamp01 = (v: number) => Math.max(0, Math.min(1, v));
@@ -77,6 +82,15 @@ export function parseAudioProjectConfig(raw: unknown): AudioProjectConfig {
}
if (Object.keys(loadTypes).length > 0) config.loadTypes = loadTypes;
}
+ const rawEncoded = (raw as { encoded?: unknown } | undefined)?.encoded;
+ if (rawEncoded && typeof rawEncoded === 'object') {
+ const encoded: Record<string, { seconds: number; lead: number }> = {};
+ for (const [path, value] of Object.entries(rawEncoded)) {
+ const v = value as { seconds?: unknown; lead?: unknown } | null;
+ if (path && v && typeof v.seconds === 'number' && v.seconds > 0 && typeof v.lead === 'number' && v.lead >= 0 && v.lead < 1) encoded[path] = { seconds: v.seconds, lead: v.lead };
+ }
+ if (Object.keys(encoded).length > 0) config.encoded = encoded;
+ }
return config;
}
@@ -90,6 +104,7 @@ export function applyAudioProjectConfig(audio: AudioAPI, config: AudioProjectCon
// while the loop below returns early where there is no mixer graph.
if (config?.maxVoices !== undefined) audio.setMaxVoices(config.maxVoices);
if (config?.loadTypes !== undefined) audio.setLoadTypes(config.loadTypes);
+ if (config?.encoded !== undefined) audio.setEncodedClips(config.encoded);
for (const bus of config?.buses ?? []) {
if (!audio.ensureBus(bus.name, bus.parent)) return; // no mixer: nothing applies
if (bus.volume !== undefined) audio.setBusVolume(bus.name, bus.volume);
diff --git a/sdk/src/audio/MiniGameAudioBackend.ts b/sdk/src/audio/MiniGameAudioBackend.ts
index 4c66ec34d..9f38d814d 100644
--- a/sdk/src/audio/MiniGameAudioBackend.ts
+++ b/sdk/src/audio/MiniGameAudioBackend.ts
@@ -11,7 +11,7 @@
* players, not a DSP graph, so routing labels are honored by the engine
* and the graph itself is absent (`mixer` is null by contract).
*/
-import type { AudioHandle, AudioBufferHandle, PlayConfig, PlatformAudioBackend, AudioBackendInitOptions } from './PlatformAudioBackend';
+import type { AudioHandle, AudioBufferHandle, PlayConfig, PlatformAudioBackend, AudioBackendInitOptions, AudioLoadOptions } from './PlatformAudioBackend';
import type { AudioMixer } from './AudioMixer';
import type { MiniGameGlobal, MiniGameInnerAudioContext } from '../platform/minigame/api';
import { scalar } from '../math/scalar';
@@ -146,6 +146,8 @@ export class MiniGameAudioBackend implements PlatformAudioBackend {
private contexts_ = new Map<number, MiniGameInnerAudioContext>();
private readonly device_: MiniGameAudioDeviceState = { suspended: false, pending: new Map() };
private urlCache_ = new Map<number, string>();
+ /** The silence encoding put before a clip the exporter re-encoded, by buffer: a player is started past it. */
+ private leads_ = new Map<number, number>();
private nextId_ = 0;
constructor(global: MiniGameGlobal, label = 'MiniGame') {
@@ -169,19 +171,22 @@ export class MiniGameAudioBackend implements PlatformAudioBackend {
// Mini-game hosts do not gate playback on a user gesture
}
- async loadBuffer(src: string): Promise<AudioBufferHandle> {
+ async loadBuffer(src: string, options?: AudioLoadOptions): Promise<AudioBufferHandle> {
const id = ++this.nextId_;
this.urlCache_.set(id, src);
- return { id, duration: 0 };
+ // The player decodes the file itself and its length is not known here, so the lead the exporter measured is taken as it is.
+ if (options?.trim && options.trim.lead > 0) this.leads_.set(id, options.trim.lead);
+ return { id, duration: options?.trim?.seconds ?? 0 };
}
/** No decoder here — the player streams from `src`, and the bytes are dropped.
* Callers avoid the wasted fetch by reading {@link delivery} first. */
- async loadBufferFromData(src: string, _data: ArrayBuffer): Promise<AudioBufferHandle> {
- return this.loadBuffer(src);
+ async loadBufferFromData(src: string, _data: ArrayBuffer, options?: AudioLoadOptions): Promise<AudioBufferHandle> {
+ return this.loadBuffer(src, options);
}
unloadBuffer(handle: AudioBufferHandle): void {
+ this.leads_.delete(handle.id);
this.urlCache_.delete(handle.id);
}
@@ -195,14 +200,16 @@ export class MiniGameAudioBackend implements PlatformAudioBackend {
ctx.loop = config.loop ?? false;
ctx.volume = scalar.clamp01(config.volume ?? 1.0);
ctx.playbackRate = config.playbackRate ?? 1.0;
- ctx.startTime = config.startOffset ?? 0;
+ const lead = this.leads_.get(buffer.id) ?? 0;
+ ctx.startTime = lead + (config.startOffset ?? 0);
ctx.obeyMuteSwitch = false;
const handleId = ++this.nextId_;
this.contexts_.set(handleId, ctx);
const handle = new MiniGameAudioHandle(handleId, ctx, this.contexts_, this.device_);
- if (config.loop && config.loopStart && config.loopStart > 0) handle.loopFrom(config.loopStart);
+ // Sent back past the lead each time round as well, where the host's own loop would play it again.
+ if (config.loop && ((config.loopStart && config.loopStart > 0) || lead > 0)) handle.loopFrom(lead + (config.loopStart ?? 0));
ctx.onEnded(() => handle.onNaturalEnd());
ctx.onError((res) => {
log.error('audio', `Playback error for "${url}"`, res.errMsg);
diff --git a/sdk/src/audio/NativeAudioBackend.ts b/sdk/src/audio/NativeAudioBackend.ts
index fb17b5afd..5c89d0f5c 100644
--- a/sdk/src/audio/NativeAudioBackend.ts
+++ b/sdk/src/audio/NativeAudioBackend.ts
@@ -22,6 +22,7 @@ import type {
} from './PlatformAudioBackend';
import type { NativeAudioBridge } from '../platform/native/bridge';
import type { AudioLoadOptions } from './PlatformAudioBackend';
+import { clipSpan } from './PlatformAudioBackend';
import { decodedPcmBytes, shouldStream } from './audioStreaming';
import { platformReadFile } from '../platform/base';
import { scalar } from '../math/scalar';
@@ -71,6 +72,8 @@ export class NativeAudioBackend implements PlatformAudioBackend {
readonly isReady = true;
private readonly voices_ = new Map<number, NativeAudioHandle>();
+ /** Where a clip the exporter re-encoded really lies in what the host decoded, by buffer. */
+ private readonly spans_ = new Map<number, { lead: number; end: number }>();
/** Reused across frames: a visualizer asks every frame, and a fresh view per
* ask would be garbage the interpreter has to collect at 60 Hz. */
private spectrumView_: Uint8Array | null = null;
@@ -102,6 +105,8 @@ export class NativeAudioBackend implements PlatformAudioBackend {
private loadFromBytes_(bytes: ArrayBuffer, options?: AudioLoadOptions): AudioBufferHandle {
const loaded = this.audio_.load(bytes, shouldStream(options?.loadType, bytes));
if (!loaded) throw new Error('[native] audio decode failed');
+ const span = clipSpan(options?.trim, loaded.duration);
+ if (span) { this.spans_.set(loaded.id, span); loaded.duration = span.end - span.lead; }
// What the host DID, not what was asked: one that could not stream the file decoded it, and its PCM
// is then what counts against the budget.
if (!loaded.streamed) return { id: loaded.id, duration: loaded.duration, bytes: loaded.bytes };
@@ -112,17 +117,22 @@ export class NativeAudioBackend implements PlatformAudioBackend {
}
unloadBuffer(handle: AudioBufferHandle): void {
+ this.spans_.delete(handle.id);
this.audio_.unload(handle.id);
}
play(buffer: AudioBufferHandle, config: PlayConfig): AudioHandle {
+ const span = this.spans_.get(buffer.id);
const voiceId = this.audio_.play(
buffer.id,
config.volume ?? 1,
scalar.clamp(config.pan ?? 0, -1, 1),
config.loop ?? false,
config.playbackRate ?? 1,
- config.loop && config.loopStart && config.loopStart > 0 ? config.loopStart : 0,
+ // Places in what the host decoded: the loop start, where the clip begins, and where it ends (0 is its natural end).
+ config.loop && config.loopStart && config.loopStart > 0 ? (span?.lead ?? 0) + config.loopStart : span?.lead ?? 0,
+ span?.lead ?? 0,
+ span?.end ?? 0,
);
const handle = new NativeAudioHandle(
voiceId, this.audio_, buffer.duration, () => this.voices_.delete(voiceId),
diff --git a/sdk/src/audio/PlatformAudioBackend.ts b/sdk/src/audio/PlatformAudioBackend.ts
index 8af488a34..3621f99fa 100644
--- a/sdk/src/audio/PlatformAudioBackend.ts
+++ b/sdk/src/audio/PlatformAudioBackend.ts
@@ -26,8 +26,26 @@ export interface AudioHandle {
*/
export type AudioLoadType = 'auto' | 'decompress' | 'streaming';
+/**
+ * What re-encoding did to a clip: `seconds` is how long it really is, `lead` the silence the encoding put in front
+ * of it. A backend plays it from past the lead and ends or loops it at lead + seconds, so a clip is in a packaged
+ * game what it was as a WAV: on time, and seamless where it was made to loop.
+ */
+export interface AudioClipTrim { seconds: number; lead: number }
+
+/**
+ * Where a clip really lies in what a decoder made of it, in seconds: from `lead` to `end`. A decoder that took the
+ * lead off itself is seen by how long its result is, and is not asked to take it off twice.
+ */
+export function clipSpan(trim: AudioClipTrim | undefined, decodedSeconds: number): { lead: number; end: number } | null {
+ if (!trim) return null;
+ const lead = decodedSeconds > trim.seconds + trim.lead * 0.5 ? trim.lead : 0;
+ return { lead, end: Math.min(decodedSeconds, lead + trim.seconds) };
+}
+
export interface AudioLoadOptions {
loadType?: AudioLoadType;
+ trim?: AudioClipTrim;
}
export interface AudioBufferHandle {
diff --git a/sdk/src/audio/WebAudioBackend.ts b/sdk/src/audio/WebAudioBackend.ts
index 96d58ba6d..4f136a9b2 100644
--- a/sdk/src/audio/WebAudioBackend.ts
+++ b/sdk/src/audio/WebAudioBackend.ts
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
import type { AudioHandle, AudioBufferHandle, PlayConfig, PlatformAudioBackend, AudioBackendInitOptions, AudioLoadOptions } from './PlatformAudioBackend';
+import { clipSpan } from './PlatformAudioBackend';
import { decodedPcmBytes, estimateAudioSeconds, shouldStream } from './audioStreaming';
import { AudioMixer } from './AudioMixer';
import { AudioPool, type PooledAudioNode } from './AudioPool';
@@ -24,8 +25,11 @@ class WebAudioHandle implements AudioHandle {
private startOffset_: number;
private playbackRate_: number;
private loop_: boolean;
- /** Where a loop comes back to, in seconds; 0 is the start. */
+ /** Where a loop comes back to, as a place in the buffer; the clip's own start where no loop start was given. */
private readonly loopStart_: number;
+ /** Where the clip begins and ends in the buffer. Times given out are counted from its beginning. */
+ private readonly lead_: number;
+ private readonly end_: number;
private readonly duration_: number;
constructor(
@@ -37,11 +41,15 @@ class WebAudioHandle implements AudioHandle {
context: AudioContext,
startOffset: number,
loopStart = 0,
+ lead = 0,
+ end = buffer.duration,
) {
this.id = id;
this.source_ = source;
this.buffer_ = buffer;
- this.duration_ = buffer.duration;
+ this.lead_ = lead;
+ this.end_ = end;
+ this.duration_ = end - lead;
this.poolNode_ = poolNode;
this.pool_ = pool;
this.context_ = context;
@@ -93,7 +101,7 @@ class WebAudioHandle implements AudioHandle {
/** A place in the clip for a time since it began: past the end, it has been round from the loop's start. */
private wrapped_(position: number): number {
- const end = this.buffer_!.duration;
+ const end = this.end_;
if (position < end) return position;
const round = end - this.loopStart_;
return round > 0 ? this.loopStart_ + ((position - this.loopStart_) % round) : position % end;
@@ -107,7 +115,7 @@ class WebAudioHandle implements AudioHandle {
try {
source.buffer = this.buffer_;
source.loop = this.loop_;
- if (this.loopStart_ > 0) { source.loopStart = this.loopStart_; source.loopEnd = this.buffer_!.duration; }
+ if (this.loop_ && (this.loopStart_ > 0 || this.end_ < this.buffer_!.duration)) { source.loopStart = this.loopStart_; source.loopEnd = this.end_; }
source.playbackRate.value = this.playbackRate_;
source.connect(this.poolNode_!.gain);
this.bindOnEnded_(source);
@@ -162,9 +170,9 @@ class WebAudioHandle implements AudioHandle {
get currentTime(): number {
if (this.stopped_) return 0;
- if (!this.playing_) return this.pausedAt_;
+ if (!this.playing_) return this.pausedAt_ - this.lead_;
const elapsed = (this.context_!.currentTime - this.startedAt_) * this.playbackRate_;
- return this.startOffset_ + elapsed;
+ return this.startOffset_ + elapsed - this.lead_;
}
get duration(): number {
@@ -329,8 +337,10 @@ function decodedBytes(buf: AudioBuffer): number {
interface BufferRecord {
buffer: AudioBuffer | null;
url: string;
+ /** Where the clip really lies in the decoded buffer, for one the exporter re-encoded: from `lead` to `end`, in seconds. */
+ span?: { lead: number; end: number };
/** A streamed clip: the object URL its media element plays, and its length. */
- stream?: { objectUrl: string; duration: number };
+ stream?: { objectUrl: string; duration: number; lead?: number };
}
interface SourceRecord {
@@ -508,7 +518,8 @@ export class WebAudioBackend implements PlatformAudioBackend {
const duration = record.stream.duration;
return { id, duration, bytes: 0, streamed: true, savedBytes: decodedPcmBytes(duration) };
}
- return { id, duration: record.buffer!.duration, bytes: decodedBytes(record.buffer!) };
+ // As long as the clip is, not as long as the decoder made it.
+ return { id, duration: record.span ? record.span.end - record.span.lead : record.buffer!.duration, bytes: decodedBytes(record.buffer!) };
}
private async loadSource_(url: string, read: () => Promise<ArrayBuffer>, options?: AudioLoadOptions): Promise<AudioBufferHandle> {
@@ -552,7 +563,7 @@ export class WebAudioBackend implements PlatformAudioBackend {
if (cached) return cached;
}
if (shouldStream(options?.loadType, data)) {
- return this.streamBuffer_(url, data, source, context, generation);
+ return this.streamBuffer_(url, data, source, context, generation, options);
}
let audioBuffer: AudioBuffer;
@@ -565,10 +576,11 @@ export class WebAudioBackend implements PlatformAudioBackend {
this.assertLoadContext_(context, generation);
const id = ++this.nextBufferId_;
- this.buffers_.set(id, { buffer: audioBuffer, url });
+ const span = clipSpan(options?.trim, audioBuffer.duration);
+ this.buffers_.set(id, { buffer: audioBuffer, url, ...(span ? { span } : {}) });
if (this.sources_.get(url) === source) source.bufferId = id;
- return { id, duration: audioBuffer.duration, bytes: decodedBytes(audioBuffer) };
+ return this.handleOf_(id, this.buffers_.get(id)!);
}
/**
@@ -577,7 +589,7 @@ export class WebAudioBackend implements PlatformAudioBackend {
* clip does not take.
*/
private async streamBuffer_(
- url: string, data: ArrayBuffer, source: SourceRecord, context: AudioContext, generation: number,
+ url: string, data: ArrayBuffer, source: SourceRecord, context: AudioContext, generation: number, options?: AudioLoadOptions,
): Promise<AudioBufferHandle> {
const objectUrl = URL.createObjectURL(new Blob([data], { type: mimeOf(url) }));
const duration = await probeDuration(objectUrl, estimateAudioSeconds(data));
@@ -586,7 +598,8 @@ export class WebAudioBackend implements PlatformAudioBackend {
throw new Error('AudioContext changed while loading audio');
}
const id = ++this.nextBufferId_;
- this.buffers_.set(id, { buffer: null, url, stream: { objectUrl, duration } });
+ const span = clipSpan(options?.trim, duration);
+ this.buffers_.set(id, { buffer: null, url, stream: { objectUrl, duration: span ? span.end - span.lead : duration, ...(span?.lead ? { lead: span.lead } : {}) } });
if (this.sources_.get(url) === source) source.bufferId = id;
return this.handleOf_(id, this.buffers_.get(id)!);
}
@@ -601,7 +614,7 @@ export class WebAudioBackend implements PlatformAudioBackend {
}
}
- private playStream_(bufferId: number, stream: { objectUrl: string; duration: number }, config: PlayConfig): AudioHandle {
+ private playStream_(bufferId: number, stream: { objectUrl: string; duration: number; lead?: number }, config: PlayConfig): AudioHandle {
const context = this.context_!;
const pool = this.pool_!;
const mixer = this.mixer_!;
@@ -628,10 +641,11 @@ export class WebAudioBackend implements PlatformAudioBackend {
const handle = new WebStreamHandle(
++this.nextHandleId_, element, node, poolNode, pool, stream.duration,
(h) => this.releaseStream_(h), () => this.ensureResumed(),
- config.loop && config.loopStart && config.loopStart < stream.duration ? config.loopStart : 0,
+ // Past the silence encoding put before it, each time round as well as the first.
+ config.loop && ((config.loopStart && config.loopStart < stream.duration) || stream.lead) ? (stream.lead ?? 0) + (config.loopStart ?? 0) : 0,
);
this.streams_.set(bufferId, handle);
- handle.begin(config.startOffset ?? 0);
+ handle.begin((stream.lead ?? 0) + (config.startOffset ?? 0));
return handle;
}
@@ -668,8 +682,12 @@ export class WebAudioBackend implements PlatformAudioBackend {
throw new Error(`Buffer ${buffer.id} not found`);
}
- const startOffset = config.startOffset ?? 0;
- const loopStart = config.loop && config.loopStart && config.loopStart > 0 && config.loopStart < audioBuffer.duration ? config.loopStart : 0;
+ // Everything below is a place in the decoded buffer: the clip begins at `lead` and is over at `end`.
+ const [lead, end] = [record!.span?.lead ?? 0, record!.span?.end ?? audioBuffer.duration];
+ const startOffset = lead + (config.startOffset ?? 0);
+ const loopStart = lead + (config.loop && config.loopStart && config.loopStart > 0 && lead + config.loopStart < end ? config.loopStart : 0);
+ // Round a part of the buffer and not the whole of it: from a loop start, or within a clip that has silence round it.
+ const region = !!config.loop && (loopStart > 0 || end < audioBuffer.duration);
const poolNode = this.pool_.acquire();
let source: AudioBufferSourceNode;
try {
@@ -678,7 +696,7 @@ export class WebAudioBackend implements PlatformAudioBackend {
source.buffer = audioBuffer;
source.loop = config.loop ?? false;
// Round from here, not from the top: the node does it between two samples.
- if (loopStart > 0) { source.loopStart = loopStart; source.loopEnd = audioBuffer.duration; }
+ if (region) { source.loopStart = loopStart; source.loopEnd = end; }
source.playbackRate.value = config.playbackRate ?? 1.0;
source.connect(poolNode.gain);
@@ -688,7 +706,9 @@ export class WebAudioBackend implements PlatformAudioBackend {
const busName = config.bus ?? 'sfx';
const bus = this.mixer_.getBus(busName) ?? this.mixer_.sfx;
poolNode.panner.connect(bus.input);
- source.start(0, startOffset);
+ // A clip with silence after it is stopped where it ends, so its end is told on time.
+ if (record!.span && !config.loop) source.start(0, startOffset, Math.max(0, end - startOffset));
+ else source.start(0, startOffset);
} catch (error) {
this.pool_.release(poolNode);
throw error;
@@ -696,7 +716,7 @@ export class WebAudioBackend implements PlatformAudioBackend {
const handleId = ++this.nextHandleId_;
const handle = new WebAudioHandle(
- handleId, source, audioBuffer, poolNode, this.pool_, this.context_, startOffset, loopStart
+ handleId, source, audioBuffer, poolNode, this.pool_, this.context_, startOffset, loopStart, lead, end
);
return handle;
diff --git a/sdk/src/platform/native/bridge.ts b/sdk/src/platform/native/bridge.ts
index 5851fe8ea..597d44fcc 100644
--- a/sdk/src/platform/native/bridge.ts
+++ b/sdk/src/platform/native/bridge.ts
@@ -148,7 +148,8 @@ export interface NativeAudioBridge {
/** Start a voice on a loaded buffer. Returns its voice id, or -1 if the buffer
* is unknown. */
/** @param loopStart seconds a looping voice comes back to; 0 is its start. A host older than this takes five arguments and loops from the start. */
- play(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number): number;
+ /** @param start, end seconds in the decoded clip that it begins from and is over at (0 for end is its natural end): a clip the exporter re-encoded has silence before and after it. */
+ play(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number): number;
stop(voiceId: number): void;
pause(voiceId: number): void;
resume(voiceId: number): void;
diff --git a/sdk/src/platform/native/hostBridge.ts b/sdk/src/platform/native/hostBridge.ts
index 9b6debae2..dabdd26b2 100644
--- a/sdk/src/platform/native/hostBridge.ts
+++ b/sdk/src/platform/native/hostBridge.ts
@@ -144,7 +144,7 @@ export interface NativeHostBindings extends NativeDeviceHost {
es_audioSpectrum?(bins: number): ArrayBuffer | null;
es_audioLoad?(bytes: ArrayBuffer, stream?: boolean): { id: number; duration: number; bytes: number; streamed?: boolean } | null;
es_audioUnload?(bufferId: number): void;
- es_audioPlay?(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number): number;
+ es_audioPlay?(bufferId: number, volume: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number): number;
es_audioStop?(voiceId: number): void;
es_audioPause?(voiceId: number): void;
es_audioResume?(voiceId: number): void;
@@ -388,7 +388,7 @@ function hostAudio(
return {
load: (bytes, stream) => bindings.es_audioLoad!(bytes, stream === true),
unload: (id) => bindings.es_audioUnload!(id),
- play: (buf, vol, pan, loop, rate, loopStart) => bindings.es_audioPlay!(buf, vol, pan, loop, rate, loopStart ?? 0),
+ play: (buf, vol, pan, loop, rate, loopStart, start, end) => bindings.es_audioPlay!(buf, vol, pan, loop, rate, loopStart ?? 0, start ?? 0, end ?? 0),
stop: (v) => bindings.es_audioStop!(v),
pause: (v) => bindings.es_audioPause!(v),
resume: (v) => bindings.es_audioResume!(v),
diff --git a/sdk/tests/audio-encoded-clips.test.ts b/sdk/tests/audio-encoded-clips.test.ts
new file mode 100644
index 000000000..2d8cac271
--- /dev/null
+++ b/sdk/tests/audio-encoded-clips.test.ts
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file audio-encoded-clips.test.ts
+ * @brief What the exporter's encoding did to a clip reaches the backend that plays it.
+ *
+ * A WAV becomes an MP3 when a game is packaged, and an MP3 decodes with silence before its sound and after
+ * it. Held here: that the exporter's note of how long a clip really is travels from the packaged config
+ * through the audio API to the backend's load, by the path the clip is played by, and what a backend makes
+ * of it.
+ */
+import { describe, it, expect, vi } from 'vitest';
+import { AudioAPI } from '../src/audio/Audio';
+import { applyAudioProjectConfig, parseAudioProjectConfig } from '../src/audio/AudioProjectConfig';
+import { clipSpan, type PlatformAudioBackend } from '../src/audio/PlatformAudioBackend';
+
+function backendOf() {
+ const loads: [string, unknown][] = [];
+ const backend = {
+ name: 'Mock', mixer: null, isReady: true, delivery: 'url',
+ initialize: vi.fn().mockResolvedValue(undefined), ensureResumed: vi.fn().mockResolvedValue(undefined),
+ loadBuffer: vi.fn(async (src: string, options: unknown) => { loads.push([src, options]); return { id: loads.length, duration: 1 }; }),
+ loadBufferFromData: vi.fn(), unloadBuffer: vi.fn(), play: vi.fn(), suspend: vi.fn(), resume: vi.fn(), dispose: vi.fn(),
+ } as unknown as PlatformAudioBackend;
+ return { backend, loads };
+}
+
+describe('a clip the exporter re-encoded', () => {
+ it('is loaded with what the encoding did to it, by the path it is played by', async () => {
+ const { backend, loads } = backendOf();
+ const audio = new AudioAPI(backend);
+ audio.setEncodedClips({ 'assets/music/theme.wav': { seconds: 53.3, lead: 0.02506 } });
+ audio.setLoadTypes({ 'assets/music/theme.wav': 'decompress' });
+ await audio.preload('assets/music/theme.wav');
+ await audio.preload('assets/sfx/plain.ogg');
+ expect(loads).toEqual([
+ ['assets/music/theme.wav', { loadType: 'decompress', trim: { seconds: 53.3, lead: 0.02506 } }],
+ ['assets/sfx/plain.ogg', undefined],
+ ]);
+ });
+
+ it('is read from the packaged config and applied, and what is not a length is left out', () => {
+ const config = parseAudioProjectConfig({ encoded: {
+ 'a.wav': { seconds: 2, lead: 0.025 }, 'bad.wav': { seconds: -1, lead: 0.025 }, 'worse.wav': 'long', 'odd.wav': { seconds: 2, lead: 3 },
+ } });
+ expect(config.encoded).toEqual({ 'a.wav': { seconds: 2, lead: 0.025 } });
+ const { backend } = backendOf();
+ const audio = new AudioAPI(backend);
+ const set = vi.spyOn(audio, 'setEncodedClips');
+ applyAudioProjectConfig(audio, config);
+ expect(set).toHaveBeenCalledWith({ 'a.wav': { seconds: 2, lead: 0.025 } });
+ // A project played from its own files has none, and nothing is asked of the API.
+ set.mockClear();
+ applyAudioProjectConfig(audio, parseAudioProjectConfig({}));
+ expect(set).not.toHaveBeenCalled();
+ });
+
+ it('lies between its lead and its real end in what a decoder made, unless the decoder took the lead off', () => {
+ const trim = { seconds: 2, lead: 0.025 };
+ // The decoder left the silence in: 2.06 s came back for a clip of 2.
+ expect(clipSpan(trim, 2.06)).toEqual({ lead: 0.025, end: 2.025 });
+ // It took the lead off and left what follows the sound: the clip begins at nothing and is over at 2.
+ expect(clipSpan(trim, 2.01)).toEqual({ lead: 0, end: 2 });
+ // It took both off.
+ expect(clipSpan(trim, 2)).toEqual({ lead: 0, end: 2 });
+ // A clip cut short by the decoder is not said to be longer than what there is.
+ expect(clipSpan(trim, 1.5)).toEqual({ lead: 0, end: 1.5 });
+ expect(clipSpan(undefined, 2.06)).toBeNull();
+ });
+});
diff --git a/sdk/tests/minigame-audio-backend.test.ts b/sdk/tests/minigame-audio-backend.test.ts
index cc1bad98e..0b439baa5 100644
--- a/sdk/tests/minigame-audio-backend.test.ts
+++ b/sdk/tests/minigame-audio-backend.test.ts
@@ -86,6 +86,25 @@ describe('MiniGameAudioBackend', () => {
expect(handle.isPlaying).toBe(false); expect(handle.currentTime).toBe(0);
expect(ctx.destroy).toHaveBeenCalledOnce();
});
+ it('starts a clip the exporter re-encoded past the silence before it, the first time and every time round', async () => {
+ const trim = { seconds: 2, lead: 0.025 };
+ const buffer = await backend.loadBuffer('theme.mp3', { trim });
+ expect(buffer.duration).toBe(2);
+ backend.play(buffer, {});
+ let ctx = (globalThis as any).wx.createInnerAudioContext.mock.results.at(-1).value;
+ expect([ctx.startTime, ctx.loop]).toEqual([0.025, false]);
+ backend.play(buffer, { loop: true });
+ ctx = (globalThis as any).wx.createInnerAudioContext.mock.results.at(-1).value;
+ // Left to loop by itself the host would play the silence again each time.
+ expect(ctx.loop).toBe(false);
+ ctx.startTime = 9; ctx.onEnded.mock.calls[0][0]();
+ expect(ctx.startTime).toBe(0.025);
+ // A clip that was not re-encoded is left to the host as before.
+ const plain = await backend.loadBuffer('plain.mp3');
+ backend.play(plain, { loop: true });
+ ctx = (globalThis as any).wx.createInnerAudioContext.mock.results.at(-1).value;
+ expect([ctx.startTime, ctx.loop]).toEqual([0, true]);
+ });
it('a loop with a loop start is sent back there when it ends, and is not left to the host to loop', async () => {
const buffer = await backend.loadBuffer('a.mp3'), handle = backend.play(buffer, { loop: true, loopStart: 12 });
const ctx = (globalThis as any).wx.createInnerAudioContext.mock.results.at(-1).value;
diff --git a/sdk/tests/native-audio.test.ts b/sdk/tests/native-audio.test.ts
index 564ade175..75f46449d 100644
--- a/sdk/tests/native-audio.test.ts
+++ b/sdk/tests/native-audio.test.ts
@@ -35,11 +35,11 @@ function installAudioHost(scope: Record<string, unknown>) {
scope.es_audioLoad = (bytes: ArrayBuffer) => ({ id: nextBuffer++, duration: 2, bytes: bytes.byteLength * 4 });
scope.es_audioUnload = () => {};
- scope.es_audioPlay = (buf: number, vol: number, pan: number, loop: boolean, rate: number, loopStart?: number) => {
+ scope.es_audioPlay = (buf: number, vol: number, pan: number, loop: boolean, rate: number, loopStart?: number, start?: number, end?: number) => {
const id = nextVoice++;
voices.set(id, { playing: true, time: 0, loop });
- // The sixth is where a loop comes back to; it is recorded only when it is somewhere.
- calls.play.push(loopStart ? [buf, vol, pan, loop, rate, loopStart] : [buf, vol, pan, loop, rate]);
+ // The sixth is where a loop comes back to, the seventh and eighth where the clip begins and ends; recorded only when they are somewhere.
+ calls.play.push(start || end ? [buf, vol, pan, loop, rate, loopStart, start, end] : loopStart ? [buf, vol, pan, loop, rate, loopStart] : [buf, vol, pan, loop, rate]);
return id;
};
scope.es_audioStop = (v: number) => { voices.delete(v); calls.stop.push(v); };
@@ -149,6 +149,28 @@ describe('NativeAudioBackend', () => {
expect(host.calls.play).toEqual([[buffer.id, 1, 0, true, 1, 8], [buffer.id, 1, 0, false, 1], [buffer.id, 1, 0, true, 1]]);
});
+ it('plays a clip the exporter re-encoded from past its silence to its real end', async () => {
+ const scope = audioScope();
+ const host = installAudioHost(scope);
+ const backend = new NativeAudioBackend(createHostBridge(scope).audio!);
+ // The fake host says every clip decodes to two seconds: 1.95 of sound, with 0.025 before it and as much after.
+ const buffer = await backend.loadBufferFromData('music', new ArrayBuffer(16), { trim: { seconds: 1.95, lead: 0.025 } });
+ expect(buffer.duration).toBeCloseTo(1.95, 9);
+ backend.play(buffer, {});
+ backend.play(buffer, { loop: true });
+ backend.play(buffer, { loop: true, loopStart: 0.5 });
+ const close = (v: number) => expect.closeTo(v, 9);
+ expect(host.calls.play).toEqual([
+ [buffer.id, 1, 0, false, 1, close(0.025), close(0.025), close(1.975)],
+ [buffer.id, 1, 0, true, 1, close(0.025), close(0.025), close(1.975)],
+ [buffer.id, 1, 0, true, 1, close(0.525), close(0.025), close(1.975)],
+ ]);
+ // A host that took the silence off itself says the clip is as long as it should be, and is asked for nothing.
+ const exact = await backend.loadBufferFromData('music2', new ArrayBuffer(16), { trim: { seconds: 2, lead: 0.025 } });
+ backend.play(exact, { loop: true });
+ expect(host.calls.play.at(-1)).toEqual([exact.id, 1, 0, true, 1, 0, 0, close(2)]);
+ });
+
it('loads, plays and reports state through the bridge', async () => {
const scope = audioScope();
const host = installAudioHost(scope); // capture the same closure the bridge will use
diff --git a/sdk/tests/web-audio-backend.test.ts b/sdk/tests/web-audio-backend.test.ts
index 132d703bc..0c38a496e 100644
--- a/sdk/tests/web-audio-backend.test.ts
+++ b/sdk/tests/web-audio-backend.test.ts
@@ -490,6 +490,64 @@ describe('WebAudioBackend', () => {
expect(handle.currentTime).toBeCloseTo(.875);
});
+ describe('a clip the exporter re-encoded', () => {
+ // Two seconds of sound that came back from the decoder with a fortieth of a second of silence before it and some after.
+ const trim = { seconds: 2, lead: 0.025 };
+ const loaded = async (decoded: number) => {
+ await backend.initialize();
+ const mockCtx = (globalThis as any).__mockAudioContext;
+ mockCtx.decodeAudioData.mockResolvedValue({ duration: decoded });
+ return { mockCtx, handle: await backend.loadBuffer('theme.mp3', { trim }) };
+ };
+
+ it('is as long as it was made, not as long as the decoder left it', async () => {
+ expect((await loaded(2.06)).handle.duration).toBeCloseTo(2, 9);
+ });
+
+ it('is begun past the silence before it, and stopped where it ends', async () => {
+ const { mockCtx, handle } = await loaded(2.06);
+ const voice = backend.play(handle, {});
+ const source = mockCtx.createBufferSource.mock.results.at(-1).value;
+ expect(source.start).toHaveBeenCalledWith(0, expect.closeTo(0.025, 9), expect.closeTo(2, 9));
+ expect(source.loopStart).toBeUndefined();
+ // Time is counted from where the clip begins.
+ mockCtx.currentTime = 0.5;
+ expect(voice.currentTime).toBeCloseTo(0.5, 9);
+ expect(voice.duration).toBeCloseTo(2, 9);
+ backend.play(handle, { startOffset: 0.5 });
+ expect(mockCtx.createBufferSource.mock.results.at(-1).value.start).toHaveBeenCalledWith(0, expect.closeTo(0.525, 9), expect.closeTo(1.5, 9));
+ });
+
+ it('goes round between its real start and its real end, so a loop made seamless stays so', async () => {
+ const { mockCtx, handle } = await loaded(2.06);
+ const voice = backend.play(handle, { loop: true });
+ const source = mockCtx.createBufferSource.mock.results.at(-1).value;
+ expect([source.loop, source.loopStart, source.loopEnd]).toEqual([true, expect.closeTo(0.025, 9), expect.closeTo(2.025, 9)]);
+ expect(source.start).toHaveBeenCalledWith(0, expect.closeTo(0.025, 9));
+ // Five seconds in it has been round twice and is one second into the third time.
+ mockCtx.currentTime = 5; voice.pause();
+ expect(voice.currentTime).toBeCloseTo(1, 6);
+ voice.resume();
+ const next = mockCtx.createBufferSource.mock.results.at(-1).value;
+ expect(next.start).toHaveBeenCalledWith(0, expect.closeTo(1.025, 6));
+ expect([next.loopStart, next.loopEnd]).toEqual([expect.closeTo(0.025, 9), expect.closeTo(2.025, 9)]);
+ // With an introduction, the loop start is counted from the clip's own beginning.
+ backend.play(handle, { loop: true, loopStart: 0.8 });
+ const led = mockCtx.createBufferSource.mock.results.at(-1).value;
+ expect([led.loopStart, led.loopEnd]).toEqual([expect.closeTo(0.825, 9), expect.closeTo(2.025, 9)]);
+ });
+
+ it('is not trimmed twice where the decoder took the silence off itself', async () => {
+ const { mockCtx, handle } = await loaded(2.0);
+ backend.play(handle, { loop: true });
+ const source = mockCtx.createBufferSource.mock.results.at(-1).value;
+ expect(source.start).toHaveBeenCalledWith(0, 0);
+ // The whole buffer is the clip: there is no part of it to go round.
+ expect(source.loopStart).toBeUndefined();
+ expect(handle.duration).toBeCloseTo(2, 9);
+ });
+ });
+
it('a loop with a loop start goes round from there, and a pause knows where in the round it is', async () => {
const { bufferHandle, mockCtx } = await initAndLoadBuffer(backend);
const handle = backend.play(bufferHandle, { loop: true, loopStart: .4 });
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index b2222f4d1..50c30a9e5 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -4652,6 +4652,10 @@ export const NOTED = {
{"note": "An AI can make background music."},
"feat(music): the tool text says the five-note scales and how a tune in one sits over a piece in seven":
{"note": "An AI can make background music."},
+ "fix(audio): a loop is seamless in a packaged game, and an effect is on time - an exported .wav became an MP3 that decodes with 1105 samples of silence before it and more after, so every loop had a gap each time round and every sound began a fortieth of a second late; the exporter now records how long each clip really is and the web, mini-game and native backends play it from past the silence to its real end":
+ {"note": "A loop is seamless in a packaged game."},
+ "feat(music): a piece that goes round is set to be decoded when a game loads it - played from its compressed file it could not loop between two samples, and the reply says what it costs in memory":
+ {"note": "An AI can make background music."},
"feat(sky): a sky of your own shades the ground under its clouds - defineSkyEffect takes groundShadow and a second function, skyShadow(world, sun), what is left of the light at a world position, and the sky pass multiplies every surface it finds by it, as Sky Atmosphere does with its own clouds":
{"note": "A sky of your own shades the ground under its clouds."},
"fix(terrain): a colour map and an occlusion map lie the way the heightmap does - Terrain.colorMap, Terrain.occlusionMap and Foliage.colorMap were sampled with their rows turned over, so a picture painted to match a heightmap, a splatmap or a density map came out mirrored north to south under them; an image over an area now has its first row along the least z whoever reads it":