Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(export): a web export packs its small local assets into a few files

esengine1144bad41e8f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4a1df5f40..d12f8d31d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,7 @@ published separately; it ships inside the editor.
- Exporting again is fast when the art has not changed: the cook keeps what it encoded under `.esengine/cache/cook/`, keyed by the source, the settings and the encoder. Re-exporting Bistro took 24 s against 131 s the first time.
- A scene's preload logs what each asset type cost it — how many, their summed load time and the slowest one — and `preloadSceneAssets` returns the same as `timings` and `wallMs`. Bistro on a Windows machine: 132 materials 54 s, 107 textures 47 s and 1505 meshes 45 s of load time over 24 s of wall time.
- KTX2 textures on the web and in the editor transcode on worker threads instead of the main thread, one to four of them by the machine's cores. Bistro's assets loaded in 14 s instead of 24 s on a Windows machine, and the page stays responsive while they do. A realm with no workers (a mini-game host) transcodes on the main thread as before, and so does one whose workers fail, which says so once.
+- Web exports pack their small local assets — meshes, materials, KTX2 textures, timelines and animation assets — into a few `packs/*.pack` files, one request each, and the runtime reads each asset out of its pack. A browser holds six connections to a site, so a scene of thousands of small files loaded at the pace of its requests: Bistro's 1505 meshes took 43 s of load time for 0.2 s of upload. Its first frame on a Windows machine went from 15.2 s to 7.3 s, pixel for pixel the same. An export with hot update configured keeps one file per asset, since an update downloads assets one by one; an update that meets a packed asset refuses it by name.
- **Export profiles.** A project can name several exports of one target in `packaging.profiles` — a test and a production WeChat build, each with its own AppID, CDN root and development or shipping configuration, writing only what differs from the project's settings. The Build dialog creates, edits and deletes them per target, marking each setting a profile overrides with a reset that hands it back to the project; `estella export --profile <name>` and the agent's `export_game` build the same one, and the size history records which profile a build used.
### Changed
diff --git a/docs/astro/src/content/docs/publishing/overview.mdx b/docs/astro/src/content/docs/publishing/overview.mdx
index 21f10f7f1..76096db0e 100644
--- a/docs/astro/src/content/docs/publishing/overview.mdx
+++ b/docs/astro/src/content/docs/publishing/overview.mdx
@@ -274,6 +274,14 @@ of its bytes — immutable URLs, automatic dedupe). **Advanced** holds the CDN r
for [hot update](/docs/publishing/hot-update/), source maps, and whether to open the
output folder when the build finishes.
+**Asset packs** (Web): a browser holds six connections to a site, so a scene of
+thousands of small assets loads at the pace of its requests. The Web export packs
+its small local assets — meshes, materials, KTX2 textures, timelines, animation
+assets — into a few `packs/*.pack` files that load in one request each; the manifest
+says where every asset sits. A scene of 1,500 meshes went from 15 s to 7 s to its
+first frame. An export with [hot update](/docs/publishing/hot-update/) configured
+keeps one file per asset, since an update downloads assets one by one.
+
<Aside type="caution" title="Assets only your code names are culled">
The cook ships what it can **reach** from the scenes in the build. Anything named
only in code — a texture in rich-text markup, a clip played by url, a prefab
diff --git a/docs/astro/src/content/docs/zh-cn/publishing/overview.mdx b/docs/astro/src/content/docs/zh-cn/publishing/overview.mdx
index bedbf213c..6c31b28f5 100644
--- a/docs/astro/src/content/docs/zh-cn/publishing/overview.mdx
+++ b/docs/astro/src/content/docs/zh-cn/publishing/overview.mdx
@@ -228,6 +228,8 @@ which a block-compressed texture must be
**高级**里放着[热更新](/docs/zh-cn/publishing/hot-update/)的 CDN 根地址、source map,
以及构建结束后是否打开输出目录。
+**资产打包**(Web):浏览器对同一个站点只开 6 条连接,所以一个有上千个小资产的场景,加载速度取决于请求数。Web 导出会把小的本地资产——网格、材质、KTX2 纹理、时间线、动画资产——打进少数几个 `packs/*.pack` 文件,每个文件一次请求取完;清单里记着每个资产在哪个包、哪个位置。一个有 1500 个网格的场景,首帧从 15 秒降到 7 秒。配置了[热更新](/docs/zh-cn/publishing/hot-update/)的导出仍然每个资产一个文件,因为热更新是按单个资产下载的。
+
<Aside type="caution" title="只有代码点名的资产会被剔除">
cook 发布的是它能从构建场景**可达**的东西。只在代码里点名的东西——富文本标记里的纹理、
按 url 播放的音频、按路径 spawn 的预制体——会被剔除,而编辑器直接从磁盘伺服项目,
diff --git a/pipeline/src/assets/addressableManifest.ts b/pipeline/src/assets/addressableManifest.ts
index 5977da968..0e38ee78d 100644
--- a/pipeline/src/assets/addressableManifest.ts
+++ b/pipeline/src/assets/addressableManifest.ts
@@ -29,6 +29,7 @@ interface FlatManifest {
compressedFormats?: string[];
importer?: Record<string, unknown>;
atlas?: { page: number; frame: { x: number; y: number; width: number; height: number }; pageWidth: number; pageHeight: number };
+ pack?: { file: string; offset: number; size: number };
}[];
}
@@ -62,6 +63,7 @@ export async function buildAddressableManifest(absOut: string): Promise<string>
compressedFormats?: string[];
textureImport?: ParsedTextureImportSettings;
spineImport?: SpineManifestContract;
+ pack?: { file: string; offset: number; size: number };
metadata?: { atlasPage?: number; atlasFrame?: { x: number; y: number; width: number; height: number }; atlasPageWidth?: number; atlasPageHeight?: number };
};
type Group = { bundleMode: string; labels: string[]; assets: Record<string, Entry> };
@@ -104,6 +106,7 @@ export async function buildAddressableManifest(absOut: string): Promise<string>
// The logical source path rides as the asset's address: path-style refs
// resolve through it. Only meaningful when staging renamed the file.
if (e.sourcePath && e.sourcePath !== e.path) entry.address = e.sourcePath;
+ if (e.pack) entry.pack = e.pack;
if (e.atlas) {
entry.metadata = {
atlasPage: e.atlas.page, atlasFrame: e.atlas.frame,
diff --git a/pipeline/src/assets/assetPacks.ts b/pipeline/src/assets/assetPacks.ts
new file mode 100644
index 000000000..5c456ccf0
--- /dev/null
+++ b/pipeline/src/assets/assetPacks.ts
@@ -0,0 +1,98 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file assetPacks.ts — a network build's small assets, packed into a few files.
+ *
+ * A browser holds six connections to an origin, so a scene of a few thousand
+ * assets loads at the pace of its requests: Bistro spent 43 s of load time
+ * fetching 1505 meshes whose upload took 0.2 s. After the cook, this moves the
+ * local assets that every read reaches through the asset server into packs, and
+ * records in the cook manifest where each one sits (the runtime reads them back
+ * through `packedBackend`).
+ */
+import { createHash } from 'node:crypto';
+import { existsSync } from 'node:fs';
+import { mkdir, readFile, rm, writeFile } from 'node:fs/promises';
+import path from 'node:path';
+import { bundleModeFor } from './addressableManifest';
+
+/** The types read only through the asset server's backend, which is what reads a pack.
+ * Not prefabs: cutting a streamed world reads them off disk after the cook. */
+const PACKABLE_TYPES = new Set([
+ 'mesh', 'material', 'timeline', 'anim-clip', 'animatorcontroller', 'avatar',
+]);
+
+export interface PackOptions {
+ /** An asset bigger than this keeps its own file: packing it saves one request among thousands. */
+ maxAssetBytes?: number;
+ /** A pack is fetched whole, so none grows past this. */
+ maxPackBytes?: number;
+}
+
+export interface PackReport { packs: number; assets: number; bytes: number }
+
+interface CookEntry {
+ path: string; type: string; group?: string; groupMode?: string; size?: number;
+ atlas?: unknown;
+ pack?: { file: string; offset: number; size: number };
+}
+
+function packable(e: CookEntry): boolean {
+ if (bundleModeFor(e.group ?? 'main', e.groupMode) !== 'local') return false;
+ if (e.atlas) return false;
+ if (e.type === 'texture') return e.path.toLowerCase().endsWith('.ktx2');
+ return PACKABLE_TYPES.has(e.type);
+}
+
+/** Pack a cook output dir's local assets; rewrites `assets.manifest.json` in place. */
+export async function packAssets(absOut: string, opts: PackOptions = {}): Promise<PackReport> {
+ const maxAsset = opts.maxAssetBytes ?? 4 * 1024 * 1024;
+ const maxPack = opts.maxPackBytes ?? 16 * 1024 * 1024;
+ const manifestPath = path.join(absOut, 'assets.manifest.json');
+ const flat = JSON.parse(await readFile(manifestPath, 'utf8')) as { entries: CookEntry[] };
+
+ // One file, one member: entries that share a path (content-deduplicated) share its slot.
+ const byPath = new Map<string, CookEntry[]>();
+ for (const e of flat.entries) {
+ if (!packable(e) || !existsSync(path.join(absOut, e.path))) continue;
+ const list = byPath.get(e.path) ?? [];
+ list.push(e);
+ byPath.set(e.path, list);
+ }
+ // A group's assets pack together, so a scene tends to read whole packs.
+ const paths = [...byPath.keys()].sort((a, b) => {
+ const ga = byPath.get(a)![0].group ?? 'main', gb = byPath.get(b)![0].group ?? 'main';
+ return ga === gb ? a.localeCompare(b) : ga.localeCompare(gb);
+ });
+
+ const report: PackReport = { packs: 0, assets: 0, bytes: 0 };
+ let parts: Uint8Array[] = [];
+ let members: Array<{ path: string; offset: number; size: number }> = [];
+ let length = 0;
+ const flush = async (): Promise<void> => {
+ if (members.length < 2) { parts = []; members = []; length = 0; return; }
+ const bytes = Buffer.concat(parts);
+ const file = `packs/${createHash('sha256').update(bytes).digest('hex').slice(0, 16)}.pack`;
+ await mkdir(path.join(absOut, 'packs'), { recursive: true });
+ await writeFile(path.join(absOut, file), bytes);
+ for (const m of members) {
+ for (const e of byPath.get(m.path)!) e.pack = { file, offset: m.offset, size: m.size };
+ await rm(path.join(absOut, m.path), { force: true });
+ }
+ report.packs++;
+ report.assets += members.length;
+ report.bytes += bytes.byteLength;
+ parts = []; members = []; length = 0;
+ };
+ for (const p of paths) {
+ const data = await readFile(path.join(absOut, p));
+ if (data.byteLength > maxAsset) continue;
+ if (length + data.byteLength > maxPack) await flush();
+ members.push({ path: p, offset: length, size: data.byteLength });
+ parts.push(data);
+ length += data.byteLength;
+ }
+ await flush();
+ await writeFile(manifestPath, JSON.stringify(flat, null, 2));
+ return report;
+}
diff --git a/pipeline/src/export/exportGame.ts b/pipeline/src/export/exportGame.ts
index d1939dfae..57386abbc 100644
--- a/pipeline/src/export/exportGame.ts
+++ b/pipeline/src/export/exportGame.ts
@@ -30,6 +30,7 @@ import path from 'node:path';
import { cookAssets, type CookManifest, type Inclusion } from '../assets/cookAssets';
import { cookWorlds, streamedScenes } from '../world/cookWorld';
import { buildAddressableManifest } from '../assets/addressableManifest';
+import { packAssets } from '../assets/assetPacks';
import type { DebugChannelConfig, PackagedGameConfig } from 'esengine';
import { packagedDebugChannel } from './debugChannel';
import { packagedHotUpdate } from './hotUpdateConfig';
@@ -559,6 +560,8 @@ export interface ExportGameOptions {
profile?: string;
/** WeChat: ship the EmscriptenGLX engine (`packaging.platforms.wechat.emscriptenGLX`). */
miniGameGlx?: boolean;
+ /** Web: pack small local assets into a few files (default on; see assetPacks). */
+ packAssets?: boolean;
/** Signing key files the project names, checked against its repository. */
secretFiles?: readonly string[];
/** iOS: where the prebuilt engine + app shell live, so the export can wrap
@@ -943,6 +946,15 @@ async function produceExport(opts: ExportGameOptions): Promise<ExportGameResult>
// model every target now shares, so `loadGroup` / remote-group / hot-update
// work on web + desktop too (not just mini-games). Additive: the eager boot
// still reads the flat manifest; this powers on-demand + hot-update delivery.
+ const hotUpdate = await packagedHotUpdate(opts.root, opts.hotUpdate);
+ // A web build pays a request per asset. An updatable one keeps its files: an
+ // update downloads assets one by one (Assets.applyUpdate refuses packed ones).
+ if (platform === 'web' && !hotUpdate && opts.packAssets !== false) {
+ const packed = await packAssets(payloadDir);
+ if (packed.assets > 0) {
+ progress({ phase: 'Packing assets', detail: `${packed.assets} in ${packed.packs} pack(s)` });
+ }
+ }
await writeFile(path.join(payloadDir, 'asset-manifest.json'), await buildAddressableManifest(payloadDir));
// A scene that declared itself streamed is cut here, once its assets are staged
// and its references are in the form the runtime resolves. The entry scene it
@@ -955,8 +967,6 @@ async function produceExport(opts: ExportGameOptions): Promise<ExportGameResult>
// asset model in the package — the mini-game export has always done this.
await rm(path.join(payloadDir, 'assets.manifest.json'), { force: true });
- const hotUpdate = await packagedHotUpdate(opts.root, opts.hotUpdate);
-
// What wasm this project's content pulls in — evidence for the plan below, and
// the filter the runtime tree is copied through further down.
const sideModuleIds = !nativeContent && existsSync(opts.wasmDir)
diff --git a/pipeline/tests/asset-packs.test.ts b/pipeline/tests/asset-packs.test.ts
new file mode 100644
index 000000000..0eba03b73
--- /dev/null
+++ b/pipeline/tests/asset-packs.test.ts
@@ -0,0 +1,72 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file A cook output's small local assets end up in packs the manifest can find.
+ */
+import { describe, it, expect, afterAll } from 'vitest';
+import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import path from 'node:path';
+import { packAssets } from '../src/assets/assetPacks';
+
+const work = mkdtempSync(path.join(tmpdir(), 'asset-packs-'));
+afterAll(() => rmSync(work, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 }));
+
+type Entry = { uuid: string; path: string; type: string; group?: string; groupMode?: string; size?: number; atlas?: unknown; pack?: { file: string; offset: number; size: number } };
+
+function cookDir(files: Array<{ path: string; type: string; bytes: number; group?: string; groupMode?: string; atlas?: boolean; dup?: string }>): string {
+ const dir = mkdtempSync(path.join(work, 'out-'));
+ const entries: Entry[] = [];
+ files.forEach((f, i) => {
+ mkdirSync(path.dirname(path.join(dir, f.path)), { recursive: true });
+ writeFileSync(path.join(dir, f.path), Buffer.alloc(f.bytes, i + 1));
+ entries.push({ uuid: `u${i}`, path: f.path, type: f.type, group: f.group, groupMode: f.groupMode, size: f.bytes, ...(f.atlas ? { atlas: {} } : {}) });
+ if (f.dup) entries.push({ uuid: f.dup, path: f.path, type: f.type, size: f.bytes });
+ });
+ writeFileSync(path.join(dir, 'assets.manifest.json'), JSON.stringify({ entries }));
+ return dir;
+}
+
+const entriesOf = (dir: string) => (JSON.parse(readFileSync(path.join(dir, 'assets.manifest.json'), 'utf8')) as { entries: Entry[] }).entries;
+
+describe('packing a cook output', () => {
+ it('packs what the asset server reads, and every packed byte reads back where the manifest says', async () => {
+ const dir = cookDir([
+ { path: 'assets/a.esmesh', type: 'mesh', bytes: 100, dup: 'twin' },
+ { path: 'assets/b.esmaterial', type: 'material', bytes: 50 },
+ { path: 'assets/c.ktx2', type: 'texture', bytes: 300 },
+ { path: 'assets/d.png', type: 'texture', bytes: 70 },
+ { path: 'assets/e.ktx2', type: 'texture', bytes: 80, atlas: true },
+ { path: 'assets/f.esmesh', type: 'mesh', bytes: 40, group: 'far', groupMode: 'remote' },
+ { path: 'assets/g.mp3', type: 'audio', bytes: 60 },
+ { path: 'assets/h.esmesh', type: 'mesh', bytes: 5000 },
+ ]);
+ const report = await packAssets(dir, { maxAssetBytes: 1000 });
+ expect(report).toMatchObject({ packs: 1, assets: 3 });
+
+ const entries = entriesOf(dir);
+ const packed = entries.filter((e) => e.pack);
+ expect(packed.map((e) => e.path).sort()).toEqual(['assets/a.esmesh', 'assets/a.esmesh', 'assets/b.esmaterial', 'assets/c.ktx2']);
+ for (const e of packed) {
+ expect(existsSync(path.join(dir, e.path))).toBe(false);
+ const pack = readFileSync(path.join(dir, e.pack!.file));
+ const fill = entries.filter((x) => x.uuid.startsWith('u')).findIndex((x) => x.path === e.path) + 1;
+ expect([...pack.subarray(e.pack!.offset, e.pack!.offset + e.pack!.size)].every((b) => b === fill)).toBe(true);
+ }
+ for (const kept of ['assets/d.png', 'assets/e.ktx2', 'assets/f.esmesh', 'assets/g.mp3', 'assets/h.esmesh']) {
+ expect(existsSync(path.join(dir, kept)), kept).toBe(true);
+ expect(entries.find((e) => e.path === kept)?.pack, kept).toBeUndefined();
+ }
+ });
+
+ it('splits packs at the size limit and leaves a lone asset its own file', async () => {
+ const dir = cookDir([
+ { path: 'assets/1.esmesh', type: 'mesh', bytes: 400 },
+ { path: 'assets/2.esmesh', type: 'mesh', bytes: 400 },
+ { path: 'assets/3.esmesh', type: 'mesh', bytes: 400 },
+ ]);
+ const report = await packAssets(dir, { maxPackBytes: 900 });
+ expect(report).toMatchObject({ packs: 1, assets: 2 });
+ expect(existsSync(path.join(dir, 'assets/3.esmesh'))).toBe(true);
+ });
+});
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index cae7fdee7..4ff223387 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -1224,4 +1224,10 @@ export const NOTED = {
{ note: 'KTX2 textures on the web and in the editor transcode on worker threads instead of the main thread, one to four of them by the machine\'s cores. Bistro\'s assets loaded in 14 s instead of 24 s on a Windows machine, and the page stays responsive while they do. A realm with no workers (a mini-game host) transcodes on the main thread as before, and so does one whose workers fail, which says so once.' },
'fix(asset): a preload times itself without a platform':
{ internal: 'The per-type preload timings landed one commit earlier and never shipped; they read the platform clock, which a realm without a platform (the SDK tests) does not have.' },
+ 'feat(asset): the runtime reads an asset out of the pack its export put it in':
+ { note: 'Web exports pack their small local assets — meshes, materials, KTX2 textures, timelines and animation assets — into a few `packs/*.pack` files, one request each, and the runtime reads each asset out of its pack. A browser holds six connections to a site, so a scene of thousands of small files loaded at the pace of its requests: Bistro\'s 1505 meshes took 43 s of load time for 0.2 s of upload. Its first frame on a Windows machine went from 15.2 s to 7.3 s, pixel for pixel the same. An export with hot update configured keeps one file per asset, since an update downloads assets one by one; an update that meets a packed asset refuses it by name.' },
+ 'feat(export): a web export packs its small local assets into a few files':
+ { note: 'Web exports pack their small local assets — meshes, materials, KTX2 textures, timelines and animation assets — into a few `packs/*.pack` files, one request each, and the runtime reads each asset out of its pack. A browser holds six connections to a site, so a scene of thousands of small files loaded at the pace of its requests: Bistro\'s 1505 meshes took 43 s of load time for 0.2 s of upload. Its first frame on a Windows machine went from 15.2 s to 7.3 s, pixel for pixel the same. An export with hot update configured keeps one file per asset, since an update downloads assets one by one; an update that meets a packed asset refuses it by name.' },
+ 'fix(eht): the generator writes LF on every platform':
+ { internal: 'The engine header tool is a repository generator; on Windows it rewrote every generated file with CRLF, which only a contributor there ever saw.' },
};