Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

test(scale): seventeen costs, each with a ceiling it may not pass

esengine21485a826c08
diff --git a/bench/scale/README.md b/bench/scale/README.md
new file mode 100644
index 000000000..2dc7a0348
--- /dev/null
+++ b/bench/scale/README.md
@@ -0,0 +1,109 @@
+# The scale corpus and its budgets
+
+Every scale problem this engine has had was reported by someone whose project was
+big enough to notice. 36,000 assets where selecting 22 files and pressing Delete
+took 52 seconds to open the confirm dialog. A folder of 788 sprites that decoded
+26 megapixels to show the twenty on screen. Nothing in this repository was ever
+that big — the examples exist to teach, so they are small, and a cost that only
+appears at 50,000 assets is invisible here until a user meets it.
+
+So there is now a project that big, and a set of ceilings it has to stay under.
+
+```sh
+node tools/perf-budget.mjs # make the corpus if needed, measure, gate
+node tools/perf-budget.mjs --table # the last run's table, without re-running
+pnpm run scale # the same thing
+```
+
+## The corpus
+
+`tools/stress-project.mjs` generates it into `build/stress-project/` — an ordinary
+Estella project you can open in the editor and press Play on.
+
+| | |
+| --- | --- |
+| assets | **50,000** (100,004 files with their `.meta` sidecars, ~42MB) |
+| textures | 31,496 — including 2,000 in **one folder**, the shape that broke the Content Browser |
+| materials / prefabs / audio / data | 6,000 / 4,000 / 3,000 / 3,000 |
+| shaders / tilesets / spine sets / fonts | 700 / 200 / 500 / 60 |
+| scenes | 43, of which 7 carry the weight |
+| `sprites.esscene` | 10,000 sprites over 8 sorting layers, 4 cameras, 200 parents |
+| `ui.esscene` | 5,000 UI nodes, six deep, masks nested inside masks |
+| `physics.esscene` | 2,000 bodies with colliders |
+| `spine.esscene` | 500 skeletons |
+| `prefabs.esscene` | 3,000 instances, most of them overriding something |
+| `tilemap.esscene` | a 256×256×4 Tiled map |
+| `everything.esscene` | all of the above in one document — 20,707 entries, 7.3MB |
+
+It is **generated, not committed**. 100,000 files would sit in front of every
+clone, every `git status` and every editor indexing the tree, to store bytes that
+are a pure function of one script. The generator is the corpus: no `Math.random`,
+no `Date`, every uuid is `sha1(seed + path)`, so the same spec produces the same
+tree byte for byte on any machine. It takes about eight seconds to write, and a
+stamp file means it is only written when the generator or the scale changes.
+
+`--scale 0.02` makes a 1/50th corpus for a quick local experiment. Budgets do not
+apply to it and the gate refuses to run against one.
+
+## The budgets
+
+Declared next to what they measure, in `desktop/tests/scale/*.scale.ts`, each with
+the reason it exists. A metric that goes over fails, and prints that reason.
+
+**Why a budget rather than a snapshot.** `tools/perf-guard.mjs` asserts ratios
+between microbenchmarks against a recorded snapshot — the right shape for an
+architectural invariant, and the wrong one here, because a snapshot can always be
+accepted with `--update`. A feature that takes scene-open from 200ms to 900ms
+would pass by rewriting the number it is compared against. A ceiling cannot be
+raised by the change that breaks it: raising one is its own commit, with its own
+reason, and somebody has to decide to write it.
+
+**Why the numbers are not milliseconds.** They would not survive the CI runner,
+which has two shared cores. Each run measures three fixed reference workloads and
+every metric is divided by the one whose shape it shares:
+
+| unit | the reference | what it stands in for |
+| --- | --- | --- |
+| `parse` | `JSON.parse` + walk of a fixed 6,000-row document | scene load, serialize, index reads |
+| `loop` | a numeric pass over 2M `Float32` lanes | ECS iteration, transform math |
+| `io` | `readdir` over 20 shards, then read+parse 10,000 `.meta` | the asset-database walk |
+
+"Cold scan = 8.3 io" means the whole-project scan costs 8.3× what reading 10,000
+metas costs on whatever machine is asking. That reads the same on a laptop and on
+a runner. The raw milliseconds are recorded beside it, because that is the number
+a human wants; it is not the number the gate uses.
+
+The statistic is `min`, for the reason the perf guard already documents: noise on
+a shared runner is one-sided, so the fastest sample is the closest to what the
+code costs.
+
+The io reference was first written as "read 2,000 files from one folder". After
+the first pass that folder is entirely in the page cache, so it measured syscall
+and scheduler cost — a regime the 100,000-file scan is never in. On a busy machine
+it slowed by 41% while the scan it denominates slowed by 15%, and the ratio moved
+the wrong way. Twenty directories and 10,000 files behave like the thing they
+divide: run to run, the scan now reads within 3%.
+
+**How the ceilings were chosen.** Each is about twice what it measures today,
+which is the same bargain the perf guard makes: run-to-run noise reaches ~10% on
+the io-denominated metrics, so this cannot see a 20% slowdown and is not meant to.
+It is meant to catch the 2×-and-up kind — the shape a scale bug actually has. The
+memory budgets are tighter because bytes are exact, not sampled.
+
+Anything above 70% of its budget is reported as headroom going, which is the only
+warning before it stops being headroom.
+
+## What is not here yet
+
+Everything that needs a renderer: draw calls, GPU uploads, frame time with
+submission, editor cold start, project scan measured through the editor rather
+than the module it calls, Play startup, build duration. Those need the electron
+harness (`ESTELLA_SHOT`, the automation surface) rather than a vitest process, and
+they are a second tier of this same contract.
+
+## Changing a budget
+
+Change the number where it is declared, in its own commit, and update the `why`
+to say what the new ceiling means. If the corpus spec changes, every budget
+measured against it changes with it — that is a change to the contract, not a
+rebase.
diff --git a/desktop/tests/scale/assets.scale.ts b/desktop/tests/scale/assets.scale.ts
new file mode 100644
index 000000000..34827ec2a
--- /dev/null
+++ b/desktop/tests/scale/assets.scale.ts
@@ -0,0 +1,144 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file Asset-database budgets against the 50k-asset corpus.
+ *
+ * These are the costs a user meets before they have done anything: opening the
+ * project, saving a file, deleting a folder's worth of sprites. Every one of
+ * them was O(whole project) at some point in this repo's history, and the one
+ * that shipped was found by someone whose project was big enough to notice —
+ * 36k assets, 52 seconds to delete 22 files. The corpus is bigger than that.
+ */
+import { describe, it, beforeAll, expect } from 'vitest';
+import { readFile, writeFile, rm } from 'node:fs/promises';
+import path from 'node:path';
+
+import {
+ scanAssetDatabase,
+ updateAssetIndex,
+ readCachedAssetIndex,
+ type AssetIndex,
+} from '../../electron/assetDb';
+import { collectAssetUsagesOfAll } from '../../src/project/assetRefs';
+import { budgeted, calibrate, corpusDir } from './harness';
+
+const GROUP = 'Asset database — 50,000 assets';
+
+let root: string;
+let index: AssetIndex;
+
+beforeAll(async () => {
+ root = corpusDir();
+ await calibrate(root);
+}, 300_000);
+
+describe(GROUP, () => {
+ it('cold scan', async () => {
+ const res = await budgeted({
+ name: 'assetdb: cold scan of the whole project',
+ group: GROUP,
+ unit: 'io',
+ budget: 18,
+ why: 'Every project open pays this once, and nothing is on screen until it lands. '
+ + 'It walks the tree, reads every .meta and builds the dependency graph.',
+ runs: 3,
+ }, () => scanAssetDatabase(root, { write: true }));
+
+ index = res.index;
+ expect(res.ok).toBe(true);
+ // The corpus is exact by construction; a scan that finds a different
+ // number is measuring a tree nobody described.
+ expect(index.entries.length).toBe(50_000);
+ }, 600_000);
+
+ it('reads the cached index', async () => {
+ await budgeted({
+ name: 'assetdb: read the cached index',
+ group: GROUP,
+ unit: 'parse',
+ budget: 16,
+ why: 'The warm path a second window or a reopened project takes instead of scanning. '
+ + 'It is one JSON.parse of 50k entries plus the dep graph.',
+ runs: 5,
+ }, async () => {
+ const cached = await readCachedAssetIndex(root);
+ expect(cached?.entries.length).toBe(50_000);
+ return cached;
+ });
+ }, 300_000);
+
+ it('incremental scan after one file changes', async () => {
+ const changed = ['assets/textures/_bulk/tex-00001.png'];
+ await budgeted({
+ name: 'assetdb: incremental scan, one file touched',
+ group: GROUP,
+ unit: 'io',
+ budget: 0.25,
+ why: 'What the watcher fires on every save. This exists so a keystroke does not re-walk '
+ + 'the project — if it approaches the cold scan, the incremental path has stopped being one.',
+ runs: 5,
+ }, async () => {
+ const res = await updateAssetIndex(root, index, changed, { write: false });
+ expect(res.fullRescan).toBe(false);
+ return res;
+ });
+ }, 300_000);
+
+ it('usages of a 22-file selection', async () => {
+ const paths = Array.from({ length: 22 }, (_, i) => `assets/textures/_bulk/tex-${String(100 + i).padStart(5, '0')}.png`);
+ await budgeted({
+ name: 'assetdb: usages of a 22-file selection',
+ group: GROUP,
+ unit: 'parse',
+ budget: 9,
+ why: 'The question the delete-confirm dialog asks before it can open. Asked once per '
+ + 'selected asset over a freshly built map, 22 files took 52 seconds on a project '
+ + 'smaller than this one. It must invert the index once for the batch, not once per file.',
+ runs: 5,
+ }, () => collectAssetUsagesOfAll(index, paths));
+ }, 300_000);
+
+ it('usages of one asset', async () => {
+ await budgeted({
+ name: 'assetdb: usages of one asset',
+ group: GROUP,
+ unit: 'parse',
+ budget: 9,
+ why: 'Find Usages on a single sprite — the interactive case, where the dialog is waiting.',
+ runs: 5,
+ }, () => collectAssetUsagesOfAll(index, ['assets/textures/_bulk/tex-00100.png']));
+ }, 300_000);
+
+ it('incremental scan after 22 files are deleted', async () => {
+ // The reported case, restored afterwards so the corpus survives the run:
+ // the removal is real (the scan reads the disk, not a list) and the exact
+ // bytes go back.
+ const rels = Array.from({ length: 22 }, (_, i) => `assets/textures/_bulk/tex-${String(200 + i).padStart(5, '0')}.png`);
+ const saved: Array<[string, Buffer]> = [];
+ for (const rel of rels) {
+ for (const f of [rel, `${rel}.meta`]) {
+ saved.push([f, await readFile(path.join(root, f))]);
+ await rm(path.join(root, f));
+ }
+ }
+ try {
+ await budgeted({
+ name: 'assetdb: incremental scan, 22 files deleted',
+ group: GROUP,
+ unit: 'io',
+ budget: 2.6,
+ why: 'Deleting a folder\'s worth. A removal cannot make a dangling reference resolve, '
+ + 'so only the documents that referenced something removed need re-reading — '
+ + 'recomputing the whole dependency graph here is the regression this pins.',
+ runs: 3,
+ }, async () => {
+ const res = await updateAssetIndex(root, index, saved.map(([f]) => f), { write: false });
+ expect(res.fullRescan).toBe(false);
+ expect(res.index.entries.length).toBe(50_000 - rels.length);
+ return res;
+ });
+ } finally {
+ for (const [f, body] of saved) await writeFile(path.join(root, f), body);
+ }
+ }, 600_000);
+});
diff --git a/desktop/tests/scale/harness.ts b/desktop/tests/scale/harness.ts
new file mode 100644
index 000000000..1dd79d3c5
--- /dev/null
+++ b/desktop/tests/scale/harness.ts
@@ -0,0 +1,282 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file The scale harness: measure one cost against the stress corpus and hold
+ * it to a declared budget.
+ *
+ * WHY A BUDGET AND NOT A SNAPSHOT. The perf snapshot (`tools/perf-guard.mjs`)
+ * answers "did this change make it worse", which is the right question for an
+ * architectural invariant and the wrong one here. A snapshot can always be
+ * accepted with `--update`, so a feature that takes scene-open from 200ms to
+ * 900ms passes by rewriting the number it is compared against. A budget cannot
+ * be updated by the change that breaks it: it is a ceiling somebody chose, with
+ * a reason next to it, and going over is a decision rather than a diff.
+ *
+ * HOW A BUDGET SURVIVES A DIFFERENT MACHINE. Milliseconds do not — the CI runner
+ * has two shared cores and this laptop does not — so a budget is denominated in
+ * CALIBRATION UNITS. Each run measures a fixed reference workload and every
+ * metric is reported as its own cost divided by that reference. There are three
+ * references because there are three kinds of cost here, and one of them cannot
+ * stand in for another: `parse` (JSON + object churn), `loop` (tight numeric
+ * iteration) and `io` (reading small files off disk). A metric names the one that
+ * matches its shape, so "cold scan = 30 io" means "thirty times what reading a
+ * fixed pile of files costs on whatever machine this is".
+ *
+ * THE STATISTIC IS `min`, for the reason the perf guard already documents: noise
+ * on a shared runner is one-sided, so the fastest sample is the one closest to
+ * what the code costs.
+ *
+ * Raw milliseconds are recorded too. They are the number a human wants when
+ * reading the report; they are not the number the gate uses.
+ */
+import { expect } from 'vitest';
+import { readFile, stat, writeFile, mkdir } from 'node:fs/promises';
+import { readdirSync, existsSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath, URL } from 'node:url';
+
+const REPO = fileURLToPath(new URL('../../../', import.meta.url));
+export const REPORT_PATH = path.join(REPO, 'build', 'scale-report.json');
+
+/**
+ * A metric's denominator. `MB` and `count` have none: bytes and draw calls are
+ * already machine-independent, so their budgets are absolute.
+ */
+export type Unit = 'parse' | 'loop' | 'io' | 'MB' | 'count';
+
+export interface MetricSpec {
+ /** What the metric measures, as it appears in the report. */
+ name: string;
+ /** The heading it is filed under. */
+ group: string;
+ unit: Unit;
+ /** The ceiling, in `unit`. Going over fails the run. */
+ budget: number;
+ /** Why this ceiling and what breaking it would mean. Printed on failure. */
+ why: string;
+ /** Timed metrics only: how many samples to take the minimum of. */
+ runs?: number;
+ /**
+ * Untimed runs first. Needed wherever the first call does less work than the
+ * rest — resetting a world that is still empty despawns nothing, and a `min`
+ * over those samples would report the one run that skipped half the job.
+ */
+ warmup?: number;
+}
+
+export interface MetricRecord extends MetricSpec {
+ measured: number;
+ rawMs: number | null;
+ pctOfBudget: number;
+}
+
+const records: MetricRecord[] = [];
+
+// --- Calibration -----------------------------------------------------------
+// Each reference is fixed and shaped like the metrics that divide by it, and
+// measured once per process.
+
+const CALIBRATION_RUNS = 7;
+
+let units: Record<'parse' | 'loop' | 'io', number> | null = null;
+
+/** A fixed document to parse — built, never read from disk, so it is identical everywhere. */
+function calibrationDocument(): string {
+ const rows = [];
+ for (let i = 0; i < 6000; i++) {
+ rows.push({
+ id: i,
+ name: `entry-${i}`,
+ nested: { a: i * 1.5, b: [i, i + 1, i + 2], c: `${i}` },
+ tags: ['alpha', 'beta', 'gamma'],
+ });
+ }
+ return JSON.stringify({ version: 1, rows });
+}
+
+function walkSum(v: unknown): number {
+ if (typeof v === 'number') return v;
+ if (typeof v === 'string') return v.length;
+ if (Array.isArray(v)) {
+ let s = 0;
+ for (const x of v) s += walkSum(x);
+ return s;
+ }
+ if (v && typeof v === 'object') {
+ let s = 0;
+ for (const x of Object.values(v)) s += walkSum(x);
+ return s;
+ }
+ return 0;
+}
+
+function loopWorkload(): number {
+ const lanes = new Float32Array(1 << 21);
+ for (let i = 0; i < lanes.length; i++) lanes[i] = i * 0.5;
+ let acc = 0;
+ for (let i = 0; i < lanes.length; i++) acc += lanes[i] * 1.0001 + 1;
+ return acc;
+}
+
+/**
+ * The io reference walks part of the corpus the way the scan walks all of it:
+ * `readdir` over twenty directories, then read and parse every `.meta`. One
+ * folder instead stays in the page cache and measures syscall cost, a regime
+ * the 100,000-file scan is never in — that reference moved the ratio backwards.
+ */
+async function ioWorkload(corpusDir: string): Promise<number> {
+ const files = ioFiles(corpusDir);
+ let bytes = 0;
+ const LIMIT = 32;
+ for (let i = 0; i < files.length; i += LIMIT) {
+ const slice = files.slice(i, i + LIMIT);
+ const results = await Promise.all(slice.map(async (f) => {
+ const [s, body] = await Promise.all([stat(f), readFile(f, 'utf8')]);
+ return s.size + JSON.parse(body).uuid.length;
+ }));
+ for (const r of results) bytes += r;
+ }
+ return bytes;
+}
+
+const IO_REFERENCE_SHARDS = 20;
+const IO_REFERENCE_FILES = 10_000;
+let ioFileList: string[] | null = null;
+
+function ioFiles(corpusDir: string): string[] {
+ if (ioFileList) return ioFileList;
+ const textures = path.join(corpusDir, 'assets/textures');
+ const shards = readdirSync(textures).filter((d) => d.startsWith('shard-')).sort().slice(0, IO_REFERENCE_SHARDS);
+ const out: string[] = [];
+ for (const shard of shards) {
+ const dir = path.join(textures, shard);
+ for (const f of readdirSync(dir).sort()) if (f.endsWith('.meta')) out.push(path.join(dir, f));
+ }
+ if (out.length !== IO_REFERENCE_FILES) {
+ throw new Error(`io calibration expects ${IO_REFERENCE_FILES} files across ${IO_REFERENCE_SHARDS} `
+ + `shards of ${textures}, found ${out.length} — the corpus was generated at a reduced `
+ + '--scale, and budgets do not apply to it.');
+ }
+ ioFileList = out;
+ return ioFileList;
+}
+
+/**
+ * Measure the three references. Every suite calls this and only the first one
+ * does the work: calibrating again half way through a run would denominate the
+ * later metrics in a different machine-state than the earlier ones, and the
+ * report is one table.
+ */
+export async function calibrate(corpusDir: string): Promise<Record<string, number>> {
+ if (units) return units;
+ const doc = calibrationDocument();
+ const parse = minOf(CALIBRATION_RUNS, () => { walkSum(JSON.parse(doc)); });
+ const loop = minOf(CALIBRATION_RUNS, () => { loopWorkload(); });
+ let io = Infinity;
+ for (let i = 0; i < 3; i++) {
+ const t = performance.now();
+ await ioWorkload(corpusDir);
+ io = Math.min(io, performance.now() - t);
+ }
+ units = { parse, loop, io };
+ return units;
+}
+
+function minOf(runs: number, fn: () => void): number {
+ let best = Infinity;
+ for (let i = 0; i < runs; i++) {
+ const t = performance.now();
+ fn();
+ best = Math.min(best, performance.now() - t);
+ }
+ return best;
+}
+
+// ---------------------------------------------------------------------------
+// Measurement
+// ---------------------------------------------------------------------------
+
+/**
+ * Time `fn` and hold the result to `spec.budget`. Returns whatever `fn` returned
+ * on its last run, so a metric can assert on the work as well as its cost.
+ */
+export async function budgeted<T>(spec: MetricSpec, fn: () => Promise<T> | T): Promise<T> {
+ if (spec.unit === 'MB' || spec.unit === 'count') {
+ throw new Error(`${spec.name}: ${spec.unit} is a value, not a duration — use budgetedValue`);
+ }
+ if (!units) throw new Error('calibrate() was never called — every budget would be meaningless');
+
+ const runs = spec.runs ?? 3;
+ let best = Infinity;
+ let last!: T;
+ for (let i = 0; i < (spec.warmup ?? 0); i++) last = await fn();
+ for (let i = 0; i < runs; i++) {
+ const t = performance.now();
+ last = await fn();
+ best = Math.min(best, performance.now() - t);
+ }
+ record(spec, best / units[spec.unit], best);
+ return last;
+}
+
+/** The same contract for a metric that is already a number: bytes, or a count. */
+export async function budgetedValue(spec: MetricSpec, fn: () => Promise<number> | number): Promise<number> {
+ if (spec.unit !== 'MB' && spec.unit !== 'count') {
+ throw new Error(`${spec.name}: ${spec.unit} is a duration — use budgeted`);
+ }
+ const value = await fn();
+ record(spec, value, null);
+ return value;
+}
+
+function record(spec: MetricSpec, measured: number, rawMs: number | null): void {
+ const rec: MetricRecord = {
+ ...spec,
+ measured: round(measured),
+ rawMs: rawMs === null ? null : round(rawMs),
+ pctOfBudget: Math.round((measured / spec.budget) * 100),
+ };
+ records.push(rec);
+ void persist();
+
+ // The failure message has to carry the reason, because whoever reads it in a
+ // CI log has neither this file nor the budget table in front of them.
+ expect(
+ measured,
+ `\n ${spec.name}\n`
+ + ` measured ${rec.measured}${suffix(spec.unit)}, budget ${spec.budget}${suffix(spec.unit)}`
+ + `${rawMs === null ? '' : ` (${rec.rawMs}ms on this machine)`}\n`
+ + ` ${spec.why}\n`
+ + ' If this cost is now correct, raise the budget in the file that declares it —\n'
+ + ' deliberately, in its own commit, with the reason updated.\n',
+ ).toBeLessThanOrEqual(spec.budget);
+}
+
+const suffix = (u: Unit) => (u === 'MB' ? 'MB' : u === 'count' ? '' : ` ${u}`);
+const round = (n: number) => Number(n.toPrecision(4));
+
+/**
+ * The report is rewritten after every metric rather than at the end, so a run
+ * that dies half way still says what it managed to measure.
+ */
+async function persist(): Promise<void> {
+ await mkdir(path.dirname(REPORT_PATH), { recursive: true });
+ await writeFile(REPORT_PATH, `${JSON.stringify({
+ note: 'Generated by the scale suite — see bench/scale/README.md.',
+ units,
+ metrics: records,
+ }, null, 2)}\n`);
+}
+
+// ---------------------------------------------------------------------------
+// The corpus
+// ---------------------------------------------------------------------------
+
+/** Where the generator put it, unless $ESTELLA_STRESS_PROJECT says otherwise. */
+export function corpusDir(): string {
+ const dir = process.env.ESTELLA_STRESS_PROJECT ?? path.join(REPO, 'build', 'stress-project');
+ if (!existsSync(path.join(dir, 'project.esproject'))) {
+ throw new Error(`No stress corpus at ${dir}. Generate it: node tools/stress-project.mjs`);
+ }
+ return dir;
+}
diff --git a/desktop/tests/scale/runtime.scale.ts b/desktop/tests/scale/runtime.scale.ts
new file mode 100644
index 000000000..82b54d0c3
--- /dev/null
+++ b/desktop/tests/scale/runtime.scale.ts
@@ -0,0 +1,182 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file Per-frame and memory budgets at corpus scale.
+ *
+ * The frame budgets are CPU-side only: a headless app has no renderer, so
+ * draw-call generation and GPU upload are not here — they need the electron
+ * harness and are a separate tier. What IS here is the half that runs before
+ * any of that, and the half a big scene makes expensive.
+ *
+ * The memory budgets are the ones worth having, because bytes do not care which
+ * machine measured them. The wasm heap is exact — it is a byte length, not a
+ * sample. The JS heap is not: V8 decides when to give memory back, so its budget
+ * is set wide enough to catch a leak and no narrower.
+ */
+import { describe, it, beforeAll, expect } from 'vitest';
+import { readFile } from 'node:fs/promises';
+import path from 'node:path';
+
+import {
+ createHeadlessApp,
+ loadEsengineModule,
+ resetWorldTo,
+ type App,
+ type SceneData,
+} from 'esengine/node';
+
+import { WASM_DIR, HAS_WASM } from '../helpers/loadWasm';
+import { budgeted, budgetedValue, calibrate, corpusDir } from './harness';
+
+const GROUP = 'Runtime — frames and memory';
+const DT = 1 / 60;
+const EMPTY: SceneData = { version: 1, name: 'empty', entities: [] };
+
+let app: App;
+let module: Awaited<ReturnType<typeof loadEsengineModule>>;
+let sprites: SceneData;
+let everything: SceneData;
+
+/** One frame of the CPU-side loop: systems, then the C++ transform propagation. */
+async function tick(): Promise<void> {
+ await app.tick(DT);
+ const registry = app.world.getCppRegistry();
+ if (registry) module.transform_update(registry);
+}
+
+async function ticks(n: number): Promise<void> {
+ for (let i = 0; i < n; i++) await tick();
+}
+
+/** heapUsed after a collection — the only reading that means anything. */
+function settledHeapMB(): number {
+ const gc = (globalThis as { gc?: () => void }).gc;
+ for (let i = 0; i < 3; i++) gc?.();
+ return process.memoryUsage().heapUsed / 1e6;
+}
+
+/**
+ * Bytes the engine has actually allocated inside wasm.
+ *
+ * NOT `HEAPU8.byteLength`: emscripten hands the module ~17MB up front and the
+ * whole corpus fits inside it, so the heap's length never moves and a budget on
+ * it would read zero however much storage a change added.
+ */
+const wasmBytesMB = (): number => (module.es_getMallocBytes?.() ?? 0) / 1e6;
+
+beforeAll(async () => {
+ const root = corpusDir();
+ await calibrate(root);
+ module = await loadEsengineModule(WASM_DIR);
+ app = createHeadlessApp(module);
+ sprites = JSON.parse(await readFile(path.join(root, 'assets/scenes/sprites.esscene'), 'utf8'));
+ everything = JSON.parse(await readFile(path.join(root, 'assets/scenes/everything.esscene'), 'utf8'));
+}, 300_000);
+
+describe.skipIf(!HAS_WASM)(GROUP, () => {
+ it('ticks the 10k-sprite scene', async () => {
+ resetWorldTo(app.world, sprites);
+ await budgeted({
+ name: 'frame: 60 update ticks over 10,000 sprites',
+ group: GROUP,
+ unit: 'loop',
+ budget: 10,
+ why: 'A second of simulation with nothing moving: system dispatch, query iteration and '
+ + 'the C++ transform propagation. Sixty ticks rather than one, because one tick is '
+ + 'short enough to be measuring the clock. GPU submission is not in here.',
+ runs: 3,
+ warmup: 1,
+ }, () => ticks(60));
+ }, 600_000);
+
+ it('ticks the everything scene', async () => {
+ resetWorldTo(app.world, everything);
+ await budgeted({
+ name: 'frame: 60 update ticks over the everything scene',
+ group: GROUP,
+ unit: 'loop',
+ budget: 16,
+ why: 'The same second on the worst scene in the corpus. The gap between this and the '
+ + 'sprite scene is what the other component families cost per frame.',
+ runs: 3,
+ warmup: 1,
+ }, () => ticks(60));
+ }, 600_000);
+
+ it('holds the 10k-sprite scene in bounded wasm storage', async () => {
+ // A module of its own, so this is the cost of the scene and not of
+ // whatever an earlier test left allocated.
+ const fresh = await loadEsengineModule(WASM_DIR);
+ const freshApp = createHeadlessApp(fresh);
+ await freshApp.tick(DT);
+ const empty = (fresh.es_getMallocBytes?.() ?? 0) / 1e6;
+ resetWorldTo(freshApp.world, sprites);
+ await freshApp.tick(DT);
+
+ await budgetedValue({
+ name: 'heap: wasm bytes for 10,000 sprites',
+ group: GROUP,
+ unit: 'MB',
+ budget: 8,
+ why: 'Component storage for 10,000 entities, measured on a module nothing else has '
+ + 'touched. Exact bytes, not a sample — the same number on every machine, so it '
+ + 'is the strictest budget here.',
+ }, () => (fresh.es_getMallocBytes?.() ?? 0) / 1e6 - empty);
+ }, 600_000);
+
+ it('holds the 10k-sprite scene in a bounded JS heap', async () => {
+ resetWorldTo(app.world, EMPTY);
+ await ticks(2);
+ const empty = settledHeapMB();
+ resetWorldTo(app.world, sprites);
+ await ticks(2);
+ await budgetedValue({
+ name: 'heap: JS growth for 10,000 sprites',
+ group: GROUP,
+ unit: 'MB',
+ budget: 14,
+ why: 'The JS-side entity bookkeeping the world keeps beside the wasm storage. Wide by '
+ + 'design: V8 decides when to release, so this catches a leak and not a regression.',
+ }, () => settledHeapMB() - empty);
+ }, 600_000);
+
+ it('gives the scene back when it is closed', async () => {
+ // The Play/Stop shape: open a scene, close it, repeat. Measured from
+ // cycle 3, so one-time caches are already paid for and what is left is
+ // what the cycle itself retains.
+ const CYCLES = 20;
+ let floorAt3 = 0;
+ let wasmAt3 = 0;
+ for (let i = 0; i < CYCLES; i++) {
+ resetWorldTo(app.world, sprites);
+ await ticks(2);
+ resetWorldTo(app.world, EMPTY);
+ await ticks(2);
+ if (i === 2) {
+ floorAt3 = settledHeapMB();
+ wasmAt3 = wasmBytesMB();
+ }
+ }
+
+ await budgetedValue({
+ name: 'heap: wasm retained over 20 open/close cycles',
+ group: GROUP,
+ unit: 'MB',
+ budget: 2,
+ why: 'Opening and closing a scene twenty times must end where it started. Component '
+ + 'pools keep their capacity by design — that is a flat floor, reached by cycle '
+ + 'three. Anything that keeps climbing after it is storage nobody reclaims.',
+ }, () => wasmBytesMB() - wasmAt3);
+
+ await budgetedValue({
+ name: 'heap: JS retained over 20 open/close cycles',
+ group: GROUP,
+ unit: 'MB',
+ budget: 8,
+ why: 'The same seventeen cycles on the JS side. A per-cycle leak of even a few hundred '
+ + 'kilobytes is a session that runs out of memory; a flat line here is the contract.',
+ }, () => settledHeapMB() - floorAt3);
+
+ expect(app.world.getAllEntities().length).toBe(0);
+ }, 900_000);
+});
diff --git a/desktop/tests/scale/scene.scale.ts b/desktop/tests/scale/scene.scale.ts
new file mode 100644
index 000000000..5b01d8337
--- /dev/null
+++ b/desktop/tests/scale/scene.scale.ts
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file Scene and prefab budgets: what it costs to open, save and instantiate
+ * at corpus size.
+ *
+ * These run against the SHIPPED SDK bundle (`esengine/node`), not `src`. A
+ * probe that only exists in source is a probe the users' engine does not have —
+ * the soak suite found exactly that, a census that tree-shaking had emptied out
+ * of `dist` while every test measuring `src` stayed green.
+ *
+ * The world here is a headless App: the real wasm core, the real component
+ * registry, no renderer. Everything below is CPU-side scene work, which is
+ * where the cost of a big scene actually lives.
+ */
+import { describe, it, beforeAll, expect } from 'vitest';
+import { readFile } from 'node:fs/promises';
+import path from 'node:path';
+
+import {
+ createHeadlessApp,
+ loadEsengineModule,
+ resetWorldTo,
+ serializeScene,
+ instantiatePrefab,
+ type App,
+ type SceneData,
+} from 'esengine/node';
+
+import { WASM_DIR, HAS_WASM } from '../helpers/loadWasm';
+import { budgeted, calibrate, corpusDir } from './harness';
+
+const GROUP = 'Scenes — 10,000 sprites and the everything scene';
+
+let root: string;
+let app: App;
+let sprites: SceneData;
+let everything: SceneData;
+let prefab: unknown;
+
+const readScene = async (rel: string): Promise<SceneData> =>
+ JSON.parse(await readFile(path.join(root, rel), 'utf8')) as SceneData;
+
+beforeAll(async () => {
+ root = corpusDir();
+ await calibrate(root);
+ const module = await loadEsengineModule(WASM_DIR);
+ app = createHeadlessApp(module);
+ sprites = await readScene('assets/scenes/sprites.esscene');
+ everything = await readScene('assets/scenes/everything.esscene');
+ prefab = JSON.parse(await readFile(path.join(root, 'assets/prefabs/shard-000/prefab-00000.esprefab'), 'utf8'));
+}, 300_000);
+
+describe.skipIf(!HAS_WASM)(GROUP, () => {
+ it('opens the 10k-sprite scene', async () => {
+ // Warmed once so every timed run despawns a full world before loading —
+ // the first reset into an empty world does half the work of the rest.
+ await budgeted({
+ name: 'scene: open 10,000 sprites',
+ group: GROUP,
+ unit: 'parse',
+ budget: 55,
+ why: 'Double-clicking a scene in the editor. The entities exist and their components '
+ + 'are in wasm storage when this returns; nothing is drawn yet.',
+ runs: 3,
+ warmup: 1,
+ }, () => {
+ const map = resetWorldTo(app.world, sprites);
+ expect(map.size).toBeGreaterThan(10_000);
+ return map;
+ });
+ }, 600_000);
+
+ it('opens the everything scene', async () => {
+ // Prefab instances are entries, not entities, until an asset-backed load
+ // expands them; this path spawns the ones that carry components inline.
+ // What expanding costs is the instantiate metric below.
+ const inline = everything.entities.filter((e) => Array.isArray(e.components)).length;
+ await budgeted({
+ name: 'scene: open the everything scene',
+ group: GROUP,
+ unit: 'parse',
+ budget: 135,
+ why: 'Sprites, UI, physics bodies, spine and a tilemap in one document — the scene '
+ + 'nobody authors on purpose and every project eventually has. The worst case for open.',
+ runs: 3,
+ warmup: 1,
+ }, () => {
+ const map = resetWorldTo(app.world, everything);
+ expect(map.size).toBe(inline);
+ return map;
+ });
+ }, 600_000);
+
+ it('serializes the everything scene', async () => {
+ // Left holding the everything scene by the test above; serialize measures
+ // the world as it stands.
+ await budgeted({
+ name: 'scene: serialize the everything scene',
+ group: GROUP,
+ unit: 'parse',
+ budget: 130,
+ why: 'Every save, every Play (the editor snapshots the world so Stop can restore it) '
+ + 'and every autosave pays this. It reads every component of every entity back out.',
+ runs: 3,
+ }, () => {
+ const data = serializeScene(app.world, 'everything');
+ expect(data.entities.length).toBeGreaterThan(17_000);
+ return data;
+ });
+ }, 600_000);
+
+ it('parses the everything scene document', async () => {
+ const text = await readFile(path.join(root, 'assets/scenes/everything.esscene'), 'utf8');
+ await budgeted({
+ name: 'scene: JSON.parse the everything document',
+ group: GROUP,
+ unit: 'parse',
+ budget: 15,
+ why: 'The floor under scene open: reading 7MB of JSON off disk into objects. '
+ + 'If opening a scene approaches a small multiple of this, the loader is close to free.',
+ runs: 5,
+ }, () => JSON.parse(text));
+ }, 300_000);
+
+ it('instantiates 1000 prefabs', async () => {
+ const overrides = [{
+ prefabEntityId: '0',
+ type: 'property' as const,
+ componentType: 'Transform',
+ propertyName: 'position',
+ value: { x: 1, y: 2, z: 0 },
+ }];
+ resetWorldTo(app.world, { version: 1, name: 'empty', entities: [] });
+ await budgeted({
+ name: 'prefab: instantiate 1000 with an override',
+ group: GROUP,
+ unit: 'parse',
+ budget: 26,
+ why: 'A spawner\'s first second — bullets, enemies, tiles. Each one flattens its source '
+ + 'and replays its overrides, so this is the cost of the prefab pipeline, not of spawn.',
+ runs: 3,
+ }, async () => {
+ for (let i = 0; i < 1000; i++) {
+ await instantiatePrefab(app.world, prefab as never, { overrides });
+ }
+ // Cleared inside the timed region on purpose: leaving 1000 entities per
+ // run would make each run measure a bigger world than the last.
+ resetWorldTo(app.world, { version: 1, name: 'empty', entities: [] });
+ });
+ }, 600_000);
+});
diff --git a/desktop/vitest.scale.config.ts b/desktop/vitest.scale.config.ts
new file mode 100644
index 000000000..fe79253f1
--- /dev/null
+++ b/desktop/vitest.scale.config.ts
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+import { defineConfig } from 'vitest/config';
+import { fileURLToPath, URL } from 'node:url';
+
+/**
+ * @file Config for the scale suite, which runs against a 50,000-asset corpus.
+ *
+ * Kept out of `vitest.config.ts` so an ordinary test run neither generates that
+ * tree nor waits for it — drive it through `node tools/perf-budget.mjs`, which
+ * makes the corpus first. One process and no parallelism, because the
+ * calibration every budget divides by is measured once per process and two
+ * suites racing for the same two cores would each be measuring the other.
+ */
+export default defineConfig({
+ resolve: {
+ alias: {
+ '@': fileURLToPath(new URL('./src', import.meta.url)),
+ },
+ },
+ test: {
+ environment: 'node',
+ include: ['tests/scale/**/*.scale.ts'],
+ fileParallelism: false,
+ // One module graph for the whole run, so the three suites share one
+ // calibration and append to one report instead of each isolating its own
+ // copy of the harness and overwriting the others' numbers.
+ isolate: false,
+ pool: 'forks',
+ poolOptions: {
+ // --expose-gc: the retained-heap budgets are meaningless without a
+ // collection they can force.
+ forks: { singleFork: true, execArgv: ['--expose-gc'] },
+ },
+ testTimeout: 600_000,
+ hookTimeout: 600_000,
+ },
+});
diff --git a/package.json b/package.json
index ca8f46e39..4e921d45b 100644
--- a/package.json
+++ b/package.json
@@ -19,6 +19,7 @@
"migrate-asset-refs": "node tools/migrate-asset-refs.js",
"verify": "pnpm --filter ./sdk exec tsc --noEmit && pnpm --filter @estella/editor exec tsc --noEmit && node tools/api-surface.mjs --check && node tools/check-cycles.mjs && node tools/check-layers.mjs && node tools/check-project-settings.mjs && node tools/check-workflows.mjs && node tools/check-tool-calls.mjs && node tools/check-inspector-door.mjs && node tools/check-dirty-source.mjs && node tools/check-path-sandbox.mjs && node tools/check-comment-style.mjs && node tools/check-release-metadata.mjs && node tools/component-reference.mjs --check",
"perf": "node tools/perf-guard.mjs --check",
+ "scale": "node tools/perf-budget.mjs",
"stress-project": "node tools/stress-project.mjs",
"prepare": "git config core.hooksPath .githooks || true",
"soak": "pnpm --filter ./sdk exec vitest run tests/soak",
diff --git a/tools/perf-budget.mjs b/tools/perf-budget.mjs
new file mode 100644
index 000000000..acf4648c4
--- /dev/null
+++ b/tools/perf-budget.mjs
@@ -0,0 +1,144 @@
+#!/usr/bin/env node
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file The scale gate — run the budgets against the stress corpus.
+ *
+ * Generates the 50,000-asset project (or reuses it), runs the scale suite, and
+ * prints the contract as a table: what each cost is allowed to be, what it is,
+ * and how much of its budget it uses.
+ *
+ * The gate itself is inside the suite — each metric asserts against its own
+ * budget, so a failure names the metric and prints the reason the budget exists.
+ * This script exists to make the corpus, to show the whole table at once, and to
+ * put that table in the CI job summary where somebody will actually read it.
+ *
+ * Its counterpart is `tools/perf-guard.mjs`, and they answer different
+ * questions. The guard asserts RATIOS between microbenchmarks — architectural
+ * invariants like "editing one UI field is far cheaper than adding a node" — and
+ * its numbers are a snapshot you accept with `--update`. This asserts CEILINGS
+ * on whole operations at a size no example reaches, and its numbers are not
+ * acceptable by the change that breaks them.
+ *
+ * node tools/perf-budget.mjs generate if needed, measure, gate
+ * node tools/perf-budget.mjs --table print the last run's table only
+ * node tools/perf-budget.mjs --regen force the corpus to be rebuilt
+ */
+
+import { existsSync, readFileSync, rmSync, appendFileSync } from 'node:fs';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { spawnSync } from 'node:child_process';
+
+import { generateStressProject, countAssets } from './stress-project.mjs';
+
+const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+const REPORT = join(ROOT, 'build', 'scale-report.json');
+
+const argv = process.argv.slice(2);
+
+if (argv.includes('--table')) {
+ if (!existsSync(REPORT)) {
+ console.error('perf-budget: no report yet — run `node tools/perf-budget.mjs` first.');
+ process.exit(1);
+ }
+ printTable(JSON.parse(readFileSync(REPORT, 'utf8')));
+ process.exit(0);
+}
+
+const corpus = await generateStressProject({
+ force: argv.includes('--regen'),
+ log: (m) => console.log(m),
+});
+if (corpus.assets !== countAssets(1)) {
+ console.error(`perf-budget: the corpus holds ${corpus.assets} assets, not ${countAssets(1)}.`
+ + ' Budgets are calibrated against the full corpus and mean nothing against a smaller one.');
+ process.exit(1);
+}
+
+// A report left by an earlier run would be read as this one's if the suite died
+// before writing anything.
+rmSync(REPORT, { force: true });
+
+const run = spawnSync('pnpm', ['exec', 'vitest', 'run', '--config', 'vitest.scale.config.ts'], {
+ cwd: join(ROOT, 'desktop'),
+ stdio: 'inherit',
+ env: { ...process.env, ESTELLA_STRESS_PROJECT: corpus.dir },
+});
+
+if (!existsSync(REPORT)) {
+ console.error('perf-budget: the scale suite produced no report — it did not reach a single metric.');
+ process.exit(1);
+}
+
+const report = JSON.parse(readFileSync(REPORT, 'utf8'));
+printTable(report);
+summarise(report);
+process.exit(run.status ?? 1);
+
+// ---------------------------------------------------------------------------
+
+function printTable(report) {
+ const { units, metrics } = report;
+ console.log('');
+ console.log(`Scale budgets — ${countAssets(1).toLocaleString('en-US')} assets`);
+ console.log(` one unit on this machine: ${Object.entries(units ?? {})
+ .map(([k, v]) => `${k} ${v.toFixed(2)}ms`).join(' ')}`);
+ console.log('');
+
+ let group = null;
+ for (const m of metrics) {
+ if (m.group !== group) {
+ group = m.group;
+ console.log(` ${group}`);
+ }
+ const bar = meter(m.pctOfBudget);
+ const raw = m.rawMs === null ? '' : ` (${fmt(m.rawMs)}ms)`;
+ console.log(` ${status(m.pctOfBudget)} ${m.name.padEnd(52)}`
+ + `${String(m.measured).padStart(9)} / ${String(m.budget).padEnd(6)}${unitName(m.unit).padEnd(6)}`
+ + ` ${bar} ${String(m.pctOfBudget).padStart(3)}%${raw}`);
+ }
+ console.log('');
+
+ const over = metrics.filter((m) => m.pctOfBudget > 100);
+ const near = metrics.filter((m) => m.pctOfBudget > 70 && m.pctOfBudget <= 100);
+ if (over.length) console.log(` ${over.length} over budget.`);
+ // Not a failure, and worth saying out loud: a metric drifting into its
+ // headroom is the only warning anybody gets before it stops being headroom.
+ if (near.length) console.log(` ${near.length} above 70% of budget — headroom is going.`);
+ if (!over.length && !near.length) console.log(' All within budget, none above 70%.');
+ console.log('');
+}
+
+/** The same table as markdown, for the GitHub job summary. */
+function summarise(report) {
+ const out = process.env.GITHUB_STEP_SUMMARY;
+ if (!out) return;
+ const lines = [
+ `### Scale budgets — ${countAssets(1).toLocaleString('en-US')} assets`,
+ '',
+ `Calibration on this runner: ${Object.entries(report.units ?? {})
+ .map(([k, v]) => `\`${k}\` ${v.toFixed(2)}ms`).join(', ')}`,
+ '',
+ '| | metric | measured | budget | used | on this runner |',
+ '| --- | --- | ---: | ---: | ---: | ---: |',
+ ];
+ for (const m of report.metrics) {
+ lines.push(`| ${m.pctOfBudget > 100 ? '❌' : m.pctOfBudget > 70 ? '⚠️' : '✅'} | ${m.name}`
+ + ` | ${m.measured} ${unitName(m.unit)} | ${m.budget} ${unitName(m.unit)}`
+ + ` | ${m.pctOfBudget}% | ${m.rawMs === null ? '—' : `${fmt(m.rawMs)}ms`} |`);
+ }
+ appendFileSync(out, `${lines.join('\n')}\n`);
+}
+
+// Declarations, not consts: printTable runs at module top level (above), so an
+// arrow bound down here would still be in its temporal dead zone.
+function unitName(u) { return u === 'MB' ? 'MB' : u === 'count' ? '' : u; }
+function fmt(n) { return n >= 100 ? n.toFixed(0) : n.toFixed(1); }
+function status(pct) { return pct > 100 ? 'FAIL' : pct > 70 ? 'near' : ' ok'; }
+
+function meter(pct) {
+ const width = 20;
+ const filled = Math.min(width, Math.round((pct / 100) * width));
+ return `[${'#'.repeat(filled)}${'.'.repeat(width - filled)}]`;
+}