Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Make the tier benchmark reproducible: idle gate, interleaved driver, table generator (Part of #904)

Nguyen Xuan An6966113f0970
diff --git a/bench/tiers/README.md b/bench/tiers/README.md
index 34995a8f..f84b66d7 100644
--- a/bench/tiers/README.md
+++ b/bench/tiers/README.md
@@ -10,7 +10,7 @@ One scene, as the first step. The other five scenes of `bench/reference/scenes.j
| Tier | For | What runs it today | 67js time | Blender equivalent and its time | Quality (FLIP vs Cycles 4096 spp, lower is better) | VRAM (card wide, shared) |
| --- | --- | --- | --- | --- | --- | --- |
-| draft (planned name) | A quick look while editing | Glaze, 0.5 render scale, 1 sample, light shading | 1.20 ms a frame (0.88 to 2.43), first frame 11.9 ms | EEVEE 5.3, 1 sample: 99 ms a render call (76 to 157) | Glaze 0.188, EEVEE 0.562 | Glaze +62 MiB (render targets 39 MB), EEVEE +2,218 MiB |
+| draft (planned name) | A quick look while editing | Glaze, 0.5 render scale, 1 sample, light shading (measured before the 8 bit target and level of detail change below) | 1.20 ms a frame (0.88 to 2.43), first frame 11.9 ms | EEVEE 5.3, 1 sample: 99 ms a render call (76 to 157) | Glaze 0.188, EEVEE 0.562 | Glaze +62 MiB (render targets 39 MB), EEVEE +2,218 MiB |
| realtime | Interactive use | Glaze, default settings | 1.11 ms a frame (1.05 to 1.24), first frame 9.7 ms | EEVEE 5.3 defaults, 64 samples: 1,127 ms a render call (716 to 1,891) | Glaze 0.197, EEVEE 0.562 | Glaze +119 MiB (render targets 65.7 MB), EEVEE +2,503 MiB |
| realtime-high | Interactive, higher quality | Planned (#902), not measurable yet | none | none | none | none |
| (not realtime-high) experimental real-time path tracer, WebGPU | Research | `src/path-tracer-realtime*.js`, not reachable from MCP | 5.6 ms a frame; FLIP under 0.20 after 16 frames (0.15 s), under 0.10 after 64 frames (0.41 s) | none | 0.056 after 256 frames | +243 MiB |
@@ -35,6 +35,36 @@ Conditions: lit-interior only, 960 by 540, 8 bounces, no denoiser on either side
Native against Cycles repeats #921 (0.91 to 0.95 there): the sign of the difference moves with the load on the card, so read it as about equal. Native draws a sample 6.0 times faster and needs 6.9 times the samples, so the two nearly cancel.
+## Reproducible runs (idle gate, interleaving, table)
+
+The table above was measured on a card that other sessions kept at 17 to 100 percent, and a re-run on a card at 99 to 100 percent measured Glaze realtime at 11 ms a frame against 1.1 ms, and EEVEE default at 6.7 to 7.3 s against 1.1 s. Numbers from a busy card are not comparable, so the benchmark now measures the load and refuses to mix it in.
+
+One command runs everything (about an hour on an idle card for all six scenes at three repetitions, plus the six Cycles references):
+
+```sh
+GPU_LOCK_WEIGHT=heavy bash ~/.agents/tools/gpu-lock.sh node bench/tiers/run-all.mjs <work directory>
+```
+
+- Idle gate ([`idle-gate.mjs`](idle-gate.mjs)): before each run it polls `nvidia-smi` once a second until utilisation is at or under 15 percent on 5 polls in a row, giving up after `--max-wait` seconds (default 600) with a message that names the load. It then samples the card for 3 s (the background figure, because the job drives utilisation itself once it starts), runs the command with a sampler beside it, and writes `{ background: { utilMean, utilMedian, utilMax, foreignProcesses }, run: { utilMedian, utilMax }, contended }` into the cell. A run whose background utilisation (mean of the 3 s) was over 30 percent is `contended`: it stays in the results file and is left out of every median. `--report-only` records the load without waiting, to check the plumbing on a busy card. The shell runners use the same helper through `node bench/tiers/idle-gate.mjs --out <file> -- <command>`.
+- Interleaving ([`run-all.mjs`](run-all.mjs)): per repetition and scene the engines run back to back in one order that rotates by one place each repetition (Glaze draft, Glaze realtime, EEVEE lowest, EEVEE default, Cook native, Cycles, Cook webgpu), all at 960 by 540, scenes from lit-interior on. `--with-cook-gpu` and `--with-cook-cpu` add the slow backends. Cook native and Cycles run at each count of `--samples` (default 256,1024) so the equal-error figure is interpolated, not extrapolated. The results file `<work directory>/matrix-960.json` is rewritten after every cell and a finished cell is skipped on the next start (`--rerun-contended` repeats the contended ones).
+- Scores: `run-all.mjs` ends by running `score-tiers.py matrix`, which adds FLIP, display relMSE and relMSE to each cell. A Glaze draft cell has two FLIP figures: against the full reference after a bilinear enlargement, and against the reference reduced to the draft's own canvas.
+- Table: `node bench/tiers/make-table.mjs <work directory>/matrix-960.json` writes a Markdown and an HTML table per scene (medians with ranges, the conditions footnote, and the equal-error statement for Cook native and webgpu, computed from the results).
+- EEVEE has two times. The table uses the whole `bpy.ops.render.render` call, because that is what produces a picture and what the Glaze frame (draw plus pixel read) is compared with. Blender's own render time, from the render_pre to render_post handlers, is recorded beside it and printed in the same cell.
+
+### What draft changes against realtime
+
+Draft was 0.5 render scale, 1 sample and light shading, and it measured the same frame time as realtime because a 1 ms frame on this scene is mostly the pixel read that ends it. It now also uses an 8 bit target (`hdr: false`) and `lodPixelError: 4`, and page.js records a second figure, the time of 30 frames queued back to back with one read at the end. Lit-interior, 3 repetitions, Glaze only, 960 by 540 (measured 2026-11-11 with the card 60 to 75 percent busy from other jobs, so read the ratios, not the milliseconds; the reference here was Cycles at 256 samples, not 4096):
+
+| Variant | Queued frame | Frame with read | FLIP, enlarged to 960 by 540 | FLIP at its own size |
+| --- | --- | --- | --- | --- |
+| realtime | 0.55 ms | 1.34 ms | 0.201 | none |
+| draft (new) | 0.32 ms | 1.12 ms | 0.191 | 0.193 |
+| draft as of #931 (hdr, default level of detail) | 0.45 ms | 1.42 ms | 0.191 | 0.193 |
+| realtime at 0.5 render scale | 0.56 ms | 1.39 ms | 0.201 | 0.204 |
+| draft with full shading | 0.44 ms | 1.34 ms | 0.201 | 0.204 |
+
+So draft is 1.7 times cheaper in queued frame time, and the 8 bit target and the level of detail account for the step from 0.45 to 0.32 ms. Draft scores better than realtime because of the light shading, not the smaller canvas: with full shading the score is realtime's 0.201 at either size, and with light shading it is 0.191 at either size. On this scene the full shading's prefiltered environment lookup and wide shadow filter sit further from Cycles than the light model does; that is one scene and one reference, and the shading terms were not taken apart further. Run `node bench/tiers/run-all.mjs <work directory> --engines glaze-draft,glaze-realtime,glaze-realtime-half,glaze-draft-full,glaze-draft-hdr,glaze-draft-before` to repeat it.
+
## Method
- `render-references.sh`: Cycles 4096 spp references through `bench/cycles-parity/render-cycles.py`. `run-final-tier.sh`: Cook cpu, gpu and webgpu (`run-cook.mjs`), native (`trace-native.mjs`) and Cycles, interleaved, three repetitions at 256 samples, once at 64 and 1,024, native also at 2,048 and 4,096. One Cook trace is limited to about 256 samples at this size ("too much ray work"), so larger counts are seeds 1 to n averaged (`run-cook-seeds.sh`, `score-tiers.py` stacks). The gpu stack stopped at 1,024 samples, because the seeds above 7 did not finish.
@@ -50,10 +80,10 @@ Native against Cycles repeats #921 (0.91 to 0.95 there): the sign of the differe
- Peak card memory is the card's `memory.used` before against its peak, polled at 100 ms (`nvidia-smi`) or 50 ms (`measure-gpu-run.py`). It includes other jobs, so it is an upper bound. `memoryReport()` of the renderer, the exact size of Glaze's render targets, is in `results/glaze-lit-interior-960.json`.
- The FLIP of Glaze here (0.197) is far below the 0.506 in `bench/reference/README.md`, which scores Glaze against a Blender 5.2.2 Cycles render with other settings. These Cycles settings and Blender 5.3 differ from that suite, so the numbers do not line up; they were not reconciled.
- EEVEE has no global illumination with ray tracing off, so on an interior it loses on quality by a wide margin (it is darker, see FLIP). EEVEE with ray tracing on was not measured.
-- Glaze at 960 by 540 on this scene takes about a millisecond, which is close to the cost of the browser's pixel read, so the draft against realtime frame time cannot be told apart here. Draft is not faster than realtime in this measurement (1.20 against 1.11 ms; ranges overlap).
+- Glaze at 960 by 540 on this scene takes about a millisecond, which is close to the cost of the browser's pixel read, so the draft against realtime frame time could not be told apart in the table above (1.20 against 1.11 ms; ranges overlap). The queued-frame figure and the cheaper draft settings below address that; the table rows above were not re-measured.
## Not done
-- The other five scenes (`spot-area`, `light-linking`, `metal`, `glass`, `outdoor-hdri`) are exported (`scenes/`) but have no Cycles references or runs. Run `render-references.sh`, `run-eevee.sh` and `run-glaze.mjs` for them.
+- The other five scenes (`spot-area`, `light-linking`, `metal`, `glass`, `outdoor-hdri`) have no committed runs; `run-all.mjs` exports, references and measures them (the full matrix has not been run yet).
- Lit-interior at 1920 by 1080 for the real-time rows, and the settle runs at that size.
- three.js 0.187 against Glaze on `bench/runtime`'s scenes: not run here; `bench/runtime/README.md` has its own results.
diff --git a/bench/tiers/idle-gate.mjs b/bench/tiers/idle-gate.mjs
new file mode 100644
index 00000000..9b58d13f
--- /dev/null
+++ b/bench/tiers/idle-gate.mjs
@@ -0,0 +1,194 @@
+// Idle gate for the tier benchmark: waits until the card is quiet, then runs one command while recording how busy the card was.
+//
+// As a command (shell and Python runners use this form):
+// node bench/tiers/idle-gate.mjs [--out <file.gate.json>] [--threshold 15] [--consecutive 5] [--max-wait 600] [--contended-above 30] [--report-only] -- <command> [arguments...]
+// without "-- <command>" it only waits and prints the gate result.
+// As a module (the Node runners): import { runGated } from './idle-gate.mjs'.
+//
+// Steps: (1) poll nvidia-smi every second until utilisation is at or under --threshold on --consecutive polls in a row, or --max-wait
+// seconds pass; (2) sample the card for 3 s as the background figure, because the job itself drives utilisation once it starts;
+// (3) run the command and sample utilisation while it runs. A run whose background mean is over --contended-above is marked contended:
+// the table generator keeps it in the raw cells and leaves it out of medians. --report-only (or TIER_GATE_REPORT_ONLY=1) never waits and
+// never fails; it only records, which is for checking the plumbing on a busy card.
+// Exit code: the command's own; 3 when the gate times out and the command was not run (gate only mode exits 3 as well).
+import { execFile, spawn } from 'node:child_process';
+import { mkdir, writeFile } from 'node:fs/promises';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { promisify } from 'node:util';
+
+const runFile = promisify(execFile);
+
+export const GATE_DEFAULTS = {
+ thresholdPercent: Number(process.env.TIER_GATE_THRESHOLD ?? 15),
+ consecutivePolls: Number(process.env.TIER_GATE_CONSECUTIVE ?? 5),
+ maxWaitSeconds: Number(process.env.TIER_GATE_MAX_WAIT ?? 600),
+ contendedAbovePercent: Number(process.env.TIER_GATE_CONTENDED_ABOVE ?? 30),
+ backgroundSeconds: 3,
+ runPollMilliseconds: 250,
+ reportOnly: process.env.TIER_GATE_REPORT_ONLY === '1',
+};
+
+// Windows desktop programs that always show up in the compute list; they are not jobs that load the card.
+const DESKTOP_SHELL_BINARIES = new Set([
+ 'applicationframehost.exe', 'cloudflare warp.exe', 'discord.exe', 'explorer.exe', 'logioptionsplus_agent.exe', 'msedgewebview2.exe',
+ 'oh-snap.exe', 'powertoys.colorpickerui.exe', 'powertoys.fancyzones.exe', 'searchhost.exe', 'shellexperiencehost.exe', 'shellhost.exe',
+ 'startmenuexperiencehost.exe', 'systemsettings.exe', 'textinputhost.exe', 'zalo.exe', 'zalocall.exe',
+]);
+
+const sleep = milliseconds => new Promise(done => setTimeout(done, milliseconds));
+
+export function median(values) {
+ if (values.length === 0) return null;
+ const ordered = [...values].sort((first, second) => first - second);
+ const middle = Math.floor(ordered.length / 2);
+ return ordered.length % 2 ? ordered[middle] : (ordered[middle - 1] + ordered[middle]) / 2;
+}
+
+export async function readCard() {
+ const { stdout } = await runFile('nvidia-smi', ['--query-gpu=utilization.gpu,memory.used', '--format=csv,noheader,nounits']);
+ const [utilizationPercent, memoryMebibytes] = stdout.trim().split(/\r?\n/)[0].split(',').map(Number);
+ return { utilizationPercent, memoryMebibytes };
+}
+
+// Programs holding the card, without the desktop shell. `available` is false when the driver reports no list (Windows often does not).
+export async function listForeignProcesses() {
+ try {
+ const { stdout } = await runFile('nvidia-smi', ['--query-compute-apps=pid,process_name', '--format=csv,noheader']);
+ const names = stdout.split(/\r?\n/).filter(Boolean).map(line => {
+ const separator = line.indexOf(',');
+ return line.slice(separator + 1).trim().split(/[\\/]/).pop();
+ });
+ const jobs = names.filter(name => !DESKTOP_SHELL_BINARIES.has(name.toLowerCase()));
+ return { available: names.length > 0, names: [...new Set(jobs)].sort() };
+ } catch (error) {
+ return { available: false, names: [], error: error.message };
+ }
+}
+
+function summarize(utilizations) {
+ return { utilMedian: median(utilizations), utilMean: utilizations.length ? utilizations.reduce((sum, value) => sum + value, 0) / utilizations.length : null, utilMax: utilizations.length ? Math.max(...utilizations) : null, samples: utilizations.length };
+}
+
+// Polls once a second until utilisation is at or under the threshold on `consecutivePolls` polls in a row.
+export async function waitForIdle(options = {}) {
+ const { thresholdPercent, consecutivePolls, maxWaitSeconds, reportOnly } = { ...GATE_DEFAULTS, ...options };
+ const started = Date.now();
+ const recent = [];
+ let quietPolls = 0;
+ while (true) {
+ const { utilizationPercent } = await readCard();
+ recent.push(utilizationPercent);
+ quietPolls = utilizationPercent <= thresholdPercent ? quietPolls + 1 : 0;
+ const waitedSeconds = (Date.now() - started) / 1000;
+ if (quietPolls >= consecutivePolls) return { idle: true, timedOut: false, reportOnly, waitedSeconds, thresholdPercent, consecutivePolls, maxWaitSeconds, lastPolls: recent.slice(-10) };
+ if (reportOnly) return { idle: false, timedOut: false, reportOnly, waitedSeconds, thresholdPercent, consecutivePolls, maxWaitSeconds, lastPolls: recent.slice(-10) };
+ if (waitedSeconds >= maxWaitSeconds) {
+ return {
+ idle: false, timedOut: true, reportOnly, waitedSeconds, thresholdPercent, consecutivePolls, maxWaitSeconds, lastPolls: recent.slice(-10),
+ message: `Card not idle after ${Math.round(waitedSeconds)} s: needs at most ${thresholdPercent}% on ${consecutivePolls} polls in a row, last polls ${recent.slice(-5).join(', ')}%. Other jobs are using it; stop them, raise --max-wait, or pass --report-only to record without waiting.`,
+ };
+ }
+ await sleep(1000);
+ }
+}
+
+// The 3 s before a run: the load other programs put on the card, measured while our job is not running.
+export async function sampleBackground(seconds = GATE_DEFAULTS.backgroundSeconds) {
+ const utilizations = [];
+ const started = Date.now();
+ while ((Date.now() - started) / 1000 < seconds) {
+ utilizations.push((await readCard()).utilizationPercent);
+ await sleep(250);
+ }
+ const foreign = await listForeignProcesses();
+ return { ...summarize(utilizations), foreignProcesses: foreign.names, computeAppsAvailable: foreign.available };
+}
+
+function startRunSampler(intervalMilliseconds) {
+ const utilizations = [];
+ let running = true;
+ const loop = (async () => {
+ while (running) {
+ utilizations.push((await readCard()).utilizationPercent);
+ await sleep(intervalMilliseconds);
+ }
+ })();
+ return async () => {
+ running = false;
+ await loop;
+ return summarize(utilizations);
+ };
+}
+
+// Gate, background sample, then the command (spawned with `spawnOptions`) with a sampler running. Returns the gate record and the exit code.
+export async function runGated(command, commandArguments, options = {}, spawnOptions = { stdio: 'inherit' }) {
+ const settings = { ...GATE_DEFAULTS, ...options };
+ const gate = await waitForIdle(settings);
+ if (gate.timedOut) {
+ console.error(gate.message);
+ return { exitCode: 3, record: { gate, ran: false, contended: true, contendedReason: 'gate timed out' } };
+ }
+ const background = await sampleBackground(settings.backgroundSeconds);
+ const stopSampler = startRunSampler(settings.runPollMilliseconds);
+ const started = Date.now();
+ const exitCode = await new Promise(done => {
+ const child = spawn(command, commandArguments, spawnOptions);
+ child.on('error', error => { console.error(`Could not start ${command}: ${error.message}`); done(127); });
+ child.on('close', code => done(code ?? 1));
+ });
+ const wallSeconds = (Date.now() - started) / 1000;
+ const run = await stopSampler();
+ // The mean, not the median, so a spike from another job inside the 3 s counts.
+ const contended = background.utilMean > settings.contendedAbovePercent;
+ const record = {
+ ran: true,
+ exitCode,
+ gate,
+ background,
+ run: { ...run, wallSeconds },
+ contended,
+ contendedReason: contended ? `background utilisation mean ${background.utilMean.toFixed(0)}% (max ${background.utilMax}%) over ${settings.contendedAbovePercent}%` : null,
+ contendedAbovePercent: settings.contendedAbovePercent,
+ };
+ return { exitCode, record };
+}
+
+function parseArguments(argumentList) {
+ const separator = argumentList.indexOf('--');
+ const flags = separator === -1 ? argumentList : argumentList.slice(0, separator);
+ const command = separator === -1 ? [] : argumentList.slice(separator + 1);
+ const value = (flag, fallback) => (flags.includes(flag) ? flags[flags.indexOf(flag) + 1] : fallback);
+ return {
+ command,
+ out: value('--out', null),
+ options: {
+ thresholdPercent: Number(value('--threshold', GATE_DEFAULTS.thresholdPercent)),
+ consecutivePolls: Number(value('--consecutive', GATE_DEFAULTS.consecutivePolls)),
+ maxWaitSeconds: Number(value('--max-wait', GATE_DEFAULTS.maxWaitSeconds)),
+ contendedAbovePercent: Number(value('--contended-above', GATE_DEFAULTS.contendedAbovePercent)),
+ reportOnly: flags.includes('--report-only') || GATE_DEFAULTS.reportOnly,
+ },
+ };
+}
+
+if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+ const { command, out, options } = parseArguments(process.argv.slice(2));
+ let exitCode;
+ let record;
+ if (command.length === 0) {
+ const gate = await waitForIdle(options);
+ if (gate.timedOut) console.error(gate.message);
+ record = { ran: false, gate };
+ exitCode = gate.timedOut ? 3 : 0;
+ } else {
+ ({ exitCode, record } = await runGated(command[0], command.slice(1), options));
+ }
+ if (out) {
+ await mkdir(dirname(resolve(out)), { recursive: true });
+ await writeFile(resolve(out), `${JSON.stringify(record, null, 1)}\n`);
+ } else {
+ console.log(JSON.stringify(record));
+ }
+ process.exit(exitCode);
+}
diff --git a/bench/tiers/make-table.mjs b/bench/tiers/make-table.mjs
new file mode 100644
index 00000000..16161f89
--- /dev/null
+++ b/bench/tiers/make-table.mjs
@@ -0,0 +1,187 @@
+// Turns the results file of run-all.mjs (after score-tiers.py matrix) into the tier table, as Markdown and as an HTML snippet.
+// node bench/tiers/make-table.mjs <results.json> [--scenes lit-interior,...] [--markdown <file>] [--html <file>]
+// One table per scene. Every figure is the median of the runs that were not contended, with the range beside it; a cell whose background
+// utilisation was over the limit stays in the results file and is left out here. When every run of a figure was contended the figure is
+// still printed, marked "contended". The Cook rows end with the equal-error statement, computed from the results.
+import { readFile, writeFile } from 'node:fs/promises';
+import { resolve } from 'node:path';
+import { median } from './idle-gate.mjs';
+
+const argumentsGiven = process.argv.slice(2);
+const valueAfter = (flag, fallback) => (argumentsGiven.includes(flag) ? argumentsGiven[argumentsGiven.indexOf(flag) + 1] : fallback);
+const resultsPath = resolve(argumentsGiven[0] ?? (() => { throw new Error('Usage: make-table.mjs <results.json> [--scenes a,b] [--markdown file] [--html file]'); })());
+const results = JSON.parse(await readFile(resultsPath, 'utf8'));
+const sceneIds = valueAfter('--scenes', null)?.split(',') ?? results.scenes;
+const markdownPath = resolve(valueAfter('--markdown', resultsPath.replace(/\.json$/, '.table.md')));
+const htmlPath = resolve(valueAfter('--html', resultsPath.replace(/\.json$/, '.table.html')));
+
+const COLUMNS = ['Tier', 'For', 'Runs on', 'Trade-off', '67js measured', 'Blender equivalent measured', 'FLIP vs Cycles (lower is better)'];
+
+// The cells of one engine on one scene: the usable ones, or all of them (flagged) when none is usable.
+function group(scene, engine, samples = null) {
+ const all = results.cells.filter(cell => cell.status === 'ok' && cell.scene === scene && cell.engine === engine && (samples === null || cell.samples === samples));
+ const usable = all.filter(cell => !cell.contended);
+ return { cells: usable.length ? usable : all, usable: usable.length, total: all.length, contendedOnly: usable.length === 0 && all.length > 0 };
+}
+
+function spread(values, digits = 1) {
+ if (values.length === 0) return null;
+ return { median: median(values), low: Math.min(...values), high: Math.max(...values), count: values.length, digits };
+}
+
+const number = (value, digits = 1) => value.toLocaleString('en-US', { minimumFractionDigits: digits, maximumFractionDigits: digits });
+function show(figure, unit) {
+ if (!figure) return 'not run';
+ const range = figure.count > 1 ? ` (${number(figure.low, figure.digits)} to ${number(figure.high, figure.digits)})` : '';
+ return `${number(figure.median, figure.digits)}${unit ? ` ${unit}` : ''}${range}`;
+}
+const flag = grouped => (grouped.contendedOnly ? ' contended' : '');
+const runsNote = grouped => (grouped.total === 0 ? '' : `, ${grouped.usable} of ${grouped.total} runs usable${flag(grouped)}`);
+
+function metric(grouped, name, digits = 1, scale = 1) {
+ return spread(grouped.cells.map(cell => cell.metrics[name]).filter(value => typeof value === 'number').map(value => value * scale), digits);
+}
+function score(grouped, name = 'flip') {
+ return spread(grouped.cells.map(cell => cell.score?.[name]).filter(value => typeof value === 'number'), 3);
+}
+
+function glazeText(grouped) {
+ if (grouped.total === 0) return 'not run';
+ const frame = metric(grouped, 'primaryMilliseconds', 2);
+ const throughput = metric(grouped, 'throughputMilliseconds', 2);
+ const first = metric(grouped, 'firstFrameMilliseconds', 1);
+ return `${show(frame, 'ms a frame with the pixel read')}; ${show(throughput, 'ms a frame queued back to back')}; first frame ${show(first, 'ms')}${runsNote(grouped)}`;
+}
+
+function eeveeText(grouped) {
+ if (grouped.total === 0) return 'not run';
+ const call = metric(grouped, 'primaryMilliseconds', 0);
+ const engine = metric(grouped, 'engineMilliseconds', 0);
+ return `${show(call, 'ms a render call')}; Blender's own render time ${engine ? show(engine, 'ms') : 'not available'}${runsNote(grouped)}`;
+}
+
+function countedText(grouped) {
+ if (grouped.total === 0) return 'not run';
+ const perSample = metric(grouped, 'perSampleMilliseconds', 2);
+ const total = metric(grouped, 'totalTraceSeconds', 2);
+ return `${show(perSample, 'ms a sample')}; ${grouped.cells[0].samples} samples take ${show(total, 's')} with setup${runsNote(grouped)}`;
+}
+
+function flipText(parts) {
+ return parts.filter(Boolean).join('; ') || 'not scored';
+}
+
+function glazeFlip(grouped, label) {
+ if (grouped.total === 0) return null;
+ const upscaled = score(grouped);
+ const native = score(grouped, 'flipNative');
+ if (!upscaled) return null;
+ return `${label} ${show(upscaled, '')}${native ? ` enlarged, ${show(native, '')} at its own ${grouped.cells[0].score.nativeSize}` : ''}`;
+}
+function plainFlip(grouped, label) {
+ const value = score(grouped);
+ return value ? `${label} ${show(value, '')}` : null;
+}
+
+// Samples needed for the same relMSE as the reference engine's, and the time that takes. points: [{ samples, relMse }].
+function samplesForError(points, targetError) {
+ const ordered = [...points].sort((first, second) => first.samples - second.samples);
+ for (let index = 0; index + 1 < ordered.length; index++) {
+ const low = ordered[index];
+ const high = ordered[index + 1];
+ if (Math.min(low.relMse, high.relMse) <= targetError && targetError <= Math.max(low.relMse, high.relMse) && low.relMse !== high.relMse) {
+ const fraction = (Math.log(targetError) - Math.log(low.relMse)) / (Math.log(high.relMse) - Math.log(low.relMse));
+ return { samples: Math.exp(Math.log(low.samples) + fraction * (Math.log(high.samples) - Math.log(low.samples))), method: 'interpolated between measured counts' };
+ }
+ }
+ const nearest = ordered.reduce((best, point) => (Math.abs(Math.log(point.relMse / targetError)) < Math.abs(Math.log(best.relMse / targetError)) ? point : best));
+ return { samples: (nearest.samples * nearest.relMse) / targetError, method: `extrapolated from ${nearest.samples} samples as 1 over N` };
+}
+
+function countedPoints(scene, engine) {
+ const sampleCounts = [...new Set(results.cells.filter(cell => cell.scene === scene && cell.engine === engine && cell.status === 'ok').map(cell => cell.samples))];
+ return sampleCounts.map(samples => {
+ const grouped = group(scene, engine, samples);
+ const relMse = score(grouped, 'relMse');
+ const sampling = metric(grouped, 'samplingSeconds', 3);
+ const total = metric(grouped, 'totalTraceSeconds', 3);
+ return relMse && sampling && total ? { samples, relMse: relMse.median, samplingSeconds: sampling.median, totalSeconds: total.median } : null;
+ }).filter(Boolean);
+}
+
+// The equal-error statement for one Cook engine against Cycles: the samples it needs and the time ratio at that error.
+function equalErrorStatement(scene, engine, label) {
+ const cycles = countedPoints(scene, 'cycles');
+ const cook = countedPoints(scene, engine);
+ const reference = cycles.find(point => point.samples === 256) ?? cycles[0];
+ // A reference error of zero (the reference compared with itself) has no equal-error point.
+ if (!reference || cook.length === 0 || !(reference.relMse > 1e-9)) return null;
+ const needed = samplesForError(cook.map(point => ({ samples: point.samples, relMse: point.relMse })), reference.relMse);
+ const perSampleCook = median(cook.map(point => point.samplingSeconds / point.samples));
+ const setupCook = median(cook.map(point => point.totalSeconds - point.samplingSeconds));
+ const perSampleCycles = reference.samplingSeconds / reference.samples;
+ const samplingTime = needed.samples * perSampleCook;
+ const totalTime = samplingTime + setupCook;
+ return {
+ engine, label, targetRelMse: reference.relMse, cyclesSamples: reference.samples, samplesNeeded: needed.samples, sampleRatio: needed.samples / reference.samples, method: needed.method,
+ perSampleSpeedup: perSampleCycles / perSampleCook, samplingSeconds: samplingTime, totalSeconds: totalTime,
+ timeRatioSampling: samplingTime / reference.samplingSeconds, timeRatioTotal: totalTime / reference.totalSeconds,
+ };
+}
+
+function equalErrorText(statement) {
+ if (!statement) return null;
+ const times = value => number(value, 2);
+ return `Equal error: ${statement.label} needs about ${number(statement.sampleRatio, 1)} times the samples of Cycles (${number(Math.round(statement.samplesNeeded), 0)} against ${statement.cyclesSamples}, to reach relMSE ${number(statement.targetRelMse, 4)}; ${statement.method}). It draws a sample ${times(statement.perSampleSpeedup)} times faster, so the time to equal error is ${times(statement.timeRatioSampling)} times Cycles' (sampling only) and ${times(statement.timeRatioTotal)} times with setup included (${number(statement.samplingSeconds, 1)} s against ${number(statement.samplingSeconds / statement.timeRatioSampling, 1)} s sampling).`;
+}
+
+function rowsForScene(scene) {
+ const draftGlaze = group(scene, 'glaze-draft');
+ const draftEevee = group(scene, 'eevee-lowest');
+ const realtimeGlaze = group(scene, 'glaze-realtime');
+ const realtimeEevee = group(scene, 'eevee-default');
+ const native = group(scene, 'cook-native');
+ const cyclesAt = group(scene, 'cycles', native.cells[0]?.samples ?? 256);
+ const nativeAt = native.cells.length ? group(scene, 'cook-native', native.cells[0].samples) : native;
+ const rows = [
+ ['draft (planned name)', 'A quick look while editing', 'Glaze, 0.5 render scale, 1 sample, light shading, 8 bit target, coarser level of detail', 'Aliased edges, no environment lookup, a quarter of the pixels enlarged', glazeText(draftGlaze), eeveeText(draftEevee),
+ flipText([glazeFlip(draftGlaze, 'Glaze'), plainFlip(draftEevee, 'EEVEE 1 sample')])],
+ ['realtime', 'Interactive use', 'Glaze, default settings', 'No path-traced global illumination: indirect light is approximate', glazeText(realtimeGlaze), eeveeText(realtimeEevee),
+ flipText([glazeFlip(realtimeGlaze, 'Glaze'), plainFlip(realtimeEevee, 'EEVEE 64 samples')])],
+ ['realtime-high', 'Interactive, higher quality', 'Planned (#902), not measurable yet', 'none', 'none', 'none', 'none'],
+ ['final', 'Offline images', 'Cook native (Vulkan ray queries)', 'Needs more samples than Cycles for the same noise (see below)', countedText(nativeAt), `Cycles: ${countedText(cyclesAt)}`,
+ flipText([plainFlip(nativeAt, 'Cook native'), plainFlip(cyclesAt, 'Cycles')])],
+ ];
+ for (const [engine, label] of [['cook-webgpu', 'Cook webgpu (WGSL compute, software BVH)'], ['cook-gpu', 'Cook gpu (WebGL2)'], ['cook-cpu', 'Cook cpu (JavaScript threads)']]) {
+ const grouped = group(scene, engine);
+ if (grouped.total === 0) continue;
+ rows.push(['final', 'Offline images', label, 'Slower per sample than native', countedText(grouped), 'same Cycles', flipText([plainFlip(grouped, label.split(' (')[0])])]);
+ }
+ return rows;
+}
+
+function conditionsText() {
+ const { conditions, gate } = results;
+ const cells = results.cells;
+ const contended = cells.filter(cell => cell.contended).length;
+ const repetitionCount = Math.max(0, ...cells.map(cell => cell.repetition));
+ return `${conditions.card ?? 'card unknown'}; ${conditions.blender ?? 'Blender unknown'}; Node ${conditions.node ?? '?'}; Chrome headless on D3D11 (ANGLE); ${results.width} by ${results.height}; measured ${conditions.date ?? 'date unknown'} on scenes ${sceneIds.join(', ')}. Idle gate: wait for at most ${gate?.thresholdPercent}% utilisation on ${gate?.consecutivePolls} polls in a row (max wait ${gate?.maxWaitSeconds} s${gate?.reportOnly ? ', REPORT ONLY: the gate did not wait' : ''}); a run whose utilisation in the 3 s before it was over ${gate?.contendedAbovePercent}% is contended and left out of medians. ${contended} of ${cells.length} runs were contended. Medians with ranges over up to ${repetitionCount} repetitions, engines interleaved in a rotating order. Reference: Cycles at ${results.referenceSamples} samples. Time columns: Glaze is a frame with a pixel read (and queued back to back); EEVEE is the whole render call including scene sync, Blender's own render time beside it; Cook and Cycles are sampling time.`;
+}
+
+function escapeHtml(text) {
+ return String(text).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
+}
+
+const markdown = [];
+const html = [];
+for (const scene of sceneIds) {
+ const rows = rowsForScene(scene);
+ const statements = [['cook-native', 'Cook native'], ['cook-webgpu', 'Cook webgpu']].map(([engine, label]) => equalErrorText(equalErrorStatement(scene, engine, label))).filter(Boolean);
+ markdown.push(`### ${scene}`, '', `| ${COLUMNS.join(' | ')} |`, `| ${COLUMNS.map(() => '---').join(' | ')} |`, ...rows.map(row => `| ${row.map(cell => String(cell).replace(/\|/g, '/')).join(' | ')} |`), '', ...statements.flatMap(text => [text, '']));
+ html.push(`<h3>${escapeHtml(scene)}</h3>`, '<table>', `<thead><tr>${COLUMNS.map(column => `<th>${escapeHtml(column)}</th>`).join('')}</tr></thead>`, '<tbody>', ...rows.map(row => `<tr>${row.map(cell => `<td>${escapeHtml(cell)}</td>`).join('')}</tr>`), '</tbody>', '</table>', ...statements.map(text => `<p>${escapeHtml(text)}</p>`));
+}
+markdown.push(`Conditions. ${conditionsText()}`, '');
+html.push(`<p class="conditions">${escapeHtml(conditionsText())}</p>`);
+await writeFile(markdownPath, `${markdown.join('\n')}\n`);
+await writeFile(htmlPath, `${html.join('\n')}\n`);
+console.log(`wrote ${markdownPath}\nwrote ${htmlPath}`);
diff --git a/bench/tiers/page.js b/bench/tiers/page.js
index ca21a8d4..fd62721a 100644
--- a/bench/tiers/page.js
+++ b/bench/tiers/page.js
@@ -41,8 +41,16 @@ window.renderTierScene = async function renderTierScene(sceneId, tier, frameCoun
frameMilliseconds.push(performance.now() - started);
if (index === frameCount - 1) image = pngOf(canvas);
}
+ // Throughput: many draws queued back to back, then one pixel read. The per-frame figure above ends in a read each time, which on a fast
+ // scene is mostly the read and the driver round trip; this one is the card's work per frame, so draft against realtime shows.
+ const throughputFrames = 30;
+ const throughputStarted = performance.now();
+ for (let index = 0; index < throughputFrames; index++) renderer.render(frame);
+ finishDrawing();
+ const throughputMilliseconds = (performance.now() - throughputStarted) / throughputFrames;
const debug = context.getExtension('WEBGL_debug_renderer_info');
const result = {
+ throughputMilliseconds,
sceneId,
tier: tier.label,
canvasWidth: canvas.width,
diff --git a/bench/tiers/render-eevee.py b/bench/tiers/render-eevee.py
index 395d6425..674471c3 100644
--- a/bench/tiers/render-eevee.py
+++ b/bench/tiers/render-eevee.py
@@ -59,6 +59,26 @@ def apply_eevee_preset(scene, preset_name):
return applied, effective
+class EngineRenderTimer:
+ """Blender's own render time: from the render_pre handler to render_post, which leaves out the Python call, the scene build and the file write."""
+
+ def __init__(self):
+ self.started = None
+ self.durations = []
+
+ def on_render_pre(self, *_):
+ self.started = time.perf_counter()
+
+ def on_render_post(self, *_):
+ if self.started is not None:
+ self.durations.append(time.perf_counter() - self.started)
+ self.started = None
+
+ def install(self):
+ bpy.app.handlers.render_pre.append(self.on_render_pre)
+ bpy.app.handlers.render_post.append(self.on_render_post)
+
+
def main():
argument_list = sys.argv[sys.argv.index("--") + 1:]
scene_json_path = os.path.abspath(argument_list[0])
@@ -80,6 +100,8 @@ def main():
builder.configure_linear_exr_output(scene, width, height)
applied, effective = apply_eevee_preset(scene, preset_name)
scene_build_seconds = time.perf_counter() - build_started
+ engine_timer = EngineRenderTimer()
+ engine_timer.install()
first_started = time.perf_counter()
bpy.ops.render.render()
first_frame_seconds = time.perf_counter() - first_started
@@ -88,6 +110,8 @@ def main():
frame_started = time.perf_counter()
bpy.ops.render.render()
frame_seconds.append(time.perf_counter() - frame_started)
+ # engine_timer.durations holds the first render, then one per frame above; the handlers may not fire in every Blender build.
+ engine_frame_seconds = engine_timer.durations[1:1 + frame_count]
exr_path = f"{output_prefix}.exr"
scene.render.filepath = exr_path
bpy.ops.render.render(write_still=True)
@@ -107,11 +131,14 @@ def main():
"firstFrameSeconds": round(first_frame_seconds, 4),
"frameSeconds": [round(value, 4) for value in frame_seconds],
"medianFrameSeconds": round(statistics.median(frame_seconds), 4),
+ "engineFirstFrameSeconds": round(engine_timer.durations[0], 4) if engine_timer.durations else None,
+ "engineFrameSeconds": [round(value, 4) for value in engine_frame_seconds],
+ "engineMedianFrameSeconds": round(statistics.median(engine_frame_seconds), 4) if engine_frame_seconds else None,
"viewTransform": scene.view_settings.view_transform,
}
with open(f"{output_prefix}.json", "w", encoding="utf-8") as handle:
json.dump(report, handle, indent=2)
- print("EEVEE_TIMING " + json.dumps({key: report[key] for key in ("preset", "firstFrameSeconds", "medianFrameSeconds", "effectiveSettings")}))
+ print("EEVEE_TIMING " + json.dumps({key: report[key] for key in ("preset", "firstFrameSeconds", "medianFrameSeconds", "engineMedianFrameSeconds", "effectiveSettings")}))
main()
diff --git a/bench/tiers/run-all.mjs b/bench/tiers/run-all.mjs
new file mode 100644
index 00000000..256ca1bb
--- /dev/null
+++ b/bench/tiers/run-all.mjs
@@ -0,0 +1,248 @@
+// The tier benchmark in one command: every engine on every scene, interleaved, behind the idle gate, resumable.
+// [gpu-lock heavy] node bench/tiers/run-all.mjs <work directory> [--scenes lit-interior,spot-area,...] [--repetitions 3] [--width 960]
+// [--samples 256,1024] [--engines glaze-draft,...] [--with-cook-gpu] [--with-cook-cpu] [--reference-samples 4096]
+// [--max-wait 600] [--report-only] [--results <file>] [--no-score] [--frames 10]
+// Per repetition and scene the engines run back to back in a fixed order that rotates by one place each repetition:
+// glaze-draft, glaze-realtime, eevee-lowest, eevee-default, cook-native, cycles, cook-webgpu (then cook-gpu, cook-cpu when asked).
+// Cook native and Cycles run once per --samples count (webgpu, gpu and cpu run at the first count only: one Cook trace is limited in size).
+// Each cell goes through bench/tiers/idle-gate.mjs, so it waits for a quiet card first and records the card's load before and during the run.
+// The results file (default <work directory>/matrix-<width>.json) is rewritten after every cell, and a cell already in it is skipped, so an
+// interrupted run continues where it stopped. Scoring (FLIP against Cycles) runs at the end and adds `score` to the cells that lack it;
+// make-table.mjs turns the file into the tier table. Needs Blender 5.3, uv, Node 24 and Chrome, like the other scripts in this folder.
+import { execFile, spawn } from 'node:child_process';
+import { existsSync } from 'node:fs';
+import { mkdir, readFile, rename, writeFile } from 'node:fs/promises';
+import { constants, setPriority } from 'node:os';
+import { join, resolve } from 'node:path';
+import { promisify } from 'node:util';
+
+const runFile = promisify(execFile);
+const argumentsGiven = process.argv.slice(2);
+const flagPresent = flag => argumentsGiven.includes(flag);
+const valueAfter = (flag, fallback) => (flagPresent(flag) ? argumentsGiven[argumentsGiven.indexOf(flag) + 1] : fallback);
+const workDirectory = resolve(argumentsGiven[0] ?? (() => { throw new Error('Usage: run-all.mjs <work directory> [options], see the top of the file.'); })());
+const ALL_SCENES = 'lit-interior,spot-area,light-linking,metal,glass,outdoor-hdri';
+const sceneIds = valueAfter('--scenes', ALL_SCENES).split(',');
+const repetitions = Number(valueAfter('--repetitions', 3));
+const width = Number(valueAfter('--width', 960));
+const height = Math.round((width * 9) / 16);
+const sampleCounts = valueAfter('--samples', '256,1024').split(',').map(Number);
+const referenceSamples = Number(valueAfter('--reference-samples', 4096));
+const frameCount = Number(valueAfter('--frames', 10));
+const maxWaitSeconds = valueAfter('--max-wait', '600');
+const reportOnly = flagPresent('--report-only');
+const resultsPath = resolve(valueAfter('--results', join(workDirectory, `matrix-${width}.json`)));
+const blenderBinary = process.env.BLENDER_BIN ?? 'C:/Program Files/Blender Foundation/Blender 5.3/blender.exe';
+const skipScoring = flagPresent('--no-score');
+
+const BASE_ORDER = ['glaze-draft', 'glaze-realtime', 'eevee-lowest', 'eevee-default', 'cook-native', 'cycles', 'cook-webgpu'];
+const OPTIONAL_ORDER = [flagPresent('--with-cook-gpu') ? 'cook-gpu' : null, flagPresent('--with-cook-cpu') ? 'cook-cpu' : null].filter(Boolean);
+const engineNames = valueAfter('--engines', null)?.split(',') ?? [...BASE_ORDER, ...OPTIONAL_ORDER];
+const COUNTED_ENGINES = new Set(['cook-native', 'cycles']);
+
+// The cells of one repetition and scene, in this repetition's order: the base order rotated by (repetition - 1) places.
+function cellsOfRepetition(repetition) {
+ const expanded = engineNames.flatMap(engine => (COUNTED_ENGINES.has(engine) ? sampleCounts.map(samples => ({ engine, samples })) : [{ engine, samples: engine.startsWith('cook-') ? sampleCounts[0] : null }]));
+ const shift = (repetition - 1) % expanded.length;
+ return [...expanded.slice(shift), ...expanded.slice(0, shift)];
+}
+
+const cellKey = (scene, engine, samples, repetition) => `${scene}|${engine}|${samples ?? '-'}|r${repetition}`;
+
+async function run(command, commandArguments, logPath) {
+ return new Promise(done => {
+ const chunks = [];
+ const child = spawn(command, commandArguments, { stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, GPU_LOCK: '' } });
+ child.stdout.on('data', chunk => chunks.push(chunk));
+ child.stderr.on('data', chunk => chunks.push(chunk));
+ child.on('error', error => { chunks.push(Buffer.from(String(error))); done(127); });
+ child.on('close', async code => {
+ if (logPath) await writeFile(logPath, Buffer.concat(chunks));
+ done(code ?? 1);
+ });
+ });
+}
+
+async function readJson(path) {
+ return JSON.parse(await readFile(path, 'utf8'));
+}
+
+async function exportSceneIfMissing(scene) {
+ const sceneJson = join(workDirectory, 'scenes', `${scene}.json`);
+ if (!existsSync(sceneJson)) {
+ console.log(`export ${scene}`);
+ const exitCode = await run('node', ['bench/cycles-parity/export-scene.mjs', workDirectory, '--scene', scene], join(workDirectory, `export-${scene}.log`));
+ if (exitCode !== 0) throw new Error(`Exporting ${scene} failed, see ${workDirectory}/export-${scene}.log.`);
+ }
+ return sceneJson;
+}
+
+// The Cycles reference the scores use. Not a timed cell, so it runs outside the gate.
+async function ensureReference(scene, sceneJson) {
+ const prefix = join(workDirectory, 'reference', `${scene}-cycles-${referenceSamples}-seed1`);
+ if (!existsSync(`${prefix}.npy`)) {
+ await mkdir(join(workDirectory, 'reference'), { recursive: true });
+ console.log(`reference ${scene} at ${referenceSamples} samples`);
+ const exitCode = await run(blenderBinary, ['-b', '--factory-startup', '--python', 'bench/cycles-parity/render-cycles.py', '--', sceneJson, prefix, '--samples', String(referenceSamples), '--bounces', '8', '--seed', '1', '--width', String(width), '--height', String(height)], `${prefix}.log`);
+ if (exitCode !== 0) throw new Error(`Reference render of ${scene} failed, see ${prefix}.log.`);
+ }
+ return `${prefix}.npy`;
+}
+
+function measuredCommand(prefix, command) {
+ return ['uv', 'run', '--no-project', '--with', 'psutil', 'python', 'bench/cycles-parity/measure-gpu-run.py', `${prefix}.metrics.json`, '--', ...command];
+}
+
+// For one cell: the command to run and a function that reads its outputs into { metrics, image }.
+function planCell(scene, sceneJson, engine, samples, repetition, directory) {
+ const prefix = join(directory, 'out');
+ if (engine.startsWith('glaze-')) {
+ const tier = engine.slice('glaze-'.length);
+ return {
+ command: ['node', 'bench/tiers/run-glaze.mjs', directory, scene, '--tiers', tier, '--repetitions', '1', '--first-repetition', String(repetition), '--width', String(width), '--height', String(height), '--frames', String(frameCount)],
+ async collect() {
+ const record = (await readJson(join(directory, `glaze-${width}.json`))).runs[0];
+ return {
+ metrics: {
+ primaryMilliseconds: record.medianFrameMilliseconds, primaryName: 'frame, draw plus pixel read',
+ throughputMilliseconds: record.throughputMilliseconds, firstFrameMilliseconds: record.firstFrameMilliseconds, setupMilliseconds: record.setupMilliseconds,
+ frameMilliseconds: record.frameMilliseconds, canvas: `${record.canvasWidth}x${record.canvasHeight}`, renderTargetMegabytes: record.memoryReport.totalMegabytes,
+ cardDeltaMebibytes: record.card.deltaMebibytes, samplesDrawn: record.capabilities.samples,
+ },
+ image: { path: join(directory, `${scene}-${tier}-${width}-r${repetition}.png`), kind: 'png-display', nativeWidth: record.canvasWidth, nativeHeight: record.canvasHeight },
+ };
+ },
+ };
+ }
+ if (engine.startsWith('eevee-')) {
+ return {
+ command: measuredCommand(prefix, [blenderBinary, '-b', '--factory-startup', '--python', 'bench/tiers/render-eevee.py', '--', sceneJson, prefix, '--preset', engine.slice('eevee-'.length), '--width', String(width), '--height', String(height), '--frames', String(frameCount)]),
+ async collect() {
+ const report = await readJson(`${prefix}.json`);
+ const processMetrics = await readJson(`${prefix}.metrics.json`);
+ return {
+ metrics: {
+ primaryMilliseconds: report.medianFrameSeconds * 1000, primaryName: 'render call, scene sync included',
+ engineMilliseconds: report.engineMedianFrameSeconds === null ? null : report.engineMedianFrameSeconds * 1000,
+ firstFrameMilliseconds: report.firstFrameSeconds * 1000, frameMilliseconds: report.frameSeconds.map(seconds => seconds * 1000),
+ sceneBuildSeconds: report.sceneBuildSeconds, effectiveSettings: report.effectiveSettings, blenderVersion: report.blenderVersion,
+ processWallSeconds: processMetrics.wallSeconds, cardDeltaMebibytes: processMetrics.peakVramDeltaMegabytes,
+ },
+ image: { path: `${prefix}.npy`, kind: 'linear-npy' },
+ };
+ },
+ };
+ }
+ if (engine === 'cook-native' || engine === 'cycles') {
+ const traceCommand = engine === 'cook-native'
+ ? ['node', 'bench/cycles-parity/trace-native.mjs', sceneJson, prefix, '--samples', String(samples), '--bounces', '8', '--seed', '1', '--firefly-clamp', '1000000000', '--noise-threshold', '0']
+ : [blenderBinary, '-b', '--factory-startup', '--python', 'bench/cycles-parity/render-cycles.py', '--', sceneJson, prefix, '--samples', String(samples), '--bounces', '8', '--seed', '1', '--width', String(width), '--height', String(height)];
+ return {
+ command: measuredCommand(prefix, traceCommand),
+ async collect() {
+ const report = await readJson(`${prefix}.json`);
+ const processMetrics = await readJson(`${prefix}.metrics.json`);
+ return {
+ metrics: {
+ samples, samplingSeconds: report.samplingSeconds, totalTraceSeconds: report.totalTraceSeconds, perSampleMilliseconds: (report.samplingSeconds / samples) * 1000,
+ primaryMilliseconds: report.samplingSeconds * 1000, primaryName: 'sampling, all samples',
+ processWallSeconds: processMetrics.wallSeconds, cardDeltaMebibytes: processMetrics.peakVramDeltaMegabytes, peakRamMegabytes: processMetrics.peakRamMegabytes,
+ },
+ image: { path: `${prefix}.npy`, kind: 'linear-npy' },
+ };
+ },
+ };
+ }
+ if (engine.startsWith('cook-')) {
+ const backend = engine.slice('cook-'.length);
+ const cookPrefix = join(directory, `${scene}-cook-${backend}-s${samples}-r${repetition}`);
+ return {
+ command: ['node', 'bench/tiers/run-cook.mjs', directory, scene, '--backend', backend, '--samples', String(samples), '--repetitions', '1', '--first-repetition', String(repetition), '--width', String(width), '--height', String(height)],
+ async collect() {
+ const report = await readJson(`${cookPrefix}.json`);
+ const compileSeconds = report.report.gpuCompileSeconds ?? 0;
+ const samplingSeconds = report.report.tracedSeconds - compileSeconds;
+ return {
+ metrics: {
+ samples, samplingSeconds, totalTraceSeconds: report.report.tracedSeconds, compileSeconds, perSampleMilliseconds: (samplingSeconds / samples) * 1000,
+ primaryMilliseconds: samplingSeconds * 1000, primaryName: 'sampling, all samples', cardDeltaMebibytes: report.card.deltaMebibytes,
+ },
+ image: { path: `${cookPrefix}.npy`, kind: 'linear-npy' },
+ };
+ },
+ };
+ }
+ throw new Error(`Unknown engine ${engine}.`);
+}
+
+async function loadResults() {
+ if (existsSync(resultsPath)) return readJson(resultsPath);
+ return { schema: 1, width, height, scenes: sceneIds, referenceSamples, created: new Date().toISOString(), conditions: {}, references: {}, cells: [] };
+}
+
+async function saveResults(results) {
+ const temporaryPath = `${resultsPath}.tmp`;
+ await writeFile(temporaryPath, `${JSON.stringify(results, null, 1)}\n`);
+ await rename(temporaryPath, resultsPath);
+}
+
+async function captureConditions() {
+ const text = async (command, commandArguments) => (await runFile(command, commandArguments).catch(() => ({ stdout: '' }))).stdout.trim();
+ const card = await text('nvidia-smi', ['--query-gpu=name,driver_version,memory.total', '--format=csv,noheader']);
+ const blender = await text(blenderBinary, ['--version']);
+ return { card, blender: blender.split(/\r?\n/)[0], node: process.version, platform: process.platform, date: new Date().toISOString().slice(0, 10) };
+}
+
+setPriority(0, constants.priority.PRIORITY_BELOW_NORMAL);
+await mkdir(workDirectory, { recursive: true });
+const results = await loadResults();
+Object.assign(results.conditions, await captureConditions());
+results.gate = { maxWaitSeconds: Number(maxWaitSeconds), reportOnly, thresholdPercent: 15, consecutivePolls: 5, contendedAbovePercent: 30 };
+const sceneFiles = {};
+for (const scene of sceneIds) {
+ sceneFiles[scene] = await exportSceneIfMissing(scene);
+ results.references[scene] = { path: await ensureReference(scene, sceneFiles[scene]), samples: referenceSamples };
+}
+await saveResults(results);
+
+// A contended cell is kept in the results; --rerun-contended runs it again (when the card is quieter) and replaces it.
+const done = new Set(results.cells.filter(cell => cell.status === 'ok' && !(flagPresent('--rerun-contended') && cell.contended)).map(cell => cell.key));
+let failures = 0;
+for (let repetition = 1; repetition <= repetitions; repetition++) {
+ for (const scene of sceneIds) {
+ for (const { engine, samples } of cellsOfRepetition(repetition)) {
+ const key = cellKey(scene, engine, samples, repetition);
+ if (done.has(key)) continue;
+ const directory = join(workDirectory, 'matrix', scene, `${engine}${samples ? `-s${samples}` : ''}-r${repetition}`);
+ await mkdir(directory, { recursive: true });
+ const plan = planCell(scene, sceneFiles[scene], engine, samples, repetition, directory);
+ const gateFile = join(directory, 'gate.json');
+ const gateFlags = ['--out', gateFile, '--max-wait', maxWaitSeconds, ...(reportOnly ? ['--report-only'] : [])];
+ const started = Date.now();
+ const exitCode = await run('node', ['bench/tiers/idle-gate.mjs', ...gateFlags, '--', ...plan.command], join(directory, 'run.log'));
+ const gate = existsSync(gateFile) ? await readJson(gateFile) : null;
+ const cell = { key, scene, engine, samples, repetition, width, height, startedAt: new Date(started).toISOString(), directory, gate, contended: gate?.contended ?? true, contendedReason: gate?.contendedReason ?? null };
+ try {
+ if (exitCode !== 0) throw new Error(gate?.gate?.timedOut ? gate.gate.message : `command exited ${exitCode}, see ${directory}/run.log`);
+ Object.assign(cell, await plan.collect(), { status: 'ok' });
+ } catch (error) {
+ cell.status = 'failed';
+ cell.error = error.message;
+ failures += 1;
+ }
+ results.cells = results.cells.filter(existing => existing.key !== key);
+ results.cells.push(cell);
+ await saveResults(results);
+ const util = gate?.background ? `background ${gate.background.utilMedian}% (max ${gate.background.utilMax}%), during ${gate.run?.utilMedian}%` : 'no gate record';
+ console.log(`${cell.status.padEnd(6)} r${repetition} ${scene} ${engine}${samples ? ` s${samples}` : ''}`, cell.status === 'ok' ? `${cell.metrics.primaryMilliseconds.toFixed(1)} ms` : cell.error, '|', util, cell.contended ? '| CONTENDED' : '');
+ }
+ }
+}
+
+if (!skipScoring) {
+ console.log('scoring');
+ const exitCode = await run('uv', ['run', '--no-project', '--python', '3.12', '--with', 'flip-evaluator', '--with', 'pillow', '--with', 'numpy', 'python', '-I', 'bench/tiers/score-tiers.py', 'matrix', resultsPath], join(workDirectory, 'score.log'));
+ if (exitCode !== 0) console.log(`scoring failed, see ${workDirectory}/score.log`);
+}
+console.log(`results ${resultsPath}: ${results.cells.length} cells, ${results.cells.filter(cell => cell.contended).length} contended, ${failures} failed this run`);
+process.exit(failures > 0 ? 1 : 0);
diff --git a/bench/tiers/run-eevee.sh b/bench/tiers/run-eevee.sh
index 53600bd1..eeb0ef07 100644
--- a/bench/tiers/run-eevee.sh
+++ b/bench/tiers/run-eevee.sh
@@ -3,6 +3,7 @@
# bash bench/tiers/run-eevee.sh <work directory> <scene,scene,...> [width] [repetitions]
# Needs <work directory>/scenes/<scene>.json. Output: <work directory>/eevee/<scene>-<preset>-<width>-r<n>.{npy,json,metrics.json}
set -u
+# Every run goes through bench/tiers/idle-gate.mjs: it waits for a quiet card and writes <prefix>.gate.json (TIER_GATE_MAX_WAIT seconds; TIER_GATE_REPORT_ONLY=1 records without waiting).
WORK_DIRECTORY="$1"
SCENE_LIST="$2"
WIDTH="${3:-960}"
@@ -18,7 +19,7 @@ for repetition in $(seq 1 "$REPETITIONS"); do
for preset in $PRESETS; do
prefix="$WORK_DIRECTORY/eevee/$scene-$preset-$WIDTH-r$repetition"
[ -f "$prefix.metrics.json" ] && continue
- $GPU_LOCK uv run --no-project --with psutil python bench/cycles-parity/measure-gpu-run.py "$prefix.metrics.json" -- \
+ $GPU_LOCK node bench/tiers/idle-gate.mjs --out "$prefix.gate.json" -- uv run --no-project --with psutil python bench/cycles-parity/measure-gpu-run.py "$prefix.metrics.json" -- \
"$BLENDER_BIN" -b --factory-startup --python bench/tiers/render-eevee.py -- "$WORK_DIRECTORY/scenes/$scene.json" "$prefix" \
--preset "$preset" --width "$WIDTH" --height "$HEIGHT" --frames 10 2>&1 | grep "EEVEE_TIMING" | cut -c1-160
done
diff --git a/bench/tiers/run-final-tier.sh b/bench/tiers/run-final-tier.sh
index bf86002f..28821798 100644
--- a/bench/tiers/run-final-tier.sh
+++ b/bench/tiers/run-final-tier.sh
@@ -4,6 +4,7 @@
# Needs <work directory>/scenes/<scene>.json and the Cycles reference from render-references.sh. Light jobs go through gpu-lock one by one.
# Per repetition and sample count the engines run back to back (cpu, gpu, webgpu, native, cycles), so a drifting load hits all of them.
set -u
+# Every run goes through bench/tiers/idle-gate.mjs: it waits for a quiet card and writes <prefix>.gate.json (TIER_GATE_MAX_WAIT seconds; TIER_GATE_REPORT_ONLY=1 records without waiting).
WORK_DIRECTORY="$1"
SCENE="${2:-lit-interior}"
REPETITIONS="${3:-3}"
@@ -16,14 +17,14 @@ mkdir -p "$LADDER_DIRECTORY"
run_cook() {
local backend="$1" samples="$2" repetition="$3"
[ -f "$LADDER_DIRECTORY/$SCENE-cook-$backend-s$samples-r$repetition.json" ] && return
- $GPU_LOCK node bench/tiers/run-cook.mjs "$LADDER_DIRECTORY" "$SCENE" --backend "$backend" --samples "$samples" --repetitions 1 --first-repetition "$repetition" 2>&1 | grep -v "^$" | tail -2
+ $GPU_LOCK node bench/tiers/idle-gate.mjs --out "$LADDER_DIRECTORY/$SCENE-cook-$backend-s$samples-r$repetition.gate.json" -- node bench/tiers/run-cook.mjs "$LADDER_DIRECTORY" "$SCENE" --backend "$backend" --samples "$samples" --repetitions 1 --first-repetition "$repetition" 2>&1 | grep -v "^$" | tail -2
}
run_measured() {
local prefix="$1"
shift
[ -f "$prefix.metrics.json" ] && return
- $GPU_LOCK uv run --no-project --with psutil python bench/cycles-parity/measure-gpu-run.py "$prefix.metrics.json" -- "$@" 2>&1 | tail -1 | cut -c1-200
+ $GPU_LOCK node bench/tiers/idle-gate.mjs --out "$prefix.gate.json" -- uv run --no-project --with psutil python bench/cycles-parity/measure-gpu-run.py "$prefix.metrics.json" -- "$@" 2>&1 | tail -1 | cut -c1-200
}
run_native() {
diff --git a/bench/tiers/run-glaze.mjs b/bench/tiers/run-glaze.mjs
index e05fddb1..356e7c1e 100644
--- a/bench/tiers/run-glaze.mjs
+++ b/bench/tiers/run-glaze.mjs
@@ -9,12 +9,12 @@ import { resolve } from 'node:path';
import { promisify } from 'node:util';
import { chromium } from 'playwright';
import { createServer } from 'vite';
-import { DRAFT_TIER, REALTIME_TIER } from './tiers.mjs';
+import { DIAGNOSTIC_TIERS, DRAFT_TIER, REALTIME_TIER } from './tiers.mjs';
const runFile = promisify(execFile);
const CHROME = process.env.CHROME_PATH ?? 'C:/Program Files/Google/Chrome/Application/chrome.exe';
const SERVER_PORT = 5340;
-const TIERS = { draft: DRAFT_TIER, realtime: REALTIME_TIER };
+const TIERS = { draft: DRAFT_TIER, realtime: REALTIME_TIER, ...DIAGNOSTIC_TIERS };
const argumentsGiven = process.argv.slice(2);
const valueAfter = (flag, fallback) => (argumentsGiven.includes(flag) ? argumentsGiven[argumentsGiven.indexOf(flag) + 1] : fallback);
const outputDirectory = resolve(argumentsGiven[0]);
@@ -24,6 +24,8 @@ const height = Number(valueAfter('--height', 540));
const repetitions = Number(valueAfter('--repetitions', 3));
const frameCount = Number(valueAfter('--frames', 10));
const tierNames = valueAfter('--tiers', 'draft,realtime').split(',');
+// A driver that interleaves engines calls this once per repetition with --repetitions 1; this sets the number in the file names.
+const firstRepetition = Number(valueAfter('--first-repetition', 1));
async function cardState() {
const { stdout } = await runFile('nvidia-smi', ['--query-gpu=memory.used,utilization.gpu', '--format=csv,noheader,nounits']);
@@ -62,7 +64,7 @@ const browser = await chromium.launch({
});
const runs = [];
try {
- for (let repetition = 1; repetition <= repetitions; repetition++) {
+ for (let repetition = firstRepetition; repetition < firstRepetition + repetitions; repetition++) {
const orderedTiers = repetition % 2 ? tierNames : [...tierNames].reverse();
for (const sceneId of sceneIds) {
for (const tierName of orderedTiers) {
@@ -76,7 +78,7 @@ try {
await writeFile(`${outputDirectory}/${sceneId}-${tierName}-${width}-r${repetition}.png`, Buffer.from(result.image, 'base64'));
const { image: _image, ...record } = result;
runs.push({ repetition, width, height, tierName, card, ...record });
- console.log(sceneId, tierName, `r${repetition}`, `first ${result.firstFrameMilliseconds.toFixed(1)} ms`, `median ${result.medianFrameMilliseconds.toFixed(2)} ms`, `canvas ${result.canvasWidth}x${result.canvasHeight}`, `card +${card.deltaMebibytes} MiB`, `util ${card.utilizationPeakPercent}%`);
+ console.log(sceneId, tierName, `r${repetition}`, `first ${result.firstFrameMilliseconds.toFixed(1)} ms`, `median ${result.medianFrameMilliseconds.toFixed(2)} ms`, `throughput ${result.throughputMilliseconds.toFixed(2)} ms`, `canvas ${result.canvasWidth}x${result.canvasHeight}`, `card +${card.deltaMebibytes} MiB`, `util ${card.utilizationPeakPercent}%`);
await page.close();
}
}
diff --git a/bench/tiers/score-tiers.py b/bench/tiers/score-tiers.py
index 051f7cf3..fae5171d 100644
--- a/bench/tiers/score-tiers.py
+++ b/bench/tiers/score-tiers.py
@@ -2,6 +2,7 @@
uv run --no-project --python 3.12 --with flip-evaluator --with pillow --with numpy python -I bench/tiers/score-tiers.py final <work directory> <scene> [--width 960]
uv run --no-project --python 3.12 --with flip-evaluator --with pillow --with numpy python -I bench/tiers/score-tiers.py raster <work directory> <scene>,<scene> [--width 960]
+ uv run --no-project --python 3.12 --with flip-evaluator --with pillow --with numpy python -I bench/tiers/score-tiers.py matrix <results json from run-all.mjs>
uv run --no-project --python 3.12 --with flip-evaluator --with pillow --with numpy python -I bench/tiers/score-tiers.py settle <work directory> <scene> --bound 0.15,0.10
final: Cook (cpu, gpu, webgpu), the native tracer and Cycles at each sample count: relMSE, FLIP, time, and the equal-error and equal-time reading.
@@ -305,8 +306,54 @@ def score_settle(work_directory, scene, bounds, width):
return result
+def downscale_display(display, width, height):
+ image = Image.fromarray(np.round(display * 255).astype(np.uint8))
+ return np.asarray(image.resize((width, height), Image.BOX), dtype=np.float32) / 255
+
+
+def score_matrix(results_path):
+ """Scores every ok cell of run-all.mjs's results file that has no `score` yet, against that scene's Cycles reference, and writes the file back.
+
+ A Glaze cell is a display-space PNG at the tier's own canvas size. flip is its score after a bilinear enlargement to the full size (what a
+ viewer sees); flipNative is its score against the reference reduced to the canvas size, which leaves the enlargement out.
+ """
+ results = read_json(results_path)
+ references = {}
+ scored = 0
+ for cell in results["cells"]:
+ if cell["status"] != "ok" or "score" in cell:
+ continue
+ scene = cell["scene"]
+ if scene not in references:
+ linear = COMPARE.load_linear_rgb_image(results["references"][scene]["path"])
+ references[scene] = (linear, display_of_linear(linear))
+ reference_linear, reference_display = references[scene]
+ width, height = cell["width"], cell["height"]
+ image = cell["image"]
+ if image["kind"] == "png-display":
+ native = (image["nativeWidth"], image["nativeHeight"])
+ test_display = load_png(image["path"], width, height)
+ score = {"flip": flip_score(reference_display, test_display), "displayRelMse": display_relative_mse(reference_display, test_display)}
+ if native != (width, height):
+ raw = load_png(image["path"], native[0], native[1])
+ score["flipNative"] = flip_score(downscale_display(reference_display, native[0], native[1]), raw)
+ score["nativeSize"] = f"{native[0]}x{native[1]}"
+ else:
+ linear = COMPARE.load_linear_rgb_image(image["path"])
+ display = display_of_linear(linear.astype(np.float32))
+ comparison = COMPARE.compare_image_pair(reference_linear, linear)
+ score = {"flip": flip_score(reference_display, display), "displayRelMse": display_relative_mse(reference_display, display), "relMse": comparison["relMse"], "channelMeanRatio": comparison["channelMeanRatio"]}
+ cell["score"] = score
+ scored += 1
+ Path(results_path).write_text(json.dumps(results, indent=1) + "\n", encoding="utf-8")
+ print(f"scored {scored} cells in {results_path}")
+
+
def main():
arguments = sys.argv[1:]
+ if arguments[0] == "matrix":
+ score_matrix(arguments[1])
+ return
command, work_directory, scene_text = arguments[0], arguments[1], arguments[2]
width = int(flag_value(arguments, "--width", "960"))
if command == "final":
diff --git a/bench/tiers/tiers.mjs b/bench/tiers/tiers.mjs
index b533042e..1e16e7c1 100644
--- a/bench/tiers/tiers.mjs
+++ b/bench/tiers/tiers.mjs
@@ -22,10 +22,14 @@ export const REALTIME_TIER = {
};
// The draft tier (planned name): the cheapest frame Glaze can draw today, with the settings that exist now.
+// Before 2026-11-11 this was only pixelRatio 0.5, 1 sample and light shading, which measured the same frame time as realtime on a small
+// scene (the cost there is the pixel read). Two more existing settings now apply: an 8 bit target instead of a half float one, and a
+// coarser level of detail.
export const DRAFT_TIER = {
label: 'draft (planned name)',
rendererOptions: {
- hdr: true,
+ // 8 bit colour target: half the bytes of the half float target on every pass. The depth buffer stays, the fog needs it.
+ hdr: false,
depth: true,
// Half the render scale: a quarter of the pixels, then shown at the viewport size by the browser or by the scorer.
pixelRatio: 0.5,
@@ -34,6 +38,8 @@ export const DRAFT_TIER = {
samples: 1,
// 'light' shading drops the prefiltered environment lookup and the wide shadow filter and uses a cheaper Fresnel term.
shading: 'light',
+ // Switch to a coarser mesh when it is 4 pixels off instead of 1 (the default), which fewer triangles draw.
+ lodPixelError: 4,
},
frameEffects: {
// No ambient occlusion pass. Bloom is off unless a frame asks for it, and this one does not.
@@ -43,6 +49,15 @@ export const DRAFT_TIER = {
notAvailable: ['smaller shadow map', 'screen space effects off by name'],
};
+// Tiers that only explain the numbers, never a table row: each changes one thing between realtime and draft, to see what the score and the
+// time follow. Run them with run-glaze.mjs --tiers realtime-half,draft-full,draft-hdr,draft-before.
+export const DIAGNOSTIC_TIERS = {
+ 'realtime-half': { label: 'realtime at 0.5 render scale', rendererOptions: { ...REALTIME_TIER.rendererOptions, pixelRatio: 0.5 }, frameEffects: REALTIME_TIER.frameEffects },
+ 'draft-full': { label: 'draft with full shading', rendererOptions: { ...DRAFT_TIER.rendererOptions, shading: 'full' }, frameEffects: DRAFT_TIER.frameEffects },
+ 'draft-hdr': { label: 'draft with a half float target', rendererOptions: { ...DRAFT_TIER.rendererOptions, hdr: true }, frameEffects: DRAFT_TIER.frameEffects },
+ 'draft-before': { label: 'draft as of #931 (scale 0.5, light, hdr, default level of detail)', rendererOptions: { ...DRAFT_TIER.rendererOptions, hdr: true, lodPixelError: 1 }, frameEffects: DRAFT_TIER.frameEffects },
+};
+
// The EEVEE 5.3 counterparts, applied by render-eevee.py. EEVEE draws its default of 64 samples for a final frame; the draft row
// uses the least it can: one sample, no shadow soft rays, no raytracing.
export const EEVEE_PRESETS = {