Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Rerun Khronos and path tracer under one sun, store baked room scores (part of #169)

Nguyen Xuan Anfc60d85e0eb6
diff --git a/bench/reference/README.md b/bench/reference/README.md
index 79813348..f58a6d03 100644
--- a/bench/reference/README.md
+++ b/bench/reference/README.md
@@ -6,15 +6,63 @@ This suite scores how close a 67js render is to Blender on 23 fixed scenes. Late
Run `npm run reference`. It prints one row for each scene, and it rewrites `baseline.json` and `images/`.
-The run needs Chrome, Blender 5.2, `uv` and an NVIDIA GPU, and it took 751 s on an RTX 3060 with other jobs on the machine (2026-11-01). Use `--scene <id>` to render one scene, then `--score-only` to score the scenes in the work directory; that also rewrites `baseline.json` with only those scenes. Use `CHROME_PATH` and `BLENDER_PATH` to change the program paths.
+The run needs Chrome, Blender 5.2, `uv` and an NVIDIA GPU, and it took 751 s on an RTX 3060 with other jobs on the machine (2026-11-01). Use `--scene <id>,<id>` to render some scenes, then `--score-only` to score the scenes in the work directory; that also rewrites `baseline.json` with only those scenes. Use `CHROME_PATH` and `BLENDER_PATH` to change the program paths.
-CI does not run this suite. `tests/reference-baseline.test.js` checks only that the committed baseline is complete.
+The other sets: `--khronos` (download the five GLB files from [glTF-Sample-Assets](https://github.com/KhronosGroup/glTF-Sample-Assets) into the gitignored `artifacts/khronos/<name>/<name>.glb` first, see below), `--path-tracer --scene <ids>` (needs the Cycles renders of an earlier plain run, for example `--skip-cpu`, which renders without scoring), and `--bake <id>,<id> --skip-blender`, which draws and scores only those scenes baked, against the Cycles renders of an earlier plain run.
+
+A baked run writes `artifacts/reference/bake.json` and compares it with the committed `bake.json` (`compare-scores.mjs`): it exits with an error when a score is worse by more than 0.01 FLIP. `--update-baseline` replaces `bake.json` and `images/<id>-baked.jpg`. The bake is deterministic on one machine (two runs gave the same four digits), so 0.01 is a margin for other GPUs and drivers, and well under the 0.047 that went unnoticed in #160.
+
+CI does not run this suite. `tests/reference-baseline.test.js` checks that the committed baseline is complete, and `tests/reference-bake-regression.test.js` reads JSON only: it checks `bake.json`, tests the comparison on the numbers of #160, and compares a fresh `artifacts/reference/bake.json` with the stored one when a run left one there.
+
+## Scores now (2026-11-03)
+
+All of them are against the Blender 5.2.2 renders taken on 2026-11-03 with the sky capped at radiance 16 (one sun, see "One sun" below); the metric is [FLIP](https://github.com/NVlabs/flip) for LDR images, mean error, 0 means identical and a higher score means a larger visible difference. The first three columns are `baseline.json` (RTX 3060 as the 67js side, unchanged by this rerun), the baked column is `bake.json` (default bake of the scene, 3 bounces, `temple-hall` with its own `bakeOptions`), and the path tracer column is `path-tracer.json` (64 samples at most, 16 scenes; the tracer skips the six newest scenes). An empty cell was not measured.
+
+| Scene | 67js against Cycles | EEVEE against Cycles | 67js against EEVEE | Baked 67js against Cycles | Path tracer against Cycles |
+| --- | --- | --- | --- | --- | --- |
+| `brick-flat` | 0.1272 | 0.1098 | 0.0462 | | 0.1152 |
+| `brick-graph` | 0.1602 | 0.1221 | 0.0896 | | 0.1476 |
+| `brick-room` | 0.5102 | 0.5020 | 0.0710 | 0.1423 | 0.2727 |
+| `brushed-film` | 0.1029 | 0.0984 | 0.0855 | | 0.1524 |
+| `car-paint` | 0.1038 | 0.0844 | 0.0650 | | 0.1343 |
+| `character` | 0.0799 | 0.0622 | 0.0553 | | 0.1289 |
+| `dark-shaft` | 0.0619 | 0.2653 | 0.2808 | 0.0550 | |
+| `fog` | 0.2257 | 0.2662 | 0.1060 | | 0.5623 |
+| `fog-room` | 0.1049 | 0.3730 | 0.3917 | | |
+| `fog-valley` | 0.1855 | 0.1952 | 0.1443 | | |
+| `foliage` | 0.1023 | 0.0888 | 0.0853 | | 0.1753 |
+| `glass` | 0.1034 | 0.1009 | 0.0996 | | 0.1406 |
+| `light-linking` | 0.0998 | 0.1558 | 0.1122 | | 0.6241 |
+| `lit-interior` | 0.5055 | 0.5159 | 0.0923 | 0.1399 | 0.2167 |
+| `long-street` | 0.1407 | 0.0976 | 0.1252 | | 0.1818 |
+| `material-spheres` | 0.1293 | 0.1053 | 0.0756 | | 0.1383 |
+| `megacity-dusk` | 0.1627 | 0.1524 | 0.1022 | | |
+| `metal` | 0.1036 | 0.0916 | 0.0782 | | 0.1492 |
+| `metal-room` | 0.5076 | 0.5262 | 0.1082 | 0.1716 | |
+| `mountain-valley` | 0.2629 | 0.1879 | 0.2981 | | |
+| `outdoor-hdri` | 0.0462 | 0.0403 | 0.0296 | | 0.1242 |
+| `spot-area` | 0.1242 | 0.1935 | 0.1707 | | 0.0658 |
+| `temple-hall` | 0.0659 | 0.2341 | 0.2394 | 0.0554 | |
+
+The mean of the 23 baseline scenes is 0.1746 for 67js and 0.1986 for EEVEE. The Khronos assets (`khronos.json`, own work directory) are scored over the pixels the asset changes, since the sky fills the frame and both sides draw it alike:
+
+| Asset | Pixels covered | 67js against Cycles (asset only) | 67js against EEVEE (asset only) | EEVEE against Cycles (asset only) |
+| --- | --- | --- | --- | --- |
+| `AnisotropyBarnLamp` | 0.0732 | 0.3355 | 0.3891 | 0.3291 |
+| `AnisotropyDiscTest` | 0.1074 | 0.2864 | 0.2771 | 0.1532 |
+| `ClearCoatTest` | 0.1354 | 0.2097 | 0.1563 | 0.1303 |
+| `SheenChair` | 0.1003 | 0.3316 | 0.3098 | 0.2299 |
+| `TransmissionTest` | 0.1350 | 0.2539 | 0.2027 | 0.1590 |
+
+## History
+
+Dated notes below, kept where they explain a decision. Numbers in them are from the date they name, and those before 2026-11-01 use the sky with two suns.
## One sun (2026-11-01)
The HDRI holds its own sun: 6 of its 524,288 pixels carry 48.1 percent of the sky light, at 48 degrees of elevation. Most scenes also add a sun light. Cycles and EEVEE therefore drew two shadows of each object, and 67js drew one because its environment casts no shadow. `scenes.js` and `render-blender.py` now cap the image at radiance 16 on both sides, which removes the sun and its halo and keeps the clouds. The sky keeps 49.2 percent of its light. A scene with a sky now has only its own sun light.
-`baseline.json` and `images/` use the cap. Scores measured before this date, including the tables and notes below, `khronos.json` and `path-tracer.json`, use the sky with two suns. Scenes with a constant world did not move when the cap went in.
+`baseline.json`, `khronos.json`, `path-tracer.json`, `bake.json` and `images/` use the cap (the last three were rerun on 2026-11-03). Scores in the notes below that were measured before 2026-11-01 use the sky with two suns. Scenes with a constant world did not move when the cap went in.
## No shadow floor (2026-11-01)
@@ -333,21 +381,7 @@ Region scores, before and after this change, against Cycles (EEVEE in brackets):
## Baseline
-Measured on 2026-10-30 with an RTX 3060, SwiftShader and Blender 5.2.2. The metric is [FLIP](https://github.com/NVlabs/flip) for LDR images: 0 means identical, and a higher score means a larger visible difference.
-
-| Scene | 67js against Cycles | EEVEE against Cycles | 67js against EEVEE | Largest visible gap against Cycles |
-| --- | --- | --- | --- | --- |
-| `outdoor-hdri` | 0.0676 | 0.0383 | 0.0520 | Soft shadow contact, and a darker ball base |
-| `character` | 0.0995 | 0.0740 | 0.0833 | No contact shadow from the environment light |
-| `glass` | 0.1450 | 0.1150 | 0.1469 | The sphere does not bend the view or reflect the sky |
-| `foliage` | 0.1594 | 0.0783 | 0.1386 | Bright leaves from a built-in translucency trick |
-| `material-spheres` | 0.1598 | 0.1006 | 0.1196 | No reflection between objects, and light shadows |
-| `metal` | 0.1868 | 0.0934 | 0.1565 | No reflection between objects |
-| `fog` | 0.2612 | 0.2662 | 0.0542 | No multiple scattering |
-| `lit-interior` | 0.4997 | 0.5159 | 0.1204 | No indirect light, and point lights cast no shadows |
-| `spot-area` | 0.1242 | 0.1935 | 0.1707 | The sphere is lit more weakly at its shaded edge, and the wall has no bounce light |
-| `brick-graph` | 0.2269 | 0.1440 | 0.1480 | Bump is stronger than Blender's Bump node (measured 2026-11-01, see below; an earlier note said weaker), and Blender's noise is not 67js noise, so brick tone differs |
-| `brushed-film` | 0.1613 | 0.0939 | 0.1601 | The film colours on the dark sphere are close to Cycles, but its reflections are softer; the brushed highlight is pinched less at the pole, and there is no ground bounce under the spheres |
+Taken on 2026-10-30, under the sky with two suns, and replaced by "Scores now". The notes below explain decisions; their numbers are from the dates they name.
Rows are the baseline taken before shadows for point lights. The `lit-interior` scene now sets `castShadow` on its three point lights. With shadows, 67js scores 0.5055 against Cycles (0.4997 before, so 0.0058 worse) and 0.0923 against EEVEE (0.1204 before, better): the shadow shape now matches, but Cycles fills its shadows with indirect light and 67js leaves them dark. The committed `lit-interior` row stays at the earlier numbers. `spot-area` is new with shadows, a spot light and an area light.
@@ -378,15 +412,7 @@ EEVEE against Cycles is a yardstick: it shows what a good rasterizer scores. In
- **Display:** Blender writes linear pixels, and `score.py` applies the tone curve and gamma from `src/post.js`. Both sides then use one display transform.
- **Lighting tricks:** the 67js fill lights are off. All light comes from the sun, the environment and point lights.
-Khronos sample assets (issue #51): `npm run reference -- --khronos` draws files from [KhronosGroup/glTF-Sample-Assets](https://github.com/KhronosGroup/glTF-Sample-Assets) that you download by hand into `artifacts/khronos/<name>/<name>.glb` (gitignored; nothing is committed). Blender imports the same GLB with its own importer, so Cycles is the reference for the file and not for our reading of it. The run uses its own work directory and writes `khronos.json` and `images/khronos-<name>.jpg`; `baseline.json` stays as it is. The sky fills most of each frame and both sides draw it alike, so the whole-frame FLIP score is about 0.05 for every asset and says little. The `asset-` columns score only the pixels the asset changes (a second render of the empty sky decides which, threshold 0.03). Licences, checked in each model's `LICENSE.md`: ClearCoatTest and AnisotropyBarnLamp are CC BY 4.0 (Analytical Graphics, Inc. and Wayfair, LLC), TransmissionTest, AnisotropyDiscTest and SheenChair are CC0 1.0 (SheenChair's model files are CC0; only its metadata files are CC BY 4.0). Measured 2026-11-01 on the RTX 3060 (SwiftShader within 0.0004 on the whole frame):
-
-| Asset | Pixels covered | 67js against Cycles (asset only) | 67js against EEVEE | EEVEE against Cycles | What differs |
-| --- | --- | --- | --- | --- | --- |
-| `ClearCoatTest` | 0.1353 | 0.2306 (was 0.2344) | 0.1808 | 0.1335 | The coat, coat roughness and coat normal textures are read; most of the gap is outside the coat |
-| `TransmissionTest` | 0.1356 | 0.2708 | 0.2371 | 0.1656 | Refraction through the front surface only, no roughness-blurred background |
-| `AnisotropyDiscTest` | 0.1074 | 0.2808 (was 0.3481) | 0.2772 | 0.1534 | `anisotropyTexture` is read: the disc pattern now draws, but the tiles are darker than Cycles; 0.2773 with anisotropy switched off |
-| `AnisotropyBarnLamp` | 0.0733 | 0.3584 (was 0.4758) | 0.4318 | 0.3765 | `anisotropyTexture` and the coat normal texture are read, and it now scores better than anisotropy switched off (0.3945) |
-| `SheenChair` | 0.1007 | 0.3745 | 0.3374 | 0.2384 | New. Cycles shows a bright red velvet rim along the top of the backrest and a more saturated orange; 67js draws the same shape and tufting but flatter, paler and with only a faint rim, so the error is a band along that edge and the seat front |
+Khronos sample assets (issue #51): `npm run reference -- --khronos` draws files from [KhronosGroup/glTF-Sample-Assets](https://github.com/KhronosGroup/glTF-Sample-Assets) that you download by hand into `artifacts/khronos/<name>/<name>.glb` (gitignored; nothing is committed). Blender imports the same GLB with its own importer, so Cycles is the reference for the file and not for our reading of it. The run uses its own work directory and writes `khronos.json` and `images/khronos-<name>.jpg`; `baseline.json` stays as it is. The sky fills most of each frame and both sides draw it alike, so the whole-frame FLIP score is about 0.05 for every asset and says little. The `asset-` columns score only the pixels the asset changes (a second render of the empty sky decides which, threshold 0.03). Licences, checked in each model's `LICENSE.md`: ClearCoatTest and AnisotropyBarnLamp are CC BY 4.0 (Analytical Graphics, Inc. and Wayfair, LLC), TransmissionTest, AnisotropyDiscTest and SheenChair are CC0 1.0 (SheenChair's model files are CC0; only its metadata files are CC BY 4.0). The scores are in "Scores now". The `AnisotropyBarnLamp` and `AnisotropyDiscTest` rows read `anisotropyTexture` since 2026-11-01 (0.4758 to 0.3584 and 0.3481 to 0.2808 then, under the sky with two suns).
Reading `anisotropyTexture` (strength in blue, direction in red and green) fixed the shade: the factor alone spread a highlight over everything, and it scored worse than no anisotropy. The disc test is still level with anisotropy switched off, so the remaining gap there is not the texture. `TransmissionTest` is unchanged at 0.2708: its `transmissionTexture` is not read, because transmission draws in the liquid pass, which has no feature code, and the scores do not show that the other slots (iridescence, specular) matter. `SheenChair` is scored since the importer accepts a normal map with its own `KHR_texture_transform` scale and offset: the fabric tiles its base color 7 times and its normal map 2 times, so the importer bakes the base color transform into the UVs and the shader applies the normal map's remainder (`normalMapTransform`, outside core; core grew by 25 bytes gzip). The score does not say that the transform is right: the visible gap is the sheen rim, and why it is weaker was not investigated. Both Blender's engines disagree on the lamp by 0.3761, so part of that gap is Blender's own anisotropy and clear coat. Two importer bugs the assets found are fixed: `KHR_xmp` and `KHR_xmp_json_ld` (metadata) stopped the whole import, and `KHR_materials_variants` is now accepted with a warning and draws the default material.
@@ -504,38 +530,13 @@ Build time, on this machine with other jobs running: a 5 km valley terrain adds
- `fog` has no noise on either side. EEVEE and 67js approximate scattering, and Cycles follows 8 volume bounces.
- Sample counts are fixed: 256 for Cycles with denoising, 64 for EEVEE and 4 MSAA samples for 67js.
-Path tracer (issue #57): `npm run reference -- --path-tracer [--samples 32]` draws each scene with `addons/path-tracer` (CPU, 960 by 540, four threads, below-normal priority) and scores it against the Cycles renders of an earlier run into `path-tracer.json`; `baseline.json` and `images/` stay as they are. Measured 2026-10-31 at 32 samples with the tracer's own denoiser, on a Ryzen 5 5600X that another session was also loading (seconds are wall time and are inflated by that):
+Path tracer (issue #57): `npm run reference -- --path-tracer [--samples 32]` draws each scene with `addons/path-tracer` (CPU, 960 by 540, four threads, below-normal priority) and scores it against the Cycles renders of an earlier run into `path-tracer.json`; `baseline.json` and `images/` stay as they are. The first pass (2026-10-31) drew 32 samples with the tracer's own denoiser; the second pass is below.
-| Scene | 67js rasterizer | Path tracer | Seconds | Not drawn by the tracer |
-| --- | --- | --- | --- | --- |
-| `brick-graph` | 0.2269 | 0.1538 | 34.2 | normal map (bump) |
-| `car-paint` | 0.1521 | 0.1372 | 44.5 | |
-| `character` | 0.0995 | 0.1143 | 35.5 | |
-| `fog` | 0.2612 | 0.5622 | 33.5 | volume fog |
-| `foliage` | 0.1594 | 0.1597 | 126.1 | leaf translucency |
-| `glass` | 0.1326 | 0.1237 | 50.3 | |
-| `lit-interior` | 0.5055 | 0.1993 | 238.0 | |
-| `material-spheres` | 0.1598 | 0.1215 | 66.8 | |
-| `metal` | 0.1868 | 0.1467 | 43.2 | |
-| `outdoor-hdri` | 0.0676 | 0.1036 | 25.1 | |
-| `spot-area` | 0.1242 | 0.0639 | 71.6 | |
-
-The mean against Cycles is 0.171 for the path tracer and 0.189 for the rasterizer (the light bake gets `lit-interior` to 0.1341 with 3 bounces (0.1444 with 2), better than the tracer's 0.1993). The tracer is worse on `outdoor-hdri` and `character`: in the images the floor is darker than in Cycles (the cause is not found), and `fog` shows the missing volume. The denoiser blurs some soft shadows. Cycles used 256 samples. See `docs/path-tracer.md` for what the tracer draws and skips.
-
-Path tracer, second pass (issue #129, measured 2026-11-01, 64 samples at most, `noiseThreshold` 0.03, four threads, 960 by 540; seconds are wall time and include scene setup). Scores against Cycles, all 17 scenes in `path-tracer.json`:
-
-| Scene | Score | Seconds | | Scene | Score | Seconds |
-| --- | --- | --- | --- | --- | --- | --- |
-| `brick-flat` | 0.1117 | 24.2 | | `lit-interior` | 0.2214 | 106.7 |
-| `brick-graph` | 0.1373 | 26.8 | | `long-street` | 0.1838 | 50.2 |
-| `brick-room` | 0.2743 | 125.9 | | `material-spheres` | 0.1187 | 52.3 |
-| `brushed-film` | 0.1462 | 47.2 | | `metal` | 0.1435 | 43.5 |
-| `car-paint` | 0.1337 | 46.5 | | `outdoor-hdri` | 0.1016 | 24.8 |
-| `character` | 0.1120 | 35.1 | | `spot-area` | 0.0900 | 21.3 |
-| `fog` | 0.5628 | 26.4 | | `light-linking` | 0.6191 | 33.9 |
-| `foliage` | 0.1577 | 69.5 | | `glass` | 0.1220 | 41.2 |
-
-The mean over the 11 scenes of the first table is 0.1728 (0.171 at 32 samples before). `light-linking` scores 0.6191 because the tracer does not trace light linking (every light lights every part).
+The tracer is worse than the rasterizer on `outdoor-hdri` and `character`: in the images the floor is darker than in Cycles (the cause is not found), and `fog` shows the missing volume. The denoiser blurs some soft shadows. Cycles used 256 samples. See `docs/path-tracer.md` for what the tracer draws and skips.
+
+Path tracer, second pass (issue #129, 2026-11-01): 64 samples at most, `noiseThreshold` 0.03, four threads, 960 by 540; seconds are wall time and include scene setup. The scores are in "Scores now".
+
+`light-linking` scores 0.6241 because the tracer does not trace light linking (every light lights every part).
Before and after on four scenes, 64 samples: every sample drawn without the box-projection and shadow-ray changes (the state of `main`), then the defaults of this change. `brick-graph` 0.1370 then 0.1373, `brick-room` 0.2794 then 0.2743 (seconds 121.2 then 126.7, the box-projected walls now read their projected UVs), `character` 0.1116 then 0.1120, `outdoor-hdri` 0.1014 then 0.1016. `brick-graph` is a UV-mapped wall, so it already drew its bump and nothing in this change touches it: against the 0.1538 of the table above (32 samples) it improved, against the same tracer at 64 samples it did not move (0.0003 worse, inside the noise). `brick-room` is the scene with box-projected materials; its normal and roughness maps used to be ignored.
diff --git a/bench/reference/bake.json b/bench/reference/bake.json
new file mode 100644
index 00000000..2b07695c
--- /dev/null
+++ b/bench/reference/bake.json
@@ -0,0 +1,42 @@
+{
+ "metric": "FLIP (LDR), mean error of the baked GPU render against Cycles, lower is better",
+ "blenderVersion": "5.2.2 LTS",
+ "graphics": {
+ "gpu": "ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 (0x00002487) Direct3D11 vs_5_0 ps_5_0, D3D11)"
+ },
+ "scores": {
+ "lit-interior": {
+ "gpu-vs-cycles": 0.1399,
+ "gpu-vs-eevee": 0.4677,
+ "eevee-vs-cycles": 0.5159
+ },
+ "brick-room": {
+ "gpu-vs-cycles": 0.1423,
+ "gpu-vs-eevee": 0.4514,
+ "eevee-vs-cycles": 0.502
+ },
+ "dark-shaft": {
+ "gpu-vs-cycles": 0.055,
+ "gpu-vs-eevee": 0.2612,
+ "eevee-vs-cycles": 0.2654,
+ "region-pixels": 0.0989,
+ "region-gpu-vs-cycles": 0.1138,
+ "region-gpu-vs-eevee": 0.2653,
+ "region-eevee-vs-cycles": 0.2645
+ },
+ "temple-hall": {
+ "gpu-vs-cycles": 0.0554,
+ "gpu-vs-eevee": 0.244,
+ "eevee-vs-cycles": 0.2341,
+ "region-pixels": 0.0961,
+ "region-gpu-vs-cycles": 0.2233,
+ "region-gpu-vs-eevee": 0.3013,
+ "region-eevee-vs-cycles": 0.2453
+ },
+ "metal-room": {
+ "gpu-vs-cycles": 0.1716,
+ "gpu-vs-eevee": 0.4623,
+ "eevee-vs-cycles": 0.5262
+ }
+ }
+}
diff --git a/bench/reference/compare-scores.mjs b/bench/reference/compare-scores.mjs
new file mode 100644
index 00000000..728811f2
--- /dev/null
+++ b/bench/reference/compare-scores.mjs
@@ -0,0 +1,40 @@
+// Compares a fresh results file with a stored one and lists every score that got worse by more than a margin.
+// node bench/reference/compare-scores.mjs <fresh.json> [<stored.json>] [--margin 0.01]
+// The stored file defaults to bench/reference/bake.json. Lower is better for every score except the "-pixels" counts, which are skipped.
+// A scene or score that the fresh file does not hold is not compared, so a run of a few scenes checks only those.
+import { readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+
+export const DEFAULT_MARGIN = 0.01;
+
+export function findRegressions(stored, fresh, margin = DEFAULT_MARGIN) {
+ const regressions = [];
+ for (const [scene, storedScores] of Object.entries(stored.scores)) {
+ const freshScores = fresh.scores[scene];
+ if (!freshScores) continue;
+ for (const [name, storedValue] of Object.entries(storedScores)) {
+ const freshValue = freshScores[name];
+ if (name.endsWith('-pixels') || typeof freshValue !== 'number') continue;
+ if (freshValue > storedValue + margin) {
+ regressions.push({ scene, name, stored: storedValue, fresh: freshValue });
+ }
+ }
+ }
+ return regressions;
+}
+
+export function describeRegressions(regressions, margin = DEFAULT_MARGIN) {
+ return regressions
+ .map(({ scene, name, stored, fresh }) => `${scene} ${name}: ${stored} stored, ${fresh} fresh (+${(fresh - stored).toFixed(4)}, margin ${margin})`)
+ .join('\n');
+}
+
+if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
+ const marginIndex = process.argv.indexOf('--margin');
+ const margin = marginIndex >= 0 ? Number(process.argv[marginIndex + 1]) : DEFAULT_MARGIN;
+ const positional = process.argv.slice(2).filter((argument, index, all) => !argument.startsWith('--') && all[index - 1] !== '--margin');
+ const [freshPath, storedPath = fileURLToPath(new URL('bake.json', import.meta.url))] = positional;
+ const regressions = findRegressions(JSON.parse(readFileSync(storedPath, 'utf8')), JSON.parse(readFileSync(freshPath, 'utf8')), margin);
+ console.log(regressions.length ? `Regressed:\n${describeRegressions(regressions, margin)}` : 'No stored score got worse by more than the margin.');
+ process.exitCode = regressions.length ? 1 : 0;
+}
diff --git a/bench/reference/images/brick-room-baked.jpg b/bench/reference/images/brick-room-baked.jpg
new file mode 100644
index 00000000..bb5931b1
Binary files /dev/null and b/bench/reference/images/brick-room-baked.jpg differ
diff --git a/bench/reference/images/dark-shaft-baked.jpg b/bench/reference/images/dark-shaft-baked.jpg
index e9c52306..3c113cca 100644
Binary files a/bench/reference/images/dark-shaft-baked.jpg and b/bench/reference/images/dark-shaft-baked.jpg differ
diff --git a/bench/reference/images/khronos-AnisotropyBarnLamp.jpg b/bench/reference/images/khronos-AnisotropyBarnLamp.jpg
index f561759e..2becdeae 100644
Binary files a/bench/reference/images/khronos-AnisotropyBarnLamp.jpg and b/bench/reference/images/khronos-AnisotropyBarnLamp.jpg differ
diff --git a/bench/reference/images/khronos-AnisotropyDiscTest.jpg b/bench/reference/images/khronos-AnisotropyDiscTest.jpg
index e9fd8d03..bb497464 100644
Binary files a/bench/reference/images/khronos-AnisotropyDiscTest.jpg and b/bench/reference/images/khronos-AnisotropyDiscTest.jpg differ
diff --git a/bench/reference/images/khronos-ClearCoatTest.jpg b/bench/reference/images/khronos-ClearCoatTest.jpg
index 828b506b..9c1174fe 100644
Binary files a/bench/reference/images/khronos-ClearCoatTest.jpg and b/bench/reference/images/khronos-ClearCoatTest.jpg differ
diff --git a/bench/reference/images/khronos-SheenChair.jpg b/bench/reference/images/khronos-SheenChair.jpg
index 033d302e..93915ed2 100644
Binary files a/bench/reference/images/khronos-SheenChair.jpg and b/bench/reference/images/khronos-SheenChair.jpg differ
diff --git a/bench/reference/images/khronos-TransmissionTest.jpg b/bench/reference/images/khronos-TransmissionTest.jpg
index 2ff11792..7edb1278 100644
Binary files a/bench/reference/images/khronos-TransmissionTest.jpg and b/bench/reference/images/khronos-TransmissionTest.jpg differ
diff --git a/bench/reference/images/lit-interior-baked.jpg b/bench/reference/images/lit-interior-baked.jpg
new file mode 100644
index 00000000..1748b827
Binary files /dev/null and b/bench/reference/images/lit-interior-baked.jpg differ
diff --git a/bench/reference/images/metal-room-baked.jpg b/bench/reference/images/metal-room-baked.jpg
new file mode 100644
index 00000000..48d02ea2
Binary files /dev/null and b/bench/reference/images/metal-room-baked.jpg differ
diff --git a/bench/reference/images/temple-hall-baked.jpg b/bench/reference/images/temple-hall-baked.jpg
index b8126967..9e7ba8cb 100644
Binary files a/bench/reference/images/temple-hall-baked.jpg and b/bench/reference/images/temple-hall-baked.jpg differ
diff --git a/bench/reference/khronos.json b/bench/reference/khronos.json
index 34ec6d0e..50f3e6eb 100644
--- a/bench/reference/khronos.json
+++ b/bench/reference/khronos.json
@@ -3,24 +3,24 @@
"blenderVersion": "5.2.2 LTS",
"seconds": {
"AnisotropyBarnLamp": {
- "cycles": 23.4,
- "eevee": 29.1
+ "cycles": 2.5,
+ "eevee": 1.2
},
"AnisotropyDiscTest": {
- "cycles": 24.1,
- "eevee": 3.7
+ "cycles": 2.7,
+ "eevee": 0.2
},
"ClearCoatTest": {
- "cycles": 13.7,
- "eevee": 10.2
+ "cycles": 2.8,
+ "eevee": 0.3
},
"SheenChair": {
- "cycles": 13.4,
- "eevee": 36.2
+ "cycles": 3.0,
+ "eevee": 0.4
},
"TransmissionTest": {
- "cycles": 22.2,
- "eevee": 10.0
+ "cycles": 3.4,
+ "eevee": 0.4
}
},
"graphics": {
@@ -29,64 +29,64 @@
},
"scores": {
"AnisotropyBarnLamp": {
- "gpu-vs-cycles": 0.041,
- "gpu-vs-eevee": 0.0462,
- "cpu-vs-cycles": 0.0408,
- "cpu-vs-eevee": 0.0458,
- "eevee-vs-cycles": 0.0398,
- "gpu-vs-cpu": 0.0065,
- "asset-pixels": 0.0733,
- "asset-gpu-vs-cycles": 0.3584,
- "asset-gpu-vs-eevee": 0.4318,
- "asset-eevee-vs-cycles": 0.3765
+ "gpu-vs-cycles": 0.0388,
+ "gpu-vs-eevee": 0.0422,
+ "cpu-vs-cycles": 0.0386,
+ "cpu-vs-eevee": 0.0419,
+ "eevee-vs-cycles": 0.035,
+ "gpu-vs-cpu": 0.0064,
+ "asset-pixels": 0.0732,
+ "asset-gpu-vs-cycles": 0.3355,
+ "asset-gpu-vs-eevee": 0.3891,
+ "asset-eevee-vs-cycles": 0.3291
},
"AnisotropyDiscTest": {
- "gpu-vs-cycles": 0.0445,
- "gpu-vs-eevee": 0.0419,
- "cpu-vs-cycles": 0.0443,
- "cpu-vs-eevee": 0.0417,
+ "gpu-vs-cycles": 0.0451,
+ "gpu-vs-eevee": 0.042,
+ "cpu-vs-cycles": 0.0449,
+ "cpu-vs-eevee": 0.0418,
"eevee-vs-cycles": 0.024,
"gpu-vs-cpu": 0.0062,
"asset-pixels": 0.1074,
- "asset-gpu-vs-cycles": 0.2808,
- "asset-gpu-vs-eevee": 0.2772,
- "asset-eevee-vs-cycles": 0.1534
+ "asset-gpu-vs-cycles": 0.2864,
+ "asset-gpu-vs-eevee": 0.2771,
+ "asset-eevee-vs-cycles": 0.1532
},
"ClearCoatTest": {
- "gpu-vs-cycles": 0.0448,
- "gpu-vs-eevee": 0.0333,
- "cpu-vs-cycles": 0.0445,
- "cpu-vs-eevee": 0.0329,
- "eevee-vs-cycles": 0.0271,
- "gpu-vs-cpu": 0.0076,
- "asset-pixels": 0.1353,
- "asset-gpu-vs-cycles": 0.2306,
- "asset-gpu-vs-eevee": 0.1808,
- "asset-eevee-vs-cycles": 0.1335
+ "gpu-vs-cycles": 0.0415,
+ "gpu-vs-eevee": 0.0298,
+ "cpu-vs-cycles": 0.0412,
+ "cpu-vs-eevee": 0.0295,
+ "eevee-vs-cycles": 0.0267,
+ "gpu-vs-cpu": 0.0073,
+ "asset-pixels": 0.1354,
+ "asset-gpu-vs-cycles": 0.2097,
+ "asset-gpu-vs-eevee": 0.1563,
+ "asset-eevee-vs-cycles": 0.1303
},
"SheenChair": {
- "gpu-vs-cycles": 0.051,
- "gpu-vs-eevee": 0.0438,
- "cpu-vs-cycles": 0.0507,
- "cpu-vs-eevee": 0.0435,
- "eevee-vs-cycles": 0.0309,
- "gpu-vs-cpu": 0.0065,
- "asset-pixels": 0.1007,
- "asset-gpu-vs-cycles": 0.3745,
- "asset-gpu-vs-eevee": 0.3374,
- "asset-eevee-vs-cycles": 0.2384
+ "gpu-vs-cycles": 0.0444,
+ "gpu-vs-eevee": 0.0398,
+ "cpu-vs-cycles": 0.0442,
+ "cpu-vs-eevee": 0.0395,
+ "eevee-vs-cycles": 0.0301,
+ "gpu-vs-cpu": 0.0063,
+ "asset-pixels": 0.1003,
+ "asset-gpu-vs-cycles": 0.3316,
+ "asset-gpu-vs-eevee": 0.3098,
+ "asset-eevee-vs-cycles": 0.2299
},
"TransmissionTest": {
- "gpu-vs-cycles": 0.0485,
- "gpu-vs-eevee": 0.042,
- "cpu-vs-cycles": 0.0483,
- "cpu-vs-eevee": 0.0417,
- "eevee-vs-cycles": 0.0286,
- "gpu-vs-cpu": 0.0063,
- "asset-pixels": 0.1356,
- "asset-gpu-vs-cycles": 0.2708,
- "asset-gpu-vs-eevee": 0.2371,
- "asset-eevee-vs-cycles": 0.1656
+ "gpu-vs-cycles": 0.0449,
+ "gpu-vs-eevee": 0.0366,
+ "cpu-vs-cycles": 0.0447,
+ "cpu-vs-eevee": 0.0363,
+ "eevee-vs-cycles": 0.0275,
+ "gpu-vs-cpu": 0.0064,
+ "asset-pixels": 0.135,
+ "asset-gpu-vs-cycles": 0.2539,
+ "asset-gpu-vs-eevee": 0.2027,
+ "asset-eevee-vs-cycles": 0.159
}
}
}
diff --git a/bench/reference/path-tracer.json b/bench/reference/path-tracer.json
index cf4ed3ed..0a3068c3 100644
--- a/bench/reference/path-tracer.json
+++ b/bench/reference/path-tracer.json
@@ -2,121 +2,121 @@
"metric": "FLIP (LDR), mean error against Cycles, lower is better",
"scores": {
"brick-flat": {
- "pathtracer-vs-cycles": 0.1117,
- "seconds": 24.2,
+ "pathtracer-vs-cycles": 0.1152,
+ "seconds": 23,
"samples": 64,
- "averageSamples": 45.7,
+ "averageSamples": 64,
"ignored": []
},
"brick-graph": {
- "pathtracer-vs-cycles": 0.1373,
- "seconds": 26.8,
+ "pathtracer-vs-cycles": 0.1476,
+ "seconds": 23.6,
"samples": 64,
- "averageSamples": 45.6,
+ "averageSamples": 64,
"ignored": []
},
"brick-room": {
- "pathtracer-vs-cycles": 0.2743,
- "seconds": 125.9,
+ "pathtracer-vs-cycles": 0.2727,
+ "seconds": 95.7,
"samples": 64,
- "averageSamples": 61.9,
+ "averageSamples": 64,
"ignored": []
},
"brushed-film": {
- "pathtracer-vs-cycles": 0.1462,
- "seconds": 47.2,
+ "pathtracer-vs-cycles": 0.1524,
+ "seconds": 35.1,
"samples": 64,
- "averageSamples": 47.4,
+ "averageSamples": 64,
"ignored": []
},
"car-paint": {
- "pathtracer-vs-cycles": 0.1337,
- "seconds": 46.5,
+ "pathtracer-vs-cycles": 0.1343,
+ "seconds": 35,
"samples": 64,
- "averageSamples": 47.7,
+ "averageSamples": 64,
"ignored": []
},
"character": {
- "pathtracer-vs-cycles": 0.112,
- "seconds": 35.1,
+ "pathtracer-vs-cycles": 0.1289,
+ "seconds": 27.6,
"samples": 64,
- "averageSamples": 43,
+ "averageSamples": 64,
"ignored": [
"gltf materials are traced as plain surfaces."
]
},
"fog": {
- "pathtracer-vs-cycles": 0.5628,
- "seconds": 26.4,
+ "pathtracer-vs-cycles": 0.5623,
+ "seconds": 24.5,
"samples": 64,
- "averageSamples": 38,
+ "averageSamples": 64,
"ignored": [
"volumeFog is not traced."
]
},
"foliage": {
- "pathtracer-vs-cycles": 0.1577,
- "seconds": 69.5,
+ "pathtracer-vs-cycles": 0.1753,
+ "seconds": 60.9,
"samples": 64,
- "averageSamples": 38.8,
+ "averageSamples": 64,
"ignored": [
"foliage materials are traced as plain surfaces."
]
},
"glass": {
- "pathtracer-vs-cycles": 0.122,
- "seconds": 41.2,
+ "pathtracer-vs-cycles": 0.1406,
+ "seconds": 30.4,
"samples": 64,
- "averageSamples": 47.7,
+ "averageSamples": 64,
"ignored": []
},
"light-linking": {
- "pathtracer-vs-cycles": 0.6191,
- "seconds": 33.9,
+ "pathtracer-vs-cycles": 0.6241,
+ "seconds": 34.6,
"samples": 64,
- "averageSamples": 36.1,
+ "averageSamples": 64,
"ignored": []
},
"lit-interior": {
- "pathtracer-vs-cycles": 0.2214,
- "seconds": 106.7,
+ "pathtracer-vs-cycles": 0.2167,
+ "seconds": 75.3,
"samples": 64,
- "averageSamples": 62.2,
+ "averageSamples": 64,
"ignored": []
},
"long-street": {
- "pathtracer-vs-cycles": 0.1838,
- "seconds": 50.2,
+ "pathtracer-vs-cycles": 0.1818,
+ "seconds": 42.1,
"samples": 64,
- "averageSamples": 48.7,
+ "averageSamples": 64,
"ignored": []
},
"material-spheres": {
- "pathtracer-vs-cycles": 0.1187,
- "seconds": 52.3,
+ "pathtracer-vs-cycles": 0.1383,
+ "seconds": 43.9,
"samples": 64,
- "averageSamples": 59.5,
+ "averageSamples": 64,
"ignored": []
},
"metal": {
- "pathtracer-vs-cycles": 0.1435,
- "seconds": 43.5,
+ "pathtracer-vs-cycles": 0.1492,
+ "seconds": 31.4,
"samples": 64,
- "averageSamples": 48.8,
+ "averageSamples": 64,
"ignored": []
},
"outdoor-hdri": {
- "pathtracer-vs-cycles": 0.1016,
- "seconds": 24.8,
+ "pathtracer-vs-cycles": 0.1242,
+ "seconds": 18.9,
"samples": 64,
- "averageSamples": 40.3,
+ "averageSamples": 64,
"ignored": []
},
"spot-area": {
- "pathtracer-vs-cycles": 0.09,
- "seconds": 21.3,
+ "pathtracer-vs-cycles": 0.0658,
+ "seconds": 26,
"samples": 64,
- "averageSamples": 27.8,
+ "averageSamples": 64,
"ignored": []
}
}
diff --git a/bench/reference/run.mjs b/bench/reference/run.mjs
index 5ac341b8..42053c4f 100644
--- a/bench/reference/run.mjs
+++ b/bench/reference/run.mjs
@@ -42,7 +42,7 @@ const argumentsGiven = process.argv.slice(2);
const scoreOnly = argumentsGiven.includes('--score-only');
const skipBlender = argumentsGiven.includes('--skip-blender');
// --skip-cpu leaves out the SwiftShader render (very slow for the 9 million triangle scenes) and the scoring, which needs it.
-const skipCpu = argumentsGiven.includes('--skip-cpu');
+const skipCpuFlag = argumentsGiven.includes('--skip-cpu');
const sceneFlag = argumentsGiven.indexOf('--scene');
const occlusionFlag = argumentsGiven.indexOf('--ambient-occlusion');
const ambientOcclusion = occlusionFlag >= 0 ? Number(argumentsGiven[occlusionFlag + 1]) : 0;
@@ -76,7 +76,18 @@ const samplesFlag = argumentsGiven.indexOf('--samples');
const pathTracerSamples = samplesFlag >= 0 ? Number(argumentsGiven[samplesFlag + 1]) : 64;
const noiseThresholdFlag = argumentsGiven.indexOf('--noise-threshold');
const pathTracerNoiseThreshold = noiseThresholdFlag >= 0 ? Number(argumentsGiven[noiseThresholdFlag + 1]) : undefined;
-const onlyScene = sceneFlag >= 0 ? argumentsGiven[sceneFlag + 1] : null;
+// --scene takes one id or several separated by commas.
+const sceneIdsGiven = sceneFlag >= 0 ? argumentsGiven[sceneFlag + 1].split(',') : null;
+// --bake <id>,<id> alone is a baked-score run: it draws and scores only those scenes, with the GPU, against the Cycles and EEVEE renders
+// of an earlier run (use --skip-blender), and writes bake.json. With an option that changes the bake (--sky-only, --bake-preset, --probes,
+// --ambient, --ambient-occlusion, --cascades) the scores go to the work directory and the committed files stay as they are.
+// The fresh bake.json lands in the work directory and is checked against the committed one (compare-scores.mjs); --update-baseline replaces
+// the committed bake.json and images/<id>-baked.jpg with it.
+const bakeScoreRun = bakedSceneIds !== null && !sceneSet && !pathTracerFlag;
+const bakeVariantFlags = ['--sky-only', '--bake-preset', '--probes', '--ambient', '--ambient-occlusion', '--cascades', '--no-object-probes', '--object-probes', '--no-water', '--ssr-water'];
+const bakeScoreIsBaseline = bakeScoreRun && argumentsGiven.includes('--update-baseline') && !bakeVariantFlags.some(flag => argumentsGiven.includes(flag));
+const onlyScenes = sceneIdsGiven ?? (bakeScoreRun ? bakedSceneIds : null);
+const skipCpu = skipCpuFlag || bakeScoreRun;
async function renderBackend(backend, serverUrl, graphics) {
const browser = await chromium.launch({ executablePath: CHROME, headless: true, args: BACKEND_ARGUMENTS[backend] });
@@ -94,7 +105,7 @@ async function renderBackend(backend, serverUrl, graphics) {
await openHarness();
const sceneIds = await page.evaluate(() => window.referenceSceneIds);
for (const id of sceneIds) {
- if (onlyScene && id !== onlyScene) continue;
+ if (onlyScenes && !onlyScenes.includes(id)) continue;
await openHarness();
const started = performance.now();
const describe = backend === 'gpu';
@@ -133,7 +144,7 @@ async function renderPathTracerScenes(serverUrl) {
await page.waitForFunction(() => window.renderReferencePathTraced, null, { timeout: PAGE_LOAD_TIMEOUT });
const sceneIds = await page.evaluate(() => window.referenceSceneIds);
for (const id of sceneIds) {
- if (onlyScene && id !== onlyScene) continue;
+ if (onlyScenes && !onlyScenes.includes(id)) continue;
const options = { width: WIDTH, height: HEIGHT, samples: pathTracerSamples, maxSeconds: 1800, ...(pathTracerNoiseThreshold === undefined ? {} : { noiseThreshold: pathTracerNoiseThreshold }) };
const result = await page.evaluate(([sceneId, tracerOptions]) => window.renderReferencePathTraced(sceneId, tracerOptions), [id, options]);
await writeFile(`${WORK_DIRECTORY}/${id}-pathtracer.png`, Buffer.from(result.image, 'base64'));
@@ -179,16 +190,20 @@ if (!scoreOnly && !pathTracerFlag) {
}
await writeFile(`${WORK_DIRECTORY}/graphics.json`, JSON.stringify(graphics, null, 1));
if (!skipBlender) {
- const sceneArguments = onlyScene ? [onlyScene] : [];
+ const sceneArguments = onlyScenes ?? [];
runChecked(BLENDER, ['-b', '--factory-startup', '--python', 'bench/reference/render-blender.py', '--', WORK_DIRECTORY, ...sceneArguments]);
}
}
-if ((!onlyScene && !skipCpu) || pathTracerFlag) {
+if ((!onlyScenes && !skipCpu) || pathTracerFlag || bakeScoreRun) {
runChecked('uv', [
'run', '--no-project', '--python', '3.12', '--with', 'flip-evaluator', '--with', 'pillow', '--with', 'numpy',
- 'python', '-I', 'bench/reference/score.py', WORK_DIRECTORY, OUTPUT_DIRECTORY, ...(pathTracerFlag ? ['--path-tracer'] : []), ...(sceneSet ? [`--${sceneSet}`] : []),
+ 'python', '-I', 'bench/reference/score.py', WORK_DIRECTORY, bakeScoreRun && !bakeScoreIsBaseline ? WORK_DIRECTORY : OUTPUT_DIRECTORY, ...(pathTracerFlag ? ['--path-tracer'] : []), ...(bakeScoreRun ? ['--bake', bakedSceneIds.join(',')] : []), ...(sceneSet ? [`--${sceneSet}`] : []),
]);
+ if (bakeScoreRun && !bakeScoreIsBaseline && !bakeVariantFlags.some(flag => argumentsGiven.includes(flag))) {
+ const comparison = spawnSync('node', ['bench/reference/compare-scores.mjs', `${WORK_DIRECTORY}/bake.json`], { stdio: 'inherit' });
+ if (comparison.status !== 0) process.exitCode = 1;
+ }
}
else {
// Scoring reads every scene in the work directory, so a run of one scene would mix new and old renders; it leaves the scores as they were.
diff --git a/bench/reference/score.py b/bench/reference/score.py
index d5f4e3e5..c27c6efa 100644
--- a/bench/reference/score.py
+++ b/bench/reference/score.py
@@ -99,7 +99,47 @@ def score_path_tracer(work_directory, output_directory):
print(scene_id.ljust(20) + f"{entry['pathtracer-vs-cycles']:.4f}".rjust(22) + f"{entry['seconds']:.1f}".rjust(10))
+def score_baked(work_directory, output_directory, scene_ids):
+ """FLIP of the GPU render of each baked scene against Cycles and EEVEE, into bake.json. The CPU render is not needed."""
+ images_directory = output_directory / "images"
+ images_directory.mkdir(parents=True, exist_ok=True)
+ results = {}
+ for scene_id in scene_ids:
+ cycles = tone_map_like_67js(np.load(work_directory / f"{scene_id}-cycles.npy"))
+ eevee = tone_map_like_67js(np.load(work_directory / f"{scene_id}-eevee.npy"))
+ gpu = load_png(work_directory / f"{scene_id}-gpu.png")
+ scores = {
+ "gpu-vs-cycles": flip_score(cycles, gpu),
+ "gpu-vs-eevee": flip_score(eevee, gpu),
+ "eevee-vs-cycles": flip_score(cycles, eevee),
+ }
+ region = json.loads((work_directory / "scenes" / f"{scene_id}.json").read_text(encoding="utf-8")).get("brightRegion")
+ if region:
+ mask = bright_region_mask(cycles, region)
+ scores["region-pixels"] = round(float(mask.mean()), 4)
+ scores["region-gpu-vs-cycles"] = masked_flip_score(cycles, gpu, mask)
+ scores["region-gpu-vs-eevee"] = masked_flip_score(eevee, gpu, mask)
+ scores["region-eevee-vs-cycles"] = masked_flip_score(cycles, eevee, mask)
+ results[scene_id] = scores
+ side_by_side(cycles, eevee, gpu, scores).save(images_directory / f"{scene_id}-baked.jpg", quality=80)
+ graphics = json.loads((work_directory / "graphics.json").read_text(encoding="utf-8"))
+ timings = json.loads((work_directory / "blender-timings.json").read_text(encoding="utf-8"))
+ report = {
+ "metric": "FLIP (LDR), mean error of the baked GPU render against Cycles, lower is better",
+ "blenderVersion": timings["blenderVersion"],
+ "graphics": {"gpu": graphics["gpu"]},
+ "scores": results,
+ }
+ (output_directory / "bake.json").write_text(json.dumps(report, indent=1) + "\n", encoding="utf-8")
+ print("scene".ljust(20) + "".join(column.rjust(18) for column in ["gpu-vs-cycles", "gpu-vs-eevee", "eevee-vs-cycles"]))
+ for scene_id, scores in results.items():
+ print(scene_id.ljust(20) + "".join(f"{scores[column]:.4f}".rjust(18) for column in ["gpu-vs-cycles", "gpu-vs-eevee", "eevee-vs-cycles"]))
+
+
def main():
+ if "--bake" in sys.argv:
+ score_baked(Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[sys.argv.index("--bake") + 1].split(","))
+ return
if "--path-tracer" in sys.argv:
score_path_tracer(Path(sys.argv[1]), Path(sys.argv[2]))
return
diff --git a/changes/issue-169-one-sun.md b/changes/issue-169-one-sun.md
new file mode 100644
index 00000000..ff1e3104
--- /dev/null
+++ b/changes/issue-169-one-sun.md
@@ -0,0 +1 @@
+- Rerun the reference suite under the capped sky with one sun (issue #169): `khronos.json` and `path-tracer.json` are rescored against fresh Cycles renders (asset-only 67js against Cycles improved on four of five Khronos assets, for example `SheenChair` 0.3745 to 0.3316 and `ClearCoatTest` 0.2306 to 0.2097; the path tracer got worse on `outdoor-hdri` 0.1016 to 0.1242, `character` 0.1120 to 0.1289 and `material-spheres` 0.1187 to 0.1383), and baked scores of `lit-interior`, `brick-room`, `dark-shaft`, `temple-hall` and `metal-room` are now stored in `bench/reference/bake.json`. `npm run reference -- --bake <ids> --skip-blender` compares a fresh run with it and fails when a score is worse by more than 0.01 FLIP (`bench/reference/compare-scores.mjs`, `tests/reference-bake-regression.test.js`, which reads JSON only). `--scene` takes several ids. The README has one current score table and a History heading.
diff --git a/tests/reference-bake-regression.test.js b/tests/reference-bake-regression.test.js
new file mode 100644
index 00000000..867a9234
--- /dev/null
+++ b/tests/reference-bake-regression.test.js
@@ -0,0 +1,41 @@
+import assert from 'node:assert/strict';
+import { existsSync, readFileSync } from 'node:fs';
+import test from 'node:test';
+import { fileURLToPath } from 'node:url';
+import { DEFAULT_MARGIN, findRegressions } from '../bench/reference/compare-scores.mjs';
+
+// Reads JSON files only; nothing here renders, so CI can run it. After `npm run reference -- --bake <ids> --skip-blender`
+// the fresh results sit in artifacts/reference/bake.json, and the last test compares them with the stored ones.
+const storedPath = fileURLToPath(new URL('../bench/reference/bake.json', import.meta.url));
+const freshPath = fileURLToPath(new URL('../artifacts/reference/bake.json', import.meta.url));
+const stored = JSON.parse(readFileSync(storedPath, 'utf8'));
+const BAKED_ROOMS = ['brick-room', 'dark-shaft', 'lit-interior', 'metal-room', 'temple-hall'];
+
+test('the stored bake scores hold every baked room, with no local paths', () => {
+ assert.match(stored.blenderVersion, /^5\.2/);
+ assert.deepEqual(Object.keys(stored.scores).sort(), BAKED_ROOMS);
+ for (const [scene, scores] of Object.entries(stored.scores)) {
+ for (const [name, score] of Object.entries(scores)) {
+ assert.ok(score >= 0 && score <= 1, `${scene} ${name} is ${score}`);
+ }
+ }
+ assert.doesNotMatch(JSON.stringify(stored), /[A-Z]:[\\/]Users/);
+});
+
+test('a score that gets worse by more than the margin is a regression, and noise inside it is not', () => {
+ // The numbers of issue #160: lit-interior baked went from 0.1444 to 0.1915 and nothing failed.
+ const before = { scores: { 'lit-interior': { 'gpu-vs-cycles': 0.1444, 'region-pixels': 0.1 }, 'brick-room': { 'gpu-vs-cycles': 0.1515 } } };
+ const regressed = { scores: { 'lit-interior': { 'gpu-vs-cycles': 0.1915, 'region-pixels': 0.5 }, 'brick-room': { 'gpu-vs-cycles': 0.1515 + DEFAULT_MARGIN / 2 } } };
+ assert.deepEqual(findRegressions(before, regressed), [{ scene: 'lit-interior', name: 'gpu-vs-cycles', stored: 0.1444, fresh: 0.1915 }]);
+});
+
+test('a better score, and a scene the fresh run did not draw, are not regressions', () => {
+ const before = { scores: { a: { 'gpu-vs-cycles': 0.2 }, b: { 'gpu-vs-cycles': 0.2 } } };
+ const fresh = { scores: { a: { 'gpu-vs-cycles': 0.1 } } };
+ assert.deepEqual(findRegressions(before, fresh), []);
+});
+
+test('a fresh bake.json from a run has no score worse than the stored one by the margin', { skip: !existsSync(freshPath) && 'no fresh run in artifacts/reference' }, () => {
+ const fresh = JSON.parse(readFileSync(freshPath, 'utf8'));
+ assert.deepEqual(findRegressions(stored, fresh), []);
+});