Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Score EEVEE at 8 bits like the 67js PNG (part of #258)

Nguyen Xuan Anff6a2f2156fa
diff --git a/bench/reference/README.md b/bench/reference/README.md
index 58e2ea85..ba860d95 100644
--- a/bench/reference/README.md
+++ b/bench/reference/README.md
@@ -1280,3 +1280,35 @@ Change tried: in the probe trace, a hit on a metal part returns what that part m
The best of them is 0.0004 better, which is inside what the choice of gain alone moves, and 0.0022 short of EEVEE. At gain 1 it overshoots the steel cylinder (0.556 to 0.590, Cycles 0.573) and brightens the chrome sphere, which is already too bright; the sphere's error is the structure of the reflection (the cloud pattern and the floor under its base), not its level. This agrees with the earlier "metal parts mirroring the sky too" row (0.0939). Not kept; no engine file, scene or `baseline.json` row changed, and the first box of #167 stays open: `metal` 0.0944 against EEVEE 0.0916. A scene without a reflective part never loads the trace, so its frame cost is as before.
Next idea: the open floor is more than half of the error and the largest gap, and a surface-level cause there needs a new input (light the metal parts reflect onto the floor, which a diffuse bounce leaves out), for example a few traced caustic-like patches stamped into the ground bounce.
+
+## Same quantisation for 67js and EEVEE (2026-11-03, #258)
+
+The scorer read 67js back as an 8-bit PNG but scored EEVEE from a float array, so every 67js against EEVEE number carried the rounding of one side only. `score.py` now rounds the EEVEE array to 8 bits in display space (`load_eevee`), which is what writing the PNG does. Cycles stays float, because it is the reference both are measured against. The alternative, reading 67js back as float, would need a float readback path in the harness for a measurement of the same size, and the PNG is what the 67js columns already use. `tests/reference-quantisation.test.js` pins the loader against an 8-bit PNG round trip.
+
+The EEVEE columns were regenerated from the stored Cycles and EEVEE arrays and the stored 67js PNGs of the baseline run, without rendering. A cell is replaced only where the old scorer on those stored files reproduces the committed number, so the 67js columns (gpu-vs-cycles, cpu-vs-cycles, gpu-vs-cpu) are untouched. Scene order against EEVEE does not flip in any scene; `metal` goes from EEVEE ahead (0.0916 against 0.0944) to a tie at 0.0944.
+
+| scene | eevee-vs-cycles | gpu-vs-eevee |
+| --- | --- | --- |
+| brick-flat | 0.1099 to 0.1129 | 0.0961 to 0.0941 |
+| brick-graph | 0.1221 to 0.1249 | 0.1001 to 0.0976 |
+| brick-room | 0.5020 to 0.5021 | same |
+| car-paint | 0.0844 to 0.0869 | same |
+| character | 0.0622 to 0.0657 | 0.0661 to 0.0626 |
+| dark-shaft | 0.2654 to 0.2658 | 0.2638 to 0.2640 |
+| fog | 0.2662 to 0.2677 | 0.1324 to 0.1334 |
+| fog-room | 0.3730 to 0.3736 | 0.3635 to 0.3638 |
+| fog-valley | 0.1952 to 0.1960 | 0.1563 to 0.1571 |
+| foliage | 0.0888 to 0.0924 | 0.0804 to 0.0771 |
+| glass | 0.1008 to 0.1056 | 0.1100 to 0.1082 |
+| light-linking | 0.1558 to 0.1574 | 0.1127 to 0.1111 |
+| lit-interior | same | 0.0926 to 0.0927 |
+| long-street | 0.0976 to 0.0990 | 0.1152 to 0.1146 |
+| material-spheres | 0.1053 to 0.1070 | same |
+| megacity-dusk | 0.1524 to 0.1532 | 0.0947 to 0.0935 |
+| metal | 0.0916 to 0.0944 | same |
+| mountain-valley | 0.1893 to 0.1895 | 0.2671 to 0.2668 |
+| outdoor-hdri | 0.0403 to 0.0441 | 0.0366 to 0.0328 |
+| spot-area | 0.1935 to 0.1957 | 0.1709 to 0.1686 |
+| temple-hall | 0.2341 to 0.2345 | same |
+
+Not regenerated, still on the old float basis: `cpu-vs-eevee` for most scenes, `gpu-vs-eevee` for `brushed-film`, `car-paint`, `material-spheres`, `metal` and `metal-room` and `eevee-vs-cycles` for `brushed-film`. The `region-*` EEVEE cells of `dark-shaft`, `fog-room` and `temple-hall` were regenerated and are not in the table. The stored images of those scenes come from a later run than the baseline, so the old scorer does not reproduce the committed number from them, and a new number would mix two changes. They need a full `npm run reference` run. `grand.json` is also not regenerated: only `megacity-dusk-a` has an EEVEE array stored, and its stored images score 0.1985 against Cycles where `grand.json` has 0.1507. The side by side images in `images/` keep their old labels until that run.
diff --git a/bench/reference/baseline.json b/bench/reference/baseline.json
index 37b0408e..f1bc19f6 100644
--- a/bench/reference/baseline.json
+++ b/bench/reference/baseline.json
@@ -43,7 +43,7 @@
"eevee": 0.9
},
"foliage": {
- "cycles": 5.0,
+ "cycles": 5,
"eevee": 0.4
},
"glass": {
@@ -71,7 +71,7 @@
"eevee": 0.7
},
"metal-room": {
- "cycles": 9.0,
+ "cycles": 9,
"eevee": 0.3
},
"metal": {
@@ -91,7 +91,7 @@
"eevee": 0.3
},
"temple-hall": {
- "cycles": 16.0,
+ "cycles": 16,
"eevee": 0.8
}
},
@@ -102,18 +102,18 @@
"scores": {
"brick-flat": {
"gpu-vs-cycles": 0.0547,
- "gpu-vs-eevee": 0.0961,
+ "gpu-vs-eevee": 0.0941,
"cpu-vs-cycles": 0.0543,
- "cpu-vs-eevee": 0.0963,
- "eevee-vs-cycles": 0.1099,
+ "cpu-vs-eevee": 0.0933,
+ "eevee-vs-cycles": 0.1129,
"gpu-vs-cpu": 0.0065
},
"brick-graph": {
"gpu-vs-cycles": 0.0982,
- "gpu-vs-eevee": 0.1001,
+ "gpu-vs-eevee": 0.0976,
"cpu-vs-cycles": 0.0982,
- "cpu-vs-eevee": 0.1001,
- "eevee-vs-cycles": 0.1221,
+ "cpu-vs-eevee": 0.0966,
+ "eevee-vs-cycles": 0.1249,
"gpu-vs-cpu": 0.0079
},
"brick-room": {
@@ -121,7 +121,7 @@
"gpu-vs-eevee": 0.0711,
"cpu-vs-cycles": 0.509,
"cpu-vs-eevee": 0.0714,
- "eevee-vs-cycles": 0.502,
+ "eevee-vs-cycles": 0.5021,
"gpu-vs-cpu": 0.0071
},
"brushed-film": {
@@ -137,84 +137,84 @@
"gpu-vs-eevee": 0.066,
"cpu-vs-cycles": 0.0975,
"cpu-vs-eevee": 0.0661,
- "eevee-vs-cycles": 0.0844,
+ "eevee-vs-cycles": 0.0869,
"gpu-vs-cpu": 0.0061
},
"character": {
"gpu-vs-cycles": 0.0567,
- "gpu-vs-eevee": 0.0661,
+ "gpu-vs-eevee": 0.0626,
"cpu-vs-cycles": 0.0565,
- "cpu-vs-eevee": 0.0663,
- "eevee-vs-cycles": 0.0622,
+ "cpu-vs-eevee": 0.0616,
+ "eevee-vs-cycles": 0.0657,
"gpu-vs-cpu": 0.006
},
"dark-shaft": {
"gpu-vs-cycles": 0.0559,
- "gpu-vs-eevee": 0.2638,
+ "gpu-vs-eevee": 0.264,
"cpu-vs-cycles": 0.0559,
"cpu-vs-eevee": 0.2636,
- "eevee-vs-cycles": 0.2654,
+ "eevee-vs-cycles": 0.2658,
"gpu-vs-cpu": 0.003,
"region-pixels": 0.0989,
"region-gpu-vs-cycles": 0.1284,
- "region-gpu-vs-eevee": 0.2562,
- "region-eevee-vs-cycles": 0.2645
+ "region-gpu-vs-eevee": 0.2564,
+ "region-eevee-vs-cycles": 0.2647
},
"fog": {
"gpu-vs-cycles": 0.2021,
- "gpu-vs-eevee": 0.1324,
+ "gpu-vs-eevee": 0.1334,
"cpu-vs-cycles": 0.2017,
- "cpu-vs-eevee": 0.1328,
- "eevee-vs-cycles": 0.2662,
+ "cpu-vs-eevee": 0.1338,
+ "eevee-vs-cycles": 0.2677,
"gpu-vs-cpu": 0.0087
},
"fog-room": {
"gpu-vs-cycles": 0.1036,
- "gpu-vs-eevee": 0.3635,
+ "gpu-vs-eevee": 0.3638,
"cpu-vs-cycles": 0.1036,
- "cpu-vs-eevee": 0.363,
- "eevee-vs-cycles": 0.373,
+ "cpu-vs-eevee": 0.3633,
+ "eevee-vs-cycles": 0.3736,
"gpu-vs-cpu": 0.0065,
"region-pixels": 0.2428,
"region-gpu-vs-cycles": 0.1614,
- "region-gpu-vs-eevee": 0.3271,
- "region-eevee-vs-cycles": 0.3147
+ "region-gpu-vs-eevee": 0.3273,
+ "region-eevee-vs-cycles": 0.3148
},
"fog-valley": {
"gpu-vs-cycles": 0.1791,
- "gpu-vs-eevee": 0.1563,
+ "gpu-vs-eevee": 0.1571,
"cpu-vs-cycles": 0.1794,
- "cpu-vs-eevee": 0.1568,
- "eevee-vs-cycles": 0.1952,
+ "cpu-vs-eevee": 0.1576,
+ "eevee-vs-cycles": 0.196,
"gpu-vs-cpu": 0.0114
},
"foliage": {
"gpu-vs-cycles": 0.0979,
- "gpu-vs-eevee": 0.0804,
+ "gpu-vs-eevee": 0.0771,
"cpu-vs-cycles": 0.0976,
"cpu-vs-eevee": 0.0802,
- "eevee-vs-cycles": 0.0888,
+ "eevee-vs-cycles": 0.0924,
"gpu-vs-cpu": 0.0095
},
"glass": {
"gpu-vs-cycles": 0.0913,
- "gpu-vs-eevee": 0.11,
+ "gpu-vs-eevee": 0.1082,
"cpu-vs-cycles": 0.0908,
- "cpu-vs-eevee": 0.1103,
- "eevee-vs-cycles": 0.1008,
+ "cpu-vs-eevee": 0.1082,
+ "eevee-vs-cycles": 0.1056,
"gpu-vs-cpu": 0.0074
},
"light-linking": {
"gpu-vs-cycles": 0.0993,
- "gpu-vs-eevee": 0.1127,
+ "gpu-vs-eevee": 0.1111,
"cpu-vs-cycles": 0.0988,
"cpu-vs-eevee": 0.1132,
- "eevee-vs-cycles": 0.1558,
+ "eevee-vs-cycles": 0.1574,
"gpu-vs-cpu": 0.0027
},
"lit-interior": {
"gpu-vs-cycles": 0.505,
- "gpu-vs-eevee": 0.0926,
+ "gpu-vs-eevee": 0.0927,
"cpu-vs-cycles": 0.5046,
"cpu-vs-eevee": 0.0932,
"eevee-vs-cycles": 0.5159,
@@ -222,10 +222,10 @@
},
"long-street": {
"gpu-vs-cycles": 0.1121,
- "gpu-vs-eevee": 0.1152,
+ "gpu-vs-eevee": 0.1146,
"cpu-vs-cycles": 0.115,
"cpu-vs-eevee": 0.0919,
- "eevee-vs-cycles": 0.0976,
+ "eevee-vs-cycles": 0.099,
"gpu-vs-cpu": 0.0066
},
"material-spheres": {
@@ -233,15 +233,15 @@
"gpu-vs-eevee": 0.0744,
"cpu-vs-cycles": 0.0949,
"cpu-vs-eevee": 0.0743,
- "eevee-vs-cycles": 0.1053,
+ "eevee-vs-cycles": 0.107,
"gpu-vs-cpu": 0.0067
},
"megacity-dusk": {
"gpu-vs-cycles": 0.1601,
- "gpu-vs-eevee": 0.0947,
+ "gpu-vs-eevee": 0.0935,
"cpu-vs-cycles": 0.1577,
"cpu-vs-eevee": 0.1076,
- "eevee-vs-cycles": 0.1524,
+ "eevee-vs-cycles": 0.1532,
"gpu-vs-cpu": 0.0276
},
"metal": {
@@ -249,7 +249,7 @@
"gpu-vs-eevee": 0.0826,
"cpu-vs-cycles": 0.0942,
"cpu-vs-eevee": 0.0828,
- "eevee-vs-cycles": 0.0916,
+ "eevee-vs-cycles": 0.0944,
"gpu-vs-cpu": 0.0067
},
"metal-room": {
@@ -262,26 +262,26 @@
},
"mountain-valley": {
"gpu-vs-cycles": 0.1769,
- "gpu-vs-eevee": 0.2671,
+ "gpu-vs-eevee": 0.2668,
"cpu-vs-cycles": 0.1743,
"cpu-vs-eevee": 0.2655,
- "eevee-vs-cycles": 0.1893,
+ "eevee-vs-cycles": 0.1895,
"gpu-vs-cpu": 0.0124
},
"outdoor-hdri": {
"gpu-vs-cycles": 0.0333,
- "gpu-vs-eevee": 0.0366,
+ "gpu-vs-eevee": 0.0328,
"cpu-vs-cycles": 0.0328,
- "cpu-vs-eevee": 0.0363,
- "eevee-vs-cycles": 0.0403,
+ "cpu-vs-eevee": 0.0311,
+ "eevee-vs-cycles": 0.0441,
"gpu-vs-cpu": 0.0066
},
"spot-area": {
"gpu-vs-cycles": 0.1244,
- "gpu-vs-eevee": 0.1709,
+ "gpu-vs-eevee": 0.1686,
"cpu-vs-cycles": 0.1245,
"cpu-vs-eevee": 0.1712,
- "eevee-vs-cycles": 0.1935,
+ "eevee-vs-cycles": 0.1957,
"gpu-vs-cpu": 0.0033
},
"temple-hall": {
@@ -289,12 +289,12 @@
"gpu-vs-eevee": 0.2319,
"cpu-vs-cycles": 0.064,
"cpu-vs-eevee": 0.2317,
- "eevee-vs-cycles": 0.2341,
+ "eevee-vs-cycles": 0.2345,
"gpu-vs-cpu": 0.0021,
"region-pixels": 0.0961,
"region-gpu-vs-cycles": 0.236,
- "region-gpu-vs-eevee": 0.2841,
- "region-eevee-vs-cycles": 0.2453
+ "region-gpu-vs-eevee": 0.2842,
+ "region-eevee-vs-cycles": 0.2454
}
}
}
diff --git a/bench/reference/score.py b/bench/reference/score.py
index c27c6efa..c51e45e7 100644
--- a/bench/reference/score.py
+++ b/bench/reference/score.py
@@ -3,8 +3,13 @@
# A scene that declares brightRegion (scenes.js) also gets region-* scores over the bright pixels of its Cycles image.
# FLIP is 0 for identical images, and higher means a larger visible difference. EEVEE against Cycles is a yardstick:
# it shows what a good rasterizer scores on the same scene.
+# Quantisation (#258): 67js is read back from an 8-bit PNG, so the EEVEE array is rounded to 8 bits in display space the same
+# way before FLIP. Cycles stays float because it is the reference both sides are measured against. 67js is not read back as
+# float because the harness reads canvas pixels, which are 8-bit by construction.
+# uv run --no-project --python 3.12 --with flip-evaluator --with pillow --with numpy python -I bench/reference/score.py --self-test
import json
import sys
+import tempfile
from pathlib import Path
import flip_evaluator
@@ -24,6 +29,34 @@ def tone_map_like_67js(linear):
return mapped ** (1 / GAMMA)
+def quantise_to_8_bit(display):
+ """Rounds display-space values to the nearest of 256 steps, as writing an 8-bit PNG does."""
+ return np.round(np.clip(display, 0, 1) * 255).astype(np.float32) / 255
+
+
+def load_eevee(path):
+ """An EEVEE render in the same 8-bit display space as a 67js PNG, so FLIP sees equal rounding on both sides."""
+ return quantise_to_8_bit(tone_map_like_67js(np.load(path)))
+
+
+def self_test():
+ rng = np.random.default_rng(258)
+ linear = rng.random((16, 16, 3), dtype=np.float32) * 3
+ with tempfile.TemporaryDirectory() as directory:
+ path = Path(directory) / "scene-eevee.npy"
+ np.save(path, linear)
+ loaded = load_eevee(path)
+ png_path = Path(directory) / "scene.png"
+ Image.fromarray(np.round(np.clip(tone_map_like_67js(linear), 0, 1) * 255).astype(np.uint8)).save(png_path)
+ written = load_png(png_path)
+ assert np.array_equal(loaded, written), "the EEVEE array does not match what an 8-bit PNG of it reads back as"
+ assert np.array_equal(np.round(loaded * 255), np.round(loaded * 255).astype(np.uint8)), "values are not whole 8-bit steps"
+ assert np.array_equal(quantise_to_8_bit(loaded), loaded), "quantising twice changed the values"
+ float_display = tone_map_like_67js(linear)
+ assert 0 < np.abs(loaded - float_display).max() <= 0.5 / 255 + 1e-6, "the rounding error is outside half a step"
+ print("score self-test ok")
+
+
def load_png(path):
return np.asarray(Image.open(path).convert("RGB"), dtype=np.float32) / 255
@@ -106,7 +139,7 @@ def score_baked(work_directory, output_directory, scene_ids):
results = {}
for scene_id in scene_ids:
cycles = tone_map_like_67js(np.load(work_directory / f"{scene_id}-cycles.npy"))
- eevee = tone_map_like_67js(np.load(work_directory / f"{scene_id}-eevee.npy"))
+ eevee = load_eevee(work_directory / f"{scene_id}-eevee.npy")
gpu = load_png(work_directory / f"{scene_id}-gpu.png")
scores = {
"gpu-vs-cycles": flip_score(cycles, gpu),
@@ -137,6 +170,9 @@ def score_baked(work_directory, output_directory, scene_ids):
def main():
+ if "--self-test" in sys.argv:
+ self_test()
+ return
if "--bake" in sys.argv:
score_baked(Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[sys.argv.index("--bake") + 1].split(","))
return
@@ -155,7 +191,7 @@ def main():
results = {}
for scene_id in scene_ids:
cycles = tone_map_like_67js(np.load(work_directory / f"{scene_id}-cycles.npy"))
- eevee = tone_map_like_67js(np.load(work_directory / f"{scene_id}-eevee.npy"))
+ eevee = load_eevee(work_directory / f"{scene_id}-eevee.npy")
gpu = load_png(work_directory / f"{scene_id}-gpu.png")
cpu = load_png(work_directory / f"{scene_id}-cpu.png")
scores = {
diff --git a/changes/258.md b/changes/258.md
new file mode 100644
index 00000000..d41bbad9
--- /dev/null
+++ b/changes/258.md
@@ -0,0 +1 @@
+- Score 67js and EEVEE with the same quantisation (part of issue #258). `bench/reference/score.py` read 67js from an 8-bit PNG but EEVEE from a float array; it now rounds the EEVEE array to 8 bits too (`load_eevee`), and `tests/reference-quantisation.test.js` pins it. The EEVEE columns of `baseline.json` were regenerated from the stored arrays where they reproduce the committed numbers: `foliage` EEVEE against Cycles goes from 0.0888 to 0.0924, and no scene changes order against 67js (`metal` becomes a tie at 0.0944). Some cells and `grand.json` need a full reference run. See `bench/reference/README.md`.
diff --git a/tests/reference-quantisation.test.js b/tests/reference-quantisation.test.js
new file mode 100644
index 00000000..40556684
--- /dev/null
+++ b/tests/reference-quantisation.test.js
@@ -0,0 +1,24 @@
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import { spawnSync } from 'node:child_process';
+import test from 'node:test';
+
+const uvAvailable = spawnSync('uv', ['--version']).status === 0;
+const scorerSource = readFileSync(new URL('../bench/reference/score.py', import.meta.url), 'utf8');
+
+// The 67js render is an 8-bit PNG, so the EEVEE array is rounded to 8 bits before FLIP (#258).
+test('score.py rounds the EEVEE array to what an 8-bit PNG holds', { skip: !uvAvailable && 'uv is not installed', timeout: 180000 }, () => {
+ const run = spawnSync(
+ 'uv',
+ ['run', '--no-project', '--python', '3.12', '--with', 'flip-evaluator', '--with', 'pillow', '--with', 'numpy', 'python', '-I', 'bench/reference/score.py', '--self-test'],
+ { encoding: 'utf8' },
+ );
+ assert.equal(run.status, 0, run.stderr);
+ assert.match(run.stdout, /score self-test ok/);
+});
+
+test('every scoring path loads EEVEE through the quantising loader', () => {
+ assert.doesNotMatch(scorerSource, /tone_map_like_67js\(np\.load\([^)]*eevee/);
+ const loads = scorerSource.match(/load_eevee\(work_directory/g) ?? [];
+ assert.equal(loads.length, 2);
+});