Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Draw 3D Gaussian splats as an optional addon (issue #288)

Nguyen Xuan An6cabb1fb93dc
diff --git a/.gitignore b/.gitignore
index 779332e6..fca31ce9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -56,3 +56,5 @@ public/agent-motion.md
public/agent-character-motion.md
public/agent-review.md
.claude/worktrees/
+bench/gaussian-splats/data/
+public/gaussian-splats.md
diff --git a/bench/gaussian-splats/CREDITS.md b/bench/gaussian-splats/CREDITS.md
new file mode 100644
index 00000000..0753cacd
--- /dev/null
+++ b/bench/gaussian-splats/CREDITS.md
@@ -0,0 +1,9 @@
+# Credits
+
+`node bench/gaussian-splats/fetch-sample.mjs` downloads one splat scene into `bench/gaussian-splats/data/`, a folder git ignores. No splat file is committed.
+
+| Scene | Author | Source | Licence |
+| --- | --- | --- | --- |
+| millipede spec. (`millipede.splat`, 116,374 splats, 3.7 MB) | Fabian Plum ([scant3d](https://superspl.at/user?id=scant3d)) | [superspl.at/scene/3d5482d4](https://superspl.at/scene/3d5482d4), through `examples/models/splat/millipede.splat` of the [three.js repository](https://github.com/mrdoob/three.js) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): the author must be credited, commercial use is allowed |
+
+The script rewrites the 32-byte `.splat` records as a standard 3DGS PLY (`millipede.ply`, 6.5 MB) so that 67js and three.js load the same bytes. The conversion keeps colour and opacity to 8 bits, as the source has them.
diff --git a/bench/gaussian-splats/README.md b/bench/gaussian-splats/README.md
new file mode 100644
index 00000000..11a5f4aa
--- /dev/null
+++ b/bench/gaussian-splats/README.md
@@ -0,0 +1,69 @@
+# Gaussian splat benchmark
+
+Frame time and memory of `67js/addons/gaussian-splats` (issue #288), and the same splats drawn by the three.js `GaussianSplat`. The guide is [docs/gaussian-splats.md](../../docs/gaussian-splats.md); the sample's licence is in [CREDITS.md](CREDITS.md).
+
+```
+node bench/gaussian-splats/fetch-sample.mjs # once: downloads the CC BY sample into data/ (git ignores it)
+bash ~/.agents/tools/gpu-lock.sh node bench/gaussian-splats/run.mjs gpu all # RTX 3060; pictures go to artifacts/gaussian-splats/
+node bench/gaussian-splats/run.mjs cpu all # SwiftShader (software GL)
+node bench/gaussian-splats/sort-time.mjs # the CPU sort alone, in Node
+```
+
+`run.mjs` takes `gpu` or `cpu`, then `engine67`, `three` or `all`, then a case-name filter. Each case loads `harness.html` (67js) or `three.html` (three.js) in Chrome at 960 by 540, with 4x MSAA in 67js and `antialias` in three.js.
+
+## What is drawn
+
+- `millipede-*`: the CC BY sample (116,374 splats), converted to a PLY and placed 2 m behind a recipe part (a plinth, a metal sphere and a ring; the picture is `*-millipede-behind-part.png`, where the ring and sphere hide the millipede). `-alone` has no part and is the case that three.js is compared with. `-sdr` is the same with `hdr: false`.
+- `synthetic-100k`, `synthetic-1m`: splats on the six faces of an 8 by 4 by 8 m room, seen from inside with the part in the middle, so the view holds several layers of overlap. The same splats go to three.js as PLY bytes.
+- `part-only`: the recipe part with no splats, for the base cost.
+
+A frame is timed from `render()` until a one-pixel `readPixels` returns (`gl.finish()` does not wait in Chrome). Each number is the median of 60 frames on the GPU, 6 on SwiftShader. `orbit` moves the camera 0.04 rad every frame, so every frame sorts: both engines re-sort at that step (three.js re-sorts when the view direction turns by about 1.8 degrees, 0.04 rad is 2.3). `still` keeps the camera, so no sort runs.
+
+## RTX 3060 (measured, one run, machine loaded by other sessions)
+
+Chrome through ANGLE and Direct3D 11, driver 621.01.
+
+| Case | splats | 67js orbit ms | 67js still ms | three.js orbit ms | three.js still ms |
+| --- | --- | --- | --- | --- | --- |
+| part only | 0 | 0.97 | 0.94 | n/a | n/a |
+| millipede behind the part | 116,374 | 4.58 | 1.66 | n/a | n/a |
+| millipede alone | 116,374 | 3.35 | 1.71 | 3.00 | 1.19 |
+| synthetic room | 100,000 | 3.29 | 1.96 | 2.51 | 1.11 |
+| synthetic room | 1,000,000 | 13.49 | 3.33 | 12.47 | 2.52 |
+
+The sort and the order upload of 67js took 1.4 ms at 116,374 splats, 2.0 ms at 100,000 and 11.2 ms at 1,000,000 inside these frames. Alone in Node (`sort-time.mjs`, 40 sorts) the sort took a median of 1.5 ms (range 1.0 to 6.5) at 100,000 and 14.8 ms (range 9.3 to 28.7) at 1,000,000.
+
+Reading it: at 100,000 splats the addon adds about 1 ms to a still frame and 2.3 ms to a moving one. At 1,000,000 it adds 2.4 ms still and 12.5 ms moving, of which the sort is about 11. Three.js is 0.5 to 1.2 ms faster in the moving rows and 0.6 to 0.8 ms in the still rows here; why is not known (guess: a cheaper per-splat draw, since the sorts take similar time).
+
+## SwiftShader (software, measured, one run)
+
+The CPU draws every pixel, so these numbers show fill cost and not what a user sees.
+
+| Case | splats | 67js orbit ms | 67js still ms | three.js orbit ms | three.js still ms |
+| --- | --- | --- | --- | --- | --- |
+| part only | 0 | 49 | 47 | n/a | n/a |
+| millipede behind the part | 116,374 | 996 | 1001 | n/a | n/a |
+| millipede alone | 116,374 | 884 | 901 | 839 | 832 |
+| synthetic room | 100,000 | 500 | 500 | 794 | 720 |
+| synthetic room | 1,000,000 | 4580 | 4471 | 5487 | 5964 |
+
+## Memory
+
+| Splats | 67js GPU (texture and order buffer, counted from sizes) | 67js CPU kept (positions and sort scratch, counted) | 67js page JS heap growth | three.js geometry attributes (CPU arrays) | three.js page JS heap growth |
+| --- | --- | --- | --- | --- | --- |
+| 100,000 | 5.0 MiB | 1.9 MiB | 12 MiB | 3.8 MiB | 54 MiB |
+| 116,374 (millipede) | 5.8 MiB | 2.2 MiB | 19 MiB | 4.4 MiB | 75 MiB |
+| 1,000,000 | 49.6 MiB | 19.1 MiB | 97 MiB | 38.1 MiB | 184 MiB |
+
+The 67js GPU figure is the size of the data it uploads: 48 bytes per splat and 4 for the order. It is not read from the driver, and three.js has no such counter, so the table does not compare GPU memory; the three.js column is the size of its CPU attribute arrays and the heap growth of its page. The heap growth is `performance.memory` of Chrome around loading and the timed frames, and includes the parsed arrays that the page still holds, so it is an upper bound on what an app must keep.
+
+## Picture against three.js
+
+`artifacts/gaussian-splats/gpu-millipede-alone-sdr.png` (67js, no HDR) and `gpu-three-millipede.png` have the same outline, position and splat shapes at the same camera: both read the same PLY bytes. They differ in colour. 67js shows the stored colour as is, as 3DGS viewers do (the DC spherical harmonic colour is already display-referred); three.js `GaussianSplat` shows the same file paler (inferred from the code: `GaussianSplatUtils.js` calls the colour `sh0ToLinear` and the renderer then encodes it to sRGB). The background also differs, since three.js has no sky here.
+
+## Limits of this comparison
+
+- The installed three.js is 0.186.1 (`package.json` pins it); it already has `GaussianSplat` and `GaussianSplatPLYLoader`, on `WebGPURenderer` with `forceWebGL`, and the issue names r187. Rerun after the dev dependency moves to 0.187.
+- Three.js draws with its own scene and no 67js part, so the part rows have no three.js column.
+- One run per cell. Frame times on this machine move with other sessions' load; treat differences under about 1 ms as noise.
+- Spherical harmonics beyond degree 0 are skipped by 67js; the sample has none.
diff --git a/bench/gaussian-splats/fetch-sample.mjs b/bench/gaussian-splats/fetch-sample.mjs
new file mode 100644
index 00000000..98db2204
--- /dev/null
+++ b/bench/gaussian-splats/fetch-sample.mjs
@@ -0,0 +1,55 @@
+// Downloads the sample splat of the Gaussian splat benchmark into bench/gaussian-splats/data/ (git ignores that folder) and
+// writes it as a standard 3DGS PLY, which is the file both 67js and three.js then load.
+// node bench/gaussian-splats/fetch-sample.mjs
+// The sample is "millipede spec." by Fabian Plum, CC BY 4.0, from https://superspl.at/scene/3d5482d4, the file that the
+// three.js repository ships in examples/models/splat/millipede.splat (MIT-licensed repository, per-file licence next to it).
+// The download is data: nothing in it is run. The .splat format holds 32 bytes per splat: position (3 floats), scale
+// (3 floats), colour (4 bytes RGBA) and rotation (4 bytes, w x y z, each (value - 128) / 128).
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const SOURCE_URL = 'https://raw.githubusercontent.com/mrdoob/three.js/dev/examples/models/splat/millipede.splat';
+const SPHERICAL_HARMONIC_DC = 0.28209479177387814;
+const BYTES_PER_SPLAT = 32;
+const PROPERTIES = ['x', 'y', 'z', 'f_dc_0', 'f_dc_1', 'f_dc_2', 'opacity', 'scale_0', 'scale_1', 'scale_2', 'rot_0', 'rot_1', 'rot_2', 'rot_3'];
+const dataDirectory = resolve(dirname(fileURLToPath(import.meta.url)), 'data');
+
+function logit(unit) {
+ const clamped = Math.min(254 / 255, Math.max(1 / 255, unit));
+ return Math.log(clamped / (1 - clamped));
+}
+
+export function splatToPly(arrayBuffer) {
+ const count = Math.floor(arrayBuffer.byteLength / BYTES_PER_SPLAT);
+ const view = new DataView(arrayBuffer);
+ const bytes = new Uint8Array(arrayBuffer);
+ const values = new Float32Array(count * PROPERTIES.length);
+ for (let index = 0; index < count; index++) {
+ const row = index * BYTES_PER_SPLAT;
+ const output = index * PROPERTIES.length;
+ for (let axis = 0; axis < 3; axis++) {
+ values[output + axis] = view.getFloat32(row + axis * 4, true);
+ values[output + 3 + axis] = (bytes[row + 24 + axis] / 255 - 0.5) / SPHERICAL_HARMONIC_DC;
+ values[output + 7 + axis] = Math.log(Math.max(view.getFloat32(row + 12 + axis * 4, true), 1e-7));
+ }
+ values[output + 6] = logit(bytes[row + 27] / 255);
+ for (let component = 0; component < 4; component++) {
+ values[output + 10 + component] = (bytes[row + 28 + component] - 128) / 128;
+ }
+ }
+ const header = ['ply', 'format binary_little_endian 1.0', `element vertex ${count}`, ...PROPERTIES.map(name => `property float ${name}`), 'end_header', ''].join('\n');
+ return Buffer.concat([Buffer.from(header, 'latin1'), Buffer.from(values.buffer)]);
+}
+
+if (process.argv[1] === fileURLToPath(import.meta.url)) {
+ await mkdir(dataDirectory, { recursive: true });
+ const splatPath = resolve(dataDirectory, 'millipede.splat');
+ const response = await fetch(SOURCE_URL);
+ if (!response.ok) throw new Error(`Download failed: HTTP ${response.status}`);
+ await writeFile(splatPath, Buffer.from(await response.arrayBuffer()));
+ const original = await readFile(splatPath);
+ const ply = splatToPly(original.buffer.slice(original.byteOffset, original.byteOffset + original.byteLength));
+ await writeFile(resolve(dataDirectory, 'millipede.ply'), ply);
+ console.log(`millipede.splat ${original.length} bytes -> millipede.ply ${ply.length} bytes, ${original.length / BYTES_PER_SPLAT} splats`);
+}
diff --git a/bench/gaussian-splats/harness.html b/bench/gaussian-splats/harness.html
new file mode 100644
index 00000000..8aa0bc5f
--- /dev/null
+++ b/bench/gaussian-splats/harness.html
@@ -0,0 +1,12 @@
+<!doctype html>
+<html>
+<head>
+<meta charset="utf-8">
+<title>67js Gaussian splat benchmark</title>
+<style>html,body{margin:0;background:#000}canvas{display:block}</style>
+</head>
+<body>
+<canvas></canvas>
+<script type="module" src="./harness.js"></script>
+</body>
+</html>
diff --git a/bench/gaussian-splats/harness.js b/bench/gaussian-splats/harness.js
new file mode 100644
index 00000000..527f9fc7
--- /dev/null
+++ b/bench/gaussian-splats/harness.js
@@ -0,0 +1,103 @@
+// Page of the Gaussian splat benchmark for 67js: a recipe part (a plinth with a sphere and a torus) drawn in front of splats.
+// window.runCase(options) draws the case, times the frames and returns a picture. run.mjs drives it.
+import { Renderer67 } from '/src/renderer.js';
+import { createAgentScene } from '/src/agent-scene.js';
+import { attachGaussianSplats, createGaussianSplats, parseGaussianSplatPly } from '/src/gaussian-splats.js';
+import { millipedeMatrix, syntheticRoom } from './scene-data.js';
+
+const RECIPE = {
+ format: '67js.scene',
+ schemaVersion: 1,
+ geometries: [{ id: 'plinth', type: 'box', bevel: 0.02 }, { id: 'ball', type: 'sphere' }, { id: 'ring', type: 'torus', tubeRadius: 0.08 }],
+ materials: [{ id: 'stone', color: [0.62, 0.6, 0.56], roughness: 0.8 }, { id: 'metal', color: [0.9, 0.55, 0.2], metallic: 1, roughness: 0.25 }],
+ nodes: [
+ { id: 'plinth', geometry: 'plinth', material: 'stone', translation: [0, 0.3, 0.6], scale: [1.2, 0.6, 1.2] },
+ { id: 'ball', geometry: 'ball', material: 'metal', translation: [0, 0.95, 0.6], scale: [0.7, 0.7, 0.7] },
+ { id: 'ring', geometry: 'ring', material: 'metal', translation: [0, 1.45, 0.6], rotation: [0.2588, 0, 0, 0.9659] },
+ ],
+};
+
+async function loadMillipede(maximumCount) {
+ const response = await fetch('/bench/gaussian-splats/data/millipede.ply');
+ if (!response.ok) throw new Error('Run node bench/gaussian-splats/fetch-sample.mjs first');
+ return parseGaussianSplatPly(await response.arrayBuffer(), { maximumCount });
+}
+
+function median(values) {
+ return values.slice().sort((first, second) => first - second)[Math.floor(values.length / 2)];
+}
+
+async function nextFrame() {
+ await new Promise(resolve => requestAnimationFrame(resolve));
+}
+
+function cameraAt(angle, source) {
+ const radius = source === 'synthetic' ? 3 : 4.2;
+ const eye = [Math.sin(angle) * radius, 1.7, Math.cos(angle) * radius];
+ const target = source === 'synthetic' ? [0, 1.5, 0] : [0, 1.3, -1];
+ return { eye, target, up: [0, 1, 0], fov: 50, near: 0.1, far: 200 };
+}
+
+const synchronizationPixel = new Uint8Array(4);
+
+// readPixels waits for the GPU to finish the frame; gl.finish() does not in Chrome.
+function timeFrames(renderer, context, frameCount, orbit, source, environment) {
+ const times = [];
+ for (let index = 0; index < frameCount; index++) {
+ const angle = orbit ? index * 0.04 : 0;
+ const start = performance.now();
+ renderer.render({ ...cameraAt(angle, source), environment, time: 0, exposure: 1 });
+ context.bindFramebuffer(context.FRAMEBUFFER, null);
+ context.readPixels(0, 0, 1, 1, context.RGBA, context.UNSIGNED_BYTE, synchronizationPixel);
+ times.push(performance.now() - start);
+ }
+ return times;
+}
+
+const GPU_STAT = 'gpuBytes';
+
+async function runCase({ source = 'synthetic', count = 100000, withPart = true, withSplats = true, hdr = true, frames = 60, width = 960, height = 540, screenshot = true }) {
+ const canvas = document.querySelector('canvas');
+ canvas.style.width = `${width}px`;
+ canvas.style.height = `${height}px`;
+ const scene = createAgentScene(RECIPE);
+ const environment = { ...scene.environment, world: true };
+ const renderer = new Renderer67(canvas, withPart ? scene.parts : [], { ground: false, hdr, depth: true, samples: 4, pixelRatio: 1, world: true, environment });
+ renderer.resize();
+ const context = renderer.context;
+ const heapBefore = performance.memory?.usedJSHeapSize ?? 0;
+ const loadStart = performance.now();
+ let splats = null;
+ if (withSplats) {
+ const data = source === 'millipede' ? await loadMillipede(count) : syntheticRoom(count);
+ const matrix = source === 'millipede' ? millipedeMatrix(data.bounds) : undefined;
+ splats = createGaussianSplats(data, { matrix, maximumCount: Math.max(count, data.count) });
+ attachGaussianSplats(renderer, splats);
+ }
+ const prepareMilliseconds = performance.now() - loadStart;
+ await nextFrame();
+ timeFrames(renderer, context, 10, true, source, environment);
+ const orbitTimes = timeFrames(renderer, context, frames, true, source, environment);
+ const stillTimes = timeFrames(renderer, context, frames, false, source, environment);
+ const image = screenshot ? (renderer.render({ ...cameraAt(0.25, source), environment, time: 0, exposure: 1 }), canvas.toDataURL('image/png').split(',')[1]) : null;
+ const debug = context.getExtension('WEBGL_debug_renderer_info');
+ const result = {
+ source, count: splats?.count ?? 0, withPart, hdr, width, height,
+ orbitMedianMilliseconds: Number(median(orbitTimes).toFixed(2)),
+ stillMedianMilliseconds: Number(median(stillTimes).toFixed(2)),
+ sortAndUploadMilliseconds: splats ? Number(splats.statistics.sortMilliseconds.toFixed(2)) : 0,
+ visibleSplats: splats?.statistics.visibleCount ?? 0,
+ gpuMebibytes: splats ? Number((splats.statistics[GPU_STAT] / 1048576).toFixed(1)) : 0,
+ cpuMebibytes: splats ? Number((splats.statistics.cpuBytes / 1048576).toFixed(1)) : 0,
+ jsHeapDeltaMebibytes: Number((((performance.memory?.usedJSHeapSize ?? 0) - heapBefore) / 1048576).toFixed(1)),
+ prepareMilliseconds: Number(prepareMilliseconds.toFixed(0)),
+ glError: context.getError(),
+ graphics: context.getParameter(debug ? debug.UNMASKED_RENDERER_WEBGL : context.RENDERER),
+ image,
+ };
+ renderer.dispose();
+ return result;
+}
+
+window.runCase = runCase;
+window.harnessReady = true;
diff --git a/bench/gaussian-splats/run.mjs b/bench/gaussian-splats/run.mjs
new file mode 100644
index 00000000..718082b5
--- /dev/null
+++ b/bench/gaussian-splats/run.mjs
@@ -0,0 +1,69 @@
+// Runs the Gaussian splat benchmark in Chrome and prints one JSON line per case; pictures go to artifacts/gaussian-splats/.
+// node bench/gaussian-splats/fetch-sample.mjs (once, for the millipede sample)
+// bash ~/.agents/tools/gpu-lock.sh node bench/gaussian-splats/run.mjs [gpu|cpu] [engine67|three|all] [case name filter]
+// gpu uses the machine's graphics card, cpu uses SwiftShader (software). Frame times are the median of 60 frames with a camera
+// that moves every frame ("orbit": a sort each frame) and 60 with a still camera (no sort). A one-pixel readPixels ends each timed frame.
+import { mkdir, writeFile } from 'node:fs/promises';
+import { constants, setPriority } from 'node:os';
+import { resolve } from 'node:path';
+import { chromium } from 'playwright';
+import { createServer } from 'vite';
+
+const CHROME = process.env.CHROME_PATH ?? 'C:/Program Files/Google/Chrome/Application/chrome.exe';
+const [backend = 'gpu', engines = 'engine67', onlyCase = ''] = process.argv.slice(2);
+const BACKEND_ARGUMENTS = {
+ gpu: ['--enable-webgl', '--ignore-gpu-blocklist', '--disable-renderer-backgrounding'],
+ cpu: ['--enable-webgl', '--ignore-gpu-blocklist', '--use-angle=swiftshader', '--enable-unsafe-swiftshader'],
+};
+const SWIFTSHADER_FRAMES = 6;
+const outputDirectory = resolve('artifacts/gaussian-splats');
+
+const ENGINE67_CASES = [
+ { name: 'part-only', source: 'synthetic', withSplats: false },
+ { name: 'millipede-behind-part', source: 'millipede', count: 200000 },
+ { name: 'millipede-alone', source: 'millipede', count: 200000, withPart: false },
+ { name: 'millipede-alone-sdr', source: 'millipede', count: 200000, withPart: false, hdr: false },
+ { name: 'millipede-behind-part-sdr', source: 'millipede', count: 200000, hdr: false },
+ { name: 'synthetic-100k', source: 'synthetic', count: 100000 },
+ { name: 'synthetic-1m', source: 'synthetic', count: 1000000 },
+];
+const THREE_CASES = [
+ { name: 'three-millipede', source: 'millipede' },
+ { name: 'three-synthetic-100k', source: 'synthetic', count: 100000 },
+ { name: 'three-synthetic-1m', source: 'synthetic', count: 1000000 },
+];
+
+setPriority(0, constants.priority.PRIORITY_BELOW_NORMAL);
+await mkdir(outputDirectory, { recursive: true });
+const server = await createServer({ server: { host: '127.0.0.1', port: 0 }, logLevel: 'silent' });
+await server.listen();
+const serverUrl = `http://127.0.0.1:${server.httpServer.address().port}`;
+const browser = await chromium.launch({ executablePath: CHROME, headless: true, args: BACKEND_ARGUMENTS[backend] });
+
+async function runCases(pagePath, cases) {
+ for (const { name, ...options } of cases.filter(entry => entry.name.includes(onlyCase))) {
+ const page = await browser.newPage({ viewport: { width: 960, height: 540 } });
+ page.on('pageerror', error => console.log('PAGE ERROR', error.message));
+ page.on('console', message => { if (message.type() === 'error') console.log('CONSOLE', message.text()); });
+ try {
+ await page.goto(`${serverUrl}/bench/gaussian-splats/${pagePath}`, { timeout: 180000 });
+ await page.waitForFunction(() => window.harnessReady, null, { timeout: 180000 });
+ const frames = backend === 'cpu' ? SWIFTSHADER_FRAMES : 60;
+ const { image, ...measurements } = await page.evaluate(parameters => window.runCase(parameters), { frames, ...options });
+ if (image) await writeFile(`${outputDirectory}/${backend}-${name}.png`, Buffer.from(image, 'base64'));
+ console.log(JSON.stringify({ name, backend, ...measurements }));
+ } catch (error) {
+ console.log(JSON.stringify({ name, backend, error: error.message.split('\n')[0] }));
+ } finally {
+ await page.close();
+ }
+ }
+}
+
+try {
+ if (engines === 'engine67' || engines === 'all') await runCases('harness.html', ENGINE67_CASES);
+ if (engines === 'three' || engines === 'all') await runCases('three.html', THREE_CASES);
+} finally {
+ await browser.close();
+ await server.close();
+}
diff --git a/bench/gaussian-splats/scene-data.js b/bench/gaussian-splats/scene-data.js
new file mode 100644
index 00000000..f21d0b82
--- /dev/null
+++ b/bench/gaussian-splats/scene-data.js
@@ -0,0 +1,72 @@
+// Splat data shared by the 67js and three.js pages of the benchmark, so both draw the same splats.
+
+function seededRandom(seed) {
+ let state = seed >>> 0;
+ return () => {
+ state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
+ return state / 4294967296;
+ };
+}
+
+// Splats on the six faces of a room 8 m wide and deep and 4 m tall, so the view holds several layers of overlap, like a captured room.
+export function syntheticRoom(count) {
+ const random = seededRandom(7);
+ const size = [8, 4, 8];
+ const splatScale = 0.67 * Math.sqrt(160 / count);
+ const positions = new Float32Array(count * 3);
+ const colors = new Uint8Array(count * 4);
+ const scales = new Float32Array(count * 3);
+ const rotations = new Float32Array(count * 4);
+ const areas = [size[1] * size[2], size[1] * size[2], size[0] * size[2], size[0] * size[2], size[0] * size[1], size[0] * size[1]];
+ const totalArea = areas.reduce((sum, area) => sum + area, 0);
+ for (let index = 0; index < count; index++) {
+ let pick = random() * totalArea;
+ let face = 0;
+ while (pick > areas[face]) pick -= areas[face++];
+ const axis = face >> 1;
+ const point = [random() - 0.5, random() - 0.5, random() - 0.5].map((value, dimension) => value * size[dimension]);
+ point[axis] = (face & 1 ? 0.5 : -0.5) * size[axis];
+ positions.set([point[0], point[1] + 2, point[2]], index * 3);
+ const tone = 0.35 + 0.4 * random();
+ colors.set([tone * 255, (tone * (0.8 + 0.2 * random())) * 255, tone * 0.7 * 255, 150 + random() * 100], index * 4);
+ const axisScale = splatScale * (0.6 + 0.8 * random());
+ scales.set([axisScale, axisScale, axisScale], index * 3);
+ rotations.set([1, 0, 0, 0], index * 4);
+ }
+ const bounds = { minimum: [-4, 0, -4], maximum: [4, 4, 4] };
+ return { count, positions, colors, scales, rotations, bounds };
+}
+
+// Places the captured millipede, which is y-down, about 4 m wide, 2 m behind the part.
+export function millipedeMatrix(bounds) {
+ const extent = [0, 1, 2].map(axis => bounds.maximum[axis] - bounds.minimum[axis]);
+ const center = [0, 1, 2].map(axis => (bounds.minimum[axis] + bounds.maximum[axis]) / 2);
+ const scale = 4 / Math.max(...extent);
+ return [scale, 0, 0, 0, 0, -scale, 0, 0, 0, 0, -scale, 0, -center[0] * scale + 0.5, center[1] * scale + 1.6, center[2] * scale - 2.2, 1];
+}
+
+const PLY_PROPERTIES = ['x', 'y', 'z', 'f_dc_0', 'f_dc_1', 'f_dc_2', 'opacity', 'scale_0', 'scale_1', 'scale_2', 'rot_0', 'rot_1', 'rot_2', 'rot_3'];
+const SPHERICAL_HARMONIC_DC = 0.28209479177387814;
+
+// Writes parsed splat data back as a standard 3DGS PLY, for loaders that only read files.
+export function splatsToPly({ count, positions, colors, scales, rotations }) {
+ const header = ['ply', 'format binary_little_endian 1.0', `element vertex ${count}`, ...PLY_PROPERTIES.map(name => `property float ${name}`), 'end_header', ''].join('\n');
+ const headerBytes = new TextEncoder().encode(header);
+ const values = new Float32Array(count * PLY_PROPERTIES.length);
+ for (let index = 0; index < count; index++) {
+ const row = index * PLY_PROPERTIES.length;
+ for (let axis = 0; axis < 3; axis++) {
+ values[row + axis] = positions[index * 3 + axis];
+ values[row + 3 + axis] = (colors[index * 4 + axis] / 255 - 0.5) / SPHERICAL_HARMONIC_DC;
+ values[row + 7 + axis] = Math.log(scales[index * 3 + axis]);
+ }
+ const opacity = Math.min(254 / 255, Math.max(1 / 255, colors[index * 4 + 3] / 255));
+ values[row + 6] = Math.log(opacity / (1 - opacity));
+ for (let component = 0; component < 4; component++) values[row + 10 + component] = rotations[index * 4 + component];
+ }
+ const file = new Uint8Array(headerBytes.length + values.byteLength);
+ file.set(headerBytes, 0);
+ file.set(new Uint8Array(values.buffer), headerBytes.length);
+ return file.buffer;
+}
+
diff --git a/bench/gaussian-splats/sort-time.mjs b/bench/gaussian-splats/sort-time.mjs
new file mode 100644
index 00000000..af666566
--- /dev/null
+++ b/bench/gaussian-splats/sort-time.mjs
@@ -0,0 +1,29 @@
+// Time of the CPU sort alone, in Node, for a camera that moves a little every frame: node bench/gaussian-splats/sort-time.mjs
+// It reports the median and the range of 40 sorts for each splat count. The browser frame time in run.mjs includes this sort.
+import { mat4 } from 'gl-matrix';
+import { createSplatSorter } from '../../src/gaussian-splat-sort.js';
+import { syntheticRoom } from './scene-data.js';
+
+const SORTS = 40;
+
+function sortTimes(count) {
+ const data = syntheticRoom(count);
+ const sorter = createSplatSorter(data.positions, count, data.bounds);
+ const view = mat4.create();
+ const times = [];
+ for (let index = 0; index < SORTS; index++) {
+ const angle = index * 0.04;
+ mat4.lookAt(view, [Math.sin(angle) * 3, 1.7, Math.cos(angle) * 3], [0, 1.5, 0], [0, 1, 0]);
+ const start = performance.now();
+ const { changed } = sorter.sort(view);
+ times.push(performance.now() - start);
+ if (!changed) throw new Error('the sort did not run');
+ }
+ return times.sort((first, second) => first - second);
+}
+
+for (const count of [100_000, 1_000_000]) {
+ const times = sortTimes(count);
+ const format = value => value.toFixed(1);
+ console.log(`${count} splats: median ${format(times[SORTS / 2])} ms, range ${format(times[0])} to ${format(times[SORTS - 1])} ms`);
+}
diff --git a/bench/gaussian-splats/three-harness.js b/bench/gaussian-splats/three-harness.js
new file mode 100644
index 00000000..d06cfe40
--- /dev/null
+++ b/bench/gaussian-splats/three-harness.js
@@ -0,0 +1,91 @@
+// Page of the Gaussian splat benchmark for three.js: the GaussianSplat object (installed three version, WebGPURenderer on its
+// WebGL 2 backend) draws the same splats, as the same PLY bytes, from the same camera as harness.js. No recipe part: three.js has
+// no 67js scene, so the comparison is splats alone against splats alone (the "millipede-alone" case of harness.js).
+import * as THREE from 'three/webgpu';
+import { GaussianSplatPLYLoader } from 'three/addons/loaders/GaussianSplatPLYLoader.js';
+import { GaussianSplat } from 'three/addons/objects/GaussianSplat.js';
+import { millipedeMatrix, splatsToPly, syntheticRoom } from './scene-data.js';
+import { parseGaussianSplatPly } from '/src/gaussian-splat-ply.js';
+
+function median(values) {
+ return values.slice().sort((first, second) => first - second)[Math.floor(values.length / 2)];
+}
+
+const synchronizationPixel = new Uint8Array(4);
+const ORBIT_STEP = 0.04;
+
+function cameraAt(angle, source) {
+ const radius = source === 'synthetic' ? 3 : 4.2;
+ const target = source === 'synthetic' ? [0, 1.5, 0] : [0, 1.3, -1];
+ return { eye: [Math.sin(angle) * radius, 1.7, Math.cos(angle) * radius], target };
+}
+
+function pointCamera(camera, angle, source) {
+ const { eye, target } = cameraAt(angle, source);
+ camera.position.set(...eye);
+ camera.lookAt(...target);
+ camera.updateMatrixWorld(true);
+}
+
+async function plyBytesFor(source, count) {
+ if (source === 'synthetic') return splatsToPly(syntheticRoom(count));
+ const response = await fetch('/bench/gaussian-splats/data/millipede.ply');
+ if (!response.ok) throw new Error('Run node bench/gaussian-splats/fetch-sample.mjs first');
+ return response.arrayBuffer();
+}
+
+async function runCase({ source = 'synthetic', count = 100000, frames = 60, width = 960, height = 540, screenshot = true }) {
+ const heapBefore = performance.memory?.usedJSHeapSize ?? 0;
+ const canvas = document.createElement('canvas');
+ canvas.width = width;
+ canvas.height = height;
+ document.body.append(canvas);
+ const renderer = new THREE.WebGPURenderer({ canvas, forceWebGL: true, antialias: true });
+ await renderer.init();
+ const scene = new THREE.Scene();
+ scene.background = new THREE.Color(0.62, 0.7, 0.67);
+ const camera = new THREE.PerspectiveCamera(50, width / height, 0.1, 200);
+ const loadStart = performance.now();
+ const bytes = await plyBytesFor(source, count);
+ const geometry = new GaussianSplatPLYLoader().parse(bytes);
+ const splats = new GaussianSplat(geometry, { autoSort: true });
+ if (source === 'millipede') {
+ const bounds = parseGaussianSplatPly(bytes).bounds;
+ splats.matrixAutoUpdate = false;
+ splats.matrix.fromArray(millipedeMatrix(bounds));
+ splats.matrixWorldNeedsUpdate = true;
+ }
+ scene.add(splats);
+ const prepareMilliseconds = performance.now() - loadStart;
+ const gl = renderer.backend.gl;
+ const frame = angle => {
+ pointCamera(camera, angle, source);
+ const start = performance.now();
+ renderer.render(scene, camera);
+ gl.bindFramebuffer(gl.FRAMEBUFFER, null);
+ gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, synchronizationPixel);
+ return performance.now() - start;
+ };
+ for (let index = 0; index < 10; index++) frame(index * ORBIT_STEP);
+ const orbitTimes = Array.from({ length: frames }, (_, index) => frame(index * ORBIT_STEP));
+ const stillTimes = Array.from({ length: frames }, () => frame(0));
+ const image = screenshot ? (frame(0.25), canvas.toDataURL('image/png').split(',')[1]) : null;
+ const geometryBytes = Object.values(geometry.attributes).reduce((total, attribute) => total + attribute.array.byteLength, 0);
+ const result = {
+ engine: 'three', source, count: geometry.attributes.position ? geometry.attributes.position.count : 0, width, height,
+ orbitMedianMilliseconds: Number(median(orbitTimes).toFixed(2)),
+ stillMedianMilliseconds: Number(median(stillTimes).toFixed(2)),
+ attributeMebibytes: Number((geometryBytes / 1048576).toFixed(1)),
+ jsHeapDeltaMebibytes: Number((((performance.memory?.usedJSHeapSize ?? 0) - heapBefore) / 1048576).toFixed(1)),
+ prepareMilliseconds: Number(prepareMilliseconds.toFixed(0)),
+ glError: gl.getError(),
+ threeRevision: THREE.REVISION,
+ image,
+ };
+ renderer.dispose();
+ canvas.remove();
+ return result;
+}
+
+window.runCase = runCase;
+window.harnessReady = true;
diff --git a/bench/gaussian-splats/three.html b/bench/gaussian-splats/three.html
new file mode 100644
index 00000000..2d12c152
--- /dev/null
+++ b/bench/gaussian-splats/three.html
@@ -0,0 +1,11 @@
+<!doctype html>
+<html>
+<head>
+<meta charset="utf-8">
+<title>three.js Gaussian splat benchmark</title>
+<style>html,body{margin:0;background:#000}canvas{display:block}</style>
+</head>
+<body>
+<script type="module" src="./three-harness.js"></script>
+</body>
+</html>
diff --git a/bench/size/budgets.json b/bench/size/budgets.json
index 9b7cc6d6..98169cf4 100644
--- a/bench/size/budgets.json
+++ b/bench/size/budgets.json
@@ -42,6 +42,7 @@
"entry:camera-effects": 8600,
"entry:water-reflection": 26580,
"entry:render-passes": 1700,
+ "entry:gaussian-splats": 11023,
"entry:material-layer-shaders": 21400,
"entry:light-linking": 19720,
"entry:cascaded-shadows": 28530,
diff --git a/changes/288.md b/changes/288.md
new file mode 100644
index 00000000..9af8f10b
--- /dev/null
+++ b/changes/288.md
@@ -0,0 +1 @@
+- Draw 3D Gaussian splats as an optional addon, `67js/addons/gaussian-splats` (issue #288). It reads a standard 3DGS PLY with a byte and splat count limit (512 MiB and 2,000,000 by default, 4,000,000 at most), and draws the splats after the opaque parts with the depth test against them, no depth write and a CPU counting sort, back to front, through a wrap of `drawBlendBatches` that needs no renderer change. The core bundle is unchanged. `docs/gaussian-splats.md` has the guide and `bench/gaussian-splats/README.md` the frame time and memory at 100,000 and 1,000,000 splats and the comparison with the three.js `GaussianSplat`. Recipes, the MCP server and the asset catalog (GLB only) do not reach it yet.
diff --git a/changes/297.md b/changes/297.md
new file mode 100644
index 00000000..8b502885
--- /dev/null
+++ b/changes/297.md
@@ -0,0 +1 @@
+- Water reflection skips the mirror draw for render pass frames (`albedo`, `normal`, `emission`, `depth`, `objectId`), which never show the reflection. A pass no longer pays for a second scene draw (closes #297).
diff --git a/docs/gaussian-splats.md b/docs/gaussian-splats.md
new file mode 100644
index 00000000..a7fff1d8
--- /dev/null
+++ b/docs/gaussian-splats.md
@@ -0,0 +1,40 @@
+# Gaussian splats
+
+`67js/addons/gaussian-splats` draws 3D Gaussian splats, the format that captured rooms, streets and products ship in, behind or among the parts of a scene. It is in the current source and not yet published. It is an optional package: the core renderer has no code for it, so the core bundle does not change, and a scene without splats never loads it. Recipes and the MCP server do not reach it yet: the local asset catalog reads GLB files only, and what the addon costs is recorded below first.
+
+```js
+import { Renderer67 } from '67js';
+import { attachGaussianSplats, createGaussianSplats, parseGaussianSplatPly } from '67js/addons/gaussian-splats';
+
+const bytes = await (await fetch('/scene.ply')).arrayBuffer();
+const data = parseGaussianSplatPly(bytes, { maximumCount: 1_000_000 });
+// Many captures are y-down: this matrix turns the scene upside down and puts it 3 m behind the origin.
+const splats = createGaussianSplats(data, { matrix: [1, 0, 0, 0, 0, -1, 0, 0, 0, 0, -1, 0, 0, 0, -3, 1] });
+
+const renderer = new Renderer67(canvas, scene.parts, { ground: false });
+const detach = attachGaussianSplats(renderer, splats);
+renderer.render(camera); // parts first, then the splats, farthest first
+detach(); // or renderer.dispose()
+```
+
+## What it reads
+
+One format: the standard 3DGS PLY that the original training code and most tools write: binary little endian with `x y z f_dc_0..2 opacity scale_0..2 rot_0..3` per vertex. Other properties (normals, `f_rest_*`) are skipped, so a splat has one colour from every direction (spherical harmonics of degree 0). `.spz`, `.ksplat` and `.splat` files are not read; `bench/gaussian-splats/fetch-sample.mjs` shows a short `.splat` to PLY conversion.
+
+A file over `maximumBytes` (default 512 MiB), or one whose header names more than `maximumCount` splats (default 2,000,000, never above 4,000,000), throws a `RangeError` before any splat is read. A file that is not a PLY, is ASCII, lacks a property or is truncated throws an `Error` that names the problem. `createGaussianSplats` applies the count limit again.
+
+## What it draws
+
+- Each splat is a screen-space quad shaped by its 3D covariance, projected through the camera as in the original paper. Colour is the stored colour, as 3DGS viewers show it, not converted from sRGB. The opacity falls off as a Gaussian toward the quad edge.
+- The splats draw after the opaque and translucent parts of the renderer, with the depth test against them and no depth write, so a part in front hides the splats behind it and splats never hide a part. A splat is tested at its centre depth, so one that pokes through a surface is not clipped along the surface.
+- Splats blend back to front. The sort is on the CPU, on the main thread: a counting sort of the view depth into 65536 steps, then one upload of the order (4 bytes per visible splat). The sort is skipped when the view depth axis is the same as in the last frame: a still camera, or one that only moves sideways or up and down. Splats behind the camera are left out.
+- `matrix` may rotate, translate and scale uniformly. Perspective cameras only. Splats are not drawn in render passes, benchmark mode or the water reflection image, and are not lit, shadowed or fogged.
+- `attachGaussianSplats` wraps `renderer.drawBlendBatches`, the step that runs once per frame after the parts and before the frame is resolved. The renderer has no other link to the addon. Attach several scenes and they draw farthest first by their bounds centre.
+
+## Memory
+
+GPU: 48 bytes per splat in one RGBA32UI texture and 4 bytes for the order buffer (52 bytes, 52 MB per million). CPU after creation: 12 bytes of positions and 8 bytes of sort scratch per splat; the parsed arrays and the packed texture data can be dropped once `createGaussianSplats` has returned and the splats are attached. The parse itself needs the file plus 44 bytes per splat.
+
+## What it costs
+
+See [bench/gaussian-splats/README.md](../bench/gaussian-splats/README.md) for frame times and memory at 100,000 and 1,000,000 splats on an RTX 3060 and on SwiftShader, and the comparison with the three.js `GaussianSplat`.
diff --git a/package.json b/package.json
index 87e319e5..0d63acfd 100644
--- a/package.json
+++ b/package.json
@@ -86,6 +86,7 @@
"./addons/material-layer-shaders": "./build/material-layer-shaders.js",
"./addons/light-linking": "./build/light-linking.js",
"./addons/cascaded-shadows": "./build/cascaded-shadows.js",
+ "./addons/gaussian-splats": "./build/gaussian-splats.js",
"./package.json": "./package.json"
},
"scripts": {
diff --git a/scripts/prepare-release.mjs b/scripts/prepare-release.mjs
index c6e4b24e..9a085d03 100644
--- a/scripts/prepare-release.mjs
+++ b/scripts/prepare-release.mjs
@@ -43,7 +43,7 @@ const sourceDocuments = [
'retarget.md',
'skeleton-pose.md',
'vehicle.md',
- 'vegetation.md', 'terrain.md', 'grass.md', 'clutter.md', 'light-bake.md', 'path-tracer.md',
+ 'vegetation.md', 'terrain.md', 'grass.md', 'clutter.md', 'light-bake.md', 'path-tracer.md', 'gaussian-splats.md',
'camera-path.md', 'lighting.md', 'volumetric.md', 'film-physics.md',
'surface-attachment.md', 'agent-scene.md', 'mcp.md', 'agent-look.md', 'agent-asset.md', 'agent-render-jobs.md', 'agent-motion.md', 'agent-character-motion.md', 'agent-review.md',
];
diff --git a/src/gaussian-splat-ply.js b/src/gaussian-splat-ply.js
new file mode 100644
index 00000000..9e4396b5
--- /dev/null
+++ b/src/gaussian-splat-ply.js
@@ -0,0 +1,180 @@
+// Reads a standard 3D Gaussian splat PLY: the file written by the original 3DGS training code and by most tools that
+// export splats. It is binary little endian with one vertex per splat and the properties x, y, z, f_dc_0 to f_dc_2,
+// opacity, scale_0 to scale_2 and rot_0 to rot_3. Higher-order spherical harmonics (f_rest_*) and normals are skipped,
+// so a splat shows one colour from every direction. Other splat formats (.spz, .ksplat, .splat) are not read here.
+
+export const GAUSSIAN_SPLAT_LIMITS = Object.freeze({
+ // A file this large cannot be read into memory safely in a browser tab.
+ maximumBytes: 512 * 1024 * 1024,
+ maximumCount: 2_000_000,
+});
+/** No option raises the splat count above this: the data texture then needs 192 MB of GPU memory. */
+export const GAUSSIAN_SPLAT_HARD_COUNT_LIMIT = 4_000_000;
+
+const REQUIRED_PROPERTIES = ['x', 'y', 'z', 'f_dc_0', 'f_dc_1', 'f_dc_2', 'opacity', 'scale_0', 'scale_1', 'scale_2', 'rot_0', 'rot_1', 'rot_2', 'rot_3'];
+const SPHERICAL_HARMONIC_DC = 0.28209479177387814;
+const HEADER_END = 'end_header\n';
+const MAXIMUM_HEADER_BYTES = 64 * 1024;
+
+const PROPERTY_TYPES = {
+ char: { bytes: 1, read: (view, offset) => view.getInt8(offset) },
+ int8: { bytes: 1, read: (view, offset) => view.getInt8(offset) },
+ uchar: { bytes: 1, read: (view, offset) => view.getUint8(offset) },
+ uint8: { bytes: 1, read: (view, offset) => view.getUint8(offset) },
+ short: { bytes: 2, read: (view, offset) => view.getInt16(offset, true) },
+ int16: { bytes: 2, read: (view, offset) => view.getInt16(offset, true) },
+ ushort: { bytes: 2, read: (view, offset) => view.getUint16(offset, true) },
+ uint16: { bytes: 2, read: (view, offset) => view.getUint16(offset, true) },
+ int: { bytes: 4, read: (view, offset) => view.getInt32(offset, true) },
+ int32: { bytes: 4, read: (view, offset) => view.getInt32(offset, true) },
+ uint: { bytes: 4, read: (view, offset) => view.getUint32(offset, true) },
+ uint32: { bytes: 4, read: (view, offset) => view.getUint32(offset, true) },
+ float: { bytes: 4, read: (view, offset) => view.getFloat32(offset, true) },
+ float32: { bytes: 4, read: (view, offset) => view.getFloat32(offset, true) },
+ double: { bytes: 8, read: (view, offset) => view.getFloat64(offset, true) },
+ float64: { bytes: 8, read: (view, offset) => view.getFloat64(offset, true) },
+};
+
+function readHeaderText(bytes) {
+ const searchLimit = Math.min(bytes.length, MAXIMUM_HEADER_BYTES);
+ const text = new TextDecoder('latin1').decode(bytes.subarray(0, searchLimit));
+ const endIndex = text.indexOf(HEADER_END);
+ if (!text.startsWith('ply') || endIndex < 0) {
+ throw new Error('Not a PLY file: the header must start with "ply" and end with "end_header"');
+ }
+ return { text: text.slice(0, endIndex), bodyStart: endIndex + HEADER_END.length };
+}
+
+function parseHeader(headerText) {
+ const lines = headerText.split(/\r?\n/).map(line => line.trim()).filter(Boolean);
+ const formatLine = lines.find(line => line.startsWith('format '));
+ if (formatLine?.split(/\s+/)[1] !== 'binary_little_endian') {
+ throw new Error(`Gaussian splat PLY must be binary_little_endian, got "${formatLine ?? 'no format line'}"`);
+ }
+ let vertexCount = 0;
+ let inVertexElement = false;
+ let vertexElementSeen = false;
+ const properties = [];
+ for (const line of lines) {
+ const words = line.split(/\s+/);
+ if (words[0] === 'element') {
+ inVertexElement = words[1] === 'vertex' && !vertexElementSeen;
+ if (inVertexElement) {
+ vertexElementSeen = true;
+ vertexCount = Number(words[2]);
+ }
+ } else if (words[0] === 'property' && inVertexElement) {
+ if (words[1] === 'list' || !PROPERTY_TYPES[words[1]]) {
+ throw new Error(`Gaussian splat PLY has an unsupported property type "${words[1]}"`);
+ }
+ properties.push({ name: words[2], type: words[1] });
+ }
+ }
+ if (!Number.isInteger(vertexCount) || vertexCount <= 0) {
+ throw new Error('Gaussian splat PLY has no vertex element with a positive count');
+ }
+ return { vertexCount, properties };
+}
+
+function propertyLayout(properties) {
+ const layout = new Map();
+ let offset = 0;
+ for (const { name, type } of properties) {
+ layout.set(name, { offset, read: PROPERTY_TYPES[type].read });
+ offset += PROPERTY_TYPES[type].bytes;
+ }
+ const missing = REQUIRED_PROPERTIES.filter(name => !layout.has(name));
+ if (missing.length > 0) {
+ throw new Error(`Gaussian splat PLY lacks the properties ${missing.join(', ')}`);
+ }
+ return { layout, stride: offset };
+}
+
+function resolveLimits(options) {
+ const maximumBytes = options.maximumBytes ?? GAUSSIAN_SPLAT_LIMITS.maximumBytes;
+ const maximumCount = options.maximumCount ?? GAUSSIAN_SPLAT_LIMITS.maximumCount;
+ if (!(maximumBytes > 0) || !(maximumCount > 0)) {
+ throw new RangeError('maximumBytes and maximumCount must be positive');
+ }
+ return { maximumBytes, maximumCount: Math.min(maximumCount, GAUSSIAN_SPLAT_HARD_COUNT_LIMIT) };
+}
+
+function toByte(unit) {
+ return Math.max(0, Math.min(255, Math.round(unit * 255)));
+}
+
+/**
+ * Parses a standard 3DGS PLY into flat typed arrays. Throws a RangeError, before reading any splat, when the file is
+ * larger than `maximumBytes` (default 512 MiB) or holds more than `maximumCount` splats (default 2,000,000; never more
+ * than 4,000,000). Returns `{ count, positions, colors, scales, rotations, bounds }`: positions are xyz, colors are
+ * RGBA bytes (RGB from the DC spherical harmonic, A from the sigmoid of opacity), scales are the exponentiated
+ * axis lengths in metres and rotations are unit quaternions in w, x, y, z order.
+ */
+export function parseGaussianSplatPly(buffer, options = {}) {
+ const { maximumBytes, maximumCount } = resolveLimits(options);
+ if (!(buffer instanceof ArrayBuffer) && !ArrayBuffer.isView(buffer)) {
+ throw new TypeError('parseGaussianSplatPly expects an ArrayBuffer or a typed array');
+ }
+ if (buffer.byteLength > maximumBytes) {
+ throw new RangeError(`Gaussian splat file is ${buffer.byteLength} bytes, over the limit of ${maximumBytes}`);
+ }
+ const bytes = buffer instanceof ArrayBuffer ? new Uint8Array(buffer) : new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength);
+ const { text, bodyStart } = readHeaderText(bytes);
+ const { vertexCount, properties } = parseHeader(text);
+ if (vertexCount > maximumCount) {
+ throw new RangeError(`Gaussian splat file holds ${vertexCount} splats, over the limit of ${maximumCount}`);
+ }
+ const { layout, stride } = propertyLayout(properties);
+ if (bytes.length - bodyStart < vertexCount * stride) {
+ throw new Error(`Gaussian splat PLY is truncated: ${vertexCount} splats need ${vertexCount * stride} bytes, ${bytes.length - bodyStart} are present`);
+ }
+ const body = new DataView(bytes.buffer, bytes.byteOffset + bodyStart, vertexCount * stride);
+ return readSplats(body, vertexCount, layout, stride);
+}
+
+function readSplats(view, count, layout, stride) {
+ const positions = new Float32Array(count * 3);
+ const colors = new Uint8Array(count * 4);
+ const scales = new Float32Array(count * 3);
+ const rotations = new Float32Array(count * 4);
+ const minimum = [Infinity, Infinity, Infinity];
+ const maximum = [-Infinity, -Infinity, -Infinity];
+ const positionFields = ['x', 'y', 'z'].map(name => layout.get(name));
+ const colorFields = ['f_dc_0', 'f_dc_1', 'f_dc_2'].map(name => layout.get(name));
+ const opacityField = layout.get('opacity');
+ const scaleFields = ['scale_0', 'scale_1', 'scale_2'].map(name => layout.get(name));
+ const rotationFields = ['rot_0', 'rot_1', 'rot_2', 'rot_3'].map(name => layout.get(name));
+ for (let index = 0; index < count; index++) {
+ const row = index * stride;
+ for (let axis = 0; axis < 3; axis++) {
+ const field = positionFields[axis];
+ const coordinate = field.read(view, row + field.offset);
+ if (!Number.isFinite(coordinate)) {
+ throw new Error(`Gaussian splat ${index} has a non-finite position`);
+ }
+ positions[index * 3 + axis] = coordinate;
+ if (coordinate < minimum[axis]) minimum[axis] = coordinate;
+ if (coordinate > maximum[axis]) maximum[axis] = coordinate;
+ const colorField = colorFields[axis];
+ colors[index * 4 + axis] = toByte(0.5 + SPHERICAL_HARMONIC_DC * colorField.read(view, row + colorField.offset));
+ const scaleField = scaleFields[axis];
+ scales[index * 3 + axis] = Math.exp(scaleField.read(view, row + scaleField.offset));
+ }
+ colors[index * 4 + 3] = toByte(1 / (1 + Math.exp(-opacityField.read(view, row + opacityField.offset))));
+ writeUnitQuaternion(rotations, index, view, row, rotationFields);
+ }
+ return { count, positions, colors, scales, rotations, bounds: { minimum, maximum } };
+}
+
+function writeUnitQuaternion(rotations, index, view, row, fields) {
+ const base = index * 4;
+ for (let component = 0; component < 4; component++) {
+ rotations[base + component] = fields[component].read(view, row + fields[component].offset);
+ }
+ const length = Math.hypot(rotations[base], rotations[base + 1], rotations[base + 2], rotations[base + 3]);
+ if (length > 1e-12) {
+ for (let component = 0; component < 4; component++) rotations[base + component] /= length;
+ } else {
+ rotations.set([1, 0, 0, 0], base);
+ }
+}
diff --git a/src/gaussian-splat-sort.js b/src/gaussian-splat-sort.js
new file mode 100644
index 00000000..72f32c9a
--- /dev/null
+++ b/src/gaussian-splat-sort.js
@@ -0,0 +1,68 @@
+// Back-to-front order of Gaussian splats for one view. Translucent splats blend correctly only when the farthest is
+// drawn first, so every change of view re-sorts them. A counting sort on the view depth quantised to 65536 steps
+// does that in three linear passes (bench/gaussian-splats/README.md has the time). Splats behind the camera are left
+// out of the order. Splats closer than one step of depth (the span divided by 65536) may draw in either order.
+
+const DEPTH_STEPS = 65536;
+const UNCHANGED_TOLERANCE = 1e-6;
+
+function viewDepthRange(modelView, { minimum, maximum }) {
+ let nearest = Infinity;
+ let farthest = -Infinity;
+ for (let corner = 0; corner < 8; corner++) {
+ const x = corner & 1 ? maximum[0] : minimum[0];
+ const y = corner & 2 ? maximum[1] : minimum[1];
+ const z = corner & 4 ? maximum[2] : minimum[2];
+ const depth = modelView[2] * x + modelView[6] * y + modelView[10] * z + modelView[14];
+ nearest = Math.min(nearest, depth);
+ farthest = Math.max(farthest, depth);
+ }
+ return { nearest, farthest };
+}
+
+/**
+ * Creates a sorter for splats at `positions` (xyz triples, `count` of them) inside `bounds`. `sort(modelView)` takes the
+ * column-major model-view matrix and returns `{ order, visibleCount, changed }`: `order[0 .. visibleCount)` holds splat
+ * indices from the farthest to the nearest in front of the camera. `order` is reused between calls, and `changed` is
+ * false when the view depth axis has not moved since the previous call, so the caller can skip its upload.
+ */
+export function createSplatSorter(positions, count, bounds) {
+ const keys = new Int32Array(count);
+ const order = new Uint32Array(count);
+ const bucketStarts = new Uint32Array(DEPTH_STEPS + 1);
+ const lastAxis = new Float64Array(4).fill(NaN);
+ let visibleCount = 0;
+
+ function sort(modelView) {
+ const axis = [modelView[2], modelView[6], modelView[10], modelView[14]];
+ const unchanged = axis.every((value, index) => Math.abs(value - lastAxis[index]) <= UNCHANGED_TOLERANCE);
+ if (unchanged) return { order, visibleCount, changed: false };
+ lastAxis.set(axis);
+ const { nearest, farthest } = viewDepthRange(modelView, bounds);
+ const frontLimit = Math.min(farthest, 0);
+ const span = frontLimit - nearest;
+ const scale = span > 0 ? (DEPTH_STEPS - 1) / span : 0;
+ bucketStarts.fill(0);
+ let kept = 0;
+ for (let index = 0; index < count; index++) {
+ const depth = axis[0] * positions[index * 3] + axis[1] * positions[index * 3 + 1] + axis[2] * positions[index * 3 + 2] + axis[3];
+ if (depth >= 0) {
+ keys[index] = -1;
+ continue;
+ }
+ const key = Math.min(DEPTH_STEPS - 1, Math.max(0, ((depth - nearest) * scale) | 0));
+ keys[index] = key;
+ bucketStarts[key + 1]++;
+ kept++;
+ }
+ for (let key = 0; key < DEPTH_STEPS; key++) bucketStarts[key + 1] += bucketStarts[key];
+ for (let index = 0; index < count; index++) {
+ const key = keys[index];
+ if (key >= 0) order[bucketStarts[key]++] = index;
+ }
+ visibleCount = kept;
+ return { order, visibleCount, changed: true };
+ }
+
+ return { sort, get visibleCount() { return visibleCount; } };
+}
diff --git a/src/gaussian-splats.js b/src/gaussian-splats.js
new file mode 100644
index 00000000..9cea8321
--- /dev/null
+++ b/src/gaussian-splats.js
@@ -0,0 +1,290 @@
+import { mat4 } from 'gl-matrix';
+import { compileProgram } from './gl-program.js';
+import { createSplatSorter } from './gaussian-splat-sort.js';
+import { GAUSSIAN_SPLAT_HARD_COUNT_LIMIT, GAUSSIAN_SPLAT_LIMITS, parseGaussianSplatPly } from './gaussian-splat-ply.js';
+
+export { GAUSSIAN_SPLAT_HARD_COUNT_LIMIT, GAUSSIAN_SPLAT_LIMITS, parseGaussianSplatPly };
+
+// Gaussian splats as an optional addon (67js/addons/gaussian-splats). A splat is an ellipsoid with a colour and an opacity;
+// each is drawn as a screen-space quad whose shape comes from projecting its 3D covariance. The splats are drawn after
+// the opaque and translucent parts of the renderer, with the depth test against them and no depth write, back to front.
+// The renderer needs no code for this: attachGaussianSplats wraps its drawBlendBatches step, which runs once per frame
+// after those parts and before the frame is resolved to the canvas. Perspective cameras only.
+// The per-splat data sits in one RGBA32UI texture, three texels per splat:
+// - texel 0: centre x, y, z as float bits and the colour as four bytes;
+// - texels 1 and 2: the six unique values of the 3D covariance (xx, xy, xz, yy, yz, zz) as float bits.
+// The only per-frame upload is the sort order, one 32-bit index per visible splat.
+
+const DATA_TEXTURE_WIDTH = 2048;
+const TEXELS_PER_SPLAT = 3;
+const WORDS_PER_TEXEL = 4;
+const MAXIMUM_SPLAT_COUNT = GAUSSIAN_SPLAT_HARD_COUNT_LIMIT;
+const DATA_TEXTURE_UNIT = 10;
+
+const VERTEX_SOURCE = `#version 300 es
+precision highp float;
+precision highp int;
+precision highp usampler2D;
+uniform usampler2D splatData;
+uniform mat4 modelView;
+uniform mat4 projection;
+uniform vec2 focal;
+uniform vec2 viewport;
+layout(location = 0) in uint splatIndex;
+out vec4 vertexColor;
+out vec2 vertexOffset;
+ivec2 texelOf(uint splat, uint texel) {
+ uint linear = splat * 3u + texel;
+ return ivec2(int(linear & 2047u), int(linear >> 11));
+}
+void main() {
+ uvec4 first = texelFetch(splatData, texelOf(splatIndex, 0u), 0);
+ uvec4 second = texelFetch(splatData, texelOf(splatIndex, 1u), 0);
+ uvec4 third = texelFetch(splatData, texelOf(splatIndex, 2u), 0);
+ vec3 center = uintBitsToFloat(first.xyz);
+ vec4 cameraSpace = modelView * vec4(center, 1.0);
+ vec4 clip = projection * cameraSpace;
+ if (clip.z < -clip.w || abs(clip.x) > 1.3 * clip.w || abs(clip.y) > 1.3 * clip.w) {
+ gl_Position = vec4(0.0, 0.0, 2.0, 1.0);
+ return;
+ }
+ vec4 firstCovariance = uintBitsToFloat(second);
+ vec2 secondCovariance = uintBitsToFloat(third.xy);
+ mat3 covariance = mat3(
+ firstCovariance.x, firstCovariance.y, firstCovariance.z,
+ firstCovariance.y, firstCovariance.w, secondCovariance.x,
+ firstCovariance.z, secondCovariance.x, secondCovariance.y);
+ float depth = -cameraSpace.z;
+ mat3 jacobian = mat3(
+ focal.x / depth, 0.0, 0.0,
+ 0.0, focal.y / depth, 0.0,
+ focal.x * cameraSpace.x / (depth * depth), focal.y * cameraSpace.y / (depth * depth), 0.0);
+ mat3 transform = jacobian * mat3(modelView);
+ mat3 screen = transform * covariance * transpose(transform);
+ float a = screen[0][0] + 0.3;
+ float b = screen[0][1];
+ float c = screen[1][1] + 0.3;
+ float middle = 0.5 * (a + c);
+ float radius = length(vec2(0.5 * (a - c), b));
+ float majorVariance = middle + radius;
+ float minorVariance = max(middle - radius, 0.1);
+ vec2 direction = vec2(b, majorVariance - a);
+ direction = dot(direction, direction) < 1e-12 ? vec2(1.0, 0.0) : normalize(direction);
+ vec2 majorAxis = min(sqrt(2.0 * majorVariance), 1024.0) * direction;
+ vec2 minorAxis = min(sqrt(2.0 * minorVariance), 1024.0) * vec2(direction.y, -direction.x);
+ vec2 corner = vec2(float(gl_VertexID & 1), float(gl_VertexID >> 1)) * 4.0 - 2.0;
+ vec2 offset = (corner.x * majorAxis + corner.y * minorAxis) * 2.0 / viewport;
+ vertexColor = vec4(float(first.w & 255u), float((first.w >> 8) & 255u), float((first.w >> 16) & 255u), float(first.w >> 24)) / 255.0;
+ vertexOffset = corner;
+ gl_Position = vec4(clip.xy / clip.w + offset, clip.z / clip.w, 1.0);
+}`;
+
+// A splat colour is display-referred, so the sRGB path takes it as is; the HDR path linearises it and applies exposure.
+const FRAGMENT_SOURCE = `#version 300 es
+precision highp float;
+uniform float hdrEnabled;
+uniform float exposure;
+in vec4 vertexColor;
+in vec2 vertexOffset;
+out vec4 outputColor;
+void main() {
+ float falloff = -dot(vertexOffset, vertexOffset);
+ if (falloff < -4.0) discard;
+ float alpha = exp(falloff) * vertexColor.a;
+ if (alpha < 1.0 / 255.0) discard;
+ vec3 color = hdrEnabled > 0.5 ? pow(vertexColor.rgb, vec3(2.2)) * exposure : vertexColor.rgb;
+ outputColor = vec4(color * alpha, alpha);
+}`;
+
+const UNIFORM_NAMES = ['splatData', 'modelView', 'projection', 'focal', 'viewport', 'hdrEnabled', 'exposure'];
+
+function covarianceOf(scales, rotations, index) {
+ const [w, x, y, z] = rotations.subarray(index * 4, index * 4 + 4);
+ const rotation = [
+ 1 - 2 * (y * y + z * z), 2 * (x * y + w * z), 2 * (x * z - w * y),
+ 2 * (x * y - w * z), 1 - 2 * (x * x + z * z), 2 * (y * z + w * x),
+ 2 * (x * z + w * y), 2 * (y * z - w * x), 1 - 2 * (x * x + y * y),
+ ];
+ // rotation is stored by columns; scaling the columns by the axis lengths gives M = R S.
+ const scaled = rotation.map((value, entry) => value * scales[index * 3 + Math.floor(entry / 3)]);
+ const product = (row, column) => scaled[row] * scaled[column] + scaled[3 + row] * scaled[3 + column] + scaled[6 + row] * scaled[6 + column];
+ return [product(0, 0), product(0, 1), product(0, 2), product(1, 1), product(1, 2), product(2, 2)];
+}
+
+function packSplats(data) {
+ const texelCount = data.count * TEXELS_PER_SPLAT;
+ const rows = Math.ceil(texelCount / DATA_TEXTURE_WIDTH);
+ const words = new Uint32Array(rows * DATA_TEXTURE_WIDTH * WORDS_PER_TEXEL);
+ const floats = new Float32Array(words.buffer);
+ const colorWords = new Uint32Array(data.colors.buffer, data.colors.byteOffset, data.count);
+ for (let index = 0; index < data.count; index++) {
+ const base = index * TEXELS_PER_SPLAT * WORDS_PER_TEXEL;
+ floats.set(data.positions.subarray(index * 3, index * 3 + 3), base);
+ words[base + 3] = colorWords[index];
+ floats.set(covarianceOf(data.scales, data.rotations, index), base + 4);
+ }
+ return { words, rows };
+}
+
+/**
+ * Splat data ready to draw, from `parseGaussianSplatPly`. `matrix` (a 16-number column-major array, default identity) places
+ * the splat scene in the world; it may rotate, translate and scale uniformly. Many captures are y-down, so a flip is
+ * usual. Throws a RangeError when `data.count` is over `maximumCount` (default 2,000,000, at most 4,000,000).
+ * The returned object has `count`, `matrix`, `bounds` and `statistics` (`count`, `visibleCount`, `sortMilliseconds`,
+ * `gpuBytes`, `cpuBytes`).
+ */
+export function createGaussianSplats(data, { matrix = mat4.create(), maximumCount = GAUSSIAN_SPLAT_LIMITS.maximumCount } = {}) {
+ const limit = Math.min(maximumCount, MAXIMUM_SPLAT_COUNT);
+ if (!(data?.count > 0)) throw new Error('createGaussianSplats needs parsed splat data with at least one splat');
+ if (data.count > limit) throw new RangeError(`Gaussian splat scene holds ${data.count} splats, over the limit of ${limit}`);
+ if (matrix.length !== 16 || !Array.from(matrix).every(Number.isFinite)) throw new TypeError('matrix must be 16 finite numbers');
+ const packed = packSplats(data);
+ const sorter = createSplatSorter(data.positions, data.count, data.bounds);
+ const statistics = { count: data.count, visibleCount: 0, sortMilliseconds: 0, gpuBytes: 0, cpuBytes: data.positions.byteLength + data.count * 8 };
+ return { count: data.count, matrix: mat4.clone(matrix), bounds: data.bounds, statistics, packed, sorter };
+}
+
+function createGpuResources(context, splats) {
+ const { words, rows } = splats.packed;
+ const texture = context.createTexture();
+ context.activeTexture(context.TEXTURE0 + DATA_TEXTURE_UNIT);
+ context.bindTexture(context.TEXTURE_2D, texture);
+ context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST);
+ context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST);
+ context.texImage2D(context.TEXTURE_2D, 0, context.RGBA32UI, DATA_TEXTURE_WIDTH, rows, 0, context.RGBA_INTEGER, context.UNSIGNED_INT, words);
+ const orderBuffer = context.createBuffer();
+ context.bindBuffer(context.ARRAY_BUFFER, orderBuffer);
+ context.bufferData(context.ARRAY_BUFFER, splats.count * 4, context.DYNAMIC_DRAW);
+ const vertexArray = context.createVertexArray();
+ context.bindVertexArray(vertexArray);
+ context.enableVertexAttribArray(0);
+ context.vertexAttribIPointer(0, 1, context.UNSIGNED_INT, 0, 0);
+ context.vertexAttribDivisor(0, 1);
+ context.bindVertexArray(null);
+ splats.statistics.gpuBytes = words.byteLength + splats.count * 4;
+ splats.packed = null;
+ return { texture, orderBuffer, vertexArray };
+}
+
+const attachments = new WeakMap();
+
+function createAttachment(renderer) {
+ const context = renderer.context;
+ const program = compileProgram(context, VERTEX_SOURCE, FRAGMENT_SOURCE);
+ const uniforms = Object.fromEntries(UNIFORM_NAMES.map(name => [name, context.getUniformLocation(program, name)]));
+ const attachment = { program, uniforms, entries: new Set(), modelView: mat4.create(), original: null, disposeOriginal: null };
+ attachment.original = renderer.drawBlendBatches;
+ attachment.disposeOriginal = renderer.dispose;
+ renderer.drawBlendBatches = function drawBlendBatchesAndSplats(frame) {
+ attachment.original.call(this, frame);
+ drawEntries(this, attachment, frame);
+ };
+ renderer.dispose = function disposeWithSplats() {
+ releaseAttachment(renderer, attachment);
+ return attachment.disposeOriginal.call(this);
+ };
+ return attachment;
+}
+
+function releaseEntry(context, entry) {
+ context.deleteTexture(entry.texture);
+ context.deleteBuffer(entry.orderBuffer);
+ context.deleteVertexArray(entry.vertexArray);
+}
+
+function releaseAttachment(renderer, attachment) {
+ for (const entry of attachment.entries) releaseEntry(renderer.context, entry);
+ attachment.entries.clear();
+ renderer.context.deleteProgram(attachment.program);
+ renderer.drawBlendBatches = attachment.original;
+ renderer.dispose = attachment.disposeOriginal;
+ attachments.delete(renderer);
+}
+
+function prepareState(renderer, attachment, frame) {
+ const context = renderer.context;
+ const width = renderer.canvas.width;
+ const height = renderer.canvas.height;
+ context.bindFramebuffer(context.FRAMEBUFFER, renderer.multisampleFramebuffer);
+ context.viewport(0, 0, width, height);
+ context.enable(context.DEPTH_TEST);
+ context.depthMask(false);
+ context.disable(context.CULL_FACE);
+ context.enable(context.BLEND);
+ context.blendFuncSeparate(context.ONE, context.ONE_MINUS_SRC_ALPHA, context.ONE, context.ONE_MINUS_SRC_ALPHA);
+ context.useProgram(attachment.program);
+ const { uniforms } = attachment;
+ context.uniformMatrix4fv(uniforms.projection, false, renderer.projection);
+ context.uniform2f(uniforms.focal, renderer.projection[0] * width * 0.5, renderer.projection[5] * height * 0.5);
+ context.uniform2f(uniforms.viewport, width, height);
+ context.uniform1f(uniforms.hdrEnabled, renderer.hdr ? 1 : 0);
+ context.uniform1f(uniforms.exposure, frame.exposure ?? 1);
+ context.uniform1i(uniforms.splatData, DATA_TEXTURE_UNIT);
+}
+
+function restoreState(renderer) {
+ const context = renderer.context;
+ context.depthMask(true);
+ context.disable(context.BLEND);
+ context.enable(context.CULL_FACE);
+ context.bindVertexArray(null);
+ context.activeTexture(context.TEXTURE0);
+}
+
+function drawEntry(renderer, attachment, entry) {
+ const context = renderer.context;
+ const { splats } = entry;
+ mat4.multiply(attachment.modelView, renderer.view, splats.matrix);
+ const start = performance.now();
+ const { order, visibleCount, changed } = splats.sorter.sort(attachment.modelView);
+ if (changed) {
+ context.bindBuffer(context.ARRAY_BUFFER, entry.orderBuffer);
+ context.bufferSubData(context.ARRAY_BUFFER, 0, order, 0, visibleCount);
+ splats.statistics.sortMilliseconds = performance.now() - start;
+ }
+ splats.statistics.visibleCount = visibleCount;
+ if (visibleCount === 0) return;
+ context.uniformMatrix4fv(attachment.uniforms.modelView, false, attachment.modelView);
+ context.activeTexture(context.TEXTURE0 + DATA_TEXTURE_UNIT);
+ context.bindTexture(context.TEXTURE_2D, entry.texture);
+ context.bindVertexArray(entry.vertexArray);
+ context.bindBuffer(context.ARRAY_BUFFER, entry.orderBuffer);
+ context.drawArraysInstanced(context.TRIANGLE_STRIP, 0, 4, visibleCount);
+}
+
+function drawEntries(renderer, attachment, frame) {
+ if (attachment.entries.size === 0) return;
+ prepareState(renderer, attachment, frame);
+ const farToNear = [...attachment.entries].map(entry => ({ entry, distance: distanceToEye(entry.splats, frame.eye) })).sort((first, second) => second.distance - first.distance);
+ for (const { entry } of farToNear) drawEntry(renderer, attachment, entry);
+ restoreState(renderer);
+}
+
+function distanceToEye(splats, eye) {
+ const { minimum, maximum } = splats.bounds;
+ const center = [0, 1, 2].map(axis => (minimum[axis] + maximum[axis]) / 2);
+ const world = [splats.matrix[0] * center[0] + splats.matrix[4] * center[1] + splats.matrix[8] * center[2] + splats.matrix[12],
+ splats.matrix[1] * center[0] + splats.matrix[5] * center[1] + splats.matrix[9] * center[2] + splats.matrix[13],
+ splats.matrix[2] * center[0] + splats.matrix[6] * center[1] + splats.matrix[10] * center[2] + splats.matrix[14]];
+ return Math.hypot(world[0] - eye[0], world[1] - eye[1], world[2] - eye[2]);
+}
+
+/**
+ * Draws `splats` (from `createGaussianSplats`) with `renderer` on every normal frame, after its parts and with their
+ * depth. Call it again for more splat scenes: they are drawn farthest first. Returns a function that removes this one
+ * and frees its GPU memory; `renderer.dispose()` removes all of them. Not drawn in render passes, benchmark mode or
+ * the water reflection image.
+ */
+export function attachGaussianSplats(renderer, splats) {
+ if (!splats?.packed) throw new Error('These splats are already attached to a renderer; create them again to attach to another');
+ if (typeof renderer?.drawBlendBatches !== 'function') throw new TypeError('attachGaussianSplats needs a Renderer67');
+ const attachment = attachments.get(renderer) ?? createAttachment(renderer);
+ attachments.set(renderer, attachment);
+ const entry = { splats, ...createGpuResources(renderer.context, splats) };
+ attachment.entries.add(entry);
+ return function detachGaussianSplats() {
+ if (!attachment.entries.delete(entry)) return;
+ releaseEntry(renderer.context, entry);
+ if (attachment.entries.size === 0 && attachments.get(renderer) === attachment) releaseAttachment(renderer, attachment);
+ };
+}
diff --git a/src/water-reflection.js b/src/water-reflection.js
index 0e9e8289..82d99026 100644
--- a/src/water-reflection.js
+++ b/src/water-reflection.js
@@ -247,7 +247,8 @@ export function attachWaterReflection(renderer, surface, { scale = WATER_REFLECT
const dispose = renderer.dispose;
const renderFrame = frame => render.call(renderer, frame);
renderer.render = frame => {
- if (!frame.reflectionTarget) {
+ // A render pass (albedo, depth, object ID...) never shows the reflection, so it skips the mirror draw.
+ if (!frame.reflectionTarget && frame.pass === undefined) {
drawMirrorImage(renderer, state, frame, renderFrame);
}
return render.call(renderer, frame);
diff --git a/tests/gaussian-splats.test.js b/tests/gaussian-splats.test.js
new file mode 100644
index 00000000..49000d53
--- /dev/null
+++ b/tests/gaussian-splats.test.js
@@ -0,0 +1,149 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import { createGaussianSplats, parseGaussianSplatPly } from '../src/gaussian-splats.js';
+import { createSplatSorter } from '../src/gaussian-splat-sort.js';
+
+const PROPERTIES = ['x', 'y', 'z', 'f_dc_0', 'f_dc_1', 'f_dc_2', 'opacity', 'scale_0', 'scale_1', 'scale_2', 'rot_0', 'rot_1', 'rot_2', 'rot_3'];
+
+// splats: arrays of the 14 values in PROPERTIES order
+function buildPly(splats, { properties = PROPERTIES, format = 'binary_little_endian', extraBytes = 0 } = {}) {
+ const header = ['ply', `format ${format} 1.0`, `element vertex ${splats.length}`, ...properties.map(name => `property float ${name}`), 'end_header', ''].join('\n');
+ const headerBytes = new TextEncoder().encode(header);
+ const body = new Float32Array(splats.length * properties.length + extraBytes / 4);
+ splats.forEach((values, index) => body.set(values, index * properties.length));
+ const file = new Uint8Array(headerBytes.length + body.byteLength);
+ file.set(headerBytes, 0);
+ file.set(new Uint8Array(body.buffer), headerBytes.length);
+ return file.buffer;
+}
+
+const IDENTITY_SPLAT = (x, y, z) => [x, y, z, 0, 0, 0, 0, Math.log(0.1), Math.log(0.2), Math.log(0.3), 1, 0, 0, 0];
+
+test('parses positions, colour, opacity, scale and rotation of a synthetic PLY', () => {
+ const data = parseGaussianSplatPly(buildPly([IDENTITY_SPLAT(1, 2, 3), [-1, 0, 4, 1, -1, 0, 3, 0, 0, 0, 2, 0, 0, 0]]));
+ assert.equal(data.count, 2);
+ assert.deepEqual(Array.from(data.positions), [1, 2, 3, -1, 0, 4]);
+ assert.deepEqual(data.bounds, { minimum: [-1, 0, 3], maximum: [1, 2, 4] });
+ assert.deepEqual(Array.from(data.colors.subarray(0, 4)), [128, 128, 128, 128]);
+ const bright = 0.5 + 0.28209479177387814;
+ assert.equal(data.colors[4], Math.round(bright * 255));
+ assert.equal(data.colors[5], Math.round((0.5 - 0.28209479177387814) * 255));
+ assert.equal(data.colors[7], Math.round(255 / (1 + Math.exp(-3))));
+ assert.deepEqual(Array.from(data.scales.subarray(0, 3)).map(value => Number(value.toFixed(4))), [0.1, 0.2, 0.3]);
+ assert.deepEqual(Array.from(data.rotations.subarray(4, 8)), [1, 0, 0, 0], 'a rotation of length 2 is normalised');
+});
+
+test('accepts properties in any order and extra properties such as normals and harmonics', () => {
+ const properties = ['nx', 'f_rest_0', ...PROPERTIES.slice().reverse()];
+ const values = [9, 9, ...IDENTITY_SPLAT(5, 6, 7).reverse()];
+ const data = parseGaussianSplatPly(buildPly([values], { properties }));
+ assert.deepEqual(Array.from(data.positions), [5, 6, 7]);
+});
+
+test('rejects files that are not 3DGS PLY with a clear message', () => {
+ assert.throws(() => parseGaussianSplatPly(new TextEncoder().encode('hello').buffer), /Not a PLY file/);
+ assert.throws(() => parseGaussianSplatPly(buildPly([IDENTITY_SPLAT(0, 0, 0)], { format: 'ascii' })), /binary_little_endian/);
+ assert.throws(() => parseGaussianSplatPly(buildPly([IDENTITY_SPLAT(0, 0, 0).slice(0, 13)], { properties: PROPERTIES.slice(0, 13) })), /lacks the properties rot_3/);
+ const truncated = buildPly([IDENTITY_SPLAT(0, 0, 0), IDENTITY_SPLAT(1, 1, 1)]).slice(0, -8);
+ assert.throws(() => parseGaussianSplatPly(truncated), /truncated/);
+ assert.throws(() => parseGaussianSplatPly(buildPly([IDENTITY_SPLAT(NaN, 0, 0)])), /non-finite position/);
+});
+
+test('enforces the byte and splat count limits before reading splats', () => {
+ const file = buildPly([IDENTITY_SPLAT(0, 0, 0), IDENTITY_SPLAT(1, 0, 0), IDENTITY_SPLAT(2, 0, 0)]);
+ assert.throws(() => parseGaussianSplatPly(file, { maximumCount: 2 }), /holds 3 splats, over the limit of 2/);
+ assert.throws(() => parseGaussianSplatPly(file, { maximumBytes: 100 }), RangeError);
+ assert.throws(() => parseGaussianSplatPly(file, { maximumBytes: 100 }), /over the limit of 100/);
+ assert.equal(parseGaussianSplatPly(file, { maximumCount: 3 }).count, 3);
+ assert.throws(() => parseGaussianSplatPly(file, { maximumCount: 0 }), /positive/);
+});
+
+test('the count limit applies to a header that claims more splats than the file holds', () => {
+ const header = ['ply', 'format binary_little_endian 1.0', 'element vertex 99999999', ...PROPERTIES.map(name => `property float ${name}`), 'end_header', ''].join('\n');
+ assert.throws(() => parseGaussianSplatPly(new TextEncoder().encode(header).buffer), /holds 99999999 splats/);
+});
+
+test('createGaussianSplats packs position, colour and covariance and limits the count', () => {
+ const data = parseGaussianSplatPly(buildPly([IDENTITY_SPLAT(1, 2, 3)]));
+ const splats = createGaussianSplats(data);
+ const floats = new Float32Array(splats.packed.words.buffer);
+ assert.deepEqual(Array.from(floats.subarray(0, 3)), [1, 2, 3]);
+ const [xx, xy, xz, yy, yz, zz] = Array.from(floats.subarray(4, 10)).map(value => Number(value.toFixed(5)));
+ assert.deepEqual([xx, xy, xz, yy, yz, zz], [0.01, 0, 0, 0.04, 0, 0.09]);
+ assert.throws(() => createGaussianSplats(data, { maximumCount: 0 }), /over the limit/);
+ assert.throws(() => createGaussianSplats(data, { matrix: [1, 2, 3] }), /16 finite/);
+});
+
+test('a rotated splat has a rotated covariance', () => {
+ const quarterTurnAboutZ = [Math.SQRT1_2, 0, 0, Math.SQRT1_2];
+ const splat = [0, 0, 0, 0, 0, 0, 0, Math.log(1), Math.log(2), Math.log(1), ...quarterTurnAboutZ];
+ const floats = new Float32Array(createGaussianSplats(parseGaussianSplatPly(buildPly([splat]))).packed.words.buffer);
+ const [xx, , , yy] = Array.from(floats.subarray(4, 8));
+ assert.ok(Math.abs(xx - 4) < 1e-5 && Math.abs(yy - 1) < 1e-5, `the long axis turned from y to x, got xx ${xx} yy ${yy}`);
+});
+
+// A camera at the origin looking down -z: the model-view is the identity, and depth is the negative z.
+const IDENTITY_VIEW = Float32Array.of(1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1);
+
+function sorterFor(points) {
+ const positions = Float32Array.from(points.flat());
+ const minimum = [0, 1, 2].map(axis => Math.min(...points.map(point => point[axis])));
+ const maximum = [0, 1, 2].map(axis => Math.max(...points.map(point => point[axis])));
+ return createSplatSorter(positions, points.length, { minimum, maximum });
+}
+
+test('sorts splats from the farthest to the nearest and drops those behind the camera', () => {
+ const sorter = sorterFor([[0, 0, -2], [0, 0, -9], [0, 0, 3], [0, 0, -5], [1, 0, -0.5]]);
+ const { order, visibleCount, changed } = sorter.sort(IDENTITY_VIEW);
+ assert.equal(changed, true);
+ assert.equal(visibleCount, 4);
+ assert.deepEqual(Array.from(order.subarray(0, visibleCount)), [1, 3, 0, 4]);
+});
+
+test('sort orders by view depth, not by distance to the eye', () => {
+ const sorter = sorterFor([[100, 0, -3], [0, 0, -4]]);
+ assert.deepEqual(Array.from(sorter.sort(IDENTITY_VIEW).order.subarray(0, 2)), [1, 0]);
+});
+
+test('sort follows the camera: turning the view around reverses the order', () => {
+ const sorter = sorterFor([[0, 0, -2], [0, 0, 2], [0, 0, -6], [0, 0, 6]]);
+ assert.deepEqual(Array.from(sorter.sort(IDENTITY_VIEW).order.subarray(0, 2)), [2, 0]);
+ const halfTurn = Float32Array.of(-1, 0, 0, 0, 0, 1, 0, 0, 0, 0, -1, 0, 0, 0, 0, 1);
+ const turned = sorter.sort(halfTurn);
+ assert.equal(turned.changed, true);
+ assert.deepEqual(Array.from(turned.order.subarray(0, turned.visibleCount)), [3, 1]);
+});
+
+test('sort reports no change for an unchanged view depth axis and keeps the order', () => {
+ const sorter = sorterFor([[0, 0, -2], [0, 0, -6]]);
+ sorter.sort(IDENTITY_VIEW);
+ const sidestep = Float32Array.from(IDENTITY_VIEW);
+ sidestep[12] = 5;
+ const second = sorter.sort(sidestep);
+ assert.equal(second.changed, false);
+ assert.deepEqual(Array.from(second.order.subarray(0, 2)), [1, 0]);
+});
+
+test('sort handles splats at the same depth and a single splat', () => {
+ const equal = sorterFor([[0, 0, -3], [1, 0, -3], [2, 0, -3]]).sort(IDENTITY_VIEW);
+ assert.equal(equal.visibleCount, 3);
+ assert.deepEqual(Array.from(equal.order.subarray(0, 3)).sort(), [0, 1, 2]);
+ assert.equal(sorterFor([[0, 0, -3]]).sort(IDENTITY_VIEW).visibleCount, 1);
+});
+
+test('200,000 random splats sort back to front', () => {
+ const count = 200_000;
+ const positions = new Float32Array(count * 3);
+ let seed = 12345;
+ const next = () => (seed = (seed * 1664525 + 1013904223) >>> 0) / 4294967296;
+ for (let index = 0; index < count; index++) positions.set([next() * 20 - 10, next() * 20 - 10, -next() * 50], index * 3);
+ const sorter = createSplatSorter(positions, count, { minimum: [-10, -10, -50], maximum: [10, 10, 0] });
+ const { order, visibleCount } = sorter.sort(IDENTITY_VIEW);
+ assert.equal(visibleCount, count);
+ const stepSize = 50 / 65535;
+ for (let index = 1; index < visibleCount; index++) {
+ const previous = positions[order[index - 1] * 3 + 2];
+ const current = positions[order[index] * 3 + 2];
+ assert.ok(previous <= current + stepSize, `splat ${index} is farther than the one drawn before it`);
+ }
+});
diff --git a/tests/water-reflection.test.js b/tests/water-reflection.test.js
index 0581f11b..c0d2b3fe 100644
--- a/tests/water-reflection.test.js
+++ b/tests/water-reflection.test.js
@@ -103,3 +103,28 @@ test('the mirror image is drawn at three quarters of the canvas in each directio
assert.deepEqual(textureSizes, [[720, 405]]);
detach();
});
+
+test('a render pass frame skips the mirror image and the next lit frame draws it again', () => {
+ const context = new Proxy({}, {
+ get: (_, name) => {
+ if (name === 'checkFramebufferStatus') {
+ return () => 1;
+ }
+ if (name === 'FRAMEBUFFER_COMPLETE') {
+ return 1;
+ }
+ return typeof name === 'string' && name === name.toUpperCase() ? 0 : () => ({});
+ },
+ });
+ const renderedFrames = [];
+ const renderer = { context, canvas: { width: 960, height: 540 }, hdr: false, viewProjection: new Float32Array(16), render: frame => renderedFrames.push(frame), dispose() {} };
+ const detach = attachWaterReflection(renderer, findWaterSurface(lakeScene()));
+ const camera = { eye: [0, 20, 100], target: [0, 0, 0] };
+ const mirrorDraws = () => renderedFrames.filter(frame => frame.reflectionTarget).length;
+ renderer.render({ ...camera, pass: 'objectId' });
+ assert.equal(mirrorDraws(), 0);
+ assert.equal(renderedFrames.length, 1);
+ renderer.render(camera);
+ assert.equal(mirrorDraws(), 1);
+ detach();
+});
diff --git a/vite.lib.config.js b/vite.lib.config.js
index 7039b352..d4ffbb63 100644
--- a/vite.lib.config.js
+++ b/vite.lib.config.js
@@ -53,6 +53,7 @@ export default defineConfig({
'material-layer-shaders': 'src/material-layer-shaders.js',
'light-linking': 'src/light-linking.js',
'cascaded-shadows': 'src/cascaded-shadows.js',
+ 'gaussian-splats': 'src/gaussian-splats.js',
// Page of the MCP preview. It imports the engine chunks above instead of bundling its own copy.
'mcp/preview': 'scripts/agent-preview-browser.js',
},