3D engine that AI agents drive
Mesh volume pass: free replaced geometry, keep far-plane stretch, load dispose hook lazily
Nguyen Xuan And7a5db8a1795
diff --git a/changes/807-raster-mesh-fit.size.md b/changes/807-raster-mesh-fit.size.mdindex f0acfa2c..77d4be11 100644--- a/changes/807-raster-mesh-fit.size.md+++ b/changes/807-raster-mesh-fit.size.md@@ -1,8 +1,6 @@- entry:fog-shader: the mesh pass (a small program, an RG32F target, the front and back face sum) and the closed-form step in the post shader, so smoke fills a modelled room in the rasterizer as it does in the path tracer instead of throwing; the code loads only with the fog addon and costs nothing in a scene without a fit "mesh" volume.-- scene:agent-scene: the same lazy fog code plus the edge check (open edges and winding) moved into the part binding, an agent-facing capability.+- scene:agent-scene: the edge check (open edges and winding) moved into the part binding, an agent-facing capability.- scene:agent-scene+lazy: the same code, counted with the lazy chunks.-- scene:lit-box: two lines in the renderer (the mesh pass call and its dispose), 26 bytes gzip; the pass itself is in the fog addon.-- goal:lit-box: 26 bytes gzip for the renderer's call into the mesh pass; the call must sit in the frame path, and the pass is lazy.-- scene:gltf-box: the same two renderer lines, 31 bytes gzip.-- goal:gltf-box: the same two renderer lines, 31 bytes gzip.-- goal:agent-scene: the scene is already over its goal; the part binding and edge check must run where the recipe is checked, and the gain is an agent-facing capability (119 bytes gzip).+- goal:agent-scene: the scene is already over its goal; the part binding and edge check must run where the recipe is checked, and the gain is an agent-facing capability (92 bytes gzip).+- scene:lit-box: the renderer passes the frame to the local volume step, 2 bytes gzip; the mesh pass and its dispose hook load lazily with the fog addon.+- scene:gltf-box: the same argument, 1 byte gzip.diff --git a/docs/volumetric.md b/docs/volumetric.mdindex e7ab19f1..fcde5670 100644--- a/docs/volumetric.md+++ b/docs/volumetric.md@@ -141,7 +141,7 @@ Each volume is marched on its own between its entry and its exit, cut at the fir- `fit: "bounds"` (the default) uses the part's bounding box in its own frame, carried through its matrix: an oriented box, tight for a rotated or scaled part. It draws in the rasterizer and the CPU path tracer, and `falloff` fades its edges as for a box.- `fit: "mesh"` uses the closed mesh itself. The CPU path tracer walks the ray from crossing to crossing, so an L-shaped room or two separate shells are filled exactly, and a ray that leaves the mesh and returns crosses both pieces. The mesh has no distance field, so `falloff` is ignored (the density is the same everywhere inside), and so is `noiseAmount`.- The rasterizer draws the closed mesh in a mesh pass before the post pass: each `fit: "mesh"` part is drawn without culling into a float buffer, adding the distance of every back face and subtracting that of every front face along each pixel's ray, each clamped to the first opaque surface. That sum is the length of the ray inside the mesh, exact for any number of shells and any shape, so there is no shell limit; the post shader then integrates the volume over that length in closed form (the light behind is dimmed by `exp(-density * absorption * length)`, the emission adds `emission * (1 - exp(-k * length)) / k`). `steps` has no effect on a mesh fit, because nothing is stepped. Volumes are still composited nearest entry first. On the L-room test (`tests/local-volume-mesh-raster.test.js`, density 1) the rasterizer's transmittance stays within 0.03 of the CPU tracer's (the test tolerance; measured differences were 0.0014 or less) and the notch of the L is exactly clear.-- The rasterizer needs the mesh to be closed and every face to wind the same way (either outward or inward; it detects which): an open mesh throws `environment.localVolumes[n].part "x" is not a closed mesh (k open edges)`, and faces wound against each other throw `... has faces wound against each other (k edges)`. It needs `EXT_float_blend` and a buffer as wide as the canvas and as tall as the canvas times the number of `fit: "mesh"` volumes (RG32F, 8 bytes per pixel each; at 1920 x 1080 that is 16.6 MB per volume), and throws with the limit when the device cannot make it. Geometry nearer than the camera's near plane is clipped, so a ray loses the stretch between the eye and the near plane. A scene with no `fit: "mesh"` volume allocates nothing and draws nothing extra.+- The rasterizer needs the mesh to be closed and every face to wind the same way (either outward or inward; it detects which): an open mesh throws `environment.localVolumes[n].part "x" is not a closed mesh (k open edges)`, and faces wound against each other throw `... has faces wound against each other (k edges)`. It needs `EXT_float_blend` and a buffer as wide as the canvas and as tall as the canvas times the number of `fit: "mesh"` volumes (RG32F, 8 bytes per pixel each; at 1920 x 1080 that is 16.6 MB per volume), and throws with the limit when the device cannot make it. Geometry nearer than the camera's near plane is clipped, so a ray loses the stretch between the eye and the near plane; the automatic near plane counts only visible parts, so a hidden volume part that sits nearer the camera than everything visible needs an explicit camera `near`. A mesh that crosses the far plane keeps the stretch up to it (the missing back face is made up from the surface or far distance), so smoke fades at the far plane like any other volume. A scene with no `fit: "mesh"` volume allocates nothing and draws nothing extra.- A part that is `visible: false` is not drawn by the rasterizer and is left out of the traced geometry when a mesh volume names it, so a hidden part can shape smoke without becoming a wall.- The part must be rigid (not skinned) and have volume. A mesh with open edges has no inside: `fit: "mesh"` throws an error with the part name and the number of open edges. A `part` that names nothing throws `environment.localVolumes[n].part "x" names no part in the scene` with the part names.- The mesh walk follows at most 64 surface crossings per ray (32 segments inside), so the cost stays bounded; the `steps` of the volume are shared between its segments by length.diff --git a/src/fog-shader.js b/src/fog-shader.jsindex 739f6be1..44db34c8 100644--- a/src/fog-shader.js+++ b/src/fog-shader.js@@ -1,7 +1,7 @@import { registerShaderFeatureCode } from './shader-registry.js';import { resolveLocalVolumes, createLocalVolumeBuffers, packLocalVolumes } from './local-volumes.js';import { bindLocalVolumeParts } from './local-volume-parts.js';-import { drawMeshVolumes, disposeMeshVolumePass } from './mesh-volume-pass.js';+import { drawMeshVolumes } from './mesh-volume-pass.js';import { FogFieldTexture, createFogLightBuffers, packFogLights, volumeShadowAmbient, DEFAULT_FLOW_STRENGTH, DEFAULT_FLOW_PERIOD, resolveFogDetail, fogUsesDetail } from './fog-field.js';// The volume fog step of the post pass: its uniforms and the march that scatters light through the fog box. Only a renderer that@@ -78,12 +78,13 @@ function bindFieldAndLights(renderer, post, volume) {// The local volumes of the frame (environment.localVolumes), checked and packed once before the post program is chosen, so a bad// field or a renderer without depth or HDR throws before any program is bound. Returns the count packed, or 0 for an empty list.-function prepareLocalVolumes(renderer, input) {+function prepareLocalVolumes(renderer, input, frame) {const resolved = resolveLocalVolumes(input, 'environment.localVolumes');if (resolved.length === 0) return 0;if (!renderer.depth || !renderer.hdr || renderer.transparent || renderer.benchmark) throw new Error('Local volumes require depth:true, active HDR and an opaque canvas');const buffers = (renderer.localVolumeBuffers ??= createLocalVolumeBuffers());packLocalVolumes(bindLocalVolumeParts(resolved, renderer.sceneParts), buffers);+ drawMeshVolumes(renderer, frame);return buffers.count;}@@ -182,9 +183,11 @@ vec3 localVolumes(vec3 background) {if (localVolumeCenters[index].w > 1.5) {// A closed mesh: the mesh pass stored the length inside this pixel's ray and the nearest entry (see mesh-volume-pass.js).vec2 inside = texelFetch(localMeshVolumes, ivec2(gl_FragCoord.xy) + ivec2(0, int(localVolumeSteps[index].y) * int(resolution.y)), 0).rg;- if (inside.x > 0.0) {+ // A back face past the far plane is clipped and leaves only its front face, a negative sum; that ray ends at the surface.+ float insideLength = inside.x < -1e-5 * surfaceDistance ? inside.x + surfaceDistance : inside.x;+ if (insideLength > 0.0) {entries[index] = inside.y < 1e29 ? inside.y : 0.0;- exits[index] = entries[index] + inside.x;+ exits[index] = entries[index] + insideLength;}continue;}@@ -244,7 +247,7 @@ function packLights(renderer) {registerShaderFeatureCode('volumeFog', {// The JavaScript half, which the renderer calls for a frame that draws a volume fog.- support: { packLights, bindUniforms, usesDetail: fogUsesDetail, prepareLocalVolumes, bindLocalVolumes, drawMeshVolumes, disposeMeshVolumes: disposeMeshVolumePass },+ support: { packLights, bindUniforms, usesDetail: fogUsesDetail, prepareLocalVolumes, bindLocalVolumes },localDeclarations: LOCAL_VOLUME_DECLARATIONS,localFunctions: LOCAL_VOLUME_FUNCTIONS,postDeclarations: `diff --git a/src/mesh-volume-pass.js b/src/mesh-volume-pass.jsindex 53914991..5d9601f3 100644--- a/src/mesh-volume-pass.js+++ b/src/mesh-volume-pass.js@@ -8,6 +8,7 @@ import { assertClosedMesh } from './local-volume-parts.js';// channel G keeps the nearest front face distance (the entry, used to order the volumes).// That sum is exact for any number of shells and any shape, so there is no shell limit. The post shader then integrates the volume// over that length in closed form, which is exact because a mesh volume has the same density everywhere inside (no falloff, no noise).+// Geometry far past the far plane is clipped, so a ray can meet a front face and lose its back face; the post shader repairs that.// The cost is two draws of the part per frame, so a scene without mesh volumes pays nothing.// Limits: the mesh must be closed and rigid; geometry nearer than the camera's near plane is clipped, so a ray loses the stretch// between the eye and the near plane; the target is as wide as the canvas and as tall as the canvas times the number of mesh volumes.@@ -97,11 +98,19 @@ function uploadGeometry(context, geometry) {};}-function createState(context) {+function createState(renderer) {+ const context = renderer.context;if (!context.getExtension('EXT_float_blend')) throw new Error('Local volumes with fit "mesh" need the WebGL extension EXT_float_blend, which this browser does not offer; use fit "bounds"');const program = compileProgram(context, MESH_VERTEX, MESH_FRAGMENT);const uniforms = Object.fromEntries(UNIFORM_NAMES.map(name => [name, context.getUniformLocation(program, name)]));- return { program, uniforms, geometries: new Map(), target: null };+ const state = { program, uniforms, geometries: new Map(), target: null };+ // The renderer's dispose gains this step here, so the core carries no call for a pass most scenes never create.+ const disposeRenderer = renderer.dispose.bind(renderer);+ renderer.dispose = () => {+ disposeMeshVolumePass(renderer, state);+ disposeRenderer();+ };+ return state;}function allocateTarget(renderer, state, layers) {@@ -136,6 +145,21 @@ function releaseTarget(renderer, state) {renderer.renderTargets.release('meshVolumes');}+function releaseGeometry(context, gpu) {+ context.deleteVertexArray(gpu.vertexArray);+ gpu.buffers.forEach(buffer => context.deleteBuffer(buffer));+}++// A part whose geometry was replaced or removed no longer draws, so its buffers go.+function pruneGeometries(context, state, meshParts) {+ const inUse = new Set(meshParts.map(({ part }) => part.geometry));+ for (const [geometry, gpu] of state.geometries) {+ if (inUse.has(geometry)) continue;+ releaseGeometry(context, gpu);+ state.geometries.delete(geometry);+ }+}+function drawPart(context, state, part) {let gpu = state.geometries.get(part.geometry);if (!gpu) {@@ -156,10 +180,14 @@ function drawPart(context, state, part) {*/export function drawMeshVolumes(renderer, frame) {const meshParts = renderer.localVolumeBuffers?.meshParts ?? [];- if (meshParts.length === 0) return null;+ if (meshParts.length === 0) {+ if (renderer.meshVolumePass) pruneGeometries(renderer.context, renderer.meshVolumePass, meshParts);+ return null;+ }const context = renderer.context;- const state = (renderer.meshVolumePass ??= createState(context));+ const state = (renderer.meshVolumePass ??= createState(renderer));for (const { part, label } of meshParts) assertClosedMesh(part, label);+ pruneGeometries(context, state, meshParts);allocateTarget(renderer, state, meshParts.length);const { width, height } = renderer.canvas;const blendWasOn = context.isEnabled(context.BLEND);@@ -201,16 +229,10 @@ export function drawMeshVolumes(renderer, frame) {return state.target.texture;}-/** Frees the program, the buffer and the uploaded meshes. */-export function disposeMeshVolumePass(renderer) {- const state = renderer.meshVolumePass;- if (!state) return;+function disposeMeshVolumePass(renderer, state) {const context = renderer.context;releaseTarget(renderer, state);- for (const gpu of state.geometries.values()) {- context.deleteVertexArray(gpu.vertexArray);- gpu.buffers.forEach(buffer => context.deleteBuffer(buffer));- }+ state.geometries.forEach(gpu => releaseGeometry(context, gpu));context.deleteProgram(state.program);renderer.meshVolumePass = null;}diff --git a/src/renderer.js b/src/renderer.jsindex d25ca99b..492ec9df 100644--- a/src/renderer.js+++ b/src/renderer.js@@ -2075,9 +2075,9 @@ export class Renderer67 {}// True when the frame draws local volumes. An empty list costs nothing; anything else is checked here, before a program is bound.- prepareLocalVolumes(input) {+ prepareLocalVolumes(input, frame) {if (input === undefined || (Array.isArray(input) && input.length === 0)) return false;- return requireFeatureCode('volumeFog').support.prepareLocalVolumes(this, input) > 0;+ return requireFeatureCode('volumeFog').support.prepareLocalVolumes(this, input, frame) > 0;}// The post steps this frame uses. A step that is off costs nothing in a program that has it compiled out.@@ -2149,10 +2149,9 @@ export class Renderer67 {context.bindVertexArray(null);const fogSky = this.fogSkyVisibility?.(this.sceneParts, volume, context, environment);const fogSupport = volume.density > 0 ? requireFeatureCode('volumeFog').support : null;- const localVolumes = this.prepareLocalVolumes(environment.localVolumes);+ const localVolumes = this.prepareLocalVolumes(environment.localVolumes, frame);if (fogSupport) fogSupport.packLights(this);else if (this.fogLightBuffers) this.fogLightBuffers.count = 0;- if (localVolumes) requireFeatureCode('volumeFog').support.drawMeshVolumes(this, frame);const post = this.selectPostProgram(this.postStepsOf(frame, effects, volume, occluded, fogSky, localVolumes));context.useProgram(post.program);if (this.look) viewTransformShader.bindLook(this, post.program);@@ -2683,7 +2682,6 @@ export class Renderer67 {dispose() {this.fogFieldTexture?.dispose();- if (this.meshVolumePass) requireFeatureCode('volumeFog').support.disposeMeshVolumes(this);this.releaseFrameTargets();this.shadowCascades?.dispose(this.context, this.renderTargets);viewTransformShader.dispose?.(this);diff --git a/tests/local-volume-mesh-raster.test.js b/tests/local-volume-mesh-raster.test.jsindex bb339dd2..a1461976 100644--- a/tests/local-volume-mesh-raster.test.js+++ b/tests/local-volume-mesh-raster.test.js@@ -107,7 +107,7 @@ async function drawInChrome(page, scenarios) {return results;};const frames = await page.evaluate(task, { scenarios, size: SIZE });- return frames.map(frame => (frame.transmittance ? { ...frame, transmittance: transmittanceOf(frame.transmittance) } : frame));+ return frames.map(frame => (frame.transmittance ? { ...frame, filled: frame.transmittance.filled, transmittance: transmittanceOf(frame.transmittance) } : frame));}// The display shows aces(light) ^ (1 / 2.2); this undoes both for a channel, so the ratio of two frames is the light that got through.@@ -234,3 +234,27 @@ test('the rasterizer sums every shell a ray crosses, from outside and from insidawait opened.close();}});++test('a mesh that straddles the far plane keeps the stretch it has this side of it', async (context) => {+ const opened = await openPage(context);+ if (!opened) return;+ try {+ // The box spans 6 to 7 units from the eye and the far plane sits at 6.5, so its back face is clipped. A box half as long ends+ // at 6.5 inside a farther far plane and loses nothing, so both pixels must glow the same.+ const view = { eye: [-6, 0.5, 0.5], target: [0, 0.5, 0.5], up: [0, 1, 0], fov: 20 };+ const glowing = { shape: 'mesh', part: 'room', fit: 'mesh', density: 0.7, falloff: 0, steps: 16, emission: [1, 1, 1] };+ const singleBox = { positions: dumbbell().positions.slice(0, 24), indices: dumbbell().indices.slice(0, 36) };+ const halfLong = [0.5, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1];+ const [clipped, whole] = await drawInChrome(opened.page, [+ { mesh: singleBox, matrix: IDENTITY, camera: { ...view, far: 6.5 }, localVolumes: [glowing] },+ { mesh: singleBox, matrix: halfLong, camera: { ...view, far: 7.5 }, localVolumes: [glowing] },+ ]);+ assert.equal(clipped.message, null);+ const middle = ((SIZE / 2) * SIZE + SIZE / 2) * 4;+ const expected = lightOf(whole.filled[middle]);+ assert.ok(lightOf(clipped.filled[middle]) > 0.4, 'the clipped volume still glows');+ assert.ok(Math.abs(lightOf(clipped.filled[middle]) - expected) < 0.02 * expected, `clipped ${lightOf(clipped.filled[middle])} against ${expected}`);+ } finally {+ await opened.close();+ }+});diff --git a/tests/mesh-volume-pass-geometry.test.js b/tests/mesh-volume-pass-geometry.test.jsnew file mode 100644index 00000000..dabf09f1--- /dev/null+++ b/tests/mesh-volume-pass-geometry.test.js@@ -0,0 +1,91 @@+// The mesh volume pass keeps one set of GPU buffers per part geometry (issue 807). A geometry that is replaced or leaves the scene+// must give its buffers back, or a scene that edits a mesh every frame leaks. Runs against a recording fake of the GL context.+import test from 'node:test';+import assert from 'node:assert/strict';+import { drawMeshVolumes } from '../src/mesh-volume-pass.js';++function recordingContext() {+ const created = { vertexArrays: new Set(), buffers: new Set() };+ const deleted = { vertexArrays: [], buffers: [] };+ let nextHandle = 0;+ const handlers = {+ getExtension: () => ({}),+ checkFramebufferStatus: () => 'FRAMEBUFFER_COMPLETE',+ getParameter: () => 16384,+ isEnabled: () => false,+ getUniformLocation: name => name,+ getShaderParameter: () => true,+ getProgramParameter: () => true,+ getShaderInfoLog: () => '',+ getProgramInfoLog: () => '',+ createVertexArray: () => {+ const handle = { kind: 'vertexArray', id: nextHandle++ };+ created.vertexArrays.add(handle);+ return handle;+ },+ createBuffer: () => {+ const handle = { kind: 'buffer', id: nextHandle++ };+ created.buffers.add(handle);+ return handle;+ },+ deleteVertexArray: handle => deleted.vertexArrays.push(handle),+ deleteBuffer: handle => deleted.buffers.push(handle),+ };+ const context = new Proxy({}, {+ get(_target, name) {+ if (name in handlers) return handlers[name];+ if (/^[A-Z0-9_]+$/.test(name)) return name;+ return () => ({ kind: String(name), id: nextHandle++ });+ },+ });+ return { context, created, deleted };+}++function closedTetrahedron(offset) {+ return {+ positions: new Float32Array([offset, 0, 0, offset + 1, 0, 0, offset, 1, 0, offset, 0, 1]),+ normals: new Float32Array(12),+ indices: new Uint32Array([0, 2, 1, 0, 1, 3, 0, 3, 2, 1, 2, 3]),+ };+}++function rendererWith(context, geometry) {+ return {+ context,+ dispose() { this.disposed = true; },+ canvas: { width: 8, height: 8 },+ depthTexture: {},+ viewProjection: new Float32Array(16),+ inverseViewProjection: new Float32Array(16),+ renderTargets: { record() {}, release() {} },+ localVolumeBuffers: { meshParts: [{ label: 'environment.localVolumes[0]', part: { name: 'room', geometry } }] },+ };+}++test('replacing a mesh volume geometry deletes the buffers of the old one', () => {+ const { context, created, deleted } = recordingContext();+ const renderer = rendererWith(context, closedTetrahedron(0));+ const frame = { eye: [0, 0, 5], projection: 'perspective' };+ drawMeshVolumes(renderer, frame);+ assert.equal(created.vertexArrays.size, 1);+ assert.equal(deleted.buffers.length, 0);+ drawMeshVolumes(renderer, frame);+ assert.equal(deleted.buffers.length, 0, 'the same geometry keeps its buffers');+ const [oldVertexArray] = created.vertexArrays;+ const [oldPositions, oldIndices] = created.buffers;+ renderer.localVolumeBuffers.meshParts[0].part.geometry = closedTetrahedron(2);+ drawMeshVolumes(renderer, frame);+ assert.deepEqual(deleted.vertexArrays, [oldVertexArray]);+ assert.deepEqual(deleted.buffers, [oldPositions, oldIndices]);+ assert.equal(created.vertexArrays.size, 2);+});++test('disposing the pass deletes the buffers still in use', () => {+ const { context, deleted } = recordingContext();+ const renderer = rendererWith(context, closedTetrahedron(0));+ drawMeshVolumes(renderer, { eye: [0, 0, 5], projection: 'perspective' });+ renderer.dispose();+ assert.equal(deleted.buffers.length, 2);+ assert.equal(renderer.meshVolumePass, null);+ assert.equal(renderer.disposed, true, 'the renderer still disposes the rest');+});