3D engine that AI agents drive
Add lighting render passes: diffuse, glossy, shadow, ao and mist (issue #601)
Nguyen Xuan Anf288392d593a
diff --git a/changes/601-lighting-passes.md b/changes/601-lighting-passes.mdnew file mode 100644index 00000000..91bbafb1--- /dev/null+++ b/changes/601-lighting-passes.md@@ -0,0 +1 @@+- render jobs: passes now include `diffuse`, `glossy`, `shadow`, `ao` and `mist` next to albedo, normal, emission, depth and objectId; diffuse + glossy + emission add back to the lit image, and `exr` keeps values above 1 (#601)diff --git a/changes/601-lighting-passes.size.md b/changes/601-lighting-passes.size.mdnew file mode 100644index 00000000..749df6d9--- /dev/null+++ b/changes/601-lighting-passes.size.md@@ -0,0 +1,2 @@+- entry:render-passes: the lighting passes (diffuse, glossy, shadow, ao, mist) bring their GLSL (lighting-pass-shader.js) into the lazy render-passes addon, which only a render job with passes loads; 524 bytes gzip for five new outputs.+- goal:agent-scene: 22 bytes gzip for the five new pass names in the render job schema and the pass mode range in the main shader; the lighting GLSL stays in the lazy addon.diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.mdindex 2bce116e..64e633e6 100644--- a/docs/agent-render-jobs.md+++ b/docs/agent-render-jobs.md@@ -105,7 +105,7 @@ and a click placed on a cut in a WAV lands on the decoded cut frame within a fra## Render passes-`image` and `stills` jobs accept `settings.passes`, a list of `albedo`, `normal`, `emission`, `depth` and `objectId`.+`image` and `stills` jobs accept `settings.passes`, a list of `albedo`, `normal`, `emission`, `depth`, `objectId`, `diffuse`, `glossy`, `shadow`, `ao` and `mist`.Each frame writes its lit image and then one PNG per pass: `render.png` and `render-<pass>.png` for an image, `frame-N.png` and `frame-N-<pass>.png` for stills.Passes are raw data at the lit image's size: no tone mapping, grading, bloom, depth of field or antialiasing, so values stay exact at silhouettes.Only opaque and cutout geometry appears. Liquid, foam and blended parts, the sky and the environment background do not.@@ -117,6 +117,13 @@ Only opaque and cutout geometry appears. Liquid, foam and blended parts, the sky| `emission` | Emissive colour times strength, clamped to 1 || `depth` | Metres from the camera over 100 m as a 16-bit value, high byte in red and low byte in green. Background is `255, 255, 0` || `objectId` | 24-bit ID, low byte in red, then green, then blue. Zero is background |+| `diffuse` | Linear diffuse light (every light, sky and bounce) times exposure, before fog and without emission. Background 0 |+| `glossy` | Linear specular light (each light's lobe and the sky reflection) times exposure, before fog. Background 0 |+| `shadow` | Sun shadow visibility, 1 lit and 0 shadowed. Background 1 |+| `ao` | Ambient occlusion the shader applies (occlusion map, vertex sky visibility, room term). Background 1 |+| `mist` | Fog amount `1 - exp(-distance * fogDensity)`. Background 1 |++The lighting passes hold linear values above 1, so write them as `exr` (a `png` or `png16` file clamps at 1). They add back to the lit image: `diffuse + glossy + emission * exposure` is the radiance before fog, and with an open world the lit image is that radiance mixed toward the fog colour by `mist`, then tone mapped. `tests/lighting-passes.test.js` checks the sum against the beauty frame (mean difference under 1.5 display levels, under 2% of samples off by more than 4). Not covered: unlit parts, the editor floor, parts that a probe or irradiance grid changes after the sky term, and anything the lit image adds after the shader (bloom, grading, depth of field).`scene_render_result` metrics carry `objectNames`, the part name for ID 1, 2 and so on (the first 1000, with `objectNamesTruncated`), and `depthRangeMetres`.IDs follow draw order, which groups parts by shared geometry, so read the names list and not node order.diff --git a/docs/parity.md b/docs/parity.mdindex 0323537d..b5f1d87c 100644--- a/docs/parity.md+++ b/docs/parity.md@@ -209,7 +209,7 @@ To move to a new Blender release, run `node scripts/parity-manual-snapshot.mjs`.| Rendering > Shader Nodes > Group | missing | No node graph. | | #50 || Rendering > Color Management | partial | View transforms `aces` (default), `agx` and `neutral` (Khronos PBR Neutral) in the rasterizer and the path tracer, and `.cube` look LUTs or four named looks; grading as before. Neutral matches Blender 5.2 within 0.01 on a ramp, AgX within 0.026 (a fit of Blender 5.2's AgX LUT; saturated colour is the part that is off). No choice of working color space or display device; no Filmic or Standard. | [view transforms](../tests/view-transform.test.js), [effects](../tests/effects.test.js) | AgX looks || Rendering > Freestyle | partial | `environment.lines` draws silhouette, boundary and crease lines in screen space, and `polylines` draws screen-width lines in the world. There are no line sets, styles or strokes along curves. Outline width is 1.65 px at 960x540 and 1.60 px at 3840x2160 for a requested 2 px, against 1.73 px in Freestyle ([comparison](line-art.md)). | [line-art](../tests/line-art.test.js), [guide](line-art.md), [reference script](../bench/reference/line-art.mjs) | |-| Rendering > View Layers & Passes | partial | Render jobs write albedo, normal, emission, depth and object ID passes as PNG files. There are no view layers, no cryptomatte and no lighting passes (diffuse, glossy, shadow). | [render-passes](../tests/render-passes.test.js) | #54 |+| Rendering > View Layers & Passes | partial | Render jobs write albedo, normal, emission, depth, object ID and the lighting passes diffuse, glossy, shadow, ao and mist as PNG, PNG16 or EXR files; diffuse + glossy + emission add back to the lit image. There are no view layers, no cryptomatte, and no transmission or volume passes. | [render-passes](../tests/render-passes.test.js), [lighting-passes](../tests/lighting-passes.test.js) | #54, #601 || Rendering > Render Output > Output Properties | partial | PNG and MP4, frame rate, duration (to 60 s) and size, and a CPU or NVENC encoder. There is no stereo or metadata. | [agent-render-jobs](../tests/agent-render-jobs.test.js) | #60 || Rendering > Render Output > Audio Rendering | partial | A video job mixes one local WAV, MP3, OGG or FLAC file with an offset and reports measured sync. No several tracks, mixing, gain or generated sound. | [film-timeline](../tests/film-timeline.test.js), `npm run smoke:film-timeline` | #60 || Rendering > Render Output > Rendering Animations | done | Snapshot render jobs with progress, cancel and verified output. | [agent-render-jobs](../tests/agent-render-jobs.test.js) | |diff --git a/skills/render-67js-media/references/jobs.md b/skills/render-67js-media/references/jobs.mdindex 3b3c9954..34a63a27 100644--- a/skills/render-67js-media/references/jobs.md+++ b/skills/render-67js-media/references/jobs.md@@ -62,7 +62,7 @@ fails the job and does not fall back, so choose `cpu` unless you know the machin## Render passes-`image` and `stills` jobs take `passes`, a list of `albedo`, `normal`, `emission`, `depth` and `objectId`. Each frame+`image` and `stills` jobs take `passes`, a list of `albedo`, `normal`, `emission`, `depth`, `objectId`, `diffuse`, `glossy`, `shadow`, `ao` and `mist` (lighting terms: use `exr` for them). Each framealso writes one raw PNG per pass at the lit image's size, with no grading, bloom, depth of field or antialiasing.```jsondiff --git a/src/agent-scene-capabilities.js b/src/agent-scene-capabilities.jsindex 4f619f87..4012ae53 100644--- a/src/agent-scene-capabilities.js+++ b/src/agent-scene-capabilities.js@@ -94,7 +94,7 @@ export const DEFAULT_LIGHTING_SETUP = 'daylight';export const DEFAULT_AMBIENT_OCCLUSION = 0;// Render passes a render job can write next to the lit image (see RENDER_PASS_MODES in renderer.js, which must list the same names).-export const RENDER_PASS_NAMES = Object.freeze(['albedo', 'normal', 'emission', 'depth', 'objectId']);+export const RENDER_PASS_NAMES = Object.freeze(['albedo', 'normal', 'emission', 'depth', 'objectId', 'diffuse', 'glossy', 'shadow', 'ao', 'mist']);// Video encoders a render job can choose (nvenc is h264_nvenc and fails the job when the GPU or ffmpeg lacks it), the// longest video, and the audio file names a job may mix in from the --audio folder: one path segment with a known// extension, no drive or protocol syntax.diff --git a/src/lighting-pass-shader.js b/src/lighting-pass-shader.jsnew file mode 100644index 00000000..8970df22--- /dev/null+++ b/src/lighting-pass-shader.js@@ -0,0 +1,26 @@+import { registerShaderFeatureCode } from './shader-registry.js';++// The GLSL of the lighting render passes (diffuse, glossy, shadow, ao, mist), spliced into the main fragment shader by the+// `lightingPass` feature. render-passes.js imports this file, so only a renderer that installs render passes carries it, and+// only a frame that asks for a lighting pass compiles it in. Modes 6 to 10 of renderPassMode, see RENDER_PASS_MODES.+//+// glossyLight collects every specular term: each light's lobe inside illuminate, and the sky or environment reflection. The+// environment term is the same function with a black surface colour, which removes its diffuse half. Diffuse is the rest of+// the lit radiance without emission, so diffuse + glossy + emission is the radiance before fog by construction.+registerShaderFeatureCode('lightingPass', {+ declarations: `#define USES_LIGHTING_PASS (renderPassMode > 5.5)+vec3 glossyLight = vec3(0.0);+`,+ illuminateSpecular: ' if (USES_LIGHTING_PASS) glossyLight += specular * radiance * normalLight;\n',+ probeReflection: ' if (USES_LIGHTING_PASS && ENVIRONMENT_ENABLED) glossyLight += environmentLighting(normal, specularNormal, baseReflectance, viewDirection, reflected, vec3(0.0), metallic, roughness, occlusion, ambientIntensity, skyReflection);\n',+ probeSurroundings: ' if (USES_LIGHTING_PASS) glossyLight += surroundings * reflectance * skyReflection * (1.0 - roughness * 0.55) * mix(1.0, 0.5 * daylight, worldMode) * occlusion * ambientIntensity;\n',+ worldFinal: ` if (USES_LIGHTING_PASS) {+ float mist = 1.0 - exp(-length(worldPosition - eye) * fogDensity);+ vec3 passValue = renderPassMode < 6.5 ? (radiance - glossyLight - emission) * exposure+ : renderPassMode < 7.5 ? glossyLight * exposure+ : vec3(renderPassMode < 8.5 ? visibility : renderPassMode < 9.5 ? occlusion : mist);+ outputColor = vec4(passValue, 1.0);+ return;+ }+`,+});diff --git a/src/render-passes.js b/src/render-passes.jsindex b3b93ab0..1747bd97 100644--- a/src/render-passes.js+++ b/src/render-passes.js@@ -1,13 +1,16 @@import { compileProgram, fullscreenVertex } from './gl-program.js';+import './lighting-pass-shader.js';// Render passes draw raw material data (albedo, normal, emission, depth, object ID) instead of the lit image.// The shader branch lives in the main fragment shader behind the `renderPass` feature, which compiles on the first// pass request; this addon owns the rest: the target, the copy to the canvas and the object ID table./** The shader's `renderPassMode` for each pass name. agent-scene-capabilities.js lists the same names. */-export const RENDER_PASS_MODES = Object.freeze({ albedo: 1, normal: 2, emission: 3, depth: 4, objectId: 5 });-// Background pixels: depth reads as the far limit; every other pass reads zero, and object ID zero means no object.-const PASS_BACKGROUNDS = Object.freeze({ depth: [1, 1, 0, 1] });+export const RENDER_PASS_MODES = Object.freeze({ albedo: 1, normal: 2, emission: 3, depth: 4, objectId: 5, diffuse: 6, glossy: 7, shadow: 8, ao: 9, mist: 10 });+// Background pixels: depth reads as the far limit, and shadow, ao and mist as open sky (one); every other pass reads zero, and object ID zero means no object.+const PASS_BACKGROUNDS = Object.freeze({ depth: [1, 1, 0, 1], shadow: [1, 1, 1, 1], ao: [1, 1, 1, 1], mist: [1, 1, 1, 1] });+// Modes above this one run the lit shader (lighting-pass-shader.js) and write one lighting term: linear, scaled by exposure.+const LAST_RAW_PASS_MODE = 5;const DEFAULT_BACKGROUND = Object.freeze([0, 0, 0, 1]);// Copies a pass texture to the canvas byte for byte: one texel per pixel, no filtering, no grading.@@ -72,7 +75,7 @@ export function installRenderPasses(renderer) {const depthRange = frame.passDepthRange ?? 1;if (!Number.isFinite(depthRange) || depthRange <= 0) throw new RangeError('Pass depth range must be positive and finite.');const context = renderer.context;- renderer.requireFeatureNames(['renderPass']);+ renderer.requireFeatureNames(mode > LAST_RAW_PASS_MODE ? ['renderPass', 'lightingPass'] : ['renderPass']);if (!copyProgram) {copyProgram = compileProgram(context, fullscreenVertex, passCopyFragment);copyTextureUniform = context.getUniformLocation(copyProgram, 'sourceTexture');diff --git a/src/shaders.js b/src/shaders.jsindex 6acf8fb8..e476e4ee 100644--- a/src/shaders.js+++ b/src/shaders.js@@ -759,7 +759,7 @@ const SHADER_FEATURE_CONDITIONS = {areaLights: ['USES_AREA_LIGHTS', 'areaLightCount > 0'],localShadows: ['USES_LOCAL_SHADOWS', 'true'],fill: ['FILL_ON', 'true'],- renderPass: ['USES_RENDER_PASS', 'renderPassMode > 0.5'],+ renderPass: ['USES_RENDER_PASS', 'renderPassMode > 0.5 && renderPassMode < 5.5'],sheen: ['USES_SHEEN', 'materialSheen.rgb != vec3(0.0)'],clearcoat: ['USES_CLEARCOAT', 'materialCoat.x > 0.0'],};diff --git a/tests/lighting-passes.test.js b/tests/lighting-passes.test.jsnew file mode 100644index 00000000..fff97879--- /dev/null+++ b/tests/lighting-passes.test.js@@ -0,0 +1,131 @@+// Lighting render passes (issue 601) drawn for real: diffuse + glossy + emission add back to the beauty frame, and shadow, ao and mist read sensibly.+// It draws in headless Chrome (SwiftShader, no GPU) from the Vite dev server, and skips when Chrome cannot start.+import test from 'node:test';+import assert from 'node:assert/strict';+import { spawn } from 'node:child_process';+import { freePort } from './helpers/free-port.js';++async function startServer(port) {+ const server = spawn(process.execPath, ['node_modules/vite/bin/vite.js', '--host', '127.0.0.1', '--port', String(port), '--strictPort'], { windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] });+ await new Promise((resolve, reject) => {+ const timeout = setTimeout(() => reject(new Error('The dev server did not start.')), 30000);+ server.stdout.on('data', (data) => {+ if (/ready in/.test(data.toString())) {+ clearTimeout(timeout);+ resolve();+ }+ });+ server.on('exit', (code) => {+ clearTimeout(timeout);+ reject(new Error(`The dev server exited ${code}.`));+ });+ });+ return server;+}++++// Draws the beauty frame and every lighting pass of a floor with a shiny, an emissive and a rough box above it.+async function drawFrames(page, withEnvironment) {+ const task = async (useEnvironment) => {+ const { box } = await import('/src/geometry.js');+ const { Renderer67 } = await import('/src/renderer.js');+ const { installRenderPasses } = await import('/src/render-passes.js');+ const { readFloatPixels } = await import('/src/float-output.js');+ const canvas = document.createElement('canvas');+ canvas.width = 64;+ canvas.height = 64;+ document.body.append(canvas);+ const copy = document.createElement('canvas');+ copy.width = 64;+ copy.height = 64;+ const copyContext = copy.getContext('2d', { willReadFrequently: true });+ const geometry = box(0, 1);+ const scaled = (sx, sy, sz, x, y, z) => new Float32Array([sx, 0, 0, 0, 0, sy, 0, 0, 0, 0, sz, 0, x, y, z, 1]);+ const parts = [+ { name: 'floor', geometry, matrix: scaled(8, 0.2, 8, 0, -0.1, 0), material: { tag: 'surface', color: [0.25, 0.25, 0.25], metallic: 0, roughness: 0.7, emission: 0 } },+ { name: 'shiny', geometry, matrix: scaled(1.2, 1.2, 1.2, -1.5, 1, 0), material: { tag: 'surface', color: [0.9, 0.2, 0.2], metallic: 0.8, roughness: 0.25, emission: 0 } },+ { name: 'rough', geometry, matrix: scaled(1.2, 1.2, 1.2, 1.5, 1, 0), material: { tag: 'surface', color: [0.2, 0.5, 0.9], metallic: 0, roughness: 0.6, emission: 0.15 } },+ ];+ const renderer = installRenderPasses(new Renderer67(canvas, parts, { ground: false, samples: 1, pixelRatio: 1 }));+ const size = renderer.canvas.width;+ copy.width = size;+ copy.height = size;+ const camera = { eye: [0, 5, 7], target: [0, 0.6, 0] };+ renderer.render(camera);+ copyContext.drawImage(canvas, 0, 0, size, size);+ const beauty = Array.from(copyContext.getImageData(0, 0, size, size).data);+ const passes = {};+ for (const pass of ['diffuse', 'glossy', 'emission', 'shadow', 'ao', 'mist', 'albedo']) {+ renderer.render({ ...camera, pass, passFloat: true });+ passes[pass] = Array.from(readFloatPixels(renderer.context, renderer.passSupport.framebuffer, size, size));+ }+ return { beauty, passes, size };+ };+ return page.evaluate(task, withEnvironment);+}++// The renderer's tone map and gamma, for the lit radiance that the passes add back up to.+function display(value) {+ const mapped = Math.min(Math.max((value * (2.51 * value + 0.03)) / (value * (2.43 * value + 0.59) + 0.14), 0), 1);+ return Math.round(255 * mapped ** (1 / 2.2));+}++test('diffuse + glossy + emission match the beauty frame, and shadow, ao and mist read as light terms', async (context) => {+ let chromium;+ try {+ ({ chromium } = await import('playwright'));+ } catch {+ context.skip('playwright is not installed');+ return;+ }+ const port = await freePort();+ const server = await startServer(port);+ let browser;+ try {+ try {+ browser = await chromium.launch({ headless: true, args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader', '--enable-webgl', '--ignore-gpu-blocklist'] });+ } catch (error) {+ context.skip(`Chrome did not start: ${error.message.split('\n')[0]}`);+ return;+ }+ const page = await browser.newPage({ viewport: { width: 64, height: 64 } });+ await page.goto(`http://127.0.0.1:${port}/`);+ const { beauty, passes, size: SIZE } = await drawFrames(page, false);+ let compared = 0;+ let totalError = 0;+ let largeErrors = 0;+ let glossyEnergy = 0;+ let shadowed = 0;+ for (let row = 0; row < SIZE; row++) {+ for (let column = 0; column < SIZE; column++) {+ const beautyIndex = (row * SIZE + column) * 4;+ // Pass rows come bottom to top, the canvas top to bottom.+ const passIndex = ((SIZE - 1 - row) * SIZE + column) * 4;+ if (passes.albedo[passIndex] + passes.albedo[passIndex + 1] + passes.albedo[passIndex + 2] === 0) continue;+ for (let channel = 0; channel < 3; channel++) {+ const sum = passes.diffuse[passIndex + channel] + passes.glossy[passIndex + channel] + passes.emission[passIndex + channel];+ const difference = Math.abs(display(sum) - beauty[beautyIndex + channel]);+ totalError += difference;+ if (difference > 4) largeErrors++;+ compared++;+ }+ glossyEnergy += passes.glossy[passIndex];+ if (passes.shadow[passIndex] < 0.5) shadowed++;+ }+ }+ assert.ok(compared > 3000, `the frame has ${compared} lit samples`);+ assert.ok(totalError / compared < 1.5, `mean difference ${totalError / compared} display levels`);+ assert.ok(largeErrors / compared < 0.02, `${largeErrors} of ${compared} samples differ by more than 4 levels`);+ assert.ok(glossyEnergy > 1, 'the glossy pass holds light');+ assert.ok(shadowed > 20, `the shadow pass marks ${shadowed} shadowed pixels`);+ for (const name of ['shadow', 'ao', 'mist']) {+ for (let index = 0; index < passes[name].length; index += 4) {+ assert.ok(passes[name][index] >= 0 && passes[name][index] <= 1.001, `${name} stays between zero and one`);+ }+ }+ } finally {+ await browser?.close();+ server.kill();+ }+});diff --git a/tests/render-passes.test.js b/tests/render-passes.test.jsindex 6d9e6bd9..18aa5d3f 100644--- a/tests/render-passes.test.js+++ b/tests/render-passes.test.js@@ -15,12 +15,12 @@ const limits = { maxOutputBytes: 1024, maxFrameBytes: 128, maxStoreBytes: 1024 *test('capabilities and the renderer list the same passes in the same order', () => {assert.deepEqual(Object.keys(RENDER_PASS_MODES), [...RENDER_PASS_NAMES]);- assert.deepEqual(Object.values(RENDER_PASS_MODES), [1, 2, 3, 4, 5]);+ assert.deepEqual(Object.values(RENDER_PASS_MODES), [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);});test('the pass branch of the main shader is compiled out unless a pass is requested', () => {assert.match(createFragmentShader([]), /#define USES_RENDER_PASS false/);- assert.match(createFragmentShader(['renderPass']), /#define USES_RENDER_PASS \(renderPassMode > 0\.5\)/);+ assert.match(createFragmentShader(['renderPass']), /#define USES_RENDER_PASS \(renderPassMode > 0\.5 && renderPassMode < 5\.5\)/);});test('camera effects default off, validate ranges and survive look normalization', () => {