Skip to content

3D engine that AI agents drive

Add a water hook to material snippets that runs after lighting (#2041)

Nguyen Xuan An5bfd624432ef
diff --git a/changes/2041.md b/changes/2041.md
new file mode 100644
index 00000000..b1d84cb1
--- /dev/null
+++ b/changes/2041.md
@@ -0,0 +1 @@
+- A material snippet on water can define `waterHook` to run after lighting with the normal, view, foam, time and light, so a few lines of GLSL draw flat tones and hard foam (#2041)
diff --git a/changes/2041.size.md b/changes/2041.size.md
new file mode 100644
index 00000000..cd594382
--- /dev/null
+++ b/changes/2041.size.md
@@ -0,0 +1,4 @@
+- scene:agent-scene+lazy: about 575 bytes gzip for the water entry point of material snippets (its pattern, the per-snippet struct, the call with the water inputs, the probe footer and the capabilities and error text), which lets a snippet draw flat tones and hard foam on water that otherwise waits for a core change (#2041).
+- goal:lit-box: 45 bytes gzip in the core for one shader slot after lighting and the renderer line that compiles the water locals in for a snippet with a water hook; the hook itself lives in the lazy snippet chunk.
+- goal:gltf-box: the same 45 bytes as goal:lit-box.
+- goal:agent-scene: the same 45 bytes as goal:lit-box.
diff --git a/docs/user-passes.md b/docs/user-passes.md
index a49bef9b..047166e4 100644
--- a/docs/user-passes.md
+++ b/docs/user-passes.md
@@ -117,7 +117,7 @@ As a recipe section:
# Material fragment hook
-A material can carry its own fragment code: a GLSL function that changes the surface colour and alpha, or discards the fragment. Use it for a dissolve, a shoreline cut from a mask, a leaf cutout or a stripe pattern. It works from the `snippet` field of a recipe material and from MCP (`scene_create`, `scene_apply` with `set-material`; `snippet: null` clears it). It follows the same rules as a post pass, and MCP accepts no host code.
+A material can carry its own fragment code: a GLSL function that changes the surface colour and alpha, or discards the fragment (water has a second entry point after lighting, [below](#a-water-hook-after-lighting)). Use it for a dissolve, a shoreline cut from a mask, a leaf cutout or a stripe pattern. It works from the `snippet` field of a recipe material and from MCP (`scene_create`, `scene_apply` with `set-material`; `snippet: null` clears it). It follows the same rules as a post pass, and MCP accepts no host code.
```json
{ "id": "stone", "color": [0.5, 0.5, 0.5], "snippet": { "fragment": "...", "uniforms": { "threshold": 0.4 }, "mask": "map" } }
@@ -125,12 +125,41 @@ A material can carry its own fragment code: a GLSL function that changes the sur
| Field | Meaning |
| --- | --- |
-| `fragment` | GLSL ES 3.00 text that defines `void materialHook(inout vec3 color, inout float alpha, vec2 uv, vec3 worldPosition)`. Helper functions and `const` values may come before it. 4096 bytes of code at most; comments and extra spaces are not counted (the text as sent may be up to 65536 bytes). |
+| `fragment` | GLSL ES 3.00 text that defines `void materialHook(inout vec3 color, inout float alpha, vec2 uv, vec3 worldPosition)`, or `waterHook` below. Helper functions and `const` values may come before it. 4096 bytes of code at most; comments and extra spaces are not counted (the text as sent may be up to 65536 bytes). |
| `uniforms` | Named values, as for a post pass: a number is a `float`, `[x, y]` to `[x, y, z, w]` are vectors, `{ "type": "int", "value": 4 }` is an int. At most 16. |
| `mask` | Optional, `"map"`: the snippet may sample `snippetMask` (a `sampler2D`), which is the material's own map or pattern. Load the image as the material `map`; there are no remote URLs. |
Inputs: `color` is the base colour after the map and the vertex tint, `alpha` is the map alpha before the material opacity, `uv` is the texture coordinate and `worldPosition` the position in the world. `discard` removes the fragment in every alpha mode. A lowered `alpha` cuts only when the material has `alphaMode: "mask"`.
+## A water hook, after lighting
+
+A snippet may define `void waterHook(inout vec3 color, WaterInputs water)` instead of `materialHook`, or next to it. It runs only on a material with `"tag": "water"`, after the engine has lit the surface and before fog and tone mapping, so it can replace the lighting with flat tones or add a hard edge. `color` is the lit linear radiance, in and out. The engine declares the `WaterInputs` type; you do not.
+
+| Field | Meaning |
+| --- | --- |
+| `water.normal` | The shading normal, ripples and waves included. |
+| `water.view` | Unit vector from the surface to the eye. |
+| `water.foam` | The foam of the water shader, 0 to 1, from the ocean `foam` vertex group. |
+| `water.depthBelow` | Metres of water over the floor below. Always 0 for now: the water shader does not resolve it yet. |
+| `water.time` | Seconds, as the engine's clock. |
+| `water.lit` | The key light facing the surface times the shadow it receives, 0 to 1. |
+
+The same limits and checks as the material hook apply: 4096 bytes, 16 uniforms, no `#` lines, counted loops only. A snippet with only `waterHook` adds nothing to the shadow and ID passes. The path tracer and exports ignore it.
+
+```glsl
+void waterHook(inout vec3 color, WaterInputs water) {
+ float tilt = 1.0 - water.normal.y;
+ vec3 tone = tone0;
+ if (tilt > 0.004) tone = tone1;
+ if (tilt > 0.012) tone = tone2;
+ if (tilt > 0.03) tone = tone3;
+ color = tone * (0.6 + 0.4 * step(0.5, water.lit));
+ if (water.foam > foamCut) color = vec3(0.9, 0.95, 1.0);
+}
+```
+
+With `tone0` to `tone3` as vec3 uniforms and `foamCut` of 0.3, that draws four flat tones from the wave slope and hard-edged foam on an ocean sea.
+
## Where the hook runs
The hook runs in the colour pass, in the shadow pass and in the ID pass. A discarded fragment therefore casts no shadow and is not picked. In the shadow pass `color` starts white and `alpha` is the map alpha, so base a discard on `alpha`, `uv`, `worldPosition` and your uniforms, not on `color`. The path tracer and exports ignore the hook. Changing a snippet makes the MCP preview build a new renderer, so edit it between looks, not in a loop.
diff --git a/src/material-snippet.js b/src/material-snippet.js
index 591c6ddd..b016453a 100644
--- a/src/material-snippet.js
+++ b/src/material-snippet.js
@@ -10,17 +10,22 @@ import {
// shadow pass runs the same hook, so a discarded fragment casts no shadow.
/** Names the engine provides to a snippet; a uniform cannot take one. */
-export const MATERIAL_SNIPPET_INPUTS = Object.freeze(['materialHook', 'snippetMask']);
+export const MATERIAL_SNIPPET_INPUTS = Object.freeze(['materialHook', 'waterHook', 'snippetMask']);
const SNIPPET_FIELDS = ['fragment', 'uniforms', 'mask', 'feature'];
+const MATERIAL_HOOK_PATTERN = /\bvoid\s+materialHook\s*\(\s*inout\s+vec3\s+[A-Za-z_]\w*\s*,\s*inout\s+float\s+[A-Za-z_]\w*\s*,\s*vec2\s+[A-Za-z_]\w*\s*,\s*vec3\s+[A-Za-z_]\w*\s*\)\s*\{/;
+// The water entry runs after lighting on water surfaces only; the engine declares the WaterInputs struct for it.
+const WATER_HOOK_PATTERN = /\bvoid\s+waterHook\s*\(\s*inout\s+vec3\s+[A-Za-z_]\w*\s*,\s*WaterInputs\s+[A-Za-z_]\w*\s*\)\s*\{/;
+const WATER_INPUTS_FIELDS = 'vec3 normal; vec3 view; float foam; float depthBelow; float time; float lit;';
const ENTRY_FUNCTION = Object.freeze({
- pattern: /\bvoid\s+materialHook\s*\(\s*inout\s+vec3\s+[A-Za-z_]\w*\s*,\s*inout\s+float\s+[A-Za-z_]\w*\s*,\s*vec2\s+[A-Za-z_]\w*\s*,\s*vec3\s+[A-Za-z_]\w*\s*\)\s*\{/,
- example: 'void materialHook(inout vec3 color, inout float alpha, vec2 uv, vec3 worldPosition) { }',
+ pattern: new RegExp(`${MATERIAL_HOOK_PATTERN.source}|${WATER_HOOK_PATTERN.source}`),
+ example: 'void materialHook(inout vec3 color, inout float alpha, vec2 uv, vec3 worldPosition) { } or void waterHook(inout vec3 color, WaterInputs water) { } (or both)',
});
/** What scene_capabilities reports under materialSnippet; it lives here so the scene entry does not carry the text. */
export const MATERIAL_SNIPPET_CAPABILITIES = Object.freeze({
fields: 'material.snippet { fragment (defines void materialHook(inout vec3 color, inout float alpha, vec2 uv, vec3 worldPosition)), uniforms, mask: "map" }; null clears it',
inputs: 'color (albedo after the map), alpha (map alpha), uv, worldPosition; snippetMask (the material map, with mask "map"); discard',
+ waterHook: 'void waterHook(inout vec3 color, WaterInputs water) on tag "water" only, after lighting, before fog and tone mapping. color is the lit linear radiance, in and out. water.normal (shading normal with ripples), water.view (surface to eye), water.foam (0 to 1, the ocean foam mask), water.depthBelow (metres to the floor below; 0 for now), water.time (seconds), water.lit (key light facing times shadow, 0 to 1)',
uniforms: 'as postPasses.uniforms',
rules: 'as postPasses.rules',
limits: Object.freeze({ maxBytes: SNIPPET_LIMITS.maxBytes, maxUniforms: SNIPPET_LIMITS.maxUniforms }),
@@ -87,10 +92,10 @@ function replaceName(code, name, replacement) {
function prefixedCode(snippet) {
const prefix = `u67_${snippet.feature}_`;
let code = snippet.fragment;
- const names = new Set([...topLevelNames(code), ...Object.keys(snippet.uniforms)]);
+ const names = new Set([...topLevelNames(code), ...Object.keys(snippet.uniforms), 'WaterInputs']);
for (const name of names) code = replaceName(code, name, `${prefix}${name}`);
const declarations = snippetUniformDeclarations(Object.fromEntries(Object.entries(snippet.uniforms).map(([name, uniform]) => [`${prefix}${name}`, uniform])));
- return { code, declarations, hook: `${prefix}materialHook` };
+ return { code, declarations, prefix, hook: `${prefix}materialHook`, waterHook: `${prefix}waterHook`, waterInputs: `${prefix}WaterInputs` };
}
const DEPTH_FRAGMENT_HEAD = 'void main() {\n if (alphaMasked < 0.5) return;\n float alpha = materialOpacity;\n if (hasBaseMap > 0.5) alpha *= texture(baseMap, mapUV(hasBaseMap)).a;\n';
@@ -117,14 +122,21 @@ function patchDepthVertex(source) {
}
function registerSnippetFeature(snippet) {
- const { code, declarations, hook } = prefixedCode(snippet);
+ const { code, declarations, hook, waterHook, waterInputs } = prefixedCode(snippet);
const flag = `u67_${snippet.feature}`;
- const mainCode = replaceName(`uniform float ${flag};\n${declarations}\n${code}\n`, 'snippetMask', 'baseMap');
+ const hasMaterialHook = MATERIAL_HOOK_PATTERN.test(snippet.fragment);
+ const hasWaterHook = WATER_HOOK_PATTERN.test(snippet.fragment);
+ const waterStruct = hasWaterHook ? `struct ${waterInputs} { ${WATER_INPUTS_FIELDS} };\n` : '';
+ const mainCode = replaceName(`uniform float ${flag};\n${declarations}\n${waterStruct}${code}\n`, 'snippetMask', 'baseMap');
const call = `if (${flag} > 0.5) ${hook}(hookColor, hookAlpha, textureUV, snippetWorld);\n`;
+ // The water values are locals of the main shader; the renderer compiles the world material code in for a snippet with a water hook.
+ const waterArguments = `${waterInputs}(normal, viewDirection, waterFoam, 0.0, time, clamp(dot(normal, normalize(lightDirection)), 0.0, 1.0) * visibility)`;
registerShaderFeatureCode(snippet.feature, Object.freeze({
libraries: mainCode,
- surfaceHook: ` if (${flag} > 0.5) ${hook}(color, textureAlpha, textureUV, worldPosition);\n`,
+ ...(hasMaterialHook ? { surfaceHook: ` if (${flag} > 0.5) ${hook}(color, textureAlpha, textureUV, worldPosition);\n` } : {}),
+ ...(hasWaterHook ? { waterHook: ` if (TAG_WATER && ${flag} > 0.5) ${waterHook}(radiance, ${waterArguments});\n` } : {}),
depth: (source) => {
+ if (!hasMaterialHook) return source;
if (source.includes('lightProjection * world')) return patchDepthVertex(source);
let text = source;
if (!text.includes(DEPTH_CALL_MARK)) {
@@ -190,7 +202,14 @@ precision highp int;
uniform sampler2D snippetMask;
out vec4 u67_output;
`;
-const PROBE_FOOTER = '\nvoid main() { vec3 color = vec3(0.5); float alpha = 1.0; materialHook(color, alpha, vec2(0.5), vec3(0.0)); u67_output = vec4(color, alpha); }\n';
+const waterStructDeclaration = (fragment) => (WATER_HOOK_PATTERN.test(fragment) ? `struct WaterInputs { ${WATER_INPUTS_FIELDS} };
+` : '');
+
+function probeFooter(fragment) {
+ const materialCall = MATERIAL_HOOK_PATTERN.test(fragment) ? 'materialHook(color, alpha, vec2(0.5), vec3(0.0));' : '';
+ const waterCall = WATER_HOOK_PATTERN.test(fragment) ? 'waterHook(color, WaterInputs(vec3(0.0, 1.0, 0.0), vec3(0.0, 1.0, 0.0), 0.5, 0.0, 0.0, 1.0));' : '';
+ return `\nvoid main() { vec3 color = vec3(0.5); float alpha = 1.0; ${materialCall} ${waterCall} u67_output = vec4(color, alpha); }\n`;
+}
function newProbeContext() {
const canvas = typeof OffscreenCanvas === 'function' ? new OffscreenCanvas(1, 1) : document.createElement('canvas');
@@ -213,9 +232,9 @@ export function probeMaterialSnippets(parts, createContext = newProbeContext) {
if (!context) return;
try {
for (const { snippet, parts: users } of snippets.values()) {
- const header = `${PROBE_HEADER}${snippetUniformDeclarations(snippet.uniforms)}\n`;
+ const header = `${PROBE_HEADER}${waterStructDeclaration(snippet.fragment)}${snippetUniformDeclarations(snippet.uniforms)}\n`;
const shader = context.createShader(context.FRAGMENT_SHADER);
- context.shaderSource(shader, `${header}${snippet.fragment}${PROBE_FOOTER}`);
+ context.shaderSource(shader, `${header}${snippet.fragment}${probeFooter(snippet.fragment)}`);
context.compileShader(shader);
const compiled = context.getShaderParameter(shader, context.COMPILE_STATUS);
const log = compiled ? '' : context.getShaderInfoLog(shader);
diff --git a/src/renderer.js b/src/renderer.js
index 2a3cdec3..e1f3d4e9 100644
--- a/src/renderer.js
+++ b/src/renderer.js
@@ -111,7 +111,7 @@ export function shaderFeatureRepresentatives(parts) {
return representatives.length > 0 ? representatives : parts.slice(0, 1);
}
-function shaderFeaturesOfParts(parts) {
+export function shaderFeaturesOfParts(parts) {
const features = new Set();
for (const part of parts) {
const { material } = part;
@@ -135,7 +135,11 @@ function shaderFeaturesOfParts(parts) {
if ((material.alphaMode ?? 'opaque') !== 'opaque') features.add('alphaMode');
if (material.subsurface?.weight > 0) features.add('subsurface');
if (material.wear) features.add('surfaceWear');
- if (material.snippet) features.add(material.snippet.feature);
+ if (material.snippet) {
+ features.add(material.snippet.feature);
+ // A water hook reads the water values the world material code declares, so that code is compiled in.
+ if (/\bvoid\s+waterHook\s*\(/.test(material.snippet.fragment)) features.add('water');
+ }
if (materialLayerValue(material, 'clearcoat') > 0) features.add('clearcoat');
if (materialLayerValue(material, 'sheenColor').some(channel => channel !== 0)) features.add('sheen');
if (TAG_SHADER_FEATURES.has(material.tag)) features.add(material.tag);
diff --git a/src/shaders.js b/src/shaders.js
index a7de9a90..5d21ef92 100644
--- a/src/shaders.js
+++ b/src/shaders.js
@@ -917,7 +917,7 @@ ${featureCode("irradianceGrid")} vec3 keyResponse = keyColor / mix(vec3(5.2, 4.
${featureCode("worldLight")} if (USES_SHEEN) radiance = sheenLayer(radiance, normal, viewDirection, visibility);
if (USES_CLEARCOAT) radiance = clearCoatLayer(radiance, normal, viewDirection, visibility, materialCoat.x * coatKept);
radiance += emission;
-${featureCode("afterEmission")}${featureCode("worldAfterEmission")} if (TAG_FLOOR) {
+${featureCode("afterEmission")}${featureCode("worldAfterEmission")}${featureCode("waterHook")} if (TAG_FLOOR) {
float contact = 1.0 - 0.45 * exp(-dot(worldPosition.xz, worldPosition.xz) * 1.4);
radiance = vec3(0.034, 0.041, 0.046) * (0.4 + visibility * 0.6) * contact;
float fog = 1.0 - exp(-length(worldPosition.xz) * 0.05);
diff --git a/tests/material-snippet.test.js b/tests/material-snippet.test.js
index df846580..850ae76c 100644
--- a/tests/material-snippet.test.js
+++ b/tests/material-snippet.test.js
@@ -2,7 +2,8 @@ import test from 'node:test';
import assert from 'node:assert/strict';
import { normalizeMaterialSnippet, probeMaterialSnippets, MATERIAL_SNIPPET_CAPABILITIES } from '../src/material-snippet.js';
import { createFragmentShader, createVertexShader, createDepthVertexShader, withDepthCode, depthFragmentShader } from '../src/shaders.js';
-import { groupParts } from '../src/renderer.js';
+import { groupParts, shaderFeaturesOfParts } from '../src/renderer.js';
+import '../src/world-material-shader.js';
import { createAgentScene, loadAgentSceneModules } from '../src/agent-scene.js';
const DISSOLVE = `float hash(vec2 p) { return fract(sin(dot(p, vec2(12.9898, 78.233))) * 43758.5453); }
@@ -148,3 +149,74 @@ test('a recipe sets, edits and clears the snippet and scene_capabilities reports
assert.equal(scene.parts.find(part => part.name === 'a').material.snippet, undefined);
assert.equal(MATERIAL_SNIPPET_CAPABILITIES.limits.maxUniforms, 16);
});
+
+const FOUR_TONES = `void waterHook(inout vec3 color, WaterInputs water) {
+ float band = floor(clamp(water.lit, 0.0, 0.999) * 4.0) / 3.0;
+ color = mix(deep, shallow, band);
+ if (water.foam > 0.5) color = vec3(1.0);
+}`;
+const fourTones = (overrides = {}) => ({ fragment: FOUR_TONES, uniforms: { deep: [0.02, 0.1, 0.3], shallow: [0.2, 0.6, 0.8] }, ...overrides });
+
+test('a snippet may define only waterHook, and the entry error names both entries', () => {
+ const snippet = normalizeMaterialSnippet(fourTones());
+ assert.match(snippet.feature, /^s[0-9a-f]{8}$/);
+ refuses(fourTones({ fragment: 'void waterHook(inout vec3 color) { }' }), /entry function is missing.*waterHook\(inout vec3 color, WaterInputs water\)/s);
+ refuses(fourTones({ uniforms: { waterHook: 1 } }), /keyword or an input the engine provides/);
+ refuses(fourTones({ fragment: FOUR_TONES.replace('water.lit', 'water.lit /* x */').replace('color = mix', '#define A 1\n color = mix') }), /Preprocessor/);
+ refuses(fourTones({ fragment: `${FOUR_TONES}\n${'x'.repeat(5000)}` }), /4096|bytes/i);
+});
+
+test('the water hook runs after lighting on water only, with its own struct, and leaves the material hook out', () => {
+ const snippet = normalizeMaterialSnippet(fourTones());
+ const prefix = `u67_${snippet.feature}_`;
+ const source = createFragmentShader(['studioLighting', 'water', snippet.feature]);
+ assert.ok(source.includes(`struct ${prefix}WaterInputs { vec3 normal; vec3 view; float foam; float depthBelow; float time; float lit; };`));
+ assert.ok(source.includes(`void ${prefix}waterHook(inout vec3 color, ${prefix}WaterInputs water)`));
+ const call = source.indexOf(`if (TAG_WATER && u67_${snippet.feature} > 0.5) ${prefix}waterHook(radiance, ${prefix}WaterInputs(normal, viewDirection, waterFoam, 0.0, time,`);
+ assert.ok(call > 0, 'the call is in the main shader');
+ assert.ok(call > source.indexOf('radiance += emission;'), 'after lighting and emission');
+ assert.ok(call < source.indexOf('float fog = 1.0 - exp(-length(worldPosition - eye)'), 'before fog and tone mapping');
+ assert.doesNotMatch(source, new RegExp(`${prefix}materialHook`));
+ assert.doesNotMatch(createFragmentShader(['studioLighting', normalizeMaterialSnippet(dissolve()).feature]), /waterHook/);
+ const shadowSource = withDepthCode(depthFragmentShader, new Set(['studioLighting', snippet.feature]));
+ assert.equal(shadowSource, depthFragmentShader, 'a water-only snippet adds nothing to the shadow pass');
+ assert.doesNotMatch(shadowSource, /waterHook|WaterInputs/);
+});
+
+test('two water snippets share one shader without clashing on the struct', () => {
+ const first = normalizeMaterialSnippet(fourTones());
+ const second = normalizeMaterialSnippet(fourTones({ uniforms: { deep: [0, 0, 0], shallow: [1, 1, 1], extra: 1 } }));
+ const source = createFragmentShader(['studioLighting', 'water', first.feature, second.feature]);
+ assert.match(source, new RegExp(`struct u67_${first.feature}_WaterInputs`));
+ assert.match(source, new RegExp(`struct u67_${second.feature}_WaterInputs`));
+});
+
+test('a snippet with a water hook asks the renderer for the water shading code, one without does not', async () => {
+ const featuresWith = async (snippet) => {
+ const recipe = {
+ format: 'crucible.scene', schemaVersion: 1,
+ geometries: [{ id: 'plane', type: 'box' }],
+ materials: [{ id: 'surface', snippet }],
+ nodes: [{ id: 'surface', geometry: 'plane', material: 'surface' }],
+ };
+ await loadAgentSceneModules(recipe);
+ return shaderFeaturesOfParts(createAgentScene(recipe).parts);
+ };
+ assert.ok((await featuresWith(fourTones())).has('water'), 'the water locals the hook reads are declared');
+ assert.ok(!(await featuresWith(dissolve())).has('water'), 'a material hook alone adds nothing');
+});
+
+test('probing a water snippet compiles it with the struct declared and reports its lines', () => {
+ const snippet = normalizeMaterialSnippet({ fragment: 'void waterHook(inout vec3 color, WaterInputs water) {\n vec3 wrong = 1.0;\n}' });
+ let compiled = '';
+ const context = {
+ FRAGMENT_SHADER: 1, COMPILE_STATUS: 2,
+ createShader: () => ({}), shaderSource: (shader, text) => { compiled = text; }, compileShader() {}, deleteShader() {},
+ getShaderParameter: () => true, getShaderInfoLog: () => '', getExtension: () => null,
+ };
+ probeMaterialSnippets([{ name: 'sea', material: { snippet } }], () => context);
+ assert.match(compiled, /struct WaterInputs \{ vec3 normal; vec3 view; float foam; float depthBelow; float time; float lit; \};/);
+ assert.match(compiled, /waterHook\(color, WaterInputs\(/);
+ assert.doesNotMatch(compiled, /\bmaterialHook\(/);
+ assert.match(MATERIAL_SNIPPET_CAPABILITIES.waterHook, /WaterInputs water/);
+});