3D engine that AI agents drive
Size the memory budget to the scene and name the bytes in budget errors (closes #470)
Nguyen Xuan Anfc9076c0da71
diff --git a/changes/470-renderer-budget.md b/changes/470-renderer-budget.mdnew file mode 100644index 00000000..ed86590c--- /dev/null+++ b/changes/470-renderer-budget.md@@ -0,0 +1,3 @@+- `Renderer67` sizes its memory budget to the scene when `memoryBudgetMiB` is not passed: room for the geometry, textures and one full frame, at least 256 MiB, at most 1024 MiB on a device that reports 8 GiB of memory or more (512 MiB otherwise). Large exports such as a 234 MB warehouse and a 244 MB paint studio now draw with no options (issue #470). Pass `memoryBudgetMiB` to keep a fixed budget.+- Memory budget errors name the geometry, texture and frame bytes they counted and the `memoryBudgetMiB` that fits, ending in `Fix: pass { memoryBudgetMiB: 365 }`, like the glTF limit errors.+- When `maxTexturePixels` scales textures down, base color and emission maps keep twice the size of the normal, roughness and occlusion maps, for the same memory. The warning names three scaled images and a count; `asset.scaledTextures` still lists every one.diff --git a/changes/470-renderer-budget.size.md b/changes/470-renderer-budget.size.mdnew file mode 100644index 00000000..3aa30ff1--- /dev/null+++ b/changes/470-renderer-budget.size.md@@ -0,0 +1,6 @@+- entry:core: +114 B so a renderer with no memoryBudgetMiB sizes its budget to the scene (bounded at 512 or 1024 MiB) and budget errors name the bytes and the memoryBudgetMiB that fits. The 234 MB warehouse and 244 MB paint studio exports now draw with default options; before, both threw.+- scene:lit-box: the same core code, +111 B.+- scene:agent-scene: the same core code, +118 B.+- scene:agent-scene+lazy: the same core code, +120 B.+- scene:gltf-box: core code plus color-first texture scaling and a three-name scaling warning, +212 B.+- scene:gltf-box+lazy: the same as gltf-box, +212 B.diff --git a/docs/gltf.md b/docs/gltf.mdindex a1141dfe..153b7c4c 100644--- a/docs/gltf.md+++ b/docs/gltf.md@@ -39,7 +39,7 @@ The optional `signal` aborts those fetches.| `maxVertices` | 8,000,000 | Declared and decoded geometry vertices. Indices have a separate ceiling of twelve times this value. || `maxParts` | 65,536 | Declared nodes and decoded mesh parts. || `maxResources` | 256 | Unique resource URIs. |-| `maxTexturePixels` | 16,777,216 | Combined texture image pixels, checked from the file headers before decoding. Over the budget, the largest images are halved together until they fit, and `asset.scaledTextures` lists each one as `{ name, from, to }` (also summarised in `asset.warnings`). |+| `maxTexturePixels` | 16,777,216 | Combined texture image pixels, checked from the file headers before decoding. Over the budget, the largest images are halved together until they fit, with base color and emission maps kept at twice the size of the normal, roughness and occlusion maps, and `asset.scaledTextures` lists each one as `{ name, from, to }` (also summarised in `asset.warnings`). || `maxTextureImagePixels` | 67,108,864 | Hard ceiling for one image, checked before decoding. Larger images are rejected, never scaled. || `maxJoints` | 4,096 | Combined declared joints and joint palettes for instantiated meshes. || `maxInstances` | 65,536 | Instances of one `EXT_mesh_gpu_instancing` node. |diff --git a/docs/guide.md b/docs/guide.mdindex 4cfc3790..a70aaf4d 100644--- a/docs/guide.md+++ b/docs/guide.md@@ -335,7 +335,7 @@ Tags select specialized behavior: `armor` allows studio finish overrides. `floor| samples | 4 | Requested MSAA count. Negotiated against hardware || hdr | false | Request floating-point color targets || (compact colour) | off | `useCompactColor(renderer)` from `@codepawlhq/67js/addons/compact-color` stores the HDR frame and its multisample colour as R11F_G11F_B10F instead of RGBA16F: 39.55 MB less at 1920 x 1080 with 4 samples (110.92 to 71.37 MB). The format has no alpha, so it needs `hdr: true` and an opaque canvas and throws otherwise; it also throws when the device does not support the renderer's sample count in that format. Mean FLIP against the default is 0.002 to 0.030 (median 0.021) over the 26 reference scenes. Without the call nothing changes. `renderer.memoryReport()` lists the formats |-| memoryBudgetMiB | 256 | Estimated render allocation budget |+| memoryBudgetMiB | sized to the scene | Estimated render allocation budget. Without it the renderer takes room for the scene and one full frame: at least 256 MiB, at most 1024 MiB on a device reporting 8 GiB of memory or more (512 MiB otherwise). A scene over the budget fails with an error that names the bytes and the `memoryBudgetMiB` that fits. || environment | {} | Sky, light and fog settings || shadowBounds | {} | Custom extent, eye and target for shadow framing || benchmark | false | Diagnostic flat shader. Not the appearance renderer |diff --git a/src/budget.js b/src/budget.jsindex b034e973..93b1a78c 100644--- a/src/budget.js+++ b/src/budget.js@@ -1,5 +1,14 @@const MEBIBYTE = 1024 * 1024;+/**+ * The error for a scene the memory budget cannot hold: the bytes it counted and the memoryBudgetMiB that fits them,+ * so one failed call says what to pass.+ */+export function memoryBudgetError(geometry, textures, frame, memoryBudgetMiB) {+ const mebibytes = bytes => (bytes / MEBIBYTE).toFixed(1);+ return new RangeError(`Memory budget cannot hold geometry ${mebibytes(geometry)} MiB, textures ${mebibytes(textures)} MiB and frame ${mebibytes(frame)} MiB, over memoryBudgetMiB ${memoryBudgetMiB}. Fix: pass { memoryBudgetMiB: ${Math.ceil((geometry + textures + frame) / MEBIBYTE)} }.`);+}+export const RENDER_PRESETS = Object.freeze({preview: { width: 960, height: 540, samples: 2, fps: 24, memoryBudgetMiB: 256 },balanced: { width: 1280, height: 720, samples: 4, fps: 24, memoryBudgetMiB: 384 },@@ -27,7 +36,15 @@ export function estimateFrameBytes(width, height, samples, shadowResolution = 20return frameBytes + bloomBytes + (depth ? width * height * 4 : 0) + shadowResolution ** 2 * 4;}-export function planRenderSize({ width, height, pixelRatio = 1, samples = 4, memoryBudgetMiB = 256, geometryBytes = 0, textureBytes = 0, hdr = false, bloom = false, depth = false, shadowResolution = 2048, maximumSize = 8192 }) {+/**+ * Without memoryBudgetMiB the budget is sized to the scene: its geometry, textures and one full frame, at least 256 MiB+ * and at most 1024 MiB on a device that reports 8 GiB of memory or more (512 MiB otherwise). The plan returns it.+ */+export function planRenderSize({ width, height, pixelRatio = 1, samples = 4, memoryBudgetMiB, geometryBytes = 0, textureBytes = 0, hdr = false, bloom = false, depth = false, shadowResolution = 2048, maximumSize = 8192 }) {+ const requestedWidth = Math.max(1, Math.round(width * pixelRatio));+ const requestedHeight = Math.max(1, Math.round(height * pixelRatio));+ const fullFrameBytes = estimateFrameBytes(Math.min(requestedWidth, maximumSize), Math.min(requestedHeight, maximumSize), samples, shadowResolution, { hdr, bloom, depth });+ memoryBudgetMiB ??= Math.min(globalThis.navigator?.deviceMemory >= 8 ? 1024 : 512, Math.max(256, Math.ceil((geometryBytes + textureBytes + fullFrameBytes) / MEBIBYTE)));for (const value of [width, height, pixelRatio, samples, memoryBudgetMiB, maximumSize]) {if (!Number.isFinite(value) || value <= 0) throw new Error('Render dimensions, sample count and budget must be positive finite numbers.');}@@ -36,9 +53,8 @@ export function planRenderSize({ width, height, pixelRatio = 1, samples = 4, memif (!Number.isFinite(shadowResolution) || shadowResolution < 0) throw new Error('Shadow resolution must be nonnegative and finite.');const available = memoryBudgetMiB * MEBIBYTE - geometryBytes - textureBytes - shadowResolution ** 2 * 4;const bytesPerPixel = (hdr ? 12 : 8) * Math.max(1, samples) + (hdr ? 24 : 20) + (bloom ? hdr ? 4 : 2 : 0) + (depth ? 4 : 0);- if (available < estimateFrameBytes(64, 64, samples, 0, { hdr, bloom, depth })) throw new Error('Memory budget cannot hold the scene and a minimum render target. Lower scene detail or increase the budget.');- const requestedWidth = Math.max(1, Math.round(width * pixelRatio));- const requestedHeight = Math.max(1, Math.round(height * pixelRatio));+ // The suggested budget fits the full requested frame, not only the minimum target.+ if (available < estimateFrameBytes(64, 64, samples, 0, { hdr, bloom, depth })) throw memoryBudgetError(geometryBytes, textureBytes, fullFrameBytes, memoryBudgetMiB);const scale = Math.min(1, maximumSize / requestedWidth, maximumSize / requestedHeight, Math.sqrt(available / (requestedWidth * requestedHeight * bytesPerPixel)));let renderWidth = Math.max(1, Math.floor(requestedWidth * scale));let renderHeight = Math.max(1, Math.floor(requestedHeight * scale));@@ -47,5 +63,5 @@ export function planRenderSize({ width, height, pixelRatio = 1, samples = 4, memrenderWidth = Math.max(1, renderWidth - 1);renderHeight = Math.max(1, renderHeight - 1);}- return { width: renderWidth, height: renderHeight, samples, hdr, bloom, depth, textureBytes, estimatedBytes: estimateFrameBytes(renderWidth, renderHeight, samples, shadowResolution, { hdr, bloom, depth }) + geometryBytes + textureBytes, reduced: renderWidth < requestedWidth || renderHeight < requestedHeight };+ return { width: renderWidth, height: renderHeight, samples, hdr, bloom, depth, textureBytes, estimatedBytes: estimateFrameBytes(renderWidth, renderHeight, samples, shadowResolution, { hdr, bloom, depth }) + geometryBytes + textureBytes, memoryBudgetMiB, reduced: renderWidth < requestedWidth || renderHeight < requestedHeight };}diff --git a/src/gltf.js b/src/gltf.jsindex 85798c2e..2ddcada6 100644--- a/src/gltf.js+++ b/src/gltf.js@@ -713,22 +713,38 @@ async function prepareResources(document, data, dataBytes, options, limits, overimageSizes.push(validateImage(bytes, imageIndex, limits, ktx2Weight, overruns));}throwOverruns(overruns);- return { resourceBytes, textureScaling: planTextureScaling(document.images ?? [], imageSizes, limits.maxTexturePixels) };+ return { resourceBytes, textureScaling: planTextureScaling(document.images ?? [], imageSizes, limits.maxTexturePixels, colorImageIndices(document)) };+}++/** Images that a material reads as base color or emission: the maps the eye sees most directly. */+function colorImageIndices(document) {+ const indices = new Set();+ for (const material of document.materials ?? []) {+ for (const reference of [material.pbrMetallicRoughness?.baseColorTexture, material.emissiveTexture]) {+ const texture = reference && document.textures?.[reference.index];+ if (texture) indices.add(textureSourceIndex(texture));+ }+ }+ return indices;}/*** Fit the combined texture pixels into the budget by halving the largest images together, so equal maps stay equal.+ * Base color and emission maps rank at a quarter of their pixels, so they keep twice the size of the normal,+ * roughness and occlusion maps beside them.* KTX2 images cannot be resized before transcoding; they count toward the budget (compressed ones at their weight) and are never scaled.*/-function planTextureScaling(imageDefinitions, imageSizes, maxTexturePixels) {+function planTextureScaling(imageDefinitions, imageSizes, maxTexturePixels, colorImages = new Set()) {const targets = imageSizes.map(({ width, height }) => ({ width, height }));const totalPixels = () => targets.reduce((sum, { width, height }, index) => sum + width * height * imageSizes[index].weight, 0);while (totalPixels() > maxTexturePixels) {- const scalable = targets.filter((target, index) => imageSizes[index].scalable && target.width * target.height > 1);+ const rank = index => targets[index].width * targets[index].height * (colorImages.has(index) ? 0.25 : 1);+ const scalable = targets.map((target, index) => index).filter(index => imageSizes[index].scalable && targets[index].width * targets[index].height > 1);if (!scalable.length) throw new Error('Texture images exceed maxTexturePixels and cannot be scaled down further.');- const largestPixels = Math.max(...scalable.map(target => target.width * target.height));- for (const target of scalable) {- if (target.width * target.height !== largestPixels) continue;+ const largestRank = Math.max(...scalable.map(rank));+ for (const index of scalable) {+ const target = targets[index];+ if (rank(index) !== largestRank) continue;target.width = Math.max(1, target.width >> 1);target.height = Math.max(1, target.height >> 1);}@@ -1021,8 +1037,11 @@ export async function parseGltf(data, options = {}) {const dataBytes = typeof data === 'string' ? new TextEncoder().encode(data).byteLength : data.byteLength;const { resourceBytes, textureScaling } = await prepareResources(document, data, dataBytes, options, limits, overruns);if (textureScaling.scaledTextures.length) {- const names = textureScaling.scaledTextures.map(({ name, from, to }) => `${name} ${from.join('x')} to ${to.join('x')}`);- warnings.push(`Scaled ${names.length} texture image(s) to fit maxTexturePixels: ${names.join(', ')}.`);+ // A large scene can scale hundreds of images; the warning names a few and asset.scaledTextures lists them all.+ const names = textureScaling.scaledTextures.slice(0, 3).map(({ name, from, to }) => `${name} ${from.join('x')} to ${to.join('x')}`);+ const more = textureScaling.scaledTextures.length - names.length;+ const rest = more ? ` and ${more} more (see asset.scaledTextures)` : '';+ warnings.push(`Scaled ${textureScaling.scaledTextures.length} texture image(s) to fit maxTexturePixels: ${names.join(', ')}${rest}.`);}let gltf;try {diff --git a/src/renderer.js b/src/renderer.jsindex 2e7f624b..654960fe 100644--- a/src/renderer.js+++ b/src/renderer.js@@ -4,7 +4,7 @@ import { SPLAT_UNITS, requireSplatSupport, layerTextureSupport, LIGHT_SHADING_FEimport { postFragment, createPostFragment, POST_STEPS, skyFragment, bloomFragment } from './post.js';import { fullscreenVertex, startProgram, finishProgram, compileProgram } from './gl-program.js';import { ambientOcclusionFragment } from './ambient-occlusion.js';-import { planRenderSize, estimateTextureBytes, estimateFrameBytes } from './budget.js';+import { planRenderSize, estimateTextureBytes, estimateFrameBytes, memoryBudgetError } from './budget.js';import { applyProjection } from './camera.js';import { resolveEffects } from './effects.js';import { frameDiagnostics } from './frame-diagnostics.js';@@ -449,7 +449,7 @@ function allFinite(values) {}export class Renderer67 {- constructor(canvas, parts, { batching = true, pixelRatio = 1.5, benchmark = false, world = false, ground = true, transparent = false, samples = 4, hdr = false, depth = false, memoryBudgetMiB = 256, environment = {}, shadowBounds = {}, shadowCascades = null, culling = true, lodPixelError = 1, createGpuTimer = null, skyOpenness = null, fogSkyVisibility = null, shading = 'full', viewTransform = 'aces', look = null } = {}) {+ constructor(canvas, parts, { batching = true, pixelRatio = 1.5, benchmark = false, world = false, ground = true, transparent = false, samples = 4, hdr = false, depth = false, memoryBudgetMiB, environment = {}, shadowBounds = {}, shadowCascades = null, culling = true, lodPixelError = 1, createGpuTimer = null, skyOpenness = null, fogSkyVisibility = null, shading = 'full', viewTransform = 'aces', look = null } = {}) {this.canvas = canvas;this.sceneParts = parts;// `skyOpenness` from closed-room.js is passed in, which keeps the ray code out of scenes without volume fog.@@ -496,7 +496,6 @@ export class Renderer67 {this.look = null;this.lookSource = null;this.selectViewTransform(viewTransform, look);- this.memoryBudgetMiB = memoryBudgetMiB;this.hdr = Boolean(hdr && !benchmark && this.context.getExtension('EXT_color_buffer_float'));if (volume.density > 0 && !this.hdr) throw new Error('Volume fog requires EXT_color_buffer_float for HDR rendering');this.multisampleSamples = benchmark ? 0 : sampleCount(this.context, this.hdr ? this.context.RGBA16F : this.context.RGBA8, Math.min(shading === 'light' ? Math.min(samples, LIGHT_SHADING_MAXIMUM_SAMPLES) : samples, this.context.getParameter(this.context.MAX_SAMPLES)));@@ -589,7 +588,7 @@ export class Renderer67 {if (geometry.colors && geometry.colors.length !== geometry.positions.length) throw new Error('Geometry colors must contain RGB for every vertex.');}const expectedGeometryBytes = groups.reduce((total, group) => total + geometryBufferBytes(group[0].geometry) + group.length * INSTANCE_STRIDE * 4, 0);- planRenderSize({ width: Math.max(1, canvas.clientWidth), height: Math.max(1, canvas.clientHeight), pixelRatio, samples: this.samples, memoryBudgetMiB, geometryBytes: expectedGeometryBytes, textureBytes: this.textureBytes, hdr: this.hdr, bloom: !benchmark, depth: this.depth, maximumSize: textureLimit });+ this.memoryBudgetMiB = planRenderSize({ width: Math.max(1, canvas.clientWidth), height: Math.max(1, canvas.clientHeight), pixelRatio, samples: this.samples, memoryBudgetMiB, geometryBytes: expectedGeometryBytes, textureBytes: this.textureBytes, hdr: this.hdr, bloom: !benchmark, depth: this.depth, maximumSize: textureLimit }).memoryBudgetMiB;this.view = mat4.create();this.projection = mat4.create();this.viewProjection = mat4.create();@@ -1062,7 +1061,7 @@ export class Renderer67 {const frameBytes = estimateFrameBytes(this.canvas.width, this.canvas.height, this.samples, 2048, { hdr: this.hdr, bloom: !this.benchmark, depth: this.depth });const peakBytes = frameBytes + this.textureBytes + this.geometryBytes + newGeometryBytes + newInstanceBytes;if (peakBytes > this.memoryBudgetMiB * 1024 * 1024) {- throw new RangeError('Memory budget cannot hold the current scene and staged replacement buffers.');+ throw memoryBudgetError(this.geometryBytes + newGeometryBytes + newInstanceBytes, this.textureBytes, frameBytes, this.memoryBudgetMiB);}const previousLocations = this.partLocations;const previousBuffers = [...this.buffers];diff --git a/tests/budget.test.js b/tests/budget.test.jsindex 7cf32007..b6d85fc0 100644--- a/tests/budget.test.js+++ b/tests/budget.test.js@@ -37,6 +37,19 @@ test('invalid or impossible memory requests fail before allocation', () => {assert.throws(() => planRenderSize({ width: Infinity, height: 720 }));});+test('a budget error names the bytes it counted and a memoryBudgetMiB that then fits', () => {+ const scene = { width: 1280, height: 720, geometryBytes: 300 * 1024 ** 2, textureBytes: 40 * 1024 ** 2, memoryBudgetMiB: 128 };+ let message = '';+ assert.throws(() => planRenderSize(scene), error => {+ message = error.message;+ return error instanceof RangeError;+ });+ assert.match(message, /geometry 300\.0 MiB, textures 40\.0 MiB and frame [0-9.]+ MiB, over memoryBudgetMiB 128\. Fix: pass \{ memoryBudgetMiB: \d+ \}/);+ const suggested = Number(message.match(/memoryBudgetMiB: (\d+) \}/)[1]);+ const plan = planRenderSize({ ...scene, memoryBudgetMiB: suggested });+ assert.equal(plan.reduced, false);+});+test('resolved depth adds exactly four bytes per pixel and participates in resolution reduction', () => {assert.equal(estimateFrameBytes(540, 960, 4, 2048, { hdr: true, depth: true }) - estimateFrameBytes(540, 960, 4, 2048, { hdr: true }), 540 * 960 * 4);const options = { width: 7680, height: 4320, hdr: true, bloom: true, samples: 4, memoryBudgetMiB: 64 };@@ -46,3 +59,12 @@ test('resolved depth adds exactly four bytes per pixel and participates in resolassert.equal(resolved.estimatedBytes, estimateFrameBytes(resolved.width, resolved.height, 4, 2048, { hdr: true, bloom: true, depth: true }));assert.ok(resolved.estimatedBytes <= 64 * 1024 ** 2);});++test('without memoryBudgetMiB the budget fits the scene, from 256 MiB up to a device ceiling', () => {+ const small = planRenderSize({ width: 1280, height: 720 });+ assert.equal(small.memoryBudgetMiB, 256);+ const large = planRenderSize({ width: 1440, height: 810, geometryBytes: 280 * 1024 ** 2, textureBytes: 60 * 1024 ** 2 });+ assert.ok(large.memoryBudgetMiB > 340 && large.memoryBudgetMiB <= 512);+ assert.equal(large.reduced, false);+ assert.throws(() => planRenderSize({ width: 1280, height: 720, geometryBytes: 600 * 1024 ** 2 }), /Fix: pass \{ memoryBudgetMiB: \d+ \}/);+});diff --git a/tests/gltf.test.js b/tests/gltf.test.jsindex cfaa4708..b5347063 100644--- a/tests/gltf.test.js+++ b/tests/gltf.test.js@@ -254,11 +254,26 @@ test('forty 1024 px maps load with default options by scaling them down and nami}));assert.equal(asset.scaledTextures.length, 40);assert.deepEqual(asset.scaledTextures[3], { name: 'map 3', from: [1024, 1024], to: [512, 512] });- assert.match(asset.warnings.join(' '), /Scaled 40 texture image\(s\)/);+ assert.match(asset.warnings.join(" "), /^Scaled 40 texture image[(]s[)] to fit maxTexturePixels: map 0 1024x1024 to 512x512, map 1 1024x1024 to 512x512, map 2 1024x1024 to 512x512 and 37 more [(]see asset.scaledTextures[)][.]$/);assert.equal(requests.length, 0, 'unreferenced images are not decoded');asset.dispose();});+test('base color keeps twice the size of a data map when textures are scaled', async context => {+ mockImageBitmapWithResize(context);+ const asset = await parseGltf(triangleAsset(document => {+ addMaterialMaps(document);+ document.images = [{ name: 'color', uri: pngHeaderUri(8, 8) }, { name: 'normal', uri: pngHeaderUri(8, 8) }];+ document.textures = [{ source: 0, sampler: 0 }, { source: 1, sampler: 0 }];+ const material = document.materials[0];+ material.pbrMetallicRoughness.metallicRoughnessTexture = { index: 1 };+ material.normalTexture = { index: 1 };+ material.occlusionTexture = { index: 1 };+ }), { maxTexturePixels: 80 });+ assert.deepEqual(asset.scaledTextures, [{ name: 'normal', from: [8, 8], to: [4, 4] }]);+ asset.dispose();+});+test('only the largest maps are scaled, and decoding requests the scaled size', async context => {const requests = mockImageBitmapWithResize(context);const asset = await parseGltf(triangleAsset(document => {diff --git a/tests/renderer-topology.test.js b/tests/renderer-topology.test.jsindex f129fc2a..4949112c 100644--- a/tests/renderer-topology.test.js+++ b/tests/renderer-topology.test.js@@ -127,7 +127,7 @@ test('invalid geometry and peak staging budgets reject before allocation and preassert.throws(() => state.renderer.replaceParts([first, invalid]), /indices/);assert.equal(state.uploads.length, allocations);state.renderer.memoryBudgetMiB = 16;- assert.throws(() => state.renderer.replaceParts([first, part('new', sphere(24, 16))]), /staged/);+ assert.throws(() => state.renderer.replaceParts([first, part('new', sphere(24, 16))]), /cannot hold geometry .* over memoryBudgetMiB 16\. Fix: pass [{] memoryBudgetMiB: 17 [}]/);assert.equal(state.renderer.batches[0], batch);assert.equal(state.uploads.length, allocations);});