Skip to content

3D engine that AI agents drive

Remove the stepped seam from baked curved meshes (#2212)

Nguyen Xuan An4ecac7f9679e
diff --git a/changes/2212.md b/changes/2212.md
new file mode 100644
index 00000000..57988d78
--- /dev/null
+++ b/changes/2212.md
@@ -0,0 +1 @@
+- Baked light no longer steps across the seams of a curved mesh: the unwrap leaves two and a half texels between charts and the smoothing averages over the same patch on both sides of a chart edge (#2212).
diff --git a/changes/2212.size.md b/changes/2212.size.md
new file mode 100644
index 00000000..d5c7a5fb
--- /dev/null
+++ b/changes/2212.size.md
@@ -0,0 +1 @@
+- scene:agent-scene+lazy: the bake's smoothing now finds neighbours by world distance and the unwrap sizes its chart gap in texels, 257 bytes gzip that remove a stepped seam from baked curved meshes (70 percent of the peak down to 4 percent on a sphere).
diff --git a/docs/light-bake.md b/docs/light-bake.md
index c992c65e..190c5916 100644
--- a/docs/light-bake.md
+++ b/docs/light-bake.md
@@ -64,6 +64,8 @@ A recipe `bake` built in a runtime on the GPU backend (the MCP server's default,
A baked part is a copy with new geometry and a new material. A plain part (no maps, pattern, layers or emission) gets lightmap UVs from the `uv_unwrap` unwrap in channel 0, and its material gets `emissiveMap` (the lightmap, `srgb`, linear filter, clamped), `emissiveColor [1, 1, 1]` and `emission` set to the brightest texel, so the renderer, its shaders and every scene without a bake stay as they are. The input parts are not changed.
+The unwrap leaves about two and a half texels between charts, so a texel read at a chart edge is the chart's own light and not its neighbour's, and the smoothing finds neighbours by distance in the world, so the two sides of a chart seam average the same patch. A curved part, a sphere for one, shows no step where its charts meet; a part with more charts or a bigger gap takes a few percent more texels.
+
A part with texture maps, a pattern, layers or its own emission cannot lend its emissive map or its UV channel 0, so it takes the light through the renderer's `lightMap` slot instead. Its geometry keeps `uvs` (the unwrap splits vertices along its seams, and each new vertex copies the `uvs` and `colors` of the vertex it came from) and gains `lightUvs`, the second UV set. The material keeps every field and gains `lightMap` (`srgb`, linear filter, clamped) and `lightMapScale` (the brightest texel). The texture holds the light that arrives, not the light times an albedo, and the fragment shader adds `color * (1 - metallic) * lightMap * lightMapScale` under the existing shading, so the pattern or texture (box projected or UV mapped) tints it at run time. The renderer uploads the two UV sets as one four-wide attribute; the shader branch, the sampler and the uniforms compile out of any scene with no `lightMap`. The bake traces such a part with its colour times the mean colour of its base map, so a brick wall reflects brick coloured light and not white light.
Skipped, with the reason in `report.skipped`: transparent parts, a part with a map or layer map (clearcoat, anisotropy, sheen) that reads the second UV set (`TEXCOORD_1`), which the bake would replace with the lightmap atlas, a material tag that draws its own look (water, fire, foliage), skinned or morphing parts, a cloth sheet, and a node that hosts a hair or particle system (its geometry is rebuilt every frame and has no UVs; the recipe bake marks these, which fixed `scene_create` failing with "Textured geometry requires UV channel 0"). Skipped parts still block and reflect light.
diff --git a/src/light-bake.js b/src/light-bake.js
index 517be178..93f6b742 100644
--- a/src/light-bake.js
+++ b/src/light-bake.js
@@ -46,6 +46,12 @@ const SMOOTH_SIGMA_TEXELS = 1.5;
const SMOOTH_MINIMUM_FACING = 0.9;
const SMOOTH_PLANE_TEXELS = 1;
const DILATION_PASSES = 2;
+// Texels of empty space between charts, so each chart edge has its own filled texels on both sides (dilation fills two).
+const CHART_GAP_TEXELS = 2.5;
+// The share of the atlas the charts cover once the gap is in; it picks the resolution the gap is sized for before the unwrap.
+const EXPECTED_UTILIZATION = 0.6;
+const MINIMUM_CHART_MARGIN = 0.01;
+const MAXIMUM_CHART_MARGIN = 0.1;
// How far outside a chart (in texels, centre to edge) a texel still takes the chart's own surface instead of a neighbour's light.
const GUTTER_TEXELS = 0.75;
const AVERAGE_GRID = 16;
@@ -601,10 +607,10 @@ function unscaledAfterUnwrap(extracted, scale) {
}
/** Gives a geometry its lightmap UVs in channel 0, from the unwrap that `uv_unwrap` uses. */
-function unwrapForLightmap(geometry, scale) {
+function unwrapForLightmap(geometry, scale, margin) {
const { positions, normals, indices } = scaledForUnwrap(geometry, scale);
const mesh = meshFromGeometry({ positions, normals, indices }, { quads: false, tags: 'none' });
- const unwrapped = unwrapUv(mesh);
+ const unwrapped = unwrapUv(mesh, { margin });
const { geometry: unwrappedGeometry } = extractRender(unwrapped.mesh);
return { geometry: unscaledAfterUnwrap(unwrappedGeometry, scale), utilization: Math.max(unwrapped.utilization, SURFACE_UTILIZATION_FLOOR) };
}
@@ -613,13 +619,13 @@ function unwrapForLightmap(geometry, scale) {
* Gives a geometry lightmap UVs in `lightUvs` and keeps its own `uvs` and `colors`. The unwrap splits vertices along its
* seams, so each source vertex index rides through it as a corner colour and the attributes are gathered back from it.
*/
-function unwrapIntoSecondUvSet(geometry, scale) {
+function unwrapIntoSecondUvSet(geometry, scale, margin) {
const { positions, normals, indices } = scaledForUnwrap(geometry, scale);
const vertexCount = positions.length / 3;
const sourceIndices = new Float32Array(vertexCount * 3);
for (let vertex = 0; vertex < vertexCount; vertex++) sourceIndices[vertex * 3] = vertex;
const mesh = meshFromGeometry({ positions, normals, indices, colors: sourceIndices }, { quads: false, tags: 'none' });
- const unwrapped = unwrapUv(mesh);
+ const unwrapped = unwrapUv(mesh, { margin });
const extracted = unscaledAfterUnwrap(extractRender(unwrapped.mesh).geometry, scale);
const newCount = extracted.positions.length / 3;
const result = { positions: extracted.positions, normals: extracted.normals, indices: extracted.indices, lightUvs: extracted.uvs };
@@ -652,6 +658,16 @@ function lightmapResolution(area, utilization, options) {
return Math.min(Math.max(rounded, BAKE_LIMITS.maxResolution.minimum), options.maxResolution);
}
+/**
+ * The gap the unwrap leaves between charts and around the atlas, as a share of it. It is sized in texels: bilinear sampling at
+ * a chart edge reads a texel on either side, so with less than a texel of the chart's own colour beyond the edge it blends in
+ * whatever chart sits next to it in the atlas, and that shows as a step across the seam of a curved mesh.
+ */
+function chartMargin(area, settings) {
+ const expectedResolution = lightmapResolution(area, EXPECTED_UTILIZATION, settings);
+ return Math.min(Math.max(CHART_GAP_TEXELS / expectedResolution, MINIMUM_CHART_MARGIN), MAXIMUM_CHART_MARGIN);
+}
+
/** The world position and facing of every texel a triangle of the geometry covers, or null where nothing covers it. */
function rasterizeTexels(geometry, matrix, resolution) {
const texelCount = resolution * resolution;
@@ -776,48 +792,86 @@ function dilate(radiance, covered, resolution) {
}
}
+/** Groups the covered texels into cubes of one smoothing radius, so a texel finds its neighbours in the world and not only in its own chart. */
+function groupTexelsByCell(covered, positions, cellSize) {
+ const cells = new Map();
+ for (let texel = 0; texel < covered.length; texel++) {
+ if (!covered[texel]) continue;
+ const key = cellKey(Math.floor(positions[texel * 3] / cellSize), Math.floor(positions[texel * 3 + 1] / cellSize), Math.floor(positions[texel * 3 + 2] / cellSize));
+ if (!cells.has(key)) cells.set(key, []);
+ cells.get(key).push(texel);
+ }
+ return cells;
+}
+
+function cellKey(cellX, cellY, cellZ) {
+ return `${cellX},${cellY},${cellZ}`;
+}
+
+/**
+ * The texels of one surface near a texel, with their weights: every covered texel within the smoothing radius in the world
+ * (any chart of the part), whose normal agrees and whose position is within `planeTolerance` of this texel's plane. Walking
+ * the world and not the texture window means a texel on a chart border averages over the same surface patch that the texel
+ * across the seam does, so the two charts agree there. Texels come out in index order, so the same input gives the same bytes.
+ */
+function surfaceNeighbours(center, cells, positions, normals, settings) {
+ const { cellSize, radiusSquared, sigmaSquared, planeTolerance } = settings;
+ const centerX = positions[center * 3];
+ const centerY = positions[center * 3 + 1];
+ const centerZ = positions[center * 3 + 2];
+ const cellX = Math.floor(centerX / cellSize);
+ const cellY = Math.floor(centerY / cellSize);
+ const cellZ = Math.floor(centerZ / cellSize);
+ const found = [];
+ for (let shiftX = -1; shiftX <= 1; shiftX++) {
+ for (let shiftY = -1; shiftY <= 1; shiftY++) {
+ for (let shiftZ = -1; shiftZ <= 1; shiftZ++) {
+ const members = cells.get(cellKey(cellX + shiftX, cellY + shiftY, cellZ + shiftZ));
+ if (!members) continue;
+ for (const neighbour of members) {
+ const deltaX = positions[neighbour * 3] - centerX;
+ const deltaY = positions[neighbour * 3 + 1] - centerY;
+ const deltaZ = positions[neighbour * 3 + 2] - centerZ;
+ const distanceSquared = deltaX * deltaX + deltaY * deltaY + deltaZ * deltaZ;
+ if (distanceSquared > radiusSquared) continue;
+ const facing = normals[center * 3] * normals[neighbour * 3] + normals[center * 3 + 1] * normals[neighbour * 3 + 1] + normals[center * 3 + 2] * normals[neighbour * 3 + 2];
+ if (facing < SMOOTH_MINIMUM_FACING) continue;
+ const offset = deltaX * normals[center * 3] + deltaY * normals[center * 3 + 1] + deltaZ * normals[center * 3 + 2];
+ if (Math.abs(offset) > planeTolerance) continue;
+ found.push({ texel: neighbour, weight: Math.exp(-distanceSquared / (2 * sigmaSquared)) });
+ }
+ }
+ }
+ }
+ return found.sort((left, right) => left.texel - right.texel);
+}
+
/**
* Smooths the gathered values (each an array of three floats per texel) over neighbouring texels of the same surface.
* A texel has only `samples` rays, so its value is noisy, and the noise reads as grain on every wall. A neighbour counts only
- * when it lies on the same plane (its normal agrees and its position is within a texel of this texel's plane), so a corner,
- * a seam between charts and the gap between two parts never leak light into each other. Same input, same bytes.
+ * when it lies on the same plane (its normal agrees and its position is within a texel of this texel's plane), so a corner
+ * and the gap between two parts never leak light into each other. Neighbours are found by world distance, so the two sides
+ * of a chart seam are smoothed over the same patch and a curved mesh keeps no step where its charts meet. Same input, same bytes.
*/
export function smoothTexels(arrays, covered, positions, normals, resolution, spacing) {
- const radius = SMOOTH_RADIUS_TEXELS;
- const sigmaSquared = SMOOTH_SIGMA_TEXELS * SMOOTH_SIGMA_TEXELS;
- const planeTolerance = spacing * SMOOTH_PLANE_TEXELS;
+ const radius = SMOOTH_RADIUS_TEXELS * spacing;
+ const settings = { cellSize: radius, radiusSquared: radius * radius, sigmaSquared: (SMOOTH_SIGMA_TEXELS * spacing) ** 2, planeTolerance: spacing * SMOOTH_PLANE_TEXELS };
+ const cells = groupTexelsByCell(covered, positions, radius);
let sources = arrays;
for (let pass = 0; pass < SMOOTH_PASSES; pass++) {
const results = sources.map(source => new Float32Array(source.length));
- for (let row = 0; row < resolution; row++) {
- for (let column = 0; column < resolution; column++) {
- const center = row * resolution + column;
- if (!covered[center]) continue;
- const totals = sources.map(() => [0, 0, 0]);
- let weightSum = 0;
- for (let rowShift = -radius; rowShift <= radius; rowShift++) {
- const neighbourRow = row + rowShift;
- if (neighbourRow < 0 || neighbourRow >= resolution) continue;
- for (let columnShift = -radius; columnShift <= radius; columnShift++) {
- const neighbourColumn = column + columnShift;
- if (neighbourColumn < 0 || neighbourColumn >= resolution) continue;
- const neighbour = neighbourRow * resolution + neighbourColumn;
- if (!covered[neighbour]) continue;
- const facing = normals[center * 3] * normals[neighbour * 3] + normals[center * 3 + 1] * normals[neighbour * 3 + 1] + normals[center * 3 + 2] * normals[neighbour * 3 + 2];
- if (facing < SMOOTH_MINIMUM_FACING) continue;
- const offset = (positions[neighbour * 3] - positions[center * 3]) * normals[center * 3] + (positions[neighbour * 3 + 1] - positions[center * 3 + 1]) * normals[center * 3 + 1] + (positions[neighbour * 3 + 2] - positions[center * 3 + 2]) * normals[center * 3 + 2];
- if (Math.abs(offset) > planeTolerance) continue;
- const weight = Math.exp(-(rowShift * rowShift + columnShift * columnShift) / (2 * sigmaSquared));
- weightSum += weight;
- sources.forEach((source, which) => {
- for (let channel = 0; channel < 3; channel++) totals[which][channel] += weight * source[neighbour * 3 + channel];
- });
- }
+ for (let center = 0; center < resolution * resolution; center++) {
+ if (!covered[center]) continue;
+ const nearby = surfaceNeighbours(center, cells, positions, normals, settings);
+ let weightSum = 0;
+ for (const { weight } of nearby) weightSum += weight;
+ sources.forEach((source, which) => {
+ for (let channel = 0; channel < 3; channel++) {
+ let total = 0;
+ for (const { texel, weight } of nearby) total += weight * source[texel * 3 + channel];
+ results[which][center * 3 + channel] = total / weightSum;
}
- results.forEach((result, which) => {
- for (let channel = 0; channel < 3; channel++) result[center * 3 + channel] = totals[which][channel] / weightSum;
- });
- }
+ });
}
sources = results;
}
@@ -933,7 +987,10 @@ export function planLightmaps(parts, lighting, options = {}) {
if (!cache.has(part.geometry)) cache.set(part.geometry, new Map());
const unwraps = cache.get(part.geometry);
const key = scaleKey(scale);
- if (!unwraps.has(key)) unwraps.set(key, slot ? unwrapIntoSecondUvSet(part.geometry, scale) : unwrapForLightmap(part.geometry, scale));
+ if (!unwraps.has(key)) {
+ const margin = chartMargin(worldTriangleArea(part), settings);
+ unwraps.set(key, slot ? unwrapIntoSecondUvSet(part.geometry, scale, margin) : unwrapForLightmap(part.geometry, scale, margin));
+ }
entries.push({ part, index, unwrapped: unwraps.get(key), area: worldTriangleArea(part), slot });
});
return { parts, lights, settings, started, entries, skipped };
diff --git a/tests/agent-bake-limits.test.js b/tests/agent-bake-limits.test.js
index c0fc6653..8e8defa3 100644
--- a/tests/agent-bake-limits.test.js
+++ b/tests/agent-bake-limits.test.js
@@ -57,7 +57,11 @@ test('an explicit bake that passes maxSeconds fails cleanly with a lower setting
return realNow.call(performance) + clockReads * 1000;
};
try {
- assert.throws(() => createAgentScene(largeRoom(slow)), error => /maxSeconds/.test(error.message) && /Try samples 128/.test(error.message));
+ // The advised samples follow how many clock reads the bake made, which follows the texel count, so the test pins that the advice is lower than the ask and not one number.
+ assert.throws(() => createAgentScene(largeRoom(slow)), (error) => {
+ const advised = Number(/Try samples (\d+)/.exec(error.message)?.[1]);
+ return /maxSeconds/.test(error.message) && advised >= 1 && advised < slow.samples;
+ });
} finally {
performance.now = realNow;
}
diff --git a/tests/light-bake-seam.test.js b/tests/light-bake-seam.test.js
new file mode 100644
index 00000000..500ed0e9
--- /dev/null
+++ b/tests/light-bake-seam.test.js
@@ -0,0 +1,107 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { box, sphere } from '../src/geometry.js';
+import { bakeLightmaps } from '../src/light-bake.js';
+
+function placed(name, geometry, translation, scale) {
+ const matrix = new Float32Array([scale[0], 0, 0, 0, 0, scale[1], 0, 0, 0, 0, scale[2], 0, translation[0], translation[1], translation[2], 1]);
+ return { name, geometry, matrix, material: { tag: 'surface', color: [0.8, 0.78, 0.72], metallic: 0, roughness: 1, emission: 0 } };
+}
+
+function srgbByteToLinear(byte) {
+ const value = byte / 255;
+ return value <= 0.04045 ? value / 12.92 : ((value + 0.055) / 1.055) ** 2.4;
+}
+
+// The bilinear read of the green channel of a lightmap, as linear light, at a texture coordinate.
+function lightmapReader({ data, width, height }) {
+ const texel = (column, row) => {
+ const clampedColumn = Math.min(Math.max(column, 0), width - 1);
+ const clampedRow = Math.min(Math.max(row, 0), height - 1);
+ return srgbByteToLinear(data[(clampedRow * width + clampedColumn) * 4 + 1]);
+ };
+ return (u, v) => {
+ const x = u * width - 0.5;
+ const y = v * height - 0.5;
+ const column = Math.floor(x);
+ const row = Math.floor(y);
+ const across = x - column;
+ const down = y - row;
+ const upper = texel(column, row) * (1 - across) + texel(column + 1, row) * across;
+ const lower = texel(column, row + 1) * (1 - across) + texel(column + 1, row + 1) * across;
+ return upper * (1 - down) + lower * down;
+ };
+}
+
+function positionKey(positions, vertex) {
+ return [0, 1, 2].map(axis => positions[vertex * 3 + axis].toFixed(5)).join(',');
+}
+
+/**
+ * Where a smooth mesh is cut into charts, one surface point has two lightmap coordinates. Reads the light at both, at five points
+ * along every edge shared by two triangles that sit in different places of the lightmap, and returns the largest and the mean
+ * difference as a share of the brightest texel. Edges that bend sharply are real corners and are left out.
+ */
+function seamJump(part) {
+ const { positions, normals, indices, uvs } = part.geometry;
+ const { image } = part.material.emissiveMap;
+ const readLight = lightmapReader(image);
+ let peak = 1e-6;
+ for (let channel = 1; channel < image.data.length; channel += 4) peak = Math.max(peak, srgbByteToLinear(image.data[channel]));
+ const sides = new Map();
+ for (let triangle = 0; triangle < indices.length / 3; triangle++) {
+ for (let corner = 0; corner < 3; corner++) {
+ const from = indices[triangle * 3 + corner];
+ const to = indices[triangle * 3 + (corner + 1) % 3];
+ const fromKey = positionKey(positions, from);
+ const toKey = positionKey(positions, to);
+ const edgeKey = fromKey < toKey ? `${fromKey}|${toKey}` : `${toKey}|${fromKey}`;
+ if (!sides.has(edgeKey)) sides.set(edgeKey, []);
+ sides.get(edgeKey).push({ from, to, fromKey });
+ }
+ }
+ let largest = 0;
+ let total = 0;
+ let count = 0;
+ for (const [first, second, ...others] of sides.values()) {
+ if (!second || others.length > 0) continue;
+ const bend = [0, 1, 2].reduce((sum, axis) => sum + normals[first.from * 3 + axis] * normals[second.from * 3 + axis], 0);
+ if (bend < 0.9) continue;
+ for (const along of [0.1, 0.3, 0.5, 0.7, 0.9]) {
+ const coordinate = (side) => {
+ const fraction = side.fromKey === first.fromKey ? along : 1 - along;
+ return [0, 1].map(axis => uvs[side.from * 2 + axis] * (1 - fraction) + uvs[side.to * 2 + axis] * fraction);
+ };
+ const [firstU, firstV] = coordinate(first);
+ const [secondU, secondV] = coordinate(second);
+ if (Math.hypot(firstU - secondU, firstV - secondV) < 1e-6) continue;
+ const jump = Math.abs(readLight(firstU, firstV) - readLight(secondU, secondV)) / peak;
+ largest = Math.max(largest, jump);
+ total += jump;
+ count++;
+ }
+ }
+ return { largest, mean: count > 0 ? total / count : 0, count };
+}
+
+function bakedBall() {
+ const parts = [placed('floor', box(0, 1), [0, -0.1, 0], [6, 0.2, 6]), placed('ball', sphere(), [1.2, 0.5, 0.4], [1, 1, 1])];
+ const lighting = { pointLights: [{ position: [0, 2.8, 0], color: [1, 0.85, 0.65], intensity: 18, radius: 0.1, range: 9 }], keyColor: [0, 0, 0] };
+ return bakeLightmaps(parts, lighting, { samples: 128, bounces: 3, texelsPerUnit: 6 }).parts[1];
+}
+
+test('the light of a baked sphere does not step where its charts meet', () => {
+ // The unwrap cut the sphere into six charts packed with less than a texel between them, and the smoothing averaged each chart
+ // edge from its own side only. Bilinear reads at a chart edge then took the neighbouring chart's texels and a biased average,
+ // so the same surface point read up to 70 percent of the brightest texel apart on its two sides.
+ const { largest, mean, count } = seamJump(bakedBall());
+ assert.ok(count > 500, `the sphere has ${count} seam samples`);
+ assert.ok(largest < 0.12, `largest seam jump is ${(largest * 100).toFixed(1)} percent of the peak`);
+ assert.ok(mean < 0.03, `mean seam jump is ${(mean * 100).toFixed(1)} percent of the peak`);
+});
+
+test('baking the same sphere twice gives the same lightmap bytes', () => {
+ const first = bakedBall().material.emissiveMap.image.data;
+ const second = bakedBall().material.emissiveMap.image.data;
+ assert.deepEqual(Buffer.from(first), Buffer.from(second));
+});