Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(modeling): a model thins to a few triangles and keeps its detail as a normal map

yhhc5ce807f3420
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 992752b5f..f8c88839c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,8 @@ published separately; it ships inside the editor.
- **Glyphs can be put on the atlas before they are shown.** The first page of a novel is hundreds of new characters, all landing in one frame. `preloadGlyphs(app, label, text, { budgetMs })` makes them ahead of time, a slice a frame, drawn as that `Text` will draw them, and returns `progress`, `done` and `release()`. On a Snapdragon 8 Elite a book's 2992 characters at 28 px took 145 ms over 25 frames with the worst frame at 9.5 ms, after which the first page's frame went from 49 to 11 ms and no page made a new glyph. Preloaded glyphs hold their atlas pages (4 MB each, a few hundred characters at that size) until `release()`.
- **Voxel GI lights what the camera does not see, and bounces light more than once.** The cache now reads the sun's shadow map back into the world, so a surface the sun lights behind or beside the camera still sends light into the view. Each surface also gives off again the bounce light it received, so a corner or a floor in shade gets light that took two or more bounces to reach it. A ray now brings what it meets at full strength however far it went (it used to fade with distance, 40% at 4 m with 12 cm voxels). Each voxel is weighed by how much of it a surface covers, so a bright patch no longer lights like the whole voxel around it. Together these keep the result close to a path tracer's whatever the voxel size. In a sunlit test room the whole frame carries 0.95 of Cycles' light at the default size and 0.74 at 12 cm voxels; before, it was 0.54 at the default and darker the finer the grid. With the lit patch behind the camera, the walls went from no bounce light to 25–57%. A closed room the sun only lights from outside stays dark. Two inspector controls tune it: **Off-screen Sun Albedo** (the colour the shadow map cannot give, assumed) and **Multi-bounce**. A fine grid no longer drops every voxel: a count truncated by the cache's ageing had kept them below the two samples a ray asks for. The WebGL2 fallback does the same in its own passes: each voxel asks the sun's map whether a sunlit surface stands in it, and traces two rays through last frame's grid for the light it is given. In the test room it carries 1.00 of Cycles' light at the default size (it was 0.56) and 0.80 at 12 cm; with the lit patch behind the camera 0.20 (it was 0). Its voxel pass costs about 1.3 ms more at 720p on a desktop.
+- **An AI building a model can make it light and keep its detail.** A `decimate` step thins a model to a share or a number of triangles, keeping paint and tags on the side they were on, and rigs and shapes are found on what it leaves. `normalBake` writes the surface as it stood before that as a normal map on the thinned one, rigged or not: a 77,000-triangle rock at 800 triangles lights close to the original, and the map bakes in about a second.
+
- **A mesh can have 256 joints, up from 64.** A rig with fingers, a face and hair no longer imports as a statue. A draw sends only the joints it has, so a small rig costs what it did.
- **A mesh can blend 64 morph targets at once, up from 8.** A face rig holds dozens open at a time. A draw pays only for the targets that are live.
diff --git a/desktop b/desktop
index b863b0796..c6b086bd5 160000
--- a/desktop
+++ b/desktop
@@ -1 +1 @@
-Subproject commit b863b0796ec55716c10a4fb913718024837d6103
+Subproject commit c6b086bd5ab414d10f3295ae5a94d84bcf8ae5c6
diff --git a/pipeline/src/modeling/booleans.ts b/pipeline/src/modeling/booleans.ts
index 00d6da945..eaab23863 100644
--- a/pipeline/src/modeling/booleans.ts
+++ b/pipeline/src/modeling/booleans.ts
@@ -7,6 +7,7 @@
* that this kernel's own report exists to catch. Its wasm loads asynchronously and
* a build is synchronous, so a caller prepares it first: see {@link prepareModeling}.
*/
+import { MeshoptSimplifier } from 'meshoptimizer';
import { triangulateFace } from './triangulate';
import { collectEdges, compact, selectFaces, type Face, type FaceSelector, type PolyMesh, type Vec3 } from './polyMesh';
@@ -33,13 +34,14 @@ interface ManifoldEngine {
let engine: ManifoldEngine | null = null;
/** Whether any step of a parsed source needs {@link prepareModeling} first. */
-export const needsPreparing = (steps: readonly { op: string }[]): boolean => steps.some((s) => s.op === 'boolean');
+export const needsPreparing = (steps: readonly { op: string }[]): boolean => steps.some((s) => s.op === 'boolean' || s.op === 'decimate');
/**
- * Loads what a `boolean` step runs on; once is enough. `wasmUrl` is for a bundled host,
+ * Loads what `boolean` and `decimate` steps run on; once is enough. `wasmUrl` is for a bundled host,
* where the module cannot find its own file beside itself.
*/
export async function prepareModeling(wasmUrl?: string): Promise<void> {
+ await MeshoptSimplifier.ready;
if (engine) return;
const { default: Module } = await import('manifold-3d/manifold') as unknown as {
default: (options?: { locateFile?: () => string }) => Promise<ManifoldEngine>;
diff --git a/pipeline/src/modeling/decimate.ts b/pipeline/src/modeling/decimate.ts
new file mode 100644
index 000000000..f396ad156
--- /dev/null
+++ b/pipeline/src/modeling/decimate.ts
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file decimate.ts — the `decimate` step: fewer triangles over the same surface.
+ *
+ * The collapse is meshoptimizer's quadric-error simplifier, as a mesh import's LOD
+ * levels use. What is here is what it is told about a vertex and what the result is
+ * made of: surviving vertices keep their places, so every triangle out is three
+ * corners of the mesh that went in, and takes its paint, tag and shading from the
+ * faces those corners were on. It needs MeshoptSimplifier.ready; prepareModeling
+ * waits for that.
+ */
+import { MeshoptSimplifier } from 'meshoptimizer';
+import { triangulateFace } from './triangulate';
+import { compact, cross, dot, faceArea, faceNormal, normalize, selectFaces, sub, type Face, type FaceSelector, type PolyMesh, type Vec3 } from './polyMesh';
+
+export interface DecimateParams {
+ faces?: FaceSelector;
+ /** Share of the triangles to keep, in (0, 1]. */
+ ratio?: number;
+ /** Triangles to end with, instead of a share. */
+ triangles?: number;
+ /** The most the surface may move, as a share of the mesh's size. It wins over the two above. */
+ error?: number;
+}
+
+const DEFAULT_ERROR = 0.02;
+/** A shading normal bends light and may give a little; a change of paint is a line someone drew. */
+const NORMAL_WEIGHT = 0.5;
+const COLOR_WEIGHT = 1;
+const ATTRIBUTES = 6;
+
+export const triangleCount = (mesh: PolyMesh): number => mesh.faces.reduce((n, f) => n + Math.max(0, f.v.length - 2), 0);
+
+/** Each vertex's normal and paint over @p faces: what a collapse must not smear. */
+function vertexAttributes(mesh: PolyMesh, faces: readonly Face[]): Float32Array {
+ const out = new Float32Array(mesh.positions.length * ATTRIBUTES);
+ const paints = new Uint32Array(mesh.positions.length);
+ for (const face of faces) {
+ const n = faceNormal(mesh, face);
+ const area = faceArea(mesh, face);
+ const color = face.color ?? [1, 1, 1, 1];
+ for (const v of face.v) {
+ for (let k = 0; k < 3; k++) {
+ out[v * ATTRIBUTES + k]! += n[k]! * area;
+ out[v * ATTRIBUTES + 3 + k]! += color[k]!;
+ }
+ paints[v]!++;
+ }
+ }
+ for (let v = 0; v < paints.length; v++) {
+ if (paints[v] === 0) continue;
+ const n = normalize([out[v * ATTRIBUTES]!, out[v * ATTRIBUTES + 1]!, out[v * ATTRIBUTES + 2]!]);
+ for (let k = 0; k < 3; k++) {
+ out[v * ATTRIBUTES + k] = n[k]!;
+ out[v * ATTRIBUTES + 3 + k]! /= paints[v]!;
+ }
+ }
+ return out;
+}
+
+export function decimate(mesh: PolyMesh, p: DecimateParams): PolyMesh {
+ if ((p.ratio === undefined) === (p.triangles === undefined)) throw new Error('give one of `ratio` and `triangles`');
+ if (p.ratio !== undefined && !(p.ratio > 0 && p.ratio <= 1)) throw new Error(`ratio ${p.ratio} must be in (0, 1]`);
+ if (p.triangles !== undefined && p.triangles < 1) throw new Error(`triangles ${p.triangles} must be at least 1`);
+ if (p.error !== undefined && !(p.error > 0)) throw new Error(`error ${p.error} must be positive`);
+
+ const selector = p.faces ?? 'all';
+ const hit = selectFaces(mesh, selector);
+ if (hit.length === 0) throw new Error(`\`faces\` ${JSON.stringify(selector)} matched no face`);
+ const chosen = new Set(hit);
+ const faces = hit.map((f) => mesh.faces[f]!);
+ const indices = Uint32Array.from(faces.flatMap((face) => triangulateFace(mesh, face).flat()));
+ const count = indices.length / 3;
+ const target = p.triangles ?? Math.round(count * p.ratio!);
+ if (target >= count) return mesh;
+
+ // The seam with faces left alone stays where it is, or the two would part.
+ const lock = new Uint8Array(mesh.positions.length);
+ mesh.faces.forEach((face, f) => {
+ if (!chosen.has(f)) for (const v of face.v) lock[v] = 1;
+ });
+ const positions = Float32Array.from(mesh.positions.flat());
+ const [kept] = MeshoptSimplifier.simplifyWithAttributes(
+ indices, positions, 3, vertexAttributes(mesh, faces), ATTRIBUTES,
+ [NORMAL_WEIGHT, NORMAL_WEIGHT, NORMAL_WEIGHT, COLOR_WEIGHT, COLOR_WEIGHT, COLOR_WEIGHT],
+ // Not RegularizeLight: under an error limit it stops a flat surface from collapsing at all.
+ lock, target * 3, p.error ?? DEFAULT_ERROR, ['LockBorder'],
+ );
+ if (kept.length === 0) throw new Error(`nothing is left at ${target} triangles; raise \`ratio\` or \`triangles\``);
+
+ const around = new Map<number, Face[]>();
+ for (const face of faces) {
+ for (const v of face.v) {
+ const at = around.get(v);
+ if (at) at.push(face); else around.set(v, [face]);
+ }
+ }
+ const made: Face[] = [];
+ for (let i = 0; i < kept.length; i += 3) {
+ const [a, b, c] = [kept[i]!, kept[i + 1]!, kept[i + 2]!];
+ const [pa, pb, pc]: Vec3[] = [mesh.positions[a]!, mesh.positions[b]!, mesh.positions[c]!];
+ const normal = normalize(cross(sub(pb, pa), sub(pc, pa)));
+ // The old face this triangle lies on the most: the one holding most of its corners, then the one facing its way.
+ const shared = new Map<Face, number>();
+ for (const v of [a, b, c]) for (const face of around.get(v)!) shared.set(face, (shared.get(face) ?? 0) + 1);
+ let from: Face | null = null;
+ let best = -Infinity;
+ for (const [face, corners] of shared) {
+ const score = corners + dot(faceNormal(mesh, face), normal);
+ if (score > best) { best = score; from = face; }
+ }
+ made.push({ ...from!, v: [a, b, c] });
+ }
+ return compact({ positions: mesh.positions, faces: [...mesh.faces.filter((_, f) => !chosen.has(f)), ...made] });
+}
diff --git a/pipeline/src/modeling/index.ts b/pipeline/src/modeling/index.ts
index e0ccfbb63..a9fbe7d4d 100644
--- a/pipeline/src/modeling/index.ts
+++ b/pipeline/src/modeling/index.ts
@@ -6,11 +6,12 @@
*/
export {
MODEL_OPS, MODEL_SOURCE_EXT, MODEL_SOURCE_VERSION, ModelBuildError, applyModelEdits, buildPolyMesh, describeModelOps, validateModelSource,
- type ModelEdit, type ModelSource, type ModelStep, type StepReport,
+ type ModelEdit, type ModelSource, type NormalBakeSource, type ModelStep, type StepReport,
} from './modelSource';
export { organic, ORGANIC_DEFAULT_RESOLUTION, ORGANIC_MAX_RESOLUTION, type OrganicParams, type OrganicPart } from './organic';
export { needsPreparing, prepareModeling } from './booleans';
export { bakeModel, type BakedModel, type ModelReport } from './modelBake';
+export { bakeNormalMap, type NormalBakeReport, type NormalMap } from './normalBake';
export type { FaceSelector } from './polyMesh';
export { DEFAULT_MODEL_VIEWS, MODEL_VIEWS, renderModelSheet, renderRigSheet, type ModelSheet, type ModelViewName, type RigViewOptions } from './modelViews';
export { controllerOf, jointPositions, morphedPositions, posedPositions, prefabOf, timelineOf, type AnimationSource, type ControllerSource, type PoseSolve, type PreviewGround, type RigSource, type ShapeSource } from './rig';
diff --git a/pipeline/src/modeling/modelBake.ts b/pipeline/src/modeling/modelBake.ts
index 9ec01680f..643055bab 100644
--- a/pipeline/src/modeling/modelBake.ts
+++ b/pipeline/src/modeling/modelBake.ts
@@ -9,9 +9,11 @@
* where the same fault in a lit picture reads as a slightly odd shadow.
*/
import { MeshChannel, MeshChannelType, packChannels, unwrapLightmapUV, type MeshData, type MeshMorphTargets } from 'esengine';
+import { generateTangents } from '../assets/meshTangents';
import { triangulateFace } from './triangulate';
import { faceUvs } from './modelUv';
import { adjustWeights, bindMatrices, computeWeights, resolveJoints, shapeDeltas, type ControllerIk, type RigGoal, type RigJoint, type SkinWeights } from './rig';
+import type { NormalBakeReport } from './normalBake';
import { buildPolyMesh, validateModelSource, type ModelSource, type StepReport } from './modelSource';
import {
boundsOf, collectEdges, cross, dot, faceArea, faceNormal, length, normalize, sub,
@@ -19,6 +21,7 @@ import {
} from './polyMesh';
const DEFAULT_SMOOTH_ANGLE = 30;
+const DEFAULT_NORMAL_SIZE = 1024;
export interface ModelReport {
vertices: number;
@@ -42,6 +45,8 @@ export interface ModelReport {
steps: StepReport[];
/** For a rigged model: each joint, and how many vertices it moves the most. */
joints?: Record<string, number>;
+ /** What a normal map carries, once one has been baked. */
+ normalBake?: NormalBakeReport;
/** The findings above that are worth stopping for, as sentences. */
warnings: string[];
}
@@ -56,6 +61,8 @@ export interface BakedModel {
shapes?: { names: string[]; deltas: Float32Array[] };
/** One triangle per entry: its three positions' indices, and the face it came from. */
triangles: { v: [number, number, number]; face: number }[];
+ /** Present where a source has a normalBake: what bakeNormalMap carries, onto a mesh already unwrapped and given tangents for it. */
+ detail?: { high: PolyMesh; size: number; distance?: number; smoothAngle: number };
}
// ---- Report -----------------------------------------------------------------------
@@ -90,7 +97,7 @@ function signedVolume(mesh: PolyMesh, faces: readonly number[], tris: readonly [
// ---- Bake -------------------------------------------------------------------------
/** Per-corner normals: each face's, averaged with those of the faces round that vertex within @p smoothAngle. */
-function cornerNormals(mesh: PolyMesh, smoothAngle: number): Vec3[][] {
+export function cornerNormals(mesh: PolyMesh, smoothAngle: number): Vec3[][] {
const normals = mesh.faces.map((f) => faceNormal(mesh, f));
const cosine = Math.cos((smoothAngle * Math.PI) / 180);
const around = new Map<number, number[]>();
@@ -224,10 +231,33 @@ function copyLightmapUvToBase(mesh: MeshData): void {
}
}
+/** The mesh with the frame a normal map is read in, from the UVs and normals as stored. */
+function withTangents(mesh: MeshData): MeshData {
+ const view = new DataView(mesh.vertices.buffer, mesh.vertices.byteOffset, mesh.vertices.byteLength);
+ const read = (semantic: number, components: number): Float32Array => {
+ const { offset } = mesh.channels.find((c) => c.semantic === semantic)!;
+ const out = new Float32Array(mesh.vertexCount * components);
+ for (let v = 0; v < mesh.vertexCount; v++) for (let k = 0; k < components; k++) out[v * components + k] = view.getFloat32(v * mesh.vertexStride + offset + k * 4, true);
+ return out;
+ };
+ const tangents = generateTangents(read(MeshChannel.Position, 3), read(MeshChannel.Normal, 3), read(MeshChannel.TexCoord0, 2), mesh.indices);
+ const stride = mesh.vertexStride + 16;
+ const vertices = new Uint8Array(mesh.vertexCount * stride);
+ const out = new DataView(vertices.buffer);
+ for (let v = 0; v < mesh.vertexCount; v++) {
+ vertices.set(mesh.vertices.subarray(v * mesh.vertexStride, (v + 1) * mesh.vertexStride), v * stride);
+ for (let k = 0; k < 4; k++) out.setFloat32(v * stride + mesh.vertexStride + k * 4, tangents[v * 4 + k]!, true);
+ }
+ return {
+ ...mesh, vertices, vertexStride: stride,
+ channels: [...mesh.channels, { semantic: MeshChannel.Tangent, components: 4, type: MeshChannelType.Float32, offset: mesh.vertexStride }],
+ };
+}
+
/** Validates, builds and bakes @p raw. Throws a ModelBuildError naming the step that failed; writes nothing, so a failure leaves nothing half made. */
export function bakeModel(raw: unknown): BakedModel {
const source = validateModelSource(raw);
- const { mesh, steps } = buildPolyMesh(source);
+ const { mesh, steps, high } = buildPolyMesh(source);
if (mesh.faces.length === 0) throw new Error('the steps leave no faces; a model starts with a box, cylinder, cone, sphere, plane or prism');
const triangles = mesh.faces.map((f) => triangulateFace(mesh, f));
@@ -237,10 +267,13 @@ export function bakeModel(raw: unknown): BakedModel {
rig = { joints, skin: adjustWeights(mesh, joints, computeWeights(mesh, joints), source.rig.adjust ?? []), ik: source.controller?.ik ?? [], goals: source.rig.goals ?? [] };
}
const shapes = source.shapes ? { names: source.shapes.map((s) => s.name), deltas: source.shapes.map((s) => shapeDeltas(mesh, s)) } : undefined;
- // A mesh that moves takes no lightmap: a bake holds still.
+ const bake = source.normalBake;
+ const size = bake?.size ?? DEFAULT_NORMAL_SIZE;
+ // A mesh that moves takes no lightmap: a bake holds still. A normal map moves with it, and needs texels of its own all the same.
const baked = toMeshData(mesh, triangles, source, rig, shapes);
- const unwrapped = rig || shapes ? baked : unwrapLightmapUV(baked).mesh;
- if (source.uvAtlas) copyLightmapUvToBase(unwrapped);
+ let unwrapped = bake ? unwrapLightmapUV(baked, { atlasSize: size, paddingTexels: 4 }).mesh : rig || shapes ? baked : unwrapLightmapUV(baked).mesh;
+ if (source.uvAtlas || bake) copyLightmapUvToBase(unwrapped);
+ if (bake) unwrapped = withTangents(unwrapped);
const edges = [...collectEdges(mesh).values()];
const openEdges = edges.filter((e) => e.faces.length === 1).length;
@@ -281,6 +314,13 @@ export function bakeModel(raw: unknown): BakedModel {
}
// The engine's delta texture is 2048 square, the most WebGL2 guarantees; past it the mesh draws, but never changes shape.
+ for (const s of steps) {
+ const step = source.steps[s.index]!;
+ if (s.op !== 'decimate' || step.faces !== undefined) continue;
+ const target = (step.triangles as number | undefined) ?? Math.round(s.trianglesBefore! * (step.ratio as number));
+ if (s.trianglesAfter! > target * 1.1) warnings.push(`step ${s.index} (decimate) stopped at ${s.trianglesAfter} triangles of the ${target} asked: the surface would have moved more than \`error\` allows. Raise it to go further`);
+ }
+
if (shapes && shapes.names.length * unwrapped.vertexCount * 2 > 2048 * 2048) {
warnings.push(`${shapes.names.length} shapes over ${unwrapped.vertexCount} vertices is more than the engine holds (${2048 * 2048} deltas, two each): it would draw but never change shape. Lower the resolution or use fewer shapes`);
}
@@ -291,6 +331,7 @@ export function bakeModel(raw: unknown): BakedModel {
data: unwrapped,
...(rig ? { rig } : {}),
...(shapes ? { shapes } : {}),
+ ...(bake && high ? { detail: { high, size, ...(bake.distance !== undefined ? { distance: bake.distance } : {}), smoothAngle: source.smoothAngle ?? DEFAULT_SMOOTH_ANGLE } } : {}),
triangles: triangles.flatMap((tris, face) => tris.map((v) => ({ v, face }))),
report: {
vertices: unwrapped.vertexCount,
diff --git a/pipeline/src/modeling/modelSource.ts b/pipeline/src/modeling/modelSource.ts
index fcee7c803..66e18cd25 100644
--- a/pipeline/src/modeling/modelSource.ts
+++ b/pipeline/src/modeling/modelSource.ts
@@ -18,6 +18,7 @@ import { booleanOp } from './booleans';
import { uvStep } from './modelUv';
import { lathe, loft } from './surfaces';
import { subdivide } from './subdivide';
+import { decimate, triangleCount } from './decimate';
import { checkAnimations, checkController, checkRig, checkShapes, type AnimationSource, type ControllerSource, type RigSource, type ShapeSource } from './rig';
import { compact, type Face, type FaceSelector, type PolyMesh, type Vec3 } from './polyMesh';
@@ -39,6 +40,8 @@ export interface ModelSource {
uvScale?: number;
/** Lay the base UV out so no two triangles share texels: for a texture painted for this one model, where tiling is not wanted. */
uvAtlas?: boolean;
+ /** Carries a denser state of the steps onto the final surface as a normal map. It lays the base UV out as uvAtlas does. */
+ normalBake?: NormalBakeSource;
/** Joints: the mesh is skinned to them, with weights found from the surface. */
rig?: RigSource;
/** Clips moving the rig's joints, each written as an `.estimeline`. */
@@ -50,6 +53,15 @@ export interface ModelSource {
steps: ModelStep[];
}
+export interface NormalBakeSource {
+ /** The step whose result holds the detail. Default: what the last `decimate` started from. */
+ from?: string;
+ /** Pixels along a side of the map, a power of two from 64 to 4096. Default 1024. */
+ size?: number;
+ /** How far from the final surface to look for the dense one, in model units. Default a twentieth of the diagonal. */
+ distance?: number;
+}
+
type Kind = 'number' | 'integer' | 'boolean' | 'string' | 'vec3' | 'number|vec3' | 'number|vec2' | 'number|numbers'
| 'color' | 'points' | 'faces' | 'edges' | 'axis' | 'pivot' | 'parts' | 'noise' | 'sections' | 'section' | 'vec3s' | 'numbers' | 'operation' | 'uvmode' | 'number|vec2only' | 'vec2';
@@ -83,6 +95,7 @@ export const MODEL_OPS: Record<string, OpSpec> = {
slice: { params: { faces: 'faces', axis: 'axis', at: 'number|numbers' }, required: ['axis', 'at'], run: ops.slice as OpSpec['run'], says: 'cuts faces across planes, making the seams a window sits between' },
boolean: { params: { operation: 'operation', with: 'faces', from: 'faces', keep: 'boolean', tag: 'string', color: 'color' }, required: ['operation', 'with'], run: booleanOp as unknown as OpSpec['run'], says: 'cuts or joins closed solids: subtract `with` from `from` makes a real hole' },
subdivide: { params: { faces: 'faces', levels: 'integer', crease: 'number' }, required: [], run: subdivide as unknown as OpSpec['run'], says: 'rounds whole shells into smooth surfaces, keeping edges sharper than `crease` degrees' },
+ decimate: { params: { faces: 'faces', ratio: 'number', triangles: 'integer', error: 'number' }, required: [], run: decimate as unknown as OpSpec['run'], says: 'leaves fewer triangles over the same surface: `ratio` to keep or `triangles` to end with, stopping sooner if the surface would move more than `error` (default 0.02) of its size' },
uv: { params: { faces: 'faces', mode: 'uvmode', axis: 'string', at: 'vec3', scale: 'number|vec2only', repeat: 'number|vec2only', offset: 'vec2', rotate: 'number' }, required: ['mode'], run: uvStep as unknown as OpSpec['run'], says: 'sets how faces take a texture: box, planar, cylinder or sphere' },
delete: { params: { faces: 'faces' }, required: ['faces'], run: ops.deleteFaces as OpSpec['run'], says: 'removes faces, leaving a hole' },
paint: { params: { faces: 'faces', color: 'color' }, required: ['faces', 'color'], run: ops.paint as OpSpec['run'], says: 'sets vertex colour: linear [r, g, b] in 0..1, multiplied into the material' },
@@ -250,6 +263,23 @@ export class ModelBuildError extends Error {
}
}
+function checkNormalBake(v: unknown, steps: readonly Partial<ModelStep>[]): string | null {
+ if (typeof v !== 'object' || v === null || Array.isArray(v)) return 'must be { from?, size?, distance? }';
+ const bake = v as Record<string, unknown>;
+ for (const key of Object.keys(bake)) if (!['from', 'size', 'distance'].includes(key)) return `has an unknown field "${key}"; it takes from, size, distance`;
+ const { size, distance, from } = bake;
+ if (size !== undefined && !(Number.isInteger(size) && (size as number) >= 64 && (size as number) <= 4096 && ((size as number) & ((size as number) - 1)) === 0)) return 'size must be a power of two, 64..4096';
+ if (distance !== undefined && !(isNum(distance) && distance > 0)) return 'distance must be a positive number';
+ if (from !== undefined) {
+ if (typeof from !== 'string') return 'from must be a step id';
+ if (!steps.some((s) => s?.id === from)) return `from "${from}" is no step's id`;
+ return null;
+ }
+ const last = steps.map((s) => s?.op).lastIndexOf('decimate');
+ if (last < 0) return 'needs a surface to carry: a decimate step, or `from` naming a step';
+ return last === 0 ? 'has nothing to carry: the decimate is the first step' : null;
+}
+
const stepLabel = (step: Partial<ModelStep> | null, index: number): string =>
`step ${index}${step?.id ? ` "${step.id}"` : ''}${step?.op ? ` (${step.op})` : ''}`;
@@ -258,8 +288,8 @@ export function validateModelSource(raw: unknown): ModelSource {
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) throw new ModelBuildError('a model source must be an object with `steps`', null, null, null);
const source = raw as Record<string, unknown>;
for (const key of Object.keys(source)) {
- if (!['version', 'smoothAngle', 'uvScale', 'uvAtlas', 'rig', 'shapes', 'animations', 'controller', 'steps'].includes(key)) {
- throw new ModelBuildError(`unknown field "${key}"; a model source has version, smoothAngle, uvScale, uvAtlas, rig, shapes, animations, controller, steps`, null, null, null);
+ if (!['version', 'smoothAngle', 'uvScale', 'uvAtlas', 'normalBake', 'rig', 'shapes', 'animations', 'controller', 'steps'].includes(key)) {
+ throw new ModelBuildError(`unknown field "${key}"; a model source has version, smoothAngle, uvScale, uvAtlas, normalBake, rig, shapes, animations, controller, steps`, null, null, null);
}
}
if (source.version !== undefined && source.version !== MODEL_SOURCE_VERSION) {
@@ -293,6 +323,10 @@ export function validateModelSource(raw: unknown): ModelSource {
if (bad) throw new ModelBuildError(bad, null, null, null);
}
if (!Array.isArray(source.steps)) throw new ModelBuildError('`steps` must be an array', null, null, null);
+ if (source.normalBake !== undefined) {
+ const bad = checkNormalBake(source.normalBake, source.steps as Partial<ModelStep>[]);
+ if (bad) throw new ModelBuildError(`normalBake ${bad}`, null, null, null);
+ }
const ids = new Set<string>();
source.steps.forEach((step: Partial<ModelStep> | null, index: number) => {
const refuse = (what: string): never => {
@@ -324,27 +358,38 @@ export interface StepReport {
op: string;
/** Faces in the mesh after it ran. */
faces: number;
+ /** A decimate: triangles it was given, and triangles it left. */
+ trianglesBefore?: number;
+ trianglesAfter?: number;
}
export interface ModelBuild {
mesh: PolyMesh;
steps: StepReport[];
+ /** The surface `normalBake` carries, as the steps stood then. */
+ high?: PolyMesh;
}
/** Runs the steps in order. A step that cannot run throws a {@link ModelBuildError} naming it. */
export function buildPolyMesh(source: ModelSource): ModelBuild {
let mesh: PolyMesh = { positions: [], faces: [] };
const steps: StepReport[] = [];
+ const bake = source.normalBake;
+ // After this step, the surface is the one to carry: the one named, or the one before the last decimate.
+ const highAfter = !bake ? -2 : bake.from !== undefined ? source.steps.findIndex((s) => s.id === bake.from) : source.steps.map((s) => s.op).lastIndexOf('decimate') - 1;
+ let high: PolyMesh | undefined;
source.steps.forEach((step, index) => {
const { id, op, ...params } = step;
+ const before = op === 'decimate' ? triangleCount(mesh) : 0;
try {
mesh = MODEL_OPS[op]!.run(mesh, params as never);
} catch (e) {
throw new ModelBuildError(`${stepLabel(step, index)}: ${e instanceof Error ? e.message : String(e)}`, index, id ?? null, op);
}
- steps.push({ index, ...(id ? { id } : {}), op, faces: mesh.faces.length });
+ steps.push({ index, ...(id ? { id } : {}), op, faces: mesh.faces.length, ...(op === 'decimate' ? { trianglesBefore: before, trianglesAfter: triangleCount(mesh) } : {}) });
+ if (index === highAfter) high = compact(mesh);
});
- return { mesh: compact(mesh), steps };
+ return { mesh: compact(mesh), steps, ...(high ? { high } : {}) };
}
// ---- Edits: changing a source without restating it ------------------------------
@@ -353,7 +398,7 @@ export type ModelEdit =
| { op: 'set'; step: string | number; fields: Record<string, unknown> }
| { op: 'insert'; step: ModelStep; before?: string | number; after?: string | number }
| { op: 'remove'; step: string | number }
- | { op: 'settings'; fields: Pick<ModelSource, 'smoothAngle' | 'uvScale' | 'uvAtlas' | 'rig' | 'shapes' | 'animations' | 'controller'> };
+ | { op: 'settings'; fields: Pick<ModelSource, 'smoothAngle' | 'uvScale' | 'uvAtlas' | 'normalBake' | 'rig' | 'shapes' | 'animations' | 'controller'> };
function indexOfStep(source: ModelSource, ref: string | number): number {
const i = typeof ref === 'number' ? ref : source.steps.findIndex((s) => s.id === ref);
@@ -389,7 +434,11 @@ export function applyModelEdits(source: ModelSource, edits: readonly ModelEdit[]
next.steps.splice(indexOfStep(next, edit.step), 1);
break;
case 'settings':
- Object.assign(next, edit.fields);
+ // A null removes, as it does in `set`.
+ for (const [key, value] of Object.entries(edit.fields ?? {})) {
+ if (value === null) delete (next as unknown as Record<string, unknown>)[key];
+ else (next as unknown as Record<string, unknown>)[key] = value;
+ }
break;
default:
throw new ModelBuildError(`unknown edit ${JSON.stringify((edit as { op?: unknown })?.op)}; there are set, insert, remove, settings`, null, null, null);
diff --git a/pipeline/src/modeling/normalBake.ts b/pipeline/src/modeling/normalBake.ts
new file mode 100644
index 000000000..64e554e54
--- /dev/null
+++ b/pipeline/src/modeling/normalBake.ts
@@ -0,0 +1,332 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file normalBake.ts — the detail of a dense surface, as a normal map on a light one.
+ *
+ * From each texel's point on the light surface a line runs along its normal, both
+ * ways, to the nearest place on the dense surface facing the same way; the dense
+ * surface's normal there is written in the frame the mesh carries. That frame is
+ * the one MESH_TANGENTS reads the map in (perturbNormal in ShaderParser.cpp): the
+ * tangent made square to the interpolated normal, and the bitangent as
+ * cross(N, T) * w. The two must stay the same arithmetic, or the map bends light
+ * the wrong way with nothing to say so.
+ *
+ * A separate call from bakeModel because it costs seconds, and only a write wants it.
+ */
+import { MeshChannel, MeshChannelType, type MeshData } from 'esengine';
+import { cornerNormals, type BakedModel } from './modelBake';
+import { triangulateFace } from './triangulate';
+import { boundsOf, type PolyMesh } from './polyMesh';
+
+/** How far the line runs when the source does not say, as a share of the model's diagonal. */
+const DEFAULT_DISTANCE = 0.05;
+/** Texels past each chart's edge that take its values: what bilinear reads and the small mips average there. */
+const GUTTER = 16;
+const LEAF = 4;
+/** Above this share of texels finding no surface, the map is probably missing what it was made for. */
+const MISSED_WARNING = 0.05;
+
+export interface NormalBakeReport {
+ size: number;
+ /** Triangles on the dense surface and on the light one. */
+ from: number;
+ to: number;
+ /** Share of the image the mesh's charts cover. */
+ coverage: number;
+ /** Share of those texels that found no surface within `distance`, and were left flat. */
+ missed: number;
+ /** How far the line ran. */
+ distance: number;
+ /** Mean bend the map adds, in degrees: near 0 means it says nothing. */
+ meanBend: number;
+}
+
+export interface NormalMap {
+ size: number;
+ /** RGBA, row by row from the top: what a PNG holds. Data, not colour: it must be read without sRGB decoding. */
+ rgba: Uint8Array;
+ report: NormalBakeReport;
+ warnings: string[];
+}
+
+/** A bounding-volume hierarchy over triangles, in flat arrays: a dense model is several hundred thousand of them. */
+class TriangleTree {
+ private readonly order: Int32Array;
+ private readonly boxes: Float64Array;
+ /** Per node: a leaf's first triangle in `order` and its count, or the right child and 0. */
+ private readonly links: Int32Array;
+ private nodes = 0;
+ private readonly stack = new Int32Array(128);
+
+ constructor(private readonly corners: Float64Array) {
+ const count = corners.length / 9;
+ this.order = Int32Array.from({ length: count }, (_, i) => i);
+ this.boxes = new Float64Array(count * 2 * 6 + 6);
+ this.links = new Int32Array(count * 2 * 2 + 2);
+ const centers = new Float64Array(count * 3);
+ for (let t = 0; t < count; t++) {
+ for (let k = 0; k < 3; k++) centers[t * 3 + k] = (corners[t * 9 + k]! + corners[t * 9 + 3 + k]! + corners[t * 9 + 6 + k]!) / 3;
+ }
+ this.build(0, count, centers);
+ }
+
+ private build(first: number, count: number, centers: Float64Array): number {
+ const node = this.nodes++;
+ const [boxes, order, corners] = [this.boxes, this.order, this.corners];
+ const lo = [Infinity, Infinity, Infinity], hi = [-Infinity, -Infinity, -Infinity];
+ const cLo = [Infinity, Infinity, Infinity], cHi = [-Infinity, -Infinity, -Infinity];
+ for (let i = first; i < first + count; i++) {
+ const t = order[i]!;
+ for (let k = 0; k < 3; k++) {
+ for (let c = 0; c < 3; c++) {
+ const x = corners[t * 9 + c * 3 + k]!;
+ if (x < lo[k]!) lo[k] = x;
+ if (x > hi[k]!) hi[k] = x;
+ }
+ const m = centers[t * 3 + k]!;
+ if (m < cLo[k]!) cLo[k] = m;
+ if (m > cHi[k]!) cHi[k] = m;
+ }
+ }
+ for (let k = 0; k < 3; k++) { boxes[node * 6 + k] = lo[k]!; boxes[node * 6 + 3 + k] = hi[k]!; }
+ const axis = cHi[0]! - cLo[0]! >= cHi[1]! - cLo[1]! && cHi[0]! - cLo[0]! >= cHi[2]! - cLo[2]! ? 0 : cHi[1]! - cLo[1]! >= cHi[2]! - cLo[2]! ? 1 : 2;
+ if (count <= LEAF || cHi[axis]! - cLo[axis]! <= 0) {
+ this.links[node * 2] = first;
+ this.links[node * 2 + 1] = count;
+ return node;
+ }
+ const half = (cLo[axis]! + cHi[axis]!) / 2;
+ let split = first;
+ for (let i = first; i < first + count; i++) {
+ if (centers[order[i]! * 3 + axis]! < half) {
+ const swap = order[i]!;
+ order[i] = order[split]!;
+ order[split] = swap;
+ split++;
+ }
+ }
+ this.build(first, split - first, centers);
+ this.links[node * 2] = this.build(split, first + count - split, centers);
+ this.links[node * 2 + 1] = 0;
+ return node;
+ }
+
+ /**
+ * The hit on the line through @p o along @p d nearest to @p o, within @p reach either way, on a triangle
+ * facing along @p d. Answers its triangle, or -1; the place on it goes in @p out as [u, v] for its 2nd and 3rd corners.
+ */
+ nearest(ox: number, oy: number, oz: number, dx: number, dy: number, dz: number, reach: number, out: Float64Array): number {
+ const [boxes, links, order, c, stack] = [this.boxes, this.links, this.order, this.corners, this.stack];
+ let best = reach, found = -1, top = 0;
+ stack[top++] = 0;
+ while (top > 0) {
+ const node = stack[--top]!;
+ const at = node * 6;
+ let near = -best, far = best;
+ let inside = true;
+ for (let k = 0; k < 3 && inside; k++) {
+ const d = k === 0 ? dx : k === 1 ? dy : dz, o = k === 0 ? ox : k === 1 ? oy : oz;
+ const lo = boxes[at + k]!, hi = boxes[at + 3 + k]!;
+ if (d === 0) { inside = o >= lo && o <= hi; continue; }
+ let t0 = (lo - o) / d, t1 = (hi - o) / d;
+ if (t0 > t1) { const s = t0; t0 = t1; t1 = s; }
+ if (t0 > near) near = t0;
+ if (t1 < far) far = t1;
+ inside = near <= far;
+ }
+ if (!inside) continue;
+ const count = links[node * 2 + 1]!;
+ if (count === 0) {
+ stack[top++] = links[node * 2]!;
+ stack[top++] = node + 1;
+ continue;
+ }
+ const first = links[node * 2]!;
+ for (let i = first; i < first + count; i++) {
+ const t = order[i]!, p = t * 9;
+ const ax = c[p]!, ay = c[p + 1]!, az = c[p + 2]!;
+ const e1x = c[p + 3]! - ax, e1y = c[p + 4]! - ay, e1z = c[p + 5]! - az;
+ const e2x = c[p + 6]! - ax, e2y = c[p + 7]! - ay, e2z = c[p + 8]! - az;
+ const px = dy * e2z - dz * e2y, py = dz * e2x - dx * e2z, pz = dx * e2y - dy * e2x;
+ const det = e1x * px + e1y * py + e1z * pz;
+ // Negative is the line running along the triangle's outward normal: the inner wall of a thin part is not this surface.
+ if (det > -1e-14) continue;
+ const inv = 1 / det;
+ const sx = ox - ax, sy = oy - ay, sz = oz - az;
+ const u = (sx * px + sy * py + sz * pz) * inv;
+ if (u < 0 || u > 1) continue;
+ const qx = sy * e1z - sz * e1y, qy = sz * e1x - sx * e1z, qz = sx * e1y - sy * e1x;
+ const v = (dx * qx + dy * qy + dz * qz) * inv;
+ if (v < 0 || u + v > 1) continue;
+ const along = (e2x * qx + e2y * qy + e2z * qz) * inv;
+ if (Math.abs(along) < best) {
+ best = Math.abs(along);
+ found = t;
+ out[0] = u;
+ out[1] = v;
+ }
+ }
+ }
+ return found;
+ }
+}
+
+/** Reads the channels the bake needs off the baked mesh, wherever the layout put them. */
+function readFloats(mesh: MeshData, semantic: number, components: number): Float32Array {
+ const channel = mesh.channels.find((c) => c.semantic === semantic && c.type === MeshChannelType.Float32);
+ if (!channel) throw new Error(`the mesh has no float channel ${semantic}`);
+ const view = new DataView(mesh.vertices.buffer, mesh.vertices.byteOffset, mesh.vertices.byteLength);
+ const out = new Float32Array(mesh.vertexCount * components);
+ for (let v = 0; v < mesh.vertexCount; v++) {
+ for (let k = 0; k < components; k++) out[v * components + k] = view.getFloat32(v * mesh.vertexStride + channel.offset + k * 4, true);
+ }
+ return out;
+}
+
+/** The dense surface as flat triangles, and the shading normal at each of their corners. */
+function denseSurface(high: PolyMesh, smoothAngle: number): { corners: Float64Array; normals: Float64Array } {
+ const normalsOf = cornerNormals(high, smoothAngle);
+ const positions: number[] = [], normals: number[] = [];
+ high.faces.forEach((face, f) => {
+ for (const tri of triangulateFace(high, face)) {
+ for (const v of tri) {
+ positions.push(...high.positions[v]!);
+ normals.push(...normalsOf[f]![face.v.indexOf(v)]!);
+ }
+ }
+ });
+ return { corners: Float64Array.from(positions), normals: Float64Array.from(normals) };
+}
+
+/** Bakes @p model's `normalBake`. Throws if it has none. */
+export function bakeNormalMap(model: BakedModel): NormalMap {
+ const detail = model.detail;
+ if (!detail) throw new Error('this model has no normalBake');
+ const { size, high, smoothAngle } = detail;
+ const { min, max } = boundsOf(high);
+ const distance = detail.distance ?? Math.hypot(max[0] - min[0], max[1] - min[1], max[2] - min[2]) * DEFAULT_DISTANCE;
+ const { corners, normals } = denseSurface(high, smoothAngle);
+ const tree = new TriangleTree(corners);
+
+ const mesh = model.data;
+ const P = readFloats(mesh, MeshChannel.Position, 3);
+ const N = readFloats(mesh, MeshChannel.Normal, 3);
+ const T = readFloats(mesh, MeshChannel.Tangent, 4);
+ const UV = readFloats(mesh, MeshChannel.TexCoord0, 2);
+
+ const rgba = new Uint8Array(size * size * 4);
+ for (let i = 0; i < size * size; i++) { rgba[i * 4] = 128; rgba[i * 4 + 1] = 128; rgba[i * 4 + 2] = 255; rgba[i * 4 + 3] = 255; }
+ const covered = new Uint8Array(size * size);
+ const place = new Float64Array(2);
+ let hits = 0, missed = 0, bend = 0;
+
+ for (let t = 0; t + 2 < mesh.indices.length; t += 3) {
+ const [a, b, c] = [mesh.indices[t]!, mesh.indices[t + 1]!, mesh.indices[t + 2]!];
+ // A texture's rows run down from the top and the engine's v runs up from the bottom.
+ const ax = UV[a * 2]! * size, ay = (1 - UV[a * 2 + 1]!) * size;
+ const bx = UV[b * 2]! * size, by = (1 - UV[b * 2 + 1]!) * size;
+ const cx = UV[c * 2]! * size, cy = (1 - UV[c * 2 + 1]!) * size;
+ const area = (bx - ax) * (cy - ay) - (cx - ax) * (by - ay);
+ if (Math.abs(area) < 1e-12) continue;
+ const x0 = Math.max(0, Math.floor(Math.min(ax, bx, cx))), x1 = Math.min(size - 1, Math.ceil(Math.max(ax, bx, cx)));
+ const y0 = Math.max(0, Math.floor(Math.min(ay, by, cy))), y1 = Math.min(size - 1, Math.ceil(Math.max(ay, by, cy)));
+ for (let y = y0; y <= y1; y++) {
+ for (let x = x0; x <= x1; x++) {
+ const texel = y * size + x;
+ if (covered[texel]) continue;
+ const px = x + 0.5, py = y + 0.5;
+ const w0 = ((bx - px) * (cy - py) - (cx - px) * (by - py)) / area;
+ const w1 = ((cx - px) * (ay - py) - (ax - px) * (cy - py)) / area;
+ const w2 = 1 - w0 - w1;
+ if (w0 < 0 || w1 < 0 || w2 < 0) continue;
+ covered[texel] = 1;
+
+ let nx = w0 * N[a * 3]! + w1 * N[b * 3]! + w2 * N[c * 3]!;
+ let ny = w0 * N[a * 3 + 1]! + w1 * N[b * 3 + 1]! + w2 * N[c * 3 + 1]!;
+ let nz = w0 * N[a * 3 + 2]! + w1 * N[b * 3 + 2]! + w2 * N[c * 3 + 2]!;
+ const nl = Math.hypot(nx, ny, nz) || 1;
+ nx /= nl; ny /= nl; nz /= nl;
+ const hit = tree.nearest(
+ w0 * P[a * 3]! + w1 * P[b * 3]! + w2 * P[c * 3]!,
+ w0 * P[a * 3 + 1]! + w1 * P[b * 3 + 1]! + w2 * P[c * 3 + 1]!,
+ w0 * P[a * 3 + 2]! + w1 * P[b * 3 + 2]! + w2 * P[c * 3 + 2]!,
+ nx, ny, nz, distance, place,
+ );
+ if (hit < 0) { missed++; continue; }
+ hits++;
+
+ const [hv, hw] = [place[0]!, place[1]!];
+ const hu = 1 - hv - hw;
+ const at = hit * 9;
+ let sx = hu * normals[at]! + hv * normals[at + 3]! + hw * normals[at + 6]!;
+ let sy = hu * normals[at + 1]! + hv * normals[at + 4]! + hw * normals[at + 7]!;
+ let sz = hu * normals[at + 2]! + hv * normals[at + 5]! + hw * normals[at + 8]!;
+ const sl = Math.hypot(sx, sy, sz) || 1;
+ sx /= sl; sy /= sl; sz /= sl;
+
+ // The frame the shader builds, in the order it builds it.
+ const tx = w0 * T[a * 4]! + w1 * T[b * 4]! + w2 * T[c * 4]!;
+ const ty = w0 * T[a * 4 + 1]! + w1 * T[b * 4 + 1]! + w2 * T[c * 4 + 1]!;
+ const tz = w0 * T[a * 4 + 2]! + w1 * T[b * 4 + 2]! + w2 * T[c * 4 + 2]!;
+ const handed = w0 * T[a * 4 + 3]! + w1 * T[b * 4 + 3]! + w2 * T[c * 4 + 3]!;
+ const lean = tx * nx + ty * ny + tz * nz;
+ let gx = tx - nx * lean, gy = ty - ny * lean, gz = tz - nz * lean;
+ const gl = Math.hypot(gx, gy, gz) || 1;
+ gx /= gl; gy /= gl; gz /= gl;
+ const ex = (ny * gz - nz * gy) * handed, ey = (nz * gx - nx * gz) * handed, ez = (nx * gy - ny * gx) * handed;
+
+ const u = sx * gx + sy * gy + sz * gz;
+ const v = sx * ex + sy * ey + sz * ez;
+ const w = sx * nx + sy * ny + sz * nz;
+ bend += Math.acos(Math.max(-1, Math.min(1, w)));
+ rgba[texel * 4] = Math.round((u * 0.5 + 0.5) * 255);
+ rgba[texel * 4 + 1] = Math.round((v * 0.5 + 0.5) * 255);
+ rgba[texel * 4 + 2] = Math.round((Math.max(w, 0) * 0.5 + 0.5) * 255);
+ }
+ }
+ }
+ const inside = hits + missed;
+ dilate(rgba, covered, size);
+
+ const from = corners.length / 9, to = mesh.indices.length / 3;
+ const warnings: string[] = [];
+ const missedShare = inside ? missed / inside : 1;
+ if (missedShare > MISSED_WARNING) warnings.push(`${Math.round(missedShare * 100)}% of the normal map found no surface within ${distance.toPrecision(3)} and is flat: raise normalBake.distance, or decimate less`);
+ if (from <= to) warnings.push(`the surface baked from has ${from} triangles and this one ${to}: there is no detail to carry. normalBake.from should name a step before the decimate`);
+ return {
+ size, rgba, warnings,
+ report: {
+ size, from, to,
+ coverage: Math.round((inside / (size * size)) * 1000) / 1000,
+ missed: Math.round(missedShare * 1000) / 1000,
+ distance: Number(distance.toPrecision(4)),
+ meanBend: hits ? Math.round(((bend / hits) * 180 / Math.PI) * 100) / 100 : 0,
+ },
+ };
+}
+
+/** Spreads each chart's edge outward, nearest first. */
+function dilate(rgba: Uint8Array, covered: Uint8Array, size: number): void {
+ let edge: number[] = [];
+ for (let i = 0; i < covered.length; i++) if (covered[i]) edge.push(i);
+ for (let pass = 0; pass < GUTTER && edge.length > 0; pass++) {
+ const next: number[] = [];
+ for (const i of edge) {
+ const x = i % size, y = (i - x) / size;
+ for (let dy = -1; dy <= 1; dy++) {
+ for (let dx = -1; dx <= 1; dx++) {
+ const [qx, qy] = [x + dx, y + dy];
+ if (qx < 0 || qy < 0 || qx >= size || qy >= size) continue;
+ const q = qy * size + qx;
+ if (covered[q]) continue;
+ covered[q] = 1;
+ rgba[q * 4] = rgba[i * 4]!;
+ rgba[q * 4 + 1] = rgba[i * 4 + 1]!;
+ rgba[q * 4 + 2] = rgba[i * 4 + 2]!;
+ next.push(q);
+ }
+ }
+ }
+ edge = next;
+ }
+}
diff --git a/pipeline/src/modeling/rig.ts b/pipeline/src/modeling/rig.ts
index 7a0f9a0bb..f6fdfbf05 100644
--- a/pipeline/src/modeling/rig.ts
+++ b/pipeline/src/modeling/rig.ts
@@ -930,7 +930,7 @@ export function timelineOf(animation: AnimationSource, joints: readonly RigJoint
* The prefab that draws and moves it: the mesh on the root, the joints below it as
* entities named so a clip's paths find them, and a player for the first clip.
*/
-export function prefabOf(name: string, meshPath: string, joints: readonly RigJoint[], cullBackfaces: boolean, firstClip: string | null, controller: string | null = null, extra: { goals?: readonly RigGoal[]; shapes?: number } = {}): Record<string, unknown> {
+export function prefabOf(name: string, meshPath: string, joints: readonly RigJoint[], cullBackfaces: boolean, firstClip: string | null, controller: string | null = null, extra: { goals?: readonly RigGoal[]; shapes?: number; normalMap?: string } = {}): Record<string, unknown> {
const idOf = (i: number): string => `j${i}`;
const goals = extra.goals ?? [];
const entities: Record<string, unknown>[] = [{
@@ -939,7 +939,7 @@ export function prefabOf(name: string, meshPath: string, joints: readonly RigJoi
visible: true,
components: [
{ type: 'Transform', data: {} },
- { type: 'MeshRenderer', data: { mesh: meshPath, lit: true, opaque: true, cullBackfaces, enabled: true } },
+ { type: 'MeshRenderer', data: { mesh: meshPath, ...(extra.normalMap ? { normalMap: extra.normalMap } : {}), lit: true, opaque: true, cullBackfaces, enabled: true } },
...(joints.length ? [{ type: 'MeshSkin', data: { joints: joints.map((_, i) => idOf(i)) } }] : []),
...(extra.shapes ? [{ type: 'MeshMorph', data: { weights: Array.from({ length: extra.shapes }, () => 0) } }] : []),
// A controller decides what plays; a player beside it would fight it for the same joints.
diff --git a/pipeline/tests/modeling-decimate-bake.test.ts b/pipeline/tests/modeling-decimate-bake.test.ts
new file mode 100644
index 000000000..f96cf6618
--- /dev/null
+++ b/pipeline/tests/modeling-decimate-bake.test.ts
@@ -0,0 +1,243 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/** Fewer triangles over the same surface, and that surface's detail read back out of the map. */
+import { beforeAll, describe, expect, it } from 'vitest';
+import { MeshChannel } from 'esengine';
+import { applyModelEdits, bakeModel, bakeNormalMap, prepareModeling, validateModelSource, type BakedModel, type ModelStep } from '../src/modeling';
+
+beforeAll(() => prepareModeling());
+
+const model = (...steps: ModelStep[]): BakedModel => bakeModel({ version: 1, steps });
+const ball: ModelStep = { op: 'sphere', radius: 1, rings: 48, sides: 96, tag: 'ball', color: [1, 1, 1] };
+
+function volume(m: BakedModel): number {
+ let v = 0;
+ for (const { v: [a, b, c] } of m.triangles) {
+ const [p, q, r] = [m.poly.positions[a]!, m.poly.positions[b]!, m.poly.positions[c]!];
+ v += p[0] * (q[1] * r[2] - q[2] * r[1]) + p[1] * (q[2] * r[0] - q[0] * r[2]) + p[2] * (q[0] * r[1] - q[1] * r[0]);
+ }
+ return v / 6;
+}
+
+function channel(m: BakedModel, semantic: number, components: number): Float32Array {
+ const { data } = m;
+ const { offset } = data.channels.find((c) => c.semantic === semantic)!;
+ const view = new DataView(data.vertices.buffer, data.vertices.byteOffset, data.vertices.byteLength);
+ const out = new Float32Array(data.vertexCount * components);
+ for (let v = 0; v < data.vertexCount; v++) for (let k = 0; k < components; k++) out[v * components + k] = view.getFloat32(v * data.vertexStride + offset + k * 4, true);
+ return out;
+}
+
+/** What the shader makes of the map at a point of a triangle: the texel there, in the frame the mesh carries, in model space. */
+function shadedNormal(m: BakedModel, rgba: Uint8Array, size: number, triangle: number, w: readonly [number, number, number]): { normal: number[]; at: number[] } {
+ const [uv, n, t, p] = [channel(m, MeshChannel.TexCoord0, 2), channel(m, MeshChannel.Normal, 3), channel(m, MeshChannel.Tangent, 4), channel(m, MeshChannel.Position, 3)];
+ const corners = [m.data.indices[triangle * 3]!, m.data.indices[triangle * 3 + 1]!, m.data.indices[triangle * 3 + 2]!];
+ const mix = (a: Float32Array, per: number, k: number): number => corners.reduce((sum, c, i) => sum + a[c * per + k]! * w[i]!, 0);
+ const x = Math.min(size - 1, Math.floor(mix(uv, 2, 0) * size));
+ const y = Math.min(size - 1, Math.floor((1 - mix(uv, 2, 1)) * size));
+ const texel = (y * size + x) * 4;
+ const tx = rgba[texel]! / 255 * 2 - 1, ty = rgba[texel + 1]! / 255 * 2 - 1;
+ const tz = Math.sqrt(Math.max(0, 1 - tx * tx - ty * ty));
+ const nl = Math.hypot(mix(n, 3, 0), mix(n, 3, 1), mix(n, 3, 2));
+ const N = [mix(n, 3, 0) / nl, mix(n, 3, 1) / nl, mix(n, 3, 2) / nl] as const;
+ const raw = [mix(t, 4, 0), mix(t, 4, 1), mix(t, 4, 2)] as const;
+ const lean = raw[0] * N[0] + raw[1] * N[1] + raw[2] * N[2];
+ const g = [raw[0] - N[0] * lean, raw[1] - N[1] * lean, raw[2] - N[2] * lean];
+ const gl = Math.hypot(g[0]!, g[1]!, g[2]!) || 1;
+ const T = [g[0]! / gl, g[1]! / gl, g[2]! / gl] as const;
+ const hand = mix(t, 4, 3);
+ const B = [(N[1] * T[2] - N[2] * T[1]) * hand, (N[2] * T[0] - N[0] * T[2]) * hand, (N[0] * T[1] - N[1] * T[0]) * hand] as const;
+ const out = [0, 1, 2].map((k) => T[k]! * tx + B[k]! * ty + N[k]! * tz);
+ const l = Math.hypot(out[0]!, out[1]!, out[2]!) || 1;
+ return { normal: out.map((c) => c / l), at: [mix(p, 3, 0), mix(p, 3, 1), mix(p, 3, 2)] };
+}
+
+const MIDDLE = [1 / 3, 1 / 3, 1 / 3] as const;
+
+const degrees = (a: readonly number[], b: readonly number[]): number =>
+ (Math.acos(Math.max(-1, Math.min(1, a[0]! * b[0]! + a[1]! * b[1]! + a[2]! * b[2]!))) * 180) / Math.PI;
+
+describe('decimate', () => {
+ it('keeps the asked share and the shape: closed, the same size, nearly the same volume', () => {
+ const full = model(ball);
+ const light = model(ball, { op: 'decimate', ratio: 0.1, error: 0.05 });
+ expect(light.report.triangles).toBeLessThanOrEqual(Math.ceil(full.report.triangles * 0.11));
+ expect(light.report.openEdges).toBe(0);
+ expect(light.report.nonManifoldEdges).toBe(0);
+ expect(light.report.insideOutShells).toBe(0);
+ expect(Math.abs(volume(light) - volume(full)) / volume(full)).toBeLessThan(0.05);
+ expect(light.report.bounds.size[0]).toBeGreaterThan(1.9);
+ });
+
+ it('`triangles` ends near that many, and the report says what it was given and what it left', () => {
+ const m = model(ball, { op: 'decimate', triangles: 400, error: 0.1 });
+ expect(m.report.triangles).toBeLessThanOrEqual(400);
+ const step = m.report.steps.at(-1)!;
+ expect(step.trianglesBefore).toBeGreaterThan(5000);
+ expect(step.trianglesAfter).toBe(m.report.triangles);
+ });
+
+ it('says when the error limit stopped it short of what was asked', () => {
+ const m = model(ball, { op: 'decimate', triangles: 20, error: 0.0005 });
+ expect(m.report.triangles).toBeGreaterThan(22);
+ expect(m.report.warnings.join()).toMatch(/stopped at \d+ triangles of the 20 asked/);
+ });
+
+ it('a flat grid goes to almost nothing, because nothing is lost', () => {
+ const m = model({ op: 'plane', size: 2 }, { op: 'slice', axis: 'x', at: [-0.5, 0, 0.5] }, { op: 'slice', axis: 'z', at: [-0.5, 0, 0.5] }, { op: 'weld' }, { op: 'decimate', ratio: 0.2 });
+ expect(m.report.bounds.size).toEqual([2, 0, 2]);
+ expect(m.report.openEdges).toBeGreaterThan(0);
+ // The rim is held, so it can only lose what lies inside it.
+ expect(m.report.triangles).toBeLessThan(18);
+ });
+
+ it('keeps paint and tags on the side they were on', () => {
+ const m = model(
+ { op: 'organic', resolution: 48, blend: 0, parts: [
+ { shape: 'sphere', radius: 1, at: [-1.2, 0, 0], tag: 'left', color: [1, 0, 0] },
+ { shape: 'sphere', radius: 1, at: [1.2, 0, 0], tag: 'right', color: [0, 0, 1] },
+ ] },
+ { op: 'decimate', ratio: 0.2, error: 0.05 },
+ );
+ expect(m.report.tags.left).toBeGreaterThan(0);
+ expect(m.report.tags.right).toBeGreaterThan(0);
+ for (const face of m.poly.faces) {
+ const x = face.v.reduce((sum, v) => sum + m.poly.positions[v]![0], 0) / face.v.length;
+ expect(face.tag).toBe(x < 0 ? 'left' : 'right');
+ expect(face.color![0]).toBe(x < 0 ? 1 : 0);
+ expect(face.smooth).toBe(true);
+ }
+ });
+
+ it('`faces` leaves the rest as they were, and the seam between them stays shut', () => {
+ const eye: ModelStep = { op: 'sphere', radius: 0.2, rings: 16, sides: 32, at: [5, 0, 0], tag: 'eye' };
+ const m = model(ball, eye, { op: 'decimate', faces: 'ball', ratio: 0.1, error: 0.05 });
+ const before = model(ball, eye);
+ expect(m.report.tags.eye).toBe(before.report.tags.eye);
+ expect(m.report.tags.ball).toBeLessThan(before.report.tags.ball! * 0.2);
+ expect(m.report.openEdges).toBe(0);
+ });
+
+ it('a rig is found on what is left, not on what was', () => {
+ const m = bakeModel({
+ version: 1,
+ rig: { joints: [{ name: 'root', at: [0, -1, 0] }, { name: 'top', at: [0, 1, 0], parent: 'root' }] },
+ steps: [ball, { op: 'decimate', ratio: 0.1, error: 0.05 }],
+ });
+ expect(m.rig!.skin.joints.length).toBe(m.poly.positions.length * 4);
+ expect(m.data.channels.some((c) => c.semantic === MeshChannel.Joints)).toBe(true);
+ });
+
+ it('refuses what cannot be meant', () => {
+ expect(() => model(ball, { op: 'decimate' })).toThrow(/one of `ratio` and `triangles`/);
+ expect(() => model(ball, { op: 'decimate', ratio: 0.5, triangles: 10 })).toThrow(/one of `ratio` and `triangles`/);
+ expect(() => model(ball, { op: 'decimate', ratio: 2 })).toThrow(/must be in \(0, 1\]/);
+ expect(() => model(ball, { op: 'decimate', faces: 'nothing', ratio: 0.5 })).toThrow(/matched no face/);
+ });
+});
+
+describe('normalBake', () => {
+ const bakeOf = (extra: Record<string, unknown>, steps: ModelStep[]) => {
+ const m = bakeModel({ version: 1, ...extra, steps });
+ return { m, map: bakeNormalMap(m) };
+ };
+
+ it('brings a round surface back on a coarse one: at the middle of each triangle the map says the true normal', () => {
+ const { m, map } = bakeOf({ normalBake: { size: 256 } }, [ball, { op: 'decimate', triangles: 300, error: 0.2 }]);
+ expect(map.report.missed).toBe(0);
+ expect(map.report.from).toBeGreaterThan(5000);
+ expect(map.report.to).toBeLessThanOrEqual(300);
+ const off: number[] = [];
+ for (let t = 0; t < m.data.indices.length / 3; t++) {
+ const { normal, at } = shadedNormal(m, map.rgba, map.size, t, MIDDLE);
+ const l = Math.hypot(at[0]!, at[1]!, at[2]!);
+ off.push(degrees(normal, at.map((x) => x / l)));
+ }
+ expect(Math.max(...off)).toBeLessThan(6);
+ });
+
+ it('carries a slope in the direction it runs: red and green are not swapped or flipped', () => {
+ // A tilted square is the detail; a flat one under it is the mesh. Tilt toward +x, then toward +z.
+ for (const [rotate, want] of [[[0, 0, -20], [Math.sin(20 * Math.PI / 180), Math.cos(20 * Math.PI / 180), 0]], [[20, 0, 0], [0, Math.cos(20 * Math.PI / 180), Math.sin(20 * Math.PI / 180)]]] as const) {
+ const { m, map } = bakeOf({ normalBake: { size: 128, from: 'tilted', distance: 2 } }, [
+ { op: 'plane', size: 4 }, { id: 'tilted', op: 'transform', rotate: rotate as unknown as number[] },
+ { op: 'delete', faces: 'all' }, { op: 'plane', size: 1 },
+ ]);
+ for (let t = 0; t < m.data.indices.length / 3; t++) expect(degrees(shadedNormal(m, map.rgba, map.size, t, MIDDLE).normal, want)).toBeLessThan(3);
+ }
+ });
+
+ it('reads a texel where its v says: the map is not upside down', () => {
+ // Only the side at -z is tilted. A row counted from the wrong end would read the other side's texels.
+ const { m, map } = bakeOf({ smoothAngle: 0, normalBake: { size: 128, from: 'bent', distance: 2 } }, [
+ { op: 'plane', size: 4 }, { op: 'slice', axis: 'z', at: 0 },
+ { id: 'bent', op: 'transform', faces: { within: { max: [null, null, 0] } }, rotate: [-25, 0, 0], pivot: [0, 0, 0] },
+ { op: 'delete', faces: 'all' }, { op: 'plane', size: 3 },
+ ]);
+ const seen = { flat: 0, bent: 0 };
+ // Near each corner of each triangle, well away from the crease at z 0.
+ for (let t = 0; t < m.data.indices.length / 3; t++) {
+ for (const w of [[0.9, 0.05, 0.05], [0.05, 0.9, 0.05], [0.05, 0.05, 0.9]] as const) {
+ const { normal, at } = shadedNormal(m, map.rgba, map.size, t, w);
+ if (Math.abs(at[2]!) < 0.5) continue;
+ const lean = degrees(normal, [0, 1, 0]);
+ if (at[2]! > 0) { expect(lean).toBeLessThan(3); seen.flat++; } else { expect(lean).toBeGreaterThan(15); seen.bent++; }
+ }
+ }
+ expect(seen.flat).toBeGreaterThan(0);
+ expect(seen.bent).toBeGreaterThan(0);
+ });
+
+ it('gives the mesh texels of its own and a frame, on a rigged and shaped mesh too', () => {
+ const { m, map } = bakeOf({
+ normalBake: { size: 128 },
+ rig: { joints: [{ name: 'root', at: [0, -1, 0] }, { name: 'top', at: [0, 1, 0], parent: 'root' }] },
+ shapes: [{ name: 'puff', moves: [{ near: { at: [0, 1, 0], radius: 0.5 }, translate: [0, 0.2, 0] }] }],
+ }, [ball, { op: 'decimate', ratio: 0.1, error: 0.05 }]);
+ for (const semantic of [MeshChannel.Tangent, MeshChannel.Joints, MeshChannel.Weights]) expect(m.data.channels.some((c) => c.semantic === semantic)).toBe(true);
+ expect(m.data.morph!.deltas.length).toBe(m.data.vertexCount * 6);
+ const uv = channel(m, MeshChannel.TexCoord0, 2);
+ for (const x of uv) { expect(x).toBeGreaterThanOrEqual(0); expect(x).toBeLessThanOrEqual(1); }
+ expect(map.report.coverage).toBeGreaterThan(0.2);
+ });
+
+ it('fills past the charts\' edges, so a bilinear read never meets the empty colour next to one', () => {
+ const { map } = bakeOf({ normalBake: { size: 256 } }, [ball, { op: 'decimate', triangles: 300, error: 0.2 }]);
+ let filled = 0;
+ for (let i = 0; i < 256 * 256; i++) if (map.rgba[i * 4] !== 128 || map.rgba[i * 4 + 1] !== 128) filled++;
+ expect(filled / (256 * 256)).toBeGreaterThan(map.report.coverage);
+ });
+
+ it('warns when the map found nothing, and when there was nothing to find', () => {
+ const far = bakeOf({ normalBake: { size: 64, from: 'away', distance: 0.1 } }, [
+ { op: 'plane', size: 4, at: [0, 5, 0] }, { id: 'away', op: 'transform', scale: 1 }, { op: 'delete', faces: 'all' }, { op: 'plane', size: 1 },
+ ]);
+ expect(far.map.report.missed).toBe(1);
+ expect(far.map.warnings.join()).toMatch(/found no surface/);
+ const same = bakeOf({ normalBake: { size: 64, from: 'same' } }, [{ id: 'same', op: 'box', size: 1 }]);
+ expect(same.map.warnings.join()).toMatch(/no detail to carry/);
+ });
+
+ it('refuses a bake with nothing to read from, a size no texture has, and a step that is not there', () => {
+ expect(() => validateModelSource({ steps: [ball], normalBake: {} })).toThrow(/a decimate step, or `from`/);
+ expect(() => validateModelSource({ steps: [{ op: 'decimate', ratio: 0.5 }], normalBake: {} })).toThrow(/first step/);
+ expect(() => validateModelSource({ steps: [ball, { op: 'decimate', ratio: 0.5 }], normalBake: { size: 1000 } })).toThrow(/power of two/);
+ expect(() => validateModelSource({ steps: [ball], normalBake: { from: 'nope' } })).toThrow(/no step's id/);
+ expect(() => validateModelSource({ steps: [ball, { op: 'decimate', ratio: 0.5 }], normalBake: { sizes: 64 } })).toThrow(/unknown field "sizes"/);
+ });
+
+ it('a model without one is drawn as it always was: no frame, and the box UVs', () => {
+ const m = model(ball);
+ expect(m.detail).toBeUndefined();
+ expect(m.data.channels.some((c) => c.semantic === MeshChannel.Tangent)).toBe(false);
+ expect(() => bakeNormalMap(m)).toThrow(/no normalBake/);
+ });
+
+ it('an edit can add it and take it away', () => {
+ const source = validateModelSource({ steps: [ball, { op: 'decimate', ratio: 0.5 }] });
+ const baked = applyModelEdits(source, [{ op: 'settings', fields: { normalBake: { size: 64 } } }]);
+ expect(baked.normalBake).toEqual({ size: 64 });
+ const off = applyModelEdits(baked, [{ op: 'settings', fields: { normalBake: null as never } }]);
+ expect(off.normalBake).toBeUndefined();
+ });
+});
diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjs
index d4c88f950..e054ac249 100644
--- a/tools/releaseNotes.mjs
+++ b/tools/releaseNotes.mjs
@@ -3872,6 +3872,10 @@ export const NOTED = {
{"note":"A linear game looks the same on every Play."},
"test(editor): a scene draws the same on every Play, with effects or without":
{"internal":"a check, for a fix the engine commit carries"},
+ "feat(modeling): a model thins to a few triangles and keeps its detail as a normal map":
+ {"note":"An AI building a model can make it light and keep its detail."},
+ "feat(mcp): model_write thins a model and bakes its detail, and a check compares it lit against the dense one":
+ {"note":"An AI building a model can make it light and keep its detail."},
"feat(render): a mesh can have 256 joints where 64 imported a rig with fingers and a face as a statue":
{"note":"A mesh can have 256 joints, up from 64."},
"feat(mcp): model_write takes 256 joints, and the modeling check poses past the old limit":