A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.
TypeScript 2d-game-engine3d3d-game-engineandroidcppecsgame-editorgame-enginegamedeviostypescriptwebassemblywebglwebgpuwechat-minigame
feat(modeling): a model says how rough and how metal it is, part by part, and a bake keeps that as a map
yhh811536596122
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 734b94231..8b3f7e3e4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -22,6 +22,8 @@ published separately; it ships inside the editor.- **Glyphs can be put on the atlas before they are shown.** The first page of a novel is hundreds of new characters, all landing in one frame. `preloadGlyphs(app, label, text, { budgetMs })` makes them ahead of time, a slice a frame, drawn as that `Text` will draw them, and returns `progress`, `done` and `release()`. On a Snapdragon 8 Elite a book's 2992 characters at 28 px took 145 ms over 25 frames with the worst frame at 9.5 ms, after which the first page's frame went from 49 to 11 ms and no page made a new glyph. Preloaded glyphs hold their atlas pages (4 MB each, a few hundred characters at that size) until `release()`.- **Voxel GI lights what the camera does not see, and bounces light more than once.** The cache now reads the sun's shadow map back into the world, so a surface the sun lights behind or beside the camera still sends light into the view. Each surface also gives off again the bounce light it received, so a corner or a floor in shade gets light that took two or more bounces to reach it. A ray now brings what it meets at full strength however far it went (it used to fade with distance, 40% at 4 m with 12 cm voxels). Each voxel is weighed by how much of it a surface covers, so a bright patch no longer lights like the whole voxel around it. Together these keep the result close to a path tracer's whatever the voxel size. In a sunlit test room the whole frame carries 0.95 of Cycles' light at the default size and 0.74 at 12 cm voxels; before, it was 0.54 at the default and darker the finer the grid. With the lit patch behind the camera, the walls went from no bounce light to 25–57%. A closed room the sun only lights from outside stays dark. Two inspector controls tune it: **Off-screen Sun Albedo** (the colour the shadow map cannot give, assumed) and **Multi-bounce**. A fine grid no longer drops every voxel: a count truncated by the cache's ageing had kept them below the two samples a ray asks for. The WebGL2 fallback does the same in its own passes: each voxel asks the sun's map whether a sunlit surface stands in it, and traces two rays through last frame's grid for the light it is given. In the test room it carries 1.00 of Cycles' light at the default size (it was 0.56) and 0.80 at 12 cm; with the lit patch behind the camera 0.20 (it was 0). Its voxel pass costs about 1.3 ms more at 720p on a desktop.+- **An AI building a model can say what it is made of.** `material` sets roughness and metal for a whole model, `paint` and `stain` for parts of it, and `normalBake` with `surface: true` keeps the differences as a map: wet moss on dry stone, bare steel where paint wore off.+- **An AI building a model can colour it finely, and keep that on a thin mesh.** A `stain` step lays a colour over a surface by a pattern, in what is sunken or on what stands out, by which way it faces or around a spot: moss on top, dirt in the cracks, a pupil. `normalBake` with `color: true` writes that as a base colour map, in linear and gamma projects alike.- **An AI building a model can carve fine detail into it.** A `displace` step pushes bumps, wrinkles, cracks, scales, pores, ribs, weave or scratches out of a surface along its normal. A pattern needs no UV and has no seams, can be limited to some faces or to around a point, stacks with others, and splits the surface as far as it needs. Followed by `decimate` and `normalBake`, half a million triangles of detail draw on 1,500.@@ -48,6 +50,8 @@ published separately; it ships inside the editor.### Fixed+- **A material's normal and metal-rough maps are read as data in the editor's Play.** A map named beside its `.esmaterial` missed its import settings there, as Play spells it by URL, so it was decoded as colour: mid-range roughness read far too smooth, and normal maps bent. Filter and wrap settings were lost the same way.+- **A rotation given to an agent as a quaternion is that rotation.** `set_field` and `apply_scene_ops` read `{ w, x, y, z }` as degrees and dropped `w`, so a character meant to turn 90 degrees turned under one, with no error. A joint posed this way looked as if the skin did not follow it.- **An agent's `get_stats` waits out a long frame.** It gave a running game two seconds to answer, so on a machine that takes that long to draw a frame it said the game had stopped.diff --git a/desktop b/desktopindex fc0f20774..1cd63d7b3 160000--- a/desktop+++ b/desktop@@ -1 +1 @@-Subproject commit fc0f20774da903313eb3d679c12ce81ab353f74d+Subproject commit 1cd63d7b327047f8d666cb1d59e918432cc2284fdiff --git a/pipeline/src/modeling/index.ts b/pipeline/src/modeling/index.tsindex a9fbe7d4d..e88dba593 100644--- a/pipeline/src/modeling/index.ts+++ b/pipeline/src/modeling/index.ts@@ -10,7 +10,7 @@ export {} from './modelSource';export { organic, ORGANIC_DEFAULT_RESOLUTION, ORGANIC_MAX_RESOLUTION, type OrganicParams, type OrganicPart } from './organic';export { needsPreparing, prepareModeling } from './booleans';-export { bakeModel, type BakedModel, type ModelReport } from './modelBake';+export { bakeModel, modelMaterialDoc, type BakedModel, type ModelReport } from './modelBake';export { bakeNormalMap, type NormalBakeReport, type NormalMap } from './normalBake';export type { FaceSelector } from './polyMesh';export { DEFAULT_MODEL_VIEWS, MODEL_VIEWS, renderModelSheet, renderRigSheet, type ModelSheet, type ModelViewName, type RigViewOptions } from './modelViews';diff --git a/pipeline/src/modeling/modelBake.ts b/pipeline/src/modeling/modelBake.tsindex 9b0834098..23a92e480 100644--- a/pipeline/src/modeling/modelBake.ts+++ b/pipeline/src/modeling/modelBake.ts@@ -8,7 +8,7 @@* "2 open edges" or "this shell is inside out" gets fixed on the next call,* where the same fault in a lit picture reads as a slightly odd shadow.*/-import { MeshChannel, MeshChannelType, packChannels, unwrapLightmapUV, type MeshData, type MeshMorphTargets } from 'esengine';+import { BlendMode, CullMode, MATERIAL_FORMAT_VERSION, MeshChannel, MeshChannelType, packChannels, unwrapLightmapUV, type MeshData, type MeshMorphTargets } from 'esengine';import { generateTangents } from '../assets/meshTangents';import { triangulateFace } from './triangulate';import { faceUvs } from './modelUv';@@ -64,7 +64,7 @@ export interface BakedModel {/** One triangle per entry: its three positions' indices, and the face it came from. */triangles: { v: [number, number, number]; face: number }[];/** Present where a source has a normalBake: what bakeNormalMap carries, onto a mesh already unwrapped and given tangents for it. */- detail?: { high: PolyMesh; size: number; distance?: number; smoothAngle: number; lowColors?: Uint8Array };+ detail?: { high: PolyMesh; size: number; distance?: number; smoothAngle: number; lowColors?: Uint8Array; surface?: { roughness: number; metallic: number } };}// ---- Report -----------------------------------------------------------------------@@ -271,6 +271,28 @@ function withTangents(mesh: MeshData): MeshData {};}+/** What a material has to say about a model, or null where a plain MeshRenderer says all of it. */+export function modelMaterialDoc(source: ModelSource, cullBackfaces: boolean, maps: { normal?: string | null; surface?: string | null }): Record<string, unknown> | null {+ const { material } = source;+ if (!material && !maps.surface) return null;+ return {+ version: MATERIAL_FORMAT_VERSION,+ type: 'material',+ shader: 'builtin:model',+ blendMode: BlendMode.None,+ depthTest: true,+ depthWrite: true,+ cull: cullBackfaces ? CullMode.Back : CullMode.None,+ properties: {+ ...(maps.normal ? { u_normalMap: maps.normal } : {}),+ // A map holds the values and the factors multiply it, so they stand aside.+ ...(maps.surface+ ? { u_metallicRoughnessMap: maps.surface, u_roughness: 1, u_metallic: 1 }+ : { u_roughness: material?.roughness ?? 1, u_metallic: material?.metallic ?? 0 }),+ },+ };+}+/** Validates, builds and bakes @p raw. Throws a ModelBuildError naming the step that failed; writes nothing, so a failure leaves nothing half made. */export function bakeModel(raw: unknown): BakedModel {const source = validateModelSource(raw);@@ -337,6 +359,10 @@ export function bakeModel(raw: unknown): BakedModel {for (const f of mesh.faces) if (f.tag) tags[f.tag] = (tags[f.tag] ?? 0) + 1;const warnings: string[] = [...notes, ...(layoutNote ? [layoutNote] : [])];+ if (!bake?.surface && (high ?? mesh).faces.some((f) => f.roughness !== undefined || f.metallic !== undefined)) {+ const uniform = mesh.faces.every((f) => f.roughness === mesh.faces[0]!.roughness && f.metallic === mesh.faces[0]!.metallic);+ warnings.push(`faces say their own roughness or metallic, and nothing carries that: give normalBake \`surface: true\`${uniform ? ', or say it once in `material`' : ''}`);+ }if (triangleCount > HEAVY_TRIANGLES) warnings.push(`${triangleCount} triangles is a lot to draw: end with a \`decimate\`, and \`normalBake\` to keep the detail`);if (openEdges) warnings.push(`${openEdges} open edge${openEdges > 1 ? 's' : ''}: the surface has holes you can see into`);if (nonManifoldEdges) warnings.push(`${nonManifoldEdges} edge${nonManifoldEdges > 1 ? 's' : ''} shared by more than two faces, which a later bevel refuses: \`weld\` may have joined things that should stay apart, or on an organic skin two parts just touch, so overlap them or raise blend`);@@ -369,7 +395,7 @@ export function bakeModel(raw: unknown): BakedModel {data: unwrapped,...(rig ? { rig } : {}),...(shapes ? { shapes } : {}),- ...(bake && high ? { detail: { high, size, ...(bake.distance !== undefined ? { distance: bake.distance } : {}), smoothAngle: source.smoothAngle ?? DEFAULT_SMOOTH_ANGLE, ...(lowColors ? { lowColors } : {}) } } : {}),+ ...(bake && high ? { detail: { high, size, ...(bake.distance !== undefined ? { distance: bake.distance } : {}), smoothAngle: source.smoothAngle ?? DEFAULT_SMOOTH_ANGLE, ...(lowColors ? { lowColors } : {}), ...(bake.surface ? { surface: { roughness: source.material?.roughness ?? 1, metallic: source.material?.metallic ?? 0 } } : {}) } } : {}),triangles: triangles.flatMap((tris, face) => tris.map((v) => ({ v, face }))),report: {vertices: unwrapped.vertexCount,diff --git a/pipeline/src/modeling/modelOps.ts b/pipeline/src/modeling/modelOps.tsindex 58283e204..6487ce8f2 100644--- a/pipeline/src/modeling/modelOps.ts+++ b/pipeline/src/modeling/modelOps.ts@@ -458,10 +458,18 @@ export function deleteFaces(mesh: PolyMesh, p: { faces: FaceSelector }): PolyMes}/** Recolours faces: linear RGB or RGBA in 0..1, multiplied into whatever material draws the mesh. */-export function paint(mesh: PolyMesh, p: { faces: FaceSelector; color: readonly number[] }): PolyMesh {- const color = toColor(p.color);+export function paint(mesh: PolyMesh, p: { faces: FaceSelector; color?: readonly number[]; roughness?: number; metallic?: number }): PolyMesh {+ if (p.color === undefined && p.roughness === undefined && p.metallic === undefined) throw new Error('give at least one of color, roughness, metallic');+ for (const [name, v] of [['roughness', p.roughness], ['metallic', p.metallic]] as const) {+ if (v !== undefined && !(v >= 0 && v <= 1)) throw new Error(`${name} ${v} must be 0..1`);+ }+ const set = {+ ...(p.color !== undefined ? { color: toColor(p.color) } : {}),+ ...(p.roughness !== undefined ? { roughness: p.roughness } : {}),+ ...(p.metallic !== undefined ? { metallic: p.metallic } : {}),+ };const hit = new Set(requireFaces(mesh, p.faces));- return { ...mesh, faces: mesh.faces.map((f, i) => (hit.has(i) ? { ...f, color } : f)) };+ return { ...mesh, faces: mesh.faces.map((f, i) => (hit.has(i) ? { ...f, ...set } : f)) };}/** Names faces, so later steps can ask for them by what they are. */diff --git a/pipeline/src/modeling/modelSource.ts b/pipeline/src/modeling/modelSource.tsindex 748fd8203..0fa3dc735 100644--- a/pipeline/src/modeling/modelSource.ts+++ b/pipeline/src/modeling/modelSource.ts@@ -42,6 +42,8 @@ export interface ModelSource {uvScale?: number;/** Lay the base UV out so no two triangles share texels: for a texture painted for this one model, where tiling is not wanted. */uvAtlas?: boolean;+ /** What every face is made of until a `paint` or `stain` says otherwise. */+ material?: ModelMaterial;/** Carries a denser state of the steps onto the final surface as a normal map. It lays the base UV out as uvAtlas does. */normalBake?: NormalBakeSource;/** Joints: the mesh is skinned to them, with weights found from the surface. */@@ -55,6 +57,13 @@ export interface ModelSource {steps: ModelStep[];}+export interface ModelMaterial {+ /** 0 is a mirror, 1 is chalk. Default 1. */+ roughness?: number;+ /** 0 is anything else, 1 is metal. Default 0. */+ metallic?: number;+}+export interface NormalBakeSource {/** The step whose result holds the detail. Default: what the last `decimate` started from. */from?: string;@@ -64,6 +73,8 @@ export interface NormalBakeSource {distance?: number;/** Also writes the dense surface's paint as a base colour map, leaving the mesh's own vertex colours white. */color?: boolean;+ /** Also writes the dense surface's roughness and metal as a map. */+ surface?: boolean;}type Kind = 'number' | 'integer' | 'boolean' | 'string' | 'vec3' | 'number|vec3' | 'number|vec2' | 'number|numbers'@@ -100,11 +111,11 @@ export const MODEL_OPS: Record<string, OpSpec> = {boolean: { params: { operation: 'operation', with: 'faces', from: 'faces', keep: 'boolean', tag: 'string', color: 'color' }, required: ['operation', 'with'], run: booleanOp as unknown as OpSpec['run'], says: 'cuts or joins closed solids: subtract `with` from `from` makes a real hole' },subdivide: { params: { faces: 'faces', levels: 'integer', crease: 'number' }, required: [], run: subdivide as unknown as OpSpec['run'], says: 'rounds whole shells into smooth surfaces, keeping edges sharper than `crease` degrees' },displace: { params: { faces: 'faces', near: 'near', pattern: 'pattern', amplitude: 'number', scale: 'number', octaves: 'integer', seed: 'integer', stretch: 'vec3', warp: 'number', width: 'number', axis: 'axis', mirror: 'axis', detail: 'number' }, required: ['pattern', 'amplitude', 'scale'], run: ((mesh: PolyMesh, params: never) => displace(mesh, params)) as OpSpec['run'], says: 'pushes fine detail out of the surface along its normal, splitting faces as far as the pattern needs: bumps, wrinkles, cracks, scales, pores, ribs, weave or scratches' },- stain: { params: { faces: 'faces', near: 'near', hard: 'boolean', color: 'color', amount: 'number', pattern: 'pattern', scale: 'number', octaves: 'integer', seed: 'integer', stretch: 'vec3', warp: 'number', width: 'number', axis: 'axis', mirror: 'axis', threshold: 'number', softness: 'number', where: 'where', reach: 'number', facing: 'vec3', detail: 'number' }, required: ['color'], run: ((mesh: PolyMesh, params: never) => stain(mesh, params)) as OpSpec['run'], says: 'lays colour over the surface by a pattern, by shape (`where`: cavity for dirt in what is sunken, edges for wear on what stands out), by which way it faces, or around a spot: pupils, moss, grime' },+ stain: { params: { faces: 'faces', near: 'near', hard: 'boolean', color: 'color', roughness: 'number', metallic: 'number', amount: 'number', pattern: 'pattern', scale: 'number', octaves: 'integer', seed: 'integer', stretch: 'vec3', warp: 'number', width: 'number', axis: 'axis', mirror: 'axis', threshold: 'number', softness: 'number', where: 'where', reach: 'number', facing: 'vec3', detail: 'number' }, required: [], run: ((mesh: PolyMesh, params: never) => stain(mesh, params)) as OpSpec['run'], says: 'lays colour, `roughness` or `metallic` over the surface by a pattern, by shape (`where`: cavity for dirt in what is sunken, edges for wear on what stands out), by which way it faces, or around a spot: pupils, moss, grime' },decimate: { params: { from: 'string', faces: 'faces', ratio: 'number', triangles: 'integer', error: 'number' }, required: [], run: decimate as unknown as OpSpec['run'], says: 'leaves fewer triangles over the same surface: `ratio` to keep or `triangles` to end with, stopping sooner if the surface would move more than `error` (default 0.02) of its size; `from` thins the surface as it stood after that step, before its fine detail' },uv: { params: { faces: 'faces', mode: 'uvmode', axis: 'string', at: 'vec3', scale: 'number|vec2only', repeat: 'number|vec2only', offset: 'vec2', rotate: 'number' }, required: ['mode'], run: uvStep as unknown as OpSpec['run'], says: 'sets how faces take a texture: box, planar, cylinder or sphere' },delete: { params: { faces: 'faces' }, required: ['faces'], run: ops.deleteFaces as OpSpec['run'], says: 'removes faces, leaving a hole' },- paint: { params: { faces: 'faces', color: 'color' }, required: ['faces', 'color'], run: ops.paint as OpSpec['run'], says: 'sets vertex colour: linear [r, g, b] in 0..1, multiplied into the material' },+ paint: { params: { faces: 'faces', color: 'color', roughness: 'number', metallic: 'number' }, required: ['faces'], run: ops.paint as OpSpec['run'], says: 'sets vertex colour: linear [r, g, b] in 0..1, multiplied into the material; and how `roughness` and `metallic` a surface is, 0..1' },tag: { params: { faces: 'faces', tag: 'string' }, required: ['faces', 'tag'], run: ops.tagFaces as OpSpec['run'], says: 'names faces for later steps' },weld: { params: { distance: 'number' }, required: [], run: ops.weld as OpSpec['run'], says: 'merges coincident vertices and removes the back-to-back faces that leaves' },};@@ -276,13 +287,14 @@ export class ModelBuildError extends Error {}function checkNormalBake(v: unknown, steps: readonly Partial<ModelStep>[]): string | null {- if (typeof v !== 'object' || v === null || Array.isArray(v)) return 'must be { from?, size?, distance?, color? }';+ if (typeof v !== 'object' || v === null || Array.isArray(v)) return 'must be { from?, size?, distance?, color?, surface? }';const bake = v as Record<string, unknown>;- for (const key of Object.keys(bake)) if (!['from', 'size', 'distance', 'color'].includes(key)) return `has an unknown field "${key}"; it takes from, size, distance, color`;+ for (const key of Object.keys(bake)) if (!['from', 'size', 'distance', 'color', 'surface'].includes(key)) return `has an unknown field "${key}"; it takes from, size, distance, color, surface`;const { size, distance, from } = bake;if (size !== undefined && !(Number.isInteger(size) && (size as number) >= 64 && (size as number) <= 4096 && ((size as number) & ((size as number) - 1)) === 0)) return 'size must be a power of two, 64..4096';if (distance !== undefined && !(isNum(distance) && distance > 0)) return 'distance must be a positive number';if (bake.color !== undefined && typeof bake.color !== 'boolean') return 'color must be true or false';+ if (bake.surface !== undefined && typeof bake.surface !== 'boolean') return 'surface must be true or false';if (from !== undefined) {if (typeof from !== 'string') return 'from must be a step id';if (!steps.some((s) => s?.id === from)) return `from "${from}" is no step's id`;@@ -301,8 +313,8 @@ export function validateModelSource(raw: unknown): ModelSource {if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) throw new ModelBuildError('a model source must be an object with `steps`', null, null, null);const source = raw as Record<string, unknown>;for (const key of Object.keys(source)) {- if (!['version', 'smoothAngle', 'uvScale', 'uvAtlas', 'normalBake', 'rig', 'shapes', 'animations', 'controller', 'steps'].includes(key)) {- throw new ModelBuildError(`unknown field "${key}"; a model source has version, smoothAngle, uvScale, uvAtlas, normalBake, rig, shapes, animations, controller, steps`, null, null, null);+ if (!['version', 'smoothAngle', 'uvScale', 'uvAtlas', 'material', 'normalBake', 'rig', 'shapes', 'animations', 'controller', 'steps'].includes(key)) {+ throw new ModelBuildError(`unknown field "${key}"; a model source has version, smoothAngle, uvScale, uvAtlas, material, normalBake, rig, shapes, animations, controller, steps`, null, null, null);}}if (source.version !== undefined && source.version !== MODEL_SOURCE_VERSION) {@@ -336,6 +348,14 @@ export function validateModelSource(raw: unknown): ModelSource {if (bad) throw new ModelBuildError(bad, null, null, null);}if (!Array.isArray(source.steps)) throw new ModelBuildError('`steps` must be an array', null, null, null);+ if (source.material !== undefined) {+ const m = source.material as Record<string, unknown> | null;+ if (typeof m !== 'object' || m === null || Array.isArray(m)) throw new ModelBuildError('material must be { roughness?, metallic? }', null, null, null);+ for (const [key, value] of Object.entries(m)) {+ if (key !== 'roughness' && key !== 'metallic') throw new ModelBuildError(`material has an unknown field "${key}"; it takes roughness, metallic`, null, null, null);+ if (!(isNum(value) && value >= 0 && value <= 1)) throw new ModelBuildError(`material.${key} must be 0..1`, null, null, null);+ }+ }if (source.normalBake !== undefined) {const bad = checkNormalBake(source.normalBake, source.steps as Partial<ModelStep>[]);if (bad) throw new ModelBuildError(`normalBake ${bad}`, null, null, null);@@ -404,7 +424,10 @@ export function buildPolyMesh(source: ModelSource): ModelBuild {const input = op === 'decimate' && from !== undefined ? kept.get(from)! : mesh;const before = op === 'decimate' ? triangleCount(input) : 0;try {- mesh = MODEL_OPS[op]!.run(input, (op === 'decimate' ? rest : params) as never);+ // A partial stain mixes from what is there, and what is there by default is the source's to say.+ mesh = op === 'stain'+ ? stain(input, params as never, undefined, { roughness: source.material?.roughness ?? 1, metallic: source.material?.metallic ?? 0 })+ : MODEL_OPS[op]!.run(input, (op === 'decimate' ? rest : params) as never);} catch (e) {throw new ModelBuildError(`${stepLabel(step, index)}: ${e instanceof Error ? e.message : String(e)}`, index, id ?? null, op);}@@ -425,7 +448,7 @@ export type ModelEdit =| { op: 'set'; step: string | number; fields: Record<string, unknown> }| { op: 'insert'; step: ModelStep; before?: string | number; after?: string | number }| { op: 'remove'; step: string | number }- | { op: 'settings'; fields: Pick<ModelSource, 'smoothAngle' | 'uvScale' | 'uvAtlas' | 'normalBake' | 'rig' | 'shapes' | 'animations' | 'controller'> };+ | { op: 'settings'; fields: Pick<ModelSource, 'smoothAngle' | 'uvScale' | 'uvAtlas' | 'material' | 'normalBake' | 'rig' | 'shapes' | 'animations' | 'controller'> };function indexOfStep(source: ModelSource, ref: string | number): number {const i = typeof ref === 'number' ? ref : source.steps.findIndex((s) => s.id === ref);diff --git a/pipeline/src/modeling/normalBake.ts b/pipeline/src/modeling/normalBake.tsindex 0ad7e0592..2824f1f98 100644--- a/pipeline/src/modeling/normalBake.ts+++ b/pipeline/src/modeling/normalBake.ts@@ -55,6 +55,8 @@ export interface NormalMap {* alike, so this reaches it as the same number in both.*/colorRgba?: Uint8Array;+ /** The dense surface's roughness in green and metal in blue, with `normalBake.surface`: glTF's packing, which u_metallicRoughnessMap reads. */+ surfaceRgba?: Uint8Array;report: NormalBakeReport;warnings: string[];}@@ -194,12 +196,13 @@ function readFloats(mesh: MeshData, semantic: number, components: number): Float}/** The dense surface as flat triangles, and the shading normal at each of their corners. */-function denseSurface(high: PolyMesh, smoothAngle: number): { corners: Float64Array; normals: Float64Array; colors: Uint8Array } {+function denseSurface(high: PolyMesh, smoothAngle: number, fallback: { roughness: number; metallic: number }): { corners: Float64Array; normals: Float64Array; colors: Uint8Array; surfaces: Uint8Array } {const normalsOf = cornerNormals(high, smoothAngle);let count = 0;for (const face of high.faces) count += Math.max(0, face.v.length - 2);const corners = new Float64Array(count * 9), normals = new Float64Array(count * 9);const colors = new Uint8Array(count * 4);+ const surfaces = new Uint8Array(count * 2);let t = 0;high.faces.forEach((face, f) => {const color = face.color ?? [1, 1, 1, 1];@@ -210,10 +213,12 @@ function denseSurface(high: PolyMesh, smoothAngle: number): { corners: Float64Arfor (let k = 0; k < 3; k++) { corners[t * 9 + c * 3 + k] = p[k]!; normals[t * 9 + c * 3 + k] = n[k]!; }});colors.set(bytes, t * 4);+ surfaces[t * 2] = Math.round((face.roughness ?? fallback.roughness) * 255);+ surfaces[t * 2 + 1] = Math.round((face.metallic ?? fallback.metallic) * 255);t++;}});- return { corners: t === count ? corners : corners.subarray(0, t * 9), normals, colors };+ return { corners: t === count ? corners : corners.subarray(0, t * 9), normals, colors, surfaces };}/** Bakes @p model's `normalBake`. Throws if it has none. */@@ -223,7 +228,8 @@ export function bakeNormalMap(model: BakedModel): NormalMap {const { size, high, smoothAngle } = detail;const { min, max } = boundsOf(high);const distance = detail.distance ?? Math.hypot(max[0] - min[0], max[1] - min[1], max[2] - min[2]) * DEFAULT_DISTANCE;- const { corners, normals, colors } = denseSurface(high, smoothAngle);+ const fallback = detail.surface ?? { roughness: 1, metallic: 0 };+ const { corners, normals, colors, surfaces } = denseSurface(high, smoothAngle, fallback);const tree = new TriangleTree(corners);const mesh = model.data;@@ -236,6 +242,16 @@ export function bakeNormalMap(model: BakedModel): NormalMap {for (let i = 0; i < size * size; i++) { rgba[i * 4] = 128; rgba[i * 4 + 1] = 128; rgba[i * 4 + 2] = 255; rgba[i * 4 + 3] = 255; }const lowColors = detail.lowColors;const colorRgba = lowColors ? new Uint8Array(size * size * 4) : undefined;+ const surfaceRgba = detail.surface ? new Uint8Array(size * size * 4) : undefined;+ if (surfaceRgba) {+ // What the source says everything is, wherever no chart lands or no surface is found.+ for (let i = 0; i < size * size; i++) {+ surfaceRgba[i * 4] = 255;+ surfaceRgba[i * 4 + 1] = Math.round(fallback.roughness * 255);+ surfaceRgba[i * 4 + 2] = Math.round(fallback.metallic * 255);+ surfaceRgba[i * 4 + 3] = 255;+ }+ }const covered = new Uint8Array(size * size);const place = new Float64Array(2);let hits = 0, missed = 0, bend = 0;@@ -280,6 +296,7 @@ export function bakeNormalMap(model: BakedModel): NormalMap {}hits++;if (colorRgba) colorRgba.set(colors.subarray(hit * 4, hit * 4 + 4), texel * 4);+ if (surfaceRgba) { surfaceRgba[texel * 4 + 1] = surfaces[hit * 2]!; surfaceRgba[texel * 4 + 2] = surfaces[hit * 2 + 1]!; }const [hv, hw] = [place[0]!, place[1]!];const hu = 1 - hv - hw;@@ -312,7 +329,7 @@ export function bakeNormalMap(model: BakedModel): NormalMap {}}const inside = hits + missed;- dilate(colorRgba ? [rgba, colorRgba] : [rgba], covered, size);+ dilate([rgba, ...(colorRgba ? [colorRgba] : []), ...(surfaceRgba ? [surfaceRgba] : [])], covered, size);const from = corners.length / 9, to = mesh.indices.length / 3;const warnings: string[] = [];@@ -322,7 +339,7 @@ export function bakeNormalMap(model: BakedModel): NormalMap {if (inside / (size * size) < SPARSE_COVERAGE) warnings.push(`the surface broke into so many pieces that they fill only ${Math.round((inside / (size * size)) * 100)}% of the map, so most of its resolution is lost: lower the amplitude before the decimate, or keep more triangles`);if (from <= to) warnings.push(`the surface baked from has ${from} triangles and this one ${to}: there is no detail to carry. normalBake.from should name a step before the decimate`);return {- size, rgba, warnings, ...(colorRgba ? { colorRgba } : {}),+ size, rgba, warnings, ...(colorRgba ? { colorRgba } : {}), ...(surfaceRgba ? { surfaceRgba } : {}),report: {size, from, to,coverage: Math.round((inside / (size * size)) * 1000) / 1000,diff --git a/pipeline/src/modeling/polyMesh.ts b/pipeline/src/modeling/polyMesh.tsindex 9008b7773..74d6a30d9 100644--- a/pipeline/src/modeling/polyMesh.ts+++ b/pipeline/src/modeling/polyMesh.ts@@ -18,6 +18,10 @@ export interface Face {tag: string;/** Linear RGBA, as MeshRenderer.color is; absent draws white. */color?: readonly [number, number, number, number];+ /** How rough, 0..1. Absent takes the source's `material`. */+ roughness?: number;+ /** How much metal, 0..1. Absent takes the source's `material`. */+ metallic?: number;/** Part of a continuous skin: shaded smooth with its neighbours whatever the angle, and its edges are not features. */smooth?: boolean;/** How this face takes its texture, from a `uv` step; absent, a box projection at the source's uvScale. */diff --git a/pipeline/src/modeling/rig.ts b/pipeline/src/modeling/rig.tsindex aefa08189..ddbc58013 100644--- a/pipeline/src/modeling/rig.ts+++ b/pipeline/src/modeling/rig.ts@@ -930,7 +930,7 @@ export function timelineOf(animation: AnimationSource, joints: readonly RigJoint* The prefab that draws and moves it: the mesh on the root, the joints below it as* entities named so a clip's paths find them, and a player for the first clip.*/-export function prefabOf(name: string, meshPath: string, joints: readonly RigJoint[], cullBackfaces: boolean, firstClip: string | null, controller: string | null = null, extra: { goals?: readonly RigGoal[]; shapes?: number; normalMap?: string; texture?: string } = {}): Record<string, unknown> {+export function prefabOf(name: string, meshPath: string, joints: readonly RigJoint[], cullBackfaces: boolean, firstClip: string | null, controller: string | null = null, extra: { goals?: readonly RigGoal[]; shapes?: number; normalMap?: string; texture?: string; material?: string } = {}): Record<string, unknown> {const idOf = (i: number): string => `j${i}`;const goals = extra.goals ?? [];const entities: Record<string, unknown>[] = [{@@ -939,7 +939,7 @@ export function prefabOf(name: string, meshPath: string, joints: readonly RigJoivisible: true,components: [{ type: 'Transform', data: {} },- { type: 'MeshRenderer', data: { mesh: meshPath, ...(extra.normalMap ? { normalMap: extra.normalMap } : {}), ...(extra.texture ? { texture: extra.texture } : {}), lit: true, opaque: true, cullBackfaces, enabled: true } },+ { type: 'MeshRenderer', data: { mesh: meshPath, ...(extra.normalMap ? { normalMap: extra.normalMap } : {}), ...(extra.texture ? { texture: extra.texture } : {}), ...(extra.material ? { material: extra.material } : {}), lit: true, opaque: true, cullBackfaces, enabled: true } },...(joints.length ? [{ type: 'MeshSkin', data: { joints: joints.map((_, i) => idOf(i)) } }] : []),...(extra.shapes ? [{ type: 'MeshMorph', data: { weights: Array.from({ length: extra.shapes }, () => 0) } }] : []),// A controller decides what plays; a player beside it would fight it for the same joints.diff --git a/pipeline/src/modeling/stain.ts b/pipeline/src/modeling/stain.tsindex c17b0b691..581326c86 100644--- a/pipeline/src/modeling/stain.ts+++ b/pipeline/src/modeling/stain.ts@@ -17,7 +17,11 @@ export interface StainParams {near?: { at: Vec3; radius: number };hard?: boolean;/** Linear [r, g, b], over what is there. */- color: readonly number[];+ color?: readonly number[];+ /** How rough to make it, 0..1: moss and dust dull a surface, wear polishes it. */+ roughness?: number;+ /** How much metal to make it, 0..1: bare steel where the paint has worn off. */+ metallic?: number;/** How strongly, 0..1, where the rest say all the way. Default 1. */amount?: number;pattern?: DisplacePattern;@@ -101,8 +105,12 @@ function relief(mesh: PolyMesh, normals: Float64Array, reach: number | undefinedreturn out;}-export function stain(mesh: PolyMesh, p: StainParams, budget = MAX_DISPLACE_TRIANGLES): PolyMesh {- if (p.color.length < 3) throw new Error('color must be [r, g, b]');+export function stain(mesh: PolyMesh, p: StainParams, budget = MAX_DISPLACE_TRIANGLES, surfaceDefaults?: { roughness: number; metallic: number }): PolyMesh {+ if (p.color === undefined && p.roughness === undefined && p.metallic === undefined) throw new Error('give at least one of color, roughness, metallic');+ if (p.color !== undefined && p.color.length < 3) throw new Error('color must be [r, g, b]');+ for (const [name, v] of [['roughness', p.roughness], ['metallic', p.metallic]] as const) {+ if (v !== undefined && !(v >= 0 && v <= 1)) throw new Error(`${name} ${v} must be 0..1`);+ }if (p.pattern !== undefined && !DISPLACE_PATTERNS.includes(p.pattern)) throw new Error(`pattern ${JSON.stringify(p.pattern)} must be one of ${DISPLACE_PATTERNS.join(', ')}`);if (p.pattern !== undefined && !(p.scale! > 0)) throw new Error('a pattern needs a positive `scale`');if (p.amount !== undefined && !(p.amount >= 0 && p.amount <= 1)) throw new Error(`amount ${p.amount} must be 0..1`);@@ -141,7 +149,8 @@ export function stain(mesh: PolyMesh, p: StainParams, budget = MAX_DISPLACE_TRIAconst field = p.pattern ? patternField(asPattern) : null;const threshold = p.threshold ?? 0.5, softness = p.softness ?? 0.15;const amount = p.amount ?? 1;- const color = [p.color[0]!, p.color[1]!, p.color[2]!] as const;+ const color = p.color ? [p.color[0]!, p.color[1]!, p.color[2]!] as const : null;+ const defaults = surfaceDefaults ?? { roughness: 1, metallic: 0 };const maskAt = (v: number): number => {let m = 1;@@ -180,7 +189,12 @@ export function stain(mesh: PolyMesh, p: StainParams, budget = MAX_DISPLACE_TRIAconst m = (sum / face.v.length) * amount;if (m <= 0) return face;const was = face.color ?? [1, 1, 1, 1];- return { ...face, color: [was[0] + (color[0] - was[0]) * m, was[1] + (color[1] - was[1]) * m, was[2] + (color[2] - was[2]) * m, was[3]] as const };+ return {+ ...face,+ ...(color ? { color: [was[0] + (color[0] - was[0]) * m, was[1] + (color[1] - was[1]) * m, was[2] + (color[2] - was[2]) * m, was[3]] as const } : {}),+ ...(p.roughness !== undefined ? { roughness: (face.roughness ?? defaults.roughness) + (p.roughness - (face.roughness ?? defaults.roughness)) * m } : {}),+ ...(p.metallic !== undefined ? { metallic: (face.metallic ?? defaults.metallic) + (p.metallic - (face.metallic ?? defaults.metallic)) * m } : {}),+ };});const out = compact({ positions: fine.positions, faces });return short ? { ...out, notes: [`this stain wants edges of ${longest!.toPrecision(3)} and stopped at ${budget} triangles, so it is coarser than asked: narrow it with \`faces\` or \`near\`, or give a larger \`detail\``] } : out;diff --git a/pipeline/tests/modeling-surface.test.ts b/pipeline/tests/modeling-surface.test.tsnew file mode 100644index 000000000..9582242de--- /dev/null+++ b/pipeline/tests/modeling-surface.test.ts@@ -0,0 +1,120 @@+// SPDX-License-Identifier: Apache-2.0+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team+/** What a surface is made of: said by faces, kept by a map in the packing the engine's material reads, and named by a material. */+import { beforeAll, describe, expect, it } from 'vitest';+import { MeshChannel } from 'esengine';+import { applyModelEdits, bakeModel, bakeNormalMap, modelMaterialDoc, prepareModeling, validateModelSource, type BakedModel, type ModelStep } from '../src/modeling';++beforeAll(() => prepareModeling());++const ball: ModelStep = { id: 'form', op: 'sphere', radius: 1, rings: 32, sides: 64, tag: 'ball' };+const thin: ModelStep = { op: 'decimate', from: 'form', triangles: 300, error: 0.5 };+const SIZE = 128;++function surfaceAt(m: BakedModel, rgba: Uint8Array, triangle: number): { texel: number[]; at: number[] } {+ const { data } = m;+ const channel = (semantic: number) => data.channels.find((c) => c.semantic === semantic)!.offset;+ const view = new DataView(data.vertices.buffer, data.vertices.byteOffset, data.vertices.byteLength);+ const mean = (offset: number, k: number): number => [0, 1, 2].reduce((sum, c) => sum + view.getFloat32(data.indices[triangle * 3 + c]! * data.vertexStride + offset + k * 4, true), 0) / 3;+ const [u, v] = [mean(channel(MeshChannel.TexCoord0), 0), mean(channel(MeshChannel.TexCoord0), 1)];+ const i = (Math.floor((1 - v) * SIZE) * SIZE + Math.floor(u * SIZE)) * 4;+ return { texel: [...rgba.subarray(i, i + 4)], at: [0, 1, 2].map((k) => mean(channel(MeshChannel.Position), k)) };+}++describe('paint and stain say what a surface is made of', () => {+ it('paint sets them on what it names, and leaves colour alone', () => {+ const m = bakeModel({ version: 1, steps: [{ op: 'box', size: 1, color: [0.5, 0.5, 0.5] }, { op: 'paint', faces: '+y', roughness: 0.2, metallic: 1 }] });+ for (const f of m.poly.faces) {+ expect(f.color).toEqual([0.5, 0.5, 0.5, 1]);+ if (f.roughness !== undefined) expect([f.roughness, f.metallic]).toEqual([0.2, 1]);+ }+ expect(m.poly.faces.filter((f) => f.metallic === 1)).toHaveLength(1);+ });++ it('stain mixes toward them by `amount`, from what the source says everything is', () => {+ const m = bakeModel({ version: 1, material: { roughness: 0.8, metallic: 0 }, steps: [{ op: 'plane', size: 1 }, { op: 'stain', near: { at: [0, 0, 0], radius: 10 }, hard: true, amount: 0.5, roughness: 0.2, metallic: 1, detail: 5 }] });+ expect(m.poly.faces[0]!.roughness).toBeCloseTo(0.5, 6);+ expect(m.poly.faces[0]!.metallic).toBeCloseTo(0.5, 6);+ expect(m.poly.faces[0]!.color).toBeUndefined();+ });++ it('refuse a step that says nothing, and values out of range', () => {+ expect(() => bakeModel({ version: 1, steps: [ball, { op: 'paint', faces: 'all' }] })).toThrow(/at least one of color, roughness, metallic/);+ expect(() => bakeModel({ version: 1, steps: [ball, { op: 'stain', facing: [0, 1, 0] }] })).toThrow(/at least one of color, roughness, metallic/);+ expect(() => bakeModel({ version: 1, steps: [ball, { op: 'paint', faces: 'all', roughness: 2 }] })).toThrow(/roughness 2 must be 0..1/);+ expect(() => validateModelSource({ steps: [ball], material: { roughness: 2 } })).toThrow(/material.roughness must be 0..1/);+ expect(() => validateModelSource({ steps: [ball], material: { shiny: 1 } })).toThrow(/unknown field "shiny"/);+ });++ it('survive a decimate on the side they were on', () => {+ const m = bakeModel({ version: 1, steps: [ball, { op: 'stain', faces: { within: { min: [null, 0, null] } }, facing: [0, 1, 0], metallic: 1, threshold: 0 }, { op: 'decimate', ratio: 0.3, error: 0.2 }] });+ const metal = m.poly.faces.filter((f) => (f.metallic ?? 0) > 0.5);+ expect(metal.length).toBeGreaterThan(5);+ for (const f of metal) expect(f.v.reduce((sum, v) => sum + m.poly.positions[v]![1], 0) / 3).toBeGreaterThan(-0.2);+ });++ it('warn when nothing carries them, and say what would', () => {+ const m = bakeModel({ version: 1, steps: [ball, { op: 'paint', faces: 'all', metallic: 1 }] });+ expect(m.report.warnings.join()).toMatch(/nothing carries that: give normalBake `surface: true`, or say it once in `material`/);+ });+});++describe('normalBake surface', () => {+ it('packs as the material reads it: full red, roughness in green, metal in blue, and where it was painted', () => {+ const m = bakeModel({ version: 1, normalBake: { size: SIZE, surface: true }, steps: [ball, { op: 'stain', near: { at: [0, 1, 0], radius: 1.2 }, hard: true, roughness: 0.2, metallic: 1 }, thin] });+ const map = bakeNormalMap(m);+ expect(map.surfaceRgba).toBeDefined();+ const seen = { steel: 0, chalk: 0 };+ for (let t = 0; t < m.data.indices.length / 3; t++) {+ const { texel, at } = surfaceAt(m, map.surfaceRgba!, t);+ if (at[1]! > 0.5) { expect(texel).toEqual([255, 51, 255, 255]); seen.steel++; }+ if (at[1]! < -0.5) { expect(texel).toEqual([255, 255, 0, 255]); seen.chalk++; }+ }+ expect(seen.steel).toBeGreaterThan(3);+ expect(seen.chalk).toBeGreaterThan(3);+ });++ it('takes the source\'s `material` for faces that say nothing, and for texels no surface reached', () => {+ const steps: ModelStep[] = [{ op: 'plane', size: 4, at: [0, 5, 0] }, { id: 'away', op: 'transform', scale: 1 }, { op: 'delete', faces: 'all' }, { op: 'plane', size: 1 }];+ const missed = bakeModel({ version: 1, material: { roughness: 0.4, metallic: 0.5 }, normalBake: { size: SIZE, surface: true, from: 'away', distance: 0.1 }, steps });+ const map = bakeNormalMap(missed);+ expect(map.report.missed).toBe(1);+ expect(surfaceAt(missed, map.surfaceRgba!, 0).texel).toEqual([255, 102, 128, 255]);+ const hit = bakeModel({ version: 1, material: { roughness: 0.4, metallic: 0.5 }, normalBake: { size: SIZE, surface: true }, steps: [ball, thin] });+ expect(surfaceAt(hit, bakeNormalMap(hit).surfaceRgba!, 0).texel).toEqual([255, 102, 128, 255]);+ });++ it('is not written unless asked for', () => {+ const m = bakeModel({ version: 1, normalBake: { size: 64 }, steps: [ball, thin] });+ expect(bakeNormalMap(m).surfaceRgba).toBeUndefined();+ });++ it('`surface` must be true or false, and an edit can take a material away', () => {+ expect(() => validateModelSource({ steps: [ball, thin], normalBake: { surface: 1 } })).toThrow(/surface must be true or false/);+ const source = validateModelSource({ steps: [ball], material: { metallic: 1 } });+ expect(applyModelEdits(source, [{ op: 'settings', fields: { material: null as never } }]).material).toBeUndefined();+ });+});++describe('the material a model writes', () => {+ it('is none where a MeshRenderer says it all', () => {+ expect(modelMaterialDoc(validateModelSource({ steps: [ball] }), true, {})).toBeNull();+ expect(modelMaterialDoc(validateModelSource({ steps: [ball] }), true, { normal: 'a_normal.png' })).toBeNull();+ });++ it('holds constants, and the normal map a renderer would otherwise hold', () => {+ const doc = modelMaterialDoc(validateModelSource({ steps: [ball], material: { roughness: 0.3, metallic: 1 } }), true, { normal: 'a_normal.png' })!;+ expect(doc.shader).toBe('builtin:model');+ expect(doc.properties).toEqual({ u_normalMap: 'a_normal.png', u_roughness: 0.3, u_metallic: 1 });+ });++ it('leaves a map\'s values unscaled: the factors multiply it, so they are 1', () => {+ const doc = modelMaterialDoc(validateModelSource({ steps: [ball, thin], material: { roughness: 0.3, metallic: 0 }, normalBake: { surface: true } }), true, { surface: 'a_surface.png' })!;+ expect(doc.properties).toEqual({ u_metallicRoughnessMap: 'a_surface.png', u_roughness: 1, u_metallic: 1 });+ });++ it('culls as the surface allows', () => {+ const source = validateModelSource({ steps: [ball], material: { metallic: 1 } });+ expect(modelMaterialDoc(source, true, {})!.cull).not.toBe(modelMaterialDoc(source, false, {})!.cull);+ });+});diff --git a/tools/releaseNotes.mjs b/tools/releaseNotes.mjsindex 8844524b8..5723cbc02 100644--- a/tools/releaseNotes.mjs+++ b/tools/releaseNotes.mjs@@ -3880,6 +3880,12 @@ export const NOTED = {{"note":"An AI building a model can make it light and keep its detail."},"fix(mcp): a rotation given as a quaternion is that rotation, where its parts were read as degrees and its w dropped":{"note":"A rotation given to an agent as a quaternion is that rotation."},+ "fix(asset): a map a material names keeps its import settings in Play, where it was decoded as colour":+ {"note":"A material's normal and metal-rough maps are read as data in the editor's Play."},+ "feat(modeling): a model says how rough and how metal it is, part by part, and a bake keeps that as a map":+ {"note":"An AI building a model can say what it is made of."},+ "feat(mcp): model_write writes a material and a surface map, and a check draws the map against its constant":+ {"note":"An AI building a model can say what it is made of."},"feat(modeling): a stain step colours a surface by what is there, and a bake keeps it as a colour map":{"note":"An AI building a model can colour it finely, and keep that on a thin mesh."},"feat(mcp): model_write stains and bakes colour, checked in both colour spaces":