Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(forge): ARKit-named face shapes, teeth and tongue, named morphs, Loop subdivision, sparse morph targets

okanaslbde11e80f196
diff --git a/.changeset/forge-human-face.md b/.changeset/forge-human-face.md
new file mode 100644
index 00000000..8348e105
--- /dev/null
+++ b/.changeset/forge-human-face.md
@@ -0,0 +1,5 @@
+---
+"framefields": minor
+---
+
+Faces for `human()`: facial blend shapes under ARKit names (blinks, brows, jaw, smiles, frowns, pucker and more, split into sides), teeth and tongue that move with the jaw, per-layer `morphs` that drive any morph target by name, Loop subdivision (`subdivide`) that carries blend shapes and skin weights exactly, and sparse morph targets in written GLBs.
diff --git a/nodes/node-compositor/src/renderers/character.ts b/nodes/node-compositor/src/renderers/character.ts
index 597c04c3..7fb2f316 100644
--- a/nodes/node-compositor/src/renderers/character.ts
+++ b/nodes/node-compositor/src/renderers/character.ts
@@ -60,6 +60,27 @@ type CharacterNode = Pick<
| "drive"
>;
+/**
+ * A mesh's morph weights: the clip's (or the file's defaults), with any
+ * target the layer names in `morphs` set from its track. Names match
+ * case-insensitively against the mesh's `morphTargetNames`.
+ */
+export function namedMorphWeights(
+ names: readonly string[] | undefined,
+ base: number[],
+ morphs: Model3DNode["morphs"],
+ clock: TrackClock,
+): number[] {
+ if (!morphs || !names?.length) return base;
+ const out = [...base];
+ const index = new Map(names.map((n, i) => [n.toLowerCase(), i]));
+ for (const [name, track] of Object.entries(morphs)) {
+ const i = index.get(name.toLowerCase());
+ if (i !== undefined) out[i] = numberAt(track, clock, out[i] ?? 0);
+ }
+ return out;
+}
+
/** Whether a node needs the rig evaluator (anything beyond one plain clip). */
export function hasRig(n: CharacterNode): boolean {
return Boolean(
diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
index d79d684c..485b9fb0 100644
--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
@@ -93,6 +93,7 @@ import {
type CrowdInstanceAnim,
crowdBake,
crowdRecord,
+ namedMorphWeights,
paletteBuffer,
poseCharacter,
} from "./character.js";
@@ -4031,7 +4032,12 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const meshOpts: DrawMesh3DOpts = {
modelMatrix: placed.matrix,
- morphWeights: sampleMorphWeights(mesh, clip, clipSec),
+ morphWeights: namedMorphWeights(
+ mesh.morphTargetNames,
+ sampleMorphWeights(mesh, clip, clipSec),
+ target?.morphs ?? lop.morphs,
+ { frame: shiftedFrame ?? 0, fps: fps || 30 },
+ ),
opacity: modelOpacity * (mat?.opacity ?? 1.0),
colorTint: meshColorTint,
transparent:
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index 220f7595..324c73e1 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -40,6 +40,8 @@ import {
LookAtSchema,
type ModelAnimationLayer,
ModelAnimationLayerSchema,
+ type NumberTrack,
+ NumberTrackSchema,
type SpringBoneChain,
SpringBoneChainSchema,
type TwoBoneIkConstraint,
@@ -1855,6 +1857,13 @@ export interface Model3DNode {
springBones?: SpringBoneChain[];
/** Joints posed directly by name (local Euler degrees, offsets), over the clips. */
pose?: Record<string, JointPose>;
+ /**
+ * Blend shape weights by target name (glTF `extras.targetNames`; case is
+ * ignored): `{ eyeBlinkLeft: blink, jawOpen: speech }`. Each value is a
+ * number, `[[frame, v]]` keys, `(frame) => v` or a signal; it replaces the
+ * clip's weight for that target.
+ */
+ morphs?: Record<string, NumberTrack>;
/** Bones pointed along scene directions, parents first (pose retargeting). */
aim?: AimConstraint[];
/** Joints driven by other joints, on top of any the model file declares (eyelids, twist bones). */
@@ -2195,6 +2204,7 @@ export const Model3DNodeSchema: z.ZodType<Model3DNode> = z
lookAt: z.array(LookAtSchema).optional(),
springBones: z.array(SpringBoneChainSchema).optional(),
pose: z.record(z.string(), JointPoseSchema).optional(),
+ morphs: z.record(z.string(), NumberTrackSchema).optional(),
aim: z.array(AimSchema).optional(),
drive: z.array(JointDriverSchema).optional(),
crowd: CrowdSchema.optional(),
diff --git a/packages/forge/src/document.ts b/packages/forge/src/document.ts
index 59bc0d93..b90db588 100644
--- a/packages/forge/src/document.ts
+++ b/packages/forge/src/document.ts
@@ -349,17 +349,17 @@ export class GltfDocument {
if (mode !== 4) json.mode = mode;
if (p.targets?.length) {
json.targets = p.targets.map((t) => {
+ const name = `${label} target ${t.name}`;
const target: Record<string, number> = {
- POSITION: this.bin.accessor(
- floats(`${label} target ${t.name}`, t.positions, 3, count),
- "VEC3",
- { bounds: true, target: "vertex" },
+ POSITION: this.targetAccessor(
+ floats(name, t.positions, 3, count),
+ true,
),
};
if (t.normals)
- target.NORMAL = vertex(
- floats(`${label} target ${t.name} normals`, t.normals, 3, count),
- "VEC3",
+ target.NORMAL = this.targetAccessor(
+ floats(`${name} normals`, t.normals, 3, count),
+ false,
);
return target;
});
@@ -367,6 +367,30 @@ export class GltfDocument {
return { json, maxJoint };
}
+ /** A morph target's deltas, sparse when they move under half the vertices. */
+ private targetAccessor(deltas: Float32Array, bounds: boolean): number {
+ const count = deltas.length / 3;
+ const moved: number[] = [];
+ for (let v = 0; v < count; v++)
+ if (deltas[v * 3] || deltas[v * 3 + 1] || deltas[v * 3 + 2])
+ moved.push(v);
+ if (moved.length === 0 || moved.length * 2 >= count)
+ return this.bin.accessor(deltas, "VEC3", { bounds, target: "vertex" });
+ const values = new Float32Array(moved.length * 3);
+ moved.forEach((v, i) => {
+ values.set(deltas.subarray(v * 3, v * 3 + 3), i * 3);
+ });
+ return this.bin.sparseAccessor(
+ count,
+ "VEC3",
+ Uint32Array.from(moved),
+ values,
+ {
+ bounds,
+ },
+ );
+ }
+
addNode(spec: NodeSpec): NodeRef {
const index = this.nodes.length;
if (spec.skin && !spec.mesh)
diff --git a/packages/forge/src/forge.test.ts b/packages/forge/src/forge.test.ts
index dd38ed5b..7c181a6e 100644
--- a/packages/forge/src/forge.test.ts
+++ b/packages/forge/src/forge.test.ts
@@ -162,6 +162,50 @@ describe("GltfDocument", () => {
);
});
+ it("stores morph targets that move few vertices as sparse accessors", async () => {
+ const s = strip(20, 100);
+ const n = s.positions.length / 3;
+ const poke = new Array<number>(n * 3).fill(0);
+ poke[7 * 3 + 2] = 4; // one vertex out along +z
+ const doc = new GltfDocument();
+ doc.addNode({
+ mesh: doc.addMesh({
+ primitives: [
+ {
+ ...s,
+ targets: [
+ {
+ name: "poke",
+ positions: poke,
+ normals: new Array(n * 3).fill(0),
+ },
+ ],
+ },
+ ],
+ }),
+ });
+ const glb = doc.toGlb();
+ const report = await validate(glb);
+ expect(report.issues).toEqual([]);
+ const jsonLength = new DataView(glb.buffer, glb.byteOffset).getUint32(
+ 12,
+ true,
+ );
+ const json = JSON.parse(
+ new TextDecoder().decode(glb.subarray(20, 20 + jsonLength)),
+ ) as {
+ meshes: { primitives: { targets: { POSITION: number }[] }[] }[];
+ accessors: { sparse?: { count: number }; max?: number[] }[];
+ };
+ const acc =
+ json.accessors[json.meshes[0]!.primitives[0]!.targets[0]!.POSITION]!;
+ expect(acc.sparse?.count).toBe(1);
+ expect(acc.max).toEqual([0, 0, 4]);
+ const m = parseGLB(glb).meshes[0]!;
+ expect(Array.from(m.morphTargets![0]!.positions!)).toEqual(poke);
+ expect(m.morphTargetNames).toEqual(["poke"]);
+ });
+
it("writes every material extension in a valid form", async () => {
const doc = new GltfDocument();
const material = doc.addMaterial({
diff --git a/packages/forge/src/glb-builder.ts b/packages/forge/src/glb-builder.ts
index b2051498..baf0e57c 100644
--- a/packages/forge/src/glb-builder.ts
+++ b/packages/forge/src/glb-builder.ts
@@ -110,6 +110,55 @@ export class GlbBuilder {
return this.accessors.length - 1;
}
+ /**
+ * A float accessor of `count` elements that are zero except at
+ * `indices` (strictly increasing), which take `values`: glTF's sparse
+ * storage, for morph targets that move a few vertices of many.
+ */
+ sparseAccessor(
+ count: number,
+ type: AccessorType,
+ indices: Uint32Array,
+ values: Float32Array,
+ options: { bounds?: boolean } = {},
+ ): number {
+ const size = COMPONENTS[type];
+ if (values.length !== indices.length * size)
+ throw new Error("sparse accessor: values don't match the indices");
+ const idx = count <= 65536 ? Uint16Array.from(indices) : indices;
+ const accessor: Record<string, unknown> = {
+ componentType: 5126,
+ count,
+ type,
+ sparse: {
+ count: indices.length,
+ indices: {
+ bufferView: this.view(
+ new Uint8Array(idx.buffer, idx.byteOffset, idx.byteLength),
+ ),
+ componentType: componentType(idx),
+ },
+ values: {
+ bufferView: this.view(
+ new Uint8Array(values.buffer, values.byteOffset, values.byteLength),
+ ),
+ },
+ },
+ };
+ if (options.bounds) {
+ const b = bounds(values, size);
+ // The elements outside `indices` are zero.
+ if (indices.length < count)
+ for (let c = 0; c < size; c++) {
+ b.min[c] = Math.min(b.min[c] as number, 0);
+ b.max[c] = Math.max(b.max[c] as number, 0);
+ }
+ Object.assign(accessor, b);
+ }
+ this.accessors.push(accessor);
+ return this.accessors.length - 1;
+ }
+
/** Raw bytes (an image) as a buffer view. Returns its index. */
image(bytes: Uint8Array): number {
return this.view(bytes);
diff --git a/packages/forge/src/human/add-human.ts b/packages/forge/src/human/add-human.ts
index 122df2fd..aecad25f 100644
--- a/packages/forge/src/human/add-human.ts
+++ b/packages/forge/src/human/add-human.ts
@@ -10,7 +10,7 @@ import type {
SkeletonRefs,
} from "../document.js";
import type { MaterialSpec } from "../material.js";
-import type { HumanModel } from "./human.js";
+import type { HumanModel, HumanPart } from "./human.js";
export interface AddHumanOptions {
/** Node name. Default "Human". */
@@ -21,12 +21,19 @@ export interface AddHumanOptions {
eyes?: MaterialSpec;
/** Cornea material. Default: clear glass (IOR 1.376, 3 mm deep). */
cornea?: MaterialSpec;
+ /** Teeth and tongue material. Default: a wet, slightly translucent ivory. */
+ mouth?: MaterialSpec;
}
export interface AddedHuman {
rig: SkeletonRefs;
- nodes: { body: NodeRef; eyes: NodeRef; cornea: NodeRef };
- materials: { skin: MaterialRef; eyes: MaterialRef; cornea: MaterialRef };
+ nodes: { body: NodeRef; eyes: NodeRef; cornea: NodeRef; mouth: NodeRef };
+ materials: {
+ skin: MaterialRef;
+ eyes: MaterialRef;
+ cornea: MaterialRef;
+ mouth: MaterialRef;
+ };
}
export function addHuman(
@@ -63,11 +70,15 @@ export function addHuman(
volume: { thickness: 0.3 },
},
);
- const part = (
- part: string,
- primitive: HumanModel["body"],
- material: MaterialRef,
- ) =>
+ const mouth = doc.addMaterial(
+ options.mouth ?? {
+ name: `${name}Mouth`,
+ color: "#e8dccb",
+ roughness: 0.3,
+ subsurface: { factor: 0.5, color: "#ff8070" },
+ },
+ );
+ const part = (part: string, primitive: HumanPart, material: MaterialRef) =>
doc.addNode({
name: `${name}${part}`,
mesh: doc.addMesh({
@@ -80,7 +91,8 @@ export function addHuman(
body: part("Body", h.body, skin),
eyes: part("Eyes", h.eyes, eyes),
cornea: part("Cornea", h.cornea, cornea),
+ mouth: part("Mouth", h.mouth, mouth),
};
for (const d of h.drivers) doc.addDriver(d);
- return { rig, nodes, materials: { skin, eyes, cornea } };
+ return { rig, nodes, materials: { skin, eyes, cornea, mouth } };
}
diff --git a/packages/forge/src/human/expressions.ts b/packages/forge/src/human/expressions.ts
new file mode 100644
index 00000000..bff4170b
--- /dev/null
+++ b/packages/forge/src/human/expressions.ts
@@ -0,0 +1,65 @@
+/**
+ * Facial blend shapes under ARKit's names, built from MakeHuman's CC0
+ * expression units (one set per ethnic base, blended like the body). Units
+ * made for one side map directly; symmetric mouth units are split into a
+ * left and a right half with a soft seam across the midline, so each side
+ * of a smile or a frown moves on its own. A face animated with ARKit names
+ * (capture apps, other rigs) drives these unchanged.
+ */
+
+/** One ARKit-named shape: MakeHuman units (path under `expression/units/<ethnicity>/`) and the half it keeps. */
+interface ShapeSource {
+ units: string[];
+ /** Keep the character's left (+x) or right (−x) half of a symmetric unit. */
+ half?: "left" | "right";
+}
+
+const sided = (name: string, unit: (side: "left" | "right") => string) => ({
+ [`${name}Left`]: { units: [unit("left")] },
+ [`${name}Right`]: { units: [unit("right")] },
+});
+const split = (name: string, unit: string) => ({
+ [`${name}Left`]: { units: [unit], half: "left" as const },
+ [`${name}Right`]: { units: [unit], half: "right" as const },
+});
+
+/** The shapes `human()` builds, by ARKit name (plus a few MakeHuman extras with descriptive names). */
+export const FACE_SHAPES: Readonly<Record<string, ShapeSource>> = {
+ ...sided("eyeBlink", (s) => `eye-${s}-closure`),
+ ...sided("eyeWide", (s) => `eye-${s}-opened-up`),
+ ...sided("eyeSquint", (s) => `eye-${s}-slit`),
+ ...sided("browDown", (s) => `eyebrows-${s}-down`),
+ ...sided("browOuterUp", (s) => `eyebrows-${s}-extern-up`),
+ browInnerUp: { units: ["eyebrows-left-inner-up", "eyebrows-right-inner-up"] },
+ ...sided("noseSneer", (s) => `nose-${s}-elevation`),
+ jawOpen: { units: ["mouth-open"] },
+ mouthPucker: { units: ["mouth-pursing"] },
+ mouthFunnel: { units: ["mouth-protusion"] },
+ mouthRollLower: { units: ["mouth-eversion"] },
+ ...split("mouthSmile", "mouth-corner-puller"),
+ ...split("mouthFrown", "mouth-depression"),
+ ...split("mouthPress", "mouth-compression"),
+ ...split("mouthStretch", "mouth-retraction"),
+ ...split("mouthUpperUp", "mouth-elevation"),
+ ...split("mouthLowerDown", "mouth-depression-retraction"),
+ ...split("mouthDimple", "mouth-upward-retraction"),
+ // Not in ARKit's set.
+ ...sided("noseFlare", (s) => `nose-${s}-dilatation`),
+ noseWrinkle: { units: ["nose-compression"] },
+ mouthPart: { units: ["mouth-parling"] },
+ neckTense: { units: ["neck-platysma"] },
+};
+
+/** Weight of the kept half at x (base mesh units, decimeters): a seam 2 cm wide across the midline. */
+export function halfWeight(
+ half: "left" | "right" | undefined,
+ x: number,
+): number {
+ if (!half) return 1;
+ const w = 0.1;
+ const t = Math.min(1, Math.max(0, (x + w) / (2 * w)));
+ const s = t * t * (3 - 2 * t);
+ return half === "left" ? s : 1 - s;
+}
+
+export const ETHNIC_UNITS = ["african", "asian", "caucasian"] as const;
diff --git a/packages/forge/src/human/human.test.ts b/packages/forge/src/human/human.test.ts
index aef8cb50..6c3eb247 100644
--- a/packages/forge/src/human/human.test.ts
+++ b/packages/forge/src/human/human.test.ts
@@ -197,6 +197,52 @@ describe.skipIf(!cached)("human (cached MakeHuman assets)", () => {
expect(Array.from(h.eyeTexture.slice(1, 4))).toEqual([0x50, 0x4e, 0x47]);
});
+ it("builds ARKit-named face shapes, split into sides, on the body and mouth", async () => {
+ const h = await human({});
+ const byName = new Map(h.body.targets.map((t) => [t.name, t.positions]));
+ for (const name of [
+ "eyeBlinkLeft",
+ "eyeBlinkRight",
+ "jawOpen",
+ "mouthSmileLeft",
+ "mouthSmileRight",
+ "browInnerUp",
+ ])
+ expect(byName.has(name)).toBe(true);
+ // Each side's shape moves its own side: the character's left is +x.
+ const sideOf = (d: ArrayLike<number>) => {
+ let left = 0;
+ let right = 0;
+ for (let v = 0; v < d.length / 3; v++) {
+ const m = Math.hypot(
+ d[v * 3] as number,
+ d[v * 3 + 1] as number,
+ d[v * 3 + 2] as number,
+ );
+ if ((h.body.positions[v * 3] as number) > 0) left += m;
+ else right += m;
+ }
+ return { left, right };
+ };
+ for (const shape of ["eyeBlink", "mouthSmile"]) {
+ const l = sideOf(byName.get(`${shape}Left`)!);
+ const r = sideOf(byName.get(`${shape}Right`)!);
+ expect(l.left).toBeGreaterThan(l.right * 5);
+ expect(r.right).toBeGreaterThan(r.left * 5);
+ }
+ // The split halves of a symmetric unit add back up to the whole unit.
+ const l = byName.get("mouthSmileLeft")!;
+ const r = byName.get("mouthSmileRight")!;
+ const sum = Array.from(l, (x, i) => x + (r[i] as number));
+ expect(sum.some((x) => Math.abs(x) > 0.05)).toBe(true);
+ // The jaw opens the teeth too; the mouth carries only shapes that move it.
+ const mouth = new Map(h.mouth.targets.map((t) => [t.name, t.positions]));
+ expect(
+ Array.from(mouth.get("jawOpen")!).some((d) => Math.abs(d) > 0.1),
+ ).toBe(true);
+ expect(mouth.has("eyeBlinkLeft")).toBe(false);
+ });
+
it("adds named targets on top of the macro shape", async () => {
const plain = await human({});
const shaped = await human({ targets: { "nose/nose-base-up": 1 } });
diff --git a/packages/forge/src/human/human.ts b/packages/forge/src/human/human.ts
index 503a219e..5d434a37 100644
--- a/packages/forge/src/human/human.ts
+++ b/packages/forge/src/human/human.ts
@@ -19,8 +19,14 @@
* doc.addNode({ mesh: doc.addMesh({ name: "Body", primitives: [{ ...elf.body, material: skin }] }), skin: rig.skin });
* ```
*/
-import type { DriverSpec, PrimitiveSpec } from "../document.js";
+import type {
+ DriverSpec,
+ MorphTargetSpec,
+ PrimitiveSpec,
+} from "../document.js";
import { Skeleton, type Vec3 } from "../skeleton.js";
+import { subdivide } from "../subdivide.js";
+import { ETHNIC_UNITS, FACE_SHAPES, halfWeight } from "./expressions.js";
import {
type BaseMesh,
fitProxy,
@@ -56,37 +62,48 @@ export interface HumanOptions {
* weights: `{ "nose/nose-base-up": 0.6, "chin/chin-jut-in": 0.3 }`.
*/
targets?: Record<string, number>;
+ /**
+ * Loop subdivision levels for the body and mouth (each multiplies the
+ * triangles by four; blend shapes and weights carry through exactly).
+ * 1 smooths faceting in close-ups. Default 0.
+ */
+ subdivide?: number;
+ /**
+ * Facial blend shapes on the body under ARKit names (`eyeBlinkLeft`,
+ * `jawOpen`, `mouthSmileRight`, …; see `FACE_SHAPES`). Default true.
+ */
+ expressions?: boolean;
/** Where the assets come from (default: a store on the shared cache). */
store?: HumanAssetStore;
}
+/** A skinned, UV-mapped part of a human. */
+export type HumanPart = Required<
+ Pick<
+ PrimitiveSpec,
+ "positions" | "normals" | "uvs" | "joints" | "weights" | "indices"
+ >
+>;
+
export interface HumanModel {
skeleton: Skeleton;
- /** The skinned body, ready for `GltfDocument.addMesh` (add a material). */
- body: Required<
- Pick<
- PrimitiveSpec,
- "positions" | "normals" | "uvs" | "joints" | "weights" | "indices"
- >
- >;
+ /** The skinned body with its facial blend shapes, ready for `GltfDocument.addMesh` (add a material). */
+ body: HumanPart & { targets: MorphTargetSpec[] };
+ /** Teeth and tongue, skinned to the head and jaw, with the expression shapes that move them. */
+ mouth: HumanPart & { targets: MorphTargetSpec[] };
/**
* Both eyeballs (sclera, iris, pupil), fitted to the shaped head and
* skinned to `LeftEye` / `RightEye`, so `lookAt` on those joints aims
* them. Give them a material with `eyeTexture` as its color texture.
*/
- eyes: Required<
- Pick<
- PrimitiveSpec,
- "positions" | "normals" | "uvs" | "joints" | "weights" | "indices"
- >
- >;
+ eyes: HumanPart;
/**
* The clear corneas over the irises, skinned like `eyes`. Give them a
* transmissive material (`transmission`, `ior` ≈ 1.376, `volume.thickness`
* ≈ 0.3 cm, the depth of the chamber in front of the iris) and they
* refract the iris: it shifts with the viewing angle like a real eye's.
*/
- cornea: HumanModel["eyes"];
+ cornea: HumanPart;
/** MakeHuman's brown eye texture (PNG, CC0): sclera, iris and pupil, mapped to `eyes`' UVs. */
eyeTexture: Uint8Array;
/**
@@ -353,6 +370,7 @@ export async function human(options: HumanOptions = {}): Promise<HumanModel> {
(shaped[v * 3 + 2] as number) * scale,
];
const positions = src.flatMap((v) => at(v));
+ const bodyNormals = vertexNormals(positions, indices, src);
// Skeleton: each bone's head is the mean of its joint group on the shaped body.
const skeleton = new Skeleton();
@@ -400,28 +418,104 @@ export async function human(options: HumanOptions = {}): Promise<HumanModel> {
l.push([j, w]);
}
});
- const joints = new Array<number>(src.length * 4).fill(0);
- const wts = new Array<number>(src.length * 4).fill(0);
- src.forEach((v, i) => {
- const l = (perVertex.get(v) ?? [[0, 1]])
- .sort((a, b) => b[1] - a[1])
- .slice(0, 4);
- const sum = l.reduce((s, x) => s + x[1], 0) || 1;
- l.forEach(([j, w], k) => {
- joints[i * 4 + k] = j;
- wts[i * 4 + k] = w / sum;
+ /** Top four painted weights per vertex, normalized. */
+ const skinOf = (list: number[]) => {
+ const joints = new Array<number>(list.length * 4).fill(0);
+ const weights = new Array<number>(list.length * 4).fill(0);
+ list.forEach((v, i) => {
+ const l = (perVertex.get(v) ?? [[0, 1]])
+ .sort((a, b) => b[1] - a[1])
+ .slice(0, 4);
+ const sum = l.reduce((t, x) => t + x[1], 0) || 1;
+ l.forEach(([j, w], k) => {
+ joints[i * 4 + k] = j;
+ weights[i * 4 + k] = w / sum;
+ });
});
- });
+ return { joints, weights };
+ };
+ const { joints, weights: wts } = skinOf(src);
- return {
+ // Teeth and tongue: helper geometry of the base mesh, skinned and shaped like the body.
+ const mouthFaces = [...base.helpers.values()].flat();
+ const mouthSrc: number[] = [];
+ const mouthUvs: number[] = [];
+ const mouthIndices: number[] = [];
+ const mouthCorner = new Map<number, number>();
+ for (const f of mouthFaces) {
+ const c = f.v.map((v, k) => {
+ const vt = f.vt[k] as number;
+ const key = v * nUv + vt;
+ let i = mouthCorner.get(key);
+ if (i === undefined) {
+ i = mouthSrc.length;
+ mouthCorner.set(key, i);
+ mouthSrc.push(v);
+ mouthUvs.push(base.uvs[vt * 2] ?? 0, 1 - (base.uvs[vt * 2 + 1] ?? 0));
+ }
+ return i;
+ });
+ for (let k = 1; k + 1 < c.length; k++)
+ mouthIndices.push(c[0] as number, c[k] as number, c[k + 1] as number);
+ }
+ const mouthPositions = mouthSrc.flatMap((v) => at(v));
+ const mouthNormals = vertexNormals(mouthPositions, mouthIndices, mouthSrc);
+ const mouthSkin = skinOf(mouthSrc);
+ const expressions = options.expressions !== false;
+ const ethnic = levels(options).ethnic;
+
+ const subdivisions = Math.max(0, Math.floor(options.subdivide ?? 0));
+ const smooth = <T extends HumanPart & { targets: MorphTargetSpec[] }>(
+ part: T,
+ weld: number[],
+ ): T =>
+ subdivisions
+ ? { ...part, ...(subdivide(part, { levels: subdivisions, weld }) as T) }
+ : part;
+ const model = {
skeleton,
body: {
positions,
- normals: vertexNormals(positions, indices, src),
+ normals: bodyNormals,
uvs,
joints,
weights: wts,
indices,
+ targets: expressions
+ ? await faceTargets(
+ store,
+ ethnic,
+ base.positions,
+ src,
+ scale,
+ positions,
+ indices,
+ bodyNormals,
+ )
+ : [],
+ },
+ mouth: {
+ positions: mouthPositions,
+ normals: mouthNormals,
+ uvs: mouthUvs,
+ joints: mouthSkin.joints,
+ weights: mouthSkin.weights,
+ indices: mouthIndices,
+ // Only the shapes that move the teeth or tongue (jaw, lips).
+ targets: expressions
+ ? (
+ await faceTargets(
+ store,
+ ethnic,
+ base.positions,
+ mouthSrc,
+ scale,
+ mouthPositions,
+ mouthIndices,
+ mouthNormals,
+ )
+ ).filter((t) => Array.from(t.positions).some((d) => d !== 0))
+ : [],
},
eyes: eyeballs(eyes.obj, fittedEyes, toModel, skeleton, false),
cornea: eyeballs(eyes.obj, fittedEyes, toModel, skeleton, true),
@@ -429,6 +523,12 @@ export async function human(options: HumanOptions = {}): Promise<HumanModel> {
drivers: LID_DRIVERS,
height: natural * (scale / 10),
};
+ // Copies of a base vertex at UV seams weld into one surface point.
+ return {
+ ...model,
+ body: smooth(model.body, src),
+ mouth: smooth(model.mouth, mouthSrc),
+ };
}
/** The eye mesh fitted to the shaped head, with each vertex's shell marked. */
@@ -553,7 +653,7 @@ function eyeballs(
toModel: (x: number, y: number, z: number) => Vec3,
skeleton: Skeleton,
cornea: boolean,
-): HumanModel["eyes"] {
+): HumanPart {
const leftEye = skeleton.joint("LeftEye").index;
const rightEye = skeleton.joint("RightEye").index;
const corner = new Map<number, number>();
@@ -647,3 +747,57 @@ function vertexNormals(
return [n[0] / l, n[1] / l, n[2] / l];
});
}
+
+/**
+ * The facial blend shapes on the body: each ARKit-named shape sums its
+ * MakeHuman units (blended by ethnicity, halved for one side where the
+ * unit is symmetric) into model units, with normal deltas so the shaded
+ * surface follows the shape.
+ */
+async function faceTargets(
+ store: HumanAssetStore,
+ ethnic: Record<string, number>,
+ basePositions: Float32Array,
+ src: number[],
+ scale: number,
+ positions: number[],
+ indices: number[],
+ normals: number[],
+): Promise<MorphTargetSpec[]> {
+ const bodyOf = new Map<number, number[]>();
+ src.forEach((v, i) => {
+ const list = bodyOf.get(v);
+ if (list) list.push(i);
+ else bodyOf.set(v, [i]);
+ });
+ const n = src.length;
+ const shapes = await Promise.all(
+ Object.entries(FACE_SHAPES).map(async ([name, source]) => {
+ const delta = new Float32Array(n * 3);
+ for (const unit of source.units)
+ for (const race of ETHNIC_UNITS) {
+ const we = ethnic[race] ?? 0;
+ if (we <= 0) continue;
+ const t = await target(store, `expression/units/${race}/${unit}`);
+ for (let k = 0; k < t.indices.length; k++) {
+ const v = t.indices[k] as number;
+ const w =
+ we *
+ scale *
+ halfWeight(source.half, basePositions[v * 3] as number);
+ if (!w) continue;
+ for (const i of bodyOf.get(v) ?? [])
+ for (let a = 0; a < 3; a++)
+ delta[i * 3 + a] =
+ (delta[i * 3 + a] as number) +
+ (t.deltas[k * 3 + a] as number) * w;
+ }
+ }
+ const moved = positions.map((p, i) => p + (delta[i] as number));
+ const shaped = vertexNormals(moved, indices, src);
+ const normalDelta = shaped.map((v, i) => v - (normals[i] as number));
+ return { name, positions: delta, normals: normalDelta };
+ }),
+ );
+ return shapes;
+}
diff --git a/packages/forge/src/human/makehuman.ts b/packages/forge/src/human/makehuman.ts
index 2bc37464..e49949d8 100644
--- a/packages/forge/src/human/makehuman.ts
+++ b/packages/forge/src/human/makehuman.ts
@@ -12,12 +12,22 @@ export interface BaseMesh {
uvs: Float32Array;
/** Faces of the "body" group: per corner, vertex and UV index. */
faces: { v: number[]; vt: number[] }[];
+ /** Faces of the helper groups the body uses (teeth, tongue), by group name. */
+ helpers: Map<string, { v: number[]; vt: number[] }[]>;
}
+/** Helper geometry that renders: the teeth and tongue inside the mouth. */
+export const MOUTH_GROUPS = new Set([
+ "helper-upper-teeth",
+ "helper-lower-teeth",
+ "helper-tongue",
+]);
+
export function parseBaseObj(text: string): BaseMesh {
const positions: number[] = [];
const uvs: number[] = [];
const faces: { v: number[]; vt: number[] }[] = [];
+ const helpers = new Map<string, { v: number[]; vt: number[] }[]>();
let group = "";
for (const line of text.split("\n")) {
if (line.startsWith("v ")) {
@@ -27,18 +37,28 @@ export function parseBaseObj(text: string): BaseMesh {
const [, u, v] = line.trim().split(/\s+/);
uvs.push(Number(u), Number(v));
} else if (line.startsWith("g ")) group = line.slice(2).trim();
- else if (line.startsWith("f ") && group === "body") {
+ else if (
+ line.startsWith("f ") &&
+ (group === "body" || MOUTH_GROUPS.has(group))
+ ) {
const corners = line.trim().split(/\s+/).slice(1);
- faces.push({
+ const face = {
v: corners.map((c) => Number(c.split("/")[0]) - 1),
vt: corners.map((c) => Number(c.split("/")[1]) - 1),
- });
+ };
+ if (group === "body") faces.push(face);
+ else {
+ const list = helpers.get(group) ?? [];
+ list.push(face);
+ helpers.set(group, list);
+ }
}
}
return {
positions: Float32Array.from(positions),
uvs: Float32Array.from(uvs),
faces,
+ helpers,
};
}
diff --git a/packages/forge/src/index.ts b/packages/forge/src/index.ts
index 22569d9c..b93834a3 100644
--- a/packages/forge/src/index.ts
+++ b/packages/forge/src/index.ts
@@ -37,7 +37,13 @@ export {
type AddHumanOptions,
addHuman,
} from "./human/add-human.js";
-export { type HumanModel, type HumanOptions, human } from "./human/human.js";
+export { FACE_SHAPES } from "./human/expressions.js";
+export {
+ type HumanModel,
+ type HumanOptions,
+ type HumanPart,
+ human,
+} from "./human/human.js";
export { MAKEHUMAN_SOURCE } from "./human/source.js";
export {
HumanAssetStore,
@@ -85,3 +91,8 @@ export {
Skeleton,
type Vec3,
} from "./skeleton.js";
+export {
+ type SubdivideOptions,
+ smoothNormals,
+ subdivide,
+} from "./subdivide.js";
diff --git a/packages/forge/src/subdivide.test.ts b/packages/forge/src/subdivide.test.ts
new file mode 100644
index 00000000..3df95815
--- /dev/null
+++ b/packages/forge/src/subdivide.test.ts
@@ -0,0 +1,86 @@
+/**
+ * Loop subdivision splits each triangle in four, relaxes the surface, and
+ * carries what rides on it exactly: a blend shape that translates the
+ * whole surface still translates it, skin weights stay normalized, and
+ * welded seam copies end up on one point.
+ */
+import { describe, expect, it } from "vitest";
+import { subdivide } from "./index.js";
+
+/** An octahedron with one vertex split at a UV seam (vertex 6 copies 0). */
+function octahedron() {
+ const positions = [
+ 1, 0, 0, -1, 0, 0, 0, 1, 0, 0, -1, 0, 0, 0, 1, 0, 0, -1, 1, 0, 0,
+ ];
+ const indices = [
+ 0, 2, 4, 2, 1, 4, 1, 3, 4, 3, 6, 4, 2, 0, 5, 1, 2, 5, 3, 1, 5, 6, 3, 5,
+ ];
+ const n = positions.length / 3;
+ const uvs = Array.from({ length: n * 2 }, (_, i) => (i === 12 ? 1 : 0));
+ const joints = Array.from({ length: n * 4 }, (_, i) => (i % 4 === 1 ? 1 : 0));
+ const weights = Array.from({ length: n }, (_, v) =>
+ v % 2 ? [0.25, 0.75, 0, 0] : [1, 0, 0, 0],
+ ).flat();
+ weights.splice(24, 4, 1, 0, 0, 0); // the seam copy weighs like vertex 0
+ return {
+ positions,
+ indices,
+ uvs,
+ joints,
+ weights,
+ weld: [0, 1, 2, 3, 4, 5, 0],
+ };
+}
+
+describe("subdivide", () => {
+ it("quadruples the triangles and moves the surface toward its limit", () => {
+ const o = octahedron();
+ const s = subdivide({ ...o }, { weld: o.weld });
+ expect(s.indices?.length).toBe(o.indices.length * 4);
+ // Corners pull in toward the smooth limit surface.
+ const r = Math.hypot(...Array.from(s.positions).slice(0, 3));
+ expect(r).toBeLessThan(1);
+ expect(r).toBeGreaterThan(0.5);
+ });
+
+ it("welds seam copies into one point, and keeps their own UVs", () => {
+ const o = octahedron();
+ const s = subdivide({ ...o }, { weld: o.weld, levels: 2 });
+ const p = Array.from(s.positions);
+ const u = Array.from(s.uvs ?? []);
+ const n = p.length / 3;
+ const at = (v: number) => p.slice(v * 3, v * 3 + 3);
+ let seamPairs = 0;
+ for (let a = 0; a < n; a++)
+ for (let b = a + 1; b < n; b++) {
+ const d = Math.hypot(...at(a).map((x, k) => x - (at(b)[k] as number)));
+ if (d < 1e-9 && u[a * 2] !== u[b * 2]) seamPairs++;
+ // Distinct surface points never coincide.
+ if (d < 1e-9) expect(u[a * 2]).not.toBe(u[b * 2]);
+ }
+ expect(seamPairs).toBeGreaterThan(0);
+ });
+
+ it("carries blend shapes and skin weights exactly", () => {
+ const o = octahedron();
+ const lift = o.positions.map((_, i) => (i % 3 === 1 ? 0.5 : 0));
+ const s = subdivide(
+ { ...o, targets: [{ name: "lift", positions: lift }] },
+ { weld: o.weld, levels: 2 },
+ );
+ const t = Array.from(s.targets?.[0]?.positions ?? []);
+ expect(t.length).toBe(s.positions.length);
+ for (let i = 0; i < t.length; i++)
+ expect(t[i]).toBeCloseTo(i % 3 === 1 ? 0.5 : 0, 12);
+ const w = Array.from(s.weights ?? []);
+ for (let v = 0; v < w.length / 4; v++)
+ expect(w.slice(v * 4, v * 4 + 4).reduce((a, b) => a + b, 0)).toBeCloseTo(
+ 1,
+ 9,
+ );
+ // Normals are unit length.
+ const nn = Array.from(s.normals ?? []);
+ for (let v = 0; v < nn.length / 3; v++)
+ expect(Math.hypot(...nn.slice(v * 3, v * 3 + 3))).toBeCloseTo(1, 9);
+ });
+});
diff --git a/packages/forge/src/subdivide.ts b/packages/forge/src/subdivide.ts
new file mode 100644
index 00000000..9fae8b73
--- /dev/null
+++ b/packages/forge/src/subdivide.ts
@@ -0,0 +1,296 @@
+/**
+ * Loop subdivision of a triangle primitive: each triangle splits in four
+ * and the surface relaxes toward its smooth limit (Loop, 1987). Everything
+ * that moves with the surface rides the same linear stencil, so it stays
+ * exact: blend shape deltas, skin weights (renormalized to four joints).
+ * UVs are interpolated linearly per face corner, so texture seams stay
+ * where they are. Vertices duplicated at a seam share one position: pass
+ * `weld` (the same id for every copy) and the seam subdivides as one
+ * surface instead of opening into a crease.
+ */
+import type { MorphTargetSpec, PrimitiveSpec } from "./document.js";
+
+export interface SubdivideOptions {
+ /** Levels (each multiplies the triangles by four). Default 1. */
+ levels?: number;
+ /** Per vertex: a topology id shared by copies of one surface point (UV seams). Default: the vertex index. */
+ weld?: ArrayLike<number>;
+}
+
+type Numbers = ArrayLike<number>;
+
+/** A vertex of the result as an affine combination of topology vertices. */
+type Stencil = [number, number][];
+
+export function subdivide(
+ p: PrimitiveSpec,
+ options: SubdivideOptions = {},
+): PrimitiveSpec {
+ let out = p;
+ let weld = options.weld;
+ for (let l = 0; l < (options.levels ?? 1); l++) {
+ const r = subdivideOnce(out, weld);
+ out = r.primitive;
+ weld = r.weld;
+ }
+ return out;
+}
+
+function subdivideOnce(
+ p: PrimitiveSpec,
+ weldIn?: Numbers,
+): { primitive: PrimitiveSpec; weld: number[] } {
+ const nv = p.positions.length / 3;
+ const idx = Array.from(p.indices ?? Array.from({ length: nv }, (_, i) => i));
+ const weld = weldIn
+ ? Array.from(weldIn)
+ : Array.from({ length: nv }, (_, i) => i);
+ // Topology vertices: one per weld id.
+ const topoOf = new Map<number, number>();
+ const repr: number[] = [];
+ const topo = weld.map((w, v) => {
+ let t = topoOf.get(w);
+ if (t === undefined) {
+ t = repr.length;
+ topoOf.set(w, t);
+ repr.push(v);
+ }
+ return t;
+ });
+ const nt = repr.length;
+
+ // Edges between topology vertices, with the opposite vertex of each face on them.
+ const edgeKey = (a: number, b: number) => (a < b ? a * nt + b : b * nt + a);
+ const opposite = new Map<number, number[]>();
+ const neighbors: Set<number>[] = Array.from({ length: nt }, () => new Set());
+ for (let f = 0; f < idx.length; f += 3)
+ for (let k = 0; k < 3; k++) {
+ const a = topo[idx[f + k] as number] as number;
+ const b = topo[idx[f + ((k + 1) % 3)] as number] as number;
+ const c = topo[idx[f + ((k + 2) % 3)] as number] as number;
+ const key = edgeKey(a, b);
+ const list = opposite.get(key) ?? [];
+ list.push(c);
+ opposite.set(key, list);
+ (neighbors[a] as Set<number>).add(b);
+ (neighbors[b] as Set<number>).add(a);
+ }
+ const boundary = (a: number, b: number) =>
+ (opposite.get(edgeKey(a, b))?.length ?? 0) < 2;
+
+ // Loop's stencils: moved old vertices, new edge vertices.
+ const vertexStencil = (t: number): Stencil => {
+ const ns = [...(neighbors[t] as Set<number>)];
+ const rim = ns.filter((n) => boundary(t, n));
+ if (rim.length >= 2)
+ return [
+ [t, 0.75],
+ [rim[0] as number, 0.125],
+ [rim[1] as number, 0.125],
+ ];
+ const n = ns.length;
+ if (!n) return [[t, 1]];
+ const beta = n === 3 ? 3 / 16 : 3 / (8 * n);
+ return [[t, 1 - n * beta], ...ns.map((m): [number, number] => [m, beta])];
+ };
+ const edgeStencil = (a: number, b: number): Stencil => {
+ const opp = opposite.get(edgeKey(a, b)) ?? [];
+ if (opp.length !== 2)
+ return [
+ [a, 0.5],
+ [b, 0.5],
+ ];
+ return [
+ [a, 0.375],
+ [b, 0.375],
+ [opp[0] as number, 0.125],
+ [opp[1] as number, 0.125],
+ ];
+ };
+
+ // Output vertices: an old corner (topology vertex + its own UV), or an
+ // edge's new vertex per distinct UV midpoint.
+ const stencils: Stencil[] = [];
+ const uvs: number[] = [];
+ const uv1: number[] = [];
+ const hasUv = Boolean(p.uvs);
+ const hasUv1 = Boolean(p.uvs1);
+ const newWeld: number[] = [];
+ const oldOut = new Map<number, number>();
+ const edgeOut = new Map<string, number>();
+ const pushUv = (
+ dst: number[],
+ src: Numbers | undefined,
+ a: number,
+ b?: number,
+ ) => {
+ if (!src) return;
+ const u =
+ b === undefined
+ ? (src[a * 2] as number)
+ : ((src[a * 2] as number) + (src[b * 2] as number)) / 2;
+ const v =
+ b === undefined
+ ? (src[a * 2 + 1] as number)
+ : ((src[a * 2 + 1] as number) + (src[b * 2 + 1] as number)) / 2;
+ dst.push(u, v);
+ };
+ const old = (v: number) => {
+ let o = oldOut.get(v);
+ if (o === undefined) {
+ o = stencils.length;
+ oldOut.set(v, o);
+ stencils.push(vertexStencil(topo[v] as number));
+ pushUv(uvs, p.uvs, v);
+ pushUv(uv1, p.uvs1, v);
+ newWeld.push(topo[v] as number);
+ }
+ return o;
+ };
+ const mid = (va: number, vb: number) => {
+ const a = topo[va] as number;
+ const b = topo[vb] as number;
+ const uvKey = hasUv
+ ? `${(((p.uvs?.[va * 2] as number) + (p.uvs?.[vb * 2] as number)) / 2).toFixed(6)},${(((p.uvs?.[va * 2 + 1] as number) + (p.uvs?.[vb * 2 + 1] as number)) / 2).toFixed(6)}`
+ : "";
+ const key = `${edgeKey(a, b)}|${uvKey}`;
+ let o = edgeOut.get(key);
+ if (o === undefined) {
+ o = stencils.length;
+ edgeOut.set(key, o);
+ stencils.push(edgeStencil(a, b));
+ pushUv(uvs, p.uvs, va, vb);
+ pushUv(uv1, p.uvs1, va, vb);
+ newWeld.push(nt + edgeKey(a, b));
+ }
+ return o;
+ };
+ const indices: number[] = [];
+ for (let f = 0; f < idx.length; f += 3) {
+ const [a, b, c] = [idx[f], idx[f + 1], idx[f + 2]] as [
+ number,
+ number,
+ number,
+ ];
+ const [oa, ob, oc] = [old(a), old(b), old(c)];
+ const [ab, bc, ca] = [mid(a, b), mid(b, c), mid(c, a)];
+ indices.push(oa, ab, ca, ab, ob, bc, ca, bc, oc, ab, bc, ca);
+ }
+
+ // Per-topology-vertex values, then the stencils.
+ const perTopo = (data: Numbers, size: number) => {
+ const out = new Float64Array(nt * size);
+ repr.forEach((v, t) => {
+ for (let k = 0; k < size; k++)
+ out[t * size + k] = data[v * size + k] as number;
+ });
+ return out;
+ };
+ const apply = (data: Float64Array, size: number) => {
+ const out = new Array<number>(stencils.length * size).fill(0);
+ stencils.forEach((s, i) => {
+ for (const [t, w] of s)
+ for (let k = 0; k < size; k++)
+ out[i * size + k] =
+ (out[i * size + k] as number) + w * (data[t * size + k] as number);
+ });
+ return out;
+ };
+ const positions = apply(perTopo(p.positions, 3), 3);
+ const result: PrimitiveSpec = { ...p, positions, indices };
+ if (hasUv) result.uvs = uvs;
+ if (hasUv1) result.uvs1 = uv1;
+ if (p.colors) result.colors = apply(perTopo(p.colors, 4), 4);
+ if (p.joints && p.weights) {
+ const j = p.joints;
+ const w = p.weights;
+ const lists = repr.map((v) => {
+ const l = new Map<number, number>();
+ for (let k = 0; k < 4; k++) {
+ const wk = w[v * 4 + k] as number;
+ if (wk > 0)
+ l.set(
+ j[v * 4 + k] as number,
+ (l.get(j[v * 4 + k] as number) ?? 0) + wk,
+ );
+ }
+ return l;
+ });
+ const joints: number[] = [];
+ const weights: number[] = [];
+ for (const s of stencils) {
+ const acc = new Map<number, number>();
+ for (const [t, sw] of s)
+ for (const [jj, ww] of lists[t] as Map<number, number>)
+ acc.set(jj, (acc.get(jj) ?? 0) + sw * ww);
+ const top = [...acc]
+ .filter(([, x]) => x > 0)
+ .sort((a, b) => b[1] - a[1])
+ .slice(0, 4);
+ const sum = top.reduce((t, x) => t + x[1], 0) || 1;
+ for (let k = 0; k < 4; k++) {
+ joints.push(top[k]?.[0] ?? 0);
+ weights.push((top[k]?.[1] ?? 0) / sum);
+ }
+ }
+ result.joints = joints;
+ result.weights = weights;
+ }
+ result.normals = smoothNormals(positions, indices, newWeld);
+ if (p.targets?.length)
+ result.targets = p.targets.map((t): MorphTargetSpec => {
+ const delta = apply(perTopo(t.positions, 3), 3);
+ if (!t.normals) return { name: t.name, positions: delta };
+ const moved = positions.map((x, i) => x + (delta[i] as number));
+ const shaped = smoothNormals(moved, indices, newWeld);
+ return {
+ name: t.name,
+ positions: delta,
+ normals: shaped.map((n, i) => n - (result.normals?.[i] as number)),
+ };
+ });
+ return { primitive: result, weld: newWeld };
+}
+
+/** Area-weighted vertex normals, shared by welded copies. */
+export function smoothNormals(
+ positions: Numbers,
+ indices: Numbers,
+ weld: Numbers,
+): number[] {
+ const acc = new Map<number, [number, number, number]>();
+ for (let f = 0; f < indices.length; f += 3) {
+ const [a, b, c] = [indices[f], indices[f + 1], indices[f + 2]] as [
+ number,
+ number,
+ number,
+ ];
+ const ux = (positions[b * 3] as number) - (positions[a * 3] as number);
+ const uy =
+ (positions[b * 3 + 1] as number) - (positions[a * 3 + 1] as number);
+ const uz =
+ (positions[b * 3 + 2] as number) - (positions[a * 3 + 2] as number);
+ const vx = (positions[c * 3] as number) - (positions[a * 3] as number);
+ const vy =
+ (positions[c * 3 + 1] as number) - (positions[a * 3 + 1] as number);
+ const vz =
+ (positions[c * 3 + 2] as number) - (positions[a * 3 + 2] as number);
+ const n: [number, number, number] = [
+ uy * vz - uz * vy,
+ uz * vx - ux * vz,
+ ux * vy - uy * vx,
+ ];
+ for (const v of [a, b, c]) {
+ const w = weld[v] as number;
+ const cur = acc.get(w) ?? [0, 0, 0];
+ acc.set(w, [cur[0] + n[0], cur[1] + n[1], cur[2] + n[2]]);
+ }
+ }
+ const out: number[] = [];
+ for (let v = 0; v < weld.length; v++) {
+ const n = acc.get(weld[v] as number) ?? [0, 1, 0];
+ const l = Math.hypot(n[0], n[1], n[2]) || 1;
+ out.push(n[0] / l, n[1] / l, n[2] / l);
+ }
+ return out;
+}
diff --git a/packages/webgpu-renderers/src/loaders/gltf-loader.ts b/packages/webgpu-renderers/src/loaders/gltf-loader.ts
index 33261370..e3e10659 100644
--- a/packages/webgpu-renderers/src/loaders/gltf-loader.ts
+++ b/packages/webgpu-renderers/src/loaders/gltf-loader.ts
@@ -141,6 +141,8 @@ interface GltfMesh {
name?: string;
weights?: number[];
primitives: GltfPrimitive[];
+ /** `targetNames`: the de-facto convention for naming morph targets. */
+ extras?: { targetNames?: string[] };
}
interface GltfTextureTransform {
@@ -178,9 +180,6 @@ interface GltfMaterial {
/** VRM 1.0 toon material: the base is lit flat, the shadowed side tinted by shadeColorFactor. */
VRMC_materials_mtoon?: { shadeColorFactor?: [number, number, number] };
KHR_materials_clearcoat?: {
- KHR_materials_transmission?: { transmissionFactor?: number };
- KHR_materials_ior?: { ior?: number };
- KHR_materials_volume?: { thicknessFactor?: number };
clearcoatFactor?: number;
clearcoatRoughnessFactor?: number;
};
@@ -192,6 +191,9 @@ interface GltfMaterial {
anisotropyStrength?: number;
anisotropyRotation?: number;
};
+ KHR_materials_transmission?: { transmissionFactor?: number };
+ KHR_materials_ior?: { ior?: number };
+ KHR_materials_volume?: { thicknessFactor?: number };
KHR_materials_iridescence?: {
iridescenceFactor?: number;
iridescenceIor?: number;
@@ -1283,6 +1285,10 @@ export class GltfLoader {
mapUvs,
morphTargets,
morphWeights,
+ ...(morphTargets &&
+ meshDef.extras?.targetNames?.length === morphTargets.length
+ ? { morphTargetNames: meshDef.extras.targetNames }
+ : {}),
colors: rgba,
jointIndices: skinned
? read(prim.attributes.JOINTS_0)?.data
diff --git a/packages/webgpu-renderers/src/loaders/types.ts b/packages/webgpu-renderers/src/loaders/types.ts
index 39595add..d13013c2 100644
--- a/packages/webgpu-renderers/src/loaders/types.ts
+++ b/packages/webgpu-renderers/src/loaders/types.ts
@@ -10,6 +10,8 @@ export interface Mesh3DData {
morphTargets?: { positions?: Float32Array; normals?: Float32Array }[];
/** Default morph weights for this mesh instance. */
morphWeights?: number[];
+ /** Morph target names (glTF mesh `extras.targetNames`), one per target. */
+ morphTargetNames?: string[];
/** Per-map UVs, with the map's own texture transform already applied. */
mapUvs?: {
normal?: Float32Array;
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index d6a4a2b9..e7684cf0 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -476,6 +476,12 @@ Layer.glb("rig.glb", { rotateX: 180, pose: { LeftArm: { rotateZ: 90 } } });
**Realistic humans (`human`)**: `await human({ sex, age, muscle, weight, proportions, ethnicity, height, targets })` builds a rigged, skinned body from MakeHuman's CC0 base mesh: `sex` 0 female … 1 male, `age` 0 baby / 0.1875 child / 0.5 adult / 1 old, `muscle` and `weight` 0 … 1 (0.5 average), `height` in cm (exact), and any MakeHuman target by path with a weight (`targets: { "nose/nose-base-up": 0.6, "ears/l-ear-shape-pointed": 1 }`). Its skeleton is placed from the shaped body with Mixamo names on the main bones (twist, extra spine and face bones keep MakeHuman names), so `pose`, `aim` and `retarget({ rig: "mixamo" })` work; weights are MakeHuman's painted weights; UVs come with the mesh. Assets download on first use (about 2 MB base plus only the targets a body needs; 131 MB if every target is used) into `~/.cache/framefields/makehuman`, verified by SHA-256 (`FRAMEFIELDS_CACHE_DIR`, `FRAMEFIELDS_HUMAN_BASE_URL` for a mirror). Use `sculpt` for costume, armor and non-human characters.
+**Faces**: the body carries facial blend shapes under ARKit names (`eyeBlinkLeft`, `eyeWideRight`, `eyeSquintLeft`, `browDownLeft`, `browInnerUp`, `browOuterUpRight`, `noseSneerLeft`, `jawOpen`, `mouthSmileLeft`, `mouthFrownRight`, `mouthPucker`, `mouthFunnel`, `mouthPressLeft`, `mouthStretchRight`, `mouthUpperUpLeft`, `mouthLowerDownRight`, `mouthDimpleLeft`, `mouthRollLower`, plus `noseFlareLeft/Right`, `noseWrinkle`, `mouthPart`, `neckTense`; the list is `FACE_SHAPES`), built from MakeHuman's expression units for the body's ethnicity; symmetric mouth units are split into sides. `human().mouth` (teeth and tongue) moves with `jawOpen` and the lips. Drive them per layer with `morphs`: names match the file's target names case-insensitively, each a number, keys or a signal; unnamed targets keep the clip's weights. `expressions: false` leaves them out; `subdivide: 1` (Loop, blend shapes and weights carried exactly) smooths faceting in close-ups. Morph targets that move few vertices are written sparse (a face-rigged body is about 2 MB, 6 MB subdivided).
+
+```ts
+Layer.glb("elf.glb", { morphs: { jawOpen: [[0, 0], [12, 0.6]], mouthSmileLeft: 0.8, mouthSmileRight: 0.8, eyeBlinkLeft: blink, eyeBlinkRight: blink } });
+```
+
```ts
import { GltfDocument, human } from "framefields/forge";
const elf = await human({ sex: 0, age: 0.6, muscle: 0.55, weight: 0.35, height: 184 });
diff --git a/specs/character-forge.md b/specs/character-forge.md
index 43898510..64bbe86f 100644
--- a/specs/character-forge.md
+++ b/specs/character-forge.md
@@ -1,6 +1,6 @@
# Character Forge: rigged characters built in code
-Status: Phases 1–2 done (GLB writer, skeleton, humanoid preset, bone gizmo; bound sculpting, manifold narrow-band meshing, simplification, smooth material borders, skin weights from the field; `packages/forge`, exposed as `framefields/forge`). Sharp-edge preservation for hard surfaces moves to Phase 7's hard-surface work. Human bodies and faces now start from MakeHuman's CC0 base (`human()`): realistic anatomy, edge loops and UVs, macro and feature targets, painted weights. `sculpt` stays for stylization, costume, armor and non-humans. The face rig (Phase 4) builds on MakeHuman's expression units and the ARKit 52 names. Phase 3 in progress: the loader reads clearcoat, sheen, anisotropy and iridescence and forge's `extras.framefields.subsurface`, the compositor uses them under layer overrides, and the subsurface lobe was rewritten to conserve energy (per-channel wrap). Eyes (first step of Phase 5): `human()` returns MakeHuman's eyeballs fitted to the shaped head (proxy binding), skinned to Mixamo `LeftEye` / `RightEye` pivoted at each ball's least-squares sphere center, plus the CC0 iris texture; a clear coat gives the wet highlight. Corneas are back as glass: glass now refracts the 3D scene behind it (a pre-pass of the batch without glass), and glTF transmission + IOR + volume thickness map to glass whose bend follows the refracted ray through that depth, so the iris shifts with the viewing angle. Lids follow the gaze through joint drivers (`drive`, carried in the GLB), and `addHuman` writes the whole character. Still to do in Phase 3: alpha-to-coverage for hair cards and morphs combined with skinning; the specular, IOR, transmission and volume extensions are written but not yet rendered. First character: **Ithrael Sunmourn** (`examples/36_emberwake/CHARACTERS.md`), an original hooded elven ranger with a long cloak, long hair, a sculpted face, real eyes, and a full set of expressions.
+Status: Phases 1–2 done (GLB writer, skeleton, humanoid preset, bone gizmo; bound sculpting, manifold narrow-band meshing, simplification, smooth material borders, skin weights from the field; `packages/forge`, exposed as `framefields/forge`). Sharp-edge preservation for hard surfaces moves to Phase 7's hard-surface work. Human bodies and faces now start from MakeHuman's CC0 base (`human()`): realistic anatomy, edge loops and UVs, macro and feature targets, painted weights. `sculpt` stays for stylization, costume, armor and non-humans. The face rig (Phase 4) builds on MakeHuman's expression units and the ARKit 52 names. Phase 3 in progress: the loader reads clearcoat, sheen, anisotropy and iridescence and forge's `extras.framefields.subsurface`, the compositor uses them under layer overrides, and the subsurface lobe was rewritten to conserve energy (per-channel wrap). Eyes (first step of Phase 5): `human()` returns MakeHuman's eyeballs fitted to the shaped head (proxy binding), skinned to Mixamo `LeftEye` / `RightEye` pivoted at each ball's least-squares sphere center, plus the CC0 iris texture; a clear coat gives the wet highlight. Corneas are back as glass: glass now refracts the 3D scene behind it (a pre-pass of the batch without glass), and glTF transmission + IOR + volume thickness map to glass whose bend follows the refracted ray through that depth, so the iris shifts with the viewing angle. Lids follow the gaze through joint drivers (`drive`, carried in the GLB), and `addHuman` writes the whole character. Face (Phase 4): ARKit-named blend shapes from MakeHuman's expression units (symmetric ones split into sides with a soft seam), a teeth-and-tongue `mouth` part that moves with the jaw and lips, per-layer `morphs` by name, Loop subdivision that carries shapes and weights exactly, and sparse morph accessors (2 MB per face-rigged body). Still to do in Phase 4: sculpted teeth to replace MakeHuman's helper blocks, a darkened mouth cavity, and wrinkle maps driven by expression weights. Still to do in Phase 3: alpha-to-coverage for hair cards and morphs combined with skinning; the specular, IOR, transmission and volume extensions are written but not yet rendered. First character: **Ithrael Sunmourn** (`examples/36_emberwake/CHARACTERS.md`), an original hooded elven ranger with a long cloak, long hair, a sculpted face, real eyes, and a full set of expressions.
## Goal