Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(3d): eyes with glass corneas, glass that refracts the 3D scene, joint drivers

okanasld545595080ef
diff --git a/.changeset/forge-human-eyes.md b/.changeset/forge-human-eyes.md
new file mode 100644
index 00000000..6cbb2346
--- /dev/null
+++ b/.changeset/forge-human-eyes.md
@@ -0,0 +1,5 @@
+---
+"framefields": minor
+---
+
+`human()` returns eyes: MakeHuman's eyeballs fitted to the shaped head, skinned to `LeftEye` / `RightEye` joints that pivot at each eyeball's center (so `lookAt` aims the gaze), with the CC0 iris texture as `eyeTexture`.
diff --git a/.changeset/glass-scene-refraction.md b/.changeset/glass-scene-refraction.md
new file mode 100644
index 00000000..0dfcd20d
--- /dev/null
+++ b/.changeset/glass-scene-refraction.md
@@ -0,0 +1,9 @@
+---
+"framefields": minor
+---
+
+Glass refracts the 3D scene behind it, not only the layers under the 3D batch. glTF materials with `KHR_materials_transmission` render as glass with their IOR, bending by their `KHR_materials_volume` thickness through the refracted ray, so the bend follows the object's size on screen.
+
+Joint drivers (`drive: [{ joint, from, rotate: { x, y, z } }]`): a joint turns by a share of another joint's turn, after look-ats and before IK. Model files carry their own drivers (forge's `GltfDocument.addDriver`).
+
+`human()` adds glass corneas over the eyes and lid drivers, so eyelids follow a gaze aimed with `lookAt`; `addHuman(doc, h)` writes the whole character with default skin, eye and cornea materials.
diff --git a/nodes/node-compositor/src/renderers/character.ts b/nodes/node-compositor/src/renderers/character.ts
index e6d58e82..597c04c3 100644
--- a/nodes/node-compositor/src/renderers/character.ts
+++ b/nodes/node-compositor/src/renderers/character.ts
@@ -50,7 +50,14 @@ const transformDir = (m: Mat4, v: Vec3): Vec3 => [
type CharacterNode = Pick<
Model3DNode,
- "animationLayers" | "ik" | "lookAt" | "springBones" | "loop" | "pose" | "aim"
+ | "animationLayers"
+ | "ik"
+ | "lookAt"
+ | "springBones"
+ | "loop"
+ | "pose"
+ | "aim"
+ | "drive"
>;
/** Whether a node needs the rig evaluator (anything beyond one plain clip). */
@@ -61,7 +68,8 @@ export function hasRig(n: CharacterNode): boolean {
n.lookAt?.length ||
n.springBones?.length ||
(n.pose && Object.keys(n.pose).length) ||
- n.aim?.length,
+ n.aim?.length ||
+ n.drive?.length,
);
}
@@ -163,6 +171,8 @@ function constraintsAt(
pole: c.poleTarget ? point(c.poleTarget) : undefined,
weight: numberAt(c.weight, clock, 1),
})),
+ // The file's own drivers, then the layer's.
+ drive: [...(p.model.drivers ?? []), ...(p.node.drive ?? [])],
lookAt: (p.node.lookAt ?? []).map((c: LookAtConstraint) => ({
joint: c.joint,
target: point(c.target),
@@ -230,7 +240,8 @@ function springsAt(
/** Model-space node matrices for this frame. */
export function poseCharacter(p: PoseInput): Mat4[] {
- if (!hasRig(p.node)) return poseNodes(p.model, p.clip, p.clipSec);
+ if (!hasRig(p.node) && !p.model.drivers?.length)
+ return poseNodes(p.model, p.clip, p.clipSec);
const toModel = Matrix4Math.invert(p.sceneRoot) ?? Matrix4Math.identity();
const rig = basePose(p, p.frame, toModel);
springsAt(p, toModel, rig)?.apply(rig);
diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
index 4fe13168..d79d684c 100644
--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
@@ -684,6 +684,13 @@ function surfaceProps(
return out;
}
+/** A transform's mean scale: the average length of its basis columns. */
+function matrixScale(m: Mat4): number {
+ const c = (i: number) =>
+ Math.hypot(m[i] as number, m[i + 1] as number, m[i + 2] as number);
+ return (c(0) + c(4) + c(8)) / 3;
+}
+
function computeContainerScissor(
matrix: DOMMatrix,
width: number,
@@ -2039,9 +2046,14 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const sceneShadow = shadow.length ? shadow : null;
/** One forward pass over the scene: floor, then the sorted draws. */
+ /**
+ * One draw of the batch. `glassBackdrop` is what glass refracts;
+ * null leaves the glass out (the pre-pass that makes that picture).
+ */
const drawScene = (
screenIndirect: "radiance" | "apply" | undefined,
writeHdr: boolean,
+ glassBackdrop: GPUTexture | null = curTex,
) => {
const begun = r3d.beginPass(
encoder,
@@ -2073,6 +2085,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
);
}
for (const item of ordered) {
+ if (glassBackdrop === null && isGlass(item)) continue;
if (item.kind === "quad") {
r3d.drawTextureQuad(
pass,
@@ -2099,7 +2112,9 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
lightsBindGroup,
item.meshBuffers ?? item.mesh,
item.texture,
- isGlass(item) ? { ...item.opts, backdrop: curTex } : item.opts,
+ isGlass(item)
+ ? { ...item.opts, backdrop: glassBackdrop ?? curTex }
+ : item.opts,
geometryEnabled,
);
}
@@ -2114,9 +2129,23 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const { cameraBuffer: radianceCamera } = drawScene("radiance", true);
r3d.traceScreenIndirect(encoder, radianceCamera, sceneEnvironment);
}
+ // Glass refracts the scene behind it, not just the layers under the
+ // batch: draw everything else first and lay it over them.
+ let glassBackdrop = curTex;
+ if (glass.length) {
+ drawScene(screenIndirectEnabled ? "apply" : undefined, false, null);
+ glassBackdrop = ctx.renderer.composite(
+ encoder,
+ curTex,
+ scene3dTex,
+ "normal",
+ [targetTexture, scene3dTex, ...curExcludes],
+ );
+ }
const { cameraBindGroup, cameraBuffer, lightsBindGroup } = drawScene(
screenIndirectEnabled ? "apply" : undefined,
Boolean(sceneBloom),
+ glassBackdrop,
);
let final3dTex = scene3dTex;
@@ -4030,9 +4059,13 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
target?.ambientIntensity ?? lop.ambientIntensity ?? 1.0,
metallic: target?.metallic ?? lop.metallic ?? mat?.metallic,
...surfaceProps(target, lop, mat),
- ior: target?.ior ?? lop.ior,
+ ior: target?.ior ?? lop.ior ?? mat?.ior,
dispersion: target?.dispersion ?? lop.dispersion,
thickness: target?.thickness ?? lop.thickness,
+ // The file's thickness is in model units; the draw wants world units.
+ ...(mat?.thickness !== undefined
+ ? { worldThickness: mat.thickness * matrixScale(placed.matrix) }
+ : {}),
audioDeform: meshAudioDeform,
wind: meshWind(lop.wind, model, (shiftedFrame ?? 0) / (fps || 30)),
};
diff --git a/packages/compositions/src/program/character-schema.ts b/packages/compositions/src/program/character-schema.ts
index a76c9466..95bae8b2 100644
--- a/packages/compositions/src/program/character-schema.ts
+++ b/packages/compositions/src/program/character-schema.ts
@@ -131,6 +131,27 @@ export const AimSchema = z
.strict();
export type AimConstraint = z.infer<typeof AimSchema>;
+/**
+ * A joint driven by another: it turns by a share of the source joint's turn
+ * from its rest, per local axis (Euler degrees, Z then Y then X). Applied
+ * after look-ats, before IK, so eyelids follow eyes a `lookAt` aims.
+ */
+export const JointDriverSchema = z
+ .object({
+ joint: z.string(),
+ from: z.string(),
+ /** Share of the source's turn about each local axis; 1 follows it fully. */
+ rotate: z
+ .object({
+ x: z.number().optional(),
+ y: z.number().optional(),
+ z: z.number().optional(),
+ })
+ .strict(),
+ })
+ .strict();
+export type JointDriver = z.infer<typeof JointDriverSchema>;
+
export const SpringBoneChainSchema = z
.object({
/** Root joints of chains (cape, tail, antenna, hair); each runs down its first child. */
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index 422adf3c..220f7595 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -32,6 +32,8 @@ import {
AimSchema,
type CrowdConfig,
CrowdSchema,
+ type JointDriver,
+ JointDriverSchema,
type JointPose,
JointPoseSchema,
type LookAtConstraint,
@@ -1855,6 +1857,8 @@ export interface Model3DNode {
pose?: Record<string, JointPose>;
/** Bones pointed along scene directions, parents first (pose retargeting). */
aim?: AimConstraint[];
+ /** Joints driven by other joints, on top of any the model file declares (eyelids, twist bones). */
+ drive?: JointDriver[];
/** Instance nodes: every instance a skinned character on its own clip, phase and speed. */
crowd?: CrowdConfig;
}
@@ -2192,6 +2196,7 @@ export const Model3DNodeSchema: z.ZodType<Model3DNode> = z
springBones: z.array(SpringBoneChainSchema).optional(),
pose: z.record(z.string(), JointPoseSchema).optional(),
aim: z.array(AimSchema).optional(),
+ drive: z.array(JointDriverSchema).optional(),
crowd: CrowdSchema.optional(),
})
.strict() as unknown as z.ZodType<Model3DNode>;
diff --git a/packages/forge/src/document.ts b/packages/forge/src/document.ts
index 32ab1578..59bc0d93 100644
--- a/packages/forge/src/document.ts
+++ b/packages/forge/src/document.ts
@@ -95,6 +95,17 @@ export interface AnimationSpec {
channels: ChannelSpec[];
}
+/**
+ * A joint driven by another (framefields rigs read it from the scene's
+ * `extras.framefields.drivers`): it turns by a share of the source joint's
+ * turn from its rest, per local axis.
+ */
+export interface DriverSpec {
+ joint: string;
+ from: string;
+ rotate: { x?: number; y?: number; z?: number };
+}
+
export interface TextureSpec {
/** Encoded image bytes. */
image: Uint8Array;
@@ -189,6 +200,7 @@ export class GltfDocument {
private readonly images: Record<string, unknown>[] = [];
private readonly samplers: Record<string, unknown>[] = [];
private readonly extensions = new Set<string>();
+ private readonly drivers: DriverSpec[] = [];
addMaterial(spec: MaterialSpec): MaterialRef {
const { json, extensions } = materialJson(spec);
@@ -509,6 +521,16 @@ export class GltfDocument {
throw new Error("material: unknown texture");
}
+ /** Drives `joint` from `from` (eyelids following eyes, twist bones). Joints are matched by name at render time. */
+ addDriver(spec: DriverSpec): void {
+ const shares = Object.values(spec.rotate);
+ if (!shares.length || shares.some((v) => !Number.isFinite(v)))
+ throw new Error(
+ `driver ${spec.joint}: give a finite share for at least one axis`,
+ );
+ this.drivers.push(spec);
+ }
+
/** The document's glTF JSON (without the binary chunk), for inspection. */
json(): Record<string, unknown> {
const roots = this.nodes
@@ -516,7 +538,14 @@ export class GltfDocument {
.filter((i) => !this.parents.has(i));
const out: Record<string, unknown> = {
scene: 0,
- scenes: [{ nodes: roots }],
+ scenes: [
+ {
+ nodes: roots,
+ ...(this.drivers.length
+ ? { extras: { framefields: { drivers: this.drivers } } }
+ : {}),
+ },
+ ],
nodes: this.nodes,
};
const put = (key: string, list: unknown[]) => {
diff --git a/packages/forge/src/forge.test.ts b/packages/forge/src/forge.test.ts
index c4e977ce..dd38ed5b 100644
--- a/packages/forge/src/forge.test.ts
+++ b/packages/forge/src/forge.test.ts
@@ -215,6 +215,30 @@ describe("GltfDocument", () => {
expect(surface?.subsurfaceColor?.[1]).toBeCloseTo(0x50 / 0xff, 2);
});
+ it("writes rig drivers and transmissive glass the loader reads", () => {
+ const doc = new GltfDocument();
+ const glass = doc.addMaterial({
+ transmission: 1,
+ ior: 1.376,
+ volume: { thickness: 0.3 },
+ });
+ doc.addNode({
+ mesh: doc.addMesh({ primitives: [{ ...strip(1, 10), material: glass }] }),
+ });
+ doc.addDriver({ joint: "lid", from: "eye", rotate: { x: 0.65 } });
+ const model = parseGLB(doc.toGlb());
+ expect(model.drivers).toEqual([
+ { joint: "lid", from: "eye", rotate: { x: 0.65 } },
+ ]);
+ const m = Object.values(model.materials)[0];
+ expect(m?.shading).toBe("glass");
+ expect(m?.ior).toBeCloseTo(1.376);
+ expect(m?.thickness).toBeCloseTo(0.3);
+ expect(() => doc.addDriver({ joint: "a", from: "b", rotate: {} })).toThrow(
+ /finite share/,
+ );
+ });
+
it("refuses inconsistent input where it is given", () => {
const doc = new GltfDocument();
expect(() =>
diff --git a/packages/forge/src/human/add-human.ts b/packages/forge/src/human/add-human.ts
new file mode 100644
index 00000000..122df2fd
--- /dev/null
+++ b/packages/forge/src/human/add-human.ts
@@ -0,0 +1,86 @@
+/**
+ * Writes a `human()` into a document in one call: skeleton, skinned body,
+ * textured eyeballs, glass corneas and the rig's drivers, each part with a
+ * material you can override.
+ */
+import type {
+ GltfDocument,
+ MaterialRef,
+ NodeRef,
+ SkeletonRefs,
+} from "../document.js";
+import type { MaterialSpec } from "../material.js";
+import type { HumanModel } from "./human.js";
+
+export interface AddHumanOptions {
+ /** Node name. Default "Human". */
+ name?: string;
+ /** Skin material. Default: a warm skin with subsurface scattering. */
+ skin?: MaterialSpec;
+ /** Eyeball material; its color texture defaults to the model's eye texture. */
+ eyes?: MaterialSpec;
+ /** Cornea material. Default: clear glass (IOR 1.376, 3 mm deep). */
+ cornea?: MaterialSpec;
+}
+
+export interface AddedHuman {
+ rig: SkeletonRefs;
+ nodes: { body: NodeRef; eyes: NodeRef; cornea: NodeRef };
+ materials: { skin: MaterialRef; eyes: MaterialRef; cornea: MaterialRef };
+}
+
+export function addHuman(
+ doc: GltfDocument,
+ h: HumanModel,
+ options: AddHumanOptions = {},
+): AddedHuman {
+ const name = options.name ?? "Human";
+ const rig = doc.addSkeleton(h.skeleton);
+ const skin = doc.addMaterial(
+ options.skin ?? {
+ name: `${name}Skin`,
+ color: "#d9b29a",
+ roughness: 0.5,
+ subsurface: { factor: 0.7, color: "#ff6a4a" },
+ },
+ );
+ const eyes = doc.addMaterial({
+ name: `${name}Eyes`,
+ roughness: 0.4,
+ subsurface: { factor: 0.3, color: "#ff8a7a" },
+ ...options.eyes,
+ colorTexture:
+ options.eyes?.colorTexture ??
+ doc.addTexture({ image: h.eyeTexture, mimeType: "image/png" }),
+ });
+ const cornea = doc.addMaterial(
+ options.cornea ?? {
+ name: `${name}Cornea`,
+ color: "#ffffff",
+ roughness: 0.02,
+ transmission: 1,
+ ior: 1.376,
+ volume: { thickness: 0.3 },
+ },
+ );
+ const part = (
+ part: string,
+ primitive: HumanModel["body"],
+ material: MaterialRef,
+ ) =>
+ doc.addNode({
+ name: `${name}${part}`,
+ mesh: doc.addMesh({
+ name: `${name}${part}`,
+ primitives: [{ ...primitive, material }],
+ }),
+ skin: rig.skin,
+ });
+ const nodes = {
+ body: part("Body", h.body, skin),
+ eyes: part("Eyes", h.eyes, eyes),
+ cornea: part("Cornea", h.cornea, cornea),
+ };
+ for (const d of h.drivers) doc.addDriver(d);
+ return { rig, nodes, materials: { skin, eyes, cornea } };
+}
diff --git a/packages/forge/src/human/human.test.ts b/packages/forge/src/human/human.test.ts
index 7f5db5d0..aef8cb50 100644
--- a/packages/forge/src/human/human.test.ts
+++ b/packages/forge/src/human/human.test.ts
@@ -12,6 +12,7 @@ import { join } from "node:path";
import { validateBytes } from "gltf-validator";
import { describe, expect, it } from "vitest";
import {
+ addHuman,
GltfDocument,
HumanAssetStore,
human,
@@ -138,15 +139,64 @@ describe.skipIf(!cached)("human (cached MakeHuman assets)", () => {
expect(Math.abs(s - 1)).toBeLessThan(1e-6);
}
const doc = new GltfDocument();
- const rig = doc.addSkeleton(h.skeleton);
- doc.addNode({
- mesh: doc.addMesh({ primitives: [h.body] }),
- skin: rig.skin,
- });
+ addHuman(doc, h);
const report = await validateBytes(doc.toGlb(), { maxIssues: 20 });
expect(report.issues.messages.filter((m) => m.severity <= 1)).toEqual([]);
});
+ it("fits eyeballs that turn about their own centers", async () => {
+ const h = await human({ sex: 0, height: 170 });
+ const left = h.skeleton.joint("LeftEye");
+ const right = h.skeleton.joint("RightEye");
+ expect(left.at[0]).toBeGreaterThan(0);
+ expect(right.at[0]).toBeCloseTo(-left.at[0], 1);
+ const { positions, joints, weights } = h.eyes;
+ const n = positions.length / 3;
+ expect(n).toBeGreaterThan(400);
+ // Every vertex follows its own side's eye, rigidly; each lies on a
+ // sphere around that eye's joint (the pivot is the ball's center).
+ const radii: number[] = [];
+ for (let v = 0; v < n; v++) {
+ const x = positions[v * 3] as number;
+ const j = joints[v * 4];
+ expect(j).toBe(x > 0 ? left.index : right.index);
+ expect(weights[v * 4]).toBe(1);
+ const c = x > 0 ? left.at : right.at;
+ radii.push(
+ Math.hypot(
+ x - c[0],
+ (positions[v * 3 + 1] as number) - c[1],
+ (positions[v * 3 + 2] as number) - c[2],
+ ),
+ );
+ }
+ const mean = radii.reduce((s, r) => s + r, 0) / n;
+ expect(mean).toBeGreaterThan(0.9);
+ expect(mean).toBeLessThan(1.8);
+ expect(Math.max(...radii.map((r) => Math.abs(r - mean)))).toBeLessThan(
+ mean * 0.15,
+ );
+ // Corneas: their own shells, in front of the eyeballs, skinned the same way.
+ expect(h.cornea.positions.length).toBeGreaterThan(300);
+ const frontEye = Math.max(
+ ...Array.from({ length: n }, (_, v) => positions[v * 3 + 2] as number),
+ );
+ const frontCornea = Math.max(
+ ...Array.from(
+ { length: h.cornea.positions.length / 3 },
+ (_, v) => h.cornea.positions[v * 3 + 2] as number,
+ ),
+ );
+ expect(frontCornea).toBeGreaterThan(frontEye);
+ // The lid drivers name joints the skeleton has.
+ for (const d of h.drivers) {
+ expect(h.skeleton.has(d.joint), d.joint).toBe(true);
+ expect(h.skeleton.has(d.from), d.from).toBe(true);
+ }
+ // The texture is a PNG.
+ expect(Array.from(h.eyeTexture.slice(1, 4))).toEqual([0x50, 0x4e, 0x47]);
+ });
+
it("adds named targets on top of the macro shape", async () => {
const plain = await human({});
const shaped = await human({ targets: { "nose/nose-base-up": 1 } });
diff --git a/packages/forge/src/human/human.ts b/packages/forge/src/human/human.ts
index e6d39ddf..503a219e 100644
--- a/packages/forge/src/human/human.ts
+++ b/packages/forge/src/human/human.ts
@@ -19,12 +19,16 @@
* doc.addNode({ mesh: doc.addMesh({ name: "Body", primitives: [{ ...elf.body, material: skin }] }), skin: rig.skin });
* ```
*/
-import type { PrimitiveSpec } from "../document.js";
+import type { DriverSpec, PrimitiveSpec } from "../document.js";
import { Skeleton, type Vec3 } from "../skeleton.js";
import {
type BaseMesh,
+ fitProxy,
+ type MhProxy,
type MhSkeleton,
parseBaseObj,
+ parseObj,
+ parseProxy,
parseSkeleton,
parseTarget,
parseWeights,
@@ -65,13 +69,51 @@ export interface HumanModel {
"positions" | "normals" | "uvs" | "joints" | "weights" | "indices"
>
>;
+ /**
+ * Both eyeballs (sclera, iris, pupil), fitted to the shaped head and
+ * skinned to `LeftEye` / `RightEye`, so `lookAt` on those joints aims
+ * them. Give them a material with `eyeTexture` as its color texture.
+ */
+ eyes: Required<
+ Pick<
+ PrimitiveSpec,
+ "positions" | "normals" | "uvs" | "joints" | "weights" | "indices"
+ >
+ >;
+ /**
+ * The clear corneas over the irises, skinned like `eyes`. Give them a
+ * transmissive material (`transmission`, `ior` ≈ 1.376, `volume.thickness`
+ * ≈ 0.3 cm, the depth of the chamber in front of the iris) and they
+ * refract the iris: it shifts with the viewing angle like a real eye's.
+ */
+ cornea: HumanModel["eyes"];
+ /** MakeHuman's brown eye texture (PNG, CC0): sclera, iris and pupil, mapped to `eyes`' UVs. */
+ eyeTexture: Uint8Array;
+ /**
+ * Rig drivers that come with the body: the upper lids follow the eyes'
+ * up-and-down turn by 65%, the lower lids by 35%, so a gaze aimed with
+ * `lookAt` carries the lids with it. Write them with `GltfDocument.addDriver`
+ * (`addHuman` does).
+ */
+ drivers: DriverSpec[];
/** Model height after shaping and scaling, model units. */
height: number;
}
+/** Lid joints per side: MakeHuman's orbicularis03 (upper) and 04 (lower) pivot at the eyeball's center. */
+const LID_DRIVERS: DriverSpec[] = (["L", "R"] as const).flatMap((side) => {
+ const eye = side === "L" ? "LeftEye" : "RightEye";
+ return [
+ { joint: `orbicularis03.${side}`, from: eye, rotate: { x: 0.65 } },
+ { joint: `orbicularis04.${side}`, from: eye, rotate: { x: 0.35 } },
+ ];
+});
+
/** MakeHuman bone → Mixamo joint name, for the bones Mixamo has. */
const MIXAMO: Record<string, string> = {
root: "Hips",
+ "eye.L": "LeftEye",
+ "eye.R": "RightEye",
spine05: "Spine",
spine03: "Spine1",
spine01: "Spine2",
@@ -200,6 +242,11 @@ const cache = new WeakMap<
base: BaseMesh;
rig: MhSkeleton;
weights: Map<string, [number, number][]>;
+ eyes: {
+ obj: ReturnType<typeof parseObj>;
+ proxy: MhProxy;
+ texture: Uint8Array;
+ };
}>
>();
const targetCache = new WeakMap<
@@ -214,10 +261,18 @@ function baseAssets(store: HumanAssetStore) {
store.text("3dobjs/base.obj"),
store.text("rigs/default.mhskel"),
store.text("rigs/default_weights.mhw"),
- ]).then(([obj, skel, w]) => ({
+ store.text("eyes/high-poly/high-poly.obj"),
+ store.text("eyes/high-poly/high-poly.mhclo"),
+ store.ensure("eyes/materials/brown_eye.png"),
+ ]).then(([obj, skel, w, eyeObj, eyeClo, eyeTex]) => ({
base: parseBaseObj(obj),
rig: parseSkeleton(skel),
weights: parseWeights(w),
+ eyes: {
+ obj: parseObj(eyeObj),
+ proxy: parseProxy(eyeClo),
+ texture: eyeTex,
+ },
}));
cache.set(store, p);
}
@@ -244,7 +299,7 @@ let defaultStore: HumanAssetStore | undefined;
export async function human(options: HumanOptions = {}): Promise<HumanModel> {
defaultStore ??= new HumanAssetStore();
const store = options.store ?? defaultStore;
- const { base, rig, weights } = await baseAssets(store);
+ const { base, rig, weights, eyes } = await baseAssets(store);
const wanted = macroTargets(options);
for (const [path, w] of Object.entries(options.targets ?? {}))
if (w) wanted.set(path, (wanted.get(path) ?? 0) + w);
@@ -315,9 +370,21 @@ export async function human(options: HumanOptions = {}): Promise<HumanModel> {
}
return [sum[0] / vs.length, sum[1] / vs.length, sum[2] / vs.length];
};
+ // Eyes turn about their centers; MakeHuman's eye bones start at the back of the eyeball.
+ const toModel = (x: number, y: number, z: number): Vec3 => [
+ x * scale,
+ (y - minY) * scale,
+ z * scale,
+ ];
+ const fittedEyes = fitEyes(eyes, shaped);
+ const centers = eyeCenters(fittedEyes, toModel);
+ const pivot: Record<string, Vec3> = {
+ "eye.L": centers.left,
+ "eye.R": centers.right,
+ };
for (const b of rig.bones)
skeleton.add(nameOf(b.name), {
- at: jointAt(b.head),
+ at: pivot[b.name] ?? jointAt(b.head),
...(b.parent ? { parent: nameOf(b.parent) } : {}),
});
@@ -356,10 +423,192 @@ export async function human(options: HumanOptions = {}): Promise<HumanModel> {
weights: wts,
indices,
},
+ eyes: eyeballs(eyes.obj, fittedEyes, toModel, skeleton, false),
+ cornea: eyeballs(eyes.obj, fittedEyes, toModel, skeleton, true),
+ eyeTexture: eyes.texture,
+ drivers: LID_DRIVERS,
height: natural * (scale / 10),
};
}
+/** The eye mesh fitted to the shaped head, with each vertex's shell marked. */
+interface FittedEyes {
+ fitted: Float32Array;
+ /** Per vertex: part of a cornea shell (left out). */
+ cornea: Uint8Array;
+}
+
+/**
+ * Each eye in MakeHuman's mesh is two shells: the eyeball and, just
+ * outside it, a clear cornea mapped to the texture's transparent corner.
+ * Per side, the shell reaching further forward is the cornea.
+ */
+function fitEyes(
+ eyes: { obj: ReturnType<typeof parseObj>; proxy: MhProxy },
+ shaped: Float32Array,
+): FittedEyes {
+ const fitted = fitProxy(eyes.proxy, shaped);
+ const nv = fitted.length / 3;
+ const parent = Array.from({ length: nv }, (_, i) => i);
+ const find = (x: number): number => {
+ while (parent[x] !== x) {
+ parent[x] = parent[parent[x] as number] as number;
+ x = parent[x] as number;
+ }
+ return x;
+ };
+ for (const f of eyes.obj.faces)
+ for (const v of f.v) parent[find(v)] = find(f.v[0] as number);
+ const shells = new Map<number, { maxZ: number; side: number }>();
+ for (let v = 0; v < nv; v++) {
+ const r = find(v);
+ const sh = shells.get(r) ?? { maxZ: Number.NEGATIVE_INFINITY, side: 0 };
+ sh.maxZ = Math.max(sh.maxZ, fitted[v * 3 + 2] as number);
+ sh.side += fitted[v * 3] as number;
+ shells.set(r, sh);
+ }
+ const corneaRoots = new Set<number>();
+ for (const side of [-1, 1]) {
+ const mine = [...shells].filter(([, sh]) => Math.sign(sh.side) === side);
+ if (mine.length > 1) {
+ mine.sort((a, b) => b[1].maxZ - a[1].maxZ);
+ corneaRoots.add((mine[0] as [number, unknown])[0]);
+ }
+ }
+ const cornea = new Uint8Array(nv);
+ for (let v = 0; v < nv; v++) if (corneaRoots.has(find(v))) cornea[v] = 1;
+ return { fitted, cornea };
+}
+
+/**
+ * Each eyeball's center, model space: where the eye turns. The shell is
+ * only the front of the ball, so its mean sits forward of the center; the
+ * least-squares sphere through its vertices finds the real one.
+ */
+function eyeCenters(
+ e: FittedEyes,
+ toModel: (x: number, y: number, z: number) => Vec3,
+) {
+ const side = (sign: number): Vec3 => {
+ const pts: Vec3[] = [];
+ for (let v = 0; v < e.cornea.length; v++)
+ if (!e.cornea[v] && Math.sign(e.fitted[v * 3] as number) === sign)
+ pts.push(
+ toModel(
+ e.fitted[v * 3] as number,
+ e.fitted[v * 3 + 1] as number,
+ e.fitted[v * 3 + 2] as number,
+ ),
+ );
+ return sphereCenter(pts);
+ };
+ return { left: side(1), right: side(-1) };
+}
+
+/** Center of the least-squares sphere through points: |p|² = 2c·p + k, solved by normal equations. */
+function sphereCenter(pts: Vec3[]): Vec3 {
+ const A = Array.from({ length: 4 }, () => [0, 0, 0, 0]);
+ const b = [0, 0, 0, 0];
+ for (const p of pts) {
+ const row = [2 * p[0], 2 * p[1], 2 * p[2], 1];
+ const rhs = p[0] * p[0] + p[1] * p[1] + p[2] * p[2];
+ for (let i = 0; i < 4; i++) {
+ b[i] = (b[i] as number) + (row[i] as number) * rhs;
+ for (let j = 0; j < 4; j++)
+ (A[i] as number[])[j] =
+ ((A[i] as number[])[j] as number) +
+ (row[i] as number) * (row[j] as number);
+ }
+ }
+ // Gaussian elimination with partial pivoting on the 4×4 system.
+ const M = A.map((r, i) => [...r, b[i] as number]);
+ for (let c = 0; c < 4; c++) {
+ let piv = c;
+ for (let r = c + 1; r < 4; r++)
+ if (
+ Math.abs((M[r] as number[])[c] as number) >
+ Math.abs((M[piv] as number[])[c] as number)
+ )
+ piv = r;
+ [M[c], M[piv]] = [M[piv] as number[], M[c] as number[]];
+ const d = (M[c] as number[])[c] as number;
+ if (Math.abs(d) < 1e-12) throw new Error("human: eye shell is degenerate");
+ for (let r = 0; r < 4; r++) {
+ if (r === c) continue;
+ const f = ((M[r] as number[])[c] as number) / d;
+ for (let k = c; k < 5; k++)
+ (M[r] as number[])[k] =
+ ((M[r] as number[])[k] as number) -
+ f * ((M[c] as number[])[k] as number);
+ }
+ }
+ const x = M.map((r, i) => (r[4] as number) / (r[i] as number));
+ return [x[0] as number, x[1] as number, x[2] as number];
+}
+
+/** The eyeballs (or their corneas) as a primitive, each moved by its eye joint. */
+function eyeballs(
+ obj: ReturnType<typeof parseObj>,
+ e: FittedEyes,
+ toModel: (x: number, y: number, z: number) => Vec3,
+ skeleton: Skeleton,
+ cornea: boolean,
+): HumanModel["eyes"] {
+ const leftEye = skeleton.joint("LeftEye").index;
+ const rightEye = skeleton.joint("RightEye").index;
+ const corner = new Map<number, number>();
+ const positions: number[] = [];
+ const uvs: number[] = [];
+ const joints: number[] = [];
+ const weights: number[] = [];
+ const indices: number[] = [];
+ const src: number[] = [];
+ const nUv = obj.uvs.length / 2;
+ const { fitted } = e;
+ for (const f of obj.faces) {
+ if (Boolean(e.cornea[f.v[0] as number]) !== cornea) continue;
+ const c = f.v.map((v, k) => {
+ const vt = f.vt[k] as number;
+ const key = v * nUv + vt;
+ let i = corner.get(key);
+ if (i === undefined) {
+ i = src.length;
+ corner.set(key, i);
+ src.push(v);
+ positions.push(
+ ...toModel(
+ fitted[v * 3] as number,
+ fitted[v * 3 + 1] as number,
+ fitted[v * 3 + 2] as number,
+ ),
+ );
+ uvs.push(
+ obj.uvs[vt * 2] as number,
+ 1 - (obj.uvs[vt * 2 + 1] as number),
+ );
+ // The character's left is +x.
+ joints.push(
+ (fitted[v * 3] as number) > 0 ? leftEye : rightEye,
+ 0,
+ 0,
+ 0,
+ );
+ weights.push(1, 0, 0, 0);
+ }
+ return i;
+ });
+ for (let k = 1; k + 1 < c.length; k++)
+ indices.push(c[0] as number, c[k] as number, c[k + 1] as number);
+ }
+ return {
+ positions,
+ normals: vertexNormals(positions, indices, src),
+ uvs,
+ joints,
+ weights,
+ indices,
+ };
+}
/** Smooth normals, averaged across UV seams (vertices with the same source vertex share one). */
function vertexNormals(
positions: number[],
diff --git a/packages/forge/src/human/makehuman.ts b/packages/forge/src/human/makehuman.ts
index bda1edd3..2bc37464 100644
--- a/packages/forge/src/human/makehuman.ts
+++ b/packages/forge/src/human/makehuman.ts
@@ -113,3 +113,115 @@ export function parseWeights(text: string): Map<string, [number, number][]> {
};
return new Map(Object.entries(json.weights));
}
+
+/** A proxy (clothes, eyes) bound to the base mesh: each vertex is a barycentric point on three base vertices plus a scaled offset. */
+export interface MhProxy {
+ /** Per proxy vertex: three base vertex indices. */
+ refs: Uint32Array;
+ /** Per proxy vertex: three weights. */
+ weights: Float32Array;
+ /** Per proxy vertex: offset (decimeters at the reference scale). */
+ offsets: Float32Array;
+ /** Per axis: two base vertices whose distance along it, over `ref`, scales the offsets. */
+ scales: { a: number; b: number; ref: number }[];
+}
+
+export function parseProxy(text: string): MhProxy {
+ const refs: number[] = [];
+ const weights: number[] = [];
+ const offsets: number[] = [];
+ const scales: { a: number; b: number; ref: number }[] = [
+ { a: 0, b: 0, ref: 0 },
+ { a: 0, b: 0, ref: 0 },
+ { a: 0, b: 0, ref: 0 },
+ ];
+ let inVerts = false;
+ for (const raw of text.split("\n")) {
+ const line = raw.trim();
+ if (!line || line.startsWith("#")) continue;
+ const parts = line.split(/\s+/);
+ const head = parts[0] as string;
+ const scale = { x_scale: 0, y_scale: 1, z_scale: 2 }[head];
+ if (scale !== undefined) {
+ scales[scale] = {
+ a: Number(parts[1]),
+ b: Number(parts[2]),
+ ref: Number(parts[3]),
+ };
+ continue;
+ }
+ if (head === "verts") {
+ inVerts = true;
+ continue;
+ }
+ if (!inVerts) continue;
+ if (parts.length < 9 || Number.isNaN(Number(head))) {
+ inVerts = false;
+ continue;
+ }
+ refs.push(Number(parts[0]), Number(parts[1]), Number(parts[2]));
+ weights.push(Number(parts[3]), Number(parts[4]), Number(parts[5]));
+ offsets.push(Number(parts[6]), Number(parts[7]), Number(parts[8]));
+ }
+ return {
+ refs: Uint32Array.from(refs),
+ weights: Float32Array.from(weights),
+ offsets: Float32Array.from(offsets),
+ scales,
+ };
+}
+
+/** The proxy's vertices placed on a (shaped) base mesh, in its units. */
+export function fitProxy(proxy: MhProxy, base: Float32Array): Float32Array {
+ const s = proxy.scales.map(({ a, b, ref }, axis) =>
+ ref
+ ? Math.abs(
+ (base[a * 3 + axis] as number) - (base[b * 3 + axis] as number),
+ ) / ref
+ : 1,
+ );
+ const n = proxy.refs.length / 3;
+ const out = new Float32Array(n * 3);
+ for (let v = 0; v < n; v++)
+ for (let axis = 0; axis < 3; axis++) {
+ let p = 0;
+ for (let k = 0; k < 3; k++)
+ p +=
+ (proxy.weights[v * 3 + k] as number) *
+ (base[(proxy.refs[v * 3 + k] as number) * 3 + axis] as number);
+ out[v * 3 + axis] =
+ p + (proxy.offsets[v * 3 + axis] as number) * (s[axis] as number);
+ }
+ return out;
+}
+
+/** A plain OBJ: positions, UVs and faces (corner vertex and UV indices). */
+export function parseObj(text: string): {
+ positions: Float32Array;
+ uvs: Float32Array;
+ faces: { v: number[]; vt: number[] }[];
+} {
+ const positions: number[] = [];
+ const uvs: number[] = [];
+ const faces: { v: number[]; vt: number[] }[] = [];
+ for (const line of text.split("\n")) {
+ if (line.startsWith("v ")) {
+ const [, x, y, z] = line.trim().split(/\s+/);
+ positions.push(Number(x), Number(y), Number(z));
+ } else if (line.startsWith("vt ")) {
+ const [, u, v] = line.trim().split(/\s+/);
+ uvs.push(Number(u), Number(v));
+ } else if (line.startsWith("f ")) {
+ const corners = line.trim().split(/\s+/).slice(1);
+ faces.push({
+ v: corners.map((c) => Number(c.split("/")[0]) - 1),
+ vt: corners.map((c) => Number(c.split("/")[1]) - 1),
+ });
+ }
+ }
+ return {
+ positions: Float32Array.from(positions),
+ uvs: Float32Array.from(uvs),
+ faces,
+ };
+}
diff --git a/packages/forge/src/index.ts b/packages/forge/src/index.ts
index 6820f4d1..22569d9c 100644
--- a/packages/forge/src/index.ts
+++ b/packages/forge/src/index.ts
@@ -12,6 +12,7 @@ export {
type AnimationPath,
type AnimationSpec,
type ChannelSpec,
+ type DriverSpec,
GltfDocument,
type MaterialRef,
type MeshRef,
@@ -31,6 +32,11 @@ export {
type AccessorType,
GlbBuilder,
} from "./glb-builder.js";
+export {
+ type AddedHuman,
+ type AddHumanOptions,
+ addHuman,
+} from "./human/add-human.js";
export { type HumanModel, type HumanOptions, human } from "./human/human.js";
export { MAKEHUMAN_SOURCE } from "./human/source.js";
export {
diff --git a/packages/framefields/src/glass-refraction-render.test.ts b/packages/framefields/src/glass-refraction-render.test.ts
new file mode 100644
index 00000000..84fec1a1
--- /dev/null
+++ b/packages/framefields/src/glass-refraction-render.test.ts
@@ -0,0 +1,67 @@
+/**
+ * Glass refracts the 3D scene behind it, not only the layers under the
+ * 3D batch: a red box behind a glass sphere shows red through it.
+ */
+import { describe, expect, it } from "vitest";
+import { Composition, Layer, Layer3D } from "./index.js";
+import { decodePngPixels } from "./test-utils/pixels.js";
+
+const S = 300;
+const C = S / 2;
+
+async function centerColor(withBox: boolean) {
+ const comp = new Composition({
+ width: S,
+ height: S,
+ fps: 30,
+ durationFrames: 1,
+ backgroundColor: "#000000",
+ });
+ comp.add([
+ Layer.camera({ x: C, y: C, z: -600, targetX: C, targetY: C, targetZ: 0 }),
+ Layer.environment("night", { intensity: 0.2 }),
+ ...(withBox
+ ? [
+ Layer3D.roundedBox({
+ id: "box",
+ x: C,
+ y: C,
+ z: 200,
+ width: 260,
+ height: 260,
+ depth: 20,
+ radius: 0,
+ color: "#ff2010",
+ material: "unlit",
+ }),
+ ]
+ : []),
+ Layer3D.sphere({
+ id: "lens",
+ x: C,
+ y: C,
+ z: 0,
+ radius: 60,
+ material: "glass",
+ color: "#ffffff",
+ roughness: 0.02,
+ }),
+ ]);
+ const { pixels } = await decodePngPixels(
+ await comp.renderFrame({ frame: 0 }),
+ );
+ const i = (C * S + C) * 4;
+ return [pixels[i] ?? 0, pixels[i + 1] ?? 0, pixels[i + 2] ?? 0];
+}
+
+describe("glass", () => {
+ it("refracts the 3D scene behind it", async () => {
+ const [r, g, b] = await centerColor(true);
+ expect(r).toBeGreaterThan(120);
+ expect(r).toBeGreaterThan(g + 60);
+ expect(r).toBeGreaterThan(b + 60);
+ // Without the box, the same pixel shows the dark background.
+ const [r0] = await centerColor(false);
+ expect(r0).toBeLessThan(60);
+ });
+});
diff --git a/packages/webgpu-renderers/src/loaders/gltf-features.test.ts b/packages/webgpu-renderers/src/loaders/gltf-features.test.ts
index 5ef083b5..40e01583 100644
--- a/packages/webgpu-renderers/src/loaders/gltf-features.test.ts
+++ b/packages/webgpu-renderers/src/loaders/gltf-features.test.ts
@@ -323,7 +323,7 @@ describe("glTF texture features", () => {
});
it("rejects unknown required extensions", () => {
- for (const ext of ["VENDOR_unknown", "KHR_materials_transmission"]) {
+ for (const ext of ["VENDOR_unknown", "KHR_materials_dispersion"]) {
expect(() => parse(quadGltf({}, { extensionsRequired: [ext] }))).toThrow(
ext,
);
diff --git a/packages/webgpu-renderers/src/loaders/gltf-loader.ts b/packages/webgpu-renderers/src/loaders/gltf-loader.ts
index 132d229c..33261370 100644
--- a/packages/webgpu-renderers/src/loaders/gltf-loader.ts
+++ b/packages/webgpu-renderers/src/loaders/gltf-loader.ts
@@ -178,6 +178,9 @@ interface GltfMaterial {
/** VRM 1.0 toon material: the base is lit flat, the shadowed side tinted by shadeColorFactor. */
VRMC_materials_mtoon?: { shadeColorFactor?: [number, number, number] };
KHR_materials_clearcoat?: {
+ KHR_materials_transmission?: { transmissionFactor?: number };
+ KHR_materials_ior?: { ior?: number };
+ KHR_materials_volume?: { thicknessFactor?: number };
clearcoatFactor?: number;
clearcoatRoughnessFactor?: number;
};
@@ -270,7 +273,11 @@ interface GltfDocument {
extensionsUsed?: string[];
extensionsRequired?: string[];
scene?: number;
- scenes?: Array<{ name?: string; nodes?: number[] }>;
+ scenes?: Array<{
+ name?: string;
+ nodes?: number[];
+ extras?: { framefields?: { drivers?: Model3DData["drivers"] } };
+ }>;
nodes?: GltfNode[];
meshes?: GltfMesh[];
materials?: GltfMaterial[];
@@ -292,6 +299,9 @@ const SUPPORTED_REQUIRED_EXTENSIONS = new Set([
"KHR_materials_sheen",
"KHR_materials_anisotropy",
"KHR_materials_iridescence",
+ "KHR_materials_transmission",
+ "KHR_materials_ior",
+ "KHR_materials_volume",
"KHR_texture_transform",
"KHR_texture_basisu",
"EXT_texture_webp",
@@ -927,6 +937,15 @@ export class GltfLoader {
material.shadeColor = mtoon.shadeColorFactor ?? [1, 1, 1];
} else if (mat.extensions?.KHR_materials_unlit) {
material.shading = "unlit";
+ } else if (
+ (mat.extensions?.KHR_materials_transmission?.transmissionFactor ?? 0) > 0
+ ) {
+ // Transmissive surfaces (corneas, windows, potions) render as glass
+ // that refracts the scene behind them.
+ material.shading = "glass";
+ material.ior = mat.extensions?.KHR_materials_ior?.ior ?? 1.5;
+ const t = mat.extensions?.KHR_materials_volume?.thicknessFactor;
+ if (t !== undefined) material.thickness = t;
}
const image = (info: GltfTextureInfo | undefined, slot: string) => {
if (!info) return undefined;
@@ -1103,6 +1122,8 @@ export class GltfLoader {
boundingSphereRadius: 0,
},
};
+ const drivers = doc.scenes?.[doc.scene ?? 0]?.extras?.framefields?.drivers;
+ if (drivers?.length) model.drivers = drivers;
const rest = poseNodes(model, undefined, 0);
let minX = Infinity,
diff --git a/packages/webgpu-renderers/src/loaders/rig.test.ts b/packages/webgpu-renderers/src/loaders/rig.test.ts
index b6f25fb5..5e4b446a 100644
--- a/packages/webgpu-renderers/src/loaders/rig.test.ts
+++ b/packages/webgpu-renderers/src/loaders/rig.test.ts
@@ -3,6 +3,7 @@ import type { Vec3 } from "../math3d/index.js";
import { sampleLocalPose } from "./pose.js";
import {
applyConstraints,
+ applyDriver,
applyJointPose,
blendClipLayers,
RigPose,
@@ -142,6 +143,28 @@ describe("rig", () => {
expect(dist(half.position(2), [0, 200, 0])).toBeGreaterThan(5);
});
+ it("drives a joint by a share of another joint's turn", () => {
+ const rig = rest();
+ // The hip turns 90° about z; tail0 (not its child) follows by half.
+ applyJointPose(rig, {
+ joint: "hip",
+ rotation: [0, 0, Math.SQRT1_2, Math.SQRT1_2],
+ offset: [0, 0, 0],
+ weight: 1,
+ mode: "additive",
+ });
+ applyDriver(rig, { joint: "tail0", from: "hip", rotate: { z: 0.5 } });
+ // tail1 sits 40 along tail0's x: after 45° about z it is at 45°.
+ const p = rig.position(4);
+ const a = (Math.atan2(p[1], p[0] - 50) * 180) / Math.PI;
+ expect(a).toBeCloseTo(45, 3);
+ // Unnamed joints and zero shares change nothing.
+ const still = rest();
+ applyDriver(still, { joint: "nope", from: "hip", rotate: { z: 1 } });
+ applyDriver(still, { joint: "tail0", from: "hip", rotate: { z: 1 } });
+ expect(still.position(4)).toEqual(rest().position(4));
+ });
+
it("turns a joint toward a target within its limit", () => {
const rig = rest();
solveLookAt(rig, {
diff --git a/packages/webgpu-renderers/src/loaders/rig.ts b/packages/webgpu-renderers/src/loaders/rig.ts
index 900219c7..a0a1eb11 100644
--- a/packages/webgpu-renderers/src/loaders/rig.ts
+++ b/packages/webgpu-renderers/src/loaders/rig.ts
@@ -133,12 +133,26 @@ export interface RigAim {
axis?: Vec3;
}
+/**
+ * A joint driven by another: it turns by a share of the source joint's turn
+ * from its rest, per local Euler axis (degrees, Z then Y then X, like
+ * layers). Eyelids follow the eye's pitch, twist bones a share of the limb's
+ * twist, a corrective joint its parent's bend.
+ */
+export interface RigDriver {
+ joint: string;
+ from: string;
+ /** Share of the source's turn about each local axis (1 follows it fully). */
+ rotate: { x?: number; y?: number; z?: number };
+}
+
export interface RigConstraints {
- /** Applied in order: poses, aims, look-ats, then IK. */
+ /** Applied in order: poses, aims, look-ats, drivers, then IK. */
pose?: RigJointPose[];
aim?: RigAim[];
ik?: RigTwoBoneIk[];
lookAt?: RigLookAt[];
+ drive?: RigDriver[];
}
/** A pose being edited: local TRS plus the matching model-space matrices. */
@@ -363,9 +377,51 @@ export function applyConstraints(rig: RigPose, c: RigConstraints): void {
for (const p of c.pose ?? []) applyJointPose(rig, p);
for (const a of c.aim ?? []) solveAim(rig, a);
for (const l of c.lookAt ?? []) solveLookAt(rig, l);
+ for (const d of c.drive ?? []) applyDriver(rig, d);
for (const k of c.ik ?? []) solveTwoBoneIk(rig, k);
}
+/** Euler degrees of R = Rz · Ry · Rx (the layer order) for a unit quaternion. */
+function eulerZyx(q: Quat): Vec3 {
+ const [x, y, z, w] = q;
+ const r11 = 1 - 2 * (y * y + z * z);
+ const r21 = 2 * (x * y + w * z);
+ const r31 = 2 * (x * z - w * y);
+ const r32 = 2 * (y * z + w * x);
+ const r33 = 1 - 2 * (x * x + y * y);
+ const d = 180 / Math.PI;
+ return [
+ Math.atan2(r32, r33) * d,
+ Math.asin(clamp(-r31, -1, 1)) * d,
+ Math.atan2(r21, r11) * d,
+ ];
+}
+
+/** qz · qy · qx for Euler degrees. */
+function quatZyx(e: Vec3): Quat {
+ const h = Math.PI / 360;
+ const qx: Quat = [Math.sin(e[0] * h), 0, 0, Math.cos(e[0] * h)];
+ const qy: Quat = [0, Math.sin(e[1] * h), 0, Math.cos(e[1] * h)];
+ const qz: Quat = [0, 0, Math.sin(e[2] * h), Math.cos(e[2] * h)];
+ return qMul(qz, qMul(qy, qx));
+}
+
+/** Turns a joint by shares of another joint's turn from its rest (see {@link RigDriver}). */
+export function applyDriver(rig: RigPose, d: RigDriver): void {
+ const i = rig.index(d.joint);
+ const src = rig.index(d.from);
+ if (i === undefined || src === undefined) return;
+ const rest = rig.nodes[src]!.rotation;
+ const turn = eulerZyx(qMul(qConj(rest), rig.local.r[src]!));
+ const share: Vec3 = [
+ turn[0] * (d.rotate.x ?? 0),
+ turn[1] * (d.rotate.y ?? 0),
+ turn[2] * (d.rotate.z ?? 0),
+ ];
+ rig.local.r[i] = qMul(rig.local.r[i]!, quatZyx(share));
+ rig.touch();
+}
+
/** Poses one joint (see {@link RigJointPose}). */
export function applyJointPose(rig: RigPose, p: RigJointPose): void {
const i = rig.index(p.joint);
diff --git a/packages/webgpu-renderers/src/loaders/types.ts b/packages/webgpu-renderers/src/loaders/types.ts
index 0fb76ecb..39595add 100644
--- a/packages/webgpu-renderers/src/loaders/types.ts
+++ b/packages/webgpu-renderers/src/loaders/types.ts
@@ -60,7 +60,11 @@ export interface Material3D {
alphaMode?: "opaque" | "mask" | "blend";
alphaCutoff?: number;
/** How the material wants to be lit when the layer leaves `material` unset. */
- shading?: "lit" | "unlit" | "toon";
+ shading?: "lit" | "unlit" | "toon" | "glass";
+ /** Glass (KHR_materials_transmission): index of refraction (KHR_materials_ior). */
+ ior?: number;
+ /** Glass: thickness in model units (KHR_materials_volume thicknessFactor). */
+ thickness?: number;
/** Toon shading: the color multiplier of the shadowed side. */
shadeColor?: [number, number, number];
/** Linear-ish emissive color 0..1 (glTF emissiveFactor). */
@@ -168,6 +172,15 @@ export interface Model3DData {
skins?: Skin3D[];
/** Centering and size normalization, applied above the scene graph instead of baked into vertices. */
rootTransform?: Mat4;
+ /**
+ * Joints the file drives from other joints (scene `extras.framefields.drivers`):
+ * eyelids following the eyes, twist bones, correctives.
+ */
+ drivers?: {
+ joint: string;
+ from: string;
+ rotate: { x?: number; y?: number; z?: number };
+ }[];
animations: AnimationClip3D[];
bounds: {
min: Vec3;
diff --git a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
index 814c2aa6..dcf4a640 100644
--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
@@ -176,6 +176,12 @@ export interface DrawMesh3DOpts extends SurfaceParams {
dispersion?: number;
/** Edge bend for glass materials, px (default 80). */
thickness?: number;
+ /**
+ * Glass thickness in world units (model files: KHR_materials_volume):
+ * the refracted ray's exit point is projected, so the bend follows the
+ * object's size on screen. Used when `thickness` (px) is unset.
+ */
+ worldThickness?: number;
/** Screen-sized picture behind this 3D batch; glass materials refract it. */
backdrop?: GPUTexture;
/**
@@ -2110,7 +2116,12 @@ export class Renderer3D {
? {
ior: opts.ior ?? (material === "acrylic" ? 1.49 : 1.5),
dispersion: opts.dispersion ?? (material === "glass" ? 0.04 : 0.015),
- thickness: opts.thickness ?? 80,
+ // A world thickness travels to the shader negative (see shadeGlass).
+ thickness:
+ opts.thickness ??
+ (opts.worldThickness !== undefined
+ ? -Math.max(1e-4, opts.worldThickness)
+ : 80),
frost: material === "frosted" ? (roughness ?? 0.5) : 0,
}
: undefined;
diff --git a/packages/webgpu-renderers/src/shaders/mesh3d.ts b/packages/webgpu-renderers/src/shaders/mesh3d.ts
index 219d69b6..f74f9a04 100644
--- a/packages/webgpu-renderers/src/shaders/mesh3d.ts
+++ b/packages/webgpu-renderers/src/shaders/mesh3d.ts
@@ -325,8 +325,20 @@ fn shadeGlass(in: VertexOutput, Nin: vec3<f32>, tint: vec3<f32>, alpha: f32, rim
let ior = max(model.pbr.w, 1.0001);
let screen = max(cam.screenParams.xy, vec2<f32>(1.0));
let uv = in.clipPos.xy / screen;
- // Screen-space bend: the normal's in-plane part, stronger toward the rim.
- let bend = N.xy * model.shade.g * (1.0 - 1.0 / ior) * (0.6 + rim) * rimAA / screen;
+ // Screen-space bend. A world thickness (model files, stored negative):
+ // where the refracted ray leaves the material, projected, against where
+ // the straight ray would; it scales with distance like the object does.
+ // Otherwise a bend in px: the normal's in-plane part, stronger toward the rim.
+ var bend = N.xy * model.shade.g * (1.0 - 1.0 / ior) * (0.6 + rim) * rimAA / screen;
+ if (model.shade.g < 0.0) {
+ let t = -model.shade.g;
+ let exitW = in.worldPos + refract(-V, N, 1.0 / ior) * t;
+ let straightW = in.worldPos - V * t;
+ let a = cam.viewProjMatrix * vec4<f32>(exitW, 1.0);
+ let b = cam.viewProjMatrix * vec4<f32>(straightW, 1.0);
+ let dNdc = a.xy / a.w - b.xy / b.w;
+ bend = -vec2<f32>(dNdc.x * 0.5, -dNdc.y * 0.5);
+ }
let disp = model.shade.r;
let frost = model.shade.b * 22.0 / screen;
var col = vec3<f32>(0.0);
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index 1e63af9a..d6a4a2b9 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -431,6 +431,8 @@ const extras = Layer3D.scatter({
});
```
+**Joint drivers** (`drive`): `{ joint, from, rotate: { x?, y?, z? } }` turns a joint by a share of another joint's turn from its rest, per local axis, after look-ats and before IK: lids following eyes, twist bones sharing a forearm's roll, correctives. Model files can carry their own (forge's `GltfDocument.addDriver`, stored in the scene's `extras.framefields.drivers`); a layer's `drive` adds to them.
+
**Posing joints directly** (`pose`, by joint name; namespace prefixes and case are ignored, so `"LeftArm"` finds `"mixamorig:LeftArm"`): local Euler degrees `rotateX/Y/Z` (Z, then Y, then X, like layers) and offsets `x/y/z`, on top of the clips (`mode: "additive"`, default) or of the rest pose (`"rest"`: clips stop moving that joint), with a `weight`. Every value is a track: a number, `[[frame, v]]` keys, `(frame) => v`, or a signal (audio, physics, vision, `Signal.*`), so a head can sway on a sine or flinch at a physics hit:
```ts
@@ -481,6 +483,10 @@ const doc = new GltfDocument();
const rig = doc.addSkeleton(elf.skeleton);
const skin = doc.addMaterial({ color: "#d9b29a", roughness: 0.5, subsurface: { factor: 0.8, color: "#ff5a3c" } });
doc.addNode({ mesh: doc.addMesh({ name: "Body", primitives: [{ ...elf.body, material: skin }] }), skin: rig.skin });
+// Or the whole character in one call: skeleton, body, textured eyes, glass corneas, lid drivers.
+addHuman(doc, elf, { skin: { color: "#e9d2b4", roughness: 0.5, subsurface: { factor: 0.7, color: "#ff6a4a" } } });
+// In the scene: aim the gaze (keep maxAngleDeg near 35 for eyes); the lids follow on their own.
+Layer.glb("elf.glb", { lookAt: [{ joint: "LeftEye", target, maxAngleDeg: 35 }, { joint: "RightEye", target, maxAngleDeg: 35 }] });
```
**Sculpting skinned bodies (`sculpt`)**: a character's body is an ordered list of parts, each an SDF shape (`sphere`, `ellipsoid`, `box`, `cylinder`, `capsule`, `roundCone`, `tube` through points, `torus`, `lathe` profile) with `at`, `rotation`, `op` (`union` / `subtract` / `intersect`), `blend` (smooth join radius), `twist`, `bend`, `displace` (noise detail), `material`, `color`, and the `joint` (or joints) that moves it (default: the nearest bone). `sculpt({ parts, skeleton, cell, triangles })` meshes it closed and manifold at hero resolution, simplifies to the triangle budget, cuts material borders on smooth curves, takes normals and blended colors from the field, and skins every vertex from the parts that reach it (smooth joins share weights, `rigid: true` parts move with one joint), returning one primitive per material plus a report (`openEdges` is 0 for a closed mesh). `limb(skeleton, "LeftForeArm", { radii: [4, 3], blend: 3 })` puts a round cone along a bone; `mirrorParts(leftParts)` adds the right side.
@@ -602,6 +608,7 @@ Layer3D.cylinder({ ..., height: 400, color: "#111", emissive: "#4da3ff", emissiv
- **Shadows:** `directional`, `spot`, `rect` (`Layer.rectLight` / `Light.rect`), and omnidirectional `point` (`Layer.pointLight` / `Light.point`, 6-face cube shadow map) lights can each cast shadows, subject to device texture-array limits. Meshes cast by default (`castShadow: false` opts out); flat layers (planes, cards) cast only with `castShadow: true`. Everything receives.
- **Rect Area Lights (`Layer.rectLight` / `Light.rect`):** studio softbox / LED panel lights with `width`, `height`, `twoSided`, `rotation`, `x/y/z`, `targetX/Y/Z`, `intensity`, `radius`, `decay`, and `castShadow`. Uses solid-angle diffuse irradiance and representative-point GGX specular highlights that stretch across rough and glossy surfaces. Numeric light props accept `Signal`s directly (e.g. `Layer.rectLight({ intensity: bassSignal, width: 300, height: 120 })`).
- **Volumetric Fog & Light Shafts (`volumetricFog`):** `Layer.environment("night", { volumetricFog: true | { density, anisotropy, heightFalloff, baseHeight, color, lightIntensity, ambientIntensity, maxDistance, steps, noise: { scale, strength, speed, octaves } } })` raymarches participating media at half resolution with depth-aware bilateral upsampling. Any shadow-casting `directional`, `spot`, `rect`, or `point` light automatically carves volumetric light shafts (god rays) through the fog behind occluders. `density`, `anisotropy`, and `noise.strength` accept numbers or live `Signal` objects directly.
+- **Glass refracts the 3D scene:** glass, acrylic and frosted meshes show the 3D objects behind them as well as the layers under the batch (the batch is drawn once without its glass to make that picture). glTF materials with `KHR_materials_transmission` render as glass with their `KHR_materials_ior`; a `KHR_materials_volume` thickness bends by the refracted ray through that depth, so the bend scales with the object on screen (corneas, potions, ice).
- **Model files bring their own lobes:** glTF materials with `KHR_materials_clearcoat`, `_sheen`, `_anisotropy` or `_iridescence`, and forge's `subsurface` (written to `extras.framefields.subsurface`), render with those lobes; a layer's own lobe props override them. Subsurface scattering conserves energy: each channel's light wraps past the terminator by how far it scatters (`subsurfaceColor`; red farthest for skin, e.g. `"#ff5a3c"`), so form keeps its shading and only shadow edges warm; thin parts glow when back-lit.
- **Substrate lobes (physical shading, needs an environment):** every `Layer3D.*` mesh, `Layer.model` and 3D card accepts `clearcoat` / `clearcoatRoughness` (car paint, lacquer), `anisotropy` (-1..1) / `anisotropyRotation` (deg; brushed metal), `subsurface` / `subsurfaceColor` (wax, jade, skin), `sheen` / `sheenColor` / `sheenRoughness` (velvet, cloth), `iridescence` (0..1 or `{ intensity, thickness, ior }`; soap, oil, beetle shell) with `iridescenceThickness` (nm) / `iridescenceIor`, and `triplanar: true | { scale, sharpness }` for world-space textures on UV-less meshes. Numeric lobes take numbers or live `Signal`s (`clearcoat: bass`). Example: `Layer3D.sphere({ radius: 90, color: "#8a0f1c", metallic: 0.6, roughness: 0.35, clearcoat: 1, clearcoatRoughness: 0.05 })`.
- **HDRI environments:** pass a `.hdr` or `.exr` path instead of a preset: `Layer.environment("assets/hdri/studio.exr", { intensity: 1.2, rotation: 90 })` (or `{ hdri: path }`). The panorama lights and reflects every physical surface; preset names keep working; relative paths resolve from the working directory. EXR files must use lossless compression (NONE, RLE, ZIPS, ZIP, PIZ or PXR24); re-save B44/DWA files as ZIP.
diff --git a/specs/character-forge.md b/specs/character-forge.md
index 74a66fce..43898510 100644
--- a/specs/character-forge.md
+++ b/specs/character-forge.md
@@ -1,6 +1,6 @@
# Character Forge: rigged characters built in code
-Status: Phases 1–2 done (GLB writer, skeleton, humanoid preset, bone gizmo; bound sculpting, manifold narrow-band meshing, simplification, smooth material borders, skin weights from the field; `packages/forge`, exposed as `framefields/forge`). Sharp-edge preservation for hard surfaces moves to Phase 7's hard-surface work. Human bodies and faces now start from MakeHuman's CC0 base (`human()`): realistic anatomy, edge loops and UVs, macro and feature targets, painted weights. `sculpt` stays for stylization, costume, armor and non-humans. The face rig (Phase 4) builds on MakeHuman's expression units and the ARKit 52 names. Phase 3 in progress: the loader reads clearcoat, sheen, anisotropy and iridescence and forge's `extras.framefields.subsurface`, the compositor uses them under layer overrides, and the subsurface lobe was rewritten to conserve energy (per-channel wrap). Still to do in Phase 3: alpha-to-coverage for hair cards and morphs combined with skinning; the specular, IOR, transmission and volume extensions are written but not yet rendered. First character: **Ithrael Sunmourn** (`examples/36_emberwake/CHARACTERS.md`), an original hooded elven ranger with a long cloak, long hair, a sculpted face, real eyes, and a full set of expressions.
+Status: Phases 1–2 done (GLB writer, skeleton, humanoid preset, bone gizmo; bound sculpting, manifold narrow-band meshing, simplification, smooth material borders, skin weights from the field; `packages/forge`, exposed as `framefields/forge`). Sharp-edge preservation for hard surfaces moves to Phase 7's hard-surface work. Human bodies and faces now start from MakeHuman's CC0 base (`human()`): realistic anatomy, edge loops and UVs, macro and feature targets, painted weights. `sculpt` stays for stylization, costume, armor and non-humans. The face rig (Phase 4) builds on MakeHuman's expression units and the ARKit 52 names. Phase 3 in progress: the loader reads clearcoat, sheen, anisotropy and iridescence and forge's `extras.framefields.subsurface`, the compositor uses them under layer overrides, and the subsurface lobe was rewritten to conserve energy (per-channel wrap). Eyes (first step of Phase 5): `human()` returns MakeHuman's eyeballs fitted to the shaped head (proxy binding), skinned to Mixamo `LeftEye` / `RightEye` pivoted at each ball's least-squares sphere center, plus the CC0 iris texture; a clear coat gives the wet highlight. Corneas are back as glass: glass now refracts the 3D scene behind it (a pre-pass of the batch without glass), and glTF transmission + IOR + volume thickness map to glass whose bend follows the refracted ray through that depth, so the iris shifts with the viewing angle. Lids follow the gaze through joint drivers (`drive`, carried in the GLB), and `addHuman` writes the whole character. Still to do in Phase 3: alpha-to-coverage for hair cards and morphs combined with skinning; the specular, IOR, transmission and volume extensions are written but not yet rendered. First character: **Ithrael Sunmourn** (`examples/36_emberwake/CHARACTERS.md`), an original hooded elven ranger with a long cloak, long hair, a sculpted face, real eyes, and a full set of expressions.
## Goal