Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(3d): direct joint posing, bone aims and pose retargeting from vision

okanasl7bfc4bc607d3
diff --git a/nodes/node-compositor/SKILL.md b/nodes/node-compositor/SKILL.md
index d2dffbf4..d7e58793 100644
--- a/nodes/node-compositor/SKILL.md
+++ b/nodes/node-compositor/SKILL.md
@@ -229,6 +229,7 @@ Each track represents animatable property modifications:
- **glTF scenes:** a glTF/GLB `model3d` keeps its node hierarchy, every skin and its animation clips (translation, rotation, scale and morph weights; linear, step and cubic-spline). Draco and meshopt compressed geometry and all seven core primitive modes are supported; per-map UV sets and texture transforms are independent. Basis KTX2 (ETC1S/UASTC) textures transcode offline to RGBA8 with their mip levels. Unsupported required extensions fail clearly. Morph targets run on the GPU before skinning and audio deformation. Skinned meshes are posed per frame from the clip (`animationName`, `animationTime`/`animationProgress`, `loop`); base-colour textures (with mipmaps), alpha `MASK`/`BLEND` and VRM MToon materials (`material: "toon"`, picked automatically when `material` is unset) render as authored. `center`/`normalizeSize` apply above the scene graph, so skinning stays intact.
- **Procedural Shards (`Layer3D.shard`):** Procedural polyhedron primitive generated by Sutherland-Hodgman convex half-space clipping against Fibonacci lattice normals with PCG jitter. Faceted flat normals, automatic planar UVs, Rapier 3D convex hull collision.
- **Character rigs (`animationLayers`, `ik`, `lookAt`, `springBones` on a glTF `model3d`):** clips blend in order from the rest pose, each over the pose beneath by `weight` (number, `[[frame, w]]` keys, or `(frame) => w` in process), slerped per joint and restricted to `mask` joints and their descendants; `speed`, `timeOffset`, `loop` per layer. Then `lookAt` (joint `axis`, default glTF +z, clamped to `maxAngleDeg`) and analytic two-bone `ik` (`chain: [root, mid, effector]`, `poleTarget` sets the bend side); targets are scene coordinates (point, keys or function) mapped into model space through the model's transform. `springBones` (each root's chain down its first child; `stiffness`, `damping`, scene `gravity`) are Verlet-simulated from frame 0 and cached per layer, so frames come out the same in any order. A node without these keeps the single-clip path.
+- **Joint poses and aims (`pose`, `aim` on a glTF `model3d`):** after clip layers, `pose` (joint name → Euler degrees Z·Y·X and offsets, `additive` over the animated pose or from `rest`, weighted) then `aim` (bone toward its first child, or `axis`, swung onto a scene direction mapped into model space) run before look-at and IK. Joint names match exactly, else without a namespace prefix and case. Every number in `pose`, `aim`, `ik`, `lookAt` and clip weights is a track (number, keys, function, or a signal read with the render's clock); points take one track per axis. Readers are shared with the SDK (`numberAt` / `vec3At` in `compositions/program/track-sampler.ts`).
- **Skinned crowds (`crowd` on an instances node):** `{ clips, fps (30), seed, speedRange, phase: "random" | "sync" }`; each mesh of each variant model bakes its joint matrices (or its node matrix, for rigid node meshes) for every clip once (`count + 1` samples per loop) into a GPU palette; instances take a second record (clip range, phase, playback rate) and skin in the vertex shader, interpolating two baked samples. Per instance `clip`, `animOffset`, `animSpeed` override the seeded defaults. One draw call per mesh, shadows included. Instance (and scatter) variants apply their own node `rotateX/Y/Z` and `scale` under each instance's transform.
- **Voronoi destruction (`fracture3d` on a model node):** `{ cells, seed, impactPoint (object space), interiorColor, interiorRoughness, track }`. The compositor fractures the loaded source model (`fractureModel`, cached per model and spec) into one rigidly skinned mesh (a joint per piece), then poses the joints from `track` (x, y, z, qx, qy, qz, qw per piece per frame, in the layer's parent space, as physics reads its x/y/z): node matrix L⁻¹ · T(p) · R(q) · S, with L the layer's own prop matrix, so pieces stay in their 3D group. While every piece is still at its frame-0 pose (or there is no track) it draws in the exact rest pose with the cut faces skipped, so the whole layer is sealed (a cut face can otherwise tie the surface's depth where pieces meet).
- **Shutter motion blur (`motionBlur` on the camera):** after the color pass (and fog, DoF), the queued meshes and planes are drawn again into a velocity target, with this and the previous frame's model matrices, skinning, instance records (GPU copies), wind and crowd time, and this and the previous camera. Then the scene is blurred (tile max, neighbor max, gather) before bloom. Previous state lives in a per-render motion history keyed by node id and item index. When frame f - 1 is missing, the compositor first renders it with `motionHistoryOnly` (3D passes only record), so output does not depend on render order.
diff --git a/nodes/node-compositor/src/renderers/character.ts b/nodes/node-compositor/src/renderers/character.ts
index 587d865d..e6d58e82 100644
--- a/nodes/node-compositor/src/renderers/character.ts
+++ b/nodes/node-compositor/src/renderers/character.ts
@@ -4,13 +4,17 @@
* spring bones (replayed from frame 0 and cached, so frames render the
* same in any order); and the per-instance records of skinned crowds.
*/
-import type {
- CrowdConfig,
- LookAtConstraint,
- Model3DNode,
- NumberTrack,
- TwoBoneIkConstraint,
- Vec3Track,
+import {
+ type AimConstraint,
+ type CrowdConfig,
+ type JointPose,
+ type LookAtConstraint,
+ type Model3DNode,
+ numberAt,
+ type TrackClock,
+ type TwoBoneIkConstraint,
+ type Vec3Track,
+ vec3At,
} from "@framefields/compositions/program";
import {
type AnimationClip3D,
@@ -23,49 +27,16 @@ import {
Matrix4Math,
type Model3DData,
poseNodes,
+ type Quat,
type RigClipLayer,
type RigConstraints,
+ type RigJointPose,
type RigPose,
resolveClips,
SpringBones,
type Vec3,
} from "@framefields/webgpu-renderers";
-/** A number track's value at `frame`. */
-export function numberAt(
- track: NumberTrack | undefined,
- frame: number,
- fallback: number,
-): number {
- if (track === undefined) return fallback;
- if (typeof track === "number") return track;
- if (typeof track === "function") return track(frame);
- return keyed(track, frame, 1)[0] ?? fallback;
-}
-
-/** A point track's value at `frame`. */
-export function vec3At(track: Vec3Track, frame: number): Vec3 {
- if (typeof track === "function") return track(frame);
- if (track.length === 3 && typeof track[0] === "number")
- return track as unknown as Vec3;
- return keyed(track as number[][], frame, 3) as Vec3;
-}
-
-/** Linear interpolation over `[[frame, …values]]` keys, held past the ends. */
-function keyed(keys: number[][], frame: number, size: number): number[] {
- const sorted = keys;
- const first = sorted[0]!;
- const last = sorted[sorted.length - 1]!;
- if (frame <= first[0]!) return first.slice(1, 1 + size);
- if (frame >= last[0]!) return last.slice(1, 1 + size);
- let k = 0;
- while (k + 1 < sorted.length && sorted[k + 1]![0]! <= frame) k++;
- const a = sorted[k]!;
- const b = sorted[k + 1]!;
- const u = (frame - a[0]!) / Math.max(1e-9, b[0]! - a[0]!);
- return a.slice(1, 1 + size).map((v, i) => v + (b[i + 1]! - v) * u);
-}
-
const transformPoint = (m: Mat4, p: Vec3): Vec3 => [
m[0] * p[0] + m[4] * p[1] + m[8] * p[2] + m[12],
m[1] * p[0] + m[5] * p[1] + m[9] * p[2] + m[13],
@@ -79,7 +50,7 @@ const transformDir = (m: Mat4, v: Vec3): Vec3 => [
type CharacterNode = Pick<
Model3DNode,
- "animationLayers" | "ik" | "lookAt" | "springBones" | "loop"
+ "animationLayers" | "ik" | "lookAt" | "springBones" | "loop" | "pose" | "aim"
>;
/** Whether a node needs the rig evaluator (anything beyond one plain clip). */
@@ -88,7 +59,44 @@ export function hasRig(n: CharacterNode): boolean {
n.animationLayers?.length ||
n.ik?.length ||
n.lookAt?.length ||
- n.springBones?.length,
+ n.springBones?.length ||
+ (n.pose && Object.keys(n.pose).length) ||
+ n.aim?.length,
+ );
+}
+
+/** Local rotation for Euler degrees, applied Z, then Y, then X (R = Rz · Ry · Rx), like layers. */
+function eulerQuat(rx: number, ry: number, rz: number): Quat {
+ const h = (d: number) => (d * Math.PI) / 360;
+ const [sx, cx] = [Math.sin(h(rx)), Math.cos(h(rx))];
+ const [sy, cy] = [Math.sin(h(ry)), Math.cos(h(ry))];
+ const [sz, cz] = [Math.sin(h(rz)), Math.cos(h(rz))];
+ // qz · qy · qx
+ return [
+ cz * cy * sx - sz * sy * cx,
+ cz * sy * cx + sz * cy * sx,
+ sz * cy * cx - cz * sy * sx,
+ cz * cy * cx + sz * sy * sx,
+ ];
+}
+
+function jointPoses(p: PoseInput, clock: TrackClock): RigJointPose[] {
+ return Object.entries(p.node.pose ?? {}).map(
+ ([joint, j]: [string, JointPose]) => ({
+ joint,
+ rotation: eulerQuat(
+ numberAt(j.rotateX, clock, 0),
+ numberAt(j.rotateY, clock, 0),
+ numberAt(j.rotateZ, clock, 0),
+ ),
+ offset: [
+ numberAt(j.x, clock, 0),
+ numberAt(j.y, clock, 0),
+ numberAt(j.z, clock, 0),
+ ] as Vec3,
+ weight: numberAt(j.weight, clock, 1),
+ mode: j.mode ?? "additive",
+ }),
);
}
@@ -126,7 +134,7 @@ function clipLayers(p: PoseInput, frame: number): RigClipLayer[] {
out.push({
clip,
time,
- weight: numberAt(l.weight, frame, 1),
+ weight: numberAt(l.weight, { frame, fps: p.fps }, 1),
mask: l.mask,
});
}
@@ -138,18 +146,27 @@ function constraintsAt(
frame: number,
toModel: Mat4,
): RigConstraints {
- const point = (t: Vec3Track) => transformPoint(toModel, vec3At(t, frame));
+ const clock: TrackClock = { frame, fps: p.fps };
+ const point = (t: Vec3Track) => transformPoint(toModel, vec3At(t, clock));
+ const direction = (t: Vec3Track) => transformDir(toModel, vec3At(t, clock));
return {
+ pose: jointPoses(p, clock),
+ aim: (p.node.aim ?? []).map((a: AimConstraint) => ({
+ joint: a.joint,
+ direction: direction(a.direction),
+ weight: numberAt(a.weight, clock, 1),
+ ...(a.axis && { axis: a.axis }),
+ })),
ik: (p.node.ik ?? []).map((c: TwoBoneIkConstraint) => ({
chain: c.chain,
target: point(c.target),
pole: c.poleTarget ? point(c.poleTarget) : undefined,
- weight: numberAt(c.weight, frame, 1),
+ weight: numberAt(c.weight, clock, 1),
})),
lookAt: (p.node.lookAt ?? []).map((c: LookAtConstraint) => ({
joint: c.joint,
target: point(c.target),
- weight: numberAt(c.weight, frame, 1),
+ weight: numberAt(c.weight, clock, 1),
maxAngleDeg: c.maxAngleDeg ?? 70,
axis: c.axis ?? [0, 0, 1],
})),
@@ -180,7 +197,13 @@ function springsAt(
): SpringBones | undefined {
const chains = p.node.springBones;
if (!chains?.length) return undefined;
- const config = JSON.stringify([chains, p.node.animationLayers, p.fps]);
+ const config = JSON.stringify([
+ chains,
+ p.node.animationLayers,
+ p.node.pose,
+ p.node.aim,
+ p.fps,
+ ]);
let s = springStates.get(p.key);
const frame = Math.max(0, Math.floor(p.frame));
if (!s || s.model !== p.model || s.config !== config || s.frame > frame) {
diff --git a/packages/compositions/src/program/character-schema.ts b/packages/compositions/src/program/character-schema.ts
index 98f39e99..a76c9466 100644
--- a/packages/compositions/src/program/character-schema.ts
+++ b/packages/compositions/src/program/character-schema.ts
@@ -4,24 +4,43 @@
*
* Time-varying values are tracks: a constant, keyframes `[[frame, …value]]`
* (linear, in the layer's own frames), or, in process, a function of the
- * frame (not serializable; a spec sent as JSON keeps constants and keys).
+ * frame or a signal (anything with `get(ctx)`: audio, physics, vision,
+ * `Signal.*`). Functions and signals are not serializable; a spec sent as
+ * JSON keeps constants and keys.
*/
import { z } from "zod";
const Vec3Schema = z.tuple([z.number(), z.number(), z.number()]);
const isFunction = (v: unknown) => typeof v === "function";
-/** A number over time: constant, `[[frame, value], …]` keys, or `(frame) => value`. */
+/** A frame signal: anything with `get(ctx)` (`Signal.*`, audio, physics, vision). */
+export interface TrackSignal {
+ get(ctx?: unknown): number;
+}
+const isSignal = (v: unknown): v is TrackSignal =>
+ typeof v === "object" &&
+ v !== null &&
+ typeof (v as { get?: unknown }).get === "function";
+
+/**
+ * A number over time: constant, `[[frame, value], …]` keys, `(frame) =>
+ * value`, or a signal.
+ */
export const NumberTrackSchema = z.union([
z.number(),
z.array(z.tuple([z.number(), z.number()])).min(1),
z.custom<(frame: number) => number>(isFunction),
+ z.custom<TrackSignal>(isSignal),
]);
export type NumberTrack = z.infer<typeof NumberTrackSchema>;
-/** A point over time: `[x, y, z]`, `[[frame, x, y, z], …]` keys, or `(frame) => [x, y, z]`. */
+/**
+ * A point over time: `[x, y, z]` where each axis is a {@link NumberTrack}
+ * (so a constant, keys, a function or a signal per axis), `[[frame, x, y,
+ * z], …]` keys, or `(frame) => [x, y, z]`.
+ */
export const Vec3TrackSchema = z.union([
- Vec3Schema,
+ z.tuple([NumberTrackSchema, NumberTrackSchema, NumberTrackSchema]),
z.array(z.tuple([z.number(), z.number(), z.number(), z.number()])).min(1),
z.custom<(frame: number) => [number, number, number]>(isFunction),
]);
@@ -74,6 +93,44 @@ export const LookAtSchema = z
.strict();
export type LookAtConstraint = z.infer<typeof LookAtSchema>;
+/**
+ * A joint posed directly, in its own local frame: Euler degrees (applied
+ * Z, then Y, then X, like layers) and a translation offset in model units,
+ * on top of the animated pose (`"additive"`, default) or of the joint's rest
+ * pose (`"rest"`: clips no longer move it).
+ */
+export const JointPoseSchema = z
+ .object({
+ rotateX: NumberTrackSchema.optional(),
+ rotateY: NumberTrackSchema.optional(),
+ rotateZ: NumberTrackSchema.optional(),
+ x: NumberTrackSchema.optional(),
+ y: NumberTrackSchema.optional(),
+ z: NumberTrackSchema.optional(),
+ /** 0..1. Default 1. */
+ weight: NumberTrackSchema.optional(),
+ mode: z.enum(["additive", "rest"]).optional(),
+ })
+ .strict();
+export type JointPose = z.infer<typeof JointPoseSchema>;
+
+/**
+ * Points a bone along a scene direction (the joint turns so the bone,
+ * toward its first child, or its local `axis` lies along it). Retargeting a
+ * performer's pose is a chain of aims, parents first.
+ */
+export const AimSchema = z
+ .object({
+ joint: z.string(),
+ /** Scene direction (any length); `[0, 0, 0]` holds the pose. */
+ direction: Vec3TrackSchema,
+ weight: NumberTrackSchema.optional(),
+ /** The joint's local axis to aim. Default: toward its first child. */
+ axis: Vec3Schema.optional(),
+ })
+ .strict();
+export type AimConstraint = z.infer<typeof AimSchema>;
+
export const SpringBoneChainSchema = z
.object({
/** Root joints of chains (cape, tail, antenna, hair); each runs down its first child. */
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index 947c13df..422adf3c 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -28,8 +28,12 @@ import {
} from "@framefields/core";
import { ColorSchema } from "@framefields/node-sdk";
import {
+ type AimConstraint,
+ AimSchema,
type CrowdConfig,
CrowdSchema,
+ type JointPose,
+ JointPoseSchema,
type LookAtConstraint,
LookAtSchema,
type ModelAnimationLayer,
@@ -1847,6 +1851,10 @@ export interface Model3DNode {
lookAt?: LookAtConstraint[];
/** Secondary motion simulated from frame 0 (capes, tails, hair). */
springBones?: SpringBoneChain[];
+ /** Joints posed directly by name (local Euler degrees, offsets), over the clips. */
+ pose?: Record<string, JointPose>;
+ /** Bones pointed along scene directions, parents first (pose retargeting). */
+ aim?: AimConstraint[];
/** Instance nodes: every instance a skinned character on its own clip, phase and speed. */
crowd?: CrowdConfig;
}
@@ -2182,6 +2190,8 @@ export const Model3DNodeSchema: z.ZodType<Model3DNode> = z
ik: z.array(TwoBoneIkSchema).optional(),
lookAt: z.array(LookAtSchema).optional(),
springBones: z.array(SpringBoneChainSchema).optional(),
+ pose: z.record(z.string(), JointPoseSchema).optional(),
+ aim: z.array(AimSchema).optional(),
crowd: CrowdSchema.optional(),
})
.strict() as unknown as z.ZodType<Model3DNode>;
diff --git a/packages/compositions/src/program/track-sampler.ts b/packages/compositions/src/program/track-sampler.ts
index d34d7bcb..0982e34b 100644
--- a/packages/compositions/src/program/track-sampler.ts
+++ b/packages/compositions/src/program/track-sampler.ts
@@ -6,6 +6,7 @@
* with the picture outside a render (physics following a layer) share it.
*/
import gsapModule from "gsap";
+import type { NumberTrack, Vec3Track } from "./character-schema.js";
import { evaluateKeyframes, evaluateTrackAtFrame } from "./evaluator.js";
import type { AnimationTrack, EaseRef } from "./schema.js";
@@ -241,3 +242,70 @@ export function sampleLayerProp(
}
return value;
}
+
+// --- Character tracks (pose, IK and aim targets, weights) -------------------
+
+/** What a track reads: the layer's frame, and the clock signals expect. */
+export interface TrackClock {
+ frame: number;
+ fps: number;
+}
+
+const signalContext = (c: TrackClock) => ({
+ frame: c.frame,
+ fps: c.fps,
+ time: c.frame / c.fps,
+ duration: 0,
+ durationMs: 0,
+ progress: 0,
+ deltaTime: 1 / c.fps,
+});
+
+/** A number track's value: constant, keys, function, or signal. */
+export function numberAt(
+ track: NumberTrack | undefined,
+ clock: TrackClock,
+ fallback: number,
+): number {
+ if (track === undefined) return fallback;
+ if (typeof track === "number") return track;
+ if (typeof track === "function") return track(clock.frame);
+ if (Array.isArray(track)) return keyed(track, clock.frame, 1)[0] ?? fallback;
+ const v = track.get(signalContext(clock));
+ return Number.isFinite(v) ? v : fallback;
+}
+
+/** Whether a point track is `[[frame, x, y, z], …]` keys (not one track per axis). */
+const isPointKeys = (track: unknown[]): track is number[][] =>
+ Array.isArray(track[0]) && (track[0] as unknown[]).length === 4;
+
+/** A point track's value: per-axis tracks, keys, or a function. */
+export function vec3At(
+ track: Vec3Track,
+ clock: TrackClock,
+): [number, number, number] {
+ if (typeof track === "function") return track(clock.frame);
+ if (isPointKeys(track))
+ return keyed(track, clock.frame, 3) as [number, number, number];
+ const axes = track as [NumberTrack, NumberTrack, NumberTrack];
+ return [
+ numberAt(axes[0], clock, 0),
+ numberAt(axes[1], clock, 0),
+ numberAt(axes[2], clock, 0),
+ ];
+}
+
+/** Linear interpolation over `[[frame, …values]]` keys, held past the ends. */
+function keyed(keys: number[][], frame: number, size: number): number[] {
+ const sorted = keys;
+ const first = sorted[0]!;
+ const last = sorted[sorted.length - 1]!;
+ if (frame <= first[0]!) return first.slice(1, 1 + size);
+ if (frame >= last[0]!) return last.slice(1, 1 + size);
+ let k = 0;
+ while (k + 1 < sorted.length && sorted[k + 1]![0]! <= frame) k++;
+ const a = sorted[k]!;
+ const b = sorted[k + 1]!;
+ const u = (frame - a[0]!) / Math.max(1e-9, b[0]! - a[0]!);
+ return a.slice(1, 1 + size).map((v, i) => v + (b[i + 1]! - v) * u);
+}
diff --git a/packages/framefields/src/character-render.test.ts b/packages/framefields/src/character-render.test.ts
index 6a7f9111..b12bdcbb 100644
--- a/packages/framefields/src/character-render.test.ts
+++ b/packages/framefields/src/character-render.test.ts
@@ -15,6 +15,9 @@ import {
Layer,
Layer3D,
type LayoutNode,
+ type PoseSource,
+ retargetPose,
+ Signal,
} from "./index.js";
import { GlbBuilder } from "./test-utils/glb.js";
import { decodePngPixels } from "./test-utils/pixels.js";
@@ -311,6 +314,28 @@ describe("character animation", () => {
ik: [{ chain: ["root", "knee", "foot"], target: [[0, 1, 2, 3]] }],
lookAt: [{ joint: "root", target: [0, 0, 0], maxAngleDeg: 30 }],
springBones: [{ joints: ["knee"], gravity: [0, 980, 0] }],
+ pose: {
+ knee: {
+ rotateZ: [
+ [0, 0],
+ [30, 45],
+ ],
+ mode: "additive",
+ },
+ },
+ aim: [
+ {
+ joint: "root",
+ direction: [
+ 1,
+ [
+ [0, 0],
+ [30, 1],
+ ],
+ 0,
+ ],
+ },
+ ],
}),
Layer3D.instances({
mesh: leg(),
@@ -322,4 +347,89 @@ describe("character animation", () => {
const bad = scene([leg({ springBones: [{ joints: [] }] })]);
expect(CompositorProgramSchema.safeParse(bad.toSpec()).success).toBe(false);
});
+
+ it("poses joints directly, from numbers, keys or signals", async () => {
+ // Additive: the knee turns +90° about z over the (still) pose.
+ const folded = await probe(
+ scene([leg({ pose: { knee: { rotateZ: 90 } } })]),
+ );
+ expect(folded(-60, 80)).toBe(true);
+ expect(folded(0, 140)).toBe(false);
+ // A signal drives it, read with the render's clock.
+ const swing = Signal.builder({
+ type: "custom",
+ fn: (ctx) => -3 * ctx.frame,
+ });
+ const comp = scene([leg({ pose: { knee: { rotateZ: swing } } })]);
+ const curled = await probe(comp, 30);
+ expect(curled(60, 80)).toBe(true);
+ // Keyed weight fades the override in.
+ const fading = scene([
+ leg({
+ pose: {
+ knee: {
+ rotateZ: 90,
+ weight: [
+ [0, 0],
+ [30, 1],
+ ],
+ },
+ },
+ }),
+ ]);
+ expect((await probe(fading, 0))(0, 140)).toBe(true);
+ // (Frame 30: the looping fold clip underneath is back at rest.)
+ expect((await probe(fading, 30))(-60, 80)).toBe(true);
+ });
+
+ it("poses from rest over a clip", async () => {
+ const lit = await probe(
+ scene([
+ leg({
+ animationLayers: [{ clip: "fold", timeOffset: 1, loop: false }],
+ pose: { knee: { mode: "rest" } },
+ }),
+ ]),
+ );
+ // The clip folds the knee; the rest pose override straightens it.
+ expect(lit(0, 140)).toBe(true);
+ });
+
+ it("aims bones along scene directions", async () => {
+ const lit = await probe(
+ scene([leg({ aim: [{ joint: "root", direction: [1, 0, 0] }] })]),
+ );
+ expect(lit(150, 0)).toBe(true);
+ expect(lit(0, 150)).toBe(false);
+ });
+
+ it("retargets a performer's 2D pose onto the rig", async () => {
+ // Screen px: hip at the origin, knee to the right, ankle below the knee.
+ const pose = (visibility: number): PoseSource => ({
+ point: (name) =>
+ ({
+ leftHip: { x: 0, y: 0, visibility },
+ leftKnee: { x: 100, y: 0, visibility },
+ leftAnkle: { x: 100, y: 100, visibility },
+ })[name as "leftHip"] ?? { x: 0, y: 0, visibility: 0 },
+ });
+ const rig = { leftUpperLeg: "root", leftLowerLeg: "knee" };
+ const lit = await probe(
+ scene([leg({ aim: retargetPose(pose(1), { rig }) })]),
+ );
+ // Thigh along +x to the knee at (80, 0), shin down the screen (model −y).
+ expect(lit(40, 0)).toBe(true);
+ expect(lit(80, -40)).toBe(true);
+ expect(lit(0, 140)).toBe(false);
+ // Unsure keypoints hand the bones back: the leg stands as animated.
+ const unsure = await probe(
+ scene([leg({ aim: retargetPose(pose(0.1), { rig }) })]),
+ );
+ expect(unsure(0, 140)).toBe(true);
+ // Mirrored: the performer's left drives the right side, unmapped here.
+ const mirrored = await probe(
+ scene([leg({ aim: retargetPose(pose(1), { rig, mirror: true }) })]),
+ );
+ expect(mirrored(0, 140)).toBe(true);
+ });
});
diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.ts
index 5290ea36..852babb3 100644
--- a/packages/framefields/src/clip-vision.ts
+++ b/packages/framefields/src/clip-vision.ts
@@ -1,3 +1,4 @@
+import type { AimConstraint } from "@framefields/compositions/program";
import {
type FrameSignal,
mediaSourceRange,
@@ -25,6 +26,7 @@ import type {
} from "@framefields/vision-sam";
import { mediaDecoderCache } from "@framefields/webgpu-renderers";
import type { VisionProps } from "./effects/index.js";
+import { type RetargetOptions, retargetPose } from "./retarget.js";
export type PoseKeypointName = (typeof COCO17_KEYPOINT_NAMES)[number];
@@ -1386,6 +1388,15 @@ export class PersonTrack {
};
}
+ /**
+ * Aim constraints that make a 3D character copy this person's pose, for
+ * a glTF model's `aim`: `Layer.glb(src, { aim: person.retarget({ rig: "mixamo" }) })`.
+ * Place the person first (`place()`), so directions read in composition px.
+ */
+ retarget(options?: RetargetOptions): AimConstraint[] {
+ return retargetPose(this, options);
+ }
+
/** Signals for a keypoint, for `.signal("x", …)`; read while rendering. */
signals(name: PoseKeypointName): {
x: FrameSignal<number>;
diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.ts
index 6de0b3bd..7ce25c5e 100644
--- a/packages/framefields/src/index.ts
+++ b/packages/framefields/src/index.ts
@@ -993,11 +993,14 @@ export {
import { scatterCoverage } from "./procedural3d.js";
export type {
+ AimConstraint,
CrowdConfig,
+ JointPose,
LookAtConstraint,
ModelAnimationLayer,
NumberTrack,
SpringBoneChain,
+ TrackSignal,
TwoBoneIkConstraint,
Vec3Track,
WindConfig,
@@ -1016,6 +1019,14 @@ export type {
Terrain3DOptions,
TerrainSurface,
} from "./procedural3d.js";
+export {
+ type PoseSource,
+ type RetargetBone,
+ type RetargetOptions,
+ RIG_PRESETS,
+ type RigMap,
+ retargetPose,
+} from "./retarget.js";
export { type StyleName, Styles } from "./styles.js";
export type { Swarm3DOptions } from "./swarm3d.js";
export {
diff --git a/packages/framefields/src/retarget.ts b/packages/framefields/src/retarget.ts
new file mode 100644
index 00000000..0993836b
--- /dev/null
+++ b/packages/framefields/src/retarget.ts
@@ -0,0 +1,191 @@
+/**
+ * Pose retargeting: a performer's 2D pose (vision keypoints, or any source
+ * of named points) drives a 3D character's bones. Each bone is aimed along
+ * its segment in the picture (shoulder → elbow, hip → knee, …), parents
+ * first, so the character copies the performer's silhouette at its own
+ * proportions. Aims run after clips and `pose`, before `lookAt` and `ik`,
+ * so clips can carry what the camera can't see and IK can plant the feet.
+ *
+ * ```ts
+ * const dancer = comp.vision("dance.mp4").person.place({ width: W, height: H });
+ * Layer.glb("robot.glb", { aim: retargetPose(dancer, { rig: "mixamo" }) });
+ * ```
+ *
+ * The picture has no depth: limbs swing in the plane facing the camera.
+ */
+import {
+ type AimConstraint,
+ type NumberTrack,
+ numberAt,
+ type TrackClock,
+} from "@framefields/compositions/program";
+import type { PoseKeypointName } from "./clip-vision.js";
+
+/** Anything that answers where a keypoint is at a frame (a vision `PersonTrack`, a mocap reader). */
+export interface PoseSource {
+ point(
+ name: PoseKeypointName,
+ frame: number,
+ ): { x: number; y: number; visibility: number };
+}
+
+/** The bones a performer's 2D pose can aim. */
+export type RetargetBone =
+ | "spine"
+ | "neck"
+ | "leftUpperArm"
+ | "leftLowerArm"
+ | "rightUpperArm"
+ | "rightLowerArm"
+ | "leftUpperLeg"
+ | "leftLowerLeg"
+ | "rightUpperLeg"
+ | "rightLowerLeg";
+
+/** Joint names per bone; names match without a namespace prefix and case (`"LeftArm"` finds `"mixamorig:LeftArm"`). */
+export type RigMap = Partial<Record<RetargetBone, string>>;
+
+/** Joint names of common rigs. */
+export const RIG_PRESETS = {
+ /** Mixamo (`mixamorig:` prefix, any number). */
+ mixamo: {
+ spine: "Spine",
+ neck: "Neck",
+ leftUpperArm: "LeftArm",
+ leftLowerArm: "LeftForeArm",
+ rightUpperArm: "RightArm",
+ rightLowerArm: "RightForeArm",
+ leftUpperLeg: "LeftUpLeg",
+ leftLowerLeg: "LeftLeg",
+ rightUpperLeg: "RightUpLeg",
+ rightLowerLeg: "RightLeg",
+ },
+ /** Unity humanoid / VRM bone names. */
+ humanoid: {
+ spine: "Spine",
+ neck: "Neck",
+ leftUpperArm: "LeftUpperArm",
+ leftLowerArm: "LeftLowerArm",
+ rightUpperArm: "RightUpperArm",
+ rightLowerArm: "RightLowerArm",
+ leftUpperLeg: "LeftUpperLeg",
+ leftLowerLeg: "LeftLowerLeg",
+ rightUpperLeg: "RightUpperLeg",
+ rightLowerLeg: "RightLowerLeg",
+ },
+ /** Blender Rigify deform bones. */
+ rigify: {
+ spine: "spine",
+ neck: "neck",
+ leftUpperArm: "upper_arm.L",
+ leftLowerArm: "forearm.L",
+ rightUpperArm: "upper_arm.R",
+ rightLowerArm: "forearm.R",
+ leftUpperLeg: "thigh.L",
+ leftLowerLeg: "shin.L",
+ rightUpperLeg: "thigh.R",
+ rightLowerLeg: "shin.R",
+ },
+} as const satisfies Record<string, RigMap>;
+
+export interface RetargetOptions {
+ /** The character's joint names: a preset or a map. Default "mixamo". */
+ rig?: keyof typeof RIG_PRESETS | RigMap;
+ /** Only these bones (e.g. arms, over a walk clip). Default: every mapped bone. */
+ bones?: RetargetBone[];
+ /**
+ * Drive the character's right side with the performer's left: for a
+ * character seen from behind. Default false (both face the camera).
+ */
+ mirror?: boolean;
+ /**
+ * Keypoints less sure than this hand their bone back to the animated
+ * pose, fading over the next 0.2 of confidence. Default 0.3.
+ */
+ minVisibility?: number;
+ /** Overall blend over the animated pose, 0..1: a number, keys, function or signal. Default 1. */
+ weight?: NumberTrack;
+}
+
+type Point = PoseKeypointName | readonly [PoseKeypointName, PoseKeypointName];
+
+/** Each bone's segment in the picture, parents first. */
+const SEGMENTS: ReadonlyArray<[RetargetBone, Point, Point]> = [
+ ["spine", ["leftHip", "rightHip"], ["leftShoulder", "rightShoulder"]],
+ ["neck", ["leftShoulder", "rightShoulder"], "nose"],
+ ["leftUpperArm", "leftShoulder", "leftElbow"],
+ ["leftLowerArm", "leftElbow", "leftWrist"],
+ ["rightUpperArm", "rightShoulder", "rightElbow"],
+ ["rightLowerArm", "rightElbow", "rightWrist"],
+ ["leftUpperLeg", "leftHip", "leftKnee"],
+ ["leftLowerLeg", "leftKnee", "leftAnkle"],
+ ["rightUpperLeg", "rightHip", "rightKnee"],
+ ["rightLowerLeg", "rightKnee", "rightAnkle"],
+];
+
+const mirrored = (name: PoseKeypointName): PoseKeypointName =>
+ (name.startsWith("left")
+ ? `right${name.slice(4)}`
+ : name.startsWith("right")
+ ? `left${name.slice(5)}`
+ : name) as PoseKeypointName;
+
+/** A keypoint, or the midpoint of two, with the lesser visibility. */
+function read(source: PoseSource, p: Point, frame: number, mirror: boolean) {
+ const name = (n: PoseKeypointName) => (mirror ? mirrored(n) : n);
+ if (typeof p === "string") return source.point(name(p), frame);
+ const a = source.point(name(p[0]), frame);
+ const b = source.point(name(p[1]), frame);
+ return {
+ x: (a.x + b.x) / 2,
+ y: (a.y + b.y) / 2,
+ visibility: Math.min(a.visibility, b.visibility),
+ };
+}
+
+const smoothstep = (lo: number, hi: number, v: number) => {
+ const t = Math.min(1, Math.max(0, (v - lo) / (hi - lo)));
+ return t * t * (3 - 2 * t);
+};
+
+/**
+ * Aim constraints that make a character copy `source`'s pose: put them on
+ * a glTF model's `aim`. Bones whose joints the rig map doesn't name, or
+ * whose keypoints the vision isn't sure of, keep the animated pose.
+ */
+export function retargetPose(
+ source: PoseSource,
+ options: RetargetOptions = {},
+): AimConstraint[] {
+ const rig: RigMap =
+ typeof options.rig === "object"
+ ? options.rig
+ : RIG_PRESETS[options.rig ?? "mixamo"];
+ const only = options.bones ? new Set(options.bones) : undefined;
+ const mirror = options.mirror ?? false;
+ const lo = options.minVisibility ?? 0.3;
+ return SEGMENTS.filter(
+ ([bone]) => rig[bone] && (!only || only.has(bone)),
+ ).map(([bone, from, to]) => ({
+ joint: rig[bone] as string,
+ direction: (frame: number): [number, number, number] => {
+ const a = read(source, from, frame, mirror);
+ const b = read(source, to, frame, mirror);
+ return [b.x - a.x, b.y - a.y, 0];
+ },
+ // A signal, so the user's weight reads with the render's own clock.
+ weight: {
+ get: (ctx?: unknown) => {
+ const clock = ctx as TrackClock;
+ const a = read(source, from, clock.frame, mirror);
+ const b = read(source, to, clock.frame, mirror);
+ const sure = smoothstep(
+ lo,
+ lo + 0.2,
+ Math.min(a.visibility, b.visibility),
+ );
+ return sure * numberAt(options.weight, clock, 1);
+ },
+ },
+ }));
+}
diff --git a/packages/webgpu-renderers/src/loaders/rig.test.ts b/packages/webgpu-renderers/src/loaders/rig.test.ts
index 6cf40aae..b6f25fb5 100644
--- a/packages/webgpu-renderers/src/loaders/rig.test.ts
+++ b/packages/webgpu-renderers/src/loaders/rig.test.ts
@@ -3,6 +3,7 @@ import type { Vec3 } from "../math3d/index.js";
import { sampleLocalPose } from "./pose.js";
import {
applyConstraints,
+ applyJointPose,
blendClipLayers,
RigPose,
SpringBones,
@@ -208,4 +209,74 @@ describe("rig", () => {
const end = blendClipLayers(MODEL, [{ clip: SWING, time: 1, weight: 1 }]);
expect(dist(settled.position(5), end.position(5))).toBeLessThan(0.5);
});
+
+ it("poses joints on the animated pose or the rest pose, by weight", () => {
+ const qz = (deg: number): [number, number, number, number] => [
+ 0,
+ 0,
+ Math.sin((deg * Math.PI) / 360),
+ Math.cos((deg * Math.PI) / 360),
+ ];
+ // The swing clip turns the hip 90°; an additive +90° adds to it.
+ const added = blendClipLayers(MODEL, [{ clip: SWING, time: 1, weight: 1 }]);
+ applyJointPose(added, {
+ joint: "hip",
+ rotation: qz(90),
+ offset: [0, 0, 0],
+ weight: 1,
+ mode: "additive",
+ });
+ expect(added.position(2)[1]).toBeCloseTo(-200, 3);
+ // From rest, the clip's turn is ignored.
+ const fromRest = blendClipLayers(MODEL, [
+ { clip: SWING, time: 1, weight: 1 },
+ ]);
+ applyJointPose(fromRest, {
+ joint: "hip",
+ rotation: qz(90),
+ offset: [0, 0, 0],
+ weight: 1,
+ mode: "rest",
+ });
+ expect(fromRest.position(2)[0]).toBeCloseTo(-200, 3);
+ // Half weight from rest: 45°; offsets move the joint.
+ const half = rest();
+ applyJointPose(half, {
+ joint: "hip",
+ rotation: qz(90),
+ offset: [10, 0, 0],
+ weight: 0.5,
+ mode: "rest",
+ });
+ expect(half.position(0)[0]).toBeCloseTo(5, 6);
+ expect(half.position(2)[0]).toBeCloseTo(5 - 200 * Math.SQRT1_2, 3);
+ });
+
+ it("aims bones along directions, parents first, by their own axis", () => {
+ const rig = rest();
+ applyConstraints(rig, {
+ aim: [
+ { joint: "hip", direction: [1, 0, 0], weight: 1 },
+ { joint: "knee", direction: [0, -1, 0], weight: 1 },
+ ],
+ });
+ // Thigh points +x, shin then points -y from the knee.
+ const knee = rig.position(1);
+ const foot = rig.position(2);
+ expect(knee[0]).toBeCloseTo(100, 3);
+ expect(knee[1]).toBeCloseTo(0, 3);
+ expect(foot[0]).toBeCloseTo(100, 3);
+ expect(foot[1]).toBeCloseTo(-100, 3);
+ // Bone lengths hold.
+ expect(dist(knee, foot)).toBeCloseTo(100, 4);
+ });
+
+ it("finds joints by name without their namespace or case", () => {
+ const nodes = NODES.map((n) => ({ ...n, name: `mixamorig:${n.name}` }));
+ const rig = new RigPose(nodes, sampleLocalPose(nodes, undefined, 0));
+ expect(rig.index("knee")).toBe(1);
+ expect(rig.index("mixamorig:Foot")).toBe(2);
+ expect(rig.index("Tail2")).toBe(5);
+ expect(rig.index("nothing")).toBeUndefined();
+ });
});
diff --git a/packages/webgpu-renderers/src/loaders/rig.ts b/packages/webgpu-renderers/src/loaders/rig.ts
index 8ada3df7..900219c7 100644
--- a/packages/webgpu-renderers/src/loaders/rig.ts
+++ b/packages/webgpu-renderers/src/loaders/rig.ts
@@ -1,6 +1,7 @@
/**
* Character rigs on a model's scene graph: layered clip blending with bone
- * masks, analytic two-bone IK with a pole target, clamped look-at, and
+ * masks, per-joint pose overrides, direction aims (retargeting), analytic
+ * two-bone IK with a pole target, clamped look-at, and
* deterministic spring bones (follow-through simulated frame by frame from
* frame 0, so any frame renders the same whether reached in order or not).
*
@@ -69,6 +70,9 @@ function qBetween(a: Vec3, b: Vec3): Quat {
}
const position = (m: Mat4): Vec3 => [m[12], m[13], m[14]];
+/** A joint name without its namespace (`mixamorig:`) and case. */
+const bare = (name: string) =>
+ name.slice(name.lastIndexOf(":") + 1).toLowerCase();
// --- Pose assembly -------------------------------------------------------
@@ -102,7 +106,37 @@ export interface RigLookAt {
axis: Vec3;
}
+/**
+ * A joint posed directly: a rotation (and offset) in the joint's local
+ * frame, on top of the animated pose (`additive`) or of its rest pose
+ * (`rest`, ignoring clips for this joint), blended by `weight`.
+ */
+export interface RigJointPose {
+ joint: string;
+ /** Local rotation. */
+ rotation: Quat;
+ /** Local translation offset. */
+ offset: Vec3;
+ weight: number;
+ mode: "additive" | "rest";
+}
+
+/**
+ * Points a bone along a direction (model space): the joint turns so its
+ * `axis` (default: toward its first child, the bone itself) lies along it.
+ * Pose retargeting is a chain of these, parents first.
+ */
+export interface RigAim {
+ joint: string;
+ direction: Vec3;
+ weight: number;
+ axis?: Vec3;
+}
+
export interface RigConstraints {
+ /** Applied in order: poses, aims, look-ats, then IK. */
+ pose?: RigJointPose[];
+ aim?: RigAim[];
ik?: RigTwoBoneIk[];
lookAt?: RigLookAt[];
}
@@ -120,8 +154,29 @@ export class RigPose {
this.byName = new Map(nodes.map((n, i) => [n.name, i]));
}
+ /**
+ * A joint by name: exact, else ignoring a namespace prefix and case
+ * (`"LeftArm"` finds `"mixamorig:LeftArm"` and `"mixamorig1:leftarm"`).
+ */
index(name: string): number | undefined {
- return this.byName.get(name);
+ const exact = this.byName.get(name);
+ if (exact !== undefined) return exact;
+ const want = bare(name);
+ this.byBare ??= new Map(
+ this.nodes.map((n, i) => [bare(n.name), i] as const).reverse(),
+ );
+ return this.byBare.get(want);
+ }
+ private byBare?: Map<string, number>;
+
+ /** Node `i`'s local direction toward its first child (the bone), if it has one. */
+ boneAxis(i: number): Vec3 | undefined {
+ for (let c = 0; c < this.nodes.length; c++) {
+ if (this.nodes[c]!.parent !== i) continue;
+ const t = this.local.t[c]!;
+ if (len(t) > 1e-9) return norm(t);
+ }
+ return undefined;
}
world(): Mat4[] {
@@ -166,7 +221,7 @@ export class RigPose {
/** Every node at or below the named joints. */
maskSet(names: string[]): Set<number> {
const roots = new Set(
- names.map((n) => this.byName.get(n)).filter((i) => i !== undefined),
+ names.map((n) => this.index(n)).filter((i) => i !== undefined),
);
const out = new Set<number>();
for (let i = 0; i < this.nodes.length; i++) {
@@ -305,10 +360,39 @@ export function solveLookAt(rig: RigPose, c: RigLookAt): void {
/** Look-at first (a head turns), then IK (limbs reach), in the given order. */
export function applyConstraints(rig: RigPose, c: RigConstraints): void {
+ for (const p of c.pose ?? []) applyJointPose(rig, p);
+ for (const a of c.aim ?? []) solveAim(rig, a);
for (const l of c.lookAt ?? []) solveLookAt(rig, l);
for (const k of c.ik ?? []) solveTwoBoneIk(rig, k);
}
+/** Poses one joint (see {@link RigJointPose}). */
+export function applyJointPose(rig: RigPose, p: RigJointPose): void {
+ const i = rig.index(p.joint);
+ const w = clamp(p.weight, 0, 1);
+ if (i === undefined || w <= 0) return;
+ const node = rig.nodes[i]!;
+ const baseR = p.mode === "rest" ? node.rotation : rig.local.r[i]!;
+ const baseT = p.mode === "rest" ? node.translation : rig.local.t[i]!;
+ const r = qMul(baseR, p.rotation);
+ const t = add(baseT, p.offset);
+ rig.local.r[i] = w >= 1 ? r : slerp(rig.local.r[i]!, r, w);
+ rig.local.t[i] = add(rig.local.t[i]!, scale(sub(t, rig.local.t[i]!), w));
+ rig.touch();
+}
+
+/** Points a bone along a direction (see {@link RigAim}). */
+export function solveAim(rig: RigPose, a: RigAim): void {
+ const i = rig.index(a.joint);
+ const w = clamp(a.weight, 0, 1);
+ if (i === undefined || w <= 0 || len(a.direction) < 1e-9) return;
+ const axis = a.axis ?? rig.boneAxis(i);
+ if (!axis) return;
+ const g = rig.globalRotation(i);
+ const swing = qBetween(qRotate(g, axis), a.direction);
+ rig.setGlobalRotation(i, qMul(swing, g), w);
+}
+
// --- Spring bones ------------------------------------------------------------
export interface RigSpringChain {
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index a983e83b..2ddfde16 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -431,6 +431,32 @@ const extras = Layer3D.scatter({
});
```
+**Posing joints directly** (`pose`, by joint name; namespace prefixes and case are ignored, so `"LeftArm"` finds `"mixamorig:LeftArm"`): local Euler degrees `rotateX/Y/Z` (Z, then Y, then X, like layers) and offsets `x/y/z`, on top of the clips (`mode: "additive"`, default) or of the rest pose (`"rest"`: clips stop moving that joint), with a `weight`. Every value is a track: a number, `[[frame, v]]` keys, `(frame) => v`, or a signal (audio, physics, vision, `Signal.*`), so a head can sway on a sine or flinch at a physics hit:
+
+```ts
+Layer.glb("hero.glb", {
+ pose: {
+ Head: { rotateY: Signal.sine({ min: -20, max: 20, period: 48 }) }, // a signal
+ Neck: { rotateX: 30, weight: ball.impact }, // flinch by a physics hit
+ "LeftForeArm": { rotateZ: [[0, 0], [24, -70]], weight: 0.8 }, // keys
+ Spine: { rotateY: (f) => 15 * Math.sin(f / 12), mode: "rest" }, // a function, over rest
+ },
+});
+```
+
+**Aiming and retargeting** (`aim`): `{ joint, direction }` turns a bone (toward its first child, or a local `axis`) along a scene direction, parents first; direction axes are tracks too (`[1, signal, 0]`). A performer's 2D pose drives a character with `person.retarget({ rig: "mixamo" | "humanoid" | "rigify" | { leftUpperArm: "…", … }, bones?, mirror?, minVisibility?, weight? })` on a placed vision person (or `retargetPose(source, options)` for any source with `point(name, frame)`):
+
+```ts
+const dancer = comp.vision("dance.mp4").person.place({ width: W, height: H, fit: "cover" });
+Layer.glb("robot.glb", {
+ x: dancer.signals("leftHip").x, y: dancer.signals("leftHip").y, // root follows (signals)
+ aim: dancer.retarget({ rig: "mixamo", bones: ["leftUpperArm", "leftLowerArm", "rightUpperArm", "rightLowerArm"] }),
+ animationLayers: [{ clip: "Walk" }], // legs keep walking
+});
+```
+
+Order of evaluation: clips, `pose`, `aim`, `lookAt`, `ik` (so IK can still plant the feet), then springs. Retargeting is 2D: limbs swing in the plane facing the camera (no depth from one view); unsure keypoints (below `minVisibility`, default 0.3) hand their bone back to the animated pose. Use `mirror: true` for a character seen from behind. Functions and signals are in-process only; JSON keeps numbers and keys.
+
Rules: a crowd mesh keeps its own `rotateX` / `scale` under each instance (y-up glTF: `rotateX: 180`). Spring bones follow the clip's motion and the model's current orientation, not the layer's keyframed travel. Per-instance `clip`, `animOffset`, `animSpeed` in `transforms` / `instanceAt` override the crowd's seeded choices.
### Procedural worlds: terrain, sculptures, scatter
diff --git a/plugins/framefields/skills/framefields-vision/SKILL.md b/plugins/framefields/skills/framefields-vision/SKILL.md
index 85ec0c2d..221c4559 100644
--- a/plugins/framefields/skills/framefields-vision/SKILL.md
+++ b/plugins/framefields/skills/framefields-vision/SKILL.md
@@ -291,6 +291,10 @@ const vision = comp.withVision({ classes: ["person"] });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });
```
+### A 3D character copies a performer
+
+`Layer.glb("robot.glb", { aim: person.retarget({ rig: "mixamo" }) })` on a placed person (`comp.vision(src).person.place({ width, height, fit })`) aims the character's spine, neck, arms and legs along the performer's keypoint segments every frame (2D: in the plane facing the camera). Presets `mixamo`, `humanoid` (Unity/VRM), `rigify`, or a `{ leftUpperArm: "…" }` map; `bones` limits it (arms over a walk clip), `mirror` swaps sides, `minVisibility` lets unsure limbs fall back to the clip. Bind the root with `person.signals("leftHip")`. See `framefields-compose` → Characters.
+
### Analyse before authoring
```typescript
const report = await comp.analyzeVisionSequence("assets/street.mp4", {
diff --git a/specs/ue58-engine-roadmap.md b/specs/ue58-engine-roadmap.md
index f3f30500..9d8c280a 100644
--- a/specs/ue58-engine-roadmap.md
+++ b/specs/ue58-engine-roadmap.md
@@ -346,7 +346,7 @@ To render 500–10,000 animated characters (robots, soldiers, pedestrians) in 1
3. In `mesh3d.wgsl` (`vs_main`), when both `isInstanced` and `isSkinned` are active, each vertex fetches its 4 influencing bone matrices from the baked joint palette storage/texture buffer at the instance's `(clipIndex, frame * speed + phaseOffset)`.
- Result: **5,000 independently walking/running skinned 3D characters with shadows, outlines, and bloom in a single draw call**.
-**Status (implemented):** rig evaluator `webgpu-renderers/src/loaders/rig.ts` (pure, unit-tested), crowd baking `loaders/crowd.ts`, compositor `renderers/character.ts`, schemas `compositions/src/program/character-schema.ts`. The field is `animationLayers` (`animation` already holds keyframe tracks). Deviations: (1) Layers blend in order, each overriding the pose beneath by its weight (slerp per joint) from the rest pose, so a crossfade is two layers with the top one's weight keyed; weights and targets are tracks: a constant, linear `[[frame, …]]` keys (JSON) or a function (in process). (2) Look-at runs before IK (a head turns, then limbs reach); look-at takes the joint's local `axis` (default glTF +z). IK targets and poles are scene coordinates, mapped through the inverse of the model's transform; the pole twists the solved chain about root→target. (3) Spring bones simulate in model space (follow-through from the clips, gravity mapped by the model's current orientation), replayed from frame 0 and cached per layer; stiffness is a per-second pull toward the animated pose, damping a per-second velocity loss. (4) Crowds bake a palette per mesh into a storage buffer (the mesh's skinning binding), not a texture; each clip bakes `count + 1` samples so playback interpolates to the loop point without wrapping; rigid node-attached meshes bake their node as joint 0. Each crowd instance carries a second 64-byte record (clip range, phase, playback rate) instead of packing bits into `emissive.w`. (5) Instance and scatter variants now apply their own node rotation and scale under each instance's, so a y-up glTF (`rotateX: 180`) stands upright in crowds and scatters.
+**Status (implemented):** rig evaluator `webgpu-renderers/src/loaders/rig.ts` (pure, unit-tested), crowd baking `loaders/crowd.ts`, compositor `renderers/character.ts`, schemas `compositions/src/program/character-schema.ts`. The field is `animationLayers` (`animation` already holds keyframe tracks). Deviations: (1) Layers blend in order, each overriding the pose beneath by its weight (slerp per joint) from the rest pose, so a crossfade is two layers with the top one's weight keyed; weights and targets are tracks: a constant, linear `[[frame, …]]` keys (JSON) or a function (in process). (2) Look-at runs before IK (a head turns, then limbs reach); look-at takes the joint's local `axis` (default glTF +z). IK targets and poles are scene coordinates, mapped through the inverse of the model's transform; the pole twists the solved chain about root→target. (3) Spring bones simulate in model space (follow-through from the clips, gravity mapped by the model's current orientation), replayed from frame 0 and cached per layer; stiffness is a per-second pull toward the animated pose, damping a per-second velocity loss. (4) Crowds bake a palette per mesh into a storage buffer (the mesh's skinning binding), not a texture; each clip bakes `count + 1` samples so playback interpolates to the loop point without wrapping; rigid node-attached meshes bake their node as joint 0. Each crowd instance carries a second 64-byte record (clip range, phase, playback rate) instead of packing bits into `emissive.w`. (5) Instance and scatter variants now apply their own node rotation and scale under each instance's, so a y-up glTF (`rotateX: 180`) stands upright in crowds and scatters. (6) Direct posing and retargeting (added after Phase 6): `pose` sets joints by name (local Euler degrees and offsets, additive over the clips or from rest, weighted) and `aim` points bones along scene directions; both run after clip layers and before look-at and IK. Every track (pose, aim, IK, look-at, clip weights) also takes a signal, and a point one track per axis; the readers live in `compositions/program/track-sampler.ts`, shared by the SDK and the compositor. `retargetPose(source, { rig })` / `person.retarget()` turns a vision person's COCO keypoints into aims (spine, neck, arms, legs; presets `mixamo`, `humanoid`, `rigify`), in the picture plane (one view has no depth), with confidence fading bones back to the animated pose. Joint names match without a namespace prefix and case.
---