Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(motion): convert generated clips from HumanML3D's rest shape so heads stay up and arms hang

nirholas9d886c7d4ce4
diff --git a/api/_lib/motion-glb.js b/api/_lib/motion-glb.js
index 764280436..28049d903 100644
--- a/api/_lib/motion-glb.js
+++ b/api/_lib/motion-glb.js
@@ -39,13 +39,14 @@
// clip's per-bone position and scale channels describe the SOURCE rig's bone
// lengths, and writing them onto another skeleton overwrites its proportions.
//
-// ROOT DRIFT is removed before the bake by default. The published generated
-// clips predate flattenRootDrift (api/_lib/motion-seed.js) and still carry the
-// lane's constant forward ramp, so a GLB baked straight from one walks a metre
-// forward while it applauds. The sellable artifact is the thing a buyer opens in
-// Blender, so it gets the corrected motion whether or not the clip JSON behind
-// it has been republished yet. Pass `flatten: false` to bake a clip exactly as
-// given, which is what the round-trip tests do.
+// ROOT DRIFT is removed before the bake for a clip that has not been through
+// the library pipeline yet (one not stamped with CLIP_BASIS): such a clip still
+// carries the lane's constant forward ramp, so a GLB baked straight from it
+// walks a metre forward while it applauds. A clip already in the library basis
+// is left alone, because that pipeline has removed the drift AND given a
+// travelling clip back the root motion its feet imply (lockRootToContacts);
+// flattening it again would turn a walk back into a treadmill. Pass `flatten`
+// explicitly to override either way, which is what the round-trip tests do.
//
// Spec: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html
@@ -262,7 +263,7 @@ export function restBasisFromGltf(gltf) {
* }} input
* @returns {{ glb: Buffer, channels: number, droppedTracks: string[], hipScale: number, duration: number, driftRemoved: number, coverage: number }}
*/
-export function bakeMotionGlb({ rig, clip: input, name, flatten = true, retarget = true }) {
+export function bakeMotionGlb({ rig, clip: input, name, flatten = needsRebase(input), retarget = true }) {
if (!input || !Array.isArray(input.tracks) || input.tracks.length === 0) {
throw new Error('clip has no tracks');
}
diff --git a/api/_lib/motion-seed.js b/api/_lib/motion-seed.js
index ca2405be9..637799a58 100644
--- a/api/_lib/motion-seed.js
+++ b/api/_lib/motion-seed.js
@@ -868,10 +868,175 @@ function slerp(a, b, t) {
// pushup, a squat, a swim stroke, a crouch, a sneak and a meditation, which are
// low-posture by definition and correct as they are.
+// ── The source skeleton's rest SHAPE (added 2026-09-18) ─────────────────────
+//
+// Treating the generator's rest as "identity, in the same shape as cz" fixed the
+// legs and nothing else, and it was measured, not guessed: rendered on the
+// default rig on 2026-09-18, every generated clip, including all 39 published by
+// the repair pass, played with the head thrown back about 90 degrees and the
+// arms raised where they should hang. The legs were right because the one thing
+// the two skeletons agree on is that legs point down.
+//
+// The Kabsch fit in mdm_sampler.py measures each joint's rotation against
+// HumanML3D's `paramUtil.t2m_raw_offsets`, and those are directions, not a
+// posed human: the upper arm, forearm and hand hang STRAIGHT DOWN, the collars
+// point straight out, and the head sits straight FORWARD of the neck. The cz rig
+// rests in a T-pose with the head above the neck. So a clip whose arms hang at
+// the sides carries identity on the arms, which the identity-shape conversion
+// played as a T-pose, and a clip whose head is upright carries a 90 degree
+// rotation on the neck that turns "forward" into "up", which it played on top
+// of a neck that already points up.
+//
+// The general conversion aligns the two rest shapes per joint. For joint j, A_j
+// is the rotation that carries the cz rest directions of j's children onto the
+// source's rest directions of the same children. Then a source world rotation
+// G_s maps onto the target as G_t = G_s * A_j * Wt_j, which puts every child
+// exactly where the source put it, and in local terms:
+//
+// q_t = (Wt_p^-1 * A_p^-1) * q_s * (A_j * Wt_j)
+//
+// With A = identity this is exactly the old conversion, which is why it was
+// right for the legs. A leaf joint (hand, head, toe) has no children in the
+// 22-joint source, so it carries identity there and inherits its parent's A,
+// which leaves it at its cz rest relative to its parent: the only honest pose
+// for a joint the source never measured.
+
+// t2m_raw_offsets from HumanML3D's paramUtil, keyed by the cz bone each SMPL
+// joint maps to (workers/model-text2motion/smpl_to_clip.py SMPL_TO_WOLF3D):
+// the direction from the parent joint to this joint in the source's rest.
+// Verified 2026-09-18 by running source forward kinematics over a live idle
+// take with these offsets: head above neck, arms hanging, toes forward.
+export const HUMANML3D_REST_DIRECTION = Object.freeze({
+ Spine: [0, 1, 0],
+ Spine1: [0, 1, 0],
+ Spine2: [0, 1, 0],
+ Neck: [0, 1, 0],
+ Head: [0, 0, 1],
+ LeftShoulder: [1, 0, 0],
+ RightShoulder: [-1, 0, 0],
+ LeftArm: [0, -1, 0],
+ RightArm: [0, -1, 0],
+ LeftForeArm: [0, -1, 0],
+ RightForeArm: [0, -1, 0],
+ LeftHand: [0, -1, 0],
+ RightHand: [0, -1, 0],
+ LeftUpLeg: [1, 0, 0],
+ RightUpLeg: [-1, 0, 0],
+ LeftLeg: [0, -1, 0],
+ RightLeg: [0, -1, 0],
+ LeftFoot: [0, -1, 0],
+ RightFoot: [0, -1, 0],
+ LeftToeBase: [0, 0, 1],
+ RightToeBase: [0, 0, 1],
+});
+
+// For a joint with several children, the child whose direction is matched
+// exactly; the others only settle the twist about it.
+const PRIMARY_CHILD = Object.freeze({ Hips: 'Spine', Spine2: 'Neck' });
+
+function normalize(v) {
+ const len = Math.hypot(v[0], v[1], v[2]) || 1;
+ return [v[0] / len, v[1] / len, v[2] / len];
+}
+
+function cross(a, b) {
+ return [a[1] * b[2] - a[2] * b[1], a[2] * b[0] - a[0] * b[2], a[0] * b[1] - a[1] * b[0]];
+}
+
+function dot(a, b) {
+ return a[0] * b[0] + a[1] * b[1] + a[2] * b[2];
+}
+
+/** Shortest-arc rotation carrying unit vector a onto unit vector b. */
+function arcBetween(a, b) {
+ const d = dot(a, b);
+ if (d < -0.999999) {
+ // Opposite vectors: any perpendicular axis is a valid half turn.
+ const axis = normalize(Math.abs(a[0]) < 0.9 ? cross(a, [1, 0, 0]) : cross(a, [0, 1, 0]));
+ return [axis[0], axis[1], axis[2], 0];
+ }
+ const c = cross(a, b);
+ const q = [c[0], c[1], c[2], 1 + d];
+ const len = Math.hypot(...q);
+ return q.map((v) => v / len);
+}
+
+/** Rotation of `angle` radians about unit axis. */
+function axisAngle(axis, angle) {
+ const s = Math.sin(angle / 2);
+ return [axis[0] * s, axis[1] * s, axis[2] * s, Math.cos(angle / 2)];
+}
+
+// A_j for every joint the source animates, computed once from the two rest
+// skeletons. Parents are resolved before children (FK_ORDER), so a leaf can
+// inherit its parent's alignment.
+const REST_ALIGNMENT = (() => {
+ const mapped = new Set(['Hips', ...Object.keys(HUMANML3D_REST_DIRECTION)]);
+ const out = {};
+ for (const bone of FK_ORDER) {
+ if (!mapped.has(bone)) continue;
+ const here = CANONICAL_REST_POSITION[bone];
+ const kids = Object.keys(CANONICAL_PARENT).filter((b) => CANONICAL_PARENT[b] === bone && mapped.has(b));
+ if (kids.length === 0) {
+ out[bone] = out[CANONICAL_PARENT[bone]] ?? [0, 0, 0, 1];
+ continue;
+ }
+ const cz = (b) => normalize(CANONICAL_REST_POSITION[b].map((v, k) => v - here[k]));
+ const src = (b) => normalize(HUMANML3D_REST_DIRECTION[b]);
+ const primary = PRIMARY_CHILD[bone] ?? kids[0];
+ let align = arcBetween(cz(primary), src(primary));
+ const others = kids.filter((b) => b !== primary);
+ if (others.length) {
+ // Twist about the primary direction so the other children land as close
+ // to their source directions as a rotation about that axis allows.
+ const axis = src(primary);
+ let sinSum = 0;
+ let cosSum = 0;
+ for (const b of others) {
+ const moved = quatApply(align, cz(b));
+ const target = src(b);
+ const u = normalize(moved.map((v, k) => v - axis[k] * dot(moved, axis)));
+ const w = normalize(target.map((v, k) => v - axis[k] * dot(target, axis)));
+ sinSum += dot(cross(u, w), axis);
+ cosSum += dot(u, w);
+ }
+ align = quatMul(axisAngle(axis, Math.atan2(sinSum, cosSum)), align);
+ }
+ out[bone] = align;
+ }
+ return Object.freeze(out);
+})();
+
+// Stamped on every clip written in the library's basis, so a repair pass can
+// tell a converted clip from one that predates the conversion and never
+// rebases the same clip twice. v1 is the identity-shape conversion (legs right,
+// arms and head wrong); a v1 clip is converted back to the source basis and
+// forward again, which is exact because both conversions are fixed rotations.
+export const CLIP_BASIS = 'canonical-rest-v2';
+export const LEGACY_CLIP_BASIS = 'canonical-rest-v1';
+
+function corrections(bone) {
+ const parent = CANONICAL_PARENT[bone];
+ const restWorld = CANONICAL_REST_WORLD[bone];
+ const align = REST_ALIGNMENT[bone];
+ if (!restWorld || !align) return null;
+ const parentWorld = parent ? CANONICAL_REST_WORLD[parent] ?? [0, 0, 0, 1] : [0, 0, 0, 1];
+ const parentAlign = parent ? REST_ALIGNMENT[parent] ?? [0, 0, 0, 1] : [0, 0, 0, 1];
+ return {
+ // q_t = L * q_s * R
+ L: quatMul(quatConjugate(parentWorld), quatConjugate(parentAlign)),
+ R: quatMul(align, restWorld),
+ // The v1 conversion, q1 = Wt_p^-1 * q_s * Wt_j, undone: q_s = Wt_p * q1 * Wt_j^-1.
+ undoL: parentWorld,
+ undoR: quatConjugate(restWorld),
+ };
+}
+
/**
- * Rewrite a clip's rotations from the generator's identity-rest basis into the
+ * Rewrite a clip's rotations from the generator's HumanML3D basis into the
* library's canonical (cz) rest basis, so it composes correctly with every other
- * library clip. Returns a NEW clip; the input is not modified.
+ * library clip. A clip stamped with the legacy v1 basis is converted back first.
+ * Returns a NEW clip; the input is not modified.
*
* @param {any} clip
* @returns {{ clip: any, rebasedTracks: number }}
@@ -879,23 +1044,19 @@ function slerp(a, b, t) {
export function rebaseToCanonicalRest(clip) {
const source = clip?.tracks;
if (!Array.isArray(source)) return { clip, rebasedTracks: 0 };
+ const legacy = clip?.userData?.basis === LEGACY_CLIP_BASIS;
let rebasedTracks = 0;
const tracks = source.map((track) => {
const name = String(track?.name || '');
if (!name.endsWith('.quaternion')) return track;
- const bone = name.slice(0, name.lastIndexOf('.'));
- const restLocal = CANONICAL_REST[bone];
- const restWorld = CANONICAL_REST_WORLD[bone];
- if (!restLocal || !restWorld) return track;
-
- // L = Rt * Wt^-1, R = Wt.
- const L = quatMul(restLocal, quatConjugate(restWorld));
- const R = restWorld;
+ const c = corrections(name.slice(0, name.lastIndexOf('.')));
+ if (!c) return track;
const values = Array.from(track.values ?? []);
for (let i = 0; i + 3 < values.length; i += 4) {
- const q = [values[i], values[i + 1], values[i + 2], values[i + 3]];
- const out = quatMul(quatMul(L, q), R);
+ let q = [values[i], values[i + 1], values[i + 2], values[i + 3]];
+ if (legacy) q = quatMul(quatMul(c.undoL, q), c.undoR);
+ const out = quatMul(quatMul(c.L, q), c.R);
values[i] = out[0];
values[i + 1] = out[1];
values[i + 2] = out[2];
@@ -905,15 +1066,12 @@ export function rebaseToCanonicalRest(clip) {
return { ...track, values };
});
- return { clip: { ...clip, tracks }, rebasedTracks };
+ const userData = clip.userData && typeof clip.userData === 'object' ? { ...clip.userData } : undefined;
+ if (userData && legacy) delete userData.basis;
+ return { clip: { ...clip, tracks, ...(userData ? { userData } : {}) }, rebasedTracks };
}
-// Stamped on every clip written in the library's basis, so a repair pass can
-// tell a converted clip from one of the 133 that predate the conversion and
-// never rebases the same clip twice.
-export const CLIP_BASIS = 'canonical-rest-v1';
-
-/** True when a clip still carries the generator's identity-rest basis. */
+/** True when a clip is not yet in the current library basis. */
export function needsRebase(clip) {
return clip?.userData?.basis !== CLIP_BASIS;
}
@@ -1497,3 +1655,24 @@ function rebuildWithBlend(clip, keep, blendFrames, fps) {
return { ...clip, duration: (keep - 1) / fps, tracks };
}
+
+// ── Library-ready conversion for a single clip ──────────────────────────────
+
+/**
+ * Convert one clip exactly as the worker wrote it into a clip that plays
+ * correctly on the library's rig: the rest-shape rebase, the foot-flick repair,
+ * the drift removal and the root travel from foot contact, in the order the
+ * batch runner applies them (scripts/gcp/seed-motion.mjs deriveClip). This is
+ * what POST /api/forge-motion's poll hands a browser, so a clip generated on
+ * demand plays the same way a published one does. No gate: an on-demand clip
+ * is the user's to judge.
+ *
+ * @param {any} workerClip
+ * @returns {any} a NEW clip stamped with CLIP_BASIS
+ */
+export function libraryReadyClip(workerClip) {
+ const rebased = needsRebase(workerClip) ? rebaseToCanonicalRest(workerClip).clip : workerClip;
+ const flattened = flattenRootDrift(despikeFootFlicks(rebased).clip).clip;
+ const clip = lockRootToContacts(flattened).clip;
+ return { ...clip, userData: { ...(clip.userData || {}), basis: CLIP_BASIS } };
+}
diff --git a/api/forge-motion.js b/api/forge-motion.js
index f2788da99..bf9cb9231 100644
--- a/api/forge-motion.js
+++ b/api/forge-motion.js
@@ -24,6 +24,8 @@
import { cors, json, method, readJson, wrap, rateLimited } from './_lib/http.js';
import { limits, clientIp } from './_lib/rate-limit.js';
import { createRegenProvider } from './_providers/gcp.js';
+import { fetchUpstream } from './_lib/upstream-fetch.js';
+import { libraryReadyClip } from './_lib/motion-seed.js';
// Provider job ids are base64url JSON envelopes (packJobId in _providers/gcp.js)
// and run several hundred chars — a 64-char cap 400s every poll.
@@ -107,16 +109,35 @@ async function pollJob(req, res, jobId) {
}
const result = await provider.status(jobId);
+ const clip = result.status === 'done' && result.resultClipUrl ? await convertedClip(result.resultClipUrl) : null;
return json(res, 200, {
job_id: jobId,
status: result.status,
clip_url: result.resultClipUrl || null,
+ // The clip in the library's basis, ready to play. The worker writes its
+ // rotations against HumanML3D's raw rest directions (arms hanging, head
+ // forward of the neck), and played as-is that throws the head back and
+ // raises the arms on every rig. `clip_url` stays for callers that convert
+ // themselves; `clip` is null only when the conversion could not run.
+ clip,
frames: typeof result.frames === 'number' ? result.frames : null,
fps: typeof result.fps === 'number' ? result.fps : null,
error: result.error || null,
});
}
+// A failed conversion degrades to the raw clip URL rather than failing a
+// generation the GPU already paid for.
+async function convertedClip(url) {
+ try {
+ const res = await fetchUpstream(url, {}, { name: 'lane:motion-clip', timeoutMs: 15_000, attempts: 2 });
+ return libraryReadyClip(await res.json());
+ } catch (err) {
+ console.warn('[forge-motion] clip conversion failed', err?.message || err);
+ return null;
+ }
+}
+
export default wrap(async (req, res) => {
if (cors(req, res, { methods: 'GET,POST,OPTIONS' })) return;
if (!method(req, res, ['GET', 'POST'])) return;
diff --git a/scripts/gcp/seed-motion.mjs b/scripts/gcp/seed-motion.mjs
index 1325cc12e..8f5982114 100644
--- a/scripts/gcp/seed-motion.mjs
+++ b/scripts/gcp/seed-motion.mjs
@@ -25,6 +25,7 @@
* node scripts/gcp/seed-motion.mjs --report # checkpoint stats only
* node scripts/gcp/seed-motion.mjs --repair --publish # fix the live library
* node scripts/gcp/seed-motion.mjs --list # list the keepers for sale
+ * node scripts/gcp/seed-motion.mjs --regate --publish # re-derive every clip, no GPU
*
* REPAIRING THE PUBLISHED SET. --repair reads every generated clip already live
* in the library, converts it out of the generator's identity rest basis into the
@@ -59,7 +60,7 @@
*/
import { createHash } from 'node:crypto';
-import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { existsSync, mkdirSync, readFileSync, unlinkSync, writeFileSync } from 'node:fs';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
@@ -101,6 +102,7 @@ const PUBLISH = !!args.publish;
const REPORT_ONLY = !!args.report;
const REPAIR = !!args.repair;
const LIST = !!args.list;
+const REGATE = !!args.regate;
// Independent samples to draw per prompt. The sampler is stochastic, so the
// same prompt yields a different take every time; several takes per prompt is
// how a 137-prompt library grows into several hundred clips while the gate
@@ -298,11 +300,30 @@ async function runPrompt(prompt) {
const clipUrl = DIRECT ? await awaitClipDirect(jobId) : await awaitClip(jobId);
const worker = await fetchClip(clipUrl);
const elapsedSeconds = Math.round((Date.now() - started) / 1000);
+ const derived = deriveClip(worker, prompt);
+ // The clip is named from its body once, at generation, and keeps that name
+ // through every later re-derivation, so the published object never moves.
+ const name = clipName(prompt, JSON.stringify({ ...derived.raw, name: 'pending' }));
+ return stageRecord(derived, prompt, {
+ name,
+ taskId: assertSelfHostedLane(jobId).taskId,
+ lane,
+ clipSourceUrl: clipUrl,
+ elapsedSeconds,
+ });
+}
- // The lane writes rotations in the HumanML3D skeleton's identity rest basis,
- // not the library's canonical one (see rebaseToCanonicalRest). Converted
- // first, because every later step, and every viewer, reads the clip in the
- // library's basis. Played unconverted, the legs fold up over the body.
+/**
+ * Turn a clip as the worker wrote it into the clip the library serves, and gate
+ * it. Pure: no GPU, no network, so a published set can be re-derived from its
+ * sources whenever this pipeline improves (--regate).
+ */
+function deriveClip(worker, prompt) {
+ // The lane writes rotations against the HumanML3D skeleton's raw rest
+ // directions, not the library's canonical rest (see rebaseToCanonicalRest).
+ // Converted first, because every later step, and every viewer, reads the clip
+ // in the library's basis. A clip stamped with the legacy v1 basis is carried
+ // back to the source basis and forward again inside the same call.
const rebased = needsRebase(worker) ? rebaseToCanonicalRest(worker).clip : worker;
// The lane's foot rotation flips to its other Kabsch solution for a frame or
@@ -310,16 +331,15 @@ async function runPrompt(prompt) {
// before anything measures contact, because every flick reads as a planted
// toe skating across the floor.
const flicks = despikeFootFlicks(rebased);
- const fetched = flicks.clip;
// The lane's root channel is a constant forward ramp carrying no prompt
- // signal (see ROOT_DRIFT in api/_lib/motion-seed.js), so it is removed first,
- // before anything else reads the clip. Order matters twice over: the gate's
+ // signal (see ROOT_DRIFT in api/_lib/motion-seed.js), so it is removed before
+ // anything else reads the clip. Order matters twice over: the gate's
// foot-slide rule divides planted-foot slide by the stride the clip covers,
// and a fake one-metre stride makes that rule vacuous, and the seam search
// below is hunting for the frame whose pose repeats frame 0, which a ramp
// guarantees no frame ever does.
- const flattened = flattenRootDrift(fetched);
+ const flattened = flattenRootDrift(flicks.clip);
// Then give the body back the travel its own feet imply. A flattened walk is a
// treadmill whose planted foot is dragged under a stationary body; the authored
@@ -336,20 +356,20 @@ async function runPrompt(prompt) {
const seam = prompt.loop === true ? closeLoopSeam(locked.clip) : null;
const raw = seam ? seam.clip : locked.clip;
- const verdict = gateWithBasis(raw, prompt);
+ return { raw, verdict: gateWithBasis(raw, prompt), flicks, flattened, locked, seam };
+}
- // The clip is renamed to its library identity before it is written, so the
- // staged file, the manifest entry and the published object always agree.
+/** Write a derived clip to staging (keepers) or the reject pile, and return its checkpoint record. */
+function stageRecord(derived, prompt, { name, taskId, lane, clipSourceUrl = null, elapsedSeconds = 0 }) {
+ const { raw, verdict, flicks, flattened, locked, seam } = derived;
// toLibraryClip stamps the basis, so no later pass ever rebases it twice.
- const body = JSON.stringify({ ...raw, name: 'pending' });
- const name = clipName(prompt, body);
const clip = toLibraryClip(raw, {
name,
promptId: prompt.id,
- prompt: prompt.prompt,
+ prompt: prompt.prompt ?? '',
category: prompt.category,
loop: prompt.loop === true,
- taskId: assertSelfHostedLane(jobId).taskId,
+ taskId,
});
const serialized = JSON.stringify(clip);
@@ -362,9 +382,10 @@ async function runPrompt(prompt) {
loop: prompt.loop === true,
name,
lane,
- clip_source_url: clipUrl,
+ clip_source_url: clipSourceUrl,
elapsed_seconds: elapsedSeconds,
gate_version: verdict.gateVersion,
+ basis: clip.userData.basis,
loop_seam: seam
? { before: seam.seamBefore, after: seam.seamAfter, trimmed_frames: seam.trimmedFrames, kept_original: seam.rejected || null }
: null,
@@ -386,14 +407,15 @@ async function runPrompt(prompt) {
decided_at: new Date().toISOString(),
};
+ const keeper = join(CLIPS_DIR, `${name}.json`);
+ const reject = join(REJECTS_DIR, `${name}.json`);
if (verdict.pass) {
- writeFileSync(join(CLIPS_DIR, `${name}.json`), serialized);
+ writeFileSync(keeper, serialized);
+ if (existsSync(reject)) unlinkSync(reject);
record.bytes = Buffer.byteLength(serialized);
} else {
- writeFileSync(
- join(REJECTS_DIR, `${name}.json`),
- JSON.stringify({ record, explanations: explainMotionGate(verdict.reasons), clip }, null, 1),
- );
+ writeFileSync(reject, JSON.stringify({ record, explanations: explainMotionGate(verdict.reasons), clip }, null, 1));
+ if (existsSync(keeper)) unlinkSync(keeper);
}
return record;
}
@@ -602,10 +624,9 @@ async function listForSale(state) {
let listed = 0;
for (const entry of entries) {
const clip = JSON.parse(readFileSync(join(CLIPS_DIR, `${entry.name}.json`), 'utf8'));
- // A clip already in the library basis has had its drift removed and, for
- // a travelling clip, its real root motion restored; flattening it again
- // would turn a walk back into a treadmill. Only a legacy clip is flattened.
- const { glb } = bakeMotionGlb({ rig, clip, name: entry.label || entry.name, flatten: needsRebase(clip) });
+ // Staged clips are in the library basis, so the bake leaves their root
+ // alone (see bakeMotionGlb): a walk keeps the travel its feet imply.
+ const { glb } = bakeMotionGlb({ rig, clip, name: entry.label || entry.name });
const artifactKey = `${market.GENERATED_GLB_PREFIX}/${entry.name}.glb`;
await client.send(
new PutObjectCommand({
@@ -678,73 +699,105 @@ async function repair(state) {
log('');
const prompts = new Map(loadPrompts().map((p) => [p.id, p]));
- let kept = 0;
- let dropped = 0;
- const tally = {};
-
+ const outcome = { kept: 0, dropped: 0, tally: {} };
for (const entry of live) {
const source = await (await fetch(entry.url)).json();
// promptIdOf: gen-<prompt id>-<12 hex>. The hash is fixed width, so the id
// is everything between the prefix and the last dash.
const stem = entry.name.slice(CLIP_NAME_PREFIX.length);
const promptId = stem.slice(0, stem.lastIndexOf('-'));
- const prompt = prompts.get(promptId);
- const loop = prompt ? prompt.loop === true : entry.loop === true;
-
- const rebased = needsRebase(source) ? rebaseToCanonicalRest(source).clip : source;
- const flattened = flattenRootDrift(rebased);
- const seam = loop ? closeLoopSeam(flattened.clip) : null;
- const repaired = seam ? seam.clip : flattened.clip;
-
- const verdict = gateWithBasis(repaired, { loop, duration_seconds: prompt?.duration_seconds });
-
- const clip = toLibraryClip(repaired, {
- name: entry.name,
- promptId,
- prompt: prompt?.prompt ?? '',
- category: entry.category ?? prompt?.category ?? 'emote',
- loop,
- taskId: entry.name,
- });
- const serialized = JSON.stringify(clip);
-
- const record = {
- prompt_id: promptId,
- label: entry.label ?? prompt?.label ?? promptId,
- category: clip.userData.category,
- icon: entry.icon || prompt?.icon || '🎬',
- loop,
+ const prompt = promptFor(prompts, promptId, entry);
+ const record = stageRecord(deriveClip(source, prompt), prompt, {
name: entry.name,
+ taskId: source.userData?.task_id ?? entry.name,
lane: 'repair',
- elapsed_seconds: 0,
- gate_version: verdict.gateVersion,
- root_drift: {
- speed_m_s: Number(flattened.speed.toFixed(4)),
- removed_m: Number(flattened.removed.toFixed(4)),
- residual_m: Number(flattened.residual.toFixed(4)),
- },
- status: verdict.pass ? 'accepted' : 'rejected',
- reasons: verdict.reasons,
- detail: verdict.detail || '',
- metrics: verdict.metrics,
- bytes: Buffer.byteLength(serialized),
- decided_at: new Date().toISOString(),
- };
+ });
state.prompts[promptId] = record;
+ tallyOutcome(outcome, record);
+ saveCheckpoint(state);
+ }
+ return summarize(outcome);
+}
- if (verdict.pass) {
- writeFileSync(join(CLIPS_DIR, `${entry.name}.json`), serialized);
- kept++;
- log(` keep ${entry.name}`);
- } else {
- writeFileSync(join(REJECTS_DIR, `${entry.name}.json`), JSON.stringify(record, null, 2));
- dropped++;
- for (const reason of verdict.reasons) tally[reason.split(':')[0]] = (tally[reason.split(':')[0]] || 0) + 1;
- log(` drop ${entry.name} ${verdict.reasons.join(',')}`);
+/**
+ * Re-derive every clip in the checkpoint from its source and re-gate it, with
+ * no GPU time. A fresh take is re-read from the worker output it was built from
+ * (clip_source_url); a clip that entered through --repair has no worker output
+ * on record, so its staged copy is carried back to the source basis instead,
+ * which is exact (see rebaseToCanonicalRest). Names never change, so a
+ * re-derived keeper replaces its published object in place on the next
+ * --publish, and one that no longer passes stops being served.
+ */
+async function regate(state) {
+ const prompts = new Map(loadPrompts().map((p) => [p.id, p]));
+ const keys = Object.keys(state.prompts).filter((k) => ['accepted', 'rejected'].includes(state.prompts[k].status));
+ log(`Re-deriving ${keys.length} clips from their sources`);
+ const outcome = { kept: 0, dropped: 0, tally: {} };
+ let index = 0;
+ const worker = async () => {
+ while (index < keys.length) {
+ const key = keys[index++];
+ const previous = state.prompts[key];
+ const source = previous.clip_source_url ? await fetchClip(previous.clip_source_url) : stagedSource(previous);
+ if (!source) {
+ log(` skip ${key}: no source on record and no staged copy`);
+ continue;
+ }
+ const prompt = { ...promptFor(prompts, previous.prompt_id, previous), sample: previous.sample ?? 0 };
+ const record = stageRecord(deriveClip(source, prompt), prompt, {
+ name: previous.name,
+ taskId: source.userData?.task_id ?? previous.name,
+ lane: previous.lane,
+ clipSourceUrl: previous.clip_source_url ?? null,
+ elapsedSeconds: previous.elapsed_seconds ?? 0,
+ });
+ if (record.status !== previous.status) {
+ log(` ${record.status === 'accepted' ? 'now keep' : 'now drop'} ${key} ${record.reasons.join(',')}`);
+ }
+ state.prompts[key] = record;
+ tallyOutcome(outcome, record);
}
- saveCheckpoint(state);
+ };
+ await Promise.all(Array.from({ length: Math.min(CONCURRENCY * 2, keys.length) }, worker));
+ saveCheckpoint(state);
+ return summarize(outcome);
+}
+
+/** The staged copy of a clip with no worker output on record (a --repair clip). */
+function stagedSource(record) {
+ const keeper = join(CLIPS_DIR, `${record.name}.json`);
+ if (existsSync(keeper)) return JSON.parse(readFileSync(keeper, 'utf8'));
+ const reject = join(REJECTS_DIR, `${record.name}.json`);
+ if (!existsSync(reject)) return null;
+ const parsed = JSON.parse(readFileSync(reject, 'utf8'));
+ return parsed.clip ?? null;
+}
+
+/** The library prompt for an id, or a stand-in built from what the record knows. */
+function promptFor(prompts, promptId, fallback) {
+ const prompt = prompts.get(promptId);
+ if (prompt) return prompt;
+ return {
+ id: promptId,
+ label: fallback.label ?? promptId,
+ category: fallback.category ?? 'emote',
+ icon: fallback.icon,
+ loop: fallback.loop === true,
+ prompt: '',
+ duration_seconds: undefined,
+ };
+}
+
+function tallyOutcome(outcome, record) {
+ if (record.status === 'accepted') {
+ outcome.kept++;
+ return;
}
+ outcome.dropped++;
+ for (const reason of record.reasons) outcome.tally[reason] = (outcome.tally[reason] || 0) + 1;
+}
+function summarize({ kept, dropped, tally }) {
log('');
log(` kept ${kept}`);
log(` dropped ${dropped}`);
@@ -768,6 +821,15 @@ async function main() {
return;
}
+ if (REGATE) {
+ const outcome = await regate(state);
+ report(state);
+ if (PUBLISH) await publish(state);
+ if (LIST) await listForSale(state);
+ if (outcome.kept === 0) process.exitCode = 4;
+ return;
+ }
+
if (REPAIR) {
const outcome = await repair(state);
if (PUBLISH) await publish(state);
diff --git a/src/animation-library.js b/src/animation-library.js
index 221efd9fa..214eabecc 100644
--- a/src/animation-library.js
+++ b/src/animation-library.js
@@ -685,6 +685,9 @@ export class AnimationLibrary {
if (token !== this._genToken) return null;
const res = await fetch(`/api/forge-motion?job=${encodeURIComponent(sub.job_id)}`);
const data = await res.json().catch(() => ({}));
+ // The server hands back the clip already converted to the library's
+ // basis; the raw worker URL is the fallback when that step failed.
+ if (data.status === 'done' && data.clip) return data.clip;
if (data.status === 'done' && data.clip_url) {
const cr = await fetch(data.clip_url);
if (!cr.ok) throw new Error(`Couldn’t fetch the generated clip (HTTP ${cr.status}).`);
diff --git a/tests/api/forge-motion.test.js b/tests/api/forge-motion.test.js
index b3873e1a3..74eeab4f8 100644
--- a/tests/api/forge-motion.test.js
+++ b/tests/api/forge-motion.test.js
@@ -29,6 +29,40 @@ vi.mock('../../api/_providers/gcp.js', () => ({
}),
}));
+// The worker's clip, as the lane writes it: every bone at the source rest
+// (identity) except the neck, which turns HumanML3D's "head forward" into "up".
+const SOURCE_BONES = [
+ 'Hips', 'Spine', 'Spine1', 'Spine2', 'Neck', 'Head',
+ 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand',
+ 'RightShoulder', 'RightArm', 'RightForeArm', 'RightHand',
+ 'LeftUpLeg', 'LeftLeg', 'LeftFoot', 'LeftToeBase',
+ 'RightUpLeg', 'RightLeg', 'RightFoot', 'RightToeBase',
+];
+const workerClip = () => {
+ const times = [0, 1 / 30, 2 / 30];
+ const neck = [-Math.SQRT1_2, 0, 0, Math.SQRT1_2];
+ return {
+ name: 'waving',
+ duration: 2 / 30,
+ tracks: [
+ ...SOURCE_BONES.map((bone) => ({
+ type: 'quaternion',
+ name: `${bone}.quaternion`,
+ times,
+ values: times.flatMap(() => (bone === 'Neck' ? neck : [0, 0, 0, 1])),
+ })),
+ { type: 'vector', name: 'Hips.position', times, values: times.flatMap(() => [0, 0.98, 0]) },
+ ],
+ };
+};
+let clipFetchFails = false;
+vi.mock('../../api/_lib/upstream-fetch.js', () => ({
+ fetchUpstream: vi.fn(async () => {
+ if (clipFetchFails) throw new Error('storage unreachable');
+ return { ok: true, json: async () => workerClip() };
+ }),
+}));
+
vi.mock('../../api/_lib/rate-limit.js', async (importActual) => {
const actual = await importActual();
return {
@@ -119,6 +153,32 @@ describe('GET /api/forge-motion?job=', () => {
expect(body.fps).toBe(30);
});
+ it('hands back the clip already converted to the library basis', async () => {
+ const res = makeRes();
+ await handler(makeReq({ method: 'GET', url: `/api/forge-motion?job=${'a'.repeat(24)}` }), res);
+ const body = JSON.parse(res.body);
+ expect(body.clip.userData.basis).toBe('canonical-rest-v2');
+ const { forwardKinematicsFrame } = await import('../../api/_lib/motion-seed.js');
+ const w = forwardKinematicsFrame(body.clip, 0);
+ // Head over the neck and arms hanging, not thrown back and raised.
+ expect(w.Head[1] - w.Neck[1]).toBeGreaterThan(0.1);
+ expect(w.LeftArm[1] - w.LeftHand[1]).toBeGreaterThan(0.4);
+ });
+
+ it('falls back to the raw clip URL when the conversion cannot run', async () => {
+ clipFetchFails = true;
+ try {
+ const res = makeRes();
+ await handler(makeReq({ method: 'GET', url: `/api/forge-motion?job=${'a'.repeat(24)}` }), res);
+ const body = JSON.parse(res.body);
+ expect(res.statusCode).toBe(200);
+ expect(body.clip).toBeNull();
+ expect(body.clip_url).toMatch(/motion-clips\/mdm/);
+ } finally {
+ clipFetchFails = false;
+ }
+ });
+
it('400s without a job id', async () => {
const res = makeRes();
await handler(makeReq({ method: 'GET', url: '/api/forge-motion' }), res);
diff --git a/tests/motion-seed.test.js b/tests/motion-seed.test.js
index fa36d535a..533beafd2 100644
--- a/tests/motion-seed.test.js
+++ b/tests/motion-seed.test.js
@@ -42,8 +42,9 @@ import {
lockRootToContacts,
forwardKinematicsFrame,
ROOT_LOCK,
+ LEGACY_CLIP_BASIS,
} from '../api/_lib/motion-seed.js';
-import { CANONICAL_REST } from '../src/animation-canonical-rest.js';
+import { CANONICAL_REST, CANONICAL_REST_WORLD, CANONICAL_PARENT } from '../src/animation-canonical-rest.js';
const BODY_BONES = [
'Hips', 'Spine', 'Spine1', 'Spine2', 'Neck', 'Head',
@@ -716,3 +717,87 @@ describe('root travel from foot contact', () => {
expect(locked.contactShare).toBeLessThan(ROOT_LOCK.MIN_CONTACT_SHARE);
});
});
+
+describe('source rest shape', () => {
+ // The lane's rest is HumanML3D's raw offsets: arms hanging, collars out, the
+ // head straight FORWARD of the neck. A standing pose in that basis is identity
+ // everywhere except the neck, which turns "forward" into "up".
+ const SOURCE_BONES = [
+ 'Hips', 'Spine', 'Spine1', 'Spine2', 'Neck', 'Head',
+ 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand',
+ 'RightShoulder', 'RightArm', 'RightForeArm', 'RightHand',
+ 'LeftUpLeg', 'LeftLeg', 'LeftFoot', 'LeftToeBase',
+ 'RightUpLeg', 'RightLeg', 'RightFoot', 'RightToeBase',
+ ];
+ const NECK_UP = quatX(-Math.PI / 2); // carries +z (forward) onto +y (up)
+
+ function standingSourceClip(frames = 4) {
+ const times = Array.from({ length: frames }, (_, i) => i / 30);
+ const tracks = SOURCE_BONES.map((bone) => ({
+ type: 'quaternion',
+ name: `${bone}.quaternion`,
+ times: [...times],
+ values: times.flatMap(() => (bone === 'Neck' ? NECK_UP : [0, 0, 0, 1])),
+ }));
+ tracks.push({ type: 'vector', name: 'Hips.position', times: [...times], values: times.flatMap(() => [0, 0.98, 0]) });
+ return { name: 'standing', duration: (frames - 1) / 30, tracks };
+ }
+
+ it('stands a source-basis pose up with the head over the neck and the arms hanging', () => {
+ const clip = rebaseToCanonicalRest(standingSourceClip()).clip;
+ const w = forwardKinematicsFrame(clip, 0);
+ // Head directly above the neck, not thrown back or forward.
+ expect(w.Head[1] - w.Neck[1]).toBeGreaterThan(0.1);
+ expect(Math.hypot(w.Head[0] - w.Neck[0], w.Head[2] - w.Neck[2])).toBeLessThan(0.05);
+ // Arms hang: each hand well below its shoulder, not out in a T-pose.
+ for (const side of ['Left', 'Right']) {
+ expect(w[`${side}Arm`][1] - w[`${side}Hand`][1]).toBeGreaterThan(0.4);
+ expect(Math.abs(w[`${side}Hand`][0] - w[`${side}Arm`][0])).toBeLessThan(0.15);
+ }
+ // Legs down, toes forward of the ankles.
+ expect(w.LeftUpLeg[1] - w.LeftFoot[1]).toBeGreaterThan(0.7);
+ expect(w.LeftToeBase[2]).toBeGreaterThan(w.LeftFoot[2]);
+ });
+
+ it('converts a legacy v1 clip exactly as if it had come straight from the source', () => {
+ const source = buildClip({ swing: 0.3 });
+ // The v1 conversion: q1 = Wt_parent^-1 * q_s * Wt_bone.
+ const conj = (q) => [-q[0], -q[1], -q[2], q[3]];
+ const v1 = {
+ ...source,
+ userData: { basis: LEGACY_CLIP_BASIS },
+ tracks: source.tracks.map((track) => {
+ if (!track.name.endsWith('.quaternion')) return track;
+ const bone = track.name.slice(0, track.name.lastIndexOf('.'));
+ const parent = CANONICAL_PARENT[bone];
+ const Wp = parent ? CANONICAL_REST_WORLD[parent] : [0, 0, 0, 1];
+ const values = Array.from(track.values);
+ for (let i = 0; i < values.length; i += 4) {
+ const q = quatMul(quatMul(conj(Wp), values.slice(i, i + 4)), CANONICAL_REST_WORLD[bone]);
+ values.splice(i, 4, ...q);
+ }
+ return { ...track, values };
+ }),
+ };
+ expect(needsRebase(v1)).toBe(true);
+ const direct = rebaseToCanonicalRest(source).clip;
+ const viaLegacy = rebaseToCanonicalRest(v1).clip;
+ expect(viaLegacy.userData.basis).toBeUndefined();
+ for (let t = 0; t < direct.tracks.length; t += 1) {
+ if (!direct.tracks[t].name.endsWith('.quaternion')) continue;
+ const a = direct.tracks[t].values;
+ const b = viaLegacy.tracks[t].values;
+ for (let i = 0; i < a.length; i += 4) {
+ // q and -q are the same rotation.
+ const d = Math.abs(a[i] * b[i] + a[i + 1] * b[i + 1] + a[i + 2] * b[i + 2] + a[i + 3] * b[i + 3]);
+ expect(d, direct.tracks[t].name).toBeCloseTo(1, 6);
+ }
+ }
+ });
+
+ it('marks the current basis as v2', () => {
+ expect(CLIP_BASIS).toBe('canonical-rest-v2');
+ expect(needsRebase({ userData: { basis: CLIP_BASIS } })).toBe(false);
+ expect(needsRebase({ userData: { basis: LEGACY_CLIP_BASIS } })).toBe(true);
+ });
+});