Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
feat(model-text2motion): add MoMask as a selectable second backend beside MDM
nirholas27586e5a40e1
diff --git a/STRUCTURE.md b/STRUCTURE.mdindex b770b2454..ba7e7cd57 100644--- a/STRUCTURE.md+++ b/STRUCTURE.md@@ -195,7 +195,7 @@ bottom.| Agent-native 3D + embeddable distribution | MCP tool [api/_mcp/tools/agents.js](api/_mcp/tools/agents.js) (`attach_avatar_to_agent`); shared oEmbed provider [api/agent-oembed.js](api/agent-oembed.js) `→ /api/oembed` + builder [api/_lib/embed.js](api/_lib/embed.js) (`forgeEmbedTarget`, shared by `get_embed_code` in [api/_mcp/tools/embed.js](api/_mcp/tools/embed.js)); Forge share page oEmbed discovery [api/forge-share.js](api/forge-share.js) `→ /forge/share/:id`; distribution snippet builders [src/forge-embed-snippets.js](src/forge-embed-snippets.js) + panel [src/forge-embed-panel.js](src/forge-embed-panel.js); autonomous demo [examples/agent-native-3d/](examples/agent-native-3d) | In-repo | Closes the loop: create → own → discover/remix → **use everywhere**. `attach_avatar_to_agent` gives a registered agent identity a persistent body (chains into `register_agent` for on-chain identity + `anchor_provenance` for a signed authenticity credential). Any Forge creation now unfurls via real oEmbed (paste `/forge/share/:id` into Notion/Discord/Slack) with the same `type:rich` iframe payload `get_embed_code` returns via MCP. The Forge "Embed this model" panel hands out five real distribution flavours from one GLB — iframe, `<model-viewer>`, `<agent-3d>`, a `@three-ws/page-agent` talking guide, and a `@three-ws/walk` companion — all built from one shared, pure snippet module so every surface stays byte-identical. `examples/agent-native-3d/run.mjs` drives the full agent-native chain live against `/api/mcp-studio` (mesh_forge → rig_mesh → create_agent_persona → persona_say → get_agent_persona → every distribution snippet) with zero mocks and a documented, logged fallback if the shared free-tier generation lane is rate-limited at run time. Spec: [specs/EMBED_SPEC.md](specs/EMBED_SPEC.md#share-routes-on-chain-agents). Docs: [docs/mcp.md#attach_avatar_to_agent](docs/mcp.md#attach_avatar_to_agent). Roadmap prompt 10 (closes the roadmap campaign). || Animation pipeline | [public/animations/](public/animations) + [scripts/build-animations.mjs](scripts/build-animations.mjs) | In-repo | Mixamo source FBX + GLB retargeting pipeline · see [docs/3d-asset-pipeline.md](docs/3d-asset-pipeline.md) || Procedural animation (runtime IK layers) | [src/procedural/](src/procedural) — analytic two-bone IK [two-bone-ik.js](src/procedural/two-bone-ik.js), gaze [look-at.js](src/procedural/look-at.js), terrain foot planting [foot-plant.js](src/procedural/foot-plant.js), drift-free re-application [pose-baseline.js](src/procedural/pose-baseline.js) | Live | Post-mixer layers that do what no baked clip can: the chest/neck/head chain turns toward a world point, and each foot lands on its own patch of uneven ground. Rig-agnostic via `canonicalNodeMapFromObject`; an unsupported rig reports `enabled === false` and no-ops. Wired into the corner companion ([walk-sdk/src/companion.js](walk-sdk/src/companion.js) `_updateGaze`), the walkaround world ([src/walk.js](src/walk.js) `rebuildFootPlant`), and agent avatars ([src/agent-avatar.js](src/agent-avatar.js) `_applyLookTarget` — which is what finally makes `setLookTarget()` and the `LOOK_AT` action move the rig). Ambient idle motion (breathing/blink/saccade/weight-shift) stays in [src/idle-animation.js](src/idle-animation.js). Docs: [docs/procedural-animation.md](docs/procedural-animation.md). Tests: [tests/procedural-anim.test.js](tests/procedural-anim.test.js), [tests/agent-avatar-look-target.test.js](tests/agent-avatar-look-target.test.js) |-| Self-hosted 3D GPU fleet (the engines behind the Forge) | [workers/model-trellis2/](workers/model-trellis2) (TRELLIS.2, MIT, the default image-to-3D lane `trellis2`: PBR GLB with alpha, `resolution` 512/1024/1536; built and tested, deploy owner-gated), [workers/model-pixal3d/](workers/model-pixal3d) (TencentARC Pixal3D, MIT, on TRELLIS.2's backbone: the `pixal3d` fidelity lane labeled "Match my photo", 1 to 4 pixel-aligned input photos with optional `view_labels` and one-photo `fill_other_sides` view synthesis via MV-Adapter (Apache-2.0); built and tested, deploy owner-gated), territory router [api/_lib/forge-territory.js](api/_lib/forge-territory.js) (EU, UK and KR never reach a Tencent-model lane; unknown country fails closed), [workers/model-trellis/](workers/model-trellis), [workers/model-hunyuan3d/](workers/model-hunyuan3d) (2.0 + 2.1 PBR + 2.1 RTX), [workers/model-triposg/](workers/model-triposg), [workers/modly/](workers/modly) (Modly v0.4.3 running its TripoSG and Hunyuan3D 2 Mini Turbo extensions as the untextured Forge lanes `modly` and `modly_hunyuan`, behind `MODEL_MODLY_URL`; weights staged, built and tested, deploy owner-gated), [workers/model-triposr/](workers/model-triposr), [workers/model-text2motion/](workers/model-text2motion), [workers/rig/](workers/rig) (auto-rigging, deployed as `model-rig`), [workers/rig-nonhuman/](workers/rig-nonhuman) (UniRig + SkinTokens, MIT, the non-humanoid rigging fallback `model-rig` reaches for when its own `mesh_classify.py` heuristic rules out a Mixamo-shaped humanoid, behind `RIG_NONHUMAN_URL`; built and tested, deploy owner-gated), [workers/model-anigen/](workers/model-anigen) (VAST-AI-Research AniGen, MIT, the rig-on-generation Forge lane `anigen` labeled "Rig on generate": predicts mesh, joint skeleton and skin weights together from a single reference image, so the GLB it returns needs no separate rerig call; opt-in only, never a default or failover target; built and tested, deploy owner-gated), [workers/model-partcrafter/](workers/model-partcrafter) (PartCrafter, MIT, LICENSE read 2026-10-11, the `partcrafter` Forge lane labeled "Editable parts": decodes a reference photo straight into N individually named part meshes in one diffusion-transformer pass; built and tested, deploy owner-gated), [workers/model-holopart/](workers/model-holopart) (HoloPart, MIT, LICENSE read 2026-10-11, the "Complete hidden parts" lane behind `/api/forge-holopart`: takes an already part-segmented mesh (from `workers/segment` or `model-partcrafter`) and fills in the occluded geometry of every part, closing each into a standalone shape; built and tested, deploy owner-gated), [workers/avatar-reconstruction/](workers/avatar-reconstruction), CPU stages [workers/rembg/](workers/rembg) · [workers/remesh/](workers/remesh) · [workers/segment/](workers/segment) · [workers/stylize/](workers/stylize), talking-avatar video [workers/longcat/](workers/longcat) (LongCat-Video-Avatar-1.5, gates the `/create/video` lane via `LONGCAT_WORKER_URL`; the one fleet member that is not a Cloud Run service, because its ~45 GB resident weight set needs an 80 GB GPU, so it is built and tested but undeployed), streaming video→3D [workers/model-video2scene/](workers/model-video2scene), selfie-pipeline CPU orchestrator [workers/avatar-pipeline-controller/](workers/avatar-pipeline-controller) | Live · Cloud Run GPU (us-central1, plus us-east4 where trellis/hunyuan/triposr hold standbys and `model-text2motion` runs as its production lane) | Every generation engine the Forge dispatches to, each with its own `cloudbuild.yaml`. The L4 quota is **per region** and both us-central1 and us-east4 are granted 3, so a warm pin in one region only starves lanes in that same region (increase to 16 filed 2026-07-16, still reconciling); the Hunyuan3D-2.1 PBR realism lane also ships an RTX PRO 6000 build (`model-hunyuan3d-21-rtx`, quota granted 1000) as the warm primary. `model-text2motion` (MDM, MIT) is the text→motion lane behind `GCP_TEXT2MOTION_URL`, the `text_to_animation` MCP tool, and `POST /api/forge-motion`: prompt in, a retargetable three.js `AnimationClip` out, applied with the same engine (`src/animation-retarget.js`) as the curated animation library. It is the one GPU service whose production URL points at us-east4, and its `cloudbuild.yaml` deploys both regions from one image so the standby never lags. Ops map: [docs/ops/gcp-credits-plan.md](docs/ops/gcp-credits-plan.md) |+| Self-hosted 3D GPU fleet (the engines behind the Forge) | [workers/model-trellis2/](workers/model-trellis2) (TRELLIS.2, MIT, the default image-to-3D lane `trellis2`: PBR GLB with alpha, `resolution` 512/1024/1536; built and tested, deploy owner-gated), [workers/model-pixal3d/](workers/model-pixal3d) (TencentARC Pixal3D, MIT, on TRELLIS.2's backbone: the `pixal3d` fidelity lane labeled "Match my photo", 1 to 4 pixel-aligned input photos with optional `view_labels` and one-photo `fill_other_sides` view synthesis via MV-Adapter (Apache-2.0); built and tested, deploy owner-gated), territory router [api/_lib/forge-territory.js](api/_lib/forge-territory.js) (EU, UK and KR never reach a Tencent-model lane; unknown country fails closed), [workers/model-trellis/](workers/model-trellis), [workers/model-hunyuan3d/](workers/model-hunyuan3d) (2.0 + 2.1 PBR + 2.1 RTX), [workers/model-triposg/](workers/model-triposg), [workers/modly/](workers/modly) (Modly v0.4.3 running its TripoSG and Hunyuan3D 2 Mini Turbo extensions as the untextured Forge lanes `modly` and `modly_hunyuan`, behind `MODEL_MODLY_URL`; weights staged, built and tested, deploy owner-gated), [workers/model-triposr/](workers/model-triposr), [workers/model-text2motion/](workers/model-text2motion), [workers/rig/](workers/rig) (auto-rigging, deployed as `model-rig`), [workers/rig-nonhuman/](workers/rig-nonhuman) (UniRig + SkinTokens, MIT, the non-humanoid rigging fallback `model-rig` reaches for when its own `mesh_classify.py` heuristic rules out a Mixamo-shaped humanoid, behind `RIG_NONHUMAN_URL`; built and tested, deploy owner-gated), [workers/model-anigen/](workers/model-anigen) (VAST-AI-Research AniGen, MIT, the rig-on-generation Forge lane `anigen` labeled "Rig on generate": predicts mesh, joint skeleton and skin weights together from a single reference image, so the GLB it returns needs no separate rerig call; opt-in only, never a default or failover target; built and tested, deploy owner-gated), [workers/model-partcrafter/](workers/model-partcrafter) (PartCrafter, MIT, LICENSE read 2026-10-11, the `partcrafter` Forge lane labeled "Editable parts": decodes a reference photo straight into N individually named part meshes in one diffusion-transformer pass; built and tested, deploy owner-gated), [workers/model-holopart/](workers/model-holopart) (HoloPart, MIT, LICENSE read 2026-10-11, the "Complete hidden parts" lane behind `/api/forge-holopart`: takes an already part-segmented mesh (from `workers/segment` or `model-partcrafter`) and fills in the occluded geometry of every part, closing each into a standalone shape; built and tested, deploy owner-gated), [workers/avatar-reconstruction/](workers/avatar-reconstruction), CPU stages [workers/rembg/](workers/rembg) · [workers/remesh/](workers/remesh) · [workers/segment/](workers/segment) · [workers/stylize/](workers/stylize), talking-avatar video [workers/longcat/](workers/longcat) (LongCat-Video-Avatar-1.5, gates the `/create/video` lane via `LONGCAT_WORKER_URL`; the one fleet member that is not a Cloud Run service, because its ~45 GB resident weight set needs an 80 GB GPU, so it is built and tested but undeployed), streaming video→3D [workers/model-video2scene/](workers/model-video2scene), selfie-pipeline CPU orchestrator [workers/avatar-pipeline-controller/](workers/avatar-pipeline-controller) | Live · Cloud Run GPU (us-central1, plus us-east4 where trellis/hunyuan/triposr hold standbys and `model-text2motion` runs as its production lane) | Every generation engine the Forge dispatches to, each with its own `cloudbuild.yaml`. The L4 quota is **per region** and both us-central1 and us-east4 are granted 3, so a warm pin in one region only starves lanes in that same region (increase to 16 filed 2026-07-16, still reconciling); the Hunyuan3D-2.1 PBR realism lane also ships an RTX PRO 6000 build (`model-hunyuan3d-21-rtx`, quota granted 1000) as the warm primary. `model-text2motion` (MDM default + MoMask, both MIT, selectable per request via `model: "mdm" | "momask"`) is the text→motion lane behind `GCP_TEXT2MOTION_URL`, the `text_to_animation` MCP tool, and `POST /api/forge-motion`: prompt in, a retargetable three.js `AnimationClip` out, applied with the same engine (`src/animation-retarget.js`) as the curated animation library. It is the one GPU service whose production URL points at us-east4, and its `cloudbuild.yaml` deploys both regions from one image so the standby never lags. Ops map: [docs/ops/gcp-credits-plan.md](docs/ops/gcp-credits-plan.md) || Self-hosted Granite Guardian (governance lane behind watsonx) | [workers/granite-guardian/](workers/granite-guardian/README.md) (Cloud Run `granite-guardian`, europe-west4, vLLM on one L4), client lane in [api/_lib/granite-guardian.js](api/_lib/granite-guardian.js) | Live · Cloud Run GPU (europe-west4) | IBM's open-weight `ibm-granite/granite-guardian-3.3-8b` served by the stock vLLM OpenAI image, so `/api/guardian/assess`, the identity integrity check, and the Granite Twin/Oracle/Attest/IRL screens return real Granite Guardian verdicts on a deployment with no IBM Cloud account. watsonx.ai stays the lead lane when configured; every verdict names the `model` and `provider` that produced it. Placed in europe-west4 because that region's L4 grant is otherwise unused, so its warm pin never starves the 3D lanes. Tests: [tests/api-guardian.test.js](tests/api-guardian.test.js), [tests/api/identity-integrity.test.js](tests/api/identity-integrity.test.js) || Reconstruct completion backstop (abandoned selfie jobs) | [api/cron/reconstruct-sweep.js](api/cron/reconstruct-sweep.js), every 5 min via `vercel.json` crons → Cloud Scheduler | Live 2026-07-25 | A reconstruct job used to be advanced **only by the user's browser**: `/create/selfie` polls `regenerate-status`, and each poll runs the finalize stages. Close the tab and the worker still finished the GLB and stored it durably, but nothing ever collected it — 26 jobs were stranded that way, oldest from 2026-05-31, invisible to `db-retention` (which prunes only terminal rows). This sweep is the server-side driver: it picks up reconstruct jobs quiet >3 min and runs **the same shared stages** in [reconstruct-finalize.js](api/_lib/reconstruct-finalize.js), so an abandoned job still lands in the user's library. 30-day rescue window (the worker's GCS blobs + Firestore job state are durable, so a weeks-old job is still deliverable); past that, or when a job's provider is no longer pollable from platform credentials, rows fail out with an honest reason so the open set can't grow unbounded. BYOK (meshy/tripo) jobs stay browser-driven — a cron has no user context to resolve their keys. Sibling of [auto-rig-sweep.js](api/cron/auto-rig-sweep.js), which plays the same role for the rerig lane. Docs: [docs/avatar-reconstruction.md](docs/avatar-reconstruction.md#who-finishes-the-job) || Avatar fidelity evaluation (selfie → avatar, scored) | [workers/avatar-reconstruction/eval/](workers/avatar-reconstruction/eval) | In-repo · gates every reconstruction change | **Identity Shape Error (ISE)** — does the reconstructed avatar have *this person's* face shape? Texture-blind, pose- and scale-invariant, dimensionless: MediaPipe's 468 landmarks and the avatar's corresponding head vertices are both Umeyama-aligned onto the neutral canonical face over a stable identity subset (no eyes/lips, so a smile is not scored as identity), then compared. A pipeline cannot score better by resizing or tilting the head — only by matching shape, a property pinned in [eval/test_identity_eval.py](workers/avatar-reconstruction/eval/test_identity_eval.py) alongside monotonicity. [`run_eval.py`](workers/avatar-reconstruction/eval/run_eval.py) scores a whole reference set against a live worker and diffs two revisions, surfacing per-face regressions a mean would hide; [`tune_morph.py`](workers/avatar-reconstruction/eval/tune_morph.py) sweeps the morph parameters against ISE offline; [`robustness.py`](workers/avatar-reconstruction/eval/robustness.py) measures the opposing axis (does a blurred / turned / occluded / dim / low-res photo of the same person still yield the same head?), and the shipped `max_displacement_frac` is the knee of those two curves. The 40-face reference set is **synthesised** by [`make_refs.py`](workers/avatar-reconstruction/eval/make_refs.py) over a deterministic grid of the platform's diversity matrix — no real biometrics, no licence terms, rebuildable by any agent — gitignored and mirrored to `gs://three-ws-avatar-reconstructions/eval-refs/`. Docs: [eval/README.md](workers/avatar-reconstruction/eval/README.md), program plan [docs/avatar-fidelity-program.md](docs/avatar-fidelity-program.md) |diff --git a/prompts/finish/_context/production-100-OWNER-ACTIONS.md b/prompts/finish/_context/production-100-OWNER-ACTIONS.mdindex cd7efa2c2..c27795012 100644--- a/prompts/finish/_context/production-100-OWNER-ACTIONS.md+++ b/prompts/finish/_context/production-100-OWNER-ACTIONS.md@@ -58,6 +58,8 @@ on it; some resolve themselves as other work lands.| 42 | **Apply the MCP client-sessions nullable-column migration** (added 2026-10-11 by the evolve triage-lane sweep). `api/_lib/migrations/20261015010000_mcp_client_sessions_client_nullable.sql` drops the `NOT NULL` constraint on `mcp_client_sessions.client` and `mcp_client_daily.client`. This is an ACTIVE production bug, not routine deploy lag: `git merge-base --is-ancestor <mcp-clients.js commit> 66a5e93bd` confirms `api/_lib/mcp-clients.js` (the code that inserts session/day rows and can leave `client` unset for some call shapes) is already deployed and running today, so every write that violates the constraint is silently failing right now (lost MCP session/day analytics rows, not a user-facing error since the insert is best-effort). `npm run db:status` lists it as the one pending migration. Command: `node scripts/apply-migrations.mjs --apply --file 20261015010000_mcp_client_sessions_client_nullable.sql` (or `npm run db:migrate`, which applies every pending migration immediately with no dry run). | one migration apply | Stops silent MCP analytics data loss | [docs/ops/production-log-triage.md](../../docs/ops/production-log-triage.md) || 45 | **Top up the HF Inference Providers credit balance for fal-ai and nscale (image generation), a separate pool from row 6's LLM-lane `HF_TOKEN` usage** (added 2026-10-12 by the evolve triage-lane sweep). Root-caused the `/api/3d/generate` 502 storm: both HF-routed image providers answer `402 "You have no remaining credits"` on every call (`api/_mcp3d/text-to-image.js`), alongside the already-tracked Vertex billing hold (row 20) on the image lane. The amplification bug (every request re-paying the full timeout of both known-dead lanes before ever reaching keyless Pollinations, often blowing the shared ladder budget) is fixed in code this sweep: a 401/402/403 now parks the lane for 5 minutes (`AUTH_COOLDOWN_SECONDS`), same breaker already used for Vertex elsewhere. That stops the 502 storm either way, but the ladder is still down two rungs (Vertex, HF) until the owner funds one or both; Pollinations alone is a lower-quality free-tier fallback with no paid Replicate backstop configured. Fix: top up the HF Inference Providers billing for the account behind `HF_TOKEN`, or fund the Vertex billing hold (row 20); either clears most image generations, both together clear all of it. | billing | `/api/3d/generate` image quality and the 502 rate | `scripts/gcp-triage.mjs` `forge-image-ladder-exhausted-502`, [docs/ops/production-log-triage.md](../../docs/ops/production-log-triage.md) §forge-image-ladder-exhausted |+| 46 | **Build and deploy `model-text2motion` with its new MoMask backend, then benchmark it against MDM** (added 2026-10-12 by the evolve loop, best3d order 077 task 1). `workers/model-text2motion/` now carries a second generation backend, MoMask (EricGuo5513/momask-codes, MIT, published FID 0.045 on HumanML3D vs. MDM-era T2M-GPT's 0.141), selectable per request via `model: "mdm" | "momask"` on `/infer`, alongside the existing default MDM backend. `main.py`'s model selector, the new `momask_sampler.py` and shared `motion_decode.py`, the updated `Dockerfile` (clones momask-codes, installs its one missing pip dependency `einops`), and the updated `cloudbuild.yaml` (adds `MOMASK_MODEL_DIR=/weights/momask` to both the primary and standby deploy steps) are all written and locally tested (49/49 passing in `workers/model-text2motion`'s own suite, including a new 12-case `test_model_selector.py`). The real checkpoints are already staged to `gs://three-ws-model-weights/momask/t2m/` (four checkpoint dirs: `length_estimator`, `rvq_nq6_dc512_nc512_noshare_qdp0.2`, `t2m_nlayer8_nhead6_ld384_ff1024_cdp0.1_rvq6ns`, `tres_nlayer8_ld384_ff1024_rvq6ns_cdp0.2_sw`), matching momask-codes' own HumanML3D checkpoint layout, so a deploy has real weights to mount the moment it lands. This is a production deploy (rebuilds and redeploys an existing Cloud Run GPU service, gate 2), so no agent runs it. Command, from the repo root: `gcloud builds submit --config workers/model-text2motion/cloudbuild.yaml --substitutions=SHORT_SHA=manual$(date +%s) .`. After it lands, run the same prompt through both backends and time the second call to each (post cold-load) for a real warm-vs-warm comparison: `curl -s -X POST "$MOTION_URL/infer" -H "authorization: Bearer $GCP_RECONSTRUCTION_KEY" -H 'content-type: application/json' -d '{"prompt":"a slow tai-chi sweep","duration_seconds":3,"fps":20,"model":"mdm"}'` and the same body with `"model":"momask"`, each followed by polling `GET $MOTION_URL/tasks/<task_id>` until `status:"done"`. That reproduced, in-house FID/latency comparison is the order's last unverified Definition of done line; the published 0.045 FID in the README is momask-codes' own number, not yet confirmed against this worker's own output. | one build + one deploy + two benchmark calls | Closes best3d order 077 task 1; gives the Animation Studio and `/api/forge-motion` a second, stronger text-to-motion backend selectable per request | [077-best3d-11-prompt-to-clip.md](../077-best3d-11-prompt-to-clip.md), [workers/model-text2motion/README.md](../../workers/model-text2motion/README.md) |+Deleted 2026-09-01: the former row 5 (`WALLET_ENCRYPTION_KEY_PREVIOUS` policy), whichdeclared itself "delete on reading" and had outlived that by three weeks.diff --git a/workers/model-text2motion/Dockerfile b/workers/model-text2motion/Dockerfileindex ebb992692..7c577ae76 100644--- a/workers/model-text2motion/Dockerfile+++ b/workers/model-text2motion/Dockerfile@@ -2,6 +2,12 @@ FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04# devel base: MDM's dependency stack (and optional CUDA ops) build cleanly with# nvcc present. TORCH_CUDA_ARCH_LIST targets the Cloud Run L4 (cc 8.9).+# PYTHONPATH only covers MDM: mdm_sampler.py relies on it being set at import+# time. momask_sampler.py manages its own sys.path at runtime instead (see its+# module docstring) specifically so momask-codes' repo dir is never a static,+# always-first PYTHONPATH entry: both repos ship an unrelated top-level+# `utils` package, and whichever was first on a static path would permanently+# shadow the other's in a long-lived container serving both backends.ENV PYTHONDONTWRITEBYTECODE=1 \PYTHONUNBUFFERED=1 \PIP_NO_CACHE_DIR=1 \@@ -36,6 +42,25 @@ RUN git clone --depth 1 https://github.com/GuyTevet/motion-diffusion-model.git /&& pip install scipy tqdm smplx \&& pip install git+https://github.com/openai/CLIP.git+# MoMask (EricGuo5513/momask-codes, MIT) is the stronger baseline (published+# FID 0.045 on HumanML3D vs. MDM-era T2M-GPT's 0.141): a residual-VQ tokenizer+# plus a masked + residual transformer, selected per request via+# `model: "momask"` (see momask_sampler.py). Not put on a static PYTHONPATH+# (see the ENV block above): momask_sampler.py inserts /opt/momask itself at+# load time. Every transitive import momask_sampler.py's `_load()` reaches+# (models.vq.model, models.vq.residual_vq, models.vq.quantizer,+# models.mask_transformer.transformer/tools, utils.get_opt,+# utils.motion_process, utils.paramUtil, common.skeleton, common.quaternion,+# utils.word_vectorizer) was walked against momask-codes' own source; the only+# dependency beyond what MDM's layer above already installs (torch, CLIP,+# scipy) is einops. momask's own requirements.txt additionally lists chumpy,+# matplotlib, scikit-learn, trimesh, gdown, vector-quantize-pytorch: those+# back its training/visualization/eval scripts and its OWN (unused, commented-+# out) VectorQuantize import, none are reachable from the inference path+# this worker calls.+RUN git clone --depth 1 https://github.com/EricGuo5513/momask-codes.git /opt/momask \+ && pip install einops+# MDM auxiliary data (glove, evaluators) and the trained checkpoint are mounted# at runtime from the weights bucket (MOTION_MODEL_DIR=/weights/mdm). The real# SMPL body model (`body_models/smpl/SMPL_NEUTRAL.pkl`) is NOT staged: it is@@ -57,7 +82,9 @@ RUN pip install -r /app/requirements.txtWORKDIR /appCOPY worker_security.py /app/worker_security.pyCOPY smpl_to_clip.py /app/smpl_to_clip.py+COPY motion_decode.py /app/motion_decode.pyCOPY mdm_sampler.py /app/mdm_sampler.py+COPY momask_sampler.py /app/momask_sampler.pyCOPY main.py /app/main.pyEXPOSE 8080diff --git a/workers/model-text2motion/README.md b/workers/model-text2motion/README.mdindex 831f953b6..0140199ed 100644--- a/workers/model-text2motion/README.md+++ b/workers/model-text2motion/README.md@@ -12,14 +12,34 @@ motion that does not pre-exist, rather than only applying preset clips.## Model-[MDM, Motion Diffusion Model](https://github.com/GuyTevet/motion-diffusion-model)-(GuyTevet), **MIT-licensed** → commercial-safe. Chosen over MoMask / T2M-GPT-specifically for the unambiguous commercial license. Swapping models touches only-`mdm_sampler.py`; the contract and the SMPL→clip conversion are model-agnostic.--The deployed checkpoint is the HumanML3D text-to-motion model: `arch=trans_enc`,-`diffusion_steps=50`, `guidance_param=2.5` (read from the checkpoint's own-`args.json` at load, not hardcoded).+Two selectable backends, both trained on HumanML3D and both **MIT-licensed**:++- **`mdm`** (default): [Motion Diffusion Model](https://github.com/GuyTevet/motion-diffusion-model)+ (GuyTevet). Diffusion sampling, one network, no residual-token stage: the+ cheaper backend, so it is the one warmed at container start.+ The deployed checkpoint is the HumanML3D text-to-motion model:+ `arch=trans_enc`, `diffusion_steps=50`, `guidance_param=2.5` (read from the+ checkpoint's own `args.json` at load, not hardcoded).+- **`momask`**: [MoMask](https://github.com/EricGuo5513/momask-codes)+ (EricGuo5513). A residual-VQ token vocabulary plus a masked transformer+ (base-layer tokens) and a residual transformer (the remaining five+ quantizer layers): the stronger HumanML3D baseline (published FID 0.045 vs.+ T2M-GPT's 0.141, the comparison point in+ [docs/research/3d-landscape-2026-10.md](../../docs/research/3d-landscape-2026-10.md)).+ Loads lazily on first request rather than at startup.++Both are MIT end to end; an earlier version of this doc framed MDM as chosen+"for the unambiguous commercial license" over MoMask, which was inaccurate+(MoMask's code license is the same MIT grant). The real historical distinction+was dependency surface, not licensing: momask-codes' own training/eval scripts+pull in SMPL/chumpy for visualization, which this worker never imports (see+the Dockerfile comment on `momask_sampler.py`'s actual reachable import set).++Select per request with `model: "mdm" | "momask"` on `POST /infer`; an absent+or unrecognized value falls back to `mdm`. Swapping/adding a backend touches+only `_load_model()` / `_generate_motion()` in `main.py` and that backend's own+`*_sampler.py`: the contract, the shared position to rotation decode+(`motion_decode.py`), and the SMPL to clip conversion are backend-agnostic.## Contract@@ -27,10 +47,11 @@ Identical shape to the other `model-*` workers. Every route except `/health`requires `Authorization: Bearer $API_KEY`.```-POST /infer { prompt, duration_seconds?=4, fps?=30, job_id? } → 202 { task_id, status }+POST /infer { prompt, duration_seconds?=4, fps?=30, model?="mdm", job_id? } → 202 { task_id, status }GET /tasks/:id → { task_id, status, model, result_url?, frames?, fps?, elapsed_ms?, error? }-GET /health → { ok, model_loaded, status: loading|ready|failed, load_error }-GET / → { service, model, health, endpoints } (identity, no auth)+GET /health → { ok, model_loaded, models_loaded, models_available, default_model,+ status: loading|ready|failed, load_error, load_errors }+GET / → { service, model, models_available, health, endpoints } (identity, no auth)````prompt` is 3 to 1000 characters, `fps` is 8 to 60, and `duration_seconds` is@@ -47,18 +68,31 @@ The platform reaches this worker through the GCP provider's `text2motion` mode## Pipeline-1. `mdm_sampler.MdmSampler.sample(prompt, n_frames)` samples the diffusion model,- denormalizes with the HumanML3D dataset stats, and calls upstream's- `recover_from_ric` to get global joint **positions** `(T,22,3)`.-2. `mdm_sampler._positions_to_local_quats` recovers each joint's **local- rotation** from where its own children landed (Kabsch fit against the rest- directions), then `_quat_to_axis_angle` + `_resample` produce SMPL-indexed- axis-angle `(T,22,3)` plus root translation `(T,3)` at the requested frame- count.+1. `mdm_sampler.MdmSampler.sample(prompt, n_frames)` samples the diffusion+ model, denormalizes with the HumanML3D dataset stats, and calls upstream's+ `recover_from_ric` to get global joint **positions** `(T,22,3)`. The+ `momask` backend takes a different route to the same shape:+ `momask_sampler.MomaskSampler.sample(prompt, n_frames)` tokenizes the+ prompt with the masked transformer, expands the residual quantizer layers+ with the residual transformer, decodes the RVQ tokens back to the+ HumanML3D feature vector, and calls the *same* upstream `recover_from_ric`+ (momask-codes' own copy, numerically identical signature) to reach the+ identical `(T,22,3)` position representation.+2. `motion_decode._positions_to_local_quats` (shared by both samplers, model-+ agnostic) recovers each joint's **local rotation** from where its own+ children landed (Kabsch fit against the rest directions), then+ `_quat_to_axis_angle` + `_resample` produce SMPL-indexed axis-angle+ `(T,22,3)` plus root translation `(T,3)` at the requested frame count. Both+ `mdm_sampler.py` and `momask_sampler.py` import this module rather than+ duplicating it, because both upstream repos separately ship an unrelated+ top-level `utils` package: keeping the shared math in a module named+ neither `utils` avoids a namespace collision in a container that serves+ both backends over its lifetime (see `momask_sampler.py`'s docstring).3. `smpl_to_clip.smpl_motion_to_clip(poses, trans, fps)` emits the AnimationClipJSON: 22 quaternion tracks with canonical Wolf3D bone names(`Hips.quaternion`, …) plus `Hips.position`.-4. Upload to `gs://$GCS_BUCKET/motion-clips/mdm/<task_id>.json`; return the URL.+4. Upload to `gs://$GCS_BUCKET/motion-clips/<model>/<task_id>.json` (`<model>`+ is `mdm` or `momask`); return the URL.## Environment@@ -66,7 +100,8 @@ The platform reaches this worker through the GCP provider's `text2motion` mode|---|---|---|| `API_KEY` | yes | Bearer secret; mounted from Secret Manager (`avatar-reconstruction-key`) || `GCS_BUCKET` | yes | Output bucket (`three-ws-avatar-reconstructions`) |-| `MOTION_MODEL_DIR` | no | Checkpoint dir, default `/weights/mdm` |+| `MOTION_MODEL_DIR` | no | MDM checkpoint dir, default `/weights/mdm` |+| `MOMASK_MODEL_DIR` | no | MoMask checkpoint dir, default `/weights/momask` || `MAX_CONCURRENT` | no | In-instance job semaphore, default 2 || `MAX_DURATION_SEC` | no | Hard cap on requested clip length, default 10 |@@ -76,13 +111,29 @@ The platform reaches this worker through the GCP provider's `text2motion` mode`/weights/mdm` by the Cloud Run GCS volume: `model.pt` (78 MiB checkpoint) and`args.json` (the checkpoint's own hyperparameters).-The real SMPL body model (`SMPL_NEUTRAL.pkl`) is deliberately **not** staged: it-is gated behind SMPL's own registration license, and this worker never needs it.-`model.mdm.MDM` eagerly constructs a `Rotation2xyz`/`SMPL` submodule that would-load it, so `mdm_sampler` patches in a real (license-file-free) `nn.Module`-stand-in purely so `.to(device)` / `.eval()` have something to walk into. That-submodule's output is never consumed: motion is decoded from the HumanML3D-feature vectors instead. See the Dockerfile comment for the full reasoning.+`gs://three-ws-model-weights/momask/t2m/` mirrors momask-codes' own released+HumanML3D checkpoint layout (its `prepare/download_models.sh` names these+exact directories), mounted at `/weights/momask/t2m/` under the same volume+(`momask_sampler.py` joins `MOMASK_MODEL_DIR` + `"t2m"` + each checkpoint name+itself, matching upstream's own `get_opt` path construction):++```+momask/t2m/+ rvq_nq6_dc512_nc512_noshare_qdp0.2/{opt.txt, meta/{mean,std}.npy, model/net_best_fid.tar}+ t2m_nlayer8_nhead6_ld384_ff1024_cdp0.1_rvq6ns/{opt.txt, model/latest.tar}+ tres_nlayer8_ld384_ff1024_rvq6ns_cdp0.2_sw/{opt.txt, model/net_best_fid.tar}+ length_estimator/model/finest.tar+```++The real SMPL body model (`SMPL_NEUTRAL.pkl`) is deliberately **not** staged for+either backend: it is gated behind SMPL's own registration license, and+neither sampler needs it. `model.mdm.MDM` eagerly constructs a+`Rotation2xyz`/`SMPL` submodule that would load it, so `mdm_sampler` patches in+a real (license-file-free) `nn.Module` stand-in purely so `.to(device)` /+`.eval()` have something to walk into. That submodule's output is never+consumed: motion is decoded from the HumanML3D feature vectors instead. MoMask+has no equivalent construction-time SMPL load in its inference path at all.+See the Dockerfile comment for the full reasoning.## Where it runs@@ -123,24 +174,43 @@ fails on `ModuleNotFoundError: smpl_to_clip`).unit-norm, the SMPL→Wolf3D bone mapping, clip JSON shape (track names, valuelengths, monotonic times, duration), rest-offset calibration, flattened-poseinput, single-frame/static, and input validation.-- `test_mdm_decode.py` (21 cases): the decode path this worker owns. The- kinematic-chain flattening, the Kabsch solve, `_quat_to_axis_angle` as the- exact inverse of `axis_angle_to_quaternion`, `_resample`, and a- forward-kinematics → `_positions_to_local_quats` → forward-kinematics round- trip that must reproduce the input positions (asserted on positions rather- than rotations, because single-child joints carry no twist signal in position- data). Two smoke cases run the whole owned chain end to end: joint positions- in, a valid AnimationClip out.+- `test_mdm_decode.py` (21 cases): the decode path this worker owns, now+ living in `motion_decode.py` and re-exported through `mdm_sampler.py` for+ import-path backward compatibility. The kinematic-chain flattening, the+ Kabsch solve, `_quat_to_axis_angle` as the exact inverse of+ `axis_angle_to_quaternion`, `_resample`, and a forward-kinematics →+ `_positions_to_local_quats` → forward-kinematics round trip that must+ reproduce the input positions (asserted on positions rather than rotations,+ because single-child joints carry no twist signal in position data). Two+ smoke cases run the whole owned chain end to end: joint positions in, a+ valid AnimationClip out. This same shared module is what `momask_sampler.py`+ calls for its own position→rotation step, so these cases cover both+ backends' decode math.+- `test_model_selector.py` (12 cases): `canonical_model()`'s resolution and+ unknown-name fallback, and the full `/infer` → `/tasks/:id` → `/health` → `/`+ request flow with a fake sampler and an in-memory recording GCS bucket+ standing in for the GPU backend and storage (no torch, no network):+ the default (`mdm`) path, explicit `model: "momask"` routing to the+ `momask`-namespaced GCS blob path, an unrecognized model name falling back+ to `mdm` rather than erroring, and `/health`'s per-backend `models_loaded`+ reporting.The JS provider mode + REST endpoint are covered by`tests/api/forge-motion.test.js` (`npm test`).-What only a deploy can validate is the pair of upstream calls in-`_decode_to_smpl` (`recover_from_ric` and the sampler itself), which need the-GPU image and the checkpoint. Those are exercised in production: verified-2026-08-11 against the us-east4 lane, a 3-second 30 fps prompt returning 90-frames in 2.7 s warm, with all 22 quaternion tracks unit-norm, 17 of 22 bones-actually animating, and `Hips.position` carrying real travel.+What only a deploy can validate is each backend's own upstream inference call+(MDM's `p_sample_loop` + `recover_from_ric`; MoMask's masked/residual+transformer `generate()` + RVQ `forward_decoder` + `recover_from_ric`), which+needs the GPU image and real checkpoints. MDM is exercised in production:+verified 2026-08-11 against the us-east4 lane, a 3-second 30 fps prompt+returning 90 frames in 2.7 s warm, with all 22 quaternion tracks unit-norm, 17+of 22 bones actually animating, and `Hips.position` carrying real travel.+MoMask has not yet had an in-house production run: `gcloud builds submit` /+`gcloud run deploy` are not available in every environment this worker is+developed from, so until a deploy lands, treat MoMask's FID 0.045 figure above+as momask-codes' own published HumanML3D number, not a number reproduced by+this deployment. See `prompts/finish/_context/production-100-OWNER-ACTIONS.md`+for the exact staging + deploy + benchmark commands.`smpl_to_clip`'s `rest_offsets` parameter defaults to identity, which emits theraw SMPL local rotations. That is the correct baseline for SMPL-rest targets;@@ -172,4 +242,12 @@ curl -s -X POST "$URL/infer" -H "authorization: Bearer $KEY" \-H 'content-type: application/json' \-d '{"prompt":"a person waves hello with the right hand","duration_seconds":3,"fps":30}'curl -s "$URL/tasks/<task_id>" -H "authorization: Bearer $KEY"++# MoMask: same contract, model: "momask". First call on a cold container pays+# the lazy-load cost (checkpoint from the weights volume); time the SECOND+# call for a real warm benchmark to compare against MDM's.+curl -s -X POST "$URL/infer" -H "authorization: Bearer $KEY" \+ -H 'content-type: application/json' \+ -d '{"prompt":"a person waves hello with the right hand","duration_seconds":3,"fps":30,"model":"momask"}'+curl -s "$URL/tasks/<task_id>" -H "authorization: Bearer $KEY"```diff --git a/workers/model-text2motion/cloudbuild.yaml b/workers/model-text2motion/cloudbuild.yamlindex 354ec8fba..cc2abb3f9 100644--- a/workers/model-text2motion/cloudbuild.yaml+++ b/workers/model-text2motion/cloudbuild.yaml@@ -63,7 +63,7 @@ steps:- --allow-unauthenticated- --add-volume=name=weights,type=cloud-storage,bucket=${_WEIGHTS_BUCKET}- --add-volume-mount=volume=weights,mount-path=/weights- - --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},MOTION_MODEL_DIR=/weights/mdm,MAX_CONCURRENT=2+ - --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},MOTION_MODEL_DIR=/weights/mdm,MOMASK_MODEL_DIR=/weights/momask,MAX_CONCURRENT=2- --set-secrets=API_KEY=avatar-reconstruction-key:latest# The dormant failback copy. Same image, same flags, min 0 so it holds no GPU@@ -92,7 +92,7 @@ steps:- --allow-unauthenticated- --add-volume=name=weights,type=cloud-storage,bucket=${_WEIGHTS_BUCKET}- --add-volume-mount=volume=weights,mount-path=/weights- - --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},MOTION_MODEL_DIR=/weights/mdm,MAX_CONCURRENT=2+ - --set-env-vars=GCS_BUCKET=${_GCS_BUCKET},MOTION_MODEL_DIR=/weights/mdm,MOMASK_MODEL_DIR=/weights/momask,MAX_CONCURRENT=2- --set-secrets=API_KEY=avatar-reconstruction-key:latestimages:diff --git a/workers/model-text2motion/main.py b/workers/model-text2motion/main.pyindex 0458133d1..06c654d75 100644--- a/workers/model-text2motion/main.py+++ b/workers/model-text2motion/main.py@@ -1,31 +1,39 @@-"""Text-to-motion service — prompt → retargetable three.js animation clip.+"""Text-to-motion service: prompt to retargetable three.js animation clip.-Wraps a text-to-motion diffusion model (MDM — Motion Diffusion Model,-GuyTevet/motion-diffusion-model, MIT) on GPU. Given a natural-language prompt-("waving confidently", "a slow tai-chi sweep") it samples SMPL-skeleton motion,-then converts it to a canonical three.js AnimationClip JSON (Wolf3D bone names —-see smpl_to_clip.py) that the platform retargets onto any rigged avatar with the+Wraps a text-to-motion model on GPU, selectable per request between two MIT-+licensed backends. Given a natural-language prompt ("waving confidently", "a+slow tai-chi sweep") it samples SMPL-skeleton motion, then converts it to a+canonical three.js AnimationClip JSON (Wolf3D bone names, see+smpl_to_clip.py) that the platform retargets onto any rigged avatar with thesame engine the curated animation library uses. Output is uploaded to GCS andthe URL returned, mirroring the model-* worker contract exactly.-Licensing- - MDM (GuyTevet/motion-diffusion-model) — MIT, commercial-OK. Chosen over- MoMask / T2M-GPT specifically because MIT is unambiguously commercial-safe;- if a higher-quality model with a commercial license is adopted later, only- `_generate_motion()` changes — the contract and the clip conversion do not.+Models+ - MDM (GuyTevet/motion-diffusion-model), MIT. The default: diffusion+ sampling, one network, no residual-token stage, so it is the cheaper+ backend to keep warm.+ - MoMask (EricGuo5513/momask-codes), MIT. The stronger baseline on+ HumanML3D (published FID 0.045 vs. T2M-GPT's 0.141): a residual-VQ token+ vocabulary plus a masked transformer and a residual transformer. Selected+ per request via `model: "momask"`; see momask_sampler.py.+ Both are trained on HumanML3D and decode to the same 22-joint position+ representation (`motion_decode.py`'s shared, pure-NumPy position→rotation+ IK is backend-agnostic), so swapping models touches only `_load_model()` /+ `_generate_motion()`; the contract and the clip conversion do not change.API contract (identical shape to the other model-* workers):- POST /infer { prompt: str, duration_seconds?: float=4, fps?: int=30, job_id?: str }+ POST /infer { prompt: str, duration_seconds?: float=4, fps?: int=30, model?: str="mdm", job_id?: str }→ 202 { task_id, status: "queued" }GET /tasks/:id → { task_id, status, result_url?, frames?, fps?, error? }GET /health → { ok }Environment:- API_KEY — bearer secret (required)- GCS_BUCKET — output bucket (required)- MAX_CONCURRENT — default 2- MOTION_MODEL_DIR — directory of MDM checkpoints (mounted from GCS)- MAX_DURATION_SEC — hard cap on requested clip length (default 10)+ API_KEY bearer secret (required)+ GCS_BUCKET output bucket (required)+ MAX_CONCURRENT default 2+ MOTION_MODEL_DIR directory of MDM checkpoints (mounted from GCS)+ MOMASK_MODEL_DIR directory of MoMask checkpoints (mounted from GCS)+ MAX_DURATION_SEC hard cap on requested clip length (default 10)"""from __future__ import annotations@@ -54,44 +62,59 @@ API_KEY = os.environ["API_KEY"]GCS_BUCKET = os.environ["GCS_BUCKET"]MAX_CONCURRENT = int(os.environ.get("MAX_CONCURRENT", "2"))MOTION_MODEL_DIR = os.environ.get("MOTION_MODEL_DIR", "/weights/mdm")+MOMASK_MODEL_DIR = os.environ.get("MOMASK_MODEL_DIR", "/weights/momask")MAX_DURATION_SEC = float(os.environ.get("MAX_DURATION_SEC", "10"))DEFAULT_FPS = 30+# Canonical backend names. The default is warmed at container start; the rest+# load lazily on first use (same pattern as workers/rembg's multi-model+# selector, see rembg_policy.canonical_model).+CANONICAL_MODELS = ("mdm", "momask")+DEFAULT_MODEL = "mdm"+_bucket: Optional[storage.Bucket] = None_sem: Optional[asyncio.Semaphore] = None-_ready: Optional[asyncio.Event] = None-_load_error: Optional[str] = None_tasks: dict[str, dict] = {}-_model = None # lazily-loaded MDM sampler+_models: dict[str, object] = {} # lazily-loaded samplers, keyed by canonical name+_model_locks: dict[str, asyncio.Lock] = {name: asyncio.Lock() for name in CANONICAL_MODELS}+_load_errors: dict[str, str] = {}+++def canonical_model(name: Optional[str]) -> str:+ """Resolve a caller-supplied model name, falling back to the default.++ An unknown name falls back rather than 400s, because a generation request+ with a stale model name still has an obviously right thing to do.+ """+ resolved = (name or "").strip().lower()+ return resolved if resolved in CANONICAL_MODELS else DEFAULT_MODEL-async def _load_model_bg() -> None:- global _load_error+async def _load_model_bg(name: str) -> None:loop = asyncio.get_event_loop()try:- await loop.run_in_executor(None, _load_model)- _ready.set()- log.info("text2motion ready — max_concurrent=%d", MAX_CONCURRENT)+ await loop.run_in_executor(None, _load_model, name)+ log.info("%s ready, max_concurrent=%d", name, MAX_CONCURRENT)except Exception as exc: # noqa: BLE001- _load_error = safe_error(exc, context="MDM checkpoint load")- log.error("MDM model load FAILED: %s", exc)+ _load_errors[name] = safe_error(exc, context=f"{name} checkpoint load")+ log.error("%s model load FAILED: %s", name, exc)@asynccontextmanagerasync def lifespan(app: FastAPI):- global _bucket, _sem, _ready+ global _bucket, _sem_bucket = storage.Client().bucket(GCS_BUCKET)_sem = asyncio.Semaphore(MAX_CONCURRENT)- _ready = asyncio.Event()- # Load the checkpoint in the BACKGROUND and yield immediately — uvicorn- # runs the ASGI lifespan before binding the socket, and the full load- # (MDM transformer + CUDA transfer from the GCS FUSE-mounted weights dir)- # can run past Cloud Run's startup TCP-probe window on a cold instance.- # Backgrounding opens the port at once; requests that arrive before the- # load completes wait on `_ready` in `_run_inference`. Same pattern as- # model-trellis / model-triposg.- asyncio.create_task(_load_model_bg())- log.info("text2motion starting, model loading in background (max_concurrent=%d)", MAX_CONCURRENT)+ # Load the DEFAULT checkpoint in the BACKGROUND and yield immediately:+ # uvicorn runs the ASGI lifespan before binding the socket, and the full+ # load (transformer weights + CUDA transfer from the GCS FUSE-mounted+ # weights dir) can run past Cloud Run's startup TCP-probe window on a cold+ # instance. Backgrounding opens the port at once; requests that arrive+ # before the load completes wait on `_ensure_loaded` in `_run_inference`.+ # Same pattern as model-trellis / model-triposg. Non-default backends+ # (momask) load on first request instead of at startup, matching rembg.+ asyncio.create_task(_load_model_bg(DEFAULT_MODEL))+ log.info("text2motion starting, %s loading in background (max_concurrent=%d)", DEFAULT_MODEL, MAX_CONCURRENT)yield@@ -105,25 +128,49 @@ def _require_api_key(authorization: str) -> None:raise HTTPException(status_code=401, detail=str(exc)) from exc-def _load_model() -> None:- """Load the MDM sampler once per container.+def _load_model(name: str) -> None:+ """Load one backend's sampler once per container.Kept import-local so the FastAPI app + the pure conversion path import- cleanly in environments without torch/MDM installed (tests, CI). On the GPU- runtime this loads the checkpoint from MOTION_MODEL_DIR.+ cleanly in environments without torch/the model repos installed (tests,+ CI). On the GPU runtime this loads the checkpoint from the backend's+ model dir."""- global _model- from mdm_sampler import MdmSampler # provided in the GPU image (see Dockerfile)+ if name == "momask":+ from momask_sampler import MomaskSampler # provided in the GPU image (see Dockerfile)- _model = MdmSampler(model_dir=MOTION_MODEL_DIR, device="cuda")- log.info("MDM checkpoint loaded from %s", MOTION_MODEL_DIR)+ _models[name] = MomaskSampler(model_dir=MOMASK_MODEL_DIR, device="cuda")+ log.info("MoMask checkpoints loaded from %s", MOMASK_MODEL_DIR)+ else:+ from mdm_sampler import MdmSampler # provided in the GPU image (see Dockerfile)+ _models[name] = MdmSampler(model_dir=MOTION_MODEL_DIR, device="cuda")+ log.info("MDM checkpoint loaded from %s", MOTION_MODEL_DIR)-def _generate_motion(prompt: str, n_frames: int):- """Sample SMPL motion for `prompt`. Returns (poses (T,24,3), trans (T,3))."""- if _model is None:- raise RuntimeError("motion model not loaded")- poses, trans = _model.sample(prompt=prompt, n_frames=n_frames)++async def _ensure_loaded(name: str) -> None:+ """Block until `name`'s sampler is loaded, loading it now if nobody has."""+ if name in _models:+ return+ async with _model_locks[name]:+ if name in _models: # a concurrent caller may have finished while we waited+ return+ if name in _load_errors:+ raise RuntimeError(_load_errors[name])+ loop = asyncio.get_event_loop()+ try:+ await loop.run_in_executor(None, _load_model, name)+ except Exception as exc: # noqa: BLE001+ _load_errors[name] = safe_error(exc, context=f"{name} checkpoint load")+ raise+++def _generate_motion(name: str, prompt: str, n_frames: int):+ """Sample SMPL motion for `prompt` on backend `name`. Returns (poses, trans)."""+ model = _models.get(name)+ if model is None:+ raise RuntimeError(f"{name} model not loaded")+ poses, trans = model.sample(prompt=prompt, n_frames=n_frames)return poses, trans@@ -131,17 +178,19 @@ class InferRequest(BaseModel):prompt: str = Field(..., min_length=3, max_length=1000)duration_seconds: float = Field(default=4.0, gt=0)fps: int = Field(default=DEFAULT_FPS, ge=8, le=60)+ model: Optional[str] = Nonejob_id: Optional[str] = None@app.post("/infer", status_code=202)async def infer(body: InferRequest, background_tasks: BackgroundTasks, authorization: str = Header(...)) -> dict:_require_api_key(authorization)+ model_name = canonical_model(body.model)task_id = body.job_id or str(uuid.uuid4())- _tasks[task_id] = {"task_id": task_id, "status": "queued", "model": "mdm"}+ _tasks[task_id] = {"task_id": task_id, "status": "queued", "model": model_name}duration = min(body.duration_seconds, MAX_DURATION_SEC)n_frames = max(2, int(round(duration * body.fps)))- background_tasks.add_task(_run_inference, task_id, body.prompt, n_frames, body.fps)+ background_tasks.add_task(_run_inference, task_id, model_name, body.prompt, n_frames, body.fps)return {"task_id": task_id, "status": "queued"}@@ -166,7 +215,8 @@ async def root() -> dict:"""return {"service": "model-text2motion",- "model": "mdm",+ "model": DEFAULT_MODEL,+ "models_available": list(CANONICAL_MODELS),"health": "/health","endpoints": ["POST /infer", "GET /tasks/{id}", "GET /health"],}@@ -174,30 +224,35 @@ async def root() -> dict:@app.get("/health")async def health() -> dict:+ loaded = sorted(_models)return {"ok": True,- "model_loaded": _model is not None,- "status": "ready" if _model is not None else ("failed" if _load_error else "loading"),- "load_error": _load_error,+ "model_loaded": DEFAULT_MODEL in _models,+ "models_loaded": loaded,+ "models_available": list(CANONICAL_MODELS),+ "default_model": DEFAULT_MODEL,+ "status": "ready" if DEFAULT_MODEL in _models else ("failed" if DEFAULT_MODEL in _load_errors else "loading"),+ "load_error": _load_errors.get(DEFAULT_MODEL),+ "load_errors": dict(_load_errors),}-async def _run_inference(task_id: str, prompt: str, n_frames: int, fps: int) -> None:- assert _sem is not None and _bucket is not None and _ready is not None+async def _run_inference(task_id: str, model_name: str, prompt: str, n_frames: int, fps: int) -> None:+ assert _sem is not None and _bucket is not Noneasync with _sem:_tasks[task_id]["status"] = "running"started = time.time()loop = asyncio.get_event_loop()try:try:- await asyncio.wait_for(_ready.wait(), timeout=600)+ await asyncio.wait_for(_ensure_loaded(model_name), timeout=600)except asyncio.TimeoutError:- raise RuntimeError(_load_error or "model load timed out")- poses, trans = await loop.run_in_executor(None, _generate_motion, prompt, n_frames)+ raise RuntimeError(_load_errors.get(model_name) or f"{model_name} load timed out")+ poses, trans = await loop.run_in_executor(None, _generate_motion, model_name, prompt, n_frames)clip = smpl_motion_to_clip(poses, trans, fps=fps, name=_safe_name(prompt))payload = json.dumps(clip).encode("utf-8")- blob_name = f"motion-clips/mdm/{task_id}.json"+ blob_name = f"motion-clips/{model_name}/{task_id}.json"blob = _bucket.blob(blob_name)await loop.run_in_executor(None,@@ -213,7 +268,7 @@ async def _run_inference(task_id: str, prompt: str, n_frames: int, fps: int) ->"elapsed_ms": int((time.time() - started) * 1000),})- log.info("task %s done — %d frames in %dms", task_id, n_frames, int((time.time() - started) * 1000))+ log.info("task %s done, %d frames in %dms", task_id, n_frames, int((time.time() - started) * 1000))except Exception as exc: # surface an opaque, correlated error to the callermsg = safe_error(exc, context=f"text2motion task {task_id}")_tasks[task_id].update({"status": "failed", "error": msg})diff --git a/workers/model-text2motion/mdm_sampler.py b/workers/model-text2motion/mdm_sampler.pyindex 44af76264..5b9b398f9 100644--- a/workers/model-text2motion/mdm_sampler.py+++ b/workers/model-text2motion/mdm_sampler.py@@ -17,6 +17,18 @@ import loggingimport numpy as np+from motion_decode import (+ _kabsch_quat,+ _parents_from_kinematic_chain,+ _positions_to_local_quats,+ _quat_conjugate,+ _quat_multiply,+ _quat_rotate,+ _quat_to_axis_angle,+ _resample,+ _rotation_matrix_to_quat,+)+log = logging.getLogger("text2motion.mdm")# HumanML3D motion is sampled at 20 fps; we resample to the requested fps in the@@ -34,7 +46,7 @@ class MdmSampler:self._load()def _load(self) -> None:- # Imports are GPU-only — kept here so the module imports without torch.+ # Imports are GPU-only: kept here so the module imports without torch.import jsonimport osfrom types import SimpleNamespace@@ -47,7 +59,7 @@ class MdmSampler:# MDM.__init__ unconditionally builds `self.rot2xyz = Rotation2xyz(...)`,# whose constructor loads the real SMPL body model (a .pkl gated behind- # SMPL's own registration/license at smpl.is.tue.mpg.de — not something+ # SMPL's own registration/license at smpl.is.tue.mpg.de, not something# an automated pipeline can fetch). This worker never calls# `rot2xyz(...)` (the thing that would actually need real SMPL data):# motion comes back from the diffusion sampler as HumanML3D feature@@ -56,7 +68,7 @@ class MdmSampler:# rotation2xyz. But `MDM._apply`/`MDM.train` (invoked by the# `.to(device)`/`.eval()` calls below, which is how the model actually# loads onto the GPU) reach into `self.rot2xyz.smpl_model._apply`/- # `.train`, so it still needs to be a real `nn.Module` — just one that+ # `.train`, so it still needs to be a real `nn.Module`, just one that# never had to load a license-gated file to exist. Patched on the# already-imported `model.mdm` module object so it takes effect# regardless of import order.@@ -76,7 +88,7 @@ class MdmSampler:mdm_module.Rotation2xyz = _StubRotation2xyz# utils.parser_util.generate_args() takes no parameters and parses- # sys.argv (a CLI-only entry point, not usable from a server process —+ # sys.argv (a CLI-only entry point, not usable from a server process:# a prior version of this file called it as `generate_args(model_path=...)`,# which doesn't match its signature at all). The released checkpoints# ship an `args.json` sibling to the `.pt` file with every field the@@ -98,14 +110,14 @@ class MdmSampler:# NOT `model.to(device).eval()`: torch's Module.to() returns# `self._apply(convert)` (verified against torch 2.3.1's own source),# but MDM's `_apply` override (`def _apply(self, fn): super()._apply(fn);- # self.rot2xyz.smpl_model._apply(fn)`) never returns `self` — an+ # self.rot2xyz.smpl_model._apply(fn)`) never returns `self`, an# upstream bug that makes `.to()` return None regardless of which# smpl_model is attached. `model` itself is still correctly mutated# in place, so call `.to()`/`.eval()` for their side effects only and# keep using the original reference.model.to(self.device)model.eval()- # Classifier-free guidance sampling wrapper — matches sample/generate.py's+ # Classifier-free guidance sampling wrapper: matches sample/generate.py's# own gating (skip the wrapper entirely when guidance_param == 1).if self._guidance_param != 1:model = ClassifierFreeSampleModel(model)@@ -132,7 +144,7 @@ class MdmSampler:# checkpoint has mask_frames=False, so the key must exist regardless. A# single unpadded prompt at exactly `horizon` frames has no padding, so# the honest mask a real (non-padded) batch's collate would produce is- # all-ones over the full length — shape (bs, 1, 1, horizon), matching+ # all-ones over the full length: shape (bs, 1, 1, horizon), matching# sample/generate.py's own `model_kwargs['y']['mask']` convention.mask = torch.ones((1, 1, 1, horizon), dtype=torch.bool, device=self.device)model_kwargs = {@@ -164,11 +176,11 @@ def _decode_to_smpl(sample):MDM's HumanML3D checkpoint emits a 263-dim feature vector per frame.`recover_from_ric` (upstream, real) turns that back into global joint- *positions* on the 22-joint HumanML3D skeleton — the SMPL body joints 0-21+ *positions* on the 22-joint HumanML3D skeleton: the SMPL body joints 0-21under the same kinematic tree (paramUtil.t2m_kinematic_chain).There is no upstream `positions_to_smpl_poses` in the MDM repo (a prior- version of this file imported one that does not exist — verified against+ version of this file imported one that does not exist, verified againstgithub.com/GuyTevet/motion-diffusion-model). Converting positions to localjoint *rotations* is done here directly by `_positions_to_local_quats`: ajoint's local rotation is recovered from where ITS OWN children ended up@@ -176,7 +188,7 @@ def _decode_to_smpl(sample):rotations), via a best-fit (Kabsch) rotation from each child's restdirection (`paramUtil.t2m_raw_offsets`, upstream data) to its measureddirection. This carries no twist signal (same limitation HumanML3D's- output has generally) — `smpl_to_clip`'s `rest_offsets` param is the+ output has generally): `smpl_to_clip`'s `rest_offsets` param is theintended place to calibrate the residual SMPL-rest vs. Wolf3D-restorientation offset."""@@ -195,7 +207,7 @@ def _decode_to_smpl(sample):feats = sample.squeeze(0).squeeze(1).permute(1, 0).float() # (T, 263)# The diffusion model samples in HumanML3D's NORMALIZED feature space- # (zero mean / unit std over the training set) — recover_from_ric expects+ # (zero mean / unit std over the training set): recover_from_ric expects# real units. Denormalize with the checkpoint-independent HumanML3D dataset# stats MDM ships in its own repo (`dataset/t2m_{mean,std}.npy`, small# precomputed arrays, not a license-gated model file), exactly the@@ -217,158 +229,3 @@ def _decode_to_smpl(sample):trans = positions[:, 0, :] # root joint IS the world root positionreturn poses, trans--def _parents_from_kinematic_chain(kinematic_chain, n_joints: int) -> np.ndarray:- """Flatten paramUtil's per-limb chains (each starting at the root, joint 0)- into a single parent-index array, e.g. [-1, 2, 5, 8, ...] for a 22-joint- skeleton. Every joint in the topology appears in exactly one chain."""- parents = np.full(n_joints, -1, dtype=np.int64)- for chain in kinematic_chain:- for i in range(1, len(chain)):- parents[chain[i]] = chain[i - 1]- return parents---def _quat_conjugate(q: np.ndarray) -> np.ndarray:- return np.concatenate([-q[..., :3], q[..., 3:4]], axis=-1)---def _quat_multiply(a: np.ndarray, b: np.ndarray) -> np.ndarray:- ax, ay, az, aw = a[..., 0], a[..., 1], a[..., 2], a[..., 3]- bx, by, bz, bw = b[..., 0], b[..., 1], b[..., 2], b[..., 3]- return np.stack(- [- aw * bx + ax * bw + ay * bz - az * by,- aw * by - ax * bz + ay * bw + az * bx,- aw * bz + ax * by - ay * bx + az * bw,- aw * bw - ax * bx - ay * by - az * bz,- ],- axis=-1,- )---def _quat_rotate(q: np.ndarray, v: np.ndarray) -> np.ndarray:- qv = np.concatenate([v, np.zeros_like(v[..., :1])], axis=-1)- return _quat_multiply(_quat_multiply(q, qv), _quat_conjugate(q))[..., :3]---def _rotation_matrix_to_quat(r: np.ndarray) -> np.ndarray:- """(...,3,3) proper rotation matrices → (...,4) xyzw quaternions."""- m00, m01, m02 = r[..., 0, 0], r[..., 0, 1], r[..., 0, 2]- m10, m11, m12 = r[..., 1, 0], r[..., 1, 1], r[..., 1, 2]- m20, m21, m22 = r[..., 2, 0], r[..., 2, 1], r[..., 2, 2]- trace = m00 + m11 + m22- w = 0.5 * np.sqrt(np.maximum(1.0 + trace, 1e-12))- safe_w = np.where(w < 1e-6, 1.0, w)- x = (m21 - m12) / (4.0 * safe_w)- y = (m02 - m20) / (4.0 * safe_w)- z = (m10 - m01) / (4.0 * safe_w)- quat = np.stack([x, y, z, w], axis=-1)- norm = np.linalg.norm(quat, axis=-1, keepdims=True)- return quat / np.where(norm < 1e-8, 1.0, norm)---def _kabsch_quat(sources: np.ndarray, targets: np.ndarray) -> np.ndarray:- """Best-fit rotation quaternion mapping unit-vector set `sources` onto- `targets`, batched over frames. `sources`/`targets`: (T, K, 3). Minimizes- sum_k |R @ sources[k] - targets[k]|^2 (Kabsch/Procrustes, reflection-- corrected so R is always a proper rotation). K == 1 degenerates to a- shortest-arc rotation with an unconstrained twist about that one axis —- the same ambiguity single-child joints have regardless of method, since- HumanML3D position data carries no twist signal for any joint."""- h = np.einsum("tka,tkb->tab", sources, targets) # (T,3,3)- u, _, vt = np.linalg.svd(h)- d = np.sign(np.linalg.det(np.einsum("tab,tbc->tac", vt.transpose(0, 2, 1), u.transpose(0, 2, 1))))- diag = np.tile(np.eye(3), (h.shape[0], 1, 1))- diag[:, 2, 2] = d- r = np.einsum("tab,tbc,tcd->tad", vt.transpose(0, 2, 1), diag, u.transpose(0, 2, 1))- return _rotation_matrix_to_quat(r)---def _positions_to_local_quats(- positions: np.ndarray, parents: np.ndarray, rest_dirs: np.ndarray-) -> np.ndarray:- """(T,J,3) global positions → (T,J,4) local xyzw quaternions per the- `parents`/`rest_dirs` topology.-- A joint's local rotation `R_j` is NOT observable from its own position —- under standard skeletal FK (global_j = global_parent(j) * R_j, and the- *fixed* rest offset rotates with the PARENT's global orientation only:- pos[j] - pos[parent(j)] = rotate(global_parent(j), rest_dirs[j])) that- offset doesn't involve R_j at all. R_j only shows up in where j's OWN- children end up. So this recovers R_j, per joint, from the directions to- j's children: best-fit (Kabsch) rotation mapping each child's rest- direction (in j's parent-relative frame) onto its measured direction,- across however many children j has (root and the chest joint branch into- 3; most joints have exactly 1, where Kabsch degenerates to a shortest-arc- solve). Leaf joints (hands, feet, head) have no children and no positional- evidence for their own twist, so they get an identity local rotation.- """- n_frames, n_joints, _ = positions.shape- local_quat = np.tile(np.array([0.0, 0.0, 0.0, 1.0]), (n_frames, n_joints, 1))- global_quat = np.tile(np.array([0.0, 0.0, 0.0, 1.0]), (n_frames, n_joints, 1))-- children_of = [[] for _ in range(n_joints)]- for j in range(1, n_joints):- children_of[parents[j]].append(j)-- # Single top-down pass: `parents[j] < j` for this topology (verified — the- # kinematic chains are built root-outward), so global_quat[parents[j]] is- # always already resolved by the time joint j is processed.- for j in range(n_joints):- kids = children_of[j]- if kids:- rest = rest_dirs[np.array(kids)] # (K,3)- rest = rest / np.linalg.norm(rest, axis=-1, keepdims=True)- rest = np.broadcast_to(rest, (n_frames,) + rest.shape).copy() # (T,K,3)-- measured = positions[:, kids, :] - positions[:, j : j + 1, :] # (T,K,3)- measured = measured / np.linalg.norm(measured, axis=-1, keepdims=True)-- if j == 0:- targets_in_parent_frame = measured # root: parent frame == world- else:- p = parents[j]- k = measured.shape[1]- parent_conj = np.broadcast_to(_quat_conjugate(global_quat[:, p])[:, None, :], (n_frames, k, 4))- targets_in_parent_frame = _quat_rotate(parent_conj, measured)-- local_quat[:, j] = _kabsch_quat(rest, targets_in_parent_frame)- # else: leaf joint, no positional evidence for its own twist — identity.-- global_quat[:, j] = (- local_quat[:, j] if j == 0 else _quat_multiply(global_quat[:, parents[j]], local_quat[:, j])- )-- return local_quat---def _quat_to_axis_angle(q: np.ndarray) -> np.ndarray:- """(...,4) xyzw quaternion → (...,3) axis-angle."""- q = q / np.where(np.linalg.norm(q, axis=-1, keepdims=True) < 1e-8, 1.0, np.linalg.norm(q, axis=-1, keepdims=True))- w = np.clip(q[..., 3], -1.0, 1.0)- angle = 2.0 * np.arccos(w)- sin_half = np.sqrt(np.maximum(1.0 - w * w, 0.0))- safe_sin = np.where(sin_half < 1e-8, 1.0, sin_half)- axis = q[..., :3] / safe_sin[..., None]- aa = axis * angle[..., None]- return np.where(sin_half[..., None] < 1e-8, 0.0, aa)---def _resample(poses: np.ndarray, trans: np.ndarray, n_frames: int):- """Linear-resample motion to exactly `n_frames` frames."""- src = poses.shape[0]- if src == n_frames:- return poses, trans- src_t = np.linspace(0.0, 1.0, src)- dst_t = np.linspace(0.0, 1.0, n_frames)- out_poses = np.empty((n_frames,) + poses.shape[1:], dtype=poses.dtype)- flat = poses.reshape(src, -1)- out = np.empty((n_frames, flat.shape[1]), dtype=poses.dtype)- for c in range(flat.shape[1]):- out[:, c] = np.interp(dst_t, src_t, flat[:, c])- out_poses = out.reshape((n_frames,) + poses.shape[1:])- out_trans = np.empty((n_frames, 3), dtype=trans.dtype)- for c in range(3):- out_trans[:, c] = np.interp(dst_t, src_t, trans[:, c])- return out_poses, out_transdiff --git a/workers/model-text2motion/momask_sampler.py b/workers/model-text2motion/momask_sampler.pynew file mode 100644index 000000000..6aecea287--- /dev/null+++ b/workers/model-text2motion/momask_sampler.py@@ -0,0 +1,251 @@+"""MoMask sampler adapter.++Thin wrapper around EricGuo5513/momask-codes (MIT) that turns a text prompt+into SMPL-skeleton motion: per-frame local joint rotations (axis-angle, 22+joints) plus a root translation, the exact same contract `mdm_sampler.+MdmSampler.sample()` returns, so `main.py` can dispatch to either backend+through one `_generate_motion()` call. Everything torch/momask-specific is+imported lazily inside `_load()` so this module imports cleanly in+environments without the GPU stack (the worker's pure conversion path ++tests). Loaded once per container by main.py's `_load_model()`.++The momask-codes repo is cloned into the image (see Dockerfile) and put on+PYTHONPATH; checkpoints are mounted from the weights bucket at `model_dir`.++Pipeline (matches upstream's own `gen_t2m.py` exactly): a masked transformer+("M-Transformer") autoregressively predicts base-layer RVQ motion tokens from+the text + a target token length, a residual transformer predicts the+remaining five quantizer layers' tokens conditioned on the first, and the RVQ+decoder turns the full token stack back into HumanML3D's 263-dim motion+feature vector. That feature vector is the same representation MDM's+diffusion sampler produces, so the position→rotation decode below reuses+`motion_decode`'s shared, pure-NumPy IK: see mdm_sampler.py's+`_positions_to_local_quats` docstring for why a joint's local rotation has to+be recovered from its children's measured directions rather than its own+position.++Namespace note: momask-codes and motion-diffusion-model both ship a top-level+`utils` package with unrelated contents. If both repos' directories are on+PYTHONPATH in the same long-lived container (one warm instance serving both+backends over its lifetime), whichever repo imported `utils` first would+poison the other's `from utils.X import Y`. `_load()` guards against this by+purging any cached `utils*` modules and pinning momask's repo to the front of+`sys.path` before importing, the one-time cost of resolving the right+`utils` package, paid again by whichever backend loads second.+"""++from __future__ import annotations++import logging++import numpy as np++from motion_decode import _parents_from_kinematic_chain, _positions_to_local_quats, _quat_to_axis_angle, _resample++log = logging.getLogger("text2motion.momask")++# HumanML3D motion is sampled at 20 fps, same as MDM; we resample to the+# requested fps in the worker via the clip times.+HUMANML_FPS = 20+MAX_MOTION_LEN = 196+UNIT_LENGTH = 4 # RVQ token stride: one token covers 4 motion frames.++# The released HumanML3D checkpoint set (momask-codes' own+# prepare/download_models.sh names these exact directories), fixed by what+# the upstream authors published, not a per-request choice.+VQ_NAME = "rvq_nq6_dc512_nc512_noshare_qdp0.2"+T2M_NAME = "t2m_nlayer8_nhead6_ld384_ff1024_cdp0.1_rvq6ns"+RES_NAME = "tres_nlayer8_ld384_ff1024_rvq6ns_cdp0.2_sw"++# Generation hyperparameters: upstream's own EvalT2MOptions CLI defaults+# (options/eval_option.py), not tuned here.+TIME_STEPS = 18+COND_SCALE = 4+TEMPERATURE = 1.0+TOPK_FILTER_THRES = 0.9+RES_COND_SCALE = 5+++class MomaskSampler:+ def __init__(self, model_dir: str, device: str = "cuda"):+ self.model_dir = model_dir+ self.device = device+ self._vq_model = None+ self._t2m_transformer = None+ self._res_model = None+ self._mean = None+ self._std = None+ self._recover_from_ric = None+ self._parents = None+ self._rest_dirs = None+ self._load()++ def _load(self) -> None:+ # Imports are GPU-only (and momask-repo-only), kept here so the+ # module imports without torch or the cloned repo present.+ import os+ import sys++ import torch++ momask_repo = "/opt/momask"+ for name in list(sys.modules):+ if name == "utils" or name.startswith("utils."):+ del sys.modules[name]+ if momask_repo in sys.path:+ sys.path.remove(momask_repo)+ sys.path.insert(0, momask_repo)++ from models.mask_transformer.transformer import MaskTransformer, ResidualTransformer+ from models.vq.model import RVQVAE+ from utils.get_opt import get_opt+ from utils.motion_process import recover_from_ric+ from utils.paramUtil import t2m_kinematic_chain, t2m_raw_offsets++ device = torch.device(self.device)++ vq_opt_path = os.path.join(self.model_dir, "t2m", VQ_NAME, "opt.txt")+ vq_opt = get_opt(vq_opt_path, device=device)+ vq_opt.dim_pose = 263++ vq_model = RVQVAE(+ vq_opt,+ vq_opt.dim_pose,+ vq_opt.nb_code,+ vq_opt.code_dim,+ vq_opt.output_emb_width,+ vq_opt.down_t,+ vq_opt.stride_t,+ vq_opt.width,+ vq_opt.depth,+ vq_opt.dilation_growth_rate,+ vq_opt.vq_act,+ vq_opt.vq_norm,+ )+ vq_ckpt = torch.load(+ os.path.join(vq_opt.checkpoints_dir, vq_opt.dataset_name, vq_opt.name, "model", "net_best_fid.tar"),+ map_location="cpu",+ )+ vq_model.load_state_dict(vq_ckpt["vq_model" if "vq_model" in vq_ckpt else "net"])+ vq_model.to(device)+ vq_model.eval()++ model_opt_path = os.path.join(self.model_dir, "t2m", T2M_NAME, "opt.txt")+ model_opt = get_opt(model_opt_path, device=device)+ model_opt.num_tokens = vq_opt.nb_code+ model_opt.num_quantizers = vq_opt.num_quantizers+ model_opt.code_dim = vq_opt.code_dim++ t2m_transformer = MaskTransformer(+ code_dim=model_opt.code_dim,+ cond_mode="text",+ latent_dim=model_opt.latent_dim,+ ff_size=model_opt.ff_size,+ num_layers=model_opt.n_layers,+ num_heads=model_opt.n_heads,+ dropout=model_opt.dropout,+ clip_dim=512,+ cond_drop_prob=model_opt.cond_drop_prob,+ clip_version="ViT-B/32",+ opt=model_opt,+ )+ t2m_ckpt = torch.load(+ os.path.join(model_opt.checkpoints_dir, model_opt.dataset_name, model_opt.name, "model", "latest.tar"),+ map_location="cpu",+ )+ model_key = "t2m_transformer" if "t2m_transformer" in t2m_ckpt else "trans"+ missing, unexpected = t2m_transformer.load_state_dict(t2m_ckpt[model_key], strict=False)+ assert not unexpected, f"unexpected t2m_transformer keys: {unexpected}"+ assert all(k.startswith("clip_model.") for k in missing), f"unexpected missing keys: {missing}"+ t2m_transformer.to(device)+ t2m_transformer.eval()++ res_opt_path = os.path.join(self.model_dir, "t2m", RES_NAME, "opt.txt")+ res_opt = get_opt(res_opt_path, device=device)+ res_opt.num_quantizers = vq_opt.num_quantizers+ res_opt.num_tokens = vq_opt.nb_code++ res_model = ResidualTransformer(+ code_dim=vq_opt.code_dim,+ cond_mode="text",+ latent_dim=res_opt.latent_dim,+ ff_size=res_opt.ff_size,+ num_layers=res_opt.n_layers,+ num_heads=res_opt.n_heads,+ dropout=res_opt.dropout,+ clip_dim=512,+ shared_codebook=vq_opt.shared_codebook,+ cond_drop_prob=res_opt.cond_drop_prob,+ share_weight=res_opt.share_weight,+ clip_version="ViT-B/32",+ opt=res_opt,+ )+ res_ckpt = torch.load(+ os.path.join(res_opt.checkpoints_dir, res_opt.dataset_name, res_opt.name, "model", "net_best_fid.tar"),+ map_location="cpu",+ )+ missing, unexpected = res_model.load_state_dict(res_ckpt["res_transformer"], strict=False)+ assert not unexpected, f"unexpected res_transformer keys: {unexpected}"+ assert all(k.startswith("clip_model.") for k in missing), f"unexpected missing keys: {missing}"+ res_model.to(device)+ res_model.eval()++ mean = np.load(os.path.join(self.model_dir, "t2m", VQ_NAME, "meta", "mean.npy"))+ std = np.load(os.path.join(self.model_dir, "t2m", VQ_NAME, "meta", "std.npy"))++ self._vq_model = vq_model+ self._t2m_transformer = t2m_transformer+ self._res_model = res_model+ self._mean = mean+ self._std = std+ self._device = device+ self._recover_from_ric = recover_from_ric+ self._parents = _parents_from_kinematic_chain(t2m_kinematic_chain, 22)+ self._rest_dirs = np.asarray(t2m_raw_offsets, dtype=np.float64)+ log.info("MoMask checkpoints loaded from %s", self.model_dir)++ def sample(self, prompt: str, n_frames: int):+ """Text → (poses (T,22,3) axis-angle, trans (T,3)).++ Generates HumanML3D motion tokens for `prompt` at a target length,+ decodes them to the HumanML3D feature vector, recovers joint+ positions via upstream's `recover_from_ric`, then this worker's own+ position→rotation IK (shared with MDM), and resamples to `n_frames`.+ """+ import torch++ if self._vq_model is None:+ raise RuntimeError("MoMask model not loaded")++ token_len = max(1, min(MAX_MOTION_LEN // UNIT_LENGTH, int(round(n_frames / UNIT_LENGTH))))+ token_lens = torch.LongTensor([token_len]).to(self._device)+ captions = [prompt]++ with torch.no_grad():+ mids = self._t2m_transformer.generate(+ captions,+ token_lens,+ timesteps=TIME_STEPS,+ cond_scale=COND_SCALE,+ temperature=TEMPERATURE,+ topk_filter_thres=TOPK_FILTER_THRES,+ gsample=False,+ )+ mids = self._res_model.generate(mids, captions, token_lens, temperature=1, cond_scale=RES_COND_SCALE)+ pred_motions = self._vq_model.forward_decoder(mids)++ data = pred_motions.detach().cpu().numpy()+ data = data * self._std + self._mean # inv_transform, same as upstream gen_t2m.py+ m_length = int(token_len * UNIT_LENGTH)+ feats = data[0][:m_length]++ positions = self._recover_from_ric(torch.from_numpy(feats).float(), 22)+ positions = positions.cpu().numpy().astype(np.float64) # (T, 22, 3)++ local_quat = _positions_to_local_quats(positions, self._parents, self._rest_dirs)+ poses = _quat_to_axis_angle(local_quat) # (T, 22, 3)+ trans = positions[:, 0, :]+ poses = np.asarray(poses, dtype=np.float64)+ trans = np.asarray(trans, dtype=np.float64)+ poses, trans = _resample(poses, trans, n_frames)+ return poses, transdiff --git a/workers/model-text2motion/motion_decode.py b/workers/model-text2motion/motion_decode.pynew file mode 100644index 000000000..5e7912216--- /dev/null+++ b/workers/model-text2motion/motion_decode.py@@ -0,0 +1,171 @@+"""Model-agnostic motion decode math: joint positions to local rotations.++Both text-to-motion backends this worker supports (MDM, MoMask) are trained on+the HumanML3D dataset and therefore emit the same thing: a 22-joint skeleton's+global joint POSITIONS per frame, recovered via each project's own upstream+`recover_from_ric`. Neither project emits local joint ROTATIONS directly, which+is what a three.js AnimationClip needs, so both samplers decode through this+shared module. Everything here is pure NumPy: no torch, no GPU, no checkpoint,+so it is testable without either upstream repo checked out (see+test_mdm_decode.py, which exercises this module's functions through its+re-export from mdm_sampler.py, the existing import path its tests already use).+"""++from __future__ import annotations++import numpy as np+++def _parents_from_kinematic_chain(kinematic_chain, n_joints: int) -> np.ndarray:+ """Flatten per-limb chains (each starting at the root, joint 0) into a+ single parent-index array, e.g. [-1, 2, 5, 8, ...] for a 22-joint+ skeleton. Every joint in the topology appears in exactly one chain."""+ parents = np.full(n_joints, -1, dtype=np.int64)+ for chain in kinematic_chain:+ for i in range(1, len(chain)):+ parents[chain[i]] = chain[i - 1]+ return parents+++def _quat_conjugate(q: np.ndarray) -> np.ndarray:+ return np.concatenate([-q[..., :3], q[..., 3:4]], axis=-1)+++def _quat_multiply(a: np.ndarray, b: np.ndarray) -> np.ndarray:+ ax, ay, az, aw = a[..., 0], a[..., 1], a[..., 2], a[..., 3]+ bx, by, bz, bw = b[..., 0], b[..., 1], b[..., 2], b[..., 3]+ return np.stack(+ [+ aw * bx + ax * bw + ay * bz - az * by,+ aw * by - ax * bz + ay * bw + az * bx,+ aw * bz + ax * by - ay * bx + az * bw,+ aw * bw - ax * bx - ay * by - az * bz,+ ],+ axis=-1,+ )+++def _quat_rotate(q: np.ndarray, v: np.ndarray) -> np.ndarray:+ qv = np.concatenate([v, np.zeros_like(v[..., :1])], axis=-1)+ return _quat_multiply(_quat_multiply(q, qv), _quat_conjugate(q))[..., :3]+++def _rotation_matrix_to_quat(r: np.ndarray) -> np.ndarray:+ """(...,3,3) proper rotation matrices → (...,4) xyzw quaternions."""+ m00, m01, m02 = r[..., 0, 0], r[..., 0, 1], r[..., 0, 2]+ m10, m11, m12 = r[..., 1, 0], r[..., 1, 1], r[..., 1, 2]+ m20, m21, m22 = r[..., 2, 0], r[..., 2, 1], r[..., 2, 2]+ trace = m00 + m11 + m22+ w = 0.5 * np.sqrt(np.maximum(1.0 + trace, 1e-12))+ safe_w = np.where(w < 1e-6, 1.0, w)+ x = (m21 - m12) / (4.0 * safe_w)+ y = (m02 - m20) / (4.0 * safe_w)+ z = (m10 - m01) / (4.0 * safe_w)+ quat = np.stack([x, y, z, w], axis=-1)+ norm = np.linalg.norm(quat, axis=-1, keepdims=True)+ return quat / np.where(norm < 1e-8, 1.0, norm)+++def _kabsch_quat(sources: np.ndarray, targets: np.ndarray) -> np.ndarray:+ """Best-fit rotation quaternion mapping unit-vector set `sources` onto+ `targets`, batched over frames. `sources`/`targets`: (T, K, 3). Minimizes+ sum_k |R @ sources[k] - targets[k]|^2 (Kabsch/Procrustes, reflection-+ corrected so R is always a proper rotation). K == 1 degenerates to a+ shortest-arc rotation with an unconstrained twist about that one axis,+ the same ambiguity single-child joints have regardless of method, since+ HumanML3D position data carries no twist signal for any joint."""+ h = np.einsum("tka,tkb->tab", sources, targets) # (T,3,3)+ u, _, vt = np.linalg.svd(h)+ d = np.sign(np.linalg.det(np.einsum("tab,tbc->tac", vt.transpose(0, 2, 1), u.transpose(0, 2, 1))))+ diag = np.tile(np.eye(3), (h.shape[0], 1, 1))+ diag[:, 2, 2] = d+ r = np.einsum("tab,tbc,tcd->tad", vt.transpose(0, 2, 1), diag, u.transpose(0, 2, 1))+ return _rotation_matrix_to_quat(r)+++def _positions_to_local_quats(+ positions: np.ndarray, parents: np.ndarray, rest_dirs: np.ndarray+) -> np.ndarray:+ """(T,J,3) global positions → (T,J,4) local xyzw quaternions per the+ `parents`/`rest_dirs` topology.++ A joint's local rotation `R_j` is NOT observable from its own position:+ under standard skeletal FK (global_j = global_parent(j) * R_j, and the+ *fixed* rest offset rotates with the PARENT's global orientation only:+ pos[j] - pos[parent(j)] = rotate(global_parent(j), rest_dirs[j])) that+ offset doesn't involve R_j at all. R_j only shows up in where j's OWN+ children end up. So this recovers R_j, per joint, from the directions to+ j's children: best-fit (Kabsch) rotation mapping each child's rest+ direction (in j's parent-relative frame) onto its measured direction,+ across however many children j has (root and the chest joint branch into+ 3; most joints have exactly 1, where Kabsch degenerates to a shortest-arc+ solve). Leaf joints (hands, feet, head) have no children and no positional+ evidence for their own twist, so they get an identity local rotation.+ """+ n_frames, n_joints, _ = positions.shape+ local_quat = np.tile(np.array([0.0, 0.0, 0.0, 1.0]), (n_frames, n_joints, 1))+ global_quat = np.tile(np.array([0.0, 0.0, 0.0, 1.0]), (n_frames, n_joints, 1))++ children_of = [[] for _ in range(n_joints)]+ for j in range(1, n_joints):+ children_of[parents[j]].append(j)++ # Single top-down pass: `parents[j] < j` for this topology (verified: the+ # kinematic chains are built root-outward), so global_quat[parents[j]] is+ # always already resolved by the time joint j is processed.+ for j in range(n_joints):+ kids = children_of[j]+ if kids:+ rest = rest_dirs[np.array(kids)] # (K,3)+ rest = rest / np.linalg.norm(rest, axis=-1, keepdims=True)+ rest = np.broadcast_to(rest, (n_frames,) + rest.shape).copy() # (T,K,3)++ measured = positions[:, kids, :] - positions[:, j : j + 1, :] # (T,K,3)+ measured = measured / np.linalg.norm(measured, axis=-1, keepdims=True)++ if j == 0:+ targets_in_parent_frame = measured # root: parent frame == world+ else:+ p = parents[j]+ k = measured.shape[1]+ parent_conj = np.broadcast_to(_quat_conjugate(global_quat[:, p])[:, None, :], (n_frames, k, 4))+ targets_in_parent_frame = _quat_rotate(parent_conj, measured)++ local_quat[:, j] = _kabsch_quat(rest, targets_in_parent_frame)+ # else: leaf joint, no positional evidence for its own twist, so identity.++ global_quat[:, j] = (+ local_quat[:, j] if j == 0 else _quat_multiply(global_quat[:, parents[j]], local_quat[:, j])+ )++ return local_quat+++def _quat_to_axis_angle(q: np.ndarray) -> np.ndarray:+ """(...,4) xyzw quaternion → (...,3) axis-angle."""+ q = q / np.where(np.linalg.norm(q, axis=-1, keepdims=True) < 1e-8, 1.0, np.linalg.norm(q, axis=-1, keepdims=True))+ w = np.clip(q[..., 3], -1.0, 1.0)+ angle = 2.0 * np.arccos(w)+ sin_half = np.sqrt(np.maximum(1.0 - w * w, 0.0))+ safe_sin = np.where(sin_half < 1e-8, 1.0, sin_half)+ axis = q[..., :3] / safe_sin[..., None]+ aa = axis * angle[..., None]+ return np.where(sin_half[..., None] < 1e-8, 0.0, aa)+++def _resample(poses: np.ndarray, trans: np.ndarray, n_frames: int):+ """Linear-resample motion to exactly `n_frames` frames."""+ src = poses.shape[0]+ if src == n_frames:+ return poses, trans+ src_t = np.linspace(0.0, 1.0, src)+ dst_t = np.linspace(0.0, 1.0, n_frames)+ flat = poses.reshape(src, -1)+ out = np.empty((n_frames, flat.shape[1]), dtype=poses.dtype)+ for c in range(flat.shape[1]):+ out[:, c] = np.interp(dst_t, src_t, flat[:, c])+ out_poses = out.reshape((n_frames,) + poses.shape[1:])+ out_trans = np.empty((n_frames, 3), dtype=trans.dtype)+ for c in range(3):+ out_trans[:, c] = np.interp(dst_t, src_t, trans[:, c])+ return out_poses, out_transdiff --git a/workers/model-text2motion/test_model_selector.py b/workers/model-text2motion/test_model_selector.pynew file mode 100644index 000000000..dda87a6e9--- /dev/null+++ b/workers/model-text2motion/test_model_selector.py@@ -0,0 +1,183 @@+"""Tests for main.py's model selector: resolution, task wiring, and routing.++No torch, no GPU, no checkpoint: `_load_model` is monkeypatched to install a+fake sampler object rather than importing mdm_sampler/momask_sampler, so this+exercises exactly what this worker owns: request validation, which backend a+job actually runs against, and the per-backend GCS path. Not either model's+real inference (covered by test_mdm_decode.py's pure-NumPy decode tests; real+on-GPU sampling is covered by server/index.mjs-level integration in the+platform's own `scripts/gcp/seed-motion.mjs` runs against the deployed lane).++Run (from this directory):+ python3 -m unittest discover -p 'test_*.py'+"""++from __future__ import annotations++import asyncio+import os+import unittest++os.environ.setdefault("API_KEY", "test-key")+os.environ.setdefault("GCS_BUCKET", "test-bucket")++import main # noqa: E402 (env vars must be set first)+from fastapi.testclient import TestClient # noqa: E402++AUTH = {"authorization": "Bearer test-key"}+++class FakeSampler:+ """Stands in for MdmSampler/MomaskSampler: same `sample()` contract,+ zero GPU/torch/checkpoint dependency."""++ def __init__(self, name: str):+ self.name = name++ def sample(self, prompt: str, n_frames: int):+ import numpy as np++ poses = np.zeros((n_frames, 22, 3), dtype=np.float64)+ trans = np.zeros((n_frames, 3), dtype=np.float64)+ return poses, trans+++class RecordingBlob:+ """Stands in for a storage.Blob so /infer's upload step can be exercised+ without a GCS project (same pattern as model-hunyuan3d's test suite)."""++ def __init__(self, store: dict, name: str):+ self._store = store+ self._name = name++ def upload_from_string(self, data, content_type=None):+ self._store[self._name] = data+++class RecordingBucket:+ def __init__(self):+ self.objects: dict[str, str] = {}++ def blob(self, name: str) -> RecordingBlob:+ return RecordingBlob(self.objects, name)+++class CanonicalModelTest(unittest.TestCase):+ def test_defaults_when_absent(self):+ self.assertEqual(main.canonical_model(None), "mdm")+ self.assertEqual(main.canonical_model(""), "mdm")++ def test_recognizes_momask(self):+ self.assertEqual(main.canonical_model("momask"), "momask")++ def test_case_insensitive(self):+ self.assertEqual(main.canonical_model("MoMask"), "momask")+ self.assertEqual(main.canonical_model(" MDM "), "mdm")++ def test_unknown_name_falls_back_to_default(self):+ # A stale/misspelled model name still has an obviously right thing to+ # do, same rationale as workers/rembg's canonical_model.+ self.assertEqual(main.canonical_model("t2m-gpt"), "mdm")+++class ModelSelectorWiringTest(unittest.TestCase):+ """Exercises /infer, /tasks/:id, /health, / with fake samplers and a+ recording GCS bucket installed in place of the real GPU + storage+ backends. Built without the TestClient `with` block (that would run the+ real `lifespan`, which opens a real `storage.Client()` bucket handle and+ would overwrite the recording one set up below), same convention as+ model-hunyuan3d's test_worker_api.py."""++ def setUp(self):+ main._tasks.clear()+ main._models.clear()+ main._load_errors.clear()+ main._sem = asyncio.Semaphore(main.MAX_CONCURRENT)+ main._bucket = RecordingBucket()+ self._real_load_model = main._load_model++ def fake_load_model(name: str) -> None:+ main._models[name] = FakeSampler(name)++ main._load_model = fake_load_model+ self.addCleanup(lambda: setattr(main, "_load_model", self._real_load_model))+ self.client = TestClient(main.app)++ def test_infer_defaults_to_mdm_and_resolves_to_done(self):+ resp = self.client.post(+ "/infer",+ json={"prompt": "a person waves hello", "duration_seconds": 1, "fps": 10},+ headers=AUTH,+ )+ self.assertEqual(resp.status_code, 202)+ task_id = resp.json()["task_id"]++ task = self.client.get(f"/tasks/{task_id}", headers=AUTH).json()+ self.assertEqual(task["status"], "done")+ self.assertEqual(task["model"], "mdm")+ self.assertIn("motion-clips/mdm/", task["result_url"])+ self.assertEqual(task["frames"], 10)++ def test_infer_with_momask_routes_to_the_momask_backend(self):+ resp = self.client.post(+ "/infer",+ json={"prompt": "a slow tai-chi sweep", "duration_seconds": 1, "fps": 10, "model": "momask"},+ headers=AUTH,+ )+ self.assertEqual(resp.status_code, 202)+ task_id = resp.json()["task_id"]++ task = self.client.get(f"/tasks/{task_id}", headers=AUTH).json()+ self.assertEqual(task["status"], "done")+ self.assertEqual(task["model"], "momask")+ self.assertIn("motion-clips/momask/", task["result_url"])++ def test_unknown_model_name_falls_back_and_still_completes(self):+ resp = self.client.post(+ "/infer",+ json={"prompt": "a person jogs forward", "duration_seconds": 1, "fps": 10, "model": "nonexistent"},+ headers=AUTH,+ )+ task_id = resp.json()["task_id"]+ task = self.client.get(f"/tasks/{task_id}", headers=AUTH).json()+ self.assertEqual(task["model"], "mdm")+ self.assertEqual(task["status"], "done")++ def test_health_reports_default_loaded_and_both_backends_available(self):+ main._models["mdm"] = FakeSampler("mdm") # normally warmed by the startup background task+ health = self.client.get("/health").json()+ self.assertTrue(health["ok"])+ self.assertTrue(health["model_loaded"])+ self.assertIn("mdm", health["models_loaded"])+ self.assertEqual(set(health["models_available"]), {"mdm", "momask"})+ self.assertEqual(health["default_model"], "mdm")++ def test_health_reflects_momask_once_it_has_been_requested(self):+ self.client.post(+ "/infer",+ json={"prompt": "a person kicks a ball", "duration_seconds": 1, "fps": 10, "model": "momask"},+ headers=AUTH,+ )+ health = self.client.get("/health").json()+ self.assertIn("momask", health["models_loaded"])++ def test_root_advertises_both_backends(self):+ root = self.client.get("/").json()+ self.assertEqual(root["model"], "mdm")+ self.assertEqual(set(root["models_available"]), {"mdm", "momask"})++ def test_missing_api_key_is_a_validation_error(self):+ resp = self.client.post("/infer", json={"prompt": "a person waves hello"})+ self.assertEqual(resp.status_code, 422)++ def test_wrong_api_key_is_rejected(self):+ resp = self.client.post(+ "/infer",+ json={"prompt": "a person waves hello"},+ headers={"authorization": "Bearer nope"},+ )+ self.assertEqual(resp.status_code, 401)+++if __name__ == "__main__":+ unittest.main()