Skip to content

Retarget human motion from video to an Adobe Mixamo-rigged 3D character.

Python

Init project

MaxLiuc64e29dcecba
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..b126c58
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,32 @@
+# ── User-supplied assets: never committed ───────────────────
+# Ignore every file under assets/; re-include directories (so git
+# can descend) plus the docs and .gitkeep placeholders. This keeps
+# SMPLX_NEUTRAL.npz / *.fbx / videos out of git no matter what.
+assets/**
+!assets/**/
+!assets/README.md
+!assets/**/.gitkeep
+
+# ── Auto-downloaded inference weights (GVHMR / 2D pose, etc.) ─
+weights/**
+!weights/.gitkeep
+
+# ── Run outputs ──────────────────────────────────────────────
+outputs/**
+!outputs/.gitkeep
+
+# ── Python ───────────────────────────────────────────────────
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+*.egg-info/
+build/
+dist/
+
+# ── Local secrets / environment variables ────────────────────
+.env
+.env.*
+
+# ── OS junk ──────────────────────────────────────────────────
+.DS_Store
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..21d5d59
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,55 @@
+# AGENTS.md
+
+Motion2MixamoRig drives an Adobe Mixamo-rigged 3D character with human motion
+extracted from a monocular video. CLI: `m2mr`.
+
+## Setup
+
+```bash
+# chumpy (legacy dep of gvhmr) breaks under PEP 517 build isolation
+# ("No module named 'pip'"); install it without isolation first.
+pip install --upgrade pip setuptools wheel \
+ && pip install "numpy>=1.26" \
+ && pip install --no-build-isolation "chumpy==0.70" \
+ && pip install -e .
+```
+
+Three things must be supplied by the user (all git-ignored, licensing-gated):
+
+| What | Where to put it | Source |
+| --- | --- | --- |
+| SMPL-X body model | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | https://smpl-x.is.tue.mpg.de/ (registration) |
+| Mixamo character | `assets/mixamo/<Name>.fbx` (FBX Binary, T-pose) | https://www.mixamo.com/ (Adobe account) |
+| Input video | `assets/video/<name>.mp4` | one clearly visible person (group shots fail preflight) |
+
+Inference weights (~5 GB) auto-download into `weights/` on the first run.
+
+## Commands
+
+```bash
+m2mr doctor # verify assets + dependencies; prints what is missing and where to get it
+m2mr run # latest video in assets/video/ + assets/mixamo/Y_Bot.fbx
+m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx
+m2mr run --skeleton outputs/<run>/skeleton_motion.npz --rig ... # reuse extraction, swap rig
+```
+
+`--video` and `--rig` combine freely. Extraction is the slow step; reuse it
+with `--skeleton` when only the rig changes.
+
+## Outputs
+
+Each run writes `outputs/<YYYYMMDD_HHMMSS>_<video>/`:
+
+- `run.json` — inputs, timings, output listing
+- `skeleton_motion.npz` — 3D human skeleton (24 SMPL joints, Y-up meters)
+- `mixamo_rotations.npz` — per-bone animation rotations (quaternion wxyz; convention string inside)
+- `mixamo_character.glb` — skinned character + clip (glTF 2.0; Blender: File → Import → glTF 2.0)
+- `videos/human_skeleton.mp4`, `videos/mixamo_skeleton.mp4`, `videos/mixamo_character.mp4`, `videos/compare.mp4` (2x2: original / mixamo skeleton / human skeleton / character)
+
+## Code map
+
+- `motion2mixamorig/paths.py` — all filesystem locations
+- `motion2mixamorig/extract.py` — video -> 3D human skeleton (GVHMR)
+- `motion2mixamorig/mixamo/` — FBX parsing, FK, retargeting, skinning, rendering
+- `motion2mixamorig/pipeline.py` — one run end to end
+- `motion2mixamorig/cli.py` — `m2mr doctor` / `m2mr run`
diff --git a/NOTICE.md b/NOTICE.md
new file mode 100644
index 0000000..eb479e8
--- /dev/null
+++ b/NOTICE.md
@@ -0,0 +1,41 @@
+# NOTICE
+
+This project's own code is released under the license in `LICENSE`. It builds
+on the following upstream components, each governed by its own terms. None of
+their gated assets are redistributed here; users obtain them directly from the
+sources below.
+
+## GVHMR
+
+World-Grounded Human Motion Recovery via Gravity-View Coordinates
+(Shen et al., SIGGRAPH Asia 2024). Installed as the `gvhmr` Python package.
+GVHMR is released for **non-commercial research purposes only** — see its
+license before any commercial use.
+https://github.com/zju3dv/GVHMR
+
+## SMPL-X body model
+
+The SMPL-X body model is registration-gated and licensed by the Max Planck
+Institute for Intelligent Systems for non-commercial scientific research.
+Users must register and download `SMPLX_NEUTRAL.npz` themselves.
+https://smpl-x.is.tue.mpg.de/
+
+## Adobe Mixamo characters
+
+Mixamo characters (Y Bot and others) are provided by Adobe under the Mixamo
+terms of use. Download them with your own Adobe account; the FBX files are
+never committed to this repository.
+https://www.mixamo.com/
+
+## Inference weights
+
+The extraction step auto-downloads model checkpoints (GVHMR, HMR2, RTMPose /
+ViTPose 2D pose, YOLOv8 person detector) into `weights/`. Note that YOLOv8
+(Ultralytics) is licensed under **AGPL-3.0**, and the pose/HMR checkpoints
+carry their respective research licenses.
+
+## Summary
+
+The end-to-end pipeline is effectively restricted to non-commercial /
+research use by its upstream components, regardless of this repository's own
+code license.
diff --git a/README.md b/README.md
index 96c83ca..aaca8cb 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,122 @@
# Motion2MixamoRig
-Retarget human motion from video to an Adobe Mixamo-rigged 3D character.
+
+[中文](README.zh-CN.md) · English
+
+Transfer the motion of a person in a video onto an Adobe Mixamo-rigged 3D character.
+
+For game developers: give it a **single-person** action video and a Mixamo character; it writes preview videos plus a skinned character file (`.glb`) you can drop into Blender or Unity.
+
+Agents should start with [`AGENTS.md`](AGENTS.md).
+
+## Prerequisites
+
+Python 3.10+. After cloning, create a virtualenv and install dependencies; the `m2mr` command is then available:
+
+```bash
+git clone https://github.com/MaxLiu22/Motion2MixamoRig.git
+cd Motion2MixamoRig
+python -m venv .venv && source .venv/bin/activate
+
+# Install dependencies and this project. chumpy (a legacy package on the
+# dependency chain) imports pip in its setup script, which fails with
+# "No module named 'pip'" under pip's default isolated build. Install it
+# first with isolation turned off, then install this project.
+pip install --upgrade pip setuptools wheel \
+ && pip install "numpy>=1.26" \
+ && pip install --no-build-isolation "chumpy==0.70" \
+ && pip install -e .
+```
+
+Inference weights (GVHMR and so on) are not downloaded here. The first `m2mr run` pulls them into `weights/` (~5 GB, once only).
+
+ffmpeg is recommended (macOS: `brew install ffmpeg`, Ubuntu: `apt install ffmpeg`).
+The pipeline runs without it, but the output videos will be **silent** and will not play in a browser.
+
+## Before you run: put three things in `assets/`
+
+This repository does not ship these files; you have to fetch them. The SMPL-X filename must match the table exactly; FBX and video names can be anything.
+
+| What | Where | Where to get it |
+|---|---|---|
+| SMPL-X body model | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | Register at [SMPL-X](https://smpl-x.is.tue.mpg.de/) and download it |
+| Mixamo character FBX | `assets/mixamo/Y_Bot.fbx` | With an Adobe account, download Y Bot (or any other Mixamo-rigged character) from [Mixamo](https://www.mixamo.com) |
+| Action video | `assets/video/<your_clip>.mp4` | A clip with **exactly one** clearly visible person (you may keep several files; each run uses one) |
+
+Note: Mixamo's download is named `Y Bot.fbx` (**space**), one character off from `Y_Bot.fbx` (**underscore**) in the table.
+Rename it to the underscore form and it becomes the default rig (`m2mr run` without `--rig`). Leaving the space is fine too — if `Y_Bot.fbx` is missing, the first `.fbx` in `assets/mixamo/` is used.
+
+`assets/video/` can hold several videos. Without `--video`, the run uses the file **most recently added** to that folder.
+Each clip must show only one person: `m2mr doctor` / `m2mr run` sample the frames and stop before extraction if two people are in view.
+
+## Quick Start
+
+Results land in `outputs/<run-time>_<video>/`. Open that folder's `videos/` to watch them.
+
+### 1. Check assets and the environment
+
+Anything missing is printed with where to download it and where to put it.
+
+```bash
+m2mr doctor
+```
+
+### 2. First run: get something on screen
+
+With no flags, this uses the **latest file in `assets/video/`** and `assets/mixamo/Y_Bot.fbx`.
+
+```bash
+m2mr run
+```
+
+This is the slow step. The first run downloads ~5 GB of inference weights (once only), then extracts human motion from the video.
+On CPU, a ~30 second clip takes about 8–15 minutes to extract; later runs of similar length, with weights already there, take about 3–5 minutes.
+Retargeting and rendering are usually a minute or two. Reusing a skeleton with `--skeleton` in the next section skips extraction entirely and finishes in tens of seconds.
+
+### 3. Same motion, different character
+
+Download another character from Mixamo into `assets/mixamo/`. **Do not rerun with only `--rig`** — that repeats the slow extraction.
+Pass `--skeleton` to reuse the skeleton from a previous run; minutes become tens of seconds:
+
+```bash
+m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/mixamo/Vampire.fbx
+```
+
+### 4. A different video
+
+`--video` picks the clip. It combines freely with `--rig`.
+
+```bash
+m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx
+```
+
+## Outputs
+
+Each `m2mr run` creates a directory named by **the time the command started**:
+
+```
+outputs/20260829_193205_dance/
+├── run.json # start time, video / rig used, full command
+├── skeleton_motion.npz # 3D human skeleton (reuse when swapping rigs; no re-extract)
+├── mixamo_rotations.npz # per-bone Mixamo rotations
+├── mixamo_character.glb # skinned character + animation, import into Blender / Unity
+└── videos/ # same aspect ratio as the input
+ ├── human_skeleton.mp4 # human skeleton
+ ├── mixamo_skeleton.mp4 # Mixamo rig skeleton
+ ├── mixamo_character.mp4 # Mixamo rig character
+ └── compare.mp4 # 2×2: original (TL) / Mixamo skeleton (TR) / human skeleton (BL) / character (BR)
+```
+
+Open `mixamo_character.glb` in Blender: File → Import → glTF 2.0 (.glb/.gltf). If the camera is framed on empty space, select the character and use View → Frame Selected. The cluster of spheres on the head and hands is the bone display, not the mesh: select the armature and turn off Shapes under Armature → Viewport Display. Set the timeline end frame to the clip length, then play. Compare against `videos/mixamo_character.mp4` from the same run.
+
+## License
+
+This repository's code is Apache-2.0. The assets you download have their own terms and must not be redistributed with this repo:
+
+- **SMPL-X**: MPI-gated; default non-commercial research / education / art; commercial use needs a separate deal
+- **Mixamo FBX**: fine in a shipped product; do not redistribute the raw FBX as an asset pack
+- **GVHMR and other inference weights**: auto-downloaded into `weights/` on the first `m2mr run`, under each upstream license
+
+## Contact
+
+If you run into a problem, need help finishing a local setup, or would like to talk about working together, email:
diff --git a/README.zh-CN.md b/README.zh-CN.md
new file mode 100644
index 0000000..3490159
--- /dev/null
+++ b/README.zh-CN.md
@@ -0,0 +1,122 @@
+# Motion2MixamoRig
+
+中文 · [English](README.md)
+
+把视频里人物的动作,转到 Adobe Mixamo rig 的 3D 角色上。
+
+面向游戏开发者:输入一段**单人**动作视频和一个 Mixamo 角色,输出预览视频,以及可直接丢进 Blender / Unity 的带皮角色文件(`.glb`)。
+
+Agent 请先读 [`AGENTS.md`](AGENTS.md)。
+
+## 前置条件
+
+需要 Python 3.10+。克隆后建虚拟环境并安装依赖,`m2mr` 命令随之可用:
+
+```bash
+git clone https://github.com/MaxLiu22/Motion2MixamoRig.git
+cd Motion2MixamoRig
+python -m venv .venv && source .venv/bin/activate
+
+# 安装依赖和本项目。chumpy(依赖链里的老包)的构建脚本会 import pip,
+# 在 pip 默认的隔离构建环境里必然报 "No module named 'pip'",
+# 所以先关掉隔离(--no-build-isolation)单独装它,再装本项目。
+pip install --upgrade pip setuptools wheel \
+ && pip install "numpy>=1.26" \
+ && pip install --no-build-isolation "chumpy==0.70" \
+ && pip install -e .
+```
+
+推理权重(GVHMR 等)不在这一步下载,首次 `m2mr run` 时自动拉到 `weights/`(约 5 GB,只下这一次)。
+
+建议再装上 ffmpeg(macOS:`brew install ffmpeg`,Ubuntu:`apt install ffmpeg`)。
+没有它也能跑,但输出的视频会**没有声音**,编码格式也没法在浏览器里直接播放。
+
+## 使用前:把三样东西放到 `assets/`
+
+本仓库不附带这些文件,需要你自己准备。SMPL-X 的文件名必须和下表一字不差;FBX 和视频叫什么都行。
+
+| 放什么 | 放到哪 | 去哪拿 |
+|---|---|---|
+| SMPL-X 人体模型 | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | 注册 [SMPL-X](https://smpl-x.is.tue.mpg.de/) 后下载 |
+| Mixamo 角色 FBX | `assets/mixamo/Y_Bot.fbx` | 用 Adobe 账号从 [Mixamo](https://www.mixamo.com) 下载 Y Bot(或其他 Mixamo rig 角色) |
+| 动作视频 | `assets/video/<your_clip>.mp4` | **只有一个人**、人清晰可见的动作视频(可放多个文件,每次跑用其中一个) |
+
+注意:Mixamo 下载下来的文件叫 `Y Bot.fbx`(中间是**空格**),和上表的 `Y_Bot.fbx`(**下划线**)差一个字符。
+改成下划线它就是默认 rig,`m2mr run` 不带 `--rig` 时自动用它;不想改名也行——找不到 `Y_Bot.fbx` 时
+会自动用 `assets/mixamo/` 里的第一个 `.fbx`。
+
+`assets/video/` 可以放多个视频文件。不指定 `--video` 时,会用**最后放进这个文件夹**的那一个。
+每个视频画面里只能有一个人:`m2mr doctor` / `m2mr run` 会抽样检测,看到两个人就直接停,不会进入提取。
+
+## Quick Start
+
+结果都在 `outputs/<运行时间>_<视频名>/`,看视频打开该目录下的 `videos/`。
+
+### 1. 检查资产和环境
+
+缺什么会告诉你去哪下、放到哪。
+
+```bash
+m2mr doctor
+```
+
+### 2. 第一次跑:先拿到一个能看的结果
+
+不带参数时,用**最后放进 `assets/video/` 的视频** + `assets/mixamo/Y_Bot.fbx`。
+
+```bash
+m2mr run
+```
+
+这步最慢。第一次跑会先下载约 5 GB 推理权重(只这一次),然后从视频里提取人体动作。
+一段约 30 秒的视频,CPU 上提取大约 8–15 分钟;权重下完之后再跑同类长度,大约 3–5 分钟。
+后面的重定向和渲染通常一两分钟。用下一节的 `--skeleton` 复用骨架时,提取整段跳过,几十秒就能出结果。
+
+### 3. 同一段动作,换个角色
+
+从 Mixamo 下载别的角色丢进 `assets/mixamo/`。**别直接只改 `--rig` 重跑**——那样最慢的
+动作提取会原样再来一遍。用 `--skeleton` 复用上一次运行提取好的骨架,几分钟的事变成几十秒:
+
+```bash
+m2mr run --skeleton outputs/<上次的运行目录>/skeleton_motion.npz --rig assets/mixamo/Vampire.fbx
+```
+
+### 4. 换视频
+
+`--video` 指定用哪一段,和 `--rig` 可任意组合。
+
+```bash
+m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx
+```
+
+## 输出
+
+每次 `m2mr run` 按**命令启动时间**新建一个目录:
+
+```
+outputs/20260829_193205_dance/
+├── run.json # 启动时间、用的视频 / rig、完整命令
+├── skeleton_motion.npz # 人体 3D 骨架(换 rig 可复用,不必重跑提取)
+├── mixamo_rotations.npz # Mixamo 骨逐帧旋转
+├── mixamo_character.glb # 带皮角色 + 动画,Blender / Unity 可直接导入
+└── videos/ # 比例跟输入视频一致
+ ├── human_skeleton.mp4 # 人体骨架
+ ├── mixamo_skeleton.mp4 # Mixamo rig 骨架
+ ├── mixamo_character.mp4 # Mixamo rig 角色
+ └── compare.mp4 # 四宫格:左上原片 / 右上 Mixamo 骨架 / 左下人体骨架 / 右下 Mixamo 角色
+```
+
+`mixamo_character.glb` 用 Blender 打开:File → Import → glTF 2.0 (.glb/.gltf)。导入后若镜头对不准,先选中角色,再 View → Frame Selected。头和手上那堆球是骨头的显示形状,不是模型:选中骨架,在 Armature → Viewport Display 里去掉 Shapes。时间轴结束帧改成和视频一样长,再点播放;对照同一次运行的 `videos/mixamo_character.mp4`。
+
+## 许可证
+
+本仓库代码为 Apache-2.0。你自取的资产各有条款,不能随本仓库再分发:
+
+- **SMPL-X**:MPI 门禁,默认非商业科研 / 教育 / 艺术;商用需另谈
+- **Mixamo FBX**:可用在成品里,不能把原始 FBX 当资源包发出去
+- **GVHMR 等推理权重**:首次 `m2mr run` 自动下载到 `weights/`,遵循各自上游许可
+
+## 联系
+
+若使用中遇到问题、需要协助完成本地部署,或希望洽谈合作,欢迎通过邮箱联系我:
diff --git a/assets/README.md b/assets/README.md
new file mode 100644
index 0000000..11d83d1
--- /dev/null
+++ b/assets/README.md
@@ -0,0 +1,18 @@
+# assets
+
+Everything in this folder is supplied by you and is never committed to git
+(see the repository `.gitignore`). Licensing is why these cannot ship with the
+project: SMPL-X and Mixamo characters must be obtained under your own account.
+
+Run `m2mr doctor` at any time to check what is present and what is missing.
+
+## What goes where
+
+| Path | What | Where to get it |
+| --- | --- | --- |
+| `body_models/smplx/SMPLX_NEUTRAL.npz` | SMPL-X neutral body model, needed to lift the video into a 3D skeleton | Register at <https://smpl-x.is.tue.mpg.de/>, download the model archive, copy `SMPLX_NEUTRAL.npz` here |
+| `mixamo/<Character>.fbx` | The character to drive, e.g. `Y_Bot.fbx` (the default) | <https://www.mixamo.com/> — pick a character, download as **FBX Binary**, **T-pose** |
+| `video/<your_clip>.mp4` | The video whose human motion you want to transfer | One clearly visible person only (`.mp4`, `.mov`, …). Group shots are rejected before extraction |
+
+`m2mr run` without flags uses the most recently added file in `video/` and
+`mixamo/Y_Bot.fbx`; use `--video` / `--rig` to pick explicitly.
diff --git a/assets/body_models/smplx/.gitkeep b/assets/body_models/smplx/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/assets/mixamo/.gitkeep b/assets/mixamo/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/assets/video/.gitkeep b/assets/video/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/motion2mixamorig/__init__.py b/motion2mixamorig/__init__.py
new file mode 100644
index 0000000..c9f970f
--- /dev/null
+++ b/motion2mixamorig/__init__.py
@@ -0,0 +1 @@
+"""Motion2MixamoRig: retarget human motion from video to a Mixamo-rigged 3D character."""
diff --git a/motion2mixamorig/checks.py b/motion2mixamorig/checks.py
new file mode 100644
index 0000000..09a6c81
--- /dev/null
+++ b/motion2mixamorig/checks.py
@@ -0,0 +1,174 @@
+"""Content checks for user-supplied files, each returning a fix-it hint.
+
+The CLI already checks that files exist. These look *inside* them, because the
+common failure mode is not a missing file but a wrong one — an ASCII FBX, an
+SMPL model renamed to SMPL-X, a clip OpenCV cannot decode — which would
+otherwise surface minutes into a run as a deep traceback.
+
+Each check returns None when the file is usable, else one line saying what is
+wrong and what to do about it.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+# Vertex counts identify the body-model family regardless of the file name.
+SMPLX_VERTS = 10475
+SMPL_VERTS = 6890
+
+FBX_BINARY_MAGIC = b"Kaydara FBX Binary"
+
+# Sampled frames for the person-count check. The pipeline maps one body to one
+# rig, so two people in frame is a hard stop — we do not wait for extraction.
+_PERSON_SAMPLES = 16
+_PERSON_CONF = 0.5
+# Ignore postage-stamp extras (audience, posters). A second dancer is larger.
+_PERSON_MIN_AREA = 0.015
+
+_yolo = None
+
+
+def _person_detector():
+ """Ultralytics YOLO, reused across checks. Same family GVHMR uses to find people."""
+ global _yolo
+ if _yolo is not None:
+ return _yolo
+ from ultralytics import YOLO
+
+ from .paths import WEIGHTS, export_gvhmr_env
+
+ export_gvhmr_env()
+ ckpt = WEIGHTS / "yolo" / "yolov8x.pt"
+ _yolo = YOLO(str(ckpt) if ckpt.exists() else "yolov8n.pt")
+ return _yolo
+
+
+def _count_people(frame) -> int:
+ h, w = frame.shape[:2]
+ min_area = _PERSON_MIN_AREA * h * w
+ result = _person_detector().predict(
+ frame, conf=_PERSON_CONF, classes=[0], verbose=False
+ )[0]
+ n = 0
+ if result.boxes is None:
+ return 0
+ for box in result.boxes:
+ x1, y1, x2, y2 = (float(v) for v in box.xyxy[0])
+ if (x2 - x1) * (y2 - y1) >= min_area:
+ n += 1
+ return n
+
+
+def check_video(path: Path) -> str | None:
+ """Decodable, and a single person on the sampled frames — not a group."""
+ import cv2
+
+ cap = cv2.VideoCapture(str(path))
+ if not cap.isOpened():
+ return (
+ "cannot be decoded as video — convert it to a plain mp4 first, e.g. "
+ "ffmpeg -i input -c:v libx264 output.mp4"
+ )
+ n_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+ if n_frames <= 0:
+ ok, _ = cap.read()
+ cap.release()
+ if not ok:
+ return (
+ "cannot be decoded as video — convert it to a plain mp4 first, e.g. "
+ "ffmpeg -i input -c:v libx264 output.mp4"
+ )
+ return "has no readable frame count — re-export the clip as a plain mp4"
+ k = min(_PERSON_SAMPLES, n_frames)
+ sample = {int(round(i * (n_frames - 1) / max(k - 1, 1))) for i in range(k)}
+ counts: list[int] = []
+ idx = 0
+ while idx <= max(sample):
+ ok, frame = cap.read()
+ if not ok:
+ break
+ if idx in sample:
+ counts.append(_count_people(frame))
+ idx += 1
+ cap.release()
+ if not counts:
+ return "could not read frames to count people — re-export the clip as a plain mp4"
+ crowded = [c for c in counts if c >= 2]
+ if crowded:
+ peak = max(counts)
+ return (
+ f"shows about {peak} people in frame — this pipeline maps one body "
+ "to one Mixamo rig. Use a clip with a single, clearly visible person"
+ )
+ if max(counts) < 1:
+ return (
+ "no person was detected — use a clip with one clearly visible dancer, "
+ "not an empty or heavily cropped shot"
+ )
+ return None
+
+
+def check_smplx(path: Path) -> str | None:
+ """SMPLX_NEUTRAL.npz must be the npz variant of SMPL-X (not SMPL, not .pkl)."""
+ import numpy as np
+
+ try:
+ with np.load(path, allow_pickle=True) as data:
+ v_template = data["v_template"]
+ except KeyError:
+ return "is an .npz but not a body model (no v_template) — re-download SMPLX_NEUTRAL.npz"
+ except Exception:
+ return (
+ "is not a valid .npz — you may have the pickle (.pkl) variant; "
+ "download the *npz* SMPL-X model from https://smpl-x.is.tue.mpg.de/"
+ )
+ n = int(v_template.shape[0])
+ if n == SMPL_VERTS:
+ return (
+ "is the older SMPL model, not SMPL-X — download SMPLX_NEUTRAL.npz "
+ "from https://smpl-x.is.tue.mpg.de/"
+ )
+ if n != SMPLX_VERTS:
+ return f"has {n} template vertices, expected {SMPLX_VERTS} (SMPL-X) — re-download SMPLX_NEUTRAL.npz"
+ return None
+
+
+def check_rig(path: Path) -> str | None:
+ """The rig must be a binary FBX containing a Mixamo skeleton."""
+ from .mixamo.fbx_skeleton import extract_skeleton
+
+ with open(path, "rb") as f:
+ head = f.read(len(FBX_BINARY_MAGIC))
+ if head != FBX_BINARY_MAGIC:
+ return (
+ "is not a *binary* FBX — on mixamo.com pick Format: 'FBX Binary(.fbx)' "
+ "when downloading the character"
+ )
+ try:
+ bones = {str(b["name"]) for b in extract_skeleton(path)["bones"]}
+ except Exception:
+ return "could not be parsed as an FBX scene — re-download the character from mixamo.com"
+ if "mixamorig:Hips" not in bones:
+ return (
+ "has no Mixamo skeleton (mixamorig:* bones) — download a *character* "
+ "from mixamo.com, T-pose, FBX Binary"
+ )
+ return None
+
+
+def check_skeleton_npz(path: Path) -> str | None:
+ """--skeleton must point at a skeleton_motion.npz from a previous run."""
+ import numpy as np
+
+ try:
+ with np.load(path, allow_pickle=True) as data:
+ missing = {"joints_3d", "joint_names", "fps"} - set(data.keys())
+ except Exception:
+ return "is not a valid .npz file"
+ if missing:
+ return (
+ f"is missing {sorted(missing)} — pass the skeleton_motion.npz "
+ "found inside a previous run's output directory"
+ )
+ return None
diff --git a/motion2mixamorig/cli.py b/motion2mixamorig/cli.py
new file mode 100644
index 0000000..5efe0f7
--- /dev/null
+++ b/motion2mixamorig/cli.py
@@ -0,0 +1,240 @@
+"""The `m2mr` command-line entry point: doctor / run.
+
+ m2mr doctor check assets, weights and dependencies
+ m2mr run latest video in assets/video/ + default Y Bot
+ m2mr run --video V --rig R pick the inputs explicitly (flags combine freely)
+
+`m2mr run` performs the same asset checks as `m2mr doctor` before doing any
+work, and reports everything that is missing in one pass.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from . import checks, paths
+
+OK = "[ok] "
+MISS = "[missing]"
+BAD = "[invalid]"
+
+SMPLX_HINT = (
+ "register at https://smpl-x.is.tue.mpg.de/ , download SMPLX_NEUTRAL.npz, "
+ "place it at assets/body_models/smplx/SMPLX_NEUTRAL.npz"
+)
+RIG_HINT = (
+ "download a character from https://www.mixamo.com/ (FBX Binary, T-pose) "
+ "into assets/mixamo/ , e.g. Y_Bot.fbx"
+)
+VIDEO_HINT = (
+ "put a *single-person* motion video (.mp4/.mov/...) into assets/video/ "
+ "— group shots are rejected before extraction"
+)
+
+
+def _check(label: str, present: bool, hint: str) -> bool:
+ print(f"{OK if present else MISS} {label}")
+ if not present:
+ print(f" -> {hint}")
+ return present
+
+
+def _check_content(label: str, path: Path, checker) -> bool:
+ """Existence is assumed; `checker` inspects the file and returns a hint or None."""
+ problem = checker(path)
+ print(f"{OK if problem is None else BAD} {label}")
+ if problem is not None:
+ print(f" -> {path.name} {problem}")
+ return problem is None
+
+
+def _module_available(name: str) -> bool:
+ import importlib.util
+
+ return importlib.util.find_spec(name) is not None
+
+
+def _available_rigs() -> list[Path]:
+ return sorted(paths.MIXAMO_DIR.glob("*.fbx")) if paths.MIXAMO_DIR.is_dir() else []
+
+
+def doctor() -> int:
+ """Check every user-supplied asset and dependency; say where to get what's missing."""
+ print(f"project root: {paths.PROJECT_ROOT}\n")
+
+ ok = True
+ smplx_label = f"SMPL-X body model {paths.SMPLX_NEUTRAL.relative_to(paths.PROJECT_ROOT)}"
+ if paths.SMPLX_NEUTRAL.exists():
+ ok &= _check_content(smplx_label, paths.SMPLX_NEUTRAL, checks.check_smplx)
+ else:
+ ok &= _check(smplx_label, False, SMPLX_HINT)
+ rigs = _available_rigs()
+ ok &= _check(
+ f"Mixamo rig .fbx assets/mixamo/ ({len(rigs)} found"
+ + (f": {', '.join(r.name for r in rigs[:5])})" if rigs else ")"),
+ bool(rigs),
+ RIG_HINT,
+ )
+ for rig in rigs:
+ ok &= _check_content(f" - {rig.name}", rig, checks.check_rig)
+ videos = paths.list_videos()
+ ok &= _check(
+ f"input video assets/video/ ({len(videos)} found"
+ + (f", latest: {videos[0].name})" if videos else ")"),
+ bool(videos),
+ VIDEO_HINT,
+ )
+ for video in videos:
+ ok &= _check_content(f" - {video.name}", video, checks.check_video)
+
+ print()
+ deps_ok = True
+ for module, hint in (
+ ("numpy", "pip install -e ."),
+ ("cv2", "pip install -e ."),
+ ("torch", "pip install -e ."),
+ ("gvhmr", "pip install -e ."),
+ ):
+ deps_ok &= _check(f"python module {module}", _module_available(module), hint)
+ import shutil as _shutil
+
+ _check(
+ "ffmpeg (optional) H.264 output with the source video's audio",
+ _shutil.which("ffmpeg") is not None,
+ "brew install ffmpeg / apt install ffmpeg — runs fine without, but videos are silent mp4v",
+ )
+
+ weights_note = (
+ "downloaded automatically into weights/ on the first `m2mr run` (~5 GB)"
+ )
+ print(f"\ninference weights: {weights_note}")
+
+ if ok and deps_ok:
+ print("\neverything in place — try: m2mr run")
+ return 0
+ print("\nfix the [missing] items above, then re-run: m2mr doctor")
+ return 1
+
+
+def preflight(args: argparse.Namespace) -> tuple[Path, Path, Path | None]:
+ """Resolve run inputs, checking everything doctor checks for this run.
+
+ All problems are collected and reported together (same format as doctor)
+ instead of failing on the first one. Exits with code 1 if anything is
+ missing.
+ """
+ problems = 0
+
+ def check(label: str, present: bool, hint: str) -> bool:
+ nonlocal problems
+ if not _check(label, present, hint):
+ problems += 1
+ return present
+
+ def check_content(label: str, path: Path | None, checker) -> None:
+ """Inspect the file's contents, but only when it exists at all."""
+ nonlocal problems
+ if path is not None and path.exists() and not _check_content(label, path, checker):
+ problems += 1
+
+ # Input video: --video wins, else the most recently added file in assets/video/.
+ if args.video:
+ video = Path(args.video)
+ check(f"input video {video}", video.exists(), "check the --video path")
+ else:
+ video = paths.latest_video()
+ check(
+ "input video assets/video/"
+ + (f" (latest: {video.name})" if video else " (0 found)"),
+ video is not None,
+ VIDEO_HINT + ", or pass --video",
+ )
+ check_content(f"video is usable {video.name if video else ''}", video, checks.check_video)
+
+ # Rig: --rig wins, else Y_Bot.fbx, else the first .fbx in assets/mixamo/.
+ if args.rig:
+ rig = Path(args.rig)
+ check(f"Mixamo rig .fbx {rig}", rig.exists(), "check the --rig path")
+ else:
+ rig = paths.DEFAULT_RIG
+ if not rig.exists():
+ rigs = _available_rigs()
+ rig = rigs[0] if rigs else None
+ check(
+ "Mixamo rig .fbx " + (str(rig) if rig else "assets/mixamo/ (0 found)"),
+ rig is not None,
+ RIG_HINT + ", or pass --rig",
+ )
+ check_content(f"rig is Mixamo {rig.name if rig else ''}", rig, checks.check_rig)
+
+ # Skeleton reuse skips extraction; only then is SMPL-X not needed.
+ skeleton = Path(args.skeleton) if args.skeleton else None
+ if skeleton is not None:
+ check(
+ f"skeleton npz {skeleton}",
+ skeleton.exists(),
+ "check the --skeleton path (a previous run's skeleton_motion.npz)",
+ )
+ check_content(f"skeleton loads {skeleton.name}", skeleton, checks.check_skeleton_npz)
+ else:
+ check(
+ f"SMPL-X body model {paths.SMPLX_NEUTRAL.relative_to(paths.PROJECT_ROOT)}",
+ paths.SMPLX_NEUTRAL.exists(),
+ SMPLX_HINT,
+ )
+ check_content(
+ "SMPL-X loads SMPLX_NEUTRAL.npz", paths.SMPLX_NEUTRAL, checks.check_smplx
+ )
+
+ if problems:
+ print(f"\n{problems} problem(s) — nothing was run. `m2mr doctor` checks everything.")
+ sys.exit(1)
+ return video, rig, skeleton
+
+
+def run(args: argparse.Namespace) -> int:
+ video, rig, skeleton = preflight(args)
+ print()
+
+ from .pipeline import run_pipeline
+
+ run_dir = run_pipeline(video, rig, device=args.device, skeleton_npz=skeleton)
+ print(f"\nresults: {run_dir / 'videos'}")
+ return 0
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ prog="m2mr",
+ description="Drive a Mixamo-rigged character with human motion from a video.",
+ )
+ sub = parser.add_subparsers(dest="command", required=True)
+
+ sub.add_parser("doctor", help="check assets, weights and dependencies")
+
+ p_run = sub.add_parser("run", help="video -> Mixamo rig, results in outputs/")
+ p_run.add_argument(
+ "--video",
+ help="input video (default: the file most recently added to assets/video/)",
+ )
+ p_run.add_argument(
+ "--rig",
+ help="Mixamo character .fbx (default: assets/mixamo/Y_Bot.fbx)",
+ )
+ p_run.add_argument(
+ "--skeleton",
+ help="reuse a previous run's skeleton_motion.npz and skip video extraction",
+ )
+ p_run.add_argument(
+ "--device",
+ default="cpu",
+ choices=["cpu", "cuda", "mps"],
+ help="inference device for the extraction step (default: cpu)",
+ )
+
+ args = parser.parse_args()
+ if args.command == "doctor":
+ sys.exit(doctor())
+ sys.exit(run(args))
diff --git a/motion2mixamorig/extract.py b/motion2mixamorig/extract.py
new file mode 100644
index 0000000..9513706
--- /dev/null
+++ b/motion2mixamorig/extract.py
@@ -0,0 +1,173 @@
+"""Video -> 3D human skeleton (GVHMR world-grounded motion recovery).
+
+Produces skeleton_motion.npz with 24 SMPL joints per frame in a Y-up,
+meter-scaled world, plus the root translation/orientation and fps.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+
+from .paths import export_gvhmr_env
+
+# SMPL 24-joint order used by GVHMR (J_regressor).
+SMPL_JOINT_NAMES = [
+ "pelvis",
+ "left_hip",
+ "right_hip",
+ "spine1",
+ "left_knee",
+ "right_knee",
+ "spine2",
+ "left_ankle",
+ "right_ankle",
+ "spine3",
+ "left_foot",
+ "right_foot",
+ "neck",
+ "left_collar",
+ "right_collar",
+ "head",
+ "left_shoulder",
+ "right_shoulder",
+ "left_elbow",
+ "right_elbow",
+ "left_wrist",
+ "right_wrist",
+ "left_hand",
+ "right_hand",
+]
+
+
+def _to_numpy(x):
+ if hasattr(x, "detach"):
+ return x.detach().cpu().numpy()
+ return np.asarray(x)
+
+
+def joints_world_from_smplx(params, device: str = "cpu") -> np.ndarray:
+ """Convert GVHMR SMPL-X params to 24 SMPL joints without the gated SMPL .pkl.
+
+ Official ``result.joints_world`` also loads SMPL faces from
+ ``SMPL_NEUTRAL.pkl``. The joint math itself only needs SMPL-X plus the
+ shipped ``smplx2smpl`` / ``J_regressor`` maps.
+ """
+ import torch
+ from einops import einsum
+ from gvhmr import PROJ_ROOT
+ from gvhmr.utils.device import to_device
+ from gvhmr.utils.smplx_utils import make_smplx
+
+ if device == "mps":
+ device = "cpu"
+ torch_device = torch.device(device)
+ smplx = make_smplx("supermotion").to(torch_device)
+ smplx2smpl = torch.load(
+ PROJ_ROOT / "gvhmr/utils/body_model/smplx2smpl_sparse.pt",
+ weights_only=False,
+ ).to(torch_device)
+ j_regressor = torch.load(
+ PROJ_ROOT / "gvhmr/utils/body_model/smpl_neutral_J_regressor.pt",
+ weights_only=False,
+ ).to(torch_device)
+ with torch.no_grad():
+ out = smplx(**to_device(dict(params), torch_device))
+ verts = torch.stack([torch.matmul(smplx2smpl, v) for v in out.vertices])
+ joints = einsum(j_regressor, verts, "j v, l v i -> l j i")
+ return joints.cpu().numpy().astype(np.float32)
+
+
+def _stabilize_pose_cache(pose_cache: Path) -> bool:
+ """Stabilize GVHMR's 2D-pose cache in place. True if the file changed.
+
+ Idempotent: a cache that was already stabilized comes back unchanged. The
+ pre-stabilization data is kept next to it (*_raw) so estimator-vs-
+ stabilizer questions stay answerable after the fact.
+ """
+ import shutil
+
+ import torch
+
+ from .stabilize_kp2d import stabilize_coco17
+
+ current = torch.load(pose_cache, weights_only=False)
+ fixed, swaps = stabilize_coco17(current)
+ if np.allclose(fixed, _to_numpy(current).astype(np.float32)):
+ return False
+ raw_backup = pose_cache.with_name(pose_cache.stem + "_raw" + pose_cache.suffix)
+ if not raw_backup.exists():
+ shutil.copy2(pose_cache, raw_backup)
+ torch.save(torch.from_numpy(fixed), pose_cache)
+ print(f"stabilized {pose_cache.name}: corrected {int(swaps.sum())}/{len(swaps)} left/right flips")
+ return True
+
+
+def extract_motion(
+ video: Path,
+ output_npz: Path,
+ *,
+ work_dir: Path,
+ device: str = "cpu",
+ static_camera: bool = True,
+ verbose: bool = True,
+) -> Path:
+ """Run GVHMR on `video` and save skeleton_motion.npz to `output_npz`.
+
+ `work_dir` holds GVHMR's per-video intermediates (2D pose cache, 3D lift).
+ Re-running with the same work_dir reuses the expensive passes.
+ """
+ export_gvhmr_env()
+ import gvhmr
+
+ video = Path(video).resolve()
+ if not video.exists():
+ raise FileNotFoundError(video)
+
+ output_dir = Path(work_dir) / video.stem
+
+ def recover():
+ return gvhmr.recover(
+ str(video),
+ static_camera=static_camera,
+ device=device,
+ pose2d="rtmpose",
+ output_dir=str(output_dir),
+ output_root=str(work_dir),
+ progress=verbose,
+ )
+
+ result = recover()
+
+ # RTMPose labels left/right per frame and can swap them when the subject
+ # turns away, which spins the recovered 3D heading. Stabilize the 2D cache
+ # and redo the 3D lift once if anything was corrected.
+ pose_cache = output_dir / "preprocess" / "vitpose.pt"
+ if pose_cache.exists() and _stabilize_pose_cache(pose_cache):
+ motion_cache = output_dir / "hmr4d_results.pt"
+ motion_cache.unlink(missing_ok=True)
+ result = recover()
+
+ world = result.smpl_params_world
+ joints_3d = joints_world_from_smplx(world, device=device)
+ root_translation = _to_numpy(world["transl"]).astype(np.float32)
+ root_orientation = _to_numpy(world["global_orient"]).astype(np.float32)
+ timestamps = np.arange(len(joints_3d), dtype=np.float64) / float(result.fps)
+
+ output_npz = Path(output_npz)
+ output_npz.parent.mkdir(parents=True, exist_ok=True)
+ np.savez_compressed(
+ output_npz,
+ joints_3d=joints_3d,
+ joint_names=np.array(SMPL_JOINT_NAMES),
+ fps=np.float32(result.fps),
+ timestamps=timestamps,
+ root_translation=root_translation,
+ root_orientation=root_orientation,
+ camera=np.array(result.camera),
+ video=np.array(str(video)),
+ )
+ if verbose:
+ print(f"saved {output_npz}")
+ return output_npz
diff --git a/motion2mixamorig/human_motion.py b/motion2mixamorig/human_motion.py
new file mode 100644
index 0000000..2e3a389
--- /dev/null
+++ b/motion2mixamorig/human_motion.py
@@ -0,0 +1,74 @@
+"""Load the skeleton_motion.npz produced by extract.py.
+
+Human world (GVHMR gravity frame):
+ +Y up, +X right, +Z forward (right-handed), units = meters.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+
+@dataclass
+class HumanMotion:
+ joints_3d: np.ndarray # (F, J, 3) meters, Y-up world
+ joint_names: list[str]
+ fps: float
+ timestamps: np.ndarray
+ root_translation: np.ndarray | None
+ path: Path
+
+ @property
+ def n_frames(self) -> int:
+ return int(self.joints_3d.shape[0])
+
+ def index(self, name: str) -> int:
+ try:
+ return self.joint_names.index(name)
+ except ValueError as exc:
+ raise KeyError(f"joint {name!r} not in {self.joint_names}") from exc
+
+ def joint(self, frame: int, name: str) -> np.ndarray:
+ return np.asarray(self.joints_3d[frame, self.index(name)], dtype=np.float64)
+
+
+def load_human_motion(path: Path) -> HumanMotion:
+ path = Path(path)
+ if not path.exists():
+ raise FileNotFoundError(f"Missing {path}. Run the extraction step first.")
+ data = np.load(path, allow_pickle=True)
+ joints = np.asarray(data["joints_3d"], dtype=np.float64)
+ if joints.ndim != 3 or joints.shape[-1] != 3:
+ raise ValueError(f"joints_3d must be (F, J, 3), got {joints.shape}")
+ names = [str(x) for x in data["joint_names"]]
+ fps = float(data["fps"]) if "fps" in data.files else 30.0
+ if "timestamps" in data.files:
+ timestamps = np.asarray(data["timestamps"], dtype=np.float64)
+ else:
+ timestamps = np.arange(len(joints), dtype=np.float64) / fps
+ root = (
+ np.asarray(data["root_translation"], dtype=np.float64)
+ if "root_translation" in data.files
+ else None
+ )
+ return HumanMotion(
+ joints_3d=joints,
+ joint_names=names,
+ fps=fps,
+ timestamps=timestamps,
+ root_translation=root,
+ path=path,
+ )
+
+
+def source_video_path(motion: HumanMotion) -> Path | None:
+ """The input video recorded inside skeleton_motion.npz, if still on disk."""
+ npz = np.load(motion.path, allow_pickle=True)
+ if "video" in npz.files:
+ path = Path(str(npz["video"]))
+ if path.exists():
+ return path
+ return None
diff --git a/motion2mixamorig/mixamo/__init__.py b/motion2mixamorig/mixamo/__init__.py
new file mode 100644
index 0000000..6f5d98a
--- /dev/null
+++ b/motion2mixamorig/mixamo/__init__.py
@@ -0,0 +1,12 @@
+"""Human skeleton -> Mixamo rig retargeting.
+
+Modules:
+ fbx_skeleton binary FBX parser, core-bone skeleton extraction
+ kinematics FK with FBX PreRotation, rotation utilities
+ retarget per-frame Human -> Mixamo Skeleton swing/aim retarget
+ tpose_calibration measured T-pose alignment skeleton <-> character rig
+ ybot_retarget rest-relative rotation transfer onto the full rig
+ skinned_mesh FBX meshes, skin clusters, linear blend skinning
+ animation whole-clip retargeting driver
+ render software cameras, skeleton/mesh drawing, video writing
+"""
diff --git a/motion2mixamorig/mixamo/animation.py b/motion2mixamorig/mixamo/animation.py
new file mode 100644
index 0000000..6ac158b
--- /dev/null
+++ b/motion2mixamorig/mixamo/animation.py
@@ -0,0 +1,113 @@
+"""Per-frame retarget of a whole clip: Human -> Mixamo Skeleton -> character rig.
+
+ human skeleton_motion.npz
+ | retarget_one_frame (swing/aim, no bone scaling, no IK)
+ Mixamo Skeleton R_anim
+ | rest-relative transfer from the T-pose calibration
+ character rig R_anim -> FK -> linear blend skinning
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+import numpy as np
+
+from .kinematics import forward_kinematics, world_positions_m
+from .retarget import joint_mixamo, retarget_one_frame
+from .skinned_mesh import YBotAsset, bone_palette, skin_mesh
+from .ybot_retarget import YBotMapping
+
+
+@dataclass
+class RetargetedAnimation:
+ frames: list[int]
+ fps: float
+ r_anim_skeleton: list[dict[str, np.ndarray]]
+ r_anim_rig: list[dict[str, np.ndarray]]
+ hips_cm_skeleton: list[np.ndarray]
+ hips_cm_rig: list[np.ndarray]
+ mixamo_pos: list[dict[str, np.ndarray]]
+ rig_pos: list[dict[str, np.ndarray]]
+ human_pos: list[dict[str, np.ndarray]]
+ twist_events: list[dict] = field(default_factory=list)
+ sign_events: list[dict] = field(default_factory=list)
+
+ @property
+ def n(self) -> int:
+ return len(self.frames)
+
+
+def build_animation(
+ motion, mapping: YBotMapping, frames: list[int], *, verbose: bool = True
+) -> RetargetedAnimation:
+ """Retarget Human -> Mixamo, then carry the rest-relative motion onto the rig."""
+ skel = mapping.skeleton_rig
+ anim = RetargetedAnimation(
+ frames=frames,
+ fps=float(motion.fps),
+ r_anim_skeleton=[],
+ r_anim_rig=[],
+ hips_cm_skeleton=[],
+ hips_cm_rig=[],
+ mixamo_pos=[],
+ rig_pos=[],
+ human_pos=[],
+ )
+ r_prev = q_prev = None
+ for i, frame in enumerate(frames):
+ result = retarget_one_frame(skel, motion, frame, prev_r_anim=r_prev, prev_q_anim=q_prev)
+ hips_cm = joint_mixamo(motion, frame, "pelvis") / skel.meters_per_unit
+
+ mix_pos = world_positions_m(
+ skel, forward_kinematics(skel, result.r_anim, hips_translation_cm=hips_cm)
+ )
+ r_rig = mapping.apply_rest_relative_rotations(result.r_anim)
+ hips_rig = mapping.hips_translation_cm(hips_cm)
+ rig_pos = mapping.ybot_positions_m(r_rig, hips_rig)
+
+ anim.r_anim_skeleton.append(result.r_anim)
+ anim.r_anim_rig.append(r_rig)
+ anim.hips_cm_skeleton.append(hips_cm)
+ anim.hips_cm_rig.append(hips_rig)
+ anim.mixamo_pos.append(mix_pos)
+ anim.rig_pos.append(rig_pos)
+ anim.human_pos.append({n: joint_mixamo(motion, frame, n) for n in motion.joint_names})
+ if result.twist_flips:
+ anim.twist_events.append({"frame": frame, "bones": result.twist_flips})
+ if result.sign_flips:
+ anim.sign_events.append({"frame": frame, "bones": result.sign_flips})
+
+ r_prev, q_prev = result.r_anim_raw, result.q_anim
+ if verbose and (i == 0 or (i + 1) % 200 == 0 or i == len(frames) - 1):
+ print(f" retarget {i + 1}/{len(frames)}")
+ return anim
+
+
+def skin_frames(
+ asset: YBotAsset, anim: RetargetedAnimation, indices: list[int]
+) -> list[list[np.ndarray]]:
+ """Linear blend skinning of the character mesh (meters) for the given indices."""
+ out: list[list[np.ndarray]] = []
+ scale = asset.rig.meters_per_unit
+ for i in indices:
+ world = forward_kinematics(
+ asset.rig, anim.r_anim_rig[i], hips_translation_cm=anim.hips_cm_rig[i]
+ )
+ out.append(
+ [
+ skin_mesh(mesh, bone_palette(world, mesh, asset.rig.order)) * scale
+ for mesh in asset.meshes
+ ]
+ )
+ return out
+
+
+def core_bones_only(pos: dict[str, np.ndarray], edges) -> dict[str, np.ndarray]:
+ """Drop the finger chains so the rig panel draws the same graph as the edges."""
+ keep = {name for edge in edges for name in edge}
+ return {name: p for name, p in pos.items() if name in keep}
+
+
+def all_points(seq: list[dict[str, np.ndarray]]) -> np.ndarray:
+ return np.stack([p for pos in seq for p in pos.values()])
diff --git a/motion2mixamorig/mixamo/fbx_skeleton.py b/motion2mixamorig/mixamo/fbx_skeleton.py
new file mode 100644
index 0000000..2c726f9
--- /dev/null
+++ b/motion2mixamorig/mixamo/fbx_skeleton.py
@@ -0,0 +1,628 @@
+"""Parse a binary Mixamo FBX and extract the core skeleton.
+
+Reads the binary FBX directly (PreRotation is a Maya/FBX property, so Assimp
+cannot be used) and returns the core-bone skeleton as a plain dict — the same
+structure `kinematics.rig_from_skeleton_data` consumes.
+"""
+
+from __future__ import annotations
+
+import re
+import struct
+import zlib
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+# Mixamo body chain only: no finger bones. End sites keep bone directions.
+CORE_BONE_SUFFIXES = (
+ "Hips",
+ "Spine",
+ "Spine1",
+ "Spine2",
+ "Neck",
+ "Head",
+ "HeadTop_End",
+ "LeftShoulder",
+ "LeftArm",
+ "LeftForeArm",
+ "LeftHand",
+ "RightShoulder",
+ "RightArm",
+ "RightForeArm",
+ "RightHand",
+ "LeftUpLeg",
+ "LeftLeg",
+ "LeftFoot",
+ "LeftToeBase",
+ "LeftToe_End",
+ "RightUpLeg",
+ "RightLeg",
+ "RightFoot",
+ "RightToeBase",
+ "RightToe_End",
+)
+
+# SMPL-24 (GVHMR skeleton_motion.npz) → Mixamo Y Bot.
+HUMAN_TO_MIXAMO = {
+ "pelvis": "mixamorig:Hips",
+ "left_hip": "mixamorig:LeftUpLeg",
+ "right_hip": "mixamorig:RightUpLeg",
+ "spine1": "mixamorig:Spine",
+ "left_knee": "mixamorig:LeftLeg",
+ "right_knee": "mixamorig:RightLeg",
+ "spine2": "mixamorig:Spine1",
+ "left_ankle": "mixamorig:LeftFoot",
+ "right_ankle": "mixamorig:RightFoot",
+ "spine3": "mixamorig:Spine2",
+ "left_foot": "mixamorig:LeftToeBase",
+ "right_foot": "mixamorig:RightToeBase",
+ "neck": "mixamorig:Neck",
+ "left_collar": "mixamorig:LeftShoulder",
+ "right_collar": "mixamorig:RightShoulder",
+ "head": "mixamorig:Head",
+ "left_shoulder": "mixamorig:LeftArm",
+ "right_shoulder": "mixamorig:RightArm",
+ "left_elbow": "mixamorig:LeftForeArm",
+ "right_elbow": "mixamorig:RightForeArm",
+ "left_wrist": "mixamorig:LeftHand",
+ "right_wrist": "mixamorig:RightHand",
+ "left_hand": "mixamorig:LeftHand",
+ "right_hand": "mixamorig:RightHand",
+}
+
+# Mixamo bones that have no 1:1 SMPL-24 joint (chain tips / extra spine).
+MIXAMO_WITHOUT_HUMAN = (
+ "mixamorig:HeadTop_End",
+ "mixamorig:LeftToe_End",
+ "mixamorig:RightToe_End",
+)
+
+_RIG_PREFIX_RE = re.compile(r"^mixamorig\d+:")
+
+FBX_HEADER = b"Kaydara FBX Binary \x00\x1a\x00"
+ARRAY_DTYPE = {
+ b"f": ("<f", 4),
+ b"d": ("<d", 8),
+ b"l": ("<q", 8),
+ b"i": ("<i", 4),
+ b"b": ("<?", 1),
+}
+
+
+class FbxError(ValueError):
+ pass
+
+
+def _read(buf: memoryview, offset: int, fmt: str) -> tuple[Any, int]:
+ size = struct.calcsize(fmt)
+ return struct.unpack_from(fmt, buf, offset)[0], offset + size
+
+
+def _read_array(buf: memoryview, offset: int, type_code: bytes) -> tuple[list[Any], int]:
+ length, offset = _read(buf, offset, "<I")
+ encoding, offset = _read(buf, offset, "<I")
+ compressed_len, offset = _read(buf, offset, "<I")
+ dtype, item_size = ARRAY_DTYPE[type_code]
+ raw = bytes(buf[offset : offset + compressed_len])
+ offset += compressed_len
+ if encoding == 1:
+ raw = zlib.decompress(raw)
+ elif encoding != 0:
+ raise FbxError(f"unsupported FBX array encoding {encoding}")
+ expected = length * item_size
+ if len(raw) != expected:
+ raise FbxError(f"array size mismatch: {len(raw)} != {expected}")
+ values = list(struct.unpack_from(f"<{length}{dtype[-1]}", raw, 0)) if length else []
+ return values, offset
+
+
+def _read_property(buf: memoryview, offset: int) -> tuple[Any, int]:
+ type_code = bytes(buf[offset : offset + 1])
+ offset += 1
+ if type_code == b"Y":
+ return _read(buf, offset, "<h")
+ if type_code == b"C":
+ value, offset = _read(buf, offset, "<B")
+ return bool(value), offset
+ if type_code == b"I":
+ return _read(buf, offset, "<i")
+ if type_code == b"F":
+ return _read(buf, offset, "<f")
+ if type_code == b"D":
+ return _read(buf, offset, "<d")
+ if type_code == b"L":
+ return _read(buf, offset, "<q")
+ if type_code in ARRAY_DTYPE:
+ return _read_array(buf, offset, type_code)
+ if type_code in (b"S", b"R"):
+ length, offset = _read(buf, offset, "<I")
+ data = bytes(buf[offset : offset + length])
+ offset += length
+ if type_code == b"S":
+ return data.split(b"\x00")[0].decode("utf-8", errors="replace"), offset
+ return data, offset
+ raise FbxError(f"unknown FBX property type {type_code!r}")
+
+
+def _is_null_record(buf: memoryview, offset: int, version: int) -> bool:
+ size = 25 if version >= 7500 else 13
+ return offset + size <= len(buf) and all(b == 0 for b in buf[offset : offset + size])
+
+
+def _read_node(buf: memoryview, offset: int, version: int) -> tuple[dict[str, Any] | None, int]:
+ start = offset
+ if version >= 7500:
+ end_offset, offset = _read(buf, offset, "<Q")
+ n_props, offset = _read(buf, offset, "<Q")
+ _prop_len, offset = _read(buf, offset, "<Q")
+ else:
+ end_offset, offset = _read(buf, offset, "<I")
+ n_props, offset = _read(buf, offset, "<I")
+ _prop_len, offset = _read(buf, offset, "<I")
+ if end_offset == 0:
+ return None, start + (25 if version >= 7500 else 13)
+ name_len = buf[offset]
+ offset += 1
+ name = bytes(buf[offset : offset + name_len]).decode("ascii", errors="replace")
+ offset += name_len
+ props: list[Any] = []
+ for _ in range(n_props):
+ value, offset = _read_property(buf, offset)
+ props.append(value)
+ children: list[dict[str, Any]] = []
+ while offset < end_offset:
+ if _is_null_record(buf, offset, version):
+ offset = end_offset
+ break
+ child, offset = _read_node(buf, offset, version)
+ if child is None:
+ break
+ children.append(child)
+ return {"name": name, "props": props, "children": children}, end_offset
+
+
+def parse_fbx(path: Path) -> tuple[int, list[dict[str, Any]]]:
+ data = path.read_bytes()
+ if not data.startswith(FBX_HEADER):
+ raise FbxError(f"{path} is not a binary FBX file")
+ version = struct.unpack_from("<I", data, 23)[0]
+ buf = memoryview(data)
+ offset = 27
+ nodes: list[dict[str, Any]] = []
+ while offset < len(buf):
+ if _is_null_record(buf, offset, version):
+ break
+ node, offset = _read_node(buf, offset, version)
+ if node is None:
+ break
+ nodes.append(node)
+ return version, nodes
+
+
+def _children_named(node: dict[str, Any], name: str) -> list[dict[str, Any]]:
+ return [c for c in node["children"] if c["name"] == name]
+
+
+def _child_named(node: dict[str, Any], name: str) -> dict[str, Any] | None:
+ for child in node["children"]:
+ if child["name"] == name:
+ return child
+ return None
+
+
+def _vec3(value: Any, default: tuple[float, float, float] = (0.0, 0.0, 0.0)) -> np.ndarray:
+ if value is None:
+ return np.array(default, dtype=np.float64)
+ arr = np.asarray(value, dtype=np.float64).reshape(-1)
+ if arr.size < 3:
+ out = np.array(default, dtype=np.float64)
+ out[: arr.size] = arr
+ return out
+ return arr[:3].copy()
+
+
+def parse_properties70(node: dict[str, Any]) -> dict[str, Any]:
+ props: dict[str, Any] = {}
+ block = _child_named(node, "Properties70")
+ if block is None:
+ return props
+ for child in block["children"]:
+ if child["name"] != "P" or not child["props"]:
+ continue
+ raw = child["props"]
+ key = str(raw[0])
+ values = raw[4:] if len(raw) > 4 else []
+ if len(values) == 1:
+ props[key] = values[0]
+ elif len(values) >= 3 and all(isinstance(v, (int, float)) for v in values[:3]):
+ props[key] = [float(v) for v in values[:3]]
+ else:
+ props[key] = values
+ return props
+
+
+def euler_xyz_deg_to_matrix(euler_deg: np.ndarray) -> np.ndarray:
+ """FBX/Maya Euler XYZ: R = Rz @ Ry @ Rx (column vectors)."""
+ x, y, z = np.deg2rad(np.asarray(euler_deg, dtype=np.float64)[:3])
+ cx, sx = np.cos(x), np.sin(x)
+ cy, sy = np.cos(y), np.sin(y)
+ cz, sz = np.cos(z), np.sin(z)
+ rx = np.array([[1.0, 0.0, 0.0], [0.0, cx, -sx], [0.0, sx, cx]])
+ ry = np.array([[cy, 0.0, sy], [0.0, 1.0, 0.0], [-sy, 0.0, cy]])
+ rz = np.array([[cz, -sz, 0.0], [sz, cz, 0.0], [0.0, 0.0, 1.0]])
+ return rz @ ry @ rx
+
+
+def matrix_to_quat_wxyz(rmat: np.ndarray) -> np.ndarray:
+ m = np.asarray(rmat, dtype=np.float64)
+ t = float(np.trace(m))
+ if t > 0.0:
+ s = 0.5 / np.sqrt(t + 1.0)
+ w = 0.25 / s
+ x = (m[2, 1] - m[1, 2]) * s
+ y = (m[0, 2] - m[2, 0]) * s
+ z = (m[1, 0] - m[0, 1]) * s
+ else:
+ i = int(np.argmax([m[0, 0], m[1, 1], m[2, 2]]))
+ if i == 0:
+ s = 2.0 * np.sqrt(1.0 + m[0, 0] - m[1, 1] - m[2, 2])
+ w = (m[2, 1] - m[1, 2]) / s
+ x = 0.25 * s
+ y = (m[0, 1] + m[1, 0]) / s
+ z = (m[0, 2] + m[2, 0]) / s
+ elif i == 1:
+ s = 2.0 * np.sqrt(1.0 + m[1, 1] - m[0, 0] - m[2, 2])
+ w = (m[0, 2] - m[2, 0]) / s
+ x = (m[0, 1] + m[1, 0]) / s
+ y = 0.25 * s
+ z = (m[1, 2] + m[2, 1]) / s
+ else:
+ s = 2.0 * np.sqrt(1.0 + m[2, 2] - m[0, 0] - m[1, 1])
+ w = (m[1, 0] - m[0, 1]) / s
+ x = (m[0, 2] + m[2, 0]) / s
+ y = (m[1, 2] + m[2, 1]) / s
+ z = 0.25 * s
+ q = np.array([w, x, y, z], dtype=np.float64)
+ n = float(np.linalg.norm(q))
+ if n < 1e-12:
+ return np.array([1.0, 0.0, 0.0, 0.0])
+ if q[0] < 0:
+ q = -q
+ return q / n
+
+
+def make_transform(
+ translation: np.ndarray,
+ pre_rot_deg: np.ndarray,
+ lcl_rot_deg: np.ndarray,
+ post_rot_deg: np.ndarray,
+ scale: np.ndarray,
+ rotation_offset: np.ndarray | None = None,
+ rotation_pivot: np.ndarray | None = None,
+) -> np.ndarray:
+ """Maya/FBX local TRS with PreRotation (pivots/offsets identity on Mixamo).
+
+ M = T * Roff * Rp * Rpre * R * Rpost^{-1} * Rp^{-1} * S
+ """
+ t = _vec3(translation)
+ s = _vec3(scale, (1.0, 1.0, 1.0))
+ roff = _vec3(rotation_offset)
+ rp = _vec3(rotation_pivot)
+ rpre = euler_xyz_deg_to_matrix(pre_rot_deg)
+ r = euler_xyz_deg_to_matrix(lcl_rot_deg)
+ rpost = euler_xyz_deg_to_matrix(post_rot_deg)
+ rot = rpre @ r @ rpost.T
+
+ def translate(vec: np.ndarray) -> np.ndarray:
+ m = np.eye(4)
+ m[:3, 3] = vec
+ return m
+
+ t_mat = translate(t)
+ roff_mat = translate(roff)
+ rp_mat = translate(rp)
+ rp_inv = translate(-rp)
+ r_mat = np.eye(4)
+ r_mat[:3, :3] = rot
+ s_mat = np.diag([float(s[0]), float(s[1]), float(s[2]), 1.0])
+ return t_mat @ roff_mat @ rp_mat @ r_mat @ rp_inv @ s_mat
+
+
+def _tolist(arr: np.ndarray, ndigits: int = 12) -> list[Any]:
+ def _round(x: float) -> float:
+ v = round(float(x), ndigits)
+ return 0.0 if abs(v) < 10 ** (-ndigits) else v
+
+ if arr.ndim == 1:
+ return [_round(x) for x in arr]
+ return [[_round(x) for x in row] for row in arr]
+
+
+def _describe_path(path: Path) -> str:
+ return str(path)
+
+
+def normalize_rig_prefix(name: str) -> str:
+ """`mixamorig8:Hips` → `mixamorig:Hips`.
+
+ Mixamo appends a digit to the namespace when a character is exported from a
+ scene that already held a rig, so the same bone arrives under a different
+ name per file. Every bone table in the project keys off the bare prefix.
+ """
+ return _RIG_PREFIX_RE.sub("mixamorig:", name, count=1)
+
+
+def bone_suffix(name: str) -> str:
+ return name.split(":")[-1]
+
+
+def is_core_bone(name: str) -> bool:
+ return bone_suffix(name) in CORE_BONE_SUFFIXES
+
+
+def extract_objects(nodes: list[dict[str, Any]]) -> dict[int, dict[str, Any]]:
+ objects: dict[int, dict[str, Any]] = {}
+ for node in nodes:
+ if node["name"] != "Objects":
+ continue
+ for child in node["children"]:
+ if not child["props"]:
+ continue
+ uid = int(child["props"][0])
+ name = normalize_rig_prefix(str(child["props"][1])) if len(child["props"]) > 1 else ""
+ class_name = str(child["props"][2]) if len(child["props"]) > 2 else ""
+ objects[uid] = {
+ "uid": uid,
+ "node_name": child["name"],
+ "name": name,
+ "class": class_name,
+ "props": parse_properties70(child),
+ "raw": child,
+ }
+ return objects
+
+
+def extract_connections(nodes: list[dict[str, Any]]) -> list[tuple[str, int, int]]:
+ links: list[tuple[str, int, int]] = []
+ for node in nodes:
+ if node["name"] != "Connections":
+ continue
+ for child in node["children"]:
+ if child["name"] != "C" or len(child["props"]) < 3:
+ continue
+ kind = str(child["props"][0])
+ src = int(child["props"][1])
+ dst = int(child["props"][2])
+ links.append((kind, src, dst))
+ return links
+
+
+def extract_global_settings(nodes: list[dict[str, Any]]) -> dict[str, Any]:
+ for node in nodes:
+ if node["name"] == "GlobalSettings":
+ return parse_properties70(node)
+ return {}
+
+
+def extract_bind_pose(objects: dict[int, dict[str, Any]]) -> dict[int, np.ndarray]:
+ poses: dict[int, np.ndarray] = {}
+ for obj in objects.values():
+ if obj["node_name"] != "Pose":
+ continue
+ for pose_node in _children_named(obj["raw"], "PoseNode"):
+ node_elem = _child_named(pose_node, "Node")
+ matrix_elem = _child_named(pose_node, "Matrix")
+ if node_elem is None or matrix_elem is None or not matrix_elem["props"]:
+ continue
+ uid = int(node_elem["props"][0])
+ values = matrix_elem["props"][0]
+ mat = np.asarray(values, dtype=np.float64).reshape(4, 4).T
+ poses[uid] = mat
+ return poses
+
+
+def collect_limb_nodes(objects: dict[int, dict[str, Any]]) -> dict[int, dict[str, Any]]:
+ limbs: dict[int, dict[str, Any]] = {}
+ for uid, obj in objects.items():
+ if obj["node_name"] != "Model":
+ continue
+ if obj["class"] != "LimbNode":
+ continue
+ limbs[uid] = obj
+ return limbs
+
+
+def build_hierarchy(
+ limbs: dict[int, dict[str, Any]],
+ connections: list[tuple[str, int, int]],
+) -> dict[int, int | None]:
+ parent_of: dict[int, int | None] = {uid: None for uid in limbs}
+ for kind, src, dst in connections:
+ if kind != "OO":
+ continue
+ if src in limbs and dst in limbs:
+ parent_of[src] = dst
+ return parent_of
+
+
+def bone_local_matrix(props: dict[str, Any]) -> tuple[np.ndarray, dict[str, np.ndarray]]:
+ translation = _vec3(props.get("Lcl Translation"))
+ lcl_rot = _vec3(props.get("Lcl Rotation"))
+ pre_rot = _vec3(props.get("PreRotation"))
+ post_rot = _vec3(props.get("PostRotation"))
+ scale = _vec3(props.get("Lcl Scaling"), (1.0, 1.0, 1.0))
+ rotation_offset = _vec3(props.get("RotationOffset"))
+ rotation_pivot = _vec3(props.get("RotationPivot"))
+ local = make_transform(
+ translation,
+ pre_rot,
+ lcl_rot,
+ post_rot,
+ scale,
+ rotation_offset=rotation_offset,
+ rotation_pivot=rotation_pivot,
+ )
+ parts = {
+ "lcl_translation": translation,
+ "lcl_rotation_euler_deg": lcl_rot,
+ "pre_rotation_euler_deg": pre_rot,
+ "post_rotation_euler_deg": post_rot,
+ "lcl_scaling": scale,
+ "rotation_offset": rotation_offset,
+ "rotation_pivot": rotation_pivot,
+ }
+ return local, parts
+
+
+def compose_world(
+ parent_of: dict[int, int | None],
+ local_of: dict[int, np.ndarray],
+) -> dict[int, np.ndarray]:
+ world: dict[int, np.ndarray] = {}
+
+ def eval_uid(uid: int) -> np.ndarray:
+ if uid in world:
+ return world[uid]
+ parent = parent_of.get(uid)
+ local = local_of[uid]
+ if parent is None:
+ world[uid] = local.copy()
+ else:
+ world[uid] = eval_uid(parent) @ local
+ return world[uid]
+
+ for uid in local_of:
+ eval_uid(uid)
+ return world
+
+
+def unit_to_meters(settings: dict[str, Any]) -> float:
+ """FBX default unit is centimeters when UnitScaleFactor == 1."""
+ scale = float(settings.get("UnitScaleFactor", 1.0) or 1.0)
+ return 0.01 * scale
+
+
+def axis_name(index: int, sign: int) -> str:
+ names = {0: "X", 1: "Y", 2: "Z"}
+ prefix = "+" if int(sign) >= 0 else "-"
+ return f"{prefix}{names.get(int(index), '?')}"
+
+
+def extract_skeleton(fbx_path: Path) -> dict[str, Any]:
+ version, nodes = parse_fbx(fbx_path)
+ objects = extract_objects(nodes)
+ connections = extract_connections(nodes)
+ settings = extract_global_settings(nodes)
+ bind_pose = extract_bind_pose(objects)
+ limbs = collect_limb_nodes(objects)
+ parent_of = build_hierarchy(limbs, connections)
+
+ local_of: dict[int, np.ndarray] = {}
+ parts_of: dict[int, dict[str, np.ndarray]] = {}
+ for uid, obj in limbs.items():
+ local, parts = bone_local_matrix(obj["props"])
+ local_of[uid] = local
+ parts_of[uid] = parts
+ world_of = compose_world(parent_of, local_of)
+
+ core_uids = [uid for uid, obj in limbs.items() if is_core_bone(obj["name"])]
+ children_of: dict[int, list[int]] = {uid: [] for uid in core_uids}
+ for uid in core_uids:
+ parent = parent_of[uid]
+ if parent in children_of:
+ children_of[parent].append(uid)
+
+ def sort_children(uids: list[int]) -> list[int]:
+ order = {name: i for i, name in enumerate(CORE_BONE_SUFFIXES)}
+ return sorted(uids, key=lambda u: order.get(bone_suffix(limbs[u]["name"]), 999))
+
+ roots = [uid for uid in core_uids if parent_of[uid] not in set(core_uids)]
+ ordered: list[int] = []
+
+ def walk(uid: int) -> None:
+ ordered.append(uid)
+ for child in sort_children(children_of[uid]):
+ walk(child)
+
+ for root in sort_children(roots):
+ walk(root)
+
+ meters_per_fbx = unit_to_meters(settings)
+ bones = []
+ for uid in ordered:
+ obj = limbs[uid]
+ parts = parts_of[uid]
+ local = local_of[uid]
+ world = world_of[uid]
+ parent_uid = parent_of[uid]
+ parent_name = limbs[parent_uid]["name"] if parent_uid in limbs else None
+ child_names = [limbs[c]["name"] for c in sort_children(children_of[uid])]
+ bind = bind_pose.get(uid)
+ bind_err = None
+ if bind is not None:
+ bind_err = float(np.linalg.norm(world - bind))
+ pre = parts["pre_rotation_euler_deg"]
+ bones.append(
+ {
+ "name": obj["name"],
+ "parent": parent_name,
+ "children": child_names,
+ "pre_rotation_euler_deg": _tolist(pre),
+ "pre_rotation_matrix": _tolist(euler_xyz_deg_to_matrix(pre)),
+ "lcl_translation": _tolist(parts["lcl_translation"]),
+ "lcl_rotation_euler_deg": _tolist(parts["lcl_rotation_euler_deg"]),
+ "lcl_scaling": _tolist(parts["lcl_scaling"]),
+ "post_rotation_euler_deg": _tolist(parts["post_rotation_euler_deg"]),
+ "rest_local": {
+ "matrix": _tolist(local),
+ "translation": _tolist(local[:3, 3]),
+ "rotation_matrix": _tolist(local[:3, :3]),
+ "quaternion_wxyz": _tolist(matrix_to_quat_wxyz(local[:3, :3])),
+ "scale": _tolist(np.array([np.linalg.norm(local[:3, i]) for i in range(3)])),
+ },
+ "rest_world": {
+ "matrix": _tolist(world),
+ "translation": _tolist(world[:3, 3]),
+ "translation_m": _tolist(world[:3, 3] * meters_per_fbx),
+ "rotation_matrix": _tolist(world[:3, :3]),
+ "quaternion_wxyz": _tolist(matrix_to_quat_wxyz(world[:3, :3])),
+ },
+ "bind_pose_world_matrix": _tolist(bind) if bind is not None else None,
+ "bind_pose_match_error": bind_err,
+ }
+ )
+
+ hierarchy = [
+ {
+ "name": limbs[uid]["name"],
+ "parent": limbs[parent_of[uid]]["name"] if parent_of[uid] in limbs else None,
+ "children": [limbs[c]["name"] for c in sort_children(children_of[uid])],
+ }
+ for uid in ordered
+ ]
+
+ return {
+ "source_fbx": _describe_path(fbx_path),
+ "fbx_version": version,
+ "character": fbx_path.stem,
+ "coord_system": {
+ "up": axis_name(int(settings.get("UpAxis", 1)), int(settings.get("UpAxisSign", 1))),
+ "front": axis_name(int(settings.get("FrontAxis", 2)), int(settings.get("FrontAxisSign", 1))),
+ "coord": axis_name(int(settings.get("CoordAxis", 0)), int(settings.get("CoordAxisSign", 1))),
+ "handedness": "right",
+ "rotation_order": "XYZ",
+ "fbx_unit": "centimeter" if abs(float(settings.get("UnitScaleFactor", 1.0)) - 1.0) < 1e-6 else "custom",
+ "unit_scale_factor": float(settings.get("UnitScaleFactor", 1.0)),
+ "meters_per_fbx_unit": meters_per_fbx,
+ },
+ "transform_convention": (
+ "Maya/FBX: M_local = T * Roff * Rp * Rpre * R * Rpost^{-1} * Rp^{-1} * S; "
+ "M_world = M_parent_world * M_local. Euler XYZ in degrees, column-vector matrices."
+ ),
+ "root": limbs[roots[0]]["name"] if roots else None,
+ "bone_count": len(bones),
+ "hierarchy": hierarchy,
+ "bones": bones,
+ }
diff --git a/motion2mixamorig/mixamo/gltf.py b/motion2mixamorig/mixamo/gltf.py
new file mode 100644
index 0000000..694cfb8
--- /dev/null
+++ b/motion2mixamorig/mixamo/gltf.py
@@ -0,0 +1,240 @@
+"""Write a skinned Mixamo character + its animation as a .glb (glTF 2.0).
+
+Blender opens this in one import: mesh, skin weights, and the clip. Node TRS
+is the full local Mixamo transform `T * Rpre * R_anim` (glTF has no
+PreRotation), so the importer's linear-blend skin matches this repo's.
+
+Units are metres (Y-up). Vertex influences are the top four weights.
+"""
+
+from __future__ import annotations
+
+import json
+import struct
+from pathlib import Path
+
+import numpy as np
+
+from .fbx_skeleton import matrix_to_quat_wxyz
+from .kinematics import continue_quaternion, local_matrix
+from .skinned_mesh import YBotAsset
+
+_GLTF_MAGIC = 0x46546C67
+_JSON_CHUNK = 0x4E4F534A
+_BIN_CHUNK = 0x004E4942
+
+
+class _Bin:
+ def __init__(self) -> None:
+ self._parts: list[bytes] = []
+ self._cursor = 0
+
+ def add(self, data: bytes) -> tuple[int, int]:
+ offset = self._cursor
+ self._parts.append(data)
+ self._cursor += len(data)
+ pad = (4 - len(data) % 4) % 4
+ if pad:
+ self._parts.append(b"\x00" * pad)
+ self._cursor += pad
+ return offset, len(data)
+
+ def add_array(self, array: np.ndarray) -> tuple[int, int]:
+ return self.add(np.ascontiguousarray(array).tobytes())
+
+ def dumps(self) -> bytes:
+ return b"".join(self._parts)
+
+
+def _quat_xyzw(rmat: np.ndarray) -> np.ndarray:
+ wxyz = matrix_to_quat_wxyz(rmat)
+ return np.array([wxyz[1], wxyz[2], wxyz[3], wxyz[0]], dtype=np.float32)
+
+
+def _scale_affine(mat: np.ndarray, scale: float) -> np.ndarray:
+ """Metres from centimetres: R stays, translation is multiplied by `scale`."""
+ out = np.asarray(mat, dtype=np.float64).reshape(4, 4).copy()
+ out[:3, 3] *= scale
+ return out
+
+
+def _top4_weights(weights: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ n_bones = weights.shape[1]
+ take = min(4, n_bones)
+ order = np.argpartition(-weights, take - 1, axis=1)[:, :take]
+ w = np.take_along_axis(weights, order, axis=1)
+ w = w / np.maximum(w.sum(axis=1, keepdims=True), 1e-8)
+ if take < 4:
+ pad_i = np.zeros((weights.shape[0], 4 - take), dtype=order.dtype)
+ pad_w = np.zeros((weights.shape[0], 4 - take), dtype=w.dtype)
+ order = np.concatenate([order, pad_i], axis=1)
+ w = np.concatenate([w, pad_w], axis=1)
+ return order.astype(np.uint16), w.astype(np.float32)
+
+
+def _vertex_normals(vertices: np.ndarray, faces: np.ndarray) -> np.ndarray:
+ normals = np.zeros_like(vertices, dtype=np.float64)
+ p0, p1, p2 = vertices[faces[:, 0]], vertices[faces[:, 1]], vertices[faces[:, 2]]
+ face_n = np.cross(p1 - p0, p2 - p0)
+ for i in range(3):
+ np.add.at(normals, faces[:, i], face_n)
+ length = np.linalg.norm(normals, axis=1, keepdims=True)
+ return (normals / np.where(length < 1e-12, 1.0, length)).astype(np.float32)
+
+
+def write_glb(
+ path: Path,
+ asset: YBotAsset,
+ r_anim_frames: list[dict[str, np.ndarray]],
+ hips_cm_frames: list[np.ndarray],
+ fps: float,
+) -> Path:
+ """Skinned character + clip → a single `.glb` file."""
+ rig = asset.rig
+ scale = float(rig.meters_per_unit)
+ identity = np.eye(3)
+ n_bones = len(rig.order)
+ bone_index = {name: i for i, name in enumerate(rig.order)}
+
+ blob = _Bin()
+ accessors: list[dict] = []
+ views: list[dict] = []
+
+ def accessor(array: np.ndarray, typ: str, component: int, *, target: int | None = None,
+ normalized: bool = False) -> int:
+ offset, length = blob.add_array(array)
+ views.append({"buffer": 0, "byteOffset": offset, "byteLength": length,
+ **({"target": target} if target is not None else {})})
+ acc: dict = {
+ "bufferView": len(views) - 1,
+ "componentType": component,
+ "count": int(array.shape[0]),
+ "type": typ,
+ }
+ if normalized:
+ acc["normalized"] = True
+ if typ in {"VEC3", "VEC4", "SCALAR"} and array.dtype.kind == "f":
+ flat = array.reshape(array.shape[0], -1)
+ acc["min"] = [float(v) for v in flat.min(axis=0)]
+ acc["max"] = [float(v) for v in flat.max(axis=0)]
+ accessors.append(acc)
+ return len(accessors) - 1
+
+ nodes: list[dict] = []
+ for name in rig.order:
+ bone = rig.bones[name]
+ rest = local_matrix(bone, identity)
+ kids = [bone_index[c] for c in bone.children if c in bone_index]
+ node = {
+ "name": name,
+ "translation": [float(v) * scale for v in rest[:3, 3]],
+ "rotation": [float(v) for v in _quat_xyzw(rest[:3, :3])],
+ }
+ if kids:
+ node["children"] = kids
+ nodes.append(node)
+
+ mesh_node = len(nodes)
+ nodes.append({"name": "skinned_mesh", "mesh": 0, "skin": 0})
+
+ ibm = np.stack(
+ [_scale_affine(asset.meshes[0].ibm[i], scale) for i in range(n_bones)],
+ axis=0,
+ ).astype(np.float32)
+ # glTF stores matrices column-major.
+ ibm_acc = accessor(np.ascontiguousarray(ibm.transpose(0, 2, 1)), "MAT4", 5126)
+
+ primitives = []
+ materials = []
+ for mesh in asset.meshes:
+ verts = (mesh.vertices * scale).astype(np.float32)
+ faces = np.ascontiguousarray(mesh.faces, dtype=np.uint32)
+ joints, weights = _top4_weights(mesh.weights)
+ normals = _vertex_normals(verts.astype(np.float64), mesh.faces)
+ color = np.clip(np.asarray(mesh.color, dtype=np.float64).reshape(3), 0, 1)
+ materials.append({
+ "name": mesh.name,
+ "pbrMetallicRoughness": {
+ "baseColorFactor": [float(color[0]), float(color[1]), float(color[2]), 1.0],
+ "metallicFactor": 0.0,
+ "roughnessFactor": 0.7,
+ },
+ })
+ primitives.append({
+ "attributes": {
+ "POSITION": accessor(verts, "VEC3", 5126, target=34962),
+ "NORMAL": accessor(normals, "VEC3", 5126, target=34962),
+ "JOINTS_0": accessor(joints, "VEC4", 5123, target=34962),
+ "WEIGHTS_0": accessor(weights, "VEC4", 5126, target=34962),
+ },
+ "indices": accessor(faces.reshape(-1), "SCALAR", 5125, target=34963),
+ "material": len(materials) - 1,
+ })
+
+ n = len(r_anim_frames)
+ times = (np.arange(n, dtype=np.float32) / max(float(fps), 1e-6))
+ time_acc = accessor(times, "SCALAR", 5126)
+
+ channels = []
+ samplers = []
+ prev_q: dict[str, np.ndarray] = {}
+ rot_tracks = {name: np.zeros((n, 4), dtype=np.float32) for name in rig.order}
+ hips_track = np.zeros((n, 3), dtype=np.float32)
+ for i, (r_anim, hips) in enumerate(zip(r_anim_frames, hips_cm_frames)):
+ hips_track[i] = np.asarray(hips, dtype=np.float64).reshape(3) * scale
+ for name in rig.order:
+ bone = rig.bones[name]
+ local = local_matrix(
+ bone,
+ r_anim.get(name, identity),
+ hips if name == rig.root else None,
+ )
+ q_wxyz, _ = continue_quaternion(matrix_to_quat_wxyz(local[:3, :3]), prev_q.get(name))
+ prev_q[name] = q_wxyz
+ rot_tracks[name][i] = [q_wxyz[1], q_wxyz[2], q_wxyz[3], q_wxyz[0]]
+
+ def add_sampler(values: np.ndarray, typ: str) -> int:
+ samplers.append({"input": time_acc, "output": accessor(values, typ, 5126), "interpolation": "LINEAR"})
+ return len(samplers) - 1
+
+ hips_s = add_sampler(hips_track.astype(np.float32), "VEC3")
+ channels.append({"sampler": hips_s, "target": {"node": bone_index[rig.root], "path": "translation"}})
+ for name in rig.order:
+ sid = add_sampler(rot_tracks[name], "VEC4")
+ channels.append({"sampler": sid, "target": {"node": bone_index[name], "path": "rotation"}})
+
+ gltf = {
+ "asset": {"version": "2.0", "generator": "motion2mixamorig"},
+ "scene": 0,
+ "scenes": [{"nodes": [bone_index[rig.root], mesh_node]}],
+ "nodes": nodes,
+ "meshes": [{"name": "character", "primitives": primitives}],
+ "materials": materials,
+ "skins": [{
+ "name": "mixamo",
+ "joints": list(range(n_bones)),
+ "skeleton": bone_index[rig.root],
+ "inverseBindMatrices": ibm_acc,
+ }],
+ "animations": [{"name": "mixamo_clip", "samplers": samplers, "channels": channels}],
+ "accessors": accessors,
+ "bufferViews": views,
+ "buffers": [{"byteLength": blob._cursor}],
+ }
+
+ json_bytes = json.dumps(gltf, separators=(",", ":")).encode("utf-8")
+ json_pad = (4 - len(json_bytes) % 4) % 4
+ json_bytes = json_bytes + b" " * json_pad
+ bin_bytes = blob.dumps()
+ bin_pad = (4 - len(bin_bytes) % 4) % 4
+ bin_bytes = bin_bytes + b"\x00" * bin_pad
+
+ total = 12 + 8 + len(json_bytes) + 8 + len(bin_bytes)
+ header = struct.pack("<III", _GLTF_MAGIC, 2, total)
+ json_chunk = struct.pack("<II", len(json_bytes), _JSON_CHUNK) + json_bytes
+ bin_chunk = struct.pack("<II", len(bin_bytes), _BIN_CHUNK) + bin_bytes
+
+ path = Path(path)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_bytes(header + json_chunk + bin_chunk)
+ return path
diff --git a/motion2mixamorig/mixamo/kinematics.py b/motion2mixamorig/mixamo/kinematics.py
new file mode 100644
index 0000000..a5dcd41
--- /dev/null
+++ b/motion2mixamorig/mixamo/kinematics.py
@@ -0,0 +1,420 @@
+"""Mixamo Y Bot FK that keeps FBX PreRotation as a separate node.
+
+Maya/FBX (and Assimp's extra PreRotation node) compose as:
+
+ M_local = T * Rpre * R_anim
+ M_world = M_parent_world * M_local
+
+R_anim is the animated Lcl Rotation. Rpre is rest-only and is never baked
+into R_anim. Bone lengths live in T (Lcl Translation) and stay constant.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+from .fbx_skeleton import matrix_to_quat_wxyz
+
+
+def _as_vec3(value: Iterable[float]) -> np.ndarray:
+ return np.asarray(list(value)[:3], dtype=np.float64)
+
+
+def _as_mat3(value) -> np.ndarray:
+ return np.asarray(value, dtype=np.float64).reshape(3, 3)
+
+
+def _as_mat4(value) -> np.ndarray:
+ return np.asarray(value, dtype=np.float64).reshape(4, 4)
+
+
+def normalize(v: np.ndarray, fallback: np.ndarray | None = None) -> np.ndarray:
+ v = np.asarray(v, dtype=np.float64).reshape(3)
+ n = float(np.linalg.norm(v))
+ if n < 1e-10:
+ if fallback is None:
+ raise ValueError("cannot normalize a near-zero vector")
+ return np.asarray(fallback, dtype=np.float64).reshape(3)
+ return v / n
+
+
+def rotation_aligning(src: np.ndarray, dst: np.ndarray) -> np.ndarray:
+ """Shortest rotation R with R @ src = dst (column vectors). Twist-free swing."""
+ a = normalize(src, np.array([0.0, 1.0, 0.0]))
+ b = normalize(dst, np.array([0.0, 1.0, 0.0]))
+ cosine = float(np.clip(a @ b, -1.0, 1.0))
+ axis = np.cross(a, b)
+ sine = float(np.linalg.norm(axis))
+ if cosine > 0.999999:
+ return np.eye(3)
+ if cosine < -0.999999:
+ helper = np.array([1.0, 0.0, 0.0]) if abs(a[0]) < 0.9 else np.array([0.0, 1.0, 0.0])
+ axis = normalize(np.cross(a, helper))
+ return 2.0 * np.outer(axis, axis) - np.eye(3)
+ skew = np.array(
+ [[0.0, -axis[2], axis[1]], [axis[2], 0.0, -axis[0]], [-axis[1], axis[0], 0.0]],
+ dtype=np.float64,
+ )
+ return np.eye(3) + skew + skew @ skew * ((1.0 - cosine) / (sine * sine))
+
+
+def orthonormal_frame(left: np.ndarray, up: np.ndarray) -> np.ndarray:
+ """Right-handed Mixamo axes: columns = (left=+X, up=+Y, forward=+Z)."""
+ up_u = normalize(up, np.array([0.0, 1.0, 0.0]))
+ left_rej = left - up_u * float(left @ up_u)
+ if float(np.linalg.norm(left_rej)) < 1e-8:
+ helper = np.array([1.0, 0.0, 0.0]) if abs(up_u[0]) < 0.9 else np.array([0.0, 0.0, 1.0])
+ left_rej = np.cross(helper, up_u)
+ left_u = normalize(left_rej)
+ fwd = normalize(np.cross(left_u, up_u))
+ left_u = normalize(np.cross(up_u, fwd))
+ return np.column_stack([left_u, up_u, fwd])
+
+
+def angle_deg(a: np.ndarray, b: np.ndarray) -> float:
+ ua = normalize(a, np.array([0.0, 1.0, 0.0]))
+ ub = normalize(b, np.array([0.0, 1.0, 0.0]))
+ return float(np.degrees(np.arccos(np.clip(ua @ ub, -1.0, 1.0))))
+
+
+def mat4(rotation: np.ndarray, translation: np.ndarray) -> np.ndarray:
+ out = np.eye(4)
+ out[:3, :3] = rotation
+ out[:3, 3] = translation
+ return out
+
+
+def rh_axes(lateral: np.ndarray, up: np.ndarray) -> np.ndarray:
+ """Right-handed columns (lateral, up, lateral × up)."""
+ up_u = normalize(up, np.array([0.0, 1.0, 0.0]))
+ lat_rej = lateral - up_u * float(lateral @ up_u)
+ if float(np.linalg.norm(lat_rej)) < 1e-8:
+ helper = np.array([1.0, 0.0, 0.0]) if abs(up_u[0]) < 0.9 else np.array([0.0, 0.0, 1.0])
+ lat_rej = np.cross(helper, up_u)
+ lat_u = normalize(lat_rej)
+ fwd = normalize(np.cross(lat_u, up_u))
+ lat_u = normalize(np.cross(up_u, fwd))
+ up_u = normalize(np.cross(fwd, lat_u))
+ return np.column_stack([lat_u, up_u, fwd])
+
+
+def human_body_frame(joints_or_right, up: np.ndarray | None = None) -> np.ndarray:
+ """Human torso columns (right, up, chest-forward).
+
+ chest-forward = right × up (matches the source video: f253 faces +X).
+ up × right is the back.
+ """
+ if up is None:
+ raise TypeError("human_body_frame(right, up)")
+ return rh_axes(joints_or_right, up)
+
+
+def mixamo_bone_frame(left: np.ndarray, up: np.ndarray) -> np.ndarray:
+ """Mixamo rest bone columns (left=+X, up=+Y, chest-forward=+Z)."""
+ return rh_axes(left, up)
+
+
+def body_frame_from_landmarks(right: np.ndarray, up: np.ndarray, chest_hint: np.ndarray) -> np.ndarray:
+ """Columns (right, up, forward). Forward is whichever of right×up / up×right matches chest_hint.
+
+ Does not recross `right` from up×forward. That would force det +1 by flipping
+ Mixamo's anatomical right onto character left.
+ """
+ up_u = normalize(up, np.array([0.0, 1.0, 0.0]))
+ right_rej = right - up_u * float(np.asarray(right, dtype=np.float64) @ up_u)
+ right_u = normalize(right_rej)
+ fwd_ru = np.cross(right_u, up_u)
+ fwd_ur = np.cross(up_u, right_u)
+ hint = np.asarray(chest_hint, dtype=np.float64).reshape(3)
+ fwd = fwd_ur if float(hint @ fwd_ur) >= float(hint @ fwd_ru) else fwd_ru
+ return np.column_stack([right_u, up_u, normalize(fwd)])
+
+
+def body_align_from_bases(r_human: np.ndarray, r_mixamo: np.ndarray, *, proper: bool = False) -> np.ndarray:
+ """Linear map R with R @ r_human ≈ r_mixamo.
+
+ The anatomical Human→Mixamo map (right→right, up→up, chest→chest) has det −1
+ because Mixamo rest +X is character left. `proper=True` polar-projects onto
+ SO(3). Identity and rotY(180) are equally close; neither matches all three axes.
+ """
+ a = np.asarray(r_mixamo, dtype=np.float64) @ np.asarray(r_human, dtype=np.float64).T
+ if not proper:
+ return a
+ u, _, vt = np.linalg.svd(a)
+ r = u @ vt
+ if float(np.linalg.det(r)) < 0.0:
+ u = u.copy()
+ u[:, -1] *= -1.0
+ r = u @ vt
+ return r
+
+
+# Maps the capture's anatomical triad onto Mixamo's:
+# Human (right, up, chest) = (+X, +Y, −Z)
+# Mixamo (right, up, chest) = (−X, +Y, +Z) # rest +X is character left
+# Anatomy obeys chest = up × right, so both triads are left-handed and the map
+# between them is a proper rotation: rotY(180), det +1, no reflection.
+#
+# The capture's chest axis is −Z, measured rather than assumed: over all frames
+# the toes-anterior and knees-bow-anterior directions agree to 5.4 deg mean and
+# both sit ~167 deg from +Z. Taking chest as +Z makes this map the reflection
+# diag(-1, 1, 1), which lands the body's front on Mixamo's back and inverts
+# laterality — the torso-backward bug.
+R_BODY_ALIGN = np.diag([-1.0, 1.0, -1.0])
+
+
+def human_to_mixamo(p: np.ndarray) -> np.ndarray:
+ """Human world (m) → Mixamo FBX world by one global body-frame map.
+
+ Human: +X = anatomical right, +Y = up, −Z = chest-forward.
+ Mixamo rest: +X = anatomical left, +Y = up, +Z = chest-forward (toes / FBX front).
+
+ p_mixamo = R_BODY_ALIGN @ p_human = [-x, y, -z]
+ """
+ p = np.asarray(p, dtype=np.float64)
+ return np.asarray(p, dtype=np.float64) @ R_BODY_ALIGN.T
+
+
+@dataclass
+class MixamoBone:
+ name: str
+ parent: str | None
+ children: list[str]
+ pre_rotation: np.ndarray
+ lcl_translation_cm: np.ndarray
+ rest_world: np.ndarray
+ rest_local: np.ndarray
+ pre_rotation_euler_deg: np.ndarray
+
+
+@dataclass
+class MixamoRig:
+ bones: dict[str, MixamoBone]
+ order: list[str]
+ meters_per_unit: float
+ source: Path
+ coord_system: dict = field(default_factory=dict)
+
+ @property
+ def root(self) -> str:
+ return self.order[0]
+
+ def child_offset_cm(self, parent: str, child: str) -> np.ndarray:
+ return self.bones[child].lcl_translation_cm.copy()
+
+
+def rig_from_skeleton_data(data: dict[str, Any], source: Path) -> MixamoRig:
+ """Build a MixamoRig from the dict `fbx_skeleton.extract_skeleton` returns."""
+ bones: dict[str, MixamoBone] = {}
+ for raw in data["bones"]:
+ name = str(raw["name"])
+ bones[name] = MixamoBone(
+ name=name,
+ parent=raw["parent"],
+ children=list(raw["children"]),
+ pre_rotation=_as_mat3(raw["pre_rotation_matrix"]),
+ lcl_translation_cm=_as_vec3(raw["lcl_translation"]),
+ rest_world=_as_mat4(raw["rest_world"]["matrix"]),
+ rest_local=_as_mat4(raw["rest_local"]["matrix"]),
+ pre_rotation_euler_deg=_as_vec3(raw["pre_rotation_euler_deg"]),
+ )
+ return MixamoRig(
+ bones=bones,
+ order=[b["name"] for b in data["bones"]],
+ meters_per_unit=float(data["coord_system"]["meters_per_fbx_unit"]),
+ source=Path(source),
+ coord_system=dict(data["coord_system"]),
+ )
+
+
+def load_mixamo_rig(path: Path) -> MixamoRig:
+ """Load a core-bone rig from a previously exported skeleton JSON."""
+ path = Path(path)
+ data = json.loads(path.read_text(encoding="utf-8"))
+ return rig_from_skeleton_data(data, path)
+
+
+def local_matrix(bone: MixamoBone, r_anim: np.ndarray, translation_cm: np.ndarray | None = None) -> np.ndarray:
+ """Assimp-equivalent local node: T * Rpre * R_anim."""
+ t = bone.lcl_translation_cm if translation_cm is None else np.asarray(translation_cm, dtype=np.float64)
+ return mat4(bone.pre_rotation @ r_anim, t)
+
+
+def forward_kinematics(
+ rig: MixamoRig,
+ r_anim: dict[str, np.ndarray] | None = None,
+ hips_translation_cm: np.ndarray | None = None,
+ lcl_translations_cm: dict[str, np.ndarray] | None = None,
+) -> dict[str, np.ndarray]:
+ """World 4x4 for every bone. Missing R_anim entries stay identity (rest)."""
+ anim = r_anim or {}
+ extra_t = lcl_translations_cm or {}
+ world: dict[str, np.ndarray] = {}
+ for name in rig.order:
+ bone = rig.bones[name]
+ rot = np.asarray(anim.get(name, np.eye(3)), dtype=np.float64)
+ if name == rig.root and hips_translation_cm is not None:
+ trans = hips_translation_cm
+ elif name in extra_t:
+ trans = extra_t[name]
+ else:
+ trans = None
+ local = local_matrix(bone, rot, trans)
+ if bone.parent is None:
+ world[name] = local
+ else:
+ world[name] = world[bone.parent] @ local
+ return world
+
+
+def world_positions_m(rig: MixamoRig, world: dict[str, np.ndarray]) -> dict[str, np.ndarray]:
+ scale = rig.meters_per_unit
+ return {name: mat[:3, 3] * scale for name, mat in world.items()}
+
+
+def bone_direction_world(rig: MixamoRig, world: dict[str, np.ndarray], parent: str, child: str) -> np.ndarray:
+ """World direction of parent→child. Independent of child's own rotation."""
+ offset = rig.child_offset_cm(parent, child)
+ return world[parent][:3, :3] @ offset
+
+
+def anim_from_world_rotation(bone: MixamoBone, r_parent_world: np.ndarray, r_world: np.ndarray) -> np.ndarray:
+ """R_anim such that R_parent * Rpre * R_anim = R_world."""
+ return bone.pre_rotation.T @ r_parent_world.T @ r_world
+
+
+def swing_anim_to_direction(
+ rig: MixamoRig,
+ world: dict[str, np.ndarray],
+ bone_name: str,
+ child_name: str | None,
+ target_dir_world: np.ndarray,
+ d_local: np.ndarray | None = None,
+) -> np.ndarray:
+ """Lcl Rotation that aims this bone's rest axis at target_dir_world.
+
+ Rest axis is the child's Lcl Translation (Mixamo local +Y) after PreRotation:
+ R_parent * Rpre * R_anim * d_local = d_target
+ """
+ bone = rig.bones[bone_name]
+ parent_r = np.eye(3) if bone.parent is None else world[bone.parent][:3, :3]
+ if d_local is None:
+ if child_name is None:
+ raise ValueError(f"{bone_name}: need child_name or d_local")
+ d_local = rig.child_offset_cm(bone_name, child_name)
+ d_in_anim = bone.pre_rotation.T @ parent_r.T @ normalize(target_dir_world)
+ return rotation_aligning(d_local, d_in_anim)
+
+
+def frame_from_primary_secondary(primary: np.ndarray, secondary: np.ndarray) -> np.ndarray:
+ """Orthonormal basis: columns = (primary, secondary_ortho, primary × secondary_ortho)."""
+ p = normalize(primary, np.array([0.0, 1.0, 0.0]))
+ s = secondary - p * float(secondary @ p)
+ if float(np.linalg.norm(s)) < 1e-8:
+ helper = np.array([0.0, 1.0, 0.0]) if abs(p[1]) < 0.9 else np.array([1.0, 0.0, 0.0])
+ s = np.cross(helper, p)
+ s = normalize(s)
+ t = np.cross(p, s)
+ return np.column_stack([p, s, t])
+
+
+def frame_anim_from_two_vectors(
+ bone: MixamoBone,
+ r_parent_world: np.ndarray,
+ rest_primary_local: np.ndarray,
+ rest_secondary_local: np.ndarray,
+ tgt_primary_world: np.ndarray,
+ tgt_secondary_world: np.ndarray,
+) -> np.ndarray:
+ """Lcl Rotation mapping rest local (primary, secondary) onto world targets.
+
+ Used for feet: primary = ankle→toe (forward), secondary = world up (twist).
+ """
+ r_rest_world = r_parent_world @ bone.pre_rotation
+ r_rest_frame = frame_from_primary_secondary(
+ r_rest_world @ rest_primary_local, r_rest_world @ rest_secondary_local
+ )
+ r_tgt_frame = frame_from_primary_secondary(tgt_primary_world, tgt_secondary_world)
+ r_world = (r_tgt_frame @ r_rest_frame.T) @ r_rest_world
+ return bone.pre_rotation.T @ r_parent_world.T @ r_world
+
+
+def frame_anim_from_axes(
+ bone: MixamoBone,
+ r_parent_world: np.ndarray,
+ rest_left_local: np.ndarray,
+ rest_up_local: np.ndarray,
+ tgt_left_world: np.ndarray,
+ tgt_up_world: np.ndarray,
+) -> np.ndarray:
+ """Lcl Rotation that maps rest local (left, up) onto the world target axes.
+
+ Used for Hips and Spine2 so clavicle attachments keep a chest twist, not a
+ free swing around the spine.
+ """
+ r_rest_world = r_parent_world @ bone.pre_rotation
+ r_rest_frame = orthonormal_frame(r_rest_world @ rest_left_local, r_rest_world @ rest_up_local)
+ r_tgt_frame = orthonormal_frame(tgt_left_world, tgt_up_world)
+ r_world = (r_tgt_frame @ r_rest_frame.T) @ r_rest_world
+ return bone.pre_rotation.T @ r_parent_world.T @ r_world
+
+
+
+def quat_wxyz(rmat: np.ndarray) -> np.ndarray:
+ return matrix_to_quat_wxyz(rmat)
+
+
+def local_aim_axis(rig: MixamoRig, bone_name: str) -> np.ndarray:
+ """Rest bone axis in the bone's local (pre-R_anim) frame. Mixamo +Y, or the first child offset."""
+ bone = rig.bones[bone_name]
+ if bone.children:
+ return rig.child_offset_cm(bone_name, bone.children[0])
+ return np.array([0.0, 1.0, 0.0])
+
+
+def geodesic_angle_deg(r_a: np.ndarray, r_b: np.ndarray) -> float:
+ rel = np.asarray(r_a, dtype=np.float64).T @ np.asarray(r_b, dtype=np.float64)
+ cosine = (float(np.trace(rel)) - 1.0) * 0.5
+ return float(np.degrees(np.arccos(np.clip(cosine, -1.0, 1.0))))
+
+
+def rotation_180_around(axis: np.ndarray) -> np.ndarray:
+ n = normalize(axis, np.array([0.0, 1.0, 0.0]))
+ return 2.0 * np.outer(n, n) - np.eye(3)
+
+
+def pick_closest_aim_twist(
+ r_curr: np.ndarray,
+ r_prev: np.ndarray | None,
+ axis: np.ndarray,
+) -> tuple[np.ndarray, bool]:
+ """Keep the independent aim. If a 180° twist around the bone axis is closer to r_prev, use it.
+
+ Does not blend or smooth: only the discrete antipodal twist that still aims the same way.
+ """
+ r_curr = np.asarray(r_curr, dtype=np.float64)
+ if r_prev is None:
+ return r_curr, False
+ r_prev = np.asarray(r_prev, dtype=np.float64)
+ r_flip = r_curr @ rotation_180_around(axis)
+ if geodesic_angle_deg(r_flip, r_prev) + 1e-9 < geodesic_angle_deg(r_curr, r_prev):
+ return r_flip, True
+ return r_curr, False
+
+
+def continue_quaternion(q_curr: np.ndarray, q_prev: np.ndarray | None) -> tuple[np.ndarray, bool]:
+ """If dot(q_t, q_{t-1}) < 0, use -q_t. Quaternion double-cover only; same rotation matrix."""
+ q = np.asarray(q_curr, dtype=np.float64).reshape(4)
+ n = float(np.linalg.norm(q))
+ if n < 1e-12:
+ q = np.array([1.0, 0.0, 0.0, 0.0])
+ else:
+ q = q / n
+ if q_prev is not None and float(np.dot(q, np.asarray(q_prev, dtype=np.float64).reshape(4))) < 0.0:
+ return -q, True
+ return q, False
diff --git a/motion2mixamorig/mixamo/render.py b/motion2mixamorig/mixamo/render.py
new file mode 100644
index 0000000..8c53e07
--- /dev/null
+++ b/motion2mixamorig/mixamo/render.py
@@ -0,0 +1,609 @@
+"""Rendering for the Y Bot rig and mesh, with cameras taken from the calibration.
+
+The anatomical FRONT / BACK / LEFT / RIGHT cameras are built from the validated
+Y Bot Right / Up / Forward axes using the same construction that produced the
+accepted orientation-check images. There is no image flip and no "camera on −Z"
+rule compensating for a mapping error.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import numpy as np
+
+from .kinematics import normalize
+
+BG_BGR = (18, 18, 20)
+GRID_BGR = (58, 56, 52)
+SHADOW_BGR = (8, 8, 9)
+
+
+@dataclass(frozen=True)
+class Theme:
+ """Panel palette. The floor runs far -> near so it reads as receding."""
+
+ bg: tuple[int, int, int]
+ floor_far: tuple[int, int, int]
+ floor_near: tuple[int, int, int]
+ grid: tuple[int, int, int]
+ shadow: tuple[int, int, int]
+ shadow_alpha: float
+ # White joint dots vanish on a pale floor and leave the limbs looking dashed.
+ joint: tuple[int, int, int]
+
+
+# Black backdrop with the floor only just lifted off it, so the grid still reads
+# as a surface while everything but the dancer stays out of the way.
+DARK_THEME = Theme((0, 0, 0), (10, 10, 11), (22, 22, 25), (52, 51, 48), (0, 0, 0), 0.75,
+ (242, 242, 242))
+# Sampled off the Mixamo viewport: flat 160 backdrop, floor 193 at the far edge
+# rising to 204 underfoot, grid lines a little darker than the floor.
+LIGHT_THEME = Theme((160, 160, 160), (193, 193, 193), (208, 208, 208), (168, 168, 168),
+ (108, 108, 108), 0.42, (48, 48, 48))
+# For the character grid. Same mood as DARK_THEME but off black, because a
+# navy or black outfit against a black backdrop has no silhouette to read.
+GRID_THEME = Theme((30, 29, 27), (44, 43, 41), (66, 65, 62), (94, 92, 88), (0, 0, 0), 0.50,
+ (242, 242, 242))
+PANEL_BAR_BGR = (16, 16, 18)
+TEXT_BGR = (240, 240, 240)
+MUTED_BGR = (136, 136, 136)
+FLAG_BGR = (60, 60, 220)
+LEFT_BGR = (232, 155, 76)
+RIGHT_BGR = (90, 90, 224)
+TORSO_BGR = (76, 200, 230)
+
+MESH_COLORS_BGR = {
+ "Alpha_Surface": np.array([200, 188, 180], dtype=np.float64),
+ "Alpha_Joints": np.array([112, 104, 98], dtype=np.float64),
+}
+
+# The teal the Mixamo viewport gives the Y Bot, sampled off a screenshot.
+MESH_COLORS_MIXAMO_BGR = {
+ "Alpha_Surface": np.array([205, 180, 140], dtype=np.float64),
+ "Alpha_Joints": np.array([120, 104, 84], dtype=np.float64),
+}
+
+# White shell over the stock grey joints. The white stops just under 255 on
+# purpose: at 255 the lit faces clip and lose their shading.
+MESH_COLORS_WHITE_BGR = {
+ "Alpha_Surface": np.array([246, 246, 248], dtype=np.float64),
+ "Alpha_Joints": np.array([112, 104, 98], dtype=np.float64),
+}
+
+VIEWS = ("front", "back", "left", "right")
+
+
+@dataclass(frozen=True)
+class Camera:
+ """Orthographic anatomical view derived from the calibrated rig axes."""
+
+ view: str
+ look: np.ndarray
+ cam_right: np.ndarray
+ cam_up: np.ndarray
+ center: np.ndarray
+ scale: float
+ width: int
+ height: int
+
+ def project(self, pts: np.ndarray) -> np.ndarray:
+ d = np.asarray(pts, dtype=np.float64).reshape(-1, 3) - self.center
+ u = d @ self.cam_right
+ v = d @ self.cam_up
+ x = self.width * 0.5 + u * self.scale
+ y = self.height * 0.5 - v * self.scale
+ return np.column_stack([x, y])
+
+ def depth(self, pts: np.ndarray) -> np.ndarray:
+ d = np.asarray(pts, dtype=np.float64).reshape(-1, 3) - self.center
+ return d @ self.look
+
+ def clip_near(self, poly: np.ndarray) -> np.ndarray:
+ """Nothing is behind an orthographic camera, so this is the identity."""
+ return np.asarray(poly, dtype=np.float64).reshape(-1, 3)
+
+ def clip_segment(self, a: np.ndarray, b: np.ndarray):
+ return a, b
+
+
+@dataclass(frozen=True)
+class PerspectiveCamera:
+ """Pinhole view. Same project / depth / clip_near surface as Camera.
+
+ An orthographic camera cannot show a floor receding to a vanishing point;
+ every horizontal line stays parallel and evenly spaced. This one can.
+ """
+
+ eye: np.ndarray
+ look: np.ndarray
+ cam_right: np.ndarray
+ cam_up: np.ndarray
+ focal: float
+ width: int
+ height: int
+ near: float = 0.05
+
+ def project(self, pts: np.ndarray) -> np.ndarray:
+ d = np.asarray(pts, dtype=np.float64).reshape(-1, 3) - self.eye
+ z = np.maximum(d @ self.look, self.near)
+ u = (d @ self.cam_right) / z * self.focal
+ v = (d @ self.cam_up) / z * self.focal
+ return np.column_stack([self.width * 0.5 + u, self.height * 0.5 - v])
+
+ def depth(self, pts: np.ndarray) -> np.ndarray:
+ d = np.asarray(pts, dtype=np.float64).reshape(-1, 3) - self.eye
+ return d @ self.look
+
+ def clip_near(self, poly: np.ndarray) -> np.ndarray:
+ """Sutherland-Hodgman against z = near, so nothing behind the eye wraps."""
+ p = np.asarray(poly, dtype=np.float64).reshape(-1, 3)
+ z = self.depth(p) - self.near
+ out = []
+ for i in range(len(p)):
+ j = (i + 1) % len(p)
+ if z[i] >= 0.0:
+ out.append(p[i])
+ if (z[i] >= 0.0) != (z[j] >= 0.0):
+ out.append(p[i] + z[i] / (z[i] - z[j]) * (p[j] - p[i]))
+ return np.asarray(out, dtype=np.float64).reshape(-1, 3)
+
+ def clip_segment(self, a: np.ndarray, b: np.ndarray):
+ """Trim a world segment to the half-space in front of the eye, or drop it."""
+ za, zb = (float(v) for v in self.depth(np.stack([a, b])) - self.near)
+ if za < 0.0 and zb < 0.0:
+ return None
+ if za < 0.0:
+ a = a + za / (za - zb) * (b - a)
+ elif zb < 0.0:
+ b = b + zb / (zb - za) * (a - b)
+ return a, b
+
+ @property
+ def fov_deg(self) -> float:
+ return float(2.0 * np.degrees(np.arctan(self.height * 0.5 / self.focal)))
+
+
+def view_basis(axes: dict, view: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ """Same anatomical camera construction as the accepted orientation check.
+
+ FRONT: stand on +Forward and look at the face/chest.
+ BACK: stand on −Forward. LEFT / RIGHT: stand on that anatomical side.
+ """
+ right = np.asarray(axes["right"], dtype=np.float64)
+ up = np.asarray(axes["up"], dtype=np.float64)
+ fwd = np.asarray(axes["forward"], dtype=np.float64)
+ eye_dir = {"front": fwd, "back": -fwd, "left": -right, "right": right}[view]
+ look = normalize(-eye_dir)
+ cam_right = np.cross(look, up)
+ if float(np.linalg.norm(cam_right)) < 1e-8:
+ cam_right = np.cross(look, fwd)
+ cam_right = normalize(cam_right)
+ cam_up = normalize(np.cross(cam_right, look))
+ return look, cam_right, cam_up
+
+
+def pitch_down(look: np.ndarray, cam_up: np.ndarray, deg: float) -> tuple[np.ndarray, np.ndarray]:
+ """Rotate a view basis down by `deg` about the unchanged camera right axis.
+
+ A level orthographic camera sees a horizontal plane exactly edge-on, so a
+ floor drawn under the character collapses to a line. Pitching down opens the
+ plane out without touching the horizontal framing.
+ """
+ t = np.radians(deg)
+ return (
+ normalize(look * np.cos(t) - cam_up * np.sin(t)),
+ normalize(cam_up * np.cos(t) + look * np.sin(t)),
+ )
+
+
+def build_camera(
+ axes: dict,
+ view: str,
+ points: np.ndarray,
+ width: int,
+ height: int,
+ *,
+ fill: float = 0.82,
+ pitch_deg: float = 0.0,
+) -> Camera:
+ """Fixed camera framing every point of the whole sequence.
+
+ `points` is the full set of world points over all frames, so the framing is
+ computed once. Nothing is re-centred per frame.
+ """
+ look, cam_right, cam_up = view_basis(axes, view)
+ if pitch_deg:
+ look, cam_up = pitch_down(look, cam_up, pitch_deg)
+ pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
+ u = pts @ cam_right
+ v = pts @ cam_up
+ center_u = 0.5 * (float(u.min()) + float(u.max()))
+ center_v = 0.5 * (float(v.min()) + float(v.max()))
+ center = center_u * cam_right + center_v * cam_up
+ span_u = max(float(u.max() - u.min()), 1e-3)
+ span_v = max(float(v.max() - v.min()), 1e-3)
+ scale = fill * min(width / span_u, height / span_v)
+ return Camera(
+ view=view,
+ look=look,
+ cam_right=cam_right,
+ cam_up=cam_up,
+ center=center,
+ scale=scale,
+ width=width,
+ height=height,
+ )
+
+
+def build_perspective_camera(
+ axes: dict,
+ points: np.ndarray,
+ width: int,
+ height: int,
+ *,
+ fov_deg: float = 28.0,
+ azimuth_deg: float = 0.0,
+ elevation_deg: float = 10.0,
+ fill: float = 0.94,
+ iterations: int = 12,
+) -> PerspectiveCamera:
+ """Pinhole camera orbiting the sequence, framed on every point of it.
+
+ azimuth turns the eye around the character's up axis from dead ahead;
+ elevation lifts it above the floor. The eye distance and the aim point are
+ then solved together so the whole clip fits, since with perspective the
+ framing depends on where the eye is and not on a scale alone.
+ """
+ up = normalize(np.asarray(axes["up"], dtype=np.float64))
+ fwd = normalize(np.asarray(axes["forward"], dtype=np.float64))
+ right = normalize(np.asarray(axes["right"], dtype=np.float64))
+
+ az, el = np.radians(azimuth_deg), np.radians(elevation_deg)
+ offset = normalize(np.cos(el) * (np.cos(az) * fwd + np.sin(az) * right) + np.sin(el) * up)
+ look = -offset
+ cam_right = normalize(np.cross(look, up))
+ cam_up = normalize(np.cross(cam_right, look))
+ focal = (height * 0.5) / np.tan(np.radians(fov_deg) * 0.5)
+
+ pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
+ target = 0.5 * (pts.min(axis=0) + pts.max(axis=0))
+ dist = float(np.linalg.norm(pts.max(axis=0) - pts.min(axis=0))) * 3.0
+
+ def place(t: np.ndarray, d: float) -> PerspectiveCamera:
+ return PerspectiveCamera(t + offset * d, look, cam_right, cam_up,
+ float(focal), width, height)
+
+ for _ in range(iterations):
+ cam = place(target, dist)
+ uv = cam.project(pts)
+ lo, hi = uv.min(axis=0), uv.max(axis=0)
+ mid = 0.5 * (lo + hi)
+ # The world point on the aim plane that lands on `mid`; aim there instead.
+ target = target + ((mid[0] - width * 0.5) * cam_right
+ - (mid[1] - height * 0.5) * cam_up) * dist / focal
+ dist *= max((hi[0] - lo[0]) / (fill * width), (hi[1] - lo[1]) / (fill * height))
+
+ # Re-aiming and rescaling interact, so guarantee the fit rather than assume it.
+ for _ in range(40):
+ cam = place(target, dist)
+ uv = cam.project(pts)
+ if (uv.min(axis=0) >= 0.0).all() and (uv.max(axis=0) < (width, height)).all():
+ break
+ dist *= 1.02
+ return place(target, dist)
+
+
+def content_panel_size(
+ axes: dict,
+ view: str,
+ points: np.ndarray,
+ height: int,
+ *,
+ min_w: int = 560,
+ max_w: int = 1280,
+) -> tuple[int, int]:
+ """Panel shape matching what the sequence actually sweeps out.
+
+ The camera stays fixed; this only stops a wandering character from being
+ rendered into a panel whose aspect leaves most of it empty.
+ """
+ _, cam_right, cam_up = view_basis(axes, view)
+ pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)
+ span_u = max(float(np.ptp(pts @ cam_right)), 1e-3)
+ span_v = max(float(np.ptp(pts @ cam_up)), 1e-3)
+ width = int(round(height * span_u / span_v))
+ width = int(np.clip(width, min_w, max_w))
+ return width + width % 2, height + height % 2
+
+
+def edge_bgr(a: str, b: str) -> tuple[int, int, int]:
+ token = f"{a}_{b}".lower()
+ if "left" in token:
+ return LEFT_BGR
+ if "right" in token:
+ return RIGHT_BGR
+ return TORSO_BGR
+
+
+def blank_panel(width: int, height: int) -> np.ndarray:
+ return np.full((height, width, 3), BG_BGR, dtype=np.uint8)
+
+
+@dataclass(frozen=True)
+class Ground:
+ """A square patch of floor the character is standing on.
+
+ `y` is a world height, so panels sharing a camera and a Ground put their
+ floors on the same pixel rows and read as one surface. The patch is wide
+ enough that under perspective its far edge crowds the vanishing point.
+ """
+
+ y: float
+ cx: float
+ cz: float
+ extent: float = 9.0
+ step: float = 0.5
+
+ def corners(self) -> np.ndarray:
+ e = self.extent
+ return np.array([
+ (self.cx - e, self.y, self.cz - e),
+ (self.cx + e, self.y, self.cz - e),
+ (self.cx + e, self.y, self.cz + e),
+ (self.cx - e, self.y, self.cz + e),
+ ], dtype=np.float64)
+
+ def lines(self) -> np.ndarray:
+ """Grid lines as world endpoint pairs, shape (n, 2, 3)."""
+ e = self.extent
+ ticks = np.arange(-e, e + 1e-9, self.step)
+ segs = [((self.cx + t, self.y, self.cz - e), (self.cx + t, self.y, self.cz + e))
+ for t in ticks]
+ segs += [((self.cx - e, self.y, self.cz + t), (self.cx + e, self.y, self.cz + t))
+ for t in ticks]
+ return np.asarray(segs, dtype=np.float64)
+
+
+def ground_backdrop(cam, ground: Ground, width: int, height: int,
+ theme: Theme = DARK_THEME) -> np.ndarray:
+ """The floor drawn once. Camera and floor are both fixed, so frames copy this."""
+ import cv2
+
+ img = np.full((height, width, 3), theme.bg, dtype=np.uint8)
+
+ quad = cam.clip_near(ground.corners())
+ if len(quad) >= 3:
+ poly = np.rint(cam.project(quad)).astype(np.int32)
+ mask = np.zeros((height, width), dtype=np.uint8)
+ cv2.fillPoly(mask, [poly], 255, cv2.LINE_8)
+ top, bottom = float(poly[:, 1].min()), float(poly[:, 1].max())
+ t = np.clip((np.arange(height) - top) / max(bottom - top, 1.0), 0.0, 1.0)[:, None]
+ ramp = (np.asarray(theme.floor_far, dtype=np.float64) * (1.0 - t)
+ + np.asarray(theme.floor_near, dtype=np.float64) * t)
+ img = np.where(mask[:, :, None] > 0, ramp[:, None, :], img).astype(np.uint8)
+
+ segs = ground.lines()
+ d = cam.depth(segs.mean(axis=1))
+ span = max(float(np.ptp(d)), 1e-6)
+ drawn: list[float] = []
+ for i in np.argsort(-d):
+ clipped = cam.clip_segment(segs[i, 0], segs[i, 1])
+ if clipped is None:
+ continue
+ a, b = (cam.project(p)[0] for p in clipped)
+ if max(abs(a[0] - b[0]), abs(a[1] - b[1])) < 3.0:
+ continue
+ # Lines crowding the vanishing point would fuse into a solid band.
+ if abs(a[1] - b[1]) < 2.0:
+ row = 0.5 * (a[1] + b[1])
+ if any(abs(row - r) < 2.0 for r in drawn):
+ continue
+ drawn.append(row)
+ near = float(np.clip((d[i] - d.min()) / span, 0.0, 1.0))
+ far = np.asarray(theme.floor_far, dtype=np.float64)
+ colour = far + (np.asarray(theme.grid, dtype=np.float64) - far) * (0.35 + 0.65 * near)
+ cv2.line(img, tuple(np.rint(a).astype(int)), tuple(np.rint(b).astype(int)),
+ tuple(int(round(v)) for v in colour), 1, cv2.LINE_AA)
+ return img
+
+
+def draw_shadow(img: np.ndarray, cam, pts: np.ndarray, ground: Ground,
+ theme: Theme = DARK_THEME) -> None:
+ """The figure flattened onto the floor: an overhead light's cast shadow."""
+ import cv2
+
+ flat = np.asarray(pts, dtype=np.float64).reshape(-1, 3).copy()
+ flat[:, 1] = ground.y
+ hull = cv2.convexHull(np.rint(cam.project(flat)).astype(np.int32))
+ mask = np.zeros(img.shape[:2], dtype=np.uint8)
+ cv2.fillConvexPoly(mask, hull, 255, cv2.LINE_8)
+ mask = cv2.GaussianBlur(mask, (0, 0), 9)
+ m = (mask.astype(np.float64) / 255.0 * theme.shadow_alpha)[:, :, None]
+ blended = (img.astype(np.float64) * (1.0 - m)
+ + np.asarray(theme.shadow, dtype=np.float64) * m)
+ img[:] = np.clip(blended, 0, 255).astype(np.uint8)
+
+
+def draw_skeleton(
+ img: np.ndarray,
+ pos: dict[str, np.ndarray],
+ edges: list[tuple[str, str]],
+ cam,
+ thickness: int = 3,
+ joint_bgr: tuple[int, int, int] = (242, 242, 242),
+) -> None:
+ import cv2
+
+ names = list(pos.keys())
+ uv = cam.project(np.stack([pos[n] for n in names]))
+ px = {n: (int(round(uv[i, 0])), int(round(uv[i, 1]))) for i, n in enumerate(names)}
+ for a, b in edges:
+ if a in px and b in px:
+ cv2.line(img, px[a], px[b], edge_bgr(a, b), thickness, cv2.LINE_AA)
+ for p in px.values():
+ cv2.circle(img, p, 3, joint_bgr, -1, cv2.LINE_AA)
+
+
+def draw_mesh(
+ img: np.ndarray,
+ vertices: list[np.ndarray],
+ faces: list[np.ndarray],
+ colors: list[np.ndarray],
+ cam: Camera,
+ ambient: float = 0.30,
+) -> None:
+ """Back-to-front painter fill with Lambert shading from a head-on key light.
+
+ A colour entry is either one BGR for the whole mesh or one per triangle.
+ `ambient` is the floor of the shading term: on a near-black texture the lit
+ side already sits low, so too little of it crushes the whole part to black.
+ """
+ import cv2
+
+ polys: list[np.ndarray] = []
+ shades: list[np.ndarray] = []
+ depths: list[np.ndarray] = []
+ light = -cam.look
+
+ for verts, face, base in zip(vertices, faces, colors):
+ uv = cam.project(verts)
+ z = cam.depth(verts)
+ v0, v1, v2 = face[:, 0], face[:, 1], face[:, 2]
+ p0, p1, p2 = verts[v0], verts[v1], verts[v2]
+ normals = np.cross(p1 - p0, p2 - p0)
+ nlen = np.linalg.norm(normals, axis=1, keepdims=True)
+ normals = normals / np.where(nlen < 1e-12, 1.0, nlen)
+ facing = normals @ light
+ keep = facing > 0.0
+ if not np.any(keep):
+ continue
+ lambert = np.clip(np.abs(facing[keep]), 0.0, 1.0)
+ shade = ambient + (1.0 - ambient) * lambert
+ tri = np.stack([uv[v0[keep]], uv[v1[keep]], uv[v2[keep]]], axis=1)
+ polys.append(np.rint(tri).astype(np.int32))
+ tint = np.asarray(base, dtype=np.float64)
+ tint = tint[None, :] if tint.ndim == 1 else tint[keep]
+ shades.append(shade[:, None] * tint)
+ depths.append((z[v0[keep]] + z[v1[keep]] + z[v2[keep]]) / 3.0)
+
+ if not polys:
+ return
+ tri = np.concatenate(polys, axis=0)
+ rgb = np.clip(np.concatenate(shades, axis=0), 0, 255)
+ depth = np.concatenate(depths, axis=0)
+
+ order = np.argsort(-depth)
+ tri = tri[order]
+ rgb = rgb[order].astype(np.int32)
+ for i in range(tri.shape[0]):
+ cv2.fillConvexPoly(img, tri[i], (int(rgb[i, 0]), int(rgb[i, 1]), int(rgb[i, 2])), cv2.LINE_8)
+
+
+def label_panel(img: np.ndarray, text: str, *, flagged: bool = False, height: int = 32) -> None:
+ import cv2
+
+ w = img.shape[1]
+ cv2.rectangle(img, (0, 0), (w, height), PANEL_BAR_BGR, -1)
+ color = FLAG_BGR if flagged else TEXT_BGR
+ cv2.putText(img, text, (10, height - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.55, color, 1, cv2.LINE_AA)
+
+
+def footer(img: np.ndarray, text: str, height: int = 28) -> None:
+ import cv2
+
+ h, w = img.shape[:2]
+ cv2.rectangle(img, (0, h - height), (w, h), PANEL_BAR_BGR, -1)
+ cv2.putText(img, text, (10, h - 9), cv2.FONT_HERSHEY_SIMPLEX, 0.48, MUTED_BGR, 1, cv2.LINE_AA)
+
+
+def hstack_panels(panels: list[np.ndarray]) -> np.ndarray:
+ import cv2
+
+ grid = np.hstack(panels)
+ x = 0
+ for panel in panels[:-1]:
+ x += panel.shape[1]
+ cv2.line(grid, (x, 0), (x, grid.shape[0]), (40, 40, 44), 1)
+ return grid
+
+
+def fit_letterbox(frame: np.ndarray, width: int, height: int) -> np.ndarray:
+ import cv2
+
+ out = np.full((height, width, 3), BG_BGR, dtype=np.uint8)
+ if frame is None:
+ return out
+ h, w = frame.shape[:2]
+ s = min(width / w, height / h)
+ nw, nh = max(1, int(round(w * s))), max(1, int(round(h * s)))
+ resized = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_AREA)
+ ox, oy = (width - nw) // 2, (height - nh) // 2
+ out[oy:oy + nh, ox:ox + nw] = resized
+ return out
+
+
+COVER_SHIFT_X = 0.0
+
+
+def cover_crop_box(
+ src_w: int, src_h: int, width: int, height: int, shift_x: float = 0.0
+) -> tuple[int, int, int, int]:
+ """Crop of a src_w x src_h frame carrying the panel's aspect: x, y, w, h.
+
+ shift_x slides the window by that fraction of its own width, negative = left,
+ clamped so the window stays inside the frame.
+ """
+ crop_w = min(src_w, max(1, int(round(src_h * width / height))))
+ crop_h = min(src_h, max(1, int(round(crop_w * height / width))))
+ x0 = (src_w - crop_w) // 2 + int(round(shift_x * crop_w))
+ y0 = (src_h - crop_h) // 2
+ return max(0, min(src_w - crop_w, x0)), max(0, min(src_h - crop_h, y0)), crop_w, crop_h
+
+
+def fit_cover(
+ frame: np.ndarray, width: int, height: int, *, shift_x: float = COVER_SHIFT_X
+) -> np.ndarray:
+ """Crop to the panel's aspect, then fill it edge to edge.
+
+ Letterboxing a 16:9 clip into a tall panel spends most of the panel on bars
+ and leaves the dancer small next to the rendered panels beside it.
+ """
+ import cv2
+
+ out = np.full((height, width, 3), BG_BGR, dtype=np.uint8)
+ if frame is None:
+ return out
+ h, w = frame.shape[:2]
+ x0, y0, crop_w, crop_h = cover_crop_box(w, h, width, height, shift_x)
+ crop = frame[y0:y0 + crop_h, x0:x0 + crop_w]
+ return cv2.resize(crop, (width, height), interpolation=cv2.INTER_AREA)
+
+
+def write_video(path, frames_bgr: list[np.ndarray], fps: float) -> None:
+ import cv2
+
+ path.parent.mkdir(parents=True, exist_ok=True)
+ h, w = frames_bgr[0].shape[:2]
+ writer = cv2.VideoWriter(str(path), cv2.VideoWriter_fourcc(*"mp4v"), float(fps), (w, h))
+ if not writer.isOpened():
+ raise RuntimeError(f"Cannot open VideoWriter for {path}")
+ for frame in frames_bgr:
+ writer.write(frame)
+ writer.release()
+
+
+def encode_h264(src, dst) -> bool:
+ """Re-encode with ffmpeg when available so the clip plays in browsers."""
+ import shutil
+ import subprocess
+
+ if shutil.which("ffmpeg") is None:
+ return False
+ cmd = [
+ "ffmpeg", "-y", "-loglevel", "error",
+ "-i", str(src),
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "20",
+ str(dst),
+ ]
+ return subprocess.run(cmd, check=False).returncode == 0
diff --git a/motion2mixamorig/mixamo/retarget.py b/motion2mixamorig/mixamo/retarget.py
new file mode 100644
index 0000000..9ee1a1e
--- /dev/null
+++ b/motion2mixamorig/mixamo/retarget.py
@@ -0,0 +1,524 @@
+"""Human → Mixamo per-frame retarget.
+
+FK is T * Rpre * R_anim. Mixamo rest bone lengths are never scaled.
+Optional previous-frame rotations pick the 180-degree twist closest to the
+last frame and enforce quaternion sign continuity. No temporal smoothing.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+import numpy as np
+
+from .kinematics import (
+ MixamoRig,
+ anim_from_world_rotation,
+ continue_quaternion,
+ forward_kinematics,
+ frame_anim_from_axes,
+ frame_anim_from_two_vectors,
+ human_to_mixamo,
+ local_aim_axis,
+ normalize,
+ orthonormal_frame,
+ pick_closest_aim_twist,
+ quat_wxyz,
+ swing_anim_to_direction,
+ world_positions_m,
+)
+
+CONTROLLED = (
+ "mixamorig:Hips",
+ "mixamorig:Spine",
+ "mixamorig:Spine1",
+ "mixamorig:Spine2",
+ "mixamorig:Neck",
+ "mixamorig:Head",
+ "mixamorig:LeftShoulder",
+ "mixamorig:RightShoulder",
+ "mixamorig:LeftArm",
+ "mixamorig:RightArm",
+ "mixamorig:LeftForeArm",
+ "mixamorig:RightForeArm",
+ "mixamorig:LeftHand",
+ "mixamorig:RightHand",
+ "mixamorig:LeftUpLeg",
+ "mixamorig:RightUpLeg",
+ "mixamorig:LeftLeg",
+ "mixamorig:RightLeg",
+ "mixamorig:LeftFoot",
+ "mixamorig:RightFoot",
+ "mixamorig:LeftToeBase",
+ "mixamorig:RightToeBase",
+)
+
+# Mixamo bone → (human parent joint, human child joint) used as the aim vector.
+# Spine2 uses a chest frame (not a single swing) so clavicle attachments keep the right twist.
+#
+# Neck/Head must not aim along SMPL neck→head. That vector is a short (~9 cm)
+# skull-center offset, often 50–70° from vertical, and flops Mixamo's longer
+# neck/head bones. spine3→head is the stable head-up (skull above the chest)
+# while still using the Human head joint. Joint mapping remains
+# neck→mixamorig:Neck and head→mixamorig:Head.
+#
+# Mixamo Neck→Head and Head→HeadTop rest translations are not along local +Y
+# (they include a forward +Z offset). Aiming that offset at head-up tilts the
+# neck/head bones. Neck/Head therefore swing local +Y.
+NECK_HEAD = ("mixamorig:Neck", "mixamorig:Head")
+NECK_HEAD_AIM_LOCAL = np.array([0.0, 1.0, 0.0])
+
+SWING_TARGETS = {
+ "mixamorig:Spine": ("spine1", "spine2"),
+ "mixamorig:Spine1": ("spine2", "spine3"),
+ "mixamorig:Neck": ("spine3", "head"),
+ "mixamorig:Head": ("spine3", "head"),
+ "mixamorig:LeftShoulder": ("left_collar", "left_shoulder"),
+ "mixamorig:LeftArm": ("left_shoulder", "left_elbow"),
+ "mixamorig:LeftForeArm": ("left_elbow", "left_wrist"),
+ "mixamorig:RightShoulder": ("right_collar", "right_shoulder"),
+ "mixamorig:RightArm": ("right_shoulder", "right_elbow"),
+ "mixamorig:RightForeArm": ("right_elbow", "right_wrist"),
+ "mixamorig:LeftUpLeg": ("left_hip", "left_knee"),
+ "mixamorig:LeftLeg": ("left_knee", "left_ankle"),
+ "mixamorig:RightUpLeg": ("right_hip", "right_knee"),
+ "mixamorig:RightLeg": ("right_knee", "right_ankle"),
+}
+
+SWING_TARGETS_NO_CLAVICLE = {
+ "mixamorig:Spine": ("spine1", "spine2"),
+ "mixamorig:Spine1": ("spine2", "spine3"),
+ "mixamorig:Spine2": ("spine3", "neck"),
+ "mixamorig:LeftArm": ("left_shoulder", "left_elbow"),
+ "mixamorig:LeftForeArm": ("left_elbow", "left_wrist"),
+ "mixamorig:RightArm": ("right_shoulder", "right_elbow"),
+ "mixamorig:RightForeArm": ("right_elbow", "right_wrist"),
+}
+
+# Child that defines this bone's rest axis (local +Y after PreRotation).
+AIM_CHILD = {
+ "mixamorig:Spine": "mixamorig:Spine1",
+ "mixamorig:Spine1": "mixamorig:Spine2",
+ "mixamorig:Spine2": "mixamorig:Neck",
+ "mixamorig:Neck": "mixamorig:Head",
+ "mixamorig:Head": "mixamorig:HeadTop_End",
+ "mixamorig:LeftShoulder": "mixamorig:LeftArm",
+ "mixamorig:LeftArm": "mixamorig:LeftForeArm",
+ "mixamorig:LeftForeArm": "mixamorig:LeftHand",
+ "mixamorig:RightShoulder": "mixamorig:RightArm",
+ "mixamorig:RightArm": "mixamorig:RightForeArm",
+ "mixamorig:RightForeArm": "mixamorig:RightHand",
+ "mixamorig:LeftUpLeg": "mixamorig:LeftLeg",
+ "mixamorig:LeftLeg": "mixamorig:LeftFoot",
+ "mixamorig:LeftFoot": "mixamorig:LeftToeBase",
+ "mixamorig:LeftToeBase": "mixamorig:LeftToe_End",
+ "mixamorig:RightUpLeg": "mixamorig:RightLeg",
+ "mixamorig:RightLeg": "mixamorig:RightFoot",
+ "mixamorig:RightFoot": "mixamorig:RightToeBase",
+ "mixamorig:RightToeBase": "mixamorig:RightToe_End",
+}
+
+
+def swing_aim_local(rig: MixamoRig, bone_name: str) -> np.ndarray:
+ """Rest axis that swing_anim_to_direction aligns with the Human target."""
+ if bone_name in NECK_HEAD:
+ return NECK_HEAD_AIM_LOCAL.copy()
+ return rig.child_offset_cm(bone_name, AIM_CHILD[bone_name])
+
+
+MIXAMO_TO_HUMAN = {
+ "mixamorig:Hips": "pelvis",
+ "mixamorig:Spine": "spine1",
+ "mixamorig:Spine1": "spine2",
+ "mixamorig:Spine2": "spine3",
+ "mixamorig:Neck": "neck",
+ "mixamorig:Head": "head",
+ "mixamorig:LeftShoulder": "left_collar",
+ "mixamorig:LeftArm": "left_shoulder",
+ "mixamorig:LeftForeArm": "left_elbow",
+ "mixamorig:LeftHand": "left_wrist",
+ "mixamorig:RightShoulder": "right_collar",
+ "mixamorig:RightArm": "right_shoulder",
+ "mixamorig:RightForeArm": "right_elbow",
+ "mixamorig:RightHand": "right_wrist",
+ "mixamorig:LeftUpLeg": "left_hip",
+ "mixamorig:LeftLeg": "left_knee",
+ "mixamorig:LeftFoot": "left_ankle",
+ "mixamorig:LeftToeBase": "left_foot",
+ "mixamorig:RightUpLeg": "right_hip",
+ "mixamorig:RightLeg": "right_knee",
+ "mixamorig:RightFoot": "right_ankle",
+ "mixamorig:RightToeBase": "right_foot",
+}
+
+ARM_POSITION_MARKERS = {
+ "left_elbow": "mixamorig:LeftForeArm",
+ "left_wrist": "mixamorig:LeftHand",
+ "right_elbow": "mixamorig:RightForeArm",
+ "right_wrist": "mixamorig:RightHand",
+}
+
+LEG_POSITION_MARKERS = {
+ "left_hip": "mixamorig:LeftUpLeg",
+ "left_knee": "mixamorig:LeftLeg",
+ "left_ankle": "mixamorig:LeftFoot",
+ "right_hip": "mixamorig:RightUpLeg",
+ "right_knee": "mixamorig:RightLeg",
+ "right_ankle": "mixamorig:RightFoot",
+}
+
+LEG_BONES = (
+ "mixamorig:LeftUpLeg",
+ "mixamorig:LeftLeg",
+ "mixamorig:LeftFoot",
+ "mixamorig:LeftToeBase",
+ "mixamorig:LeftToe_End",
+ "mixamorig:RightUpLeg",
+ "mixamorig:RightLeg",
+ "mixamorig:RightFoot",
+ "mixamorig:RightToeBase",
+ "mixamorig:RightToe_End",
+)
+
+WORLD_UP = np.array([0.0, 1.0, 0.0])
+HAND_LOCAL_AXIS = np.array([0.0, 1.0, 0.0])
+
+MIXAMO_EDGES = [
+ ("mixamorig:Hips", "mixamorig:Spine"),
+ ("mixamorig:Spine", "mixamorig:Spine1"),
+ ("mixamorig:Spine1", "mixamorig:Spine2"),
+ ("mixamorig:Spine2", "mixamorig:Neck"),
+ ("mixamorig:Neck", "mixamorig:Head"),
+ ("mixamorig:Head", "mixamorig:HeadTop_End"),
+ ("mixamorig:Spine2", "mixamorig:LeftShoulder"),
+ ("mixamorig:LeftShoulder", "mixamorig:LeftArm"),
+ ("mixamorig:LeftArm", "mixamorig:LeftForeArm"),
+ ("mixamorig:LeftForeArm", "mixamorig:LeftHand"),
+ ("mixamorig:Spine2", "mixamorig:RightShoulder"),
+ ("mixamorig:RightShoulder", "mixamorig:RightArm"),
+ ("mixamorig:RightArm", "mixamorig:RightForeArm"),
+ ("mixamorig:RightForeArm", "mixamorig:RightHand"),
+ ("mixamorig:Hips", "mixamorig:LeftUpLeg"),
+ ("mixamorig:LeftUpLeg", "mixamorig:LeftLeg"),
+ ("mixamorig:LeftLeg", "mixamorig:LeftFoot"),
+ ("mixamorig:LeftFoot", "mixamorig:LeftToeBase"),
+ ("mixamorig:LeftToeBase", "mixamorig:LeftToe_End"),
+ ("mixamorig:Hips", "mixamorig:RightUpLeg"),
+ ("mixamorig:RightUpLeg", "mixamorig:RightLeg"),
+ ("mixamorig:RightLeg", "mixamorig:RightFoot"),
+ ("mixamorig:RightFoot", "mixamorig:RightToeBase"),
+ ("mixamorig:RightToeBase", "mixamorig:RightToe_End"),
+]
+
+HUMAN_EDGES = [
+ ("pelvis", "spine1"),
+ ("spine1", "spine2"),
+ ("spine2", "spine3"),
+ ("spine3", "neck"),
+ ("neck", "head"),
+ ("spine3", "left_collar"),
+ ("left_collar", "left_shoulder"),
+ ("left_shoulder", "left_elbow"),
+ ("left_elbow", "left_wrist"),
+ ("left_wrist", "left_hand"),
+ ("spine3", "right_collar"),
+ ("right_collar", "right_shoulder"),
+ ("right_shoulder", "right_elbow"),
+ ("right_elbow", "right_wrist"),
+ ("right_wrist", "right_hand"),
+ ("pelvis", "left_hip"),
+ ("left_hip", "left_knee"),
+ ("left_knee", "left_ankle"),
+ ("left_ankle", "left_foot"),
+ ("pelvis", "right_hip"),
+ ("right_hip", "right_knee"),
+ ("right_knee", "right_ankle"),
+ ("right_ankle", "right_foot"),
+]
+
+LENGTH_EDGES = [
+ ("mixamorig:Spine2", "mixamorig:LeftShoulder"),
+ ("mixamorig:LeftShoulder", "mixamorig:LeftArm"),
+ ("mixamorig:LeftArm", "mixamorig:LeftForeArm"),
+ ("mixamorig:LeftForeArm", "mixamorig:LeftHand"),
+ ("mixamorig:Spine2", "mixamorig:RightShoulder"),
+ ("mixamorig:RightShoulder", "mixamorig:RightArm"),
+ ("mixamorig:RightArm", "mixamorig:RightForeArm"),
+ ("mixamorig:RightForeArm", "mixamorig:RightHand"),
+ ("mixamorig:Hips", "mixamorig:Spine"),
+ ("mixamorig:Spine", "mixamorig:Spine1"),
+ ("mixamorig:Spine1", "mixamorig:Spine2"),
+ ("mixamorig:Spine2", "mixamorig:Neck"),
+ ("mixamorig:Neck", "mixamorig:Head"),
+ ("mixamorig:Head", "mixamorig:HeadTop_End"),
+ ("mixamorig:Hips", "mixamorig:LeftUpLeg"),
+ ("mixamorig:LeftUpLeg", "mixamorig:LeftLeg"),
+ ("mixamorig:LeftLeg", "mixamorig:LeftFoot"),
+ ("mixamorig:LeftFoot", "mixamorig:LeftToeBase"),
+ ("mixamorig:Hips", "mixamorig:RightUpLeg"),
+ ("mixamorig:RightUpLeg", "mixamorig:RightLeg"),
+ ("mixamorig:RightLeg", "mixamorig:RightFoot"),
+ ("mixamorig:RightFoot", "mixamorig:RightToeBase"),
+]
+
+
+@dataclass
+class FrameRetarget:
+ r_anim: dict[str, np.ndarray]
+ q_anim: dict[str, np.ndarray]
+ r_anim_raw: dict[str, np.ndarray] = field(default_factory=dict)
+ twist_flips: list[str] = field(default_factory=list)
+ sign_flips: list[str] = field(default_factory=list)
+
+
+def joint_mixamo(motion, frame: int, name: str) -> np.ndarray:
+ return human_to_mixamo(motion.joint(frame, name))
+
+
+def human_dir(motion, frame: int, start: str, end: str) -> np.ndarray:
+ return joint_mixamo(motion, frame, end) - joint_mixamo(motion, frame, start)
+
+
+def verify_rest_fk(rig: MixamoRig, atol: float = 1e-4) -> float:
+ """Identity R_anim must reproduce the FBX rest pose (PreRotation included)."""
+ world = forward_kinematics(rig, {})
+ err = 0.0
+ for name, bone in rig.bones.items():
+ err = max(err, float(np.linalg.norm(world[name] - bone.rest_world)))
+ if err > atol:
+ raise RuntimeError(f"rest FK mismatch {err:.4g} > {atol} — PreRotation composition is wrong")
+ return err
+
+
+def mixamo_proportioned_from(
+ rig: MixamoRig,
+ motion,
+ frame: int,
+ rest_pos: dict[str, np.ndarray],
+ root_name: str,
+ root_pos_m: np.ndarray,
+) -> dict[str, np.ndarray]:
+ """Rebuild descendants of root_name with Human directions and Mixamo rest lengths."""
+ target: dict[str, np.ndarray] = {root_name: np.asarray(root_pos_m, dtype=np.float64).copy()}
+ for name in rig.order:
+ if name == root_name:
+ continue
+ parent = rig.bones[name].parent
+ if parent is None or parent not in target:
+ continue
+ length = float(np.linalg.norm(rest_pos[name] - rest_pos[parent]))
+ if name in MIXAMO_TO_HUMAN and parent in MIXAMO_TO_HUMAN:
+ direction = human_dir(motion, frame, MIXAMO_TO_HUMAN[parent], MIXAMO_TO_HUMAN[name])
+ else:
+ direction = rest_pos[name] - rest_pos[parent]
+ target[name] = target[parent] + normalize(direction) * length
+ return target
+
+
+def position_errors(
+ posed: dict[str, np.ndarray],
+ target: dict[str, np.ndarray],
+ markers: dict[str, str],
+) -> dict[str, dict[str, float]]:
+ out: dict[str, dict[str, float]] = {}
+ for label, bone in markers.items():
+ delta = posed[bone] - target[bone]
+ out[label] = {
+ "error_m": float(np.linalg.norm(delta)),
+ "dx": float(delta[0]),
+ "dy": float(delta[1]),
+ "dz": float(delta[2]),
+ "mixamo_bone": bone,
+ }
+ return out
+
+
+def _hips_rotation(rig: MixamoRig, motion, frame: int) -> np.ndarray:
+ rest_world = forward_kinematics(rig, {})
+ rest_pos = world_positions_m(rig, rest_world)
+ r_rest_frame = orthonormal_frame(
+ rest_pos["mixamorig:LeftUpLeg"] - rest_pos["mixamorig:RightUpLeg"],
+ rest_pos["mixamorig:Neck"] - rest_pos["mixamorig:Hips"],
+ )
+ r_tgt_frame = orthonormal_frame(
+ joint_mixamo(motion, frame, "left_hip") - joint_mixamo(motion, frame, "right_hip"),
+ joint_mixamo(motion, frame, "neck") - joint_mixamo(motion, frame, "pelvis"),
+ )
+ r_hips_world = (r_tgt_frame @ r_rest_frame.T) @ rest_world["mixamorig:Hips"][:3, :3]
+ return anim_from_world_rotation(rig.bones["mixamorig:Hips"], np.eye(3), r_hips_world)
+
+
+def _spine2_chest_rotation(rig: MixamoRig, world: dict[str, np.ndarray], motion, frame: int) -> np.ndarray:
+ bone = rig.bones["mixamorig:Spine2"]
+ parent_r = world[bone.parent][:3, :3] if bone.parent else np.eye(3)
+ rest_left = rig.child_offset_cm("mixamorig:Spine2", "mixamorig:LeftShoulder") - rig.child_offset_cm(
+ "mixamorig:Spine2", "mixamorig:RightShoulder"
+ )
+ rest_up = rig.child_offset_cm("mixamorig:Spine2", "mixamorig:Neck")
+ return frame_anim_from_axes(
+ bone,
+ parent_r,
+ rest_left,
+ rest_up,
+ joint_mixamo(motion, frame, "left_collar") - joint_mixamo(motion, frame, "right_collar"),
+ joint_mixamo(motion, frame, "neck") - joint_mixamo(motion, frame, "spine3"),
+ )
+
+
+def _foot_rotation(rig: MixamoRig, world: dict[str, np.ndarray], motion, frame: int, side: str) -> np.ndarray:
+ foot = f"mixamorig:{side}Foot"
+ toe = f"mixamorig:{side}ToeBase"
+ bone = rig.bones[foot]
+ parent_r = world[bone.parent][:3, :3] if bone.parent else np.eye(3)
+ along_local = rig.child_offset_cm(foot, toe)
+ r_rest = parent_r @ bone.pre_rotation
+ up_local = r_rest.T @ WORLD_UP
+ ankle = "left_ankle" if side == "Left" else "right_ankle"
+ foot_j = "left_foot" if side == "Left" else "right_foot"
+ knee = "left_knee" if side == "Left" else "right_knee"
+ tgt_along = human_dir(motion, frame, ankle, foot_j)
+ tgt_up = WORLD_UP.copy()
+ if abs(float(normalize(tgt_along) @ normalize(tgt_up))) > 0.95:
+ tgt_up = human_dir(motion, frame, knee, ankle)
+ if bone.parent:
+ up_local = r_rest.T @ (parent_r @ rig.child_offset_cm(bone.parent, foot))
+ return frame_anim_from_two_vectors(
+ bone,
+ parent_r,
+ along_local,
+ up_local,
+ tgt_along,
+ tgt_up,
+ )
+
+
+def _hand_rotation(rig: MixamoRig, world: dict[str, np.ndarray], motion, frame: int, side: str) -> np.ndarray:
+ """Aim Mixamo Hand local +Y along Human wrist→hand. No finger bones in the core rig."""
+ hand = f"mixamorig:{side}Hand"
+ wrist_j = "left_wrist" if side == "Left" else "right_wrist"
+ palm_j = "left_hand" if side == "Left" else "right_hand"
+ tgt = human_dir(motion, frame, wrist_j, palm_j)
+ if float(np.linalg.norm(tgt)) < 1e-4:
+ return np.eye(3)
+ return swing_anim_to_direction(rig, world, hand, None, tgt, d_local=HAND_LOCAL_AXIS)
+
+
+# Bones whose rotation is built from two independent human directions (or is
+# constant). They are fully determined by the current frame's data, so there is
+# no twist ambiguity for the continuity rule to resolve. Letting the rule
+# prefer the 180-degree twist variant anyway lets one transient glitch in the
+# extracted data latch permanently: once the previous frame is in the flipped
+# state, the flipped variant of every later (correct) rotation stays closest
+# to it, and the character finishes the clip facing backwards.
+FULLY_DETERMINED = frozenset({
+ "mixamorig:Hips",
+ "mixamorig:Spine2",
+ "mixamorig:LeftFoot",
+ "mixamorig:RightFoot",
+ "mixamorig:LeftToeBase",
+ "mixamorig:RightToeBase",
+})
+
+
+def retarget_one_frame(
+ rig: MixamoRig,
+ motion,
+ frame: int,
+ *,
+ clavicles: bool = True,
+ legs: bool = True,
+ prev_r_anim: dict[str, np.ndarray] | None = None,
+ prev_q_anim: dict[str, np.ndarray] | None = None,
+) -> FrameRetarget:
+ hips_t_cm = joint_mixamo(motion, frame, "pelvis") / rig.meters_per_unit
+ r_anim = {name: np.eye(3) for name in rig.order}
+ q_anim = {name: np.array([1.0, 0.0, 0.0, 0.0]) for name in rig.order}
+ twist_flips: list[str] = []
+ sign_flips: list[str] = []
+
+ def fk() -> dict[str, np.ndarray]:
+ return forward_kinematics(rig, r_anim, hips_translation_cm=hips_t_cm)
+
+ def commit(name: str, rotation: np.ndarray) -> None:
+ q_prev = None if prev_q_anim is None else prev_q_anim.get(name)
+ if name in FULLY_DETERMINED:
+ # Trust the data over history; only the aim-ambiguous bones below
+ # get the anti-twist continuity.
+ r_pick, did_twist = np.asarray(rotation, dtype=np.float64), False
+ else:
+ axis = swing_aim_local(rig, name) if name in AIM_CHILD or name in NECK_HEAD else local_aim_axis(rig, name)
+ r_prev = None if prev_r_anim is None else prev_r_anim.get(name)
+ r_pick, did_twist = pick_closest_aim_twist(rotation, r_prev, axis)
+ q = quat_wxyz(r_pick)
+ q, did_sign = continue_quaternion(q, q_prev)
+ r_anim[name] = r_pick
+ q_anim[name] = q
+ if did_twist:
+ twist_flips.append(name)
+ if did_sign:
+ sign_flips.append(name)
+
+ def apply_swings(targets: dict[str, tuple[str, str]]) -> None:
+ for bone_name, (h_a, h_b) in targets.items():
+ rotation = swing_anim_to_direction(
+ rig,
+ fk(),
+ bone_name,
+ AIM_CHILD.get(bone_name),
+ human_dir(motion, frame, h_a, h_b),
+ d_local=swing_aim_local(rig, bone_name),
+ )
+ commit(bone_name, rotation)
+
+ commit("mixamorig:Hips", _hips_rotation(rig, motion, frame))
+ if clavicles:
+ apply_swings(
+ {
+ "mixamorig:Spine": SWING_TARGETS["mixamorig:Spine"],
+ "mixamorig:Spine1": SWING_TARGETS["mixamorig:Spine1"],
+ }
+ )
+ commit("mixamorig:Spine2", _spine2_chest_rotation(rig, fk(), motion, frame))
+ apply_swings(
+ {
+ "mixamorig:Neck": SWING_TARGETS["mixamorig:Neck"],
+ "mixamorig:Head": SWING_TARGETS["mixamorig:Head"],
+ "mixamorig:LeftShoulder": SWING_TARGETS["mixamorig:LeftShoulder"],
+ "mixamorig:LeftArm": SWING_TARGETS["mixamorig:LeftArm"],
+ "mixamorig:LeftForeArm": SWING_TARGETS["mixamorig:LeftForeArm"],
+ "mixamorig:RightShoulder": SWING_TARGETS["mixamorig:RightShoulder"],
+ "mixamorig:RightArm": SWING_TARGETS["mixamorig:RightArm"],
+ "mixamorig:RightForeArm": SWING_TARGETS["mixamorig:RightForeArm"],
+ }
+ )
+ commit("mixamorig:LeftHand", _hand_rotation(rig, fk(), motion, frame, "Left"))
+ commit("mixamorig:RightHand", _hand_rotation(rig, fk(), motion, frame, "Right"))
+ else:
+ apply_swings(SWING_TARGETS_NO_CLAVICLE)
+ apply_swings(
+ {
+ "mixamorig:Neck": SWING_TARGETS["mixamorig:Neck"],
+ "mixamorig:Head": SWING_TARGETS["mixamorig:Head"],
+ }
+ )
+ if legs:
+ apply_swings(
+ {
+ "mixamorig:LeftUpLeg": SWING_TARGETS["mixamorig:LeftUpLeg"],
+ "mixamorig:LeftLeg": SWING_TARGETS["mixamorig:LeftLeg"],
+ "mixamorig:RightUpLeg": SWING_TARGETS["mixamorig:RightUpLeg"],
+ "mixamorig:RightLeg": SWING_TARGETS["mixamorig:RightLeg"],
+ }
+ )
+ commit("mixamorig:LeftFoot", _foot_rotation(rig, fk(), motion, frame, "Left"))
+ commit("mixamorig:LeftToeBase", np.eye(3))
+ commit("mixamorig:RightFoot", _foot_rotation(rig, fk(), motion, frame, "Right"))
+ commit("mixamorig:RightToeBase", np.eye(3))
+ r_raw = {name: np.asarray(rot, dtype=np.float64).copy() for name, rot in r_anim.items()}
+ return FrameRetarget(
+ r_anim=r_anim,
+ q_anim=q_anim,
+ r_anim_raw=r_raw,
+ twist_flips=twist_flips,
+ sign_flips=sign_flips,
+ )
diff --git a/motion2mixamorig/mixamo/skinned_mesh.py b/motion2mixamorig/mixamo/skinned_mesh.py
new file mode 100644
index 0000000..6ae9f73
--- /dev/null
+++ b/motion2mixamorig/mixamo/skinned_mesh.py
@@ -0,0 +1,543 @@
+"""Load Mixamo character meshes + skin clusters from binary FBX.
+
+Does not change retargeting. FK is the existing T * Rpre * R_anim chain;
+finger bones stay at rest (identity R_anim) and follow the hands.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from pathlib import Path
+
+import numpy as np
+
+from .fbx_skeleton import (
+ _child_named,
+ bone_local_matrix,
+ build_hierarchy,
+ collect_limb_nodes,
+ compose_world,
+ euler_xyz_deg_to_matrix,
+ extract_connections,
+ extract_global_settings,
+ extract_objects,
+ parse_fbx,
+ unit_to_meters,
+)
+from .kinematics import MixamoBone, MixamoRig, forward_kinematics
+
+
+def _fbx_name(raw: str) -> str:
+ return str(raw).split("\x00", 1)[0]
+
+
+def _mat4_fbx(values) -> np.ndarray:
+ return np.asarray(values, dtype=np.float64).reshape(4, 4).T
+
+
+def _triangulate(indices: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ """Fan-triangulate PolygonVertexIndex.
+
+ Returns the vertex ids per triangle, the *flat* polygon-vertex slots they
+ came from, and the polygon each triangle belongs to. The slots index the
+ per-polygon-vertex layers (UV), and the polygon id indexes the per-polygon
+ ones (material), so both can be read back after triangulation.
+ """
+ faces: list[tuple[int, int, int]] = []
+ slots: list[tuple[int, int, int]] = []
+ polys: list[int] = []
+ corner: list[int] = []
+ corner_slot: list[int] = []
+ poly_id = 0
+ for flat, raw in enumerate(indices):
+ value = int(raw)
+ corner_slot.append(flat)
+ if value < 0:
+ corner.append(~value)
+ for i in range(1, len(corner) - 1):
+ faces.append((corner[0], corner[i], corner[i + 1]))
+ slots.append((corner_slot[0], corner_slot[i], corner_slot[i + 1]))
+ polys.append(poly_id)
+ corner = []
+ corner_slot = []
+ poly_id += 1
+ else:
+ corner.append(value)
+ if not faces:
+ return (np.zeros((0, 3), dtype=np.int32),
+ np.zeros((0, 3), dtype=np.int64),
+ np.zeros((0,), dtype=np.int64))
+ return (np.asarray(faces, dtype=np.int32),
+ np.asarray(slots, dtype=np.int64),
+ np.asarray(polys, dtype=np.int64))
+
+
+# Mixamo ships diffuse alongside normal/spec/emission maps under one material,
+# and only the file names tell them apart.
+_DIFFUSE_HINTS = ("diffuse", "albedo", "basecolor", "base_color")
+_OTHER_MAP_HINTS = ("normal", "spec", "gloss", "rough", "metal", "emission",
+ "occlusion", "_ao", "bump", "displace", "opacity", "transparen")
+
+
+def _texture_rank(filename: str) -> int:
+ low = filename.lower()
+ if any(h in low for h in _DIFFUSE_HINTS):
+ return 0
+ if any(h in low for h in _OTHER_MAP_HINTS):
+ return 2
+ return 1
+
+
+def _decode_video(obj: dict) -> "np.ndarray | None":
+ """Decoded texture, keeping any alpha channel.
+
+ Alpha matters: the facial decals and hair/cape cards are masked art painted
+ on an opaque backing, so discarding alpha stamps that backing onto the model.
+ """
+ import cv2
+
+ content = _child_named(obj["raw"], "Content")
+ if content is None or not content["props"]:
+ return None
+ blob = content["props"][0]
+ if not isinstance(blob, (bytes, bytearray)):
+ return None
+ if len(blob) == 0:
+ return None
+ image = cv2.imdecode(np.frombuffer(bytes(blob), dtype=np.uint8), cv2.IMREAD_UNCHANGED)
+ if image is None:
+ return None
+ if image.ndim == 2:
+ image = cv2.cvtColor(image, cv2.COLOR_GRAY2BGR)
+ return image
+
+
+def _videos_by_filename(objects: dict) -> dict[str, list[int]]:
+ """Several Video nodes can share one file with only one holding the bytes."""
+ out: dict[str, list[int]] = {}
+ for uid, obj in objects.items():
+ if obj["node_name"] != "Video":
+ continue
+ node = _child_named(obj["raw"], "RelativeFilename")
+ if node is None or not node["props"]:
+ continue
+ key = _fbx_name(str(node["props"][0]))
+ if key:
+ out.setdefault(key, []).append(uid)
+ return out
+
+
+def _material_textures(objects: dict, connections: list) -> dict[int, list[tuple[int, str]]]:
+ """material uid → its textures as (video uid, filename), best diffuse first."""
+ tex_uids = {u for u, o in objects.items() if o["node_name"] == "Texture"}
+ video_uids = {u for u, o in objects.items() if o["node_name"] == "Video"}
+ mat_uids = {u for u, o in objects.items() if o["node_name"] == "Material"}
+
+ tex_video: dict[int, int] = {}
+ for kind, src, dst in connections:
+ if kind == "OO" and src in video_uids and dst in tex_uids:
+ tex_video[dst] = src
+
+ out: dict[int, list[tuple[int, str]]] = {}
+ for kind, src, dst in connections:
+ if src not in tex_uids or dst not in mat_uids:
+ continue
+ node = _child_named(objects[src]["raw"], "RelativeFilename")
+ filename = _fbx_name(str(node["props"][0])) if node is not None and node["props"] else ""
+ video = tex_video.get(src)
+ if video is None:
+ continue
+ entry = (video, filename)
+ bucket = out.setdefault(dst, [])
+ if entry not in bucket:
+ bucket.append(entry)
+ for uid, bucket in out.items():
+ bucket.sort(key=lambda e: _texture_rank(e[1]))
+ return out
+
+
+def _geometry_materials(objects: dict, connections: list) -> dict[int, list[int]]:
+ """geometry uid → material uids in the order the Model declares them.
+
+ A `ByPolygon` material layer indexes this list, and FBX defines the order by
+ the sequence of material→model connections.
+ """
+ geom_uids = {u for u, o in objects.items() if o["node_name"] == "Geometry"}
+ mat_uids = {u for u, o in objects.items() if o["node_name"] == "Material"}
+ model_uids = {u for u, o in objects.items()
+ if o["node_name"] == "Model" and o.get("class") == "Mesh"}
+
+ geom_model: dict[int, int] = {}
+ model_mats: dict[int, list[int]] = {}
+ for kind, src, dst in connections:
+ if kind != "OO":
+ continue
+ if src in geom_uids and dst in model_uids:
+ geom_model[src] = dst
+ if src in mat_uids and dst in model_uids:
+ model_mats.setdefault(dst, []).append(src)
+ return {g: model_mats.get(m, []) for g, m in geom_model.items()}
+
+
+def _uv_per_slot(geom_raw: dict) -> np.ndarray | None:
+ """(n_polygon_vertex, 2) UVs, resolved through UVIndex when present."""
+ layer = _child_named(geom_raw, "LayerElementUV")
+ if layer is None:
+ return None
+ uv_node = _child_named(layer, "UV")
+ if uv_node is None or not uv_node["props"]:
+ return None
+ uvs = np.asarray(uv_node["props"][0], dtype=np.float64).reshape(-1, 2)
+ idx_node = _child_named(layer, "UVIndex")
+ if idx_node is not None and idx_node["props"]:
+ idx = np.asarray(idx_node["props"][0], dtype=np.int64)
+ idx = np.clip(idx, 0, uvs.shape[0] - 1)
+ return uvs[idx]
+ return uvs
+
+
+def _material_per_polygon(geom_raw: dict, n_poly: int) -> np.ndarray:
+ layer = _child_named(geom_raw, "LayerElementMaterial")
+ if layer is None:
+ return np.zeros(n_poly, dtype=np.int64)
+ arr = _child_named(layer, "Materials")
+ if arr is None or not arr["props"]:
+ return np.zeros(n_poly, dtype=np.int64)
+ mats = np.asarray(arr["props"][0], dtype=np.int64)
+ if mats.size == 1:
+ return np.full(n_poly, int(mats[0]), dtype=np.int64)
+ if mats.size < n_poly:
+ mats = np.pad(mats, (0, n_poly - mats.size), mode="edge")
+ return mats[:n_poly]
+
+
+def _sample(image: np.ndarray, uv: np.ndarray) -> np.ndarray:
+ """Nearest-texel BGR at UV. FBX UV origin is bottom-left, images are top-down.
+
+ UVs are wrapped, not clamped: Mixamo reuses one half of an atlas by offsetting
+ the other half a whole tile in U, so clamping folds every one of those faces
+ onto the texture's edge column.
+ """
+ h, w = image.shape[:2]
+ u = np.mod(uv[:, 0], 1.0)
+ v = np.mod(1.0 - uv[:, 1], 1.0)
+ xs = np.clip((u * (w - 1)).astype(np.int64), 0, w - 1)
+ ys = np.clip((v * (h - 1)).astype(np.int64), 0, h - 1)
+ return image[ys, xs].astype(np.float64)
+
+
+# Barycentric weights of points strictly inside a triangle. One texel at the
+# centroid lands in the black gutter between UV islands often enough to speckle
+# a whole character, so each triangle is probed at several interior points and
+# the median is kept, which ignores a stray sample or two.
+_PROBES = np.array([
+ [1 / 3, 1 / 3, 1 / 3],
+ [0.6, 0.2, 0.2], [0.2, 0.6, 0.2], [0.2, 0.2, 0.6],
+ [0.45, 0.45, 0.10], [0.45, 0.10, 0.45], [0.10, 0.45, 0.45],
+])
+
+
+def _sample_triangles(image: np.ndarray, corners: np.ndarray) -> np.ndarray:
+ """(T, 3, 2) triangle UVs -> (T, C) sample, median over the interior probes."""
+ channels = image.shape[2]
+ stack = np.empty((_PROBES.shape[0], corners.shape[0], channels), dtype=np.float64)
+ for i, w in enumerate(_PROBES):
+ uv = (corners[:, 0] * w[0] + corners[:, 1] * w[1] + corners[:, 2] * w[2])
+ stack[i] = _sample(image, uv)
+ return np.median(stack, axis=0)
+
+
+def face_colors_from_textures(
+ objects: dict,
+ connections: list,
+ geom_uid: int,
+ geom_raw: dict,
+ slots: np.ndarray,
+ poly_ids: np.ndarray,
+ lift: float = 1.0,
+) -> tuple[np.ndarray, np.ndarray] | None:
+ """One BGR per triangle, read from the diffuse map at the UV centroid.
+
+ UVs never move, so this is paid once per character rather than per frame,
+ and the painter fill stays exactly as cheap as a flat colour.
+
+ `lift` below 1 is a gamma on the sampled colour. Several of these characters
+ wear near-black clothing that no amount of ambient light separates from a
+ dark backdrop; the curve raises those without touching the bright end.
+ """
+ uv_slots = _uv_per_slot(geom_raw)
+ if uv_slots is None or slots.shape[0] == 0:
+ return None
+ mats = _geometry_materials(objects, connections).get(geom_uid, [])
+ if not mats:
+ return None
+ textures = _material_textures(objects, connections)
+ shared = _videos_by_filename(objects)
+
+ n_poly = int(poly_ids.max()) + 1 if poly_ids.size else 0
+ poly_mat = _material_per_polygon(geom_raw, n_poly)
+ tri_mat = poly_mat[poly_ids]
+
+ slot_clip = np.clip(slots, 0, uv_slots.shape[0] - 1)
+ corners = np.stack([uv_slots[slot_clip[:, 0]], uv_slots[slot_clip[:, 1]],
+ uv_slots[slot_clip[:, 2]]], axis=1)
+
+ out = np.full((slots.shape[0], 3), np.nan, dtype=np.float64)
+ opacity = np.full(slots.shape[0], 255.0, dtype=np.float64)
+ for local_idx, mat_uid in enumerate(mats):
+ picked = tri_mat == local_idx
+ if not np.any(picked):
+ continue
+ image = None
+ for video_uid, filename in textures.get(mat_uid, []):
+ for candidate in [video_uid, *shared.get(filename, [])]:
+ image = _decode_video(objects[candidate])
+ if image is not None:
+ break
+ if image is not None:
+ break
+ if image is None:
+ continue
+ sampled = _sample_triangles(image, corners[picked])
+ out[picked] = sampled[:, :3]
+ if sampled.shape[1] == 4:
+ opacity[picked] = sampled[:, 3]
+ del image
+ if np.all(np.isnan(out)):
+ return None
+ fallback = np.nanmean(out.reshape(-1, 3), axis=0)
+ bad = np.isnan(out[:, 0])
+ out[bad] = fallback
+ if abs(lift - 1.0) > 1e-9:
+ out = 255.0 * np.power(np.clip(out / 255.0, 0.0, 1.0), lift)
+ return out, opacity >= 128.0
+
+
+@dataclass
+class SkinnedMesh:
+ name: str
+ vertices: np.ndarray # (V, 3) bind pose, FBX cm
+ faces: np.ndarray # (F, 3)
+ weights: np.ndarray # (V, B)
+ ibm: np.ndarray # (B, 4, 4) inverse bind = Cluster.Transform
+ color: np.ndarray
+ face_colors: np.ndarray | None = None # (F, 3) BGR from the diffuse map
+
+
+@dataclass
+class YBotAsset:
+ rig: MixamoRig
+ meshes: list[SkinnedMesh]
+ bone_index: dict[str, int] = field(default_factory=dict)
+
+ @property
+ def n_bones(self) -> int:
+ return len(self.rig.order)
+
+
+def load_full_rig(fbx_path: Path) -> MixamoRig:
+ """Every Mixamo limb, same PreRotation local-matrix convention as the core rig."""
+ fbx_path = Path(fbx_path)
+ version, nodes = parse_fbx(fbx_path)
+ objects = extract_objects(nodes)
+ connections = extract_connections(nodes)
+ settings = extract_global_settings(nodes)
+ limbs = collect_limb_nodes(objects)
+ parent_of = build_hierarchy(limbs, connections)
+ local_of: dict[int, np.ndarray] = {}
+ parts_of: dict[int, dict[str, np.ndarray]] = {}
+ for uid, obj in limbs.items():
+ local, parts = bone_local_matrix(obj["props"])
+ local_of[uid] = local
+ parts_of[uid] = parts
+ world_of = compose_world(parent_of, local_of)
+
+ children_of: dict[int, list[int]] = {uid: [] for uid in limbs}
+ for uid, parent in parent_of.items():
+ if parent in children_of:
+ children_of[parent].append(uid)
+ roots = [uid for uid, parent in parent_of.items() if parent is None]
+ ordered: list[int] = []
+
+ def walk(uid: int) -> None:
+ ordered.append(uid)
+ for child in children_of[uid]:
+ walk(child)
+
+ for root in roots:
+ walk(root)
+
+ bones: dict[str, MixamoBone] = {}
+ order: list[str] = []
+ for uid in ordered:
+ obj = limbs[uid]
+ name = _fbx_name(obj["name"])
+ parts = parts_of[uid]
+ parent_uid = parent_of[uid]
+ parent_name = _fbx_name(limbs[parent_uid]["name"]) if parent_uid in limbs else None
+ child_names = [_fbx_name(limbs[c]["name"]) for c in children_of[uid]]
+ bones[name] = MixamoBone(
+ name=name,
+ parent=parent_name,
+ children=child_names,
+ pre_rotation=euler_xyz_deg_to_matrix(parts["pre_rotation_euler_deg"]),
+ lcl_translation_cm=parts["lcl_translation"].copy(),
+ rest_world=world_of[uid],
+ rest_local=local_of[uid],
+ pre_rotation_euler_deg=parts["pre_rotation_euler_deg"].copy(),
+ )
+ order.append(name)
+ return MixamoRig(
+ bones=bones,
+ order=order,
+ meters_per_unit=unit_to_meters(settings),
+ source=fbx_path,
+ coord_system={"fbx_unit": "centimeter", "meters_per_fbx_unit": unit_to_meters(settings)},
+ )
+
+
+def load_character_asset(
+ fbx_path: Path, *, textures: bool = False, lift: float = 1.0
+) -> YBotAsset:
+ """Rig + skinned meshes (optionally with diffuse-sampled face colors)."""
+ fbx_path = Path(fbx_path)
+ rig = load_full_rig(fbx_path)
+ _, nodes = parse_fbx(fbx_path)
+ objects = extract_objects(nodes)
+ connections = extract_connections(nodes)
+ bone_index = {name: i for i, name in enumerate(rig.order)}
+ n_bones = len(rig.order)
+
+ uid_name = {uid: _fbx_name(obj["name"]) for uid, obj in objects.items()}
+
+ limb_uids = {uid for uid, obj in objects.items() if obj["node_name"] == "Model" and obj.get("class") == "LimbNode"}
+ skin_uids = {uid for uid, obj in objects.items() if obj["node_name"] == "Deformer" and obj.get("class") == "Skin"}
+ geom_uids = {uid for uid, obj in objects.items() if obj["node_name"] == "Geometry"}
+ cluster_uids = {uid for uid, obj in objects.items() if obj["node_name"] == "Deformer" and obj.get("class") == "Cluster"}
+
+ skin_to_geom: dict[int, int] = {}
+ cluster_to_skin: dict[int, int] = {}
+ cluster_to_limb: dict[int, int] = {}
+ for kind, src, dst in connections:
+ if kind != "OO":
+ continue
+ if src in cluster_uids and dst in skin_uids:
+ cluster_to_skin[src] = dst
+ if src in limb_uids and dst in cluster_uids:
+ cluster_to_limb[dst] = src
+ if src in skin_uids and dst in geom_uids:
+ skin_to_geom[src] = dst
+ if src in geom_uids and dst in skin_uids:
+ skin_to_geom[dst] = src
+
+ colors = {
+ "Alpha_Surface": np.array([0.72, 0.74, 0.78]),
+ "Alpha_Joints": np.array([0.38, 0.40, 0.44]),
+ }
+ meshes: list[SkinnedMesh] = []
+ for geom_uid in geom_uids:
+ geom = objects[geom_uid]
+ verts_node = _child_named(geom["raw"], "Vertices")
+ idx_node = _child_named(geom["raw"], "PolygonVertexIndex")
+ if verts_node is None or idx_node is None:
+ continue
+ vertices = np.asarray(verts_node["props"][0], dtype=np.float64).reshape(-1, 3)
+ faces, slots, poly_ids = _triangulate(np.asarray(idx_node["props"][0], dtype=np.int64))
+ tinted = None
+ if textures:
+ sampled = face_colors_from_textures(
+ objects, connections, geom_uid, geom["raw"], slots, poly_ids, lift
+ )
+ if sampled is not None:
+ tinted, visible = sampled
+ # Masked-out faces are dropped outright rather than blended, so
+ # the painter fill stays a single opaque pass.
+ if not np.all(visible):
+ faces = faces[visible]
+ tinted = tinted[visible]
+ n_verts = vertices.shape[0]
+ weights = np.zeros((n_verts, n_bones), dtype=np.float64)
+ ibm = np.repeat(np.eye(4)[None, ...], n_bones, axis=0)
+ for cluster_uid, skin_uid in cluster_to_skin.items():
+ if skin_to_geom.get(skin_uid) != geom_uid:
+ continue
+ limb_uid = cluster_to_limb.get(cluster_uid)
+ if limb_uid is None:
+ continue
+ bone_name = uid_name[limb_uid]
+ if bone_name not in bone_index:
+ continue
+ cluster = objects[cluster_uid]
+ idx_elem = _child_named(cluster["raw"], "Indexes")
+ w_elem = _child_named(cluster["raw"], "Weights")
+ t_elem = _child_named(cluster["raw"], "Transform")
+ b = bone_index[bone_name]
+ if t_elem is not None and t_elem["props"]:
+ ibm[b] = _mat4_fbx(t_elem["props"][0])
+ if idx_elem is None or w_elem is None:
+ continue
+ indexes = np.asarray(idx_elem["props"][0], dtype=np.int64)
+ wts = np.asarray(w_elem["props"][0], dtype=np.float64)
+ weights[indexes, b] = wts
+ row = weights.sum(axis=1, keepdims=True)
+ row = np.where(row < 1e-8, 1.0, row)
+ weights = weights / row
+ meshes.append(
+ SkinnedMesh(
+ name=_fbx_name(geom["name"]),
+ vertices=vertices,
+ faces=faces,
+ weights=weights,
+ ibm=ibm,
+ color=colors.get(_fbx_name(geom["name"]), np.array([0.6, 0.6, 0.6])),
+ face_colors=tinted,
+ )
+ )
+ return YBotAsset(rig=rig, meshes=meshes, bone_index=bone_index)
+
+
+def bone_palette(world: dict[str, np.ndarray], mesh: SkinnedMesh, order: list[str]) -> np.ndarray:
+ mats = np.stack([world[name] for name in order], axis=0)
+ return mats @ mesh.ibm
+
+
+def skin_mesh(mesh: SkinnedMesh, palette: np.ndarray) -> np.ndarray:
+ """Linear blend skinning. palette is (B, 4, 4) = M_world @ IBM."""
+ ones = np.ones((mesh.vertices.shape[0], 1), dtype=np.float64)
+ vh = np.concatenate([mesh.vertices, ones], axis=1)
+ posed = np.zeros((mesh.vertices.shape[0], 4), dtype=np.float64)
+ for b in range(palette.shape[0]):
+ w = mesh.weights[:, b]
+ if float(w.max()) < 1e-12:
+ continue
+ posed += w[:, None] * (vh @ palette[b].T)
+ return posed[:, :3]
+
+
+def posed_vertices(
+ asset: YBotAsset,
+ r_anim: dict[str, np.ndarray],
+ hips_translation_cm: np.ndarray,
+ lcl_translations_cm: dict[str, np.ndarray] | None = None,
+) -> list[np.ndarray]:
+ world = forward_kinematics(
+ asset.rig,
+ r_anim,
+ hips_translation_cm=hips_translation_cm,
+ lcl_translations_cm=lcl_translations_cm,
+ )
+ out: list[np.ndarray] = []
+ for mesh in asset.meshes:
+ palette = bone_palette(world, mesh, asset.rig.order)
+ out.append(skin_mesh(mesh, palette))
+ return out
+
+
+def rest_skin_error(asset: YBotAsset) -> float:
+ """Identity R_anim must reproduce bind vertices (PreRotation FK + IBM)."""
+ world = forward_kinematics(asset.rig, {})
+ err = 0.0
+ for mesh in asset.meshes:
+ posed = skin_mesh(mesh, bone_palette(world, mesh, asset.rig.order))
+ err = max(err, float(np.linalg.norm(posed - mesh.vertices, axis=1).max()))
+ return err
diff --git a/motion2mixamorig/mixamo/tpose_calibration.py b/motion2mixamorig/mixamo/tpose_calibration.py
new file mode 100644
index 0000000..44ea203
--- /dev/null
+++ b/motion2mixamorig/mixamo/tpose_calibration.py
@@ -0,0 +1,382 @@
+"""The single source of truth for the Mixamo Skeleton ↔ character Rig T-pose relationship.
+
+ Mixamo Skeleton T-pose
+ ↕ one global calibration
+ character Rig rest T-pose
+
+Both the T-pose inspection programs and the animation retargeting import this
+module, so there is exactly one Front / Right / Up convention in the project.
+
+Nothing here is a manual correction: the anatomical axes are measured from bone
+positions, and the alignment is `B_rig @ B_skel.T` with no polar/SVD det fix and
+no per-bone 90/180 patches.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from .fbx_skeleton import extract_skeleton
+from .kinematics import (
+ MixamoRig,
+ forward_kinematics,
+ normalize,
+ rig_from_skeleton_data,
+ world_positions_m,
+)
+
+# Mixamo skeleton world (validated Human → Mixamo space):
+# +X = anatomical LEFT, +Y = UP, +Z = chest-forward (toes / FBX front).
+LEFT_DIR = np.array([1.0, 0.0, 0.0])
+RIGHT_DIR = np.array([-1.0, 0.0, 0.0])
+UP_DIR = np.array([0.0, 1.0, 0.0])
+DOWN_DIR = np.array([0.0, -1.0, 0.0])
+FWD_DIR = np.array([0.0, 0.0, 1.0])
+
+REQUIRED_BONES = (
+ "mixamorig:Hips",
+ "mixamorig:Spine",
+ "mixamorig:Spine1",
+ "mixamorig:Spine2",
+ "mixamorig:Neck",
+ "mixamorig:Head",
+ "mixamorig:LeftShoulder",
+ "mixamorig:LeftArm",
+ "mixamorig:LeftForeArm",
+ "mixamorig:LeftHand",
+ "mixamorig:RightShoulder",
+ "mixamorig:RightArm",
+ "mixamorig:RightForeArm",
+ "mixamorig:RightHand",
+ "mixamorig:LeftUpLeg",
+ "mixamorig:LeftLeg",
+ "mixamorig:LeftFoot",
+ "mixamorig:RightUpLeg",
+ "mixamorig:RightLeg",
+ "mixamorig:RightFoot",
+)
+
+SPINE_CHAIN = (
+ "mixamorig:Spine",
+ "mixamorig:Spine1",
+ "mixamorig:Spine2",
+ "mixamorig:Neck",
+ "mixamorig:Head",
+ "mixamorig:HeadTop_End",
+)
+LEFT_ARM_CHAIN = (
+ "mixamorig:LeftShoulder",
+ "mixamorig:LeftArm",
+ "mixamorig:LeftForeArm",
+ "mixamorig:LeftHand",
+)
+RIGHT_ARM_CHAIN = (
+ "mixamorig:RightShoulder",
+ "mixamorig:RightArm",
+ "mixamorig:RightForeArm",
+ "mixamorig:RightHand",
+)
+LEFT_LEG_CHAIN = ("mixamorig:LeftUpLeg", "mixamorig:LeftLeg", "mixamorig:LeftFoot")
+RIGHT_LEG_CHAIN = ("mixamorig:RightUpLeg", "mixamorig:RightLeg", "mixamorig:RightFoot")
+LEFT_TOE_CHAIN = ("mixamorig:LeftToeBase", "mixamorig:LeftToe_End")
+RIGHT_TOE_CHAIN = ("mixamorig:RightToeBase", "mixamorig:RightToe_End")
+
+# Anatomical bone correspondence. Left stays Left, Right stays Right.
+# Human joint → Mixamo/Y Bot bone; the two rigs share these bone names.
+ANATOMICAL_BONE_MAP = {
+ "pelvis": "mixamorig:Hips",
+ "spine1": "mixamorig:Spine",
+ "spine2": "mixamorig:Spine1",
+ "spine3": "mixamorig:Spine2",
+ "neck": "mixamorig:Neck",
+ "head": "mixamorig:Head",
+ "left_collar": "mixamorig:LeftShoulder",
+ "left_shoulder": "mixamorig:LeftArm",
+ "left_elbow": "mixamorig:LeftForeArm",
+ "left_wrist": "mixamorig:LeftHand",
+ "right_collar": "mixamorig:RightShoulder",
+ "right_shoulder": "mixamorig:RightArm",
+ "right_elbow": "mixamorig:RightForeArm",
+ "right_wrist": "mixamorig:RightHand",
+ "left_hip": "mixamorig:LeftUpLeg",
+ "left_knee": "mixamorig:LeftLeg",
+ "left_ankle": "mixamorig:LeftFoot",
+ "left_foot": "mixamorig:LeftToeBase",
+ "right_hip": "mixamorig:RightUpLeg",
+ "right_knee": "mixamorig:RightLeg",
+ "right_ankle": "mixamorig:RightFoot",
+ "right_foot": "mixamorig:RightToeBase",
+}
+
+
+def rest_length(rest: dict[str, np.ndarray], parent: str, child: str) -> float:
+ return float(np.linalg.norm(rest[child] - rest[parent]))
+
+
+def canonical_mixamo_tpose(rig: MixamoRig) -> dict[str, np.ndarray]:
+ """Explicit Mixamo-skeleton T-pose. Not an animation frame.
+
+ Uses Mixamo rest bone *lengths* and the validated Mixamo world axes:
+ arms along ±X (left = +X), spine +Y, legs −Y, feet +Z. Hip / clavicle
+ widths keep the rest lateral offsets so proportions stay Mixamo's.
+ """
+ rest = world_positions_m(rig, forward_kinematics(rig, {}))
+ pos: dict[str, np.ndarray] = {"mixamorig:Hips": np.zeros(3)}
+
+ def place(child: str, parent: str, direction: np.ndarray) -> None:
+ length = rest_length(rest, parent, child)
+ pos[child] = pos[parent] + normalize(direction) * length
+
+ for child in SPINE_CHAIN:
+ parent = rig.bones[child].parent
+ place(child, parent, UP_DIR)
+
+ def lateral_then_chain(first: str, chain: tuple[str, ...], side_sign: float, along: np.ndarray) -> None:
+ parent = rig.bones[first].parent
+ offset = rest[first] - rest[parent]
+ lateral = np.array([side_sign * abs(float(offset[0])), float(offset[1]), 0.0])
+ if float(np.linalg.norm(lateral)) < 1e-8:
+ lateral = np.array([side_sign, 0.0, 0.0])
+ place(first, parent, lateral)
+ for child in chain[1:]:
+ p = rig.bones[child].parent
+ place(child, p, along)
+
+ lateral_then_chain("mixamorig:LeftShoulder", LEFT_ARM_CHAIN, +1.0, LEFT_DIR)
+ for child in LEFT_ARM_CHAIN[1:]:
+ place(child, rig.bones[child].parent, LEFT_DIR)
+ lateral_then_chain("mixamorig:RightShoulder", RIGHT_ARM_CHAIN, -1.0, RIGHT_DIR)
+ for child in RIGHT_ARM_CHAIN[1:]:
+ place(child, rig.bones[child].parent, RIGHT_DIR)
+
+ lateral_then_chain("mixamorig:LeftUpLeg", LEFT_LEG_CHAIN, +1.0, DOWN_DIR)
+ for child in LEFT_LEG_CHAIN[1:]:
+ place(child, rig.bones[child].parent, DOWN_DIR)
+ lateral_then_chain("mixamorig:RightUpLeg", RIGHT_LEG_CHAIN, -1.0, DOWN_DIR)
+ for child in RIGHT_LEG_CHAIN[1:]:
+ place(child, rig.bones[child].parent, DOWN_DIR)
+
+ for child in LEFT_TOE_CHAIN:
+ place(child, rig.bones[child].parent, FWD_DIR)
+ for child in RIGHT_TOE_CHAIN:
+ place(child, rig.bones[child].parent, FWD_DIR)
+
+ return pos
+
+
+def rig_rest_from_skeleton_data(data: dict, fbx_path: Path) -> tuple[dict[str, np.ndarray], dict]:
+ """Core Mixamo bones from the character FBX rest pose.
+
+ PreRotation is already inside the local matrix. Assimp
+ `$AssimpFbx$_PreRotation` helpers are not anatomical bones; extract_skeleton
+ applies Maya/FBX `T * Rpre * R * ...` so those extra nodes are not listed.
+ """
+ pos: dict[str, np.ndarray] = {}
+ for bone in data["bones"]:
+ pos[str(bone["name"])] = np.asarray(bone["rest_world"]["translation_m"], dtype=np.float64)
+ missing = [name for name in REQUIRED_BONES if name not in pos]
+ if missing:
+ raise RuntimeError(f"{fbx_path.name} is missing required Mixamo bones: {missing}")
+ meta = {
+ "source_fbx": data.get("source_fbx"),
+ "fbx_version": data.get("fbx_version"),
+ "coord_system": data.get("coord_system"),
+ "transform_convention": data.get("transform_convention"),
+ "bone_count": data.get("bone_count"),
+ "character": data.get("character"),
+ }
+ return pos, meta
+
+
+def hips_origin(pos: dict[str, np.ndarray]) -> dict[str, np.ndarray]:
+ origin = np.asarray(pos["mixamorig:Hips"], dtype=np.float64)
+ return {name: np.asarray(p, dtype=np.float64) - origin for name, p in pos.items()}
+
+
+def body_height(pos: dict[str, np.ndarray]) -> float:
+ return float(np.linalg.norm(pos["mixamorig:Head"] - pos["mixamorig:Hips"]))
+
+
+def scale_to_height(pos: dict[str, np.ndarray], height: float) -> dict[str, np.ndarray]:
+ h = body_height(pos)
+ if h < 1e-8:
+ raise RuntimeError("degenerate pelvis→head height")
+ s = float(height / h)
+ return {name: p * s for name, p in pos.items()}
+
+
+def anatomical_axes(pos: dict[str, np.ndarray], label: str, *, verbose: bool = True) -> dict:
+ """Right = right_hip − left_hip, Up = neck − pelvis. Front from the two crosses."""
+ left_hip = pos["mixamorig:LeftUpLeg"]
+ right_hip = pos["mixamorig:RightUpLeg"]
+ pelvis = pos["mixamorig:Hips"]
+ neck = pos["mixamorig:Neck"]
+ mid_sh = 0.5 * (pos["mixamorig:LeftArm"] + pos["mixamorig:RightArm"])
+ toe = pos["mixamorig:LeftToeBase"] - pos["mixamorig:LeftFoot"]
+ chest_hint = mid_sh - pelvis
+ if float(np.linalg.norm(chest_hint)) < 1e-6:
+ chest_hint = toe
+
+ right_raw = right_hip - left_hip
+ up_raw = neck - pelvis
+ up = normalize(up_raw, UP_DIR)
+ right = right_raw - up * float(right_raw @ up)
+ right = normalize(right, RIGHT_DIR)
+
+ cross_right_up = np.cross(right, up)
+ cross_up_right = np.cross(up, right)
+ n_ru = float(np.linalg.norm(cross_right_up))
+ n_ur = float(np.linalg.norm(cross_up_right))
+ fwd_ru = cross_right_up / n_ru if n_ru > 1e-12 else np.zeros(3)
+ fwd_ur = cross_up_right / n_ur if n_ur > 1e-12 else np.zeros(3)
+
+ hint = normalize(chest_hint, FWD_DIR)
+ toe_u = normalize(toe, FWD_DIR)
+ score_ru = float(hint @ fwd_ru) + float(toe_u @ fwd_ru)
+ score_ur = float(hint @ fwd_ur) + float(toe_u @ fwd_ur)
+ use_up_cross_right = score_ur >= score_ru
+ forward = fwd_ur if use_up_cross_right else fwd_ru
+ formula = "up × right" if use_up_cross_right else "right × up"
+
+ basis = np.column_stack([right, up, forward])
+ det = float(np.linalg.det(basis))
+ if verbose:
+ print(f"\n=== {label} anatomical axes ===")
+ print(f" right (RightUpLeg − LeftUpLeg, ortho to up) {np.round(right, 6)}")
+ print(f" up (Neck − Hips) {np.round(up, 6)}")
+ print(f" right × up {np.round(fwd_ru, 6)} chest· = {float(hint @ fwd_ru):+.4f} toe· = {float(toe_u @ fwd_ru):+.4f}")
+ print(f" up × right {np.round(fwd_ur, 6)} chest· = {float(hint @ fwd_ur):+.4f} toe· = {float(toe_u @ fwd_ur):+.4f}")
+ print(f" FRONT chosen {formula}")
+ print(f" forward {np.round(forward, 6)}")
+ print(f" basis columns [Right | Up | Forward]")
+ print(f" det(basis) {det:+.6f}")
+ if det < 0.0:
+ print(" FLAG det < 0 — anatomical [Right|Up|Forward] is left-handed in this world.")
+ print(" Expected for Mixamo: anatomical right is world −X (world +X is character left).")
+ print(" This labels the existing world; it is not a transform we apply.")
+ elif abs(det - 1.0) > 1e-3:
+ print(f" FLAG |det| not 1 — basis is not orthonormal-proper.")
+ else:
+ print(" det ≈ +1 right-handed orthonormal frame.")
+ return {
+ "right": right,
+ "up": up,
+ "forward": forward,
+ "basis": basis,
+ "det": det,
+ "front_formula": formula,
+ "cross_right_up": fwd_ru,
+ "cross_up_right": fwd_ur,
+ "chest_hint": hint,
+ "toe_hint": toe_u,
+ }
+
+
+def global_alignment(skel: dict, rig: dict, *, verbose: bool = True) -> dict:
+ """R such that R @ skeleton_axis ≈ rig_axis for Right, Up, Forward.
+
+ R = B_rig @ B_skel.T. Not polar-projected. A det −1 result is a reflection
+ and is flagged, not silently corrected with a 180° yaw.
+ """
+ b_s = skel["basis"]
+ b_r = rig["basis"]
+ r = b_r @ b_s.T
+ det = float(np.linalg.det(r))
+ if verbose:
+ print("\n=== candidate global alignment R_skeleton_to_rig ===")
+ print(" maps Skeleton Right/Up/Forward → Rig Right/Up/Forward")
+ print(r)
+ print(f" det(R_skeleton_to_rig) = {det:+.6f}")
+ if det < 0.0:
+ print(" FLAG det < 0 — this is a REFLECTION / mirror. Not applied as a rotation.")
+ print(" No SVD/polar flip, no rotY(180) substitute.")
+ elif abs(det - 1.0) > 5e-3:
+ print(" FLAG det is not +1 — not a proper rotation.")
+ else:
+ print(" det ≈ +1 proper rotation (no reflection).")
+ mapped = r @ b_s
+ print(" R @ skeleton_right ", np.round(mapped[:, 0], 6), " vs rig", np.round(b_r[:, 0], 6))
+ print(" R @ skeleton_up ", np.round(mapped[:, 1], 6), " vs rig", np.round(b_r[:, 1], 6))
+ print(" R @ skeleton_forward ", np.round(mapped[:, 2], 6), " vs rig", np.round(b_r[:, 2], 6))
+ return {"R": r, "det": det}
+
+
+def apply_global(
+ pos: dict[str, np.ndarray],
+ r: np.ndarray,
+ scale: float,
+ translation: np.ndarray,
+) -> dict[str, np.ndarray]:
+ return {name: scale * (r @ p) + translation for name, p in pos.items()}
+
+
+@dataclass(frozen=True)
+class TposeCalibration:
+ """Frozen result of the accepted T-pose orientation check.
+
+ `skeleton_axes` / `rig_axes` are the anatomical [Right | Up | Forward] bases
+ that produced the accepted FRONT/BACK/LEFT/RIGHT images. `r_skeleton_to_rig`
+ maps one onto the other. `height_scale` stretches the Mixamo pelvis→head
+ length onto the character rig's for visualisation only.
+ """
+
+ skeleton_rig: MixamoRig
+ skeleton_tpose: dict[str, np.ndarray]
+ rig_rest: dict[str, np.ndarray]
+ skeleton_axes: dict
+ rig_axes: dict
+ r_skeleton_to_rig: np.ndarray
+ height_scale: float
+ fbx_meta: dict
+ fbx_path: Path
+
+ @property
+ def forward(self) -> np.ndarray:
+ """Anatomical FRONT in the character rig world. Used by the fixed render camera."""
+ return np.asarray(self.rig_axes["forward"], dtype=np.float64)
+
+ @property
+ def up(self) -> np.ndarray:
+ return np.asarray(self.rig_axes["up"], dtype=np.float64)
+
+ @property
+ def right(self) -> np.ndarray:
+ return np.asarray(self.rig_axes["right"], dtype=np.float64)
+
+
+def load_calibration(fbx_path: Path, *, verbose: bool = False) -> TposeCalibration:
+ """T-pose calibration between the Mixamo core skeleton and the character rig.
+
+ Both sides come from the same character FBX: the core-bone skeleton drives
+ the retarget, the full rig receives the motion. Everything is measured from
+ the file; nothing is hand-authored.
+ """
+ fbx_path = Path(fbx_path)
+ data = extract_skeleton(fbx_path)
+ rig = rig_from_skeleton_data(data, fbx_path)
+ skel = hips_origin(canonical_mixamo_tpose(rig))
+ rest_raw, fbx_meta = rig_rest_from_skeleton_data(data, fbx_path)
+ rest = hips_origin(rest_raw)
+
+ ref_h = body_height(rest)
+ skel_n = scale_to_height(skel, ref_h)
+ height_scale = float(ref_h / body_height(skel))
+
+ skel_axes = anatomical_axes(skel_n, "Mixamo skeleton T-pose", verbose=verbose)
+ rig_axes = anatomical_axes(rest, f"{fbx_path.stem} FBX rest rig", verbose=verbose)
+ align = global_alignment(skel_axes, rig_axes, verbose=verbose)
+
+ return TposeCalibration(
+ skeleton_rig=rig,
+ skeleton_tpose=skel_n,
+ rig_rest=rest,
+ skeleton_axes=skel_axes,
+ rig_axes=rig_axes,
+ r_skeleton_to_rig=align["R"],
+ height_scale=height_scale,
+ fbx_meta=fbx_meta,
+ fbx_path=fbx_path,
+ )
diff --git a/motion2mixamorig/mixamo/ybot_retarget.py b/motion2mixamorig/mixamo/ybot_retarget.py
new file mode 100644
index 0000000..63c9aca
--- /dev/null
+++ b/motion2mixamorig/mixamo/ybot_retarget.py
@@ -0,0 +1,234 @@
+"""Mixamo Skeleton motion → Y Bot Rig motion, derived from the T-pose calibration.
+
+ validated_mixamo_motion
+ ↓
+ tpose_calibration
+ ↓
+ apply_rest_relative_rotations()
+ ↓
+ ybot_rig_fk
+
+For every controlled bone we measure how the Mixamo Skeleton bone moved away
+from its own rest/T-pose, and apply that same rest-relative motion to the
+corresponding Y Bot bone:
+
+ delta_skeleton = inverse(skeleton_rest_local_rotation) * skeleton_frame_local_rotation
+ ybot_frame_local = ybot_rest_local_rotation * (C * delta_skeleton * Cᵀ)
+
+`C` is the per-bone rest-frame basis conversion. It is *derived* from the two
+validated T-pose bone frames (see `bone_basis_change`), never authored by hand.
+No world-space Euler angles are copied, and there are no per-bone 90°/180°
+constants, laterality swaps or camera-based assumptions anywhere in this module.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from .kinematics import MixamoRig, forward_kinematics, world_positions_m
+from .skinned_mesh import load_full_rig
+from .tpose_calibration import (
+ TposeCalibration,
+ anatomical_axes,
+ global_alignment,
+ hips_origin,
+)
+
+# Bones whose motion is authored by the validated Human → Mixamo retarget.
+# Anatomical LEFT stays LEFT and RIGHT stays RIGHT.
+CONTROLLED_BONES = (
+ "mixamorig:Hips",
+ "mixamorig:Spine",
+ "mixamorig:Spine1",
+ "mixamorig:Spine2",
+ "mixamorig:Neck",
+ "mixamorig:Head",
+ "mixamorig:LeftShoulder",
+ "mixamorig:LeftArm",
+ "mixamorig:LeftForeArm",
+ "mixamorig:LeftHand",
+ "mixamorig:RightShoulder",
+ "mixamorig:RightArm",
+ "mixamorig:RightForeArm",
+ "mixamorig:RightHand",
+ "mixamorig:LeftUpLeg",
+ "mixamorig:LeftLeg",
+ "mixamorig:LeftFoot",
+ "mixamorig:LeftToeBase",
+ "mixamorig:RightUpLeg",
+ "mixamorig:RightLeg",
+ "mixamorig:RightFoot",
+ "mixamorig:RightToeBase",
+)
+
+
+def rest_world_rotations(rig: MixamoRig) -> dict[str, np.ndarray]:
+ """World rotation of each bone's rest frame.
+
+ FK is `M_local = T * Rpre * R_anim`, so at rest (`R_anim = I`) the bone's
+ world rotation is `R_parent_world @ Rpre`. PreRotation therefore stays part
+ of the rest transform and is never folded into the animation channel.
+ """
+ world = forward_kinematics(rig, {})
+ return {name: np.asarray(mat[:3, :3], dtype=np.float64) for name, mat in world.items()}
+
+
+def bone_basis_change(
+ skel_rest_rot: dict[str, np.ndarray],
+ ybot_rest_rot: dict[str, np.ndarray],
+ bone: str,
+ r_world_align: np.ndarray,
+) -> np.ndarray:
+ """Rest-frame basis conversion `C` for one bone, from the two T-pose frames.
+
+ A rest-relative local rotation `D` acts on the bone's own rest frame. In the
+ Mixamo Skeleton that frame sits in the world as `F = skel_rest_rot[bone]`;
+ in the Y Bot as `F' = ybot_rest_rot[bone]`. Requiring the *world* motion to
+ match (up to the calibrated world alignment `R`) gives
+
+ F' D' F'ᵀ = R (F D Fᵀ) Rᵀ ⇒ D' = C D Cᵀ, C = F'ᵀ R F
+
+ so `C` falls straight out of the two validated rest poses.
+ """
+ f_skel = np.asarray(skel_rest_rot[bone], dtype=np.float64)
+ f_ybot = np.asarray(ybot_rest_rot[bone], dtype=np.float64)
+ return f_ybot.T @ np.asarray(r_world_align, dtype=np.float64) @ f_skel
+
+
+@dataclass(frozen=True)
+class YBotMapping:
+ """Frozen Mixamo Skeleton → Y Bot transfer, built once from the calibration."""
+
+ skeleton_rig: MixamoRig
+ ybot_rig: MixamoRig
+ basis_change: dict[str, np.ndarray]
+ translation_scale: float
+ r_world_align: np.ndarray
+ max_basis_deviation_deg: float
+
+ @property
+ def controlled(self) -> tuple[str, ...]:
+ return tuple(name for name in CONTROLLED_BONES if name in self.ybot_rig.bones)
+
+ def apply_rest_relative_rotations(
+ self, r_anim_skeleton: dict[str, np.ndarray]
+ ) -> dict[str, np.ndarray]:
+ """Transfer one frame of rest-relative local rotations onto the Y Bot rig.
+
+ Input and output are both animation-channel rotations, i.e. the `R_anim`
+ of `M_local = T * Rpre * R_anim`. Bones the Mixamo retarget does not
+ author (the finger chains) are left at rest.
+ """
+ out: dict[str, np.ndarray] = {}
+ for bone, delta in r_anim_skeleton.items():
+ change = self.basis_change.get(bone)
+ if change is None:
+ continue
+ out[bone] = change @ np.asarray(delta, dtype=np.float64) @ change.T
+ return out
+
+ def hips_translation_cm(self, pelvis_translation_cm: np.ndarray) -> np.ndarray:
+ """Root translation carried across with the calibrated scale and world alignment."""
+ p = np.asarray(pelvis_translation_cm, dtype=np.float64).reshape(3)
+ return self.translation_scale * (self.r_world_align @ p)
+
+ def ybot_rig_fk(
+ self,
+ r_anim_ybot: dict[str, np.ndarray],
+ hips_translation_cm: np.ndarray,
+ ) -> dict[str, np.ndarray]:
+ """World 4x4 per Y Bot bone. Raw FBX rest hierarchy, animation on top."""
+ return forward_kinematics(
+ self.ybot_rig,
+ r_anim_ybot,
+ hips_translation_cm=np.asarray(hips_translation_cm, dtype=np.float64),
+ )
+
+ def ybot_positions_m(
+ self,
+ r_anim_ybot: dict[str, np.ndarray],
+ hips_translation_cm: np.ndarray,
+ ) -> dict[str, np.ndarray]:
+ return world_positions_m(self.ybot_rig, self.ybot_rig_fk(r_anim_ybot, hips_translation_cm))
+
+
+def _rotation_angle_deg(rot: np.ndarray) -> float:
+ cosine = (float(np.trace(np.asarray(rot, dtype=np.float64))) - 1.0) * 0.5
+ return float(np.degrees(np.arccos(np.clip(cosine, -1.0, 1.0))))
+
+
+def build_mapping(
+ calibration: TposeCalibration,
+ fbx_path: Path | None = None,
+ *,
+ verbose: bool = True,
+) -> YBotMapping:
+ """Derive the Mixamo Skeleton → character-rig transfer from the calibration."""
+ fbx_path = Path(fbx_path) if fbx_path is not None else calibration.fbx_path
+ skel_rig = calibration.skeleton_rig
+ ybot_rig = load_full_rig(fbx_path)
+
+ skel_rest_rot = rest_world_rotations(skel_rig)
+ ybot_rest_rot = rest_world_rotations(ybot_rig)
+
+ # World alignment between the two rigs, measured with the same
+ # `anatomical_axes` / `global_alignment` that produced the accepted
+ # orientation-check images — so the animation cannot invent a different
+ # Front/Right/Up convention. Both sides are the rigs' own rest poses.
+ skel_rest_pos = hips_origin(world_positions_m(skel_rig, forward_kinematics(skel_rig, {})))
+ ybot_rest_pos = hips_origin(world_positions_m(ybot_rig, forward_kinematics(ybot_rig, {})))
+ skel_rest_axes = anatomical_axes(skel_rest_pos, "Mixamo Skeleton rest", verbose=False)
+ ybot_rest_axes = anatomical_axes(ybot_rest_pos, "Y Bot rig rest", verbose=False)
+ r_world_align = global_alignment(skel_rest_axes, ybot_rest_axes, verbose=False)["R"]
+
+ # The Y Bot anatomical frame used here must be the one the calibration
+ # accepted; otherwise the render camera and the mapping would disagree.
+ front_drift = float(
+ np.degrees(
+ np.arccos(
+ np.clip(float(np.asarray(ybot_rest_axes["forward"]) @ calibration.forward), -1.0, 1.0)
+ )
+ )
+ )
+ if front_drift > 1e-6:
+ raise RuntimeError(
+ f"rig FRONT disagrees with the accepted calibration by {front_drift:.6f} deg"
+ )
+
+ basis_change: dict[str, np.ndarray] = {}
+ worst = 0.0
+ for bone in CONTROLLED_BONES:
+ if bone not in skel_rest_rot or bone not in ybot_rest_rot:
+ continue
+ change = bone_basis_change(skel_rest_rot, ybot_rest_rot, bone, r_world_align)
+ basis_change[bone] = change
+ worst = max(worst, _rotation_angle_deg(change))
+
+ translation_scale = float(skel_rig.meters_per_unit / ybot_rig.meters_per_unit)
+
+ if verbose:
+ print("\n=== Mixamo Skeleton → Y Bot rest-relative mapping ===")
+ print(f" skeleton rig {skel_rig.source.name} {len(skel_rig.order)} bones")
+ print(f" Y Bot rig {Path(fbx_path).name} {len(ybot_rig.order)} bones")
+ print(f" controlled bones {len(basis_change)}")
+ print(f" untouched (rest) bones {len(ybot_rig.order) - len(basis_change)} (finger chains)")
+ print(f" translation scale {translation_scale:.6f}")
+ print(f" world alignment {_rotation_angle_deg(r_world_align):.6f} deg (measured, det={np.linalg.det(r_world_align):+.6f})")
+ print(f" max |C| rotation {worst:.6f} deg (0 = identical rest frames)")
+ if worst < 1e-6:
+ print(" the two validated T-pose bone frames coincide, so C = I for every bone:")
+ print(" the Mixamo rest-relative rotation is applied to the Y Bot unchanged.")
+ else:
+ print(" C is a measured rest-frame basis change, not a hand-authored offset.")
+
+ return YBotMapping(
+ skeleton_rig=skel_rig,
+ ybot_rig=ybot_rig,
+ basis_change=basis_change,
+ translation_scale=translation_scale,
+ r_world_align=r_world_align,
+ max_basis_deviation_deg=worst,
+ )
diff --git a/motion2mixamorig/paths.py b/motion2mixamorig/paths.py
new file mode 100644
index 0000000..f9b9998
--- /dev/null
+++ b/motion2mixamorig/paths.py
@@ -0,0 +1,59 @@
+"""Single source of truth for project paths: assets, weights, outputs."""
+
+from __future__ import annotations
+
+import os
+from datetime import datetime
+from pathlib import Path
+
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+
+ASSETS = PROJECT_ROOT / "assets"
+BODY_MODELS = ASSETS / "body_models"
+SMPLX_NEUTRAL = BODY_MODELS / "smplx" / "SMPLX_NEUTRAL.npz"
+MIXAMO_DIR = ASSETS / "mixamo"
+DEFAULT_RIG = MIXAMO_DIR / "Y_Bot.fbx"
+VIDEO_DIR = ASSETS / "video"
+
+WEIGHTS = PROJECT_ROOT / "weights"
+OUTPUTS = PROJECT_ROOT / "outputs"
+
+VIDEO_SUFFIXES = (".mp4", ".mov", ".avi", ".mkv", ".webm")
+
+
+def list_videos() -> list[Path]:
+ """User-supplied videos in assets/video/, most recently added first."""
+ if not VIDEO_DIR.is_dir():
+ return []
+ videos = [
+ p
+ for p in VIDEO_DIR.iterdir()
+ if p.suffix.lower() in VIDEO_SUFFIXES and not p.name.startswith(".")
+ ]
+ return sorted(videos, key=lambda p: p.stat().st_mtime, reverse=True)
+
+
+def latest_video() -> Path | None:
+ """The video most recently placed into assets/video/ (by file mtime)."""
+ videos = list_videos()
+ return videos[0] if videos else None
+
+
+def new_run_dir(video: Path, now: datetime | None = None) -> Path:
+ """Create outputs/<YYYYMMDD_HHMMSS>_<video-stem>/ for one `m2mr run`."""
+ stamp = (now or datetime.now()).strftime("%Y%m%d_%H%M%S")
+ run_dir = OUTPUTS / f"{stamp}_{video.stem}"
+ run_dir.mkdir(parents=True, exist_ok=True)
+ (run_dir / "videos").mkdir(exist_ok=True)
+ return run_dir
+
+
+def export_gvhmr_env() -> None:
+ """Point GVHMR at this project's weights/ and assets/ before importing it.
+
+ The gvhmr package resolves its asset roots from these variables at import
+ time. Auto-downloaded checkpoints land in weights/; the gated SMPL-X body
+ model is read from assets/body_models/ where the user placed it.
+ """
+ os.environ.setdefault("GVHMR_CHECKPOINTS", str(WEIGHTS))
+ os.environ.setdefault("GVHMR_BODY_MODELS", str(BODY_MODELS))
diff --git a/motion2mixamorig/pipeline.py b/motion2mixamorig/pipeline.py
new file mode 100644
index 0000000..aff185b
--- /dev/null
+++ b/motion2mixamorig/pipeline.py
@@ -0,0 +1,158 @@
+"""One `m2mr run`: video -> human skeleton -> Mixamo rig -> result videos.
+
+Everything a run produces lands in its own timestamped folder:
+
+ outputs/<YYYYMMDD_HHMMSS>_<video>/
+ ├── run.json what ran, with what inputs, producing what
+ ├── skeleton_motion.npz 3D human skeleton (24 SMPL joints per frame)
+ ├── mixamo_rotations.npz per-bone animation rotations for the rig
+ ├── mixamo_character.glb skinned character + animation (Blender / Unity)
+ ├── cache/ GVHMR intermediates (2D pose, 3D lift)
+ └── videos/ the four result videos
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+import time
+from datetime import datetime
+from pathlib import Path
+
+import numpy as np
+
+from . import paths
+from .human_motion import HumanMotion, load_human_motion
+from .mixamo.animation import build_animation
+from .mixamo.gltf import write_glb
+from .mixamo.kinematics import quat_wxyz
+from .mixamo.retarget import verify_rest_fk
+from .mixamo.skinned_mesh import load_character_asset, rest_skin_error
+from .mixamo.tpose_calibration import load_calibration
+from .mixamo.ybot_retarget import build_mapping
+from .videos import render_run_videos
+
+ROTATION_CONVENTION = (
+ "Animation-channel rotations: M_local = T * Rpre * R_anim, world = parent @ local. "
+ "Quaternions are wxyz. FBX rest translations (T, Rpre) come from the rig file."
+)
+
+
+def save_mixamo_rotations(anim, mapping, rig_path: Path, out_npz: Path) -> Path:
+ """Per-bone animation rotations, both the portable Mixamo Skeleton channel
+ and the channel already converted for this rig file."""
+ bones = list(mapping.controlled)
+ n = anim.n
+ q_skel = np.zeros((n, len(bones), 4), dtype=np.float64)
+ q_rig = np.zeros((n, len(bones), 4), dtype=np.float64)
+ for i in range(n):
+ for b, name in enumerate(bones):
+ q_skel[i, b] = quat_wxyz(anim.r_anim_skeleton[i].get(name, np.eye(3)))
+ q_rig[i, b] = quat_wxyz(anim.r_anim_rig[i].get(name, np.eye(3)))
+ out_npz = Path(out_npz)
+ out_npz.parent.mkdir(parents=True, exist_ok=True)
+ np.savez_compressed(
+ out_npz,
+ bone_names=np.array(bones),
+ quat_wxyz_skeleton=q_skel,
+ quat_wxyz_rig=q_rig,
+ hips_translation_cm_skeleton=np.stack(anim.hips_cm_skeleton),
+ hips_translation_cm_rig=np.stack(anim.hips_cm_rig),
+ frames=np.array(anim.frames, dtype=np.int64),
+ fps=np.float64(anim.fps),
+ rig=np.array(str(rig_path)),
+ convention=np.array(ROTATION_CONVENTION),
+ )
+ return out_npz
+
+
+def run_pipeline(
+ video: Path,
+ rig: Path,
+ *,
+ device: str = "cpu",
+ skeleton_npz: Path | None = None,
+ verbose: bool = True,
+) -> Path:
+ """Full pipeline for one video + one rig. Returns the run directory.
+
+ `skeleton_npz` reuses a previous run's skeleton_motion.npz and skips the
+ (slow) video extraction entirely.
+ """
+ video = Path(video).resolve()
+ rig = Path(rig).resolve()
+ started = datetime.now()
+ t0 = time.time()
+ run_dir = paths.new_run_dir(video, started)
+ print(f"run directory {run_dir}")
+
+ motion_npz = run_dir / "skeleton_motion.npz"
+ if skeleton_npz is not None:
+ import shutil
+
+ shutil.copy2(skeleton_npz, motion_npz)
+ print(f"reusing skeleton from {skeleton_npz}")
+ else:
+ from .extract import extract_motion
+
+ print(f"extracting human motion from {video.name}")
+ extract_motion(
+ video,
+ motion_npz,
+ work_dir=run_dir / "cache",
+ device=device,
+ verbose=verbose,
+ )
+ motion: HumanMotion = load_human_motion(motion_npz)
+ print(f"human motion {motion.n_frames} frames @ {motion.fps:g} fps")
+
+ print(f"calibrating rig {rig.name}")
+ calibration = load_calibration(rig)
+ rest_err = verify_rest_fk(calibration.skeleton_rig)
+ mapping = build_mapping(calibration, verbose=verbose)
+ asset = load_character_asset(rig, textures=True)
+ skin_err = rest_skin_error(asset)
+ if verbose:
+ print(f"rest FK error {rest_err:.3e} rest skin error {skin_err:.3e} cm")
+
+ print("retargeting")
+ frames = list(range(motion.n_frames))
+ anim = build_animation(motion, mapping, frames, verbose=verbose)
+
+ rotations_npz = run_dir / "mixamo_rotations.npz"
+ save_mixamo_rotations(anim, mapping, rig, rotations_npz)
+ print(f"saved {rotations_npz}")
+
+ glb_path = run_dir / "mixamo_character.glb"
+ write_glb(glb_path, asset, anim.r_anim_rig, anim.hips_cm_rig, anim.fps)
+ print(f"saved {glb_path}")
+
+ print("rendering result videos")
+ videos = render_run_videos(
+ anim, asset, calibration, video, run_dir / "videos", verbose=verbose
+ )
+
+ report = {
+ "started": started.isoformat(timespec="seconds"),
+ "elapsed_s": round(time.time() - t0, 1),
+ "command": sys.argv,
+ "video": str(video),
+ "rig": str(rig),
+ "device": device,
+ "fps": float(motion.fps),
+ "n_frames": motion.n_frames,
+ "skeleton_reused_from": str(skeleton_npz) if skeleton_npz else None,
+ "rest_fk_error": rest_err,
+ "rest_skin_error_cm": skin_err,
+ "twist_events": len(anim.twist_events),
+ "quat_sign_flips": len(anim.sign_events),
+ "outputs": {
+ "skeleton_motion": motion_npz.name,
+ "mixamo_rotations": rotations_npz.name,
+ "mixamo_character_glb": glb_path.name,
+ "videos": {k: f"videos/{v.name}" for k, v in videos.items()},
+ },
+ }
+ (run_dir / "run.json").write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+ print(f"done in {report['elapsed_s']}s -> {run_dir}")
+ return run_dir
diff --git a/motion2mixamorig/stabilize_kp2d.py b/motion2mixamorig/stabilize_kp2d.py
new file mode 100644
index 0000000..19199ae
--- /dev/null
+++ b/motion2mixamorig/stabilize_kp2d.py
@@ -0,0 +1,96 @@
+"""Keep COCO-17 left/right identity consistent across time.
+
+RTMPose/ViTPose label left/right independently per frame. When the subject
+turns to face away, those labels often swap. GVHMR then reads a jumping 2D
+skeleton and spins the 3D heading. This assigns each left/right pair to the
+previous frame's positions so anatomical identity is tracked, not re-guessed.
+"""
+
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+
+import numpy as np
+
+# Full COCO-17 left/right permutation (nose stays).
+COCO17_SWAP = np.array([0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 16, 15], dtype=np.int64)
+
+
+def _as_numpy(x) -> np.ndarray:
+ if hasattr(x, "detach"):
+ return x.detach().cpu().numpy()
+ return np.asarray(x)
+
+
+def _lr_x(frame: np.ndarray) -> float:
+ """Shoulder left-minus-right in image x. Hips if shoulders are collapsed."""
+ sh = float(frame[5, 0] - frame[6, 0])
+ if abs(sh) >= 40.0:
+ return sh
+ return float(frame[11, 0] - frame[12, 0])
+
+
+def stabilize_coco17(kp2d: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ """Return (stabilized (F,17,3), per-frame swap flags (F,)).
+
+ A real 180-degree turn drives left-right x through a narrow side view. A
+ label flip jumps the sign while the width stays large. Those jumps are
+ un-swapped so GVHMR sees one anatomical identity.
+
+ "Narrow" is relative to how wide this subject usually reads: on a mid-spin
+ frame the estimator can report the shoulders 50-60 px apart instead of ~0
+ (one such borderline frame per turn is common), and a fixed 40 px cut then
+ misreads the whole genuine turn as a label flip and un-swaps it away. The
+ width is invariant under label swaps, so the scale can be measured on the
+ raw input up front.
+ """
+ src = np.asarray(_as_numpy(kp2d), dtype=np.float64)
+ if src.ndim != 3 or src.shape[1] < 17:
+ raise ValueError(f"expected (F, 17, 3) COCO-17, got {src.shape}")
+ out = src.copy()
+ n = len(out)
+ narrow = float(np.clip(0.45 * np.median([abs(_lr_x(f)) for f in src]), 40.0, 80.0))
+ swapped = np.zeros(n, dtype=np.int32)
+ last_sign = np.sign(_lr_x(out[0])) or 1.0
+ last_turn = -(10**9)
+ for i in range(n):
+ raw = src[i]
+ width = abs(_lr_x(raw))
+ sign = np.sign(_lr_x(raw)) or last_sign
+ if width >= 40.0 and sign != last_sign:
+ start = max(0, last_turn + 1, i - 18)
+ widths = [abs(_lr_x(f)) for f in out[start:i]]
+ run = best = 0
+ for w in widths:
+ run = run + 1 if w < narrow else 0
+ best = max(best, run)
+ if best >= (1 if last_turn < 0 else 4):
+ last_sign = sign
+ last_turn = i
+ else:
+ raw = raw[COCO17_SWAP]
+ swapped[i] = 1
+ out[i] = raw
+ sh = out[i, 5, 0] - out[i, 6, 0]
+ hip = out[i, 11, 0] - out[i, 12, 0]
+ if abs(sh) >= 40.0 and abs(hip) >= 40.0 and np.sign(sh) != np.sign(hip):
+ out[i, [11, 12, 13, 14, 15, 16]] = out[i, [12, 11, 14, 13, 16, 15]]
+ return out.astype(np.float32), swapped
+
+
+def stabilize_cache(path: Path) -> np.ndarray:
+ """Stabilize a GVHMR 2D-pose cache file in place (keeping a *_raw backup)."""
+ import torch
+
+ raw = path.with_name(path.stem + "_raw" + path.suffix)
+ src = raw if raw.exists() else path
+ if not src.exists():
+ raise FileNotFoundError(path)
+ if src == path and not raw.exists():
+ shutil.copy2(path, raw)
+ tensor = torch.load(src, weights_only=False)
+ fixed, swaps = stabilize_coco17(tensor)
+ print(f"stabilized {path.name}: corrected {int(swaps.sum())}/{len(swaps)} left/right flips")
+ torch.save(torch.from_numpy(fixed), path)
+ return fixed
diff --git a/motion2mixamorig/videos.py b/motion2mixamorig/videos.py
new file mode 100644
index 0000000..c31d50f
--- /dev/null
+++ b/motion2mixamorig/videos.py
@@ -0,0 +1,312 @@
+"""Render the four result videos of a run, all in the input video's aspect ratio.
+
+ videos/human_skeleton.mp4 3D human skeleton recovered from the video
+ videos/mixamo_skeleton.mp4 retargeted Mixamo Skeleton
+ videos/mixamo_character.mp4 skinned character mesh
+ videos/compare.mp4 2x2 grid: original | mixamo skeleton
+ human | character
+
+The three rendered panels share one perspective camera and one floor grid, so
+they read as views into the same space: an orthographic camera cannot show a
+floor receding to a vanishing point, and without a floor and a contact shadow
+the figure just hangs in black. All of it is fitted once over the whole clip;
+nothing is re-centred per frame.
+
+Frames are streamed straight into ffmpeg (which also muxes the source video's
+audio into every output), so memory stays flat no matter how long the clip is.
+When ffmpeg is missing the videos still get written via OpenCV, just silent.
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+from pathlib import Path
+
+import numpy as np
+
+from .mixamo.animation import RetargetedAnimation, all_points, skin_frames
+from .mixamo.render import (
+ DARK_THEME,
+ GRID_THEME,
+ MESH_COLORS_BGR,
+ Ground,
+ blank_panel,
+ build_perspective_camera,
+ draw_mesh,
+ draw_shadow,
+ draw_skeleton,
+ fit_letterbox,
+ ground_backdrop,
+ label_panel,
+)
+from .mixamo.retarget import HUMAN_EDGES, MIXAMO_EDGES
+
+# Rendered panel height in pixels; width follows the input video's aspect.
+PANEL_H = 720
+MAX_PANEL_W = 1280
+
+# Frames sampled only to bound the mesh for the camera fit and the floor
+# height; never all kept, or the streaming render would gain nothing.
+CAMERA_MESH_SAMPLES = 60
+# Fraction of the panel the whole clip's motion is fitted into.
+CAMERA_FILL = 0.94
+# Pinhole view matching the Mixamo viewport the look was taken from. Level
+# (elevation 0): a pinhole camera standing above the floor still sees it
+# recede, and tilting down makes the chest overhang the hips.
+CAMERA_FOV_DEG = 28.0
+CAMERA_AZIMUTH_DEG = 0.0
+CAMERA_ELEVATION_DEG = 0.0
+# The sole wanders over ~20 cm across a clip, mostly on lifted-leg frames, so
+# the floor goes at a low percentile of it rather than the minimum. Higher and
+# the character sinks in on many frames; lower and it visibly floats on all.
+FLOOR_PERCENTILE = 10.0
+# The character keeps its own texture colours, and a navy or black outfit
+# against a black backdrop has no silhouette: its panel gets the lifted
+# backdrop and a little more ambient fill than the white-on-black skeletons.
+CHARACTER_AMBIENT = 0.34
+
+
+def video_size(video: Path) -> tuple[int, int]:
+ import cv2
+
+ cap = cv2.VideoCapture(str(video))
+ if not cap.isOpened():
+ raise RuntimeError(f"Cannot open video: {video}")
+ w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+ h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+ cap.release()
+ return w, h
+
+
+def panel_size(video: Path) -> tuple[int, int]:
+ """Panel with the same aspect ratio as the input video, even dimensions."""
+ w, h = video_size(video)
+ pw = int(round(PANEL_H * w / max(h, 1)))
+ pw = min(max(pw, 2), MAX_PANEL_W)
+ return pw + pw % 2, PANEL_H + PANEL_H % 2
+
+
+class _OriginalReader:
+ """Sequential reader that letterboxes each requested source frame."""
+
+ def __init__(self, src: Path, width: int, height: int):
+ import cv2
+
+ self.blank = blank_panel(width, height)
+ self.width, self.height = width, height
+ self.cap = cv2.VideoCapture(str(src))
+ self.next_idx = 0
+
+ def frame(self, idx: int) -> np.ndarray:
+ out = None
+ while self.cap.isOpened() and self.next_idx <= idx:
+ ok, frame = self.cap.read()
+ if not ok:
+ break
+ if self.next_idx == idx:
+ out = fit_letterbox(frame, self.width, self.height)
+ self.next_idx += 1
+ return self.blank.copy() if out is None else out
+
+ def close(self) -> None:
+ self.cap.release()
+
+
+class _StreamWriter:
+ """Raw BGR frames piped into ffmpeg, which encodes H.264 and muxes the
+ source video's audio over the rendered frame window.
+
+ Falls back to a plain (silent) OpenCV writer when ffmpeg is unavailable.
+ """
+
+ def __init__(
+ self,
+ out_path: Path,
+ fps: float,
+ width: int,
+ height: int,
+ *,
+ audio_src: Path | None = None,
+ audio_start_s: float = 0.0,
+ audio_dur_s: float | None = None,
+ ):
+ self.out_path = Path(out_path)
+ self.out_path.parent.mkdir(parents=True, exist_ok=True)
+ self.proc = None
+ self.writer = None
+ ffmpeg = shutil.which("ffmpeg")
+ if ffmpeg is not None:
+ cmd = [
+ ffmpeg, "-y", "-loglevel", "error",
+ "-f", "rawvideo", "-pix_fmt", "bgr24",
+ "-s", f"{width}x{height}", "-r", f"{fps}",
+ "-i", "-",
+ ]
+ if audio_src is not None and Path(audio_src).exists():
+ cmd += ["-ss", f"{audio_start_s:.6f}"]
+ if audio_dur_s is not None:
+ cmd += ["-t", f"{audio_dur_s:.6f}"]
+ # `0:v` is the rendered frames, `1:a:0?` the source audio if
+ # it has any; -shortest stops the audio with the last frame.
+ cmd += ["-i", str(audio_src), "-map", "0:v:0", "-map", "1:a:0?",
+ "-c:a", "aac", "-b:a", "192k", "-shortest"]
+ cmd += ["-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18",
+ "-preset", "medium", "-movflags", "+faststart",
+ str(self.out_path)]
+ self.proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)
+ else:
+ import cv2
+
+ self.writer = cv2.VideoWriter(
+ str(self.out_path), cv2.VideoWriter_fourcc(*"mp4v"),
+ float(fps), (width, height),
+ )
+ if not self.writer.isOpened():
+ raise RuntimeError(f"Cannot open VideoWriter for {self.out_path}")
+
+ def write(self, frame: np.ndarray) -> None:
+ if self.proc is not None:
+ self.proc.stdin.write(frame.tobytes())
+ else:
+ self.writer.write(frame)
+
+ def close(self) -> Path:
+ if self.proc is not None:
+ self.proc.stdin.close()
+ if self.proc.wait() != 0:
+ raise RuntimeError(f"ffmpeg failed encoding {self.out_path}")
+ if self.writer is not None:
+ self.writer.release()
+ return self.out_path
+
+
+def _grid2x2(tl: np.ndarray, tr: np.ndarray, bl: np.ndarray, br: np.ndarray) -> np.ndarray:
+ return np.vstack([np.hstack([tl, tr]), np.hstack([bl, br])])
+
+
+def render_run_videos(
+ anim: RetargetedAnimation,
+ asset,
+ calibration,
+ source_video: Path,
+ out_dir: Path,
+ *,
+ verbose: bool = True,
+) -> dict[str, Path]:
+ """Write the four result videos into `out_dir` (videos/ of a run)."""
+ out_dir = Path(out_dir)
+ out_dir.mkdir(parents=True, exist_ok=True)
+ width, height = panel_size(source_video)
+ fps = anim.fps
+
+ # One camera and one floor over human joints, Mixamo joints and the mesh,
+ # so the three rendered panels agree on where everything stands. Human,
+ # Mixamo Skeleton and character all live in the calibrated rig world, so
+ # the shared anatomical axes come from the calibration.
+ axes = calibration.rig_axes
+ stride = max(1, anim.n // CAMERA_MESH_SAMPLES)
+ probe = skin_frames(asset, anim, list(range(0, anim.n, stride)))
+ frames_pts = [np.concatenate(v, axis=0) for v in probe]
+ del probe
+ soles = np.array([float(pts[:, 1].min()) for pts in frames_pts])
+ mesh_pts = np.concatenate(frames_pts, axis=0)
+ del frames_pts
+
+ cam = build_perspective_camera(
+ axes,
+ np.concatenate(
+ [all_points(anim.human_pos), all_points(anim.mixamo_pos), mesh_pts],
+ axis=0,
+ ),
+ width,
+ height,
+ fov_deg=CAMERA_FOV_DEG,
+ azimuth_deg=CAMERA_AZIMUTH_DEG,
+ elevation_deg=CAMERA_ELEVATION_DEG,
+ fill=CAMERA_FILL,
+ )
+ del mesh_pts
+ hips = np.stack([anim.mixamo_pos[i]["mixamorig:Hips"] for i in range(anim.n)])
+ ground = Ground(
+ y=float(np.percentile(soles, FLOOR_PERCENTILE)),
+ cx=float(np.median(hips[:, 0])),
+ cz=float(np.median(hips[:, 2])),
+ )
+ if verbose:
+ print(f" camera eye {np.round(cam.eye, 3)} fov {cam.fov_deg:.1f} deg"
+ f" floor y {ground.y:+.4f} m")
+
+ # Camera and floor are both fixed, so the floor is drawn once per theme
+ # and every frame starts as a copy of it.
+ backdrop_dark = ground_backdrop(cam, ground, width, height, DARK_THEME)
+ backdrop_grid = ground_backdrop(cam, ground, width, height, GRID_THEME)
+
+ faces = [m.faces for m in asset.meshes]
+ colors = [
+ m.face_colors
+ if m.face_colors is not None
+ else MESH_COLORS_BGR.get(m.name, np.array([150.0, 150.0, 150.0]))
+ for m in asset.meshes
+ ]
+
+ audio_start = anim.frames[0] / fps
+ audio_dur = anim.n / fps
+
+ def writer(name: str, w: int, h: int) -> _StreamWriter:
+ return _StreamWriter(
+ out_dir / name, fps, w, h,
+ audio_src=source_video,
+ audio_start_s=audio_start,
+ audio_dur_s=audio_dur,
+ )
+
+ character = calibration.fbx_path.stem
+ writers = {
+ "human_skeleton": writer("human_skeleton.mp4", width, height),
+ "mixamo_skeleton": writer("mixamo_skeleton.mp4", width, height),
+ "mixamo_character": writer("mixamo_character.mp4", width, height),
+ "compare": writer("compare.mp4", 2 * width, 2 * height),
+ }
+ originals = _OriginalReader(source_video, width, height)
+ try:
+ for i in range(anim.n):
+ t = anim.frames[i] / fps
+ stamp = f"f{anim.frames[i]} t={t:.2f}s"
+
+ p_hum = backdrop_dark.copy()
+ pos = anim.human_pos[i]
+ draw_shadow(p_hum, cam, np.stack(list(pos.values())), ground, DARK_THEME)
+ draw_skeleton(p_hum, pos, HUMAN_EDGES, cam, 2, DARK_THEME.joint)
+ label_panel(p_hum, f"Human Skeleton {stamp}")
+
+ p_mix = backdrop_dark.copy()
+ pos = anim.mixamo_pos[i]
+ draw_shadow(p_mix, cam, np.stack(list(pos.values())), ground, DARK_THEME)
+ draw_skeleton(p_mix, pos, MIXAMO_EDGES, cam, 3, DARK_THEME.joint)
+ label_panel(p_mix, f"Mixamo Skeleton {stamp}")
+
+ p_char = backdrop_grid.copy()
+ verts = skin_frames(asset, anim, [i])[0]
+ draw_shadow(p_char, cam, np.concatenate(verts, axis=0), ground, GRID_THEME)
+ draw_mesh(p_char, verts, faces, colors, cam, CHARACTER_AMBIENT)
+ label_panel(p_char, f"{character} {stamp}")
+
+ p_orig = originals.frame(anim.frames[i])
+ label_panel(p_orig, f"Original Video {stamp}")
+
+ writers["human_skeleton"].write(p_hum)
+ writers["mixamo_skeleton"].write(p_mix)
+ writers["mixamo_character"].write(p_char)
+ writers["compare"].write(_grid2x2(p_orig, p_mix, p_hum, p_char))
+
+ if verbose and (i == 0 or (i + 1) % 100 == 0 or i == anim.n - 1):
+ print(f" render {i + 1}/{anim.n}")
+ finally:
+ originals.close()
+
+ out = {name: w.close() for name, w in writers.items()}
+ if verbose:
+ for path in out.values():
+ print(f"saved {path}")
+ return out
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..379490b
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,23 @@
+[project]
+name = "motion2mixamorig"
+version = "0.1.0"
+description = "Retarget human motion from video to an Adobe Mixamo-rigged 3D character."
+requires-python = ">=3.10"
+dependencies = [
+ "numpy",
+ "opencv-python",
+ "einops",
+ # Video -> 3D human skeleton. `preproc` = person detection/tracking,
+ # `rtmpose` = the ungated 2D-pose backend the pipeline uses.
+ "gvhmr[preproc,rtmpose]>=1.6",
+]
+
+[project.scripts]
+m2mr = "motion2mixamorig.cli:main"
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+include = ["motion2mixamorig*"]
diff --git a/weights/.gitkeep b/weights/.gitkeep
new file mode 100644
index 0000000..e69de29