Retarget human motion from video to an Adobe Mixamo-rigged 3D character.
Python
Add still-image pose input via --image
MaxLiu631d752f925a
diff --git a/AGENTS.md b/AGENTS.mdindex 21d5d59..4c0d52f 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -1,7 +1,8 @@# AGENTS.mdMotion2MixamoRig drives an Adobe Mixamo-rigged 3D character with human motion-extracted from a monocular video. CLI: `m2mr`.+extracted from a monocular video, or a static 3D pose from a still photo.+CLI: `m2mr`.## Setup@@ -14,13 +15,15 @@ pip install --upgrade pip setuptools wheel \&& pip install -e .```-Three things must be supplied by the user (all git-ignored, licensing-gated):+These must be supplied by the user (all git-ignored, licensing-gated).+A run needs SMPL-X, a Mixamo character, and either a video or a still:| What | Where to put it | Source || --- | --- | --- || SMPL-X body model | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | https://smpl-x.is.tue.mpg.de/ (registration) || Mixamo character | `assets/mixamo/<Name>.fbx` (FBX Binary, T-pose) | https://www.mixamo.com/ (Adobe account) || Input video | `assets/video/<name>.mp4` | one clearly visible person (group shots fail preflight) |+| Input still | `assets/image/<name>.jpg` | one clearly visible person, head to toe (optional; `--image`) |Inference weights (~5 GB) auto-download into `weights/` on the first run.@@ -30,26 +33,28 @@ Inference weights (~5 GB) auto-download into `weights/` on the first run.m2mr doctor # verify assets + dependencies; prints what is missing and where to get itm2mr run # latest video in assets/video/ + assets/mixamo/Y_Bot.fbxm2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Y_Bot.fbxm2mr run --skeleton outputs/<run>/skeleton_motion.npz --rig ... # reuse extraction, swap rig```-`--video` and `--rig` combine freely. Extraction is the slow step; reuse it-with `--skeleton` when only the rig changes.+`--video` / `--image` and `--rig` combine freely (not `--video` with `--image`).+Extraction is the slow step; reuse it with `--skeleton` when only the rig changes.## Outputs-Each run writes `outputs/<YYYYMMDD_HHMMSS>_<video>/`:+Each run writes `outputs/<YYYYMMDD_HHMMSS>_<source>/`:- `run.json` — inputs, timings, output listing- `skeleton_motion.npz` — 3D human skeleton (24 SMPL joints, Y-up meters)- `mixamo_rotations.npz` — per-bone animation rotations (quaternion wxyz; convention string inside)- `mixamo_character.glb` — skinned character + clip (glTF 2.0; Blender: File → Import → glTF 2.0)-- `videos/human_skeleton.mp4`, `videos/mixamo_skeleton.mp4`, `videos/mixamo_character.mp4`, `videos/compare.mp4` (2x2: original / mixamo skeleton / human skeleton / character)+- Video in: `videos/human_skeleton.mp4`, `videos/mixamo_skeleton.mp4`, `videos/mixamo_character.mp4`, `videos/compare.mp4` (2x2: original / mixamo skeleton / human skeleton / character)+- Photo in: the same four views as PNG under `images/`, plus `images/before_after_360_compare.mp4` (left original / right 10° orbit of the rig)## Code map- `motion2mixamorig/paths.py` — all filesystem locations-- `motion2mixamorig/extract.py` — video -> 3D human skeleton (GVHMR)+- `motion2mixamorig/extract.py` — video or still -> 3D human skeleton (GVHMR)- `motion2mixamorig/mixamo/` — FBX parsing, FK, retargeting, skinning, rendering- `motion2mixamorig/pipeline.py` — one run end to end- `motion2mixamorig/cli.py` — `m2mr doctor` / `m2mr run`diff --git a/README.ar.md b/README.ar.mdindex ed76b60..8cd780b 100644--- a/README.ar.md+++ b/README.ar.md@@ -6,9 +6,9 @@<a href="README.zh-CN.md">中文</a> · <a href="README.md">English</a> · <a href="README.ko.md">한국어</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.ru.md">Русский</a> · العربية</p>-ينقل حركة شخص في مقطع فيديو إلى شخصية ثلاثية الأبعاد بهيكل تحريك Adobe Mixamo.+ينقل حركة شخص في مقطع فيديو — أو الوضعية في صورة ثابتة — إلى شخصية ثلاثية الأبعاد بهيكل تحريك Adobe Mixamo.-للمطوّرين في مجال الألعاب: قدّم فيديو حركة لـ**شخص واحد** وشخصية Mixamo؛ يكتب مقاطع معاينة وملف شخصية مكسوّة (`.glb`) يمكن وضعه في Blender أو Unity.+للمطوّرين في مجال الألعاب: قدّم فيديو حركة لـ**شخص واحد** (أو صورة) وشخصية Mixamo؛ يكتب معاينات وملف شخصية مكسوّة (`.glb`) يمكن وضعه في Blender أو Unity.ينبغي للوكلاء البرمجيين (Agents) البدء من [`AGENTS.md`](AGENTS.md).@@ -46,25 +46,26 @@ pip install --upgrade pip setuptools wheel \يُفضَّل تثبيت ffmpeg (macOS: `brew install ffmpeg`، Ubuntu: `apt install ffmpeg`).يعمل مسار المعالجة بدونه، لكن مقاطع الإخراج ستكون **صامتة** ولن تُشغَّل في المتصفح.-## قبل التشغيل: ضع ثلاثة أشياء في `assets/`+## قبل التشغيل: ضع هذه في `assets/`-هذا المستودع لا يتضمّن هذه الملفات؛ ينبغي جلبها بنفسك. يجب أن يطابق اسم ملف SMPL-X الجدول حرفيًا؛ أما أسماء FBX والفيديو فحرة.+هذا المستودع لا يتضمّن هذه الملفات؛ ينبغي جلبها بنفسك. يجب أن يطابق اسم ملف SMPL-X الجدول حرفيًا؛ أما أسماء FBX والفيديو والصورة فحرة. يحتاج التشغيل إلى SMPL-X وشخصية Mixamo و**إما** فيديو **أو** صورة.| ماذا | أين | من أين ||---|---|---|| نموذج جسم SMPL-X | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | سجّل في [SMPL-X](https://smpl-x.is.tue.mpg.de/) ثم نزّله || ملف FBX لشخصية Mixamo | `assets/mixamo/Y_Bot.fbx` | بحساب Adobe نزّل Y Bot (أو أي شخصية بهيكل Mixamo) من [Mixamo](https://www.mixamo.com) || فيديو الحركة | `assets/video/<your_clip>.mp4` | مقطع فيه **شخص واحد بالضبط** ظاهر بوضوح، **من الرأس إلى القدمين داخل الإطار طوال المقطع** (يمكن وضع عدة ملفات؛ كل تشغيل يستخدم آخر ملف أُضيف) |+| صورة وضعية (اختياري) | `assets/image/<your_photo>.jpg` | صورة فيها **شخص واحد بالضبط** ظاهر بوضوح، **من الرأس إلى القدمين داخل الإطار**. مع `--image` |ملاحظة: ملف التنزيل من Mixamo اسمه `Y Bot.fbx` (**مسافة**)، ويختلف بحرف واحد عن `Y_Bot.fbx` (**شرطة سفلية**) في الجدول.أعد تسميته بالشرطة السفلية فيصبح الهيكل الافتراضي (`m2mr run` دون `--rig`). الإبقاء على المسافة جائز أيضًا — إن غاب `Y_Bot.fbx` يُستخدم أول `.fbx` في `assets/mixamo/`.-يمكن وضع عدة فيديوهات في `assets/video/`. دون `--video` يستخدم التشغيل **آخر ملف أُضيف إلى هذا المجلد**.-يجب أن يظهر في كل مقطع شخص واحد فقط، **من الرأس إلى القدمين داخل الإطار طوال المقطع** — دون قص عند الحواف. يأخذ `m2mr doctor` / `m2mr run` عيّنات من الإطارات ويتوقفان قبل الاستخراج إن ظهر شخصان.+يمكن وضع عدة فيديوهات في `assets/video/`. دون `--video` أو `--image` يستخدم التشغيل **آخر ملف أُضيف إلى `assets/video/`**؛ إن كان المجلد فارغًا يعود إلى أحدث صورة في `assets/image/`.+يجب أن يظهر في كل مقطع أو صورة شخص واحد فقط، **من الرأس إلى القدمين داخل الإطار** — دون قص عند الحواف. يتوقف `m2mr doctor` / `m2mr run` قبل الاستخراج إن ظهر شخصان.## البدء السريع-النتائج في `outputs/<وقت-التشغيل>_<الفيديو>/`. افتح مجلد `videos/` لمشاهدتها.+النتائج في `outputs/<وقت-التشغيل>_<المصدر>/`. تشغيل الفيديو يكتب `videos/`؛ تشغيل الصورة يكتب `images/`.### 1. تحقق من الأصول والبيئة@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. صورة ثابتة (وضعية ساكنة)++`--image` يستعيد **وضعية ثلاثية الأبعاد ساكنة** من صورة لشخص واحد، ثم ينقلها إلى شخصية Mixamo كما يفعل تشغيل الفيديو. المعاينات أربع صور في `images/` (نفس المناظر كتشغيل الفيديو) إضافة إلى مقطع المدار يمين/يسار `before_after_360_compare.mp4`. لا تُجمع مع `--video`.++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++`--image` دون مسار يستخدم آخر ملف أُضيف إلى `assets/image/`.+## المخرجاتكل `m2mr run` ينشئ مجلدًا باسم **وقت بدء الأمر**:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # هيكل إنسان ثلاثي الأبعاد (يُعاد استخدامه عند تبديل الهيكل؛ بلا إعادة استخراج)├── mixamo_rotations.npz # دورانات عظام Mixamo لكل عظم├── mixamo_character.glb # شخصية مكسوّة + تحريك، استيراد إلى Blender / Unity-└── videos/ # نسبة العرض إلى الارتفاع نفسها كفيديو الإدخال- ├── human_skeleton.mp4 # هيكل الإنسان- ├── mixamo_skeleton.mp4 # هيكل Mixamo- ├── mixamo_character.mp4 # شخصية Mixamo+└── videos/ # إدخال فيديو: نسبة العرض إلى الارتفاع نفسها كالمقطع+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 2×2: الأصل (أعلى اليسار) / هيكل Mixamo (أعلى اليمين) / هيكل الإنسان (أسفل اليسار) / الشخصية (أسفل اليمين)++# إدخال الصورة يكتب images/ بنفس المناظر الأربعة:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# إضافة إلى before_after_360_compare.mp4 (اليسار الأصل / اليمين مدار 10°)```افتح `mixamo_character.glb` في Blender: File → Import → glTF 2.0 (.glb/.gltf). إن وُجّهت الكاميرا إلى فراغ، حدّد الشخصية ثم View → Frame Selected. مجموعة الكرات على الرأس واليدين هي عرض للعظام وليست الشبكة: حدّد الهيكل (Armature) وأوقف Shapes تحت Armature → Viewport Display. اضبط إطار نهاية الخط الزمني على طول المقطع ثم شغّل. قارن مع `videos/mixamo_character.mp4` من التشغيل نفسه.diff --git a/README.de.md b/README.de.mdindex adc0d8a..c008a1c 100644--- a/README.de.md+++ b/README.de.md@@ -6,9 +6,9 @@<a href="README.zh-CN.md">中文</a> · <a href="README.md">English</a> · <a href="README.ko.md">한국어</a> · <a href="README.ja.md">日本語</a> · Deutsch · <a href="README.ru.md">Русский</a> · <a href="README.ar.md">العربية</a></p>-Überträgt die Bewegung einer Person aus einem Video auf einen Adobe-Mixamo-geriggten 3D-Charakter.+Überträgt die Bewegung einer Person aus einem Video — oder die Pose aus einem Foto — auf einen Adobe-Mixamo-geriggten 3D-Charakter.-Für Spieleentwickler: ein Video mit **genau einer Person** und ein Mixamo-Charakter; Ausgabe sind Vorschauvideos plus eine Charakterdatei mit Skinning (`.glb`) für Blender oder Unity.+Für Spieleentwickler: ein Video mit **genau einer Person** (oder ein Still) und ein Mixamo-Charakter; Ausgabe sind Vorschauen plus eine Charakterdatei mit Skinning (`.glb`) für Blender oder Unity.Agents beginnen mit [`AGENTS.md`](AGENTS.md).@@ -46,25 +46,26 @@ Inferenzgewichte (GVHMR usw.) werden hier nicht geladen. Der erste `m2mr run` ziffmpeg wird empfohlen (macOS: `brew install ffmpeg`, Ubuntu: `apt install ffmpeg`).Ohne ffmpeg läuft die Pipeline, aber die Ausgabevideos sind **stumm** und spielen im Browser nicht.-## Vor dem Lauf: drei Dinge nach `assets/`+## Vor dem Lauf: dies nach `assets/` legen-Dieses Repository enthält diese Dateien nicht; sie müssen selbst besorgt werden. Der SMPL-X-Dateiname muss exakt zur Tabelle passen; FBX- und Videonamen sind frei.+Dieses Repository enthält diese Dateien nicht; sie müssen selbst besorgt werden. Der SMPL-X-Dateiname muss exakt zur Tabelle passen; FBX-, Video- und Bildnamen sind frei. Ein Lauf braucht SMPL-X, einen Mixamo-Charakter und **entweder** ein Video **oder** ein Still.| Was | Wohin | Woher ||---|---|---|| SMPL-X-Körpermodell | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | Bei [SMPL-X](https://smpl-x.is.tue.mpg.de/) registrieren und herunterladen || Mixamo-Charakter-FBX | `assets/mixamo/Y_Bot.fbx` | Mit Adobe-Konto Y Bot (oder einen anderen Mixamo-geriggten Charakter) von [Mixamo](https://www.mixamo.com) laden || Bewegungsvideo | `assets/video/<your_clip>.mp4` | Clip mit **genau einer** klar sichtbaren Person, **Kopf bis Fuß die ganze Zeit im Bild** (mehrere Dateien möglich; jeder Lauf nimmt die zuletzt hinzugefügte) |+| Pose-Foto (optional) | `assets/image/<your_photo>.jpg` | Foto mit **genau einer** klar sichtbaren Person, **Kopf bis Fuß im Bild**. Mit `--image` |Hinweis: Der Download von Mixamo heißt `Y Bot.fbx` (**Leerzeichen**) und weicht um ein Zeichen von `Y_Bot.fbx` (**Unterstrich**) in der Tabelle ab.Umbenennen zum Unterstrich macht es zum Standard-Rig (`m2mr run` ohne `--rig`). Der Name mit Leerzeichen ist ebenfalls in Ordnung — fehlt `Y_Bot.fbx`, wird die erste `.fbx` in `assets/mixamo/` verwendet.-In `assets/video/` können mehrere Videos liegen. Ohne `--video` nimmt der Lauf die **zuletzt in diesen Ordner gelegte** Datei.-Jeder Clip darf nur eine Person zeigen, **Kopf bis Fuß die ganze Zeit im Bild** — nichts am Rand abgeschnitten. `m2mr doctor` / `m2mr run` prüfen Frames stichprobenartig und stoppen vor der Extraktion, wenn zwei Personen zu sehen sind.+In `assets/video/` können mehrere Videos liegen. Ohne `--video` oder `--image` nimmt der Lauf die **zuletzt in `assets/video/` gelegte** Datei; ist der Ordner leer, das neueste Still in `assets/image/`.+Jeder Clip oder jedes Foto darf nur eine Person zeigen, **Kopf bis Fuß im Bild** — nichts am Rand abgeschnitten. `m2mr doctor` / `m2mr run` stoppen vor der Extraktion, wenn zwei Personen zu sehen sind.## Schnellstart-Ergebnisse liegen in `outputs/<Laufzeit>_<Video>/`. Videos im Unterordner `videos/` ansehen.+Ergebnisse liegen in `outputs/<Laufzeit>_<Quelle>/`. Videoläufe schreiben `videos/`; Fotoläufe schreiben `images/`.### 1. Assets und Umgebung prüfen@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. Ein Still (statische Pose)++`--image` rekonstruiert eine **statische 3D-Pose** aus einem Foto mit einer Person und überträgt sie wie ein Videolauf auf den Mixamo-Charakter. Vorschauen sind vier Stills in `images/` (gleiche Ansichten wie der Videolauf) plus der Links/Rechts-Orbit-Clip `before_after_360_compare.mp4`. Nicht zusammen mit `--video` verwenden.++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++`--image` ohne Pfad nimmt die zuletzt nach `assets/image/` gelegte Datei.+## AusgabenJeder `m2mr run` legt ein Verzeichnis an, benannt nach **Startzeit des Befehls**:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # 3D-Körperskelett (beim Rig-Wechsel wiederverwenden; keine Neu-Extraktion)├── mixamo_rotations.npz # Mixamo-Rotationen pro Bone├── mixamo_character.glb # Charakterdatei mit Skinning + Animation, Import in Blender / Unity-└── videos/ # gleiches Seitenverhältnis wie das Eingangsvideo- ├── human_skeleton.mp4 # Körperskelett- ├── mixamo_skeleton.mp4 # Mixamo-Rig-Skelett- ├── mixamo_character.mp4 # Mixamo-Rig-Charakter+└── videos/ # Videoeingabe: gleiches Seitenverhältnis wie der Clip+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 2×2: Original (oben links) / Mixamo-Skelett (oben rechts) / Körperskelett (unten links) / Charakter (unten rechts)++# Fotoeingabe schreibt images/ mit denselben vier Ansichten:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# plus before_after_360_compare.mp4 (links Original / rechts 10°-Orbit)````mixamo_character.glb` in Blender öffnen: File → Import → glTF 2.0 (.glb/.gltf). Zeigt die Kamera ins Leere, den Charakter wählen und View → Frame Selected. Die Kugeln an Kopf und Händen sind die Knochenanzeige, nicht das Mesh: Armature wählen und unter Armature → Viewport Display die Shapes ausschalten. Das letzte Timeline-Frame auf die Cliplänge setzen, dann abspielen. Mit `videos/mixamo_character.mp4` desselben Laufs vergleichen.diff --git a/README.ja.md b/README.ja.mdindex 90ce7b4..8f1bfd0 100644--- a/README.ja.md+++ b/README.ja.md@@ -6,9 +6,9 @@<a href="README.zh-CN.md">中文</a> · <a href="README.md">English</a> · <a href="README.ko.md">한국어</a> · 日本語 · <a href="README.de.md">Deutsch</a> · <a href="README.ru.md">Русский</a> · <a href="README.ar.md">العربية</a></p>-動画の人物の動きを、Adobe Mixamo rig の 3D キャラクターに移します。+動画の人物の動き — または静止画のポーズ — を、Adobe Mixamo rig の 3D キャラクターに移します。-ゲーム開発者向け:入力は**一人**の動作動画と Mixamo キャラクター。プレビュー動画と、Blender / Unity にそのまま入れられるスキン付きキャラクターファイル(`.glb`)を書き出します。+ゲーム開発者向け:入力は**一人**の動作動画(または写真)と Mixamo キャラクター。プレビューと、Blender / Unity にそのまま入れられるスキン付きキャラクターファイル(`.glb`)を書き出します。Agent は先に [`AGENTS.md`](AGENTS.md) を読んでください。@@ -46,25 +46,26 @@ pip install --upgrade pip setuptools wheel \ffmpeg も推奨です(macOS: `brew install ffmpeg`、Ubuntu: `apt install ffmpeg`)。なくても動きますが、出力動画は**無音**になり、ブラウザでは再生できません。-## 実行前:`assets/` に 3 つのファイルを入れてください+## 実行前:これらを `assets/` に入れてください-このリポジトリにはこれらのファイルは入っていません。自分で用意してください。SMPL-X のファイル名は下表と一字一句同じである必要があります。FBX と動画の名前は何でも構いません。+このリポジトリにはこれらのファイルは入っていません。自分で用意してください。SMPL-X のファイル名は下表と一字一句同じである必要があります。FBX、動画、写真の名前は何でも構いません。1 回の実行には SMPL-X、Mixamo キャラクター、そして**動画か写真のどちらか**が必要です。| 何を | どこへ | どこから ||---|---|---|| SMPL-X ボディモデル | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | [SMPL-X](https://smpl-x.is.tue.mpg.de/) で登録してダウンロード || Mixamo キャラクター FBX | `assets/mixamo/Y_Bot.fbx` | Adobe アカウントで [Mixamo](https://www.mixamo.com) から Y Bot(または他の Mixamo rig キャラクター)をダウンロード || 動作動画 | `assets/video/<your_clip>.mp4` | **一人だけ**、はっきり見え、**頭から足まで全編フレーム内**(複数可、実行ごとに最後に入れたファイルを使用) |+| ポーズ写真(任意) | `assets/image/<your_photo>.jpg` | **一人だけ**、はっきり見え、**頭から足までフレーム内**。`--image` と一緒に使う |注意:Mixamo のダウンロード名は `Y Bot.fbx`(**スペース**)で、表の `Y_Bot.fbx`(**アンダースコア**)と一字違います。アンダースコアにすればデフォルト rig になり、`m2mr run` は `--rig` なしでそれを使います。名前を変えなくても構いません——`Y_Bot.fbx` が無いときは `assets/mixamo/` の最初の `.fbx` を使います。-`assets/video/` には動画を複数置けます。`--video` を指定しないと、**このフォルダに最後に入れた**ファイルを使います。-各クリップは一人だけ、**頭から足まで全編フレーム内**に収まっている必要があります。はみ出してはいけません。`m2mr doctor` / `m2mr run` はフレームをサンプリングし、二人が写っていれば抽出に入らず止まります。+`assets/video/` には動画を複数置けます。`--video` も `--image` も指定しないと、**`assets/video/` に最後に入れた**ファイルを使い、そのフォルダが空なら `assets/image/` の最新の写真に戻ります。+各クリップや写真は一人だけ、**頭から足までフレーム内**に収まっている必要があります。はみ出してはいけません。`m2mr doctor` / `m2mr run` は二人が写っていれば抽出に入らず止まります。## クイックスタート-結果は `outputs/<実行時刻>_<動画名>/` です。動画を見るにはその中の `videos/` を開きます。+結果は `outputs/<実行時刻>_<入力名>/` です。動画入力は `videos/`、写真入力は `images/` に書き出します。### 1. アセットと環境を確認@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. 静止画(静的ポーズ)++`--image` は**一人**の写真から**静的な 3D ポーズ**を復元し、動画実行と同じ経路で Mixamo キャラクターに載せます。プレビューは `images/` の 4 枚(動画実行と同じ視点)と、左右比較の周回クリップ `before_after_360_compare.mp4` です。`--video` と同時に指定しないでください。++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++パスなしの `--image` は、`assets/image/` に最後に入れたファイルを使います。+## 出力各 `m2mr run` は**コマンド開始時刻**でディレクトリを作ります:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # 人体 3D スケルトン(rig 差し替え時に再利用、再抽出不要)├── mixamo_rotations.npz # Mixamo ボーンごとの回転├── mixamo_character.glb # スキン付きキャラクター + アニメーション、Blender / Unity に直接インポート-└── videos/ # 入力動画と同じアスペクト比- ├── human_skeleton.mp4 # 人体スケルトン- ├── mixamo_skeleton.mp4 # Mixamo rig スケルトン- ├── mixamo_character.mp4 # Mixamo rig キャラクター+└── videos/ # 動画入力:クリップと同じアスペクト比+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 2×2:元映像(左上)/ Mixamo スケルトン(右上)/ 人体スケルトン(左下)/ キャラクター(右下)++# 写真入力は images/ に同じ 4 ビューを書き出します:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# 加えて before_after_360_compare.mp4(左が原画 / 右が 10° 周回)````mixamo_character.glb` は Blender で File → Import → glTF 2.0 (.glb/.gltf) で開きます。カメラが空を向いているときはキャラクターを選び、View → Frame Selected。頭と手の球の塊はメッシュではなくボーン表示です:アーマチュアを選び、Armature → Viewport Display で Shapes をオフにします。タイムラインの終了フレームをクリップ長に合わせて再生。同じ実行の `videos/mixamo_character.mp4` と見比べてください。diff --git a/README.ko.md b/README.ko.mdindex 1eb8a52..10d87aa 100644--- a/README.ko.md+++ b/README.ko.md@@ -6,9 +6,9 @@<a href="README.zh-CN.md">中文</a> · <a href="README.md">English</a> · 한국어 · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.ru.md">Русский</a> · <a href="README.ar.md">العربية</a></p>-영상 속 사람의 동작을 Adobe Mixamo rig 3D 캐릭터로 옮깁니다.+영상 속 사람의 동작 — 또는 사진 속 자세 — 를 Adobe Mixamo rig 3D 캐릭터로 옮깁니다.-게임 개발자를 위한 도구입니다. **한 사람만** 나오는 동작 영상과 Mixamo 캐릭터를 넣으면 미리보기 영상과, Blender / Unity에 바로 넣을 수 있는 스키닝된 캐릭터 파일(`.glb`)을 생성합니다.+게임 개발자를 위한 도구입니다. **한 사람만** 나오는 동작 영상(또는 사진)과 Mixamo 캐릭터를 넣으면 미리보기와, Blender / Unity에 바로 넣을 수 있는 스키닝된 캐릭터 파일(`.glb`)을 생성합니다.Agent는 먼저 [`AGENTS.md`](AGENTS.md)를 읽어 주세요.@@ -46,25 +46,26 @@ pip install --upgrade pip setuptools wheel \ffmpeg도 권장합니다(macOS: `brew install ffmpeg`, Ubuntu: `apt install ffmpeg`).없어도 실행되지만, 출력 영상에 **소리가 없고** 브라우저에서 바로 재생되지 않습니다.-## 실행 전: `assets/`에 세 가지를 넣으세요+## 실행 전: 이것을 `assets/`에 넣으세요-이 저장소는 해당 파일을 포함하지 않습니다. 직접 준비해야 합니다. SMPL-X 파일명은 아래 표와 한 글자도 같아야 합니다. FBX와 영상 이름은 상관없습니다.+이 저장소는 해당 파일을 포함하지 않습니다. 직접 준비해야 합니다. SMPL-X 파일명은 아래 표와 한 글자도 같아야 합니다. FBX, 영상, 사진 이름은 상관없습니다. 한 번의 실행에는 SMPL-X, Mixamo 캐릭터, 그리고 **영상 또는 사진 중 하나**가 필요합니다.| 무엇 | 어디에 | 어디서 ||---|---|---|| SMPL-X 바디 모델 | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | [SMPL-X](https://smpl-x.is.tue.mpg.de/)에서 등록한 뒤 다운로드 || Mixamo 캐릭터 FBX | `assets/mixamo/Y_Bot.fbx` | Adobe 계정으로 [Mixamo](https://www.mixamo.com)에서 Y Bot(또는 다른 Mixamo rig 캐릭터) 다운로드 || 동작 영상 | `assets/video/<your_clip>.mp4` | **사람 한 명만**, 잘 보이고, **머리부터 발끝까지 영상 내내 화면 안에** (여러 파일 가능, 실행마다 가장 최근에 넣은 것 사용) |+| 자세 사진(선택) | `assets/image/<your_photo>.jpg` | **사람 한 명만**, 잘 보이고, **머리부터 발끝까지 화면 안에**. `--image`와 함께 사용 |주의: Mixamo에서 받은 파일은 `Y Bot.fbx`(**공백**)이고, 표의 `Y_Bot.fbx`(**밑줄**)과 한 글자 다릅니다.밑줄로 바꾸면 기본 rig가 되어 `m2mr run`에 `--rig` 없이 씁니다. 이름을 안 바꿔도 됩니다 — `Y_Bot.fbx`가 없으면 `assets/mixamo/`의 첫 `.fbx`를 씁니다.-`assets/video/`에는 영상을 여러 개 넣을 수 있습니다. `--video`를 지정하지 않으면 **이 폴더에 가장 마지막에 넣은** 파일을 씁니다.-각 영상에는 사람이 한 명만 나와야 하고, **머리부터 발끝까지 영상 내내 화면 안**에 있어야 하며 화면 밖으로 나가면 안 됩니다. `m2mr doctor` / `m2mr run`이 프레임을 샘플링해 두 명이 보이면 추출 전에 멈춥니다.+`assets/video/`에는 영상을 여러 개 넣을 수 있습니다. `--video`나 `--image`를 지정하지 않으면 **`assets/video/`에 가장 마지막에 넣은** 파일을 쓰고, 그 폴더가 비어 있으면 `assets/image/`의 최신 사진을 씁니다.+각 영상이나 사진에는 사람이 한 명만 나와야 하고, **머리부터 발끝까지 화면 안**에 있어야 하며 화면 밖으로 나가면 안 됩니다. `m2mr doctor` / `m2mr run`이 두 명이 보이면 추출 전에 멈춥니다.## 빠른 시작-결과는 `outputs/<실행시각>_<영상명>/`에 있습니다. 영상은 그 안의 `videos/`를 여세요.+결과는 `outputs/<실행시각>_<입력명>/`에 있습니다. 영상 입력은 `videos/`, 사진 입력은 `images/`에 씁니다.### 1. 자산과 환경 확인@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. 사진(정적 자세)++`--image`는 **한 사람** 사진에서 **정적 3D 자세**를 복원한 뒤, 영상 실행과 같은 경로로 Mixamo 캐릭터에 옮깁니다. 미리보기는 `images/`의 네 장(영상 실행과 같은 뷰)과 좌우 비교 궤도 클립 `before_after_360_compare.mp4`입니다. `--video`와 함께 쓰지 마세요.++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++경로 없이 `--image`만 쓰면 `assets/image/`에 가장 최근에 넣은 파일을 씁니다.+## 출력각 `m2mr run`은 **명령이 시작된 시각**으로 디렉터리를 만듭니다:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # 인체 3D 스켈레톤 (rig 교체 시 재사용, 추출 생략)├── mixamo_rotations.npz # Mixamo 본별 회전├── mixamo_character.glb # 스키닝된 캐릭터 + 애니메이션, Blender / Unity에 바로 임포트-└── videos/ # 입력 영상과 같은 화면비- ├── human_skeleton.mp4 # 인체 스켈레톤- ├── mixamo_skeleton.mp4 # Mixamo rig 스켈레톤- ├── mixamo_character.mp4 # Mixamo rig 캐릭터+└── videos/ # 영상 입력: 클립과 같은 화면비+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 2×2: 원본(좌상) / Mixamo 스켈레톤(우상) / 인체 스켈레톤(좌하) / 캐릭터(우하)++# 사진 입력은 images/에 같은 네 뷰를 씁니다:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# 그리고 before_after_360_compare.mp4 (왼쪽 원본 / 오른쪽 10° 궤도)````mixamo_character.glb`는 Blender에서 File → Import → glTF 2.0 (.glb/.gltf)로 엽니다. 카메라가 빈 곳을 보고 있으면 캐릭터를 선택한 뒤 View → Frame Selected. 머리와 손의 구체들은 메시가 아니라 본 표시입니다. Armature를 선택하고 Armature → Viewport Display에서 Shapes를 끄세요. 타임라인 끝 프레임을 클립 길이에 맞춘 뒤 재생하세요. 같은 실행의 `videos/mixamo_character.mp4`와 비교하면 됩니다.diff --git a/README.md b/README.mdindex c1e09a0..8f76537 100644--- a/README.md+++ b/README.md@@ -6,9 +6,9 @@<a href="README.zh-CN.md">中文</a> · English · <a href="README.ko.md">한국어</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.ru.md">Русский</a> · <a href="README.ar.md">العربية</a></p>-Transfer the motion of a person in a video onto an Adobe Mixamo-rigged 3D character.+Transfer the motion of a person in a video — or the pose in a still photo — onto an Adobe Mixamo-rigged 3D character.-For game developers: give it a **single-person** action video and a Mixamo character; it writes preview videos plus a skinned character file (`.glb`) you can drop into Blender or Unity.+For game developers: give it a **single-person** action video (or a still) and a Mixamo character; it writes preview videos plus a skinned character file (`.glb`) you can drop into Blender or Unity.Agents should start with [`AGENTS.md`](AGENTS.md).@@ -46,25 +46,26 @@ Inference weights (GVHMR and so on) are not downloaded here. The first `m2mr runffmpeg is recommended (macOS: `brew install ffmpeg`, Ubuntu: `apt install ffmpeg`).The pipeline runs without it, but the output videos will be **silent** and will not play in a browser.-## Before you run: put three things in `assets/`+## Before you run: put these in `assets/`-This repository does not ship these files; you have to fetch them. The SMPL-X filename must match the table exactly; FBX and video names can be anything.+This repository does not ship these files; you have to fetch them. The SMPL-X filename must match the table exactly; FBX, video, and image names can be anything. A run needs SMPL-X, a Mixamo character, and **either** a video **or** a still.| What | Where | Where to get it ||---|---|---|| SMPL-X body model | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | Register at [SMPL-X](https://smpl-x.is.tue.mpg.de/) and download it || Mixamo character FBX | `assets/mixamo/Y_Bot.fbx` | With an Adobe account, download Y Bot (or any other Mixamo-rigged character) from [Mixamo](https://www.mixamo.com) || Action video | `assets/video/<your_clip>.mp4` | A clip with **exactly one** clearly visible person, **head to toe in frame the whole time** (you may keep several files; each run uses the most recently added) |+| Pose still (optional) | `assets/image/<your_photo>.jpg` | A photo with **exactly one** clearly visible person, **head to toe in frame**. Use with `--image` |Note: Mixamo's download is named `Y Bot.fbx` (**space**), one character off from `Y_Bot.fbx` (**underscore**) in the table.Rename it to the underscore form and it becomes the default rig (`m2mr run` without `--rig`). Leaving the space is fine too — if `Y_Bot.fbx` is missing, the first `.fbx` in `assets/mixamo/` is used.-`assets/video/` can hold several videos. Without `--video`, the run uses the file **most recently added** to that folder.-Each clip must show only one person, **head to toe in frame for the whole clip** — nothing cropped at the edges. `m2mr doctor` / `m2mr run` sample the frames and stop before extraction if two people are in view.+`assets/video/` can hold several videos. Without `--video` or `--image`, the run uses the file **most recently added** to `assets/video/`; if that folder is empty it falls back to the latest still in `assets/image/`.+Each clip or photo must show only one person, **head to toe in frame** — nothing cropped at the edges. `m2mr doctor` / `m2mr run` stop before extraction if two people are in view.## Quick Start-Results land in `outputs/<run-time>_<video>/`. Open that folder's `videos/` to watch them.+Results land in `outputs/<run-time>_<source>/`. Video runs write `videos/`; still runs write `images/`.### 1. Check assets and the environment@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. A still photo (static pose)++`--image` recovers a **static 3D pose** from a photo of one person, then retargets it onto the Mixamo character the same way a video run does. Previews are four stills in `images/` (same views as the video run), plus a left/right orbit clip `before_after_360_compare.mp4`. Do not pass `--video` at the same time.++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++`--image` with no path uses the file most recently added to `assets/image/`.+## OutputsEach `m2mr run` creates a directory named by **the time the command started**:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # 3D human skeleton (reuse when swapping rigs; no re-extract)├── mixamo_rotations.npz # per-bone Mixamo rotations├── mixamo_character.glb # skinned character + animation, import into Blender / Unity-└── videos/ # same aspect ratio as the input- ├── human_skeleton.mp4 # human skeleton- ├── mixamo_skeleton.mp4 # Mixamo rig skeleton- ├── mixamo_character.mp4 # Mixamo rig character+└── videos/ # video input: same aspect ratio as the clip+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 2×2: original (TL) / Mixamo skeleton (TR) / human skeleton (BL) / character (BR)++# still input writes images/ instead, same four views:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# plus before_after_360_compare.mp4 (left original / right 10° orbit of the rig)```Open `mixamo_character.glb` in Blender: File → Import → glTF 2.0 (.glb/.gltf). If the camera is framed on empty space, select the character and use View → Frame Selected. The cluster of spheres on the head and hands is the bone display, not the mesh: select the armature and turn off Shapes under Armature → Viewport Display. Set the timeline end frame to the clip length, then play. Compare against `videos/mixamo_character.mp4` from the same run.diff --git a/README.ru.md b/README.ru.mdindex 84760f9..6d536dd 100644--- a/README.ru.md+++ b/README.ru.md@@ -6,9 +6,9 @@<a href="README.zh-CN.md">中文</a> · <a href="README.md">English</a> · <a href="README.ko.md">한국어</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · Русский · <a href="README.ar.md">العربية</a></p>-Переносит движение человека из видео на 3D-персонажа с ригом Adobe Mixamo.+Переносит движение человека из видео — или позу с фото — на 3D-персонажа с ригом Adobe Mixamo.-Для разработчиков игр: видео с **одним человеком** и персонаж Mixamo; на выходе превью-видео и файл персонажа со скином (`.glb`) для Blender или Unity.+Для разработчиков игр: видео с **одним человеком** (или фото) и персонаж Mixamo; на выходе превью и файл персонажа со скином (`.glb`) для Blender или Unity.Агентам следует начать с [`AGENTS.md`](AGENTS.md).@@ -46,25 +46,26 @@ pip install --upgrade pip setuptools wheel \Рекомендуется ffmpeg (macOS: `brew install ffmpeg`, Ubuntu: `apt install ffmpeg`).Без него пайплайн работает, но выходные видео будут **без звука** и не откроются в браузере.-## Перед запуском: положите три вещи в `assets/`+## Перед запуском: положите это в `assets/`-Репозиторий эти файлы не содержит — их нужно получить самостоятельно. Имя файла SMPL-X должно совпадать с таблицей буква в букву; имена FBX и видео любые.+Репозиторий эти файлы не содержит — их нужно получить самостоятельно. Имя файла SMPL-X должно совпадать с таблицей буква в букву; имена FBX, видео и фото любые. Для запуска нужны SMPL-X, персонаж Mixamo и **либо** видео, **либо** фото.| Что | Куда | Откуда ||---|---|---|| Модель тела SMPL-X | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | Зарегистрироваться на [SMPL-X](https://smpl-x.is.tue.mpg.de/) и скачать || FBX-персонаж Mixamo | `assets/mixamo/Y_Bot.fbx` | С Adobe-аккаунтом скачать Y Bot (или другого персонажа с ригом Mixamo) с [Mixamo](https://www.mixamo.com) || Видео с действием | `assets/video/<your_clip>.mp4` | Ролик с **ровно одним** хорошо видимым человеком, **с головы до ног весь ролик в кадре** (можно несколько файлов; каждый запуск берёт последний добавленный) |+| Фото позы (необязательно) | `assets/image/<your_photo>.jpg` | Фото с **ровно одним** хорошо видимым человеком, **с головы до ног в кадре**. Вместе с `--image` |Замечание: скачанный файл Mixamo называется `Y Bot.fbx` (**пробел**), на один символ отличается от `Y_Bot.fbx` (**подчёркивание**) в таблице.Переименуйте с подчёркиванием — это станет ригом по умолчанию (`m2mr run` без `--rig`). Можно не переименовывать: если `Y_Bot.fbx` нет, берётся первый `.fbx` в `assets/mixamo/`.-В `assets/video/` можно класть несколько видео. Без `--video` запуск берёт файл, **последним положенный в эту папку**.-В каждом ролике должен быть один человек, **с головы до ног весь ролик в кадре** — ничего не обрезано по краям. `m2mr doctor` / `m2mr run` выборочно проверяют кадры и останавливаются до извлечения, если в кадре двое.+В `assets/video/` можно класть несколько видео. Без `--video` или `--image` запуск берёт файл, **последним положенный в `assets/video/`**; если папка пуста — последнее фото в `assets/image/`.+В каждом ролике или фото должен быть один человек, **с головы до ног в кадре** — ничего не обрезано по краям. `m2mr doctor` / `m2mr run` останавливаются до извлечения, если в кадре двое.## Быстрый старт-Результаты в `outputs/<время-запуска>_<видео>/`. Видео смотреть в `videos/`.+Результаты в `outputs/<время-запуска>_<источник>/`. Видео-запуск пишет в `videos/`; фото-запуск — в `images/`.### 1. Проверить ассеты и окружение@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<previous-run-dir>/skeleton_motion.npz --rig assets/m2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. Фото (статическая поза)++`--image` восстанавливает **статическую 3D-позу** с фото одного человека и переносит её на персонажа Mixamo так же, как видео-запуск. Превью — четыре кадра в `images/` (те же виды, что у видео) плюс клип орбиты слева/справа `before_after_360_compare.mp4`. Не сочетать с `--video`.++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++`--image` без пути берёт файл, последним положенный в `assets/image/`.+## РезультатыКаждый `m2mr run` создаёт каталог по **времени старта команды**:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # 3D-скелет человека (переиспользовать при смене рига; без повторного извлечения)├── mixamo_rotations.npz # вращения костей Mixamo├── mixamo_character.glb # персонаж со скином + анимация, импорт в Blender / Unity-└── videos/ # то же соотношение сторон, что у входного видео- ├── human_skeleton.mp4 # скелет человека- ├── mixamo_skeleton.mp4 # скелет рига Mixamo- ├── mixamo_character.mp4 # персонаж с ригом Mixamo+└── videos/ # видео-вход: то же соотношение сторон, что у клипа+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 2×2: оригинал (верхний левый) / скелет Mixamo (верхний правый) / скелет человека (нижний левый) / персонаж (нижний правый)++# фото-вход пишет images/ с теми же четырьмя видами:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# плюс before_after_360_compare.mp4 (слева оригинал / справа орбита 10°)````mixamo_character.glb` в Blender: File → Import → glTF 2.0 (.glb/.gltf). Если камера смотрит в пустоту, выберите персонажа и View → Frame Selected. Скопление сфер на голове и руках — отображение костей, не меш: выберите арматуру и отключите Shapes в Armature → Viewport Display. Конечный кадр таймлайна поставьте по длине клипа, затем воспроизведите. Сверяйте с `videos/mixamo_character.mp4` того же запуска.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex edf1cb3..7549727 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -6,9 +6,9 @@中文 · <a href="README.md">English</a> · <a href="README.ko.md">한국어</a> · <a href="README.ja.md">日本語</a> · <a href="README.de.md">Deutsch</a> · <a href="README.ru.md">Русский</a> · <a href="README.ar.md">العربية</a></p>-把视频里人物的动作,转到 Adobe Mixamo rig 的 3D 角色上。+把视频里人物的动作,或一张照片里的姿态,转到 Adobe Mixamo rig 的 3D 角色上。-面向游戏开发者:输入一段**单人**动作视频和一个 Mixamo 角色,输出预览视频,以及可直接丢进 Blender / Unity 的带皮角色文件(`.glb`)。+面向游戏开发者:输入一段**单人**动作视频(或一张人物照片)和一个 Mixamo 角色,输出预览视频,以及可直接丢进 Blender / Unity 的带皮角色文件(`.glb`)。Agent 请先读 [`AGENTS.md`](AGENTS.md)。@@ -45,26 +45,27 @@ pip install --upgrade pip setuptools wheel \建议再装上 ffmpeg(macOS:`brew install ffmpeg`,Ubuntu:`apt install ffmpeg`)。没有它也能跑,但输出的视频会**没有声音**,编码格式也没法在浏览器里直接播放。-## 使用前:把三样东西放到 `assets/`+## 使用前:把这些东西放到 `assets/`-本仓库不附带这些文件,需要你自己准备。SMPL-X 的文件名必须和下表一字不差;FBX 和视频叫什么都行。+本仓库不附带这些文件,需要你自己准备。SMPL-X 的文件名必须和下表一字不差;FBX、视频和照片叫什么都行。一次运行需要 SMPL-X、一个 Mixamo 角色,以及**视频或照片二选一**。| 放什么 | 放到哪 | 去哪拿 ||---|---|---|| SMPL-X 人体模型 | `assets/body_models/smplx/SMPLX_NEUTRAL.npz` | 注册 [SMPL-X](https://smpl-x.is.tue.mpg.de/) 后下载 || Mixamo 角色 FBX | `assets/mixamo/Y_Bot.fbx` | 用 Adobe 账号从 [Mixamo](https://www.mixamo.com) 下载 Y Bot(或其他 Mixamo rig 角色) || 动作视频 | `assets/video/<your_clip>.mp4` | **只有一个人**、人清晰可见、**从头到脚全程都在画框内**(可放多个文件,每次跑用最新的那一个) |+| 姿态照片(可选) | `assets/image/<your_photo>.jpg` | **只有一个人**、人清晰可见、**从头到脚都在画框内**。配合 `--image` 使用 |注意:Mixamo 下载下来的文件叫 `Y Bot.fbx`(中间是**空格**),和上表的 `Y_Bot.fbx`(**下划线**)差一个字符。改成下划线它就是默认 rig,`m2mr run` 不带 `--rig` 时自动用它;不想改名也行——找不到 `Y_Bot.fbx` 时会自动用 `assets/mixamo/` 里的第一个 `.fbx`。-`assets/video/` 可以放多个视频文件。不指定 `--video` 时,会用**最后放进这个文件夹**的那一个。-每个视频画面里只能有一个人,而且人物必须**从头到脚全程都在画框内**,不能出画。`m2mr doctor` / `m2mr run` 会抽样检测人数,看到两个人就直接停,不会进入提取。+`assets/video/` 可以放多个视频文件。不指定 `--video` 或 `--image` 时,会用**最后放进 `assets/video/` 的视频**;这个文件夹是空的,就退回到 `assets/image/` 里最新的一张照片。+每个视频或照片画面里只能有一个人,而且人物必须**从头到脚都在画框内**,不能出画。`m2mr doctor` / `m2mr run` 检测到两个人就直接停,不会进入提取。## Quick Start-结果都在 `outputs/<运行时间>_<视频名>/`,看视频打开该目录下的 `videos/`。+结果都在 `outputs/<运行时间>_<输入名>/`。视频输入写 `videos/`,照片输入写 `images/`。### 1. 检查资产和环境@@ -103,6 +104,17 @@ m2mr run --skeleton outputs/<上次的运行目录>/skeleton_motion.npz --rig asm2mr run --video assets/video/dance.mp4 --rig assets/mixamo/Vampire.fbx```+### 5. 用一张照片(静态姿态)++`--image` 从**单人照片**恢复一个静态 3D 姿态,再按和视频相同的路径转到 Mixamo 角色上。预览是 `images/` 里的四张静帧(种类和视频 run 一样),外加左右对比绕转视频 `before_after_360_compare.mp4`。不要和 `--video` 一起用。++```bash+m2mr run --image assets/image/pose.jpg+m2mr run --image assets/image/pose.jpg --rig assets/mixamo/Vampire.fbx+```++`--image` 不带路径时,用 `assets/image/` 里最后放进去的那一张。+## 输出每次 `m2mr run` 按**命令启动时间**新建一个目录:@@ -113,11 +125,15 @@ outputs/20260829_193205_dance/├── skeleton_motion.npz # 人体 3D 骨架(换 rig 可复用,不必重跑提取)├── mixamo_rotations.npz # Mixamo 骨骼逐帧旋转├── mixamo_character.glb # 带皮角色 + 动画,Blender / Unity 可直接导入-└── videos/ # 比例跟输入视频一致- ├── human_skeleton.mp4 # 人体骨架- ├── mixamo_skeleton.mp4 # Mixamo rig 骨架- ├── mixamo_character.mp4 # Mixamo rig 角色+└── videos/ # 视频输入:比例跟输入视频一致+ ├── human_skeleton.mp4+ ├── mixamo_skeleton.mp4+ ├── mixamo_character.mp4└── compare.mp4 # 四宫格:左上原片 / 右上 Mixamo 骨架 / 左下人体骨架 / 右下 Mixamo 角色++# 照片输入则写 images/,四种图和上面一一对应:+# human_skeleton.png / mixamo_skeleton.png / mixamo_character.png / compare.png+# 另外还有 before_after_360_compare.mp4(左原图 / 右 10° 俯视绕角色转一圈)````mixamo_character.glb` 用 Blender 打开:File → Import → glTF 2.0 (.glb/.gltf)。导入后若镜头对不准,先选中角色,再 View → Frame Selected。头和手上那堆球是骨头的显示形状,不是模型:选中骨架,在 Armature → Viewport Display 里去掉 Shapes。时间轴结束帧改成和视频一样长,再点播放;对照同一次运行的 `videos/mixamo_character.mp4`。diff --git a/assets/README.md b/assets/README.mdindex 11d83d1..ed386a7 100644--- a/assets/README.md+++ b/assets/README.md@@ -13,6 +13,7 @@ Run `m2mr doctor` at any time to check what is present and what is missing.| `body_models/smplx/SMPLX_NEUTRAL.npz` | SMPL-X neutral body model, needed to lift the video into a 3D skeleton | Register at <https://smpl-x.is.tue.mpg.de/>, download the model archive, copy `SMPLX_NEUTRAL.npz` here || `mixamo/<Character>.fbx` | The character to drive, e.g. `Y_Bot.fbx` (the default) | <https://www.mixamo.com/> — pick a character, download as **FBX Binary**, **T-pose** || `video/<your_clip>.mp4` | The video whose human motion you want to transfer | One clearly visible person only (`.mp4`, `.mov`, …). Group shots are rejected before extraction |+| `image/<your_photo>.jpg` | A still whose pose you want to transfer | One clearly visible person (`.jpg`, `.png`, …). Group photos are rejected before extraction |`m2mr run` without flags uses the most recently added file in `video/` and-`mixamo/Y_Bot.fbx`; use `--video` / `--rig` to pick explicitly.+`mixamo/Y_Bot.fbx`; use `--video` / `--image` / `--rig` to pick explicitly.diff --git a/assets/image/.gitkeep b/assets/image/.gitkeepnew file mode 100644index 0000000..e69de29diff --git a/motion2mixamorig/__init__.py b/motion2mixamorig/__init__.pyindex c9f970f..823aa74 100644--- a/motion2mixamorig/__init__.py+++ b/motion2mixamorig/__init__.py@@ -1 +1 @@-"""Motion2MixamoRig: retarget human motion from video to a Mixamo-rigged 3D character."""+"""Motion2MixamoRig: retarget human motion (or a still pose) to a Mixamo-rigged 3D character."""diff --git a/motion2mixamorig/checks.py b/motion2mixamorig/checks.pyindex 09a6c81..017dcdf 100644--- a/motion2mixamorig/checks.py+++ b/motion2mixamorig/checks.py@@ -109,6 +109,29 @@ def check_video(path: Path) -> str | None:return None+def check_image(path: Path) -> str | None:+ """Decodable still, and exactly one person — same rule as check_video."""+ import cv2++ frame = cv2.imread(str(path), cv2.IMREAD_COLOR)+ if frame is None:+ return (+ "cannot be decoded as an image — use a jpg/png/webp still of one person"+ )+ n = _count_people(frame)+ if n >= 2:+ return (+ f"shows about {n} people — this pipeline maps one body "+ "to one Mixamo rig. Use a photo with a single, clearly visible person"+ )+ if n < 1:+ return (+ "no person was detected — use a photo with one clearly visible person, "+ "head to toe in frame"+ )+ return None++def check_smplx(path: Path) -> str | None:"""SMPLX_NEUTRAL.npz must be the npz variant of SMPL-X (not SMPL, not .pkl)."""import numpy as npdiff --git a/motion2mixamorig/cli.py b/motion2mixamorig/cli.pyindex 5efe0f7..72ccd06 100644--- a/motion2mixamorig/cli.py+++ b/motion2mixamorig/cli.py@@ -3,6 +3,7 @@m2mr doctor check assets, weights and dependenciesm2mr run latest video in assets/video/ + default Y Botm2mr run --video V --rig R pick the inputs explicitly (flags combine freely)+ m2mr run --image I --rig R still photo -> 3D pose on the same Mixamo path`m2mr run` performs the same asset checks as `m2mr doctor` before doing anywork, and reports everything that is missing in one pass.@@ -32,6 +33,11 @@ VIDEO_HINT = ("put a *single-person* motion video (.mp4/.mov/...) into assets/video/ ""— group shots are rejected before extraction")+IMAGE_HINT = (+ "put a *single-person* still (.jpg/.png/...) into assets/image/ "+ "— group photos are rejected before extraction"+)+SOURCE_HINT = VIDEO_HINT + "; or " + IMAGE_HINTdef _check(label: str, present: bool, hint: str) -> bool:@@ -80,14 +86,21 @@ def doctor() -> int:for rig in rigs:ok &= _check_content(f" - {rig.name}", rig, checks.check_rig)videos = paths.list_videos()+ images = paths.list_images()ok &= _check(- f"input video assets/video/ ({len(videos)} found"- + (f", latest: {videos[0].name})" if videos else ")"),- bool(videos),- VIDEO_HINT,+ "input source assets/video/ + assets/image/ "+ f"({len(videos)} videos"+ + (f", latest {videos[0].name}" if videos else "")+ + f"; {len(images)} images"+ + (f", latest {images[0].name}" if images else "")+ + ")",+ bool(videos) or bool(images),+ SOURCE_HINT,)for video in videos:ok &= _check_content(f" - {video.name}", video, checks.check_video)+ for image in images:+ ok &= _check_content(f" - {image.name}", image, checks.check_image)print()deps_ok = True@@ -118,12 +131,12 @@ def doctor() -> int:return 1-def preflight(args: argparse.Namespace) -> tuple[Path, Path, Path | None]:+def preflight(args: argparse.Namespace) -> tuple[Path | None, Path, Path | None, Path | None]:"""Resolve run inputs, checking everything doctor checks for this run.All problems are collected and reported together (same format as doctor)instead of failing on the first one. Exits with code 1 if anything is- missing.+ missing. Returns (video, rig, skeleton, image)."""problems = 0@@ -139,19 +152,67 @@ def preflight(args: argparse.Namespace) -> tuple[Path, Path, Path | None]:if path is not None and path.exists() and not _check_content(label, path, checker):problems += 1- # Input video: --video wins, else the most recently added file in assets/video/.- if args.video:+ video: Path | None = None+ image: Path | None = None+ want_image = getattr(args, "image", None) is not None+ want_video = bool(getattr(args, "video", None))++ if want_image and want_video:+ check(+ "input source --video / --image",+ False,+ "pass only one of --video or --image",+ )+ elif want_image:+ if args.image:+ image = Path(args.image)+ check(f"input image {image}", image.exists(), "check the --image path")+ else:+ image = paths.latest_image()+ check(+ "input image assets/image/"+ + (f" (latest: {image.name})" if image else " (0 found)"),+ image is not None,+ IMAGE_HINT + ", or pass --image PATH",+ )+ if image is not None and image.exists() and not paths.is_image(image):+ check(+ f"image suffix {image.name}",+ False,+ "use a still (.jpg/.png/webp/bmp), or pass a clip with --video",+ )+ check_content(f"image is usable {image.name if image else ''}", image, checks.check_image)+ elif want_video:video = Path(args.video)- check(f"input video {video}", video.exists(), "check the --video path")+ if video.exists() and paths.is_image(video):+ image = video+ video = None+ print(f"{OK} input image {image} (detected from --video path)")+ check_content(f"image is usable {image.name}", image, checks.check_image)+ else:+ check(f"input video {video}", video.exists(), "check the --video path")+ check_content(f"video is usable {video.name}", video, checks.check_video)else:video = paths.latest_video()- check(- "input video assets/video/"- + (f" (latest: {video.name})" if video else " (0 found)"),- video is not None,- VIDEO_HINT + ", or pass --video",- )- check_content(f"video is usable {video.name if video else ''}", video, checks.check_video)+ if video is None:+ image = paths.latest_image()+ check(+ "input image assets/image/"+ + (f" (latest: {image.name})" if image else " (0 found)"),+ image is not None,+ SOURCE_HINT + ", or pass --video / --image",+ )+ check_content(+ f"image is usable {image.name if image else ''}", image, checks.check_image+ )+ else:+ check(+ "input video assets/video/"+ + f" (latest: {video.name})",+ True,+ VIDEO_HINT + ", or pass --video",+ )+ check_content(f"video is usable {video.name}", video, checks.check_video)# Rig: --rig wins, else Y_Bot.fbx, else the first .fbx in assets/mixamo/.if args.rig:@@ -191,41 +252,57 @@ def preflight(args: argparse.Namespace) -> tuple[Path, Path, Path | None]:if problems:print(f"\n{problems} problem(s) — nothing was run. `m2mr doctor` checks everything.")sys.exit(1)- return video, rig, skeleton+ return video, rig, skeleton, imagedef run(args: argparse.Namespace) -> int:- video, rig, skeleton = preflight(args)+ video, rig, skeleton, image = preflight(args)print()from .pipeline import run_pipeline- run_dir = run_pipeline(video, rig, device=args.device, skeleton_npz=skeleton)- print(f"\nresults: {run_dir / 'videos'}")+ run_dir = run_pipeline(+ video, rig, image=image, device=args.device, skeleton_npz=skeleton+ )+ preview = "images" if image is not None else "videos"+ print(f"\nresults: {run_dir / preview}")return 0def main() -> None:parser = argparse.ArgumentParser(prog="m2mr",- description="Drive a Mixamo-rigged character with human motion from a video.",+ description=(+ "Drive a Mixamo-rigged character with human motion from a video, "+ "or a static 3D pose from a still photo."+ ),)sub = parser.add_subparsers(dest="command", required=True)sub.add_parser("doctor", help="check assets, weights and dependencies")- p_run = sub.add_parser("run", help="video -> Mixamo rig, results in outputs/")+ p_run = sub.add_parser("run", help="video or still -> Mixamo rig, results in outputs/")p_run.add_argument("--video",help="input video (default: the file most recently added to assets/video/)",)+ p_run.add_argument(+ "--image",+ nargs="?",+ const="",+ default=None,+ help=(+ "still photo of one person (default: the file most recently added "+ "to assets/image/). Recovers a static 3D pose; do not combine with --video"+ ),+ )p_run.add_argument("--rig",help="Mixamo character .fbx (default: assets/mixamo/Y_Bot.fbx)",)p_run.add_argument("--skeleton",- help="reuse a previous run's skeleton_motion.npz and skip video extraction",+ help="reuse a previous run's skeleton_motion.npz and skip extraction",)p_run.add_argument("--device",diff --git a/motion2mixamorig/extract.py b/motion2mixamorig/extract.pyindex 9513706..3351fdf 100644--- a/motion2mixamorig/extract.py+++ b/motion2mixamorig/extract.py@@ -1,16 +1,21 @@-"""Video -> 3D human skeleton (GVHMR world-grounded motion recovery).+"""Video or still -> 3D human skeleton (GVHMR world-grounded motion recovery).Produces skeleton_motion.npz with 24 SMPL joints per frame in a Y-up,meter-scaled world, plus the root translation/orientation and fps.++A still photo is repeated into a short hold clip first: GVHMR is a temporal+model and a one-frame "video" is too short to recover a stable pose."""from __future__ import annotations+from contextlib import contextmanagerfrom pathlib import Path+from typing import Iteratorimport numpy as np-from .paths import export_gvhmr_env+from .paths import VIDEO_SUFFIXES, export_gvhmr_env# SMPL 24-joint order used by GVHMR (J_regressor).SMPL_JOINT_NAMES = [@@ -40,6 +45,15 @@ SMPL_JOINT_NAMES = ["right_hand",]+# GVHMR is a temporal model: a one-frame "video" is too short and often+# unstable. Repeating the still for about a second gives it a static clip+# without inventing motion, and the preview / .glb then hold the pose.+HOLD_FRAMES = 32+HOLD_FPS = 30.0+# Phone photos are often 4000px+; YOLO / RTMPose do not need that, and a+# huge still makes the hold-video encode the slow part of an image run.+HOLD_MAX_SIDE = 1920+def _to_numpy(x):if hasattr(x, "detach"):@@ -104,6 +118,103 @@ def _stabilize_pose_cache(pose_cache: Path) -> bool:return True+def still_to_hold_video(+ image: Path,+ out_mp4: Path,+ *,+ n_frames: int = HOLD_FRAMES,+ fps: float = HOLD_FPS,+) -> Path:+ """Repeat a still as a short silent mp4 so GVHMR can treat it as a clip."""+ import shutil+ import subprocess++ import cv2++ image = Path(image)+ frame = cv2.imread(str(image), cv2.IMREAD_COLOR)+ if frame is None:+ raise ValueError(f"cannot decode image: {image}")++ h, w = frame.shape[:2]+ longest = max(h, w)+ if longest > HOLD_MAX_SIDE:+ scale = HOLD_MAX_SIDE / longest+ w = int(round(w * scale))+ h = int(round(h * scale))+ frame = cv2.resize(frame, (w, h), interpolation=cv2.INTER_AREA)+ h, w = frame.shape[:2]+ w -= w % 2+ h -= h % 2+ if w < 2 or h < 2:+ raise ValueError(f"image is too small: {image}")+ frame = frame[:h, :w]++ out_mp4 = Path(out_mp4)+ out_mp4.parent.mkdir(parents=True, exist_ok=True)++ ffmpeg = shutil.which("ffmpeg")+ if ffmpeg is not None:+ cmd = [+ ffmpeg, "-y", "-loglevel", "error",+ "-f", "rawvideo", "-pix_fmt", "bgr24",+ "-s", f"{w}x{h}", "-r", f"{fps:g}",+ "-i", "-",+ "-frames:v", str(n_frames),+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18",+ "-preset", "veryfast", "-movflags", "+faststart",+ str(out_mp4),+ ]+ proc = subprocess.Popen(cmd, stdin=subprocess.PIPE)+ assert proc.stdin is not None+ for _ in range(n_frames):+ proc.stdin.write(frame.tobytes())+ proc.stdin.close()+ if proc.wait() != 0:+ raise RuntimeError(f"ffmpeg failed writing {out_mp4}")+ return out_mp4++ writer = cv2.VideoWriter(+ str(out_mp4), cv2.VideoWriter_fourcc(*"mp4v"), float(fps), (w, h)+ )+ if not writer.isOpened():+ raise RuntimeError(f"Cannot open VideoWriter for {out_mp4}")+ for _ in range(n_frames):+ writer.write(frame)+ writer.release()+ return out_mp4+++@contextmanager+def temp_hold_video(image: Path) -> Iterator[Path]:+ """Write the hold clip under the system temp dir; delete it afterwards.++ GVHMR needs a real video file, but that file is not a run artifact and+ must not land in cache/.+ """+ import shutil+ import tempfile++ tmp = Path(tempfile.mkdtemp(prefix="m2mr-hold-"))+ try:+ yield still_to_hold_video(image, tmp / f"{Path(image).stem}.mp4")+ finally:+ shutil.rmtree(tmp, ignore_errors=True)+++def _strip_cache_videos(root: Path) -> None:+ """Drop staged / overlay mp4s from a still-image extract.++ GVHMR always stages ``0_input_video.mp4`` (and may write overlay clips+ under preprocess/). Those are only useful for a video run.+ """+ if not root.is_dir():+ return+ for path in root.rglob("*"):+ if path.suffix.lower() in VIDEO_SUFFIXES and (path.is_file() or path.is_symlink()):+ path.unlink(missing_ok=True)++def extract_motion(video: Path,output_npz: Path,@@ -112,11 +223,13 @@ def extract_motion(device: str = "cpu",static_camera: bool = True,verbose: bool = True,+ source_image: Path | None = None,) -> Path:"""Run GVHMR on `video` and save skeleton_motion.npz to `output_npz`.`work_dir` holds GVHMR's per-video intermediates (2D pose cache, 3D lift).Re-running with the same work_dir reuses the expensive passes.+ `source_image` is recorded when the clip was built from a still."""export_gvhmr_env()import gvhmr@@ -157,8 +270,7 @@ def extract_motion(output_npz = Path(output_npz)output_npz.parent.mkdir(parents=True, exist_ok=True)- np.savez_compressed(- output_npz,+ payload = dict(joints_3d=joints_3d,joint_names=np.array(SMPL_JOINT_NAMES),fps=np.float32(result.fps),@@ -168,6 +280,10 @@ def extract_motion(camera=np.array(result.camera),video=np.array(str(video)),)+ if source_image is not None:+ payload["image"] = np.array(str(source_image))+ _strip_cache_videos(output_dir)+ np.savez_compressed(output_npz, **payload)if verbose:print(f"saved {output_npz}")return output_npzdiff --git a/motion2mixamorig/paths.py b/motion2mixamorig/paths.pyindex f9b9998..45f2aba 100644--- a/motion2mixamorig/paths.py+++ b/motion2mixamorig/paths.py@@ -14,23 +14,33 @@ SMPLX_NEUTRAL = BODY_MODELS / "smplx" / "SMPLX_NEUTRAL.npz"MIXAMO_DIR = ASSETS / "mixamo"DEFAULT_RIG = MIXAMO_DIR / "Y_Bot.fbx"VIDEO_DIR = ASSETS / "video"+IMAGE_DIR = ASSETS / "image"WEIGHTS = PROJECT_ROOT / "weights"OUTPUTS = PROJECT_ROOT / "outputs"VIDEO_SUFFIXES = (".mp4", ".mov", ".avi", ".mkv", ".webm")+IMAGE_SUFFIXES = (".jpg", ".jpeg", ".png", ".webp", ".bmp")-def list_videos() -> list[Path]:- """User-supplied videos in assets/video/, most recently added first."""- if not VIDEO_DIR.is_dir():+def _list_media(folder: Path, suffixes: tuple[str, ...]) -> list[Path]:+ if not folder.is_dir():return []- videos = [+ files = [p- for p in VIDEO_DIR.iterdir()- if p.suffix.lower() in VIDEO_SUFFIXES and not p.name.startswith(".")+ for p in folder.iterdir()+ if p.suffix.lower() in suffixes and not p.name.startswith(".")]- return sorted(videos, key=lambda p: p.stat().st_mtime, reverse=True)+ return sorted(files, key=lambda p: p.stat().st_mtime, reverse=True)+++def is_image(path: Path) -> bool:+ return Path(path).suffix.lower() in IMAGE_SUFFIXES+++def list_videos() -> list[Path]:+ """User-supplied videos in assets/video/, most recently added first."""+ return _list_media(VIDEO_DIR, VIDEO_SUFFIXES)def latest_video() -> Path | None:@@ -39,12 +49,22 @@ def latest_video() -> Path | None:return videos[0] if videos else None-def new_run_dir(video: Path, now: datetime | None = None) -> Path:- """Create outputs/<YYYYMMDD_HHMMSS>_<video-stem>/ for one `m2mr run`."""+def list_images() -> list[Path]:+ """User-supplied stills in assets/image/, most recently added first."""+ return _list_media(IMAGE_DIR, IMAGE_SUFFIXES)+++def latest_image() -> Path | None:+ """The still most recently placed into assets/image/ (by file mtime)."""+ images = list_images()+ return images[0] if images else None+++def new_run_dir(source: Path, now: datetime | None = None) -> Path:+ """Create outputs/<YYYYMMDD_HHMMSS>_<source-stem>/ for one `m2mr run`."""stamp = (now or datetime.now()).strftime("%Y%m%d_%H%M%S")- run_dir = OUTPUTS / f"{stamp}_{video.stem}"+ run_dir = OUTPUTS / f"{stamp}_{source.stem}"run_dir.mkdir(parents=True, exist_ok=True)- (run_dir / "videos").mkdir(exist_ok=True)return run_dirdiff --git a/motion2mixamorig/pipeline.py b/motion2mixamorig/pipeline.pyindex aff185b..a5669a5 100644--- a/motion2mixamorig/pipeline.py+++ b/motion2mixamorig/pipeline.py@@ -1,14 +1,14 @@-"""One `m2mr run`: video -> human skeleton -> Mixamo rig -> result videos.+"""One `m2mr run`: video or still -> human skeleton -> Mixamo rig -> result videos.Everything a run produces lands in its own timestamped folder:- outputs/<YYYYMMDD_HHMMSS>_<video>/+ outputs/<YYYYMMDD_HHMMSS>_<source>/├── run.json what ran, with what inputs, producing what├── skeleton_motion.npz 3D human skeleton (24 SMPL joints per frame)├── mixamo_rotations.npz per-bone animation rotations for the rig├── mixamo_character.glb skinned character + animation (Blender / Unity)├── cache/ GVHMR intermediates (2D pose, 3D lift)- └── videos/ the four result videos+ └── videos/ or images/ four result clips (video in) or four stills (photo in)"""from __future__ import annotations@@ -30,7 +30,7 @@ from .mixamo.retarget import verify_rest_fkfrom .mixamo.skinned_mesh import load_character_asset, rest_skin_errorfrom .mixamo.tpose_calibration import load_calibrationfrom .mixamo.ybot_retarget import build_mapping-from .videos import render_run_videos+from .videos import render_run_images, render_run_videosROTATION_CONVENTION = ("Animation-channel rotations: M_local = T * Rpre * R_anim, world = parent @ local. "@@ -67,37 +67,61 @@ def save_mixamo_rotations(anim, mapping, rig_path: Path, out_npz: Path) -> Path:def run_pipeline(- video: Path,+ video: Path | None,rig: Path,*,+ image: Path | None = None,device: str = "cpu",skeleton_npz: Path | None = None,verbose: bool = True,) -> Path:- """Full pipeline for one video + one rig. Returns the run directory.+ """Full pipeline for one video or still + one rig. Returns the run directory.`skeleton_npz` reuses a previous run's skeleton_motion.npz and skips the- (slow) video extraction entirely.+ (slow) extraction entirely. A still is turned into a short hold clip so+ the same GVHMR path can recover a static 3D pose."""- video = Path(video).resolve()+ if image is None and video is None:+ raise ValueError("run_pipeline needs a video or an image")+ if image is not None:+ image = Path(image).resolve()+ if video is not None:+ video = Path(video).resolve()rig = Path(rig).resolve()started = datetime.now()t0 = time.time()- run_dir = paths.new_run_dir(video, started)+ source = image if image is not None else video+ assert source is not None+ run_dir = paths.new_run_dir(source, started)print(f"run directory {run_dir}")+ extract_video = videomotion_npz = run_dir / "skeleton_motion.npz"if skeleton_npz is not None:import shutilshutil.copy2(skeleton_npz, motion_npz)print(f"reusing skeleton from {skeleton_npz}")+ elif image is not None:+ from .extract import extract_motion, temp_hold_video++ print(f"extracting human pose from {image.name}")+ with temp_hold_video(image) as hold:+ extract_motion(+ hold,+ motion_npz,+ work_dir=run_dir / "cache",+ device=device,+ verbose=verbose,+ source_image=image,+ )else:from .extract import extract_motion- print(f"extracting human motion from {video.name}")+ assert extract_video is not None+ print(f"extracting human motion from {extract_video.name}")extract_motion(- video,+ extract_video,motion_npz,work_dir=run_dir / "cache",device=device,@@ -127,16 +151,34 @@ def run_pipeline(write_glb(glb_path, asset, anim.r_anim_rig, anim.hips_cm_rig, anim.fps)print(f"saved {glb_path}")- print("rendering result videos")- videos = render_run_videos(- anim, asset, calibration, video, run_dir / "videos", verbose=verbose- )+ if image is not None:+ print("rendering result images")+ previews = render_run_images(+ anim, asset, calibration, image, run_dir / "images", verbose=verbose+ )+ preview_key = "images"+ preview_dir = "images"+ else:+ print("rendering result videos")+ assert extract_video is not None+ previews = render_run_videos(+ anim,+ asset,+ calibration,+ extract_video,+ run_dir / "videos",+ original_label="Original Video",+ verbose=verbose,+ )+ preview_key = "videos"+ preview_dir = "videos"report = {"started": started.isoformat(timespec="seconds"),"elapsed_s": round(time.time() - t0, 1),"command": sys.argv,- "video": str(video),+ "video": None if image is not None else str(video),+ "image": str(image) if image is not None else None,"rig": str(rig),"device": device,"fps": float(motion.fps),@@ -150,7 +192,7 @@ def run_pipeline("skeleton_motion": motion_npz.name,"mixamo_rotations": rotations_npz.name,"mixamo_character_glb": glb_path.name,- "videos": {k: f"videos/{v.name}" for k, v in videos.items()},+ preview_key: {k: f"{preview_dir}/{v.name}" for k, v in previews.items()},},}(run_dir / "run.json").write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")diff --git a/motion2mixamorig/videos.py b/motion2mixamorig/videos.pyindex c31d50f..ae183e8 100644--- a/motion2mixamorig/videos.py+++ b/motion2mixamorig/videos.py@@ -1,10 +1,14 @@-"""Render the four result videos of a run, all in the input video's aspect ratio.+"""Render the four result views of a run, all in the input's aspect ratio.- videos/human_skeleton.mp4 3D human skeleton recovered from the video- videos/mixamo_skeleton.mp4 retargeted Mixamo Skeleton- videos/mixamo_character.mp4 skinned character mesh- videos/compare.mp4 2x2 grid: original | mixamo skeleton- human | character+Video input writes `videos/*.mp4`; a still writes the same four views as+`images/*.png`:++ human_skeleton 3D human skeleton recovered from the source+ mixamo_skeleton retargeted Mixamo Skeleton+ mixamo_character skinned character mesh+ compare 2x2: original | mixamo skeleton+ human | character+ before_after_360_compare.mp4 image runs only: original | orbiting rigThe three rendered panels share one perspective camera and one floor grid, sothey read as views into the same space: an orthographic camera cannot show a@@ -12,9 +16,10 @@ floor receding to a vanishing point, and without a floor and a contact shadowthe figure just hangs in black. All of it is fitted once over the whole clip;nothing is re-centred per frame.-Frames are streamed straight into ffmpeg (which also muxes the source video's-audio into every output), so memory stays flat no matter how long the clip is.-When ffmpeg is missing the videos still get written via OpenCV, just silent.+Video frames are streamed straight into ffmpeg (which also muxes the source+video's audio into every output), so memory stays flat no matter how long the+clip is. When ffmpeg is missing the videos still get written via OpenCV, just+silent."""from __future__ import annotations@@ -26,11 +31,13 @@ from pathlib import Pathimport numpy as npfrom .mixamo.animation import RetargetedAnimation, all_points, skin_frames+from .mixamo.kinematics import normalizefrom .mixamo.render import (DARK_THEME,GRID_THEME,MESH_COLORS_BGR,Ground,+ PerspectiveCamera,blank_panel,build_perspective_camera,draw_mesh,@@ -66,6 +73,17 @@ FLOOR_PERCENTILE = 10.0# backdrop and a little more ambient fill than the white-on-black skeletons.CHARACTER_AMBIENT = 0.34+# Image-only turntable: original still on the left, orbiting rig on the right.+# Elevation 10° looks down from about the character's own height; a slightly+# wider FOV and looser fill keep hands and feet inside the frame as the+# camera goes around.+ORBIT_ELEVATION_DEG = 10.0+ORBIT_FOV_DEG = 38.0+ORBIT_FILL = 0.86+ORBIT_SECONDS = 6.0+ORBIT_FPS = 30.0+ORBIT_PAD = 1.06+def video_size(video: Path) -> tuple[int, int]:import cv2@@ -79,9 +97,25 @@ def video_size(video: Path) -> tuple[int, int]:return w, h-def panel_size(video: Path) -> tuple[int, int]:- """Panel with the same aspect ratio as the input video, even dimensions."""- w, h = video_size(video)+def image_size(image: Path) -> tuple[int, int]:+ import cv2++ frame = cv2.imread(str(image), cv2.IMREAD_COLOR)+ if frame is None:+ raise RuntimeError(f"Cannot open image: {image}")+ h, w = frame.shape[:2]+ return w, h+++def source_size(path: Path) -> tuple[int, int]:+ from .paths import is_image++ return image_size(path) if is_image(path) else video_size(path)+++def panel_size(source: Path) -> tuple[int, int]:+ """Panel with the same aspect ratio as the input, even dimensions."""+ w, h = source_size(source)pw = int(round(PANEL_H * w / max(h, 1)))pw = min(max(pw, 2), MAX_PANEL_W)return pw + pw % 2, PANEL_H + PANEL_H % 2@@ -185,25 +219,27 @@ def _grid2x2(tl: np.ndarray, tr: np.ndarray, bl: np.ndarray, br: np.ndarray) ->return np.vstack([np.hstack([tl, tr]), np.hstack([bl, br])])-def render_run_videos(- anim: RetargetedAnimation,- asset,- calibration,- source_video: Path,- out_dir: Path,- *,- verbose: bool = True,-) -> dict[str, Path]:- """Write the four result videos into `out_dir` (videos/ of a run)."""- out_dir = Path(out_dir)- out_dir.mkdir(parents=True, exist_ok=True)- width, height = panel_size(source_video)- fps = anim.fps+def _write_png(path: Path, bgr: np.ndarray) -> Path:+ import cv2- # One camera and one floor over human joints, Mixamo joints and the mesh,- # so the three rendered panels agree on where everything stands. Human,- # Mixamo Skeleton and character all live in the calibrated rig world, so- # the shared anatomical axes come from the calibration.+ path = Path(path)+ path.parent.mkdir(parents=True, exist_ok=True)+ if not cv2.imwrite(str(path), bgr):+ raise RuntimeError(f"Cannot write {path}")+ return path+++def _letterbox_still(image: Path, width: int, height: int) -> np.ndarray:+ import cv2++ frame = cv2.imread(str(image), cv2.IMREAD_COLOR)+ if frame is None:+ raise RuntimeError(f"Cannot open image: {image}")+ return fit_letterbox(frame, width, height)+++def _setup_view(anim: RetargetedAnimation, asset, calibration, width: int, height: int):+ """Shared camera, floor and backdrops for the three rendered panels."""axes = calibration.rig_axesstride = max(1, anim.n // CAMERA_MESH_SAMPLES)probe = skin_frames(asset, anim, list(range(0, anim.n, stride)))@@ -233,14 +269,201 @@ def render_run_videos(cx=float(np.median(hips[:, 0])),cz=float(np.median(hips[:, 2])),)+ backdrop_dark = ground_backdrop(cam, ground, width, height, DARK_THEME)+ backdrop_grid = ground_backdrop(cam, ground, width, height, GRID_THEME)+ faces = [m.faces for m in asset.meshes]+ colors = [+ m.face_colors+ if m.face_colors is not None+ else MESH_COLORS_BGR.get(m.name, np.array([150.0, 150.0, 150.0]))+ for m in asset.meshes+ ]+ return cam, ground, backdrop_dark, backdrop_grid, faces, colors+++def _draw_panels(+ anim: RetargetedAnimation,+ asset,+ calibration,+ i: int,+ *,+ original: np.ndarray,+ original_label: str,+ stamp: str,+ cam,+ ground,+ backdrop_dark,+ backdrop_grid,+ faces,+ colors,+) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:+ character = calibration.fbx_path.stem++ p_hum = backdrop_dark.copy()+ pos = anim.human_pos[i]+ draw_shadow(p_hum, cam, np.stack(list(pos.values())), ground, DARK_THEME)+ draw_skeleton(p_hum, pos, HUMAN_EDGES, cam, 2, DARK_THEME.joint)+ label_panel(p_hum, f"Human Skeleton{stamp}")++ p_mix = backdrop_dark.copy()+ pos = anim.mixamo_pos[i]+ draw_shadow(p_mix, cam, np.stack(list(pos.values())), ground, DARK_THEME)+ draw_skeleton(p_mix, pos, MIXAMO_EDGES, cam, 3, DARK_THEME.joint)+ label_panel(p_mix, f"Mixamo Skeleton{stamp}")++ p_char = backdrop_grid.copy()+ verts = skin_frames(asset, anim, [i])[0]+ draw_shadow(p_char, cam, np.concatenate(verts, axis=0), ground, GRID_THEME)+ draw_mesh(p_char, verts, faces, colors, cam, CHARACTER_AMBIENT)+ label_panel(p_char, f"{character}{stamp}")++ p_orig = original.copy()+ label_panel(p_orig, f"{original_label}{stamp}")+ return p_orig, p_mix, p_hum, p_char+++def render_run_images(+ anim: RetargetedAnimation,+ asset,+ calibration,+ source_image: Path,+ out_dir: Path,+ *,+ original_label: str = "Original Photo",+ verbose: bool = True,+) -> dict[str, Path]:+ """Write the four stills into `out_dir` (images/ of a still run).++ Uses the middle hold frame: GVHMR is temporal, so the centre of the+ repeated still is the most settled pose.+ """+ out_dir = Path(out_dir)+ out_dir.mkdir(parents=True, exist_ok=True)+ width, height = panel_size(source_image)+ cam, ground, backdrop_dark, backdrop_grid, faces, colors = _setup_view(+ anim, asset, calibration, width, height+ )if verbose:print(f" camera eye {np.round(cam.eye, 3)} fov {cam.fov_deg:.1f} deg"f" floor y {ground.y:+.4f} m")- # Camera and floor are both fixed, so the floor is drawn once per theme- # and every frame starts as a copy of it.- backdrop_dark = ground_backdrop(cam, ground, width, height, DARK_THEME)- backdrop_grid = ground_backdrop(cam, ground, width, height, GRID_THEME)+ i = anim.n // 2+ p_orig, p_mix, p_hum, p_char = _draw_panels(+ anim, asset, calibration, i,+ original=_letterbox_still(source_image, width, height),+ original_label=original_label,+ stamp="",+ cam=cam, ground=ground,+ backdrop_dark=backdrop_dark, backdrop_grid=backdrop_grid,+ faces=faces, colors=colors,+ )+ out = {+ "human_skeleton": _write_png(out_dir / "human_skeleton.png", p_hum),+ "mixamo_skeleton": _write_png(out_dir / "mixamo_skeleton.png", p_mix),+ "mixamo_character": _write_png(out_dir / "mixamo_character.png", p_char),+ "compare": _write_png(out_dir / "compare.png", _grid2x2(p_orig, p_mix, p_hum, p_char)),+ }+ if verbose:+ print("rendering 360 compare")+ out["before_after_360_compare"] = render_orbit_compare(+ anim, asset, calibration, source_image, out_dir, verbose=verbose+ )+ if verbose:+ for path in out.values():+ print(f"saved {path}")+ return out+++def _orbit_camera(+ axes: dict,+ target: np.ndarray,+ dist: float,+ azimuth_deg: float,+ width: int,+ height: int,+ *,+ elevation_deg: float = ORBIT_ELEVATION_DEG,+ fov_deg: float = ORBIT_FOV_DEG,+) -> PerspectiveCamera:+ """Pinhole eye on a circle around `target`, looking down at `elevation_deg`."""+ up = normalize(np.asarray(axes["up"], dtype=np.float64))+ fwd = normalize(np.asarray(axes["forward"], dtype=np.float64))+ right = normalize(np.asarray(axes["right"], dtype=np.float64))+ az, el = np.radians(azimuth_deg), np.radians(elevation_deg)+ offset = normalize(np.cos(el) * (np.cos(az) * fwd + np.sin(az) * right) + np.sin(el) * up)+ look = -offset+ cam_right = normalize(np.cross(look, up))+ cam_up = normalize(np.cross(cam_right, look))+ focal = (height * 0.5) / np.tan(np.radians(fov_deg) * 0.5)+ return PerspectiveCamera(+ target + offset * dist, look, cam_right, cam_up, float(focal), width, height+ )+++def _fit_orbit_distance(+ axes: dict,+ points: np.ndarray,+ target: np.ndarray,+ width: int,+ height: int,+) -> float:+ """One radius that keeps the whole mesh on screen at every heading.++ Starts at the distance that puts the eye at about the character's height+ (elevation 10° looking at the torso centre), then grows if any heading+ would clip a hand or a foot.+ """+ pts = np.asarray(points, dtype=np.float64).reshape(-1, 3)+ up = normalize(np.asarray(axes["up"], dtype=np.float64))+ along = pts @ up+ head = float(along.max())+ mid = float(target @ up)+ el = np.radians(ORBIT_ELEVATION_DEG)+ dist = max((head - mid) / max(np.sin(el), 1e-3), 1e-3)+ center = pts.mean(axis=0)+ padded = center + (pts - center) * ORBIT_PAD+ for az in np.linspace(0.0, 360.0, 12, endpoint=False):+ for _ in range(60):+ cam = _orbit_camera(axes, target, dist, float(az), width, height)+ uv = cam.project(padded)+ lo, hi = uv.min(axis=0), uv.max(axis=0)+ need = max(+ (hi[0] - lo[0]) / (ORBIT_FILL * width),+ (hi[1] - lo[1]) / (ORBIT_FILL * height),+ 1.0,+ )+ inside = (lo >= 0.0).all() and (hi < (width, height)).all()+ if need <= 1.0 and inside:+ break+ dist *= max(need, 1.02)+ return float(dist)+++def render_orbit_compare(+ anim: RetargetedAnimation,+ asset,+ calibration,+ source_image: Path,+ out_dir: Path,+ *,+ verbose: bool = True,+) -> Path:+ """Left = original photo, right = 10° orbit of the skinned rig. Image runs only."""+ out_dir = Path(out_dir)+ out_dir.mkdir(parents=True, exist_ok=True)+ width, height = panel_size(source_image)+ i = anim.n // 2+ verts = skin_frames(asset, anim, [i])[0]+ mesh_pts = np.concatenate(verts, axis=0)+ axes = calibration.rig_axes+ target = 0.5 * (mesh_pts.min(axis=0) + mesh_pts.max(axis=0))+ dist = _fit_orbit_distance(axes, mesh_pts, target, width, height)+ hips = anim.mixamo_pos[i]["mixamorig:Hips"]+ ground = Ground(+ y=float(np.percentile(mesh_pts[:, 1], FLOOR_PERCENTILE)),+ cx=float(hips[0]),+ cz=float(hips[2]),+ )faces = [m.faces for m in asset.meshes]colors = [@@ -249,6 +472,52 @@ def render_run_videos(else MESH_COLORS_BGR.get(m.name, np.array([150.0, 150.0, 150.0]))for m in asset.meshes]+ left = _letterbox_still(source_image, width, height)+ label_panel(left, "Original Photo")+ character = calibration.fbx_path.stem+ n = max(2, int(round(ORBIT_SECONDS * ORBIT_FPS)))+ out_path = out_dir / "before_after_360_compare.mp4"+ writer = _StreamWriter(out_path, ORBIT_FPS, 2 * width, height)+ if verbose:+ print(f" orbit dist {dist:.3f} m {n} frames @ {ORBIT_FPS:g} fps"+ f" elev {ORBIT_ELEVATION_DEG:g} deg")+ try:+ for k in range(n):+ az = 360.0 * k / n+ cam = _orbit_camera(axes, target, dist, az, width, height)+ right = ground_backdrop(cam, ground, width, height, GRID_THEME)+ draw_shadow(right, cam, mesh_pts, ground, GRID_THEME)+ draw_mesh(right, verts, faces, colors, cam, CHARACTER_AMBIENT)+ label_panel(right, character)+ writer.write(np.hstack([left, right]))+ if verbose and (k == 0 or (k + 1) % 30 == 0 or k == n - 1):+ print(f" orbit {k + 1}/{n}")+ finally:+ writer.close()+ return out_path+++def render_run_videos(+ anim: RetargetedAnimation,+ asset,+ calibration,+ source_video: Path,+ out_dir: Path,+ *,+ original_label: str = "Original Video",+ verbose: bool = True,+) -> dict[str, Path]:+ """Write the four result videos into `out_dir` (videos/ of a run)."""+ out_dir = Path(out_dir)+ out_dir.mkdir(parents=True, exist_ok=True)+ width, height = panel_size(source_video)+ fps = anim.fps+ cam, ground, backdrop_dark, backdrop_grid, faces, colors = _setup_view(+ anim, asset, calibration, width, height+ )+ if verbose:+ print(f" camera eye {np.round(cam.eye, 3)} fov {cam.fov_deg:.1f} deg"+ f" floor y {ground.y:+.4f} m")audio_start = anim.frames[0] / fpsaudio_dur = anim.n / fps@@ -261,7 +530,6 @@ def render_run_videos(audio_dur_s=audio_dur,)- character = calibration.fbx_path.stemwriters = {"human_skeleton": writer("human_skeleton.mp4", width, height),"mixamo_skeleton": writer("mixamo_skeleton.mp4", width, height),@@ -272,29 +540,15 @@ def render_run_videos(try:for i in range(anim.n):t = anim.frames[i] / fps- stamp = f"f{anim.frames[i]} t={t:.2f}s"-- p_hum = backdrop_dark.copy()- pos = anim.human_pos[i]- draw_shadow(p_hum, cam, np.stack(list(pos.values())), ground, DARK_THEME)- draw_skeleton(p_hum, pos, HUMAN_EDGES, cam, 2, DARK_THEME.joint)- label_panel(p_hum, f"Human Skeleton {stamp}")-- p_mix = backdrop_dark.copy()- pos = anim.mixamo_pos[i]- draw_shadow(p_mix, cam, np.stack(list(pos.values())), ground, DARK_THEME)- draw_skeleton(p_mix, pos, MIXAMO_EDGES, cam, 3, DARK_THEME.joint)- label_panel(p_mix, f"Mixamo Skeleton {stamp}")-- p_char = backdrop_grid.copy()- verts = skin_frames(asset, anim, [i])[0]- draw_shadow(p_char, cam, np.concatenate(verts, axis=0), ground, GRID_THEME)- draw_mesh(p_char, verts, faces, colors, cam, CHARACTER_AMBIENT)- label_panel(p_char, f"{character} {stamp}")-- p_orig = originals.frame(anim.frames[i])- label_panel(p_orig, f"Original Video {stamp}")-+ p_orig, p_mix, p_hum, p_char = _draw_panels(+ anim, asset, calibration, i,+ original=originals.frame(anim.frames[i]),+ original_label=original_label,+ stamp=f" f{anim.frames[i]} t={t:.2f}s",+ cam=cam, ground=ground,+ backdrop_dark=backdrop_dark, backdrop_grid=backdrop_grid,+ faces=faces, colors=colors,+ )writers["human_skeleton"].write(p_hum)writers["mixamo_skeleton"].write(p_mix)writers["mixamo_character"].write(p_char)diff --git a/pyproject.toml b/pyproject.tomlindex 379490b..8e2aec7 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,7 +1,7 @@[project]name = "motion2mixamorig"version = "0.1.0"-description = "Retarget human motion from video to an Adobe Mixamo-rigged 3D character."+description = "Retarget human motion from video or a still pose to an Adobe Mixamo-rigged 3D character."requires-python = ">=3.10"dependencies = ["numpy",