Skip to content

[SIGGRAPH Asia 2026] UniMate: One Unified Model to Animate Diverse Skeletons

Python

Release the UniML3D dataset processing pipeline

Friedrich-Mf697a3386ed3
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..98b31db
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,95 @@
+# ---------- Not yet open-sourced ----------
+# Only the data-processing pipeline (data_process/) is public at this stage;
+# the training / inference code lives in the private motion-gen repo and will
+# be released here later. Patterns are root-anchored on purpose.
+/src/
+/configs/
+/scripts/
+/docs/
+/CLAUDE.md
+/reference/
+
+# ---------- Data / experiments (root-level only) ----------
+/dataset/
+/baselines/
+/outputs/
+/output/
+/logs/
+/*debug/
+/animation_debugs/
+wandb/
+runs/
+mlruns/
+lightning_logs/
+tensorboard/
+tb_logs/
+
+# ---------- ML artifacts ----------
+*.ckpt
+*.pt
+*.pth
+*.safetensors
+*.onnx
+
+# ---------- Python ----------
+__pycache__/
+*.py[cod]
+*.so
+.ipynb_checkpoints/
+*.egg-info/
+dist/
+build/
+*.egg
+
+# ---------- Virtual environments ----------
+.venv/
+venv/
+env/
+ENV/
+.conda/
+
+# ---------- Caches / coverage ----------
+.cache/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.dmypy.json
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+.tox/
+
+# ---------- SLURM / cluster ----------
+# Cluster-specific launchers โ€” not part of the public pipeline.
+/data_process/scripts/slurm/
+slurm-*.out
+slurm-*.err
+*.slurm.log
+core
+core.*
+
+# ---------- Editor / IDE ----------
+.claude/
+.vscode/
+.idea/
+*.iml
+.cursor/
+.history/
+
+# ---------- OS ----------
+.DS_Store
+Thumbs.db
+Desktop.ini
+
+# ---------- Logs / temp ----------
+*.log
+*.tmp
+*.temp
+*.bak
+*.old
+*.swp
+*.swo
+*~
+.env
+.env.local
diff --git a/README.md b/README.md
index ad8e7a2..b5513df 100644
--- a/README.md
+++ b/README.md
@@ -4,9 +4,10 @@
<p align="center">
<a href="https://linzhanm.github.io/unimate/"><img alt="Project Page" src="https://img.shields.io/badge/Project_Page-6D28D9?style=for-the-badge&logo=githubpages&logoColor=white"></a>
- <a href="https://linzhanmou.com/unimate/resources/unimate.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-BF40FF?style=for-the-badge&logo=googledocs&logoColor=white"></a>
+ <a href="https://linzhanmou.com/unimate/resources/unimate.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-A855F7?style=for-the-badge&logo=googledocs&logoColor=white"></a>
<a href="https://linzhanmou.com/unimate/interactive.html"><img alt="Interactive Demo" src="https://img.shields.io/badge/Interactive_Demo-0EA5E9?style=for-the-badge&logo=threedotjs&logoColor=white"></a>
- <img alt="Venue" src="https://img.shields.io/badge/SIGGRAPH_Asia-2026-1E1B4B?style=for-the-badge">
+ <a href="https://huggingface.co/Linzhan"><img alt="Hugging Face Dataset" src="https://img.shields.io/badge/Dataset-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000000"></a>
+ <a href="https://asia.siggraph.org/2026/"><img alt="Venue" src="https://img.shields.io/badge/SIGGRAPH_Asia-2026-6D28D9?style=for-the-badge&labelColor=1E1B4B"></a>
</p>
<p align="center">
@@ -25,41 +26,45 @@
<img src="assets/teaser.png" alt="UniMate teaser" width="100%">
</div>
-<!--
-<div align="center">
-
-https://github.com/user-attachments/assets/a4efca07-8673-4ddf-8057-b8be7b008265
-
-</div>
--->
-
---
-## ๐Ÿ“œ News
+## ๐Ÿ”ฅ News
-- **[TODO]** Training code and the UniML3D dataset will be released in this repository around August 2026.
+- **[TODO]** Training and inference code will be released in this repository soon.
+- **[2026-08-30]** The **UniML3D dataset** and its [data-processing pipeline](data_process/) are released. ๐Ÿš€
- **[2026-08-01]** Our [Interactive Demo](https://linzhanmou.com/unimate/interactive.html) is live โ€” browse our animation results in 3D. ๐ŸŽฎ
- **[2026-07-18]** UniMate is accepted to SIGGRAPH Asia 2026! ๐ŸŽ‰
-## ๐Ÿ–ผ๏ธ Inference
+## ๐Ÿ› ๏ธ Environment Setup
-Given a rigged 3D asset and a text prompt, UniMate generates articulated motion for arbitrary skeletons in real time โ€” with no per-skeleton retraining and no test-time optimization.
+All components share a single conda environment, specified in [`requirements.txt`](requirements.txt):
-<div align="center">
- <img src="assets/qualitative.png" alt="Qualitative results" width="100%">
-</div>
+```bash
+conda create -n unimate python=3.10 -y
+conda activate unimate
+pip install "setuptools<81"
+pip install -r requirements.txt --no-build-isolation
+```
-## ๐Ÿ“Š Dataset
+## ๐Ÿ“Š Dataset & Data Processing
We introduce **UniML3D**, a large-scale dataset of 13,006 text-paired motion sequences covering diverse skeletal topologies โ€” bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects โ€” all brought into a unified canonicalization.
+The raw source assets are available on the Hugging Face Hub: [Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters), [Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) and [Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata and renders only). The Truebones ZOO animal motions themselves are a commercial asset pack whose license does not permit redistribution โ€” please purchase the pack directly from [Truebones](https://truebones.com); our pipeline consumes the stock `Truebone_Z-OO` folder layout as-is.
+
<div align="center">
<img src="assets/dataset_overview.png" alt="UniML3D dataset overview" width="100%">
</div>
-## ๐Ÿ“„ License
+See [`data_process/README.md`](data_process/README.md) for the full data processing pipeline that turns the raw assets into UniML3D (download โ†’ export โ†’ rendering โ†’ captioning โ†’ joint annotation โ†’ feature extraction โ†’ animation).
+
+## ๐ŸŽฌ Inference
-This project is released under the [MIT License](LICENSE).
+Given a rigged 3D asset and a text prompt, UniMate generates articulated motion for arbitrary skeletons in real time โ€” with no per-skeleton retraining and no test-time optimization.
+
+<div align="center">
+ <img src="assets/qualitative.png" alt="Qualitative results" width="100%">
+</div>
## ๐Ÿ“ Citation
@@ -73,3 +78,9 @@ If you find UniMate useful in your research, please consider citing our work:
year={2026}
}
```
+
+## โš–๏ธ License
+
+The code in this repository is released under the [MIT License](LICENSE).
+
+The datasets remain governed by the licenses of their original sources: the [Mixamo](https://www.mixamo.com/) assets by Adobe's Mixamo terms of use, the [Objaverse-XL](https://objaverse.allenai.org/) assets by the license attached to each original object, and the Truebones ZOO motions by [Truebones](https://truebones.com)' commercial license. Please review and comply with the respective source licenses before using the data.
diff --git a/data_process/README.md b/data_process/README.md
new file mode 100644
index 0000000..c2dfeda
--- /dev/null
+++ b/data_process/README.md
@@ -0,0 +1,344 @@
+# UniML3D Data Processing
+
+The pipeline that turns raw rigged assets (Truebones FBX, Mixamo FBX, Objaverse GLB) into **UniML3D** โ€” the text-paired, topology-annotated motion clips used to train UniMate.
+
+All commands are run **from the repository root**.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Requirements](#requirements)
+- [Directory Layout](#directory-layout)
+- [Getting the Raw Data](#getting-the-raw-data)
+- [Quick Start](#quick-start)
+- [Pipeline Stages](#pipeline-stages)
+- [Data Formats](#data-formats)
+- [Conventions](#conventions)
+- [Troubleshooting](#troubleshooting)
+
+## Overview
+
+```mermaid
+%%{init: {
+ "theme": "base",
+ "themeVariables": {
+ "fontFamily": "-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif",
+ "fontSize": "14px",
+ "lineColor": "#94A3B8",
+ "edgeLabelBackground": "#F1F5F9"
+ },
+ "flowchart": { "curve": "basis", "nodeSpacing": 40, "rankSpacing": 60 }
+}}%%
+flowchart LR
+ raw(["๐Ÿ“ฆ raw FBX / GLB"])
+ export("<b>1 ยท Export</b><br/>Blender โ†’ NPZ")
+ render("<b>2a ยท Render</b><br/>EEVEE multi-view")
+ caption("<b>2b ยท Caption</b><br/>VLM")
+ joints("<b>3 ยท Joints</b><br/>names + facing")
+ extract("<b>4 ยท Extract</b><br/>canonicalize")
+ animate("<b>5 ยท Animate</b><br/>rigged GLB/FBX")
+ clips(["โœจ training clips + cond.npy"])
+
+ raw --> export
+ raw --> render
+ export -- "joint_names.json" --> joints
+ render -- "frames" --> caption
+ export -- "motions/*.npz" --> extract
+ caption -- "captions / categories" --> extract
+ joints -- "clean / face joints" --> extract
+ extract --> clips
+ clips --> animate
+ raw -- "rigged mesh" --> animate
+
+ classDef data fill:#1E1B4B,stroke:#1E1B4B,color:#F8FAFC
+ classDef blender fill:#6D28D9,stroke:#5B21B6,color:#FFFFFF
+ classDef vision fill:#0EA5E9,stroke:#0284C7,color:#FFFFFF
+ classDef llm fill:#A855F7,stroke:#9333EA,color:#FFFFFF
+ classDef out fill:#FFD21E,stroke:#CA8A04,color:#1E1B4B
+
+ class raw data
+ class export,animate blender
+ class render,caption vision
+ class joints,extract llm
+ class clips out
+
+ linkStyle default stroke:#94A3B8,stroke-width:1.5px
+```
+
+| Stage | Wrapper(s) | Input | Output |
+|-------|-----------|-------|--------|
+| 0. Download | `run_download.sh` | Hugging Face Hub | `dataset/raw/<dataset>/` |
+| 1. Export | `run_export.sh`, `run_export_general.sh` | raw FBX / GLB | `export/<dataset>/{motions/*.npz, videos/*.mp4, tpose/*.png, joint_names.json}` |
+| 2a. Render | `run_render_motion.sh`, `run_render_tpose.sh` | raw FBX / GLB (mixamo: animated characters) | `render/<dataset>/<clip>/v00{0..3}/*.png`, `render/<dataset>_tpose/<name>.png` |
+| 2b. Caption | `run_caption_motion.sh`, `run_caption_category.sh` | multi-view renders | `motion_captions.json`, `category_groups.json` |
+| 3. Joints | `run_joints_*.sh` | `joint_names.json` | `clean_joint_names.json`, `face_joint_names.json` |
+| 4. Extract | `run_extract_features.sh` | export NPZs + stage-2/3 JSONs | per-clip NPZs, `cond.npy`, `captions.json` |
+| 5. Animate | `run_animate_{motion,npz,fbx,mixamo}.sh` | motion clip + rigged mesh | animated `.glb` / `.fbx` |
+
+Stage 4 reads the caption and joint-annotation JSONs from the export directory, so stages 2 and 3 must run first.
+
+> [!NOTE]
+> **Mixamo runs stage 5 in the middle of the pipeline.** Raw Mixamo FBXs are animation-only (armature without mesh), so the render/caption input is produced by `run_animate_mixamo.sh`: export โ†’ animate a character โ†’ render โ†’ caption. The result also ships in the mixamo repo as `animation_motion_ybot/`, so this step can be skipped by downloading it.
+
+## Requirements
+
+The pipeline uses the shared `unimate` conda environment โ€” see [Environment Setup](../README.md#%EF%B8%8F-environment-setup) in the top-level README. Beyond it:
+
+| Tool | Used by |
+|------|---------|
+| [Blender](https://www.blender.org/) on `PATH` | stages 1 & 5 (headless `blender -b -P`) |
+| pip [`bpy`](https://pypi.org/project/bpy/) module (in the conda env) | stage 2a (EEVEE needs the module's GPU context) |
+| `ffmpeg` on `PATH` | video visualizations |
+| [`hf` CLI](https://hf.co/cli) | stage 0 |
+| CUDA GPU | stages 2a & 2b |
+| `OPENAI_API_KEY` / `GOOGLE_API_KEY` / `DEEPSEEK_API_KEY` | stage 2b (OpenAI / Gemini captioning) and stage 3 (OpenAI / DeepSeek joint annotation) API backends |
+
+## Directory Layout
+
+Code โ€” one directory per stage, bash wrappers in `scripts/`, shared code in `utils/`:
+
+```
+data_process/
+โ”œโ”€โ”€ scripts/ Bash entry points โ€” one run_<stage>_<task>.sh per task
+โ”œโ”€โ”€ motion_export/ Stage 1 raw assets โ†’ NPZ (Blender headless)
+โ”œโ”€โ”€ motion_rendering/ Stage 2a rigged assets โ†’ multi-view renders + T-pose grids (EEVEE)
+โ”œโ”€โ”€ vlm_caption/ Stage 2b multi-view renders โ†’ captions / body-plan categories (VLM)
+โ”œโ”€โ”€ joint_annotation/ Stage 3 joint-name cleanup + facing-direction joint selection
+โ”œโ”€โ”€ feature_extraction/ Stage 4 NPZ + metadata โ†’ canonical training clips + cond.npy
+โ”œโ”€โ”€ mesh_animation/ Stage 5 motion clip + rigged mesh โ†’ animated GLB/FBX (Blender)
+โ”œโ”€โ”€ tools/ Standalone utilities (summary merging, FBXโ†’GLB, QA visualizers)
+โ””โ”€โ”€ utils/ Shared library code (no entry points)
+```
+
+Data โ€” all wrappers share one convention (override any path through the environment variables each wrapper documents in its header):
+
+```
+dataset/raw/<dataset>/ raw FBX / GLB assets
+dataset/export/<dataset>/ stage-1 NPZs + previews + stage-2/3 metadata
+dataset/render/<dataset>/<clip>/v00{0..3}/ multi-view renders (captioning input)
+dataset/render/<dataset>_tpose/ T-pose 2x2 grids (category input)
+dataset/features/<dataset>/ stage-4 training clips + cond.npy
+```
+
+`<dataset>` is one of `truebones`, `mixamo`, `objaverse`.
+
+## Getting the Raw Data
+
+The source assets are hosted on the Hugging Face Hub and download straight into the layout above:
+
+```bash
+bash data_process/scripts/run_download.sh mixamo # โ†’ dataset/raw/mixamo/{animation_motion,character_refined,...}
+bash data_process/scripts/run_download.sh objaverse # โ†’ dataset/raw/objaverse/glb
+bash data_process/scripts/run_download.sh truebones # โ†’ dataset/raw/truebones (annotations only โ€” see below)
+```
+
+| Dataset | Source |
+|---------|--------|
+| `mixamo` | [Linzhan/Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters) |
+| `objaverse` | [Linzhan/Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) |
+| `truebones` | [Linzhan/Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata, renders, build scripts โ€” no motion files) |
+
+The mixamo and truebones repos also carry the stage-2a multi-view renders as per-view MP4 previews plus camera JSONs (`animation_motion_render/`, `animation_render/` โ€” see each repo's README). The per-frame PNGs the caption stage reads are not hosted; stage 2a regenerates them (or extract stills from the MP4s).
+
+> [!IMPORTANT]
+> **The Truebones motion files are not downloadable from us.** The Truebones ZOO animal pack is a commercial product whose license does not allow redistribution, so its repo above ships annotations only. To reproduce the truebones part of the pipeline, purchase the pack from [Truebones](https://truebones.com), unpack it to `dataset/raw/truebones/Truebone_Z-OO/{Animal}/`, and run the downloaded `scripts/pipeline/` (see the repo's README) to rebuild the per-clip layout โ€” one binary FBX per clip, flat in a single directory:
+>
+> ```
+> dataset/raw/truebones/animation/{Species}-{Action}.fbx # e.g. Alligator-Big_Mouth.fbx
+> ```
+>
+> (Species names must not contain `-` โ€” the first dash separates species from action.)
+>
+> Stage 5 additionally needs the **original per-animal layout** for the character meshes, since the flat per-clip files above are the animation source only:
+>
+> ```
+> dataset/raw/truebones/Truebone_Z-OO/{Animal}/*.fbx # e.g. Truebone_Z-OO/Dog-2/
+> ```
+>
+> Directory names are matched ignoring separators and case, so the clip-name object type `Dog2` resolves to the `Dog-2` directory.
+
+## Quick Start
+
+End-to-end run for Objaverse (Truebones is identical with `objaverse` โ†’ `truebones`; see the [note above](#overview) for Mixamo's extra animate step):
+
+```bash
+# 0. Raw data
+bash data_process/scripts/run_download.sh objaverse
+
+# 1. Export raw GLBs to NPZ motion data
+bash data_process/scripts/run_export.sh objaverse --multi-worker 8
+
+# 2a. Render multi-view frames + T-pose grids
+bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8
+bash data_process/scripts/run_render_tpose.sh objaverse
+
+# 2b. Caption motions + classify body plans (local Qwen3-VL by default)
+bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu
+bash data_process/scripts/run_caption_category.sh objaverse
+
+# 3. Joint-name cleanup + facing-direction pair
+bash data_process/scripts/run_joints_names_clean_llm.sh objaverse
+bash data_process/scripts/run_joints_face_select_llm.sh objaverse
+
+# 4. Canonicalized training clips + topology condition
+bash data_process/scripts/run_extract_features.sh objaverse
+```
+
+Every stage is resumable โ€” rerunning a wrapper skips already-complete outputs.
+
+## Pipeline Stages
+
+### Stage 1 โ€” Export: raw assets โ†’ NPZ
+
+```bash
+bash data_process/scripts/run_export.sh truebones # flat per-clip {Species}-{Action}.fbx
+bash data_process/scripts/run_export.sh mixamo # animation-only FBX (no mesh)
+bash data_process/scripts/run_export.sh objaverse # GLB/GLTF
+bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # parallel workers (mixamo too)
+bash data_process/scripts/run_export.sh mixamo --no-vis # skip per-clip MP4 previews (much faster)
+DATA_DIR=my_assets bash data_process/scripts/run_export.sh # auto mode: .glb/.gltf and .fbx side by side
+```
+
+Skeletons are pruned of control/helper bones that carry no skin weight and converted to Y-up. Mixamo needs its explicit preset: animation-only FBXs carry no mesh, so auto mode cannot detect them. For custom assets there is a general exporter (single file or a directory of mixed GLB/GLTF/FBX):
+
+```bash
+bash data_process/scripts/run_export_general.sh my_model.glb
+bash data_process/scripts/run_export_general.sh my_assets/ --multi-worker 8
+```
+
+### Stage 2a โ€” Render: multi-view frames & T-pose grids
+
+Renders each motion from four fixed views (front / back / left / right) with EEVEE, plus a T-pose 2x2 grid per asset for body-plan classification:
+
+```bash
+bash data_process/scripts/run_render_motion.sh truebones
+bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8
+bash data_process/scripts/run_render_motion.sh objaverse --missing-only # resume: skip complete clips
+bash data_process/scripts/run_render_tpose.sh objaverse
+DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_motion.sh mixamo
+```
+
+Render folders use the same clip naming as stage 1, so they line up with the exported NPZs. Rendering is capped at the first 200 frames per clip (~6.7 s at 30 fps) โ€” enough context for captioning without paying for full-length renders.
+
+`--fps` (default 30) must match the export stage's `--fps`: glTF stores keyframe times in seconds, so the importer resamples them at the scene frame rate, and a mismatch makes the renders cover a different time window than the NPZs.
+
+### Stage 2b โ€” Caption: renders โ†’ text
+
+One wrapper; the backend is picked from `MODEL`:
+
+```bash
+bash data_process/scripts/run_caption_motion.sh truebones # local Qwen3-VL (default)
+bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu # shard across all visible GPUs
+MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse # Gemini (GOOGLE_API_KEY)
+MODEL=gpt-5-mini bash data_process/scripts/run_caption_motion.sh mixamo # OpenAI (OPENAI_API_KEY)
+bash data_process/scripts/run_caption_category.sh objaverse # body-plan classification
+```
+
+`--multi-gpu` applies to the local Qwen backend only; API backends scale via `NUM_WORKERS`. The local backend feeds each clip's four camera views as **native video** at the render frame rate (all frames, `--vision_input video`, the default); API backends receive frame sequences instead, capped by `--max_frames_per_view` (default 64, uniformly sampled including the first and last frame). For mixamo and truebones the official catalogue prompts (`animation_motion_prompts.json` / `animation_prompts.json`) are attached as reference hints automatically (`HINTS_JSON` overrides, `HINTS_JSON=""` disables). Clips whose renders are incomplete are recorded as failures and retried later rather than captioned from a partial set.
+
+### Stage 3 โ€” Joint annotation: name cleanup & facing direction
+
+Two `<task>_<step>` trios โ€” joint-name cleaning (`names_*`) and facing-direction pair selection (`face_*`) โ€” each with a rule-based and an LLM variant. The LLM variants default to `deepseek-v4-flash` (also `gpt-*` or a local HF model id via `MODEL=`) and fall back to the rules on failure.
+
+**Required** โ€” these two produce the artifacts stage 4 consumes (`clean_joint_names.json`, `face_joint_names.json`):
+
+```bash
+bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # LLM names (falls back to rules)
+bash data_process/scripts/run_joints_face_select_llm.sh objaverse # LLM facing pair
+```
+
+**Optional** โ€” refinement passes over the files above. Skip them entirely on a clean LLM run; reach for them when the required passes recorded failures (`failed_*.txt`) or when spot checks reveal residual label / pairing errors:
+
+```bash
+bash data_process/scripts/run_joints_names_correct_llm.sh objaverse # re-check every label, correct in place
+bash data_process/scripts/run_joints_face_correct_llm.sh objaverse # retry rigs whose facing pair came back "empty"
+```
+
+`names_correct_llm` sends each `{raw, current}` label pair back to the LLM to keep or fix (most useful for rigs that fell back to the rules โ€” restrict with `FAILED_LIST=.../failed_clean_names.txt`); `face_correct_llm` re-attempts only `source == "empty"` entries. Both edit in place and create a `.bak` sibling on first run.
+
+Pure rule-based variants exist as `run_joints_names_clean_rule.sh` / `run_joints_face_select_rule.sh` (no API needed โ€” useful offline or as a fast first pass). The QA visualizers behind this stage live in `data_process/tools/`: `run_joints_vis_tpose.sh` renders annotated T-pose PNGs (`tools/vis_tpose.py`), and to sanity-check a picked pair on an actual clip, `python -m data_process.tools.vis_motion <motion.npz> --direction face --face-joints I J` renders an MP4 with the implied heading arrow.
+
+Both cleaners default to the same `clean_joint_names.json`, so the LLM pass skips rigs the rule-based pass already filled in; pass `--overwrite` to redo everything or `--redo_failed` to redo only the rigs listed in `failed_clean_names.txt`.
+
+### Stage 4 โ€” Extract: NPZ + metadata โ†’ training clips
+
+```bash
+bash data_process/scripts/run_extract_features.sh objaverse
+
+# crop long motions into overlapping fixed-length clips instead of truncating
+APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones
+
+# parallel over object types, skip MP4 previews
+NUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse
+```
+
+Mixamo is reduced to a built-in **22-joint humanoid core** (finger chains and End bones dropped) via `--mixamo_core_joints` (default on); pass `--no-mixamo_core_joints` to keep the full 65-joint rig. Truebones and objaverse always use the full skeleton.
+
+Every object type is canonicalized against its T-pose (facing โ†’ XZ-centering โ†’ diameter scaling โ†’ grounding), static pre/post-roll is trimmed, low-activity clips are filtered, and the per-object topology condition is written to `cond.npy`. Finished object types are cached under `cond_parts/`, so interrupted runs resume where they stopped; the cache is keyed on the clip/threshold settings **and** on digests of the caption / joint-annotation inputs, so changing any of them (including a stage-2/3 rerun) re-processes the affected object types instead of silently mixing old and new results.
+
+Per-clip preview MP4s use a ground-plane view โ€” checkerboard stage, smoothly following camera and root trajectory (`--vis_ground`, default on; `--no-vis_ground` renders the plain cubic view instead).
+
+Each motion is grounded on its own lowest joint by default; pass `--use_tpos_ground_height` to reuse the T-pose's ground height for every clip instead. Either way `cond['ground_height']` and `cond['ground_height_mode']` record what was actually applied. Object types that fail are logged to `extract_errors.log` and skipped rather than aborting the run, and are retried on the next run.
+
+### Stage 5 โ€” Animate: drive a mesh with a motion
+
+Used at inference time to render generated motions onto their rigged meshes:
+
+```bash
+ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse # stage-4 / generated clip + cond.npy
+ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ†’ rig
+bash data_process/scripts/run_animate_fbx.sh # raw animation FBXs โ†’ any character
+bash data_process/scripts/run_animate_mixamo.sh # batch over Mixamo export NPZs (Y Bot)
+```
+
+`run_animate_fbx.sh` needs no export stage at all: it bakes a raw animation clip โ€” or a whole directory of them โ€” onto any character that shares the clips' bone names (the Mixamo convention), e.g. the full library on a different character:
+
+```bash
+CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \
+ bash data_process/scripts/run_animate_fbx.sh # โ†’ outputs/animated_Amy/*.fbx
+```
+
+The character is auto-resolved for truebones / objaverse. For mixamo it defaults to `character_refined/Y_Bot.fbx` โ€” the character whose skeleton the animation FBXs are authored on (identical 65-bone rig, so no bone reconciliation is needed); override with `CHAR_PATH` to use a different character. Armature bones absent from the motion are handled by `--extra_bones_strategy` (`merge` weights into the nearest kept ancestor, `remove` bones and their vertices, or `keep` them un-keyed).
+
+## Data Formats
+
+**Stage-1 export NPZ** (`dataset/export/<dataset>/motions/*.npz`) โ€” one file per clip:
+
+| Field | Shape | Description |
+|-------|-------|-------------|
+| `rest_local_pos` / `rest_local_rot` | `(J, 3)` / `(J, 4)` | rest-pose local translations / rotations (quaternions) |
+| `anim_local_pos` / `anim_local_rot` | `(T, J, 3)` / `(T, J, 4)` | per-frame local translations / rotations |
+| `offsets` | `(J, 3)` | bind-pose bone offsets |
+| `parents` | `(J,)` | parent indices, `-1` for the root |
+| `names` | `(J,)` | bone name strings |
+| `skin_matrix` | `(V, J)` | skinning weights (empty when no mesh) |
+| `fps`, `action_name` | scalar | frame rate and source action |
+
+**Stage-4 feature NPZ** (`dataset/features/<dataset>/motions/*.npz`) โ€” canonicalized per-clip arrays, named `{object}-{motion}-{clip_idx}.npz`:
+
+| Field | Shape | Description |
+|-------|-------|-------------|
+| `global_positions` | `(F, J, 3)` | global joint positions at canonical scale |
+| `local_rotations` | `(F, J, 4)` | local joint rotations (quaternions) |
+| `root_facing_quat` | `(F, 4)` | per-frame root quaternion aligning the skeleton to face Z+ |
+| `fps` | scalar | frame rate of the clip (after any downsampling) |
+
+**`cond.npy`** โ€” one dict per object type holding everything the training dataloader needs about the skeleton: topology (`parents`, `offsets`, joint names), T-pose positions/rotations, graph structure (relations, distances, depths, spectral features, kinematic chains), calibration (`scale_factor`, `ground_height`), face joints and per-clip captions.
+
+## Conventions
+
+- **Run from the repo root.** Wrappers `cd` there themselves.
+- **Resumable by default.** Every batch stage skips outputs that already exist and records failures in sibling error logs, so failed items are retried on the next run without redoing finished work.
+- **Env-var overrides.** Every wrapper documents its overridable paths/knobs in its header comment; run any wrapper with `-h` to print it.
+- **Parallelism.** Blender stages shard with `--multi-worker N`; local-model captioning shards with `--multi-gpu`; API backends use concurrent workers via `NUM_WORKERS`.
+- **Naming.** Clips are named `{object_type}-{action}` (stage 1) and `{object_type}-{motion}-{clip_idx}` (stage 4). Mixamo is a single shared skeleton, so its files carry no object-type prefix.
+
+## Troubleshooting
+
+- **EEVEE renders fail / produce black frames under `blender -b`** โ€” stage 2a must run with plain `python` and the pip `bpy` module (the wrappers already do); headless `blender -b` has no GPU display surface for EEVEE.
+- **Offline compute nodes** โ€” with a populated HuggingFace cache, export `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` before the local Qwen caption/classify runs.
+- **`joint_names.json` is missing object types that have clips** โ€” an export killed mid-run (Slurm walltime, OOM) can leave the summary JSONs behind the completion markers. Rebuild them from the markers:
+ ```bash
+ python -m data_process.tools.merge_summaries --output_dir dataset/export/<dataset>
+ ```
+ Additive and idempotent.
diff --git a/data_process/__init__.py b/data_process/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/feature_extraction/__init__.py b/data_process/feature_extraction/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/feature_extraction/extract_features.py b/data_process/feature_extraction/extract_features.py
new file mode 100644
index 0000000..c77c419
--- /dev/null
+++ b/data_process/feature_extraction/extract_features.py
@@ -0,0 +1,469 @@
+"""Process exported NPZ motions into canonicalized training clips.
+
+Supports three dataset layouts via ``--dataset_type``:
+
+- ``truebones``: per-object NPZ files under ``data_dir/motions``, named
+ ``{object_type}-{motion}.npz``. Object types are discovered from filename
+ prefixes.
+- ``objaverse``: same layout as truebones, plus an optional
+ ``filtered_objects.txt`` listing object types to skip.
+- ``mixamo``: single skeleton / object type. All NPZs live directly under
+ ``data_dir/motions`` with no ``{object_type}-`` prefix; caption keys are
+ bare motion stems.
+
+Stage-2 (captions, category groups) and stage-3 (clean / face joint names)
+metadata JSONs are read from ``data_dir`` โ€” see
+:mod:`data_process.feature_extraction.metadata`.
+
+Object types are independent, so they can be processed in parallel
+(``--num_workers``) and are resumable: each finished object type caches its
+result under ``<save_dir>/cond_parts/`` and is skipped on rerun. The cache
+records the clip / threshold / topology settings it was built with and is
+re-processed automatically when they change; delete ``cond_parts/`` (or one
+entry) to force re-processing for any other reason.
+
+Object types that fail are logged to ``<save_dir>/extract_errors.log``,
+recorded in ``filtered_clips.json`` and skipped (no cache entry is written,
+so they are retried on the next run) โ€” one bad object never aborts the run.
+
+Usage:
+ python -m data_process.feature_extraction.extract_features \\
+ --dataset_type truebones \\
+ --data_dir dataset/export/truebones \\
+ --save_dir dataset/features/truebones
+"""
+
+import argparse
+import hashlib
+import json
+import multiprocessing as mp
+import os
+from os.path import join as pjoin
+
+import numpy as np
+from loguru import logger
+from tqdm import tqdm
+
+from data_process.feature_extraction.metadata import (
+ MIXAMO_CORE_JOINTS,
+ atomic_np_save,
+ build_stats,
+ check_metadata_object_types_consistent,
+ get_object_captions,
+ get_object_face_joints,
+ get_object_metadata,
+ load_all_metadata,
+ print_summary,
+ save_metadata_report,
+ save_outputs,
+)
+from data_process.utils.motion_features import process_object
+
+
+MIXAMO_OBJECT_TYPE = 'mixamo'
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Process exported NPZ motions into canonicalized training clips.")
+ # Dataset selection
+ parser.add_argument("--dataset_type", type=str, required=True,
+ choices=['truebones', 'objaverse', 'mixamo'],
+ help="Which dataset layout to process")
+ # Directories
+ parser.add_argument("--data_dir", type=str, required=True,
+ help="Root data directory (motions under data_dir/motions, "
+ "metadata JSONs in data_dir)")
+ parser.add_argument("--save_dir", type=str, required=True,
+ help="Output directory for processed clips and metadata")
+ # Clip windowing
+ parser.add_argument("--max_clip_len", type=int, default=180,
+ help="Max frames per saved clip")
+ parser.add_argument("--diffusion_max_len", type=int, default=60,
+ help="Model training max length (dataloader random-crops to this)")
+ parser.add_argument("--apply_clip", action="store_true", default=False,
+ help="Crop long motions into overlapping fixed-length clips of "
+ "max_clip_len frames (stride = max_clip_len - "
+ "diffusion_max_len). Otherwise (default), keep only the "
+ "first max_clip_len frames of each motion.")
+ # Topology
+ parser.add_argument("--max_path_len", type=int, default=5,
+ help="Max path length for topology edge relations")
+ parser.add_argument("--max_freqs", type=int, default=8,
+ help="Number of Laplacian eigenvector frequencies for "
+ "spectral joint features")
+ # Scaling / grounding
+ parser.add_argument("--target_diameter", type=float, default=2.0,
+ help="Target skeleton diameter for leaf-to-leaf scaling")
+ parser.add_argument("--use_tpos_ground_height", action="store_true", default=False,
+ help="Ground every motion of an object type on the T-pose's "
+ "floor height. Otherwise (default), each motion is "
+ "grounded on its own minimum Y. The applied value is "
+ "recorded in cond as ground_height / ground_height_mode.")
+ # Filtering
+ parser.add_argument("--activity_threshold", type=float, default=0.02,
+ help="Min joint activity (temporal spread at canonical "
+ "scale) to keep a clip")
+ parser.add_argument("--static_threshold", type=float, default=1e-5,
+ help="Per-frame max-joint displacement below which a frame "
+ "is static; leading/trailing static frames are trimmed")
+ parser.add_argument("--min_frames", type=int, default=8,
+ help="Minimum frame count for a motion to be kept "
+ "(checked after downsampling and static trimming)")
+ parser.add_argument("--mixamo_core_joints", action=argparse.BooleanOptionalAction,
+ default=True,
+ help="(mixamo only) Restrict the rig to the built-in "
+ "22-joint humanoid core (drops finger chains and "
+ "End bones) โ€” replaces the retired "
+ "corps_joint_names.json. Use --no-mixamo_core_joints "
+ "to keep the full 65-joint skeleton.")
+ parser.add_argument("--min_joints", type=int, default=8,
+ help="Skip object types with fewer joints than this "
+ "(counted after corps filtering)")
+ parser.add_argument("--max_joints", type=int, default=150,
+ help="Skip object types with more joints than this "
+ "(counted after corps filtering)")
+ # Objaverse-only filtering
+ parser.add_argument("--filtered_objects", type=str, default="auto",
+ help="(objaverse only) Path to a txt file listing object "
+ "types to skip (one per line; blanks and '#' comments "
+ "ignored). Default 'auto' uses "
+ "<data_dir>/filtered_objects.txt if present. Pass an "
+ "empty string to disable.")
+ # Execution
+ parser.add_argument("--num_workers", type=int, default=1,
+ help="Parallel worker processes (object types are independent)")
+ parser.add_argument("--vis", action=argparse.BooleanOptionalAction, default=True,
+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs)")
+ parser.add_argument("--vis_ground", action=argparse.BooleanOptionalAction,
+ default=True,
+ help="Render previews with a checkerboard ground plane, "
+ "follow camera and root trajectory (default). "
+ "--no-vis_ground falls back to the legacy plain "
+ "cubic view.")
+ return parser.parse_args()
+
+
+def resolve_filtered_objects_path(path, data_dir):
+ if path == "auto":
+ default = pjoin(data_dir, 'filtered_objects.txt')
+ return default if os.path.isfile(default) else None
+ return path or None
+
+
+def load_filtered_objects(path):
+ if not path:
+ return set()
+ with open(path, 'r') as f:
+ return {
+ line.strip() for line in f
+ if line.strip() and not line.startswith('#')
+ }
+
+
+def discover_object_types(motion_dir, dataset_type, args):
+ """Return (object_types, all_motions) given the dataset layout.
+
+ Only ``.npz`` files are considered, so stray side files in ``motions/``
+ never turn into object types with zero motions.
+ """
+ entries = os.listdir(motion_dir)
+ all_motions = [f for f in entries if f.endswith('.npz')]
+ if len(all_motions) != len(entries):
+ logger.info(f'Ignoring {len(entries) - len(all_motions)} non-NPZ '
+ f'entries in {motion_dir}')
+
+ if dataset_type == 'mixamo':
+ # Single object type; no prefix-based grouping.
+ return [MIXAMO_OBJECT_TYPE], all_motions
+
+ object_types = sorted(set(m.split('-')[0] for m in all_motions))
+ logger.info(f'Found {len(all_motions)} motion files, {len(object_types)} object types')
+
+ if dataset_type == 'objaverse':
+ filtered_path = resolve_filtered_objects_path(args.filtered_objects, args.data_dir)
+ filtered_object_types = load_filtered_objects(filtered_path)
+ if filtered_object_types:
+ before = len(object_types)
+ object_types = [o for o in object_types if o not in filtered_object_types]
+ logger.info(f'Loaded {len(filtered_object_types)} entries from {filtered_path}; '
+ f'{before} -> {len(object_types)} object types after filtering')
+ elif args.filtered_objects:
+ logger.info(f'No filtered_objects file found '
+ f'(--filtered_objects={args.filtered_objects!r}); '
+ f'processing all {len(object_types)} object types')
+
+ return object_types, all_motions
+
+
+def collect_object_npzs(object_type, dataset_type, motion_dir, all_motions):
+ if dataset_type == 'mixamo':
+ return sorted(pjoin(motion_dir, f) for f in all_motions if f.endswith('.npz'))
+ return sorted(
+ pjoin(motion_dir, f) for f in all_motions
+ if f.startswith(object_type + '-') and f.endswith('.npz')
+ )
+
+
+def resolve_captions(object_type, dataset_type, motion_captions):
+ # Mixamo caption keys are bare motion stems (no "{object_type}-" prefix),
+ # so the full captions dict is passed through untouched.
+ if dataset_type == 'mixamo':
+ return motion_captions
+ return get_object_captions(object_type, motion_captions)
+
+
+def _object_part_path(save_dir, object_type):
+ return pjoin(save_dir, 'cond_parts', f'{object_type}.npy')
+
+
+def _error_log_path(save_dir):
+ return pjoin(save_dir, 'extract_errors.log')
+
+
+# Settings that determine a cached object result's contents. A cache entry
+# built with different values is stale and gets re-processed. ``save_vis``
+# is deliberately excluded: it only controls the MP4 previews, not the clips
+# or the cond.
+CACHE_PARAM_KEYS = (
+ 'max_clip_len', 'clip_stride', 'apply_clip',
+ 'max_path_len', 'max_freqs',
+ 'target_diameter', 'activity_threshold', 'static_threshold',
+ 'min_frames', 'min_joints', 'max_joints',
+ 'use_tpos_ground_height',
+ 'corps_names',
+)
+
+# Per-object METADATA inputs baked into the result (captions land in the
+# cond, face joints steer the facing canonicalization, clean names are
+# stored per joint). Hashed into the cache key so a stage-2/3 rerun โ€”
+# new captions, corrected face pairs, recleaned labels โ€” re-processes the
+# affected object types instead of silently resuming stale results.
+CACHE_DATA_KEYS = ('captions', 'face_joints', 'clean_names')
+
+
+def _digest(value):
+ """Stable short digest of a JSON-serializable metadata value."""
+ blob = json.dumps(value, sort_keys=True, default=str)
+ return hashlib.sha1(blob.encode('utf-8')).hexdigest()[:16]
+
+
+def _cache_params(task):
+ """Extract the cache-invalidating settings from a task dict."""
+ params = {k: task[k] for k in CACHE_PARAM_KEYS}
+ for k in CACHE_DATA_KEYS:
+ params[f'{k}_digest'] = _digest(task[k])
+ return params
+
+
+def _params_hash(params):
+ """Stable hash of the cache-invalidating settings."""
+ blob = json.dumps(params, sort_keys=True, default=str)
+ return hashlib.sha1(blob.encode('utf-8')).hexdigest()
+
+
+def _load_cached_result(part_path, object_type, params, params_hash):
+ """Return a cached object result, or None when absent / stale / unreadable."""
+ if not os.path.isfile(part_path):
+ return None
+ try:
+ payload = np.load(part_path, allow_pickle=True).item()
+ except Exception as e: # noqa: BLE001 โ€” a bad cache entry is just a miss
+ logger.warning(f'[{object_type}] unreadable cache {part_path} ({e}); '
+ f're-processing')
+ return None
+
+ if 'params_hash' not in payload:
+ logger.info(f'[{object_type}] cache invalidated (entry predates settings '
+ f'tracking); re-processing')
+ return None
+ if payload['params_hash'] == params_hash:
+ return payload.get('result')
+
+ cached_params = payload.get('params') or {}
+ diff = ', '.join(
+ f'{k}: {cached_params.get(k, "<missing>")} -> {v}'
+ for k, v in sorted(params.items())
+ if cached_params.get(k, '<missing>') != v)
+ logger.info(f'[{object_type}] cache invalidated (settings changed: {diff}); '
+ f're-processing')
+ return None
+
+
+def process_object_task(task):
+ """Process one object type, caching its result for resume.
+
+ ``task`` bundles every :func:`process_object` argument plus the part
+ path. The cached result is reused only when it was produced with the same
+ clip / threshold / topology settings (see ``CACHE_PARAM_KEYS``).
+
+ Failures are contained here: the object is reported as failed (and its
+ error appended to ``extract_errors.log``) instead of propagating, which
+ with ``mp.Pool`` would kill the whole run. No cache entry is written for
+ a failed object, so it is retried on the next run.
+
+ Returns ``(object_type, result_dict, from_cache)``.
+ """
+ object_type = task.pop('object_type')
+ part_path = task.pop('part_path')
+ params = _cache_params(task)
+ params_hash = _params_hash(params)
+
+ cached = _load_cached_result(part_path, object_type, params, params_hash)
+ if cached is not None:
+ return object_type, cached, True
+
+ try:
+ obj_cond, n_clips, n_frames, n_joints, filtered = process_object(
+ object_type, **task)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ logger.exception(f'[{object_type}] failed to process')
+ try:
+ os.makedirs(task['save_dir'], exist_ok=True)
+ with open(_error_log_path(task['save_dir']), 'a') as log_file:
+ log_file.write(f'{object_type}\t{type(e).__name__}: {e}\n')
+ except OSError as log_err:
+ logger.warning(f'[{object_type}] could not write error log: {log_err}')
+ return object_type, {
+ 'cond': None, 'n_clips': 0, 'n_frames': 0, 'n_joints': 0,
+ 'filtered': [{'name': f'{object_type} (all clips)',
+ 'reason': f'processing failed: {type(e).__name__}: {e}'}],
+ 'error': f'{type(e).__name__}: {e}',
+ }, False
+
+ result = {'cond': obj_cond, 'n_clips': n_clips, 'n_frames': n_frames,
+ 'n_joints': n_joints, 'filtered': filtered}
+ # Atomic: a worker killed mid-write can never leave a half-written cache
+ # entry behind (unreadable ones are treated as misses, but this keeps them
+ # from happening in the first place).
+ atomic_np_save(part_path, {'params_hash': params_hash, 'params': params,
+ 'result': result})
+ return object_type, result, False
+
+
+def build_object_tasks(args, clip_stride, motion_dir, metadata):
+ face_joint_names = metadata.get('face_joint_names')
+ clean_joint_names = metadata.get('clean_joint_names')
+ joint_names = metadata.get('joint_names')
+ motion_captions = metadata.get('motion_captions')
+
+ object_types, all_motions = discover_object_types(motion_dir, args.dataset_type, args)
+
+ tasks = []
+ for object_type in object_types:
+ face_joints = get_object_face_joints(object_type, face_joint_names)
+ if not face_joints:
+ if args.dataset_type == 'mixamo':
+ raise ValueError(f'No face_joints entry for {object_type}; '
+ f'face_joint_names.json is required for mixamo')
+ logger.info(f'No face_joints entry for {object_type}; using identity facing')
+
+ tasks.append(dict(
+ object_type=object_type,
+ part_path=_object_part_path(args.save_dir, object_type),
+ object_npzs=collect_object_npzs(object_type, args.dataset_type,
+ motion_dir, all_motions),
+ save_dir=args.save_dir,
+ max_clip_len=args.max_clip_len, clip_stride=clip_stride,
+ apply_clip=args.apply_clip,
+ max_path_len=args.max_path_len, max_freqs=args.max_freqs,
+ captions=resolve_captions(object_type, args.dataset_type, motion_captions),
+ face_joints=face_joints,
+ corps_names=(MIXAMO_CORE_JOINTS
+ if args.dataset_type == 'mixamo'
+ and args.mixamo_core_joints else None),
+ clean_names=get_object_metadata(object_type, clean_joint_names),
+ expected_names=get_object_metadata(object_type, joint_names),
+ require_face=args.dataset_type == 'mixamo',
+ target_diameter=args.target_diameter,
+ use_tpos_ground_height=args.use_tpos_ground_height,
+ activity_threshold=args.activity_threshold,
+ static_threshold=args.static_threshold,
+ min_frames=args.min_frames,
+ min_joints=args.min_joints, max_joints=args.max_joints,
+ save_vis=args.vis, vis_ground=args.vis_ground,
+ ))
+ return tasks
+
+
+def main(args):
+ clip_stride = args.max_clip_len - args.diffusion_max_len
+ if clip_stride <= 0:
+ message = (f'--diffusion_max_len ({args.diffusion_max_len}) must be smaller '
+ f'than --max_clip_len ({args.max_clip_len}): the clip stride is '
+ f'their difference, and {clip_stride} would '
+ + ('never advance the window'
+ if clip_stride == 0 else 'walk the window backwards'))
+ if args.apply_clip:
+ raise ValueError(message)
+ logger.warning(f'{message}. Harmless here because --apply_clip is off '
+ f'(the stride is unused), but fix it before enabling it.')
+
+ os.makedirs(args.save_dir, exist_ok=True)
+
+ motion_dir = pjoin(args.data_dir, 'motions')
+
+ metadata = load_all_metadata(args.data_dir)
+ logger.info(f'Loaded metadata from {args.data_dir}: '
+ + ', '.join(f'{k}={len(v)}' for k, v in metadata.items()))
+ check_metadata_object_types_consistent(metadata)
+ category_groups = metadata.get('category_groups')
+
+ tasks = build_object_tasks(args, clip_stride, motion_dir, metadata)
+
+ cond = {}
+ all_filtered_clips = {}
+ clips_per_object = {}
+ joints_per_object = {}
+ total_frames = 0
+ max_njoints = 0
+ n_cached = 0
+ failed_objects = {}
+
+ def accumulate(object_type, result, from_cache):
+ nonlocal total_frames, max_njoints, n_cached
+ n_cached += int(from_cache)
+ if result.get('error'):
+ failed_objects[object_type] = result['error']
+ if result['n_clips']:
+ cond[object_type] = result['cond']
+ clips_per_object[object_type] = result['n_clips']
+ joints_per_object[object_type] = result['n_joints']
+ total_frames += result['n_frames']
+ max_njoints = max(max_njoints, result['n_joints'])
+ if result['filtered']:
+ all_filtered_clips[object_type] = result['filtered']
+
+ pbar = tqdm(total=len(tasks), desc="Processing objects", unit="obj")
+ if args.num_workers > 1:
+ with mp.Pool(args.num_workers) as pool:
+ for object_type, result, from_cache in pool.imap_unordered(
+ process_object_task, tasks):
+ accumulate(object_type, result, from_cache)
+ pbar.update(1)
+ else:
+ for task in tasks:
+ accumulate(*process_object_task(task))
+ pbar.update(1)
+ pbar.close()
+
+ if n_cached:
+ logger.info(f'Resumed {n_cached}/{len(tasks)} object types from cond_parts/ '
+ f'(delete that directory to force re-processing)')
+ if failed_objects:
+ logger.error(f'{len(failed_objects)}/{len(tasks)} object types failed and were '
+ f'skipped (retried on the next run); see '
+ f'{_error_log_path(args.save_dir)}: '
+ + ', '.join(sorted(failed_objects)))
+
+ all_captions = save_outputs(args.save_dir, cond, all_filtered_clips,
+ category_groups=category_groups)
+ stats = build_stats(clips_per_object, joints_per_object, total_frames, max_njoints)
+ print_summary(stats)
+ save_metadata_report(args.save_dir, stats, all_filtered_clips,
+ all_captions=all_captions, category_groups=category_groups)
+ logger.info('Dataset processing complete.')
+
+
+if __name__ == "__main__":
+ main(parse_args())
diff --git a/data_process/feature_extraction/metadata.py b/data_process/feature_extraction/metadata.py
new file mode 100644
index 0000000..fc83854
--- /dev/null
+++ b/data_process/feature_extraction/metadata.py
@@ -0,0 +1,387 @@
+"""Metadata I/O and clip windowing for the feature-extraction stage.
+
+Loads the per-dataset JSON side files produced by the caption and
+joint-annotation stages (``motion_captions.json``, ``category_groups.json``,
+``clean_joint_names.json``, ``face_joint_names.json``, ...), provides
+case-insensitive per-object lookups, writes the stage outputs (``cond.npy``,
+``captions.json``, ``filtered_clips.json``, ``metadata.txt``) and computes
+clip windows. Used by :mod:`data_process.feature_extraction.extract_features`.
+"""
+
+import json
+import os
+import tempfile
+from contextlib import contextmanager
+from os.path import join as pjoin
+
+import numpy as np
+
+
+# ---------------------------------------------------------------------------
+# Atomic output writes
+# ---------------------------------------------------------------------------
+
+@contextmanager
+def atomic_output_path(path):
+ # type: (str) -> Iterator[str]
+ """Yield a temp path next to *path*, renamed into place on clean exit.
+
+ Stage outputs land at the end of runs that can take hours, so writing them
+ in place risks leaving a truncated ``cond.npy`` / ``captions.json`` behind
+ if the run is interrupted. Writers fill the yielded temp path instead; the
+ final ``os.replace`` is atomic, so readers see either the previous file or
+ the complete new one, and a failed write leaves the previous file intact.
+ """
+ directory = os.path.dirname(os.path.abspath(path))
+ os.makedirs(directory, exist_ok=True)
+ fd, tmp_path = tempfile.mkstemp(
+ dir=directory, prefix=f'.{os.path.basename(path)}.', suffix='.tmp')
+ os.close(fd)
+ try:
+ yield tmp_path
+ os.replace(tmp_path, path)
+ except BaseException:
+ try:
+ os.remove(tmp_path)
+ except OSError:
+ pass
+ raise
+
+
+def atomic_np_save(path, data):
+ # type: (str, object) -> None
+ """``np.save`` *data* to *path* atomically (see :func:`atomic_output_path`)."""
+ with atomic_output_path(path) as tmp_path:
+ # Passing a file object keeps np.save from appending '.npy' to the
+ # temp name (it only does that for string paths).
+ with open(tmp_path, 'wb') as f:
+ np.save(f, data)
+
+
+# ---------------------------------------------------------------------------
+# JSON loading & metadata helpers
+# ---------------------------------------------------------------------------
+
+def load_json(path):
+ # type: (str) -> Optional[dict]
+ """Load a JSON file if it exists, otherwise return None."""
+ if path and os.path.isfile(path):
+ with open(path, 'r') as f:
+ return json.load(f)
+ return None
+
+
+def load_motion_captions(source_dir):
+ # type: (str) -> Optional[Dict[str, str]]
+ """Load motion captions from ``motion_captions.json`` in *source_dir*."""
+ return load_json(pjoin(source_dir, 'motion_captions.json'))
+
+
+def load_category_groups(source_dir):
+ # type: (str) -> Optional[Dict[str, List[str]]]
+ """Load category-to-object-type groupings from ``category_groups.json``."""
+ return load_json(pjoin(source_dir, 'category_groups.json'))
+
+
+def load_clean_joint_names(source_dir):
+ # type: (str) -> Optional[Dict[str, List[str]]]
+ """Load cleaned joint names from ``clean_joint_names.json``."""
+ return load_json(pjoin(source_dir, 'clean_joint_names.json'))
+
+
+def load_face_joint_names(source_dir):
+ # type: (str) -> Optional[Dict[str, dict]]
+ """Load face joint info from ``face_joint_names.json``."""
+ return load_json(pjoin(source_dir, 'face_joint_names.json'))
+
+
+# The 22-joint humanoid core the mixamo rig is reduced to at extraction time
+# (drops the 40 finger bones plus the End bones). Replaces the retired
+# corps_joint_names.json side file; toggled by --mixamo_core_joints.
+MIXAMO_CORE_JOINTS = [
+ "mixamorig:Hips",
+ "mixamorig:Spine", "mixamorig:Spine1", "mixamorig:Spine2",
+ "mixamorig:Neck", "mixamorig:Head",
+ "mixamorig:LeftShoulder", "mixamorig:LeftArm",
+ "mixamorig:LeftForeArm", "mixamorig:LeftHand",
+ "mixamorig:RightShoulder", "mixamorig:RightArm",
+ "mixamorig:RightForeArm", "mixamorig:RightHand",
+ "mixamorig:LeftUpLeg", "mixamorig:LeftLeg",
+ "mixamorig:LeftFoot", "mixamorig:LeftToeBase",
+ "mixamorig:RightUpLeg", "mixamorig:RightLeg",
+ "mixamorig:RightFoot", "mixamorig:RightToeBase",
+]
+
+
+def load_joint_names(source_dir):
+ # type: (str) -> Optional[Dict[str, List[str]]]
+ """Load raw joint names from ``joint_names.json``."""
+ return load_json(pjoin(source_dir, 'joint_names.json'))
+
+
+def load_all_metadata(source_dir):
+ # type: (str) -> dict
+ """Load all available JSON metadata files from *source_dir*."""
+ loaders = {
+ 'motion_captions': load_motion_captions,
+ 'category_groups': load_category_groups,
+ 'clean_joint_names': load_clean_joint_names,
+ 'face_joint_names': load_face_joint_names,
+ 'joint_names': load_joint_names,
+ }
+ meta = {}
+ for key, loader in loaders.items():
+ data = loader(source_dir)
+ if data is not None:
+ meta[key] = data
+ return meta
+
+
+def check_metadata_object_types_consistent(
+ metadata,
+ keys=('clean_joint_names', 'face_joint_names', 'joint_names'),
+):
+ # type: (dict, tuple) -> None
+ """Verify per-object-type metadata dicts cover the same object types.
+
+ Compares the case-insensitive key sets of each present dict named by
+ *keys* in *metadata*; absent or empty dicts are skipped. Raises
+ ``ValueError`` on mismatch so drivers fail loudly on misaligned JSON
+ sources, and prints a consistency-OK line when all present dicts agree.
+ """
+ present = {n: {k.lower() for k in metadata[n].keys()}
+ for n in keys if metadata.get(n)}
+ if len(present) < 2:
+ return
+ ref_name, ref_set = next(iter(present.items()))
+ for other_name, other_set in present.items():
+ if other_name == ref_name:
+ continue
+ only_ref = sorted(ref_set - other_set)
+ only_other = sorted(other_set - ref_set)
+ if only_ref or only_other:
+ raise ValueError(
+ f"Object type mismatch between '{ref_name}' and '{other_name}': "
+ f"only in {ref_name}: {only_ref}; "
+ f"only in {other_name}: {only_other}"
+ )
+ print(f'Metadata consistency OK across {list(present.keys())}: '
+ f'{len(ref_set)} object types')
+
+
+def _dict_get_ci(d, key):
+ # type: (dict, str) -> object
+ """Case-insensitive dict lookup. Tries exact match first, then lowercase."""
+ if key in d:
+ return d[key]
+ key_lower = key.lower()
+ for k, v in d.items():
+ if k.lower() == key_lower:
+ return v
+ return None
+
+
+def get_object_metadata(object_type, metadata_dict):
+ # type: (str, Optional[dict]) -> object
+ """Case-insensitive lookup of *object_type* in a metadata dict, or None."""
+ if metadata_dict:
+ return _dict_get_ci(metadata_dict, object_type)
+ return None
+
+
+def get_object_face_joints(object_type, face_joint_names=None):
+ # type: (str, Optional[Dict[str, dict]]) -> Optional[dict]
+ """Return face joint info for *object_type*, or None."""
+ return get_object_metadata(object_type, face_joint_names)
+
+
+def get_object_captions(object_type, motion_captions=None):
+ # type: (str, Optional[Dict[str, str]]) -> Dict[str, str]
+ """Filter motion captions to entries for *object_type*.
+
+ Caption keys follow the ``{object_type}-{motion_name}`` base-name
+ convention used throughout the pipeline; matching is case-insensitive
+ on the prefix. Returns an empty dict when no captions are supplied or
+ none match.
+ """
+ if not motion_captions:
+ return {}
+ prefix = f'{object_type.lower()}-'
+ return {k: v for k, v in motion_captions.items()
+ if k.lower().startswith(prefix)}
+
+
+def save_json(path, data):
+ """Write *data* as indented JSON to *path*, atomically."""
+ with atomic_output_path(path) as tmp_path:
+ with open(tmp_path, 'w') as f:
+ json.dump(data, f, indent=2)
+
+
+def collect_captions(cond):
+ # type: (Dict[str, dict]) -> Dict[str, str]
+ """Merge per-object ``captions`` sub-dicts into a single flat dict."""
+ all_captions = {}
+ for obj_cond in cond.values():
+ if 'captions' in obj_cond:
+ all_captions.update(obj_cond['captions'])
+ return all_captions
+
+
+def save_outputs(save_dir, cond, all_filtered_clips, category_groups=None):
+ """Save ``cond.npy``, filtered clips, captions, and category groupings.
+
+ Captions and category groups are only written when present. Every file is
+ written atomically, so an interrupted run cannot leave a truncated output
+ behind for the training loader to trip over. Returns the merged captions
+ dict so callers can pass it to ``save_metadata_report``.
+ """
+ atomic_np_save(pjoin(save_dir, 'cond.npy'), cond)
+
+ if all_filtered_clips:
+ save_json(pjoin(save_dir, 'filtered_clips.json'), all_filtered_clips)
+ total = sum(len(v) for v in all_filtered_clips.values())
+ print(f'Saved {total} filtered clips to filtered_clips.json')
+
+ all_captions = collect_captions(cond)
+ if all_captions:
+ save_json(pjoin(save_dir, 'captions.json'), all_captions)
+ print(f'Saved {len(all_captions)} clip captions')
+
+ if category_groups:
+ save_json(pjoin(save_dir, 'category_groups.json'), category_groups)
+
+ return all_captions
+
+
+def build_stats(clips_per_object, joints_per_object, total_frames, max_njoints, fps=30):
+ """Aggregate per-object counts into the dataset summary dict.
+
+ ``duration_min`` is a nominal figure: clips carry their own ``fps`` field
+ (30 for every source, 30 after the >=60 fps down-sampling in
+ :func:`~data_process.utils.motion_features._load_npz_anim`), but the
+ duration here simply divides the frame total by *fps*. The assumption is
+ reported alongside the value, so ``fps`` is echoed back in the stats.
+ """
+ return {
+ 'total_clips': sum(clips_per_object.values()),
+ 'total_frames': total_frames,
+ 'duration_min': total_frames / fps / 60,
+ 'fps': fps,
+ 'max_njoints': max_njoints,
+ 'clips_per_object': clips_per_object,
+ 'joints_per_object': joints_per_object,
+ }
+
+
+def save_metadata_report(save_dir, stats, all_filtered_clips,
+ all_captions=None, category_groups=None):
+ """Write a human-readable metadata summary to ``metadata.txt``.
+
+ ``all_captions`` and ``category_groups`` are optional; their sections are
+ omitted when not supplied. The file is written atomically.
+ """
+ total_filtered = sum(len(v) for v in all_filtered_clips.values())
+
+ with atomic_output_path(pjoin(save_dir, 'metadata.txt')) as tmp_path, \
+ open(tmp_path, 'w') as f:
+ f.write(f'Total clips: {stats["total_clips"]}\n')
+ f.write(f'Total frames: {stats["total_frames"]}\n')
+ f.write(f'Total duration (minutes, assuming {stats.get("fps", 30)} fps): '
+ f'{stats["duration_min"]:.2f}\n')
+ f.write(f'Max joints in dataset: {stats["max_njoints"]}\n')
+ if all_captions is not None:
+ f.write(f'Total captions: {len(all_captions)}\n')
+
+ f.write('Clips per object type:\n')
+ for obj, count in stats['clips_per_object'].items():
+ f.write(f' {obj}: {count}\n')
+
+ f.write('Joints per object type:\n')
+ for obj, count in stats['joints_per_object'].items():
+ f.write(f' {obj}: {count}\n')
+
+ f.write(f'Total filtered clips: {total_filtered}\n')
+ if all_filtered_clips:
+ f.write('Filtered clips per object type:\n')
+ for obj, clips in all_filtered_clips.items():
+ f.write(f' {obj}: {len(clips)}\n')
+
+ if category_groups:
+ f.write('Categories:\n')
+ for cat, members in category_groups.items():
+ processed = [m for m in members if m in stats['clips_per_object']]
+ f.write(f' {cat}: {len(processed)} object types\n')
+
+
+# ---------------------------------------------------------------------------
+# Clip windowing
+# ---------------------------------------------------------------------------
+
+def compute_clip_windows(nframes, max_clip_len, clip_stride):
+ """Compute (start, end) windows for cropping a sequence into clips.
+
+ Windows are ``max_clip_len`` frames long and start every ``clip_stride``
+ frames. No returned window is ever longer than ``max_clip_len`` (the whole
+ sequence is returned unsliced only when it already fits).
+
+ Leftover frames past the last strided window are handled by tail length:
+
+ - at least half a clip: emitted as their own (shorter) trailing window;
+ - shorter than that, but still adding at least as many new frames as a
+ normal window's overlap (``max_clip_len - clip_stride``): covered by a
+ full-length window anchored at the final frame โ€” replacing the last
+ window when the frames it gives up stay covered by the one before it,
+ otherwise appended;
+ - shorter than that: dropped, rather than glued onto the previous window
+ (which is what used to push clips past ``max_clip_len``).
+ """
+ if not max_clip_len or nframes <= max_clip_len:
+ return [(0, nframes)]
+
+ clips = []
+ for start in range(0, nframes - max_clip_len + 1, clip_stride):
+ clips.append((start, start + max_clip_len))
+
+ last_end = clips[-1][1]
+ if last_end < nframes:
+ tail_len = nframes - last_end
+ if tail_len >= max_clip_len // 2:
+ clips.append((last_end, nframes))
+ else:
+ # Cover the tail with a full-length window anchored at the end.
+ slid_start = nframes - max_clip_len
+ prev_end = clips[-2][1] if len(clips) >= 2 else 0
+ if slid_start <= prev_end:
+ # Free: the frames the last window gives up are still covered.
+ clips[-1] = (slid_start, nframes)
+ elif tail_len >= max(1, max_clip_len - clip_stride):
+ # Worth an extra window: it adds at least as many new frames
+ # as a normal window's overlap.
+ clips.append((slid_start, nframes))
+ # Otherwise the tail is too small to pay for a window of its own.
+
+ return clips
+
+
+def get_motion_clip_windows(nframes, max_clip_len, clip_stride, apply_clip=True):
+ """Return clip windows for a motion sequence.
+
+ When ``apply_clip`` is True, crops into overlapping fixed-length clips via
+ :func:`compute_clip_windows`. When False, returns a single window covering
+ at most the first ``max_clip_len`` frames (i.e., truncation, not cropping).
+ """
+ if apply_clip:
+ return compute_clip_windows(nframes, max_clip_len, clip_stride)
+ clip_end = min(nframes, max_clip_len) if max_clip_len else nframes
+ return [(0, clip_end)]
+
+
+# ---------------------------------------------------------------------------
+# Summary
+# ---------------------------------------------------------------------------
+
+def print_summary(stats):
+ print(f'Total clips: {stats["total_clips"]}, Frames: {stats["total_frames"]}, '
+ f'Duration: {stats["duration_min"]:.1f}m (@{stats.get("fps", 30)} fps), '
+ f'Max joints: {stats["max_njoints"]}')
diff --git a/data_process/joint_annotation/__init__.py b/data_process/joint_annotation/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/joint_annotation/face_correct_llm.py b/data_process/joint_annotation/face_correct_llm.py
new file mode 100644
index 0000000..d31d328
--- /dev/null
+++ b/data_process/joint_annotation/face_correct_llm.py
@@ -0,0 +1,235 @@
+"""Re-run LLM face-joint selection on rigs whose ``source == "empty"``.
+
+:mod:`face_select_llm` writes the empty sentinel
+``{r_hip: {raw:"", clean:""}, l_hip: {raw:"", clean:""}, source: "empty"}``
+whenever neither the rule-based hint nor the LLM finds a bilateral pair or
+a head/tail body axis. This script re-attempts those rigs only (by default
+every ``source == "empty"`` entry in ``--face``; optionally a subset listed
+in ``--failed_list``) and, when the LLM now finds a valid pair, overwrites
+the entry in place. Rigs that still fail keep their existing empty entry.
+
+A ``.bak`` sibling of ``--face`` is created on first run. Rigs that remain
+empty are written to ``still_empty_face_joints.txt``.
+
+Usage:
+ python -m data_process.joint_annotation.face_correct_llm \\
+ --raw <export_dir>/joint_names.json \\
+ --cleaned <export_dir>/clean_joint_names.json \\
+ --face <export_dir>/face_joint_names.json [--model gpt-5]
+"""
+
+import argparse
+import json
+import os
+import shutil
+import sys
+
+from loguru import logger
+from tqdm import tqdm
+
+_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+from data_process.joint_annotation.face_select_llm import ( # noqa: E402
+ SYSTEM_PROMPT,
+ resolve_rig,
+)
+from data_process.joint_annotation.llm import ( # noqa: E402
+ LLMClient,
+ MAX_RETRIES,
+ add_llm_args,
+ append_id_list,
+ read_id_list,
+ save_json,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Entry classification
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _hips(entry):
+ if not isinstance(entry, dict):
+ return None, None
+ r, l = entry.get("r_hip", {}), entry.get("l_hip", {})
+ if not isinstance(r, dict) or not isinstance(l, dict):
+ return None, None
+ return r, l
+
+
+def is_non_empty_entry(entry):
+ # type: (object) -> bool
+ """True iff ``entry`` has a real (non-blank) r_hip/l_hip pair."""
+ r, l = _hips(entry)
+ return bool(r and l and r.get("raw") and l.get("raw"))
+
+
+def is_empty_entry(entry):
+ # type: (object) -> bool
+ """True for the canonical 'source==empty, both hips blank' sentinel
+ (and for malformed entries, which are treated as empty)."""
+ if not isinstance(entry, dict):
+ return True
+ if entry.get("source") != "empty":
+ return False
+ r, l = _hips(entry)
+ if r is None:
+ return True
+ return not r.get("raw") and not l.get("raw")
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Driver
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def correct_all(raw_path, clean_path, face_path, client, system_prompt=SYSTEM_PROMPT,
+ failed_list_path=None, max_tokens=512, max_retries=MAX_RETRIES,
+ timeout_seconds=15, still_empty_path=None, only_if_currently_empty=True):
+ # type: (...) -> Dict[str, Dict]
+ """Re-run LLM face-joint selection and write results back IN-PLACE to
+ ``face_path``.
+
+ Targets every ``source == "empty"`` rig in ``face_path``, or only the
+ ids in ``failed_list_path`` when given. Rigs where the LLM fails or
+ still returns an empty entry are left untouched.
+
+ A missing ``failed_list_path`` is a hard error: a typo must not silently
+ escalate a targeted repair into a full sweep.
+ """
+ with open(raw_path, "r") as f:
+ raw_data = json.load(f)
+ with open(clean_path, "r") as f:
+ clean_data = json.load(f)
+ with open(face_path, "r") as f:
+ face_data = json.load(f)
+
+ if failed_list_path:
+ if not os.path.exists(failed_list_path):
+ raise FileNotFoundError(
+ "--failed_list {!r} does not exist. Drop the flag to re-attempt "
+ "every source=='empty' rig in {}.".format(failed_list_path, face_path))
+ target_ids = read_id_list(failed_list_path)
+ scope_desc = "{} rigs from {}".format(len(target_ids), failed_list_path)
+ else:
+ target_ids = sorted(k for k, v in face_data.items() if is_empty_entry(v))
+ scope_desc = "{} empty-source rigs in {}".format(len(target_ids), face_path)
+
+ backup = face_path + ".bak"
+ if not os.path.exists(backup):
+ shutil.copy2(face_path, backup)
+ logger.info("Backed up {} -> {}".format(face_path, backup))
+
+ if still_empty_path is None:
+ still_empty_path = os.path.join(
+ os.path.dirname(face_path) or ".", "still_empty_face_joints.txt")
+
+ logger.info("Re-attempting {} (in-place update of {})".format(scope_desc, face_path))
+
+ updated = skipped_not_empty = missing = 0
+ still_empty = [] # type: List[str]
+ pbar = tqdm(target_ids, desc="Correcting", unit="rig", dynamic_ncols=True)
+ for rig_id in pbar:
+ pbar.set_postfix_str(rig_id, refresh=False)
+
+ if rig_id not in raw_data or rig_id not in clean_data:
+ missing += 1
+ logger.warning("[{}] missing from raw/clean; skipping".format(rig_id))
+ continue
+
+ # The failed list may be stale: only touch entries still empty now.
+ if only_if_currently_empty and not is_empty_entry(face_data.get(rig_id)):
+ skipped_not_empty += 1
+ continue
+
+ try:
+ entry, used_fallback = resolve_rig(
+ rig_id, raw_data[rig_id], clean_data[rig_id], client,
+ system_prompt=system_prompt, max_tokens=max_tokens,
+ max_retries=max_retries, timeout_seconds=timeout_seconds,
+ fallback_on_error=False,
+ )
+ except Exception as err: # noqa: BLE001 โ€” exhausted retries: keep the empty entry
+ logger.warning("[{}] LLM resolve failed: {}".format(rig_id, err))
+ entry, used_fallback = None, True
+
+ # Accept only clean non-empty LLM wins. used_fallback=True means the
+ # rule-based hint came back on timeout โ€” the same hint that already
+ # produced the empty entry.
+ if entry is not None and not used_fallback and is_non_empty_entry(entry):
+ face_data[rig_id] = entry
+ updated += 1
+ else:
+ still_empty.append(rig_id)
+
+ save_json(face_data, face_path)
+
+ # Append (like every other id log in this stage) rather than truncate: a
+ # --failed_list run only sees a subset, and a truncating write would
+ # replace the global record with just that subset.
+ known_empty = (set(read_id_list(still_empty_path))
+ if os.path.exists(still_empty_path) else set())
+ if still_empty:
+ new_ids = [rig_id for rig_id in still_empty if rig_id not in known_empty]
+ if new_ids:
+ append_id_list(still_empty_path, new_ids)
+ logger.info("{} rigs still empty ({} newly recorded) -> {}".format(
+ len(still_empty), len(new_ids), still_empty_path))
+ elif known_empty:
+ logger.warning(
+ "No rig came back empty in this run, but {} still lists {} id(s) "
+ "from earlier runs โ€” it may be stale.".format(
+ still_empty_path, len(known_empty)))
+
+ logger.info(
+ "Done. updated={} | still_empty={} | skipped_not_empty={} | "
+ "missing_raw_or_clean={} | total={}".format(
+ updated, len(still_empty), skipped_not_empty, missing, len(target_ids)))
+ return face_data
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# CLI
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Re-run LLM face-joint selection on rigs whose source is 'empty'; "
+ "updates face_joint_names.json in place when a pair is found.")
+ parser.add_argument("--raw", required=True, help="Raw joint_names.json.")
+ parser.add_argument("--cleaned", required=True, help="clean_joint_names.json.")
+ parser.add_argument("--face", required=True,
+ help="face_joint_names.json โ€” updated IN PLACE; a '.bak' sibling "
+ "is made on first run.")
+ parser.add_argument("--failed_list", default=None,
+ help="Optional txt file with one rig id per line. Default: every "
+ "rig whose entry in --face has source=='empty'.")
+ parser.add_argument("--still_empty_log", default=None,
+ help="Path to write rig ids the LLM still couldn't fill (default: "
+ "sibling 'still_empty_face_joints.txt').")
+ parser.add_argument("--force_reattempt", action="store_true",
+ help="Re-run even if the current entry isn't source=='empty' anymore.")
+ parser.add_argument("--rig_timeout", type=int, default=15,
+ help="Per-rig wall-time budget in seconds (SIGALRM). 0 disables.")
+ add_llm_args(parser, default_max_tokens=512)
+ args = parser.parse_args()
+
+ client = LLMClient.from_args(args)
+ logger.info("{} | Writing in-place to: {}".format(client, args.face))
+
+ correct_all(
+ raw_path=args.raw,
+ clean_path=args.cleaned,
+ face_path=args.face,
+ client=client,
+ failed_list_path=args.failed_list,
+ max_tokens=args.max_tokens,
+ max_retries=args.max_retries,
+ timeout_seconds=args.rig_timeout,
+ still_empty_path=args.still_empty_log,
+ only_if_currently_empty=not args.force_reattempt,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/joint_annotation/face_select_llm.py b/data_process/joint_annotation/face_select_llm.py
new file mode 100644
index 0000000..0d2b5d2
--- /dev/null
+++ b/data_process/joint_annotation/face_select_llm.py
@@ -0,0 +1,647 @@
+"""Pick the facing-direction joint pair for each rig with a Language Model.
+
+Complement to the rule-based :mod:`face_select_rule`: asks an LLM to choose ONE
+bilaterally symmetric joint pair (hip / shoulder / wing / fin / ...) that
+defines the rig's lateral axis for heading-angle computation. For serpentine
+rigs with no bilateral symmetry, the LLM picks longitudinal body-axis
+endpoints (head-tip / tail-tip) and sets ``body_axis: true``.
+
+Backends (see :mod:`llm`): local HF causal LM, OpenAI API (default:
+gpt-5-mini), DeepSeek API.
+
+Inputs :
+ ``joint_names.json`` โ€” ``{rig_id: [raw_joint_name, ...]}``
+ ``clean_joint_names.json`` โ€” ``{rig_id: [clean_joint_name, ...]}``
+Output :
+ ``face_joint_names.json`` โ€” ``{rig_id: {r_hip, l_hip, source, [body_axis]}}``
+
+Pipeline per rig:
+ 1. Pre-filter the joint list into four small buckets (Right / Left /
+ Head / Tail candidates). Dozens-of-joint rigs collapse to a handful.
+ 2. Short-circuit 'empty' if no candidates survive (no LLM call).
+ 3. Otherwise send only the compact candidate list to the LLM, together
+ with the rule-based pick as a hint, and ask it to apply a three-step
+ decision (bilateral -> body-axis -> empty).
+ 4. Validate the response (raw names must appear verbatim in the input);
+ retry with a structured correction on failure.
+ 5. Fall back to the rule-based resolver after max retries / timeout.
+
+Usage:
+ python -m data_process.joint_annotation.face_select_llm \\
+ --input_dir <export_dir> # OpenAI gpt-5-mini
+ python -m data_process.joint_annotation.face_select_llm \\
+ --input_dir <export_dir> --model deepseek-v4-flash
+ python -m data_process.joint_annotation.face_select_llm \\
+ --input_dir <export_dir> --model Qwen/Qwen3-8B
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+import time
+
+from loguru import logger
+from tqdm import tqdm
+
+_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+from data_process.joint_annotation.llm import ( # noqa: E402
+ FatalLLMError,
+ LLMClient,
+ MAX_RETRIES,
+ RigTimeout,
+ add_llm_args,
+ append_id_list,
+ install_rig_alarm,
+ load_json,
+ read_id_list,
+ save_json,
+ strip_llm_noise,
+ uninstall_rig_alarm,
+)
+from data_process.joint_annotation.face_select_rule import ( # noqa: E402
+ HEAD_KEYWORDS,
+ TAIL_KEYWORDS,
+ empty_entry,
+ resolve_face_joints,
+ strip_trailing_num,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# System prompt
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+SYSTEM_PROMPT = (
+ "# TASK\n"
+ "Pick the joint pair that defines one 3D rig's lateral facing axis. "
+ "Rigs come from Objaverse (Mixamo, 3ds Max Biped, Maya QuickRig, "
+ "Unreal mannequin, CAT, custom) and Truebones animals. All clean "
+ "names are already normalised to a canonical vocabulary โ€” reason on "
+ "CLEAN names; RAW names are only for verbatim copy-back.\n\n"
+ "# INPUT (user message)\n"
+ "Four pre-filtered buckets (any may be empty):\n"
+ " RIGHT : rows whose clean name starts with 'Right '\n"
+ " LEFT : rows whose clean name starts with 'Left '\n"
+ " HEAD : rows whose clean name (minus any trailing ' N') is one of\n"
+ " {Head, Skull, Skull Base, Head End, Jaw, Upper Jaw,\n"
+ " Lower Jaw, Tongue, Muzzle, Nose, Chin}\n"
+ " TAIL : rows whose clean name (minus any trailing ' N') is 'Tail'\n"
+ " or 'Tail Twist'\n"
+ "Plus a 'Rule-based hint' JSON object โ€” the deterministic resolver's "
+ "pick. Use it as a sanity check, not ground truth (see HINT below).\n\n"
+ "# ALGORITHM\n"
+ "Execute the steps in order. Do not skip ahead.\n"
+ "STEP 1. Compute OVERLAP.\n"
+ " - For each row in RIGHT, suffix = clean_name minus 'Right ' "
+ "minus any trailing ' <digits>'. Collect into set R_SUFFIXES.\n"
+ " - For each row in LEFT, suffix = clean_name minus 'Left ' "
+ "minus any trailing ' <digits>'. Collect into set L_SUFFIXES.\n"
+ " - OVERLAP = R_SUFFIXES โˆฉ L_SUFFIXES.\n"
+ "STEP 2. If OVERLAP is non-empty โ€” APPLY RULE A AND RETURN.\n"
+ " a. Pick suffix `s` = first element of OVERLAP that appears in "
+ "PRIORITY (see below). If none appear, pick the alphabetically "
+ "first element of OVERLAP.\n"
+ " b. R_ROWS = RIGHT rows whose suffix == s; L_ROWS = LEFT rows "
+ "whose suffix == s.\n"
+ " c. If len(R_ROWS)==1 and len(L_ROWS)==1, pair them.\n"
+ " Otherwise (chain duplicates, e.g. scorpion legs) match by "
+ "DIGIT SIGNATURE: the tuple of ALL digit groups in the RAW name "
+ "('Bip01_R_Thigh_4' -> (01,4)). Tier 1 โ€” pair rows whose full "
+ "signatures are equal. Tier 2 โ€” no tier-1 match: drop the LAST "
+ "group (the Objaverse per-joint global index: 'Bip01_R_Thigh_1_053' "
+ "and 'Bip01_L_Thigh_1_054' both reduce to (01,1)) and pair on the "
+ "rest. If neither tier matches, take R_ROWS[0] and L_ROWS[0].\n"
+ " d. Output: source = s.lower(); body_axis = false.\n"
+ " e. DO NOT consider HEAD or TAIL. Even if HEAD+TAIL look perfect, "
+ "rule A wins whenever OVERLAP is non-empty.\n"
+ "STEP 3. OVERLAP is empty. If HEAD is non-empty AND TAIL is non-"
+ "empty โ€” APPLY RULE B AND RETURN.\n"
+ " - r_hip = LAST HEAD row; l_hip = LAST TAIL row (chain endpoints).\n"
+ " - source = 'body_axis'; body_axis = true.\n"
+ "STEP 4. Otherwise โ€” APPLY RULE C (empty).\n"
+ " - Both hips are {raw: '', clean: ''}. source = 'empty'; "
+ "body_axis = false.\n"
+ " - Vehicles, props, and abstract rigs land here. NEVER pair an "
+ "unrelated Right/Left entry (e.g. lone 'Right Eye' without left "
+ "mate) just to avoid emitting empty.\n\n"
+ "# PRIORITY (highest โ†’ lowest; first match in OVERLAP wins)\n"
+ "Hip-level : thigh, shoulder, front shoulder, back hip, hip, "
+ "scapula\n"
+ "Whole-limb : upper arm, arm, front leg, hind leg, middle leg, "
+ "back leg, wing, leg\n"
+ "Aquatic : pectoral fin, pelvic fin, fin, gill\n"
+ "Arthropod : pincer, mandible, large mandible, lower mandible, "
+ "stinger, claw, hand claw, antenna\n"
+ "Mid-limb : forearm, shin, knee, elbow, ankle, wrist\n"
+ "Extremity : hand, palm, foot, heel, front paw, back paw, paw, "
+ "front hoof, rear hoof, hoof, fetlock, cannon, metacarpus, "
+ "pastern, toe\n"
+ "Fingers : thumb finger, index finger, middle finger, ring "
+ "finger, pinky finger, finger, neck\n"
+ "Weak : eye, eyeball, eyelid, eyebrow, ear, horn, cheek, "
+ "whisker, fang, barbel, tentacle, feather\n"
+ "Last : tail\n"
+ "Suffix comparison is case-insensitive but uses the clean-name form "
+ "('Upper Arm', 'Front Shoulder', 'Pectoral Fin'). Read the list above "
+ "LITERALLY, left to right, top to bottom โ€” it is exactly the order the "
+ "rule-based resolver uses, so do not re-rank quadruped markers "
+ "('Front Shoulder', 'Back Hip', 'Front Leg') against their plain "
+ "counterparts ('Shoulder', 'Hip', 'Leg'). Note in particular that "
+ "'shoulder' comes BEFORE 'front shoulder', while 'front leg' comes "
+ "BEFORE 'leg'.\n\n"
+ "# HINT\n"
+ "The hint is the rule-based resolver's output. It is usually right "
+ "but can err on edge cases:\n"
+ " - Hint says 'body_axis' or 'empty' yet OVERLAP is non-empty โ†’ "
+ "hint is wrong, apply STEP 2 (rule A wins).\n"
+ " - Hint swapped sides (Right/Left in wrong fields) โ†’ fix.\n"
+ " - Hint chose a lower-priority suffix than STEP 2a finds โ†’ "
+ "override with the higher-priority one.\n"
+ " - Hint paired mismatched chain indices on a multi-leg rig โ†’ fix "
+ "via STEP 2c raw-suffix match.\n"
+ " - Your algorithm yields the same answer as the hint โ†’ return "
+ "the hint verbatim.\n\n"
+ "# OUTPUT\n"
+ "Exactly one JSON object, no prose, no markdown fences, no comments:\n"
+ ' {"r_hip": {"raw": "<exact>", "clean": "<exact>"},\n'
+ ' "l_hip": {"raw": "<exact>", "clean": "<exact>"},\n'
+ ' "source": "<lowercase suffix or \'body_axis\' or \'empty\'>",\n'
+ ' "body_axis": <true|false>}\n\n'
+ "# INVARIANTS (verify before emitting)\n"
+ " I1. r_hip.raw != l_hip.raw, UNLESS both are '' (empty case).\n"
+ " I2. r_hip.raw and l_hip.raw each appear verbatim in the rig's "
+ "input rows (copied character-for-character).\n"
+ " I3. r_hip always holds the Right (or head) side; l_hip always "
+ "holds the Left (or tail) side. Never swap.\n"
+ " I4. body_axis == true IFF source == 'body_axis'.\n"
+ " I5. source is lowercase. If rule A fired, source is the suffix "
+ "lowercased with single spaces (e.g. 'thigh', 'front shoulder', "
+ "'pectoral fin').\n"
+ " I6. Unless body_axis, r_hip and l_hip mirror the SAME part: "
+ "their clean names minus the side prefix and any trailing digits "
+ "are identical. Never pair different parts ('Right Thigh' with "
+ "'Left Shoulder' is invalid even though the sides are correct).\n\n"
+ "# WORKED EXAMPLES\n"
+ "## Ex1 โ€” Mixamo humanoid (rule A, hip-level pick)\n"
+ "RIGHT has 'Right Thigh' + 'Right Shoulder'; LEFT has the mirrors. "
+ "OVERLAP = {Thigh, Shoulder}. Thigh outranks Shoulder โ†’ pick Thigh.\n"
+ ' {"r_hip": {"raw": "mixamorig:RightUpLeg_056", "clean": '
+ '"Right Thigh"},\n'
+ ' "l_hip": {"raw": "mixamorig:LeftUpLeg_056", "clean": '
+ '"Left Thigh"},\n'
+ ' "source": "thigh", "body_axis": false}\n'
+ "## Ex2 โ€” Objaverse quadruped (Front Shoulder beats Back Hip)\n"
+ "No Thigh pair; OVERLAP = {Front Shoulder, Back Hip}. Front "
+ "Shoulder outranks Back Hip.\n"
+ ' {"r_hip": {"raw": "F_R_Shoulder_012", "clean": '
+ '"Right Front Shoulder"},\n'
+ ' "l_hip": {"raw": "F_L_Shoulder_012", "clean": '
+ '"Left Front Shoulder"},\n'
+ ' "source": "front shoulder", "body_axis": false}\n'
+ "## Ex3 โ€” Scorpion multi-leg (STEP 2c raw-suffix match)\n"
+ "RIGHT has three 'Right Thigh' rows (Bip01_R_Thigh_1/_2/_4); LEFT "
+ "the mirrors. OVERLAP = {Thigh}. Digit signatures: (01,1) on both "
+ "sides -> tier-1 match.\n"
+ ' {"r_hip": {"raw": "Bip01_R_Thigh_1", "clean": "Right Thigh"},\n'
+ ' "l_hip": {"raw": "Bip01_L_Thigh_1", "clean": "Left Thigh"},\n'
+ ' "source": "thigh", "body_axis": false}\n'
+ "## Ex4 โ€” OVERRIDE a mistaken body_axis hint\n"
+ "RIGHT has 'Right Thigh'; LEFT has 'Left Thigh'; HEAD has 'Tongue'; "
+ "TAIL has 'Tail'. Hint = body_axis Tongue+Tail. OVERLAP = {Thigh} "
+ "is non-empty โ†’ STEP 2 wins, hint is wrong.\n"
+ ' {"r_hip": {"raw": "RightUpLeg_033", "clean": "Right Thigh"},\n'
+ ' "l_hip": {"raw": "LeftUpLeg_028", "clean": "Left Thigh"},\n'
+ ' "source": "thigh", "body_axis": false}\n'
+ "## Ex5 โ€” Snake / serpent (rule B fires)\n"
+ "RIGHT and LEFT are empty; HEAD has 'Head'; TAIL has 'Tail_30'. "
+ "STEP 3 applies.\n"
+ ' {"r_hip": {"raw": "Head", "clean": "Head"},\n'
+ ' "l_hip": {"raw": "Tail_30", "clean": "Tail"},\n'
+ ' "source": "body_axis", "body_axis": true}\n'
+ "## Ex6 โ€” Vehicle / prop (rule C fires)\n"
+ "All sections empty, or only a lone 'Right Eye' with no left mate.\n"
+ ' {"r_hip": {"raw": "", "clean": ""},\n'
+ ' "l_hip": {"raw": "", "clean": ""},\n'
+ ' "source": "empty", "body_axis": false}\n'
+)
+
+
+def prefilter_candidates(raw_names, clean_names):
+ # type: (List[str], List[str]) -> Dict[str, List[Dict]]
+ """Partition joints into LLM-relevant sections.
+
+ Returns a dict with four keys โ€” ``right``, ``left``, ``head``, ``tail``
+ โ€” each a list of ``{"raw": ..., "clean": ...}`` dicts preserving input
+ order. Most rigs have 60+ joints but only a handful are candidates for
+ the facing direction, so this drastically shortens the LLM's context.
+ """
+ buckets = {"right": [], "left": [], "head": [], "tail": []}
+ for raw, clean in zip(raw_names, clean_names):
+ if clean.startswith("Right "):
+ buckets["right"].append({"raw": raw, "clean": clean})
+ continue
+ if clean.startswith("Left "):
+ buckets["left"].append({"raw": raw, "clean": clean})
+ continue
+ base = strip_trailing_num(clean)
+ if base in HEAD_KEYWORDS:
+ buckets["head"].append({"raw": raw, "clean": clean})
+ if base in TAIL_KEYWORDS:
+ buckets["tail"].append({"raw": raw, "clean": clean})
+ return buckets
+
+
+def _format_section(name, rows):
+ # type: (str, List[Dict]) -> str
+ if not rows:
+ return "{}: (none)".format(name)
+ lines = ["{}:".format(name)]
+ for row in rows:
+ lines.append(" raw={!r:<40s} clean={!r}".format(row["raw"], row["clean"]))
+ return "\n".join(lines)
+
+
+def build_user_prompt(rig_id, candidates, rule_hint=None, correction=None):
+ # type: (str, Dict[str, List[Dict]], Optional[Dict], Optional[str]) -> str
+ sections = [
+ "Rig identifier: {}".format(rig_id),
+ "Candidate counts: right={} left={} head={} tail={}".format(
+ len(candidates["right"]), len(candidates["left"]),
+ len(candidates["head"]), len(candidates["tail"])),
+ "",
+ _format_section("RIGHT", candidates["right"]),
+ "",
+ _format_section("LEFT", candidates["left"]),
+ "",
+ _format_section("HEAD", candidates["head"]),
+ "",
+ _format_section("TAIL", candidates["tail"]),
+ "",
+ ]
+ if rule_hint is not None:
+ sections.extend([
+ "Rule-based hint (deterministic resolver's pick โ€” HINT, not ground truth):",
+ json.dumps(rule_hint, ensure_ascii=False, indent=2),
+ "",
+ ])
+ sections.append(
+ "Apply rules A -> B -> C and return the JSON object only. If the "
+ "hint already satisfies A/B/C, return it verbatim; otherwise override."
+ )
+ prompt = "\n".join(sections)
+ if correction:
+ prompt += "\n\nIMPORTANT: " + correction
+ return prompt
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Response parsing
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _hip_fields(hip, side):
+ # type: (object, str) -> Tuple[str, str]
+ if not isinstance(hip, dict):
+ raise ValueError("{} is not a JSON object".format(side))
+ raw = hip.get("raw", "")
+ clean = hip.get("clean", "")
+ if not isinstance(raw, str) or not isinstance(clean, str):
+ raise ValueError("{} raw/clean must be strings".format(side))
+ return raw, clean
+
+
+def _clean_side(clean_name):
+ # type: (str) -> Optional[str]
+ """``"Left"`` / ``"Right"`` if the clean label carries a side, else None."""
+ if clean_name.startswith("Left "):
+ return "Left"
+ if clean_name.startswith("Right "):
+ return "Right"
+ return None
+
+
+def _validate_sides(r_clean, l_clean, body_axis):
+ # type: (str, str, bool) -> None
+ """Enforce invariant I3: ``r_hip`` holds the Right (or head) side and
+ ``l_hip`` the Left (or tail) side.
+
+ The prompt states I3 but a model can still return a swapped pair, and a
+ swap is invisible downstream: ``get_root_facing_quat`` just builds the
+ lateral axis the other way round, silently rotating every clip of that
+ object type by 180 degrees. Raise ``ValueError`` so the caller retries
+ with a correction and, if that fails too, falls back to the rule-based
+ resolver.
+ """
+ if body_axis:
+ if (strip_trailing_num(r_clean) in TAIL_KEYWORDS
+ or strip_trailing_num(l_clean) in HEAD_KEYWORDS):
+ raise ValueError(
+ "body-axis pair is swapped: r_hip must be the HEAD end and "
+ "l_hip the TAIL end, got r_hip={!r} / l_hip={!r}".format(
+ r_clean, l_clean))
+ return
+ if _clean_side(r_clean) == "Left" or _clean_side(l_clean) == "Right":
+ raise ValueError(
+ "left/right pair is swapped: r_hip must be the Right-side joint "
+ "and l_hip the Left-side one, got r_hip={!r} / l_hip={!r}".format(
+ r_clean, l_clean))
+
+
+def parse_response(text, raw_names, clean_names):
+ # type: (str, List[str], List[str]) -> Dict
+ """Extract and validate a ``face_joint_names.json``-shaped object.
+
+ Tolerates markdown fences and ``<think>โ€ฆ</think>`` blocks. Raises
+ ``ValueError`` on structural mismatch so the caller can retry or fall
+ back. Validation rules:
+
+ - ``r_hip.raw`` / ``l_hip.raw`` must exist verbatim in ``raw_names``
+ (or both be ``""`` for the empty sentinel).
+ - ``r_hip.raw != l_hip.raw``.
+ - The clean name is re-sourced from ``clean_names`` so the LLM can't
+ drift the canonical label.
+ - The sides are not swapped (invariant I3, see :func:`_validate_sides`).
+ """
+ if not text or not text.strip():
+ raise ValueError("empty response")
+
+ cleaned = strip_llm_noise(text)
+ try:
+ data = json.loads(cleaned)
+ except json.JSONDecodeError:
+ start = cleaned.find("{")
+ end = cleaned.rfind("}")
+ if start == -1 or end == -1 or end <= start:
+ raise ValueError("no JSON object found in response")
+ data = json.loads(cleaned[start:end + 1])
+
+ if not isinstance(data, dict):
+ raise ValueError("response is not a JSON object")
+ for key in ("r_hip", "l_hip", "source"):
+ if key not in data:
+ raise ValueError("missing key '{}'".format(key))
+
+ r_raw, _ = _hip_fields(data["r_hip"], "r_hip")
+ l_raw, _ = _hip_fields(data["l_hip"], "l_hip")
+ source = str(data["source"]).strip().lower()
+ body_axis = bool(data.get("body_axis", False))
+
+ if not r_raw and not l_raw:
+ return empty_entry()
+ if not r_raw or not l_raw:
+ raise ValueError("one of r_hip/l_hip is empty; both must be filled "
+ "or both empty for 'empty' source")
+ if r_raw == l_raw:
+ raise ValueError("r_hip.raw == l_hip.raw ({!r}): pair must be two "
+ "distinct joints".format(r_raw))
+ if r_raw not in raw_names:
+ raise ValueError("r_hip.raw {!r} not in input raw joint list".format(r_raw))
+ if l_raw not in raw_names:
+ raise ValueError("l_hip.raw {!r} not in input raw joint list".format(l_raw))
+
+ r_idx = raw_names.index(r_raw)
+ l_idx = raw_names.index(l_raw)
+ is_axis = body_axis or source == "body_axis"
+ _validate_sides(clean_names[r_idx], clean_names[l_idx], is_axis)
+ if not is_axis:
+ # Invariant I6: a lateral pair must mirror the SAME part โ€” a
+ # cross-part pair ('Right Thigh' + 'Left Shoulder') passes the side
+ # check yet yields a skewed facing axis.
+ r_base = strip_trailing_num(
+ re.sub(r'^(Left|Right)\s+', '', clean_names[r_idx]))
+ l_base = strip_trailing_num(
+ re.sub(r'^(Left|Right)\s+', '', clean_names[l_idx]))
+ if r_base.lower() != l_base.lower():
+ raise ValueError(
+ "r_hip and l_hip are different parts ({!r} vs {!r}); the "
+ "pair must be the mirrored copies of one joint".format(
+ clean_names[r_idx], clean_names[l_idx]))
+ entry = {
+ "r_hip": {"raw": raw_names[r_idx], "clean": clean_names[r_idx]},
+ "l_hip": {"raw": raw_names[l_idx], "clean": clean_names[l_idx]},
+ "source": source or "pair",
+ }
+ if body_axis or source == "body_axis":
+ entry["body_axis"] = True
+ return entry
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-rig resolver (retry + rule-based hint + fallback + timeout)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def resolve_rig(rig_id, raw_names, clean_names, client, system_prompt=SYSTEM_PROMPT,
+ max_tokens=512, max_retries=MAX_RETRIES, timeout_seconds=15,
+ fallback_on_error=True):
+ # type: (...) -> Tuple[Dict, bool]
+ """Pick the face-direction joint pair for one rig.
+
+ Returns ``(entry, used_fallback)``. Rigs with no Right/Left or
+ Head/Tail candidates short-circuit to the empty entry without an LLM
+ call. On exhausted retries or timeout the rule-based pick is returned
+ (or ``RuntimeError`` is raised when ``fallback_on_error`` is False and
+ no timeout occurred).
+ """
+ if not raw_names:
+ return empty_entry(), False
+ if len(raw_names) != len(clean_names):
+ # clean_joint_names is indexed positionally against raw_names; a
+ # stale/short list silently pairs the wrong bones (same guard as the
+ # rule-based CLI). Abort instead of quietly emitting 'empty'.
+ raise ValueError(
+ "{!r}: {} cleaned names vs {} raw names โ€” clean_joint_names.json "
+ "is stale; re-run the joint-name cleaner.".format(
+ rig_id, len(clean_names), len(raw_names)))
+
+ candidates = prefilter_candidates(raw_names, clean_names)
+ has_bilateral = bool(candidates["right"]) and bool(candidates["left"])
+ has_body_axis = bool(candidates["head"]) and bool(candidates["tail"])
+ if not has_bilateral and not has_body_axis:
+ return empty_entry(), False
+
+ # Rule-based hint (cheap, deterministic); also the fallback result.
+ rule_hint = resolve_face_joints(clean_names, raw_names)
+
+ last_err = None
+ correction = None
+ timed_out = False
+
+ prev_handler = install_rig_alarm(timeout_seconds)
+ try:
+ for attempt in range(1, max_retries + 1):
+ try:
+ raw_text = client.generate(
+ system_prompt,
+ build_user_prompt(rig_id, candidates, rule_hint=rule_hint,
+ correction=correction),
+ max_tokens)
+ return parse_response(raw_text, raw_names, clean_names), False
+ except (RigTimeout, FatalLLMError):
+ raise
+ except Exception as err: # noqa: BLE001 โ€” retry on any backend/parse error
+ last_err = err
+ if isinstance(err, ValueError):
+ correction = (
+ "Your previous response failed validation ({}). Return "
+ "exactly one JSON object with keys r_hip, l_hip, source, "
+ "body_axis. r_hip and l_hip must be objects with 'raw' "
+ "and 'clean' string fields. 'raw' must be an exact copy "
+ "of one of the rig's raw joint names (or \"\" for the "
+ "'empty' source). r_hip.raw != l_hip.raw. r_hip MUST "
+ "be the Right-side joint (or, for body_axis, the HEAD "
+ "end) and l_hip the Left-side one (the TAIL end) โ€” "
+ "never swap them, and both must mirror the SAME part. "
+ "No prose, no code fences."
+ ).format(err)
+ logger.warning("[{}] attempt {}/{} failed: {}".format(
+ rig_id, attempt, max_retries, err))
+ if attempt < max_retries:
+ time.sleep(1.5)
+ except RigTimeout:
+ timed_out = True
+ finally:
+ uninstall_rig_alarm(prev_handler)
+
+ if not fallback_on_error and not timed_out:
+ raise RuntimeError("LLM face-joint resolution failed for {}: {}".format(
+ rig_id, last_err))
+
+ if timed_out:
+ logger.warning("[{}] exceeded {}s budget; using rule-based hint".format(
+ rig_id, timeout_seconds))
+ else:
+ logger.error("[{}] all {} attempts failed ({}); falling back to "
+ "rule-based resolver".format(rig_id, max_retries, last_err))
+ return rule_hint, True
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Batch driver
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _is_complete(entry):
+ # type: (Dict) -> bool
+ """A cached entry is complete if it's well-formed (even 'empty' counts)."""
+ return (isinstance(entry, dict)
+ and "r_hip" in entry and "l_hip" in entry and "source" in entry)
+
+
+def resolve_all(raw_path, clean_path, output_path, client, system_prompt=SYSTEM_PROMPT,
+ max_tokens=512, max_retries=MAX_RETRIES, timeout_seconds=15,
+ fallback_on_error=True, failed_path=None):
+ # type: (...) -> Dict[str, Dict]
+ """Resolve face joints for every rig in ``raw_path`` / ``clean_path``.
+
+ Resumes from a partial output file. Rig ids that fall back to the
+ rule-based resolver are appended to ``failed_path`` (one per line).
+ """
+ with open(raw_path, "r") as f:
+ raw_data = json.load(f)
+ with open(clean_path, "r") as f:
+ clean_data = json.load(f)
+
+ result = load_json(output_path)
+ rig_ids = sorted(set(raw_data.keys()) & set(clean_data.keys()))
+ missing = set(raw_data) ^ set(clean_data)
+ if missing:
+ logger.warning("{} rig ids are not in both raw+clean; skipping: {}"
+ .format(len(missing), sorted(missing)[:5]))
+
+ if failed_path is None:
+ failed_path = os.path.join(
+ os.path.dirname(output_path) or ".", "failed_face_joints.txt")
+ already_failed = set(read_id_list(failed_path)) if os.path.exists(failed_path) else set()
+
+ fallback_count = 0
+ pbar = tqdm(rig_ids, desc="Resolving", unit="rig", dynamic_ncols=True)
+ for rig_id in pbar:
+ pbar.set_postfix_str(rig_id, refresh=False)
+ if _is_complete(result.get(rig_id)):
+ continue
+
+ entry, used_fallback = resolve_rig(
+ rig_id, raw_data[rig_id], clean_data[rig_id], client,
+ system_prompt=system_prompt, max_tokens=max_tokens,
+ max_retries=max_retries, timeout_seconds=timeout_seconds,
+ fallback_on_error=fallback_on_error,
+ )
+ fallback_count += int(used_fallback)
+ result[rig_id] = entry
+ save_json(result, output_path)
+
+ if used_fallback and rig_id not in already_failed:
+ append_id_list(failed_path, [rig_id])
+ already_failed.add(rig_id)
+
+ logger.info("Done. {} rigs resolved ({} via rule-based fallback) -> {}"
+ .format(len(result), fallback_count, output_path))
+ if fallback_count:
+ logger.info("Failed rig ids recorded at {}".format(failed_path))
+ return result
+
+
+def print_source_summary(result):
+ # type: (Dict[str, Dict]) -> None
+ sources = {}
+ for rig_id, entry in result.items():
+ sources.setdefault(entry["source"], []).append(rig_id)
+ print("\nResolved {} rigs. Breakdown:".format(len(result)))
+ for src, rigs in sorted(sources.items()):
+ print(" [{}] {} rigs".format(src, len(rigs)))
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# CLI
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Pick facing-direction joint pairs via an LLM (local / openai / deepseek).")
+ parser.add_argument("--input_dir", type=str, required=True,
+ help="Directory containing joint_names.json and clean_joint_names.json.")
+ parser.add_argument("--raw_name", type=str, default="joint_names.json",
+ help="Filename of raw joint names inside --input_dir.")
+ parser.add_argument("--clean_name", type=str, default="clean_joint_names.json",
+ help="Filename of cleaned joint names inside --input_dir.")
+ parser.add_argument("--output_name", type=str, default="face_joint_names.json",
+ help="Output filename written inside --input_dir (ignored if "
+ "--output is set).")
+ parser.add_argument("--output", type=str, default=None,
+ help="Explicit output path (default: <input_dir>/<output_name>).")
+ parser.add_argument("--no_fallback", action="store_true",
+ help="Raise on LLM failure instead of falling back to the "
+ "rule-based resolver.")
+ parser.add_argument("--failed_log", default=None,
+ help="Path to append rig ids that hit the rule-based fallback "
+ "(default: sibling 'failed_face_joints.txt').")
+ parser.add_argument("--rig_timeout", type=int, default=15,
+ help="Per-rig wall-time budget in seconds (SIGALRM). 0 disables.")
+ add_llm_args(parser, default_max_tokens=512)
+ args = parser.parse_args()
+
+ raw_path = os.path.join(args.input_dir, args.raw_name)
+ clean_path = os.path.join(args.input_dir, args.clean_name)
+ output_path = args.output or os.path.join(args.input_dir, args.output_name)
+ client = LLMClient.from_args(args)
+ logger.info("{} | Dir: {} | Output: {}".format(client, args.input_dir, output_path))
+
+ result = resolve_all(
+ raw_path=raw_path,
+ clean_path=clean_path,
+ output_path=output_path,
+ client=client,
+ max_tokens=args.max_tokens,
+ max_retries=args.max_retries,
+ timeout_seconds=args.rig_timeout,
+ fallback_on_error=not args.no_fallback,
+ failed_path=args.failed_log,
+ )
+ print_source_summary(result)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/joint_annotation/face_select_rule.py b/data_process/joint_annotation/face_select_rule.py
new file mode 100644
index 0000000..dc0ac9d
--- /dev/null
+++ b/data_process/joint_annotation/face_select_rule.py
@@ -0,0 +1,372 @@
+"""Rule-based facing-direction joint selection โ†’ ``face_joint_names.json``.
+
+For each rig this script identifies one symmetric joint pair that defines
+the lateral (left-right) axis used to compute the facing direction:
+
+ 1. Search clean joint names for any symmetrical "Right <X>" / "Left <X>"
+ pair, preferring anatomically meaningful joints (Thigh, Shoulder, ...).
+ 2. For body-axis animals (snakes like Anaconda) that lack bilateral
+ symmetry, use longitudinal body-axis endpoints (tail-tip / head-tip)
+ and flag ``"body_axis": true``. The tail-to-head vector is directly
+ the facing direction (no cross product with Y-up needed).
+ 3. Animals with opaque joint names where no meaningful pair can be
+ extracted get ``"source": "empty"``.
+
+:mod:`face_select_llm` is the LLM-based complement and uses this
+module both as a hint and as its fallback.
+
+Usage:
+ python -m data_process.joint_annotation.face_select_rule \
+ --input_dir <export_dir> [--output PATH]
+"""
+
+import argparse
+import copy
+import json
+import os
+import re
+from typing import Dict, List, Optional, Tuple
+
+# Clean-name keywords for body-axis endpoint detection (serpentine skeletons
+# and any rig with no bilateral symmetry but a clear longitudinal axis).
+TAIL_KEYWORDS = {"Tail", "Tail Twist"}
+HEAD_KEYWORDS = {"Head", "Skull", "Skull Base", "Head End", "Jaw",
+ "Upper Jaw", "Lower Jaw", "Tongue", "Muzzle", "Nose", "Chin"}
+
+# Priority order for symmetrical pair search (suffix after stripping the
+# "Right "/"Left " prefix). Higher entries win.
+#
+# Ordering rationale (top โ†’ bottom):
+# 1. Hip-level limbs closest to the rig root give the most stable facing
+# axis (Thigh, Shoulder, and the quadruped "Front Shoulder"/"Back Hip"
+# markers Objaverse quadrupeds use after names_clean_rule.py).
+# 2. Whole-limb roots (Upper Arm, Front Leg, Hind Leg, Wing, ...) come
+# next โ€” they're unique per side and rarely ambiguous.
+# 3. Aquatic / arthropod homologues (fins, pincers, mandibles) appear
+# where they're anatomically equivalent to the limbs above.
+# 4. Mid-limb joints (Forearm, Shin, Knee, Elbow, ...) are good fallbacks
+# but drift a few cm from the true hip axis.
+# 5. Extremities (Hand, Foot, Paw, Hoof, Claw) are weakest โ€” they bend a
+# lot during motion โ€” so they sit near the bottom.
+# 6. Head-adjacent bilateral parts (Eye, Ear, Whisker, Antenna) are last
+# resorts; they give a cross-head axis, not a hip axis.
+# 7. Tail is the absolute bottom (often a chain; the 'body_axis' fallback
+# uses it as a longitudinal endpoint instead).
+SYMMETRIC_PAIR_PRIORITY = [
+ # --- Hip / shoulder level (best) -----------------------------------
+ "Thigh",
+ "Shoulder",
+ "Front Shoulder", # quadruped pec (Objaverse)
+ "Back Hip", # quadruped pelvis (Objaverse)
+ "Hip",
+ "Scapula",
+ # --- Whole-limb roots ----------------------------------------------
+ "Upper Arm",
+ "Arm",
+ "Front Leg",
+ "Hind Leg",
+ "Middle Leg", # insect / arachnid
+ "Back Leg",
+ "Wing",
+ "Leg",
+ # --- Aquatic / arthropod homologues --------------------------------
+ "Pectoral Fin",
+ "Pelvic Fin",
+ "Fin", # generic fish fin (L_Fin1-style rigs)
+ "Gill",
+ "Pincer",
+ "Mandible",
+ "Large Mandible",
+ "Lower Mandible",
+ "Stinger",
+ "Claw",
+ "Hand Claw",
+ "Antenna",
+ # --- Mid-limb joints -----------------------------------------------
+ "Forearm",
+ "Shin",
+ "Knee",
+ "Elbow",
+ "Ankle",
+ "Wrist",
+ # --- Extremities ---------------------------------------------------
+ "Hand",
+ "Palm",
+ "Foot",
+ "Heel",
+ "Front Paw",
+ "Back Paw",
+ "Paw",
+ "Front Hoof",
+ "Rear Hoof",
+ "Hoof",
+ "Fetlock",
+ "Cannon",
+ "Metacarpus",
+ "Pastern",
+ "Toe",
+ # --- Fingers (hand-to-hand axis; both hands move, so below feet) ---
+ "Thumb Finger",
+ "Index Finger",
+ "Middle Finger",
+ "Ring Finger",
+ "Pinky Finger",
+ "Finger",
+ # --- Coat / soft ---------------------------------------------------
+ "Neck",
+ # --- Head-adjacent bilateral parts (weak) --------------------------
+ "Eye",
+ "Eyeball",
+ "Eyelid",
+ "Eyebrow",
+ "Ear",
+ "Horn",
+ "Cheek",
+ "Whisker",
+ "Fang",
+ "Barbel",
+ "Tentacle",
+ "Feather",
+ # --- Last resort ---------------------------------------------------
+ "Tail",
+]
+
+
+def strip_trailing_num(name: str) -> str:
+ """'Thigh 1' -> 'Thigh'."""
+ return re.sub(r"\s+\d+$", "", name)
+
+
+def _digit_runs(raw_name: str) -> Tuple[str, ...]:
+ """All digit groups in a raw name: 'Bip01_R_Thigh_4' -> ('01', '4')."""
+ return tuple(re.findall(r"\d+", raw_name))
+
+
+def _match_pair_by_suffix(
+ r_indices: List[int],
+ l_indices: List[int],
+ raw_names: List[str],
+) -> Tuple[int, int]:
+ """Pair one Right index with one Left index by matching raw-name digits.
+
+ When multiple joints share the same clean name (e.g. multiple "Right
+ Thigh" in Scorpion), pairs are matched by the digit signature of the raw
+ name so that 'Bip01_R_Thigh_4' pairs with 'Bip01_L_Thigh_4', not
+ 'Bip01_L_Thigh_1'. Two tiers:
+
+ 1. Full digit signature โ€” rigs whose mirrored bones share every index.
+ 2. Signature minus the LAST group โ€” the Objaverse export stamps a
+ per-joint global index on every name ('Bip01_R_Thigh_1_053' vs
+ 'Bip01_L_Thigh_1_054'), so the full signatures never match; dropping
+ the final group compares only the chain indices.
+
+ Falls back to the first of each side when neither tier matches.
+ """
+ if len(r_indices) == 1 and len(l_indices) == 1:
+ return r_indices[0], l_indices[0]
+
+ for key_fn in (lambda runs: runs, lambda runs: runs[:-1]):
+ l_by_sig: Dict[Tuple[str, ...], int] = {}
+ for li in l_indices:
+ l_by_sig.setdefault(key_fn(_digit_runs(raw_names[li])), li)
+ for ri in r_indices:
+ li = l_by_sig.get(key_fn(_digit_runs(raw_names[ri])))
+ if li is not None:
+ return ri, li
+
+ return r_indices[0], l_indices[0]
+
+
+def find_symmetric_pair(
+ clean_names: List[str],
+ raw_names: List[str],
+) -> Tuple[Optional[int], Optional[int], Optional[str]]:
+ """Find the highest-priority "Right <X>" / "Left <X>" pair.
+
+ Returns ``(r_idx, l_idx, suffix)`` or ``(None, None, None)``.
+ """
+ right_map: Dict[str, List[int]] = {}
+ left_map: Dict[str, List[int]] = {}
+ for i, name in enumerate(clean_names):
+ if name.startswith("Right "):
+ right_map.setdefault(strip_trailing_num(name[6:]), []).append(i)
+ elif name.startswith("Left "):
+ left_map.setdefault(strip_trailing_num(name[5:]), []).append(i)
+
+ common = set(right_map) & set(left_map)
+ if not common:
+ return None, None, None
+
+ ordered = [s for s in SYMMETRIC_PAIR_PRIORITY if s in common]
+ ordered += sorted(common - set(SYMMETRIC_PAIR_PRIORITY))
+ suffix = ordered[0]
+ ri, li = _match_pair_by_suffix(right_map[suffix], left_map[suffix], raw_names)
+ return ri, li, suffix
+
+
+def find_body_axis_pair(
+ clean_names: List[str],
+) -> Tuple[Optional[int], Optional[int]]:
+ """Find body-axis endpoints (tail-tip and head-end) from clean names.
+
+ For serpentine animals without bilateral symmetry, facing direction is
+ the longitudinal axis from tail-tip to head-end. Uses the last occurrence
+ of each keyword set to land on the extremities.
+ """
+ tail_idx: Optional[int] = None
+ head_idx: Optional[int] = None
+ for i, name in enumerate(clean_names):
+ base = strip_trailing_num(name)
+ if base in TAIL_KEYWORDS:
+ tail_idx = i
+ if base in HEAD_KEYWORDS:
+ head_idx = i
+ if tail_idx is not None and head_idx is not None:
+ return tail_idx, head_idx
+ return None, None
+
+
+# Sentinel for "no facing pair could be determined". Always hand out a
+# ``copy.deepcopy`` of it โ€” a shallow ``dict()`` copy shares the r_hip/l_hip
+# sub-dicts with this module-level constant, so one in-place mutation of a
+# returned entry would poison every other 'empty' entry.
+EMPTY_ENTRY = {
+ "r_hip": {"raw": "", "clean": ""},
+ "l_hip": {"raw": "", "clean": ""},
+ "source": "empty",
+}
+
+
+def empty_entry() -> Dict:
+ """A fresh, independent copy of :data:`EMPTY_ENTRY`."""
+ return copy.deepcopy(EMPTY_ENTRY)
+
+
+def _make_entry(
+ raw_names: List[str],
+ clean_names: List[str],
+ r_idx: int,
+ l_idx: int,
+ source: str,
+) -> Dict:
+ return {
+ "r_hip": {"raw": raw_names[r_idx], "clean": clean_names[r_idx]},
+ "l_hip": {"raw": raw_names[l_idx], "clean": clean_names[l_idx]},
+ "source": source,
+ }
+
+
+def resolve_face_joints(
+ clean_names: List[str],
+ raw_names: List[str],
+) -> Dict:
+ """Resolve one joint pair defining the facing direction."""
+ r_idx, l_idx, suffix = find_symmetric_pair(clean_names, raw_names)
+ if r_idx is not None:
+ return _make_entry(raw_names, clean_names, r_idx, l_idx, suffix.lower())
+
+ # r_hip = head-end (facing toward), l_hip = tail-tip (facing away from).
+ tail_idx, head_idx = find_body_axis_pair(clean_names)
+ if tail_idx is not None:
+ entry = _make_entry(raw_names, clean_names, head_idx, tail_idx, "body_axis")
+ entry["body_axis"] = True
+ return entry
+
+ return empty_entry()
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ parser.add_argument(
+ "--input_dir", type=str, nargs="+", required=True,
+ help="Directory(ies) containing joint_names.json and clean_joint_names.json",
+ )
+ parser.add_argument(
+ "--raw_name", type=str, default="joint_names.json",
+ help="Filename of raw joint names inside each --input_dir",
+ )
+ parser.add_argument(
+ "--clean_name", type=str, default="clean_joint_names.json",
+ help="Filename of cleaned joint names inside each --input_dir",
+ )
+ parser.add_argument(
+ "--output_name", type=str, default="face_joint_names.json",
+ help="Output filename written inside each --input_dir (ignored if --output is set)",
+ )
+ parser.add_argument(
+ "--output", type=str, default=None,
+ help="Explicit output path (default: <input_dir>/<output_name> per directory)",
+ )
+ args = parser.parse_args()
+
+ # A single --output cannot hold the results of several --input_dir: only
+ # the last directory would survive. Make that an error instead.
+ if args.output and len(args.input_dir) > 1:
+ parser.error(
+ "--output takes a single path but {} --input_dir were given; drop "
+ "--output (use --output_name, written inside each directory) or "
+ "pass one directory at a time.".format(len(args.input_dir)))
+
+ for input_dir in args.input_dir:
+ raw_path = os.path.join(input_dir, args.raw_name)
+ clean_path = os.path.join(input_dir, args.clean_name)
+ with open(raw_path) as f:
+ raw_data = json.load(f)
+ with open(clean_path) as f:
+ clean_data = json.load(f)
+
+ # Drive the loop off the RAW key set: it is the authority on which
+ # object types exist, and stage 4 raises when one has no face entry.
+ extra = sorted(set(clean_data) - set(raw_data))
+ if extra:
+ print("WARNING: {} keys in {} are absent from {} and are ignored: "
+ "{}".format(len(extra), clean_path, raw_path, extra[:5]))
+
+ result = {}
+ for animal in raw_data:
+ raw_names = raw_data[animal]
+ if animal not in clean_data:
+ raise KeyError(
+ "{!r} is in {} but missing from {}; re-run the joint-name "
+ "cleaner so every object type has cleaned names."
+ .format(animal, raw_path, clean_path))
+ clean_names = clean_data[animal]
+ # clean_joint_names is indexed POSITIONALLY against raw_names, so a
+ # stale/short list would silently emit an r_hip.raw taken from the
+ # wrong position โ€” a real bone name that passes stage 4's
+ # membership check and yields a wrong facing axis.
+ if len(clean_names) != len(raw_names):
+ raise ValueError(
+ "{!r}: {} has {} cleaned names but {} has {} raw names; "
+ "{} is stale โ€” re-run the joint-name cleaner."
+ .format(animal, clean_path, len(clean_names),
+ raw_path, len(raw_names), clean_path))
+ result[animal] = resolve_face_joints(clean_names, raw_names)
+
+ output_path = args.output or os.path.join(input_dir, args.output_name)
+ with open(output_path, "w") as f:
+ json.dump(result, f, indent=2)
+
+ sources: Dict[str, List[str]] = {}
+ for animal, entry in result.items():
+ sources.setdefault(entry["source"], []).append(animal)
+
+ print("Written {} entries to {}\n".format(len(result), output_path))
+ for src, animals in sorted(sources.items()):
+ print(" [{}] {} entries".format(src, len(animals)))
+ for a in animals:
+ e = result[a]
+ r_clean = e["r_hip"]["clean"] or "(none)"
+ l_clean = e["l_hip"]["clean"] or "(none)"
+ r_raw = e["r_hip"]["raw"] or "(none)"
+ l_raw = e["l_hip"]["raw"] or "(none)"
+ body = " [BODY_AXIS]" if e.get("body_axis") else ""
+ print(" {:20s} r={:25s} ({}){}".format(a, r_clean, r_raw, body))
+ print(" {:20s} l={:25s} ({})".format("", l_clean, l_raw))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/joint_annotation/llm.py b/data_process/joint_annotation/llm.py
new file mode 100644
index 0000000..ff9d034
--- /dev/null
+++ b/data_process/joint_annotation/llm.py
@@ -0,0 +1,396 @@
+"""Shared LLM scaffolding for the ``joint_annotation/*_llm.py`` entry points.
+
+Every LLM-assisted joint-annotation script (joint-name cleaning, name
+correction, face-joint selection) needs the same plumbing: backend detection
+from the model name, a local HuggingFace causal-LM loader, one generate
+function per backend (local / OpenAI-compatible), response-noise stripping, a
+per-rig SIGALRM timeout, and simple JSON persistence. :class:`LLMClient`
+bundles the backend state so task scripts only own their prompts, parsers
+and per-rig orchestration.
+"""
+
+import json
+import os
+import re
+import signal
+import tempfile
+
+from loguru import logger
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Backend detection
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+BACKEND_LOCAL = "local"
+BACKEND_OPENAI = "openai"
+VALID_BACKENDS = [BACKEND_LOCAL, BACKEND_OPENAI]
+
+MAX_RETRIES = 4
+# gpt-5-mini is the alternative once OPENAI credits are topped up; DeepSeek
+# think-low reached quality parity (0 side errors, 99.6% format compliance,
+# A/B 2026-08-30) at a fraction of the cost.
+DEFAULT_MODEL = "deepseek-v4-flash"
+
+# DeepSeek's API speaks the OpenAI protocol, so ``deepseek-*`` models ride the
+# OpenAI backend with this base URL and DEEPSEEK_API_KEY (see generate_openai).
+DEEPSEEK_BASE_URL = "https://api.deepseek.com"
+
+
+def detect_backend(model_name):
+ # type: (str) -> str
+ """Infer the backend from a model name (``gpt-*`` / ``deepseek-*`` /
+ HF id)."""
+ if model_name.startswith(("gpt-", "o1", "o3", "o4", "deepseek")):
+ return BACKEND_OPENAI
+ return BACKEND_LOCAL
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Local LLM loading & generation
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def load_local_model(model_name, device_map="auto", torch_dtype=None):
+ """Load a text-only causal LM + tokenizer from HuggingFace."""
+ import torch
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ dtype = torch.float16 if torch_dtype == "float16" else torch.bfloat16
+
+ logger.info("Loading LLM: {} (dtype={}, device_map={})".format(
+ model_name, dtype, device_map))
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
+ model = AutoModelForCausalLM.from_pretrained(
+ model_name,
+ torch_dtype=dtype,
+ device_map=device_map,
+ )
+ model.eval()
+ return model, tokenizer
+
+
+def generate_local(model, tokenizer, system_prompt, user_prompt, max_tokens):
+ """Run a single chat completion through a local causal LM.
+
+ Qwen3 and other "thinking" models emit long ``<think>...</think>`` blocks
+ by default, which easily exhausts the token budget before the answer is
+ produced; ``enable_thinking=False`` is passed when the chat template
+ supports it. Sampling follows the Qwen3 model-card recommendation for
+ non-thinking mode (temperature=0.7, top_p=0.8, top_k=20) โ€” greedy
+ decoding is documented to cause endless repetitions.
+ """
+ import torch
+
+ messages = [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": user_prompt},
+ ]
+ try:
+ text = tokenizer.apply_chat_template(
+ messages, tokenize=False, add_generation_prompt=True,
+ enable_thinking=False,
+ )
+ except TypeError:
+ text = tokenizer.apply_chat_template(
+ messages, tokenize=False, add_generation_prompt=True,
+ )
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
+
+ with torch.no_grad():
+ generated = model.generate(
+ **inputs,
+ max_new_tokens=max_tokens,
+ do_sample=True,
+ temperature=0.7,
+ top_p=0.8,
+ top_k=20,
+ pad_token_id=tokenizer.eos_token_id,
+ )
+ trimmed = generated[0, inputs["input_ids"].shape[1]:]
+ return tokenizer.decode(trimmed, skip_special_tokens=True).strip()
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# API backends
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def generate_openai(system_prompt, user_prompt, model_name, api_key, base_url,
+ max_tokens, reasoning_effort=None):
+ from openai import OpenAI
+
+ is_deepseek = model_name.startswith("deepseek")
+ if api_key is None:
+ key_var = "DEEPSEEK_API_KEY" if is_deepseek else "OPENAI_API_KEY"
+ api_key = os.environ.get(key_var)
+ if api_key is None:
+ # Fail loudly: with api_key=None the OpenAI client silently falls
+ # back to OPENAI_API_KEY, which sends the wrong vendor's key to a
+ # DeepSeek endpoint (and produces a confusing 401).
+ raise RuntimeError(
+ "{} is not set (and no --api_key given) for model {!r}.".format(
+ key_var, model_name))
+ if base_url is None and is_deepseek:
+ base_url = DEEPSEEK_BASE_URL
+ client = OpenAI(api_key=api_key, base_url=base_url)
+ messages = [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": user_prompt},
+ ]
+ kwargs = dict(model=model_name, messages=messages)
+ is_gpt5 = model_name.startswith("gpt-5") and "chat" not in model_name
+ is_o_series = model_name.startswith(("o1", "o3", "o4"))
+ if is_gpt5 or is_o_series:
+ # Reasoning models reject max_tokens and non-default temperature;
+ # max_completion_tokens counts reasoning + visible output together.
+ kwargs["max_completion_tokens"] = max_tokens
+ if is_gpt5:
+ # The default effort burns hidden thinking tokens on every call
+ # and the joint-annotation tasks are classification, not
+ # multi-step reasoning โ€” force the lowest effort. The original
+ # gpt-5 family calls it "minimal"; gpt-5.1+ dropped "minimal"
+ # for "none" and errors on the old value. o-series models keep
+ # their default effort ("minimal"/"none" don't exist there, and
+ # o1-mini rejects the parameter entirely).
+ versioned = re.match(r"gpt-5\.\d", model_name)
+ kwargs["reasoning_effort"] = reasoning_effort or (
+ "none" if versioned else "minimal")
+ elif is_deepseek:
+ # DeepSeek V4 thinks by default (effort 'high') and the reasoning
+ # tokens count against max_tokens. Effort 'none' disables thinking โ€”
+ # but V4-Flash WITHOUT thinking drifts on long batched arrays
+ # (systematic label/side misalignment on ~10% of truebones rigs,
+ # A/B'd 2026-08-30), so default to the cheapest thinking tier.
+ # Thinking mode ignores temperature.
+ kwargs["reasoning_effort"] = reasoning_effort or "low"
+ if kwargs["reasoning_effort"] == "none":
+ kwargs["max_tokens"] = max_tokens
+ kwargs["temperature"] = 0.0
+ else:
+ # Reasoning shares the max_tokens budget; a small task budget
+ # (512 for face selection) gets eaten by thought and returns
+ # EMPTY content with finish_reason='length'. Billing is per
+ # token used, so a generous floor costs nothing extra.
+ kwargs["max_tokens"] = max(max_tokens, 8192)
+ else:
+ kwargs["max_tokens"] = max_tokens
+ kwargs["temperature"] = 0.0
+ try:
+ response = client.chat.completions.create(**kwargs)
+ except Exception as err:
+ msg = str(err)
+ if any(marker in msg for marker in (
+ "insufficient_quota", "credit_balance_exhausted",
+ "invalid_api_key", "Authentication Fails",
+ "account_deactivated")):
+ raise FatalLLMError(msg) from err
+ raise
+ choice = response.choices[0]
+ content = (choice.message.content or "").strip()
+ if not content and choice.finish_reason == "length":
+ # Thinking consumed the whole budget โ€” retrying with identical
+ # parameters cannot succeed; surface an actionable error instead.
+ raise RuntimeError(
+ "empty content with finish_reason='length': the reasoning "
+ "budget is exhausted โ€” raise --max_tokens or lower "
+ "--reasoning_effort (model {}).".format(model_name))
+ return content
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Unified client
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+class LLMClient:
+ """One text-in / text-out generate() over a local or OpenAI-compatible model.
+
+ Args:
+ model_name: OpenAI model (``gpt-5-mini``), DeepSeek model
+ (``deepseek-v4-flash``) or HuggingFace model id for the local
+ backend (``Qwen/Qwen3-8B``).
+ backend: One of :data:`VALID_BACKENDS`; auto-detected when ``None``.
+ api_key: API key for the OpenAI-compatible backends (defaults to
+ the env vars).
+ base_url: OpenAI-compatible base URL override.
+ device_map, torch_dtype: Local-backend loading knobs.
+ """
+
+ def __init__(self, model_name, backend=None, api_key=None, base_url=None,
+ device_map="auto", torch_dtype=None, reasoning_effort=None):
+ self.model_name = model_name
+ self.backend = backend or detect_backend(model_name)
+ if self.backend not in VALID_BACKENDS:
+ raise ValueError("Unknown backend {!r}; expected one of {}".format(
+ self.backend, VALID_BACKENDS))
+ self.api_key = api_key
+ self.base_url = base_url
+ self.reasoning_effort = reasoning_effort
+ self._model = self._tokenizer = None
+ if self.backend == BACKEND_LOCAL:
+ self._model, self._tokenizer = load_local_model(
+ model_name, device_map=device_map, torch_dtype=torch_dtype)
+
+ @classmethod
+ def from_args(cls, args):
+ """Build a client from the namespace produced by :func:`add_llm_args`."""
+ return cls(
+ args.model, backend=args.backend,
+ api_key=args.api_key, base_url=args.base_url,
+ device_map=args.device_map, torch_dtype=args.torch_dtype,
+ reasoning_effort=getattr(args, "reasoning_effort", None),
+ )
+
+ def generate(self, system_prompt, user_prompt, max_tokens):
+ # type: (str, str, int) -> str
+ if self.backend == BACKEND_LOCAL:
+ return generate_local(self._model, self._tokenizer,
+ system_prompt, user_prompt, max_tokens)
+ return generate_openai(system_prompt, user_prompt,
+ self.model_name, self.api_key, self.base_url,
+ max_tokens, reasoning_effort=self.reasoning_effort)
+
+ def __repr__(self):
+ return "LLMClient(backend={!r}, model={!r})".format(self.backend, self.model_name)
+
+
+def add_llm_args(parser, default_model=DEFAULT_MODEL, default_max_tokens=2048):
+ """Add the backend / model / retry flags shared by every ``*_llm.py`` CLI."""
+ group = parser.add_argument_group("LLM backend")
+ group.add_argument("--backend", type=str, default=None, choices=VALID_BACKENDS,
+ help="LLM backend. Auto-detected from --model if unset.")
+ group.add_argument("--model", type=str, default=default_model,
+ help="OpenAI model (e.g. gpt-5-mini), DeepSeek model (e.g. "
+ "deepseek-v4-flash), or HF model id for the local "
+ "backend (e.g. Qwen/Qwen3-8B).")
+ group.add_argument("--max_tokens", type=int, default=default_max_tokens,
+ help="Max new tokens per LLM call.")
+ group.add_argument("--max_retries", type=int, default=MAX_RETRIES,
+ help="Max LLM retries per rig before falling back.")
+ group.add_argument("--torch_dtype", type=str, default=None, choices=["float16", "bfloat16"],
+ help="[local] Model dtype override (default: bfloat16).")
+ group.add_argument("--device_map", type=str, default="auto",
+ help="[local] Device placement strategy (default: auto).")
+ group.add_argument("--api_key", type=str, default=None,
+ help="[openai] Defaults to OPENAI_API_KEY / DEEPSEEK_API_KEY, "
+ "matched to the model.")
+ group.add_argument("--base_url", type=str, default=None,
+ help="[openai] Override API base URL (deepseek-* models default "
+ "to the DeepSeek endpoint; use with --backend openai for "
+ "any other OpenAI-compatible server, e.g. a local vLLM).")
+ group.add_argument("--reasoning_effort", type=str, default=None,
+ help="[openai/deepseek] Reasoning effort override. Defaults: "
+ "gpt-5 'minimal'/'none' (version-dependent); deepseek "
+ "'low' ('none' disables thinking but degrades batched "
+ "labeling accuracy).")
+ return group
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Response hygiene
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+_FENCE_RE = re.compile(r"^\s*```(?:json)?\s*|\s*```\s*$", re.MULTILINE)
+_THINK_RE = re.compile(r"<think>.*?(?:</think>|$)", re.DOTALL | re.IGNORECASE)
+
+
+def strip_llm_noise(text):
+ # type: (str) -> str
+ """Remove ``<think>`` blocks and markdown fences from an LLM response."""
+ cleaned = _THINK_RE.sub("", text)
+ return _FENCE_RE.sub("", cleaned).strip()
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-rig timeout (SIGALRM, Unix main thread only)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+class FatalLLMError(Exception):
+ """Non-retryable backend failure (exhausted credits, bad API key).
+
+ Retry loops must re-raise this instead of swallowing it: falling back to
+ the rule-based cleaner on a quota outage would silently degrade EVERY
+ rig of a batch run to rule-based output."""
+
+
+class RigTimeout(Exception):
+ """Raised by the per-rig SIGALRM handler to abort an over-budget rig."""
+
+
+def _alarm_handler(signum, frame):
+ raise RigTimeout()
+
+
+def install_rig_alarm(timeout_seconds):
+ # type: (int) -> Optional[object]
+ """Arm a SIGALRM for *timeout_seconds*; returns the previous handler, or
+ ``None`` when no alarm was installed (disabled, unsupported, or not in
+ the main thread)."""
+ if not timeout_seconds or timeout_seconds <= 0:
+ return None
+ if not hasattr(signal, "SIGALRM"):
+ return None
+ try:
+ prev = signal.signal(signal.SIGALRM, _alarm_handler)
+ except ValueError:
+ return None
+ signal.alarm(int(timeout_seconds))
+ return prev
+
+
+def uninstall_rig_alarm(prev_handler):
+ if prev_handler is None:
+ return
+ signal.alarm(0)
+ signal.signal(signal.SIGALRM, prev_handler)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# JSON persistence
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def load_json(path):
+ # type: (str) -> Dict
+ """Load a JSON dict if *path* exists, else return ``{}``."""
+ if os.path.exists(path):
+ with open(path, "r") as f:
+ data = json.load(f)
+ logger.info("Loaded {} existing entries from {}".format(len(data), path))
+ return data
+ return {}
+
+
+def save_json(data, path):
+ # type: (Dict, str) -> None
+ """Write *data* as JSON to *path*, atomically.
+
+ The batch drivers re-save the whole file after every rig (3676 times for
+ an Objaverse sweep), so a plain truncate-rewrite leaves a truncated,
+ unparseable file if the process is killed mid-write โ€” and the two batch
+ stages keep no ``.bak``. Write a temp file in the same directory and
+ ``os.replace`` it into place instead, so readers only ever see the old
+ or the new file.
+ """
+ directory = os.path.dirname(path) or "."
+ fd, tmp_path = tempfile.mkstemp(
+ dir=directory, prefix=os.path.basename(path) + ".", suffix=".tmp")
+ try:
+ with os.fdopen(fd, "w") as f:
+ json.dump(data, f, indent=2)
+ os.replace(tmp_path, path)
+ except BaseException:
+ if os.path.exists(tmp_path):
+ os.remove(tmp_path)
+ raise
+
+
+def read_id_list(path):
+ # type: (str) -> List[str]
+ """Read one id per line, skipping blanks and ``#`` comments."""
+ with open(path, "r") as f:
+ return [line.strip() for line in f if line.strip() and not line.startswith("#")]
+
+
+def append_id_list(path, ids):
+ # type: (str, Iterable[str]) -> None
+ with open(path, "a") as f:
+ for rig_id in ids:
+ f.write(rig_id + "\n")
diff --git a/data_process/joint_annotation/names_clean_llm.py b/data_process/joint_annotation/names_clean_llm.py
new file mode 100644
index 0000000..44ca613
--- /dev/null
+++ b/data_process/joint_annotation/names_clean_llm.py
@@ -0,0 +1,474 @@
+"""Clean joint names with a Language Model.
+
+Complement to the rule-based :mod:`names_clean_rule`: sends raw joint names to an
+LLM and asks it to map each one to a canonical anatomical label from
+:mod:`vocab`. Pure text in / text out โ€” no images needed.
+
+Backends (see :mod:`llm`): local HF causal LM, OpenAI API (default:
+gpt-5-mini), DeepSeek API.
+
+Input : ``joint_names.json`` โ€” ``{rig_id: [raw_joint_name, ...]}`` where
+ ``rig_id`` keys may be humans, animals, props, vehicles or any other
+ skeletal asset (Objaverse uses opaque UUIDs; Truebones uses animal
+ names).
+Output : ``clean_joint_names.json`` โ€” same shape, with cleaned names.
+
+Each rig is processed as a single batched request: the full joint list is
+sent together so the LLM can reason about the rig as a whole, and the
+response must be a JSON array of the same length. Rigs that fail after
+``--max_retries`` fall back to the rule-based cleaner and are recorded in
+``failed_clean_names.txt`` next to the output.
+
+Runs resume: a rig that already has a correct-length entry in the output is
+skipped. Because the rule-based stage writes the SAME default file
+(``<export_dir>/clean_joint_names.json``), that makes this stage a no-op after
+it โ€” and it also skips the very rigs listed in ``failed_clean_names.txt``, since a
+fallen-back rig has a full-length entry too. Use ``--overwrite`` to redo every
+rig or ``--redo_failed`` to redo just the recorded failures.
+
+Usage:
+ python -m data_process.joint_annotation.names_clean_llm \\
+ --input <export_dir>/joint_names.json # OpenAI gpt-5-mini
+ python -m data_process.joint_annotation.names_clean_llm \\
+ --input <export_dir>/joint_names.json --model deepseek-v4-flash
+ python -m data_process.joint_annotation.names_clean_llm \\
+ --input <export_dir>/joint_names.json --model Qwen/Qwen3-8B
+"""
+
+import argparse
+import json
+import os
+import sys
+import time
+
+from loguru import logger
+from tqdm import tqdm
+
+_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+from data_process.joint_annotation.names_clean_rule import ( # noqa: E402
+ clean_joint_name,
+ post_process,
+ report_coverage,
+)
+from data_process.joint_annotation.llm import ( # noqa: E402
+ FatalLLMError,
+ LLMClient,
+ MAX_RETRIES,
+ add_llm_args,
+ append_id_list,
+ load_json,
+ read_id_list,
+ save_json,
+ strip_llm_noise,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# System prompt โ€” encodes the vocab.py canonical vocabulary
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+SYSTEM_PROMPT = (
+ "Standardize 3D rig joint names to canonical anatomical labels. Inputs "
+ "come from Mixamo, Maya, Blender, Unreal, Truebones and custom rigs. "
+ "Focus on semantic meaning, not surface syntax.\n\n"
+ "OUTPUT: a single JSON array of strings, same length and order as the "
+ "input. No prose, no fences, no extra text. One input entry -> one "
+ "output entry; never dedupe, merge, skip or reorder, even when "
+ "neighbours produce identical labels.\n\n"
+ "CLEAN each name by removing rig noise and extracting meaning:\n"
+ "1. Digits and Blender '.NNN' counters are meaningless โ€” rig-internal "
+ "bookkeeping (chain index, mirror id, duplicate counter). Ignore them "
+ "when matching, and never include a digit, underscore, or dot in the "
+ "final label. 'Spine', 'Spine1', 'Spine_02', 'Spine.003' all map to "
+ "'Spine'. Finger-chain segments ('Thumb1/Thumb2/Thumb3') all map to "
+ "'Thumb Finger'. The Objaverse export pipeline also stamps a trailing "
+ "global index on EVERY name ('_NN' or '_0NN': '_01', '_010', '_063'); "
+ "indices can STACK ('Hip_01_41', 'Head_1_016', 'Index.R.001_013_6', "
+ "'Bone.001_01', 'Spine_1_013') โ€” strip ALL of them, in any position. Editor decorations are noise too: strip '(mirrored)' anywhere and a trailing '.x' center marker ('spine_01.x' -> 'Spine').\n"
+ "2. Drop any leading '<Word>:' namespace (case-insensitive, trailing "
+ "digits in the namespace OK): mixamorig:, Mixamorig1:, Mutant:, Sif:. "
+ "The same words are noise without the colon too โ€” leading "
+ "'mixamorig_'/'Mixamorig'/'Character'/'Rig'/'QuickRigCharacter_' "
+ "segments are dropped, never echoed into the label. The same goes "
+ "for embedded ASSET/CHARACTER names and their decorations โ€” "
+ "'rp_karl_animated_006_warmingUp_spine_01' -> 'Spine', "
+ "'CMan0205-M4-CS_Hips L Finger0' -> 'Left Thumb Finger': keep only "
+ "the anatomical tokens, drop every name-like or counter token.\n"
+ "3. Drop rig prefixes (match before ignoring digits so 'Bip01_' still "
+ "strips cleanly): any Bip<digits> container with any separator (Bip01_, Bip002 , Bip01-, and separator-free Bip001LFinger0), BN_Bip01_, BN_, Bn_, NPC_, jt_, Elk, "
+ "Sabrecat_, QuickRigCharacter_, Bind_, Skeleton_, Root_, DEF-, def_.\n"
+ "4. Drop Maya suffixes: _jnt, _jt, _Jt, _JNT, _joint, _bone, _bn, _C.\n"
+ "5. Extract side as explicit 'Left '/'Right ' prefix. Recognise:\n"
+ " prefix L_ / R_, Lt_ / Rt_, Left_ / Right_, Left<UpperWord> / "
+ "Right<UpperWord> (LeftHand, RightArm), L<UpperLetter> / R<UpperLetter> "
+ "(LArm, RHand) โ€” but NOT when followed by a lowercase letter (Lower, "
+ "Ribcage).\n"
+ " 3ds Max Biped space-separated single-letter side: 'Bip001 L "
+ "UpperArm' -> Left Upper Arm, 'Bip001 R Thigh' -> Right Thigh. Token "
+ "boundaries are spaces, not underscores.\n"
+ " suffix _L / _R / _l / _r, .L / .R, Japanese trailing L/R.\n"
+ " PRECEDENCE: a trailing .L/.R/_L/_R token OVERRIDES a leading side word โ€” Blender's symmetrize renames only the suffix, leaving the prefix text stale: 'mixamorig:LeftShoulder.R' -> 'Right Shoulder', 'r_toe.L' -> 'Left Toe'.\n"
+ " quadruped F_/B_ = Front/Back (e.g. 'F_R_Shoulder' -> "
+ "'Right Front Shoulder', 'B_L_Foot' -> 'Left Back Foot').\n"
+ "6. Common body-part roots โ€” translate case-insensitively, accepting "
+ "Unreal snake_case AND CamelCase short-forms: pelvis/spine/neck/head/jaw/"
+ "eye -> Pelvis/Spine/Neck/Head/Jaw/Eye; clavicle/collar -> Shoulder; "
+ "upperarm / UpperArm / UpArm -> Upper Arm; lowerarm / LowerArm / LowArm / "
+ "ForeArm -> Forearm; hand -> Hand; thigh / upleg / UpLeg -> Thigh; "
+ "calf / lowleg / LowLeg / lowerleg -> Shin; upperleg -> Thigh; "
+ "toes -> Toe; in a mixamo chain "
+ "(UpLeg -> Leg -> Foot) the mid-bone 'Leg' is the shin -> Shin; foot -> "
+ "Foot; toebase / Toe0 / toe -> Toe; ball -> Toe; eyelid -> Eyelid. "
+ "Finger roots index/middle/ring/pinky/thumb -> '<Root> Finger'. "
+ "*_twist -> '<Root> Twist'.\n"
+ "6b. 3ds Max Biped fingers use numeric codes: Finger0*=Thumb, Finger1*="
+ "Index, Finger2*=Middle, Finger3*=Ring, Finger4*=Pinky. Any trailing "
+ "digits after that code are chain position โ€” ignore. 'Bip001 L Finger0' "
+ "and 'Bip001 L Finger01' both -> 'Left Thumb Finger'; 'Bip001 R Finger21'"
+ " -> 'Right Middle Finger'; 'Bip001 R Toe0' -> 'Right Toe'.\n"
+ "6c. Mocap-segmented fingers are 1-BASED and carry a segment word: Finger1..Finger5 + Metacarpal/Proximal/Medial/Distal/Tip, with Finger1=Thumb ... Finger5=Pinky. 'LeftFinger1Metacarpal' -> 'Left Thumb Finger'; the Tip segment -> '<Root> Finger End'. Rule 6b's 0-based codes apply only to BARE Finger<digit> names with no segment word. Segment words after a NAMED finger ('IndexDistal', 'thumb_proximal_l', 'RingIntermediate') are likewise chain position โ€” drop them: 'IndexDistal' -> 'Index Finger'.\n"
+ "7. Other direction words: Top->Upper, Low->Lower ('Topjaw'->'Upper Jaw'). "
+ "'HeadTop_End' / '*_End' / '*Nub' -> '<Root> End'. Animal 'Hair*/Mane*' -> 'Mane'; humanoid accessories 'Ponytail*/Cape*/Cloth*/Skirt*' -> 'Appendage'.\n"
+ "8. Placeholders -> 'Bone': Bone, joint, Xtra*, MagicEffectsNode, "
+ "and any token with no clear anatomy (meshok, Capuche, ...). Do NOT "
+ "fabricate body parts. EXCEPTION: a name that is ONLY digits, "
+ "optionally with a leading underscore ('_00', '12'), is copied "
+ "through UNCHANGED โ€” it marks a rig with unnamed bones.\n"
+ "9. Japanese roots (Alligator/Pirrana/Tukan):\n"
+ " body momo=Thigh, hiza=Knee, ashi=Foot, hiji=Elbow, te=Hand, "
+ "kata=Shoulder, mune=Chest, hara=Abdomen, koshi/kosi=Hips, kubi=Neck, "
+ "atama/kao=Head, ago=Jaw\n"
+ " tail sippo/shippo=Tail, o=Tail\n"
+ " fish munabire=Pectoral Fin, harabire=Pelvic Fin, sebire=Dorsal "
+ "Fin, obire=Caudal Fin, shiribire=Anal Fin, era=Gill\n"
+ " Trailing L/R on any of these -> Left/Right prefix.\n"
+ "10. Output Title Case, single spaces. Prefer the CANONICAL vocabulary; "
+ "if nothing fits, use 'Bone'.\n\n"
+ "ALIASES & TYPOS (map to the canonical term): Spline=Spine, Scull=Skull, Nek=Neck, Tai=Tail, Tone/Thouge/Tunge=Tongue, Eyeleds=Eyelid, HorseLink=Fetlock, LargeCannon=Cannon, PhalanxPrima=Pastern, PhalangesManus=Phalanges, Foreleg=Front Leg, Hindleg=Hind Leg, Digit=Finger, Hair=Mane, Little=Pinky (finger), locator/Trajectory/Cog=Root, Clavicle/Collarbone=Shoulder. Species terms: insect 'Clip'/'Shall'=Mandible, 'Pliers'/'Piers'=Pincer, cricket 'Feeler'=Antenna but fish 'Feelers'=Barbel, bird 'ponitail'=Crest.\n\n"
+ "CANONICAL VOCABULARY (use these exact terms verbatim):\n"
+ " Core: Pelvis, Hips, Spine, Ribcage, Neck, Head, Skull, Skull Base, "
+ "Head End, Body, Upper Body, Lower Body, Chest, Abdomen, Waist, Collar, "
+ "Hip, Belly, Root, Center\n"
+ " Arm : Shoulder, Scapula, Arm, Upper Arm, Forearm, Elbow, Wrist, Hand, Palm\n"
+ " Leg : Thigh, Shin, Leg, Knee, Ankle, Foot, Heel, Toe, Paw, Hoof, "
+ "Fetlock, Cannon, Metacarpus, Phalanges, Pastern\n"
+ " Finger: Finger, Thumb Finger, Index Finger, Middle Finger, Ring Finger, "
+ "Pinky Finger\n"
+ " Head: Jaw, Upper Jaw, Lower Jaw, Tongue, Ear, Eye, Eyeball, Eyebrow, "
+ "Eyelid, Mouth, Lip, Upper Lip, Lower Lip, Nose, Muzzle, Chin, Cheek\n"
+ " Appendage: Tail, Wing, Feather, Antenna, Barbel, Tentacle, Claw, "
+ "Hand Claw, Fang, Mandible, Large Mandible, Lower Mandible, Pincer, "
+ "Stinger, Appendage\n"
+ " Fins: Fin, Pectoral Fin, Pelvic Fin, Dorsal Fin, Caudal Fin, Anal Fin, Gill\n"
+ " Coat/Equipment: Mane, Fur, Whisker, Shell, Dorsal Plate, Crest, Horn, Reins, Halter\n"
+ " Quadruped: Front Leg, Middle Leg, Hind Leg, Front Paw, Back Paw, "
+ "Front Hoof, Rear Hoof, Front Shoulder, Back Hip\n"
+ " Physics: Twist, Upper Arm Twist, Forearm Twist, Thigh Twist, Shin Twist, "
+ "Thigh Muscle, Neck Muscle, Tail Twist, Jiggle, Handle, IK Chain, Bone\n"
+ "Side goes first ('Left Upper Arm', 'Right Pinky Finger'); quadruped "
+ "markers sit between side and part ('Right Front Shoulder'). "
+ "COMPOSED labels are also valid: '<part> End' for chain tips/Nub "
+ "bones, 'Inner/Middle/Outer <part>' (raptor toes, claws, fingers), "
+ "'Upper/Lower Eyelid', 'Upper/Lower Left/Right/Front Lip', "
+ "'Wrist Back', 'Elbow Back', 'Front/Middle/Hind Leg End'.\n\n"
+ "EXAMPLES (one per rig family):\n"
+ " mixamorig:LeftUpLeg -> Left Thigh\n"
+ " Mutant:RightHandThumb1 -> Right Thumb Finger\n"
+ " Sif:calf_twist_01_r -> Right Shin Twist\n"
+ " index_01_l -> Left Index Finger\n"
+ " Bip01_L_Thigh -> Left Thigh\n"
+ " BN_Bip01_R_Forearm_03 -> Right Forearm\n"
+ " NPC_L_Finger02 -> Left Finger\n"
+ " Elk_RearHoof_L -> Left Rear Hoof\n"
+ " jt_FrontLeg1_R_C -> Right Front Leg\n"
+ " Lt_Thumb1_jt -> Left Thumb Finger\n"
+ " R_toeBase_jnt -> Right Toe\n"
+ " Eye.R.001 -> Right Eye\n"
+ " mixamorig:LeftShoulder.R -> Right Shoulder\n"
+ " LeftHandIndex1 -> Left Index Finger\n"
+ " mixamorig:RightLeg -> Right Shin\n"
+ " LeftFinger2Distal -> Left Index Finger\n"
+ " BN_Spline_03 -> Spine\n"
+ " Bone.001 -> Bone\n"
+ " F_R_Shoulder -> Right Front Shoulder\n"
+ " B_L_Foot -> Left Back Foot\n"
+ " Topjaw -> Upper Jaw\n"
+ " momoR -> Right Thigh\n"
+ " munabireL -> Left Pectoral Fin\n"
+ " joint12 / Xtra01 / meshok -> Bone\n"
+ "OBJAVERSE (trailing global '_NN', often stacked with a chain index):\n"
+ " mixamorig:LeftUpLeg_056 -> Left Thigh\n"
+ " mixamorig:LeftHandThumb1_012 -> Left Thumb Finger\n"
+ " QuickRigCharacter_LeftForeArm_014 -> Left Forearm\n"
+ " Bip001 L UpperArm_07 -> Left Upper Arm\n"
+ " Bip001 L Finger01_011 -> Left Thumb Finger\n"
+ " Bip001 R Finger21_036 -> Right Middle Finger\n"
+ " Bip001 R Toe0_054 -> Right Toe\n"
+ " UpArm.R_010_15 -> Right Upper Arm\n"
+ " LowLeg.L_038_39 -> Left Shin\n"
+ " Hip_01_41 -> Pelvis\n"
+ " Index.R.001_013_6 -> Right Index Finger\n"
+ " Rt_Eyelid_jt_08 -> Right Eyelid\n"
+ " Skeleton_Root_02 -> Root\n"
+ " joint1_2 / Bone.001_01 -> Bone\n\n"
+ "BATCH EXAMPLE (notice identical outputs are preserved, not merged):\n"
+ " input : [\"Spine\", \"Spine1\", \"Spine2\", \"Spine3\", \"Neck\", "
+ "\"Tail_01\", \"Tail_02\", \"Tail_03\"]\n"
+ " output : [\"Spine\", \"Spine\", \"Spine\", \"Spine\", \"Neck\", "
+ "\"Tail\", \"Tail\", \"Tail\"]\n"
+ " WRONG : [\"Spine\", \"Neck\", \"Tail\"] (8 inputs must yield 8 "
+ "outputs โ€” deduping is a failure)\n"
+)
+
+
+def build_user_prompt(rig_id, raw_names, correction=None):
+ # type: (str, List[str], Optional[str]) -> str
+ prompt = (
+ "Rig identifier: {}\n"
+ "Joint count: {}\n\n"
+ "Raw joint names (JSON array, order matters):\n"
+ "{}\n\n"
+ "Return a JSON array of the SAME length with each raw name replaced by "
+ "its cleaned canonical label."
+ ).format(rig_id, len(raw_names), json.dumps(raw_names, ensure_ascii=False))
+ if correction:
+ prompt += "\n\nIMPORTANT: " + correction
+ return prompt
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Response parsing
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_response(text, expected_len):
+ # type: (str, int) -> List[str]
+ """Extract a JSON string array of the expected length from an LLM response.
+
+ Tolerates markdown fences, ``<think>โ€ฆ</think>`` reasoning blocks (Qwen3
+ and other thinking models) and trailing prose; raises ``ValueError`` on
+ structural mismatch so the caller can retry or fall back.
+ """
+ if not text or not text.strip():
+ raise ValueError("empty response")
+
+ cleaned = strip_llm_noise(text)
+ try:
+ data = json.loads(cleaned)
+ except json.JSONDecodeError:
+ start = cleaned.find("[")
+ end = cleaned.rfind("]")
+ if start == -1 or end == -1 or end <= start:
+ raise ValueError("no JSON array found in response")
+ data = json.loads(cleaned[start:end + 1])
+
+ if not isinstance(data, list):
+ raise ValueError("response is not a JSON array")
+ if len(data) != expected_len:
+ raise ValueError("length mismatch: got {}, expected {}".format(
+ len(data), expected_len))
+ result = []
+ for i, item in enumerate(data):
+ if not isinstance(item, str) or not item.strip():
+ raise ValueError("element {} is not a non-empty string".format(i))
+ result.append(post_process(item.strip()))
+ return result
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-rig cleaner (with retry + rule-based fallback)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def rule_based_clean(rig_id, raw_names):
+ # type: (str, List[str]) -> List[str]
+ """Rule-based fallback. ``rig_id`` drives animal-specific dispatch for
+ Truebones (e.g. Spider, Pirrana); opaque Objaverse UUIDs fall through to
+ the generic cleaner."""
+ return [post_process(clean_joint_name(raw, rig_id)) for raw in raw_names]
+
+
+def clean_rig(rig_id, raw_names, client, system_prompt=SYSTEM_PROMPT,
+ max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True):
+ # type: (...) -> Tuple[List[str], bool]
+ """Clean one rig's joint list via the LLM. Returns ``(names, used_fallback)``.
+
+ Structural errors (length mismatch, bad JSON, ...) are fed back to the
+ LLM on the next attempt so it can self-correct.
+ """
+ if not raw_names:
+ return [], False
+
+ last_err = None
+ correction = None
+ for attempt in range(1, max_retries + 1):
+ try:
+ raw_text = client.generate(
+ system_prompt, build_user_prompt(rig_id, raw_names, correction=correction),
+ max_tokens)
+ return parse_response(raw_text, len(raw_names)), False
+ except FatalLLMError:
+ raise
+ except Exception as err: # noqa: BLE001 โ€” retry on any backend/parse error
+ last_err = err
+ if isinstance(err, ValueError):
+ correction = (
+ "Your previous response failed validation ({}). Return a "
+ "JSON array of EXACTLY {} strings โ€” one cleaned label per "
+ "input joint, in the same order. Do not add, merge, split, "
+ "or skip any entries. Output only the JSON array, no prose "
+ "or code fences."
+ ).format(err, len(raw_names))
+ logger.warning("[{}] attempt {}/{} failed: {}".format(
+ rig_id, attempt, max_retries, err))
+ if attempt < max_retries:
+ time.sleep(1.5)
+
+ if not fallback_on_error:
+ raise RuntimeError("LLM cleaning failed for {}: {}".format(rig_id, last_err))
+
+ logger.error("[{}] all {} attempts failed ({}); falling back to rule-based "
+ "cleaner".format(rig_id, max_retries, last_err))
+ return rule_based_clean(rig_id, raw_names), True
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Batch driver
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def clean_all(input_path, output_path, client, system_prompt=SYSTEM_PROMPT,
+ max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True,
+ failed_path=None, overwrite=False, redo_failed=False):
+ # type: (...) -> Dict[str, List[str]]
+ """Clean every rig in ``input_path`` and write to ``output_path``.
+
+ Resumes from a partial output file: rigs whose cleaned list is already
+ present with the correct length are skipped. Rig ids that fall back to
+ the rule-based cleaner are appended to ``failed_path`` (one per line) so
+ the batch can be re-run later with a stronger prompt/model.
+
+ That resume rule makes this stage a silent no-op when ``output_path``
+ was already filled in by the rule-based cleaner (both wrappers default to
+ ``<export_dir>/clean_joint_names.json``), and it also skips exactly the
+ rigs recorded in ``failed_path`` โ€” a fallen-back rig already has a
+ full-length list. ``overwrite`` re-processes every rig; ``redo_failed``
+ re-processes only the ids in ``failed_path``.
+ """
+ with open(input_path, "r") as f:
+ raw_data = json.load(f)
+
+ cleaned = load_json(output_path)
+ rig_ids = sorted(raw_data.keys())
+
+ if failed_path is None:
+ failed_path = os.path.join(os.path.dirname(output_path) or ".", "failed_clean_names.txt")
+ already_failed = set(read_id_list(failed_path)) if os.path.exists(failed_path) else set()
+ redo_ids = already_failed if redo_failed else set()
+ if redo_failed:
+ logger.info("--redo_failed: re-processing {} rig ids listed in {}".format(
+ len(redo_ids), failed_path))
+
+ fallback_count = 0
+ processed = skipped = 0
+ pbar = tqdm(rig_ids, desc="Cleaning", unit="rig", dynamic_ncols=True)
+ for rig_id in pbar:
+ raw_names = raw_data[rig_id]
+ pbar.set_postfix_str(rig_id, refresh=False)
+
+ existing = cleaned.get(rig_id)
+ if (not overwrite and rig_id not in redo_ids
+ and isinstance(existing, list) and len(existing) == len(raw_names)):
+ skipped += 1
+ continue
+
+ processed += 1
+ names_out, used_fallback = clean_rig(
+ rig_id, raw_names, client,
+ system_prompt=system_prompt, max_tokens=max_tokens,
+ max_retries=max_retries, fallback_on_error=fallback_on_error,
+ )
+ fallback_count += int(used_fallback)
+ cleaned[rig_id] = names_out
+ save_json(cleaned, output_path)
+
+ if used_fallback and rig_id not in already_failed:
+ append_id_list(failed_path, [rig_id])
+ already_failed.add(rig_id)
+
+ logger.info("Done. {}/{} rigs cleaned by the LLM ({} via rule-based "
+ "fallback), {} skipped (already complete); {} now holds {} "
+ "rigs.".format(processed, len(rig_ids), fallback_count,
+ skipped, output_path, len(cleaned)))
+ if processed == 0 and skipped:
+ logger.warning(
+ "Nothing was re-processed: every rig in {} already had a "
+ "correct-length entry in {} (the rule-based stage writes the same "
+ "file). Pass --overwrite to redo them all, or --redo_failed to "
+ "redo just the ids in {}.".format(input_path, output_path, failed_path))
+ if fallback_count:
+ logger.info("Failed rig ids recorded at {}".format(failed_path))
+ return cleaned
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# CLI
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Clean joint names via an LLM (local / openai / deepseek).")
+ parser.add_argument("--input", required=True,
+ help="Input joint_names.json path.")
+ parser.add_argument("--output", default=None,
+ help="Output JSON path (default: sibling 'clean_joint_names.json').")
+ parser.add_argument("--report", action="store_true",
+ help="Print a residual-token coverage report after cleaning.")
+ parser.add_argument("--no_fallback", action="store_true",
+ help="Raise on LLM failure instead of falling back to the "
+ "rule-based cleaner.")
+ parser.add_argument("--failed_log", default=None,
+ help="Path to append rig ids that hit the rule-based fallback "
+ "(default: sibling 'failed_clean_names.txt').")
+ parser.add_argument("--overwrite", action="store_true",
+ help="Re-process every rig, even one that already has a "
+ "correct-length entry in the output. Without this the "
+ "stage is a no-op when the rule-based cleaner already "
+ "wrote the same file.")
+ parser.add_argument("--redo_failed", action="store_true",
+ help="Also re-process the rig ids listed in --failed_log "
+ "(they always have a full-length entry from the "
+ "rule-based fallback, so they are otherwise skipped).")
+ add_llm_args(parser, default_max_tokens=2048)
+ args = parser.parse_args()
+
+ output_path = args.output or os.path.join(
+ os.path.dirname(args.input), "clean_joint_names.json")
+ client = LLMClient.from_args(args)
+ logger.info("{} | Input: {} | Output: {}".format(client, args.input, output_path))
+
+ clean_data = clean_all(
+ input_path=args.input,
+ output_path=output_path,
+ client=client,
+ max_tokens=args.max_tokens,
+ max_retries=args.max_retries,
+ fallback_on_error=not args.no_fallback,
+ failed_path=args.failed_log,
+ overwrite=args.overwrite,
+ redo_failed=args.redo_failed,
+ )
+
+ with open(args.input, "r") as f:
+ raw_data = json.load(f)
+ for rig_id in list(clean_data.keys())[:3]:
+ print("\n--- {} ---".format(rig_id))
+ for raw, out in zip(raw_data.get(rig_id, []), clean_data[rig_id]):
+ print(" {:50s} -> {}".format(raw, out))
+
+ if args.report:
+ report_coverage(clean_data)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/joint_annotation/names_clean_rule.py b/data_process/joint_annotation/names_clean_rule.py
new file mode 100644
index 0000000..9c00973
--- /dev/null
+++ b/data_process/joint_annotation/names_clean_rule.py
@@ -0,0 +1,601 @@
+"""Rule-based joint-name cleaning: ``joint_names.json`` โ†’ ``clean_joint_names.json``.
+
+Applies prefix stripping, CamelCase splitting, Japanese translation and
+canonical anatomical vocabulary mapping. The vocabulary lives in
+:mod:`vocab`; :mod:`names_clean_llm` is the LLM-based complement and uses
+this module as its fallback.
+
+Usage:
+ python -m data_process.joint_annotation.names_clean_rule \
+ --input <export_dir>/joint_names.json [--output ...] [--report]
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+
+_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+from data_process.joint_annotation.vocab import (
+ CANONICAL,
+ ELK_MAP,
+ JAPANESE_COMPOUND_ANIMALS,
+ JAPANESE_COMPOUND_NAMES,
+ JAPANESE_WORDS_LOWER,
+ JT_MAP,
+ MIXAMO_MAP,
+ NPC_DIRECT,
+ PIRRANA_COMPOUNDS,
+ REMOVE_PREFIXES,
+ SABRECAT_MAP,
+ SPIDER_MAP,
+ STANDALONE_MAP,
+)
+
+
+# ---------------------------------------------------------------------------
+# Common helpers
+# ---------------------------------------------------------------------------
+
+# Explicit ``_L_`` / ``_R_`` / ``.L`` / ``.R`` side token: an isolated L or R
+# delimited by an underscore/dot on the left and by a delimiter, a digit or
+# the end of the name on the right. This is the strongest side marker a raw
+# bone name carries, so it outranks a bare single-letter prefix.
+_SIDE_TOKEN_RE = re.compile(r'[._]([LR])(?=[._]|\d|$)')
+
+# A leading single L/R only marks a side when what follows reads as a word
+# ("LThigh", "LEar", "LWing01") or as a short side abbreviation followed by a
+# word ("LB_foot", "LT_BackFoot_09", "LFT_ankle_09"). All-caps tokens
+# ("ROOT_01", "RIG_01", "LOD", "LEG", "RUG_01") and rig namespaces
+# ("LP1:Head") must NOT match โ€” there the L/R is part of the word itself.
+_SINGLE_SIDE_RE = re.compile(r'^[LR](?:[A-Z][a-z]|[A-Z]{0,2}_[A-Za-z])')
+
+# Trailing Blender/Rigify decorations: '.NNN' duplicate counters, '.NN' chain
+# indices, Objaverse '_NN' global indices, '.x' center markers and '.L'/'.R'
+# ('.l'/'.r') side tokens stack in any order at the end of a name
+# ('thumb.01.R_037', 'Bone.005.R', 'spine_01.x', 'Index.R.001_013_6'). The
+# separator requirement keeps lookup keys like 'Spine2' / 'Tail01' intact.
+_TRAILING_TOKEN_RE = re.compile(r'[._](\d+|[LRlr]|x)$')
+
+# 3ds Max Biped numeric finger codes (0-based): 'Finger0'/'Finger01' = thumb
+# chain; trailing digits after the code are chain position.
+_FINGER_CODE_RE = re.compile(r'^[Ff]inger([0-4])\d*(Nub)?$')
+_FINGER_CODE = {"0": "Thumb", "1": "Index", "2": "Middle", "3": "Ring",
+ "4": "Pinky"}
+# CMU / mocap-style segmented fingers (1-based): 'Finger1Metacarpal' etc.
+_FINGER_SEG_RE = re.compile(
+ r'^[Ff]inger([1-5])(Metacarpal|Proximal|Medial|Distal|Tip)\d*$')
+_FINGER_ORD = {"1": "Thumb", "2": "Index", "3": "Middle", "4": "Ring",
+ "5": "Pinky"}
+
+# 'Bone001(mirrored)' and similar editor bookkeeping โ€” anywhere in the name,
+# since the Objaverse global index can trail it: 'Bone018(mirrored)_079'.
+_PAREN_DECOR_RE = re.compile(r'\s*\([^)]*\)')
+
+# Any leading '<word>:' rig namespace ('RenWu2:', 'candi_aaaaaa2:'). Known
+# ones (mixamorig:, Mutant:, Sif:) are handled earlier with their own maps.
+_NAMESPACE_RE = re.compile(r'^[A-Za-z][\w .-]*:')
+
+# Generic 3ds Max Biped container prefix: Bip01_/Bip002 /Bip01-/Bip001L...
+# (any index; separator optional when a side letter follows), optionally
+# behind a BN_ namespace.
+_BIP_PREFIX_RE = re.compile(r'^(?:BN_)?Bip\d+(?:[-_ ]+|(?=[LR][A-Z]))')
+
+
+def strip_trailing_decorations(name):
+ # type: (str) -> Tuple[str, str]
+ """Peel stacked trailing ``[._]``-separated tokens off *name*.
+
+ Digit groups and ``x`` center markers are dropped; the innermost ``L``/
+ ``R`` token wins as the side. Returns ``(side, base)`` with ``side`` one
+ of ``""`` / ``"Left"`` / ``"Right"``.
+ """
+ side = ""
+ while True:
+ m = _TRAILING_TOKEN_RE.search(name)
+ if not m:
+ return side, name
+ tok = m.group(1)
+ if tok in ("L", "l"):
+ side = "Left"
+ elif tok in ("R", "r"):
+ side = "Right"
+ name = name[:m.start()]
+
+
+def _canon(token):
+ # type: (str) -> Optional[str]
+ """CANONICAL lookup, tolerating all-lower/all-upper raw tokens
+ ('thumb' -> 'Thumb Finger'). Exact match wins so CamelCase keys like
+ 'UpLeg' are never shadowed by their capitalize() form."""
+ value = CANONICAL.get(token)
+ if value is None and token and token != token.capitalize():
+ value = CANONICAL.get(token.capitalize())
+ return value
+
+
+def extract_side_prefix(name):
+ # type: (str) -> Tuple[str, str]
+ """Extract Left/Right side from common prefix/suffix patterns.
+
+ Handles: R_/L_ prefix, Left_/Right_ prefix, single R/L followed by
+ uppercase, _L_NN/_R_NN suffix, _L/_R suffix. Guards against false
+ positives on words that begin with R or L followed by a lowercase letter
+ (e.g. "Ribcage", "Lower..."), on all-caps tokens that merely start with
+ one ("ROOT_01", "RIG_01", "LEG") and on rig namespaces ("LP1:Head").
+
+ Markers are ranked: an explicit ``_L_``/``_R_``/``.L``/``.R`` token beats
+ a bare single-letter prefix, because rigs that carry both use the token as
+ the real side ("LLeg.R_08" is the RIGHT leg, "LB_R_Shoulder_DEF" the RIGHT
+ shoulder). A wrong side here silently rotates the clip 180 degrees, since
+ the side flows into face_select -> get_root_facing_quat.
+
+ Returns:
+ (side, remaining_name) where side is "" | "Left" | "Right".
+ """
+ # Trailing Blender/Rigify decorations first: '.L'/'.R' side tokens mixed
+ # with '.NNN'/'_NN' counters ('thumb.01.R_037' -> Right 'thumb'). When no
+ # side is found the stripped base still flows into the prefix checks.
+ side, name = strip_trailing_decorations(name)
+ if side:
+ # A symmetrized bone keeps its pre-mirror text: Blender renames only
+ # the suffix, so 'mixamorig:LeftShoulder.R' / 'r_toe.L' carry a STALE
+ # leading side word. The trailing token is the real side โ€” drop the
+ # leftover prefix so it can't pollute the vocabulary lookup.
+ stripped = re.sub(
+ r'^(?:Left|Right)(?=[A-Z])|^(?:Left|Right)[_ ]|^[LR][-_]|^[lr]_',
+ '', name)
+ return side, stripped or name
+
+ # R_X / L_X / R-X / L-X / R.X / L.X prefix
+ if re.match(r'^[RL][-_.]', name):
+ side = "Left" if name[0] == "L" else "Right"
+ return side, name[2:]
+
+ # "R X" / "L X" prefix with space separator (e.g. "Bip01 L UpperArm"
+ # becomes "L UpperArm" after prefix strip, then "UpperArm").
+ m = re.match(r'^([RL]) +', name)
+ if m:
+ side = "Left" if m.group(1) == "L" else "Right"
+ return side, name[m.end():]
+
+ # Left_X or Right_X prefix (case-insensitive)
+ m = re.match(r'^(Left|Right)_', name, re.IGNORECASE)
+ if m:
+ return m.group(1).capitalize(), name[len(m.group(0)):]
+
+ # "Left" / "Right" directly followed by uppercase (Mixamo: LeftHandThumb1).
+ m = re.match(r'^(Left|Right)(?=[A-Z])', name)
+ if m:
+ return m.group(1), name[len(m.group(1)):]
+
+ # Single L/R followed by uppercase (e.g., LThigh, RArm). The weakest of
+ # the side markers, so it is deliberately narrow:
+ # * an explicit _L_/_R_/.L/.R token later in the name is the real
+ # marker and wins ("LLeg.R_08" is the RIGHT leg, "LB_R_Shoulder_DEF"
+ # the RIGHT shoulder);
+ # * all-caps tokens ("ROOT_01", "RIG_01", "LEG") and rig namespaces
+ # ("LP1:Head") carry no side at all.
+ if len(name) > 1 and name[0] in "LR" and name[1].isupper():
+ m = _SIDE_TOKEN_RE.search(name, 1)
+ if m:
+ side = "Left" if m.group(1) == "L" else "Right"
+ rest = (name[:m.start()] + name[m.end():]).strip("._")
+ return side, rest or name
+ if _SINGLE_SIDE_RE.match(name):
+ side = "Left" if name[0] == "L" else "Right"
+ return side, name[1:]
+
+ # _L_NN or _R_NN suffix
+ m = re.search(r'_([LR])_(\d+)$', name)
+ if m:
+ side = "Left" if m.group(1) == "L" else "Right"
+ return side, name[:m.start()] + "_" + m.group(2)
+
+ # _L or _R suffix
+ m = re.search(r'_([LR])$', name)
+ if m:
+ side = "Left" if m.group(1) == "L" else "Right"
+ return side, name[:m.start()]
+
+ return "", name
+
+
+def split_and_map_tokens(name):
+ # type: (str) -> str
+ """Split a CamelCase/underscore name and map tokens via CANONICAL.
+
+ Tokens mapping to ``""`` (rig noise like 'Joint'/'Jnt') are dropped.
+ """
+ tokens = re.split(r'(?=[A-Z])|_', name)
+ mapped = []
+ for t in tokens:
+ if not t:
+ continue
+ base = re.sub(r'\d+$', '', t)
+ value = _canon(base)
+ if value is not None:
+ if value:
+ mapped.append(value)
+ elif base and len(base) > 1:
+ mapped.append(base)
+ return " ".join(mapped)
+
+
+def with_side(side, result):
+ # type: (str, str) -> str
+ """Prepend side to result, replacing any side the mapping already carries.
+
+ A side extracted from the raw name (``R_``/``L_``, ``_L_``/``_R_``, ...)
+ is an explicit marker and outranks one baked into a vocabulary entry, so
+ ``with_side("Right", "Left Wing")`` yields ``"Right Wing"`` โ€” otherwise
+ ``R_LWing`` would clean to ``Left Wing`` via ``CANONICAL["LWing"]``.
+ """
+ if not side:
+ return result
+ m = re.match(r'^(Left|Right)\b\s*', result)
+ if m:
+ result = result[m.end():]
+ return (side + " " + result).strip()
+
+
+# ---------------------------------------------------------------------------
+# Skeleton-specific cleaners
+# ---------------------------------------------------------------------------
+
+def clean_japanese_name(name):
+ """Handle fully Japanese joint names (Alligator, Pirrana, Tukan style)."""
+ side, name = extract_side_prefix(name)
+
+ base = re.sub(r'\d+$', '', name)
+ lower = base.lower()
+ if lower in JAPANESE_WORDS_LOWER:
+ return with_side(side, JAPANESE_WORDS_LOWER[lower])
+
+ # Pirrana-specific compound words (side already encoded in the value)
+ if name in PIRRANA_COMPOUNDS:
+ return PIRRANA_COMPOUNDS[name]
+
+ return with_side(side, name)
+
+
+def clean_prefixed_name(raw, prefix, rig_map, strip_trailing_c=False, skip_substrs=()):
+ # type: (str, str, dict, bool, tuple) -> str
+ """Generic cleaner for rigs with a fixed prefix and a lookup table.
+
+ Strips ``prefix``, optionally drops a trailing ``_C`` (jt center bones),
+ extracts Left/Right side, and maps via ``rig_map`` โ€” exact match first,
+ then digit-stripped base. Falls back to CANONICAL via
+ :func:`split_and_map_tokens`. Names containing any of ``skip_substrs``
+ (e.g. ``"Jiggle"`` for NPC physics bones) are dropped.
+ """
+ name = raw[len(prefix):]
+ if strip_trailing_c and name.endswith("_C"):
+ name = name[:-2]
+ side, name = extract_side_prefix(name)
+
+ for sub in skip_substrs:
+ if sub in name:
+ return "" # post_process converts empty -> "Bone"
+
+ result = rig_map.get(name)
+ if result is None:
+ result = rig_map.get(re.sub(r'\d+$', '', name))
+ if result is None:
+ result = split_and_map_tokens(name) or name
+
+ return with_side(side, result).strip()
+
+
+def clean_spider_name(raw):
+ """Special handling for Spider skeleton names."""
+ if raw in SPIDER_MAP:
+ return SPIDER_MAP[raw]
+
+ m = re.match(r'Fang([RL])_(\d+)_', raw)
+ if m:
+ side = "Right" if m.group(1) == "R" else "Left"
+ return "{} Fang".format(side)
+
+ m = re.match(r'Leg_([RL])_(\d)(\d)_', raw)
+ if m:
+ side = "Right" if m.group(1) == "R" else "Left"
+ return "{} Leg".format(side)
+
+ m = re.match(r'_([RL])Toe(\d)_', raw)
+ if m:
+ side = "Right" if m.group(1) == "R" else "Left"
+ return "{} Leg Tip".format(side)
+
+ return raw.strip("_")
+
+
+def clean_standard_name(raw):
+ """Clean standard Bip01/BN prefixed names."""
+ name = _NAMESPACE_RE.sub("", raw) or raw
+ name = _BIP_PREFIX_RE.sub("", name)
+
+ for prefix in REMOVE_PREFIXES:
+ if name.startswith(prefix):
+ name = name[len(prefix):]
+ break
+
+ name = name.lstrip("_").strip()
+ if not name:
+ return raw.strip("_")
+
+ side, name = extract_side_prefix(name)
+
+ # Numeric finger codes, before the chain-index strip eats the code digit:
+ # CMU segments ('Finger1Metacarpal', 1-based) and 3ds Max Biped codes
+ # ('Finger0'/'Finger01'/'Finger21', 0-based).
+ m = _FINGER_SEG_RE.match(name)
+ if m:
+ label = _FINGER_ORD[m.group(1)] + " Finger"
+ if m.group(2) == "Tip":
+ label += " End"
+ return with_side(side, label)
+ m = _FINGER_CODE_RE.match(name)
+ if m:
+ label = _FINGER_CODE[m.group(1)] + " Finger"
+ if m.group(2):
+ label += " End"
+ return with_side(side, label)
+
+ # Strip a trailing chain index: "Thigh_4" -> "Thigh", "Spine02" -> "Spine".
+ m = re.match(r'^(.+?)_?(\d+)$', name)
+ base = m.group(1).rstrip("_") if m else name
+
+ mapped_base = _canon(base)
+ if mapped_base is not None:
+ # Placeholders and skip-tokens carry no side ('Bone.005.R' -> 'Bone').
+ if not mapped_base or mapped_base == "Bone":
+ return "Bone"
+ return with_side(side, mapped_base)
+
+ # Handle compound names with underscores
+ parts = [p for p in base.split("_") if p]
+ mapped = []
+ for p in parts:
+ # A bare L/R token only fills in a side that extract_side_prefix did
+ # not already find; it must NOT override one, because the side it
+ # returned came from a stronger marker (an explicit _L_/_R_/.L/.R
+ # token). 'r_index_00_L_049' is the LEFT rear-paw index finger โ€” the
+ # leading 'r' means "rear", not "right".
+ if p in ("L", "l"):
+ side = side or "Left"
+ continue
+ if p in ("R", "r"):
+ side = side or "Right"
+ continue
+
+ sub_base = re.sub(r'\d+$', '', p)
+ sub_mapped = _canon(sub_base)
+ if sub_mapped is not None:
+ if sub_mapped:
+ mapped.append(sub_mapped)
+ elif sub_base:
+ sub_tokens = [t for t in re.split(r'(?=[A-Z])', sub_base) if t]
+ for st in sub_tokens:
+ st_mapped = _canon(st)
+ if st_mapped is not None:
+ if st_mapped:
+ mapped.append(st_mapped)
+ elif st and len(st) > 1:
+ mapped.append(st)
+
+ result = " ".join(mapped)
+ if not result:
+ # Nothing anatomical survived (pure counters like '1.L', skip tokens
+ # like 'internal50.L') -> placeholder, without a side.
+ return "Bone"
+ if result == "Bone":
+ return "Bone"
+ return with_side(side, result).strip() or raw
+
+
+# ---------------------------------------------------------------------------
+# Main dispatcher
+# ---------------------------------------------------------------------------
+
+# Matches any mixamorig-style namespace: mixamorig:, mixamorig1:, mixamorig7:,
+# Mixamorig1:, MIXAMORIG:, ... The trailing digits and case both vary across
+# rigs but the vocabulary beneath is the same.
+_MIXAMORIG_RE = re.compile(r'^mixamorig\d*[:_]', re.IGNORECASE)
+
+
+def clean_joint_name(raw, animal):
+ """Main dispatcher for cleaning a single joint name."""
+ if not raw or not raw.strip():
+ return raw
+
+ # Editor bookkeeping suffixes: 'Bone001(mirrored)' -> 'Bone001'
+ raw = _PAREN_DECOR_RE.sub("", raw).strip() or raw
+
+ # Generic "BoneNN" placeholders -> "Bone" (dot forms via clean_standard_name)
+ if re.match(r'^Bone\d+$', raw):
+ return "Bone"
+
+ # Numeric only names โ€” return unchanged
+ if re.match(r'^_?\d+$', raw.strip("_")):
+ return raw
+
+ m = _MIXAMORIG_RE.match(raw)
+ if m:
+ return clean_prefixed_name(raw, m.group(0), MIXAMO_MAP)
+
+ if raw in STANDALONE_MAP:
+ return STANDALONE_MAP[raw]
+
+ if raw in SABRECAT_MAP:
+ return SABRECAT_MAP[raw]
+
+ if animal == "Spider":
+ return clean_spider_name(raw)
+
+ if raw.startswith("Sabrecat"):
+ return SABRECAT_MAP.get(raw, raw)
+
+ if raw.startswith("NPC_"):
+ return clean_prefixed_name(raw, "NPC_", NPC_DIRECT, skip_substrs=("Jiggle",))
+
+ if raw.startswith("Elk"):
+ return clean_prefixed_name(raw, "Elk", ELK_MAP)
+
+ if raw.startswith("jt_"):
+ return clean_prefixed_name(raw, "jt_", JT_MAP, strip_trailing_c=True)
+
+ # Japanese names: strip side + trailing digits, check vocabulary
+ base_lower = re.sub(r'^[RL]_', '', raw).lower()
+ base_lower = re.sub(r'\d+$', '', base_lower)
+ if base_lower in JAPANESE_WORDS_LOWER:
+ return clean_japanese_name(raw)
+
+ if animal in JAPANESE_COMPOUND_ANIMALS and raw in JAPANESE_COMPOUND_NAMES:
+ return clean_japanese_name(raw)
+
+ return clean_standard_name(raw)
+
+
+# ---------------------------------------------------------------------------
+# Post-processing
+# ---------------------------------------------------------------------------
+
+def post_process(name):
+ """Final cleanup pass."""
+ # rstrip only: a leading '_' marks the untouched numeric passthrough
+ # names ('_00') and must survive.
+ name = re.sub(r'\s+', ' ', name).strip().rstrip('._ ')
+
+ # Strip trailing chain indices: "Tail 3" -> "Tail"
+ name = re.sub(r'\s+\d+$', '', name)
+
+ # Strip numbers before End: "Toe0 End" -> "Toe End"
+ name = re.sub(r'(\w)\d+\s+End', r'\1 End', name)
+
+ # Canonical-format rewrites for compositional leftovers. The finger
+ # chain's 'Hand' segment is context, not a label word; UE/mixamo split
+ # forms collapse to their canonical terms.
+ name = re.sub(r'\bHand (Thumb|Index|Middle|Ring|Pinky) Finger\b', r'\1 Finger', name)
+ name = re.sub(r'\b(Thumb|Index|Middle|Ring|Pinky) Finger Finger\b', r'\1 Finger', name)
+ name = re.sub(r'\bFinger (Thumb|Index|Middle|Ring|Pinky) Finger\b', r'\1 Finger', name)
+ name = re.sub(r'\b(?:Up|Upper) Leg$', 'Thigh', name)
+ name = re.sub(r'\bLower Leg$', 'Shin', name)
+ name = re.sub(r'\b(?:Lower|Fore) Arm$', 'Forearm', name)
+ name = re.sub(r'\bToe Base$', 'Toe', name)
+ # Trailing side word -> canonical leading position ('Index Finger Left')
+ name = re.sub(r'^(?!Left |Right )(.+) (Left|Right)$', r'\2 \1', name)
+ name = re.sub(r'^Head Top End$', 'Head End', name)
+ name = name.replace('. ', ' ')
+
+ # Dog-specific patches for misrouted head/spine subparts (with or
+ # without the chain digit, which earlier passes may already strip)
+ name = re.sub(r'^Spine ?\d* ?Tail$', 'Tail', name)
+ name = re.sub(r'^Head ?\d* ?Jaw$', 'Jaw', name)
+ name = re.sub(r'^Head ?\d* ?Eyelid$', 'Eyelid', name)
+ name = re.sub(r'^Head Muzzle$', 'Muzzle', name)
+ name = re.sub(r'^Head Jaw End$', 'Jaw End', name)
+ name = re.sub(r'^Head Brain$', 'Head', name)
+
+ # Clean leftover Bip01 fragments
+ name = re.sub(r'Bip \d+ ', '', name)
+
+ # "Spine Spine" -> "Spine"
+ name = re.sub(r'^(\w+) \1$', r'\1', name)
+
+ if re.match(r'^Xtra', name):
+ name = "Bone"
+
+ name = re.sub(r'Ponytail\d*.*', 'Appendage', name)
+
+ # Title case consistency: "Right leg" -> "Right Leg"
+ words = name.split()
+ name = " ".join(w.capitalize() if w[0].islower() and len(w) > 1 else w for w in words)
+
+ name = re.sub(r'^Spine (Left|Right) Wing$', r'\1 Wing', name)
+ name = re.sub(r'\s+\d+$', '', name)
+
+ if not name:
+ return "Bone"
+ return name
+
+
+# ---------------------------------------------------------------------------
+# Coverage reporting
+# ---------------------------------------------------------------------------
+
+_UNMAPPED_PATTERN = re.compile(r'[_\d]|[a-z]{2,}[A-Z]')
+
+
+def _looks_unmapped(cleaned):
+ """Heuristic: residual underscores, digits, or CamelCase runs suggest
+ the name fell through the rules without hitting a canonical mapping."""
+ return bool(_UNMAPPED_PATTERN.search(cleaned))
+
+
+def report_coverage(clean_data):
+ """Per-rig conformance to the shared canonical label format
+ (:func:`vocab.is_canonical_label`); 'Bone' counts as conforming."""
+ from data_process.joint_annotation.vocab import is_canonical_label
+ total = 0
+ unmapped = 0
+ print("\n=== Canonical-format report ===")
+ for animal, labels in clean_data.items():
+ bad = [l for l in labels
+ if l != "Bone" and not is_canonical_label(l)]
+ total += len(labels)
+ unmapped += len(bad)
+ if bad:
+ preview = ", ".join(sorted(set(bad))[:5])
+ print(" {:20s} {:3d}/{:3d} residual: {}".format(
+ animal, len(bad), len(labels), preview))
+ pct = 100.0 * (1 - unmapped / max(total, 1))
+ print("Overall: {}/{} labels clean ({:.1f}%)".format(
+ total - unmapped, total, pct))
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def main():
+ parser = argparse.ArgumentParser(description="Rule-based joint-name cleaning.")
+ parser.add_argument("--input", required=True,
+ help="Path to the raw joint_names.json")
+ parser.add_argument("--output", default=None,
+ help="Output path (default: sibling 'clean_joint_names.json')")
+ parser.add_argument("--report", action="store_true",
+ help="Print a per-rig coverage report")
+ args = parser.parse_args()
+
+ input_path = args.input
+ output_path = args.output or os.path.join(
+ os.path.dirname(input_path), "clean_joint_names.json")
+
+ with open(input_path, "r") as f:
+ data = json.load(f)
+
+ clean_data = {}
+ for animal, joints in data.items():
+ clean_data[animal] = [post_process(clean_joint_name(j, animal)) for j in joints]
+
+ with open(output_path, "w") as f:
+ json.dump(clean_data, f, indent=2)
+
+ print("Written {} entries to {}".format(len(clean_data), output_path))
+
+ for animal in list(data.keys())[:5]:
+ print("\n--- {} ---".format(animal))
+ for raw, clean in zip(data[animal], clean_data[animal]):
+ print(" {:50s} -> {}".format(raw, clean))
+
+ if args.report:
+ report_coverage(clean_data)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/joint_annotation/names_correct_llm.py b/data_process/joint_annotation/names_correct_llm.py
new file mode 100644
index 0000000..578db7b
--- /dev/null
+++ b/data_process/joint_annotation/names_correct_llm.py
@@ -0,0 +1,379 @@
+"""LLM check-and-correct pass over ``clean_joint_names.json``.
+
+Feeds the LLM BOTH the raw joint name and the current cleaned label for
+every joint, and asks it to either keep the current label or replace it with
+a corrected canonical one. Useful for:
+
+ - correcting rigs that fell back to the rule-based cleaner (digits,
+ missing side prefixes, ``'Bone'`` for tokens the rules don't recognise),
+ - sweeping the whole dataset for residual vocabulary / rule violations.
+
+Inputs:
+ --raw raw ``joint_names.json``.
+ --cleaned current ``clean_joint_names.json``.
+ --failed_list OPTIONAL text file with one rig id per line (e.g. the
+ ``failed_clean_names.txt`` written by ``names_clean_llm``). If
+ omitted, EVERY rig in ``--raw`` is checked and corrected.
+
+Output: the script WRITES BACK IN-PLACE to ``--cleaned``. A ``.bak`` sibling
+is created on first run so the original is always recoverable.
+
+Usage:
+ python -m data_process.joint_annotation.names_correct_llm \\
+ --raw <export_dir>/joint_names.json \\
+ --cleaned <export_dir>/clean_joint_names.json [--failed_list failed_clean_names.txt]
+"""
+
+import argparse
+import json
+import os
+import shutil
+import sys
+import time
+
+from loguru import logger
+from tqdm import tqdm
+
+_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if _REPO_ROOT not in sys.path:
+ sys.path.insert(0, _REPO_ROOT)
+
+from data_process.joint_annotation.names_clean_llm import ( # noqa: E402
+ SYSTEM_PROMPT,
+ parse_response,
+ rule_based_clean,
+)
+from data_process.joint_annotation.llm import ( # noqa: E402
+ FatalLLMError,
+ LLMClient,
+ MAX_RETRIES,
+ RigTimeout,
+ add_llm_args,
+ append_id_list,
+ install_rig_alarm,
+ read_id_list,
+ save_json,
+ uninstall_rig_alarm,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Correction-mode system prompt addendum
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+CORRECTION_PREAMBLE = (
+ "=========================================================================\n"
+ "CHECK & CORRECT MODE โ€” INPUT/OUTPUT SHAPE OVERRIDES THE ABOVE.\n"
+ "=========================================================================\n"
+ "INPUT change: instead of a plain JSON array of raw names you receive a "
+ "JSON array of objects, one per joint:\n"
+ " {\"raw\": \"<raw joint name>\", \"current\": \"<current cleaned label>\"}\n"
+ "The ``current`` label is whatever is already stored in "
+ "clean_joint_names.json โ€” it may be hand-curated, LLM-cleaned, or a "
+ "rule-based fallback. Treat it as a candidate, not ground truth.\n\n"
+ "PROCESS EACH JOINT IN TWO STEPS.\n\n"
+ "STEP 1 โ€” VALIDATE the current label against ALL cleaning rules above. "
+ "The label is VALID only if every check below passes:\n"
+ " (a) It appears verbatim in the CANONICAL VOCABULARY (exact term, "
+ "Title Case, single spaces).\n"
+ " (b) It contains NO digits, underscores, dots, or Blender '.NNN' "
+ "counters โ€” rule 1.\n"
+ " (c) It carries NO leftover namespace (mixamorig:, Mutant:, Sif:, ...) "
+ "or rig/Maya prefix/suffix (Bip01_, BN_, NPC_, jt_, DEF-, _jnt, _C, "
+ "...) โ€” rules 2-4.\n"
+ " (d) If the raw name encodes a side (L_/R_, Lt_/Rt_, Left<Word>, "
+ "'Bip001 L/R ', trailing L/R on Japanese roots, F_/B_ for quadrupeds), "
+ "the label starts with 'Left '/'Right ' (and includes the Front/Back "
+ "marker between side and part when applicable) โ€” rule 5.\n"
+ " (e) If the raw name clearly encodes a recognisable body part, the "
+ "label is NOT the 'Bone'/'Appendage' fallback โ€” rules 6-9.\n"
+ " (f) Finger roots, mixamo chain bones (UpLeg/Leg/Foot โ†’ Thigh/Shin/"
+ "Foot), 3ds Max Biped finger codes (Finger0*=Thumb, 1*=Index, ...), "
+ "Japanese roots, and 'Hair*/Ponytail*/Cape*/Skirt*' โ†’ 'Appendage' are "
+ "all resolved correctly โ€” rules 6-7.\n\n"
+ "STEP 2 โ€” DECIDE:\n"
+ " - If VALID on every check in Step 1: OUTPUT THE CURRENT LABEL "
+ "VERBATIM (skip โ€” do not rewrite).\n"
+ " - If INVALID on any check: IGNORE the current label and apply the "
+ "cleaning rules above to the raw name from scratch; output the "
+ "CORRECTED canonical label.\n\n"
+ "OUTPUT unchanged (identical to the clean-mode contract): a single FLAT "
+ "JSON array of STRINGS โ€” one label per joint, same order, same length "
+ "as the input array. Critical constraints:\n"
+ " - Output is an array of strings, NOT objects. Do NOT echo 'raw' or "
+ "'current' keys. Do NOT wrap each entry in an object.\n"
+ " - Same length as input (number of objects in). One input object -> "
+ "one output string.\n"
+ " - No prose, no markdown fences, no comments, no trailing text.\n"
+ " - Never dedupe, merge, split, reorder or skip โ€” identical neighbouring "
+ "outputs (e.g. 'Spine', 'Spine', 'Spine') are CORRECT.\n"
+ " - Title Case with single spaces. Canonical vocabulary terms only.\n"
+ " - Side first ('Left Upper Arm'); quadruped marker between side and "
+ "part ('Right Front Shoulder').\n\n"
+ "CHECK/CORRECT EXAMPLES (current -> final, with verdict):\n"
+ " raw='Spine', current='Spine' -> 'Spine' (VALID, skip)\n"
+ " raw='mixamorig:LeftUpLeg', current='Left Thigh' -> 'Left Thigh' (VALID, skip)\n"
+ " raw='ToeBase.L', current='Left Toe' -> 'Left Toe' (VALID, skip)\n"
+ " raw='mixamorig:LeftLeg', current='Left Leg' -> 'Left Shin' (INVALID: mixamo mid-bone)\n"
+ " raw='mixamorig:Spine_02', current='Spine 02' -> 'Spine' (INVALID: digits)\n"
+ " raw='Bip01_L_Thigh_07', current='Bip01 L Thigh' -> 'Left Thigh' (INVALID: prefix + digits)\n"
+ " raw='QuickRigCharacter_LeftForeArm_014',\n"
+ " current='Bone' -> 'Left Forearm' (INVALID: lost anatomy)\n"
+ " raw='Bone.001_01', current='Bone 001 01' -> 'Bone' (INVALID: digits)\n"
+ " raw='munabireL', current='MunabireL' -> 'Left Pectoral Fin' (INVALID: non-canonical)\n"
+ " raw='Hair_03', current='Hair' -> 'Mane' (INVALID: animal Hair->Mane)\n"
+ " raw='Reins_02', current='Reins' -> 'Reins' (VALID: equipment label)\n"
+ " raw='LegTip_R', current='Right Leg Tip' -> 'Right Leg End' (INVALID: Tip not canonical)\n"
+)
+
+CORRECTION_SYSTEM_PROMPT = SYSTEM_PROMPT + "\n\n" + CORRECTION_PREAMBLE
+
+
+def build_correction_prompt(rig_id, raw_names, current_labels, correction=None):
+ # type: (str, List[str], List[str], Optional[str]) -> str
+ pairs = [{"raw": raw, "current": current}
+ for raw, current in zip(raw_names, current_labels)]
+ prompt = (
+ "Rig identifier: {rig}\n"
+ "Joint count: {n}\n\n"
+ "Joints (JSON array of {{raw, current}} objects, order matters):\n"
+ "{pairs}\n\n"
+ "For each joint: validate the 'current' label against the cleaning "
+ "rules. If VALID, output it verbatim; if INVALID, output a corrected "
+ "canonical label. Return a FLAT JSON array of EXACTLY {n} STRINGS, "
+ "same order as the input. Do NOT output objects; strings only."
+ ).format(rig=rig_id, n=len(raw_names),
+ pairs=json.dumps(pairs, ensure_ascii=False, indent=2))
+ if correction:
+ prompt += "\n\nIMPORTANT: " + correction
+ return prompt
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-chunk + per-rig corrector (with retry + keep-rule-based fallback)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+#
+# Long rigs (50+ joints) reliably break the "emit EXACTLY N entries" contract
+# on both Qwen3 and GPT-5 โ€” the LLM drifts by ยฑ1-2 when near-duplicate names
+# sit next to each other (Spine/Spine1/Spine2 โ†’ "Spine" ร— 3). Shrinking the
+# per-call enumeration fixes it: at chunk_size โ‰ค 16 the count stays exact.
+# The per-rig SIGALRM budget wraps the whole chunk sequence; a timeout keeps
+# already-corrected chunks and leaves the rest as rule-based.
+
+def _run_chunk(rig_id, chunk_raw, chunk_current, client, system_prompt,
+ max_tokens, max_retries, chunk_tag=""):
+ # type: (...) -> Optional[List[str]]
+ """Retry loop for a single chunk. Returns the parsed list on success, or
+ ``None`` after exhausting retries. Lets ``RigTimeout`` propagate so the
+ caller can stop early and preserve already-finished chunks."""
+ correction = None
+ last_err = None
+ for attempt in range(1, max_retries + 1):
+ try:
+ user_prompt = build_correction_prompt(
+ rig_id, chunk_raw, chunk_current, correction=correction)
+ raw_text = client.generate(system_prompt, user_prompt, max_tokens)
+ return parse_response(raw_text, len(chunk_raw))
+ except (RigTimeout, FatalLLMError):
+ raise
+ except Exception as err: # noqa: BLE001 โ€” retry on any backend/parse error
+ last_err = err
+ if isinstance(err, ValueError):
+ correction = (
+ "Your previous response failed validation ({}). Return a "
+ "JSON array of EXACTLY {} strings โ€” one corrected label "
+ "per raw joint, same order. No prose, no code fences."
+ ).format(err, len(chunk_raw))
+ logger.warning("[{}{}] attempt {}/{} failed: {}".format(
+ rig_id, chunk_tag, attempt, max_retries, err))
+ if attempt < max_retries:
+ time.sleep(1.5)
+ logger.error("[{}{}] chunk failed after {} attempts ({})".format(
+ rig_id, chunk_tag, max_retries, last_err))
+ return None
+
+
+def correct_rig(rig_id, raw_names, current_labels, client,
+ system_prompt=CORRECTION_SYSTEM_PROMPT, max_tokens=4096,
+ max_retries=MAX_RETRIES, timeout_seconds=15, chunk_size=16):
+ # type: (...) -> Tuple[List[str], bool]
+ """Correct a single rig via the LLM. Returns ``(labels, still_failed)``.
+
+ Rigs longer than ``chunk_size`` are split into consecutive chunks; each
+ chunk is an independent LLM call with its own retries. ``timeout_seconds``
+ is the total SIGALRM budget for the whole rig; successfully-corrected
+ chunks stay corrected even if a later chunk times out.
+ """
+ if not raw_names:
+ return [], False
+
+ n = len(raw_names)
+ result = list(current_labels) # failed / timed-out chunks keep their labels
+ chunk_size = max(1, int(chunk_size))
+ n_chunks = (n + chunk_size - 1) // chunk_size
+ failed_chunks = 0
+ timed_out = False
+
+ prev_handler = install_rig_alarm(timeout_seconds)
+ try:
+ for i in range(n_chunks):
+ start = i * chunk_size
+ end = min(start + chunk_size, n)
+ chunk_tag = (" [{}/{} {}..{}]".format(i + 1, n_chunks, start, end - 1)
+ if n_chunks > 1 else "")
+ chunk_out = _run_chunk(
+ rig_id, raw_names[start:end], current_labels[start:end],
+ client, system_prompt, max_tokens, max_retries, chunk_tag=chunk_tag)
+ if chunk_out is not None:
+ result[start:end] = chunk_out
+ else:
+ failed_chunks += 1
+ except RigTimeout:
+ timed_out = True
+ finally:
+ uninstall_rig_alarm(prev_handler)
+
+ if timed_out:
+ logger.warning("[{}] exceeded {}s budget; keeping current labels for "
+ "unfinished chunks".format(rig_id, timeout_seconds))
+ elif failed_chunks:
+ logger.warning("[{}] {}/{} chunks still failed; keeping current labels "
+ "for those".format(rig_id, failed_chunks, n_chunks))
+
+ return result, timed_out or failed_chunks > 0
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Driver
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def correct_all(raw_path, cleaned_path, client, system_prompt=CORRECTION_SYSTEM_PROMPT,
+ failed_list_path=None, max_tokens=4096, max_retries=MAX_RETRIES,
+ timeout_seconds=15, chunk_size=16, still_failed_path=None):
+ # type: (...) -> Dict[str, List[str]]
+ """Check + correct joint labels in ``cleaned_path`` via the LLM.
+
+ When ``failed_list_path`` is provided, only rigs listed there are
+ processed; otherwise every rig in ``raw_path``. The result is written
+ BACK IN-PLACE to ``cleaned_path`` (a ``.bak`` sibling is created on first
+ run). Rigs the LLM still couldn't fully correct are appended to
+ ``still_failed_path``.
+
+ A missing ``failed_list_path`` is a hard error (a typo must not silently
+ escalate a targeted repair into a full sweep), and so is a missing
+ ``cleaned_path`` in that mode โ€” the run would otherwise replace the whole
+ cleaned file with one holding ONLY the listed rigs, and with no ``.bak``.
+ """
+ with open(raw_path, "r") as f:
+ raw_data = json.load(f)
+
+ if failed_list_path:
+ if not os.path.exists(failed_list_path):
+ raise FileNotFoundError(
+ "--failed_list {!r} does not exist. Drop the flag to correct "
+ "every rig in {}.".format(failed_list_path, raw_path))
+ if not os.path.exists(cleaned_path):
+ raise FileNotFoundError(
+ "--failed_list restricts this run to a subset of rigs, but "
+ "--cleaned {!r} does not exist: the run would write a cleaned "
+ "file containing ONLY the listed rigs (and make no .bak). Run "
+ "the joint-name cleaner first.".format(cleaned_path))
+ target_ids = read_id_list(failed_list_path)
+ scope_desc = "{} rigs from {}".format(len(target_ids), failed_list_path)
+ else:
+ target_ids = sorted(raw_data.keys())
+ scope_desc = "all {} rigs in {}".format(len(target_ids), raw_path)
+
+ if os.path.exists(cleaned_path):
+ with open(cleaned_path, "r") as f:
+ cleaned_data = json.load(f)
+ else:
+ cleaned_data = {}
+ logger.info("No existing {}; starting from rule-based labels.".format(cleaned_path))
+
+ backup = cleaned_path + ".bak"
+ if os.path.exists(cleaned_path) and not os.path.exists(backup):
+ shutil.copy2(cleaned_path, backup)
+ logger.info("Backed up {} -> {}".format(cleaned_path, backup))
+
+ if still_failed_path is None:
+ still_failed_path = os.path.join(
+ os.path.dirname(cleaned_path) or ".", "still_failed_clean_names.txt")
+
+ logger.info("Correcting {} (in-place) -> {}".format(scope_desc, cleaned_path))
+
+ still_failed = []
+ corrected_count = 0
+ pbar = tqdm(target_ids, desc="Correcting", unit="rig", dynamic_ncols=True)
+ for rig_id in pbar:
+ pbar.set_postfix_str(rig_id, refresh=False)
+ if rig_id not in raw_data:
+ logger.warning("[{}] missing from raw file; skipping".format(rig_id))
+ continue
+ raw_names = raw_data[rig_id]
+ current = cleaned_data.get(rig_id)
+ if not isinstance(current, list) or len(current) != len(raw_names):
+ current = rule_based_clean(rig_id, raw_names)
+
+ new_labels, still_bad = correct_rig(
+ rig_id, raw_names, current, client,
+ system_prompt=system_prompt, max_tokens=max_tokens,
+ max_retries=max_retries, timeout_seconds=timeout_seconds,
+ chunk_size=chunk_size,
+ )
+ cleaned_data[rig_id] = new_labels
+ if still_bad:
+ still_failed.append(rig_id)
+ else:
+ corrected_count += 1
+ save_json(cleaned_data, cleaned_path)
+
+ if still_failed:
+ append_id_list(still_failed_path, still_failed)
+ logger.info("{} rigs still failed; appended to {}".format(
+ len(still_failed), still_failed_path))
+
+ logger.info("Done. Corrected {}/{} rigs (still failed: {}).".format(
+ corrected_count, len(target_ids), len(still_failed)))
+ return cleaned_data
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="LLM check-and-correct pass over clean_joint_names.json (in place).")
+ parser.add_argument("--raw", required=True, help="Raw joint_names.json.")
+ parser.add_argument("--cleaned", required=True,
+ help="clean_joint_names.json โ€” updated IN PLACE; a '.bak' "
+ "sibling is made on first run.")
+ parser.add_argument("--failed_list", default=None,
+ help="Optional txt file with one rig id per line. When omitted, "
+ "ALL rigs in --raw are checked and corrected.")
+ parser.add_argument("--still_failed_log", default=None,
+ help="Append rig ids the LLM still couldn't fix (default: "
+ "sibling 'still_failed_clean_names.txt').")
+ parser.add_argument("--rig_timeout", type=int, default=15,
+ help="Per-rig wall-time budget in seconds (SIGALRM). 0 disables.")
+ parser.add_argument("--chunk_size", type=int, default=16,
+ help="Joints per LLM call. Smaller chunks avoid length drift on "
+ "long arrays; larger chunks save API calls.")
+ add_llm_args(parser, default_max_tokens=4096)
+ args = parser.parse_args()
+
+ client = LLMClient.from_args(args)
+ logger.info("{} | Writing in-place to: {}".format(client, args.cleaned))
+
+ correct_all(
+ raw_path=args.raw,
+ cleaned_path=args.cleaned,
+ client=client,
+ failed_list_path=args.failed_list,
+ max_tokens=args.max_tokens,
+ max_retries=args.max_retries,
+ timeout_seconds=args.rig_timeout,
+ chunk_size=args.chunk_size,
+ still_failed_path=args.still_failed_log,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/joint_annotation/vocab.py b/data_process/joint_annotation/vocab.py
new file mode 100644
index 0000000..b45880c
--- /dev/null
+++ b/data_process/joint_annotation/vocab.py
@@ -0,0 +1,439 @@
+"""Vocabulary data for Truebones / Mixamo joint-name cleaning.
+
+Kept separate from the cleaning logic so mapping edits don't require touching
+code. All maps are module-level constants; import them from the cleaner.
+"""
+
+# ---------------------------------------------------------------------------
+# Prefixes to strip (order matters: longest first)
+# ---------------------------------------------------------------------------
+REMOVE_PREFIXES = [
+ "BN_Bip01_", "BN_Bip01 ",
+ "Base Human ",
+ "Bip001_", "Bip001 ",
+ "Bip01_", "Bip01 ",
+ "Preset01_", "Preset01 ",
+ "Mutant:", "Sif:",
+ "BN_", "NPC_", "jt_",
+ "Bn_", "bn_", "b_",
+]
+
+
+# ---------------------------------------------------------------------------
+# Mixamo-specific mappings (humanoid rig, uses CamelCase without underscores)
+# ---------------------------------------------------------------------------
+MIXAMO_MAP = {
+ "Hips": "Hips",
+ "Spine": "Spine", "Spine1": "Spine", "Spine2": "Spine",
+ "Neck": "Neck",
+ "Head": "Head", "HeadTop_End": "Head End",
+ "Eye": "Eye",
+ "Shoulder": "Shoulder",
+ "Arm": "Upper Arm", "ForeArm": "Forearm", "Hand": "Hand",
+ "HandThumb": "Thumb Finger", "HandIndex": "Index Finger",
+ "HandMiddle": "Middle Finger", "HandRing": "Ring Finger",
+ "HandPinky": "Pinky Finger",
+ # Mixamo chain UpLeg -> Leg -> Foot: the mid bone "Leg" is the shin.
+ "UpLeg": "Thigh", "Leg": "Shin", "Foot": "Foot",
+ "ToeBase": "Toe", "Toe_End": "Toe End",
+}
+
+
+# ---------------------------------------------------------------------------
+# Japanese vocabulary (Alligator, Pirrana, Tukan rigs)
+# ---------------------------------------------------------------------------
+JAPANESE_WORDS = {
+ "momo": "Thigh", "sippo": "Tail", "mune": "Chest", "hiza": "Knee",
+ "hara": "Abdomen", "ashi": "Foot", "hiji": "Elbow", "koshi": "Hips",
+ "kosi": "Hips", "te": "Hand", "kubi": "Neck", "atama": "Head",
+ "ago": "Jaw", "kata": "Shoulder", "kao": "Head", "o": "Tail",
+ # Fish-specific (Pirrana)
+ "munabire": "Pectoral Fin", "era": "Gill", "obire": "Caudal Fin",
+ "sebire": "Dorsal Fin", "harabire": "Pelvic Fin",
+ "shiribire": "Anal Fin", "shippo": "Tail",
+}
+
+# Precomputed lowercase index for fast dispatch
+JAPANESE_WORDS_LOWER = {k.lower(): v for k, v in JAPANESE_WORDS.items()}
+
+
+# ---------------------------------------------------------------------------
+# Canonical anatomical vocabulary (applied after prefix/side stripping)
+# ---------------------------------------------------------------------------
+CANONICAL = {
+ # Body core
+ "Pelvis": "Pelvis", "pelv": "Pelvis",
+ "Spine": "Spine", "Spn": "Spine", "Spline": "Spine",
+ "Ribcage": "Ribcage",
+ "Neck": "Neck", "Nek": "Neck",
+ "Head": "Head", "Scull": "Skull", "Skull": "Skull",
+ "ScullBase": "Skull Base", "SkullBase": "Skull Base",
+ "HeadNub": "Head End", "Nub": "End", "Tip": "End",
+ "Hips": "Hips", "Cog": "Root",
+ # Limb upper
+ "Clavicle": "Shoulder", "Collarbone": "Shoulder",
+ "Scapula": "Scapula",
+ "UpperArm": "Upper Arm", "Upperarm": "Upper Arm",
+ "Humerus": "Upper Arm",
+ "Forearm": "Forearm", "ForeArm": "Forearm",
+ "LowerArm": "Forearm", "Lowerarm": "Forearm",
+ "Radius": "Forearm",
+ "Hand": "Hand", "Hnd": "Hand", "Palm": "Palm",
+ "Wrist": "Wrist", "Elbow": "Elbow", "Shoulder": "Shoulder",
+ # Limb lower
+ "Thigh": "Thigh", "Femur": "Thigh", "UpLeg": "Thigh",
+ "UpperLeg": "Thigh", "Upperleg": "Thigh",
+ "Calf": "Shin", "Tibia": "Shin", "Leg": "Leg",
+ "LowerLeg": "Shin", "Lowerleg": "Shin",
+ "HorseLink": "Fetlock", "LargeCannon": "Cannon",
+ "Metacarpus": "Metacarpus", "PhalangesManus": "Phalanges",
+ "PhalanxPrima": "Pastern",
+ "Foot": "Foot", "Ankle": "Ankle", "Heel": "Heel",
+ "Toe": "Toe", "Toes": "Toe", "ToeBase": "Toe", "Ball": "Toe",
+ "Hoof": "Hoof", "Knee": "Knee",
+ "Foreleg": "Front Leg", "Hindleg": "Hind Leg",
+ # Fingers
+ "Finger": "Finger", "Thumb": "Thumb Finger",
+ "Pinky": "Pinky Finger", "Little": "Pinky Finger",
+ "Ring": "Ring Finger",
+ "Middle": "Middle Finger", "Index": "Index Finger",
+ # Head features
+ "Jaw": "Jaw",
+ "Tongue": "Tongue", "Thouge": "Tongue", "Tone": "Tongue",
+ "tunge": "Tongue", "Tunge": "Tongue",
+ "Ear": "Ear",
+ "Eye": "Eye", "Eyeball": "Eyeball", "EyeBall": "Eyeball",
+ "Eyebrow": "Eyebrow", "Eyelid": "Eyelid",
+ "Eyeleds": "Eyelid",
+ "Mouth": "Mouth", "Lip": "Lip",
+ "Nose": "Nose", "Muzzle": "Muzzle",
+ "Chin": "Chin", "Cheek": "Cheek",
+ "Torso": "Body", "LegAnkle": "Ankle", "Fingers": "Finger",
+ # Appendages
+ "Tail": "Tail", "Tai": "Tail",
+ "Wing": "Wing", "RWing": "Right Wing", "LWing": "Left Wing",
+ "Feather": "Feather",
+ # Species-driven, NOT typos โ€” do not unify: Cricket "Feeler" bones are
+ # antennae, Catfish "Feelers" are barbels.
+ "Feeler": "Antenna", "Antenna": "Antenna", "Feelers": "Barbel",
+ "Tentacle": "Tentacle", "Tentacles": "Tentacle",
+ "Claw": "Claw", "HandClaw": "Hand Claw",
+ "Fang": "Fang", "Fangs": "Fang",
+ "Mandible": "Mandible", "BigMandible": "Large Mandible",
+ "LowerMandible": "Lower Mandible",
+ "Pincer": "Pincer", "pincers": "Pincer", "Pliers": "Pincer",
+ "Piers": "Pincer",
+ # Anatomy extras
+ "Mane": "Mane", "Fur": "Fur", "Hair": "Mane",
+ "Beard": "Whisker", "Mascara": "Whisker",
+ "Shell": "Shell", "Stinger": "Stinger", "Horn": "Horn",
+ "dorsal": "Dorsal Plate", "Fin": "Fin",
+ # Species-driven, NOT typos: Tukan "ponitail" is the bird's crest,
+ # humanoid "Ponytail" chains are generic appendages.
+ "ponitail": "Crest", "Ponytail": "Appendage",
+ "Downbody": "Lower Body", "Down": "Lower Body",
+ "Body": "Body",
+ # Equipment (horse)
+ "Reins": "Reins", "Halter": "Halter",
+ # Physics/IK
+ "Jiggle": "Jiggle", "TwistBone": "Twist",
+ "UpperArmTwist": "Upper Arm Twist", "ForearmTwist": "Forearm Twist",
+ "ThighMuscle": "Thigh Muscle", "NeckMuscle": "Neck Muscle",
+ # Insect
+ "Clip": "Mandible", "Wings": "Wing", "Shall": "Mandible",
+ # Misc
+ "Trajectory": "Root", "locator": "Root", "locator2": "Root",
+ "center": "Center",
+ "MagicEffectsNode": "Bone",
+ "Handle": "Handle", "IK_Chain": "IK Chain",
+ # Rig-noise tokens mapped to "" are DROPPED by the cleaners; a name made
+ # only of them collapses to the "Bone" placeholder ('joint16.001',
+ # 'internal50.L', 'l_foreleg_jnt01').
+ "Joint": "", "Jnt": "", "Internal": "", "Def": "",
+ "Part": "", "Armature": "",
+ "Mixamorig": "", "Character": "", "Rig": "", "Bip": "",
+ "Base": "", "Human": "", "Mid": "",
+ "Controller": "", "Controler": "", "Ctrl": "", "Quick": "",
+ "Untitled": "", "Pasted": "",
+ "Bind": "", "Skeleton": "", "Reference": "", "Res": "", "Main": "",
+ # UE-style per-finger segment words: chain position, not anatomy
+ # ('IndexDistal', 'thumb_proximal_l' -> the finger label alone).
+ "Metacarpal": "", "Proximal": "", "Intermediate": "", "Medial": "",
+ "Distal": "",
+ "Digit": "Finger",
+ "FrontLeg": "Front Leg", "MiddleLeg": "Middle Leg", "HindLeg": "Hind Leg",
+ "Crab_pincers": "Pincer",
+ "SpineR": "Right Abdomen", "SpineL": "Left Abdomen",
+}
+
+
+# ---------------------------------------------------------------------------
+# Standalone exact-match mappings (checked early in dispatch).
+# A few names (locator*, Trajectory, MagicEffectsNode) appear in CANONICAL
+# too: here they catch the whole-name form before any prefix/side stripping.
+# ---------------------------------------------------------------------------
+STANDALONE_MAP = {
+ "locator": "Root", "locator2": "Root",
+ "EyesBlue_2": "Eye",
+ "MagicEffectsNode": "Bone",
+ "Handle": "Handle", "IK_Chain01": "IK Chain",
+ "BN_P": "Belly",
+ "Hips": "Hips", "Spine": "Spine", "Head": "Head",
+ "Trajectory": "Root",
+ "RightArm": "Right Upper Arm", "RightForeArm": "Right Forearm",
+ "LeftArm": "Left Upper Arm", "LeftForeArm": "Left Forearm",
+ # Standalone mixamo-style chain: bare "Leg" is the shin (see MIXAMO_MAP).
+ "RightLeg": "Right Shin", "LeftLeg": "Left Shin",
+ "RightUpLeg": "Right Thigh", "LeftUpLeg": "Left Thigh",
+ "RightFoot": "Right Foot", "LeftFoot": "Left Foot",
+ "RightHand": "Right Hand", "LeftHand": "Left Hand",
+ "Tail01": "Tail", "Tail02": "Tail",
+}
+
+
+# ---------------------------------------------------------------------------
+# SabreToothTiger: full explicit mapping (abbreviation codes are too messy)
+# ---------------------------------------------------------------------------
+SABRECAT_MAP = {
+ "Sabrecat__pelv_": "Pelvis",
+ "Sabrecat_LeftThigh_LThi_": "Left Thigh",
+ "Sabrecat_LeftCalf_LClf_": "Left Shin",
+ "Sabrecat_LeftFoot_LFot_": "Left Foot",
+ "Sabrecat_LeftToe0_LT00_": "Left Toe",
+ "Sabrecat_LeftToe0_LT01_": "Left Toe",
+ "Sabrecat_RightThigh_RThi_": "Right Thigh",
+ "Sabrecat_RightCalf_RClf_": "Right Shin",
+ "Sabrecat_RightFoot_RFot_": "Right Foot",
+ "Sabrecat_RightToe0_RT00_": "Right Toe",
+ "Sabrecat_RightToe0_RT01_": "Right Toe",
+ "Sabrecat_Tail0_Tal0_": "Tail",
+ "Sabrecat_Tail1_Tal1_": "Tail",
+ "Sabrecat_Tail2_Tal2_": "Tail",
+ "Sabrecat_Spine_Spn0_": "Spine",
+ "Sabrecat_Spine_Spn1_": "Spine",
+ "MagicEffectsNode": "Bone",
+ "Sabrecat_Spine_Spn2_": "Spine",
+ "Sabrecat_Spine_Spn3_": "Spine",
+ "Sabrecat_Ribcage_Spn4_": "Ribcage",
+ "Sabrecat_Ribcage_Spn1_": "Ribcage",
+ "Sabrecat_Neck_Nek0_": "Neck",
+ "Sabrecat_Neck_Nek1_": "Neck",
+ "Sabrecat_Neck_Nek2_": "Neck",
+ "Sabrecat_Head__Head_": "Head",
+ "Sabrecat_Head__LEye_": "Left Eye",
+ "Sabrecat_Head__RChk_": "Right Cheek",
+ "Sabrecat_Head_Head__LChk_": "Left Cheek",
+ "Sabrecat_HeadLeftEar_LEar_": "Left Ear",
+ "Sabrecat_Head_EyeLid_HELT_": "Upper Eyelid",
+ "Sabrecat_Head__REye_": "Right Eye",
+ "Sabrecat_Head_jaw_": "Jaw",
+ "Sabrecat_HeadEyeLid__HELB_": "Lower Eyelid",
+ "Sabrecat_HeadRightEar_REar_": "Right Ear",
+ "Sabrecat_Head_LM01_": "Left Mouth",
+ "Sabrecat_Head_RM01_": "Right Mouth",
+ "Sabrecat_RightClavicle_RClv_": "Right Shoulder",
+ "Sabrecat_RightUpperArm_RUar_": "Right Upper Arm",
+ "Sabrecat_RightForearm_RFar_": "Right Forearm",
+ "Sabrecat_RightTwistBone_RFTB_": "Right Twist",
+ "Sabrecat_RightHand_RHnd_": "Right Hand",
+ "Sabrecat_RightFinger3_RF30_": "Right Finger",
+ "Sabrecat_RightFinger3_RF31_": "Right Finger",
+ "Sabrecat_RightFinger2_RF20_": "Right Finger",
+ "Sabrecat_RightFinger2_RF21_": "Right Finger",
+ "Sabrecat_Finger4_RF04_": "Right Finger",
+ "Sabrecat_RightFinger1_RF10_": "Right Finger",
+ "Sabrecat_RightFinger1_RF11_": "Right Finger",
+ "Sabrecat_RightFinger0_RF00_": "Right Finger",
+ "Sabrecat_RightFinger0_RF01_": "Right Finger",
+ "Sabrecat_LeftClavicle_LClv_": "Left Shoulder",
+ "Sabrecat_LeftUpperArm_LUar_": "Left Upper Arm",
+ "Sabrecat_LeftForearm_LFar_": "Left Forearm",
+ "Sabrecat_LeftTwistBone_LFTB_": "Left Twist",
+ "Sabrecat_LeftHand_LHnd_": "Left Hand",
+ "Sabrecat_Finger4_LF04_": "Left Finger",
+ "Sabrecat_LeftFinger1_LF10_": "Left Finger",
+ "Sabrecat_LeftFinger1_LF11_": "Left Finger",
+ "Sabrecat_LeftFinger2_LF20_": "Left Finger",
+ "Sabrecat_LeftFinger2_LF21_": "Left Finger",
+ "Sabrecat_LeftFinger3_RF30_": "Left Finger",
+ "Sabrecat_LeftFinger3_RF31_": "Left Finger",
+ "Sabrecat_LeftFinger0_LF00_": "Left Finger",
+ "Sabrecat_LeftFinger0_LF01_": "Left Finger",
+}
+
+
+# ---------------------------------------------------------------------------
+# Pirrana/Tukan compound Japanese names
+# ---------------------------------------------------------------------------
+PIRRANA_COMPOUNDS = {
+ "munabireR": "Right Pectoral Fin", "munabireL": "Left Pectoral Fin",
+ "eraR": "Right Gill", "eraL": "Left Gill",
+ "shippoA": "Tail", "shippoB": "Tail",
+ "shiribire": "Anal Fin", "shirihireB": "Anal Fin",
+ "shiribireA": "Anal Fin",
+ "obire": "Caudal Fin", "obireB": "Caudal Fin", "obireA": "Caudal Fin",
+ "sebire": "Dorsal Fin",
+ "harabireR": "Right Pelvic Fin", "harabireL": "Left Pelvic Fin",
+}
+
+# Raw names that route through the Japanese cleaner even when the lowercase
+# base isn't a direct JAPANESE_WORDS hit (Pirrana/Tukan/Alligator compounds).
+JAPANESE_COMPOUND_NAMES = frozenset({
+ "munabireR", "munabireL", "eraR", "eraL", "shippoA", "shippoB",
+ "shiribire", "shirihireB", "shiribireA", "obire", "obireB", "obireA",
+ "sebire", "harabireR", "harabireL", "locator", "locator2",
+ "kosi", "kao",
+})
+JAPANESE_COMPOUND_ANIMALS = frozenset({"Pirrana", "Tukan", "Alligator"})
+
+
+# ---------------------------------------------------------------------------
+# NPC_ prefix (Bear / Skyrim-style)
+# ---------------------------------------------------------------------------
+NPC_DIRECT = {
+ "Pelvis": "Pelvis",
+ "Ribcage": "Ribcage",
+ "Spine1": "Spine", "Spine2": "Spine", "Spine3": "Spine", "Spine4": "Spine",
+ "Neck1": "Neck", "Neck2": "Neck",
+ "Head": "Head", "Jaw": "Jaw", "Nose": "Nose",
+ "UpperRightLip": "Upper Right Lip", "UpperLeftLip": "Upper Left Lip",
+ "UpperLip": "Upper Lip",
+ "LowerLeftLip": "Lower Left Lip", "LowerFrontLip": "Lower Front Lip",
+ "LowerRightLip": "Lower Right Lip",
+ "Eyebrow": "Eyebrow",
+ "Leg1": "Thigh", "Leg2": "Shin", "LegAnkle": "Ankle",
+ "LegBall1": "Foot",
+ "Toe": "Toe",
+ "Arm1": "Upper Arm", "Arm2": "Forearm",
+ "ArmCollarbone": "Shoulder",
+ "ArmPalm": "Hand", "ArmBall1": "Foot",
+ "Arm1_UpperArmTwist1": "Upper Arm Twist",
+ "Arm1_UpperArmTwist2": "Upper Arm Twist",
+ "Arm2_ForearmTwist1": "Forearm Twist",
+ "Arm2_ForearmTwist2": "Forearm Twist",
+ "Pinky01": "Pinky Finger", "Pinky02": "Pinky Finger",
+ "Ring01": "Ring Finger", "Ring02": "Ring Finger",
+ "Middle01": "Middle Finger", "Middle02": "Middle Finger",
+ "Thumb01": "Thumb Finger", "Thumb02": "Thumb Finger",
+ "Index01": "Index Finger", "Index02": "Index Finger",
+}
+
+
+# ---------------------------------------------------------------------------
+# Elk prefix (Deer)
+# ---------------------------------------------------------------------------
+ELK_MAP = {
+ "Femur": "Thigh", "Tibia": "Shin",
+ "LargeCannon": "Cannon", "PhalanxPrima": "Pastern",
+ "RearHoof": "Rear Hoof", "FrontHoof": "Front Hoof",
+ "Scapula": "Scapula", "Humerus": "Upper Arm",
+ "Radius": "Forearm", "Metacarpus": "Metacarpus",
+ "PhalangesManus": "Phalanges",
+ "Spine1": "Spine", "Spine2": "Spine", "Spine3": "Spine",
+ "Ribcage": "Ribcage",
+ "Neck1": "Neck", "Neck2": "Neck", "Neck3": "Neck", "Neck4": "Neck",
+ "ScullBase": "Skull Base", "Scull": "Skull",
+ "Ear": "Ear", "REar": "Right Ear", "LEar": "Left Ear",
+ "Jaw": "Jaw", "UpperLip": "Upper Lip",
+ "Pelvis": "Pelvis", "Tail1": "Tail", "Tail2": "Tail",
+}
+
+
+# ---------------------------------------------------------------------------
+# jt_ prefix (Pteranodon, Raptor2/3, Trex, Scorpion-2)
+# ---------------------------------------------------------------------------
+JT_MAP = {
+ "Cog": "Root", "Spine1": "Spine", "Spine2": "Spine",
+ "Hips": "Hips", "Hip": "Hip",
+ "Thigh": "Thigh", "Knee": "Knee", "Ankle": "Ankle", "Foot": "Foot",
+ "ToeMiddle": "Middle Toe", "ToeInner": "Inner Toe", "ToeOutter": "Outer Toe",
+ "ClawMiddle": "Middle Claw", "ClawInner": "Inner Claw",
+ "ClawOutter": "Outer Claw", "ClawBack": "Back Claw",
+ "ClawMiddle2": "Middle Claw", "ClawInner2": "Inner Claw",
+ "ClawOutter2": "Outer Claw", "ClawBack2": "Back Claw",
+ "Neck1": "Neck", "Neck2": "Neck", "Neck3": "Neck",
+ "Head": "Head", "Jaw": "Jaw",
+ "Tongue1": "Tongue", "Tongue2": "Tongue",
+ "Eye": "Eye", "EyeBall": "Eyeball",
+ "Shoulder": "Shoulder", "Elbow": "Elbow", "Wrist": "Wrist",
+ "WristBack": "Wrist Back", "ElbowBack": "Elbow Back",
+ "Clavicle": "Shoulder",
+ "FingerMiddle": "Middle Finger", "FingerInner": "Inner Finger",
+ "FingerOutter": "Outer Finger",
+ "HandClawMiddle": "Middle Hand Claw", "HandClawInner": "Inner Hand Claw",
+ "HandClawOutter": "Outer Hand Claw",
+ "Tail1": "Tail", "Tail2": "Tail", "Tail3": "Tail",
+ "Tail4": "Tail", "Tail5": "Tail", "Tail6": "Tail",
+ "ThighMuscle": "Thigh Muscle", "NeckMuscle": "Neck Muscle",
+ "Tail01": "Tail", "Tail02": "Tail", "Tail03": "Tail",
+ "Tail04": "Tail", "Tail05": "Tail", "Tail06": "Tail",
+ "Tail07": "Tail", "Tail08": "Tail", "Tail09": "Tail",
+ "Tail01x": "Tail Twist", "Tail02x": "Tail Twist",
+ "Tail03x": "Tail Twist", "Tail04x": "Tail Twist",
+ "Tail05x": "Tail Twist", "Tail06x": "Tail Twist",
+ "Tail07x": "Tail Twist", "Tail08x": "Tail Twist",
+ "FrontLeg1": "Front Leg", "FrontLeg2": "Front Leg",
+ "FrontLeg3": "Front Leg", "FrontLeg4End": "Front Leg End",
+ "MiddleLeg1": "Middle Leg", "MiddleLeg2": "Middle Leg",
+ "MiddleLeg3": "Middle Leg", "MiddleLeg4End": "Middle Leg End",
+ "HindLeg1": "Hind Leg", "HindLeg2": "Hind Leg",
+ "HindLeg3": "Hind Leg", "HindLeg4End": "Hind Leg End",
+ "BigMandible": "Large Mandible", "BigMandibleMid": "Large Mandible",
+ "LowerMandible": "Lower Mandible", "LowerMandibleMid": "Lower Mandible",
+ "Fangs": "Fang", "FangsMid": "Fang",
+}
+
+
+# ---------------------------------------------------------------------------
+# Spider skeleton (unique naming)
+# ---------------------------------------------------------------------------
+SPIDER_MAP = {
+ "_body_": "Body",
+ "NPC_L_MagicNode__LMag_": "Bone",
+ "ArmRCollarbone": "Right Shoulder",
+ "ArmR_01_": "Right Arm", "ArmR_02_": "Right Arm",
+ "ArmRClaw": "Right Claw",
+ "ArmLCollarbone": "Left Shoulder",
+ "ArmL_01_": "Left Arm", "ArmL_02_": "Left Arm",
+ "ArmLClaw": "Left Claw",
+ "Tail1": "Tail", "Tail2": "Tail", "Tail3": "Tail",
+ "R_Jaw_": "Right Jaw", "L_Jaw_": "Left Jaw",
+}
+
+
+# ---------------------------------------------------------------------------
+# Canonical label format โ€” ONE format shared by truebones / mixamo / objaverse
+#
+# label := [Left |Right ] <part> [ End]
+#
+# where <part> is any base label the maps above emit (qualifiers like
+# Front/Hind/Inner/Middle/Outer/Upper/Lower are already baked into those
+# values: 'Front Leg', 'Upper Eyelid', ...) and ' End' marks a chain tip
+# (Nub/Tip/*_End sources). Two sentinels sit outside the grammar:
+# - "Bone" placeholder for rig noise with no recognizable anatomy
+# - "_00", "12" numeric passthrough for rigs whose bones were never named
+# ---------------------------------------------------------------------------
+
+def canonical_parts():
+ """Base part labels (side stripped) that the vocabulary can emit."""
+ import re as _re
+ parts = set()
+ for mapping in (CANONICAL, MIXAMO_MAP, NPC_DIRECT, ELK_MAP, JT_MAP,
+ SPIDER_MAP, STANDALONE_MAP, SABRECAT_MAP,
+ PIRRANA_COMPOUNDS, JAPANESE_WORDS):
+ for value in mapping.values():
+ if value:
+ parts.add(_re.sub(r'^(Left|Right)\s+', '', value))
+ return parts
+
+
+def is_canonical_label(label, _parts_cache=[]):
+ """True iff *label* conforms to the shared canonical format."""
+ import re as _re
+ if not _parts_cache:
+ _parts_cache.append(canonical_parts())
+ if _re.match(r'^_?\d+$', label): # numeric passthrough
+ return True
+ base = _re.sub(r'^(Left|Right)\s+', '', label)
+ base = _re.sub(r'\s+End$', '', base)
+ return base in _parts_cache[0]
diff --git a/data_process/mesh_animation/__init__.py b/data_process/mesh_animation/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/mesh_animation/animate_fbx.py b/data_process/mesh_animation/animate_fbx.py
new file mode 100644
index 0000000..53d26b6
--- /dev/null
+++ b/data_process/mesh_animation/animate_fbx.py
@@ -0,0 +1,184 @@
+"""Bake raw animation FBX/GLB clips onto a chosen rigged character.
+
+Unlike :mod:`animate_npz` (which replays an export-stage NPZ), this works
+straight from animation files: each clip's action is moved onto the
+character's armature by bone name โ€” the Mixamo convention, where every clip
+is authored on the shared ``mixamorig:`` skeleton so any compatible
+character can play it without remapping. Bones the character lacks are left
+undriven; bones the clip does not touch keep their rest pose.
+
+``--anim_path`` accepts a single clip or a directory of clips, so one
+command can bake a whole animation library onto any character. Per-file
+failures are logged to ``animate_errors.log`` inside the output dir.
+
+Usage (Blender headless):
+ blender -b -P data_process/mesh_animation/animate_fbx.py -- \\
+ --char_path dataset/raw/mixamo/character_refined/Amy.fbx \\
+ --anim_path dataset/raw/mixamo/animation_motion \\
+ --output_dir outputs/animated_Amy \\
+ --char_anim_type fbx
+"""
+
+import argparse
+import os
+import re
+import sys
+
+import bpy
+from loguru import logger
+from tqdm import tqdm
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import reset_scene # noqa: E402
+from data_process.utils.blender_rig import ( # noqa: E402
+ RECONSTRUCTED_ACTION_NAME,
+ export_animated_character,
+ get_armature_obj,
+ load_file,
+ repair_and_pack_textures,
+ update_scene,
+)
+
+_BONE_PATH_RE = re.compile(r'^pose\.bones\["(.+?)"\]')
+
+
+def _action_bone_names(action):
+ """Bone names an action's F-curves drive."""
+ names = set()
+ for fcurve in action.fcurves:
+ match = _BONE_PATH_RE.match(fcurve.data_path)
+ if match:
+ names.add(match.group(1))
+ return names
+
+
+def _adopt_action(char_armature, action):
+ """Assign *action* to the character armature and mark it for export.
+
+ ``export_animated_character`` keeps only the action named
+ ``RECONSTRUCTED_ACTION_NAME``, so the transferred action takes that name
+ (after removing any stale holder of it).
+ """
+ for stale in list(bpy.data.actions):
+ if stale is not action and stale.name == RECONSTRUCTED_ACTION_NAME:
+ bpy.data.actions.remove(stale)
+ action.name = RECONSTRUCTED_ACTION_NAME
+
+ # The importer keys the animation armature's *object* transform (unit
+ # scale, axis correction) into the same action; the character must keep
+ # its own object transform, so only bone channels come along.
+ for fcurve in [f for f in action.fcurves
+ if not f.data_path.startswith('pose.bones')]:
+ action.fcurves.remove(fcurve)
+
+ if char_armature.animation_data is None:
+ char_armature.animation_data_create()
+ char_armature.animation_data.action = action
+ # Blender >= 4.4 slotted actions: bind the (single) slot the importer made.
+ anim_data = char_armature.animation_data
+ if getattr(anim_data, 'action_slot', True) is None and action.slots:
+ anim_data.action_slot = action.slots[0]
+
+
+def animate_character_fbx(char_path, anim_path, output_dir, char_anim_type='glb',
+ fps=30, skip_if_exists=False):
+ """Transfer one clip's action onto the character and export GLB/FBX.
+
+ Returns the fraction of the clip's driven bones that exist on the
+ character (1.0 = every animated bone landed).
+ """
+ anim_base = os.path.splitext(os.path.basename(anim_path))[0]
+ output_path = os.path.join(output_dir, f"{anim_base}.{char_anim_type}")
+ if skip_if_exists and os.path.exists(output_path):
+ logger.info(f"Output already exists, skipping: {output_path}")
+ return 1.0
+ os.makedirs(output_dir, exist_ok=True)
+
+ reset_scene()
+ char_armature = get_armature_obj(load_file(char_path))
+ assert char_armature is not None, "No armature found in character file"
+ repair_and_pack_textures(char_path)
+
+ anim_objs = load_file(anim_path)
+ anim_armature = get_armature_obj(anim_objs)
+ assert anim_armature is not None, "No armature found in animation file"
+ assert anim_armature.animation_data and anim_armature.animation_data.action, \
+ "Animation file carries no action"
+ action = anim_armature.animation_data.action
+
+ anim_bones = _action_bone_names(action)
+ char_bones = {b.name for b in char_armature.data.bones}
+ coverage = len(anim_bones & char_bones) / len(anim_bones) if anim_bones else 0.0
+ if coverage < 0.5:
+ logger.warning(
+ f"{anim_base}: only {coverage:.0%} of the clip's bones exist on the "
+ f"character โ€” bone naming likely differs; the result will barely move.")
+
+ _adopt_action(char_armature, action)
+ for obj in anim_objs:
+ bpy.data.objects.remove(obj, do_unlink=True)
+
+ frame_start, frame_end = action.frame_range
+ scene = bpy.context.scene
+ scene.render.fps = int(fps)
+ scene.frame_start = int(round(frame_start))
+ scene.frame_end = int(round(frame_end))
+ update_scene()
+
+ export_animated_character(os.path.splitext(output_path)[0],
+ formats=(char_anim_type,))
+ return coverage
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Bake raw animation FBX/GLB clips onto a rigged character.")
+ parser.add_argument("--char_path", type=str, required=True,
+ help="Rigged character FBX/GLB (mesh + armature).")
+ parser.add_argument("--anim_path", type=str, required=True,
+ help="Animation clip, or a directory of clips, whose "
+ "bone names match the character's rig.")
+ parser.add_argument("--output_dir", type=str, default='outputs/animated',
+ help="Destination directory, one file per clip.")
+ parser.add_argument("--char_anim_type", type=str, default='glb',
+ choices=['glb', 'fbx'],
+ help="Export format for the animated character.")
+ parser.add_argument("--fps", type=int, default=30,
+ help="Scene frame rate for the export (Mixamo is 30).")
+ parser.add_argument("--overwrite", action='store_true',
+ help="Re-export clips whose output already exists.")
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ args = parse_args()
+ assert os.path.exists(args.char_path), f"Character file not found: {args.char_path}"
+ assert os.path.exists(args.anim_path), f"Animation path not found: {args.anim_path}"
+
+ if os.path.isdir(args.anim_path):
+ clips = sorted(os.path.join(args.anim_path, f)
+ for f in os.listdir(args.anim_path)
+ if f.lower().endswith(('.fbx', '.glb')))
+ else:
+ clips = [args.anim_path]
+ logger.info(f"Baking {len(clips)} clip(s) onto {args.char_path}")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ err_log = os.path.join(args.output_dir, "animate_errors.log")
+
+ for clip in tqdm(clips, desc="Animating character"):
+ try:
+ animate_character_fbx(
+ char_path=args.char_path,
+ anim_path=clip,
+ output_dir=args.output_dir,
+ char_anim_type=args.char_anim_type,
+ fps=args.fps,
+ skip_if_exists=not args.overwrite,
+ )
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ logger.error(f"Error processing {os.path.basename(clip)}: {e}")
+ with open(err_log, "a") as f:
+ f.write(f"{os.path.basename(clip)}: {e}\n")
diff --git a/data_process/mesh_animation/animate_mixamo.py b/data_process/mesh_animation/animate_mixamo.py
new file mode 100644
index 0000000..32c0838
--- /dev/null
+++ b/data_process/mesh_animation/animate_mixamo.py
@@ -0,0 +1,66 @@
+"""Batch-animate a Mixamo character mesh with a directory of exported NPZs.
+
+Raw Mixamo data only contains animation (armature + action) without a mesh,
+so every NPZ in ``--npz_dir`` is paired with the same ``--char_path`` and
+written to ``--output_dir`` via :func:`animate_npz.animate_character`.
+Per-file failures are logged to ``animate_errors.log`` inside the output dir.
+
+Usage (Blender headless):
+ blender -b -P data_process/mesh_animation/animate_mixamo.py -- \\
+ --char_path <character.fbx> \\
+ --npz_dir <export_dir>/motions \\
+ --output_dir outputs/mixamo_characters
+"""
+
+import argparse
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from loguru import logger # noqa: E402
+from tqdm import tqdm # noqa: E402
+
+from data_process.mesh_animation.animate_npz import animate_character # noqa: E402
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Batch-animate a Mixamo character mesh from exported NPZ motions.")
+ parser.add_argument("--char_path", type=str, required=True,
+ help="Character FBX file (provides mesh + armature).")
+ parser.add_argument("--npz_dir", type=str, required=True,
+ help="Directory containing export-stage motion NPZ files.")
+ parser.add_argument("--output_dir", type=str, default='outputs/mixamo_characters',
+ help="Output directory for animated GLB/FBX files.")
+ parser.add_argument("--char_anim_type", type=str, default='glb', choices=['glb', 'fbx'],
+ help="Export format for the animated character.")
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ args = parse_args()
+ assert os.path.exists(args.char_path), f"Character file not found: {args.char_path}"
+ assert os.path.isdir(args.npz_dir), f"NPZ directory not found: {args.npz_dir}"
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ err_log = os.path.join(args.output_dir, "animate_errors.log")
+
+ npz_files = sorted(f for f in os.listdir(args.npz_dir) if f.endswith(".npz"))
+ logger.info(f"Found {len(npz_files)} NPZ files in {args.npz_dir}")
+
+ for npz_file in tqdm(npz_files, desc="Animating meshes"):
+ npz_path = os.path.join(args.npz_dir, npz_file)
+ try:
+ animate_character(
+ char_path=args.char_path,
+ anim_path=npz_path,
+ output_dir=args.output_dir,
+ char_anim_type=args.char_anim_type,
+ skip_if_exists=True,
+ )
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ logger.error(f"Error processing {npz_file}: {e}")
+ with open(err_log, "a") as f:
+ f.write(f"{npz_file}: {e}\n")
diff --git a/data_process/mesh_animation/animate_motion.py b/data_process/mesh_animation/animate_motion.py
new file mode 100644
index 0000000..7533a7c
--- /dev/null
+++ b/data_process/mesh_animation/animate_motion.py
@@ -0,0 +1,343 @@
+"""Drive a rigged character with a canonicalized motion clip and save GLB + FBX.
+
+This is the inference-time path: it takes a motion in the *feature* format
+written by the feature-extraction stage (``global_positions`` /
+``local_rotations`` / ``root_facing_quat``) โ€” or generated by the model in
+that same format โ€” together with the dataset's ``cond.npy`` (skeleton
+topology, joint names, scale) and drives the original rigged asset with it.
+
+Pipeline:
+ 1. Load the cond entry for the requested ``--dataset_type`` and the motion
+ file (NPZ; legacy ``.npy`` model features are also accepted, see below).
+ 2. Reconcile the character armature against the animation's bone list
+ (see :func:`data_process.utils.blender_rig.sync_armature_bones`).
+ 3. Compute per-bone keyframes and bind them as a Blender Action.
+ 4. Export the animated character as ``.glb`` and ``.fbx``.
+
+Usage (Blender headless):
+ blender -b -P data_process/mesh_animation/animate_motion.py -- \\
+ --dataset_type objaverse \\
+ --anim_path dataset/features/objaverse/motions/<clip>.npz \\
+ --output_dir outputs/animated
+
+``--char_path`` is auto-resolved for truebones / objaverse from the raw
+asset directories; it is required for mixamo.
+"""
+
+import argparse
+import os
+import sys
+
+import numpy as np
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from Animation import Animation, Quaternions, transforms_local # noqa: E402
+
+from data_process.utils.blender_export import reset_scene # noqa: E402
+from data_process.utils.blender_rig import ( # noqa: E402
+ EXTRA_BONES_STRATEGIES,
+ compute_bone_keyframes,
+ export_animated_character,
+ get_armature_obj,
+ load_file,
+ rebuild_action_from_data,
+ repair_and_pack_textures,
+ set_scene_timing,
+ sync_armature_bones,
+ update_scene,
+)
+from data_process.utils.skeleton import scale_anim # noqa: E402
+
+
+MIXAMO_OBJECT_TYPE = 'mixamo'
+
+# Default raw-asset locations used for --char_path auto-resolution.
+TRUEBONES_RAW_DIR = 'dataset/raw/truebones/Truebone_Z-OO'
+OBJAVERSE_RAW_DIR = 'dataset/raw/objaverse/glb'
+
+# Default cond.npy location per dataset (written by the feature-extraction stage).
+COND_PATH_TEMPLATE = 'dataset/features/{dataset_type}/cond.npy'
+
+
+# ---------------------------------------------------------------------------
+# Animation loading
+# ---------------------------------------------------------------------------
+
+def _get_scalar(d, key, default):
+ """Read a scalar from an NpzFile / dict, falling back to *default*."""
+ return float(d[key]) if key in d else default
+
+
+def build_anim_from_npz(anim_path, cond_data):
+ """Build ``(anim, rest_anim, tpos_global_rot, raw_anim_data)`` from a
+ feature-format NPZ (``global_positions`` + ``local_rotations``)."""
+ parents = cond_data['parents']
+ scale_factor = _get_scalar(cond_data, 'scale_factor', 1.0)
+
+ offsets_scaled = cond_data['offsets'] / scale_factor
+ tpos_local_rot = cond_data['tpos_local_rotations']
+ tpos_positions = cond_data['tpos_first_frame'] / scale_factor
+ tpos_positions[1:] = offsets_scaled[1:]
+
+ anim_data = np.load(anim_path, allow_pickle=True)
+ motion_local_rot = anim_data['local_rotations'] # (nframes, nbones, 4)
+ nbones = motion_local_rot.shape[1]
+
+ anim_positions = anim_data['global_positions'] / scale_factor
+ anim_positions[:, 1:] = offsets_scaled[1:][None]
+
+ anim = Animation(
+ rotations=Quaternions(motion_local_rot),
+ positions=anim_positions,
+ orients=Quaternions.id(nbones),
+ offsets=offsets_scaled,
+ parents=parents,
+ )
+ rest_anim = Animation(
+ rotations=Quaternions(tpos_local_rot[None]),
+ positions=tpos_positions[None],
+ orients=Quaternions.id(nbones),
+ offsets=offsets_scaled,
+ parents=parents,
+ )
+ return anim, rest_anim, None, anim_data
+
+
+def build_anim_from_npy(anim_path, cond_data, anim_mode):
+ """Build ``(anim, rest_anim, tpos_global_rot, raw_anim_data)`` from a
+ legacy ``.npy`` of model motion features.
+
+ Decoding model features back into an ``Animation`` requires the
+ training/inference package (``src.utils.motion_utils``), which is
+ imported lazily here so the NPZ path stays usable without it.
+ """
+ try:
+ from src.utils.motion_utils import (
+ recover_anytop_anim_from_rot,
+ recover_anytop_joint_pos_from_ric,
+ )
+ except ImportError as exc: # pragma: no cover โ€” depends on the model package
+ raise RuntimeError(
+ "Legacy .npy motion features need the training/inference package "
+ "(src.utils.motion_utils); pass a feature-format .npz instead."
+ ) from exc
+
+ parents = cond_data['parents']
+ scale_factor = _get_scalar(cond_data, 'scale_factor', 1.0)
+ tpos_global_rot = cond_data['tpos_global_rotations']
+
+ offsets = cond_data['tpos_offsets']
+ offsets_scaled = offsets / scale_factor
+
+ anim_data = np.load(anim_path)
+ nbones = anim_data.shape[1]
+
+ if anim_mode == 'fk':
+ logger.info("Recovering animation using FK mode")
+ anim = recover_anytop_anim_from_rot(anim_data, parents, offsets)
+ anim = scale_anim(anim, 1. / scale_factor)
+ elif anim_mode == 'ik':
+ logger.info("Recovering animation using IK mode")
+ from InverseKinematics import animation_from_positions
+ recover_pos_ric = recover_anytop_joint_pos_from_ric(anim_data)
+ ik_anim, ik_order, _ = animation_from_positions(
+ recover_pos_ric, parents, offsets, iterations=150,
+ )
+ inverse_order = np.argsort(ik_order)
+ ik_anim_reordered = Animation(
+ rotations=ik_anim.rotations[:, inverse_order],
+ positions=ik_anim.positions[:, inverse_order],
+ orients=ik_anim.orients[:, inverse_order],
+ offsets=offsets,
+ parents=parents,
+ )
+ anim = scale_anim(ik_anim_reordered, 1. / scale_factor)
+ else:
+ raise RuntimeError(f"Unsupported anim_mode: {anim_mode}")
+
+ rest_anim = Animation(
+ rotations=Quaternions.id((1, nbones)),
+ positions=offsets_scaled[None],
+ orients=Quaternions.id(nbones),
+ offsets=offsets_scaled,
+ parents=parents,
+ )
+ return anim, rest_anim, tpos_global_rot, anim_data
+
+
+def load_animation(anim_path, cond_data, anim_mode):
+ """Dispatch to the NPZ or NPY loader based on file extension."""
+ if anim_path.endswith('.npz'):
+ return build_anim_from_npz(anim_path, cond_data)
+ if anim_path.endswith('.npy'):
+ return build_anim_from_npy(anim_path, cond_data, anim_mode)
+ raise RuntimeError(f"Unsupported animation file format: {anim_path}")
+
+
+# ---------------------------------------------------------------------------
+# Path resolution
+# ---------------------------------------------------------------------------
+
+def resolve_object_type(anim_path, dataset_type):
+ """Map an animation filename to its cond-dict key.
+
+ - truebones / objaverse: filename is ``{object_type}-{motion}[-{clip}].npz``,
+ so the object type is the prefix before the first '-'.
+ - mixamo: motion filenames carry no ``{object_type}-`` prefix; the cond
+ dict always uses ``'mixamo'`` as its single key.
+ """
+ if dataset_type == 'mixamo':
+ return MIXAMO_OBJECT_TYPE
+ return os.path.basename(anim_path).split("-")[0]
+
+
+def resolve_char_path(anim_path, dataset_type):
+ """Auto-resolve a character mesh path from ``anim_path`` + ``dataset_type``.
+
+ - truebones: ``TRUEBONES_RAW_DIR/{object_type}/*.fbx`` (prefers an
+ ``Idle`` / T-pose variant when available).
+ - objaverse: ``OBJAVERSE_RAW_DIR/{object_type}.glb``.
+ - mixamo: no auto-resolution โ€” ``--char_path`` is required.
+ """
+ if dataset_type == 'mixamo':
+ raise ValueError(
+ "dataset_type='mixamo' has no auto-resolved character mesh; "
+ "pass --char_path explicitly (or set CHAR_PATH in the bash wrapper)."
+ )
+
+ object_type = resolve_object_type(anim_path, dataset_type)
+
+ if dataset_type == 'truebones':
+ char_dir = os.path.join(TRUEBONES_RAW_DIR, object_type)
+ if not os.path.isdir(char_dir):
+ raise FileNotFoundError(
+ f"No truebones character directory for object_type='{object_type}' "
+ f"(resolved from '{anim_path}') at {char_dir}")
+ fbx_files = sorted(f for f in os.listdir(char_dir) if f.lower().endswith('.fbx'))
+ if not fbx_files:
+ raise FileNotFoundError(f"No FBX file found in {char_dir}")
+ idle_files = [f for f in fbx_files if 'idle' in f.lower()]
+ return os.path.join(char_dir, idle_files[0] if idle_files else fbx_files[0])
+
+ if dataset_type == 'objaverse':
+ char_path = os.path.join(OBJAVERSE_RAW_DIR, f'{object_type}.glb')
+ if not os.path.isfile(char_path):
+ raise FileNotFoundError(
+ f"No objaverse character file for object_type='{object_type}' "
+ f"(resolved from '{anim_path}') at {char_path}")
+ return char_path
+
+ raise ValueError(f"Unknown dataset_type: {dataset_type}")
+
+
+def load_cond_data(cond_path, anim_path, dataset_type):
+ object_type = resolve_object_type(anim_path, dataset_type)
+ cond_dict = np.load(cond_path, allow_pickle=True).item()
+ if object_type not in cond_dict:
+ raise KeyError(
+ f"object_type '{object_type}' (resolved from '{anim_path}' under "
+ f"dataset_type='{dataset_type}') not found in cond file {cond_path}; "
+ f"available keys: {sorted(cond_dict.keys())}")
+ return cond_dict[object_type]
+
+
+# ---------------------------------------------------------------------------
+# Main pipeline
+# ---------------------------------------------------------------------------
+
+def animate_character(char_path, anim_path, cond_path, output_dir,
+ dataset_type, anim_mode='fk',
+ extra_bones_strategy='merge'):
+ """Drive a character mesh with a motion clip and export GLB + FBX.
+
+ Args:
+ char_path: Character FBX/GLB (provides mesh + armature).
+ anim_path: Feature-format motion NPZ (or legacy NPY).
+ cond_path: ``cond.npy`` for the dataset.
+ output_dir: Destination directory (created if missing).
+ dataset_type: ``'truebones'`` | ``'objaverse'`` | ``'mixamo'``;
+ controls how the cond-dict key is derived from ``anim_path``.
+ anim_mode: Recovery mode for legacy NPY input, ``'fk'`` or ``'ik'``.
+ extra_bones_strategy: How to handle armature bones not present in
+ the animation (see :func:`sync_armature_bones`).
+ """
+ assert os.path.exists(char_path), f"Character file not found: {char_path}"
+ assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"
+ assert os.path.exists(cond_path), f"Condition file not found: {cond_path}"
+
+ os.makedirs(output_dir, exist_ok=True)
+ anim_base = os.path.splitext(os.path.basename(anim_path))[0]
+ output_path_no_ext = os.path.join(output_dir, anim_base)
+
+ reset_scene()
+ char_armature = get_armature_obj(load_file(char_path))
+ assert char_armature is not None, "No armature found in character file"
+ repair_and_pack_textures(char_path)
+
+ cond_data = load_cond_data(cond_path, anim_path, dataset_type)
+ bone_names = cond_data['joint_names']
+ anim, rest_anim, tpos_global_rot, anim_data = load_animation(
+ anim_path, cond_data, anim_mode)
+
+ anim_local_mat = transforms_local(anim) # (nframes, nbones, 4, 4)
+ rest_local_mat = transforms_local(rest_anim)[0] # (nbones, 4, 4)
+
+ sync_armature_bones(char_armature, bone_names,
+ extra_bones_strategy=extra_bones_strategy)
+ set_scene_timing(anim_local_mat.shape[0], _get_scalar(anim_data, 'fps', 30))
+
+ keyframes = compute_bone_keyframes(
+ rest_local_mat, anim_local_mat, bone_names, tpos_global_rot)
+ rebuild_action_from_data(char_armature, keyframes)
+ update_scene()
+
+ export_animated_character(output_path_no_ext, formats=('glb', 'fbx'))
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Drive a rigged character with a canonicalized motion clip.")
+ parser.add_argument("--dataset_type", type=str, required=True,
+ choices=['truebones', 'objaverse', 'mixamo'],
+ help="Dataset that produced the cond/anim files; controls "
+ "how the cond-dict key is resolved.")
+ parser.add_argument("--anim_path", type=str, required=True,
+ help="Feature-format motion NPZ (or legacy .npy).")
+ parser.add_argument("--char_path", type=str, default=None,
+ help="Character FBX/GLB. If omitted, auto-resolved from "
+ f"--dataset_type and --anim_path (truebones -> "
+ f"{TRUEBONES_RAW_DIR}/{{object_type}}/*.fbx, objaverse -> "
+ f"{OBJAVERSE_RAW_DIR}/{{object_type}}.glb). Required for mixamo.")
+ parser.add_argument("--cond_path", type=str, default=None,
+ help=f"cond.npy for the dataset (default: {COND_PATH_TEMPLATE}).")
+ parser.add_argument("--output_dir", type=str, default='outputs/animated')
+ parser.add_argument("--anim_mode", type=str, default='fk', choices=['fk', 'ik'],
+ help="Recovery mode for legacy .npy input.")
+ parser.add_argument("--extra_bones_strategy", type=str, default='merge',
+ choices=list(EXTRA_BONES_STRATEGIES),
+ help="What to do with armature bones missing from the "
+ "animation. 'merge' (default): transfer vertex weights to "
+ "the nearest kept ancestor, then remove the bones. "
+ "'remove': delete bones AND mesh vertices dominantly "
+ "weighted to them. 'keep': leave in place, no keyframes.")
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ args = parse_args()
+ char_path = args.char_path or resolve_char_path(args.anim_path, args.dataset_type)
+ cond_path = args.cond_path or COND_PATH_TEMPLATE.format(dataset_type=args.dataset_type)
+ logger.info(f"Using character mesh: {char_path}")
+ logger.info(f"Using cond file: {cond_path}")
+ animate_character(
+ char_path=char_path,
+ anim_path=args.anim_path,
+ cond_path=cond_path,
+ output_dir=args.output_dir,
+ dataset_type=args.dataset_type,
+ anim_mode=args.anim_mode,
+ extra_bones_strategy=args.extra_bones_strategy,
+ )
diff --git a/data_process/mesh_animation/animate_npz.py b/data_process/mesh_animation/animate_npz.py
new file mode 100644
index 0000000..d5d9943
--- /dev/null
+++ b/data_process/mesh_animation/animate_npz.py
@@ -0,0 +1,143 @@
+"""Drive a rigged character with an *exported* motion NPZ and save GLB/FBX.
+
+Takes the NPZ format written by the export stage
+(``rest_local_pos`` / ``rest_local_rot`` / ``anim_local_pos`` /
+``anim_local_rot`` / ``names`` / ``parents`` / ``offsets``) and re-applies it
+to a character whose armature uses the same bone names. Useful for
+round-trip checks of the export and for re-rendering exported clips.
+
+For *generated* motions (feature NPZ + ``cond.npy``) use
+:mod:`data_process.mesh_animation.animate_motion` instead.
+
+Usage (Blender headless):
+ blender -b -P data_process/mesh_animation/animate_npz.py -- \\
+ --char_path my_model.glb \\
+ --anim_path <export_dir>/motions/my_model-Walk.npz \\
+ --output_dir outputs/animated
+"""
+
+import argparse
+import os
+import sys
+
+import numpy as np
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from Animation import Animation, Quaternions, transforms_local # noqa: E402
+
+from data_process.utils.blender_export import reset_scene # noqa: E402
+from data_process.utils.blender_rig import ( # noqa: E402
+ compute_bone_keyframes,
+ export_animated_character,
+ get_armature_obj,
+ load_file,
+ rebuild_action_from_data,
+ set_scene_timing,
+ sync_armature_bones,
+ update_scene,
+)
+
+
+def _npz_get_scalar(npz, key, default):
+ """Read a scalar from an NpzFile, falling back to *default*."""
+ return float(npz[key]) if key in npz else default
+
+
+def build_anims_from_export_npz(anim_data):
+ """Build ``(anim, rest_anim)`` Animation objects from an export NPZ."""
+ bone_names = anim_data['names'].tolist()
+ nbones = anim_data['anim_local_rot'].shape[1]
+ scale_factor = _npz_get_scalar(anim_data, 'scale_factor', 1.0)
+
+ anim_positions = anim_data['anim_local_pos'] / scale_factor
+ rest_positions = anim_data['rest_local_pos'][None] / scale_factor
+ offsets = anim_data['offsets'] / scale_factor
+ parents = anim_data['parents']
+
+ anim = Animation(
+ rotations=Quaternions(anim_data['anim_local_rot']),
+ positions=anim_positions,
+ orients=Quaternions.id(nbones),
+ offsets=offsets,
+ parents=parents,
+ )
+ rest_anim = Animation(
+ rotations=Quaternions(anim_data['rest_local_rot'][None]),
+ positions=rest_positions,
+ orients=Quaternions.id(nbones),
+ offsets=offsets,
+ parents=parents,
+ )
+ return anim, rest_anim, bone_names
+
+
+def animate_character(char_path, anim_path, output_dir, char_anim_type='glb',
+ extra_bones_strategy='merge', skip_if_exists=False):
+ """Drive a character mesh with an export NPZ and save as GLB/FBX.
+
+ Args:
+ char_path: Character FBX/GLB (provides mesh + armature).
+ anim_path: Motion NPZ produced by the export stage.
+ output_dir: Destination directory (created if missing).
+ char_anim_type: Export format, ``'glb'`` or ``'fbx'``.
+ extra_bones_strategy: How to treat armature bones absent from the
+ animation (see :func:`sync_armature_bones`).
+ skip_if_exists: Return early when the output file already exists.
+ """
+ assert os.path.exists(char_path), f"Character file not found: {char_path}"
+ assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"
+
+ anim_base = os.path.splitext(os.path.basename(anim_path))[0]
+ output_path = os.path.join(output_dir, f"{anim_base}.{char_anim_type}")
+ if skip_if_exists and os.path.exists(output_path):
+ logger.info(f"Output already exists, skipping: {output_path}")
+ return
+ os.makedirs(output_dir, exist_ok=True)
+
+ reset_scene()
+ char_armature = get_armature_obj(load_file(char_path))
+ assert char_armature is not None, "No armature found in character file"
+
+ anim_data = np.load(anim_path, allow_pickle=True)
+ anim, rest_anim, bone_names = build_anims_from_export_npz(anim_data)
+ anim_local_mat = transforms_local(anim) # (nframes, nbones, 4, 4)
+ rest_local_mat = transforms_local(rest_anim)[0] # (nbones, 4, 4)
+
+ sync_armature_bones(char_armature, bone_names,
+ extra_bones_strategy=extra_bones_strategy)
+ set_scene_timing(anim_local_mat.shape[0], _npz_get_scalar(anim_data, 'fps', 30))
+
+ keyframes = compute_bone_keyframes(rest_local_mat, anim_local_mat, bone_names)
+ rebuild_action_from_data(char_armature, keyframes)
+ update_scene()
+
+ export_animated_character(os.path.splitext(output_path)[0], formats=(char_anim_type,))
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Drive a rigged character with an exported motion NPZ.")
+ parser.add_argument("--char_path", type=str, required=True,
+ help="Character FBX/GLB (mesh + armature).")
+ parser.add_argument("--anim_path", type=str, required=True,
+ help="Export-stage motion NPZ whose bone names match the rig.")
+ parser.add_argument("--output_dir", type=str, default='outputs/animated')
+ parser.add_argument("--char_anim_type", type=str, default='glb', choices=['glb', 'fbx'])
+ parser.add_argument("--extra_bones_strategy", type=str, default='merge',
+ choices=['keep', 'merge', 'remove'],
+ help="What to do with armature bones missing from the animation.")
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ args = parse_args()
+ animate_character(
+ char_path=args.char_path,
+ anim_path=args.anim_path,
+ output_dir=args.output_dir,
+ char_anim_type=args.char_anim_type,
+ extra_bones_strategy=args.extra_bones_strategy,
+ )
diff --git a/data_process/motion_export/__init__.py b/data_process/motion_export/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/motion_export/export_general.py b/data_process/motion_export/export_general.py
new file mode 100644
index 0000000..9b5d427
--- /dev/null
+++ b/data_process/motion_export/export_general.py
@@ -0,0 +1,288 @@
+"""Export rigged assets (GLB/GLTF or FBX) to NPZ motion data.
+
+Generic counterpart to the per-dataset batch exporters: ``--input`` is either
+a single file or a directory of ``.glb``/``.gltf``/``.fbx`` assets (mixed
+formats welcome). The importer is picked per file by extension, the character
+mesh is optional, and every pose action found in a file is exported as its
+own clip. Directory runs mirror the Objaverse exporter: per-asset completion
+markers for resume, worker sharding, a per-worker error log, and per-worker
+summary shards (merge with ``data_process.tools.merge_summaries``).
+
+Pipeline: import scene โ†’ select armature (+ mesh if present) โ†’ discover
+actions โ†’ build skeleton arrays โ†’ extract animations โ†’ optional skeleton
+pruning โ†’ coordinate conversion โ†’ save per-action NPZ + MP4 visualization.
+
+Behavior notes:
+ - Mesh optional: with a mesh, skin weights are extracted and skeleton
+ pruning is enabled by default; without one, ``skin_matrix`` is saved
+ as an empty ``(0, nbones)`` array and pruning is skipped (the pruning
+ passes rely on skin weights to decide which bones matter).
+ - Actions: all relevant pose actions are exported. Files whose action
+ does not pass the pose-action filter (e.g. Mixamo animation-only
+ clips) fall back to the armature's single bound action.
+ - No joint-count filtering: stage 4 (feature_extraction --min_joints /
+ --max_joints) decides which skeletons enter training.
+
+Usage (Blender headless):
+ blender -b -P data_process/motion_export/export_general.py -- \
+ --input assets_dir --output_dir outputs/export
+
+Usage (pip-installed bpy):
+ python data_process/motion_export/export_general.py \
+ --input assets/dragon.fbx --output_dir outputs/export
+"""
+
+import bpy
+import sys
+import os
+import numpy as np
+import argparse
+from pathlib import Path
+from tqdm import tqdm
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import (
+ import_fbx, import_gltf, sanitize_action_name, unique_clip_names,
+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,
+ prune_skeleton_shared, remove_tpose_frames, discover_pose_actions,
+ action_frame_range, save_motion, write_export_summary,
+ is_asset_complete, mark_asset_complete,
+)
+
+GLTF_EXTS = {'.glb', '.gltf'}
+FBX_EXTS = {'.fbx'}
+ASSET_EXTS = GLTF_EXTS | FBX_EXTS
+
+
+def pick_importer(path):
+ """Return the importer matching *path*'s extension, or raise."""
+ ext = Path(path).suffix.lower()
+ if ext in GLTF_EXTS:
+ return import_gltf
+ if ext in FBX_EXTS:
+ return import_fbx
+ raise ValueError(f"Unsupported extension '{ext}' (expected one of "
+ f"{sorted(ASSET_EXTS)}): {path}")
+
+
+def list_asset_files(input_dir):
+ """List .glb/.gltf/.fbx files directly under *input_dir*, sorted."""
+ input_dir = Path(input_dir)
+ return sorted(p for p in input_dir.iterdir()
+ if p.is_file() and p.suffix.lower() in ASSET_EXTS)
+
+
+def export_asset(input_path, output_dir, save_name=None, fps=30,
+ dtype=np.float64, prune=None, consider_parent_rotate=True,
+ min_frames=5, remove_tpose=True, save_vis=True):
+ """Export every pose action in a single GLB/GLTF/FBX file to NPZ clips.
+
+ Args:
+ prune: True/False to force skeleton pruning on/off; None (default)
+ enables it iff the file contains a skinned mesh.
+
+ Returns:
+ {save_name: joint_names} when at least one clip was saved, else {}.
+ """
+ assert os.path.exists(input_path), f"Input file not found: {input_path}"
+ save_name = save_name or Path(input_path).stem
+
+ motion_save_dir = os.path.join(output_dir, "motions")
+ vis_save_dir = os.path.join(output_dir, "videos")
+ tpos_save_dir = os.path.join(output_dir, "tpose")
+
+ if is_asset_complete(output_dir, save_name):
+ logger.info(f"Asset '{save_name}' already complete, skipping: {input_path}")
+ return {}
+ existing = sorted(Path(motion_save_dir).glob(f"{save_name}-*.npz")) if os.path.isdir(motion_save_dir) else []
+ if existing:
+ # Pruning is joint across all of an asset's clips, so a partial
+ # (or pre-marker) export must be redone as a whole.
+ logger.warning(f"'{save_name}' has {len(existing)} clip(s) but no completion "
+ f"marker (partial or pre-marker export); re-exporting.")
+
+ logger.info(f"Processing asset: {input_path}")
+ os.makedirs(motion_save_dir, exist_ok=True)
+ os.makedirs(vis_save_dir, exist_ok=True)
+
+ armature, mesh = load_scene(pick_importer(input_path), input_path, fps=fps)
+ if mesh is None:
+ logger.info(f"No mesh in {input_path}; exporting armature only "
+ f"(empty skin matrix, pruning {'forced on' if prune else 'off'}).")
+ if prune is None:
+ prune = mesh is not None
+
+ # All relevant pose actions; fall back to the armature's bound action
+ # for single-action files that the pose filter rejects.
+ action_names, frame_ranges = discover_pose_actions()
+ if not action_names and armature.animation_data is not None \
+ and armature.animation_data.action is not None:
+ action = armature.animation_data.action
+ action_names = [action.name]
+ frame_ranges = {action.name: action_frame_range(action)}
+ logger.info(f"No pose actions discovered; falling back to bound action '{action.name}'.")
+ logger.info(f"Total actions to export: {len(action_names)}")
+ if not action_names:
+ logger.info(f"No actions in {input_path}; skipping.")
+ mark_asset_complete(output_dir, save_name, status="skipped",
+ reason="no pose actions")
+ return {}
+
+ skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)
+ extracted = extract_all_actions(
+ armature, {n: frame_ranges[n] for n in action_names},
+ skel['rest_local_pos'], skel['parents_array'],
+ skel['rest_anim'], skel['nbones'], dtype=dtype,
+ )
+
+ anims_list = [anim for _, anim in extracted]
+ rest_anim_shared = skel['rest_anim_shared']
+ names = list(skel['bone_names'])
+ skin_matrix = skel['skin_matrix']
+ if prune:
+ anims_list, rest_anim_shared, names, skin_matrix = prune_skeleton_shared(
+ anims_list, rest_anim_shared, names, skin_matrix.copy(),
+ consider_parent_rotate=consider_parent_rotate,
+ )
+
+ save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_shared)
+
+ # Distinct actions can sanitize to the same name and would then overwrite
+ # each other's NPZ; disambiguate over the full discovered list.
+ clip_suffixes = unique_clip_names(action_names)
+
+ scene_fps = bpy.context.scene.render.fps
+ saved_clips = []
+ for i, (action_name, _) in enumerate(extracted):
+ anim = anims_list[i]
+
+ if remove_tpose:
+ anim, n_tpose = remove_tpose_frames(anim, rest_anim_shared)
+ if n_tpose > 0:
+ logger.info(f"Removed {n_tpose} T-pose frames from '{action_name}', "
+ f"{anim.positions.shape[0]} frames remaining")
+ if anim.positions.shape[0] < min_frames:
+ logger.info(f"Skipping '{action_name}': only {anim.positions.shape[0]} frames "
+ f"(min={min_frames})")
+ continue
+
+ clip_name = f"{save_name}-{clip_suffixes.get(action_name) or sanitize_action_name(action_name) or 'action'}"
+ save_motion(
+ os.path.join(motion_save_dir, f"{clip_name}.npz"),
+ os.path.join(vis_save_dir, f"{clip_name}.mp4"),
+ anim, rest_anim_shared, names, skin_matrix, scene_fps,
+ action_name=action_name, save_vis=save_vis,
+ )
+ saved_clips.append(clip_name)
+
+ mark_asset_complete(output_dir, save_name,
+ status="exported" if saved_clips else "skipped",
+ n_clips=len(saved_clips),
+ reason=None if saved_clips else "no clips survived filtering",
+ joint_names=names if saved_clips else None)
+ logger.info(f"Saved {len(saved_clips)} clip(s) for '{save_name}'")
+ if saved_clips:
+ return {save_name: names}
+ return {}
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Entry point
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Export rigged GLB/GLTF/FBX assets (a file or a directory) "
+ "to NPZ motion data.")
+ parser.add_argument('--input', type=str, required=True,
+ help='A .glb/.gltf/.fbx file, or a directory of them.')
+ parser.add_argument('--output_dir', type=str, required=True,
+ help='Output directory (motions/, videos/, tpose/ created inside).')
+ parser.add_argument('--name', type=str, default=None,
+ help='Save name / clip prefix (single-file input only; '
+ 'default: input file stem).')
+ parser.add_argument('--fps', type=int, default=30)
+ parser.add_argument('--prune', action=argparse.BooleanOptionalAction, default=None,
+ help='Force skeleton pruning on/off '
+ '(default: on iff the file has a skinned mesh).')
+ parser.add_argument('--consider_parent_rotate', action=argparse.BooleanOptionalAction, default=True,
+ help='Preserve non-rotating leaves whose parent rotates (see prune_skeleton_shared).')
+ parser.add_argument('--min_frames', type=int, default=5,
+ help='Skip clips shorter than this many frames '
+ '(shared minimum across all dataset exporters).')
+ parser.add_argument('--keep_tpose_frames', action='store_true',
+ help='Keep frames matching the rest pose instead of removing them.')
+ parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,
+ help='Render the per-clip MP4 preview (use --no-vis for bulk runs).')
+ parser.add_argument('--worker_id', type=int, default=0,
+ help='Worker index for sharding files across workers (directory input).')
+ parser.add_argument('--num_workers', type=int, default=1,
+ help='Total number of workers (directory input).')
+
+ # Blender passes script args after "--"; plain python passes them directly.
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+def main():
+ args = parse_args()
+ export_kwargs = dict(
+ fps=args.fps, prune=args.prune,
+ consider_parent_rotate=args.consider_parent_rotate,
+ min_frames=args.min_frames, remove_tpose=not args.keep_tpose_frames,
+ save_vis=args.vis,
+ )
+
+ if os.path.isdir(args.input):
+ asset_paths = list_asset_files(args.input)
+ assert asset_paths, f"No {sorted(ASSET_EXTS)} files found in {args.input}"
+ logger.info(f"Found {len(asset_paths)} asset files in {args.input}")
+
+ if args.num_workers > 1:
+ asset_paths = asset_paths[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "
+ f"processing {len(asset_paths)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'export_errors_worker{args.worker_id}.log')
+ all_joint_names = {}
+ n_failed = 0
+ for asset_path in tqdm(asset_paths, desc="Exporting assets"):
+ try:
+ saved = export_asset(str(asset_path), args.output_dir,
+ **export_kwargs)
+ if saved:
+ all_joint_names.update(saved)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ logger.exception(f"Failed to export {asset_path}")
+ with open(error_log, 'a') as log_file:
+ log_file.write(f"Failed to export {asset_path}: {e}\n")
+
+ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""
+ write_export_summary(args.output_dir, all_joint_names,
+ worker_suffix=worker_suffix)
+ else:
+ saved = export_asset(args.input, args.output_dir, save_name=args.name,
+ **export_kwargs)
+ if saved:
+ write_export_summary(args.output_dir, saved)
+
+ if n_failed:
+ logger.error(f"{n_failed}/{len(asset_paths)} assets failed; "
+ f"see {error_log}")
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ # Blender exits 0 even on an uncaught exception, which hides failures from
+ # `set -e` and from Slurm's afterok dependencies.
+ try:
+ main()
+ except SystemExit:
+ raise
+ except Exception:
+ logger.exception("export_general failed")
+ sys.exit(1)
diff --git a/data_process/motion_export/export_mixamo.py b/data_process/motion_export/export_mixamo.py
new file mode 100644
index 0000000..0cc849a
--- /dev/null
+++ b/data_process/motion_export/export_mixamo.py
@@ -0,0 +1,236 @@
+"""Export Mixamo animation FBX files to NPZ motion data.
+
+Pipeline: import animation FBX (armature + action, no mesh) โ†’ build skeleton
+arrays โ†’ extract animation โ†’ Z-up โ†’ Y-up โ†’ save per-clip NPZ + MP4.
+
+Skin weights are not extracted (there is no character mesh); ``skin_matrix``
+is saved as an empty ``(0, nbones)`` array. All Mixamo animations share the
+same skeleton, so ``joint_names.json`` and the rest-pose PNG are written once
+(by worker 0) and every freshly-extracted clip's rest pose is checked against
+the first one.
+
+Files can be sharded across parallel Blender processes with
+``--worker_id`` / ``--num_workers``; each worker writes its own summary
+JSONs (``*_worker{i}.json``) โ€” merge them afterwards with
+``python -m data_process.tools.merge_summaries`` (the
+``run_export.sh`` wrapper does both automatically).
+
+Usage (Blender headless):
+ blender -b -P data_process/motion_export/export_mixamo.py -- \
+ --anim_dir dataset/raw/mixamo/animation_motion \
+ --output_dir dataset/export/mixamo
+"""
+
+import bpy
+import sys
+import os
+import json
+import numpy as np
+import argparse
+from tqdm import tqdm
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import (
+ import_fbx,
+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,
+ action_frame_range, save_motion, write_export_summary,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Main export function
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def export_mixamo(anim_path, output_dir, fps=30, dtype=np.float64,
+ min_frames=5, skip_if_exists=True, save_vis=True):
+ """Export a single Mixamo animation clip.
+
+ Args:
+ anim_path: Path to animation FBX (provides armature + action).
+ output_dir: Output root directory.
+ fps: Scene FPS.
+ dtype: Numpy dtype for arrays.
+ skip_if_exists: Return early when the output NPZ already exists.
+ save_vis: Also render the per-clip MP4 preview.
+
+ Returns:
+ (bone_names, rest_anim) if extracted, else None.
+ """
+ assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"
+
+ motion_save_dir = os.path.join(output_dir, "motions")
+ vis_save_dir = os.path.join(output_dir, "videos")
+ os.makedirs(motion_save_dir, exist_ok=True)
+ os.makedirs(vis_save_dir, exist_ok=True)
+
+ anim_base_name = os.path.splitext(os.path.basename(anim_path))[0]
+ motion_save_path = os.path.join(motion_save_dir, f"{anim_base_name}.npz")
+ vis_save_path = os.path.join(vis_save_dir, f"{anim_base_name}.mp4")
+
+ # Skip if output already exists (unless caller still needs metadata)
+ output_exists = os.path.exists(motion_save_path)
+ if output_exists and skip_if_exists:
+ logger.info(f"Output already exists for '{anim_base_name}', skipping.")
+ return None
+
+ armature, _ = load_scene(import_fbx, anim_path, fps=fps)
+ assert armature.animation_data is not None and armature.animation_data.action is not None, \
+ f"Armature '{armature.name}' has no animation data or action."
+
+ action = armature.animation_data.action
+ start_frame, end_frame = action_frame_range(action)
+ logger.info(f"Action frame range: {start_frame} to {end_frame}")
+
+ skel = prepare_skeleton(armature, mesh=None, dtype=dtype, apply_world=True)
+ extracted = extract_all_actions(
+ armature, {action.name: (start_frame, end_frame)},
+ skel['rest_local_pos'], skel['parents_array'],
+ skel['rest_anim'], skel['nbones'], dtype=dtype,
+ )
+ _, anim_obj = extracted[0]
+
+ nframes = anim_obj.positions.shape[0]
+ if nframes < min_frames:
+ logger.info(f"Skipping '{anim_base_name}': only {nframes} frames "
+ f"(min={min_frames})")
+ return None
+
+ if not output_exists:
+ scene_fps = bpy.context.scene.render.fps
+ save_motion(motion_save_path, vis_save_path, anim_obj, skel['rest_anim_shared'],
+ list(skel['bone_names']), skel['skin_matrix'], scene_fps,
+ action_name=anim_base_name, save_vis=save_vis)
+
+ return list(skel['bone_names']), skel['rest_anim_shared']
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Entry point
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_args():
+ parser = argparse.ArgumentParser(description="Export Mixamo animation FBX files to NPZ motion data.")
+ parser.add_argument("--anim_dir", type=str, required=True,
+ help='Directory containing animation FBX files.')
+ parser.add_argument("--output_dir", type=str, required=True,
+ help='Output directory (motions/, videos/, tpose/ created inside).')
+ parser.add_argument("--fps", type=int, default=30,
+ help="Scene FPS for import and export (dataset is 30 fps).")
+ parser.add_argument("--min_frames", type=int, default=5,
+ help="Skip clips shorter than this many frames "
+ "(shared minimum across all dataset exporters).")
+ parser.add_argument("--worker_id", type=int, default=0,
+ help="Worker index for sharding files across workers.")
+ parser.add_argument("--num_workers", type=int, default=1,
+ help="Total number of workers.")
+ parser.add_argument("--vis", action=argparse.BooleanOptionalAction, default=True,
+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+def main(args):
+ os.makedirs(args.output_dir, exist_ok=True)
+ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""
+ error_log = os.path.join(args.output_dir, f"export_errors{worker_suffix}.log")
+
+ n_failed = 0
+ anim_files = sorted([f for f in os.listdir(args.anim_dir) if f.endswith(".fbx")])
+ anim_paths = [os.path.join(args.anim_dir, f) for f in anim_files]
+ logger.info(f"Found {len(anim_paths)} animation files")
+
+ if args.num_workers > 1:
+ anim_paths = anim_paths[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "
+ f"processing {len(anim_paths)} files")
+
+ joint_names_path = os.path.join(args.output_dir, "joint_names.json")
+ tpos_path = os.path.join(args.output_dir, "tpose", "mixamo.png")
+
+ # Only worker 0 writes the shared skeleton metadata (single skeleton for
+ # the whole dataset). While it is missing, worker 0 extracts its first
+ # file even if the NPZ already exists, to recover the reference skeleton.
+ is_primary = args.worker_id == 0
+ need_metadata = is_primary and not (
+ os.path.exists(joint_names_path) and os.path.exists(tpos_path))
+
+ ref_bone_names = None
+ ref_rest_pos = None
+ ref_rest_rot = None
+ tpos_mismatches = []
+
+ for anim_path in tqdm(anim_paths, desc="Processing animations"):
+ try:
+ result = export_mixamo(anim_path, args.output_dir, fps=args.fps,
+ min_frames=args.min_frames,
+ skip_if_exists=not need_metadata,
+ save_vis=args.vis)
+ if result is None:
+ continue
+
+ bone_names, rest_anim = result
+ rest_pos = rest_anim.positions[0] # (nbones, 3)
+ rest_rot = rest_anim.rotations[0].qs # (nbones, 4)
+
+ if ref_bone_names is None:
+ ref_bone_names = bone_names
+ ref_rest_pos = rest_pos.copy()
+ ref_rest_rot = rest_rot.copy()
+ if need_metadata:
+ with open(joint_names_path, 'w') as f:
+ json.dump({"mixamo": bone_names}, f, indent=2)
+ logger.info(f"Saved joint names to {joint_names_path}")
+ if not os.path.exists(tpos_path):
+ save_rest_pose_vis(os.path.dirname(tpos_path), "mixamo", rest_anim)
+ need_metadata = False
+ else:
+ # Check T-pose consistency against the reference skeleton
+ anim_name = os.path.basename(anim_path)
+ if bone_names != ref_bone_names:
+ logger.warning(f"Bone names mismatch: {anim_name}")
+ tpos_mismatches.append(anim_name)
+ else:
+ pos_diff = np.max(np.abs(rest_pos - ref_rest_pos))
+ rot_diff = np.max(np.abs(rest_rot - ref_rest_rot))
+ if pos_diff > 1e-6 or rot_diff > 1e-6:
+ logger.warning(f"T-pose mismatch: {anim_name} "
+ f"(pos_diff={pos_diff:.8f}, rot_diff={rot_diff:.8f})")
+ tpos_mismatches.append(anim_name)
+
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Error processing {anim_path}")
+ with open(error_log, "a") as f:
+ f.write(f"Error processing {anim_path}: {e}\n")
+
+ if tpos_mismatches:
+ logger.warning(f"{len(tpos_mismatches)} animations have mismatched T-pose: {tpos_mismatches}")
+ else:
+ logger.info("All extracted animations share the same T-pose.")
+
+ # On a fully-resumed run nothing is extracted; fall back to the saved
+ # joint names instead of overwriting the summary files with {}.
+ if ref_bone_names is None and os.path.exists(joint_names_path):
+ with open(joint_names_path) as f:
+ ref_bone_names = json.load(f).get("mixamo")
+
+ all_joint_names = {"mixamo": ref_bone_names} if ref_bone_names else {}
+ write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)
+
+ if n_failed:
+ logger.error(f"{n_failed}/{len(anim_paths)} clips failed; see {error_log}")
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ # Blender exits 0 even on an uncaught exception, which hides failures from
+ # `set -e` and from Slurm's afterok dependencies.
+ try:
+ main(parse_args())
+ except SystemExit:
+ raise
+ except Exception:
+ logger.exception("export_mixamo failed")
+ sys.exit(1)
diff --git a/data_process/motion_export/export_objaverse.py b/data_process/motion_export/export_objaverse.py
new file mode 100644
index 0000000..c2b0818
--- /dev/null
+++ b/data_process/motion_export/export_objaverse.py
@@ -0,0 +1,207 @@
+"""Export a directory of rigged GLB/GLTF files (e.g. Objaverse) to NPZ motion data.
+
+Pipeline per file: import scene โ†’ select armature/mesh โ†’ discover actions โ†’
+extract animations โ†’ Z-up โ†’ Y-up โ†’ prune skeleton (shared across clips) โ†’
+remove T-pose frames โ†’ save per-clip NPZ + MP4 + rest-pose PNG.
+
+Files can be sharded across parallel Blender processes with
+``--worker_id`` / ``--num_workers``; each worker writes its own summary
+JSONs (``*_worker{i}.json``).
+
+Usage (Blender headless):
+ blender -b -P data_process/motion_export/export_objaverse.py -- \
+ --data_dir dataset/raw/objaverse/glb \
+ --output_dir dataset/export/objaverse
+"""
+
+import bpy
+import sys
+import os
+import numpy as np
+import argparse
+from pathlib import Path
+from loguru import logger
+from tqdm import tqdm
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import (
+ import_gltf, list_gltf_files, sanitize_action_name, unique_clip_names,
+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,
+ prune_skeleton_shared, remove_tpose_frames, discover_pose_actions,
+ save_motion, write_export_summary,
+ is_asset_complete, mark_asset_complete,
+)
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Main export pipeline
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def export_objaverse(gltf_path, save_name, output_dir, fps=30, dtype=np.float64,
+ min_frames=5, consider_parent_rotate=True, save_vis=True):
+ """Export every pose action in one GLB/GLTF file. Returns ``{save_name: joint_names}``."""
+ assert os.path.exists(gltf_path), f"GLTF file not found: {gltf_path}"
+
+ motion_save_dir = os.path.join(output_dir, "motions")
+ vis_save_dir = os.path.join(output_dir, "videos")
+ tpos_save_dir = os.path.join(output_dir, "tpose")
+
+ if is_asset_complete(output_dir, save_name):
+ logger.info(f"Asset '{save_name}' already complete, skipping: {gltf_path}")
+ return {}
+ existing = sorted(Path(motion_save_dir).glob(f"{save_name}-*.npz")) if os.path.isdir(motion_save_dir) else []
+ if existing:
+ # Pruning is joint across all of an asset's clips, so a partial
+ # (or pre-marker) export must be redone as a whole.
+ logger.warning(f"'{save_name}' has {len(existing)} clip(s) but no completion "
+ f"marker (partial or pre-marker export); re-exporting.")
+
+ logger.info(f"Processing GLTF: {gltf_path}")
+ os.makedirs(motion_save_dir, exist_ok=True)
+ os.makedirs(vis_save_dir, exist_ok=True)
+
+ armature, mesh = load_scene(import_gltf, gltf_path, fps=fps)
+ assert mesh is not None, f"No mesh found in {gltf_path}; cannot build skin matrix."
+
+ action_names, frame_ranges = discover_pose_actions()
+ logger.info(f"Total relevant actions: {len(action_names)}")
+ if not action_names:
+ logger.info(f"No valid actions in {gltf_path}; skipping.")
+ mark_asset_complete(output_dir, save_name, status="skipped",
+ reason="no pose actions")
+ return {}
+
+ skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)
+ extracted = extract_all_actions(
+ armature, frame_ranges,
+ skel['rest_local_pos'], skel['parents_array'],
+ skel['rest_anim'], skel['nbones'], dtype=dtype,
+ )
+
+ anims_list = [anim for _, anim in extracted]
+ anims_list, rest_anim_pruned, names_pruned, skin_matrix_pruned = prune_skeleton_shared(
+ anims_list, skel['rest_anim_shared'], skel['bone_names'].copy(), skel['skin_matrix'].copy(),
+ consider_parent_rotate=consider_parent_rotate,
+ )
+
+ # No joint-count gate here: stage 4 (feature_extraction --min_joints /
+ # --max_joints) decides which skeletons enter training, so the export
+ # stays complete.
+ save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_pruned)
+
+ # Distinct actions can sanitize to the same name; disambiguate over the
+ # full discovered list so the renderer derives identical clip names.
+ clip_suffixes = unique_clip_names(action_names)
+
+ scene_fps = bpy.context.scene.render.fps
+ saved_clips = []
+ for i, (action_name, _) in enumerate(extracted):
+ anim = anims_list[i]
+
+ anim, n_tpose = remove_tpose_frames(anim, rest_anim_pruned)
+ if n_tpose > 0:
+ logger.info(f"Removed {n_tpose} T-pose frames from '{action_name}', "
+ f"{anim.positions.shape[0]} frames remaining")
+ if anim.positions.shape[0] < min_frames:
+ logger.info(f"Skipping '{action_name}': only {anim.positions.shape[0]} frames "
+ f"after T-pose removal (min={min_frames})")
+ continue
+
+ clean_action_name = clip_suffixes.get(
+ action_name, sanitize_action_name(action_name) or "action")
+ clip_name = f"{save_name}-{clean_action_name}"
+ motion_path = os.path.join(motion_save_dir, f"{clip_name}.npz")
+ vis_path = os.path.join(vis_save_dir, f"{clip_name}.mp4")
+ save_motion(motion_path, vis_path, anim, rest_anim_pruned,
+ names_pruned, skin_matrix_pruned, scene_fps,
+ action_name=action_name, save_vis=save_vis)
+ saved_clips.append(clip_name)
+
+ mark_asset_complete(output_dir, save_name,
+ status="exported" if saved_clips else "skipped",
+ n_clips=len(saved_clips),
+ reason="" if saved_clips else "no clips above min frame count",
+ joint_names=names_pruned if saved_clips else None)
+
+ # Return once per asset (skeleton type), not per clip
+ if saved_clips:
+ return {save_name: names_pruned}
+ return {}
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Entry point
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_args():
+ parser = argparse.ArgumentParser(description="Export GLB/GLTF files to NPZ motion data.")
+ parser.add_argument('--data_dir', type=str, required=True,
+ help='Directory containing GLB/GLTF files.')
+ parser.add_argument('--output_dir', type=str, required=True,
+ help='Output directory for NPZ motion data.')
+ parser.add_argument('--fps', type=int, default=30,
+ help='Scene FPS for import and export.')
+ parser.add_argument('--consider_parent_rotate', action=argparse.BooleanOptionalAction, default=True,
+ help='Preserve non-rotating leaves whose parent rotates during pruning '
+ '(see prune_skeleton_shared; use --no-consider_parent_rotate to disable).')
+ parser.add_argument('--min_frames', type=int, default=5,
+ help='Skip clips shorter than this after T-pose removal.')
+ parser.add_argument('--worker_id', type=int, default=0,
+ help="Worker index for sharding files across workers.")
+ parser.add_argument('--num_workers', type=int, default=1,
+ help="Total number of workers.")
+ parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,
+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")
+
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+def main():
+ args = parse_args()
+
+ gltf_paths = list_gltf_files(Path(args.data_dir))
+ assert len(gltf_paths) > 0, f"No GLB/GLTF files found in {args.data_dir}"
+ logger.info(f"Found {len(gltf_paths)} GLB/GLTF files in {args.data_dir}")
+
+ # Shard files across workers
+ if args.num_workers > 1:
+ gltf_paths = gltf_paths[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: processing {len(gltf_paths)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir, f'export_errors_worker{args.worker_id}.log')
+ all_joint_names = {}
+ n_failed = 0
+ for glb_path in tqdm(gltf_paths, desc="Exporting GLB/GLTF files"):
+ try:
+ saved = export_objaverse(glb_path, glb_path.stem, output_dir=args.output_dir,
+ fps=args.fps, min_frames=args.min_frames,
+ consider_parent_rotate=args.consider_parent_rotate,
+ save_vis=args.vis)
+ if saved:
+ all_joint_names.update(saved)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Failed to export {glb_path}")
+ with open(error_log, 'a') as log_file:
+ log_file.write(f"Failed to export {glb_path}: {e}\n")
+
+ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""
+ write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)
+
+ if n_failed:
+ logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ # Blender exits 0 even on an uncaught exception, which hides failures from
+ # `set -e` and from Slurm's afterok dependencies.
+ try:
+ main()
+ except SystemExit:
+ raise
+ except Exception:
+ logger.exception("export_objaverse failed")
+ sys.exit(1)
diff --git a/data_process/motion_export/export_truebones.py b/data_process/motion_export/export_truebones.py
new file mode 100644
index 0000000..ecd6562
--- /dev/null
+++ b/data_process/motion_export/export_truebones.py
@@ -0,0 +1,271 @@
+"""Export Truebones FBX clips to NPZ motion data.
+
+Consumes the curated flat layout of the Truebones ZOO dataset: one FBX per
+clip, named ``{Species}-{Action}.fbx`` (species never contains ``-``), all in
+a single directory. Files are grouped by species so skeleton pruning is
+shared across every clip of that species.
+
+Pipeline: import โ†’ select armature/mesh โ†’ extract animation โ†’ Z-up โ†’ Y-up โ†’
+prune skeleton (shared across the species' clips) โ†’ save per-clip NPZ + MP4
++ rest-pose PNG.
+
+Usage (Blender headless):
+ blender -b -P data_process/motion_export/export_truebones.py -- \
+ --data_dir dataset/raw/truebones/animation \
+ --output_dir dataset/export/truebones
+"""
+
+import bpy
+import sys
+import os
+import numpy as np
+from tqdm import tqdm
+import argparse
+from pathlib import Path
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import (
+ import_fbx,
+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,
+ prune_skeleton_shared,
+ action_frame_range, sanitize_action_name, sanitize_object_type,
+ save_motion, write_export_summary,
+ is_asset_complete, mark_asset_complete,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-FBX extraction (one action per file)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def extract_single_fbx(fbx_path, fps=30, dtype=np.float64):
+ """Import one FBX file and return its (anim, rest_anim, names, skin_matrix, nbones)."""
+ assert os.path.exists(fbx_path), f"FBX file not found: {fbx_path}"
+
+ armature, mesh = load_scene(import_fbx, fbx_path, fps=fps)
+ assert armature.animation_data is not None and armature.animation_data.action is not None, \
+ f"Armature '{armature.name}' has no animation data or action."
+
+ action = armature.animation_data.action
+ start_frame, end_frame = action_frame_range(action)
+ logger.info(f"Action frame range: {start_frame} to {end_frame}")
+
+ skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)
+ extracted = extract_all_actions(
+ armature, {action.name: (start_frame, end_frame)},
+ skel['rest_local_pos'], skel['parents_array'],
+ skel['rest_anim'], skel['nbones'], dtype=dtype,
+ )
+ _, anim = extracted[0]
+ return {
+ 'anim': anim,
+ 'rest_anim': skel['rest_anim_shared'],
+ 'names': skel['bone_names'],
+ 'skin_matrix': skel['skin_matrix'],
+ 'nbones': skel['nbones'],
+ }
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Shared post-processing: pruning, visualization, saving
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def postprocess_and_save(obj_type, extracted, rest_anim_shared, bone_names,
+ skin_matrix, output_dir, fps,
+ consider_parent_rotate=True, save_vis=True,
+ save_tpose=True):
+ """Shared skeleton pruning, visualization, and per-clip NPZ saving."""
+ motion_save_dir = os.path.join(output_dir, "motions")
+ vis_save_dir = os.path.join(output_dir, "videos")
+ tpos_save_dir = os.path.join(output_dir, "tpose")
+ os.makedirs(motion_save_dir, exist_ok=True)
+ os.makedirs(vis_save_dir, exist_ok=True)
+
+ anims_list = [anim for _, anim in extracted]
+ anims_list, rest_anim_pruned, names_pruned, skin_matrix_pruned = prune_skeleton_shared(
+ anims_list, rest_anim_shared, list(bone_names), skin_matrix.copy(),
+ consider_parent_rotate=consider_parent_rotate,
+ )
+
+ # No joint-count gate here: stage 4 (feature_extraction --min_joints /
+ # --max_joints) decides which skeletons enter training, so the export
+ # stays complete.
+ if save_tpose:
+ save_rest_pose_vis(tpos_save_dir, obj_type, rest_anim_pruned)
+
+ for i, (action_name, _) in enumerate(extracted):
+ clean_action_name = sanitize_action_name(action_name)
+ clip_name = f"{obj_type}-{clean_action_name}"
+ save_motion(
+ os.path.join(motion_save_dir, f"{clip_name}.npz"),
+ os.path.join(vis_save_dir, f"{clip_name}.mp4"),
+ anims_list[i], rest_anim_pruned,
+ names_pruned, skin_matrix_pruned, fps,
+ action_name=clean_action_name, save_vis=save_vis,
+ )
+
+ logger.info(f"Saved {len(extracted)} clips for '{obj_type}'")
+ return {obj_type: names_pruned}
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-species export
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def export_species(species, fbx_paths, output_dir, fps=30, dtype=np.float64,
+ min_frames=5, consider_parent_rotate=True, save_vis=True):
+ """Export one species: extract each of its per-clip FBX files, then prune
+ the skeleton jointly across all clips and save them."""
+
+ obj_type = sanitize_object_type(species)
+
+ if is_asset_complete(output_dir, obj_type):
+ logger.info(f"Species '{obj_type}' already complete, skipping.")
+ return {}
+ motion_save_dir = os.path.join(output_dir, "motions")
+ existing = sorted(Path(motion_save_dir).glob(f"{obj_type}-*.npz")) if os.path.isdir(motion_save_dir) else []
+ if existing:
+ # Pruning is joint across all of a species' clips, so a partial
+ # (or pre-marker) export must be redone as a whole.
+ logger.warning(f"'{obj_type}' has {len(existing)} clip(s) but no completion "
+ f"marker (partial or pre-marker export); re-exporting.")
+
+ # A few species ship rig variants (different node counts or container
+ # names โ€” e.g. KingCobra's extra Null, Monkey's A01/B01/B02 containers).
+ # prune_skeleton_shared requires identical topology across its inputs, so
+ # clips are grouped by joint-name signature and each group is pruned
+ # separately. The dominant group defines the species' joint_names entry
+ # and rest-pose PNG.
+ groups = {} # names signature -> {'rest_anim', 'skin_matrix', 'extracted'}
+ for fbx_path in fbx_paths:
+ stem = Path(fbx_path).stem
+ species_part, sep, action_name = stem.partition('-')
+ if not sep:
+ # Without the separator there is no action name; skipping one file
+ # beats aborting the whole species on an IndexError.
+ logger.warning(f"Skipping {fbx_path}: filename has no '-' separating "
+ f"species from action (expected "
+ f"'{{Species}}-{{Action}}.fbx').")
+ continue
+
+ try:
+ result = extract_single_fbx(fbx_path, fps=fps, dtype=dtype)
+ except Exception as e:
+ logger.warning(f"Failed to extract {fbx_path}: {e}")
+ continue
+
+ nframes = result['anim'].positions.shape[0]
+ if nframes < min_frames:
+ logger.info(f"Skipping {fbx_path}: only {nframes} frames "
+ f"(min={min_frames})")
+ continue
+
+ sig = tuple(result['names'])
+ group = groups.setdefault(sig, {
+ 'rest_anim': result['rest_anim'],
+ 'names': list(result['names']),
+ 'skin_matrix': result['skin_matrix'],
+ 'extracted': [],
+ })
+ group['extracted'].append((action_name, result['anim']))
+
+ saved = {}
+ if groups:
+ by_size = sorted(groups.values(), key=lambda g: len(g['extracted']),
+ reverse=True)
+ if len(by_size) > 1:
+ logger.warning(f"'{obj_type}' has {len(by_size)} rig variants "
+ f"(clip counts: {[len(g['extracted']) for g in by_size]}); "
+ f"pruning each separately.")
+ for i, group in enumerate(by_size):
+ logger.info(f"Extracted {len(group['extracted'])} animations for "
+ f"'{obj_type}' (variant {i})")
+ result = postprocess_and_save(
+ obj_type, group['extracted'],
+ group['rest_anim'], group['names'], group['skin_matrix'],
+ output_dir, fps=fps,
+ consider_parent_rotate=consider_parent_rotate, save_vis=save_vis,
+ save_tpose=(i == 0),
+ )
+ if i == 0:
+ saved = result
+ else:
+ logger.info(f"No valid FBX files extracted for '{obj_type}'; skipping.")
+
+ mark_asset_complete(output_dir, obj_type,
+ status="exported" if saved else "skipped",
+ joint_names=saved.get(obj_type))
+ return saved
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Entry point
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_args():
+ parser = argparse.ArgumentParser(description="Export Truebones FBX clips to NPZ motion data.")
+ parser.add_argument('--data_dir', type=str, required=True,
+ help='Flat directory of per-clip {Species}-{Action}.fbx files.')
+ parser.add_argument('--output_dir', type=str, required=True,
+ help='Output directory (motions/, videos/, tpose/ created inside).')
+ parser.add_argument('--fps', type=int, default=30,
+ help='Scene FPS for import and export (dataset is 30 fps).')
+ parser.add_argument('--min_frames', type=int, default=5,
+ help='Skip clips shorter than this many frames '
+ '(shared minimum across all dataset exporters).')
+ parser.add_argument('--consider_parent_rotate', action=argparse.BooleanOptionalAction, default=True,
+ help="Preserve non-rotating leaves whose parent rotates during pruning "
+ "(see prune_skeleton_shared; use --no-consider_parent_rotate to disable).")
+ parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,
+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+def main():
+ args = parse_args()
+
+ fbx_files = sorted(f for f in os.listdir(args.data_dir) if f.lower().endswith('.fbx'))
+ by_species = {}
+ for f in fbx_files:
+ species = os.path.splitext(f)[0].split('-', 1)[0]
+ by_species.setdefault(species, []).append(os.path.join(args.data_dir, f))
+ logger.info(f"Found {len(fbx_files)} clips across {len(by_species)} species "
+ f"in {args.data_dir}")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir, 'export_errors.log')
+ all_joint_names = {}
+ n_failed = 0
+ for species, fbx_paths in tqdm(sorted(by_species.items()), desc="Exporting species"):
+ try:
+ saved = export_species(species, fbx_paths, output_dir=args.output_dir,
+ fps=args.fps, min_frames=args.min_frames,
+ consider_parent_rotate=args.consider_parent_rotate,
+ save_vis=args.vis)
+ if saved:
+ all_joint_names.update(saved)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Failed to export species {species}")
+ with open(error_log, 'a') as log_file:
+ log_file.write(f"Failed to export species {species}: {e}\n")
+
+ write_export_summary(args.output_dir, all_joint_names)
+ if n_failed:
+ logger.error(f"{n_failed}/{len(by_species)} species failed; see {error_log}")
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ # Blender exits 0 even on an uncaught exception, which hides failures from
+ # `set -e` and from Slurm's afterok dependencies.
+ try:
+ main()
+ except SystemExit:
+ raise
+ except Exception:
+ logger.exception("export_truebones failed")
+ sys.exit(1)
diff --git a/data_process/motion_rendering/__init__.py b/data_process/motion_rendering/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/motion_rendering/render_mixamo.py b/data_process/motion_rendering/render_mixamo.py
new file mode 100644
index 0000000..b991a34
--- /dev/null
+++ b/data_process/motion_rendering/render_mixamo.py
@@ -0,0 +1,174 @@
+"""Render multi-view frames for pre-animated Mixamo characters (EEVEE).
+
+Consumes animated GLB/FBX files produced by
+``mesh_animation/animate_mixamo.py`` (mesh + armature + baked animation)
+and renders each as ``{file_stem}/v00{0..3}/*.png`` โ€” the layout consumed
+by ``vlm_caption/caption_motion.py``.
+
+Character meshes are kept intact (body / hair / clothes are often separate
+meshes, so no main-mesh filtering is applied).
+
+Usage (plain python โ€” EEVEE needs the pip ``bpy`` module's GPU context):
+ python -m data_process.motion_rendering.render_mixamo \
+ --data_dir outputs/mixamo_characters --output_dir <render_dir>
+"""
+
+import os
+import sys
+import argparse
+from pathlib import Path
+
+import bpy
+from loguru import logger
+
+from data_process.utils.blender_export import discover_pose_actions
+from data_process.utils.blender_render import (
+ RENDERABLE_EXTS,
+ RENDER_FPS,
+ init_render_engine,
+ is_asset_render_skipped,
+ is_render_complete,
+ load_and_prep_asset,
+ mark_asset_render_skipped,
+ render_action_multiview,
+)
+
+
+def render_animated_file(path, output_dir, args):
+ """Render multi-view frames for one pre-animated character file."""
+ save_name = Path(path).stem
+ save_root = os.path.join(output_dir, save_name)
+
+ if is_render_complete(save_root, require_video=args.video):
+ logger.info(f"Skipping complete: {save_root}")
+ return
+ if is_asset_render_skipped(output_dir, save_name):
+ logger.info(f"Previously marked as no-render, skipping: {path}")
+ return
+
+ logger.info(f"Processing: {path}")
+
+ armature = load_and_prep_asset(str(path), fps=args.fps)
+ if armature is None:
+ logger.warning(f"No armature/mesh in {path}; skipping.")
+ mark_asset_render_skipped(output_dir, save_name, reason="no armature/mesh")
+ return
+
+ # The animated file carries exactly one baked action: the one bound to
+ # the armature. Falling back to a different action would render a motion
+ # the matching NPZ never contains.
+ action_names, frame_ranges = discover_pose_actions(min_frames=args.min_action_frames)
+ if not action_names:
+ logger.warning(f"No animation data in {path}; skipping.")
+ mark_asset_render_skipped(output_dir, save_name, reason="no pose actions")
+ return
+
+ bound = None
+ if armature.animation_data and armature.animation_data.action:
+ bound = armature.animation_data.action.name
+ if bound not in action_names:
+ logger.warning(
+ f"Bound action {bound!r} is not among the renderable pose actions "
+ f"{action_names} for {path} (too short, or none bound); skipping "
+ f"rather than rendering a different action than the export used.")
+ mark_asset_render_skipped(output_dir, save_name,
+ reason=f"bound action {bound!r} not renderable")
+ return
+ action_name = bound
+
+ start, end = frame_ranges[action_name]
+ logger.info(f"Rendering '{action_name}' -> '{save_name}' frames=[{start},{end}]")
+
+ render_action_multiview(
+ armature=armature,
+ action_name=action_name,
+ start_frame=start,
+ end_frame=end,
+ save_root=save_root,
+ camera_dist=args.camera_dist,
+ scene_scale=args.scale,
+ resolution=args.resolution,
+ max_render_frames=args.max_render_frames,
+ compose_video=args.video,
+ fps=args.fps,
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Render multi-view frames for animated Mixamo characters (EEVEE).")
+ parser.add_argument("--data_dir", type=str, required=True,
+ help="Directory of animated GLB/FBX files from "
+ "mesh_animation/animate_mixamo.py.")
+ parser.add_argument("--output_dir", type=str, required=True,
+ help="Output directory.")
+ parser.add_argument("--scale", type=float, default=1.0,
+ help="Scene normalization scale.")
+ parser.add_argument("--camera_dist", type=float, default=1.5,
+ help="Camera distance from origin.")
+ parser.add_argument("--resolution", type=int, default=512,
+ help="Render resolution (square).")
+ parser.add_argument("--fps", type=int, default=RENDER_FPS,
+ help="Scene FPS for import and preview video "
+ "(must match the export stage's --fps).")
+ parser.add_argument("--samples", type=int, default=64,
+ help="EEVEE TAA render samples.")
+ parser.add_argument("--max_render_frames", type=int, default=200,
+ help="Cap on rendered frames per clip (per view).")
+ parser.add_argument("--min_action_frames", type=int, default=5,
+ help="Skip actions shorter than this many frames "
+ "(default 5 โ€” the shared minimum clip length "
+ "across export and rendering).")
+ parser.add_argument("--video", action=argparse.BooleanOptionalAction, default=True,
+ help="Compose each view's frames into v00x.mp4 "
+ "(use --no-video to skip).")
+ parser.add_argument("--worker_id", type=int, default=0,
+ help="Worker index for sharding across workers.")
+ parser.add_argument("--num_workers", type=int, default=1,
+ help="Total number of workers.")
+ parser.add_argument("--gpu_id", type=int, default=None,
+ help="GPU device ID to pin this worker to "
+ "(default: inherit CUDA_VISIBLE_DEVICES).")
+ args = parser.parse_args()
+
+ if args.gpu_id is not None:
+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)
+
+ init_render_engine(render_samples=args.samples)
+
+ data_dir = Path(args.data_dir)
+ files = sorted(p for p in data_dir.iterdir()
+ if p.is_file() and p.suffix.lower() in RENDERABLE_EXTS)
+ logger.info(f"Found {len(files)} animated files in {data_dir}")
+
+ if args.num_workers > 1:
+ files = files[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(files)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'render_errors_worker{args.worker_id}.log')
+
+ n_failed = 0
+ for path in files:
+ try:
+ render_animated_file(path, args.output_dir, args)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Failed to render {path}")
+ with open(error_log, 'a') as f:
+ f.write(f"{path}\t{e}\n")
+
+ if n_failed:
+ logger.error(f"{n_failed}/{len(files)} clips failed; see {error_log}")
+ return 1 if n_failed else 0
+
+
+if __name__ == "__main__":
+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures
+ # from `set -e` and from Slurm's afterok dependencies.
+ try:
+ sys.exit(main())
+ except Exception:
+ logger.exception("render_mixamo failed")
+ sys.exit(1)
diff --git a/data_process/motion_rendering/render_objaverse.py b/data_process/motion_rendering/render_objaverse.py
new file mode 100644
index 0000000..8b7c9d7
--- /dev/null
+++ b/data_process/motion_rendering/render_objaverse.py
@@ -0,0 +1,241 @@
+"""Render multi-view frames for Objaverse GLB/GLTF animations (EEVEE).
+
+For each GLB, discovers all pose actions and renders each one as
+``{glb_stem}-{action}/v00{0..3}/*.png`` โ€” the layout consumed by
+``vlm_caption/caption_motion.py``.
+
+Usage (plain python โ€” EEVEE needs the pip ``bpy`` module's GPU context):
+ python -m data_process.motion_rendering.render_objaverse \
+ --data_dir dataset/raw/objaverse/glb --output_dir <render_dir>
+"""
+
+import os
+import sys
+import argparse
+from pathlib import Path
+
+import bpy
+from loguru import logger
+
+from data_process.utils.blender_export import (
+ discover_pose_actions,
+ list_gltf_files,
+ unique_clip_names,
+)
+from data_process.utils.blender_render import (
+ RENDER_FPS,
+ get_expected_clip_dirs,
+ init_render_engine,
+ is_asset_render_skipped,
+ is_render_complete,
+ load_and_prep_asset,
+ mark_asset_render_done,
+ mark_asset_render_skipped,
+ render_action_multiview,
+)
+
+
+def find_pending_stems(output_dir, gltf_stems, require_video=False):
+ """Return the subset of GLTF stems that still need rendering.
+
+ A stem is done only when a ``.rendered/`` marker lists the clip
+ directories it is expected to produce AND every one of them passes
+ :func:`is_render_complete`. Requiring the marker is what distinguishes
+ "all actions rendered" from "a worker was killed partway through": the
+ directories that do exist are complete in both cases, so judging by
+ on-disk directories alone would retire the asset with actions still
+ unrendered, permanently. Stems that are done skip the expensive GLB load.
+
+ Action folders are named ``{stem}-{sanitized_action}``; stems may
+ themselves contain ``-``, so directory names are matched against the
+ known stem set by longest-prefix-at-hyphen rather than splitting on
+ the first ``-``.
+ """
+ output_dir = Path(output_dir)
+ stems_set = set(gltf_stems)
+ if not output_dir.is_dir():
+ return stems_set
+
+ # Assets marked as producing no renders (no armature / actions) are done.
+ stems_set = {s for s in stems_set
+ if not is_asset_render_skipped(str(output_dir), s)}
+
+ pending = set()
+ for stem in stems_set:
+ expected = get_expected_clip_dirs(str(output_dir), stem)
+ if not expected:
+ # Never finished an end-to-end pass (or predates the marker):
+ # re-open it. Completed clip dirs are still skipped per-action.
+ pending.add(stem)
+ continue
+ if not all(is_render_complete(str(output_dir / d), require_video=require_video)
+ for d in expected):
+ pending.add(stem)
+
+ return pending
+
+
+def render_glb(glb_path, output_dir, args):
+ """Render multi-view frames for every pose action in one GLB/GLTF file."""
+ save_name = Path(glb_path).stem
+ if is_asset_render_skipped(output_dir, save_name):
+ logger.info(f"Previously marked as no-render, skipping: {glb_path}")
+ return
+
+ logger.info(f"Processing: {glb_path}")
+
+ armature = load_and_prep_asset(str(glb_path), fps=args.fps)
+ if armature is None:
+ logger.warning(f"No armature/mesh in {glb_path}; skipping.")
+ mark_asset_render_skipped(output_dir, save_name, reason="no armature/mesh")
+ return
+
+ # Names are derived from the UNFILTERED action list so they match the
+ # export stage, which applies no frame-count filter โ€” deriving them from
+ # a filtered subset would shift the collision suffixes out of step.
+ all_action_names, frame_ranges = discover_pose_actions()
+ clip_suffixes = unique_clip_names(all_action_names)
+
+ action_names = [
+ n for n in all_action_names
+ if (frame_ranges[n][1] - frame_ranges[n][0] + 1) >= args.min_action_frames
+ ]
+ logger.info(f"Found {len(action_names)} pose actions "
+ f"({len(all_action_names)} before the frame-count filter)")
+ if not action_names:
+ mark_asset_render_skipped(output_dir, save_name, reason="no pose actions")
+ return
+
+ expected_dirs = []
+ for action_name in action_names:
+ clean_action_name = clip_suffixes[action_name]
+ clip_dir_name = f"{save_name}-{clean_action_name}"
+ action_dir = os.path.join(output_dir, clip_dir_name)
+ expected_dirs.append(clip_dir_name)
+
+ if is_render_complete(action_dir, require_video=args.video):
+ logger.info(f"Skipping complete: {action_dir}")
+ continue
+
+ start, end = frame_ranges[action_name]
+ logger.info(f"Rendering '{action_name}' -> '{clip_dir_name}' "
+ f"frames=[{start},{end}]")
+
+ render_action_multiview(
+ armature=armature,
+ action_name=action_name,
+ start_frame=start,
+ end_frame=end,
+ save_root=action_dir,
+ camera_dist=args.camera_dist,
+ scene_scale=args.scale,
+ resolution=args.resolution,
+ max_render_frames=args.max_render_frames,
+ compose_video=args.video,
+ fps=args.fps,
+ )
+
+ # Only now is the asset known to have had every action handled.
+ mark_asset_render_done(output_dir, save_name, expected_dirs)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Render multi-view frames for Objaverse GLB/GLTF animations (EEVEE).")
+ parser.add_argument("--data_dir", type=str, required=True,
+ help="Directory containing GLB/GLTF files.")
+ parser.add_argument("--obj_name", type=str, default=None,
+ help="Single GLB stem to render (default: all files).")
+ parser.add_argument("--output_dir", type=str, required=True,
+ help="Output directory.")
+ parser.add_argument("--scale", type=float, default=1.0,
+ help="Scene normalization scale.")
+ parser.add_argument("--camera_dist", type=float, default=1.5,
+ help="Camera distance from origin.")
+ parser.add_argument("--resolution", type=int, default=512,
+ help="Render resolution (square).")
+ parser.add_argument("--fps", type=int, default=RENDER_FPS,
+ help="Scene FPS for import and preview video. Must match "
+ "the export stage's --fps: glTF keyframes are stored "
+ "in seconds, so the importer resamples at this rate.")
+ parser.add_argument("--samples", type=int, default=64,
+ help="EEVEE TAA render samples.")
+ parser.add_argument("--max_render_frames", type=int, default=200,
+ help="Cap on rendered frames per clip (per view).")
+ parser.add_argument("--min_action_frames", type=int, default=5,
+ help="Skip actions shorter than this many raw frames "
+ "(default 5 โ€” the shared minimum clip length "
+ "across export and rendering; raw length >= "
+ "post-T-pose-removal length, so no exported clip "
+ "is ever skipped).")
+ parser.add_argument("--video", action=argparse.BooleanOptionalAction, default=True,
+ help="Compose each view's frames into v00x.mp4 "
+ "(use --no-video to skip).")
+ parser.add_argument("--missing-only", dest="missing_only", action="store_true",
+ help="Scan output_dir first; only load GLBs that have "
+ "at least one missing or incomplete action folder.")
+ parser.add_argument("--worker_id", type=int, default=0,
+ help="Worker index for sharding across workers.")
+ parser.add_argument("--num_workers", type=int, default=1,
+ help="Total number of workers.")
+ parser.add_argument("--gpu_id", type=int, default=None,
+ help="GPU device ID to pin this worker to "
+ "(default: inherit CUDA_VISIBLE_DEVICES).")
+ args = parser.parse_args()
+
+ if args.gpu_id is not None:
+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)
+
+ init_render_engine(render_samples=args.samples)
+
+ data_dir = Path(args.data_dir)
+ if args.obj_name:
+ glb_path = data_dir / f"{args.obj_name}.glb"
+ if not glb_path.is_file():
+ glb_path = data_dir / f"{args.obj_name}.gltf"
+ if not glb_path.is_file():
+ parser.error(f"Cannot find {args.obj_name}.glb or .gltf in {data_dir}")
+ gltf_paths = [glb_path]
+ else:
+ gltf_paths = list_gltf_files(data_dir)
+ logger.info(f"Found {len(gltf_paths)} GLB/GLTF files in {data_dir}")
+
+ if args.missing_only:
+ pending = find_pending_stems(args.output_dir, [p.stem for p in gltf_paths],
+ require_video=args.video)
+ before = len(gltf_paths)
+ gltf_paths = [p for p in gltf_paths if p.stem in pending]
+ logger.info(f"[missing-only] {len(gltf_paths)}/{before} GLBs have pending renders.")
+
+ if args.num_workers > 1:
+ gltf_paths = gltf_paths[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "
+ f"{len(gltf_paths)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'render_errors_worker{args.worker_id}.log')
+
+ n_failed = 0
+ for glb_path in gltf_paths:
+ try:
+ render_glb(glb_path, args.output_dir, args)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Failed to render {glb_path}")
+ with open(error_log, 'a') as f:
+ f.write(f"{glb_path}\t{e}\n")
+
+ if n_failed:
+ logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")
+ return 1 if n_failed else 0
+
+
+if __name__ == "__main__":
+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures
+ # from `set -e` and from Slurm's afterok dependencies.
+ try:
+ sys.exit(main())
+ except Exception:
+ logger.exception("render_objaverse failed")
+ sys.exit(1)
diff --git a/data_process/motion_rendering/render_tpose.py b/data_process/motion_rendering/render_tpose.py
new file mode 100644
index 0000000..eb2cd58
--- /dev/null
+++ b/data_process/motion_rendering/render_tpose.py
@@ -0,0 +1,168 @@
+"""Render T-pose 2x2 grid images for rigged assets (EEVEE).
+
+One PNG per asset, saved flat as ``{output_dir}/{name}.png`` (front/back
+top row, left/right bottom row) โ€” the "flat" layout consumed by
+``vlm_caption/classify_category.py``.
+
+Two input layouts are auto-detected per entry of ``--data_dir``:
+ - files: ``<data_dir>/*.{glb,gltf,fbx}`` -> one grid per file stem
+ - subdirectory: ``<data_dir>/<obj_type>/*.fbx`` -> one grid per object
+ type, rendered from its main (all-in-one) FBX โ€” Truebones layout.
+
+Usage (plain python โ€” EEVEE needs the pip ``bpy`` module's GPU context):
+ python -m data_process.motion_rendering.render_tpose \
+ --data_dir dataset/raw/objaverse/glb --output_dir <tpose_dir>
+"""
+
+import os
+import sys
+import argparse
+from pathlib import Path
+
+from loguru import logger
+
+from data_process.utils.blender_export import find_main_fbx, sanitize_object_type
+from data_process.utils.blender_render import (
+ RENDERABLE_EXTS,
+ RENDER_FPS,
+ init_render_engine,
+ load_and_prep_asset,
+ render_tpose_grid,
+)
+
+
+def discover_assets(data_dir, species_grids=False):
+ """Return (name, path) pairs to render, auto-detecting the layout.
+
+ Files directly under *data_dir* are rendered by stem; each
+ subdirectory is treated as a Truebones-style object type and rendered
+ from its main FBX (fallback: first FBX in the subdirectory).
+
+ With *species_grids*, files named ``{Species}-{Action}.*`` are grouped
+ by the stem's prefix before the first ``-`` and one grid is rendered
+ per species, from its first file (curated Truebones layout).
+ """
+ data_dir = Path(data_dir)
+ assets = []
+ seen_species = set()
+
+ for p in sorted(data_dir.iterdir()):
+ if p.is_file() and p.suffix.lower() in RENDERABLE_EXTS:
+ if species_grids:
+ species = p.stem.split('-', 1)[0]
+ if species in seen_species:
+ continue
+ seen_species.add(species)
+ assets.append((sanitize_object_type(species), p))
+ else:
+ assets.append((p.stem, p))
+ elif p.is_dir():
+ main_fbx = find_main_fbx(str(p))
+ if main_fbx is None:
+ fbxs = sorted(f for f in p.iterdir()
+ if f.suffix.lower() == ".fbx")
+ main_fbx = str(fbxs[0]) if fbxs else None
+ if main_fbx is None:
+ logger.warning(f"No FBX found in {p}; skipping.")
+ continue
+ assets.append((sanitize_object_type(p.name), Path(main_fbx)))
+
+ return assets
+
+
+def render_asset_tpose(name, path, output_dir, args):
+ """Render the T-pose grid for one asset to ``{output_dir}/{name}.png``."""
+ tpose_path = os.path.join(output_dir, f"{name}.png")
+ if os.path.isfile(tpose_path):
+ logger.info(f"Skipping complete: {tpose_path}")
+ return
+
+ logger.info(f"Processing: {path} -> {name}")
+
+ armature = load_and_prep_asset(str(path),
+ keep_only_main_mesh=args.keep_main_mesh_only,
+ fps=args.fps)
+ if armature is None:
+ logger.warning(f"No armature/mesh in {path}; skipping.")
+ return
+
+ render_tpose_grid(
+ save_path=tpose_path,
+ camera_dist=args.camera_dist,
+ scene_scale=args.scale,
+ resolution=args.resolution,
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Render T-pose 2x2 grid images for rigged assets (EEVEE).")
+ parser.add_argument("--data_dir", type=str, required=True,
+ help="Directory of GLB/GLTF/FBX files, or of "
+ "per-object-type FBX subdirectories (Truebones).")
+ parser.add_argument("--obj_name", type=str, default=None,
+ help="Single asset name (file stem or object type) "
+ "to render (default: all).")
+ parser.add_argument("--output_dir", type=str, required=True,
+ help="Flat directory for the <name>.png grids.")
+ parser.add_argument("--scale", type=float, default=1.0,
+ help="Scene normalization scale.")
+ parser.add_argument("--camera_dist", type=float, default=1.5,
+ help="Camera distance from origin.")
+ parser.add_argument("--fps", type=int, default=RENDER_FPS,
+ help="Scene FPS for import (must match the export "
+ "stage's --fps).")
+ parser.add_argument("--resolution", type=int, default=512,
+ help="Render resolution per view (square).")
+ parser.add_argument("--samples", type=int, default=64,
+ help="EEVEE TAA render samples.")
+ parser.add_argument("--keep-all-meshes", dest="keep_main_mesh_only",
+ action="store_false",
+ help="Keep every mesh instead of only the "
+ "highest-vertex one (multi-mesh characters).")
+ parser.add_argument("--species_grids", action="store_true",
+ help="Group {Species}-{Action}.fbx files by species "
+ "and render one grid per species (curated "
+ "Truebones layout).")
+ parser.add_argument("--worker_id", type=int, default=0,
+ help="Worker index for sharding across workers.")
+ parser.add_argument("--num_workers", type=int, default=1,
+ help="Total number of workers.")
+ parser.add_argument("--gpu_id", type=int, default=None,
+ help="GPU device ID to pin this worker to "
+ "(default: inherit CUDA_VISIBLE_DEVICES).")
+ args = parser.parse_args()
+
+ if args.gpu_id is not None:
+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)
+
+ init_render_engine(render_samples=args.samples)
+
+ assets = discover_assets(args.data_dir, species_grids=args.species_grids)
+ logger.info(f"Found {len(assets)} assets in {args.data_dir}")
+
+ if args.obj_name:
+ assets = [(n, p) for n, p in assets if n == args.obj_name]
+ if not assets:
+ parser.error(f"Asset '{args.obj_name}' not found in {args.data_dir}")
+
+ if args.num_workers > 1:
+ assets = assets[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "
+ f"{len(assets)} assets")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'tpose_errors_worker{args.worker_id}.log')
+
+ for name, path in assets:
+ try:
+ render_asset_tpose(name, path, args.output_dir, args)
+ except Exception as e:
+ logger.error(f"Failed to render {path}: {e}")
+ with open(error_log, 'a') as f:
+ f.write(f"{path}\t{e}\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/motion_rendering/render_truebones.py b/data_process/motion_rendering/render_truebones.py
new file mode 100644
index 0000000..4570e99
--- /dev/null
+++ b/data_process/motion_rendering/render_truebones.py
@@ -0,0 +1,180 @@
+"""Render multi-view frames for Truebones FBX animations (EEVEE).
+
+Consumes the curated flat layout of the Truebones ZOO dataset: one FBX per
+clip, named ``{Species}-{Action}.fbx``, all in a single directory (the same
+input as ``motion_export/export_truebones.py``). Each file is rendered as
+``{file_stem}/v00{0..3}/*.png`` โ€” the layout consumed by
+``vlm_caption/caption_motion.py``.
+
+Usage (plain python โ€” EEVEE needs the pip ``bpy`` module's GPU context):
+ python -m data_process.motion_rendering.render_truebones \
+ --data_dir dataset/raw/truebones/animation --output_dir <render_dir>
+"""
+
+import os
+import sys
+import argparse
+from pathlib import Path
+
+import bpy
+from loguru import logger
+
+from data_process.utils.blender_export import (
+ discover_pose_actions,
+ truebones_clip_name,
+)
+from data_process.utils.blender_render import (
+ RENDER_FPS,
+ init_render_engine,
+ is_asset_render_skipped,
+ is_render_complete,
+ load_and_prep_asset,
+ mark_asset_render_skipped,
+ render_action_multiview,
+)
+
+
+def render_clip_file(path, output_dir, args):
+ """Render multi-view frames for one per-clip FBX file."""
+ # Must match export_truebones' clip naming exactly โ€” stage 4 looks the
+ # caption up by the NPZ stem with no fuzzy matching, so a raw stem here
+ # (spaces, extra dashes) silently produces an uncaptioned clip.
+ save_name = truebones_clip_name(Path(path).stem)
+ save_root = os.path.join(output_dir, save_name)
+
+ if is_render_complete(save_root, require_video=args.video):
+ logger.info(f"Skipping complete: {save_root}")
+ return
+ if is_asset_render_skipped(output_dir, save_name):
+ logger.info(f"Previously marked as no-render, skipping: {path}")
+ return
+
+ logger.info(f"Processing: {path}")
+
+ armature = load_and_prep_asset(str(path), fps=args.fps)
+ if armature is None:
+ logger.warning(f"No armature/mesh in {path}; skipping.")
+ mark_asset_render_skipped(output_dir, save_name, reason="no armature/mesh")
+ return
+
+ # Each curated file carries exactly one take: the action bound to the
+ # armature. Export always uses that one, so falling back to a different
+ # action here would render a motion the NPZ never contains.
+ action_names, frame_ranges = discover_pose_actions(min_frames=args.min_action_frames)
+ if not action_names:
+ logger.warning(f"No animation data in {path}; skipping.")
+ mark_asset_render_skipped(output_dir, save_name, reason="no pose actions")
+ return
+
+ bound = None
+ if armature.animation_data and armature.animation_data.action:
+ bound = armature.animation_data.action.name
+ if bound not in action_names:
+ logger.warning(
+ f"Bound action {bound!r} is not among the renderable pose actions "
+ f"{action_names} for {path} (too short, or none bound); skipping "
+ f"rather than rendering a different action than the export used.")
+ mark_asset_render_skipped(output_dir, save_name,
+ reason=f"bound action {bound!r} not renderable")
+ return
+ action_name = bound
+
+ start, end = frame_ranges[action_name]
+ logger.info(f"Rendering '{action_name}' -> '{save_name}' frames=[{start},{end}]")
+
+ render_action_multiview(
+ armature=armature,
+ action_name=action_name,
+ start_frame=start,
+ end_frame=end,
+ save_root=save_root,
+ camera_dist=args.camera_dist,
+ scene_scale=args.scale,
+ resolution=args.resolution,
+ max_render_frames=args.max_render_frames,
+ compose_video=args.video,
+ fps=args.fps,
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Render multi-view frames for Truebones FBX animations (EEVEE).")
+ parser.add_argument("--data_dir", type=str, required=True,
+ help="Flat directory of per-clip {Species}-{Action}.fbx files.")
+ parser.add_argument("--obj_type", type=str, default=None,
+ help="Single species to render (default: all).")
+ parser.add_argument("--output_dir", type=str, required=True,
+ help="Output directory.")
+ parser.add_argument("--scale", type=float, default=1.0,
+ help="Scene normalization scale.")
+ parser.add_argument("--camera_dist", type=float, default=1.5,
+ help="Camera distance from origin.")
+ parser.add_argument("--resolution", type=int, default=512,
+ help="Render resolution (square).")
+ parser.add_argument("--fps", type=int, default=RENDER_FPS,
+ help="Scene FPS for import and preview video "
+ "(must match the export stage's --fps).")
+ parser.add_argument("--samples", type=int, default=64,
+ help="EEVEE TAA render samples.")
+ parser.add_argument("--max_render_frames", type=int, default=200,
+ help="Cap on rendered frames per clip (per view).")
+ parser.add_argument("--min_action_frames", type=int, default=5,
+ help="Skip actions shorter than this many frames "
+ "(default 5 โ€” the shared minimum clip length "
+ "across export and rendering).")
+ parser.add_argument("--video", action=argparse.BooleanOptionalAction, default=True,
+ help="Compose each view's frames into v00x.mp4 "
+ "(use --no-video to skip).")
+ parser.add_argument("--worker_id", type=int, default=0,
+ help="Worker index for sharding across workers.")
+ parser.add_argument("--num_workers", type=int, default=1,
+ help="Total number of workers.")
+ parser.add_argument("--gpu_id", type=int, default=None,
+ help="GPU device ID to pin this worker to "
+ "(default: inherit CUDA_VISIBLE_DEVICES).")
+ args = parser.parse_args()
+
+ if args.gpu_id is not None:
+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)
+
+ init_render_engine(render_samples=args.samples)
+
+ data_dir = Path(args.data_dir)
+ files = sorted(p for p in data_dir.iterdir()
+ if p.is_file() and p.suffix.lower() == ".fbx")
+ if args.obj_type:
+ files = [p for p in files if p.stem.split('-', 1)[0] == args.obj_type]
+ logger.info(f"Found {len(files)} clip files in {data_dir}")
+
+ if args.num_workers > 1:
+ files = files[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(files)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'render_errors_worker{args.worker_id}.log')
+
+ n_failed = 0
+ for path in files:
+ try:
+ render_clip_file(path, args.output_dir, args)
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Failed to render {path}")
+ with open(error_log, 'a') as f:
+ f.write(f"{path}\t{e}\n")
+
+ if n_failed:
+ logger.error(f"{n_failed}/{len(files)} clips failed; see {error_log}")
+ return 1 if n_failed else 0
+
+
+if __name__ == "__main__":
+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures
+ # from `set -e` and from Slurm's afterok dependencies.
+ try:
+ sys.exit(main())
+ except Exception:
+ logger.exception("render_truebones failed")
+ sys.exit(1)
diff --git a/data_process/scripts/_common.sh b/data_process/scripts/_common.sh
new file mode 100644
index 0000000..77c22fd
--- /dev/null
+++ b/data_process/scripts/_common.sh
@@ -0,0 +1,84 @@
+# shellcheck shell=bash
+# Shared prologue for every data_process/scripts/run_*.sh wrapper.
+#
+# source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+#
+# Resolves PROJECT_ROOT (the repository root), cds into it, activates the
+# conda environment named by $CONDA_ENV (default: unimate), and defines the
+# default dataset layout used by every stage. Override any DEFAULT_* path by
+# exporting the corresponding variable before calling a wrapper.
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+# Resolve the sourcing wrapper's own path *before* the cd below, so that
+# print_usage still finds it when the wrapper was invoked by a relative path
+# (e.g. `cd data_process/scripts && bash run_export.sh -h`).
+WRAPPER_PATH="${BASH_SOURCE[1]:-$0}"
+[[ "$WRAPPER_PATH" == /* ]] || WRAPPER_PATH="$PWD/$WRAPPER_PATH"
+cd "$PROJECT_ROOT"
+
+CONDA_ENV=${CONDA_ENV:-unimate}
+if command -v conda >/dev/null 2>&1; then
+ # conda activation hooks reference unset variables; relax `set -u` around them.
+ case $- in *u*) _had_nounset=1 ;; *) _had_nounset=0 ;; esac
+ set +u
+ eval "$(conda shell.bash hook 2>/dev/null)"
+ conda activate "$CONDA_ENV" 2>/dev/null \
+ || echo "warning: could not activate conda env '$CONDA_ENV'; using current python" >&2
+ [[ "$_had_nounset" == 1 ]] && set -u
+fi
+
+# Avoid user startup hooks leaking into Blender / worker processes.
+unset PYTHONSTARTUP PYTHONBREAKPOINT
+
+# โ”€โ”€ Default dataset layout (relative to PROJECT_ROOT) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# dataset/raw/<dataset>/ raw FBX / GLB assets
+# dataset/export/<dataset>/ stage-1 export + stage-2/3 metadata
+# dataset/render/<dataset>/ multi-view renders (captioning input)
+# dataset/render/<dataset>_tpose/ T-pose grids (category input)
+# dataset/features/<dataset>/ stage-4 training clips + cond.npy
+export_dir() { echo "dataset/export/$1"; }
+render_dir() { echo "dataset/render/$1"; }
+tpose_dir() { echo "dataset/render/${1}_tpose"; }
+features_dir() { echo "dataset/features/$1"; }
+
+# โ”€โ”€ Helpers โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# require_dataset <name> : validate a dataset name.
+require_dataset() {
+ case "$1" in
+ truebones|mixamo|objaverse) ;;
+ *) echo "Unknown dataset '$1' (expected: truebones | mixamo | objaverse)" >&2; exit 2 ;;
+ esac
+}
+
+# require_env VAR : abort if the environment variable is unset/empty.
+require_env() {
+ if [[ -z "${!1:-}" ]]; then
+ echo "ERROR: $1 is not set. Export it in your shell." >&2
+ exit 1
+ fi
+}
+
+# print_usage [script] : print the leading comment block of a wrapper
+# (defaults to the wrapper that sourced this file).
+print_usage() {
+ sed -n '2,/^$/p' "${1:-$WRAPPER_PATH}" | sed 's/^# \{0,1\}//'
+}
+
+# handle_help "$@" : print the wrapper usage and exit 0 when the first
+# argument is -h/--help. Call it right after sourcing this file, before any
+# other argument validation, so `-h` works on every wrapper.
+handle_help() {
+ case "${1:-}" in
+ -h|--help) print_usage; exit 0 ;;
+ esac
+}
+
+# worker_log_dir <name> : create and echo a directory for per-worker logs, so
+# that a failing shard of a parallel run can be attributed. Override with
+# WORKER_LOG_DIR.
+worker_log_dir() {
+ local dir="${WORKER_LOG_DIR:-outputs/logs/$1}"
+ mkdir -p "$dir"
+ echo "$dir"
+}
diff --git a/data_process/scripts/run_animate_fbx.sh b/data_process/scripts/run_animate_fbx.sh
new file mode 100755
index 0000000..a0c2fac
--- /dev/null
+++ b/data_process/scripts/run_animate_fbx.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+# Stage 5 โ€” bake raw animation FBX/GLB clips onto a chosen rigged character
+# (action transfer by bone name โ€” made for Mixamo-style shared skeletons,
+# where any clip can drive any compatible character without remapping).
+#
+# Usage:
+# bash data_process/scripts/run_animate_fbx.sh # Y Bot x animation_motion/
+# CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \
+# bash data_process/scripts/run_animate_fbx.sh # whole library on Amy
+# ANIM_PATH=dataset/raw/mixamo/animation_motion/Jab_Cross.fbx \
+# CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \
+# bash data_process/scripts/run_animate_fbx.sh # a single clip
+#
+# Env overrides:
+# CHAR_PATH rigged character FBX/GLB (default: character_refined/Y_Bot.fbx)
+# ANIM_PATH animation clip or directory of clips
+# (default: dataset/raw/mixamo/animation_motion)
+# OUTPUT_DIR destination directory (default: outputs/animated_<character stem>)
+# CHAR_ANIM_TYPE glb|fbx (default: fbx)
+#
+# Extra arguments are passed through to animate_fbx.py (e.g. --overwrite).
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+CHAR_PATH=${CHAR_PATH:-dataset/raw/mixamo/character_refined/Y_Bot.fbx}
+if [[ ! -f "$CHAR_PATH" ]]; then
+ echo "Character file not found: $CHAR_PATH" >&2
+ echo "Download the mixamo raw data (run_download.sh mixamo) or set CHAR_PATH." >&2
+ exit 1
+fi
+ANIM_PATH=${ANIM_PATH:-dataset/raw/mixamo/animation_motion}
+CHAR_STEM=$(basename "${CHAR_PATH%.*}")
+OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated_${CHAR_STEM}}
+CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-fbx}
+
+blender -b -P data_process/mesh_animation/animate_fbx.py -- \
+ --char_path="$CHAR_PATH" \
+ --anim_path="$ANIM_PATH" \
+ --output_dir="$OUTPUT_DIR" \
+ --char_anim_type="$CHAR_ANIM_TYPE" "$@"
diff --git a/data_process/scripts/run_animate_mixamo.sh b/data_process/scripts/run_animate_mixamo.sh
new file mode 100755
index 0000000..7f69a6c
--- /dev/null
+++ b/data_process/scripts/run_animate_mixamo.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+# Stage 5 โ€” batch-animate a Mixamo character mesh with a directory of exported
+# motion NPZs (raw Mixamo animations carry no mesh, so a character FBX is required).
+#
+# Usage:
+# bash data_process/scripts/run_animate_mixamo.sh # default character: Y Bot
+# CHAR_PATH=char.fbx bash data_process/scripts/run_animate_mixamo.sh
+#
+# Env overrides:
+# CHAR_PATH rigged Mixamo character FBX (default: Y_Bot โ€” the character
+# whose skeleton the animation FBXs are authored on)
+# NPZ_DIR (default: dataset/export/mixamo/motions)
+# OUTPUT_DIR (default: outputs/mixamo_characters)
+# CHAR_ANIM_TYPE glb|fbx (default: fbx)
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DEFAULT_CHAR=dataset/raw/mixamo/character_refined/Y_Bot.fbx
+CHAR_PATH=${CHAR_PATH:-$DEFAULT_CHAR}
+if [[ ! -f "$CHAR_PATH" ]]; then
+ echo "Character FBX not found: $CHAR_PATH" >&2
+ echo "Download the mixamo raw data (run_download.sh mixamo) or set CHAR_PATH." >&2
+ exit 1
+fi
+NPZ_DIR=${NPZ_DIR:-$(export_dir mixamo)/motions}
+OUTPUT_DIR=${OUTPUT_DIR:-outputs/mixamo_characters}
+CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-fbx}
+
+blender -b -P data_process/mesh_animation/animate_mixamo.py -- \
+ --char_path="$CHAR_PATH" \
+ --npz_dir="$NPZ_DIR" \
+ --output_dir="$OUTPUT_DIR" \
+ --char_anim_type="$CHAR_ANIM_TYPE" "$@"
diff --git a/data_process/scripts/run_animate_motion.sh b/data_process/scripts/run_animate_motion.sh
new file mode 100755
index 0000000..295de1d
--- /dev/null
+++ b/data_process/scripts/run_animate_motion.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Stage 5 โ€” drive a rigged character with a canonicalized motion clip (stage-4
+# format, or a generated motion in that format) and export GLB + FBX. The
+# character is auto-resolved from the dataset + clip name for truebones and
+# objaverse; mixamo needs CHAR_PATH.
+#
+# Usage:
+# ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse
+# ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh truebones
+# ANIM_PATH=clip.npz CHAR_PATH=char.fbx bash data_process/scripts/run_animate_motion.sh mixamo
+#
+# Env overrides:
+# ANIM_PATH (required) motion .npz (or legacy .npy)
+# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; required for mixamo)
+# COND_PATH cond.npy (default: dataset/features/<dataset>/cond.npy)
+# OUTPUT_DIR (default: outputs/animated)
+# ANIM_MODE fk|ik, legacy .npy only (default: fk)
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_animate_motion.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion .npz/.npy for dataset=$DATASET}
+OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}
+ANIM_MODE=${ANIM_MODE:-fk}
+
+EXTRA_ARGS=()
+[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")
+[[ -n "${COND_PATH:-}" ]] && EXTRA_ARGS+=(--cond_path="$COND_PATH")
+if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then
+ echo "dataset=mixamo requires CHAR_PATH (no auto-resolved character mesh)" >&2
+ exit 1
+fi
+
+blender -b -P data_process/mesh_animation/animate_motion.py -- \
+ --dataset_type="$DATASET" \
+ --anim_path="$ANIM_PATH" \
+ --output_dir="$OUTPUT_DIR" \
+ --anim_mode="$ANIM_MODE" \
+ "${EXTRA_ARGS[@]}" "$@"
diff --git a/data_process/scripts/run_animate_npz.sh b/data_process/scripts/run_animate_npz.sh
new file mode 100755
index 0000000..6fe9f9b
--- /dev/null
+++ b/data_process/scripts/run_animate_npz.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+# Stage 5 โ€” drive a rigged character with an *exported* motion NPZ (stage-1
+# format) and save the animated character as GLB/FBX.
+#
+# Usage:
+# CHAR_PATH=char.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh
+#
+# Env overrides:
+# CHAR_PATH (required โ€” rigged character GLB/FBX matching the NPZ skeleton)
+# ANIM_PATH (required โ€” NPZ whose skeleton matches the rig)
+# OUTPUT_DIR (default: outputs/animated)
+# CHAR_ANIM_TYPE glb|fbx (default: glb)
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+CHAR_PATH=${CHAR_PATH:?Set CHAR_PATH to a rigged character GLB/FBX}
+ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion NPZ matching the rig}
+OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}
+CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-glb}
+
+blender -b -P data_process/mesh_animation/animate_npz.py -- \
+ --char_path="$CHAR_PATH" \
+ --anim_path="$ANIM_PATH" \
+ --output_dir="$OUTPUT_DIR" \
+ --char_anim_type="$CHAR_ANIM_TYPE" "$@"
diff --git a/data_process/scripts/run_caption_category.sh b/data_process/scripts/run_caption_category.sh
new file mode 100755
index 0000000..1a8269b
--- /dev/null
+++ b/data_process/scripts/run_caption_category.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Stage 2 โ€” classify assets into body-plan categories with a local Qwen3-VL model.
+# Single-GPU only โ€” one classification per object, no sharding.
+#
+# Usage:
+# bash data_process/scripts/run_caption_category.sh objaverse
+# MODEL=Qwen/Qwen3-VL-2B-Instruct bash data_process/scripts/run_caption_category.sh truebones
+#
+# Env overrides:
+# MODEL, RENDER_ROOT (default: dataset/render/<dataset>_tpose),
+# CATEGORY_GROUPS_JSON (default: dataset/export/<dataset>/category_groups.json),
+# MAX_TOKENS
+# Extra arguments are passed through to classify_category.py.
+# On offline compute nodes with a populated HuggingFace cache, export
+# HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+DATASET=${1:?Usage: run_caption_category.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+RENDER_ROOT=${RENDER_ROOT:-$(tpose_dir "$DATASET")}
+CATEGORY_GROUPS_JSON=${CATEGORY_GROUPS_JSON:-$(export_dir "$DATASET")/category_groups.json}
+if [[ ! -d "$RENDER_ROOT" ]]; then
+ echo "RENDER_ROOT does not exist: $RENDER_ROOT" >&2
+ exit 2
+fi
+mkdir -p "$(dirname "$CATEGORY_GROUPS_JSON")"
+
+# Local Qwen3-VL only. Sizes: Qwen/Qwen3-VL-{2B,8B,32B,72B}-Instruct.
+MODEL=${MODEL:-Qwen/Qwen3-VL-8B-Instruct}
+MAX_TOKENS=${MAX_TOKENS:-10}
+
+echo "Dataset: $DATASET | Model: $MODEL"
+echo "Render root: $RENDER_ROOT"
+echo "Output JSON: $CATEGORY_GROUPS_JSON"
+
+python -m data_process.vlm_caption.classify_category \
+ --render_root="$RENDER_ROOT" \
+ --model="$MODEL" \
+ --max_tokens="$MAX_TOKENS" \
+ --category_groups_json="$CATEGORY_GROUPS_JSON" "$@"
diff --git a/data_process/scripts/run_caption_motion.sh b/data_process/scripts/run_caption_motion.sh
new file mode 100755
index 0000000..f2dca46
--- /dev/null
+++ b/data_process/scripts/run_caption_motion.sh
@@ -0,0 +1,156 @@
+#!/bin/bash
+# Stage 2 โ€” caption multi-view motion renders with a VLM.
+# The backend is picked from MODEL, with the same rules as detect_backend() in
+# vlm_caption/backends.py:
+# *qwen* (default, any case) -> local Qwen3-VL (GPU-bound; supports --multi-gpu)
+# gemini* -> Gemini API (needs GOOGLE_API_KEY)
+# anything else -> OpenAI-compatible API (needs OPENAI_API_KEY)
+#
+# Usage:
+# bash data_process/scripts/run_caption_motion.sh objaverse # local Qwen3-VL
+# bash data_process/scripts/run_caption_motion.sh truebones --multi-gpu # all visible GPUs (local backend only)
+# MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse
+# MODEL=gpt-5-mini NUM_WORKERS=16 bash data_process/scripts/run_caption_motion.sh mixamo
+#
+# Env overrides:
+# MODEL, RENDER_ROOT (default: dataset/render/<dataset>),
+# OUTPUT_JSON (default: dataset/export/<dataset>/motion_captions.json),
+# DOWNSAMPLE_RATE, MAX_TOKENS, NUM_WORKERS, MAX_RETRIES, BASE_URL (OpenAI-compatible proxy),
+# THINKING_LEVEL / MEDIA_RESOLUTION (gemini-3), THINKING_BUDGET (gemini-2.5)
+# Unrecognized arguments are passed through to caption_motion.py.
+# On offline compute nodes with a populated HuggingFace cache, export
+# HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+# โ”€โ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+handle_help "$@"
+DATASET=${1:?Usage: run_caption_motion.sh <truebones|mixamo|objaverse> [--multi-gpu] [extra args...]}
+shift
+require_dataset "$DATASET"
+
+MODE=single
+PASSTHRU=()
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --multi-gpu) MODE=multi-gpu; shift ;;
+ -h|--help) print_usage; exit 0 ;;
+ # Everything else (e.g. --views v000 v002, --downsample_rate 4) goes to
+ # the Python entry point, which owns the full flag set.
+ *) PASSTHRU+=("$1"); shift ;;
+ esac
+done
+
+# โ”€โ”€ Backend from MODEL (mirrors vlm_caption/backends.py detect_backend) โ”€
+MODEL=${MODEL:-Qwen/Qwen3-VL-8B-Instruct}
+if [[ "${MODEL,,}" == *qwen* ]]; then
+ backend=local # backends.py: BACKEND_QWEN
+elif [[ "$MODEL" == gemini* ]]; then
+ backend=gemini
+else
+ backend=openai
+fi
+
+if [[ "$MODE" == "multi-gpu" && "$backend" != "local" ]]; then
+ echo "ERROR: --multi-gpu only applies to the local Qwen backend "\
+"(API backends scale via NUM_WORKERS instead)." >&2
+ exit 2
+fi
+[[ "$backend" == "gemini" ]] && require_env GOOGLE_API_KEY
+[[ "$backend" == "openai" ]] && require_env OPENAI_API_KEY
+
+# โ”€โ”€ Paths โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+RENDER_ROOT=${RENDER_ROOT:-$(render_dir "$DATASET")}
+OUTPUT_JSON=${OUTPUT_JSON:-$(export_dir "$DATASET")/motion_captions.json}
+if [[ ! -d "$RENDER_ROOT" ]]; then
+ echo "RENDER_ROOT does not exist: $RENDER_ROOT" >&2
+ exit 2
+fi
+mkdir -p "$(dirname "$OUTPUT_JSON")"
+
+# โ”€โ”€ Backend-specific settings โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Full 30 fps for every dataset: temporal continuity is what lets the model
+# track heading turns and posture on appearance-poor subjects (measured on
+# mixamo: 180-turn and sit/stand errors gone at 1x vs 2x downsampling).
+DOWNSAMPLE_RATE=${DOWNSAMPLE_RATE:-1}
+MAX_RETRIES=${MAX_RETRIES:-4}
+EXTRA=()
+case "$backend" in
+ local)
+ MAX_TOKENS=${MAX_TOKENS:-800} ;;
+ gemini)
+ # Reasoning/answer tokens share one budget -> generous default.
+ # Default 2 workers โ€” token/minute quotas are easy to exhaust with
+ # many concurrent image-heavy requests.
+ MAX_TOKENS=${MAX_TOKENS:-1024}
+ NUM_WORKERS=${NUM_WORKERS:-2}
+ EXTRA+=(--thinking_level="${THINKING_LEVEL:-low}"
+ --media_resolution="${MEDIA_RESOLUTION:-medium}"
+ --thinking_budget="${THINKING_BUDGET:-0}"
+ --max_retries="$MAX_RETRIES"
+ --num_workers="$NUM_WORKERS") ;;
+ openai)
+ # GPT-5 reasoning tokens count against max_completion_tokens.
+ MAX_TOKENS=${MAX_TOKENS:-1024}
+ NUM_WORKERS=${NUM_WORKERS:-4}
+ EXTRA+=(--max_retries="$MAX_RETRIES" --num_workers="$NUM_WORKERS")
+ [[ -n "${BASE_URL:-}" ]] && EXTRA+=(--base_url="$BASE_URL") ;;
+esac
+
+# Mixamo ships official catalogue labels and truebones the T2M4LVO
+# annotations; pass them as verify-against-frames hints
+# (HINTS_JSON overrides, HINTS_JSON="" disables).
+case "$DATASET" in
+ mixamo) HINTS_JSON=${HINTS_JSON-dataset/raw/mixamo/animation_motion_prompts.json} ;;
+ truebones) HINTS_JSON=${HINTS_JSON-dataset/raw/truebones/animation_prompts.json} ;;
+ *) HINTS_JSON=${HINTS_JSON:-} ;;
+esac
+
+COMMON=(--task="$DATASET" --render_root="$RENDER_ROOT" --model="$MODEL"
+ --downsample_rate="$DOWNSAMPLE_RATE" --max_tokens="$MAX_TOKENS"
+ --output_json="$OUTPUT_JSON")
+[[ -n "$HINTS_JSON" ]] && COMMON+=(--hints_json="$HINTS_JSON")
+
+echo "Dataset: $DATASET | Backend: $backend | Model: $MODEL | Mode: $MODE"
+[[ -n "$HINTS_JSON" ]] && echo "Caption hints: $HINTS_JSON"
+echo "Render root: $RENDER_ROOT"
+echo "Output JSON: $OUTPUT_JSON"
+
+# โ”€โ”€ Run โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+if [[ "$MODE" == "multi-gpu" ]]; then
+ num_gpus=$(python -c "import torch; print(torch.cuda.device_count())")
+ if ! [[ "$num_gpus" =~ ^[0-9]+$ ]] || [[ "$num_gpus" -lt 1 ]]; then
+ echo "No visible CUDA devices (num_gpus=$num_gpus). Aborting --multi-gpu." >&2
+ exit 1
+ fi
+ LOG_DIR=$(worker_log_dir "caption-motion-$DATASET")
+ echo "Launching $num_gpus GPU processes (per-GPU logs: $LOG_DIR)..."
+
+ pids=()
+ for gpu_id in $(seq 0 $((num_gpus - 1))); do
+ python -m data_process.vlm_caption.caption_motion "${COMMON[@]}" \
+ --num_gpus="$num_gpus" --gpu_id="$gpu_id" \
+ "${PASSTHRU[@]}" > >(tee "$LOG_DIR/gpu${gpu_id}.log") 2>&1 &
+ pids+=($!)
+ echo " Started GPU $gpu_id (PID ${pids[-1]})"
+ done
+
+ failed=0
+ for gpu_id in "${!pids[@]}"; do
+ wait "${pids[gpu_id]}" \
+ || { echo "GPU $gpu_id failed (log: $LOG_DIR/gpu${gpu_id}.log)" >&2; failed=1; }
+ done
+ if [[ $failed -ne 0 ]]; then
+ echo "Some processes failed. Shard files preserved for inspection." >&2
+ exit 1
+ fi
+
+ echo "Merging shards..."
+ python -m data_process.vlm_caption.caption_motion \
+ --task="$DATASET" --merge_shards --output_json="$OUTPUT_JSON" --num_gpus="$num_gpus"
+ echo "Done! Output: $OUTPUT_JSON"
+else
+ python -m data_process.vlm_caption.caption_motion \
+ "${COMMON[@]}" "${EXTRA[@]}" "${PASSTHRU[@]}"
+fi
diff --git a/data_process/scripts/run_download.sh b/data_process/scripts/run_download.sh
new file mode 100755
index 0000000..ccc9aa2
--- /dev/null
+++ b/data_process/scripts/run_download.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+# Stage 0 โ€” download raw source assets from the Hugging Face Hub into
+# dataset/raw/<dataset>/, the layout every downstream wrapper expects.
+#
+# Usage:
+# bash data_process/scripts/run_download.sh mixamo # animations + rigged characters
+# bash data_process/scripts/run_download.sh objaverse # rigged + animated GLBs
+# bash data_process/scripts/run_download.sh objaverse --include 'glb/*' # skip metadata
+#
+# Sources:
+# mixamo https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters
+# objaverse https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated
+# truebones https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations
+#
+# The truebones repo carries annotations only (prompts, per-clip metadata,
+# T-pose renders, build scripts) โ€” the Truebones ZOO animal pack itself is a
+# commercial product that we cannot redistribute. Purchase it from Truebones
+# (https://truebones.com), unpack it to dataset/raw/truebones/Truebone_Z-OO/
+# (one folder per animal), and rebuild the per-clip animation/ layout with the
+# downloaded scripts/pipeline/ (see the repo's README).
+#
+# Needs the `hf` CLI (https://hf.co/cli); gated/private repos also need
+# `hf auth login` or HF_TOKEN.
+#
+# Env overrides: RAW_DIR (target directory), TRUEBONES_REPO
+# Extra arguments are passed through to `hf download`.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+DATASET=${1:-}
+[[ -n "$DATASET" ]] || { print_usage; exit 0; }
+require_dataset "$DATASET"
+shift
+
+if ! command -v hf >/dev/null 2>&1; then
+ echo "ERROR: the 'hf' CLI is not installed. Install it with:" >&2
+ echo " curl -LsSf https://hf.co/cli/install.sh | bash -s" >&2
+ exit 1
+fi
+
+case "$DATASET" in
+ mixamo) REPO=Linzhan/Mixamo-Animations-Characters ;;
+ objaverse) REPO=Linzhan/Objaverse-XL-Rigged-Animated ;;
+ truebones)
+ REPO=${TRUEBONES_REPO:-Linzhan/Truebones-ZOO-Annotations}
+ cat >&2 <<'EOF'
+NOTE: Truebones ZOO is a commercial asset pack and is not redistributed; this
+ downloads annotations only (prompts, metadata, renders, build scripts).
+ To obtain the motion data, purchase it from https://truebones.com, unpack
+ it to dataset/raw/truebones/Truebone_Z-OO/ (one folder per animal), and
+ rebuild animation/ with the downloaded scripts/pipeline/.
+EOF
+ ;;
+esac
+RAW_DIR=${RAW_DIR:-dataset/raw/$DATASET}
+
+echo "Downloading $REPO -> $RAW_DIR"
+if ! hf download "$REPO" --type dataset --local-dir "$RAW_DIR" "$@"; then
+ if [[ "$DATASET" == "truebones" ]]; then
+ echo "Download failed. Note that only annotations are hosted; the motion files" >&2
+ echo "are commercial: purchase the Truebones ZOO pack from https://truebones.com" >&2
+ echo "and unpack it to dataset/raw/truebones/Truebone_Z-OO/<Animal>/..." >&2
+ fi
+ exit 1
+fi
diff --git a/data_process/scripts/run_export.sh b/data_process/scripts/run_export.sh
new file mode 100755
index 0000000..352a167
--- /dev/null
+++ b/data_process/scripts/run_export.sh
@@ -0,0 +1,126 @@
+#!/bin/bash
+# Stage 1 โ€” export raw rigged assets (GLB/GLTF and FBX) to NPZ motion data.
+#
+# Usage:
+# bash data_process/scripts/run_export.sh truebones # dataset preset (flat {Species}-{Action}.fbx clips)
+# bash data_process/scripts/run_export.sh mixamo # dataset preset (animation-only FBX)
+# bash data_process/scripts/run_export.sh objaverse # dataset preset (GLB/GLTF)
+# bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # parallel (mixamo too)
+# bash data_process/scripts/run_export.sh mixamo --multi-worker 8 --no-vis # skip MP4 previews
+# DATA_DIR=my_assets bash data_process/scripts/run_export.sh # auto: .glb/.gltf and .fbx side by side
+#
+# --multi-worker only applies to a *directory* input; truebones is exported by
+# a single-process exporter and ignores it (with a warning).
+#
+# Auto mode runs export_general.py on the directory: GLB/GLTF and FBX side
+# by side, importer picked per file by extension, mesh optional (armature-only
+# FBX works too).
+#
+# Env overrides: DATA_DIR, OUTPUT_DIR
+# Extra arguments are passed through to the exporter.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+EXPORTERS=data_process/motion_export
+
+# โ”€โ”€ Parse mode + flags โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+handle_help "$@"
+
+MODE=auto
+case "${1:-}" in
+ truebones|mixamo|objaverse|auto) MODE="$1"; shift ;;
+esac
+
+NUM_WORKERS=1
+EXTRA_ARGS=()
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;
+ *) EXTRA_ARGS+=("$1"); shift ;;
+ esac
+done
+
+# โ”€โ”€ Per-mode defaults โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+case "$MODE" in
+ truebones)
+ DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation}
+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir truebones)} ;;
+ mixamo)
+ DATA_DIR=${DATA_DIR:-dataset/raw/mixamo/animation_motion}
+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir mixamo)} ;;
+ objaverse)
+ DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb}
+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir objaverse)} ;;
+ auto)
+ DATA_DIR=${DATA_DIR:?auto mode requires DATA_DIR (directory of .glb/.gltf/.fbx assets)}
+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir custom)} ;;
+esac
+
+# โ”€โ”€ Exporter invocations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# launch_workers <script> <input flag>: run one exporter, sharding over
+# NUM_WORKERS parallel Blender processes and merging the per-worker summary
+# JSONs afterwards. Only a directory input is sharded โ€” the exporters write
+# unsuffixed canonical summaries for a single file, which N workers would
+# concurrently overwrite.
+launch_workers() {
+ local script="$1" input_flag="$2"
+ if [[ "$NUM_WORKERS" -gt 1 && -d "$DATA_DIR" ]]; then
+ local log_dir
+ log_dir=$(worker_log_dir "export-$MODE")
+ echo "Launching $NUM_WORKERS export workers (per-worker logs: $log_dir)..."
+ local pids=() worker_id failed=0
+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do
+ blender -b -P "$EXPORTERS/$script" -- \
+ "$input_flag=$DATA_DIR" --output_dir="$OUTPUT_DIR" \
+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \
+ "${EXTRA_ARGS[@]}" > >(tee "$log_dir/worker${worker_id}.log") 2>&1 &
+ pids+=($!)
+ echo " Started worker $worker_id (PID ${pids[-1]})"
+ done
+
+ # A bare `wait` always returns 0 โ€” check every worker individually.
+ # Merging a truncated set of shards would silently shrink the
+ # canonical JSONs and then delete the shards, so never merge on failure.
+ for worker_id in "${!pids[@]}"; do
+ wait "${pids[worker_id]}" \
+ || { echo "Worker $worker_id failed (log: $log_dir/worker${worker_id}.log)" >&2; failed=1; }
+ done
+ if [[ $failed -ne 0 ]]; then
+ echo "Some export workers failed โ€” NOT merging summary JSONs." >&2
+ echo "Per-worker shards are preserved in $OUTPUT_DIR; rerun to finish," >&2
+ echo "then merge with: python -m data_process.tools.merge_summaries --output_dir=$OUTPUT_DIR" >&2
+ exit 1
+ fi
+ echo "All workers finished. Merging summary JSONs..."
+ python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"
+ else
+ if [[ "$NUM_WORKERS" -gt 1 ]]; then
+ echo "WARNING: --multi-worker is ignored for a single-file input" \
+ "($DATA_DIR); running one exporter process." >&2
+ fi
+ blender -b -P "$EXPORTERS/$script" -- \
+ "$input_flag=$DATA_DIR" --output_dir="$OUTPUT_DIR" "${EXTRA_ARGS[@]}"
+ fi
+}
+
+export_glb() { launch_workers export_objaverse.py --data_dir; }
+export_mixamo() { launch_workers export_mixamo.py --anim_dir; }
+
+export_fbx() {
+ if [[ "$NUM_WORKERS" -gt 1 ]]; then
+ echo "WARNING: --multi-worker is ignored for truebones โ€” export_truebones.py" \
+ "is single-process; running one exporter process." >&2
+ fi
+ blender -b -P "$EXPORTERS/export_truebones.py" -- \
+ --data_dir="$DATA_DIR" --output_dir="$OUTPUT_DIR" "$@" "${EXTRA_ARGS[@]}"
+}
+
+# โ”€โ”€ Dispatch โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+echo "Mode: $MODE | Input: $DATA_DIR | Output: $OUTPUT_DIR"
+case "$MODE" in
+ objaverse) export_glb ;;
+ mixamo) export_mixamo ;;
+ truebones) export_fbx ;;
+ auto) launch_workers export_general.py --input ;;
+esac
diff --git a/data_process/scripts/run_export_general.sh b/data_process/scripts/run_export_general.sh
new file mode 100644
index 0000000..0ff2602
--- /dev/null
+++ b/data_process/scripts/run_export_general.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+# Stage 1 โ€” export rigged GLB/GLTF/FBX assets to NPZ motion data.
+# Input is a single file or a directory of assets (mixed formats welcome).
+# Mesh optional (armature-only FBX works); every pose action becomes a clip.
+#
+# Usage:
+# bash data_process/scripts/run_export_general.sh <asset.glb|.fbx|dir> [extra args...]
+# bash data_process/scripts/run_export_general.sh my_assets/ --multi-worker 8 --no-vis
+# bash data_process/scripts/run_export_general.sh model.fbx --no-prune
+#
+# Env overrides: OUTPUT_DIR (default: dataset/export/custom)
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+INPUT="${1:?Usage: run_export_general.sh <asset.glb|.fbx|dir> [extra args...]}"
+shift
+OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir custom)}
+
+NUM_WORKERS=1
+EXTRA_ARGS=()
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;
+ *) EXTRA_ARGS+=("$1"); shift ;;
+ esac
+done
+
+if [[ "$NUM_WORKERS" -gt 1 && -d "$INPUT" ]]; then
+ LOG_DIR=$(worker_log_dir "export-general")
+ echo "Launching $NUM_WORKERS export workers (per-worker logs: $LOG_DIR)..."
+ pids=()
+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do
+ blender -b -P data_process/motion_export/export_general.py -- \
+ --input="$INPUT" --output_dir="$OUTPUT_DIR" \
+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \
+ "${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &
+ pids+=($!)
+ echo " Started worker $worker_id (PID ${pids[-1]})"
+ done
+
+ # A bare `wait` always returns 0 โ€” check every worker individually, and
+ # never merge a truncated set of shards (merging deletes them).
+ failed=0
+ for worker_id in "${!pids[@]}"; do
+ wait "${pids[worker_id]}" \
+ || { echo "Worker $worker_id failed (log: $LOG_DIR/worker${worker_id}.log)" >&2; failed=1; }
+ done
+ if [[ $failed -ne 0 ]]; then
+ echo "Some export workers failed โ€” NOT merging summary JSONs." >&2
+ echo "Per-worker shards are preserved in $OUTPUT_DIR; rerun to finish." >&2
+ exit 1
+ fi
+ echo "All workers finished. Merging summary JSONs..."
+ python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"
+else
+ if [[ "$NUM_WORKERS" -gt 1 ]]; then
+ echo "WARNING: --multi-worker is ignored for a single-file input" \
+ "($INPUT); running one exporter process." >&2
+ fi
+ blender -b -P data_process/motion_export/export_general.py -- \
+ --input="$INPUT" --output_dir="$OUTPUT_DIR" "${EXTRA_ARGS[@]}"
+fi
diff --git a/data_process/scripts/run_extract_features.sh b/data_process/scripts/run_extract_features.sh
new file mode 100755
index 0000000..021d783
--- /dev/null
+++ b/data_process/scripts/run_extract_features.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+# Stage 4 โ€” process exported NPZ motions into canonicalized training clips.
+#
+# Usage:
+# bash data_process/scripts/run_extract_features.sh truebones
+# bash data_process/scripts/run_extract_features.sh objaverse
+# bash data_process/scripts/run_extract_features.sh mixamo
+# APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones
+# NUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse
+#
+# Env overrides:
+# DATA_DIR Stage-1 export directory (default: dataset/export/<dataset>)
+# SAVE_DIR Output directory (default: dataset/features/<dataset>)
+# APPLY_CLIP If 1/true, crop long motions into overlapping fixed-length
+# clips; otherwise (default) keep only the first max_clip_len frames.
+# NUM_WORKERS Parallel worker processes over object types (default: 1)
+# NO_VIS If 1/true, skip the per-clip MP4 previews
+# Extra arguments are passed through to extract_features.py.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_extract_features.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+DATA_DIR=${DATA_DIR:-$(export_dir "$DATASET")}
+SAVE_DIR=${SAVE_DIR:-$(features_dir "$DATASET")}
+
+EXTRA_ARGS=()
+# Per-dataset joint-count range (skeletons outside it are skipped here, not
+# at export โ€” the exports stay complete). Defaults in extract_features.py
+# are 8..150; objaverse keeps its wider historical range.
+case "$DATASET" in
+ objaverse) EXTRA_ARGS+=(--min_joints 4 --max_joints 180) ;;
+esac
+case "${APPLY_CLIP:-0}" in
+ 1|true|TRUE) EXTRA_ARGS+=(--apply_clip) ;;
+esac
+case "${NO_VIS:-0}" in
+ 1|true|TRUE) EXTRA_ARGS+=(--no-vis) ;;
+esac
+[[ "${NUM_WORKERS:-1}" -gt 1 ]] && EXTRA_ARGS+=(--num_workers="$NUM_WORKERS")
+
+python -m data_process.feature_extraction.extract_features \
+ --dataset_type="$DATASET" \
+ --data_dir="$DATA_DIR" \
+ --save_dir="$SAVE_DIR" \
+ "${EXTRA_ARGS[@]}" "$@"
diff --git a/data_process/scripts/run_joints_face_correct_llm.sh b/data_process/scripts/run_joints_face_correct_llm.sh
new file mode 100755
index 0000000..7cca9b0
--- /dev/null
+++ b/data_process/scripts/run_joints_face_correct_llm.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+# Stage 3 โ€” re-run LLM face-joint selection on rigs whose source == "empty".
+#
+# For each empty entry in face_joint_names.json the LLM is asked again to
+# pick a bilateral pair (or body-axis endpoints). Successful picks overwrite
+# the entry in place (a *.bak copy is made on first run); rigs that stay
+# empty are listed in still_empty_face_joints.txt.
+#
+# Usage:
+# bash data_process/scripts/run_joints_face_correct_llm.sh objaverse
+# MODEL=gpt-5 bash data_process/scripts/run_joints_face_correct_llm.sh objaverse
+#
+# Env overrides: RAW, CLEANED, FACE, FAILED_LIST (optional id subset), MODEL,
+# MAX_TOKENS, RIG_TIMEOUT. Extra args are passed through.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_joints_face_correct_llm.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+RAW=${RAW:-$(export_dir "$DATASET")/joint_names.json}
+CLEANED=${CLEANED:-$(export_dir "$DATASET")/clean_joint_names.json}
+FACE=${FACE:-$(export_dir "$DATASET")/face_joint_names.json}
+MODEL=${MODEL:-deepseek-v4-flash}
+MAX_TOKENS=${MAX_TOKENS:-512}
+# Thinking backends need a much larger per-rig budget (see names_correct wrapper).
+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi
+
+for f in "$RAW" "$CLEANED" "$FACE"; do
+ [[ -f "$f" ]] || { echo "MISSING FILE: $f" >&2; exit 1; }
+done
+
+EXTRA_ARGS=()
+[[ -n "${FAILED_LIST:-}" ]] && EXTRA_ARGS+=(--failed_list="$FAILED_LIST")
+
+python -m data_process.joint_annotation.face_correct_llm \
+ --raw="$RAW" --cleaned="$CLEANED" --face="$FACE" \
+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \
+ "${EXTRA_ARGS[@]}" "$@"
diff --git a/data_process/scripts/run_joints_face_select_llm.sh b/data_process/scripts/run_joints_face_select_llm.sh
new file mode 100755
index 0000000..63950dc
--- /dev/null
+++ b/data_process/scripts/run_joints_face_select_llm.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+# Stage 3 โ€” LLM facing-direction joint pair per rig: -> face_joint_names.json.
+#
+# Usage:
+# bash data_process/scripts/run_joints_face_select_llm.sh objaverse # DeepSeek v4-flash (default)
+# MODEL=Qwen/Qwen3-8B bash data_process/scripts/run_joints_face_select_llm.sh objaverse # local HF model
+#
+# Env overrides: INPUT_DIR, OUTPUT_NAME, MODEL, MAX_TOKENS. Extra args are passed
+# through (e.g. --no_fallback). API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_joints_face_select_llm.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+INPUT_DIR=${INPUT_DIR:-$(export_dir "$DATASET")}
+OUTPUT_NAME=${OUTPUT_NAME:-face_joint_names.json}
+MODEL=${MODEL:-deepseek-v4-flash}
+# Thinking backends hold a call for tens of seconds (DeepSeek low-effort
+# ~19s, plus server keep-alive under load), so the per-rig SIGALRM budget
+# scales with the backend; override with RIG_TIMEOUT.
+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi
+MAX_TOKENS=${MAX_TOKENS:-512}
+
+python -m data_process.joint_annotation.face_select_llm \
+ --input_dir="$INPUT_DIR" --output_name="$OUTPUT_NAME" \
+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \
+"$@"
diff --git a/data_process/scripts/run_joints_face_select_rule.sh b/data_process/scripts/run_joints_face_select_rule.sh
new file mode 100755
index 0000000..0056a82
--- /dev/null
+++ b/data_process/scripts/run_joints_face_select_rule.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+# Stage 3 โ€” rule-based facing-direction joint pair: -> face_joint_names.json.
+# Reads joint_names.json + clean_joint_names.json from the export directory.
+#
+# Usage:
+# bash data_process/scripts/run_joints_face_select_rule.sh <truebones|mixamo|objaverse>
+#
+# Env overrides: INPUT_DIR, OUTPUT
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_joints_face_select_rule.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+INPUT_DIR=${INPUT_DIR:-$(export_dir "$DATASET")}
+OUTPUT=${OUTPUT:-$INPUT_DIR/face_joint_names.json}
+
+python -m data_process.joint_annotation.face_select_rule \
+ --input_dir "$INPUT_DIR" --output "$OUTPUT" "$@"
diff --git a/data_process/scripts/run_joints_names_clean_llm.sh b/data_process/scripts/run_joints_names_clean_llm.sh
new file mode 100755
index 0000000..53b688d
--- /dev/null
+++ b/data_process/scripts/run_joints_names_clean_llm.sh
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Stage 3 โ€” LLM joint-name cleaning: joint_names.json -> clean_joint_names.json.
+#
+# Usage:
+# bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # DeepSeek v4-flash (default)
+# MODEL=deepseek-v4-flash bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # DeepSeek API
+# MODEL=Qwen/Qwen3-8B bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # local HF model
+#
+# Env overrides: INPUT, OUTPUT, MODEL, MAX_TOKENS. Extra args are passed through
+# (e.g. --no_fallback). API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_joints_names_clean_llm.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+INPUT=${INPUT:-$(export_dir "$DATASET")/joint_names.json}
+OUTPUT=${OUTPUT:-$(export_dir "$DATASET")/clean_joint_names.json}
+MODEL=${MODEL:-deepseek-v4-flash}
+MAX_TOKENS=${MAX_TOKENS:-2048}
+
+python -m data_process.joint_annotation.names_clean_llm \
+ --input="$INPUT" --output="$OUTPUT" \
+ --model="$MODEL" --max_tokens="$MAX_TOKENS" \
+ --report "$@"
diff --git a/data_process/scripts/run_joints_names_clean_rule.sh b/data_process/scripts/run_joints_names_clean_rule.sh
new file mode 100755
index 0000000..07d9966
--- /dev/null
+++ b/data_process/scripts/run_joints_names_clean_rule.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+# Stage 3 โ€” rule-based joint-name cleaning: joint_names.json -> clean_joint_names.json.
+#
+# Usage:
+# bash data_process/scripts/run_joints_names_clean_rule.sh <truebones|mixamo|objaverse> [--report]
+#
+# Env overrides: INPUT, OUTPUT (default: inside dataset/export/<dataset>)
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_joints_names_clean_rule.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+INPUT=${INPUT:-$(export_dir "$DATASET")/joint_names.json}
+OUTPUT=${OUTPUT:-$(export_dir "$DATASET")/clean_joint_names.json}
+
+python -m data_process.joint_annotation.names_clean_rule \
+ --input="$INPUT" --output="$OUTPUT" "$@"
diff --git a/data_process/scripts/run_joints_names_correct_llm.sh b/data_process/scripts/run_joints_names_correct_llm.sh
new file mode 100755
index 0000000..4472d62
--- /dev/null
+++ b/data_process/scripts/run_joints_names_correct_llm.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+# Stage 3 โ€” LLM check-and-correct sweep over clean_joint_names.json (in place).
+#
+# Every rig in joint_names.json is sent to the LLM together with its current
+# cleaned label; the LLM keeps or corrects each label. The cleaned file is
+# updated IN PLACE (a *.bak copy is made on first run). Set FAILED_LIST to
+# restrict the sweep to the rig ids listed in a text file.
+#
+# Usage:
+# bash data_process/scripts/run_joints_names_correct_llm.sh objaverse
+# FAILED_LIST=.../failed_clean_names.txt bash data_process/scripts/run_joints_names_correct_llm.sh objaverse
+#
+# Env overrides: RAW, CLEANED, FAILED_LIST, MODEL, MAX_TOKENS. Extra args are
+# passed through. API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+DATASET=${1:?Usage: run_joints_names_correct_llm.sh <truebones|mixamo|objaverse> [extra args...]}
+shift
+require_dataset "$DATASET"
+
+RAW=${RAW:-$(export_dir "$DATASET")/joint_names.json}
+CLEANED=${CLEANED:-$(export_dir "$DATASET")/clean_joint_names.json}
+MODEL=${MODEL:-deepseek-v4-flash}
+# Thinking backends hold a call for tens of seconds (DeepSeek low-effort
+# ~19s, plus server keep-alive under load), so the per-rig SIGALRM budget
+# scales with the backend; override with RIG_TIMEOUT.
+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi
+MAX_TOKENS=${MAX_TOKENS:-4096}
+
+EXTRA_ARGS=()
+[[ -n "${FAILED_LIST:-}" ]] && EXTRA_ARGS+=(--failed_list="$FAILED_LIST")
+
+python -m data_process.joint_annotation.names_correct_llm \
+ --raw="$RAW" --cleaned="$CLEANED" \
+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \
+ "${EXTRA_ARGS[@]}" "$@"
diff --git a/data_process/scripts/run_joints_vis_tpose.sh b/data_process/scripts/run_joints_vis_tpose.sh
new file mode 100755
index 0000000..e27fe1c
--- /dev/null
+++ b/data_process/scripts/run_joints_vis_tpose.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+# Stage 3 โ€” render annotated T-pose PNGs from exported motion NPZs.
+# Use the PNGs to identify left/right symmetric face joints by eye.
+#
+# Usage:
+# # Batch mode โ€” one PNG per unique skeleton in the export directory:
+# bash data_process/scripts/run_joints_vis_tpose.sh truebones
+# LIMIT=50 bash data_process/scripts/run_joints_vis_tpose.sh truebones
+#
+# # Single mode โ€” one NPZ:
+# NPZ_PATH=/path/to/motion.npz bash data_process/scripts/run_joints_vis_tpose.sh
+# NPZ_PATH=/path/to/motion.npz OUTPUT_PATH=out.png bash data_process/scripts/run_joints_vis_tpose.sh
+#
+# Env overrides:
+# NPZ_PATH single mode: path to one motion NPZ (takes precedence over batch mode)
+# OUTPUT_PATH [single] output PNG path
+# MOTIONS_DIR [batch] motion NPZ directory (default: <export_dir>/motions)
+# OUTPUT_DIR [batch] PNG output directory (default: <export_dir>/face_joints_vis)
+# LIMIT [batch] stop after N skeletons
+# FONT_SIZE label font size (default: 7)
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+
+cmd=(python -m data_process.tools.vis_tpose)
+
+if [[ -n "${NPZ_PATH:-}" ]]; then
+ [[ -f "$NPZ_PATH" ]] || { echo "NPZ_PATH not found: $NPZ_PATH" >&2; exit 1; }
+ cmd+=(--npz_path "$NPZ_PATH")
+ [[ -n "${OUTPUT_PATH:-}" ]] && cmd+=(--output "$OUTPUT_PATH")
+else
+ DATASET=${1:?Usage: run_joints_vis_tpose.sh <truebones|mixamo|objaverse> (or set NPZ_PATH)}
+ require_dataset "$DATASET"
+ MOTIONS_DIR=${MOTIONS_DIR:-$(export_dir "$DATASET")/motions}
+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir "$DATASET")/face_joints_vis}
+ [[ -d "$MOTIONS_DIR" ]] || { echo "MOTIONS_DIR not found: $MOTIONS_DIR" >&2; exit 1; }
+ cmd+=(--motions_dir "$MOTIONS_DIR" --output_dir "$OUTPUT_DIR")
+ [[ -n "${LIMIT:-}" ]] && cmd+=(--limit "$LIMIT")
+fi
+[[ -n "${FONT_SIZE:-}" ]] && cmd+=(--font-size "$FONT_SIZE")
+
+"${cmd[@]}"
diff --git a/data_process/scripts/run_render_motion.sh b/data_process/scripts/run_render_motion.sh
new file mode 100755
index 0000000..3056677
--- /dev/null
+++ b/data_process/scripts/run_render_motion.sh
@@ -0,0 +1,98 @@
+#!/bin/bash
+# Stage 2a โ€” render multi-view frames of every motion (EEVEE), the input of
+# VLM motion captioning (run_caption_motion.sh).
+#
+# Usage:
+# bash data_process/scripts/run_render_motion.sh truebones
+# bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8
+# bash data_process/scripts/run_render_motion.sh objaverse --missing-only
+# DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_motion.sh mixamo
+#
+# Inputs per dataset: truebones/objaverse render the raw FBX/GLB assets;
+# mixamo renders the animated character files produced by
+# run_animate_mixamo.sh (raw Mixamo FBXs carry no mesh).
+#
+# Runs with plain python + the pip `bpy` module: EEVEE needs the module's
+# GPU context (`blender -b` has no display surface). --multi-worker N
+# shards the assets over N processes; with NUM_GPUS>1 the workers are
+# spread round-robin over the visible GPUs (worker i -> GPU i % NUM_GPUS),
+# so N is the *total* worker count, not the per-GPU count.
+#
+# KNOWN LIMITATION: the CUDA_VISIBLE_DEVICES pinning does NOT bind EEVEE's
+# EGL context โ€” on a multi-GPU machine every worker still renders on the
+# first GPU. Give each render process (or Slurm job) a machine/cgroup that
+# exposes exactly one GPU instead of relying on NUM_GPUS>1.
+#
+# Env overrides: DATA_DIR, OUTPUT_DIR, RESOLUTION, SAMPLES, CAMERA_DIST,
+# MAX_RENDER_FRAMES (per-clip frame cap, default 200), MIN_ACTION_FRAMES,
+# NUM_GPUS (default 1). Extra arguments are passed through to the renderer.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+DATASET=${1:-}
+[[ -n "$DATASET" ]] || { print_usage; exit 0; }
+require_dataset "$DATASET"
+shift
+
+NUM_WORKERS=1
+NUM_GPUS=${NUM_GPUS:-1}
+EXTRA_ARGS=()
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;
+ *) EXTRA_ARGS+=("$1"); shift ;;
+ esac
+done
+
+case "$DATASET" in
+ truebones) DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation} ;;
+ mixamo) DATA_DIR=${DATA_DIR:-outputs/mixamo_characters} ;;
+ objaverse) DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb} ;;
+esac
+OUTPUT_DIR=${OUTPUT_DIR:-$(render_dir "$DATASET")}
+
+RENDERER="data_process.motion_rendering.render_${DATASET}"
+COMMON_ARGS=(
+ --data_dir="$DATA_DIR"
+ --output_dir="$OUTPUT_DIR"
+ --resolution="${RESOLUTION:-512}"
+ --samples="${SAMPLES:-64}"
+ --camera_dist="${CAMERA_DIST:-1.5}"
+)
+[[ -n "${MAX_RENDER_FRAMES:-}" ]] && COMMON_ARGS+=(--max_render_frames="$MAX_RENDER_FRAMES")
+[[ -n "${MIN_ACTION_FRAMES:-}" ]] && COMMON_ARGS+=(--min_action_frames="$MIN_ACTION_FRAMES")
+
+echo "Dataset: $DATASET | Input: $DATA_DIR | Output: $OUTPUT_DIR"
+if [[ "$NUM_WORKERS" -gt 1 ]]; then
+ LOG_DIR=$(worker_log_dir "render-motion-$DATASET")
+ echo "Launching $NUM_WORKERS render workers (per-worker logs: $LOG_DIR)..."
+ pids=()
+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do
+ # Pin each worker to one GPU before the process starts: EEVEE picks
+ # its device when the bpy module builds its GL context, so the
+ # restriction has to be in the environment, not set from Python.
+ gpu=$((worker_id % NUM_GPUS))
+ CUDA_VISIBLE_DEVICES="$gpu" \
+ python -m "$RENDERER" "${COMMON_ARGS[@]}" \
+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \
+ "${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &
+ pids+=($!)
+ echo " Started worker $worker_id on GPU $gpu (PID ${pids[-1]})"
+ done
+
+ # A bare `wait` always returns 0 โ€” check every worker individually.
+ failed=0
+ for worker_id in "${!pids[@]}"; do
+ wait "${pids[worker_id]}" \
+ || { echo "Worker $worker_id failed (log: $LOG_DIR/worker${worker_id}.log)" >&2; failed=1; }
+ done
+ if [[ $failed -ne 0 ]]; then
+ echo "Some render workers failed. Rerun to retry the missing frames." >&2
+ exit 1
+ fi
+ echo "All workers finished."
+else
+ python -m "$RENDERER" "${COMMON_ARGS[@]}" "${EXTRA_ARGS[@]}"
+fi
diff --git a/data_process/scripts/run_render_tpose.sh b/data_process/scripts/run_render_tpose.sh
new file mode 100755
index 0000000..bd16345
--- /dev/null
+++ b/data_process/scripts/run_render_tpose.sh
@@ -0,0 +1,77 @@
+#!/bin/bash
+# Stage 2a โ€” render a T-pose 2x2 grid per asset (EEVEE), the input of
+# body-plan category classification (run_caption_category.sh).
+#
+# Usage:
+# bash data_process/scripts/run_render_tpose.sh truebones
+# bash data_process/scripts/run_render_tpose.sh objaverse --multi-worker 8
+# DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_tpose.sh mixamo
+#
+# Grids are written flat as <tpose_dir>/<name>.png (one per GLB stem, or
+# per Truebones object type). Runs with plain python + the pip `bpy`
+# module: EEVEE needs the module's GPU context.
+#
+# Env overrides: DATA_DIR, OUTPUT_DIR, RESOLUTION, SAMPLES, CAMERA_DIST
+# Extra arguments are passed through to the renderer.
+
+set -euo pipefail
+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
+
+handle_help "$@"
+DATASET=${1:-}
+[[ -n "$DATASET" ]] || { print_usage; exit 0; }
+require_dataset "$DATASET"
+shift
+
+NUM_WORKERS=1
+EXTRA_ARGS=()
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;
+ *) EXTRA_ARGS+=("$1"); shift ;;
+ esac
+done
+
+case "$DATASET" in
+ truebones) DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation}
+ EXTRA_ARGS+=(--species_grids) ;; # one grid per species
+ mixamo) DATA_DIR=${DATA_DIR:-dataset/raw/mixamo/character_refined} ;;
+ objaverse) DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb} ;;
+esac
+OUTPUT_DIR=${OUTPUT_DIR:-$(tpose_dir "$DATASET")}
+
+COMMON_ARGS=(
+ --data_dir="$DATA_DIR"
+ --output_dir="$OUTPUT_DIR"
+ --resolution="${RESOLUTION:-512}"
+ --samples="${SAMPLES:-64}"
+ --camera_dist="${CAMERA_DIST:-1.5}"
+)
+
+echo "Dataset: $DATASET | Input: $DATA_DIR | Output: $OUTPUT_DIR"
+if [[ "$NUM_WORKERS" -gt 1 ]]; then
+ LOG_DIR=$(worker_log_dir "render-tpose-$DATASET")
+ echo "Launching $NUM_WORKERS render workers (per-worker logs: $LOG_DIR)..."
+ pids=()
+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do
+ python -m data_process.motion_rendering.render_tpose "${COMMON_ARGS[@]}" \
+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \
+ "${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &
+ pids+=($!)
+ echo " Started worker $worker_id (PID ${pids[-1]})"
+ done
+
+ # A bare `wait` always returns 0 โ€” check every worker individually.
+ failed=0
+ for worker_id in "${!pids[@]}"; do
+ wait "${pids[worker_id]}" \
+ || { echo "Worker $worker_id failed (log: $LOG_DIR/worker${worker_id}.log)" >&2; failed=1; }
+ done
+ if [[ $failed -ne 0 ]]; then
+ echo "Some render workers failed. Rerun to retry the missing grids." >&2
+ exit 1
+ fi
+ echo "All workers finished."
+else
+ python -m data_process.motion_rendering.render_tpose "${COMMON_ARGS[@]}" "${EXTRA_ARGS[@]}"
+fi
diff --git a/data_process/tools/__init__.py b/data_process/tools/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/tools/character_fbx2glb.py b/data_process/tools/character_fbx2glb.py
new file mode 100644
index 0000000..e402d07
--- /dev/null
+++ b/data_process/tools/character_fbx2glb.py
@@ -0,0 +1,110 @@
+"""Convert rigged T-pose character FBX files to GLB, one to one.
+
+Consumes a flat directory of character FBXs (mesh + armature, no animation
+required โ€” e.g. ``dataset/raw/mixamo/character_refined``) and writes
+``{name}.glb`` per file. Any actions the FBX importer binds (static
+object-transform takes) are stripped so the GLB is a clean T-pose rig.
+Resumable: existing GLBs are skipped.
+
+Usage (Blender headless):
+ blender -b -P data_process/tools/character_fbx2glb.py -- \
+ --data_dir dataset/raw/mixamo/character_refined \
+ --output_dir dataset/raw/mixamo/character_refined_glb
+"""
+
+import argparse
+import os
+import sys
+
+import bpy
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import import_fbx, reset_scene
+
+
+def convert_character_to_glb(fbx_path, glb_path):
+ """Import one character FBX and export it as a T-pose GLB (no animation)."""
+ reset_scene()
+ import_fbx(fbx_path)
+
+ armatures = [o for o in bpy.data.objects if o.type == 'ARMATURE']
+ assert armatures, f"No armature found in {fbx_path}"
+ meshes = [o for o in bpy.data.objects if o.type == 'MESH']
+ assert meshes, f"No mesh found in {fbx_path}"
+
+ # Strip every action/binding so the export carries only the rig in its
+ # rest pose โ€” character files should have no animation, but the FBX
+ # importer still binds static object-transform takes.
+ for obj in bpy.data.objects:
+ if obj.animation_data is not None:
+ obj.animation_data_clear()
+ for action in list(bpy.data.actions):
+ bpy.data.actions.remove(action)
+
+ for obj in bpy.data.objects:
+ obj.select_set(obj.type in ('MESH', 'ARMATURE'))
+
+ kwargs = dict(
+ filepath=glb_path,
+ check_existing=False,
+ use_selection=True,
+ export_format='GLB',
+ export_animations=False,
+ )
+ bpy.ops.export_scene.gltf(**kwargs)
+ logger.info(f"Saved T-pose GLB: {glb_path}")
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Convert rigged T-pose character FBX files to GLB, one to one.")
+ parser.add_argument('--data_dir', type=str, required=True,
+ help='Flat directory of character FBX files.')
+ parser.add_argument('--output_dir', type=str, required=True,
+ help='Destination directory for the per-character GLB files.')
+ parser.add_argument('--worker_id', type=int, default=0,
+ help='Worker index for sharding across workers.')
+ parser.add_argument('--num_workers', type=int, default=1,
+ help='Total number of workers.')
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ args = parse_args()
+
+ fbx_files = sorted(f for f in os.listdir(args.data_dir)
+ if f.lower().endswith('.fbx'))
+ logger.info(f"Found {len(fbx_files)} character FBXs in {args.data_dir}")
+
+ if args.num_workers > 1:
+ fbx_files = fbx_files[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(fbx_files)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'fbx2glb_errors_worker{args.worker_id}.log')
+
+ n_ok = n_skip = n_fail = 0
+ for fbx_file in fbx_files:
+ glb_path = os.path.join(args.output_dir,
+ os.path.splitext(fbx_file)[0] + '.glb')
+ if os.path.exists(glb_path):
+ n_skip += 1
+ continue
+ try:
+ convert_character_to_glb(os.path.join(args.data_dir, fbx_file), glb_path)
+ n_ok += 1
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ logger.error(f"Failed to convert {fbx_file}: {e}")
+ with open(error_log, 'a') as f:
+ f.write(f"{fbx_file}\t{e}\n")
+ n_fail += 1
+
+ logger.info(f"Done: {n_ok} converted, {n_skip} skipped, {n_fail} failed "
+ f"-> {args.output_dir}")
+
+ if n_fail:
+ sys.exit(1)
diff --git a/data_process/tools/merge_summaries.py b/data_process/tools/merge_summaries.py
new file mode 100644
index 0000000..2223f24
--- /dev/null
+++ b/data_process/tools/merge_summaries.py
@@ -0,0 +1,137 @@
+"""Merge per-worker export summary JSONs into the canonical files.
+
+Multi-worker export runs write per-worker summary shards
+(``joint_names_worker{i}.json``, ``joint_count_worker{i}.json``,
+``clip_frames_worker{i}.json``) so parallel Blender processes never race
+on one file. This script merges them โ€” together with any existing canonical
+files โ€” into ``joint_names.json`` / ``joint_count.json`` /
+``clip_frames.json`` (plus ``summary.json`` with the dataset-level
+summary) and removes the shards. Idempotent: rerunning with no shards present
+just rewrites the canonical files from themselves.
+
+Plain Python (no bpy), so it runs in the conda env directly:
+ python -m data_process.tools.merge_summaries --output_dir <export_dir>
+"""
+
+import argparse
+import glob
+import json
+import os
+import sys
+
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.export_markers import collect_joint_names_from_markers # noqa: E402
+
+
+def _load(path):
+ with open(path) as f:
+ return json.load(f)
+
+
+def _save(path, data):
+ tmp = f"{path}.tmp.{os.getpid()}"
+ with open(tmp, "w") as f:
+ json.dump(data, f, indent=2)
+ os.replace(tmp, path)
+
+
+def _scan_missing_frame_counts(motion_dir, known):
+ """Frame counts for clips in *motion_dir* that *known* doesn't cover."""
+ if not os.path.isdir(motion_dir):
+ return {}
+ import numpy as np # local: keeps the module importable without numpy
+
+ found = {}
+ for npz_file in sorted(os.listdir(motion_dir)):
+ if not npz_file.endswith(".npz"):
+ continue
+ clip_name = os.path.splitext(npz_file)[0]
+ if clip_name in known:
+ continue
+ try:
+ data = np.load(os.path.join(motion_dir, npz_file), allow_pickle=True)
+ found[clip_name] = int(data["anim_local_rot"].shape[0])
+ except Exception as exc: # noqa: BLE001 โ€” a bad clip shouldn't stop the merge
+ logger.warning(f"Could not read frame count from {npz_file}: {exc}")
+ return found
+
+
+def merge_summaries(output_dir, fps=30, keep_shards=False):
+ jn_path = os.path.join(output_dir, "joint_names.json")
+ joint_names = _load(jn_path) if os.path.isfile(jn_path) else {}
+
+ # Per-asset completion markers are written as each asset finishes, so they
+ # survive a killed worker whose in-memory shard never reached disk. Folding
+ # them in here is what makes a truncated joint_names.json self-healing.
+ from_markers = collect_joint_names_from_markers(output_dir)
+ recovered = len(set(from_markers) - set(joint_names))
+ joint_names.update(from_markers)
+ if recovered:
+ logger.info(f"Recovered {recovered} joint-name entries from completion markers")
+
+ jn_shards = sorted(glob.glob(os.path.join(output_dir, "joint_names_worker*.json")))
+ for shard in jn_shards:
+ joint_names.update(_load(shard))
+ if joint_names:
+ _save(jn_path, joint_names)
+ _save(os.path.join(output_dir, "joint_count.json"),
+ {k: len(v) for k, v in joint_names.items()})
+ logger.info(f"joint_names.json: {len(joint_names)} entries "
+ f"({len(jn_shards)} shard(s) merged)")
+
+ af_path = os.path.join(output_dir, "clip_frames.json")
+ frames = {}
+ if os.path.isfile(af_path):
+ # Tolerate the legacy in-file summary block from older exports.
+ frames = {k: v for k, v in _load(af_path).items() if k != "_summary"}
+ af_shards = sorted(glob.glob(os.path.join(output_dir, "clip_frames_worker*.json")))
+ for shard in af_shards:
+ frames.update({k: v for k, v in _load(shard).items() if k != "_summary"})
+
+ # Clips written after the last summary flush (or by a worker that was
+ # killed before writing its shard) are on disk but absent here. Scan for
+ # those only โ€” the NPZ read is the expensive part, so already-recorded
+ # clips are left alone.
+ scanned = _scan_missing_frame_counts(os.path.join(output_dir, "motions"), frames)
+ if scanned:
+ logger.info(f"Recovered {len(scanned)} clip frame counts by scanning motions/")
+ frames.update(scanned)
+
+ if frames:
+ _save(af_path, frames)
+ total_frames = sum(frames.values())
+ duration_sec = total_frames / fps if fps else 0.0
+ _save(os.path.join(output_dir, "summary.json"), {
+ "total_clips": len(frames),
+ "total_frames": total_frames,
+ "total_duration_sec": round(duration_sec, 2),
+ "total_duration_min": round(duration_sec / 60, 2),
+ "fps": fps,
+ })
+ logger.info(f"clip_frames.json: {len(frames)} clips "
+ f"({len(af_shards)} shard(s) merged)")
+
+ if not keep_shards:
+ jc_shards = sorted(glob.glob(os.path.join(output_dir, "joint_count_worker*.json")))
+ for shard in jn_shards + af_shards + jc_shards:
+ os.remove(shard)
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Merge per-worker export summary JSONs into the canonical files.")
+ parser.add_argument("--output_dir", type=str, required=True,
+ help="Export directory holding the *_worker{i}.json shards.")
+ parser.add_argument("--fps", type=int, default=30,
+ help="Frame rate for the clip_frames duration summary.")
+ parser.add_argument("--keep_shards", action="store_true",
+ help="Keep the per-worker shard files after merging.")
+ args = parser.parse_args()
+ merge_summaries(args.output_dir, fps=args.fps, keep_shards=args.keep_shards)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/tools/truebones_fbx2glb.py b/data_process/tools/truebones_fbx2glb.py
new file mode 100644
index 0000000..b93386f
--- /dev/null
+++ b/data_process/tools/truebones_fbx2glb.py
@@ -0,0 +1,127 @@
+"""Convert curated Truebones per-clip FBX files to GLB, one to one.
+
+Consumes the flat ``{Species}-{Action}.fbx`` layout (the same input as
+``export_truebones.py``) and writes ``{Species}-{Action}.glb`` next to it,
+preserving mesh, armature, and the clip's single animation take. Resumable:
+existing GLBs are skipped.
+
+Usage (Blender headless):
+ blender -b -P data_process/tools/truebones_fbx2glb.py -- \
+ --data_dir dataset/raw/truebones/animation \
+ --output_dir dataset/raw/truebones/animation_glb
+"""
+
+import argparse
+import os
+import sys
+
+import bpy
+from loguru import logger
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from data_process.utils.blender_export import import_fbx, reset_scene
+
+
+def convert_fbx_to_glb(fbx_path, glb_path):
+ """Import one per-clip FBX and export it as GLB with its animation."""
+ reset_scene()
+ import_fbx(fbx_path)
+
+ armatures = [o for o in bpy.data.objects if o.type == 'ARMATURE']
+ assert armatures, f"No armature found in {fbx_path}"
+ armature = armatures[0]
+ assert armature.animation_data is not None and armature.animation_data.action is not None, \
+ f"Armature '{armature.name}' has no bound action in {fbx_path}"
+
+ # Name the GLB animation after the clip's action; drop the static
+ # object-transform actions the FBX importer binds to mesh objects so the
+ # skeletal take is the only animation in the file.
+ stem = os.path.splitext(os.path.basename(fbx_path))[0]
+ _, sep, action_part = stem.partition('-')
+ if not sep:
+ raise ValueError(
+ f"Clip filename '{stem}' has no '-' separating species from action "
+ f"(expected '{{Species}}-{{Action}}.fbx')")
+ take = armature.animation_data.action
+ take.name = action_part
+ for obj in bpy.data.objects:
+ if obj.type != 'ARMATURE' and obj.animation_data is not None:
+ obj.animation_data_clear()
+ for action in list(bpy.data.actions):
+ if action is not take:
+ bpy.data.actions.remove(action)
+
+ for obj in bpy.data.objects:
+ obj.select_set(obj.type in ('MESH', 'ARMATURE'))
+
+ kwargs = dict(
+ filepath=glb_path,
+ check_existing=False,
+ use_selection=True,
+ export_format='GLB',
+ export_animations=True,
+ )
+ # One animation per GLB. On Blender 3.2 `export_nla_strips=False` merges
+ # everything into a single clip; the option was replaced by
+ # `export_animation_mode` in 3.6 (ACTIVE_ACTIONS emits the armature's
+ # bound action as the only animation).
+ if bpy.app.version >= (3, 6, 0):
+ kwargs['export_animation_mode'] = 'ACTIVE_ACTIONS'
+ else:
+ kwargs['export_nla_strips'] = False
+ bpy.ops.export_scene.gltf(**kwargs)
+ logger.info(f"Saved animated GLB: {glb_path}")
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Convert curated Truebones per-clip FBX files to GLB, one to one.")
+ parser.add_argument('--data_dir', type=str, required=True,
+ help='Flat directory of per-clip {Species}-{Action}.fbx files.')
+ parser.add_argument('--output_dir', type=str, required=True,
+ help='Destination directory for the per-clip GLB files.')
+ parser.add_argument('--worker_id', type=int, default=0,
+ help='Worker index for sharding across workers.')
+ parser.add_argument('--num_workers', type=int, default=1,
+ help='Total number of workers.')
+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]
+ return parser.parse_args(argv)
+
+
+if __name__ == "__main__":
+ args = parse_args()
+
+ fbx_files = sorted(f for f in os.listdir(args.data_dir)
+ if f.lower().endswith('.fbx'))
+ logger.info(f"Found {len(fbx_files)} FBX clips in {args.data_dir}")
+
+ if args.num_workers > 1:
+ fbx_files = fbx_files[args.worker_id::args.num_workers]
+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(fbx_files)} files")
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ error_log = os.path.join(args.output_dir,
+ f'fbx2glb_errors_worker{args.worker_id}.log')
+
+ n_ok = n_skip = n_fail = 0
+ for fbx_file in fbx_files:
+ glb_path = os.path.join(args.output_dir,
+ os.path.splitext(fbx_file)[0] + '.glb')
+ if os.path.exists(glb_path):
+ n_skip += 1
+ continue
+ try:
+ convert_fbx_to_glb(os.path.join(args.data_dir, fbx_file), glb_path)
+ n_ok += 1
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ logger.error(f"Failed to convert {fbx_file}: {e}")
+ with open(error_log, 'a') as f:
+ f.write(f"{fbx_file}\t{e}\n")
+ n_fail += 1
+
+ logger.info(f"Done: {n_ok} converted, {n_skip} skipped, {n_fail} failed "
+ f"-> {args.output_dir}")
+
+ if n_fail:
+ sys.exit(1)
diff --git a/data_process/tools/vis_motion.py b/data_process/tools/vis_motion.py
new file mode 100644
index 0000000..b375673
--- /dev/null
+++ b/data_process/tools/vis_motion.py
@@ -0,0 +1,197 @@
+"""Visualize an exported motion with a root-anchored orientation arrow.
+
+Sanity-check tool for facing-direction annotations: loads an export-stage
+motion NPZ, reconstructs per-frame global joint positions via forward
+kinematics, derives a per-frame orientation vector, and saves an MP4 where
+an arrow at the root joint points along that orientation. With
+``--direction face --face-joints i j`` the arrow shows the heading implied
+by a candidate face-joint pair.
+
+Expected NPZ keys:
+
+ rest_local_pos : (J, 3) rest bone offsets
+ anim_local_pos : (T, J, 3) per-frame local joint positions
+ anim_local_rot : (T, J, 4) per-frame local joint rotations (quaternions)
+ parents : (J,) parent joint indices (-1 for root)
+
+The orientation vector is chosen by ``--direction``:
+
+ velocity (default) โ€” root horizontal velocity, held forward on stalls
+ x / y / z โ€” fixed unit axis (world frame, pre-rotate)
+ face โ€” vector from root to the mean of two face joints
+ specified via ``--face-joints i j``
+
+Usage:
+ python -m data_process.tools.vis_motion \\
+ <path/to/motion.npz> [--output out.mp4] [--fps 30] \\
+ [--direction velocity|x|y|z|face] [--face-joints i j]
+"""
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from Animation import Animation, Quaternions, positions_global # noqa: E402
+from data_process.utils.plotting import render_skeleton_motion_directed # noqa: E402
+import imageio # noqa: E402
+
+
+def load_motion_from_npz(npz_path):
+ """Load a motion NPZ and compute per-frame global joint positions.
+
+ Returns:
+ (parents, global_positions) where ``global_positions`` is a
+ float32 (T, J, 3) array.
+ """
+ data = np.load(npz_path, allow_pickle=True)
+ required = ('rest_local_pos', 'anim_local_pos', 'anim_local_rot', 'parents')
+ missing = [k for k in required if k not in data.files]
+ if missing:
+ raise KeyError(
+ f"{npz_path} is missing required keys {missing}; "
+ f"available keys: {list(data.files)}"
+ )
+
+ rest_local_pos = data['rest_local_pos'] # (J, 3)
+ anim_local_pos = data['anim_local_pos'] # (T, J, 3)
+ anim_local_rot = data['anim_local_rot'] # (T, J, 4)
+ parents = np.asarray(data['parents']) # (J,)
+ njoints = len(parents)
+
+ motion_anim = Animation(
+ rotations=Quaternions(anim_local_rot),
+ positions=anim_local_pos,
+ orients=Quaternions.id(njoints),
+ offsets=rest_local_pos,
+ parents=parents,
+ )
+ global_pos = positions_global(motion_anim).astype(np.float32) # (T, J, 3)
+ return parents, global_pos
+
+
+def compute_directions(global_pos, mode, face_joints=None):
+ """Build a per-frame orientation vector from global joint positions.
+
+ Args:
+ global_pos: (T, J, 3) float array.
+ mode: One of ``'velocity'``, ``'x'``, ``'y'``, ``'z'``, ``'face'``.
+ face_joints: Tuple ``(i, j)`` of joint indices used when
+ ``mode == 'face'``; the direction points from the root joint
+ toward the midpoint of joints ``i`` and ``j``.
+
+ Returns:
+ (T, 3) float32 array. Not normalized โ€” the renderer normalizes
+ internally and skips zero-length frames.
+ """
+ T = global_pos.shape[0]
+
+ if mode == 'velocity':
+ # Horizontal velocity of the root joint, held forward on stalls so
+ # the arrow doesn't flicker off on stationary frames.
+ roots = global_pos[:, 0] # (T, 3)
+ vel = np.zeros_like(roots)
+ vel[1:] = roots[1:] - roots[:-1]
+ if T >= 2:
+ vel[0] = vel[1]
+ vel[:, 1] = 0.0 # project to ground plane (y=up)
+ # Forward-fill through stalls so the arrow stays visible.
+ last = np.zeros(3, dtype=np.float32)
+ for t in range(T):
+ if np.linalg.norm(vel[t]) < 1e-6:
+ vel[t] = last
+ else:
+ last = vel[t]
+ return vel.astype(np.float32)
+
+ if mode in ('x', 'y', 'z'):
+ axis = {'x': [1, 0, 0], 'y': [0, 1, 0], 'z': [0, 0, 1]}[mode]
+ return np.tile(np.array(axis, dtype=np.float32)[None, :], (T, 1))
+
+ if mode == 'face':
+ if face_joints is None or len(face_joints) != 2:
+ raise ValueError("--face-joints i j required when --direction=face")
+ i, j = face_joints
+ mid = 0.5 * (global_pos[:, i] + global_pos[:, j]) # (T, 3)
+ return (mid - global_pos[:, 0]).astype(np.float32)
+
+ raise ValueError(f"unknown direction mode: {mode}")
+
+
+DEFAULT_OUTPUT_DIR = "outputs/motion_vis"
+
+
+def default_output_path(npz_path):
+ """Default output: ``<DEFAULT_OUTPUT_DIR>/<stem>_motion_directed.mp4``."""
+ return os.path.join(DEFAULT_OUTPUT_DIR, f"{Path(npz_path).stem}_motion_directed.mp4")
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ parser.add_argument(
+ 'npz_path', type=str,
+ help='Path to a motion NPZ containing anim_local_pos / anim_local_rot / rest_local_pos / parents',
+ )
+ parser.add_argument(
+ '--output', type=str, default=None,
+ help=f'Output MP4 path (default: {DEFAULT_OUTPUT_DIR}/<input_stem>_motion_directed.mp4)',
+ )
+ parser.add_argument('--fps', type=int, default=30, help='Output video fps (default: 30)')
+ parser.add_argument(
+ '--direction', type=str, default='velocity',
+ choices=['velocity', 'x', 'y', 'z', 'face'],
+ help='Source of the orientation vector (default: velocity)',
+ )
+ parser.add_argument(
+ '--face-joints', type=int, nargs=2, default=None, metavar=('I', 'J'),
+ help='Joint index pair for --direction=face (midpoint target)',
+ )
+ parser.add_argument('--elev', type=float, default=30.0, help='Camera elevation')
+ parser.add_argument('--azim', type=float, default=-60.0, help='Camera azimuth')
+ parser.add_argument(
+ '--no-rotate-root', action='store_true',
+ help='Disable Z-up -> Y-up swap (default: enabled, matches save_skeleton_motion)',
+ )
+ parser.add_argument(
+ '--arrow-scale', type=float, default=0.25,
+ help='Arrow length as a fraction of the scene side (default: 0.25)',
+ )
+ parser.add_argument('--arrow-color', type=str, default='green', help='Arrow color')
+ args = parser.parse_args()
+
+ if not os.path.exists(args.npz_path):
+ parser.error(f"Input NPZ not found: {args.npz_path}")
+
+ parents, global_pos = load_motion_from_npz(args.npz_path)
+ directions = compute_directions(global_pos, args.direction, args.face_joints)
+
+ frames = render_skeleton_motion_directed(
+ parents=parents,
+ positions=global_pos,
+ directions=directions,
+ elev=args.elev,
+ azim=args.azim,
+ rotate_root=not args.no_rotate_root,
+ arrow_scale=args.arrow_scale,
+ arrow_color=args.arrow_color,
+ )
+
+ output_path = args.output or default_output_path(args.npz_path)
+ Path(output_path).parent.mkdir(parents=True, exist_ok=True)
+ imageio.mimwrite(output_path, frames, fps=args.fps)
+ print(
+ f"Saved directed motion ({frames.shape[0]} frames, "
+ f"{global_pos.shape[1]} joints, direction={args.direction}) "
+ f"to {output_path}"
+ )
+
+
+if __name__ == '__main__':
+ main()
diff --git a/data_process/tools/vis_tpose.py b/data_process/tools/vis_tpose.py
new file mode 100644
index 0000000..4c6b432
--- /dev/null
+++ b/data_process/tools/vis_tpose.py
@@ -0,0 +1,296 @@
+"""Visualize skeleton T-poses with joint names annotated, for manual review.
+
+Workflow companion for ``face_select*``: when automatic face-joint
+resolution produces ``"source": "empty"`` (or picks the wrong pair), render
+the T-pose with every joint labeled so the left/right symmetric pair can be
+identified by eye. Each motion NPZ produces one PNG.
+
+Accepts the export-stage NPZ layout (see
+:func:`data_process.utils.blender_export.save_motion`) whose relevant fields are:
+
+ rest_local_pos : (J, 3) local bone offsets at rest
+ rest_local_rot : (J, 4) local bone rotations at rest (quaternions)
+ names : (J,) joint name strings
+ parents : (J,) parent joint indices (-1 for root)
+ offsets : (J, 3) bind-pose offsets (optional โ€” falls back to
+ rest_local_pos when absent)
+
+The T-pose global positions are reconstructed via forward kinematics on the
+rest pose and passed to :func:`save_skeleton_tpose_annotated`.
+
+Modes:
+ - Single: ``--npz_path X`` โ†’ one PNG.
+ - Batch (default): ``--motions_dir M [--output_dir O]`` โ†’ one PNG per
+ unique key (prefix before the first ``-``) discovered in ``M``.
+ - Batch (face_json): ``--face_json F --motions_dir M [--source empty] [--output_dir O]``
+ โ†’ one PNG per key in F whose ``source`` field matches ``--source``.
+
+See ``data_process/scripts/run_joints_vis_tpose.sh`` for the typical batch call.
+"""
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+import numpy as np
+from tqdm import tqdm
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
+
+from Animation import Animation, Quaternions, positions_global # noqa: E402
+from data_process.utils.plotting import save_skeleton_tpose_annotated # noqa: E402
+
+
+REQUIRED_KEYS = ("rest_local_pos", "rest_local_rot", "names", "parents")
+
+
+def load_tpose_from_npz(npz_path):
+ # type: (str) -> Tuple[np.ndarray, List[str], np.ndarray]
+ """Load rest-pose data from a motion NPZ and compute global T-pose positions.
+
+ Returns:
+ ``(parents, joint_names, tpose_positions)`` where ``tpose_positions``
+ is a float32 ``(J, 3)`` array of global joint coordinates at rest.
+ """
+ data = np.load(npz_path, allow_pickle=True)
+ missing = [k for k in REQUIRED_KEYS if k not in data.files]
+ if missing:
+ raise KeyError(
+ "{} is missing required keys {}; available: {}".format(
+ npz_path, missing, list(data.files)
+ )
+ )
+
+ rest_local_pos = data['rest_local_pos'] # (J, 3)
+ rest_local_rot = data['rest_local_rot'] # (J, 4) quaternions
+ parents = np.asarray(data['parents']) # (J,)
+ names = [str(n) for n in data['names']] # length J
+ # Prefer the explicit bind-pose offsets when the export recorded them;
+ # fall back to the rest-pose local translation when absent.
+ offsets = (np.asarray(data['offsets']) if 'offsets' in data.files
+ else rest_local_pos)
+
+ njoints = len(parents)
+
+ rest_anim = Animation(
+ rotations=Quaternions(rest_local_rot[None]), # (1, J, 4)
+ positions=rest_local_pos[None], # (1, J, 3)
+ orients=Quaternions.id(njoints),
+ offsets=offsets,
+ parents=parents,
+ )
+ tpose_positions = positions_global(rest_anim)[0].astype(np.float32) # (J, 3)
+
+ return parents, names, tpose_positions
+
+
+def render_one(npz_path, output_path, *, font_size, joint_marker_size,
+ bone_linewidth):
+ # type: (str, str, int, float, float) -> None
+ """Render one NPZ as an annotated T-pose PNG."""
+ parents, names, tpose_positions = load_tpose_from_npz(npz_path)
+ os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
+ save_skeleton_tpose_annotated(
+ save_path=output_path,
+ parents=parents,
+ positions=tpose_positions,
+ joint_names=names,
+ font_size=font_size,
+ joint_marker_size=joint_marker_size,
+ bone_linewidth=bone_linewidth,
+ )
+
+
+def find_motion_npz(motions_dir, key):
+ # type: (str, str) -> Optional[Path]
+ """Resolve the first motion NPZ whose prefix-before-'-' matches ``key``.
+
+ Motion files are named ``<key>-<clip_name>.npz`` (e.g.
+ ``669a728e57bbceb71e6b8ce6_fbx-mixamo_com.npz``). Any single clip is
+ enough to render the rest-pose skeleton, so we take the first sorted
+ match. Returns ``None`` if no file matches.
+ """
+ hits = sorted(Path(motions_dir).glob("{}-*.npz".format(key)))
+ return hits[0] if hits else None
+
+
+def collect_keys_from_motions_dir(motions_dir):
+ # type: (str) -> List[str]
+ """Collect unique keys (prefix before the first ``-``) from NPZs in ``motions_dir``.
+
+ NPZs without a ``-`` in their stem are treated as their own key.
+ """
+ keys = set()
+ for p in Path(motions_dir).glob("*.npz"):
+ stem = p.stem
+ keys.add(stem.split("-", 1)[0] if "-" in stem else stem)
+ return sorted(keys)
+
+
+DEFAULT_OUTPUT_DIR = "outputs/face_joints_vis"
+
+
+def default_single_output_path(npz_path):
+ # type: (str) -> str
+ """Default single-mode output: ``<DEFAULT_OUTPUT_DIR>/<stem>_tpose_annotated.png``."""
+ return os.path.join(DEFAULT_OUTPUT_DIR,
+ "{}_tpose_annotated.png".format(Path(npz_path).stem))
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ mode = parser.add_mutually_exclusive_group(required=False)
+ mode.add_argument(
+ '--npz_path', type=str, default=None,
+ help='Single mode: path to a motion NPZ (skips batch logic).',
+ )
+ mode.add_argument(
+ '--face_json', type=str, default=None,
+ help='Batch mode (alt): path to face_joint_names.json. Motion files '
+ 'are matched as <motions_dir>/<key>-*.npz (prefix before the '
+ 'first "-"). Without this flag, batch mode scans --motions_dir '
+ 'directly for unique keys.',
+ )
+
+ # Batch-only
+ parser.add_argument(
+ '--motions_dir', type=str, default=None,
+ help='[batch] Directory of motion NPZs (default: sibling "motions/" '
+ 'of --face_json).',
+ )
+ parser.add_argument(
+ '--source', type=str, default='empty',
+ help='[batch] Only render uuids whose "source" field equals this '
+ '(default: "empty"). Use "ALL" to render every key.',
+ )
+ parser.add_argument(
+ '--output_dir', type=str, default=DEFAULT_OUTPUT_DIR,
+ help='[batch] Output directory (default: {}).'.format(DEFAULT_OUTPUT_DIR),
+ )
+ parser.add_argument(
+ '--limit', type=int, default=None,
+ help='[batch] Stop after N uuids (debug aid).',
+ )
+ parser.add_argument(
+ '--overwrite', action='store_true',
+ help='[batch] Re-render even if the output PNG already exists.',
+ )
+
+ # Single-only
+ parser.add_argument(
+ '--output', type=str, default=None,
+ help='[single] Output PNG path (default: '
+ '<output_dir>/<input_stem>_tpose_annotated.png).',
+ )
+
+ # Plot knobs
+ parser.add_argument('--font-size', type=int, default=7,
+ help='Font size for joint name annotations.')
+ parser.add_argument('--joint-marker-size', type=float, default=80.0,
+ help='Joint dot marker size in points^2.')
+ parser.add_argument('--bone-linewidth', type=float, default=2.0,
+ help='Bone line width.')
+ args = parser.parse_args()
+
+ plot_kwargs = dict(
+ font_size=args.font_size,
+ joint_marker_size=args.joint_marker_size,
+ bone_linewidth=args.bone_linewidth,
+ )
+
+ # โ”€โ”€ Single mode โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ if args.npz_path:
+ if not os.path.exists(args.npz_path):
+ parser.error("Input NPZ not found: {}".format(args.npz_path))
+ out = args.output or default_single_output_path(args.npz_path)
+ render_one(args.npz_path, out, **plot_kwargs)
+ print("Saved: {}".format(out))
+ return
+
+ # โ”€โ”€ Batch mode: resolve the list of keys โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ # face_json takes precedence when explicitly provided; otherwise we scan
+ # --motions_dir directly for unique keys.
+ if args.face_json:
+ if not os.path.exists(args.face_json):
+ parser.error("face_json not found: {}".format(args.face_json))
+ motions_dir = args.motions_dir or str(Path(args.face_json).parent / "motions")
+ if not os.path.isdir(motions_dir):
+ parser.error("motions_dir not found: {}".format(motions_dir))
+ with open(args.face_json) as f:
+ face = json.load(f)
+ if args.source == "ALL":
+ keys = sorted(face.keys())
+ else:
+ keys = sorted(
+ k for k, v in face.items()
+ if isinstance(v, dict) and v.get("source") == args.source
+ )
+ source_desc = "face_json={} source={!r}".format(args.face_json, args.source)
+ else:
+ if args.motions_dir is None:
+ parser.error("--motions_dir is required in batch mode")
+ motions_dir = args.motions_dir
+ if not os.path.isdir(motions_dir):
+ parser.error("motions_dir not found: {}".format(motions_dir))
+ keys = collect_keys_from_motions_dir(motions_dir)
+ source_desc = "motions_dir={}".format(motions_dir)
+
+ if args.limit is not None:
+ keys = keys[: args.limit]
+
+ if not keys:
+ print("No entries selected ({})".format(source_desc))
+ return
+
+ os.makedirs(args.output_dir, exist_ok=True)
+ print("motions_dir : {}".format(motions_dir))
+ print("output_dir : {}".format(args.output_dir))
+ print("Rendering {} T-poses ({})".format(len(keys), source_desc))
+
+ rendered = skipped_existing = missing_npz = failed = 0
+ missing_examples = [] # type: List[str]
+ failed_examples = [] # type: List[str]
+
+ for key in tqdm(keys, unit="seq", dynamic_ncols=True):
+ out_path = os.path.join(args.output_dir, "{}.png".format(key))
+ if (not args.overwrite) and os.path.exists(out_path):
+ skipped_existing += 1
+ continue
+ npz = find_motion_npz(motions_dir, key)
+ if npz is None:
+ missing_npz += 1
+ if len(missing_examples) < 5:
+ missing_examples.append(key)
+ continue
+ try:
+ render_one(str(npz), out_path, **plot_kwargs)
+ rendered += 1
+ except Exception as e: # noqa: BLE001
+ failed += 1
+ if len(failed_examples) < 5:
+ failed_examples.append("{}: {}".format(key, e))
+
+ print()
+ print("rendered : {}".format(rendered))
+ print("skipped (exists): {}".format(skipped_existing))
+ print("missing NPZ : {}".format(missing_npz))
+ if missing_examples:
+ for k in missing_examples:
+ print(" - {}".format(k))
+ if missing_npz > len(missing_examples):
+ print(" ... ({} total)".format(missing_npz))
+ print("failed : {}".format(failed))
+ if failed_examples:
+ for line in failed_examples:
+ print(" - {}".format(line))
+ if failed > len(failed_examples):
+ print(" ... ({} total)".format(failed))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/data_process/utils/__init__.py b/data_process/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/data_process/utils/blender_export.py b/data_process/utils/blender_export.py
new file mode 100644
index 0000000..ad9276d
--- /dev/null
+++ b/data_process/utils/blender_export.py
@@ -0,0 +1,1230 @@
+"""Blender utilities for exporting rigged assets to NPZ motion data.
+
+Format-agnostic helpers shared by the Truebones (FBX), Mixamo (FBX),
+Objaverse (GLB/GLTF) and single-asset exporters: scene loading, action
+discovery, skeleton/skin extraction, skeleton pruning, Z-up โ†’ Y-up
+conversion and NPZ/summary writing.
+
+The counterpart for the *animate* direction (NPZ โ†’ rigged character) lives
+in :mod:`data_process.utils.blender_rig`.
+"""
+
+import csv
+import os
+import re
+import json
+import numpy as np
+import bpy
+from math import floor, ceil
+from pathlib import Path
+from loguru import logger
+from Animation import Animation, Quaternions, positions_global
+from data_process.utils.plotting import save_skeleton_motion, save_skeleton_tpose
+from data_process.utils.export_markers import ( # noqa: F401 (re-exported)
+ collect_joint_names_from_markers,
+ is_asset_complete,
+ mark_asset_complete,
+ merge_joint_names as _merge_joint_names,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# GLTF I/O
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def import_gltf(path: str):
+ bpy.ops.import_scene.gltf(filepath=path)
+
+
+def import_fbx(path: str):
+ bpy.ops.import_scene.fbx(filepath=path, use_anim=True)
+
+
+def action_frame_range(action):
+ """Return (start, end) integer frame range for an action."""
+ fr = action.frame_range
+ start = int(floor(fr[0]))
+ end = int(ceil(fr[1]))
+ return start, max(start, end)
+
+
+def find_main_fbx(fbx_dir):
+ """Find the single FBX file that contains all actions for the object type.
+
+ Priority:
+ 1. FBX files without '-': if exactly one, use it; if multiple, pick the one
+ with 'all' (case-insensitive) in its name.
+ 2. If no files without '-', pick the one with '-all' in its name (e.g., Foo-all.fbx).
+ """
+ fbx_files = [f for f in os.listdir(fbx_dir) if f.lower().endswith('.fbx')]
+
+ # Try files without '-' first
+ no_dash = [f for f in fbx_files if '-' not in os.path.splitext(f)[0]]
+ if len(no_dash) == 1:
+ return os.path.join(fbx_dir, no_dash[0])
+ if len(no_dash) > 1:
+ with_all = [f for f in no_dash if 'all' in os.path.splitext(f)[0].lower()]
+ if len(with_all) >= 1:
+ return os.path.join(fbx_dir, with_all[0])
+
+ # Fallback: file with '-all' in name (e.g., Foo-all.fbx)
+ dash_all = [f for f in fbx_files if '-all' in os.path.splitext(f)[0].lower()]
+ if len(dash_all) >= 1:
+ return os.path.join(fbx_dir, dash_all[0])
+
+ return None
+
+
+EXCLUDED_CSV = "excluded.csv"
+
+
+def load_excluded_stems(input_dir: Path) -> set:
+ """Asset stems listed in ``excluded.csv`` beside (or inside) *input_dir*.
+
+ Assets no importer can read โ€” corrupt rigs, singular transforms, NaN
+ axis limits โ€” are recorded there once so every later stage skips them
+ instead of failing on them again on each rerun. The CSV carries a
+ header with at least ``file`` or ``object_id``; ``stage`` and ``reason``
+ are informational.
+ """
+ stems = set()
+ for candidate in (input_dir / EXCLUDED_CSV, input_dir.parent / EXCLUDED_CSV):
+ if not candidate.is_file():
+ continue
+ with open(candidate, newline="") as f:
+ for row in csv.DictReader(f):
+ stem = (row.get("object_id") or "").strip()
+ if not stem:
+ stem = Path((row.get("file") or "").strip()).stem
+ if stem:
+ stems.add(stem)
+ return stems
+
+
+def list_gltf_files(input_dir: Path):
+ exts = {".glb", ".gltf"}
+ paths = sorted(p for p in input_dir.glob("*") if p.is_file() and p.suffix.lower() in exts)
+ excluded = load_excluded_stems(input_dir)
+ if not excluded:
+ return paths
+ kept = [p for p in paths if p.stem not in excluded]
+ if len(kept) != len(paths):
+ logger.info(f"Skipping {len(paths) - len(kept)} asset(s) listed in {EXCLUDED_CSV}")
+ return kept
+
+
+def action_is_relevant_pose(action) -> bool:
+ """Return True if the action has fcurves that animate pose bones."""
+ if action is None or len(action.fcurves) == 0:
+ return False
+ return any(fc.data_path.startswith("pose.bones[") for fc in action.fcurves)
+
+
+def discover_pose_actions(min_frames: int = 0, max_frames: int = float('inf')):
+ """Find all pose-type actions in the current Blender scene.
+
+ By default no frame-count filter is applied; callers may pass
+ *min_frames* / *max_frames* to reject clips outside a desired range.
+
+ Returns:
+ (action_names, frame_ranges) where frame_ranges maps action name to
+ (start_frame, end_frame) tuples.
+ """
+ action_names = []
+ frame_ranges = {}
+
+ for action_name, action in bpy.data.actions.items():
+ if not action_is_relevant_pose(action):
+ logger.info(f"Skipping irrelevant action: {action_name}")
+ continue
+
+ start, end = action_frame_range(action)
+ n_frames = end - start + 1
+
+ if n_frames < min_frames or n_frames > max_frames:
+ logger.info(f"Skipping action '{action_name}': {n_frames} frames "
+ f"(valid range: [{min_frames}, {max_frames}])")
+ continue
+
+ logger.info(f"Found action: {action_name}, frames=[{start},{end}]")
+
+ if action_name not in frame_ranges:
+ action_names.append(action_name)
+ frame_ranges[action_name] = (start, end)
+ else:
+ prev_start, prev_end = frame_ranges[action_name]
+ frame_ranges[action_name] = (min(start, prev_start), max(end, prev_end))
+
+ return action_names, frame_ranges
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Naming helpers
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def sanitize_object_type(obj_type):
+ """Normalize a Truebones object-type directory name.
+
+ Replaces '-' and whitespace with '_' so compound type names (e.g.
+ 'Dog-2') don't collide with the object-action separator dash used in
+ clip names (e.g. 'Dog_2-Walk').
+ """
+ return obj_type.replace('-', '_').replace(' ', '_')
+
+
+def sanitize_action_name(raw_name):
+ """Extract a clean action name from Blender action names.
+
+ Strips Blender prefixes (``Armature|``), colon suffixes (``:Base``),
+ known domain prefixes (``mixamo.com_``), then replaces all
+ non-alphanumeric characters with ``_``.
+
+ Examples:
+ 'Armature|Walk' -> 'Walk'
+ 'Armature|Run|Run:Base' -> 'Run'
+ 'Idle' -> 'Idle'
+ 'Armature|Big-Mouth Attack' -> 'Big_Mouth_Attack'
+ 'mixamo.com_Walk' -> 'Walk'
+ 'Armature|mixamo.com_Run:Base' -> 'Run'
+ """
+ parts = raw_name.split('|')
+ name = parts[1] if len(parts) > 1 else parts[0]
+ name = name.split(':')[0]
+ # Remove known domain prefixes
+ name = re.sub(r'^mixamo\.com_', '', name)
+ # Remove GLTF importer suffixes (e.g. '_GLTF_created_0')
+ name = re.sub(r'_GLTF_created_\d+$', '', name)
+ # Replace whitespace with '_'
+ name = re.sub(r'\s+', '_', name)
+ # Replace all non-alphanumeric characters with '_'
+ name = re.sub(r'[^a-zA-Z0-9]+', '_', name)
+ # Strip leading/trailing underscores
+ name = name.strip('_')
+ return name
+
+
+def truebones_clip_name(fbx_stem):
+ """Clip name for one curated Truebones per-clip FBX stem.
+
+ The curated layout is ``{Species}-{Action}.fbx`` with the first dash
+ separating the two. Export and rendering must derive the same name from
+ it or stage 4's exact-match caption lookup silently yields an empty
+ caption, so both go through this one helper.
+
+ Raises:
+ ValueError: when the stem has no ``-`` to split on.
+ """
+ species, sep, action = fbx_stem.partition('-')
+ if not sep:
+ raise ValueError(
+ f"Truebones clip filename '{fbx_stem}' has no '-' separating "
+ f"species from action; expected '{{Species}}-{{Action}}.fbx'")
+ return f"{sanitize_object_type(species)}-{sanitize_action_name(action)}"
+
+
+def unique_clip_names(action_names):
+ """Map each action name to a collision-free sanitized clip suffix.
+
+ ``sanitize_action_name`` keeps only the second ``|``-separated segment,
+ so distinct actions routinely collapse onto one name โ€” e.g.
+ ``Armature|Take 001|BaseLayer_Obj`` and
+ ``Armature|Take 001|BaseLayer.002_Obj`` both yield ``Take_001``. Export
+ then overwrites ``motions/{clip}.npz`` (last wins) while the renderer's
+ completeness check skips the later actions (first wins), so the NPZ and
+ the render under one clip name end up being different animations and the
+ caption describes the wrong motion.
+
+ Colliding names get a ``_00``/``_01``โ€ฆ suffix in list order. Callers on
+ both sides must pass the SAME ordered action list (the full set of pose
+ actions, before any frame-count filtering) or the suffixes diverge.
+
+ Args:
+ action_names: Ordered action names, as returned by
+ :func:`discover_pose_actions`.
+
+ Returns:
+ ``{action_name: clip_suffix}``.
+ """
+ bases = [sanitize_action_name(n) or "action" for n in action_names]
+ total = {}
+ for base in bases:
+ total[base] = total.get(base, 0) + 1
+
+ seen = {}
+ mapping = {}
+ for action_name, base in zip(action_names, bases):
+ if total[base] == 1:
+ mapping[action_name] = base
+ else:
+ idx = seen.get(base, 0)
+ seen[base] = idx + 1
+ mapping[action_name] = f"{base}_{idx:02d}"
+ return mapping
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Constants (skinning threshold and Blender rigging naming conventions)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+_SKINNING_WEIGHT_EPS = 1e-5
+
+_CONTROL_PREFIXES = ('MCH-', 'ORG-', 'VIS-')
+_CONTROL_SUFFIXES_LOWER = (
+ '_ik', '.ik', '_pole', '.pole',
+ '_target', '_tip',
+ '_ctrl', '.ctrl',
+ '_tweak', '_helper',
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Blender scene utilities
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def reset_scene():
+ bpy.ops.wm.read_factory_settings(use_empty=True)
+
+
+def set_scene_fps(fps: int):
+ bpy.context.scene.render.fps = int(fps)
+
+
+def find_armatures():
+ return [o for o in bpy.data.objects if o.type == "ARMATURE"]
+
+
+def find_meshes():
+ return [o for o in bpy.data.objects if o.type == "MESH"]
+
+
+def choose_main_armature(armatures):
+ """Heuristic: armature with most bones."""
+ if not armatures:
+ return None
+ return sorted(armatures, key=lambda a: len(a.data.bones), reverse=True)[0]
+
+
+def choose_main_mesh(meshes):
+ """Heuristic: mesh with most vertices."""
+ if not meshes:
+ return None
+ return sorted(meshes, key=lambda m: len(m.data.vertices), reverse=True)[0]
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Animation state management
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def clear_animation_state(arm_obj: bpy.types.Object):
+ """Clear action and NLA tracks to avoid cross-action contamination."""
+ ad = arm_obj.animation_data
+ if ad is None:
+ return
+ ad.action = None
+ if ad.nla_tracks:
+ for tr in list(ad.nla_tracks):
+ ad.nla_tracks.remove(tr)
+
+
+def bind_action(arm_obj: bpy.types.Object, action: bpy.types.Action):
+ if arm_obj.animation_data is None:
+ arm_obj.animation_data_create()
+ arm_obj.animation_data.action = action
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Skinning utilities
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def build_vgroup_to_bone_mapping(mesh_obj, arm_obj):
+ """Return dict {vgroup_index -> bone_name} for groups that match a bone."""
+ bone_names = set(arm_obj.data.bones.keys())
+ vg_to_bone = {}
+ for i, vg in enumerate(mesh_obj.vertex_groups):
+ if vg.name in bone_names:
+ vg_to_bone[i] = vg.name
+ return vg_to_bone
+
+
+def build_vertex_groups_sparse(mesh_obj, vg_to_bone: dict, bone_name_to_index: dict, dtype=np.float64):
+ """
+ Returns:
+ vert_groups: len=V list[list[(bone_idx, weight)]]
+ skin_matrix: (V, B) dense skinning weight matrix
+ """
+ verts = mesh_obj.data.vertices
+ V = len(verts)
+ vert_groups = [[] for _ in range(V)]
+ skin_matrix = np.zeros((V, len(bone_name_to_index)), dtype=dtype)
+
+ for i, vert in enumerate(verts):
+ groups = []
+ for g in vert.groups:
+ w = float(g.weight)
+ if w <= 0.0:
+ continue
+ bname = vg_to_bone.get(g.group)
+ if bname is None:
+ continue
+ bidx = bone_name_to_index.get(bname)
+ if bidx is None:
+ continue
+ skin_matrix[i, bidx] = w
+ groups.append((int(bidx), w))
+ vert_groups[i] = groups
+
+ return vert_groups, skin_matrix
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Skeleton topology utilities
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def children_from_parents(parents):
+ ch = [[] for _ in range(len(parents))]
+ for j, p in enumerate(parents):
+ if p >= 0:
+ ch[p].append(j)
+ return ch
+
+
+def name_is_control_bone(name: str) -> bool:
+ return (
+ any(name.startswith(p) for p in _CONTROL_PREFIXES)
+ or any(name.lower().endswith(s) for s in _CONTROL_SUFFIXES_LOWER)
+ )
+
+
+def rotation_amplitude(j, anim):
+ """
+ Maximum rotation angle (radians) of joint j relative to its first-frame
+ orientation, across all frames.
+ """
+ rot_0 = anim.rotations[0:1, j]
+ delta = (-rot_0) * anim.rotations[:, j]
+ cos_half = np.abs(delta.qs[:, 0]).clip(0.0, 1.0)
+ return float(np.max(2.0 * np.arccos(cos_half)))
+
+
+def prune_secondary_roots(armature, name_to_weight_sum):
+ def get_subtree_weight(bone, weight_dict):
+ w = weight_dict.get(bone.name, 0)
+ for child in bone.children:
+ w += get_subtree_weight(child, weight_dict)
+ return w
+
+ all_roots = [bone for bone in armature.data.bones if bone.parent is None]
+ if len(all_roots) <= 1:
+ logger.info("Only one or no root bone found. Skipping prune_secondary_roots.")
+ return
+
+ root_weights = []
+ for rb in all_roots:
+ total_w = get_subtree_weight(rb, name_to_weight_sum)
+ root_weights.append((rb.name, total_w))
+ logger.info(f"Root bone '{rb.name}' has total subtree weight: {total_w:.4f}")
+
+ best_root_name = max(root_weights, key=lambda x: x[1])[0]
+ logger.info(f"Keeping best root: '{best_root_name}'")
+
+ bones_to_remove = []
+ for rb_name, _ in root_weights:
+ if rb_name != best_root_name:
+ rb_obj = armature.data.bones.get(rb_name)
+ if rb_obj:
+ bones_to_remove.extend([rb_obj.name] + [b.name for b in rb_obj.children_recursive])
+
+ if bones_to_remove:
+ logger.info(f"Removing {len(bones_to_remove)} bones from secondary root subtrees.")
+ bpy.context.view_layer.objects.active = armature
+ bpy.ops.object.mode_set(mode='EDIT')
+ eb = armature.data.edit_bones
+ for b_name in bones_to_remove:
+ if b_name in eb:
+ eb.remove(eb[b_name])
+ bpy.ops.object.mode_set(mode='OBJECT')
+ bpy.context.view_layer.update()
+
+
+def collapse_unused_root(anim: Animation):
+ """Remove an unskinned root with a single child; the child becomes the new root.
+
+ Uses direct local-space composition instead of global FK round-trip
+ to avoid floating-point drift.
+ """
+ parents = anim.parents.copy()
+ F, J = anim.positions.shape[:2]
+ assert J == len(parents)
+
+ root_ids = np.where(np.array(parents) == -1)[0]
+ assert len(root_ids) == 1, "Multiple root joints found."
+ root_id = root_ids[0]
+
+ all_children = children_from_parents(parents)
+ assert len(all_children[root_id]) == 1, "Root must have exactly one child."
+ true_root_id = all_children[root_id][0]
+
+ def get_all_descendants(idx):
+ res = [idx]
+ for c in all_children[idx]:
+ res.extend(get_all_descendants(c))
+ return res
+
+ keep = sorted(get_all_descendants(true_root_id))
+ new_index = {old: new for new, old in enumerate(keep)}
+ newJ = len(keep)
+
+ # Build new parent array
+ new_parents = np.full((newJ,), -1, dtype=np.int32)
+ for old_j in keep:
+ if old_j != true_root_id:
+ new_parents[new_index[old_j]] = new_index[parents[old_j]]
+
+ # Copy local transforms for all kept bones
+ new_rot_qs = anim.rotations.qs[:, keep].copy()
+ new_pos = anim.positions[:, keep].copy()
+ new_offsets = anim.offsets[keep].copy()
+
+ # Compose root transform into the new root (former child):
+ # new_child_local = old_root_local * old_child_local
+ new_root_idx = new_index[true_root_id]
+ root_rot = anim.rotations[:, root_id]
+ new_rot_qs[:, new_root_idx] = (root_rot * anim.rotations[:, true_root_id]).qs
+ new_pos[:, new_root_idx] = (
+ anim.positions[:, root_id] + root_rot * anim.positions[:, true_root_id]
+ )
+
+ # Compose rest-pose offset
+ rest_root_rot = Quaternions(anim.rotations.qs[0:1, root_id])
+ new_offsets[new_root_idx] = (
+ anim.offsets[root_id] + (rest_root_rot * anim.offsets[true_root_id:true_root_id+1])[0]
+ )
+
+ new_anim = Animation(
+ rotations=Quaternions(new_rot_qs),
+ positions=new_pos,
+ orients=anim.orients[keep].copy(),
+ offsets=new_offsets,
+ parents=new_parents,
+ )
+ return new_anim, keep
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Skeleton pruning
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+# --- Prunable bone detection ------------------------------------------------
+
+def _is_prunable(j, skin_matrix, anims_list, rest_anim, names,
+ offset_eps, pos_var_eps, rot_eps,
+ mean_bone_len=1.0, is_leaf=True,
+ consider_parent_rotate=True):
+ """
+ Return True if bone j should be removed from the skeleton.
+ Works with one or more animation clips for aggregated decisions.
+
+ Conditions (checked in order, first match wins):
+
+ (0) Name convention โ€” Blender control/mechanism prefix or suffix.
+ (1) Near-zero offset โ€” bone head coincident with parent in rest pose.
+ (2) Near-zero positionโ€” bone stays at parent origin in ALL clips.
+ (3) Non-rigid (leaf) โ€” bone length varies from rest in ANY clip;
+ likely an IK-stretch control.
+ (4) Zero rotation DOF โ€” bone never rotates above threshold in ALL clips.
+ For leaf bones, when *consider_parent_rotate* is True, only prune
+ if the parent also doesn't rotate in ANY clip โ€” a static leaf on a
+ rotating parent is a meaningful end-effector (e.g. spider toe tip).
+ When False, prune such leaves regardless of parent motion.
+
+ Distance conditions are normalized by mean_bone_len (scale-agnostic).
+ """
+ # Skinned bones are never prunable
+ if skin_matrix[:, j].max() >= _SKINNING_WEIGHT_EPS:
+ return False
+
+ if name_is_control_bone(names[j]): # (0)
+ return True
+
+ norm = max(mean_bone_len, 1e-9)
+ rest_pos_j = rest_anim.positions[0, j]
+
+ if np.linalg.norm(rest_pos_j) / norm < offset_eps: # (1)
+ return True
+
+ if all(np.max(np.linalg.norm(a.positions[:, j], axis=-1)) / norm # (2)
+ < offset_eps for a in anims_list):
+ return True
+
+ if is_leaf: # (3)
+ if any(np.max(np.linalg.norm(a.positions[:, j] - rest_pos_j,
+ axis=-1)) / norm > pos_var_eps
+ for a in anims_list):
+ return True
+
+ if rot_eps > 0.0 and all(rotation_amplitude(j, a) < rot_eps # (4)
+ for a in anims_list):
+ if is_leaf and consider_parent_rotate:
+ parent_j = anims_list[0].parents[j]
+ if parent_j >= 0 and any(rotation_amplitude(parent_j, a) >= rot_eps
+ for a in anims_list):
+ return False # parent rotates โ†’ keep this end-effector
+ return True
+
+ return False
+
+
+# --- Bone merge operation ---------------------------------------------------
+
+def _merge_passthrough_bone(anims_list, rest_anim, names, skin_matrix, j):
+ """Collapse pass-through bone j into its single child across all clips.
+
+ The child inherits j's transform so that global positions are preserved.
+ """
+ parents = list(anims_list[0].parents)
+ childrens = children_from_parents(parents)
+ c = childrens[j][0]
+
+ # Compose rest-pose offset: child_new = j_offset + rot_j * child_offset
+ new_offsets = anims_list[0].offsets.copy()
+ rest_rot_j0 = Quaternions(rest_anim.rotations.qs[0:1, j])
+ new_offsets[c] = new_offsets[j] + (rest_rot_j0 * new_offsets[c:c+1])[0]
+
+ # Re-parent child to j's parent, build index mapping
+ new_parents = list(parents)
+ new_parents[c] = parents[j]
+ keep_idxs = [i for i in range(len(parents)) if i != j]
+ old_to_new = {old: new for new, old in enumerate(keep_idxs)}
+ final_parents = np.array(
+ [old_to_new[new_parents[i]] if new_parents[i] >= 0 else -1
+ for i in keep_idxs],
+ dtype=np.int32,
+ )
+
+ # Update rest anim
+ rest_rot_j = rest_anim.rotations[:, j]
+ new_rest_pos = rest_anim.positions.copy()
+ new_rest_rots = rest_anim.rotations.qs.copy()
+ new_rest_pos[:, c] = rest_anim.positions[:, j] + rest_rot_j * rest_anim.positions[:, c]
+ new_rest_rots[:, c] = (rest_rot_j * rest_anim.rotations[:, c]).qs
+ new_rest_anim = Animation(
+ rotations=Quaternions(new_rest_rots[:, keep_idxs]),
+ positions=new_rest_pos[:, keep_idxs],
+ orients=rest_anim.orients[keep_idxs],
+ offsets=new_offsets[keep_idxs],
+ parents=final_parents,
+ )
+
+ # Update each animation clip
+ new_anims = []
+ for anim in anims_list:
+ rot_j = anim.rotations[:, j]
+ new_anim_pos = anim.positions.copy()
+ new_anim_rots = anim.rotations.qs.copy()
+ new_anim_pos[:, c] = anim.positions[:, j] + rot_j * anim.positions[:, c]
+ new_anim_rots[:, c] = (rot_j * anim.rotations[:, c]).qs
+ new_anims.append(Animation(
+ rotations=Quaternions(new_anim_rots[:, keep_idxs]),
+ positions=new_anim_pos[:, keep_idxs],
+ orients=anim.orients[keep_idxs],
+ offsets=new_offsets[keep_idxs],
+ parents=final_parents,
+ ))
+
+ new_names = [names[i] for i in keep_idxs]
+ new_skin = skin_matrix[:, keep_idxs]
+ return new_anims, new_rest_anim, new_names, new_skin
+
+
+# --- Helpers ----------------------------------------------------------------
+
+def _compute_mean_bone_len(rest_anim):
+ """Mean bone length from rest-pose offsets, excluding root."""
+ bone_lens = np.linalg.norm(rest_anim.offsets[1:], axis=-1)
+ valid_lens = bone_lens[bone_lens > 1e-6]
+ return float(valid_lens.mean()) if len(valid_lens) > 0 else 1.0
+
+
+def _find_deepest_in_chain(j, children_map, cand_set):
+ """Walk from candidate j down the single-child chain to the deepest candidate."""
+ while children_map[j][0] in cand_set:
+ j = children_map[j][0]
+ return j
+
+
+def _find_root(parents):
+ """Return the index of the root joint (parent == -1)."""
+ return int(np.where(np.array(parents) == -1)[0][0])
+
+
+# --- Pruning passes --------------------------------------------------------
+
+def _pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,
+ offset_eps, pos_var_eps, rot_eps, min_joints,
+ consider_parent_rotate=True):
+ """Pass 1: iteratively remove prunable leaf bones."""
+ count = 0
+ mbl = _compute_mean_bone_len(rest_anim)
+ while True:
+ n_joints = len(anims_list[0].parents)
+ childrens = children_from_parents(anims_list[0].parents)
+ candidates = [j for j in range(n_joints)
+ if len(childrens[j]) == 0
+ and _is_prunable(j, skin_matrix, anims_list, rest_anim, names,
+ offset_eps, pos_var_eps, rot_eps,
+ mean_bone_len=mbl, is_leaf=True,
+ consider_parent_rotate=consider_parent_rotate)]
+ if not candidates:
+ break
+ keep = [j for j in range(n_joints) if j not in set(candidates)]
+ if len(keep) < min_joints:
+ break
+ logger.info(f"Pruning leaf bones: {[names[j] for j in candidates]}")
+ anims_list = [a[:, keep].copy() for a in anims_list]
+ rest_anim = rest_anim[:, keep].copy()
+ names = [names[j] for j in keep]
+ skin_matrix = skin_matrix[:, keep]
+ count += len(candidates)
+ return anims_list, rest_anim, names, skin_matrix, count
+
+
+def _pass_promote_root(anims_list, rest_anim, names, skin_matrix):
+ """Pass 2: collapse unskinned single-child root into its child."""
+ count = 0
+ while True:
+ root_idx = _find_root(anims_list[0].parents)
+ children = children_from_parents(anims_list[0].parents)[root_idx]
+ if not (len(children) == 1
+ and skin_matrix[:, root_idx].max() < _SKINNING_WEIGHT_EPS):
+ break
+ logger.info(f"Promoting root child (removing '{names[root_idx]}')")
+ new_anims = []
+ keep_idxs = None
+ for anim in anims_list:
+ new_anim, ki = collapse_unused_root(anim)
+ if keep_idxs is None:
+ keep_idxs = ki
+ else:
+ assert ki == keep_idxs, "Root-clean keep_idxs mismatch across clips"
+ new_anims.append(new_anim)
+ rest_anim, rest_ki = collapse_unused_root(rest_anim)
+ assert rest_ki == keep_idxs, "Root-clean keep_idxs mismatch for rest_anim"
+ anims_list = new_anims
+ names = [names[j] for j in keep_idxs]
+ skin_matrix = skin_matrix[:, keep_idxs]
+ count += 1
+ return anims_list, rest_anim, names, skin_matrix, count
+
+
+def _pass_merge_passthrough(anims_list, rest_anim, names, skin_matrix,
+ offset_eps, pos_var_eps, rot_eps, min_joints):
+ """Pass 3: merge prunable single-child (non-root) bones, deepest first."""
+ count = 0
+ mbl = _compute_mean_bone_len(rest_anim)
+ while True:
+ n_joints = len(anims_list[0].parents)
+ childrens = children_from_parents(anims_list[0].parents)
+ root_idx = _find_root(anims_list[0].parents)
+ candidates = [j for j in range(n_joints)
+ if j != root_idx
+ and len(childrens[j]) == 1
+ and _is_prunable(j, skin_matrix, anims_list, rest_anim, names,
+ offset_eps, pos_var_eps, rot_eps,
+ mean_bone_len=mbl, is_leaf=False)]
+ if not candidates:
+ break
+ cand_set = set(candidates)
+ j = _find_deepest_in_chain(candidates[0], childrens, cand_set)
+ logger.info(f"Merging pass-through bone '{names[j]}' into its child")
+ anims_list, rest_anim, names, skin_matrix = \
+ _merge_passthrough_bone(anims_list, rest_anim, names, skin_matrix, j)
+ count += 1
+ return anims_list, rest_anim, names, skin_matrix, count
+
+
+# --- Main entry points ------------------------------------------------------
+
+def prune_skeleton_shared(anims_list, rest_anim, names, skin_matrix,
+ offset_eps=1e-3, pos_var_eps=1e-3,
+ rot_eps=np.deg2rad(2.0), min_joints=4,
+ consider_parent_rotate=True):
+ """Iterative skeleton pruning across multiple animation clips.
+
+ Guarantees identical topology across all clips. Three passes per
+ iteration, repeated until no pass makes progress:
+
+ 1. Remove prunable leaf bones
+ 2. Promote child when root is unskinned with single child
+ 3. Merge prunable pass-through bones into their children
+
+ Args:
+ consider_parent_rotate: If True (default), leaf bones that never
+ rotate are preserved when their parent rotates โ€” treating them
+ as meaningful end-effectors (e.g. spider toe tips). If False,
+ such leaves are pruned regardless of parent motion.
+ """
+ stats = {"leaf": 0, "root_promoted": 0, "passthrough": 0}
+
+ while len(anims_list[0].parents) > min_joints:
+ anims_list, rest_anim, names, skin_matrix, n_leaf = \
+ _pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,
+ offset_eps, pos_var_eps, rot_eps, min_joints,
+ consider_parent_rotate=consider_parent_rotate)
+
+ anims_list, rest_anim, names, skin_matrix, n_root = \
+ _pass_promote_root(anims_list, rest_anim, names, skin_matrix)
+
+ anims_list, rest_anim, names, skin_matrix, n_pass = \
+ _pass_merge_passthrough(anims_list, rest_anim, names, skin_matrix,
+ offset_eps, pos_var_eps, rot_eps, min_joints)
+
+ stats["leaf"] += n_leaf
+ stats["root_promoted"] += n_root
+ stats["passthrough"] += n_pass
+
+ if n_leaf + n_root + n_pass == 0:
+ break
+
+ logger.info(f"Pruning summary: {stats}, final joints: {len(anims_list[0].parents)}")
+ return anims_list, rest_anim, names, skin_matrix
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Motion filtering (T-pose frame removal)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def remove_tpose_frames(anim, rest_anim, rot_thresh=np.deg2rad(1.0)):
+ """Remove frames whose non-root joint rotations match the rest pose.
+
+ For each frame, computes the maximum rotation angle (over all non-root
+ joints) between the frame's local rotation and the rest-pose rotation.
+ Frames where this maximum is below *rot_thresh* are considered T-pose
+ duplicates and removed.
+
+ Args:
+ anim: Animation with shape (F, J, ...).
+ rest_anim: Single-frame rest-pose Animation (1, J, ...).
+ rot_thresh: Maximum allowed rotation deviation (radians) for a
+ frame to be considered identical to the T-pose.
+
+ Returns:
+ (pruned_anim, n_removed) โ€” the filtered Animation and the count
+ of removed frames. Returns the original if no frames are removed
+ or if removing them would leave fewer than 2 frames.
+ """
+ F, J = anim.rotations.qs.shape[:2]
+ if J < 2:
+ return anim, 0
+
+ # rest rotation for non-root joints: (1, J-1, 4)
+ rest_rot = rest_anim.rotations[:, 1:] # Quaternions (1, J-1)
+ frame_rot = anim.rotations[:, 1:] # Quaternions (F, J-1)
+
+ # relative rotation: rest^-1 * frame โ†’ identity when poses match
+ delta = (-rest_rot) * frame_rot # (F, J-1)
+ # angle = 2 * arccos(|w|)
+ cos_half = np.abs(delta.qs[..., 0]).clip(0.0, 1.0) # (F, J-1)
+ angles = 2.0 * np.arccos(cos_half) # (F, J-1)
+
+ max_angle_per_frame = angles.max(axis=1) # (F,)
+ keep_mask = max_angle_per_frame >= rot_thresh
+
+ n_removed = int((~keep_mask).sum())
+ if n_removed == 0:
+ return anim, 0
+
+ keep_indices = np.where(keep_mask)[0]
+ if len(keep_indices) < 2:
+ # Don't strip if it would leave fewer than 2 frames
+ return anim, 0
+
+ pruned = Animation(
+ rotations=Quaternions(anim.rotations.qs[keep_indices]),
+ positions=anim.positions[keep_indices],
+ orients=anim.orients.copy(),
+ offsets=anim.offsets.copy(),
+ parents=anim.parents.copy(),
+ )
+ return pruned, n_removed
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Animation extraction helpers
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def extract_rest_pose(anim_data_bones, armature, dtype=np.float64, apply_world=True):
+ """Extract local rest-pose positions and rotations for all bones."""
+ nbones = len(anim_data_bones)
+ rest_local_pos = np.zeros((nbones, 3), dtype=dtype)
+ rest_local_rot = np.zeros((nbones, 4), dtype=dtype)
+
+ for idxb, bone in enumerate(anim_data_bones):
+ if bone.parent:
+ mat = bone.parent.matrix_local.inverted() @ bone.matrix_local
+ rest_local_pos[idxb] = np.array(mat.to_translation(), dtype=dtype)
+ rest_local_rot[idxb] = np.array(mat.to_quaternion(), dtype=dtype)
+ else:
+ if apply_world:
+ mat = armature.matrix_world @ bone.matrix_local
+ scale = np.array(armature.matrix_world.to_scale(), dtype=dtype)
+ rest_local_pos[idxb] = np.array(mat.to_translation(), dtype=dtype) / scale
+ else:
+ mat = bone.matrix_local
+ rest_local_pos[idxb] = np.array(mat.to_translation(), dtype=dtype)
+ rest_local_rot[idxb] = np.array(mat.to_quaternion(), dtype=dtype)
+
+ return rest_local_pos, rest_local_rot
+
+
+def extract_animation_frames(armature, start_frame, end_frame, nbones, dtype=np.float64, apply_world=True):
+ """Scrub through frames and extract per-frame local bone transforms."""
+ nframes = end_frame - start_frame + 1
+ anim_local_pos = np.zeros((nframes, nbones, 3), dtype=dtype)
+ anim_local_rot = np.zeros((nframes, nbones, 4), dtype=dtype)
+
+ pose_bones = armature.pose.bones
+ assert len(pose_bones) > 0, f"No pose bones in armature {armature.name}"
+
+ for idxf, f in enumerate(range(start_frame, end_frame + 1)):
+ bpy.context.scene.frame_set(f) # evaluates the depsgraph
+
+ for idxb, bone in enumerate(pose_bones):
+ if bone.parent:
+ mat = bone.parent.matrix.inverted() @ bone.matrix
+ anim_local_pos[idxf, idxb] = np.array(mat.to_translation(), dtype=dtype)
+ anim_local_rot[idxf, idxb] = np.array(mat.to_quaternion(), dtype=dtype)
+ else:
+ if apply_world:
+ mat = armature.matrix_world @ bone.matrix
+ scale = np.array(armature.matrix_world.to_scale(), dtype=dtype)
+ anim_local_pos[idxf, idxb] = np.array(mat.to_translation(), dtype=dtype) / scale
+ else:
+ mat = bone.matrix
+ anim_local_pos[idxf, idxb] = np.array(mat.to_translation(), dtype=dtype)
+ anim_local_rot[idxf, idxb] = np.array(mat.to_quaternion(), dtype=dtype)
+
+ return anim_local_pos, anim_local_rot
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Skeleton setup (Blender scene โ†’ numpy arrays)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def compute_bone_weight_sums(mesh, armature):
+ """Sum skin weights per bone across all vertices."""
+ vg_to_bone = build_vgroup_to_bone_mapping(mesh, armature)
+ weight_sums = {bname: 0.0 for bname in armature.data.bones.keys()}
+ for v in mesh.data.vertices:
+ for g in v.groups:
+ bname = vg_to_bone.get(g.group)
+ if bname:
+ weight_sums[bname] += g.weight
+ return weight_sums
+
+
+def build_skeleton_arrays(armature, mesh, dtype=np.float64, apply_world=True):
+ """Extract skeleton topology, rest pose, and skin matrix from Blender objects.
+
+ Returns:
+ bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix
+ """
+ bones = armature.data.bones
+ bone_names = list(bones.keys())
+ bone_name_to_index = {b.name: i for i, b in enumerate(bones)}
+
+ if mesh is None:
+ skin_matrix = np.zeros((0, len(bone_names)), dtype=dtype)
+ else:
+ vg_to_bone = build_vgroup_to_bone_mapping(mesh, armature)
+ _, skin_matrix = build_vertex_groups_sparse(mesh, vg_to_bone, bone_name_to_index)
+
+ parents_array = np.array([
+ -1 if bone.parent is None else bones.find(bone.parent.name)
+ for bone in bones
+ ], dtype=np.int32)
+
+ rest_local_pos, rest_local_rot = extract_rest_pose(bones, armature, dtype, apply_world=apply_world)
+
+ return bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix
+
+
+def build_rest_animation(parents_array, rest_local_pos, rest_local_rot, nbones):
+ """Build a single-frame Animation object from rest pose arrays."""
+ return Animation(
+ rotations=Quaternions(rest_local_rot[None]),
+ positions=rest_local_pos[None],
+ orients=Quaternions.id(nbones),
+ offsets=rest_local_pos,
+ parents=parents_array,
+ )
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Coordinate conversion (Z-up โ†’ Y-up)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _apply_root_rotation(anim, root_quat):
+ """Apply *root_quat* to the root joint of an Animation (rotations, positions, offsets)."""
+ rots = anim.rotations.copy()
+ pos = anim.positions.copy()
+ off = anim.offsets.copy()
+ rots[:, 0] = root_quat * anim.rotations[:, 0]
+ pos[:, 0] = root_quat * anim.positions[:, 0]
+ off[0] = (root_quat * anim.offsets[0:1])[0]
+ return Animation(
+ rotations=rots, positions=pos,
+ orients=anim.orients, offsets=off, parents=anim.parents,
+ )
+
+
+def apply_zup_to_yup(anim, rest_anim):
+ """Rotate root bone -90 deg around X to convert Z-up to Y-up coordinate system."""
+ root_quat = Quaternions.from_euler(np.array([[-np.pi / 2, 0, 0]]))
+ return _apply_root_rotation(anim, root_quat), _apply_root_rotation(rest_anim, root_quat)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# High-level pipeline helpers (shared by FBX and GLB exporters)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def load_scene(import_fn, path, fps=30):
+ """Reset scene, set FPS, import via *import_fn*, select main armature/mesh.
+
+ Returns (armature, mesh). Mesh may be None if none found.
+ """
+ reset_scene()
+ set_scene_fps(fps=fps)
+ import_fn(str(path))
+
+ armatures = find_armatures()
+ assert len(armatures) > 0, f"No armature found in {path}."
+ armature = choose_main_armature(armatures)
+ logger.info(f"Using armature: {armature.name}, bones={len(armature.data.bones)}")
+
+ meshes = find_meshes()
+ mesh = choose_main_mesh(meshes) if meshes else None
+ if mesh is not None:
+ logger.info(f"Using mesh: {mesh.name}, vertices={len(mesh.data.vertices)}")
+
+ return armature, mesh
+
+
+def prepare_skeleton(armature, mesh=None, dtype=np.float64, apply_world=True):
+ """Prune secondary roots, build skeleton arrays, and Zโ†’Y rest poses.
+
+ When *mesh* is None (e.g. Mixamo clips without a character mesh), the
+ secondary-root prune step is skipped and skin_matrix is returned empty.
+
+ Returns a dict with keys: nbones, bone_names, parents_array,
+ rest_local_pos, rest_local_rot, skin_matrix, rest_anim, rest_anim_shared.
+ """
+ if mesh is not None:
+ weight_sums = compute_bone_weight_sums(mesh, armature)
+ prune_secondary_roots(armature, weight_sums)
+ bpy.context.view_layer.update()
+
+ nbones = len(armature.data.bones)
+ bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix = \
+ build_skeleton_arrays(armature, mesh, dtype, apply_world=apply_world)
+
+ rest_anim = build_rest_animation(parents_array, rest_local_pos, rest_local_rot, nbones)
+ _, rest_anim_shared = apply_zup_to_yup(rest_anim, rest_anim)
+
+ return {
+ 'nbones': nbones,
+ 'bone_names': bone_names,
+ 'parents_array': parents_array,
+ 'rest_local_pos': rest_local_pos,
+ 'rest_local_rot': rest_local_rot,
+ 'skin_matrix': skin_matrix,
+ 'rest_anim': rest_anim,
+ 'rest_anim_shared': rest_anim_shared,
+ }
+
+
+def extract_all_actions(armature, frame_ranges, rest_local_pos, parents_array,
+ rest_anim, nbones, dtype=np.float64, apply_world=True):
+ """Extract (name, Animation) for every action in *frame_ranges*.
+
+ Args:
+ frame_ranges: dict mapping action_name -> (start_frame, end_frame).
+
+ Applies Z-up โ†’ Y-up conversion to each extracted clip.
+ """
+ extracted = []
+ for action_name, (start_frame, end_frame) in frame_ranges.items():
+ action = bpy.data.actions.get(action_name)
+ if action is None:
+ logger.warning(f"Action '{action_name}' not found; skipping.")
+ continue
+
+ clear_animation_state(armature)
+ bind_action(armature, action)
+
+ anim_local_pos, anim_local_rot = extract_animation_frames(
+ armature, start_frame, end_frame, nbones, dtype, apply_world=apply_world
+ )
+
+ anim = Animation(
+ rotations=Quaternions(anim_local_rot),
+ positions=anim_local_pos,
+ orients=Quaternions.id(nbones),
+ offsets=rest_local_pos,
+ parents=parents_array,
+ )
+ anim, _ = apply_zup_to_yup(anim, rest_anim)
+ extracted.append((action_name, anim))
+
+ return extracted
+
+
+def save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_pruned):
+ """Render and save the rest-pose skeleton PNG for one asset."""
+ os.makedirs(tpos_save_dir, exist_ok=True)
+ tpos_path = os.path.join(tpos_save_dir, f"{save_name}.png")
+ rest_global_pos = positions_global(rest_anim_pruned)
+ save_skeleton_tpose(
+ save_path=tpos_path,
+ parents=rest_anim_pruned.parents,
+ positions=rest_global_pos[0],
+ )
+ logger.info(f"Saved rest pose visualization: {tpos_path}")
+ return tpos_path
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Shared NPZ save and summary helpers
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def save_motion(motion_path, vis_path, anim, rest_anim, names, skin_matrix, fps,
+ action_name="", save_vis=True):
+ """Save motion NPZ (float64) and optional MP4.
+
+ The MP4 preview dominates per-clip export time; pass ``save_vis=False``
+ to skip it on bulk runs.
+ """
+ if save_vis:
+ global_pos = positions_global(anim)
+ save_skeleton_motion(
+ save_path=vis_path,
+ parents=anim.parents,
+ positions=global_pos,
+ fps=fps,
+ )
+ logger.info(f"Saved visualization: {vis_path}")
+
+ np.savez(
+ motion_path,
+ rest_local_pos=rest_anim.positions[0],
+ rest_local_rot=rest_anim.rotations[0].qs,
+ anim_local_pos=anim.positions,
+ anim_local_rot=anim.rotations.qs,
+ offsets=anim.offsets,
+ names=np.array(names),
+ skin_matrix=skin_matrix,
+ fps=fps,
+ parents=anim.parents,
+ action_name=action_name,
+ )
+ logger.info(f"Saved motion data: {motion_path}")
+
+
+def scan_clip_frames(motion_dir):
+ """Scan saved NPZ files in motion_dir and return {clip_name: nframes}."""
+ frames = {}
+ if not os.path.isdir(motion_dir):
+ return frames
+ for npz_file in sorted(os.listdir(motion_dir)):
+ if not npz_file.endswith('.npz'):
+ continue
+ clip_name = os.path.splitext(npz_file)[0]
+ try:
+ data = np.load(os.path.join(motion_dir, npz_file), allow_pickle=True)
+ frames[clip_name] = int(data['anim_local_rot'].shape[0])
+ except Exception:
+ pass
+ return frames
+
+
+def write_clip_frames_json(path, frames, fps=30):
+ """Write clip_frames.json: a pure {clip_name: nframes} mapping."""
+ total_frames = sum(frames.values())
+ total_clips = len(frames)
+ total_duration_sec = total_frames / fps if fps else 0.0
+ with open(path, 'w') as f:
+ json.dump(frames, f, indent=2)
+ logger.info(f"Saved clip frames ({total_clips} clips, {total_frames} frames, "
+ f"{total_duration_sec / 60:.1f} min) to {path}")
+
+
+def write_summary_json(path, frames, fps=30):
+ """Write summary.json with the dataset-level summary of the export."""
+ total_frames = sum(frames.values())
+ total_duration_sec = total_frames / fps if fps else 0.0
+ with open(path, 'w') as f:
+ json.dump({
+ "total_clips": len(frames),
+ "total_frames": total_frames,
+ "total_duration_sec": round(total_duration_sec, 2),
+ "total_duration_min": round(total_duration_sec / 60, 2),
+ "fps": fps,
+ }, f, indent=2)
+ logger.info(f"Saved export metadata to {path}")
+
+
+def write_export_summary(output_dir, all_joint_names, fps=30, worker_suffix=""):
+ """Write per-run summary JSONs: clip_frames, joint_names, joint_count,
+ and (for non-sharded runs) summary.json with the dataset-level summary.
+
+ ``all_joint_names`` only covers assets exported by *this* run โ€” assets
+ skipped via their completion marker return nothing. Writing it verbatim
+ would therefore truncate ``joint_names.json`` on every resumed run, so
+ for the canonical (non-sharded) file it is merged over the durable
+ per-asset markers and whatever the file already held. Shard files stay
+ run-local; :func:`~data_process.tools.merge_summaries.merge_summaries`
+ does the same union for them.
+
+ Args:
+ output_dir: Root directory containing the "motions/" NPZ folder.
+ all_joint_names: dict mapping obj_type/save_name -> list[str] bone names.
+ fps: Frame rate used for duration summary.
+ worker_suffix: Optional suffix (e.g. "_worker0") appended to filenames.
+ When set, summary.json is left to merge_summaries so
+ parallel workers never race on it.
+ """
+ frames = scan_clip_frames(os.path.join(output_dir, "motions"))
+ write_clip_frames_json(
+ os.path.join(output_dir, f"clip_frames{worker_suffix}.json"),
+ frames, fps=fps,
+ )
+ if not worker_suffix:
+ write_summary_json(os.path.join(output_dir, "summary.json"), frames, fps=fps)
+
+ joint_names_path = os.path.join(output_dir, f"joint_names{worker_suffix}.json")
+ if worker_suffix:
+ merged = dict(all_joint_names)
+ else:
+ merged = _merge_joint_names(output_dir, all_joint_names, log=logger.info)
+ with open(joint_names_path, 'w') as f:
+ json.dump(merged, f, indent=2)
+ logger.info(f"Saved joint names ({len(merged)} entries) to {joint_names_path}")
+ all_joint_names = merged
+
+ joint_count = {k: len(v) for k, v in all_joint_names.items()}
+ joint_count_path = os.path.join(output_dir, f"joint_count{worker_suffix}.json")
+ with open(joint_count_path, 'w') as f:
+ json.dump(joint_count, f, indent=2)
+ logger.info(f"Saved joint count to {joint_count_path}")
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Per-asset completion markers (resume bookkeeping)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+#
+# Re-exported from data_process.utils.export_markers, which is deliberately
+# free of any bpy import so plain-Python tools (merge_summaries) can use it.
+
+__all__ = [n for n in dir() if not n.startswith('_')] + [
+ "is_asset_complete", "mark_asset_complete",
+ "collect_joint_names_from_markers", "merge_joint_names",
+]
diff --git a/data_process/utils/blender_render.py b/data_process/utils/blender_render.py
new file mode 100644
index 0000000..5c8a25b
--- /dev/null
+++ b/data_process/utils/blender_render.py
@@ -0,0 +1,1190 @@
+"""EEVEE multi-view rendering utilities (pip ``bpy`` module).
+
+Shared by the :mod:`data_process.motion_rendering` stage: render-engine
+init, scene reset, asset import, material fixes, studio lighting, scene
+normalization, spherical camera placement, per-action multi-view frame
+rendering and T-pose 2x2 grids.
+
+These helpers target the pip ``bpy`` module driven by plain ``python``
+(EEVEE needs the module's GPU context; ``blender -b`` has no display
+surface), unlike :mod:`data_process.utils.blender_export`, which also runs
+under Blender's bundled interpreter.
+"""
+
+import fcntl
+import os
+import json
+import math
+import re
+import tempfile
+from contextlib import contextmanager
+from typing import List, Literal, Optional, Tuple
+
+import bpy
+import imageio
+import numpy as np
+from mathutils import Euler, Matrix, Vector
+from PIL import Image
+from loguru import logger
+
+
+# Frame rate for import, preview video and the per-view metadata JSON. Must
+# match the export stage's --fps (also 30) โ€” see load_and_prep_asset.
+RENDER_FPS = 30
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# GPU serialization
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+# Several EEVEE contexts rendering on one device can wedge the NVIDIA driver
+# (Xid 109 CTX SWITCH TIMEOUT, then Xid 31 MMU fault): every worker spins at
+# 100% CPU in os_acquire_rwlock and not a single frame is ever written. An
+# exclusive lock around the render call keeps GPU work single-file while
+# asset import and video encoding still overlap across workers. The lock is
+# node-local and per-GPU; set RENDER_GPU_LOCK=0 to disable it.
+
+_gpu_lock_handle = None
+
+
+def _gpu_lock_path() -> str:
+ job = os.environ.get("SLURM_JOB_ID", "local")
+ gpu = (os.environ.get("CUDA_VISIBLE_DEVICES", "0").split(",")[0] or "0")
+ name = f"unimate_render_{job}_gpu{gpu}.lock"
+ # Node-local by preference: the contended resource is one node's GPU, and
+ # TMPDIR often points at shared scratch, where the lock would be slower
+ # and โ€” for a multi-node job โ€” serialize workers that share no device.
+ for directory in ("/tmp", tempfile.gettempdir()):
+ if os.path.isdir(directory) and os.access(directory, os.W_OK):
+ return os.path.join(directory, name)
+ return os.path.join(tempfile.gettempdir(), name)
+
+
+@contextmanager
+def gpu_render_lock():
+ """Serialize GPU rendering across worker processes sharing one GPU."""
+ global _gpu_lock_handle
+ if os.environ.get("RENDER_GPU_LOCK", "1") == "0":
+ yield
+ return
+ if _gpu_lock_handle is None:
+ _gpu_lock_handle = open(_gpu_lock_path(), "w")
+ fcntl.flock(_gpu_lock_handle, fcntl.LOCK_EX)
+ try:
+ yield
+ finally:
+ fcntl.flock(_gpu_lock_handle, fcntl.LOCK_UN)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Render engine
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def init_render_engine(render_samples: int = 64):
+ """Initialize EEVEE with the given sample count. Call once per process."""
+ bpy.context.scene.render.engine = "BLENDER_EEVEE"
+ eevee = bpy.context.scene.eevee
+
+ # Anti-aliasing
+ eevee.taa_render_samples = render_samples
+
+ # Ambient occlusion
+ eevee.use_gtao = True
+ eevee.gtao_distance = 1.0
+ eevee.gtao_quality = 0.5
+
+ # Screen-space reflections (half-res for efficiency at 512px)
+ eevee.use_ssr = True
+ eevee.use_ssr_refraction = True
+ eevee.use_ssr_halfres = True
+ eevee.ssr_quality = 0.5
+ eevee.ssr_max_roughness = 0.5
+ eevee.ssr_thickness = 0.2
+
+ # Shadows
+ eevee.use_soft_shadows = True
+ eevee.shadow_cascade_size = "2048"
+ eevee.shadow_cube_size = "1024"
+
+ eevee.use_bloom = True
+
+ bpy.context.scene.render.use_high_quality_normals = True
+ bpy.context.scene.render.use_persistent_data = True
+
+ # Color management โ€” Filmic handles HDR values from PBR materials
+ # properly (Standard clips > 1.0, causing dark metallic surfaces)
+ bpy.context.scene.view_settings.view_transform = "Filmic"
+ bpy.context.scene.view_settings.look = "Medium Contrast"
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Scene reset
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def clear_scene():
+ """Remove all objects, compositor nodes, actions; purge orphan data.
+
+ Removing objects leaves their mesh/material/image datablocks behind
+ with zero users; without the purge they accumulate across assets and
+ grow memory unboundedly over long in-process batches.
+ """
+ for obj in list(bpy.data.objects):
+ bpy.data.objects.remove(obj, do_unlink=True)
+
+ bpy.context.scene.use_nodes = True
+ for node in list(bpy.context.scene.node_tree.nodes):
+ bpy.context.scene.node_tree.nodes.remove(node)
+
+ bpy.context.scene.frame_start = 0
+ bpy.context.scene.frame_end = 0
+ for action in list(bpy.data.actions):
+ bpy.data.actions.remove(action)
+
+ bpy.data.orphans_purge(do_recursive=True)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Asset import
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+_IMPORT_OPS = {
+ ".fbx": lambda p: bpy.ops.import_scene.fbx(filepath=p, use_anim=True),
+ ".glb": lambda p: bpy.ops.import_scene.gltf(filepath=p),
+ ".gltf": lambda p: bpy.ops.import_scene.gltf(filepath=p),
+}
+
+RENDERABLE_EXTS = frozenset(_IMPORT_OPS)
+
+
+def load_file(path: str):
+ """Import a rigged asset (.fbx, .glb, .gltf)."""
+ ext = os.path.splitext(path)[1].lower()
+ op = _IMPORT_OPS.get(ext)
+ if op is None:
+ raise RuntimeError(f"Unsupported file format: {path}")
+ op(path)
+
+
+def load_armature(path: str):
+ """Import a file and return its main newly added armature (or None).
+
+ Picks the armature with the most bones, matching the export stage's
+ ``choose_main_armature`` โ€” on multi-armature assets both stages must
+ agree, or the render animates a different skeleton than the NPZ.
+ """
+ existing = set(bpy.data.objects)
+ load_file(path)
+ armatures = [
+ obj for obj in bpy.data.objects
+ if obj not in existing and obj.type == "ARMATURE"
+ ]
+ if not armatures:
+ return None
+ return max(armatures, key=lambda a: len(a.data.bones))
+
+
+def find_meshes():
+ return [o for o in bpy.data.objects if o.type == "MESH"]
+
+
+def get_scene_armatures():
+ """Return all ARMATURE objects in the current scene."""
+ return [o for o in bpy.context.scene.objects if o.type == "ARMATURE"]
+
+
+def get_root_objects():
+ """Yield all top-level (parentless) objects in the current scene."""
+ for obj in bpy.context.scene.objects.values():
+ if not obj.parent:
+ yield obj
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Material fixes (post-import)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def fix_materials():
+ """Fix common GLTF/FBX material issues after import.
+
+ 1. Clamps Metallic to <= 0.9 so surfaces don't render black when
+ film_transparent=True (pure mirrors reflect the empty background).
+ 2. Connects vertex color attributes when Base Color is at a default
+ value (white or black), indicating the importer left it unset.
+ 3. Connects disconnected Image Texture nodes found in the shader tree.
+ """
+ # Pass 1: clamp metallic on all Principled BSDF materials
+ for material in bpy.data.materials:
+ if not material.use_nodes:
+ continue
+ for node in material.node_tree.nodes:
+ if node.type != 'BSDF_PRINCIPLED':
+ continue
+ metallic_input = node.inputs.get('Metallic')
+ if metallic_input is None:
+ continue
+ # Only fix unlinked (constant) metallic values
+ if not metallic_input.is_linked and metallic_input.default_value > 0.9:
+ logger.info(f"Clamping Metallic {metallic_input.default_value:.2f} -> 0.9 "
+ f"on material '{material.name}'")
+ metallic_input.default_value = 0.9
+
+ # Pass 2: fix unlinked Base Color defaults per mesh
+ for obj in find_meshes():
+ mesh = obj.data
+
+ # Check if mesh has color attributes (vertex colors)
+ color_attr_name = None
+ if hasattr(mesh, 'color_attributes') and len(mesh.color_attributes) > 0:
+ color_attr_name = mesh.color_attributes[0].name
+ elif hasattr(mesh, 'vertex_colors') and len(mesh.vertex_colors) > 0:
+ color_attr_name = mesh.vertex_colors[0].name
+
+ for slot in obj.material_slots:
+ mat = slot.material
+ if mat is None or not mat.use_nodes:
+ continue
+
+ tree = mat.node_tree
+ nodes = tree.nodes
+ links = tree.links
+
+ principled = None
+ for node in nodes:
+ if node.type == 'BSDF_PRINCIPLED':
+ principled = node
+ break
+ if principled is None:
+ continue
+
+ base_color_input = principled.inputs.get('Base Color')
+ if base_color_input is None or base_color_input.is_linked:
+ continue
+
+ existing_color = list(base_color_input.default_value)
+ is_default_white = all(c > 0.95 for c in existing_color[:3])
+ is_default_black = all(c < 0.05 for c in existing_color[:3])
+ if not (is_default_white or is_default_black):
+ continue
+
+ # Priority 1: connect vertex colors if available
+ if color_attr_name is not None:
+ vc_node = nodes.new(type='ShaderNodeVertexColor')
+ vc_node.layer_name = color_attr_name
+ links.new(vc_node.outputs['Color'], base_color_input)
+ logger.info(f"Connected vertex colors '{color_attr_name}' to "
+ f"material '{mat.name}' on '{obj.name}'")
+ continue
+
+ # Priority 2: find disconnected Image Texture nodes in the tree
+ img_tex = None
+ for node in nodes:
+ if node.type == 'TEX_IMAGE' and node.image is not None:
+ if not any(l.from_node == node and l.from_socket == node.outputs['Color']
+ for l in links):
+ img_tex = node
+ break
+ if img_tex is not None:
+ links.new(img_tex.outputs['Color'], base_color_input)
+ logger.info(f"Connected orphan image texture '{img_tex.image.name}' to "
+ f"material '{mat.name}' on '{obj.name}'")
+
+
+def clear_normal_maps():
+ """Remove normal map connections from all Principled BSDF materials.
+
+ Some GLTF normal maps cause rendering artifacts; removing them
+ produces cleaner results for motion-capture rendering.
+ """
+ for material in bpy.data.materials:
+ if not material.use_nodes:
+ continue
+ tree = material.node_tree
+ for node in tree.nodes:
+ if node.type == 'BSDF_PRINCIPLED':
+ normal_input = node.inputs.get("Normal")
+ if normal_input and normal_input.is_linked:
+ for link in list(normal_input.links):
+ tree.links.remove(link)
+
+
+def set_materials_opaque():
+ """Force materials to OPAQUE blend mode, keeping real alpha as HASHED.
+
+ Prevents transparency artifacts that cause parts of meshes to
+ disappear or render incorrectly in multi-view rendering. Materials
+ whose Principled Alpha input is actually driven (linked, or set below
+ 1.0) genuinely need transparency โ€” e.g. Mixamo facial-feature decal
+ planes layered over the face, whose atlas would otherwise cover it as
+ an opaque black quad. Those get HASHED, which EEVEE resolves without
+ the depth-sorting artifacts of BLEND.
+ """
+ for material in bpy.data.materials:
+ if not material.use_nodes:
+ continue
+ has_alpha = False
+ for node in material.node_tree.nodes:
+ if node.type == 'BSDF_PRINCIPLED':
+ alpha_input = node.inputs.get('Alpha')
+ if alpha_input is not None and (
+ alpha_input.is_linked or alpha_input.default_value < 1.0):
+ has_alpha = True
+ break
+ material.blend_method = "HASHED" if has_alpha else "OPAQUE"
+ material.show_transparent_back = False
+
+
+def smooth_meshes():
+ """Enable auto-smooth on all meshes (30-degree threshold)."""
+ for obj in find_meshes():
+ obj.data.use_auto_smooth = True
+ obj.data.auto_smooth_angle = np.deg2rad(30)
+
+
+def keep_main_mesh_only(main_mesh):
+ """Remove every mesh except *main_mesh*.
+
+ Dropping stray helper geometry (icospheres, prop meshes, low-vertex
+ duplicates) keeps the render clean and the scene bbox tight.
+ """
+ for m in find_meshes():
+ if m is not main_mesh:
+ bpy.data.objects.remove(m, do_unlink=True)
+
+
+def keep_armature_meshes(armature):
+ """Remove meshes that do not belong to *armature*.
+
+ A mesh belongs to the rig when it is skinned to it (Armature
+ modifier) or parented under it (bone-parented props, meshes grouped
+ beneath the armature node). Stray helper geometry (icospheres, prop
+ meshes of another rig) matches neither โ€” so this drops the junk
+ without amputating multi-mesh characters the way keeping the single
+ largest mesh would. Falls back to the largest mesh when nothing
+ belongs to the rig.
+ """
+ def belongs(mesh):
+ if any(mod.type == 'ARMATURE' and mod.object == armature
+ for mod in mesh.modifiers):
+ return True
+ parent = mesh.parent
+ while parent is not None:
+ if parent == armature:
+ return True
+ parent = parent.parent
+ return False
+
+ meshes = find_meshes()
+ keep = [m for m in meshes if belongs(m)]
+ if keep:
+ for m in meshes:
+ if m not in keep:
+ bpy.data.objects.remove(m, do_unlink=True)
+ else:
+ keep_main_mesh_only(max(meshes, key=lambda m: len(m.data.vertices)))
+
+
+def load_and_prep_asset(path: str, keep_only_main_mesh: bool = True,
+ fps: int = RENDER_FPS):
+ """Clear the scene, import an asset, and fix it up for rendering.
+
+ Sets up lighting, optionally drops meshes that do not belong to the
+ main armature (stray helper geometry that would pollute the render
+ and the scene bbox โ€” multi-mesh characters keep every skinned or
+ bone-parented part), then applies material fixes and auto-smooth.
+
+ The scene frame rate is set *before* the import, matching the export
+ stage (``blender_export.load_scene``). glTF stores keyframe times in
+ seconds, so the importer resamples them at the current scene fps โ€”
+ leaving it at Blender's default 24 makes every GLB render 0.8x as many
+ frames as the 30 fps NPZ the same asset exports to, which shifts the
+ frame window and makes the previews play 25% fast. (The FBX importer
+ takes the rate from the file, so only glTF was affected.)
+
+ Args:
+ path: Asset file (.fbx / .glb / .gltf).
+ keep_only_main_mesh: Drop meshes that are neither skinned to nor
+ parented under the main armature (falling back to the
+ highest-vertex mesh if nothing belongs to it). Disable to
+ keep every mesh regardless.
+ fps: Scene frame rate to import and render at. Must match the
+ export stage's ``--fps`` or renders and NPZs desynchronize.
+
+ Returns:
+ The imported armature, or None if the file has no armature or
+ no mesh.
+ """
+ clear_scene()
+ bpy.context.scene.render.fps = int(fps)
+ setup_lighting()
+
+ armature = load_armature(path)
+ if armature is None:
+ return None
+
+ # Drop lights bundled in the asset โ€” they stack on the studio lighting
+ # and make per-asset exposure inconsistent. Cameras go too: add_camera
+ # only creates one when scene.camera is unset, so an asset camera would
+ # otherwise take over the render and follow the asset's own camera path.
+ studio_lights = {"Key_Light", "Fill_Light", "Rim_Light", "Bottom_Light"}
+ for obj in list(bpy.data.objects):
+ if obj.type == "LIGHT" and obj.name not in studio_lights:
+ bpy.data.objects.remove(obj, do_unlink=True)
+ elif obj.type == "CAMERA":
+ bpy.data.objects.remove(obj, do_unlink=True)
+
+ meshes = find_meshes()
+ if not meshes:
+ return None
+ if keep_only_main_mesh:
+ keep_armature_meshes(armature)
+
+ fix_materials()
+ clear_normal_maps()
+ set_materials_opaque()
+ smooth_meshes()
+ return armature
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Lighting
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _create_light(
+ name: str,
+ light_type: Literal["POINT", "SUN", "SPOT", "AREA"],
+ rotation: Tuple[float, float, float],
+ energy: float,
+ use_shadow: bool = False,
+ specular_factor: float = 1.0,
+ angle: float = 0.0,
+):
+ """Create and link a light object to the scene."""
+ light_data = bpy.data.lights.new(name=name, type=light_type)
+ light_object = bpy.data.objects.new(name, light_data)
+ bpy.context.collection.objects.link(light_object)
+ light_object.rotation_euler = rotation
+ light_data.use_shadow = use_shadow
+ light_data.specular_factor = specular_factor
+ light_data.energy = energy
+ if light_type == "SUN" and angle > 0:
+ light_data.angle = angle # angular diameter โ†’ soft shadow edges
+ return light_object
+
+
+def _setup_world_environment(strength: float = 0.7):
+ """Soft top-to-bottom gradient world for even ambient illumination."""
+ world = bpy.data.worlds.get("World")
+ if world is None:
+ world = bpy.data.worlds.new("World")
+ bpy.context.scene.world = world
+ world.use_nodes = True
+ nodes = world.node_tree.nodes
+ links = world.node_tree.links
+ nodes.clear()
+
+ bg_node = nodes.new(type="ShaderNodeBackground")
+ bg_node.inputs["Strength"].default_value = strength
+
+ output_node = nodes.new(type="ShaderNodeOutputWorld")
+
+ tex_coord = nodes.new(type="ShaderNodeTexCoord")
+ mapping = nodes.new(type="ShaderNodeMapping")
+ gradient = nodes.new(type="ShaderNodeTexGradient")
+ gradient.gradient_type = "LINEAR"
+
+ color_ramp = nodes.new(type="ShaderNodeValToRGB")
+ color_ramp.color_ramp.elements[0].position = 0.3
+ color_ramp.color_ramp.elements[0].color = (0.9, 0.9, 0.9, 1.0)
+ color_ramp.color_ramp.elements[1].position = 0.7
+ color_ramp.color_ramp.elements[1].color = (1.0, 1.0, 1.0, 1.0)
+
+ links.new(tex_coord.outputs["Generated"], mapping.inputs["Vector"])
+ links.new(mapping.outputs["Vector"], gradient.inputs["Vector"])
+ links.new(gradient.outputs["Color"], color_ramp.inputs["Fac"])
+ links.new(color_ramp.outputs["Color"], bg_node.inputs["Color"])
+ links.new(bg_node.outputs["Background"], output_node.inputs["Surface"])
+
+
+def setup_lighting():
+ """Studio-style lighting for multi-view rendering.
+
+ Three-point neutral sun lights plus a soft ambient world fill so no
+ view is completely dark:
+
+ - Key light: upper-front-left, shadow enabled for depth
+ - Fill light: upper-front-right, softer, no shadow
+ - Rim light: upper-rear, edge separation
+ - Bottom fill: subtle upward fill to prevent harsh under-shadows
+ """
+ # Clear existing lights
+ for obj in list(bpy.data.objects):
+ if obj.type == "LIGHT":
+ bpy.data.objects.remove(obj, do_unlink=True)
+
+ _setup_world_environment()
+
+ _create_light("Key_Light", "SUN",
+ rotation=(0.785, 0, -0.785), # 45ยฐ elevation, 45ยฐ left
+ energy=1.5, use_shadow=True, angle=0.05)
+ _create_light("Fill_Light", "SUN",
+ rotation=(0.785, 0, 2.356), # 45ยฐ elevation, 135ยฐ right
+ energy=0.7, specular_factor=0.5)
+ _create_light("Rim_Light", "SUN",
+ rotation=(-0.524, 0, -3.927), # 30ยฐ above rear
+ energy=1.0, specular_factor=0.8)
+ _create_light("Bottom_Light", "SUN",
+ rotation=(2.618, 0, 0), # pointing upward from below
+ energy=0.3, specular_factor=0.3)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Bounding box & scene normalization
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def get_scene_bbox(single_obj=None, ignore_matrix=False):
+ """Axis-aligned bounding box of all meshes (or a single object)."""
+ bbox_min = (math.inf,) * 3
+ bbox_max = (-math.inf,) * 3
+
+ meshes = find_meshes() if single_obj is None else [single_obj]
+ if not meshes:
+ raise RuntimeError("No objects in scene to compute bounding box for")
+
+ for obj in meshes:
+ for coord in obj.bound_box:
+ coord = Vector(coord)
+ if not ignore_matrix:
+ coord = obj.matrix_world @ coord
+ bbox_min = tuple(min(a, b) for a, b in zip(bbox_min, coord))
+ bbox_max = tuple(max(a, b) for a, b in zip(bbox_max, coord))
+
+ return Vector(bbox_min), Vector(bbox_max)
+
+
+def get_scene_bbox_all_frames():
+ """Bounding box that encloses the entire animation sequence."""
+ bbox_min = (math.inf,) * 3
+ bbox_max = (-math.inf,) * 3
+
+ saved_frame = bpy.context.scene.frame_current
+ for frame in range(bpy.context.scene.frame_start, bpy.context.scene.frame_end + 1):
+ bpy.context.scene.frame_set(frame)
+ fmin, fmax = get_scene_bbox()
+ bbox_min = tuple(min(a, b) for a, b in zip(bbox_min, fmin))
+ bbox_max = tuple(max(a, b) for a, b in zip(bbox_max, fmax))
+
+ bpy.context.scene.frame_set(saved_frame)
+ return Vector(bbox_min), Vector(bbox_max)
+
+
+def normalize_scene(scene_scale: float = 1.0, process_frames: bool = False):
+ """Scale and center the scene to fit within *scene_scale* at the origin.
+
+ Parents all root objects under a single "NormalizationNode" empty and
+ applies scale/offset there. Safe to call multiple times โ€” resets the
+ node to identity first, so repeated calls don't compound.
+
+ Args:
+ scene_scale: Target bounding box size.
+ process_frames: If True, the bbox covers the whole animation range.
+ """
+ node = bpy.data.objects.get("NormalizationNode")
+ if node is not None:
+ node.location = (0, 0, 0)
+ node.rotation_euler = (0, 0, 0)
+ node.scale = (1, 1, 1)
+ bpy.context.view_layer.update()
+
+ if process_frames:
+ bbox_min, bbox_max = get_scene_bbox_all_frames()
+ else:
+ bbox_min, bbox_max = get_scene_bbox()
+
+ scale = scene_scale / max(bbox_max - bbox_min)
+ offset = -(bbox_min + bbox_max) / 2
+
+ if node is None:
+ node = bpy.data.objects.new("NormalizationNode", None)
+ bpy.context.scene.collection.objects.link(node)
+ for obj in get_root_objects():
+ if obj is not node and obj.type != "CAMERA":
+ obj.parent = node
+ obj.matrix_parent_inverse = node.matrix_world.inverted()
+ node.scale = (scale, scale, scale)
+ node.location = offset * scale
+ bpy.context.view_layer.update()
+
+ bpy.ops.object.select_all(action="DESELECT")
+ return scale, offset
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Camera
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def build_transformation_mat(translation, rotation) -> np.ndarray:
+ """Build a 4x4 transform from a (3,) translation and a (3,3) rotation or (3,) Euler."""
+ translation = np.array(translation)
+ rotation = np.array(rotation)
+
+ mat = np.eye(4)
+ if translation.shape[0] != 3:
+ raise RuntimeError(
+ f"Translation has invalid shape: {translation.shape}. Expected (3,).")
+ mat[:3, 3] = translation
+
+ if rotation.shape == (3, 3):
+ mat[:3, :3] = rotation
+ elif rotation.shape[0] == 3:
+ mat[:3, :3] = np.array(Euler(rotation).to_matrix())
+ else:
+ raise RuntimeError(
+ f"Rotation has invalid shape: {rotation.shape}. Expected (3,3) or (3,).")
+
+ return mat
+
+
+def camera_pose_on_sphere(
+ azimuth_deg: float,
+ elevation_deg: float,
+ radius: float,
+ center: Tuple[float, float, float] = (0, 0, 0),
+) -> np.ndarray:
+ """Cam-to-world 4x4 matrix for a camera on a sphere looking at *center*."""
+ elev = math.radians(elevation_deg)
+ azim = math.radians(azimuth_deg)
+ phi = 0.5 * math.pi - elev
+
+ x = center[0] + radius * math.sin(phi) * math.cos(azim)
+ y = center[1] + radius * math.sin(phi) * math.sin(azim)
+ z = center[2] + radius * math.cos(phi)
+ cam_pos = Vector((x, y, z))
+
+ look_dir = Vector(center) - cam_pos
+ rot_euler = look_dir.to_track_quat("-Z", "Y").to_euler()
+ return build_transformation_mat(cam_pos, rot_euler)
+
+
+def fit_camera_distance(camera_dist: float, scene_scale: float = 1.0,
+ camera_sensor_width: int = 32, camera_lens: int = 35,
+ margin: float = 1.05) -> float:
+ """Raise *camera_dist* if it would crop the normalized scene bbox.
+
+ ``normalize_scene`` fits the subject into a cube of side *scene_scale*,
+ whose worst case in frame is a near-face corner at depth
+ ``d - scene_scale/2`` and lateral offset ``scene_scale/2``. With the
+ fixed 35mm lens on a 32mm sensor the half-FOV is only atan(16/35) =
+ 24.6 degrees, so the default distance of 1.5 is marginally too close and
+ clips the occasional wide subject (measured: ~0.3% of rendered frames,
+ e.g. a crab's claw).
+
+ Returns the larger of *camera_dist* and the distance that fits, so
+ subjects that already frame correctly keep their exact previous framing
+ and only the ones that would crop get pushed back.
+ """
+ half_fov = math.atan((camera_sensor_width / 2.0) / camera_lens)
+ half_extent = scene_scale / 2.0
+ required = half_extent + (half_extent / math.tan(half_fov)) * margin
+ if required > camera_dist:
+ logger.info(f"Camera distance {camera_dist:.3f} would crop a "
+ f"{scene_scale:.3f} bbox; using {required:.3f}")
+ return required
+ return camera_dist
+
+
+def add_camera(
+ cam2world_matrix,
+ camera_sensor_width: int = 32,
+ camera_lens: int = 35,
+):
+ """Set (or create) the scene camera from a cam-to-world matrix.
+
+ Keyframes the pose at the current frame_end so per-view placement
+ survives animation rendering (constant extrapolation holds it fixed).
+ """
+ if not isinstance(cam2world_matrix, Matrix):
+ cam2world_matrix = Matrix(cam2world_matrix)
+
+ if bpy.context.scene.camera is None:
+ # camera_add leaves scene.camera unset, so the new camera has to be
+ # picked up explicitly. Selecting "the last CAMERA in bpy.data.objects"
+ # instead would hand the render to a camera bundled with the asset
+ # (objects iterate by name), which then inherits its parent's animated
+ # transform and flies along the asset's own camera path.
+ before = set(bpy.data.objects)
+ bpy.ops.object.camera_add(location=(0, 0, 0))
+ new_cams = [o for o in bpy.data.objects
+ if o not in before and o.type == "CAMERA"]
+ if not new_cams:
+ raise RuntimeError("camera_add did not create a camera object")
+ bpy.context.scene.camera = new_cams[0]
+
+ cam = bpy.context.scene.camera
+ cam.data.type = "PERSP"
+ cam.data.sensor_width = camera_sensor_width
+ cam.data.lens = camera_lens
+ cam.matrix_world = cam2world_matrix
+
+ frame = bpy.context.scene.frame_end
+ cam.keyframe_insert(data_path="location", frame=frame)
+ cam.keyframe_insert(data_path="rotation_euler", frame=frame)
+ cam.data.keyframe_insert(data_path="lens", frame=frame)
+ cam.data.keyframe_insert(data_path="sensor_width", frame=frame)
+
+ return cam
+
+
+def remove_camera():
+ """Delete the active scene camera (clears stale keyframes between actions)."""
+ cam = bpy.context.scene.camera
+ if cam is not None:
+ bpy.data.objects.remove(cam, do_unlink=True)
+ bpy.context.scene.camera = None
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Render output
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def enable_color_output(
+ resolution: int,
+ output_dir: str,
+ file_prefix: str = "",
+ mode: Literal["IMAGE", "VIDEO"] = "IMAGE",
+ film_transparent: bool = True,
+ fps: int = RENDER_FPS,
+):
+ """Configure color render output (square resolution).
+
+ Args:
+ resolution: Render resolution in pixels.
+ output_dir: Directory for output files.
+ file_prefix: Filename prefix (IMAGE) or MP4 stem (VIDEO).
+ mode: 'IMAGE' for per-frame PNGs, 'VIDEO' for an MP4.
+ film_transparent: Transparent background (IMAGE mode).
+ fps: Video frame rate (VIDEO mode).
+ """
+ scene = bpy.context.scene
+ scene.render.resolution_x = resolution
+ scene.render.resolution_y = resolution
+ scene.render.resolution_percentage = 100
+ scene.render.film_transparent = film_transparent
+ scene.render.image_settings.quality = 100
+
+ if mode == "IMAGE":
+ scene.render.image_settings.file_format = "PNG"
+ scene.render.image_settings.color_mode = "RGBA"
+ scene.render.filepath = os.path.join(output_dir, file_prefix)
+ elif mode == "VIDEO":
+ scene.render.image_settings.file_format = "FFMPEG"
+ scene.render.ffmpeg.format = "MPEG4"
+ scene.render.ffmpeg.codec = "H264"
+ scene.render.image_settings.color_mode = "RGB"
+ scene.render.fps = fps
+ scene.render.filepath = os.path.join(output_dir, file_prefix + ".mp4")
+
+
+def render_animation():
+ """Render the scene's frame range with compositing enabled."""
+ bpy.context.scene.render.use_compositing = True
+ bpy.context.scene.use_nodes = True
+
+ tree = bpy.context.scene.node_tree
+ if "Render Layers" not in tree.nodes:
+ tree.nodes.new("CompositorNodeRLayers")
+
+ with gpu_render_lock():
+ bpy.ops.render.render(animation=True, write_still=True)
+
+
+def render_still(filepath: str):
+ """Render the current frame to *filepath* with compositing enabled."""
+ bpy.context.scene.render.use_compositing = True
+ bpy.context.scene.use_nodes = True
+
+ tree = bpy.context.scene.node_tree
+ if "Render Layers" not in tree.nodes:
+ tree.nodes.new("CompositorNodeRLayers")
+
+ bpy.context.scene.render.filepath = filepath
+ with gpu_render_lock():
+ bpy.ops.render.render(write_still=True)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Multi-view rendering
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+VIEW_NAMES = ["front", "back", "left", "right"]
+VIEW_AZIMUTHS = [-90, 90, 180, 0] # matches VIEW_NAMES order
+NUM_VIEWS = len(VIEW_NAMES)
+MAX_RENDER_FRAMES = 200
+
+
+VIDEO_BACKGROUND = (235, 235, 235)
+
+
+def _reset_frames_dir(frames_dir: str):
+ """Create *frames_dir*, removing any PNGs left by a previous render.
+
+ Blender overwrites ``{frame:04d}.png`` but never deletes frames outside
+ the new range. Re-rendering a clip with a shorter range would otherwise
+ leave the old tail in place โ€” and since ``is_render_complete`` only
+ checks that the four views hold the same non-zero count (all four are
+ stale by the same amount), the clip still reports complete while every
+ ``v00x.mp4`` ends with frames from the previous animation.
+ """
+ if os.path.isdir(frames_dir):
+ for fname in os.listdir(frames_dir):
+ if fname.endswith(".png"):
+ os.remove(os.path.join(frames_dir, fname))
+ else:
+ os.makedirs(frames_dir, exist_ok=True)
+
+
+def compose_frames_to_video(frames_dir: str, out_path: str, fps: int = RENDER_FPS,
+ background=VIDEO_BACKGROUND):
+ """Compose a view's rendered PNG frames into an H.264 MP4.
+
+ Reuses the already-rendered frames (no extra Blender render pass).
+ Transparent-film PNGs are flattened onto *background* โ€” H.264 carries
+ no alpha channel.
+ """
+ frames = sorted(f for f in os.listdir(frames_dir) if f.endswith(".png"))
+ if not frames:
+ logger.warning(f"No frames in {frames_dir}; skipping video compose.")
+ return
+ writer = imageio.get_writer(out_path, fps=fps)
+ try:
+ for fname in frames:
+ img = Image.open(os.path.join(frames_dir, fname))
+ if img.mode in ("RGBA", "LA", "P"):
+ img = img.convert("RGBA")
+ bg = Image.new("RGB", img.size, background)
+ bg.paste(img, mask=img.split()[-1])
+ img = bg
+ else:
+ img = img.convert("RGB")
+ writer.append_data(np.asarray(img))
+ finally:
+ writer.close()
+ logger.info(f"Composed video: {out_path} ({len(frames)} frames @ {fps} fps)")
+
+
+def _skip_marker_path(output_dir: str, name: str) -> str:
+ return os.path.join(output_dir, ".skipped", name)
+
+
+def is_asset_render_skipped(output_dir: str, name: str) -> bool:
+ """True when *name* was marked as producing no renders (see below)."""
+ return os.path.isfile(_skip_marker_path(output_dir, name))
+
+
+def mark_asset_render_skipped(output_dir: str, name: str, reason: str = ""):
+ """Record that *name* legitimately produces no renders (no armature /
+ mesh / pose actions), so reruns don't reload the asset just to rediscover
+ that. Delete the marker (or ``.skipped/``) to retry the asset."""
+ path = _skip_marker_path(output_dir, name)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ with open(path, "w") as f:
+ f.write(reason + "\n")
+
+
+def _rendered_marker_path(output_dir: str, name: str) -> str:
+ return os.path.join(output_dir, ".rendered", f"{name}.json")
+
+
+def mark_asset_render_done(output_dir: str, name: str, clip_dirs):
+ """Record the full set of clip directories *name* is expected to produce.
+
+ ``--missing-only`` cannot otherwise tell "this asset rendered all five of
+ its actions" from "a worker was killed after three of them" โ€” both leave
+ only complete directories on disk, and without the expected set the
+ partially-rendered asset is treated as done and its remaining actions are
+ never rendered. Written only after every action of the asset has been
+ handled.
+ """
+ path = _rendered_marker_path(output_dir, name)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ tmp = f"{path}.tmp.{os.getpid()}"
+ with open(tmp, "w") as f:
+ json.dump({"clip_dirs": sorted(clip_dirs)}, f)
+ os.replace(tmp, path)
+
+
+def get_expected_clip_dirs(output_dir: str, name: str):
+ """Expected clip directory names for *name*, or None when unrecorded."""
+ path = _rendered_marker_path(output_dir, name)
+ if not os.path.isfile(path):
+ return None
+ try:
+ with open(path) as f:
+ return json.load(f).get("clip_dirs")
+ except (OSError, ValueError):
+ return None
+
+
+def is_render_complete(save_root: str, num_views: int = NUM_VIEWS,
+ require_video: bool = False) -> bool:
+ """Check if all expected multi-view outputs exist for one clip.
+
+ Requires *num_views* view directories (v000..) with the matching camera
+ metadata JSONs (v000.json..), each holding the same non-zero number of
+ PNGs โ€” plus the composed v000.mp4.. when *require_video* is set. Views
+ render sequentially, so an interrupted render always leaves either a
+ missing view directory or a view with fewer frames than the first โ€”
+ both fail the equal-count check.
+ """
+ if not os.path.isdir(save_root):
+ return False
+ frame_counts = []
+ for view_idx in range(num_views):
+ view_name = f"v{view_idx:03d}"
+ if not os.path.isfile(os.path.join(save_root, f"{view_name}.json")):
+ return False
+ if require_video and not os.path.isfile(os.path.join(save_root, f"{view_name}.mp4")):
+ return False
+ view_dir = os.path.join(save_root, view_name)
+ if not os.path.isdir(view_dir):
+ return False
+ frame_counts.append(sum(1 for f in os.listdir(view_dir) if f.endswith(".png")))
+ return frame_counts[0] > 0 and len(set(frame_counts)) == 1
+
+
+def render_action_multiview(
+ armature,
+ action_name: str,
+ start_frame: int,
+ end_frame: int,
+ save_root: str,
+ camera_dist: float = 1.5,
+ scene_scale: float = 1.0,
+ resolution: int = 512,
+ views: Optional[List[str]] = None,
+ render_video: bool = False,
+ render_frames: bool = True,
+ compose_video: bool = True,
+ fps: int = RENDER_FPS,
+ max_render_frames: int = MAX_RENDER_FRAMES,
+ save_camera_metadata: bool = True,
+):
+ """Render one action from multiple views.
+
+ Output structure::
+
+ save_root/
+ v000/ 0001.png ... (front)
+ v001/ ... (back)
+ v002/ ... (left)
+ v003/ ... (right)
+ v000.mp4 ... (if render_video=True)
+ v000.json ... (camera metadata per view)
+
+ Args:
+ armature: Blender armature object to animate.
+ action_name: Name of the action in bpy.data.actions.
+ start_frame / end_frame: Frame range of the action.
+ save_root: Output directory for this clip's renders.
+ camera_dist: Camera distance from origin.
+ scene_scale: Target bounding box size for normalization.
+ resolution: Render resolution (square).
+ views: Subset of VIEW_NAMES to render (default: all four).
+ render_video: Also render an MP4 per view via Blender (second full
+ render pass; prefer *compose_video*).
+ render_frames: Render per-frame PNGs per view.
+ compose_video: Compose the rendered PNGs into v00x.mp4 per view
+ (no extra render pass).
+ fps: Video frame rate.
+ max_render_frames: Clamp on the number of rendered frames.
+ save_camera_metadata: Write a JSON per view with camera pose.
+ """
+ if views is None:
+ views = list(VIEW_NAMES)
+ view_azimuths = [VIEW_AZIMUTHS[VIEW_NAMES.index(v)] for v in views]
+
+ # Clamp frame range
+ if end_frame - start_frame + 1 > max_render_frames:
+ end_frame = start_frame + max_render_frames - 1
+
+ # Ensure armatures are in POSE mode (may be stuck in REST from a
+ # preceding T-pose render)
+ for arm in get_scene_armatures():
+ arm.data.pose_position = "POSE"
+
+ # Bind the action
+ if armature.animation_data:
+ armature.animation_data_clear()
+ armature.animation_data_create()
+ action = bpy.data.actions[action_name]
+ armature.animation_data.action = action
+
+ # A discovered action can belong to another armature in the file; the
+ # render then silently shows a static pose. Warn so it's diagnosable.
+ target_bones = {
+ m.group(1)
+ for fc in action.fcurves
+ for m in [re.match(r'pose\.bones\["(.+?)"\]', fc.data_path)]
+ if m
+ }
+ if target_bones and not (target_bones & set(armature.pose.bones.keys())):
+ logger.warning(f"Action '{action_name}' animates none of armature "
+ f"'{armature.name}'s bones; render will be static.")
+
+ # Set the frame range and flush the depsgraph so the mesh deforms
+ # before normalization measures it
+ scene = bpy.context.scene
+ scene.frame_start = start_frame
+ scene.frame_end = end_frame
+ scene.frame_set(start_frame)
+ bpy.context.view_layer.update()
+
+ # Reset camera so the previous action's keyframes don't carry over
+ remove_camera()
+
+ # Normalize per action (accounts for the full motion extent). The
+ # returned transform maps original asset coordinates into the rendered
+ # (normalized) space: p_rendered = norm_scale * (p_original + norm_offset).
+ norm_scale, norm_offset = normalize_scene(scene_scale, process_frames=True)
+ camera_dist = fit_camera_distance(camera_dist, scene_scale)
+
+ os.makedirs(save_root, exist_ok=True)
+
+ for view_idx, azimuth in enumerate(view_azimuths):
+ cam_mat = camera_pose_on_sphere(azimuth, 0.0, camera_dist)
+ camera = add_camera(cam_mat)
+ view_name = f"v{view_idx:03d}"
+
+ if save_camera_metadata:
+ meta = {
+ # view identity
+ "view": views[view_idx],
+ "azimuth": math.radians(azimuth),
+ "elevation": 0.0,
+ # camera intrinsics
+ "resolution": resolution,
+ "camera_angle_x": camera.data.angle_x,
+ # camera extrinsics (cam-to-world, in normalized scene space)
+ "transform_matrix": cam_mat.tolist(),
+ # scene normalization: p_rendered = scene_scale * (p_original + scene_offset)
+ "scene_scale": norm_scale,
+ "scene_offset": list(norm_offset),
+ # rendered clip timing (after the max_render_frames clamp)
+ "frame_start": start_frame,
+ "frame_end": end_frame,
+ "fps": fps,
+ }
+ with open(os.path.join(save_root, f"{view_name}.json"), "w") as f:
+ json.dump(meta, f, indent=2)
+ if render_video:
+ enable_color_output(resolution, save_root, file_prefix=view_name,
+ mode="VIDEO", fps=fps)
+ render_animation()
+
+ if render_frames:
+ frames_dir = os.path.join(save_root, view_name)
+ _reset_frames_dir(frames_dir)
+ enable_color_output(resolution, frames_dir, mode="IMAGE")
+ render_animation()
+ if compose_video:
+ compose_frames_to_video(
+ frames_dir, os.path.join(save_root, f"{view_name}.mp4"), fps=fps)
+
+ logger.info(f"Rendered view '{views[view_idx]}' -> {save_root}/{view_name}")
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# T-pose grid rendering
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def compose_grid_2x2(image_paths: List[str], output_path: str):
+ """Arrange 4 images into a 2x2 grid and save the result.
+
+ Layout::
+
+ [front] [back]
+ [left] [right]
+ """
+ assert len(image_paths) == 4, f"Expected 4 images, got {len(image_paths)}"
+ images = [Image.open(p) for p in image_paths]
+ w, h = images[0].size
+
+ grid = Image.new("RGBA", (w * 2, h * 2))
+ grid.paste(images[0], (0, 0))
+ grid.paste(images[1], (w, 0))
+ grid.paste(images[2], (0, h))
+ grid.paste(images[3], (w, h))
+ # Written atomically: the T-pose resume check is a bare os.path.isfile, so
+ # a grid truncated by a kill mid-write would be treated as complete and
+ # never regenerated.
+ tmp_path = f"{output_path}.tmp.{os.getpid()}.png"
+ grid.save(tmp_path)
+ os.replace(tmp_path, output_path)
+ for img in images:
+ img.close()
+
+
+def render_tpose_grid(
+ save_path: str,
+ camera_dist: float = 1.5,
+ scene_scale: float = 1.0,
+ resolution: int = 512,
+):
+ """Render the rest pose from 4 views and save a 2x2 grid PNG.
+
+ Forces every armature into ``pose_position='REST'`` and clears bound
+ actions so the evaluated mesh reflects the true rest pose; the bbox
+ used for normalization is measured after that, so the T-pose is
+ centered and scaled independently of any animation state. The caller
+ is expected to clear the scene afterwards (state is not restored).
+ """
+ scene = bpy.context.scene
+
+ # Force rest pose and drop bound actions
+ for arm in get_scene_armatures():
+ if arm.animation_data and arm.animation_data.action is not None:
+ arm.animation_data.action = None
+ arm.data.pose_position = "REST"
+ arm.data.update_tag()
+
+ # Flush the depsgraph on a single frame so bbox reads rest-pose coords
+ frame = scene.frame_current
+ scene.frame_start = frame
+ scene.frame_end = frame
+ scene.frame_set(frame)
+ bpy.context.view_layer.update()
+ bpy.context.evaluated_depsgraph_get()
+
+ remove_camera()
+ normalize_scene(scene_scale, process_frames=False)
+
+ scene.render.resolution_x = resolution
+ scene.render.resolution_y = resolution
+ scene.render.resolution_percentage = 100
+ scene.render.image_settings.file_format = "PNG"
+ scene.render.image_settings.color_mode = "RGBA"
+ scene.render.film_transparent = True
+
+ tmp_dir = tempfile.mkdtemp(prefix="tpose_")
+ image_paths = []
+ try:
+ for view_name, azimuth in zip(VIEW_NAMES, VIEW_AZIMUTHS):
+ add_camera(camera_pose_on_sphere(
+ azimuth, 0.0, fit_camera_distance(camera_dist, scene_scale)))
+ out_file = os.path.join(tmp_dir, f"{view_name}.png")
+ scene.frame_set(frame)
+ render_still(out_file)
+ image_paths.append(out_file)
+
+ os.makedirs(os.path.dirname(save_path) or ".", exist_ok=True)
+ compose_grid_2x2(image_paths, save_path)
+ logger.info(f"Saved T-pose grid: {save_path}")
+ finally:
+ for p in image_paths:
+ try:
+ os.remove(p)
+ except OSError:
+ pass
+ try:
+ os.rmdir(tmp_dir)
+ except OSError:
+ pass
diff --git a/data_process/utils/blender_rig.py b/data_process/utils/blender_rig.py
new file mode 100644
index 0000000..8a3ff4a
--- /dev/null
+++ b/data_process/utils/blender_rig.py
@@ -0,0 +1,673 @@
+"""Blender utilities for driving a rigged character with an NPZ animation.
+
+Organized as five sections:
+
+1. Scene / object helpers โ€” import meshes, select objects, pack textures.
+2. Keyframe computation โ€” per-bone loc/rot keyframes from 4x4 local
+ transforms.
+3. Action reconstruction โ€” turn per-bone keyframe arrays into a Blender
+ Action bound to the armature.
+4. Armature reconciliation โ€” handle armature bones that are absent from
+ the driving animation (keep them, merge their weights, or delete them).
+5. Character export โ€” export the animated mesh + armature to GLB/FBX.
+
+The counterpart for the *export* direction (Blender scene โ†’ NPZ) lives in
+:mod:`data_process.utils.blender_export`.
+"""
+
+import os
+import re
+
+import bmesh
+import bpy
+import numpy as np
+from loguru import logger
+from mathutils import Matrix, Quaternion
+
+from data_process.utils.blender_export import (
+ bind_action,
+ clear_animation_state,
+ reset_scene,
+)
+
+RECONSTRUCTED_ACTION_NAME = "Reconstructed_Action"
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# 1. Scene / object helpers
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def update_scene():
+ """Force a full scene dependency-graph update."""
+ bpy.context.view_layer.update()
+ bpy.context.scene.update_tag()
+ for obj in bpy.context.scene.objects:
+ obj.update_tag()
+
+
+def deselect():
+ """Deselect every object in the scene."""
+ bpy.ops.object.select_all(action="DESELECT")
+
+
+def select_objs(obj_list=None, deselect_first=False):
+ """Select Blender objects, optionally deselecting everything first."""
+ if not obj_list:
+ obj_list = bpy.context.scene.objects
+ if deselect_first:
+ deselect()
+ for obj in obj_list:
+ obj.select_set(True)
+
+
+def load_file(filepath, *args, **kwargs):
+ """Import a 3D file and return the list of newly-added scene objects."""
+ old_objs = set(bpy.context.scene.objects)
+ if filepath.endswith(".glb"):
+ bpy.ops.import_scene.gltf(filepath=filepath, *args, **kwargs)
+ elif filepath.endswith(".fbx"):
+ bpy.ops.import_scene.fbx(filepath=filepath, *args, **kwargs)
+ elif filepath.endswith(".obj"):
+ bpy.ops.wm.obj_import(filepath=filepath, *args, **kwargs)
+ elif filepath.endswith(".ply"):
+ bpy.ops.wm.ply_import(filepath=filepath, *args, **kwargs)
+ else:
+ raise RuntimeError(f"Invalid input file: {filepath}")
+ imported = sorted(set(bpy.context.scene.objects) - old_objs, key=lambda x: x.name)
+ logger.info(f"Imported: {[obj.name for obj in imported]}")
+ return imported
+
+
+_IMAGE_FORMAT_EXTENSIONS = {
+ 'PNG': '.png', 'JPEG': '.jpg', 'JPEG2000': '.jp2', 'TARGA': '.tga',
+ 'TARGA_RAW': '.tga', 'BMP': '.bmp', 'TIFF': '.tif', 'OPEN_EXR': '.exr',
+ 'HDR': '.hdr', 'WEBP': '.webp',
+}
+
+
+def repair_and_pack_textures(source_path):
+ """Re-link missing texture files and pack every image into memory.
+
+ Imported assets frequently reference textures by paths from the
+ original author's machine (absolute Windows paths are common in
+ Truebones FBX). When the importer can't locate a file, the image
+ datablock has no pixel data, and both the GLB and FBX exporters
+ silently drop it. This helper:
+
+ 1. searches for missing image files next to *source_path*, and
+ 2. packs every loadable image into the .blend so exports no
+ longer depend on external files.
+
+ Returns the list of image names that still have no usable data
+ (their texture files could not be found on disk).
+ """
+ src_dir = os.path.dirname(os.path.abspath(source_path))
+ try:
+ bpy.ops.file.find_missing_files(directory=src_dir)
+ except RuntimeError as exc:
+ logger.warning(f"find_missing_files failed: {exc}")
+
+ broken = []
+ for img in bpy.data.images:
+ if img.source not in ('FILE', 'SEQUENCE', 'TILED'):
+ continue
+ if img.packed_file is None:
+ try:
+ img.pack()
+ except RuntimeError as exc:
+ logger.warning(f"Could not pack image '{img.name}': {exc}")
+ if img.packed_file is None:
+ filepath = bpy.path.abspath(img.filepath) if img.filepath else ''
+ if not (filepath and os.path.isfile(filepath)):
+ broken.append(img.name)
+
+ if broken:
+ logger.warning(
+ f"{len(broken)} image(s) have no pixel data and will be missing "
+ f"from exports (texture files not found near {src_dir}): {broken}"
+ )
+ return broken
+
+
+def write_packed_images_to_dir(texture_dir):
+ """Write packed images that have no on-disk file out to *texture_dir*.
+
+ The FBX exporter's COPY/embed path reads texture bytes from the
+ image's file path on disk; images that exist only as packed data
+ (typical after a GLB import) are silently skipped, losing textures.
+ Writing them out right before FBX export makes embedding work.
+
+ Returns the list of file paths written.
+ """
+ written = []
+ for idx, img in enumerate(bpy.data.images):
+ if img.packed_file is None:
+ continue
+ filepath = bpy.path.abspath(img.filepath) if img.filepath else ''
+ if filepath and os.path.isfile(filepath):
+ continue
+ ext = _IMAGE_FORMAT_EXTENSIONS.get(img.file_format, '.png')
+ safe_name = re.sub(r'[^A-Za-z0-9_.-]+', '_', img.name).strip('_') or f'image_{idx}'
+ if not safe_name.lower().endswith(ext):
+ safe_name += ext
+ os.makedirs(texture_dir, exist_ok=True)
+ out_path = os.path.join(texture_dir, safe_name)
+ img.filepath_raw = out_path
+ try:
+ # Dump the packed bytes verbatim (no re-encode) exactly where
+ # filepath_raw points, so the exporter's embed path finds them.
+ # (img.unpack() writes to the image's *original* path instead.)
+ with open(out_path, 'wb') as fh:
+ fh.write(img.packed_file.data)
+ written.append(out_path)
+ except OSError as exc:
+ logger.warning(f"Could not write packed image '{img.name}' to disk: {exc}")
+ if written:
+ logger.info(f"Wrote {len(written)} packed image(s) to {texture_dir} for FBX embedding")
+ return written
+
+
+def _find_upstream_image_node(socket, max_depth=8):
+ """BFS upstream from *socket* to the first Image Texture node with an image."""
+ queue = [(socket, 0)]
+ visited = set()
+ while queue:
+ sock, depth = queue.pop(0)
+ if depth > max_depth:
+ continue
+ for link in sock.links:
+ node = link.from_node
+ if node in visited:
+ continue
+ visited.add(node)
+ if node.type == 'TEX_IMAGE' and node.image is not None:
+ return node
+ queue.extend((inp, depth + 1) for inp in node.inputs)
+ return None
+
+
+def simplify_materials_for_fbx(objs):
+ """Rewire each material's base-color texture directly into a Principled BSDF.
+
+ The FBX exporter only recognizes a texture when its Image Texture node
+ feeds a Principled BSDF input directly. glTF-imported materials often
+ interpose extra nodes โ€” a Mix/Multiply for baseColorFactor or vertex
+ colors, or (for KHR_materials_unlit) an Emission chain with no
+ Principled node at all โ€” which makes the FBX exporter silently drop
+ the texture. This flattens those graphs: find the first image texture
+ upstream and connect it straight to Base Color, creating a Principled
+ BSDF if the material lacks one.
+ """
+ mats = {slot.material for obj in objs if obj.type == 'MESH'
+ for slot in obj.material_slots if slot.material}
+ for mat in mats:
+ if not mat.use_nodes:
+ continue
+ tree = mat.node_tree
+ principled = next((n for n in tree.nodes if n.type == 'BSDF_PRINCIPLED'), None)
+
+ if principled is not None:
+ base = principled.inputs['Base Color']
+ if any(l.from_node.type == 'TEX_IMAGE' and l.from_node.image
+ for l in base.links):
+ continue # already directly wired; FBX export handles it
+ img_node = _find_upstream_image_node(base)
+ if img_node is None:
+ continue
+ for l in list(base.links):
+ tree.links.remove(l)
+ tree.links.new(img_node.outputs['Color'], base)
+ logger.info(f"Material '{mat.name}': rewired image "
+ f"'{img_node.image.name}' directly to Base Color for FBX")
+ continue
+
+ # No Principled BSDF (e.g. unlit glTF material): build one from the
+ # image feeding the output surface, so FBX gets a textured material.
+ output = next((n for n in tree.nodes
+ if n.type == 'OUTPUT_MATERIAL' and n.is_active_output),
+ None) or next((n for n in tree.nodes
+ if n.type == 'OUTPUT_MATERIAL'), None)
+ if output is None:
+ continue
+ img_node = _find_upstream_image_node(output.inputs['Surface'])
+ if img_node is None:
+ continue
+ principled = tree.nodes.new('ShaderNodeBsdfPrincipled')
+ tree.links.new(img_node.outputs['Color'], principled.inputs['Base Color'])
+ for l in list(output.inputs['Surface'].links):
+ tree.links.remove(l)
+ tree.links.new(principled.outputs['BSDF'], output.inputs['Surface'])
+ logger.info(f"Material '{mat.name}': no Principled BSDF; created one "
+ f"with image '{img_node.image.name}' for FBX")
+
+
+def get_armature_obj(obj_list=None):
+ """Return the first ARMATURE object from a list (or the scene), else None."""
+ if not obj_list:
+ obj_list = bpy.context.scene.objects
+ for obj in obj_list:
+ if obj.type == "ARMATURE":
+ return obj
+ return None
+
+
+def set_scene_timing(nframes, fps):
+ """Set the scene frame rate and frame range ``[0, nframes - 1]``."""
+ bpy.context.scene.render.fps = int(fps)
+ bpy.context.scene.frame_start = 0
+ bpy.context.scene.frame_end = nframes - 1
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# 2. Keyframe computation
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def compute_bone_keyframes(rest_local_mat, anim_local_mat, bone_names,
+ tpos_global_rot=None):
+ """Compute per-bone location/rotation keyframes from local 4x4 transforms.
+
+ The relative transform ``rest_inv @ anim`` is computed in NumPy for all
+ frames at once; per-keyframe loc/rot are then extracted via ``mathutils``
+ so they match Blender's conventions exactly.
+
+ Args:
+ rest_local_mat: ``(nbones, 4, 4)`` rest-pose local transforms.
+ anim_local_mat: ``(nframes, nbones, 4, 4)`` animated local transforms.
+ bone_names: Bone names, length ``nbones``.
+ tpos_global_rot: Optional ``(nbones, 4)`` T-pose global rotations.
+ When given, each keyframe is conjugated by the bone's T-pose
+ global rotation.
+
+ Returns:
+ ``{bone_name: {'location': [(frame, Vector), ...],
+ 'rotation': [(frame, Quaternion), ...]}}``
+ """
+ nframes, nbones = anim_local_mat.shape[:2]
+
+ rest_inv = np.linalg.inv(rest_local_mat) # (nbones, 4, 4)
+ rel_mats = rest_inv[None] @ anim_local_mat # (nframes, nbones, 4, 4)
+
+ tpos_quats = tpos_quats_inv = None
+ if tpos_global_rot is not None:
+ tpos_quats = [Quaternion(tpos_global_rot[b]) for b in range(nbones)]
+ tpos_quats_inv = [q.inverted() for q in tpos_quats]
+
+ anim_data_dict = {}
+ for idxb in range(nbones):
+ locs, rots = [], []
+ for idxf in range(nframes):
+ m = Matrix(rel_mats[idxf, idxb].tolist())
+ loc = m.to_translation()
+ rot = m.to_quaternion()
+ if tpos_quats is not None:
+ rot = tpos_quats_inv[idxb] @ rot @ tpos_quats[idxb]
+ loc = tpos_quats_inv[idxb] @ loc
+ locs.append((idxf, loc))
+ rots.append((idxf, rot))
+ anim_data_dict[str(bone_names[idxb])] = {'location': locs, 'rotation': rots}
+
+ return anim_data_dict
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# 3. Action reconstruction
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _add_fcurves(action, bone_name, attr, n_components, keyframes):
+ """Create per-channel fcurves for one pose-bone attribute."""
+ data_path = f'pose.bones["{bone_name}"].{attr}'
+ for i in range(n_components):
+ fc = action.fcurves.new(data_path=data_path, index=i)
+ fc.keyframe_points.add(len(keyframes))
+ for idx, (frame, val) in enumerate(keyframes):
+ kp = fc.keyframe_points[idx]
+ kp.co = (frame, val[i])
+ kp.interpolation = 'LINEAR'
+
+
+def rebuild_action_from_data(armature, anim_data_dict):
+ """Replace the armature's animation with a new Action built from keyframes.
+
+ ``anim_data_dict`` has the format returned by
+ :func:`compute_bone_keyframes`. Any existing action / NLA tracks on the
+ armature are cleared first; bones missing from the armature are skipped
+ with a warning.
+ """
+ clear_animation_state(armature)
+ new_action = bpy.data.actions.new(name=RECONSTRUCTED_ACTION_NAME)
+ bind_action(armature, new_action)
+
+ valid_bones = {bone.name for bone in armature.data.bones}
+ for bone_name, data in anim_data_dict.items():
+ if bone_name not in valid_bones:
+ logger.warning(f"Skipping keyframes for '{bone_name}': not in armature")
+ continue
+ _add_fcurves(new_action, bone_name, 'location', 3, data['location'])
+ _add_fcurves(new_action, bone_name, 'rotation_quaternion', 4, data['rotation'])
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# 4. Armature reconciliation: handle bones not present in the animation
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+EXTRA_BONES_STRATEGIES = ('keep', 'merge', 'remove')
+
+
+def find_skinned_meshes(armature):
+ """Return all mesh objects whose Armature modifier targets *armature*."""
+ meshes = []
+ for obj in bpy.context.scene.objects:
+ if obj.type != 'MESH':
+ continue
+ for mod in obj.modifiers:
+ if mod.type == 'ARMATURE' and mod.object == armature:
+ meshes.append(obj)
+ break
+ return meshes
+
+
+def build_ancestor_map(armature, kept_bone_set):
+ """Map each armature bone NOT in *kept_bone_set* to its nearest kept ancestor.
+
+ Returns ``{dropped_name: ancestor_name_or_None}``. ``None`` indicates
+ the chain to the root is entirely dropped (orphan bone).
+ """
+ mapping = {}
+ for bone in armature.data.bones:
+ if bone.name in kept_bone_set:
+ continue
+ parent = bone.parent
+ while parent is not None and parent.name not in kept_bone_set:
+ parent = parent.parent
+ mapping[bone.name] = parent.name if parent is not None else None
+ return mapping
+
+
+def transfer_vertex_group(mesh_obj, src_name, dst_name):
+ """Merge weights of vertex group *src_name* into *dst_name* and delete src.
+
+ If *dst_name* is ``None``, the src group is just removed. Existing dst
+ weights are preserved and incremented (ADD semantics).
+ """
+ vgs = mesh_obj.vertex_groups
+ src = vgs.get(src_name)
+ if src is None:
+ return
+ if dst_name is None:
+ vgs.remove(src)
+ return
+
+ dst = vgs.get(dst_name) or vgs.new(name=dst_name)
+ src_idx = src.index
+
+ # Snapshot (vert_idx, weight) pairs before mutating the groups.
+ transfers = []
+ for vert in mesh_obj.data.vertices:
+ for grp in vert.groups:
+ if grp.group == src_idx:
+ if grp.weight > 0.0:
+ transfers.append((vert.index, grp.weight))
+ break # a vertex appears at most once per group
+
+ for v_idx, weight in transfers:
+ dst.add([v_idx], weight, 'ADD')
+ vgs.remove(src)
+
+
+def merge_extra_bone_weights(armature, kept_bone_set):
+ """Transfer weights of dropped bones to their nearest kept ancestor.
+
+ Returns the ancestor map used (for logging).
+ """
+ ancestor_map = build_ancestor_map(armature, kept_bone_set)
+ if not ancestor_map:
+ return ancestor_map
+ meshes = find_skinned_meshes(armature)
+ if not meshes:
+ logger.warning("No skinned meshes found targeting this armature; "
+ "weight transfer is a no-op")
+ for src_name, dst_name in ancestor_map.items():
+ for mesh_obj in meshes:
+ transfer_vertex_group(mesh_obj, src_name, dst_name)
+ return ancestor_map
+
+
+def delete_dominant_weighted_vertices(mesh_obj, dropped_bone_names):
+ """Delete vertices whose dominant (max-weight) group is in *dropped_bone_names*.
+
+ Vertices with mixed influences whose primary bone survives are kept,
+ so we don't punch holes near boundary regions (e.g. wrist verts that
+ have minor finger-bone influences but are mostly weighted to the
+ forearm).
+
+ Returns the number of vertices deleted from *mesh_obj*.
+ """
+ vgs = mesh_obj.vertex_groups
+ dropped_idxs = {vgs[name].index for name in dropped_bone_names if name in vgs}
+ if not dropped_idxs:
+ return 0
+
+ verts_to_delete = []
+ for vert in mesh_obj.data.vertices:
+ if not vert.groups:
+ continue
+ max_weight = -1.0
+ max_group = -1
+ for g in vert.groups:
+ if g.weight > max_weight:
+ max_weight = g.weight
+ max_group = g.group
+ if max_group in dropped_idxs:
+ verts_to_delete.append(vert.index)
+
+ if not verts_to_delete:
+ return 0
+
+ bm = bmesh.new()
+ bm.from_mesh(mesh_obj.data)
+ bm.verts.ensure_lookup_table()
+ bmesh.ops.delete(
+ bm,
+ geom=[bm.verts[i] for i in verts_to_delete],
+ context='VERTS',
+ )
+ bm.to_mesh(mesh_obj.data)
+ bm.free()
+ mesh_obj.data.update()
+ return len(verts_to_delete)
+
+
+def delete_skin_vertices_for_bones(armature, dropped_bone_names):
+ """Delete dominantly-weighted vertices for *dropped_bone_names* across all meshes.
+
+ Returns ``{mesh_name: n_deleted}`` for logging.
+ """
+ if not dropped_bone_names:
+ return {}
+ meshes = find_skinned_meshes(armature)
+ if not meshes:
+ logger.warning("No skinned meshes found targeting this armature; "
+ "vertex deletion is a no-op")
+ return {}
+ deleted = {}
+ for mesh_obj in meshes:
+ n = delete_dominant_weighted_vertices(mesh_obj, dropped_bone_names)
+ if n:
+ deleted[mesh_obj.name] = n
+ return deleted
+
+
+def remove_armature_bones(armature, bone_names):
+ """Delete *bone_names* from *armature* in EDIT mode, keeping the
+ hierarchy connected.
+
+ Blender does NOT re-parent children when an edit bone is removed โ€”
+ they silently become parentless roots. So before deleting, every
+ surviving bone whose parent chain passes through a dropped bone is
+ re-parented to its nearest surviving ancestor (the same bone its
+ vertex weights are merged into under the 'merge' strategy). Edit-bone
+ head/tail are stored in armature space, so re-parenting with
+ ``use_connect=False`` preserves each bone's rest transform exactly.
+ """
+ bpy.context.view_layer.objects.active = armature
+ bpy.ops.object.mode_set(mode='EDIT')
+ edit_bones = armature.data.edit_bones
+ dropped = {name for name in bone_names if name in edit_bones}
+
+ for bone in edit_bones:
+ if bone.name in dropped:
+ continue
+ if bone.parent is None or bone.parent.name not in dropped:
+ continue
+ ancestor = bone.parent
+ while ancestor is not None and ancestor.name in dropped:
+ ancestor = ancestor.parent
+ bone.use_connect = False
+ bone.parent = ancestor
+ if ancestor is None:
+ logger.warning(
+ f"Bone '{bone.name}': entire parent chain was dropped; "
+ f"it becomes a root bone"
+ )
+ else:
+ logger.info(f"Re-parented '{bone.name}' to surviving ancestor "
+ f"'{ancestor.name}'")
+
+ for bone_name in dropped:
+ edit_bones.remove(edit_bones[bone_name])
+ bpy.ops.object.mode_set(mode='OBJECT')
+
+
+def sync_armature_bones(armature, anim_bone_names, extra_bones_strategy='merge'):
+ """Reconcile armature bones with the animation bone list.
+
+ *extra_bones_strategy* controls what happens to armature bones that are
+ NOT present in *anim_bone_names*:
+
+ - ``'merge'`` (default): transfer each extra bone's vertex weights to
+ its nearest kept ancestor on every skinned mesh, then delete the
+ bone. Mesh stays attached; articulation in the merged region is lost.
+ - ``'remove'``: delete the bones AND every mesh vertex whose dominant
+ (max-weight) bone is one of them. Vertices with mixed influences
+ whose primary bone survives are kept, so we don't punch holes near
+ boundary regions.
+ - ``'keep'``: leave the bones in place with no keyframes. They sit at
+ identity local pose and inherit their parent's animated transform
+ through Blender's hierarchical pose evaluation.
+
+ Animation bones absent from the armature are always logged and skipped
+ by :func:`rebuild_action_from_data`.
+ """
+ if extra_bones_strategy not in EXTRA_BONES_STRATEGIES:
+ raise ValueError(
+ f"extra_bones_strategy must be one of {EXTRA_BONES_STRATEGIES}, "
+ f"got {extra_bones_strategy!r}"
+ )
+
+ armature_bone_set = {bone.name for bone in armature.data.bones}
+ anim_bone_set = {str(name) for name in anim_bone_names}
+
+ extra = armature_bone_set - anim_bone_set
+ if extra:
+ if extra_bones_strategy == 'keep':
+ logger.info(
+ f"Keeping {len(extra)} armature bones not in animation "
+ f"(no keyframes; will follow parent pose): {sorted(extra)}"
+ )
+ elif extra_bones_strategy == 'merge':
+ ancestor_map = merge_extra_bone_weights(armature, anim_bone_set)
+ summary = ', '.join(
+ f"{src}->{dst or '<orphan>'}"
+ for src, dst in sorted(ancestor_map.items())
+ )
+ logger.warning(
+ f"Merged {len(extra)} extra armature bones into nearest kept "
+ f"ancestor and removed them: {summary}"
+ )
+ remove_armature_bones(armature, extra)
+ else: # 'remove'
+ deleted = delete_skin_vertices_for_bones(armature, extra)
+ if deleted:
+ total = sum(deleted.values())
+ summary = ', '.join(f"{name}: {n}" for name, n in sorted(deleted.items()))
+ logger.warning(
+ f"Deleted {total} mesh vertices dominantly weighted to "
+ f"dropped bones ({summary})"
+ )
+ logger.warning(f"Removing armature bones not in animation: {sorted(extra)}")
+ remove_armature_bones(armature, extra)
+
+ missing = anim_bone_set - armature_bone_set
+ if missing:
+ logger.warning(f"Animation bones not in armature (will be skipped): {missing}")
+
+ update_scene()
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# 5. Character export
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def export_animated_character(output_path_no_ext, formats=('glb',)):
+ """Export every MESH + ARMATURE object to ``<output_path_no_ext>.<fmt>``.
+
+ Stale actions other than the reconstructed one are removed first so the
+ exporters only bake the driving animation.
+ """
+ export_objs = [obj for obj in bpy.context.scene.objects
+ if obj.type in ('MESH', 'ARMATURE')]
+ select_objs(export_objs, deselect_first=True)
+
+ for action in list(bpy.data.actions):
+ if action.name != RECONSTRUCTED_ACTION_NAME:
+ bpy.data.actions.remove(action)
+ # Imported characters can carry object-level animation data (static
+ # transform keys); the glTF exporter samples it into a stray extra
+ # animation, so drop everything but the reconstructed armature action.
+ for obj in export_objs:
+ anim_data = obj.animation_data
+ if anim_data is not None and (
+ anim_data.action is None
+ or anim_data.action.name != RECONSTRUCTED_ACTION_NAME):
+ obj.animation_data_clear()
+
+ for fmt in formats:
+ export_selected_to_file(f"{output_path_no_ext}.{fmt}", fmt)
+
+
+def export_selected_to_file(filepath, char_anim_type='glb'):
+ """Export currently-selected scene objects to GLB or FBX."""
+ if char_anim_type == 'glb':
+ bpy.ops.export_scene.gltf(
+ filepath=filepath,
+ check_existing=False,
+ use_selection=True,
+ export_format='GLB',
+ # Export the action's own F-curves rather than resampling every
+ # object: sampling also bakes the armature *object's* static
+ # transform into a stray second glTF animation.
+ export_force_sampling=False,
+ )
+ elif char_anim_type == 'fbx':
+ # The FBX exporter drops textures that aren't wired directly into a
+ # Principled BSDF (Mix nodes, unlit materials); flatten those first.
+ simplify_materials_for_fbx(bpy.context.selected_objects)
+ # Sidecar copies for FBX consumers that ignore embedded media;
+ # embedding itself reads packed data or these files.
+ write_packed_images_to_dir(
+ os.path.join(os.path.dirname(os.path.abspath(filepath)), 'textures'))
+ bpy.ops.export_scene.fbx(
+ filepath=filepath,
+ check_existing=False,
+ use_selection=True,
+ use_triangles=True,
+ add_leaf_bones=False,
+ bake_anim=True,
+ path_mode="COPY",
+ embed_textures=True,
+ )
+ else:
+ raise RuntimeError(f"Invalid char_anim_type: {char_anim_type}")
+ logger.info(f"Saved animated character: {filepath}")
+
+
diff --git a/data_process/utils/export_markers.py b/data_process/utils/export_markers.py
new file mode 100644
index 0000000..bf9bc1f
--- /dev/null
+++ b/data_process/utils/export_markers.py
@@ -0,0 +1,106 @@
+"""Per-asset export completion markers (resume bookkeeping).
+
+Skeleton pruning is joint across all of an asset's clips, so an asset only
+completes atomically. A marker under ``<output_dir>/.completed/`` records
+that โ€” including assets that legitimately produce no clips
+(``status="skipped"``), so reruns don't reload them. Delete a marker (or the
+directory) to force a re-export, e.g. after changing joint-count limits.
+
+Each marker also carries the asset's pruned joint names, which makes the
+canonical ``joint_names.json`` rebuildable from disk at any time. That
+matters because markers are durable and written per asset, while the
+in-memory joint-name dict is only flushed at the very end of a run: without
+the marker copy, a worker killed by a Slurm walltime or an OOM leaves
+markers that suppress re-export but no joint names, losing those entries
+permanently.
+
+Deliberately free of any ``bpy`` import so plain-Python tools (notably
+:mod:`data_process.tools.merge_summaries`) can use it outside Blender.
+"""
+
+import json
+import os
+
+
+def _asset_marker_path(output_dir, save_name):
+ return os.path.join(output_dir, ".completed", f"{save_name}.json")
+
+
+def is_asset_complete(output_dir, save_name):
+ """True when a completion marker exists for *save_name*."""
+ return os.path.isfile(_asset_marker_path(output_dir, save_name))
+
+
+def mark_asset_complete(output_dir, save_name, status="exported", n_clips=None,
+ reason="", joint_names=None):
+ """Write the completion marker for one fully-processed asset.
+
+ Written atomically โ€” a half-written marker would otherwise be
+ indistinguishable from a complete one on the next run.
+ """
+ path = _asset_marker_path(output_dir, save_name)
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ payload = {"status": status}
+ if n_clips is not None:
+ payload["n_clips"] = int(n_clips)
+ if reason:
+ payload["reason"] = reason
+ if joint_names is not None:
+ payload["joint_names"] = [str(n) for n in joint_names]
+ tmp = f"{path}.tmp.{os.getpid()}"
+ with open(tmp, "w") as f:
+ json.dump(payload, f)
+ os.replace(tmp, path)
+
+
+def collect_joint_names_from_markers(output_dir):
+ """Rebuild ``{save_name: joint_names}`` from the per-asset markers.
+
+ Only markers that actually recorded joint names contribute; assets that
+ produced no clips (status ``"skipped"``) never do.
+ """
+ marker_dir = os.path.join(output_dir, ".completed")
+ if not os.path.isdir(marker_dir):
+ return {}
+
+ recovered = {}
+ for fname in sorted(os.listdir(marker_dir)):
+ if not fname.endswith(".json"):
+ continue
+ try:
+ with open(os.path.join(marker_dir, fname)) as f:
+ payload = json.load(f)
+ except (OSError, ValueError):
+ continue
+ names = payload.get("joint_names")
+ if names:
+ recovered[os.path.splitext(fname)[0]] = names
+ return recovered
+
+
+def merge_joint_names(output_dir, all_joint_names, log=None):
+ """Union this run's joint names over the markers and the existing file.
+
+ Precedence (lowest to highest): the canonical ``joint_names.json`` on
+ disk, the per-asset completion markers, then *all_joint_names* from the
+ current run.
+ """
+ merged = {}
+
+ canonical = os.path.join(output_dir, "joint_names.json")
+ if os.path.isfile(canonical):
+ try:
+ with open(canonical) as f:
+ merged.update(json.load(f))
+ except (OSError, ValueError) as exc:
+ if log:
+ log(f"Could not read {canonical} ({exc}); rebuilding from markers only.")
+
+ from_markers = collect_joint_names_from_markers(output_dir)
+ new_from_markers = len(set(from_markers) - set(merged))
+ merged.update(from_markers)
+ if new_from_markers and log:
+ log(f"Recovered {new_from_markers} joint-name entries from completion markers")
+
+ merged.update(all_joint_names)
+ return merged
diff --git a/data_process/utils/motion_features.py b/data_process/utils/motion_features.py
new file mode 100644
index 0000000..36ece22
--- /dev/null
+++ b/data_process/utils/motion_features.py
@@ -0,0 +1,803 @@
+"""Per-object motion feature extraction (export NPZ โ†’ canonical clips).
+
+Pipeline:
+ 1. ``process_tpose`` โ€” load / canonicalize the T-pose, compute
+ scale + rest offsets.
+ 2. ``build_topology_cond`` โ€” build the per-object topology condition
+ dict (joint graph, spectral feats, โ€ฆ).
+ 3. ``load_and_canonicalize_motion`` + ``trim_static_ends``
+ โ€” load / normalize each motion, strip
+ static pre/post-roll.
+ 4. ``extract_clip_feats`` + ``save_clip``
+ โ€” slice into clips, dump per-frame arrays
+ (positions / rotations / facing) and an
+ MP4 visualization.
+
+``process_object`` orchestrates 1โ€“4 for one object type and returns an
+``ObjectResult`` (see below) summarizing the saved artifacts.
+
+Output layout under ``save_dir``:
+ motions/{object}-{motion}-{clip_idx}.npz # per-clip per-frame arrays
+ videos/{object}-{motion}-{clip_idx}.mp4
+ tpose/{object}.png # T-pose still
+ (driver-level) cond.npy, captions.json, filtered_clips.json, metadata.txt
+
+The driver is :mod:`data_process.feature_extraction.extract_features`.
+"""
+
+import os
+from os.path import join as pjoin
+from pathlib import Path
+
+import numpy as np
+from loguru import logger
+
+from Animation import (
+ Animation,
+ positions_global,
+ rotations_global,
+ offsets_from_positions
+)
+from Quaternions import Quaternions
+
+from data_process.utils.topology import (
+ compute_edge_relations_and_distances,
+ compute_kinematic_chains,
+ compute_joint_depths,
+ compute_edge_indexs,
+ compute_laplacian_eigenvectors,
+)
+from data_process.utils.skeleton import (
+ reorder_anim_bfs,
+ get_root_facing_quat,
+ canonicalize_anim,
+ get_motion_activity_metrics,
+ get_skeleton_diameter,
+)
+from data_process.utils.plotting import (
+ save_skeleton_motion,
+ save_skeleton_motion_ground,
+ save_skeleton_motion_spectral,
+ save_skeleton_tpose,
+ save_skeleton_tpos_spectral,
+)
+from data_process.feature_extraction.metadata import (
+ atomic_output_path,
+ get_motion_clip_windows,
+)
+
+
+# Smallest leaf-to-leaf diameter a skeleton may have and still be scaled to
+# the canonical target diameter (below this the scale factor blows up / divides
+# by zero).
+MIN_SKELETON_DIAMETER = 1e-8
+
+
+class DegenerateSkeletonError(ValueError):
+ """Raised when a T-pose cannot be canonicalized (e.g. zero-length bones).
+
+ Callers treat this as a skip-with-reason for the whole object type rather
+ than a hard failure.
+ """
+
+
+# ===========================================================================
+# Joint-name resolution
+# ===========================================================================
+
+def resolve_face_joint_idxs(face_joints, names):
+ """Resolve face joint names to indices from a face_joints dict.
+
+ Returns the sentinel ``[-1, -1]`` when the entry is missing, empty, or
+ its names don't appear in the skeleton โ€” downstream code treats that as
+ identity facing rather than skipping the object.
+
+ Returns:
+ Tuple of (face_joint_idxs, body_axis).
+ """
+ if not face_joints or 'r_hip' not in face_joints or 'l_hip' not in face_joints:
+ return [-1, -1], False
+ r_hip_name = face_joints['r_hip']['raw']
+ l_hip_name = face_joints['l_hip']['raw']
+ if r_hip_name in names and l_hip_name in names:
+ idxs = [names.index(r_hip_name), names.index(l_hip_name)]
+ return idxs, face_joints.get('body_axis', False)
+ return [-1, -1], False
+
+
+def resolve_joint_names(object_type, tpos_data, corps_names=None,
+ clean_names=None, expected_names=None):
+ """Resolve joint names and corps subset indices for an object type.
+
+ Returns:
+ Tuple of (corps_idxs, corps_names, clean_names_list, raw_parents,
+ raw_njoints).
+ """
+ raw_parents = tpos_data['parents'].tolist()
+ raw_njoints = len(raw_parents)
+ raw_names = tpos_data['names'].tolist()
+
+ # clean_names comes from clean_joint_names.json, whose entries are
+ # positionally aligned with joint_names.json โ€” NOT necessarily with this
+ # NPZ: rig variants can enumerate the same skeleton in a different order
+ # (e.g. Truebones KingCobra), and the T-pose reference is simply the
+ # first NPZ in sorted order. Align by the expected name list when given;
+ # a silent positional paste would tag every joint with the wrong label.
+ raw_clean = clean_names if clean_names else raw_names
+ if clean_names and expected_names is not None:
+ if len(clean_names) != len(expected_names):
+ raise ValueError(
+ f'[{object_type}] clean_joint_names has {len(clean_names)} '
+ f'entries but joint_names.json has {len(expected_names)}; '
+ f'clean_joint_names.json is stale โ€” re-run the joint-name '
+ f'cleaner.')
+ if raw_names != expected_names:
+ index = {n: i for i, n in enumerate(expected_names)}
+ try:
+ raw_clean = [clean_names[index[n]] for n in raw_names]
+ except KeyError as e:
+ raise ValueError(
+ f'[{object_type}] T-pose reference NPZ contains joint '
+ f'{e} that is absent from joint_names.json; the export '
+ f'metadata is out of date.')
+ logger.info(f'[{object_type}] T-pose NPZ joint order differs '
+ f'from joint_names.json; clean names realigned by '
+ f'name.')
+
+ corps_idxs = None
+ if corps_names is not None:
+ try:
+ corps_idxs = [raw_names.index(n) for n in corps_names]
+ except ValueError as e:
+ raise ValueError(
+ f'Failed to resolve the requested joint subset (mixamo core '
+ f'joints / corps_names) for {object_type} in tpos_data: {e}. '
+ f'Raw names: {raw_names}')
+ clean_names_list = [raw_clean[i] for i in corps_idxs]
+ else:
+ clean_names_list = list(raw_clean)
+
+ return corps_idxs, corps_names, clean_names_list, raw_parents, raw_njoints
+
+
+# ===========================================================================
+# NPZ loading
+# ===========================================================================
+
+def _names_permutation(ref_names, ref_parents, names, parents):
+ """Permutation aligning a clip's joint order to the reference skeleton.
+
+ Returns ``perm`` such that ``array[:, perm]`` reorders per-joint data
+ into the reference joint order, or None when the clip is not the same
+ skeleton (different name set, duplicate names, or a topology that
+ disagrees under the name mapping).
+ """
+ if sorted(names) != sorted(ref_names) or len(set(names)) != len(names):
+ return None
+ index = {n: i for i, n in enumerate(names)}
+ perm = [index[n] for n in ref_names]
+ for i, p in enumerate(ref_parents):
+ q = parents[perm[i]]
+ if (q == -1) != (p == -1) or (p != -1 and q != perm[p]):
+ return None
+ return perm
+
+
+def _load_npz_anim(object_npz_path, raw_parents, raw_njoints,
+ corps_idxs, bfs_order, fps, raw_names=None):
+ """Load a motion NPZ, apply joint subset/reorder, and downsample.
+
+ Returns:
+ Tuple of (motion_anim, motion_fps, None) on success, or
+ (None, None, reason_string) when the motion should be skipped.
+ """
+ anim_data = np.load(object_npz_path, allow_pickle=True)
+ nframes, njoints = anim_data['anim_local_rot'].shape[:2]
+
+ if njoints != raw_njoints:
+ reason = f'joint count mismatch: expected {raw_njoints}, got {njoints}'
+ logger.warning(f'Skipping clip ({reason}): {object_npz_path}')
+ return None, None, reason
+
+ anim_local_rot = anim_data['anim_local_rot']
+ anim_local_pos = anim_data['anim_local_pos']
+ rest_local_pos = anim_data['rest_local_pos']
+
+ # corps_idxs/bfs_order are positional indices into the T-pose clip's
+ # joint order; a source file enumerating the same skeleton in a
+ # different order would silently misassign joints, so align by name.
+ if raw_names is not None:
+ clip_names = anim_data['names'].tolist()
+ if clip_names != raw_names:
+ perm = _names_permutation(raw_names, raw_parents, clip_names,
+ anim_data['parents'].tolist())
+ if perm is None:
+ reason = 'joint names mismatch with T-pose skeleton'
+ logger.warning(f'Skipping clip ({reason}): {object_npz_path}')
+ return None, None, reason
+ logger.info(f'Joint order differs from T-pose skeleton; '
+ f'reordering by name: {object_npz_path}')
+ anim_local_rot = anim_local_rot[:, perm]
+ anim_local_pos = anim_local_pos[:, perm]
+ rest_local_pos = rest_local_pos[perm]
+
+ motion_positions = rest_local_pos.copy()[None, :].repeat(nframes, axis=0)
+ motion_positions[:, 0] = anim_local_pos[:, 0] # root joint's local position is the animated global position
+
+ motion_anim = Animation(
+ rotations=Quaternions(anim_local_rot),
+ positions=motion_positions,
+ orients=Quaternions.id(raw_njoints),
+ offsets=rest_local_pos,
+ parents=raw_parents,
+ )
+
+ if corps_idxs is not None:
+ motion_anim = motion_anim[:, corps_idxs]
+ motion_anim = motion_anim[:, bfs_order]
+
+ if fps >= 60:
+ motion_anim = motion_anim[::2]
+ motion_fps = fps // 2
+ else:
+ motion_fps = fps
+
+ return motion_anim, motion_fps, None
+
+
+def _load_tpos_npz(tpos_data, corps_idxs=None, corps_names=None):
+ """Load T-pose from NPZ data, apply joint subset and BFS reorder.
+
+ Returns:
+ Tuple of (tpos_anim, parents, names, fps, bfs_order).
+ """
+ names, fps = tpos_data['names'].tolist(), tpos_data['fps'].item()
+ parents = tpos_data['parents'].tolist()
+ njoints = len(parents)
+
+ tpos_anim = Animation(
+ rotations=Quaternions(tpos_data['rest_local_rot'][None]),
+ positions=tpos_data['rest_local_pos'][None],
+ orients=Quaternions.id(njoints),
+ offsets=tpos_data['rest_local_pos'],
+ parents=tpos_data['parents'],
+ )
+
+ if corps_idxs is not None:
+ tpos_anim = tpos_anim[:, corps_idxs]
+ names = corps_names if corps_names is not None else [names[i] for i in corps_idxs]
+
+ tpos_anim, bfs_order, parents = reorder_anim_bfs(tpos_anim)
+ names = [names[i] for i in bfs_order]
+
+ return tpos_anim, parents, names, fps, bfs_order
+
+
+def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,
+ corps_idxs, bfs_order, fps, scale_factor,
+ ground_height, face_joint_idxs,
+ body_axis=False, raw_names=None):
+ """Load a motion NPZ and canonicalize facing / XZ / ground + scale.
+
+ ``scale_factor`` and ``ground_height`` are reused from T-pose
+ calibration so every motion of this object type shares a
+ diameter-consistent scale and a consistent floor height (matching the
+ npz pipeline); ``root_pose_init_xz`` is recomputed fresh per
+ motion so each motion starts centered at the origin. Clips are then
+ simple slices of the canonicalized motion.
+
+ Returns:
+ Tuple of ((motion_anim, motion_fps), None) on success, or
+ (None, reason_string) when the motion should be skipped.
+ """
+ motion_anim, motion_fps, skip_reason = _load_npz_anim(
+ object_npz_path, raw_parents, raw_njoints, corps_idxs, bfs_order, fps,
+ raw_names=raw_names)
+ if motion_anim is None:
+ return None, skip_reason
+
+ motion_anim, _, _, _ = canonicalize_anim(
+ motion_anim,
+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,
+ ground_height=ground_height,
+ scale_factor=scale_factor)
+ return (motion_anim, motion_fps), None
+
+
+# ===========================================================================
+# T-pose processing and topology conditioning
+# ===========================================================================
+
+def process_tpose(tpos_data, corps_idxs=None, corps_names=None,
+ face_joints=None, target_diameter=2.0):
+ """Load T-pose from NPZ with facing -> center XZ -> scale -> ground.
+
+ ``scale_factor``, ``ground_height`` (and ``offsets``) are threaded
+ downstream so motions share a diameter-consistent scale and a
+ consistent floor height across the object type (matching the npz
+ pipeline); each motion's ``root_pose_init_xz`` is still
+ recomputed fresh so every motion starts centered at origin.
+
+ Raises:
+ DegenerateSkeletonError: if the skeleton has no measurable extent
+ (all-zero bone offsets), which would make diameter scaling divide
+ by zero.
+
+ Returns:
+ Tuple of (tpos_anim, offsets, scale_factor, ground_height,
+ parents, names, fps, bfs_order,
+ face_joint_idxs, body_axis).
+ """
+ tpos_anim, parents, names, fps, bfs_order = _load_tpos_npz(
+ tpos_data, corps_idxs, corps_names)
+
+ diameter = float(get_skeleton_diameter(tpos_anim))
+ if not np.isfinite(diameter) or diameter <= MIN_SKELETON_DIAMETER:
+ raise DegenerateSkeletonError(
+ f'degenerate skeleton: leaf-to-leaf diameter {diameter:.3e} '
+ f'(zero-length bones?), cannot scale to target diameter '
+ f'{target_diameter}')
+
+ face_joint_idxs, body_axis = resolve_face_joint_idxs(face_joints, names)
+ tpos_anim, _, ground_height, scale_factor = canonicalize_anim(
+ tpos_anim,
+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,
+ target_scale_metric=target_diameter)
+
+ offsets = np.array(tpos_anim.offsets).copy() # (J, 3)
+
+ return (tpos_anim, offsets, scale_factor, ground_height,
+ parents, names, fps, bfs_order,
+ face_joint_idxs, body_axis)
+
+
+def build_topology_cond(object_type, parents, offsets, names, clean_joint_names,
+ tpos_global_pos, tpos_local_rot, tpos_global_rot,
+ max_path_len=5, max_freqs=8,
+ face_joint_idxs=None, body_axis=False,
+ scale_factor=1.0, ground_height=None):
+ """Build the topology conditioning dict from a canonicalized T-pose.
+
+ Produces every field consumed by the training dataset loader plus the
+ joint-name and optional face-joint metadata fields.
+
+ ``ground_height`` is the vertical offset actually subtracted from every
+ clip of this object type, or None when each motion was grounded on its
+ own minimum Y (the default). It is recorded together with
+ ``ground_height_mode`` (``'tpos'`` / ``'per_motion'``) so the cond stays
+ self-describing โ€” do not pass the T-pose's own ground height here unless
+ that same value was reused for the motions.
+ """
+ parents_arr = np.asarray(parents)
+ joint_relations, joint_graph_dists = compute_edge_relations_and_distances(
+ parents_arr, max_path_len=max_path_len,
+ )
+ joint_depths = compute_joint_depths(parents_arr)
+ edge_indexs = compute_edge_indexs(parents_arr)
+ spectral_feats, _ = compute_laplacian_eigenvectors(parents_arr, max_freqs=max_freqs)
+ kinematic_chains = compute_kinematic_chains(parents_arr)
+
+ tpos_offsets = offsets_from_positions(tpos_global_pos, parents_arr)
+
+ cond = {
+ 'object_type': object_type,
+ 'parents': parents,
+ 'offsets': offsets, # (J, 3)
+ 'tpos_offsets': tpos_offsets, # (J, 3)
+ 'joint_names': names,
+ 'clean_joint_names': clean_joint_names,
+ 'tpos_first_frame': tpos_global_pos, # (J, 3)
+ 'tpos_local_rotations': tpos_local_rot, # (J, 4) quaternion
+ 'tpos_global_rotations': tpos_global_rot, # (J, 4) quaternion
+ 'joint_relations': joint_relations, # (J, J)
+ 'joint_graph_dists': joint_graph_dists, # (J, J)
+ 'joint_depths': joint_depths, # (J,)
+ 'edge_indexs': edge_indexs, # (2, 2*(J-1))
+ 'spectral_feats': spectral_feats, # (J, K)
+ 'kinematic_chains': kinematic_chains, # list of lists of joint indices
+ 'scale_factor': scale_factor, # float
+ 'ground_height': ground_height, # float, or None per-motion
+ 'ground_height_mode': 'tpos' if ground_height is not None else 'per_motion',
+ }
+
+ if face_joint_idxs is not None:
+ cond['face_joint_idxs'] = {
+ 'r_hip': face_joint_idxs[0],
+ 'l_hip': face_joint_idxs[1],
+ 'body_axis': body_axis,
+ }
+
+ return cond
+
+
+# ===========================================================================
+# Motion clip processing
+# ===========================================================================
+
+def trim_static_ends(motion_anim, static_threshold=1e-5):
+ """Strip leading/trailing frames where every joint is nearly stationary.
+
+ Per-frame activity is the max over joints of consecutive global-position
+ displacement. Frames at the head/tail with activity below
+ ``static_threshold`` are removed; interior static spans are preserved.
+
+ Returns:
+ Tuple of (trimmed_anim, start_idx, end_idx). ``end_idx`` is
+ exclusive; returns an empty slice when the entire motion is static.
+ """
+ nframes = len(motion_anim)
+ if nframes < 2:
+ return motion_anim, 0, nframes
+
+ global_positions = positions_global(motion_anim)
+ step = np.linalg.norm(np.diff(global_positions, axis=0), axis=-1) # (F-1, J)
+ max_joint_step = step.max(axis=-1) # (F-1,)
+
+ active = max_joint_step > static_threshold
+ if not active.any():
+ return motion_anim[:0], 0, 0
+
+ active_idx = np.where(active)[0]
+ start = int(active_idx[0])
+ end = min(int(active_idx[-1]) + 2, nframes) # +1 diff offset, +1 exclusive
+ return motion_anim[start:end], start, end
+
+
+def extract_clip_feats(motion_anim, clip_start, clip_end,
+ face_joint_idxs, body_axis=False):
+ """Slice a canonicalized motion into a clip and extract per-frame data.
+
+ The motion is assumed to already be canonicalized against T-pose
+ calibration (facing / XZ / ground / scale) by
+ :func:`load_and_canonicalize_motion`, so clips are simple slices โ€” no
+ per-clip re-centering.
+
+ Returns:
+ Tuple of (local_rotations (F, J, 4), global_positions (F, J, 3),
+ root_facing_quat (F, 4)).
+ """
+ clip_anim = motion_anim[clip_start:clip_end]
+ local_rotations = clip_anim.rotations.copy() # (F, J, 4)
+ global_positions = positions_global(clip_anim)
+ root_facing_quat = get_root_facing_quat(
+ global_positions, face_joint_idxs, body_axis=body_axis)
+
+ return local_rotations.qs, global_positions, root_facing_quat.qs
+
+
+def _activity_below_threshold(global_positions, threshold):
+ """Return (is_low, metrics) from joint-activity metrics for a sequence."""
+ metrics = get_motion_activity_metrics(global_positions)
+ is_low = (metrics['joint_activity_topk'] < threshold
+ and metrics['root_activity'] < threshold)
+ return is_low, metrics
+
+
+def _format_activity(metrics):
+ return (f'joint={metrics["joint_activity"]:.4f}, '
+ f'topk={metrics["joint_activity_topk"]:.4f}, '
+ f'root={metrics["root_activity"]:.4f}')
+
+
+# ===========================================================================
+# Output I/O
+# ===========================================================================
+
+def save_clip(clip_name,
+ global_positions, local_rotations, root_facing_quat,
+ parents, motion_fps,
+ motions_save_dir, animations_save_dir,
+ spectral_feats=None, caption=None, save_vis=True,
+ vis_ground=False):
+ """Save per-clip canonicalized positions/rotations/facing + MP4.
+
+ When ``spectral_feats`` is supplied, the MP4 colors joints by PCA-projected
+ spectral features so topologically close joints share colors across
+ object types; otherwise falls back to the default root/non-root palette.
+ ``caption``, if non-empty, is rendered as the video title. The MP4
+ dominates per-clip time; pass ``save_vis=False`` to skip it.
+
+ The NPZ is written atomically: nothing downstream re-validates clip files,
+ so an interrupted run must not leave a truncated one for the training
+ loader to read.
+ """
+ with atomic_output_path(pjoin(motions_save_dir, clip_name + '.npz')) as tmp_path:
+ # A file object keeps np.savez from appending '.npz' to the temp name.
+ with open(tmp_path, 'wb') as f:
+ np.savez(
+ f,
+ global_positions=global_positions,
+ local_rotations=local_rotations,
+ root_facing_quat=root_facing_quat,
+ fps=int(motion_fps),
+ )
+
+ if not save_vis:
+ return
+ clip_vis_path = pjoin(animations_save_dir, clip_name + '.mp4')
+ title = caption or None
+ if vis_ground:
+ # Checkerboard ground + follow camera + trajectory/shadows; keeps
+ # the spectral palette when available.
+ save_skeleton_motion_ground(
+ save_path=clip_vis_path, parents=parents,
+ positions=global_positions, spectral_feats=spectral_feats,
+ fps=int(motion_fps), title=title,
+ figsize=(6.4, 6.4), dpi=100,
+ )
+ return
+ if spectral_feats is not None:
+ save_skeleton_motion_spectral(
+ save_path=clip_vis_path, parents=parents,
+ positions=global_positions, spectral_feats=spectral_feats,
+ fps=int(motion_fps), title=title,
+ figsize=(6.4, 6.4), dpi=100,
+ )
+ else:
+ save_skeleton_motion(
+ save_path=clip_vis_path, parents=parents,
+ positions=global_positions, fps=int(motion_fps), title=title,
+ figsize=(6.4, 6.4), dpi=100,
+ )
+
+
+def _save_tpose_vis(save_dir, object_type, parents, tpos_anim,
+ spectral_feats=None):
+ tpos_save_dir = pjoin(save_dir, 'tpose')
+ os.makedirs(tpos_save_dir, exist_ok=True)
+ tpos_vis_path = pjoin(tpos_save_dir, f'{object_type}.png')
+ tpos_global_pos = positions_global(tpos_anim)[0]
+ if spectral_feats is not None:
+ save_skeleton_tpos_spectral(
+ save_path=tpos_vis_path, parents=parents,
+ positions=tpos_global_pos, spectral_feats=spectral_feats,
+ figsize=(6.4, 6.4), dpi=100,
+ )
+ else:
+ save_skeleton_tpose(
+ save_path=tpos_vis_path, parents=parents,
+ positions=tpos_global_pos,
+ figsize=(6.4, 6.4), dpi=100,
+ )
+
+
+# ===========================================================================
+# Per-object pipeline
+# ===========================================================================
+
+def _process_motion_file(object_npz_path, ctx):
+ """Process one motion NPZ into saved clips.
+
+ ``ctx`` bundles the per-object state shared across every motion file of
+ an object type (skeleton topology, scale, thresholds, save dirs,
+ captions). Returns a dict with:
+ n_saved : int โ€” number of clips written to disk
+ frames : int โ€” total kept frames across clips
+ clip_captions : Dict[str, str] โ€” per-clip caption map
+ filtered : List[dict] โ€” entries dropped (with reason)
+ """
+ base_name = str(Path(object_npz_path).stem)
+ min_frames = ctx['min_frames']
+
+ # --- Load + motion-level canonicalize ----------------------------------
+ result, skip_reason = load_and_canonicalize_motion(
+ object_npz_path, ctx['raw_parents'], ctx['raw_njoints'],
+ ctx['corps_idxs'], ctx['bfs_order'], ctx['fps'], ctx['scale_factor'],
+ ctx['ground_height'], ctx['face_joint_idxs'],
+ body_axis=ctx['body_axis'], raw_names=ctx.get('raw_names'))
+ if result is None:
+ return {'n_saved': 0, 'frames': 0, 'clip_captions': {},
+ 'filtered': [{'name': base_name, 'reason': skip_reason}]}
+ motion_anim, motion_fps = result
+
+ # --- Strip static pre/post-roll ----------------------------------------
+ orig_nframes = len(motion_anim)
+ motion_anim, _, _ = trim_static_ends(
+ motion_anim, static_threshold=ctx['static_threshold'])
+ if len(motion_anim) < min_frames:
+ reason = (f'too few frames after static trim: {len(motion_anim)} '
+ f'< {min_frames} (orig {orig_nframes})')
+ return {'n_saved': 0, 'frames': 0, 'clip_captions': {},
+ 'filtered': [{'name': base_name, 'reason': reason}]}
+
+ # --- Clip windowing ----------------------------------------------------
+ nframes = len(motion_anim)
+ clips = get_motion_clip_windows(
+ nframes, ctx['max_clip_len'], ctx['clip_stride'], apply_clip=ctx['apply_clip'])
+ caption = ctx['captions'].get(base_name, '') if ctx['captions'] else ''
+
+ clip_captions, filtered, n_saved, frames = {}, [], 0, 0
+ for clip_idx, (clip_start, clip_end) in enumerate(clips):
+ (clip_local_rotations, clip_global_positions,
+ clip_root_facing_quat) = extract_clip_feats(
+ motion_anim, clip_start, clip_end,
+ ctx['face_joint_idxs'],
+ body_axis=ctx['body_axis'],
+ )
+
+ is_low, clip_metrics = _activity_below_threshold(
+ clip_global_positions, ctx['activity_threshold'])
+ clip_name = f'{base_name}-{clip_idx:03d}'
+ if is_low:
+ filtered.append({
+ 'name': clip_name,
+ 'reason': f'low activity: {_format_activity(clip_metrics)}',
+ })
+ continue
+
+ save_clip(
+ clip_name,
+ clip_global_positions, clip_local_rotations,
+ clip_root_facing_quat,
+ ctx['parents'], motion_fps,
+ ctx['motions_save_dir'], ctx['animations_save_dir'],
+ spectral_feats=ctx.get('spectral_feats'),
+ caption=caption, save_vis=ctx['save_vis'],
+ vis_ground=ctx.get('vis_ground', False))
+ n_saved += 1
+ frames += clip_end - clip_start
+ if caption:
+ clip_captions[clip_name] = caption
+
+ return {
+ 'n_saved': n_saved,
+ 'frames': frames,
+ 'clip_captions': clip_captions,
+ 'filtered': filtered,
+ }
+
+
+def process_object(object_type, object_npzs, save_dir,
+ max_path_len=5, max_freqs=8,
+ max_clip_len=100, clip_stride=50, apply_clip=True,
+ captions=None,
+ face_joints=None, corps_names=None, clean_names=None,
+ expected_names=None, require_face=False,
+ target_diameter=2.0, activity_threshold=0.02,
+ static_threshold=1e-05, min_frames=8,
+ min_joints=None, max_joints=None,
+ use_tpos_ground_height=False, save_vis=True,
+ vis_ground=False):
+ """Process all motion clips for one object type.
+
+ Emits per-clip NPZs holding raw canonicalized arrays โ€” global
+ positions, local rotation quaternions, root facing quaternions โ€”
+ from which the training side builds its motion representation.
+
+ When ``apply_clip`` is False, each motion yields a single clip truncated
+ to the first ``max_clip_len`` frames; when True, motions are cropped
+ into overlapping fixed-length clips with stride ``clip_stride``.
+
+ ``use_tpos_ground_height`` controls whether the T-pose's ground height
+ is reused for every motion (True) or recomputed per motion from its own
+ minimum Y so each motion's lowest joint sits on the ground (False,
+ default). Whichever is used is recorded in the cond as ``ground_height``
+ / ``ground_height_mode``.
+
+ Object types with no motion NPZs, a joint count outside
+ ``[min_joints, max_joints]`` or a degenerate (zero-extent) skeleton are
+ skipped with a reason instead of raising.
+
+ Returns:
+ Tuple of (cond, n_clips, n_frames, n_joints, filtered).
+ """
+ def _skip(reason, njoints=0):
+ """Skip the whole object type with a recorded reason."""
+ logger.info(f'[{object_type}] skipped: {reason}')
+ return None, 0, 0, njoints, [{'name': f'{object_type} (all clips)',
+ 'reason': reason}]
+
+ if not object_npzs:
+ return _skip('no motion NPZ files found')
+
+ motions_save_dir = pjoin(save_dir, 'motions')
+ animations_save_dir = pjoin(save_dir, 'videos')
+ for d in (motions_save_dir, animations_save_dir):
+ os.makedirs(d, exist_ok=True)
+
+ # ---- T-pose + topology ----
+ tpos_data = np.load(object_npzs[0], allow_pickle=True)
+ (corps_idxs, corps_names_list, clean_names_list,
+ raw_parents, raw_njoints) = resolve_joint_names(
+ object_type, tpos_data, corps_names, clean_names,
+ expected_names=expected_names)
+
+ # Joint count is fixed by the corps subset (BFS reordering preserves it),
+ # so gate before the T-pose canonicalization rather than after.
+ njoints = len(corps_idxs) if corps_idxs is not None else raw_njoints
+ if ((min_joints is not None and njoints < min_joints) or
+ (max_joints is not None and njoints > max_joints)):
+ return _skip(f'{njoints} joints outside [{min_joints}, {max_joints}]',
+ njoints)
+
+ try:
+ (tpos_anim, offsets, scale_factor, tpos_ground_height,
+ parents, names, fps, bfs_order,
+ face_joint_idxs, body_axis) = process_tpose(
+ tpos_data, corps_idxs=corps_idxs,
+ corps_names=corps_names_list,
+ face_joints=face_joints,
+ target_diameter=target_diameter)
+ except DegenerateSkeletonError as e:
+ return _skip(str(e), njoints)
+
+ face_named = bool(face_joints) and bool(
+ face_joints.get('r_hip', {}).get('raw')
+ or face_joints.get('l_hip', {}).get('raw'))
+ if face_named and face_joint_idxs == [-1, -1]:
+ # The entry exists but its raw names are not in this (possibly
+ # corps-filtered) skeleton โ€” canonicalization silently ran with
+ # identity facing above. Never acceptable where facing is required.
+ detail = (f'face_joints entry did not resolve: '
+ f'r_hip={face_joints.get("r_hip", {}).get("raw")!r}, '
+ f'l_hip={face_joints.get("l_hip", {}).get("raw")!r} '
+ f'not both present in the processed skeleton')
+ if require_face:
+ raise ValueError(f'[{object_type}] {detail}; refusing to fall '
+ f'back to identity facing for this dataset.')
+ logger.warning(f'[{object_type}] {detail}; using identity facing.')
+
+ # Ground height actually applied to the clips: the T-pose's when it is
+ # reused, otherwise None (each motion is grounded on its own minimum Y).
+ ground_height = tpos_ground_height if use_tpos_ground_height else None
+
+ clean_names_bfs = [clean_names_list[i] for i in bfs_order]
+ tpos_global_pos = positions_global(tpos_anim)[0]
+ tpos_local_rot = tpos_anim.rotations.qs[0]
+ tpos_global_rot = rotations_global(tpos_anim).qs[0]
+ object_cond = build_topology_cond(
+ object_type, parents, offsets, names, clean_names_bfs,
+ tpos_global_pos, tpos_local_rot, tpos_global_rot,
+ max_path_len=max_path_len, max_freqs=max_freqs,
+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,
+ scale_factor=scale_factor, ground_height=ground_height
+ )
+ spectral_feats = object_cond.get('spectral_feats')
+
+ _save_tpose_vis(save_dir, object_type, parents, tpos_anim,
+ spectral_feats=spectral_feats)
+
+ logger.info(f'[{object_type}] n_joints={njoints} fps={fps} '
+ f'scale={scale_factor:.4f} files={len(object_npzs)}')
+
+ # ---- Per-motion processing ----
+ ctx = {
+ # skeleton / scale / topology
+ 'raw_parents': raw_parents, 'raw_njoints': raw_njoints,
+ 'raw_names': tpos_data['names'].tolist(),
+ 'corps_idxs': corps_idxs, 'bfs_order': bfs_order,
+ 'parents': parents,
+ 'fps': fps, 'scale_factor': scale_factor,
+ 'ground_height': ground_height, # None => per-motion grounding
+ 'face_joint_idxs': face_joint_idxs, 'body_axis': body_axis,
+ 'spectral_feats': spectral_feats,
+ # clip windowing / filtering
+ 'max_clip_len': max_clip_len, 'clip_stride': clip_stride,
+ 'apply_clip': apply_clip,
+ 'activity_threshold': activity_threshold,
+ 'static_threshold': static_threshold,
+ 'min_frames': min_frames,
+ # I/O
+ 'motions_save_dir': motions_save_dir,
+ 'animations_save_dir': animations_save_dir,
+ 'captions': captions,
+ 'save_vis': save_vis, 'vis_ground': vis_ground,
+ }
+
+ filtered_clips, clip_captions = [], {}
+ n_clips, n_frames = 0, 0
+
+ for object_npz_path in object_npzs:
+ result = _process_motion_file(object_npz_path, ctx)
+ filtered_clips.extend(result.get('filtered', []))
+ n_clips += result.get('n_saved', 0)
+ n_frames += result.get('frames', 0)
+ clip_captions.update(result.get('clip_captions', {}))
+
+ if clip_captions:
+ object_cond['captions'] = clip_captions
+
+ logger.info(f'[{object_type}] saved {n_clips} clips ({n_frames} frames), '
+ f'filtered {len(filtered_clips)}')
+
+ return object_cond, n_clips, n_frames, njoints, filtered_clips
diff --git a/data_process/utils/plotting.py b/data_process/utils/plotting.py
new file mode 100644
index 0000000..19f3c87
--- /dev/null
+++ b/data_process/utils/plotting.py
@@ -0,0 +1,1039 @@
+"""3D skeleton visualization utilities (matplotlib, headless Agg backend).
+
+All public functions operate on a parent-array skeleton representation:
+
+ parents : array_like of shape (J,)
+ parents[j] is the index of joint j's parent, or -1 for the root.
+ positions : array_like
+ Joint positions. Single pose: (J, 3) or (J, D) with D >= 3 (extra
+ feature dims are stripped). Motion: (T, J, 3) or (T, J, D).
+
+Two families of functions:
+
+ render_skeleton_* -> RGB numpy array(s) in memory
+ save_skeleton_* -> write PNG (pose) / MP4 (motion) to disk
+"""
+
+from pathlib import Path
+from typing import List, Optional, Sequence, Tuple
+
+import numpy as np
+import matplotlib
+import imageio
+from matplotlib.backends.backend_agg import FigureCanvasAgg
+from matplotlib.figure import Figure
+from matplotlib.ticker import LinearLocator
+from mpl_toolkits.mplot3d.art3d import Line3DCollection
+from mpl_toolkits.mplot3d import proj3d
+
+matplotlib.use('Agg')
+
+
+# ============================================================================
+# Geometry / array helpers
+# ============================================================================
+
+def _as_xyz(positions) -> np.ndarray:
+ """Cast to float32 and truncate the last axis to xyz."""
+ pts = np.asarray(positions, dtype=np.float32)
+ if pts.shape[-1] > 3:
+ pts = pts[..., :3]
+ return pts
+
+
+def _maybe_rotate_y_up(pts: np.ndarray, rotate: bool) -> np.ndarray:
+ """Swap Y/Z so a Z-up skeleton renders upright in matplotlib's Y-up view."""
+ if not rotate:
+ return pts
+ out = pts.copy()
+ out[..., 1] = -pts[..., 2]
+ out[..., 2] = pts[..., 1]
+ return out
+
+
+def _edges_from_parents(parents: Sequence[int]) -> np.ndarray:
+ """(E, 2) edge list of (child, parent) pairs, skipping the root."""
+ return np.array(
+ [(i, p) for i, p in enumerate(parents) if p != -1],
+ dtype=np.int32,
+ )
+
+
+def _cubic_bounds(points: np.ndarray):
+ """Axis-aligned cubic bounding box centered on the points (preserves aspect)."""
+ x_min, x_max = points[..., 0].min(), points[..., 0].max()
+ y_min, y_max = points[..., 1].min(), points[..., 1].max()
+ z_min, z_max = points[..., 2].min(), points[..., 2].max()
+ side = max(x_max - x_min, y_max - y_min, z_max - z_min)
+ cx, cy, cz = (x_min + x_max) / 2, (y_min + y_max) / 2, (z_min + z_max) / 2
+ half = side / 2
+ return cx - half, cx + half, cy - half, cy + half, cz - half, cz + half
+
+
+def _tight_bounds(points: np.ndarray, pad_ratio: float = 0.08,
+ abs_pad_ratio: float = 0.0):
+ """Per-axis bounding box with proportional padding; pairs with non-cubic box_aspect.
+
+ ``pad_ratio`` adds *proportional* padding (``pad_ratio * axis_length``) โ€”
+ tiny on a thin axis.
+ ``abs_pad_ratio`` adds *absolute* padding tied to the overall scale
+ (``abs_pad_ratio * max_axis_length``) โ€” the right knob when labels or
+ annotations are placed at offsets proportional to the full skeleton size.
+ """
+ x_min, x_max = points[..., 0].min(), points[..., 0].max()
+ y_min, y_max = points[..., 1].min(), points[..., 1].max()
+ z_min, z_max = points[..., 2].min(), points[..., 2].max()
+ # Guard against degenerate flat axes (e.g. planar skeleton).
+ dx = max(x_max - x_min, 1e-6)
+ dy = max(y_max - y_min, 1e-6)
+ dz = max(z_max - z_min, 1e-6)
+ scale = max(dx, dy, dz)
+ abs_pad = abs_pad_ratio * scale
+ px = dx * pad_ratio + abs_pad
+ py = dy * pad_ratio + abs_pad
+ pz = dz * pad_ratio + abs_pad
+ return (x_min - px, x_max + px,
+ y_min - py, y_max + py,
+ z_min - pz, z_max + pz)
+
+
+def _box_aspect_from_bounds(bounds) -> Tuple[float, float, float]:
+ x_min, x_max, y_min, y_max, z_min, z_max = bounds
+ return (max(x_max - x_min, 1e-6),
+ max(y_max - y_min, 1e-6),
+ max(z_max - z_min, 1e-6))
+
+
+# ============================================================================
+# Matplotlib figure helpers
+# ============================================================================
+
+def _new_figure(bounds, elev: Optional[float], azim: Optional[float],
+ figsize=(6.4, 4.8), dpi: int = 120,
+ box_aspect=(1, 1, 1)):
+ """Create a 3D figure/axes configured for skeleton rendering.
+
+ Use ``box_aspect=(1,1,1)`` with :func:`_cubic_bounds`, or per-axis extents
+ with :func:`_tight_bounds` to fill the figure with a long-thin subject.
+ """
+ fig = Figure(figsize=figsize, dpi=dpi)
+ canvas = FigureCanvasAgg(fig)
+ ax = fig.add_subplot(111, projection='3d', computed_zorder=False)
+
+ x_min, x_max, y_min, y_max, z_min, z_max = bounds
+ ax.set_xlim([x_min, x_max])
+ ax.set_ylim([y_min, y_max])
+ ax.set_zlim([z_min, z_max])
+
+ ax.set_xticks([]); ax.set_yticks([]); ax.set_zticks([])
+ ax.set_xticklabels([]); ax.set_yticklabels([]); ax.set_zticklabels([])
+ ax.xaxis.set_major_locator(LinearLocator(8))
+ ax.yaxis.set_major_locator(LinearLocator(8))
+ ax.zaxis.set_major_locator(LinearLocator(8))
+ ax.grid(True, color='gray', linestyle='--', linewidth=0.5)
+ ax.set_box_aspect(box_aspect)
+
+ if elev is not None and azim is not None:
+ ax.view_init(elev=elev, azim=azim)
+
+ fig.subplots_adjust(left=-0.05, right=1.05, top=1.05, bottom=-0.05)
+ return fig, canvas, ax
+
+
+def _capture_rgb(canvas: FigureCanvasAgg) -> np.ndarray:
+ """Draw the canvas and return its pixels as an (H, W, 3) uint8 array."""
+ canvas.draw()
+ w, h = canvas.get_width_height()
+ rgba = np.frombuffer(canvas.buffer_rgba(), dtype=np.uint8).reshape(h, w, 4)
+ return rgba[..., :3].copy()
+
+
+# ============================================================================
+# Skeleton drawing primitives
+# ============================================================================
+
+def _draw_static_skeleton(ax, pts: np.ndarray, parents: Sequence[int],
+ edges: np.ndarray) -> None:
+ """Draw bones (gray) and joints (root red, others blue) for a single pose."""
+ segs = np.zeros((len(edges), 2, 3), dtype=np.float32)
+ segs[:, 0] = pts[edges[:, 0]]
+ segs[:, 1] = pts[edges[:, 1]]
+ ax.add_collection3d(Line3DCollection(segs, colors='gray', linewidths=1.0))
+
+ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')
+ ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2], s=10, c=joint_colors)
+ ax.scatter(pts[0:1, 0], pts[0:1, 1], pts[0:1, 2], s=15, c='r')
+
+
+def _init_motion_artists(ax, edges: np.ndarray, first_frame: np.ndarray,
+ joint_colors, joint_size: float,
+ bone_color: str = 'gray',
+ bone_linewidth: float = 1.0,
+ joint_edgecolors=None,
+ joint_edgewidth: float = 0.0):
+ """Create reusable bone/joint artists for a motion render.
+
+ Returns ``(segs, lc, sc)`` where ``segs`` is the mutable (E, 2, 3) buffer
+ backing the Line3DCollection ``lc``, and ``sc`` is the joint scatter.
+ Callers mutate ``segs`` and ``sc._offsets3d`` per frame.
+ """
+ segs = np.zeros((len(edges), 2, 3), dtype=np.float32)
+ lc = Line3DCollection(segs, colors=bone_color, linewidths=bone_linewidth)
+ ax.add_collection3d(lc)
+
+ kwargs = dict(s=joint_size, c=joint_colors, depthshade=False, zorder=2)
+ if joint_edgecolors is not None:
+ kwargs['edgecolors'] = joint_edgecolors
+ kwargs['linewidths'] = joint_edgewidth
+ sc = ax.scatter(first_frame[:, 0], first_frame[:, 1], first_frame[:, 2],
+ **kwargs)
+ return segs, lc, sc
+
+
+def _fibonacci_sphere_dirs(n: int) -> np.ndarray:
+ """``n`` approximately uniform unit vectors on the sphere (Fibonacci spiral)."""
+ phi = np.pi * (3.0 - np.sqrt(5.0)) # golden angle
+ idx = np.arange(n, dtype=np.float32)
+ y = 1.0 - 2.0 * idx / max(n - 1, 1)
+ radius = np.sqrt(np.maximum(1.0 - y * y, 0.0))
+ theta = phi * idx
+ x = np.cos(theta) * radius
+ z = np.sin(theta) * radius
+ return np.stack([x, y, z], axis=1).astype(np.float32)
+
+
+def _anchor_for_dir(d: np.ndarray):
+ """Pick matching (ha, va) text anchors so text grows *away* from the joint."""
+ ha = 'left' if d[0] > 0.15 else 'right' if d[0] < -0.15 else 'center'
+ va = 'bottom' if d[1] > 0.15 else 'top' if d[1] < -0.15 else 'center'
+ return ha, va
+
+
+def _sample_bone_obstacles(pts: np.ndarray, parents: Sequence[int],
+ samples_per_bone: int = 3) -> np.ndarray:
+ """Sample interior points along every bone segment; used as label obstacles."""
+ pts = np.asarray(pts, dtype=np.float32)
+ samples: List[np.ndarray] = []
+ ts = np.linspace(0.0, 1.0, samples_per_bone + 2)[1:-1] # interior only
+ for c, p in enumerate(parents):
+ if p == -1:
+ continue
+ a, b = pts[c], pts[p]
+ for t in ts:
+ samples.append(a * (1.0 - t) + b * t)
+ if not samples:
+ return np.zeros((0, 3), dtype=np.float32)
+ return np.stack(samples, axis=0)
+
+
+def _annotate_joints(ax, pts: np.ndarray, joint_names: Sequence[str],
+ bounds, font_size: int,
+ parents: Optional[Sequence[int]] = None) -> None:
+ """Place each joint's name next to its joint without overlapping bones/labels.
+
+ For each joint, scores 24 ร— 4 candidate offset directions (Fibonacci sphere
+ ร— multiple radii) in screen space by distance from other joints, bone
+ samples, and already-placed labels. The chosen anchor gets a colored
+ leader line back to the joint so the association stays visible even when
+ labels are pushed far out into empty pockets.
+ """
+ x_min, x_max, y_min, y_max, z_min, z_max = bounds
+ scale = max(x_max - x_min, y_max - y_min, z_max - z_min)
+ base = 0.08 * scale # leader lines let us push labels well clear of bones
+
+ unit_dirs_single = _fibonacci_sphere_dirs(36)
+ radii = np.array([1.0, 1.5, 2.2, 3.0], dtype=np.float32)
+ unit_dirs = np.repeat(unit_dirs_single, len(radii), axis=0)
+ radius_scales = np.tile(radii, len(unit_dirs_single)).astype(np.float32)
+
+ # Force a draw so box_aspect/view_init propagate into get_proj().
+ ax.figure.canvas.draw()
+ proj_matrix = ax.get_proj()
+
+ def project2d(xyz: np.ndarray) -> np.ndarray:
+ xs, ys, _ = proj3d.proj_transform(
+ xyz[..., 0], xyz[..., 1], xyz[..., 2], proj_matrix,
+ )
+ return np.stack([xs, ys], axis=-1)
+
+ J = len(pts)
+ parents_arr = (np.asarray(parents) if parents is not None
+ else np.full(J, -1, dtype=int))
+ joint_colors = [
+ ('red' if parents_arr[i] == -1 else 'steelblue') for i in range(J)
+ ]
+
+ joint_obstacles = pts
+ bone_obstacles = (_sample_bone_obstacles(pts, parents)
+ if parents is not None
+ else np.zeros((0, 3), dtype=np.float32))
+ placed_2d: List[np.ndarray] = []
+
+ joint_obstacles_2d = project2d(joint_obstacles)
+ bone_obstacles_2d = (project2d(bone_obstacles)
+ if len(bone_obstacles) > 0
+ else np.zeros((0, 2), dtype=np.float32))
+
+ for i, (p, name) in enumerate(zip(pts, joint_names)):
+ offsets = unit_dirs * (radius_scales[:, None] * base)
+ candidates = p[None, :] + offsets # (N, 3)
+ candidates_2d = project2d(candidates) # (N, 2)
+
+ # Avoidance set in 2D: all joints (except self) + bone samples + placed labels.
+ mask = np.ones(len(joint_obstacles_2d), dtype=bool)
+ mask[i] = False
+ avoid_parts = [joint_obstacles_2d[mask]]
+ if len(bone_obstacles_2d) > 0:
+ avoid_parts.append(bone_obstacles_2d)
+ if placed_2d:
+ avoid_parts.append(np.stack(placed_2d, axis=0))
+ avoid_2d = np.concatenate(avoid_parts, axis=0)
+
+ diffs = candidates_2d[:, None, :] - avoid_2d[None, :, :]
+ min_dists = np.linalg.norm(diffs, axis=-1).min(axis=1)
+
+ # Small penalty on longer radii โ€” prefer shorter leaders when free.
+ screen_span = float(
+ max(np.ptp(candidates_2d[:, 0]), np.ptp(candidates_2d[:, 1]), 1e-6)
+ )
+ score = min_dists - 0.03 * radius_scales * screen_span
+ slot = int(np.argmax(score))
+
+ lp = candidates[slot]
+ ha, va = _anchor_for_dir(unit_dirs[slot])
+ color = joint_colors[i]
+
+ leader, = ax.plot([p[0], lp[0]], [p[1], lp[1]], [p[2], lp[2]],
+ color=color, linewidth=0.9, linestyle='-',
+ alpha=0.9, zorder=9)
+ leader.set_clip_on(False)
+ txt = ax.text(lp[0], lp[1], lp[2],
+ f"[{i}] {name}", fontsize=font_size, color='black',
+ ha=ha, va=va, zorder=10,
+ bbox=dict(boxstyle='round,pad=0.2', facecolor='white',
+ edgecolor=color, linewidth=0.8, alpha=0.85))
+ txt.set_clip_on(False)
+ placed_2d.append(candidates_2d[slot])
+
+
+def _spectral_to_rgb(spectral_feats: np.ndarray) -> np.ndarray:
+ """Map (J, K) spectral features to (J, 3) RGB via PCA + per-channel min-max.
+
+ Topologically close joints get similar colors; distant joints contrast.
+ """
+ J, K = spectral_feats.shape
+ if K >= 3:
+ centered = spectral_feats - spectral_feats.mean(axis=0, keepdims=True)
+ _, _, Vt = np.linalg.svd(centered, full_matrices=False)
+ rgb = centered @ Vt[:3].T
+ else:
+ rgb = np.zeros((J, 3), dtype=np.float32)
+ rgb[:, :K] = spectral_feats
+
+ for c in range(3):
+ lo, hi = rgb[:, c].min(), rgb[:, c].max()
+ if hi - lo > 1e-8:
+ rgb[:, c] = (rgb[:, c] - lo) / (hi - lo)
+ else:
+ rgb[:, c] = 0.5
+ return rgb.astype(np.float32)
+
+
+# ============================================================================
+# Renderers: in-memory RGB arrays
+# ============================================================================
+
+def render_skeleton_tpose(parents: Sequence[int], positions,
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> np.ndarray:
+ """Render a single skeleton pose as an (H, W, 3) uint8 image."""
+ pts = _as_xyz(positions).reshape(-1, 3)
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ edges = _edges_from_parents(parents)
+ bounds = _cubic_bounds(pts)
+
+ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)
+ _draw_static_skeleton(ax, pts, parents, edges)
+ if title:
+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.98)
+ fig.subplots_adjust(top=0.90)
+ return _capture_rgb(canvas)
+
+
+def render_skeleton_tpose_annotated(parents: Sequence[int], positions,
+ joint_names: Sequence[str],
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ font_size: int = 6,
+ joint_marker_size: float = 80.0,
+ bone_linewidth: float = 2.0,
+ figsize: Tuple[float, float] = (12, 12),
+ dpi: int = 120) -> np.ndarray:
+ """Render a T-pose with each joint labeled by name for manual inspection.
+
+ Uses tight per-axis bounds + matching box_aspect so long-thin subjects
+ (snakes, centipedes) fill the figure without distortion, leaving more
+ room for labels.
+ """
+ pts = _as_xyz(positions).reshape(-1, 3)
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ assert len(joint_names) == pts.shape[0], (
+ f"joint_names has {len(joint_names)} entries but positions has "
+ f"{pts.shape[0]} joints"
+ )
+ edges = _edges_from_parents(parents)
+ # Labels are placed at offsets up to ~0.24*scale from each joint (see
+ # ``_annotate_joints``: base=0.08*scale ร— radii up to 3.0). Add 0.32*scale
+ # of absolute padding so the label bboxes and leader endpoints stay inside
+ # the plotted volume even on thin/long subjects (snakes, tentacles).
+ bounds = _tight_bounds(pts, abs_pad_ratio=0.32)
+ box_aspect = _box_aspect_from_bounds(bounds)
+
+ _, canvas, ax = _new_figure(
+ bounds, elev, azim,
+ figsize=figsize, dpi=dpi, box_aspect=box_aspect,
+ )
+
+ # Drop the dashed grid + axis panes so labels stand out against a clean
+ # background. Scoped to this renderer only โ€” other plots keep their grids.
+ ax.grid(False)
+ for pane in (ax.xaxis.pane, ax.yaxis.pane, ax.zaxis.pane):
+ pane.set_visible(False)
+ ax.set_axis_off()
+
+ segs = np.zeros((len(edges), 2, 3), dtype=np.float32)
+ segs[:, 0] = pts[edges[:, 0]]
+ segs[:, 1] = pts[edges[:, 1]]
+ ax.add_collection3d(Line3DCollection(
+ segs, colors='dimgray', linewidths=bone_linewidth, zorder=5,
+ ))
+
+ # Root drawn separately (red, larger) so skeleton origin is obvious.
+ parents_arr = np.asarray(parents)
+ non_root = parents_arr != -1
+ root = ~non_root
+ ax.scatter(pts[non_root, 0], pts[non_root, 1], pts[non_root, 2],
+ s=joint_marker_size, c='steelblue',
+ edgecolors='black', linewidths=0.8, depthshade=False, zorder=8)
+ ax.scatter(pts[root, 0], pts[root, 1], pts[root, 2],
+ s=joint_marker_size * 1.8, c='red',
+ edgecolors='black', linewidths=1.2, depthshade=False, zorder=9)
+
+ _annotate_joints(ax, pts, joint_names, bounds, font_size, parents=parents)
+ return _capture_rgb(canvas)
+
+
+def render_skeleton_tpos_spectral(parents: Sequence[int], positions,
+ spectral_feats,
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (8, 8),
+ dpi: int = 120) -> np.ndarray:
+ """Render T-pose with joints PCA-colored from (J, K) spectral features."""
+ pts = _as_xyz(positions).reshape(-1, 3)
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)
+ edges = _edges_from_parents(parents)
+ bounds = _cubic_bounds(pts)
+
+ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)
+ colors = _spectral_to_rgb(spectral_feats)
+
+ for child, parent in edges:
+ ax.plot([pts[child, 0], pts[parent, 0]],
+ [pts[child, 1], pts[parent, 1]],
+ [pts[child, 2], pts[parent, 2]],
+ color='black', linewidth=2.0, alpha=0.8, zorder=1)
+
+ ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2],
+ c=colors, s=80, edgecolors='k', linewidths=0.5,
+ zorder=2, depthshade=False)
+
+ if title:
+ fig.suptitle(title, fontsize=12, fontweight='bold', y=0.95)
+ return _capture_rgb(canvas)
+
+
+def render_skeleton_spectral_grid(parents: Sequence[int], positions,
+ spectral_feats,
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Optional[Tuple[float, float]] = None,
+ cell_size: float = 5.0,
+ dpi: int = 120) -> np.ndarray:
+ """Grid of skeleton subplots, one per eigenvector frequency (coolwarm).
+
+ ``figsize`` overrides the full figure size; when ``None``, the figure is
+ sized as ``(cell_size * ncols, cell_size * nrows)``.
+ """
+ pts = _as_xyz(positions).reshape(-1, 3)
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)
+ K = spectral_feats.shape[1]
+ edges = _edges_from_parents(parents)
+ bounds = _tight_bounds(pts, pad_ratio=0.10)
+ x_min, x_max, y_min, y_max, z_min, z_max = bounds
+ box_aspect = _box_aspect_from_bounds(bounds)
+
+ ncols = min(K, 4)
+ nrows = (K + ncols - 1) // ncols
+
+ if figsize is None:
+ figsize = (cell_size * ncols, cell_size * nrows)
+ fig = Figure(figsize=figsize, dpi=dpi)
+ canvas = FigureCanvasAgg(fig)
+
+ top_margin = 0.04 if title else 0.01
+ usable_h = 1.0 - top_margin
+
+ for k in range(K):
+ row, col = divmod(k, ncols)
+ cell_w = 1.0 / ncols
+ cell_h = usable_h / nrows
+ x0 = col * cell_w
+ y0 = 1.0 - top_margin - (row + 1) * cell_h
+ ax = fig.add_axes([x0, y0, cell_w, cell_h], projection='3d',
+ computed_zorder=False)
+ ax.set_xlim([x_min, x_max])
+ ax.set_ylim([y_min, y_max])
+ ax.set_zlim([z_min, z_max])
+ ax.set_box_aspect(box_aspect)
+ ax.set_xticklabels([]); ax.set_yticklabels([]); ax.set_zticklabels([])
+ ax.xaxis.set_major_locator(LinearLocator(6))
+ ax.yaxis.set_major_locator(LinearLocator(6))
+ ax.zaxis.set_major_locator(LinearLocator(6))
+ ax.tick_params(axis='both', length=0)
+ ax.grid(True, color='gray', linestyle='--', linewidth=0.4)
+ if elev is not None and azim is not None:
+ ax.view_init(elev=elev, azim=azim)
+
+ for child, parent in edges:
+ ax.plot([pts[child, 0], pts[parent, 0]],
+ [pts[child, 1], pts[parent, 1]],
+ [pts[child, 2], pts[parent, 2]],
+ color='black', linewidth=1.5, alpha=0.9, zorder=1)
+
+ vals = spectral_feats[:, k]
+ vmax = max(np.abs(vals).max(), 1e-8)
+ sc = ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2],
+ c=vals, cmap='coolwarm', vmin=-vmax, vmax=vmax,
+ s=50, edgecolors='k', linewidths=0.4,
+ zorder=2, depthshade=False)
+ fig.colorbar(sc, ax=ax, shrink=0.5, pad=0.0, aspect=15)
+ ax.set_title(f'Frequency {k + 1}', fontsize=11, fontweight='bold', pad=0)
+
+ if title:
+ fig.suptitle(title, fontsize=14, fontweight='bold',
+ y=1.0 - top_margin * 0.3)
+ return _capture_rgb(canvas)
+
+
+def render_skeleton_motion(parents: Sequence[int], positions,
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> np.ndarray:
+ """Render a motion sequence as (T, H, W, 3). Reuses artists across frames."""
+ pts = _as_xyz(positions)
+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ T = pts.shape[0]
+
+ edges = _edges_from_parents(parents)
+ bounds = _cubic_bounds(pts)
+ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)
+ if title:
+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.98)
+ fig.subplots_adjust(top=0.90)
+
+ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')
+ segs, lc, sc = _init_motion_artists(ax, edges, pts[0], joint_colors,
+ joint_size=10.0)
+ sc_root = ax.scatter(pts[0, 0:1, 0], pts[0, 0:1, 1], pts[0, 0:1, 2],
+ s=15, c='r')
+
+ frames: List[np.ndarray] = []
+ for t in range(T):
+ P = pts[t]
+ segs[:, 0] = P[edges[:, 0]]
+ segs[:, 1] = P[edges[:, 1]]
+ lc.set_segments(segs)
+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])
+ sc_root._offsets3d = (P[0:1, 0], P[0:1, 1], P[0:1, 2])
+ frames.append(_capture_rgb(canvas))
+
+ return np.stack(frames, axis=0)
+
+
+def render_skeleton_motion_spectral(parents: Sequence[int], positions,
+ spectral_feats,
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ joint_marker_size: float = 30.0,
+ bone_linewidth: float = 1.5,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> np.ndarray:
+ """Motion sequence with joints PCA-colored from spectral features (fixed across frames)."""
+ pts = _as_xyz(positions)
+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ T, J, _ = pts.shape
+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)
+ assert spectral_feats.shape[0] == J, (
+ f"spectral_feats has {spectral_feats.shape[0]} joints but positions "
+ f"has {J} joints"
+ )
+
+ edges = _edges_from_parents(parents)
+ bounds = _cubic_bounds(pts)
+ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)
+ if title:
+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.98)
+ fig.subplots_adjust(top=0.90)
+
+ colors = _spectral_to_rgb(spectral_feats)
+ segs, lc, sc = _init_motion_artists(
+ ax, edges, pts[0], colors,
+ joint_size=joint_marker_size, bone_linewidth=bone_linewidth,
+ joint_edgecolors='k', joint_edgewidth=0.4,
+ )
+
+ frames: List[np.ndarray] = []
+ for t in range(T):
+ P = pts[t]
+ segs[:, 0] = P[edges[:, 0]]
+ segs[:, 1] = P[edges[:, 1]]
+ lc.set_segments(segs)
+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])
+ frames.append(_capture_rgb(canvas))
+
+ return np.stack(frames, axis=0)
+
+
+def render_skeleton_motion_directed(parents: Sequence[int], positions,
+ directions,
+ elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ arrow_scale: float = 0.25,
+ arrow_color: str = 'green',
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> np.ndarray:
+ """Motion sequence with a per-frame orientation arrow anchored at the root.
+
+ ``directions`` is (3,) (constant) or (T, 3). Zero-length vectors render
+ no arrow for that frame. ``arrow_scale`` is a fraction of the scene's
+ cubic-bound side length.
+ """
+ pts = _as_xyz(positions)
+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ T = pts.shape[0]
+
+ dirs = np.asarray(directions, dtype=np.float32)
+ if dirs.ndim == 1:
+ dirs = np.broadcast_to(dirs, (T, 3)).copy()
+ assert dirs.shape == (T, 3), (
+ f"directions must be (3,) or (T, 3) with T={T}, got {dirs.shape}"
+ )
+ dirs = _maybe_rotate_y_up(dirs, rotate_root)
+ norms = np.linalg.norm(dirs, axis=-1, keepdims=True)
+ unit_dirs = np.where(norms > 1e-8, dirs / np.maximum(norms, 1e-8), 0.0)
+
+ edges = _edges_from_parents(parents)
+ bounds = _cubic_bounds(pts)
+ x_min, x_max, y_min, y_max, z_min, z_max = bounds
+ side = max(x_max - x_min, y_max - y_min, z_max - z_min)
+ arrow_len = side * arrow_scale
+
+ _, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)
+
+ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')
+ segs, lc, sc = _init_motion_artists(ax, edges, pts[0], joint_colors,
+ joint_size=10.0)
+ sc_root = ax.scatter(pts[0, 0:1, 0], pts[0, 0:1, 1], pts[0, 0:1, 2],
+ s=15, c='r')
+
+ # 3D quiver can't be mutated in place; remove+re-add each frame.
+ quiver = None
+ frames: List[np.ndarray] = []
+ for t in range(T):
+ P = pts[t]
+ segs[:, 0] = P[edges[:, 0]]
+ segs[:, 1] = P[edges[:, 1]]
+ lc.set_segments(segs)
+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])
+ sc_root._offsets3d = (P[0:1, 0], P[0:1, 1], P[0:1, 2])
+
+ if quiver is not None:
+ quiver.remove()
+ quiver = None
+ d = unit_dirs[t]
+ if np.any(d != 0.0):
+ root = P[0]
+ quiver = ax.quiver(
+ root[0], root[1], root[2],
+ d[0], d[1], d[2],
+ length=arrow_len, normalize=False,
+ color=arrow_color, linewidth=2.0,
+ arrow_length_ratio=0.25, zorder=11,
+ )
+ frames.append(_capture_rgb(canvas))
+
+ return np.stack(frames, axis=0)
+
+
+_CHECKER_COLORS = ((0.93, 0.93, 0.93), (0.885, 0.885, 0.885))
+
+
+def _checker_platform_local(platform_half: float, n_tiles: int, z: float):
+ """A RIGID checker platform in platform-local coordinates.
+
+ Returns ``(quads, colors)`` for an ``n_tiles`` ร— ``n_tiles`` board
+ centered on the origin. The pattern is glued to the platform: the whole
+ board is translated rigidly per frame, so nothing about the ground ever
+ changes on screen (world-anchored tiles clipped to a sliding window
+ made the edge tiles boil). Keeping the board small and near the camera
+ also avoids matplotlib's broken projection of far/behind-camera quads.
+ """
+ tile = 2.0 * platform_half / n_tiles
+ quads, colors = [], []
+ for i in range(n_tiles):
+ xa = -platform_half + i * tile
+ for j in range(n_tiles):
+ ya = -platform_half + j * tile
+ quads.append([(xa, ya, z), (xa + tile, ya, z),
+ (xa + tile, ya + tile, z), (xa, ya + tile, z)])
+ colors.append(_CHECKER_COLORS[(i + j) % 2])
+ return np.asarray(quads, dtype=np.float32), colors
+
+
+def _checker_tiles_window(cx: float, cy: float, view_half: float,
+ tile: float, z: float, margin_tiles: int = 2):
+ """WHOLE world-anchored tiles covering the view window plus a margin.
+
+ Every tile is drawn complete (never clipped), and the drawn set only
+ changes by whole tiles entering/leaving beyond the visible edge โ€” so a
+ sliding camera sees a rigid floor scrolling underfoot with no edge
+ boiling. The margin keeps the set's boundary off screen while staying
+ small enough to avoid matplotlib's broken far/behind-camera projection.
+ """
+ i0 = int(np.floor((cx - view_half) / tile)) - margin_tiles
+ i1 = int(np.floor((cx + view_half) / tile)) + margin_tiles
+ j0 = int(np.floor((cy - view_half) / tile)) - margin_tiles
+ j1 = int(np.floor((cy + view_half) / tile)) + margin_tiles
+ quads, colors = [], []
+ for i in range(i0, i1 + 1):
+ xa = i * tile
+ for j in range(j0, j1 + 1):
+ ya = j * tile
+ quads.append([(xa, ya, z), (xa + tile, ya, z),
+ (xa + tile, ya + tile, z), (xa, ya + tile, z)])
+ colors.append(_CHECKER_COLORS[(i + j) % 2])
+ return quads, colors
+
+
+def _smooth_path(path: np.ndarray, window: int) -> np.ndarray:
+ """Moving-average smoothing with edge padding; kills gait sway so the
+ follow camera and platform glide instead of bobbing with each step."""
+ if window <= 1 or len(path) < 3:
+ return path
+ kernel = np.ones(window, dtype=np.float32) / window
+ pad = window // 2
+ padded = np.pad(path, ((pad, window - 1 - pad), (0, 0)), mode='edge')
+ return np.stack([np.convolve(padded[:, d], kernel, mode='valid')
+ for d in range(path.shape[1])], axis=1)
+
+
+def render_skeleton_motion_ground(parents: Sequence[int], positions,
+ spectral_feats=None,
+ elev: float = 20,
+ azim: float = -60,
+ rotate_root: bool = True,
+ follow: bool = True,
+ title: Optional[str] = None,
+ joint_marker_size: float = 14.0,
+ bone_linewidth: float = 1.6,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> np.ndarray:
+ """Motion render with a checkerboard ground plane and root trajectory;
+ the camera follows the subject.
+
+ The ground sits at the motion's lowest point, the view window is sized to
+ the subject (not the whole trajectory, so a long walk doesn't shrink the
+ character to a speck), and each frame re-centers the window on the root โ€”
+ the checker tiles scrolling past convey world-space travel. Joints use
+ the spectral PCA palette when ``spectral_feats`` is given, else the
+ default root-red / joint-blue scheme.
+ """
+ pts = _as_xyz(positions)
+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"
+ pts = _maybe_rotate_y_up(pts, rotate_root)
+ T, J, _ = pts.shape
+
+ edges = _edges_from_parents(parents)
+ ground_z = float(pts[..., 2].min())
+ root_xy = pts[:, 0, :2] # (T, 2)
+
+ # Window half-extent from the subject's own size, not the trajectory.
+ radius_xy = float(np.abs(pts[..., :2] - root_xy[:, None, :]).max())
+ height = float(pts[..., 2].max() - ground_z)
+ half = max(radius_xy * 1.08, height * 0.52, 1e-3)
+
+ # Coordinated motion: the camera GLIDES after the subject (smoothed root
+ # path) while the ground stays fixed in the WORLD โ€” subject, ground and
+ # camera all move together, and world travel reads off the stage sliding
+ # through the frame plus the trail. The ground itself has two forms:
+ # stage โ€” modest travel: one rigid bounded platform covering the
+ # whole trajectory, built once, world-fixed.
+ # scroll โ€” extreme travel: world-anchored whole-tile floor windowed
+ # around the camera (a trajectory-sized stage would need far
+ # too many quads and far-away quads project badly).
+ traj_center = 0.5 * (root_xy.max(axis=0) + root_xy.min(axis=0))
+ traj_half = float((root_xy.max(axis=0) - root_xy.min(axis=0)).max()) / 2.0
+ stage_half = traj_half + half * 1.05
+ scroll_mode = stage_half > 4.0 * half
+ view_half = half if follow else max(stage_half, half)
+ centers = _smooth_path(root_xy, window=max(5, T // 12)) if follow \
+ else np.repeat(traj_center[None, :], T, axis=0)
+ center0 = centers[0]
+
+ # Vertical extent hugs the subject (tall bipeds and flat crawlers both
+ # fill the frame); the box aspect mirrors the ranges to avoid distortion.
+ z_top = ground_z + max(height * 1.18, view_half * 0.3)
+ bounds = (center0[0] - view_half, center0[0] + view_half,
+ center0[1] - view_half, center0[1] + view_half,
+ ground_z, z_top)
+ fig, canvas, ax = _new_figure(
+ bounds, elev, azim, figsize=figsize, dpi=dpi,
+ box_aspect=(1.0, 1.0, (z_top - ground_z) / (2 * view_half)))
+ ax.set_axis_off()
+ # Fill the canvas: with the axes cage hidden there is nothing to clip,
+ # so spill the 3D box past the figure edges.
+ fig.subplots_adjust(left=-0.22, right=1.22, top=1.18, bottom=-0.18)
+ if title:
+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.97)
+ fig.subplots_adjust(top=1.02)
+
+ from mpl_toolkits.mplot3d.art3d import Poly3DCollection
+ # Tile size from BOTH body dimensions โ€” the geometric mean of height
+ # and horizontal span: an upright biped and a sprawling low body each
+ # get tiles proportioned to their overall bulk, and neither dimension
+ # alone can blow the scale up. View-based clamps keep it sane on
+ # degenerate shapes.
+ body_span = 2.0 * radius_xy
+ tile = float(np.clip(0.35 * np.sqrt(max(height * body_span, 1e-12)),
+ half / 6.0, half / 2.0))
+ if scroll_mode:
+ # World-anchored whole tiles, refreshed per frame around the camera.
+ platform_half = 10 * view_half # trail clip bound (window handles it)
+ quads, cols = _checker_tiles_window(*center0, view_half, tile, ground_z)
+ platform = Poly3DCollection(quads, facecolors=cols,
+ edgecolors='none', zorder=0)
+ ax.add_collection3d(platform)
+ else:
+ # One rigid bounded stage, world-fixed, tiles glued to it. The
+ # following camera slides over it; the stage edges sweeping through
+ # the frame are what makes the travel visible.
+ platform_half = stage_half
+ n_tiles = int(np.clip(round(2 * platform_half / tile), 4, 16))
+ local_quads, quad_colors = _checker_platform_local(
+ platform_half, n_tiles=n_tiles, z=ground_z)
+ platform = Poly3DCollection(local_quads + np.array(
+ [traj_center[0], traj_center[1], 0.0], dtype=np.float32),
+ facecolors=quad_colors, edgecolors='none', zorder=0)
+ ax.add_collection3d(platform)
+
+ if spectral_feats is not None:
+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)
+ assert spectral_feats.shape[0] == J
+ joint_colors = _spectral_to_rgb(spectral_feats)
+ edge_kwargs = dict(joint_edgecolors='k', joint_edgewidth=0.4)
+ else:
+ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')
+ edge_kwargs = {}
+
+ # The trajectory sits just above the tiles to avoid z-fighting.
+ eps = half * 4e-3
+ trail, = ax.plot(root_xy[:1, 0], root_xy[:1, 1], [ground_z + eps],
+ color='tab:orange', linewidth=1.6, alpha=0.9, zorder=1)
+
+ segs, lc, sc = _init_motion_artists(
+ ax, edges, pts[0], joint_colors,
+ joint_size=joint_marker_size, bone_color='dimgray',
+ bone_linewidth=bone_linewidth, **edge_kwargs)
+ # The root must never be occluded: inside a single scatter, points paint
+ # in array order, so joints drawn after index 0 cover it. Re-draw the
+ # root as its own topmost scatter (same size; red in the default
+ # palette, its own spectral color otherwise).
+ root_color = ([joint_colors[0]] if spectral_feats is not None else 'r')
+ sc_root = ax.scatter(pts[0, 0:1, 0], pts[0, 0:1, 1], pts[0, 0:1, 2],
+ s=joint_marker_size, c=root_color,
+ depthshade=False, zorder=10, **(
+ dict(edgecolors='k', linewidths=0.4)
+ if spectral_feats is not None else {}))
+
+ frames: List[np.ndarray] = []
+ for t in range(T):
+ P = pts[t]
+ segs[:, 0] = P[edges[:, 0]]
+ segs[:, 1] = P[edges[:, 1]]
+ lc.set_segments(segs)
+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])
+ sc_root._offsets3d = (P[0:1, 0], P[0:1, 1], P[0:1, 2])
+ cx, cy = centers[t]
+ tx = root_xy[:t + 1, 0].astype(np.float32).copy()
+ ty = root_xy[:t + 1, 1].astype(np.float32).copy()
+ off = (np.abs(tx - traj_center[0]) > platform_half) | \
+ (np.abs(ty - traj_center[1]) > platform_half)
+ tx[off] = np.nan
+ trail.set_data_3d(tx, ty, np.full(t + 1, ground_z + eps))
+ if scroll_mode:
+ quads, cols = _checker_tiles_window(cx, cy, view_half, tile,
+ ground_z)
+ platform.set_verts(quads)
+ platform.set_facecolor(cols)
+ if follow:
+ ax.set_xlim(cx - view_half, cx + view_half)
+ ax.set_ylim(cy - view_half, cy + view_half)
+ frames.append(_capture_rgb(canvas))
+
+ return np.stack(frames, axis=0)
+
+
+# ============================================================================
+# Savers: write PNG / MP4 to disk
+# ============================================================================
+
+def _write_image(save_path: str, frame: np.ndarray) -> None:
+ Path(save_path).parent.mkdir(parents=True, exist_ok=True)
+ imageio.imwrite(save_path, frame)
+
+
+def _write_video(save_path: str, frames: np.ndarray, fps: int) -> None:
+ Path(save_path).parent.mkdir(parents=True, exist_ok=True)
+ imageio.mimwrite(save_path, frames, fps=fps)
+
+
+# Defaults (elev=30, azim=-60, rotate_root=True) match the pipeline convention.
+
+def save_skeleton_tpose(save_path: str, parents: Sequence[int], positions,
+ elev: float = 30, azim: float = -60,
+ rotate_root: bool = True,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> None:
+ """Render a T-pose and save as PNG."""
+ _write_image(save_path, render_skeleton_tpose(
+ parents, positions, elev=elev, azim=azim, rotate_root=rotate_root,
+ figsize=figsize, dpi=dpi,
+ ))
+
+
+def save_skeleton_tpose_annotated(save_path: str, parents: Sequence[int],
+ positions, joint_names: Sequence[str],
+ elev: float = 30, azim: float = -60,
+ rotate_root: bool = True,
+ font_size: int = 6,
+ joint_marker_size: float = 80.0,
+ bone_linewidth: float = 2.0,
+ figsize: Tuple[float, float] = (12, 12),
+ dpi: int = 120) -> None:
+ """Render an annotated T-pose and save as PNG."""
+ _write_image(save_path, render_skeleton_tpose_annotated(
+ parents, positions, joint_names,
+ elev=elev, azim=azim, rotate_root=rotate_root, font_size=font_size,
+ joint_marker_size=joint_marker_size, bone_linewidth=bone_linewidth,
+ figsize=figsize, dpi=dpi,
+ ))
+
+
+def save_skeleton_tpos_spectral(save_path: str, parents: Sequence[int], positions,
+ spectral_feats,
+ elev: float = 30, azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (8, 8),
+ dpi: int = 120) -> None:
+ """Render spectral-colored T-pose and save as PNG."""
+ _write_image(save_path, render_skeleton_tpos_spectral(
+ parents, positions, spectral_feats,
+ elev=elev, azim=azim, rotate_root=rotate_root, title=title,
+ figsize=figsize, dpi=dpi,
+ ))
+
+
+def save_skeleton_spectral_grid(save_path: str, parents: Sequence[int],
+ positions, spectral_feats,
+ elev: float = 30, azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Optional[Tuple[float, float]] = None,
+ cell_size: float = 5.0,
+ dpi: int = 120) -> None:
+ """Render per-frequency spectral grid and save as PNG."""
+ _write_image(save_path, render_skeleton_spectral_grid(
+ parents, positions, spectral_feats,
+ elev=elev, azim=azim, rotate_root=rotate_root, title=title,
+ figsize=figsize, cell_size=cell_size, dpi=dpi,
+ ))
+
+
+def save_skeleton_motion(save_path: str, parents: Sequence[int], positions,
+ fps: int = 30, elev: float = 30, azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> None:
+ """Render a motion sequence and save as MP4."""
+ frames = render_skeleton_motion(
+ parents, positions, elev=elev, azim=azim, rotate_root=rotate_root,
+ title=title, figsize=figsize, dpi=dpi,
+ )
+ _write_video(save_path, frames, fps)
+
+
+def save_skeleton_motion_spectral(save_path: str, parents: Sequence[int],
+ positions, spectral_feats,
+ fps: int = 30, elev: float = 30,
+ azim: float = -60,
+ rotate_root: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> None:
+ """Render a spectral-colored motion sequence and save as MP4."""
+ frames = render_skeleton_motion_spectral(
+ parents, positions, spectral_feats,
+ elev=elev, azim=azim, rotate_root=rotate_root, title=title,
+ figsize=figsize, dpi=dpi,
+ )
+ _write_video(save_path, frames, fps)
+
+
+def save_skeleton_motion_ground(save_path: str, parents: Sequence[int],
+ positions, spectral_feats=None,
+ fps: int = 20, elev: float = 20,
+ azim: float = -60, rotate_root: bool = True,
+ follow: bool = True,
+ title: Optional[str] = None,
+ figsize: Tuple[float, float] = (6, 6),
+ dpi: int = 120) -> None:
+ """Render and save a ground-plane motion MP4 (see
+ :func:`render_skeleton_motion_ground`)."""
+ frames = render_skeleton_motion_ground(
+ parents, positions, spectral_feats=spectral_feats, elev=elev,
+ azim=azim, rotate_root=rotate_root, follow=follow, title=title,
+ figsize=figsize, dpi=dpi)
+ _write_video(save_path, frames, fps)
diff --git a/data_process/utils/skeleton.py b/data_process/utils/skeleton.py
new file mode 100644
index 0000000..ac1ca5a
--- /dev/null
+++ b/data_process/utils/skeleton.py
@@ -0,0 +1,533 @@
+"""Skeleton processing utilities.
+
+Topology-agnostic helpers that operate on ``Animation`` objects: BFS joint
+reordering, geometric measurements, uniform scaling, root alignment,
+facing-axis alignment and motion activity metrics.
+
+Naming conventions:
+ get_* โ€” pure computations that return a value (no mutation).
+ scale_* โ€” uniform scaling transforms returning a new Animation.
+ align_* โ€” positional / rotational alignment transforms returning a
+ new Animation (together with the offset/quaternion applied).
+"""
+
+import statistics
+from collections import deque
+
+import numpy as np
+from Quaternions import Quaternions
+from Animation import Animation, positions_global, offset_lengths
+
+
+# ---------------------------------------------------------------------------
+# BFS joint reordering
+# ---------------------------------------------------------------------------
+
+def bfs_reorder_joints(parents, offsets=None):
+ """Compute a canonical breadth-first ordering of joints starting from the root.
+
+ Children sorted by subtree size (descending), bone length as tiebreaker (ascending).
+ """
+ njoints = len(parents)
+
+ if offsets is not None:
+ bone_lengths = np.linalg.norm(offsets, axis=-1)
+ else:
+ bone_lengths = None
+
+ # Build children adjacency and find root.
+ children = [[] for _ in range(njoints)]
+ root = None
+ for j, p in enumerate(parents):
+ if p == -1:
+ root = j
+ else:
+ children[p].append(j)
+ assert root is not None, "No root joint found (parent == -1)"
+
+ # Compute subtree sizes via BFS topological order (bottom-up accumulation).
+ subtree_size = [1] * njoints
+ topo = []
+ q = deque([root])
+ while q:
+ node = q.popleft()
+ topo.append(node)
+ q.extend(children[node])
+ for node in reversed(topo):
+ for child in children[node]:
+ subtree_size[node] += subtree_size[child]
+
+ # Sort children: largest subtree first, shortest bone as tiebreaker.
+ for node in range(njoints):
+ if children[node]:
+ children[node].sort(
+ key=lambda idx: (-subtree_size[idx],
+ bone_lengths[idx] if bone_lengths is not None else 0))
+
+ # Final BFS traversal with sorted children โ†’ canonical ordering.
+ bfs_order = []
+ q = deque([root])
+ while q:
+ node = q.popleft()
+ bfs_order.append(node)
+ q.extend(children[node])
+
+ assert len(bfs_order) == njoints
+
+ # Build oldโ†’new index mapping and remap parents.
+ old_to_new = [0] * njoints
+ for new_idx, old_idx in enumerate(bfs_order):
+ old_to_new[old_idx] = new_idx
+
+ new_parents = []
+ for new_idx, old_idx in enumerate(bfs_order):
+ old_parent = parents[old_idx]
+ new_parents.append(-1 if old_parent == -1 else old_to_new[old_parent])
+
+ return bfs_order, new_parents
+
+
+def reorder_anim_bfs(anim):
+ """Reorder an Animation's joints into BFS order."""
+ parents_list = anim.parents.tolist() if hasattr(anim.parents, 'tolist') else list(anim.parents)
+ offsets = np.array(anim.offsets) if hasattr(anim, 'offsets') else None
+ bfs_order, new_parents = bfs_reorder_joints(parents_list, offsets=offsets)
+
+ if bfs_order == list(range(len(parents_list))):
+ return anim, bfs_order, parents_list
+
+ reordered = anim[:, bfs_order]
+ return reordered, bfs_order, new_parents
+
+
+# ---------------------------------------------------------------------------
+# Geometry: skeleton diameter
+# ---------------------------------------------------------------------------
+
+def get_skeleton_diameter(anim):
+ """Return the leaf-to-leaf geodesic diameter of a skeleton.
+
+ Uses the classic two-pass BFS trick on the bone-length-weighted skeleton
+ tree:
+ 1. BFS from the root to find an extreme leaf ``u``.
+ 2. BFS from ``u`` to find the farthest node โ€” that distance is the
+ tree diameter.
+
+ The BFS formulation is robust to arbitrary root placements and
+ non-humanoid topologies (multi-limb creatures, snakes, etc.).
+
+ Args:
+ anim: Animation object whose ``parents`` / ``offsets`` define the
+ rest-pose skeleton.
+
+ Returns:
+ Diameter as a float. Returns 0.0 for degenerate single-joint skeletons.
+ """
+ parents = anim.parents
+ offsets = anim.offsets
+ n_joints = len(parents)
+
+ # Build a bidirectional, bone-length-weighted adjacency list.
+ adj = {i: [] for i in range(n_joints)}
+ bone_lengths = np.linalg.norm(offsets, axis=-1)
+ for i, p in enumerate(parents):
+ if p != -1:
+ adj[p].append((i, bone_lengths[i]))
+ adj[i].append((p, bone_lengths[i]))
+
+ def _bfs_farthest(start):
+ """BFS returning (farthest_node, distance) from ``start``."""
+ distances = {start: 0.0}
+ visited = {start}
+ queue = [start]
+ farthest_node = start
+ max_dist = 0.0
+
+ ptr = 0
+ while ptr < len(queue):
+ u = queue[ptr]
+ ptr += 1
+ for v, weight in adj[u]:
+ if v in visited:
+ continue
+ visited.add(v)
+ dist = distances[u] + weight
+ distances[v] = dist
+ queue.append(v)
+ if dist > max_dist:
+ max_dist = dist
+ farthest_node = v
+ return farthest_node, max_dist
+
+ u, _ = _bfs_farthest(0) # pass 1: one extreme leaf
+ _, diameter = _bfs_farthest(u) # pass 2: farthest from u โ†’ diameter
+ return diameter
+
+
+# ---------------------------------------------------------------------------
+# Uniform scaling
+# ---------------------------------------------------------------------------
+
+def scale_anim(anim, scale_factor):
+ """Return a copy of *anim* with positions and offsets scaled uniformly."""
+ return Animation(
+ rotations=anim.rotations.copy(),
+ positions=anim.positions.copy() * scale_factor,
+ orients=anim.orients.copy(),
+ offsets=anim.offsets.copy() * scale_factor,
+ parents=anim.parents.copy(),
+ )
+
+
+def scale_by_diameter(anim, scale_factor=None, target_diameter=2.0):
+ """Scale an animation so that its leaf-to-leaf diameter matches a target.
+
+ Args:
+ anim: Animation to scale.
+ scale_factor: Explicit scale. If None, computed as
+ ``target_diameter / get_skeleton_diameter(anim)``.
+ target_diameter: Desired diameter (default 2.0).
+
+ Returns:
+ Tuple of (scaled Animation, scale_factor applied).
+ """
+ if scale_factor is None:
+ scale_factor = target_diameter / get_skeleton_diameter(anim)
+ return scale_anim(anim, scale_factor), scale_factor
+
+
+def scale_by_bone_length(anim, scale_factor=None, target_bone_len=None):
+ """Scale an animation so that the mean bone length matches a target.
+
+ If ``scale_factor`` is not provided it is derived as
+ ``target_bone_len / mean(offset_lengths(anim))``. Rotations are untouched.
+
+ Args:
+ anim: Animation to scale.
+ scale_factor: Pre-computed scale. If None, derived from *target_bone_len*.
+ target_bone_len: Desired mean bone length (used only when *scale_factor*
+ is None).
+
+ Returns:
+ Tuple of (scaled Animation, scale_factor applied).
+ """
+ if scale_factor is None:
+ mean_len = statistics.mean(offset_lengths(anim))
+ scale_factor = target_bone_len / mean_len
+ return scale_anim(anim, scale_factor), scale_factor
+
+
+# ---------------------------------------------------------------------------
+# Root alignment (translation)
+# ---------------------------------------------------------------------------
+
+def _translated_anim(anim, new_positions, new_offsets):
+ """Return a copy of *anim* with replacement positions/offsets."""
+ return Animation(
+ rotations=anim.rotations.copy(),
+ positions=new_positions,
+ orients=anim.orients.copy(),
+ offsets=new_offsets,
+ parents=anim.parents.copy(),
+ )
+
+
+def align_root_xz_to_origin(anim, root_pose_init_xz=None):
+ """Translate the skeleton so the root's XZ position is at the origin at frame 0.
+
+ Only the horizontal (XZ) components are subtracted โ€” the absolute Y height
+ is preserved.
+
+ Args:
+ anim: Animation to translate.
+ root_pose_init_xz: Pre-computed XZ offset (3,) with Y component zero.
+ If None, auto-detected from the frame-0 root position.
+
+ Returns:
+ Tuple of (translated Animation, root_pose_init_xz actually subtracted).
+ """
+ if root_pose_init_xz is None:
+ # Zero-out the Y component so only XZ is subtracted.
+ root_pose_init_xz = anim.positions[0, 0] * np.array([1, 0, 1])
+
+ new_positions = anim.positions.copy()
+ new_positions[:, 0] -= root_pose_init_xz
+ new_offsets = anim.offsets.copy()
+ new_offsets[0] -= root_pose_init_xz
+
+ return _translated_anim(anim, new_positions, new_offsets), root_pose_init_xz
+
+
+def align_root_to_ground(anim, ground_height=None):
+ """Translate the skeleton vertically so the lowest joint sits on Y=0.
+
+ If ``ground_height`` is not provided, it is the global minimum Y across
+ all joints and frames (computed via forward kinematics).
+
+ Args:
+ anim: Animation to translate.
+ ground_height: Pre-computed Y offset. If None, auto-detected.
+
+ Returns:
+ Tuple of (translated Animation, ground_height subtracted).
+ """
+ if ground_height is None:
+ global_positions = positions_global(anim)
+ ground_height = global_positions.min(axis=0).min(axis=0)[1]
+
+ new_positions = anim.positions.copy()
+ new_positions[:, 0, 1] -= ground_height
+ new_offsets = anim.offsets.copy()
+ new_offsets[0, 1] -= ground_height
+
+ return _translated_anim(anim, new_positions, new_offsets), ground_height
+
+
+# ---------------------------------------------------------------------------
+# Root orientation alignment (facing axis)
+# ---------------------------------------------------------------------------
+
+def _normalize(v, eps=1e-8):
+ """Return ``v`` normalized along its last axis with a numerical floor."""
+ norm = np.sqrt((v ** 2).sum(axis=-1, keepdims=True))
+ return v / np.maximum(norm, eps)
+
+
+def get_root_facing_quat(joints, face_joint_idxs, body_axis=False):
+ """Compute per-frame root quaternion that rotates the skeleton to face Z+.
+
+ The left-right axis ``across`` is assembled from the given face joint
+ indices:
+
+ - 2 indices ``[r_hip, l_hip]``: ``joints[r_hip] - joints[l_hip]``.
+ - 4 indices ``[r_hip, l_hip, r_shoulder, l_shoulder]``: the sum of the
+ hip and shoulder left-right vectors.
+
+ Forward direction is ``Y_up x across``, and the returned quaternion
+ rotates that forward vector onto Z+. When *body_axis* is True (e.g.
+ snakes), ``across`` is treated as a body-axis (forward) vector rather
+ than a lateral vector: it is rotated by -90ยฐ Y to convert it into a
+ lateral axis before the cross product.
+
+ If ``face_joint_idxs`` is None or the sentinel ``[-1, -1]`` (face joint
+ names unavailable for this skeleton), returns per-frame identity
+ quaternions so the caller can skip facing alignment.
+
+ Args:
+ joints: Global joint positions (F, J, 3).
+ face_joint_idxs: Sequence of 2 or 4 joint indices, or ``[-1, -1]``
+ sentinel. For body-axis creatures, convention is r_hip=head/tongue,
+ l_hip=tail.
+ body_axis: If True, treat the joint vector as body-axis (forward)
+ rather than lateral.
+
+ Returns:
+ Quaternions of shape (F,) aligning the skeleton to face Z+.
+ """
+ if face_joint_idxs is None or (
+ len(face_joint_idxs) >= 2
+ and face_joint_idxs[0] == -1
+ and face_joint_idxs[1] == -1
+ ):
+ return Quaternions.id(len(joints))
+
+ if len(face_joint_idxs) == 4:
+ r_hip, l_hip, sdr_r, sdr_l = face_joint_idxs
+ across = (joints[:, r_hip] - joints[:, l_hip]) + (
+ joints[:, sdr_r] - joints[:, sdr_l]
+ )
+ elif len(face_joint_idxs) == 2:
+ r_hip, l_hip = face_joint_idxs
+ across = joints[:, r_hip] - joints[:, l_hip]
+ else:
+ raise ValueError(
+ f"face_joint_idxs must have 2 or 4 entries, got {len(face_joint_idxs)}"
+ )
+
+ across = _normalize(across)
+ forward = _normalize(np.cross(np.array([[0, 1, 0]]), across, axis=-1))
+
+ if body_axis:
+ rot_correction = Quaternions.from_euler(np.array([0, -np.pi / 2, 0]), "xyz")
+ forward = rot_correction * forward
+
+ target = np.array([[0, 0, 1]]).repeat(len(forward), axis=0)
+ root_quat = Quaternions.between(forward, target)
+
+ return root_quat
+
+
+def align_initial_facing(anim, face_joint_idxs, body_axis=False):
+ """Rotate the entire motion so the skeleton faces Z+ at the first frame.
+
+ Computes the facing-direction quaternion from the first frame using
+ ``get_root_facing_quat`` and applies it to the root joint across all
+ frames. Non-root joints are unaffected at the rotation level (the
+ rotation propagates through the root's orientation at render time).
+
+ When ``face_joint_idxs`` is None or the sentinel ``[-1, -1]`` (face
+ joint names unavailable), the animation is returned unchanged
+ (identity facing rotation).
+
+ Args:
+ anim: Animation object.
+ face_joint_idxs: Indices [r_hip, l_hip] or
+ [r_hip, l_hip, r_shoulder, l_shoulder], or ``[-1, -1]`` sentinel.
+ body_axis: If True, apply an extra -90ยฐ Y correction (e.g. snakes).
+
+ Returns:
+ New Animation with rotated root.
+ """
+ if face_joint_idxs is None or (
+ len(face_joint_idxs) >= 2
+ and face_joint_idxs[0] == -1
+ and face_joint_idxs[1] == -1
+ ):
+ return Animation(
+ rotations=anim.rotations.copy(),
+ positions=anim.positions.copy(),
+ orients=anim.orients.copy(),
+ offsets=anim.offsets.copy(),
+ parents=anim.parents.copy(),
+ )
+
+ global_pos = positions_global(anim)
+ quat = get_root_facing_quat(
+ global_pos, face_joint_idxs, body_axis=body_axis,
+ )[0]
+
+ nframes = anim.shape[0]
+ quat_tiled = quat.repeat(nframes, axis=0)
+
+ new_rots = anim.rotations.copy()
+ new_pos = anim.positions.copy()
+ new_offsets = anim.offsets.copy()
+ new_rots[:, 0] = quat_tiled * new_rots[:, 0]
+ new_pos[:, 0] = quat_tiled * new_pos[:, 0]
+ new_offsets[0] = quat * new_offsets[0]
+
+ return Animation(
+ rotations=new_rots,
+ positions=new_pos,
+ orients=anim.orients.copy(),
+ offsets=new_offsets,
+ parents=anim.parents.copy(),
+ )
+
+
+# ---------------------------------------------------------------------------
+# Full alignment pipeline
+# ---------------------------------------------------------------------------
+
+def canonicalize_anim(anim,
+ face_joint_idxs, body_axis=False,
+ root_pose_init_xz=None, ground_height=None,
+ scale_method="diameter", scale_factor=None, target_scale_metric=None):
+ """Full alignment pipeline: facing -> center XZ -> scale -> ground.
+
+ Each step accepts an optional pre-computed parameter so T-pose calibration
+ values can be reused for motion clips.
+
+ Args:
+ anim: Raw Animation object.
+ face_joint_idxs: Joint indices defining the facing direction.
+ body_axis: If True, apply an extra -90ยฐ Y correction (e.g. snakes).
+ root_pose_init_xz: XZ offset for centering (from T-pose calibration).
+ ground_height: Vertical ground offset (from T-pose calibration).
+ scale_method: ``"diameter"`` (default) or ``"bone_length"``.
+ scale_factor: Uniform scale (from T-pose calibration).
+ target_scale_metric: Target value for the chosen scale method. Interpreted
+ as target diameter when ``scale_method="diameter"`` or target mean bone
+ length when ``scale_method="bone_length"``. Only used when
+ *scale_factor* is None.
+
+ Returns:
+ (aligned_anim, root_pose_init_xz, ground_height, scale_factor).
+ """
+ rotated = align_initial_facing(anim, face_joint_idxs, body_axis=body_axis)
+ centered, root_pose_init_xz = align_root_xz_to_origin(rotated, root_pose_init_xz)
+
+ if scale_method == "diameter":
+ scaled, scale_factor = scale_by_diameter(
+ centered, scale_factor=scale_factor, target_diameter=target_scale_metric or 2.0)
+ elif scale_method == "bone_length":
+ scaled, scale_factor = scale_by_bone_length(
+ centered, scale_factor=scale_factor, target_bone_len=target_scale_metric)
+ else:
+ raise ValueError(f"Unknown scale_method: {scale_method!r}")
+
+ grounded, ground_height = align_root_to_ground(scaled, ground_height)
+ return grounded, root_pose_init_xz, ground_height, scale_factor
+
+
+# ---------------------------------------------------------------------------
+# Motion activity metrics
+# ---------------------------------------------------------------------------
+
+
+def get_motion_activity_metrics(positions, top_k_ratio=0.25, top_k_min=2):
+ """Compute motion activity metrics from joint positions.
+
+ Activity is measured as the **temporal spread** (std over frames) of each
+ joint's position. Non-root joint activity is computed from *root-relative*
+ positions, so it isolates articulation from global translation. Root
+ activity uses the global root position.
+
+ Assumes positions are already at a canonical scale (e.g. via
+ ``scale_by_diameter``), so no bone-length normalization is applied.
+
+ This spread-based formulation is robust to frame rate and to motions that
+ have tiny jitter but no real movement: a jittery-but-static motion has
+ near-zero std even if its frame-to-frame step is nonzero, so thresholding
+ on these metrics reliably filters out inactive clips.
+
+ In addition to the mean over all non-root joints, a top-k mean is returned
+ โ€” the average spread of the most active joints. This catches long-tailed /
+ multi-limb skeletons where only a few joints move (e.g. snake tail tip,
+ single-limb gestures); the mean gets diluted by the many static joints,
+ but the top-k mean stays high. ``k`` scales with skeleton size as
+ ``max(top_k_min, round(top_k_ratio * (J - 1)))`` so it is meaningful for
+ both small skeletons (~10 joints) and large ones (100+ joints).
+
+ Args:
+ positions: Joint global positions (F, J, 3), at canonical scale.
+ top_k_ratio: Fraction of non-root joints to include in the top-k mean
+ (default 0.25 โ†’ top quartile of most-active joints).
+ top_k_min: Minimum number of joints in the top-k window, so tiny
+ skeletons still get a meaningful average (default 2).
+
+ Returns:
+ Dict with keys ``joint_activity``, ``joint_activity_topk`` and
+ ``root_activity`` (floats).
+ """
+ assert positions.ndim == 3, "positions must be (F, J, 3)"
+
+ # Root activity: temporal spread of the global root trajectory.
+ root_pos = positions[:, 0, :] # (F, 3)
+ root_activity = float(np.linalg.norm(root_pos.std(axis=0)))
+
+ # Joint activity: temporal spread of *root-relative* joint positions,
+ # which isolates articulation from global translation.
+ if positions.shape[1] > 1:
+ rel_pos = positions[:, 1:, :] - positions[:, 0:1, :] # (F, J-1, 3)
+ per_joint_std = rel_pos.std(axis=0) # (J-1, 3)
+ per_joint_spread = np.linalg.norm(per_joint_std, axis=-1) # (J-1,)
+ joint_activity = float(per_joint_spread.mean())
+
+ # Top-k mean: robust to long-tailed skeletons where the mean is
+ # diluted by many static joints but a few limbs carry the motion.
+ # k scales with joint count so it stays meaningful across skeleton sizes.
+ n_non_root = per_joint_spread.shape[0]
+ k = max(top_k_min, int(round(top_k_ratio * n_non_root)))
+ k = min(k, n_non_root)
+ top_k_vals = np.partition(per_joint_spread, -k)[-k:]
+ joint_activity_topk = float(top_k_vals.mean())
+ else:
+ joint_activity = 0.0
+ joint_activity_topk = 0.0
+
+ return {
+ "joint_activity": joint_activity,
+ "joint_activity_topk": joint_activity_topk,
+ "root_activity": root_activity,
+ }
diff --git a/data_process/utils/topology.py b/data_process/utils/topology.py
new file mode 100644
index 0000000..0872218
--- /dev/null
+++ b/data_process/utils/topology.py
@@ -0,0 +1,333 @@
+"""Skeleton topology utilities.
+
+All public functions accept a parent index array ``(J,)`` (list or numpy
+array) where ``parents[j]`` is the parent of joint *j* and the root has
+parent ``-1``.
+
+Sections:
+ 1. Graph representations โ€” adjacency, edge index, joint depths
+ 2. Spectral encodings โ€” Laplacian eigenvectors
+ 3. Pairwise relations โ€” edge types, topology distances
+ 4. Kinematic chains
+"""
+
+import bisect
+from collections import deque
+
+import numpy as np
+
+
+# ---------------------------------------------------------------------------
+# Internal helpers
+# ---------------------------------------------------------------------------
+
+def _parse_parents(parents):
+ """Normalize *parents* to a 1-D int64 numpy array."""
+ return np.asarray(parents, dtype=np.int64).ravel()
+
+
+def _children_map(parents):
+ """List-of-lists mapping each joint to its children."""
+ n = len(parents)
+ children = [[] for _ in range(n)]
+ for c, p in enumerate(parents):
+ if p is None or p == -1:
+ continue
+ if not (0 <= p < n):
+ raise ValueError(f"Invalid parent index parents[{c}]={p}")
+ children[p].append(c)
+ return children
+
+
+def _adjacency_list(parents):
+ """Undirected adjacency list built from *parents*."""
+ n = len(parents)
+ adj = [[] for _ in range(n)]
+ for c, p in enumerate(parents):
+ if p is None or p == -1:
+ continue
+ adj[c].append(p)
+ adj[p].append(c)
+ return adj
+
+
+# ---------------------------------------------------------------------------
+# 1. Graph representations
+# ---------------------------------------------------------------------------
+
+def compute_adjacency_matrix(parents, self_loops=True):
+ """Dense undirected adjacency matrix ``(J, J)`` (float32)."""
+ parents = _parse_parents(parents)
+ n = len(parents)
+ adj = np.eye(n, dtype=np.float32) if self_loops else np.zeros((n, n), dtype=np.float32)
+ for child, parent in enumerate(parents):
+ if child == parent or parent < 0:
+ continue
+ adj[child, parent] = 1.0
+ adj[parent, child] = 1.0
+ return adj
+
+
+def compute_edge_indexs(parents):
+ """PyG-style edge index ``[2, 2*(J-1)]`` (int64).
+
+ Undirected: includes both parent->child and child->parent edges.
+ Self-loops are excluded.
+ """
+ parents = _parse_parents(parents)
+ children = np.arange(len(parents))
+ valid = (children != parents) & (parents >= 0)
+ p = parents[valid]
+ c = children[valid]
+ return np.stack([np.concatenate([p, c]), np.concatenate([c, p])], axis=0)
+
+
+def compute_joint_depths(parents):
+ """Depth of each joint in the kinematic tree ``(J,)`` (int64, root=0).
+
+ Handles arbitrary joint ordering (parent index may be > child index).
+ """
+ parents = list(parents)
+ n = len(parents)
+ depths = np.full(n, -1, dtype=np.int64)
+
+ for j in range(n):
+ if parents[j] == -1 or parents[j] == j:
+ depths[j] = 0
+
+ changed = True
+ while changed:
+ changed = False
+ for j in range(n):
+ if depths[j] >= 0:
+ continue
+ p = parents[j]
+ if 0 <= p < n and depths[p] >= 0:
+ depths[j] = depths[p] + 1
+ changed = True
+
+ return depths
+
+
+# ---------------------------------------------------------------------------
+# 2. Spectral encodings
+# ---------------------------------------------------------------------------
+
+def _build_laplacian(parents, norm='none'):
+ """Graph Laplacian ``(J, J)`` of the undirected skeleton graph.
+
+ Args:
+ parents: Already-parsed (J,) int64 array.
+ norm: ``'none'`` (combinatorial ``L = D - A``), ``'sym'``
+ (``I - D^{-1/2} A D^{-1/2}``) or ``'rw'`` (``I - D^{-1} A``).
+ """
+ adj = compute_adjacency_matrix(parents, self_loops=False).astype(np.float64)
+ deg = adj.sum(axis=1)
+ if norm == 'none':
+ return np.diag(deg) - adj
+ with np.errstate(divide='ignore'):
+ if norm == 'sym':
+ d_inv_sqrt = np.where(deg > 0, deg ** -0.5, 0.0)
+ return np.eye(len(deg)) - d_inv_sqrt[:, None] * adj * d_inv_sqrt[None, :]
+ if norm == 'rw':
+ d_inv = np.where(deg > 0, 1.0 / deg, 0.0)
+ return np.eye(len(deg)) - d_inv[:, None] * adj
+ raise ValueError(f"Unsupported laplacian norm: '{norm}'. Choose from 'none', 'sym', 'rw'.")
+
+
+def _normalize_eigvecs(eigvecs, eigvals, norm='L2', eps=1e-12):
+ """Normalize eigenvector columns (``'sign'``, ``'L1'``, ``'L2'``,
+ ``'abs-max'`` or ``'wavelength'``)."""
+ eigvecs = eigvecs.copy()
+
+ if norm == 'sign':
+ for k in range(eigvecs.shape[1]):
+ max_idx = np.argmax(np.abs(eigvecs[:, k]))
+ if eigvecs[max_idx, k] < 0:
+ eigvecs[:, k] *= -1
+ return eigvecs
+
+ for k in range(eigvecs.shape[1]):
+ col = eigvecs[:, k]
+ if norm == 'L1':
+ denom = np.abs(col).sum()
+ elif norm == 'L2':
+ denom = np.sqrt((col ** 2).sum())
+ elif norm == 'abs-max':
+ denom = np.abs(col).max()
+ elif norm == 'wavelength':
+ denom = np.abs(col).max()
+ lam = np.sqrt(max(eigvals[k], eps))
+ denom = denom * lam * 2.0 / np.pi
+ else:
+ raise ValueError(f"Unsupported eigvec norm: '{norm}'. "
+ f"Choose from 'sign', 'L1', 'L2', 'abs-max', 'wavelength'.")
+ if denom > eps:
+ eigvecs[:, k] = col / denom
+
+ return eigvecs
+
+
+def compute_laplacian_eigenvectors(parents, max_freqs=8,
+ laplacian_norm='sym', eigvec_norm='L2'):
+ """Smallest non-trivial eigenvectors of the graph Laplacian.
+
+ Follows the GRIT / GraphGPS decomposition: build the Laplacian, run an
+ eigendecomposition, clamp eigenvalues and normalize eigenvectors.
+
+ Args:
+ parents: Parent indices (J,).
+ max_freqs: Number of frequencies to return (clamped to J-1).
+ laplacian_norm: ``'none'``, ``'sym'`` or ``'rw'``.
+ eigvec_norm: ``'sign'``, ``'L1'``, ``'L2'``, ``'abs-max'`` or
+ ``'wavelength'``.
+
+ Returns:
+ ``(eigvecs (J, max_freqs) float32, eigvals (max_freqs,) float32)``.
+ Skeletons with fewer than ``max_freqs + 1`` joints are zero-padded on
+ the frequency axis so every skeleton yields the same width.
+ """
+ parents = _parse_parents(parents)
+ n = len(parents)
+ k = min(n - 1, max_freqs)
+
+ laplacian = _build_laplacian(parents, norm=laplacian_norm.lower())
+ eigvals_all, eigvecs_all = np.linalg.eigh(laplacian)
+
+ # Skip the trivial (constant) eigenvector at eigenvalue ~0.
+ eigvals = np.maximum(eigvals_all[1:1 + k], 0.0)
+ eigvecs = np.real(eigvecs_all[:, 1:1 + k])
+ eigvecs = _normalize_eigvecs(eigvecs, eigvals, norm=eigvec_norm)
+
+ if k < max_freqs:
+ eigvecs = np.pad(eigvecs, ((0, 0), (0, max_freqs - k)))
+ eigvals = np.pad(eigvals, (0, max_freqs - k))
+
+ return eigvecs.astype(np.float32), eigvals.astype(np.float32)
+
+
+# ---------------------------------------------------------------------------
+# 3. Pairwise relations and topology distances
+# ---------------------------------------------------------------------------
+
+EDGE_TYPES = {
+ 'self': 0, 'parent': 1, 'child': 2, 'sibling': 3,
+ 'no_relation': 4, 'end_effector': 5, 'ts_token_conn': 6,
+}
+
+
+def compute_edge_relations_and_distances(parents, max_path_len=5):
+ """Pairwise edge relation types and clamped topology distances.
+
+ Edge relation types follow ``EDGE_TYPES``: 0=self, 1=parent, 2=child,
+ 3=sibling, 4=no_relation, 5=end_effector, 6=ts_token_conn.
+
+ Returns:
+ ``(edge_rel, topo_rel)``, each ``(J, J)`` int16.
+ """
+ parents = list(parents)
+ n = len(parents)
+ children = _children_map(parents)
+
+ edge_rel = np.full((n, n), EDGE_TYPES['no_relation'], dtype=np.int16)
+ for i in range(n):
+ pi = parents[i]
+ for j in range(n):
+ pj = parents[j]
+ if i == j:
+ edge_rel[i, j] = EDGE_TYPES['self']
+ elif pj == i:
+ edge_rel[i, j] = EDGE_TYPES['child']
+ elif j == pi and pi != -1:
+ edge_rel[i, j] = EDGE_TYPES['parent']
+ elif pi != -1 and pj == pi:
+ edge_rel[i, j] = EDGE_TYPES['sibling']
+ if len(children[i]) == 0:
+ edge_rel[i, i] = EDGE_TYPES['end_effector']
+
+ adj = _adjacency_list(parents)
+ topo_rel = np.full((n, n), max_path_len, dtype=np.int16)
+ for s in range(n):
+ dist = np.full(n, 32767, dtype=np.int16)
+ dist[s] = 0
+ q = deque([s])
+ while q:
+ u = q.popleft()
+ if dist[u] >= max_path_len:
+ continue
+ for v in adj[u]:
+ if dist[v] > dist[u] + 1:
+ dist[v] = dist[u] + 1
+ q.append(v)
+ topo_rel[s] = np.minimum(dist, max_path_len)
+
+ return edge_rel, topo_rel
+
+
+# ---------------------------------------------------------------------------
+# 4. Kinematic chain decomposition
+# ---------------------------------------------------------------------------
+
+def compute_kinematic_chains(parents, policy='h_first', root=0):
+ """Decompose a skeleton tree into kinematic chains (root-to-leaf paths).
+
+ Args:
+ parents: Parent indices (J,).
+ policy: Branching priority โ€” ``'h_first'`` (highest index first)
+ or ``'l_first'`` (lowest index first).
+ root: Index of the root joint.
+
+ Returns:
+ List of chains, each a list of joint indices from root to leaf.
+ """
+ parents = list(parents)
+ n = len(parents)
+ children_dict = {i: [] for i in range(n)}
+
+ for j in range(n):
+ p = parents[j]
+ if j == root or p is None or p == -1:
+ continue
+ if not (0 <= p < n):
+ raise ValueError(f"Invalid parent index parents[{j}]={p}")
+ if policy == 'h_first':
+ _reverse_insort(children_dict[p], j)
+ else:
+ bisect.insort(children_dict[p], j)
+
+ chains = []
+ _build_kinchains([], root, children_dict, chains, policy)
+ return chains
+
+
+def _reverse_insort(a, x, lo=0, hi=None):
+ """Insert *x* into descending-sorted list *a*, keeping order."""
+ if hi is None:
+ hi = len(a)
+ while lo < hi:
+ mid = (lo + hi) // 2
+ if x > a[mid]:
+ hi = mid
+ else:
+ lo = mid + 1
+ a.insert(lo, x)
+
+
+def _build_kinchains(chain, j, children_dict, chains, policy):
+ """Recursively build kinematic chains from the skeleton tree."""
+ children = children_dict[j]
+ chain2 = chain + [j]
+
+ if len(children) == 0:
+ chains.append(chain2)
+ return
+ if len(children) == 1:
+ _build_kinchains(chain2, children[0], children_dict, chains, policy)
+ return
+
+ main_child = children[0]
+ for child in children:
+ if child == main_child:
+ _build_kinchains(chain2, child, children_dict, chains, policy)
+ else:
+ _build_kinchains([j], child, children_dict, chains, policy)
diff --git a/data_process/utils/vlm.py b/data_process/utils/vlm.py
new file mode 100644
index 0000000..ad2cfb2
--- /dev/null
+++ b/data_process/utils/vlm.py
@@ -0,0 +1,759 @@
+"""Shared utilities for the VLM captioning stage.
+
+Frame loading / sampling / base64 encoding, multi-view prompt text,
+render-directory discovery + completeness checks, resumable JSON I/O,
+multi-GPU sharding, Qwen3-VL loading + generation, caption validation, and
+the cached API clients used by :mod:`data_process.vlm_caption`.
+"""
+
+import base64
+import json
+import os
+from functools import lru_cache
+from io import BytesIO
+from pathlib import Path
+
+import torch
+from loguru import logger
+from PIL import Image
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Constants
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+# Camera azimuth of each view (v000..v003) relative to v000, about the
+# vertical axis. The renderer's on-disk metadata names these views
+# front/back/left/right, but those are WORLD-space camera positions โ€” the
+# subject's orientation is arbitrary (normalize_scene never rotates it), so
+# the words carry no information about the subject and would only bias a
+# VLM into reading 'front view' as the subject's front. The model is
+# therefore shown neutral azimuth labels instead.
+VIEW_REL_AZIMUTHS = [0, 180, 270, 90]
+NUM_VIEWS = len(VIEW_REL_AZIMUTHS)
+
+# Rendered clips run to a few hundred frames per view; a fixed stride alone
+# still sends 160 images at the median and 400 worst-case in a single request.
+# Cap the per-view budget and sample uniformly across the clip instead.
+# 64 frames x 4 views = 256 images/request: measured 31.6 GiB peak on
+# Qwen3-VL-8B (A100-80GB), well within budget; the OOM retry ladder in
+# generate_qwen halves this per attempt if a device is tighter.
+DEFAULT_MAX_FRAMES_PER_VIEW = 64
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Frame loading
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+# Renders are saved with a transparent film; a plain .convert("RGB") drops
+# alpha onto black, hiding dark subjects. Composite onto light grey instead
+# (close to the render's own world-background tone).
+RGBA_BACKGROUND = (235, 235, 235)
+
+
+def flatten_rgba(img, background=RGBA_BACKGROUND):
+ # type: (Image.Image, tuple) -> Image.Image
+ """Composite a possibly-transparent image onto an opaque RGB background."""
+ if img.mode == "P":
+ img = img.convert("RGBA")
+ if img.mode in ("RGBA", "LA"):
+ bg = Image.new("RGB", img.size, background)
+ bg.paste(img, mask=img.split()[-1])
+ return bg
+ return img.convert("RGB")
+
+
+def uniform_subsample(items, max_items):
+ # type: (List, Optional[int]) -> List
+ """Uniformly pick at most ``max_items`` entries, keeping first and last.
+
+ Used both to cap frames per view and to shrink a request after a CUDA
+ OOM. A non-positive or None ``max_items`` means "no cap".
+ """
+ n = len(items)
+ if not max_items or max_items <= 0 or n <= max_items:
+ return list(items)
+ if max_items == 1:
+ return [items[0]]
+ # round() spreads the picks evenly over [0, n-1] and pins both ends, so
+ # the prompt's "frame 1 is the start, last frame is the end" holds.
+ idxs = sorted({int(round(i * (n - 1) / (max_items - 1)))
+ for i in range(max_items)})
+ return [items[i] for i in idxs]
+
+
+def select_frame_paths(paths, downsample_rate=1, max_frames=None):
+ # type: (List[Path], int, Optional[int]) -> List[Path]
+ """Choose which frames of one view to send to the VLM.
+
+ ``downsample_rate`` applies the legacy fixed stride first (the last
+ frame is re-appended โ€” a bare ``[::rate]`` drops it whenever the count
+ is not a multiple of the stride), then ``max_frames`` uniformly samples
+ the remainder down to that budget.
+ """
+ if not paths:
+ return []
+ selected = list(paths)
+ if downsample_rate > 1:
+ selected = selected[::downsample_rate]
+ if selected[-1] != paths[-1]:
+ selected.append(paths[-1])
+ return uniform_subsample(selected, max_frames)
+
+
+def load_frames_as_base64(view_dir, max_size=None, jpeg_quality=85,
+ downsample_rate=1, max_frames=None):
+ # type: (str, Optional[int], int, int, Optional[int]) -> List[str]
+ """Load PNG frames from a view directory as base64 strings.
+
+ Frame selection (stride + ``max_frames`` uniform cap, see
+ :func:`select_frame_paths`) happens at the path level before decoding,
+ so discarded frames are never read. When ``max_size`` is set, frames are
+ resized so the longest side is at most ``max_size`` px and re-encoded as
+ JPEG at ``jpeg_quality`` (10-20x payload shrink, quadratic savings in
+ GPT-5 patch tokens). When ``max_size`` is None, raw PNG bytes pass
+ through.
+ """
+ paths = select_frame_paths(
+ sorted(Path(view_dir).glob("*.png")),
+ downsample_rate=downsample_rate, max_frames=max_frames,
+ )
+ result = []
+ for frame_path in paths:
+ if max_size is None:
+ with open(frame_path, "rb") as f:
+ result.append(base64.b64encode(f.read()).decode("utf-8"))
+ continue
+ img = flatten_rgba(Image.open(frame_path))
+ w, h = img.size
+ long_side = max(w, h)
+ if long_side > max_size:
+ scale = max_size / long_side
+ img = img.resize(
+ (max(1, int(round(w * scale))), max(1, int(round(h * scale)))),
+ Image.LANCZOS,
+ )
+ buf = BytesIO()
+ img.save(buf, "JPEG", quality=jpeg_quality, optimize=True)
+ result.append(base64.b64encode(buf.getvalue()).decode("utf-8"))
+ return result
+
+
+def frame_mime_type(max_size):
+ # type: (Optional[int]) -> str
+ """Mime type produced by ``load_frames_as_base64`` for a given ``max_size``."""
+ return "image/jpeg" if max_size is not None else "image/png"
+
+
+def _view_label_from_id(view_id):
+ # type: (str) -> str
+ # Label is keyed by the numeric index in the view id (v000โ†’azimuth 0ยฐ),
+ # NOT by iteration order โ€” so subsetting --views keeps labels correct.
+ try:
+ idx = int(view_id.lstrip("v"))
+ except ValueError:
+ return view_id
+ if 0 <= idx < len(VIEW_REL_AZIMUTHS):
+ return "camera azimuth {}ยฐ".format(VIEW_REL_AZIMUTHS[idx])
+ return "view_{}".format(idx)
+
+
+def load_multiview_frames(
+ render_dir,
+ downsample_rate=2,
+ views=None,
+ max_size=None,
+ jpeg_quality=85,
+ max_frames_per_view=DEFAULT_MAX_FRAMES_PER_VIEW,
+):
+ # type: (str, int, Optional[List[str]], Optional[int], int, Optional[int]) -> Dict[str, List[str]]
+ """Load and downsample frames from every view of a rendered motion.
+
+ Returns {"v000 (camera azimuth 0ยฐ)": [b64, โ€ฆ], โ€ฆ}. Bytes are PNG when
+ ``max_size`` is None, JPEG otherwise (see ``frame_mime_type``). At most
+ ``max_frames_per_view`` frames per view are kept (0/None = no cap).
+
+ Raises:
+ ValueError: a requested view directory is missing or holds no
+ frames. Callers gate on :func:`is_render_complete` first, so
+ this means the render is broken โ€” captioning a subset of the
+ views would silently freeze a partial-render caption.
+ """
+ render_path = Path(render_dir)
+ if views is None:
+ views = sorted([d.name for d in render_path.iterdir()
+ if d.is_dir() and d.name.startswith("v")])
+
+ result = {}
+ for view_id in views:
+ view_dir = render_path / view_id
+ if not view_dir.exists():
+ raise ValueError("View directory not found: {}".format(view_dir))
+ frames = load_frames_as_base64(
+ str(view_dir), max_size=max_size, jpeg_quality=jpeg_quality,
+ downsample_rate=downsample_rate, max_frames=max_frames_per_view,
+ )
+ if not frames:
+ raise ValueError("No frames found in {}".format(view_dir))
+ view_label = _view_label_from_id(view_id)
+ result["{} ({})".format(view_id, view_label)] = frames
+ logger.info("Loaded {} ({}): {} frames (downsample_rate={}, "
+ "max_frames_per_view={}, max_size={})".format(
+ view_id, view_label, len(frames), downsample_rate,
+ max_frames_per_view, max_size))
+
+ return result
+
+
+RENDER_SOURCE_FPS = 30.0 # fallback when a view's metadata JSON is absent
+
+
+def _view_source_fps(render_path, view_id):
+ # type: (Path, str) -> float
+ """Source fps of one view, from the renderer's ``v00X.json`` metadata."""
+ meta = render_path / "{}.json".format(view_id)
+ try:
+ with open(meta) as f:
+ return float(json.load(f)["fps"])
+ except (OSError, KeyError, ValueError):
+ return RENDER_SOURCE_FPS
+
+
+def load_multiview_videos(
+ render_dir,
+ downsample_rate=2,
+ views=None,
+ max_frames_per_view=DEFAULT_MAX_FRAMES_PER_VIEW,
+):
+ # type: (str, int, Optional[List[str]], Optional[int]) -> Tuple[Dict[str, List], Dict[str, float]]
+ """Load each view's frame PNGs as PIL lists for native video input.
+
+ Same discovery, downsampling and per-view cap as
+ :func:`load_multiview_frames`, but returns raw PIL frames (qwen video
+ mode feeds them to the video processor, which does its own resize and
+ 2-frame temporal patching) plus each view's effective fps after
+ sampling, so timestamps stay truthful.
+
+ Returns:
+ ``({view_key: [PIL.Image, ...]}, {view_key: effective_fps})``.
+ """
+ render_path = Path(render_dir)
+ if views is None:
+ views = sorted([d.name for d in render_path.iterdir()
+ if d.is_dir() and d.name.startswith("v")])
+
+ videos, fps_map = {}, {}
+ for view_id in views:
+ view_dir = render_path / view_id
+ if not view_dir.exists():
+ raise ValueError("View directory not found: {}".format(view_dir))
+ frame_paths = sorted(
+ p for p in view_dir.iterdir() if p.suffix.lower() == ".png")
+ frame_paths = frame_paths[::downsample_rate]
+ n_downsampled = len(frame_paths)
+ frame_paths = uniform_subsample(frame_paths, max_frames_per_view)
+ if not frame_paths:
+ raise ValueError("No frames found in {}".format(view_dir))
+ fps = _view_source_fps(render_path, view_id) / downsample_rate
+ if n_downsampled:
+ fps *= len(frame_paths) / float(n_downsampled)
+ key = "{} ({})".format(view_id, _view_label_from_id(view_id))
+ videos[key] = [flatten_rgba(Image.open(str(p))) for p in frame_paths]
+ fps_map[key] = fps
+ logger.info("Loaded {} as video: {} frames @ {:.1f} fps".format(
+ key, len(videos[key]), fps))
+ return videos, fps_map
+
+
+def b64_to_pil(b64_str):
+ # type: (str) -> Image.Image
+ """Convert a base64-encoded PNG string to an opaque RGB PIL Image."""
+ return flatten_rgba(Image.open(BytesIO(base64.b64decode(b64_str))))
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Multiview prompt text (shared by API and local backends)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def build_multiview_intro():
+ # type: () -> str
+ """Build the intro text for a multiview caption prompt.
+
+ The clip directory name is never shown to the model โ€” dataset clip
+ names are (or contain) the action label itself (mixamo:
+ ``Acknowledging``; objaverse: ``{hex}-{Action}``; truebones:
+ ``{Species}-{Action}``), so embedding one would leak the answer or a
+ species word. Per-clip reference labels ride
+ :func:`build_hint_footer` after the views instead.
+ """
+ intro = (
+ "Below are multi-view video frames of a single motion clip. "
+ "Each view shows the same motion from a different camera angle. "
+ "Frames within each view are in chronological order (frame 1 is the "
+ "start, last frame is the end). Analyze the temporal progression "
+ "across all views and describe the motion.\n"
+ )
+ return intro
+
+
+def build_hint_footer(hint):
+ # type: (Optional[str]) -> Optional[str]
+ """Closing text carrying the per-clip reference label, placed AFTER the
+ view sections so the frames are read unprimed and the label sits next
+ to the generation point (image-first VQA layout; how much weight it
+ gets is defined by the task's system prompt)."""
+ if not hint:
+ return None
+ return ("Reference label from the dataset catalogue: '{}'.".format(hint))
+
+
+def build_view_header(view_name, num_frames):
+ # type: (str, int) -> str
+ """Build the header text for a single view section."""
+ return "\n=== {} (frames 1-{}, chronological order) ===".format(
+ view_name, num_frames)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Directory discovery
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def find_motion_dirs(render_root):
+ # type: (str) -> List[Path]
+ """Find all motion directories under a root (must contain view subdirs).
+
+ A motion directory is one that has at least one child directory whose
+ name starts with "v" (e.g., v000, v001). Partially rendered clips are
+ intentionally still returned so callers can report them as (retryable)
+ failures instead of silently dropping them โ€” use
+ :func:`render_incomplete_reason` to gate captioning.
+
+ Returns:
+ Sorted list of Path objects.
+ """
+ render_root = Path(render_root)
+ return sorted([
+ d for d in render_root.iterdir()
+ if d.is_dir() and any(
+ sub.is_dir() and sub.name.startswith("v")
+ for sub in d.iterdir()
+ )
+ ])
+
+
+def render_incomplete_reason(render_dir, num_views=NUM_VIEWS):
+ # type: (str, int) -> Optional[str]
+ """Return None when a clip's multi-view render is complete, else why not.
+
+ Mirrors ``data_process.utils.blender_render.is_render_complete``:
+ ``num_views`` view directories (v000..) with their camera metadata JSONs
+ (v000.json..), each holding the same non-zero number of PNGs. Views
+ render sequentially, so an interrupted render always leaves either a
+ missing view directory or a view with fewer frames than the first.
+ Reimplemented here (rather than imported) because ``blender_render``
+ imports ``bpy``, which the captioning stage must not require.
+ """
+ render_path = Path(render_dir)
+ if not render_path.is_dir():
+ return "not a directory: {}".format(render_dir)
+
+ frame_counts = []
+ for view_idx in range(num_views):
+ view_name = "v{:03d}".format(view_idx)
+ if not (render_path / "{}.json".format(view_name)).is_file():
+ return "missing camera metadata {}.json".format(view_name)
+ view_dir = render_path / view_name
+ if not view_dir.is_dir():
+ return "missing view directory {}".format(view_name)
+ frame_counts.append(len(list(view_dir.glob("*.png"))))
+
+ if frame_counts[0] == 0:
+ return "view v000 has no frames"
+ if len(set(frame_counts)) != 1:
+ return "unequal frame counts across views: {}".format(
+ ", ".join("v{:03d}={}".format(i, c)
+ for i, c in enumerate(frame_counts)))
+ return None
+
+
+def is_render_complete(render_dir, num_views=NUM_VIEWS):
+ # type: (str, int) -> bool
+ """True when every expected view of a clip rendered to the same length."""
+ return render_incomplete_reason(render_dir, num_views=num_views) is None
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# JSON I/O (resumable batch processing)
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def load_json(path):
+ # type: (str) -> Dict
+ """Load a JSON file, returning an empty dict if it doesn't exist."""
+ if os.path.exists(path):
+ with open(path) as f:
+ data = json.load(f)
+ logger.info("Loaded {} existing entries from {}".format(len(data), path))
+ return data
+ return {}
+
+
+def save_json(data, path):
+ # type: (Dict, str) -> None
+ """Atomically save a dict to a JSON file with indentation."""
+ tmp = "{}.tmp.{}".format(path, os.getpid())
+ with open(tmp, "w") as f:
+ json.dump(data, f, indent=2)
+ os.replace(tmp, path)
+
+
+def is_completed(result):
+ # type: (Optional[str]) -> bool
+ """Check if a result string represents a successful completion."""
+ return bool(result) and not result.startswith("ERROR:")
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Multi-GPU sharding
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def shard_output_path(output_json, gpu_id):
+ # type: (str, int) -> str
+ """Return the shard-specific output path for a given GPU."""
+ base, ext = os.path.splitext(output_json)
+ return "{}.shard{}{}".format(base, gpu_id, ext)
+
+
+def find_shard_paths(output_json):
+ # type: (str) -> List[str]
+ """Every ``<base>.shard<N><ext>`` file sitting next to ``output_json``.
+
+ Globbing (rather than trusting a ``num_gpus`` argument) means a merge
+ run launched with fewer GPUs than the run that produced the shards can
+ no longer silently drop the extra shards.
+ """
+ base, ext = os.path.splitext(output_json)
+ pattern = "{}.shard*{}".format(os.path.basename(base), ext)
+ paths = Path(base).parent.glob(pattern)
+
+ def _gid(path):
+ # type: (Path) -> Tuple[int, str]
+ tail = path.name[len(os.path.basename(base)) + len(".shard"):]
+ digits = tail[:-len(ext)] if ext else tail
+ return (int(digits), path.name) if digits.isdigit() else (2 ** 31, path.name)
+
+ return [str(p) for p in sorted(paths, key=_gid)]
+
+
+def merge_shards(output_json, num_gpus=None):
+ # type: (str, Optional[int]) -> Dict[str, str]
+ """Merge per-GPU shard files into a single output JSON.
+
+ Loads every ``output_json.shard*`` file found next to ``output_json``,
+ merges them on top of the existing ``output_json`` contents, and writes
+ the combined result back. ``num_gpus`` is only used to warn when fewer
+ shards than expected are present.
+
+ Returns:
+ The merged dict.
+ """
+ merged = {} # type: Dict[str, str]
+
+ if os.path.exists(output_json):
+ with open(output_json) as f:
+ merged.update(json.load(f))
+ logger.info("Loaded {} existing entries from {}".format(
+ len(merged), output_json))
+
+ shard_paths = find_shard_paths(output_json)
+ for spath in shard_paths:
+ with open(spath) as f:
+ shard_data = json.load(f)
+ merged.update(shard_data)
+ logger.info("Merged shard {} ({} entries)".format(
+ spath, len(shard_data)))
+
+ if not shard_paths:
+ logger.warning("No shard files found for {}".format(output_json))
+ elif num_gpus is not None and len(shard_paths) < num_gpus:
+ logger.warning("Found {} shard files but --num_gpus={}".format(
+ len(shard_paths), num_gpus))
+
+ save_json(merged, output_json)
+ logger.info("Merged {} shards -> {} total entries in {}".format(
+ len(shard_paths), len(merged), output_json))
+ return merged
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Qwen model loading & generation
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def load_qwen_model(
+ model_name,
+ device_map="auto",
+ torch_dtype=None,
+ min_pixels=256 * 28 * 28,
+ max_pixels=256 * 28 * 28,
+ gpu_id=None,
+):
+ # type: (str, str, Optional[str], int, int, Optional[int]) -> Tuple
+ """Load Qwen3-VL model and processor.
+
+ Args:
+ model_name: HuggingFace model ID or local path
+ (e.g., "Qwen/Qwen3-VL-8B-Instruct").
+ device_map: Device placement strategy (default "auto").
+ torch_dtype: Override dtype ("float16", "bfloat16", or None for auto).
+ min_pixels: Minimum pixels per image (default 200704 = 256*28*28).
+ Lower = fewer visual tokens = faster + less memory.
+ max_pixels: Maximum pixels per image (default 200704 = 256*28*28).
+ The Qwen3-VL default is 1843200 (1280*1440) which is very
+ expensive for multi-image inputs. 256*28*28 yields ~256 tokens
+ per image, a good balance for rendered motion frames.
+ gpu_id: If set, load the model on this specific GPU (overrides
+ device_map). Used for multi-GPU data-parallel captioning.
+
+ Returns:
+ (model, processor) tuple.
+ """
+ from transformers import AutoModelForImageTextToText, AutoProcessor
+
+ dtype = torch.bfloat16
+ if torch_dtype == "float16":
+ dtype = torch.float16
+ elif torch_dtype == "bfloat16":
+ dtype = torch.bfloat16
+
+ if gpu_id is not None:
+ device_map = "cuda:{}".format(gpu_id)
+
+ # flash_attn is an optional compiled extra; fall back to PyTorch SDPA
+ # (still fused attention) when it isn't installed rather than failing.
+ try:
+ import flash_attn # noqa: F401
+ attn_implementation = "flash_attention_2"
+ except ImportError:
+ attn_implementation = "sdpa"
+
+ logger.info("Loading model: {} (dtype={}, device_map={}, attn={})".format(
+ model_name, dtype, device_map, attn_implementation))
+
+ model = AutoModelForImageTextToText.from_pretrained(
+ model_name,
+ dtype=dtype,
+ device_map=device_map,
+ attn_implementation=attn_implementation,
+ )
+ processor = AutoProcessor.from_pretrained(
+ model_name,
+ min_pixels=min_pixels,
+ max_pixels=max_pixels,
+ )
+
+ logger.info("Model loaded (min_pixels={}, max_pixels={}, "
+ "~{} visual tokens per image)".format(
+ min_pixels, max_pixels, max_pixels // (28 * 28)))
+ return model, processor
+
+
+def qwen_generate(model, processor, messages, max_tokens=300,
+ return_num_tokens=False, do_sample=False,
+ videos=None, video_fps=None):
+ # type: (object, object, List[Dict], int, bool, bool, Optional[List], Optional[List[float]]) -> Union[str, Tuple[str, int]]
+ """Run Qwen3-VL generation on chat messages and return decoded text.
+
+ Decoding is greedy by default: the shipped Qwen generation_config has
+ ``do_sample=True``, which makes captions drift between identical runs
+ (chirality flips, stray adverbs). Callers enable ``do_sample`` on
+ retry attempts only, so a rejected output gets a genuinely different
+ second chance while the first pass stays reproducible โ€” mirroring the
+ fixed temperatures of the API backends.
+
+ Args:
+ model: Loaded Qwen3-VL model.
+ processor: Loaded Qwen3-VL processor.
+ messages: Chat messages in Qwen format (with PIL Images inline).
+ max_tokens: Maximum new tokens to generate.
+ return_num_tokens: Also return the number of generated tokens, so
+ callers can detect truncation (``n == max_tokens``).
+ do_sample: Sample (temperature 0.7) instead of greedy decoding.
+ videos: Native video inputs, one ``[PIL.Image, ...]`` frame list per
+ ``{"type": "video"}`` item in *messages*. When given, the chat
+ template is rendered to text and the frames go through the
+ processor's video path (2-frame temporal patching, timestamp
+ alignment) instead of the per-image path.
+ video_fps: Per-video effective fps matching *videos*, so the
+ processor's timestamps reflect the true frame spacing.
+
+ Returns:
+ Decoded output string (stripped), or ``(text, num_new_tokens)`` when
+ ``return_num_tokens`` is set.
+ """
+ if videos is None:
+ inputs = processor.apply_chat_template(
+ messages,
+ tokenize=True,
+ add_generation_prompt=True,
+ return_dict=True,
+ return_tensors="pt",
+ )
+ else:
+ text = processor.apply_chat_template(
+ messages, tokenize=False, add_generation_prompt=True)
+ video_kwargs = {"fps": video_fps} if video_fps else {}
+ try:
+ inputs = processor(text=[text], videos=videos,
+ return_tensors="pt", **video_kwargs)
+ except TypeError:
+ # Older processors without an fps kwarg.
+ inputs = processor(text=[text], videos=videos,
+ return_tensors="pt")
+ inputs.pop("token_type_ids", None)
+ logger.debug("Qwen prompt tokens: {}".format(inputs["input_ids"].shape[-1]))
+ inputs = inputs.to(model.device)
+
+ gen_kwargs = dict(max_new_tokens=max_tokens)
+ if do_sample:
+ gen_kwargs.update(do_sample=True, temperature=0.7, top_p=0.8, top_k=20)
+ else:
+ gen_kwargs.update(do_sample=False, temperature=None, top_p=None,
+ top_k=None)
+
+ with torch.no_grad():
+ generated_ids = model.generate(**inputs, **gen_kwargs)
+
+ generated_ids_trimmed = [
+ out_ids[len(in_ids):]
+ for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+ ]
+ text = processor.batch_decode(
+ generated_ids_trimmed, skip_special_tokens=True,
+ clean_up_tokenization_spaces=False,
+ )[0].strip()
+ if return_num_tokens:
+ return text, len(generated_ids_trimmed[0])
+ return text
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Qwen argparse helpers
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def add_qwen_model_args(parser):
+ """Add common Qwen model CLI arguments to an argparse parser."""
+ parser.add_argument('--model', type=str,
+ default='Qwen/Qwen3-VL-8B-Instruct',
+ help='HuggingFace model ID or local path '
+ '(e.g., Qwen/Qwen3-VL-8B-Instruct, '
+ 'Qwen/Qwen3-VL-72B-Instruct).')
+ parser.add_argument('--torch_dtype', type=str, default=None,
+ choices=['float16', 'bfloat16'],
+ help='Override model dtype (default: bfloat16).')
+ parser.add_argument('--device_map', type=str, default='auto',
+ help='Device placement strategy (default: auto).')
+ parser.add_argument('--min_pixels', type=int, default=256 * 28 * 28,
+ help='Min pixels per image (default: 200704 = 256*28*28).')
+ parser.add_argument('--max_pixels', type=int, default=256 * 28 * 28,
+ help='Max pixels per image (default: 200704 = 256*28*28).')
+
+
+def add_multi_gpu_args(parser):
+ """Add multi-GPU sharding CLI arguments to an argparse parser."""
+ parser.add_argument('--num_gpus', type=int, default=1,
+ help='Total number of GPUs for data-parallel captioning.')
+ parser.add_argument('--gpu_id', type=int, default=0,
+ help='GPU index for this process (0-indexed).')
+ parser.add_argument('--merge_shards', action='store_true',
+ help='Merge per-GPU shard files into a single '
+ 'output JSON and exit.')
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Caption validation
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+# finish_reason values meaning "stopped at the token cap" โ€” the caption is
+# truncated mid-sentence and must never be frozen into the output JSON.
+TRUNCATED_FINISH_REASONS = ("length", "max_tokens")
+
+MIN_CAPTION_CHARS = 8
+
+# Matched against the START of the response only, so a legitimate caption
+# that happens to contain one of these words is never rejected.
+REFUSAL_PREFIXES = (
+ "i'm sorry", "i am sorry", "sorry,", "sorry.", "i cannot", "i can't",
+ "i can not", "i'm unable", "i am unable", "i'm not able", "i am not able",
+ "unable to", "as an ai", "i apologize", "there is no image",
+ "no image", "i don't see", "i do not see",
+)
+
+
+def caption_rejection_reason(text, finish_reason=None):
+ # type: (Optional[str], object) -> Optional[str]
+ """Return None when *text* is a usable caption, else why it is rejected.
+
+ Rejects empty/whitespace output, truncation at the token cap (OpenAI's
+ ``finish_reason='length'``, Gemini's ``MAX_TOKENS``, Qwen hitting
+ ``max_new_tokens``) and plain-text refusals. Deliberately conservative:
+ anything else โ€” including an oddly worded caption โ€” is accepted, since
+ a rejected caption costs another API call.
+ """
+ if not text or not text.strip():
+ return "empty response (finish_reason={})".format(finish_reason)
+
+ if finish_reason is not None:
+ # Gemini reports an enum (types.FinishReason.MAX_TOKENS), OpenAI a str.
+ token = str(finish_reason).rsplit(".", 1)[-1].lower()
+ if token in TRUNCATED_FINISH_REASONS:
+ return "truncated at the token cap (finish_reason={})".format(
+ finish_reason)
+
+ stripped = text.strip()
+ if len(stripped) < MIN_CAPTION_CHARS:
+ return "caption too short: {!r}".format(stripped)
+
+ lowered = stripped.lower().lstrip("\"'*` ")
+ if lowered.startswith(REFUSAL_PREFIXES):
+ return "looks like a refusal: {!r}".format(stripped[:120])
+
+ return None
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# API clients + retry classification
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+GEMINI_HTTP_TIMEOUT_MS = 90_000
+RETRYABLE_ERROR_TOKENS = (
+ "429", "503", "500", "504",
+ "RESOURCE_EXHAUSTED", "UNAVAILABLE", "INTERNAL", "DEADLINE_EXCEEDED",
+)
+
+
+@lru_cache(maxsize=4)
+def openai_client(api_key, base_url):
+ """Cache one thread-safe client per (api_key, base_url) tuple.
+
+ A fresh ``OpenAI()`` per request leaks its connection pool (and the file
+ descriptors behind it) across a 10k-clip run.
+ """
+ from openai import OpenAI
+
+ return OpenAI(api_key=api_key, base_url=base_url)
+
+
+@lru_cache(maxsize=4)
+def gemini_client(api_key, is_gemini3):
+ """Cache one thread-safe Client per (api_key, api_version) tuple."""
+ from google import genai
+
+ http_options = {"timeout": GEMINI_HTTP_TIMEOUT_MS}
+ if is_gemini3:
+ http_options["api_version"] = "v1alpha"
+ return genai.Client(api_key=api_key, http_options=http_options)
+
+
+def is_retryable_error(exc):
+ """True for transient VLM API failures worth retrying (rate limit / 5xx)."""
+ msg = str(exc)
+ return any(tok in msg for tok in RETRYABLE_ERROR_TOKENS)
diff --git a/data_process/vlm_caption/__init__.py b/data_process/vlm_caption/__init__.py
new file mode 100644
index 0000000..10fd52a
--- /dev/null
+++ b/data_process/vlm_caption/__init__.py
@@ -0,0 +1,10 @@
+"""Stage 2b โ€” VLM captioning of multi-view renders.
+
+Entry points (also available through ``data_process/scripts/run_caption_*.sh``):
+ caption_motion โ€” per-clip motion captions from ``render/<dataset>/``.
+ classify_category โ€” per-asset body-plan categories from T-pose grids.
+
+Supporting modules: ``backends`` (Qwen / OpenAI / Gemini adapters) and
+``prompts`` (system prompts + task registry). Shared VLM plumbing lives in
+:mod:`data_process.utils.vlm`.
+"""
diff --git a/data_process/vlm_caption/backends.py b/data_process/vlm_caption/backends.py
new file mode 100644
index 0000000..65a85a4
--- /dev/null
+++ b/data_process/vlm_caption/backends.py
@@ -0,0 +1,423 @@
+"""VLM backend adapters for motion captioning.
+
+One captioning request is the same everywhere โ€” a system prompt plus an
+interleaved [view header, frame label, image, ...] user turn โ€” but each
+backend speaks a different dialect. This module owns those dialects:
+
+ qwen โ€” local Qwen3-VL via HuggingFace transformers; PIL images
+ inline, OOM-aware retries that halve the frame count.
+ openai โ€” OpenAI-compatible chat API; base64 ``data:`` image URLs,
+ GPT-5 vs GPT-4o parameter differences handled here.
+ gemini โ€” Google Gemini API; raw image bytes as Parts, thinking and
+ media-resolution controls for Gemini 2.5 / 3.
+
+Each ``generate_*`` function takes the loaded multi-view frames and returns
+a caption string ("" after all retries fail); the caller decides what to do
+with failures. :func:`detect_backend` maps a model name to a backend and
+:func:`image_params` picks the frame loader's resize/quality settings per
+backend, so the orchestrator (:mod:`.caption_motion`) never needs
+backend-specific knowledge beyond dispatching on the backend name.
+"""
+
+import base64
+import os
+import random
+import time
+
+from loguru import logger
+
+from data_process.utils.vlm import (
+ b64_to_pil,
+ build_hint_footer,
+ build_multiview_intro,
+ build_view_header,
+ caption_rejection_reason,
+ gemini_client,
+ is_retryable_error,
+ openai_client,
+ qwen_generate,
+ uniform_subsample,
+)
+
+
+BACKEND_QWEN = "qwen"
+BACKEND_OPENAI = "openai"
+BACKEND_GEMINI = "gemini"
+VALID_BACKENDS = [BACKEND_QWEN, BACKEND_OPENAI, BACKEND_GEMINI]
+
+MAX_RETRIES = 3
+
+
+def detect_backend(model_name):
+ # type: (str) -> str
+ """Auto-detect backend from model name.
+
+ - Contains 'qwen' (case-insensitive) -> qwen
+ - Starts with 'gemini' -> gemini
+ - Everything else -> openai
+ """
+ if "qwen" in model_name.lower():
+ return BACKEND_QWEN
+ elif model_name.startswith("gemini"):
+ return BACKEND_GEMINI
+ else:
+ return BACKEND_OPENAI
+
+
+# Loader-side resize targets per (backend, model). Qwen relies on its
+# AutoProcessor (min/max_pixels) for server-side resize; API backends pre-shrink
+# to keep payloads small and, for GPT-5 patch vision, to cut tokens quadratically.
+# GPT-5: ceil(w/32)ยฒยทmult โ€” 384ยฒ โ†’ ~233 tok vs 415 at 512ยฒ, q85 stays crisp.
+# GPT-4o tile scheme with detail='low' is flat 85 tok โ€” size only affects bandwidth.
+# Gemini: server-side media_resolution sets tokens; 768 caps the inline upload.
+def image_params(backend, model_name):
+ # type: (str, str) -> tuple
+ """Return ``(max_size, jpeg_quality)`` for the frame loader."""
+ if backend == BACKEND_QWEN:
+ return None, 85
+ if backend == BACKEND_GEMINI:
+ return 768, 85
+ if model_name.startswith("gpt-5"):
+ return 384, 85
+ return 512, 80
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Message building
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _build_qwen_messages(multiview_frames, system_prompt, hint=None):
+ # type: (Dict[str, List[str]], str, Optional[str]) -> List[Dict]
+ """Build Qwen3-VL chat messages with PIL Images inline."""
+ user_content = []
+ user_content.append({
+ "type": "text",
+ "text": build_multiview_intro(),
+ })
+
+ for view_name, frames in multiview_frames.items():
+ user_content.append({
+ "type": "text",
+ "text": build_view_header(view_name, len(frames)),
+ })
+ for idx, frame_b64 in enumerate(frames):
+ user_content.append({
+ "type": "text",
+ "text": "[frame {}]".format(idx + 1),
+ })
+ user_content.append({
+ "type": "image",
+ "image": b64_to_pil(frame_b64),
+ })
+
+ footer = build_hint_footer(hint)
+ if footer:
+ user_content.append({"type": "text", "text": footer})
+
+ return [
+ {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
+ {"role": "user", "content": user_content},
+ ]
+
+
+def _build_qwen_video_messages(multiview_videos, system_prompt, hint=None):
+ # type: (Dict[str, List], str, Optional[str]) -> List[Dict]
+ """Build Qwen3-VL chat messages with one native video item per view.
+
+ Each view's PIL frame list rides the processor's video path (2-frame
+ temporal patching + timestamp alignment) instead of being spelled out
+ as individually labeled images โ€” half the visual tokens and native
+ temporal encoding for the same frames.
+ """
+ user_content = [{
+ "type": "text",
+ "text": build_multiview_intro(),
+ }]
+ for view_name, frames in multiview_videos.items():
+ user_content.append({
+ "type": "text",
+ "text": build_view_header(view_name, len(frames)),
+ })
+ user_content.append({"type": "video", "video": frames})
+
+ footer = build_hint_footer(hint)
+ if footer:
+ user_content.append({"type": "text", "text": footer})
+
+ return [
+ {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
+ {"role": "user", "content": user_content},
+ ]
+
+
+def _build_openai_messages(multiview_frames, system_prompt, mime_type,
+ hint=None):
+ # type: (Dict[str, List[str]], str, str, Optional[str]) -> List[Dict]
+ """Build OpenAI-compatible chat messages with base64 image URLs.
+
+ Frames are already resized + encoded by the loader; this just wraps them
+ in the ``data:`` URL scheme.
+ """
+ user_content = []
+ user_content.append({
+ "type": "text",
+ "text": build_multiview_intro(),
+ })
+
+ for view_name, frames in multiview_frames.items():
+ user_content.append({
+ "type": "text",
+ "text": build_view_header(view_name, len(frames)),
+ })
+ for idx, frame_b64 in enumerate(frames):
+ user_content.append({
+ "type": "text",
+ "text": "[frame {}]".format(idx + 1),
+ })
+ user_content.append({
+ "type": "image_url",
+ "image_url": {
+ "url": "data:{};base64,{}".format(mime_type, frame_b64),
+ "detail": "low",
+ },
+ })
+
+ footer = build_hint_footer(hint)
+ if footer:
+ user_content.append({"type": "text", "text": footer})
+
+ return [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": user_content},
+ ]
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Generation
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def generate_qwen(multiview_frames, motion_name, system_prompt,
+ model, processor, max_tokens, max_retries=MAX_RETRIES,
+ hint=None, video_fps=None):
+ # type: (Dict[str, List], str, str, object, object, int, int, Optional[str], Optional[Dict[str, float]]) -> str
+ """Generate caption using local Qwen3-VL model.
+
+ ``multiview_frames`` carries base64 PNG/JPEG strings in image mode, or
+ PIL frame lists when ``video_fps`` is given โ€” the latter selects the
+ native video input path (one video item per view).
+
+ Retries transient failures (OOM, empty/truncated/refusal output) up to
+ max_retries times. An OOM halves the frames per view before retrying โ€”
+ resubmitting the identical image stack would OOM again deterministically.
+ """
+ import torch # local import; keeps module light if never hit
+
+ frames = multiview_frames
+ video_mode = video_fps is not None
+
+ def _assemble(f, fps_scale=1.0):
+ if video_mode:
+ msgs = _build_qwen_video_messages(f, system_prompt, hint=hint)
+ fps = [video_fps[v] * fps_scale for v in f]
+ return msgs, {"videos": [f[v] for v in f], "video_fps": fps}
+ return _build_qwen_messages(f, system_prompt, hint=hint), {}
+
+ fps_scale = 1.0
+ messages, vid_kwargs = _assemble(frames)
+
+ for attempt in range(1, max_retries + 1):
+ try:
+ # First attempt is greedy (reproducible); retries sample so a
+ # rejected output gets a genuinely different second chance.
+ text, num_tokens = qwen_generate(
+ model, processor, messages, max_tokens=max_tokens,
+ return_num_tokens=True, do_sample=attempt > 1,
+ **vid_kwargs,
+ )
+ # Qwen exposes no finish_reason; hitting the cap means truncation.
+ finish_reason = "length" if num_tokens >= max_tokens else "stop"
+ reason = caption_rejection_reason(text, finish_reason)
+ if reason is None:
+ return text.strip()
+ logger.warning("Attempt {}/{}: rejected Qwen output โ€” {}".format(
+ attempt, max_retries, reason))
+ except torch.cuda.OutOfMemoryError as e:
+ logger.warning("Attempt {}/{}: CUDA OOM: {}".format(
+ attempt, max_retries, e))
+ torch.cuda.empty_cache()
+ frames = {
+ view: uniform_subsample(f, max(1, len(f) // 2))
+ for view, f in frames.items()
+ }
+ # Halving the frames doubles their temporal spacing.
+ fps_scale *= 0.5
+ logger.warning("Retrying with {} frames (halved per view)".format(
+ sum(len(f) for f in frames.values())))
+ messages, vid_kwargs = _assemble(frames, fps_scale=fps_scale)
+ except Exception as e:
+ logger.warning("Attempt {}/{}: Qwen error: {}".format(
+ attempt, max_retries, e))
+
+ if attempt < max_retries:
+ time.sleep(1)
+
+ logger.error("All {} Qwen attempts failed or returned empty for '{}'".format(
+ max_retries, motion_name))
+ return ""
+
+
+def generate_openai(multiview_frames, motion_name, system_prompt,
+ model_name, api_key, base_url, max_tokens, max_retries,
+ mime_type, hint=None):
+ # type: (Dict[str, List[str]], str, str, str, Optional[str], Optional[str], int, int, str, Optional[str]) -> str
+ """Generate caption using OpenAI-compatible API."""
+ client = openai_client(
+ api_key or os.environ.get("OPENAI_API_KEY"), base_url,
+ )
+
+ messages = _build_openai_messages(
+ multiview_frames, system_prompt, mime_type=mime_type, hint=hint,
+ )
+
+ # gpt-5+ reasoning models use max_completion_tokens and don't support
+ # temperature; older models (gpt-4o, etc.) use max_tokens.
+ is_gpt5 = "gpt-5" in model_name
+ create_kwargs = dict(model=model_name, messages=messages)
+ if is_gpt5:
+ create_kwargs["max_completion_tokens"] = max_tokens
+ else:
+ create_kwargs["max_tokens"] = max_tokens
+ create_kwargs["temperature"] = 0.3
+
+ for attempt in range(1, max_retries + 1):
+ try:
+ response = client.chat.completions.create(**create_kwargs)
+
+ choice = response.choices[0]
+ content = choice.message.content
+ refusal = getattr(choice.message, "refusal", None)
+ logger.debug("Attempt {}: finish_reason={}, refusal={}".format(
+ attempt, choice.finish_reason, refusal))
+
+ if refusal:
+ logger.warning("Attempt {}/{}: model refused: {}".format(
+ attempt, max_retries, refusal))
+ else:
+ reason = caption_rejection_reason(content, choice.finish_reason)
+ if reason is None:
+ return content.strip()
+ logger.warning("Attempt {}/{}: rejected response โ€” {}".format(
+ attempt, max_retries, reason))
+ except Exception as e:
+ logger.warning("Attempt {}/{}: OpenAI API error: {}".format(
+ attempt, max_retries, e))
+
+ if attempt < max_retries:
+ time.sleep(2)
+
+ logger.error("All {} attempts failed or returned empty for '{}'".format(
+ max_retries, motion_name))
+ return ""
+
+
+def generate_gemini(multiview_frames, motion_name, system_prompt,
+ model_name, api_key, max_tokens, max_retries,
+ mime_type="image/png",
+ thinking_level="low", media_resolution="medium",
+ thinking_budget=0, hint=None):
+ # type: (Dict[str, List[str]], str, str, str, Optional[str], int, int, str, str, str, int, Optional[str]) -> str
+ """Generate caption using Google Gemini API.
+
+ Gemini 3 uses ``thinking_level`` (enum) + ``media_resolution``; Gemini 2.5
+ uses ``thinking_budget`` (int token cap, 0 disables, -1 dynamic). For a
+ one-sentence caption, dynamic thinking is wasteful โ€” default is 0.
+ """
+ from google.genai import types
+
+ is_gemini3 = "gemini-3" in model_name
+ is_gemini25 = "gemini-2.5" in model_name
+
+ # Gemini 3's thinking_config draws reasoning tokens from the same budget
+ # as the final answer. At thinking_level='low' the reasoning pass commonly
+ # burns 200โ€“500 tokens, so <1024 total risks returning an empty answer.
+ if is_gemini3 and max_tokens < 1024:
+ logger.warning(
+ "max_tokens={} is tight for Gemini 3 + thinking; raising to 1024."
+ .format(max_tokens))
+ max_tokens = 1024
+
+ client = gemini_client(
+ api_key or os.environ.get("GOOGLE_API_KEY"), is_gemini3
+ )
+
+ contents = [types.Part.from_text(text=build_multiview_intro())]
+ for view_name, frames in multiview_frames.items():
+ contents.append(types.Part.from_text(
+ text=build_view_header(view_name, len(frames)),
+ ))
+ for idx, frame_b64 in enumerate(frames):
+ contents.append(types.Part.from_text(
+ text="[frame {}]".format(idx + 1),
+ ))
+ contents.append(types.Part.from_bytes(
+ data=base64.b64decode(frame_b64),
+ mime_type=mime_type,
+ ))
+ footer = build_hint_footer(hint)
+ if footer:
+ contents.append(types.Part.from_text(text=footer))
+
+ config_kwargs = dict(
+ system_instruction=system_prompt,
+ max_output_tokens=max_tokens,
+ temperature=0.0,
+ stop_sequences=["\n\n"],
+ )
+ if is_gemini3:
+ config_kwargs["thinking_config"] = types.ThinkingConfig(
+ thinking_level=thinking_level,
+ )
+ config_kwargs["media_resolution"] = "MEDIA_RESOLUTION_{}".format(
+ media_resolution.upper()
+ )
+ elif is_gemini25:
+ config_kwargs["thinking_config"] = types.ThinkingConfig(
+ thinking_budget=thinking_budget,
+ )
+ config = types.GenerateContentConfig(**config_kwargs)
+
+ for attempt in range(1, max_retries + 1):
+ try:
+ response = client.models.generate_content(
+ model=model_name,
+ contents=contents,
+ config=config,
+ )
+ candidates = getattr(response, "candidates", None) or []
+ finish_reason = getattr(candidates[0], "finish_reason", None) \
+ if candidates else None
+ reason = caption_rejection_reason(response.text, finish_reason)
+ if reason is None:
+ return response.text.strip()
+ # A rejected response (empty, truncated at MAX_TOKENS, refusal)
+ # is worth another shot; fall through to the backoff.
+ logger.warning("Attempt {}/{}: rejected Gemini response โ€” {}".format(
+ attempt, max_retries, reason))
+ except Exception as e:
+ retryable = is_retryable_error(e)
+ level = "warning" if retryable else "error"
+ getattr(logger, level)(
+ "Attempt {}/{}: Gemini API error ({}): {}".format(
+ attempt, max_retries,
+ "retryable" if retryable else "fatal", e))
+ if not retryable:
+ break
+
+ if attempt < max_retries:
+ # Exponential backoff with jitter: 5s, 15s, 45s, โ€ฆ capped at 60s.
+ delay = min(60.0, 5.0 * (3 ** (attempt - 1))) + random.uniform(0, 2)
+ time.sleep(delay)
+
+ logger.error("All {} attempts failed or returned empty for '{}'".format(
+ max_retries, motion_name))
+ return ""
diff --git a/data_process/vlm_caption/caption_motion.py b/data_process/vlm_caption/caption_motion.py
new file mode 100644
index 0000000..911df7f
--- /dev/null
+++ b/data_process/vlm_caption/caption_motion.py
@@ -0,0 +1,563 @@
+"""Caption multi-view rendered motion sequences with a Vision-Language Model.
+
+Supports three backends (see :mod:`data_process.vlm_caption.backends`):
+ qwen โ€” Local Qwen3-VL via HuggingFace transformers (default).
+ openai โ€” OpenAI-compatible API (GPT-4o, GPT-5, ...).
+ gemini โ€” Google Gemini API.
+
+The backend is auto-detected from the model name, or set explicitly via
+``--backend``. ``--task`` selects the dataset system prompt (see
+:mod:`data_process.vlm_caption.prompts`): ``mixamo``, ``objaverse`` or
+``truebones``.
+
+Input layout: ``<render_root>/<motion_name>/v00{0..3}/*.png`` โ€” one
+directory per motion with one sub-directory of frame PNGs per camera view.
+
+``--hints_json`` optionally feeds per-clip reference labels from dataset
+metadata (Mixamo's official catalogue prompts, truebones' T2M4LVO
+annotations) to the model as frame-grounded reference labels;
+``run_caption_motion.sh`` wires this up for both by default.
+
+Usage:
+ # Local Qwen (auto-detected from model name)
+ python -m data_process.vlm_caption.caption_motion \
+ --render_root dataset/render/truebones --task truebones
+
+ # Gemini 3 Flash API
+ python -m data_process.vlm_caption.caption_motion \
+ --render_root dataset/render/objaverse \
+ --model gemini-3-flash-preview --task objaverse --num_workers 16
+
+ # Multi-GPU (qwen backend only)
+ python -m data_process.vlm_caption.caption_motion \
+ --render_root dataset/render/objaverse --task objaverse --num_gpus 4 --gpu_id 0
+
+ # Merge shards after multi-GPU
+ python -m data_process.vlm_caption.caption_motion \
+ --merge_shards --output_json motion_captions.json --num_gpus 4
+"""
+
+import argparse
+import os
+import re
+import threading
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from loguru import logger
+from tqdm import tqdm
+
+from data_process.vlm_caption.backends import (
+ BACKEND_GEMINI,
+ BACKEND_QWEN,
+ MAX_RETRIES,
+ VALID_BACKENDS,
+ detect_backend,
+ generate_gemini,
+ generate_openai,
+ generate_qwen,
+ image_params,
+)
+from data_process.vlm_caption.prompts import TASK_PROMPTS, get_prompt
+from data_process.utils.vlm import (
+ DEFAULT_MAX_FRAMES_PER_VIEW,
+ add_multi_gpu_args,
+ find_motion_dirs,
+ frame_mime_type,
+ is_completed,
+ load_json,
+ load_multiview_frames,
+ load_multiview_videos,
+ load_qwen_model,
+ merge_shards,
+ render_incomplete_reason,
+ save_json,
+ shard_output_path,
+)
+
+
+SAVE_EVERY = 20 # captions per incremental JSON flush
+
+
+def load_hints(hints_json):
+ # type: (Optional[str]) -> Dict[str, str]
+ """Load per-clip reference labels from a dataset metadata JSON.
+
+ Accepts either flat ``{name: label}`` or record values like Mixamo's
+ ``animation_motion_prompts.json`` (``{name: {prompt, description, ...}}``,
+ where ``description`` is preferred over ``prompt`` as the more specific
+ label). Keys are normalised to the render clip-directory name by
+ stripping a file extension, so ``Jab_Cross.fbx`` matches the render dir
+ ``Jab_Cross``.
+ """
+ if not hints_json:
+ return {}
+ hints = {}
+ for key, value in load_json(hints_json).items():
+ name = os.path.splitext(key)[0]
+ if isinstance(value, dict):
+ label = value.get("description") or value.get("prompt") or ""
+ # Keep only the ACTION as reference: records that name their
+ # subject (truebones' object_name, e.g. 'Tyrannosaurus Rex')
+ # get it replaced with the neutral word so no species reaches
+ # the model.
+ subject = value.get("object_name")
+ if subject and label:
+ label = re.sub(re.escape(subject), "animal", label,
+ flags=re.IGNORECASE)
+ label = (label.replace("A animal", "An animal")
+ .replace("a animal", "an animal"))
+ else:
+ label = str(value)
+ if label:
+ hints[name] = label
+ logger.info("Loaded {} caption hints from {}".format(len(hints), hints_json))
+ return hints
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Batch captioning
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def caption_batch(
+ render_root,
+ backend,
+ system_prompt,
+ output_json=None,
+ downsample_rate=1,
+ max_frames_per_view=None,
+ max_tokens=300,
+ views=None,
+ hints=None,
+ vision_input="video",
+ gpu_id=0,
+ num_gpus=1,
+ num_workers=1,
+ # qwen-specific
+ qwen_model=None,
+ qwen_processor=None,
+ # api-specific
+ model_name=None,
+ api_key=None,
+ base_url=None,
+ max_retries=MAX_RETRIES,
+ # gemini-specific
+ thinking_level="low",
+ media_resolution="medium",
+ thinking_budget=0,
+):
+ """Caption all motion sequences under a root render directory.
+
+ Skips sequences that already have captions in the output JSON (for
+ resumability). Clips whose render is incomplete are reported as failures
+ and left out of the JSON, so they are retried once rendering finishes.
+ Results are saved incrementally (every SAVE_EVERY captions).
+
+ Args:
+ max_frames_per_view: Cap on frames sent per view after
+ ``downsample_rate`` (0 = no cap; frames are sampled
+ uniformly across the clip, first and last always kept).
+ Default None resolves per mode: no cap for 'video' (its
+ 2-frame temporal patching keeps even a 200-frame clip
+ cheap), DEFAULT_MAX_FRAMES_PER_VIEW for 'images'.
+ vision_input: 'video' (default) sends each view through the
+ qwen backend's native video path (2-frame temporal patching
+ + timestamp alignment, ~half the visual tokens of 'images');
+ 'images' sends individually labeled frames. Video is
+ qwen-only; API backends fall back to images with a warning.
+ hints: Optional ``{clip_name: reference_label}`` mapping (see
+ ``load_hints``); a clip's label is passed to the model as a
+ verify-against-frames hint.
+ gpu_id: GPU/process index for multi-GPU sharding (default 0).
+ num_gpus: Total number of GPUs/processes (default 1 = no sharding).
+ When num_gpus > 1, each process handles every num_gpus-th motion
+ and saves to a shard file. Use merge_shards() afterwards.
+ num_workers: Concurrent in-process workers. Only effective for HTTP
+ backends (gemini/openai); Qwen ignores this since one local model
+ can't be shared across threads safely.
+ """
+ render_root = Path(render_root)
+
+ if output_json is None:
+ output_json = str(render_root / "motion_captions.json")
+
+ save_path = shard_output_path(output_json, gpu_id) if num_gpus > 1 else output_json
+ save_path_obj = Path(save_path)
+ failed_path = str(save_path_obj.with_name(save_path_obj.stem + "_failed.txt"))
+
+ captions = load_json(save_path)
+
+ # Resume from the merged output as well: shard files are not cleaned up
+ # after a merge, so a tidied-up directory would otherwise re-caption the
+ # whole dataset. The shard wins on conflict; only the shard's own entries
+ # are written back to save_path.
+ already_done = dict(captions)
+ if save_path != output_json:
+ for name, caption in load_json(output_json).items():
+ already_done.setdefault(name, caption)
+
+ motion_dirs = find_motion_dirs(str(render_root))
+
+ failed_set = set()
+ if os.path.exists(failed_path):
+ with open(failed_path, "r") as f:
+ for line in f:
+ name = line.strip()
+ if name:
+ failed_set.add(name)
+
+ if num_gpus > 1:
+ motion_dirs = motion_dirs[gpu_id::num_gpus]
+ logger.info("[GPU {}/{}] Assigned {} motion sequences".format(
+ gpu_id, num_gpus, len(motion_dirs)))
+ else:
+ logger.info("Found {} motion sequences in {}".format(
+ len(motion_dirs), render_root))
+
+ pending = [d for d in motion_dirs if not is_completed(already_done.get(d.name))]
+ logger.info("{} already done, {} pending.".format(
+ len(motion_dirs) - len(pending), len(pending)))
+
+ if backend == BACKEND_QWEN and num_workers > 1:
+ logger.warning("Qwen backend does not support num_workers > 1; using 1.")
+ num_workers = 1
+
+ if vision_input == "video" and backend != BACKEND_QWEN:
+ logger.warning("--vision_input video is qwen-only; "
+ "falling back to images for backend '{}'.".format(backend))
+ vision_input = "images"
+
+ if max_frames_per_view is None:
+ max_frames_per_view = 0 if vision_input == "video" \
+ else DEFAULT_MAX_FRAMES_PER_VIEW
+ logger.info("vision_input={} max_frames_per_view={}".format(
+ vision_input, max_frames_per_view or "uncapped"))
+
+ max_size, jpeg_quality = image_params(backend, model_name or "")
+ mime_type = frame_mime_type(max_size)
+ logger.info("Frame loader: max_size={} jpeg_q={} mime={} "
+ "downsample_rate={} max_frames_per_view={}".format(
+ max_size, jpeg_quality, mime_type, downsample_rate,
+ max_frames_per_view))
+
+ state_lock = threading.Lock()
+ n_ok = 0
+ n_err = 0
+ n_unsaved = 0
+
+ def _process(motion_dir):
+ motion_name = motion_dir.name
+ try:
+ # A partially rendered clip must never be captioned: the caption
+ # would be frozen forever (is_completed() accepts any non-ERROR
+ # string). Report it as a failure so a later run retries it once
+ # rendering has finished.
+ incomplete = render_incomplete_reason(str(motion_dir))
+ if incomplete is not None:
+ return motion_name, None, "incomplete render: {}".format(
+ incomplete)
+
+ logger.info("Captioning [{}/{}]: {}".format(
+ backend, vision_input, motion_name))
+ video_fps = None
+ if vision_input == "video":
+ multiview_frames, video_fps = load_multiview_videos(
+ str(motion_dir),
+ downsample_rate=downsample_rate, views=views,
+ max_frames_per_view=max_frames_per_view,
+ )
+ else:
+ multiview_frames = load_multiview_frames(
+ str(motion_dir),
+ downsample_rate=downsample_rate, views=views,
+ max_size=max_size, jpeg_quality=jpeg_quality,
+ max_frames_per_view=max_frames_per_view,
+ )
+ if not multiview_frames:
+ return motion_name, None, "no frames"
+
+ total_images = sum(len(f) for f in multiview_frames.values())
+ logger.info("Total frames for VLM: {}".format(total_images))
+
+ hint = hints.get(motion_name) if hints else None
+ if backend == BACKEND_QWEN:
+ caption = generate_qwen(
+ multiview_frames, motion_name, system_prompt,
+ model=qwen_model, processor=qwen_processor,
+ max_tokens=max_tokens, max_retries=max_retries,
+ hint=hint, video_fps=video_fps,
+ )
+ elif backend == BACKEND_GEMINI:
+ caption = generate_gemini(
+ multiview_frames, motion_name, system_prompt,
+ model_name=model_name, api_key=api_key,
+ max_tokens=max_tokens, max_retries=max_retries,
+ mime_type=mime_type,
+ thinking_level=thinking_level,
+ media_resolution=media_resolution,
+ thinking_budget=thinking_budget,
+ hint=hint,
+ )
+ else:
+ caption = generate_openai(
+ multiview_frames, motion_name, system_prompt,
+ model_name=model_name, api_key=api_key, base_url=base_url,
+ max_tokens=max_tokens, max_retries=max_retries,
+ mime_type=mime_type, hint=hint,
+ )
+
+ logger.info("Caption for '{}': {}".format(motion_name, caption))
+ if caption:
+ return motion_name, caption, None
+ return motion_name, None, "empty response"
+ except Exception as e:
+ logger.error("Failed to caption {}: {}".format(motion_name, e))
+ return motion_name, None, str(e)
+
+ gpu_label = " GPU {}".format(gpu_id) if num_gpus > 1 else ""
+ pbar = tqdm(total=len(pending), desc="Captioning{}".format(gpu_label),
+ unit="seq", dynamic_ncols=True)
+
+ def _record(motion_name, caption, error):
+ nonlocal n_ok, n_err, n_unsaved
+ with state_lock:
+ if caption is not None:
+ captions[motion_name] = caption
+ n_ok += 1
+ n_unsaved += 1
+ # Rewriting the full JSON per caption is O(n^2) over a large
+ # run; batch it (a crash loses at most SAVE_EVERY captions).
+ if n_unsaved >= SAVE_EVERY:
+ save_json(captions, save_path)
+ n_unsaved = 0
+ else:
+ n_err += 1
+ if motion_name not in failed_set:
+ failed_set.add(motion_name)
+ with open(failed_path, "a") as f:
+ f.write(motion_name + "\n")
+ logger.warning("Skipped {} (not saved to JSON): {}".format(
+ motion_name, error))
+ pbar.set_postfix_str("{} ok / {} err".format(n_ok, n_err),
+ refresh=False)
+ pbar.update(1)
+
+ try:
+ if num_workers <= 1:
+ for d in pending:
+ name, caption, error = _process(d)
+ _record(name, caption, error)
+ else:
+ with ThreadPoolExecutor(max_workers=num_workers) as ex:
+ futures = [ex.submit(_process, d) for d in pending]
+ try:
+ for fut in as_completed(futures):
+ name, caption, error = fut.result()
+ _record(name, caption, error)
+ except KeyboardInterrupt:
+ # Every item is queued upfront; without cancel_futures the
+ # pool's shutdown would still run (and bill) all of them.
+ logger.warning("Interrupted โ€” cancelling queued requests.")
+ ex.shutdown(wait=False, cancel_futures=True)
+ raise
+ finally:
+ with state_lock:
+ if n_unsaved:
+ save_json(captions, save_path)
+ n_unsaved = 0
+
+ pbar.close()
+ logger.info("Done. {} ok / {} err this run; {} total saved to {}".format(
+ n_ok, n_err, len(captions), save_path))
+ if n_err:
+ logger.info("Failed motion names appended to {}".format(failed_path))
+ return captions
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# CLI
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Caption multi-view motion renders using a VLM "
+ "(local Qwen, OpenAI API, or Gemini API)."
+ )
+
+ parser.add_argument('--render_root', type=str,
+ help='Root directory for batch captioning all motions.')
+
+ parser.add_argument('--output_json', type=str, default=None,
+ help='Output JSON file. '
+ 'Defaults to <render_root>/motion_captions.json.')
+
+ # โ”€โ”€ Backend & task โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ parser.add_argument('--backend', type=str, default=None,
+ choices=VALID_BACKENDS,
+ help='VLM backend. Auto-detected from --model if '
+ 'not specified (qwen/openai/gemini).')
+ parser.add_argument('--task', type=str, required=True,
+ choices=sorted(TASK_PROMPTS.keys()),
+ help='Captioning task / system prompt to use.')
+ parser.add_argument('--hints_json', type=str, default=None,
+ help='Optional JSON with per-clip reference labels '
+ 'shown to the model as verify-against-frames '
+ 'hints (e.g. dataset/raw/mixamo/'
+ 'animation_motion_prompts.json). Keys may carry '
+ 'a file extension; dict values use description '
+ 'over prompt.')
+
+ # โ”€โ”€ Model โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ parser.add_argument('--model', type=str,
+ default='Qwen/Qwen3-VL-8B-Instruct',
+ help='Model name. For qwen: HuggingFace model ID '
+ '(default Qwen/Qwen3-VL-8B-Instruct). For openai: '
+ 'API model (e.g., gpt-5-mini). For gemini: API model '
+ '(e.g., gemini-3-flash-preview).')
+
+ # โ”€โ”€ Qwen-specific โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ parser.add_argument('--torch_dtype', type=str, default=None,
+ choices=['float16', 'bfloat16'],
+ help='[qwen] Override model dtype (default: bfloat16).')
+ parser.add_argument('--device_map', type=str, default='auto',
+ help='[qwen] Device placement strategy (default: auto).')
+ parser.add_argument('--min_pixels', type=int, default=256 * 28 * 28,
+ help='[qwen] Min pixels per image '
+ '(default: 200704 = 256*28*28).')
+ parser.add_argument('--max_pixels', type=int, default=256 * 28 * 28,
+ help='[qwen] Max pixels per image '
+ '(default: 200704 = 256*28*28).')
+
+ # โ”€โ”€ API-specific โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ parser.add_argument('--api_key', type=str, default=None,
+ help='[openai/gemini] API key (defaults to '
+ 'OPENAI_API_KEY or GOOGLE_API_KEY env var).')
+ parser.add_argument('--base_url', type=str, default=None,
+ help='[openai] Override API base URL.')
+ parser.add_argument('--max_retries', type=int, default=MAX_RETRIES,
+ help='[openai/gemini] Max retry attempts for '
+ 'empty responses.')
+
+ # โ”€โ”€ Gemini 3-specific โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ parser.add_argument('--thinking_level', type=str, default='low',
+ choices=['minimal', 'low', 'medium', 'high'],
+ help='[gemini-3] Reasoning depth. Captioning works '
+ 'well at "low" (default); raise to "medium" if '
+ 'captions are noisy.')
+ parser.add_argument('--media_resolution', type=str, default='medium',
+ choices=['low', 'medium', 'high', 'ultra_high'],
+ help='[gemini-3] Vision tokens per image. "medium" '
+ '(560 tok) is a good default for multi-view '
+ 'frames; "high" (1120 tok) for fine details. '
+ '"ultra_high" is Gemini 3 Flash only.')
+ parser.add_argument('--thinking_budget', type=int, default=0,
+ help='[gemini-2.5] Thinking token cap. 0 disables '
+ '(recommended for one-sentence captions); -1 '
+ 'enables dynamic thinking; up to 24576.')
+
+ # โ”€โ”€ Common โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ parser.add_argument('--vision_input', type=str, default='video',
+ choices=['images', 'video'],
+ help="How frames reach the model. 'video' (default) "
+ "uses the qwen backend's native video path "
+ "(temporal patching + timestamps, ~half the "
+ "visual tokens; API backends fall back to "
+ "images); 'images' sends individually labeled "
+ "frames.")
+ parser.add_argument('--views', type=str, nargs='+', default=None,
+ help='Subset of views to use (e.g., v000 v002). '
+ 'Default: all views.')
+ parser.add_argument('--downsample_rate', type=int, default=1,
+ help='Take every N-th frame per view (default 1 = '
+ 'full 30 FPS; 2 -> 15 FPS).')
+ parser.add_argument('--max_frames_per_view', type=int, default=None,
+ help='Cap on frames sent per view after '
+ '--downsample_rate (0 = no cap). Default: '
+ 'no cap in video mode (temporal patching '
+ 'keeps it cheap), {} in images mode.'
+ .format(DEFAULT_MAX_FRAMES_PER_VIEW))
+ parser.add_argument('--max_tokens', type=int, default=1024,
+ help='Max new tokens to generate.')
+ parser.add_argument('--num_workers', type=int, default=1,
+ help='Concurrent in-process workers for HTTP backends '
+ '(gemini/openai). Qwen ignores this.')
+
+ add_multi_gpu_args(parser)
+
+ return parser, parser.parse_args()
+
+
+def main():
+ parser, args = parse_args()
+
+ # โ”€โ”€ Merge mode โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ if args.merge_shards:
+ if not args.output_json:
+ parser.error("--merge_shards requires --output_json")
+ merge_shards(args.output_json, args.num_gpus)
+ return
+
+ if not args.render_root:
+ parser.error("--render_root is required")
+
+ # โ”€โ”€ Resolve backend โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ backend = args.backend or detect_backend(args.model)
+
+ task = args.task
+ logger.info("Backend: {} | Model: {} | Task: {}".format(
+ backend, args.model, task))
+
+ system_prompt = get_prompt(task)
+
+ # โ”€โ”€ Multi-GPU is only for qwen (local model) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ num_gpus = args.num_gpus
+ gpu_id = args.gpu_id
+ if backend != BACKEND_QWEN and num_gpus > 1:
+ logger.warning("Multi-GPU sharding is only supported for the qwen "
+ "backend. Ignoring --num_gpus/--gpu_id.")
+ num_gpus = 1
+ gpu_id = 0
+
+ # โ”€โ”€ Load qwen model (once) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ qwen_model = None
+ qwen_processor = None
+
+ if backend == BACKEND_QWEN:
+ gpu_id_for_model = gpu_id if num_gpus > 1 else None
+ qwen_model, qwen_processor = load_qwen_model(
+ args.model,
+ device_map=args.device_map,
+ torch_dtype=args.torch_dtype,
+ min_pixels=args.min_pixels,
+ max_pixels=args.max_pixels,
+ gpu_id=gpu_id_for_model,
+ )
+
+ # โ”€โ”€ Run โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+ caption_batch(
+ render_root=args.render_root,
+ backend=backend,
+ system_prompt=system_prompt,
+ output_json=args.output_json,
+ downsample_rate=args.downsample_rate,
+ max_frames_per_view=args.max_frames_per_view,
+ max_tokens=args.max_tokens,
+ views=args.views,
+ hints=load_hints(args.hints_json),
+ vision_input=args.vision_input,
+ gpu_id=gpu_id, num_gpus=num_gpus,
+ num_workers=args.num_workers,
+ qwen_model=qwen_model,
+ qwen_processor=qwen_processor,
+ model_name=args.model,
+ api_key=args.api_key,
+ base_url=args.base_url,
+ max_retries=args.max_retries,
+ thinking_level=args.thinking_level,
+ media_resolution=args.media_resolution,
+ thinking_budget=args.thinking_budget,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/vlm_caption/classify_category.py b/data_process/vlm_caption/classify_category.py
new file mode 100644
index 0000000..c236663
--- /dev/null
+++ b/data_process/vlm_caption/classify_category.py
@@ -0,0 +1,366 @@
+"""Classify 3D assets into body-plan categories using a local Qwen3-VL model.
+
+Input is a 2x2 multi-view grid PNG per asset (front, back, plus two side
+views, T-pose). Two render layouts are supported and auto-detected:
+
+ Flat (e.g. objaverse): <render_root>/<asset_id>.png
+ Nested (e.g. truebones): <render_root>/<NAME-MOTION>/tpose_grid.png
+ Object types are grouped by the segment before '-'; one
+ classification per type is then propagated to every clip of that
+ type.
+
+Output:
+ * ``category_groups.json`` โ€” {category: [asset_id, ...]} for assets
+ that classified successfully.
+ * ``category_groups_errors.json`` (sibling, same directory)
+ โ€” {asset_id: error_message} for assets
+ whose retries were exhausted. ERROR
+ entries are NOT in category_groups.json
+ so a re-run will retry them automatically;
+ the error log persists the last reason.
+
+Usage:
+ # Objaverse (flat layout)
+ python -m data_process.vlm_caption.classify_category \
+ --render_root dataset/render/objaverse_tpose
+
+ # Truebones (nested layout)
+ python -m data_process.vlm_caption.classify_category \
+ --render_root dataset/render/truebones_tpose
+"""
+
+import argparse
+import time
+from collections import defaultdict
+from pathlib import Path
+
+from loguru import logger
+from PIL import Image
+from tqdm import tqdm
+
+from data_process.vlm_caption.prompts import CATEGORY_PROMPT
+from data_process.utils.vlm import (
+ add_qwen_model_args,
+ flatten_rgba,
+ load_json,
+ load_qwen_model,
+ qwen_generate,
+ save_json,
+)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Constants
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+VALID_CATEGORIES = [
+ "bipedal", "quadrupedal", "insectoid", "avian", "marine",
+ "serpentine", "articulated_rigid",
+]
+UNKNOWN_CATEGORY = "unknown" # fallback when the model output matches no valid category
+
+# Per-asset retry policy: up to NUM_RETRIES additional attempts after the
+# initial try (5 total). Retries cover both exceptions (image I/O, CUDA OOM,
+# generation failure) and unparseable model outputs (UNKNOWN_CATEGORY) so
+# stochastic decoding gets a second chance to land on a valid category.
+NUM_RETRIES = 4
+RETRY_BACKOFF_SECONDS = 2.0
+
+SAVE_EVERY = 20 # assets per incremental JSON flush (both output files)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Core classification
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def classify_asset(
+ tpose_image,
+ asset_id,
+ model,
+ processor,
+ max_tokens=10,
+ system_prompt=CATEGORY_PROMPT,
+ do_sample=False,
+):
+ # type: (Image.Image, str, object, object, int, str, bool) -> str
+ """Classify one asset from its 2x2 T-pose grid PNG.
+
+ Returns one of ``VALID_CATEGORIES`` or ``UNKNOWN_CATEGORY`` if the
+ model output cannot be parsed into a known category.
+ """
+ messages = [
+ {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
+ {"role": "user", "content": [
+ {"type": "text", "text": (
+ "Classify this 3D asset (id: '{}'). The image is a 2x2 grid "
+ "of four canonical T-pose views.".format(asset_id)
+ )},
+ {"type": "image", "image": tpose_image},
+ ]},
+ ]
+
+ raw = qwen_generate(model, processor, messages, max_tokens=max_tokens,
+ do_sample=do_sample).lower()
+ # Accept "articulated rigid" (space) as well as "articulated_rigid" (underscore).
+ raw_normalized = raw.replace(" ", "_").strip()
+ # Bare token after stripping quotes/punctuation โ€” the expected reply.
+ bare = raw_normalized.strip("_*`'\".,;:!?()[]\n\t ")
+
+ if bare in VALID_CATEGORIES:
+ category = bare
+ else:
+ # Take the category that appears EARLIEST in the response, not the
+ # first one in VALID_CATEGORIES order: "quadrupedal (not bipedal)"
+ # must resolve to quadrupedal.
+ hits = sorted((raw_normalized.find(v), v)
+ for v in VALID_CATEGORIES if v in raw_normalized)
+ if len(hits) > 1:
+ logger.warning("Ambiguous category response '{}' for '{}': {} โ€” "
+ "taking the earliest match '{}'".format(
+ raw, asset_id,
+ ", ".join(v for _, v in hits), hits[0][1]))
+ category = hits[0][1] if hits else None
+ if category is None:
+ logger.warning("Unexpected category '{}' for '{}', mapping to '{}'".format(
+ raw, asset_id, UNKNOWN_CATEGORY))
+ category = UNKNOWN_CATEGORY
+
+ logger.info("Category for '{}': {}".format(asset_id, category))
+ return category
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# Asset discovery + batch processing
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def _discover_assets(render_root):
+ # type: (Path) -> Tuple[str, List[Tuple[str, Path]]]
+ """Locate (asset_id, tpose_png) pairs under ``render_root``.
+
+ Auto-detects the layout:
+ * flat โ€” ``<render_root>/<asset_id>.png`` (one render per asset).
+ * nested โ€” ``<render_root>/<NAME-MOTION>/tpose_grid.png`` (multi-clip;
+ grouped by the segment before '-' so every object type is
+ classified exactly once via its first available render).
+
+ Returns ``(layout, items)`` where ``items`` is sorted by asset_id.
+ """
+ flat_pngs = sorted(p for p in render_root.glob("*.png") if p.is_file())
+ if flat_pngs:
+ items = [(p.stem, p) for p in flat_pngs]
+ return "flat", items
+
+ char_dirs = sorted(
+ d for d in render_root.iterdir()
+ if d.is_dir() and (d / "tpose_grid.png").exists()
+ )
+ type_to_png = {} # type: Dict[str, Path]
+ for d in char_dirs:
+ obj_type = d.name.split("-", 1)[0] if "-" in d.name else d.name
+ type_to_png.setdefault(obj_type, d / "tpose_grid.png")
+ items = sorted(type_to_png.items())
+ return "nested", items
+
+
+def _error_log_path_for(category_groups_json):
+ # type: (str) -> str
+ """Sibling error-log path next to ``category_groups.json``."""
+ p = Path(category_groups_json)
+ return str(p.with_name(p.stem + "_errors.json"))
+
+
+def classify_batch(
+ render_root,
+ model,
+ processor,
+ category_groups_json=None,
+ max_tokens=10,
+ system_prompt=CATEGORY_PROMPT,
+):
+ """Classify every asset under ``render_root``.
+
+ Outputs:
+ * ``category_groups.json`` โ€” {category: [asset_id, ...]} for the
+ assets that classified successfully.
+ * ``<stem>_errors.json`` โ€” {asset_id: error_message} for assets
+ whose retries were exhausted (siblings of the groups JSON).
+
+ Both files are read on entry for resumability โ€” successful asset_ids
+ are skipped, and stale error entries (now-successful assets) are
+ pruned. Re-runs only re-attempt assets in the error log plus any
+ assets the discovery pass found that aren't yet classified.
+ """
+ render_root = Path(render_root)
+
+ if category_groups_json is None:
+ category_groups_json = str(render_root / "category_groups.json")
+ error_log_path = _error_log_path_for(category_groups_json)
+
+ existing_groups = load_json(category_groups_json) # type: Dict[str, List[str]]
+ already_classified = {} # type: Dict[str, str]
+ for cat, members in existing_groups.items():
+ for asset_id in members:
+ already_classified[asset_id] = cat
+
+ # Prune stale errors (assets now successfully classified) and retain
+ # the rest as the running error log for this run.
+ existing_errors = load_json(error_log_path) # type: Dict[str, str]
+ asset_errors = {
+ aid: err for aid, err in existing_errors.items()
+ if aid not in already_classified
+ } # type: Dict[str, str]
+
+ layout, items = _discover_assets(render_root)
+ logger.info("Layout: {} | Found {} assets in {}".format(
+ layout, len(items), render_root))
+
+ asset_categories = dict(already_classified) # type: Dict[str, str]
+ to_classify = [(aid, p) for aid, p in items if aid not in already_classified]
+ logger.info("{} already classified, {} remaining ({} previously errored)".format(
+ len(already_classified), len(to_classify), len(asset_errors)))
+
+ pbar = tqdm(to_classify, desc="Classifying", unit="asset",
+ dynamic_ncols=True)
+ total_attempts = NUM_RETRIES + 1
+ n_unsaved = 0
+
+ def _flush():
+ """Write both output files (called every SAVE_EVERY assets)."""
+ _save_category_groups(asset_categories, category_groups_json)
+ save_json(asset_errors, error_log_path)
+
+ try:
+ for asset_id, png_path in pbar:
+ pbar.set_postfix_str(asset_id, refresh=False)
+ result = None
+ last_error = None
+ for attempt in range(1, total_attempts + 1):
+ try:
+ tpose_image = flatten_rgba(Image.open(str(png_path)))
+ # Greedy first, sampled on retries โ€” a greedy retry would
+ # deterministically repeat the same unparseable output.
+ category = classify_asset(
+ tpose_image, asset_id,
+ model=model, processor=processor,
+ max_tokens=max_tokens, system_prompt=system_prompt,
+ do_sample=attempt > 1,
+ )
+ if category != UNKNOWN_CATEGORY:
+ result = category
+ break
+ last_error = ValueError(
+ "model output did not match any valid category")
+ except Exception as e:
+ last_error = e
+ logger.warning("Attempt {}/{} failed for {}: {}".format(
+ attempt, total_attempts, asset_id, e))
+
+ if attempt < total_attempts:
+ time.sleep(RETRY_BACKOFF_SECONDS)
+
+ if result is not None:
+ asset_categories[asset_id] = result
+ asset_errors.pop(asset_id, None) # success clears any prior error
+ else:
+ # Exhausted retries on either exceptions or unparseable output.
+ # Mark as ERROR so _save_category_groups drops it from the JSON
+ # and a later re-run will retry from scratch. Record the reason
+ # in the sibling error log for visibility.
+ if isinstance(last_error, ValueError):
+ err_msg = "All {} attempts produced unparseable output".format(
+ total_attempts)
+ else:
+ err_msg = "All {} attempts failed: {}".format(
+ total_attempts, last_error)
+ logger.error("[{}] {}".format(asset_id, err_msg))
+ asset_categories[asset_id] = "ERROR: {}".format(last_error)
+ asset_errors[asset_id] = err_msg
+
+ # Rewriting both JSONs per asset is O(n^2) over a large run;
+ # batch it (a crash loses at most SAVE_EVERY classifications).
+ n_unsaved += 1
+ if n_unsaved >= SAVE_EVERY:
+ _flush()
+ n_unsaved = 0
+ finally:
+ _flush()
+ n_ok = sum(
+ 1 for c in asset_categories.values()
+ if isinstance(c, str) and not c.startswith("ERROR:")
+ )
+ logger.info("Done. {} classified, {} errored (logged to {})".format(
+ n_ok, len(asset_errors), error_log_path))
+ return asset_categories
+
+
+def _save_category_groups(asset_categories, output_path):
+ # type: (Dict[str, str], str) -> None
+ """Invert {asset_id: category} into {category: [asset_id, ...]} and save.
+
+ ERROR entries are dropped so a re-run will retry those assets. After
+ retries are exhausted (in classify_batch) the asset is marked ERROR
+ and recorded in the sibling error log instead of leaking into the
+ groups JSON.
+ """
+ assets_by_category = defaultdict(list) # type: Dict[str, List[str]]
+ for asset_id, cat in sorted(asset_categories.items()):
+ if isinstance(cat, str) and cat.startswith("ERROR:"):
+ continue
+ assets_by_category[cat].append(asset_id)
+
+ category_assets = {
+ cat: sorted(assets) for cat, assets in sorted(assets_by_category.items())
+ } # type: Dict[str, List[str]]
+ save_json(category_assets, output_path)
+
+
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# CLI
+# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ description="Classify 3D assets into body-plan categories "
+ "using a local Qwen3-VL model."
+ )
+
+ parser.add_argument('--render_root', type=str, required=True,
+ help='Root directory holding T-pose 2x2 grid PNGs '
+ '(flat <asset_id>.png or nested '
+ '<NAME-MOTION>/tpose_grid.png โ€” auto-detected).')
+
+ parser.add_argument('--category_groups_json', type=str, default=None,
+ help='Output JSON file mapping '
+ '{category: [asset_id, ...]}. '
+ 'Defaults to <render_root>/category_groups.json. '
+ 'A sibling <stem>_errors.json records assets '
+ 'whose retries were exhausted. Both files are '
+ 're-read on entry for resumability.')
+ add_qwen_model_args(parser)
+ parser.add_argument('--max_tokens', type=int, default=10,
+ help='Max new tokens to generate (default: 10).')
+
+ return parser.parse_args()
+
+
+def main():
+ args = parse_args()
+
+ qwen_model, qwen_processor = load_qwen_model(
+ args.model,
+ device_map=args.device_map,
+ torch_dtype=args.torch_dtype,
+ min_pixels=args.min_pixels,
+ max_pixels=args.max_pixels,
+ )
+
+ classify_batch(
+ render_root=args.render_root,
+ model=qwen_model, processor=qwen_processor,
+ category_groups_json=args.category_groups_json,
+ max_tokens=args.max_tokens,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/data_process/vlm_caption/prompts.py b/data_process/vlm_caption/prompts.py
new file mode 100644
index 0000000..91aa04c
--- /dev/null
+++ b/data_process/vlm_caption/prompts.py
@@ -0,0 +1,397 @@
+"""System prompts for the VLM captioning stage.
+
+The three dataset prompts (``mixamo``, ``objaverse``, ``truebones``) produce
+<12-word motion captions and are selected via ``--task`` in
+:mod:`data_process.vlm_caption.caption_motion`. ``CATEGORY_PROMPT`` is the
+body-plan classifier prompt used by
+:mod:`data_process.vlm_caption.classify_category`.
+
+The caption prompts share one scaffold โ€” camera-input explanation,
+observation protocol steps 2โ€“5, the core caption rules, and the closing
+line โ€” assembled by :func:`_caption_prompt` so a protocol change lands in
+every dataset at once. Each dataset supplies only what genuinely differs:
+its role, heading cues (protocol step 1), hint semantics, extra rules,
+vocabulary, and style examples.
+"""
+
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+# Shared caption-prompt scaffold
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+
+_INPUT_CAMERAS = (
+ "# Input\n"
+ "Four synchronized cameras 90 degrees apart at fixed elevation, labeled "
+ "only by their relative azimuth about the vertical axis (0ยฐ, 90ยฐ, 180ยฐ, "
+ "270ยฐ; 0ยฐ and 180ยฐ are opposite cameras, as are 90ยฐ and 270ยฐ). The "
+ "labels carry NO information about {heading} โ€” the subject's world "
+ "orientation is arbitrary, so any camera may be seeing the front, back, "
+ "side, or an oblique angle. Determine facing from the body itself "
+ "(protocol step 1). The cameras are STATIC: when the subject shifts "
+ "across the frame or grows/shrinks, that is the subject translating, "
+ "never camera motion. A view looking straight along an elongated body "
+ "may show only a compact silhouette โ€” rely on the other views. The "
+ "frames of each view are uniformly sampled across the clip in "
+ "chronological order, and the views are synchronized: frame k of every "
+ "view shows the same moment. Pose can change noticeably between "
+ "consecutive sampled frames.\n\n"
+)
+
+_PROTOCOL_HEADER = "# Observation protocol (silent โ€” write only the caption)\n"
+
+# Steps 2โ€“5 are identical logic everywhere; step 1 (heading cues) is the
+# genuinely dataset-specific part and is supplied by each spec.
+_PROTOCOL_STEPS_2_TO_5 = (
+ "2. CHIRALITY. From the front, {possessive} left side is on the viewer's "
+ "right (mirror). Apply consistently โ€” never label a limb by which side of "
+ "the camera frame it sits on.\n"
+ "3. SCAN every frame across all four views; do not infer from first/last "
+ "alone. Note which body parts change pose and how.\n"
+ "4. CLASSIFY: translation (moves through space), rotation in place, "
+ "articulation only ({articulation}), or held pose. Direction is read "
+ "relative to the heading from step 1. Say 'in place' only for rotation "
+ "without translation, or for locomotion-style movement without "
+ "translation ({in_place_example}); never as a default. Compare the "
+ "subject's facing at the START and END of the clip: if it differs, a "
+ "turn happened and belongs in the caption ('turns around and walks "
+ "away').\n"
+ "5. PICK the most specific verb that fits the kinematics.\n\n"
+)
+
+_RULES_CORE = (
+ "# Rules\n"
+ "- Format: '{subject} [action].' (one sentence, <12 words)\n"
+ "- ONE dominant action โ€” or a short two-phase sequence ('X and then Y') "
+ "when the clip clearly has two stages.\n"
+ "- A cyclic motion (walk cycle, idle sway) is described ONCE as a "
+ "continuous action, never per repetition; reserve the two-phase form "
+ "for genuinely distinct stages.\n"
+ "- A concurrent pose or secondary movement may be attached with "
+ "'while ...' / 'with ...' ({concurrent_examples}).\n"
+ "- Mention direction (forward/backward/left/right/up/down/in place) only "
+ "if clearly observed; omit rather than guess.\n"
+ "- All directions and chirality are object-relative (step 1), never "
+ "camera-relative. {uncertain_fallback}\n"
+)
+
+_RULES_TAIL = (
+ "- No adverbs, no appearance{no_appearance_extra}.\n"
+ "- Only say 'stands still' if the pose is truly unchanged across ALL "
+ "frames; {still_clause}.\n\n"
+)
+
+_EXAMPLES_HEADER = (
+ "# Style examples (format only โ€” do NOT copy unless the motion matches)\n"
+)
+
+_CLOSING = ("Respond with ONLY the caption sentence โ€” plain text, no "
+ "quotation marks.")
+
+
+def _caption_prompt(role, heading, blocks_after_input, step1, possessive,
+ articulation, in_place_example, vocab_block, subject,
+ uncertain_fallback, concurrent_examples, specific_rules,
+ no_appearance_extra, still_clause, examples):
+ # type: (...) -> str
+ """Assemble one dataset's caption system prompt from the shared scaffold."""
+ return (
+ "# Role\n" + role + "\n\n"
+ + _INPUT_CAMERAS.format(heading=heading)
+ + blocks_after_input
+ + _PROTOCOL_HEADER
+ + step1
+ + _PROTOCOL_STEPS_2_TO_5.format(
+ possessive=possessive, articulation=articulation,
+ in_place_example=in_place_example)
+ + vocab_block
+ + _RULES_CORE.format(
+ subject=subject, uncertain_fallback=uncertain_fallback,
+ concurrent_examples=concurrent_examples)
+ + specific_rules
+ + _RULES_TAIL.format(
+ no_appearance_extra=no_appearance_extra, still_clause=still_clause)
+ + _EXAMPLES_HEADER
+ + examples + "\n"
+ + _CLOSING
+ )
+
+
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+# Mixamo: concise human motion captioning
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+
+MIXAMO_PROMPT = _caption_prompt(
+ role=(
+ "You caption clips from the Mixamo 3D human animation dataset "
+ "(locomotion, combat, sports, dance, gestures, idle poses). Every "
+ "clip is a single human character, rendered as a faceless "
+ "untextured mannequin."
+ ),
+ heading="the character's facing direction",
+ blocks_after_input=(
+ "A reference label from the Mixamo catalogue may be provided in "
+ "the text after the views (e.g., '180 Turn With Briefcase', "
+ "'Jab Cross'). Treat it as "
+ "a helpful HINT, not ground truth: describe what the frames show, "
+ "and use the label to disambiguate when the frames are unclear or "
+ "to pick a more specific verb for an action you can verify. Do not "
+ "copy it verbatim, and never carry over an element the frames do "
+ "not support. Props named in the label are not rendered โ€” never "
+ "name them; when the pose clearly implies one, express it as "
+ "'as if holding/carrying something'.\n\n"
+ ),
+ step1=(
+ "1. ESTABLISH HEADING. The mannequin has no face or eyes โ€” read "
+ "heading from body geometry: toes point forward, knees bend "
+ "forward while elbows bend backward, and the chest is convex where "
+ "the back is flatter. Whichever camera the chest and toes most "
+ "directly point at shows the person's FRONT; the opposite shows "
+ "the BACK; the other two show their own LEFT and RIGHT.\n"
+ ),
+ possessive="the person's",
+ articulation="limbs move, pelvis fixed",
+ in_place_example="'runs in place', 'marches in place'",
+ vocab_block="",
+ subject="A person",
+ uncertain_fallback=(
+ "If heading is uncertain, say 'one arm' / 'one leg' instead of "
+ "guessing left/right."
+ ),
+ concurrent_examples=("'walks forward with arms raised', "
+ "'claps while seated'"),
+ specific_rules=(
+ "- Mention a body part only when essential to disambiguate.\n"
+ ),
+ no_appearance_extra=", no clothing",
+ still_clause="subtle sway/breathing still counts as motion",
+ examples=(
+ "- A person walks forward.\n"
+ "- A person runs in place.\n"
+ "- A person punches with the right fist.\n"
+ "- A person raises one arm while seated.\n"
+ "- A person turns around as if carrying something.\n"
+ "- A person crouches and then rises back up.\n"
+ ),
+)
+
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+# Objaverse: concise topology-agnostic captioning
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+
+OBJAVERSE_PROMPT = _caption_prompt(
+ role=(
+ "You caption clips from the Objaverse 3D animation dataset โ€” "
+ "humanoids (majority), quadrupeds, avians, marine creatures, "
+ "insectoids, serpentines, and articulated rigid objects."
+ ),
+ heading="the asset's canonical heading",
+ blocks_after_input=(
+ "# Task\n"
+ "ONE short sentence describing the dominant motion in "
+ "OBJECT-RELATIVE terms. Subject is 'An object', body parts use "
+ "natural anatomy words (arm, wing, tail, ...). When topology is "
+ "ambiguous, default to humanoid vocabulary โ€” most assets in this "
+ "dataset are humanoid.\n\n"
+ ),
+ step1=(
+ "1. ESTABLISH HEADING. Cues, in priority order: (a) "
+ "head/face/eye/snout direction, (b) spine direction shouldersโ†’hips "
+ "for quadrupeds, (c) beak/head for avians, (d) head vs. tail end for "
+ "serpentines, (e) principal translation axis for faceless rigid "
+ "assets.\n"
+ ),
+ possessive="the object's",
+ articulation="limbs/wings/tail move, body fixed",
+ in_place_example="'walks in place'",
+ vocab_block=(
+ "# Body-part vocabulary (descriptions of shape, not category labels)\n"
+ "- Humanoid/bipedal: arm, hand, leg, foot, torso, head, hip, shoulder\n"
+ "- Quadruped: front leg, hind leg, head, torso, tail\n"
+ "- Winged/flying: wing, head, torso, tail, leg (if visible)\n"
+ "- Serpentine: head, body, tail\n"
+ "- Aquatic: fin, tail, head, body\n"
+ "- Insectoid: leg, body, head\n"
+ "- Articulated rigid: part, segment, base, top, arm (mechanical)\n"
+ "If shape fits no row: upper/lower part, left/right side, front, "
+ "back.\n\n"
+ ),
+ subject="An object",
+ uncertain_fallback="If heading is uncertain, omit chirality.",
+ concurrent_examples=("'walks forward with arms swinging', "
+ "'rotates its base while extending an arm'"),
+ specific_rules=(
+ "- Mention a body part only when essential to disambiguate.\n"
+ "- No category names ('person', 'dog', 'dragon', 'car', 'robot', "
+ "...); anatomy words (arm, wing, tail) are allowed.\n"
+ ),
+ no_appearance_extra="/color/material/texture, no scene/lighting",
+ still_clause="subtle sway, breathing, or limb shifts still count as motion",
+ examples=(
+ "- An object walks forward with arms swinging.\n"
+ "- An object kicks with the right leg while pivoting.\n"
+ "- An object flaps its wings and rises.\n"
+ "- An object slithers forward.\n"
+ "- An object rotates its upper segment in place.\n"
+ "- An object crouches down and springs upward.\n"
+ ),
+)
+
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+# Truebones: concise animal motion captioning
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+
+TRUEBONES_PROMPT = _caption_prompt(
+ role=(
+ "You caption clips from the Truebones 3D animal animation dataset โ€” "
+ "quadrupeds, avians, marine creatures, insectoids, serpentines. "
+ "Every clip is a single non-human animal."
+ ),
+ heading="the animal's heading",
+ blocks_after_input=(
+ "A reference label may be provided in the text after the views โ€” "
+ "a human annotation that reliably names the motion (e.g., 'A "
+ "running tyrannosaurus "
+ "rex loses balance and falls to the right.'). Its ACTION is "
+ "authoritative: make it the caption's MAIN ACTION and override "
+ "it only when the frames unambiguously show a different one. Its "
+ "DIRECTIONS and left/right are not โ€” the annotator's viewpoint "
+ "is unknown, so re-derive every direction from step 1 and keep "
+ "the label's directional words only when the frames confirm "
+ "them. Do not copy the label "
+ "verbatim: rewrite it in the required format and never carry a "
+ "species word into the caption โ€” the example label above becomes "
+ "'An animal runs forward and then falls to the right.'\n\n"
+ ),
+ step1=(
+ "1. ESTABLISH HEADING. Cues, in priority order: (a) "
+ "head/snout/eyes/beak direction, (b) spine direction "
+ "shoulders/withersโ†’hips for quadrupeds and winged animals, (c) head "
+ "end vs. tail end for serpentines/fish, (d) principal direction of "
+ "locomotion (animals move forward along their facing axis).\n"
+ ),
+ possessive="the animal's",
+ articulation="limbs/wings/tail/jaw move, torso fixed",
+ in_place_example="'trots in place'",
+ vocab_block="",
+ subject="An animal",
+ uncertain_fallback=(
+ "If heading is uncertain, say 'one foreleg' / 'a wing' instead of "
+ "guessing left/right."
+ ),
+ concurrent_examples=("'walks forward with its head turned right', "
+ "'crouches with its tail raised'"),
+ specific_rules=(
+ "- Use generic body-part terms: front legs, hind legs, head, torso, "
+ "tail, wings, fins, jaw, left/right side. Mention only when "
+ "essential.\n"
+ "- Do NOT identify the species in the caption (no 'dog', 'horse', "
+ "'bear', 'snake') even if the reference label mentions one.\n"
+ ),
+ no_appearance_extra="/color/texture",
+ still_clause="subtle sway, breathing, tail flicks, ear twitches still count as motion",
+ examples=(
+ "- An animal gallops forward.\n"
+ "- An animal turns to the right.\n"
+ "- An animal rears upward on its hind legs.\n"
+ "- An animal flaps its wings and rises.\n"
+ "- An animal swims forward with its tail.\n"
+ "- An animal strikes forward with its tail curled.\n"
+ ),
+)
+
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+# Category: body-type classification
+# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+
+CATEGORY_PROMPT = (
+ "# Role\n"
+ "You classify the body plan of a single 3D asset into exactly ONE of "
+ "seven categories. The output is consumed by an automated pipeline โ€” "
+ "it must be one exact token, no prose, no punctuation.\n\n"
+ "# Input\n"
+ "ONE image: a 2x2 grid of four canonical views of the asset in T-pose "
+ "(front, back, and two orthogonal side / top views). The exact panel "
+ "ordering is not guaranteed. Use ALL four panels โ€” never decide from a "
+ "single panel โ€” and reason from silhouette, limb count, and proportions. "
+ "T-pose means a canonical rest pose with limbs spread; there is no "
+ "motion to read. If the asset appears horizontal in some panels, that "
+ "is just a top-down or side view of the same rest pose โ€” judge body "
+ "plan from the panel that most clearly shows the limb topology.\n\n"
+ "# Categories\n"
+ "- bipedal: organism with TWO legs as primary stance / locomotion. "
+ "Humans, humanoids, apes, bipedal dinosaurs (T-Rex, raptors), "
+ "flightless upright birds (ostrich, penguin), humanoid robots and "
+ "mechs (head + torso + two arms + two legs).\n"
+ "- quadrupedal: organism with FOUR legs as primary stance. Dogs, "
+ "cats, horses, bears, deer, lizards, turtles, four-legged dinosaurs, "
+ "quadruped robots. Centaurs and four-leg hybrids belong here.\n"
+ "- insectoid: arthropod body plan โ€” SIX OR MORE articulated legs, "
+ "typically segmented thorax/abdomen, often antennae. Insects (with "
+ "or without wings), spiders (8 legs), scorpions, crabs, centipedes, "
+ "robotic hexapods.\n"
+ "- avian: vertebrate defined by TWO LARGE WINGS used for flight, "
+ "usually paired with legs. Flying birds, bats, pterosaurs, dragons, "
+ "wyverns, winged humanoids / angels. Wings must be a dominant "
+ "feature, not vestigial decoration.\n"
+ "- marine: streamlined aquatic body propelled by fins/flippers/tail, "
+ "with NO walking legs. Fish, sharks, rays, dolphins, whales, "
+ "octopuses, seahorses. Mermaids belong here.\n"
+ "- serpentine: elongated, LIMBLESS, tubular body. Snakes, eels, "
+ "worms, legless lizards. Any visible legs disqualify this category.\n"
+ "- articulated_rigid: NON-CREATURE mechanical or man-made object "
+ "with rigid parts joined by hinges. Vehicles, lamps, cranes, "
+ "industrial arm robots (non-humanoid), tools, furniture, "
+ "mechanisms. Use this ONLY when the asset clearly does not read as "
+ "a creature.\n\n"
+ "# Decision priority (apply in order; stop at the first match)\n"
+ "1. Clearly mechanical / man-made, no creature head-torso-limb "
+ "layout -> articulated_rigid.\n"
+ "2. Six or more legs OR arthropod silhouette (segmented body, "
+ "exoskeleton plates, antennae) -> insectoid. THIS OVERRIDES WINGS โ€” "
+ "a bee or butterfly is insectoid, not avian.\n"
+ "3. Two large flight wings paired with the torso, wings are a "
+ "dominant feature -> avian.\n"
+ "4. No walking limbs, body shaped for swimming with fins/tail "
+ "-> marine.\n"
+ "5. No limbs at all, long tubular body -> serpentine.\n"
+ "6. Four legs as primary ground stance -> quadrupedal.\n"
+ "7. Two legs as primary ground stance -> bipedal.\n"
+ "8. Anything still ambiguous -> articulated_rigid.\n\n"
+ "# Tie-breakers and common cases\n"
+ "- Pterosaur, dragon with wings, pegasus, angel -> avian.\n"
+ "- Wingless dragon with four legs -> quadrupedal.\n"
+ "- Wyvern (two legs + two wings) -> avian.\n"
+ "- Penguin, ostrich, T-Rex, raptor -> bipedal.\n"
+ "- Bee, butterfly, dragonfly, ladybug -> insectoid.\n"
+ "- Centaur -> quadrupedal. Mermaid -> marine.\n"
+ "- Spider robot / hexapod robot -> insectoid.\n"
+ "- Humanoid robot / mech with two arms and two legs -> bipedal.\n"
+ "- Quadruped robot (Spot-style, four legs) -> quadrupedal.\n"
+ "- Industrial robot arm, crane, lamp, loose mechanism -> "
+ "articulated_rigid.\n"
+ "- Lizard with small limbs (gecko, salamander) -> quadrupedal.\n"
+ "- Snake, eel, worm -> serpentine.\n\n"
+ "# Output\n"
+ "Reply with EXACTLY one of these tokens, lowercase, no punctuation, "
+ "no explanation, no quotes:\n"
+ "bipedal | quadrupedal | insectoid | avian | marine | serpentine | "
+ "articulated_rigid"
+)
+
+# โ”€โ”€ Caption task registry โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+TASK_PROMPTS = {
+ "mixamo": MIXAMO_PROMPT,
+ "objaverse": OBJAVERSE_PROMPT,
+ "truebones": TRUEBONES_PROMPT,
+}
+
+def get_prompt(task):
+ # type: (str) -> str
+ """Return the caption system prompt for *task*; raises ValueError if unknown."""
+ if task not in TASK_PROMPTS:
+ raise ValueError(
+ "Unknown task '{}'. Available: {}".format(
+ task, ", ".join(sorted(TASK_PROMPTS.keys()))
+ )
+ )
+ return TASK_PROMPTS[task]
+
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..f08c08e
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,71 @@
+# UniMate unified environment โ€” one env for everything: data processing,
+# rendering, training, inference and evaluation (Python 3.10, CUDA 12.4).
+#
+# conda create -n unimate python=3.10 -y
+# conda activate unimate
+# pip install "setuptools<81"
+# pip install -r requirements.txt --no-build-isolation
+#
+# The Motion git dependency has a legacy setup.py that imports pkg_resources,
+# which setuptools >= 81 no longer ships. Isolated builds always use the
+# newest setuptools, so the install must run with --no-build-isolation
+# against an env setuptools < 81 โ€” installed by the step above (fresh conda
+# envs ship setuptools >= 81 too); the pin below keeps it from upgrading.
+#
+# Only direct dependencies are listed; everything else (CUDA runtime libs,
+# tokenizers, HTTP stack, ...) is resolved transitively by pip.
+
+# PyTorch cu124 wheels (torch / torchvision "+cu124" pins below)
+--extra-index-url https://download.pytorch.org/whl/cu124
+# Blender official wheel index โ€” PyPI no longer hosts cp310 bpy wheels
+--extra-index-url https://download.blender.org/pypi/
+
+setuptools<81 # keeps pkg_resources available (Motion build)
+
+# โ”€โ”€ Core / training โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+numpy==2.2.6
+torch==2.5.1+cu124
+torchvision==0.20.1+cu124
+torch-geometric==2.7.0 # GCN graph embedding (use_graph_emb)
+torchdiffeq==0.2.5 # ODE/SDE integrators for flow matching
+einops==0.8.2
+accelerate==0.30.1 # multi-GPU + mixed-precision training
+tensorboard==2.20.0 # training logs (SummaryWriter)
+
+# โ”€โ”€ Text conditioning (T5 / CLIP / BERT) & VLM captioning โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+transformers==4.57.6 # T5/BERT text encoders; >=4.57 required by the local Qwen3-VL captioner
+sentencepiece==0.2.1 # T5 tokenizer backend
+clip @ git+https://github.com/openai/CLIP.git # CLIP text encoder
+spacy # T5 conditioner text normalization
+num2words # number โ†’ word expansion in text normalization
+openai # OpenAI-compatible backends (captioning; DeepSeek/GPT joint annotation)
+google-genai # Gemini caption backend ("from google import genai")
+
+# OPTIONAL โ€” fused attention for the local Qwen3-VL captioner. The loader
+# falls back to SDPA when absent, and the prebuilt wheels only exist for a
+# few exact torch/CUDA/Python combinations (a source build takes an hour+),
+# so it is not installed by default:
+# pip install flash-attn==2.7.4.post1 --no-build-isolation
+
+# โ”€โ”€ Motion / animation library (Holden-style Animation / Quaternions /
+# InverseKinematics; used across data_process/) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+Motion @ git+https://github.com/inbar-2344/Motion.git
+
+# โ”€โ”€ Blender as a Python module (motion_export/, motion_rendering/,
+# mesh_animation/). The EEVEE render stage requires the pip module;
+# export/animate can alternatively run via `blender -b -P`. Pinned to the
+# last cp310 release, served from download.blender.org (see index above);
+# 4.1+ drops Python 3.10 and 4.2+ changes the EEVEE API. โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+bpy==4.0.0
+
+# โ”€โ”€ Visualization / imaging โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+matplotlib==3.10.8
+imageio==2.37.3
+imageio-ffmpeg==0.6.0 # mp4 encoding backend for imageio
+pillow==12.2.0
+
+# โ”€โ”€ CLI / logging โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+tyro==1.0.12 # config + CLI overrides
+rich==14.3.3
+loguru
+tqdm==4.67.3