[SIGGRAPH Asia 2026] UniMate: One Unified Model to Animate Diverse Skeletons
Python
Release the UniML3D dataset processing pipeline
Friedrich-Mf697a3386ed3
diff --git a/.gitignore b/.gitignorenew file mode 100644index 0000000..98b31db--- /dev/null+++ b/.gitignore@@ -0,0 +1,95 @@+# ---------- Not yet open-sourced ----------+# Only the data-processing pipeline (data_process/) is public at this stage;+# the training / inference code lives in the private motion-gen repo and will+# be released here later. Patterns are root-anchored on purpose.+/src/+/configs/+/scripts/+/docs/+/CLAUDE.md+/reference/++# ---------- Data / experiments (root-level only) ----------+/dataset/+/baselines/+/outputs/+/output/+/logs/+/*debug/+/animation_debugs/+wandb/+runs/+mlruns/+lightning_logs/+tensorboard/+tb_logs/++# ---------- ML artifacts ----------+*.ckpt+*.pt+*.pth+*.safetensors+*.onnx++# ---------- Python ----------+__pycache__/+*.py[cod]+*.so+.ipynb_checkpoints/+*.egg-info/+dist/+build/+*.egg++# ---------- Virtual environments ----------+.venv/+venv/+env/+ENV/+.conda/++# ---------- Caches / coverage ----------+.cache/+.pytest_cache/+.mypy_cache/+.ruff_cache/+.dmypy.json+.coverage+.coverage.*+htmlcov/+coverage.xml+.tox/++# ---------- SLURM / cluster ----------+# Cluster-specific launchers โ not part of the public pipeline.+/data_process/scripts/slurm/+slurm-*.out+slurm-*.err+*.slurm.log+core+core.*++# ---------- Editor / IDE ----------+.claude/+.vscode/+.idea/+*.iml+.cursor/+.history/++# ---------- OS ----------+.DS_Store+Thumbs.db+Desktop.ini++# ---------- Logs / temp ----------+*.log+*.tmp+*.temp+*.bak+*.old+*.swp+*.swo+*~+.env+.env.localdiff --git a/README.md b/README.mdindex ad8e7a2..b5513df 100644--- a/README.md+++ b/README.md@@ -4,9 +4,10 @@<p align="center"><a href="https://linzhanm.github.io/unimate/"><img alt="Project Page" src="https://img.shields.io/badge/Project_Page-6D28D9?style=for-the-badge&logo=githubpages&logoColor=white"></a>- <a href="https://linzhanmou.com/unimate/resources/unimate.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-BF40FF?style=for-the-badge&logo=googledocs&logoColor=white"></a>+ <a href="https://linzhanmou.com/unimate/resources/unimate.pdf"><img alt="Paper" src="https://img.shields.io/badge/Paper-A855F7?style=for-the-badge&logo=googledocs&logoColor=white"></a><a href="https://linzhanmou.com/unimate/interactive.html"><img alt="Interactive Demo" src="https://img.shields.io/badge/Interactive_Demo-0EA5E9?style=for-the-badge&logo=threedotjs&logoColor=white"></a>- <img alt="Venue" src="https://img.shields.io/badge/SIGGRAPH_Asia-2026-1E1B4B?style=for-the-badge">+ <a href="https://huggingface.co/Linzhan"><img alt="Hugging Face Dataset" src="https://img.shields.io/badge/Dataset-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000000"></a>+ <a href="https://asia.siggraph.org/2026/"><img alt="Venue" src="https://img.shields.io/badge/SIGGRAPH_Asia-2026-6D28D9?style=for-the-badge&labelColor=1E1B4B"></a></p><p align="center">@@ -25,41 +26,45 @@<img src="assets/teaser.png" alt="UniMate teaser" width="100%"></div>-<!---<div align="center">--https://github.com/user-attachments/assets/a4efca07-8673-4ddf-8057-b8be7b008265--</div>--->-----## ๐ News+## ๐ฅ News-- **[TODO]** Training code and the UniML3D dataset will be released in this repository around August 2026.+- **[TODO]** Training and inference code will be released in this repository soon.+- **[2026-08-30]** The **UniML3D dataset** and its [data-processing pipeline](data_process/) are released. ๐- **[2026-08-01]** Our [Interactive Demo](https://linzhanmou.com/unimate/interactive.html) is live โ browse our animation results in 3D. ๐ฎ- **[2026-07-18]** UniMate is accepted to SIGGRAPH Asia 2026! ๐-## ๐ผ๏ธ Inference+## ๐ ๏ธ Environment Setup-Given a rigged 3D asset and a text prompt, UniMate generates articulated motion for arbitrary skeletons in real time โ with no per-skeleton retraining and no test-time optimization.+All components share a single conda environment, specified in [`requirements.txt`](requirements.txt):-<div align="center">- <img src="assets/qualitative.png" alt="Qualitative results" width="100%">-</div>+```bash+conda create -n unimate python=3.10 -y+conda activate unimate+pip install "setuptools<81"+pip install -r requirements.txt --no-build-isolation+```-## ๐ Dataset+## ๐ Dataset & Data ProcessingWe introduce **UniML3D**, a large-scale dataset of 13,006 text-paired motion sequences covering diverse skeletal topologies โ bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects โ all brought into a unified canonicalization.+The raw source assets are available on the Hugging Face Hub: [Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters), [Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) and [Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata and renders only). The Truebones ZOO animal motions themselves are a commercial asset pack whose license does not permit redistribution โ please purchase the pack directly from [Truebones](https://truebones.com); our pipeline consumes the stock `Truebone_Z-OO` folder layout as-is.+<div align="center"><img src="assets/dataset_overview.png" alt="UniML3D dataset overview" width="100%"></div>-## ๐ License+See [`data_process/README.md`](data_process/README.md) for the full data processing pipeline that turns the raw assets into UniML3D (download โ export โ rendering โ captioning โ joint annotation โ feature extraction โ animation).++## ๐ฌ Inference-This project is released under the [MIT License](LICENSE).+Given a rigged 3D asset and a text prompt, UniMate generates articulated motion for arbitrary skeletons in real time โ with no per-skeleton retraining and no test-time optimization.++<div align="center">+ <img src="assets/qualitative.png" alt="Qualitative results" width="100%">+</div>## ๐ Citation@@ -73,3 +78,9 @@ If you find UniMate useful in your research, please consider citing our work:year={2026}}```++## โ๏ธ License++The code in this repository is released under the [MIT License](LICENSE).++The datasets remain governed by the licenses of their original sources: the [Mixamo](https://www.mixamo.com/) assets by Adobe's Mixamo terms of use, the [Objaverse-XL](https://objaverse.allenai.org/) assets by the license attached to each original object, and the Truebones ZOO motions by [Truebones](https://truebones.com)' commercial license. Please review and comply with the respective source licenses before using the data.diff --git a/data_process/README.md b/data_process/README.mdnew file mode 100644index 0000000..c2dfeda--- /dev/null+++ b/data_process/README.md@@ -0,0 +1,344 @@+# UniML3D Data Processing++The pipeline that turns raw rigged assets (Truebones FBX, Mixamo FBX, Objaverse GLB) into **UniML3D** โ the text-paired, topology-annotated motion clips used to train UniMate.++All commands are run **from the repository root**.++## Table of Contents++- [Overview](#overview)+- [Requirements](#requirements)+- [Directory Layout](#directory-layout)+- [Getting the Raw Data](#getting-the-raw-data)+- [Quick Start](#quick-start)+- [Pipeline Stages](#pipeline-stages)+- [Data Formats](#data-formats)+- [Conventions](#conventions)+- [Troubleshooting](#troubleshooting)++## Overview++```mermaid+%%{init: {+ "theme": "base",+ "themeVariables": {+ "fontFamily": "-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif",+ "fontSize": "14px",+ "lineColor": "#94A3B8",+ "edgeLabelBackground": "#F1F5F9"+ },+ "flowchart": { "curve": "basis", "nodeSpacing": 40, "rankSpacing": 60 }+}}%%+flowchart LR+ raw(["๐ฆ raw FBX / GLB"])+ export("<b>1 ยท Export</b><br/>Blender โ NPZ")+ render("<b>2a ยท Render</b><br/>EEVEE multi-view")+ caption("<b>2b ยท Caption</b><br/>VLM")+ joints("<b>3 ยท Joints</b><br/>names + facing")+ extract("<b>4 ยท Extract</b><br/>canonicalize")+ animate("<b>5 ยท Animate</b><br/>rigged GLB/FBX")+ clips(["โจ training clips + cond.npy"])++ raw --> export+ raw --> render+ export -- "joint_names.json" --> joints+ render -- "frames" --> caption+ export -- "motions/*.npz" --> extract+ caption -- "captions / categories" --> extract+ joints -- "clean / face joints" --> extract+ extract --> clips+ clips --> animate+ raw -- "rigged mesh" --> animate++ classDef data fill:#1E1B4B,stroke:#1E1B4B,color:#F8FAFC+ classDef blender fill:#6D28D9,stroke:#5B21B6,color:#FFFFFF+ classDef vision fill:#0EA5E9,stroke:#0284C7,color:#FFFFFF+ classDef llm fill:#A855F7,stroke:#9333EA,color:#FFFFFF+ classDef out fill:#FFD21E,stroke:#CA8A04,color:#1E1B4B++ class raw data+ class export,animate blender+ class render,caption vision+ class joints,extract llm+ class clips out++ linkStyle default stroke:#94A3B8,stroke-width:1.5px+```++| Stage | Wrapper(s) | Input | Output |+|-------|-----------|-------|--------|+| 0. Download | `run_download.sh` | Hugging Face Hub | `dataset/raw/<dataset>/` |+| 1. Export | `run_export.sh`, `run_export_general.sh` | raw FBX / GLB | `export/<dataset>/{motions/*.npz, videos/*.mp4, tpose/*.png, joint_names.json}` |+| 2a. Render | `run_render_motion.sh`, `run_render_tpose.sh` | raw FBX / GLB (mixamo: animated characters) | `render/<dataset>/<clip>/v00{0..3}/*.png`, `render/<dataset>_tpose/<name>.png` |+| 2b. Caption | `run_caption_motion.sh`, `run_caption_category.sh` | multi-view renders | `motion_captions.json`, `category_groups.json` |+| 3. Joints | `run_joints_*.sh` | `joint_names.json` | `clean_joint_names.json`, `face_joint_names.json` |+| 4. Extract | `run_extract_features.sh` | export NPZs + stage-2/3 JSONs | per-clip NPZs, `cond.npy`, `captions.json` |+| 5. Animate | `run_animate_{motion,npz,fbx,mixamo}.sh` | motion clip + rigged mesh | animated `.glb` / `.fbx` |++Stage 4 reads the caption and joint-annotation JSONs from the export directory, so stages 2 and 3 must run first.++> [!NOTE]+> **Mixamo runs stage 5 in the middle of the pipeline.** Raw Mixamo FBXs are animation-only (armature without mesh), so the render/caption input is produced by `run_animate_mixamo.sh`: export โ animate a character โ render โ caption. The result also ships in the mixamo repo as `animation_motion_ybot/`, so this step can be skipped by downloading it.++## Requirements++The pipeline uses the shared `unimate` conda environment โ see [Environment Setup](../README.md#%EF%B8%8F-environment-setup) in the top-level README. Beyond it:++| Tool | Used by |+|------|---------|+| [Blender](https://www.blender.org/) on `PATH` | stages 1 & 5 (headless `blender -b -P`) |+| pip [`bpy`](https://pypi.org/project/bpy/) module (in the conda env) | stage 2a (EEVEE needs the module's GPU context) |+| `ffmpeg` on `PATH` | video visualizations |+| [`hf` CLI](https://hf.co/cli) | stage 0 |+| CUDA GPU | stages 2a & 2b |+| `OPENAI_API_KEY` / `GOOGLE_API_KEY` / `DEEPSEEK_API_KEY` | stage 2b (OpenAI / Gemini captioning) and stage 3 (OpenAI / DeepSeek joint annotation) API backends |++## Directory Layout++Code โ one directory per stage, bash wrappers in `scripts/`, shared code in `utils/`:++```+data_process/+โโโ scripts/ Bash entry points โ one run_<stage>_<task>.sh per task+โโโ motion_export/ Stage 1 raw assets โ NPZ (Blender headless)+โโโ motion_rendering/ Stage 2a rigged assets โ multi-view renders + T-pose grids (EEVEE)+โโโ vlm_caption/ Stage 2b multi-view renders โ captions / body-plan categories (VLM)+โโโ joint_annotation/ Stage 3 joint-name cleanup + facing-direction joint selection+โโโ feature_extraction/ Stage 4 NPZ + metadata โ canonical training clips + cond.npy+โโโ mesh_animation/ Stage 5 motion clip + rigged mesh โ animated GLB/FBX (Blender)+โโโ tools/ Standalone utilities (summary merging, FBXโGLB, QA visualizers)+โโโ utils/ Shared library code (no entry points)+```++Data โ all wrappers share one convention (override any path through the environment variables each wrapper documents in its header):++```+dataset/raw/<dataset>/ raw FBX / GLB assets+dataset/export/<dataset>/ stage-1 NPZs + previews + stage-2/3 metadata+dataset/render/<dataset>/<clip>/v00{0..3}/ multi-view renders (captioning input)+dataset/render/<dataset>_tpose/ T-pose 2x2 grids (category input)+dataset/features/<dataset>/ stage-4 training clips + cond.npy+```++`<dataset>` is one of `truebones`, `mixamo`, `objaverse`.++## Getting the Raw Data++The source assets are hosted on the Hugging Face Hub and download straight into the layout above:++```bash+bash data_process/scripts/run_download.sh mixamo # โ dataset/raw/mixamo/{animation_motion,character_refined,...}+bash data_process/scripts/run_download.sh objaverse # โ dataset/raw/objaverse/glb+bash data_process/scripts/run_download.sh truebones # โ dataset/raw/truebones (annotations only โ see below)+```++| Dataset | Source |+|---------|--------|+| `mixamo` | [Linzhan/Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters) |+| `objaverse` | [Linzhan/Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) |+| `truebones` | [Linzhan/Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata, renders, build scripts โ no motion files) |++The mixamo and truebones repos also carry the stage-2a multi-view renders as per-view MP4 previews plus camera JSONs (`animation_motion_render/`, `animation_render/` โ see each repo's README). The per-frame PNGs the caption stage reads are not hosted; stage 2a regenerates them (or extract stills from the MP4s).++> [!IMPORTANT]+> **The Truebones motion files are not downloadable from us.** The Truebones ZOO animal pack is a commercial product whose license does not allow redistribution, so its repo above ships annotations only. To reproduce the truebones part of the pipeline, purchase the pack from [Truebones](https://truebones.com), unpack it to `dataset/raw/truebones/Truebone_Z-OO/{Animal}/`, and run the downloaded `scripts/pipeline/` (see the repo's README) to rebuild the per-clip layout โ one binary FBX per clip, flat in a single directory:+>+> ```+> dataset/raw/truebones/animation/{Species}-{Action}.fbx # e.g. Alligator-Big_Mouth.fbx+> ```+>+> (Species names must not contain `-` โ the first dash separates species from action.)+>+> Stage 5 additionally needs the **original per-animal layout** for the character meshes, since the flat per-clip files above are the animation source only:+>+> ```+> dataset/raw/truebones/Truebone_Z-OO/{Animal}/*.fbx # e.g. Truebone_Z-OO/Dog-2/+> ```+>+> Directory names are matched ignoring separators and case, so the clip-name object type `Dog2` resolves to the `Dog-2` directory.++## Quick Start++End-to-end run for Objaverse (Truebones is identical with `objaverse` โ `truebones`; see the [note above](#overview) for Mixamo's extra animate step):++```bash+# 0. Raw data+bash data_process/scripts/run_download.sh objaverse++# 1. Export raw GLBs to NPZ motion data+bash data_process/scripts/run_export.sh objaverse --multi-worker 8++# 2a. Render multi-view frames + T-pose grids+bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8+bash data_process/scripts/run_render_tpose.sh objaverse++# 2b. Caption motions + classify body plans (local Qwen3-VL by default)+bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu+bash data_process/scripts/run_caption_category.sh objaverse++# 3. Joint-name cleanup + facing-direction pair+bash data_process/scripts/run_joints_names_clean_llm.sh objaverse+bash data_process/scripts/run_joints_face_select_llm.sh objaverse++# 4. Canonicalized training clips + topology condition+bash data_process/scripts/run_extract_features.sh objaverse+```++Every stage is resumable โ rerunning a wrapper skips already-complete outputs.++## Pipeline Stages++### Stage 1 โ Export: raw assets โ NPZ++```bash+bash data_process/scripts/run_export.sh truebones # flat per-clip {Species}-{Action}.fbx+bash data_process/scripts/run_export.sh mixamo # animation-only FBX (no mesh)+bash data_process/scripts/run_export.sh objaverse # GLB/GLTF+bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # parallel workers (mixamo too)+bash data_process/scripts/run_export.sh mixamo --no-vis # skip per-clip MP4 previews (much faster)+DATA_DIR=my_assets bash data_process/scripts/run_export.sh # auto mode: .glb/.gltf and .fbx side by side+```++Skeletons are pruned of control/helper bones that carry no skin weight and converted to Y-up. Mixamo needs its explicit preset: animation-only FBXs carry no mesh, so auto mode cannot detect them. For custom assets there is a general exporter (single file or a directory of mixed GLB/GLTF/FBX):++```bash+bash data_process/scripts/run_export_general.sh my_model.glb+bash data_process/scripts/run_export_general.sh my_assets/ --multi-worker 8+```++### Stage 2a โ Render: multi-view frames & T-pose grids++Renders each motion from four fixed views (front / back / left / right) with EEVEE, plus a T-pose 2x2 grid per asset for body-plan classification:++```bash+bash data_process/scripts/run_render_motion.sh truebones+bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8+bash data_process/scripts/run_render_motion.sh objaverse --missing-only # resume: skip complete clips+bash data_process/scripts/run_render_tpose.sh objaverse+DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_motion.sh mixamo+```++Render folders use the same clip naming as stage 1, so they line up with the exported NPZs. Rendering is capped at the first 200 frames per clip (~6.7 s at 30 fps) โ enough context for captioning without paying for full-length renders.++`--fps` (default 30) must match the export stage's `--fps`: glTF stores keyframe times in seconds, so the importer resamples them at the scene frame rate, and a mismatch makes the renders cover a different time window than the NPZs.++### Stage 2b โ Caption: renders โ text++One wrapper; the backend is picked from `MODEL`:++```bash+bash data_process/scripts/run_caption_motion.sh truebones # local Qwen3-VL (default)+bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu # shard across all visible GPUs+MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse # Gemini (GOOGLE_API_KEY)+MODEL=gpt-5-mini bash data_process/scripts/run_caption_motion.sh mixamo # OpenAI (OPENAI_API_KEY)+bash data_process/scripts/run_caption_category.sh objaverse # body-plan classification+```++`--multi-gpu` applies to the local Qwen backend only; API backends scale via `NUM_WORKERS`. The local backend feeds each clip's four camera views as **native video** at the render frame rate (all frames, `--vision_input video`, the default); API backends receive frame sequences instead, capped by `--max_frames_per_view` (default 64, uniformly sampled including the first and last frame). For mixamo and truebones the official catalogue prompts (`animation_motion_prompts.json` / `animation_prompts.json`) are attached as reference hints automatically (`HINTS_JSON` overrides, `HINTS_JSON=""` disables). Clips whose renders are incomplete are recorded as failures and retried later rather than captioned from a partial set.++### Stage 3 โ Joint annotation: name cleanup & facing direction++Two `<task>_<step>` trios โ joint-name cleaning (`names_*`) and facing-direction pair selection (`face_*`) โ each with a rule-based and an LLM variant. The LLM variants default to `deepseek-v4-flash` (also `gpt-*` or a local HF model id via `MODEL=`) and fall back to the rules on failure.++**Required** โ these two produce the artifacts stage 4 consumes (`clean_joint_names.json`, `face_joint_names.json`):++```bash+bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # LLM names (falls back to rules)+bash data_process/scripts/run_joints_face_select_llm.sh objaverse # LLM facing pair+```++**Optional** โ refinement passes over the files above. Skip them entirely on a clean LLM run; reach for them when the required passes recorded failures (`failed_*.txt`) or when spot checks reveal residual label / pairing errors:++```bash+bash data_process/scripts/run_joints_names_correct_llm.sh objaverse # re-check every label, correct in place+bash data_process/scripts/run_joints_face_correct_llm.sh objaverse # retry rigs whose facing pair came back "empty"+```++`names_correct_llm` sends each `{raw, current}` label pair back to the LLM to keep or fix (most useful for rigs that fell back to the rules โ restrict with `FAILED_LIST=.../failed_clean_names.txt`); `face_correct_llm` re-attempts only `source == "empty"` entries. Both edit in place and create a `.bak` sibling on first run.++Pure rule-based variants exist as `run_joints_names_clean_rule.sh` / `run_joints_face_select_rule.sh` (no API needed โ useful offline or as a fast first pass). The QA visualizers behind this stage live in `data_process/tools/`: `run_joints_vis_tpose.sh` renders annotated T-pose PNGs (`tools/vis_tpose.py`), and to sanity-check a picked pair on an actual clip, `python -m data_process.tools.vis_motion <motion.npz> --direction face --face-joints I J` renders an MP4 with the implied heading arrow.++Both cleaners default to the same `clean_joint_names.json`, so the LLM pass skips rigs the rule-based pass already filled in; pass `--overwrite` to redo everything or `--redo_failed` to redo only the rigs listed in `failed_clean_names.txt`.++### Stage 4 โ Extract: NPZ + metadata โ training clips++```bash+bash data_process/scripts/run_extract_features.sh objaverse++# crop long motions into overlapping fixed-length clips instead of truncating+APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones++# parallel over object types, skip MP4 previews+NUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse+```++Mixamo is reduced to a built-in **22-joint humanoid core** (finger chains and End bones dropped) via `--mixamo_core_joints` (default on); pass `--no-mixamo_core_joints` to keep the full 65-joint rig. Truebones and objaverse always use the full skeleton.++Every object type is canonicalized against its T-pose (facing โ XZ-centering โ diameter scaling โ grounding), static pre/post-roll is trimmed, low-activity clips are filtered, and the per-object topology condition is written to `cond.npy`. Finished object types are cached under `cond_parts/`, so interrupted runs resume where they stopped; the cache is keyed on the clip/threshold settings **and** on digests of the caption / joint-annotation inputs, so changing any of them (including a stage-2/3 rerun) re-processes the affected object types instead of silently mixing old and new results.++Per-clip preview MP4s use a ground-plane view โ checkerboard stage, smoothly following camera and root trajectory (`--vis_ground`, default on; `--no-vis_ground` renders the plain cubic view instead).++Each motion is grounded on its own lowest joint by default; pass `--use_tpos_ground_height` to reuse the T-pose's ground height for every clip instead. Either way `cond['ground_height']` and `cond['ground_height_mode']` record what was actually applied. Object types that fail are logged to `extract_errors.log` and skipped rather than aborting the run, and are retried on the next run.++### Stage 5 โ Animate: drive a mesh with a motion++Used at inference time to render generated motions onto their rigged meshes:++```bash+ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse # stage-4 / generated clip + cond.npy+ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ rig+bash data_process/scripts/run_animate_fbx.sh # raw animation FBXs โ any character+bash data_process/scripts/run_animate_mixamo.sh # batch over Mixamo export NPZs (Y Bot)+```++`run_animate_fbx.sh` needs no export stage at all: it bakes a raw animation clip โ or a whole directory of them โ onto any character that shares the clips' bone names (the Mixamo convention), e.g. the full library on a different character:++```bash+CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \+ bash data_process/scripts/run_animate_fbx.sh # โ outputs/animated_Amy/*.fbx+```++The character is auto-resolved for truebones / objaverse. For mixamo it defaults to `character_refined/Y_Bot.fbx` โ the character whose skeleton the animation FBXs are authored on (identical 65-bone rig, so no bone reconciliation is needed); override with `CHAR_PATH` to use a different character. Armature bones absent from the motion are handled by `--extra_bones_strategy` (`merge` weights into the nearest kept ancestor, `remove` bones and their vertices, or `keep` them un-keyed).++## Data Formats++**Stage-1 export NPZ** (`dataset/export/<dataset>/motions/*.npz`) โ one file per clip:++| Field | Shape | Description |+|-------|-------|-------------|+| `rest_local_pos` / `rest_local_rot` | `(J, 3)` / `(J, 4)` | rest-pose local translations / rotations (quaternions) |+| `anim_local_pos` / `anim_local_rot` | `(T, J, 3)` / `(T, J, 4)` | per-frame local translations / rotations |+| `offsets` | `(J, 3)` | bind-pose bone offsets |+| `parents` | `(J,)` | parent indices, `-1` for the root |+| `names` | `(J,)` | bone name strings |+| `skin_matrix` | `(V, J)` | skinning weights (empty when no mesh) |+| `fps`, `action_name` | scalar | frame rate and source action |++**Stage-4 feature NPZ** (`dataset/features/<dataset>/motions/*.npz`) โ canonicalized per-clip arrays, named `{object}-{motion}-{clip_idx}.npz`:++| Field | Shape | Description |+|-------|-------|-------------|+| `global_positions` | `(F, J, 3)` | global joint positions at canonical scale |+| `local_rotations` | `(F, J, 4)` | local joint rotations (quaternions) |+| `root_facing_quat` | `(F, 4)` | per-frame root quaternion aligning the skeleton to face Z+ |+| `fps` | scalar | frame rate of the clip (after any downsampling) |++**`cond.npy`** โ one dict per object type holding everything the training dataloader needs about the skeleton: topology (`parents`, `offsets`, joint names), T-pose positions/rotations, graph structure (relations, distances, depths, spectral features, kinematic chains), calibration (`scale_factor`, `ground_height`), face joints and per-clip captions.++## Conventions++- **Run from the repo root.** Wrappers `cd` there themselves.+- **Resumable by default.** Every batch stage skips outputs that already exist and records failures in sibling error logs, so failed items are retried on the next run without redoing finished work.+- **Env-var overrides.** Every wrapper documents its overridable paths/knobs in its header comment; run any wrapper with `-h` to print it.+- **Parallelism.** Blender stages shard with `--multi-worker N`; local-model captioning shards with `--multi-gpu`; API backends use concurrent workers via `NUM_WORKERS`.+- **Naming.** Clips are named `{object_type}-{action}` (stage 1) and `{object_type}-{motion}-{clip_idx}` (stage 4). Mixamo is a single shared skeleton, so its files carry no object-type prefix.++## Troubleshooting++- **EEVEE renders fail / produce black frames under `blender -b`** โ stage 2a must run with plain `python` and the pip `bpy` module (the wrappers already do); headless `blender -b` has no GPU display surface for EEVEE.+- **Offline compute nodes** โ with a populated HuggingFace cache, export `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` before the local Qwen caption/classify runs.+- **`joint_names.json` is missing object types that have clips** โ an export killed mid-run (Slurm walltime, OOM) can leave the summary JSONs behind the completion markers. Rebuild them from the markers:+ ```bash+ python -m data_process.tools.merge_summaries --output_dir dataset/export/<dataset>+ ```+ Additive and idempotent.diff --git a/data_process/__init__.py b/data_process/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/feature_extraction/__init__.py b/data_process/feature_extraction/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/feature_extraction/extract_features.py b/data_process/feature_extraction/extract_features.pynew file mode 100644index 0000000..c77c419--- /dev/null+++ b/data_process/feature_extraction/extract_features.py@@ -0,0 +1,469 @@+"""Process exported NPZ motions into canonicalized training clips.++Supports three dataset layouts via ``--dataset_type``:++- ``truebones``: per-object NPZ files under ``data_dir/motions``, named+ ``{object_type}-{motion}.npz``. Object types are discovered from filename+ prefixes.+- ``objaverse``: same layout as truebones, plus an optional+ ``filtered_objects.txt`` listing object types to skip.+- ``mixamo``: single skeleton / object type. All NPZs live directly under+ ``data_dir/motions`` with no ``{object_type}-`` prefix; caption keys are+ bare motion stems.++Stage-2 (captions, category groups) and stage-3 (clean / face joint names)+metadata JSONs are read from ``data_dir`` โ see+:mod:`data_process.feature_extraction.metadata`.++Object types are independent, so they can be processed in parallel+(``--num_workers``) and are resumable: each finished object type caches its+result under ``<save_dir>/cond_parts/`` and is skipped on rerun. The cache+records the clip / threshold / topology settings it was built with and is+re-processed automatically when they change; delete ``cond_parts/`` (or one+entry) to force re-processing for any other reason.++Object types that fail are logged to ``<save_dir>/extract_errors.log``,+recorded in ``filtered_clips.json`` and skipped (no cache entry is written,+so they are retried on the next run) โ one bad object never aborts the run.++Usage:+ python -m data_process.feature_extraction.extract_features \\+ --dataset_type truebones \\+ --data_dir dataset/export/truebones \\+ --save_dir dataset/features/truebones+"""++import argparse+import hashlib+import json+import multiprocessing as mp+import os+from os.path import join as pjoin++import numpy as np+from loguru import logger+from tqdm import tqdm++from data_process.feature_extraction.metadata import (+ MIXAMO_CORE_JOINTS,+ atomic_np_save,+ build_stats,+ check_metadata_object_types_consistent,+ get_object_captions,+ get_object_face_joints,+ get_object_metadata,+ load_all_metadata,+ print_summary,+ save_metadata_report,+ save_outputs,+)+from data_process.utils.motion_features import process_object+++MIXAMO_OBJECT_TYPE = 'mixamo'+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Process exported NPZ motions into canonicalized training clips.")+ # Dataset selection+ parser.add_argument("--dataset_type", type=str, required=True,+ choices=['truebones', 'objaverse', 'mixamo'],+ help="Which dataset layout to process")+ # Directories+ parser.add_argument("--data_dir", type=str, required=True,+ help="Root data directory (motions under data_dir/motions, "+ "metadata JSONs in data_dir)")+ parser.add_argument("--save_dir", type=str, required=True,+ help="Output directory for processed clips and metadata")+ # Clip windowing+ parser.add_argument("--max_clip_len", type=int, default=180,+ help="Max frames per saved clip")+ parser.add_argument("--diffusion_max_len", type=int, default=60,+ help="Model training max length (dataloader random-crops to this)")+ parser.add_argument("--apply_clip", action="store_true", default=False,+ help="Crop long motions into overlapping fixed-length clips of "+ "max_clip_len frames (stride = max_clip_len - "+ "diffusion_max_len). Otherwise (default), keep only the "+ "first max_clip_len frames of each motion.")+ # Topology+ parser.add_argument("--max_path_len", type=int, default=5,+ help="Max path length for topology edge relations")+ parser.add_argument("--max_freqs", type=int, default=8,+ help="Number of Laplacian eigenvector frequencies for "+ "spectral joint features")+ # Scaling / grounding+ parser.add_argument("--target_diameter", type=float, default=2.0,+ help="Target skeleton diameter for leaf-to-leaf scaling")+ parser.add_argument("--use_tpos_ground_height", action="store_true", default=False,+ help="Ground every motion of an object type on the T-pose's "+ "floor height. Otherwise (default), each motion is "+ "grounded on its own minimum Y. The applied value is "+ "recorded in cond as ground_height / ground_height_mode.")+ # Filtering+ parser.add_argument("--activity_threshold", type=float, default=0.02,+ help="Min joint activity (temporal spread at canonical "+ "scale) to keep a clip")+ parser.add_argument("--static_threshold", type=float, default=1e-5,+ help="Per-frame max-joint displacement below which a frame "+ "is static; leading/trailing static frames are trimmed")+ parser.add_argument("--min_frames", type=int, default=8,+ help="Minimum frame count for a motion to be kept "+ "(checked after downsampling and static trimming)")+ parser.add_argument("--mixamo_core_joints", action=argparse.BooleanOptionalAction,+ default=True,+ help="(mixamo only) Restrict the rig to the built-in "+ "22-joint humanoid core (drops finger chains and "+ "End bones) โ replaces the retired "+ "corps_joint_names.json. Use --no-mixamo_core_joints "+ "to keep the full 65-joint skeleton.")+ parser.add_argument("--min_joints", type=int, default=8,+ help="Skip object types with fewer joints than this "+ "(counted after corps filtering)")+ parser.add_argument("--max_joints", type=int, default=150,+ help="Skip object types with more joints than this "+ "(counted after corps filtering)")+ # Objaverse-only filtering+ parser.add_argument("--filtered_objects", type=str, default="auto",+ help="(objaverse only) Path to a txt file listing object "+ "types to skip (one per line; blanks and '#' comments "+ "ignored). Default 'auto' uses "+ "<data_dir>/filtered_objects.txt if present. Pass an "+ "empty string to disable.")+ # Execution+ parser.add_argument("--num_workers", type=int, default=1,+ help="Parallel worker processes (object types are independent)")+ parser.add_argument("--vis", action=argparse.BooleanOptionalAction, default=True,+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs)")+ parser.add_argument("--vis_ground", action=argparse.BooleanOptionalAction,+ default=True,+ help="Render previews with a checkerboard ground plane, "+ "follow camera and root trajectory (default). "+ "--no-vis_ground falls back to the legacy plain "+ "cubic view.")+ return parser.parse_args()+++def resolve_filtered_objects_path(path, data_dir):+ if path == "auto":+ default = pjoin(data_dir, 'filtered_objects.txt')+ return default if os.path.isfile(default) else None+ return path or None+++def load_filtered_objects(path):+ if not path:+ return set()+ with open(path, 'r') as f:+ return {+ line.strip() for line in f+ if line.strip() and not line.startswith('#')+ }+++def discover_object_types(motion_dir, dataset_type, args):+ """Return (object_types, all_motions) given the dataset layout.++ Only ``.npz`` files are considered, so stray side files in ``motions/``+ never turn into object types with zero motions.+ """+ entries = os.listdir(motion_dir)+ all_motions = [f for f in entries if f.endswith('.npz')]+ if len(all_motions) != len(entries):+ logger.info(f'Ignoring {len(entries) - len(all_motions)} non-NPZ '+ f'entries in {motion_dir}')++ if dataset_type == 'mixamo':+ # Single object type; no prefix-based grouping.+ return [MIXAMO_OBJECT_TYPE], all_motions++ object_types = sorted(set(m.split('-')[0] for m in all_motions))+ logger.info(f'Found {len(all_motions)} motion files, {len(object_types)} object types')++ if dataset_type == 'objaverse':+ filtered_path = resolve_filtered_objects_path(args.filtered_objects, args.data_dir)+ filtered_object_types = load_filtered_objects(filtered_path)+ if filtered_object_types:+ before = len(object_types)+ object_types = [o for o in object_types if o not in filtered_object_types]+ logger.info(f'Loaded {len(filtered_object_types)} entries from {filtered_path}; '+ f'{before} -> {len(object_types)} object types after filtering')+ elif args.filtered_objects:+ logger.info(f'No filtered_objects file found '+ f'(--filtered_objects={args.filtered_objects!r}); '+ f'processing all {len(object_types)} object types')++ return object_types, all_motions+++def collect_object_npzs(object_type, dataset_type, motion_dir, all_motions):+ if dataset_type == 'mixamo':+ return sorted(pjoin(motion_dir, f) for f in all_motions if f.endswith('.npz'))+ return sorted(+ pjoin(motion_dir, f) for f in all_motions+ if f.startswith(object_type + '-') and f.endswith('.npz')+ )+++def resolve_captions(object_type, dataset_type, motion_captions):+ # Mixamo caption keys are bare motion stems (no "{object_type}-" prefix),+ # so the full captions dict is passed through untouched.+ if dataset_type == 'mixamo':+ return motion_captions+ return get_object_captions(object_type, motion_captions)+++def _object_part_path(save_dir, object_type):+ return pjoin(save_dir, 'cond_parts', f'{object_type}.npy')+++def _error_log_path(save_dir):+ return pjoin(save_dir, 'extract_errors.log')+++# Settings that determine a cached object result's contents. A cache entry+# built with different values is stale and gets re-processed. ``save_vis``+# is deliberately excluded: it only controls the MP4 previews, not the clips+# or the cond.+CACHE_PARAM_KEYS = (+ 'max_clip_len', 'clip_stride', 'apply_clip',+ 'max_path_len', 'max_freqs',+ 'target_diameter', 'activity_threshold', 'static_threshold',+ 'min_frames', 'min_joints', 'max_joints',+ 'use_tpos_ground_height',+ 'corps_names',+)++# Per-object METADATA inputs baked into the result (captions land in the+# cond, face joints steer the facing canonicalization, clean names are+# stored per joint). Hashed into the cache key so a stage-2/3 rerun โ+# new captions, corrected face pairs, recleaned labels โ re-processes the+# affected object types instead of silently resuming stale results.+CACHE_DATA_KEYS = ('captions', 'face_joints', 'clean_names')+++def _digest(value):+ """Stable short digest of a JSON-serializable metadata value."""+ blob = json.dumps(value, sort_keys=True, default=str)+ return hashlib.sha1(blob.encode('utf-8')).hexdigest()[:16]+++def _cache_params(task):+ """Extract the cache-invalidating settings from a task dict."""+ params = {k: task[k] for k in CACHE_PARAM_KEYS}+ for k in CACHE_DATA_KEYS:+ params[f'{k}_digest'] = _digest(task[k])+ return params+++def _params_hash(params):+ """Stable hash of the cache-invalidating settings."""+ blob = json.dumps(params, sort_keys=True, default=str)+ return hashlib.sha1(blob.encode('utf-8')).hexdigest()+++def _load_cached_result(part_path, object_type, params, params_hash):+ """Return a cached object result, or None when absent / stale / unreadable."""+ if not os.path.isfile(part_path):+ return None+ try:+ payload = np.load(part_path, allow_pickle=True).item()+ except Exception as e: # noqa: BLE001 โ a bad cache entry is just a miss+ logger.warning(f'[{object_type}] unreadable cache {part_path} ({e}); '+ f're-processing')+ return None++ if 'params_hash' not in payload:+ logger.info(f'[{object_type}] cache invalidated (entry predates settings '+ f'tracking); re-processing')+ return None+ if payload['params_hash'] == params_hash:+ return payload.get('result')++ cached_params = payload.get('params') or {}+ diff = ', '.join(+ f'{k}: {cached_params.get(k, "<missing>")} -> {v}'+ for k, v in sorted(params.items())+ if cached_params.get(k, '<missing>') != v)+ logger.info(f'[{object_type}] cache invalidated (settings changed: {diff}); '+ f're-processing')+ return None+++def process_object_task(task):+ """Process one object type, caching its result for resume.++ ``task`` bundles every :func:`process_object` argument plus the part+ path. The cached result is reused only when it was produced with the same+ clip / threshold / topology settings (see ``CACHE_PARAM_KEYS``).++ Failures are contained here: the object is reported as failed (and its+ error appended to ``extract_errors.log``) instead of propagating, which+ with ``mp.Pool`` would kill the whole run. No cache entry is written for+ a failed object, so it is retried on the next run.++ Returns ``(object_type, result_dict, from_cache)``.+ """+ object_type = task.pop('object_type')+ part_path = task.pop('part_path')+ params = _cache_params(task)+ params_hash = _params_hash(params)++ cached = _load_cached_result(part_path, object_type, params, params_hash)+ if cached is not None:+ return object_type, cached, True++ try:+ obj_cond, n_clips, n_frames, n_joints, filtered = process_object(+ object_type, **task)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ logger.exception(f'[{object_type}] failed to process')+ try:+ os.makedirs(task['save_dir'], exist_ok=True)+ with open(_error_log_path(task['save_dir']), 'a') as log_file:+ log_file.write(f'{object_type}\t{type(e).__name__}: {e}\n')+ except OSError as log_err:+ logger.warning(f'[{object_type}] could not write error log: {log_err}')+ return object_type, {+ 'cond': None, 'n_clips': 0, 'n_frames': 0, 'n_joints': 0,+ 'filtered': [{'name': f'{object_type} (all clips)',+ 'reason': f'processing failed: {type(e).__name__}: {e}'}],+ 'error': f'{type(e).__name__}: {e}',+ }, False++ result = {'cond': obj_cond, 'n_clips': n_clips, 'n_frames': n_frames,+ 'n_joints': n_joints, 'filtered': filtered}+ # Atomic: a worker killed mid-write can never leave a half-written cache+ # entry behind (unreadable ones are treated as misses, but this keeps them+ # from happening in the first place).+ atomic_np_save(part_path, {'params_hash': params_hash, 'params': params,+ 'result': result})+ return object_type, result, False+++def build_object_tasks(args, clip_stride, motion_dir, metadata):+ face_joint_names = metadata.get('face_joint_names')+ clean_joint_names = metadata.get('clean_joint_names')+ joint_names = metadata.get('joint_names')+ motion_captions = metadata.get('motion_captions')++ object_types, all_motions = discover_object_types(motion_dir, args.dataset_type, args)++ tasks = []+ for object_type in object_types:+ face_joints = get_object_face_joints(object_type, face_joint_names)+ if not face_joints:+ if args.dataset_type == 'mixamo':+ raise ValueError(f'No face_joints entry for {object_type}; '+ f'face_joint_names.json is required for mixamo')+ logger.info(f'No face_joints entry for {object_type}; using identity facing')++ tasks.append(dict(+ object_type=object_type,+ part_path=_object_part_path(args.save_dir, object_type),+ object_npzs=collect_object_npzs(object_type, args.dataset_type,+ motion_dir, all_motions),+ save_dir=args.save_dir,+ max_clip_len=args.max_clip_len, clip_stride=clip_stride,+ apply_clip=args.apply_clip,+ max_path_len=args.max_path_len, max_freqs=args.max_freqs,+ captions=resolve_captions(object_type, args.dataset_type, motion_captions),+ face_joints=face_joints,+ corps_names=(MIXAMO_CORE_JOINTS+ if args.dataset_type == 'mixamo'+ and args.mixamo_core_joints else None),+ clean_names=get_object_metadata(object_type, clean_joint_names),+ expected_names=get_object_metadata(object_type, joint_names),+ require_face=args.dataset_type == 'mixamo',+ target_diameter=args.target_diameter,+ use_tpos_ground_height=args.use_tpos_ground_height,+ activity_threshold=args.activity_threshold,+ static_threshold=args.static_threshold,+ min_frames=args.min_frames,+ min_joints=args.min_joints, max_joints=args.max_joints,+ save_vis=args.vis, vis_ground=args.vis_ground,+ ))+ return tasks+++def main(args):+ clip_stride = args.max_clip_len - args.diffusion_max_len+ if clip_stride <= 0:+ message = (f'--diffusion_max_len ({args.diffusion_max_len}) must be smaller '+ f'than --max_clip_len ({args.max_clip_len}): the clip stride is '+ f'their difference, and {clip_stride} would '+ + ('never advance the window'+ if clip_stride == 0 else 'walk the window backwards'))+ if args.apply_clip:+ raise ValueError(message)+ logger.warning(f'{message}. Harmless here because --apply_clip is off '+ f'(the stride is unused), but fix it before enabling it.')++ os.makedirs(args.save_dir, exist_ok=True)++ motion_dir = pjoin(args.data_dir, 'motions')++ metadata = load_all_metadata(args.data_dir)+ logger.info(f'Loaded metadata from {args.data_dir}: '+ + ', '.join(f'{k}={len(v)}' for k, v in metadata.items()))+ check_metadata_object_types_consistent(metadata)+ category_groups = metadata.get('category_groups')++ tasks = build_object_tasks(args, clip_stride, motion_dir, metadata)++ cond = {}+ all_filtered_clips = {}+ clips_per_object = {}+ joints_per_object = {}+ total_frames = 0+ max_njoints = 0+ n_cached = 0+ failed_objects = {}++ def accumulate(object_type, result, from_cache):+ nonlocal total_frames, max_njoints, n_cached+ n_cached += int(from_cache)+ if result.get('error'):+ failed_objects[object_type] = result['error']+ if result['n_clips']:+ cond[object_type] = result['cond']+ clips_per_object[object_type] = result['n_clips']+ joints_per_object[object_type] = result['n_joints']+ total_frames += result['n_frames']+ max_njoints = max(max_njoints, result['n_joints'])+ if result['filtered']:+ all_filtered_clips[object_type] = result['filtered']++ pbar = tqdm(total=len(tasks), desc="Processing objects", unit="obj")+ if args.num_workers > 1:+ with mp.Pool(args.num_workers) as pool:+ for object_type, result, from_cache in pool.imap_unordered(+ process_object_task, tasks):+ accumulate(object_type, result, from_cache)+ pbar.update(1)+ else:+ for task in tasks:+ accumulate(*process_object_task(task))+ pbar.update(1)+ pbar.close()++ if n_cached:+ logger.info(f'Resumed {n_cached}/{len(tasks)} object types from cond_parts/ '+ f'(delete that directory to force re-processing)')+ if failed_objects:+ logger.error(f'{len(failed_objects)}/{len(tasks)} object types failed and were '+ f'skipped (retried on the next run); see '+ f'{_error_log_path(args.save_dir)}: '+ + ', '.join(sorted(failed_objects)))++ all_captions = save_outputs(args.save_dir, cond, all_filtered_clips,+ category_groups=category_groups)+ stats = build_stats(clips_per_object, joints_per_object, total_frames, max_njoints)+ print_summary(stats)+ save_metadata_report(args.save_dir, stats, all_filtered_clips,+ all_captions=all_captions, category_groups=category_groups)+ logger.info('Dataset processing complete.')+++if __name__ == "__main__":+ main(parse_args())diff --git a/data_process/feature_extraction/metadata.py b/data_process/feature_extraction/metadata.pynew file mode 100644index 0000000..fc83854--- /dev/null+++ b/data_process/feature_extraction/metadata.py@@ -0,0 +1,387 @@+"""Metadata I/O and clip windowing for the feature-extraction stage.++Loads the per-dataset JSON side files produced by the caption and+joint-annotation stages (``motion_captions.json``, ``category_groups.json``,+``clean_joint_names.json``, ``face_joint_names.json``, ...), provides+case-insensitive per-object lookups, writes the stage outputs (``cond.npy``,+``captions.json``, ``filtered_clips.json``, ``metadata.txt``) and computes+clip windows. Used by :mod:`data_process.feature_extraction.extract_features`.+"""++import json+import os+import tempfile+from contextlib import contextmanager+from os.path import join as pjoin++import numpy as np+++# ---------------------------------------------------------------------------+# Atomic output writes+# ---------------------------------------------------------------------------++@contextmanager+def atomic_output_path(path):+ # type: (str) -> Iterator[str]+ """Yield a temp path next to *path*, renamed into place on clean exit.++ Stage outputs land at the end of runs that can take hours, so writing them+ in place risks leaving a truncated ``cond.npy`` / ``captions.json`` behind+ if the run is interrupted. Writers fill the yielded temp path instead; the+ final ``os.replace`` is atomic, so readers see either the previous file or+ the complete new one, and a failed write leaves the previous file intact.+ """+ directory = os.path.dirname(os.path.abspath(path))+ os.makedirs(directory, exist_ok=True)+ fd, tmp_path = tempfile.mkstemp(+ dir=directory, prefix=f'.{os.path.basename(path)}.', suffix='.tmp')+ os.close(fd)+ try:+ yield tmp_path+ os.replace(tmp_path, path)+ except BaseException:+ try:+ os.remove(tmp_path)+ except OSError:+ pass+ raise+++def atomic_np_save(path, data):+ # type: (str, object) -> None+ """``np.save`` *data* to *path* atomically (see :func:`atomic_output_path`)."""+ with atomic_output_path(path) as tmp_path:+ # Passing a file object keeps np.save from appending '.npy' to the+ # temp name (it only does that for string paths).+ with open(tmp_path, 'wb') as f:+ np.save(f, data)+++# ---------------------------------------------------------------------------+# JSON loading & metadata helpers+# ---------------------------------------------------------------------------++def load_json(path):+ # type: (str) -> Optional[dict]+ """Load a JSON file if it exists, otherwise return None."""+ if path and os.path.isfile(path):+ with open(path, 'r') as f:+ return json.load(f)+ return None+++def load_motion_captions(source_dir):+ # type: (str) -> Optional[Dict[str, str]]+ """Load motion captions from ``motion_captions.json`` in *source_dir*."""+ return load_json(pjoin(source_dir, 'motion_captions.json'))+++def load_category_groups(source_dir):+ # type: (str) -> Optional[Dict[str, List[str]]]+ """Load category-to-object-type groupings from ``category_groups.json``."""+ return load_json(pjoin(source_dir, 'category_groups.json'))+++def load_clean_joint_names(source_dir):+ # type: (str) -> Optional[Dict[str, List[str]]]+ """Load cleaned joint names from ``clean_joint_names.json``."""+ return load_json(pjoin(source_dir, 'clean_joint_names.json'))+++def load_face_joint_names(source_dir):+ # type: (str) -> Optional[Dict[str, dict]]+ """Load face joint info from ``face_joint_names.json``."""+ return load_json(pjoin(source_dir, 'face_joint_names.json'))+++# The 22-joint humanoid core the mixamo rig is reduced to at extraction time+# (drops the 40 finger bones plus the End bones). Replaces the retired+# corps_joint_names.json side file; toggled by --mixamo_core_joints.+MIXAMO_CORE_JOINTS = [+ "mixamorig:Hips",+ "mixamorig:Spine", "mixamorig:Spine1", "mixamorig:Spine2",+ "mixamorig:Neck", "mixamorig:Head",+ "mixamorig:LeftShoulder", "mixamorig:LeftArm",+ "mixamorig:LeftForeArm", "mixamorig:LeftHand",+ "mixamorig:RightShoulder", "mixamorig:RightArm",+ "mixamorig:RightForeArm", "mixamorig:RightHand",+ "mixamorig:LeftUpLeg", "mixamorig:LeftLeg",+ "mixamorig:LeftFoot", "mixamorig:LeftToeBase",+ "mixamorig:RightUpLeg", "mixamorig:RightLeg",+ "mixamorig:RightFoot", "mixamorig:RightToeBase",+]+++def load_joint_names(source_dir):+ # type: (str) -> Optional[Dict[str, List[str]]]+ """Load raw joint names from ``joint_names.json``."""+ return load_json(pjoin(source_dir, 'joint_names.json'))+++def load_all_metadata(source_dir):+ # type: (str) -> dict+ """Load all available JSON metadata files from *source_dir*."""+ loaders = {+ 'motion_captions': load_motion_captions,+ 'category_groups': load_category_groups,+ 'clean_joint_names': load_clean_joint_names,+ 'face_joint_names': load_face_joint_names,+ 'joint_names': load_joint_names,+ }+ meta = {}+ for key, loader in loaders.items():+ data = loader(source_dir)+ if data is not None:+ meta[key] = data+ return meta+++def check_metadata_object_types_consistent(+ metadata,+ keys=('clean_joint_names', 'face_joint_names', 'joint_names'),+):+ # type: (dict, tuple) -> None+ """Verify per-object-type metadata dicts cover the same object types.++ Compares the case-insensitive key sets of each present dict named by+ *keys* in *metadata*; absent or empty dicts are skipped. Raises+ ``ValueError`` on mismatch so drivers fail loudly on misaligned JSON+ sources, and prints a consistency-OK line when all present dicts agree.+ """+ present = {n: {k.lower() for k in metadata[n].keys()}+ for n in keys if metadata.get(n)}+ if len(present) < 2:+ return+ ref_name, ref_set = next(iter(present.items()))+ for other_name, other_set in present.items():+ if other_name == ref_name:+ continue+ only_ref = sorted(ref_set - other_set)+ only_other = sorted(other_set - ref_set)+ if only_ref or only_other:+ raise ValueError(+ f"Object type mismatch between '{ref_name}' and '{other_name}': "+ f"only in {ref_name}: {only_ref}; "+ f"only in {other_name}: {only_other}"+ )+ print(f'Metadata consistency OK across {list(present.keys())}: '+ f'{len(ref_set)} object types')+++def _dict_get_ci(d, key):+ # type: (dict, str) -> object+ """Case-insensitive dict lookup. Tries exact match first, then lowercase."""+ if key in d:+ return d[key]+ key_lower = key.lower()+ for k, v in d.items():+ if k.lower() == key_lower:+ return v+ return None+++def get_object_metadata(object_type, metadata_dict):+ # type: (str, Optional[dict]) -> object+ """Case-insensitive lookup of *object_type* in a metadata dict, or None."""+ if metadata_dict:+ return _dict_get_ci(metadata_dict, object_type)+ return None+++def get_object_face_joints(object_type, face_joint_names=None):+ # type: (str, Optional[Dict[str, dict]]) -> Optional[dict]+ """Return face joint info for *object_type*, or None."""+ return get_object_metadata(object_type, face_joint_names)+++def get_object_captions(object_type, motion_captions=None):+ # type: (str, Optional[Dict[str, str]]) -> Dict[str, str]+ """Filter motion captions to entries for *object_type*.++ Caption keys follow the ``{object_type}-{motion_name}`` base-name+ convention used throughout the pipeline; matching is case-insensitive+ on the prefix. Returns an empty dict when no captions are supplied or+ none match.+ """+ if not motion_captions:+ return {}+ prefix = f'{object_type.lower()}-'+ return {k: v for k, v in motion_captions.items()+ if k.lower().startswith(prefix)}+++def save_json(path, data):+ """Write *data* as indented JSON to *path*, atomically."""+ with atomic_output_path(path) as tmp_path:+ with open(tmp_path, 'w') as f:+ json.dump(data, f, indent=2)+++def collect_captions(cond):+ # type: (Dict[str, dict]) -> Dict[str, str]+ """Merge per-object ``captions`` sub-dicts into a single flat dict."""+ all_captions = {}+ for obj_cond in cond.values():+ if 'captions' in obj_cond:+ all_captions.update(obj_cond['captions'])+ return all_captions+++def save_outputs(save_dir, cond, all_filtered_clips, category_groups=None):+ """Save ``cond.npy``, filtered clips, captions, and category groupings.++ Captions and category groups are only written when present. Every file is+ written atomically, so an interrupted run cannot leave a truncated output+ behind for the training loader to trip over. Returns the merged captions+ dict so callers can pass it to ``save_metadata_report``.+ """+ atomic_np_save(pjoin(save_dir, 'cond.npy'), cond)++ if all_filtered_clips:+ save_json(pjoin(save_dir, 'filtered_clips.json'), all_filtered_clips)+ total = sum(len(v) for v in all_filtered_clips.values())+ print(f'Saved {total} filtered clips to filtered_clips.json')++ all_captions = collect_captions(cond)+ if all_captions:+ save_json(pjoin(save_dir, 'captions.json'), all_captions)+ print(f'Saved {len(all_captions)} clip captions')++ if category_groups:+ save_json(pjoin(save_dir, 'category_groups.json'), category_groups)++ return all_captions+++def build_stats(clips_per_object, joints_per_object, total_frames, max_njoints, fps=30):+ """Aggregate per-object counts into the dataset summary dict.++ ``duration_min`` is a nominal figure: clips carry their own ``fps`` field+ (30 for every source, 30 after the >=60 fps down-sampling in+ :func:`~data_process.utils.motion_features._load_npz_anim`), but the+ duration here simply divides the frame total by *fps*. The assumption is+ reported alongside the value, so ``fps`` is echoed back in the stats.+ """+ return {+ 'total_clips': sum(clips_per_object.values()),+ 'total_frames': total_frames,+ 'duration_min': total_frames / fps / 60,+ 'fps': fps,+ 'max_njoints': max_njoints,+ 'clips_per_object': clips_per_object,+ 'joints_per_object': joints_per_object,+ }+++def save_metadata_report(save_dir, stats, all_filtered_clips,+ all_captions=None, category_groups=None):+ """Write a human-readable metadata summary to ``metadata.txt``.++ ``all_captions`` and ``category_groups`` are optional; their sections are+ omitted when not supplied. The file is written atomically.+ """+ total_filtered = sum(len(v) for v in all_filtered_clips.values())++ with atomic_output_path(pjoin(save_dir, 'metadata.txt')) as tmp_path, \+ open(tmp_path, 'w') as f:+ f.write(f'Total clips: {stats["total_clips"]}\n')+ f.write(f'Total frames: {stats["total_frames"]}\n')+ f.write(f'Total duration (minutes, assuming {stats.get("fps", 30)} fps): '+ f'{stats["duration_min"]:.2f}\n')+ f.write(f'Max joints in dataset: {stats["max_njoints"]}\n')+ if all_captions is not None:+ f.write(f'Total captions: {len(all_captions)}\n')++ f.write('Clips per object type:\n')+ for obj, count in stats['clips_per_object'].items():+ f.write(f' {obj}: {count}\n')++ f.write('Joints per object type:\n')+ for obj, count in stats['joints_per_object'].items():+ f.write(f' {obj}: {count}\n')++ f.write(f'Total filtered clips: {total_filtered}\n')+ if all_filtered_clips:+ f.write('Filtered clips per object type:\n')+ for obj, clips in all_filtered_clips.items():+ f.write(f' {obj}: {len(clips)}\n')++ if category_groups:+ f.write('Categories:\n')+ for cat, members in category_groups.items():+ processed = [m for m in members if m in stats['clips_per_object']]+ f.write(f' {cat}: {len(processed)} object types\n')+++# ---------------------------------------------------------------------------+# Clip windowing+# ---------------------------------------------------------------------------++def compute_clip_windows(nframes, max_clip_len, clip_stride):+ """Compute (start, end) windows for cropping a sequence into clips.++ Windows are ``max_clip_len`` frames long and start every ``clip_stride``+ frames. No returned window is ever longer than ``max_clip_len`` (the whole+ sequence is returned unsliced only when it already fits).++ Leftover frames past the last strided window are handled by tail length:++ - at least half a clip: emitted as their own (shorter) trailing window;+ - shorter than that, but still adding at least as many new frames as a+ normal window's overlap (``max_clip_len - clip_stride``): covered by a+ full-length window anchored at the final frame โ replacing the last+ window when the frames it gives up stay covered by the one before it,+ otherwise appended;+ - shorter than that: dropped, rather than glued onto the previous window+ (which is what used to push clips past ``max_clip_len``).+ """+ if not max_clip_len or nframes <= max_clip_len:+ return [(0, nframes)]++ clips = []+ for start in range(0, nframes - max_clip_len + 1, clip_stride):+ clips.append((start, start + max_clip_len))++ last_end = clips[-1][1]+ if last_end < nframes:+ tail_len = nframes - last_end+ if tail_len >= max_clip_len // 2:+ clips.append((last_end, nframes))+ else:+ # Cover the tail with a full-length window anchored at the end.+ slid_start = nframes - max_clip_len+ prev_end = clips[-2][1] if len(clips) >= 2 else 0+ if slid_start <= prev_end:+ # Free: the frames the last window gives up are still covered.+ clips[-1] = (slid_start, nframes)+ elif tail_len >= max(1, max_clip_len - clip_stride):+ # Worth an extra window: it adds at least as many new frames+ # as a normal window's overlap.+ clips.append((slid_start, nframes))+ # Otherwise the tail is too small to pay for a window of its own.++ return clips+++def get_motion_clip_windows(nframes, max_clip_len, clip_stride, apply_clip=True):+ """Return clip windows for a motion sequence.++ When ``apply_clip`` is True, crops into overlapping fixed-length clips via+ :func:`compute_clip_windows`. When False, returns a single window covering+ at most the first ``max_clip_len`` frames (i.e., truncation, not cropping).+ """+ if apply_clip:+ return compute_clip_windows(nframes, max_clip_len, clip_stride)+ clip_end = min(nframes, max_clip_len) if max_clip_len else nframes+ return [(0, clip_end)]+++# ---------------------------------------------------------------------------+# Summary+# ---------------------------------------------------------------------------++def print_summary(stats):+ print(f'Total clips: {stats["total_clips"]}, Frames: {stats["total_frames"]}, '+ f'Duration: {stats["duration_min"]:.1f}m (@{stats.get("fps", 30)} fps), '+ f'Max joints: {stats["max_njoints"]}')diff --git a/data_process/joint_annotation/__init__.py b/data_process/joint_annotation/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/joint_annotation/face_correct_llm.py b/data_process/joint_annotation/face_correct_llm.pynew file mode 100644index 0000000..d31d328--- /dev/null+++ b/data_process/joint_annotation/face_correct_llm.py@@ -0,0 +1,235 @@+"""Re-run LLM face-joint selection on rigs whose ``source == "empty"``.++:mod:`face_select_llm` writes the empty sentinel+``{r_hip: {raw:"", clean:""}, l_hip: {raw:"", clean:""}, source: "empty"}``+whenever neither the rule-based hint nor the LLM finds a bilateral pair or+a head/tail body axis. This script re-attempts those rigs only (by default+every ``source == "empty"`` entry in ``--face``; optionally a subset listed+in ``--failed_list``) and, when the LLM now finds a valid pair, overwrites+the entry in place. Rigs that still fail keep their existing empty entry.++A ``.bak`` sibling of ``--face`` is created on first run. Rigs that remain+empty are written to ``still_empty_face_joints.txt``.++Usage:+ python -m data_process.joint_annotation.face_correct_llm \\+ --raw <export_dir>/joint_names.json \\+ --cleaned <export_dir>/clean_joint_names.json \\+ --face <export_dir>/face_joint_names.json [--model gpt-5]+"""++import argparse+import json+import os+import shutil+import sys++from loguru import logger+from tqdm import tqdm++_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++from data_process.joint_annotation.face_select_llm import ( # noqa: E402+ SYSTEM_PROMPT,+ resolve_rig,+)+from data_process.joint_annotation.llm import ( # noqa: E402+ LLMClient,+ MAX_RETRIES,+ add_llm_args,+ append_id_list,+ read_id_list,+ save_json,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Entry classification+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _hips(entry):+ if not isinstance(entry, dict):+ return None, None+ r, l = entry.get("r_hip", {}), entry.get("l_hip", {})+ if not isinstance(r, dict) or not isinstance(l, dict):+ return None, None+ return r, l+++def is_non_empty_entry(entry):+ # type: (object) -> bool+ """True iff ``entry`` has a real (non-blank) r_hip/l_hip pair."""+ r, l = _hips(entry)+ return bool(r and l and r.get("raw") and l.get("raw"))+++def is_empty_entry(entry):+ # type: (object) -> bool+ """True for the canonical 'source==empty, both hips blank' sentinel+ (and for malformed entries, which are treated as empty)."""+ if not isinstance(entry, dict):+ return True+ if entry.get("source") != "empty":+ return False+ r, l = _hips(entry)+ if r is None:+ return True+ return not r.get("raw") and not l.get("raw")+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Driver+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def correct_all(raw_path, clean_path, face_path, client, system_prompt=SYSTEM_PROMPT,+ failed_list_path=None, max_tokens=512, max_retries=MAX_RETRIES,+ timeout_seconds=15, still_empty_path=None, only_if_currently_empty=True):+ # type: (...) -> Dict[str, Dict]+ """Re-run LLM face-joint selection and write results back IN-PLACE to+ ``face_path``.++ Targets every ``source == "empty"`` rig in ``face_path``, or only the+ ids in ``failed_list_path`` when given. Rigs where the LLM fails or+ still returns an empty entry are left untouched.++ A missing ``failed_list_path`` is a hard error: a typo must not silently+ escalate a targeted repair into a full sweep.+ """+ with open(raw_path, "r") as f:+ raw_data = json.load(f)+ with open(clean_path, "r") as f:+ clean_data = json.load(f)+ with open(face_path, "r") as f:+ face_data = json.load(f)++ if failed_list_path:+ if not os.path.exists(failed_list_path):+ raise FileNotFoundError(+ "--failed_list {!r} does not exist. Drop the flag to re-attempt "+ "every source=='empty' rig in {}.".format(failed_list_path, face_path))+ target_ids = read_id_list(failed_list_path)+ scope_desc = "{} rigs from {}".format(len(target_ids), failed_list_path)+ else:+ target_ids = sorted(k for k, v in face_data.items() if is_empty_entry(v))+ scope_desc = "{} empty-source rigs in {}".format(len(target_ids), face_path)++ backup = face_path + ".bak"+ if not os.path.exists(backup):+ shutil.copy2(face_path, backup)+ logger.info("Backed up {} -> {}".format(face_path, backup))++ if still_empty_path is None:+ still_empty_path = os.path.join(+ os.path.dirname(face_path) or ".", "still_empty_face_joints.txt")++ logger.info("Re-attempting {} (in-place update of {})".format(scope_desc, face_path))++ updated = skipped_not_empty = missing = 0+ still_empty = [] # type: List[str]+ pbar = tqdm(target_ids, desc="Correcting", unit="rig", dynamic_ncols=True)+ for rig_id in pbar:+ pbar.set_postfix_str(rig_id, refresh=False)++ if rig_id not in raw_data or rig_id not in clean_data:+ missing += 1+ logger.warning("[{}] missing from raw/clean; skipping".format(rig_id))+ continue++ # The failed list may be stale: only touch entries still empty now.+ if only_if_currently_empty and not is_empty_entry(face_data.get(rig_id)):+ skipped_not_empty += 1+ continue++ try:+ entry, used_fallback = resolve_rig(+ rig_id, raw_data[rig_id], clean_data[rig_id], client,+ system_prompt=system_prompt, max_tokens=max_tokens,+ max_retries=max_retries, timeout_seconds=timeout_seconds,+ fallback_on_error=False,+ )+ except Exception as err: # noqa: BLE001 โ exhausted retries: keep the empty entry+ logger.warning("[{}] LLM resolve failed: {}".format(rig_id, err))+ entry, used_fallback = None, True++ # Accept only clean non-empty LLM wins. used_fallback=True means the+ # rule-based hint came back on timeout โ the same hint that already+ # produced the empty entry.+ if entry is not None and not used_fallback and is_non_empty_entry(entry):+ face_data[rig_id] = entry+ updated += 1+ else:+ still_empty.append(rig_id)++ save_json(face_data, face_path)++ # Append (like every other id log in this stage) rather than truncate: a+ # --failed_list run only sees a subset, and a truncating write would+ # replace the global record with just that subset.+ known_empty = (set(read_id_list(still_empty_path))+ if os.path.exists(still_empty_path) else set())+ if still_empty:+ new_ids = [rig_id for rig_id in still_empty if rig_id not in known_empty]+ if new_ids:+ append_id_list(still_empty_path, new_ids)+ logger.info("{} rigs still empty ({} newly recorded) -> {}".format(+ len(still_empty), len(new_ids), still_empty_path))+ elif known_empty:+ logger.warning(+ "No rig came back empty in this run, but {} still lists {} id(s) "+ "from earlier runs โ it may be stale.".format(+ still_empty_path, len(known_empty)))++ logger.info(+ "Done. updated={} | still_empty={} | skipped_not_empty={} | "+ "missing_raw_or_clean={} | total={}".format(+ updated, len(still_empty), skipped_not_empty, missing, len(target_ids)))+ return face_data+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# CLI+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def main():+ parser = argparse.ArgumentParser(+ description="Re-run LLM face-joint selection on rigs whose source is 'empty'; "+ "updates face_joint_names.json in place when a pair is found.")+ parser.add_argument("--raw", required=True, help="Raw joint_names.json.")+ parser.add_argument("--cleaned", required=True, help="clean_joint_names.json.")+ parser.add_argument("--face", required=True,+ help="face_joint_names.json โ updated IN PLACE; a '.bak' sibling "+ "is made on first run.")+ parser.add_argument("--failed_list", default=None,+ help="Optional txt file with one rig id per line. Default: every "+ "rig whose entry in --face has source=='empty'.")+ parser.add_argument("--still_empty_log", default=None,+ help="Path to write rig ids the LLM still couldn't fill (default: "+ "sibling 'still_empty_face_joints.txt').")+ parser.add_argument("--force_reattempt", action="store_true",+ help="Re-run even if the current entry isn't source=='empty' anymore.")+ parser.add_argument("--rig_timeout", type=int, default=15,+ help="Per-rig wall-time budget in seconds (SIGALRM). 0 disables.")+ add_llm_args(parser, default_max_tokens=512)+ args = parser.parse_args()++ client = LLMClient.from_args(args)+ logger.info("{} | Writing in-place to: {}".format(client, args.face))++ correct_all(+ raw_path=args.raw,+ clean_path=args.cleaned,+ face_path=args.face,+ client=client,+ failed_list_path=args.failed_list,+ max_tokens=args.max_tokens,+ max_retries=args.max_retries,+ timeout_seconds=args.rig_timeout,+ still_empty_path=args.still_empty_log,+ only_if_currently_empty=not args.force_reattempt,+ )+++if __name__ == "__main__":+ main()diff --git a/data_process/joint_annotation/face_select_llm.py b/data_process/joint_annotation/face_select_llm.pynew file mode 100644index 0000000..0d2b5d2--- /dev/null+++ b/data_process/joint_annotation/face_select_llm.py@@ -0,0 +1,647 @@+"""Pick the facing-direction joint pair for each rig with a Language Model.++Complement to the rule-based :mod:`face_select_rule`: asks an LLM to choose ONE+bilaterally symmetric joint pair (hip / shoulder / wing / fin / ...) that+defines the rig's lateral axis for heading-angle computation. For serpentine+rigs with no bilateral symmetry, the LLM picks longitudinal body-axis+endpoints (head-tip / tail-tip) and sets ``body_axis: true``.++Backends (see :mod:`llm`): local HF causal LM, OpenAI API (default:+gpt-5-mini), DeepSeek API.++Inputs :+ ``joint_names.json`` โ ``{rig_id: [raw_joint_name, ...]}``+ ``clean_joint_names.json`` โ ``{rig_id: [clean_joint_name, ...]}``+Output :+ ``face_joint_names.json`` โ ``{rig_id: {r_hip, l_hip, source, [body_axis]}}``++Pipeline per rig:+ 1. Pre-filter the joint list into four small buckets (Right / Left /+ Head / Tail candidates). Dozens-of-joint rigs collapse to a handful.+ 2. Short-circuit 'empty' if no candidates survive (no LLM call).+ 3. Otherwise send only the compact candidate list to the LLM, together+ with the rule-based pick as a hint, and ask it to apply a three-step+ decision (bilateral -> body-axis -> empty).+ 4. Validate the response (raw names must appear verbatim in the input);+ retry with a structured correction on failure.+ 5. Fall back to the rule-based resolver after max retries / timeout.++Usage:+ python -m data_process.joint_annotation.face_select_llm \\+ --input_dir <export_dir> # OpenAI gpt-5-mini+ python -m data_process.joint_annotation.face_select_llm \\+ --input_dir <export_dir> --model deepseek-v4-flash+ python -m data_process.joint_annotation.face_select_llm \\+ --input_dir <export_dir> --model Qwen/Qwen3-8B+"""++import argparse+import json+import os+import re+import sys+import time++from loguru import logger+from tqdm import tqdm++_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++from data_process.joint_annotation.llm import ( # noqa: E402+ FatalLLMError,+ LLMClient,+ MAX_RETRIES,+ RigTimeout,+ add_llm_args,+ append_id_list,+ install_rig_alarm,+ load_json,+ read_id_list,+ save_json,+ strip_llm_noise,+ uninstall_rig_alarm,+)+from data_process.joint_annotation.face_select_rule import ( # noqa: E402+ HEAD_KEYWORDS,+ TAIL_KEYWORDS,+ empty_entry,+ resolve_face_joints,+ strip_trailing_num,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# System prompt+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++SYSTEM_PROMPT = (+ "# TASK\n"+ "Pick the joint pair that defines one 3D rig's lateral facing axis. "+ "Rigs come from Objaverse (Mixamo, 3ds Max Biped, Maya QuickRig, "+ "Unreal mannequin, CAT, custom) and Truebones animals. All clean "+ "names are already normalised to a canonical vocabulary โ reason on "+ "CLEAN names; RAW names are only for verbatim copy-back.\n\n"+ "# INPUT (user message)\n"+ "Four pre-filtered buckets (any may be empty):\n"+ " RIGHT : rows whose clean name starts with 'Right '\n"+ " LEFT : rows whose clean name starts with 'Left '\n"+ " HEAD : rows whose clean name (minus any trailing ' N') is one of\n"+ " {Head, Skull, Skull Base, Head End, Jaw, Upper Jaw,\n"+ " Lower Jaw, Tongue, Muzzle, Nose, Chin}\n"+ " TAIL : rows whose clean name (minus any trailing ' N') is 'Tail'\n"+ " or 'Tail Twist'\n"+ "Plus a 'Rule-based hint' JSON object โ the deterministic resolver's "+ "pick. Use it as a sanity check, not ground truth (see HINT below).\n\n"+ "# ALGORITHM\n"+ "Execute the steps in order. Do not skip ahead.\n"+ "STEP 1. Compute OVERLAP.\n"+ " - For each row in RIGHT, suffix = clean_name minus 'Right ' "+ "minus any trailing ' <digits>'. Collect into set R_SUFFIXES.\n"+ " - For each row in LEFT, suffix = clean_name minus 'Left ' "+ "minus any trailing ' <digits>'. Collect into set L_SUFFIXES.\n"+ " - OVERLAP = R_SUFFIXES โฉ L_SUFFIXES.\n"+ "STEP 2. If OVERLAP is non-empty โ APPLY RULE A AND RETURN.\n"+ " a. Pick suffix `s` = first element of OVERLAP that appears in "+ "PRIORITY (see below). If none appear, pick the alphabetically "+ "first element of OVERLAP.\n"+ " b. R_ROWS = RIGHT rows whose suffix == s; L_ROWS = LEFT rows "+ "whose suffix == s.\n"+ " c. If len(R_ROWS)==1 and len(L_ROWS)==1, pair them.\n"+ " Otherwise (chain duplicates, e.g. scorpion legs) match by "+ "DIGIT SIGNATURE: the tuple of ALL digit groups in the RAW name "+ "('Bip01_R_Thigh_4' -> (01,4)). Tier 1 โ pair rows whose full "+ "signatures are equal. Tier 2 โ no tier-1 match: drop the LAST "+ "group (the Objaverse per-joint global index: 'Bip01_R_Thigh_1_053' "+ "and 'Bip01_L_Thigh_1_054' both reduce to (01,1)) and pair on the "+ "rest. If neither tier matches, take R_ROWS[0] and L_ROWS[0].\n"+ " d. Output: source = s.lower(); body_axis = false.\n"+ " e. DO NOT consider HEAD or TAIL. Even if HEAD+TAIL look perfect, "+ "rule A wins whenever OVERLAP is non-empty.\n"+ "STEP 3. OVERLAP is empty. If HEAD is non-empty AND TAIL is non-"+ "empty โ APPLY RULE B AND RETURN.\n"+ " - r_hip = LAST HEAD row; l_hip = LAST TAIL row (chain endpoints).\n"+ " - source = 'body_axis'; body_axis = true.\n"+ "STEP 4. Otherwise โ APPLY RULE C (empty).\n"+ " - Both hips are {raw: '', clean: ''}. source = 'empty'; "+ "body_axis = false.\n"+ " - Vehicles, props, and abstract rigs land here. NEVER pair an "+ "unrelated Right/Left entry (e.g. lone 'Right Eye' without left "+ "mate) just to avoid emitting empty.\n\n"+ "# PRIORITY (highest โ lowest; first match in OVERLAP wins)\n"+ "Hip-level : thigh, shoulder, front shoulder, back hip, hip, "+ "scapula\n"+ "Whole-limb : upper arm, arm, front leg, hind leg, middle leg, "+ "back leg, wing, leg\n"+ "Aquatic : pectoral fin, pelvic fin, fin, gill\n"+ "Arthropod : pincer, mandible, large mandible, lower mandible, "+ "stinger, claw, hand claw, antenna\n"+ "Mid-limb : forearm, shin, knee, elbow, ankle, wrist\n"+ "Extremity : hand, palm, foot, heel, front paw, back paw, paw, "+ "front hoof, rear hoof, hoof, fetlock, cannon, metacarpus, "+ "pastern, toe\n"+ "Fingers : thumb finger, index finger, middle finger, ring "+ "finger, pinky finger, finger, neck\n"+ "Weak : eye, eyeball, eyelid, eyebrow, ear, horn, cheek, "+ "whisker, fang, barbel, tentacle, feather\n"+ "Last : tail\n"+ "Suffix comparison is case-insensitive but uses the clean-name form "+ "('Upper Arm', 'Front Shoulder', 'Pectoral Fin'). Read the list above "+ "LITERALLY, left to right, top to bottom โ it is exactly the order the "+ "rule-based resolver uses, so do not re-rank quadruped markers "+ "('Front Shoulder', 'Back Hip', 'Front Leg') against their plain "+ "counterparts ('Shoulder', 'Hip', 'Leg'). Note in particular that "+ "'shoulder' comes BEFORE 'front shoulder', while 'front leg' comes "+ "BEFORE 'leg'.\n\n"+ "# HINT\n"+ "The hint is the rule-based resolver's output. It is usually right "+ "but can err on edge cases:\n"+ " - Hint says 'body_axis' or 'empty' yet OVERLAP is non-empty โ "+ "hint is wrong, apply STEP 2 (rule A wins).\n"+ " - Hint swapped sides (Right/Left in wrong fields) โ fix.\n"+ " - Hint chose a lower-priority suffix than STEP 2a finds โ "+ "override with the higher-priority one.\n"+ " - Hint paired mismatched chain indices on a multi-leg rig โ fix "+ "via STEP 2c raw-suffix match.\n"+ " - Your algorithm yields the same answer as the hint โ return "+ "the hint verbatim.\n\n"+ "# OUTPUT\n"+ "Exactly one JSON object, no prose, no markdown fences, no comments:\n"+ ' {"r_hip": {"raw": "<exact>", "clean": "<exact>"},\n'+ ' "l_hip": {"raw": "<exact>", "clean": "<exact>"},\n'+ ' "source": "<lowercase suffix or \'body_axis\' or \'empty\'>",\n'+ ' "body_axis": <true|false>}\n\n'+ "# INVARIANTS (verify before emitting)\n"+ " I1. r_hip.raw != l_hip.raw, UNLESS both are '' (empty case).\n"+ " I2. r_hip.raw and l_hip.raw each appear verbatim in the rig's "+ "input rows (copied character-for-character).\n"+ " I3. r_hip always holds the Right (or head) side; l_hip always "+ "holds the Left (or tail) side. Never swap.\n"+ " I4. body_axis == true IFF source == 'body_axis'.\n"+ " I5. source is lowercase. If rule A fired, source is the suffix "+ "lowercased with single spaces (e.g. 'thigh', 'front shoulder', "+ "'pectoral fin').\n"+ " I6. Unless body_axis, r_hip and l_hip mirror the SAME part: "+ "their clean names minus the side prefix and any trailing digits "+ "are identical. Never pair different parts ('Right Thigh' with "+ "'Left Shoulder' is invalid even though the sides are correct).\n\n"+ "# WORKED EXAMPLES\n"+ "## Ex1 โ Mixamo humanoid (rule A, hip-level pick)\n"+ "RIGHT has 'Right Thigh' + 'Right Shoulder'; LEFT has the mirrors. "+ "OVERLAP = {Thigh, Shoulder}. Thigh outranks Shoulder โ pick Thigh.\n"+ ' {"r_hip": {"raw": "mixamorig:RightUpLeg_056", "clean": '+ '"Right Thigh"},\n'+ ' "l_hip": {"raw": "mixamorig:LeftUpLeg_056", "clean": '+ '"Left Thigh"},\n'+ ' "source": "thigh", "body_axis": false}\n'+ "## Ex2 โ Objaverse quadruped (Front Shoulder beats Back Hip)\n"+ "No Thigh pair; OVERLAP = {Front Shoulder, Back Hip}. Front "+ "Shoulder outranks Back Hip.\n"+ ' {"r_hip": {"raw": "F_R_Shoulder_012", "clean": '+ '"Right Front Shoulder"},\n'+ ' "l_hip": {"raw": "F_L_Shoulder_012", "clean": '+ '"Left Front Shoulder"},\n'+ ' "source": "front shoulder", "body_axis": false}\n'+ "## Ex3 โ Scorpion multi-leg (STEP 2c raw-suffix match)\n"+ "RIGHT has three 'Right Thigh' rows (Bip01_R_Thigh_1/_2/_4); LEFT "+ "the mirrors. OVERLAP = {Thigh}. Digit signatures: (01,1) on both "+ "sides -> tier-1 match.\n"+ ' {"r_hip": {"raw": "Bip01_R_Thigh_1", "clean": "Right Thigh"},\n'+ ' "l_hip": {"raw": "Bip01_L_Thigh_1", "clean": "Left Thigh"},\n'+ ' "source": "thigh", "body_axis": false}\n'+ "## Ex4 โ OVERRIDE a mistaken body_axis hint\n"+ "RIGHT has 'Right Thigh'; LEFT has 'Left Thigh'; HEAD has 'Tongue'; "+ "TAIL has 'Tail'. Hint = body_axis Tongue+Tail. OVERLAP = {Thigh} "+ "is non-empty โ STEP 2 wins, hint is wrong.\n"+ ' {"r_hip": {"raw": "RightUpLeg_033", "clean": "Right Thigh"},\n'+ ' "l_hip": {"raw": "LeftUpLeg_028", "clean": "Left Thigh"},\n'+ ' "source": "thigh", "body_axis": false}\n'+ "## Ex5 โ Snake / serpent (rule B fires)\n"+ "RIGHT and LEFT are empty; HEAD has 'Head'; TAIL has 'Tail_30'. "+ "STEP 3 applies.\n"+ ' {"r_hip": {"raw": "Head", "clean": "Head"},\n'+ ' "l_hip": {"raw": "Tail_30", "clean": "Tail"},\n'+ ' "source": "body_axis", "body_axis": true}\n'+ "## Ex6 โ Vehicle / prop (rule C fires)\n"+ "All sections empty, or only a lone 'Right Eye' with no left mate.\n"+ ' {"r_hip": {"raw": "", "clean": ""},\n'+ ' "l_hip": {"raw": "", "clean": ""},\n'+ ' "source": "empty", "body_axis": false}\n'+)+++def prefilter_candidates(raw_names, clean_names):+ # type: (List[str], List[str]) -> Dict[str, List[Dict]]+ """Partition joints into LLM-relevant sections.++ Returns a dict with four keys โ ``right``, ``left``, ``head``, ``tail``+ โ each a list of ``{"raw": ..., "clean": ...}`` dicts preserving input+ order. Most rigs have 60+ joints but only a handful are candidates for+ the facing direction, so this drastically shortens the LLM's context.+ """+ buckets = {"right": [], "left": [], "head": [], "tail": []}+ for raw, clean in zip(raw_names, clean_names):+ if clean.startswith("Right "):+ buckets["right"].append({"raw": raw, "clean": clean})+ continue+ if clean.startswith("Left "):+ buckets["left"].append({"raw": raw, "clean": clean})+ continue+ base = strip_trailing_num(clean)+ if base in HEAD_KEYWORDS:+ buckets["head"].append({"raw": raw, "clean": clean})+ if base in TAIL_KEYWORDS:+ buckets["tail"].append({"raw": raw, "clean": clean})+ return buckets+++def _format_section(name, rows):+ # type: (str, List[Dict]) -> str+ if not rows:+ return "{}: (none)".format(name)+ lines = ["{}:".format(name)]+ for row in rows:+ lines.append(" raw={!r:<40s} clean={!r}".format(row["raw"], row["clean"]))+ return "\n".join(lines)+++def build_user_prompt(rig_id, candidates, rule_hint=None, correction=None):+ # type: (str, Dict[str, List[Dict]], Optional[Dict], Optional[str]) -> str+ sections = [+ "Rig identifier: {}".format(rig_id),+ "Candidate counts: right={} left={} head={} tail={}".format(+ len(candidates["right"]), len(candidates["left"]),+ len(candidates["head"]), len(candidates["tail"])),+ "",+ _format_section("RIGHT", candidates["right"]),+ "",+ _format_section("LEFT", candidates["left"]),+ "",+ _format_section("HEAD", candidates["head"]),+ "",+ _format_section("TAIL", candidates["tail"]),+ "",+ ]+ if rule_hint is not None:+ sections.extend([+ "Rule-based hint (deterministic resolver's pick โ HINT, not ground truth):",+ json.dumps(rule_hint, ensure_ascii=False, indent=2),+ "",+ ])+ sections.append(+ "Apply rules A -> B -> C and return the JSON object only. If the "+ "hint already satisfies A/B/C, return it verbatim; otherwise override."+ )+ prompt = "\n".join(sections)+ if correction:+ prompt += "\n\nIMPORTANT: " + correction+ return prompt+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Response parsing+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _hip_fields(hip, side):+ # type: (object, str) -> Tuple[str, str]+ if not isinstance(hip, dict):+ raise ValueError("{} is not a JSON object".format(side))+ raw = hip.get("raw", "")+ clean = hip.get("clean", "")+ if not isinstance(raw, str) or not isinstance(clean, str):+ raise ValueError("{} raw/clean must be strings".format(side))+ return raw, clean+++def _clean_side(clean_name):+ # type: (str) -> Optional[str]+ """``"Left"`` / ``"Right"`` if the clean label carries a side, else None."""+ if clean_name.startswith("Left "):+ return "Left"+ if clean_name.startswith("Right "):+ return "Right"+ return None+++def _validate_sides(r_clean, l_clean, body_axis):+ # type: (str, str, bool) -> None+ """Enforce invariant I3: ``r_hip`` holds the Right (or head) side and+ ``l_hip`` the Left (or tail) side.++ The prompt states I3 but a model can still return a swapped pair, and a+ swap is invisible downstream: ``get_root_facing_quat`` just builds the+ lateral axis the other way round, silently rotating every clip of that+ object type by 180 degrees. Raise ``ValueError`` so the caller retries+ with a correction and, if that fails too, falls back to the rule-based+ resolver.+ """+ if body_axis:+ if (strip_trailing_num(r_clean) in TAIL_KEYWORDS+ or strip_trailing_num(l_clean) in HEAD_KEYWORDS):+ raise ValueError(+ "body-axis pair is swapped: r_hip must be the HEAD end and "+ "l_hip the TAIL end, got r_hip={!r} / l_hip={!r}".format(+ r_clean, l_clean))+ return+ if _clean_side(r_clean) == "Left" or _clean_side(l_clean) == "Right":+ raise ValueError(+ "left/right pair is swapped: r_hip must be the Right-side joint "+ "and l_hip the Left-side one, got r_hip={!r} / l_hip={!r}".format(+ r_clean, l_clean))+++def parse_response(text, raw_names, clean_names):+ # type: (str, List[str], List[str]) -> Dict+ """Extract and validate a ``face_joint_names.json``-shaped object.++ Tolerates markdown fences and ``<think>โฆ</think>`` blocks. Raises+ ``ValueError`` on structural mismatch so the caller can retry or fall+ back. Validation rules:++ - ``r_hip.raw`` / ``l_hip.raw`` must exist verbatim in ``raw_names``+ (or both be ``""`` for the empty sentinel).+ - ``r_hip.raw != l_hip.raw``.+ - The clean name is re-sourced from ``clean_names`` so the LLM can't+ drift the canonical label.+ - The sides are not swapped (invariant I3, see :func:`_validate_sides`).+ """+ if not text or not text.strip():+ raise ValueError("empty response")++ cleaned = strip_llm_noise(text)+ try:+ data = json.loads(cleaned)+ except json.JSONDecodeError:+ start = cleaned.find("{")+ end = cleaned.rfind("}")+ if start == -1 or end == -1 or end <= start:+ raise ValueError("no JSON object found in response")+ data = json.loads(cleaned[start:end + 1])++ if not isinstance(data, dict):+ raise ValueError("response is not a JSON object")+ for key in ("r_hip", "l_hip", "source"):+ if key not in data:+ raise ValueError("missing key '{}'".format(key))++ r_raw, _ = _hip_fields(data["r_hip"], "r_hip")+ l_raw, _ = _hip_fields(data["l_hip"], "l_hip")+ source = str(data["source"]).strip().lower()+ body_axis = bool(data.get("body_axis", False))++ if not r_raw and not l_raw:+ return empty_entry()+ if not r_raw or not l_raw:+ raise ValueError("one of r_hip/l_hip is empty; both must be filled "+ "or both empty for 'empty' source")+ if r_raw == l_raw:+ raise ValueError("r_hip.raw == l_hip.raw ({!r}): pair must be two "+ "distinct joints".format(r_raw))+ if r_raw not in raw_names:+ raise ValueError("r_hip.raw {!r} not in input raw joint list".format(r_raw))+ if l_raw not in raw_names:+ raise ValueError("l_hip.raw {!r} not in input raw joint list".format(l_raw))++ r_idx = raw_names.index(r_raw)+ l_idx = raw_names.index(l_raw)+ is_axis = body_axis or source == "body_axis"+ _validate_sides(clean_names[r_idx], clean_names[l_idx], is_axis)+ if not is_axis:+ # Invariant I6: a lateral pair must mirror the SAME part โ a+ # cross-part pair ('Right Thigh' + 'Left Shoulder') passes the side+ # check yet yields a skewed facing axis.+ r_base = strip_trailing_num(+ re.sub(r'^(Left|Right)\s+', '', clean_names[r_idx]))+ l_base = strip_trailing_num(+ re.sub(r'^(Left|Right)\s+', '', clean_names[l_idx]))+ if r_base.lower() != l_base.lower():+ raise ValueError(+ "r_hip and l_hip are different parts ({!r} vs {!r}); the "+ "pair must be the mirrored copies of one joint".format(+ clean_names[r_idx], clean_names[l_idx]))+ entry = {+ "r_hip": {"raw": raw_names[r_idx], "clean": clean_names[r_idx]},+ "l_hip": {"raw": raw_names[l_idx], "clean": clean_names[l_idx]},+ "source": source or "pair",+ }+ if body_axis or source == "body_axis":+ entry["body_axis"] = True+ return entry+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-rig resolver (retry + rule-based hint + fallback + timeout)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def resolve_rig(rig_id, raw_names, clean_names, client, system_prompt=SYSTEM_PROMPT,+ max_tokens=512, max_retries=MAX_RETRIES, timeout_seconds=15,+ fallback_on_error=True):+ # type: (...) -> Tuple[Dict, bool]+ """Pick the face-direction joint pair for one rig.++ Returns ``(entry, used_fallback)``. Rigs with no Right/Left or+ Head/Tail candidates short-circuit to the empty entry without an LLM+ call. On exhausted retries or timeout the rule-based pick is returned+ (or ``RuntimeError`` is raised when ``fallback_on_error`` is False and+ no timeout occurred).+ """+ if not raw_names:+ return empty_entry(), False+ if len(raw_names) != len(clean_names):+ # clean_joint_names is indexed positionally against raw_names; a+ # stale/short list silently pairs the wrong bones (same guard as the+ # rule-based CLI). Abort instead of quietly emitting 'empty'.+ raise ValueError(+ "{!r}: {} cleaned names vs {} raw names โ clean_joint_names.json "+ "is stale; re-run the joint-name cleaner.".format(+ rig_id, len(clean_names), len(raw_names)))++ candidates = prefilter_candidates(raw_names, clean_names)+ has_bilateral = bool(candidates["right"]) and bool(candidates["left"])+ has_body_axis = bool(candidates["head"]) and bool(candidates["tail"])+ if not has_bilateral and not has_body_axis:+ return empty_entry(), False++ # Rule-based hint (cheap, deterministic); also the fallback result.+ rule_hint = resolve_face_joints(clean_names, raw_names)++ last_err = None+ correction = None+ timed_out = False++ prev_handler = install_rig_alarm(timeout_seconds)+ try:+ for attempt in range(1, max_retries + 1):+ try:+ raw_text = client.generate(+ system_prompt,+ build_user_prompt(rig_id, candidates, rule_hint=rule_hint,+ correction=correction),+ max_tokens)+ return parse_response(raw_text, raw_names, clean_names), False+ except (RigTimeout, FatalLLMError):+ raise+ except Exception as err: # noqa: BLE001 โ retry on any backend/parse error+ last_err = err+ if isinstance(err, ValueError):+ correction = (+ "Your previous response failed validation ({}). Return "+ "exactly one JSON object with keys r_hip, l_hip, source, "+ "body_axis. r_hip and l_hip must be objects with 'raw' "+ "and 'clean' string fields. 'raw' must be an exact copy "+ "of one of the rig's raw joint names (or \"\" for the "+ "'empty' source). r_hip.raw != l_hip.raw. r_hip MUST "+ "be the Right-side joint (or, for body_axis, the HEAD "+ "end) and l_hip the Left-side one (the TAIL end) โ "+ "never swap them, and both must mirror the SAME part. "+ "No prose, no code fences."+ ).format(err)+ logger.warning("[{}] attempt {}/{} failed: {}".format(+ rig_id, attempt, max_retries, err))+ if attempt < max_retries:+ time.sleep(1.5)+ except RigTimeout:+ timed_out = True+ finally:+ uninstall_rig_alarm(prev_handler)++ if not fallback_on_error and not timed_out:+ raise RuntimeError("LLM face-joint resolution failed for {}: {}".format(+ rig_id, last_err))++ if timed_out:+ logger.warning("[{}] exceeded {}s budget; using rule-based hint".format(+ rig_id, timeout_seconds))+ else:+ logger.error("[{}] all {} attempts failed ({}); falling back to "+ "rule-based resolver".format(rig_id, max_retries, last_err))+ return rule_hint, True+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Batch driver+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _is_complete(entry):+ # type: (Dict) -> bool+ """A cached entry is complete if it's well-formed (even 'empty' counts)."""+ return (isinstance(entry, dict)+ and "r_hip" in entry and "l_hip" in entry and "source" in entry)+++def resolve_all(raw_path, clean_path, output_path, client, system_prompt=SYSTEM_PROMPT,+ max_tokens=512, max_retries=MAX_RETRIES, timeout_seconds=15,+ fallback_on_error=True, failed_path=None):+ # type: (...) -> Dict[str, Dict]+ """Resolve face joints for every rig in ``raw_path`` / ``clean_path``.++ Resumes from a partial output file. Rig ids that fall back to the+ rule-based resolver are appended to ``failed_path`` (one per line).+ """+ with open(raw_path, "r") as f:+ raw_data = json.load(f)+ with open(clean_path, "r") as f:+ clean_data = json.load(f)++ result = load_json(output_path)+ rig_ids = sorted(set(raw_data.keys()) & set(clean_data.keys()))+ missing = set(raw_data) ^ set(clean_data)+ if missing:+ logger.warning("{} rig ids are not in both raw+clean; skipping: {}"+ .format(len(missing), sorted(missing)[:5]))++ if failed_path is None:+ failed_path = os.path.join(+ os.path.dirname(output_path) or ".", "failed_face_joints.txt")+ already_failed = set(read_id_list(failed_path)) if os.path.exists(failed_path) else set()++ fallback_count = 0+ pbar = tqdm(rig_ids, desc="Resolving", unit="rig", dynamic_ncols=True)+ for rig_id in pbar:+ pbar.set_postfix_str(rig_id, refresh=False)+ if _is_complete(result.get(rig_id)):+ continue++ entry, used_fallback = resolve_rig(+ rig_id, raw_data[rig_id], clean_data[rig_id], client,+ system_prompt=system_prompt, max_tokens=max_tokens,+ max_retries=max_retries, timeout_seconds=timeout_seconds,+ fallback_on_error=fallback_on_error,+ )+ fallback_count += int(used_fallback)+ result[rig_id] = entry+ save_json(result, output_path)++ if used_fallback and rig_id not in already_failed:+ append_id_list(failed_path, [rig_id])+ already_failed.add(rig_id)++ logger.info("Done. {} rigs resolved ({} via rule-based fallback) -> {}"+ .format(len(result), fallback_count, output_path))+ if fallback_count:+ logger.info("Failed rig ids recorded at {}".format(failed_path))+ return result+++def print_source_summary(result):+ # type: (Dict[str, Dict]) -> None+ sources = {}+ for rig_id, entry in result.items():+ sources.setdefault(entry["source"], []).append(rig_id)+ print("\nResolved {} rigs. Breakdown:".format(len(result)))+ for src, rigs in sorted(sources.items()):+ print(" [{}] {} rigs".format(src, len(rigs)))+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# CLI+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def main():+ parser = argparse.ArgumentParser(+ description="Pick facing-direction joint pairs via an LLM (local / openai / deepseek).")+ parser.add_argument("--input_dir", type=str, required=True,+ help="Directory containing joint_names.json and clean_joint_names.json.")+ parser.add_argument("--raw_name", type=str, default="joint_names.json",+ help="Filename of raw joint names inside --input_dir.")+ parser.add_argument("--clean_name", type=str, default="clean_joint_names.json",+ help="Filename of cleaned joint names inside --input_dir.")+ parser.add_argument("--output_name", type=str, default="face_joint_names.json",+ help="Output filename written inside --input_dir (ignored if "+ "--output is set).")+ parser.add_argument("--output", type=str, default=None,+ help="Explicit output path (default: <input_dir>/<output_name>).")+ parser.add_argument("--no_fallback", action="store_true",+ help="Raise on LLM failure instead of falling back to the "+ "rule-based resolver.")+ parser.add_argument("--failed_log", default=None,+ help="Path to append rig ids that hit the rule-based fallback "+ "(default: sibling 'failed_face_joints.txt').")+ parser.add_argument("--rig_timeout", type=int, default=15,+ help="Per-rig wall-time budget in seconds (SIGALRM). 0 disables.")+ add_llm_args(parser, default_max_tokens=512)+ args = parser.parse_args()++ raw_path = os.path.join(args.input_dir, args.raw_name)+ clean_path = os.path.join(args.input_dir, args.clean_name)+ output_path = args.output or os.path.join(args.input_dir, args.output_name)+ client = LLMClient.from_args(args)+ logger.info("{} | Dir: {} | Output: {}".format(client, args.input_dir, output_path))++ result = resolve_all(+ raw_path=raw_path,+ clean_path=clean_path,+ output_path=output_path,+ client=client,+ max_tokens=args.max_tokens,+ max_retries=args.max_retries,+ timeout_seconds=args.rig_timeout,+ fallback_on_error=not args.no_fallback,+ failed_path=args.failed_log,+ )+ print_source_summary(result)+++if __name__ == "__main__":+ main()diff --git a/data_process/joint_annotation/face_select_rule.py b/data_process/joint_annotation/face_select_rule.pynew file mode 100644index 0000000..dc0ac9d--- /dev/null+++ b/data_process/joint_annotation/face_select_rule.py@@ -0,0 +1,372 @@+"""Rule-based facing-direction joint selection โ ``face_joint_names.json``.++For each rig this script identifies one symmetric joint pair that defines+the lateral (left-right) axis used to compute the facing direction:++ 1. Search clean joint names for any symmetrical "Right <X>" / "Left <X>"+ pair, preferring anatomically meaningful joints (Thigh, Shoulder, ...).+ 2. For body-axis animals (snakes like Anaconda) that lack bilateral+ symmetry, use longitudinal body-axis endpoints (tail-tip / head-tip)+ and flag ``"body_axis": true``. The tail-to-head vector is directly+ the facing direction (no cross product with Y-up needed).+ 3. Animals with opaque joint names where no meaningful pair can be+ extracted get ``"source": "empty"``.++:mod:`face_select_llm` is the LLM-based complement and uses this+module both as a hint and as its fallback.++Usage:+ python -m data_process.joint_annotation.face_select_rule \+ --input_dir <export_dir> [--output PATH]+"""++import argparse+import copy+import json+import os+import re+from typing import Dict, List, Optional, Tuple++# Clean-name keywords for body-axis endpoint detection (serpentine skeletons+# and any rig with no bilateral symmetry but a clear longitudinal axis).+TAIL_KEYWORDS = {"Tail", "Tail Twist"}+HEAD_KEYWORDS = {"Head", "Skull", "Skull Base", "Head End", "Jaw",+ "Upper Jaw", "Lower Jaw", "Tongue", "Muzzle", "Nose", "Chin"}++# Priority order for symmetrical pair search (suffix after stripping the+# "Right "/"Left " prefix). Higher entries win.+#+# Ordering rationale (top โ bottom):+# 1. Hip-level limbs closest to the rig root give the most stable facing+# axis (Thigh, Shoulder, and the quadruped "Front Shoulder"/"Back Hip"+# markers Objaverse quadrupeds use after names_clean_rule.py).+# 2. Whole-limb roots (Upper Arm, Front Leg, Hind Leg, Wing, ...) come+# next โ they're unique per side and rarely ambiguous.+# 3. Aquatic / arthropod homologues (fins, pincers, mandibles) appear+# where they're anatomically equivalent to the limbs above.+# 4. Mid-limb joints (Forearm, Shin, Knee, Elbow, ...) are good fallbacks+# but drift a few cm from the true hip axis.+# 5. Extremities (Hand, Foot, Paw, Hoof, Claw) are weakest โ they bend a+# lot during motion โ so they sit near the bottom.+# 6. Head-adjacent bilateral parts (Eye, Ear, Whisker, Antenna) are last+# resorts; they give a cross-head axis, not a hip axis.+# 7. Tail is the absolute bottom (often a chain; the 'body_axis' fallback+# uses it as a longitudinal endpoint instead).+SYMMETRIC_PAIR_PRIORITY = [+ # --- Hip / shoulder level (best) -----------------------------------+ "Thigh",+ "Shoulder",+ "Front Shoulder", # quadruped pec (Objaverse)+ "Back Hip", # quadruped pelvis (Objaverse)+ "Hip",+ "Scapula",+ # --- Whole-limb roots ----------------------------------------------+ "Upper Arm",+ "Arm",+ "Front Leg",+ "Hind Leg",+ "Middle Leg", # insect / arachnid+ "Back Leg",+ "Wing",+ "Leg",+ # --- Aquatic / arthropod homologues --------------------------------+ "Pectoral Fin",+ "Pelvic Fin",+ "Fin", # generic fish fin (L_Fin1-style rigs)+ "Gill",+ "Pincer",+ "Mandible",+ "Large Mandible",+ "Lower Mandible",+ "Stinger",+ "Claw",+ "Hand Claw",+ "Antenna",+ # --- Mid-limb joints -----------------------------------------------+ "Forearm",+ "Shin",+ "Knee",+ "Elbow",+ "Ankle",+ "Wrist",+ # --- Extremities ---------------------------------------------------+ "Hand",+ "Palm",+ "Foot",+ "Heel",+ "Front Paw",+ "Back Paw",+ "Paw",+ "Front Hoof",+ "Rear Hoof",+ "Hoof",+ "Fetlock",+ "Cannon",+ "Metacarpus",+ "Pastern",+ "Toe",+ # --- Fingers (hand-to-hand axis; both hands move, so below feet) ---+ "Thumb Finger",+ "Index Finger",+ "Middle Finger",+ "Ring Finger",+ "Pinky Finger",+ "Finger",+ # --- Coat / soft ---------------------------------------------------+ "Neck",+ # --- Head-adjacent bilateral parts (weak) --------------------------+ "Eye",+ "Eyeball",+ "Eyelid",+ "Eyebrow",+ "Ear",+ "Horn",+ "Cheek",+ "Whisker",+ "Fang",+ "Barbel",+ "Tentacle",+ "Feather",+ # --- Last resort ---------------------------------------------------+ "Tail",+]+++def strip_trailing_num(name: str) -> str:+ """'Thigh 1' -> 'Thigh'."""+ return re.sub(r"\s+\d+$", "", name)+++def _digit_runs(raw_name: str) -> Tuple[str, ...]:+ """All digit groups in a raw name: 'Bip01_R_Thigh_4' -> ('01', '4')."""+ return tuple(re.findall(r"\d+", raw_name))+++def _match_pair_by_suffix(+ r_indices: List[int],+ l_indices: List[int],+ raw_names: List[str],+) -> Tuple[int, int]:+ """Pair one Right index with one Left index by matching raw-name digits.++ When multiple joints share the same clean name (e.g. multiple "Right+ Thigh" in Scorpion), pairs are matched by the digit signature of the raw+ name so that 'Bip01_R_Thigh_4' pairs with 'Bip01_L_Thigh_4', not+ 'Bip01_L_Thigh_1'. Two tiers:++ 1. Full digit signature โ rigs whose mirrored bones share every index.+ 2. Signature minus the LAST group โ the Objaverse export stamps a+ per-joint global index on every name ('Bip01_R_Thigh_1_053' vs+ 'Bip01_L_Thigh_1_054'), so the full signatures never match; dropping+ the final group compares only the chain indices.++ Falls back to the first of each side when neither tier matches.+ """+ if len(r_indices) == 1 and len(l_indices) == 1:+ return r_indices[0], l_indices[0]++ for key_fn in (lambda runs: runs, lambda runs: runs[:-1]):+ l_by_sig: Dict[Tuple[str, ...], int] = {}+ for li in l_indices:+ l_by_sig.setdefault(key_fn(_digit_runs(raw_names[li])), li)+ for ri in r_indices:+ li = l_by_sig.get(key_fn(_digit_runs(raw_names[ri])))+ if li is not None:+ return ri, li++ return r_indices[0], l_indices[0]+++def find_symmetric_pair(+ clean_names: List[str],+ raw_names: List[str],+) -> Tuple[Optional[int], Optional[int], Optional[str]]:+ """Find the highest-priority "Right <X>" / "Left <X>" pair.++ Returns ``(r_idx, l_idx, suffix)`` or ``(None, None, None)``.+ """+ right_map: Dict[str, List[int]] = {}+ left_map: Dict[str, List[int]] = {}+ for i, name in enumerate(clean_names):+ if name.startswith("Right "):+ right_map.setdefault(strip_trailing_num(name[6:]), []).append(i)+ elif name.startswith("Left "):+ left_map.setdefault(strip_trailing_num(name[5:]), []).append(i)++ common = set(right_map) & set(left_map)+ if not common:+ return None, None, None++ ordered = [s for s in SYMMETRIC_PAIR_PRIORITY if s in common]+ ordered += sorted(common - set(SYMMETRIC_PAIR_PRIORITY))+ suffix = ordered[0]+ ri, li = _match_pair_by_suffix(right_map[suffix], left_map[suffix], raw_names)+ return ri, li, suffix+++def find_body_axis_pair(+ clean_names: List[str],+) -> Tuple[Optional[int], Optional[int]]:+ """Find body-axis endpoints (tail-tip and head-end) from clean names.++ For serpentine animals without bilateral symmetry, facing direction is+ the longitudinal axis from tail-tip to head-end. Uses the last occurrence+ of each keyword set to land on the extremities.+ """+ tail_idx: Optional[int] = None+ head_idx: Optional[int] = None+ for i, name in enumerate(clean_names):+ base = strip_trailing_num(name)+ if base in TAIL_KEYWORDS:+ tail_idx = i+ if base in HEAD_KEYWORDS:+ head_idx = i+ if tail_idx is not None and head_idx is not None:+ return tail_idx, head_idx+ return None, None+++# Sentinel for "no facing pair could be determined". Always hand out a+# ``copy.deepcopy`` of it โ a shallow ``dict()`` copy shares the r_hip/l_hip+# sub-dicts with this module-level constant, so one in-place mutation of a+# returned entry would poison every other 'empty' entry.+EMPTY_ENTRY = {+ "r_hip": {"raw": "", "clean": ""},+ "l_hip": {"raw": "", "clean": ""},+ "source": "empty",+}+++def empty_entry() -> Dict:+ """A fresh, independent copy of :data:`EMPTY_ENTRY`."""+ return copy.deepcopy(EMPTY_ENTRY)+++def _make_entry(+ raw_names: List[str],+ clean_names: List[str],+ r_idx: int,+ l_idx: int,+ source: str,+) -> Dict:+ return {+ "r_hip": {"raw": raw_names[r_idx], "clean": clean_names[r_idx]},+ "l_hip": {"raw": raw_names[l_idx], "clean": clean_names[l_idx]},+ "source": source,+ }+++def resolve_face_joints(+ clean_names: List[str],+ raw_names: List[str],+) -> Dict:+ """Resolve one joint pair defining the facing direction."""+ r_idx, l_idx, suffix = find_symmetric_pair(clean_names, raw_names)+ if r_idx is not None:+ return _make_entry(raw_names, clean_names, r_idx, l_idx, suffix.lower())++ # r_hip = head-end (facing toward), l_hip = tail-tip (facing away from).+ tail_idx, head_idx = find_body_axis_pair(clean_names)+ if tail_idx is not None:+ entry = _make_entry(raw_names, clean_names, head_idx, tail_idx, "body_axis")+ entry["body_axis"] = True+ return entry++ return empty_entry()+++def main() -> None:+ parser = argparse.ArgumentParser(+ description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter,+ )+ parser.add_argument(+ "--input_dir", type=str, nargs="+", required=True,+ help="Directory(ies) containing joint_names.json and clean_joint_names.json",+ )+ parser.add_argument(+ "--raw_name", type=str, default="joint_names.json",+ help="Filename of raw joint names inside each --input_dir",+ )+ parser.add_argument(+ "--clean_name", type=str, default="clean_joint_names.json",+ help="Filename of cleaned joint names inside each --input_dir",+ )+ parser.add_argument(+ "--output_name", type=str, default="face_joint_names.json",+ help="Output filename written inside each --input_dir (ignored if --output is set)",+ )+ parser.add_argument(+ "--output", type=str, default=None,+ help="Explicit output path (default: <input_dir>/<output_name> per directory)",+ )+ args = parser.parse_args()++ # A single --output cannot hold the results of several --input_dir: only+ # the last directory would survive. Make that an error instead.+ if args.output and len(args.input_dir) > 1:+ parser.error(+ "--output takes a single path but {} --input_dir were given; drop "+ "--output (use --output_name, written inside each directory) or "+ "pass one directory at a time.".format(len(args.input_dir)))++ for input_dir in args.input_dir:+ raw_path = os.path.join(input_dir, args.raw_name)+ clean_path = os.path.join(input_dir, args.clean_name)+ with open(raw_path) as f:+ raw_data = json.load(f)+ with open(clean_path) as f:+ clean_data = json.load(f)++ # Drive the loop off the RAW key set: it is the authority on which+ # object types exist, and stage 4 raises when one has no face entry.+ extra = sorted(set(clean_data) - set(raw_data))+ if extra:+ print("WARNING: {} keys in {} are absent from {} and are ignored: "+ "{}".format(len(extra), clean_path, raw_path, extra[:5]))++ result = {}+ for animal in raw_data:+ raw_names = raw_data[animal]+ if animal not in clean_data:+ raise KeyError(+ "{!r} is in {} but missing from {}; re-run the joint-name "+ "cleaner so every object type has cleaned names."+ .format(animal, raw_path, clean_path))+ clean_names = clean_data[animal]+ # clean_joint_names is indexed POSITIONALLY against raw_names, so a+ # stale/short list would silently emit an r_hip.raw taken from the+ # wrong position โ a real bone name that passes stage 4's+ # membership check and yields a wrong facing axis.+ if len(clean_names) != len(raw_names):+ raise ValueError(+ "{!r}: {} has {} cleaned names but {} has {} raw names; "+ "{} is stale โ re-run the joint-name cleaner."+ .format(animal, clean_path, len(clean_names),+ raw_path, len(raw_names), clean_path))+ result[animal] = resolve_face_joints(clean_names, raw_names)++ output_path = args.output or os.path.join(input_dir, args.output_name)+ with open(output_path, "w") as f:+ json.dump(result, f, indent=2)++ sources: Dict[str, List[str]] = {}+ for animal, entry in result.items():+ sources.setdefault(entry["source"], []).append(animal)++ print("Written {} entries to {}\n".format(len(result), output_path))+ for src, animals in sorted(sources.items()):+ print(" [{}] {} entries".format(src, len(animals)))+ for a in animals:+ e = result[a]+ r_clean = e["r_hip"]["clean"] or "(none)"+ l_clean = e["l_hip"]["clean"] or "(none)"+ r_raw = e["r_hip"]["raw"] or "(none)"+ l_raw = e["l_hip"]["raw"] or "(none)"+ body = " [BODY_AXIS]" if e.get("body_axis") else ""+ print(" {:20s} r={:25s} ({}){}".format(a, r_clean, r_raw, body))+ print(" {:20s} l={:25s} ({})".format("", l_clean, l_raw))+++if __name__ == "__main__":+ main()diff --git a/data_process/joint_annotation/llm.py b/data_process/joint_annotation/llm.pynew file mode 100644index 0000000..ff9d034--- /dev/null+++ b/data_process/joint_annotation/llm.py@@ -0,0 +1,396 @@+"""Shared LLM scaffolding for the ``joint_annotation/*_llm.py`` entry points.++Every LLM-assisted joint-annotation script (joint-name cleaning, name+correction, face-joint selection) needs the same plumbing: backend detection+from the model name, a local HuggingFace causal-LM loader, one generate+function per backend (local / OpenAI-compatible), response-noise stripping, a+per-rig SIGALRM timeout, and simple JSON persistence. :class:`LLMClient`+bundles the backend state so task scripts only own their prompts, parsers+and per-rig orchestration.+"""++import json+import os+import re+import signal+import tempfile++from loguru import logger+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Backend detection+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++BACKEND_LOCAL = "local"+BACKEND_OPENAI = "openai"+VALID_BACKENDS = [BACKEND_LOCAL, BACKEND_OPENAI]++MAX_RETRIES = 4+# gpt-5-mini is the alternative once OPENAI credits are topped up; DeepSeek+# think-low reached quality parity (0 side errors, 99.6% format compliance,+# A/B 2026-08-30) at a fraction of the cost.+DEFAULT_MODEL = "deepseek-v4-flash"++# DeepSeek's API speaks the OpenAI protocol, so ``deepseek-*`` models ride the+# OpenAI backend with this base URL and DEEPSEEK_API_KEY (see generate_openai).+DEEPSEEK_BASE_URL = "https://api.deepseek.com"+++def detect_backend(model_name):+ # type: (str) -> str+ """Infer the backend from a model name (``gpt-*`` / ``deepseek-*`` /+ HF id)."""+ if model_name.startswith(("gpt-", "o1", "o3", "o4", "deepseek")):+ return BACKEND_OPENAI+ return BACKEND_LOCAL+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Local LLM loading & generation+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def load_local_model(model_name, device_map="auto", torch_dtype=None):+ """Load a text-only causal LM + tokenizer from HuggingFace."""+ import torch+ from transformers import AutoModelForCausalLM, AutoTokenizer++ dtype = torch.float16 if torch_dtype == "float16" else torch.bfloat16++ logger.info("Loading LLM: {} (dtype={}, device_map={})".format(+ model_name, dtype, device_map))+ tokenizer = AutoTokenizer.from_pretrained(model_name)+ model = AutoModelForCausalLM.from_pretrained(+ model_name,+ torch_dtype=dtype,+ device_map=device_map,+ )+ model.eval()+ return model, tokenizer+++def generate_local(model, tokenizer, system_prompt, user_prompt, max_tokens):+ """Run a single chat completion through a local causal LM.++ Qwen3 and other "thinking" models emit long ``<think>...</think>`` blocks+ by default, which easily exhausts the token budget before the answer is+ produced; ``enable_thinking=False`` is passed when the chat template+ supports it. Sampling follows the Qwen3 model-card recommendation for+ non-thinking mode (temperature=0.7, top_p=0.8, top_k=20) โ greedy+ decoding is documented to cause endless repetitions.+ """+ import torch++ messages = [+ {"role": "system", "content": system_prompt},+ {"role": "user", "content": user_prompt},+ ]+ try:+ text = tokenizer.apply_chat_template(+ messages, tokenize=False, add_generation_prompt=True,+ enable_thinking=False,+ )+ except TypeError:+ text = tokenizer.apply_chat_template(+ messages, tokenize=False, add_generation_prompt=True,+ )+ inputs = tokenizer(text, return_tensors="pt").to(model.device)++ with torch.no_grad():+ generated = model.generate(+ **inputs,+ max_new_tokens=max_tokens,+ do_sample=True,+ temperature=0.7,+ top_p=0.8,+ top_k=20,+ pad_token_id=tokenizer.eos_token_id,+ )+ trimmed = generated[0, inputs["input_ids"].shape[1]:]+ return tokenizer.decode(trimmed, skip_special_tokens=True).strip()+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# API backends+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def generate_openai(system_prompt, user_prompt, model_name, api_key, base_url,+ max_tokens, reasoning_effort=None):+ from openai import OpenAI++ is_deepseek = model_name.startswith("deepseek")+ if api_key is None:+ key_var = "DEEPSEEK_API_KEY" if is_deepseek else "OPENAI_API_KEY"+ api_key = os.environ.get(key_var)+ if api_key is None:+ # Fail loudly: with api_key=None the OpenAI client silently falls+ # back to OPENAI_API_KEY, which sends the wrong vendor's key to a+ # DeepSeek endpoint (and produces a confusing 401).+ raise RuntimeError(+ "{} is not set (and no --api_key given) for model {!r}.".format(+ key_var, model_name))+ if base_url is None and is_deepseek:+ base_url = DEEPSEEK_BASE_URL+ client = OpenAI(api_key=api_key, base_url=base_url)+ messages = [+ {"role": "system", "content": system_prompt},+ {"role": "user", "content": user_prompt},+ ]+ kwargs = dict(model=model_name, messages=messages)+ is_gpt5 = model_name.startswith("gpt-5") and "chat" not in model_name+ is_o_series = model_name.startswith(("o1", "o3", "o4"))+ if is_gpt5 or is_o_series:+ # Reasoning models reject max_tokens and non-default temperature;+ # max_completion_tokens counts reasoning + visible output together.+ kwargs["max_completion_tokens"] = max_tokens+ if is_gpt5:+ # The default effort burns hidden thinking tokens on every call+ # and the joint-annotation tasks are classification, not+ # multi-step reasoning โ force the lowest effort. The original+ # gpt-5 family calls it "minimal"; gpt-5.1+ dropped "minimal"+ # for "none" and errors on the old value. o-series models keep+ # their default effort ("minimal"/"none" don't exist there, and+ # o1-mini rejects the parameter entirely).+ versioned = re.match(r"gpt-5\.\d", model_name)+ kwargs["reasoning_effort"] = reasoning_effort or (+ "none" if versioned else "minimal")+ elif is_deepseek:+ # DeepSeek V4 thinks by default (effort 'high') and the reasoning+ # tokens count against max_tokens. Effort 'none' disables thinking โ+ # but V4-Flash WITHOUT thinking drifts on long batched arrays+ # (systematic label/side misalignment on ~10% of truebones rigs,+ # A/B'd 2026-08-30), so default to the cheapest thinking tier.+ # Thinking mode ignores temperature.+ kwargs["reasoning_effort"] = reasoning_effort or "low"+ if kwargs["reasoning_effort"] == "none":+ kwargs["max_tokens"] = max_tokens+ kwargs["temperature"] = 0.0+ else:+ # Reasoning shares the max_tokens budget; a small task budget+ # (512 for face selection) gets eaten by thought and returns+ # EMPTY content with finish_reason='length'. Billing is per+ # token used, so a generous floor costs nothing extra.+ kwargs["max_tokens"] = max(max_tokens, 8192)+ else:+ kwargs["max_tokens"] = max_tokens+ kwargs["temperature"] = 0.0+ try:+ response = client.chat.completions.create(**kwargs)+ except Exception as err:+ msg = str(err)+ if any(marker in msg for marker in (+ "insufficient_quota", "credit_balance_exhausted",+ "invalid_api_key", "Authentication Fails",+ "account_deactivated")):+ raise FatalLLMError(msg) from err+ raise+ choice = response.choices[0]+ content = (choice.message.content or "").strip()+ if not content and choice.finish_reason == "length":+ # Thinking consumed the whole budget โ retrying with identical+ # parameters cannot succeed; surface an actionable error instead.+ raise RuntimeError(+ "empty content with finish_reason='length': the reasoning "+ "budget is exhausted โ raise --max_tokens or lower "+ "--reasoning_effort (model {}).".format(model_name))+ return content+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Unified client+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++class LLMClient:+ """One text-in / text-out generate() over a local or OpenAI-compatible model.++ Args:+ model_name: OpenAI model (``gpt-5-mini``), DeepSeek model+ (``deepseek-v4-flash``) or HuggingFace model id for the local+ backend (``Qwen/Qwen3-8B``).+ backend: One of :data:`VALID_BACKENDS`; auto-detected when ``None``.+ api_key: API key for the OpenAI-compatible backends (defaults to+ the env vars).+ base_url: OpenAI-compatible base URL override.+ device_map, torch_dtype: Local-backend loading knobs.+ """++ def __init__(self, model_name, backend=None, api_key=None, base_url=None,+ device_map="auto", torch_dtype=None, reasoning_effort=None):+ self.model_name = model_name+ self.backend = backend or detect_backend(model_name)+ if self.backend not in VALID_BACKENDS:+ raise ValueError("Unknown backend {!r}; expected one of {}".format(+ self.backend, VALID_BACKENDS))+ self.api_key = api_key+ self.base_url = base_url+ self.reasoning_effort = reasoning_effort+ self._model = self._tokenizer = None+ if self.backend == BACKEND_LOCAL:+ self._model, self._tokenizer = load_local_model(+ model_name, device_map=device_map, torch_dtype=torch_dtype)++ @classmethod+ def from_args(cls, args):+ """Build a client from the namespace produced by :func:`add_llm_args`."""+ return cls(+ args.model, backend=args.backend,+ api_key=args.api_key, base_url=args.base_url,+ device_map=args.device_map, torch_dtype=args.torch_dtype,+ reasoning_effort=getattr(args, "reasoning_effort", None),+ )++ def generate(self, system_prompt, user_prompt, max_tokens):+ # type: (str, str, int) -> str+ if self.backend == BACKEND_LOCAL:+ return generate_local(self._model, self._tokenizer,+ system_prompt, user_prompt, max_tokens)+ return generate_openai(system_prompt, user_prompt,+ self.model_name, self.api_key, self.base_url,+ max_tokens, reasoning_effort=self.reasoning_effort)++ def __repr__(self):+ return "LLMClient(backend={!r}, model={!r})".format(self.backend, self.model_name)+++def add_llm_args(parser, default_model=DEFAULT_MODEL, default_max_tokens=2048):+ """Add the backend / model / retry flags shared by every ``*_llm.py`` CLI."""+ group = parser.add_argument_group("LLM backend")+ group.add_argument("--backend", type=str, default=None, choices=VALID_BACKENDS,+ help="LLM backend. Auto-detected from --model if unset.")+ group.add_argument("--model", type=str, default=default_model,+ help="OpenAI model (e.g. gpt-5-mini), DeepSeek model (e.g. "+ "deepseek-v4-flash), or HF model id for the local "+ "backend (e.g. Qwen/Qwen3-8B).")+ group.add_argument("--max_tokens", type=int, default=default_max_tokens,+ help="Max new tokens per LLM call.")+ group.add_argument("--max_retries", type=int, default=MAX_RETRIES,+ help="Max LLM retries per rig before falling back.")+ group.add_argument("--torch_dtype", type=str, default=None, choices=["float16", "bfloat16"],+ help="[local] Model dtype override (default: bfloat16).")+ group.add_argument("--device_map", type=str, default="auto",+ help="[local] Device placement strategy (default: auto).")+ group.add_argument("--api_key", type=str, default=None,+ help="[openai] Defaults to OPENAI_API_KEY / DEEPSEEK_API_KEY, "+ "matched to the model.")+ group.add_argument("--base_url", type=str, default=None,+ help="[openai] Override API base URL (deepseek-* models default "+ "to the DeepSeek endpoint; use with --backend openai for "+ "any other OpenAI-compatible server, e.g. a local vLLM).")+ group.add_argument("--reasoning_effort", type=str, default=None,+ help="[openai/deepseek] Reasoning effort override. Defaults: "+ "gpt-5 'minimal'/'none' (version-dependent); deepseek "+ "'low' ('none' disables thinking but degrades batched "+ "labeling accuracy).")+ return group+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Response hygiene+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++_FENCE_RE = re.compile(r"^\s*```(?:json)?\s*|\s*```\s*$", re.MULTILINE)+_THINK_RE = re.compile(r"<think>.*?(?:</think>|$)", re.DOTALL | re.IGNORECASE)+++def strip_llm_noise(text):+ # type: (str) -> str+ """Remove ``<think>`` blocks and markdown fences from an LLM response."""+ cleaned = _THINK_RE.sub("", text)+ return _FENCE_RE.sub("", cleaned).strip()+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-rig timeout (SIGALRM, Unix main thread only)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++class FatalLLMError(Exception):+ """Non-retryable backend failure (exhausted credits, bad API key).++ Retry loops must re-raise this instead of swallowing it: falling back to+ the rule-based cleaner on a quota outage would silently degrade EVERY+ rig of a batch run to rule-based output."""+++class RigTimeout(Exception):+ """Raised by the per-rig SIGALRM handler to abort an over-budget rig."""+++def _alarm_handler(signum, frame):+ raise RigTimeout()+++def install_rig_alarm(timeout_seconds):+ # type: (int) -> Optional[object]+ """Arm a SIGALRM for *timeout_seconds*; returns the previous handler, or+ ``None`` when no alarm was installed (disabled, unsupported, or not in+ the main thread)."""+ if not timeout_seconds or timeout_seconds <= 0:+ return None+ if not hasattr(signal, "SIGALRM"):+ return None+ try:+ prev = signal.signal(signal.SIGALRM, _alarm_handler)+ except ValueError:+ return None+ signal.alarm(int(timeout_seconds))+ return prev+++def uninstall_rig_alarm(prev_handler):+ if prev_handler is None:+ return+ signal.alarm(0)+ signal.signal(signal.SIGALRM, prev_handler)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# JSON persistence+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def load_json(path):+ # type: (str) -> Dict+ """Load a JSON dict if *path* exists, else return ``{}``."""+ if os.path.exists(path):+ with open(path, "r") as f:+ data = json.load(f)+ logger.info("Loaded {} existing entries from {}".format(len(data), path))+ return data+ return {}+++def save_json(data, path):+ # type: (Dict, str) -> None+ """Write *data* as JSON to *path*, atomically.++ The batch drivers re-save the whole file after every rig (3676 times for+ an Objaverse sweep), so a plain truncate-rewrite leaves a truncated,+ unparseable file if the process is killed mid-write โ and the two batch+ stages keep no ``.bak``. Write a temp file in the same directory and+ ``os.replace`` it into place instead, so readers only ever see the old+ or the new file.+ """+ directory = os.path.dirname(path) or "."+ fd, tmp_path = tempfile.mkstemp(+ dir=directory, prefix=os.path.basename(path) + ".", suffix=".tmp")+ try:+ with os.fdopen(fd, "w") as f:+ json.dump(data, f, indent=2)+ os.replace(tmp_path, path)+ except BaseException:+ if os.path.exists(tmp_path):+ os.remove(tmp_path)+ raise+++def read_id_list(path):+ # type: (str) -> List[str]+ """Read one id per line, skipping blanks and ``#`` comments."""+ with open(path, "r") as f:+ return [line.strip() for line in f if line.strip() and not line.startswith("#")]+++def append_id_list(path, ids):+ # type: (str, Iterable[str]) -> None+ with open(path, "a") as f:+ for rig_id in ids:+ f.write(rig_id + "\n")diff --git a/data_process/joint_annotation/names_clean_llm.py b/data_process/joint_annotation/names_clean_llm.pynew file mode 100644index 0000000..44ca613--- /dev/null+++ b/data_process/joint_annotation/names_clean_llm.py@@ -0,0 +1,474 @@+"""Clean joint names with a Language Model.++Complement to the rule-based :mod:`names_clean_rule`: sends raw joint names to an+LLM and asks it to map each one to a canonical anatomical label from+:mod:`vocab`. Pure text in / text out โ no images needed.++Backends (see :mod:`llm`): local HF causal LM, OpenAI API (default:+gpt-5-mini), DeepSeek API.++Input : ``joint_names.json`` โ ``{rig_id: [raw_joint_name, ...]}`` where+ ``rig_id`` keys may be humans, animals, props, vehicles or any other+ skeletal asset (Objaverse uses opaque UUIDs; Truebones uses animal+ names).+Output : ``clean_joint_names.json`` โ same shape, with cleaned names.++Each rig is processed as a single batched request: the full joint list is+sent together so the LLM can reason about the rig as a whole, and the+response must be a JSON array of the same length. Rigs that fail after+``--max_retries`` fall back to the rule-based cleaner and are recorded in+``failed_clean_names.txt`` next to the output.++Runs resume: a rig that already has a correct-length entry in the output is+skipped. Because the rule-based stage writes the SAME default file+(``<export_dir>/clean_joint_names.json``), that makes this stage a no-op after+it โ and it also skips the very rigs listed in ``failed_clean_names.txt``, since a+fallen-back rig has a full-length entry too. Use ``--overwrite`` to redo every+rig or ``--redo_failed`` to redo just the recorded failures.++Usage:+ python -m data_process.joint_annotation.names_clean_llm \\+ --input <export_dir>/joint_names.json # OpenAI gpt-5-mini+ python -m data_process.joint_annotation.names_clean_llm \\+ --input <export_dir>/joint_names.json --model deepseek-v4-flash+ python -m data_process.joint_annotation.names_clean_llm \\+ --input <export_dir>/joint_names.json --model Qwen/Qwen3-8B+"""++import argparse+import json+import os+import sys+import time++from loguru import logger+from tqdm import tqdm++_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++from data_process.joint_annotation.names_clean_rule import ( # noqa: E402+ clean_joint_name,+ post_process,+ report_coverage,+)+from data_process.joint_annotation.llm import ( # noqa: E402+ FatalLLMError,+ LLMClient,+ MAX_RETRIES,+ add_llm_args,+ append_id_list,+ load_json,+ read_id_list,+ save_json,+ strip_llm_noise,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# System prompt โ encodes the vocab.py canonical vocabulary+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++SYSTEM_PROMPT = (+ "Standardize 3D rig joint names to canonical anatomical labels. Inputs "+ "come from Mixamo, Maya, Blender, Unreal, Truebones and custom rigs. "+ "Focus on semantic meaning, not surface syntax.\n\n"+ "OUTPUT: a single JSON array of strings, same length and order as the "+ "input. No prose, no fences, no extra text. One input entry -> one "+ "output entry; never dedupe, merge, skip or reorder, even when "+ "neighbours produce identical labels.\n\n"+ "CLEAN each name by removing rig noise and extracting meaning:\n"+ "1. Digits and Blender '.NNN' counters are meaningless โ rig-internal "+ "bookkeeping (chain index, mirror id, duplicate counter). Ignore them "+ "when matching, and never include a digit, underscore, or dot in the "+ "final label. 'Spine', 'Spine1', 'Spine_02', 'Spine.003' all map to "+ "'Spine'. Finger-chain segments ('Thumb1/Thumb2/Thumb3') all map to "+ "'Thumb Finger'. The Objaverse export pipeline also stamps a trailing "+ "global index on EVERY name ('_NN' or '_0NN': '_01', '_010', '_063'); "+ "indices can STACK ('Hip_01_41', 'Head_1_016', 'Index.R.001_013_6', "+ "'Bone.001_01', 'Spine_1_013') โ strip ALL of them, in any position. Editor decorations are noise too: strip '(mirrored)' anywhere and a trailing '.x' center marker ('spine_01.x' -> 'Spine').\n"+ "2. Drop any leading '<Word>:' namespace (case-insensitive, trailing "+ "digits in the namespace OK): mixamorig:, Mixamorig1:, Mutant:, Sif:. "+ "The same words are noise without the colon too โ leading "+ "'mixamorig_'/'Mixamorig'/'Character'/'Rig'/'QuickRigCharacter_' "+ "segments are dropped, never echoed into the label. The same goes "+ "for embedded ASSET/CHARACTER names and their decorations โ "+ "'rp_karl_animated_006_warmingUp_spine_01' -> 'Spine', "+ "'CMan0205-M4-CS_Hips L Finger0' -> 'Left Thumb Finger': keep only "+ "the anatomical tokens, drop every name-like or counter token.\n"+ "3. Drop rig prefixes (match before ignoring digits so 'Bip01_' still "+ "strips cleanly): any Bip<digits> container with any separator (Bip01_, Bip002 , Bip01-, and separator-free Bip001LFinger0), BN_Bip01_, BN_, Bn_, NPC_, jt_, Elk, "+ "Sabrecat_, QuickRigCharacter_, Bind_, Skeleton_, Root_, DEF-, def_.\n"+ "4. Drop Maya suffixes: _jnt, _jt, _Jt, _JNT, _joint, _bone, _bn, _C.\n"+ "5. Extract side as explicit 'Left '/'Right ' prefix. Recognise:\n"+ " prefix L_ / R_, Lt_ / Rt_, Left_ / Right_, Left<UpperWord> / "+ "Right<UpperWord> (LeftHand, RightArm), L<UpperLetter> / R<UpperLetter> "+ "(LArm, RHand) โ but NOT when followed by a lowercase letter (Lower, "+ "Ribcage).\n"+ " 3ds Max Biped space-separated single-letter side: 'Bip001 L "+ "UpperArm' -> Left Upper Arm, 'Bip001 R Thigh' -> Right Thigh. Token "+ "boundaries are spaces, not underscores.\n"+ " suffix _L / _R / _l / _r, .L / .R, Japanese trailing L/R.\n"+ " PRECEDENCE: a trailing .L/.R/_L/_R token OVERRIDES a leading side word โ Blender's symmetrize renames only the suffix, leaving the prefix text stale: 'mixamorig:LeftShoulder.R' -> 'Right Shoulder', 'r_toe.L' -> 'Left Toe'.\n"+ " quadruped F_/B_ = Front/Back (e.g. 'F_R_Shoulder' -> "+ "'Right Front Shoulder', 'B_L_Foot' -> 'Left Back Foot').\n"+ "6. Common body-part roots โ translate case-insensitively, accepting "+ "Unreal snake_case AND CamelCase short-forms: pelvis/spine/neck/head/jaw/"+ "eye -> Pelvis/Spine/Neck/Head/Jaw/Eye; clavicle/collar -> Shoulder; "+ "upperarm / UpperArm / UpArm -> Upper Arm; lowerarm / LowerArm / LowArm / "+ "ForeArm -> Forearm; hand -> Hand; thigh / upleg / UpLeg -> Thigh; "+ "calf / lowleg / LowLeg / lowerleg -> Shin; upperleg -> Thigh; "+ "toes -> Toe; in a mixamo chain "+ "(UpLeg -> Leg -> Foot) the mid-bone 'Leg' is the shin -> Shin; foot -> "+ "Foot; toebase / Toe0 / toe -> Toe; ball -> Toe; eyelid -> Eyelid. "+ "Finger roots index/middle/ring/pinky/thumb -> '<Root> Finger'. "+ "*_twist -> '<Root> Twist'.\n"+ "6b. 3ds Max Biped fingers use numeric codes: Finger0*=Thumb, Finger1*="+ "Index, Finger2*=Middle, Finger3*=Ring, Finger4*=Pinky. Any trailing "+ "digits after that code are chain position โ ignore. 'Bip001 L Finger0' "+ "and 'Bip001 L Finger01' both -> 'Left Thumb Finger'; 'Bip001 R Finger21'"+ " -> 'Right Middle Finger'; 'Bip001 R Toe0' -> 'Right Toe'.\n"+ "6c. Mocap-segmented fingers are 1-BASED and carry a segment word: Finger1..Finger5 + Metacarpal/Proximal/Medial/Distal/Tip, with Finger1=Thumb ... Finger5=Pinky. 'LeftFinger1Metacarpal' -> 'Left Thumb Finger'; the Tip segment -> '<Root> Finger End'. Rule 6b's 0-based codes apply only to BARE Finger<digit> names with no segment word. Segment words after a NAMED finger ('IndexDistal', 'thumb_proximal_l', 'RingIntermediate') are likewise chain position โ drop them: 'IndexDistal' -> 'Index Finger'.\n"+ "7. Other direction words: Top->Upper, Low->Lower ('Topjaw'->'Upper Jaw'). "+ "'HeadTop_End' / '*_End' / '*Nub' -> '<Root> End'. Animal 'Hair*/Mane*' -> 'Mane'; humanoid accessories 'Ponytail*/Cape*/Cloth*/Skirt*' -> 'Appendage'.\n"+ "8. Placeholders -> 'Bone': Bone, joint, Xtra*, MagicEffectsNode, "+ "and any token with no clear anatomy (meshok, Capuche, ...). Do NOT "+ "fabricate body parts. EXCEPTION: a name that is ONLY digits, "+ "optionally with a leading underscore ('_00', '12'), is copied "+ "through UNCHANGED โ it marks a rig with unnamed bones.\n"+ "9. Japanese roots (Alligator/Pirrana/Tukan):\n"+ " body momo=Thigh, hiza=Knee, ashi=Foot, hiji=Elbow, te=Hand, "+ "kata=Shoulder, mune=Chest, hara=Abdomen, koshi/kosi=Hips, kubi=Neck, "+ "atama/kao=Head, ago=Jaw\n"+ " tail sippo/shippo=Tail, o=Tail\n"+ " fish munabire=Pectoral Fin, harabire=Pelvic Fin, sebire=Dorsal "+ "Fin, obire=Caudal Fin, shiribire=Anal Fin, era=Gill\n"+ " Trailing L/R on any of these -> Left/Right prefix.\n"+ "10. Output Title Case, single spaces. Prefer the CANONICAL vocabulary; "+ "if nothing fits, use 'Bone'.\n\n"+ "ALIASES & TYPOS (map to the canonical term): Spline=Spine, Scull=Skull, Nek=Neck, Tai=Tail, Tone/Thouge/Tunge=Tongue, Eyeleds=Eyelid, HorseLink=Fetlock, LargeCannon=Cannon, PhalanxPrima=Pastern, PhalangesManus=Phalanges, Foreleg=Front Leg, Hindleg=Hind Leg, Digit=Finger, Hair=Mane, Little=Pinky (finger), locator/Trajectory/Cog=Root, Clavicle/Collarbone=Shoulder. Species terms: insect 'Clip'/'Shall'=Mandible, 'Pliers'/'Piers'=Pincer, cricket 'Feeler'=Antenna but fish 'Feelers'=Barbel, bird 'ponitail'=Crest.\n\n"+ "CANONICAL VOCABULARY (use these exact terms verbatim):\n"+ " Core: Pelvis, Hips, Spine, Ribcage, Neck, Head, Skull, Skull Base, "+ "Head End, Body, Upper Body, Lower Body, Chest, Abdomen, Waist, Collar, "+ "Hip, Belly, Root, Center\n"+ " Arm : Shoulder, Scapula, Arm, Upper Arm, Forearm, Elbow, Wrist, Hand, Palm\n"+ " Leg : Thigh, Shin, Leg, Knee, Ankle, Foot, Heel, Toe, Paw, Hoof, "+ "Fetlock, Cannon, Metacarpus, Phalanges, Pastern\n"+ " Finger: Finger, Thumb Finger, Index Finger, Middle Finger, Ring Finger, "+ "Pinky Finger\n"+ " Head: Jaw, Upper Jaw, Lower Jaw, Tongue, Ear, Eye, Eyeball, Eyebrow, "+ "Eyelid, Mouth, Lip, Upper Lip, Lower Lip, Nose, Muzzle, Chin, Cheek\n"+ " Appendage: Tail, Wing, Feather, Antenna, Barbel, Tentacle, Claw, "+ "Hand Claw, Fang, Mandible, Large Mandible, Lower Mandible, Pincer, "+ "Stinger, Appendage\n"+ " Fins: Fin, Pectoral Fin, Pelvic Fin, Dorsal Fin, Caudal Fin, Anal Fin, Gill\n"+ " Coat/Equipment: Mane, Fur, Whisker, Shell, Dorsal Plate, Crest, Horn, Reins, Halter\n"+ " Quadruped: Front Leg, Middle Leg, Hind Leg, Front Paw, Back Paw, "+ "Front Hoof, Rear Hoof, Front Shoulder, Back Hip\n"+ " Physics: Twist, Upper Arm Twist, Forearm Twist, Thigh Twist, Shin Twist, "+ "Thigh Muscle, Neck Muscle, Tail Twist, Jiggle, Handle, IK Chain, Bone\n"+ "Side goes first ('Left Upper Arm', 'Right Pinky Finger'); quadruped "+ "markers sit between side and part ('Right Front Shoulder'). "+ "COMPOSED labels are also valid: '<part> End' for chain tips/Nub "+ "bones, 'Inner/Middle/Outer <part>' (raptor toes, claws, fingers), "+ "'Upper/Lower Eyelid', 'Upper/Lower Left/Right/Front Lip', "+ "'Wrist Back', 'Elbow Back', 'Front/Middle/Hind Leg End'.\n\n"+ "EXAMPLES (one per rig family):\n"+ " mixamorig:LeftUpLeg -> Left Thigh\n"+ " Mutant:RightHandThumb1 -> Right Thumb Finger\n"+ " Sif:calf_twist_01_r -> Right Shin Twist\n"+ " index_01_l -> Left Index Finger\n"+ " Bip01_L_Thigh -> Left Thigh\n"+ " BN_Bip01_R_Forearm_03 -> Right Forearm\n"+ " NPC_L_Finger02 -> Left Finger\n"+ " Elk_RearHoof_L -> Left Rear Hoof\n"+ " jt_FrontLeg1_R_C -> Right Front Leg\n"+ " Lt_Thumb1_jt -> Left Thumb Finger\n"+ " R_toeBase_jnt -> Right Toe\n"+ " Eye.R.001 -> Right Eye\n"+ " mixamorig:LeftShoulder.R -> Right Shoulder\n"+ " LeftHandIndex1 -> Left Index Finger\n"+ " mixamorig:RightLeg -> Right Shin\n"+ " LeftFinger2Distal -> Left Index Finger\n"+ " BN_Spline_03 -> Spine\n"+ " Bone.001 -> Bone\n"+ " F_R_Shoulder -> Right Front Shoulder\n"+ " B_L_Foot -> Left Back Foot\n"+ " Topjaw -> Upper Jaw\n"+ " momoR -> Right Thigh\n"+ " munabireL -> Left Pectoral Fin\n"+ " joint12 / Xtra01 / meshok -> Bone\n"+ "OBJAVERSE (trailing global '_NN', often stacked with a chain index):\n"+ " mixamorig:LeftUpLeg_056 -> Left Thigh\n"+ " mixamorig:LeftHandThumb1_012 -> Left Thumb Finger\n"+ " QuickRigCharacter_LeftForeArm_014 -> Left Forearm\n"+ " Bip001 L UpperArm_07 -> Left Upper Arm\n"+ " Bip001 L Finger01_011 -> Left Thumb Finger\n"+ " Bip001 R Finger21_036 -> Right Middle Finger\n"+ " Bip001 R Toe0_054 -> Right Toe\n"+ " UpArm.R_010_15 -> Right Upper Arm\n"+ " LowLeg.L_038_39 -> Left Shin\n"+ " Hip_01_41 -> Pelvis\n"+ " Index.R.001_013_6 -> Right Index Finger\n"+ " Rt_Eyelid_jt_08 -> Right Eyelid\n"+ " Skeleton_Root_02 -> Root\n"+ " joint1_2 / Bone.001_01 -> Bone\n\n"+ "BATCH EXAMPLE (notice identical outputs are preserved, not merged):\n"+ " input : [\"Spine\", \"Spine1\", \"Spine2\", \"Spine3\", \"Neck\", "+ "\"Tail_01\", \"Tail_02\", \"Tail_03\"]\n"+ " output : [\"Spine\", \"Spine\", \"Spine\", \"Spine\", \"Neck\", "+ "\"Tail\", \"Tail\", \"Tail\"]\n"+ " WRONG : [\"Spine\", \"Neck\", \"Tail\"] (8 inputs must yield 8 "+ "outputs โ deduping is a failure)\n"+)+++def build_user_prompt(rig_id, raw_names, correction=None):+ # type: (str, List[str], Optional[str]) -> str+ prompt = (+ "Rig identifier: {}\n"+ "Joint count: {}\n\n"+ "Raw joint names (JSON array, order matters):\n"+ "{}\n\n"+ "Return a JSON array of the SAME length with each raw name replaced by "+ "its cleaned canonical label."+ ).format(rig_id, len(raw_names), json.dumps(raw_names, ensure_ascii=False))+ if correction:+ prompt += "\n\nIMPORTANT: " + correction+ return prompt+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Response parsing+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_response(text, expected_len):+ # type: (str, int) -> List[str]+ """Extract a JSON string array of the expected length from an LLM response.++ Tolerates markdown fences, ``<think>โฆ</think>`` reasoning blocks (Qwen3+ and other thinking models) and trailing prose; raises ``ValueError`` on+ structural mismatch so the caller can retry or fall back.+ """+ if not text or not text.strip():+ raise ValueError("empty response")++ cleaned = strip_llm_noise(text)+ try:+ data = json.loads(cleaned)+ except json.JSONDecodeError:+ start = cleaned.find("[")+ end = cleaned.rfind("]")+ if start == -1 or end == -1 or end <= start:+ raise ValueError("no JSON array found in response")+ data = json.loads(cleaned[start:end + 1])++ if not isinstance(data, list):+ raise ValueError("response is not a JSON array")+ if len(data) != expected_len:+ raise ValueError("length mismatch: got {}, expected {}".format(+ len(data), expected_len))+ result = []+ for i, item in enumerate(data):+ if not isinstance(item, str) or not item.strip():+ raise ValueError("element {} is not a non-empty string".format(i))+ result.append(post_process(item.strip()))+ return result+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-rig cleaner (with retry + rule-based fallback)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def rule_based_clean(rig_id, raw_names):+ # type: (str, List[str]) -> List[str]+ """Rule-based fallback. ``rig_id`` drives animal-specific dispatch for+ Truebones (e.g. Spider, Pirrana); opaque Objaverse UUIDs fall through to+ the generic cleaner."""+ return [post_process(clean_joint_name(raw, rig_id)) for raw in raw_names]+++def clean_rig(rig_id, raw_names, client, system_prompt=SYSTEM_PROMPT,+ max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True):+ # type: (...) -> Tuple[List[str], bool]+ """Clean one rig's joint list via the LLM. Returns ``(names, used_fallback)``.++ Structural errors (length mismatch, bad JSON, ...) are fed back to the+ LLM on the next attempt so it can self-correct.+ """+ if not raw_names:+ return [], False++ last_err = None+ correction = None+ for attempt in range(1, max_retries + 1):+ try:+ raw_text = client.generate(+ system_prompt, build_user_prompt(rig_id, raw_names, correction=correction),+ max_tokens)+ return parse_response(raw_text, len(raw_names)), False+ except FatalLLMError:+ raise+ except Exception as err: # noqa: BLE001 โ retry on any backend/parse error+ last_err = err+ if isinstance(err, ValueError):+ correction = (+ "Your previous response failed validation ({}). Return a "+ "JSON array of EXACTLY {} strings โ one cleaned label per "+ "input joint, in the same order. Do not add, merge, split, "+ "or skip any entries. Output only the JSON array, no prose "+ "or code fences."+ ).format(err, len(raw_names))+ logger.warning("[{}] attempt {}/{} failed: {}".format(+ rig_id, attempt, max_retries, err))+ if attempt < max_retries:+ time.sleep(1.5)++ if not fallback_on_error:+ raise RuntimeError("LLM cleaning failed for {}: {}".format(rig_id, last_err))++ logger.error("[{}] all {} attempts failed ({}); falling back to rule-based "+ "cleaner".format(rig_id, max_retries, last_err))+ return rule_based_clean(rig_id, raw_names), True+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Batch driver+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def clean_all(input_path, output_path, client, system_prompt=SYSTEM_PROMPT,+ max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True,+ failed_path=None, overwrite=False, redo_failed=False):+ # type: (...) -> Dict[str, List[str]]+ """Clean every rig in ``input_path`` and write to ``output_path``.++ Resumes from a partial output file: rigs whose cleaned list is already+ present with the correct length are skipped. Rig ids that fall back to+ the rule-based cleaner are appended to ``failed_path`` (one per line) so+ the batch can be re-run later with a stronger prompt/model.++ That resume rule makes this stage a silent no-op when ``output_path``+ was already filled in by the rule-based cleaner (both wrappers default to+ ``<export_dir>/clean_joint_names.json``), and it also skips exactly the+ rigs recorded in ``failed_path`` โ a fallen-back rig already has a+ full-length list. ``overwrite`` re-processes every rig; ``redo_failed``+ re-processes only the ids in ``failed_path``.+ """+ with open(input_path, "r") as f:+ raw_data = json.load(f)++ cleaned = load_json(output_path)+ rig_ids = sorted(raw_data.keys())++ if failed_path is None:+ failed_path = os.path.join(os.path.dirname(output_path) or ".", "failed_clean_names.txt")+ already_failed = set(read_id_list(failed_path)) if os.path.exists(failed_path) else set()+ redo_ids = already_failed if redo_failed else set()+ if redo_failed:+ logger.info("--redo_failed: re-processing {} rig ids listed in {}".format(+ len(redo_ids), failed_path))++ fallback_count = 0+ processed = skipped = 0+ pbar = tqdm(rig_ids, desc="Cleaning", unit="rig", dynamic_ncols=True)+ for rig_id in pbar:+ raw_names = raw_data[rig_id]+ pbar.set_postfix_str(rig_id, refresh=False)++ existing = cleaned.get(rig_id)+ if (not overwrite and rig_id not in redo_ids+ and isinstance(existing, list) and len(existing) == len(raw_names)):+ skipped += 1+ continue++ processed += 1+ names_out, used_fallback = clean_rig(+ rig_id, raw_names, client,+ system_prompt=system_prompt, max_tokens=max_tokens,+ max_retries=max_retries, fallback_on_error=fallback_on_error,+ )+ fallback_count += int(used_fallback)+ cleaned[rig_id] = names_out+ save_json(cleaned, output_path)++ if used_fallback and rig_id not in already_failed:+ append_id_list(failed_path, [rig_id])+ already_failed.add(rig_id)++ logger.info("Done. {}/{} rigs cleaned by the LLM ({} via rule-based "+ "fallback), {} skipped (already complete); {} now holds {} "+ "rigs.".format(processed, len(rig_ids), fallback_count,+ skipped, output_path, len(cleaned)))+ if processed == 0 and skipped:+ logger.warning(+ "Nothing was re-processed: every rig in {} already had a "+ "correct-length entry in {} (the rule-based stage writes the same "+ "file). Pass --overwrite to redo them all, or --redo_failed to "+ "redo just the ids in {}.".format(input_path, output_path, failed_path))+ if fallback_count:+ logger.info("Failed rig ids recorded at {}".format(failed_path))+ return cleaned+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# CLI+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def main():+ parser = argparse.ArgumentParser(+ description="Clean joint names via an LLM (local / openai / deepseek).")+ parser.add_argument("--input", required=True,+ help="Input joint_names.json path.")+ parser.add_argument("--output", default=None,+ help="Output JSON path (default: sibling 'clean_joint_names.json').")+ parser.add_argument("--report", action="store_true",+ help="Print a residual-token coverage report after cleaning.")+ parser.add_argument("--no_fallback", action="store_true",+ help="Raise on LLM failure instead of falling back to the "+ "rule-based cleaner.")+ parser.add_argument("--failed_log", default=None,+ help="Path to append rig ids that hit the rule-based fallback "+ "(default: sibling 'failed_clean_names.txt').")+ parser.add_argument("--overwrite", action="store_true",+ help="Re-process every rig, even one that already has a "+ "correct-length entry in the output. Without this the "+ "stage is a no-op when the rule-based cleaner already "+ "wrote the same file.")+ parser.add_argument("--redo_failed", action="store_true",+ help="Also re-process the rig ids listed in --failed_log "+ "(they always have a full-length entry from the "+ "rule-based fallback, so they are otherwise skipped).")+ add_llm_args(parser, default_max_tokens=2048)+ args = parser.parse_args()++ output_path = args.output or os.path.join(+ os.path.dirname(args.input), "clean_joint_names.json")+ client = LLMClient.from_args(args)+ logger.info("{} | Input: {} | Output: {}".format(client, args.input, output_path))++ clean_data = clean_all(+ input_path=args.input,+ output_path=output_path,+ client=client,+ max_tokens=args.max_tokens,+ max_retries=args.max_retries,+ fallback_on_error=not args.no_fallback,+ failed_path=args.failed_log,+ overwrite=args.overwrite,+ redo_failed=args.redo_failed,+ )++ with open(args.input, "r") as f:+ raw_data = json.load(f)+ for rig_id in list(clean_data.keys())[:3]:+ print("\n--- {} ---".format(rig_id))+ for raw, out in zip(raw_data.get(rig_id, []), clean_data[rig_id]):+ print(" {:50s} -> {}".format(raw, out))++ if args.report:+ report_coverage(clean_data)+++if __name__ == "__main__":+ main()diff --git a/data_process/joint_annotation/names_clean_rule.py b/data_process/joint_annotation/names_clean_rule.pynew file mode 100644index 0000000..9c00973--- /dev/null+++ b/data_process/joint_annotation/names_clean_rule.py@@ -0,0 +1,601 @@+"""Rule-based joint-name cleaning: ``joint_names.json`` โ ``clean_joint_names.json``.++Applies prefix stripping, CamelCase splitting, Japanese translation and+canonical anatomical vocabulary mapping. The vocabulary lives in+:mod:`vocab`; :mod:`names_clean_llm` is the LLM-based complement and uses+this module as its fallback.++Usage:+ python -m data_process.joint_annotation.names_clean_rule \+ --input <export_dir>/joint_names.json [--output ...] [--report]+"""++import argparse+import json+import os+import re+import sys++_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++from data_process.joint_annotation.vocab import (+ CANONICAL,+ ELK_MAP,+ JAPANESE_COMPOUND_ANIMALS,+ JAPANESE_COMPOUND_NAMES,+ JAPANESE_WORDS_LOWER,+ JT_MAP,+ MIXAMO_MAP,+ NPC_DIRECT,+ PIRRANA_COMPOUNDS,+ REMOVE_PREFIXES,+ SABRECAT_MAP,+ SPIDER_MAP,+ STANDALONE_MAP,+)+++# ---------------------------------------------------------------------------+# Common helpers+# ---------------------------------------------------------------------------++# Explicit ``_L_`` / ``_R_`` / ``.L`` / ``.R`` side token: an isolated L or R+# delimited by an underscore/dot on the left and by a delimiter, a digit or+# the end of the name on the right. This is the strongest side marker a raw+# bone name carries, so it outranks a bare single-letter prefix.+_SIDE_TOKEN_RE = re.compile(r'[._]([LR])(?=[._]|\d|$)')++# A leading single L/R only marks a side when what follows reads as a word+# ("LThigh", "LEar", "LWing01") or as a short side abbreviation followed by a+# word ("LB_foot", "LT_BackFoot_09", "LFT_ankle_09"). All-caps tokens+# ("ROOT_01", "RIG_01", "LOD", "LEG", "RUG_01") and rig namespaces+# ("LP1:Head") must NOT match โ there the L/R is part of the word itself.+_SINGLE_SIDE_RE = re.compile(r'^[LR](?:[A-Z][a-z]|[A-Z]{0,2}_[A-Za-z])')++# Trailing Blender/Rigify decorations: '.NNN' duplicate counters, '.NN' chain+# indices, Objaverse '_NN' global indices, '.x' center markers and '.L'/'.R'+# ('.l'/'.r') side tokens stack in any order at the end of a name+# ('thumb.01.R_037', 'Bone.005.R', 'spine_01.x', 'Index.R.001_013_6'). The+# separator requirement keeps lookup keys like 'Spine2' / 'Tail01' intact.+_TRAILING_TOKEN_RE = re.compile(r'[._](\d+|[LRlr]|x)$')++# 3ds Max Biped numeric finger codes (0-based): 'Finger0'/'Finger01' = thumb+# chain; trailing digits after the code are chain position.+_FINGER_CODE_RE = re.compile(r'^[Ff]inger([0-4])\d*(Nub)?$')+_FINGER_CODE = {"0": "Thumb", "1": "Index", "2": "Middle", "3": "Ring",+ "4": "Pinky"}+# CMU / mocap-style segmented fingers (1-based): 'Finger1Metacarpal' etc.+_FINGER_SEG_RE = re.compile(+ r'^[Ff]inger([1-5])(Metacarpal|Proximal|Medial|Distal|Tip)\d*$')+_FINGER_ORD = {"1": "Thumb", "2": "Index", "3": "Middle", "4": "Ring",+ "5": "Pinky"}++# 'Bone001(mirrored)' and similar editor bookkeeping โ anywhere in the name,+# since the Objaverse global index can trail it: 'Bone018(mirrored)_079'.+_PAREN_DECOR_RE = re.compile(r'\s*\([^)]*\)')++# Any leading '<word>:' rig namespace ('RenWu2:', 'candi_aaaaaa2:'). Known+# ones (mixamorig:, Mutant:, Sif:) are handled earlier with their own maps.+_NAMESPACE_RE = re.compile(r'^[A-Za-z][\w .-]*:')++# Generic 3ds Max Biped container prefix: Bip01_/Bip002 /Bip01-/Bip001L...+# (any index; separator optional when a side letter follows), optionally+# behind a BN_ namespace.+_BIP_PREFIX_RE = re.compile(r'^(?:BN_)?Bip\d+(?:[-_ ]+|(?=[LR][A-Z]))')+++def strip_trailing_decorations(name):+ # type: (str) -> Tuple[str, str]+ """Peel stacked trailing ``[._]``-separated tokens off *name*.++ Digit groups and ``x`` center markers are dropped; the innermost ``L``/+ ``R`` token wins as the side. Returns ``(side, base)`` with ``side`` one+ of ``""`` / ``"Left"`` / ``"Right"``.+ """+ side = ""+ while True:+ m = _TRAILING_TOKEN_RE.search(name)+ if not m:+ return side, name+ tok = m.group(1)+ if tok in ("L", "l"):+ side = "Left"+ elif tok in ("R", "r"):+ side = "Right"+ name = name[:m.start()]+++def _canon(token):+ # type: (str) -> Optional[str]+ """CANONICAL lookup, tolerating all-lower/all-upper raw tokens+ ('thumb' -> 'Thumb Finger'). Exact match wins so CamelCase keys like+ 'UpLeg' are never shadowed by their capitalize() form."""+ value = CANONICAL.get(token)+ if value is None and token and token != token.capitalize():+ value = CANONICAL.get(token.capitalize())+ return value+++def extract_side_prefix(name):+ # type: (str) -> Tuple[str, str]+ """Extract Left/Right side from common prefix/suffix patterns.++ Handles: R_/L_ prefix, Left_/Right_ prefix, single R/L followed by+ uppercase, _L_NN/_R_NN suffix, _L/_R suffix. Guards against false+ positives on words that begin with R or L followed by a lowercase letter+ (e.g. "Ribcage", "Lower..."), on all-caps tokens that merely start with+ one ("ROOT_01", "RIG_01", "LEG") and on rig namespaces ("LP1:Head").++ Markers are ranked: an explicit ``_L_``/``_R_``/``.L``/``.R`` token beats+ a bare single-letter prefix, because rigs that carry both use the token as+ the real side ("LLeg.R_08" is the RIGHT leg, "LB_R_Shoulder_DEF" the RIGHT+ shoulder). A wrong side here silently rotates the clip 180 degrees, since+ the side flows into face_select -> get_root_facing_quat.++ Returns:+ (side, remaining_name) where side is "" | "Left" | "Right".+ """+ # Trailing Blender/Rigify decorations first: '.L'/'.R' side tokens mixed+ # with '.NNN'/'_NN' counters ('thumb.01.R_037' -> Right 'thumb'). When no+ # side is found the stripped base still flows into the prefix checks.+ side, name = strip_trailing_decorations(name)+ if side:+ # A symmetrized bone keeps its pre-mirror text: Blender renames only+ # the suffix, so 'mixamorig:LeftShoulder.R' / 'r_toe.L' carry a STALE+ # leading side word. The trailing token is the real side โ drop the+ # leftover prefix so it can't pollute the vocabulary lookup.+ stripped = re.sub(+ r'^(?:Left|Right)(?=[A-Z])|^(?:Left|Right)[_ ]|^[LR][-_]|^[lr]_',+ '', name)+ return side, stripped or name++ # R_X / L_X / R-X / L-X / R.X / L.X prefix+ if re.match(r'^[RL][-_.]', name):+ side = "Left" if name[0] == "L" else "Right"+ return side, name[2:]++ # "R X" / "L X" prefix with space separator (e.g. "Bip01 L UpperArm"+ # becomes "L UpperArm" after prefix strip, then "UpperArm").+ m = re.match(r'^([RL]) +', name)+ if m:+ side = "Left" if m.group(1) == "L" else "Right"+ return side, name[m.end():]++ # Left_X or Right_X prefix (case-insensitive)+ m = re.match(r'^(Left|Right)_', name, re.IGNORECASE)+ if m:+ return m.group(1).capitalize(), name[len(m.group(0)):]++ # "Left" / "Right" directly followed by uppercase (Mixamo: LeftHandThumb1).+ m = re.match(r'^(Left|Right)(?=[A-Z])', name)+ if m:+ return m.group(1), name[len(m.group(1)):]++ # Single L/R followed by uppercase (e.g., LThigh, RArm). The weakest of+ # the side markers, so it is deliberately narrow:+ # * an explicit _L_/_R_/.L/.R token later in the name is the real+ # marker and wins ("LLeg.R_08" is the RIGHT leg, "LB_R_Shoulder_DEF"+ # the RIGHT shoulder);+ # * all-caps tokens ("ROOT_01", "RIG_01", "LEG") and rig namespaces+ # ("LP1:Head") carry no side at all.+ if len(name) > 1 and name[0] in "LR" and name[1].isupper():+ m = _SIDE_TOKEN_RE.search(name, 1)+ if m:+ side = "Left" if m.group(1) == "L" else "Right"+ rest = (name[:m.start()] + name[m.end():]).strip("._")+ return side, rest or name+ if _SINGLE_SIDE_RE.match(name):+ side = "Left" if name[0] == "L" else "Right"+ return side, name[1:]++ # _L_NN or _R_NN suffix+ m = re.search(r'_([LR])_(\d+)$', name)+ if m:+ side = "Left" if m.group(1) == "L" else "Right"+ return side, name[:m.start()] + "_" + m.group(2)++ # _L or _R suffix+ m = re.search(r'_([LR])$', name)+ if m:+ side = "Left" if m.group(1) == "L" else "Right"+ return side, name[:m.start()]++ return "", name+++def split_and_map_tokens(name):+ # type: (str) -> str+ """Split a CamelCase/underscore name and map tokens via CANONICAL.++ Tokens mapping to ``""`` (rig noise like 'Joint'/'Jnt') are dropped.+ """+ tokens = re.split(r'(?=[A-Z])|_', name)+ mapped = []+ for t in tokens:+ if not t:+ continue+ base = re.sub(r'\d+$', '', t)+ value = _canon(base)+ if value is not None:+ if value:+ mapped.append(value)+ elif base and len(base) > 1:+ mapped.append(base)+ return " ".join(mapped)+++def with_side(side, result):+ # type: (str, str) -> str+ """Prepend side to result, replacing any side the mapping already carries.++ A side extracted from the raw name (``R_``/``L_``, ``_L_``/``_R_``, ...)+ is an explicit marker and outranks one baked into a vocabulary entry, so+ ``with_side("Right", "Left Wing")`` yields ``"Right Wing"`` โ otherwise+ ``R_LWing`` would clean to ``Left Wing`` via ``CANONICAL["LWing"]``.+ """+ if not side:+ return result+ m = re.match(r'^(Left|Right)\b\s*', result)+ if m:+ result = result[m.end():]+ return (side + " " + result).strip()+++# ---------------------------------------------------------------------------+# Skeleton-specific cleaners+# ---------------------------------------------------------------------------++def clean_japanese_name(name):+ """Handle fully Japanese joint names (Alligator, Pirrana, Tukan style)."""+ side, name = extract_side_prefix(name)++ base = re.sub(r'\d+$', '', name)+ lower = base.lower()+ if lower in JAPANESE_WORDS_LOWER:+ return with_side(side, JAPANESE_WORDS_LOWER[lower])++ # Pirrana-specific compound words (side already encoded in the value)+ if name in PIRRANA_COMPOUNDS:+ return PIRRANA_COMPOUNDS[name]++ return with_side(side, name)+++def clean_prefixed_name(raw, prefix, rig_map, strip_trailing_c=False, skip_substrs=()):+ # type: (str, str, dict, bool, tuple) -> str+ """Generic cleaner for rigs with a fixed prefix and a lookup table.++ Strips ``prefix``, optionally drops a trailing ``_C`` (jt center bones),+ extracts Left/Right side, and maps via ``rig_map`` โ exact match first,+ then digit-stripped base. Falls back to CANONICAL via+ :func:`split_and_map_tokens`. Names containing any of ``skip_substrs``+ (e.g. ``"Jiggle"`` for NPC physics bones) are dropped.+ """+ name = raw[len(prefix):]+ if strip_trailing_c and name.endswith("_C"):+ name = name[:-2]+ side, name = extract_side_prefix(name)++ for sub in skip_substrs:+ if sub in name:+ return "" # post_process converts empty -> "Bone"++ result = rig_map.get(name)+ if result is None:+ result = rig_map.get(re.sub(r'\d+$', '', name))+ if result is None:+ result = split_and_map_tokens(name) or name++ return with_side(side, result).strip()+++def clean_spider_name(raw):+ """Special handling for Spider skeleton names."""+ if raw in SPIDER_MAP:+ return SPIDER_MAP[raw]++ m = re.match(r'Fang([RL])_(\d+)_', raw)+ if m:+ side = "Right" if m.group(1) == "R" else "Left"+ return "{} Fang".format(side)++ m = re.match(r'Leg_([RL])_(\d)(\d)_', raw)+ if m:+ side = "Right" if m.group(1) == "R" else "Left"+ return "{} Leg".format(side)++ m = re.match(r'_([RL])Toe(\d)_', raw)+ if m:+ side = "Right" if m.group(1) == "R" else "Left"+ return "{} Leg Tip".format(side)++ return raw.strip("_")+++def clean_standard_name(raw):+ """Clean standard Bip01/BN prefixed names."""+ name = _NAMESPACE_RE.sub("", raw) or raw+ name = _BIP_PREFIX_RE.sub("", name)++ for prefix in REMOVE_PREFIXES:+ if name.startswith(prefix):+ name = name[len(prefix):]+ break++ name = name.lstrip("_").strip()+ if not name:+ return raw.strip("_")++ side, name = extract_side_prefix(name)++ # Numeric finger codes, before the chain-index strip eats the code digit:+ # CMU segments ('Finger1Metacarpal', 1-based) and 3ds Max Biped codes+ # ('Finger0'/'Finger01'/'Finger21', 0-based).+ m = _FINGER_SEG_RE.match(name)+ if m:+ label = _FINGER_ORD[m.group(1)] + " Finger"+ if m.group(2) == "Tip":+ label += " End"+ return with_side(side, label)+ m = _FINGER_CODE_RE.match(name)+ if m:+ label = _FINGER_CODE[m.group(1)] + " Finger"+ if m.group(2):+ label += " End"+ return with_side(side, label)++ # Strip a trailing chain index: "Thigh_4" -> "Thigh", "Spine02" -> "Spine".+ m = re.match(r'^(.+?)_?(\d+)$', name)+ base = m.group(1).rstrip("_") if m else name++ mapped_base = _canon(base)+ if mapped_base is not None:+ # Placeholders and skip-tokens carry no side ('Bone.005.R' -> 'Bone').+ if not mapped_base or mapped_base == "Bone":+ return "Bone"+ return with_side(side, mapped_base)++ # Handle compound names with underscores+ parts = [p for p in base.split("_") if p]+ mapped = []+ for p in parts:+ # A bare L/R token only fills in a side that extract_side_prefix did+ # not already find; it must NOT override one, because the side it+ # returned came from a stronger marker (an explicit _L_/_R_/.L/.R+ # token). 'r_index_00_L_049' is the LEFT rear-paw index finger โ the+ # leading 'r' means "rear", not "right".+ if p in ("L", "l"):+ side = side or "Left"+ continue+ if p in ("R", "r"):+ side = side or "Right"+ continue++ sub_base = re.sub(r'\d+$', '', p)+ sub_mapped = _canon(sub_base)+ if sub_mapped is not None:+ if sub_mapped:+ mapped.append(sub_mapped)+ elif sub_base:+ sub_tokens = [t for t in re.split(r'(?=[A-Z])', sub_base) if t]+ for st in sub_tokens:+ st_mapped = _canon(st)+ if st_mapped is not None:+ if st_mapped:+ mapped.append(st_mapped)+ elif st and len(st) > 1:+ mapped.append(st)++ result = " ".join(mapped)+ if not result:+ # Nothing anatomical survived (pure counters like '1.L', skip tokens+ # like 'internal50.L') -> placeholder, without a side.+ return "Bone"+ if result == "Bone":+ return "Bone"+ return with_side(side, result).strip() or raw+++# ---------------------------------------------------------------------------+# Main dispatcher+# ---------------------------------------------------------------------------++# Matches any mixamorig-style namespace: mixamorig:, mixamorig1:, mixamorig7:,+# Mixamorig1:, MIXAMORIG:, ... The trailing digits and case both vary across+# rigs but the vocabulary beneath is the same.+_MIXAMORIG_RE = re.compile(r'^mixamorig\d*[:_]', re.IGNORECASE)+++def clean_joint_name(raw, animal):+ """Main dispatcher for cleaning a single joint name."""+ if not raw or not raw.strip():+ return raw++ # Editor bookkeeping suffixes: 'Bone001(mirrored)' -> 'Bone001'+ raw = _PAREN_DECOR_RE.sub("", raw).strip() or raw++ # Generic "BoneNN" placeholders -> "Bone" (dot forms via clean_standard_name)+ if re.match(r'^Bone\d+$', raw):+ return "Bone"++ # Numeric only names โ return unchanged+ if re.match(r'^_?\d+$', raw.strip("_")):+ return raw++ m = _MIXAMORIG_RE.match(raw)+ if m:+ return clean_prefixed_name(raw, m.group(0), MIXAMO_MAP)++ if raw in STANDALONE_MAP:+ return STANDALONE_MAP[raw]++ if raw in SABRECAT_MAP:+ return SABRECAT_MAP[raw]++ if animal == "Spider":+ return clean_spider_name(raw)++ if raw.startswith("Sabrecat"):+ return SABRECAT_MAP.get(raw, raw)++ if raw.startswith("NPC_"):+ return clean_prefixed_name(raw, "NPC_", NPC_DIRECT, skip_substrs=("Jiggle",))++ if raw.startswith("Elk"):+ return clean_prefixed_name(raw, "Elk", ELK_MAP)++ if raw.startswith("jt_"):+ return clean_prefixed_name(raw, "jt_", JT_MAP, strip_trailing_c=True)++ # Japanese names: strip side + trailing digits, check vocabulary+ base_lower = re.sub(r'^[RL]_', '', raw).lower()+ base_lower = re.sub(r'\d+$', '', base_lower)+ if base_lower in JAPANESE_WORDS_LOWER:+ return clean_japanese_name(raw)++ if animal in JAPANESE_COMPOUND_ANIMALS and raw in JAPANESE_COMPOUND_NAMES:+ return clean_japanese_name(raw)++ return clean_standard_name(raw)+++# ---------------------------------------------------------------------------+# Post-processing+# ---------------------------------------------------------------------------++def post_process(name):+ """Final cleanup pass."""+ # rstrip only: a leading '_' marks the untouched numeric passthrough+ # names ('_00') and must survive.+ name = re.sub(r'\s+', ' ', name).strip().rstrip('._ ')++ # Strip trailing chain indices: "Tail 3" -> "Tail"+ name = re.sub(r'\s+\d+$', '', name)++ # Strip numbers before End: "Toe0 End" -> "Toe End"+ name = re.sub(r'(\w)\d+\s+End', r'\1 End', name)++ # Canonical-format rewrites for compositional leftovers. The finger+ # chain's 'Hand' segment is context, not a label word; UE/mixamo split+ # forms collapse to their canonical terms.+ name = re.sub(r'\bHand (Thumb|Index|Middle|Ring|Pinky) Finger\b', r'\1 Finger', name)+ name = re.sub(r'\b(Thumb|Index|Middle|Ring|Pinky) Finger Finger\b', r'\1 Finger', name)+ name = re.sub(r'\bFinger (Thumb|Index|Middle|Ring|Pinky) Finger\b', r'\1 Finger', name)+ name = re.sub(r'\b(?:Up|Upper) Leg$', 'Thigh', name)+ name = re.sub(r'\bLower Leg$', 'Shin', name)+ name = re.sub(r'\b(?:Lower|Fore) Arm$', 'Forearm', name)+ name = re.sub(r'\bToe Base$', 'Toe', name)+ # Trailing side word -> canonical leading position ('Index Finger Left')+ name = re.sub(r'^(?!Left |Right )(.+) (Left|Right)$', r'\2 \1', name)+ name = re.sub(r'^Head Top End$', 'Head End', name)+ name = name.replace('. ', ' ')++ # Dog-specific patches for misrouted head/spine subparts (with or+ # without the chain digit, which earlier passes may already strip)+ name = re.sub(r'^Spine ?\d* ?Tail$', 'Tail', name)+ name = re.sub(r'^Head ?\d* ?Jaw$', 'Jaw', name)+ name = re.sub(r'^Head ?\d* ?Eyelid$', 'Eyelid', name)+ name = re.sub(r'^Head Muzzle$', 'Muzzle', name)+ name = re.sub(r'^Head Jaw End$', 'Jaw End', name)+ name = re.sub(r'^Head Brain$', 'Head', name)++ # Clean leftover Bip01 fragments+ name = re.sub(r'Bip \d+ ', '', name)++ # "Spine Spine" -> "Spine"+ name = re.sub(r'^(\w+) \1$', r'\1', name)++ if re.match(r'^Xtra', name):+ name = "Bone"++ name = re.sub(r'Ponytail\d*.*', 'Appendage', name)++ # Title case consistency: "Right leg" -> "Right Leg"+ words = name.split()+ name = " ".join(w.capitalize() if w[0].islower() and len(w) > 1 else w for w in words)++ name = re.sub(r'^Spine (Left|Right) Wing$', r'\1 Wing', name)+ name = re.sub(r'\s+\d+$', '', name)++ if not name:+ return "Bone"+ return name+++# ---------------------------------------------------------------------------+# Coverage reporting+# ---------------------------------------------------------------------------++_UNMAPPED_PATTERN = re.compile(r'[_\d]|[a-z]{2,}[A-Z]')+++def _looks_unmapped(cleaned):+ """Heuristic: residual underscores, digits, or CamelCase runs suggest+ the name fell through the rules without hitting a canonical mapping."""+ return bool(_UNMAPPED_PATTERN.search(cleaned))+++def report_coverage(clean_data):+ """Per-rig conformance to the shared canonical label format+ (:func:`vocab.is_canonical_label`); 'Bone' counts as conforming."""+ from data_process.joint_annotation.vocab import is_canonical_label+ total = 0+ unmapped = 0+ print("\n=== Canonical-format report ===")+ for animal, labels in clean_data.items():+ bad = [l for l in labels+ if l != "Bone" and not is_canonical_label(l)]+ total += len(labels)+ unmapped += len(bad)+ if bad:+ preview = ", ".join(sorted(set(bad))[:5])+ print(" {:20s} {:3d}/{:3d} residual: {}".format(+ animal, len(bad), len(labels), preview))+ pct = 100.0 * (1 - unmapped / max(total, 1))+ print("Overall: {}/{} labels clean ({:.1f}%)".format(+ total - unmapped, total, pct))+++# ---------------------------------------------------------------------------+# CLI+# ---------------------------------------------------------------------------++def main():+ parser = argparse.ArgumentParser(description="Rule-based joint-name cleaning.")+ parser.add_argument("--input", required=True,+ help="Path to the raw joint_names.json")+ parser.add_argument("--output", default=None,+ help="Output path (default: sibling 'clean_joint_names.json')")+ parser.add_argument("--report", action="store_true",+ help="Print a per-rig coverage report")+ args = parser.parse_args()++ input_path = args.input+ output_path = args.output or os.path.join(+ os.path.dirname(input_path), "clean_joint_names.json")++ with open(input_path, "r") as f:+ data = json.load(f)++ clean_data = {}+ for animal, joints in data.items():+ clean_data[animal] = [post_process(clean_joint_name(j, animal)) for j in joints]++ with open(output_path, "w") as f:+ json.dump(clean_data, f, indent=2)++ print("Written {} entries to {}".format(len(clean_data), output_path))++ for animal in list(data.keys())[:5]:+ print("\n--- {} ---".format(animal))+ for raw, clean in zip(data[animal], clean_data[animal]):+ print(" {:50s} -> {}".format(raw, clean))++ if args.report:+ report_coverage(clean_data)+++if __name__ == "__main__":+ main()diff --git a/data_process/joint_annotation/names_correct_llm.py b/data_process/joint_annotation/names_correct_llm.pynew file mode 100644index 0000000..578db7b--- /dev/null+++ b/data_process/joint_annotation/names_correct_llm.py@@ -0,0 +1,379 @@+"""LLM check-and-correct pass over ``clean_joint_names.json``.++Feeds the LLM BOTH the raw joint name and the current cleaned label for+every joint, and asks it to either keep the current label or replace it with+a corrected canonical one. Useful for:++ - correcting rigs that fell back to the rule-based cleaner (digits,+ missing side prefixes, ``'Bone'`` for tokens the rules don't recognise),+ - sweeping the whole dataset for residual vocabulary / rule violations.++Inputs:+ --raw raw ``joint_names.json``.+ --cleaned current ``clean_joint_names.json``.+ --failed_list OPTIONAL text file with one rig id per line (e.g. the+ ``failed_clean_names.txt`` written by ``names_clean_llm``). If+ omitted, EVERY rig in ``--raw`` is checked and corrected.++Output: the script WRITES BACK IN-PLACE to ``--cleaned``. A ``.bak`` sibling+is created on first run so the original is always recoverable.++Usage:+ python -m data_process.joint_annotation.names_correct_llm \\+ --raw <export_dir>/joint_names.json \\+ --cleaned <export_dir>/clean_joint_names.json [--failed_list failed_clean_names.txt]+"""++import argparse+import json+import os+import shutil+import sys+import time++from loguru import logger+from tqdm import tqdm++_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))+if _REPO_ROOT not in sys.path:+ sys.path.insert(0, _REPO_ROOT)++from data_process.joint_annotation.names_clean_llm import ( # noqa: E402+ SYSTEM_PROMPT,+ parse_response,+ rule_based_clean,+)+from data_process.joint_annotation.llm import ( # noqa: E402+ FatalLLMError,+ LLMClient,+ MAX_RETRIES,+ RigTimeout,+ add_llm_args,+ append_id_list,+ install_rig_alarm,+ read_id_list,+ save_json,+ uninstall_rig_alarm,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Correction-mode system prompt addendum+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++CORRECTION_PREAMBLE = (+ "=========================================================================\n"+ "CHECK & CORRECT MODE โ INPUT/OUTPUT SHAPE OVERRIDES THE ABOVE.\n"+ "=========================================================================\n"+ "INPUT change: instead of a plain JSON array of raw names you receive a "+ "JSON array of objects, one per joint:\n"+ " {\"raw\": \"<raw joint name>\", \"current\": \"<current cleaned label>\"}\n"+ "The ``current`` label is whatever is already stored in "+ "clean_joint_names.json โ it may be hand-curated, LLM-cleaned, or a "+ "rule-based fallback. Treat it as a candidate, not ground truth.\n\n"+ "PROCESS EACH JOINT IN TWO STEPS.\n\n"+ "STEP 1 โ VALIDATE the current label against ALL cleaning rules above. "+ "The label is VALID only if every check below passes:\n"+ " (a) It appears verbatim in the CANONICAL VOCABULARY (exact term, "+ "Title Case, single spaces).\n"+ " (b) It contains NO digits, underscores, dots, or Blender '.NNN' "+ "counters โ rule 1.\n"+ " (c) It carries NO leftover namespace (mixamorig:, Mutant:, Sif:, ...) "+ "or rig/Maya prefix/suffix (Bip01_, BN_, NPC_, jt_, DEF-, _jnt, _C, "+ "...) โ rules 2-4.\n"+ " (d) If the raw name encodes a side (L_/R_, Lt_/Rt_, Left<Word>, "+ "'Bip001 L/R ', trailing L/R on Japanese roots, F_/B_ for quadrupeds), "+ "the label starts with 'Left '/'Right ' (and includes the Front/Back "+ "marker between side and part when applicable) โ rule 5.\n"+ " (e) If the raw name clearly encodes a recognisable body part, the "+ "label is NOT the 'Bone'/'Appendage' fallback โ rules 6-9.\n"+ " (f) Finger roots, mixamo chain bones (UpLeg/Leg/Foot โ Thigh/Shin/"+ "Foot), 3ds Max Biped finger codes (Finger0*=Thumb, 1*=Index, ...), "+ "Japanese roots, and 'Hair*/Ponytail*/Cape*/Skirt*' โ 'Appendage' are "+ "all resolved correctly โ rules 6-7.\n\n"+ "STEP 2 โ DECIDE:\n"+ " - If VALID on every check in Step 1: OUTPUT THE CURRENT LABEL "+ "VERBATIM (skip โ do not rewrite).\n"+ " - If INVALID on any check: IGNORE the current label and apply the "+ "cleaning rules above to the raw name from scratch; output the "+ "CORRECTED canonical label.\n\n"+ "OUTPUT unchanged (identical to the clean-mode contract): a single FLAT "+ "JSON array of STRINGS โ one label per joint, same order, same length "+ "as the input array. Critical constraints:\n"+ " - Output is an array of strings, NOT objects. Do NOT echo 'raw' or "+ "'current' keys. Do NOT wrap each entry in an object.\n"+ " - Same length as input (number of objects in). One input object -> "+ "one output string.\n"+ " - No prose, no markdown fences, no comments, no trailing text.\n"+ " - Never dedupe, merge, split, reorder or skip โ identical neighbouring "+ "outputs (e.g. 'Spine', 'Spine', 'Spine') are CORRECT.\n"+ " - Title Case with single spaces. Canonical vocabulary terms only.\n"+ " - Side first ('Left Upper Arm'); quadruped marker between side and "+ "part ('Right Front Shoulder').\n\n"+ "CHECK/CORRECT EXAMPLES (current -> final, with verdict):\n"+ " raw='Spine', current='Spine' -> 'Spine' (VALID, skip)\n"+ " raw='mixamorig:LeftUpLeg', current='Left Thigh' -> 'Left Thigh' (VALID, skip)\n"+ " raw='ToeBase.L', current='Left Toe' -> 'Left Toe' (VALID, skip)\n"+ " raw='mixamorig:LeftLeg', current='Left Leg' -> 'Left Shin' (INVALID: mixamo mid-bone)\n"+ " raw='mixamorig:Spine_02', current='Spine 02' -> 'Spine' (INVALID: digits)\n"+ " raw='Bip01_L_Thigh_07', current='Bip01 L Thigh' -> 'Left Thigh' (INVALID: prefix + digits)\n"+ " raw='QuickRigCharacter_LeftForeArm_014',\n"+ " current='Bone' -> 'Left Forearm' (INVALID: lost anatomy)\n"+ " raw='Bone.001_01', current='Bone 001 01' -> 'Bone' (INVALID: digits)\n"+ " raw='munabireL', current='MunabireL' -> 'Left Pectoral Fin' (INVALID: non-canonical)\n"+ " raw='Hair_03', current='Hair' -> 'Mane' (INVALID: animal Hair->Mane)\n"+ " raw='Reins_02', current='Reins' -> 'Reins' (VALID: equipment label)\n"+ " raw='LegTip_R', current='Right Leg Tip' -> 'Right Leg End' (INVALID: Tip not canonical)\n"+)++CORRECTION_SYSTEM_PROMPT = SYSTEM_PROMPT + "\n\n" + CORRECTION_PREAMBLE+++def build_correction_prompt(rig_id, raw_names, current_labels, correction=None):+ # type: (str, List[str], List[str], Optional[str]) -> str+ pairs = [{"raw": raw, "current": current}+ for raw, current in zip(raw_names, current_labels)]+ prompt = (+ "Rig identifier: {rig}\n"+ "Joint count: {n}\n\n"+ "Joints (JSON array of {{raw, current}} objects, order matters):\n"+ "{pairs}\n\n"+ "For each joint: validate the 'current' label against the cleaning "+ "rules. If VALID, output it verbatim; if INVALID, output a corrected "+ "canonical label. Return a FLAT JSON array of EXACTLY {n} STRINGS, "+ "same order as the input. Do NOT output objects; strings only."+ ).format(rig=rig_id, n=len(raw_names),+ pairs=json.dumps(pairs, ensure_ascii=False, indent=2))+ if correction:+ prompt += "\n\nIMPORTANT: " + correction+ return prompt+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-chunk + per-rig corrector (with retry + keep-rule-based fallback)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+#+# Long rigs (50+ joints) reliably break the "emit EXACTLY N entries" contract+# on both Qwen3 and GPT-5 โ the LLM drifts by ยฑ1-2 when near-duplicate names+# sit next to each other (Spine/Spine1/Spine2 โ "Spine" ร 3). Shrinking the+# per-call enumeration fixes it: at chunk_size โค 16 the count stays exact.+# The per-rig SIGALRM budget wraps the whole chunk sequence; a timeout keeps+# already-corrected chunks and leaves the rest as rule-based.++def _run_chunk(rig_id, chunk_raw, chunk_current, client, system_prompt,+ max_tokens, max_retries, chunk_tag=""):+ # type: (...) -> Optional[List[str]]+ """Retry loop for a single chunk. Returns the parsed list on success, or+ ``None`` after exhausting retries. Lets ``RigTimeout`` propagate so the+ caller can stop early and preserve already-finished chunks."""+ correction = None+ last_err = None+ for attempt in range(1, max_retries + 1):+ try:+ user_prompt = build_correction_prompt(+ rig_id, chunk_raw, chunk_current, correction=correction)+ raw_text = client.generate(system_prompt, user_prompt, max_tokens)+ return parse_response(raw_text, len(chunk_raw))+ except (RigTimeout, FatalLLMError):+ raise+ except Exception as err: # noqa: BLE001 โ retry on any backend/parse error+ last_err = err+ if isinstance(err, ValueError):+ correction = (+ "Your previous response failed validation ({}). Return a "+ "JSON array of EXACTLY {} strings โ one corrected label "+ "per raw joint, same order. No prose, no code fences."+ ).format(err, len(chunk_raw))+ logger.warning("[{}{}] attempt {}/{} failed: {}".format(+ rig_id, chunk_tag, attempt, max_retries, err))+ if attempt < max_retries:+ time.sleep(1.5)+ logger.error("[{}{}] chunk failed after {} attempts ({})".format(+ rig_id, chunk_tag, max_retries, last_err))+ return None+++def correct_rig(rig_id, raw_names, current_labels, client,+ system_prompt=CORRECTION_SYSTEM_PROMPT, max_tokens=4096,+ max_retries=MAX_RETRIES, timeout_seconds=15, chunk_size=16):+ # type: (...) -> Tuple[List[str], bool]+ """Correct a single rig via the LLM. Returns ``(labels, still_failed)``.++ Rigs longer than ``chunk_size`` are split into consecutive chunks; each+ chunk is an independent LLM call with its own retries. ``timeout_seconds``+ is the total SIGALRM budget for the whole rig; successfully-corrected+ chunks stay corrected even if a later chunk times out.+ """+ if not raw_names:+ return [], False++ n = len(raw_names)+ result = list(current_labels) # failed / timed-out chunks keep their labels+ chunk_size = max(1, int(chunk_size))+ n_chunks = (n + chunk_size - 1) // chunk_size+ failed_chunks = 0+ timed_out = False++ prev_handler = install_rig_alarm(timeout_seconds)+ try:+ for i in range(n_chunks):+ start = i * chunk_size+ end = min(start + chunk_size, n)+ chunk_tag = (" [{}/{} {}..{}]".format(i + 1, n_chunks, start, end - 1)+ if n_chunks > 1 else "")+ chunk_out = _run_chunk(+ rig_id, raw_names[start:end], current_labels[start:end],+ client, system_prompt, max_tokens, max_retries, chunk_tag=chunk_tag)+ if chunk_out is not None:+ result[start:end] = chunk_out+ else:+ failed_chunks += 1+ except RigTimeout:+ timed_out = True+ finally:+ uninstall_rig_alarm(prev_handler)++ if timed_out:+ logger.warning("[{}] exceeded {}s budget; keeping current labels for "+ "unfinished chunks".format(rig_id, timeout_seconds))+ elif failed_chunks:+ logger.warning("[{}] {}/{} chunks still failed; keeping current labels "+ "for those".format(rig_id, failed_chunks, n_chunks))++ return result, timed_out or failed_chunks > 0+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Driver+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def correct_all(raw_path, cleaned_path, client, system_prompt=CORRECTION_SYSTEM_PROMPT,+ failed_list_path=None, max_tokens=4096, max_retries=MAX_RETRIES,+ timeout_seconds=15, chunk_size=16, still_failed_path=None):+ # type: (...) -> Dict[str, List[str]]+ """Check + correct joint labels in ``cleaned_path`` via the LLM.++ When ``failed_list_path`` is provided, only rigs listed there are+ processed; otherwise every rig in ``raw_path``. The result is written+ BACK IN-PLACE to ``cleaned_path`` (a ``.bak`` sibling is created on first+ run). Rigs the LLM still couldn't fully correct are appended to+ ``still_failed_path``.++ A missing ``failed_list_path`` is a hard error (a typo must not silently+ escalate a targeted repair into a full sweep), and so is a missing+ ``cleaned_path`` in that mode โ the run would otherwise replace the whole+ cleaned file with one holding ONLY the listed rigs, and with no ``.bak``.+ """+ with open(raw_path, "r") as f:+ raw_data = json.load(f)++ if failed_list_path:+ if not os.path.exists(failed_list_path):+ raise FileNotFoundError(+ "--failed_list {!r} does not exist. Drop the flag to correct "+ "every rig in {}.".format(failed_list_path, raw_path))+ if not os.path.exists(cleaned_path):+ raise FileNotFoundError(+ "--failed_list restricts this run to a subset of rigs, but "+ "--cleaned {!r} does not exist: the run would write a cleaned "+ "file containing ONLY the listed rigs (and make no .bak). Run "+ "the joint-name cleaner first.".format(cleaned_path))+ target_ids = read_id_list(failed_list_path)+ scope_desc = "{} rigs from {}".format(len(target_ids), failed_list_path)+ else:+ target_ids = sorted(raw_data.keys())+ scope_desc = "all {} rigs in {}".format(len(target_ids), raw_path)++ if os.path.exists(cleaned_path):+ with open(cleaned_path, "r") as f:+ cleaned_data = json.load(f)+ else:+ cleaned_data = {}+ logger.info("No existing {}; starting from rule-based labels.".format(cleaned_path))++ backup = cleaned_path + ".bak"+ if os.path.exists(cleaned_path) and not os.path.exists(backup):+ shutil.copy2(cleaned_path, backup)+ logger.info("Backed up {} -> {}".format(cleaned_path, backup))++ if still_failed_path is None:+ still_failed_path = os.path.join(+ os.path.dirname(cleaned_path) or ".", "still_failed_clean_names.txt")++ logger.info("Correcting {} (in-place) -> {}".format(scope_desc, cleaned_path))++ still_failed = []+ corrected_count = 0+ pbar = tqdm(target_ids, desc="Correcting", unit="rig", dynamic_ncols=True)+ for rig_id in pbar:+ pbar.set_postfix_str(rig_id, refresh=False)+ if rig_id not in raw_data:+ logger.warning("[{}] missing from raw file; skipping".format(rig_id))+ continue+ raw_names = raw_data[rig_id]+ current = cleaned_data.get(rig_id)+ if not isinstance(current, list) or len(current) != len(raw_names):+ current = rule_based_clean(rig_id, raw_names)++ new_labels, still_bad = correct_rig(+ rig_id, raw_names, current, client,+ system_prompt=system_prompt, max_tokens=max_tokens,+ max_retries=max_retries, timeout_seconds=timeout_seconds,+ chunk_size=chunk_size,+ )+ cleaned_data[rig_id] = new_labels+ if still_bad:+ still_failed.append(rig_id)+ else:+ corrected_count += 1+ save_json(cleaned_data, cleaned_path)++ if still_failed:+ append_id_list(still_failed_path, still_failed)+ logger.info("{} rigs still failed; appended to {}".format(+ len(still_failed), still_failed_path))++ logger.info("Done. Corrected {}/{} rigs (still failed: {}).".format(+ corrected_count, len(target_ids), len(still_failed)))+ return cleaned_data+++def main():+ parser = argparse.ArgumentParser(+ description="LLM check-and-correct pass over clean_joint_names.json (in place).")+ parser.add_argument("--raw", required=True, help="Raw joint_names.json.")+ parser.add_argument("--cleaned", required=True,+ help="clean_joint_names.json โ updated IN PLACE; a '.bak' "+ "sibling is made on first run.")+ parser.add_argument("--failed_list", default=None,+ help="Optional txt file with one rig id per line. When omitted, "+ "ALL rigs in --raw are checked and corrected.")+ parser.add_argument("--still_failed_log", default=None,+ help="Append rig ids the LLM still couldn't fix (default: "+ "sibling 'still_failed_clean_names.txt').")+ parser.add_argument("--rig_timeout", type=int, default=15,+ help="Per-rig wall-time budget in seconds (SIGALRM). 0 disables.")+ parser.add_argument("--chunk_size", type=int, default=16,+ help="Joints per LLM call. Smaller chunks avoid length drift on "+ "long arrays; larger chunks save API calls.")+ add_llm_args(parser, default_max_tokens=4096)+ args = parser.parse_args()++ client = LLMClient.from_args(args)+ logger.info("{} | Writing in-place to: {}".format(client, args.cleaned))++ correct_all(+ raw_path=args.raw,+ cleaned_path=args.cleaned,+ client=client,+ failed_list_path=args.failed_list,+ max_tokens=args.max_tokens,+ max_retries=args.max_retries,+ timeout_seconds=args.rig_timeout,+ chunk_size=args.chunk_size,+ still_failed_path=args.still_failed_log,+ )+++if __name__ == "__main__":+ main()diff --git a/data_process/joint_annotation/vocab.py b/data_process/joint_annotation/vocab.pynew file mode 100644index 0000000..b45880c--- /dev/null+++ b/data_process/joint_annotation/vocab.py@@ -0,0 +1,439 @@+"""Vocabulary data for Truebones / Mixamo joint-name cleaning.++Kept separate from the cleaning logic so mapping edits don't require touching+code. All maps are module-level constants; import them from the cleaner.+"""++# ---------------------------------------------------------------------------+# Prefixes to strip (order matters: longest first)+# ---------------------------------------------------------------------------+REMOVE_PREFIXES = [+ "BN_Bip01_", "BN_Bip01 ",+ "Base Human ",+ "Bip001_", "Bip001 ",+ "Bip01_", "Bip01 ",+ "Preset01_", "Preset01 ",+ "Mutant:", "Sif:",+ "BN_", "NPC_", "jt_",+ "Bn_", "bn_", "b_",+]+++# ---------------------------------------------------------------------------+# Mixamo-specific mappings (humanoid rig, uses CamelCase without underscores)+# ---------------------------------------------------------------------------+MIXAMO_MAP = {+ "Hips": "Hips",+ "Spine": "Spine", "Spine1": "Spine", "Spine2": "Spine",+ "Neck": "Neck",+ "Head": "Head", "HeadTop_End": "Head End",+ "Eye": "Eye",+ "Shoulder": "Shoulder",+ "Arm": "Upper Arm", "ForeArm": "Forearm", "Hand": "Hand",+ "HandThumb": "Thumb Finger", "HandIndex": "Index Finger",+ "HandMiddle": "Middle Finger", "HandRing": "Ring Finger",+ "HandPinky": "Pinky Finger",+ # Mixamo chain UpLeg -> Leg -> Foot: the mid bone "Leg" is the shin.+ "UpLeg": "Thigh", "Leg": "Shin", "Foot": "Foot",+ "ToeBase": "Toe", "Toe_End": "Toe End",+}+++# ---------------------------------------------------------------------------+# Japanese vocabulary (Alligator, Pirrana, Tukan rigs)+# ---------------------------------------------------------------------------+JAPANESE_WORDS = {+ "momo": "Thigh", "sippo": "Tail", "mune": "Chest", "hiza": "Knee",+ "hara": "Abdomen", "ashi": "Foot", "hiji": "Elbow", "koshi": "Hips",+ "kosi": "Hips", "te": "Hand", "kubi": "Neck", "atama": "Head",+ "ago": "Jaw", "kata": "Shoulder", "kao": "Head", "o": "Tail",+ # Fish-specific (Pirrana)+ "munabire": "Pectoral Fin", "era": "Gill", "obire": "Caudal Fin",+ "sebire": "Dorsal Fin", "harabire": "Pelvic Fin",+ "shiribire": "Anal Fin", "shippo": "Tail",+}++# Precomputed lowercase index for fast dispatch+JAPANESE_WORDS_LOWER = {k.lower(): v for k, v in JAPANESE_WORDS.items()}+++# ---------------------------------------------------------------------------+# Canonical anatomical vocabulary (applied after prefix/side stripping)+# ---------------------------------------------------------------------------+CANONICAL = {+ # Body core+ "Pelvis": "Pelvis", "pelv": "Pelvis",+ "Spine": "Spine", "Spn": "Spine", "Spline": "Spine",+ "Ribcage": "Ribcage",+ "Neck": "Neck", "Nek": "Neck",+ "Head": "Head", "Scull": "Skull", "Skull": "Skull",+ "ScullBase": "Skull Base", "SkullBase": "Skull Base",+ "HeadNub": "Head End", "Nub": "End", "Tip": "End",+ "Hips": "Hips", "Cog": "Root",+ # Limb upper+ "Clavicle": "Shoulder", "Collarbone": "Shoulder",+ "Scapula": "Scapula",+ "UpperArm": "Upper Arm", "Upperarm": "Upper Arm",+ "Humerus": "Upper Arm",+ "Forearm": "Forearm", "ForeArm": "Forearm",+ "LowerArm": "Forearm", "Lowerarm": "Forearm",+ "Radius": "Forearm",+ "Hand": "Hand", "Hnd": "Hand", "Palm": "Palm",+ "Wrist": "Wrist", "Elbow": "Elbow", "Shoulder": "Shoulder",+ # Limb lower+ "Thigh": "Thigh", "Femur": "Thigh", "UpLeg": "Thigh",+ "UpperLeg": "Thigh", "Upperleg": "Thigh",+ "Calf": "Shin", "Tibia": "Shin", "Leg": "Leg",+ "LowerLeg": "Shin", "Lowerleg": "Shin",+ "HorseLink": "Fetlock", "LargeCannon": "Cannon",+ "Metacarpus": "Metacarpus", "PhalangesManus": "Phalanges",+ "PhalanxPrima": "Pastern",+ "Foot": "Foot", "Ankle": "Ankle", "Heel": "Heel",+ "Toe": "Toe", "Toes": "Toe", "ToeBase": "Toe", "Ball": "Toe",+ "Hoof": "Hoof", "Knee": "Knee",+ "Foreleg": "Front Leg", "Hindleg": "Hind Leg",+ # Fingers+ "Finger": "Finger", "Thumb": "Thumb Finger",+ "Pinky": "Pinky Finger", "Little": "Pinky Finger",+ "Ring": "Ring Finger",+ "Middle": "Middle Finger", "Index": "Index Finger",+ # Head features+ "Jaw": "Jaw",+ "Tongue": "Tongue", "Thouge": "Tongue", "Tone": "Tongue",+ "tunge": "Tongue", "Tunge": "Tongue",+ "Ear": "Ear",+ "Eye": "Eye", "Eyeball": "Eyeball", "EyeBall": "Eyeball",+ "Eyebrow": "Eyebrow", "Eyelid": "Eyelid",+ "Eyeleds": "Eyelid",+ "Mouth": "Mouth", "Lip": "Lip",+ "Nose": "Nose", "Muzzle": "Muzzle",+ "Chin": "Chin", "Cheek": "Cheek",+ "Torso": "Body", "LegAnkle": "Ankle", "Fingers": "Finger",+ # Appendages+ "Tail": "Tail", "Tai": "Tail",+ "Wing": "Wing", "RWing": "Right Wing", "LWing": "Left Wing",+ "Feather": "Feather",+ # Species-driven, NOT typos โ do not unify: Cricket "Feeler" bones are+ # antennae, Catfish "Feelers" are barbels.+ "Feeler": "Antenna", "Antenna": "Antenna", "Feelers": "Barbel",+ "Tentacle": "Tentacle", "Tentacles": "Tentacle",+ "Claw": "Claw", "HandClaw": "Hand Claw",+ "Fang": "Fang", "Fangs": "Fang",+ "Mandible": "Mandible", "BigMandible": "Large Mandible",+ "LowerMandible": "Lower Mandible",+ "Pincer": "Pincer", "pincers": "Pincer", "Pliers": "Pincer",+ "Piers": "Pincer",+ # Anatomy extras+ "Mane": "Mane", "Fur": "Fur", "Hair": "Mane",+ "Beard": "Whisker", "Mascara": "Whisker",+ "Shell": "Shell", "Stinger": "Stinger", "Horn": "Horn",+ "dorsal": "Dorsal Plate", "Fin": "Fin",+ # Species-driven, NOT typos: Tukan "ponitail" is the bird's crest,+ # humanoid "Ponytail" chains are generic appendages.+ "ponitail": "Crest", "Ponytail": "Appendage",+ "Downbody": "Lower Body", "Down": "Lower Body",+ "Body": "Body",+ # Equipment (horse)+ "Reins": "Reins", "Halter": "Halter",+ # Physics/IK+ "Jiggle": "Jiggle", "TwistBone": "Twist",+ "UpperArmTwist": "Upper Arm Twist", "ForearmTwist": "Forearm Twist",+ "ThighMuscle": "Thigh Muscle", "NeckMuscle": "Neck Muscle",+ # Insect+ "Clip": "Mandible", "Wings": "Wing", "Shall": "Mandible",+ # Misc+ "Trajectory": "Root", "locator": "Root", "locator2": "Root",+ "center": "Center",+ "MagicEffectsNode": "Bone",+ "Handle": "Handle", "IK_Chain": "IK Chain",+ # Rig-noise tokens mapped to "" are DROPPED by the cleaners; a name made+ # only of them collapses to the "Bone" placeholder ('joint16.001',+ # 'internal50.L', 'l_foreleg_jnt01').+ "Joint": "", "Jnt": "", "Internal": "", "Def": "",+ "Part": "", "Armature": "",+ "Mixamorig": "", "Character": "", "Rig": "", "Bip": "",+ "Base": "", "Human": "", "Mid": "",+ "Controller": "", "Controler": "", "Ctrl": "", "Quick": "",+ "Untitled": "", "Pasted": "",+ "Bind": "", "Skeleton": "", "Reference": "", "Res": "", "Main": "",+ # UE-style per-finger segment words: chain position, not anatomy+ # ('IndexDistal', 'thumb_proximal_l' -> the finger label alone).+ "Metacarpal": "", "Proximal": "", "Intermediate": "", "Medial": "",+ "Distal": "",+ "Digit": "Finger",+ "FrontLeg": "Front Leg", "MiddleLeg": "Middle Leg", "HindLeg": "Hind Leg",+ "Crab_pincers": "Pincer",+ "SpineR": "Right Abdomen", "SpineL": "Left Abdomen",+}+++# ---------------------------------------------------------------------------+# Standalone exact-match mappings (checked early in dispatch).+# A few names (locator*, Trajectory, MagicEffectsNode) appear in CANONICAL+# too: here they catch the whole-name form before any prefix/side stripping.+# ---------------------------------------------------------------------------+STANDALONE_MAP = {+ "locator": "Root", "locator2": "Root",+ "EyesBlue_2": "Eye",+ "MagicEffectsNode": "Bone",+ "Handle": "Handle", "IK_Chain01": "IK Chain",+ "BN_P": "Belly",+ "Hips": "Hips", "Spine": "Spine", "Head": "Head",+ "Trajectory": "Root",+ "RightArm": "Right Upper Arm", "RightForeArm": "Right Forearm",+ "LeftArm": "Left Upper Arm", "LeftForeArm": "Left Forearm",+ # Standalone mixamo-style chain: bare "Leg" is the shin (see MIXAMO_MAP).+ "RightLeg": "Right Shin", "LeftLeg": "Left Shin",+ "RightUpLeg": "Right Thigh", "LeftUpLeg": "Left Thigh",+ "RightFoot": "Right Foot", "LeftFoot": "Left Foot",+ "RightHand": "Right Hand", "LeftHand": "Left Hand",+ "Tail01": "Tail", "Tail02": "Tail",+}+++# ---------------------------------------------------------------------------+# SabreToothTiger: full explicit mapping (abbreviation codes are too messy)+# ---------------------------------------------------------------------------+SABRECAT_MAP = {+ "Sabrecat__pelv_": "Pelvis",+ "Sabrecat_LeftThigh_LThi_": "Left Thigh",+ "Sabrecat_LeftCalf_LClf_": "Left Shin",+ "Sabrecat_LeftFoot_LFot_": "Left Foot",+ "Sabrecat_LeftToe0_LT00_": "Left Toe",+ "Sabrecat_LeftToe0_LT01_": "Left Toe",+ "Sabrecat_RightThigh_RThi_": "Right Thigh",+ "Sabrecat_RightCalf_RClf_": "Right Shin",+ "Sabrecat_RightFoot_RFot_": "Right Foot",+ "Sabrecat_RightToe0_RT00_": "Right Toe",+ "Sabrecat_RightToe0_RT01_": "Right Toe",+ "Sabrecat_Tail0_Tal0_": "Tail",+ "Sabrecat_Tail1_Tal1_": "Tail",+ "Sabrecat_Tail2_Tal2_": "Tail",+ "Sabrecat_Spine_Spn0_": "Spine",+ "Sabrecat_Spine_Spn1_": "Spine",+ "MagicEffectsNode": "Bone",+ "Sabrecat_Spine_Spn2_": "Spine",+ "Sabrecat_Spine_Spn3_": "Spine",+ "Sabrecat_Ribcage_Spn4_": "Ribcage",+ "Sabrecat_Ribcage_Spn1_": "Ribcage",+ "Sabrecat_Neck_Nek0_": "Neck",+ "Sabrecat_Neck_Nek1_": "Neck",+ "Sabrecat_Neck_Nek2_": "Neck",+ "Sabrecat_Head__Head_": "Head",+ "Sabrecat_Head__LEye_": "Left Eye",+ "Sabrecat_Head__RChk_": "Right Cheek",+ "Sabrecat_Head_Head__LChk_": "Left Cheek",+ "Sabrecat_HeadLeftEar_LEar_": "Left Ear",+ "Sabrecat_Head_EyeLid_HELT_": "Upper Eyelid",+ "Sabrecat_Head__REye_": "Right Eye",+ "Sabrecat_Head_jaw_": "Jaw",+ "Sabrecat_HeadEyeLid__HELB_": "Lower Eyelid",+ "Sabrecat_HeadRightEar_REar_": "Right Ear",+ "Sabrecat_Head_LM01_": "Left Mouth",+ "Sabrecat_Head_RM01_": "Right Mouth",+ "Sabrecat_RightClavicle_RClv_": "Right Shoulder",+ "Sabrecat_RightUpperArm_RUar_": "Right Upper Arm",+ "Sabrecat_RightForearm_RFar_": "Right Forearm",+ "Sabrecat_RightTwistBone_RFTB_": "Right Twist",+ "Sabrecat_RightHand_RHnd_": "Right Hand",+ "Sabrecat_RightFinger3_RF30_": "Right Finger",+ "Sabrecat_RightFinger3_RF31_": "Right Finger",+ "Sabrecat_RightFinger2_RF20_": "Right Finger",+ "Sabrecat_RightFinger2_RF21_": "Right Finger",+ "Sabrecat_Finger4_RF04_": "Right Finger",+ "Sabrecat_RightFinger1_RF10_": "Right Finger",+ "Sabrecat_RightFinger1_RF11_": "Right Finger",+ "Sabrecat_RightFinger0_RF00_": "Right Finger",+ "Sabrecat_RightFinger0_RF01_": "Right Finger",+ "Sabrecat_LeftClavicle_LClv_": "Left Shoulder",+ "Sabrecat_LeftUpperArm_LUar_": "Left Upper Arm",+ "Sabrecat_LeftForearm_LFar_": "Left Forearm",+ "Sabrecat_LeftTwistBone_LFTB_": "Left Twist",+ "Sabrecat_LeftHand_LHnd_": "Left Hand",+ "Sabrecat_Finger4_LF04_": "Left Finger",+ "Sabrecat_LeftFinger1_LF10_": "Left Finger",+ "Sabrecat_LeftFinger1_LF11_": "Left Finger",+ "Sabrecat_LeftFinger2_LF20_": "Left Finger",+ "Sabrecat_LeftFinger2_LF21_": "Left Finger",+ "Sabrecat_LeftFinger3_RF30_": "Left Finger",+ "Sabrecat_LeftFinger3_RF31_": "Left Finger",+ "Sabrecat_LeftFinger0_LF00_": "Left Finger",+ "Sabrecat_LeftFinger0_LF01_": "Left Finger",+}+++# ---------------------------------------------------------------------------+# Pirrana/Tukan compound Japanese names+# ---------------------------------------------------------------------------+PIRRANA_COMPOUNDS = {+ "munabireR": "Right Pectoral Fin", "munabireL": "Left Pectoral Fin",+ "eraR": "Right Gill", "eraL": "Left Gill",+ "shippoA": "Tail", "shippoB": "Tail",+ "shiribire": "Anal Fin", "shirihireB": "Anal Fin",+ "shiribireA": "Anal Fin",+ "obire": "Caudal Fin", "obireB": "Caudal Fin", "obireA": "Caudal Fin",+ "sebire": "Dorsal Fin",+ "harabireR": "Right Pelvic Fin", "harabireL": "Left Pelvic Fin",+}++# Raw names that route through the Japanese cleaner even when the lowercase+# base isn't a direct JAPANESE_WORDS hit (Pirrana/Tukan/Alligator compounds).+JAPANESE_COMPOUND_NAMES = frozenset({+ "munabireR", "munabireL", "eraR", "eraL", "shippoA", "shippoB",+ "shiribire", "shirihireB", "shiribireA", "obire", "obireB", "obireA",+ "sebire", "harabireR", "harabireL", "locator", "locator2",+ "kosi", "kao",+})+JAPANESE_COMPOUND_ANIMALS = frozenset({"Pirrana", "Tukan", "Alligator"})+++# ---------------------------------------------------------------------------+# NPC_ prefix (Bear / Skyrim-style)+# ---------------------------------------------------------------------------+NPC_DIRECT = {+ "Pelvis": "Pelvis",+ "Ribcage": "Ribcage",+ "Spine1": "Spine", "Spine2": "Spine", "Spine3": "Spine", "Spine4": "Spine",+ "Neck1": "Neck", "Neck2": "Neck",+ "Head": "Head", "Jaw": "Jaw", "Nose": "Nose",+ "UpperRightLip": "Upper Right Lip", "UpperLeftLip": "Upper Left Lip",+ "UpperLip": "Upper Lip",+ "LowerLeftLip": "Lower Left Lip", "LowerFrontLip": "Lower Front Lip",+ "LowerRightLip": "Lower Right Lip",+ "Eyebrow": "Eyebrow",+ "Leg1": "Thigh", "Leg2": "Shin", "LegAnkle": "Ankle",+ "LegBall1": "Foot",+ "Toe": "Toe",+ "Arm1": "Upper Arm", "Arm2": "Forearm",+ "ArmCollarbone": "Shoulder",+ "ArmPalm": "Hand", "ArmBall1": "Foot",+ "Arm1_UpperArmTwist1": "Upper Arm Twist",+ "Arm1_UpperArmTwist2": "Upper Arm Twist",+ "Arm2_ForearmTwist1": "Forearm Twist",+ "Arm2_ForearmTwist2": "Forearm Twist",+ "Pinky01": "Pinky Finger", "Pinky02": "Pinky Finger",+ "Ring01": "Ring Finger", "Ring02": "Ring Finger",+ "Middle01": "Middle Finger", "Middle02": "Middle Finger",+ "Thumb01": "Thumb Finger", "Thumb02": "Thumb Finger",+ "Index01": "Index Finger", "Index02": "Index Finger",+}+++# ---------------------------------------------------------------------------+# Elk prefix (Deer)+# ---------------------------------------------------------------------------+ELK_MAP = {+ "Femur": "Thigh", "Tibia": "Shin",+ "LargeCannon": "Cannon", "PhalanxPrima": "Pastern",+ "RearHoof": "Rear Hoof", "FrontHoof": "Front Hoof",+ "Scapula": "Scapula", "Humerus": "Upper Arm",+ "Radius": "Forearm", "Metacarpus": "Metacarpus",+ "PhalangesManus": "Phalanges",+ "Spine1": "Spine", "Spine2": "Spine", "Spine3": "Spine",+ "Ribcage": "Ribcage",+ "Neck1": "Neck", "Neck2": "Neck", "Neck3": "Neck", "Neck4": "Neck",+ "ScullBase": "Skull Base", "Scull": "Skull",+ "Ear": "Ear", "REar": "Right Ear", "LEar": "Left Ear",+ "Jaw": "Jaw", "UpperLip": "Upper Lip",+ "Pelvis": "Pelvis", "Tail1": "Tail", "Tail2": "Tail",+}+++# ---------------------------------------------------------------------------+# jt_ prefix (Pteranodon, Raptor2/3, Trex, Scorpion-2)+# ---------------------------------------------------------------------------+JT_MAP = {+ "Cog": "Root", "Spine1": "Spine", "Spine2": "Spine",+ "Hips": "Hips", "Hip": "Hip",+ "Thigh": "Thigh", "Knee": "Knee", "Ankle": "Ankle", "Foot": "Foot",+ "ToeMiddle": "Middle Toe", "ToeInner": "Inner Toe", "ToeOutter": "Outer Toe",+ "ClawMiddle": "Middle Claw", "ClawInner": "Inner Claw",+ "ClawOutter": "Outer Claw", "ClawBack": "Back Claw",+ "ClawMiddle2": "Middle Claw", "ClawInner2": "Inner Claw",+ "ClawOutter2": "Outer Claw", "ClawBack2": "Back Claw",+ "Neck1": "Neck", "Neck2": "Neck", "Neck3": "Neck",+ "Head": "Head", "Jaw": "Jaw",+ "Tongue1": "Tongue", "Tongue2": "Tongue",+ "Eye": "Eye", "EyeBall": "Eyeball",+ "Shoulder": "Shoulder", "Elbow": "Elbow", "Wrist": "Wrist",+ "WristBack": "Wrist Back", "ElbowBack": "Elbow Back",+ "Clavicle": "Shoulder",+ "FingerMiddle": "Middle Finger", "FingerInner": "Inner Finger",+ "FingerOutter": "Outer Finger",+ "HandClawMiddle": "Middle Hand Claw", "HandClawInner": "Inner Hand Claw",+ "HandClawOutter": "Outer Hand Claw",+ "Tail1": "Tail", "Tail2": "Tail", "Tail3": "Tail",+ "Tail4": "Tail", "Tail5": "Tail", "Tail6": "Tail",+ "ThighMuscle": "Thigh Muscle", "NeckMuscle": "Neck Muscle",+ "Tail01": "Tail", "Tail02": "Tail", "Tail03": "Tail",+ "Tail04": "Tail", "Tail05": "Tail", "Tail06": "Tail",+ "Tail07": "Tail", "Tail08": "Tail", "Tail09": "Tail",+ "Tail01x": "Tail Twist", "Tail02x": "Tail Twist",+ "Tail03x": "Tail Twist", "Tail04x": "Tail Twist",+ "Tail05x": "Tail Twist", "Tail06x": "Tail Twist",+ "Tail07x": "Tail Twist", "Tail08x": "Tail Twist",+ "FrontLeg1": "Front Leg", "FrontLeg2": "Front Leg",+ "FrontLeg3": "Front Leg", "FrontLeg4End": "Front Leg End",+ "MiddleLeg1": "Middle Leg", "MiddleLeg2": "Middle Leg",+ "MiddleLeg3": "Middle Leg", "MiddleLeg4End": "Middle Leg End",+ "HindLeg1": "Hind Leg", "HindLeg2": "Hind Leg",+ "HindLeg3": "Hind Leg", "HindLeg4End": "Hind Leg End",+ "BigMandible": "Large Mandible", "BigMandibleMid": "Large Mandible",+ "LowerMandible": "Lower Mandible", "LowerMandibleMid": "Lower Mandible",+ "Fangs": "Fang", "FangsMid": "Fang",+}+++# ---------------------------------------------------------------------------+# Spider skeleton (unique naming)+# ---------------------------------------------------------------------------+SPIDER_MAP = {+ "_body_": "Body",+ "NPC_L_MagicNode__LMag_": "Bone",+ "ArmRCollarbone": "Right Shoulder",+ "ArmR_01_": "Right Arm", "ArmR_02_": "Right Arm",+ "ArmRClaw": "Right Claw",+ "ArmLCollarbone": "Left Shoulder",+ "ArmL_01_": "Left Arm", "ArmL_02_": "Left Arm",+ "ArmLClaw": "Left Claw",+ "Tail1": "Tail", "Tail2": "Tail", "Tail3": "Tail",+ "R_Jaw_": "Right Jaw", "L_Jaw_": "Left Jaw",+}+++# ---------------------------------------------------------------------------+# Canonical label format โ ONE format shared by truebones / mixamo / objaverse+#+# label := [Left |Right ] <part> [ End]+#+# where <part> is any base label the maps above emit (qualifiers like+# Front/Hind/Inner/Middle/Outer/Upper/Lower are already baked into those+# values: 'Front Leg', 'Upper Eyelid', ...) and ' End' marks a chain tip+# (Nub/Tip/*_End sources). Two sentinels sit outside the grammar:+# - "Bone" placeholder for rig noise with no recognizable anatomy+# - "_00", "12" numeric passthrough for rigs whose bones were never named+# ---------------------------------------------------------------------------++def canonical_parts():+ """Base part labels (side stripped) that the vocabulary can emit."""+ import re as _re+ parts = set()+ for mapping in (CANONICAL, MIXAMO_MAP, NPC_DIRECT, ELK_MAP, JT_MAP,+ SPIDER_MAP, STANDALONE_MAP, SABRECAT_MAP,+ PIRRANA_COMPOUNDS, JAPANESE_WORDS):+ for value in mapping.values():+ if value:+ parts.add(_re.sub(r'^(Left|Right)\s+', '', value))+ return parts+++def is_canonical_label(label, _parts_cache=[]):+ """True iff *label* conforms to the shared canonical format."""+ import re as _re+ if not _parts_cache:+ _parts_cache.append(canonical_parts())+ if _re.match(r'^_?\d+$', label): # numeric passthrough+ return True+ base = _re.sub(r'^(Left|Right)\s+', '', label)+ base = _re.sub(r'\s+End$', '', base)+ return base in _parts_cache[0]diff --git a/data_process/mesh_animation/__init__.py b/data_process/mesh_animation/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/mesh_animation/animate_fbx.py b/data_process/mesh_animation/animate_fbx.pynew file mode 100644index 0000000..53d26b6--- /dev/null+++ b/data_process/mesh_animation/animate_fbx.py@@ -0,0 +1,184 @@+"""Bake raw animation FBX/GLB clips onto a chosen rigged character.++Unlike :mod:`animate_npz` (which replays an export-stage NPZ), this works+straight from animation files: each clip's action is moved onto the+character's armature by bone name โ the Mixamo convention, where every clip+is authored on the shared ``mixamorig:`` skeleton so any compatible+character can play it without remapping. Bones the character lacks are left+undriven; bones the clip does not touch keep their rest pose.++``--anim_path`` accepts a single clip or a directory of clips, so one+command can bake a whole animation library onto any character. Per-file+failures are logged to ``animate_errors.log`` inside the output dir.++Usage (Blender headless):+ blender -b -P data_process/mesh_animation/animate_fbx.py -- \\+ --char_path dataset/raw/mixamo/character_refined/Amy.fbx \\+ --anim_path dataset/raw/mixamo/animation_motion \\+ --output_dir outputs/animated_Amy \\+ --char_anim_type fbx+"""++import argparse+import os+import re+import sys++import bpy+from loguru import logger+from tqdm import tqdm++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import reset_scene # noqa: E402+from data_process.utils.blender_rig import ( # noqa: E402+ RECONSTRUCTED_ACTION_NAME,+ export_animated_character,+ get_armature_obj,+ load_file,+ repair_and_pack_textures,+ update_scene,+)++_BONE_PATH_RE = re.compile(r'^pose\.bones\["(.+?)"\]')+++def _action_bone_names(action):+ """Bone names an action's F-curves drive."""+ names = set()+ for fcurve in action.fcurves:+ match = _BONE_PATH_RE.match(fcurve.data_path)+ if match:+ names.add(match.group(1))+ return names+++def _adopt_action(char_armature, action):+ """Assign *action* to the character armature and mark it for export.++ ``export_animated_character`` keeps only the action named+ ``RECONSTRUCTED_ACTION_NAME``, so the transferred action takes that name+ (after removing any stale holder of it).+ """+ for stale in list(bpy.data.actions):+ if stale is not action and stale.name == RECONSTRUCTED_ACTION_NAME:+ bpy.data.actions.remove(stale)+ action.name = RECONSTRUCTED_ACTION_NAME++ # The importer keys the animation armature's *object* transform (unit+ # scale, axis correction) into the same action; the character must keep+ # its own object transform, so only bone channels come along.+ for fcurve in [f for f in action.fcurves+ if not f.data_path.startswith('pose.bones')]:+ action.fcurves.remove(fcurve)++ if char_armature.animation_data is None:+ char_armature.animation_data_create()+ char_armature.animation_data.action = action+ # Blender >= 4.4 slotted actions: bind the (single) slot the importer made.+ anim_data = char_armature.animation_data+ if getattr(anim_data, 'action_slot', True) is None and action.slots:+ anim_data.action_slot = action.slots[0]+++def animate_character_fbx(char_path, anim_path, output_dir, char_anim_type='glb',+ fps=30, skip_if_exists=False):+ """Transfer one clip's action onto the character and export GLB/FBX.++ Returns the fraction of the clip's driven bones that exist on the+ character (1.0 = every animated bone landed).+ """+ anim_base = os.path.splitext(os.path.basename(anim_path))[0]+ output_path = os.path.join(output_dir, f"{anim_base}.{char_anim_type}")+ if skip_if_exists and os.path.exists(output_path):+ logger.info(f"Output already exists, skipping: {output_path}")+ return 1.0+ os.makedirs(output_dir, exist_ok=True)++ reset_scene()+ char_armature = get_armature_obj(load_file(char_path))+ assert char_armature is not None, "No armature found in character file"+ repair_and_pack_textures(char_path)++ anim_objs = load_file(anim_path)+ anim_armature = get_armature_obj(anim_objs)+ assert anim_armature is not None, "No armature found in animation file"+ assert anim_armature.animation_data and anim_armature.animation_data.action, \+ "Animation file carries no action"+ action = anim_armature.animation_data.action++ anim_bones = _action_bone_names(action)+ char_bones = {b.name for b in char_armature.data.bones}+ coverage = len(anim_bones & char_bones) / len(anim_bones) if anim_bones else 0.0+ if coverage < 0.5:+ logger.warning(+ f"{anim_base}: only {coverage:.0%} of the clip's bones exist on the "+ f"character โ bone naming likely differs; the result will barely move.")++ _adopt_action(char_armature, action)+ for obj in anim_objs:+ bpy.data.objects.remove(obj, do_unlink=True)++ frame_start, frame_end = action.frame_range+ scene = bpy.context.scene+ scene.render.fps = int(fps)+ scene.frame_start = int(round(frame_start))+ scene.frame_end = int(round(frame_end))+ update_scene()++ export_animated_character(os.path.splitext(output_path)[0],+ formats=(char_anim_type,))+ return coverage+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Bake raw animation FBX/GLB clips onto a rigged character.")+ parser.add_argument("--char_path", type=str, required=True,+ help="Rigged character FBX/GLB (mesh + armature).")+ parser.add_argument("--anim_path", type=str, required=True,+ help="Animation clip, or a directory of clips, whose "+ "bone names match the character's rig.")+ parser.add_argument("--output_dir", type=str, default='outputs/animated',+ help="Destination directory, one file per clip.")+ parser.add_argument("--char_anim_type", type=str, default='glb',+ choices=['glb', 'fbx'],+ help="Export format for the animated character.")+ parser.add_argument("--fps", type=int, default=30,+ help="Scene frame rate for the export (Mixamo is 30).")+ parser.add_argument("--overwrite", action='store_true',+ help="Re-export clips whose output already exists.")+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++if __name__ == "__main__":+ args = parse_args()+ assert os.path.exists(args.char_path), f"Character file not found: {args.char_path}"+ assert os.path.exists(args.anim_path), f"Animation path not found: {args.anim_path}"++ if os.path.isdir(args.anim_path):+ clips = sorted(os.path.join(args.anim_path, f)+ for f in os.listdir(args.anim_path)+ if f.lower().endswith(('.fbx', '.glb')))+ else:+ clips = [args.anim_path]+ logger.info(f"Baking {len(clips)} clip(s) onto {args.char_path}")++ os.makedirs(args.output_dir, exist_ok=True)+ err_log = os.path.join(args.output_dir, "animate_errors.log")++ for clip in tqdm(clips, desc="Animating character"):+ try:+ animate_character_fbx(+ char_path=args.char_path,+ anim_path=clip,+ output_dir=args.output_dir,+ char_anim_type=args.char_anim_type,+ fps=args.fps,+ skip_if_exists=not args.overwrite,+ )+ except Exception as e: # noqa: BLE001 โ keep the batch going+ logger.error(f"Error processing {os.path.basename(clip)}: {e}")+ with open(err_log, "a") as f:+ f.write(f"{os.path.basename(clip)}: {e}\n")diff --git a/data_process/mesh_animation/animate_mixamo.py b/data_process/mesh_animation/animate_mixamo.pynew file mode 100644index 0000000..32c0838--- /dev/null+++ b/data_process/mesh_animation/animate_mixamo.py@@ -0,0 +1,66 @@+"""Batch-animate a Mixamo character mesh with a directory of exported NPZs.++Raw Mixamo data only contains animation (armature + action) without a mesh,+so every NPZ in ``--npz_dir`` is paired with the same ``--char_path`` and+written to ``--output_dir`` via :func:`animate_npz.animate_character`.+Per-file failures are logged to ``animate_errors.log`` inside the output dir.++Usage (Blender headless):+ blender -b -P data_process/mesh_animation/animate_mixamo.py -- \\+ --char_path <character.fbx> \\+ --npz_dir <export_dir>/motions \\+ --output_dir outputs/mixamo_characters+"""++import argparse+import os+import sys++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from loguru import logger # noqa: E402+from tqdm import tqdm # noqa: E402++from data_process.mesh_animation.animate_npz import animate_character # noqa: E402+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Batch-animate a Mixamo character mesh from exported NPZ motions.")+ parser.add_argument("--char_path", type=str, required=True,+ help="Character FBX file (provides mesh + armature).")+ parser.add_argument("--npz_dir", type=str, required=True,+ help="Directory containing export-stage motion NPZ files.")+ parser.add_argument("--output_dir", type=str, default='outputs/mixamo_characters',+ help="Output directory for animated GLB/FBX files.")+ parser.add_argument("--char_anim_type", type=str, default='glb', choices=['glb', 'fbx'],+ help="Export format for the animated character.")+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++if __name__ == "__main__":+ args = parse_args()+ assert os.path.exists(args.char_path), f"Character file not found: {args.char_path}"+ assert os.path.isdir(args.npz_dir), f"NPZ directory not found: {args.npz_dir}"++ os.makedirs(args.output_dir, exist_ok=True)+ err_log = os.path.join(args.output_dir, "animate_errors.log")++ npz_files = sorted(f for f in os.listdir(args.npz_dir) if f.endswith(".npz"))+ logger.info(f"Found {len(npz_files)} NPZ files in {args.npz_dir}")++ for npz_file in tqdm(npz_files, desc="Animating meshes"):+ npz_path = os.path.join(args.npz_dir, npz_file)+ try:+ animate_character(+ char_path=args.char_path,+ anim_path=npz_path,+ output_dir=args.output_dir,+ char_anim_type=args.char_anim_type,+ skip_if_exists=True,+ )+ except Exception as e: # noqa: BLE001 โ keep the batch going+ logger.error(f"Error processing {npz_file}: {e}")+ with open(err_log, "a") as f:+ f.write(f"{npz_file}: {e}\n")diff --git a/data_process/mesh_animation/animate_motion.py b/data_process/mesh_animation/animate_motion.pynew file mode 100644index 0000000..7533a7c--- /dev/null+++ b/data_process/mesh_animation/animate_motion.py@@ -0,0 +1,343 @@+"""Drive a rigged character with a canonicalized motion clip and save GLB + FBX.++This is the inference-time path: it takes a motion in the *feature* format+written by the feature-extraction stage (``global_positions`` /+``local_rotations`` / ``root_facing_quat``) โ or generated by the model in+that same format โ together with the dataset's ``cond.npy`` (skeleton+topology, joint names, scale) and drives the original rigged asset with it.++Pipeline:+ 1. Load the cond entry for the requested ``--dataset_type`` and the motion+ file (NPZ; legacy ``.npy`` model features are also accepted, see below).+ 2. Reconcile the character armature against the animation's bone list+ (see :func:`data_process.utils.blender_rig.sync_armature_bones`).+ 3. Compute per-bone keyframes and bind them as a Blender Action.+ 4. Export the animated character as ``.glb`` and ``.fbx``.++Usage (Blender headless):+ blender -b -P data_process/mesh_animation/animate_motion.py -- \\+ --dataset_type objaverse \\+ --anim_path dataset/features/objaverse/motions/<clip>.npz \\+ --output_dir outputs/animated++``--char_path`` is auto-resolved for truebones / objaverse from the raw+asset directories; it is required for mixamo.+"""++import argparse+import os+import sys++import numpy as np+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from Animation import Animation, Quaternions, transforms_local # noqa: E402++from data_process.utils.blender_export import reset_scene # noqa: E402+from data_process.utils.blender_rig import ( # noqa: E402+ EXTRA_BONES_STRATEGIES,+ compute_bone_keyframes,+ export_animated_character,+ get_armature_obj,+ load_file,+ rebuild_action_from_data,+ repair_and_pack_textures,+ set_scene_timing,+ sync_armature_bones,+ update_scene,+)+from data_process.utils.skeleton import scale_anim # noqa: E402+++MIXAMO_OBJECT_TYPE = 'mixamo'++# Default raw-asset locations used for --char_path auto-resolution.+TRUEBONES_RAW_DIR = 'dataset/raw/truebones/Truebone_Z-OO'+OBJAVERSE_RAW_DIR = 'dataset/raw/objaverse/glb'++# Default cond.npy location per dataset (written by the feature-extraction stage).+COND_PATH_TEMPLATE = 'dataset/features/{dataset_type}/cond.npy'+++# ---------------------------------------------------------------------------+# Animation loading+# ---------------------------------------------------------------------------++def _get_scalar(d, key, default):+ """Read a scalar from an NpzFile / dict, falling back to *default*."""+ return float(d[key]) if key in d else default+++def build_anim_from_npz(anim_path, cond_data):+ """Build ``(anim, rest_anim, tpos_global_rot, raw_anim_data)`` from a+ feature-format NPZ (``global_positions`` + ``local_rotations``)."""+ parents = cond_data['parents']+ scale_factor = _get_scalar(cond_data, 'scale_factor', 1.0)++ offsets_scaled = cond_data['offsets'] / scale_factor+ tpos_local_rot = cond_data['tpos_local_rotations']+ tpos_positions = cond_data['tpos_first_frame'] / scale_factor+ tpos_positions[1:] = offsets_scaled[1:]++ anim_data = np.load(anim_path, allow_pickle=True)+ motion_local_rot = anim_data['local_rotations'] # (nframes, nbones, 4)+ nbones = motion_local_rot.shape[1]++ anim_positions = anim_data['global_positions'] / scale_factor+ anim_positions[:, 1:] = offsets_scaled[1:][None]++ anim = Animation(+ rotations=Quaternions(motion_local_rot),+ positions=anim_positions,+ orients=Quaternions.id(nbones),+ offsets=offsets_scaled,+ parents=parents,+ )+ rest_anim = Animation(+ rotations=Quaternions(tpos_local_rot[None]),+ positions=tpos_positions[None],+ orients=Quaternions.id(nbones),+ offsets=offsets_scaled,+ parents=parents,+ )+ return anim, rest_anim, None, anim_data+++def build_anim_from_npy(anim_path, cond_data, anim_mode):+ """Build ``(anim, rest_anim, tpos_global_rot, raw_anim_data)`` from a+ legacy ``.npy`` of model motion features.++ Decoding model features back into an ``Animation`` requires the+ training/inference package (``src.utils.motion_utils``), which is+ imported lazily here so the NPZ path stays usable without it.+ """+ try:+ from src.utils.motion_utils import (+ recover_anytop_anim_from_rot,+ recover_anytop_joint_pos_from_ric,+ )+ except ImportError as exc: # pragma: no cover โ depends on the model package+ raise RuntimeError(+ "Legacy .npy motion features need the training/inference package "+ "(src.utils.motion_utils); pass a feature-format .npz instead."+ ) from exc++ parents = cond_data['parents']+ scale_factor = _get_scalar(cond_data, 'scale_factor', 1.0)+ tpos_global_rot = cond_data['tpos_global_rotations']++ offsets = cond_data['tpos_offsets']+ offsets_scaled = offsets / scale_factor++ anim_data = np.load(anim_path)+ nbones = anim_data.shape[1]++ if anim_mode == 'fk':+ logger.info("Recovering animation using FK mode")+ anim = recover_anytop_anim_from_rot(anim_data, parents, offsets)+ anim = scale_anim(anim, 1. / scale_factor)+ elif anim_mode == 'ik':+ logger.info("Recovering animation using IK mode")+ from InverseKinematics import animation_from_positions+ recover_pos_ric = recover_anytop_joint_pos_from_ric(anim_data)+ ik_anim, ik_order, _ = animation_from_positions(+ recover_pos_ric, parents, offsets, iterations=150,+ )+ inverse_order = np.argsort(ik_order)+ ik_anim_reordered = Animation(+ rotations=ik_anim.rotations[:, inverse_order],+ positions=ik_anim.positions[:, inverse_order],+ orients=ik_anim.orients[:, inverse_order],+ offsets=offsets,+ parents=parents,+ )+ anim = scale_anim(ik_anim_reordered, 1. / scale_factor)+ else:+ raise RuntimeError(f"Unsupported anim_mode: {anim_mode}")++ rest_anim = Animation(+ rotations=Quaternions.id((1, nbones)),+ positions=offsets_scaled[None],+ orients=Quaternions.id(nbones),+ offsets=offsets_scaled,+ parents=parents,+ )+ return anim, rest_anim, tpos_global_rot, anim_data+++def load_animation(anim_path, cond_data, anim_mode):+ """Dispatch to the NPZ or NPY loader based on file extension."""+ if anim_path.endswith('.npz'):+ return build_anim_from_npz(anim_path, cond_data)+ if anim_path.endswith('.npy'):+ return build_anim_from_npy(anim_path, cond_data, anim_mode)+ raise RuntimeError(f"Unsupported animation file format: {anim_path}")+++# ---------------------------------------------------------------------------+# Path resolution+# ---------------------------------------------------------------------------++def resolve_object_type(anim_path, dataset_type):+ """Map an animation filename to its cond-dict key.++ - truebones / objaverse: filename is ``{object_type}-{motion}[-{clip}].npz``,+ so the object type is the prefix before the first '-'.+ - mixamo: motion filenames carry no ``{object_type}-`` prefix; the cond+ dict always uses ``'mixamo'`` as its single key.+ """+ if dataset_type == 'mixamo':+ return MIXAMO_OBJECT_TYPE+ return os.path.basename(anim_path).split("-")[0]+++def resolve_char_path(anim_path, dataset_type):+ """Auto-resolve a character mesh path from ``anim_path`` + ``dataset_type``.++ - truebones: ``TRUEBONES_RAW_DIR/{object_type}/*.fbx`` (prefers an+ ``Idle`` / T-pose variant when available).+ - objaverse: ``OBJAVERSE_RAW_DIR/{object_type}.glb``.+ - mixamo: no auto-resolution โ ``--char_path`` is required.+ """+ if dataset_type == 'mixamo':+ raise ValueError(+ "dataset_type='mixamo' has no auto-resolved character mesh; "+ "pass --char_path explicitly (or set CHAR_PATH in the bash wrapper)."+ )++ object_type = resolve_object_type(anim_path, dataset_type)++ if dataset_type == 'truebones':+ char_dir = os.path.join(TRUEBONES_RAW_DIR, object_type)+ if not os.path.isdir(char_dir):+ raise FileNotFoundError(+ f"No truebones character directory for object_type='{object_type}' "+ f"(resolved from '{anim_path}') at {char_dir}")+ fbx_files = sorted(f for f in os.listdir(char_dir) if f.lower().endswith('.fbx'))+ if not fbx_files:+ raise FileNotFoundError(f"No FBX file found in {char_dir}")+ idle_files = [f for f in fbx_files if 'idle' in f.lower()]+ return os.path.join(char_dir, idle_files[0] if idle_files else fbx_files[0])++ if dataset_type == 'objaverse':+ char_path = os.path.join(OBJAVERSE_RAW_DIR, f'{object_type}.glb')+ if not os.path.isfile(char_path):+ raise FileNotFoundError(+ f"No objaverse character file for object_type='{object_type}' "+ f"(resolved from '{anim_path}') at {char_path}")+ return char_path++ raise ValueError(f"Unknown dataset_type: {dataset_type}")+++def load_cond_data(cond_path, anim_path, dataset_type):+ object_type = resolve_object_type(anim_path, dataset_type)+ cond_dict = np.load(cond_path, allow_pickle=True).item()+ if object_type not in cond_dict:+ raise KeyError(+ f"object_type '{object_type}' (resolved from '{anim_path}' under "+ f"dataset_type='{dataset_type}') not found in cond file {cond_path}; "+ f"available keys: {sorted(cond_dict.keys())}")+ return cond_dict[object_type]+++# ---------------------------------------------------------------------------+# Main pipeline+# ---------------------------------------------------------------------------++def animate_character(char_path, anim_path, cond_path, output_dir,+ dataset_type, anim_mode='fk',+ extra_bones_strategy='merge'):+ """Drive a character mesh with a motion clip and export GLB + FBX.++ Args:+ char_path: Character FBX/GLB (provides mesh + armature).+ anim_path: Feature-format motion NPZ (or legacy NPY).+ cond_path: ``cond.npy`` for the dataset.+ output_dir: Destination directory (created if missing).+ dataset_type: ``'truebones'`` | ``'objaverse'`` | ``'mixamo'``;+ controls how the cond-dict key is derived from ``anim_path``.+ anim_mode: Recovery mode for legacy NPY input, ``'fk'`` or ``'ik'``.+ extra_bones_strategy: How to handle armature bones not present in+ the animation (see :func:`sync_armature_bones`).+ """+ assert os.path.exists(char_path), f"Character file not found: {char_path}"+ assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"+ assert os.path.exists(cond_path), f"Condition file not found: {cond_path}"++ os.makedirs(output_dir, exist_ok=True)+ anim_base = os.path.splitext(os.path.basename(anim_path))[0]+ output_path_no_ext = os.path.join(output_dir, anim_base)++ reset_scene()+ char_armature = get_armature_obj(load_file(char_path))+ assert char_armature is not None, "No armature found in character file"+ repair_and_pack_textures(char_path)++ cond_data = load_cond_data(cond_path, anim_path, dataset_type)+ bone_names = cond_data['joint_names']+ anim, rest_anim, tpos_global_rot, anim_data = load_animation(+ anim_path, cond_data, anim_mode)++ anim_local_mat = transforms_local(anim) # (nframes, nbones, 4, 4)+ rest_local_mat = transforms_local(rest_anim)[0] # (nbones, 4, 4)++ sync_armature_bones(char_armature, bone_names,+ extra_bones_strategy=extra_bones_strategy)+ set_scene_timing(anim_local_mat.shape[0], _get_scalar(anim_data, 'fps', 30))++ keyframes = compute_bone_keyframes(+ rest_local_mat, anim_local_mat, bone_names, tpos_global_rot)+ rebuild_action_from_data(char_armature, keyframes)+ update_scene()++ export_animated_character(output_path_no_ext, formats=('glb', 'fbx'))+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Drive a rigged character with a canonicalized motion clip.")+ parser.add_argument("--dataset_type", type=str, required=True,+ choices=['truebones', 'objaverse', 'mixamo'],+ help="Dataset that produced the cond/anim files; controls "+ "how the cond-dict key is resolved.")+ parser.add_argument("--anim_path", type=str, required=True,+ help="Feature-format motion NPZ (or legacy .npy).")+ parser.add_argument("--char_path", type=str, default=None,+ help="Character FBX/GLB. If omitted, auto-resolved from "+ f"--dataset_type and --anim_path (truebones -> "+ f"{TRUEBONES_RAW_DIR}/{{object_type}}/*.fbx, objaverse -> "+ f"{OBJAVERSE_RAW_DIR}/{{object_type}}.glb). Required for mixamo.")+ parser.add_argument("--cond_path", type=str, default=None,+ help=f"cond.npy for the dataset (default: {COND_PATH_TEMPLATE}).")+ parser.add_argument("--output_dir", type=str, default='outputs/animated')+ parser.add_argument("--anim_mode", type=str, default='fk', choices=['fk', 'ik'],+ help="Recovery mode for legacy .npy input.")+ parser.add_argument("--extra_bones_strategy", type=str, default='merge',+ choices=list(EXTRA_BONES_STRATEGIES),+ help="What to do with armature bones missing from the "+ "animation. 'merge' (default): transfer vertex weights to "+ "the nearest kept ancestor, then remove the bones. "+ "'remove': delete bones AND mesh vertices dominantly "+ "weighted to them. 'keep': leave in place, no keyframes.")+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++if __name__ == "__main__":+ args = parse_args()+ char_path = args.char_path or resolve_char_path(args.anim_path, args.dataset_type)+ cond_path = args.cond_path or COND_PATH_TEMPLATE.format(dataset_type=args.dataset_type)+ logger.info(f"Using character mesh: {char_path}")+ logger.info(f"Using cond file: {cond_path}")+ animate_character(+ char_path=char_path,+ anim_path=args.anim_path,+ cond_path=cond_path,+ output_dir=args.output_dir,+ dataset_type=args.dataset_type,+ anim_mode=args.anim_mode,+ extra_bones_strategy=args.extra_bones_strategy,+ )diff --git a/data_process/mesh_animation/animate_npz.py b/data_process/mesh_animation/animate_npz.pynew file mode 100644index 0000000..d5d9943--- /dev/null+++ b/data_process/mesh_animation/animate_npz.py@@ -0,0 +1,143 @@+"""Drive a rigged character with an *exported* motion NPZ and save GLB/FBX.++Takes the NPZ format written by the export stage+(``rest_local_pos`` / ``rest_local_rot`` / ``anim_local_pos`` /+``anim_local_rot`` / ``names`` / ``parents`` / ``offsets``) and re-applies it+to a character whose armature uses the same bone names. Useful for+round-trip checks of the export and for re-rendering exported clips.++For *generated* motions (feature NPZ + ``cond.npy``) use+:mod:`data_process.mesh_animation.animate_motion` instead.++Usage (Blender headless):+ blender -b -P data_process/mesh_animation/animate_npz.py -- \\+ --char_path my_model.glb \\+ --anim_path <export_dir>/motions/my_model-Walk.npz \\+ --output_dir outputs/animated+"""++import argparse+import os+import sys++import numpy as np+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from Animation import Animation, Quaternions, transforms_local # noqa: E402++from data_process.utils.blender_export import reset_scene # noqa: E402+from data_process.utils.blender_rig import ( # noqa: E402+ compute_bone_keyframes,+ export_animated_character,+ get_armature_obj,+ load_file,+ rebuild_action_from_data,+ set_scene_timing,+ sync_armature_bones,+ update_scene,+)+++def _npz_get_scalar(npz, key, default):+ """Read a scalar from an NpzFile, falling back to *default*."""+ return float(npz[key]) if key in npz else default+++def build_anims_from_export_npz(anim_data):+ """Build ``(anim, rest_anim)`` Animation objects from an export NPZ."""+ bone_names = anim_data['names'].tolist()+ nbones = anim_data['anim_local_rot'].shape[1]+ scale_factor = _npz_get_scalar(anim_data, 'scale_factor', 1.0)++ anim_positions = anim_data['anim_local_pos'] / scale_factor+ rest_positions = anim_data['rest_local_pos'][None] / scale_factor+ offsets = anim_data['offsets'] / scale_factor+ parents = anim_data['parents']++ anim = Animation(+ rotations=Quaternions(anim_data['anim_local_rot']),+ positions=anim_positions,+ orients=Quaternions.id(nbones),+ offsets=offsets,+ parents=parents,+ )+ rest_anim = Animation(+ rotations=Quaternions(anim_data['rest_local_rot'][None]),+ positions=rest_positions,+ orients=Quaternions.id(nbones),+ offsets=offsets,+ parents=parents,+ )+ return anim, rest_anim, bone_names+++def animate_character(char_path, anim_path, output_dir, char_anim_type='glb',+ extra_bones_strategy='merge', skip_if_exists=False):+ """Drive a character mesh with an export NPZ and save as GLB/FBX.++ Args:+ char_path: Character FBX/GLB (provides mesh + armature).+ anim_path: Motion NPZ produced by the export stage.+ output_dir: Destination directory (created if missing).+ char_anim_type: Export format, ``'glb'`` or ``'fbx'``.+ extra_bones_strategy: How to treat armature bones absent from the+ animation (see :func:`sync_armature_bones`).+ skip_if_exists: Return early when the output file already exists.+ """+ assert os.path.exists(char_path), f"Character file not found: {char_path}"+ assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"++ anim_base = os.path.splitext(os.path.basename(anim_path))[0]+ output_path = os.path.join(output_dir, f"{anim_base}.{char_anim_type}")+ if skip_if_exists and os.path.exists(output_path):+ logger.info(f"Output already exists, skipping: {output_path}")+ return+ os.makedirs(output_dir, exist_ok=True)++ reset_scene()+ char_armature = get_armature_obj(load_file(char_path))+ assert char_armature is not None, "No armature found in character file"++ anim_data = np.load(anim_path, allow_pickle=True)+ anim, rest_anim, bone_names = build_anims_from_export_npz(anim_data)+ anim_local_mat = transforms_local(anim) # (nframes, nbones, 4, 4)+ rest_local_mat = transforms_local(rest_anim)[0] # (nbones, 4, 4)++ sync_armature_bones(char_armature, bone_names,+ extra_bones_strategy=extra_bones_strategy)+ set_scene_timing(anim_local_mat.shape[0], _npz_get_scalar(anim_data, 'fps', 30))++ keyframes = compute_bone_keyframes(rest_local_mat, anim_local_mat, bone_names)+ rebuild_action_from_data(char_armature, keyframes)+ update_scene()++ export_animated_character(os.path.splitext(output_path)[0], formats=(char_anim_type,))+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Drive a rigged character with an exported motion NPZ.")+ parser.add_argument("--char_path", type=str, required=True,+ help="Character FBX/GLB (mesh + armature).")+ parser.add_argument("--anim_path", type=str, required=True,+ help="Export-stage motion NPZ whose bone names match the rig.")+ parser.add_argument("--output_dir", type=str, default='outputs/animated')+ parser.add_argument("--char_anim_type", type=str, default='glb', choices=['glb', 'fbx'])+ parser.add_argument("--extra_bones_strategy", type=str, default='merge',+ choices=['keep', 'merge', 'remove'],+ help="What to do with armature bones missing from the animation.")+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++if __name__ == "__main__":+ args = parse_args()+ animate_character(+ char_path=args.char_path,+ anim_path=args.anim_path,+ output_dir=args.output_dir,+ char_anim_type=args.char_anim_type,+ extra_bones_strategy=args.extra_bones_strategy,+ )diff --git a/data_process/motion_export/__init__.py b/data_process/motion_export/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/motion_export/export_general.py b/data_process/motion_export/export_general.pynew file mode 100644index 0000000..9b5d427--- /dev/null+++ b/data_process/motion_export/export_general.py@@ -0,0 +1,288 @@+"""Export rigged assets (GLB/GLTF or FBX) to NPZ motion data.++Generic counterpart to the per-dataset batch exporters: ``--input`` is either+a single file or a directory of ``.glb``/``.gltf``/``.fbx`` assets (mixed+formats welcome). The importer is picked per file by extension, the character+mesh is optional, and every pose action found in a file is exported as its+own clip. Directory runs mirror the Objaverse exporter: per-asset completion+markers for resume, worker sharding, a per-worker error log, and per-worker+summary shards (merge with ``data_process.tools.merge_summaries``).++Pipeline: import scene โ select armature (+ mesh if present) โ discover+actions โ build skeleton arrays โ extract animations โ optional skeleton+pruning โ coordinate conversion โ save per-action NPZ + MP4 visualization.++Behavior notes:+ - Mesh optional: with a mesh, skin weights are extracted and skeleton+ pruning is enabled by default; without one, ``skin_matrix`` is saved+ as an empty ``(0, nbones)`` array and pruning is skipped (the pruning+ passes rely on skin weights to decide which bones matter).+ - Actions: all relevant pose actions are exported. Files whose action+ does not pass the pose-action filter (e.g. Mixamo animation-only+ clips) fall back to the armature's single bound action.+ - No joint-count filtering: stage 4 (feature_extraction --min_joints /+ --max_joints) decides which skeletons enter training.++Usage (Blender headless):+ blender -b -P data_process/motion_export/export_general.py -- \+ --input assets_dir --output_dir outputs/export++Usage (pip-installed bpy):+ python data_process/motion_export/export_general.py \+ --input assets/dragon.fbx --output_dir outputs/export+"""++import bpy+import sys+import os+import numpy as np+import argparse+from pathlib import Path+from tqdm import tqdm+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import (+ import_fbx, import_gltf, sanitize_action_name, unique_clip_names,+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,+ prune_skeleton_shared, remove_tpose_frames, discover_pose_actions,+ action_frame_range, save_motion, write_export_summary,+ is_asset_complete, mark_asset_complete,+)++GLTF_EXTS = {'.glb', '.gltf'}+FBX_EXTS = {'.fbx'}+ASSET_EXTS = GLTF_EXTS | FBX_EXTS+++def pick_importer(path):+ """Return the importer matching *path*'s extension, or raise."""+ ext = Path(path).suffix.lower()+ if ext in GLTF_EXTS:+ return import_gltf+ if ext in FBX_EXTS:+ return import_fbx+ raise ValueError(f"Unsupported extension '{ext}' (expected one of "+ f"{sorted(ASSET_EXTS)}): {path}")+++def list_asset_files(input_dir):+ """List .glb/.gltf/.fbx files directly under *input_dir*, sorted."""+ input_dir = Path(input_dir)+ return sorted(p for p in input_dir.iterdir()+ if p.is_file() and p.suffix.lower() in ASSET_EXTS)+++def export_asset(input_path, output_dir, save_name=None, fps=30,+ dtype=np.float64, prune=None, consider_parent_rotate=True,+ min_frames=5, remove_tpose=True, save_vis=True):+ """Export every pose action in a single GLB/GLTF/FBX file to NPZ clips.++ Args:+ prune: True/False to force skeleton pruning on/off; None (default)+ enables it iff the file contains a skinned mesh.++ Returns:+ {save_name: joint_names} when at least one clip was saved, else {}.+ """+ assert os.path.exists(input_path), f"Input file not found: {input_path}"+ save_name = save_name or Path(input_path).stem++ motion_save_dir = os.path.join(output_dir, "motions")+ vis_save_dir = os.path.join(output_dir, "videos")+ tpos_save_dir = os.path.join(output_dir, "tpose")++ if is_asset_complete(output_dir, save_name):+ logger.info(f"Asset '{save_name}' already complete, skipping: {input_path}")+ return {}+ existing = sorted(Path(motion_save_dir).glob(f"{save_name}-*.npz")) if os.path.isdir(motion_save_dir) else []+ if existing:+ # Pruning is joint across all of an asset's clips, so a partial+ # (or pre-marker) export must be redone as a whole.+ logger.warning(f"'{save_name}' has {len(existing)} clip(s) but no completion "+ f"marker (partial or pre-marker export); re-exporting.")++ logger.info(f"Processing asset: {input_path}")+ os.makedirs(motion_save_dir, exist_ok=True)+ os.makedirs(vis_save_dir, exist_ok=True)++ armature, mesh = load_scene(pick_importer(input_path), input_path, fps=fps)+ if mesh is None:+ logger.info(f"No mesh in {input_path}; exporting armature only "+ f"(empty skin matrix, pruning {'forced on' if prune else 'off'}).")+ if prune is None:+ prune = mesh is not None++ # All relevant pose actions; fall back to the armature's bound action+ # for single-action files that the pose filter rejects.+ action_names, frame_ranges = discover_pose_actions()+ if not action_names and armature.animation_data is not None \+ and armature.animation_data.action is not None:+ action = armature.animation_data.action+ action_names = [action.name]+ frame_ranges = {action.name: action_frame_range(action)}+ logger.info(f"No pose actions discovered; falling back to bound action '{action.name}'.")+ logger.info(f"Total actions to export: {len(action_names)}")+ if not action_names:+ logger.info(f"No actions in {input_path}; skipping.")+ mark_asset_complete(output_dir, save_name, status="skipped",+ reason="no pose actions")+ return {}++ skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)+ extracted = extract_all_actions(+ armature, {n: frame_ranges[n] for n in action_names},+ skel['rest_local_pos'], skel['parents_array'],+ skel['rest_anim'], skel['nbones'], dtype=dtype,+ )++ anims_list = [anim for _, anim in extracted]+ rest_anim_shared = skel['rest_anim_shared']+ names = list(skel['bone_names'])+ skin_matrix = skel['skin_matrix']+ if prune:+ anims_list, rest_anim_shared, names, skin_matrix = prune_skeleton_shared(+ anims_list, rest_anim_shared, names, skin_matrix.copy(),+ consider_parent_rotate=consider_parent_rotate,+ )++ save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_shared)++ # Distinct actions can sanitize to the same name and would then overwrite+ # each other's NPZ; disambiguate over the full discovered list.+ clip_suffixes = unique_clip_names(action_names)++ scene_fps = bpy.context.scene.render.fps+ saved_clips = []+ for i, (action_name, _) in enumerate(extracted):+ anim = anims_list[i]++ if remove_tpose:+ anim, n_tpose = remove_tpose_frames(anim, rest_anim_shared)+ if n_tpose > 0:+ logger.info(f"Removed {n_tpose} T-pose frames from '{action_name}', "+ f"{anim.positions.shape[0]} frames remaining")+ if anim.positions.shape[0] < min_frames:+ logger.info(f"Skipping '{action_name}': only {anim.positions.shape[0]} frames "+ f"(min={min_frames})")+ continue++ clip_name = f"{save_name}-{clip_suffixes.get(action_name) or sanitize_action_name(action_name) or 'action'}"+ save_motion(+ os.path.join(motion_save_dir, f"{clip_name}.npz"),+ os.path.join(vis_save_dir, f"{clip_name}.mp4"),+ anim, rest_anim_shared, names, skin_matrix, scene_fps,+ action_name=action_name, save_vis=save_vis,+ )+ saved_clips.append(clip_name)++ mark_asset_complete(output_dir, save_name,+ status="exported" if saved_clips else "skipped",+ n_clips=len(saved_clips),+ reason=None if saved_clips else "no clips survived filtering",+ joint_names=names if saved_clips else None)+ logger.info(f"Saved {len(saved_clips)} clip(s) for '{save_name}'")+ if saved_clips:+ return {save_name: names}+ return {}+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Entry point+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_args():+ parser = argparse.ArgumentParser(+ description="Export rigged GLB/GLTF/FBX assets (a file or a directory) "+ "to NPZ motion data.")+ parser.add_argument('--input', type=str, required=True,+ help='A .glb/.gltf/.fbx file, or a directory of them.')+ parser.add_argument('--output_dir', type=str, required=True,+ help='Output directory (motions/, videos/, tpose/ created inside).')+ parser.add_argument('--name', type=str, default=None,+ help='Save name / clip prefix (single-file input only; '+ 'default: input file stem).')+ parser.add_argument('--fps', type=int, default=30)+ parser.add_argument('--prune', action=argparse.BooleanOptionalAction, default=None,+ help='Force skeleton pruning on/off '+ '(default: on iff the file has a skinned mesh).')+ parser.add_argument('--consider_parent_rotate', action=argparse.BooleanOptionalAction, default=True,+ help='Preserve non-rotating leaves whose parent rotates (see prune_skeleton_shared).')+ parser.add_argument('--min_frames', type=int, default=5,+ help='Skip clips shorter than this many frames '+ '(shared minimum across all dataset exporters).')+ parser.add_argument('--keep_tpose_frames', action='store_true',+ help='Keep frames matching the rest pose instead of removing them.')+ parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,+ help='Render the per-clip MP4 preview (use --no-vis for bulk runs).')+ parser.add_argument('--worker_id', type=int, default=0,+ help='Worker index for sharding files across workers (directory input).')+ parser.add_argument('--num_workers', type=int, default=1,+ help='Total number of workers (directory input).')++ # Blender passes script args after "--"; plain python passes them directly.+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++def main():+ args = parse_args()+ export_kwargs = dict(+ fps=args.fps, prune=args.prune,+ consider_parent_rotate=args.consider_parent_rotate,+ min_frames=args.min_frames, remove_tpose=not args.keep_tpose_frames,+ save_vis=args.vis,+ )++ if os.path.isdir(args.input):+ asset_paths = list_asset_files(args.input)+ assert asset_paths, f"No {sorted(ASSET_EXTS)} files found in {args.input}"+ logger.info(f"Found {len(asset_paths)} asset files in {args.input}")++ if args.num_workers > 1:+ asset_paths = asset_paths[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "+ f"processing {len(asset_paths)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'export_errors_worker{args.worker_id}.log')+ all_joint_names = {}+ n_failed = 0+ for asset_path in tqdm(asset_paths, desc="Exporting assets"):+ try:+ saved = export_asset(str(asset_path), args.output_dir,+ **export_kwargs)+ if saved:+ all_joint_names.update(saved)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ logger.exception(f"Failed to export {asset_path}")+ with open(error_log, 'a') as log_file:+ log_file.write(f"Failed to export {asset_path}: {e}\n")++ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""+ write_export_summary(args.output_dir, all_joint_names,+ worker_suffix=worker_suffix)+ else:+ saved = export_asset(args.input, args.output_dir, save_name=args.name,+ **export_kwargs)+ if saved:+ write_export_summary(args.output_dir, saved)++ if n_failed:+ logger.error(f"{n_failed}/{len(asset_paths)} assets failed; "+ f"see {error_log}")+ sys.exit(1)+++if __name__ == "__main__":+ # Blender exits 0 even on an uncaught exception, which hides failures from+ # `set -e` and from Slurm's afterok dependencies.+ try:+ main()+ except SystemExit:+ raise+ except Exception:+ logger.exception("export_general failed")+ sys.exit(1)diff --git a/data_process/motion_export/export_mixamo.py b/data_process/motion_export/export_mixamo.pynew file mode 100644index 0000000..0cc849a--- /dev/null+++ b/data_process/motion_export/export_mixamo.py@@ -0,0 +1,236 @@+"""Export Mixamo animation FBX files to NPZ motion data.++Pipeline: import animation FBX (armature + action, no mesh) โ build skeleton+arrays โ extract animation โ Z-up โ Y-up โ save per-clip NPZ + MP4.++Skin weights are not extracted (there is no character mesh); ``skin_matrix``+is saved as an empty ``(0, nbones)`` array. All Mixamo animations share the+same skeleton, so ``joint_names.json`` and the rest-pose PNG are written once+(by worker 0) and every freshly-extracted clip's rest pose is checked against+the first one.++Files can be sharded across parallel Blender processes with+``--worker_id`` / ``--num_workers``; each worker writes its own summary+JSONs (``*_worker{i}.json``) โ merge them afterwards with+``python -m data_process.tools.merge_summaries`` (the+``run_export.sh`` wrapper does both automatically).++Usage (Blender headless):+ blender -b -P data_process/motion_export/export_mixamo.py -- \+ --anim_dir dataset/raw/mixamo/animation_motion \+ --output_dir dataset/export/mixamo+"""++import bpy+import sys+import os+import json+import numpy as np+import argparse+from tqdm import tqdm+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import (+ import_fbx,+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,+ action_frame_range, save_motion, write_export_summary,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Main export function+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def export_mixamo(anim_path, output_dir, fps=30, dtype=np.float64,+ min_frames=5, skip_if_exists=True, save_vis=True):+ """Export a single Mixamo animation clip.++ Args:+ anim_path: Path to animation FBX (provides armature + action).+ output_dir: Output root directory.+ fps: Scene FPS.+ dtype: Numpy dtype for arrays.+ skip_if_exists: Return early when the output NPZ already exists.+ save_vis: Also render the per-clip MP4 preview.++ Returns:+ (bone_names, rest_anim) if extracted, else None.+ """+ assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"++ motion_save_dir = os.path.join(output_dir, "motions")+ vis_save_dir = os.path.join(output_dir, "videos")+ os.makedirs(motion_save_dir, exist_ok=True)+ os.makedirs(vis_save_dir, exist_ok=True)++ anim_base_name = os.path.splitext(os.path.basename(anim_path))[0]+ motion_save_path = os.path.join(motion_save_dir, f"{anim_base_name}.npz")+ vis_save_path = os.path.join(vis_save_dir, f"{anim_base_name}.mp4")++ # Skip if output already exists (unless caller still needs metadata)+ output_exists = os.path.exists(motion_save_path)+ if output_exists and skip_if_exists:+ logger.info(f"Output already exists for '{anim_base_name}', skipping.")+ return None++ armature, _ = load_scene(import_fbx, anim_path, fps=fps)+ assert armature.animation_data is not None and armature.animation_data.action is not None, \+ f"Armature '{armature.name}' has no animation data or action."++ action = armature.animation_data.action+ start_frame, end_frame = action_frame_range(action)+ logger.info(f"Action frame range: {start_frame} to {end_frame}")++ skel = prepare_skeleton(armature, mesh=None, dtype=dtype, apply_world=True)+ extracted = extract_all_actions(+ armature, {action.name: (start_frame, end_frame)},+ skel['rest_local_pos'], skel['parents_array'],+ skel['rest_anim'], skel['nbones'], dtype=dtype,+ )+ _, anim_obj = extracted[0]++ nframes = anim_obj.positions.shape[0]+ if nframes < min_frames:+ logger.info(f"Skipping '{anim_base_name}': only {nframes} frames "+ f"(min={min_frames})")+ return None++ if not output_exists:+ scene_fps = bpy.context.scene.render.fps+ save_motion(motion_save_path, vis_save_path, anim_obj, skel['rest_anim_shared'],+ list(skel['bone_names']), skel['skin_matrix'], scene_fps,+ action_name=anim_base_name, save_vis=save_vis)++ return list(skel['bone_names']), skel['rest_anim_shared']+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Entry point+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_args():+ parser = argparse.ArgumentParser(description="Export Mixamo animation FBX files to NPZ motion data.")+ parser.add_argument("--anim_dir", type=str, required=True,+ help='Directory containing animation FBX files.')+ parser.add_argument("--output_dir", type=str, required=True,+ help='Output directory (motions/, videos/, tpose/ created inside).')+ parser.add_argument("--fps", type=int, default=30,+ help="Scene FPS for import and export (dataset is 30 fps).")+ parser.add_argument("--min_frames", type=int, default=5,+ help="Skip clips shorter than this many frames "+ "(shared minimum across all dataset exporters).")+ parser.add_argument("--worker_id", type=int, default=0,+ help="Worker index for sharding files across workers.")+ parser.add_argument("--num_workers", type=int, default=1,+ help="Total number of workers.")+ parser.add_argument("--vis", action=argparse.BooleanOptionalAction, default=True,+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++def main(args):+ os.makedirs(args.output_dir, exist_ok=True)+ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""+ error_log = os.path.join(args.output_dir, f"export_errors{worker_suffix}.log")++ n_failed = 0+ anim_files = sorted([f for f in os.listdir(args.anim_dir) if f.endswith(".fbx")])+ anim_paths = [os.path.join(args.anim_dir, f) for f in anim_files]+ logger.info(f"Found {len(anim_paths)} animation files")++ if args.num_workers > 1:+ anim_paths = anim_paths[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "+ f"processing {len(anim_paths)} files")++ joint_names_path = os.path.join(args.output_dir, "joint_names.json")+ tpos_path = os.path.join(args.output_dir, "tpose", "mixamo.png")++ # Only worker 0 writes the shared skeleton metadata (single skeleton for+ # the whole dataset). While it is missing, worker 0 extracts its first+ # file even if the NPZ already exists, to recover the reference skeleton.+ is_primary = args.worker_id == 0+ need_metadata = is_primary and not (+ os.path.exists(joint_names_path) and os.path.exists(tpos_path))++ ref_bone_names = None+ ref_rest_pos = None+ ref_rest_rot = None+ tpos_mismatches = []++ for anim_path in tqdm(anim_paths, desc="Processing animations"):+ try:+ result = export_mixamo(anim_path, args.output_dir, fps=args.fps,+ min_frames=args.min_frames,+ skip_if_exists=not need_metadata,+ save_vis=args.vis)+ if result is None:+ continue++ bone_names, rest_anim = result+ rest_pos = rest_anim.positions[0] # (nbones, 3)+ rest_rot = rest_anim.rotations[0].qs # (nbones, 4)++ if ref_bone_names is None:+ ref_bone_names = bone_names+ ref_rest_pos = rest_pos.copy()+ ref_rest_rot = rest_rot.copy()+ if need_metadata:+ with open(joint_names_path, 'w') as f:+ json.dump({"mixamo": bone_names}, f, indent=2)+ logger.info(f"Saved joint names to {joint_names_path}")+ if not os.path.exists(tpos_path):+ save_rest_pose_vis(os.path.dirname(tpos_path), "mixamo", rest_anim)+ need_metadata = False+ else:+ # Check T-pose consistency against the reference skeleton+ anim_name = os.path.basename(anim_path)+ if bone_names != ref_bone_names:+ logger.warning(f"Bone names mismatch: {anim_name}")+ tpos_mismatches.append(anim_name)+ else:+ pos_diff = np.max(np.abs(rest_pos - ref_rest_pos))+ rot_diff = np.max(np.abs(rest_rot - ref_rest_rot))+ if pos_diff > 1e-6 or rot_diff > 1e-6:+ logger.warning(f"T-pose mismatch: {anim_name} "+ f"(pos_diff={pos_diff:.8f}, rot_diff={rot_diff:.8f})")+ tpos_mismatches.append(anim_name)++ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Error processing {anim_path}")+ with open(error_log, "a") as f:+ f.write(f"Error processing {anim_path}: {e}\n")++ if tpos_mismatches:+ logger.warning(f"{len(tpos_mismatches)} animations have mismatched T-pose: {tpos_mismatches}")+ else:+ logger.info("All extracted animations share the same T-pose.")++ # On a fully-resumed run nothing is extracted; fall back to the saved+ # joint names instead of overwriting the summary files with {}.+ if ref_bone_names is None and os.path.exists(joint_names_path):+ with open(joint_names_path) as f:+ ref_bone_names = json.load(f).get("mixamo")++ all_joint_names = {"mixamo": ref_bone_names} if ref_bone_names else {}+ write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)++ if n_failed:+ logger.error(f"{n_failed}/{len(anim_paths)} clips failed; see {error_log}")+ sys.exit(1)+++if __name__ == "__main__":+ # Blender exits 0 even on an uncaught exception, which hides failures from+ # `set -e` and from Slurm's afterok dependencies.+ try:+ main(parse_args())+ except SystemExit:+ raise+ except Exception:+ logger.exception("export_mixamo failed")+ sys.exit(1)diff --git a/data_process/motion_export/export_objaverse.py b/data_process/motion_export/export_objaverse.pynew file mode 100644index 0000000..c2b0818--- /dev/null+++ b/data_process/motion_export/export_objaverse.py@@ -0,0 +1,207 @@+"""Export a directory of rigged GLB/GLTF files (e.g. Objaverse) to NPZ motion data.++Pipeline per file: import scene โ select armature/mesh โ discover actions โ+extract animations โ Z-up โ Y-up โ prune skeleton (shared across clips) โ+remove T-pose frames โ save per-clip NPZ + MP4 + rest-pose PNG.++Files can be sharded across parallel Blender processes with+``--worker_id`` / ``--num_workers``; each worker writes its own summary+JSONs (``*_worker{i}.json``).++Usage (Blender headless):+ blender -b -P data_process/motion_export/export_objaverse.py -- \+ --data_dir dataset/raw/objaverse/glb \+ --output_dir dataset/export/objaverse+"""++import bpy+import sys+import os+import numpy as np+import argparse+from pathlib import Path+from loguru import logger+from tqdm import tqdm++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import (+ import_gltf, list_gltf_files, sanitize_action_name, unique_clip_names,+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,+ prune_skeleton_shared, remove_tpose_frames, discover_pose_actions,+ save_motion, write_export_summary,+ is_asset_complete, mark_asset_complete,+)++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Main export pipeline+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def export_objaverse(gltf_path, save_name, output_dir, fps=30, dtype=np.float64,+ min_frames=5, consider_parent_rotate=True, save_vis=True):+ """Export every pose action in one GLB/GLTF file. Returns ``{save_name: joint_names}``."""+ assert os.path.exists(gltf_path), f"GLTF file not found: {gltf_path}"++ motion_save_dir = os.path.join(output_dir, "motions")+ vis_save_dir = os.path.join(output_dir, "videos")+ tpos_save_dir = os.path.join(output_dir, "tpose")++ if is_asset_complete(output_dir, save_name):+ logger.info(f"Asset '{save_name}' already complete, skipping: {gltf_path}")+ return {}+ existing = sorted(Path(motion_save_dir).glob(f"{save_name}-*.npz")) if os.path.isdir(motion_save_dir) else []+ if existing:+ # Pruning is joint across all of an asset's clips, so a partial+ # (or pre-marker) export must be redone as a whole.+ logger.warning(f"'{save_name}' has {len(existing)} clip(s) but no completion "+ f"marker (partial or pre-marker export); re-exporting.")++ logger.info(f"Processing GLTF: {gltf_path}")+ os.makedirs(motion_save_dir, exist_ok=True)+ os.makedirs(vis_save_dir, exist_ok=True)++ armature, mesh = load_scene(import_gltf, gltf_path, fps=fps)+ assert mesh is not None, f"No mesh found in {gltf_path}; cannot build skin matrix."++ action_names, frame_ranges = discover_pose_actions()+ logger.info(f"Total relevant actions: {len(action_names)}")+ if not action_names:+ logger.info(f"No valid actions in {gltf_path}; skipping.")+ mark_asset_complete(output_dir, save_name, status="skipped",+ reason="no pose actions")+ return {}++ skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)+ extracted = extract_all_actions(+ armature, frame_ranges,+ skel['rest_local_pos'], skel['parents_array'],+ skel['rest_anim'], skel['nbones'], dtype=dtype,+ )++ anims_list = [anim for _, anim in extracted]+ anims_list, rest_anim_pruned, names_pruned, skin_matrix_pruned = prune_skeleton_shared(+ anims_list, skel['rest_anim_shared'], skel['bone_names'].copy(), skel['skin_matrix'].copy(),+ consider_parent_rotate=consider_parent_rotate,+ )++ # No joint-count gate here: stage 4 (feature_extraction --min_joints /+ # --max_joints) decides which skeletons enter training, so the export+ # stays complete.+ save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_pruned)++ # Distinct actions can sanitize to the same name; disambiguate over the+ # full discovered list so the renderer derives identical clip names.+ clip_suffixes = unique_clip_names(action_names)++ scene_fps = bpy.context.scene.render.fps+ saved_clips = []+ for i, (action_name, _) in enumerate(extracted):+ anim = anims_list[i]++ anim, n_tpose = remove_tpose_frames(anim, rest_anim_pruned)+ if n_tpose > 0:+ logger.info(f"Removed {n_tpose} T-pose frames from '{action_name}', "+ f"{anim.positions.shape[0]} frames remaining")+ if anim.positions.shape[0] < min_frames:+ logger.info(f"Skipping '{action_name}': only {anim.positions.shape[0]} frames "+ f"after T-pose removal (min={min_frames})")+ continue++ clean_action_name = clip_suffixes.get(+ action_name, sanitize_action_name(action_name) or "action")+ clip_name = f"{save_name}-{clean_action_name}"+ motion_path = os.path.join(motion_save_dir, f"{clip_name}.npz")+ vis_path = os.path.join(vis_save_dir, f"{clip_name}.mp4")+ save_motion(motion_path, vis_path, anim, rest_anim_pruned,+ names_pruned, skin_matrix_pruned, scene_fps,+ action_name=action_name, save_vis=save_vis)+ saved_clips.append(clip_name)++ mark_asset_complete(output_dir, save_name,+ status="exported" if saved_clips else "skipped",+ n_clips=len(saved_clips),+ reason="" if saved_clips else "no clips above min frame count",+ joint_names=names_pruned if saved_clips else None)++ # Return once per asset (skeleton type), not per clip+ if saved_clips:+ return {save_name: names_pruned}+ return {}+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Entry point+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_args():+ parser = argparse.ArgumentParser(description="Export GLB/GLTF files to NPZ motion data.")+ parser.add_argument('--data_dir', type=str, required=True,+ help='Directory containing GLB/GLTF files.')+ parser.add_argument('--output_dir', type=str, required=True,+ help='Output directory for NPZ motion data.')+ parser.add_argument('--fps', type=int, default=30,+ help='Scene FPS for import and export.')+ parser.add_argument('--consider_parent_rotate', action=argparse.BooleanOptionalAction, default=True,+ help='Preserve non-rotating leaves whose parent rotates during pruning '+ '(see prune_skeleton_shared; use --no-consider_parent_rotate to disable).')+ parser.add_argument('--min_frames', type=int, default=5,+ help='Skip clips shorter than this after T-pose removal.')+ parser.add_argument('--worker_id', type=int, default=0,+ help="Worker index for sharding files across workers.")+ parser.add_argument('--num_workers', type=int, default=1,+ help="Total number of workers.")+ parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")++ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++def main():+ args = parse_args()++ gltf_paths = list_gltf_files(Path(args.data_dir))+ assert len(gltf_paths) > 0, f"No GLB/GLTF files found in {args.data_dir}"+ logger.info(f"Found {len(gltf_paths)} GLB/GLTF files in {args.data_dir}")++ # Shard files across workers+ if args.num_workers > 1:+ gltf_paths = gltf_paths[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: processing {len(gltf_paths)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir, f'export_errors_worker{args.worker_id}.log')+ all_joint_names = {}+ n_failed = 0+ for glb_path in tqdm(gltf_paths, desc="Exporting GLB/GLTF files"):+ try:+ saved = export_objaverse(glb_path, glb_path.stem, output_dir=args.output_dir,+ fps=args.fps, min_frames=args.min_frames,+ consider_parent_rotate=args.consider_parent_rotate,+ save_vis=args.vis)+ if saved:+ all_joint_names.update(saved)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Failed to export {glb_path}")+ with open(error_log, 'a') as log_file:+ log_file.write(f"Failed to export {glb_path}: {e}\n")++ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""+ write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)++ if n_failed:+ logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")+ sys.exit(1)+++if __name__ == "__main__":+ # Blender exits 0 even on an uncaught exception, which hides failures from+ # `set -e` and from Slurm's afterok dependencies.+ try:+ main()+ except SystemExit:+ raise+ except Exception:+ logger.exception("export_objaverse failed")+ sys.exit(1)diff --git a/data_process/motion_export/export_truebones.py b/data_process/motion_export/export_truebones.pynew file mode 100644index 0000000..ecd6562--- /dev/null+++ b/data_process/motion_export/export_truebones.py@@ -0,0 +1,271 @@+"""Export Truebones FBX clips to NPZ motion data.++Consumes the curated flat layout of the Truebones ZOO dataset: one FBX per+clip, named ``{Species}-{Action}.fbx`` (species never contains ``-``), all in+a single directory. Files are grouped by species so skeleton pruning is+shared across every clip of that species.++Pipeline: import โ select armature/mesh โ extract animation โ Z-up โ Y-up โ+prune skeleton (shared across the species' clips) โ save per-clip NPZ + MP4++ rest-pose PNG.++Usage (Blender headless):+ blender -b -P data_process/motion_export/export_truebones.py -- \+ --data_dir dataset/raw/truebones/animation \+ --output_dir dataset/export/truebones+"""++import bpy+import sys+import os+import numpy as np+from tqdm import tqdm+import argparse+from pathlib import Path+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import (+ import_fbx,+ load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,+ prune_skeleton_shared,+ action_frame_range, sanitize_action_name, sanitize_object_type,+ save_motion, write_export_summary,+ is_asset_complete, mark_asset_complete,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-FBX extraction (one action per file)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def extract_single_fbx(fbx_path, fps=30, dtype=np.float64):+ """Import one FBX file and return its (anim, rest_anim, names, skin_matrix, nbones)."""+ assert os.path.exists(fbx_path), f"FBX file not found: {fbx_path}"++ armature, mesh = load_scene(import_fbx, fbx_path, fps=fps)+ assert armature.animation_data is not None and armature.animation_data.action is not None, \+ f"Armature '{armature.name}' has no animation data or action."++ action = armature.animation_data.action+ start_frame, end_frame = action_frame_range(action)+ logger.info(f"Action frame range: {start_frame} to {end_frame}")++ skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)+ extracted = extract_all_actions(+ armature, {action.name: (start_frame, end_frame)},+ skel['rest_local_pos'], skel['parents_array'],+ skel['rest_anim'], skel['nbones'], dtype=dtype,+ )+ _, anim = extracted[0]+ return {+ 'anim': anim,+ 'rest_anim': skel['rest_anim_shared'],+ 'names': skel['bone_names'],+ 'skin_matrix': skel['skin_matrix'],+ 'nbones': skel['nbones'],+ }+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Shared post-processing: pruning, visualization, saving+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def postprocess_and_save(obj_type, extracted, rest_anim_shared, bone_names,+ skin_matrix, output_dir, fps,+ consider_parent_rotate=True, save_vis=True,+ save_tpose=True):+ """Shared skeleton pruning, visualization, and per-clip NPZ saving."""+ motion_save_dir = os.path.join(output_dir, "motions")+ vis_save_dir = os.path.join(output_dir, "videos")+ tpos_save_dir = os.path.join(output_dir, "tpose")+ os.makedirs(motion_save_dir, exist_ok=True)+ os.makedirs(vis_save_dir, exist_ok=True)++ anims_list = [anim for _, anim in extracted]+ anims_list, rest_anim_pruned, names_pruned, skin_matrix_pruned = prune_skeleton_shared(+ anims_list, rest_anim_shared, list(bone_names), skin_matrix.copy(),+ consider_parent_rotate=consider_parent_rotate,+ )++ # No joint-count gate here: stage 4 (feature_extraction --min_joints /+ # --max_joints) decides which skeletons enter training, so the export+ # stays complete.+ if save_tpose:+ save_rest_pose_vis(tpos_save_dir, obj_type, rest_anim_pruned)++ for i, (action_name, _) in enumerate(extracted):+ clean_action_name = sanitize_action_name(action_name)+ clip_name = f"{obj_type}-{clean_action_name}"+ save_motion(+ os.path.join(motion_save_dir, f"{clip_name}.npz"),+ os.path.join(vis_save_dir, f"{clip_name}.mp4"),+ anims_list[i], rest_anim_pruned,+ names_pruned, skin_matrix_pruned, fps,+ action_name=clean_action_name, save_vis=save_vis,+ )++ logger.info(f"Saved {len(extracted)} clips for '{obj_type}'")+ return {obj_type: names_pruned}+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-species export+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def export_species(species, fbx_paths, output_dir, fps=30, dtype=np.float64,+ min_frames=5, consider_parent_rotate=True, save_vis=True):+ """Export one species: extract each of its per-clip FBX files, then prune+ the skeleton jointly across all clips and save them."""++ obj_type = sanitize_object_type(species)++ if is_asset_complete(output_dir, obj_type):+ logger.info(f"Species '{obj_type}' already complete, skipping.")+ return {}+ motion_save_dir = os.path.join(output_dir, "motions")+ existing = sorted(Path(motion_save_dir).glob(f"{obj_type}-*.npz")) if os.path.isdir(motion_save_dir) else []+ if existing:+ # Pruning is joint across all of a species' clips, so a partial+ # (or pre-marker) export must be redone as a whole.+ logger.warning(f"'{obj_type}' has {len(existing)} clip(s) but no completion "+ f"marker (partial or pre-marker export); re-exporting.")++ # A few species ship rig variants (different node counts or container+ # names โ e.g. KingCobra's extra Null, Monkey's A01/B01/B02 containers).+ # prune_skeleton_shared requires identical topology across its inputs, so+ # clips are grouped by joint-name signature and each group is pruned+ # separately. The dominant group defines the species' joint_names entry+ # and rest-pose PNG.+ groups = {} # names signature -> {'rest_anim', 'skin_matrix', 'extracted'}+ for fbx_path in fbx_paths:+ stem = Path(fbx_path).stem+ species_part, sep, action_name = stem.partition('-')+ if not sep:+ # Without the separator there is no action name; skipping one file+ # beats aborting the whole species on an IndexError.+ logger.warning(f"Skipping {fbx_path}: filename has no '-' separating "+ f"species from action (expected "+ f"'{{Species}}-{{Action}}.fbx').")+ continue++ try:+ result = extract_single_fbx(fbx_path, fps=fps, dtype=dtype)+ except Exception as e:+ logger.warning(f"Failed to extract {fbx_path}: {e}")+ continue++ nframes = result['anim'].positions.shape[0]+ if nframes < min_frames:+ logger.info(f"Skipping {fbx_path}: only {nframes} frames "+ f"(min={min_frames})")+ continue++ sig = tuple(result['names'])+ group = groups.setdefault(sig, {+ 'rest_anim': result['rest_anim'],+ 'names': list(result['names']),+ 'skin_matrix': result['skin_matrix'],+ 'extracted': [],+ })+ group['extracted'].append((action_name, result['anim']))++ saved = {}+ if groups:+ by_size = sorted(groups.values(), key=lambda g: len(g['extracted']),+ reverse=True)+ if len(by_size) > 1:+ logger.warning(f"'{obj_type}' has {len(by_size)} rig variants "+ f"(clip counts: {[len(g['extracted']) for g in by_size]}); "+ f"pruning each separately.")+ for i, group in enumerate(by_size):+ logger.info(f"Extracted {len(group['extracted'])} animations for "+ f"'{obj_type}' (variant {i})")+ result = postprocess_and_save(+ obj_type, group['extracted'],+ group['rest_anim'], group['names'], group['skin_matrix'],+ output_dir, fps=fps,+ consider_parent_rotate=consider_parent_rotate, save_vis=save_vis,+ save_tpose=(i == 0),+ )+ if i == 0:+ saved = result+ else:+ logger.info(f"No valid FBX files extracted for '{obj_type}'; skipping.")++ mark_asset_complete(output_dir, obj_type,+ status="exported" if saved else "skipped",+ joint_names=saved.get(obj_type))+ return saved+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Entry point+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_args():+ parser = argparse.ArgumentParser(description="Export Truebones FBX clips to NPZ motion data.")+ parser.add_argument('--data_dir', type=str, required=True,+ help='Flat directory of per-clip {Species}-{Action}.fbx files.')+ parser.add_argument('--output_dir', type=str, required=True,+ help='Output directory (motions/, videos/, tpose/ created inside).')+ parser.add_argument('--fps', type=int, default=30,+ help='Scene FPS for import and export (dataset is 30 fps).')+ parser.add_argument('--min_frames', type=int, default=5,+ help='Skip clips shorter than this many frames '+ '(shared minimum across all dataset exporters).')+ parser.add_argument('--consider_parent_rotate', action=argparse.BooleanOptionalAction, default=True,+ help="Preserve non-rotating leaves whose parent rotates during pruning "+ "(see prune_skeleton_shared; use --no-consider_parent_rotate to disable).")+ parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,+ help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++def main():+ args = parse_args()++ fbx_files = sorted(f for f in os.listdir(args.data_dir) if f.lower().endswith('.fbx'))+ by_species = {}+ for f in fbx_files:+ species = os.path.splitext(f)[0].split('-', 1)[0]+ by_species.setdefault(species, []).append(os.path.join(args.data_dir, f))+ logger.info(f"Found {len(fbx_files)} clips across {len(by_species)} species "+ f"in {args.data_dir}")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir, 'export_errors.log')+ all_joint_names = {}+ n_failed = 0+ for species, fbx_paths in tqdm(sorted(by_species.items()), desc="Exporting species"):+ try:+ saved = export_species(species, fbx_paths, output_dir=args.output_dir,+ fps=args.fps, min_frames=args.min_frames,+ consider_parent_rotate=args.consider_parent_rotate,+ save_vis=args.vis)+ if saved:+ all_joint_names.update(saved)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Failed to export species {species}")+ with open(error_log, 'a') as log_file:+ log_file.write(f"Failed to export species {species}: {e}\n")++ write_export_summary(args.output_dir, all_joint_names)+ if n_failed:+ logger.error(f"{n_failed}/{len(by_species)} species failed; see {error_log}")+ sys.exit(1)+++if __name__ == "__main__":+ # Blender exits 0 even on an uncaught exception, which hides failures from+ # `set -e` and from Slurm's afterok dependencies.+ try:+ main()+ except SystemExit:+ raise+ except Exception:+ logger.exception("export_truebones failed")+ sys.exit(1)diff --git a/data_process/motion_rendering/__init__.py b/data_process/motion_rendering/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/motion_rendering/render_mixamo.py b/data_process/motion_rendering/render_mixamo.pynew file mode 100644index 0000000..b991a34--- /dev/null+++ b/data_process/motion_rendering/render_mixamo.py@@ -0,0 +1,174 @@+"""Render multi-view frames for pre-animated Mixamo characters (EEVEE).++Consumes animated GLB/FBX files produced by+``mesh_animation/animate_mixamo.py`` (mesh + armature + baked animation)+and renders each as ``{file_stem}/v00{0..3}/*.png`` โ the layout consumed+by ``vlm_caption/caption_motion.py``.++Character meshes are kept intact (body / hair / clothes are often separate+meshes, so no main-mesh filtering is applied).++Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):+ python -m data_process.motion_rendering.render_mixamo \+ --data_dir outputs/mixamo_characters --output_dir <render_dir>+"""++import os+import sys+import argparse+from pathlib import Path++import bpy+from loguru import logger++from data_process.utils.blender_export import discover_pose_actions+from data_process.utils.blender_render import (+ RENDERABLE_EXTS,+ RENDER_FPS,+ init_render_engine,+ is_asset_render_skipped,+ is_render_complete,+ load_and_prep_asset,+ mark_asset_render_skipped,+ render_action_multiview,+)+++def render_animated_file(path, output_dir, args):+ """Render multi-view frames for one pre-animated character file."""+ save_name = Path(path).stem+ save_root = os.path.join(output_dir, save_name)++ if is_render_complete(save_root, require_video=args.video):+ logger.info(f"Skipping complete: {save_root}")+ return+ if is_asset_render_skipped(output_dir, save_name):+ logger.info(f"Previously marked as no-render, skipping: {path}")+ return++ logger.info(f"Processing: {path}")++ armature = load_and_prep_asset(str(path), fps=args.fps)+ if armature is None:+ logger.warning(f"No armature/mesh in {path}; skipping.")+ mark_asset_render_skipped(output_dir, save_name, reason="no armature/mesh")+ return++ # The animated file carries exactly one baked action: the one bound to+ # the armature. Falling back to a different action would render a motion+ # the matching NPZ never contains.+ action_names, frame_ranges = discover_pose_actions(min_frames=args.min_action_frames)+ if not action_names:+ logger.warning(f"No animation data in {path}; skipping.")+ mark_asset_render_skipped(output_dir, save_name, reason="no pose actions")+ return++ bound = None+ if armature.animation_data and armature.animation_data.action:+ bound = armature.animation_data.action.name+ if bound not in action_names:+ logger.warning(+ f"Bound action {bound!r} is not among the renderable pose actions "+ f"{action_names} for {path} (too short, or none bound); skipping "+ f"rather than rendering a different action than the export used.")+ mark_asset_render_skipped(output_dir, save_name,+ reason=f"bound action {bound!r} not renderable")+ return+ action_name = bound++ start, end = frame_ranges[action_name]+ logger.info(f"Rendering '{action_name}' -> '{save_name}' frames=[{start},{end}]")++ render_action_multiview(+ armature=armature,+ action_name=action_name,+ start_frame=start,+ end_frame=end,+ save_root=save_root,+ camera_dist=args.camera_dist,+ scene_scale=args.scale,+ resolution=args.resolution,+ max_render_frames=args.max_render_frames,+ compose_video=args.video,+ fps=args.fps,+ )+++def main():+ parser = argparse.ArgumentParser(+ description="Render multi-view frames for animated Mixamo characters (EEVEE).")+ parser.add_argument("--data_dir", type=str, required=True,+ help="Directory of animated GLB/FBX files from "+ "mesh_animation/animate_mixamo.py.")+ parser.add_argument("--output_dir", type=str, required=True,+ help="Output directory.")+ parser.add_argument("--scale", type=float, default=1.0,+ help="Scene normalization scale.")+ parser.add_argument("--camera_dist", type=float, default=1.5,+ help="Camera distance from origin.")+ parser.add_argument("--resolution", type=int, default=512,+ help="Render resolution (square).")+ parser.add_argument("--fps", type=int, default=RENDER_FPS,+ help="Scene FPS for import and preview video "+ "(must match the export stage's --fps).")+ parser.add_argument("--samples", type=int, default=64,+ help="EEVEE TAA render samples.")+ parser.add_argument("--max_render_frames", type=int, default=200,+ help="Cap on rendered frames per clip (per view).")+ parser.add_argument("--min_action_frames", type=int, default=5,+ help="Skip actions shorter than this many frames "+ "(default 5 โ the shared minimum clip length "+ "across export and rendering).")+ parser.add_argument("--video", action=argparse.BooleanOptionalAction, default=True,+ help="Compose each view's frames into v00x.mp4 "+ "(use --no-video to skip).")+ parser.add_argument("--worker_id", type=int, default=0,+ help="Worker index for sharding across workers.")+ parser.add_argument("--num_workers", type=int, default=1,+ help="Total number of workers.")+ parser.add_argument("--gpu_id", type=int, default=None,+ help="GPU device ID to pin this worker to "+ "(default: inherit CUDA_VISIBLE_DEVICES).")+ args = parser.parse_args()++ if args.gpu_id is not None:+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)++ init_render_engine(render_samples=args.samples)++ data_dir = Path(args.data_dir)+ files = sorted(p for p in data_dir.iterdir()+ if p.is_file() and p.suffix.lower() in RENDERABLE_EXTS)+ logger.info(f"Found {len(files)} animated files in {data_dir}")++ if args.num_workers > 1:+ files = files[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(files)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'render_errors_worker{args.worker_id}.log')++ n_failed = 0+ for path in files:+ try:+ render_animated_file(path, args.output_dir, args)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Failed to render {path}")+ with open(error_log, 'a') as f:+ f.write(f"{path}\t{e}\n")++ if n_failed:+ logger.error(f"{n_failed}/{len(files)} clips failed; see {error_log}")+ return 1 if n_failed else 0+++if __name__ == "__main__":+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures+ # from `set -e` and from Slurm's afterok dependencies.+ try:+ sys.exit(main())+ except Exception:+ logger.exception("render_mixamo failed")+ sys.exit(1)diff --git a/data_process/motion_rendering/render_objaverse.py b/data_process/motion_rendering/render_objaverse.pynew file mode 100644index 0000000..8b7c9d7--- /dev/null+++ b/data_process/motion_rendering/render_objaverse.py@@ -0,0 +1,241 @@+"""Render multi-view frames for Objaverse GLB/GLTF animations (EEVEE).++For each GLB, discovers all pose actions and renders each one as+``{glb_stem}-{action}/v00{0..3}/*.png`` โ the layout consumed by+``vlm_caption/caption_motion.py``.++Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):+ python -m data_process.motion_rendering.render_objaverse \+ --data_dir dataset/raw/objaverse/glb --output_dir <render_dir>+"""++import os+import sys+import argparse+from pathlib import Path++import bpy+from loguru import logger++from data_process.utils.blender_export import (+ discover_pose_actions,+ list_gltf_files,+ unique_clip_names,+)+from data_process.utils.blender_render import (+ RENDER_FPS,+ get_expected_clip_dirs,+ init_render_engine,+ is_asset_render_skipped,+ is_render_complete,+ load_and_prep_asset,+ mark_asset_render_done,+ mark_asset_render_skipped,+ render_action_multiview,+)+++def find_pending_stems(output_dir, gltf_stems, require_video=False):+ """Return the subset of GLTF stems that still need rendering.++ A stem is done only when a ``.rendered/`` marker lists the clip+ directories it is expected to produce AND every one of them passes+ :func:`is_render_complete`. Requiring the marker is what distinguishes+ "all actions rendered" from "a worker was killed partway through": the+ directories that do exist are complete in both cases, so judging by+ on-disk directories alone would retire the asset with actions still+ unrendered, permanently. Stems that are done skip the expensive GLB load.++ Action folders are named ``{stem}-{sanitized_action}``; stems may+ themselves contain ``-``, so directory names are matched against the+ known stem set by longest-prefix-at-hyphen rather than splitting on+ the first ``-``.+ """+ output_dir = Path(output_dir)+ stems_set = set(gltf_stems)+ if not output_dir.is_dir():+ return stems_set++ # Assets marked as producing no renders (no armature / actions) are done.+ stems_set = {s for s in stems_set+ if not is_asset_render_skipped(str(output_dir), s)}++ pending = set()+ for stem in stems_set:+ expected = get_expected_clip_dirs(str(output_dir), stem)+ if not expected:+ # Never finished an end-to-end pass (or predates the marker):+ # re-open it. Completed clip dirs are still skipped per-action.+ pending.add(stem)+ continue+ if not all(is_render_complete(str(output_dir / d), require_video=require_video)+ for d in expected):+ pending.add(stem)++ return pending+++def render_glb(glb_path, output_dir, args):+ """Render multi-view frames for every pose action in one GLB/GLTF file."""+ save_name = Path(glb_path).stem+ if is_asset_render_skipped(output_dir, save_name):+ logger.info(f"Previously marked as no-render, skipping: {glb_path}")+ return++ logger.info(f"Processing: {glb_path}")++ armature = load_and_prep_asset(str(glb_path), fps=args.fps)+ if armature is None:+ logger.warning(f"No armature/mesh in {glb_path}; skipping.")+ mark_asset_render_skipped(output_dir, save_name, reason="no armature/mesh")+ return++ # Names are derived from the UNFILTERED action list so they match the+ # export stage, which applies no frame-count filter โ deriving them from+ # a filtered subset would shift the collision suffixes out of step.+ all_action_names, frame_ranges = discover_pose_actions()+ clip_suffixes = unique_clip_names(all_action_names)++ action_names = [+ n for n in all_action_names+ if (frame_ranges[n][1] - frame_ranges[n][0] + 1) >= args.min_action_frames+ ]+ logger.info(f"Found {len(action_names)} pose actions "+ f"({len(all_action_names)} before the frame-count filter)")+ if not action_names:+ mark_asset_render_skipped(output_dir, save_name, reason="no pose actions")+ return++ expected_dirs = []+ for action_name in action_names:+ clean_action_name = clip_suffixes[action_name]+ clip_dir_name = f"{save_name}-{clean_action_name}"+ action_dir = os.path.join(output_dir, clip_dir_name)+ expected_dirs.append(clip_dir_name)++ if is_render_complete(action_dir, require_video=args.video):+ logger.info(f"Skipping complete: {action_dir}")+ continue++ start, end = frame_ranges[action_name]+ logger.info(f"Rendering '{action_name}' -> '{clip_dir_name}' "+ f"frames=[{start},{end}]")++ render_action_multiview(+ armature=armature,+ action_name=action_name,+ start_frame=start,+ end_frame=end,+ save_root=action_dir,+ camera_dist=args.camera_dist,+ scene_scale=args.scale,+ resolution=args.resolution,+ max_render_frames=args.max_render_frames,+ compose_video=args.video,+ fps=args.fps,+ )++ # Only now is the asset known to have had every action handled.+ mark_asset_render_done(output_dir, save_name, expected_dirs)+++def main():+ parser = argparse.ArgumentParser(+ description="Render multi-view frames for Objaverse GLB/GLTF animations (EEVEE).")+ parser.add_argument("--data_dir", type=str, required=True,+ help="Directory containing GLB/GLTF files.")+ parser.add_argument("--obj_name", type=str, default=None,+ help="Single GLB stem to render (default: all files).")+ parser.add_argument("--output_dir", type=str, required=True,+ help="Output directory.")+ parser.add_argument("--scale", type=float, default=1.0,+ help="Scene normalization scale.")+ parser.add_argument("--camera_dist", type=float, default=1.5,+ help="Camera distance from origin.")+ parser.add_argument("--resolution", type=int, default=512,+ help="Render resolution (square).")+ parser.add_argument("--fps", type=int, default=RENDER_FPS,+ help="Scene FPS for import and preview video. Must match "+ "the export stage's --fps: glTF keyframes are stored "+ "in seconds, so the importer resamples at this rate.")+ parser.add_argument("--samples", type=int, default=64,+ help="EEVEE TAA render samples.")+ parser.add_argument("--max_render_frames", type=int, default=200,+ help="Cap on rendered frames per clip (per view).")+ parser.add_argument("--min_action_frames", type=int, default=5,+ help="Skip actions shorter than this many raw frames "+ "(default 5 โ the shared minimum clip length "+ "across export and rendering; raw length >= "+ "post-T-pose-removal length, so no exported clip "+ "is ever skipped).")+ parser.add_argument("--video", action=argparse.BooleanOptionalAction, default=True,+ help="Compose each view's frames into v00x.mp4 "+ "(use --no-video to skip).")+ parser.add_argument("--missing-only", dest="missing_only", action="store_true",+ help="Scan output_dir first; only load GLBs that have "+ "at least one missing or incomplete action folder.")+ parser.add_argument("--worker_id", type=int, default=0,+ help="Worker index for sharding across workers.")+ parser.add_argument("--num_workers", type=int, default=1,+ help="Total number of workers.")+ parser.add_argument("--gpu_id", type=int, default=None,+ help="GPU device ID to pin this worker to "+ "(default: inherit CUDA_VISIBLE_DEVICES).")+ args = parser.parse_args()++ if args.gpu_id is not None:+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)++ init_render_engine(render_samples=args.samples)++ data_dir = Path(args.data_dir)+ if args.obj_name:+ glb_path = data_dir / f"{args.obj_name}.glb"+ if not glb_path.is_file():+ glb_path = data_dir / f"{args.obj_name}.gltf"+ if not glb_path.is_file():+ parser.error(f"Cannot find {args.obj_name}.glb or .gltf in {data_dir}")+ gltf_paths = [glb_path]+ else:+ gltf_paths = list_gltf_files(data_dir)+ logger.info(f"Found {len(gltf_paths)} GLB/GLTF files in {data_dir}")++ if args.missing_only:+ pending = find_pending_stems(args.output_dir, [p.stem for p in gltf_paths],+ require_video=args.video)+ before = len(gltf_paths)+ gltf_paths = [p for p in gltf_paths if p.stem in pending]+ logger.info(f"[missing-only] {len(gltf_paths)}/{before} GLBs have pending renders.")++ if args.num_workers > 1:+ gltf_paths = gltf_paths[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "+ f"{len(gltf_paths)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'render_errors_worker{args.worker_id}.log')++ n_failed = 0+ for glb_path in gltf_paths:+ try:+ render_glb(glb_path, args.output_dir, args)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Failed to render {glb_path}")+ with open(error_log, 'a') as f:+ f.write(f"{glb_path}\t{e}\n")++ if n_failed:+ logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")+ return 1 if n_failed else 0+++if __name__ == "__main__":+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures+ # from `set -e` and from Slurm's afterok dependencies.+ try:+ sys.exit(main())+ except Exception:+ logger.exception("render_objaverse failed")+ sys.exit(1)diff --git a/data_process/motion_rendering/render_tpose.py b/data_process/motion_rendering/render_tpose.pynew file mode 100644index 0000000..eb2cd58--- /dev/null+++ b/data_process/motion_rendering/render_tpose.py@@ -0,0 +1,168 @@+"""Render T-pose 2x2 grid images for rigged assets (EEVEE).++One PNG per asset, saved flat as ``{output_dir}/{name}.png`` (front/back+top row, left/right bottom row) โ the "flat" layout consumed by+``vlm_caption/classify_category.py``.++Two input layouts are auto-detected per entry of ``--data_dir``:+ - files: ``<data_dir>/*.{glb,gltf,fbx}`` -> one grid per file stem+ - subdirectory: ``<data_dir>/<obj_type>/*.fbx`` -> one grid per object+ type, rendered from its main (all-in-one) FBX โ Truebones layout.++Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):+ python -m data_process.motion_rendering.render_tpose \+ --data_dir dataset/raw/objaverse/glb --output_dir <tpose_dir>+"""++import os+import sys+import argparse+from pathlib import Path++from loguru import logger++from data_process.utils.blender_export import find_main_fbx, sanitize_object_type+from data_process.utils.blender_render import (+ RENDERABLE_EXTS,+ RENDER_FPS,+ init_render_engine,+ load_and_prep_asset,+ render_tpose_grid,+)+++def discover_assets(data_dir, species_grids=False):+ """Return (name, path) pairs to render, auto-detecting the layout.++ Files directly under *data_dir* are rendered by stem; each+ subdirectory is treated as a Truebones-style object type and rendered+ from its main FBX (fallback: first FBX in the subdirectory).++ With *species_grids*, files named ``{Species}-{Action}.*`` are grouped+ by the stem's prefix before the first ``-`` and one grid is rendered+ per species, from its first file (curated Truebones layout).+ """+ data_dir = Path(data_dir)+ assets = []+ seen_species = set()++ for p in sorted(data_dir.iterdir()):+ if p.is_file() and p.suffix.lower() in RENDERABLE_EXTS:+ if species_grids:+ species = p.stem.split('-', 1)[0]+ if species in seen_species:+ continue+ seen_species.add(species)+ assets.append((sanitize_object_type(species), p))+ else:+ assets.append((p.stem, p))+ elif p.is_dir():+ main_fbx = find_main_fbx(str(p))+ if main_fbx is None:+ fbxs = sorted(f for f in p.iterdir()+ if f.suffix.lower() == ".fbx")+ main_fbx = str(fbxs[0]) if fbxs else None+ if main_fbx is None:+ logger.warning(f"No FBX found in {p}; skipping.")+ continue+ assets.append((sanitize_object_type(p.name), Path(main_fbx)))++ return assets+++def render_asset_tpose(name, path, output_dir, args):+ """Render the T-pose grid for one asset to ``{output_dir}/{name}.png``."""+ tpose_path = os.path.join(output_dir, f"{name}.png")+ if os.path.isfile(tpose_path):+ logger.info(f"Skipping complete: {tpose_path}")+ return++ logger.info(f"Processing: {path} -> {name}")++ armature = load_and_prep_asset(str(path),+ keep_only_main_mesh=args.keep_main_mesh_only,+ fps=args.fps)+ if armature is None:+ logger.warning(f"No armature/mesh in {path}; skipping.")+ return++ render_tpose_grid(+ save_path=tpose_path,+ camera_dist=args.camera_dist,+ scene_scale=args.scale,+ resolution=args.resolution,+ )+++def main():+ parser = argparse.ArgumentParser(+ description="Render T-pose 2x2 grid images for rigged assets (EEVEE).")+ parser.add_argument("--data_dir", type=str, required=True,+ help="Directory of GLB/GLTF/FBX files, or of "+ "per-object-type FBX subdirectories (Truebones).")+ parser.add_argument("--obj_name", type=str, default=None,+ help="Single asset name (file stem or object type) "+ "to render (default: all).")+ parser.add_argument("--output_dir", type=str, required=True,+ help="Flat directory for the <name>.png grids.")+ parser.add_argument("--scale", type=float, default=1.0,+ help="Scene normalization scale.")+ parser.add_argument("--camera_dist", type=float, default=1.5,+ help="Camera distance from origin.")+ parser.add_argument("--fps", type=int, default=RENDER_FPS,+ help="Scene FPS for import (must match the export "+ "stage's --fps).")+ parser.add_argument("--resolution", type=int, default=512,+ help="Render resolution per view (square).")+ parser.add_argument("--samples", type=int, default=64,+ help="EEVEE TAA render samples.")+ parser.add_argument("--keep-all-meshes", dest="keep_main_mesh_only",+ action="store_false",+ help="Keep every mesh instead of only the "+ "highest-vertex one (multi-mesh characters).")+ parser.add_argument("--species_grids", action="store_true",+ help="Group {Species}-{Action}.fbx files by species "+ "and render one grid per species (curated "+ "Truebones layout).")+ parser.add_argument("--worker_id", type=int, default=0,+ help="Worker index for sharding across workers.")+ parser.add_argument("--num_workers", type=int, default=1,+ help="Total number of workers.")+ parser.add_argument("--gpu_id", type=int, default=None,+ help="GPU device ID to pin this worker to "+ "(default: inherit CUDA_VISIBLE_DEVICES).")+ args = parser.parse_args()++ if args.gpu_id is not None:+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)++ init_render_engine(render_samples=args.samples)++ assets = discover_assets(args.data_dir, species_grids=args.species_grids)+ logger.info(f"Found {len(assets)} assets in {args.data_dir}")++ if args.obj_name:+ assets = [(n, p) for n, p in assets if n == args.obj_name]+ if not assets:+ parser.error(f"Asset '{args.obj_name}' not found in {args.data_dir}")++ if args.num_workers > 1:+ assets = assets[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: "+ f"{len(assets)} assets")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'tpose_errors_worker{args.worker_id}.log')++ for name, path in assets:+ try:+ render_asset_tpose(name, path, args.output_dir, args)+ except Exception as e:+ logger.error(f"Failed to render {path}: {e}")+ with open(error_log, 'a') as f:+ f.write(f"{path}\t{e}\n")+++if __name__ == "__main__":+ main()diff --git a/data_process/motion_rendering/render_truebones.py b/data_process/motion_rendering/render_truebones.pynew file mode 100644index 0000000..4570e99--- /dev/null+++ b/data_process/motion_rendering/render_truebones.py@@ -0,0 +1,180 @@+"""Render multi-view frames for Truebones FBX animations (EEVEE).++Consumes the curated flat layout of the Truebones ZOO dataset: one FBX per+clip, named ``{Species}-{Action}.fbx``, all in a single directory (the same+input as ``motion_export/export_truebones.py``). Each file is rendered as+``{file_stem}/v00{0..3}/*.png`` โ the layout consumed by+``vlm_caption/caption_motion.py``.++Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):+ python -m data_process.motion_rendering.render_truebones \+ --data_dir dataset/raw/truebones/animation --output_dir <render_dir>+"""++import os+import sys+import argparse+from pathlib import Path++import bpy+from loguru import logger++from data_process.utils.blender_export import (+ discover_pose_actions,+ truebones_clip_name,+)+from data_process.utils.blender_render import (+ RENDER_FPS,+ init_render_engine,+ is_asset_render_skipped,+ is_render_complete,+ load_and_prep_asset,+ mark_asset_render_skipped,+ render_action_multiview,+)+++def render_clip_file(path, output_dir, args):+ """Render multi-view frames for one per-clip FBX file."""+ # Must match export_truebones' clip naming exactly โ stage 4 looks the+ # caption up by the NPZ stem with no fuzzy matching, so a raw stem here+ # (spaces, extra dashes) silently produces an uncaptioned clip.+ save_name = truebones_clip_name(Path(path).stem)+ save_root = os.path.join(output_dir, save_name)++ if is_render_complete(save_root, require_video=args.video):+ logger.info(f"Skipping complete: {save_root}")+ return+ if is_asset_render_skipped(output_dir, save_name):+ logger.info(f"Previously marked as no-render, skipping: {path}")+ return++ logger.info(f"Processing: {path}")++ armature = load_and_prep_asset(str(path), fps=args.fps)+ if armature is None:+ logger.warning(f"No armature/mesh in {path}; skipping.")+ mark_asset_render_skipped(output_dir, save_name, reason="no armature/mesh")+ return++ # Each curated file carries exactly one take: the action bound to the+ # armature. Export always uses that one, so falling back to a different+ # action here would render a motion the NPZ never contains.+ action_names, frame_ranges = discover_pose_actions(min_frames=args.min_action_frames)+ if not action_names:+ logger.warning(f"No animation data in {path}; skipping.")+ mark_asset_render_skipped(output_dir, save_name, reason="no pose actions")+ return++ bound = None+ if armature.animation_data and armature.animation_data.action:+ bound = armature.animation_data.action.name+ if bound not in action_names:+ logger.warning(+ f"Bound action {bound!r} is not among the renderable pose actions "+ f"{action_names} for {path} (too short, or none bound); skipping "+ f"rather than rendering a different action than the export used.")+ mark_asset_render_skipped(output_dir, save_name,+ reason=f"bound action {bound!r} not renderable")+ return+ action_name = bound++ start, end = frame_ranges[action_name]+ logger.info(f"Rendering '{action_name}' -> '{save_name}' frames=[{start},{end}]")++ render_action_multiview(+ armature=armature,+ action_name=action_name,+ start_frame=start,+ end_frame=end,+ save_root=save_root,+ camera_dist=args.camera_dist,+ scene_scale=args.scale,+ resolution=args.resolution,+ max_render_frames=args.max_render_frames,+ compose_video=args.video,+ fps=args.fps,+ )+++def main():+ parser = argparse.ArgumentParser(+ description="Render multi-view frames for Truebones FBX animations (EEVEE).")+ parser.add_argument("--data_dir", type=str, required=True,+ help="Flat directory of per-clip {Species}-{Action}.fbx files.")+ parser.add_argument("--obj_type", type=str, default=None,+ help="Single species to render (default: all).")+ parser.add_argument("--output_dir", type=str, required=True,+ help="Output directory.")+ parser.add_argument("--scale", type=float, default=1.0,+ help="Scene normalization scale.")+ parser.add_argument("--camera_dist", type=float, default=1.5,+ help="Camera distance from origin.")+ parser.add_argument("--resolution", type=int, default=512,+ help="Render resolution (square).")+ parser.add_argument("--fps", type=int, default=RENDER_FPS,+ help="Scene FPS for import and preview video "+ "(must match the export stage's --fps).")+ parser.add_argument("--samples", type=int, default=64,+ help="EEVEE TAA render samples.")+ parser.add_argument("--max_render_frames", type=int, default=200,+ help="Cap on rendered frames per clip (per view).")+ parser.add_argument("--min_action_frames", type=int, default=5,+ help="Skip actions shorter than this many frames "+ "(default 5 โ the shared minimum clip length "+ "across export and rendering).")+ parser.add_argument("--video", action=argparse.BooleanOptionalAction, default=True,+ help="Compose each view's frames into v00x.mp4 "+ "(use --no-video to skip).")+ parser.add_argument("--worker_id", type=int, default=0,+ help="Worker index for sharding across workers.")+ parser.add_argument("--num_workers", type=int, default=1,+ help="Total number of workers.")+ parser.add_argument("--gpu_id", type=int, default=None,+ help="GPU device ID to pin this worker to "+ "(default: inherit CUDA_VISIBLE_DEVICES).")+ args = parser.parse_args()++ if args.gpu_id is not None:+ os.environ["CUDA_VISIBLE_DEVICES"] = str(args.gpu_id)++ init_render_engine(render_samples=args.samples)++ data_dir = Path(args.data_dir)+ files = sorted(p for p in data_dir.iterdir()+ if p.is_file() and p.suffix.lower() == ".fbx")+ if args.obj_type:+ files = [p for p in files if p.stem.split('-', 1)[0] == args.obj_type]+ logger.info(f"Found {len(files)} clip files in {data_dir}")++ if args.num_workers > 1:+ files = files[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(files)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'render_errors_worker{args.worker_id}.log')++ n_failed = 0+ for path in files:+ try:+ render_clip_file(path, args.output_dir, args)+ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Failed to render {path}")+ with open(error_log, 'a') as f:+ f.write(f"{path}\t{e}\n")++ if n_failed:+ logger.error(f"{n_failed}/{len(files)} clips failed; see {error_log}")+ return 1 if n_failed else 0+++if __name__ == "__main__":+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures+ # from `set -e` and from Slurm's afterok dependencies.+ try:+ sys.exit(main())+ except Exception:+ logger.exception("render_truebones failed")+ sys.exit(1)diff --git a/data_process/scripts/_common.sh b/data_process/scripts/_common.shnew file mode 100644index 0000000..77c22fd--- /dev/null+++ b/data_process/scripts/_common.sh@@ -0,0 +1,84 @@+# shellcheck shell=bash+# Shared prologue for every data_process/scripts/run_*.sh wrapper.+#+# source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"+#+# Resolves PROJECT_ROOT (the repository root), cds into it, activates the+# conda environment named by $CONDA_ENV (default: unimate), and defines the+# default dataset layout used by every stage. Override any DEFAULT_* path by+# exporting the corresponding variable before calling a wrapper.++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"+# Resolve the sourcing wrapper's own path *before* the cd below, so that+# print_usage still finds it when the wrapper was invoked by a relative path+# (e.g. `cd data_process/scripts && bash run_export.sh -h`).+WRAPPER_PATH="${BASH_SOURCE[1]:-$0}"+[[ "$WRAPPER_PATH" == /* ]] || WRAPPER_PATH="$PWD/$WRAPPER_PATH"+cd "$PROJECT_ROOT"++CONDA_ENV=${CONDA_ENV:-unimate}+if command -v conda >/dev/null 2>&1; then+ # conda activation hooks reference unset variables; relax `set -u` around them.+ case $- in *u*) _had_nounset=1 ;; *) _had_nounset=0 ;; esac+ set +u+ eval "$(conda shell.bash hook 2>/dev/null)"+ conda activate "$CONDA_ENV" 2>/dev/null \+ || echo "warning: could not activate conda env '$CONDA_ENV'; using current python" >&2+ [[ "$_had_nounset" == 1 ]] && set -u+fi++# Avoid user startup hooks leaking into Blender / worker processes.+unset PYTHONSTARTUP PYTHONBREAKPOINT++# โโ Default dataset layout (relative to PROJECT_ROOT) โโโโโโโโโโโโโโโโโโโโโโโ+# dataset/raw/<dataset>/ raw FBX / GLB assets+# dataset/export/<dataset>/ stage-1 export + stage-2/3 metadata+# dataset/render/<dataset>/ multi-view renders (captioning input)+# dataset/render/<dataset>_tpose/ T-pose grids (category input)+# dataset/features/<dataset>/ stage-4 training clips + cond.npy+export_dir() { echo "dataset/export/$1"; }+render_dir() { echo "dataset/render/$1"; }+tpose_dir() { echo "dataset/render/${1}_tpose"; }+features_dir() { echo "dataset/features/$1"; }++# โโ Helpers โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# require_dataset <name> : validate a dataset name.+require_dataset() {+ case "$1" in+ truebones|mixamo|objaverse) ;;+ *) echo "Unknown dataset '$1' (expected: truebones | mixamo | objaverse)" >&2; exit 2 ;;+ esac+}++# require_env VAR : abort if the environment variable is unset/empty.+require_env() {+ if [[ -z "${!1:-}" ]]; then+ echo "ERROR: $1 is not set. Export it in your shell." >&2+ exit 1+ fi+}++# print_usage [script] : print the leading comment block of a wrapper+# (defaults to the wrapper that sourced this file).+print_usage() {+ sed -n '2,/^$/p' "${1:-$WRAPPER_PATH}" | sed 's/^# \{0,1\}//'+}++# handle_help "$@" : print the wrapper usage and exit 0 when the first+# argument is -h/--help. Call it right after sourcing this file, before any+# other argument validation, so `-h` works on every wrapper.+handle_help() {+ case "${1:-}" in+ -h|--help) print_usage; exit 0 ;;+ esac+}++# worker_log_dir <name> : create and echo a directory for per-worker logs, so+# that a failing shard of a parallel run can be attributed. Override with+# WORKER_LOG_DIR.+worker_log_dir() {+ local dir="${WORKER_LOG_DIR:-outputs/logs/$1}"+ mkdir -p "$dir"+ echo "$dir"+}diff --git a/data_process/scripts/run_animate_fbx.sh b/data_process/scripts/run_animate_fbx.shnew file mode 100755index 0000000..a0c2fac--- /dev/null+++ b/data_process/scripts/run_animate_fbx.sh@@ -0,0 +1,43 @@+#!/bin/bash+# Stage 5 โ bake raw animation FBX/GLB clips onto a chosen rigged character+# (action transfer by bone name โ made for Mixamo-style shared skeletons,+# where any clip can drive any compatible character without remapping).+#+# Usage:+# bash data_process/scripts/run_animate_fbx.sh # Y Bot x animation_motion/+# CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \+# bash data_process/scripts/run_animate_fbx.sh # whole library on Amy+# ANIM_PATH=dataset/raw/mixamo/animation_motion/Jab_Cross.fbx \+# CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \+# bash data_process/scripts/run_animate_fbx.sh # a single clip+#+# Env overrides:+# CHAR_PATH rigged character FBX/GLB (default: character_refined/Y_Bot.fbx)+# ANIM_PATH animation clip or directory of clips+# (default: dataset/raw/mixamo/animation_motion)+# OUTPUT_DIR destination directory (default: outputs/animated_<character stem>)+# CHAR_ANIM_TYPE glb|fbx (default: fbx)+#+# Extra arguments are passed through to animate_fbx.py (e.g. --overwrite).++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++CHAR_PATH=${CHAR_PATH:-dataset/raw/mixamo/character_refined/Y_Bot.fbx}+if [[ ! -f "$CHAR_PATH" ]]; then+ echo "Character file not found: $CHAR_PATH" >&2+ echo "Download the mixamo raw data (run_download.sh mixamo) or set CHAR_PATH." >&2+ exit 1+fi+ANIM_PATH=${ANIM_PATH:-dataset/raw/mixamo/animation_motion}+CHAR_STEM=$(basename "${CHAR_PATH%.*}")+OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated_${CHAR_STEM}}+CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-fbx}++blender -b -P data_process/mesh_animation/animate_fbx.py -- \+ --char_path="$CHAR_PATH" \+ --anim_path="$ANIM_PATH" \+ --output_dir="$OUTPUT_DIR" \+ --char_anim_type="$CHAR_ANIM_TYPE" "$@"diff --git a/data_process/scripts/run_animate_mixamo.sh b/data_process/scripts/run_animate_mixamo.shnew file mode 100755index 0000000..7f69a6c--- /dev/null+++ b/data_process/scripts/run_animate_mixamo.sh@@ -0,0 +1,36 @@+#!/bin/bash+# Stage 5 โ batch-animate a Mixamo character mesh with a directory of exported+# motion NPZs (raw Mixamo animations carry no mesh, so a character FBX is required).+#+# Usage:+# bash data_process/scripts/run_animate_mixamo.sh # default character: Y Bot+# CHAR_PATH=char.fbx bash data_process/scripts/run_animate_mixamo.sh+#+# Env overrides:+# CHAR_PATH rigged Mixamo character FBX (default: Y_Bot โ the character+# whose skeleton the animation FBXs are authored on)+# NPZ_DIR (default: dataset/export/mixamo/motions)+# OUTPUT_DIR (default: outputs/mixamo_characters)+# CHAR_ANIM_TYPE glb|fbx (default: fbx)++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DEFAULT_CHAR=dataset/raw/mixamo/character_refined/Y_Bot.fbx+CHAR_PATH=${CHAR_PATH:-$DEFAULT_CHAR}+if [[ ! -f "$CHAR_PATH" ]]; then+ echo "Character FBX not found: $CHAR_PATH" >&2+ echo "Download the mixamo raw data (run_download.sh mixamo) or set CHAR_PATH." >&2+ exit 1+fi+NPZ_DIR=${NPZ_DIR:-$(export_dir mixamo)/motions}+OUTPUT_DIR=${OUTPUT_DIR:-outputs/mixamo_characters}+CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-fbx}++blender -b -P data_process/mesh_animation/animate_mixamo.py -- \+ --char_path="$CHAR_PATH" \+ --npz_dir="$NPZ_DIR" \+ --output_dir="$OUTPUT_DIR" \+ --char_anim_type="$CHAR_ANIM_TYPE" "$@"diff --git a/data_process/scripts/run_animate_motion.sh b/data_process/scripts/run_animate_motion.shnew file mode 100755index 0000000..295de1d--- /dev/null+++ b/data_process/scripts/run_animate_motion.sh@@ -0,0 +1,45 @@+#!/bin/bash+# Stage 5 โ drive a rigged character with a canonicalized motion clip (stage-4+# format, or a generated motion in that format) and export GLB + FBX. The+# character is auto-resolved from the dataset + clip name for truebones and+# objaverse; mixamo needs CHAR_PATH.+#+# Usage:+# ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse+# ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh truebones+# ANIM_PATH=clip.npz CHAR_PATH=char.fbx bash data_process/scripts/run_animate_motion.sh mixamo+#+# Env overrides:+# ANIM_PATH (required) motion .npz (or legacy .npy)+# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; required for mixamo)+# COND_PATH cond.npy (default: dataset/features/<dataset>/cond.npy)+# OUTPUT_DIR (default: outputs/animated)+# ANIM_MODE fk|ik, legacy .npy only (default: fk)++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_animate_motion.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion .npz/.npy for dataset=$DATASET}+OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}+ANIM_MODE=${ANIM_MODE:-fk}++EXTRA_ARGS=()+[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")+[[ -n "${COND_PATH:-}" ]] && EXTRA_ARGS+=(--cond_path="$COND_PATH")+if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then+ echo "dataset=mixamo requires CHAR_PATH (no auto-resolved character mesh)" >&2+ exit 1+fi++blender -b -P data_process/mesh_animation/animate_motion.py -- \+ --dataset_type="$DATASET" \+ --anim_path="$ANIM_PATH" \+ --output_dir="$OUTPUT_DIR" \+ --anim_mode="$ANIM_MODE" \+ "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_animate_npz.sh b/data_process/scripts/run_animate_npz.shnew file mode 100755index 0000000..6fe9f9b--- /dev/null+++ b/data_process/scripts/run_animate_npz.sh@@ -0,0 +1,28 @@+#!/bin/bash+# Stage 5 โ drive a rigged character with an *exported* motion NPZ (stage-1+# format) and save the animated character as GLB/FBX.+#+# Usage:+# CHAR_PATH=char.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh+#+# Env overrides:+# CHAR_PATH (required โ rigged character GLB/FBX matching the NPZ skeleton)+# ANIM_PATH (required โ NPZ whose skeleton matches the rig)+# OUTPUT_DIR (default: outputs/animated)+# CHAR_ANIM_TYPE glb|fbx (default: glb)++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++CHAR_PATH=${CHAR_PATH:?Set CHAR_PATH to a rigged character GLB/FBX}+ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion NPZ matching the rig}+OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}+CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-glb}++blender -b -P data_process/mesh_animation/animate_npz.py -- \+ --char_path="$CHAR_PATH" \+ --anim_path="$ANIM_PATH" \+ --output_dir="$OUTPUT_DIR" \+ --char_anim_type="$CHAR_ANIM_TYPE" "$@"diff --git a/data_process/scripts/run_caption_category.sh b/data_process/scripts/run_caption_category.shnew file mode 100755index 0000000..1a8269b--- /dev/null+++ b/data_process/scripts/run_caption_category.sh@@ -0,0 +1,45 @@+#!/bin/bash+# Stage 2 โ classify assets into body-plan categories with a local Qwen3-VL model.+# Single-GPU only โ one classification per object, no sharding.+#+# Usage:+# bash data_process/scripts/run_caption_category.sh objaverse+# MODEL=Qwen/Qwen3-VL-2B-Instruct bash data_process/scripts/run_caption_category.sh truebones+#+# Env overrides:+# MODEL, RENDER_ROOT (default: dataset/render/<dataset>_tpose),+# CATEGORY_GROUPS_JSON (default: dataset/export/<dataset>/category_groups.json),+# MAX_TOKENS+# Extra arguments are passed through to classify_category.py.+# On offline compute nodes with a populated HuggingFace cache, export+# HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"+DATASET=${1:?Usage: run_caption_category.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++RENDER_ROOT=${RENDER_ROOT:-$(tpose_dir "$DATASET")}+CATEGORY_GROUPS_JSON=${CATEGORY_GROUPS_JSON:-$(export_dir "$DATASET")/category_groups.json}+if [[ ! -d "$RENDER_ROOT" ]]; then+ echo "RENDER_ROOT does not exist: $RENDER_ROOT" >&2+ exit 2+fi+mkdir -p "$(dirname "$CATEGORY_GROUPS_JSON")"++# Local Qwen3-VL only. Sizes: Qwen/Qwen3-VL-{2B,8B,32B,72B}-Instruct.+MODEL=${MODEL:-Qwen/Qwen3-VL-8B-Instruct}+MAX_TOKENS=${MAX_TOKENS:-10}++echo "Dataset: $DATASET | Model: $MODEL"+echo "Render root: $RENDER_ROOT"+echo "Output JSON: $CATEGORY_GROUPS_JSON"++python -m data_process.vlm_caption.classify_category \+ --render_root="$RENDER_ROOT" \+ --model="$MODEL" \+ --max_tokens="$MAX_TOKENS" \+ --category_groups_json="$CATEGORY_GROUPS_JSON" "$@"diff --git a/data_process/scripts/run_caption_motion.sh b/data_process/scripts/run_caption_motion.shnew file mode 100755index 0000000..f2dca46--- /dev/null+++ b/data_process/scripts/run_caption_motion.sh@@ -0,0 +1,156 @@+#!/bin/bash+# Stage 2 โ caption multi-view motion renders with a VLM.+# The backend is picked from MODEL, with the same rules as detect_backend() in+# vlm_caption/backends.py:+# *qwen* (default, any case) -> local Qwen3-VL (GPU-bound; supports --multi-gpu)+# gemini* -> Gemini API (needs GOOGLE_API_KEY)+# anything else -> OpenAI-compatible API (needs OPENAI_API_KEY)+#+# Usage:+# bash data_process/scripts/run_caption_motion.sh objaverse # local Qwen3-VL+# bash data_process/scripts/run_caption_motion.sh truebones --multi-gpu # all visible GPUs (local backend only)+# MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse+# MODEL=gpt-5-mini NUM_WORKERS=16 bash data_process/scripts/run_caption_motion.sh mixamo+#+# Env overrides:+# MODEL, RENDER_ROOT (default: dataset/render/<dataset>),+# OUTPUT_JSON (default: dataset/export/<dataset>/motion_captions.json),+# DOWNSAMPLE_RATE, MAX_TOKENS, NUM_WORKERS, MAX_RETRIES, BASE_URL (OpenAI-compatible proxy),+# THINKING_LEVEL / MEDIA_RESOLUTION (gemini-3), THINKING_BUDGET (gemini-2.5)+# Unrecognized arguments are passed through to caption_motion.py.+# On offline compute nodes with a populated HuggingFace cache, export+# HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++# โโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+handle_help "$@"+DATASET=${1:?Usage: run_caption_motion.sh <truebones|mixamo|objaverse> [--multi-gpu] [extra args...]}+shift+require_dataset "$DATASET"++MODE=single+PASSTHRU=()+while [[ $# -gt 0 ]]; do+ case "$1" in+ --multi-gpu) MODE=multi-gpu; shift ;;+ -h|--help) print_usage; exit 0 ;;+ # Everything else (e.g. --views v000 v002, --downsample_rate 4) goes to+ # the Python entry point, which owns the full flag set.+ *) PASSTHRU+=("$1"); shift ;;+ esac+done++# โโ Backend from MODEL (mirrors vlm_caption/backends.py detect_backend) โ+MODEL=${MODEL:-Qwen/Qwen3-VL-8B-Instruct}+if [[ "${MODEL,,}" == *qwen* ]]; then+ backend=local # backends.py: BACKEND_QWEN+elif [[ "$MODEL" == gemini* ]]; then+ backend=gemini+else+ backend=openai+fi++if [[ "$MODE" == "multi-gpu" && "$backend" != "local" ]]; then+ echo "ERROR: --multi-gpu only applies to the local Qwen backend "\+"(API backends scale via NUM_WORKERS instead)." >&2+ exit 2+fi+[[ "$backend" == "gemini" ]] && require_env GOOGLE_API_KEY+[[ "$backend" == "openai" ]] && require_env OPENAI_API_KEY++# โโ Paths โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+RENDER_ROOT=${RENDER_ROOT:-$(render_dir "$DATASET")}+OUTPUT_JSON=${OUTPUT_JSON:-$(export_dir "$DATASET")/motion_captions.json}+if [[ ! -d "$RENDER_ROOT" ]]; then+ echo "RENDER_ROOT does not exist: $RENDER_ROOT" >&2+ exit 2+fi+mkdir -p "$(dirname "$OUTPUT_JSON")"++# โโ Backend-specific settings โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Full 30 fps for every dataset: temporal continuity is what lets the model+# track heading turns and posture on appearance-poor subjects (measured on+# mixamo: 180-turn and sit/stand errors gone at 1x vs 2x downsampling).+DOWNSAMPLE_RATE=${DOWNSAMPLE_RATE:-1}+MAX_RETRIES=${MAX_RETRIES:-4}+EXTRA=()+case "$backend" in+ local)+ MAX_TOKENS=${MAX_TOKENS:-800} ;;+ gemini)+ # Reasoning/answer tokens share one budget -> generous default.+ # Default 2 workers โ token/minute quotas are easy to exhaust with+ # many concurrent image-heavy requests.+ MAX_TOKENS=${MAX_TOKENS:-1024}+ NUM_WORKERS=${NUM_WORKERS:-2}+ EXTRA+=(--thinking_level="${THINKING_LEVEL:-low}"+ --media_resolution="${MEDIA_RESOLUTION:-medium}"+ --thinking_budget="${THINKING_BUDGET:-0}"+ --max_retries="$MAX_RETRIES"+ --num_workers="$NUM_WORKERS") ;;+ openai)+ # GPT-5 reasoning tokens count against max_completion_tokens.+ MAX_TOKENS=${MAX_TOKENS:-1024}+ NUM_WORKERS=${NUM_WORKERS:-4}+ EXTRA+=(--max_retries="$MAX_RETRIES" --num_workers="$NUM_WORKERS")+ [[ -n "${BASE_URL:-}" ]] && EXTRA+=(--base_url="$BASE_URL") ;;+esac++# Mixamo ships official catalogue labels and truebones the T2M4LVO+# annotations; pass them as verify-against-frames hints+# (HINTS_JSON overrides, HINTS_JSON="" disables).+case "$DATASET" in+ mixamo) HINTS_JSON=${HINTS_JSON-dataset/raw/mixamo/animation_motion_prompts.json} ;;+ truebones) HINTS_JSON=${HINTS_JSON-dataset/raw/truebones/animation_prompts.json} ;;+ *) HINTS_JSON=${HINTS_JSON:-} ;;+esac++COMMON=(--task="$DATASET" --render_root="$RENDER_ROOT" --model="$MODEL"+ --downsample_rate="$DOWNSAMPLE_RATE" --max_tokens="$MAX_TOKENS"+ --output_json="$OUTPUT_JSON")+[[ -n "$HINTS_JSON" ]] && COMMON+=(--hints_json="$HINTS_JSON")++echo "Dataset: $DATASET | Backend: $backend | Model: $MODEL | Mode: $MODE"+[[ -n "$HINTS_JSON" ]] && echo "Caption hints: $HINTS_JSON"+echo "Render root: $RENDER_ROOT"+echo "Output JSON: $OUTPUT_JSON"++# โโ Run โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+if [[ "$MODE" == "multi-gpu" ]]; then+ num_gpus=$(python -c "import torch; print(torch.cuda.device_count())")+ if ! [[ "$num_gpus" =~ ^[0-9]+$ ]] || [[ "$num_gpus" -lt 1 ]]; then+ echo "No visible CUDA devices (num_gpus=$num_gpus). Aborting --multi-gpu." >&2+ exit 1+ fi+ LOG_DIR=$(worker_log_dir "caption-motion-$DATASET")+ echo "Launching $num_gpus GPU processes (per-GPU logs: $LOG_DIR)..."++ pids=()+ for gpu_id in $(seq 0 $((num_gpus - 1))); do+ python -m data_process.vlm_caption.caption_motion "${COMMON[@]}" \+ --num_gpus="$num_gpus" --gpu_id="$gpu_id" \+ "${PASSTHRU[@]}" > >(tee "$LOG_DIR/gpu${gpu_id}.log") 2>&1 &+ pids+=($!)+ echo " Started GPU $gpu_id (PID ${pids[-1]})"+ done++ failed=0+ for gpu_id in "${!pids[@]}"; do+ wait "${pids[gpu_id]}" \+ || { echo "GPU $gpu_id failed (log: $LOG_DIR/gpu${gpu_id}.log)" >&2; failed=1; }+ done+ if [[ $failed -ne 0 ]]; then+ echo "Some processes failed. Shard files preserved for inspection." >&2+ exit 1+ fi++ echo "Merging shards..."+ python -m data_process.vlm_caption.caption_motion \+ --task="$DATASET" --merge_shards --output_json="$OUTPUT_JSON" --num_gpus="$num_gpus"+ echo "Done! Output: $OUTPUT_JSON"+else+ python -m data_process.vlm_caption.caption_motion \+ "${COMMON[@]}" "${EXTRA[@]}" "${PASSTHRU[@]}"+fidiff --git a/data_process/scripts/run_download.sh b/data_process/scripts/run_download.shnew file mode 100755index 0000000..ccc9aa2--- /dev/null+++ b/data_process/scripts/run_download.sh@@ -0,0 +1,67 @@+#!/bin/bash+# Stage 0 โ download raw source assets from the Hugging Face Hub into+# dataset/raw/<dataset>/, the layout every downstream wrapper expects.+#+# Usage:+# bash data_process/scripts/run_download.sh mixamo # animations + rigged characters+# bash data_process/scripts/run_download.sh objaverse # rigged + animated GLBs+# bash data_process/scripts/run_download.sh objaverse --include 'glb/*' # skip metadata+#+# Sources:+# mixamo https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters+# objaverse https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated+# truebones https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations+#+# The truebones repo carries annotations only (prompts, per-clip metadata,+# T-pose renders, build scripts) โ the Truebones ZOO animal pack itself is a+# commercial product that we cannot redistribute. Purchase it from Truebones+# (https://truebones.com), unpack it to dataset/raw/truebones/Truebone_Z-OO/+# (one folder per animal), and rebuild the per-clip animation/ layout with the+# downloaded scripts/pipeline/ (see the repo's README).+#+# Needs the `hf` CLI (https://hf.co/cli); gated/private repos also need+# `hf auth login` or HF_TOKEN.+#+# Env overrides: RAW_DIR (target directory), TRUEBONES_REPO+# Extra arguments are passed through to `hf download`.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"+DATASET=${1:-}+[[ -n "$DATASET" ]] || { print_usage; exit 0; }+require_dataset "$DATASET"+shift++if ! command -v hf >/dev/null 2>&1; then+ echo "ERROR: the 'hf' CLI is not installed. Install it with:" >&2+ echo " curl -LsSf https://hf.co/cli/install.sh | bash -s" >&2+ exit 1+fi++case "$DATASET" in+ mixamo) REPO=Linzhan/Mixamo-Animations-Characters ;;+ objaverse) REPO=Linzhan/Objaverse-XL-Rigged-Animated ;;+ truebones)+ REPO=${TRUEBONES_REPO:-Linzhan/Truebones-ZOO-Annotations}+ cat >&2 <<'EOF'+NOTE: Truebones ZOO is a commercial asset pack and is not redistributed; this+ downloads annotations only (prompts, metadata, renders, build scripts).+ To obtain the motion data, purchase it from https://truebones.com, unpack+ it to dataset/raw/truebones/Truebone_Z-OO/ (one folder per animal), and+ rebuild animation/ with the downloaded scripts/pipeline/.+EOF+ ;;+esac+RAW_DIR=${RAW_DIR:-dataset/raw/$DATASET}++echo "Downloading $REPO -> $RAW_DIR"+if ! hf download "$REPO" --type dataset --local-dir "$RAW_DIR" "$@"; then+ if [[ "$DATASET" == "truebones" ]]; then+ echo "Download failed. Note that only annotations are hosted; the motion files" >&2+ echo "are commercial: purchase the Truebones ZOO pack from https://truebones.com" >&2+ echo "and unpack it to dataset/raw/truebones/Truebone_Z-OO/<Animal>/..." >&2+ fi+ exit 1+fidiff --git a/data_process/scripts/run_export.sh b/data_process/scripts/run_export.shnew file mode 100755index 0000000..352a167--- /dev/null+++ b/data_process/scripts/run_export.sh@@ -0,0 +1,126 @@+#!/bin/bash+# Stage 1 โ export raw rigged assets (GLB/GLTF and FBX) to NPZ motion data.+#+# Usage:+# bash data_process/scripts/run_export.sh truebones # dataset preset (flat {Species}-{Action}.fbx clips)+# bash data_process/scripts/run_export.sh mixamo # dataset preset (animation-only FBX)+# bash data_process/scripts/run_export.sh objaverse # dataset preset (GLB/GLTF)+# bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # parallel (mixamo too)+# bash data_process/scripts/run_export.sh mixamo --multi-worker 8 --no-vis # skip MP4 previews+# DATA_DIR=my_assets bash data_process/scripts/run_export.sh # auto: .glb/.gltf and .fbx side by side+#+# --multi-worker only applies to a *directory* input; truebones is exported by+# a single-process exporter and ignores it (with a warning).+#+# Auto mode runs export_general.py on the directory: GLB/GLTF and FBX side+# by side, importer picked per file by extension, mesh optional (armature-only+# FBX works too).+#+# Env overrides: DATA_DIR, OUTPUT_DIR+# Extra arguments are passed through to the exporter.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++EXPORTERS=data_process/motion_export++# โโ Parse mode + flags โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+handle_help "$@"++MODE=auto+case "${1:-}" in+ truebones|mixamo|objaverse|auto) MODE="$1"; shift ;;+esac++NUM_WORKERS=1+EXTRA_ARGS=()+while [[ $# -gt 0 ]]; do+ case "$1" in+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ *) EXTRA_ARGS+=("$1"); shift ;;+ esac+done++# โโ Per-mode defaults โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+case "$MODE" in+ truebones)+ DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation}+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir truebones)} ;;+ mixamo)+ DATA_DIR=${DATA_DIR:-dataset/raw/mixamo/animation_motion}+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir mixamo)} ;;+ objaverse)+ DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb}+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir objaverse)} ;;+ auto)+ DATA_DIR=${DATA_DIR:?auto mode requires DATA_DIR (directory of .glb/.gltf/.fbx assets)}+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir custom)} ;;+esac++# โโ Exporter invocations โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# launch_workers <script> <input flag>: run one exporter, sharding over+# NUM_WORKERS parallel Blender processes and merging the per-worker summary+# JSONs afterwards. Only a directory input is sharded โ the exporters write+# unsuffixed canonical summaries for a single file, which N workers would+# concurrently overwrite.+launch_workers() {+ local script="$1" input_flag="$2"+ if [[ "$NUM_WORKERS" -gt 1 && -d "$DATA_DIR" ]]; then+ local log_dir+ log_dir=$(worker_log_dir "export-$MODE")+ echo "Launching $NUM_WORKERS export workers (per-worker logs: $log_dir)..."+ local pids=() worker_id failed=0+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do+ blender -b -P "$EXPORTERS/$script" -- \+ "$input_flag=$DATA_DIR" --output_dir="$OUTPUT_DIR" \+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \+ "${EXTRA_ARGS[@]}" > >(tee "$log_dir/worker${worker_id}.log") 2>&1 &+ pids+=($!)+ echo " Started worker $worker_id (PID ${pids[-1]})"+ done++ # A bare `wait` always returns 0 โ check every worker individually.+ # Merging a truncated set of shards would silently shrink the+ # canonical JSONs and then delete the shards, so never merge on failure.+ for worker_id in "${!pids[@]}"; do+ wait "${pids[worker_id]}" \+ || { echo "Worker $worker_id failed (log: $log_dir/worker${worker_id}.log)" >&2; failed=1; }+ done+ if [[ $failed -ne 0 ]]; then+ echo "Some export workers failed โ NOT merging summary JSONs." >&2+ echo "Per-worker shards are preserved in $OUTPUT_DIR; rerun to finish," >&2+ echo "then merge with: python -m data_process.tools.merge_summaries --output_dir=$OUTPUT_DIR" >&2+ exit 1+ fi+ echo "All workers finished. Merging summary JSONs..."+ python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"+ else+ if [[ "$NUM_WORKERS" -gt 1 ]]; then+ echo "WARNING: --multi-worker is ignored for a single-file input" \+ "($DATA_DIR); running one exporter process." >&2+ fi+ blender -b -P "$EXPORTERS/$script" -- \+ "$input_flag=$DATA_DIR" --output_dir="$OUTPUT_DIR" "${EXTRA_ARGS[@]}"+ fi+}++export_glb() { launch_workers export_objaverse.py --data_dir; }+export_mixamo() { launch_workers export_mixamo.py --anim_dir; }++export_fbx() {+ if [[ "$NUM_WORKERS" -gt 1 ]]; then+ echo "WARNING: --multi-worker is ignored for truebones โ export_truebones.py" \+ "is single-process; running one exporter process." >&2+ fi+ blender -b -P "$EXPORTERS/export_truebones.py" -- \+ --data_dir="$DATA_DIR" --output_dir="$OUTPUT_DIR" "$@" "${EXTRA_ARGS[@]}"+}++# โโ Dispatch โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+echo "Mode: $MODE | Input: $DATA_DIR | Output: $OUTPUT_DIR"+case "$MODE" in+ objaverse) export_glb ;;+ mixamo) export_mixamo ;;+ truebones) export_fbx ;;+ auto) launch_workers export_general.py --input ;;+esacdiff --git a/data_process/scripts/run_export_general.sh b/data_process/scripts/run_export_general.shnew file mode 100644index 0000000..0ff2602--- /dev/null+++ b/data_process/scripts/run_export_general.sh@@ -0,0 +1,64 @@+#!/bin/bash+# Stage 1 โ export rigged GLB/GLTF/FBX assets to NPZ motion data.+# Input is a single file or a directory of assets (mixed formats welcome).+# Mesh optional (armature-only FBX works); every pose action becomes a clip.+#+# Usage:+# bash data_process/scripts/run_export_general.sh <asset.glb|.fbx|dir> [extra args...]+# bash data_process/scripts/run_export_general.sh my_assets/ --multi-worker 8 --no-vis+# bash data_process/scripts/run_export_general.sh model.fbx --no-prune+#+# Env overrides: OUTPUT_DIR (default: dataset/export/custom)++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"+INPUT="${1:?Usage: run_export_general.sh <asset.glb|.fbx|dir> [extra args...]}"+shift+OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir custom)}++NUM_WORKERS=1+EXTRA_ARGS=()+while [[ $# -gt 0 ]]; do+ case "$1" in+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ *) EXTRA_ARGS+=("$1"); shift ;;+ esac+done++if [[ "$NUM_WORKERS" -gt 1 && -d "$INPUT" ]]; then+ LOG_DIR=$(worker_log_dir "export-general")+ echo "Launching $NUM_WORKERS export workers (per-worker logs: $LOG_DIR)..."+ pids=()+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do+ blender -b -P data_process/motion_export/export_general.py -- \+ --input="$INPUT" --output_dir="$OUTPUT_DIR" \+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \+ "${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &+ pids+=($!)+ echo " Started worker $worker_id (PID ${pids[-1]})"+ done++ # A bare `wait` always returns 0 โ check every worker individually, and+ # never merge a truncated set of shards (merging deletes them).+ failed=0+ for worker_id in "${!pids[@]}"; do+ wait "${pids[worker_id]}" \+ || { echo "Worker $worker_id failed (log: $LOG_DIR/worker${worker_id}.log)" >&2; failed=1; }+ done+ if [[ $failed -ne 0 ]]; then+ echo "Some export workers failed โ NOT merging summary JSONs." >&2+ echo "Per-worker shards are preserved in $OUTPUT_DIR; rerun to finish." >&2+ exit 1+ fi+ echo "All workers finished. Merging summary JSONs..."+ python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"+else+ if [[ "$NUM_WORKERS" -gt 1 ]]; then+ echo "WARNING: --multi-worker is ignored for a single-file input" \+ "($INPUT); running one exporter process." >&2+ fi+ blender -b -P data_process/motion_export/export_general.py -- \+ --input="$INPUT" --output_dir="$OUTPUT_DIR" "${EXTRA_ARGS[@]}"+fidiff --git a/data_process/scripts/run_extract_features.sh b/data_process/scripts/run_extract_features.shnew file mode 100755index 0000000..021d783--- /dev/null+++ b/data_process/scripts/run_extract_features.sh@@ -0,0 +1,51 @@+#!/bin/bash+# Stage 4 โ process exported NPZ motions into canonicalized training clips.+#+# Usage:+# bash data_process/scripts/run_extract_features.sh truebones+# bash data_process/scripts/run_extract_features.sh objaverse+# bash data_process/scripts/run_extract_features.sh mixamo+# APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones+# NUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse+#+# Env overrides:+# DATA_DIR Stage-1 export directory (default: dataset/export/<dataset>)+# SAVE_DIR Output directory (default: dataset/features/<dataset>)+# APPLY_CLIP If 1/true, crop long motions into overlapping fixed-length+# clips; otherwise (default) keep only the first max_clip_len frames.+# NUM_WORKERS Parallel worker processes over object types (default: 1)+# NO_VIS If 1/true, skip the per-clip MP4 previews+# Extra arguments are passed through to extract_features.py.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_extract_features.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++DATA_DIR=${DATA_DIR:-$(export_dir "$DATASET")}+SAVE_DIR=${SAVE_DIR:-$(features_dir "$DATASET")}++EXTRA_ARGS=()+# Per-dataset joint-count range (skeletons outside it are skipped here, not+# at export โ the exports stay complete). Defaults in extract_features.py+# are 8..150; objaverse keeps its wider historical range.+case "$DATASET" in+ objaverse) EXTRA_ARGS+=(--min_joints 4 --max_joints 180) ;;+esac+case "${APPLY_CLIP:-0}" in+ 1|true|TRUE) EXTRA_ARGS+=(--apply_clip) ;;+esac+case "${NO_VIS:-0}" in+ 1|true|TRUE) EXTRA_ARGS+=(--no-vis) ;;+esac+[[ "${NUM_WORKERS:-1}" -gt 1 ]] && EXTRA_ARGS+=(--num_workers="$NUM_WORKERS")++python -m data_process.feature_extraction.extract_features \+ --dataset_type="$DATASET" \+ --data_dir="$DATA_DIR" \+ --save_dir="$SAVE_DIR" \+ "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_joints_face_correct_llm.sh b/data_process/scripts/run_joints_face_correct_llm.shnew file mode 100755index 0000000..7cca9b0--- /dev/null+++ b/data_process/scripts/run_joints_face_correct_llm.sh@@ -0,0 +1,43 @@+#!/bin/bash+# Stage 3 โ re-run LLM face-joint selection on rigs whose source == "empty".+#+# For each empty entry in face_joint_names.json the LLM is asked again to+# pick a bilateral pair (or body-axis endpoints). Successful picks overwrite+# the entry in place (a *.bak copy is made on first run); rigs that stay+# empty are listed in still_empty_face_joints.txt.+#+# Usage:+# bash data_process/scripts/run_joints_face_correct_llm.sh objaverse+# MODEL=gpt-5 bash data_process/scripts/run_joints_face_correct_llm.sh objaverse+#+# Env overrides: RAW, CLEANED, FACE, FAILED_LIST (optional id subset), MODEL,+# MAX_TOKENS, RIG_TIMEOUT. Extra args are passed through.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_joints_face_correct_llm.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++RAW=${RAW:-$(export_dir "$DATASET")/joint_names.json}+CLEANED=${CLEANED:-$(export_dir "$DATASET")/clean_joint_names.json}+FACE=${FACE:-$(export_dir "$DATASET")/face_joint_names.json}+MODEL=${MODEL:-deepseek-v4-flash}+MAX_TOKENS=${MAX_TOKENS:-512}+# Thinking backends need a much larger per-rig budget (see names_correct wrapper).+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi++for f in "$RAW" "$CLEANED" "$FACE"; do+ [[ -f "$f" ]] || { echo "MISSING FILE: $f" >&2; exit 1; }+done++EXTRA_ARGS=()+[[ -n "${FAILED_LIST:-}" ]] && EXTRA_ARGS+=(--failed_list="$FAILED_LIST")++python -m data_process.joint_annotation.face_correct_llm \+ --raw="$RAW" --cleaned="$CLEANED" --face="$FACE" \+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \+ "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_joints_face_select_llm.sh b/data_process/scripts/run_joints_face_select_llm.shnew file mode 100755index 0000000..63950dc--- /dev/null+++ b/data_process/scripts/run_joints_face_select_llm.sh@@ -0,0 +1,32 @@+#!/bin/bash+# Stage 3 โ LLM facing-direction joint pair per rig: -> face_joint_names.json.+#+# Usage:+# bash data_process/scripts/run_joints_face_select_llm.sh objaverse # DeepSeek v4-flash (default)+# MODEL=Qwen/Qwen3-8B bash data_process/scripts/run_joints_face_select_llm.sh objaverse # local HF model+#+# Env overrides: INPUT_DIR, OUTPUT_NAME, MODEL, MAX_TOKENS. Extra args are passed+# through (e.g. --no_fallback). API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_joints_face_select_llm.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++INPUT_DIR=${INPUT_DIR:-$(export_dir "$DATASET")}+OUTPUT_NAME=${OUTPUT_NAME:-face_joint_names.json}+MODEL=${MODEL:-deepseek-v4-flash}+# Thinking backends hold a call for tens of seconds (DeepSeek low-effort+# ~19s, plus server keep-alive under load), so the per-rig SIGALRM budget+# scales with the backend; override with RIG_TIMEOUT.+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi+MAX_TOKENS=${MAX_TOKENS:-512}++python -m data_process.joint_annotation.face_select_llm \+ --input_dir="$INPUT_DIR" --output_name="$OUTPUT_NAME" \+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \+"$@"diff --git a/data_process/scripts/run_joints_face_select_rule.sh b/data_process/scripts/run_joints_face_select_rule.shnew file mode 100755index 0000000..0056a82--- /dev/null+++ b/data_process/scripts/run_joints_face_select_rule.sh@@ -0,0 +1,23 @@+#!/bin/bash+# Stage 3 โ rule-based facing-direction joint pair: -> face_joint_names.json.+# Reads joint_names.json + clean_joint_names.json from the export directory.+#+# Usage:+# bash data_process/scripts/run_joints_face_select_rule.sh <truebones|mixamo|objaverse>+#+# Env overrides: INPUT_DIR, OUTPUT++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_joints_face_select_rule.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++INPUT_DIR=${INPUT_DIR:-$(export_dir "$DATASET")}+OUTPUT=${OUTPUT:-$INPUT_DIR/face_joint_names.json}++python -m data_process.joint_annotation.face_select_rule \+ --input_dir "$INPUT_DIR" --output "$OUTPUT" "$@"diff --git a/data_process/scripts/run_joints_names_clean_llm.sh b/data_process/scripts/run_joints_names_clean_llm.shnew file mode 100755index 0000000..53b688d--- /dev/null+++ b/data_process/scripts/run_joints_names_clean_llm.sh@@ -0,0 +1,29 @@+#!/bin/bash+# Stage 3 โ LLM joint-name cleaning: joint_names.json -> clean_joint_names.json.+#+# Usage:+# bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # DeepSeek v4-flash (default)+# MODEL=deepseek-v4-flash bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # DeepSeek API+# MODEL=Qwen/Qwen3-8B bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # local HF model+#+# Env overrides: INPUT, OUTPUT, MODEL, MAX_TOKENS. Extra args are passed through+# (e.g. --no_fallback). API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_joints_names_clean_llm.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++INPUT=${INPUT:-$(export_dir "$DATASET")/joint_names.json}+OUTPUT=${OUTPUT:-$(export_dir "$DATASET")/clean_joint_names.json}+MODEL=${MODEL:-deepseek-v4-flash}+MAX_TOKENS=${MAX_TOKENS:-2048}++python -m data_process.joint_annotation.names_clean_llm \+ --input="$INPUT" --output="$OUTPUT" \+ --model="$MODEL" --max_tokens="$MAX_TOKENS" \+ --report "$@"diff --git a/data_process/scripts/run_joints_names_clean_rule.sh b/data_process/scripts/run_joints_names_clean_rule.shnew file mode 100755index 0000000..07d9966--- /dev/null+++ b/data_process/scripts/run_joints_names_clean_rule.sh@@ -0,0 +1,22 @@+#!/bin/bash+# Stage 3 โ rule-based joint-name cleaning: joint_names.json -> clean_joint_names.json.+#+# Usage:+# bash data_process/scripts/run_joints_names_clean_rule.sh <truebones|mixamo|objaverse> [--report]+#+# Env overrides: INPUT, OUTPUT (default: inside dataset/export/<dataset>)++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_joints_names_clean_rule.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++INPUT=${INPUT:-$(export_dir "$DATASET")/joint_names.json}+OUTPUT=${OUTPUT:-$(export_dir "$DATASET")/clean_joint_names.json}++python -m data_process.joint_annotation.names_clean_rule \+ --input="$INPUT" --output="$OUTPUT" "$@"diff --git a/data_process/scripts/run_joints_names_correct_llm.sh b/data_process/scripts/run_joints_names_correct_llm.shnew file mode 100755index 0000000..4472d62--- /dev/null+++ b/data_process/scripts/run_joints_names_correct_llm.sh@@ -0,0 +1,40 @@+#!/bin/bash+# Stage 3 โ LLM check-and-correct sweep over clean_joint_names.json (in place).+#+# Every rig in joint_names.json is sent to the LLM together with its current+# cleaned label; the LLM keeps or corrects each label. The cleaned file is+# updated IN PLACE (a *.bak copy is made on first run). Set FAILED_LIST to+# restrict the sweep to the rig ids listed in a text file.+#+# Usage:+# bash data_process/scripts/run_joints_names_correct_llm.sh objaverse+# FAILED_LIST=.../failed_clean_names.txt bash data_process/scripts/run_joints_names_correct_llm.sh objaverse+#+# Env overrides: RAW, CLEANED, FAILED_LIST, MODEL, MAX_TOKENS. Extra args are+# passed through. API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++DATASET=${1:?Usage: run_joints_names_correct_llm.sh <truebones|mixamo|objaverse> [extra args...]}+shift+require_dataset "$DATASET"++RAW=${RAW:-$(export_dir "$DATASET")/joint_names.json}+CLEANED=${CLEANED:-$(export_dir "$DATASET")/clean_joint_names.json}+MODEL=${MODEL:-deepseek-v4-flash}+# Thinking backends hold a call for tens of seconds (DeepSeek low-effort+# ~19s, plus server keep-alive under load), so the per-rig SIGALRM budget+# scales with the backend; override with RIG_TIMEOUT.+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi+MAX_TOKENS=${MAX_TOKENS:-4096}++EXTRA_ARGS=()+[[ -n "${FAILED_LIST:-}" ]] && EXTRA_ARGS+=(--failed_list="$FAILED_LIST")++python -m data_process.joint_annotation.names_correct_llm \+ --raw="$RAW" --cleaned="$CLEANED" \+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \+ "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_joints_vis_tpose.sh b/data_process/scripts/run_joints_vis_tpose.shnew file mode 100755index 0000000..e27fe1c--- /dev/null+++ b/data_process/scripts/run_joints_vis_tpose.sh@@ -0,0 +1,44 @@+#!/bin/bash+# Stage 3 โ render annotated T-pose PNGs from exported motion NPZs.+# Use the PNGs to identify left/right symmetric face joints by eye.+#+# Usage:+# # Batch mode โ one PNG per unique skeleton in the export directory:+# bash data_process/scripts/run_joints_vis_tpose.sh truebones+# LIMIT=50 bash data_process/scripts/run_joints_vis_tpose.sh truebones+#+# # Single mode โ one NPZ:+# NPZ_PATH=/path/to/motion.npz bash data_process/scripts/run_joints_vis_tpose.sh+# NPZ_PATH=/path/to/motion.npz OUTPUT_PATH=out.png bash data_process/scripts/run_joints_vis_tpose.sh+#+# Env overrides:+# NPZ_PATH single mode: path to one motion NPZ (takes precedence over batch mode)+# OUTPUT_PATH [single] output PNG path+# MOTIONS_DIR [batch] motion NPZ directory (default: <export_dir>/motions)+# OUTPUT_DIR [batch] PNG output directory (default: <export_dir>/face_joints_vis)+# LIMIT [batch] stop after N skeletons+# FONT_SIZE label font size (default: 7)++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"++cmd=(python -m data_process.tools.vis_tpose)++if [[ -n "${NPZ_PATH:-}" ]]; then+ [[ -f "$NPZ_PATH" ]] || { echo "NPZ_PATH not found: $NPZ_PATH" >&2; exit 1; }+ cmd+=(--npz_path "$NPZ_PATH")+ [[ -n "${OUTPUT_PATH:-}" ]] && cmd+=(--output "$OUTPUT_PATH")+else+ DATASET=${1:?Usage: run_joints_vis_tpose.sh <truebones|mixamo|objaverse> (or set NPZ_PATH)}+ require_dataset "$DATASET"+ MOTIONS_DIR=${MOTIONS_DIR:-$(export_dir "$DATASET")/motions}+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir "$DATASET")/face_joints_vis}+ [[ -d "$MOTIONS_DIR" ]] || { echo "MOTIONS_DIR not found: $MOTIONS_DIR" >&2; exit 1; }+ cmd+=(--motions_dir "$MOTIONS_DIR" --output_dir "$OUTPUT_DIR")+ [[ -n "${LIMIT:-}" ]] && cmd+=(--limit "$LIMIT")+fi+[[ -n "${FONT_SIZE:-}" ]] && cmd+=(--font-size "$FONT_SIZE")++"${cmd[@]}"diff --git a/data_process/scripts/run_render_motion.sh b/data_process/scripts/run_render_motion.shnew file mode 100755index 0000000..3056677--- /dev/null+++ b/data_process/scripts/run_render_motion.sh@@ -0,0 +1,98 @@+#!/bin/bash+# Stage 2a โ render multi-view frames of every motion (EEVEE), the input of+# VLM motion captioning (run_caption_motion.sh).+#+# Usage:+# bash data_process/scripts/run_render_motion.sh truebones+# bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8+# bash data_process/scripts/run_render_motion.sh objaverse --missing-only+# DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_motion.sh mixamo+#+# Inputs per dataset: truebones/objaverse render the raw FBX/GLB assets;+# mixamo renders the animated character files produced by+# run_animate_mixamo.sh (raw Mixamo FBXs carry no mesh).+#+# Runs with plain python + the pip `bpy` module: EEVEE needs the module's+# GPU context (`blender -b` has no display surface). --multi-worker N+# shards the assets over N processes; with NUM_GPUS>1 the workers are+# spread round-robin over the visible GPUs (worker i -> GPU i % NUM_GPUS),+# so N is the *total* worker count, not the per-GPU count.+#+# KNOWN LIMITATION: the CUDA_VISIBLE_DEVICES pinning does NOT bind EEVEE's+# EGL context โ on a multi-GPU machine every worker still renders on the+# first GPU. Give each render process (or Slurm job) a machine/cgroup that+# exposes exactly one GPU instead of relying on NUM_GPUS>1.+#+# Env overrides: DATA_DIR, OUTPUT_DIR, RESOLUTION, SAMPLES, CAMERA_DIST,+# MAX_RENDER_FRAMES (per-clip frame cap, default 200), MIN_ACTION_FRAMES,+# NUM_GPUS (default 1). Extra arguments are passed through to the renderer.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"+DATASET=${1:-}+[[ -n "$DATASET" ]] || { print_usage; exit 0; }+require_dataset "$DATASET"+shift++NUM_WORKERS=1+NUM_GPUS=${NUM_GPUS:-1}+EXTRA_ARGS=()+while [[ $# -gt 0 ]]; do+ case "$1" in+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ *) EXTRA_ARGS+=("$1"); shift ;;+ esac+done++case "$DATASET" in+ truebones) DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation} ;;+ mixamo) DATA_DIR=${DATA_DIR:-outputs/mixamo_characters} ;;+ objaverse) DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb} ;;+esac+OUTPUT_DIR=${OUTPUT_DIR:-$(render_dir "$DATASET")}++RENDERER="data_process.motion_rendering.render_${DATASET}"+COMMON_ARGS=(+ --data_dir="$DATA_DIR"+ --output_dir="$OUTPUT_DIR"+ --resolution="${RESOLUTION:-512}"+ --samples="${SAMPLES:-64}"+ --camera_dist="${CAMERA_DIST:-1.5}"+)+[[ -n "${MAX_RENDER_FRAMES:-}" ]] && COMMON_ARGS+=(--max_render_frames="$MAX_RENDER_FRAMES")+[[ -n "${MIN_ACTION_FRAMES:-}" ]] && COMMON_ARGS+=(--min_action_frames="$MIN_ACTION_FRAMES")++echo "Dataset: $DATASET | Input: $DATA_DIR | Output: $OUTPUT_DIR"+if [[ "$NUM_WORKERS" -gt 1 ]]; then+ LOG_DIR=$(worker_log_dir "render-motion-$DATASET")+ echo "Launching $NUM_WORKERS render workers (per-worker logs: $LOG_DIR)..."+ pids=()+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do+ # Pin each worker to one GPU before the process starts: EEVEE picks+ # its device when the bpy module builds its GL context, so the+ # restriction has to be in the environment, not set from Python.+ gpu=$((worker_id % NUM_GPUS))+ CUDA_VISIBLE_DEVICES="$gpu" \+ python -m "$RENDERER" "${COMMON_ARGS[@]}" \+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \+ "${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &+ pids+=($!)+ echo " Started worker $worker_id on GPU $gpu (PID ${pids[-1]})"+ done++ # A bare `wait` always returns 0 โ check every worker individually.+ failed=0+ for worker_id in "${!pids[@]}"; do+ wait "${pids[worker_id]}" \+ || { echo "Worker $worker_id failed (log: $LOG_DIR/worker${worker_id}.log)" >&2; failed=1; }+ done+ if [[ $failed -ne 0 ]]; then+ echo "Some render workers failed. Rerun to retry the missing frames." >&2+ exit 1+ fi+ echo "All workers finished."+else+ python -m "$RENDERER" "${COMMON_ARGS[@]}" "${EXTRA_ARGS[@]}"+fidiff --git a/data_process/scripts/run_render_tpose.sh b/data_process/scripts/run_render_tpose.shnew file mode 100755index 0000000..bd16345--- /dev/null+++ b/data_process/scripts/run_render_tpose.sh@@ -0,0 +1,77 @@+#!/bin/bash+# Stage 2a โ render a T-pose 2x2 grid per asset (EEVEE), the input of+# body-plan category classification (run_caption_category.sh).+#+# Usage:+# bash data_process/scripts/run_render_tpose.sh truebones+# bash data_process/scripts/run_render_tpose.sh objaverse --multi-worker 8+# DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_tpose.sh mixamo+#+# Grids are written flat as <tpose_dir>/<name>.png (one per GLB stem, or+# per Truebones object type). Runs with plain python + the pip `bpy`+# module: EEVEE needs the module's GPU context.+#+# Env overrides: DATA_DIR, OUTPUT_DIR, RESOLUTION, SAMPLES, CAMERA_DIST+# Extra arguments are passed through to the renderer.++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"+DATASET=${1:-}+[[ -n "$DATASET" ]] || { print_usage; exit 0; }+require_dataset "$DATASET"+shift++NUM_WORKERS=1+EXTRA_ARGS=()+while [[ $# -gt 0 ]]; do+ case "$1" in+ --multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ *) EXTRA_ARGS+=("$1"); shift ;;+ esac+done++case "$DATASET" in+ truebones) DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation}+ EXTRA_ARGS+=(--species_grids) ;; # one grid per species+ mixamo) DATA_DIR=${DATA_DIR:-dataset/raw/mixamo/character_refined} ;;+ objaverse) DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb} ;;+esac+OUTPUT_DIR=${OUTPUT_DIR:-$(tpose_dir "$DATASET")}++COMMON_ARGS=(+ --data_dir="$DATA_DIR"+ --output_dir="$OUTPUT_DIR"+ --resolution="${RESOLUTION:-512}"+ --samples="${SAMPLES:-64}"+ --camera_dist="${CAMERA_DIST:-1.5}"+)++echo "Dataset: $DATASET | Input: $DATA_DIR | Output: $OUTPUT_DIR"+if [[ "$NUM_WORKERS" -gt 1 ]]; then+ LOG_DIR=$(worker_log_dir "render-tpose-$DATASET")+ echo "Launching $NUM_WORKERS render workers (per-worker logs: $LOG_DIR)..."+ pids=()+ for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do+ python -m data_process.motion_rendering.render_tpose "${COMMON_ARGS[@]}" \+ --worker_id="$worker_id" --num_workers="$NUM_WORKERS" \+ "${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &+ pids+=($!)+ echo " Started worker $worker_id (PID ${pids[-1]})"+ done++ # A bare `wait` always returns 0 โ check every worker individually.+ failed=0+ for worker_id in "${!pids[@]}"; do+ wait "${pids[worker_id]}" \+ || { echo "Worker $worker_id failed (log: $LOG_DIR/worker${worker_id}.log)" >&2; failed=1; }+ done+ if [[ $failed -ne 0 ]]; then+ echo "Some render workers failed. Rerun to retry the missing grids." >&2+ exit 1+ fi+ echo "All workers finished."+else+ python -m data_process.motion_rendering.render_tpose "${COMMON_ARGS[@]}" "${EXTRA_ARGS[@]}"+fidiff --git a/data_process/tools/__init__.py b/data_process/tools/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/tools/character_fbx2glb.py b/data_process/tools/character_fbx2glb.pynew file mode 100644index 0000000..e402d07--- /dev/null+++ b/data_process/tools/character_fbx2glb.py@@ -0,0 +1,110 @@+"""Convert rigged T-pose character FBX files to GLB, one to one.++Consumes a flat directory of character FBXs (mesh + armature, no animation+required โ e.g. ``dataset/raw/mixamo/character_refined``) and writes+``{name}.glb`` per file. Any actions the FBX importer binds (static+object-transform takes) are stripped so the GLB is a clean T-pose rig.+Resumable: existing GLBs are skipped.++Usage (Blender headless):+ blender -b -P data_process/tools/character_fbx2glb.py -- \+ --data_dir dataset/raw/mixamo/character_refined \+ --output_dir dataset/raw/mixamo/character_refined_glb+"""++import argparse+import os+import sys++import bpy+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import import_fbx, reset_scene+++def convert_character_to_glb(fbx_path, glb_path):+ """Import one character FBX and export it as a T-pose GLB (no animation)."""+ reset_scene()+ import_fbx(fbx_path)++ armatures = [o for o in bpy.data.objects if o.type == 'ARMATURE']+ assert armatures, f"No armature found in {fbx_path}"+ meshes = [o for o in bpy.data.objects if o.type == 'MESH']+ assert meshes, f"No mesh found in {fbx_path}"++ # Strip every action/binding so the export carries only the rig in its+ # rest pose โ character files should have no animation, but the FBX+ # importer still binds static object-transform takes.+ for obj in bpy.data.objects:+ if obj.animation_data is not None:+ obj.animation_data_clear()+ for action in list(bpy.data.actions):+ bpy.data.actions.remove(action)++ for obj in bpy.data.objects:+ obj.select_set(obj.type in ('MESH', 'ARMATURE'))++ kwargs = dict(+ filepath=glb_path,+ check_existing=False,+ use_selection=True,+ export_format='GLB',+ export_animations=False,+ )+ bpy.ops.export_scene.gltf(**kwargs)+ logger.info(f"Saved T-pose GLB: {glb_path}")+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Convert rigged T-pose character FBX files to GLB, one to one.")+ parser.add_argument('--data_dir', type=str, required=True,+ help='Flat directory of character FBX files.')+ parser.add_argument('--output_dir', type=str, required=True,+ help='Destination directory for the per-character GLB files.')+ parser.add_argument('--worker_id', type=int, default=0,+ help='Worker index for sharding across workers.')+ parser.add_argument('--num_workers', type=int, default=1,+ help='Total number of workers.')+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++if __name__ == "__main__":+ args = parse_args()++ fbx_files = sorted(f for f in os.listdir(args.data_dir)+ if f.lower().endswith('.fbx'))+ logger.info(f"Found {len(fbx_files)} character FBXs in {args.data_dir}")++ if args.num_workers > 1:+ fbx_files = fbx_files[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(fbx_files)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'fbx2glb_errors_worker{args.worker_id}.log')++ n_ok = n_skip = n_fail = 0+ for fbx_file in fbx_files:+ glb_path = os.path.join(args.output_dir,+ os.path.splitext(fbx_file)[0] + '.glb')+ if os.path.exists(glb_path):+ n_skip += 1+ continue+ try:+ convert_character_to_glb(os.path.join(args.data_dir, fbx_file), glb_path)+ n_ok += 1+ except Exception as e: # noqa: BLE001 โ keep the batch going+ logger.error(f"Failed to convert {fbx_file}: {e}")+ with open(error_log, 'a') as f:+ f.write(f"{fbx_file}\t{e}\n")+ n_fail += 1++ logger.info(f"Done: {n_ok} converted, {n_skip} skipped, {n_fail} failed "+ f"-> {args.output_dir}")++ if n_fail:+ sys.exit(1)diff --git a/data_process/tools/merge_summaries.py b/data_process/tools/merge_summaries.pynew file mode 100644index 0000000..2223f24--- /dev/null+++ b/data_process/tools/merge_summaries.py@@ -0,0 +1,137 @@+"""Merge per-worker export summary JSONs into the canonical files.++Multi-worker export runs write per-worker summary shards+(``joint_names_worker{i}.json``, ``joint_count_worker{i}.json``,+``clip_frames_worker{i}.json``) so parallel Blender processes never race+on one file. This script merges them โ together with any existing canonical+files โ into ``joint_names.json`` / ``joint_count.json`` /+``clip_frames.json`` (plus ``summary.json`` with the dataset-level+summary) and removes the shards. Idempotent: rerunning with no shards present+just rewrites the canonical files from themselves.++Plain Python (no bpy), so it runs in the conda env directly:+ python -m data_process.tools.merge_summaries --output_dir <export_dir>+"""++import argparse+import glob+import json+import os+import sys++from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.export_markers import collect_joint_names_from_markers # noqa: E402+++def _load(path):+ with open(path) as f:+ return json.load(f)+++def _save(path, data):+ tmp = f"{path}.tmp.{os.getpid()}"+ with open(tmp, "w") as f:+ json.dump(data, f, indent=2)+ os.replace(tmp, path)+++def _scan_missing_frame_counts(motion_dir, known):+ """Frame counts for clips in *motion_dir* that *known* doesn't cover."""+ if not os.path.isdir(motion_dir):+ return {}+ import numpy as np # local: keeps the module importable without numpy++ found = {}+ for npz_file in sorted(os.listdir(motion_dir)):+ if not npz_file.endswith(".npz"):+ continue+ clip_name = os.path.splitext(npz_file)[0]+ if clip_name in known:+ continue+ try:+ data = np.load(os.path.join(motion_dir, npz_file), allow_pickle=True)+ found[clip_name] = int(data["anim_local_rot"].shape[0])+ except Exception as exc: # noqa: BLE001 โ a bad clip shouldn't stop the merge+ logger.warning(f"Could not read frame count from {npz_file}: {exc}")+ return found+++def merge_summaries(output_dir, fps=30, keep_shards=False):+ jn_path = os.path.join(output_dir, "joint_names.json")+ joint_names = _load(jn_path) if os.path.isfile(jn_path) else {}++ # Per-asset completion markers are written as each asset finishes, so they+ # survive a killed worker whose in-memory shard never reached disk. Folding+ # them in here is what makes a truncated joint_names.json self-healing.+ from_markers = collect_joint_names_from_markers(output_dir)+ recovered = len(set(from_markers) - set(joint_names))+ joint_names.update(from_markers)+ if recovered:+ logger.info(f"Recovered {recovered} joint-name entries from completion markers")++ jn_shards = sorted(glob.glob(os.path.join(output_dir, "joint_names_worker*.json")))+ for shard in jn_shards:+ joint_names.update(_load(shard))+ if joint_names:+ _save(jn_path, joint_names)+ _save(os.path.join(output_dir, "joint_count.json"),+ {k: len(v) for k, v in joint_names.items()})+ logger.info(f"joint_names.json: {len(joint_names)} entries "+ f"({len(jn_shards)} shard(s) merged)")++ af_path = os.path.join(output_dir, "clip_frames.json")+ frames = {}+ if os.path.isfile(af_path):+ # Tolerate the legacy in-file summary block from older exports.+ frames = {k: v for k, v in _load(af_path).items() if k != "_summary"}+ af_shards = sorted(glob.glob(os.path.join(output_dir, "clip_frames_worker*.json")))+ for shard in af_shards:+ frames.update({k: v for k, v in _load(shard).items() if k != "_summary"})++ # Clips written after the last summary flush (or by a worker that was+ # killed before writing its shard) are on disk but absent here. Scan for+ # those only โ the NPZ read is the expensive part, so already-recorded+ # clips are left alone.+ scanned = _scan_missing_frame_counts(os.path.join(output_dir, "motions"), frames)+ if scanned:+ logger.info(f"Recovered {len(scanned)} clip frame counts by scanning motions/")+ frames.update(scanned)++ if frames:+ _save(af_path, frames)+ total_frames = sum(frames.values())+ duration_sec = total_frames / fps if fps else 0.0+ _save(os.path.join(output_dir, "summary.json"), {+ "total_clips": len(frames),+ "total_frames": total_frames,+ "total_duration_sec": round(duration_sec, 2),+ "total_duration_min": round(duration_sec / 60, 2),+ "fps": fps,+ })+ logger.info(f"clip_frames.json: {len(frames)} clips "+ f"({len(af_shards)} shard(s) merged)")++ if not keep_shards:+ jc_shards = sorted(glob.glob(os.path.join(output_dir, "joint_count_worker*.json")))+ for shard in jn_shards + af_shards + jc_shards:+ os.remove(shard)+++def main():+ parser = argparse.ArgumentParser(+ description="Merge per-worker export summary JSONs into the canonical files.")+ parser.add_argument("--output_dir", type=str, required=True,+ help="Export directory holding the *_worker{i}.json shards.")+ parser.add_argument("--fps", type=int, default=30,+ help="Frame rate for the clip_frames duration summary.")+ parser.add_argument("--keep_shards", action="store_true",+ help="Keep the per-worker shard files after merging.")+ args = parser.parse_args()+ merge_summaries(args.output_dir, fps=args.fps, keep_shards=args.keep_shards)+++if __name__ == "__main__":+ main()diff --git a/data_process/tools/truebones_fbx2glb.py b/data_process/tools/truebones_fbx2glb.pynew file mode 100644index 0000000..b93386f--- /dev/null+++ b/data_process/tools/truebones_fbx2glb.py@@ -0,0 +1,127 @@+"""Convert curated Truebones per-clip FBX files to GLB, one to one.++Consumes the flat ``{Species}-{Action}.fbx`` layout (the same input as+``export_truebones.py``) and writes ``{Species}-{Action}.glb`` next to it,+preserving mesh, armature, and the clip's single animation take. Resumable:+existing GLBs are skipped.++Usage (Blender headless):+ blender -b -P data_process/tools/truebones_fbx2glb.py -- \+ --data_dir dataset/raw/truebones/animation \+ --output_dir dataset/raw/truebones/animation_glb+"""++import argparse+import os+import sys++import bpy+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.blender_export import import_fbx, reset_scene+++def convert_fbx_to_glb(fbx_path, glb_path):+ """Import one per-clip FBX and export it as GLB with its animation."""+ reset_scene()+ import_fbx(fbx_path)++ armatures = [o for o in bpy.data.objects if o.type == 'ARMATURE']+ assert armatures, f"No armature found in {fbx_path}"+ armature = armatures[0]+ assert armature.animation_data is not None and armature.animation_data.action is not None, \+ f"Armature '{armature.name}' has no bound action in {fbx_path}"++ # Name the GLB animation after the clip's action; drop the static+ # object-transform actions the FBX importer binds to mesh objects so the+ # skeletal take is the only animation in the file.+ stem = os.path.splitext(os.path.basename(fbx_path))[0]+ _, sep, action_part = stem.partition('-')+ if not sep:+ raise ValueError(+ f"Clip filename '{stem}' has no '-' separating species from action "+ f"(expected '{{Species}}-{{Action}}.fbx')")+ take = armature.animation_data.action+ take.name = action_part+ for obj in bpy.data.objects:+ if obj.type != 'ARMATURE' and obj.animation_data is not None:+ obj.animation_data_clear()+ for action in list(bpy.data.actions):+ if action is not take:+ bpy.data.actions.remove(action)++ for obj in bpy.data.objects:+ obj.select_set(obj.type in ('MESH', 'ARMATURE'))++ kwargs = dict(+ filepath=glb_path,+ check_existing=False,+ use_selection=True,+ export_format='GLB',+ export_animations=True,+ )+ # One animation per GLB. On Blender 3.2 `export_nla_strips=False` merges+ # everything into a single clip; the option was replaced by+ # `export_animation_mode` in 3.6 (ACTIVE_ACTIONS emits the armature's+ # bound action as the only animation).+ if bpy.app.version >= (3, 6, 0):+ kwargs['export_animation_mode'] = 'ACTIVE_ACTIONS'+ else:+ kwargs['export_nla_strips'] = False+ bpy.ops.export_scene.gltf(**kwargs)+ logger.info(f"Saved animated GLB: {glb_path}")+++def parse_args():+ parser = argparse.ArgumentParser(+ description="Convert curated Truebones per-clip FBX files to GLB, one to one.")+ parser.add_argument('--data_dir', type=str, required=True,+ help='Flat directory of per-clip {Species}-{Action}.fbx files.')+ parser.add_argument('--output_dir', type=str, required=True,+ help='Destination directory for the per-clip GLB files.')+ parser.add_argument('--worker_id', type=int, default=0,+ help='Worker index for sharding across workers.')+ parser.add_argument('--num_workers', type=int, default=1,+ help='Total number of workers.')+ argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]+ return parser.parse_args(argv)+++if __name__ == "__main__":+ args = parse_args()++ fbx_files = sorted(f for f in os.listdir(args.data_dir)+ if f.lower().endswith('.fbx'))+ logger.info(f"Found {len(fbx_files)} FBX clips in {args.data_dir}")++ if args.num_workers > 1:+ fbx_files = fbx_files[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(fbx_files)} files")++ os.makedirs(args.output_dir, exist_ok=True)+ error_log = os.path.join(args.output_dir,+ f'fbx2glb_errors_worker{args.worker_id}.log')++ n_ok = n_skip = n_fail = 0+ for fbx_file in fbx_files:+ glb_path = os.path.join(args.output_dir,+ os.path.splitext(fbx_file)[0] + '.glb')+ if os.path.exists(glb_path):+ n_skip += 1+ continue+ try:+ convert_fbx_to_glb(os.path.join(args.data_dir, fbx_file), glb_path)+ n_ok += 1+ except Exception as e: # noqa: BLE001 โ keep the batch going+ logger.error(f"Failed to convert {fbx_file}: {e}")+ with open(error_log, 'a') as f:+ f.write(f"{fbx_file}\t{e}\n")+ n_fail += 1++ logger.info(f"Done: {n_ok} converted, {n_skip} skipped, {n_fail} failed "+ f"-> {args.output_dir}")++ if n_fail:+ sys.exit(1)diff --git a/data_process/tools/vis_motion.py b/data_process/tools/vis_motion.pynew file mode 100644index 0000000..b375673--- /dev/null+++ b/data_process/tools/vis_motion.py@@ -0,0 +1,197 @@+"""Visualize an exported motion with a root-anchored orientation arrow.++Sanity-check tool for facing-direction annotations: loads an export-stage+motion NPZ, reconstructs per-frame global joint positions via forward+kinematics, derives a per-frame orientation vector, and saves an MP4 where+an arrow at the root joint points along that orientation. With+``--direction face --face-joints i j`` the arrow shows the heading implied+by a candidate face-joint pair.++Expected NPZ keys:++ rest_local_pos : (J, 3) rest bone offsets+ anim_local_pos : (T, J, 3) per-frame local joint positions+ anim_local_rot : (T, J, 4) per-frame local joint rotations (quaternions)+ parents : (J,) parent joint indices (-1 for root)++The orientation vector is chosen by ``--direction``:++ velocity (default) โ root horizontal velocity, held forward on stalls+ x / y / z โ fixed unit axis (world frame, pre-rotate)+ face โ vector from root to the mean of two face joints+ specified via ``--face-joints i j``++Usage:+ python -m data_process.tools.vis_motion \\+ <path/to/motion.npz> [--output out.mp4] [--fps 30] \\+ [--direction velocity|x|y|z|face] [--face-joints i j]+"""++import argparse+import os+import sys+from pathlib import Path++import numpy as np++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from Animation import Animation, Quaternions, positions_global # noqa: E402+from data_process.utils.plotting import render_skeleton_motion_directed # noqa: E402+import imageio # noqa: E402+++def load_motion_from_npz(npz_path):+ """Load a motion NPZ and compute per-frame global joint positions.++ Returns:+ (parents, global_positions) where ``global_positions`` is a+ float32 (T, J, 3) array.+ """+ data = np.load(npz_path, allow_pickle=True)+ required = ('rest_local_pos', 'anim_local_pos', 'anim_local_rot', 'parents')+ missing = [k for k in required if k not in data.files]+ if missing:+ raise KeyError(+ f"{npz_path} is missing required keys {missing}; "+ f"available keys: {list(data.files)}"+ )++ rest_local_pos = data['rest_local_pos'] # (J, 3)+ anim_local_pos = data['anim_local_pos'] # (T, J, 3)+ anim_local_rot = data['anim_local_rot'] # (T, J, 4)+ parents = np.asarray(data['parents']) # (J,)+ njoints = len(parents)++ motion_anim = Animation(+ rotations=Quaternions(anim_local_rot),+ positions=anim_local_pos,+ orients=Quaternions.id(njoints),+ offsets=rest_local_pos,+ parents=parents,+ )+ global_pos = positions_global(motion_anim).astype(np.float32) # (T, J, 3)+ return parents, global_pos+++def compute_directions(global_pos, mode, face_joints=None):+ """Build a per-frame orientation vector from global joint positions.++ Args:+ global_pos: (T, J, 3) float array.+ mode: One of ``'velocity'``, ``'x'``, ``'y'``, ``'z'``, ``'face'``.+ face_joints: Tuple ``(i, j)`` of joint indices used when+ ``mode == 'face'``; the direction points from the root joint+ toward the midpoint of joints ``i`` and ``j``.++ Returns:+ (T, 3) float32 array. Not normalized โ the renderer normalizes+ internally and skips zero-length frames.+ """+ T = global_pos.shape[0]++ if mode == 'velocity':+ # Horizontal velocity of the root joint, held forward on stalls so+ # the arrow doesn't flicker off on stationary frames.+ roots = global_pos[:, 0] # (T, 3)+ vel = np.zeros_like(roots)+ vel[1:] = roots[1:] - roots[:-1]+ if T >= 2:+ vel[0] = vel[1]+ vel[:, 1] = 0.0 # project to ground plane (y=up)+ # Forward-fill through stalls so the arrow stays visible.+ last = np.zeros(3, dtype=np.float32)+ for t in range(T):+ if np.linalg.norm(vel[t]) < 1e-6:+ vel[t] = last+ else:+ last = vel[t]+ return vel.astype(np.float32)++ if mode in ('x', 'y', 'z'):+ axis = {'x': [1, 0, 0], 'y': [0, 1, 0], 'z': [0, 0, 1]}[mode]+ return np.tile(np.array(axis, dtype=np.float32)[None, :], (T, 1))++ if mode == 'face':+ if face_joints is None or len(face_joints) != 2:+ raise ValueError("--face-joints i j required when --direction=face")+ i, j = face_joints+ mid = 0.5 * (global_pos[:, i] + global_pos[:, j]) # (T, 3)+ return (mid - global_pos[:, 0]).astype(np.float32)++ raise ValueError(f"unknown direction mode: {mode}")+++DEFAULT_OUTPUT_DIR = "outputs/motion_vis"+++def default_output_path(npz_path):+ """Default output: ``<DEFAULT_OUTPUT_DIR>/<stem>_motion_directed.mp4``."""+ return os.path.join(DEFAULT_OUTPUT_DIR, f"{Path(npz_path).stem}_motion_directed.mp4")+++def main():+ parser = argparse.ArgumentParser(+ description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter,+ )+ parser.add_argument(+ 'npz_path', type=str,+ help='Path to a motion NPZ containing anim_local_pos / anim_local_rot / rest_local_pos / parents',+ )+ parser.add_argument(+ '--output', type=str, default=None,+ help=f'Output MP4 path (default: {DEFAULT_OUTPUT_DIR}/<input_stem>_motion_directed.mp4)',+ )+ parser.add_argument('--fps', type=int, default=30, help='Output video fps (default: 30)')+ parser.add_argument(+ '--direction', type=str, default='velocity',+ choices=['velocity', 'x', 'y', 'z', 'face'],+ help='Source of the orientation vector (default: velocity)',+ )+ parser.add_argument(+ '--face-joints', type=int, nargs=2, default=None, metavar=('I', 'J'),+ help='Joint index pair for --direction=face (midpoint target)',+ )+ parser.add_argument('--elev', type=float, default=30.0, help='Camera elevation')+ parser.add_argument('--azim', type=float, default=-60.0, help='Camera azimuth')+ parser.add_argument(+ '--no-rotate-root', action='store_true',+ help='Disable Z-up -> Y-up swap (default: enabled, matches save_skeleton_motion)',+ )+ parser.add_argument(+ '--arrow-scale', type=float, default=0.25,+ help='Arrow length as a fraction of the scene side (default: 0.25)',+ )+ parser.add_argument('--arrow-color', type=str, default='green', help='Arrow color')+ args = parser.parse_args()++ if not os.path.exists(args.npz_path):+ parser.error(f"Input NPZ not found: {args.npz_path}")++ parents, global_pos = load_motion_from_npz(args.npz_path)+ directions = compute_directions(global_pos, args.direction, args.face_joints)++ frames = render_skeleton_motion_directed(+ parents=parents,+ positions=global_pos,+ directions=directions,+ elev=args.elev,+ azim=args.azim,+ rotate_root=not args.no_rotate_root,+ arrow_scale=args.arrow_scale,+ arrow_color=args.arrow_color,+ )++ output_path = args.output or default_output_path(args.npz_path)+ Path(output_path).parent.mkdir(parents=True, exist_ok=True)+ imageio.mimwrite(output_path, frames, fps=args.fps)+ print(+ f"Saved directed motion ({frames.shape[0]} frames, "+ f"{global_pos.shape[1]} joints, direction={args.direction}) "+ f"to {output_path}"+ )+++if __name__ == '__main__':+ main()diff --git a/data_process/tools/vis_tpose.py b/data_process/tools/vis_tpose.pynew file mode 100644index 0000000..4c6b432--- /dev/null+++ b/data_process/tools/vis_tpose.py@@ -0,0 +1,296 @@+"""Visualize skeleton T-poses with joint names annotated, for manual review.++Workflow companion for ``face_select*``: when automatic face-joint+resolution produces ``"source": "empty"`` (or picks the wrong pair), render+the T-pose with every joint labeled so the left/right symmetric pair can be+identified by eye. Each motion NPZ produces one PNG.++Accepts the export-stage NPZ layout (see+:func:`data_process.utils.blender_export.save_motion`) whose relevant fields are:++ rest_local_pos : (J, 3) local bone offsets at rest+ rest_local_rot : (J, 4) local bone rotations at rest (quaternions)+ names : (J,) joint name strings+ parents : (J,) parent joint indices (-1 for root)+ offsets : (J, 3) bind-pose offsets (optional โ falls back to+ rest_local_pos when absent)++The T-pose global positions are reconstructed via forward kinematics on the+rest pose and passed to :func:`save_skeleton_tpose_annotated`.++Modes:+ - Single: ``--npz_path X`` โ one PNG.+ - Batch (default): ``--motions_dir M [--output_dir O]`` โ one PNG per+ unique key (prefix before the first ``-``) discovered in ``M``.+ - Batch (face_json): ``--face_json F --motions_dir M [--source empty] [--output_dir O]``+ โ one PNG per key in F whose ``source`` field matches ``--source``.++See ``data_process/scripts/run_joints_vis_tpose.sh`` for the typical batch call.+"""++import argparse+import json+import os+import sys+from pathlib import Path++import numpy as np+from tqdm import tqdm++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from Animation import Animation, Quaternions, positions_global # noqa: E402+from data_process.utils.plotting import save_skeleton_tpose_annotated # noqa: E402+++REQUIRED_KEYS = ("rest_local_pos", "rest_local_rot", "names", "parents")+++def load_tpose_from_npz(npz_path):+ # type: (str) -> Tuple[np.ndarray, List[str], np.ndarray]+ """Load rest-pose data from a motion NPZ and compute global T-pose positions.++ Returns:+ ``(parents, joint_names, tpose_positions)`` where ``tpose_positions``+ is a float32 ``(J, 3)`` array of global joint coordinates at rest.+ """+ data = np.load(npz_path, allow_pickle=True)+ missing = [k for k in REQUIRED_KEYS if k not in data.files]+ if missing:+ raise KeyError(+ "{} is missing required keys {}; available: {}".format(+ npz_path, missing, list(data.files)+ )+ )++ rest_local_pos = data['rest_local_pos'] # (J, 3)+ rest_local_rot = data['rest_local_rot'] # (J, 4) quaternions+ parents = np.asarray(data['parents']) # (J,)+ names = [str(n) for n in data['names']] # length J+ # Prefer the explicit bind-pose offsets when the export recorded them;+ # fall back to the rest-pose local translation when absent.+ offsets = (np.asarray(data['offsets']) if 'offsets' in data.files+ else rest_local_pos)++ njoints = len(parents)++ rest_anim = Animation(+ rotations=Quaternions(rest_local_rot[None]), # (1, J, 4)+ positions=rest_local_pos[None], # (1, J, 3)+ orients=Quaternions.id(njoints),+ offsets=offsets,+ parents=parents,+ )+ tpose_positions = positions_global(rest_anim)[0].astype(np.float32) # (J, 3)++ return parents, names, tpose_positions+++def render_one(npz_path, output_path, *, font_size, joint_marker_size,+ bone_linewidth):+ # type: (str, str, int, float, float) -> None+ """Render one NPZ as an annotated T-pose PNG."""+ parents, names, tpose_positions = load_tpose_from_npz(npz_path)+ os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)+ save_skeleton_tpose_annotated(+ save_path=output_path,+ parents=parents,+ positions=tpose_positions,+ joint_names=names,+ font_size=font_size,+ joint_marker_size=joint_marker_size,+ bone_linewidth=bone_linewidth,+ )+++def find_motion_npz(motions_dir, key):+ # type: (str, str) -> Optional[Path]+ """Resolve the first motion NPZ whose prefix-before-'-' matches ``key``.++ Motion files are named ``<key>-<clip_name>.npz`` (e.g.+ ``669a728e57bbceb71e6b8ce6_fbx-mixamo_com.npz``). Any single clip is+ enough to render the rest-pose skeleton, so we take the first sorted+ match. Returns ``None`` if no file matches.+ """+ hits = sorted(Path(motions_dir).glob("{}-*.npz".format(key)))+ return hits[0] if hits else None+++def collect_keys_from_motions_dir(motions_dir):+ # type: (str) -> List[str]+ """Collect unique keys (prefix before the first ``-``) from NPZs in ``motions_dir``.++ NPZs without a ``-`` in their stem are treated as their own key.+ """+ keys = set()+ for p in Path(motions_dir).glob("*.npz"):+ stem = p.stem+ keys.add(stem.split("-", 1)[0] if "-" in stem else stem)+ return sorted(keys)+++DEFAULT_OUTPUT_DIR = "outputs/face_joints_vis"+++def default_single_output_path(npz_path):+ # type: (str) -> str+ """Default single-mode output: ``<DEFAULT_OUTPUT_DIR>/<stem>_tpose_annotated.png``."""+ return os.path.join(DEFAULT_OUTPUT_DIR,+ "{}_tpose_annotated.png".format(Path(npz_path).stem))+++def main():+ parser = argparse.ArgumentParser(+ description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter,+ )+ mode = parser.add_mutually_exclusive_group(required=False)+ mode.add_argument(+ '--npz_path', type=str, default=None,+ help='Single mode: path to a motion NPZ (skips batch logic).',+ )+ mode.add_argument(+ '--face_json', type=str, default=None,+ help='Batch mode (alt): path to face_joint_names.json. Motion files '+ 'are matched as <motions_dir>/<key>-*.npz (prefix before the '+ 'first "-"). Without this flag, batch mode scans --motions_dir '+ 'directly for unique keys.',+ )++ # Batch-only+ parser.add_argument(+ '--motions_dir', type=str, default=None,+ help='[batch] Directory of motion NPZs (default: sibling "motions/" '+ 'of --face_json).',+ )+ parser.add_argument(+ '--source', type=str, default='empty',+ help='[batch] Only render uuids whose "source" field equals this '+ '(default: "empty"). Use "ALL" to render every key.',+ )+ parser.add_argument(+ '--output_dir', type=str, default=DEFAULT_OUTPUT_DIR,+ help='[batch] Output directory (default: {}).'.format(DEFAULT_OUTPUT_DIR),+ )+ parser.add_argument(+ '--limit', type=int, default=None,+ help='[batch] Stop after N uuids (debug aid).',+ )+ parser.add_argument(+ '--overwrite', action='store_true',+ help='[batch] Re-render even if the output PNG already exists.',+ )++ # Single-only+ parser.add_argument(+ '--output', type=str, default=None,+ help='[single] Output PNG path (default: '+ '<output_dir>/<input_stem>_tpose_annotated.png).',+ )++ # Plot knobs+ parser.add_argument('--font-size', type=int, default=7,+ help='Font size for joint name annotations.')+ parser.add_argument('--joint-marker-size', type=float, default=80.0,+ help='Joint dot marker size in points^2.')+ parser.add_argument('--bone-linewidth', type=float, default=2.0,+ help='Bone line width.')+ args = parser.parse_args()++ plot_kwargs = dict(+ font_size=args.font_size,+ joint_marker_size=args.joint_marker_size,+ bone_linewidth=args.bone_linewidth,+ )++ # โโ Single mode โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ if args.npz_path:+ if not os.path.exists(args.npz_path):+ parser.error("Input NPZ not found: {}".format(args.npz_path))+ out = args.output or default_single_output_path(args.npz_path)+ render_one(args.npz_path, out, **plot_kwargs)+ print("Saved: {}".format(out))+ return++ # โโ Batch mode: resolve the list of keys โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ # face_json takes precedence when explicitly provided; otherwise we scan+ # --motions_dir directly for unique keys.+ if args.face_json:+ if not os.path.exists(args.face_json):+ parser.error("face_json not found: {}".format(args.face_json))+ motions_dir = args.motions_dir or str(Path(args.face_json).parent / "motions")+ if not os.path.isdir(motions_dir):+ parser.error("motions_dir not found: {}".format(motions_dir))+ with open(args.face_json) as f:+ face = json.load(f)+ if args.source == "ALL":+ keys = sorted(face.keys())+ else:+ keys = sorted(+ k for k, v in face.items()+ if isinstance(v, dict) and v.get("source") == args.source+ )+ source_desc = "face_json={} source={!r}".format(args.face_json, args.source)+ else:+ if args.motions_dir is None:+ parser.error("--motions_dir is required in batch mode")+ motions_dir = args.motions_dir+ if not os.path.isdir(motions_dir):+ parser.error("motions_dir not found: {}".format(motions_dir))+ keys = collect_keys_from_motions_dir(motions_dir)+ source_desc = "motions_dir={}".format(motions_dir)++ if args.limit is not None:+ keys = keys[: args.limit]++ if not keys:+ print("No entries selected ({})".format(source_desc))+ return++ os.makedirs(args.output_dir, exist_ok=True)+ print("motions_dir : {}".format(motions_dir))+ print("output_dir : {}".format(args.output_dir))+ print("Rendering {} T-poses ({})".format(len(keys), source_desc))++ rendered = skipped_existing = missing_npz = failed = 0+ missing_examples = [] # type: List[str]+ failed_examples = [] # type: List[str]++ for key in tqdm(keys, unit="seq", dynamic_ncols=True):+ out_path = os.path.join(args.output_dir, "{}.png".format(key))+ if (not args.overwrite) and os.path.exists(out_path):+ skipped_existing += 1+ continue+ npz = find_motion_npz(motions_dir, key)+ if npz is None:+ missing_npz += 1+ if len(missing_examples) < 5:+ missing_examples.append(key)+ continue+ try:+ render_one(str(npz), out_path, **plot_kwargs)+ rendered += 1+ except Exception as e: # noqa: BLE001+ failed += 1+ if len(failed_examples) < 5:+ failed_examples.append("{}: {}".format(key, e))++ print()+ print("rendered : {}".format(rendered))+ print("skipped (exists): {}".format(skipped_existing))+ print("missing NPZ : {}".format(missing_npz))+ if missing_examples:+ for k in missing_examples:+ print(" - {}".format(k))+ if missing_npz > len(missing_examples):+ print(" ... ({} total)".format(missing_npz))+ print("failed : {}".format(failed))+ if failed_examples:+ for line in failed_examples:+ print(" - {}".format(line))+ if failed > len(failed_examples):+ print(" ... ({} total)".format(failed))+++if __name__ == '__main__':+ main()diff --git a/data_process/utils/__init__.py b/data_process/utils/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/data_process/utils/blender_export.py b/data_process/utils/blender_export.pynew file mode 100644index 0000000..ad9276d--- /dev/null+++ b/data_process/utils/blender_export.py@@ -0,0 +1,1230 @@+"""Blender utilities for exporting rigged assets to NPZ motion data.++Format-agnostic helpers shared by the Truebones (FBX), Mixamo (FBX),+Objaverse (GLB/GLTF) and single-asset exporters: scene loading, action+discovery, skeleton/skin extraction, skeleton pruning, Z-up โ Y-up+conversion and NPZ/summary writing.++The counterpart for the *animate* direction (NPZ โ rigged character) lives+in :mod:`data_process.utils.blender_rig`.+"""++import csv+import os+import re+import json+import numpy as np+import bpy+from math import floor, ceil+from pathlib import Path+from loguru import logger+from Animation import Animation, Quaternions, positions_global+from data_process.utils.plotting import save_skeleton_motion, save_skeleton_tpose+from data_process.utils.export_markers import ( # noqa: F401 (re-exported)+ collect_joint_names_from_markers,+ is_asset_complete,+ mark_asset_complete,+ merge_joint_names as _merge_joint_names,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# GLTF I/O+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def import_gltf(path: str):+ bpy.ops.import_scene.gltf(filepath=path)+++def import_fbx(path: str):+ bpy.ops.import_scene.fbx(filepath=path, use_anim=True)+++def action_frame_range(action):+ """Return (start, end) integer frame range for an action."""+ fr = action.frame_range+ start = int(floor(fr[0]))+ end = int(ceil(fr[1]))+ return start, max(start, end)+++def find_main_fbx(fbx_dir):+ """Find the single FBX file that contains all actions for the object type.++ Priority:+ 1. FBX files without '-': if exactly one, use it; if multiple, pick the one+ with 'all' (case-insensitive) in its name.+ 2. If no files without '-', pick the one with '-all' in its name (e.g., Foo-all.fbx).+ """+ fbx_files = [f for f in os.listdir(fbx_dir) if f.lower().endswith('.fbx')]++ # Try files without '-' first+ no_dash = [f for f in fbx_files if '-' not in os.path.splitext(f)[0]]+ if len(no_dash) == 1:+ return os.path.join(fbx_dir, no_dash[0])+ if len(no_dash) > 1:+ with_all = [f for f in no_dash if 'all' in os.path.splitext(f)[0].lower()]+ if len(with_all) >= 1:+ return os.path.join(fbx_dir, with_all[0])++ # Fallback: file with '-all' in name (e.g., Foo-all.fbx)+ dash_all = [f for f in fbx_files if '-all' in os.path.splitext(f)[0].lower()]+ if len(dash_all) >= 1:+ return os.path.join(fbx_dir, dash_all[0])++ return None+++EXCLUDED_CSV = "excluded.csv"+++def load_excluded_stems(input_dir: Path) -> set:+ """Asset stems listed in ``excluded.csv`` beside (or inside) *input_dir*.++ Assets no importer can read โ corrupt rigs, singular transforms, NaN+ axis limits โ are recorded there once so every later stage skips them+ instead of failing on them again on each rerun. The CSV carries a+ header with at least ``file`` or ``object_id``; ``stage`` and ``reason``+ are informational.+ """+ stems = set()+ for candidate in (input_dir / EXCLUDED_CSV, input_dir.parent / EXCLUDED_CSV):+ if not candidate.is_file():+ continue+ with open(candidate, newline="") as f:+ for row in csv.DictReader(f):+ stem = (row.get("object_id") or "").strip()+ if not stem:+ stem = Path((row.get("file") or "").strip()).stem+ if stem:+ stems.add(stem)+ return stems+++def list_gltf_files(input_dir: Path):+ exts = {".glb", ".gltf"}+ paths = sorted(p for p in input_dir.glob("*") if p.is_file() and p.suffix.lower() in exts)+ excluded = load_excluded_stems(input_dir)+ if not excluded:+ return paths+ kept = [p for p in paths if p.stem not in excluded]+ if len(kept) != len(paths):+ logger.info(f"Skipping {len(paths) - len(kept)} asset(s) listed in {EXCLUDED_CSV}")+ return kept+++def action_is_relevant_pose(action) -> bool:+ """Return True if the action has fcurves that animate pose bones."""+ if action is None or len(action.fcurves) == 0:+ return False+ return any(fc.data_path.startswith("pose.bones[") for fc in action.fcurves)+++def discover_pose_actions(min_frames: int = 0, max_frames: int = float('inf')):+ """Find all pose-type actions in the current Blender scene.++ By default no frame-count filter is applied; callers may pass+ *min_frames* / *max_frames* to reject clips outside a desired range.++ Returns:+ (action_names, frame_ranges) where frame_ranges maps action name to+ (start_frame, end_frame) tuples.+ """+ action_names = []+ frame_ranges = {}++ for action_name, action in bpy.data.actions.items():+ if not action_is_relevant_pose(action):+ logger.info(f"Skipping irrelevant action: {action_name}")+ continue++ start, end = action_frame_range(action)+ n_frames = end - start + 1++ if n_frames < min_frames or n_frames > max_frames:+ logger.info(f"Skipping action '{action_name}': {n_frames} frames "+ f"(valid range: [{min_frames}, {max_frames}])")+ continue++ logger.info(f"Found action: {action_name}, frames=[{start},{end}]")++ if action_name not in frame_ranges:+ action_names.append(action_name)+ frame_ranges[action_name] = (start, end)+ else:+ prev_start, prev_end = frame_ranges[action_name]+ frame_ranges[action_name] = (min(start, prev_start), max(end, prev_end))++ return action_names, frame_ranges+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Naming helpers+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def sanitize_object_type(obj_type):+ """Normalize a Truebones object-type directory name.++ Replaces '-' and whitespace with '_' so compound type names (e.g.+ 'Dog-2') don't collide with the object-action separator dash used in+ clip names (e.g. 'Dog_2-Walk').+ """+ return obj_type.replace('-', '_').replace(' ', '_')+++def sanitize_action_name(raw_name):+ """Extract a clean action name from Blender action names.++ Strips Blender prefixes (``Armature|``), colon suffixes (``:Base``),+ known domain prefixes (``mixamo.com_``), then replaces all+ non-alphanumeric characters with ``_``.++ Examples:+ 'Armature|Walk' -> 'Walk'+ 'Armature|Run|Run:Base' -> 'Run'+ 'Idle' -> 'Idle'+ 'Armature|Big-Mouth Attack' -> 'Big_Mouth_Attack'+ 'mixamo.com_Walk' -> 'Walk'+ 'Armature|mixamo.com_Run:Base' -> 'Run'+ """+ parts = raw_name.split('|')+ name = parts[1] if len(parts) > 1 else parts[0]+ name = name.split(':')[0]+ # Remove known domain prefixes+ name = re.sub(r'^mixamo\.com_', '', name)+ # Remove GLTF importer suffixes (e.g. '_GLTF_created_0')+ name = re.sub(r'_GLTF_created_\d+$', '', name)+ # Replace whitespace with '_'+ name = re.sub(r'\s+', '_', name)+ # Replace all non-alphanumeric characters with '_'+ name = re.sub(r'[^a-zA-Z0-9]+', '_', name)+ # Strip leading/trailing underscores+ name = name.strip('_')+ return name+++def truebones_clip_name(fbx_stem):+ """Clip name for one curated Truebones per-clip FBX stem.++ The curated layout is ``{Species}-{Action}.fbx`` with the first dash+ separating the two. Export and rendering must derive the same name from+ it or stage 4's exact-match caption lookup silently yields an empty+ caption, so both go through this one helper.++ Raises:+ ValueError: when the stem has no ``-`` to split on.+ """+ species, sep, action = fbx_stem.partition('-')+ if not sep:+ raise ValueError(+ f"Truebones clip filename '{fbx_stem}' has no '-' separating "+ f"species from action; expected '{{Species}}-{{Action}}.fbx'")+ return f"{sanitize_object_type(species)}-{sanitize_action_name(action)}"+++def unique_clip_names(action_names):+ """Map each action name to a collision-free sanitized clip suffix.++ ``sanitize_action_name`` keeps only the second ``|``-separated segment,+ so distinct actions routinely collapse onto one name โ e.g.+ ``Armature|Take 001|BaseLayer_Obj`` and+ ``Armature|Take 001|BaseLayer.002_Obj`` both yield ``Take_001``. Export+ then overwrites ``motions/{clip}.npz`` (last wins) while the renderer's+ completeness check skips the later actions (first wins), so the NPZ and+ the render under one clip name end up being different animations and the+ caption describes the wrong motion.++ Colliding names get a ``_00``/``_01``โฆ suffix in list order. Callers on+ both sides must pass the SAME ordered action list (the full set of pose+ actions, before any frame-count filtering) or the suffixes diverge.++ Args:+ action_names: Ordered action names, as returned by+ :func:`discover_pose_actions`.++ Returns:+ ``{action_name: clip_suffix}``.+ """+ bases = [sanitize_action_name(n) or "action" for n in action_names]+ total = {}+ for base in bases:+ total[base] = total.get(base, 0) + 1++ seen = {}+ mapping = {}+ for action_name, base in zip(action_names, bases):+ if total[base] == 1:+ mapping[action_name] = base+ else:+ idx = seen.get(base, 0)+ seen[base] = idx + 1+ mapping[action_name] = f"{base}_{idx:02d}"+ return mapping+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Constants (skinning threshold and Blender rigging naming conventions)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++_SKINNING_WEIGHT_EPS = 1e-5++_CONTROL_PREFIXES = ('MCH-', 'ORG-', 'VIS-')+_CONTROL_SUFFIXES_LOWER = (+ '_ik', '.ik', '_pole', '.pole',+ '_target', '_tip',+ '_ctrl', '.ctrl',+ '_tweak', '_helper',+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Blender scene utilities+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def reset_scene():+ bpy.ops.wm.read_factory_settings(use_empty=True)+++def set_scene_fps(fps: int):+ bpy.context.scene.render.fps = int(fps)+++def find_armatures():+ return [o for o in bpy.data.objects if o.type == "ARMATURE"]+++def find_meshes():+ return [o for o in bpy.data.objects if o.type == "MESH"]+++def choose_main_armature(armatures):+ """Heuristic: armature with most bones."""+ if not armatures:+ return None+ return sorted(armatures, key=lambda a: len(a.data.bones), reverse=True)[0]+++def choose_main_mesh(meshes):+ """Heuristic: mesh with most vertices."""+ if not meshes:+ return None+ return sorted(meshes, key=lambda m: len(m.data.vertices), reverse=True)[0]+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Animation state management+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def clear_animation_state(arm_obj: bpy.types.Object):+ """Clear action and NLA tracks to avoid cross-action contamination."""+ ad = arm_obj.animation_data+ if ad is None:+ return+ ad.action = None+ if ad.nla_tracks:+ for tr in list(ad.nla_tracks):+ ad.nla_tracks.remove(tr)+++def bind_action(arm_obj: bpy.types.Object, action: bpy.types.Action):+ if arm_obj.animation_data is None:+ arm_obj.animation_data_create()+ arm_obj.animation_data.action = action+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Skinning utilities+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def build_vgroup_to_bone_mapping(mesh_obj, arm_obj):+ """Return dict {vgroup_index -> bone_name} for groups that match a bone."""+ bone_names = set(arm_obj.data.bones.keys())+ vg_to_bone = {}+ for i, vg in enumerate(mesh_obj.vertex_groups):+ if vg.name in bone_names:+ vg_to_bone[i] = vg.name+ return vg_to_bone+++def build_vertex_groups_sparse(mesh_obj, vg_to_bone: dict, bone_name_to_index: dict, dtype=np.float64):+ """+ Returns:+ vert_groups: len=V list[list[(bone_idx, weight)]]+ skin_matrix: (V, B) dense skinning weight matrix+ """+ verts = mesh_obj.data.vertices+ V = len(verts)+ vert_groups = [[] for _ in range(V)]+ skin_matrix = np.zeros((V, len(bone_name_to_index)), dtype=dtype)++ for i, vert in enumerate(verts):+ groups = []+ for g in vert.groups:+ w = float(g.weight)+ if w <= 0.0:+ continue+ bname = vg_to_bone.get(g.group)+ if bname is None:+ continue+ bidx = bone_name_to_index.get(bname)+ if bidx is None:+ continue+ skin_matrix[i, bidx] = w+ groups.append((int(bidx), w))+ vert_groups[i] = groups++ return vert_groups, skin_matrix+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Skeleton topology utilities+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def children_from_parents(parents):+ ch = [[] for _ in range(len(parents))]+ for j, p in enumerate(parents):+ if p >= 0:+ ch[p].append(j)+ return ch+++def name_is_control_bone(name: str) -> bool:+ return (+ any(name.startswith(p) for p in _CONTROL_PREFIXES)+ or any(name.lower().endswith(s) for s in _CONTROL_SUFFIXES_LOWER)+ )+++def rotation_amplitude(j, anim):+ """+ Maximum rotation angle (radians) of joint j relative to its first-frame+ orientation, across all frames.+ """+ rot_0 = anim.rotations[0:1, j]+ delta = (-rot_0) * anim.rotations[:, j]+ cos_half = np.abs(delta.qs[:, 0]).clip(0.0, 1.0)+ return float(np.max(2.0 * np.arccos(cos_half)))+++def prune_secondary_roots(armature, name_to_weight_sum):+ def get_subtree_weight(bone, weight_dict):+ w = weight_dict.get(bone.name, 0)+ for child in bone.children:+ w += get_subtree_weight(child, weight_dict)+ return w++ all_roots = [bone for bone in armature.data.bones if bone.parent is None]+ if len(all_roots) <= 1:+ logger.info("Only one or no root bone found. Skipping prune_secondary_roots.")+ return++ root_weights = []+ for rb in all_roots:+ total_w = get_subtree_weight(rb, name_to_weight_sum)+ root_weights.append((rb.name, total_w))+ logger.info(f"Root bone '{rb.name}' has total subtree weight: {total_w:.4f}")++ best_root_name = max(root_weights, key=lambda x: x[1])[0]+ logger.info(f"Keeping best root: '{best_root_name}'")++ bones_to_remove = []+ for rb_name, _ in root_weights:+ if rb_name != best_root_name:+ rb_obj = armature.data.bones.get(rb_name)+ if rb_obj:+ bones_to_remove.extend([rb_obj.name] + [b.name for b in rb_obj.children_recursive])++ if bones_to_remove:+ logger.info(f"Removing {len(bones_to_remove)} bones from secondary root subtrees.")+ bpy.context.view_layer.objects.active = armature+ bpy.ops.object.mode_set(mode='EDIT')+ eb = armature.data.edit_bones+ for b_name in bones_to_remove:+ if b_name in eb:+ eb.remove(eb[b_name])+ bpy.ops.object.mode_set(mode='OBJECT')+ bpy.context.view_layer.update()+++def collapse_unused_root(anim: Animation):+ """Remove an unskinned root with a single child; the child becomes the new root.++ Uses direct local-space composition instead of global FK round-trip+ to avoid floating-point drift.+ """+ parents = anim.parents.copy()+ F, J = anim.positions.shape[:2]+ assert J == len(parents)++ root_ids = np.where(np.array(parents) == -1)[0]+ assert len(root_ids) == 1, "Multiple root joints found."+ root_id = root_ids[0]++ all_children = children_from_parents(parents)+ assert len(all_children[root_id]) == 1, "Root must have exactly one child."+ true_root_id = all_children[root_id][0]++ def get_all_descendants(idx):+ res = [idx]+ for c in all_children[idx]:+ res.extend(get_all_descendants(c))+ return res++ keep = sorted(get_all_descendants(true_root_id))+ new_index = {old: new for new, old in enumerate(keep)}+ newJ = len(keep)++ # Build new parent array+ new_parents = np.full((newJ,), -1, dtype=np.int32)+ for old_j in keep:+ if old_j != true_root_id:+ new_parents[new_index[old_j]] = new_index[parents[old_j]]++ # Copy local transforms for all kept bones+ new_rot_qs = anim.rotations.qs[:, keep].copy()+ new_pos = anim.positions[:, keep].copy()+ new_offsets = anim.offsets[keep].copy()++ # Compose root transform into the new root (former child):+ # new_child_local = old_root_local * old_child_local+ new_root_idx = new_index[true_root_id]+ root_rot = anim.rotations[:, root_id]+ new_rot_qs[:, new_root_idx] = (root_rot * anim.rotations[:, true_root_id]).qs+ new_pos[:, new_root_idx] = (+ anim.positions[:, root_id] + root_rot * anim.positions[:, true_root_id]+ )++ # Compose rest-pose offset+ rest_root_rot = Quaternions(anim.rotations.qs[0:1, root_id])+ new_offsets[new_root_idx] = (+ anim.offsets[root_id] + (rest_root_rot * anim.offsets[true_root_id:true_root_id+1])[0]+ )++ new_anim = Animation(+ rotations=Quaternions(new_rot_qs),+ positions=new_pos,+ orients=anim.orients[keep].copy(),+ offsets=new_offsets,+ parents=new_parents,+ )+ return new_anim, keep+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Skeleton pruning+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++# --- Prunable bone detection ------------------------------------------------++def _is_prunable(j, skin_matrix, anims_list, rest_anim, names,+ offset_eps, pos_var_eps, rot_eps,+ mean_bone_len=1.0, is_leaf=True,+ consider_parent_rotate=True):+ """+ Return True if bone j should be removed from the skeleton.+ Works with one or more animation clips for aggregated decisions.++ Conditions (checked in order, first match wins):++ (0) Name convention โ Blender control/mechanism prefix or suffix.+ (1) Near-zero offset โ bone head coincident with parent in rest pose.+ (2) Near-zero positionโ bone stays at parent origin in ALL clips.+ (3) Non-rigid (leaf) โ bone length varies from rest in ANY clip;+ likely an IK-stretch control.+ (4) Zero rotation DOF โ bone never rotates above threshold in ALL clips.+ For leaf bones, when *consider_parent_rotate* is True, only prune+ if the parent also doesn't rotate in ANY clip โ a static leaf on a+ rotating parent is a meaningful end-effector (e.g. spider toe tip).+ When False, prune such leaves regardless of parent motion.++ Distance conditions are normalized by mean_bone_len (scale-agnostic).+ """+ # Skinned bones are never prunable+ if skin_matrix[:, j].max() >= _SKINNING_WEIGHT_EPS:+ return False++ if name_is_control_bone(names[j]): # (0)+ return True++ norm = max(mean_bone_len, 1e-9)+ rest_pos_j = rest_anim.positions[0, j]++ if np.linalg.norm(rest_pos_j) / norm < offset_eps: # (1)+ return True++ if all(np.max(np.linalg.norm(a.positions[:, j], axis=-1)) / norm # (2)+ < offset_eps for a in anims_list):+ return True++ if is_leaf: # (3)+ if any(np.max(np.linalg.norm(a.positions[:, j] - rest_pos_j,+ axis=-1)) / norm > pos_var_eps+ for a in anims_list):+ return True++ if rot_eps > 0.0 and all(rotation_amplitude(j, a) < rot_eps # (4)+ for a in anims_list):+ if is_leaf and consider_parent_rotate:+ parent_j = anims_list[0].parents[j]+ if parent_j >= 0 and any(rotation_amplitude(parent_j, a) >= rot_eps+ for a in anims_list):+ return False # parent rotates โ keep this end-effector+ return True++ return False+++# --- Bone merge operation ---------------------------------------------------++def _merge_passthrough_bone(anims_list, rest_anim, names, skin_matrix, j):+ """Collapse pass-through bone j into its single child across all clips.++ The child inherits j's transform so that global positions are preserved.+ """+ parents = list(anims_list[0].parents)+ childrens = children_from_parents(parents)+ c = childrens[j][0]++ # Compose rest-pose offset: child_new = j_offset + rot_j * child_offset+ new_offsets = anims_list[0].offsets.copy()+ rest_rot_j0 = Quaternions(rest_anim.rotations.qs[0:1, j])+ new_offsets[c] = new_offsets[j] + (rest_rot_j0 * new_offsets[c:c+1])[0]++ # Re-parent child to j's parent, build index mapping+ new_parents = list(parents)+ new_parents[c] = parents[j]+ keep_idxs = [i for i in range(len(parents)) if i != j]+ old_to_new = {old: new for new, old in enumerate(keep_idxs)}+ final_parents = np.array(+ [old_to_new[new_parents[i]] if new_parents[i] >= 0 else -1+ for i in keep_idxs],+ dtype=np.int32,+ )++ # Update rest anim+ rest_rot_j = rest_anim.rotations[:, j]+ new_rest_pos = rest_anim.positions.copy()+ new_rest_rots = rest_anim.rotations.qs.copy()+ new_rest_pos[:, c] = rest_anim.positions[:, j] + rest_rot_j * rest_anim.positions[:, c]+ new_rest_rots[:, c] = (rest_rot_j * rest_anim.rotations[:, c]).qs+ new_rest_anim = Animation(+ rotations=Quaternions(new_rest_rots[:, keep_idxs]),+ positions=new_rest_pos[:, keep_idxs],+ orients=rest_anim.orients[keep_idxs],+ offsets=new_offsets[keep_idxs],+ parents=final_parents,+ )++ # Update each animation clip+ new_anims = []+ for anim in anims_list:+ rot_j = anim.rotations[:, j]+ new_anim_pos = anim.positions.copy()+ new_anim_rots = anim.rotations.qs.copy()+ new_anim_pos[:, c] = anim.positions[:, j] + rot_j * anim.positions[:, c]+ new_anim_rots[:, c] = (rot_j * anim.rotations[:, c]).qs+ new_anims.append(Animation(+ rotations=Quaternions(new_anim_rots[:, keep_idxs]),+ positions=new_anim_pos[:, keep_idxs],+ orients=anim.orients[keep_idxs],+ offsets=new_offsets[keep_idxs],+ parents=final_parents,+ ))++ new_names = [names[i] for i in keep_idxs]+ new_skin = skin_matrix[:, keep_idxs]+ return new_anims, new_rest_anim, new_names, new_skin+++# --- Helpers ----------------------------------------------------------------++def _compute_mean_bone_len(rest_anim):+ """Mean bone length from rest-pose offsets, excluding root."""+ bone_lens = np.linalg.norm(rest_anim.offsets[1:], axis=-1)+ valid_lens = bone_lens[bone_lens > 1e-6]+ return float(valid_lens.mean()) if len(valid_lens) > 0 else 1.0+++def _find_deepest_in_chain(j, children_map, cand_set):+ """Walk from candidate j down the single-child chain to the deepest candidate."""+ while children_map[j][0] in cand_set:+ j = children_map[j][0]+ return j+++def _find_root(parents):+ """Return the index of the root joint (parent == -1)."""+ return int(np.where(np.array(parents) == -1)[0][0])+++# --- Pruning passes --------------------------------------------------------++def _pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,+ offset_eps, pos_var_eps, rot_eps, min_joints,+ consider_parent_rotate=True):+ """Pass 1: iteratively remove prunable leaf bones."""+ count = 0+ mbl = _compute_mean_bone_len(rest_anim)+ while True:+ n_joints = len(anims_list[0].parents)+ childrens = children_from_parents(anims_list[0].parents)+ candidates = [j for j in range(n_joints)+ if len(childrens[j]) == 0+ and _is_prunable(j, skin_matrix, anims_list, rest_anim, names,+ offset_eps, pos_var_eps, rot_eps,+ mean_bone_len=mbl, is_leaf=True,+ consider_parent_rotate=consider_parent_rotate)]+ if not candidates:+ break+ keep = [j for j in range(n_joints) if j not in set(candidates)]+ if len(keep) < min_joints:+ break+ logger.info(f"Pruning leaf bones: {[names[j] for j in candidates]}")+ anims_list = [a[:, keep].copy() for a in anims_list]+ rest_anim = rest_anim[:, keep].copy()+ names = [names[j] for j in keep]+ skin_matrix = skin_matrix[:, keep]+ count += len(candidates)+ return anims_list, rest_anim, names, skin_matrix, count+++def _pass_promote_root(anims_list, rest_anim, names, skin_matrix):+ """Pass 2: collapse unskinned single-child root into its child."""+ count = 0+ while True:+ root_idx = _find_root(anims_list[0].parents)+ children = children_from_parents(anims_list[0].parents)[root_idx]+ if not (len(children) == 1+ and skin_matrix[:, root_idx].max() < _SKINNING_WEIGHT_EPS):+ break+ logger.info(f"Promoting root child (removing '{names[root_idx]}')")+ new_anims = []+ keep_idxs = None+ for anim in anims_list:+ new_anim, ki = collapse_unused_root(anim)+ if keep_idxs is None:+ keep_idxs = ki+ else:+ assert ki == keep_idxs, "Root-clean keep_idxs mismatch across clips"+ new_anims.append(new_anim)+ rest_anim, rest_ki = collapse_unused_root(rest_anim)+ assert rest_ki == keep_idxs, "Root-clean keep_idxs mismatch for rest_anim"+ anims_list = new_anims+ names = [names[j] for j in keep_idxs]+ skin_matrix = skin_matrix[:, keep_idxs]+ count += 1+ return anims_list, rest_anim, names, skin_matrix, count+++def _pass_merge_passthrough(anims_list, rest_anim, names, skin_matrix,+ offset_eps, pos_var_eps, rot_eps, min_joints):+ """Pass 3: merge prunable single-child (non-root) bones, deepest first."""+ count = 0+ mbl = _compute_mean_bone_len(rest_anim)+ while True:+ n_joints = len(anims_list[0].parents)+ childrens = children_from_parents(anims_list[0].parents)+ root_idx = _find_root(anims_list[0].parents)+ candidates = [j for j in range(n_joints)+ if j != root_idx+ and len(childrens[j]) == 1+ and _is_prunable(j, skin_matrix, anims_list, rest_anim, names,+ offset_eps, pos_var_eps, rot_eps,+ mean_bone_len=mbl, is_leaf=False)]+ if not candidates:+ break+ cand_set = set(candidates)+ j = _find_deepest_in_chain(candidates[0], childrens, cand_set)+ logger.info(f"Merging pass-through bone '{names[j]}' into its child")+ anims_list, rest_anim, names, skin_matrix = \+ _merge_passthrough_bone(anims_list, rest_anim, names, skin_matrix, j)+ count += 1+ return anims_list, rest_anim, names, skin_matrix, count+++# --- Main entry points ------------------------------------------------------++def prune_skeleton_shared(anims_list, rest_anim, names, skin_matrix,+ offset_eps=1e-3, pos_var_eps=1e-3,+ rot_eps=np.deg2rad(2.0), min_joints=4,+ consider_parent_rotate=True):+ """Iterative skeleton pruning across multiple animation clips.++ Guarantees identical topology across all clips. Three passes per+ iteration, repeated until no pass makes progress:++ 1. Remove prunable leaf bones+ 2. Promote child when root is unskinned with single child+ 3. Merge prunable pass-through bones into their children++ Args:+ consider_parent_rotate: If True (default), leaf bones that never+ rotate are preserved when their parent rotates โ treating them+ as meaningful end-effectors (e.g. spider toe tips). If False,+ such leaves are pruned regardless of parent motion.+ """+ stats = {"leaf": 0, "root_promoted": 0, "passthrough": 0}++ while len(anims_list[0].parents) > min_joints:+ anims_list, rest_anim, names, skin_matrix, n_leaf = \+ _pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,+ offset_eps, pos_var_eps, rot_eps, min_joints,+ consider_parent_rotate=consider_parent_rotate)++ anims_list, rest_anim, names, skin_matrix, n_root = \+ _pass_promote_root(anims_list, rest_anim, names, skin_matrix)++ anims_list, rest_anim, names, skin_matrix, n_pass = \+ _pass_merge_passthrough(anims_list, rest_anim, names, skin_matrix,+ offset_eps, pos_var_eps, rot_eps, min_joints)++ stats["leaf"] += n_leaf+ stats["root_promoted"] += n_root+ stats["passthrough"] += n_pass++ if n_leaf + n_root + n_pass == 0:+ break++ logger.info(f"Pruning summary: {stats}, final joints: {len(anims_list[0].parents)}")+ return anims_list, rest_anim, names, skin_matrix+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Motion filtering (T-pose frame removal)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def remove_tpose_frames(anim, rest_anim, rot_thresh=np.deg2rad(1.0)):+ """Remove frames whose non-root joint rotations match the rest pose.++ For each frame, computes the maximum rotation angle (over all non-root+ joints) between the frame's local rotation and the rest-pose rotation.+ Frames where this maximum is below *rot_thresh* are considered T-pose+ duplicates and removed.++ Args:+ anim: Animation with shape (F, J, ...).+ rest_anim: Single-frame rest-pose Animation (1, J, ...).+ rot_thresh: Maximum allowed rotation deviation (radians) for a+ frame to be considered identical to the T-pose.++ Returns:+ (pruned_anim, n_removed) โ the filtered Animation and the count+ of removed frames. Returns the original if no frames are removed+ or if removing them would leave fewer than 2 frames.+ """+ F, J = anim.rotations.qs.shape[:2]+ if J < 2:+ return anim, 0++ # rest rotation for non-root joints: (1, J-1, 4)+ rest_rot = rest_anim.rotations[:, 1:] # Quaternions (1, J-1)+ frame_rot = anim.rotations[:, 1:] # Quaternions (F, J-1)++ # relative rotation: rest^-1 * frame โ identity when poses match+ delta = (-rest_rot) * frame_rot # (F, J-1)+ # angle = 2 * arccos(|w|)+ cos_half = np.abs(delta.qs[..., 0]).clip(0.0, 1.0) # (F, J-1)+ angles = 2.0 * np.arccos(cos_half) # (F, J-1)++ max_angle_per_frame = angles.max(axis=1) # (F,)+ keep_mask = max_angle_per_frame >= rot_thresh++ n_removed = int((~keep_mask).sum())+ if n_removed == 0:+ return anim, 0++ keep_indices = np.where(keep_mask)[0]+ if len(keep_indices) < 2:+ # Don't strip if it would leave fewer than 2 frames+ return anim, 0++ pruned = Animation(+ rotations=Quaternions(anim.rotations.qs[keep_indices]),+ positions=anim.positions[keep_indices],+ orients=anim.orients.copy(),+ offsets=anim.offsets.copy(),+ parents=anim.parents.copy(),+ )+ return pruned, n_removed+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Animation extraction helpers+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def extract_rest_pose(anim_data_bones, armature, dtype=np.float64, apply_world=True):+ """Extract local rest-pose positions and rotations for all bones."""+ nbones = len(anim_data_bones)+ rest_local_pos = np.zeros((nbones, 3), dtype=dtype)+ rest_local_rot = np.zeros((nbones, 4), dtype=dtype)++ for idxb, bone in enumerate(anim_data_bones):+ if bone.parent:+ mat = bone.parent.matrix_local.inverted() @ bone.matrix_local+ rest_local_pos[idxb] = np.array(mat.to_translation(), dtype=dtype)+ rest_local_rot[idxb] = np.array(mat.to_quaternion(), dtype=dtype)+ else:+ if apply_world:+ mat = armature.matrix_world @ bone.matrix_local+ scale = np.array(armature.matrix_world.to_scale(), dtype=dtype)+ rest_local_pos[idxb] = np.array(mat.to_translation(), dtype=dtype) / scale+ else:+ mat = bone.matrix_local+ rest_local_pos[idxb] = np.array(mat.to_translation(), dtype=dtype)+ rest_local_rot[idxb] = np.array(mat.to_quaternion(), dtype=dtype)++ return rest_local_pos, rest_local_rot+++def extract_animation_frames(armature, start_frame, end_frame, nbones, dtype=np.float64, apply_world=True):+ """Scrub through frames and extract per-frame local bone transforms."""+ nframes = end_frame - start_frame + 1+ anim_local_pos = np.zeros((nframes, nbones, 3), dtype=dtype)+ anim_local_rot = np.zeros((nframes, nbones, 4), dtype=dtype)++ pose_bones = armature.pose.bones+ assert len(pose_bones) > 0, f"No pose bones in armature {armature.name}"++ for idxf, f in enumerate(range(start_frame, end_frame + 1)):+ bpy.context.scene.frame_set(f) # evaluates the depsgraph++ for idxb, bone in enumerate(pose_bones):+ if bone.parent:+ mat = bone.parent.matrix.inverted() @ bone.matrix+ anim_local_pos[idxf, idxb] = np.array(mat.to_translation(), dtype=dtype)+ anim_local_rot[idxf, idxb] = np.array(mat.to_quaternion(), dtype=dtype)+ else:+ if apply_world:+ mat = armature.matrix_world @ bone.matrix+ scale = np.array(armature.matrix_world.to_scale(), dtype=dtype)+ anim_local_pos[idxf, idxb] = np.array(mat.to_translation(), dtype=dtype) / scale+ else:+ mat = bone.matrix+ anim_local_pos[idxf, idxb] = np.array(mat.to_translation(), dtype=dtype)+ anim_local_rot[idxf, idxb] = np.array(mat.to_quaternion(), dtype=dtype)++ return anim_local_pos, anim_local_rot+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Skeleton setup (Blender scene โ numpy arrays)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def compute_bone_weight_sums(mesh, armature):+ """Sum skin weights per bone across all vertices."""+ vg_to_bone = build_vgroup_to_bone_mapping(mesh, armature)+ weight_sums = {bname: 0.0 for bname in armature.data.bones.keys()}+ for v in mesh.data.vertices:+ for g in v.groups:+ bname = vg_to_bone.get(g.group)+ if bname:+ weight_sums[bname] += g.weight+ return weight_sums+++def build_skeleton_arrays(armature, mesh, dtype=np.float64, apply_world=True):+ """Extract skeleton topology, rest pose, and skin matrix from Blender objects.++ Returns:+ bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix+ """+ bones = armature.data.bones+ bone_names = list(bones.keys())+ bone_name_to_index = {b.name: i for i, b in enumerate(bones)}++ if mesh is None:+ skin_matrix = np.zeros((0, len(bone_names)), dtype=dtype)+ else:+ vg_to_bone = build_vgroup_to_bone_mapping(mesh, armature)+ _, skin_matrix = build_vertex_groups_sparse(mesh, vg_to_bone, bone_name_to_index)++ parents_array = np.array([+ -1 if bone.parent is None else bones.find(bone.parent.name)+ for bone in bones+ ], dtype=np.int32)++ rest_local_pos, rest_local_rot = extract_rest_pose(bones, armature, dtype, apply_world=apply_world)++ return bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix+++def build_rest_animation(parents_array, rest_local_pos, rest_local_rot, nbones):+ """Build a single-frame Animation object from rest pose arrays."""+ return Animation(+ rotations=Quaternions(rest_local_rot[None]),+ positions=rest_local_pos[None],+ orients=Quaternions.id(nbones),+ offsets=rest_local_pos,+ parents=parents_array,+ )+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Coordinate conversion (Z-up โ Y-up)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _apply_root_rotation(anim, root_quat):+ """Apply *root_quat* to the root joint of an Animation (rotations, positions, offsets)."""+ rots = anim.rotations.copy()+ pos = anim.positions.copy()+ off = anim.offsets.copy()+ rots[:, 0] = root_quat * anim.rotations[:, 0]+ pos[:, 0] = root_quat * anim.positions[:, 0]+ off[0] = (root_quat * anim.offsets[0:1])[0]+ return Animation(+ rotations=rots, positions=pos,+ orients=anim.orients, offsets=off, parents=anim.parents,+ )+++def apply_zup_to_yup(anim, rest_anim):+ """Rotate root bone -90 deg around X to convert Z-up to Y-up coordinate system."""+ root_quat = Quaternions.from_euler(np.array([[-np.pi / 2, 0, 0]]))+ return _apply_root_rotation(anim, root_quat), _apply_root_rotation(rest_anim, root_quat)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# High-level pipeline helpers (shared by FBX and GLB exporters)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def load_scene(import_fn, path, fps=30):+ """Reset scene, set FPS, import via *import_fn*, select main armature/mesh.++ Returns (armature, mesh). Mesh may be None if none found.+ """+ reset_scene()+ set_scene_fps(fps=fps)+ import_fn(str(path))++ armatures = find_armatures()+ assert len(armatures) > 0, f"No armature found in {path}."+ armature = choose_main_armature(armatures)+ logger.info(f"Using armature: {armature.name}, bones={len(armature.data.bones)}")++ meshes = find_meshes()+ mesh = choose_main_mesh(meshes) if meshes else None+ if mesh is not None:+ logger.info(f"Using mesh: {mesh.name}, vertices={len(mesh.data.vertices)}")++ return armature, mesh+++def prepare_skeleton(armature, mesh=None, dtype=np.float64, apply_world=True):+ """Prune secondary roots, build skeleton arrays, and ZโY rest poses.++ When *mesh* is None (e.g. Mixamo clips without a character mesh), the+ secondary-root prune step is skipped and skin_matrix is returned empty.++ Returns a dict with keys: nbones, bone_names, parents_array,+ rest_local_pos, rest_local_rot, skin_matrix, rest_anim, rest_anim_shared.+ """+ if mesh is not None:+ weight_sums = compute_bone_weight_sums(mesh, armature)+ prune_secondary_roots(armature, weight_sums)+ bpy.context.view_layer.update()++ nbones = len(armature.data.bones)+ bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix = \+ build_skeleton_arrays(armature, mesh, dtype, apply_world=apply_world)++ rest_anim = build_rest_animation(parents_array, rest_local_pos, rest_local_rot, nbones)+ _, rest_anim_shared = apply_zup_to_yup(rest_anim, rest_anim)++ return {+ 'nbones': nbones,+ 'bone_names': bone_names,+ 'parents_array': parents_array,+ 'rest_local_pos': rest_local_pos,+ 'rest_local_rot': rest_local_rot,+ 'skin_matrix': skin_matrix,+ 'rest_anim': rest_anim,+ 'rest_anim_shared': rest_anim_shared,+ }+++def extract_all_actions(armature, frame_ranges, rest_local_pos, parents_array,+ rest_anim, nbones, dtype=np.float64, apply_world=True):+ """Extract (name, Animation) for every action in *frame_ranges*.++ Args:+ frame_ranges: dict mapping action_name -> (start_frame, end_frame).++ Applies Z-up โ Y-up conversion to each extracted clip.+ """+ extracted = []+ for action_name, (start_frame, end_frame) in frame_ranges.items():+ action = bpy.data.actions.get(action_name)+ if action is None:+ logger.warning(f"Action '{action_name}' not found; skipping.")+ continue++ clear_animation_state(armature)+ bind_action(armature, action)++ anim_local_pos, anim_local_rot = extract_animation_frames(+ armature, start_frame, end_frame, nbones, dtype, apply_world=apply_world+ )++ anim = Animation(+ rotations=Quaternions(anim_local_rot),+ positions=anim_local_pos,+ orients=Quaternions.id(nbones),+ offsets=rest_local_pos,+ parents=parents_array,+ )+ anim, _ = apply_zup_to_yup(anim, rest_anim)+ extracted.append((action_name, anim))++ return extracted+++def save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_pruned):+ """Render and save the rest-pose skeleton PNG for one asset."""+ os.makedirs(tpos_save_dir, exist_ok=True)+ tpos_path = os.path.join(tpos_save_dir, f"{save_name}.png")+ rest_global_pos = positions_global(rest_anim_pruned)+ save_skeleton_tpose(+ save_path=tpos_path,+ parents=rest_anim_pruned.parents,+ positions=rest_global_pos[0],+ )+ logger.info(f"Saved rest pose visualization: {tpos_path}")+ return tpos_path+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Shared NPZ save and summary helpers+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def save_motion(motion_path, vis_path, anim, rest_anim, names, skin_matrix, fps,+ action_name="", save_vis=True):+ """Save motion NPZ (float64) and optional MP4.++ The MP4 preview dominates per-clip export time; pass ``save_vis=False``+ to skip it on bulk runs.+ """+ if save_vis:+ global_pos = positions_global(anim)+ save_skeleton_motion(+ save_path=vis_path,+ parents=anim.parents,+ positions=global_pos,+ fps=fps,+ )+ logger.info(f"Saved visualization: {vis_path}")++ np.savez(+ motion_path,+ rest_local_pos=rest_anim.positions[0],+ rest_local_rot=rest_anim.rotations[0].qs,+ anim_local_pos=anim.positions,+ anim_local_rot=anim.rotations.qs,+ offsets=anim.offsets,+ names=np.array(names),+ skin_matrix=skin_matrix,+ fps=fps,+ parents=anim.parents,+ action_name=action_name,+ )+ logger.info(f"Saved motion data: {motion_path}")+++def scan_clip_frames(motion_dir):+ """Scan saved NPZ files in motion_dir and return {clip_name: nframes}."""+ frames = {}+ if not os.path.isdir(motion_dir):+ return frames+ for npz_file in sorted(os.listdir(motion_dir)):+ if not npz_file.endswith('.npz'):+ continue+ clip_name = os.path.splitext(npz_file)[0]+ try:+ data = np.load(os.path.join(motion_dir, npz_file), allow_pickle=True)+ frames[clip_name] = int(data['anim_local_rot'].shape[0])+ except Exception:+ pass+ return frames+++def write_clip_frames_json(path, frames, fps=30):+ """Write clip_frames.json: a pure {clip_name: nframes} mapping."""+ total_frames = sum(frames.values())+ total_clips = len(frames)+ total_duration_sec = total_frames / fps if fps else 0.0+ with open(path, 'w') as f:+ json.dump(frames, f, indent=2)+ logger.info(f"Saved clip frames ({total_clips} clips, {total_frames} frames, "+ f"{total_duration_sec / 60:.1f} min) to {path}")+++def write_summary_json(path, frames, fps=30):+ """Write summary.json with the dataset-level summary of the export."""+ total_frames = sum(frames.values())+ total_duration_sec = total_frames / fps if fps else 0.0+ with open(path, 'w') as f:+ json.dump({+ "total_clips": len(frames),+ "total_frames": total_frames,+ "total_duration_sec": round(total_duration_sec, 2),+ "total_duration_min": round(total_duration_sec / 60, 2),+ "fps": fps,+ }, f, indent=2)+ logger.info(f"Saved export metadata to {path}")+++def write_export_summary(output_dir, all_joint_names, fps=30, worker_suffix=""):+ """Write per-run summary JSONs: clip_frames, joint_names, joint_count,+ and (for non-sharded runs) summary.json with the dataset-level summary.++ ``all_joint_names`` only covers assets exported by *this* run โ assets+ skipped via their completion marker return nothing. Writing it verbatim+ would therefore truncate ``joint_names.json`` on every resumed run, so+ for the canonical (non-sharded) file it is merged over the durable+ per-asset markers and whatever the file already held. Shard files stay+ run-local; :func:`~data_process.tools.merge_summaries.merge_summaries`+ does the same union for them.++ Args:+ output_dir: Root directory containing the "motions/" NPZ folder.+ all_joint_names: dict mapping obj_type/save_name -> list[str] bone names.+ fps: Frame rate used for duration summary.+ worker_suffix: Optional suffix (e.g. "_worker0") appended to filenames.+ When set, summary.json is left to merge_summaries so+ parallel workers never race on it.+ """+ frames = scan_clip_frames(os.path.join(output_dir, "motions"))+ write_clip_frames_json(+ os.path.join(output_dir, f"clip_frames{worker_suffix}.json"),+ frames, fps=fps,+ )+ if not worker_suffix:+ write_summary_json(os.path.join(output_dir, "summary.json"), frames, fps=fps)++ joint_names_path = os.path.join(output_dir, f"joint_names{worker_suffix}.json")+ if worker_suffix:+ merged = dict(all_joint_names)+ else:+ merged = _merge_joint_names(output_dir, all_joint_names, log=logger.info)+ with open(joint_names_path, 'w') as f:+ json.dump(merged, f, indent=2)+ logger.info(f"Saved joint names ({len(merged)} entries) to {joint_names_path}")+ all_joint_names = merged++ joint_count = {k: len(v) for k, v in all_joint_names.items()}+ joint_count_path = os.path.join(output_dir, f"joint_count{worker_suffix}.json")+ with open(joint_count_path, 'w') as f:+ json.dump(joint_count, f, indent=2)+ logger.info(f"Saved joint count to {joint_count_path}")+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Per-asset completion markers (resume bookkeeping)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+#+# Re-exported from data_process.utils.export_markers, which is deliberately+# free of any bpy import so plain-Python tools (merge_summaries) can use it.++__all__ = [n for n in dir() if not n.startswith('_')] + [+ "is_asset_complete", "mark_asset_complete",+ "collect_joint_names_from_markers", "merge_joint_names",+]diff --git a/data_process/utils/blender_render.py b/data_process/utils/blender_render.pynew file mode 100644index 0000000..5c8a25b--- /dev/null+++ b/data_process/utils/blender_render.py@@ -0,0 +1,1190 @@+"""EEVEE multi-view rendering utilities (pip ``bpy`` module).++Shared by the :mod:`data_process.motion_rendering` stage: render-engine+init, scene reset, asset import, material fixes, studio lighting, scene+normalization, spherical camera placement, per-action multi-view frame+rendering and T-pose 2x2 grids.++These helpers target the pip ``bpy`` module driven by plain ``python``+(EEVEE needs the module's GPU context; ``blender -b`` has no display+surface), unlike :mod:`data_process.utils.blender_export`, which also runs+under Blender's bundled interpreter.+"""++import fcntl+import os+import json+import math+import re+import tempfile+from contextlib import contextmanager+from typing import List, Literal, Optional, Tuple++import bpy+import imageio+import numpy as np+from mathutils import Euler, Matrix, Vector+from PIL import Image+from loguru import logger+++# Frame rate for import, preview video and the per-view metadata JSON. Must+# match the export stage's --fps (also 30) โ see load_and_prep_asset.+RENDER_FPS = 30+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# GPU serialization+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++# Several EEVEE contexts rendering on one device can wedge the NVIDIA driver+# (Xid 109 CTX SWITCH TIMEOUT, then Xid 31 MMU fault): every worker spins at+# 100% CPU in os_acquire_rwlock and not a single frame is ever written. An+# exclusive lock around the render call keeps GPU work single-file while+# asset import and video encoding still overlap across workers. The lock is+# node-local and per-GPU; set RENDER_GPU_LOCK=0 to disable it.++_gpu_lock_handle = None+++def _gpu_lock_path() -> str:+ job = os.environ.get("SLURM_JOB_ID", "local")+ gpu = (os.environ.get("CUDA_VISIBLE_DEVICES", "0").split(",")[0] or "0")+ name = f"unimate_render_{job}_gpu{gpu}.lock"+ # Node-local by preference: the contended resource is one node's GPU, and+ # TMPDIR often points at shared scratch, where the lock would be slower+ # and โ for a multi-node job โ serialize workers that share no device.+ for directory in ("/tmp", tempfile.gettempdir()):+ if os.path.isdir(directory) and os.access(directory, os.W_OK):+ return os.path.join(directory, name)+ return os.path.join(tempfile.gettempdir(), name)+++@contextmanager+def gpu_render_lock():+ """Serialize GPU rendering across worker processes sharing one GPU."""+ global _gpu_lock_handle+ if os.environ.get("RENDER_GPU_LOCK", "1") == "0":+ yield+ return+ if _gpu_lock_handle is None:+ _gpu_lock_handle = open(_gpu_lock_path(), "w")+ fcntl.flock(_gpu_lock_handle, fcntl.LOCK_EX)+ try:+ yield+ finally:+ fcntl.flock(_gpu_lock_handle, fcntl.LOCK_UN)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Render engine+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def init_render_engine(render_samples: int = 64):+ """Initialize EEVEE with the given sample count. Call once per process."""+ bpy.context.scene.render.engine = "BLENDER_EEVEE"+ eevee = bpy.context.scene.eevee++ # Anti-aliasing+ eevee.taa_render_samples = render_samples++ # Ambient occlusion+ eevee.use_gtao = True+ eevee.gtao_distance = 1.0+ eevee.gtao_quality = 0.5++ # Screen-space reflections (half-res for efficiency at 512px)+ eevee.use_ssr = True+ eevee.use_ssr_refraction = True+ eevee.use_ssr_halfres = True+ eevee.ssr_quality = 0.5+ eevee.ssr_max_roughness = 0.5+ eevee.ssr_thickness = 0.2++ # Shadows+ eevee.use_soft_shadows = True+ eevee.shadow_cascade_size = "2048"+ eevee.shadow_cube_size = "1024"++ eevee.use_bloom = True++ bpy.context.scene.render.use_high_quality_normals = True+ bpy.context.scene.render.use_persistent_data = True++ # Color management โ Filmic handles HDR values from PBR materials+ # properly (Standard clips > 1.0, causing dark metallic surfaces)+ bpy.context.scene.view_settings.view_transform = "Filmic"+ bpy.context.scene.view_settings.look = "Medium Contrast"+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Scene reset+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def clear_scene():+ """Remove all objects, compositor nodes, actions; purge orphan data.++ Removing objects leaves their mesh/material/image datablocks behind+ with zero users; without the purge they accumulate across assets and+ grow memory unboundedly over long in-process batches.+ """+ for obj in list(bpy.data.objects):+ bpy.data.objects.remove(obj, do_unlink=True)++ bpy.context.scene.use_nodes = True+ for node in list(bpy.context.scene.node_tree.nodes):+ bpy.context.scene.node_tree.nodes.remove(node)++ bpy.context.scene.frame_start = 0+ bpy.context.scene.frame_end = 0+ for action in list(bpy.data.actions):+ bpy.data.actions.remove(action)++ bpy.data.orphans_purge(do_recursive=True)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Asset import+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++_IMPORT_OPS = {+ ".fbx": lambda p: bpy.ops.import_scene.fbx(filepath=p, use_anim=True),+ ".glb": lambda p: bpy.ops.import_scene.gltf(filepath=p),+ ".gltf": lambda p: bpy.ops.import_scene.gltf(filepath=p),+}++RENDERABLE_EXTS = frozenset(_IMPORT_OPS)+++def load_file(path: str):+ """Import a rigged asset (.fbx, .glb, .gltf)."""+ ext = os.path.splitext(path)[1].lower()+ op = _IMPORT_OPS.get(ext)+ if op is None:+ raise RuntimeError(f"Unsupported file format: {path}")+ op(path)+++def load_armature(path: str):+ """Import a file and return its main newly added armature (or None).++ Picks the armature with the most bones, matching the export stage's+ ``choose_main_armature`` โ on multi-armature assets both stages must+ agree, or the render animates a different skeleton than the NPZ.+ """+ existing = set(bpy.data.objects)+ load_file(path)+ armatures = [+ obj for obj in bpy.data.objects+ if obj not in existing and obj.type == "ARMATURE"+ ]+ if not armatures:+ return None+ return max(armatures, key=lambda a: len(a.data.bones))+++def find_meshes():+ return [o for o in bpy.data.objects if o.type == "MESH"]+++def get_scene_armatures():+ """Return all ARMATURE objects in the current scene."""+ return [o for o in bpy.context.scene.objects if o.type == "ARMATURE"]+++def get_root_objects():+ """Yield all top-level (parentless) objects in the current scene."""+ for obj in bpy.context.scene.objects.values():+ if not obj.parent:+ yield obj+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Material fixes (post-import)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def fix_materials():+ """Fix common GLTF/FBX material issues after import.++ 1. Clamps Metallic to <= 0.9 so surfaces don't render black when+ film_transparent=True (pure mirrors reflect the empty background).+ 2. Connects vertex color attributes when Base Color is at a default+ value (white or black), indicating the importer left it unset.+ 3. Connects disconnected Image Texture nodes found in the shader tree.+ """+ # Pass 1: clamp metallic on all Principled BSDF materials+ for material in bpy.data.materials:+ if not material.use_nodes:+ continue+ for node in material.node_tree.nodes:+ if node.type != 'BSDF_PRINCIPLED':+ continue+ metallic_input = node.inputs.get('Metallic')+ if metallic_input is None:+ continue+ # Only fix unlinked (constant) metallic values+ if not metallic_input.is_linked and metallic_input.default_value > 0.9:+ logger.info(f"Clamping Metallic {metallic_input.default_value:.2f} -> 0.9 "+ f"on material '{material.name}'")+ metallic_input.default_value = 0.9++ # Pass 2: fix unlinked Base Color defaults per mesh+ for obj in find_meshes():+ mesh = obj.data++ # Check if mesh has color attributes (vertex colors)+ color_attr_name = None+ if hasattr(mesh, 'color_attributes') and len(mesh.color_attributes) > 0:+ color_attr_name = mesh.color_attributes[0].name+ elif hasattr(mesh, 'vertex_colors') and len(mesh.vertex_colors) > 0:+ color_attr_name = mesh.vertex_colors[0].name++ for slot in obj.material_slots:+ mat = slot.material+ if mat is None or not mat.use_nodes:+ continue++ tree = mat.node_tree+ nodes = tree.nodes+ links = tree.links++ principled = None+ for node in nodes:+ if node.type == 'BSDF_PRINCIPLED':+ principled = node+ break+ if principled is None:+ continue++ base_color_input = principled.inputs.get('Base Color')+ if base_color_input is None or base_color_input.is_linked:+ continue++ existing_color = list(base_color_input.default_value)+ is_default_white = all(c > 0.95 for c in existing_color[:3])+ is_default_black = all(c < 0.05 for c in existing_color[:3])+ if not (is_default_white or is_default_black):+ continue++ # Priority 1: connect vertex colors if available+ if color_attr_name is not None:+ vc_node = nodes.new(type='ShaderNodeVertexColor')+ vc_node.layer_name = color_attr_name+ links.new(vc_node.outputs['Color'], base_color_input)+ logger.info(f"Connected vertex colors '{color_attr_name}' to "+ f"material '{mat.name}' on '{obj.name}'")+ continue++ # Priority 2: find disconnected Image Texture nodes in the tree+ img_tex = None+ for node in nodes:+ if node.type == 'TEX_IMAGE' and node.image is not None:+ if not any(l.from_node == node and l.from_socket == node.outputs['Color']+ for l in links):+ img_tex = node+ break+ if img_tex is not None:+ links.new(img_tex.outputs['Color'], base_color_input)+ logger.info(f"Connected orphan image texture '{img_tex.image.name}' to "+ f"material '{mat.name}' on '{obj.name}'")+++def clear_normal_maps():+ """Remove normal map connections from all Principled BSDF materials.++ Some GLTF normal maps cause rendering artifacts; removing them+ produces cleaner results for motion-capture rendering.+ """+ for material in bpy.data.materials:+ if not material.use_nodes:+ continue+ tree = material.node_tree+ for node in tree.nodes:+ if node.type == 'BSDF_PRINCIPLED':+ normal_input = node.inputs.get("Normal")+ if normal_input and normal_input.is_linked:+ for link in list(normal_input.links):+ tree.links.remove(link)+++def set_materials_opaque():+ """Force materials to OPAQUE blend mode, keeping real alpha as HASHED.++ Prevents transparency artifacts that cause parts of meshes to+ disappear or render incorrectly in multi-view rendering. Materials+ whose Principled Alpha input is actually driven (linked, or set below+ 1.0) genuinely need transparency โ e.g. Mixamo facial-feature decal+ planes layered over the face, whose atlas would otherwise cover it as+ an opaque black quad. Those get HASHED, which EEVEE resolves without+ the depth-sorting artifacts of BLEND.+ """+ for material in bpy.data.materials:+ if not material.use_nodes:+ continue+ has_alpha = False+ for node in material.node_tree.nodes:+ if node.type == 'BSDF_PRINCIPLED':+ alpha_input = node.inputs.get('Alpha')+ if alpha_input is not None and (+ alpha_input.is_linked or alpha_input.default_value < 1.0):+ has_alpha = True+ break+ material.blend_method = "HASHED" if has_alpha else "OPAQUE"+ material.show_transparent_back = False+++def smooth_meshes():+ """Enable auto-smooth on all meshes (30-degree threshold)."""+ for obj in find_meshes():+ obj.data.use_auto_smooth = True+ obj.data.auto_smooth_angle = np.deg2rad(30)+++def keep_main_mesh_only(main_mesh):+ """Remove every mesh except *main_mesh*.++ Dropping stray helper geometry (icospheres, prop meshes, low-vertex+ duplicates) keeps the render clean and the scene bbox tight.+ """+ for m in find_meshes():+ if m is not main_mesh:+ bpy.data.objects.remove(m, do_unlink=True)+++def keep_armature_meshes(armature):+ """Remove meshes that do not belong to *armature*.++ A mesh belongs to the rig when it is skinned to it (Armature+ modifier) or parented under it (bone-parented props, meshes grouped+ beneath the armature node). Stray helper geometry (icospheres, prop+ meshes of another rig) matches neither โ so this drops the junk+ without amputating multi-mesh characters the way keeping the single+ largest mesh would. Falls back to the largest mesh when nothing+ belongs to the rig.+ """+ def belongs(mesh):+ if any(mod.type == 'ARMATURE' and mod.object == armature+ for mod in mesh.modifiers):+ return True+ parent = mesh.parent+ while parent is not None:+ if parent == armature:+ return True+ parent = parent.parent+ return False++ meshes = find_meshes()+ keep = [m for m in meshes if belongs(m)]+ if keep:+ for m in meshes:+ if m not in keep:+ bpy.data.objects.remove(m, do_unlink=True)+ else:+ keep_main_mesh_only(max(meshes, key=lambda m: len(m.data.vertices)))+++def load_and_prep_asset(path: str, keep_only_main_mesh: bool = True,+ fps: int = RENDER_FPS):+ """Clear the scene, import an asset, and fix it up for rendering.++ Sets up lighting, optionally drops meshes that do not belong to the+ main armature (stray helper geometry that would pollute the render+ and the scene bbox โ multi-mesh characters keep every skinned or+ bone-parented part), then applies material fixes and auto-smooth.++ The scene frame rate is set *before* the import, matching the export+ stage (``blender_export.load_scene``). glTF stores keyframe times in+ seconds, so the importer resamples them at the current scene fps โ+ leaving it at Blender's default 24 makes every GLB render 0.8x as many+ frames as the 30 fps NPZ the same asset exports to, which shifts the+ frame window and makes the previews play 25% fast. (The FBX importer+ takes the rate from the file, so only glTF was affected.)++ Args:+ path: Asset file (.fbx / .glb / .gltf).+ keep_only_main_mesh: Drop meshes that are neither skinned to nor+ parented under the main armature (falling back to the+ highest-vertex mesh if nothing belongs to it). Disable to+ keep every mesh regardless.+ fps: Scene frame rate to import and render at. Must match the+ export stage's ``--fps`` or renders and NPZs desynchronize.++ Returns:+ The imported armature, or None if the file has no armature or+ no mesh.+ """+ clear_scene()+ bpy.context.scene.render.fps = int(fps)+ setup_lighting()++ armature = load_armature(path)+ if armature is None:+ return None++ # Drop lights bundled in the asset โ they stack on the studio lighting+ # and make per-asset exposure inconsistent. Cameras go too: add_camera+ # only creates one when scene.camera is unset, so an asset camera would+ # otherwise take over the render and follow the asset's own camera path.+ studio_lights = {"Key_Light", "Fill_Light", "Rim_Light", "Bottom_Light"}+ for obj in list(bpy.data.objects):+ if obj.type == "LIGHT" and obj.name not in studio_lights:+ bpy.data.objects.remove(obj, do_unlink=True)+ elif obj.type == "CAMERA":+ bpy.data.objects.remove(obj, do_unlink=True)++ meshes = find_meshes()+ if not meshes:+ return None+ if keep_only_main_mesh:+ keep_armature_meshes(armature)++ fix_materials()+ clear_normal_maps()+ set_materials_opaque()+ smooth_meshes()+ return armature+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Lighting+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _create_light(+ name: str,+ light_type: Literal["POINT", "SUN", "SPOT", "AREA"],+ rotation: Tuple[float, float, float],+ energy: float,+ use_shadow: bool = False,+ specular_factor: float = 1.0,+ angle: float = 0.0,+):+ """Create and link a light object to the scene."""+ light_data = bpy.data.lights.new(name=name, type=light_type)+ light_object = bpy.data.objects.new(name, light_data)+ bpy.context.collection.objects.link(light_object)+ light_object.rotation_euler = rotation+ light_data.use_shadow = use_shadow+ light_data.specular_factor = specular_factor+ light_data.energy = energy+ if light_type == "SUN" and angle > 0:+ light_data.angle = angle # angular diameter โ soft shadow edges+ return light_object+++def _setup_world_environment(strength: float = 0.7):+ """Soft top-to-bottom gradient world for even ambient illumination."""+ world = bpy.data.worlds.get("World")+ if world is None:+ world = bpy.data.worlds.new("World")+ bpy.context.scene.world = world+ world.use_nodes = True+ nodes = world.node_tree.nodes+ links = world.node_tree.links+ nodes.clear()++ bg_node = nodes.new(type="ShaderNodeBackground")+ bg_node.inputs["Strength"].default_value = strength++ output_node = nodes.new(type="ShaderNodeOutputWorld")++ tex_coord = nodes.new(type="ShaderNodeTexCoord")+ mapping = nodes.new(type="ShaderNodeMapping")+ gradient = nodes.new(type="ShaderNodeTexGradient")+ gradient.gradient_type = "LINEAR"++ color_ramp = nodes.new(type="ShaderNodeValToRGB")+ color_ramp.color_ramp.elements[0].position = 0.3+ color_ramp.color_ramp.elements[0].color = (0.9, 0.9, 0.9, 1.0)+ color_ramp.color_ramp.elements[1].position = 0.7+ color_ramp.color_ramp.elements[1].color = (1.0, 1.0, 1.0, 1.0)++ links.new(tex_coord.outputs["Generated"], mapping.inputs["Vector"])+ links.new(mapping.outputs["Vector"], gradient.inputs["Vector"])+ links.new(gradient.outputs["Color"], color_ramp.inputs["Fac"])+ links.new(color_ramp.outputs["Color"], bg_node.inputs["Color"])+ links.new(bg_node.outputs["Background"], output_node.inputs["Surface"])+++def setup_lighting():+ """Studio-style lighting for multi-view rendering.++ Three-point neutral sun lights plus a soft ambient world fill so no+ view is completely dark:++ - Key light: upper-front-left, shadow enabled for depth+ - Fill light: upper-front-right, softer, no shadow+ - Rim light: upper-rear, edge separation+ - Bottom fill: subtle upward fill to prevent harsh under-shadows+ """+ # Clear existing lights+ for obj in list(bpy.data.objects):+ if obj.type == "LIGHT":+ bpy.data.objects.remove(obj, do_unlink=True)++ _setup_world_environment()++ _create_light("Key_Light", "SUN",+ rotation=(0.785, 0, -0.785), # 45ยฐ elevation, 45ยฐ left+ energy=1.5, use_shadow=True, angle=0.05)+ _create_light("Fill_Light", "SUN",+ rotation=(0.785, 0, 2.356), # 45ยฐ elevation, 135ยฐ right+ energy=0.7, specular_factor=0.5)+ _create_light("Rim_Light", "SUN",+ rotation=(-0.524, 0, -3.927), # 30ยฐ above rear+ energy=1.0, specular_factor=0.8)+ _create_light("Bottom_Light", "SUN",+ rotation=(2.618, 0, 0), # pointing upward from below+ energy=0.3, specular_factor=0.3)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Bounding box & scene normalization+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def get_scene_bbox(single_obj=None, ignore_matrix=False):+ """Axis-aligned bounding box of all meshes (or a single object)."""+ bbox_min = (math.inf,) * 3+ bbox_max = (-math.inf,) * 3++ meshes = find_meshes() if single_obj is None else [single_obj]+ if not meshes:+ raise RuntimeError("No objects in scene to compute bounding box for")++ for obj in meshes:+ for coord in obj.bound_box:+ coord = Vector(coord)+ if not ignore_matrix:+ coord = obj.matrix_world @ coord+ bbox_min = tuple(min(a, b) for a, b in zip(bbox_min, coord))+ bbox_max = tuple(max(a, b) for a, b in zip(bbox_max, coord))++ return Vector(bbox_min), Vector(bbox_max)+++def get_scene_bbox_all_frames():+ """Bounding box that encloses the entire animation sequence."""+ bbox_min = (math.inf,) * 3+ bbox_max = (-math.inf,) * 3++ saved_frame = bpy.context.scene.frame_current+ for frame in range(bpy.context.scene.frame_start, bpy.context.scene.frame_end + 1):+ bpy.context.scene.frame_set(frame)+ fmin, fmax = get_scene_bbox()+ bbox_min = tuple(min(a, b) for a, b in zip(bbox_min, fmin))+ bbox_max = tuple(max(a, b) for a, b in zip(bbox_max, fmax))++ bpy.context.scene.frame_set(saved_frame)+ return Vector(bbox_min), Vector(bbox_max)+++def normalize_scene(scene_scale: float = 1.0, process_frames: bool = False):+ """Scale and center the scene to fit within *scene_scale* at the origin.++ Parents all root objects under a single "NormalizationNode" empty and+ applies scale/offset there. Safe to call multiple times โ resets the+ node to identity first, so repeated calls don't compound.++ Args:+ scene_scale: Target bounding box size.+ process_frames: If True, the bbox covers the whole animation range.+ """+ node = bpy.data.objects.get("NormalizationNode")+ if node is not None:+ node.location = (0, 0, 0)+ node.rotation_euler = (0, 0, 0)+ node.scale = (1, 1, 1)+ bpy.context.view_layer.update()++ if process_frames:+ bbox_min, bbox_max = get_scene_bbox_all_frames()+ else:+ bbox_min, bbox_max = get_scene_bbox()++ scale = scene_scale / max(bbox_max - bbox_min)+ offset = -(bbox_min + bbox_max) / 2++ if node is None:+ node = bpy.data.objects.new("NormalizationNode", None)+ bpy.context.scene.collection.objects.link(node)+ for obj in get_root_objects():+ if obj is not node and obj.type != "CAMERA":+ obj.parent = node+ obj.matrix_parent_inverse = node.matrix_world.inverted()+ node.scale = (scale, scale, scale)+ node.location = offset * scale+ bpy.context.view_layer.update()++ bpy.ops.object.select_all(action="DESELECT")+ return scale, offset+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Camera+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def build_transformation_mat(translation, rotation) -> np.ndarray:+ """Build a 4x4 transform from a (3,) translation and a (3,3) rotation or (3,) Euler."""+ translation = np.array(translation)+ rotation = np.array(rotation)++ mat = np.eye(4)+ if translation.shape[0] != 3:+ raise RuntimeError(+ f"Translation has invalid shape: {translation.shape}. Expected (3,).")+ mat[:3, 3] = translation++ if rotation.shape == (3, 3):+ mat[:3, :3] = rotation+ elif rotation.shape[0] == 3:+ mat[:3, :3] = np.array(Euler(rotation).to_matrix())+ else:+ raise RuntimeError(+ f"Rotation has invalid shape: {rotation.shape}. Expected (3,3) or (3,).")++ return mat+++def camera_pose_on_sphere(+ azimuth_deg: float,+ elevation_deg: float,+ radius: float,+ center: Tuple[float, float, float] = (0, 0, 0),+) -> np.ndarray:+ """Cam-to-world 4x4 matrix for a camera on a sphere looking at *center*."""+ elev = math.radians(elevation_deg)+ azim = math.radians(azimuth_deg)+ phi = 0.5 * math.pi - elev++ x = center[0] + radius * math.sin(phi) * math.cos(azim)+ y = center[1] + radius * math.sin(phi) * math.sin(azim)+ z = center[2] + radius * math.cos(phi)+ cam_pos = Vector((x, y, z))++ look_dir = Vector(center) - cam_pos+ rot_euler = look_dir.to_track_quat("-Z", "Y").to_euler()+ return build_transformation_mat(cam_pos, rot_euler)+++def fit_camera_distance(camera_dist: float, scene_scale: float = 1.0,+ camera_sensor_width: int = 32, camera_lens: int = 35,+ margin: float = 1.05) -> float:+ """Raise *camera_dist* if it would crop the normalized scene bbox.++ ``normalize_scene`` fits the subject into a cube of side *scene_scale*,+ whose worst case in frame is a near-face corner at depth+ ``d - scene_scale/2`` and lateral offset ``scene_scale/2``. With the+ fixed 35mm lens on a 32mm sensor the half-FOV is only atan(16/35) =+ 24.6 degrees, so the default distance of 1.5 is marginally too close and+ clips the occasional wide subject (measured: ~0.3% of rendered frames,+ e.g. a crab's claw).++ Returns the larger of *camera_dist* and the distance that fits, so+ subjects that already frame correctly keep their exact previous framing+ and only the ones that would crop get pushed back.+ """+ half_fov = math.atan((camera_sensor_width / 2.0) / camera_lens)+ half_extent = scene_scale / 2.0+ required = half_extent + (half_extent / math.tan(half_fov)) * margin+ if required > camera_dist:+ logger.info(f"Camera distance {camera_dist:.3f} would crop a "+ f"{scene_scale:.3f} bbox; using {required:.3f}")+ return required+ return camera_dist+++def add_camera(+ cam2world_matrix,+ camera_sensor_width: int = 32,+ camera_lens: int = 35,+):+ """Set (or create) the scene camera from a cam-to-world matrix.++ Keyframes the pose at the current frame_end so per-view placement+ survives animation rendering (constant extrapolation holds it fixed).+ """+ if not isinstance(cam2world_matrix, Matrix):+ cam2world_matrix = Matrix(cam2world_matrix)++ if bpy.context.scene.camera is None:+ # camera_add leaves scene.camera unset, so the new camera has to be+ # picked up explicitly. Selecting "the last CAMERA in bpy.data.objects"+ # instead would hand the render to a camera bundled with the asset+ # (objects iterate by name), which then inherits its parent's animated+ # transform and flies along the asset's own camera path.+ before = set(bpy.data.objects)+ bpy.ops.object.camera_add(location=(0, 0, 0))+ new_cams = [o for o in bpy.data.objects+ if o not in before and o.type == "CAMERA"]+ if not new_cams:+ raise RuntimeError("camera_add did not create a camera object")+ bpy.context.scene.camera = new_cams[0]++ cam = bpy.context.scene.camera+ cam.data.type = "PERSP"+ cam.data.sensor_width = camera_sensor_width+ cam.data.lens = camera_lens+ cam.matrix_world = cam2world_matrix++ frame = bpy.context.scene.frame_end+ cam.keyframe_insert(data_path="location", frame=frame)+ cam.keyframe_insert(data_path="rotation_euler", frame=frame)+ cam.data.keyframe_insert(data_path="lens", frame=frame)+ cam.data.keyframe_insert(data_path="sensor_width", frame=frame)++ return cam+++def remove_camera():+ """Delete the active scene camera (clears stale keyframes between actions)."""+ cam = bpy.context.scene.camera+ if cam is not None:+ bpy.data.objects.remove(cam, do_unlink=True)+ bpy.context.scene.camera = None+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Render output+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def enable_color_output(+ resolution: int,+ output_dir: str,+ file_prefix: str = "",+ mode: Literal["IMAGE", "VIDEO"] = "IMAGE",+ film_transparent: bool = True,+ fps: int = RENDER_FPS,+):+ """Configure color render output (square resolution).++ Args:+ resolution: Render resolution in pixels.+ output_dir: Directory for output files.+ file_prefix: Filename prefix (IMAGE) or MP4 stem (VIDEO).+ mode: 'IMAGE' for per-frame PNGs, 'VIDEO' for an MP4.+ film_transparent: Transparent background (IMAGE mode).+ fps: Video frame rate (VIDEO mode).+ """+ scene = bpy.context.scene+ scene.render.resolution_x = resolution+ scene.render.resolution_y = resolution+ scene.render.resolution_percentage = 100+ scene.render.film_transparent = film_transparent+ scene.render.image_settings.quality = 100++ if mode == "IMAGE":+ scene.render.image_settings.file_format = "PNG"+ scene.render.image_settings.color_mode = "RGBA"+ scene.render.filepath = os.path.join(output_dir, file_prefix)+ elif mode == "VIDEO":+ scene.render.image_settings.file_format = "FFMPEG"+ scene.render.ffmpeg.format = "MPEG4"+ scene.render.ffmpeg.codec = "H264"+ scene.render.image_settings.color_mode = "RGB"+ scene.render.fps = fps+ scene.render.filepath = os.path.join(output_dir, file_prefix + ".mp4")+++def render_animation():+ """Render the scene's frame range with compositing enabled."""+ bpy.context.scene.render.use_compositing = True+ bpy.context.scene.use_nodes = True++ tree = bpy.context.scene.node_tree+ if "Render Layers" not in tree.nodes:+ tree.nodes.new("CompositorNodeRLayers")++ with gpu_render_lock():+ bpy.ops.render.render(animation=True, write_still=True)+++def render_still(filepath: str):+ """Render the current frame to *filepath* with compositing enabled."""+ bpy.context.scene.render.use_compositing = True+ bpy.context.scene.use_nodes = True++ tree = bpy.context.scene.node_tree+ if "Render Layers" not in tree.nodes:+ tree.nodes.new("CompositorNodeRLayers")++ bpy.context.scene.render.filepath = filepath+ with gpu_render_lock():+ bpy.ops.render.render(write_still=True)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Multi-view rendering+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++VIEW_NAMES = ["front", "back", "left", "right"]+VIEW_AZIMUTHS = [-90, 90, 180, 0] # matches VIEW_NAMES order+NUM_VIEWS = len(VIEW_NAMES)+MAX_RENDER_FRAMES = 200+++VIDEO_BACKGROUND = (235, 235, 235)+++def _reset_frames_dir(frames_dir: str):+ """Create *frames_dir*, removing any PNGs left by a previous render.++ Blender overwrites ``{frame:04d}.png`` but never deletes frames outside+ the new range. Re-rendering a clip with a shorter range would otherwise+ leave the old tail in place โ and since ``is_render_complete`` only+ checks that the four views hold the same non-zero count (all four are+ stale by the same amount), the clip still reports complete while every+ ``v00x.mp4`` ends with frames from the previous animation.+ """+ if os.path.isdir(frames_dir):+ for fname in os.listdir(frames_dir):+ if fname.endswith(".png"):+ os.remove(os.path.join(frames_dir, fname))+ else:+ os.makedirs(frames_dir, exist_ok=True)+++def compose_frames_to_video(frames_dir: str, out_path: str, fps: int = RENDER_FPS,+ background=VIDEO_BACKGROUND):+ """Compose a view's rendered PNG frames into an H.264 MP4.++ Reuses the already-rendered frames (no extra Blender render pass).+ Transparent-film PNGs are flattened onto *background* โ H.264 carries+ no alpha channel.+ """+ frames = sorted(f for f in os.listdir(frames_dir) if f.endswith(".png"))+ if not frames:+ logger.warning(f"No frames in {frames_dir}; skipping video compose.")+ return+ writer = imageio.get_writer(out_path, fps=fps)+ try:+ for fname in frames:+ img = Image.open(os.path.join(frames_dir, fname))+ if img.mode in ("RGBA", "LA", "P"):+ img = img.convert("RGBA")+ bg = Image.new("RGB", img.size, background)+ bg.paste(img, mask=img.split()[-1])+ img = bg+ else:+ img = img.convert("RGB")+ writer.append_data(np.asarray(img))+ finally:+ writer.close()+ logger.info(f"Composed video: {out_path} ({len(frames)} frames @ {fps} fps)")+++def _skip_marker_path(output_dir: str, name: str) -> str:+ return os.path.join(output_dir, ".skipped", name)+++def is_asset_render_skipped(output_dir: str, name: str) -> bool:+ """True when *name* was marked as producing no renders (see below)."""+ return os.path.isfile(_skip_marker_path(output_dir, name))+++def mark_asset_render_skipped(output_dir: str, name: str, reason: str = ""):+ """Record that *name* legitimately produces no renders (no armature /+ mesh / pose actions), so reruns don't reload the asset just to rediscover+ that. Delete the marker (or ``.skipped/``) to retry the asset."""+ path = _skip_marker_path(output_dir, name)+ os.makedirs(os.path.dirname(path), exist_ok=True)+ with open(path, "w") as f:+ f.write(reason + "\n")+++def _rendered_marker_path(output_dir: str, name: str) -> str:+ return os.path.join(output_dir, ".rendered", f"{name}.json")+++def mark_asset_render_done(output_dir: str, name: str, clip_dirs):+ """Record the full set of clip directories *name* is expected to produce.++ ``--missing-only`` cannot otherwise tell "this asset rendered all five of+ its actions" from "a worker was killed after three of them" โ both leave+ only complete directories on disk, and without the expected set the+ partially-rendered asset is treated as done and its remaining actions are+ never rendered. Written only after every action of the asset has been+ handled.+ """+ path = _rendered_marker_path(output_dir, name)+ os.makedirs(os.path.dirname(path), exist_ok=True)+ tmp = f"{path}.tmp.{os.getpid()}"+ with open(tmp, "w") as f:+ json.dump({"clip_dirs": sorted(clip_dirs)}, f)+ os.replace(tmp, path)+++def get_expected_clip_dirs(output_dir: str, name: str):+ """Expected clip directory names for *name*, or None when unrecorded."""+ path = _rendered_marker_path(output_dir, name)+ if not os.path.isfile(path):+ return None+ try:+ with open(path) as f:+ return json.load(f).get("clip_dirs")+ except (OSError, ValueError):+ return None+++def is_render_complete(save_root: str, num_views: int = NUM_VIEWS,+ require_video: bool = False) -> bool:+ """Check if all expected multi-view outputs exist for one clip.++ Requires *num_views* view directories (v000..) with the matching camera+ metadata JSONs (v000.json..), each holding the same non-zero number of+ PNGs โ plus the composed v000.mp4.. when *require_video* is set. Views+ render sequentially, so an interrupted render always leaves either a+ missing view directory or a view with fewer frames than the first โ+ both fail the equal-count check.+ """+ if not os.path.isdir(save_root):+ return False+ frame_counts = []+ for view_idx in range(num_views):+ view_name = f"v{view_idx:03d}"+ if not os.path.isfile(os.path.join(save_root, f"{view_name}.json")):+ return False+ if require_video and not os.path.isfile(os.path.join(save_root, f"{view_name}.mp4")):+ return False+ view_dir = os.path.join(save_root, view_name)+ if not os.path.isdir(view_dir):+ return False+ frame_counts.append(sum(1 for f in os.listdir(view_dir) if f.endswith(".png")))+ return frame_counts[0] > 0 and len(set(frame_counts)) == 1+++def render_action_multiview(+ armature,+ action_name: str,+ start_frame: int,+ end_frame: int,+ save_root: str,+ camera_dist: float = 1.5,+ scene_scale: float = 1.0,+ resolution: int = 512,+ views: Optional[List[str]] = None,+ render_video: bool = False,+ render_frames: bool = True,+ compose_video: bool = True,+ fps: int = RENDER_FPS,+ max_render_frames: int = MAX_RENDER_FRAMES,+ save_camera_metadata: bool = True,+):+ """Render one action from multiple views.++ Output structure::++ save_root/+ v000/ 0001.png ... (front)+ v001/ ... (back)+ v002/ ... (left)+ v003/ ... (right)+ v000.mp4 ... (if render_video=True)+ v000.json ... (camera metadata per view)++ Args:+ armature: Blender armature object to animate.+ action_name: Name of the action in bpy.data.actions.+ start_frame / end_frame: Frame range of the action.+ save_root: Output directory for this clip's renders.+ camera_dist: Camera distance from origin.+ scene_scale: Target bounding box size for normalization.+ resolution: Render resolution (square).+ views: Subset of VIEW_NAMES to render (default: all four).+ render_video: Also render an MP4 per view via Blender (second full+ render pass; prefer *compose_video*).+ render_frames: Render per-frame PNGs per view.+ compose_video: Compose the rendered PNGs into v00x.mp4 per view+ (no extra render pass).+ fps: Video frame rate.+ max_render_frames: Clamp on the number of rendered frames.+ save_camera_metadata: Write a JSON per view with camera pose.+ """+ if views is None:+ views = list(VIEW_NAMES)+ view_azimuths = [VIEW_AZIMUTHS[VIEW_NAMES.index(v)] for v in views]++ # Clamp frame range+ if end_frame - start_frame + 1 > max_render_frames:+ end_frame = start_frame + max_render_frames - 1++ # Ensure armatures are in POSE mode (may be stuck in REST from a+ # preceding T-pose render)+ for arm in get_scene_armatures():+ arm.data.pose_position = "POSE"++ # Bind the action+ if armature.animation_data:+ armature.animation_data_clear()+ armature.animation_data_create()+ action = bpy.data.actions[action_name]+ armature.animation_data.action = action++ # A discovered action can belong to another armature in the file; the+ # render then silently shows a static pose. Warn so it's diagnosable.+ target_bones = {+ m.group(1)+ for fc in action.fcurves+ for m in [re.match(r'pose\.bones\["(.+?)"\]', fc.data_path)]+ if m+ }+ if target_bones and not (target_bones & set(armature.pose.bones.keys())):+ logger.warning(f"Action '{action_name}' animates none of armature "+ f"'{armature.name}'s bones; render will be static.")++ # Set the frame range and flush the depsgraph so the mesh deforms+ # before normalization measures it+ scene = bpy.context.scene+ scene.frame_start = start_frame+ scene.frame_end = end_frame+ scene.frame_set(start_frame)+ bpy.context.view_layer.update()++ # Reset camera so the previous action's keyframes don't carry over+ remove_camera()++ # Normalize per action (accounts for the full motion extent). The+ # returned transform maps original asset coordinates into the rendered+ # (normalized) space: p_rendered = norm_scale * (p_original + norm_offset).+ norm_scale, norm_offset = normalize_scene(scene_scale, process_frames=True)+ camera_dist = fit_camera_distance(camera_dist, scene_scale)++ os.makedirs(save_root, exist_ok=True)++ for view_idx, azimuth in enumerate(view_azimuths):+ cam_mat = camera_pose_on_sphere(azimuth, 0.0, camera_dist)+ camera = add_camera(cam_mat)+ view_name = f"v{view_idx:03d}"++ if save_camera_metadata:+ meta = {+ # view identity+ "view": views[view_idx],+ "azimuth": math.radians(azimuth),+ "elevation": 0.0,+ # camera intrinsics+ "resolution": resolution,+ "camera_angle_x": camera.data.angle_x,+ # camera extrinsics (cam-to-world, in normalized scene space)+ "transform_matrix": cam_mat.tolist(),+ # scene normalization: p_rendered = scene_scale * (p_original + scene_offset)+ "scene_scale": norm_scale,+ "scene_offset": list(norm_offset),+ # rendered clip timing (after the max_render_frames clamp)+ "frame_start": start_frame,+ "frame_end": end_frame,+ "fps": fps,+ }+ with open(os.path.join(save_root, f"{view_name}.json"), "w") as f:+ json.dump(meta, f, indent=2)+ if render_video:+ enable_color_output(resolution, save_root, file_prefix=view_name,+ mode="VIDEO", fps=fps)+ render_animation()++ if render_frames:+ frames_dir = os.path.join(save_root, view_name)+ _reset_frames_dir(frames_dir)+ enable_color_output(resolution, frames_dir, mode="IMAGE")+ render_animation()+ if compose_video:+ compose_frames_to_video(+ frames_dir, os.path.join(save_root, f"{view_name}.mp4"), fps=fps)++ logger.info(f"Rendered view '{views[view_idx]}' -> {save_root}/{view_name}")+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# T-pose grid rendering+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def compose_grid_2x2(image_paths: List[str], output_path: str):+ """Arrange 4 images into a 2x2 grid and save the result.++ Layout::++ [front] [back]+ [left] [right]+ """+ assert len(image_paths) == 4, f"Expected 4 images, got {len(image_paths)}"+ images = [Image.open(p) for p in image_paths]+ w, h = images[0].size++ grid = Image.new("RGBA", (w * 2, h * 2))+ grid.paste(images[0], (0, 0))+ grid.paste(images[1], (w, 0))+ grid.paste(images[2], (0, h))+ grid.paste(images[3], (w, h))+ # Written atomically: the T-pose resume check is a bare os.path.isfile, so+ # a grid truncated by a kill mid-write would be treated as complete and+ # never regenerated.+ tmp_path = f"{output_path}.tmp.{os.getpid()}.png"+ grid.save(tmp_path)+ os.replace(tmp_path, output_path)+ for img in images:+ img.close()+++def render_tpose_grid(+ save_path: str,+ camera_dist: float = 1.5,+ scene_scale: float = 1.0,+ resolution: int = 512,+):+ """Render the rest pose from 4 views and save a 2x2 grid PNG.++ Forces every armature into ``pose_position='REST'`` and clears bound+ actions so the evaluated mesh reflects the true rest pose; the bbox+ used for normalization is measured after that, so the T-pose is+ centered and scaled independently of any animation state. The caller+ is expected to clear the scene afterwards (state is not restored).+ """+ scene = bpy.context.scene++ # Force rest pose and drop bound actions+ for arm in get_scene_armatures():+ if arm.animation_data and arm.animation_data.action is not None:+ arm.animation_data.action = None+ arm.data.pose_position = "REST"+ arm.data.update_tag()++ # Flush the depsgraph on a single frame so bbox reads rest-pose coords+ frame = scene.frame_current+ scene.frame_start = frame+ scene.frame_end = frame+ scene.frame_set(frame)+ bpy.context.view_layer.update()+ bpy.context.evaluated_depsgraph_get()++ remove_camera()+ normalize_scene(scene_scale, process_frames=False)++ scene.render.resolution_x = resolution+ scene.render.resolution_y = resolution+ scene.render.resolution_percentage = 100+ scene.render.image_settings.file_format = "PNG"+ scene.render.image_settings.color_mode = "RGBA"+ scene.render.film_transparent = True++ tmp_dir = tempfile.mkdtemp(prefix="tpose_")+ image_paths = []+ try:+ for view_name, azimuth in zip(VIEW_NAMES, VIEW_AZIMUTHS):+ add_camera(camera_pose_on_sphere(+ azimuth, 0.0, fit_camera_distance(camera_dist, scene_scale)))+ out_file = os.path.join(tmp_dir, f"{view_name}.png")+ scene.frame_set(frame)+ render_still(out_file)+ image_paths.append(out_file)++ os.makedirs(os.path.dirname(save_path) or ".", exist_ok=True)+ compose_grid_2x2(image_paths, save_path)+ logger.info(f"Saved T-pose grid: {save_path}")+ finally:+ for p in image_paths:+ try:+ os.remove(p)+ except OSError:+ pass+ try:+ os.rmdir(tmp_dir)+ except OSError:+ passdiff --git a/data_process/utils/blender_rig.py b/data_process/utils/blender_rig.pynew file mode 100644index 0000000..8a3ff4a--- /dev/null+++ b/data_process/utils/blender_rig.py@@ -0,0 +1,673 @@+"""Blender utilities for driving a rigged character with an NPZ animation.++Organized as five sections:++1. Scene / object helpers โ import meshes, select objects, pack textures.+2. Keyframe computation โ per-bone loc/rot keyframes from 4x4 local+ transforms.+3. Action reconstruction โ turn per-bone keyframe arrays into a Blender+ Action bound to the armature.+4. Armature reconciliation โ handle armature bones that are absent from+ the driving animation (keep them, merge their weights, or delete them).+5. Character export โ export the animated mesh + armature to GLB/FBX.++The counterpart for the *export* direction (Blender scene โ NPZ) lives in+:mod:`data_process.utils.blender_export`.+"""++import os+import re++import bmesh+import bpy+import numpy as np+from loguru import logger+from mathutils import Matrix, Quaternion++from data_process.utils.blender_export import (+ bind_action,+ clear_animation_state,+ reset_scene,+)++RECONSTRUCTED_ACTION_NAME = "Reconstructed_Action"+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# 1. Scene / object helpers+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def update_scene():+ """Force a full scene dependency-graph update."""+ bpy.context.view_layer.update()+ bpy.context.scene.update_tag()+ for obj in bpy.context.scene.objects:+ obj.update_tag()+++def deselect():+ """Deselect every object in the scene."""+ bpy.ops.object.select_all(action="DESELECT")+++def select_objs(obj_list=None, deselect_first=False):+ """Select Blender objects, optionally deselecting everything first."""+ if not obj_list:+ obj_list = bpy.context.scene.objects+ if deselect_first:+ deselect()+ for obj in obj_list:+ obj.select_set(True)+++def load_file(filepath, *args, **kwargs):+ """Import a 3D file and return the list of newly-added scene objects."""+ old_objs = set(bpy.context.scene.objects)+ if filepath.endswith(".glb"):+ bpy.ops.import_scene.gltf(filepath=filepath, *args, **kwargs)+ elif filepath.endswith(".fbx"):+ bpy.ops.import_scene.fbx(filepath=filepath, *args, **kwargs)+ elif filepath.endswith(".obj"):+ bpy.ops.wm.obj_import(filepath=filepath, *args, **kwargs)+ elif filepath.endswith(".ply"):+ bpy.ops.wm.ply_import(filepath=filepath, *args, **kwargs)+ else:+ raise RuntimeError(f"Invalid input file: {filepath}")+ imported = sorted(set(bpy.context.scene.objects) - old_objs, key=lambda x: x.name)+ logger.info(f"Imported: {[obj.name for obj in imported]}")+ return imported+++_IMAGE_FORMAT_EXTENSIONS = {+ 'PNG': '.png', 'JPEG': '.jpg', 'JPEG2000': '.jp2', 'TARGA': '.tga',+ 'TARGA_RAW': '.tga', 'BMP': '.bmp', 'TIFF': '.tif', 'OPEN_EXR': '.exr',+ 'HDR': '.hdr', 'WEBP': '.webp',+}+++def repair_and_pack_textures(source_path):+ """Re-link missing texture files and pack every image into memory.++ Imported assets frequently reference textures by paths from the+ original author's machine (absolute Windows paths are common in+ Truebones FBX). When the importer can't locate a file, the image+ datablock has no pixel data, and both the GLB and FBX exporters+ silently drop it. This helper:++ 1. searches for missing image files next to *source_path*, and+ 2. packs every loadable image into the .blend so exports no+ longer depend on external files.++ Returns the list of image names that still have no usable data+ (their texture files could not be found on disk).+ """+ src_dir = os.path.dirname(os.path.abspath(source_path))+ try:+ bpy.ops.file.find_missing_files(directory=src_dir)+ except RuntimeError as exc:+ logger.warning(f"find_missing_files failed: {exc}")++ broken = []+ for img in bpy.data.images:+ if img.source not in ('FILE', 'SEQUENCE', 'TILED'):+ continue+ if img.packed_file is None:+ try:+ img.pack()+ except RuntimeError as exc:+ logger.warning(f"Could not pack image '{img.name}': {exc}")+ if img.packed_file is None:+ filepath = bpy.path.abspath(img.filepath) if img.filepath else ''+ if not (filepath and os.path.isfile(filepath)):+ broken.append(img.name)++ if broken:+ logger.warning(+ f"{len(broken)} image(s) have no pixel data and will be missing "+ f"from exports (texture files not found near {src_dir}): {broken}"+ )+ return broken+++def write_packed_images_to_dir(texture_dir):+ """Write packed images that have no on-disk file out to *texture_dir*.++ The FBX exporter's COPY/embed path reads texture bytes from the+ image's file path on disk; images that exist only as packed data+ (typical after a GLB import) are silently skipped, losing textures.+ Writing them out right before FBX export makes embedding work.++ Returns the list of file paths written.+ """+ written = []+ for idx, img in enumerate(bpy.data.images):+ if img.packed_file is None:+ continue+ filepath = bpy.path.abspath(img.filepath) if img.filepath else ''+ if filepath and os.path.isfile(filepath):+ continue+ ext = _IMAGE_FORMAT_EXTENSIONS.get(img.file_format, '.png')+ safe_name = re.sub(r'[^A-Za-z0-9_.-]+', '_', img.name).strip('_') or f'image_{idx}'+ if not safe_name.lower().endswith(ext):+ safe_name += ext+ os.makedirs(texture_dir, exist_ok=True)+ out_path = os.path.join(texture_dir, safe_name)+ img.filepath_raw = out_path+ try:+ # Dump the packed bytes verbatim (no re-encode) exactly where+ # filepath_raw points, so the exporter's embed path finds them.+ # (img.unpack() writes to the image's *original* path instead.)+ with open(out_path, 'wb') as fh:+ fh.write(img.packed_file.data)+ written.append(out_path)+ except OSError as exc:+ logger.warning(f"Could not write packed image '{img.name}' to disk: {exc}")+ if written:+ logger.info(f"Wrote {len(written)} packed image(s) to {texture_dir} for FBX embedding")+ return written+++def _find_upstream_image_node(socket, max_depth=8):+ """BFS upstream from *socket* to the first Image Texture node with an image."""+ queue = [(socket, 0)]+ visited = set()+ while queue:+ sock, depth = queue.pop(0)+ if depth > max_depth:+ continue+ for link in sock.links:+ node = link.from_node+ if node in visited:+ continue+ visited.add(node)+ if node.type == 'TEX_IMAGE' and node.image is not None:+ return node+ queue.extend((inp, depth + 1) for inp in node.inputs)+ return None+++def simplify_materials_for_fbx(objs):+ """Rewire each material's base-color texture directly into a Principled BSDF.++ The FBX exporter only recognizes a texture when its Image Texture node+ feeds a Principled BSDF input directly. glTF-imported materials often+ interpose extra nodes โ a Mix/Multiply for baseColorFactor or vertex+ colors, or (for KHR_materials_unlit) an Emission chain with no+ Principled node at all โ which makes the FBX exporter silently drop+ the texture. This flattens those graphs: find the first image texture+ upstream and connect it straight to Base Color, creating a Principled+ BSDF if the material lacks one.+ """+ mats = {slot.material for obj in objs if obj.type == 'MESH'+ for slot in obj.material_slots if slot.material}+ for mat in mats:+ if not mat.use_nodes:+ continue+ tree = mat.node_tree+ principled = next((n for n in tree.nodes if n.type == 'BSDF_PRINCIPLED'), None)++ if principled is not None:+ base = principled.inputs['Base Color']+ if any(l.from_node.type == 'TEX_IMAGE' and l.from_node.image+ for l in base.links):+ continue # already directly wired; FBX export handles it+ img_node = _find_upstream_image_node(base)+ if img_node is None:+ continue+ for l in list(base.links):+ tree.links.remove(l)+ tree.links.new(img_node.outputs['Color'], base)+ logger.info(f"Material '{mat.name}': rewired image "+ f"'{img_node.image.name}' directly to Base Color for FBX")+ continue++ # No Principled BSDF (e.g. unlit glTF material): build one from the+ # image feeding the output surface, so FBX gets a textured material.+ output = next((n for n in tree.nodes+ if n.type == 'OUTPUT_MATERIAL' and n.is_active_output),+ None) or next((n for n in tree.nodes+ if n.type == 'OUTPUT_MATERIAL'), None)+ if output is None:+ continue+ img_node = _find_upstream_image_node(output.inputs['Surface'])+ if img_node is None:+ continue+ principled = tree.nodes.new('ShaderNodeBsdfPrincipled')+ tree.links.new(img_node.outputs['Color'], principled.inputs['Base Color'])+ for l in list(output.inputs['Surface'].links):+ tree.links.remove(l)+ tree.links.new(principled.outputs['BSDF'], output.inputs['Surface'])+ logger.info(f"Material '{mat.name}': no Principled BSDF; created one "+ f"with image '{img_node.image.name}' for FBX")+++def get_armature_obj(obj_list=None):+ """Return the first ARMATURE object from a list (or the scene), else None."""+ if not obj_list:+ obj_list = bpy.context.scene.objects+ for obj in obj_list:+ if obj.type == "ARMATURE":+ return obj+ return None+++def set_scene_timing(nframes, fps):+ """Set the scene frame rate and frame range ``[0, nframes - 1]``."""+ bpy.context.scene.render.fps = int(fps)+ bpy.context.scene.frame_start = 0+ bpy.context.scene.frame_end = nframes - 1+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# 2. Keyframe computation+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def compute_bone_keyframes(rest_local_mat, anim_local_mat, bone_names,+ tpos_global_rot=None):+ """Compute per-bone location/rotation keyframes from local 4x4 transforms.++ The relative transform ``rest_inv @ anim`` is computed in NumPy for all+ frames at once; per-keyframe loc/rot are then extracted via ``mathutils``+ so they match Blender's conventions exactly.++ Args:+ rest_local_mat: ``(nbones, 4, 4)`` rest-pose local transforms.+ anim_local_mat: ``(nframes, nbones, 4, 4)`` animated local transforms.+ bone_names: Bone names, length ``nbones``.+ tpos_global_rot: Optional ``(nbones, 4)`` T-pose global rotations.+ When given, each keyframe is conjugated by the bone's T-pose+ global rotation.++ Returns:+ ``{bone_name: {'location': [(frame, Vector), ...],+ 'rotation': [(frame, Quaternion), ...]}}``+ """+ nframes, nbones = anim_local_mat.shape[:2]++ rest_inv = np.linalg.inv(rest_local_mat) # (nbones, 4, 4)+ rel_mats = rest_inv[None] @ anim_local_mat # (nframes, nbones, 4, 4)++ tpos_quats = tpos_quats_inv = None+ if tpos_global_rot is not None:+ tpos_quats = [Quaternion(tpos_global_rot[b]) for b in range(nbones)]+ tpos_quats_inv = [q.inverted() for q in tpos_quats]++ anim_data_dict = {}+ for idxb in range(nbones):+ locs, rots = [], []+ for idxf in range(nframes):+ m = Matrix(rel_mats[idxf, idxb].tolist())+ loc = m.to_translation()+ rot = m.to_quaternion()+ if tpos_quats is not None:+ rot = tpos_quats_inv[idxb] @ rot @ tpos_quats[idxb]+ loc = tpos_quats_inv[idxb] @ loc+ locs.append((idxf, loc))+ rots.append((idxf, rot))+ anim_data_dict[str(bone_names[idxb])] = {'location': locs, 'rotation': rots}++ return anim_data_dict+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# 3. Action reconstruction+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _add_fcurves(action, bone_name, attr, n_components, keyframes):+ """Create per-channel fcurves for one pose-bone attribute."""+ data_path = f'pose.bones["{bone_name}"].{attr}'+ for i in range(n_components):+ fc = action.fcurves.new(data_path=data_path, index=i)+ fc.keyframe_points.add(len(keyframes))+ for idx, (frame, val) in enumerate(keyframes):+ kp = fc.keyframe_points[idx]+ kp.co = (frame, val[i])+ kp.interpolation = 'LINEAR'+++def rebuild_action_from_data(armature, anim_data_dict):+ """Replace the armature's animation with a new Action built from keyframes.++ ``anim_data_dict`` has the format returned by+ :func:`compute_bone_keyframes`. Any existing action / NLA tracks on the+ armature are cleared first; bones missing from the armature are skipped+ with a warning.+ """+ clear_animation_state(armature)+ new_action = bpy.data.actions.new(name=RECONSTRUCTED_ACTION_NAME)+ bind_action(armature, new_action)++ valid_bones = {bone.name for bone in armature.data.bones}+ for bone_name, data in anim_data_dict.items():+ if bone_name not in valid_bones:+ logger.warning(f"Skipping keyframes for '{bone_name}': not in armature")+ continue+ _add_fcurves(new_action, bone_name, 'location', 3, data['location'])+ _add_fcurves(new_action, bone_name, 'rotation_quaternion', 4, data['rotation'])+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# 4. Armature reconciliation: handle bones not present in the animation+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++EXTRA_BONES_STRATEGIES = ('keep', 'merge', 'remove')+++def find_skinned_meshes(armature):+ """Return all mesh objects whose Armature modifier targets *armature*."""+ meshes = []+ for obj in bpy.context.scene.objects:+ if obj.type != 'MESH':+ continue+ for mod in obj.modifiers:+ if mod.type == 'ARMATURE' and mod.object == armature:+ meshes.append(obj)+ break+ return meshes+++def build_ancestor_map(armature, kept_bone_set):+ """Map each armature bone NOT in *kept_bone_set* to its nearest kept ancestor.++ Returns ``{dropped_name: ancestor_name_or_None}``. ``None`` indicates+ the chain to the root is entirely dropped (orphan bone).+ """+ mapping = {}+ for bone in armature.data.bones:+ if bone.name in kept_bone_set:+ continue+ parent = bone.parent+ while parent is not None and parent.name not in kept_bone_set:+ parent = parent.parent+ mapping[bone.name] = parent.name if parent is not None else None+ return mapping+++def transfer_vertex_group(mesh_obj, src_name, dst_name):+ """Merge weights of vertex group *src_name* into *dst_name* and delete src.++ If *dst_name* is ``None``, the src group is just removed. Existing dst+ weights are preserved and incremented (ADD semantics).+ """+ vgs = mesh_obj.vertex_groups+ src = vgs.get(src_name)+ if src is None:+ return+ if dst_name is None:+ vgs.remove(src)+ return++ dst = vgs.get(dst_name) or vgs.new(name=dst_name)+ src_idx = src.index++ # Snapshot (vert_idx, weight) pairs before mutating the groups.+ transfers = []+ for vert in mesh_obj.data.vertices:+ for grp in vert.groups:+ if grp.group == src_idx:+ if grp.weight > 0.0:+ transfers.append((vert.index, grp.weight))+ break # a vertex appears at most once per group++ for v_idx, weight in transfers:+ dst.add([v_idx], weight, 'ADD')+ vgs.remove(src)+++def merge_extra_bone_weights(armature, kept_bone_set):+ """Transfer weights of dropped bones to their nearest kept ancestor.++ Returns the ancestor map used (for logging).+ """+ ancestor_map = build_ancestor_map(armature, kept_bone_set)+ if not ancestor_map:+ return ancestor_map+ meshes = find_skinned_meshes(armature)+ if not meshes:+ logger.warning("No skinned meshes found targeting this armature; "+ "weight transfer is a no-op")+ for src_name, dst_name in ancestor_map.items():+ for mesh_obj in meshes:+ transfer_vertex_group(mesh_obj, src_name, dst_name)+ return ancestor_map+++def delete_dominant_weighted_vertices(mesh_obj, dropped_bone_names):+ """Delete vertices whose dominant (max-weight) group is in *dropped_bone_names*.++ Vertices with mixed influences whose primary bone survives are kept,+ so we don't punch holes near boundary regions (e.g. wrist verts that+ have minor finger-bone influences but are mostly weighted to the+ forearm).++ Returns the number of vertices deleted from *mesh_obj*.+ """+ vgs = mesh_obj.vertex_groups+ dropped_idxs = {vgs[name].index for name in dropped_bone_names if name in vgs}+ if not dropped_idxs:+ return 0++ verts_to_delete = []+ for vert in mesh_obj.data.vertices:+ if not vert.groups:+ continue+ max_weight = -1.0+ max_group = -1+ for g in vert.groups:+ if g.weight > max_weight:+ max_weight = g.weight+ max_group = g.group+ if max_group in dropped_idxs:+ verts_to_delete.append(vert.index)++ if not verts_to_delete:+ return 0++ bm = bmesh.new()+ bm.from_mesh(mesh_obj.data)+ bm.verts.ensure_lookup_table()+ bmesh.ops.delete(+ bm,+ geom=[bm.verts[i] for i in verts_to_delete],+ context='VERTS',+ )+ bm.to_mesh(mesh_obj.data)+ bm.free()+ mesh_obj.data.update()+ return len(verts_to_delete)+++def delete_skin_vertices_for_bones(armature, dropped_bone_names):+ """Delete dominantly-weighted vertices for *dropped_bone_names* across all meshes.++ Returns ``{mesh_name: n_deleted}`` for logging.+ """+ if not dropped_bone_names:+ return {}+ meshes = find_skinned_meshes(armature)+ if not meshes:+ logger.warning("No skinned meshes found targeting this armature; "+ "vertex deletion is a no-op")+ return {}+ deleted = {}+ for mesh_obj in meshes:+ n = delete_dominant_weighted_vertices(mesh_obj, dropped_bone_names)+ if n:+ deleted[mesh_obj.name] = n+ return deleted+++def remove_armature_bones(armature, bone_names):+ """Delete *bone_names* from *armature* in EDIT mode, keeping the+ hierarchy connected.++ Blender does NOT re-parent children when an edit bone is removed โ+ they silently become parentless roots. So before deleting, every+ surviving bone whose parent chain passes through a dropped bone is+ re-parented to its nearest surviving ancestor (the same bone its+ vertex weights are merged into under the 'merge' strategy). Edit-bone+ head/tail are stored in armature space, so re-parenting with+ ``use_connect=False`` preserves each bone's rest transform exactly.+ """+ bpy.context.view_layer.objects.active = armature+ bpy.ops.object.mode_set(mode='EDIT')+ edit_bones = armature.data.edit_bones+ dropped = {name for name in bone_names if name in edit_bones}++ for bone in edit_bones:+ if bone.name in dropped:+ continue+ if bone.parent is None or bone.parent.name not in dropped:+ continue+ ancestor = bone.parent+ while ancestor is not None and ancestor.name in dropped:+ ancestor = ancestor.parent+ bone.use_connect = False+ bone.parent = ancestor+ if ancestor is None:+ logger.warning(+ f"Bone '{bone.name}': entire parent chain was dropped; "+ f"it becomes a root bone"+ )+ else:+ logger.info(f"Re-parented '{bone.name}' to surviving ancestor "+ f"'{ancestor.name}'")++ for bone_name in dropped:+ edit_bones.remove(edit_bones[bone_name])+ bpy.ops.object.mode_set(mode='OBJECT')+++def sync_armature_bones(armature, anim_bone_names, extra_bones_strategy='merge'):+ """Reconcile armature bones with the animation bone list.++ *extra_bones_strategy* controls what happens to armature bones that are+ NOT present in *anim_bone_names*:++ - ``'merge'`` (default): transfer each extra bone's vertex weights to+ its nearest kept ancestor on every skinned mesh, then delete the+ bone. Mesh stays attached; articulation in the merged region is lost.+ - ``'remove'``: delete the bones AND every mesh vertex whose dominant+ (max-weight) bone is one of them. Vertices with mixed influences+ whose primary bone survives are kept, so we don't punch holes near+ boundary regions.+ - ``'keep'``: leave the bones in place with no keyframes. They sit at+ identity local pose and inherit their parent's animated transform+ through Blender's hierarchical pose evaluation.++ Animation bones absent from the armature are always logged and skipped+ by :func:`rebuild_action_from_data`.+ """+ if extra_bones_strategy not in EXTRA_BONES_STRATEGIES:+ raise ValueError(+ f"extra_bones_strategy must be one of {EXTRA_BONES_STRATEGIES}, "+ f"got {extra_bones_strategy!r}"+ )++ armature_bone_set = {bone.name for bone in armature.data.bones}+ anim_bone_set = {str(name) for name in anim_bone_names}++ extra = armature_bone_set - anim_bone_set+ if extra:+ if extra_bones_strategy == 'keep':+ logger.info(+ f"Keeping {len(extra)} armature bones not in animation "+ f"(no keyframes; will follow parent pose): {sorted(extra)}"+ )+ elif extra_bones_strategy == 'merge':+ ancestor_map = merge_extra_bone_weights(armature, anim_bone_set)+ summary = ', '.join(+ f"{src}->{dst or '<orphan>'}"+ for src, dst in sorted(ancestor_map.items())+ )+ logger.warning(+ f"Merged {len(extra)} extra armature bones into nearest kept "+ f"ancestor and removed them: {summary}"+ )+ remove_armature_bones(armature, extra)+ else: # 'remove'+ deleted = delete_skin_vertices_for_bones(armature, extra)+ if deleted:+ total = sum(deleted.values())+ summary = ', '.join(f"{name}: {n}" for name, n in sorted(deleted.items()))+ logger.warning(+ f"Deleted {total} mesh vertices dominantly weighted to "+ f"dropped bones ({summary})"+ )+ logger.warning(f"Removing armature bones not in animation: {sorted(extra)}")+ remove_armature_bones(armature, extra)++ missing = anim_bone_set - armature_bone_set+ if missing:+ logger.warning(f"Animation bones not in armature (will be skipped): {missing}")++ update_scene()+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# 5. Character export+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def export_animated_character(output_path_no_ext, formats=('glb',)):+ """Export every MESH + ARMATURE object to ``<output_path_no_ext>.<fmt>``.++ Stale actions other than the reconstructed one are removed first so the+ exporters only bake the driving animation.+ """+ export_objs = [obj for obj in bpy.context.scene.objects+ if obj.type in ('MESH', 'ARMATURE')]+ select_objs(export_objs, deselect_first=True)++ for action in list(bpy.data.actions):+ if action.name != RECONSTRUCTED_ACTION_NAME:+ bpy.data.actions.remove(action)+ # Imported characters can carry object-level animation data (static+ # transform keys); the glTF exporter samples it into a stray extra+ # animation, so drop everything but the reconstructed armature action.+ for obj in export_objs:+ anim_data = obj.animation_data+ if anim_data is not None and (+ anim_data.action is None+ or anim_data.action.name != RECONSTRUCTED_ACTION_NAME):+ obj.animation_data_clear()++ for fmt in formats:+ export_selected_to_file(f"{output_path_no_ext}.{fmt}", fmt)+++def export_selected_to_file(filepath, char_anim_type='glb'):+ """Export currently-selected scene objects to GLB or FBX."""+ if char_anim_type == 'glb':+ bpy.ops.export_scene.gltf(+ filepath=filepath,+ check_existing=False,+ use_selection=True,+ export_format='GLB',+ # Export the action's own F-curves rather than resampling every+ # object: sampling also bakes the armature *object's* static+ # transform into a stray second glTF animation.+ export_force_sampling=False,+ )+ elif char_anim_type == 'fbx':+ # The FBX exporter drops textures that aren't wired directly into a+ # Principled BSDF (Mix nodes, unlit materials); flatten those first.+ simplify_materials_for_fbx(bpy.context.selected_objects)+ # Sidecar copies for FBX consumers that ignore embedded media;+ # embedding itself reads packed data or these files.+ write_packed_images_to_dir(+ os.path.join(os.path.dirname(os.path.abspath(filepath)), 'textures'))+ bpy.ops.export_scene.fbx(+ filepath=filepath,+ check_existing=False,+ use_selection=True,+ use_triangles=True,+ add_leaf_bones=False,+ bake_anim=True,+ path_mode="COPY",+ embed_textures=True,+ )+ else:+ raise RuntimeError(f"Invalid char_anim_type: {char_anim_type}")+ logger.info(f"Saved animated character: {filepath}")++diff --git a/data_process/utils/export_markers.py b/data_process/utils/export_markers.pynew file mode 100644index 0000000..bf9bc1f--- /dev/null+++ b/data_process/utils/export_markers.py@@ -0,0 +1,106 @@+"""Per-asset export completion markers (resume bookkeeping).++Skeleton pruning is joint across all of an asset's clips, so an asset only+completes atomically. A marker under ``<output_dir>/.completed/`` records+that โ including assets that legitimately produce no clips+(``status="skipped"``), so reruns don't reload them. Delete a marker (or the+directory) to force a re-export, e.g. after changing joint-count limits.++Each marker also carries the asset's pruned joint names, which makes the+canonical ``joint_names.json`` rebuildable from disk at any time. That+matters because markers are durable and written per asset, while the+in-memory joint-name dict is only flushed at the very end of a run: without+the marker copy, a worker killed by a Slurm walltime or an OOM leaves+markers that suppress re-export but no joint names, losing those entries+permanently.++Deliberately free of any ``bpy`` import so plain-Python tools (notably+:mod:`data_process.tools.merge_summaries`) can use it outside Blender.+"""++import json+import os+++def _asset_marker_path(output_dir, save_name):+ return os.path.join(output_dir, ".completed", f"{save_name}.json")+++def is_asset_complete(output_dir, save_name):+ """True when a completion marker exists for *save_name*."""+ return os.path.isfile(_asset_marker_path(output_dir, save_name))+++def mark_asset_complete(output_dir, save_name, status="exported", n_clips=None,+ reason="", joint_names=None):+ """Write the completion marker for one fully-processed asset.++ Written atomically โ a half-written marker would otherwise be+ indistinguishable from a complete one on the next run.+ """+ path = _asset_marker_path(output_dir, save_name)+ os.makedirs(os.path.dirname(path), exist_ok=True)+ payload = {"status": status}+ if n_clips is not None:+ payload["n_clips"] = int(n_clips)+ if reason:+ payload["reason"] = reason+ if joint_names is not None:+ payload["joint_names"] = [str(n) for n in joint_names]+ tmp = f"{path}.tmp.{os.getpid()}"+ with open(tmp, "w") as f:+ json.dump(payload, f)+ os.replace(tmp, path)+++def collect_joint_names_from_markers(output_dir):+ """Rebuild ``{save_name: joint_names}`` from the per-asset markers.++ Only markers that actually recorded joint names contribute; assets that+ produced no clips (status ``"skipped"``) never do.+ """+ marker_dir = os.path.join(output_dir, ".completed")+ if not os.path.isdir(marker_dir):+ return {}++ recovered = {}+ for fname in sorted(os.listdir(marker_dir)):+ if not fname.endswith(".json"):+ continue+ try:+ with open(os.path.join(marker_dir, fname)) as f:+ payload = json.load(f)+ except (OSError, ValueError):+ continue+ names = payload.get("joint_names")+ if names:+ recovered[os.path.splitext(fname)[0]] = names+ return recovered+++def merge_joint_names(output_dir, all_joint_names, log=None):+ """Union this run's joint names over the markers and the existing file.++ Precedence (lowest to highest): the canonical ``joint_names.json`` on+ disk, the per-asset completion markers, then *all_joint_names* from the+ current run.+ """+ merged = {}++ canonical = os.path.join(output_dir, "joint_names.json")+ if os.path.isfile(canonical):+ try:+ with open(canonical) as f:+ merged.update(json.load(f))+ except (OSError, ValueError) as exc:+ if log:+ log(f"Could not read {canonical} ({exc}); rebuilding from markers only.")++ from_markers = collect_joint_names_from_markers(output_dir)+ new_from_markers = len(set(from_markers) - set(merged))+ merged.update(from_markers)+ if new_from_markers and log:+ log(f"Recovered {new_from_markers} joint-name entries from completion markers")++ merged.update(all_joint_names)+ return mergeddiff --git a/data_process/utils/motion_features.py b/data_process/utils/motion_features.pynew file mode 100644index 0000000..36ece22--- /dev/null+++ b/data_process/utils/motion_features.py@@ -0,0 +1,803 @@+"""Per-object motion feature extraction (export NPZ โ canonical clips).++Pipeline:+ 1. ``process_tpose`` โ load / canonicalize the T-pose, compute+ scale + rest offsets.+ 2. ``build_topology_cond`` โ build the per-object topology condition+ dict (joint graph, spectral feats, โฆ).+ 3. ``load_and_canonicalize_motion`` + ``trim_static_ends``+ โ load / normalize each motion, strip+ static pre/post-roll.+ 4. ``extract_clip_feats`` + ``save_clip``+ โ slice into clips, dump per-frame arrays+ (positions / rotations / facing) and an+ MP4 visualization.++``process_object`` orchestrates 1โ4 for one object type and returns an+``ObjectResult`` (see below) summarizing the saved artifacts.++Output layout under ``save_dir``:+ motions/{object}-{motion}-{clip_idx}.npz # per-clip per-frame arrays+ videos/{object}-{motion}-{clip_idx}.mp4+ tpose/{object}.png # T-pose still+ (driver-level) cond.npy, captions.json, filtered_clips.json, metadata.txt++The driver is :mod:`data_process.feature_extraction.extract_features`.+"""++import os+from os.path import join as pjoin+from pathlib import Path++import numpy as np+from loguru import logger++from Animation import (+ Animation,+ positions_global,+ rotations_global,+ offsets_from_positions+)+from Quaternions import Quaternions++from data_process.utils.topology import (+ compute_edge_relations_and_distances,+ compute_kinematic_chains,+ compute_joint_depths,+ compute_edge_indexs,+ compute_laplacian_eigenvectors,+)+from data_process.utils.skeleton import (+ reorder_anim_bfs,+ get_root_facing_quat,+ canonicalize_anim,+ get_motion_activity_metrics,+ get_skeleton_diameter,+)+from data_process.utils.plotting import (+ save_skeleton_motion,+ save_skeleton_motion_ground,+ save_skeleton_motion_spectral,+ save_skeleton_tpose,+ save_skeleton_tpos_spectral,+)+from data_process.feature_extraction.metadata import (+ atomic_output_path,+ get_motion_clip_windows,+)+++# Smallest leaf-to-leaf diameter a skeleton may have and still be scaled to+# the canonical target diameter (below this the scale factor blows up / divides+# by zero).+MIN_SKELETON_DIAMETER = 1e-8+++class DegenerateSkeletonError(ValueError):+ """Raised when a T-pose cannot be canonicalized (e.g. zero-length bones).++ Callers treat this as a skip-with-reason for the whole object type rather+ than a hard failure.+ """+++# ===========================================================================+# Joint-name resolution+# ===========================================================================++def resolve_face_joint_idxs(face_joints, names):+ """Resolve face joint names to indices from a face_joints dict.++ Returns the sentinel ``[-1, -1]`` when the entry is missing, empty, or+ its names don't appear in the skeleton โ downstream code treats that as+ identity facing rather than skipping the object.++ Returns:+ Tuple of (face_joint_idxs, body_axis).+ """+ if not face_joints or 'r_hip' not in face_joints or 'l_hip' not in face_joints:+ return [-1, -1], False+ r_hip_name = face_joints['r_hip']['raw']+ l_hip_name = face_joints['l_hip']['raw']+ if r_hip_name in names and l_hip_name in names:+ idxs = [names.index(r_hip_name), names.index(l_hip_name)]+ return idxs, face_joints.get('body_axis', False)+ return [-1, -1], False+++def resolve_joint_names(object_type, tpos_data, corps_names=None,+ clean_names=None, expected_names=None):+ """Resolve joint names and corps subset indices for an object type.++ Returns:+ Tuple of (corps_idxs, corps_names, clean_names_list, raw_parents,+ raw_njoints).+ """+ raw_parents = tpos_data['parents'].tolist()+ raw_njoints = len(raw_parents)+ raw_names = tpos_data['names'].tolist()++ # clean_names comes from clean_joint_names.json, whose entries are+ # positionally aligned with joint_names.json โ NOT necessarily with this+ # NPZ: rig variants can enumerate the same skeleton in a different order+ # (e.g. Truebones KingCobra), and the T-pose reference is simply the+ # first NPZ in sorted order. Align by the expected name list when given;+ # a silent positional paste would tag every joint with the wrong label.+ raw_clean = clean_names if clean_names else raw_names+ if clean_names and expected_names is not None:+ if len(clean_names) != len(expected_names):+ raise ValueError(+ f'[{object_type}] clean_joint_names has {len(clean_names)} '+ f'entries but joint_names.json has {len(expected_names)}; '+ f'clean_joint_names.json is stale โ re-run the joint-name '+ f'cleaner.')+ if raw_names != expected_names:+ index = {n: i for i, n in enumerate(expected_names)}+ try:+ raw_clean = [clean_names[index[n]] for n in raw_names]+ except KeyError as e:+ raise ValueError(+ f'[{object_type}] T-pose reference NPZ contains joint '+ f'{e} that is absent from joint_names.json; the export '+ f'metadata is out of date.')+ logger.info(f'[{object_type}] T-pose NPZ joint order differs '+ f'from joint_names.json; clean names realigned by '+ f'name.')++ corps_idxs = None+ if corps_names is not None:+ try:+ corps_idxs = [raw_names.index(n) for n in corps_names]+ except ValueError as e:+ raise ValueError(+ f'Failed to resolve the requested joint subset (mixamo core '+ f'joints / corps_names) for {object_type} in tpos_data: {e}. '+ f'Raw names: {raw_names}')+ clean_names_list = [raw_clean[i] for i in corps_idxs]+ else:+ clean_names_list = list(raw_clean)++ return corps_idxs, corps_names, clean_names_list, raw_parents, raw_njoints+++# ===========================================================================+# NPZ loading+# ===========================================================================++def _names_permutation(ref_names, ref_parents, names, parents):+ """Permutation aligning a clip's joint order to the reference skeleton.++ Returns ``perm`` such that ``array[:, perm]`` reorders per-joint data+ into the reference joint order, or None when the clip is not the same+ skeleton (different name set, duplicate names, or a topology that+ disagrees under the name mapping).+ """+ if sorted(names) != sorted(ref_names) or len(set(names)) != len(names):+ return None+ index = {n: i for i, n in enumerate(names)}+ perm = [index[n] for n in ref_names]+ for i, p in enumerate(ref_parents):+ q = parents[perm[i]]+ if (q == -1) != (p == -1) or (p != -1 and q != perm[p]):+ return None+ return perm+++def _load_npz_anim(object_npz_path, raw_parents, raw_njoints,+ corps_idxs, bfs_order, fps, raw_names=None):+ """Load a motion NPZ, apply joint subset/reorder, and downsample.++ Returns:+ Tuple of (motion_anim, motion_fps, None) on success, or+ (None, None, reason_string) when the motion should be skipped.+ """+ anim_data = np.load(object_npz_path, allow_pickle=True)+ nframes, njoints = anim_data['anim_local_rot'].shape[:2]++ if njoints != raw_njoints:+ reason = f'joint count mismatch: expected {raw_njoints}, got {njoints}'+ logger.warning(f'Skipping clip ({reason}): {object_npz_path}')+ return None, None, reason++ anim_local_rot = anim_data['anim_local_rot']+ anim_local_pos = anim_data['anim_local_pos']+ rest_local_pos = anim_data['rest_local_pos']++ # corps_idxs/bfs_order are positional indices into the T-pose clip's+ # joint order; a source file enumerating the same skeleton in a+ # different order would silently misassign joints, so align by name.+ if raw_names is not None:+ clip_names = anim_data['names'].tolist()+ if clip_names != raw_names:+ perm = _names_permutation(raw_names, raw_parents, clip_names,+ anim_data['parents'].tolist())+ if perm is None:+ reason = 'joint names mismatch with T-pose skeleton'+ logger.warning(f'Skipping clip ({reason}): {object_npz_path}')+ return None, None, reason+ logger.info(f'Joint order differs from T-pose skeleton; '+ f'reordering by name: {object_npz_path}')+ anim_local_rot = anim_local_rot[:, perm]+ anim_local_pos = anim_local_pos[:, perm]+ rest_local_pos = rest_local_pos[perm]++ motion_positions = rest_local_pos.copy()[None, :].repeat(nframes, axis=0)+ motion_positions[:, 0] = anim_local_pos[:, 0] # root joint's local position is the animated global position++ motion_anim = Animation(+ rotations=Quaternions(anim_local_rot),+ positions=motion_positions,+ orients=Quaternions.id(raw_njoints),+ offsets=rest_local_pos,+ parents=raw_parents,+ )++ if corps_idxs is not None:+ motion_anim = motion_anim[:, corps_idxs]+ motion_anim = motion_anim[:, bfs_order]++ if fps >= 60:+ motion_anim = motion_anim[::2]+ motion_fps = fps // 2+ else:+ motion_fps = fps++ return motion_anim, motion_fps, None+++def _load_tpos_npz(tpos_data, corps_idxs=None, corps_names=None):+ """Load T-pose from NPZ data, apply joint subset and BFS reorder.++ Returns:+ Tuple of (tpos_anim, parents, names, fps, bfs_order).+ """+ names, fps = tpos_data['names'].tolist(), tpos_data['fps'].item()+ parents = tpos_data['parents'].tolist()+ njoints = len(parents)++ tpos_anim = Animation(+ rotations=Quaternions(tpos_data['rest_local_rot'][None]),+ positions=tpos_data['rest_local_pos'][None],+ orients=Quaternions.id(njoints),+ offsets=tpos_data['rest_local_pos'],+ parents=tpos_data['parents'],+ )++ if corps_idxs is not None:+ tpos_anim = tpos_anim[:, corps_idxs]+ names = corps_names if corps_names is not None else [names[i] for i in corps_idxs]++ tpos_anim, bfs_order, parents = reorder_anim_bfs(tpos_anim)+ names = [names[i] for i in bfs_order]++ return tpos_anim, parents, names, fps, bfs_order+++def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,+ corps_idxs, bfs_order, fps, scale_factor,+ ground_height, face_joint_idxs,+ body_axis=False, raw_names=None):+ """Load a motion NPZ and canonicalize facing / XZ / ground + scale.++ ``scale_factor`` and ``ground_height`` are reused from T-pose+ calibration so every motion of this object type shares a+ diameter-consistent scale and a consistent floor height (matching the+ npz pipeline); ``root_pose_init_xz`` is recomputed fresh per+ motion so each motion starts centered at the origin. Clips are then+ simple slices of the canonicalized motion.++ Returns:+ Tuple of ((motion_anim, motion_fps), None) on success, or+ (None, reason_string) when the motion should be skipped.+ """+ motion_anim, motion_fps, skip_reason = _load_npz_anim(+ object_npz_path, raw_parents, raw_njoints, corps_idxs, bfs_order, fps,+ raw_names=raw_names)+ if motion_anim is None:+ return None, skip_reason++ motion_anim, _, _, _ = canonicalize_anim(+ motion_anim,+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,+ ground_height=ground_height,+ scale_factor=scale_factor)+ return (motion_anim, motion_fps), None+++# ===========================================================================+# T-pose processing and topology conditioning+# ===========================================================================++def process_tpose(tpos_data, corps_idxs=None, corps_names=None,+ face_joints=None, target_diameter=2.0):+ """Load T-pose from NPZ with facing -> center XZ -> scale -> ground.++ ``scale_factor``, ``ground_height`` (and ``offsets``) are threaded+ downstream so motions share a diameter-consistent scale and a+ consistent floor height across the object type (matching the npz+ pipeline); each motion's ``root_pose_init_xz`` is still+ recomputed fresh so every motion starts centered at origin.++ Raises:+ DegenerateSkeletonError: if the skeleton has no measurable extent+ (all-zero bone offsets), which would make diameter scaling divide+ by zero.++ Returns:+ Tuple of (tpos_anim, offsets, scale_factor, ground_height,+ parents, names, fps, bfs_order,+ face_joint_idxs, body_axis).+ """+ tpos_anim, parents, names, fps, bfs_order = _load_tpos_npz(+ tpos_data, corps_idxs, corps_names)++ diameter = float(get_skeleton_diameter(tpos_anim))+ if not np.isfinite(diameter) or diameter <= MIN_SKELETON_DIAMETER:+ raise DegenerateSkeletonError(+ f'degenerate skeleton: leaf-to-leaf diameter {diameter:.3e} '+ f'(zero-length bones?), cannot scale to target diameter '+ f'{target_diameter}')++ face_joint_idxs, body_axis = resolve_face_joint_idxs(face_joints, names)+ tpos_anim, _, ground_height, scale_factor = canonicalize_anim(+ tpos_anim,+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,+ target_scale_metric=target_diameter)++ offsets = np.array(tpos_anim.offsets).copy() # (J, 3)++ return (tpos_anim, offsets, scale_factor, ground_height,+ parents, names, fps, bfs_order,+ face_joint_idxs, body_axis)+++def build_topology_cond(object_type, parents, offsets, names, clean_joint_names,+ tpos_global_pos, tpos_local_rot, tpos_global_rot,+ max_path_len=5, max_freqs=8,+ face_joint_idxs=None, body_axis=False,+ scale_factor=1.0, ground_height=None):+ """Build the topology conditioning dict from a canonicalized T-pose.++ Produces every field consumed by the training dataset loader plus the+ joint-name and optional face-joint metadata fields.++ ``ground_height`` is the vertical offset actually subtracted from every+ clip of this object type, or None when each motion was grounded on its+ own minimum Y (the default). It is recorded together with+ ``ground_height_mode`` (``'tpos'`` / ``'per_motion'``) so the cond stays+ self-describing โ do not pass the T-pose's own ground height here unless+ that same value was reused for the motions.+ """+ parents_arr = np.asarray(parents)+ joint_relations, joint_graph_dists = compute_edge_relations_and_distances(+ parents_arr, max_path_len=max_path_len,+ )+ joint_depths = compute_joint_depths(parents_arr)+ edge_indexs = compute_edge_indexs(parents_arr)+ spectral_feats, _ = compute_laplacian_eigenvectors(parents_arr, max_freqs=max_freqs)+ kinematic_chains = compute_kinematic_chains(parents_arr)++ tpos_offsets = offsets_from_positions(tpos_global_pos, parents_arr)++ cond = {+ 'object_type': object_type,+ 'parents': parents,+ 'offsets': offsets, # (J, 3)+ 'tpos_offsets': tpos_offsets, # (J, 3)+ 'joint_names': names,+ 'clean_joint_names': clean_joint_names,+ 'tpos_first_frame': tpos_global_pos, # (J, 3)+ 'tpos_local_rotations': tpos_local_rot, # (J, 4) quaternion+ 'tpos_global_rotations': tpos_global_rot, # (J, 4) quaternion+ 'joint_relations': joint_relations, # (J, J)+ 'joint_graph_dists': joint_graph_dists, # (J, J)+ 'joint_depths': joint_depths, # (J,)+ 'edge_indexs': edge_indexs, # (2, 2*(J-1))+ 'spectral_feats': spectral_feats, # (J, K)+ 'kinematic_chains': kinematic_chains, # list of lists of joint indices+ 'scale_factor': scale_factor, # float+ 'ground_height': ground_height, # float, or None per-motion+ 'ground_height_mode': 'tpos' if ground_height is not None else 'per_motion',+ }++ if face_joint_idxs is not None:+ cond['face_joint_idxs'] = {+ 'r_hip': face_joint_idxs[0],+ 'l_hip': face_joint_idxs[1],+ 'body_axis': body_axis,+ }++ return cond+++# ===========================================================================+# Motion clip processing+# ===========================================================================++def trim_static_ends(motion_anim, static_threshold=1e-5):+ """Strip leading/trailing frames where every joint is nearly stationary.++ Per-frame activity is the max over joints of consecutive global-position+ displacement. Frames at the head/tail with activity below+ ``static_threshold`` are removed; interior static spans are preserved.++ Returns:+ Tuple of (trimmed_anim, start_idx, end_idx). ``end_idx`` is+ exclusive; returns an empty slice when the entire motion is static.+ """+ nframes = len(motion_anim)+ if nframes < 2:+ return motion_anim, 0, nframes++ global_positions = positions_global(motion_anim)+ step = np.linalg.norm(np.diff(global_positions, axis=0), axis=-1) # (F-1, J)+ max_joint_step = step.max(axis=-1) # (F-1,)++ active = max_joint_step > static_threshold+ if not active.any():+ return motion_anim[:0], 0, 0++ active_idx = np.where(active)[0]+ start = int(active_idx[0])+ end = min(int(active_idx[-1]) + 2, nframes) # +1 diff offset, +1 exclusive+ return motion_anim[start:end], start, end+++def extract_clip_feats(motion_anim, clip_start, clip_end,+ face_joint_idxs, body_axis=False):+ """Slice a canonicalized motion into a clip and extract per-frame data.++ The motion is assumed to already be canonicalized against T-pose+ calibration (facing / XZ / ground / scale) by+ :func:`load_and_canonicalize_motion`, so clips are simple slices โ no+ per-clip re-centering.++ Returns:+ Tuple of (local_rotations (F, J, 4), global_positions (F, J, 3),+ root_facing_quat (F, 4)).+ """+ clip_anim = motion_anim[clip_start:clip_end]+ local_rotations = clip_anim.rotations.copy() # (F, J, 4)+ global_positions = positions_global(clip_anim)+ root_facing_quat = get_root_facing_quat(+ global_positions, face_joint_idxs, body_axis=body_axis)++ return local_rotations.qs, global_positions, root_facing_quat.qs+++def _activity_below_threshold(global_positions, threshold):+ """Return (is_low, metrics) from joint-activity metrics for a sequence."""+ metrics = get_motion_activity_metrics(global_positions)+ is_low = (metrics['joint_activity_topk'] < threshold+ and metrics['root_activity'] < threshold)+ return is_low, metrics+++def _format_activity(metrics):+ return (f'joint={metrics["joint_activity"]:.4f}, '+ f'topk={metrics["joint_activity_topk"]:.4f}, '+ f'root={metrics["root_activity"]:.4f}')+++# ===========================================================================+# Output I/O+# ===========================================================================++def save_clip(clip_name,+ global_positions, local_rotations, root_facing_quat,+ parents, motion_fps,+ motions_save_dir, animations_save_dir,+ spectral_feats=None, caption=None, save_vis=True,+ vis_ground=False):+ """Save per-clip canonicalized positions/rotations/facing + MP4.++ When ``spectral_feats`` is supplied, the MP4 colors joints by PCA-projected+ spectral features so topologically close joints share colors across+ object types; otherwise falls back to the default root/non-root palette.+ ``caption``, if non-empty, is rendered as the video title. The MP4+ dominates per-clip time; pass ``save_vis=False`` to skip it.++ The NPZ is written atomically: nothing downstream re-validates clip files,+ so an interrupted run must not leave a truncated one for the training+ loader to read.+ """+ with atomic_output_path(pjoin(motions_save_dir, clip_name + '.npz')) as tmp_path:+ # A file object keeps np.savez from appending '.npz' to the temp name.+ with open(tmp_path, 'wb') as f:+ np.savez(+ f,+ global_positions=global_positions,+ local_rotations=local_rotations,+ root_facing_quat=root_facing_quat,+ fps=int(motion_fps),+ )++ if not save_vis:+ return+ clip_vis_path = pjoin(animations_save_dir, clip_name + '.mp4')+ title = caption or None+ if vis_ground:+ # Checkerboard ground + follow camera + trajectory/shadows; keeps+ # the spectral palette when available.+ save_skeleton_motion_ground(+ save_path=clip_vis_path, parents=parents,+ positions=global_positions, spectral_feats=spectral_feats,+ fps=int(motion_fps), title=title,+ figsize=(6.4, 6.4), dpi=100,+ )+ return+ if spectral_feats is not None:+ save_skeleton_motion_spectral(+ save_path=clip_vis_path, parents=parents,+ positions=global_positions, spectral_feats=spectral_feats,+ fps=int(motion_fps), title=title,+ figsize=(6.4, 6.4), dpi=100,+ )+ else:+ save_skeleton_motion(+ save_path=clip_vis_path, parents=parents,+ positions=global_positions, fps=int(motion_fps), title=title,+ figsize=(6.4, 6.4), dpi=100,+ )+++def _save_tpose_vis(save_dir, object_type, parents, tpos_anim,+ spectral_feats=None):+ tpos_save_dir = pjoin(save_dir, 'tpose')+ os.makedirs(tpos_save_dir, exist_ok=True)+ tpos_vis_path = pjoin(tpos_save_dir, f'{object_type}.png')+ tpos_global_pos = positions_global(tpos_anim)[0]+ if spectral_feats is not None:+ save_skeleton_tpos_spectral(+ save_path=tpos_vis_path, parents=parents,+ positions=tpos_global_pos, spectral_feats=spectral_feats,+ figsize=(6.4, 6.4), dpi=100,+ )+ else:+ save_skeleton_tpose(+ save_path=tpos_vis_path, parents=parents,+ positions=tpos_global_pos,+ figsize=(6.4, 6.4), dpi=100,+ )+++# ===========================================================================+# Per-object pipeline+# ===========================================================================++def _process_motion_file(object_npz_path, ctx):+ """Process one motion NPZ into saved clips.++ ``ctx`` bundles the per-object state shared across every motion file of+ an object type (skeleton topology, scale, thresholds, save dirs,+ captions). Returns a dict with:+ n_saved : int โ number of clips written to disk+ frames : int โ total kept frames across clips+ clip_captions : Dict[str, str] โ per-clip caption map+ filtered : List[dict] โ entries dropped (with reason)+ """+ base_name = str(Path(object_npz_path).stem)+ min_frames = ctx['min_frames']++ # --- Load + motion-level canonicalize ----------------------------------+ result, skip_reason = load_and_canonicalize_motion(+ object_npz_path, ctx['raw_parents'], ctx['raw_njoints'],+ ctx['corps_idxs'], ctx['bfs_order'], ctx['fps'], ctx['scale_factor'],+ ctx['ground_height'], ctx['face_joint_idxs'],+ body_axis=ctx['body_axis'], raw_names=ctx.get('raw_names'))+ if result is None:+ return {'n_saved': 0, 'frames': 0, 'clip_captions': {},+ 'filtered': [{'name': base_name, 'reason': skip_reason}]}+ motion_anim, motion_fps = result++ # --- Strip static pre/post-roll ----------------------------------------+ orig_nframes = len(motion_anim)+ motion_anim, _, _ = trim_static_ends(+ motion_anim, static_threshold=ctx['static_threshold'])+ if len(motion_anim) < min_frames:+ reason = (f'too few frames after static trim: {len(motion_anim)} '+ f'< {min_frames} (orig {orig_nframes})')+ return {'n_saved': 0, 'frames': 0, 'clip_captions': {},+ 'filtered': [{'name': base_name, 'reason': reason}]}++ # --- Clip windowing ----------------------------------------------------+ nframes = len(motion_anim)+ clips = get_motion_clip_windows(+ nframes, ctx['max_clip_len'], ctx['clip_stride'], apply_clip=ctx['apply_clip'])+ caption = ctx['captions'].get(base_name, '') if ctx['captions'] else ''++ clip_captions, filtered, n_saved, frames = {}, [], 0, 0+ for clip_idx, (clip_start, clip_end) in enumerate(clips):+ (clip_local_rotations, clip_global_positions,+ clip_root_facing_quat) = extract_clip_feats(+ motion_anim, clip_start, clip_end,+ ctx['face_joint_idxs'],+ body_axis=ctx['body_axis'],+ )++ is_low, clip_metrics = _activity_below_threshold(+ clip_global_positions, ctx['activity_threshold'])+ clip_name = f'{base_name}-{clip_idx:03d}'+ if is_low:+ filtered.append({+ 'name': clip_name,+ 'reason': f'low activity: {_format_activity(clip_metrics)}',+ })+ continue++ save_clip(+ clip_name,+ clip_global_positions, clip_local_rotations,+ clip_root_facing_quat,+ ctx['parents'], motion_fps,+ ctx['motions_save_dir'], ctx['animations_save_dir'],+ spectral_feats=ctx.get('spectral_feats'),+ caption=caption, save_vis=ctx['save_vis'],+ vis_ground=ctx.get('vis_ground', False))+ n_saved += 1+ frames += clip_end - clip_start+ if caption:+ clip_captions[clip_name] = caption++ return {+ 'n_saved': n_saved,+ 'frames': frames,+ 'clip_captions': clip_captions,+ 'filtered': filtered,+ }+++def process_object(object_type, object_npzs, save_dir,+ max_path_len=5, max_freqs=8,+ max_clip_len=100, clip_stride=50, apply_clip=True,+ captions=None,+ face_joints=None, corps_names=None, clean_names=None,+ expected_names=None, require_face=False,+ target_diameter=2.0, activity_threshold=0.02,+ static_threshold=1e-05, min_frames=8,+ min_joints=None, max_joints=None,+ use_tpos_ground_height=False, save_vis=True,+ vis_ground=False):+ """Process all motion clips for one object type.++ Emits per-clip NPZs holding raw canonicalized arrays โ global+ positions, local rotation quaternions, root facing quaternions โ+ from which the training side builds its motion representation.++ When ``apply_clip`` is False, each motion yields a single clip truncated+ to the first ``max_clip_len`` frames; when True, motions are cropped+ into overlapping fixed-length clips with stride ``clip_stride``.++ ``use_tpos_ground_height`` controls whether the T-pose's ground height+ is reused for every motion (True) or recomputed per motion from its own+ minimum Y so each motion's lowest joint sits on the ground (False,+ default). Whichever is used is recorded in the cond as ``ground_height``+ / ``ground_height_mode``.++ Object types with no motion NPZs, a joint count outside+ ``[min_joints, max_joints]`` or a degenerate (zero-extent) skeleton are+ skipped with a reason instead of raising.++ Returns:+ Tuple of (cond, n_clips, n_frames, n_joints, filtered).+ """+ def _skip(reason, njoints=0):+ """Skip the whole object type with a recorded reason."""+ logger.info(f'[{object_type}] skipped: {reason}')+ return None, 0, 0, njoints, [{'name': f'{object_type} (all clips)',+ 'reason': reason}]++ if not object_npzs:+ return _skip('no motion NPZ files found')++ motions_save_dir = pjoin(save_dir, 'motions')+ animations_save_dir = pjoin(save_dir, 'videos')+ for d in (motions_save_dir, animations_save_dir):+ os.makedirs(d, exist_ok=True)++ # ---- T-pose + topology ----+ tpos_data = np.load(object_npzs[0], allow_pickle=True)+ (corps_idxs, corps_names_list, clean_names_list,+ raw_parents, raw_njoints) = resolve_joint_names(+ object_type, tpos_data, corps_names, clean_names,+ expected_names=expected_names)++ # Joint count is fixed by the corps subset (BFS reordering preserves it),+ # so gate before the T-pose canonicalization rather than after.+ njoints = len(corps_idxs) if corps_idxs is not None else raw_njoints+ if ((min_joints is not None and njoints < min_joints) or+ (max_joints is not None and njoints > max_joints)):+ return _skip(f'{njoints} joints outside [{min_joints}, {max_joints}]',+ njoints)++ try:+ (tpos_anim, offsets, scale_factor, tpos_ground_height,+ parents, names, fps, bfs_order,+ face_joint_idxs, body_axis) = process_tpose(+ tpos_data, corps_idxs=corps_idxs,+ corps_names=corps_names_list,+ face_joints=face_joints,+ target_diameter=target_diameter)+ except DegenerateSkeletonError as e:+ return _skip(str(e), njoints)++ face_named = bool(face_joints) and bool(+ face_joints.get('r_hip', {}).get('raw')+ or face_joints.get('l_hip', {}).get('raw'))+ if face_named and face_joint_idxs == [-1, -1]:+ # The entry exists but its raw names are not in this (possibly+ # corps-filtered) skeleton โ canonicalization silently ran with+ # identity facing above. Never acceptable where facing is required.+ detail = (f'face_joints entry did not resolve: '+ f'r_hip={face_joints.get("r_hip", {}).get("raw")!r}, '+ f'l_hip={face_joints.get("l_hip", {}).get("raw")!r} '+ f'not both present in the processed skeleton')+ if require_face:+ raise ValueError(f'[{object_type}] {detail}; refusing to fall '+ f'back to identity facing for this dataset.')+ logger.warning(f'[{object_type}] {detail}; using identity facing.')++ # Ground height actually applied to the clips: the T-pose's when it is+ # reused, otherwise None (each motion is grounded on its own minimum Y).+ ground_height = tpos_ground_height if use_tpos_ground_height else None++ clean_names_bfs = [clean_names_list[i] for i in bfs_order]+ tpos_global_pos = positions_global(tpos_anim)[0]+ tpos_local_rot = tpos_anim.rotations.qs[0]+ tpos_global_rot = rotations_global(tpos_anim).qs[0]+ object_cond = build_topology_cond(+ object_type, parents, offsets, names, clean_names_bfs,+ tpos_global_pos, tpos_local_rot, tpos_global_rot,+ max_path_len=max_path_len, max_freqs=max_freqs,+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,+ scale_factor=scale_factor, ground_height=ground_height+ )+ spectral_feats = object_cond.get('spectral_feats')++ _save_tpose_vis(save_dir, object_type, parents, tpos_anim,+ spectral_feats=spectral_feats)++ logger.info(f'[{object_type}] n_joints={njoints} fps={fps} '+ f'scale={scale_factor:.4f} files={len(object_npzs)}')++ # ---- Per-motion processing ----+ ctx = {+ # skeleton / scale / topology+ 'raw_parents': raw_parents, 'raw_njoints': raw_njoints,+ 'raw_names': tpos_data['names'].tolist(),+ 'corps_idxs': corps_idxs, 'bfs_order': bfs_order,+ 'parents': parents,+ 'fps': fps, 'scale_factor': scale_factor,+ 'ground_height': ground_height, # None => per-motion grounding+ 'face_joint_idxs': face_joint_idxs, 'body_axis': body_axis,+ 'spectral_feats': spectral_feats,+ # clip windowing / filtering+ 'max_clip_len': max_clip_len, 'clip_stride': clip_stride,+ 'apply_clip': apply_clip,+ 'activity_threshold': activity_threshold,+ 'static_threshold': static_threshold,+ 'min_frames': min_frames,+ # I/O+ 'motions_save_dir': motions_save_dir,+ 'animations_save_dir': animations_save_dir,+ 'captions': captions,+ 'save_vis': save_vis, 'vis_ground': vis_ground,+ }++ filtered_clips, clip_captions = [], {}+ n_clips, n_frames = 0, 0++ for object_npz_path in object_npzs:+ result = _process_motion_file(object_npz_path, ctx)+ filtered_clips.extend(result.get('filtered', []))+ n_clips += result.get('n_saved', 0)+ n_frames += result.get('frames', 0)+ clip_captions.update(result.get('clip_captions', {}))++ if clip_captions:+ object_cond['captions'] = clip_captions++ logger.info(f'[{object_type}] saved {n_clips} clips ({n_frames} frames), '+ f'filtered {len(filtered_clips)}')++ return object_cond, n_clips, n_frames, njoints, filtered_clipsdiff --git a/data_process/utils/plotting.py b/data_process/utils/plotting.pynew file mode 100644index 0000000..19f3c87--- /dev/null+++ b/data_process/utils/plotting.py@@ -0,0 +1,1039 @@+"""3D skeleton visualization utilities (matplotlib, headless Agg backend).++All public functions operate on a parent-array skeleton representation:++ parents : array_like of shape (J,)+ parents[j] is the index of joint j's parent, or -1 for the root.+ positions : array_like+ Joint positions. Single pose: (J, 3) or (J, D) with D >= 3 (extra+ feature dims are stripped). Motion: (T, J, 3) or (T, J, D).++Two families of functions:++ render_skeleton_* -> RGB numpy array(s) in memory+ save_skeleton_* -> write PNG (pose) / MP4 (motion) to disk+"""++from pathlib import Path+from typing import List, Optional, Sequence, Tuple++import numpy as np+import matplotlib+import imageio+from matplotlib.backends.backend_agg import FigureCanvasAgg+from matplotlib.figure import Figure+from matplotlib.ticker import LinearLocator+from mpl_toolkits.mplot3d.art3d import Line3DCollection+from mpl_toolkits.mplot3d import proj3d++matplotlib.use('Agg')+++# ============================================================================+# Geometry / array helpers+# ============================================================================++def _as_xyz(positions) -> np.ndarray:+ """Cast to float32 and truncate the last axis to xyz."""+ pts = np.asarray(positions, dtype=np.float32)+ if pts.shape[-1] > 3:+ pts = pts[..., :3]+ return pts+++def _maybe_rotate_y_up(pts: np.ndarray, rotate: bool) -> np.ndarray:+ """Swap Y/Z so a Z-up skeleton renders upright in matplotlib's Y-up view."""+ if not rotate:+ return pts+ out = pts.copy()+ out[..., 1] = -pts[..., 2]+ out[..., 2] = pts[..., 1]+ return out+++def _edges_from_parents(parents: Sequence[int]) -> np.ndarray:+ """(E, 2) edge list of (child, parent) pairs, skipping the root."""+ return np.array(+ [(i, p) for i, p in enumerate(parents) if p != -1],+ dtype=np.int32,+ )+++def _cubic_bounds(points: np.ndarray):+ """Axis-aligned cubic bounding box centered on the points (preserves aspect)."""+ x_min, x_max = points[..., 0].min(), points[..., 0].max()+ y_min, y_max = points[..., 1].min(), points[..., 1].max()+ z_min, z_max = points[..., 2].min(), points[..., 2].max()+ side = max(x_max - x_min, y_max - y_min, z_max - z_min)+ cx, cy, cz = (x_min + x_max) / 2, (y_min + y_max) / 2, (z_min + z_max) / 2+ half = side / 2+ return cx - half, cx + half, cy - half, cy + half, cz - half, cz + half+++def _tight_bounds(points: np.ndarray, pad_ratio: float = 0.08,+ abs_pad_ratio: float = 0.0):+ """Per-axis bounding box with proportional padding; pairs with non-cubic box_aspect.++ ``pad_ratio`` adds *proportional* padding (``pad_ratio * axis_length``) โ+ tiny on a thin axis.+ ``abs_pad_ratio`` adds *absolute* padding tied to the overall scale+ (``abs_pad_ratio * max_axis_length``) โ the right knob when labels or+ annotations are placed at offsets proportional to the full skeleton size.+ """+ x_min, x_max = points[..., 0].min(), points[..., 0].max()+ y_min, y_max = points[..., 1].min(), points[..., 1].max()+ z_min, z_max = points[..., 2].min(), points[..., 2].max()+ # Guard against degenerate flat axes (e.g. planar skeleton).+ dx = max(x_max - x_min, 1e-6)+ dy = max(y_max - y_min, 1e-6)+ dz = max(z_max - z_min, 1e-6)+ scale = max(dx, dy, dz)+ abs_pad = abs_pad_ratio * scale+ px = dx * pad_ratio + abs_pad+ py = dy * pad_ratio + abs_pad+ pz = dz * pad_ratio + abs_pad+ return (x_min - px, x_max + px,+ y_min - py, y_max + py,+ z_min - pz, z_max + pz)+++def _box_aspect_from_bounds(bounds) -> Tuple[float, float, float]:+ x_min, x_max, y_min, y_max, z_min, z_max = bounds+ return (max(x_max - x_min, 1e-6),+ max(y_max - y_min, 1e-6),+ max(z_max - z_min, 1e-6))+++# ============================================================================+# Matplotlib figure helpers+# ============================================================================++def _new_figure(bounds, elev: Optional[float], azim: Optional[float],+ figsize=(6.4, 4.8), dpi: int = 120,+ box_aspect=(1, 1, 1)):+ """Create a 3D figure/axes configured for skeleton rendering.++ Use ``box_aspect=(1,1,1)`` with :func:`_cubic_bounds`, or per-axis extents+ with :func:`_tight_bounds` to fill the figure with a long-thin subject.+ """+ fig = Figure(figsize=figsize, dpi=dpi)+ canvas = FigureCanvasAgg(fig)+ ax = fig.add_subplot(111, projection='3d', computed_zorder=False)++ x_min, x_max, y_min, y_max, z_min, z_max = bounds+ ax.set_xlim([x_min, x_max])+ ax.set_ylim([y_min, y_max])+ ax.set_zlim([z_min, z_max])++ ax.set_xticks([]); ax.set_yticks([]); ax.set_zticks([])+ ax.set_xticklabels([]); ax.set_yticklabels([]); ax.set_zticklabels([])+ ax.xaxis.set_major_locator(LinearLocator(8))+ ax.yaxis.set_major_locator(LinearLocator(8))+ ax.zaxis.set_major_locator(LinearLocator(8))+ ax.grid(True, color='gray', linestyle='--', linewidth=0.5)+ ax.set_box_aspect(box_aspect)++ if elev is not None and azim is not None:+ ax.view_init(elev=elev, azim=azim)++ fig.subplots_adjust(left=-0.05, right=1.05, top=1.05, bottom=-0.05)+ return fig, canvas, ax+++def _capture_rgb(canvas: FigureCanvasAgg) -> np.ndarray:+ """Draw the canvas and return its pixels as an (H, W, 3) uint8 array."""+ canvas.draw()+ w, h = canvas.get_width_height()+ rgba = np.frombuffer(canvas.buffer_rgba(), dtype=np.uint8).reshape(h, w, 4)+ return rgba[..., :3].copy()+++# ============================================================================+# Skeleton drawing primitives+# ============================================================================++def _draw_static_skeleton(ax, pts: np.ndarray, parents: Sequence[int],+ edges: np.ndarray) -> None:+ """Draw bones (gray) and joints (root red, others blue) for a single pose."""+ segs = np.zeros((len(edges), 2, 3), dtype=np.float32)+ segs[:, 0] = pts[edges[:, 0]]+ segs[:, 1] = pts[edges[:, 1]]+ ax.add_collection3d(Line3DCollection(segs, colors='gray', linewidths=1.0))++ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')+ ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2], s=10, c=joint_colors)+ ax.scatter(pts[0:1, 0], pts[0:1, 1], pts[0:1, 2], s=15, c='r')+++def _init_motion_artists(ax, edges: np.ndarray, first_frame: np.ndarray,+ joint_colors, joint_size: float,+ bone_color: str = 'gray',+ bone_linewidth: float = 1.0,+ joint_edgecolors=None,+ joint_edgewidth: float = 0.0):+ """Create reusable bone/joint artists for a motion render.++ Returns ``(segs, lc, sc)`` where ``segs`` is the mutable (E, 2, 3) buffer+ backing the Line3DCollection ``lc``, and ``sc`` is the joint scatter.+ Callers mutate ``segs`` and ``sc._offsets3d`` per frame.+ """+ segs = np.zeros((len(edges), 2, 3), dtype=np.float32)+ lc = Line3DCollection(segs, colors=bone_color, linewidths=bone_linewidth)+ ax.add_collection3d(lc)++ kwargs = dict(s=joint_size, c=joint_colors, depthshade=False, zorder=2)+ if joint_edgecolors is not None:+ kwargs['edgecolors'] = joint_edgecolors+ kwargs['linewidths'] = joint_edgewidth+ sc = ax.scatter(first_frame[:, 0], first_frame[:, 1], first_frame[:, 2],+ **kwargs)+ return segs, lc, sc+++def _fibonacci_sphere_dirs(n: int) -> np.ndarray:+ """``n`` approximately uniform unit vectors on the sphere (Fibonacci spiral)."""+ phi = np.pi * (3.0 - np.sqrt(5.0)) # golden angle+ idx = np.arange(n, dtype=np.float32)+ y = 1.0 - 2.0 * idx / max(n - 1, 1)+ radius = np.sqrt(np.maximum(1.0 - y * y, 0.0))+ theta = phi * idx+ x = np.cos(theta) * radius+ z = np.sin(theta) * radius+ return np.stack([x, y, z], axis=1).astype(np.float32)+++def _anchor_for_dir(d: np.ndarray):+ """Pick matching (ha, va) text anchors so text grows *away* from the joint."""+ ha = 'left' if d[0] > 0.15 else 'right' if d[0] < -0.15 else 'center'+ va = 'bottom' if d[1] > 0.15 else 'top' if d[1] < -0.15 else 'center'+ return ha, va+++def _sample_bone_obstacles(pts: np.ndarray, parents: Sequence[int],+ samples_per_bone: int = 3) -> np.ndarray:+ """Sample interior points along every bone segment; used as label obstacles."""+ pts = np.asarray(pts, dtype=np.float32)+ samples: List[np.ndarray] = []+ ts = np.linspace(0.0, 1.0, samples_per_bone + 2)[1:-1] # interior only+ for c, p in enumerate(parents):+ if p == -1:+ continue+ a, b = pts[c], pts[p]+ for t in ts:+ samples.append(a * (1.0 - t) + b * t)+ if not samples:+ return np.zeros((0, 3), dtype=np.float32)+ return np.stack(samples, axis=0)+++def _annotate_joints(ax, pts: np.ndarray, joint_names: Sequence[str],+ bounds, font_size: int,+ parents: Optional[Sequence[int]] = None) -> None:+ """Place each joint's name next to its joint without overlapping bones/labels.++ For each joint, scores 24 ร 4 candidate offset directions (Fibonacci sphere+ ร multiple radii) in screen space by distance from other joints, bone+ samples, and already-placed labels. The chosen anchor gets a colored+ leader line back to the joint so the association stays visible even when+ labels are pushed far out into empty pockets.+ """+ x_min, x_max, y_min, y_max, z_min, z_max = bounds+ scale = max(x_max - x_min, y_max - y_min, z_max - z_min)+ base = 0.08 * scale # leader lines let us push labels well clear of bones++ unit_dirs_single = _fibonacci_sphere_dirs(36)+ radii = np.array([1.0, 1.5, 2.2, 3.0], dtype=np.float32)+ unit_dirs = np.repeat(unit_dirs_single, len(radii), axis=0)+ radius_scales = np.tile(radii, len(unit_dirs_single)).astype(np.float32)++ # Force a draw so box_aspect/view_init propagate into get_proj().+ ax.figure.canvas.draw()+ proj_matrix = ax.get_proj()++ def project2d(xyz: np.ndarray) -> np.ndarray:+ xs, ys, _ = proj3d.proj_transform(+ xyz[..., 0], xyz[..., 1], xyz[..., 2], proj_matrix,+ )+ return np.stack([xs, ys], axis=-1)++ J = len(pts)+ parents_arr = (np.asarray(parents) if parents is not None+ else np.full(J, -1, dtype=int))+ joint_colors = [+ ('red' if parents_arr[i] == -1 else 'steelblue') for i in range(J)+ ]++ joint_obstacles = pts+ bone_obstacles = (_sample_bone_obstacles(pts, parents)+ if parents is not None+ else np.zeros((0, 3), dtype=np.float32))+ placed_2d: List[np.ndarray] = []++ joint_obstacles_2d = project2d(joint_obstacles)+ bone_obstacles_2d = (project2d(bone_obstacles)+ if len(bone_obstacles) > 0+ else np.zeros((0, 2), dtype=np.float32))++ for i, (p, name) in enumerate(zip(pts, joint_names)):+ offsets = unit_dirs * (radius_scales[:, None] * base)+ candidates = p[None, :] + offsets # (N, 3)+ candidates_2d = project2d(candidates) # (N, 2)++ # Avoidance set in 2D: all joints (except self) + bone samples + placed labels.+ mask = np.ones(len(joint_obstacles_2d), dtype=bool)+ mask[i] = False+ avoid_parts = [joint_obstacles_2d[mask]]+ if len(bone_obstacles_2d) > 0:+ avoid_parts.append(bone_obstacles_2d)+ if placed_2d:+ avoid_parts.append(np.stack(placed_2d, axis=0))+ avoid_2d = np.concatenate(avoid_parts, axis=0)++ diffs = candidates_2d[:, None, :] - avoid_2d[None, :, :]+ min_dists = np.linalg.norm(diffs, axis=-1).min(axis=1)++ # Small penalty on longer radii โ prefer shorter leaders when free.+ screen_span = float(+ max(np.ptp(candidates_2d[:, 0]), np.ptp(candidates_2d[:, 1]), 1e-6)+ )+ score = min_dists - 0.03 * radius_scales * screen_span+ slot = int(np.argmax(score))++ lp = candidates[slot]+ ha, va = _anchor_for_dir(unit_dirs[slot])+ color = joint_colors[i]++ leader, = ax.plot([p[0], lp[0]], [p[1], lp[1]], [p[2], lp[2]],+ color=color, linewidth=0.9, linestyle='-',+ alpha=0.9, zorder=9)+ leader.set_clip_on(False)+ txt = ax.text(lp[0], lp[1], lp[2],+ f"[{i}] {name}", fontsize=font_size, color='black',+ ha=ha, va=va, zorder=10,+ bbox=dict(boxstyle='round,pad=0.2', facecolor='white',+ edgecolor=color, linewidth=0.8, alpha=0.85))+ txt.set_clip_on(False)+ placed_2d.append(candidates_2d[slot])+++def _spectral_to_rgb(spectral_feats: np.ndarray) -> np.ndarray:+ """Map (J, K) spectral features to (J, 3) RGB via PCA + per-channel min-max.++ Topologically close joints get similar colors; distant joints contrast.+ """+ J, K = spectral_feats.shape+ if K >= 3:+ centered = spectral_feats - spectral_feats.mean(axis=0, keepdims=True)+ _, _, Vt = np.linalg.svd(centered, full_matrices=False)+ rgb = centered @ Vt[:3].T+ else:+ rgb = np.zeros((J, 3), dtype=np.float32)+ rgb[:, :K] = spectral_feats++ for c in range(3):+ lo, hi = rgb[:, c].min(), rgb[:, c].max()+ if hi - lo > 1e-8:+ rgb[:, c] = (rgb[:, c] - lo) / (hi - lo)+ else:+ rgb[:, c] = 0.5+ return rgb.astype(np.float32)+++# ============================================================================+# Renderers: in-memory RGB arrays+# ============================================================================++def render_skeleton_tpose(parents: Sequence[int], positions,+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> np.ndarray:+ """Render a single skeleton pose as an (H, W, 3) uint8 image."""+ pts = _as_xyz(positions).reshape(-1, 3)+ pts = _maybe_rotate_y_up(pts, rotate_root)+ edges = _edges_from_parents(parents)+ bounds = _cubic_bounds(pts)++ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)+ _draw_static_skeleton(ax, pts, parents, edges)+ if title:+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.98)+ fig.subplots_adjust(top=0.90)+ return _capture_rgb(canvas)+++def render_skeleton_tpose_annotated(parents: Sequence[int], positions,+ joint_names: Sequence[str],+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ font_size: int = 6,+ joint_marker_size: float = 80.0,+ bone_linewidth: float = 2.0,+ figsize: Tuple[float, float] = (12, 12),+ dpi: int = 120) -> np.ndarray:+ """Render a T-pose with each joint labeled by name for manual inspection.++ Uses tight per-axis bounds + matching box_aspect so long-thin subjects+ (snakes, centipedes) fill the figure without distortion, leaving more+ room for labels.+ """+ pts = _as_xyz(positions).reshape(-1, 3)+ pts = _maybe_rotate_y_up(pts, rotate_root)+ assert len(joint_names) == pts.shape[0], (+ f"joint_names has {len(joint_names)} entries but positions has "+ f"{pts.shape[0]} joints"+ )+ edges = _edges_from_parents(parents)+ # Labels are placed at offsets up to ~0.24*scale from each joint (see+ # ``_annotate_joints``: base=0.08*scale ร radii up to 3.0). Add 0.32*scale+ # of absolute padding so the label bboxes and leader endpoints stay inside+ # the plotted volume even on thin/long subjects (snakes, tentacles).+ bounds = _tight_bounds(pts, abs_pad_ratio=0.32)+ box_aspect = _box_aspect_from_bounds(bounds)++ _, canvas, ax = _new_figure(+ bounds, elev, azim,+ figsize=figsize, dpi=dpi, box_aspect=box_aspect,+ )++ # Drop the dashed grid + axis panes so labels stand out against a clean+ # background. Scoped to this renderer only โ other plots keep their grids.+ ax.grid(False)+ for pane in (ax.xaxis.pane, ax.yaxis.pane, ax.zaxis.pane):+ pane.set_visible(False)+ ax.set_axis_off()++ segs = np.zeros((len(edges), 2, 3), dtype=np.float32)+ segs[:, 0] = pts[edges[:, 0]]+ segs[:, 1] = pts[edges[:, 1]]+ ax.add_collection3d(Line3DCollection(+ segs, colors='dimgray', linewidths=bone_linewidth, zorder=5,+ ))++ # Root drawn separately (red, larger) so skeleton origin is obvious.+ parents_arr = np.asarray(parents)+ non_root = parents_arr != -1+ root = ~non_root+ ax.scatter(pts[non_root, 0], pts[non_root, 1], pts[non_root, 2],+ s=joint_marker_size, c='steelblue',+ edgecolors='black', linewidths=0.8, depthshade=False, zorder=8)+ ax.scatter(pts[root, 0], pts[root, 1], pts[root, 2],+ s=joint_marker_size * 1.8, c='red',+ edgecolors='black', linewidths=1.2, depthshade=False, zorder=9)++ _annotate_joints(ax, pts, joint_names, bounds, font_size, parents=parents)+ return _capture_rgb(canvas)+++def render_skeleton_tpos_spectral(parents: Sequence[int], positions,+ spectral_feats,+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (8, 8),+ dpi: int = 120) -> np.ndarray:+ """Render T-pose with joints PCA-colored from (J, K) spectral features."""+ pts = _as_xyz(positions).reshape(-1, 3)+ pts = _maybe_rotate_y_up(pts, rotate_root)+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)+ edges = _edges_from_parents(parents)+ bounds = _cubic_bounds(pts)++ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)+ colors = _spectral_to_rgb(spectral_feats)++ for child, parent in edges:+ ax.plot([pts[child, 0], pts[parent, 0]],+ [pts[child, 1], pts[parent, 1]],+ [pts[child, 2], pts[parent, 2]],+ color='black', linewidth=2.0, alpha=0.8, zorder=1)++ ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2],+ c=colors, s=80, edgecolors='k', linewidths=0.5,+ zorder=2, depthshade=False)++ if title:+ fig.suptitle(title, fontsize=12, fontweight='bold', y=0.95)+ return _capture_rgb(canvas)+++def render_skeleton_spectral_grid(parents: Sequence[int], positions,+ spectral_feats,+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Optional[Tuple[float, float]] = None,+ cell_size: float = 5.0,+ dpi: int = 120) -> np.ndarray:+ """Grid of skeleton subplots, one per eigenvector frequency (coolwarm).++ ``figsize`` overrides the full figure size; when ``None``, the figure is+ sized as ``(cell_size * ncols, cell_size * nrows)``.+ """+ pts = _as_xyz(positions).reshape(-1, 3)+ pts = _maybe_rotate_y_up(pts, rotate_root)+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)+ K = spectral_feats.shape[1]+ edges = _edges_from_parents(parents)+ bounds = _tight_bounds(pts, pad_ratio=0.10)+ x_min, x_max, y_min, y_max, z_min, z_max = bounds+ box_aspect = _box_aspect_from_bounds(bounds)++ ncols = min(K, 4)+ nrows = (K + ncols - 1) // ncols++ if figsize is None:+ figsize = (cell_size * ncols, cell_size * nrows)+ fig = Figure(figsize=figsize, dpi=dpi)+ canvas = FigureCanvasAgg(fig)++ top_margin = 0.04 if title else 0.01+ usable_h = 1.0 - top_margin++ for k in range(K):+ row, col = divmod(k, ncols)+ cell_w = 1.0 / ncols+ cell_h = usable_h / nrows+ x0 = col * cell_w+ y0 = 1.0 - top_margin - (row + 1) * cell_h+ ax = fig.add_axes([x0, y0, cell_w, cell_h], projection='3d',+ computed_zorder=False)+ ax.set_xlim([x_min, x_max])+ ax.set_ylim([y_min, y_max])+ ax.set_zlim([z_min, z_max])+ ax.set_box_aspect(box_aspect)+ ax.set_xticklabels([]); ax.set_yticklabels([]); ax.set_zticklabels([])+ ax.xaxis.set_major_locator(LinearLocator(6))+ ax.yaxis.set_major_locator(LinearLocator(6))+ ax.zaxis.set_major_locator(LinearLocator(6))+ ax.tick_params(axis='both', length=0)+ ax.grid(True, color='gray', linestyle='--', linewidth=0.4)+ if elev is not None and azim is not None:+ ax.view_init(elev=elev, azim=azim)++ for child, parent in edges:+ ax.plot([pts[child, 0], pts[parent, 0]],+ [pts[child, 1], pts[parent, 1]],+ [pts[child, 2], pts[parent, 2]],+ color='black', linewidth=1.5, alpha=0.9, zorder=1)++ vals = spectral_feats[:, k]+ vmax = max(np.abs(vals).max(), 1e-8)+ sc = ax.scatter(pts[:, 0], pts[:, 1], pts[:, 2],+ c=vals, cmap='coolwarm', vmin=-vmax, vmax=vmax,+ s=50, edgecolors='k', linewidths=0.4,+ zorder=2, depthshade=False)+ fig.colorbar(sc, ax=ax, shrink=0.5, pad=0.0, aspect=15)+ ax.set_title(f'Frequency {k + 1}', fontsize=11, fontweight='bold', pad=0)++ if title:+ fig.suptitle(title, fontsize=14, fontweight='bold',+ y=1.0 - top_margin * 0.3)+ return _capture_rgb(canvas)+++def render_skeleton_motion(parents: Sequence[int], positions,+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> np.ndarray:+ """Render a motion sequence as (T, H, W, 3). Reuses artists across frames."""+ pts = _as_xyz(positions)+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"+ pts = _maybe_rotate_y_up(pts, rotate_root)+ T = pts.shape[0]++ edges = _edges_from_parents(parents)+ bounds = _cubic_bounds(pts)+ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)+ if title:+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.98)+ fig.subplots_adjust(top=0.90)++ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')+ segs, lc, sc = _init_motion_artists(ax, edges, pts[0], joint_colors,+ joint_size=10.0)+ sc_root = ax.scatter(pts[0, 0:1, 0], pts[0, 0:1, 1], pts[0, 0:1, 2],+ s=15, c='r')++ frames: List[np.ndarray] = []+ for t in range(T):+ P = pts[t]+ segs[:, 0] = P[edges[:, 0]]+ segs[:, 1] = P[edges[:, 1]]+ lc.set_segments(segs)+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])+ sc_root._offsets3d = (P[0:1, 0], P[0:1, 1], P[0:1, 2])+ frames.append(_capture_rgb(canvas))++ return np.stack(frames, axis=0)+++def render_skeleton_motion_spectral(parents: Sequence[int], positions,+ spectral_feats,+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ joint_marker_size: float = 30.0,+ bone_linewidth: float = 1.5,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> np.ndarray:+ """Motion sequence with joints PCA-colored from spectral features (fixed across frames)."""+ pts = _as_xyz(positions)+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"+ pts = _maybe_rotate_y_up(pts, rotate_root)+ T, J, _ = pts.shape+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)+ assert spectral_feats.shape[0] == J, (+ f"spectral_feats has {spectral_feats.shape[0]} joints but positions "+ f"has {J} joints"+ )++ edges = _edges_from_parents(parents)+ bounds = _cubic_bounds(pts)+ fig, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)+ if title:+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.98)+ fig.subplots_adjust(top=0.90)++ colors = _spectral_to_rgb(spectral_feats)+ segs, lc, sc = _init_motion_artists(+ ax, edges, pts[0], colors,+ joint_size=joint_marker_size, bone_linewidth=bone_linewidth,+ joint_edgecolors='k', joint_edgewidth=0.4,+ )++ frames: List[np.ndarray] = []+ for t in range(T):+ P = pts[t]+ segs[:, 0] = P[edges[:, 0]]+ segs[:, 1] = P[edges[:, 1]]+ lc.set_segments(segs)+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])+ frames.append(_capture_rgb(canvas))++ return np.stack(frames, axis=0)+++def render_skeleton_motion_directed(parents: Sequence[int], positions,+ directions,+ elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ arrow_scale: float = 0.25,+ arrow_color: str = 'green',+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> np.ndarray:+ """Motion sequence with a per-frame orientation arrow anchored at the root.++ ``directions`` is (3,) (constant) or (T, 3). Zero-length vectors render+ no arrow for that frame. ``arrow_scale`` is a fraction of the scene's+ cubic-bound side length.+ """+ pts = _as_xyz(positions)+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"+ pts = _maybe_rotate_y_up(pts, rotate_root)+ T = pts.shape[0]++ dirs = np.asarray(directions, dtype=np.float32)+ if dirs.ndim == 1:+ dirs = np.broadcast_to(dirs, (T, 3)).copy()+ assert dirs.shape == (T, 3), (+ f"directions must be (3,) or (T, 3) with T={T}, got {dirs.shape}"+ )+ dirs = _maybe_rotate_y_up(dirs, rotate_root)+ norms = np.linalg.norm(dirs, axis=-1, keepdims=True)+ unit_dirs = np.where(norms > 1e-8, dirs / np.maximum(norms, 1e-8), 0.0)++ edges = _edges_from_parents(parents)+ bounds = _cubic_bounds(pts)+ x_min, x_max, y_min, y_max, z_min, z_max = bounds+ side = max(x_max - x_min, y_max - y_min, z_max - z_min)+ arrow_len = side * arrow_scale++ _, canvas, ax = _new_figure(bounds, elev, azim, figsize=figsize, dpi=dpi)++ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')+ segs, lc, sc = _init_motion_artists(ax, edges, pts[0], joint_colors,+ joint_size=10.0)+ sc_root = ax.scatter(pts[0, 0:1, 0], pts[0, 0:1, 1], pts[0, 0:1, 2],+ s=15, c='r')++ # 3D quiver can't be mutated in place; remove+re-add each frame.+ quiver = None+ frames: List[np.ndarray] = []+ for t in range(T):+ P = pts[t]+ segs[:, 0] = P[edges[:, 0]]+ segs[:, 1] = P[edges[:, 1]]+ lc.set_segments(segs)+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])+ sc_root._offsets3d = (P[0:1, 0], P[0:1, 1], P[0:1, 2])++ if quiver is not None:+ quiver.remove()+ quiver = None+ d = unit_dirs[t]+ if np.any(d != 0.0):+ root = P[0]+ quiver = ax.quiver(+ root[0], root[1], root[2],+ d[0], d[1], d[2],+ length=arrow_len, normalize=False,+ color=arrow_color, linewidth=2.0,+ arrow_length_ratio=0.25, zorder=11,+ )+ frames.append(_capture_rgb(canvas))++ return np.stack(frames, axis=0)+++_CHECKER_COLORS = ((0.93, 0.93, 0.93), (0.885, 0.885, 0.885))+++def _checker_platform_local(platform_half: float, n_tiles: int, z: float):+ """A RIGID checker platform in platform-local coordinates.++ Returns ``(quads, colors)`` for an ``n_tiles`` ร ``n_tiles`` board+ centered on the origin. The pattern is glued to the platform: the whole+ board is translated rigidly per frame, so nothing about the ground ever+ changes on screen (world-anchored tiles clipped to a sliding window+ made the edge tiles boil). Keeping the board small and near the camera+ also avoids matplotlib's broken projection of far/behind-camera quads.+ """+ tile = 2.0 * platform_half / n_tiles+ quads, colors = [], []+ for i in range(n_tiles):+ xa = -platform_half + i * tile+ for j in range(n_tiles):+ ya = -platform_half + j * tile+ quads.append([(xa, ya, z), (xa + tile, ya, z),+ (xa + tile, ya + tile, z), (xa, ya + tile, z)])+ colors.append(_CHECKER_COLORS[(i + j) % 2])+ return np.asarray(quads, dtype=np.float32), colors+++def _checker_tiles_window(cx: float, cy: float, view_half: float,+ tile: float, z: float, margin_tiles: int = 2):+ """WHOLE world-anchored tiles covering the view window plus a margin.++ Every tile is drawn complete (never clipped), and the drawn set only+ changes by whole tiles entering/leaving beyond the visible edge โ so a+ sliding camera sees a rigid floor scrolling underfoot with no edge+ boiling. The margin keeps the set's boundary off screen while staying+ small enough to avoid matplotlib's broken far/behind-camera projection.+ """+ i0 = int(np.floor((cx - view_half) / tile)) - margin_tiles+ i1 = int(np.floor((cx + view_half) / tile)) + margin_tiles+ j0 = int(np.floor((cy - view_half) / tile)) - margin_tiles+ j1 = int(np.floor((cy + view_half) / tile)) + margin_tiles+ quads, colors = [], []+ for i in range(i0, i1 + 1):+ xa = i * tile+ for j in range(j0, j1 + 1):+ ya = j * tile+ quads.append([(xa, ya, z), (xa + tile, ya, z),+ (xa + tile, ya + tile, z), (xa, ya + tile, z)])+ colors.append(_CHECKER_COLORS[(i + j) % 2])+ return quads, colors+++def _smooth_path(path: np.ndarray, window: int) -> np.ndarray:+ """Moving-average smoothing with edge padding; kills gait sway so the+ follow camera and platform glide instead of bobbing with each step."""+ if window <= 1 or len(path) < 3:+ return path+ kernel = np.ones(window, dtype=np.float32) / window+ pad = window // 2+ padded = np.pad(path, ((pad, window - 1 - pad), (0, 0)), mode='edge')+ return np.stack([np.convolve(padded[:, d], kernel, mode='valid')+ for d in range(path.shape[1])], axis=1)+++def render_skeleton_motion_ground(parents: Sequence[int], positions,+ spectral_feats=None,+ elev: float = 20,+ azim: float = -60,+ rotate_root: bool = True,+ follow: bool = True,+ title: Optional[str] = None,+ joint_marker_size: float = 14.0,+ bone_linewidth: float = 1.6,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> np.ndarray:+ """Motion render with a checkerboard ground plane and root trajectory;+ the camera follows the subject.++ The ground sits at the motion's lowest point, the view window is sized to+ the subject (not the whole trajectory, so a long walk doesn't shrink the+ character to a speck), and each frame re-centers the window on the root โ+ the checker tiles scrolling past convey world-space travel. Joints use+ the spectral PCA palette when ``spectral_feats`` is given, else the+ default root-red / joint-blue scheme.+ """+ pts = _as_xyz(positions)+ assert pts.ndim == 3, f"motion positions must be (T, J, D), got {pts.shape}"+ pts = _maybe_rotate_y_up(pts, rotate_root)+ T, J, _ = pts.shape++ edges = _edges_from_parents(parents)+ ground_z = float(pts[..., 2].min())+ root_xy = pts[:, 0, :2] # (T, 2)++ # Window half-extent from the subject's own size, not the trajectory.+ radius_xy = float(np.abs(pts[..., :2] - root_xy[:, None, :]).max())+ height = float(pts[..., 2].max() - ground_z)+ half = max(radius_xy * 1.08, height * 0.52, 1e-3)++ # Coordinated motion: the camera GLIDES after the subject (smoothed root+ # path) while the ground stays fixed in the WORLD โ subject, ground and+ # camera all move together, and world travel reads off the stage sliding+ # through the frame plus the trail. The ground itself has two forms:+ # stage โ modest travel: one rigid bounded platform covering the+ # whole trajectory, built once, world-fixed.+ # scroll โ extreme travel: world-anchored whole-tile floor windowed+ # around the camera (a trajectory-sized stage would need far+ # too many quads and far-away quads project badly).+ traj_center = 0.5 * (root_xy.max(axis=0) + root_xy.min(axis=0))+ traj_half = float((root_xy.max(axis=0) - root_xy.min(axis=0)).max()) / 2.0+ stage_half = traj_half + half * 1.05+ scroll_mode = stage_half > 4.0 * half+ view_half = half if follow else max(stage_half, half)+ centers = _smooth_path(root_xy, window=max(5, T // 12)) if follow \+ else np.repeat(traj_center[None, :], T, axis=0)+ center0 = centers[0]++ # Vertical extent hugs the subject (tall bipeds and flat crawlers both+ # fill the frame); the box aspect mirrors the ranges to avoid distortion.+ z_top = ground_z + max(height * 1.18, view_half * 0.3)+ bounds = (center0[0] - view_half, center0[0] + view_half,+ center0[1] - view_half, center0[1] + view_half,+ ground_z, z_top)+ fig, canvas, ax = _new_figure(+ bounds, elev, azim, figsize=figsize, dpi=dpi,+ box_aspect=(1.0, 1.0, (z_top - ground_z) / (2 * view_half)))+ ax.set_axis_off()+ # Fill the canvas: with the axes cage hidden there is nothing to clip,+ # so spill the 3D box past the figure edges.+ fig.subplots_adjust(left=-0.22, right=1.22, top=1.18, bottom=-0.18)+ if title:+ fig.suptitle(title, fontsize=12, fontweight='bold', wrap=True, y=0.97)+ fig.subplots_adjust(top=1.02)++ from mpl_toolkits.mplot3d.art3d import Poly3DCollection+ # Tile size from BOTH body dimensions โ the geometric mean of height+ # and horizontal span: an upright biped and a sprawling low body each+ # get tiles proportioned to their overall bulk, and neither dimension+ # alone can blow the scale up. View-based clamps keep it sane on+ # degenerate shapes.+ body_span = 2.0 * radius_xy+ tile = float(np.clip(0.35 * np.sqrt(max(height * body_span, 1e-12)),+ half / 6.0, half / 2.0))+ if scroll_mode:+ # World-anchored whole tiles, refreshed per frame around the camera.+ platform_half = 10 * view_half # trail clip bound (window handles it)+ quads, cols = _checker_tiles_window(*center0, view_half, tile, ground_z)+ platform = Poly3DCollection(quads, facecolors=cols,+ edgecolors='none', zorder=0)+ ax.add_collection3d(platform)+ else:+ # One rigid bounded stage, world-fixed, tiles glued to it. The+ # following camera slides over it; the stage edges sweeping through+ # the frame are what makes the travel visible.+ platform_half = stage_half+ n_tiles = int(np.clip(round(2 * platform_half / tile), 4, 16))+ local_quads, quad_colors = _checker_platform_local(+ platform_half, n_tiles=n_tiles, z=ground_z)+ platform = Poly3DCollection(local_quads + np.array(+ [traj_center[0], traj_center[1], 0.0], dtype=np.float32),+ facecolors=quad_colors, edgecolors='none', zorder=0)+ ax.add_collection3d(platform)++ if spectral_feats is not None:+ spectral_feats = np.asarray(spectral_feats, dtype=np.float32)+ assert spectral_feats.shape[0] == J+ joint_colors = _spectral_to_rgb(spectral_feats)+ edge_kwargs = dict(joint_edgecolors='k', joint_edgewidth=0.4)+ else:+ joint_colors = np.where(np.asarray(parents) == -1, 'r', 'b')+ edge_kwargs = {}++ # The trajectory sits just above the tiles to avoid z-fighting.+ eps = half * 4e-3+ trail, = ax.plot(root_xy[:1, 0], root_xy[:1, 1], [ground_z + eps],+ color='tab:orange', linewidth=1.6, alpha=0.9, zorder=1)++ segs, lc, sc = _init_motion_artists(+ ax, edges, pts[0], joint_colors,+ joint_size=joint_marker_size, bone_color='dimgray',+ bone_linewidth=bone_linewidth, **edge_kwargs)+ # The root must never be occluded: inside a single scatter, points paint+ # in array order, so joints drawn after index 0 cover it. Re-draw the+ # root as its own topmost scatter (same size; red in the default+ # palette, its own spectral color otherwise).+ root_color = ([joint_colors[0]] if spectral_feats is not None else 'r')+ sc_root = ax.scatter(pts[0, 0:1, 0], pts[0, 0:1, 1], pts[0, 0:1, 2],+ s=joint_marker_size, c=root_color,+ depthshade=False, zorder=10, **(+ dict(edgecolors='k', linewidths=0.4)+ if spectral_feats is not None else {}))++ frames: List[np.ndarray] = []+ for t in range(T):+ P = pts[t]+ segs[:, 0] = P[edges[:, 0]]+ segs[:, 1] = P[edges[:, 1]]+ lc.set_segments(segs)+ sc._offsets3d = (P[:, 0], P[:, 1], P[:, 2])+ sc_root._offsets3d = (P[0:1, 0], P[0:1, 1], P[0:1, 2])+ cx, cy = centers[t]+ tx = root_xy[:t + 1, 0].astype(np.float32).copy()+ ty = root_xy[:t + 1, 1].astype(np.float32).copy()+ off = (np.abs(tx - traj_center[0]) > platform_half) | \+ (np.abs(ty - traj_center[1]) > platform_half)+ tx[off] = np.nan+ trail.set_data_3d(tx, ty, np.full(t + 1, ground_z + eps))+ if scroll_mode:+ quads, cols = _checker_tiles_window(cx, cy, view_half, tile,+ ground_z)+ platform.set_verts(quads)+ platform.set_facecolor(cols)+ if follow:+ ax.set_xlim(cx - view_half, cx + view_half)+ ax.set_ylim(cy - view_half, cy + view_half)+ frames.append(_capture_rgb(canvas))++ return np.stack(frames, axis=0)+++# ============================================================================+# Savers: write PNG / MP4 to disk+# ============================================================================++def _write_image(save_path: str, frame: np.ndarray) -> None:+ Path(save_path).parent.mkdir(parents=True, exist_ok=True)+ imageio.imwrite(save_path, frame)+++def _write_video(save_path: str, frames: np.ndarray, fps: int) -> None:+ Path(save_path).parent.mkdir(parents=True, exist_ok=True)+ imageio.mimwrite(save_path, frames, fps=fps)+++# Defaults (elev=30, azim=-60, rotate_root=True) match the pipeline convention.++def save_skeleton_tpose(save_path: str, parents: Sequence[int], positions,+ elev: float = 30, azim: float = -60,+ rotate_root: bool = True,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> None:+ """Render a T-pose and save as PNG."""+ _write_image(save_path, render_skeleton_tpose(+ parents, positions, elev=elev, azim=azim, rotate_root=rotate_root,+ figsize=figsize, dpi=dpi,+ ))+++def save_skeleton_tpose_annotated(save_path: str, parents: Sequence[int],+ positions, joint_names: Sequence[str],+ elev: float = 30, azim: float = -60,+ rotate_root: bool = True,+ font_size: int = 6,+ joint_marker_size: float = 80.0,+ bone_linewidth: float = 2.0,+ figsize: Tuple[float, float] = (12, 12),+ dpi: int = 120) -> None:+ """Render an annotated T-pose and save as PNG."""+ _write_image(save_path, render_skeleton_tpose_annotated(+ parents, positions, joint_names,+ elev=elev, azim=azim, rotate_root=rotate_root, font_size=font_size,+ joint_marker_size=joint_marker_size, bone_linewidth=bone_linewidth,+ figsize=figsize, dpi=dpi,+ ))+++def save_skeleton_tpos_spectral(save_path: str, parents: Sequence[int], positions,+ spectral_feats,+ elev: float = 30, azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (8, 8),+ dpi: int = 120) -> None:+ """Render spectral-colored T-pose and save as PNG."""+ _write_image(save_path, render_skeleton_tpos_spectral(+ parents, positions, spectral_feats,+ elev=elev, azim=azim, rotate_root=rotate_root, title=title,+ figsize=figsize, dpi=dpi,+ ))+++def save_skeleton_spectral_grid(save_path: str, parents: Sequence[int],+ positions, spectral_feats,+ elev: float = 30, azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Optional[Tuple[float, float]] = None,+ cell_size: float = 5.0,+ dpi: int = 120) -> None:+ """Render per-frequency spectral grid and save as PNG."""+ _write_image(save_path, render_skeleton_spectral_grid(+ parents, positions, spectral_feats,+ elev=elev, azim=azim, rotate_root=rotate_root, title=title,+ figsize=figsize, cell_size=cell_size, dpi=dpi,+ ))+++def save_skeleton_motion(save_path: str, parents: Sequence[int], positions,+ fps: int = 30, elev: float = 30, azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> None:+ """Render a motion sequence and save as MP4."""+ frames = render_skeleton_motion(+ parents, positions, elev=elev, azim=azim, rotate_root=rotate_root,+ title=title, figsize=figsize, dpi=dpi,+ )+ _write_video(save_path, frames, fps)+++def save_skeleton_motion_spectral(save_path: str, parents: Sequence[int],+ positions, spectral_feats,+ fps: int = 30, elev: float = 30,+ azim: float = -60,+ rotate_root: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> None:+ """Render a spectral-colored motion sequence and save as MP4."""+ frames = render_skeleton_motion_spectral(+ parents, positions, spectral_feats,+ elev=elev, azim=azim, rotate_root=rotate_root, title=title,+ figsize=figsize, dpi=dpi,+ )+ _write_video(save_path, frames, fps)+++def save_skeleton_motion_ground(save_path: str, parents: Sequence[int],+ positions, spectral_feats=None,+ fps: int = 20, elev: float = 20,+ azim: float = -60, rotate_root: bool = True,+ follow: bool = True,+ title: Optional[str] = None,+ figsize: Tuple[float, float] = (6, 6),+ dpi: int = 120) -> None:+ """Render and save a ground-plane motion MP4 (see+ :func:`render_skeleton_motion_ground`)."""+ frames = render_skeleton_motion_ground(+ parents, positions, spectral_feats=spectral_feats, elev=elev,+ azim=azim, rotate_root=rotate_root, follow=follow, title=title,+ figsize=figsize, dpi=dpi)+ _write_video(save_path, frames, fps)diff --git a/data_process/utils/skeleton.py b/data_process/utils/skeleton.pynew file mode 100644index 0000000..ac1ca5a--- /dev/null+++ b/data_process/utils/skeleton.py@@ -0,0 +1,533 @@+"""Skeleton processing utilities.++Topology-agnostic helpers that operate on ``Animation`` objects: BFS joint+reordering, geometric measurements, uniform scaling, root alignment,+facing-axis alignment and motion activity metrics.++Naming conventions:+ get_* โ pure computations that return a value (no mutation).+ scale_* โ uniform scaling transforms returning a new Animation.+ align_* โ positional / rotational alignment transforms returning a+ new Animation (together with the offset/quaternion applied).+"""++import statistics+from collections import deque++import numpy as np+from Quaternions import Quaternions+from Animation import Animation, positions_global, offset_lengths+++# ---------------------------------------------------------------------------+# BFS joint reordering+# ---------------------------------------------------------------------------++def bfs_reorder_joints(parents, offsets=None):+ """Compute a canonical breadth-first ordering of joints starting from the root.++ Children sorted by subtree size (descending), bone length as tiebreaker (ascending).+ """+ njoints = len(parents)++ if offsets is not None:+ bone_lengths = np.linalg.norm(offsets, axis=-1)+ else:+ bone_lengths = None++ # Build children adjacency and find root.+ children = [[] for _ in range(njoints)]+ root = None+ for j, p in enumerate(parents):+ if p == -1:+ root = j+ else:+ children[p].append(j)+ assert root is not None, "No root joint found (parent == -1)"++ # Compute subtree sizes via BFS topological order (bottom-up accumulation).+ subtree_size = [1] * njoints+ topo = []+ q = deque([root])+ while q:+ node = q.popleft()+ topo.append(node)+ q.extend(children[node])+ for node in reversed(topo):+ for child in children[node]:+ subtree_size[node] += subtree_size[child]++ # Sort children: largest subtree first, shortest bone as tiebreaker.+ for node in range(njoints):+ if children[node]:+ children[node].sort(+ key=lambda idx: (-subtree_size[idx],+ bone_lengths[idx] if bone_lengths is not None else 0))++ # Final BFS traversal with sorted children โ canonical ordering.+ bfs_order = []+ q = deque([root])+ while q:+ node = q.popleft()+ bfs_order.append(node)+ q.extend(children[node])++ assert len(bfs_order) == njoints++ # Build oldโnew index mapping and remap parents.+ old_to_new = [0] * njoints+ for new_idx, old_idx in enumerate(bfs_order):+ old_to_new[old_idx] = new_idx++ new_parents = []+ for new_idx, old_idx in enumerate(bfs_order):+ old_parent = parents[old_idx]+ new_parents.append(-1 if old_parent == -1 else old_to_new[old_parent])++ return bfs_order, new_parents+++def reorder_anim_bfs(anim):+ """Reorder an Animation's joints into BFS order."""+ parents_list = anim.parents.tolist() if hasattr(anim.parents, 'tolist') else list(anim.parents)+ offsets = np.array(anim.offsets) if hasattr(anim, 'offsets') else None+ bfs_order, new_parents = bfs_reorder_joints(parents_list, offsets=offsets)++ if bfs_order == list(range(len(parents_list))):+ return anim, bfs_order, parents_list++ reordered = anim[:, bfs_order]+ return reordered, bfs_order, new_parents+++# ---------------------------------------------------------------------------+# Geometry: skeleton diameter+# ---------------------------------------------------------------------------++def get_skeleton_diameter(anim):+ """Return the leaf-to-leaf geodesic diameter of a skeleton.++ Uses the classic two-pass BFS trick on the bone-length-weighted skeleton+ tree:+ 1. BFS from the root to find an extreme leaf ``u``.+ 2. BFS from ``u`` to find the farthest node โ that distance is the+ tree diameter.++ The BFS formulation is robust to arbitrary root placements and+ non-humanoid topologies (multi-limb creatures, snakes, etc.).++ Args:+ anim: Animation object whose ``parents`` / ``offsets`` define the+ rest-pose skeleton.++ Returns:+ Diameter as a float. Returns 0.0 for degenerate single-joint skeletons.+ """+ parents = anim.parents+ offsets = anim.offsets+ n_joints = len(parents)++ # Build a bidirectional, bone-length-weighted adjacency list.+ adj = {i: [] for i in range(n_joints)}+ bone_lengths = np.linalg.norm(offsets, axis=-1)+ for i, p in enumerate(parents):+ if p != -1:+ adj[p].append((i, bone_lengths[i]))+ adj[i].append((p, bone_lengths[i]))++ def _bfs_farthest(start):+ """BFS returning (farthest_node, distance) from ``start``."""+ distances = {start: 0.0}+ visited = {start}+ queue = [start]+ farthest_node = start+ max_dist = 0.0++ ptr = 0+ while ptr < len(queue):+ u = queue[ptr]+ ptr += 1+ for v, weight in adj[u]:+ if v in visited:+ continue+ visited.add(v)+ dist = distances[u] + weight+ distances[v] = dist+ queue.append(v)+ if dist > max_dist:+ max_dist = dist+ farthest_node = v+ return farthest_node, max_dist++ u, _ = _bfs_farthest(0) # pass 1: one extreme leaf+ _, diameter = _bfs_farthest(u) # pass 2: farthest from u โ diameter+ return diameter+++# ---------------------------------------------------------------------------+# Uniform scaling+# ---------------------------------------------------------------------------++def scale_anim(anim, scale_factor):+ """Return a copy of *anim* with positions and offsets scaled uniformly."""+ return Animation(+ rotations=anim.rotations.copy(),+ positions=anim.positions.copy() * scale_factor,+ orients=anim.orients.copy(),+ offsets=anim.offsets.copy() * scale_factor,+ parents=anim.parents.copy(),+ )+++def scale_by_diameter(anim, scale_factor=None, target_diameter=2.0):+ """Scale an animation so that its leaf-to-leaf diameter matches a target.++ Args:+ anim: Animation to scale.+ scale_factor: Explicit scale. If None, computed as+ ``target_diameter / get_skeleton_diameter(anim)``.+ target_diameter: Desired diameter (default 2.0).++ Returns:+ Tuple of (scaled Animation, scale_factor applied).+ """+ if scale_factor is None:+ scale_factor = target_diameter / get_skeleton_diameter(anim)+ return scale_anim(anim, scale_factor), scale_factor+++def scale_by_bone_length(anim, scale_factor=None, target_bone_len=None):+ """Scale an animation so that the mean bone length matches a target.++ If ``scale_factor`` is not provided it is derived as+ ``target_bone_len / mean(offset_lengths(anim))``. Rotations are untouched.++ Args:+ anim: Animation to scale.+ scale_factor: Pre-computed scale. If None, derived from *target_bone_len*.+ target_bone_len: Desired mean bone length (used only when *scale_factor*+ is None).++ Returns:+ Tuple of (scaled Animation, scale_factor applied).+ """+ if scale_factor is None:+ mean_len = statistics.mean(offset_lengths(anim))+ scale_factor = target_bone_len / mean_len+ return scale_anim(anim, scale_factor), scale_factor+++# ---------------------------------------------------------------------------+# Root alignment (translation)+# ---------------------------------------------------------------------------++def _translated_anim(anim, new_positions, new_offsets):+ """Return a copy of *anim* with replacement positions/offsets."""+ return Animation(+ rotations=anim.rotations.copy(),+ positions=new_positions,+ orients=anim.orients.copy(),+ offsets=new_offsets,+ parents=anim.parents.copy(),+ )+++def align_root_xz_to_origin(anim, root_pose_init_xz=None):+ """Translate the skeleton so the root's XZ position is at the origin at frame 0.++ Only the horizontal (XZ) components are subtracted โ the absolute Y height+ is preserved.++ Args:+ anim: Animation to translate.+ root_pose_init_xz: Pre-computed XZ offset (3,) with Y component zero.+ If None, auto-detected from the frame-0 root position.++ Returns:+ Tuple of (translated Animation, root_pose_init_xz actually subtracted).+ """+ if root_pose_init_xz is None:+ # Zero-out the Y component so only XZ is subtracted.+ root_pose_init_xz = anim.positions[0, 0] * np.array([1, 0, 1])++ new_positions = anim.positions.copy()+ new_positions[:, 0] -= root_pose_init_xz+ new_offsets = anim.offsets.copy()+ new_offsets[0] -= root_pose_init_xz++ return _translated_anim(anim, new_positions, new_offsets), root_pose_init_xz+++def align_root_to_ground(anim, ground_height=None):+ """Translate the skeleton vertically so the lowest joint sits on Y=0.++ If ``ground_height`` is not provided, it is the global minimum Y across+ all joints and frames (computed via forward kinematics).++ Args:+ anim: Animation to translate.+ ground_height: Pre-computed Y offset. If None, auto-detected.++ Returns:+ Tuple of (translated Animation, ground_height subtracted).+ """+ if ground_height is None:+ global_positions = positions_global(anim)+ ground_height = global_positions.min(axis=0).min(axis=0)[1]++ new_positions = anim.positions.copy()+ new_positions[:, 0, 1] -= ground_height+ new_offsets = anim.offsets.copy()+ new_offsets[0, 1] -= ground_height++ return _translated_anim(anim, new_positions, new_offsets), ground_height+++# ---------------------------------------------------------------------------+# Root orientation alignment (facing axis)+# ---------------------------------------------------------------------------++def _normalize(v, eps=1e-8):+ """Return ``v`` normalized along its last axis with a numerical floor."""+ norm = np.sqrt((v ** 2).sum(axis=-1, keepdims=True))+ return v / np.maximum(norm, eps)+++def get_root_facing_quat(joints, face_joint_idxs, body_axis=False):+ """Compute per-frame root quaternion that rotates the skeleton to face Z+.++ The left-right axis ``across`` is assembled from the given face joint+ indices:++ - 2 indices ``[r_hip, l_hip]``: ``joints[r_hip] - joints[l_hip]``.+ - 4 indices ``[r_hip, l_hip, r_shoulder, l_shoulder]``: the sum of the+ hip and shoulder left-right vectors.++ Forward direction is ``Y_up x across``, and the returned quaternion+ rotates that forward vector onto Z+. When *body_axis* is True (e.g.+ snakes), ``across`` is treated as a body-axis (forward) vector rather+ than a lateral vector: it is rotated by -90ยฐ Y to convert it into a+ lateral axis before the cross product.++ If ``face_joint_idxs`` is None or the sentinel ``[-1, -1]`` (face joint+ names unavailable for this skeleton), returns per-frame identity+ quaternions so the caller can skip facing alignment.++ Args:+ joints: Global joint positions (F, J, 3).+ face_joint_idxs: Sequence of 2 or 4 joint indices, or ``[-1, -1]``+ sentinel. For body-axis creatures, convention is r_hip=head/tongue,+ l_hip=tail.+ body_axis: If True, treat the joint vector as body-axis (forward)+ rather than lateral.++ Returns:+ Quaternions of shape (F,) aligning the skeleton to face Z+.+ """+ if face_joint_idxs is None or (+ len(face_joint_idxs) >= 2+ and face_joint_idxs[0] == -1+ and face_joint_idxs[1] == -1+ ):+ return Quaternions.id(len(joints))++ if len(face_joint_idxs) == 4:+ r_hip, l_hip, sdr_r, sdr_l = face_joint_idxs+ across = (joints[:, r_hip] - joints[:, l_hip]) + (+ joints[:, sdr_r] - joints[:, sdr_l]+ )+ elif len(face_joint_idxs) == 2:+ r_hip, l_hip = face_joint_idxs+ across = joints[:, r_hip] - joints[:, l_hip]+ else:+ raise ValueError(+ f"face_joint_idxs must have 2 or 4 entries, got {len(face_joint_idxs)}"+ )++ across = _normalize(across)+ forward = _normalize(np.cross(np.array([[0, 1, 0]]), across, axis=-1))++ if body_axis:+ rot_correction = Quaternions.from_euler(np.array([0, -np.pi / 2, 0]), "xyz")+ forward = rot_correction * forward++ target = np.array([[0, 0, 1]]).repeat(len(forward), axis=0)+ root_quat = Quaternions.between(forward, target)++ return root_quat+++def align_initial_facing(anim, face_joint_idxs, body_axis=False):+ """Rotate the entire motion so the skeleton faces Z+ at the first frame.++ Computes the facing-direction quaternion from the first frame using+ ``get_root_facing_quat`` and applies it to the root joint across all+ frames. Non-root joints are unaffected at the rotation level (the+ rotation propagates through the root's orientation at render time).++ When ``face_joint_idxs`` is None or the sentinel ``[-1, -1]`` (face+ joint names unavailable), the animation is returned unchanged+ (identity facing rotation).++ Args:+ anim: Animation object.+ face_joint_idxs: Indices [r_hip, l_hip] or+ [r_hip, l_hip, r_shoulder, l_shoulder], or ``[-1, -1]`` sentinel.+ body_axis: If True, apply an extra -90ยฐ Y correction (e.g. snakes).++ Returns:+ New Animation with rotated root.+ """+ if face_joint_idxs is None or (+ len(face_joint_idxs) >= 2+ and face_joint_idxs[0] == -1+ and face_joint_idxs[1] == -1+ ):+ return Animation(+ rotations=anim.rotations.copy(),+ positions=anim.positions.copy(),+ orients=anim.orients.copy(),+ offsets=anim.offsets.copy(),+ parents=anim.parents.copy(),+ )++ global_pos = positions_global(anim)+ quat = get_root_facing_quat(+ global_pos, face_joint_idxs, body_axis=body_axis,+ )[0]++ nframes = anim.shape[0]+ quat_tiled = quat.repeat(nframes, axis=0)++ new_rots = anim.rotations.copy()+ new_pos = anim.positions.copy()+ new_offsets = anim.offsets.copy()+ new_rots[:, 0] = quat_tiled * new_rots[:, 0]+ new_pos[:, 0] = quat_tiled * new_pos[:, 0]+ new_offsets[0] = quat * new_offsets[0]++ return Animation(+ rotations=new_rots,+ positions=new_pos,+ orients=anim.orients.copy(),+ offsets=new_offsets,+ parents=anim.parents.copy(),+ )+++# ---------------------------------------------------------------------------+# Full alignment pipeline+# ---------------------------------------------------------------------------++def canonicalize_anim(anim,+ face_joint_idxs, body_axis=False,+ root_pose_init_xz=None, ground_height=None,+ scale_method="diameter", scale_factor=None, target_scale_metric=None):+ """Full alignment pipeline: facing -> center XZ -> scale -> ground.++ Each step accepts an optional pre-computed parameter so T-pose calibration+ values can be reused for motion clips.++ Args:+ anim: Raw Animation object.+ face_joint_idxs: Joint indices defining the facing direction.+ body_axis: If True, apply an extra -90ยฐ Y correction (e.g. snakes).+ root_pose_init_xz: XZ offset for centering (from T-pose calibration).+ ground_height: Vertical ground offset (from T-pose calibration).+ scale_method: ``"diameter"`` (default) or ``"bone_length"``.+ scale_factor: Uniform scale (from T-pose calibration).+ target_scale_metric: Target value for the chosen scale method. Interpreted+ as target diameter when ``scale_method="diameter"`` or target mean bone+ length when ``scale_method="bone_length"``. Only used when+ *scale_factor* is None.++ Returns:+ (aligned_anim, root_pose_init_xz, ground_height, scale_factor).+ """+ rotated = align_initial_facing(anim, face_joint_idxs, body_axis=body_axis)+ centered, root_pose_init_xz = align_root_xz_to_origin(rotated, root_pose_init_xz)++ if scale_method == "diameter":+ scaled, scale_factor = scale_by_diameter(+ centered, scale_factor=scale_factor, target_diameter=target_scale_metric or 2.0)+ elif scale_method == "bone_length":+ scaled, scale_factor = scale_by_bone_length(+ centered, scale_factor=scale_factor, target_bone_len=target_scale_metric)+ else:+ raise ValueError(f"Unknown scale_method: {scale_method!r}")++ grounded, ground_height = align_root_to_ground(scaled, ground_height)+ return grounded, root_pose_init_xz, ground_height, scale_factor+++# ---------------------------------------------------------------------------+# Motion activity metrics+# ---------------------------------------------------------------------------+++def get_motion_activity_metrics(positions, top_k_ratio=0.25, top_k_min=2):+ """Compute motion activity metrics from joint positions.++ Activity is measured as the **temporal spread** (std over frames) of each+ joint's position. Non-root joint activity is computed from *root-relative*+ positions, so it isolates articulation from global translation. Root+ activity uses the global root position.++ Assumes positions are already at a canonical scale (e.g. via+ ``scale_by_diameter``), so no bone-length normalization is applied.++ This spread-based formulation is robust to frame rate and to motions that+ have tiny jitter but no real movement: a jittery-but-static motion has+ near-zero std even if its frame-to-frame step is nonzero, so thresholding+ on these metrics reliably filters out inactive clips.++ In addition to the mean over all non-root joints, a top-k mean is returned+ โ the average spread of the most active joints. This catches long-tailed /+ multi-limb skeletons where only a few joints move (e.g. snake tail tip,+ single-limb gestures); the mean gets diluted by the many static joints,+ but the top-k mean stays high. ``k`` scales with skeleton size as+ ``max(top_k_min, round(top_k_ratio * (J - 1)))`` so it is meaningful for+ both small skeletons (~10 joints) and large ones (100+ joints).++ Args:+ positions: Joint global positions (F, J, 3), at canonical scale.+ top_k_ratio: Fraction of non-root joints to include in the top-k mean+ (default 0.25 โ top quartile of most-active joints).+ top_k_min: Minimum number of joints in the top-k window, so tiny+ skeletons still get a meaningful average (default 2).++ Returns:+ Dict with keys ``joint_activity``, ``joint_activity_topk`` and+ ``root_activity`` (floats).+ """+ assert positions.ndim == 3, "positions must be (F, J, 3)"++ # Root activity: temporal spread of the global root trajectory.+ root_pos = positions[:, 0, :] # (F, 3)+ root_activity = float(np.linalg.norm(root_pos.std(axis=0)))++ # Joint activity: temporal spread of *root-relative* joint positions,+ # which isolates articulation from global translation.+ if positions.shape[1] > 1:+ rel_pos = positions[:, 1:, :] - positions[:, 0:1, :] # (F, J-1, 3)+ per_joint_std = rel_pos.std(axis=0) # (J-1, 3)+ per_joint_spread = np.linalg.norm(per_joint_std, axis=-1) # (J-1,)+ joint_activity = float(per_joint_spread.mean())++ # Top-k mean: robust to long-tailed skeletons where the mean is+ # diluted by many static joints but a few limbs carry the motion.+ # k scales with joint count so it stays meaningful across skeleton sizes.+ n_non_root = per_joint_spread.shape[0]+ k = max(top_k_min, int(round(top_k_ratio * n_non_root)))+ k = min(k, n_non_root)+ top_k_vals = np.partition(per_joint_spread, -k)[-k:]+ joint_activity_topk = float(top_k_vals.mean())+ else:+ joint_activity = 0.0+ joint_activity_topk = 0.0++ return {+ "joint_activity": joint_activity,+ "joint_activity_topk": joint_activity_topk,+ "root_activity": root_activity,+ }diff --git a/data_process/utils/topology.py b/data_process/utils/topology.pynew file mode 100644index 0000000..0872218--- /dev/null+++ b/data_process/utils/topology.py@@ -0,0 +1,333 @@+"""Skeleton topology utilities.++All public functions accept a parent index array ``(J,)`` (list or numpy+array) where ``parents[j]`` is the parent of joint *j* and the root has+parent ``-1``.++Sections:+ 1. Graph representations โ adjacency, edge index, joint depths+ 2. Spectral encodings โ Laplacian eigenvectors+ 3. Pairwise relations โ edge types, topology distances+ 4. Kinematic chains+"""++import bisect+from collections import deque++import numpy as np+++# ---------------------------------------------------------------------------+# Internal helpers+# ---------------------------------------------------------------------------++def _parse_parents(parents):+ """Normalize *parents* to a 1-D int64 numpy array."""+ return np.asarray(parents, dtype=np.int64).ravel()+++def _children_map(parents):+ """List-of-lists mapping each joint to its children."""+ n = len(parents)+ children = [[] for _ in range(n)]+ for c, p in enumerate(parents):+ if p is None or p == -1:+ continue+ if not (0 <= p < n):+ raise ValueError(f"Invalid parent index parents[{c}]={p}")+ children[p].append(c)+ return children+++def _adjacency_list(parents):+ """Undirected adjacency list built from *parents*."""+ n = len(parents)+ adj = [[] for _ in range(n)]+ for c, p in enumerate(parents):+ if p is None or p == -1:+ continue+ adj[c].append(p)+ adj[p].append(c)+ return adj+++# ---------------------------------------------------------------------------+# 1. Graph representations+# ---------------------------------------------------------------------------++def compute_adjacency_matrix(parents, self_loops=True):+ """Dense undirected adjacency matrix ``(J, J)`` (float32)."""+ parents = _parse_parents(parents)+ n = len(parents)+ adj = np.eye(n, dtype=np.float32) if self_loops else np.zeros((n, n), dtype=np.float32)+ for child, parent in enumerate(parents):+ if child == parent or parent < 0:+ continue+ adj[child, parent] = 1.0+ adj[parent, child] = 1.0+ return adj+++def compute_edge_indexs(parents):+ """PyG-style edge index ``[2, 2*(J-1)]`` (int64).++ Undirected: includes both parent->child and child->parent edges.+ Self-loops are excluded.+ """+ parents = _parse_parents(parents)+ children = np.arange(len(parents))+ valid = (children != parents) & (parents >= 0)+ p = parents[valid]+ c = children[valid]+ return np.stack([np.concatenate([p, c]), np.concatenate([c, p])], axis=0)+++def compute_joint_depths(parents):+ """Depth of each joint in the kinematic tree ``(J,)`` (int64, root=0).++ Handles arbitrary joint ordering (parent index may be > child index).+ """+ parents = list(parents)+ n = len(parents)+ depths = np.full(n, -1, dtype=np.int64)++ for j in range(n):+ if parents[j] == -1 or parents[j] == j:+ depths[j] = 0++ changed = True+ while changed:+ changed = False+ for j in range(n):+ if depths[j] >= 0:+ continue+ p = parents[j]+ if 0 <= p < n and depths[p] >= 0:+ depths[j] = depths[p] + 1+ changed = True++ return depths+++# ---------------------------------------------------------------------------+# 2. Spectral encodings+# ---------------------------------------------------------------------------++def _build_laplacian(parents, norm='none'):+ """Graph Laplacian ``(J, J)`` of the undirected skeleton graph.++ Args:+ parents: Already-parsed (J,) int64 array.+ norm: ``'none'`` (combinatorial ``L = D - A``), ``'sym'``+ (``I - D^{-1/2} A D^{-1/2}``) or ``'rw'`` (``I - D^{-1} A``).+ """+ adj = compute_adjacency_matrix(parents, self_loops=False).astype(np.float64)+ deg = adj.sum(axis=1)+ if norm == 'none':+ return np.diag(deg) - adj+ with np.errstate(divide='ignore'):+ if norm == 'sym':+ d_inv_sqrt = np.where(deg > 0, deg ** -0.5, 0.0)+ return np.eye(len(deg)) - d_inv_sqrt[:, None] * adj * d_inv_sqrt[None, :]+ if norm == 'rw':+ d_inv = np.where(deg > 0, 1.0 / deg, 0.0)+ return np.eye(len(deg)) - d_inv[:, None] * adj+ raise ValueError(f"Unsupported laplacian norm: '{norm}'. Choose from 'none', 'sym', 'rw'.")+++def _normalize_eigvecs(eigvecs, eigvals, norm='L2', eps=1e-12):+ """Normalize eigenvector columns (``'sign'``, ``'L1'``, ``'L2'``,+ ``'abs-max'`` or ``'wavelength'``)."""+ eigvecs = eigvecs.copy()++ if norm == 'sign':+ for k in range(eigvecs.shape[1]):+ max_idx = np.argmax(np.abs(eigvecs[:, k]))+ if eigvecs[max_idx, k] < 0:+ eigvecs[:, k] *= -1+ return eigvecs++ for k in range(eigvecs.shape[1]):+ col = eigvecs[:, k]+ if norm == 'L1':+ denom = np.abs(col).sum()+ elif norm == 'L2':+ denom = np.sqrt((col ** 2).sum())+ elif norm == 'abs-max':+ denom = np.abs(col).max()+ elif norm == 'wavelength':+ denom = np.abs(col).max()+ lam = np.sqrt(max(eigvals[k], eps))+ denom = denom * lam * 2.0 / np.pi+ else:+ raise ValueError(f"Unsupported eigvec norm: '{norm}'. "+ f"Choose from 'sign', 'L1', 'L2', 'abs-max', 'wavelength'.")+ if denom > eps:+ eigvecs[:, k] = col / denom++ return eigvecs+++def compute_laplacian_eigenvectors(parents, max_freqs=8,+ laplacian_norm='sym', eigvec_norm='L2'):+ """Smallest non-trivial eigenvectors of the graph Laplacian.++ Follows the GRIT / GraphGPS decomposition: build the Laplacian, run an+ eigendecomposition, clamp eigenvalues and normalize eigenvectors.++ Args:+ parents: Parent indices (J,).+ max_freqs: Number of frequencies to return (clamped to J-1).+ laplacian_norm: ``'none'``, ``'sym'`` or ``'rw'``.+ eigvec_norm: ``'sign'``, ``'L1'``, ``'L2'``, ``'abs-max'`` or+ ``'wavelength'``.++ Returns:+ ``(eigvecs (J, max_freqs) float32, eigvals (max_freqs,) float32)``.+ Skeletons with fewer than ``max_freqs + 1`` joints are zero-padded on+ the frequency axis so every skeleton yields the same width.+ """+ parents = _parse_parents(parents)+ n = len(parents)+ k = min(n - 1, max_freqs)++ laplacian = _build_laplacian(parents, norm=laplacian_norm.lower())+ eigvals_all, eigvecs_all = np.linalg.eigh(laplacian)++ # Skip the trivial (constant) eigenvector at eigenvalue ~0.+ eigvals = np.maximum(eigvals_all[1:1 + k], 0.0)+ eigvecs = np.real(eigvecs_all[:, 1:1 + k])+ eigvecs = _normalize_eigvecs(eigvecs, eigvals, norm=eigvec_norm)++ if k < max_freqs:+ eigvecs = np.pad(eigvecs, ((0, 0), (0, max_freqs - k)))+ eigvals = np.pad(eigvals, (0, max_freqs - k))++ return eigvecs.astype(np.float32), eigvals.astype(np.float32)+++# ---------------------------------------------------------------------------+# 3. Pairwise relations and topology distances+# ---------------------------------------------------------------------------++EDGE_TYPES = {+ 'self': 0, 'parent': 1, 'child': 2, 'sibling': 3,+ 'no_relation': 4, 'end_effector': 5, 'ts_token_conn': 6,+}+++def compute_edge_relations_and_distances(parents, max_path_len=5):+ """Pairwise edge relation types and clamped topology distances.++ Edge relation types follow ``EDGE_TYPES``: 0=self, 1=parent, 2=child,+ 3=sibling, 4=no_relation, 5=end_effector, 6=ts_token_conn.++ Returns:+ ``(edge_rel, topo_rel)``, each ``(J, J)`` int16.+ """+ parents = list(parents)+ n = len(parents)+ children = _children_map(parents)++ edge_rel = np.full((n, n), EDGE_TYPES['no_relation'], dtype=np.int16)+ for i in range(n):+ pi = parents[i]+ for j in range(n):+ pj = parents[j]+ if i == j:+ edge_rel[i, j] = EDGE_TYPES['self']+ elif pj == i:+ edge_rel[i, j] = EDGE_TYPES['child']+ elif j == pi and pi != -1:+ edge_rel[i, j] = EDGE_TYPES['parent']+ elif pi != -1 and pj == pi:+ edge_rel[i, j] = EDGE_TYPES['sibling']+ if len(children[i]) == 0:+ edge_rel[i, i] = EDGE_TYPES['end_effector']++ adj = _adjacency_list(parents)+ topo_rel = np.full((n, n), max_path_len, dtype=np.int16)+ for s in range(n):+ dist = np.full(n, 32767, dtype=np.int16)+ dist[s] = 0+ q = deque([s])+ while q:+ u = q.popleft()+ if dist[u] >= max_path_len:+ continue+ for v in adj[u]:+ if dist[v] > dist[u] + 1:+ dist[v] = dist[u] + 1+ q.append(v)+ topo_rel[s] = np.minimum(dist, max_path_len)++ return edge_rel, topo_rel+++# ---------------------------------------------------------------------------+# 4. Kinematic chain decomposition+# ---------------------------------------------------------------------------++def compute_kinematic_chains(parents, policy='h_first', root=0):+ """Decompose a skeleton tree into kinematic chains (root-to-leaf paths).++ Args:+ parents: Parent indices (J,).+ policy: Branching priority โ ``'h_first'`` (highest index first)+ or ``'l_first'`` (lowest index first).+ root: Index of the root joint.++ Returns:+ List of chains, each a list of joint indices from root to leaf.+ """+ parents = list(parents)+ n = len(parents)+ children_dict = {i: [] for i in range(n)}++ for j in range(n):+ p = parents[j]+ if j == root or p is None or p == -1:+ continue+ if not (0 <= p < n):+ raise ValueError(f"Invalid parent index parents[{j}]={p}")+ if policy == 'h_first':+ _reverse_insort(children_dict[p], j)+ else:+ bisect.insort(children_dict[p], j)++ chains = []+ _build_kinchains([], root, children_dict, chains, policy)+ return chains+++def _reverse_insort(a, x, lo=0, hi=None):+ """Insert *x* into descending-sorted list *a*, keeping order."""+ if hi is None:+ hi = len(a)+ while lo < hi:+ mid = (lo + hi) // 2+ if x > a[mid]:+ hi = mid+ else:+ lo = mid + 1+ a.insert(lo, x)+++def _build_kinchains(chain, j, children_dict, chains, policy):+ """Recursively build kinematic chains from the skeleton tree."""+ children = children_dict[j]+ chain2 = chain + [j]++ if len(children) == 0:+ chains.append(chain2)+ return+ if len(children) == 1:+ _build_kinchains(chain2, children[0], children_dict, chains, policy)+ return++ main_child = children[0]+ for child in children:+ if child == main_child:+ _build_kinchains(chain2, child, children_dict, chains, policy)+ else:+ _build_kinchains([j], child, children_dict, chains, policy)diff --git a/data_process/utils/vlm.py b/data_process/utils/vlm.pynew file mode 100644index 0000000..ad2cfb2--- /dev/null+++ b/data_process/utils/vlm.py@@ -0,0 +1,759 @@+"""Shared utilities for the VLM captioning stage.++Frame loading / sampling / base64 encoding, multi-view prompt text,+render-directory discovery + completeness checks, resumable JSON I/O,+multi-GPU sharding, Qwen3-VL loading + generation, caption validation, and+the cached API clients used by :mod:`data_process.vlm_caption`.+"""++import base64+import json+import os+from functools import lru_cache+from io import BytesIO+from pathlib import Path++import torch+from loguru import logger+from PIL import Image+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Constants+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++# Camera azimuth of each view (v000..v003) relative to v000, about the+# vertical axis. The renderer's on-disk metadata names these views+# front/back/left/right, but those are WORLD-space camera positions โ the+# subject's orientation is arbitrary (normalize_scene never rotates it), so+# the words carry no information about the subject and would only bias a+# VLM into reading 'front view' as the subject's front. The model is+# therefore shown neutral azimuth labels instead.+VIEW_REL_AZIMUTHS = [0, 180, 270, 90]+NUM_VIEWS = len(VIEW_REL_AZIMUTHS)++# Rendered clips run to a few hundred frames per view; a fixed stride alone+# still sends 160 images at the median and 400 worst-case in a single request.+# Cap the per-view budget and sample uniformly across the clip instead.+# 64 frames x 4 views = 256 images/request: measured 31.6 GiB peak on+# Qwen3-VL-8B (A100-80GB), well within budget; the OOM retry ladder in+# generate_qwen halves this per attempt if a device is tighter.+DEFAULT_MAX_FRAMES_PER_VIEW = 64+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Frame loading+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++# Renders are saved with a transparent film; a plain .convert("RGB") drops+# alpha onto black, hiding dark subjects. Composite onto light grey instead+# (close to the render's own world-background tone).+RGBA_BACKGROUND = (235, 235, 235)+++def flatten_rgba(img, background=RGBA_BACKGROUND):+ # type: (Image.Image, tuple) -> Image.Image+ """Composite a possibly-transparent image onto an opaque RGB background."""+ if img.mode == "P":+ img = img.convert("RGBA")+ if img.mode in ("RGBA", "LA"):+ bg = Image.new("RGB", img.size, background)+ bg.paste(img, mask=img.split()[-1])+ return bg+ return img.convert("RGB")+++def uniform_subsample(items, max_items):+ # type: (List, Optional[int]) -> List+ """Uniformly pick at most ``max_items`` entries, keeping first and last.++ Used both to cap frames per view and to shrink a request after a CUDA+ OOM. A non-positive or None ``max_items`` means "no cap".+ """+ n = len(items)+ if not max_items or max_items <= 0 or n <= max_items:+ return list(items)+ if max_items == 1:+ return [items[0]]+ # round() spreads the picks evenly over [0, n-1] and pins both ends, so+ # the prompt's "frame 1 is the start, last frame is the end" holds.+ idxs = sorted({int(round(i * (n - 1) / (max_items - 1)))+ for i in range(max_items)})+ return [items[i] for i in idxs]+++def select_frame_paths(paths, downsample_rate=1, max_frames=None):+ # type: (List[Path], int, Optional[int]) -> List[Path]+ """Choose which frames of one view to send to the VLM.++ ``downsample_rate`` applies the legacy fixed stride first (the last+ frame is re-appended โ a bare ``[::rate]`` drops it whenever the count+ is not a multiple of the stride), then ``max_frames`` uniformly samples+ the remainder down to that budget.+ """+ if not paths:+ return []+ selected = list(paths)+ if downsample_rate > 1:+ selected = selected[::downsample_rate]+ if selected[-1] != paths[-1]:+ selected.append(paths[-1])+ return uniform_subsample(selected, max_frames)+++def load_frames_as_base64(view_dir, max_size=None, jpeg_quality=85,+ downsample_rate=1, max_frames=None):+ # type: (str, Optional[int], int, int, Optional[int]) -> List[str]+ """Load PNG frames from a view directory as base64 strings.++ Frame selection (stride + ``max_frames`` uniform cap, see+ :func:`select_frame_paths`) happens at the path level before decoding,+ so discarded frames are never read. When ``max_size`` is set, frames are+ resized so the longest side is at most ``max_size`` px and re-encoded as+ JPEG at ``jpeg_quality`` (10-20x payload shrink, quadratic savings in+ GPT-5 patch tokens). When ``max_size`` is None, raw PNG bytes pass+ through.+ """+ paths = select_frame_paths(+ sorted(Path(view_dir).glob("*.png")),+ downsample_rate=downsample_rate, max_frames=max_frames,+ )+ result = []+ for frame_path in paths:+ if max_size is None:+ with open(frame_path, "rb") as f:+ result.append(base64.b64encode(f.read()).decode("utf-8"))+ continue+ img = flatten_rgba(Image.open(frame_path))+ w, h = img.size+ long_side = max(w, h)+ if long_side > max_size:+ scale = max_size / long_side+ img = img.resize(+ (max(1, int(round(w * scale))), max(1, int(round(h * scale)))),+ Image.LANCZOS,+ )+ buf = BytesIO()+ img.save(buf, "JPEG", quality=jpeg_quality, optimize=True)+ result.append(base64.b64encode(buf.getvalue()).decode("utf-8"))+ return result+++def frame_mime_type(max_size):+ # type: (Optional[int]) -> str+ """Mime type produced by ``load_frames_as_base64`` for a given ``max_size``."""+ return "image/jpeg" if max_size is not None else "image/png"+++def _view_label_from_id(view_id):+ # type: (str) -> str+ # Label is keyed by the numeric index in the view id (v000โazimuth 0ยฐ),+ # NOT by iteration order โ so subsetting --views keeps labels correct.+ try:+ idx = int(view_id.lstrip("v"))+ except ValueError:+ return view_id+ if 0 <= idx < len(VIEW_REL_AZIMUTHS):+ return "camera azimuth {}ยฐ".format(VIEW_REL_AZIMUTHS[idx])+ return "view_{}".format(idx)+++def load_multiview_frames(+ render_dir,+ downsample_rate=2,+ views=None,+ max_size=None,+ jpeg_quality=85,+ max_frames_per_view=DEFAULT_MAX_FRAMES_PER_VIEW,+):+ # type: (str, int, Optional[List[str]], Optional[int], int, Optional[int]) -> Dict[str, List[str]]+ """Load and downsample frames from every view of a rendered motion.++ Returns {"v000 (camera azimuth 0ยฐ)": [b64, โฆ], โฆ}. Bytes are PNG when+ ``max_size`` is None, JPEG otherwise (see ``frame_mime_type``). At most+ ``max_frames_per_view`` frames per view are kept (0/None = no cap).++ Raises:+ ValueError: a requested view directory is missing or holds no+ frames. Callers gate on :func:`is_render_complete` first, so+ this means the render is broken โ captioning a subset of the+ views would silently freeze a partial-render caption.+ """+ render_path = Path(render_dir)+ if views is None:+ views = sorted([d.name for d in render_path.iterdir()+ if d.is_dir() and d.name.startswith("v")])++ result = {}+ for view_id in views:+ view_dir = render_path / view_id+ if not view_dir.exists():+ raise ValueError("View directory not found: {}".format(view_dir))+ frames = load_frames_as_base64(+ str(view_dir), max_size=max_size, jpeg_quality=jpeg_quality,+ downsample_rate=downsample_rate, max_frames=max_frames_per_view,+ )+ if not frames:+ raise ValueError("No frames found in {}".format(view_dir))+ view_label = _view_label_from_id(view_id)+ result["{} ({})".format(view_id, view_label)] = frames+ logger.info("Loaded {} ({}): {} frames (downsample_rate={}, "+ "max_frames_per_view={}, max_size={})".format(+ view_id, view_label, len(frames), downsample_rate,+ max_frames_per_view, max_size))++ return result+++RENDER_SOURCE_FPS = 30.0 # fallback when a view's metadata JSON is absent+++def _view_source_fps(render_path, view_id):+ # type: (Path, str) -> float+ """Source fps of one view, from the renderer's ``v00X.json`` metadata."""+ meta = render_path / "{}.json".format(view_id)+ try:+ with open(meta) as f:+ return float(json.load(f)["fps"])+ except (OSError, KeyError, ValueError):+ return RENDER_SOURCE_FPS+++def load_multiview_videos(+ render_dir,+ downsample_rate=2,+ views=None,+ max_frames_per_view=DEFAULT_MAX_FRAMES_PER_VIEW,+):+ # type: (str, int, Optional[List[str]], Optional[int]) -> Tuple[Dict[str, List], Dict[str, float]]+ """Load each view's frame PNGs as PIL lists for native video input.++ Same discovery, downsampling and per-view cap as+ :func:`load_multiview_frames`, but returns raw PIL frames (qwen video+ mode feeds them to the video processor, which does its own resize and+ 2-frame temporal patching) plus each view's effective fps after+ sampling, so timestamps stay truthful.++ Returns:+ ``({view_key: [PIL.Image, ...]}, {view_key: effective_fps})``.+ """+ render_path = Path(render_dir)+ if views is None:+ views = sorted([d.name for d in render_path.iterdir()+ if d.is_dir() and d.name.startswith("v")])++ videos, fps_map = {}, {}+ for view_id in views:+ view_dir = render_path / view_id+ if not view_dir.exists():+ raise ValueError("View directory not found: {}".format(view_dir))+ frame_paths = sorted(+ p for p in view_dir.iterdir() if p.suffix.lower() == ".png")+ frame_paths = frame_paths[::downsample_rate]+ n_downsampled = len(frame_paths)+ frame_paths = uniform_subsample(frame_paths, max_frames_per_view)+ if not frame_paths:+ raise ValueError("No frames found in {}".format(view_dir))+ fps = _view_source_fps(render_path, view_id) / downsample_rate+ if n_downsampled:+ fps *= len(frame_paths) / float(n_downsampled)+ key = "{} ({})".format(view_id, _view_label_from_id(view_id))+ videos[key] = [flatten_rgba(Image.open(str(p))) for p in frame_paths]+ fps_map[key] = fps+ logger.info("Loaded {} as video: {} frames @ {:.1f} fps".format(+ key, len(videos[key]), fps))+ return videos, fps_map+++def b64_to_pil(b64_str):+ # type: (str) -> Image.Image+ """Convert a base64-encoded PNG string to an opaque RGB PIL Image."""+ return flatten_rgba(Image.open(BytesIO(base64.b64decode(b64_str))))+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Multiview prompt text (shared by API and local backends)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def build_multiview_intro():+ # type: () -> str+ """Build the intro text for a multiview caption prompt.++ The clip directory name is never shown to the model โ dataset clip+ names are (or contain) the action label itself (mixamo:+ ``Acknowledging``; objaverse: ``{hex}-{Action}``; truebones:+ ``{Species}-{Action}``), so embedding one would leak the answer or a+ species word. Per-clip reference labels ride+ :func:`build_hint_footer` after the views instead.+ """+ intro = (+ "Below are multi-view video frames of a single motion clip. "+ "Each view shows the same motion from a different camera angle. "+ "Frames within each view are in chronological order (frame 1 is the "+ "start, last frame is the end). Analyze the temporal progression "+ "across all views and describe the motion.\n"+ )+ return intro+++def build_hint_footer(hint):+ # type: (Optional[str]) -> Optional[str]+ """Closing text carrying the per-clip reference label, placed AFTER the+ view sections so the frames are read unprimed and the label sits next+ to the generation point (image-first VQA layout; how much weight it+ gets is defined by the task's system prompt)."""+ if not hint:+ return None+ return ("Reference label from the dataset catalogue: '{}'.".format(hint))+++def build_view_header(view_name, num_frames):+ # type: (str, int) -> str+ """Build the header text for a single view section."""+ return "\n=== {} (frames 1-{}, chronological order) ===".format(+ view_name, num_frames)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Directory discovery+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def find_motion_dirs(render_root):+ # type: (str) -> List[Path]+ """Find all motion directories under a root (must contain view subdirs).++ A motion directory is one that has at least one child directory whose+ name starts with "v" (e.g., v000, v001). Partially rendered clips are+ intentionally still returned so callers can report them as (retryable)+ failures instead of silently dropping them โ use+ :func:`render_incomplete_reason` to gate captioning.++ Returns:+ Sorted list of Path objects.+ """+ render_root = Path(render_root)+ return sorted([+ d for d in render_root.iterdir()+ if d.is_dir() and any(+ sub.is_dir() and sub.name.startswith("v")+ for sub in d.iterdir()+ )+ ])+++def render_incomplete_reason(render_dir, num_views=NUM_VIEWS):+ # type: (str, int) -> Optional[str]+ """Return None when a clip's multi-view render is complete, else why not.++ Mirrors ``data_process.utils.blender_render.is_render_complete``:+ ``num_views`` view directories (v000..) with their camera metadata JSONs+ (v000.json..), each holding the same non-zero number of PNGs. Views+ render sequentially, so an interrupted render always leaves either a+ missing view directory or a view with fewer frames than the first.+ Reimplemented here (rather than imported) because ``blender_render``+ imports ``bpy``, which the captioning stage must not require.+ """+ render_path = Path(render_dir)+ if not render_path.is_dir():+ return "not a directory: {}".format(render_dir)++ frame_counts = []+ for view_idx in range(num_views):+ view_name = "v{:03d}".format(view_idx)+ if not (render_path / "{}.json".format(view_name)).is_file():+ return "missing camera metadata {}.json".format(view_name)+ view_dir = render_path / view_name+ if not view_dir.is_dir():+ return "missing view directory {}".format(view_name)+ frame_counts.append(len(list(view_dir.glob("*.png"))))++ if frame_counts[0] == 0:+ return "view v000 has no frames"+ if len(set(frame_counts)) != 1:+ return "unequal frame counts across views: {}".format(+ ", ".join("v{:03d}={}".format(i, c)+ for i, c in enumerate(frame_counts)))+ return None+++def is_render_complete(render_dir, num_views=NUM_VIEWS):+ # type: (str, int) -> bool+ """True when every expected view of a clip rendered to the same length."""+ return render_incomplete_reason(render_dir, num_views=num_views) is None+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# JSON I/O (resumable batch processing)+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def load_json(path):+ # type: (str) -> Dict+ """Load a JSON file, returning an empty dict if it doesn't exist."""+ if os.path.exists(path):+ with open(path) as f:+ data = json.load(f)+ logger.info("Loaded {} existing entries from {}".format(len(data), path))+ return data+ return {}+++def save_json(data, path):+ # type: (Dict, str) -> None+ """Atomically save a dict to a JSON file with indentation."""+ tmp = "{}.tmp.{}".format(path, os.getpid())+ with open(tmp, "w") as f:+ json.dump(data, f, indent=2)+ os.replace(tmp, path)+++def is_completed(result):+ # type: (Optional[str]) -> bool+ """Check if a result string represents a successful completion."""+ return bool(result) and not result.startswith("ERROR:")+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Multi-GPU sharding+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def shard_output_path(output_json, gpu_id):+ # type: (str, int) -> str+ """Return the shard-specific output path for a given GPU."""+ base, ext = os.path.splitext(output_json)+ return "{}.shard{}{}".format(base, gpu_id, ext)+++def find_shard_paths(output_json):+ # type: (str) -> List[str]+ """Every ``<base>.shard<N><ext>`` file sitting next to ``output_json``.++ Globbing (rather than trusting a ``num_gpus`` argument) means a merge+ run launched with fewer GPUs than the run that produced the shards can+ no longer silently drop the extra shards.+ """+ base, ext = os.path.splitext(output_json)+ pattern = "{}.shard*{}".format(os.path.basename(base), ext)+ paths = Path(base).parent.glob(pattern)++ def _gid(path):+ # type: (Path) -> Tuple[int, str]+ tail = path.name[len(os.path.basename(base)) + len(".shard"):]+ digits = tail[:-len(ext)] if ext else tail+ return (int(digits), path.name) if digits.isdigit() else (2 ** 31, path.name)++ return [str(p) for p in sorted(paths, key=_gid)]+++def merge_shards(output_json, num_gpus=None):+ # type: (str, Optional[int]) -> Dict[str, str]+ """Merge per-GPU shard files into a single output JSON.++ Loads every ``output_json.shard*`` file found next to ``output_json``,+ merges them on top of the existing ``output_json`` contents, and writes+ the combined result back. ``num_gpus`` is only used to warn when fewer+ shards than expected are present.++ Returns:+ The merged dict.+ """+ merged = {} # type: Dict[str, str]++ if os.path.exists(output_json):+ with open(output_json) as f:+ merged.update(json.load(f))+ logger.info("Loaded {} existing entries from {}".format(+ len(merged), output_json))++ shard_paths = find_shard_paths(output_json)+ for spath in shard_paths:+ with open(spath) as f:+ shard_data = json.load(f)+ merged.update(shard_data)+ logger.info("Merged shard {} ({} entries)".format(+ spath, len(shard_data)))++ if not shard_paths:+ logger.warning("No shard files found for {}".format(output_json))+ elif num_gpus is not None and len(shard_paths) < num_gpus:+ logger.warning("Found {} shard files but --num_gpus={}".format(+ len(shard_paths), num_gpus))++ save_json(merged, output_json)+ logger.info("Merged {} shards -> {} total entries in {}".format(+ len(shard_paths), len(merged), output_json))+ return merged+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Qwen model loading & generation+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def load_qwen_model(+ model_name,+ device_map="auto",+ torch_dtype=None,+ min_pixels=256 * 28 * 28,+ max_pixels=256 * 28 * 28,+ gpu_id=None,+):+ # type: (str, str, Optional[str], int, int, Optional[int]) -> Tuple+ """Load Qwen3-VL model and processor.++ Args:+ model_name: HuggingFace model ID or local path+ (e.g., "Qwen/Qwen3-VL-8B-Instruct").+ device_map: Device placement strategy (default "auto").+ torch_dtype: Override dtype ("float16", "bfloat16", or None for auto).+ min_pixels: Minimum pixels per image (default 200704 = 256*28*28).+ Lower = fewer visual tokens = faster + less memory.+ max_pixels: Maximum pixels per image (default 200704 = 256*28*28).+ The Qwen3-VL default is 1843200 (1280*1440) which is very+ expensive for multi-image inputs. 256*28*28 yields ~256 tokens+ per image, a good balance for rendered motion frames.+ gpu_id: If set, load the model on this specific GPU (overrides+ device_map). Used for multi-GPU data-parallel captioning.++ Returns:+ (model, processor) tuple.+ """+ from transformers import AutoModelForImageTextToText, AutoProcessor++ dtype = torch.bfloat16+ if torch_dtype == "float16":+ dtype = torch.float16+ elif torch_dtype == "bfloat16":+ dtype = torch.bfloat16++ if gpu_id is not None:+ device_map = "cuda:{}".format(gpu_id)++ # flash_attn is an optional compiled extra; fall back to PyTorch SDPA+ # (still fused attention) when it isn't installed rather than failing.+ try:+ import flash_attn # noqa: F401+ attn_implementation = "flash_attention_2"+ except ImportError:+ attn_implementation = "sdpa"++ logger.info("Loading model: {} (dtype={}, device_map={}, attn={})".format(+ model_name, dtype, device_map, attn_implementation))++ model = AutoModelForImageTextToText.from_pretrained(+ model_name,+ dtype=dtype,+ device_map=device_map,+ attn_implementation=attn_implementation,+ )+ processor = AutoProcessor.from_pretrained(+ model_name,+ min_pixels=min_pixels,+ max_pixels=max_pixels,+ )++ logger.info("Model loaded (min_pixels={}, max_pixels={}, "+ "~{} visual tokens per image)".format(+ min_pixels, max_pixels, max_pixels // (28 * 28)))+ return model, processor+++def qwen_generate(model, processor, messages, max_tokens=300,+ return_num_tokens=False, do_sample=False,+ videos=None, video_fps=None):+ # type: (object, object, List[Dict], int, bool, bool, Optional[List], Optional[List[float]]) -> Union[str, Tuple[str, int]]+ """Run Qwen3-VL generation on chat messages and return decoded text.++ Decoding is greedy by default: the shipped Qwen generation_config has+ ``do_sample=True``, which makes captions drift between identical runs+ (chirality flips, stray adverbs). Callers enable ``do_sample`` on+ retry attempts only, so a rejected output gets a genuinely different+ second chance while the first pass stays reproducible โ mirroring the+ fixed temperatures of the API backends.++ Args:+ model: Loaded Qwen3-VL model.+ processor: Loaded Qwen3-VL processor.+ messages: Chat messages in Qwen format (with PIL Images inline).+ max_tokens: Maximum new tokens to generate.+ return_num_tokens: Also return the number of generated tokens, so+ callers can detect truncation (``n == max_tokens``).+ do_sample: Sample (temperature 0.7) instead of greedy decoding.+ videos: Native video inputs, one ``[PIL.Image, ...]`` frame list per+ ``{"type": "video"}`` item in *messages*. When given, the chat+ template is rendered to text and the frames go through the+ processor's video path (2-frame temporal patching, timestamp+ alignment) instead of the per-image path.+ video_fps: Per-video effective fps matching *videos*, so the+ processor's timestamps reflect the true frame spacing.++ Returns:+ Decoded output string (stripped), or ``(text, num_new_tokens)`` when+ ``return_num_tokens`` is set.+ """+ if videos is None:+ inputs = processor.apply_chat_template(+ messages,+ tokenize=True,+ add_generation_prompt=True,+ return_dict=True,+ return_tensors="pt",+ )+ else:+ text = processor.apply_chat_template(+ messages, tokenize=False, add_generation_prompt=True)+ video_kwargs = {"fps": video_fps} if video_fps else {}+ try:+ inputs = processor(text=[text], videos=videos,+ return_tensors="pt", **video_kwargs)+ except TypeError:+ # Older processors without an fps kwarg.+ inputs = processor(text=[text], videos=videos,+ return_tensors="pt")+ inputs.pop("token_type_ids", None)+ logger.debug("Qwen prompt tokens: {}".format(inputs["input_ids"].shape[-1]))+ inputs = inputs.to(model.device)++ gen_kwargs = dict(max_new_tokens=max_tokens)+ if do_sample:+ gen_kwargs.update(do_sample=True, temperature=0.7, top_p=0.8, top_k=20)+ else:+ gen_kwargs.update(do_sample=False, temperature=None, top_p=None,+ top_k=None)++ with torch.no_grad():+ generated_ids = model.generate(**inputs, **gen_kwargs)++ generated_ids_trimmed = [+ out_ids[len(in_ids):]+ for in_ids, out_ids in zip(inputs.input_ids, generated_ids)+ ]+ text = processor.batch_decode(+ generated_ids_trimmed, skip_special_tokens=True,+ clean_up_tokenization_spaces=False,+ )[0].strip()+ if return_num_tokens:+ return text, len(generated_ids_trimmed[0])+ return text+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Qwen argparse helpers+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def add_qwen_model_args(parser):+ """Add common Qwen model CLI arguments to an argparse parser."""+ parser.add_argument('--model', type=str,+ default='Qwen/Qwen3-VL-8B-Instruct',+ help='HuggingFace model ID or local path '+ '(e.g., Qwen/Qwen3-VL-8B-Instruct, '+ 'Qwen/Qwen3-VL-72B-Instruct).')+ parser.add_argument('--torch_dtype', type=str, default=None,+ choices=['float16', 'bfloat16'],+ help='Override model dtype (default: bfloat16).')+ parser.add_argument('--device_map', type=str, default='auto',+ help='Device placement strategy (default: auto).')+ parser.add_argument('--min_pixels', type=int, default=256 * 28 * 28,+ help='Min pixels per image (default: 200704 = 256*28*28).')+ parser.add_argument('--max_pixels', type=int, default=256 * 28 * 28,+ help='Max pixels per image (default: 200704 = 256*28*28).')+++def add_multi_gpu_args(parser):+ """Add multi-GPU sharding CLI arguments to an argparse parser."""+ parser.add_argument('--num_gpus', type=int, default=1,+ help='Total number of GPUs for data-parallel captioning.')+ parser.add_argument('--gpu_id', type=int, default=0,+ help='GPU index for this process (0-indexed).')+ parser.add_argument('--merge_shards', action='store_true',+ help='Merge per-GPU shard files into a single '+ 'output JSON and exit.')+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Caption validation+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++# finish_reason values meaning "stopped at the token cap" โ the caption is+# truncated mid-sentence and must never be frozen into the output JSON.+TRUNCATED_FINISH_REASONS = ("length", "max_tokens")++MIN_CAPTION_CHARS = 8++# Matched against the START of the response only, so a legitimate caption+# that happens to contain one of these words is never rejected.+REFUSAL_PREFIXES = (+ "i'm sorry", "i am sorry", "sorry,", "sorry.", "i cannot", "i can't",+ "i can not", "i'm unable", "i am unable", "i'm not able", "i am not able",+ "unable to", "as an ai", "i apologize", "there is no image",+ "no image", "i don't see", "i do not see",+)+++def caption_rejection_reason(text, finish_reason=None):+ # type: (Optional[str], object) -> Optional[str]+ """Return None when *text* is a usable caption, else why it is rejected.++ Rejects empty/whitespace output, truncation at the token cap (OpenAI's+ ``finish_reason='length'``, Gemini's ``MAX_TOKENS``, Qwen hitting+ ``max_new_tokens``) and plain-text refusals. Deliberately conservative:+ anything else โ including an oddly worded caption โ is accepted, since+ a rejected caption costs another API call.+ """+ if not text or not text.strip():+ return "empty response (finish_reason={})".format(finish_reason)++ if finish_reason is not None:+ # Gemini reports an enum (types.FinishReason.MAX_TOKENS), OpenAI a str.+ token = str(finish_reason).rsplit(".", 1)[-1].lower()+ if token in TRUNCATED_FINISH_REASONS:+ return "truncated at the token cap (finish_reason={})".format(+ finish_reason)++ stripped = text.strip()+ if len(stripped) < MIN_CAPTION_CHARS:+ return "caption too short: {!r}".format(stripped)++ lowered = stripped.lower().lstrip("\"'*` ")+ if lowered.startswith(REFUSAL_PREFIXES):+ return "looks like a refusal: {!r}".format(stripped[:120])++ return None+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# API clients + retry classification+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++GEMINI_HTTP_TIMEOUT_MS = 90_000+RETRYABLE_ERROR_TOKENS = (+ "429", "503", "500", "504",+ "RESOURCE_EXHAUSTED", "UNAVAILABLE", "INTERNAL", "DEADLINE_EXCEEDED",+)+++@lru_cache(maxsize=4)+def openai_client(api_key, base_url):+ """Cache one thread-safe client per (api_key, base_url) tuple.++ A fresh ``OpenAI()`` per request leaks its connection pool (and the file+ descriptors behind it) across a 10k-clip run.+ """+ from openai import OpenAI++ return OpenAI(api_key=api_key, base_url=base_url)+++@lru_cache(maxsize=4)+def gemini_client(api_key, is_gemini3):+ """Cache one thread-safe Client per (api_key, api_version) tuple."""+ from google import genai++ http_options = {"timeout": GEMINI_HTTP_TIMEOUT_MS}+ if is_gemini3:+ http_options["api_version"] = "v1alpha"+ return genai.Client(api_key=api_key, http_options=http_options)+++def is_retryable_error(exc):+ """True for transient VLM API failures worth retrying (rate limit / 5xx)."""+ msg = str(exc)+ return any(tok in msg for tok in RETRYABLE_ERROR_TOKENS)diff --git a/data_process/vlm_caption/__init__.py b/data_process/vlm_caption/__init__.pynew file mode 100644index 0000000..10fd52a--- /dev/null+++ b/data_process/vlm_caption/__init__.py@@ -0,0 +1,10 @@+"""Stage 2b โ VLM captioning of multi-view renders.++Entry points (also available through ``data_process/scripts/run_caption_*.sh``):+ caption_motion โ per-clip motion captions from ``render/<dataset>/``.+ classify_category โ per-asset body-plan categories from T-pose grids.++Supporting modules: ``backends`` (Qwen / OpenAI / Gemini adapters) and+``prompts`` (system prompts + task registry). Shared VLM plumbing lives in+:mod:`data_process.utils.vlm`.+"""diff --git a/data_process/vlm_caption/backends.py b/data_process/vlm_caption/backends.pynew file mode 100644index 0000000..65a85a4--- /dev/null+++ b/data_process/vlm_caption/backends.py@@ -0,0 +1,423 @@+"""VLM backend adapters for motion captioning.++One captioning request is the same everywhere โ a system prompt plus an+interleaved [view header, frame label, image, ...] user turn โ but each+backend speaks a different dialect. This module owns those dialects:++ qwen โ local Qwen3-VL via HuggingFace transformers; PIL images+ inline, OOM-aware retries that halve the frame count.+ openai โ OpenAI-compatible chat API; base64 ``data:`` image URLs,+ GPT-5 vs GPT-4o parameter differences handled here.+ gemini โ Google Gemini API; raw image bytes as Parts, thinking and+ media-resolution controls for Gemini 2.5 / 3.++Each ``generate_*`` function takes the loaded multi-view frames and returns+a caption string ("" after all retries fail); the caller decides what to do+with failures. :func:`detect_backend` maps a model name to a backend and+:func:`image_params` picks the frame loader's resize/quality settings per+backend, so the orchestrator (:mod:`.caption_motion`) never needs+backend-specific knowledge beyond dispatching on the backend name.+"""++import base64+import os+import random+import time++from loguru import logger++from data_process.utils.vlm import (+ b64_to_pil,+ build_hint_footer,+ build_multiview_intro,+ build_view_header,+ caption_rejection_reason,+ gemini_client,+ is_retryable_error,+ openai_client,+ qwen_generate,+ uniform_subsample,+)+++BACKEND_QWEN = "qwen"+BACKEND_OPENAI = "openai"+BACKEND_GEMINI = "gemini"+VALID_BACKENDS = [BACKEND_QWEN, BACKEND_OPENAI, BACKEND_GEMINI]++MAX_RETRIES = 3+++def detect_backend(model_name):+ # type: (str) -> str+ """Auto-detect backend from model name.++ - Contains 'qwen' (case-insensitive) -> qwen+ - Starts with 'gemini' -> gemini+ - Everything else -> openai+ """+ if "qwen" in model_name.lower():+ return BACKEND_QWEN+ elif model_name.startswith("gemini"):+ return BACKEND_GEMINI+ else:+ return BACKEND_OPENAI+++# Loader-side resize targets per (backend, model). Qwen relies on its+# AutoProcessor (min/max_pixels) for server-side resize; API backends pre-shrink+# to keep payloads small and, for GPT-5 patch vision, to cut tokens quadratically.+# GPT-5: ceil(w/32)ยฒยทmult โ 384ยฒ โ ~233 tok vs 415 at 512ยฒ, q85 stays crisp.+# GPT-4o tile scheme with detail='low' is flat 85 tok โ size only affects bandwidth.+# Gemini: server-side media_resolution sets tokens; 768 caps the inline upload.+def image_params(backend, model_name):+ # type: (str, str) -> tuple+ """Return ``(max_size, jpeg_quality)`` for the frame loader."""+ if backend == BACKEND_QWEN:+ return None, 85+ if backend == BACKEND_GEMINI:+ return 768, 85+ if model_name.startswith("gpt-5"):+ return 384, 85+ return 512, 80+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Message building+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _build_qwen_messages(multiview_frames, system_prompt, hint=None):+ # type: (Dict[str, List[str]], str, Optional[str]) -> List[Dict]+ """Build Qwen3-VL chat messages with PIL Images inline."""+ user_content = []+ user_content.append({+ "type": "text",+ "text": build_multiview_intro(),+ })++ for view_name, frames in multiview_frames.items():+ user_content.append({+ "type": "text",+ "text": build_view_header(view_name, len(frames)),+ })+ for idx, frame_b64 in enumerate(frames):+ user_content.append({+ "type": "text",+ "text": "[frame {}]".format(idx + 1),+ })+ user_content.append({+ "type": "image",+ "image": b64_to_pil(frame_b64),+ })++ footer = build_hint_footer(hint)+ if footer:+ user_content.append({"type": "text", "text": footer})++ return [+ {"role": "system", "content": [{"type": "text", "text": system_prompt}]},+ {"role": "user", "content": user_content},+ ]+++def _build_qwen_video_messages(multiview_videos, system_prompt, hint=None):+ # type: (Dict[str, List], str, Optional[str]) -> List[Dict]+ """Build Qwen3-VL chat messages with one native video item per view.++ Each view's PIL frame list rides the processor's video path (2-frame+ temporal patching + timestamp alignment) instead of being spelled out+ as individually labeled images โ half the visual tokens and native+ temporal encoding for the same frames.+ """+ user_content = [{+ "type": "text",+ "text": build_multiview_intro(),+ }]+ for view_name, frames in multiview_videos.items():+ user_content.append({+ "type": "text",+ "text": build_view_header(view_name, len(frames)),+ })+ user_content.append({"type": "video", "video": frames})++ footer = build_hint_footer(hint)+ if footer:+ user_content.append({"type": "text", "text": footer})++ return [+ {"role": "system", "content": [{"type": "text", "text": system_prompt}]},+ {"role": "user", "content": user_content},+ ]+++def _build_openai_messages(multiview_frames, system_prompt, mime_type,+ hint=None):+ # type: (Dict[str, List[str]], str, str, Optional[str]) -> List[Dict]+ """Build OpenAI-compatible chat messages with base64 image URLs.++ Frames are already resized + encoded by the loader; this just wraps them+ in the ``data:`` URL scheme.+ """+ user_content = []+ user_content.append({+ "type": "text",+ "text": build_multiview_intro(),+ })++ for view_name, frames in multiview_frames.items():+ user_content.append({+ "type": "text",+ "text": build_view_header(view_name, len(frames)),+ })+ for idx, frame_b64 in enumerate(frames):+ user_content.append({+ "type": "text",+ "text": "[frame {}]".format(idx + 1),+ })+ user_content.append({+ "type": "image_url",+ "image_url": {+ "url": "data:{};base64,{}".format(mime_type, frame_b64),+ "detail": "low",+ },+ })++ footer = build_hint_footer(hint)+ if footer:+ user_content.append({"type": "text", "text": footer})++ return [+ {"role": "system", "content": system_prompt},+ {"role": "user", "content": user_content},+ ]+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Generation+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def generate_qwen(multiview_frames, motion_name, system_prompt,+ model, processor, max_tokens, max_retries=MAX_RETRIES,+ hint=None, video_fps=None):+ # type: (Dict[str, List], str, str, object, object, int, int, Optional[str], Optional[Dict[str, float]]) -> str+ """Generate caption using local Qwen3-VL model.++ ``multiview_frames`` carries base64 PNG/JPEG strings in image mode, or+ PIL frame lists when ``video_fps`` is given โ the latter selects the+ native video input path (one video item per view).++ Retries transient failures (OOM, empty/truncated/refusal output) up to+ max_retries times. An OOM halves the frames per view before retrying โ+ resubmitting the identical image stack would OOM again deterministically.+ """+ import torch # local import; keeps module light if never hit++ frames = multiview_frames+ video_mode = video_fps is not None++ def _assemble(f, fps_scale=1.0):+ if video_mode:+ msgs = _build_qwen_video_messages(f, system_prompt, hint=hint)+ fps = [video_fps[v] * fps_scale for v in f]+ return msgs, {"videos": [f[v] for v in f], "video_fps": fps}+ return _build_qwen_messages(f, system_prompt, hint=hint), {}++ fps_scale = 1.0+ messages, vid_kwargs = _assemble(frames)++ for attempt in range(1, max_retries + 1):+ try:+ # First attempt is greedy (reproducible); retries sample so a+ # rejected output gets a genuinely different second chance.+ text, num_tokens = qwen_generate(+ model, processor, messages, max_tokens=max_tokens,+ return_num_tokens=True, do_sample=attempt > 1,+ **vid_kwargs,+ )+ # Qwen exposes no finish_reason; hitting the cap means truncation.+ finish_reason = "length" if num_tokens >= max_tokens else "stop"+ reason = caption_rejection_reason(text, finish_reason)+ if reason is None:+ return text.strip()+ logger.warning("Attempt {}/{}: rejected Qwen output โ {}".format(+ attempt, max_retries, reason))+ except torch.cuda.OutOfMemoryError as e:+ logger.warning("Attempt {}/{}: CUDA OOM: {}".format(+ attempt, max_retries, e))+ torch.cuda.empty_cache()+ frames = {+ view: uniform_subsample(f, max(1, len(f) // 2))+ for view, f in frames.items()+ }+ # Halving the frames doubles their temporal spacing.+ fps_scale *= 0.5+ logger.warning("Retrying with {} frames (halved per view)".format(+ sum(len(f) for f in frames.values())))+ messages, vid_kwargs = _assemble(frames, fps_scale=fps_scale)+ except Exception as e:+ logger.warning("Attempt {}/{}: Qwen error: {}".format(+ attempt, max_retries, e))++ if attempt < max_retries:+ time.sleep(1)++ logger.error("All {} Qwen attempts failed or returned empty for '{}'".format(+ max_retries, motion_name))+ return ""+++def generate_openai(multiview_frames, motion_name, system_prompt,+ model_name, api_key, base_url, max_tokens, max_retries,+ mime_type, hint=None):+ # type: (Dict[str, List[str]], str, str, str, Optional[str], Optional[str], int, int, str, Optional[str]) -> str+ """Generate caption using OpenAI-compatible API."""+ client = openai_client(+ api_key or os.environ.get("OPENAI_API_KEY"), base_url,+ )++ messages = _build_openai_messages(+ multiview_frames, system_prompt, mime_type=mime_type, hint=hint,+ )++ # gpt-5+ reasoning models use max_completion_tokens and don't support+ # temperature; older models (gpt-4o, etc.) use max_tokens.+ is_gpt5 = "gpt-5" in model_name+ create_kwargs = dict(model=model_name, messages=messages)+ if is_gpt5:+ create_kwargs["max_completion_tokens"] = max_tokens+ else:+ create_kwargs["max_tokens"] = max_tokens+ create_kwargs["temperature"] = 0.3++ for attempt in range(1, max_retries + 1):+ try:+ response = client.chat.completions.create(**create_kwargs)++ choice = response.choices[0]+ content = choice.message.content+ refusal = getattr(choice.message, "refusal", None)+ logger.debug("Attempt {}: finish_reason={}, refusal={}".format(+ attempt, choice.finish_reason, refusal))++ if refusal:+ logger.warning("Attempt {}/{}: model refused: {}".format(+ attempt, max_retries, refusal))+ else:+ reason = caption_rejection_reason(content, choice.finish_reason)+ if reason is None:+ return content.strip()+ logger.warning("Attempt {}/{}: rejected response โ {}".format(+ attempt, max_retries, reason))+ except Exception as e:+ logger.warning("Attempt {}/{}: OpenAI API error: {}".format(+ attempt, max_retries, e))++ if attempt < max_retries:+ time.sleep(2)++ logger.error("All {} attempts failed or returned empty for '{}'".format(+ max_retries, motion_name))+ return ""+++def generate_gemini(multiview_frames, motion_name, system_prompt,+ model_name, api_key, max_tokens, max_retries,+ mime_type="image/png",+ thinking_level="low", media_resolution="medium",+ thinking_budget=0, hint=None):+ # type: (Dict[str, List[str]], str, str, str, Optional[str], int, int, str, str, str, int, Optional[str]) -> str+ """Generate caption using Google Gemini API.++ Gemini 3 uses ``thinking_level`` (enum) + ``media_resolution``; Gemini 2.5+ uses ``thinking_budget`` (int token cap, 0 disables, -1 dynamic). For a+ one-sentence caption, dynamic thinking is wasteful โ default is 0.+ """+ from google.genai import types++ is_gemini3 = "gemini-3" in model_name+ is_gemini25 = "gemini-2.5" in model_name++ # Gemini 3's thinking_config draws reasoning tokens from the same budget+ # as the final answer. At thinking_level='low' the reasoning pass commonly+ # burns 200โ500 tokens, so <1024 total risks returning an empty answer.+ if is_gemini3 and max_tokens < 1024:+ logger.warning(+ "max_tokens={} is tight for Gemini 3 + thinking; raising to 1024."+ .format(max_tokens))+ max_tokens = 1024++ client = gemini_client(+ api_key or os.environ.get("GOOGLE_API_KEY"), is_gemini3+ )++ contents = [types.Part.from_text(text=build_multiview_intro())]+ for view_name, frames in multiview_frames.items():+ contents.append(types.Part.from_text(+ text=build_view_header(view_name, len(frames)),+ ))+ for idx, frame_b64 in enumerate(frames):+ contents.append(types.Part.from_text(+ text="[frame {}]".format(idx + 1),+ ))+ contents.append(types.Part.from_bytes(+ data=base64.b64decode(frame_b64),+ mime_type=mime_type,+ ))+ footer = build_hint_footer(hint)+ if footer:+ contents.append(types.Part.from_text(text=footer))++ config_kwargs = dict(+ system_instruction=system_prompt,+ max_output_tokens=max_tokens,+ temperature=0.0,+ stop_sequences=["\n\n"],+ )+ if is_gemini3:+ config_kwargs["thinking_config"] = types.ThinkingConfig(+ thinking_level=thinking_level,+ )+ config_kwargs["media_resolution"] = "MEDIA_RESOLUTION_{}".format(+ media_resolution.upper()+ )+ elif is_gemini25:+ config_kwargs["thinking_config"] = types.ThinkingConfig(+ thinking_budget=thinking_budget,+ )+ config = types.GenerateContentConfig(**config_kwargs)++ for attempt in range(1, max_retries + 1):+ try:+ response = client.models.generate_content(+ model=model_name,+ contents=contents,+ config=config,+ )+ candidates = getattr(response, "candidates", None) or []+ finish_reason = getattr(candidates[0], "finish_reason", None) \+ if candidates else None+ reason = caption_rejection_reason(response.text, finish_reason)+ if reason is None:+ return response.text.strip()+ # A rejected response (empty, truncated at MAX_TOKENS, refusal)+ # is worth another shot; fall through to the backoff.+ logger.warning("Attempt {}/{}: rejected Gemini response โ {}".format(+ attempt, max_retries, reason))+ except Exception as e:+ retryable = is_retryable_error(e)+ level = "warning" if retryable else "error"+ getattr(logger, level)(+ "Attempt {}/{}: Gemini API error ({}): {}".format(+ attempt, max_retries,+ "retryable" if retryable else "fatal", e))+ if not retryable:+ break++ if attempt < max_retries:+ # Exponential backoff with jitter: 5s, 15s, 45s, โฆ capped at 60s.+ delay = min(60.0, 5.0 * (3 ** (attempt - 1))) + random.uniform(0, 2)+ time.sleep(delay)++ logger.error("All {} attempts failed or returned empty for '{}'".format(+ max_retries, motion_name))+ return ""diff --git a/data_process/vlm_caption/caption_motion.py b/data_process/vlm_caption/caption_motion.pynew file mode 100644index 0000000..911df7f--- /dev/null+++ b/data_process/vlm_caption/caption_motion.py@@ -0,0 +1,563 @@+"""Caption multi-view rendered motion sequences with a Vision-Language Model.++Supports three backends (see :mod:`data_process.vlm_caption.backends`):+ qwen โ Local Qwen3-VL via HuggingFace transformers (default).+ openai โ OpenAI-compatible API (GPT-4o, GPT-5, ...).+ gemini โ Google Gemini API.++The backend is auto-detected from the model name, or set explicitly via+``--backend``. ``--task`` selects the dataset system prompt (see+:mod:`data_process.vlm_caption.prompts`): ``mixamo``, ``objaverse`` or+``truebones``.++Input layout: ``<render_root>/<motion_name>/v00{0..3}/*.png`` โ one+directory per motion with one sub-directory of frame PNGs per camera view.++``--hints_json`` optionally feeds per-clip reference labels from dataset+metadata (Mixamo's official catalogue prompts, truebones' T2M4LVO+annotations) to the model as frame-grounded reference labels;+``run_caption_motion.sh`` wires this up for both by default.++Usage:+ # Local Qwen (auto-detected from model name)+ python -m data_process.vlm_caption.caption_motion \+ --render_root dataset/render/truebones --task truebones++ # Gemini 3 Flash API+ python -m data_process.vlm_caption.caption_motion \+ --render_root dataset/render/objaverse \+ --model gemini-3-flash-preview --task objaverse --num_workers 16++ # Multi-GPU (qwen backend only)+ python -m data_process.vlm_caption.caption_motion \+ --render_root dataset/render/objaverse --task objaverse --num_gpus 4 --gpu_id 0++ # Merge shards after multi-GPU+ python -m data_process.vlm_caption.caption_motion \+ --merge_shards --output_json motion_captions.json --num_gpus 4+"""++import argparse+import os+import re+import threading+from concurrent.futures import ThreadPoolExecutor, as_completed+from pathlib import Path++from loguru import logger+from tqdm import tqdm++from data_process.vlm_caption.backends import (+ BACKEND_GEMINI,+ BACKEND_QWEN,+ MAX_RETRIES,+ VALID_BACKENDS,+ detect_backend,+ generate_gemini,+ generate_openai,+ generate_qwen,+ image_params,+)+from data_process.vlm_caption.prompts import TASK_PROMPTS, get_prompt+from data_process.utils.vlm import (+ DEFAULT_MAX_FRAMES_PER_VIEW,+ add_multi_gpu_args,+ find_motion_dirs,+ frame_mime_type,+ is_completed,+ load_json,+ load_multiview_frames,+ load_multiview_videos,+ load_qwen_model,+ merge_shards,+ render_incomplete_reason,+ save_json,+ shard_output_path,+)+++SAVE_EVERY = 20 # captions per incremental JSON flush+++def load_hints(hints_json):+ # type: (Optional[str]) -> Dict[str, str]+ """Load per-clip reference labels from a dataset metadata JSON.++ Accepts either flat ``{name: label}`` or record values like Mixamo's+ ``animation_motion_prompts.json`` (``{name: {prompt, description, ...}}``,+ where ``description`` is preferred over ``prompt`` as the more specific+ label). Keys are normalised to the render clip-directory name by+ stripping a file extension, so ``Jab_Cross.fbx`` matches the render dir+ ``Jab_Cross``.+ """+ if not hints_json:+ return {}+ hints = {}+ for key, value in load_json(hints_json).items():+ name = os.path.splitext(key)[0]+ if isinstance(value, dict):+ label = value.get("description") or value.get("prompt") or ""+ # Keep only the ACTION as reference: records that name their+ # subject (truebones' object_name, e.g. 'Tyrannosaurus Rex')+ # get it replaced with the neutral word so no species reaches+ # the model.+ subject = value.get("object_name")+ if subject and label:+ label = re.sub(re.escape(subject), "animal", label,+ flags=re.IGNORECASE)+ label = (label.replace("A animal", "An animal")+ .replace("a animal", "an animal"))+ else:+ label = str(value)+ if label:+ hints[name] = label+ logger.info("Loaded {} caption hints from {}".format(len(hints), hints_json))+ return hints+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Batch captioning+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def caption_batch(+ render_root,+ backend,+ system_prompt,+ output_json=None,+ downsample_rate=1,+ max_frames_per_view=None,+ max_tokens=300,+ views=None,+ hints=None,+ vision_input="video",+ gpu_id=0,+ num_gpus=1,+ num_workers=1,+ # qwen-specific+ qwen_model=None,+ qwen_processor=None,+ # api-specific+ model_name=None,+ api_key=None,+ base_url=None,+ max_retries=MAX_RETRIES,+ # gemini-specific+ thinking_level="low",+ media_resolution="medium",+ thinking_budget=0,+):+ """Caption all motion sequences under a root render directory.++ Skips sequences that already have captions in the output JSON (for+ resumability). Clips whose render is incomplete are reported as failures+ and left out of the JSON, so they are retried once rendering finishes.+ Results are saved incrementally (every SAVE_EVERY captions).++ Args:+ max_frames_per_view: Cap on frames sent per view after+ ``downsample_rate`` (0 = no cap; frames are sampled+ uniformly across the clip, first and last always kept).+ Default None resolves per mode: no cap for 'video' (its+ 2-frame temporal patching keeps even a 200-frame clip+ cheap), DEFAULT_MAX_FRAMES_PER_VIEW for 'images'.+ vision_input: 'video' (default) sends each view through the+ qwen backend's native video path (2-frame temporal patching+ + timestamp alignment, ~half the visual tokens of 'images');+ 'images' sends individually labeled frames. Video is+ qwen-only; API backends fall back to images with a warning.+ hints: Optional ``{clip_name: reference_label}`` mapping (see+ ``load_hints``); a clip's label is passed to the model as a+ verify-against-frames hint.+ gpu_id: GPU/process index for multi-GPU sharding (default 0).+ num_gpus: Total number of GPUs/processes (default 1 = no sharding).+ When num_gpus > 1, each process handles every num_gpus-th motion+ and saves to a shard file. Use merge_shards() afterwards.+ num_workers: Concurrent in-process workers. Only effective for HTTP+ backends (gemini/openai); Qwen ignores this since one local model+ can't be shared across threads safely.+ """+ render_root = Path(render_root)++ if output_json is None:+ output_json = str(render_root / "motion_captions.json")++ save_path = shard_output_path(output_json, gpu_id) if num_gpus > 1 else output_json+ save_path_obj = Path(save_path)+ failed_path = str(save_path_obj.with_name(save_path_obj.stem + "_failed.txt"))++ captions = load_json(save_path)++ # Resume from the merged output as well: shard files are not cleaned up+ # after a merge, so a tidied-up directory would otherwise re-caption the+ # whole dataset. The shard wins on conflict; only the shard's own entries+ # are written back to save_path.+ already_done = dict(captions)+ if save_path != output_json:+ for name, caption in load_json(output_json).items():+ already_done.setdefault(name, caption)++ motion_dirs = find_motion_dirs(str(render_root))++ failed_set = set()+ if os.path.exists(failed_path):+ with open(failed_path, "r") as f:+ for line in f:+ name = line.strip()+ if name:+ failed_set.add(name)++ if num_gpus > 1:+ motion_dirs = motion_dirs[gpu_id::num_gpus]+ logger.info("[GPU {}/{}] Assigned {} motion sequences".format(+ gpu_id, num_gpus, len(motion_dirs)))+ else:+ logger.info("Found {} motion sequences in {}".format(+ len(motion_dirs), render_root))++ pending = [d for d in motion_dirs if not is_completed(already_done.get(d.name))]+ logger.info("{} already done, {} pending.".format(+ len(motion_dirs) - len(pending), len(pending)))++ if backend == BACKEND_QWEN and num_workers > 1:+ logger.warning("Qwen backend does not support num_workers > 1; using 1.")+ num_workers = 1++ if vision_input == "video" and backend != BACKEND_QWEN:+ logger.warning("--vision_input video is qwen-only; "+ "falling back to images for backend '{}'.".format(backend))+ vision_input = "images"++ if max_frames_per_view is None:+ max_frames_per_view = 0 if vision_input == "video" \+ else DEFAULT_MAX_FRAMES_PER_VIEW+ logger.info("vision_input={} max_frames_per_view={}".format(+ vision_input, max_frames_per_view or "uncapped"))++ max_size, jpeg_quality = image_params(backend, model_name or "")+ mime_type = frame_mime_type(max_size)+ logger.info("Frame loader: max_size={} jpeg_q={} mime={} "+ "downsample_rate={} max_frames_per_view={}".format(+ max_size, jpeg_quality, mime_type, downsample_rate,+ max_frames_per_view))++ state_lock = threading.Lock()+ n_ok = 0+ n_err = 0+ n_unsaved = 0++ def _process(motion_dir):+ motion_name = motion_dir.name+ try:+ # A partially rendered clip must never be captioned: the caption+ # would be frozen forever (is_completed() accepts any non-ERROR+ # string). Report it as a failure so a later run retries it once+ # rendering has finished.+ incomplete = render_incomplete_reason(str(motion_dir))+ if incomplete is not None:+ return motion_name, None, "incomplete render: {}".format(+ incomplete)++ logger.info("Captioning [{}/{}]: {}".format(+ backend, vision_input, motion_name))+ video_fps = None+ if vision_input == "video":+ multiview_frames, video_fps = load_multiview_videos(+ str(motion_dir),+ downsample_rate=downsample_rate, views=views,+ max_frames_per_view=max_frames_per_view,+ )+ else:+ multiview_frames = load_multiview_frames(+ str(motion_dir),+ downsample_rate=downsample_rate, views=views,+ max_size=max_size, jpeg_quality=jpeg_quality,+ max_frames_per_view=max_frames_per_view,+ )+ if not multiview_frames:+ return motion_name, None, "no frames"++ total_images = sum(len(f) for f in multiview_frames.values())+ logger.info("Total frames for VLM: {}".format(total_images))++ hint = hints.get(motion_name) if hints else None+ if backend == BACKEND_QWEN:+ caption = generate_qwen(+ multiview_frames, motion_name, system_prompt,+ model=qwen_model, processor=qwen_processor,+ max_tokens=max_tokens, max_retries=max_retries,+ hint=hint, video_fps=video_fps,+ )+ elif backend == BACKEND_GEMINI:+ caption = generate_gemini(+ multiview_frames, motion_name, system_prompt,+ model_name=model_name, api_key=api_key,+ max_tokens=max_tokens, max_retries=max_retries,+ mime_type=mime_type,+ thinking_level=thinking_level,+ media_resolution=media_resolution,+ thinking_budget=thinking_budget,+ hint=hint,+ )+ else:+ caption = generate_openai(+ multiview_frames, motion_name, system_prompt,+ model_name=model_name, api_key=api_key, base_url=base_url,+ max_tokens=max_tokens, max_retries=max_retries,+ mime_type=mime_type, hint=hint,+ )++ logger.info("Caption for '{}': {}".format(motion_name, caption))+ if caption:+ return motion_name, caption, None+ return motion_name, None, "empty response"+ except Exception as e:+ logger.error("Failed to caption {}: {}".format(motion_name, e))+ return motion_name, None, str(e)++ gpu_label = " GPU {}".format(gpu_id) if num_gpus > 1 else ""+ pbar = tqdm(total=len(pending), desc="Captioning{}".format(gpu_label),+ unit="seq", dynamic_ncols=True)++ def _record(motion_name, caption, error):+ nonlocal n_ok, n_err, n_unsaved+ with state_lock:+ if caption is not None:+ captions[motion_name] = caption+ n_ok += 1+ n_unsaved += 1+ # Rewriting the full JSON per caption is O(n^2) over a large+ # run; batch it (a crash loses at most SAVE_EVERY captions).+ if n_unsaved >= SAVE_EVERY:+ save_json(captions, save_path)+ n_unsaved = 0+ else:+ n_err += 1+ if motion_name not in failed_set:+ failed_set.add(motion_name)+ with open(failed_path, "a") as f:+ f.write(motion_name + "\n")+ logger.warning("Skipped {} (not saved to JSON): {}".format(+ motion_name, error))+ pbar.set_postfix_str("{} ok / {} err".format(n_ok, n_err),+ refresh=False)+ pbar.update(1)++ try:+ if num_workers <= 1:+ for d in pending:+ name, caption, error = _process(d)+ _record(name, caption, error)+ else:+ with ThreadPoolExecutor(max_workers=num_workers) as ex:+ futures = [ex.submit(_process, d) for d in pending]+ try:+ for fut in as_completed(futures):+ name, caption, error = fut.result()+ _record(name, caption, error)+ except KeyboardInterrupt:+ # Every item is queued upfront; without cancel_futures the+ # pool's shutdown would still run (and bill) all of them.+ logger.warning("Interrupted โ cancelling queued requests.")+ ex.shutdown(wait=False, cancel_futures=True)+ raise+ finally:+ with state_lock:+ if n_unsaved:+ save_json(captions, save_path)+ n_unsaved = 0++ pbar.close()+ logger.info("Done. {} ok / {} err this run; {} total saved to {}".format(+ n_ok, n_err, len(captions), save_path))+ if n_err:+ logger.info("Failed motion names appended to {}".format(failed_path))+ return captions+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# CLI+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_args():+ parser = argparse.ArgumentParser(+ description="Caption multi-view motion renders using a VLM "+ "(local Qwen, OpenAI API, or Gemini API)."+ )++ parser.add_argument('--render_root', type=str,+ help='Root directory for batch captioning all motions.')++ parser.add_argument('--output_json', type=str, default=None,+ help='Output JSON file. '+ 'Defaults to <render_root>/motion_captions.json.')++ # โโ Backend & task โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ parser.add_argument('--backend', type=str, default=None,+ choices=VALID_BACKENDS,+ help='VLM backend. Auto-detected from --model if '+ 'not specified (qwen/openai/gemini).')+ parser.add_argument('--task', type=str, required=True,+ choices=sorted(TASK_PROMPTS.keys()),+ help='Captioning task / system prompt to use.')+ parser.add_argument('--hints_json', type=str, default=None,+ help='Optional JSON with per-clip reference labels '+ 'shown to the model as verify-against-frames '+ 'hints (e.g. dataset/raw/mixamo/'+ 'animation_motion_prompts.json). Keys may carry '+ 'a file extension; dict values use description '+ 'over prompt.')++ # โโ Model โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ parser.add_argument('--model', type=str,+ default='Qwen/Qwen3-VL-8B-Instruct',+ help='Model name. For qwen: HuggingFace model ID '+ '(default Qwen/Qwen3-VL-8B-Instruct). For openai: '+ 'API model (e.g., gpt-5-mini). For gemini: API model '+ '(e.g., gemini-3-flash-preview).')++ # โโ Qwen-specific โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ parser.add_argument('--torch_dtype', type=str, default=None,+ choices=['float16', 'bfloat16'],+ help='[qwen] Override model dtype (default: bfloat16).')+ parser.add_argument('--device_map', type=str, default='auto',+ help='[qwen] Device placement strategy (default: auto).')+ parser.add_argument('--min_pixels', type=int, default=256 * 28 * 28,+ help='[qwen] Min pixels per image '+ '(default: 200704 = 256*28*28).')+ parser.add_argument('--max_pixels', type=int, default=256 * 28 * 28,+ help='[qwen] Max pixels per image '+ '(default: 200704 = 256*28*28).')++ # โโ API-specific โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ parser.add_argument('--api_key', type=str, default=None,+ help='[openai/gemini] API key (defaults to '+ 'OPENAI_API_KEY or GOOGLE_API_KEY env var).')+ parser.add_argument('--base_url', type=str, default=None,+ help='[openai] Override API base URL.')+ parser.add_argument('--max_retries', type=int, default=MAX_RETRIES,+ help='[openai/gemini] Max retry attempts for '+ 'empty responses.')++ # โโ Gemini 3-specific โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ parser.add_argument('--thinking_level', type=str, default='low',+ choices=['minimal', 'low', 'medium', 'high'],+ help='[gemini-3] Reasoning depth. Captioning works '+ 'well at "low" (default); raise to "medium" if '+ 'captions are noisy.')+ parser.add_argument('--media_resolution', type=str, default='medium',+ choices=['low', 'medium', 'high', 'ultra_high'],+ help='[gemini-3] Vision tokens per image. "medium" '+ '(560 tok) is a good default for multi-view '+ 'frames; "high" (1120 tok) for fine details. '+ '"ultra_high" is Gemini 3 Flash only.')+ parser.add_argument('--thinking_budget', type=int, default=0,+ help='[gemini-2.5] Thinking token cap. 0 disables '+ '(recommended for one-sentence captions); -1 '+ 'enables dynamic thinking; up to 24576.')++ # โโ Common โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ parser.add_argument('--vision_input', type=str, default='video',+ choices=['images', 'video'],+ help="How frames reach the model. 'video' (default) "+ "uses the qwen backend's native video path "+ "(temporal patching + timestamps, ~half the "+ "visual tokens; API backends fall back to "+ "images); 'images' sends individually labeled "+ "frames.")+ parser.add_argument('--views', type=str, nargs='+', default=None,+ help='Subset of views to use (e.g., v000 v002). '+ 'Default: all views.')+ parser.add_argument('--downsample_rate', type=int, default=1,+ help='Take every N-th frame per view (default 1 = '+ 'full 30 FPS; 2 -> 15 FPS).')+ parser.add_argument('--max_frames_per_view', type=int, default=None,+ help='Cap on frames sent per view after '+ '--downsample_rate (0 = no cap). Default: '+ 'no cap in video mode (temporal patching '+ 'keeps it cheap), {} in images mode.'+ .format(DEFAULT_MAX_FRAMES_PER_VIEW))+ parser.add_argument('--max_tokens', type=int, default=1024,+ help='Max new tokens to generate.')+ parser.add_argument('--num_workers', type=int, default=1,+ help='Concurrent in-process workers for HTTP backends '+ '(gemini/openai). Qwen ignores this.')++ add_multi_gpu_args(parser)++ return parser, parser.parse_args()+++def main():+ parser, args = parse_args()++ # โโ Merge mode โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ if args.merge_shards:+ if not args.output_json:+ parser.error("--merge_shards requires --output_json")+ merge_shards(args.output_json, args.num_gpus)+ return++ if not args.render_root:+ parser.error("--render_root is required")++ # โโ Resolve backend โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ backend = args.backend or detect_backend(args.model)++ task = args.task+ logger.info("Backend: {} | Model: {} | Task: {}".format(+ backend, args.model, task))++ system_prompt = get_prompt(task)++ # โโ Multi-GPU is only for qwen (local model) โโโโโโโโโโโโโโโโโโโโโโโโโโโ+ num_gpus = args.num_gpus+ gpu_id = args.gpu_id+ if backend != BACKEND_QWEN and num_gpus > 1:+ logger.warning("Multi-GPU sharding is only supported for the qwen "+ "backend. Ignoring --num_gpus/--gpu_id.")+ num_gpus = 1+ gpu_id = 0++ # โโ Load qwen model (once) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ qwen_model = None+ qwen_processor = None++ if backend == BACKEND_QWEN:+ gpu_id_for_model = gpu_id if num_gpus > 1 else None+ qwen_model, qwen_processor = load_qwen_model(+ args.model,+ device_map=args.device_map,+ torch_dtype=args.torch_dtype,+ min_pixels=args.min_pixels,+ max_pixels=args.max_pixels,+ gpu_id=gpu_id_for_model,+ )++ # โโ Run โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+ caption_batch(+ render_root=args.render_root,+ backend=backend,+ system_prompt=system_prompt,+ output_json=args.output_json,+ downsample_rate=args.downsample_rate,+ max_frames_per_view=args.max_frames_per_view,+ max_tokens=args.max_tokens,+ views=args.views,+ hints=load_hints(args.hints_json),+ vision_input=args.vision_input,+ gpu_id=gpu_id, num_gpus=num_gpus,+ num_workers=args.num_workers,+ qwen_model=qwen_model,+ qwen_processor=qwen_processor,+ model_name=args.model,+ api_key=args.api_key,+ base_url=args.base_url,+ max_retries=args.max_retries,+ thinking_level=args.thinking_level,+ media_resolution=args.media_resolution,+ thinking_budget=args.thinking_budget,+ )+++if __name__ == "__main__":+ main()diff --git a/data_process/vlm_caption/classify_category.py b/data_process/vlm_caption/classify_category.pynew file mode 100644index 0000000..c236663--- /dev/null+++ b/data_process/vlm_caption/classify_category.py@@ -0,0 +1,366 @@+"""Classify 3D assets into body-plan categories using a local Qwen3-VL model.++Input is a 2x2 multi-view grid PNG per asset (front, back, plus two side+views, T-pose). Two render layouts are supported and auto-detected:++ Flat (e.g. objaverse): <render_root>/<asset_id>.png+ Nested (e.g. truebones): <render_root>/<NAME-MOTION>/tpose_grid.png+ Object types are grouped by the segment before '-'; one+ classification per type is then propagated to every clip of that+ type.++Output:+ * ``category_groups.json`` โ {category: [asset_id, ...]} for assets+ that classified successfully.+ * ``category_groups_errors.json`` (sibling, same directory)+ โ {asset_id: error_message} for assets+ whose retries were exhausted. ERROR+ entries are NOT in category_groups.json+ so a re-run will retry them automatically;+ the error log persists the last reason.++Usage:+ # Objaverse (flat layout)+ python -m data_process.vlm_caption.classify_category \+ --render_root dataset/render/objaverse_tpose++ # Truebones (nested layout)+ python -m data_process.vlm_caption.classify_category \+ --render_root dataset/render/truebones_tpose+"""++import argparse+import time+from collections import defaultdict+from pathlib import Path++from loguru import logger+from PIL import Image+from tqdm import tqdm++from data_process.vlm_caption.prompts import CATEGORY_PROMPT+from data_process.utils.vlm import (+ add_qwen_model_args,+ flatten_rgba,+ load_json,+ load_qwen_model,+ qwen_generate,+ save_json,+)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Constants+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++VALID_CATEGORIES = [+ "bipedal", "quadrupedal", "insectoid", "avian", "marine",+ "serpentine", "articulated_rigid",+]+UNKNOWN_CATEGORY = "unknown" # fallback when the model output matches no valid category++# Per-asset retry policy: up to NUM_RETRIES additional attempts after the+# initial try (5 total). Retries cover both exceptions (image I/O, CUDA OOM,+# generation failure) and unparseable model outputs (UNKNOWN_CATEGORY) so+# stochastic decoding gets a second chance to land on a valid category.+NUM_RETRIES = 4+RETRY_BACKOFF_SECONDS = 2.0++SAVE_EVERY = 20 # assets per incremental JSON flush (both output files)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Core classification+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def classify_asset(+ tpose_image,+ asset_id,+ model,+ processor,+ max_tokens=10,+ system_prompt=CATEGORY_PROMPT,+ do_sample=False,+):+ # type: (Image.Image, str, object, object, int, str, bool) -> str+ """Classify one asset from its 2x2 T-pose grid PNG.++ Returns one of ``VALID_CATEGORIES`` or ``UNKNOWN_CATEGORY`` if the+ model output cannot be parsed into a known category.+ """+ messages = [+ {"role": "system", "content": [{"type": "text", "text": system_prompt}]},+ {"role": "user", "content": [+ {"type": "text", "text": (+ "Classify this 3D asset (id: '{}'). The image is a 2x2 grid "+ "of four canonical T-pose views.".format(asset_id)+ )},+ {"type": "image", "image": tpose_image},+ ]},+ ]++ raw = qwen_generate(model, processor, messages, max_tokens=max_tokens,+ do_sample=do_sample).lower()+ # Accept "articulated rigid" (space) as well as "articulated_rigid" (underscore).+ raw_normalized = raw.replace(" ", "_").strip()+ # Bare token after stripping quotes/punctuation โ the expected reply.+ bare = raw_normalized.strip("_*`'\".,;:!?()[]\n\t ")++ if bare in VALID_CATEGORIES:+ category = bare+ else:+ # Take the category that appears EARLIEST in the response, not the+ # first one in VALID_CATEGORIES order: "quadrupedal (not bipedal)"+ # must resolve to quadrupedal.+ hits = sorted((raw_normalized.find(v), v)+ for v in VALID_CATEGORIES if v in raw_normalized)+ if len(hits) > 1:+ logger.warning("Ambiguous category response '{}' for '{}': {} โ "+ "taking the earliest match '{}'".format(+ raw, asset_id,+ ", ".join(v for _, v in hits), hits[0][1]))+ category = hits[0][1] if hits else None+ if category is None:+ logger.warning("Unexpected category '{}' for '{}', mapping to '{}'".format(+ raw, asset_id, UNKNOWN_CATEGORY))+ category = UNKNOWN_CATEGORY++ logger.info("Category for '{}': {}".format(asset_id, category))+ return category+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Asset discovery + batch processing+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def _discover_assets(render_root):+ # type: (Path) -> Tuple[str, List[Tuple[str, Path]]]+ """Locate (asset_id, tpose_png) pairs under ``render_root``.++ Auto-detects the layout:+ * flat โ ``<render_root>/<asset_id>.png`` (one render per asset).+ * nested โ ``<render_root>/<NAME-MOTION>/tpose_grid.png`` (multi-clip;+ grouped by the segment before '-' so every object type is+ classified exactly once via its first available render).++ Returns ``(layout, items)`` where ``items`` is sorted by asset_id.+ """+ flat_pngs = sorted(p for p in render_root.glob("*.png") if p.is_file())+ if flat_pngs:+ items = [(p.stem, p) for p in flat_pngs]+ return "flat", items++ char_dirs = sorted(+ d for d in render_root.iterdir()+ if d.is_dir() and (d / "tpose_grid.png").exists()+ )+ type_to_png = {} # type: Dict[str, Path]+ for d in char_dirs:+ obj_type = d.name.split("-", 1)[0] if "-" in d.name else d.name+ type_to_png.setdefault(obj_type, d / "tpose_grid.png")+ items = sorted(type_to_png.items())+ return "nested", items+++def _error_log_path_for(category_groups_json):+ # type: (str) -> str+ """Sibling error-log path next to ``category_groups.json``."""+ p = Path(category_groups_json)+ return str(p.with_name(p.stem + "_errors.json"))+++def classify_batch(+ render_root,+ model,+ processor,+ category_groups_json=None,+ max_tokens=10,+ system_prompt=CATEGORY_PROMPT,+):+ """Classify every asset under ``render_root``.++ Outputs:+ * ``category_groups.json`` โ {category: [asset_id, ...]} for the+ assets that classified successfully.+ * ``<stem>_errors.json`` โ {asset_id: error_message} for assets+ whose retries were exhausted (siblings of the groups JSON).++ Both files are read on entry for resumability โ successful asset_ids+ are skipped, and stale error entries (now-successful assets) are+ pruned. Re-runs only re-attempt assets in the error log plus any+ assets the discovery pass found that aren't yet classified.+ """+ render_root = Path(render_root)++ if category_groups_json is None:+ category_groups_json = str(render_root / "category_groups.json")+ error_log_path = _error_log_path_for(category_groups_json)++ existing_groups = load_json(category_groups_json) # type: Dict[str, List[str]]+ already_classified = {} # type: Dict[str, str]+ for cat, members in existing_groups.items():+ for asset_id in members:+ already_classified[asset_id] = cat++ # Prune stale errors (assets now successfully classified) and retain+ # the rest as the running error log for this run.+ existing_errors = load_json(error_log_path) # type: Dict[str, str]+ asset_errors = {+ aid: err for aid, err in existing_errors.items()+ if aid not in already_classified+ } # type: Dict[str, str]++ layout, items = _discover_assets(render_root)+ logger.info("Layout: {} | Found {} assets in {}".format(+ layout, len(items), render_root))++ asset_categories = dict(already_classified) # type: Dict[str, str]+ to_classify = [(aid, p) for aid, p in items if aid not in already_classified]+ logger.info("{} already classified, {} remaining ({} previously errored)".format(+ len(already_classified), len(to_classify), len(asset_errors)))++ pbar = tqdm(to_classify, desc="Classifying", unit="asset",+ dynamic_ncols=True)+ total_attempts = NUM_RETRIES + 1+ n_unsaved = 0++ def _flush():+ """Write both output files (called every SAVE_EVERY assets)."""+ _save_category_groups(asset_categories, category_groups_json)+ save_json(asset_errors, error_log_path)++ try:+ for asset_id, png_path in pbar:+ pbar.set_postfix_str(asset_id, refresh=False)+ result = None+ last_error = None+ for attempt in range(1, total_attempts + 1):+ try:+ tpose_image = flatten_rgba(Image.open(str(png_path)))+ # Greedy first, sampled on retries โ a greedy retry would+ # deterministically repeat the same unparseable output.+ category = classify_asset(+ tpose_image, asset_id,+ model=model, processor=processor,+ max_tokens=max_tokens, system_prompt=system_prompt,+ do_sample=attempt > 1,+ )+ if category != UNKNOWN_CATEGORY:+ result = category+ break+ last_error = ValueError(+ "model output did not match any valid category")+ except Exception as e:+ last_error = e+ logger.warning("Attempt {}/{} failed for {}: {}".format(+ attempt, total_attempts, asset_id, e))++ if attempt < total_attempts:+ time.sleep(RETRY_BACKOFF_SECONDS)++ if result is not None:+ asset_categories[asset_id] = result+ asset_errors.pop(asset_id, None) # success clears any prior error+ else:+ # Exhausted retries on either exceptions or unparseable output.+ # Mark as ERROR so _save_category_groups drops it from the JSON+ # and a later re-run will retry from scratch. Record the reason+ # in the sibling error log for visibility.+ if isinstance(last_error, ValueError):+ err_msg = "All {} attempts produced unparseable output".format(+ total_attempts)+ else:+ err_msg = "All {} attempts failed: {}".format(+ total_attempts, last_error)+ logger.error("[{}] {}".format(asset_id, err_msg))+ asset_categories[asset_id] = "ERROR: {}".format(last_error)+ asset_errors[asset_id] = err_msg++ # Rewriting both JSONs per asset is O(n^2) over a large run;+ # batch it (a crash loses at most SAVE_EVERY classifications).+ n_unsaved += 1+ if n_unsaved >= SAVE_EVERY:+ _flush()+ n_unsaved = 0+ finally:+ _flush()+ n_ok = sum(+ 1 for c in asset_categories.values()+ if isinstance(c, str) and not c.startswith("ERROR:")+ )+ logger.info("Done. {} classified, {} errored (logged to {})".format(+ n_ok, len(asset_errors), error_log_path))+ return asset_categories+++def _save_category_groups(asset_categories, output_path):+ # type: (Dict[str, str], str) -> None+ """Invert {asset_id: category} into {category: [asset_id, ...]} and save.++ ERROR entries are dropped so a re-run will retry those assets. After+ retries are exhausted (in classify_batch) the asset is marked ERROR+ and recorded in the sibling error log instead of leaking into the+ groups JSON.+ """+ assets_by_category = defaultdict(list) # type: Dict[str, List[str]]+ for asset_id, cat in sorted(asset_categories.items()):+ if isinstance(cat, str) and cat.startswith("ERROR:"):+ continue+ assets_by_category[cat].append(asset_id)++ category_assets = {+ cat: sorted(assets) for cat, assets in sorted(assets_by_category.items())+ } # type: Dict[str, List[str]]+ save_json(category_assets, output_path)+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# CLI+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++def parse_args():+ parser = argparse.ArgumentParser(+ description="Classify 3D assets into body-plan categories "+ "using a local Qwen3-VL model."+ )++ parser.add_argument('--render_root', type=str, required=True,+ help='Root directory holding T-pose 2x2 grid PNGs '+ '(flat <asset_id>.png or nested '+ '<NAME-MOTION>/tpose_grid.png โ auto-detected).')++ parser.add_argument('--category_groups_json', type=str, default=None,+ help='Output JSON file mapping '+ '{category: [asset_id, ...]}. '+ 'Defaults to <render_root>/category_groups.json. '+ 'A sibling <stem>_errors.json records assets '+ 'whose retries were exhausted. Both files are '+ 're-read on entry for resumability.')+ add_qwen_model_args(parser)+ parser.add_argument('--max_tokens', type=int, default=10,+ help='Max new tokens to generate (default: 10).')++ return parser.parse_args()+++def main():+ args = parse_args()++ qwen_model, qwen_processor = load_qwen_model(+ args.model,+ device_map=args.device_map,+ torch_dtype=args.torch_dtype,+ min_pixels=args.min_pixels,+ max_pixels=args.max_pixels,+ )++ classify_batch(+ render_root=args.render_root,+ model=qwen_model, processor=qwen_processor,+ category_groups_json=args.category_groups_json,+ max_tokens=args.max_tokens,+ )+++if __name__ == "__main__":+ main()diff --git a/data_process/vlm_caption/prompts.py b/data_process/vlm_caption/prompts.pynew file mode 100644index 0000000..91aa04c--- /dev/null+++ b/data_process/vlm_caption/prompts.py@@ -0,0 +1,397 @@+"""System prompts for the VLM captioning stage.++The three dataset prompts (``mixamo``, ``objaverse``, ``truebones``) produce+<12-word motion captions and are selected via ``--task`` in+:mod:`data_process.vlm_caption.caption_motion`. ``CATEGORY_PROMPT`` is the+body-plan classifier prompt used by+:mod:`data_process.vlm_caption.classify_category`.++The caption prompts share one scaffold โ camera-input explanation,+observation protocol steps 2โ5, the core caption rules, and the closing+line โ assembled by :func:`_caption_prompt` so a protocol change lands in+every dataset at once. Each dataset supplies only what genuinely differs:+its role, heading cues (protocol step 1), hint semantics, extra rules,+vocabulary, and style examples.+"""++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Shared caption-prompt scaffold+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++_INPUT_CAMERAS = (+ "# Input\n"+ "Four synchronized cameras 90 degrees apart at fixed elevation, labeled "+ "only by their relative azimuth about the vertical axis (0ยฐ, 90ยฐ, 180ยฐ, "+ "270ยฐ; 0ยฐ and 180ยฐ are opposite cameras, as are 90ยฐ and 270ยฐ). The "+ "labels carry NO information about {heading} โ the subject's world "+ "orientation is arbitrary, so any camera may be seeing the front, back, "+ "side, or an oblique angle. Determine facing from the body itself "+ "(protocol step 1). The cameras are STATIC: when the subject shifts "+ "across the frame or grows/shrinks, that is the subject translating, "+ "never camera motion. A view looking straight along an elongated body "+ "may show only a compact silhouette โ rely on the other views. The "+ "frames of each view are uniformly sampled across the clip in "+ "chronological order, and the views are synchronized: frame k of every "+ "view shows the same moment. Pose can change noticeably between "+ "consecutive sampled frames.\n\n"+)++_PROTOCOL_HEADER = "# Observation protocol (silent โ write only the caption)\n"++# Steps 2โ5 are identical logic everywhere; step 1 (heading cues) is the+# genuinely dataset-specific part and is supplied by each spec.+_PROTOCOL_STEPS_2_TO_5 = (+ "2. CHIRALITY. From the front, {possessive} left side is on the viewer's "+ "right (mirror). Apply consistently โ never label a limb by which side of "+ "the camera frame it sits on.\n"+ "3. SCAN every frame across all four views; do not infer from first/last "+ "alone. Note which body parts change pose and how.\n"+ "4. CLASSIFY: translation (moves through space), rotation in place, "+ "articulation only ({articulation}), or held pose. Direction is read "+ "relative to the heading from step 1. Say 'in place' only for rotation "+ "without translation, or for locomotion-style movement without "+ "translation ({in_place_example}); never as a default. Compare the "+ "subject's facing at the START and END of the clip: if it differs, a "+ "turn happened and belongs in the caption ('turns around and walks "+ "away').\n"+ "5. PICK the most specific verb that fits the kinematics.\n\n"+)++_RULES_CORE = (+ "# Rules\n"+ "- Format: '{subject} [action].' (one sentence, <12 words)\n"+ "- ONE dominant action โ or a short two-phase sequence ('X and then Y') "+ "when the clip clearly has two stages.\n"+ "- A cyclic motion (walk cycle, idle sway) is described ONCE as a "+ "continuous action, never per repetition; reserve the two-phase form "+ "for genuinely distinct stages.\n"+ "- A concurrent pose or secondary movement may be attached with "+ "'while ...' / 'with ...' ({concurrent_examples}).\n"+ "- Mention direction (forward/backward/left/right/up/down/in place) only "+ "if clearly observed; omit rather than guess.\n"+ "- All directions and chirality are object-relative (step 1), never "+ "camera-relative. {uncertain_fallback}\n"+)++_RULES_TAIL = (+ "- No adverbs, no appearance{no_appearance_extra}.\n"+ "- Only say 'stands still' if the pose is truly unchanged across ALL "+ "frames; {still_clause}.\n\n"+)++_EXAMPLES_HEADER = (+ "# Style examples (format only โ do NOT copy unless the motion matches)\n"+)++_CLOSING = ("Respond with ONLY the caption sentence โ plain text, no "+ "quotation marks.")+++def _caption_prompt(role, heading, blocks_after_input, step1, possessive,+ articulation, in_place_example, vocab_block, subject,+ uncertain_fallback, concurrent_examples, specific_rules,+ no_appearance_extra, still_clause, examples):+ # type: (...) -> str+ """Assemble one dataset's caption system prompt from the shared scaffold."""+ return (+ "# Role\n" + role + "\n\n"+ + _INPUT_CAMERAS.format(heading=heading)+ + blocks_after_input+ + _PROTOCOL_HEADER+ + step1+ + _PROTOCOL_STEPS_2_TO_5.format(+ possessive=possessive, articulation=articulation,+ in_place_example=in_place_example)+ + vocab_block+ + _RULES_CORE.format(+ subject=subject, uncertain_fallback=uncertain_fallback,+ concurrent_examples=concurrent_examples)+ + specific_rules+ + _RULES_TAIL.format(+ no_appearance_extra=no_appearance_extra, still_clause=still_clause)+ + _EXAMPLES_HEADER+ + examples + "\n"+ + _CLOSING+ )+++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Mixamo: concise human motion captioning+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++MIXAMO_PROMPT = _caption_prompt(+ role=(+ "You caption clips from the Mixamo 3D human animation dataset "+ "(locomotion, combat, sports, dance, gestures, idle poses). Every "+ "clip is a single human character, rendered as a faceless "+ "untextured mannequin."+ ),+ heading="the character's facing direction",+ blocks_after_input=(+ "A reference label from the Mixamo catalogue may be provided in "+ "the text after the views (e.g., '180 Turn With Briefcase', "+ "'Jab Cross'). Treat it as "+ "a helpful HINT, not ground truth: describe what the frames show, "+ "and use the label to disambiguate when the frames are unclear or "+ "to pick a more specific verb for an action you can verify. Do not "+ "copy it verbatim, and never carry over an element the frames do "+ "not support. Props named in the label are not rendered โ never "+ "name them; when the pose clearly implies one, express it as "+ "'as if holding/carrying something'.\n\n"+ ),+ step1=(+ "1. ESTABLISH HEADING. The mannequin has no face or eyes โ read "+ "heading from body geometry: toes point forward, knees bend "+ "forward while elbows bend backward, and the chest is convex where "+ "the back is flatter. Whichever camera the chest and toes most "+ "directly point at shows the person's FRONT; the opposite shows "+ "the BACK; the other two show their own LEFT and RIGHT.\n"+ ),+ possessive="the person's",+ articulation="limbs move, pelvis fixed",+ in_place_example="'runs in place', 'marches in place'",+ vocab_block="",+ subject="A person",+ uncertain_fallback=(+ "If heading is uncertain, say 'one arm' / 'one leg' instead of "+ "guessing left/right."+ ),+ concurrent_examples=("'walks forward with arms raised', "+ "'claps while seated'"),+ specific_rules=(+ "- Mention a body part only when essential to disambiguate.\n"+ ),+ no_appearance_extra=", no clothing",+ still_clause="subtle sway/breathing still counts as motion",+ examples=(+ "- A person walks forward.\n"+ "- A person runs in place.\n"+ "- A person punches with the right fist.\n"+ "- A person raises one arm while seated.\n"+ "- A person turns around as if carrying something.\n"+ "- A person crouches and then rises back up.\n"+ ),+)++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Objaverse: concise topology-agnostic captioning+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++OBJAVERSE_PROMPT = _caption_prompt(+ role=(+ "You caption clips from the Objaverse 3D animation dataset โ "+ "humanoids (majority), quadrupeds, avians, marine creatures, "+ "insectoids, serpentines, and articulated rigid objects."+ ),+ heading="the asset's canonical heading",+ blocks_after_input=(+ "# Task\n"+ "ONE short sentence describing the dominant motion in "+ "OBJECT-RELATIVE terms. Subject is 'An object', body parts use "+ "natural anatomy words (arm, wing, tail, ...). When topology is "+ "ambiguous, default to humanoid vocabulary โ most assets in this "+ "dataset are humanoid.\n\n"+ ),+ step1=(+ "1. ESTABLISH HEADING. Cues, in priority order: (a) "+ "head/face/eye/snout direction, (b) spine direction shouldersโhips "+ "for quadrupeds, (c) beak/head for avians, (d) head vs. tail end for "+ "serpentines, (e) principal translation axis for faceless rigid "+ "assets.\n"+ ),+ possessive="the object's",+ articulation="limbs/wings/tail move, body fixed",+ in_place_example="'walks in place'",+ vocab_block=(+ "# Body-part vocabulary (descriptions of shape, not category labels)\n"+ "- Humanoid/bipedal: arm, hand, leg, foot, torso, head, hip, shoulder\n"+ "- Quadruped: front leg, hind leg, head, torso, tail\n"+ "- Winged/flying: wing, head, torso, tail, leg (if visible)\n"+ "- Serpentine: head, body, tail\n"+ "- Aquatic: fin, tail, head, body\n"+ "- Insectoid: leg, body, head\n"+ "- Articulated rigid: part, segment, base, top, arm (mechanical)\n"+ "If shape fits no row: upper/lower part, left/right side, front, "+ "back.\n\n"+ ),+ subject="An object",+ uncertain_fallback="If heading is uncertain, omit chirality.",+ concurrent_examples=("'walks forward with arms swinging', "+ "'rotates its base while extending an arm'"),+ specific_rules=(+ "- Mention a body part only when essential to disambiguate.\n"+ "- No category names ('person', 'dog', 'dragon', 'car', 'robot', "+ "...); anatomy words (arm, wing, tail) are allowed.\n"+ ),+ no_appearance_extra="/color/material/texture, no scene/lighting",+ still_clause="subtle sway, breathing, or limb shifts still count as motion",+ examples=(+ "- An object walks forward with arms swinging.\n"+ "- An object kicks with the right leg while pivoting.\n"+ "- An object flaps its wings and rises.\n"+ "- An object slithers forward.\n"+ "- An object rotates its upper segment in place.\n"+ "- An object crouches down and springs upward.\n"+ ),+)++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Truebones: concise animal motion captioning+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++TRUEBONES_PROMPT = _caption_prompt(+ role=(+ "You caption clips from the Truebones 3D animal animation dataset โ "+ "quadrupeds, avians, marine creatures, insectoids, serpentines. "+ "Every clip is a single non-human animal."+ ),+ heading="the animal's heading",+ blocks_after_input=(+ "A reference label may be provided in the text after the views โ "+ "a human annotation that reliably names the motion (e.g., 'A "+ "running tyrannosaurus "+ "rex loses balance and falls to the right.'). Its ACTION is "+ "authoritative: make it the caption's MAIN ACTION and override "+ "it only when the frames unambiguously show a different one. Its "+ "DIRECTIONS and left/right are not โ the annotator's viewpoint "+ "is unknown, so re-derive every direction from step 1 and keep "+ "the label's directional words only when the frames confirm "+ "them. Do not copy the label "+ "verbatim: rewrite it in the required format and never carry a "+ "species word into the caption โ the example label above becomes "+ "'An animal runs forward and then falls to the right.'\n\n"+ ),+ step1=(+ "1. ESTABLISH HEADING. Cues, in priority order: (a) "+ "head/snout/eyes/beak direction, (b) spine direction "+ "shoulders/withersโhips for quadrupeds and winged animals, (c) head "+ "end vs. tail end for serpentines/fish, (d) principal direction of "+ "locomotion (animals move forward along their facing axis).\n"+ ),+ possessive="the animal's",+ articulation="limbs/wings/tail/jaw move, torso fixed",+ in_place_example="'trots in place'",+ vocab_block="",+ subject="An animal",+ uncertain_fallback=(+ "If heading is uncertain, say 'one foreleg' / 'a wing' instead of "+ "guessing left/right."+ ),+ concurrent_examples=("'walks forward with its head turned right', "+ "'crouches with its tail raised'"),+ specific_rules=(+ "- Use generic body-part terms: front legs, hind legs, head, torso, "+ "tail, wings, fins, jaw, left/right side. Mention only when "+ "essential.\n"+ "- Do NOT identify the species in the caption (no 'dog', 'horse', "+ "'bear', 'snake') even if the reference label mentions one.\n"+ ),+ no_appearance_extra="/color/texture",+ still_clause="subtle sway, breathing, tail flicks, ear twitches still count as motion",+ examples=(+ "- An animal gallops forward.\n"+ "- An animal turns to the right.\n"+ "- An animal rears upward on its hind legs.\n"+ "- An animal flaps its wings and rises.\n"+ "- An animal swims forward with its tail.\n"+ "- An animal strikes forward with its tail curled.\n"+ ),+)++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Category: body-type classification+# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++CATEGORY_PROMPT = (+ "# Role\n"+ "You classify the body plan of a single 3D asset into exactly ONE of "+ "seven categories. The output is consumed by an automated pipeline โ "+ "it must be one exact token, no prose, no punctuation.\n\n"+ "# Input\n"+ "ONE image: a 2x2 grid of four canonical views of the asset in T-pose "+ "(front, back, and two orthogonal side / top views). The exact panel "+ "ordering is not guaranteed. Use ALL four panels โ never decide from a "+ "single panel โ and reason from silhouette, limb count, and proportions. "+ "T-pose means a canonical rest pose with limbs spread; there is no "+ "motion to read. If the asset appears horizontal in some panels, that "+ "is just a top-down or side view of the same rest pose โ judge body "+ "plan from the panel that most clearly shows the limb topology.\n\n"+ "# Categories\n"+ "- bipedal: organism with TWO legs as primary stance / locomotion. "+ "Humans, humanoids, apes, bipedal dinosaurs (T-Rex, raptors), "+ "flightless upright birds (ostrich, penguin), humanoid robots and "+ "mechs (head + torso + two arms + two legs).\n"+ "- quadrupedal: organism with FOUR legs as primary stance. Dogs, "+ "cats, horses, bears, deer, lizards, turtles, four-legged dinosaurs, "+ "quadruped robots. Centaurs and four-leg hybrids belong here.\n"+ "- insectoid: arthropod body plan โ SIX OR MORE articulated legs, "+ "typically segmented thorax/abdomen, often antennae. Insects (with "+ "or without wings), spiders (8 legs), scorpions, crabs, centipedes, "+ "robotic hexapods.\n"+ "- avian: vertebrate defined by TWO LARGE WINGS used for flight, "+ "usually paired with legs. Flying birds, bats, pterosaurs, dragons, "+ "wyverns, winged humanoids / angels. Wings must be a dominant "+ "feature, not vestigial decoration.\n"+ "- marine: streamlined aquatic body propelled by fins/flippers/tail, "+ "with NO walking legs. Fish, sharks, rays, dolphins, whales, "+ "octopuses, seahorses. Mermaids belong here.\n"+ "- serpentine: elongated, LIMBLESS, tubular body. Snakes, eels, "+ "worms, legless lizards. Any visible legs disqualify this category.\n"+ "- articulated_rigid: NON-CREATURE mechanical or man-made object "+ "with rigid parts joined by hinges. Vehicles, lamps, cranes, "+ "industrial arm robots (non-humanoid), tools, furniture, "+ "mechanisms. Use this ONLY when the asset clearly does not read as "+ "a creature.\n\n"+ "# Decision priority (apply in order; stop at the first match)\n"+ "1. Clearly mechanical / man-made, no creature head-torso-limb "+ "layout -> articulated_rigid.\n"+ "2. Six or more legs OR arthropod silhouette (segmented body, "+ "exoskeleton plates, antennae) -> insectoid. THIS OVERRIDES WINGS โ "+ "a bee or butterfly is insectoid, not avian.\n"+ "3. Two large flight wings paired with the torso, wings are a "+ "dominant feature -> avian.\n"+ "4. No walking limbs, body shaped for swimming with fins/tail "+ "-> marine.\n"+ "5. No limbs at all, long tubular body -> serpentine.\n"+ "6. Four legs as primary ground stance -> quadrupedal.\n"+ "7. Two legs as primary ground stance -> bipedal.\n"+ "8. Anything still ambiguous -> articulated_rigid.\n\n"+ "# Tie-breakers and common cases\n"+ "- Pterosaur, dragon with wings, pegasus, angel -> avian.\n"+ "- Wingless dragon with four legs -> quadrupedal.\n"+ "- Wyvern (two legs + two wings) -> avian.\n"+ "- Penguin, ostrich, T-Rex, raptor -> bipedal.\n"+ "- Bee, butterfly, dragonfly, ladybug -> insectoid.\n"+ "- Centaur -> quadrupedal. Mermaid -> marine.\n"+ "- Spider robot / hexapod robot -> insectoid.\n"+ "- Humanoid robot / mech with two arms and two legs -> bipedal.\n"+ "- Quadruped robot (Spot-style, four legs) -> quadrupedal.\n"+ "- Industrial robot arm, crane, lamp, loose mechanism -> "+ "articulated_rigid.\n"+ "- Lizard with small limbs (gecko, salamander) -> quadrupedal.\n"+ "- Snake, eel, worm -> serpentine.\n\n"+ "# Output\n"+ "Reply with EXACTLY one of these tokens, lowercase, no punctuation, "+ "no explanation, no quotes:\n"+ "bipedal | quadrupedal | insectoid | avian | marine | serpentine | "+ "articulated_rigid"+)++# โโ Caption task registry โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+TASK_PROMPTS = {+ "mixamo": MIXAMO_PROMPT,+ "objaverse": OBJAVERSE_PROMPT,+ "truebones": TRUEBONES_PROMPT,+}++def get_prompt(task):+ # type: (str) -> str+ """Return the caption system prompt for *task*; raises ValueError if unknown."""+ if task not in TASK_PROMPTS:+ raise ValueError(+ "Unknown task '{}'. Available: {}".format(+ task, ", ".join(sorted(TASK_PROMPTS.keys()))+ )+ )+ return TASK_PROMPTS[task]+diff --git a/requirements.txt b/requirements.txtnew file mode 100644index 0000000..f08c08e--- /dev/null+++ b/requirements.txt@@ -0,0 +1,71 @@+# UniMate unified environment โ one env for everything: data processing,+# rendering, training, inference and evaluation (Python 3.10, CUDA 12.4).+#+# conda create -n unimate python=3.10 -y+# conda activate unimate+# pip install "setuptools<81"+# pip install -r requirements.txt --no-build-isolation+#+# The Motion git dependency has a legacy setup.py that imports pkg_resources,+# which setuptools >= 81 no longer ships. Isolated builds always use the+# newest setuptools, so the install must run with --no-build-isolation+# against an env setuptools < 81 โ installed by the step above (fresh conda+# envs ship setuptools >= 81 too); the pin below keeps it from upgrading.+#+# Only direct dependencies are listed; everything else (CUDA runtime libs,+# tokenizers, HTTP stack, ...) is resolved transitively by pip.++# PyTorch cu124 wheels (torch / torchvision "+cu124" pins below)+--extra-index-url https://download.pytorch.org/whl/cu124+# Blender official wheel index โ PyPI no longer hosts cp310 bpy wheels+--extra-index-url https://download.blender.org/pypi/++setuptools<81 # keeps pkg_resources available (Motion build)++# โโ Core / training โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+numpy==2.2.6+torch==2.5.1+cu124+torchvision==0.20.1+cu124+torch-geometric==2.7.0 # GCN graph embedding (use_graph_emb)+torchdiffeq==0.2.5 # ODE/SDE integrators for flow matching+einops==0.8.2+accelerate==0.30.1 # multi-GPU + mixed-precision training+tensorboard==2.20.0 # training logs (SummaryWriter)++# โโ Text conditioning (T5 / CLIP / BERT) & VLM captioning โโโโโโโโโโโโโโโโโโโโ+transformers==4.57.6 # T5/BERT text encoders; >=4.57 required by the local Qwen3-VL captioner+sentencepiece==0.2.1 # T5 tokenizer backend+clip @ git+https://github.com/openai/CLIP.git # CLIP text encoder+spacy # T5 conditioner text normalization+num2words # number โ word expansion in text normalization+openai # OpenAI-compatible backends (captioning; DeepSeek/GPT joint annotation)+google-genai # Gemini caption backend ("from google import genai")++# OPTIONAL โ fused attention for the local Qwen3-VL captioner. The loader+# falls back to SDPA when absent, and the prebuilt wheels only exist for a+# few exact torch/CUDA/Python combinations (a source build takes an hour+),+# so it is not installed by default:+# pip install flash-attn==2.7.4.post1 --no-build-isolation++# โโ Motion / animation library (Holden-style Animation / Quaternions /+# InverseKinematics; used across data_process/) โโโโโโโโโโโโโโโโโโโโโโโโโโโ+Motion @ git+https://github.com/inbar-2344/Motion.git++# โโ Blender as a Python module (motion_export/, motion_rendering/,+# mesh_animation/). The EEVEE render stage requires the pip module;+# export/animate can alternatively run via `blender -b -P`. Pinned to the+# last cp310 release, served from download.blender.org (see index above);+# 4.1+ drops Python 3.10 and 4.2+ changes the EEVEE API. โโโโโโโโโโโโโโโโโโ+bpy==4.0.0++# โโ Visualization / imaging โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+matplotlib==3.10.8+imageio==2.37.3+imageio-ffmpeg==0.6.0 # mp4 encoding backend for imageio+pillow==12.2.0++# โโ CLI / logging โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+tyro==1.0.12 # config + CLI overrides+rich==14.3.3+loguru+tqdm==4.67.3