[SIGGRAPH Asia 2026] UniMate: One Unified Model to Animate Diverse Skeletons
Python
Add rig_preprocess for custom rigs with LLM labelling, a review step and verification, example assets and README inference guides; align the uniml3d configs with the v3 recipe and fix data-pipeline, annotation and sampling issues
Friedrich-M2f605cd47634
diff --git a/README.md b/README.mdindex 77292e5..ca368af 100644--- a/README.md+++ b/README.md@@ -30,14 +30,13 @@## ๐ฅ News+- **[2026-10-05]** **Bring your own rig**: [`rig_preprocess`](data_process/rig_preprocess/) turns any rigged GLB / glTF / FBX, animated or not, into an asset UniMate can animate, with a review step for its joint labels and facing; the sampler takes dataset skeletons and your own assets by reference. [UniML3D](https://huggingface.co/datasets/Linzhan/UniML3D) adds training-ready features, canonical rest-pose meshes and generic / detailed captions. ๐ ๏ธ- **[2026-09-27]** Preview checkpoints are released at [HuggingFace](https://huggingface.co/Linzhan/UniMate); new checkpoints will be synced there. ๐ฆ- **[2026-09-06]** The **training and inference code** is released. ๐- **[2026-08-30]** The [UniML3D dataset](https://huggingface.co/collections/Linzhan/unimate) and its [data-processing pipeline](data_process/) are released. ๐- **[2026-08-01]** Our [Interactive Demo](https://linzhanmou.com/unimate/interactive.html) is live โ browse our animation results in 3D. ๐ฎ- **[2026-07-18]** UniMate is accepted to SIGGRAPH Asia 2026! ๐-> **[TODO]** We will release an official preprocessing pipeline for new (out-of-distribution) rigs by this week.-## ๐ ๏ธ Environment SetupAll components share a single conda environment, specified in [`requirements.txt`](requirements.txt):@@ -51,19 +50,24 @@ pip install -r requirements.txt --no-build-isolation## ๐ Dataset & Data Processing-We introduce **UniML3D**, a large-scale dataset of 13,006 text-paired motion sequences covering diverse skeletal topologies โ bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects โ all brought into a unified canonicalization.+We introduce **UniML3D**, a large-scale dataset of 13,769 text-paired motion clips on 7,430 skeletons covering diverse skeletal topologies โ bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects โ all brought into a unified canonicalization.-The raw source assets are available on the Hugging Face Hub (collected under [UniMate](https://huggingface.co/collections/Linzhan/unimate)): [Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters), [Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) and [Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata and renders only). The Truebones ZOO animal motions themselves are a commercial asset pack whose license does not permit redistribution โ please purchase the pack directly from [Truebones](https://truebones.com); our pipeline consumes the stock `Truebone_Z-OO` folder layout as-is.+The raw source assets are available on the Hugging Face Hub (collected under [UniMate](https://huggingface.co/collections/Linzhan/unimate)): [Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters), [Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) and [Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata and renders only). The Truebones ZOO animal motions themselves are a commercial asset pack whose license does not permit redistribution โ please purchase the pack directly from [Truebones](https://truebones.com) and rebuild the per-clip layout with the build scripts in Truebones-ZOO-Annotations.<div align="center"><img src="assets/dataset_overview.png" alt="UniML3D dataset overview" width="100%"></div>-See [`data_process/README.md`](data_process/README.md) for the full data processing pipeline that turns the raw assets into UniML3D (download โ export โ rendering โ captioning โ joint annotation โ feature extraction โ animation).+The processed dataset is on the Hub as [UniML3D](https://huggingface.co/datasets/Linzhan/UniML3D): the exported clips with all captions and annotations, the training-ready features, and canonical rest-pose meshes (everything except the Truebones motions and meshes). See [`data_process/README.md`](data_process/README.md) for the full data processing pipeline that turns the raw assets into UniML3D (download โ export โ rendering โ captioning โ joint annotation โ feature extraction โ animation).## ๐๏ธ Training-Training reads the canonicalized clips under `dataset/features/<dataset>/`, produced by stage 4 of the [data-processing pipeline](data_process/README.md#stage-4--extract-npz--metadata--training-clips).+Training reads the canonicalized clips under `dataset/features/<dataset>/`. Download them, or build them with stage 4 of the [data-processing pipeline](data_process/README.md#stage-4-extract) (the Truebones clips must be built from the purchased pack):++```bash+hf download Linzhan/UniML3D --repo-type dataset --local-dir dataset \+ --include "features/*" --exclude "features/*/videos/*" --exclude "features/*/tpose/*"+```Runs are configured by the JSON files in [`configs/`](configs/).@@ -88,7 +92,9 @@ Runs are configured by the JSON files in [`configs/`](configs/).The two axes are independent and all four combinations are implemented, so `full` x `adaln` and `graph` x `cross_attn` also run if you set them in a config; the two shipped pairings are the ones the paper compares.-Not shared across configs: `training.batch_size`, `training.num_steps`, `model.num_layers` and `dataset.max_joints` are tuned per data combination (GPU memory tracks batch x frames x joints; depth grows with the data: 6 layers for a single Truebones / Mixamo source, 8 for Objaverse, 10 for the full mixture). `dataset.max_joints` is `100` for `truebones_*` / `mixamo_*` and `60` for `objaverse_*` / `uniml3d_*`; `dataset.min_joints` is `5` everywhere. Together they bound the skeleton sizes a run admits (object types outside the range are dropped) and, through what survives, the joint-axis padding width. Mixamo is a single skeleton, so its configs also turn off the object-type balancing (a no-op with one type) and, by choice, the topology augmentations. Every other setting is identical.+Tuned per data combination: `training.batch_size` (16 / 32 / 24 / 16 for Truebones / Mixamo / Objaverse / UniML3D), `training.num_steps` (80k / 120k / 100k / 120k), `model.num_layers` (6 / 6 / 8 / 10) and `dataset.max_joints` (100 / 100 / 60 / 70; `dataset.min_joints` is `5` everywhere). The joint range bounds the skeleton sizes a run admits (object types outside it are dropped) and, through what survives, the joint-axis padding width. Mixamo is a single skeleton, so its configs also turn off the object-type balancing (a no-op with one type) and, by choice, the topology augmentations.++The `uniml3d_*` configs train the recommended model's recipe: fixed per-dataset sampling weights (`dataset.sampler_dataset_weights`), one normalization pool across datasets (`dataset.use_dataset_stats: false`), the generic and detailed captions drawn with probability 0.35 and 0.25 besides the normal one, and logit-normal flow time (`training.t_sampling`). The single-source configs keep per-dataset statistics, the normal caption and uniform flow time.</details>@@ -104,9 +110,12 @@ Multi-GPU on one node (e.g. 8 GPUs):accelerate launch --num_processes 8 -m unimate.training.train --config configs/uniml3d_60frames_graph_adaln.json```+<details>+<summary><b>Options and run outputs</b></summary>+`scripts/run_train.sh <config> [-- extra args]` wraps the single-GPU command with the conda environment activated, the GPU with the most free memory selected, and anything after `--` forwarded to the training module.-`--output_dir`, `--batch_size`, `--num_workers` and `--resume <checkpoint.pt>` override the config from the command line. Resuming restores model, EMA, optimizer, LR-scheduler and step counter, so a run continues exactly where it stopped.+`--output_dir`, `--batch_size`, `--num_workers`, `--resume <checkpoint.pt>` and `--stats_path <dataset_stats.npy>` override the config from the command line. Resuming restores model, EMA, optimizer, LR-scheduler and step counter, and reuses the run's `dataset_stats.npy`, so a run continues exactly where it stopped. `--stats_path` loads normalization statistics from a file instead of computing them, e.g. to fine-tune a model on new data in the normalization it was trained with.Each run writes to `outputs/<experiment name>/`:@@ -118,10 +127,12 @@ Each run writes to `outputs/<experiment name>/`:| `debug/` | Sample visualizations, rendered once before training and at every checkpoint (EMA weights, `sampling.cfg_scale`) || `logs/` | TensorBoard scalars (`tensorboard --logdir outputs/<experiment name>/logs`) |+</details>+<details><summary><b>What a training step does</b></summary>-Clips are drawn by a power-law-balanced sampler when `training.balanced` is set โ a type with `n` clips is sampled in proportion to `n^(1-sampler_alpha)`, so at the default `sampler_alpha = 0.5` a species with 100 clips is seen ten times as often as one with a single clip rather than a hundred times โ then augmented on the fly โ joint addition, leaf removal, chain pooling and per-bone length perturbation (`dataset.use_*_aug`) โ so the model sees more topologies than the data literally contains. Every clip is padded to `max_joints` on the joint axis and `max_motion_length` on the time axis, with masks carried alongside; nothing padded ever contributes to attention or to the loss.+Clips are drawn by a power-law-balanced sampler when `training.balanced` is set โ a type with `n` clips is sampled in proportion to `n^(1-sampler_alpha)`, so at the default `sampler_alpha = 0.5` a species with 100 clips is seen ten times as often as one with a single clip rather than a hundred times; with `dataset.sampler_dataset_weights` each dataset first gets its fixed share โ then augmented on the fly โ joint addition, leaf removal, chain pooling and per-bone length perturbation (`dataset.use_*_aug`) โ so the model sees more topologies than the data literally contains. Every clip is padded to `max_joints` on the joint axis and `max_motion_length` on the time axis, with masks carried alongside; nothing padded ever contributes to attention or to the loss.Training is **flow matching** (`training.diff_model = "flow"`): the network predicts the velocity of a linear interpolant between noise and data, under a masked L2 loss plus two auxiliary terms computed on the reconstructed clean motion โ a geodesic rotation loss (`training.lambda_geo`) and a velocity-smoothness loss (`training.lambda_smooth`). Conditioning is dropped with probability `model.cond_mask_prob` so the same weights serve the conditional and unconditional branches that classifier-free guidance interpolates at sampling time. AdamW with a cosine schedule and warmup, gradient clipping at `training.max_grad_norm`, and an EMA copy of the weights (`training.use_ema`) โ the copy inference loads by default.@@ -136,7 +147,7 @@ The text encoder (`google/flan-t5-base` by default) is fetched from the Huggingpython -m unimate.tools.precompute_text_emb --config configs/uniml3d_60frames_graph_adaln.json```-This writes `caption_emb_cache.npz` and `joint_emb_cache.npz` into each `dataset/features/<dataset>/` the config uses. Captions are cached per token (the sequence `cross_attn` attends; `adaln` mean-pools it), joint names as one pooled vector each, keyed by the **cleaned** joint vocabulary that stage 3 produces โ the shared naming is what lets the same anatomical joint embed identically across rigs. Re-run it after regenerating captions or joint names: anything the cache misses is still encoded at load time, so a stale cache costs speed rather than correctness.+This writes `caption_emb_cache.npz` (and `caption_generic_emb_cache.npz` / `caption_detail_emb_cache.npz` for the other two captions) and `joint_emb_cache.npz` into each `dataset/features/<dataset>/` the config uses. Captions are cached per token (the sequence `cross_attn` attends; `adaln` mean-pools it), joint names as one pooled vector each, keyed by the **cleaned** joint vocabulary that stage 3 produces โ the shared naming is what lets the same anatomical joint embed identically across rigs. Re-run it after regenerating captions or joint names: anything the cache misses is still encoded at load time, so a stale cache costs speed rather than correctness.</details>@@ -145,13 +156,13 @@ This writes `caption_emb_cache.npz` and `joint_emb_cache.npz` into each `datasetA non-trivial share of the Objaverse-XL rigs and clips are defective: rest poses that lie flat, are rotated or are inverted, and clips that stitch several unrelated actions together. Training on them can destabilize or collapse a run, and isolated spikes in the training loss are usually a symptom of bad data rather than of optimization.-To localize the problem, first train on Mixamo and Truebones alone โ set `dataset.dataset_list` to `["truebones", "mixamo"]` in a copy of a config. If that run is healthy, the fault is on the Objaverse side. From there, inspect the skeleton preview videos of the suspect object types under `dataset/features/objaverse/videos/`, by eye or with an automated pass, and add the offending rigs and clips to the stage-4 skip lists that [`tools/patch_annotations.py`](data_process/README.md#stage-3--joint-annotation-name-cleanup--facing-direction) maintains.+To localize the problem, first train on Mixamo and Truebones alone โ set `dataset.dataset_list` to `["truebones", "mixamo"]` in a copy of a config. If that run is healthy, the fault is on the Objaverse side. From there, inspect the skeleton preview videos of the suspect object types under `dataset/features/objaverse/videos/`, by eye or with an automated pass, and add the offending rigs and clips to the stage-4 skip lists that [`tools/patch_annotations.py`](data_process/README.md#stage-3-joint-annotation) maintains.</details>## ๐ Note-The processed **UniML3D** dataset is being prepared for open release. Its captions were **re-processed** for this release, so they do not necessarily match the prompts shown on the project page or in the paper. See the released caption style in [Truebones](https://huggingface.co/datasets/Linzhan/UniML3D/blob/main/export/truebones/motion_captions.json) ยท [Mixamo](https://huggingface.co/datasets/Linzhan/UniML3D/blob/main/export/mixamo/motion_captions.json) ยท [Objaverse](https://huggingface.co/datasets/Linzhan/UniML3D/blob/main/export/objaverse/motion_captions.json). New prompts start with "An object" to generalize across objects.+The processed **UniML3D** dataset is [released](https://huggingface.co/datasets/Linzhan/UniML3D). Its captions were **re-processed** for this release, so they do not necessarily match the prompts shown on the project page or in the paper. See the released caption style in [Truebones](https://huggingface.co/datasets/Linzhan/UniML3D/blob/main/export/truebones/motion_captions.json) ยท [Mixamo](https://huggingface.co/datasets/Linzhan/UniML3D/blob/main/export/mixamo/motion_captions.json) ยท [Objaverse](https://huggingface.co/datasets/Linzhan/UniML3D/blob/main/export/objaverse/motion_captions.json). New prompts start with "An object" to generalize across objects.UniMate is an **early step** toward text-to-animation for any skeleton, and many motions and skeletons **still fail**. We believe that scaling up training data โ **distilled from agents or generated from videos** โ is a promising direction to close this gap. If you run into failure cases, please open an issue or contact us; they help us improve.@@ -159,69 +170,67 @@ UniMate is an **early step** toward text-to-animation for any skeleton, and manyGiven a rigged 3D asset and a text prompt, UniMate generates articulated motion for arbitrary skeletons in real time โ with no per-skeleton retraining and no test-time optimization.-<div align="center">- <img src="assets/qualitative.png" alt="Qualitative results" width="100%">-</div>--Sampling starts from the output directory of a training run (`config.json`, `dataset_stats.npy`, `checkpoints/`) โ the [released checkpoints](https://huggingface.co/Linzhan/UniMate) use the same layout. The target skeleton โ T-pose and topology conditioning โ is taken from the dataset, so the `dataset/features/<dataset>/` directory the model was trained on must be present.+**1. Get a model.** Download a [released checkpoint](https://huggingface.co/Linzhan/UniMate) (configuration, normalization statistics, latest checkpoint), or use the output directory of your own training run:```bash-python -m unimate.inference.sample \- --exp_dir outputs/uniml3d_60frames_graph_adaln \- --test_cases_json test_cases.json \- --num_repetitions 3+hf download Linzhan/UniMate --repo-type model --local-dir outputs \+ --include "unimate_uniml3d_f60_v3_preview/*.json" --include "unimate_uniml3d_f60_v3_preview/*.npy" \+ --include "unimate_uniml3d_f60_v3_preview/checkpoints/checkpoint_step_100000.pt"```-<details>-<summary><b>Test cases, flags and outputs</b></summary>--Test cases are a JSON map from `<object_type>-<case_id>` to a prompt. `object_type` must exist in the dataset; `case_id` is a free-form tag that names the output files:+**2a. Animate a dataset skeleton.** Download [UniML3D](https://huggingface.co/datasets/Linzhan/UniML3D) into `dataset/` (โ 88 GB; its [dataset card](https://huggingface.co/datasets/Linzhan/UniML3D#download) shows smaller partial downloads):-```json-{- "Dog-walk": "a dog walks forward at a steady pace",- "Dragon-takeoff": "a dragon flaps its wings and takes off"-}+```bash+hf download Linzhan/UniML3D --repo-type dataset --local-dir dataset```-`--test_cases_json` is itself optional: without it every clip of the dataset's eval split is enumerated as a test case (falling back to unique `(object_type, caption)` pairs on train when there is no eval split). `--test_cases_txt` (one `object_type` per line) drives unconditional sampling, which requires `--cfg_scale 1.0`.--| Flag | Effect |-|------|--------|-| `--cfg_scale` | Classifier-free guidance scale (>= 1.0); defaults to the value saved in the run's config |-| `--model_path` | A specific checkpoint; defaults to the latest step |-| `--output_dir` | Defaults to `<exp_dir>/samples` |-| `--num_repetitions` | Samples generated per test case |-| `--batch_size` | Per-chunk inference batch; caps GPU memory regardless of how many cases there are |-| `--seed` | Fixes the sampling noise |-| `--only_save_motion` | Skip the MP4 renders, write only the `.npy` features |-| `--save_ric` | Add the RIC-recovered render beside the FK one |--Each run writes:+Then name skeletons as `<dataset>:<object_type>` (`truebones:Horse`, `mixamo`, `objaverse:<uid>`). For example, Garfield, a Gundam, Baymax and a flower from Objaverse-XL, the Mixamo humanoid and a Truebones reindeer ([`assets/examples/examples.json`](assets/examples/examples.json)):-| Path | Content |-|------|---------|-| `motions/<case_id>-rep_<r>-<i>.npy` | Generated motion features `(T, J, 12)`, one file per repetition |-| `animations/<case_id>-rep_<r>-sample<i>_fk.mp4` | Skeleton render of each sample (`_ric.mp4` variants with `--save_ric`) |-| `animations/<object_type>_tpos.png` | The conditioning T-pose |-| `captions.json` | Prompt used for every saved `.npy` |+<details>+<summary><code>assets/examples/examples.json</code></summary>-`scripts/run_sample_motion_text.sh <exp_dir> [test_cases_json] [cfg_scale]` runs the command above with the conda environment activated, the GPU with the most free memory selected, and an output directory named after the test-case file. Run it with `-h` for its options.+```json+[{"asset": "objaverse:01fcb4e4c36548ca86624b63dfc6b255", "id": "garfield-dance", "prompt": "An object dances in place."},+ {"asset": "objaverse:01fcb4e4c36548ca86624b63dfc6b255", "id": "garfield-fall", "prompt": "An object falls forward."},+ {"asset": "objaverse:42eb6e70ce024c4c9b6ab24118c737b3", "id": "gundam-kick", "prompt": "An object kicks with the right leg."},+ {"asset": "objaverse:42eb6e70ce024c4c9b6ab24118c737b3", "id": "gundam-jump", "prompt": "An object jumps in place."},+ {"asset": "objaverse:e87c7d42f0b3475c9a698da8295244d1", "id": "baymax-walk", "prompt": "An object walks forward."},+ {"asset": "objaverse:e87c7d42f0b3475c9a698da8295244d1", "id": "baymax-punch", "prompt": "An object punches forward with its right arm."},+ {"asset": "objaverse:669a74dd57bbceb71e6e8b1d_fbx", "id": "flower-close", "prompt": "An object closes its petals and bends its stem."},+ {"asset": "mixamo", "id": "mixamo-run", "prompt": "An object runs forward."},+ {"asset": "mixamo", "id": "mixamo-cheer", "prompt": "An object cheers with its arms raised."},+ {"asset": "truebones:Raindeer", "id": "raindeer-walk", "prompt": "An object walks in place."},+ {"asset": "truebones:Raindeer", "id": "raindeer-attack", "prompt": "An object attacks forward."}]+```</details>-<details>-<summary><b>Driving a rigged mesh</b></summary>+```bash+python -m unimate.inference.sample --exp_dir outputs/unimate_uniml3d_f60_v3_preview \+ --test_cases_json assets/examples/examples.json --num_repetitions 3 --output_dir outputs/samples/examples+# drive each skeleton's canonical mesh: one animated GLB + FBX per motion+# (Truebones meshes are not distributed; SKIP_INVALID=1 skips motions without a mesh)+SKIP_INVALID=1 bash scripts/run_animate_motion.sh outputs/samples/examples+```-To animate the original mesh with a generated motion, hand the `.npy` files to stage 5 of the data pipeline, which exports an animated GLB + FBX:+`--asset <refs> --prompt <prompts>` samples every prompt on every asset without a file. Motions are written to `--output_dir` (default `<exp_dir>/samples`); runs into the same directory add to it, and `run_animate_motion.sh <dir>` drives them all. Training captions, good prompts to start from, are in `dataset/features/<dataset>/captions*.json`. Mesh driving checks first that motion, conditioning and mesh share one joint order.++**2b. Animate your own rigged asset.** [`rig_preprocess`](data_process/rig_preprocess/README.md) turns a rigged GLB / glTF / FBX, animated or not, into an asset directory that `--asset` takes. It labels the joints with an LLM and stops so you can review the labels and the facing pair before building, since the model is conditioned on both; its [guide](data_process/rig_preprocess/README.md) walks through the review and every option. Three assets processed this way are in [`assets/examples`](assets/examples): a Unitree Go2 quadruped, an eagle and a shark:```bash-bash scripts/run_animate_motion.sh objaverse \- outputs/uniml3d_60frames_graph_adaln/samples/motions/Dog-walk-rep_0-0.npy \- outputs/animated+python -m unimate.inference.sample --exp_dir outputs/unimate_uniml3d_f60_v3_preview --asset assets/examples/unitree_go2 \+ --prompt "An object trots forward." "An object rears up on its hind legs." --output_dir outputs/samples/custom+python -m unimate.inference.sample --exp_dir outputs/unimate_uniml3d_f60_v3_preview --asset assets/examples/eagle \+ --prompt "An object flaps its wings." "An object strikes forward." --output_dir outputs/samples/custom+python -m unimate.inference.sample --exp_dir outputs/unimate_uniml3d_f60_v3_preview --asset assets/examples/shark \+ --prompt "An object swims and turns around." "An object bites forward." --output_dir outputs/samples/custom+bash scripts/run_animate_motion.sh outputs/samples/custom # each asset's mesh, animated```-It accepts several files or a directory, and reads the rig from `dataset/features/<dataset>/cond.npy` by default.+<details>+<summary><b>Prompting</b></summary>++Prompts matter. Write short captions in the style of the training captions (`dataset/features/<dataset>/captions*.json`): start with "An object" and describe one motion, not the character ("An object flaps its wings.", not "A blue parrot flaps its wings in the sky."). The closer a prompt is to that style, the better it is followed. Samples vary, so generate several (`--num_repetitions`) and try a few phrasings. `--cfg_scale` (default 3) sets how strongly the prompt is followed: raise it (5 to 7) when a prompt is ignored, lower it for smoother, calmer motion.</details>@@ -243,12 +252,12 @@ Hold chosen keyframes at their ground truth and generate the transitions between`--keep_frames` takes signed indices (negatives count back from the generation window), so `"0,-1"` fills in everything between a clip's first and last pose.```json-{ "mixamo-Squat-000": "A human squats and then rises back up" }+{ "mixamo-Air_Squat-000": "An object squats and then stands up." }``````bashKEEP_FRAMES="0,-1" bash scripts/run_sample_motion_inbetween.sh \- outputs/uniml3d_60frames_graph_adaln cases.json+ outputs/unimate_uniml3d_f60_v3_preview cases.json```</details>@@ -267,12 +276,12 @@ Hold chosen joints at their ground-truth motion for every frame and regenerate t`--keep_joints` matches case-insensitively against either the rig's own bone names or the cleaned vocabulary.```json-{ "<objaverse_uid>-turn-head-000": "The robot walks forward." }+{ "mixamo-Cocky_Head_Turn-000": "An object walks forward." }``````bash-KEEP_JOINTS="Hips,Spine,Neck,Head" bash scripts/run_sample_motion_edit.sh \- outputs/uniml3d_60frames_graph_adaln cases.json+KEEP_JOINTS="Neck,Head" bash scripts/run_sample_motion_edit.sh \+ outputs/unimate_uniml3d_f60_v3_preview cases.json```</details>@@ -291,12 +300,12 @@ Chain several prompts into one long motion. The first segment is generated freelTest-case values become *lists* of prompts, one per segment; `--expand_overlap` sets how many frames consecutive segments share.```json-{ "mixamo-sequence": ["A human stands up.", "A human walks forward.", "A human turns around in place."] }+{ "mixamo-sequence": ["An object stands up.", "An object walks forward.", "An object turns around in place."] }``````bashEXPAND_OVERLAP=10 bash scripts/run_sample_motion_expand.sh \- outputs/uniml3d_60frames_graph_adaln cases.json+ outputs/unimate_uniml3d_f60_v3_preview cases.json```</details>@@ -306,7 +315,7 @@ EXPAND_OVERLAP=10 bash scripts/run_sample_motion_expand.sh \Each mode writes into its own subdirectory of `--output_dir` (`inbetween/`, `motion_edit/`, `motion_expand/`) alongside a small JSON recording the constraint that produced it, and each has a wrapper in `scripts/` โ run any of them with `-h` for the full option list.-In-betweening and editing clamp against a real clip, so their test-case keys must be `<object_type>-<clip_id>` naming a clip the dataset actually holds; that clip's motion is saved beside the result as `<case_id>-gt_rep_<r>-<i>.npy` for side-by-side comparison. `--gt_start_frame` pins which window of the clip is used instead of a random one. Editing trims both the sample and the GT to the clip's true length, while in-betweening generates the full window and trims only the GT โ so align the two on frame 0 rather than assuming equal lengths. All three modes need `--cfg_scale > 1.0` and are mutually exclusive with each other.+In-betweening and editing clamp against a real clip, so their test-case keys must be `<object_type>-<clip_id>` naming a clip on disk: a dataset clip under `dataset/features/<dataset>/motions/`, or one of a `rig_preprocess` asset saved with `--save_clips`; that clip's motion is saved beside the result as `<case_id>-gt_rep_<r>-<i>.npy` for side-by-side comparison. `--gt_start_frame` pins which window of the clip is used instead of a random one. Editing trims both the sample and the GT to the clip's true length, while in-betweening generates the full window and trims only the GT โ so align the two on frame 0 rather than assuming equal lengths. All three modes need `--cfg_scale > 1.0` and are mutually exclusive with each other.</details>@@ -331,4 +340,4 @@ The datasets remain governed by the licenses of their original sources: the [Mix## ๐ค Acknowledgement-We thank the authors of [AnyTop](https://github.com/Anytop2025/Anytop) for open-sourcing their codebase, on which parts of this repository build.+We thank the authors of [AnyTop](https://github.com/Anytop2025/Anytop) for open-sourcing their codebase, on which parts of this repository build. We thank [@chutch1122](https://github.com/chutch1122) for suggesting several captions per clip, which led to the normal, generic and detailed captions of UniML3D.diff --git a/assets/examples/eagle/Eagle.glb b/assets/examples/eagle/Eagle.glbnew file mode 100644index 0000000..8998a82Binary files /dev/null and b/assets/examples/eagle/Eagle.glb differdiff --git a/assets/examples/eagle/annotation.json b/assets/examples/eagle/annotation.jsonnew file mode 100644index 0000000..2ec21ff--- /dev/null+++ b/assets/examples/eagle/annotation.json@@ -0,0 +1,634 @@+{+ "format": "rig_preprocess-annotation/1",+ "asset": "Eagle",+ "notes": [],+ "face_pair": {+ "right": "Bip01_R_Thigh",+ "left": "Bip01_L_Thigh",+ "body_axis": false,+ "source": "thigh",+ "check": false+ },+ "face_pair_alternatives": [+ [+ "Bip01_R_Thigh",+ "Bip01_L_Thigh",+ "Thigh"+ ],+ [+ "BN_Wing_R_01",+ "BN_Wing_L_01",+ "Wing"+ ],+ [+ "Bip01_R_Calf",+ "Bip01_L_Calf",+ "Shin"+ ],+ [+ "Bip01_R_Foot",+ "Bip01_L_Foot",+ "Foot"+ ],+ [+ "Bip01_R_HorseLink",+ "Bip01_L_HorseLink",+ "Fetlock"+ ],+ [+ "BN_Toe_R_30",+ "BN_Toe_L_30",+ "Toe"+ ],+ [+ "BN_Tai_R_01",+ "BN_Tai_L_01",+ "Tail"+ ]+ ],+ "n_flagged": 0,+ "joints": [+ {+ "index": 0,+ "raw": "Bip01_Pelvis",+ "parent": null,+ "label": "Hips",+ "source": "template",+ "check": false+ },+ {+ "index": 1,+ "raw": "Bip01_Spine",+ "parent": "Bip01_Pelvis",+ "label": "Spine",+ "source": "template",+ "check": false+ },+ {+ "index": 2,+ "raw": "BN_Tai_01",+ "parent": "Bip01_Pelvis",+ "label": "Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 3,+ "raw": "Bip01_Spine1",+ "parent": "Bip01_Spine",+ "label": "Spine",+ "source": "template",+ "check": false+ },+ {+ "index": 4,+ "raw": "Bip01_L_Thigh",+ "parent": "Bip01_Spine",+ "label": "Left Thigh",+ "source": "template",+ "check": false+ },+ {+ "index": 5,+ "raw": "Bip01_R_Thigh",+ "parent": "Bip01_Spine",+ "label": "Right Thigh",+ "source": "template",+ "check": false+ },+ {+ "index": 6,+ "raw": "BN_Tai_L_01",+ "parent": "BN_Tai_01",+ "label": "Left Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 7,+ "raw": "BN_Tai_R_01",+ "parent": "BN_Tai_01",+ "label": "Right Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 8,+ "raw": "BN_Wing_R_01",+ "parent": "Bip01_Spine1",+ "label": "Right Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 9,+ "raw": "BN_Wing_L_01",+ "parent": "Bip01_Spine1",+ "label": "Left Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 10,+ "raw": "Bip01_Neck",+ "parent": "Bip01_Spine1",+ "label": "Neck",+ "source": "template",+ "check": false+ },+ {+ "index": 11,+ "raw": "Bip01_L_Calf",+ "parent": "Bip01_L_Thigh",+ "label": "Left Shin",+ "source": "template",+ "check": false+ },+ {+ "index": 12,+ "raw": "Bip01_R_Calf",+ "parent": "Bip01_R_Thigh",+ "label": "Right Shin",+ "source": "template",+ "check": false+ },+ {+ "index": 13,+ "raw": "BN_Tai_L_02",+ "parent": "BN_Tai_L_01",+ "label": "Left Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 14,+ "raw": "BN_Tai_R_02",+ "parent": "BN_Tai_R_01",+ "label": "Right Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 15,+ "raw": "BN_Wing_R_02",+ "parent": "BN_Wing_R_01",+ "label": "Right Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 16,+ "raw": "BN_Wing_L_02",+ "parent": "BN_Wing_L_01",+ "label": "Left Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 17,+ "raw": "Bip01_Neck1",+ "parent": "Bip01_Neck",+ "label": "Neck",+ "source": "template",+ "check": false+ },+ {+ "index": 18,+ "raw": "Bip01_L_HorseLink",+ "parent": "Bip01_L_Calf",+ "label": "Left Fetlock",+ "source": "template",+ "check": false+ },+ {+ "index": 19,+ "raw": "Bip01_R_HorseLink",+ "parent": "Bip01_R_Calf",+ "label": "Right Fetlock",+ "source": "template",+ "check": false+ },+ {+ "index": 20,+ "raw": "BN_Wing_R_03",+ "parent": "BN_Wing_R_02",+ "label": "Right Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 21,+ "raw": "BN_Wing_L_03",+ "parent": "BN_Wing_L_02",+ "label": "Left Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 22,+ "raw": "Bip01_Neck2",+ "parent": "Bip01_Neck1",+ "label": "Neck",+ "source": "template",+ "check": false+ },+ {+ "index": 23,+ "raw": "Bip01_L_Foot",+ "parent": "Bip01_L_HorseLink",+ "label": "Left Foot",+ "source": "template",+ "check": false+ },+ {+ "index": 24,+ "raw": "Bip01_R_Foot",+ "parent": "Bip01_R_HorseLink",+ "label": "Right Foot",+ "source": "template",+ "check": false+ },+ {+ "index": 25,+ "raw": "BN_Wing_R_05",+ "parent": "BN_Wing_R_03",+ "label": "Right Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 26,+ "raw": "BN_Wing_R_04",+ "parent": "BN_Wing_R_03",+ "label": "Right Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 27,+ "raw": "BN_Wing_L_05",+ "parent": "BN_Wing_L_03",+ "label": "Left Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 28,+ "raw": "BN_Wing_L_04",+ "parent": "BN_Wing_L_03",+ "label": "Left Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 29,+ "raw": "Bip01_Head",+ "parent": "Bip01_Neck2",+ "label": "Head",+ "source": "template",+ "check": false+ },+ {+ "index": 30,+ "raw": "BN_Toe_L_30",+ "parent": "Bip01_L_Foot",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 31,+ "raw": "BN_Toe_L_10",+ "parent": "Bip01_L_Foot",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 32,+ "raw": "BN_Toe_L_20",+ "parent": "Bip01_L_Foot",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 33,+ "raw": "BN_Toe_L_00",+ "parent": "Bip01_L_Foot",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 34,+ "raw": "Bip01_L_Toe0",+ "parent": "Bip01_L_Foot",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 35,+ "raw": "BN_Toe_R_30",+ "parent": "Bip01_R_Foot",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 36,+ "raw": "BN_Toe_R_10",+ "parent": "Bip01_R_Foot",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 37,+ "raw": "BN_Toe_R_20",+ "parent": "Bip01_R_Foot",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 38,+ "raw": "BN_Toe_R_00",+ "parent": "Bip01_R_Foot",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 39,+ "raw": "Bip01_R_Toe0",+ "parent": "Bip01_R_Foot",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 40,+ "raw": "BN_Wing_R_06",+ "parent": "BN_Wing_R_05",+ "label": "Right Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 41,+ "raw": "BN_Wing_L_06",+ "parent": "BN_Wing_L_05",+ "label": "Left Wing",+ "source": "template",+ "check": false+ },+ {+ "index": 42,+ "raw": "BN_Jaw",+ "parent": "Bip01_Head",+ "label": "Jaw",+ "source": "template",+ "check": false+ },+ {+ "index": 43,+ "raw": "BN_Toe_L_31",+ "parent": "BN_Toe_L_30",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 44,+ "raw": "BN_Toe_L_11",+ "parent": "BN_Toe_L_10",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 45,+ "raw": "BN_Toe_L_21",+ "parent": "BN_Toe_L_20",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 46,+ "raw": "BN_Toe_L_01",+ "parent": "BN_Toe_L_00",+ "label": "Left Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 47,+ "raw": "BN_Toe_R_31",+ "parent": "BN_Toe_R_30",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 48,+ "raw": "BN_Toe_R_11",+ "parent": "BN_Toe_R_10",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 49,+ "raw": "BN_Toe_R_21",+ "parent": "BN_Toe_R_20",+ "label": "Right Toe",+ "source": "template",+ "check": false+ },+ {+ "index": 50,+ "raw": "BN_Toe_R_01",+ "parent": "BN_Toe_R_00",+ "label": "Right Toe",+ "source": "template",+ "check": false+ }+ ],+ "vocabulary": {+ "format": "[Left |Right ][Qualifier ]Part[ End], e.g. 'Left Front Leg', 'Right Index Finger End'. Qualifier: Front, Back, Middle, Rear, Hind, Inner, Outer, Upper, Lower. No numbers: every joint of a chain takes the same label (Spine, Spine, Spine; Tail, Tail), its last joint 'End' ('Tail End'). 'Bone' only for a joint that is no body part.",+ "parts": [+ "Abdomen",+ "Anal Fin",+ "Ankle",+ "Antenna",+ "Appendage",+ "Arm",+ "Back Claw",+ "Back Hip",+ "Back Paw",+ "Bag",+ "Barbel",+ "Base",+ "Belly",+ "Belt",+ "Body",+ "Bone",+ "Branch",+ "Cannon",+ "Cape",+ "Caudal Fin",+ "Center",+ "Chain",+ "Cheek",+ "Chest",+ "Chin",+ "Claw",+ "Cloth",+ "Collar",+ "Crest",+ "Door",+ "Dorsal Fin",+ "Dorsal Plate",+ "Ear",+ "Elbow",+ "Elbow Back",+ "End",+ "Eye",+ "Eyeball",+ "Eyebrow",+ "Eyelid",+ "Fang",+ "Feather",+ "Fetlock",+ "Fin",+ "Finger",+ "Foot",+ "Forearm",+ "Forearm Twist",+ "Front Hoof",+ "Front Leg",+ "Front Leg End",+ "Front Paw",+ "Front Shoulder",+ "Fur",+ "Gill",+ "Hair",+ "Halter",+ "Hand",+ "Hand Claw",+ "Handle",+ "Hat",+ "Head",+ "Head End",+ "Heel",+ "Hind Leg",+ "Hind Leg End",+ "Hip",+ "Hips",+ "Hoof",+ "Horn",+ "IK Chain",+ "Index Finger",+ "Inner Claw",+ "Inner Finger",+ "Inner Hand Claw",+ "Inner Toe",+ "Jaw",+ "Jiggle",+ "Knee",+ "Large Mandible",+ "Leaf",+ "Leg",+ "Lever",+ "Lid",+ "Lip",+ "Lower Body",+ "Lower Eyelid",+ "Lower Front Lip",+ "Lower Jaw",+ "Lower Left Lip",+ "Lower Lip",+ "Lower Mandible",+ "Lower Right Lip",+ "Mandible",+ "Mane",+ "Metacarpus",+ "Middle Claw",+ "Middle Finger",+ "Middle Hand Claw",+ "Middle Leg",+ "Middle Leg End",+ "Middle Toe",+ "Mouth",+ "Muzzle",+ "Neck",+ "Neck Muscle",+ "Nose",+ "Outer Claw",+ "Outer Finger",+ "Outer Hand Claw",+ "Outer Toe",+ "Palm",+ "Pastern",+ "Paw",+ "Pectoral Fin",+ "Pelvic Fin",+ "Pelvis",+ "Petal",+ "Phalanges",+ "Pincer",+ "Pinky Finger",+ "Prop",+ "Propeller",+ "Rear Hoof",+ "Reins",+ "Ribcage",+ "Ring Finger",+ "Root",+ "Rope",+ "Scapula",+ "Scarf",+ "Shell",+ "Shield",+ "Shin",+ "Shin Twist",+ "Shoulder",+ "Skirt",+ "Skull",+ "Skull Base",+ "Spine",+ "Stem",+ "Stinger",+ "Strap",+ "Tail",+ "Tail Twist",+ "Tentacle",+ "Thigh",+ "Thigh Muscle",+ "Thigh Twist",+ "Thumb Finger",+ "Toe",+ "Toe End",+ "Tongue",+ "Twist",+ "Upper Arm",+ "Upper Arm Twist",+ "Upper Body",+ "Upper Eyelid",+ "Upper Jaw",+ "Upper Left Lip",+ "Upper Lip",+ "Upper Right Lip",+ "Waist",+ "Weapon",+ "Wheel",+ "Whisker",+ "Wing",+ "Wrist",+ "Wrist Back"+ ]+ }+}\ No newline at end of filediff --git a/assets/examples/eagle/cond.npy b/assets/examples/eagle/cond.npynew file mode 100644index 0000000..73e04feBinary files /dev/null and b/assets/examples/eagle/cond.npy differdiff --git a/assets/examples/eagle/preview.png b/assets/examples/eagle/preview.pngnew file mode 100644index 0000000..28c59beBinary files /dev/null and b/assets/examples/eagle/preview.png differdiff --git a/assets/examples/eagle/summary.json b/assets/examples/eagle/summary.jsonnew file mode 100644index 0000000..3b47aec--- /dev/null+++ b/assets/examples/eagle/summary.json@@ -0,0 +1,43 @@+{+ "name": "Eagle",+ "profile": "general",+ "inputs": [+ "Eagle.glb"+ ],+ "annotate": "file",+ "annotation": "assets/examples/eagle/annotation.json",+ "animated": false,+ "n_joints_export": 51,+ "rig_glb": null,+ "stats_dataset": "general",+ "dropped_translation": {+ "max_share_of_extent": 0.0,+ "clip": null,+ "joint": null+ },+ "face_pair": {+ "r_hip": {+ "raw": "Bip01_R_Thigh",+ "clean": "Right Thigh"+ },+ "l_hip": {+ "raw": "Bip01_L_Thigh",+ "clean": "Left Thigh"+ },+ "source": "thigh"+ },+ "annotation_notes": [+ "annotation from assets/examples/eagle/annotation.json"+ ],+ "n_joints": 51,+ "model_joint_limit": {+ "max_joints": 71,+ "source": "released UniML3D models",+ "fits": true+ },+ "n_clips": 0,+ "cond": "assets/examples/eagle/cond.npy",+ "canonical_glb": "assets/examples/eagle/Eagle.glb",+ "notes": [],+ "preview": "assets/examples/eagle/preview.png"+}diff --git a/assets/examples/examples.json b/assets/examples/examples.jsonnew file mode 100644index 0000000..12fa065--- /dev/null+++ b/assets/examples/examples.json@@ -0,0 +1,57 @@+[+ {+ "asset": "objaverse:01fcb4e4c36548ca86624b63dfc6b255",+ "id": "garfield-dance",+ "prompt": "An object dances in place."+ },+ {+ "asset": "objaverse:01fcb4e4c36548ca86624b63dfc6b255",+ "id": "garfield-fall",+ "prompt": "An object falls forward."+ },+ {+ "asset": "objaverse:42eb6e70ce024c4c9b6ab24118c737b3",+ "id": "gundam-kick",+ "prompt": "An object kicks with the right leg."+ },+ {+ "asset": "objaverse:42eb6e70ce024c4c9b6ab24118c737b3",+ "id": "gundam-jump",+ "prompt": "An object jumps in place."+ },+ {+ "asset": "objaverse:e87c7d42f0b3475c9a698da8295244d1",+ "id": "baymax-walk",+ "prompt": "An object walks forward."+ },+ {+ "asset": "objaverse:e87c7d42f0b3475c9a698da8295244d1",+ "id": "baymax-punch",+ "prompt": "An object punches forward with its right arm."+ },+ {+ "asset": "objaverse:669a74dd57bbceb71e6e8b1d_fbx",+ "id": "flower-close",+ "prompt": "An object closes its petals and bends its stem."+ },+ {+ "asset": "mixamo",+ "id": "mixamo-run",+ "prompt": "An object runs forward."+ },+ {+ "asset": "mixamo",+ "id": "mixamo-cheer",+ "prompt": "An object cheers with its arms raised."+ },+ {+ "asset": "truebones:Raindeer",+ "id": "raindeer-walk",+ "prompt": "An object walks in place."+ },+ {+ "asset": "truebones:Raindeer",+ "id": "raindeer-attack",+ "prompt": "An object attacks forward."+ }+]diff --git a/assets/examples/shark/annotation.json b/assets/examples/shark/annotation.jsonnew file mode 100644index 0000000..3566a56--- /dev/null+++ b/assets/examples/shark/annotation.json@@ -0,0 +1,329 @@+{+ "format": "rig_preprocess-annotation/1",+ "asset": "shark",+ "notes": [],+ "face_pair": {+ "right": "_05",+ "left": "_04",+ "body_axis": false,+ "source": "pectoral fin",+ "check": false+ },+ "face_pair_alternatives": [+ [+ "_05",+ "_04",+ "Pectoral Fin"+ ],+ [+ "_11",+ "_10",+ "Pelvic Fin"+ ]+ ],+ "n_flagged": 0,+ "joints": [+ {+ "index": 0,+ "raw": "_00",+ "parent": null,+ "label": "Root",+ "source": "template",+ "check": false+ },+ {+ "index": 1,+ "raw": "_07",+ "parent": "_00",+ "label": "Spine",+ "source": "template",+ "check": false+ },+ {+ "index": 2,+ "raw": "_01",+ "parent": "_00",+ "label": "Spine",+ "source": "template",+ "check": false+ },+ {+ "index": 3,+ "raw": "_08",+ "parent": "_07",+ "label": "Spine",+ "source": "template",+ "check": false+ },+ {+ "index": 4,+ "raw": "_02",+ "parent": "_01",+ "label": "Head",+ "source": "template",+ "check": false+ },+ {+ "index": 5,+ "raw": "_05",+ "parent": "_01",+ "label": "Right Pectoral Fin",+ "source": "file",+ "check": false+ },+ {+ "index": 6,+ "raw": "_04",+ "parent": "_01",+ "label": "Left Pectoral Fin",+ "source": "file",+ "check": false+ },+ {+ "index": 7,+ "raw": "_06",+ "parent": "_01",+ "label": "Dorsal Fin",+ "source": "file",+ "check": false+ },+ {+ "index": 8,+ "raw": "_09",+ "parent": "_08",+ "label": "Spine",+ "source": "template",+ "check": false+ },+ {+ "index": 9,+ "raw": "_03",+ "parent": "_02",+ "label": "Jaw",+ "source": "template",+ "check": false+ },+ {+ "index": 10,+ "raw": "_12",+ "parent": "_09",+ "label": "Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 11,+ "raw": "_11",+ "parent": "_09",+ "label": "Right Pelvic Fin",+ "source": "file",+ "check": false+ },+ {+ "index": 12,+ "raw": "_10",+ "parent": "_09",+ "label": "Left Pelvic Fin",+ "source": "file",+ "check": false+ },+ {+ "index": 13,+ "raw": "_13",+ "parent": "_12",+ "label": "Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 14,+ "raw": "_14",+ "parent": "_13",+ "label": "Tail",+ "source": "template",+ "check": false+ },+ {+ "index": 15,+ "raw": "_15",+ "parent": "_14",+ "label": "Tail",+ "source": "template",+ "check": false+ }+ ],+ "vocabulary": {+ "format": "[Left |Right ][Qualifier ]Part[ End], e.g. 'Left Front Leg', 'Right Index Finger End'. Qualifier: Front, Back, Middle, Rear, Hind, Inner, Outer, Upper, Lower. No numbers: every joint of a chain takes the same label (Spine, Spine, Spine; Tail, Tail), its last joint 'End' ('Tail End'). 'Bone' only for a joint that is no body part.",+ "parts": [+ "Abdomen",+ "Anal Fin",+ "Ankle",+ "Antenna",+ "Appendage",+ "Arm",+ "Back Claw",+ "Back Hip",+ "Back Paw",+ "Bag",+ "Barbel",+ "Base",+ "Belly",+ "Belt",+ "Body",+ "Bone",+ "Branch",+ "Cannon",+ "Cape",+ "Caudal Fin",+ "Center",+ "Chain",+ "Cheek",+ "Chest",+ "Chin",+ "Claw",+ "Cloth",+ "Collar",+ "Crest",+ "Door",+ "Dorsal Fin",+ "Dorsal Plate",+ "Ear",+ "Elbow",+ "Elbow Back",+ "End",+ "Eye",+ "Eyeball",+ "Eyebrow",+ "Eyelid",+ "Fang",+ "Feather",+ "Fetlock",+ "Fin",+ "Finger",+ "Foot",+ "Forearm",+ "Forearm Twist",+ "Front Hoof",+ "Front Leg",+ "Front Leg End",+ "Front Paw",+ "Front Shoulder",+ "Fur",+ "Gill",+ "Hair",+ "Halter",+ "Hand",+ "Hand Claw",+ "Handle",+ "Hat",+ "Head",+ "Head End",+ "Heel",+ "Hind Leg",+ "Hind Leg End",+ "Hip",+ "Hips",+ "Hoof",+ "Horn",+ "IK Chain",+ "Index Finger",+ "Inner Claw",+ "Inner Finger",+ "Inner Hand Claw",+ "Inner Toe",+ "Jaw",+ "Jiggle",+ "Knee",+ "Large Mandible",+ "Leaf",+ "Leg",+ "Lever",+ "Lid",+ "Lip",+ "Lower Body",+ "Lower Eyelid",+ "Lower Front Lip",+ "Lower Jaw",+ "Lower Left Lip",+ "Lower Lip",+ "Lower Mandible",+ "Lower Right Lip",+ "Mandible",+ "Mane",+ "Metacarpus",+ "Middle Claw",+ "Middle Finger",+ "Middle Hand Claw",+ "Middle Leg",+ "Middle Leg End",+ "Middle Toe",+ "Mouth",+ "Muzzle",+ "Neck",+ "Neck Muscle",+ "Nose",+ "Outer Claw",+ "Outer Finger",+ "Outer Hand Claw",+ "Outer Toe",+ "Palm",+ "Pastern",+ "Paw",+ "Pectoral Fin",+ "Pelvic Fin",+ "Pelvis",+ "Petal",+ "Phalanges",+ "Pincer",+ "Pinky Finger",+ "Prop",+ "Propeller",+ "Rear Hoof",+ "Reins",+ "Ribcage",+ "Ring Finger",+ "Root",+ "Rope",+ "Scapula",+ "Scarf",+ "Shell",+ "Shield",+ "Shin",+ "Shin Twist",+ "Shoulder",+ "Skirt",+ "Skull",+ "Skull Base",+ "Spine",+ "Stem",+ "Stinger",+ "Strap",+ "Tail",+ "Tail Twist",+ "Tentacle",+ "Thigh",+ "Thigh Muscle",+ "Thigh Twist",+ "Thumb Finger",+ "Toe",+ "Toe End",+ "Tongue",+ "Twist",+ "Upper Arm",+ "Upper Arm Twist",+ "Upper Body",+ "Upper Eyelid",+ "Upper Jaw",+ "Upper Left Lip",+ "Upper Lip",+ "Upper Right Lip",+ "Waist",+ "Weapon",+ "Wheel",+ "Whisker",+ "Wing",+ "Wrist",+ "Wrist Back"+ ]+ }+}\ No newline at end of filediff --git a/assets/examples/shark/cond.npy b/assets/examples/shark/cond.npynew file mode 100644index 0000000..f76a196Binary files /dev/null and b/assets/examples/shark/cond.npy differdiff --git a/assets/examples/shark/preview.png b/assets/examples/shark/preview.pngnew file mode 100644index 0000000..b1fb75aBinary files /dev/null and b/assets/examples/shark/preview.png differdiff --git a/assets/examples/shark/shark.glb b/assets/examples/shark/shark.glbnew file mode 100644index 0000000..f688000Binary files /dev/null and b/assets/examples/shark/shark.glb differdiff --git a/assets/examples/shark/summary.json b/assets/examples/shark/summary.jsonnew file mode 100644index 0000000..ab13530--- /dev/null+++ b/assets/examples/shark/summary.json@@ -0,0 +1,43 @@+{+ "name": "shark",+ "profile": "general",+ "inputs": [+ "shark.glb"+ ],+ "annotate": "file",+ "annotation": "assets/examples/shark/annotation.json",+ "animated": false,+ "n_joints_export": 16,+ "rig_glb": null,+ "stats_dataset": "general",+ "dropped_translation": {+ "max_share_of_extent": 0.0,+ "clip": null,+ "joint": null+ },+ "face_pair": {+ "r_hip": {+ "raw": "_05",+ "clean": "Right Pectoral Fin"+ },+ "l_hip": {+ "raw": "_04",+ "clean": "Left Pectoral Fin"+ },+ "source": "pectoral fin"+ },+ "annotation_notes": [+ "annotation from assets/examples/shark/annotation.json"+ ],+ "n_joints": 16,+ "model_joint_limit": {+ "max_joints": 71,+ "source": "released UniML3D models",+ "fits": true+ },+ "n_clips": 0,+ "cond": "assets/examples/shark/cond.npy",+ "canonical_glb": "assets/examples/shark/shark.glb",+ "notes": [],+ "preview": "assets/examples/shark/preview.png"+}diff --git a/assets/examples/unitree_go2/annotation.json b/assets/examples/unitree_go2/annotation.jsonnew file mode 100644index 0000000..a89946f--- /dev/null+++ b/assets/examples/unitree_go2/annotation.json@@ -0,0 +1,378 @@+{+ "format": "rig_preprocess-annotation/1",+ "asset": "unitree_go2",+ "notes": [],+ "face_pair": {+ "right": "FR_hip",+ "left": "FL_hip",+ "body_axis": false,+ "source": "front hip",+ "check": true,+ "reasons": [+ "chosen from 'front hip' joints, not hip or shoulder level: prefer the thighs or shoulders if the asset has them"+ ]+ },+ "face_pair_alternatives": [+ [+ "RR_hip",+ "RL_hip",+ "Back Hip"+ ],+ [+ "RR_foot",+ "RL_foot",+ "Back Foot"+ ],+ [+ "RR_calf",+ "RL_calf",+ "Back Shin"+ ],+ [+ "RR_thigh",+ "RL_thigh",+ "Back Thigh"+ ],+ [+ "FR_foot",+ "FL_foot",+ "Front Foot"+ ],+ [+ "FR_hip",+ "FL_hip",+ "Front Hip"+ ],+ [+ "FR_calf",+ "FL_calf",+ "Front Shin"+ ],+ [+ "FR_thigh",+ "FL_thigh",+ "Front Thigh"+ ]+ ],+ "n_flagged": 0,+ "joints": [+ {+ "index": 0,+ "raw": "base",+ "parent": null,+ "label": "Body",+ "source": "file",+ "check": false+ },+ {+ "index": 1,+ "raw": "FR_hip",+ "parent": "base",+ "label": "Right Front Hip",+ "source": "file",+ "check": false+ },+ {+ "index": 2,+ "raw": "RL_hip",+ "parent": "base",+ "label": "Left Back Hip",+ "source": "file",+ "check": false+ },+ {+ "index": 3,+ "raw": "FL_hip",+ "parent": "base",+ "label": "Left Front Hip",+ "source": "file",+ "check": false+ },+ {+ "index": 4,+ "raw": "RR_hip",+ "parent": "base",+ "label": "Right Back Hip",+ "source": "file",+ "check": false+ },+ {+ "index": 5,+ "raw": "Head_upper",+ "parent": "base",+ "label": "Head",+ "source": "file",+ "check": false+ },+ {+ "index": 6,+ "raw": "FR_thigh",+ "parent": "FR_hip",+ "label": "Right Front Thigh",+ "source": "file",+ "check": false+ },+ {+ "index": 7,+ "raw": "RL_thigh",+ "parent": "RL_hip",+ "label": "Left Back Thigh",+ "source": "file",+ "check": false+ },+ {+ "index": 8,+ "raw": "FL_thigh",+ "parent": "FL_hip",+ "label": "Left Front Thigh",+ "source": "file",+ "check": false+ },+ {+ "index": 9,+ "raw": "RR_thigh",+ "parent": "RR_hip",+ "label": "Right Back Thigh",+ "source": "file",+ "check": false+ },+ {+ "index": 10,+ "raw": "FR_calf",+ "parent": "FR_thigh",+ "label": "Right Front Shin",+ "source": "file",+ "check": false+ },+ {+ "index": 11,+ "raw": "RL_calf",+ "parent": "RL_thigh",+ "label": "Left Back Shin",+ "source": "file",+ "check": false+ },+ {+ "index": 12,+ "raw": "FL_calf",+ "parent": "FL_thigh",+ "label": "Left Front Shin",+ "source": "file",+ "check": false+ },+ {+ "index": 13,+ "raw": "RR_calf",+ "parent": "RR_thigh",+ "label": "Right Back Shin",+ "source": "file",+ "check": false+ },+ {+ "index": 14,+ "raw": "FR_foot",+ "parent": "FR_calf",+ "label": "Right Front Foot",+ "source": "names",+ "check": false+ },+ {+ "index": 15,+ "raw": "RL_foot",+ "parent": "RL_calf",+ "label": "Left Back Foot",+ "source": "file",+ "check": false+ },+ {+ "index": 16,+ "raw": "FL_foot",+ "parent": "FL_calf",+ "label": "Left Front Foot",+ "source": "names",+ "check": false+ },+ {+ "index": 17,+ "raw": "RR_foot",+ "parent": "RR_calf",+ "label": "Right Back Foot",+ "source": "file",+ "check": false+ }+ ],+ "vocabulary": {+ "format": "[Left |Right ][Qualifier ]Part[ End], e.g. 'Left Front Leg', 'Right Index Finger End'. Qualifier: Front, Back, Middle, Rear, Hind, Inner, Outer, Upper, Lower. No numbers: every joint of a chain takes the same label (Spine, Spine, Spine; Tail, Tail), its last joint 'End' ('Tail End'). 'Bone' only for a joint that is no body part.",+ "parts": [+ "Abdomen",+ "Anal Fin",+ "Ankle",+ "Antenna",+ "Appendage",+ "Arm",+ "Back Claw",+ "Back Hip",+ "Back Paw",+ "Bag",+ "Barbel",+ "Base",+ "Belly",+ "Belt",+ "Body",+ "Bone",+ "Branch",+ "Cannon",+ "Cape",+ "Caudal Fin",+ "Center",+ "Chain",+ "Cheek",+ "Chest",+ "Chin",+ "Claw",+ "Cloth",+ "Collar",+ "Crest",+ "Door",+ "Dorsal Fin",+ "Dorsal Plate",+ "Ear",+ "Elbow",+ "Elbow Back",+ "End",+ "Eye",+ "Eyeball",+ "Eyebrow",+ "Eyelid",+ "Fang",+ "Feather",+ "Fetlock",+ "Fin",+ "Finger",+ "Foot",+ "Forearm",+ "Forearm Twist",+ "Front Hoof",+ "Front Leg",+ "Front Leg End",+ "Front Paw",+ "Front Shoulder",+ "Fur",+ "Gill",+ "Hair",+ "Halter",+ "Hand",+ "Hand Claw",+ "Handle",+ "Hat",+ "Head",+ "Head End",+ "Heel",+ "Hind Leg",+ "Hind Leg End",+ "Hip",+ "Hips",+ "Hoof",+ "Horn",+ "IK Chain",+ "Index Finger",+ "Inner Claw",+ "Inner Finger",+ "Inner Hand Claw",+ "Inner Toe",+ "Jaw",+ "Jiggle",+ "Knee",+ "Large Mandible",+ "Leaf",+ "Leg",+ "Lever",+ "Lid",+ "Lip",+ "Lower Body",+ "Lower Eyelid",+ "Lower Front Lip",+ "Lower Jaw",+ "Lower Left Lip",+ "Lower Lip",+ "Lower Mandible",+ "Lower Right Lip",+ "Mandible",+ "Mane",+ "Metacarpus",+ "Middle Claw",+ "Middle Finger",+ "Middle Hand Claw",+ "Middle Leg",+ "Middle Leg End",+ "Middle Toe",+ "Mouth",+ "Muzzle",+ "Neck",+ "Neck Muscle",+ "Nose",+ "Outer Claw",+ "Outer Finger",+ "Outer Hand Claw",+ "Outer Toe",+ "Palm",+ "Pastern",+ "Paw",+ "Pectoral Fin",+ "Pelvic Fin",+ "Pelvis",+ "Petal",+ "Phalanges",+ "Pincer",+ "Pinky Finger",+ "Prop",+ "Propeller",+ "Rear Hoof",+ "Reins",+ "Ribcage",+ "Ring Finger",+ "Root",+ "Rope",+ "Scapula",+ "Scarf",+ "Shell",+ "Shield",+ "Shin",+ "Shin Twist",+ "Shoulder",+ "Skirt",+ "Skull",+ "Skull Base",+ "Spine",+ "Stem",+ "Stinger",+ "Strap",+ "Tail",+ "Tail Twist",+ "Tentacle",+ "Thigh",+ "Thigh Muscle",+ "Thigh Twist",+ "Thumb Finger",+ "Toe",+ "Toe End",+ "Tongue",+ "Twist",+ "Upper Arm",+ "Upper Arm Twist",+ "Upper Body",+ "Upper Eyelid",+ "Upper Jaw",+ "Upper Left Lip",+ "Upper Lip",+ "Upper Right Lip",+ "Waist",+ "Weapon",+ "Wheel",+ "Whisker",+ "Wing",+ "Wrist",+ "Wrist Back"+ ]+ }+}\ No newline at end of filediff --git a/assets/examples/unitree_go2/cond.npy b/assets/examples/unitree_go2/cond.npynew file mode 100644index 0000000..ecf8f65Binary files /dev/null and b/assets/examples/unitree_go2/cond.npy differdiff --git a/assets/examples/unitree_go2/preview.png b/assets/examples/unitree_go2/preview.pngnew file mode 100644index 0000000..08480d0Binary files /dev/null and b/assets/examples/unitree_go2/preview.png differdiff --git a/assets/examples/unitree_go2/summary.json b/assets/examples/unitree_go2/summary.jsonnew file mode 100644index 0000000..32865c1--- /dev/null+++ b/assets/examples/unitree_go2/summary.json@@ -0,0 +1,43 @@+{+ "name": "unitree_go2",+ "profile": "general",+ "inputs": [+ "unitree_go2.glb"+ ],+ "annotate": "file",+ "annotation": "assets/examples/unitree_go2/annotation.json",+ "animated": false,+ "n_joints_export": 18,+ "rig_glb": null,+ "stats_dataset": "general",+ "dropped_translation": {+ "max_share_of_extent": 0.0,+ "clip": null,+ "joint": null+ },+ "face_pair": {+ "r_hip": {+ "raw": "FR_hip",+ "clean": "Right Front Hip"+ },+ "l_hip": {+ "raw": "FL_hip",+ "clean": "Left Front Hip"+ },+ "source": "front hip"+ },+ "annotation_notes": [+ "annotation from assets/examples/unitree_go2/annotation.json"+ ],+ "n_joints": 18,+ "model_joint_limit": {+ "max_joints": 71,+ "source": "released UniML3D models",+ "fits": true+ },+ "n_clips": 0,+ "cond": "assets/examples/unitree_go2/cond.npy",+ "canonical_glb": "assets/examples/unitree_go2/unitree_go2.glb",+ "notes": [],+ "preview": "assets/examples/unitree_go2/preview.png"+}diff --git a/assets/examples/unitree_go2/unitree_go2.glb b/assets/examples/unitree_go2/unitree_go2.glbnew file mode 100644index 0000000..20757d2Binary files /dev/null and b/assets/examples/unitree_go2/unitree_go2.glb differdiff --git a/assets/qualitative.png b/assets/qualitative.pngdeleted file mode 100644index eb303f4..0000000Binary files a/assets/qualitative.png and /dev/null differdiff --git a/configs/uniml3d_60frames_full_cross_attn.json b/configs/uniml3d_60frames_full_cross_attn.jsonindex 403a626..ef1503e 100644--- a/configs/uniml3d_60frames_full_cross_attn.json+++ b/configs/uniml3d_60frames_full_cross_attn.json@@ -25,17 +25,24 @@"motion_repr": "unimate","test_split_ratio": 0.0,"max_motion_length": 60,- "max_joints": 60,+ "max_joints": 70,"min_joints": 5,- "use_dataset_stats": true,- "balanced_stats": true,+ "use_dataset_stats": false,+ "balanced_stats": false,"sampler_alpha": 0.5,"use_addition_aug": true,"use_removal_aug": true,"use_pooling_aug": true,"use_perturbation_aug": true,"ground_motion_height": true,- "realign_feature": true+ "realign_feature": true,+ "sampler_dataset_weights": {+ "truebones": 0.183,+ "mixamo": 0.223,+ "objaverse": 0.594+ },+ "generic_caption_prob": 0.35,+ "detail_caption_prob": 0.25},"model": {"attention": "full",@@ -78,7 +85,10 @@"max_grad_norm": 1.0,"warmup_ratio": 0.03,"min_lr_ratio": 0.05,- "balanced": true+ "balanced": true,+ "t_sampling": "logit_normal",+ "t_logit_mean": 0.0,+ "t_logit_std": 1.0},"sampling": {"num_samples": 2,diff --git a/configs/uniml3d_60frames_graph_adaln.json b/configs/uniml3d_60frames_graph_adaln.jsonindex 9dfdbd4..c0dca0f 100644--- a/configs/uniml3d_60frames_graph_adaln.json+++ b/configs/uniml3d_60frames_graph_adaln.json@@ -25,17 +25,24 @@"motion_repr": "unimate","test_split_ratio": 0.0,"max_motion_length": 60,- "max_joints": 60,+ "max_joints": 70,"min_joints": 5,- "use_dataset_stats": true,- "balanced_stats": true,+ "use_dataset_stats": false,+ "balanced_stats": false,"sampler_alpha": 0.5,"use_addition_aug": true,"use_removal_aug": true,"use_pooling_aug": true,"use_perturbation_aug": true,"ground_motion_height": true,- "realign_feature": true+ "realign_feature": true,+ "sampler_dataset_weights": {+ "truebones": 0.183,+ "mixamo": 0.223,+ "objaverse": 0.594+ },+ "generic_caption_prob": 0.35,+ "detail_caption_prob": 0.25},"model": {"attention": "graph",@@ -78,7 +85,10 @@"max_grad_norm": 1.0,"warmup_ratio": 0.03,"min_lr_ratio": 0.05,- "balanced": true+ "balanced": true,+ "t_sampling": "logit_normal",+ "t_logit_mean": 0.0,+ "t_logit_std": 1.0},"sampling": {"num_samples": 2,diff --git a/data_process/README.md b/data_process/README.mdindex c0492c6..6012193 100644--- a/data_process/README.md+++ b/data_process/README.md@@ -1,25 +1,26 @@-# UniML3D Data Processing+# UniML3D Data Pipeline-The pipeline that turns raw rigged assets (Truebones FBX, Mixamo FBX, Objaverse GLB) into **[UniML3D](https://huggingface.co/datasets/Linzhan/UniML3D)** โ the text-paired, topology-annotated motion clips used to train UniMate.+This pipeline turns rigged assets (Truebones FBX, Mixamo FBX, Objaverse GLB, or your own) into **[UniML3D](https://huggingface.co/datasets/Linzhan/UniML3D)**: text-paired, topology-annotated motion clips for training UniMate. It also prepares a single rigged asset for a trained model ([`rig_preprocess`](rig_preprocess/README.md)).-**How to read this document.** All commands are run **from the repository root**. Each stage lists the commands you need first; the collapsible **Details / Reference** blocks hold the knobs and field tables. Every wrapper also prints its own usage with `-h`.+Run every command from the repository root. Every wrapper in `data_process/scripts/` prints its options with `-h`.-## Table of Contents+> [!TIP]+> To train on UniML3D you do not need to run stages 1 to 3. Their output is published, so stage 4 alone rebuilds the training clips ([Prebuilt export](#prebuilt-export)).++## Contents- [Overview](#overview)-- [Requirements](#requirements)-- [Directory Layout](#directory-layout)-- [Getting the Raw Data](#getting-the-raw-data)-- [Quick Start](#quick-start)-- [Pipeline Stages](#pipeline-stages)- - [Stage 1 โ Export](#stage-1--export-raw-assets--npz)- - [Stage 2a โ Render](#stage-2a--render-multi-view-frames--t-pose-grids)- - [Stage 2b โ Caption](#stage-2b--caption-renders--text)- - [Stage 3 โ Joint annotation](#stage-3--joint-annotation-name-cleanup--facing-direction)- - [Stage 4 โ Extract](#stage-4--extract-npz--metadata--training-clips)- - [Stage 5 โ Animate](#stage-5--animate-drive-a-mesh-with-a-motion)-- [Custom Assets](#custom-assets)-- [Data Formats](#data-formats)+- [Setup](#setup)+- [Raw data](#raw-data)+- [Quick start](#quick-start)+- [Stage 1: Export](#stage-1-export)+- [Stage 2a: Render](#stage-2a-render)+- [Stage 2b: Caption](#stage-2b-caption)+- [Stage 3: Joint annotation](#stage-3-joint-annotation)+- [Stage 4: Extract](#stage-4-extract)+- [Stage 5: Animate](#stage-5-animate)+- [Custom assets](#custom-assets)+- [Data formats](#data-formats)- [Tools](#tools)- [Conventions](#conventions)- [Troubleshooting](#troubleshooting)@@ -27,608 +28,467 @@ The pipeline that turns raw rigged assets (Truebones FBX, Mixamo FBX, Objaverse## Overview```mermaid-%%{init: {"theme": "base", "themeVariables": {"fontFamily": "-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif", "fontSize": "14px", "lineColor": "#94A3B8", "edgeLabelBackground": "#F1F5F9"}, "flowchart": {"curve": "basis", "nodeSpacing": 40, "rankSpacing": 60}}}%%flowchart LR- raw(["๐ฆ raw FBX / GLB"])- export("<b>1 ยท Export</b><br/>Blender โ NPZ")- render("<b>2a ยท Render</b><br/>EEVEE multi-view")- caption("<b>2b ยท Caption</b><br/>VLM")- joints("<b>3 ยท Joints</b><br/>names + facing")- extract("<b>4 ยท Extract</b><br/>canonicalize")- animate("<b>5 ยท Animate</b><br/>rigged GLB/FBX")- clips(["โจ training clips + cond.npy"])-+ raw(["raw FBX / GLB"])+ export["1 ยท Export<br/>Blender โ NPZ"]+ render["2a ยท Render<br/>multi-view frames"]+ caption["2b ยท Caption<br/>VLM"]+ joints["3 ยท Joints<br/>labels, facing pair"]+ extract["4 ยท Extract<br/>canonical clips"]+ clips(["training clips + cond.npy"])+ animate["5 ยท Animate<br/>rigged GLB / FBX"]raw --> export- raw --> render- export -- "joint_names.json" --> joints- render -- "frames" --> caption- export -- "motions/*.npz" --> extract- caption -- "captions / categories" --> extract- joints -- "clean / face joints" --> extract- extract --> clips- clips --> animate- raw -- "rigged mesh" --> animate-- classDef data fill:#1E1B4B,stroke:#1E1B4B,color:#F8FAFC- classDef blender fill:#6D28D9,stroke:#5B21B6,color:#FFFFFF- classDef vision fill:#0EA5E9,stroke:#0284C7,color:#FFFFFF- classDef llm fill:#A855F7,stroke:#9333EA,color:#FFFFFF- classDef out fill:#FFD21E,stroke:#CA8A04,color:#1E1B4B-- class raw data- class export,animate blender- class render,caption vision- class joints,extract llm- class clips out-- linkStyle default stroke:#94A3B8,stroke-width:1.5px+ raw --> render --> caption --> extract+ export -- joint_names.json --> joints --> extract+ export -- motions/*.npz --> extract+ extract --> clips --> animate```-| Stage | Wrapper(s) | Input | Output |-|-------|-----------|-------|--------|-| 0. Download | `run_download.sh` | Hugging Face Hub | `dataset/raw/<dataset>/` |-| 1. Export | `run_export.sh`, `run_export_general.sh` | raw FBX / GLB | `export/<dataset>/{motions/*.npz, videos/*.mp4, tpose/*.png}` + summary JSONs |-| 2a. Render | `run_render_motion.sh`, `run_render_tpose.sh` | raw FBX / GLB (mixamo: animated characters) | `render/<dataset>/<clip>/v00{0..3}/`, `render/<dataset>_tpose/<name>.png` |-| 2b. Caption | `run_caption_motion.sh`, `run_caption_category.sh` | multi-view renders | `motion_captions.json`, `category_groups.json` |-| 3. Joints | `run_joints_*.sh` | `joint_names.json` | `clean_joint_names.json`, `face_joint_names.json` |-| 4. Extract | `run_extract_features.sh` | export NPZs + stage-2/3 JSONs | per-clip NPZs, `cond.npy`, `captions.json` |-| 5. Animate | `run_animate_{motion,npz,fbx,mixamo,lbs}.sh`, `run_preprocess_char.sh` | motion clip + rigged mesh | animated `.glb` / `.fbx` |+| Stage | Wrapper | Input | Output |+|---|---|---|---|+| 0 Download | `run_download.sh` | Hugging Face Hub | `dataset/raw/<dataset>/` |+| 1 Export | `run_export.sh` | raw FBX / GLB | `export/<dataset>/motions/*.npz`, `joint_names.json`, previews |+| 2a Render | `run_render_motion.sh`, `run_render_tpose.sh` | raw assets | `render/<dataset>/<clip>/v00{0..3}/`, T-pose grids |+| 2b Caption | `run_caption_motion.sh`, `run_caption_category.sh` | renders | `motion_captions.json`, `category_groups.json` |+| 3 Joints | `run_joints_*.sh` | `joint_names.json` | `clean_joint_names.json`, `face_joint_names.json` |+| 4 Extract | `run_extract_features.sh` | stage 1 to 3 output | `features/<dataset>/motions/*.npz`, `cond.npy`, `captions.json` |+| 5 Animate | `run_animate_*.sh` | a motion and a rigged mesh | animated `.glb` / `.fbx` |-Stages 1 and 2a both read the raw assets and can run in either order or in parallel. Stage 3 needs stage 1's `joint_names.json`, stage 2b needs stage 2a's renders, and stage 4 needs all three.+Stages 1 and 2a both read the raw assets and are independent. Stage 3 needs stage 1, stage 2b needs stage 2a, and stage 4 needs all of them.> [!NOTE]-> **Mixamo runs stage 5 in the middle of the pipeline.** Raw Mixamo FBXs are animation-only (armature without mesh), so the render/caption input has to be produced first by `run_animate_mixamo.sh`: export โ animate a character โ render โ caption. The result also ships in the Mixamo repo as `animation_motion_ybot/`, so this step can be skipped by downloading it.+> Raw Mixamo FBXs are animations without a mesh, so Mixamo renders an animated character first: export, then `run_animate_mixamo.sh`, then render and caption. Its output ships in the Mixamo repository as `animation_motion_ybot/`.-## Requirements+## Setup-The pipeline uses the shared `unimate` conda environment โ see [Environment Setup](../README.md#%EF%B8%8F-environment-setup) in the top-level README. Beyond it:+The pipeline uses the repository's `unimate` conda environment ([setup](../README.md#%EF%B8%8F-environment-setup)), plus:-| Tool | Used by |-|------|---------|-| [Blender](https://www.blender.org/) on `PATH` | stages 1 & 5 (headless `blender -b -P`; developed against 3.2) |-| pip [`bpy`](https://pypi.org/project/bpy/) module (in the conda env) | stage 2a โ pinned to `bpy==4.0.0`, the last release for Python 3.10; EEVEE needs the module's GPU context, so this stage does *not* run under `blender -b` |-| `ffmpeg` | video previews โ bundled by `imageio-ffmpeg` from `requirements.txt`; no system install needed |-| [`hf` CLI](https://hf.co/cli) | stage 0 โ installed with the env's `huggingface_hub` |-| CUDA GPU | stages 2a & 2b (and stage 3 when a local LLM is used) |-| `OPENAI_API_KEY` / `GOOGLE_API_KEY` / `DEEPSEEK_API_KEY` | stage 2b (OpenAI / Gemini captioning) and stage 3 (DeepSeek / OpenAI joint annotation) API backends |+| Requirement | Used by |+|---|---|+| [Blender](https://www.blender.org/) โฅ 3.2 on `PATH` | stages 1 and 5 (`blender -b`) |+| pip [`bpy`](https://pypi.org/project/bpy/) 4.0.0 (in the environment) | stage 2a: EEVEE needs the module's GPU context, so this stage runs under plain `python` |+| CUDA GPU | stages 2a and 2b, and stage 3 with a local LLM |+| `OPENAI_API_KEY`, `GOOGLE_API_KEY` or `DEEPSEEK_API_KEY` | API backends of stages 2b and 3 |-## Directory Layout+`ffmpeg` (through `imageio-ffmpeg`) and the [`hf` CLI](https://hf.co/cli) come with the environment.-Code โ one directory per stage, bash wrappers in `scripts/`, shared library code in `utils/`:+**Code.** One package per stage; wrappers in `scripts/`, shared code in `utils/`:```data_process/-โโโ scripts/ Bash entry points โ one run_<stage>_<task>.sh per task-โโโ motion_export/ Stage 1 raw assets โ NPZ (Blender headless)-โโโ motion_rendering/ Stage 2a rigged assets โ multi-view renders + T-pose grids (EEVEE)-โโโ vlm_caption/ Stage 2b multi-view renders โ captions / body-plan categories (VLM)-โโโ joint_annotation/ Stage 3 joint-name cleanup + facing-direction joint selection-โโโ feature_extraction/ Stage 4 NPZ + metadata โ canonical training clips + cond.npy-โโโ mesh_animation/ Stage 5 motion clip + rigged mesh โ animated GLB/FBX (Blender);-โ single-asset preprocessing + manual NumPy FK/LBS-โโโ tools/ Standalone utilities (summary merging, QA patch/visualizers, FBXโGLB)-โโโ utils/ Shared library code (no entry points)+โโโ scripts/ run_<stage>_<task>.sh entry points+โโโ motion_export/ stage 1 raw assets โ NPZ (Blender)+โโโ motion_rendering/ stage 2a multi-view renders, T-pose grids (EEVEE)+โโโ vlm_caption/ stage 2b captions, body-plan categories (VLM)+โโโ joint_annotation/ stage 3 joint labels, facing pair (rules, LLM)+โโโ feature_extraction/ stage 4 canonical training clips, cond.npy, canonical assets+โโโ mesh_animation/ stage 5 drive a rigged mesh with a motion+โโโ rig_preprocess/ one rigged asset โ cond.npy + canonical GLB, for a trained model+โโโ tools/ QA patching, visualizers, summary merging, FBX โ GLB+โโโ utils/ shared library code```-Data โ all wrappers share one convention (override any path through the environment variables each wrapper documents in its header):+**Data.** All wrappers use this layout; each wrapper's header lists the environment variables that override it:```-dataset/raw/<dataset>/ raw FBX / GLB assets-dataset/export/<dataset>/ stage-1 NPZs + previews + stage-2/3 metadata-dataset/render/<dataset>/<clip>/v00{0..3}/ multi-view renders (captioning input)-dataset/render/<dataset>_tpose/ T-pose 2x2 grids (category input)-dataset/features/<dataset>/ stage-4 training clips + cond.npy+dataset/raw/<dataset>/ raw assets+dataset/export/<dataset>/ stage 1 output and the stage 2b / 3 annotations+dataset/render/<dataset>/<clip>/v00{0..3}/ stage 2a renders+dataset/render/<dataset>_tpose/ T-pose grids+dataset/features/<dataset>/ stage 4 output: training clips, cond.npy+dataset/canonical_assets/<dataset>/ stage 4, optional: rest-pose assets in the canonical frame```-`<dataset>` is one of `truebones`, `mixamo`, `objaverse`. For the file-by-file contents of an export directory, see [Reference โ files in `dataset/export/<dataset>/`](#data-formats).+`<dataset>` is `truebones`, `mixamo` or `objaverse`, or `general` for your own data ([Train on your own data](#train-on-your-own-data-optional)). The `dataset/render/` paths point into the raw mirrors: `raw/truebones/{animation_render,species_tpose}`, `raw/mixamo/{animation_motion_render,character_tpose}` and `raw/objaverse_renders/{glb_render,tpose}`.-Both `dataset/render/` paths are symlinks into each dataset's own Hub mirror, so renders sit beside the assets they came from: `truebones` โ `raw/truebones/{animation_render,species_tpose}`, `mixamo` โ `raw/mixamo/{animation_motion_render,character_tpose}`, `objaverse` โ `raw/objaverse_renders/{glb_render,tpose}` (a separate repo, because there are 10,355 clip folders).--## Getting the Raw Data--The source assets are hosted on the Hugging Face Hub (collected under [UniMate](https://huggingface.co/collections/Linzhan/unimate)) and download straight into the layout above:+## Raw data```bash-bash data_process/scripts/run_download.sh mixamo # โ dataset/raw/mixamo/{animation_motion,character_refined,...}-bash data_process/scripts/run_download.sh objaverse # โ dataset/raw/objaverse/glb-bash data_process/scripts/run_download.sh truebones # โ dataset/raw/truebones (annotations only โ see below)-bash data_process/scripts/run_download.sh objaverse_renders # โ dataset/raw/objaverse_renders (optional, ~6 GB)+bash data_process/scripts/run_download.sh mixamo # dataset/raw/mixamo+bash data_process/scripts/run_download.sh objaverse # dataset/raw/objaverse/glb+bash data_process/scripts/run_download.sh truebones # dataset/raw/truebones (annotations only)+bash data_process/scripts/run_download.sh objaverse_renders # dataset/raw/objaverse_renders (optional, ~6 GB)```-| Dataset | Source |-|---------|--------|+| Name | Repository |+|---|---|| `mixamo` | [Linzhan/Mixamo-Animations-Characters](https://huggingface.co/datasets/Linzhan/Mixamo-Animations-Characters) || `objaverse` | [Linzhan/Objaverse-XL-Rigged-Animated](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated) |-| `truebones` | [Linzhan/Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations) (prompts, metadata, renders, build scripts โ no motion files) |-| `objaverse_renders` | [Linzhan/Objaverse-XL-Rigged-Animated-Renders](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated-Renders) (four-view clip MP4s + T-pose grids; download-only companion, not a pipeline dataset name) |+| `truebones` | [Linzhan/Truebones-ZOO-Annotations](https://huggingface.co/datasets/Linzhan/Truebones-ZOO-Annotations): prompts, metadata, renders and build scripts; no motion files |+| `objaverse_renders` | [Linzhan/Objaverse-XL-Rigged-Animated-Renders](https://huggingface.co/datasets/Linzhan/Objaverse-XL-Rigged-Animated-Renders): four-view clip MP4s and T-pose grids |-The Mixamo and Truebones repos also carry the stage-2a renders as per-view MP4 previews plus camera JSONs. The per-frame PNGs the caption stage reads are not hosted; stage 2a regenerates them.--> [!TIP]-> **You may not need to run stages 1-3 at all.** Their combined output is published as [Linzhan/UniML3D](https://huggingface.co/datasets/Linzhan/UniML3D), so stage 4 on its own reproduces the training clips:->-> ```bash-> hf download Linzhan/UniML3D --type dataset --local-dir dataset --include "export/*"-> bash data_process/scripts/run_extract_features.sh objaverse-> ```->-> The Truebones motion NPZs are held back there for the licensing reason below; everything else, including all captions and annotations, is included.+The Mixamo and Truebones repositories include the stage 2a renders as MP4s with camera JSONs. The per-frame PNGs that captioning reads are not hosted; stage 2a regenerates them.> [!IMPORTANT]-> **The Truebones motion files are not downloadable from us.** The Truebones ZOO animal pack is a commercial product whose license does not allow redistribution, so its repo ships annotations only. Purchase the pack from [Truebones](https://truebones.com) and rebuild the per-clip layout as described below.--<details>-<summary><b>Details โ</b> rebuilding the Truebones layout from a purchased copy</summary>+> **Truebones motion files are not redistributed.** The Truebones ZOO pack is a commercial product. Purchase it from [Truebones](https://truebones.com), unpack it to `dataset/raw/truebones/Truebone_Z-OO/{Animal}/`, and run the downloaded `scripts/pipeline/` (see that repository's README) to build the per-clip layout `dataset/raw/truebones/animation/{Species}-{Action}.fbx`. Species names contain no `-`. Stage 5 also reads the meshes from the original per-animal layout; directory names are matched ignoring case and separators (object type `Dog2` โ `Truebone_Z-OO/Dog-2/`).-Unpack the pack to `dataset/raw/truebones/Truebone_Z-OO/{Animal}/` and run the downloaded `scripts/pipeline/` (see the repo's README) to rebuild the per-clip layout โ one binary FBX per clip, flat in a single directory:+### Prebuilt export-```-dataset/raw/truebones/animation/{Species}-{Action}.fbx # e.g. Alligator-Big_Mouth.fbx-```+The output of stages 1 to 3 (motions, captions, joint labels, facing pairs, QA lists) is part of UniML3D, so stage 4 alone rebuilds the training clips:-Species names must not contain `-` โ the first dash separates species from action.--Stage 5 additionally needs the **original per-animal layout** for the character meshes, since the flat per-clip files above are the animation source only:--```-dataset/raw/truebones/Truebone_Z-OO/{Animal}/*.fbx # e.g. Truebone_Z-OO/Dog-2/+```bash+hf download Linzhan/UniML3D --repo-type dataset --local-dir dataset --include "export/*"+# the Hub splits export/objaverse/{motions,videos} into 64 hash buckets; flatten them (hardlinks)+for d in dataset/export/objaverse/{motions,videos}; do find "$d" -mindepth 2 -type f -exec ln -f -t "$d" -- {} +; done+bash data_process/scripts/run_extract_features.sh objaverse```-Directory names are matched ignoring separators and case, so the clip-name object type `Dog2` resolves to the `Dog-2` directory.+The Truebones motion NPZs are withheld for the license reason above; its captions and annotations are included.-</details>--## Quick Start+## Quick start-End-to-end run for Objaverse (Truebones is identical with `objaverse` โ `truebones`; see the [note above](#overview) for Mixamo's extra animate step):+All stages for Objaverse. Truebones is the same with `truebones`; Mixamo adds the animate step noted above.```bash-# 0. Raw data-bash data_process/scripts/run_download.sh objaverse--# 1. Export raw GLBs to NPZ motion data-bash data_process/scripts/run_export.sh objaverse --multi-worker 8--# 2a. Render multi-view frames + T-pose grids-bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8+bash data_process/scripts/run_download.sh objaverse # 0 raw data+bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # 1 export+bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8 # 2a renderbash data_process/scripts/run_render_tpose.sh objaverse--# 2b. Caption motions + classify body plans (local Qwen3.5-9B by default)-bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu+bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu # 2b caption (local Qwen3.5-9B)bash data_process/scripts/run_caption_category.sh objaverse--# 3. Joint-name cleanup + facing-direction pair-bash data_process/scripts/run_joints_names_clean_llm.sh objaverse-bash data_process/scripts/run_joints_face_select_llm.sh objaverse--# 4. Canonicalized training clips + topology condition-bash data_process/scripts/run_extract_features.sh objaverse+bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # 3 joint labels+bash data_process/scripts/run_joints_face_select_llm.sh objaverse # facing pair+python data_process/tools/patch_annotations.py --datasets objaverse # QA fixes+bash data_process/scripts/run_extract_features.sh objaverse # 4 training clips + cond.npy```-Every stage is resumable โ rerunning a wrapper skips already-complete outputs and retries only what failed.+Every stage is resumable: a rerun skips finished outputs and retries failures.-## Pipeline Stages--### Stage 1 โ Export: raw assets โ NPZ+## Stage 1: Export```bash-bash data_process/scripts/run_export.sh truebones # flat per-clip {Species}-{Action}.fbx-bash data_process/scripts/run_export.sh mixamo # animation-only FBX (no mesh)-bash data_process/scripts/run_export.sh objaverse # GLB/GLTF-bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # parallel workers (mixamo too)-bash data_process/scripts/run_export.sh mixamo --no-vis # skip per-clip MP4 previews (much faster)+bash data_process/scripts/run_export.sh truebones # {Species}-{Action}.fbx clips+bash data_process/scripts/run_export.sh mixamo --no-vis # skip MP4 previews (much faster)+bash data_process/scripts/run_export.sh objaverse --multi-worker 8+bash data_process/scripts/run_export.sh objaverse --save_glb # also rigs/<asset>.glb+bash data_process/scripts/run_export.sh mixamo --glb_only # only add the GLBs to an existing export```-Skeletons are pruned of control/helper bones that carry no skin weight and converted to Y-up. Mixamo needs its explicit preset: animation-only FBXs carry no mesh, so auto mode cannot detect them. No joint-count filtering happens here โ stage 4 decides which skeletons enter training, so the exports stay complete.+Each asset is imported in Blender, pruned of control and helper bones that carry no skin weight, converted to Y-up, and written as one NPZ per animation. Mixamo has its own preset, since its FBXs have no mesh. Joint counts are not filtered here; stage 4 decides which skeletons enter training.-For assets outside the three datasets, use `run_export_general.sh` โ see [Custom Assets](#custom-assets).+`--save_glb` writes `rigs/<asset>.glb`: the asset's mesh and armature in its rest pose on the pruned skeleton, textures embedded, without animation. Its skeleton is that of the asset's clips, so every clip drives it directly. Mixamo writes one GLB per character in `CHAR_DIR` (default `dataset/raw/mixamo/character_refined`), in the character's rest pose on the shared clip skeleton; a character needs the 22 core joints, and one without finger bones is exported without them. Missing textures are listed in `rigs/texture_issues/` and failed builds in `rigs/glb_errors/`; neither fails the export. `--glb_only` adds the GLBs to an existing export, including the released one, and writes nothing else.<details>-<summary><b>Details โ</b> outputs, parallel workers, and resume markers</summary>--Each export directory receives:+<summary>Outputs, parallelism, resume</summary>| Path | Content |-|------|---------|-| `motions/{clip}.npz` | one NPZ per clip (see [Data Formats](#data-formats)) |-| `videos/{clip}.mp4`, `tpose/{name}.png` | skeleton preview per clip and rest-pose still per asset (skip with `--no-vis`) |-| `joint_names.json` | `{asset: [pruned bone names]}` โ the input of stage 3 |-| `joint_count.json`, `clip_frames.json`, `summary.json` | joints per skeleton, frames per clip, dataset-level summary |-| `.completed/{asset}.json` | per-asset completion markers; delete one to force a re-export |--`--multi-worker N` shards a *directory* input over N Blender processes (objaverse, mixamo, auto mode). Each worker writes its own `*_worker{i}.json` summary shards, merged afterwards by `tools/merge_summaries.py` โ only when every worker exited cleanly; a worker that had failed assets exits non-zero and the merge is left for the next run. Truebones is exported by a single-process exporter and ignores `--multi-worker`.--An optional `excluded.csv` (column `object_id` or `file`) in the input directory or its parent lists asset stems the objaverse exporter skips entirely (`run_export_general.sh` does not read it).--Skeleton pruning is joint across all of an asset's clips, so an asset only completes atomically โ the marker is written once every clip of that asset is on disk. Markers also carry the pruned joint names, which is what makes `joint_names.json` rebuildable from disk after an interrupted run.+|---|---|+| `motions/<clip>.npz` | one clip ([format](#data-formats)) |+| `videos/<clip>.mp4`, `tpose/<asset>.png` | skeleton previews (`--no-vis` skips them) |+| `joint_names.json` | `{asset: [bone names]}`, the input of stage 3 |+| `joint_count.json`, `clip_frames.json`, `summary.json` | joints per skeleton, frames per clip, totals |+| `.completed/<asset>.json` | completion markers; delete one to re-export that asset |+| `rigs/` | `--save_glb` / `--glb_only` output |++- `--multi-worker N` shards a directory over N Blender processes (objaverse, mixamo, general). Their summary shards are merged by `tools/merge_summaries.py` only when every worker succeeded. Truebones runs in one process, except with `--glb_only`.+- An `excluded.csv` (column `object_id` or `file`) in the input directory or its parent lists assets the objaverse and general exporters skip.+- Pruning is shared by all clips of an asset, so an asset completes as a whole. Its marker stores the pruned joint names, from which `joint_names.json` can be rebuilt.</details>-### Stage 2a โ Render: multi-view frames & T-pose grids--Renders each motion with EEVEE from four cameras 90ยฐ apart at a fixed elevation (`v000`โ`v003`, nominally front / back / left / right in the asset's own frame), plus a T-pose 2x2 grid per asset for body-plan classification. The captioner treats the four as unlabeled azimuths โ the real front is only established by the stage-3 facing annotation.+## Stage 2a: Render```bash-bash data_process/scripts/run_render_motion.sh truebonesbash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8-bash data_process/scripts/run_render_motion.sh objaverse --missing-only # resume: skip complete clips+bash data_process/scripts/run_render_motion.sh objaverse --missing-only # skip finished clipsbash data_process/scripts/run_render_tpose.sh objaverseDATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_motion.sh mixamo```-Each clip becomes `<clip>/v00{0..3}/*.png` (the per-frame PNGs the caption stage reads) plus a composed `<clip>/v00{0..3}.mp4` per view (`--no-video` to skip). Render folders use the same clip naming as stage 1, so they line up with the exported NPZs.+Each clip is rendered with EEVEE from four cameras 90ยฐ apart (`v000` to `v003`), as per-frame PNGs and one MP4 per view (`--no-video` skips the MP4s). The captioner treats the views as unlabelled; the asset's front is defined later by its facing pair. `run_render_tpose.sh` renders a 2ร2 T-pose grid per asset for body-plan classification.<details>-<summary><b>Details โ</b> frame cap, fps consistency, multi-GPU</summary>+<summary>Settings</summary>-- **Frame cap.** Rendering stops at the first 200 frames per clip (`MAX_RENDER_FRAMES`, ~6.7 s at 30 fps) โ enough context for captioning without paying for full-length renders. Clips shorter than `MIN_ACTION_FRAMES` (default 5) are skipped.-- **`--fps` must match stage 1.** glTF stores keyframe times in seconds, so the importer resamples them at the scene frame rate; a mismatch makes the renders cover a different time window than the NPZs. Default 30 in both stages.-- **Multi-GPU.** `--multi-worker N` shards assets over N processes, and `NUM_GPUS=k` assigns worker `i` to `CUDA_VISIBLE_DEVICES=i % k`. That pinning does **not** bind EEVEE's EGL context, so on a shared multi-GPU machine every worker still renders on the first GPU โ give each render job a cgroup / scheduler allocation that exposes exactly one GPU instead. `--missing-only` exists for objaverse only; the wrapper rejects it for the other datasets, which resume per asset anyway.-- Quality knobs (`RESOLUTION`, `SAMPLES`, `CAMERA_DIST`) are environment overrides on both wrappers; run either with `-h`.+- At most 200 frames are rendered per clip (`MAX_RENDER_FRAMES`); clips under 5 frames (`MIN_ACTION_FRAMES`) are skipped.+- `--fps` must match stage 1 (default 30 in both): glTF stores keyframe times in seconds.+- `--multi-worker N` with `NUM_GPUS=k` sets `CUDA_VISIBLE_DEVICES=i % k` per worker, but EEVEE's EGL context ignores it: on a shared machine, give each render job an allocation with one GPU.+- `--missing-only` applies to objaverse and general; the other datasets resume per asset.+- `RESOLUTION`, `SAMPLES` and `CAMERA_DIST` set the quality.</details>-### Stage 2b โ Caption: renders โ text--Two independent passes over the stage-2a output: per-clip motion captions, and one body-plan category per skeleton.+## Stage 2b: Caption```bash-# motion captions โ the backend is picked from MODEL-bash data_process/scripts/run_caption_motion.sh truebones # local Qwen3.5-9B (default)-bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu # shard across all visible GPUs-MODEL=Qwen/Qwen3.8-27B bash data_process/scripts/run_caption_motion.sh objaverse # local 27B (one 80 GB GPU)-MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse # Gemini (GOOGLE_API_KEY)-MODEL=gpt-5-mini bash data_process/scripts/run_caption_motion.sh mixamo # OpenAI (OPENAI_API_KEY)--# body-plan categories (single GPU, local model)-bash data_process/scripts/run_caption_category.sh objaverse+bash data_process/scripts/run_caption_motion.sh objaverse --multi-gpu # local Qwen/Qwen3.5-9B+MODEL=Qwen/Qwen3.8-27B bash data_process/scripts/run_caption_motion.sh objaverse # local, one 80 GB GPU+MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse # GOOGLE_API_KEY+MODEL=gpt-5-mini bash data_process/scripts/run_caption_motion.sh mixamo # OPENAI_API_KEY+bash data_process/scripts/run_caption_category.sh objaverse # body-plan categories```-Both write into the export directory next to the stage-1 output:--| File | Content |-|------|---------|-| `motion_captions.json` | `{clip: caption}` โ one short caption per clip |-| `motion_captions_failed.txt` | clips that failed or whose renders were incomplete; retried on the next run |-| `category_groups.json` | `{category: [assets]}` over `bipedal`, `quadrupedal`, `insectoid`, `avian`, `marine`, `serpentine`, `articulated_rigid`, plus an `uncertain` bucket |-| `category_groups_review.json`, `category_groups_errors.json` | per-asset votes and evidence; assets whose retries were exhausted |+| Output (in `export/<dataset>/`) | Content |+|---|---|+| `motion_captions.json` | `{clip: caption}` |+| `motion_captions_failed.txt` | failed clips and clips with incomplete renders; retried on the next run |+| `category_groups.json` | `{category: [assets]}` over `bipedal`, `quadrupedal`, `insectoid`, `avian`, `marine`, `serpentine`, `articulated_rigid` and `uncertain` |+| `category_groups_review.json`, `category_groups_errors.json` | per-asset votes and evidence; assets whose retries ran out |-Clips whose renders are incomplete are recorded as failures rather than captioned from a partial set. Mixamo is a single shared humanoid rig, so its wrapper writes `category_groups.json` directly instead of classifying.+Mixamo is one humanoid rig, so its wrapper writes `category_groups.json` without classifying.<details>-<summary><b>Details โ</b> backends, parallelism, and what the models see</summary>+<summary>Backends and inputs</summary>-- **Backend selection.** A `MODEL` containing `qwen` runs locally through HuggingFace transformers; `gemini*` uses the Gemini API; anything else is treated as an OpenAI-compatible model (`BASE_URL` points it at a proxy). The local default is `Qwen/Qwen3.5-9B`; `MODEL=Qwen/Qwen3.8-27B` selects the 27B model (โ56 GB in bf16, so one 80 GB GPU).-- **Parallelism.** `--multi-gpu` applies to the local backend only: one process per visible GPU writes a shard file, merged into `motion_captions.json` when all of them succeed. API backends scale with `NUM_WORKERS` concurrent requests instead.-- **Caption inputs.** The local backend receives each clip's four camera views as native video at the render frame rate; API backends receive frame sequences sampled down to `--max_frames_per_view`. For mixamo and truebones the official catalogue prompts are attached as frame-grounded hints (`HINTS_JSON=""` disables them).-- **Category inputs.** Per asset, `classify_category.py` shows four T-pose views, the first frame of one clip from the same four cameras (a rest pose can lie flat, so the natural stance matters), skeleton facts (joint count, grounded limb chains, label histogram, body extents) and up to `--max_captions` motion captions. `VOTES` (default 3) sampled answers are majority-voted; a tie, a low-confidence majority or the model's own `uncertain` verdict lands in the `uncertain` bucket instead of being silently mis-filed. Hand corrections go in `<dataset>_categories.json` in the patch directory; `tools/eval_category_groups.py` scores a run against a truth set.+- **Backend.** A `MODEL` containing `qwen` runs locally with transformers, a model containing `gemini` uses the Gemini API, and anything else an OpenAI-compatible API (`BASE_URL` for a proxy). `--multi-gpu` runs one local process per visible GPU and merges their shards when all succeed; API backends use `NUM_WORKERS` concurrent requests.+- **Captions.** The local backend receives the four views as video; API backends receive frames sampled down to `--max_frames_per_view`. For Mixamo and Truebones the catalogue action names are added as hints (`HINTS_JSON=""` disables them).+- **Categories.** Per asset, `classify_category.py` shows four T-pose views, the first frame of one clip, skeleton facts and up to `--max_captions` captions. `VOTES` (default 3) answers are majority-voted; a tie, a low-confidence majority or an `uncertain` answer goes to `uncertain`. `tools/eval_category_groups.py` scores a run against a truth set.+- **Prop words.** A catalogue hint such as *Rifle Run Left* can put a prop into a caption. `vlm_caption/caption_rewrite_llm.py --captions <motion_captions.json> --output <patch_dir>/<dataset>_captions_llm.json` rewrites such captions into body-only wording; after review, `tools/patch_annotations.py` applies the validated rewrites whose original caption is unchanged. Nothing is modified in place.</details>-<details>-<summary><b>Details โ</b> rewriting captions that name props</summary>+## Stage 3: Joint annotation-Nothing but the body is rendered, yet a catalogue hint such as *Rifle Run Left* can leak a prop or scenery word into a caption. `caption_rewrite_llm.py` finds those with a word list, asks a text LLM to rewrite each into body-only wording (`aims a rifle` โ `extends one arm forward as if aiming`), validates the answer and writes a patch JSON that `tools/patch_annotations.py` applies after review โ nothing is modified in place:+Two tasks. **Joint-name cleaning** maps raw bone names onto one anatomical vocabulary (`mixamorig:LeftUpLeg` โ `Left Thigh`). **Facing selection** picks the joint pair that defines the rig's heading: a left / right pair, or a head / tail axis for limbless bodies. Each has an LLM and a rule variant. The LLM default is `deepseek-v4-flash` (`MODEL=` takes a `gpt-*` model or a local Hugging Face model id).```bash-python data_process/vlm_caption/caption_rewrite_llm.py \- --captions dataset/export/<dataset>/motion_captions.json \- --output <patch_dir>/<dataset>_captions.json+bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # โ clean_joint_names.json+bash data_process/scripts/run_joints_face_select_llm.sh objaverse # โ face_joint_names.json+python data_process/tools/patch_annotations.py [--dry_run] [--datasets objaverse]```-</details>--### Stage 3 โ Joint annotation: name cleanup & facing direction--Two tasks, each with a rule-based and an LLM variant: joint-name cleaning (`names_*`) maps raw bone names onto a canonical anatomical vocabulary, and facing-direction selection (`face_*`) picks the bilateral joint pair โ or head/tail body axis, for serpentine rigs โ that defines each rig's heading. The LLM variants default to `deepseek-v4-flash` (`MODEL=` switches to any `gpt-*` model or a local HF model id). The two required passes fall back to the rules when a rig fails or exceeds its per-rig budget (`RIG_TIMEOUT`); the two refinement passes below instead keep the entry they were asked to correct.--**Required** โ these two produce the artifacts stage 4 consumes:--```bash-bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # โ clean_joint_names.json-bash data_process/scripts/run_joints_face_select_llm.sh objaverse # โ face_joint_names.json-```--Rigs that fell back to the rules are listed in `failed_clean_names.txt` / `failed_face_joints.txt` next to the outputs, so a refinement pass can be restricted to them.+A rig whose LLM call fails or exceeds `RIG_TIMEOUT` gets the rule result and is listed in `failed_clean_names.txt` / `failed_face_joints.txt`.<details>-<summary><b>Details โ</b> refinement passes, rule variants, and QA visualizers</summary>--Skip the refinement passes on a clean LLM run; reach for them when the required passes recorded failures or when spot checks reveal residual errors. Both edit in place, create a `.bak` sibling on first run, and list rigs they still could not fix in `still_*` text files.+<summary>Refinement passes, rule variants, visual checks</summary>```bash-bash data_process/scripts/run_joints_names_correct_llm.sh objaverse # re-check every label, correct in place-bash data_process/scripts/run_joints_face_correct_llm.sh objaverse # retry rigs whose facing pair came back "empty"+bash data_process/scripts/run_joints_names_correct_llm.sh objaverse # re-check every label (FAILED_LIST= to restrict)+bash data_process/scripts/run_joints_face_correct_llm.sh objaverse # retry rigs with an empty facing pair+bash data_process/scripts/run_joints_names_clean_rule.sh objaverse # rules only, no API+bash data_process/scripts/run_joints_face_select_rule.sh objaverse+bash data_process/scripts/run_joints_vis_tpose.sh objaverse # one PNG per skeleton, every joint labelled+bash data_process/scripts/run_joints_vis_facing.sh objaverse # rest pose vs. canonicalized pose```-`names_correct_llm` sends each `{raw, current}` label pair back to the LLM to keep or fix (restrict with `FAILED_LIST=.../failed_clean_names.txt`). `face_correct_llm` re-attempts only `source == "empty"` entries by default; `--force_reattempt` also retries the rule-fallback rigs. Both cleaners write the same `clean_joint_names.json`, so the LLM pass skips rigs the rule pass already filled in (`--overwrite` / `--redo_failed`).--Pure rule-based variants exist as `run_joints_names_clean_rule.sh` / `run_joints_face_select_rule.sh` โ no API needed, and the LLM variants use them as hint and fallback.--Three visualizers check a facing pair by eye:--```bash-bash data_process/scripts/run_joints_vis_tpose.sh <dataset> # one annotated PNG per skeleton, every joint labeled-bash data_process/scripts/run_joints_vis_facing.sh <dataset> # rest pose vs. canonicalized pose-python -m data_process.tools.vis_motion <motion.npz> --direction face --face-joints I J-```+The correction passes edit in place, keep a `.bak` copy, and list rigs they could not fix in `still_*.txt`; `face_correct_llm --force_reattempt` also retries rule-fallback rigs. The LLM cleaner skips rigs that already have labels unless given `--overwrite` or `--redo_failed`.-`run_joints_vis_facing.sh` runs the stage-4 canonicalization on every rest pose and saves a side-by-side `[rest pose with the pair marked and its forward arrow | the same pose corrected, facing +Z]` under `outputs/tpose_facing_vis/<dataset>/`, plus a `facing_summary.tsv` flagging `COINCIDENT` pairs (both joints on the midline โ pick another pair), `REST-DEGENERATE` rest poses (lateral at frame 0 but not in the rest pose: the exported rest pose lies on its side, so the stage-4 T-pose facing is unreliable) and `UNRESOLVED` names. `FACE_JSON=<patch_dir>/<ds>_face_pairs.json` previews a patch before applying it.+`run_joints_vis_facing.sh` writes `outputs/tpose_facing_vis/<dataset>/` and a `facing_summary.tsv` that marks `COINCIDENT` pairs (both joints on the midline), `REST-DEGENERATE` rest poses (lying on their side) and `UNRESOLVED` names. `FACE_JSON=<face_pairs.json>` previews a facing patch.</details><details>-<summary><b>Details โ</b> QA patch (<code>tools/patch_annotations.py</code>, run last)</summary>+<summary>QA patch (<code>tools/patch_annotations.py</code>, run after stage 3)</summary>-```bash-python data_process/tools/patch_annotations.py [--dry_run] [--datasets truebones mixamo] [--report_dir DIR]-```--Applies the corrections found in a manual audit of the stage-2/3 outputs, so they are reproducible on a fresh run and recorded in one place. Rerun it after regenerating any stage-2/3 output.--**Rule fixes**, built into the script:+Rerun it after regenerating any stage 2 or 3 output. Built-in fixes:| Fix | Effect |-|-----|--------|-| Pelvis unification | unsided `Pelvis` / `Hip` โ `Hips` (`Left Hip` / `Right Hip` untouched) |+|---|---|+| Pelvis | unsided `Pelvis` / `Hip` โ `Hips` || Numeric labels | pass-through labels (`_2`, `_1045`) โ `Bone` |-| Arthropod chains | numbered leg / claw-arm chains get anatomical positions (Thigh, Shin, Foot, Toe โฆ / Upper Arm, Forearm, Hand, Claw) instead of `Leg` repeated along the chain |-| Duplicate sides | rigs whose second side is a Blender/Maya duplicate (`LeftUpLeg.001`, `Leg_L1`, โฆ) get side prefixes from the rest-pose X coordinate |-| Face resync | the `clean` fields of `face_joint_names.json` are re-synced against the labels |-| Captions | grammar and subject fixes, plus locomotion qualifiers grounded in the root trajectory (`walks forward` with no root travel โ `walks in place`) |--**Hand overrides** are read from `--patch_dir` (default `dataset/UniML3D/patches/`) as `<dataset>_{joint_labels,face_pairs,captions,categories}.json`. Missing files are skipped, so the rule fixes apply on their own.+| Arthropod chains | numbered leg and claw chains get positions (Thigh, Shin, Foot, Toe; Upper Arm, Forearm, Hand, Claw) |+| Duplicate sides | a side duplicated in Blender / Maya (`LeftUpLeg.001`, `Leg_L1`) gets its side from the rest-pose X coordinate |+| Face resync | the label text in `face_joint_names.json` follows the joint labels |+| Captions | grammar fixes; locomotion words checked against the root path (`walks forward` without travel โ `walks in place`) |-**Skip lists** the patch writes into the export directory for stage 4:+Hand corrections are read from `--patch_dir` when present: `<dataset>_{joint_labels,face_pairs,captions,categories}.json`, and the reviewed clip and rig lists from which the script writes the files stage 4 reads. The released export already contains their result:-| Output | Scope |-|--------|-------|-| `export/<ds>/filtered_clips.txt` | individual clips, any dataset |-| `export/<ds>/activity_keep.txt` | `<clip>`: exempt from stage 4's low-activity filter โ a real motion confined to a few joints (a head turn, a clap, a wave) that the joint-activity gate cannot tell from a held pose; every other filter still applies |-| `export/<ds>/clip_trims.txt` | `<clip> <N>`: the clip's export NPZ starts N frames late โ stage 4 drops `motions/<clip>.npz` frames `0..N-1` on load โ for a bind-pose or foreign opening frame that snaps into the motion, so the clip is kept instead of filtered |-| `export/objaverse/filtered_objects.txt` | whole rigs โ only the `tpose_wrong` flag (rest pose lies flat / rotated / upside-down) and `not_in_legacy_raw` (raw GLB absent from the legacy raw set, held out of training) |-| `export/objaverse/rig_flags.json` | informational: `empty_pair`, `bone_pair`, `body_axis_unnamed`, `facing_wrong`, `object_no_front` |--Every other flag is informational โ those rigs are kept and trained with the facing they have. Per-rig evidence (rest-pose metrics, mesh T-pose grids and corrected-skeleton panels, all checked by eye) lives under `export/objaverse/tpose_abnormal_vis/`.--Truebones uses `filtered_clips.txt` for the three clips whose rig disagrees with their object's reference skeleton; objaverse uses it for the motion-discontinuity review.+| File | Effect in stage 4 |+|---|---|+| `filtered_clips.txt` | clips skipped |+| `clip_trims.txt` | `<clip> <N>`: the first N frames dropped (a bind pose or foreign opening frame) |+| `activity_keep.txt` | clips exempt from the low-activity filter (a real motion of a few joints, such as a head turn) |+| `filtered_objects.txt` | objaverse: whole rigs skipped (rest pose lying, rotated or upside down; asset held out of the release) |+| `rig_flags.json` | objaverse, informational: `empty_pair`, `bone_pair`, `body_axis_unnamed`, `facing_wrong`, `object_no_front` |</details>-### Stage 4 โ Extract: NPZ + metadata โ training clips+## Stage 4: Extract```bashbash data_process/scripts/run_extract_features.sh objaverse--# crop long motions into overlapping fixed-length clips instead of truncating-APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones--# parallel over object types, skip MP4 previews+APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones # crop long motions into windowsNUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse+SAVE_GLB=1 bash data_process/scripts/run_extract_features.sh truebones # also the canonical assets+python -m data_process.feature_extraction.canonical_assets \+ --export_dir dataset/export/truebones --features_dir dataset/features/truebones # canonical assets only```-Every object type is canonicalized against its T-pose (facing โ XZ-centering โ diameter scaling โ grounding), static pre/post-roll is trimmed, low-activity and motion-discontinuous clips are filtered, and the per-object topology condition is written to `cond.npy`. Output goes to `dataset/features/<dataset>/`: `motions/{object}-{motion}-{clip_idx}.npz`, `videos/`, `tpose/`, plus `cond.npy` and a `metadata.txt` report, and โ whenever they have content โ `captions.json`, `category_groups.json` and `filtered_clips.json` (runtime-filtered clips and skipped object types; clips pre-filtered through `filtered_clips.txt` are dropped before processing and not listed).--<details>-<summary><b>Details โ</b> filters and thresholds</summary>--| Setting | Default | Effect |-|---------|---------|--------|-| `--min_joints` / `--max_joints` | 8 / 150 (objaverse: 4 / 180 via the wrapper) | object types outside the range are skipped โ this, not stage 1, decides which skeletons enter training |-| `--max_clip_len` | 200 | frames kept per motion; without `APPLY_CLIP` only the first 200 frames survive |-| `APPLY_CLIP=1` (`--apply_clip`) | off | crop long motions into overlapping windows of `max_clip_len` frames with stride `max_clip_len โ diffusion_max_len` (200 โ 90 = 110), so every training crop of up to `--diffusion_max_len` frames fits inside some saved clip |-| `--activity_threshold` | 0.02 | minimum joint activity (temporal spread at canonical scale) for a clip to be kept |-| `--jump_step_threshold` | 0.20 | discontinuity filter: a clip is dropped when one frame moves the skeleton at least this many body lengths **and** the step is at least `--jump_ratio_threshold` times the clip's median; `0` disables it |-| `--jump_ratio_threshold` | 8.0 | max/median per-frame joint displacement required alongside `--jump_step_threshold` |-| `--static_threshold` | 1e-5 | per-frame displacement below which a frame counts as static; leading / trailing static frames are trimmed |-| `--min_frames` | 8 | minimum frame count after trimming |-| `--target_diameter` | 2.0 | leaf-to-leaf skeleton diameter every object type is scaled to |-| `--max_freqs` / `--max_path_len` | 8 / 5 | Laplacian eigenvectors per joint and the clamp on topology distances in `cond.npy` |--Clips dropped at runtime and skipped object types are recorded in `filtered_clips.json`; `metadata.txt` reports the dataset totals.+Each object type is canonicalized against its T-pose (facing +Z, centred in XZ, scaled to diameter 2, grounded). Static lead-in and lead-out frames are trimmed, low-activity and discontinuous clips are dropped, and the topology condition of each object type goes into `cond.npy`. Output in `dataset/features/<dataset>/`: `motions/<object>-<motion>-<clip_idx>.npz`, `cond.npy`, `captions.json` (plus `captions_generic.json` / `captions_detail.json` when the export has those captions), `category_groups.json`, `filtered_clips.json` (clips and object types dropped at run time), `metadata.txt`, and previews in `videos/` and `tpose/`.-Two hand-reviewed skip lists are read from the export directory before any processing, both **optional**: `filtered_clips.txt` (individual clips, every dataset) and `filtered_objects.txt` (whole rigs, objaverse only), both written by `tools/patch_annotations.py`, plus `clip_trims.txt` (`<clip> <N>`: drop the first N frames of a clip's export NPZ; `--clip_trims`) and `activity_keep.txt` (clips exempt from the low-activity filter; `--activity_keep`). The matching flags take a path, `"auto"` (the default) or an empty string to disable. `--category_groups` follows the same convention for `category_groups.json`, which this stage only copies through.--</details>+**Canonical assets** (`SAVE_GLB=1`, or `canonical_assets.py`): each `export/<dataset>/rigs/<asset>.glb` rebuilt in the canonical frame of its clips (same joints, facing, centring, scale and grounding), as `dataset/canonical_assets/<dataset>/<object_type>.glb`, without animation. Generated motion drives it without retargeting. Mixamo gets one per character on the 22 core joints. A GLB is rebuilt only when its cond entry or source asset changes. For a large dataset, shard with `--worker_id` / `--num_workers`.<details>-<summary><b>Details โ</b> Mixamo core joints, grounding, previews, and resume</summary>--- **Mixamo** is reduced to a built-in **22-joint humanoid core** (finger chains and End bones dropped) via `--mixamo_core_joints` (default on); pass `--no-mixamo_core_joints` to keep the full 65-joint rig. Truebones and objaverse always use the full skeleton.-- **Grounding.** Each motion is grounded on its own lowest joint by default; `--use_tpos_ground_height` reuses the T-pose's ground height for every clip of the object type instead. Either way `cond['ground_height']` and `cond['ground_height_mode']` record what was applied.-- **Previews.** Per-clip MP4s use a ground-plane view with a following camera, contact shadow and fading root trajectory (`--no-vis_ground` for the plain cubic view; `NO_VIS=1` skips them entirely).-- **Resume.** Finished object types are cached under `cond_parts/`, keyed on the clip / threshold / topology settings, on digests of the caption and joint-annotation inputs, **and** on the object's source clip list โ so a stage-2/3 rerun, a new QA patch or an added clip re-processes the affected object types instead of silently mixing old and new results. Re-processing first deletes the clip NPZs a previous run wrote for that object, since the training loader enumerates `motions/` rather than `captions.json`. Object types that fail are appended to `extract_errors.log` and retried next run โ one bad object never aborts the run.+<summary>Filters and options</summary>++| Option | Default | Effect |+|---|---|---|+| `--min_joints` / `--max_joints` | 8 / 150 (objaverse, general: 4 / 180) | object types outside the range are skipped |+| `--max_clip_len` | 200 | frames kept per motion; the rest is dropped unless `APPLY_CLIP=1` |+| `APPLY_CLIP=1` | off | windows of `max_clip_len` frames with stride `max_clip_len โ diffusion_max_len` (110), so every training crop fits in a saved clip |+| `--activity_threshold` | 0.02 | minimum joint activity at canonical scale |+| `--jump_step_threshold` / `--jump_ratio_threshold` | 0.20 / 8.0 | drop a clip when one frame moves the skeleton this many body lengths and this many times its median step (`0` disables) |+| `--static_threshold` | 1e-5 | per-frame displacement below which lead-in / lead-out frames are trimmed |+| `--min_frames` | 8 | minimum frames after trimming |+| `--target_diameter` | 2.0 | skeleton diameter after scaling |+| `--max_freqs` / `--max_path_len` | 8 / 5 | Laplacian eigenvectors per joint; clamp of the graph distances |++- **Skip lists.** `filtered_clips.txt`, `filtered_objects.txt`, `clip_trims.txt` and `activity_keep.txt` from the export directory are optional; each flag takes a path, `auto` (default) or `""` to disable.+- **Mixamo** is reduced to its 22-joint core (`--no-mixamo_core_joints` keeps all 65 joints).+- **Grounding.** Each clip is grounded on its own lowest joint; `--use_tpos_ground_height` uses the T-pose's instead. `cond['ground_height_mode']` records which.+- **Resume.** Finished object types are cached in `cond_parts/`, keyed on the settings, the annotation inputs and the clip list, so changed inputs are reprocessed. Failures go to `extract_errors.log` and are retried on the next run.</details>-### Stage 5 โ Animate: drive a mesh with a motion--Used at inference time to render generated motions onto their rigged meshes:+## Stage 5: Animate```bash-ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse # stage-4 / generated clip + cond.npy-CHAR_PATH=char.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ rig-bash data_process/scripts/run_animate_fbx.sh # raw animation FBXs โ any character-bash data_process/scripts/run_animate_mixamo.sh # batch over Mixamo export NPZs (Y Bot)-CHAR_PATH=asset.glb OUTPUT_DIR=outputs/asset bash data_process/scripts/run_preprocess_char.sh # one asset โ cond.npy + canonical rig-CHAR_PATH=outputs/asset/asset_canonical.glb ANIM_PATH=motion.npz bash data_process/scripts/run_animate_lbs.sh # NumPy FK + LBS, no cond needed+bash scripts/run_animate_motion.sh outputs/<run>/samples # every sampled motion onto its asset+ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse # one feature clip or sample + cond.npy+ANIM_PATH=clip.npz DATASET_TYPE=truebones bash data_process/scripts/run_animate_npz.sh # stage 1 NPZ โ its export asset+CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx bash data_process/scripts/run_animate_fbx.sh # raw FBX clips โ a character+bash data_process/scripts/run_animate_mixamo.sh # Mixamo export NPZs โ Y Bot```-`run_animate_motion.sh` accepts both the stage-4 feature NPZ and the `.npy` motion features written by the model's sampler. The character is auto-resolved for truebones / objaverse from the raw asset directories; mixamo defaults to `CHAR_PATH=dataset/raw/mixamo/character_refined/Y_Bot.fbx` (the rig the animation FBXs are authored on) when that file has been downloaded, and requires `CHAR_PATH` otherwise. The last two wrappers are described under [Custom Assets](#custom-assets).+`run_animate_motion.sh` takes stage 4 clips and the sampler's `.npy` motions. With a sampler output directory, `scripts/run_animate_motion.sh` reads its `manifest.json`, drives each motion's own asset, and stops before Blender starts if a motion, its cond and its asset disagree on the joint order.-Armature bones absent from the motion are handled by `--extra_bones_strategy` on `animate_motion.py` / `animate_npz.py` (`EXTRA_BONES_STRATEGY=` on their wrappers and on `scripts/run_animate_motion.sh`): `merge` (default) transfers their vertex weights to the nearest kept ancestor and removes them, `remove` deletes the bones and their vertices, `keep` leaves them un-keyed. The FBX / raw-clip paths (`animate_fbx`, `animate_mixamo`, `animate_lbs`) leave undriven bones at rest.+The character is chosen by `ASSET`:-`run_animate_fbx.sh` needs no export stage at all: it bakes a raw animation clip โ or a whole directory of them โ onto any character that shares the clips' bone names (the Mixamo convention):+- `canonical` (default): `dataset/canonical_assets/<dataset>/<object_type>.glb` (Mixamo: the character `CHARACTER`, default `Michelle`), at canonical scale. Without one it falls back to `export` with a warning; `CANONICAL=1` makes that an error.+- `export` (`CANONICAL=0`): the stage 1 `rigs/<asset>.glb` if present, else the raw asset (Mixamo: the processed character, else, for the default character only, `character_refined/Y_Bot.fbx`), with the motion scaled to the export's size.+- `CHAR_PATH` overrides both.-```bash-CHAR_PATH=dataset/raw/mixamo/character_refined/Amy.fbx \- bash data_process/scripts/run_animate_fbx.sh # โ outputs/animated_Amy/*.fbx-```+In `run_animate_motion.sh` and `run_animate_npz.sh`, armature bones the motion lacks follow `EXTRA_BONES_STRATEGY`: `merge` (default; their skin weights go to the nearest driven ancestor), `remove` (bones and their vertices deleted) or `keep` (left at rest). The other paths leave them at rest.-## Custom Assets+## Custom assets-Nothing in the pipeline is tied to the three datasets. There are three entry points for your own rigged assets, in increasing order of integration.+### Train on your own data (optional)-**1 ยท Export only** โ turn assets into stage-1 NPZs. A single file or a directory of mixed GLB/GLTF/FBX; mesh optional (armature-only FBX works), every pose action becomes a clip.+To develop a model with your own data, the pipeline has an optional dataset slot, `general`, processed like Objaverse. Put one rigged, animated asset per file (GLB with named animations; glTF and FBX work too; a mesh is needed for rendering) in `dataset/raw/general/animation/` and run every stage with `general`:```bash-bash data_process/scripts/run_export_general.sh my_model.glb-bash data_process/scripts/run_export_general.sh my_assets/ --multi-worker 8 # โ dataset/export/custom+bash data_process/scripts/run_export.sh general --multi-worker 8+bash data_process/scripts/run_render_motion.sh general && bash data_process/scripts/run_render_tpose.sh general+bash data_process/scripts/run_caption_motion.sh general && bash data_process/scripts/run_caption_category.sh general+bash data_process/scripts/run_joints_names_clean_llm.sh general && bash data_process/scripts/run_joints_face_select_llm.sh general+bash data_process/scripts/run_extract_features.sh general```-The general exporter uses the same `{asset}-{action}.npz` clip naming as the Objaverse layout, so the later stages run over a custom export by borrowing `objaverse` as the layout name and overriding the paths:--```bash-EXPORT=dataset/export/custom-INPUT=$EXPORT/joint_names.json OUTPUT=$EXPORT/clean_joint_names.json \- bash data_process/scripts/run_joints_names_clean_llm.sh objaverse-INPUT_DIR=$EXPORT bash data_process/scripts/run_joints_face_select_llm.sh objaverse-DATA_DIR=$EXPORT SAVE_DIR=dataset/features/custom \- bash data_process/scripts/run_extract_features.sh objaverse-```+Clips are named `<asset>-<animation>`, the asset name being the file stem with `-` and spaces replaced by `_`; names must be unique and must not clash with an object type of another dataset. Add `"general"` to `dataset.dataset_list` in a config to train on it.-The dataset argument selects the layout and the per-dataset defaults, not the data โ every wrapper validates it against `truebones | mixamo | objaverse`, and each documents the environment variables that redirect its input and output.+### Animate your asset with a trained model-**2 ยท Preprocess one asset for inference** โ `run_preprocess_char.sh` runs the real export and extraction stages in-process on a single rigged, animated asset:+[`rig_preprocess`](rig_preprocess/README.md) turns one rigged GLB, glTF or FBX, animated or not, into an asset directory the sampler takes (`cond.npy`, a canonical `<name>.glb`, `preview.png`, `summary.json`), using the code of stages 1, 3 and 4:```bash-CHAR_PATH=asset.glb OUTPUT_DIR=outputs/asset \- FACE_R=R_Thigh FACE_L=L_Thigh FORMATS=glb,fbx \- bash data_process/scripts/run_preprocess_char.sh+INPUT=robot.glb OUTPUT_DIR=outputs/rig/robot bash data_process/scripts/run_rig_preprocess.sh # stops for review+INPUT=robot.glb OUTPUT_DIR=outputs/rig/robot ANNOTATION=outputs/rig/robot/annotation.json \+ bash data_process/scripts/run_rig_preprocess.sh # builds+python -m unimate.inference.sample --exp_dir <run> --asset outputs/rig/robot --prompt "An object walks forward."+bash scripts/run_animate_motion.sh <run>/samples```-It leaves exactly three deliverables: `<name>_canonical.{glb,fbx}` (the rest-pose asset rebuilt to the canonical T-pose, joint order stored as a custom property), `cond.npy`, and `motions/<clip>.npz` per action. `FACE_R` / `FACE_L` name the asset's facing pair (`BODY_AXIS=1` for a head/tail axis). An asset **without** any action falls back to a rest-only cond and writes no `motions/`.+Joint labels and the facing pair come from an LLM (`DEEPSEEK_API_KEY` or `OPENAI_API_KEY`; offline rules without one). A run stops before the cond is built so they can be checked and corrected in `annotation.json`, by hand or with an LLM or coding agent (see its `REVIEW.md`); `ANNOTATION=<file>` continues from the corrected file, `REVIEW=0` builds directly. See its [README](rig_preprocess/README.md) for every option.++### Drive a mesh with a feature clip-**3 ยท Animate the canonical asset** โ a feature-format motion NPZ (generated by the model, or extracted above) then drives it directly, with no `cond.npy` or export NPZ at animate time:+`run_animate_lbs.sh` drives a canonical asset with stage 4 feature clips, without `cond.npy`, computing forward kinematics and linear blend skinning in NumPy:```bash-CHAR_PATH=outputs/asset/<name>_canonical.glb ANIM_PATH=<motion.npz-or-directory> \- bash data_process/scripts/run_animate_lbs.sh # one animated GLB per action+CHAR_PATH=outputs/rig/robot/robot.glb ANIM_PATH=<clip.npz or directory> bash data_process/scripts/run_animate_lbs.sh```-`animate_lbs` computes FK and Linear Blend Skinning manually in NumPy (Blender only parses the asset; the math is numerically identical to Blender's armature modifier). `SAVE` is the full list of outputs: `SAVE=glb,npz,obj` writes the rigged GLB plus raw deformed vertex sequences and per-frame OBJs (`SAVE=npz,obj` alone drops the GLB). A feature NPZ that is not from a canonical asset can still be driven by passing `DATASET_TYPE=` and/or `COND_PATH=` (a single-entry `cond.npy` is accepted whatever the clip is named).+`SAVE=glb,npz,obj` adds the deformed vertices as an NPZ and per-frame OBJs. Sampled `.npy` motions go through `run_animate_motion.sh` instead.-## Data Formats--Three artifacts matter downstream: the **stage-1 export NPZ** (raw per-clip skeleton animation), the **stage-4 feature NPZ** (canonicalized per-clip arrays the training loader reads), and **`cond.npy`** (one topology-conditioning dict per object type).+## Data formats<details>-<summary><b>Reference โ</b> stage-1 export NPZ (<code>dataset/export/<dataset>/motions/*.npz</code>)</summary>+<summary>Stage 1 export NPZ (<code>dataset/export/<dataset>/motions/*.npz</code>)</summary>| Field | Shape | Description |-|-------|-------|-------------|+|---|---|---|| `rest_local_pos` / `rest_local_rot` | `(J, 3)` / `(J, 4)` | rest-pose local translations / rotations (quaternions) || `anim_local_pos` / `anim_local_rot` | `(T, J, 3)` / `(T, J, 4)` | per-frame local translations / rotations || `offsets` | `(J, 3)` | bind-pose bone offsets || `parents` | `(J,)` | parent indices, `-1` for the root |-| `names` | `(J,)` | bone name strings |-| `skin_matrix` | `(V, J)` | skinning weights (empty when no mesh) |-| `fps`, `action_name` | scalar | frame rate and source action |+| `names` | `(J,)` | bone names |+| `skin_matrix` | `(V, J)` | skin weights (empty without a mesh) |+| `fps`, `action_name` | scalar | frame rate, source action |</details><details>-<summary><b>Reference โ</b> stage-4 feature NPZ (<code>dataset/features/<dataset>/motions/{object}-{motion}-{clip_idx}.npz</code>)</summary>+<summary>Stage 4 feature NPZ (<code>dataset/features/<dataset>/motions/*.npz</code>)</summary>| Field | Shape | Description |-|-------|-------|-------------|+|---|---|---|| `global_positions` | `(F, J, 3)` | global joint positions at canonical scale || `local_rotations` | `(F, J, 4)` | local joint rotations (quaternions) |-| `root_facing_quat` | `(F, 4)` | per-frame root quaternion aligning the skeleton to face Z+ |-| `fps` | scalar | frame rate of the clip, after halving anything โฅ 60 fps down below it (30 for every current source) |+| `root_facing_quat` | `(F, 4)` | per-frame root rotation that faces the skeleton to +Z |+| `fps` | scalar | frame rate; a source at 60 fps or more is halved until below 60 (30 for every current source) |-The motion representation stores per-frame deltas without dividing by the frame rate, so the frame rate is part of the feature scale โ the training loader rejects clips that are not 30 fps rather than mixing scales.+The motion features store per-frame differences, so the frame rate is part of their scale: the training loader rejects clips that are not 30 fps.</details><details>-<summary><b>Reference โ</b> <code>cond.npy</code> (one dict per object type)</summary>+<summary><code>cond.npy</code> (one dict per object type)</summary>| Key | Shape | Description |-|-----|-------|-------------|-| `object_type` | str | object-type name (clip-name prefix) |-| `parents` | list, len `J` | parent indices in canonical BFS joint order (`-1` = root) |-| `offsets`, `tpos_offsets` | `(J, 3)` float64 | local bone offsets of the canonicalized T-pose (scaled, root moved by centering / grounding) and offsets recomputed from `tpos_first_frame` |-| `joint_names`, `clean_joint_names` | list, len `J` | raw and cleaned bone names (stage 3) |-| `tpos_first_frame` | `(J, 3)` float64 | canonical T-pose global joint positions |-| `tpos_local_rotations`, `tpos_global_rotations` | `(J, 4)` float64 | T-pose local / global rotations (quaternions, `w, x, y, z`) |-| `joint_relations`, `joint_graph_dists` | `(J, J)` int16 | pairwise edge-relation types and topology distances clamped at `--max_path_len` |-| `joint_depths` | `(J,)` int64 | depth in the kinematic tree |-| `edge_indexs` | `(2, 2(Jโ1))` int64 | undirected edge list |-| `spectral_feats` | `(J, K)` float32 | Laplacian eigenvectors (`K = --max_freqs`) |-| `kinematic_chains` | list | root-to-leaf joint chains |-| `scale_factor`, `ground_height`, `ground_height_mode` | scalar | calibration applied to every clip of the type |-| `face_joint_idxs` | dict | `{r_hip, l_hip, body_axis}` indices of the facing pair; `-1, -1` when the rig has no resolved pair (identity facing) |-| `captions` | dict | `{clip_stem: caption}` for the type's saved clips (also flattened into `captions.json`) |+|---|---|---|+| `object_type` | str | object-type name (the clip-name prefix) |+| `parents` | list, `J` | parent indices in canonical BFS order (`-1` = root) |+| `offsets`, `tpos_offsets` | `(J, 3)` | local offsets of the canonical T-pose; offsets recomputed from `tpos_first_frame` |+| `joint_names`, `clean_joint_names` | list, `J` | raw and cleaned joint names |+| `tpos_first_frame` | `(J, 3)` | canonical T-pose joint positions |+| `tpos_local_rotations`, `tpos_global_rotations` | `(J, 4)` | T-pose rotations (quaternions `w, x, y, z`) |+| `joint_relations`, `joint_graph_dists` | `(J, J)` int16 | edge-relation types; graph distances clamped at `--max_path_len` |+| `joint_depths` | `(J,)` | depth in the tree |+| `edge_indexs` | `(2, 2(Jโ1))` | undirected edges |+| `spectral_feats` | `(J, K)` | Laplacian eigenvectors (`K = --max_freqs`) |+| `kinematic_chains` | list | root-to-leaf chains |+| `scale_factor`, `ground_height`, `ground_height_mode` | scalar | normalization applied to every clip of the type |+| `face_joint_idxs` | dict | `{r_hip, l_hip, body_axis}`; `-1, -1` without a facing pair (identity facing) |+| `captions` | dict | `{clip: caption}` for the saved clips (also in `captions.json`) |</details><details>-<summary><b>Reference โ</b> files in <code>dataset/export/<dataset>/</code></summary>--The export directory is where every stage before feature extraction accumulates its output, and it is the single input directory stage 4 reads.+<summary>Files in <code>dataset/export/<dataset>/</code></summary>| Path | Stage | Content |-|------|-------|---------|-| `motions/{clip}.npz` | 1 | per-clip skeleton animation |-| `videos/{clip}.mp4`, `tpose/{asset}.png` | 1 | per-clip skeleton preview, rest-pose still per asset |-| `joint_names.json` | 1 | `{asset: [pruned bone names]}` |-| `joint_count.json`, `clip_frames.json`, `summary.json` | 1 | joints per skeleton, frames per clip, dataset totals |-| `.completed/{asset}.json` | 1 | resume markers |-| `motion_captions.json`, `motion_captions_failed.txt` | 2b | `{clip: caption}`, clips to retry |-| `category_groups.json` | 2b | `{category: [assets]}` |-| `category_groups_review.json`, `category_groups_errors.json` | 2b | per-asset votes/evidence, exhausted retries |-| `clean_joint_names.json` | 3 | canonical anatomical labels per rig |-| `face_joint_names.json` | 3 | facing pair (or body axis) per rig |+|---|---|---|+| `motions/<clip>.npz`, `videos/`, `tpose/` | 1 | clips and previews |+| `joint_names.json`, `joint_count.json`, `clip_frames.json`, `summary.json` | 1 | bone names, counts, totals |+| `rigs/<asset>.glb` | 1 | with `--save_glb`: the rest-pose asset |+| `motion_captions.json`, `category_groups.json` | 2b | captions, body-plan categories |+| `motion_captions_generic.json`, `motion_captions_detail.json` | QA | a second, generic and a third, detailed caption per clip, sampled in training with `generic_caption_prob` / `detail_caption_prob` (released with the export) |+| `clean_joint_names.json`, `face_joint_names.json` | 3 | joint labels, facing pair (or body axis) || `failed_clean_names.txt`, `failed_face_joints.txt` | 3 | rigs that fell back to the rules |-| `filtered_clips.txt` | QA patch | clips stage 4 skips |-| `clip_trims.txt` | QA patch | clips whose first N frames stage 4 drops |-| `activity_keep.txt` | QA patch | clips stage 4's low-activity filter keeps |-| `filtered_objects.txt`, `rig_flags.json` | QA patch | objaverse only: rigs stage 4 skips, and the full flag list |+| `filtered_clips.txt`, `clip_trims.txt`, `activity_keep.txt`, `filtered_objects.txt`, `rig_flags.json` | QA | stage 4 skip lists and rig flags |</details>## Tools-Standalone utilities in `data_process/tools/`. `vis_tpose.py` and `vis_tpose_facing.py` have wrappers (listed under [Stage 3](#stage-3--joint-annotation-name-cleanup--facing-direction)); the rest are run directly.--| Tool | Purpose |-|------|---------|-| `merge_summaries.py` | Merge per-worker export summary shards into the canonical `joint_names` / `joint_count` / `clip_frames` / `summary` JSONs, self-healing from the `.completed/` markers. Additive and idempotent |-| `patch_annotations.py` | Post-stage-3 QA patch: rule fixes plus hand overrides for joint labels, face pairs, captions and categories; writes the stage-4 skip lists |-| `eval_category_groups.py` | Score a `category_groups.json` against a truth set, or diff two runs |-| `vis_tpose.py` | Annotated T-pose PNG with every joint labeled, per skeleton |-| `vis_tpose_facing.py` | Rest pose vs. facing-canonicalized pose per skeleton, plus `facing_summary.tsv` |-| `vis_motion.py` | MP4 of an export clip with a root-anchored heading arrow |-| `vis_clip_frames.py` / `vis_joint_count.py` | Distribution figures for frames per clip / joints per skeleton. Both default over every `dataset/export/*`, write each dataset's figure beside its JSON, and put the comparison figure under `outputs/{clip_frames,joint_count}_vis/` |-| `truebones_fbx2glb.py` / `character_fbx2glb.py` | Per-clip Truebones FBX โ GLB (mesh, armature and take preserved); rigged T-pose character FBX โ GLB |--```bash-python -m data_process.tools.merge_summaries --output_dir dataset/export/<dataset>-python data_process/tools/patch_annotations.py --dry_run-python -m data_process.tools.vis_joint_count # defaults over every dataset/export/*-blender -b -P data_process/tools/truebones_fbx2glb.py -- --data_dir <in> --output_dir <out>-```+| Tool (`data_process/tools/`) | Purpose |+|---|---|+| `patch_annotations.py` | QA fixes and hand corrections for labels, facing pairs, captions and categories; writes the stage 4 skip lists |+| `merge_summaries.py` | Rebuild an export's summary JSONs from worker shards and completion markers (idempotent) |+| `sync_captions.py` | Refresh `features/<dataset>/captions*.json` from the export's captions without rerunning stage 4 |+| `eval_category_groups.py` | Score `category_groups.json` against a truth set, or compare two runs |+| `vis_tpose.py`, `vis_tpose_facing.py` | Labelled T-pose per skeleton; rest vs. canonical facing (wrapped by `run_joints_vis_*.sh`) |+| `vis_motion.py` | MP4 of an export clip with a heading arrow (`--direction face --face-joints I J`) |+| `vis_clip_frames.py`, `vis_joint_count.py` | Distributions of frames per clip and joints per skeleton |+| `truebones_fbx2glb.py`, `character_fbx2glb.py` | Truebones clip FBX โ GLB; rigged character FBX โ GLB (run with `blender -b -P`) |## Conventions-- **Run from the repo root.** Wrappers `cd` there themselves and activate the `unimate` conda env (`CONDA_ENV` overrides).-- **Self-documenting wrappers.** Every wrapper documents its overridable paths and knobs in its header comment; run any wrapper with `-h` to print it.-- **Resumable by default.** Every batch stage skips outputs that already exist and records failures in sibling error logs, so failed items are retried on the next run without redoing finished work.-- **Parallelism.** Blender stages shard with `--multi-worker N`; local-model captioning shards with `--multi-gpu`; API backends use concurrent workers via `NUM_WORKERS`; stage 4 parallelizes over object types with `NUM_WORKERS`.-- **Naming.** Clips are named `{object_type}-{action}` (stage 1) and `{object_type}-{motion}-{clip_idx}` (stage 4). Mixamo is a single shared skeleton, so its files carry no object-type prefix.-- **Dependency direction.** `data_process/` is self-contained and never imports from the training package; kinematics come from the external [`Motion`](https://github.com/inbar-2344/Motion) library.+- **Wrappers** activate the `unimate` environment (`CONDA_ENV` overrides) and document their variables in their header (`-h`).+- **Resumable.** Batch stages skip finished outputs and log failures for the next run.+- **Parallelism.** Blender stages: `--multi-worker N`. Local captioning: `--multi-gpu`. API backends and stage 4: `NUM_WORKERS`.+- **Naming.** Stage 1 clips are `<object_type>-<action>`, stage 4 clips `<object_type>-<motion>-<clip_idx>`; object types contain no `-`. Mixamo clips have no prefix (one shared skeleton).+- **Dependencies.** `data_process/` does not import the training package; kinematics come from the [`Motion`](https://github.com/inbar-2344/Motion) library.## Troubleshooting-- **EEVEE renders fail / produce black frames under `blender -b`** โ stage 2a must run with plain `python` and the pip `bpy` module (the wrappers already do); headless `blender -b` has no GPU display surface for EEVEE.-- **All render workers land on one GPU** โ `CUDA_VISIBLE_DEVICES` does not bind EEVEE's EGL context. Run one render job per GPU through a cgroup / scheduler allocation that exposes a single device.-- **Offline compute nodes** โ with a populated HuggingFace cache, export `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` before the local Qwen caption/classify runs.-- **A multi-worker export exited non-zero, or `joint_names.json` is missing object types that have clips** โ the wrapper deliberately does not merge summary shards when a worker failed, and an export killed mid-run can leave the summary JSONs behind the completion markers. Rerun to finish the missing assets, then rebuild the JSONs from the markers (additive and idempotent):- ```bash- python -m data_process.tools.merge_summaries --output_dir dataset/export/<dataset>- ```-- **Stage 4 re-processes objects that look unchanged** โ the `cond_parts/` cache key covers thresholds, topology settings, the caption/annotation digests and the object's clip list. The run logs the exact diff per object (`cache invalidated (settings changed: ...)`).-- **`Object type mismatch between ...`** โ stage 4 refuses to run when `joint_names.json`, `clean_joint_names.json` and `face_joint_names.json` cover different object types. Rerun the stage-3 wrappers on the current export directory.--If you run into a problem not covered here, please open an issue or email [[email protected]](mailto:[email protected]). Citation and license information is in the [top-level README](../README.md).+| Problem | Fix |+|---|---|+| EEVEE renders fail or are black | Run stage 2a with `python` and the pip `bpy` module (the wrappers do), not `blender -b` |+| All render workers use one GPU | EEVEE's EGL context ignores `CUDA_VISIBLE_DEVICES`: run one job per single-GPU allocation |+| Local captioning on an offline node | Populate the Hugging Face cache, then set `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` |+| A multi-worker export exited non-zero, or `joint_names.json` lacks assets that have clips | Rerun to finish the failed assets, then `python -m data_process.tools.merge_summaries --output_dir dataset/export/<dataset>` |+| Stage 4 reprocesses unchanged objects | Its cache key changed; the log names the cause (`cache invalidated (settings changed: ...)`) |+| `Object type mismatch between ...` | `joint_names.json` and the stage 3 files cover different assets: rerun stage 3 on this export |++For other problems, open an issue or email [[email protected]](mailto:[email protected]). Citation and license: [top-level README](../README.md).diff --git a/data_process/feature_extraction/canonical_assets.py b/data_process/feature_extraction/canonical_assets.pynew file mode 100644index 0000000..f097e76--- /dev/null+++ b/data_process/feature_extraction/canonical_assets.py@@ -0,0 +1,335 @@+"""Bake the export stage's rest-pose assets into the stage-4 canonical frame.++For every object type in ``<features_dir>/cond.npy`` whose processed asset+exists in ``<export_dir>/rigs/`` (``run_export.sh <ds> --save_glb`` /+``--glb_only``), write ``<assets_dir>/<object_type>.glb``: the asset rebuilt+to the cond's canonical T-pose (the joint order, facing, XZ centring, scale+and grounding of the training clips), without animation+(``mesh_animation.canonical_rig.bake_canonical_asset``). A stage-4 feature+clip of that object type (``<features_dir>/motions/*.npz``: ``local_rotations``++ ``global_positions``) then drives it with no ``cond.npy``; the sampler's+``.npy`` features go through ``animate_motion`` and the cond instead::++ python -m data_process.mesh_animation.animate_lbs \\+ --char_path dataset/canonical_assets/<ds>/<type>.glb \\+ --anim_path dataset/features/<ds>/motions/<type>-<motion>-000.npz++The joint order rides in the GLB as ``canonical_joint_order`` and a digest of+its inputs (the cond entry, its diameter, the source GLB's bytes and+``BAKE_VERSION``) as ``canonical_cond_digest``: an asset whose cond or source+changed is rebuilt on the next run, the others are skipped. A skeleton whose+roll about its joint line the fit cannot determine (collinear joints, or nearly+collinear ones for the fit's residual) is refused.++Mixamo has one object type, ``mixamo`` (the 22 core joints of the rig the clips+were made on, or the full rig if stage 4 kept it), but many characters: every+character in the export's ``rigs/`` gets ``<assets_dir>/<character>.glb``.+Its joint positions are its OWN rest pose through the stage-4 T-pose+canonicalization (the cond's joints, facing pair and diameter, grounding), and+its bone frames are the shared cond's canonical rotations, so a Mixamo feature+clip's local rotations drive every character directly as the change from rest,+the character keeping its proportions (Y_Bot, the clips' rig, reproduces the+clip exactly). The clips' rest root (the cond's T-pose root) rides along as+``canonical_motion_rest_root``: the root then moves by the clip's change from+that rest, from the character's own rest root (``animate_lbs`` reads it), so a character whose hips sit lower than Y_Bot's does not float.++Bones the cond does not have are merged into kept ones. Unskinned meshes+parented under the armature (rigid parts) are bound to their parent bone+before the bake, and an asset without any mesh on the armature fails. A+failure is recorded in ``<assets_dir>/glb_errors/<name>.txt`` (``<name>``: the+object type, for Mixamo the character) and removes a GLB built from older+inputs; an image the canonical GLB lost relative to its source goes to+``<assets_dir>/texture_issues/<name>.txt``.++``<assets_dir>`` defaults to :func:`default_assets_dir`: ``canonical_assets/<ds>``+beside a features root, ``dataset/features/truebones`` giving+``dataset/canonical_assets/truebones``; ``--assets_dir`` overrides.++Usage (pip bpy or Blender; GLB-only, never touches the stage-4 outputs):+ python -m data_process.feature_extraction.canonical_assets \\+ --export_dir dataset/export/truebones --features_dir dataset/features/truebones+``extract_features.py --save_glb`` runs the same after stage 4, in one process;+for a large dataset run this module sharded instead (``--worker_id`` /+``--num_workers``, one short process per shard).+"""++import argparse+import copy+import hashlib+import json+import os+import sys++import numpy as np+from loguru import logger++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from Animation import Animation, positions_global # noqa: E402+from Quaternions import Quaternions # noqa: E402++from data_process.mesh_animation.canonical_rig import UndeterminedRoll, bake_canonical_asset # noqa: E402+from data_process.utils.asset_files import ( # noqa: E402+ GLB_ERRORS_DIR, PROCESSED_GLB_DIR, TEXTURE_ISSUES_DIR,+ default_assets_dir, glb_image_digests, glb_node_extra, write_note,+)+from data_process.utils.blender_export import ( # noqa: E402+ import_gltf, load_scene, prepare_skeleton,+)+from data_process.utils.motion_features import process_tpose # noqa: E402+from data_process.utils.skeleton import get_skeleton_diameter, line_ratio # noqa: E402++DIGEST_KEY = 'canonical_cond_digest'+# Mixamo characters: the clips' rest root in the canonical frame (module docstring).+MOTION_ROOT_KEY = 'canonical_motion_rest_root'+# Part of every digest: bump it when the bake's output changes for the same+# inputs, so the next run rebuilds every GLB.+BAKE_VERSION = 'v3'+++def cond_digest(entry):+ """sha1 of what the baked asset depends on: joint names, hierarchy and+ the canonical T-pose (rounded to 1e-6)."""+ h = hashlib.sha1()+ h.update(json.dumps([str(n) for n in entry['joint_names']]).encode())+ h.update(np.asarray(entry['parents'], dtype=np.int64).tobytes())+ for key in ('tpos_first_frame', 'tpos_global_rotations'):+ h.update(np.round(np.asarray(entry[key], dtype=np.float64), 6).tobytes())+ return h.hexdigest()+++def cond_diameter(entry):+ """The leaf-to-leaf diameter of the cond's canonical T-pose: stage 4's+ ``--target_diameter``, recovered from the cond itself (rounded to 1e-4)."""+ offsets = np.asarray(entry['tpos_offsets'], dtype=np.float64)+ J = len(offsets)+ anim = Animation(Quaternions.id((1, J)), offsets[None], Quaternions.id(J),+ offsets, np.asarray(entry['parents']))+ return round(float(get_skeleton_diameter(anim)), 4)+++def cond_face_joints(entry):+ """The cond's facing pair in the ``face_joint_names.json`` form that+ :func:`process_tpose` reads (raw names), or ValueError when it has none."""+ face = entry.get('face_joint_idxs') or {}+ names = [str(n) for n in entry['joint_names']]+ r, l = int(face.get('r_hip', -1)), int(face.get('l_hip', -1))+ if not (0 <= r < len(names) and 0 <= l < len(names)):+ raise ValueError("the cond has no facing pair (face_joint_idxs); the characters "+ "could not be faced like the clips")+ return {'r_hip': {'raw': names[r]}, 'l_hip': {'raw': names[l]},+ 'body_axis': bool(face.get('body_axis', False))}+++def file_digest(path):+ """sha1 of a file's bytes."""+ h = hashlib.sha1()+ with open(path, 'rb') as f:+ for block in iter(lambda: f.read(1 << 20), b''):+ h.update(block)+ return h.hexdigest()+++def mixamo_character_cond(char_glb, cond):+ """The cond a Mixamo character is baked to (module docstring): joint+ positions from its own rest pose canonicalized like stage 4's T-pose (the+ cond's joints, facing pair and diameter), bone rotations from the shared+ *cond*.++ Raises:+ ValueError: the character lacks one of the cond's joints, or its+ joints have other names or another hierarchy than the cond's+ (compared by name: sibling order may differ).+ """+ armature, mesh = load_scene(import_gltf, char_glb, fps=30)+ skel = prepare_skeleton(armature, mesh, apply_world=True)+ names = [str(n) for n in skel['bone_names']]+ # The cond's joints: the 22 MIXAMO_CORE_JOINTS by default, the full rig+ # when stage 4 ran with --no-mixamo_core_joints.+ want = [str(n) for n in cond['joint_names']]+ missing = [n for n in want if n not in names]+ if missing:+ raise ValueError(f"{len(missing)} of the cond's {len(want)} joints missing: {missing[:3]}")+ rest = skel['rest_anim_shared'] # export convention (Y-up), as the NPZs+ tpos_data = {+ 'names': np.array(names), 'parents': np.asarray(skel['parents_array']),+ 'rest_local_pos': np.asarray(rest.positions[0]),+ 'rest_local_rot': np.asarray(rest.rotations.qs[0]), 'fps': np.array(30),+ }+ (canon_anim, _offsets, _scale, _ground, canon_parents, canon_names,+ *_rest) = process_tpose(+ tpos_data, corps_idxs=[names.index(n) for n in want],+ corps_names=want, face_joints=cond_face_joints(cond),+ target_diameter=cond_diameter(cond))+ # Same joints and hierarchy, matched by NAME: BFS breaks sibling ties by+ # the source's bone order, which differs between files (a character GLB+ # lists Right* before Left*, the clips' FBX the other way round), and the+ # canonicalization itself does not depend on joint order.+ got = [str(n) for n in canon_names]+ parent_of = {n: (got[p] if p >= 0 else None) for n, p in zip(got, canon_parents)}+ want_parent = {n: (want[int(p)] if int(p) >= 0 else None)+ for n, p in zip(want, cond['parents'])}+ if sorted(got) != sorted(want) or parent_of != want_parent:+ raise ValueError("joints differ from the shared cond's (names or hierarchy)")+ positions = np.asarray(positions_global(canon_anim)[0])+ out = copy.copy(cond)+ out['tpos_first_frame'] = positions[[got.index(n) for n in want]]+ return out+++def glb_digest(path):+ """The ``canonical_cond_digest`` stored in a baked GLB (glTF extras), or None."""+ return glb_node_extra(path, DIGEST_KEY)+++class CollinearSkeleton(UndeterminedRoll):+ """A skeleton the bake refuses (see :func:`_check_not_collinear`)."""+++def _check_not_collinear(entry, rel_tol=1e-3):+ """The bake fits the asset onto the canonical joints with one similarity;+ joints on a straight line leave the roll about that line undetermined, and+ the mesh would come out at an arbitrary roll, so such skeletons are refused."""+ if line_ratio(entry['tpos_first_frame']) <= rel_tol:+ raise CollinearSkeleton("collinear skeleton: its roll about the joint line is "+ "undetermined, so the mesh cannot be fitted to the canonical frame")+++def _texture_note(assets_dir, name, source, baked):+ """Images the baked GLB lost relative to its source GLB. The glTF exporter+ may rename an image it re-embeds (``dressrun.2`` -> ``dressrun``), so an+ image counts as kept when its bytes are embedded under any name."""+ kept = {digest for _, digest in glb_image_digests(baked)}+ lost = sorted({n for n, digest in glb_image_digests(source) if digest is None or digest not in kept})+ write_note(assets_dir, TEXTURE_ISSUES_DIR, name,+ [f"images of {os.path.basename(source)} not in the canonical GLB: {lost}"] if lost else [])+ if lost:+ logger.warning(f"'{name}': canonical GLB lost image(s) {lost}")+++def bake_canonical_assets(export_dir, features_dir, assets_dir=None, dataset_type=None,+ object_types=None, worker_id=0, num_workers=1, overwrite=False,+ formats=('glb',)):+ """Write ``<assets_dir>/<object_type>.glb`` (module docstring; *assets_dir*+ defaults to :func:`default_assets_dir`), plus ``<object_type>.fbx`` when+ *formats* holds ``'fbx'`` (the GLB is always written: it carries the digest).++ Returns ``{'baked': n, 'up_to_date': n, 'no_source': n, 'refused': n,+ 'failed': n}`` (refused: collinear skeletons, an expected outcome).+ """+ if not 0 <= worker_id < num_workers:+ raise ValueError(f"worker_id {worker_id} not in [0, {num_workers})")+ cond = np.load(os.path.join(features_dir, 'cond.npy'), allow_pickle=True).item()+ out_dir = assets_dir or default_assets_dir(features_dir)+ src_dir = os.path.join(export_dir, PROCESSED_GLB_DIR)+ mixamo = dataset_type == 'mixamo' or list(cond) == ['mixamo']+ if mixamo:+ # (asset name, source GLB, object type) per character.+ if 'mixamo' not in cond:+ raise ValueError(f"{features_dir}/cond.npy has no 'mixamo' entry")+ chars = sorted(f[:-4] for f in os.listdir(src_dir)+ if f.endswith('.glb') and not f.startswith('.')) \+ if os.path.isdir(src_dir) else []+ jobs = [(c, os.path.join(src_dir, f'{c}.glb'), 'mixamo') for c in chars+ if object_types is None or c in object_types]+ else:+ types = sorted(cond) if object_types is None else [t for t in object_types if t in cond]+ jobs = [(t, os.path.join(src_dir, f'{t}.glb'), t) for t in types]+ jobs = jobs[worker_id::num_workers]+ counts = {'baked': 0, 'up_to_date': 0, 'no_source': 0, 'refused': 0, 'failed': 0}+ for name, source, object_type in jobs:+ if not os.path.isfile(source):+ counts['no_source'] += 1+ continue+ final = os.path.join(out_dir, f'{name}.glb')+ extra_files = [(fmt, os.path.join(out_dir, f'{name}.{fmt}'))+ for fmt in formats if fmt != 'glb']+ # What the GLB is built from: the cond entry (and its diameter, which+ # scales a Mixamo character), the source asset, the bake version and,+ # for a Mixamo character, the clips' rest root it stores.+ extra = {}+ if mixamo:+ extra[MOTION_ROOT_KEY] = json.dumps(+ [float(v) for v in np.asarray(cond[object_type]['tpos_first_frame'])[0]])+ digest = hashlib.sha1((cond_digest(cond[object_type])+ + str(cond_diameter(cond[object_type]))+ + file_digest(source) + ''.join(extra.values())+ + BAKE_VERSION).encode()).hexdigest()+ if (not overwrite and os.path.isfile(final) and glb_digest(final) == digest+ and all(os.path.isfile(path) for _, path in extra_files)):+ write_note(out_dir, GLB_ERRORS_DIR, name, [])+ counts['up_to_date'] += 1+ continue+ partial = os.path.join(out_dir, f'.{name}.partial')+ try:+ os.makedirs(out_dir, exist_ok=True)+ target = mixamo_character_cond(source, cond[object_type]) if mixamo \+ else cond[object_type]+ _check_not_collinear(target)+ bake_canonical_asset(source, target, partial,+ formats=('glb',) + tuple(fmt for fmt, _ in extra_files),+ extra_props={DIGEST_KEY: digest, **extra})+ for fmt, path in extra_files:+ os.replace(f'{partial}.{fmt}', path)+ os.replace(partial + '.glb', final)+ except Exception as exc: # noqa: BLE001 โ keep the batch going+ if isinstance(exc, UndeterminedRoll): # incl. CollinearSkeleton+ logger.info(f"Canonical GLB of '{name}' refused: {exc}")+ counts['refused'] += 1+ else:+ logger.exception(f"Canonical GLB of '{name}' failed")+ counts['failed'] += 1+ write_note(out_dir, GLB_ERRORS_DIR, name, [f'{type(exc).__name__}: {exc}'])+ # A GLB of older inputs would be driven as if current: remove it.+ for stale in [partial + '.glb', final] + [p for _, p in extra_files] \+ + [f'{partial}.{fmt}' for fmt, _ in extra_files]:+ if os.path.isfile(stale):+ os.remove(stale)+ write_note(out_dir, TEXTURE_ISSUES_DIR, name, [])+ continue+ write_note(out_dir, GLB_ERRORS_DIR, name, [])+ try:+ _texture_note(out_dir, name, source, final)+ except Exception as exc: # noqa: BLE001 โ the GLB is written; keep it+ logger.warning(f"'{name}': texture comparison failed: {exc}")+ counts['baked'] += 1+ logger.info(f"Canonical GLBs in {out_dir}: {counts} "+ f"({len(jobs)} asset(s){' on this worker' if num_workers > 1 else ''}; "+ f"'no_source' = no {export_dir}/{PROCESSED_GLB_DIR}/<asset>.glb)")+ return counts+++def parse_args():+ ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])+ ap.add_argument('--export_dir', required=True,+ help='Stage-1 export directory (its rigs/ holds the rest-pose GLBs).')+ ap.add_argument('--features_dir', required=True,+ help='Stage-4 feature directory (its cond.npy).')+ ap.add_argument('--assets_dir', default=None,+ help='Output directory (default: canonical_assets/<dataset> beside the '+ 'features root, e.g. dataset/canonical_assets/truebones).')+ ap.add_argument('--dataset_type', default=None,+ help="'mixamo' (also detected from a cond with the single key 'mixamo'): "+ "one GLB per character in the export's rigs/; else per object type.")+ ap.add_argument('--object_types', nargs='+', default=None,+ help='Only these object types (Mixamo: character names).')+ ap.add_argument('--overwrite', action='store_true',+ help='Rebuild even the GLBs whose inputs are unchanged.')+ ap.add_argument('--worker_id', type=int, default=0)+ ap.add_argument('--num_workers', type=int, default=1)+ argv = sys.argv[sys.argv.index('--') + 1:] if '--' in sys.argv else sys.argv[1:]+ return ap.parse_args(argv)+++def main():+ args = parse_args()+ counts = bake_canonical_assets(+ args.export_dir, args.features_dir, assets_dir=args.assets_dir,+ dataset_type=args.dataset_type,+ object_types=args.object_types, worker_id=args.worker_id,+ num_workers=args.num_workers, overwrite=args.overwrite)+ return 1 if counts['failed'] else 0+++if __name__ == '__main__':+ sys.exit(main())diff --git a/data_process/feature_extraction/extract_features.py b/data_process/feature_extraction/extract_features.pyindex d909a3b..02f6bad 100644--- a/data_process/feature_extraction/extract_features.py+++ b/data_process/feature_extraction/extract_features.py@@ -1,12 +1,14 @@"""Process exported NPZ motions into canonicalized training clips.-Supports three dataset layouts via ``--dataset_type``:+Supports four dataset layouts via ``--dataset_type``:- ``truebones``: per-object NPZ files under ``data_dir/motions``, named``{object_type}-{motion}.npz``. Object types are discovered from filenameprefixes.- ``objaverse``: same layout as truebones, plus an optional``filtered_objects.txt`` listing object types to skip.+- ``general``: extra assets of your own (``dataset/raw/general/animation``, exported by+ ``export_general.py``); processed exactly like ``objaverse``.- ``mixamo``: single skeleton / object type. All NPZs live directly under``data_dir/motions`` with no ``{object_type}-`` prefix; caption keys arebare motion stems.@@ -20,14 +22,21 @@ three are generated from the hand-reviewed patches byStage-2 (captions, category groups) and stage-3 (clean / face joint names)metadata JSONs are read from ``data_dir`` โ see-:mod:`data_process.feature_extraction.metadata`.+:mod:`data_process.feature_extraction.metadata`. Every clip has up to three+captions: the normal one (``motion_captions.json`` -> ``captions.json``;+a clip without one is still saved, but left out of ``captions.json``, and the+training loader skips it) and, when the export has them, a short+generic and a longer detail one (``motion_captions_<v>.json`` ->+``captions_<v>.json``, see ``--generic_captions`` / ``--detail_captions``).+Training can sample any of them per step.Object types are independent, so they can be processed in parallel(``--num_workers``) and are resumable: each finished object type caches itsresult under ``<save_dir>/cond_parts/`` and is skipped on rerun. The cache-records the clip / threshold / topology settings it was built with and is-re-processed automatically when they change; delete ``cond_parts/`` (or one-entry) to force re-processing for any other reason.+records the clip / threshold / topology settings and the export files (name,+size, mtime) it was built from and is re-processed automatically when they+change; delete ``cond_parts/`` (or one entry) to force re-processing for any+other reason.Object types that fail are logged to ``<save_dir>/extract_errors.log``,recorded in ``filtered_clips.json`` and skipped (no cache entry is written,@@ -53,6 +62,7 @@ from loguru import loggerfrom tqdm import tqdmfrom data_process.feature_extraction.metadata import (+ EXTRA_CAPTION_VERSIONS,MIXAMO_CORE_JOINTS,atomic_np_save,build_stats,@@ -72,12 +82,13 @@ from data_process.utils.motion_features import process_objectMIXAMO_OBJECT_TYPE = 'mixamo'-def parse_args():+def parse_args(argv=None):+ """Stage-4 arguments from *argv* (default: the command line)."""parser = argparse.ArgumentParser(description="Process exported NPZ motions into canonicalized training clips.")# Dataset selectionparser.add_argument("--dataset_type", type=str, required=True,- choices=['truebones', 'objaverse', 'mixamo'],+ choices=['truebones', 'objaverse', 'mixamo', 'general'],help="Which dataset layout to process")# Directoriesparser.add_argument("--data_dir", type=str, required=True,@@ -123,8 +134,9 @@ def parse_args():"is static; leading/trailing static frames are trimmed")parser.add_argument("--jump_step_threshold", type=float, default=0.2,help="Discontinuity filter: a clip is dropped when one "- "frame moves the skeleton at least this many body "- "lengths AND that step is at least "+ "frame moves the skeleton at least this many clip "+ "extents (the box the clip sweeps, root travel "+ "included) AND that step is at least ""--jump_ratio_threshold times the clip median ""(concatenated actions / teleports). 0 disables.")parser.add_argument("--jump_ratio_threshold", type=float, default=8.0,@@ -157,7 +169,7 @@ def parse_args():"any object is processed, so they do NOT appear in ""filtered_clips.json (which records runtime filters).")parser.add_argument("--filtered_objects", type=str, default="auto",- help="(objaverse only) Path to a txt file listing object "+ help="(objaverse / general) Path to a txt file listing object ""types to skip (one per line; blanks and '#' comments ""ignored). Default 'auto' uses ""<data_dir>/filtered_objects.txt if present. Pass an "@@ -178,6 +190,21 @@ def parse_args():"turn or a wave. Default 'auto' uses ""<data_dir>/activity_keep.txt if present. Pass an empty ""string to disable.")+ parser.add_argument("--generic_captions", type=str, default="auto",+ help="JSON {export clip: generic caption}, the second "+ "caption of every clip (written by "+ "tools/patch_annotations.py). Default 'auto' uses "+ "<data_dir>/motion_captions_generic.json if present. "+ "Saved as captions_generic.json beside captions.json, "+ "keyed the same way. Pass an empty string to disable "+ "(a stale captions_generic.json is then removed). "+ "Not part of the per-object cache: changing it never "+ "re-processes an object.")+ parser.add_argument("--detail_captions", type=str, default="auto",+ help="Like --generic_captions for the third, detail "+ "caption (7-19 words): default 'auto' uses "+ "<data_dir>/motion_captions_detail.json if present, "+ "saved as captions_detail.json.")parser.add_argument("--category_groups", type=str, default="auto",help="Path to the body-plan category JSON copied into ""the feature dir (the training loader reads it for "@@ -199,7 +226,18 @@ def parse_args():"follow camera, contact shadow and root ""trajectory (default). --no-vis_ground renders ""the plain cubic view instead.")- return parser.parse_args()+ parser.add_argument("--save_glb", action="store_true",+ help="Afterwards bake <data_dir>/rigs/<asset>.glb (the export stage's "+ "rest-pose assets, run_export.sh --save_glb / --glb_only) into the "+ "canonical frame: <canonical_assets_dir>/<object_type>.glb, driven "+ "by the feature clips with no cond.npy (canonical_assets.py; needs "+ "bpy; one process, so shard canonical_assets.py for a large dataset).")+ parser.add_argument("--canonical_assets_dir", type=str, default=None,+ help="Where --save_glb writes (default: canonical_assets/<dataset> "+ "beside the features root, e.g. dataset/canonical_assets/truebones "+ "for dataset/features/truebones; <save_dir>_canonical_assets for "+ "a save_dir outside a features root).")+ return parser.parse_args(argv)def _resolve_skip_list_path(path, data_dir, filename):@@ -239,6 +277,30 @@ def resolve_activity_keep_path(path, data_dir):return _resolve_skip_list_path(path, data_dir, 'activity_keep.txt')+def resolve_extra_captions(path, data_dir, version):+ """Load the *version* captions named by ``--<version>_captions`` (or None).++ ``"auto"`` uses ``<data_dir>/motion_captions_<version>.json`` when present+ and an empty string disables them; both then remove a stale+ ``captions_<version>.json``. An explicit path that does not exist raises+ instead: a typo must not silently delete the captions already in the+ feature directory.+ """+ name = f'motion_captions_{version}.json'+ if path == 'auto':+ path = pjoin(data_dir, name)+ if not os.path.isfile(path):+ logger.info(f'No {name} in {data_dir}; no captions_{version}.json this run')+ return None+ elif not path:+ return None+ elif not os.path.isfile(path):+ raise FileNotFoundError(f'--{version}_captions {path!r} does not exist')+ captions = load_json(path)+ logger.info(f'{version.capitalize()} captions: {len(captions)} from {path}')+ return captions++def load_clip_trims(path):"""Read ``<clip stem> <N>`` head trims; a falsy path yields an empty dict.@@ -324,7 +386,7 @@ def discover_object_types(motion_dir, dataset_type, args):object_types = sorted(set(m.split('-')[0] for m in all_motions))logger.info(f'Found {len(all_motions)} motion files, {len(object_types)} object types')- if dataset_type == 'objaverse':+ if dataset_type in ('objaverse', 'general'):filtered_path = resolve_filtered_objects_path(args.filtered_objects, args.data_dir)filtered_object_types = load_filtered_objects(filtered_path)if filtered_object_types:@@ -420,26 +482,31 @@ def _cache_params(task):# re-processes exactly the owning object type.if task.get('root_offsets'):params['root_offsets_digest'] = _digest(task['root_offsets'])+ # Rest orientation fixes are baked into the NPZs the same way.+ if task.get('rest_orientations'):+ params['rest_orientations_digest'] = _digest(task['rest_orientations'])return params# Written into an export NPZ by tools/patch_annotations.py (apply_root_offsets)# when it bakes a root orientation fix in; keep the name in sync.ROOT_OFFSET_KEY = 'root_offset_applied'+# Written by tools/patch_annotations.py (apply_rest_orientations); keep in sync.+REST_ORIENTATION_KEY = 'rest_orientation_applied'-def read_root_offsets(object_npzs):+def read_root_offsets(object_npzs, key=ROOT_OFFSET_KEY):"""``{clip stem: [w, x, y, z]}`` for the export NPZs carrying a baked-in- root offset. Only the zip directory is read for the others, so this stays- cheap over a whole dataset."""+ fix under *key* (a root offset by default, or a rest orientation). Only the+ zip directory is read for the others, so this stays cheap over a whole+ dataset."""offsets = {}for path in object_npzs:with zipfile.ZipFile(path) as zf:- if ROOT_OFFSET_KEY + '.npy' not in zf.namelist():+ if key + '.npy' not in zf.namelist():continuewith np.load(path) as d:- offsets[os.path.basename(path)[:-4]] = [- round(float(x), 6) for x in d[ROOT_OFFSET_KEY]]+ offsets[os.path.basename(path)[:-4]] = [round(float(x), 6) for x in d[key]]return offsets@@ -449,8 +516,23 @@ def _params_hash(params):return hashlib.sha1(blob.encode('utf-8')).hexdigest()-def _load_cached_result(part_path, object_type, params, params_hash):- """Return a cached object result, or None when absent / stale / unreadable."""+def _export_digest(task):+ """Digest of the inputs the settings hash cannot see: each clip NPZ's name,+ size and mtime (a re-export under the same clip names) and the object's+ ``joint_names.json`` entry (its order drives the clean-name realignment).+ Kept beside ``params_hash`` rather than in it (see :func:`_load_cached_result`)."""+ files = []+ for path in sorted(task['object_npzs']):+ st = os.stat(path)+ files.append([os.path.basename(path), st.st_size, st.st_mtime_ns])+ return _digest([files, task.get('expected_names')])+++def _load_cached_result(part_path, object_type, params, params_hash, export_digest):+ """Return a cached object result, or None when absent / stale / unreadable.++ An entry without an ``export_digest`` is taken as built from the current+ export and stamped with its digest."""if not os.path.isfile(part_path):return Nonetry:@@ -465,7 +547,16 @@ def _load_cached_result(part_path, object_type, params, params_hash):f'tracking); re-processing')return Noneif payload['params_hash'] == params_hash:- return payload.get('result')+ cached_digest = payload.get('export_digest')+ if cached_digest is None:+ payload['export_digest'] = export_digest+ atomic_np_save(part_path, payload)+ return payload.get('result')+ if cached_digest == export_digest:+ return payload.get('result')+ logger.info(f'[{object_type}] cache invalidated (export NPZs or joint_names.json '+ f'entry changed); re-processing')+ return Nonecached_params = payload.get('params') or {}diff = ', '.join(@@ -527,9 +618,11 @@ def process_object_task(task):clip_prefix = task.pop('clip_prefix')params = _cache_params(task)params_hash = _params_hash(params)+ export_digest = _export_digest(task)task.pop('root_offsets', None) # cache key only; process_object reads the NPZs+ task.pop('rest_orientations', None)- cached = _load_cached_result(part_path, object_type, params, params_hash)+ cached = _load_cached_result(part_path, object_type, params, params_hash, export_digest)if cached is not None:return object_type, cached, True@@ -562,7 +655,7 @@ def process_object_task(task):# entry behind (unreadable ones are treated as misses, but this keeps them# from happening in the first place).atomic_np_save(part_path, {'params_hash': params_hash, 'params': params,- 'result': result})+ 'export_digest': export_digest, 'result': result})return object_type, result, False@@ -614,6 +707,7 @@ def build_object_tasks(args, clip_stride, motion_dir, metadata):head_trims=head_trims,activity_keep=keep,root_offsets=read_root_offsets(object_npzs),+ rest_orientations=read_root_offsets(object_npzs, key=REST_ORIENTATION_KEY),# Mixamo clip files carry no "{object_type}-" prefix; its single# object type owns the whole directory.clip_prefix=('' if args.dataset_type == 'mixamo'@@ -665,6 +759,9 @@ def main(args):+ ', '.join(f'{k}={len(v)}' for k, v in metadata.items()))check_metadata_object_types_consistent(metadata)category_groups = resolve_category_groups(args.category_groups, metadata)+ # Resolved up front so a bad --<v>_captions path fails before any work.+ extra_captions = {v: resolve_extra_captions(getattr(args, f'{v}_captions'), args.data_dir, v)+ for v in EXTRA_CAPTION_VERSIONS}tasks = build_object_tasks(args, clip_stride, motion_dir, metadata)@@ -714,11 +811,21 @@ def main(args):+ ', '.join(sorted(failed_objects)))all_captions = save_outputs(args.save_dir, cond, all_filtered_clips,- category_groups=category_groups)+ category_groups=category_groups,+ extra_captions=extra_captions)stats = build_stats(clips_per_object, joints_per_object, total_frames, max_njoints)print_summary(stats)save_metadata_report(args.save_dir, stats, all_filtered_clips,all_captions=all_captions, category_groups=category_groups)+ if args.save_glb:+ # bpy only here, so the stage itself stays Blender-free.+ from data_process.feature_extraction.canonical_assets import bake_canonical_assets+ glb_counts = bake_canonical_assets(args.data_dir, args.save_dir,+ assets_dir=args.canonical_assets_dir,+ dataset_type=args.dataset_type)+ if glb_counts['failed']:+ logger.error(f"{glb_counts['failed']} canonical GLB(s) failed; see glb_errors/ in "+ f"the canonical_assets directory (the features are complete)")logger.info('Dataset processing complete.')diff --git a/data_process/feature_extraction/metadata.py b/data_process/feature_extraction/metadata.pyindex 7d5de44..aceb1cc 100644--- a/data_process/feature_extraction/metadata.py+++ b/data_process/feature_extraction/metadata.py@@ -77,6 +77,47 @@ def load_motion_captions(source_dir):return load_json(pjoin(source_dir, 'motion_captions.json'))+# Extra caption versions beside the normal one (``motion_captions.json`` ->+# ``captions.json``): the short generic caption and the longer detail caption,+# HumanML3D-style. Each comes only from its reviewed patch file through+# ``patch_annotations.py`` (export ``motion_captions_<v>.json``) and is saved by+# stage 4 as ``captions_<v>.json``; training may sample one per step.+EXTRA_CAPTION_VERSIONS = ('generic', 'detail')+++def load_extra_captions(source_dir, version):+ # type: (str, str) -> Optional[Dict[str, str]]+ """Load one extra caption version from ``motion_captions_<version>.json``+ (written by ``patch_annotations.py``; None when the export has none)."""+ if version not in EXTRA_CAPTION_VERSIONS:+ raise ValueError(f'unknown caption version {version!r}; '+ f'expected one of {EXTRA_CAPTION_VERSIONS}')+ return load_json(pjoin(source_dir, f'motion_captions_{version}.json'))+++def load_generic_captions(source_dir):+ # type: (str) -> Optional[Dict[str, str]]+ """``load_extra_captions(source_dir, 'generic')``."""+ return load_extra_captions(source_dir, 'generic')+++def by_feature_key(feature_captions, export_captions):+ # type: (Dict[str, str], Dict[str, str]) -> Dict[str, str]+ """Re-key an export ``{clip: caption}`` map (normal, generic or detail) by+ the feature clip keys.++ Feature keys are ``<export clip>-<NNN>`` (one per saved segment), the keys+ of ``captions.json``; every segment of a clip gets the clip's caption.+ Feature clips without an export caption are left out.+ """+ out = {}+ for key in feature_captions:+ stem = key.rsplit('-', 1)[0]+ if stem in export_captions:+ out[key] = export_captions[stem]+ return out++def load_category_groups(source_dir):# type: (str) -> Optional[Dict[str, List[str]]]"""Load category-to-object-type groupings from ``category_groups.json``."""@@ -227,9 +268,38 @@ def collect_captions(cond):return all_captions-def save_outputs(save_dir, cond, all_filtered_clips, category_groups=None):+def save_extra_captions(save_dir, all_captions, version, export_captions):+ """Write ``captions_<version>.json`` (keyed like ``captions.json``) from+ the export's ``motion_captions_<version>.json`` map, or remove a stale+ one when no saved clip has a caption of that version. Returns the dict+ written (empty when none)."""+ path = pjoin(save_dir, f'captions_{version}.json')+ out = (by_feature_key(all_captions, export_captions)+ if all_captions and export_captions else {})+ if out:+ save_json(path, out)+ missing = sorted(set(all_captions) - set(out))+ print(f'Saved {len(out)}/{len(all_captions)} {version} clip captions'+ + (f'; {len(missing)} saved clips have none, e.g. {missing[:3]}' if missing else ''))+ elif os.path.isfile(path):+ os.remove(path)+ print(f'Removed stale {path} (no {version} captions this run)')+ return out+++def save_outputs(save_dir, cond, all_filtered_clips, category_groups=None,+ extra_captions=None):"""Save ``cond.npy``, filtered clips, captions, and category groupings.+ Caption files, keyed alike by saved clip (``<clip>-<NNN>``):+ ``captions.json`` from ``motion_captions.json`` (the normal caption; a+ clip without one is not saved) and, for each version ``v`` of+ ``EXTRA_CAPTION_VERSIONS``, ``captions_<v>.json`` from+ ``extra_captions[v]`` (the export's ``motion_captions_<v>.json``). Extra+ captions never enter ``cond`` or the ``cond_parts`` cache, so changing them+ never re-processes an object; a version without captions has its stale+ ``captions_<v>.json`` removed.+Captions and category groups are only written when present. Every file iswritten atomically, so an interrupted run cannot leave a truncated outputbehind for the training loader to trip over. Returns the merged captions@@ -246,6 +316,9 @@ def save_outputs(save_dir, cond, all_filtered_clips, category_groups=None):if all_captions:save_json(pjoin(save_dir, 'captions.json'), all_captions)print(f'Saved {len(all_captions)} clip captions')+ for version in EXTRA_CAPTION_VERSIONS:+ save_extra_captions(save_dir, all_captions, version,+ (extra_captions or {}).get(version))if category_groups:save_json(pjoin(save_dir, 'category_groups.json'), category_groups)@@ -278,8 +351,12 @@ def save_metadata_report(save_dir, stats, all_filtered_clips,"""Write a human-readable metadata summary to ``metadata.txt``.``all_captions`` and ``category_groups`` are optional; their sections are- omitted when not supplied. The file is written atomically.+ omitted when not supplied. The extra-caption counts are read back from+ the ``captions_<v>.json`` files ``save_outputs`` wrote. The file is+ written atomically."""+ extra = {v: load_json(pjoin(save_dir, f'captions_{v}.json'))+ for v in EXTRA_CAPTION_VERSIONS}total_filtered = sum(len(v) for v in all_filtered_clips.values())with atomic_output_path(pjoin(save_dir, 'metadata.txt')) as tmp_path, \@@ -291,6 +368,9 @@ def save_metadata_report(save_dir, stats, all_filtered_clips,f.write(f'Max joints in dataset: {stats["max_njoints"]}\n')if all_captions is not None:f.write(f'Total captions: {len(all_captions)}\n')+ for version, caps in extra.items():+ if caps is not None:+ f.write(f'Total {version} captions: {len(caps)}\n')f.write('Clips per object type:\n')for obj, count in stats['clips_per_object'].items():diff --git a/data_process/joint_annotation/face_select_llm.py b/data_process/joint_annotation/face_select_llm.pyindex ad1fe60..c881695 100644--- a/data_process/joint_annotation/face_select_llm.py+++ b/data_process/joint_annotation/face_select_llm.py@@ -369,6 +369,9 @@ def parse_response(text, raw_names, clean_names):- The clean name is re-sourced from ``clean_names`` so the LLM can'tdrift the canonical label.- The sides are not swapped (invariant I3, see :func:`_validate_sides`).+ - ``body_axis``, when given, is a JSON boolean, true exactly when+ ``source`` is ``'body_axis'`` (invariant I4); omitted, it follows+ ``source``."""if not text or not text.strip():raise ValueError("empty response")@@ -392,7 +395,7 @@ def parse_response(text, raw_names, clean_names):r_raw, _ = _hip_fields(data["r_hip"], "r_hip")l_raw, _ = _hip_fields(data["l_hip"], "l_hip")source = str(data["source"]).strip().lower()- body_axis = bool(data.get("body_axis", False))+ body_axis = data.get("body_axis")if not r_raw and not l_raw:return empty_entry()@@ -407,9 +410,20 @@ def parse_response(text, raw_names, clean_names):if l_raw not in raw_names:raise ValueError("l_hip.raw {!r} not in input raw joint list".format(l_raw))+ if body_axis is None:+ body_axis = source == "body_axis"+ elif not isinstance(body_axis, bool):+ # bool("false") is True: a string would turn a lateral pair into a+ # body axis and rotate every clip of the rig by 90 degrees.+ raise ValueError("body_axis must be a JSON boolean, got {!r}".format(body_axis))+ elif body_axis != (source == "body_axis"):+ # Invariant I4: body_axis iff source == 'body_axis'.+ raise ValueError("body_axis={} disagrees with source={!r}".format(+ body_axis, source))+r_idx = raw_names.index(r_raw)l_idx = raw_names.index(l_raw)- is_axis = body_axis or source == "body_axis"+ is_axis = body_axis_validate_sides(clean_names[r_idx], clean_names[l_idx], is_axis)if not is_axis:# Invariant I6: a lateral pair must mirror the SAME part โ a@@ -429,7 +443,7 @@ def parse_response(text, raw_names, clean_names):"l_hip": {"raw": raw_names[l_idx], "clean": clean_names[l_idx]},"source": source or "pair",}- if body_axis or source == "body_axis":+ if body_axis:entry["body_axis"] = Truereturn entrydiff --git a/data_process/joint_annotation/face_select_rule.py b/data_process/joint_annotation/face_select_rule.pyindex dc0ac9d..856f218 100644--- a/data_process/joint_annotation/face_select_rule.py+++ b/data_process/joint_annotation/face_select_rule.py@@ -7,8 +7,9 @@ the lateral (left-right) axis used to compute the facing direction:pair, preferring anatomically meaningful joints (Thigh, Shoulder, ...).2. For body-axis animals (snakes like Anaconda) that lack bilateralsymmetry, use longitudinal body-axis endpoints (tail-tip / head-tip)- and flag ``"body_axis": true``. The tail-to-head vector is directly- the facing direction (no cross product with Y-up needed).+ and flag ``"body_axis": true``. Stage 4 turns the tail-to-head vector+ by -90 degrees about Y before the usual cross product with Y-up, so it+ ends up as the facing direction.3. Animals with opaque joint names where no meaningful pair can beextracted get ``"source": "empty"``.@@ -180,10 +181,13 @@ def _match_pair_by_suffix(def find_symmetric_pair(clean_names: List[str],raw_names: List[str],+ usable=None,) -> Tuple[Optional[int], Optional[int], Optional[str]]:"""Find the highest-priority "Right <X>" / "Left <X>" pair.- Returns ``(r_idx, l_idx, suffix)`` or ``(None, None, None)``.+ *usable* ``(r_idx, l_idx) -> bool`` rejects a pair (e.g. one without+ horizontal separation in the rest pose); the next part in priority order+ is tried. Returns ``(r_idx, l_idx, suffix)`` or ``(None, None, None)``."""right_map: Dict[str, List[int]] = {}left_map: Dict[str, List[int]] = {}@@ -199,9 +203,11 @@ def find_symmetric_pair(ordered = [s for s in SYMMETRIC_PAIR_PRIORITY if s in common]ordered += sorted(common - set(SYMMETRIC_PAIR_PRIORITY))- suffix = ordered[0]- ri, li = _match_pair_by_suffix(right_map[suffix], left_map[suffix], raw_names)- return ri, li, suffix+ for suffix in ordered:+ ri, li = _match_pair_by_suffix(right_map[suffix], left_map[suffix], raw_names)+ if usable is None or usable(ri, li):+ return ri, li, suffix+ return None, None, Nonedef find_body_axis_pair(@@ -259,9 +265,11 @@ def _make_entry(def resolve_face_joints(clean_names: List[str],raw_names: List[str],+ usable=None,) -> Dict:- """Resolve one joint pair defining the facing direction."""- r_idx, l_idx, suffix = find_symmetric_pair(clean_names, raw_names)+ """Resolve one joint pair defining the facing direction (*usable*: see+ ``find_symmetric_pair``)."""+ r_idx, l_idx, suffix = find_symmetric_pair(clean_names, raw_names, usable)if r_idx is not None:return _make_entry(raw_names, clean_names, r_idx, l_idx, suffix.lower())diff --git a/data_process/joint_annotation/learn_vocab.py b/data_process/joint_annotation/learn_vocab.pynew file mode 100644index 0000000..e62dd3d--- /dev/null+++ b/data_process/joint_annotation/learn_vocab.py@@ -0,0 +1,116 @@+"""Learn a raw-name vocabulary from reviewed joint labels: ``learned_vocab.json``.++For every rig of the given dataset exports, each raw joint name is reduced to+its words (``names_clean_rule.name_words``: namespace, Biped prefix,+counters and side markers dropped) and paired with the part of its reviewed+label in ``clean_joint_names.json`` (LLM output, human review and patches;+side removed, ``End`` kept). Every word tail of a name (``doberman_ref_haunch``,+``ref_haunch``, ``haunch``) is counted. A key enters the vocabulary when the+rigs it occurs in agree on one part: at least ``--min_rigs`` distinct rigs and+a share of at least ``--min_purity`` of its joints. Placeholder labels+('Bone', numeric passthroughs) are not learned, nor single-letter keys+('t'), nor a generic word (``link``, ``joint``), nor a one-word key whose own use as a whole joint name disagrees with+what it was learned as (``base`` is the tail of ``toe_base``, Toe, but alone+names a root).++``names_clean_rule.clean_joint_name`` consults the vocabulary only for a name+its rules leave as a placeholder or a non-canonical label, longest key first.++ python -m data_process.joint_annotation.learn_vocab # all rigs+ python -m data_process.joint_annotation.learn_vocab --split dev --out /tmp/v.json+"""++import argparse+import datetime+import json+import os+import re+import sys+from collections import Counter, defaultdict++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++import hashlib # noqa: E402++from data_process.joint_annotation.names_clean_rule import name_words # noqa: E402++DEFAULT_OUT = os.path.join(os.path.dirname(__file__), 'learned_vocab.json')+# Generic structural words that name no body part on their own (robot '<part>_link').+GENERIC_WORDS = frozenset({'link', 'joint', 'jnt', 'bone', 'node'})+++def in_split(rig, split):+ """Deterministic half of the rigs: 'dev' or 'test' (None: all)."""+ if split is None:+ return True+ bucket = int(hashlib.sha1(rig.encode()).hexdigest(), 16) % 2+ return bucket == (0 if split == 'dev' else 1)+++def part_of(label):+ """Reviewed label without its side ('Right Upper Arm' -> 'Upper Arm')."""+ return re.sub(r'^(Left|Right)\s+', '', label).strip()+++def learn(datasets, export_root='dataset/export', split=None, min_rigs=3, min_purity=0.9):+ parts = defaultdict(Counter) # key -> part -> joints+ rigs = defaultdict(set) # key -> rigs it occurs in+ alone = defaultdict(Counter) # one-word key -> part, as a whole name+ for ds in datasets:+ root = os.path.join(export_root, ds)+ if not os.path.isfile(os.path.join(root, 'clean_joint_names.json')):+ continue+ with open(os.path.join(root, 'joint_names.json')) as f:+ names = json.load(f)+ with open(os.path.join(root, 'clean_joint_names.json')) as f:+ clean = json.load(f)+ for rig, raw in names.items():+ labels = clean.get(rig)+ if not labels or len(labels) != len(raw) or not in_split(rig, split):+ continue+ for name, label in zip(raw, labels):+ words = name_words(name)+ if len(words) == 1:+ alone[words[0]][part_of(label)] += 1+ part = part_of(label)+ if not part or part == 'Bone' or re.match(r'^_?\d+$', part):+ continue+ for i in range(len(words)):+ key = '_'.join(words[i:])+ parts[key][part] += 1+ rigs[key].add(f'{ds}/{rig}')+ vocab = {}+ for key, counter in parts.items():+ part, n = counter.most_common(1)[0]+ if len(rigs[key]) < min_rigs or n / sum(counter.values()) < min_purity:+ continue+ if key in GENERIC_WORDS:+ continue+ if '_' not in key and (len(key) < 2 or (+ alone[key] and alone[key].most_common(1)[0][0] != part)):+ continue+ vocab[key] = part+ return dict(sorted(vocab.items()))+++def main():+ ap = argparse.ArgumentParser(description="Learn learned_vocab.json from reviewed labels.")+ ap.add_argument('--datasets', nargs='+', default=['truebones', 'mixamo', 'objaverse'])+ ap.add_argument('--export_root', default='dataset/export')+ ap.add_argument('--split', choices=('dev', 'test'), default=None,+ help="Learn from one half of the rigs only (to evaluate on the other).")+ ap.add_argument('--min_rigs', type=int, default=3)+ ap.add_argument('--min_purity', type=float, default=0.9)+ ap.add_argument('--out', default=DEFAULT_OUT)+ args = ap.parse_args()+ vocab = learn(args.datasets, args.export_root, args.split, args.min_rigs, args.min_purity)+ meta = {'datasets': args.datasets, 'split': args.split, 'min_rigs': args.min_rigs,+ 'min_purity': args.min_purity, 'keys': len(vocab),+ 'date': datetime.date.today().isoformat()}+ with open(args.out, 'w') as f:+ json.dump({'_meta': meta, 'keys': vocab}, f, indent=1, sort_keys=False)+ print(f"{len(vocab)} keys -> {args.out}")+++if __name__ == '__main__':+ main()diff --git a/data_process/joint_annotation/learned_name_map.json.gz b/data_process/joint_annotation/learned_name_map.json.gznew file mode 100644index 0000000..f1454c2Binary files /dev/null and b/data_process/joint_annotation/learned_name_map.json.gz differdiff --git a/data_process/joint_annotation/learned_vocab.json b/data_process/joint_annotation/learned_vocab.jsonnew file mode 100644index 0000000..1b20189--- /dev/null+++ b/data_process/joint_annotation/learned_vocab.json@@ -0,0 +1,1228 @@+{+ "_meta": {+ "datasets": [+ "truebones",+ "mixamo",+ "objaverse"+ ],+ "split": null,+ "min_rigs": 3,+ "min_purity": 0.9,+ "keys": 1212,+ "date": "2026-10-05"+ },+ "keys": {+ "a_m": "Spine",+ "abdomen": "Abdomen",+ "abdomen_jnt": "Abdomen",+ "abdomen_root_jnt": "Abdomen",+ "ago": "Jaw",+ "anelar": "Ring Finger",+ "ankle_bone": "Toe",+ "ankle_jnt": "Ankle",+ "antenna": "Antenna",+ "antenna_jnt": "Antenna",+ "arm_a": "Upper Arm",+ "arm_clavicle_sh_jnt": "Shoulder",+ "arm_collarbone": "Shoulder",+ "arm_elbow_sh_jnt": "Elbow",+ "arm_forearm": "Forearm",+ "arm_hand": "Hand",+ "arm_index": "Index Finger",+ "arm_jnt": "Upper Arm",+ "arm_lower": "Forearm",+ "arm_pinky": "Pinky Finger",+ "arm_shoulder": "Shoulder",+ "arm_shoulder_sh_jnt": "Shoulder",+ "arm_stretch": "Upper Arm",+ "arm_thumb": "Thumb Finger",+ "arm_twist": "Upper Arm Twist",+ "arm_upper": "Upper Arm",+ "arm_upperarm": "Upper Arm",+ "arm_wrist_sh_jnt": "Wrist",+ "armature_foot": "Foot",+ "armature_hand": "Hand",+ "armature_head": "Head",+ "armature_mixamorig_arm": "Upper Arm",+ "armature_mixamorig_foot": "Foot",+ "armature_mixamorig_fore_arm": "Forearm",+ "armature_mixamorig_hand": "Hand",+ "armature_mixamorig_head": "Head",+ "armature_mixamorig_hips": "Hips",+ "armature_mixamorig_leg": "Shin",+ "armature_mixamorig_neck": "Neck",+ "armature_mixamorig_shoulder": "Shoulder",+ "armature_mixamorig_spine": "Spine",+ "armature_mixamorig_toe_base": "Toe",+ "armature_mixamorig_toe_end": "Toe End",+ "armature_mixamorig_up_leg": "Thigh",+ "armature_neck": "Neck",+ "arrow": "Weapon",+ "atama": "Head",+ "b_arm": "Upper Arm",+ "b_calf": "Shin",+ "b_clav": "Shoulder",+ "b_clavicle": "Shoulder",+ "b_elbow": "Elbow",+ "b_eye": "Eye",+ "b_foot": "Foot",+ "b_fore_arm": "Forearm",+ "b_forearm": "Forearm",+ "b_hand": "Hand",+ "b_head": "Head",+ "b_hips": "Hips",+ "b_index": "Index Finger",+ "b_indexend": "Index Finger End",+ "b_jaw": "Jaw",+ "b_leg_upper": "Thigh",+ "b_lower_arm": "Forearm",+ "b_middle": "Middle Finger",+ "b_middleend": "Middle Finger End",+ "b_neck": "Neck",+ "b_pelvis": "Hips",+ "b_pinky": "Pinky Finger",+ "b_pinkyend": "Pinky Finger End",+ "b_ring": "Ring Finger",+ "b_ringend": "Ring Finger End",+ "b_root": "Root",+ "b_shoulder": "Shoulder",+ "b_skart": "Skirt",+ "b_spine": "Spine",+ "b_tail": "Tail",+ "b_thigh": "Thigh",+ "b_thumb": "Thumb Finger",+ "b_thumbend": "Thumb Finger End",+ "b_toe": "Toe",+ "b_toes": "Toe",+ "b_upper_arm": "Upper Arm",+ "back_a_m": "Spine",+ "back_ankle": "Back Ankle",+ "back_b_m": "Spine",+ "back_hoof": "Back Hoof",+ "back_lower_leg": "Back Shin",+ "back_paw": "Back Paw",+ "back_rump": "Back Hip",+ "back_toe": "Back Toe",+ "back_upper_leg": "Back Thigh",+ "bag": "Bag",+ "bag_bones": "Bag",+ "bag_init": "Bag",+ "bag_mid": "Bag",+ "ball": "Toe",+ "ball_end": "Toe End",+ "base_armature": "Toe",+ "base_end": "Toe End",+ "base_human_arm_collarbone": "Shoulder",+ "base_human_arm_forearm": "Forearm",+ "base_human_arm_palm": "Palm",+ "base_human_arm_upperarm": "Upper Arm",+ "base_human_calf": "Shin",+ "base_human_collarbone": "Shoulder",+ "base_human_foot": "Foot",+ "base_human_forearm": "Forearm",+ "base_human_head": "Head",+ "base_human_leg_calf": "Shin",+ "base_human_leg_thigh": "Thigh",+ "base_human_neck": "Neck",+ "base_human_palm": "Palm",+ "base_human_pelvis": "Hips",+ "base_human_ribcage": "Ribcage",+ "base_human_spine": "Spine",+ "base_human_thigh": "Thigh",+ "base_human_upperarm": "Upper Arm",+ "basic_rig_calf": "Shin",+ "basic_rig_clavicle": "Shoulder",+ "basic_rig_foot": "Foot",+ "basic_rig_forearm": "Forearm",+ "basic_rig_hand": "Hand",+ "basic_rig_head": "Head",+ "basic_rig_neck": "Neck",+ "basic_rig_pelvis": "Hips",+ "basic_rig_spine": "Spine",+ "basic_rig_thigh": "Thigh",+ "basic_rig_toe": "Toe",+ "basic_rig_upper_arm": "Upper Arm",+ "bear_clavicle_sh_jnt": "Shoulder",+ "bear_ear_sh_jnt": "Ear",+ "bear_head_jaw_end_sh_jnt": "Jaw End",+ "bear_head_jaw_sh_jnt": "Jaw",+ "bear_head_top_sh_jnt": "Head",+ "bear_neck_sh_jnt": "Neck",+ "bear_rootsh_jnt": "Root",+ "bear_spine_sh_jnt": "Spine",+ "bear_spine_top_sh_jnt": "Spine",+ "bear_tail_sh_jnt": "Tail",+ "bedro": "Thigh",+ "belly": "Belly",+ "bicep": "Upper Arm",+ "bicept": "Upper Arm",+ "big_toe": "Toe",+ "bind_arm": "Upper Arm",+ "bind_femur": "Thigh",+ "bind_foot": "Foot",+ "bind_fore_arm": "Forearm",+ "bind_hand": "Hand",+ "bind_hand_index": "Index Finger",+ "bind_hand_middle": "Middle Finger",+ "bind_hand_ring": "Ring Finger",+ "bind_hand_thumb": "Thumb Finger",+ "bind_head": "Head",+ "bind_head_top_end": "Head End",+ "bind_hips": "Hips",+ "bind_leg": "Shin",+ "bind_neck": "Neck",+ "bind_pelvis": "Hips",+ "bind_shoulder": "Shoulder",+ "bind_spine": "Spine",+ "bind_toe_base": "Toe",+ "bind_toe_end": "Toe End",+ "bind_up_leg": "Thigh",+ "bip": "Root",+ "bip_calf": "Shin",+ "bip_clavicle": "Shoulder",+ "bip_foot": "Foot",+ "bip_forearm": "Forearm",+ "bip_hand": "Hand",+ "bip_head": "Head",+ "bip_head_nub": "Head End",+ "bip_neck": "Neck",+ "bip_pelvis": "Hips",+ "bip_spine": "Spine",+ "bip_thigh": "Thigh",+ "bip_toe": "Toe",+ "bip_upper_arm": "Upper Arm",+ "bl_foot": "Back Foot",+ "bl_leg": "Back Leg",+ "bn_ball": "Toe",+ "bn_chest": "Chest",+ "bn_chin": "Chin",+ "bn_downbody": "Lower Body",+ "bn_ear": "Ear",+ "bn_eye": "Eye",+ "bn_eyebrow": "Eyebrow",+ "bn_eyelid": "Eyelid",+ "bn_foot": "Foot",+ "bn_fore_arm": "Forearm",+ "bn_forearm": "Forearm",+ "bn_hair": "Mane",+ "bn_hand": "Hand",+ "bn_hand_index": "Index Finger",+ "bn_hand_middle": "Middle Finger",+ "bn_head": "Head",+ "bn_jaw": "Jaw",+ "bn_lip": "Lip",+ "bn_lower_eye_lid": "Lower Eyelid",+ "bn_mane": "Mane",+ "bn_mouth": "Mouth",+ "bn_neck": "Neck",+ "bn_nose": "Nose",+ "bn_shoulder": "Shoulder",+ "bn_spine": "Spine",+ "bn_tai": "Tail",+ "bn_tail": "Tail",+ "bn_toe": "Toe",+ "bn_tongue": "Tongue",+ "bn_up_leg": "Thigh",+ "bn_up_leg_twist": "Thigh Twist",+ "bn_upper_arm": "Upper Arm",+ "bn_upper_eye_lid": "Upper Eyelid",+ "bn_wing": "Wing",+ "bone_brow": "Eyebrow",+ "bone_brow_c": "Eyebrow",+ "bone_calf": "Shin",+ "bone_clavicle": "Shoulder",+ "bone_eye": "Eye",+ "bone_foot": "Foot",+ "bone_forearm": "Forearm",+ "bone_hand": "Hand",+ "bone_head": "Head",+ "bone_jaw": "Jaw",+ "bone_lid_down": "Lower Eyelid",+ "bone_lid_up": "Upper Eyelid",+ "bone_lip": "Lip",+ "bone_mouth": "Mouth",+ "bone_neck": "Neck",+ "bone_spine": "Spine",+ "bone_tail": "Tail",+ "bone_thigh": "Thigh",+ "br_foot": "Back Foot",+ "br_leg": "Back Leg",+ "breast": "Chest",+ "brow": "Eyebrow",+ "brow_c": "Eyebrow",+ "buffbone_glb_hand_loc": "Hand",+ "buffbone_glb_weapon": "Weapon",+ "c_index": "Index Finger",+ "c_middle": "Middle Finger",+ "c_pinky": "Pinky Finger",+ "c_ring": "Ring Finger",+ "c_skull": "Skull",+ "c_tail": "Tail",+ "c_thumb": "Thumb Finger",+ "cabe_a": "Head",+ "cabeca": "Head",+ "cabeza": "Head",+ "calf": "Shin",+ "calf_twist": "Shin Twist",+ "cam": "Hand",+ "cam_end": "Hand End",+ "camel_hump_m_clavicle_sh_jnt": "Shoulder",+ "camel_hump_m_front_leg_ankle_sh_jnt": "Front Ankle",+ "camel_hump_m_front_leg_ball_sh_jnt": "Front Toe",+ "camel_hump_m_front_leg_knee_sh_jnt": "Front Knee",+ "camel_hump_m_head_jaw_sh_jnt": "Jaw",+ "camel_hump_m_head_top_sh_jnt": "Head",+ "camel_hump_m_neck_sh_jnt": "Neck",+ "camel_hump_m_neck_top_sh_jnt": "Neck",+ "camel_hump_m_rootsh_jnt": "Root",+ "camel_hump_m_spine_sh_jnt": "Spine",+ "camel_hump_m_spine_top_sh_jnt": "Spine",+ "camel_hump_m_tail_sh_jnt": "Tail",+ "cat_rig_ankle_bone": "Toe",+ "cat_rig_arm_collarbone": "Shoulder",+ "cat_rig_chest": "Chest",+ "cat_rig_head": "Head",+ "cat_rig_leg_digit": "Toe",+ "cat_rig_neck": "Neck",+ "cat_rig_pelvis": "Hips",+ "cat_rig_tail": "Tail",+ "character_arm": "Upper Arm",+ "character_foot": "Foot",+ "character_fore_arm": "Forearm",+ "character_hand": "Hand",+ "character_hand_index": "Index Finger",+ "character_hand_middle": "Middle Finger",+ "character_hand_pinky": "Pinky Finger",+ "character_hand_ring": "Ring Finger",+ "character_hand_thumb": "Thumb Finger",+ "character_head": "Head",+ "character_hips": "Hips",+ "character_leg": "Shin",+ "character_neck": "Neck",+ "character_shoulder": "Shoulder",+ "character_spine": "Spine",+ "character_toe_base": "Toe",+ "character_up_leg": "Thigh",+ "cheek": "Cheek",+ "chest": "Chest",+ "chest_armature": "Chest",+ "chest_jnt": "Chest",+ "chest_m": "Chest",+ "chin_end": "Chin End",+ "clav": "Shoulder",+ "clav_jnt": "Shoulder",+ "clavicle": "Shoulder",+ "clavicle_bn": "Shoulder",+ "clavicle_jnt": "Shoulder",+ "clavicle_sh_jnt": "Shoulder",+ "clavicula": "Shoulder",+ "codo": "Elbow",+ "cog": "Root",+ "cog_c": "Root",+ "cola": "Tail",+ "collarbone": "Shoulder",+ "columna": "Spine",+ "coluna": "Spine",+ "costela": "Ribcage",+ "cotovelo": "Elbow",+ "cou": "Neck",+ "created_root_joint": "Root",+ "crista": "Hip",+ "ctrl_hips": "Hips",+ "ctrl_hips_free": "Hips",+ "ctrl_index": "Index Finger",+ "ctrl_index_end": "Index Finger End",+ "ctrl_middle": "Middle Finger",+ "ctrl_pinky": "Pinky Finger",+ "ctrl_ring": "Ring Finger",+ "ctrl_spine": "Spine",+ "ctrl_thumb": "Thumb Finger",+ "ctrl_thumb_end": "Thumb Finger End",+ "cuello": "Neck",+ "dau": "Head",+ "def_arm": "Upper Arm",+ "def_chest": "Chest",+ "def_foot": "Foot",+ "def_forarm": "Forearm",+ "def_forearm": "Forearm",+ "def_hand": "Hand",+ "def_hand_end": "Hand End",+ "def_head": "Head",+ "def_head_end": "Head End",+ "def_hips": "Hips",+ "def_neck": "Neck",+ "def_shin": "Shin",+ "def_shoulder": "Shoulder",+ "def_spine": "Spine",+ "def_thigh": "Thigh",+ "def_thumb": "Thumb Finger",+ "def_toe": "Toe",+ "def_toe_end": "Toe End",+ "def_upper_arm": "Upper Arm",+ "dos": "Spine",+ "downbody": "Lower Body",+ "ea": "Ear",+ "ear": "Ear",+ "ear_base": "Ear",+ "ear_bn": "Ear",+ "ear_end": "Ear End",+ "ear_jnt": "Ear",+ "ear_mid": "Ear",+ "ear_nub": "Ear End",+ "ear_sh_jnt": "Ear",+ "ears": "Ear",+ "elbow_jnt": "Elbow",+ "elbow_joint": "Elbow",+ "elbow_sh_jnt": "Elbow",+ "end_sh_jnt": "Jaw End",+ "eye": "Eye",+ "eye_ball": "Eyeball",+ "eye_ball_joint": "Eyeball",+ "eye_bn": "Eye",+ "eye_brow": "Eyebrow",+ "eye_end": "Eye End",+ "eye_jnt": "Eye",+ "eye_joint": "Eye",+ "eyeball": "Eyeball",+ "eyebrows": "Eyebrow",+ "eyelip_b": "Lower Eyelid",+ "eyelip_t": "Upper Eyelid",+ "eyes": "Eye",+ "f_ankle": "Front Ankle",+ "f_foot": "Front Foot",+ "f_index": "Index Finger",+ "f_index_end": "Index Finger End",+ "f_inger": "Thumb Finger",+ "f_leg": "Front Leg",+ "f_middle": "Middle Finger",+ "f_middle_end": "Middle Finger End",+ "f_pinky": "Pinky Finger",+ "f_pinky_end": "Pinky Finger End",+ "f_ring": "Ring Finger",+ "f_ring_end": "Ring Finger End",+ "f_skart": "Skirt",+ "feather": "Feather",+ "feeler": "Antenna",+ "feet": "Foot",+ "feet_end": "Foot End",+ "femur": "Thigh",+ "fibula": "Shin",+ "finger_index": "Index Finger",+ "finger_inner": "Inner Finger",+ "finger_little": "Pinky Finger",+ "finger_lower": "Finger",+ "finger_mid": "Middle Finger",+ "finger_middle": "Middle Finger",+ "finger_outter": "Outer Finger",+ "finger_pinky": "Pinky Finger",+ "finger_ring": "Ring Finger",+ "finger_thumb": "Thumb Finger",+ "fk_arm": "Upper Arm",+ "fk_up_leg": "Thigh",+ "fl_foot": "Front Foot",+ "fl_leg": "Front Leg",+ "foot": "Foot",+ "foot_end_end": "Foot End",+ "foot_f": "Front Foot",+ "foot_front": "Front Foot",+ "foot_heel": "Heel",+ "foot_jnt": "Foot",+ "foot_joint": "Foot",+ "foot_roll_ik": "Foot",+ "foot_toe": "Toe",+ "forarm": "Forearm",+ "fore_arm": "Forearm",+ "fore_arm_armature": "Forearm",+ "fore_arm_roll": "Forearm Twist",+ "fore_twist": "Forearm Twist",+ "forearm": "Forearm",+ "forearm_end": "Forearm End",+ "forearm_jnt": "Forearm",+ "forearm_stretch": "Forearm",+ "forearm_twist": "Forearm Twist",+ "foreleg_jnt": "Front Leg",+ "fr_foot": "Front Foot",+ "fr_leg": "Front Leg",+ "free": "Hips",+ "front_foot": "Front Foot",+ "front_foot_end": "Front Foot End",+ "front_leg": "Front Leg",+ "front_shin": "Front Shin",+ "front_shoulder": "Front Shoulder",+ "front_spine": "Spine",+ "front_thigh": "Front Thigh",+ "front_toe": "Front Toe",+ "front_toe_end": "Front Toe End",+ "front_upper_leg": "Front Thigh",+ "g_b_skart": "Skirt",+ "g_breast": "Chest",+ "g_f_skart": "Skirt",+ "g_skart": "Skirt",+ "g_spine": "Spine",+ "generic_rig_arm": "Upper Arm",+ "generic_rig_foot": "Foot",+ "generic_rig_fore_arm": "Forearm",+ "generic_rig_hand": "Hand",+ "generic_rig_head": "Head",+ "generic_rig_hips": "Hips",+ "generic_rig_leg": "Shin",+ "generic_rig_neck": "Neck",+ "generic_rig_shoulder": "Shoulder",+ "generic_rig_spine": "Spine",+ "generic_rig_toe_base": "Toe",+ "generic_rig_up_leg": "Thigh",+ "glb_hand_loc": "Hand",+ "glb_weapon": "Weapon",+ "gltf_created_root_joint": "Root",+ "golen": "Shin",+ "guy_arm": "Upper Arm",+ "guy_foot": "Foot",+ "guy_fore_arm": "Forearm",+ "guy_hand": "Hand",+ "guy_head": "Head",+ "guy_leg": "Shin",+ "guy_neck": "Neck",+ "guy_shoulder": "Shoulder",+ "guy_spine": "Spine",+ "guy_up_leg": "Thigh",+ "hand": "Hand",+ "hand_armature": "Hand",+ "hand_bone": "Hand",+ "hand_claw_inner": "Inner Hand Claw",+ "hand_claw_outter": "Outer Hand Claw",+ "hand_control": "Hand",+ "hand_control_end": "Hand End",+ "hand_end": "Hand End",+ "hand_end_end": "Hand End",+ "hand_index": "Index Finger",+ "hand_index_armature": "Index Finger",+ "hand_index_end": "Index Finger End",+ "hand_index_jnt": "Index Finger",+ "hand_jnt": "Hand",+ "hand_little": "Pinky Finger",+ "hand_loc": "Hand",+ "hand_middle": "Middle Finger",+ "hand_middle_armature": "Middle Finger",+ "hand_middle_end": "Middle Finger End",+ "hand_middle_jnt": "Middle Finger",+ "hand_pinky": "Pinky Finger",+ "hand_pinky_armature": "Pinky Finger",+ "hand_pinky_end": "Pinky Finger End",+ "hand_pinky_jnt": "Pinky Finger",+ "hand_ring": "Ring Finger",+ "hand_ring_armature": "Ring Finger",+ "hand_ring_end": "Ring Finger End",+ "hand_ring_jnt": "Ring Finger",+ "hand_thumb": "Thumb Finger",+ "hand_thumb_armature": "Thumb Finger",+ "hand_thumb_end": "Thumb Finger End",+ "hand_thumb_jnt": "Thumb Finger",+ "hand_top": "Hand",+ "handed_weapon": "Weapon",+ "hara": "Abdomen",+ "head": "Head",+ "head_armature": "Head",+ "head_bn": "Head",+ "head_c": "Head",+ "head_end": "Head End",+ "head_end_m": "Head End",+ "head_eye": "Eye",+ "head_head": "Head",+ "head_j": "Head",+ "head_jaw": "Jaw",+ "head_jaw_end_sh_jnt": "Jaw End",+ "head_jaw_sh_jnt": "Jaw",+ "head_jnt": "Head",+ "head_joint": "Head",+ "head_m": "Head",+ "head_neck": "Neck",+ "head_nub": "Head End",+ "head_point": "Head End",+ "head_top_end": "Head End",+ "head_top_end_armature": "Head End",+ "head_top_end_end": "Head End",+ "heel": "Heel",+ "heel_end": "Heel End",+ "hi_res_hips": "Hips",+ "hi_res_spine": "Spine",+ "hiji": "Elbow",+ "hindleg_jnt": "Hind Leg",+ "hip_armature": "Hips",+ "hip_bn": "Hips",+ "hips": "Hips",+ "hips_armature": "Hips",+ "hips_c": "Hips",+ "hips_free": "Hips",+ "hips_jnt": "Hips",+ "hiza": "Knee",+ "hombro": "Shoulder",+ "hoofd": "Head",+ "horn": "Horn",+ "horse_link": "Fetlock",+ "human_arm_collarbone": "Shoulder",+ "human_arm_forearm": "Forearm",+ "human_arm_palm": "Palm",+ "human_arm_upperarm": "Upper Arm",+ "human_calf": "Shin",+ "human_collarbone": "Shoulder",+ "human_foot": "Foot",+ "human_forearm": "Forearm",+ "human_head": "Head",+ "human_leg_calf": "Shin",+ "human_leg_thigh": "Thigh",+ "human_neck": "Neck",+ "human_palm": "Palm",+ "human_pelvis": "Hips",+ "human_ribcage": "Ribcage",+ "human_spine": "Spine",+ "human_thigh": "Thigh",+ "human_upperarm": "Upper Arm",+ "humerus": "Upper Arm",+ "hump_m_clavicle_sh_jnt": "Shoulder",+ "hump_m_front_leg_ankle_sh_jnt": "Front Ankle",+ "hump_m_front_leg_ball_sh_jnt": "Front Toe",+ "hump_m_front_leg_knee_sh_jnt": "Front Knee",+ "hump_m_head_jaw_sh_jnt": "Jaw",+ "hump_m_head_top_sh_jnt": "Head",+ "hump_m_neck_sh_jnt": "Neck",+ "hump_m_neck_top_sh_jnt": "Neck",+ "hump_m_rootsh_jnt": "Root",+ "hump_m_spine_sh_jnt": "Spine",+ "hump_m_spine_top_sh_jnt": "Spine",+ "hump_m_tail_sh_jnt": "Tail",+ "ik_arm": "Upper Arm",+ "ik_foot": "Foot",+ "ik_fore_arm": "Forearm",+ "ik_hand": "Hand",+ "ik_leg": "Shin",+ "ik_toe_base": "Toe",+ "ik_toe_end": "Toe End",+ "ik_up_leg": "Thigh",+ "in_hand_index": "Index Finger",+ "in_hand_middle": "Middle Finger",+ "in_hand_ring": "Ring Finger",+ "ind": "Index Finger",+ "index": "Index Finger",+ "index_armature": "Index Finger",+ "index_base": "Index Finger",+ "index_bn": "Index Finger",+ "index_distal": "Index Finger",+ "index_end": "Index Finger End",+ "index_end_end": "Index Finger End",+ "index_finger": "Index Finger",+ "index_intermediate": "Index Finger",+ "index_j": "Index Finger",+ "index_jnt": "Index Finger",+ "index_mid": "Index Finger",+ "index_proximal": "Index Finger",+ "index_tip": "Index Finger End",+ "indexend": "Index Finger End",+ "indicador": "Index Finger",+ "inger": "Thumb Finger",+ "init": "Bag",+ "j_ago": "Jaw",+ "j_atama": "Head",+ "j_codo": "Elbow",+ "j_ear": "Ear",+ "j_finger": "Finger",+ "j_foot": "Foot",+ "j_hair": "Mane",+ "j_hand": "Hand",+ "j_hara": "Abdomen",+ "j_head": "Head",+ "j_hiji": "Elbow",+ "j_hiza": "Knee",+ "j_kakato": "Heel",+ "j_momo": "Thigh",+ "j_mouth": "Mouth",+ "j_mune": "Chest",+ "j_neck": "Neck",+ "j_shippo": "Tail",+ "j_shoulder": "Shoulder",+ "j_spine": "Spine",+ "j_tail": "Tail",+ "j_te": "Hand",+ "j_toe": "Toe",+ "j_tuma": "Toe",+ "j_ude": "Upper Arm",+ "j_wing": "Wing",+ "j_yubi": "Finger",+ "jaw_bn": "Jaw",+ "jaw_c": "Jaw",+ "jaw_down": "Lower Jaw",+ "jaw_end": "Jaw End",+ "jaw_end_m": "Jaw End",+ "jaw_end_sh_jnt": "Jaw End",+ "jaw_m": "Jaw",+ "jaw_nub": "Jaw End",+ "jaw_sh_jnt": "Jaw",+ "jaw_tip": "Jaw End",+ "jaw_up": "Upper Jaw",+ "jaws": "Jaw",+ "jig_heel": "Heel",+ "jnt_eye_brow": "Eyebrow",+ "jnt_eyelid": "Eyelid",+ "jnt_head": "Head",+ "jnt_neck": "Neck",+ "jnt_spine": "Spine",+ "joint_ankle": "Ankle",+ "joint_head": "Head",+ "joint_knee": "Knee",+ "joint_neck": "Neck",+ "joint_shoulder": "Shoulder",+ "joint_tail": "Tail",+ "joint_toe": "Toe",+ "jt_ankle": "Ankle",+ "jt_claw_inner": "Inner Claw",+ "jt_claw_middle": "Middle Claw",+ "jt_claw_outter": "Outer Claw",+ "jt_cog_c": "Root",+ "jt_elbow": "Elbow",+ "jt_finger_inner": "Inner Finger",+ "jt_finger_outter": "Outer Finger",+ "jt_foot": "Foot",+ "jt_hand_claw_inner": "Inner Hand Claw",+ "jt_hand_claw_outter": "Outer Hand Claw",+ "jt_head_c": "Head",+ "jt_hip": "Hip",+ "jt_hips_c": "Hips",+ "jt_jaw_c": "Jaw",+ "jt_knee": "Knee",+ "jt_neck_c": "Neck",+ "jt_shoulder": "Shoulder",+ "jt_spine_c": "Spine",+ "jt_tail_c": "Tail",+ "jt_thigh": "Thigh",+ "jt_thigh_muscle": "Thigh Muscle",+ "jt_toe_inner": "Inner Toe",+ "jt_toe_middle": "Middle Toe",+ "jt_toe_outter": "Outer Toe",+ "jt_tongue_c": "Tongue",+ "jt_wrist": "Wrist",+ "kakato": "Heel",+ "kata": "Shoulder",+ "kepala": "Head",+ "kist": "Hand",+ "knee_jnt": "Knee",+ "kopf": "Head",+ "kosi": "Hips",+ "kubi": "Neck",+ "leaf": "Leaf",+ "lef_hand_end": "Hand End",+ "lefthand": "Hand",+ "leg_calf": "Shin",+ "leg_control": "Shin",+ "leg_hip": "Hip",+ "leg_j": "Thigh",+ "leg_lo_res_foot": "Foot",+ "leg_lo_res_lower_leg": "Shin",+ "leg_lo_res_upper_leg": "Thigh",+ "leg_low": "Shin",+ "leg_stretch": "Shin",+ "leg_thigh": "Thigh",+ "leg_toe": "Toe",+ "leg_top": "Thigh",+ "leg_up": "Thigh",+ "leher": "Neck",+ "leye": "Eye",+ "lf_leg": "Front Leg",+ "lfingers": "Finger",+ "lfoot": "Foot",+ "lh": "Hand",+ "lhand": "Hand",+ "lid_down": "Lower Eyelid",+ "lid_up": "Upper Eyelid",+ "lindex": "Index Finger",+ "little": "Pinky Finger",+ "little_distal": "Pinky Finger",+ "little_finger": "Pinky Finger",+ "little_intermediate": "Pinky Finger",+ "little_proximal": "Pinky Finger",+ "lo_res_foot": "Foot",+ "lo_res_lower_leg": "Shin",+ "lo_res_pelvis": "Hips",+ "lo_res_spine": "Spine",+ "lo_res_upper_leg": "Thigh",+ "locator": "Root",+ "low_arm": "Forearm",+ "low_man_arm": "Upper Arm",+ "low_man_foot": "Foot",+ "low_man_fore_arm": "Forearm",+ "low_man_hand": "Hand",+ "low_man_hand_index": "Index Finger",+ "low_man_hand_middle": "Middle Finger",+ "low_man_hand_pinky": "Pinky Finger",+ "low_man_hand_ring": "Ring Finger",+ "low_man_hand_thumb": "Thumb Finger",+ "low_man_head": "Head",+ "low_man_head_end": "Head End",+ "low_man_hips": "Hips",+ "low_man_leg": "Shin",+ "low_man_neck": "Neck",+ "low_man_shoulder": "Shoulder",+ "low_man_spine": "Spine",+ "low_man_toe_base": "Toe",+ "low_man_toe_base_end": "Toe End",+ "low_man_up_leg": "Thigh",+ "low_wing": "Lower Wing",+ "lower_arm": "Forearm",+ "lower_arm_armature": "Forearm",+ "lower_eye_lid": "Lower Eyelid",+ "lower_eyelid": "Lower Eyelid",+ "lower_jaw": "Lower Jaw",+ "lower_leg": "Shin",+ "lower_leg_armature": "Shin",+ "lower_lid": "Lower Eyelid",+ "lower_lid_bn": "Lower Eyelid",+ "lowerarm": "Forearm",+ "lowerarm_twist": "Forearm Twist",+ "lowerleg": "Shin",+ "lowerleg_twist": "Shin Twist",+ "lthumb": "Thumb Finger",+ "m_clavicle_sh_jnt": "Shoulder",+ "m_front_leg_ankle_sh_jnt": "Front Ankle",+ "m_front_leg_ball_sh_jnt": "Front Toe",+ "m_front_leg_knee_sh_jnt": "Front Knee",+ "m_head_jaw_sh_jnt": "Jaw",+ "m_head_top_sh_jnt": "Head",+ "m_neck_jnt": "Neck",+ "m_neck_sh_jnt": "Neck",+ "m_neck_top_sh_jnt": "Neck",+ "m_rootsh_jnt": "Root",+ "m_spine_sh_jnt": "Spine",+ "m_spine_top_sh_jnt": "Spine",+ "m_tail_sh_jnt": "Tail",+ "main_body": "Body",+ "main_character_arm": "Upper Arm",+ "main_character_foot": "Foot",+ "main_character_fore_arm": "Forearm",+ "main_character_hand": "Hand",+ "main_character_head": "Head",+ "main_character_hips": "Hips",+ "main_character_leg": "Shin",+ "main_character_neck": "Neck",+ "main_character_shoulder": "Shoulder",+ "main_character_spine": "Spine",+ "main_character_toe_base": "Toe",+ "main_character_up_leg": "Thigh",+ "man_arm": "Upper Arm",+ "man_foot": "Foot",+ "man_fore_arm": "Forearm",+ "man_hand": "Hand",+ "man_hand_index": "Index Finger",+ "man_hand_middle": "Middle Finger",+ "man_hand_pinky": "Pinky Finger",+ "man_hand_ring": "Ring Finger",+ "man_hand_thumb": "Thumb Finger",+ "man_head": "Head",+ "man_head_end": "Head End",+ "man_hips": "Hips",+ "man_leg": "Shin",+ "man_neck": "Neck",+ "man_shoulder": "Shoulder",+ "man_spine": "Spine",+ "man_toe_base": "Toe",+ "man_toe_base_end": "Toe End",+ "man_up_leg": "Thigh",+ "mandible_jnt": "Mandible",+ "mandibre_jnt": "Mandible",+ "mane": "Mane",+ "mano": "Hand",+ "mao": "Hand",+ "mao_end": "Hand End",+ "master": "Root",+ "medio": "Middle Finger",+ "middle": "Middle Finger",+ "middle_armature": "Middle Finger",+ "middle_base": "Middle Finger",+ "middle_bn": "Middle Finger",+ "middle_distal": "Middle Finger",+ "middle_end": "Middle Finger End",+ "middle_finger": "Middle Finger",+ "middle_hand": "Middle Finger",+ "middle_intermediate": "Middle Finger",+ "middle_j": "Middle Finger",+ "middle_proximal": "Middle Finger",+ "middle_tip": "Middle Finger End",+ "middleend": "Middle Finger End",+ "minimo": "Pinky Finger",+ "mixamorig_arm": "Upper Arm",+ "mixamorig_eye": "Eye",+ "mixamorig_foot": "Foot",+ "mixamorig_fore_arm": "Forearm",+ "mixamorig_hand": "Hand",+ "mixamorig_hand_index": "Index Finger",+ "mixamorig_hand_middle": "Middle Finger",+ "mixamorig_hand_pinky": "Pinky Finger",+ "mixamorig_hand_ring": "Ring Finger",+ "mixamorig_hand_thumb": "Thumb Finger",+ "mixamorig_head": "Head",+ "mixamorig_head_top_end": "Head End",+ "mixamorig_hips": "Hips",+ "mixamorig_leg": "Shin",+ "mixamorig_neck": "Neck",+ "mixamorig_shoulder": "Shoulder",+ "mixamorig_spine": "Spine",+ "mixamorig_toe_base": "Toe",+ "mixamorig_toe_end": "Toe End",+ "mixamorig_up_leg": "Thigh",+ "momo": "Thigh",+ "mune": "Chest",+ "muslo": "Thigh",+ "muzzle": "Muzzle",+ "neck": "Neck",+ "neck_armature": "Neck",+ "neck_base": "Neck",+ "neck_bn": "Neck",+ "neck_bone": "Neck",+ "neck_c": "Neck",+ "neck_jnt": "Neck",+ "neck_m": "Neck",+ "neck_sh_jnt": "Neck",+ "neck_top": "Neck",+ "nek": "Neck",+ "nguc": "Chest",+ "nose": "Nose",+ "nose_jnt": "Nose",+ "nose_m": "Nose",+ "ombro": "Shoulder",+ "orelha": "Ear",+ "org_chest": "Chest",+ "org_foot": "Foot",+ "org_forearm": "Forearm",+ "org_hand": "Hand",+ "org_head": "Head",+ "org_hips": "Hips",+ "org_neck": "Neck",+ "org_shin": "Shin",+ "org_shoulder": "Shoulder",+ "org_spine": "Spine",+ "org_thigh": "Thigh",+ "org_toe": "Toe",+ "org_upper_arm": "Upper Arm",+ "panther_head": "Head",+ "panther_leg_collarbone": "Shoulder",+ "panther_neck": "Neck",+ "panther_pelvis": "Hips",+ "panther_spine": "Spine",+ "panther_tail": "Tail",+ "parenting_hand_loc": "Hand",+ "pe_end": "Foot End",+ "pecho": "Chest",+ "pelvis_c": "Hips",+ "pelvis_jnt": "Hips",+ "pelvis_joint": "Hips",+ "pesco_o": "Neck",+ "pink": "Pinky Finger",+ "pinkie": "Pinky Finger",+ "pinky": "Pinky Finger",+ "pinky_armature": "Pinky Finger",+ "pinky_base": "Pinky Finger",+ "pinky_bn": "Pinky Finger",+ "pinky_end": "Pinky Finger End",+ "pinky_finger": "Pinky Finger",+ "pinky_jnt": "Pinky Finger",+ "pinky_tip": "Pinky Finger End",+ "pinkyend": "Pinky Finger End",+ "pnk": "Pinky Finger",+ "pointer": "Index Finger",+ "polegar": "Thumb Finger",+ "queue": "Tail",+ "quick_rig_character_arm": "Upper Arm",+ "quick_rig_character_foot": "Foot",+ "quick_rig_character_fore_arm": "Forearm",+ "quick_rig_character_hand": "Hand",+ "quick_rig_character_head": "Head",+ "quick_rig_character_hips": "Hips",+ "quick_rig_character_leg": "Shin",+ "quick_rig_character_neck": "Neck",+ "quick_rig_character_shoulder": "Shoulder",+ "quick_rig_character_spine": "Spine",+ "quick_rig_character_toe_base": "Toe",+ "quick_rig_character_up_leg": "Thigh",+ "quiver": "Bag",+ "radio": "Forearm",+ "res_foot": "Foot",+ "res_hips": "Hips",+ "res_lower_leg": "Shin",+ "res_pelvis": "Hips",+ "res_spine": "Spine",+ "res_toe": "Toe",+ "res_upper_leg": "Thigh",+ "rex_arm_clavicle_sh_jnt": "Shoulder",+ "rex_arm_elbow_sh_jnt": "Elbow",+ "rex_arm_shoulder_sh_jnt": "Shoulder",+ "rex_arm_wrist_sh_jnt": "Wrist",+ "rex_head_jaw_sh_jnt": "Jaw",+ "rex_leg_ankle_sh_jnt": "Ankle",+ "rex_leg_ball_sh_jnt": "Toe",+ "rex_leg_hip_sh_jnt": "Hip",+ "rex_leg_knee_sh_jnt": "Knee",+ "rex_leg_toe_sh_jnt": "Toe",+ "rex_neck_sh_jnt": "Neck",+ "rex_rootsh_jnt": "Root",+ "rex_spine_sh_jnt": "Spine",+ "rex_tail_sh_jnt": "Tail",+ "rex_toes_sh_jnt": "Toe",+ "rex_tounq_sh_jnt": "Tongue",+ "reye": "Eye",+ "rf_leg": "Front Leg",+ "rfingers": "Finger",+ "rfoot": "Foot",+ "rh": "Hand",+ "rhand": "Hand",+ "ribcage": "Ribcage",+ "ribs": "Ribcage",+ "rig_ankle_bone": "Toe",+ "rig_arm_collarbone": "Shoulder",+ "rig_arm_index": "Index Finger",+ "rig_arm_pinky": "Pinky Finger",+ "rig_arm_thumb": "Thumb Finger",+ "rig_calf": "Shin",+ "rig_character_arm": "Upper Arm",+ "rig_character_foot": "Foot",+ "rig_character_fore_arm": "Forearm",+ "rig_character_hand": "Hand",+ "rig_character_head": "Head",+ "rig_character_hips": "Hips",+ "rig_character_leg": "Shin",+ "rig_character_neck": "Neck",+ "rig_character_shoulder": "Shoulder",+ "rig_character_spine": "Spine",+ "rig_character_toe_base": "Toe",+ "rig_character_up_leg": "Thigh",+ "rig_chest": "Chest",+ "rig_clavicle": "Shoulder",+ "rig_ear": "Ear",+ "rig_foot": "Foot",+ "rig_fore_arm": "Forearm",+ "rig_forearm": "Forearm",+ "rig_hand": "Hand",+ "rig_head": "Head",+ "rig_head_end": "Head End",+ "rig_hips": "Hips",+ "rig_jaw": "Jaw",+ "rig_leg_digit": "Toe",+ "rig_leg_toes": "Toe",+ "rig_neck": "Neck",+ "rig_pelvis": "Hips",+ "rig_ribcage": "Ribcage",+ "rig_shoulder": "Shoulder",+ "rig_tail": "Tail",+ "rig_tail_end": "Tail End",+ "rig_thigh": "Thigh",+ "rig_toe": "Toe",+ "rig_toe_base": "Toe",+ "rig_tongue": "Tongue",+ "rig_up_leg": "Thigh",+ "rig_upper_arm": "Upper Arm",+ "rig_wing": "Wing",+ "rig_wing_end": "Wing End",+ "righthand": "Hand",+ "rindex": "Index Finger",+ "ring": "Ring Finger",+ "ring_armature": "Ring Finger",+ "ring_base": "Ring Finger",+ "ring_bn": "Ring Finger",+ "ring_distal": "Ring Finger",+ "ring_end": "Ring Finger End",+ "ring_finger": "Ring Finger",+ "ring_intermediate": "Ring Finger",+ "ring_j": "Ring Finger",+ "ring_jnt": "Ring Finger",+ "ring_mid": "Ring Finger",+ "ring_proximal": "Ring Finger",+ "ring_tip": "Ring Finger End",+ "ringend": "Ring Finger End",+ "rng": "Ring Finger",+ "robo_arm": "Upper Arm",+ "robo_foot": "Foot",+ "robo_fore_arm": "Forearm",+ "robo_hand": "Hand",+ "robo_hand_middle": "Middle Finger",+ "robo_hand_pinky": "Pinky Finger",+ "robo_hand_thumb": "Thumb Finger",+ "robo_hips": "Hips",+ "robo_leg": "Shin",+ "robo_neck": "Neck",+ "robo_shoulder": "Shoulder",+ "robo_spine": "Spine",+ "robo_toe_base": "Toe",+ "robo_up_leg": "Thigh",+ "roll_ik": "Foot",+ "root_joint": "Root",+ "rootsh_jnt": "Root",+ "rright_ear": "Ear",+ "rt_elbow_jnt": "Elbow",+ "rt_hip_jnt": "Hip",+ "rt_knee_jnt": "Knee",+ "rthumb": "Thumb Finger",+ "scapula": "Scapula",+ "schulter": "Shoulder",+ "shank": "Shin",+ "shell": "Shell",+ "shin": "Shin",+ "shin_end": "Shin End",+ "shippo": "Tail",+ "sholder_jnt": "Shoulder",+ "shoulder": "Shoulder",+ "shoulder_armature": "Shoulder",+ "shoulder_jnt": "Shoulder",+ "shoulder_sh_jnt": "Shoulder",+ "skart": "Skirt",+ "skeleton_root": "Root",+ "skirt_bn": "Appendage",+ "skull": "Skull",+ "snout": "Muzzle",+ "spin": "Spine",+ "spine": "Spine",+ "spine_a": "Spine",+ "spine_armature": "Spine",+ "spine_b": "Spine",+ "spine_bn": "Spine",+ "spine_c": "Spine",+ "spine_end": "Spine End",+ "spine_hi_res_hips": "Hips",+ "spine_hi_res_spine": "Spine",+ "spine_jnt": "Spine",+ "spine_joint": "Spine",+ "spine_lo_res_pelvis": "Hips",+ "spine_lo_res_spine": "Spine",+ "spine_m": "Spine",+ "spine_part_m": "Spine",+ "spine_sh_jnt": "Spine",+ "spline": "Spine",+ "stick": "Weapon",+ "stomach": "Abdomen",+ "stopa": "Foot",+ "t_rex_arm_clavicle_sh_jnt": "Shoulder",+ "t_rex_arm_elbow_sh_jnt": "Elbow",+ "t_rex_arm_shoulder_sh_jnt": "Shoulder",+ "t_rex_arm_wrist_sh_jnt": "Wrist",+ "t_rex_head_jaw_sh_jnt": "Jaw",+ "t_rex_leg_ankle_sh_jnt": "Ankle",+ "t_rex_leg_ball_sh_jnt": "Toe",+ "t_rex_leg_hip_sh_jnt": "Hip",+ "t_rex_leg_knee_sh_jnt": "Knee",+ "t_rex_leg_toe_sh_jnt": "Toe",+ "t_rex_neck_sh_jnt": "Neck",+ "t_rex_rootsh_jnt": "Root",+ "t_rex_spine_sh_jnt": "Spine",+ "t_rex_tail_sh_jnt": "Tail",+ "t_rex_toes_sh_jnt": "Toe",+ "t_rex_tounq_sh_jnt": "Tongue",+ "tail": "Tail",+ "tail_a": "Tail",+ "tail_b": "Tail",+ "tail_base": "Tail",+ "tail_bone": "Tail",+ "tail_c": "Tail",+ "tail_end": "Tail End",+ "tail_end_jnt": "Tail End",+ "tail_jnt": "Tail",+ "tail_joint": "Tail",+ "tail_m": "Tail",+ "tail_nub": "Tail End",+ "tail_root": "Tail",+ "tail_sh_jnt": "Tail",+ "te": "Hand",+ "tete": "Head",+ "thigh": "Thigh",+ "thigh_jnt": "Thigh",+ "thigh_muscle": "Thigh Muscle",+ "thigh_stretch": "Thigh",+ "thigh_twist": "Thigh Twist",+ "thm": "Thumb Finger",+ "thorax": "Chest",+ "thorax_jnt": "Chest",+ "thumb": "Thumb Finger",+ "thumb_armature": "Thumb Finger",+ "thumb_b": "Thumb Finger",+ "thumb_base": "Thumb Finger",+ "thumb_bn": "Thumb Finger",+ "thumb_distal": "Thumb Finger",+ "thumb_end": "Thumb Finger End",+ "thumb_f_inger": "Thumb Finger",+ "thumb_finger": "Thumb Finger",+ "thumb_intermediate": "Thumb Finger",+ "thumb_j": "Thumb Finger",+ "thumb_jnt": "Thumb Finger",+ "thumb_knuckle": "Thumb Finger",+ "thumb_lower": "Thumb Finger",+ "thumb_mid": "Thumb Finger",+ "thumb_middle": "Thumb Finger",+ "thumb_proximal": "Thumb Finger",+ "thumb_tip": "Thumb Finger End",+ "thumbend": "Thumb Finger End",+ "tiptoe": "Toe",+ "tobillo": "Ankle",+ "toe": "Toe",+ "toe_base": "Toe",+ "toe_base_armature": "Toe",+ "toe_base_end": "Toe End",+ "toe_end": "Toe End",+ "toe_end_armature": "Toe End",+ "toe_end_end": "Toe End",+ "toe_inner": "Inner Toe",+ "toe_j": "Toe",+ "toe_jnt": "Toe",+ "toe_middle": "Middle Toe",+ "toe_nub": "Toe End",+ "toe_outter": "Outer Toe",+ "toe_tip": "Toe End",+ "toebase_jnt": "Toe",+ "toes_armature": "Toe",+ "toes_end": "Toe End",+ "toes_sh_jnt": "Toe",+ "tongue": "Tongue",+ "tongue_c": "Tongue",+ "tongue_end": "Tongue End",+ "tongue_m": "Tongue",+ "top_end": "Head End",+ "top_end_armature": "Head End",+ "top_end_end": "Head End",+ "tounq_sh_jnt": "Tongue",+ "tuma": "Toe",+ "two_handed_weapon": "Weapon",+ "umero": "Upper Arm",+ "up_arm": "Upper Arm",+ "up_arm_twist": "Upper Arm Twist",+ "up_leg": "Thigh",+ "up_leg_armature": "Thigh",+ "up_leg_roll": "Thigh Twist",+ "up_leg_twist": "Thigh Twist",+ "up_reg": "Thigh",+ "up_wing": "Upper Wing",+ "uparm": "Upper Arm",+ "upleg_jnt": "Thigh",+ "upper_arm": "Upper Arm",+ "upper_arm_armature": "Upper Arm",+ "upper_body": "Upper Body",+ "upper_eye_lid": "Upper Eyelid",+ "upper_eyelid": "Upper Eyelid",+ "upper_jaw": "Upper Jaw",+ "upper_leg": "Thigh",+ "upper_leg_armature": "Thigh",+ "upper_lid": "Upper Eyelid",+ "upper_lid_bn": "Upper Eyelid",+ "upper_torso": "Upper Body",+ "upperarm": "Upper Arm",+ "upperarm_twist": "Upper Arm Twist",+ "upperleg": "Thigh",+ "upperleg_twist": "Thigh Twist",+ "waist_j": "Waist",+ "wattle": "Chin",+ "weapon": "Weapon",+ "weiba": "Tail",+ "wing_endjnt": "Wing End",+ "wing_jnt": "Wing",+ "wing_joint": "Wing",+ "wingroot_jnt": "Wing",+ "wings": "Wing",+ "wrist_j": "Wrist",+ "wrist_jnt": "Wrist",+ "wrist_joint": "Wrist",+ "wrist_sh_jnt": "Wrist",+ "yubi": "Finger"+ }+}\ No newline at end of filediff --git a/data_process/joint_annotation/llm.py b/data_process/joint_annotation/llm.pyindex 62a8bd9..bf74a86 100644--- a/data_process/joint_annotation/llm.py+++ b/data_process/joint_annotation/llm.py@@ -158,8 +158,8 @@ def generate_openai(system_prompt, user_prompt, model_name, api_key, base_url,# DeepSeek V4 thinks by default (effort 'high') and the reasoning# tokens count against max_tokens. Effort 'none' disables thinking โ# but V4-Flash WITHOUT thinking drifts on long batched arrays- # (systematic label/side misalignment on ~10% of truebones rigs,- # A/B'd 2026-08-30), so default to the cheapest thinking tier.+ # (systematic label/side misalignment on ~10% of truebones rigs in+ # an A/B test), so default to the cheapest thinking tier.# Thinking mode ignores temperature.kwargs["reasoning_effort"] = reasoning_effort or "low"if kwargs["reasoning_effort"] == "none":@@ -311,8 +311,12 @@ class FatalLLMError(Exception):rig of a batch run to rule-based output."""-class RigTimeout(Exception):- """Raised by the per-rig SIGALRM handler to abort an over-budget rig."""+class RigTimeout(BaseException):+ """Raised by the per-rig SIGALRM handler to abort an over-budget rig.++ A ``BaseException``: the alarm usually fires inside the HTTP client, whose+ ``except Exception`` would otherwise catch it and retry the request with no+ budget left. Callers catch it by name."""def _alarm_handler(signum, frame):diff --git a/data_process/joint_annotation/name_map.py b/data_process/joint_annotation/name_map.pynew file mode 100644index 0000000..a23c1e4--- /dev/null+++ b/data_process/joint_annotation/name_map.py@@ -0,0 +1,250 @@+"""Reviewed joint labels as a lookup for rule annotation: ``learned_name_map.json.gz``.++UniML3D's reviewed stage-3 labels (LLM output, human review and patches) are+very consistent across rigs built from the same template: Mixamo, Biped,+Rigify and other rigging tools give many assets the same bones. Two lookups,+learned from a dataset export and applied after the rules+(``rig_preprocess.annotate.rule_annotation``):++- **templates**: a rig whose bone tree equals a reviewed rig's, by normalized+ bone names (:func:`norm_name`) and their parents, takes that template's+ majority label at every joint. The tree is compared in a canonical order+ (:func:`canonical_order`), so the same rig with its bones enumerated in+ another order (an FBX and the GLB made from it) still matches. Where sibling+ subtrees are identical by names (two legs whose bones carry no side) and+ their reviewed labels differ, the template cannot tell them apart and the+ rule label is kept;+- **names**: otherwise, a normalized bone name whose reviewed label agreed in at+ least ``min_purity`` of its occurrences takes that label, but only where the+ rules left a placeholder or a non-canonical label: a valid rule label is+ never overridden by a name seen in another dataset's convention.++Learned from the released datasets (``DATASETS``) only.++ python -m data_process.joint_annotation.name_map # all reviewed rigs+ python -m data_process.joint_annotation.name_map --split dev --out /tmp/m.json.gz+"""++import argparse+import datetime+import gzip+import hashlib+import json+import os+import re+import sys+from collections import Counter, defaultdict++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++import numpy as np # noqa: E402++from data_process.joint_annotation.learn_vocab import in_split # noqa: E402+from data_process.joint_annotation.names_clean_rule import is_canonical_label # noqa: E402+from data_process.tools.patch_annotations import rig_npz # noqa: E402++DEFAULT_PATH = os.path.join(os.path.dirname(__file__), 'learned_name_map.json.gz')+# Datasets the map is learned from.+DATASETS = ('objaverse', 'mixamo', 'truebones')+MIN_PURITY = 0.9++_NAMESPACE_RE = re.compile(r'^.*[:|]')+_COUNTER_RE = re.compile(r'(?:[._]\d+)+$')+# Rule labels the per-name lookup may replace: placeholders ('Bone', numeric+# pass-throughs, sided or not) and labels outside the canonical vocabulary.+_PLACEHOLDER_RE = re.compile(r'^(?:Left |Right )?(?:Bone|_?\d+)$')+++def norm_name(raw):+ """A bone name without namespace and trailing counters, lower-cased+ ('mixamorig:LeftUpLeg' -> 'leftupleg', 'Foot.R_035_34' -> 'foot.r'). Side+ markers stay: they are part of the label."""+ return _COUNTER_RE.sub('', _NAMESPACE_RE.sub('', str(raw).strip())).lower()+++def _tree(norm_names, parents):+ """``(children, roots, sig)``: child lists, roots, and each joint's subtree+ signature (its name and its children's signatures, sorted)."""+ children, roots = defaultdict(list), []+ for i, p in enumerate(parents):+ (children[p] if p >= 0 else roots).append(i)+ signature = {}++ def sig(i):+ if i not in signature:+ signature[i] = norm_names[i] + '(' + ','.join(sorted(sig(c) for c in children[i])) + ')'+ return signature[i]++ return children, roots, sig+++def canonical_order(norm_names, parents):+ """Joint indices in a depth-first order that does not depend on how the+ bones were enumerated: roots and siblings sorted by their subtree (names,+ recursively). Sibling subtrees that are identical by names (see+ :func:`tied_blocks`) keep their enumeration order."""+ children, roots, sig = _tree(norm_names, parents)+ order, stack = [], sorted(roots, key=sig, reverse=True)+ while stack:+ i = stack.pop()+ order.append(i)+ stack.extend(sorted(children[i], key=sig, reverse=True))+ return order+++def tied_blocks(norm_names, parents, order):+ """Sibling subtrees identical by names, whose relative order is the+ enumeration's (the two legs of a rig whose bones carry no side): one list+ of ``(start, size)`` ranges of *order* per set of such siblings."""+ children, roots, sig = _tree(norm_names, parents)+ size = {}+ for i in reversed(order):+ size[i] = 1 + sum(size[c] for c in children[i])+ pos = {i: k for k, i in enumerate(order)}+ groups = []+ for siblings in [roots, *children.values()]:+ by_sig = defaultdict(list)+ for i in siblings:+ by_sig[sig(i)].append(i)+ groups += [sorted((pos[i], size[i]) for i in same)+ for same in by_sig.values() if len(same) > 1]+ return groups+++def _template(norm_names, parents):+ """``(key, order)``: the tree's template key and its canonical joint order.+ Each joint enters the key as its name and its parent's position in that+ order, so trees whose bones share one name (``Bone.001``, ``Bone.002``)+ still get different keys when their shapes differ."""+ order = canonical_order(norm_names, parents)+ pos = {i: k for k, i in enumerate(order)}+ edges = [f'{norm_names[i]}<{pos[parents[i]] if parents[i] >= 0 else ""}' for i in order]+ return hashlib.sha1('\n'.join(edges).encode()).hexdigest()[:20], order+++def _ambiguous(seq, groups):+ """Positions of label sequence *seq* (canonical order) inside tied sibling+ subtrees whose labels differ at that position: there the template cannot+ tell which subtree is which."""+ out = set()+ for group in groups:+ for k in range(group[0][1]):+ if len({seq[start + k] for start, _ in group}) > 1:+ out.update(start + k for start, _ in group)+ return out+++def weak_label(label):+ """Whether a rule label is a placeholder or outside the canonical vocabulary."""+ return bool(_PLACEHOLDER_RE.match(label)) or not is_canonical_label(label)+++def learn_name_map(export_root='dataset/export', split=None, min_purity=MIN_PURITY,+ datasets=DATASETS):+ """Learn the map from ``<export_root>/<ds>/{joint_names,clean_joint_names}.json``+ and each rig's export NPZ (its parents), for one *split* half of the rigs,+ or all."""+ labels, label_index = [], {}++ def idx(label):+ if label not in label_index:+ label_index[label] = len(labels)+ labels.append(label)+ return label_index[label]++ templates, ambiguous, names = defaultdict(list), defaultdict(set), defaultdict(Counter)+ for ds in datasets:+ root = os.path.join(export_root, ds)+ path = os.path.join(root, 'clean_joint_names.json')+ if not os.path.isfile(path):+ continue+ with open(os.path.join(root, 'joint_names.json')) as f:+ raw_names = json.load(f)+ with open(path) as f:+ clean = json.load(f)+ for rig, raw in raw_names.items():+ lab = clean.get(rig)+ if not lab or len(lab) != len(raw) or not in_split(rig, split):+ continue+ normed = [norm_name(n) for n in raw]+ npz = rig_npz(root, ds, rig)+ if npz:+ with np.load(npz, allow_pickle=True) as data:+ parents = [int(p) for p in data['parents']]+ if len(parents) == len(raw):+ key, order = _template(normed, parents)+ seq = [lab[i] for i in order]+ templates[key].append(seq)+ ambiguous[key] |= _ambiguous(seq, tied_blocks(normed, parents, order))+ for n, label in zip(normed, lab):+ names[n][label] += 1+ out_templates = {+ key: [-1 if i in ambiguous[key] else idx(Counter(rig[i] for rig in rigs).most_common(1)[0][0])+ for i in range(len(rigs[0]))]+ for key, rigs in templates.items()}+ out_names = {}+ for n, counts in names.items():+ label, count = counts.most_common(1)[0]+ if count / sum(counts.values()) >= min_purity:+ out_names[n] = idx(label)+ return {'labels': labels, 'templates': out_templates, 'names': out_names}+++def save_name_map(name_map, path, meta):+ with gzip.open(path, 'wt', encoding='utf-8') as f:+ json.dump({'_meta': meta, **name_map}, f, separators=(',', ':'))+++def load_name_map(path=DEFAULT_PATH, _cache={}):+ """The shipped (or given) map; an empty one when the file is absent."""+ if path not in _cache:+ if os.path.isfile(path):+ with gzip.open(path, 'rt', encoding='utf-8') as f:+ data = json.load(f)+ data.pop('_meta', None)+ else:+ data = {'labels': [], 'templates': {}, 'names': {}}+ _cache[path] = data+ return _cache[path]+++def apply_name_map(name_map, raw_names, labels, parents=None):+ """``(labels, source)``: *labels* (one per raw name) with the template of the+ rig's tree (needs *parents*), else with the confident per-name labels in+ place of weak rule labels. *source* is 'template', 'names' or 'none'."""+ vocab = name_map.get('labels', [])+ normed = [norm_name(n) for n in raw_names]+ if parents is not None and len(parents) == len(labels):+ key, order = _template(normed, [int(p) for p in parents])+ tmpl = name_map.get('templates', {}).get(key)+ if tmpl is not None and len(tmpl) == len(labels):+ out = list(labels)+ for pos, i in enumerate(order):+ if tmpl[pos] >= 0:+ out[i] = vocab[tmpl[pos]]+ return out, 'template'+ table = name_map.get('names', {})+ out = [vocab[table[n]] if n in table and weak_label(label) else label+ for n, label in zip(normed, labels)]+ return out, ('names' if out != list(labels) else 'none')+++def main():+ ap = argparse.ArgumentParser(description="Learn learned_name_map.json.gz from reviewed labels.")+ ap.add_argument('--export_root', default='dataset/export')+ ap.add_argument('--split', choices=('dev', 'test'), default=None,+ help="Learn from one half of the rigs only (to evaluate on the other).")+ ap.add_argument('--min_purity', type=float, default=MIN_PURITY)+ ap.add_argument('--out', default=DEFAULT_PATH)+ args = ap.parse_args()+ name_map = learn_name_map(args.export_root, args.split, args.min_purity)+ meta = {'datasets': list(DATASETS), 'split': args.split, 'min_purity': args.min_purity,+ 'date': datetime.date.today().isoformat(),+ 'templates': len(name_map['templates']), 'names': len(name_map['names'])}+ save_name_map(name_map, args.out, meta)+ print(f"{meta['templates']} templates, {meta['names']} names -> {args.out} "+ f"({os.path.getsize(args.out) / 1e6:.2f} MB)")+++if __name__ == '__main__':+ main()diff --git a/data_process/joint_annotation/names_clean_rule.py b/data_process/joint_annotation/names_clean_rule.pyindex 00660c9..5a4e222 100644--- a/data_process/joint_annotation/names_clean_rule.py+++ b/data_process/joint_annotation/names_clean_rule.py@@ -16,6 +16,7 @@ import osimport reimport sysimport tempfile+import unicodedata_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))if _REPO_ROOT not in sys.path:@@ -24,6 +25,7 @@ if _REPO_ROOT not in sys.path:from data_process.joint_annotation.vocab import (CANONICAL,ELK_MAP,+ FOREIGN_PARTS,JAPANESE_COMPOUND_ANIMALS,JAPANESE_COMPOUND_NAMES,JAPANESE_WORDS_LOWER,@@ -33,8 +35,10 @@ from data_process.joint_annotation.vocab import (PIRRANA_COMPOUNDS,REMOVE_PREFIXES,SABRECAT_MAP,+ SIDE_WORDS,SPIDER_MAP,STANDALONE_MAP,+ is_canonical_label,)@@ -67,6 +71,8 @@ _TRAILING_TOKEN_RE = re.compile(r'[._](\d+|[LRlr]|x)$')_FINGER_CODE_RE = re.compile(r'^[Ff]inger([0-4])\d*(Nub)?$')_FINGER_CODE = {"0": "Thumb", "1": "Index", "2": "Middle", "3": "Ring","4": "Pinky"}+# 3ds Max CAT digits (1-based finger, then segment): 'Digit21' = index.+_DIGIT_CODE_RE = re.compile(r'^Digit([1-5])\d*$')# CMU / mocap-style segmented fingers (1-based): 'Finger1Metacarpal' etc._FINGER_SEG_RE = re.compile(r'^[Ff]inger([1-5])(Metacarpal|Proximal|Medial|Distal|Tip)\d*$')@@ -87,6 +93,108 @@ _NAMESPACE_RE = re.compile(r'^[A-Za-z][\w .-]*:')_BIP_PREFIX_RE = re.compile(r'^(?:BN_)?Bip\d+(?:[-_ ]+|(?=[LR][A-Z]))')+# An end bone's name ending: 'LeftToesEND', 'Foot.L_end', 'Bip01 R Toe0Nub'.+_END_SUFFIX_RE = re.compile(r'(?:[_.\s]end|End|END|Nub)$')+# FBX ASCII escapes in names written by some exporters ('FBXASC032' = space).+_FBX_ESCAPE_RE = re.compile(r'FBXASC(\d{3})')+# Words of a raw name: capital runs ('CAT', 'R'), Capitalized or lower-case+# words, digit runs ('LowManRightUpLeg' -> Low Man Right Up Leg).+_WORD_RE = re.compile(r'[A-Z]+(?![a-z])|[A-Z]?[a-z]+|\d+')+# Leg codes of legged robots and quadruped rigs: Front / Rear / Back / Hind,+# then the side ('FR_hip', 'RL_calf', 'fl_knee'); the reviewed labels write+# every hind leg as 'Back'. Accepted only when the rest is a known part.+_LEG_CODE_RE = re.compile(r'^([FRBHfrbh])([LRlr])[_.](\w+)$')+_LEG_QUALIFIER = {'F': 'Front', 'R': 'Back', 'B': 'Back', 'H': 'Back'}+++def extract_leg_code(name):+ # type: (str) -> Tuple[str, str]+ """``(side, label)`` of a leg-coded name ('RL_calf' -> Left, 'Back Shin'),+ else ``('', '')``. Runs before the generic side rules, which would read+ 'RL_' as Right."""+ m = _LEG_CODE_RE.match(name)+ if not m:+ return '', ''+ label = _part_label(_WORD_RE.findall(m.group(3)))+ if not label or label == 'Bone':+ return '', ''+ return ('Left' if m.group(2).upper() == 'L' else 'Right'), \+ f'{_LEG_QUALIFIER[m.group(1).upper()]} {label}'+# A lower-case side letter glued to, or dotted onto, a part word ('rthigh',+# 'l.brazo'); accepted only when the rest is a known part.+_LOWER_SIDE_RE = re.compile(r'^([lr])[._]?([a-z]{2,})\d*$')+++def _part_label(tokens):+ """Canonical label of a run of word tokens ('Up', 'Leg' -> 'Thigh';+ 'Muslo' -> 'Thigh'), or '' when none of it is a known part."""+ words = [t for t in tokens if not t.isdigit()]+ if not words:+ return ''+ joined = _canon(''.join(w[:1].upper() + w[1:] for w in words))+ if joined:+ return joined+ out = []+ for w in words:+ value = _canon(w)+ if value is None:+ plain = unicodedata.normalize('NFKD', w).encode('ascii', 'ignore').decode().lower()+ value = FOREIGN_PARTS.get(plain)+ if value:+ out.append(value)+ if not out:+ # A canonical part no map lists ('hip'), when nothing else names it.+ out = [w.capitalize() for w in words+ if w.lower() not in ('end', 'nub') and is_canonical_label(w.capitalize())]+ return ' '.join(out)+++def extract_inner_side(name):+ # type: (str) -> Tuple[str, str]+ """Side word inside a name that carries rig words around it, for names the+ prefix/suffix rules miss: 'LowManRightUpLeg', 'robo_RightUpLeg',+ 'HumanRCollarbone', 'CATRigRLeg1', 'Muslo_Der_01', 'jambe_droite',+ 'rthigh_12', 'l.brazo_1'.++ A side word (``vocab.SIDE_WORDS``, any of its languages) or a single+ capital L/R between words marks the side; the label comes from the words+ after the marker, else before it, and must be a known part (``CANONICAL``+ or ``vocab.FOREIGN_PARTS``) โ rig words alone never produce a side.+ Returns ``(side, label)``, or ``('', '')``.+ """+ m = _LOWER_SIDE_RE.match(name)+ if m:+ label = _part_label([m.group(2)])+ if label and label != 'Bone':+ return ('Left' if m.group(1) == 'l' else 'Right'), label+ tokens = _WORD_RE.findall(name)+ marks, sides = [], set()+ for i, tok in enumerate(tokens):+ side = SIDE_WORDS.get(tok.lower())+ if side is None and tok in ('L', 'R') and i > 0:+ side = 'Left' if tok == 'L' else 'Right'+ if side:+ marks.append(i)+ sides.add(side)+ if len(sides) != 1:+ return '', ''+ m = re.search(r'Digit([1-5])', name)+ if m:+ return sides.pop(), _FINGER_ORD[m.group(1)] + ' Finger'+ # Words between markers, those after a marker first (a rig prefix sits+ # before the side, a Romance-language side after the part).+ bounds = [-1] + marks + [len(tokens)]+ segments = [tokens[bounds[k] + 1:bounds[k + 1]] for k in range(len(bounds) - 1)]+ labels = [_part_label(segment) for segment in segments[1:] + segments[:1]]+ labels = [label for label in labels if label and label != 'Bone']+ if not labels:+ return '', ''+ if labels[0] == 'End' and len(labels) > 1:+ # 'ankle_left_tip': the end marker after the side ends the part before it+ return sides.pop(), f'{labels[1]} End'+ return sides.pop(), labels[0]++def strip_trailing_decorations(name):# type: (str) -> Tuple[str, str]"""Peel stacked trailing ``[._]``-separated tokens off *name*.@@ -329,7 +437,14 @@ def clean_standard_name(raw):if not name:return raw.strip("_")+ code_side, label = extract_leg_code(name)+ if code_side:+ return with_side(code_side, label)side, name = extract_side_prefix(name)+ if not side:+ inner_side, label = extract_inner_side(name)+ if inner_side:+ return with_side(inner_side, label)# Numeric finger codes, before the chain-index strip eats the code digit:# CMU segments ('Finger1Metacarpal', 1-based) and 3ds Max Biped codes@@ -346,6 +461,9 @@ def clean_standard_name(raw):if m.group(2):label += " End"return with_side(side, label)+ m = _DIGIT_CODE_RE.match(name)+ if m:+ return with_side(side, _FINGER_ORD[m.group(1)] + " Finger")# Strip a trailing chain index: "Thigh_4" -> "Thigh", "Spine02" -> "Spine".m = re.match(r'^(.+?)_?(\d+)$', name)@@ -409,13 +527,87 @@ def clean_standard_name(raw):_MIXAMORIG_RE = re.compile(r'^mixamorig\d*[:_]', re.IGNORECASE)-def clean_joint_name(raw, animal):+def name_words(raw):+ """Words of a raw joint name for the learned vocabulary: namespace,+ Biped prefix, counters and side markers dropped, lower-cased+ ('mixamorig:LeftUpLeg_012' -> up, leg; 'JNT_Muslo_Der_01' -> jnt, muslo)."""+ name = _PAREN_DECOR_RE.sub('', raw).strip() or raw+ name = _FBX_ESCAPE_RE.sub(lambda m: chr(int(m.group(1))), name)+ name = re.sub(r'^.*:', '', name)+ name = _BIP_PREFIX_RE.sub('', name)+ _, name = strip_trailing_decorations(name)+ words = [w.lower() for w in _WORD_RE.findall(name) if not w.isdigit()]+ return [w for w in words if w not in SIDE_WORDS and w not in ('l', 'r')]+++def name_keys(raw):+ """Lookup keys of a raw name, longest first: its words and every shorter+ tail of them, so a rig-specific prefix ('doberman_ref_haunch') still+ reaches a learned tail ('haunch')."""+ words = name_words(raw)+ return ['_'.join(words[i:]) for i in range(len(words))]+++def _learned_vocab(_cache=[]):+ """``{name_key: part label}`` learned from reviewed annotations+ (``learned_vocab.json``, written by ``learn_vocab.py``); empty if absent."""+ if not _cache:+ path = os.path.join(os.path.dirname(__file__), 'learned_vocab.json')+ keys = {}+ if os.path.isfile(path):+ with open(path) as f:+ keys = json.load(f).get('keys', {})+ _cache.append(keys)+ return _cache[0]+++def _name_side(raw):+ bare = _NAMESPACE_RE.sub('', raw) or raw+ side, _ = extract_side_prefix(_BIP_PREFIX_RE.sub('', bare))+ return side or extract_inner_side(bare)[0]+++def clean_joint_name(raw, animal, refine=True):+ """Clean one joint name: the rules below, then (*refine*) the label+ refinements the reviewed annotations support:++ - a placeholder or non-canonical label takes the learned vocabulary's part+ for the name (``learned_vocab.json``), keeping the name's side;+ - a placeholder keeps the side its name marks ('1.L' -> 'Left Bone');+ - a name ending in ``END`` / ``_end`` / ``Nub`` is an end bone+ ('LeftToesEND' -> 'Left Toe End').++ ``refine=False`` is the label set the face resolver uses+ (``rig_preprocess.annotate.rule_annotation``).+ """+ out = _clean_joint_name_rules(raw, animal)+ if not refine or not raw or not raw.strip():+ return out+ cleaned = post_process(out)+ if cleaned == 'Bone' or not is_canonical_label(cleaned):+ vocab = _learned_vocab()+ label = next((vocab[k] for k in name_keys(raw) if k in vocab), None)+ if label:+ side = re.match(r'^(Left|Right)\b', cleaned)+ out = with_side(side.group(1) if side else _name_side(raw), label)+ cleaned = post_process(out)+ if cleaned == 'Bone':+ side, _ = strip_trailing_decorations(_PAREN_DECOR_RE.sub('', raw).strip() or raw)+ return with_side(side, 'Bone')+ if (_END_SUFFIX_RE.search(re.sub(r'[._]\d+$', '', raw)) and not cleaned.endswith(' End')+ and not re.match(r'^_?\d+$', cleaned)):+ return out + ' End'+ return out+++def _clean_joint_name_rules(raw, animal):"""Main dispatcher for cleaning a single joint name."""if not raw or not raw.strip():return raw# Editor bookkeeping suffixes: 'Bone001(mirrored)' -> 'Bone001'raw = _PAREN_DECOR_RE.sub("", raw).strip() or raw+ raw = _FBX_ESCAPE_RE.sub(lambda m: chr(int(m.group(1))), raw)# Generic "BoneNN" placeholders -> "Bone" (dot forms via clean_standard_name)if re.match(r'^Bone\d+$', raw):@@ -462,6 +654,112 @@ def clean_joint_name(raw, animal):return clean_standard_name(raw)+def limb_context(labels, parents):+ """Resolve the generic limb words a rig's hierarchy disambiguates (Mixamo+ convention, which the reviewed labels follow): a ``Leg`` whose parent is+ a ``Thigh`` is the shin; an ``Arm`` with a ``Forearm`` child is the upper+ arm. Returns a new list."""+ def part(label):+ return re.sub(r'^(Left|Right) ', '', label)++ def side(label):+ m = re.match(r'^(Left|Right) ', label)+ return m.group(1) if m else ''++ out = list(labels)+ for j, label in enumerate(labels):+ p = int(parents[j])+ if part(label) == 'Leg' and p >= 0 and part(labels[p]) == 'Thigh':+ out[j] = with_side(side(label), 'Shin')+ elif part(label) == 'Arm' and any(int(parents[k]) == j and part(labels[k]) == 'Forearm'+ for k in range(len(labels))):+ out[j] = with_side(side(label), 'Upper Arm')+ return quadruped_leg_chains(out, parents)+++# Segment labels of a quadruped leg chain, root to tip, in the Truebones+# convention the reviewed labels follow: hind legs by chain length, front+# legs after an optional shoulder (a paw tip is a 'Thumb Finger').+_HIND_CHAINS = {1: ['Thigh'], 2: ['Thigh', 'Shin'], 3: ['Thigh', 'Shin', 'Foot'],+ 4: ['Thigh', 'Shin', 'Fetlock', 'Foot']}+_HIND_LONG = ['Thigh', 'Shin', 'Fetlock', 'Foot', 'Toe']+_FRONT_CHAIN = ['Upper Arm', 'Forearm', 'Hand', 'Thumb Finger']+_LEG_SEGMENT_RE = re.compile(r'^(Left|Right) (Front|Back|Hind|Rear) Leg\b(.*)$')+++def _chain_labels(template, n):+ return [template[min(i, len(template) - 1)] for i in range(n)]+++def quadruped_leg_chains(labels, parents):+ """Relabel leg chains named by position rather than by part+ (``Back_Leg_Upper_L`` -> 'Left Back Leg Upper'): every non-canonical+ '<Side> <Front|Back> Leg ...' joint, the joints of its chain (single+ same-side children below, a same-side Shoulder / Hips-like joint above)+ get the segment labels of their position in the chain, except joints+ that already carry a canonical part label. Chains with no such joint are+ left alone. Returns a new list."""+ parents = [int(p) for p in parents]+ children = {}+ for j, p in enumerate(parents):+ children.setdefault(p, []).append(j)+ out = list(labels)+ done = set()+ for j, label in enumerate(labels):+ m = _LEG_SEGMENT_RE.match(label)+ if j in done or not m or is_canonical_label(label):+ continue+ side, front = m.group(1), m.group(2) == 'Front'+ top = j # climb to the chain's first leg-segment joint+ while parents[top] >= 0:+ pm = _LEG_SEGMENT_RE.match(labels[parents[top]])+ if not (pm and pm.group(1) == side and (pm.group(2) == 'Front') == front):+ break+ top = parents[top]+ p = parents[top]+ if front and p >= 0 and labels[p] in (f'{side} Shoulder', f'{side} Clavicle'):+ top = p+ chain = [top]+ while True:+ kids = [k for k in children.get(chain[-1], [])+ if labels[k].startswith(side + ' ')]+ if len(kids) != 1:+ break+ chain.append(kids[0])+ done.update(chain)+ ends = [labels[k].endswith(' End') for k in chain]+ core = [k for k, e in zip(chain, ends) if not e]+ if not core:+ continue+ if front:+ # A shoulder-like first joint, or one more joint than the arm+ # template names before the paw tip, is the shoulder.+ shoulder = (any(w in labels[core[0]] for w in ('Shoulder', 'Clavicle', 'Scapula'))+ or len(core) > len(_FRONT_CHAIN))+ names = (['Shoulder'] if shoulder else []) + _chain_labels(+ _FRONT_CHAIN, len(core) - shoulder)+ else:+ names = _HIND_CHAINS.get(len(core)) or _chain_labels(_HIND_LONG, len(core))+ last = None+ for k, e in zip(chain, ends):+ if e:+ if last and _replaceable(labels[k][:-len(' End')]):+ out[k] = with_side(side, f'{last} End')+ else:+ last = names.pop(0)+ if _replaceable(labels[k]):+ out[k] = with_side(side, last)+ else:+ last = re.sub(r'^(Left|Right) ', '', labels[k])+ return out+++def _replaceable(label):+ """A chain joint's label the position label replaces: a positional leg+ name or any non-canonical one; a canonical part label is kept."""+ return bool(_LEG_SEGMENT_RE.match(label)) or not is_canonical_label(label)++# ---------------------------------------------------------------------------# Post-processing# ---------------------------------------------------------------------------diff --git a/data_process/joint_annotation/skeleton_parse.py b/data_process/joint_annotation/skeleton_parse.pynew file mode 100644index 0000000..0b3f5f6--- /dev/null+++ b/data_process/joint_annotation/skeleton_parse.py@@ -0,0 +1,311 @@+"""Body-part labels for unnamed joints from the rest-pose geometry (bpy-free).++``parse_labels(parents, rest)`` reads a skeleton the way the reviewed labels+of numerically named rigs ('Bone.012', 'joint5') were made: from where the+joints are. It returns a label per joint, or '' where the geometry does not+say ('Bone' stays). Y is up; the rest pose is in the export frame.++1. Symmetry. The lateral axis is X or Z, whichever mirrors more joints onto+ joints; joints on the mirror plane are the midline, the others pair up.+2. Limbs. An off-midline joint with a midline parent starts a limb; its main+ path follows the largest child subtree. A limb whose lowest joint is near+ the floor is a leg; others hanging off the trunk are arms (or, on the+ head, ears); a joint with five or more limbs of one kind is left alone+ (tentacles, spider legs) unless they hang below it (tentacles).+3. Trunk. The midline joint the legs hang from is the hips; the midline path+ from it to the highest midline joint is spine, then neck from the arm+ attachment up, then head; a midline chain leaving the hips that does not+ lead to the head is the tail.+4. Facing. Forward along the non-lateral horizontal axis is where the head+ is when it sits well ahead of the hips, else the majority of where the+ toes point, where the face sticks out of the head and away from the tail;+ that vote is used only for a rig lying sideways (lateral axis Z): with+ the lateral axis X the asset faces +Z, as exports do. Left is up x+ forward.+5. Segments. Legs: Thigh, Shin, Foot, Toe; arms back from the hand (the+ first joint with several children, else the chain's fourth): Hand,+ Forearm, Upper Arm, Shoulder; the hand's chains are fingers, thumb first+ (the one set apart from the others), then index to pinky by distance+ from the thumb; a foot's chains are toes. A horizontal body (head ahead+ of the hips more than above them) with two leg pairs has arm chains for+ front legs (Upper Arm, Forearm, Hand, Finger, after a Shoulder on a long+ chain); with one leg pair, its other limb pair is a pair of wings.+"""++import numpy as np++MIRROR_TOL = 0.03 # of the rig's extent+MIDLINE_TOL = 0.02+FLOOR_TOL = 0.1+FINGERS = ['Thumb Finger', 'Index Finger', 'Middle Finger', 'Ring Finger', 'Pinky Finger']+++def _children(parents):+ kids = [[] for _ in parents]+ for j, p in enumerate(parents):+ if p >= 0:+ kids[p].append(j)+ return kids+++def _subtree(kids, j):+ out, stack = [], [j]+ while stack:+ k = stack.pop()+ out.append(k)+ stack.extend(kids[k])+ return out+++def _main_path(kids, sizes, j):+ path = [j]+ while kids[path[-1]]:+ path.append(max(kids[path[-1]], key=lambda k: sizes[k]))+ return path+++def _mirror(P, axis, center, ext):+ """Joint -> mirrored joint along *axis* (or -1), and the matched count."""+ Q = P.copy()+ Q[:, axis] = 2 * center - Q[:, axis]+ d = np.linalg.norm(P[:, None, :] - Q[None, :, :], axis=-1)+ m = d.argmin(axis=1)+ ok = d[np.arange(len(P)), m] <= MIRROR_TOL * ext+ mirror = np.where(ok, m, -1)+ off = np.abs(P[:, axis] - center) > MIDLINE_TOL * ext+ return mirror, int((ok & off).sum())+++def parse_labels(parents, rest):+ parents = [int(p) for p in parents]+ P = np.asarray(rest, dtype=float)+ J = len(parents)+ out = [''] * J+ if J < 3:+ return out+ ext = float(np.ptp(P, axis=0).max()) or 1.0+ kids = _children(parents)+ sizes = [len(_subtree(kids, j)) for j in range(J)]++ # 1. symmetry+ best = None+ for axis in (0, 2):+ center = float(np.median(P[:, axis]))+ mirror, n = _mirror(P, axis, center, ext)+ if best is None or n > best[2]:+ best = (axis, center, n, mirror)+ lat, center, n_mirrored, mirror = best+ if n_mirrored < 2:+ return out+ fwd_axis = 2 if lat == 0 else 0+ off = P[:, lat] - center+ midline = np.abs(off) <= MIDLINE_TOL * ext+ floor = float(P[:, 1].min())++ # 2. limbs+ limbs = []+ for j in range(J):+ p = parents[j]+ if midline[j] or p < 0 or not midline[p]:+ continue+ sub = _subtree(kids, j)+ path = _main_path(kids, sizes, j)+ low = float(P[sub, 1].min())+ limbs.append({'root': j, 'attach': p, 'path': path, 'sub': sub,+ 'floor': (low - floor) <= FLOOR_TOL * ext,+ 'top': float(P[sub, 1].max())})+ if not limbs:+ return out+ by_attach = {}+ for limb in limbs:+ by_attach.setdefault(limb['attach'], []).append(limb)++ # 3. trunk+ legs = [l for l in limbs if l['floor']]+ if legs:+ hips = max({l['attach'] for l in legs},+ key=lambda a: sum(l['attach'] == a for l in legs))+ else:+ hips = 0 if midline[0] else None+ mid_nodes = [j for j in range(J) if midline[j]]+ head = max(mid_nodes, key=lambda j: P[j, 1]) if mid_nodes else None+ trunk = []+ if hips is not None and head is not None:+ # head's midline ancestry back to hips (or to the root)+ a, chain = head, []+ while a >= 0 and midline[a]:+ chain.append(a)+ if a == hips:+ break+ a = parents[a]+ if chain and chain[-1] == hips:+ trunk = chain[::-1]+ arms = [l for l in limbs if not l['floor'] and l['attach'] in trunk and l['attach'] != hips+ and l['attach'] != head]+ ears = [l for l in limbs if not l['floor'] and head is not None and l['attach'] == head]++ # crowds of limbs from one joint: leave them unless they hang (tentacles)+ for a, group in by_attach.items():+ if len(group) >= 5:+ hang = all(P[l['path'][-1], 1] < P[a, 1] for l in group)+ for l in group:+ l['crowd'] = True+ if hang and not l['floor']:+ for k in l['sub']:+ out[k] = 'Tentacle'+ legs = [l for l in legs if not l.get('crowd')]+ arms = [l for l in arms if not l.get('crowd')]+ ears = [l for l in ears if not l.get('crowd')]++ # 4. facing+ votes = []+ for l in legs:+ if len(l['path']) >= 3:+ a, b = P[l['path'][-2]], P[l['path'][-1]]+ v = b[fwd_axis] - a[fwd_axis]+ if abs(v) > 0.03 * ext:+ votes.append(np.sign(v))+ if head is not None:+ front = [k for k in kids[head] if midline[k]]+ for k in front:+ v = P[k, fwd_axis] - P[head, fwd_axis]+ if abs(v) > 0.03 * ext:+ votes.append(np.sign(v))+ tails = []+ if hips is not None:+ for k in kids[hips]:+ if midline[k] and k not in trunk:+ tails.append(k)+ v = P[_main_path(kids, sizes, k)[-1], fwd_axis] - P[hips, fwd_axis]+ if abs(v) > 0.05 * ext:+ votes.append(-np.sign(v))+ if hips is not None and head is not None:+ # a head well ahead of the hips (a horizontal body) decides+ v = P[head, fwd_axis] - P[hips, fwd_axis]+ if abs(v) > 0.15 * ext:+ votes = [np.sign(v)]+ # Exported assets face +Z unless they lie sideways: with the lateral+ # axis X the cues are not trusted over that (they turn more right+ # facings around than they fix); with the lateral axis Z they pick +X+ # or -X.+ sign = 1.0 if fwd_axis == 2 or sum(votes) >= 0 else -1.0+ forward = np.zeros(3)+ forward[fwd_axis] = sign+ left = np.cross([0.0, 1.0, 0.0], forward)++ def side(j):+ return 'Left' if (P[j] - P[hips if hips is not None else 0]) @ left > 0 else 'Right'++ # 5. labels+ if hips is not None:+ out[hips] = 'Hips'+ a = parents[hips]+ while a >= 0:+ out[a] = 'Root'+ a = parents[a]+ if trunk:+ arm_attach = [i for i, j in enumerate(trunk) if any(l['attach'] == j for l in arms)]+ top_arm = max(arm_attach) if arm_attach else None+ for i, j in enumerate(trunk[1:], start=1):+ if j == head:+ out[j] = 'Head'+ elif top_arm is not None and i > top_arm:+ out[j] = 'Neck'+ else:+ out[j] = 'Spine'+ if head is not None:+ for k in _subtree(kids, head)[1:]:+ if midline[k]:+ out[k] = 'Head'+ for t in tails:+ for k in _subtree(kids, t):+ if midline[k]:+ out[k] = 'Tail'+ leg_names = {1: ['Thigh'], 2: ['Thigh', 'Shin'], 3: ['Thigh', 'Shin', 'Foot'],+ 4: ['Thigh', 'Shin', 'Foot', 'Toe']}+ # a body whose head is ahead of the hips more than above them is horizontal:+ # its front legs are arm chains, its non-floor limb pair wings+ horizontal = False+ if hips is not None and head is not None:+ d = P[head] - P[hips]+ horizontal = abs(d[fwd_axis]) > abs(d[1])+ front = []+ if len({l['attach'] for l in legs}) >= 2 and head is not None:+ attach_pos = {a: (P[a] - P[hips]) @ forward for a in {l['attach'] for l in legs}}+ front_attach = max(attach_pos, key=attach_pos.get)+ if front_attach != hips:+ front = [l for l in legs if l['attach'] == front_attach]+ for l in front:+ path = l['path']+ s = side(l['root'])+ names = ['Upper Arm', 'Forearm', 'Hand', 'Finger']+ if len(path) > 4:+ names = ['Shoulder'] + names+ for i, k in enumerate(path):+ out[k] = f'{s} {names[min(i, len(names) - 1)]}'+ for k in l['sub']:+ if not out[k]:+ out[k] = f'{s} Finger'+ legs = [l for l in legs if l not in front]+ if horizontal and not front:+ for l in arms:+ s = side(l['root'])+ for k in l['sub']:+ out[k] = f'{s} Wing'+ arms = []+ for l in legs:+ path = l['path']+ names = leg_names.get(len(path)) or (['Thigh', 'Shin', 'Foot'] + ['Toe'] * (len(path) - 3))+ s = side(l['root'])+ for k, name in zip(path, names):+ out[k] = f'{s} {name}'+ foot = next((k for k, n in zip(path, names) if n == 'Foot'), None)+ for k in l['sub']:+ if not out[k]:+ out[k] = f'{s} Toe' if foot is not None else ''+ for l in arms:+ path = l['path']+ s = side(l['root'])+ # the hand: the first branching joint past the upper arm+ hand_i = next((i for i, k in enumerate(path) if i >= 2 and len(kids[k]) >= 2), None)+ if hand_i is None:+ hand_i = min(len(path) - 1, 3 if len(path) >= 4 else len(path) - 1)+ seg = ['Hand', 'Forearm', 'Upper Arm', 'Shoulder']+ for i in range(hand_i, -1, -1):+ out[path[i]] = f'{s} {seg[min(hand_i - i, 3)]}'+ hand = path[hand_i]+ fingers = [k for k in kids[hand]]+ _label_fingers(out, kids, P, hand, fingers, s)+ if len(path) > hand_i + 1 and len(fingers) == 1:+ for k in path[hand_i + 1:]:+ out[k] = f'{s} Finger'+ for l in ears:+ s = side(l['root'])+ if l['top'] > P[l['attach'], 1]:+ for k in l['sub']:+ out[k] = f'{s} Ear'+ return out+++def _label_fingers(out, kids, P, hand, fingers, s):+ chains = {f: _subtree(kids, f) for f in fingers}+ if len(fingers) == 1:+ for k in chains[fingers[0]]:+ out[k] = f'{s} Finger'+ return+ base = np.array([P[f] for f in fingers])+ if len(fingers) >= 3:+ # thumb: the base farthest from the others' mean+ dists = [np.linalg.norm(base[i] - np.delete(base, i, 0).mean(0)) for i in range(len(fingers))]+ thumb = int(np.argmax(dists))+ order = sorted(range(len(fingers)), key=lambda i: np.linalg.norm(base[i] - base[thumb]))+ names = FINGERS if len(fingers) == 5 else (['Thumb Finger'] + FINGERS[1:len(fingers)])+ if len(fingers) > 5:+ names = ['Finger'] * len(fingers)+ for rank, i in enumerate(order):+ for k in chains[fingers[i]]:+ out[k] = f'{s} {names[min(rank, len(names) - 1)]}'+ else:+ for f in fingers:+ for k in chains[f]:+ out[k] = f'{s} Finger'diff --git a/data_process/joint_annotation/structure_rule.py b/data_process/joint_annotation/structure_rule.pynew file mode 100644index 0000000..a56be2c--- /dev/null+++ b/data_process/joint_annotation/structure_rule.py@@ -0,0 +1,430 @@+"""Rig-level refinements of rule joint labels: what one name cannot tell but+the rig's other names and its hierarchy can.++``rig_context(labels, raw, parents)`` runs after the per-name cleaner+(``names_clean_rule.clean_joint_name``) and ``limb_context``; every step+follows a convention of the reviewed labels in ``dataset/export/*/+clean_joint_names.json`` and none is keyed by rig name:++- labels that are a word of the asset's name rather than a part (a word the+ cleaner could not map, repeated over much of the rig) become placeholders;+- a known part behind or before unknown words keeps the part ('Rex Toe' ->+ 'Toe', 'Index Finger Bn' -> 'Index Finger'; 'Thigh Rear' -> 'Hind Thigh');+- joint words map to the part the reviewed labels use for the bone+ ('Collar' -> 'Shoulder', a sided 'Point' -> 'Index Finger', single foreign+ words through ``vocab.FOREIGN_PARTS``);+- front / hind markers in a leg name qualify its Leg / Toe / Foot / Paw;+- numbered fingers follow the rig's numbering: 0-based (3ds Max Biped,+ 'Finger0' = thumb) when any finger is numbered 0, else 1-based from the+ thumb when five are numbered, else from the index finger;+- positional limb names along a chain get segment labels ('Arm1', 'Arm2' ->+ Upper Arm, Forearm; 'Leg1', 'Leg2' -> Thigh, Shin), except the legs of a+ rig with more than four of them;+- finger labels below a foot are toes; 'Middle' fingers of a rig with legs+ and no hands are middle legs;+- a side marked in a way the per-name cleaner skips ('lIndex3', 'lefthand',+ 'Lf_Wrist', 'PinkR3') is kept when the rig then has the part on both sides.++``fill_unnamed(labels, parents, rest)`` gives a rig whose joints are mostly+placeholders the geometric labels of ``skeleton_parse``.+"""++import re+from collections import Counter++from data_process.joint_annotation.skeleton_parse import parse_labels+from data_process.joint_annotation.vocab import FOREIGN_PARTS, is_canonical_label++_SIDE_RE = re.compile(r'^(Left|Right) ')+++def split_side(label):+ m = _SIDE_RE.match(label)+ return (m.group(1), label[m.end():]) if m else ('', label)+++def join_side(side, part):+ return f'{side} {part}' if side else part+++def _children(parents):+ kids = {}+ for j, p in enumerate(parents):+ kids.setdefault(int(p), []).append(j)+ return kids+++# ---------------------------------------------------------------------------+# Label-level fixes+# ---------------------------------------------------------------------------++# Joint words whose reviewed label is the bone they start.+LABEL_MAP = {+ 'Collar': 'Shoulder',+ 'Collar Bone': 'Shoulder',+ 'Coler Bone': 'Shoulder',+ 'Clavical': 'Shoulder',+ 'Back Leg': 'Hind Leg',+ 'Side Leg': 'Middle Leg',+ 'Patte Arr': 'Hind Leg',+ 'Patte Av': 'Front Leg',+ 'Leg Upper': 'Thigh',+ 'Leg Lower': 'Shin',+ 'Leg Trochanter': 'Thigh',+ 'Leg Twist': 'Shin Twist',+ 'Ankle Bone': 'Toe',+ 'Talons': 'Claw',+ 'Talon': 'Claw',+ 'Braid': 'Hair',+ 'Proboscis': 'Appendage',+ 'Mustache': 'Whisker',+ 'Moustache': 'Whisker',+ 'Lit': 'Pinky Finger',+ 'Fins': 'Fin',+ 'Ornamentalhorn': 'Horn',+ 'Head Top': 'Head',+ 'Head Vertex': 'Head',+ 'Labrum': 'Lip',+}+# Unsided joint words.+UNSIDED_MAP = {+ 'Back': 'Spine',+ 'Point': 'Bone', # a 3ds Max helper+}+# Sided joint words.+SIDED_MAP = {+ 'Point': 'Index Finger',+}+++def _end(part):+ return part.endswith(' End'), (part[:-4] if part.endswith(' End') else part)+++def map_labels(labels):+ out = list(labels)+ for j, label in enumerate(labels):+ side, part = split_side(label)+ end, base = _end(part)+ base = base.rstrip('-_. ')+ new = LABEL_MAP.get(base)+ if new is None and ' ' not in base and not is_canonical_label(base):+ new = FOREIGN_PARTS.get(base.lower())+ if new is None and not side:+ new = UNSIDED_MAP.get(base)+ if new is None and side:+ new = SIDED_MAP.get(base)+ if new:+ out[j] = join_side(side, new + (' End' if end else ''))+ return out+++def canonical_tail(labels):+ """'Rex Toe' -> 'Toe', 'Index Finger Bn' -> 'Index Finger': a non-canonical+ label whose last words (else first words) form a canonical part keeps+ that part; a trailing position word moves in front ('Thigh Rear' ->+ 'Hind Thigh')."""+ out = list(labels)+ for j, label in enumerate(labels):+ side, part = split_side(label)+ if part == 'Bone' or is_canonical_label(part) or part in LABEL_MAP:+ continue+ end, base = _end(part)+ m = re.match(r'^(.+) (Front|Rear|Hind|Back)$', base)+ if m:+ q = 'Front' if m.group(2) == 'Front' else 'Hind'+ if is_canonical_label(f'{q} {m.group(1)}'):+ out[j] = join_side(side, f'{q} {m.group(1)}' + (' End' if end else ''))+ continue+ words = base.split()+ spans = [words[k:] for k in range(1, len(words))] + \+ [words[:k] for k in range(len(words) - 1, 0, -1)]+ for span in spans:+ cand = ' '.join(span)+ if cand != 'Bone' and is_canonical_label(cand):+ out[j] = join_side(side, cand + (' End' if end else ''))+ break+ return out+++def asset_word_placeholders(labels, min_share=0.3, min_joints=4):+ """A non-canonical label carried by much of the rig names the asset, not+ a part ('Frog', 'Racoon'): placeholder."""+ parts = Counter(_end(split_side(l)[1])[1] for l in labels)+ common = {p for p, n in parts.items()+ if n >= min_joints and n >= min_share * len(labels)+ and p != 'Bone' and not is_canonical_label(p)}+ if not common:+ return list(labels)+ out = []+ for label in labels:+ side, part = split_side(label)+ end, base = _end(part)+ out.append(join_side(side, 'Bone') if base in common else label)+ return out+++# Front / hind markers in a raw leg name: words ('FrontLeg', 'frount toe',+# 'HindLeg', 'Rear'), a capital F/B/H glued to the part ('FLeg01', 'BLeg03'),+# or a side+position code ('b_LF_leg', 'LB_Leg1').+_FRONT_RE = re.compile(r'(?i:front|frnt|frount|fore)|(?<![A-Za-z])F(?=Leg|Foot|Toe|Paw)'+ r'|(?:^|_)[LR]F(?=_|$)|(?:^|_)F[LR](?=_|$)')+_HIND_RE = re.compile(r'(?i:hind|rear|back)|(?<![A-Za-z])[BH](?=Leg|Foot|Toe|Paw)'+ r'|(?:^|_)[LR][BH](?=_|$)|(?:^|_)[BH][LR](?=_|$)')+_QUALIFIABLE = {'Leg', 'Toe', 'Foot', 'Paw'}+++def leg_qualifiers(labels, raw):+ """'Leg' / 'Toe' / 'Foot' / 'Paw' of a name marked front or hind take the+ qualifier ('FLeg01' -> 'Front Leg', 'BLeg01' -> 'Hind Leg',+ 'frount toe.L' -> 'Front Toe'); hind legs are 'Hind', other hind parts+ 'Back', as the reviewed labels write them."""+ out = list(labels)+ for j, label in enumerate(labels):+ side, part = split_side(label)+ end, base = _end(part)+ if base not in _QUALIFIABLE:+ continue+ name = re.sub(r'^.*:', '', raw[j])+ front, hind = bool(_FRONT_RE.search(name)), bool(_HIND_RE.search(name))+ if front == hind:+ continue+ q = 'Front' if front else ('Hind' if base == 'Leg' else 'Back')+ out[j] = join_side(side, f'{q} {base}' + (' End' if end else ''))+ return out+++# ---------------------------------------------------------------------------+# Finger numbering+# ---------------------------------------------------------------------------++# 'Finger1' / 'Finger12' (finger 1, segment 2) or 'finger_2' / 'finger_2_1';+# behind a separator the whole number is the finger ('Finger_07' is a chain+# counter, not finger 0).+_FINGER_NUM_RE = re.compile(r'finger(?:(\d)|[ _-](\d+))(?!\d*[A-Za-z]{2})', re.IGNORECASE)+_FINGERS_1 = ['Thumb', 'Index', 'Middle', 'Ring', 'Pinky']+++def finger_numbering(labels, raw):+ """Generic 'Finger' labels of numbered finger names get the finger the+ rig's numbering gives them."""+ nums = {}+ for j, (label, name) in enumerate(zip(labels, raw)):+ _, part = split_side(label)+ if _end(part)[1] != 'Finger':+ continue+ m = _FINGER_NUM_RE.search(name)+ if m:+ n = int(m.group(1) if m.group(1) is not None else m.group(2))+ if n <= 5:+ nums[j] = n+ if not nums:+ return list(labels)+ digits = set(nums.values())+ has_thumb = any(_end(split_side(l)[1])[1] == 'Thumb Finger' for l in labels)+ if 0 in digits:+ base = 0 # 0 = thumb+ elif 5 in digits or (not has_thumb and max(digits) >= 5):+ base = 1 # 1 = thumb+ else:+ base = 2 # 1 = index (no thumb among the numbered fingers)+ out = list(labels)+ for j, n in nums.items():+ k = n - 1 if base == 1 else n # index into _FINGERS_1 (0 = thumb)+ if 0 <= k < 5:+ side, part = split_side(labels[j])+ end, _ = _end(part)+ out[j] = join_side(side, _FINGERS_1[k] + ' Finger' + (' End' if end else ''))+ return out+++# ---------------------------------------------------------------------------+# Hierarchy+# ---------------------------------------------------------------------------++_CHAIN_SEGMENTS = {+ 'Arm': ['Upper Arm', 'Forearm'],+ 'Leg': ['Thigh', 'Shin'],+}+++def limb_chains(labels, parents, max_len=2, max_leg_runs=4):+ """Runs of one same-side positional limb word down a chain ('Arm1',+ 'Arm2'): segment labels by position. A run longer than *max_len* (a+ tentacle-like limb) keeps the word, except that 'Arm' becomes 'Upper Arm'."""+ parents = [int(p) for p in parents]+ kids = _children(parents)+ starts = [j for j, label in enumerate(labels)+ if split_side(label)[1] in _CHAIN_SEGMENTS+ and not (parents[j] >= 0 and labels[parents[j]] == label)]+ many_legs = sum(split_side(labels[j])[1] == 'Leg' for j in starts) > max_leg_runs+ out = list(labels)+ for j in starts:+ label = labels[j]+ side, part = split_side(label)+ if part == 'Leg' and many_legs:+ continue # a spider or insect keeps its legs+ p = parents[j]+ run = [j]+ while True:+ nxt = [k for k in kids.get(run[-1], []) if labels[k] == label]+ if len(nxt) != 1:+ break+ run.append(nxt[0])+ seg = _CHAIN_SEGMENTS[part]+ parent_part = split_side(labels[p])[1] if p >= 0 else ''+ if len(run) <= max_len:+ # a run that continues a chain already holding its first segment+ start = 1 if parent_part in (seg[0], 'Hip' if part == 'Leg' else '') else 0+ for i, k in enumerate(run):+ out[k] = join_side(side, seg[min(start + i, len(seg) - 1)])+ elif part == 'Arm':+ for k in run:+ out[k] = join_side(side, 'Upper Arm')+ return out+++_HAND_PARTS = {'Hand', 'Wrist', 'Palm', 'Forearm', 'Upper Arm', 'Elbow'}+_FOOT_PARTS = {'Foot', 'Ankle', 'Toe', 'Heel'}+_LEG_PARTS = {'Shin', 'Thigh', 'Knee'}+_FINGER_PARTS = {'Thumb Finger', 'Index Finger', 'Middle Finger', 'Ring Finger',+ 'Pinky Finger', 'Finger'}+++def _limb_ancestor(labels, parents, j):+ """'hand' / 'foot' / '' for the nearest ancestor that tells a hand chain+ from a foot chain."""+ p = int(parents[j])+ while p >= 0:+ part = _end(split_side(labels[p])[1])[1]+ part = re.sub(r'^(Front|Back|Hind|Rear|Middle) ', '', part)+ if part in _HAND_PARTS:+ return 'hand'+ if part in _FOOT_PARTS:+ return 'foot'+ if part in _LEG_PARTS or part.endswith('Leg'):+ return 'leg'+ p = int(parents[p])+ return ''+++def hand_and_foot_digits(labels, parents):+ """Finger labels below a foot are toes."""+ out = list(labels)+ for j, label in enumerate(labels):+ side, part = split_side(label)+ end, base = _end(part)+ if base in _FINGER_PARTS and _limb_ancestor(labels, parents, j) == 'foot':+ out[j] = join_side(side, 'Toe' + (' End' if end else ''))+ return out+++def middle_legs(labels, parents):+ """'Middle Finger' joints of a rig with legged parts and no hand chain+ above them are the middle legs of an insect or spider."""+ has_legs = any(re.search(r'\b(Front|Back|Hind) Leg\b', l) for l in labels)+ if not has_legs:+ return list(labels)+ out = list(labels)+ for j, label in enumerate(labels):+ side, part = split_side(label)+ end, base = _end(part)+ if base == 'Middle Finger' and _limb_ancestor(labels, parents, j) != 'hand':+ out[j] = join_side(side, 'Middle Leg' + (' End' if end else ''))+ return out+++# ---------------------------------------------------------------------------+# Sides+# ---------------------------------------------------------------------------++# Parts that sit on the body's midline and never take a side.+CENTER_PARTS = {'Root', 'Hips', 'Pelvis', 'Spine', 'Chest', 'Body', 'Neck', 'Head', 'Jaw',+ 'Tail', 'Bone', 'Center', 'Abdomen', 'Ribcage', 'Waist', 'Upper Body',+ 'Lower Body', 'Torso', 'Mouth', 'Tongue', 'Nose', 'Chin', 'Base'}+_SIDE_TOKENS = {'l': 'Left', 'r': 'Right', 'lf': 'Left', 'rt': 'Right', 'lft': 'Left',+ 'rgt': 'Right', 'lt': 'Left', 'left': 'Left', 'right': 'Right'}+++def token_side(raw):+ """Side marked anywhere in a raw name in the less common ways the+ per-name cleaner skips ('lIndex3', 'lefthand', 'Lf_Wrist', 'PinkR3',+ 'finger middle 3 l', 'RrightEar'); '' when none or both sides."""+ name = re.sub(r'^.*:', '', raw).lstrip('-_ ')+ found = set()+ m = re.match(r'^([LR])(right|left)', name, re.IGNORECASE)+ if m:+ found.add(_SIDE_TOKENS[m.group(2).lower()])+ m = re.match(r'^(left|right)(?=[a-z])', name)+ if m:+ found.add(_SIDE_TOKENS[m.group(1)])+ for m in re.finditer(r'(?:^|[^A-Za-z])([lr])(?=[A-Z][a-z])', name):+ found.add(_SIDE_TOKENS[m.group(1)])+ m = re.search(r'[a-z]([LR])\d*$', name)+ if m:+ found.add(_SIDE_TOKENS[m.group(1).lower()])+ for tok in re.split(r'[^A-Za-z]+|(?<=[a-z])(?=[A-Z])', name):+ side = _SIDE_TOKENS.get(tok.lower())+ if side:+ found.add(side)+ return found.pop() if len(found) == 1 else ''+++def fill_sides(labels, raw):+ """Sides the per-name cleaner leaves out, from side markers anywhere in+ the raw name (``token_side``), for parts the rig then has on both sides.+ Unmarked joints stay unsided: the reviewed labels do not side them by+ position either."""+ cand = {}+ for j, label in enumerate(labels):+ side, part = split_side(label)+ if side or _end(part)[1] in CENTER_PARTS:+ continue+ t = token_side(raw[j])+ if t:+ cand[j] = join_side(t, part)+ trial = list(labels)+ for j, l in cand.items():+ trial[j] = l+ seen = {}+ for l in trial:+ side, part = split_side(l)+ if side:+ seen.setdefault(_end(part)[1], set()).add(side)+ out = list(labels)+ for j, l in cand.items():+ if len(seen.get(_end(split_side(l)[1])[1], ())) == 2:+ out[j] = l+ return out++def rig_context(labels, raw, parents, steps=None):+ """All rig-level refinements, in order. *steps* restricts them (names of+ the functions above) for evaluation."""+ pipeline = [+ ('asset_word_placeholders', lambda l: asset_word_placeholders(l)),+ ('canonical_tail', canonical_tail),+ ('map_labels', map_labels),+ ('leg_qualifiers', lambda l: leg_qualifiers(l, raw)),+ ('finger_numbering', lambda l: finger_numbering(l, raw)),+ ('limb_chains', lambda l: limb_chains(l, parents)),+ ('hand_and_foot_digits', lambda l: hand_and_foot_digits(l, parents)),+ ('middle_legs', lambda l: middle_legs(l, parents)),+ ('fill_sides', lambda l: fill_sides(l, raw)),+ ]+ out = list(labels)+ for name, fn in pipeline:+ if steps is None or name in steps:+ out = fn(out)+ return out+++def fill_unnamed(labels, parents, rest, min_share=0.5):+ """A rig whose joints are mostly placeholders ('Bone.012', 'joint5') takes+ the geometric labels of ``skeleton_parse.parse_labels`` for them. On a rig+ with names the placeholders stay: there the geometric guess loses more+ reviewed labels than it gains."""+ bone = [_end(split_side(l)[1])[1] == 'Bone' for l in labels]+ if not labels or sum(bone) <= min_share * len(labels):+ return list(labels)+ proposal = parse_labels(parents, rest)+ return [p if b and p else l for l, p, b in zip(labels, proposal, bone)]diff --git a/data_process/joint_annotation/vocab.py b/data_process/joint_annotation/vocab.pyindex b45880c..4080022 100644--- a/data_process/joint_annotation/vocab.py+++ b/data_process/joint_annotation/vocab.py@@ -156,6 +156,9 @@ CANONICAL = {"Controller": "", "Controler": "", "Ctrl": "", "Quick": "","Untitled": "", "Pasted": "","Bind": "", "Skeleton": "", "Reference": "", "Res": "", "Main": "",+ # URDF robots: every body is a '<part>_link', joints split per axis+ # ('left_hip_pitch_link' -> the part alone).+ "Link": "", "Pitch": "", "Roll": "", "Yaw": "",# UE-style per-finger segment words: chain position, not anatomy# ('IndexDistal', 'thumb_proximal_l' -> the finger label alone)."Metacarpal": "", "Proximal": "", "Intermediate": "", "Medial": "",@@ -180,6 +183,9 @@ STANDALONE_MAP = {"BN_P": "Belly","Hips": "Hips", "Spine": "Spine", "Head": "Head","Trajectory": "Root",+ # A joint named only 'base' is mostly the root; inside a name the word+ # is a modifier ('ToeBase').+ "Base": "Root", "base": "Root","RightArm": "Right Upper Arm", "RightForeArm": "Right Forearm","LeftArm": "Left Upper Arm", "LeftForeArm": "Left Forearm",# Standalone mixamo-style chain: bare "Leg" is the shin (see MIXAMO_MAP).@@ -404,20 +410,83 @@ SPIDER_MAP = {# ---------------------------------------------------------------------------# Canonical label format โ ONE format shared by truebones / mixamo / objaverse#-# label := [Left |Right ] <part> [ End]+# label := [Left |Right ] [<qualifier> ] <part> [ End]#-# where <part> is any base label the maps above emit (qualifiers like-# Front/Hind/Inner/Middle/Outer/Upper/Lower are already baked into those-# values: 'Front Leg', 'Upper Eyelid', ...) and ' End' marks a chain tip-# (Nub/Tip/*_End sources). Two sentinels sit outside the grammar:+# where <part> is any base label the maps above emit, plus PROMPT_PARTS and+# REVIEW_PARTS; <qualifier> is one of QUALIFIERS ('Right Front Toe',+# 'Back Hip', 'Inner Claw'), the composed form names_clean_llm's prompt+# allows; and ' End' marks a chain tip (Nub/Tip/*_End sources). Some+# qualified forms are parts in their own right ('Front Leg', 'Upper+# Eyelid'). Two sentinels sit outside the grammar:# - "Bone" placeholder for rig noise with no recognizable anatomy# - "_00", "12" numeric passthrough for rigs whose bones were never named# ---------------------------------------------------------------------------+# Non-anatomical parts that hand-reviewed joint labels use (a closed list:+# garments and carried items, plant parts, object parts). No raw-name rule emits them, so they are not in the maps above; they+# reach clean_joint_names.json only through patches/<ds>_joint_labels.json.+REVIEW_PARTS = (+ "Hair", "Cape", "Skirt", "Cloth", "Hat", "Scarf", "Belt", "Bag", "Strap",+ "Weapon", "Shield", "Prop",+ "Stem", "Leaf", "Petal", "Branch",+ "Base", "Lid", "Door", "Wheel", "Propeller", "Lever", "Chain", "Rope",+)++# Terms names_clean_llm's prompt lists in its CANONICAL VOCABULARY that no rule+# map emits; without them the LLM's own vocabulary would fail the check.+PROMPT_PARTS = (+ "Upper Body", "Waist", "Collar", "Paw", "Upper Jaw", "Lower Jaw", "Lower Lip",+ "Front Paw", "Back Paw", "Front Shoulder", "Back Hip", "Thigh Twist", "Shin Twist",+)++# Side words inside a raw name (whole tokens, lower case): English and the+# Spanish, Portuguese, French, German and Italian words rigs use.+SIDE_WORDS = {+ 'right': 'Right', 'der': 'Right', 'derecha': 'Right', 'derecho': 'Right',+ 'dcha': 'Right', 'direita': 'Right', 'direito': 'Right', 'droite': 'Right',+ 'droit': 'Right', 'rechts': 'Right', 'destra': 'Right', 'destro': 'Right',+ 'left': 'Left', 'izq': 'Left', 'izquierda': 'Left', 'izquierdo': 'Left',+ 'izda': 'Left', 'iz': 'Left', 'esquerda': 'Left', 'esquerdo': 'Left',+ 'gauche': 'Left', 'links': 'Left', 'sinistra': 'Left', 'sinistro': 'Left',+}++# Body parts in those languages (lower case, accents dropped) -> canonical part.+FOREIGN_PARTS = {+ # Spanish+ 'brazo': 'Upper Arm', 'antebrazo': 'Forearm', 'pierna': 'Leg', 'muslo': 'Thigh',+ 'pie': 'Foot', 'mano': 'Hand', 'cabeza': 'Head', 'cuello': 'Neck', 'hombro': 'Shoulder',+ 'codo': 'Elbow', 'rodilla': 'Knee', 'ala': 'Wing', 'alas': 'Wing', 'cola': 'Tail',+ 'dedo': 'Finger', 'tobillo': 'Ankle', 'cadera': 'Hips', 'pecho': 'Chest',+ # Portuguese+ 'braco': 'Upper Arm', 'perna': 'Leg', 'coxa': 'Thigh', 'mao': 'Hand',+ 'cabeca': 'Head', 'pescoco': 'Neck', 'ombro': 'Shoulder', 'cotovelo': 'Elbow',+ 'joelho': 'Knee', 'asa': 'Wing', 'cauda': 'Tail', 'quadril': 'Hips',+ # French+ 'bras': 'Upper Arm', 'avantbras': 'Forearm', 'jambe': 'Leg', 'cuisse': 'Thigh',+ 'pied': 'Foot', 'main': 'Hand', 'tete': 'Head', 'cou': 'Neck', 'epaule': 'Shoulder',+ 'coude': 'Elbow', 'genou': 'Knee', 'aile': 'Wing', 'queue': 'Tail', 'hanche': 'Hips',+ # German+ 'bein': 'Leg', 'oberschenkel': 'Thigh', 'fuss': 'Foot',+ 'kopf': 'Head', 'hals': 'Neck', 'schulter': 'Shoulder', 'knie': 'Knee',+ 'flugel': 'Wing', 'schwanz': 'Tail',+ # Italian+ 'braccio': 'Upper Arm', 'gamba': 'Leg', 'coscia': 'Thigh', 'piede': 'Foot',+ 'testa': 'Head', 'collo': 'Neck', 'spalla': 'Shoulder', 'gomito': 'Elbow',+ 'ginocchio': 'Knee', 'coda': 'Tail',+ # Slavic, Vietnamese and pinyin words seen in rigs+ 'hvost': 'Tail', 'khvost': 'Tail', 'uho': 'Ear', 'ucho': 'Ear', 'hobot': 'Appendage',+ 'canh': 'Wing', 'chibang': 'Wing', 'weiba': 'Tail', 'datui': 'Thigh', 'xiaotui': 'Shin',+ 'dabi': 'Upper Arm', 'xiaobi': 'Forearm',+}++# Position qualifiers allowed between the side and the part.+QUALIFIERS = ("Front", "Back", "Middle", "Rear", "Hind", "Inner", "Outer", "Upper", "Lower")++def canonical_parts():"""Base part labels (side stripped) that the vocabulary can emit."""import re as _re- parts = set()+ parts = set(REVIEW_PARTS) | set(PROMPT_PARTS)for mapping in (CANONICAL, MIXAMO_MAP, NPC_DIRECT, ELK_MAP, JT_MAP,SPIDER_MAP, STANDALONE_MAP, SABRECAT_MAP,PIRRANA_COMPOUNDS, JAPANESE_WORDS):@@ -436,4 +505,7 @@ def is_canonical_label(label, _parts_cache=[]):return Truebase = _re.sub(r'^(Left|Right)\s+', '', label)base = _re.sub(r'\s+End$', '', base)- return base in _parts_cache[0]+ if base in _parts_cache[0]:+ return True+ head, _, rest = base.partition(' ')+ return head in QUALIFIERS and rest in _parts_cache[0]diff --git a/data_process/mesh_animation/__init__.py b/data_process/mesh_animation/__init__.pyindex ab99b0b..fa93648 100644--- a/data_process/mesh_animation/__init__.py+++ b/data_process/mesh_animation/__init__.py@@ -8,11 +8,10 @@ Entry points (all run under Blender headless, ``blender -b -P <script> -- <args>- ``animate_mixamo`` โ batch wrapper pairing one character with a directory of NPZs- ``animate_lbs`` โ manual NumPy FK+LBS: deform a rigged asset's skinned meshdirectly with a motion NPZ (either flavor) and save the vertex animation-- ``preprocess_char`` โ run one rigged, animated asset through the real export +- feature pipeline in-process: writes export NPZs, motion-feature NPZs, the- model-side ``cond.npy``, and a canonical rest-pose GLB/FBX that feature NPZs- drive through ``animate_lbs`` cond-free+- one rigged asset โ ``cond.npy`` + canonical rest-pose GLB: ``data_process.rig_preprocess``-Shared plumbing lives in :mod:`.common`; rig primitives in+Shared plumbing lives in :mod:`.common` (processed-asset lookup, keyframe-and-+export core); the canonical rest-pose bake in :mod:`.canonical_rig` (also used+by ``feature_extraction/canonical_assets.py``); rig primitives in:mod:`data_process.utils.blender_rig`."""diff --git a/data_process/mesh_animation/animate_lbs.py b/data_process/mesh_animation/animate_lbs.pyindex 9093d2a..70255df 100644--- a/data_process/mesh_animation/animate_lbs.py+++ b/data_process/mesh_animation/animate_lbs.py@@ -22,8 +22,16 @@ Accepts both motion formats, auto-detected from the NPZ keys::mod:`animate_motion`;* WITHOUT them, the asset itself is taken as the canonical motionskeleton โ the cond-free path for assets baked by- :mod:`preprocess_char` (joint order = bone order, rest = the+ ``rig_preprocess`` (joint order = bone order, rest = theasset's rest, scale = 1).+ * A canonical asset (``canonical_joint_order`` property: from+ ``feature_extraction/canonical_assets.py`` or rig_preprocess) always+ takes the cond-free path, ``--dataset_type`` or not.++``--char_path`` may be omitted with ``--dataset_type``: a feature NPZ then+drives the canonical asset ``dataset/canonical_assets/<ds>/<name>.glb``, an+export NPZ the export asset ``dataset/export/<ds>/rigs/<name>.glb``; <name>+is the clip's object type, for Mixamo ``--character`` (default Michelle).Bones are matched to the asset by name. Asset bones missing from the motionkeep their rest local transform (they follow their parent, the ``'keep'``@@ -55,13 +63,19 @@ from loguru import loggersys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))-from Animation import transforms_local # noqa: E402+from Animation import transforms_global, transforms_local # noqa: E402from data_process.mesh_animation.common import ( # noqa: E402+ DEFAULT_MIXAMO_CHARACTER,+ canonical_joint_order,+ is_canonical_asset,+ is_feature_motion,load_character,npz_scalar,parse_blender_argv,quat_pos_to_mats,+ resolve_processed_asset,+ rotations_to_quats,)from data_process.utils.blender_rig import ( # noqa: E402compute_bone_keyframes,@@ -71,6 +85,7 @@ from data_process.utils.blender_rig import ( # noqa: E402set_scene_timing,update_scene,)+from data_process.utils.skeleton import motion_axis_alignment # noqa: E402# ---------------------------------------------------------------------------@@ -139,20 +154,24 @@ def extract_skin(char_armature):logger.info(f"Extracted {len(vertices)} vertices ({(~skinned).sum()} unskinned), "f"{len(all_faces)} faces, {len(bones)} bones from {len(meshes)} mesh(es)")- # Canonical joint order baked in by preprocess_char (glTF extras carry+ # Canonical joint order baked into a canonical asset (glTF extras carry# object-level custom properties); bone enumeration order itself is not# stable across glTF round trips.import json- order = (char_armature.get('canonical_joint_order')- or char_armature.data.get('canonical_joint_order'))- if isinstance(order, str):- order = json.loads(order)+ order = canonical_joint_order(char_armature)+ # The clips' rest root, on a Mixamo character baked by canonical_assets.py+ # (its own rest root differs: motion_from_canonical_char).+ motion_root = (char_armature.get('canonical_motion_rest_root')+ or char_armature.data.get('canonical_motion_rest_root'))+ if isinstance(motion_root, str):+ motion_root = json.loads(motion_root)return {'bone_names': bone_names, 'parents': parents,'rest_global': rest_global, 'rest_local': rest_local,'vertices': vertices, 'weights': weights, 'faces': all_faces,'arm_world': arm_world, 'canonical_order': order,+ 'motion_rest_root': None if motion_root is None else np.asarray(motion_root, dtype=np.float64),}@@ -161,7 +180,8 @@ def extract_skin(char_armature):# ---------------------------------------------------------------------------def load_motion_local_mats(anim_path, dataset_type=None, cond_path=None):- """Return ``(anim_local (F,J,4,4), rest_local (J,4,4), bone_names, fps)``.+ """Return ``(anim_local (F,J,4,4), rest_local (J,4,4), bone_names, fps,+ rest_global (J,4,4))``.Auto-detects the NPZ flavor: export-stage (self-contained) vsfeature-format (generated motion; resolved through ``cond.npy`` exactly@@ -181,11 +201,12 @@ def load_motion_local_mats(anim_path, dataset_type=None, cond_path=None):# Either flag resolves the cond: --dataset_type picks the dataset's# default cond.npy (and the 'mixamo' key), --cond_path names the file# directly (object type from the clip's filename prefix, or the file's- # single entry โ a preprocess_char cond has exactly one).+ # single entry โ a rig_preprocess cond has exactly one).if not dataset_type and not cond_path:raise RuntimeError("feature-format motion needs --dataset_type and/or --cond_path "- "(omit both only for a canonical asset from preprocess_char)")+ "(omit both only for a canonical asset: rig_preprocess, "+ "canonical_assets.py)")cond_path = cond_path or COND_PATH_TEMPLATE.format(dataset_type=dataset_type)cond_data = load_cond_data(cond_path, anim_path, dataset_type)anim, rest_anim, _tpos, raw = build_anim_from_npz(anim_path, cond_data)@@ -197,16 +218,20 @@ def load_motion_local_mats(anim_path, dataset_type=None, cond_path=None):"export-stage ('anim_local_rot', ...) or feature format ""('local_rotations', ...)")return (transforms_local(anim), transforms_local(rest_anim)[0],- [str(n) for n in names], fps)+ [str(n) for n in names], fps, transforms_global(rest_anim)[0])def motion_from_canonical_char(data, skin):- """Feature NPZ driving a canonical asset (:mod:`preprocess_char` output).+ """Feature NPZ driving a canonical asset (``rig_preprocess`` or ``canonical_assets.py`` output).The asset's own skeleton IS the motion skeleton: joint order = boneorder, rest pose = the asset's rest, scale = 1 โ so no ``cond.npy`` isneeded. Root positions come from ``global_positions``; every other jointkeeps its rest offset (rigid bones), rotations from ``local_rotations``.+ A Mixamo character other than the clips' rig stores the clips' rest root+ (``canonical_motion_rest_root``): its root then moves by the clip's change+ from that rest, starting from its own rest root, as :mod:`animate_motion`+ retargets the root, so its feet stay on the ground."""q = np.asarray(data['local_rotations'], dtype=np.float64) # (F, J, 4) wxyzgp = np.asarray(data['global_positions'], dtype=np.float64) # (F, J, 3)@@ -214,14 +239,15 @@ def motion_from_canonical_char(data, skin):order = skin.get('canonical_order')if order is None:logger.warning("Asset carries no canonical_joint_order property (not "- "produced by preprocess_char?); assuming its bone "+ "produced by rig_preprocess / canonical_assets.py?); "+ "assuming its bone ""enumeration order matches the motion's joint order")order = list(skin['bone_names'])J = len(order)assert q.shape[1] == J, (f"motion has {q.shape[1]} joints but the asset's canonical skeleton "f"has {J} bones; the cond-free path needs an asset baked to the "- f"motion skeleton (see preprocess_char.py) โ or pass "+ f"motion skeleton (rig_preprocess, canonical_assets.py) โ or pass "f"--dataset_type/--cond_path")missing = [n for n in order if n not in skin['bone_names']]assert not missing, f"canonical_joint_order names missing from armature: {missing}"@@ -230,6 +256,8 @@ def motion_from_canonical_char(data, skin):rest_local = skin['rest_local'][idx] # motion joint orderpos = np.repeat(rest_local[None, :, :3, 3], q.shape[0], axis=0)pos[:, 0] = gp[:, 0]+ if skin.get('motion_rest_root') is not None:+ pos[:, 0] += rest_local[0, :3, 3] - skin['motion_rest_root']anim_local = quat_pos_to_mats(q, pos)return anim_local, rest_local, list(order), npz_scalar(data, 'fps', 30)@@ -238,10 +266,11 @@ def motion_from_canonical_char(data, skin):# Manual LBS# ----------------------------------------------------------------------------def lbs_deform(skin, anim_local, motion_rest_local, motion_bone_names):+def lbs_deform(skin, anim_local, motion_rest_local, motion_bone_names, conj=None):"""Run manual Linear Blend Skinning; returns world-space ``(F, V, 3)``.- Per frame and asset bone::+ Per frame and asset bone (*conj*: optional ``(J, 3, 3)`` per-motion-bone+ rotations from :func:`motion_axis_alignment`, conjugating ``rel``)::pose_local[b] = rest_local[b] @ inv(m_rest[b]) @ m_anim[f, b] (driven)pose_local[b] = rest_local[b] (undriven)@@ -274,6 +303,10 @@ def lbs_deform(skin, anim_local, motion_rest_local, motion_bone_names):pose_local = np.broadcast_to(skin['rest_local'], (nframes, nbones, 4, 4)).copy()for b, j in driven:rel = np.linalg.inv(motion_rest_local[j])[None] @ anim_local[:, j]+ if conj is not None:+ C = np.eye(4)+ C[:3, :3] = conj[j]+ rel = C.T[None] @ rel @ C[None]pose_local[:, b] = skin['rest_local'][b][None] @ rel# FK to armature space (bones are stored parent-before-child in Blender).@@ -311,26 +344,50 @@ def save_obj_sequence(dirpath, vertices, faces):logger.info(f"Saved {vertices.shape[0]} OBJ frames to {dirpath}")+def resolve_lbs_char(anim_path, dataset_type, character=None):+ """The processed asset for a motion (module docstring): canonical for a+ feature NPZ, export for an export NPZ."""+ if not dataset_type:+ raise ValueError("--char_path is required without --dataset_type")+ kind = 'canonical' if is_feature_motion(anim_path) else 'export'+ return resolve_processed_asset(anim_path, dataset_type, kind, character)++def animate_lbs(char_path, anim_path, output_dir, dataset_type=None,- cond_path=None, save=('glb',)):- """Deform a rigged asset with a motion NPZ via manual LBS and save."""+ cond_path=None, save=('glb',), character=None):+ """Deform a rigged asset with a motion NPZ via manual LBS and save.+ *char_path* None: :func:`resolve_lbs_char`."""assert os.path.exists(anim_path), f"Animation file not found: {anim_path}"+ if not anim_path.endswith('.npz'):+ raise ValueError(+ f"{anim_path}: animate_lbs reads motion NPZs (stage-1 export or stage-4 "+ f"feature format). Drive a sampler .npy with animate_motion "+ f"(scripts/run_animate_motion.sh <samples_dir>).")os.makedirs(output_dir, exist_ok=True)+ char_path = char_path or resolve_lbs_char(anim_path, dataset_type, character)rig_formats = tuple(f for f in save if f in ('glb', 'fbx'))char_armature = load_character(char_path, pack_textures=bool(rig_formats))skin = extract_skin(char_armature)data = np.load(anim_path, allow_pickle=True)- if 'local_rotations' in data and dataset_type is None and cond_path is None:- logger.info("Feature NPZ with no cond: treating the asset as the "- "canonical motion skeleton (preprocess_char output)")+ if 'local_rotations' in data and (is_canonical_asset(char_armature)+ or (dataset_type is None and cond_path is None)):+ logger.info("Feature NPZ on a canonical asset: the asset is the motion "+ "skeleton (no cond)")+ if cond_path is not None:+ logger.warning(f"--cond_path {cond_path} not used: the canonical asset "+ f"carries the motion's skeleton")anim_local, rest_local, motion_names, fps = motion_from_canonical_char(data, skin)+ conj = Noneelse:- anim_local, rest_local, motion_names, fps = load_motion_local_mats(+ anim_local, rest_local, motion_names, fps, rest_global = load_motion_local_mats(anim_path, dataset_type=dataset_type, cond_path=cond_path)+ # The asset's bone axes may differ from the motion's.+ conj = motion_axis_alignment(rest_global, motion_names,+ skin['rest_global'], skin['bone_names'])- vertices = lbs_deform(skin, anim_local, rest_local, motion_names)+ vertices = lbs_deform(skin, anim_local, rest_local, motion_names, conj=conj)stem = os.path.splitext(os.path.basename(anim_path))[0] + '_lbs'faces_arr = np.array(skin['faces'], dtype=object)@@ -350,7 +407,9 @@ def animate_lbs(char_path, anim_path, output_dir, dataset_type=None,# keep their rest local transform (no keyframes = follow parent),# matching the LBS 'keep' semantics.set_scene_timing(anim_local.shape[0], fps)- keyframes = compute_bone_keyframes(rest_local, anim_local, motion_names)+ keyframes = compute_bone_keyframes(+ rest_local, anim_local, motion_names,+ None if conj is None else rotations_to_quats(conj))rebuild_action_from_data(char_armature, keyframes)update_scene()export_animated_character(os.path.join(output_dir, stem),@@ -358,19 +417,28 @@ def animate_lbs(char_path, anim_path, output_dir, dataset_type=None,return vertices+# Outputs ``--save`` can ask for (see animate_lbs).+SAVE_FORMATS = ('glb', 'fbx', 'npz', 'obj')++def parse_args():parser = argparse.ArgumentParser(description="Animate a rigged asset via manual NumPy LBS.")- parser.add_argument("--char_path", type=str, required=True,- help="Rigged asset GLB/FBX (mesh + armature + weights).")+ parser.add_argument("--char_path", type=str, default=None,+ help="Rigged asset GLB/FBX (mesh + armature + weights). Default with "+ "--dataset_type: the processed asset (module docstring).")+ parser.add_argument("--character", type=str, default=None,+ help=f"Mixamo character for the auto-resolved asset "+ f"(default {DEFAULT_MIXAMO_CHARACTER}).")parser.add_argument("--anim_path", type=str, required=True,help="Motion NPZ (export-stage or feature-format, auto-detected), ""or a DIRECTORY of NPZs โ one output set per action clip.")parser.add_argument("--dataset_type", type=str, default=None,- choices=['truebones', 'objaverse', 'mixamo'],- help="For feature-format motion: resolves cond.npy. Omit it "- "(and --cond_path) to drive a canonical asset baked by "- "preprocess_char.py cond-free.")+ choices=['truebones', 'objaverse', 'mixamo', 'general'],+ help="Resolves --char_path when omitted (the processed asset of "+ "the clip: canonical for a feature NPZ, export for an export "+ "NPZ) and, for a feature NPZ on a non-canonical asset, "+ "cond.npy. A canonical asset is always driven cond-free.")parser.add_argument("--cond_path", type=str, default=None,help="cond.npy override for feature-format motion.")parser.add_argument("--output_dir", type=str, default='outputs/lbs')@@ -384,6 +452,10 @@ def parse_args():if __name__ == "__main__":args = parse_args()save = tuple(s.strip() for s in args.save.split(',') if s.strip())+ unknown = sorted(set(save) - set(SAVE_FORMATS))+ if unknown or not save:+ raise SystemExit(f"--save {args.save!r}: expected a comma-separated subset of "+ f"{', '.join(SAVE_FORMATS)}")if os.path.isdir(args.anim_path):import glob as _glob@@ -396,7 +468,7 @@ if __name__ == "__main__":lambda clip: animate_lbs(char_path=args.char_path, anim_path=clip,output_dir=args.output_dir, dataset_type=args.dataset_type,- cond_path=args.cond_path, save=save),+ cond_path=args.cond_path, save=save, character=args.character),output_dir=args.output_dir,desc="LBS animating",)@@ -408,4 +480,5 @@ if __name__ == "__main__":dataset_type=args.dataset_type,cond_path=args.cond_path,save=save,+ character=args.character,)diff --git a/data_process/mesh_animation/animate_motion.py b/data_process/mesh_animation/animate_motion.pyindex 60b23c4..d29918b 100644--- a/data_process/mesh_animation/animate_motion.py+++ b/data_process/mesh_animation/animate_motion.py@@ -4,14 +4,18 @@ This is the inference-time path: it takes a motion in the *feature* formatwritten by the feature-extraction stage (``global_positions`` /``local_rotations`` / ``root_facing_quat``) โ or generated by the model inthat same format โ together with the dataset's ``cond.npy`` (skeleton-topology, joint names, scale) and drives the original rigged asset with it.+topology, joint names, scale) and drives a rigged asset of that object type+with it (resolution below).Pipeline:1. Load the cond entry for the requested ``--dataset_type`` and the motionfile (NPZ, or the ``.npy`` motion features written by the model's sampler).2. Reconcile the character armature against the animation's bone list(see :func:`data_process.utils.blender_rig.sync_armature_bones`).- 3. Compute per-bone keyframes and bind them as a Blender Action.+ 3. Compute per-bone keyframes, re-expressed in the rig's bone axes when they+ differ from the motion's+ (:func:`data_process.utils.skeleton.motion_axis_alignment`), and bind+ them as a Blender Action.4. Export the animated character as ``.glb`` and ``.fbx``.Usage (Blender headless):@@ -20,8 +24,24 @@ Usage (Blender headless):--anim_path dataset/features/objaverse/motions/<clip>.npz \\--output_dir outputs/animated-``--char_path`` is auto-resolved for truebones / objaverse from the raw-asset directories; it is required for mixamo.+``--char_path`` is auto-resolved by ``--asset``:++- ``canonical`` (default): the canonical asset of the object type+ (``dataset/canonical_assets/<ds>/<type>.glb``, feature_extraction/+ canonical_assets.py; mixamo: of ``--character``, default ``Michelle``). An+ object type without one (e.g. a skeleton the bake refused) falls back to+ the ``export`` resolution with a warning; ``--canonical`` makes it an error.+- ``export``: the processed GLB of the object type+ (``dataset/export/<ds>/rigs/<type>.glb``, written by the export stage's+ ``--save_glb`` / ``--glb_only``: the asset in its rest pose on the exported+ skeleton) when it exists, else the raw asset (truebones / objaverse /+ general); for mixamo the processed character ``--character`` (default+ ``Michelle``, else the raw Y Bot FBX).++A canonical character is detected from its ``canonical_joint_order``+property, and the motion then stays in the canonical frame and scale (any+other character gets it divided by the cond's ``scale_factor``, back to the+export's units)."""import argparse@@ -36,11 +56,18 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))from Animation import Animation, Quaternions # noqa: E402from data_process.mesh_animation.common import ( # noqa: E402+ DEFAULT_MIXAMO_CHARACTER,+ check_canonical_order,drive_and_export,+ find_processed_asset,+ is_canonical_asset,load_character,npz_scalar,parse_blender_argv,+ processed_asset_path,+ resolve_processed_asset,)+from data_process.utils.blender_export import asset_name, list_asset_files # noqa: E402from data_process.utils.blender_rig import EXTRA_BONES_STRATEGIES # noqa: E402from data_process.utils.skeleton import scale_anim # noqa: E402@@ -50,6 +77,9 @@ MIXAMO_OBJECT_TYPE = 'mixamo'# Default raw-asset locations used for --char_path auto-resolution.TRUEBONES_RAW_DIR = 'dataset/raw/truebones/Truebone_Z-OO'OBJAVERSE_RAW_DIR = 'dataset/raw/objaverse/glb'+GENERAL_RAW_DIR = 'dataset/raw/general/animation'+# Mixamo's clips carry no mesh; the character they were made on.+MIXAMO_RAW_DEFAULT = 'dataset/raw/mixamo/character_refined/Y_Bot.fbx'# Default cond.npy location per dataset (written by the feature-extraction stage).COND_PATH_TEMPLATE = 'dataset/features/{dataset_type}/cond.npy'@@ -59,11 +89,15 @@ COND_PATH_TEMPLATE = 'dataset/features/{dataset_type}/cond.npy'# Animation loading# ----------------------------------------------------------------------------def build_anim_from_npz(anim_path, cond_data):+def build_anim_from_npz(anim_path, cond_data, scale_factor=None):"""Build ``(anim, rest_anim, tpos_global_rot, raw_anim_data)`` from a- feature-format NPZ (``global_positions`` + ``local_rotations``)."""+ feature-format NPZ (``global_positions`` + ``local_rotations``).++ Positions are divided by *scale_factor* (default: the cond's, back to the+ export's units; 1.0 keeps the canonical scale, for a canonical asset)."""parents = cond_data['parents']- scale_factor = npz_scalar(cond_data, 'scale_factor', 1.0)+ if scale_factor is None:+ scale_factor = npz_scalar(cond_data, 'scale_factor', 1.0)offsets_scaled = cond_data['offsets'] / scale_factortpos_local_rot = cond_data['tpos_local_rotations']@@ -94,9 +128,10 @@ def build_anim_from_npz(anim_path, cond_data):return anim, rest_anim, None, anim_data-def build_anim_from_npy(anim_path, cond_data, anim_mode):+def build_anim_from_npy(anim_path, cond_data, anim_mode, scale_factor=None):"""Build ``(anim, rest_anim, tpos_global_rot, raw_anim_data)`` from a``.npy`` of model motion features (the format ``unimate.inference.sample`` saves).+ *scale_factor* as in :func:`build_anim_from_npz`.Decoding model features back into an ``Animation`` requires thetraining/inference package (``unimate.utils.motion_utils``), which is@@ -114,7 +149,8 @@ def build_anim_from_npy(anim_path, cond_data, anim_mode):) from excparents = cond_data['parents']- scale_factor = npz_scalar(cond_data, 'scale_factor', 1.0)+ if scale_factor is None:+ scale_factor = npz_scalar(cond_data, 'scale_factor', 1.0)tpos_global_rot = cond_data['tpos_global_rotations']offsets = cond_data['tpos_offsets']@@ -156,12 +192,12 @@ def build_anim_from_npy(anim_path, cond_data, anim_mode):return anim, rest_anim, tpos_global_rot, anim_data-def load_animation(anim_path, cond_data, anim_mode):+def load_animation(anim_path, cond_data, anim_mode, scale_factor=None):"""Dispatch to the NPZ or NPY loader based on file extension."""if anim_path.endswith('.npz'):- return build_anim_from_npz(anim_path, cond_data)+ return build_anim_from_npz(anim_path, cond_data, scale_factor)if anim_path.endswith('.npy'):- return build_anim_from_npy(anim_path, cond_data, anim_mode)+ return build_anim_from_npy(anim_path, cond_data, anim_mode, scale_factor)raise RuntimeError(f"Unsupported animation file format: {anim_path}")@@ -172,7 +208,7 @@ def load_animation(anim_path, cond_data, anim_mode):def resolve_object_type(anim_path, dataset_type):"""Map an animation filename to its cond-dict key.- - truebones / objaverse: filename is ``{object_type}-{motion}[-{clip}].npz``,+ - truebones / objaverse / general: filename is ``{object_type}-{motion}[-{clip}].npz``,so the object type is the prefix before the first '-'.- mixamo: motion filenames carry no ``{object_type}-`` prefix; the conddict always uses ``'mixamo'`` as its single key.@@ -182,19 +218,60 @@ def resolve_object_type(anim_path, dataset_type):return os.path.basename(anim_path).split("-")[0]-def resolve_char_path(anim_path, dataset_type):+# Characters --asset can auto-resolve (see resolve_char_path).+ASSET_KINDS = ('canonical', 'export')+++def resolve_char_path(anim_path, dataset_type, character=None, asset='canonical',+ strict=False):"""Auto-resolve a character mesh path from ``anim_path`` + ``dataset_type``.+ *asset* 'canonical' (default): the canonical asset+ (``dataset/canonical_assets/<ds>/``; Mixamo: of *character*, default+ Michelle). When the object type has none, an error with *strict*, else the+ 'export' resolution with a warning.++ *asset* 'export': the processed export GLB of the object type+ (``dataset/export/<ds>/rigs/``; Mixamo: of *character*, default Michelle)+ wins over the raw asset: its armature already is the exported skeleton,+ so no bones are merged away at animate time. The raw assets:+- truebones: ``TRUEBONES_RAW_DIR/{object_type}/*.fbx`` (prefers an``Idle`` / T-pose variant when available).- objaverse: ``OBJAVERSE_RAW_DIR/{object_type}.glb``.- - mixamo: no auto-resolution โ ``--char_path`` is required.+ - general: the file in ``GENERAL_RAW_DIR`` whose asset name (stem with+ ``-`` and whitespace replaced by ``_``, as the exporter names it) is+ the object type.+ - mixamo: the raw Y Bot (``MIXAMO_RAW_DEFAULT``, the rig the clips were+ made on) when the default character is asked for; any other raw+ character needs ``--char_path``."""+ if asset not in ASSET_KINDS:+ raise ValueError(f"asset must be one of {ASSET_KINDS}, got {asset!r}")+ if asset == 'canonical':+ processed = find_processed_asset(anim_path, dataset_type, 'canonical', character)+ if processed is not None:+ logger.info(f"Using the canonical asset: {processed}")+ return processed+ if strict:+ return resolve_processed_asset(anim_path, dataset_type, 'canonical',+ character) # raises+ logger.warning(+ f"No canonical asset "+ f"{processed_asset_path(anim_path, dataset_type, 'canonical', character)}; "+ f"using the export / raw asset instead (motion at the export's scale)")++ processed = find_processed_asset(anim_path, dataset_type, 'export', character)+ if processed is not None:+ logger.info(f"Using the processed export asset: {processed}")+ return processed+ if (dataset_type == 'mixamo'+ and (character or DEFAULT_MIXAMO_CHARACTER) == DEFAULT_MIXAMO_CHARACTER+ and os.path.isfile(MIXAMO_RAW_DEFAULT)):+ logger.info(f"No processed Mixamo characters; using the raw {MIXAMO_RAW_DEFAULT}")+ return MIXAMO_RAW_DEFAULTif dataset_type == 'mixamo':- raise ValueError(- "dataset_type='mixamo' has no auto-resolved character mesh; "- "pass --char_path explicitly (or set CHAR_PATH in the bash wrapper)."- )+ return resolve_processed_asset(anim_path, dataset_type, 'export', character) # raisesobject_type = resolve_object_type(anim_path, dataset_type)@@ -218,6 +295,16 @@ def resolve_char_path(anim_path, dataset_type):f"(resolved from '{anim_path}') at {char_path}")return char_path+ if dataset_type == 'general':+ matches = [p for p in (list_asset_files(GENERAL_RAW_DIR)+ if os.path.isdir(GENERAL_RAW_DIR) else [])+ if asset_name(p) == object_type]+ if not matches:+ raise FileNotFoundError(+ f"No general asset file for object_type='{object_type}' "+ f"(resolved from '{anim_path}') in {GENERAL_RAW_DIR}")+ return str(matches[0])+raise ValueError(f"Unknown dataset_type: {dataset_type}")@@ -225,7 +312,7 @@ def load_cond_data(cond_path, anim_path, dataset_type):"""Pick the clip's cond entry out of ``cond_path``.The key is the object type resolved from the filename (or ``'mixamo'``).- A cond file holding exactly one entry (``preprocess_char.py`` output, or+ A cond file holding exactly one entry (``rig_preprocess`` output, ora custom single-asset export) is accepted whatever the clip is called,since there is nothing to disambiguate."""@@ -259,7 +346,7 @@ def animate_character(char_path, anim_path, cond_path, output_dir,anim_path: Feature-format motion NPZ, or a ``.npy`` of model motion features.cond_path: ``cond.npy`` for the dataset.output_dir: Destination directory (created if missing).- dataset_type: ``'truebones'`` | ``'objaverse'`` | ``'mixamo'``;+ dataset_type: ``'truebones'`` | ``'objaverse'`` | ``'mixamo'`` | ``'general'``;controls how the cond-dict key is derived from ``anim_path``.anim_mode: Recovery mode for ``.npy`` model-feature input, ``'fk'`` or ``'ik'``.extra_bones_strategy: How to handle armature bones not present in@@ -272,10 +359,17 @@ def animate_character(char_path, anim_path, cond_path, output_dir,anim_base = os.path.splitext(os.path.basename(anim_path))[0]char_armature = load_character(char_path)+ # A canonical asset already is at the cond's canonical scale: keep the+ # motion there instead of dividing it back to the export's units.+ canonical = is_canonical_asset(char_armature)+ if canonical:+ logger.info("Canonical asset: motion kept in the canonical frame and scale")cond_data = load_cond_data(cond_path, anim_path, dataset_type)+ if canonical:+ check_canonical_order(char_armature, cond_data['joint_names'], cond_path)anim, rest_anim, tpos_global_rot, anim_data = load_animation(- anim_path, cond_data, anim_mode)+ anim_path, cond_data, anim_mode, scale_factor=1.0 if canonical else None)drive_and_export(char_armature, anim, rest_anim, cond_data['joint_names'],@@ -291,16 +385,29 @@ def parse_args():parser = argparse.ArgumentParser(description="Drive a rigged character with a canonicalized motion clip.")parser.add_argument("--dataset_type", type=str, required=True,- choices=['truebones', 'objaverse', 'mixamo'],+ choices=['truebones', 'objaverse', 'mixamo', 'general'],help="Dataset that produced the cond/anim files; controls ""how the cond-dict key is resolved.")parser.add_argument("--anim_path", type=str, required=True,help="Feature-format motion NPZ, or a .npy of model motion features.")parser.add_argument("--char_path", type=str, default=None,help="Character FBX/GLB. If omitted, auto-resolved from "- f"--dataset_type and --anim_path (truebones -> "+ "--dataset_type and --anim_path by --asset.")+ parser.add_argument("--asset", type=str, default='canonical', choices=list(ASSET_KINDS),+ help="Auto-resolved character: 'canonical' (default) the canonical "+ "asset in dataset/canonical_assets/<ds>/ (mixamo: of "+ "--character), falling back to 'export' when the object type "+ "has none; 'export' the processed GLB in dataset/export/<ds>/rigs/ "+ "when present, else the raw asset (truebones -> "f"{TRUEBONES_RAW_DIR}/{{object_type}}/*.fbx, objaverse -> "- f"{OBJAVERSE_RAW_DIR}/{{object_type}}.glb). Required for mixamo.")+ f"{OBJAVERSE_RAW_DIR}/{{object_type}}.glb, general -> the "+ f"{GENERAL_RAW_DIR} file named like the object type).")+ parser.add_argument("--character", type=str, default=None,+ help=f"Mixamo character to animate (default {DEFAULT_MIXAMO_CHARACTER}); "+ "mixamo only (elsewhere the asset is the motion's object type).")+ parser.add_argument("--canonical", action="store_true",+ help="Require the canonical asset: --asset canonical without the "+ "export fallback (an error when it is missing).")parser.add_argument("--cond_path", type=str, default=None,help=f"cond.npy for the dataset (default: {COND_PATH_TEMPLATE}).")parser.add_argument("--output_dir", type=str, default='outputs/animated')@@ -318,7 +425,11 @@ def parse_args():if __name__ == "__main__":args = parse_args()- char_path = args.char_path or resolve_char_path(args.anim_path, args.dataset_type)+ if args.canonical and args.asset != 'canonical':+ raise SystemExit("--canonical contradicts --asset export")+ char_path = args.char_path or resolve_char_path(+ args.anim_path, args.dataset_type, character=args.character,+ asset=args.asset, strict=args.canonical)cond_path = args.cond_path or COND_PATH_TEMPLATE.format(dataset_type=args.dataset_type)logger.info(f"Using character mesh: {char_path}")logger.info(f"Using cond file: {cond_path}")diff --git a/data_process/mesh_animation/animate_npz.py b/data_process/mesh_animation/animate_npz.pyindex 052151f..e49f5d2 100644--- a/data_process/mesh_animation/animate_npz.py+++ b/data_process/mesh_animation/animate_npz.py@@ -9,6 +9,11 @@ round-trip checks of the export and for re-rendering exported clips.For *generated* motions (feature NPZ + ``cond.npy``) use:mod:`data_process.mesh_animation.animate_motion` instead.+``--char_path`` may be omitted with ``--dataset_type``: the processed export+asset ``dataset/export/<ds>/rigs/<name>.glb`` is used (run_export.sh+--save_glb / --glb_only), whose skeleton is the NPZ's, so nothing is merged;+<name> is the clip's object type, for Mixamo ``--character`` (default Michelle).+Usage (Blender headless):blender -b -P data_process/mesh_animation/animate_npz.py -- \\--char_path my_model.glb \\@@ -28,11 +33,13 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))from Animation import Animation, Quaternions # noqa: E402from data_process.mesh_animation.common import ( # noqa: E402+ DEFAULT_MIXAMO_CHARACTER,clip_output_path,drive_and_export,load_character,npz_scalar,parse_blender_argv,+ resolve_processed_asset,)from data_process.utils.blender_rig import EXTRA_BONES_STRATEGIES # noqa: E402@@ -103,8 +110,14 @@ def animate_character(char_path, anim_path, output_dir, char_anim_type='glb',def parse_args():parser = argparse.ArgumentParser(description="Drive a rigged character with an exported motion NPZ.")- parser.add_argument("--char_path", type=str, required=True,- help="Character FBX/GLB (mesh + armature).")+ parser.add_argument("--char_path", type=str, default=None,+ help="Character FBX/GLB (mesh + armature). Default with "+ "--dataset_type: the processed export asset.")+ parser.add_argument("--dataset_type", type=str, default=None,+ choices=['truebones', 'objaverse', 'mixamo', 'general'],+ help="Resolve --char_path from dataset/export/<ds>/rigs/.")+ parser.add_argument("--character", type=str, default=None,+ help=f"Mixamo character (default {DEFAULT_MIXAMO_CHARACTER}).")parser.add_argument("--anim_path", type=str, required=True,help="Export-stage motion NPZ whose bone names match the rig.")parser.add_argument("--output_dir", type=str, default='outputs/animated')@@ -117,8 +130,14 @@ def parse_args():if __name__ == "__main__":args = parse_args()+ char_path = args.char_path+ if char_path is None:+ if not args.dataset_type:+ raise SystemExit("--char_path is required without --dataset_type")+ char_path = resolve_processed_asset(args.anim_path, args.dataset_type, 'export',+ args.character)animate_character(- char_path=args.char_path,+ char_path=char_path,anim_path=args.anim_path,output_dir=args.output_dir,char_anim_type=args.char_anim_type,diff --git a/data_process/mesh_animation/canonical_rig.py b/data_process/mesh_animation/canonical_rig.pynew file mode 100644index 0000000..0270def--- /dev/null+++ b/data_process/mesh_animation/canonical_rig.py@@ -0,0 +1,193 @@+"""Rebuild a rigged asset to a cond's canonical T-pose (bpy).++:func:`bake_canonical_asset` writes the rest-pose asset whose armature is the+stage-4 motion skeleton itself: the cond's joint order, canonical rest positions+and ``tpos_global_rotations``, in the Y-up feature frame (scale and grounding of+the training clips), so a feature clip drives it with no ``cond.npy``+(``animate_lbs``, cond-free). Used by ``feature_extraction/canonical_assets.py`` (a dataset's export assets,+also run by ``data_process.rig_preprocess`` for one asset).+"""++import json+import math++import bpy+import numpy as np+from loguru import logger+from mathutils import Matrix++from data_process.mesh_animation.common import load_character, quat_pos_to_mats+from data_process.utils.blender_export import clear_animation_state+from data_process.utils.blender_rig import (+ export_selected_to_file,+ find_skinned_meshes,+ rename_truncated_bones,+ select_objs,+ skin_rigid_parts,+ sync_armature_bones,+ update_scene,+)+from data_process.utils.skeleton import line_ratio, similarity_fit++# Largest rig-to-cond joint residual (relative to the extent) the fit accepts.+FIT_TOL = 5e-3+# Largest roll (radians) the fit may leave undetermined about a nearly straight+# skeleton, estimated as residual / line_ratio.+ROLL_TOL = 0.02+++class UndeterminedRoll(ValueError):+ """The similarity fit cannot pin the skeleton's roll about its line."""+++def transform_mesh_points(mesh, T):+ """Apply the affine ``(4, 4)`` *T* to every vertex of *mesh* and to the+ points of every shape key (a mesh with shape keys is evaluated and+ exported from its key blocks, not from ``vertices``). *T* being affine,+ each key's offsets from the basis move with it."""+ def moved(points):+ co = np.empty(len(points) * 3)+ points.foreach_get('co', co)+ co = co.reshape(-1, 3) @ T[:3, :3].T + T[:3, 3]+ points.foreach_set('co', co.ravel())++ moved(mesh.vertices)+ if mesh.shape_keys is not None:+ for key_block in mesh.shape_keys.key_blocks:+ moved(key_block.data)+ mesh.update()+++def bake_canonical_asset(char_path, cond, out_base, formats=('glb',), extra_props=None):+ """Rebuild the asset's armature to the cond's canonical T-pose and save.++ The canonical skeleton (joint order, rest positions/orientations) is+ taken verbatim from the topology cond so the baked asset and the motion+ features agree by construction. The skinned mesh gets the same+ similarity transform, so binding is untouched; asset bones outside the+ motion skeleton are weight-merged away first. *extra_props* are stored+ as further object-level custom properties on the armature (glTF extras),+ beside ``canonical_joint_order``.+ """+ names = list(cond['joint_names'])+ canon_parents = [int(p) for p in cond['parents']]+ canon_pos = np.asarray(cond['tpos_first_frame'], dtype=np.float64)+ canon_mats = quat_pos_to_mats(+ np.asarray(cond['tpos_global_rotations'], dtype=np.float64), canon_pos)++ armature = load_character(char_path)+ rename_truncated_bones(armature, names)+ # Rigid parts (bone- or armature-parented meshes) would otherwise be+ # dropped below with the props; bind them to their bone first.+ skin_rigid_parts(armature)+ meshes = find_skinned_meshes(armature)+ assert meshes, "No mesh with an Armature modifier targeting the asset's armature"++ # The canonical asset holds ONLY the rigged character: drop unskinned+ # props, lights, cameras, helper empties (e.g. the egg Icosphere in+ # Truebones' Chicken-EggLaying) โ they can't be driven by the motion and+ # would linger as static junk in the export.+ keep = {armature.name} | {m.name for m in meshes}+ dropped = [o.name for o in bpy.context.scene.objects if o.name not in keep]+ for name_ in dropped:+ bpy.data.objects.remove(bpy.data.objects[name_], do_unlink=True)+ if dropped:+ logger.info(f"Dropped {len(dropped)} non-skinned scene object(s): {dropped[:6]}")++ missing = [n for n in names if n not in armature.data.bones]+ assert not missing, f"cond joints missing from the asset: {missing}"+ sync_armature_bones(armature, names, extra_bones_strategy='merge')++ # Similarity mapping Blender world -> canonical frame: the whole export ++ # extract chain is one similarity of the world frame; fit it on the bone+ # heads and verify the residual (a current import may read the bind pose+ # of an older export up to ~2e-3 apart; more is another skeleton).+ arm_world = np.array(armature.matrix_world)+ bones = armature.data.bones+ world_pos = np.stack([+ (arm_world @ np.array(bones[n].matrix_local))[:3, 3] for n in names])+ R, sim_scale, t, resid = similarity_fit(world_pos, canon_pos)+ if resid > FIT_TOL:+ raise ValueError(f"world->canonical is not a clean similarity (residual {resid:.2e} "+ f"of the extent); non-uniform armature scale, or the cond belongs "+ f"to another asset?")+ straightness = line_ratio(canon_pos)+ if resid > ROLL_TOL * straightness:+ raise UndeterminedRoll(+ f"nearly straight skeleton (line ratio {straightness:.1e}) fitted with residual "+ f"{resid:.1e}: its roll about the joint line is uncertain by about "+ f"{resid / max(straightness, 1e-12):.2f} rad")+ if resid > 1e-4:+ logger.warning(f"Rig joints are {resid:.1e} of the extent from the cond's T-pose; "+ f"the bones are placed at the cond's")+ M = np.eye(4)+ M[:3, :3] = sim_scale * R+ M[:3, 3] = t++ # Transform every skinned mesh into the canonical frame. Un-parent BEFORE+ # zeroing transforms: the meshes hang under the armature, so identity-ing+ # the armature afterwards would shift any still-parented mesh again. A+ # mesh shared by several objects is copied first, or its vertices would+ # be moved once per object.+ for mesh_obj in meshes:+ if mesh_obj.data.users > 1:+ mesh_obj.data = mesh_obj.data.copy()+ for mesh_obj in meshes:+ transform_mesh_points(mesh_obj.data, M @ np.array(mesh_obj.matrix_world))+ mesh_obj.parent = None+ mesh_obj.matrix_world = Matrix.Identity(4)++ # Rebuild the armature: canonical rest, canonical (BFS) bone order.+ old_lengths = {n: bones[n].length for n in names}+ bpy.context.view_layer.objects.active = armature+ bpy.ops.object.mode_set(mode='EDIT')+ edit_bones = armature.data.edit_bones+ for eb in list(edit_bones):+ edit_bones.remove(eb)+ child_of = {p: j for j, p in enumerate(canon_parents) if p >= 0}+ for j, name in enumerate(names):+ eb = edit_bones.new(name)+ eb.head = (0.0, 0.0, 0.0)+ eb.tail = (0.0, 1.0, 0.0)+ length = (np.linalg.norm(canon_pos[child_of[j]] - canon_pos[j])+ if j in child_of else old_lengths[name] * sim_scale)+ eb.matrix = Matrix(canon_mats[j].tolist())+ eb.length = max(float(length), 1e-5)+ for j, name in enumerate(names):+ if canon_parents[j] >= 0:+ eb = edit_bones[name]+ eb.use_connect = False+ eb.parent = edit_bones[names[canon_parents[j]]]+ bpy.ops.object.mode_set(mode='OBJECT')+ # ARMATURE-SPACE data stays exactly canonical (Y-up feature frame) โ the+ # cond-free animate math lives there. The +90ยฐX OBJECT rotation stands+ # the asset up in Blender's Z-up world; verified with an independent+ # glTF evaluator, the exported file is upright BOTH in standard Y-up+ # viewers and when re-imported into Blender.+ yup_to_zup = Matrix.Rotation(math.radians(90.0), 4, 'X')+ armature.matrix_world = yup_to_zup+ for mesh_obj in meshes: # re-parent with identity LOCAL transforms+ mesh_obj.parent = armature+ mesh_obj.matrix_world = yup_to_zup.copy()++ # Strip animation; store the canonical joint ORDER as an OBJECT-level+ # custom property (glTF extras / FBX user properties) โ importers do not+ # keep bone enumeration order stable.+ clear_animation_state(armature)+ for obj in meshes:+ if obj.animation_data:+ obj.animation_data_clear()+ # Unbinding is not enough: the glTF exporter's default animation mode+ # exports every matchable ACTION datablock, bound or not โ delete them+ # all so the canonical asset is truly rest-only.+ for action in list(bpy.data.actions):+ bpy.data.actions.remove(action)+ update_scene()+ armature['canonical_joint_order'] = json.dumps(names)+ for key, value in (extra_props or {}).items():+ armature[key] = value++ select_objs([armature] + meshes, deselect_first=True)+ for fmt in formats:+ assert fmt in ('glb', 'fbx'), f"unsupported output format: {fmt}"+ export_selected_to_file(f'{out_base}.{fmt}', fmt, custom_props=True)diff --git a/data_process/mesh_animation/common.py b/data_process/mesh_animation/common.pyindex be04087..fa5f391 100644--- a/data_process/mesh_animation/common.py+++ b/data_process/mesh_animation/common.py@@ -7,23 +7,35 @@ where the driving animation comes from:animate_motion feature NPZ (or model-feature ``.npy``) + ``cond.npy``animate_npz export-stage NPZ (bone names carried in the file)+ animate_lbs either NPZ, deformed with NumPy LBS (cond-free on a+ canonical asset)animate_fbx raw animation FBX/GLB clips (action transfer by name)animate_mixamo batch wrapper over ``animate_npz``-This module holds the pieces they share: Blender-style CLI parsing, character-loading, the keyframe-and-export core, and the error-logged batch loop. Rig-primitives (keyframe math, armature reconciliation, exporters) stay in-:mod:`data_process.utils.blender_rig`.+This module holds the pieces they share: Blender-style CLI parsing, the+processed-asset lookup (``ASSET_DIR_TEMPLATES``), character loading, the+keyframe-and-export core (with the bone-axis alignment of+:func:`data_process.utils.skeleton.motion_axis_alignment`), and the+error-logged batch loop. Rig primitives (keyframe math, armature+reconciliation, exporters) stay in :mod:`data_process.utils.blender_rig`."""+import jsonimport osimport sys+import numpy as npfrom loguru import loggerfrom tqdm import tqdm-from Animation import transforms_local+from Animation import Quaternions, transforms_global, transforms_local+from data_process.utils.asset_files import (+ CANONICAL_ASSETS_DIR,+ CANONICAL_ORDER_KEY,+ DEFAULT_MIXAMO_CHARACTER,+ PROCESSED_GLB_DIR,+)from data_process.utils.blender_export import reset_scenefrom data_process.utils.blender_rig import (compute_bone_keyframes,@@ -36,8 +48,28 @@ from data_process.utils.blender_rig import (sync_armature_bones,update_scene,)+from data_process.utils.skeleton import motion_axis_alignment+from mathutils import Matrix # after bpy (imported above): the pip module registers itERROR_LOG_NAME = 'animate_errors.log'+# drive_and_export refuses a character holding fewer of the motion's joints+# than this share: it is the wrong character for the motion.+MIN_JOINT_MATCH = 0.5++# Processed assets (repo-relative, run from the repo root):+# export dataset/export/<ds>/rigs/<name>.glb rest pose on the exported+# skeleton (run_export.sh --save_glb / --glb_only); driven by the+# export NPZs, or by feature motions through the cond+# canonical dataset/canonical_assets/<ds>/<name>.glb the same in the stage-4+# canonical frame (feature_extraction/canonical_assets.py); driven+# by feature NPZs with no cond+# <name> is the object type (the motion file's prefix before the first '-');+# Mixamo clips carry none, so its assets are characters (default Michelle; any+# character of CHAR_DIR / canonical_assets/mixamo can be chosen).+ASSET_DIR_TEMPLATES = {+ 'export': os.path.join('dataset', 'export', '{dataset_type}', PROCESSED_GLB_DIR),+ 'canonical': os.path.join('dataset', CANONICAL_ASSETS_DIR, '{dataset_type}'),+}# ---------------------------------------------------------------------------@@ -66,15 +98,100 @@ def npz_scalar(data, key, default):return default+def motion_asset_name(anim_path, dataset_type, character=None):+ """Name of the processed asset a motion file belongs to (see ASSET_DIR_TEMPLATES)."""+ if dataset_type == 'mixamo':+ return character or DEFAULT_MIXAMO_CHARACTER+ if character:+ # Elsewhere a clip drives only its own object type's skeleton.+ raise ValueError(f"a character is chosen for mixamo only (dataset_type={dataset_type!r}); "+ "pass the asset file as --char_path")+ return os.path.basename(anim_path).split('-')[0]+++def processed_asset_path(anim_path, dataset_type, kind='export', character=None):+ """Where the processed GLB of the motion's asset lives (*kind* 'export' or+ 'canonical'; see ``ASSET_DIR_TEMPLATES``), whether or not it exists."""+ return os.path.join(ASSET_DIR_TEMPLATES[kind].format(dataset_type=dataset_type),+ motion_asset_name(anim_path, dataset_type, character) + '.glb')+++def find_processed_asset(anim_path, dataset_type, kind='export', character=None):+ """The processed GLB of the motion's asset, or None when it has not been written."""+ path = processed_asset_path(anim_path, dataset_type, kind, character)+ return path if os.path.isfile(path) else None+++def resolve_processed_asset(anim_path, dataset_type, kind='export', character=None):+ """:func:`find_processed_asset`, or FileNotFoundError saying how to build it."""+ path = find_processed_asset(anim_path, dataset_type, kind, character)+ if path is None:+ build = (f"run_export.sh {dataset_type} --glb_only" if kind == 'export'+ else "feature_extraction/canonical_assets.py")+ raise FileNotFoundError(+ f"No processed {kind} asset {processed_asset_path(anim_path, dataset_type, kind, character)} "+ f"for {anim_path!r}: build it ({build}) or pass --char_path")+ logger.info(f"Using the processed {kind} asset: {path}")+ return path+++def canonical_joint_order(armature):+ """The joint order a canonical asset stores (``canonical_joint_order``+ custom property, a JSON list in glTF extras), or None."""+ order = armature.get(CANONICAL_ORDER_KEY) or armature.data.get(CANONICAL_ORDER_KEY)+ if not order:+ return None+ if isinstance(order, str):+ order = json.loads(order)+ return [str(n) for n in order]+++def is_canonical_asset(armature):+ """True for an asset baked to the stage-4 canonical frame (it carries the+ ``canonical_joint_order`` custom property: canonical_assets.py,+ rig_preprocess)."""+ return canonical_joint_order(armature) is not None+++def check_canonical_order(armature, joint_names, source):+ """Raise when a canonical asset was baked from another skeleton than the+ motion's: index j of the motion must be its stored joint j."""+ order = canonical_joint_order(armature)+ names = [str(n) for n in joint_names]+ if order is not None and order != names:+ diff = next((i for i, (a, b) in enumerate(zip(order, names)) if a != b),+ min(len(order), len(names)))+ raise ValueError(+ f"the canonical asset's joint order ({len(order)} joints) is not the motion's "+ f"({len(names)} joints, from {source}); first difference at joint {diff}. "+ f"The asset was baked from another cond: rebuild it, or use the cond it was "+ f"baked from.")+++def is_feature_motion(anim_path):+ """Stage-4 feature format: an NPZ with ``local_rotations`` (and+ ``global_positions``), as opposed to an export NPZ or model ``.npy``."""+ if not anim_path.endswith('.npz'):+ return False+ with np.load(anim_path, allow_pickle=True) as data:+ return 'local_rotations' in data.files++def clip_output_path(anim_path, output_dir, ext=None):"""``<output_dir>/<clip stem>[.<ext>]`` for a driving-animation file."""base = os.path.splitext(os.path.basename(anim_path))[0]return os.path.join(output_dir, base + (f'.{ext}' if ext else ''))+def rotations_to_quats(rotations):+ """``(J, 3, 3)`` rotation matrices -> ``(J, 4)`` wxyz quaternions, through+ mathutils (``Quaternions.from_transforms`` returns the inverse when two+ components tie, e.g. for an exact -90 deg turn)."""+ return np.array([tuple(Matrix(r.tolist()).to_quaternion()) for r in rotations])++def quat_pos_to_mats(q, pos):"""Batch (โฆ, 4) wxyz quaternions + (โฆ, 3) translations -> (โฆ, 4, 4)."""- import numpy as npq = q / np.linalg.norm(q, axis=-1, keepdims=True)w, x, y, z = q[..., 0], q[..., 1], q[..., 2], q[..., 3]m = np.zeros(q.shape[:-1] + (4, 4))@@ -135,14 +252,34 @@ def drive_and_export(char_armature, anim, rest_anim, bone_names, fps,*bone_names* (see :func:`sync_armature_bones`).tpos_global_rot: Optional ``(nbones, 4)`` T-pose global rotations toconjugate keyframes by (``.npy`` model-feature path in animate_motion).+ When the rig's bone axes differ from the motion's+ (:func:`motion_axis_alignment`), the conjugation also re-expresses+ the keys in the rig's axes."""anim_local_mat = transforms_local(anim) # (nframes, nbones, 4, 4)rest_local_mat = transforms_local(rest_anim)[0] # (nbones, 4, 4)sync_armature_bones(char_armature, bone_names,extra_bones_strategy=extra_bones_strategy)+ present = {b.name for b in char_armature.data.bones}+ matched = sum(str(n) in present for n in bone_names)+ if matched < MIN_JOINT_MATCH * len(bone_names):+ raise ValueError(+ f"the character has only {matched} of the motion's {len(bone_names)} joints; "+ f"it is not the character this motion was made for")set_scene_timing(anim_local_mat.shape[0], fps)+ # The rig's bone axes may differ from the motion's (motion_axis_alignment).+ bones = char_armature.data.bones+ tpos_rot = None if tpos_global_rot is None else \+ Quaternions(np.asarray(tpos_global_rot)).transforms()+ conj = motion_axis_alignment(+ transforms_global(rest_anim)[0], bone_names,+ np.stack([np.array(b.matrix_local) for b in bones]), [b.name for b in bones],+ conj_rot=tpos_rot)+ if conj is not None:+ tpos_global_rot = rotations_to_quats(conj)+keyframes = compute_bone_keyframes(rest_local_mat, anim_local_mat, bone_names, tpos_global_rot)rebuild_action_from_data(char_armature, keyframes)diff --git a/data_process/mesh_animation/preprocess_char.py b/data_process/mesh_animation/preprocess_char.pydeleted file mode 100644index 88bacaf..0000000--- a/data_process/mesh_animation/preprocess_char.py+++ /dev/null@@ -1,409 +0,0 @@-"""Preprocess one rigged, animated asset through the real export + feature pipeline.--Runs the actual pipeline stages in-process on a single GLB/FBX:-- 1. EXPORT (:func:`export_general.export_asset`) โ discover the asset's- pose actions, extract rest + animations, motion-driven skeleton pruning,- Z-up -> Y-up; writes export-stage NPZs.- 2. EXTRACT (:func:`motion_features.process_object`) โ canonical T-pose- (BFS order, facing -> center -> diameter scale -> ground), per-clip- motion FEATURES, and the full topology conditioning dict the model- consumes.- 3. BAKE โ rebuild the asset's armature to the canonical T-pose taken- straight from the cond (same joint order, rest, scale), transform the- skinned mesh by the same similarity, and save GLB/FBX. The canonical- joint order rides inside the file as a custom property.--Outputs under ``--output_dir`` (exactly three by default):- <name>_canonical.{glb,fbx} canonical rest-pose asset (drives cond-free)- cond.npy ``{asset_name: topology cond}`` โ model-side input- motions/<clip>.npz motion features (``global_positions`` /- ``local_rotations`` / ``root_facing_quat`` / ``fps``)-Intermediates (export-stage NPZs, T-pose/preview visuals) are cleaned up on-success; pass ``--keep_intermediate`` to keep them (also makes reruns skip-the export stage via its completion markers).--Animate afterwards with ONLY a motion-feature NPZ (generated or extracted):-- python -m data_process.mesh_animation.animate_lbs \\- --char_path <output_dir>/<name>_canonical.glb \\- --anim_path <feature_motion.npz>--Facing canonicalization needs the asset's face-joint pair (``--face_r`` /-``--face_l`` raw bone names); omit for identity facing (pipeline sentinel).--Assets WITHOUT any action are supported too: a rest-only fallback builds the-cond from the rest pose (full armature skeleton โ motion-driven pruning-needs animations โ and no ``motions/`` output).--Usage (Blender headless):- blender -b -P data_process/mesh_animation/preprocess_char.py -- \\- --char_path my_asset.glb --output_dir outputs/my_asset \\- --face_r R_Thigh --face_l L_Thigh --formats glb,fbx-"""--import argparse-import glob-import json-import math-import os-import sys--import bpy-import numpy as np-from loguru import logger-from mathutils import Matrix--sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))--from data_process.joint_annotation.names_clean_rule import ( # noqa: E402- clean_joint_name,- post_process,-)-from data_process.mesh_animation.common import ( # noqa: E402- load_character,- parse_blender_argv,- quat_pos_to_mats,-)-from data_process.motion_export.export_general import export_asset # noqa: E402-from data_process.utils.blender_export import ( # noqa: E402- apply_zup_to_yup,- build_rest_animation,- build_skeleton_arrays,- clear_animation_state,-)-from data_process.utils.blender_rig import ( # noqa: E402- export_selected_to_file,- find_skinned_meshes,- select_objs,- sync_armature_bones,- update_scene,-)-from data_process.utils.motion_features import ( # noqa: E402- build_topology_cond,- process_object,- process_tpose,-)---def _umeyama(X, Y):- """Similarity (s, R, t) with ``Y ~= s * X @ R + t``; returns (4x4, residual)."""- muX, muY = X.mean(0), Y.mean(0)- Xc, Yc = X - muX, Y - muY- U, S, Vt = np.linalg.svd(Xc.T @ Yc)- D = np.eye(3)- D[2, 2] = np.sign(np.linalg.det(U @ Vt))- R = U @ D @ Vt- s = (S * D.diagonal()).sum() / (Xc ** 2).sum()- M = np.eye(4)- M[:3, :3] = s * R.T- M[:3, 3] = muY - s * muX @ R- resid = np.abs(s * X @ R + (muY - s * muX @ R) - Y).max()- return M, resid---# ----------------------------------------------------------------------------# Rest-only fallback: cond from the rest pose alone (asset has no actions)-# -----------------------------------------------------------------------------def cond_from_rest(char_path, name, face_joints=None, target_diameter=2.0):- """Build the topology cond from the asset's rest pose alone.-- Used when the asset carries no animation: the export stage's- motion-driven skeleton pruning cannot run, so the FULL armature skeleton- is kept; everything else (world-applied rest, Z-up -> Y-up, BFS reorder,- facing/scale/ground canonicalization, cond fields) matches the pipeline.- """- from Animation import positions_global, rotations_global-- armature = load_character(char_path)- names, parents, rest_pos, rest_rot, _ = build_skeleton_arrays(- armature, None, apply_world=True)- rest_anim = build_rest_animation(parents, rest_pos, rest_rot, len(names))- _, rest_anim = apply_zup_to_yup(rest_anim, rest_anim)-- tpos_data = {- 'names': np.array(names), 'parents': np.asarray(parents),- 'rest_local_pos': np.asarray(rest_anim.positions[0]),- 'rest_local_rot': np.asarray(rest_anim.rotations.qs[0]),- 'fps': np.array(30),- }- clean = [post_process(clean_joint_name(n, name)) for n in names]-- (canon_anim, offsets, scale_factor, _gh, canon_parents, names_bfs,- _fps, bfs, face_idxs, baxis) = process_tpose(- tpos_data, face_joints=face_joints, target_diameter=target_diameter)- clean_bfs = [clean[i] for i in bfs]-- cond = build_topology_cond(- name, canon_parents, offsets, names_bfs, clean_bfs,- positions_global(canon_anim)[0],- canon_anim.rotations.qs[0],- rotations_global(canon_anim).qs[0],- face_joint_idxs=face_idxs, body_axis=baxis,- scale_factor=scale_factor, ground_height=None)- logger.warning(- f"'{name}' has no animation: canonical skeleton = FULL armature "- f"({len(names_bfs)} joints, no motion-driven pruning) and no motion "- f"features are produced.")- return cond---# ----------------------------------------------------------------------------# Stage 3: bake the canonical rest pose into the asset-# -----------------------------------------------------------------------------def bake_canonical_asset(char_path, cond, out_base, formats=('glb',)):- """Rebuild the asset's armature to the cond's canonical T-pose and save.-- The canonical skeleton (joint order, rest positions/orientations) is- taken verbatim from the topology cond so the baked asset and the motion- features agree by construction. The skinned mesh gets the same- similarity transform, so binding is untouched; asset bones outside the- motion skeleton are weight-merged away first.- """- names = list(cond['joint_names'])- canon_parents = [int(p) for p in cond['parents']]- canon_pos = np.asarray(cond['tpos_first_frame'], dtype=np.float64)- canon_mats = quat_pos_to_mats(- np.asarray(cond['tpos_global_rotations'], dtype=np.float64), canon_pos)-- armature = load_character(char_path)- meshes = find_skinned_meshes(armature)- assert meshes, "No mesh with an Armature modifier targeting the asset's armature"-- # The canonical asset holds ONLY the rigged character: drop unskinned- # props, lights, cameras, helper empties (e.g. the egg Icosphere in- # Truebones' Chicken-EggLaying) โ they can't be driven by the motion and- # would linger as static junk in the export.- keep = {armature.name} | {m.name for m in meshes}- dropped = [o.name for o in bpy.context.scene.objects if o.name not in keep]- for name_ in dropped:- bpy.data.objects.remove(bpy.data.objects[name_], do_unlink=True)- if dropped:- logger.info(f"Dropped {len(dropped)} non-skinned scene object(s): {dropped[:6]}")-- missing = [n for n in names if n not in armature.data.bones]- assert not missing, f"cond joints missing from the asset: {missing}"- sync_armature_bones(armature, names, extra_bones_strategy='merge')-- # Similarity mapping Blender world -> canonical frame: the whole export +- # extract chain is one similarity of the world frame; fit it exactly on- # the bone heads and verify the residual.- arm_world = np.array(armature.matrix_world)- bones = armature.data.bones- world_pos = np.stack([- (arm_world @ np.array(bones[n].matrix_local))[:3, 3] for n in names])- M, _ = _umeyama(world_pos, canon_pos)- check = (np.concatenate([world_pos, np.ones((len(names), 1))], 1) @ M.T)[:, :3]- resid = np.abs(check - canon_pos).max() / max(np.abs(canon_pos).max(), 1e-9)- assert resid < 1e-4, (- f"world->canonical is not a clean similarity (residual {resid:.2e}); "- f"non-uniform armature scale, or the cond belongs to another asset?")- sim_scale = float(np.cbrt(abs(np.linalg.det(M[:3, :3]))))-- # Transform every skinned mesh into the canonical frame. Un-parent BEFORE- # zeroing transforms: the meshes hang under the armature, so identity-ing- # the armature afterwards would shift any still-parented mesh again.- for mesh_obj in meshes:- T = M @ np.array(mesh_obj.matrix_world)- for vert in mesh_obj.data.vertices:- v = T @ np.array([*vert.co, 1.0])- vert.co = v[:3]- mesh_obj.data.update()- mesh_obj.parent = None- mesh_obj.matrix_world = Matrix.Identity(4)-- # Rebuild the armature: canonical rest, canonical (BFS) bone order.- old_lengths = {n: bones[n].length for n in names}- bpy.context.view_layer.objects.active = armature- bpy.ops.object.mode_set(mode='EDIT')- edit_bones = armature.data.edit_bones- for eb in list(edit_bones):- edit_bones.remove(eb)- child_of = {p: j for j, p in enumerate(canon_parents) if p >= 0}- for j, name in enumerate(names):- eb = edit_bones.new(name)- eb.head = (0.0, 0.0, 0.0)- eb.tail = (0.0, 1.0, 0.0)- length = (np.linalg.norm(canon_pos[child_of[j]] - canon_pos[j])- if j in child_of else old_lengths[name] * sim_scale)- eb.matrix = Matrix(canon_mats[j].tolist())- eb.length = max(float(length), 1e-5)- for j, name in enumerate(names):- if canon_parents[j] >= 0:- eb = edit_bones[name]- eb.use_connect = False- eb.parent = edit_bones[names[canon_parents[j]]]- bpy.ops.object.mode_set(mode='OBJECT')- # ARMATURE-SPACE data stays exactly canonical (Y-up feature frame) โ the- # cond-free animate math lives there. The +90ยฐX OBJECT rotation stands- # the asset up in Blender's Z-up world; verified with an independent- # glTF evaluator, the exported file is upright BOTH in standard Y-up- # viewers and when re-imported into Blender.- yup_to_zup = Matrix.Rotation(math.radians(90.0), 4, 'X')- armature.matrix_world = yup_to_zup- for mesh_obj in meshes: # re-parent with identity LOCAL transforms- mesh_obj.parent = armature- mesh_obj.matrix_world = yup_to_zup.copy()-- # Strip animation; store the canonical joint ORDER as an OBJECT-level- # custom property (glTF extras / FBX user properties) โ importers do not- # keep bone enumeration order stable.- clear_animation_state(armature)- for obj in meshes:- if obj.animation_data:- obj.animation_data_clear()- # Unbinding is not enough: the glTF exporter's default animation mode- # exports every matchable ACTION datablock, bound or not โ delete them- # all so the canonical asset is truly rest-only.- for action in list(bpy.data.actions):- bpy.data.actions.remove(action)- update_scene()- armature['canonical_joint_order'] = json.dumps(names)-- select_objs([armature] + meshes, deselect_first=True)- for fmt in formats:- assert fmt in ('glb', 'fbx'), f"unsupported output format: {fmt}"- export_selected_to_file(f'{out_base}.{fmt}', fmt, custom_props=True)---# ----------------------------------------------------------------------------# Full per-asset preprocessing-# -----------------------------------------------------------------------------def preprocess_asset(char_path, output_dir, face_r=None, face_l=None,- body_axis=False, target_diameter=2.0, formats=('glb',),- save_vis=False, apply_clip=False, max_clip_len=200,- clip_stride=110, keep_intermediate=False):- """Export + extract + bake one asset; see the module docstring.-- ``max_clip_len`` / ``clip_stride`` mirror stage 4's ``--max_clip_len`` and- ``max_clip_len - diffusion_max_len``: without ``apply_clip`` a motion is- truncated to ``max_clip_len`` frames, with it the motion is cropped into- overlapping windows of that length.- """- assert os.path.exists(char_path), f"Asset not found: {char_path}"- name = os.path.splitext(os.path.basename(char_path))[0]- os.makedirs(output_dir, exist_ok=True)-- face_joints = None- if face_r and face_l:- face_joints = {'r_hip': {'raw': face_r}, 'l_hip': {'raw': face_l},- 'body_axis': body_axis}- else:- logger.info("No face joints given; identity facing (pipeline sentinel).")-- # --- 1. Export stage (real pipeline; resumes via its own markers) ------- export_dir = os.path.join(output_dir, 'export')- export_asset(char_path, export_dir, save_name=name, save_vis=False)- npzs = sorted(glob.glob(os.path.join(export_dir, 'motions', f'{name}-*.npz')))-- if npzs:- # --- 2a. Feature stage (real pipeline) ------------------------------- logger.info(f"Export stage: {len(npzs)} clip NPZ(s)")- tpos_data = np.load(npzs[0], allow_pickle=True) # first sorted = T-pose ref- raw_names = tpos_data['names'].tolist()- clean_names = [post_process(clean_joint_name(n, name)) for n in raw_names]-- cond, n_clips, n_frames, n_joints, _filtered = process_object(- name, npzs, save_dir=output_dir,- face_joints=face_joints, clean_names=clean_names,- expected_names=raw_names,- apply_clip=apply_clip, max_clip_len=max_clip_len,- clip_stride=clip_stride,- target_diameter=target_diameter, save_vis=save_vis)- assert cond is not None, ("feature extraction skipped this asset "- "(degenerate skeleton or no usable clips โ see log)")- logger.info(f"Feature stage: {n_clips} clip(s), {n_frames} frames, "- f"{n_joints} joints")- else:- # --- 2b. Rest-only fallback: asset carries no animation -------------- cond = cond_from_rest(char_path, name, face_joints=face_joints,- target_diameter=target_diameter)-- cond_path = os.path.join(output_dir, 'cond.npy')- np.save(cond_path, {name: cond})- logger.info(f"Saved cond -> {cond_path}")-- # --- 3. Bake the canonical rest pose into the asset ---------------------- bake_canonical_asset(char_path, cond,- os.path.join(output_dir, f'{name}_canonical'),- formats=formats)-- # --- 4. Keep only the three deliverables --------------------------------- if not keep_intermediate:- import shutil- for sub in ('export', 'tpose', 'videos', 'animations'):- shutil.rmtree(os.path.join(output_dir, sub), ignore_errors=True)- motions_dir = os.path.join(output_dir, 'motions')- if os.path.isdir(motions_dir) and not os.listdir(motions_dir):- os.rmdir(motions_dir) # rest-only run: no motion features- logger.info("Removed intermediates; kept: "- f"{name}_canonical.{{{','.join(formats)}}}, cond.npy, motions/")- return cond---def parse_args():- parser = argparse.ArgumentParser(- description="Preprocess a rigged, animated asset: export + feature "- "extraction + canonical rest-pose bake.")- parser.add_argument("--char_path", type=str, required=True,- help="Rigged asset GLB/FBX (mesh + armature + weights). "- "Assets WITH actions get the full pipeline incl. "- "motion-driven skeleton pruning and motion features; "- "action-less assets fall back to a rest-only cond "- "(full skeleton, no motions/).")- parser.add_argument("--output_dir", type=str, required=True,- help="Output directory (export/, motions/, cond.npy, "- "<name>_canonical.{glb,fbx}).")- parser.add_argument("--face_r", type=str, default=None,- help="Raw name of the right face joint (facing canonicalization).")- parser.add_argument("--face_l", type=str, default=None,- help="Raw name of the left face joint.")- parser.add_argument("--body_axis", action='store_true',- help="Face pair is a head/tail body axis (serpentine rigs).")- parser.add_argument("--target_diameter", type=float, default=2.0,- help="Canonical skeleton diameter (pipeline default: 2.0).")- parser.add_argument("--formats", type=str, default='glb',- help="Comma-separated canonical-asset formats (glb,fbx).")- parser.add_argument("--save_vis", action='store_true',- help="Also render per-clip preview MP4s (slow).")- parser.add_argument("--apply_clip", action='store_true',- help="Crop motions into overlapping windows of "- "--max_clip_len frames (stride max_clip_len - "- "diffusion_max_len) instead of truncating each "- "motion to --max_clip_len frames.")- parser.add_argument("--max_clip_len", type=int, default=200,- help="Frames kept per motion / window length "- "(stage-4 default: 200).")- parser.add_argument("--diffusion_max_len", type=int, default=90,- help="Training crop length used to derive the window "- "stride max_clip_len - diffusion_max_len "- "(stage-4 default: 90).")- parser.add_argument("--keep_intermediate", action='store_true',- help="Keep export/ NPZs and T-pose/preview visuals (default: "- "only canonical asset + cond.npy + motions/ remain).")- return parse_blender_argv(parser)---if __name__ == "__main__":- args = parse_args()- if args.apply_clip and args.max_clip_len <= args.diffusion_max_len:- raise SystemExit(- f"--apply_clip needs max_clip_len ({args.max_clip_len}) > "- f"diffusion_max_len ({args.diffusion_max_len}) for a positive stride")- preprocess_asset(- char_path=args.char_path,- output_dir=args.output_dir,- face_r=args.face_r,- face_l=args.face_l,- body_axis=args.body_axis,- target_diameter=args.target_diameter,- formats=tuple(f.strip().lower() for f in args.formats.split(',') if f.strip()),- save_vis=args.save_vis,- apply_clip=args.apply_clip,- max_clip_len=args.max_clip_len,- clip_stride=args.max_clip_len - args.diffusion_max_len,- keep_intermediate=args.keep_intermediate,- )diff --git a/data_process/mesh_animation/sample_manifest.py b/data_process/mesh_animation/sample_manifest.pynew file mode 100644index 0000000..a980b29--- /dev/null+++ b/data_process/mesh_animation/sample_manifest.py@@ -0,0 +1,206 @@+"""Mesh-driving jobs from a sampler run's ``manifest.json`` (bpy-free).++``python -m unimate.inference.sample`` writes ``<samples>/manifest.json``+beside ``<samples>/motions/*.npy``: for every motion the asset it was+generated for (cond file and key, joint order, canonical GLB). This module+turns it into one job per motion for ``animate_motion.py`` and checks, before+any Blender process starts, that the three still belong together:++- the cond file still holds the asset's entry with the recorded joint order+ (a cond rebuilt since sampling can reorder or rename joints);+- the canonical GLB stores that same joint order (``canonical_joint_order``);+- the motion has as many joints.++A mismatch is an error rather than a silently scrambled animation.++ python -m data_process.mesh_animation.sample_manifest <samples_dir | manifest.json> \\+ [--include_gt] [--char_path FILE] [--character NAME] [--skip_invalid]++prints ``anim_path<TAB>char_path<TAB>cond_path<TAB>dataset_type`` per job+(``scripts/run_animate_motion.sh <samples_dir>`` runs them) and exits 1 on any+problem; with ``--skip_invalid`` the problems are warnings and the other+motions are still listed. A samples directory without a manifest of its own+but exactly one subdirectory with one (``inbetween/``, ``motion_edit/``,+``motion_expand/``) uses that one. Paths in the manifest are relative to the+directory sampling ran in (the repo root).+"""++import argparse+import json+import os+import sys++import numpy as np++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.utils.asset_files import ( # noqa: E402+ DEFAULT_MIXAMO_CHARACTER, default_assets_dir, glb_joint_order)++MANIFEST_FILE = 'manifest.json'+FORMAT_PREFIX = 'unimate-samples/'+# dataset_type animate_motion.py takes for an asset that is no dataset object+# type: the object type is then the motion file's prefix, and the cond file+# holds that key (or a single entry).+CUSTOM_DATASET_TYPE = 'general'+++def manifest_path(path):+ """``<path>/manifest.json`` for a samples directory (or the manifest of its+ one mode subdirectory that has one), else *path*."""+ if not os.path.isdir(path):+ return path+ own = os.path.join(path, MANIFEST_FILE)+ if os.path.isfile(own):+ return own+ subs = sorted(os.path.join(path, d, MANIFEST_FILE) for d in os.listdir(path)+ if os.path.isfile(os.path.join(path, d, MANIFEST_FILE)))+ if len(subs) > 1:+ raise ValueError(f"several manifests under {path}: pass one of "+ f"{', '.join(os.path.dirname(p) for p in subs)}")+ return subs[0] if subs else own+++def load_manifest(path):+ path = manifest_path(path)+ with open(path) as f:+ data = json.load(f)+ if not str(data.get('format', '')).startswith(FORMAT_PREFIX):+ raise ValueError(f"{path} is not a sampler manifest (format {data.get('format')!r}).")+ return path, data+++def build_jobs(path, include_gt=False, char_path=None, character=None):+ """``(jobs, problems)``: one ``(anim, char, cond, dataset_type)`` per motion+ of the manifest at *path*, and every reason one cannot be driven.+ *char_path* drives every motion with that file instead of the asset's+ canonical GLB (only its joint names are then checked, by animate_motion);+ *character* picks the Mixamo character whose canonical GLB drives the+ ``mixamo`` asset (the file beside the recorded one)."""+ mpath, data = load_manifest(path)+ motions_dir = os.path.join(os.path.dirname(os.path.abspath(mpath)),+ data.get('motions_dir', 'motions'))+ assets = data.get('assets', {})+ jobs, problems = [], []+ checked = {}+ conds = {}++ def asset_glb(info):+ glb = info.get('canonical_glb')+ if info.get('dataset') == 'mixamo' and (character or not glb):+ root = (os.path.dirname(glb) if glb else+ default_assets_dir(os.path.dirname(info.get('cond_path') or+ os.path.join('dataset', 'features', 'mixamo', 'cond.npy'))))+ glb = os.path.join(root, f'{character or DEFAULT_MIXAMO_CHARACTER}.glb')+ return glb++ def load_cond(path):+ if path not in conds:+ conds[path] = np.load(path, allow_pickle=True).item()+ return conds[path]++ def check_asset(name):+ if name in checked:+ return checked[name]+ info = assets.get(name)+ issues = []+ if info is None:+ issues.append(f"asset {name!r} is not described in the manifest")+ checked[name] = issues+ return issues+ names = [str(n) for n in info.get('joint_names', [])]+ cond_path = info.get('cond_path')+ if not cond_path or not os.path.isfile(cond_path):+ issues.append(f"{name}: cond file {cond_path!r} not found")+ else:+ entry = load_cond(cond_path).get(info.get('cond_key', name))+ if entry is None:+ issues.append(f"{name}: {cond_path} no longer holds {info.get('cond_key')!r}")+ elif [str(n) for n in entry['joint_names']] != names:+ issues.append(f"{name}: {cond_path} changed since sampling (joint order "+ f"differs); sample again")+ if char_path is None:+ glb = asset_glb(info)+ if not glb or not os.path.isfile(glb):+ no_dir = not (glb and os.path.isdir(os.path.dirname(glb)))+ hint = ("build the canonical assets first (stage 4 with SAVE_GLB=1, or "+ "feature_extraction/canonical_assets.py)" if no_dir else+ f"CHARACTER={character or DEFAULT_MIXAMO_CHARACTER} is not among the "+ f"Mixamo canonical assets" if info.get('dataset') == 'mixamo' else+ "pass a character (CHAR_PATH) or build it (rig_preprocess, or "+ "stage 4 with SAVE_GLB=1)")+ issues.append(f"{name}: no canonical GLB ({glb!r}); {hint}")+ else:+ order = glb_joint_order(glb)+ if order is None:+ issues.append(f"{name}: {glb} carries no canonical joint order")+ elif order != names:+ issues.append(f"{name}: {glb} was baked from another cond (joint order "+ f"differs from the sampled one)")+ checked[name] = issues+ return issues++ for npy, sample in sorted(data.get('samples', {}).items()):+ if sample.get('kind', 'sample') != 'sample' and not include_gt:+ continue+ name = sample.get('asset')+ issues = list(check_asset(name))+ anim = os.path.join(motions_dir, npy)+ if not os.path.isfile(anim):+ issues.append(f"{npy}: motion file not found in {motions_dir}")+ elif not issues:+ n_motion = np.load(anim, mmap_mode='r').shape[1]+ n_asset = len(assets[name]['joint_names'])+ if n_motion != n_asset:+ issues.append(f"{npy}: {n_motion} joints, asset {name!r} has {n_asset}")+ if issues:+ problems.extend(issues)+ continue+ info = assets[name]+ jobs.append((anim, char_path or asset_glb(info), info['cond_path'],+ info.get('dataset') or CUSTOM_DATASET_TYPE))+ return jobs, sorted(set(problems))+++def main(argv=None):+ ap = argparse.ArgumentParser(description="List (and check) the mesh-driving jobs of a "+ "sampler run's manifest.json.")+ ap.add_argument('path', help="Samples directory (holding manifest.json) or the manifest.")+ ap.add_argument('--include_gt', action='store_true',+ help="Also drive the saved ground truth of in-betweening / motion editing.")+ ap.add_argument('--char_path', default=None,+ help="Drive every motion with this character instead of each asset's "+ "canonical GLB.")+ ap.add_argument('--character', default=None,+ help="Mixamo character to drive the mixamo motions with (its canonical GLB "+ "beside the recorded one), instead of the default.")+ ap.add_argument('--skip_invalid', action='store_true',+ help="Report a motion that cannot be driven and list the others.")+ args = ap.parse_args(argv)+ try:+ jobs, problems = build_jobs(args.path, args.include_gt, args.char_path, args.character)+ except (OSError, ValueError) as exc:+ print(f"error: {exc}", file=sys.stderr)+ return 1+ if args.character and args.char_path:+ print(f"warning: --character {args.character} is ignored: --char_path drives every "+ f"motion", file=sys.stderr)+ elif args.character and not any(info.get('dataset') == 'mixamo'+ for info in load_manifest(args.path)[1].get('assets', {}).values()):+ print(f"warning: --character {args.character} is ignored: no mixamo motion here",+ file=sys.stderr)+ level = 'warning' if args.skip_invalid else 'error'+ for problem in problems:+ print(f"{level}: {problem}", file=sys.stderr)+ if problems and not args.skip_invalid:+ return 1+ if not jobs:+ print(f"error: no motions to drive in {manifest_path(args.path)}", file=sys.stderr)+ return 1+ for job in jobs:+ print('\t'.join(job))+ return 0+++if __name__ == '__main__':+ sys.exit(main())diff --git a/data_process/motion_export/export_general.py b/data_process/motion_export/export_general.pyindex 55d2224..0902418 100644--- a/data_process/motion_export/export_general.py+++ b/data_process/motion_export/export_general.py@@ -1,25 +1,46 @@"""Export rigged assets (GLB/GLTF or FBX) to NPZ motion data.-Generic counterpart to the per-dataset batch exporters: ``--input`` is either-a single file or a directory of ``.glb``/``.gltf``/``.fbx`` assets (mixed-formats welcome). The importer is picked per file by extension, the character-mesh is optional, and every pose action found in a file is exported as its-own clip. Directory runs mirror the Objaverse exporter: per-asset completion-markers for resume, worker sharding, a per-worker error log, and per-worker-summary shards (merge with ``data_process.tools.merge_summaries``).+The exporter of the ``general`` dataset (extra training assets in+``dataset/raw/general/animation/``, see ``run_export.sh general``) and of any single+asset: ``--input`` is either a single file or a directory of+``.glb``/``.gltf``/``.fbx`` assets (mixed formats welcome). The importer is+picked per file by extension, the character mesh is optional, and every pose+action found in a file is exported as its own clip. Directory runs mirror the+Objaverse exporter: per-asset completion markers for resume, worker sharding,+a per-worker error log, per-worker summary shards (merge with+``data_process.tools.merge_summaries``), and assets listed in an+``excluded.csv`` beside or inside the directory are skipped.++``--save_glb`` also writes ``rigs/<asset>.glb`` per asset: the asset in its+rest pose on the pruned skeleton, without animation+(``processed_assets.export_asset_glb``); every clip NPZ of the asset drives it as a+character (``mesh_animation``). On an existing export it only adds the missing+GLBs (the NPZs are not rewritten). ``--glb_only`` does only that, also for an+export without completion markers (the released ones): it builds the missing+GLBs of the assets whose NPZs are already in ``motions/`` and writes nothing+else.++Asset names: clips are ``{asset}-{action}`` and the object type is the text+before the first ``-``, so the asset name is the file stem with ``-`` and+whitespace replaced by ``_`` (``blender_export.asset_name``); two files that+map to one name stop a directory run before anything is exported.Pipeline: import scene โ select armature (+ mesh if present) โ discoveractions โ build skeleton arrays โ extract animations โ optional skeletonpruning โ coordinate conversion โ save per-action NPZ + MP4 visualization.Behavior notes:- - Mesh optional: with a mesh, skin weights are extracted and skeleton- pruning is enabled by default; without one, ``skin_matrix`` is saved+ - Mesh optional: with a mesh, skin weights are extracted from every mesh+ the armature deforms (``find_skinned_meshes``) and skeleton pruning is+ enabled by default; without one, ``skin_matrix`` is savedas an empty ``(0, nbones)`` array and pruning is skipped (the pruningpasses rely on skin weights to decide which bones matter).- - Actions: all relevant pose actions are exported. Files whose action+ - Actions: all relevant pose actions are exported, named after their+ glTF animation (the importer's ``_<armature>`` suffix is stripped, so+ ``eve.glb``'s animation ``alert`` is the clip ``eve-alert``). Files whose actiondoes not pass the pose-action filter (e.g. Mixamo animation-only- clips) fall back to the armature's single bound action.+ clips) fall back to the armature's single bound action+ (``blender_export.discover_clip_actions``, shared with render_general).- No joint-count filtering: stage 4 (feature_extraction --min_joints /--max_joints) decides which skeletons enter training.@@ -34,6 +55,7 @@ Usage (pip-installed bpy):import bpyimport sys+import jsonimport osimport numpy as npimport argparse@@ -44,16 +66,23 @@ from loguru import loggersys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))from data_process.utils.blender_export import (- import_fbx, import_gltf, sanitize_action_name, unique_clip_names,+ GENERAL_ASSET_EXTS, import_fbx, import_gltf, sanitize_action_name, unique_clip_names,load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,- prune_skeleton_shared, remove_tpose_frames, discover_pose_actions,- action_frame_range, save_motion, write_export_summary,- is_asset_complete, mark_asset_complete,+ prune_skeleton_shared, remove_tpose_frames, discover_clip_actions, find_skinned_meshes,+ save_motion, write_export_summary, asset_name, asset_names,+ list_asset_files, is_asset_complete, mark_asset_complete,)+from data_process.utils.asset_files import (+ REPLACED_DIR, asset_glb_missing, remove_asset_glb, report_glb_errors, set_aside_old_clips,+)+from data_process.utils.processed_assets import add_asset_glb, build_asset_glbGLTF_EXTS = {'.glb', '.gltf'}FBX_EXTS = {'.fbx'}-ASSET_EXTS = GLTF_EXTS | FBX_EXTS+ASSET_EXTS = set(GENERAL_ASSET_EXTS)+# Leaf bones named like this are a link's end point (a foot or tool tip),+# kept when their parent rotates, not control bones.+END_EFFECTOR_SUFFIXES = ('_tip',)def pick_importer(path):@@ -67,41 +96,60 @@ def pick_importer(path):f"{sorted(ASSET_EXTS)}): {path}")-def list_asset_files(input_dir):- """List .glb/.gltf/.fbx files directly under *input_dir*, sorted."""- input_dir = Path(input_dir)- return sorted(p for p in input_dir.iterdir()- if p.is_file() and p.suffix.lower() in ASSET_EXTS)--def export_asset(input_path, output_dir, save_name=None, fps=30,dtype=np.float64, prune=None, consider_parent_rotate=True,- min_frames=5, remove_tpose=True, save_vis=True):+ min_frames=5, remove_tpose=True, save_vis=True, save_glb=False,+ glb_only=False):"""Export every pose action in a single GLB/GLTF/FBX file to NPZ clips.Args:+ save_name: Asset name / clip prefix (default: ``asset_name(input_path)``,+ the stem with ``-`` and whitespace replaced by ``_``). Must not+ contain ``-``.prune: True/False to force skeleton pruning on/off; None (default)enables it iff the file contains a skinned mesh.+ save_glb: Also write the asset's processed GLB (module docstring).+ glb_only: Only write that, for an asset already exported.Returns:{save_name: joint_names} when at least one clip was saved, else {}."""assert os.path.exists(input_path), f"Input file not found: {input_path}"- save_name = save_name or Path(input_path).stem+ save_name = save_name or asset_name(input_path)+ if '-' in save_name:+ raise ValueError(f"Asset name '{save_name}' contains '-', the separator between "+ f"asset and action in clip names; pick another name.")++ if glb_only:+ # Existing NPZs only, markers or not; never (re-)exports.+ if asset_glb_missing(output_dir, save_name):+ add_asset_glb(save_name, output_dir,+ lambda: load_scene(pick_importer(input_path), input_path, fps=fps)[0],+ source_path=input_path)+ return {}motion_save_dir = os.path.join(output_dir, "motions")vis_save_dir = os.path.join(output_dir, "videos")tpos_save_dir = os.path.join(output_dir, "tpose")if is_asset_complete(output_dir, save_name):- logger.info(f"Asset '{save_name}' already complete, skipping: {input_path}")+ if not (save_glb and asset_glb_missing(output_dir, save_name)):+ logger.info(f"Asset '{save_name}' already complete, skipping: {input_path}")+ return {}+ logger.info(f"Asset '{save_name}' already exported; adding its processed GLB")+ add_asset_glb(save_name, output_dir,+ lambda: load_scene(pick_importer(input_path), input_path, fps=fps)[0],+ source_path=input_path)return {}+ remove_asset_glb(output_dir, save_name)existing = sorted(Path(motion_save_dir).glob(f"{save_name}-*.npz")) if os.path.isdir(motion_save_dir) else []if existing:# Pruning is joint across all of an asset's clips, so a partial# (or pre-marker) export must be redone as a whole.logger.warning(f"'{save_name}' has {len(existing)} clip(s) but no completion "- f"marker (partial or pre-marker export); re-exporting.")+ f"marker (partial or pre-marker export); re-exporting; the old clips go to "+ f"{REPLACED_DIR}/.")+ set_aside_old_clips(output_dir, save_name)logger.info(f"Processing asset: {input_path}")os.makedirs(motion_save_dir, exist_ok=True)@@ -116,13 +164,7 @@ def export_asset(input_path, output_dir, save_name=None, fps=30,# All relevant pose actions; fall back to the armature's bound action# for single-action files that the pose filter rejects.- action_names, frame_ranges = discover_pose_actions()- if not action_names and armature.animation_data is not None \- and armature.animation_data.action is not None:- action = armature.animation_data.action- action_names = [action.name]- frame_ranges = {action.name: action_frame_range(action)}- logger.info(f"No pose actions discovered; falling back to bound action '{action.name}'.")+ action_names, frame_ranges = discover_clip_actions(armature)logger.info(f"Total actions to export: {len(action_names)}")if not action_names:logger.info(f"No actions in {input_path}; skipping.")@@ -130,7 +172,11 @@ def export_asset(input_path, output_dir, save_name=None, fps=30,reason="no pose actions")return {}- skel = prepare_skeleton(armature, mesh, dtype=dtype, apply_world=True)+ # Skin weights of every mesh the armature deforms, not only the largest:+ # a rig built from one skinned mesh per part would otherwise look+ # unskinned and lose its moving links to pruning.+ skin_meshes = find_skinned_meshes(armature) or ([mesh] if mesh is not None else [])+ skel = prepare_skeleton(armature, skin_meshes or None, dtype=dtype, apply_world=True)extracted = extract_all_actions(armature, {n: frame_ranges[n] for n in action_names},skel['rest_local_pos'], skel['parents_array'],@@ -145,6 +191,7 @@ def export_asset(input_path, output_dir, save_name=None, fps=30,anims_list, rest_anim_shared, names, skin_matrix = prune_skeleton_shared(anims_list, rest_anim_shared, names, skin_matrix.copy(),consider_parent_rotate=consider_parent_rotate,+ end_effector_suffixes=END_EFFECTOR_SUFFIXES,)save_rest_pose_vis(tpos_save_dir, save_name, rest_anim_shared)@@ -182,6 +229,15 @@ def export_asset(input_path, output_dir, save_name=None, fps=30,n_clips=len(saved_clips),reason=None if saved_clips else "no clips survived filtering",joint_names=names if saved_clips else None)+ # After the marker, from a clip this run saved and a fresh import (the+ # scene now sits at the last extracted frame, and an animated armature+ # object or parent empty would carry that pose): exactly what --glb_only+ # builds. A failure is recorded, not raised.+ if save_glb and saved_clips:+ build_asset_glb(save_name, os.path.join(motion_save_dir, f"{saved_clips[0]}.npz"),+ output_dir,+ lambda: load_scene(pick_importer(input_path), input_path, fps=fps)[0],+ source_path=input_path)logger.info(f"Saved {len(saved_clips)} clip(s) for '{save_name}'")if saved_clips:return {save_name: names}@@ -192,6 +248,29 @@ def export_asset(input_path, output_dir, save_name=None, fps=30,# Entry point# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+def check_other_datasets(names, output_dir):+ """Refuse asset names another training dataset already uses, before any+ work: an export into the general dataset (``<export>/general``) whose+ asset is a Truebones or Objaverse object type (their ``joint_names.json``+ beside it), or ``mixamo`` / ``mixamo_*`` (Mixamo's clip keys are+ ``mixamo_<clip>``), would collide in the training loader's one namespace."""+ root, ds = os.path.split(os.path.abspath(output_dir.rstrip(os.sep)))+ if ds != 'general':+ return+ taken = {}+ for other in ('truebones', 'objaverse'):+ path = os.path.join(root, other, 'joint_names.json')+ if os.path.isfile(path):+ with open(path) as f:+ taken.update({n: other for n in json.load(f)})+ clashes = sorted(f"{n} ({taken[n]})" if n in taken else f"{n} (mixamo clip keys)"+ for n in set(names)+ if n in taken or n == 'mixamo' or n.startswith('mixamo_'))+ if clashes:+ raise ValueError(f"{len(clashes)} general asset name(s) are used by another dataset "+ f"(rename the files): {clashes[:10]}")++def parse_args():parser = argparse.ArgumentParser(description="Export rigged GLB/GLTF/FBX assets (a file or a directory) "@@ -201,8 +280,9 @@ def parse_args():parser.add_argument('--output_dir', type=str, required=True,help='Output directory (motions/, videos/, tpose/ created inside).')parser.add_argument('--name', type=str, default=None,- help='Save name / clip prefix (single-file input only; '- 'default: input file stem).')+ help="Save name / clip prefix (single-file input only; "+ "default: input file stem with '-' and whitespace "+ "replaced by '_').")parser.add_argument('--fps', type=int, default=30)parser.add_argument('--prune', action=argparse.BooleanOptionalAction, default=None,help='Force skeleton pruning on/off '@@ -216,6 +296,14 @@ def parse_args():help='Keep frames matching the rest pose instead of removing them.')parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,help='Render the per-clip MP4 preview (use --no-vis for bulk runs).')+ parser.add_argument('--save_glb', action='store_true',+ help="Also write rigs/<asset>.glb: the asset in its rest pose on the "+ "pruned skeleton (no animation; the clip NPZs drive it), for "+ "mesh_animation. On an existing export, only the missing GLBs are added.")+ parser.add_argument('--glb_only', action='store_true',+ help="Only add missing rigs/<asset>.glb to an existing export, for the "+ "assets whose NPZs are already in motions/: no NPZ, completion marker "+ "or summary JSON is written, and none is required.")parser.add_argument('--worker_id', type=int, default=0,help='Worker index for sharding files across workers (directory input).')parser.add_argument('--num_workers', type=int, default=1,@@ -232,13 +320,16 @@ def main():fps=args.fps, prune=args.prune,consider_parent_rotate=args.consider_parent_rotate,min_frames=args.min_frames, remove_tpose=not args.keep_tpose_frames,- save_vis=args.vis,+ save_vis=args.vis, save_glb=args.save_glb, glb_only=args.glb_only,)if os.path.isdir(args.input):asset_paths = list_asset_files(args.input)assert asset_paths, f"No {sorted(ASSET_EXTS)} files found in {args.input}"logger.info(f"Found {len(asset_paths)} asset files in {args.input}")+ # Checked over the full list, before sharding, so every worker agrees.+ names = asset_names(asset_paths)+ check_other_datasets(names.values(), args.output_dir)if args.num_workers > 1:asset_paths = asset_paths[args.worker_id::args.num_workers]@@ -253,7 +344,7 @@ def main():for asset_path in tqdm(asset_paths, desc="Exporting assets"):try:saved = export_asset(str(asset_path), args.output_dir,- **export_kwargs)+ save_name=names[asset_path], **export_kwargs)if saved:all_joint_names.update(saved)except Exception as e: # noqa: BLE001 โ keep the batch going@@ -262,22 +353,28 @@ def main():with open(error_log, 'a') as log_file:log_file.write(f"Failed to export {asset_path}: {e}\n")- worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""- write_export_summary(args.output_dir, all_joint_names, fps=args.fps,- worker_suffix=worker_suffix)+ if not args.glb_only:+ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,+ worker_suffix=worker_suffix)# Summaries are written first so a partial batch still leaves the# canonical JSONs consistent; the non-zero exit then tells the# wrapper not to merge shards and Slurm afterok chains to stop.+ if args.save_glb or args.glb_only:+ report_glb_errors(args.output_dir)if n_failed:logger.error(f"{n_failed}/{len(asset_paths)} assets failed; "f"see {error_log}")sys.exit(1)else:+ check_other_datasets([args.name or asset_name(args.input)], args.output_dir)saved = export_asset(args.input, args.output_dir, save_name=args.name,**export_kwargs)- if saved:+ if saved and not args.glb_only:write_export_summary(args.output_dir, saved, fps=args.fps)+ if args.save_glb or args.glb_only:+ report_glb_errors(args.output_dir)if __name__ == "__main__":diff --git a/data_process/motion_export/export_mixamo.py b/data_process/motion_export/export_mixamo.pyindex dbc67a6..ea82a51 100644--- a/data_process/motion_export/export_mixamo.py+++ b/data_process/motion_export/export_mixamo.py@@ -15,6 +15,18 @@ JSONs (``*_worker{i}.json``) โ merge them afterwards with``python -m data_process.tools.merge_summaries`` (the``run_export.sh`` wrapper does both automatically).+``--save_glb`` (with ``--char_dir``, the characters: one rigged FBX/GLB each,+e.g. ``dataset/raw/mixamo/character_refined``) also writes+``rigs/<character>.glb`` per character: the character in its own rest pose+on the export skeleton (the bones of every clip NPZ; extra bones such as eyes+merged away), without animation (``processed_assets.export_asset_glb``), so any+clip NPZ drives it. A character must have the 22 core joints stage 4 keeps+(``MIXAMO_CORE_JOINTS``); missing finger or end bones are logged, a missing+core joint is recorded in ``rigs/glb_errors/`` (the non-Mixamo-named rigs,+e.g. a 3ds Max Biped). ``--glb_only`` writes only the missing character GLBs,+from the NPZs already in ``motions/``, sharded over the workers, and nothing+else. Existing GLBs are kept.+Usage (Blender headless):blender -b -P data_process/motion_export/export_mixamo.py -- \--anim_dir dataset/raw/mixamo/animation_motion \@@ -32,11 +44,16 @@ from loguru import loggersys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))+from data_process.feature_extraction.metadata import MIXAMO_CORE_JOINTSfrom data_process.utils.blender_export import (- import_fbx,+ asset_names, import_fbx, import_gltf, list_asset_files,load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,action_frame_range, save_motion, write_export_summary,)+from data_process.utils.asset_files import (+ processed_glb_path, record_glb_error, report_glb_errors,+)+from data_process.utils.processed_assets import build_asset_glb# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ@@ -106,6 +123,57 @@ def export_mixamo(anim_path, output_dir, fps=30, dtype=np.float64,return list(skel['bone_names']), skel['rest_anim_shared']+# A character standing more than this factor taller or shorter than the export+# skeleton was authored in other units and is scaled to it; real characters+# stand about 0.7x (Mousey) to 1.25x its height.+CHARACTER_RESCALE_TOL = 1.8+++def save_character_glbs(char_dir, output_dir, fps=30, worker_id=0, num_workers=1,+ reference=None):+ """``rigs/<character>.glb`` for every character in *char_dir* that has+ none (module docstring); failures are recorded, not raised.++ *reference* is the clip NPZ giving the export skeleton (default: the+ first in ``motions/``; a worker of a running export passes one it wrote+ itself, since another worker's NPZ may still be being written).+ """+ if reference is None:+ motion_dir = os.path.join(output_dir, "motions")+ npzs = sorted(f for f in os.listdir(motion_dir) if f.endswith(".npz")) \+ if os.path.isdir(motion_dir) else []+ if not npzs:+ logger.warning(f"No clip NPZ in {motion_dir} to take the export skeleton from; "+ f"character GLBs skipped.")+ return+ reference = os.path.join(motion_dir, npzs[0]) # every clip shares the skeleton+ if not os.path.isdir(char_dir):+ record_glb_error(output_dir, "_characters",+ f"Character directory {char_dir} does not exist; no character GLBs.")+ return++ char_paths = list_asset_files(char_dir)+ try:+ names = asset_names(char_paths) # a name clash, before sharding+ except ValueError as exc:+ record_glb_error(output_dir, "_characters", f"{char_dir}: {exc}")+ return+ record_glb_error(output_dir, "_characters", None)+ char_paths = char_paths[worker_id::num_workers]+ logger.info(f"Character GLBs: {len(char_paths)} character(s) in {char_dir}"+ + (f" (worker {worker_id}/{num_workers})" if num_workers > 1 else ""))+ for path in char_paths:+ name = names[path]+ if os.path.isfile(processed_glb_path(output_dir, name)):+ continue+ importer = import_fbx if path.suffix.lower() == ".fbx" else import_gltf+ build_asset_glb(name, reference, output_dir,+ lambda: load_scene(importer, path, fps=fps)[0],+ source_path=path, required=MIXAMO_CORE_JOINTS,+ rest_tol=None, # each character keeps its own proportions+ rescale_tol=CHARACTER_RESCALE_TOL)++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ# Entry point# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ@@ -127,12 +195,27 @@ def parse_args():help="Total number of workers.")parser.add_argument("--vis", action=argparse.BooleanOptionalAction, default=True,help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")+ parser.add_argument("--char_dir", type=str, default=None,+ help="Characters (one rigged FBX/GLB each) for --save_glb / --glb_only.")+ parser.add_argument("--save_glb", action="store_true",+ help="Also write rigs/<character>.glb per character in --char_dir: the "+ "character in its rest pose on the export skeleton, no animation.")+ parser.add_argument("--glb_only", action="store_true",+ help="Only write the missing character GLBs (from the NPZs already in "+ "motions/); no NPZ or summary JSON is written.")argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]return parser.parse_args(argv)def main(args):+ if (args.save_glb or args.glb_only) and not args.char_dir:+ raise SystemExit("--save_glb / --glb_only need --char_dir (the characters).")os.makedirs(args.output_dir, exist_ok=True)+ if args.glb_only:+ save_character_glbs(args.char_dir, args.output_dir, fps=args.fps,+ worker_id=args.worker_id, num_workers=args.num_workers)+ report_glb_errors(args.output_dir)+ returnworker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""error_log = os.path.join(args.output_dir, f"export_errors{worker_suffix}.log")@@ -157,6 +240,7 @@ def main(args):os.path.exists(joint_names_path) and os.path.exists(tpos_path))ref_bone_names = None+ own_npz = None # a clip this worker wrote: --save_glb's skeleton referenceref_rest_pos = Noneref_rest_rot = Nonetpos_mismatches = []@@ -171,6 +255,9 @@ def main(args):continuebone_names, rest_anim = result+ if own_npz is None:+ stem = os.path.splitext(os.path.basename(anim_path))[0]+ own_npz = os.path.join(args.output_dir, "motions", f"{stem}.npz")rest_pos = rest_anim.positions[0] # (nbones, 3)rest_rot = rest_anim.rotations[0].qs # (nbones, 4)@@ -220,6 +307,12 @@ def main(args):write_export_summary(args.output_dir, all_joint_names, fps=args.fps,worker_suffix=worker_suffix)+ if args.save_glb:+ save_character_glbs(args.char_dir, args.output_dir, fps=args.fps,+ worker_id=args.worker_id, num_workers=args.num_workers,+ reference=own_npz if own_npz and os.path.isfile(own_npz) else None)+ report_glb_errors(args.output_dir)+if n_failed:logger.error(f"{n_failed}/{len(anim_paths)} clips failed; see {error_log}")sys.exit(1)diff --git a/data_process/motion_export/export_objaverse.py b/data_process/motion_export/export_objaverse.pyindex 7493cc2..181f85a 100644--- a/data_process/motion_export/export_objaverse.py+++ b/data_process/motion_export/export_objaverse.py@@ -8,6 +8,15 @@ Files can be sharded across parallel Blender processes with``--worker_id`` / ``--num_workers``; each worker writes its own summaryJSONs (``*_worker{i}.json``).+``--save_glb`` also writes ``rigs/<asset>.glb`` per asset: the asset in its+rest pose on the pruned skeleton, without animation+(``processed_assets.export_asset_glb``); every clip NPZ of the asset drives it as a+character (``mesh_animation``). On an existing export it only adds the missing+GLBs (the NPZs are not rewritten). ``--glb_only`` does only that, also for an+export without completion markers (the released ones): it builds the missing+GLBs of the assets whose NPZs are already in ``motions/`` and writes nothing+else.+Usage (Blender headless):blender -b -P data_process/motion_export/export_objaverse.py -- \--data_dir dataset/raw/objaverse/glb \@@ -26,35 +35,62 @@ from tqdm import tqdmsys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))from data_process.utils.blender_export import (+ asset_names,import_gltf, list_gltf_files, sanitize_action_name, unique_clip_names,load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,prune_skeleton_shared, remove_tpose_frames, discover_pose_actions,save_motion, write_export_summary,is_asset_complete, mark_asset_complete,)+from data_process.utils.asset_files import (+ REPLACED_DIR, asset_glb_missing, remove_asset_glb, report_glb_errors, set_aside_old_clips,+)+from data_process.utils.processed_assets import add_asset_glb, build_asset_glb# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ# Main export pipeline# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโdef export_objaverse(gltf_path, save_name, output_dir, fps=30, dtype=np.float64,- min_frames=5, consider_parent_rotate=True, save_vis=True):- """Export every pose action in one GLB/GLTF file. Returns ``{save_name: joint_names}``."""+ min_frames=5, consider_parent_rotate=True, save_vis=True,+ save_glb=False, glb_only=False):+ """Export every pose action in one GLB/GLTF file. Returns ``{save_name: joint_names}``.++ ``save_glb`` also writes the asset's processed GLB, ``glb_only`` only that+ (module docstring).+ """assert os.path.exists(gltf_path), f"GLTF file not found: {gltf_path}"+ if glb_only:+ # Existing NPZs only, markers or not; never (re-)exports.+ if asset_glb_missing(output_dir, save_name):+ add_asset_glb(save_name, output_dir,+ lambda: load_scene(import_gltf, gltf_path, fps=fps)[0],+ source_path=gltf_path)+ return {}+motion_save_dir = os.path.join(output_dir, "motions")vis_save_dir = os.path.join(output_dir, "videos")tpos_save_dir = os.path.join(output_dir, "tpose")if is_asset_complete(output_dir, save_name):- logger.info(f"Asset '{save_name}' already complete, skipping: {gltf_path}")+ if not (save_glb and asset_glb_missing(output_dir, save_name)):+ logger.info(f"Asset '{save_name}' already complete, skipping: {gltf_path}")+ return {}+ logger.info(f"Asset '{save_name}' already exported; adding its processed GLB")+ add_asset_glb(save_name, output_dir,+ lambda: load_scene(import_gltf, gltf_path, fps=fps)[0],+ source_path=gltf_path)return {}+ remove_asset_glb(output_dir, save_name)existing = sorted(Path(motion_save_dir).glob(f"{save_name}-*.npz")) if os.path.isdir(motion_save_dir) else []if existing:# Pruning is joint across all of an asset's clips, so a partial# (or pre-marker) export must be redone as a whole.logger.warning(f"'{save_name}' has {len(existing)} clip(s) but no completion "- f"marker (partial or pre-marker export); re-exporting.")+ f"marker (partial or pre-marker export); re-exporting; the old clips go to "+ f"{REPLACED_DIR}/.")+ set_aside_old_clips(output_dir, save_name)logger.info(f"Processing GLTF: {gltf_path}")os.makedirs(motion_save_dir, exist_ok=True)@@ -122,6 +158,14 @@ def export_objaverse(gltf_path, save_name, output_dir, fps=30, dtype=np.float64,n_clips=len(saved_clips),reason="" if saved_clips else "no clips above min frame count",joint_names=names_pruned if saved_clips else None)+ # After the marker, from a clip this run saved and a fresh import (the+ # scene now sits at the last extracted frame, and an animated armature+ # object or parent empty would carry that pose): exactly what --glb_only+ # builds. A failure is recorded, not raised.+ if save_glb and saved_clips:+ build_asset_glb(save_name, os.path.join(motion_save_dir, f"{saved_clips[0]}.npz"),+ output_dir, lambda: load_scene(import_gltf, gltf_path, fps=fps)[0],+ source_path=gltf_path)# Return once per asset (skeleton type), not per clipif saved_clips:@@ -152,6 +196,14 @@ def parse_args():help="Total number of workers.")parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")+ parser.add_argument('--save_glb', action='store_true',+ help="Also write rigs/<asset>.glb: the asset in its rest pose on the "+ "pruned skeleton (no animation; the clip NPZs drive it), for "+ "mesh_animation. On an existing export, only the missing GLBs are added.")+ parser.add_argument('--glb_only', action='store_true',+ help="Only add missing rigs/<asset>.glb to an existing export, for the "+ "assets whose NPZs are already in motions/: no NPZ, completion marker "+ "or summary JSON is written, and none is required.")argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]return parser.parse_args(argv)@@ -163,6 +215,9 @@ def main():gltf_paths = list_gltf_files(Path(args.data_dir))assert len(gltf_paths) > 0, f"No GLB/GLTF files found in {args.data_dir}"logger.info(f"Found {len(gltf_paths)} GLB/GLTF files in {args.data_dir}")+ # Same names as render_objaverse (identity for objaverse's hex ids);+ # checked over the full list, before sharding, so every worker agrees.+ names = asset_names(gltf_paths)# Shard files across workersif args.num_workers > 1:@@ -175,10 +230,11 @@ def main():n_failed = 0for glb_path in tqdm(gltf_paths, desc="Exporting GLB/GLTF files"):try:- saved = export_objaverse(glb_path, glb_path.stem, output_dir=args.output_dir,+ saved = export_objaverse(glb_path, names[glb_path], output_dir=args.output_dir,fps=args.fps, min_frames=args.min_frames,consider_parent_rotate=args.consider_parent_rotate,- save_vis=args.vis)+ save_vis=args.vis, save_glb=args.save_glb,+ glb_only=args.glb_only)if saved:all_joint_names.update(saved)except Exception as e: # noqa: BLE001 โ keep the batch going@@ -187,10 +243,13 @@ def main():with open(error_log, 'a') as log_file:log_file.write(f"Failed to export {glb_path}: {e}\n")- worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""- write_export_summary(args.output_dir, all_joint_names, fps=args.fps,- worker_suffix=worker_suffix)+ if not args.glb_only:+ worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,+ worker_suffix=worker_suffix)+ if args.save_glb or args.glb_only:+ report_glb_errors(args.output_dir)if n_failed:logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")sys.exit(1)diff --git a/data_process/motion_export/export_truebones.py b/data_process/motion_export/export_truebones.pyindex f578146..4d753eb 100644--- a/data_process/motion_export/export_truebones.py+++ b/data_process/motion_export/export_truebones.py@@ -9,6 +9,16 @@ Pipeline: import โ select armature/mesh โ extract animation โ Z-up โ Y-uprune skeleton (shared across the species' clips) โ save per-clip NPZ + MP4+ rest-pose PNG.+``--save_glb`` also writes ``rigs/<species>.glb``: the species in its rest+pose on the pruned skeleton, without animation (``processed_assets.export_asset_glb``),+built from the FBX of one clip on the species' main skeleton (its+``joint_names.json`` entry); every clip NPZ of that skeleton drives it as a+character (``mesh_animation``). Clips of another rig variant (KingCobra and+Monkey ship two) are logged; drive those from their FBX. On an existing export+it only adds the missing GLBs (the NPZs are not rewritten). ``--glb_only`` does+only that, also for an export without completion markers (the released one),+and writes nothing else; it shards species with ``--num_workers``.+Usage (Blender headless):blender -b -P data_process/motion_export/export_truebones.py -- \--data_dir dataset/raw/truebones/animation \@@ -31,9 +41,14 @@ from data_process.utils.blender_export import (load_scene, prepare_skeleton, extract_all_actions, save_rest_pose_vis,prune_skeleton_shared,action_frame_range, sanitize_action_name, sanitize_object_type,- save_motion, write_export_summary,+ save_motion, write_export_summary, truebones_clip_name,is_asset_complete, mark_asset_complete,)+from data_process.utils.asset_files import (+ asset_glb_missing, log_other_skeletons, main_skeleton_clips, record_glb_error,+ REPLACED_DIR, remove_asset_glb, report_glb_errors, set_aside_old_clips,+)+from data_process.utils.processed_assets import build_asset_glb# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ@@ -114,23 +129,70 @@ def postprocess_and_save(obj_type, extracted, rest_anim_shared, bone_names,# Per-species export# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Clips (source FBXs) tried for a species' processed GLB before giving up.+MAX_GLB_ATTEMPTS = 3+++def save_species_glb(obj_type, fbx_paths, output_dir, fps=30, want=None):+ """The species' processed GLB, from the FBX of its first clip on the main+ skeleton (*want* joint names, else its ``joint_names.json`` entry; see+ ``asset_files.main_skeleton_clips``). A failure is recorded, not raised."""+ main, others = main_skeleton_clips(output_dir, obj_type, want=want)+ log_other_skeletons(obj_type, others)+ fbx_by_clip = {}+ for fbx_path in fbx_paths:+ try:+ fbx_by_clip[truebones_clip_name(Path(fbx_path).stem)] = fbx_path+ except ValueError:+ continue+ candidates = [(npz_path, fbx_by_clip[os.path.basename(npz_path)[:-4]]) for npz_path in main+ if os.path.basename(npz_path)[:-4] in fbx_by_clip]+ # The next main-skeleton clip if one fails (a bad FBX), but not every one:+ # a failure that repeats is the asset's, and is recorded by build_asset_glb.+ for npz_path, fbx_path in candidates[:MAX_GLB_ATTEMPTS]:+ if build_asset_glb(obj_type, npz_path, output_dir,+ lambda: load_scene(import_fbx, fbx_path, fps=fps)[0],+ source_path=fbx_path):+ return+ if not candidates:+ record_glb_error(output_dir, obj_type,+ f"'{obj_type}': no source FBX in the input for any of its "+ f"{len(main)} main-skeleton clip(s); processed GLB not built")++def export_species(species, fbx_paths, output_dir, fps=30, dtype=np.float64,- min_frames=5, consider_parent_rotate=True, save_vis=True):+ min_frames=5, consider_parent_rotate=True, save_vis=True,+ save_glb=False, glb_only=False):"""Export one species: extract each of its per-clip FBX files, then prune- the skeleton jointly across all clips and save them."""+ the skeleton jointly across all clips and save them. ``save_glb`` also+ writes the species' processed GLB, ``glb_only`` only that (module+ docstring)."""obj_type = sanitize_object_type(species)+ if glb_only:+ # Existing NPZs only, markers or not; never (re-)exports.+ if asset_glb_missing(output_dir, obj_type):+ save_species_glb(obj_type, fbx_paths, output_dir, fps=fps)+ return {}+if is_asset_complete(output_dir, obj_type):- logger.info(f"Species '{obj_type}' already complete, skipping.")+ if save_glb and asset_glb_missing(output_dir, obj_type):+ logger.info(f"Species '{obj_type}' already exported; adding its processed GLB.")+ save_species_glb(obj_type, fbx_paths, output_dir, fps=fps)+ else:+ logger.info(f"Species '{obj_type}' already complete, skipping.")return {}+ remove_asset_glb(output_dir, obj_type)motion_save_dir = os.path.join(output_dir, "motions")existing = sorted(Path(motion_save_dir).glob(f"{obj_type}-*.npz")) if os.path.isdir(motion_save_dir) else []if existing:# Pruning is joint across all of a species' clips, so a partial# (or pre-marker) export must be redone as a whole.logger.warning(f"'{obj_type}' has {len(existing)} clip(s) but no completion "- f"marker (partial or pre-marker export); re-exporting.")+ f"marker (partial or pre-marker export); re-exporting; the old clips go to "+ f"{REPLACED_DIR}/.")+ set_aside_old_clips(output_dir, obj_type)# A few species ship rig variants (different node counts or container# names โ e.g. KingCobra's extra Null, Monkey's A01/B01/B02 containers).@@ -139,6 +201,7 @@ def export_species(species, fbx_paths, output_dir, fps=30, dtype=np.float64,# separately. The dominant group defines the species' joint_names entry# and rest-pose PNG.groups = {} # names signature -> {'rest_anim', 'skin_matrix', 'extracted'}+ import_failures = []for fbx_path in fbx_paths:stem = Path(fbx_path).stemspecies_part, sep, action_name = stem.partition('-')@@ -152,8 +215,9 @@ def export_species(species, fbx_paths, output_dir, fps=30, dtype=np.float64,try:result = extract_single_fbx(fbx_path, fps=fps, dtype=dtype)- except Exception as e:+ except Exception as e: # noqa: BLE001 โ finish the species, then fail itlogger.warning(f"Failed to extract {fbx_path}: {e}")+ import_failures.append(f"{os.path.basename(fbx_path)}: {e}")continuenframes = result['anim'].positions.shape[0]@@ -194,9 +258,19 @@ def export_species(species, fbx_paths, output_dir, fps=30, dtype=np.float64,else:logger.info(f"No valid FBX files extracted for '{obj_type}'; skipping.")+ if import_failures:+ # No completion marker: a rerun redoes the species with every clip+ # (pruning is joint across them), and the batch exits non-zero.+ raise RuntimeError(f"{len(import_failures)} of {len(fbx_paths)} clip(s) of "+ f"'{obj_type}' failed to extract: " + '; '.join(import_failures))mark_asset_complete(output_dir, obj_type,status="exported" if saved else "skipped",joint_names=saved.get(obj_type))+ # After the marker, on the skeleton this run saved as the species' own+ # (joint_names.json is only rewritten at the end of the run): a failure is+ # recorded, not raised, and a rerun with --glb_only adds the missing GLB.+ if save_glb and saved:+ save_species_glb(obj_type, fbx_paths, output_dir, fps=fps, want=saved[obj_type])return saved@@ -220,12 +294,29 @@ def parse_args():"(see prune_skeleton_shared; use --no-consider_parent_rotate to disable).")parser.add_argument('--vis', action=argparse.BooleanOptionalAction, default=True,help="Render the per-clip MP4 preview (use --no-vis for bulk runs).")+ parser.add_argument('--save_glb', action='store_true',+ help="Also write rigs/<species>.glb: the species in its rest pose on the "+ "pruned skeleton (no animation; the clip NPZs drive it), for "+ "mesh_animation. On an existing export, only the missing GLBs are added.")+ parser.add_argument('--glb_only', action='store_true',+ help="Only add missing rigs/<species>.glb to an existing export, for the "+ "species whose NPZs are already in motions/: no NPZ, completion marker "+ "or summary JSON is written, and none is required.")+ parser.add_argument('--worker_id', type=int, default=0,+ help="Worker index when sharding species across processes "+ "(--glb_only only).")+ parser.add_argument('--num_workers', type=int, default=1,+ help="Total number of workers (--glb_only only: a full export writes "+ "one set of summary JSONs, so it stays single-process).")argv = sys.argv[sys.argv.index("--") + 1:] if "--" in sys.argv else sys.argv[1:]return parser.parse_args(argv)def main():args = parse_args()+ if args.num_workers > 1 and not args.glb_only:+ raise SystemExit("--num_workers > 1 needs --glb_only: a full Truebones export "+ "is single-process (it writes one set of summary JSONs).")fbx_files = sorted(f for f in os.listdir(args.data_dir) if f.lower().endswith('.fbx'))by_species = {}@@ -239,12 +330,17 @@ def main():error_log = os.path.join(args.output_dir, 'export_errors.log')all_joint_names = {}n_failed = 0- for species, fbx_paths in tqdm(sorted(by_species.items()), desc="Exporting species"):+ species_items = sorted(by_species.items())+ if args.num_workers > 1:+ species_items = species_items[args.worker_id::args.num_workers]+ logger.info(f"Worker {args.worker_id}/{args.num_workers}: {len(species_items)} species")+ for species, fbx_paths in tqdm(species_items, desc="Exporting species"):try:saved = export_species(species, fbx_paths, output_dir=args.output_dir,fps=args.fps, min_frames=args.min_frames,consider_parent_rotate=args.consider_parent_rotate,- save_vis=args.vis)+ save_vis=args.vis, save_glb=args.save_glb,+ glb_only=args.glb_only)if saved:all_joint_names.update(saved)except Exception as e: # noqa: BLE001 โ keep the batch going@@ -253,7 +349,10 @@ def main():with open(error_log, 'a') as log_file:log_file.write(f"Failed to export species {species}: {e}\n")- write_export_summary(args.output_dir, all_joint_names, fps=args.fps)+ if not args.glb_only:+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps)+ if args.save_glb or args.glb_only:+ report_glb_errors(args.output_dir)if n_failed:logger.error(f"{n_failed}/{len(by_species)} species failed; see {error_log}")sys.exit(1)diff --git a/data_process/motion_rendering/render_general.py b/data_process/motion_rendering/render_general.pynew file mode 100644index 0000000..fe52908--- /dev/null+++ b/data_process/motion_rendering/render_general.py@@ -0,0 +1,49 @@+"""Render multi-view frames for the general dataset's assets (EEVEE).++The ``general`` dataset is extra training data in the objaverse layout: one+rigged asset per file in ``dataset/raw/general/animation/`` (``.glb``/``.gltf``,+``.fbx`` also accepted; every animation of a file is a clip), exported by+``motion_export/export_general.py``. This is+:mod:`render_objaverse` with that exporter's file list and action discovery,+so both stages derive the same ``{asset}-{action}`` clip names:++ * every ``.glb``/``.gltf``/``.fbx`` file, minus ``excluded.csv``+ (``blender_export.list_asset_files``);+ * asset name = file stem with ``-`` and whitespace replaced by ``_``+ (``blender_export.asset_name``);+ * actions renamed to their glTF animation names, and the bound action when+ no pose action is found (``blender_export.discover_clip_actions``).++Assets without a mesh export fine but have nothing to render, so they get no+caption and stage 4 drops their clips.++Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):+ python -m data_process.motion_rendering.render_general \+ --data_dir dataset/raw/general/animation --output_dir dataset/render/general+"""++import sys++from loguru import logger++from data_process.motion_rendering.render_objaverse import main as render_main+from data_process.utils.blender_export import discover_clip_actions, list_asset_files+++def main():+ return render_main(+ list_assets=list_asset_files,+ discover=discover_clip_actions,+ description="Render multi-view frames for the general dataset's "+ "GLB/GLTF/FBX animations (EEVEE).",+ )+++if __name__ == "__main__":+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures+ # from `set -e` and from Slurm's afterok dependencies.+ try:+ sys.exit(main())+ except Exception:+ logger.exception("render_general failed")+ sys.exit(1)diff --git a/data_process/motion_rendering/render_objaverse.py b/data_process/motion_rendering/render_objaverse.pyindex 6e9de91..2629df6 100644--- a/data_process/motion_rendering/render_objaverse.py+++ b/data_process/motion_rendering/render_objaverse.py@@ -2,7 +2,8 @@For each GLB, discovers all pose actions and renders each one as``{glb_stem}-{action}/v00{0..3}/*.png`` โ the layout consumed by-``vlm_caption/caption_motion.py``.+``vlm_caption/caption_motion.py``. :mod:`render_general` reuses+:func:`main` with the general dataset's file list and action discovery.Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):python -m data_process.motion_rendering.render_objaverse \@@ -18,6 +19,7 @@ import bpy # noqa: F401 โ fail fast when the bpy module is unavailablefrom loguru import loggerfrom data_process.utils.blender_export import (+ asset_names,discover_pose_actions,list_gltf_files,unique_clip_names,@@ -75,9 +77,16 @@ def find_pending_stems(output_dir, gltf_stems, require_video=False):return pending-def render_glb(glb_path, output_dir, args):- """Render multi-view frames for every pose action in one GLB/GLTF file."""- save_name = Path(glb_path).stem+def render_glb(glb_path, output_dir, args, save_name, discover=None):+ """Render multi-view frames for every pose action in one asset file.++ Args:+ save_name: Asset name / clip-folder prefix (``asset_name`` of the+ file, as the exporter names it).+ discover: ``armature -> (action_names, frame_ranges)``; must match+ the exporter of the same dataset (default: every pose action,+ :func:`discover_pose_actions`).+ """if is_asset_render_skipped(output_dir, save_name):logger.info(f"Previously marked as no-render, skipping: {glb_path}")return@@ -93,7 +102,10 @@ def render_glb(glb_path, output_dir, args):# Names are derived from the UNFILTERED action list so they match the# export stage, which applies no frame-count filter โ deriving them from# a filtered subset would shift the collision suffixes out of step.- all_action_names, frame_ranges = discover_pose_actions()+ if discover is None:+ all_action_names, frame_ranges = discover_pose_actions()+ else:+ all_action_names, frame_ranges = discover(armature)clip_suffixes = unique_clip_names(all_action_names)action_names = [@@ -139,13 +151,21 @@ def render_glb(glb_path, output_dir, args):mark_asset_render_done(output_dir, save_name, expected_dirs)-def main():- parser = argparse.ArgumentParser(- description="Render multi-view frames for Objaverse GLB/GLTF animations (EEVEE).")+def main(list_assets=list_gltf_files, discover=None,+ description="Render multi-view frames for Objaverse GLB/GLTF animations (EEVEE)."):+ """CLI entry point.++ Args:+ list_assets: ``data_dir -> [asset paths]``. Each asset renders under+ its :func:`asset_name` (the stem for objaverse's hex ids).+ discover: Action discovery passed to :func:`render_glb`.+ """+ parser = argparse.ArgumentParser(description=description)parser.add_argument("--data_dir", type=str, required=True,- help="Directory containing GLB/GLTF files.")+ help="Directory containing the asset files.")parser.add_argument("--obj_name", type=str, default=None,- help="Single GLB stem to render (default: all files).")+ help="Single asset name to render (the file stem with "+ "'-' and whitespace replaced by '_'; default: all files).")parser.add_argument("--output_dir", type=str, required=True,help="Output directory.")parser.add_argument("--scale", type=float, default=1.0,@@ -189,23 +209,24 @@ def main():init_render_engine(render_samples=args.samples)data_dir = Path(args.data_dir)+ gltf_paths = list_assets(data_dir)+ # Raises when two files share a name; checked before sharding so every+ # worker agrees.+ names = asset_names(gltf_paths)if args.obj_name:- glb_path = data_dir / f"{args.obj_name}.glb"- if not glb_path.is_file():- glb_path = data_dir / f"{args.obj_name}.gltf"- if not glb_path.is_file():- parser.error(f"Cannot find {args.obj_name}.glb or .gltf in {data_dir}")- gltf_paths = [glb_path]+ gltf_paths = [p for p in gltf_paths if names[p] == args.obj_name]+ if not gltf_paths:+ parser.error(f"No asset named {args.obj_name!r} in {data_dir} "+ f"(or it is listed in excluded.csv)")else:- gltf_paths = list_gltf_files(data_dir)- logger.info(f"Found {len(gltf_paths)} GLB/GLTF files in {data_dir}")+ logger.info(f"Found {len(gltf_paths)} asset files in {data_dir}")if args.missing_only:- pending = find_pending_stems(args.output_dir, [p.stem for p in gltf_paths],+ pending = find_pending_stems(args.output_dir, [names[p] for p in gltf_paths],require_video=args.video)before = len(gltf_paths)- gltf_paths = [p for p in gltf_paths if p.stem in pending]- logger.info(f"[missing-only] {len(gltf_paths)}/{before} GLBs have pending renders.")+ gltf_paths = [p for p in gltf_paths if names[p] in pending]+ logger.info(f"[missing-only] {len(gltf_paths)}/{before} assets have pending renders.")if args.num_workers > 1:gltf_paths = gltf_paths[args.worker_id::args.num_workers]@@ -219,7 +240,8 @@ def main():n_failed = 0for glb_path in gltf_paths:try:- render_glb(glb_path, args.output_dir, args)+ render_glb(glb_path, args.output_dir, args,+ save_name=names[glb_path], discover=discover)except Exception as e: # noqa: BLE001 โ keep the batch goingn_failed += 1logger.exception(f"Failed to render {glb_path}")diff --git a/data_process/motion_rendering/render_tpose.py b/data_process/motion_rendering/render_tpose.pyindex 8de3e2e..fe50f53 100644--- a/data_process/motion_rendering/render_tpose.py+++ b/data_process/motion_rendering/render_tpose.py@@ -5,7 +5,9 @@ top row, left/right bottom row) โ the "flat" layout consumed by``vlm_caption/classify_category.py``.Two input layouts are auto-detected per entry of ``--data_dir``:- - files: ``<data_dir>/*.{glb,gltf,fbx}`` -> one grid per file stem+ - files: ``<data_dir>/*.{glb,gltf,fbx}`` -> one grid per asset name+ (the stem with ``-`` and whitespace replaced by ``_``, as the general+ exporter names it; objaverse stems are unchanged)- subdirectory: ``<data_dir>/<obj_type>/*.fbx`` -> one grid per objecttype, rendered from its main (all-in-one) FBX โ Truebones layout.@@ -21,7 +23,9 @@ from pathlib import Pathfrom loguru import logger-from data_process.utils.blender_export import find_main_fbx, sanitize_object_type+from data_process.utils.blender_export import (+ asset_names, find_main_fbx, load_excluded_stems, sanitize_object_type,+)from data_process.utils.blender_render import (RENDERABLE_EXTS,RENDER_FPS,@@ -34,7 +38,8 @@ from data_process.utils.blender_render import (def discover_assets(data_dir, species_grids=False):"""Return (name, path) pairs to render, auto-detecting the layout.- Files directly under *data_dir* are rendered by stem; each+ Files directly under *data_dir* (minus ``excluded.csv``) are rendered+ by asset name; eachsubdirectory is treated as a Truebones-style object type and renderedfrom its main FBX (fallback: first FBX in the subdirectory).@@ -45,18 +50,25 @@ def discover_assets(data_dir, species_grids=False):data_dir = Path(data_dir)assets = []seen_species = set()-+ # Per-file grids are named like the exporters name them (asset_names+ # raises when two share a name) and skip excluded.csv, as the objaverse+ # and general stages do; species grids keep the Truebones behaviour.+ excluded = set() if species_grids else load_excluded_stems(data_dir)+ files = [p for p in sorted(data_dir.iterdir())+ if p.is_file() and p.suffix.lower() in RENDERABLE_EXTS and p.stem not in excluded]+ names = {} if species_grids else asset_names(files)++ for p in files:+ if species_grids:+ species = p.stem.split('-', 1)[0]+ if species in seen_species:+ continue+ seen_species.add(species)+ assets.append((sanitize_object_type(species), p))+ else:+ assets.append((names[p], p))for p in sorted(data_dir.iterdir()):- if p.is_file() and p.suffix.lower() in RENDERABLE_EXTS:- if species_grids:- species = p.stem.split('-', 1)[0]- if species in seen_species:- continue- seen_species.add(species)- assets.append((sanitize_object_type(species), p))- else:- assets.append((p.stem, p))- elif p.is_dir():+ if p.is_dir():main_fbx = find_main_fbx(str(p))if main_fbx is None:fbxs = sorted(f for f in p.iterdir()@@ -101,7 +113,7 @@ def main():help="Directory of GLB/GLTF/FBX files, or of ""per-object-type FBX subdirectories (Truebones).")parser.add_argument("--obj_name", type=str, default=None,- help="Single asset name (file stem or object type) "+ help="Single asset name (asset name or object type) ""to render (default: all).")parser.add_argument("--output_dir", type=str, required=True,help="Flat directory for the <name>.png grids.")@@ -118,8 +130,9 @@ def main():help="EEVEE TAA render samples.")parser.add_argument("--keep-all-meshes", dest="keep_main_mesh_only",action="store_false",- help="Keep every mesh instead of only the "- "highest-vertex one (multi-mesh characters).")+ help="Keep every mesh, instead of only those skinned to "+ "or parented under the armature (stray helper "+ "geometry included).")parser.add_argument("--species_grids", action="store_true",help="Group {Species}-{Action}.fbx files by species ""and render one grid per species (curated "diff --git a/data_process/rig_preprocess/AGENTS.md b/data_process/rig_preprocess/AGENTS.mdnew file mode 100644index 0000000..02512e0--- /dev/null+++ b/data_process/rig_preprocess/AGENTS.md@@ -0,0 +1,75 @@+# AGENTS.md: preparing a rig with `rig_preprocess`++For a coding agent helping a user animate their own rigged asset with UniMate. Usage and every option are in [README.md](README.md). Run from the repository root in the `unimate` environment, write only under the output directory you choose, and do not change files under `dataset/`.++The model was trained on skeletons whose joints carry labels from one shared vocabulary and whose forward direction comes from a pair of joints. Your job is to make the user's asset look like that training data: the closer the labels and the facing pair follow the conventions below, the better the generated motion.++## Workflow++```bash+# 1. label the joints (an LLM when DEEPSEEK_API_KEY is set for the default model, else rules) and stop for review+python -m data_process.rig_preprocess run --input <asset.glb|.gltf|.fbx> --output_dir outputs/rig/<name>+# 2. read outputs/rig/<name>/REVIEW.md and annotation_preview.png, correct outputs/rig/<name>/annotation.json+# 3. build the asset from the corrected file (same command, plus --annotation)+python -m data_process.rig_preprocess run --input <asset> --output_dir outputs/rig/<name> \+ --annotation outputs/rig/<name>/annotation.json+# 4. check summary.json and preview.png, then sample and animate+python -m unimate.inference.sample --exp_dir outputs/unimate_uniml3d_f60_v3_preview --asset outputs/rig/<name> \+ --prompt "An object walks forward." --num_repetitions 3 --output_dir outputs/samples/<name>+bash scripts/run_animate_motion.sh outputs/samples/<name>+```++`REVIEW.md` gives, under *Continue*, the exact command that continues (it keeps the first run's options); use it. For another LLM pass `--model gpt-โฆ` with `OPENAI_API_KEY`, or a local model. If a build is refused, fix what the message names and rerun the same command; `annotation.json` is never deleted.++## Correcting `annotation.json`++Edit only each joint's `label` and the `face_pair` fields (`right`, `left`, `body_axis`). Keep every `raw` name, the joint order and the JSON structure: the file is matched to the skeleton by raw name, and a changed raw name is refused.++### Joint labels++- **Use the training vocabulary.** `annotation.json` lists it under `vocabulary`; prefer a label from it. A label outside it is kept with a note, but the model has never seen it and treats it as an unknown part.+- **Sides are a prefix**: `Left` / `Right` from the asset's own point of view (its left is on the +X side when it faces +Z). Joints on the midline have no side (`Spine`, `Neck`, `Head`, `Tail`, `Jaw`).+- **Name the bone segment, not the joint.** Legs are `Thigh`, `Shin`, `Foot`, `Toe`; arms `Shoulder`, `Upper Arm`, `Forearm`, `Hand`; the root of a body is `Hips`. Use `Knee`, `Elbow`, `Ankle`, `Wrist` only when the rig really has an extra joint there.+- **Repeat a label along a chain**: several spine joints are all `Spine`, several tail joints all `Tail`, several neck joints all `Neck`.+- **Fingers and toes**: `Thumb Finger`, `Index Finger`, `Middle Finger`, `Ring Finger`, `Pinky Finger`, each repeated along its chain; toes are `Toe`.+- **Leaf tips** (an end bone with nothing below it) take the parent's part plus `End`: `Head End`, `Left Toe End`, `Left Hand End`.+- **Four-legged animals** label the hind legs like human legs (`Thigh`, `Shin`, `Fetlock` if present, `Foot`, `Toe`) and the forelegs like arms (`Shoulder`, `Upper Arm`, `Forearm`, `Hand`), as in the Horse row below. Use `Front` / `Back` (`Left Front Thigh`) only for robot legs whose bones are coded that way (`FR_hip`, `RL_calf`). Insects and spiders with more legs use `Front Leg`, `Middle Leg`, `Hind Leg`.+- **Other body plans** use the plain part name: `Wing`, `Feather`, `Fin`, `Pectoral Fin`, `Dorsal Fin`, `Tentacle`, `Antenna`, `Ear`, `Eye`, `Mane`, `Claw`, `Pincer`, `Stem`, `Leaf`.+- **Machines and objects**: label by the body part a joint plays (a robot's hip link is `Left Thigh`, its gripper `Left Hand`); a part with no anatomical role gets a short plain name (`Wheel`, `Lid`, `Arm`).+- **Unknown or helper bones** (IK targets, controls, props) are `Bone`.++Examples from the training data:++| Body | Labels (in joint order, abridged) | Facing pair (right / left) |+|---|---|---|+| Humanoid | `Hips`, `Spine`, `Spine`, `Spine`, `Neck`, `Head`, `Head End`, `Left Shoulder`, `Left Upper Arm`, `Left Forearm`, `Left Hand`, `Left Thumb Finger` โฆ | `Right Thigh` / `Left Thigh` |+| Horse | `Hips`, `Tail` ร5, `Spine`, `Right Thigh`, `Right Shin`, `Right Fetlock`, `Right Foot`, `Right Toe`, โฆ, `Left Shoulder`, `Left Upper Arm`, `Left Forearm`, `Left Hand`, โฆ `Mane` | `Right Thigh` / `Left Thigh` |+| Bird | `Hips`, `Tail`, `Spine`, `Right Thigh`, `Right Shin`, `Right Fetlock`, `Right Foot`, `Right Toe` โฆ, `Left Wing` โฆ | `Right Thigh` / `Left Thigh` |+| Shark | `Root`, `Spine`, `Head`, `Jaw`, `Left Pectoral Fin`, `Right Pectoral Fin`, `Dorsal Fin`, `Spine` ร3, `Left Pelvic Fin`, `Right Pelvic Fin`, `Tail` ร4 | `Right Pectoral Fin` / `Left Pectoral Fin` |+| Snake | `Hips`, `Tail` ร13, `Spine` ร6, `Neck`, `Head`, `Jaw`, `Tongue` | head-end joint / tail-end joint, `body_axis: true` |++### Facing pair++The pair defines where the asset faces; the model generates motion relative to it. Pick, in this order:++1. the two **thighs** (`Right Thigh` / `Left Thigh`), for any body with legs, including birds and robots;+2. otherwise two other mirrored joints near the body's centre: shoulders, upper arms, front legs, wings, pectoral fins;+3. a body without mirrored limbs (snake, worm, fish without fins): a joint at the head end as `right`, one at the tail end as `left`, and `body_axis: true`.++`right` must be the asset's right-hand joint. The two joints must be side by side in the rest pose (not one above the other); a pair without horizontal separation is refused. Leave both empty only if the asset has no front at all. Check the arrow in `annotation_preview.png`: it must point where the asset looks.++## Before sampling++Read `summary.json` and `preview.png`:++- `model_joint_limit.fits` must be true: the models take at most 71 joints (trained on up to 70). A larger rig (full fingers, face bones) must be simplified first.+- `notes` lists what the model cannot reproduce. If it says the rest pose is upside down or lying, add `--rest_rotation x180 --overwrite` (or another rotation) to the `--annotation` command and rebuild until the preview stands upright and faces +Z.+- In `preview.png` the asset should stand on the ground, upright, facing +Z, like the training skeletons.++## Prompts++Write prompts in the style of the training captions (`dataset/features/<dataset>/captions*.json` after downloading UniML3D): start with "An object", describe one motion, not the character ("An object flaps its wings.", not "A golden eagle flaps its wings in the sky."). Short captions that already occur in the training data for similar skeletons work best. Generate several samples (`--num_repetitions`) and try a few phrasings; left / right in a prompt is followed unreliably.++## If you change this package++Keep `rig_preprocess` a driver of the dataset pipeline's own functions (export, annotation, feature extraction, canonical bake) so an asset is prepared exactly like the training data; never copy pipeline code into it. The output directory is what `unimate/inference/assets.py` reads (`cond.npy` with one entry, `summary.json`, `<name>.glb`, `motions/`); keep those names stable. After a change, run `python -m py_compile data_process/rig_preprocess/*.py`, process one training asset with `--annotate dataset --profile <dataset>` and check it with `python -m data_process.rig_preprocess verify --output_dir <out>` (the canonical GLB must be `identical`, the cond `identical` or `equivalent`).diff --git a/data_process/rig_preprocess/README.md b/data_process/rig_preprocess/README.mdnew file mode 100644index 0000000..2cc0974--- /dev/null+++ b/data_process/rig_preprocess/README.md@@ -0,0 +1,296 @@+# rig_preprocess: bring your own rigged asset++`rig_preprocess` turns one rigged 3D asset (GLB, glTF or FBX, animated or not) into an **asset directory** that a trained UniMate model can animate from text: the skeleton conditioning (`cond.npy`), the rest-pose mesh in the model's canonical frame (`<name>.glb`) and a preview to check. It runs the export, joint-annotation, feature-extraction and canonical-bake code of the [UniML3D pipeline](../README.md), so an unseen asset is prepared exactly like the training data, and a training asset reproduces the dataset's copy.++```bash+python -m data_process.rig_preprocess run --input robot.glb --output_dir outputs/rig/robot # labels, then stops for review+# read outputs/rig/robot/REVIEW.md, correct annotation.json (by hand or with an LLM / coding agent), then build:+python -m data_process.rig_preprocess run --input robot.glb --output_dir outputs/rig/robot \+ --annotation outputs/rig/robot/annotation.json+python -m unimate.inference.sample --exp_dir outputs/unimate_uniml3d_f60_v3_preview --asset outputs/rig/robot \+ --prompt "An object walks forward." --num_repetitions 3 --output_dir outputs/samples/custom+bash scripts/run_animate_motion.sh outputs/samples/custom # robot.glb, animated per motion+```++## Contents++- [How it works](#how-it-works)+- [Quick start](#quick-start)+- [Inputs](#inputs)+- [Joint labels and facing](#joint-labels-and-facing)+- [Review before the cond](#review-before-the-cond)+- [Outputs](#outputs)+- [Check the result](#check-the-result)+- [Command reference](#command-reference)+- [Reproducing a training asset](#reproducing-a-training-asset)+- [Accuracy](#accuracy)+- [Limitations](#limitations)+- [Troubleshooting](#troubleshooting)++## How it works++```mermaid+flowchart LR+ asset(["rigged GLB / glTF / FBX"])+ export["1 ยท Export<br/>skeleton, clips"]+ annotate["2 ยท Annotate<br/>joint labels, facing pair"]+ review["Review (optional)<br/>you, an LLM or an agent"]+ features["3 ยท Features<br/>canonical T-pose, topology"]+ bake["4 ยท Bake<br/>canonical GLB"]+ out(["asset directory"])+ asset --> export --> annotate --> features --> bake --> out+ annotate -. "default" .-> review -. "--annotation" .-> features+```++1. **Export** (Blender): imports the asset, prunes control and helper bones, converts to Y-up, and writes one clip per animation; an asset without animation becomes one rest-pose clip.+2. **Annotate**: maps each bone name onto UniMate's joint vocabulary (`Bip01_L_Thigh` โ `Left Thigh`) and picks the *facing pair*, two joints that define which way the asset faces.+3. **Features**: canonicalizes the rest pose (facing +Z, centred, scaled to diameter 2, grounded) and computes the topology condition.+4. **Bake**: rebuilds armature and mesh in that canonical frame, so generated motion drives the asset without retargeting.++**Requirements.** The repository's `unimate` environment, which includes the pip `bpy` 4.0 module (no separate Blender needed); run from the repository root. CPU only: an asset takes seconds to a minute. LLM labels (the default) need an LLM backend, by default `deepseek-v4-flash` with `DEEPSEEK_API_KEY` ([LLM options](#llm-options)); without one a run falls back to the offline rules with a warning.++## Quick start++```bash+# a GLB, animated or not: stops after the annotation for review (see "Review before the cond")+python -m data_process.rig_preprocess run --input robot.glb --output_dir outputs/rig/robot+# continue from the checked annotation.json+python -m data_process.rig_preprocess run --input robot.glb --output_dir outputs/rig/robot \+ --annotation outputs/rig/robot/annotation.json+# or build directly from the automatic annotation+python -m data_process.rig_preprocess run --input robot.glb --output_dir outputs/rig/robot --no_review++# a known facing pair (raw bone names, right side first)+python -m data_process.rig_preprocess run --input creature.fbx --output_dir outputs/rig/creature \+ --face_r R_Thigh --face_l L_Thigh++# one species' {Species}-{Action}.fbx clips, keeping them for in-betweening and editing+python -m data_process.rig_preprocess run --input clips/Parrot-*.fbx --output_dir outputs/rig/parrot --save_clips++# the wrapper: options as environment variables, arguments after -- passed to run+INPUT=robot.glb OUTPUT_DIR=outputs/rig/robot SAVE_CLIPS=1 bash data_process/scripts/run_rig_preprocess.sh -- --overwrite+```++## Inputs++**Formats.** GLB, glTF or FBX with one armature. A skinned mesh is optional; without one there is no canonical GLB.++**Animation.** Each animation (glTF animation or FBX action) becomes a clip, and the skeleton is pruned with the dataset's motion-aware rules (bones that never move, static or stretching leaves). An asset without animation is pruned by the motion-free rules only, so it keeps a few unskinned end bones (`*_End`, `*Nub`) an animated export would drop.++**Profile.** The dataset whose processing to reproduce. The default, `auto`, suits almost every input.++| Profile | Reads | Use for | Differences |+|---|---|---|---|+| `auto` (default) | any | everything | `truebones` for several `{Species}-{Action}.fbx` clips of one species (with `--annotate dataset`: the dataset of `--dataset_export_dir`), else `general` |+| `general` | GLB, glTF, FBX | robots, multi-mesh characters, any single file | skin weights from every skinned mesh; `_tip` leaves kept; rest-pose frames kept; 4 to 180 joints |+| `objaverse` | GLB, glTF | single-mesh GLBs like Objaverse-XL | skin weights from the largest mesh; rest-pose frames removed; 4 to 180 joints |+| `truebones` | FBX | one species' `{Species}-{Action}.fbx` clips | the species exporter; 8 to 150 joints |++**Name.** The asset name (its *object type*) is the file stem with `-`, whitespace and `[ ] * ?` replaced by `_` (`my-robot.glb` โ `my_robot`); a Truebones clip set is named after its species. `--name` sets another; it may not contain `- / \ [ ] * ?`.++**Joint budget.** The UniML3D models are trained on skeletons of up to 70 joints and take at most 71 (one slot for the joint-addition augmentation). `summary.json` records whether the asset fits (`--exp_dir <run>` checks against that run instead). A larger rig, common for game characters with full fingers and face bones, must be simplified before sampling.++## Joint labels and facing++The model is conditioned on two things a bare rig does not provide:++- **Joint labels.** Every bone is mapped onto one anatomical vocabulary shared by all training skeletons (`mixamorig:LeftUpLeg`, `Bip01_L_Thigh`, `left_hip_pitch_link` โ `Left Thigh` / `Left Hip`), so the same body part has the same embedding on every rig.+- **Facing pair.** Two joints, a left / right pair such as the thighs or a head / tail pair for snakes and fish, define the asset's forward direction. The canonical rest pose faces +Z, and generated motion moves relative to it.++| `--annotate` | Source |+|---|---|+| `llm` (default when an LLM is configured) | The pipeline's LLM labeller and facing-pair selector, the source of the training labels before review; a failed call, or a pair that defines no facing, keeps the rule result |+| `rule` (default without an LLM) | Offline and deterministic: a rule-based cleaner with a vocabulary learned from the reviewed training labels, rig-level refinements (limb chains, sides, finger numbering), geometric labels for mostly unnamed rigs, then a lookup of the reviewed training labels: a rig with the same bone tree as a training rig (Mixamo, Biped, CAT, Rigify and similar templates) takes that rig's labels, and bone names with a consistent reviewed label replace placeholder labels |+| `dataset` | A training asset's own reviewed labels and facing pair, matched by raw bone name; reproduces the dataset exactly |++`--annotate_names` / `--annotate_face` take the labels or the pair from another source; given alone, the other source is `rule` (`--annotate_face llm`: rule labels with an LLM pair). `--face_r` / `--face_l` set the pair by hand in any mode, and no LLM pair is requested then (raw bone names, right or head first; `--body_axis` for a head / tail pair). A selected pair whose joints are not separated horizontally in the rest pose defines no facing: the asset keeps its authored facing, with a note. A hand-given pair like that is an error.++## Review before the cond++Automatic labels cover common rigs well, but an unusual rig can get wrong labels or a wrong facing pair (abbreviated bone names such as `fr_uleg` or `arm_link_sh0` are the usual cause), and both are baked into `cond.npy`. So by default a run stops after the annotation, before anything is built, so the result can be checked by hand or by your own LLM or coding agent (`--no_review` skips this):++```bash+python -m data_process.rig_preprocess run --input creature.glb --output_dir outputs/rig/creature+# read outputs/rig/creature/REVIEW.md and correct annotation.json, then:+python -m data_process.rig_preprocess run --input creature.glb --output_dir outputs/rig/creature \+ --annotation outputs/rig/creature/annotation.json+```++| File | Content |+|---|---|+| `annotation.json` | Every joint's raw name, parent, label and the label's `source`: `template` (a training rig with the same bone tree), `names` (a bone name with a consistent reviewed label), or `rule` / `llm` (`file` when built from an annotation file). Joints worth a look have `"check": true` and `reasons`: a placeholder label (`Bone`), a label outside the vocabulary, or Left / Right limbs that lie the other way round from the facing pair. The facing pair has its own flags (missing, a head / tail axis, a part other than hips, shoulders, limb roots or fins, contradicted by most mirrored limbs). Also: the alternative pairs the labels offer, the run's notes (e.g. an upside-down rest pose), and the vocabulary |+| `annotation_preview.png` | The exported rest pose with the facing pair and the forward direction it gives (green arrow), and every joint as `[index] label` |+| `REVIEW.md` | The flagged joints and notes, what to check, the command that continues (with the first run's options), and a request to give an LLM or coding agent together with `annotation.json` |++Edit only `label`, `face_pair.right`, `face_pair.left` and `face_pair.body_axis` (`true` or `false`), and keep `format`. `--annotation` validates the file before building: raw names unchanged, every label a non-empty string, a facing pair of two different joints of the asset that define a facing (or none). Labels outside the vocabulary are kept, with a note. Add `--review` to an `--annotation` run to regenerate the review files, `annotation.json` included, from the edited file without building. `--annotate dataset` takes already reviewed labels and builds directly; `--review` is not available with it.++Continuing a review (`--annotation`, with or without `--review`) replaces it without `--overwrite`; the review files stay until the edited file has loaded, so a rejected edit can be fixed and the same command rerun. `annotation.json` is never deleted: a run that would not read it stops, and with `--overwrite` moves it to `annotation.json.bak` (`.bak2`, ...). A run from an annotation file elsewhere keeps a copy as `annotation.json`.++The flags point at likely errors; they are not a complete check ([Accuracy](#accuracy)). Read the whole list.++## Outputs++```+<output_dir>/+โโโ cond.npy {name: cond}: the skeleton conditioning the sampler reads+โโโ <name>.glb the canonical rest-pose asset, with its joint order stored in the file+โโโ <name>.fbx with --formats glb,fbx+โโโ preview.png what to check before sampling+โโโ motions/ with --save_clips: the asset's clips as stage 4 feature NPZs+โโโ summary.json what was done, and what to watch+โโโ work/ with --keep_intermediate: every stage's output in the dataset layout+```++A review run writes `annotation.json`, `annotation_preview.png`, `REVIEW.md` and `summary.json` instead. `cond.npy` is the primary output: when the canonical GLB cannot be built (no skinned mesh, a degenerate skeleton), the cond is still written and `summary.json` says why.++<details>+<summary><code>summary.json</code> fields</summary>++| Field | Meaning |+|---|---|+| `name` | Asset name, the key of `cond.npy` |+| `inputs`, `profile`, `profile_auto` | Files read, profile used, whether `auto` chose it |+| `animated` | Whether the asset had animation |+| `annotate`, `annotate_names`, `annotate_face`, `annotation`, `annotation_notes` | Label and facing-pair sources (`annotate: file` with `--annotation`, whose path is `annotation`) and the annotation step's notes |+| `face_pair` | Raw and cleaned names of the facing pair, and how it was chosen |+| `n_joints_export`, `n_joints` | Joints after export and in the cond |+| `n_clips`, `clips` | Clips that passed stage 4's filters; those kept in `motions/` |+| `stats_dataset` | Dataset whose normalization statistics the sampler applies |+| `model_joint_limit` | `{max_joints, source, fits}` |+| `dropped_translation` | Largest translation of a non-root joint in the clips, as a share of the skeleton's size |+| `rest_rotation` | The `--rest_rotation` applied |+| `notes` | Everything worth a look: dropped translation, an upside-down rest pose, too many joints, no canonical GLB |+| `cond`, `canonical_glb`, `canonical_fbx`, `canonical_glb_error`, `preview` | Output paths, or why the GLB is missing |+| `rig_glb` | The export stage's rest-pose GLB when kept (`--keep_intermediate`), else null |+| `stage`, `review` | review runs only: `stage: review`; the annotation path, flag counts and the command that continues |+| `reference`, `dataset_sidecars`, `root_offsets_applied`, `rest_orientations_applied` | `--annotate dataset` only: the dataset reproduced and what was taken from it |++</details>++## Check the result++`preview.png` has three panels: the canonical rest pose with the facing pair (red: right or head; blue: left or tail) and the forward direction (grey arrow, +Z); a front view, where the asset's face or chest should be visible with its right side on the image's left; and every joint with its label. Then read `notes` in `summary.json`.++| You see | Do |+|---|---|+| The asset faces sideways or backwards | Set the pair: `--face_r <right bone> --face_l <left bone>`, or correct it with `--review` |+| Wrong or many `Bone` labels | Correct `annotation.json` before building; with `rule` labels, set an LLM key and run again |+| The rest pose lies down or is upside down | `--rest_rotation x180` (upside down) or `x90` / `x-90` (on its back / front); turns compose, e.g. `x90,y180` |+| `model_joint_limit.fits` is false | Remove finger, face or accessory bones from the rig |+| A note that a joint translates | Expected: the model generates rotations and the root path, so sliding or stretching bones keep their rest length |+| No canonical GLB | The asset has no skinned mesh; the cond still samples, but there is no mesh to drive |++Re-run with `--overwrite` after changing anything.++**Using the asset.** Pass the directory to the sampler; only the named assets are read, so no dataset is needed. The sampler's `manifest.json` records the asset of every motion, and `scripts/run_animate_motion.sh <samples_dir>` checks motion, cond and mesh before driving the mesh. With `--save_clips`, in-betweening and motion editing can hold the asset's own clips. See [Inference](../../README.md#-inference) in the top-level README.++## Command reference++### `run`++`python -m data_process.rig_preprocess run --input FILE [FILE ...] --output_dir DIR [options]`++| Option | Default | Description |+|---|---|---|+| `--input` | required | The asset; several `{Species}-{Action}.fbx` clips for the Truebones profile |+| `--output_dir` | required | The asset directory to write |+| `--profile` | `auto` | `auto`, `general`, `objaverse` or `truebones` |+| `--name` | from the file | Asset name (not with `--annotate dataset` or the Truebones profile) |+| `--annotate` | `llm` when the `--model` key is set (`DEEPSEEK_API_KEY` for the default, `OPENAI_API_KEY` for a `gpt-*` model) or a local model is used, else `rule` | `llm`, `rule` or `dataset` |+| `--annotate_names`, `--annotate_face` | `--annotate` | `rule` or `llm` for one of the two |+| `--face_r`, `--face_l`, `--body_axis` | none | The facing pair by hand |+| `--review` / `--no_review` | review, except with `--annotation` or `--annotate dataset` | Stop after the annotation and write the review files / build directly |+| `--annotation` | none | Build from a reviewed `annotation.json` (not with `--annotate llm` / `dataset`, `--annotate_names`, `--annotate_face`, `--face_r`, `--face_l`, `--body_axis`) |+| `--rest_rotation` | none | Axis turns in degrees, applied in order (`x180`, `x90`, `x90,y180`) |+| `--save_clips` | off | Keep the clips in `motions/` |+| `--formats` | `glb` | `glb` (always) and `fbx` |+| `--no_preview` | off | Skip `preview.png` |+| `--exp_dir` | none | Check the joint count against this run (default 71, the UniML3D models' limit) |+| `--fps` | `30` | Frame rate of the exported clips |+| `--keep_intermediate` | off | Keep `work/` |+| `--overwrite` | off | Replace an earlier output of this tool |+| `--dataset_export_dir` | `dataset/export/<profile>` | `--annotate dataset`: the export with the asset's reviewed entries |+| `--patch_dir` | `dataset/UniML3D/patches` | `--annotate dataset`: root-offset and rest-orientation patch files, applied when present (the released export already includes them) |++<a id="llm-options"></a>**LLM options** (an `llm` source): `--model` (an OpenAI, DeepSeek or Hugging Face model id), `--backend local|openai` (default: from the model), `--api_key` (default `OPENAI_API_KEY` / `DEEPSEEK_API_KEY`), `--base_url`, `--max_tokens` (default 8192), `--max_retries`, `--reasoning_effort`, `--torch_dtype`, `--device_map`, and `--rig_timeout` (seconds per call before the rule fallback, default 300). `run -h` lists defaults.++### `verify` and `evaluate`++```bash+python -m data_process.rig_preprocess verify --output_dir DIR [--profile DATASET] [--dataset_root dataset] [--report FILE]+python -m data_process.rig_preprocess.evaluate [--datasets objaverse truebones] [--split dev|test] [--rows FILE]+```++`verify` compares an output with the dataset's copy of the asset ([below](#reproducing-a-training-asset)) and exits 1 on a difference. `evaluate` scores `rule` annotation against the reviewed labels ([Accuracy](#accuracy)).++### Wrapper and Python API++`data_process/scripts/run_rig_preprocess.sh` maps environment variables to options and passes arguments after `--` to `run`:++| Variable | Option |+|---|---|+| `INPUT` (required; a glob or space-separated list, no spaces in paths), `OUTPUT_DIR` (required) | `--input`, `--output_dir` |+| `PROFILE`, `ANNOTATE`, `ANNOTATE_NAMES`, `ANNOTATE_FACE`, `NAME` | `--profile`, `--annotate`, `--annotate_names`, `--annotate_face`, `--name` |+| `FACE_R`, `FACE_L`, `BODY_AXIS=1` | `--face_r`, `--face_l`, `--body_axis` |+| `REVIEW=1` / `REVIEW=0`, `ANNOTATION` | `--review` / `--no_review`, `--annotation` |+| `REST_ROTATION`, `SAVE_CLIPS=1`, `FORMATS`, `EXP_DIR` | `--rest_rotation`, `--save_clips`, `--formats`, `--exp_dir` |+| `VERIFY=1` | run `verify` afterwards, when the asset was built |+| `CONDA_ENV` | environment to activate (default `unimate`) |++```python+from data_process.rig_preprocess.pipeline import preprocess # needs bpy+summary = preprocess(['robot.glb'], 'outputs/rig/robot', save_clips=True, face_r='R_hip', face_l='L_hip')+```++`preprocess` takes the `run` options as keyword arguments (`review=True`, `annotation=path`; unlike `run`, it builds directly unless `review=True`, and labels with `annotate_mode='rule'` unless told otherwise) and returns the `summary.json` dict. `verify.verify`, `evaluate.evaluate` and `review.build_annotation` / `review.load_annotation` do not need bpy.++## Reproducing a training asset++`--annotate dataset` takes a UniML3D asset's reviewed labels and fixes; `verify` compares the result with `<dataset_root>/features/<ds>/cond.npy` and `<dataset_root>/canonical_assets/<ds>/`:++```bash+python -m data_process.rig_preprocess run --input <training asset> --output_dir out --annotate dataset --profile objaverse+python -m data_process.rig_preprocess verify --output_dir out+```++Verdicts, for the cond and the canonical GLB separately: **`identical`** (within 1e-4 of the extent for positions and meshes, 1e-5 for rotations and eigenvectors); **`equivalent`**, the same skeleton described differently (sibling order, quaternion sign, bone axes with equal joint positions, eigenvector sign or another basis of a repeated Laplacian eigenvalue, chains split differently at a branch; listed under `notes`); **`different`** (listed under `differences`).++## Accuracy++**Rule annotation** against the reviewed UniML3D labels (LLM output, human review and fixes). Rigs are split into two halves by name hash; each half is scored with the vocabulary and label lookup learned on the other half only. Measured 2026-10-04 (`evaluate --split test`, `--split dev`):++| Held-out half | Joint labels | Same facing pair | Same facing (โค 10ยฐ) |+|---|---|---|---|+| Objaverse, test (3,669 rigs) / dev (3,686) | 93.9% / 94.8% | 91.5% / 92.8% | 97.9% / 98.4% |+| Truebones, test (33 species) / dev (41) | 89.9% / 91.6% | 97.0% / 92.7% | 97.0% / 97.6% |++The facing is what reaches the model; different pairs (hips or shoulders) usually give the same facing. Template rigs (Mixamo, Biped, CAT, Rigify) come out almost exactly as reviewed; errors concentrate in one-off rigs with unusual names and in robots, whose link names have the least reviewed vocabulary. Use `--review` for those.++**Review flags** on the same held-out test halves: on Objaverse, 84% of flagged joints were ones the reviewers relabelled, and the flags caught 42% of all relabelled joints; on Truebones, 44 of 44 flagged joints were relabelled (27% of all relabelled joints). The facing pair is flagged on 5% of Objaverse rigs, which hold a third of the wrong facings (25 of 77): a flagged pair means "check the arrow", not "wrong", and an unflagged one still deserves a look at the preview.++**Effect on generated motion.** Training assets were sampled with the same prompts and noise twice: with the dataset's reviewed conditioning and with this tool's `rule` output. With the same labels and facing pair, the motions are the same (mean joint difference below 0.2% of body size). With 57 to 92% of labels the same and the same pair, the motion changes less than a second sample of the prompt would. A different facing pair, or mostly wrong labels, gives a different motion.++## Limitations++- **Rotations only.** The model generates joint rotations and the root trajectory; sliding or stretching bones keep their rest length (reported in `summary.json`).+- **Rest orientation.** A rig authored lying down or upside down is not stood up automatically (an upside-down pose is flagged); use `--rest_rotation`.+- **No skinned mesh.** The cond is written, but there is no canonical GLB, and pruning keeps fewer joints, since skin weights mark deforming bones.+- **Joint budget.** Rigs over the model's limit are reported, not simplified.+- **Full character skeletons.** A Mixamo character keeps all its bones (65 for Y Bot), while UniML3D trains Mixamo motion on 22 core joints; such an asset is sampled as an unseen skeleton.+- **Automatic labels can be wrong** on rigs with unusual names (often robots), from the LLM or the rules: check them in the review step.++## Troubleshooting++| Message | Cause and fix |+|---|---|+| `profile 'objaverse' reads .glb/.gltf files, got โฆ` | Use `--profile general` or the default `auto` |+| `โฆ exists; pass --overwrite to redo the asset` | `--output_dir` holds an earlier output: add `--overwrite` or choose another directory |+| `โฆ is not a rig_preprocess output` | `--overwrite` only replaces a directory this tool wrote |+| `โฆ holds a review of โฆ` / `โฆ annotation.json exists and may hold edits` | Continue with the `--annotation` command shown, or add `--overwrite` (the old file becomes `annotation.json.bak`) |+| `โฆ: joints do not match the exported skeleton` / `empty label for โฆ` / `face_pair โฆ` / `must be true or false` | The edited `annotation.json` changed a raw name, left a label empty, has a wrong type, or names an incomplete, unknown or vertical facing pair: fix it and rerun the same command |+| `asset name โฆ must be non-empty and hold none of โฆ` | Pass a valid `--name` |+| `the exported skeleton has N root bones` | Parent the roots under one, or add the skinned mesh that tells which one deforms |+| `stage 4 produced no cond for โฆ` | Re-run with `--keep_intermediate`; read `work/features/extract_errors.log` and `filtered_clips.json` |+| `face pair โฆ has no horizontal separation in the rest pose` | The hand-given joints lie on a vertical line: choose two side by side, or a head / tail pair with `--body_axis` |+| `--annotate dataset needs a stage-3 dataset export` / `โฆ has no stage-3 entry` | `dataset` mode needs a training asset and its dataset export (`--dataset_export_dir`) |+| `--rest_rotation is for unseen assets` | `--annotate dataset` applies the dataset's own rest orientation |diff --git a/data_process/rig_preprocess/__init__.py b/data_process/rig_preprocess/__init__.pynew file mode 100644index 0000000..940e6f6--- /dev/null+++ b/data_process/rig_preprocess/__init__.py@@ -0,0 +1,11 @@+"""Turn one rigged 3D asset, with or without animation, into a UniMate asset.++An asset directory holds what sampling and mesh driving need about a skeleton:+its stage-4 cond (``cond.npy``), its canonical rest-pose GLB, a preview and a+summary. ``pipeline.preprocess`` (the entry point; CLI in ``cli``) chains+``export`` (stage 1), ``annotate`` (stage 3), stage 4 and the canonical bake+for one asset, reusing the dataset pipeline's own code; ``profiles`` holds the+per-dataset settings, ``verify`` compares an output with the dataset's copy,+``evaluate`` scores rule annotation. User guide: ``README.md``; maintainer+guide: ``AGENTS.md``.+"""diff --git a/data_process/rig_preprocess/__main__.py b/data_process/rig_preprocess/__main__.pynew file mode 100644index 0000000..46f6611--- /dev/null+++ b/data_process/rig_preprocess/__main__.py@@ -0,0 +1,10 @@+"""``python -m data_process.rig_preprocess`` -> :mod:`data_process.rig_preprocess.cli`."""++import os+import sys++sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))++from data_process.rig_preprocess.cli import main # noqa: E402++sys.exit(main())diff --git a/data_process/rig_preprocess/annotate.py b/data_process/rig_preprocess/annotate.pynew file mode 100644index 0000000..ee10754--- /dev/null+++ b/data_process/rig_preprocess/annotate.py@@ -0,0 +1,262 @@+"""Stage 3 for one asset: cleaned joint names and the facing pair.++Three sources, each producing the two stage-3 entries stage 4 reads+(``clean_joint_names.json``, ``face_joint_names.json``):++- ``dataset``: the dataset's own (reviewed and patched) entries for the asset,+ matched by raw joint name (a joint the dataset's skeleton lacks gets the rule+ label); reproduces a training asset exactly.+- ``rule``: the rule-based cleaner, its rig-level refinements+ (``joint_annotation/structure_rule.py``; geometric labels of+ ``skeleton_parse`` for a mostly unnamed rig) and the face resolver, then the+ automatic fixes ``tools/patch_annotations.py`` applies to every rig (root+ label unification, side-from-x on object datasets, face-label resync), then+ the reviewed-label lookup (``joint_annotation/name_map.py``: a known rig+ tree's labels, else confident per-name labels where the rules left a+ placeholder). Offline.+- ``llm``: the LLM cleaner and face selector (``joint_annotation/llm.py``+ backends), then the same automatic fixes; a failed or timed-out call, or+ an LLM pair without horizontal separation, keeps the ``rule`` result.++Outside ``dataset`` mode the joint names and the face pair can come from+different sources (``names_mode`` / ``face_mode``), e.g. rule names with an+LLM face pair.++A face pair given by hand (``face_r`` / ``face_l`` raw names) replaces the+selected one in any mode. A selected pair without horizontal separation in the+rest pose (coincident or vertical joints, which define no facing) is dropped+for the identity facing; a hand-given one is an error.+"""++import json+import os+import re+from collections import Counter, OrderedDict++from loguru import logger++import numpy as np++from data_process.joint_annotation.face_select_rule import empty_entry, resolve_face_joints+from data_process.joint_annotation.name_map import apply_name_map, load_name_map+from data_process.joint_annotation.names_clean_rule import (+ clean_joint_name,+ limb_context,+ post_process,+)+from data_process.joint_annotation.structure_rule import fill_unnamed, rig_context+from data_process.tools import patch_annotations+from data_process.utils.kinematics import rest_global+from data_process.utils.skeleton import FACING_EPS++MODES = ('rule', 'llm', 'dataset')+++def _patch_labels(ds, root, clean, names, log, generic_only):+ stats = Counter()+ saved = patch_annotations.CHAIN_RIGS, patch_annotations.X_SIDE_RIGS+ if generic_only:+ patch_annotations.CHAIN_RIGS, patch_annotations.X_SIDE_RIGS = {}, {}+ try:+ patch_annotations.patch_joint_labels(ds, root, clean, names, {}, stats, log)+ finally:+ patch_annotations.CHAIN_RIGS, patch_annotations.X_SIDE_RIGS = saved+++def rule_annotation(ds, root, names, log=None, generic_only=False, name_map=None):+ """Rule labels and face pairs for the rigs of *names* (``{rig: raw names}``)+ exported under *root*.++ Labels: the rule cleaner with its refinements (learned vocabulary, sided+ placeholders, end bones), the hierarchy's limb context, then the automatic+ label fixes of ``patch_annotations`` (root unification, side-from-x), then+ the reviewed-label lookup of *name_map* (default: the shipped+ ``learned_name_map.json.gz``; ``False``: none): a known rig tree's labels,+ else confident per-name labels in place of weak rule labels.++ The face pair is resolved on the rule labels without the learned+ vocabulary (richer labels steer the resolver to other parts more often than+ they help), except that a thigh pair of the refined labels wins over a+ plain pair of another part. The lookup does not move the pair. *generic_only* leaves out the+ fixes keyed by dataset rig name (``CHAIN_RIGS``, ``X_SIDE_RIGS``), which an+ unseen asset never gets.+ """+ log = [] if log is None else log+ clean = {r: [post_process(clean_joint_name(n, r)) for n in raw] for r, raw in names.items()}+ plain = {r: [post_process(clean_joint_name(n, r, refine=False)) for n in raw]+ for r, raw in names.items()}+ rest, parents = {}, {}+ for rig in names:+ npz = patch_annotations.rig_npz(root, ds, rig)+ if npz:+ with np.load(npz, allow_pickle=True) as data:+ parents[rig] = data['parents']+ rest[rig] = rest_global(data)+ clean[rig] = rig_context(limb_context(clean[rig], parents[rig]), list(names[rig]),+ parents[rig])+ clean[rig] = fill_unnamed(clean[rig], parents[rig], rest[rig])+ _patch_labels(ds, root, clean, names, log, generic_only)+ _patch_labels(ds, root, plain, names, [], generic_only)+ face = {}+ for rig in names:+ raw = list(names[rig])+ usable = _separation_check(rest[rig]) if rig in rest else None+ # the plain rule labels first; a rig they give no pair (unnamed joints),+ # or a pair of another part where the refined labels find thighs, takes+ # the refined labels' pair+ face[rig] = resolve_face_joints(plain[rig], raw, usable)+ refined = resolve_face_joints(clean[rig], raw, usable)+ if not face[rig]['r_hip']['raw'] or (+ _part(refined, raw, clean[rig]) == 'Thigh'+ and _part(face[rig], raw, clean[rig]) != 'Thigh'):+ face[rig] = refined+ if name_map is not False:+ lookup = load_name_map() if name_map is None else name_map+ for rig in names:+ clean[rig], _ = apply_name_map(lookup, list(names[rig]), clean[rig], parents.get(rig))+ patch_annotations.patch_face_pairs(ds, face, clean, names, {}, Counter(), log)+ return clean, face+++def _part(entry, raw, labels):+ """The refined label of a face entry's right joint, without its side+ ('' for an empty pair)."""+ joint = entry.get('r_hip', {}).get('raw')+ if not joint or entry.get('body_axis'):+ return ''+ return re.sub(r'^(Left|Right) ', '', labels[raw.index(joint)])+++def _separation_check(pos):+ """The resolver's pair predicate: horizontal separation in rest pose *pos*."""+ def usable(ri, li):+ return _separated(pos[ri], pos[li])+ return usable+++def _separated(a, b):+ across = np.asarray(a, dtype=float) - np.asarray(b, dtype=float)+ across = across / max(np.linalg.norm(across), 1e-8)+ return bool(np.linalg.norm(np.cross([0.0, 1.0, 0.0], across)) > FACING_EPS)+++def manual_face(raw, clean, face_r, face_l, body_axis=False):+ """A ``face_joint_names.json`` entry for the hand-given pair *face_r* /+ *face_l* (raw names of the exported skeleton, right or head first)."""+ for joint in (face_r, face_l):+ if joint not in raw:+ raise ValueError(f"face joint {joint!r} is not a joint of the exported skeleton")+ entry = OrderedDict([+ ('r_hip', {'raw': face_r, 'clean': clean[raw.index(face_r)]}),+ ('l_hip', {'raw': face_l, 'clean': clean[raw.index(face_l)]}),+ ('source', 'body_axis' if body_axis else 'manual'),+ ])+ if body_axis:+ entry['body_axis'] = True+ return entry+++def facing_defined(face, raw, rest_pos):+ """Whether the pair has horizontal separation in the rest pose (export+ space, Y-up), the condition stage 4's facing needs; an empty pair counts."""+ r, l = face.get('r_hip', {}).get('raw'), face.get('l_hip', {}).get('raw')+ if not (r and l):+ return True+ across = np.asarray(rest_pos[raw.index(r)]) - np.asarray(rest_pos[raw.index(l)])+ across = across / max(np.linalg.norm(across), 1e-8)+ return bool(np.linalg.norm(np.cross([0.0, 1.0, 0.0], across)) > FACING_EPS)+++def annotate(mode, ds, name, raw, export_dir, dataset_export_dir=None,+ face_r=None, face_l=None, body_axis=False, llm_client=None, llm_options=None,+ rest_pos=None, names_mode=None, face_mode=None):+ """Return ``(clean_names, face_entry, notes)`` for one asset.++ *raw* is the exported skeleton's joint names; *ds* the profile name (it+ selects dataset-specific automatic fixes); *dataset_export_dir* the+ dataset export holding the entries for ``mode='dataset'``; *llm_options*+ ``max_tokens`` / ``max_retries`` / ``timeout_seconds`` for both LLM calls.+ Outside ``dataset`` mode, *names_mode* and *face_mode* ('rule' / 'llm',+ default *mode*) choose the source of the joint names and of the face pair+ separately; an LLM call that fails falls back to the rule result.+ """+ log = []+ if mode == 'dataset':+ if not dataset_export_dir:+ raise ValueError("mode 'dataset' needs the dataset export directory")+ entries = {}+ for key, fname in (('raw', 'joint_names.json'), ('clean', 'clean_joint_names.json'),+ ('face', 'face_joint_names.json')):+ with open(os.path.join(dataset_export_dir, fname)) as f:+ entries[key] = json.load(f).get(name)+ if any(v is None for v in entries.values()):+ raise ValueError(f"'{name}' has no stage-3 entry in {dataset_export_dir}")+ # By raw joint name: a skeleton exported without animation can keep+ # joints the dataset's export pruned; those get the rule label.+ label = dict(zip(entries['raw'], entries['clean']))+ extra = [n for n in raw if n not in label]+ clean = [label[n] if n in label else post_process(clean_joint_name(n, name)) for n in raw]+ if extra:+ log.append(f'{len(extra)} joint(s) not in the dataset skeleton, rule-labelled: '+ f'{extra[:5]}')+ face = entries['face']+ for key in ('r_hip', 'l_hip'):+ joint = face.get(key, {}).get('raw')+ if joint and joint not in raw:+ raise ValueError(f"'{name}': dataset face joint {joint!r} is not in the "+ f"exported skeleton")+ elif mode in ('rule', 'llm'):+ names_mode, face_mode = names_mode or mode, face_mode or mode+ for m in (names_mode, face_mode):+ if m not in ('rule', 'llm'):+ raise ValueError(f"names / face source must be 'rule' or 'llm', got {m!r}")+ if 'llm' in (names_mode, face_mode) and llm_client is None:+ raise ValueError("an 'llm' names / face source needs an LLM client")+ # The rule result: the 'rule' source, and the fallback of a failed LLM call.+ clean_d, face_d = rule_annotation(ds, export_dir, {name: list(raw)}, log)+ clean, face = clean_d[name], face_d[name]+ opts = dict(llm_options or {})+ names = {name: list(raw)}+ if names_mode == 'llm':+ from data_process.joint_annotation.names_clean_llm import clean_rig+ llm_clean, fell_back = clean_rig(name, list(raw), llm_client, **opts)+ if fell_back:+ log.append('joint names: LLM failed, rule labels kept')+ else:+ llm_d = {name: list(llm_clean)}+ _patch_labels(ds, export_dir, llm_d, names, log, generic_only=False)+ clean = llm_d[name]+ if face_mode == 'llm':+ from data_process.joint_annotation.face_select_llm import resolve_rig+ llm_face, fell_back = resolve_rig(name, list(raw), list(clean), llm_client, **opts)+ if fell_back:+ log.append('face pair: LLM failed, rule pair kept')+ elif rest_pos is not None and not facing_defined(llm_face, list(raw), rest_pos):+ log.append(f"face pair: the LLM's {llm_face['r_hip']['raw']} / "+ f"{llm_face['l_hip']['raw']} has no horizontal separation in "+ f"the rest pose, rule pair kept")+ else:+ face = llm_face+ face_d = {name: face}+ patch_annotations.patch_face_pairs(ds, face_d, {name: clean}, names, {}, Counter(), log)+ face = face_d[name]+ log.append(f'sources: joint names {names_mode}, face pair {face_mode}')+ else:+ raise ValueError(f"mode must be one of {MODES}, got {mode!r}")++ if face_r or face_l:+ if not (face_r and face_l):+ raise ValueError("give both face_r and face_l")+ face = manual_face(list(raw), clean, face_r, face_l, body_axis)+ log.append(f'face pair given by hand: {face_r} / {face_l}')+ if rest_pos is not None and not facing_defined(face, list(raw), rest_pos):+ raise ValueError(f"face pair {face_r} / {face_l} has no horizontal separation "+ f"in the rest pose; choose another")+ elif rest_pos is not None and not facing_defined(face, list(raw), rest_pos):+ # Stage 4 would skip the asset; the authored facing is the better fallback.+ log.append(f"face pair {face['r_hip']['raw']} / {face['l_hip']['raw']} has no "+ f"horizontal separation in the rest pose; using none (identity facing)")+ face = empty_entry()+ for line in log:+ logger.info(f"[annotate] {line}")+ return list(clean), face, logdiff --git a/data_process/rig_preprocess/cli.py b/data_process/rig_preprocess/cli.pynew file mode 100644index 0000000..20b52fc--- /dev/null+++ b/data_process/rig_preprocess/cli.py@@ -0,0 +1,231 @@+"""Command line of ``rig_preprocess``.++ python -m data_process.rig_preprocess run --input ASSET [...] --output_dir DIR [options]+ python -m data_process.rig_preprocess verify --output_dir DIR [--profile P] [--report FILE]++``run`` needs the bpy module (the ``unimate`` env, or Blender's own Python);+``verify`` does not. Both exit non-zero on failure (``verify``: on any+``different`` verdict).+"""++import argparse+import json+import os+import sys++from loguru import logger++from data_process.rig_preprocess.annotate import MODES+from data_process.rig_preprocess.profiles import AUTO, PROFILES, pick_profile+++def _add_run(sub):+ ap = sub.add_parser('run', help="Process one asset into cond.npy + <name>.glb.")+ ap.add_argument('--input', nargs='+', required=True,+ help="The asset file (GLB/GLTF/FBX); for --profile truebones, the "+ "{Species}-{Action}.fbx clips of one species.")+ ap.add_argument('--output_dir', required=True,+ help="The asset directory to write (cond.npy, <name>.glb, preview.png, "+ "summary.json).")+ ap.add_argument('--profile', default=AUTO, choices=[AUTO] + sorted(PROFILES),+ help="Dataset whose processing to reproduce. auto (default): truebones "+ "for several {Species}-{Action}.fbx clips, the --dataset_export_dir "+ "dataset with --annotate dataset, else general.")+ ap.add_argument('--annotate', default=None, choices=MODES,+ help="Joint-name / facing-pair source: llm (default when an LLM is "+ "configured: an API key for --model, or a local model), rule "+ "(offline; the default otherwise), or dataset (the dataset's own "+ "entries; training assets only).")+ ap.add_argument('--annotate_names', default=None, choices=('rule', 'llm'),+ help="Joint-name source when it differs from --annotate (rule / llm modes).")+ ap.add_argument('--annotate_face', default=None, choices=('rule', 'llm'),+ help="Face-pair source when it differs from --annotate (rule / llm modes).")+ ap.add_argument('--name', default=None,+ help="Object type to give the asset (default: the exporter's name for it).")+ ap.add_argument('--face_r', default=None, help="Raw name of the right (or head) face joint.")+ ap.add_argument('--face_l', default=None, help="Raw name of the left (or tail) face joint.")+ ap.add_argument('--body_axis', action='store_true',+ help="The face pair is a head / tail axis.")+ ap.add_argument('--dataset_export_dir', default=None,+ help="Dataset export for --annotate dataset (default: dataset/export/<profile>).")+ ap.add_argument('--patch_dir', default='dataset/UniML3D/patches',+ help="Patch files for --annotate dataset.")+ ap.add_argument('--keep_intermediate', action='store_true',+ help="Keep work/ (export/, features/ with the training clips, "+ "canonical_assets/) in the dataset layout.")+ review = ap.add_mutually_exclusive_group()+ review.add_argument('--review', dest='review', action='store_true', default=None,+ help="Stop after the annotation, before the cond: write annotation.json, "+ "annotation_preview.png and REVIEW.md to check the joint labels "+ "and the facing pair (by hand, or with an LLM / coding agent). The "+ "default, except with --annotation or --annotate dataset.")+ review.add_argument('--no_review', dest='review', action='store_false',+ help="Build the cond directly from the automatic annotation.")+ ap.add_argument('--annotation', default=None,+ help="A reviewed annotation.json: its labels and facing pair replace the "+ "annotation (with --review: the review files are written from it).")+ ap.add_argument('--overwrite', action='store_true', help="Replace an earlier run's outputs.")+ ap.add_argument('--formats', default='glb',+ help="Canonical asset files, comma-separated: glb (always) and fbx.")+ ap.add_argument('--save_clips', action='store_true',+ help="Keep the asset's stage-4 feature clips in <output_dir>/motions/: "+ "the ground truth in-betweening and motion editing hold.")+ ap.add_argument('--no_preview', action='store_true', help="Skip preview.png.")+ ap.add_argument('--rest_rotation', default=None,+ help="Stand up a rest pose authored lying down or upside down: axis turns "+ "in degrees applied in order, e.g. x180, x90, x-90, x90,y180 "+ "(not with --annotate dataset).")+ ap.add_argument('--exp_dir', default=None,+ help="Training run whose joint width (config.json) the asset is checked "+ "against (default: the released UniML3D models' width, 71).")+ ap.add_argument('--fps', type=int, default=30,+ help="Sample rate of the exported clips (training reads 30 fps clips; "+ "the cond and canonical GLB do not depend on it).")+ ap.add_argument('--rig_timeout', type=int, default=300,+ help="LLM sources: seconds per LLM call before the rule fallback (0: none).")+ from data_process.joint_annotation.llm import add_llm_args+ add_llm_args(ap, default_max_tokens=8192)+++def _add_verify(sub):+ ap = sub.add_parser('verify', help="Compare an output with the dataset's copy of the asset.")+ ap.add_argument('--output_dir', required=True, help="A rig_preprocess output directory.")+ ap.add_argument('--profile', default=None,+ help="Dataset to compare with (default: the dataset annotated from, "+ "else the output's profile).")+ ap.add_argument('--dataset_root', default='dataset',+ help="Root holding features/<ds>/cond.npy and canonical_assets/<ds>/.")+ ap.add_argument('--report', default=None, help="Also write the report JSON here.")+++def _llm_configured(args):+ """Whether the LLM backend of *args* can run: a local model, or an API key+ for the model (``--api_key`` or its environment variable)."""+ from data_process.joint_annotation.llm import BACKEND_LOCAL, detect_backend+ if (args.backend or detect_backend(args.model)) == BACKEND_LOCAL or args.api_key:+ return True+ var = 'DEEPSEEK_API_KEY' if args.model.startswith('deepseek') else 'OPENAI_API_KEY'+ return bool(os.environ.get(var))+++def _annotate_mode(args):+ """``--annotate`` as given; otherwise ``rule`` when ``--annotation`` or a+ single-source flag (``--annotate_names`` / ``--annotate_face``) decides, so+ the other source stays the offline rules, else ``llm`` when an LLM is+ configured, else ``rule``."""+ if args.annotate is not None:+ return args.annotate+ if args.annotation or args.annotate_names or args.annotate_face:+ return 'rule'+ if _llm_configured(args):+ return 'llm'+ var = 'DEEPSEEK_API_KEY' if args.model.startswith('deepseek') else 'OPENAI_API_KEY'+ logger.warning(f"No LLM configured for --model {args.model} ({var} unset, no --api_key): "+ f"labelling with the offline rules; check the labels closely, or set the "+ f"key for LLM labels.")+ return 'rule'+++def _run(args):+ from data_process.rig_preprocess.pipeline import preprocess # needs bpy+ if bool(args.face_r) != bool(args.face_l):+ logger.error("--face_r and --face_l give the pair together; pass both or neither.")+ return 2+ if args.body_axis and not args.face_r:+ logger.error("--body_axis marks a hand-given head / tail pair; pass it with --face_r / --face_l.")+ return 2+ if args.face_r and args.annotate_face == 'llm':+ logger.error("--face_r / --face_l give the facing pair by hand; drop --annotate_face llm.")+ return 2+ args.annotate = _annotate_mode(args)+ if (args.face_r and args.face_l and args.annotate_face is None+ and args.annotate != 'dataset' and not args.annotation):+ args.annotate_face = 'rule' # the hand-given pair replaces any selected one+ llm_sources = [flag for flag, mode in (('--annotate_names', args.annotate_names or args.annotate),+ ('--annotate_face', args.annotate_face or args.annotate))+ if mode == 'llm']+ if args.annotate != 'dataset' and not args.annotation and llm_sources \+ and not _llm_configured(args):+ asked = '--annotate llm' if args.annotate == 'llm' else ' / '.join(f'{f} llm' for f in llm_sources)+ logger.error(f"{asked} needs an LLM for --model {args.model}: set its API key "+ f"(DEEPSEEK_API_KEY / OPENAI_API_KEY, or --api_key) or use a local model.")+ return 2+ client = None+ if args.annotate != 'dataset' and not args.annotation and 'llm' in (+ args.annotate_names or args.annotate, args.annotate_face or args.annotate):+ from data_process.joint_annotation.llm import LLMClient+ client = LLMClient.from_args(args)+ profile = args.profile+ if profile == AUTO:+ # what auto stands for, to default --dataset_export_dir; preprocess+ # resolves it again (and records that it was chosen automatically)+ profile = pick_profile(args.input, args.dataset_export_dir+ if args.annotate == 'dataset' else None)+ preprocess(args.input, args.output_dir, profile=args.profile,+ annotate_mode=args.annotate, name=args.name, face_r=args.face_r,+ face_l=args.face_l, body_axis=args.body_axis,+ dataset_export_dir=args.dataset_export_dir or os.path.join(+ 'dataset', 'export', profile),+ patch_dir=args.patch_dir, llm_client=client,+ llm_options=None if client is None else {'max_tokens': args.max_tokens,+ 'max_retries': args.max_retries,+ 'timeout_seconds': args.rig_timeout},+ overwrite=args.overwrite,+ keep_intermediate=args.keep_intermediate, fps=args.fps,+ names_mode=args.annotate_names, face_mode=args.annotate_face,+ exp_dir=args.exp_dir,+ formats=tuple(f.strip() for f in args.formats.split(',') if f.strip()),+ save_clips=args.save_clips, preview=not args.no_preview,+ rest_rotation=args.rest_rotation, review=_review(args),+ annotation=args.annotation)+ return 0+++def _review(args):+ """``--review`` / ``--no_review`` as given; otherwise review, unless the run+ continues from an ``--annotation`` file or takes the dataset's reviewed+ labels (``--annotate dataset``)."""+ if args.review is not None:+ return args.review+ return not (args.annotation or args.annotate == 'dataset')+++def _verify(args):+ from data_process.rig_preprocess.verify import verify+ with open(os.path.join(args.output_dir, 'summary.json')) as f:+ summary = json.load(f)+ report = verify(args.output_dir, summary['name'], args.profile or summary.get('reference') or summary['profile'],+ args.dataset_root)+ text = json.dumps(report, indent=2, default=str)+ print(text)+ if args.report:+ with open(args.report, 'w') as f:+ f.write(text)+ return int(any(report[k].get('verdict') == 'different' for k in ('cond', 'canonical_glb')))+++def main(argv=None):+ """Parse *argv* (default ``sys.argv``) and run ``run`` or ``verify``;+ returns the exit code."""+ ap = argparse.ArgumentParser(+ prog='python -m data_process.rig_preprocess',+ description="Process one rigged 3D asset (with or without animation) into the "+ "model's cond and canonical GLB, and check it against the dataset.")+ sub = ap.add_subparsers(dest='command', required=True)+ _add_run(sub)+ _add_verify(sub)+ args = ap.parse_args(argv)+ from data_process.joint_annotation.llm import FatalLLMError+ try:+ return _run(args) if args.command == 'run' else _verify(args)+ except FatalLLMError as exc:+ logger.error(f"rig_preprocess {args.command} failed: the LLM call failed ({exc}); fix the "+ f"API key or account, or label offline with --annotate rule.")+ return 1+ except (FileNotFoundError, FileExistsError, ValueError, RuntimeError) as exc:+ logger.error(f"rig_preprocess {args.command} failed: {exc}")+ return 1+++if __name__ == '__main__':+ sys.exit(main())diff --git a/data_process/rig_preprocess/evaluate.py b/data_process/rig_preprocess/evaluate.pynew file mode 100644index 0000000..8b18e2c--- /dev/null+++ b/data_process/rig_preprocess/evaluate.py@@ -0,0 +1,165 @@+"""Score the rule annotation against a dataset's reviewed stage-3 entries (bpy-free).++For every rig of ``dataset/export/<ds>`` it runs ``annotate.rule_annotation``+(what ``rule`` mode does; without the fixes keyed by dataset rig name, which an+unseen asset never gets) on the exported joint names and compares+with ``clean_joint_names.json`` / ``face_joint_names.json`` (LLM output plus+human review and patches):++- joint labels: fraction equal to the reviewed label;+- face pair: the same raw pair (and body-axis flag);+- facing: the root rotation the pair gives the rest pose (the one stage 4+ applies) within ``FACING_TOL_DEG`` of the reviewed pair's. Two different+ pairs often face the same way (shoulders vs hips), and only the facing+ reaches the cond and the canonical GLB. An empty pair is the identity.++Rigs can be restricted to a hash split (``--split dev|test``) so rules are+tuned on one half and judged on the other. With a split, the two learned+artifacts rule mode reads, the vocabulary (``learn_vocab``) and the reviewed-label+lookup (``name_map``), are learned on the other half for the run, so no scored+rig's own labels are looked up (``--shipped`` uses the shipped files instead,+which have seen every rig).++ python -m data_process.rig_preprocess.evaluate --datasets objaverse truebones --split test+"""++import argparse+import json+import os+import sys+from contextlib import contextmanager++import numpy as np++from data_process.joint_annotation import names_clean_rule+from data_process.joint_annotation.learn_vocab import in_split, learn+from data_process.joint_annotation.name_map import learn_name_map+from data_process.rig_preprocess.annotate import rule_annotation+from data_process.tools import patch_annotations+from data_process.utils.kinematics import rest_global+from data_process.utils.skeleton import DegenerateSkeletonError, get_root_facing_quat++FACING_TOL_DEG = 10.0+# Datasets the held-out vocabulary is learned from (learn_vocab's default).+VOCAB_DATASETS = ('truebones', 'mixamo', 'objaverse')+++@contextmanager+def _vocabulary(keys):+ """Rule labelling with the learned vocabulary *keys* instead of the shipped one."""+ cache = names_clean_rule._learned_vocab.__defaults__[0]+ saved = list(cache)+ cache[:] = [keys]+ try:+ yield+ finally:+ cache[:] = saved+++def _pair(entry):+ return (entry.get('r_hip', {}).get('raw') or '', entry.get('l_hip', {}).get('raw') or '',+ bool(entry.get('body_axis')))+++def facing_quat(entry, raw, rest_pos):+ """Root rotation (w, x, y, z) the pair gives the rest pose; None when undefined."""+ r, l, body_axis = _pair(entry)+ if not (r and l):+ return np.array([1.0, 0.0, 0.0, 0.0])+ if r not in raw or l not in raw:+ return None+ try:+ q = get_root_facing_quat(np.asarray(rest_pos)[None], [raw.index(r), raw.index(l)],+ body_axis=body_axis)+ except DegenerateSkeletonError:+ return None+ return q.qs[0]+++def quat_angle_deg(a, b):+ """Angle in degrees between two unit quaternions (sign-insensitive)."""+ return float(np.degrees(2 * np.arccos(min(1.0, abs(float(np.dot(a, b)))))))+++def evaluate(ds, export_root='dataset/export', split=None, rigs=None, held_out=True):+ """Score rule annotation on the rigs of ``<export_root>/<ds>`` (optionally+ one *split* half, or the rigs in *rigs*). With a split and *held_out*, the+ vocabulary and the reviewed-label lookup are learned on the other half.+ Returns ``(summary, rows)``: label accuracy, same pair and same facing over+ the rigs, and one row per rig."""+ root = os.path.join(export_root, ds)+ with open(os.path.join(root, 'joint_names.json')) as f:+ names = json.load(f)+ with open(os.path.join(root, 'clean_joint_names.json')) as f:+ ref_clean = json.load(f)+ with open(os.path.join(root, 'face_joint_names.json')) as f:+ ref_face = json.load(f)+ keep = [r for r in names if r in ref_clean and r in ref_face and in_split(r, split)+ and (rigs is None or r in rigs)]+ names = {r: names[r] for r in keep}+ if split and held_out:+ other = 'test' if split == 'dev' else 'dev'+ with _vocabulary(learn(VOCAB_DATASETS, export_root, split=other)):+ clean, face = rule_annotation(ds, root, names, generic_only=True,+ name_map=learn_name_map(export_root, split=other))+ else:+ clean, face = rule_annotation(ds, root, names, generic_only=True)++ n_lab = ok_lab = same_pair = same_facing = undefined = 0+ rows = []+ for rig in keep:+ raw = names[rig]+ ok = sum(a == b for a, b in zip(clean[rig], ref_clean[rig]))+ n_lab += len(raw)+ ok_lab += ok+ pair_ok = _pair(face[rig]) == _pair(ref_face[rig])+ same_pair += pair_ok+ npz = patch_annotations.rig_npz(root, ds, rig)+ angle = None+ if npz:+ with np.load(npz, allow_pickle=True) as d:+ rest = rest_global(d)+ qa, qb = facing_quat(face[rig], raw, rest), facing_quat(ref_face[rig], raw, rest)+ if qa is None or qb is None:+ undefined += 1+ else:+ angle = quat_angle_deg(qa, qb)+ facing_ok = pair_ok or (angle is not None and angle <= FACING_TOL_DEG)+ same_facing += facing_ok+ rows.append({'rig': rig, 'label_acc': ok / max(len(raw), 1), 'same_pair': pair_ok,+ 'facing_ok': facing_ok, 'angle': angle,+ 'rule_pair': _pair(face[rig]), 'ref_pair': _pair(ref_face[rig])})+ n = max(len(keep), 1)+ summary = {'dataset': ds, 'split': split, 'held_out': bool(split and held_out), 'rigs': len(keep),+ 'label_acc': ok_lab / max(n_lab, 1), 'same_pair': same_pair / n,+ 'same_facing': same_facing / n, 'facing_undefined': undefined}+ return summary, rows+++def main():+ """Command line: one summary line (JSON) per dataset."""+ ap = argparse.ArgumentParser(description="Score the rule annotation against a dataset.")+ ap.add_argument('--datasets', nargs='+', default=['truebones', 'mixamo', 'objaverse'])+ ap.add_argument('--export_root', default='dataset/export')+ ap.add_argument('--split', choices=('dev', 'test'), default=None)+ ap.add_argument('--rows', default=None, help="Write the per-rig rows (JSON) here.")+ ap.add_argument('--shipped', action='store_true',+ help="With --split: use the shipped vocabulary and lookup (learned on "+ "every rig, so they have seen the scored half) instead of learning "+ "them on the other half.")+ args = ap.parse_args()+ all_rows = {}+ for ds in args.datasets:+ if not os.path.isfile(os.path.join(args.export_root, ds, 'clean_joint_names.json')):+ print(f"skipping {ds}: no annotated export under {args.export_root}", file=sys.stderr)+ continue+ summary, rows = evaluate(ds, args.export_root, args.split, held_out=not args.shipped)+ all_rows[ds] = rows+ print(json.dumps(summary))+ if args.rows:+ with open(args.rows, 'w') as f:+ json.dump(all_rows, f, indent=1, default=str)+++if __name__ == '__main__':+ main()diff --git a/data_process/rig_preprocess/export.py b/data_process/rig_preprocess/export.pynew file mode 100644index 0000000..ec694cd--- /dev/null+++ b/data_process/rig_preprocess/export.py@@ -0,0 +1,148 @@+"""Stage 1 for one asset: export NPZs and the processed rest-pose GLB (bpy).++An animated asset goes through its profile's own exporter, so its NPZs are+what the dataset export wrote. An asset without animation gets the same+skeleton steps (import, secondary-root pruning, world-applied rest, Z-up ->+Y-up) and one placeholder clip holding the rest pose, ``<name>-rest_pose``,+so every later stage reads the usual export NPZ. Its skeleton is pruned by+the motion-free rules only (control-bone names, bones on their parent's+origin, unskinned single-child roots); the rules that need animation (static+leaves, stretching leaves, bones that never rotate) cannot be decided, so+unskinned bones that would fall to them (``*_End``, ``*Nub``, finger tips,+controllers) are kept. Treating the still pose as motion instead would prune+unskinned bones that an animated asset's clips do move.+"""++import os+from pathlib import Path++import numpy as np+from loguru import logger++from Animation import Animation+from Quaternions import Quaternions++from data_process.motion_export import export_general, export_objaverse, export_truebones+from data_process.utils.blender_export import (+ find_skinned_meshes,+ import_fbx,+ import_gltf,+ load_scene,+ mark_asset_complete,+ prepare_skeleton,+ prune_skeleton_shared,+ sanitize_object_type,+ save_motion,+)+from data_process.rig_preprocess.profiles import IMPORTER_EXTS+from data_process.utils.processed_assets import build_asset_glb++# The placeholder clip of an asset without animation.+REST_CLIP = 'rest_pose'+# Frames in that clip; any count works, stage 4 trims it as static.+REST_CLIP_FRAMES = 2+++def asset_key(profile, inputs):+ """The asset name the profile's exporter gives *inputs* (also the object type)."""+ exts = IMPORTER_EXTS[profile.importer]+ for p in inputs:+ if Path(p).suffix.lower() not in exts:+ raise ValueError(f"profile '{profile.name}' reads {'/'.join(exts)} files, got {p}")+ if profile.exporter == 'truebones':+ species = {Path(p).stem.split('-', 1)[0] for p in inputs}+ if len(species) != 1:+ raise ValueError(f"truebones inputs must be clips of one species, got {sorted(species)}")+ return sanitize_object_type(species.pop())+ if len(inputs) != 1:+ raise ValueError(f"profile '{profile.name}' takes one asset file, got {len(inputs)}")+ return export_general.asset_name(inputs[0])+++def _importer(profile, path):+ if profile.importer == 'gltf':+ return import_gltf+ if profile.importer == 'fbx':+ return import_fbx+ return export_general.pick_importer(path)+++def _has_bound_action(profile, path, fps):+ """Whether the file's armature carries an action (a Truebones clip FBX does)."""+ armature, _ = load_scene(_importer(profile, path), path, fps=fps)+ data = armature.animation_data+ return data is not None and data.action is not None+++def export_animated(profile, inputs, export_dir, name, fps=30):+ """Run the profile's exporter. Returns the joint names, or None when the+ asset has no clip (no pose action, or none long enough)."""+ if profile.exporter == 'objaverse':+ saved = export_objaverse.export_objaverse(inputs[0], name, export_dir, fps=fps,+ save_vis=False)+ elif profile.exporter == 'general':+ saved = export_general.export_asset(inputs[0], export_dir, save_name=name, fps=fps,+ remove_tpose=profile.remove_tpose, save_vis=False)+ elif profile.exporter == 'truebones':+ if len(inputs) == 1 and not _has_bound_action(profile, inputs[0], fps):+ return None # a rigged FBX without animation: the static path+ species = Path(inputs[0]).stem.split('-', 1)[0]+ saved = export_truebones.export_species(species, list(inputs), export_dir, fps=fps,+ save_vis=False)+ else:+ raise ValueError(f"unknown exporter {profile.exporter!r}")+ return saved.get(name)+++def export_static(profile, path, export_dir, name, fps=30):+ """Export an asset without animation as one rest-pose clip. Returns the joint names."""+ armature, mesh = load_scene(_importer(profile, path), path, fps=fps)+ if profile.all_skinned_meshes:+ meshes = find_skinned_meshes(armature) or ([mesh] if mesh is not None else [])+ skel = prepare_skeleton(armature, meshes or None, apply_world=True)+ else:+ if mesh is None:+ raise ValueError(f"no mesh in {path}; profile '{profile.name}' needs one")+ skel = prepare_skeleton(armature, mesh, apply_world=True)++ rest = skel['rest_anim_shared']+ clip = Animation(+ rotations=Quaternions(np.repeat(rest.rotations.qs, REST_CLIP_FRAMES, axis=0)),+ positions=np.repeat(rest.positions, REST_CLIP_FRAMES, axis=0),+ orients=rest.orients.copy(), offsets=rest.offsets.copy(),+ parents=rest.parents.copy())+ names = list(skel['bone_names'])+ skin_matrix = skel['skin_matrix']+ if mesh is not None:+ # With a still clip the position rules reduce to the rest-pose ones,+ # and rot_eps=0 switches off the rotation rule, which would otherwise+ # prune every unskinned bone as never moving (also ones the clips+ # move, such as a Mixamo rig's unskinned legs; see the module docstring).+ anims, rest, names, skin_matrix = prune_skeleton_shared(+ [clip], rest, names, skin_matrix.copy(), rot_eps=0.0,+ end_effector_suffixes=profile.end_effector_suffixes)+ clip = anims[0]++ clip_name = f'{name}-{REST_CLIP}'+ save_motion(os.path.join(export_dir, 'motions', f'{clip_name}.npz'),+ os.path.join(export_dir, 'videos', f'{clip_name}.mp4'),+ clip, rest, names, skin_matrix, fps, action_name=REST_CLIP, save_vis=False)+ mark_asset_complete(export_dir, name, n_clips=1, joint_names=names)+ logger.info(f"'{name}' has no animation: exported its rest pose as '{clip_name}' "+ f"({len(names)} joints, motion-free pruning)")+ return names+++def build_processed_glb(profile, inputs, export_dir, name, names, animated=True, fps=30):+ """``<export_dir>/rigs/<name>.glb``, as ``--save_glb`` builds it; after any+ rest-orientation patch, which the GLB follows. A static asset builds it+ from its one file and rest-pose clip. Returns its path or None."""+ if profile.exporter == 'truebones' and animated:+ export_truebones.save_species_glb(name, list(inputs), export_dir, fps=fps, want=names)+ else:+ clips = sorted((Path(export_dir) / 'motions').glob(f'{name}-*.npz'))+ build_asset_glb(name, str(clips[0]), export_dir,+ lambda: load_scene(_importer(profile, inputs[0]), inputs[0], fps=fps)[0],+ source_path=inputs[0])+ path = os.path.join(export_dir, 'rigs', f'{name}.glb')+ return path if os.path.isfile(path) else Nonediff --git a/data_process/rig_preprocess/glb.py b/data_process/rig_preprocess/glb.pynew file mode 100644index 0000000..b7890ea--- /dev/null+++ b/data_process/rig_preprocess/glb.py@@ -0,0 +1,112 @@+"""Minimal bpy-free GLB reader: node transforms, skin joints, skinned rest geometry."""++import json+import struct++import numpy as np++_COMPONENTS = {5120: np.int8, 5121: np.uint8, 5122: np.int16, 5123: np.uint16,+ 5125: np.uint32, 5126: np.float32}+_SIZES = {'SCALAR': 1, 'VEC2': 2, 'VEC3': 3, 'VEC4': 4, 'MAT4': 16}+_JSON_CHUNK, _BIN_CHUNK = 0x4E4F534A, 0x004E4942+++class Glb:+ """One GLB file: its glTF JSON (``json``), binary chunk and node list."""++ def __init__(self, path):+ with open(path, 'rb') as f:+ data = f.read()+ self.json, self.bin = {}, b''+ offset = 12+ while offset < len(data):+ length, kind = struct.unpack_from('<II', data, offset)+ chunk = data[offset + 8:offset + 8 + length]+ if kind == _JSON_CHUNK:+ self.json = json.loads(chunk)+ elif kind == _BIN_CHUNK:+ self.bin = chunk+ offset += 8 + length+ self.nodes = self.json.get('nodes', [])++ def accessor(self, index):+ """Accessor *index* as a float64 ``(count, components)`` array."""+ acc = self.json['accessors'][index]+ view = self.json['bufferViews'][acc['bufferView']]+ dtype = np.dtype(_COMPONENTS[acc['componentType']])+ width = _SIZES[acc['type']]+ start = view.get('byteOffset', 0) + acc.get('byteOffset', 0)+ stride = view.get('byteStride', 0) or dtype.itemsize * width+ raw = np.frombuffer(self.bin, dtype=np.uint8, offset=start,+ count=stride * (acc['count'] - 1) + dtype.itemsize * width)+ rows = np.lib.stride_tricks.as_strided(raw, (acc['count'], dtype.itemsize * width),+ (stride, 1))+ arr = np.ascontiguousarray(rows).view(dtype).reshape(acc['count'], width)+ arr = arr.astype(np.float64)+ if acc.get('normalized'):+ arr /= np.iinfo(dtype).max+ return arr++ @staticmethod+ def _local(node):+ if 'matrix' in node:+ return np.array(node['matrix'], dtype=np.float64).reshape(4, 4).T+ x, y, z, w = node.get('rotation', [0, 0, 0, 1])+ rot = np.array([[1 - 2 * (y * y + z * z), 2 * (x * y - z * w), 2 * (x * z + y * w)],+ [2 * (x * y + z * w), 1 - 2 * (x * x + z * z), 2 * (y * z - x * w)],+ [2 * (x * z - y * w), 2 * (y * z + x * w), 1 - 2 * (x * x + y * y)]])+ mat = np.eye(4)+ mat[:3, :3] = rot * np.array(node.get('scale', [1, 1, 1]))+ mat[:3, 3] = node.get('translation', [0, 0, 0])+ return mat++ def world(self):+ """World matrices of every node, ``(N, 4, 4)``."""+ parent = {c: i for i, n in enumerate(self.nodes) for c in n.get('children', [])}+ out = [None] * len(self.nodes)++ def get(i):+ if out[i] is None:+ local = self._local(self.nodes[i])+ out[i] = local if i not in parent else get(parent[i]) @ local+ return out[i]+ return np.stack([get(i) for i in range(len(self.nodes))]) if self.nodes \+ else np.zeros((0, 4, 4))++ def joints(self):+ """``{name: world 4x4}`` of every skin joint."""+ world = self.world()+ return {self.nodes[j].get('name'): world[j]+ for skin in self.json.get('skins', []) for j in skin['joints']}++ def skinned_triangles(self):+ """Rest-pose triangles of every skinned primitive as ``(T, 9)``, and the+ dominant joint name of each of their corners, ``(T, 3)``."""+ world = self.world()+ tris, dominant = [], []+ for node in self.nodes:+ if 'mesh' not in node or 'skin' not in node:+ continue+ skin = self.json['skins'][node['skin']]+ joints = skin['joints']+ ibm = self.accessor(skin['inverseBindMatrices']).reshape(-1, 4, 4).transpose(0, 2, 1)+ joint_mats = world[joints] @ ibm+ names = np.array([self.nodes[j].get('name') for j in joints])+ for prim in self.json['meshes'][node['mesh']]['primitives']:+ attrs = prim['attributes']+ if 'JOINTS_0' not in attrs:+ continue+ pos = self.accessor(attrs['POSITION'])+ jidx = self.accessor(attrs['JOINTS_0']).astype(int)+ wts = self.accessor(attrs['WEIGHTS_0'])+ wts = wts / np.maximum(wts.sum(1, keepdims=True), 1e-12)+ homo = np.c_[pos, np.ones(len(pos))]+ verts = sum(wts[:, k:k + 1] * np.einsum('nij,nj->ni', joint_mats[jidx[:, k]], homo)[:, :3]+ for k in range(jidx.shape[1]))+ faces = (self.accessor(prim['indices']).astype(int).reshape(-1, 3)+ if 'indices' in prim else np.arange(len(pos)).reshape(-1, 3))+ tris.append(verts[faces].reshape(-1, 9))+ dominant.append(names[jidx[np.arange(len(jidx)), wts.argmax(1)]][faces])+ if not tris:+ return np.zeros((0, 9)), np.zeros((0, 3), dtype=object)+ return np.concatenate(tris), np.concatenate(dominant)diff --git a/data_process/rig_preprocess/pipeline.py b/data_process/rig_preprocess/pipeline.pynew file mode 100644index 0000000..04c149a--- /dev/null+++ b/data_process/rig_preprocess/pipeline.py@@ -0,0 +1,637 @@+"""Turn one rigged 3D asset, with or without animation, into training data.++One command runs, for a single asset, the stages the dataset pipeline runs+over a whole dataset, calling the same code so the result is what that+pipeline would have produced. Every stage writes into ``<output_dir>/work/``+in the dataset layout:++ 1. export the profile's stage-1 exporter (``rig_preprocess.export``);+ an asset without animation becomes one rest-pose clip. A+ skeleton with several roots is refused.+ 2. patches ``--annotate dataset``: the dataset's root-offset and+ rest-orientation fixes, filter / trim / activity lists and+ captions for this asset. ``--rest_rotation``: the same+ rest-orientation fix with a hand-given turn.+ 3. rig GLB ``export/rigs/<name>.glb``, as ``--save_glb`` builds it.+ 4. annotate cleaned joint names and the facing pair+ (``rig_preprocess.annotate``, or a reviewed ``--annotation``+ file), plus the upside-down check. ``--review`` stops here and+ writes ``annotation.json``, ``annotation_preview.png`` and+ ``REVIEW.md`` (``rig_preprocess.review``) instead of the+ outputs below.+ 5. features stage 4 (``feature_extraction.extract_features``) with the+ profile's arguments: ``features/cond.npy`` and ``motions/``.+ An asset whose clips are all filtered (or that has none) still+ gets its cond entry, built from its rest pose.+ 6. canonical ``canonical_assets/<name>.glb`` (``canonical_assets.bake_canonical_assets``).+ An asset without a mesh (or whose GLB cannot be built) still+ gets its cond; ``summary.json`` says why there is no GLB.+ 7. deliver the outputs below, then ``work/`` is removed (unless+ ``--keep_intermediate``).++Outputs in ``--output_dir``::++ cond.npy {name: cond}, the stage-4 topology condition of the asset+ <name>.glb the canonical asset: rebuilt to the cond's T-pose, joint order+ stored in the file (what the model's motions drive);+ ``<name>.fbx`` beside it with ``--formats glb,fbx``+ preview.png the canonical T-pose with its facing pair, forward direction+ and cleaned joint labels (``rig_preprocess/preview.py``)+ motions/ with ``--save_clips``: the asset's stage-4 feature clips, the+ ground truth sampling can hold (in-betweening, motion editing)+ summary.json the asset name, profile, stats dataset, annotation sources,+ face pair, counts, and notes on what the model cannot+ reproduce (README.md lists every field)++The output directory is an *asset*: ``python -m unimate.inference.sample+--asset <output_dir>`` samples on it, and the samples' manifest drives+``<name>.glb`` (``scripts/run_animate_motion.sh <samples_dir>``).++``--keep_intermediate`` keeps ``work/`` (``export/``: NPZs, ``rigs/<name>.glb``,+the stage-3 JSONs; ``features/``: ``cond.npy``, the training clips+``motions/*.npz``; ``canonical_assets/``).++Run it through the package CLI (``python -m data_process.rig_preprocess run``,+see ``cli.py``) or call :func:`preprocess`.+"""++import glob+import json+import os+import re+import shlex+import shutil++from pathlib import Path++import numpy as np+from loguru import logger++from data_process.feature_extraction import extract_features+from data_process.rig_preprocess.annotate import annotate+from data_process.rig_preprocess.export import (+ asset_key,+ build_processed_glb,+ export_animated,+ export_static,+)+from data_process.rig_preprocess.profiles import AUTO, PROFILES, pick_profile+from data_process.rig_preprocess.review import (+ ANNOTATION_FILE,+ REVIEW_FILE,+ REVIEW_PREVIEW,+ build_annotation,+ load_annotation,+ read_annotation,+ write_review,+)+from data_process.tools import patch_annotations+from data_process.utils.kinematics import rest_global++# Per-asset sidecars of a dataset export that stage 4 reads (copied whole,+# stage 4 looks up only this asset's clips) and the caption files (copied+# down to this asset's clips).+LIST_SIDECARS = ('filtered_clips.txt', 'clip_trims.txt', 'activity_keep.txt',+ 'filtered_objects.txt')+CAPTION_SIDECARS = ('motion_captions.json', 'motion_captions_generic.json',+ 'motion_captions_detail.json')+# The dataset-layout intermediates, under <output_dir>/<WORK_DIR>/.+WORK_DIR = 'work'+SUBDIRS = ('export', 'features', 'canonical_assets')+# Characters an asset name cannot hold: '-' separates the object type from the+# clip id, '/' a directory, and glob patterns ('{name}-*.npz') would read+# [ ] * ? as wildcards and find none of the asset's clips.+_GLOB_CHARS = re.compile(r'[\[\]*?]')+_BAD_NAME = re.compile(r'[-/\\\[\]*?]')+# Joints a model can take: the padded joint width of the released UniML3D+# models (dataset.max_joints 70, +1 for joint addition); a run's own config.json+# (--exp_dir) gives its exact width.+MODEL_MAX_JOINTS = 71+# Stage 4 keeps only the root's animated translation; a clip whose other joints+# translate by more than this share of the skeleton extent loses visible motion.+TRANSLATION_NOTE_TOL = 0.01+CANONICAL_FORMATS = ('glb', 'fbx')+# Clean-label words of the joints the upside-down check compares.+_HEAD_WORDS = ('Head',)+_FOOT_WORDS = ('Foot', 'Toe')+# Feet this far above the head (share of the rest-pose extent) mean the rest+# pose was authored upside down.+UPSIDE_DOWN_TOL = 0.1+++def parse_rest_rotation(spec):+ """Quaternion (w, x, y, z) of ``--rest_rotation``: comma-separated axis turns+ applied in order, e.g. ``x180`` (upside down), ``x90`` / ``x-90`` (lying on+ its back / front), ``x90,y180``. Angles in degrees, about the export's+ Y-up axes."""+ q = np.array([1.0, 0.0, 0.0, 0.0])+ for part in [p.strip().lower() for p in spec.split(',') if p.strip()]:+ m = re.fullmatch(r'([xyz])\s*([+-]?\d+(?:\.\d+)?)', part)+ if not m:+ raise ValueError(f"--rest_rotation {spec!r}: expected turns like 'x180' or 'x90,y180'")+ half = np.radians(float(m.group(2))) / 2+ axis = np.eye(3)['xyz'.index(m.group(1))]+ turn = np.concatenate([[np.cos(half)], np.sin(half) * axis])+ w1, v1, w2, v2 = turn[0], turn[1:], q[0], q[1:]+ q = np.concatenate([[w1 * w2 - v1 @ v2], w1 * v2 + w2 * v1 + np.cross(v1, v2)])+ return q / np.linalg.norm(q)+++def upside_down_note(clean, rest_pos):+ """A note when the rest pose stands on its head (the mean height of the+ Foot / Toe joints above that of the Head joints), else None."""+ pos = np.asarray(rest_pos, dtype=np.float64)+ head = [i for i, n in enumerate(clean) if any(w in n.split() for w in _HEAD_WORDS)+ and not n.endswith(' End')]+ feet = [i for i, n in enumerate(clean) if any(w in n.split() for w in _FOOT_WORDS)]+ if not head or not feet:+ return None+ extent = float(np.ptp(pos, axis=0).max()) or 1.0+ if pos[feet, 1].mean() - pos[head, 1].mean() > UPSIDE_DOWN_TOL * extent:+ return ("the rest pose looks upside down (its feet are above its head); check "+ "preview.png and rerun with --rest_rotation x180 (or x90 / x-90 for a "+ "pose lying on its back / front)")+ return None+++def _asset_name(prof, inputs, name):+ """The asset's name: *name*, else the exporter's name for the file with+ glob characters replaced. Raises on a name stage 1 / stage 4 cannot use."""+ key = asset_key(prof, inputs)+ if _GLOB_CHARS.search(key):+ if prof.exporter == 'truebones':+ raise ValueError(f"species name {key!r} holds one of '[]*?'; rename the clip files")+ safe = _GLOB_CHARS.sub('_', key)+ logger.info(f"asset name {key!r} -> {safe!r} ('[]*?' would act as glob wildcards)")+ key = safe+ name = name or key+ if not name or _BAD_NAME.search(name):+ raise ValueError(f"asset name {name!r} must be non-empty and hold none of '-/\\[]*?'")+ return name, key+++def model_joint_limit(exp_dir=None):+ """``(limit, source)``: the joint count a model takes, from a run directory's+ config.json (its padded ``dataset.max_joints``), else ``MODEL_MAX_JOINTS``."""+ if exp_dir:+ path = os.path.join(exp_dir, 'config.json')+ with open(path) as f:+ return int(json.load(f)['dataset']['max_joints']), path+ return MODEL_MAX_JOINTS, 'released UniML3D models'+++def _save_json(path, obj):+ with open(path, 'w') as f:+ json.dump(obj, f, indent=2)+++def _copy_dataset_sidecars(name, dataset_export_dir, export_dir):+ copied = []+ for fname in LIST_SIDECARS:+ src = os.path.join(dataset_export_dir, fname)+ if os.path.isfile(src):+ shutil.copyfile(src, os.path.join(export_dir, fname))+ copied.append(fname)+ for fname in CAPTION_SIDECARS:+ src = os.path.join(dataset_export_dir, fname)+ if os.path.isfile(src):+ with open(src) as f:+ caps = json.load(f)+ _save_json(os.path.join(export_dir, fname),+ {k: v for k, v in caps.items() if k.split('-', 1)[0] == name})+ copied.append(fname)+ return copied+++def _dropped_translation(export_dir, name):+ """Largest animated translation of a non-root joint away from its rest+ offset over the asset's export clips, relative to the rest-pose extent:+ ``(share, clip, joint)``. Stage 4 keeps rotations and the root's path+ only, so this motion (sliding joints, stretching bones) is lost."""+ worst = (0.0, None, None)+ for path in sorted(Path(export_dir, 'motions').glob(f'{glob.escape(name)}-*.npz')):+ with np.load(path, allow_pickle=True) as npz:+ rest = np.asarray(npz['rest_local_pos'], dtype=np.float64)+ anim = np.asarray(npz['anim_local_pos'], dtype=np.float64)+ extent = float(np.ptp(rest_global(npz), axis=0).max()) or 1.0+ if anim.shape[1] < 2:+ continue+ dev = np.linalg.norm(anim[:, 1:] - rest[None, 1:], axis=-1).max(axis=0) / extent+ j = int(np.argmax(dev))+ if dev[j] > worst[0]:+ worst = (float(dev[j]), path.stem, str(npz['names'][j + 1]))+ return worst+++def _is_own_summary(path):+ """Whether *path* is a summary.json this package wrote (not, e.g., an+ export directory's merged summary)."""+ try:+ with open(path) as f:+ info = json.load(f)+ except (OSError, ValueError):+ return False+ return isinstance(info, dict) and {'name', 'profile', 'inputs'} <= set(info)+++def _summary_field(path, key):+ """Field *key* of an earlier output's summary.json (``stage`` is 'review'+ for a ``--review`` run), or None."""+ if not _is_own_summary(path):+ return None+ with open(path) as f:+ return json.load(f).get(key)+++def _unused_path(path):+ """*path*, or *path* with the first free number appended."""+ n, candidate = 1, path+ while os.path.exists(candidate):+ n += 1+ candidate = f'{path}{n}'+ return candidate+++def _display_path(path):+ """*path* relative to the working directory when it lies below it (as the+ other summary paths are), else absolute."""+ rel = os.path.relpath(path)+ return path if rel == os.pardir or rel.startswith(os.pardir + os.sep) else rel+++def _resume_command(inputs, output_dir, profile, name, key, rest_rotation, annotation_path,+ options):+ """The command that continues a ``--review`` run from its annotation, with+ the run's other non-default options (*options*: flag -> value, True for a+ switch)."""+ def q(arg):+ return shlex.quote(os.fspath(arg))+ cmd = ['python -m data_process.rig_preprocess run --input', *map(q, inputs),+ '--output_dir', q(output_dir), '--profile', profile]+ if name != key:+ cmd += ['--name', q(name)]+ if rest_rotation:+ cmd += ['--rest_rotation', q(rest_rotation)]+ for flag, value in options.items():+ if value is True:+ cmd.append(flag)+ elif value and value is not False and not (flag == '--formats' and value == 'glb'):+ cmd += [flag, q(str(value))]+ cmd += ['--annotation', q(annotation_path)]+ return ' '.join(cmd)+++def _stop_for_review(summary, notes, output_dir, work, keep_intermediate, names, parents,+ rest_pos, clean, face, annotation_path, resume):+ """Write the review files and the review-stage summary.json; no cond yet."""+ mode = summary.get('annotate_names') or summary['annotate']+ data = build_annotation(summary['name'], names, parents, rest_pos, clean, face, mode=mode,+ notes=notes)+ write_review(output_dir, data, names, parents, rest_pos, face, resume)+ summary.update({'stage': 'review', 'review': {+ 'annotation': annotation_path, 'flagged_joints': data['n_flagged'],+ 'face_pair_flagged': data['face_pair']['check'], 'resume': resume}})+ summary['notes'] = notes+ if not keep_intermediate:+ shutil.rmtree(work)+ summary['rig_glb'] = None+ _save_json(os.path.join(output_dir, 'summary.json'), summary)+ logger.info(f"'{summary['name']}': annotation ready for review, {data['n_flagged']} of "+ f"{len(names)} joints flagged"+ + (", the facing pair flagged" if data['face_pair']['check'] else '')+ + f". Read {os.path.join(output_dir, REVIEW_FILE)}, edit {annotation_path}, "+ f"then continue:\n {resume}")+ return summary+++def _root_count(npz_path):+ with np.load(npz_path, allow_pickle=True) as npz:+ return int((np.asarray(npz['parents']) < 0).sum())+++def _cond_entry(features_dir, name):+ """This asset's cond: from cond.npy, else from stage 4's cache entry, which+ holds it even when every clip was filtered (cond.npy lists only objects+ with clips)."""+ cond = np.load(os.path.join(features_dir, 'cond.npy'), allow_pickle=True).item()+ if name in cond:+ return cond[name], True+ part = os.path.join(features_dir, 'cond_parts', f'{name}.npy')+ if os.path.isfile(part):+ entry = np.load(part, allow_pickle=True).item().get('result', {}).get('cond')+ if entry is not None:+ return entry, False+ return None, False+++def preprocess(inputs, output_dir, profile=AUTO, annotate_mode='rule', name=None,+ face_r=None, face_l=None, body_axis=False, dataset_export_dir=None,+ patch_dir=None, llm_client=None, llm_options=None, overwrite=False,+ keep_intermediate=False, fps=30, names_mode=None, face_mode=None,+ exp_dir=None, formats=('glb',), save_clips=False, preview=True,+ rest_rotation=None, review=False, annotation=None):+ """Run every stage on one asset; see the module docstring. Returns the summary dict.+ *profile* ``auto`` is :func:`profiles.pick_profile`; *names_mode* /+ *face_mode* ('rule' / 'llm', default *annotate_mode*) pick the joint-name and+ face-pair sources separately outside ``dataset`` mode; *exp_dir* a training+ run whose joint width the asset is checked against; *formats* the canonical+ asset's files (``glb`` always); *save_clips* keeps the feature clips in+ ``<output_dir>/motions/``; *preview* writes ``preview.png``; *rest_rotation*+ (:func:`parse_rest_rotation`) stands up a rest pose authored lying down or+ upside down, as the dataset's rest-orientation patches do. *review* stops+ after the annotation and writes ``annotation.json``, ``annotation_preview.png``+ and ``REVIEW.md`` for a person, an LLM or an agent to check+ (``rig_preprocess/review.py``); *annotation* takes the joint labels and the+ facing pair from such a file, edited, instead of annotating."""+ requested_profile = profile+ if profile == AUTO:+ profile = pick_profile(inputs, dataset_export_dir if annotate_mode == 'dataset' else None)+ logger.info(f"profile: {profile} (auto)")+ if profile not in PROFILES:+ raise ValueError(f"unknown profile {profile!r}; one of {sorted(PROFILES)} or {AUTO!r}")+ formats = tuple(dict.fromkeys(('glb',) + tuple(formats)))+ bad = sorted(set(formats) - set(CANONICAL_FORMATS))+ if bad:+ raise ValueError(f"canonical asset formats {bad}: expected a subset of {CANONICAL_FORMATS}")+ prof = PROFILES[profile]+ inputs = [os.fspath(p) for p in inputs]+ for p in inputs:+ if not os.path.isfile(p):+ raise FileNotFoundError(p)+ name, key = _asset_name(prof, inputs, name)+ joint_limit, limit_source = model_joint_limit(exp_dir)+ if annotate_mode == 'dataset' and (names_mode or face_mode):+ raise ValueError("--annotate dataset takes both the names and the face pair from "+ "the dataset; drop --annotate_names / --annotate_face")+ if annotate_mode == 'dataset' and not (dataset_export_dir and os.path.isfile(+ os.path.join(dataset_export_dir, 'clean_joint_names.json'))):+ raise ValueError(f"--annotate dataset needs a stage-3 dataset export, got "+ f"{dataset_export_dir!r}")+ rest_quat = parse_rest_rotation(rest_rotation) if rest_rotation else None+ if rest_quat is not None and annotate_mode == 'dataset':+ raise ValueError("--rest_rotation is for unseen assets; --annotate dataset applies the "+ "dataset's own rest-orientation patch")+ if annotation is not None:+ if annotate_mode != 'rule' or names_mode or face_mode or face_r or face_l or body_axis:+ raise ValueError("--annotation gives the labels and the face pair; drop --annotate / "+ "--annotate_names / --annotate_face / --face_r / --face_l / "+ "--body_axis")+ if not os.path.isfile(annotation):+ raise FileNotFoundError(annotation)+ annotation = os.path.abspath(annotation)+ read_annotation(annotation) # structure, before anything is removed+ if review and annotate_mode == 'dataset':+ raise ValueError("--review checks rule / llm annotation; --annotate dataset already "+ "takes the dataset's reviewed labels")+ if name != key and (annotate_mode == 'dataset' or prof.exporter == 'truebones'):+ raise ValueError("--name cannot rename an asset annotated from the dataset or a "+ "Truebones species (named after its clip files)")++ cond_path = os.path.join(output_dir, 'cond.npy')+ asset_paths = {fmt: os.path.join(output_dir, f'{name}.{fmt}') for fmt in CANONICAL_FORMATS}+ glb_path = asset_paths['glb']+ work = os.path.join(output_dir, WORK_DIR)+ summary_path = os.path.join(output_dir, 'summary.json')+ preview_path = os.path.join(output_dir, 'preview.png')+ clips_dir = os.path.join(output_dir, 'motions')+ review_paths = [os.path.join(output_dir, f) for f in (REVIEW_FILE, REVIEW_PREVIEW)]+ annotation_path = os.path.join(output_dir, ANNOTATION_FILE)+ present = [p for p in (cond_path, *asset_paths.values(), work, summary_path, preview_path,+ clips_dir, *review_paths) if os.path.exists(p)]+ # Only an earlier output (its own summary.json) or an interrupted run's+ # work/ is ever removed: a feature or export directory also holds cond.npy /+ # motions/ / a summary.json of its own.+ if present and not (_is_own_summary(summary_path) or present == [work]):+ raise FileExistsError(+ f"{output_dir} holds {', '.join(os.path.basename(p) for p in present)} but is "+ f"not a rig_preprocess output; choose another --output_dir")+ # A review of this asset is the first half of the job: continuing it+ # (--annotation, or --review again) replaces it without --overwrite.+ review_stage = (_summary_field(summary_path, 'stage') == 'review'+ and _summary_field(summary_path, 'name') == name)+ continuing = review_stage and bool(annotation or review)+ if present and not (overwrite or continuing):+ if review_stage:+ raise FileExistsError(+ f"{output_dir} holds a review of '{name}': continue with --annotation "+ f"{os.path.join(output_dir, ANNOTATION_FILE)}, or pass --overwrite to redo it")+ raise FileExistsError(f"{present[0]} exists; pass --overwrite to redo the asset")+ # annotation.json may hold a reviewer's edits: it is never removed, and a+ # run that does not read it sets it aside only with --overwrite.+ if os.path.isfile(annotation_path) and not (+ annotation and os.path.samefile(annotation, annotation_path)):+ if not overwrite:+ hint = f"--annotation {annotation_path}" + (+ '' if review_stage or not present else ' --overwrite')+ raise FileExistsError(+ f"{annotation_path} exists and may hold edits this run would not use; continue "+ f"with {hint}, or pass --overwrite to set it aside")+ backup = _unused_path(annotation_path + '.bak')+ shutil.move(annotation_path, backup)+ logger.warning(f"{annotation_path} moved to {backup}")+ # While continuing, the review files and summary stay until the annotation+ # file has loaded, so a rejected edit can be fixed and the command rerun.+ keep_review = continuing and not overwrite+ for p in present:+ if not (keep_review and p in (summary_path, *review_paths)):+ shutil.rmtree(p) if os.path.isdir(p) else os.remove(p)+ export_dir, features_dir, canon_dir = (os.path.join(work, sub) for sub in SUBDIRS)+ os.makedirs(os.path.join(export_dir, 'motions'), exist_ok=True)+ summary = {'name': name, 'profile': profile, 'inputs': inputs,+ 'annotate': 'file' if annotation else annotate_mode}+ if requested_profile == AUTO:+ summary['profile_auto'] = True+ if annotation:+ summary['annotation'] = _display_path(annotation)+ elif annotate_mode != 'dataset':+ summary['annotate_names'] = names_mode or annotate_mode+ summary['annotate_face'] = face_mode or annotate_mode++ # 1. export+ names = export_animated(prof, inputs, export_dir, name, fps=fps)+ summary['animated'] = names is not None+ if names is None:+ if len(inputs) != 1:+ raise ValueError("no clip was exported from the given files")+ names = export_static(prof, inputs[0], export_dir, name, fps=fps)+ _save_json(os.path.join(export_dir, 'joint_names.json'), {name: names})+ summary['n_joints_export'] = len(names)+ first_clip = sorted(Path(export_dir, 'motions').glob(f'{glob.escape(name)}-*.npz'))[0]+ n_roots = _root_count(first_clip)+ if n_roots > 1:+ raise ValueError(+ f"'{name}': the exported skeleton has {n_roots} root bones. With a mesh the "+ f"exporter keeps the one that deforms it; this asset has none skinned. Give it "+ f"a single root (parent the others under it) or a skinned mesh.")++ # 2. dataset patches and sidecars+ if annotate_mode == 'dataset':+ # The dataset whose cond / canonical GLB this output reproduces (verify's default).+ summary['reference'] = os.path.basename(os.path.normpath(+ os.path.realpath(dataset_export_dir)))+ log = []+ summary['dataset_sidecars'] = _copy_dataset_sidecars(name, dataset_export_dir, export_dir)+ # The patch files are keyed by the dataset reproduced, which can differ+ # from the profile (a GLB of objaverse under the general profile).+ summary['root_offsets_applied'] = patch_annotations.apply_root_offsets(+ summary['reference'], export_dir, patch_dir, False, log)+ summary['rest_orientations_applied'] = patch_annotations.apply_rest_orientations(+ summary['reference'], export_dir, patch_dir, False, log)+ for line in log:+ # the patch files list every rig of the dataset; only this one's lines matter+ if name in line or 'no export NPZ' not in line:+ logger.info(f"[patch] {line}")++ if rest_quat is not None:+ # The dataset's patch mechanism, with the turn given by hand.+ patch_tmp = os.path.join(work, 'patches')+ os.makedirs(patch_tmp, exist_ok=True)+ _save_json(os.path.join(patch_tmp, f'{profile}_rest_orientation.json'),+ {name: {'quat': [float(v) for v in rest_quat],+ 'reason': f'--rest_rotation {rest_rotation}'}})+ log = []+ patch_annotations.apply_rest_orientations(profile, export_dir, patch_tmp, False, log)+ summary['rest_rotation'] = {'spec': rest_rotation,+ 'quat': [round(float(v), 6) for v in rest_quat]}+ for line in log:+ logger.info(f"[rest_rotation] {line}")++ # 3. processed rig GLB+ summary['rig_glb'] = build_processed_glb(prof, inputs, export_dir, name, names,+ animated=summary['animated'], fps=fps)++ summary['stats_dataset'] = summary.get('reference', profile)+ share, clip, joint = _dropped_translation(export_dir, name)+ summary['dropped_translation'] = {'max_share_of_extent': round(share, 4), 'clip': clip,+ 'joint': joint}+ notes = []+ if share > TRANSLATION_NOTE_TOL:+ notes.append(f"joint '{joint}' translates by {share:.1%} of the skeleton in "+ f"'{clip}'; the model keeps rotations and the root path only")+ logger.warning(f"'{name}': {notes[-1]}")++ # 4. annotate+ with np.load(first_clip, allow_pickle=True) as npz:+ rest_pos = rest_global(npz)+ parents = [int(p) for p in npz['parents']]+ if annotation:+ clean, face, notes_ann = load_annotation(annotation, names, rest_pos)+ notes_ann = [line.replace(annotation, _display_path(annotation)) for line in notes_ann]+ for line in notes_ann:+ logger.info(f"[annotate] {line}")+ if not review:+ for p in review_paths:+ if os.path.exists(p):+ os.remove(p)+ else:+ clean, face, notes_ann = annotate(annotate_mode, profile, name, names, export_dir,+ dataset_export_dir=dataset_export_dir, face_r=face_r,+ face_l=face_l, body_axis=body_axis,+ llm_client=llm_client, llm_options=llm_options,+ rest_pos=rest_pos, names_mode=names_mode,+ face_mode=face_mode)+ flipped = upside_down_note(clean, rest_pos)+ if flipped:+ notes.append(flipped)+ logger.warning(f"'{name}': {flipped}")+ _save_json(os.path.join(export_dir, 'clean_joint_names.json'), {name: clean})+ _save_json(os.path.join(export_dir, 'face_joint_names.json'), {name: face})+ summary['face_pair'] = face+ summary['annotation_notes'] = notes_ann+ if review:+ return _stop_for_review(summary, notes, output_dir, work, keep_intermediate, names,+ parents, rest_pos, clean, face, annotation_path,+ _resume_command(inputs, output_dir, profile, name, key,+ rest_rotation, annotation_path, {+ '--formats': ','.join(formats),+ '--save_clips': save_clips,+ '--no_preview': not preview,+ '--keep_intermediate': keep_intermediate,+ '--exp_dir': exp_dir,+ '--fps': fps if fps != 30 else None}))++ # 5. features (stage 4, the profile's arguments)+ args = extract_features.parse_args(+ ['--dataset_type', profile, '--data_dir', export_dir, '--save_dir', features_dir,+ '--no-vis', *prof.stage4_args])+ extract_features.main(args)+ entry, has_clips = _cond_entry(features_dir, name)+ if entry is None:+ raise RuntimeError(f"stage 4 produced no cond for '{name}'; see "+ f"{features_dir}/extract_errors.log and filtered_clips.json")+ if not has_clips:+ # The rest-pose cond, so the asset can still be driven and conditioned on.+ np.save(os.path.join(features_dir, 'cond.npy'), {name: entry})+ summary['n_joints'] = len(entry['joint_names'])+ summary['model_joint_limit'] = {'max_joints': joint_limit, 'source': limit_source,+ 'fits': summary['n_joints'] <= joint_limit}+ if not summary['model_joint_limit']['fits']:+ notes.append(f"{summary['n_joints']} joints: more than the {joint_limit} a model "+ f"takes ({limit_source}); sampling refuses it")+ logger.warning(f"'{name}' has {summary['n_joints']} joints, more than the "+ f"{joint_limit} a model takes ({limit_source}): sampling will "+ f"refuse it. Simplify the rig (fewer finger / face / accessory "+ f"bones) before using it.")+ summary['n_clips'] = len([f for f in os.listdir(os.path.join(features_dir, 'motions'))+ if f.startswith(f'{name}-')]) \+ if os.path.isdir(os.path.join(features_dir, 'motions')) else 0++ # 6. canonical asset (the cond stands without it: the motion can still be+ # sampled and rendered as a skeleton)+ canon = os.path.join(canon_dir, f'{name}.glb')+ if summary['rig_glb']:+ from data_process.feature_extraction.canonical_assets import bake_canonical_assets+ counts = bake_canonical_assets(export_dir, features_dir, assets_dir=canon_dir,+ dataset_type=profile, object_types=[name],+ formats=formats)+ if not os.path.isfile(canon):+ note = os.path.join(canon_dir, 'glb_errors', f'{name}.txt')+ reason = open(note).read().strip() if os.path.isfile(note) else f'{counts}'+ summary['canonical_glb_error'] = reason+ else:+ note = os.path.join(export_dir, 'rigs', 'glb_errors', f'{name}.txt')+ summary['canonical_glb_error'] = (open(note).read().strip() if os.path.isfile(note)+ else 'no processed rig GLB')+ if 'canonical_glb_error' in summary:+ notes.append(f"no canonical GLB: {summary['canonical_glb_error']}")+ logger.warning(f"'{name}': no canonical GLB ({summary['canonical_glb_error']}); "+ f"the cond is written, mesh driving needs a GLB")++ # 7. the deliverables+ np.save(cond_path, {name: entry})+ summary['cond'] = cond_path+ if annotation and not os.path.exists(annotation_path):+ shutil.copyfile(annotation, annotation_path) # the annotation that built the asset+ summary['canonical_glb'] = None+ if os.path.isfile(canon):+ for fmt in formats:+ shutil.copyfile(os.path.join(canon_dir, f'{name}.{fmt}'), asset_paths[fmt])+ summary['canonical_glb'] = glb_path+ if 'fbx' in formats:+ summary['canonical_fbx'] = asset_paths['fbx']+ if save_clips:+ motions = sorted(Path(features_dir, 'motions').glob(f'{glob.escape(name)}-*.npz')) \+ if os.path.isdir(os.path.join(features_dir, 'motions')) else []+ if motions:+ os.makedirs(clips_dir, exist_ok=True)+ for path in motions:+ shutil.copyfile(path, os.path.join(clips_dir, path.name))+ summary['clips'] = [p.stem for p in motions]+ summary['notes'] = notes+ if preview:+ from data_process.rig_preprocess.preview import render_preview+ try:+ summary['preview'] = render_preview(entry, name, preview_path, notes=notes)+ except Exception as exc: # noqa: BLE001 โ the deliverables are written+ logger.warning(f"'{name}': preview failed: {exc}")+ if not keep_intermediate:+ shutil.rmtree(work)+ summary['rig_glb'] = None # it lived in work/+ _save_json(summary_path, summary)+ logger.info(f"'{name}': {summary['n_joints']} joints -> {cond_path}"+ + (f", {glb_path}" if summary['canonical_glb'] else ' (no canonical GLB)')+ + (f", {len(summary['clips'])} clip(s) in {clips_dir}" if save_clips else '')+ + (f" ({summary['n_clips']} training clip(s) in {work})" if keep_intermediate else ''))+ logger.info(f"Check {preview_path}, then sample: python -m unimate.inference.sample "+ f"--exp_dir <run> --asset {output_dir} --prompt \"...\"")+ return summarydiff --git a/data_process/rig_preprocess/preview.py b/data_process/rig_preprocess/preview.pynew file mode 100644index 0000000..5b1054e--- /dev/null+++ b/data_process/rig_preprocess/preview.py@@ -0,0 +1,71 @@+"""``preview.png`` of a processed asset: what the model will be told (bpy-free).++Three panels of the cond's canonical T-pose (Y-up, facing +Z, grounded):++- an oblique view with the facing pair (red: right / head, blue: left / tail)+ and the forward direction the model generates toward (grey arrow, +Z);+- the same from the front (looking along -Z): the asset's face or chest should+ be the side shown here, its right side on the image's left;+- every joint with its cleaned label, the names the model embeds.++The two most likely errors on an unseen rig, a wrong facing and wrong joint+labels, are visible at a glance here and invisible in ``cond.npy``.+"""++import imageio+import numpy as np++from data_process.utils.plotting import (+ render_skeleton_tpose_annotated,+ render_skeleton_tpose_facing,+)++PANEL_FIGSIZE = (5.5, 5.5)+PANEL_DPI = 110+# matplotlib camera looking at the asset's front: the plot shows world -Z as+# its +Y axis (plotting._maybe_rotate_y_up), so azimuth -90 looks along -Z.+FRONT_AZIM, FRONT_ELEV = -90, 8+++def _fit_height(img, height):+ """Pad *img* (H, W, 3) with white rows to *height* (centred)."""+ if img.shape[0] >= height:+ return img[:height]+ pad = height - img.shape[0]+ top = np.full((pad // 2, img.shape[1], 3), 255, dtype=img.dtype)+ bottom = np.full((pad - pad // 2, img.shape[1], 3), 255, dtype=img.dtype)+ return np.concatenate([top, img, bottom], axis=0)+++def render_preview(cond, name, out_path, notes=()):+ """Write the three-panel preview of cond entry *cond* to *out_path*."""+ parents = np.asarray(cond['parents'])+ pos = np.asarray(cond['tpos_first_frame'], dtype=np.float32)+ clean = cond.get('clean_joint_names')+ labels = [str(n) for n in (cond['joint_names'] if clean is None or len(clean) == 0+ else clean)]+ face = cond.get('face_joint_idxs') or {}+ pair = [int(face.get('r_hip', -1)), int(face.get('l_hip', -1))]+ pair_txt = ('facing pair: {} / {}{}'.format(labels[pair[0]], labels[pair[1]],+ ' (head / tail axis)' if face.get('body_axis') else '')+ if min(pair) >= 0 else 'facing pair: none (the asset keeps its authored facing)')+ head = f'{name}: {len(labels)} joints\n{pair_txt}'+ if notes:+ head += '\n' + '\n'.join(notes)+ forward = np.array([0.0, 0.0, 1.0])+ oblique = render_skeleton_tpose_facing(+ parents, pos, pair, target_forward=forward,+ title=head + '\ngrey arrow: forward (+Z), where the model moves the asset',+ figsize=PANEL_FIGSIZE, dpi=PANEL_DPI)+ front = render_skeleton_tpose_facing(+ parents, pos, pair, elev=FRONT_ELEV, azim=FRONT_AZIM,+ title=f'{name}: front view (looking along -Z)\nthe face / chest should be this side, '+ f'right on the image left',+ figsize=PANEL_FIGSIZE, dpi=PANEL_DPI)+ labelled = render_skeleton_tpose_annotated(+ parents, pos, labels, figsize=(PANEL_FIGSIZE[0] * 1.4, PANEL_FIGSIZE[1] * 1.4),+ dpi=PANEL_DPI)+ height = max(p.shape[0] for p in (oblique, front, labelled))+ image = np.concatenate([_fit_height(p, height) for p in (oblique, front, labelled)], axis=1)+ imageio.imwrite(out_path, image)+ return out_pathdiff --git a/data_process/rig_preprocess/profiles.py b/data_process/rig_preprocess/profiles.pynew file mode 100644index 0000000..025b01f--- /dev/null+++ b/data_process/rig_preprocess/profiles.py@@ -0,0 +1,70 @@+"""Per-dataset processing profiles for one rigged asset.++A profile reproduces what the dataset's own pipeline does to an asset: which+stage-1 exporter (and so which importer, skeleton and frame rules) and which+stage-4 arguments (the ``run_extract_features.sh`` overrides). An unseen asset+takes the profile whose data it resembles; ``auto`` (the default,+:func:`pick_profile`) chooses ``truebones`` for a set of ``{Species}-{Action}.fbx``+clips and ``general``, the catch-all for GLB/GLTF/FBX files, otherwise.+"""++from dataclasses import dataclass, field+from pathlib import Path+from typing import Sequence, Tuple+++@dataclass(frozen=True)+class Profile:+ """How one dataset's pipeline processes an asset (see the module docstring)."""+ name: str+ # Stage-1 exporter: 'objaverse' (export_objaverse), 'general'+ # (export_general) or 'truebones' (export_truebones.export_species).+ exporter: str+ # Importer: 'gltf', 'fbx' or 'auto' (by extension).+ importer: str+ # Skin weights from every skinned mesh (general) or the largest one only.+ all_skinned_meshes: bool+ # Leaf-name suffixes that mark an end effector rather than a control bone.+ end_effector_suffixes: Tuple[str, ...]+ # Drop rest-pose frames from clips (export_general --keep_tpose_frames off).+ remove_tpose: bool+ # Extra stage-4 arguments, as run_extract_features.sh passes them.+ stage4_args: Tuple[str, ...] = field(default_factory=tuple)+++PROFILES = {+ 'objaverse': Profile(+ name='objaverse', exporter='objaverse', importer='gltf',+ all_skinned_meshes=False, end_effector_suffixes=(), remove_tpose=True,+ stage4_args=('--min_joints', '4', '--max_joints', '180')),+ # Rest-pose frames are kept (stage 1 --keep_tpose_frames).+ 'general': Profile(+ name='general', exporter='general', importer='auto',+ all_skinned_meshes=True, end_effector_suffixes=('_tip',), remove_tpose=False,+ stage4_args=('--min_joints', '4', '--max_joints', '180')),+ 'truebones': Profile(+ name='truebones', exporter='truebones', importer='fbx',+ all_skinned_meshes=False, end_effector_suffixes=(), remove_tpose=False),+}+++# Input file extensions each importer accepts.+IMPORTER_EXTS = {'gltf': ('.glb', '.gltf'), 'fbx': ('.fbx',), 'auto': ('.glb', '.gltf', '.fbx')}+++AUTO = 'auto'+++def pick_profile(inputs: Sequence[str], dataset_export_dir: str = None) -> str:+ """The profile ``auto`` stands for: the dataset of *dataset_export_dir* when+ it has a profile (``--annotate dataset``); ``truebones`` for several FBX+ clips of one species (``{Species}-{Action}.fbx``); else ``general``."""+ if dataset_export_dir:+ name = Path(dataset_export_dir.rstrip('/')).resolve().name+ if name in PROFILES:+ return name+ paths = [Path(p) for p in inputs]+ if (len(paths) > 1 and all(p.suffix.lower() == '.fbx' and '-' in p.stem for p in paths)+ and len({p.stem.split('-', 1)[0] for p in paths}) == 1):+ return 'truebones'+ return 'general'diff --git a/data_process/rig_preprocess/review.py b/data_process/rig_preprocess/review.pynew file mode 100644index 0000000..c4ed942--- /dev/null+++ b/data_process/rig_preprocess/review.py@@ -0,0 +1,424 @@+"""Review the joint annotation before the cond is built (bpy-free).++``run --review`` stops after stage 3 and writes, in the output directory:++- ``annotation.json``: every joint (raw name, parent, label, where the label+ came from, and ``check`` with reasons when it deserves a look), the facing+ pair with the alternatives the labels offer, and the label vocabulary;+- ``annotation_preview.png``: the export rest pose with the facing pair, the+ forward direction it implies, and every joint's index and label;+- ``REVIEW.md``: what to check and how, written for a person, an LLM or a+ coding agent, and the command that continues.++A reviewer edits ``annotation.json`` (labels, ``face_pair``); ``run --annotation+annotation.json`` then builds the asset from it (:func:`load_annotation`+validates it against the exported skeleton).+"""++import json+import os+import re++import numpy as np++from data_process.joint_annotation.face_select_rule import SYMMETRIC_PAIR_PRIORITY+from data_process.joint_annotation.name_map import apply_name_map, load_name_map, norm_name+from data_process.joint_annotation.vocab import QUALIFIERS, canonical_parts, is_canonical_label+from data_process.rig_preprocess.annotate import facing_defined+from data_process.rig_preprocess.evaluate import facing_quat++ANNOTATION_FILE = 'annotation.json'+REVIEW_FILE = 'REVIEW.md'+REVIEW_PREVIEW = 'annotation_preview.png'+FORMAT = 'rig_preprocess-annotation/1'+# Mirrored limbs (the same part, Left and Right) whose centres are this far+# apart (share of the skeleton's extent) and point the other way from the+# facing pair (cosine below -SIDE_COS) are flagged.+SIDE_TOL = 0.02+SIDE_COS = 0.5+# Face-pair parts that need no second look: hip / shoulder level, whole-limb+# roots and fins. A pair of any other part (hands, knees, eyes, ...) is flagged.+_SOLID_PARTS = {p.lower() for p in+ SYMMETRIC_PAIR_PRIORITY[:SYMMETRIC_PAIR_PRIORITY.index('Fin') + 1]}+_SOLID_PARTS |= {'manual', 'reviewed', 'body_axis'}+_SIDE_RE = re.compile(r'^(Left|Right) ')+_PLACEHOLDER_RE = re.compile(r'^(?:Left |Right )?(?:Bone|_?\d+)$')+++def _side(label):+ m = _SIDE_RE.match(label)+ return m.group(1) if m else ''+++def _sources(raw, parents, labels, mode):+ """Per joint: 'template' / 'names' where the reviewed-label lookup gives the+ label, else *mode* ('rule', 'llm', 'dataset', or 'file' for an annotation+ file)."""+ if mode == 'dataset':+ return ['dataset'] * len(raw)+ lookup = load_name_map()+ unset = '\0'+ tmpl, src = apply_name_map(lookup, raw, [unset] * len(raw), parents)+ if src == 'template':+ return ['template' if a != unset and a == b else mode for a, b in zip(tmpl, labels)]+ table, vocab = lookup.get('names', {}), lookup.get('labels', [])+ return ['names' if norm_name(n) in table and vocab[table[norm_name(n)]] == l else mode+ for n, l in zip(raw, labels)]+++def _limbs(labels, parents):+ """Sided limbs: ``{(side, part of the limb's first joint): [[joints], ...]}``,+ a limb being a connected run of joints labelled with the same side."""+ limbs = {}+ for i, label in enumerate(labels):+ side, p = _side(label), parents[i]+ if side and not (p >= 0 and _side(labels[p]) == side):+ members, stack = [], [i]+ while stack:+ j = stack.pop()+ members.append(j)+ stack.extend(c for c, q in enumerate(parents) if q == j and _side(labels[c]) == side)+ limbs.setdefault((side, label[len(side) + 1:]), []).append(members)+ return limbs+++def _mirror_conflicts(labels, parents, pos, face, raw, extent):+ """``(conflicts, pairs)``: the mirrored limb pairs whose Right limb lies on+ the left of the Left limb, judged by the facing pair's lateral axis (limb+ centres, so a crossing clavicle in an otherwise sided arm does not count),+ as ``(right root, left root, part)``; *pairs* how many pairs were judged."""+ r, l = face.get('r_hip', {}).get('raw'), face.get('l_hip', {}).get('raw')+ if not (r and l) or face.get('body_axis'):+ return [], 0+ across = pos[raw.index(r)] - pos[raw.index(l)]+ across[1] = 0.0+ if np.linalg.norm(across) < 1e-8:+ return [], 0+ across /= np.linalg.norm(across)+ limbs, conflicts, pairs = _limbs(labels, parents), [], 0+ for (side, part), right in limbs.items():+ left = limbs.get(('Left', part), [])+ if side != 'Right' or len(right) != 1 or len(left) != 1:+ continue+ gap = pos[right[0]].mean(0) - pos[left[0]].mean(0)+ gap[1] = 0.0+ if np.linalg.norm(gap) < SIDE_TOL * extent:+ continue+ pairs += 1+ if float(gap @ across) / np.linalg.norm(gap) < -SIDE_COS:+ conflicts.append((right[0][0], left[0][0], part))+ return conflicts, pairs+++def _candidates(raw, labels, rest_pos):+ """Mirrored pairs the labels offer: ``[right raw, left raw, part]``, ranked+ parts first, each only when it defines a facing."""+ by_part = {}+ for i, label in enumerate(labels):+ side = _side(label)+ if side:+ by_part.setdefault(label[len(side) + 1:], {}).setdefault(side, []).append(i)+ out = []+ for part, sides in by_part.items():+ if 'Left' in sides and 'Right' in sides:+ r, l = sides['Right'][0], sides['Left'][0]+ entry = {'r_hip': {'raw': raw[r]}, 'l_hip': {'raw': raw[l]}}+ if facing_defined(entry, raw, rest_pos):+ out.append([raw[r], raw[l], part])+ rank = {p: i for i, p in enumerate(SYMMETRIC_PAIR_PRIORITY)}+ return sorted(out, key=lambda c: (rank.get(c[2], len(rank)), c[2]))+++def build_annotation(name, raw, parents, rest_pos, labels, face, mode='rule', notes=()):+ """The ``annotation.json`` dict for one asset: *raw* the exported joint+ names, *parents* and *rest_pos* (export space, Y-up) the skeleton, *labels*+ and *face* the stage-3 result, *mode* where it came from, *notes* the run's+ notes (``summary.json``)."""+ raw = [str(n) for n in raw]+ pos = np.asarray(rest_pos, dtype=float)+ extent = float(np.ptp(pos, axis=0).max()) or 1.0+ sources = _sources(raw, parents, labels, mode)+ r, l = face.get('r_hip', {}).get('raw'), face.get('l_hip', {}).get('raw')+ conflicts, judged = _mirror_conflicts(labels, parents, pos, face, raw, extent)+ mirrored = {}+ for ri, li, part in conflicts:+ for i in (ri, li):+ mirrored[i] = (f"the Left / Right {part} limbs lie on the opposite sides from the "+ f"facing pair's: swap their sides, or the pair")+ joints = []+ for i, (n, label) in enumerate(zip(raw, labels)):+ reasons = []+ if sources[i] not in ('template', 'dataset'):+ if _PLACEHOLDER_RE.match(label):+ reasons.append('placeholder: give it a part name if it is a body part')+ elif not is_canonical_label(label):+ reasons.append('not in the label vocabulary')+ if i in mirrored:+ reasons.append(mirrored[i])+ joints.append({'index': i, 'raw': n,+ 'parent': raw[parents[i]] if parents[i] >= 0 else None,+ 'label': label, 'source': sources[i], 'check': bool(reasons),+ **({'reasons': reasons} if reasons else {})})+ face_reasons = []+ if not (r and l):+ face_reasons.append('no facing pair: the asset keeps its authored facing')+ elif face.get('body_axis'):+ face_reasons.append('head / tail axis: check that `right` is the head end')+ elif str(face.get('source', '')).lower() not in _SOLID_PARTS:+ face_reasons.append(f"chosen from '{face.get('source') or 'fallback'}' joints, not hip "+ f"or shoulder level: prefer the thighs or shoulders if the asset "+ f"has them")+ if judged >= 2 and len(conflicts) * 2 > judged:+ face_reasons.append(f'{len(conflicts)} of {judged} mirrored limb pairs lie the other '+ f'way round: the pair is likely swapped')+ return {+ 'format': FORMAT,+ 'asset': name,+ 'notes': list(notes),+ 'face_pair': {'right': r or '', 'left': l or '', 'body_axis': bool(face.get('body_axis')),+ 'source': face.get('source', ''), 'check': bool(face_reasons),+ **({'reasons': face_reasons} if face_reasons else {})},+ 'face_pair_alternatives': _candidates(raw, labels, pos),+ 'n_flagged': sum(j['check'] for j in joints),+ 'joints': joints,+ 'vocabulary': {+ 'format': "[Left |Right ][Qualifier ]Part[ End], e.g. 'Left Front Leg', "+ "'Right Index Finger End'. Qualifier: " + ', '.join(QUALIFIERS)+ + ". No numbers: every joint of a chain takes the same label "+ "(Spine, Spine, Spine; Tail, Tail), its last joint 'End' "+ "('Tail End'). 'Bone' only for a joint that is no body part.",+ 'parts': sorted(canonical_parts()),+ },+ }+++def read_annotation(path):+ """The dict of annotation file *path*, with its structure checked: the+ format, a list of joints with string ``raw`` and non-empty string ``label``,+ and a ``face_pair`` of two different joint names (or none) with a boolean+ ``body_axis``. Raises ValueError saying what is wrong."""+ try:+ with open(path) as f:+ data = json.load(f)+ except ValueError as exc:+ raise ValueError(f"{path}: not valid JSON ({exc})") from None+ if not isinstance(data, dict) or data.get('format') != FORMAT:+ raise ValueError(f"{path}: not a rig_preprocess annotation (needs \"format\": "+ f"\"{FORMAT}\")")+ joints = data.get('joints')+ if not isinstance(joints, list) or not joints:+ raise ValueError(f"{path}: 'joints' must be a non-empty list")+ seen = set()+ for j in joints:+ if not (isinstance(j, dict) and isinstance(j.get('raw'), str)+ and isinstance(j.get('label'), str)):+ raise ValueError(f"{path}: every joint needs a string 'raw' and a string 'label' "+ f"(got {j!r:.80})")+ if not j['label'].strip():+ raise ValueError(f"{path}: empty label for joint {j['raw']!r}")+ if j['raw'] in seen:+ raise ValueError(f"{path}: joint {j['raw']!r} is listed twice")+ seen.add(j['raw'])+ fp = data.get('face_pair')+ if not isinstance(fp, dict) or not {'right', 'left'} <= set(fp):+ raise ValueError(f"{path}: 'face_pair' must be an object with 'right', 'left' and "+ f"'body_axis' (both names empty for no pair)")+ r, l, axis = fp.get('right') or '', fp.get('left') or '', fp.get('body_axis', False)+ if not (isinstance(r, str) and isinstance(l, str)):+ raise ValueError(f"{path}: face_pair 'right' and 'left' must be joint names (strings)")+ if not isinstance(axis, bool):+ raise ValueError(f"{path}: face_pair 'body_axis' must be true or false, got {axis!r}")+ if bool(r) != bool(l):+ raise ValueError(f"{path}: face_pair needs both 'right' and 'left', or neither")+ if r and r == l:+ raise ValueError(f"{path}: face_pair names {r!r} twice; give two different joints")+ if axis and not r:+ raise ValueError(f"{path}: face_pair 'body_axis' is true but no joints are given")+ return data+++def load_annotation(path, raw, rest_pos):+ """``(labels, face_entry, notes)`` from a reviewed ``annotation.json``,+ labels in the order of the exported skeleton's raw names *raw*. Raises+ ValueError saying what is wrong."""+ data = read_annotation(path)+ by_raw = {j['raw']: j['label'].strip() for j in data['joints']}+ raw = [str(n) for n in raw]+ missing = [n for n in raw if n not in by_raw]+ unknown = [n for n in by_raw if n not in raw]+ if missing or unknown:+ raise ValueError(f"{path}: joints do not match the exported skeleton (missing "+ f"{missing[:5]}, unknown {unknown[:5]}); edit labels only, never the "+ f"raw names")+ labels = [by_raw[n] for n in raw]+ fp = data.get('face_pair', {})+ r, l = fp.get('right') or '', fp.get('left') or ''+ if r:+ for joint in (r, l):+ if joint not in raw:+ raise ValueError(f"{path}: face_pair joint {joint!r} is not a joint of the asset")+ face = {'r_hip': {'raw': r, 'clean': labels[raw.index(r)]},+ 'l_hip': {'raw': l, 'clean': labels[raw.index(l)]},+ 'source': 'reviewed'}+ if fp.get('body_axis'):+ face['body_axis'] = True+ face['source'] = 'body_axis'+ if not facing_defined(face, raw, rest_pos):+ raise ValueError(f"{path}: face_pair {r} / {l} has no horizontal separation in the "+ f"rest pose; choose two joints side by side (or a head / tail pair "+ f"with body_axis true)")+ else:+ face = {'r_hip': {'raw': '', 'clean': ''}, 'l_hip': {'raw': '', 'clean': ''},+ 'source': 'empty'}+ notes = [f'annotation from {path}']+ off = sorted({lab for lab in labels if not is_canonical_label(lab)})+ if off:+ notes.append(f'{len(off)} label(s) outside the vocabulary kept as given: {off[:5]}')+ return labels, face, notes+++def _forward(face, raw, rest_pos):+ q = facing_quat(face, raw, rest_pos)+ if q is None:+ return None+ from Quaternions import Quaternions+ return (-Quaternions(np.asarray(q)[None]) * np.array([[0.0, 0.0, 1.0]]))[0]+++def write_review(out_dir, annotation, raw, parents, rest_pos, face, resume_cmd):+ """Write ``annotation.json``, ``annotation_preview.png`` and ``REVIEW.md``."""+ path = os.path.join(out_dir, ANNOTATION_FILE)+ with open(path, 'w') as f:+ json.dump(annotation, f, indent=1)+ labels = [j['label'] for j in annotation['joints']]+ try:+ _render(os.path.join(out_dir, REVIEW_PREVIEW), annotation['asset'], raw, parents,+ rest_pos, labels, face)+ except Exception as exc: # noqa: BLE001 โ the annotation file is what matters+ from loguru import logger+ logger.warning(f"annotation preview failed: {exc}")+ with open(os.path.join(out_dir, REVIEW_FILE), 'w') as f:+ f.write(_review_text(annotation, resume_cmd))+ return path+++def _render(path, name, raw, parents, rest_pos, labels, face):+ import imageio+ from data_process.rig_preprocess.preview import PANEL_DPI, PANEL_FIGSIZE, _fit_height+ from data_process.utils.plotting import (+ render_skeleton_tpose_annotated,+ render_skeleton_tpose_facing,+ )+ pos = np.asarray(rest_pos, dtype=np.float32)+ r, l = face.get('r_hip', {}).get('raw'), face.get('l_hip', {}).get('raw')+ pair = [raw.index(r), raw.index(l)] if r and l else [-1, -1]+ forward = _forward(face, list(raw), rest_pos)+ title = (f'{name}: rest pose as exported (not yet canonical)\n'+ f'facing pair: {r or "none"} / {l or "none"}'+ f'{" (head / tail)" if face.get("body_axis") else ""}\n'+ f'green arrow: the forward direction this pair gives')+ facing = render_skeleton_tpose_facing(np.asarray(parents), pos, pair, forward=forward,+ title=title, figsize=PANEL_FIGSIZE, dpi=PANEL_DPI)+ labelled = render_skeleton_tpose_annotated(+ np.asarray(parents), pos, labels,+ figsize=(PANEL_FIGSIZE[0] * 1.4, PANEL_FIGSIZE[1] * 1.4), dpi=PANEL_DPI)+ height = max(facing.shape[0], labelled.shape[0])+ imageio.imwrite(path, np.concatenate([_fit_height(facing, height),+ _fit_height(labelled, height)], axis=1))+++def _review_text(annotation, resume_cmd):+ fp = annotation['face_pair']+ flagged = [j for j in annotation['joints'] if j['check']]+ pair = (f"`{fp['right']}` / `{fp['left']}`" + (" (head / tail axis)" if fp['body_axis'] else '')+ if fp['right'] else 'none')+ alternatives = ', '.join(f"`{r}` / `{l}` ({part})"+ for r, l, part in annotation['face_pair_alternatives'][:6])+ lines = [+ f"# Review the joint annotation of `{annotation['asset']}`",+ "",+ "The model is conditioned on two things decided here: a **label** for every joint,",+ "from one anatomical vocabulary shared by all training assets, and a **facing pair**,",+ "the two joints that say which way the asset faces. The labels and pair in",+ "`annotation.json` come from the annotation step (rules and a lookup of reviewed",+ "training rigs, or an LLM). Check them now: they are easy to fix here and invisible",+ "in the finished `cond.npy`.",+ "",+ f"- {len(annotation['joints'])} joints, {len(flagged)} flagged (`\"check\": true`).",+ f"- Facing pair: {pair}"+ + (f". **Flagged**: {'; '.join(fp.get('reasons', []))}" if fp['check'] else ''),+ "- `annotation_preview.png`: left, the rest pose with the pair (red right, blue left)",+ " and the forward direction it gives (green arrow); right, every joint as",+ " `[index] label`.",+ "",+ ]+ if annotation.get('notes'):+ lines += ["Notes from the run (an upside-down or lying rest pose needs `--rest_rotation`,",+ "e.g. `x180`, added to the command under *Continue*):", ""]+ lines += [f"- {note}" for note in annotation['notes']] + [""]+ if flagged:+ lines += ["Flagged joints:", "", "| index | raw | label | why |", "|---|---|---|---|"]+ for j in flagged[:40]:+ lines.append(f"| {j['index']} | `{j['raw']}` | {j['label']} | "+ f"{'; '.join(j['reasons'])} |")+ if len(flagged) > 40:+ lines.append(f"| ... | {len(flagged) - 40} more in `annotation.json` | | |")+ lines.append("")+ lines += [+ "## What to check",+ "",+ "1. **Labels** (`joints[].label`). Flags catch the obvious cases only; read the whole",+ " list against the raw names and the hierarchy (`parent`).",+ " - Form: `[Left |Right ][Qualifier ]Part[ End]`, with parts from",+ " `vocabulary.parts`: `Left Thigh`, `Right Front Leg`, `Left Index Finger End`.",+ " - **Left and Right are the asset's own sides**, as seen by the character, not by",+ " the viewer.",+ " - No numbers: every joint of a chain takes the same label (`Spine`, `Spine`,",+ " `Spine`), and a chain's last joint may add `End` (`Tail End`).",+ " - `Bone` only for joints that are not body parts (props, helpers).",+ " - `source` says where a label came from: `template` (a training rig with the same",+ " bone tree) and `names` (bone names whose reviewed label is consistent) are",+ " usually right; `rule` labels deserve the closer look.",+ "2. **Facing pair** (`face_pair`). Two joints side by side, the asset's right one in",+ " `right`: preferably the thighs, else the shoulders or the upper legs."+ + (f" Candidates the labels offer: {alternatives}." if alternatives else ''),+ " For a body with no left / right (snake, fish, worm) give a head / tail pair, the",+ " head end in `right`, and set `body_axis` to true. Leave both empty only if no pair",+ " makes sense: the asset then keeps its authored facing.",+ " The green arrow must point where the asset faces. If it points backwards, the",+ " rig's Left / Right names are mirrored: swap `right` and `left`, and the sides of",+ " the labels.",+ "",+ "Edit only `label`, `face_pair.right`, `face_pair.left` and `face_pair.body_axis`",+ "(`true` or `false`). Never change `raw` names and keep `format`; the other fields",+ "are information, ignored when loading.",+ "",+ "## Continue",+ "",+ "```bash",+ resume_cmd,+ "```",+ "",+ "The file is validated first (every joint labelled, the pair two joints of the asset",+ "that define a facing); the run then builds `cond.npy`, the canonical GLB and",+ "`preview.png` from it. To check an edited file before building, add `--review` to",+ "the command: the review files, `annotation.json` included, are written again from it",+ "(fields you added are dropped).",+ "",+ "## Asking an LLM or a coding agent",+ "",+ "Give it `annotation.json` (and `annotation_preview.png` if it reads images) with:",+ "",+ "> This is the joint annotation of a rigged 3D asset for a motion model. Check every",+ "> joint's `label` against its raw name, its parent and its place in the hierarchy, and",+ "> correct wrong ones using only the form and parts in `vocabulary` and the asset's own",+ "> left / right; the joints with `\"check\": true` list reasons. Then check",+ "> `face_pair`: two side-by-side joints, the asset's right one first, preferably the",+ "> thighs (for a limbless body a head / tail pair with `body_axis: true`). Change only",+ "> `label` values and the `face_pair` fields, keep every `raw` name and the JSON",+ "> structure, and return the whole corrected file.",+ "",+ "Save the answer over `annotation.json` and continue.",+ "",+ ]+ return '\n'.join(lines)diff --git a/data_process/rig_preprocess/verify.py b/data_process/rig_preprocess/verify.pynew file mode 100644index 0000000..7aafaa6--- /dev/null+++ b/data_process/rig_preprocess/verify.py@@ -0,0 +1,265 @@+"""Compare a ``rig_preprocess`` output with the dataset's copy of the asset (bpy-free).++For an asset that is also training data, both deliverables are checked:++- ``cond.npy`` against the stage-4 cond entry, field by field, and+- ``<name>.glb`` against the canonical GLB: joints, skinned rest triangles and+ the dominant joint of every vertex.++A cond can differ from the dataset's without describing a different skeleton.+These differences are tolerated and listed under ``notes``; anything else is a+``difference``:++- joint order: the same joints with siblings in another order (Blender's bone+ enumeration differs between versions); compared after reordering by name.+- quaternion sign: ``q`` and ``-q`` are one rotation.+- bone frames: ``offsets`` and the T-pose rotations are expressed in each+ bone's own axes, which another Blender version may orient differently (the+ released Objaverse NPZs carry an older one's). Accepted when the joint+ positions agree and each cond is self-consistent+ (``pos[j] - pos[parent] == rotate(global_rot[parent], offsets[j])`` and+ ``global_rot[j] == global_rot[parent] * local_rot[j]``).+- spectral basis: the Laplacian eigenvectors are unique only up to sign and,+ for repeated eigenvalues, up to a rotation of the eigenspace; when a tie+ straddles the ``max_freqs`` cut-off any subset of it is valid. Accepted when+ each column of both conds lies in the eigenspace of its own eigenvalue+ (ties included) and the columns are orthonormal, so columns of distinct+ eigenvalues may not trade places.++Verdict: ``identical``, ``equivalent`` (notes only) or ``different``.+"""++import os++import numpy as np++from data_process.rig_preprocess.glb import Glb+from data_process.utils.topology import _build_laplacian, _parse_parents++# Joint positions and other geometry, relative to the skeleton's extent. The+# released exports were made with an older Blender, whose float32 rest poses+# differ from a current import by up to about 1.5e-5 of the extent.+POS_TOL = 1e-4+# Scalars and unit quantities (rotations, eigenvectors), absolute.+VAL_TOL = 1e-5+# Skinned triangles of the canonical GLB, relative to the mesh's extent.+MESH_TOL = 1e-4+# Orthonormality of stored (float32) eigenvector columns.+SPEC_ORTHO_TOL = 1e-4++QUAT_FIELDS = ('tpos_local_rotations', 'tpos_global_rotations')+FRAME_FIELDS = ('offsets',) + QUAT_FIELDS+PER_JOINT_FIELDS = ('tpos_first_frame', 'tpos_offsets', 'joint_depths', 'clean_joint_names')+PAIRWISE_FIELDS = ('joint_relations', 'joint_graph_dists')+++def _qrot(q, v):+ """Rotate vectors *v* by quaternions *q* ``(w, x, y, z)``."""+ w, u = q[..., :1], q[..., 1:]+ t = 2 * np.cross(u, v)+ return v + w * t + np.cross(u, t)+++def _qmul(a, b):+ """Quaternion products ``a * b`` ``(w, x, y, z)``."""+ w1, v1, w2, v2 = a[..., :1], a[..., 1:], b[..., :1], b[..., 1:]+ return np.concatenate([w1 * w2 - (v1 * v2).sum(-1, keepdims=True),+ w1 * v2 + w2 * v1 + np.cross(v1, v2)], -1)+++def _frame_consistency(cond):+ """Whether a cond's bone frames agree with its joint positions:+ ``|pos[j] - pos[parent] - rotate(G[parent], offsets[j])|`` within ``POS_TOL``+ of the extent and ``G[j] == G[parent] * L[j]`` within ``VAL_TOL``."""+ pos = np.asarray(cond['tpos_first_frame'], float)+ glob = np.asarray(cond['tpos_global_rotations'], float)+ loc = np.asarray(cond['tpos_local_rotations'], float)+ off = np.asarray(cond['offsets'], float)+ parents = np.asarray(cond['parents'])+ j = np.flatnonzero(parents >= 0)+ ext = float(np.ptp(pos, 0).max()) or 1.0+ pos_err = float(np.abs(pos[j] - pos[parents[j]] - _qrot(glob[parents[j]], off[j])).max(initial=0) / ext)+ rot_err = _quat_err(glob[j], _qmul(glob[parents[j]], loc[j]))+ return pos_err <= POS_TOL and rot_err <= VAL_TOL+++def _quat_err(a, b):+ """Largest component difference of two quaternion arrays, up to the sign of each."""+ if not len(a):+ return 0.0+ return float(np.minimum(np.abs(a - b).max(-1), np.abs(a + b).max(-1)).max())+++def _valid_eigenbasis(parents, spec, tol=1e-6):+ """Whether column ``c`` of *spec* lies in the eigenspace of the ``c+1``-th+ smallest eigenvalue of the skeleton's Laplacian (every eigenvector tied with+ it included), and the columns are orthonormal."""+ if not spec.shape[1]:+ return True+ vals, vecs = np.linalg.eigh(_build_laplacian(_parse_parents(parents), norm='sym'))+ if np.abs(spec.T @ spec - np.eye(spec.shape[1])).max() > SPEC_ORTHO_TOL:+ return False+ for c in range(spec.shape[1]):+ space = vecs[:, np.abs(vals - vals[c + 1]) <= tol]+ if np.abs(spec[:, c] - space @ (space.T @ spec[:, c])).max() > VAL_TOL:+ return False+ return True+++def _parent_names(cond, names):+ return [names[p] if p >= 0 else None for p in cond['parents']]+++def _edges(cond, names):+ return {(names[i], names[j]) for i, j in np.asarray(cond['edge_indexs']).T}+++def _chains(cond, names):+ return sorted(tuple(names[i] for i in chain) for chain in cond['kinematic_chains'])+++def _chain_edges(cond, names):+ return {(names[c[i]], names[c[i + 1]]) for c in cond['kinematic_chains']+ for i in range(len(c) - 1)}+++def _face(face, names):+ return tuple(names[face[k]] if face[k] >= 0 else None for k in ('r_hip', 'l_hip')) \+ + (bool(face.get('body_axis')),)+++def compare_cond(ours, theirs):+ """Compare two cond entries of one asset; returns a dict with ``verdict``."""+ na = [str(n) for n in ours['joint_names']]+ nb = [str(n) for n in theirs['joint_names']]+ notes, diffs = [], []+ if sorted(na) != sorted(nb):+ missing, extra = sorted(set(nb) - set(na)), sorted(set(na) - set(nb))+ return {'verdict': 'different', 'notes': [], 'differences': [+ f'joint sets differ ({len(na)} vs {len(nb)}; missing {missing[:5]}, extra {extra[:5]})']}+ perm = np.array([na.index(n) for n in nb]) # ours' index of each dataset joint+ if not np.array_equal(perm, np.arange(len(nb))):+ notes.append('joint order: same joints, siblings ordered differently')+ parents_a = _parent_names(ours, na)+ if [parents_a[i] for i in perm] != _parent_names(theirs, nb):+ diffs.append('hierarchy')++ pos_b = np.asarray(theirs['tpos_first_frame'], float)+ ext = float(np.ptp(pos_b, 0).max()) or 1.0+ pos_err = float(np.abs(np.asarray(ours['tpos_first_frame'], float)[perm] - pos_b).max() / ext)+ for key in PER_JOINT_FIELDS:+ a, b = np.asarray(ours[key])[perm], np.asarray(theirs[key])+ same = (np.abs(a - b).max() / ext <= POS_TOL) if a.dtype.kind == 'f' \+ else np.array_equal(a.astype(str), b.astype(str))+ if not same:+ diffs.append(key)+ for key in PAIRWISE_FIELDS:+ if not np.array_equal(np.asarray(ours[key])[np.ix_(perm, perm)], np.asarray(theirs[key])):+ diffs.append(key)+ if _edges(ours, na) != _edges(theirs, nb):+ diffs.append('edge_indexs')+ if _chains(ours, na) != _chains(theirs, nb):+ # Chains continue through a branch's first child, so another sibling+ # order splits the same tree differently.+ if _chain_edges(ours, na) == _chain_edges(theirs, nb):+ notes.append('kinematic_chains: same tree, split differently at a branch')+ else:+ diffs.append('kinematic_chains')++ # Bone-frame fields: exact (quaternions up to sign), else frame-equivalent.+ frame_exact = True+ for key in FRAME_FIELDS:+ a, b = np.asarray(ours[key], float)[perm], np.asarray(theirs[key], float)+ if key in QUAT_FIELDS:+ if _quat_err(a, b) > VAL_TOL:+ frame_exact = False+ elif np.abs(a - b).max() > VAL_TOL:+ notes.append(f'{key}: some quaternions stored with the opposite sign')+ elif np.abs(a - b).max() / ext > POS_TOL:+ frame_exact = False+ if not frame_exact:+ if pos_err <= POS_TOL and _frame_consistency(ours) and _frame_consistency(theirs):+ notes.append('bone frames: offsets / T-pose rotations in other bone axes, '+ 'joint positions identical')+ else:+ diffs.append('offsets / T-pose rotations')++ # Spectral features: within the (possibly degenerate) eigenspace.+ spec_a = np.asarray(ours['spectral_feats'], float)[perm]+ spec_b = np.asarray(theirs['spectral_feats'], float)+ if spec_a.shape != spec_b.shape:+ diffs.append('spectral_feats shape')+ else:+ k = int((np.abs(spec_b).sum(0) > 0).sum()) # zero-padded for tiny skeletons+ a, b = spec_a[:, :k], spec_b[:, :k]+ if np.abs(spec_a[:, k:]).max(initial=0) > 0:+ diffs.append('spectral_feats')+ elif np.minimum(np.abs(a - b).max(0), np.abs(a + b).max(0)).max(initial=0) <= VAL_TOL:+ if np.abs(a - b).max(initial=0) > VAL_TOL:+ notes.append('spectral_feats: some eigenvectors stored with the opposite sign')+ elif _valid_eigenbasis(theirs['parents'], a) and _valid_eigenbasis(theirs['parents'], b):+ notes.append('spectral_feats: another basis of a repeated eigenvalue')+ else:+ diffs.append('spectral_feats')++ fa, fb = ours.get('face_joint_idxs'), theirs.get('face_joint_idxs')+ if (fa is None) != (fb is None) or (fa is not None and _face(fa, na) != _face(fb, nb)):+ diffs.append('face_joint_idxs')+ for key in ('scale_factor', 'ground_height'):+ a, b = ours.get(key), theirs.get(key)+ if (a is None) != (b is None) or (a is not None and abs(a - b) > VAL_TOL * max(1, abs(b))):+ diffs.append(key)+ if ours.get('ground_height_mode') != theirs.get('ground_height_mode'):+ diffs.append('ground_height_mode')+ if dict(ours.get('captions') or {}) != dict(theirs.get('captions') or {}):+ notes.append('captions differ (training-clip captions, not a skeleton property)')++ verdict = 'different' if diffs else ('equivalent' if notes else 'identical')+ return {'verdict': verdict, 'differences': diffs, 'notes': notes,+ 'joint_position_err': pos_err}+++def compare_glb(ours, theirs):+ """Joints and skinned rest triangles of two GLBs; returns a dict with ``verdict``."""+ a, b = Glb(ours), Glb(theirs)+ ja, jb = a.joints(), b.joints()+ if sorted(ja) != sorted(jb):+ return {'verdict': 'different', 'differences': ['joint names']}+ names = sorted(jb)+ pa = np.stack([ja[n][:3, 3] for n in names])+ pb = np.stack([jb[n][:3, 3] for n in names])+ ext = float(np.ptp(pb, 0).max()) or 1.0+ out = {'joint_err': float(np.abs(pa - pb).max() / ext)}+ ta, da = a.skinned_triangles()+ tb, db = b.skinned_triangles()+ out['triangles'] = [len(ta), len(tb)]+ diffs = [] if out['joint_err'] <= POS_TOL else ['joint positions']+ if len(ta) != len(tb):+ diffs.append('triangle count')+ elif len(ta):+ from scipy.spatial import cKDTree+ mext = float(np.ptp(tb.reshape(-1, 3), 0).max()) or 1.0+ dist, idx = cKDTree(tb).query(ta)+ out['triangle_err'] = float(dist.max() / mext)+ out['dominant_joint_agree'] = float((da == db[idx]).all(1).mean())+ if out['triangle_err'] > MESH_TOL:+ diffs.append('mesh')+ out.update(verdict='different' if diffs else 'identical', differences=diffs)+ return out+++def verify(output_dir, name, profile, dataset_root='dataset'):+ """Compare ``<output_dir>/cond.npy`` and ``<output_dir>/<name>.glb`` with the+ dataset's cond entry and canonical GLB of the same asset."""+ report = {'name': name, 'profile': profile}+ ours = np.load(os.path.join(output_dir, 'cond.npy'), allow_pickle=True).item()[name]+ theirs = np.load(os.path.join(dataset_root, 'features', profile, 'cond.npy'),+ allow_pickle=True).item().get(name)+ report['cond'] = (compare_cond(ours, theirs) if theirs is not None+ else {'verdict': 'no reference', 'differences': ['not in the dataset cond']})+ glb = os.path.join(output_dir, f'{name}.glb')+ ref = os.path.join(dataset_root, 'canonical_assets', profile, f'{name}.glb')+ report['canonical_glb'] = (compare_glb(glb, ref) if os.path.isfile(ref)+ else {'verdict': 'no reference',+ 'differences': ['no dataset canonical GLB']})+ return reportdiff --git a/data_process/scripts/_common.sh b/data_process/scripts/_common.shindex 77c22fd..1ba08c8 100644--- a/data_process/scripts/_common.sh+++ b/data_process/scripts/_common.sh@@ -32,6 +32,11 @@ fiunset PYTHONSTARTUP PYTHONBREAKPOINT# โโ Default dataset layout (relative to PROJECT_ROOT) โโโโโโโโโโโโโโโโโโโโโโโ+# <dataset> is truebones | mixamo | objaverse, or general: extra training+# assets of your own, one rigged GLB (GLTF / FBX also accepted) per object in+# dataset/raw/general/animation/ (GENERAL_RAW_DIR), each holding that asset's+# clips as named animations, processed like objaverse. No shipped config+# trains on it; add "general" to dataset.dataset_list to use it.# dataset/raw/<dataset>/ raw FBX / GLB assets# dataset/export/<dataset>/ stage-1 export + stage-2/3 metadata# dataset/render/<dataset>/ multi-view renders (captioning input)@@ -41,13 +46,16 @@ export_dir() { echo "dataset/export/$1"; }render_dir() { echo "dataset/render/$1"; }tpose_dir() { echo "dataset/render/${1}_tpose"; }features_dir() { echo "dataset/features/$1"; }+# The general dataset's input assets: the only part of dataset/raw/general the+# pipeline reads.+GENERAL_RAW_DIR=dataset/raw/general/animation# โโ Helpers โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ# require_dataset <name> : validate a dataset name.require_dataset() {case "$1" in- truebones|mixamo|objaverse) ;;- *) echo "Unknown dataset '$1' (expected: truebones | mixamo | objaverse)" >&2; exit 2 ;;+ truebones|mixamo|objaverse|general) ;;+ *) echo "Unknown dataset '$1' (expected: truebones | mixamo | objaverse | general)" >&2; exit 2 ;;esac}diff --git a/data_process/scripts/run_animate_fbx.sh b/data_process/scripts/run_animate_fbx.shindex a0c2fac..27be0e5 100755--- a/data_process/scripts/run_animate_fbx.sh+++ b/data_process/scripts/run_animate_fbx.sh@@ -36,7 +36,7 @@ CHAR_STEM=$(basename "${CHAR_PATH%.*}")OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated_${CHAR_STEM}}CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-fbx}-blender -b -P data_process/mesh_animation/animate_fbx.py -- \+blender -b --python-exit-code 1 -P data_process/mesh_animation/animate_fbx.py -- \--char_path="$CHAR_PATH" \--anim_path="$ANIM_PATH" \--output_dir="$OUTPUT_DIR" \diff --git a/data_process/scripts/run_animate_lbs.sh b/data_process/scripts/run_animate_lbs.shindex d52280b..a250ea2 100755--- a/data_process/scripts/run_animate_lbs.sh+++ b/data_process/scripts/run_animate_lbs.sh@@ -1,19 +1,27 @@#!/bin/bash# Stage 5 โ deform a rigged asset with a motion NPZ via manual NumPy LBS and-# save the vertex animation (NPZ, optionally per-frame OBJ).+# save the animated rigged asset (GLB by default; FBX, vertex NPZ and per-frame+# OBJ on request, SAVE).## Accepts both motion flavors (auto-detected): export-stage NPZ, or-# feature-format (generated) NPZ โ the latter needs DATASET_TYPE.+# feature-format NPZ โ the latter through the cond (DATASET_TYPE / COND_PATH),+# or with no cond on a canonical asset (dataset/canonical_assets/<ds>/).## Usage:# CHAR_PATH=asset.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_lbs.sh-# CHAR_PATH=asset.glb ANIM_PATH=gen.npz DATASET_TYPE=objaverse bash data_process/scripts/run_animate_lbs.sh+# ANIM_PATH=dataset/features/truebones/motions/Dog-Walk-000.npz DATASET_TYPE=truebones \+# bash data_process/scripts/run_animate_lbs.sh # -> canonical_assets/truebones/Dog.glb+# ANIM_PATH=<feature>.npz DATASET_TYPE=mixamo CHARACTER=Abe bash data_process/scripts/run_animate_lbs.sh## Env overrides:-# CHAR_PATH (required โ rigged asset GLB/FBX with skinning weights)+# CHAR_PATH rigged asset GLB/FBX with skinning weights; optional with+# DATASET_TYPE: the processed asset of the clip (canonical for a+# feature NPZ, export for an export NPZ)+# CHARACTER Mixamo character for that (default Michelle)# ANIM_PATH (required โ motion NPZ, or a directory of NPZs: one output# set per action clip)-# DATASET_TYPE truebones|objaverse|mixamo (feature-format motion only)+# DATASET_TYPE truebones|objaverse|mixamo|general: resolves CHAR_PATH when+# unset, and the cond of a feature NPZ on a non-canonical asset# COND_PATH cond.npy override (feature-format motion only)# OUTPUT_DIR (default: outputs/lbs)# SAVE comma-separated: glb,fbx,npz,obj (default: glb)@@ -24,17 +32,21 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-CHAR_PATH=${CHAR_PATH:?Set CHAR_PATH to a rigged asset GLB/FBX}+if [[ -z "${CHAR_PATH:-}" && -z "${DATASET_TYPE:-}" ]]; then+ echo "Set CHAR_PATH, or DATASET_TYPE to use the processed asset of the clip" >&2+ exit 1+fiANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion NPZ}OUTPUT_DIR=${OUTPUT_DIR:-outputs/lbs}SAVE=${SAVE:-glb}EXTRA_ARGS=()+[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")+[[ -n "${CHARACTER:-}" ]] && EXTRA_ARGS+=(--character="$CHARACTER")[[ -n "${DATASET_TYPE:-}" ]] && EXTRA_ARGS+=(--dataset_type="$DATASET_TYPE")[[ -n "${COND_PATH:-}" ]] && EXTRA_ARGS+=(--cond_path="$COND_PATH")-blender -b -P data_process/mesh_animation/animate_lbs.py -- \- --char_path="$CHAR_PATH" \+blender -b --python-exit-code 1 -P data_process/mesh_animation/animate_lbs.py -- \--anim_path="$ANIM_PATH" \--output_dir="$OUTPUT_DIR" \--save="$SAVE" \diff --git a/data_process/scripts/run_animate_mixamo.sh b/data_process/scripts/run_animate_mixamo.shindex 7f69a6c..c1f9a2c 100755--- a/data_process/scripts/run_animate_mixamo.sh+++ b/data_process/scripts/run_animate_mixamo.sh@@ -29,7 +29,7 @@ NPZ_DIR=${NPZ_DIR:-$(export_dir mixamo)/motions}OUTPUT_DIR=${OUTPUT_DIR:-outputs/mixamo_characters}CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-fbx}-blender -b -P data_process/mesh_animation/animate_mixamo.py -- \+blender -b --python-exit-code 1 -P data_process/mesh_animation/animate_mixamo.py -- \--char_path="$CHAR_PATH" \--npz_dir="$NPZ_DIR" \--output_dir="$OUTPUT_DIR" \diff --git a/data_process/scripts/run_animate_motion.sh b/data_process/scripts/run_animate_motion.shindex 7c9ab7b..fa47858 100755--- a/data_process/scripts/run_animate_motion.sh+++ b/data_process/scripts/run_animate_motion.sh@@ -1,18 +1,30 @@#!/bin/bash# Stage 5 โ drive a rigged character with a canonicalized motion clip (stage-4# format, or a generated motion in that format) and export GLB + FBX. The-# character is auto-resolved from the dataset + clip name for truebones and-# objaverse; mixamo needs CHAR_PATH.+# character is auto-resolved from the dataset + clip name (Mixamo clips carry+# no object type, so there it is CHARACTER, default Michelle):+# ASSET=canonical (default): the canonical asset (dataset/canonical_assets/<ds>/),+# motion kept at canonical scale; an object type without one falls back+# to the export resolution with a warning.+# ASSET=export: the processed GLB (dataset/export/<ds>/rigs/, from+# `run_export.sh <ds> --glb_only`) when one exists, else the raw asset+# (Mixamo: rigs/<CHARACTER>.glb, else, for the default character only,+# the raw Y_Bot.fbx), motion scaled+# back to the export's units.## Usage:-# ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse+# ANIM_PATH=sample.npy bash data_process/scripts/run_animate_motion.sh objaverse# ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh truebones-# ANIM_PATH=clip.npz CHAR_PATH=char.fbx bash data_process/scripts/run_animate_motion.sh mixamo+# ANIM_PATH=clip.npz CHARACTER=Abe bash data_process/scripts/run_animate_motion.sh mixamo+# ANIM_PATH=clip.npz ASSET=export bash data_process/scripts/run_animate_motion.sh truebones## Env overrides:# ANIM_PATH (required) motion .npz, or a .npy of model motion features-# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; mixamo-# defaults to dataset/raw/mixamo/character_refined/Y_Bot.fbx)+# CHAR_PATH character mesh (default: auto-resolved, above)+# CHARACTER Mixamo character (default Michelle)+# ASSET canonical (default) | export, see above+# CANONICAL 1: require the canonical asset (no export fallback);+# 0: same as ASSET=export# COND_PATH cond.npy (default: dataset/features/<dataset>/cond.npy)# OUTPUT_DIR (default: outputs/animated)# ANIM_MODE fk|ik, .npy input only (default: fk)@@ -24,7 +36,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_animate_motion.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_animate_motion.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"@@ -32,25 +44,19 @@ ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion .npz/.npy for dataset=$DATASET}OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}ANIM_MODE=${ANIM_MODE:-fk}-# Mixamo has no per-object mesh to auto-resolve; the animation FBXs are-# authored on Y Bot, so that character is the default when it is downloaded.-MIXAMO_DEFAULT_CHAR=dataset/raw/mixamo/character_refined/Y_Bot.fbx-if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then- if [[ -f "$MIXAMO_DEFAULT_CHAR" ]]; then- CHAR_PATH=$MIXAMO_DEFAULT_CHAR- else- echo "dataset=mixamo requires CHAR_PATH (default $MIXAMO_DEFAULT_CHAR not found;" \- "download it with run_download.sh mixamo)" >&2- exit 1- fi-fi-EXTRA_ARGS=()+[[ -n "${CHARACTER:-}" ]] && EXTRA_ARGS+=(--character="$CHARACTER")+ASSET=${ASSET:-canonical}+case "${CANONICAL:-}" in+ 1) ASSET=canonical; EXTRA_ARGS+=(--canonical) ;;+ 0) ASSET=export ;;+esac+EXTRA_ARGS+=(--asset="$ASSET")[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")[[ -n "${COND_PATH:-}" ]] && EXTRA_ARGS+=(--cond_path="$COND_PATH")[[ -n "${EXTRA_BONES_STRATEGY:-}" ]] && EXTRA_ARGS+=(--extra_bones_strategy="$EXTRA_BONES_STRATEGY")-blender -b -P data_process/mesh_animation/animate_motion.py -- \+blender -b --python-exit-code 1 -P data_process/mesh_animation/animate_motion.py -- \--dataset_type="$DATASET" \--anim_path="$ANIM_PATH" \--output_dir="$OUTPUT_DIR" \diff --git a/data_process/scripts/run_animate_npz.sh b/data_process/scripts/run_animate_npz.shindex c30055b..c32bc9e 100755--- a/data_process/scripts/run_animate_npz.sh+++ b/data_process/scripts/run_animate_npz.sh@@ -4,9 +4,14 @@## Usage:# CHAR_PATH=char.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh+# ANIM_PATH=dataset/export/truebones/motions/Dog-Walk.npz DATASET_TYPE=truebones \+# bash data_process/scripts/run_animate_npz.sh # -> export/truebones/rigs/Dog.glb## Env overrides:-# CHAR_PATH (required โ rigged character GLB/FBX matching the NPZ skeleton)+# CHAR_PATH rigged character GLB/FBX matching the NPZ skeleton; optional+# with DATASET_TYPE: the processed export asset of the clip+# DATASET_TYPE truebones|objaverse|mixamo|general (to resolve CHAR_PATH)+# CHARACTER Mixamo character for that (default Michelle)# ANIM_PATH (required โ NPZ whose skeleton matches the rig)# OUTPUT_DIR (default: outputs/animated)# CHAR_ANIM_TYPE glb|fbx (default: glb)@@ -18,16 +23,21 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-CHAR_PATH=${CHAR_PATH:?Set CHAR_PATH to a rigged character GLB/FBX}+if [[ -z "${CHAR_PATH:-}" && -z "${DATASET_TYPE:-}" ]]; then+ echo "Set CHAR_PATH, or DATASET_TYPE to use the processed export asset of the clip" >&2+ exit 1+fiANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion NPZ matching the rig}OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-glb}EXTRA_ARGS=()+[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")+[[ -n "${DATASET_TYPE:-}" ]] && EXTRA_ARGS+=(--dataset_type="$DATASET_TYPE")+[[ -n "${CHARACTER:-}" ]] && EXTRA_ARGS+=(--character="$CHARACTER")[[ -n "${EXTRA_BONES_STRATEGY:-}" ]] && EXTRA_ARGS+=(--extra_bones_strategy="$EXTRA_BONES_STRATEGY")-blender -b -P data_process/mesh_animation/animate_npz.py -- \- --char_path="$CHAR_PATH" \+blender -b --python-exit-code 1 -P data_process/mesh_animation/animate_npz.py -- \--anim_path="$ANIM_PATH" \--output_dir="$OUTPUT_DIR" \--char_anim_type="$CHAR_ANIM_TYPE" "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_caption_category.sh b/data_process/scripts/run_caption_category.shindex 742f38a..cb1a1b0 100755--- a/data_process/scripts/run_caption_category.sh+++ b/data_process/scripts/run_caption_category.sh@@ -20,7 +20,7 @@ set -euo pipefailsource "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_caption_category.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_caption_category.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_caption_motion.sh b/data_process/scripts/run_caption_motion.shindex de38b24..a1b71ab 100755--- a/data_process/scripts/run_caption_motion.sh+++ b/data_process/scripts/run_caption_motion.sh@@ -3,7 +3,7 @@# The backend is picked from MODEL, with the same rules as detect_backend() in# vlm_caption/backends.py:# *qwen* (default, any case) -> local Qwen3.5 / Qwen3.8 / Qwen3-VL (GPU-bound; supports --multi-gpu)-# gemini* -> Gemini API (needs GOOGLE_API_KEY)+# *gemini* -> Gemini API (needs GOOGLE_API_KEY)# anything else -> OpenAI-compatible API (needs OPENAI_API_KEY)## Usage:@@ -13,6 +13,7 @@# MODEL=Qwen/Qwen3-VL-8B-Instruct bash data_process/scripts/run_caption_motion.sh objaverse# MODEL=gemini-3-flash-preview bash data_process/scripts/run_caption_motion.sh objaverse# MODEL=gpt-5-mini NUM_WORKERS=16 bash data_process/scripts/run_caption_motion.sh mixamo+# bash data_process/scripts/run_caption_motion.sh general # extra assets (objaverse prompt)## Env overrides:# MODEL, RENDER_ROOT (default: dataset/render/<dataset>),@@ -20,6 +21,9 @@# DOWNSAMPLE_RATE, MAX_TOKENS, NUM_WORKERS, MAX_RETRIES, BASE_URL (OpenAI-compatible proxy),# THINKING_LEVEL / MEDIA_RESOLUTION (gemini-3), THINKING_BUDGET (gemini-2.5)# Unrecognized arguments are passed through to caption_motion.py.+# Exit status: 0 when every clip got a caption, 3 when the run finished but some+# clips did not (listed in the *_failed.txt next to OUTPUT_JSON; rerun to retry+# them; --multi-gpu still merges the shards), 1 on a crash.# On offline compute nodes with a populated HuggingFace cache, export# HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1.@@ -28,7 +32,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"# โโ Arguments โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโhandle_help "$@"-DATASET=${1:?Usage: run_caption_motion.sh <truebones|mixamo|objaverse> [--multi-gpu] [extra args...]}+DATASET=${1:?Usage: run_caption_motion.sh <truebones|mixamo|objaverse|general> [--multi-gpu] [extra args...]}shiftrequire_dataset "$DATASET"@@ -139,10 +143,19 @@ if [[ "$MODE" == "multi-gpu" ]]; thenecho " Started GPU $gpu_id (PID ${pids[-1]})"done+ # Exit 3 = finished with some clips uncaptioned (caption_motion.EXIT_CLIPS_FAILED):+ # merge anyway, then report it; any other non-zero status is a crash.failed=0+ partial=0for gpu_id in "${!pids[@]}"; do- wait "${pids[gpu_id]}" \- || { echo "GPU $gpu_id failed (log: $LOG_DIR/gpu${gpu_id}.log)" >&2; failed=1; }+ status=0+ wait "${pids[gpu_id]}" || status=$?+ if [[ $status -eq 3 ]]; then+ partial=1+ elif [[ $status -ne 0 ]]; then+ echo "GPU $gpu_id failed (log: $LOG_DIR/gpu${gpu_id}.log)" >&2+ failed=1+ fidoneif [[ $failed -ne 0 ]]; thenecho "Some processes failed. Shard files preserved for inspection." >&2@@ -153,6 +166,10 @@ if [[ "$MODE" == "multi-gpu" ]]; thenpython -m data_process.vlm_caption.caption_motion \--task="$DATASET" --merge_shards --output_json="$OUTPUT_JSON" --num_gpus="$num_gpus"echo "Done! Output: $OUTPUT_JSON"+ if [[ $partial -ne 0 ]]; then+ echo "Some clips failed to caption (see the *_failed.txt next to $OUTPUT_JSON); rerun to retry them." >&2+ exit 3+ fielsepython -m data_process.vlm_caption.caption_motion \"${COMMON[@]}" "${EXTRA[@]}" "${PASSTHRU[@]}"diff --git a/data_process/scripts/run_download.sh b/data_process/scripts/run_download.shindex 4c74833..6d5ac3a 100755--- a/data_process/scripts/run_download.sh+++ b/data_process/scripts/run_download.sh@@ -44,7 +44,7 @@ if [[ "$DATASET" == "objaverse_renders" ]]; thenREPO=Linzhan/Objaverse-XL-Rigged-Animated-RendersRAW_DIR=${RAW_DIR:-dataset/raw/objaverse_renders}echo "Downloading $REPO -> $RAW_DIR"- exec hf download "$REPO" --type dataset --local-dir "$RAW_DIR" "$@"+ exec hf download "$REPO" --repo-type dataset --local-dir "$RAW_DIR" "$@"firequire_dataset "$DATASET"@@ -73,7 +73,7 @@ esacRAW_DIR=${RAW_DIR:-dataset/raw/$DATASET}echo "Downloading $REPO -> $RAW_DIR"-if ! hf download "$REPO" --type dataset --local-dir "$RAW_DIR" "$@"; then+if ! hf download "$REPO" --repo-type dataset --local-dir "$RAW_DIR" "$@"; thenif [[ "$DATASET" == "truebones" ]]; thenecho "Download failed. Note that only annotations are hosted; the motion files" >&2echo "are commercial: purchase the Truebones ZOO pack from https://truebones.com" >&2diff --git a/data_process/scripts/run_export.sh b/data_process/scripts/run_export.shindex 685b924..386ed98 100755--- a/data_process/scripts/run_export.sh+++ b/data_process/scripts/run_export.sh@@ -7,16 +7,26 @@# bash data_process/scripts/run_export.sh objaverse # dataset preset (GLB/GLTF)# bash data_process/scripts/run_export.sh objaverse --multi-worker 8 # parallel (mixamo too)# bash data_process/scripts/run_export.sh mixamo --multi-worker 8 --no-vis # skip MP4 previews-# DATA_DIR=my_assets bash data_process/scripts/run_export.sh # auto: .glb/.gltf and .fbx side by side+# bash data_process/scripts/run_export.sh general --multi-worker 8 # your extra assets+# DATA_DIR=my_assets bash data_process/scripts/run_export.sh general # ... from another folder+# bash data_process/scripts/run_export.sh truebones --save_glb # + rigs/<asset>.glb (rest pose, no animation)+# bash data_process/scripts/run_export.sh truebones --glb_only # only add missing GLBs to an existing export## --multi-worker only applies to a *directory* input; truebones is exported by-# a single-process exporter and ignores it (with a warning).+# a single-process exporter and ignores it (with a warning), except under+# --glb_only, which shards its species.#-# Auto mode runs export_general.py on the directory: GLB/GLTF and FBX side-# by side, importer picked per file by extension, mesh optional (armature-only-# FBX works too).+# general is extra training data of your own: one rigged asset per file in+# dataset/raw/general/animation/ (objaverse-style GLB/GLTF, each animation a+# clip; FBX side by side works too),+# exported by export_general.py to dataset/export/general. The importer is+# picked per file by extension, the mesh is optional (armature-only FBX works,+# but only meshed assets can be rendered and captioned later), and the asset+# name is the file stem with '-' and whitespace replaced by '_'. Calling the+# script without a dataset (or with `auto`, an alias) means general.#-# Env overrides: DATA_DIR, OUTPUT_DIR+# Env overrides: DATA_DIR, OUTPUT_DIR, CHAR_DIR (mixamo: the characters whose+# GLBs --save_glb / --glb_only write; default dataset/raw/mixamo/character_refined)# Extra arguments are passed through to the exporter.set -euo pipefail@@ -27,23 +37,26 @@ EXPORTERS=data_process/motion_export# โโ Parse mode + flags โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโhandle_help "$@"-MODE=auto+MODE=generalcase "${1:-}" in- truebones|mixamo|objaverse|auto) MODE="$1"; shift ;;- ""|--*) ;; # no dataset given: auto mode (DATA_DIR required), flags follow+ truebones|mixamo|objaverse|general) MODE="$1"; shift ;;+ auto) MODE=general; shift ;; # alias of general+ ""|--*) ;; # no dataset given: general, flags follow*)# A bare word that is not a dataset name is almost always a typo;- # silently treating it as an exporter flag would land in auto mode- # and fail later on a missing DATA_DIR.- echo "ERROR: unknown dataset '$1' (expected truebones | mixamo | objaverse | auto)" >&2+ # silently treating it as an exporter flag would export the general+ # dataset instead.+ echo "ERROR: unknown dataset '$1' (expected truebones | mixamo | objaverse | general)" >&2exit 2 ;;esacNUM_WORKERS=1+GLB_ONLY=0EXTRA_ARGS=()while [[ $# -gt 0 ]]; docase "$1" in--multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ --glb_only) GLB_ONLY=1; EXTRA_ARGS+=("$1"); shift ;;*) EXTRA_ARGS+=("$1"); shift ;;esacdone@@ -55,13 +68,25 @@ case "$MODE" inOUTPUT_DIR=${OUTPUT_DIR:-$(export_dir truebones)} ;;mixamo)DATA_DIR=${DATA_DIR:-dataset/raw/mixamo/animation_motion}- OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir mixamo)} ;;+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir mixamo)}+ # The characters the processed GLBs are made of (--save_glb / --glb_only).+ CHAR_DIR=${CHAR_DIR:-dataset/raw/mixamo/character_refined} ;;objaverse)DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb}OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir objaverse)} ;;- auto)- DATA_DIR=${DATA_DIR:?auto mode requires DATA_DIR (directory of .glb/.gltf/.fbx assets)}- OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir custom)} ;;+ general)+ DATA_DIR=${DATA_DIR:-$GENERAL_RAW_DIR}+ OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir general)}+ if [[ ! -e "$DATA_DIR" ]]; then+ echo "ERROR: $DATA_DIR does not exist; put your rigged assets there" \+ "(one .glb per object) or set DATA_DIR." >&2+ exit 2+ fi+ if [[ "$(realpath -m "$DATA_DIR")" != "$(realpath -m "$GENERAL_RAW_DIR")" ]]; then+ echo "NOTE: general assets from $DATA_DIR: pass the same DATA_DIR to" \+ "run_render_motion.sh and run_render_tpose.sh, which read $GENERAL_RAW_DIR" \+ "by default (a clip without renders gets no caption and stage 4 drops it)."+ fi ;;esac# โโ Exporter invocations โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ@@ -78,7 +103,7 @@ launch_workers() {echo "Launching $NUM_WORKERS export workers (per-worker logs: $log_dir)..."local pids=() worker_id failed=0for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do- blender -b -P "$EXPORTERS/$script" -- \+ blender -b --python-exit-code 1 -P "$EXPORTERS/$script" -- \"$input_flag=$DATA_DIR" --output_dir="$OUTPUT_DIR" \--worker_id="$worker_id" --num_workers="$NUM_WORKERS" \"${EXTRA_ARGS[@]}" > >(tee "$log_dir/worker${worker_id}.log") 2>&1 &@@ -99,6 +124,10 @@ launch_workers() {echo "then merge with: python -m data_process.tools.merge_summaries --output_dir=$OUTPUT_DIR" >&2exit 1fi+ if [[ "$GLB_ONLY" == 1 ]]; then+ echo "All workers finished (--glb_only: no summaries to merge)."+ return+ fiecho "All workers finished. Merging summary JSONs..."python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"else@@ -106,20 +135,31 @@ launch_workers() {echo "WARNING: --multi-worker is ignored for a single-file input" \"($DATA_DIR); running one exporter process." >&2fi- blender -b -P "$EXPORTERS/$script" -- \+ blender -b --python-exit-code 1 -P "$EXPORTERS/$script" -- \"$input_flag=$DATA_DIR" --output_dir="$OUTPUT_DIR" "${EXTRA_ARGS[@]}"fi}export_glb() { launch_workers export_objaverse.py --data_dir; }-export_mixamo() { launch_workers export_mixamo.py --anim_dir; }+export_mixamo() {+ case " ${EXTRA_ARGS[*]} " in+ # Default first: an explicit --char_dir later in the arguments wins.+ *" --save_glb "*|*" --glb_only "*) EXTRA_ARGS=(--char_dir="$CHAR_DIR" "${EXTRA_ARGS[@]}") ;;+ esac+ launch_workers export_mixamo.py --anim_dir+}export_fbx() {+ # A GLB-only run writes per-clip files only, so its species can be sharded.+ if [[ "$GLB_ONLY" == 1 && "$NUM_WORKERS" -gt 1 ]]; then+ launch_workers export_truebones.py --data_dir+ return+ fiif [[ "$NUM_WORKERS" -gt 1 ]]; thenecho "WARNING: --multi-worker is ignored for truebones โ export_truebones.py" \- "is single-process; running one exporter process." >&2+ "is single-process; running one exporter process (except with --glb_only)." >&2fi- blender -b -P "$EXPORTERS/export_truebones.py" -- \+ blender -b --python-exit-code 1 -P "$EXPORTERS/export_truebones.py" -- \--data_dir="$DATA_DIR" --output_dir="$OUTPUT_DIR" "$@" "${EXTRA_ARGS[@]}"}@@ -129,5 +169,5 @@ case "$MODE" inobjaverse) export_glb ;;mixamo) export_mixamo ;;truebones) export_fbx ;;- auto) launch_workers export_general.py --input ;;+ general) launch_workers export_general.py --input ;;esacdiff --git a/data_process/scripts/run_export_general.sh b/data_process/scripts/run_export_general.shindex 0ff2602..983561e 100755--- a/data_process/scripts/run_export_general.sh+++ b/data_process/scripts/run_export_general.sh@@ -2,13 +2,18 @@# Stage 1 โ export rigged GLB/GLTF/FBX assets to NPZ motion data.# Input is a single file or a directory of assets (mixed formats welcome).# Mesh optional (armature-only FBX works); every pose action becomes a clip.+# `run_export.sh general` is the same exporter on dataset/raw/general/animation/,+# the general dataset's inputs: only files there are rendered, captioned and+# extracted by the later stages, so an input inside that directory exports+# into dataset/export/general and any other input into a one-off+# dataset/export/custom.## Usage:# bash data_process/scripts/run_export_general.sh <asset.glb|.fbx|dir> [extra args...]# bash data_process/scripts/run_export_general.sh my_assets/ --multi-worker 8 --no-vis-# bash data_process/scripts/run_export_general.sh model.fbx --no-prune+# OUTPUT_DIR=outputs/my_export bash data_process/scripts/run_export_general.sh model.fbx --no-prune#-# Env overrides: OUTPUT_DIR (default: dataset/export/custom)+# Env overrides: OUTPUT_DIR (default above)set -euo pipefailsource "$(dirname "${BASH_SOURCE[0]}")/_common.sh"@@ -16,13 +21,22 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"INPUT="${1:?Usage: run_export_general.sh <asset.glb|.fbx|dir> [extra args...]}"shift-OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir custom)}+if [[ -z "${OUTPUT_DIR:-}" ]]; then+ case "$(realpath -m "$INPUT")/" in+ "$(realpath -m "$GENERAL_RAW_DIR")"/*) OUTPUT_DIR=$(export_dir general) ;;+ *) OUTPUT_DIR=$(export_dir custom)+ echo "Input outside $GENERAL_RAW_DIR: exporting to $OUTPUT_DIR (one-off;" \+ "put the files in $GENERAL_RAW_DIR to add them to the general dataset)." ;;+ esac+fiNUM_WORKERS=1+GLB_ONLY=0EXTRA_ARGS=()while [[ $# -gt 0 ]]; docase "$1" in--multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ --glb_only) GLB_ONLY=1; EXTRA_ARGS+=("$1"); shift ;;*) EXTRA_ARGS+=("$1"); shift ;;esacdone@@ -32,7 +46,7 @@ if [[ "$NUM_WORKERS" -gt 1 && -d "$INPUT" ]]; thenecho "Launching $NUM_WORKERS export workers (per-worker logs: $LOG_DIR)..."pids=()for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do- blender -b -P data_process/motion_export/export_general.py -- \+ blender -b --python-exit-code 1 -P data_process/motion_export/export_general.py -- \--input="$INPUT" --output_dir="$OUTPUT_DIR" \--worker_id="$worker_id" --num_workers="$NUM_WORKERS" \"${EXTRA_ARGS[@]}" > >(tee "$LOG_DIR/worker${worker_id}.log") 2>&1 &@@ -52,13 +66,17 @@ if [[ "$NUM_WORKERS" -gt 1 && -d "$INPUT" ]]; thenecho "Per-worker shards are preserved in $OUTPUT_DIR; rerun to finish." >&2exit 1fi- echo "All workers finished. Merging summary JSONs..."- python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"+ if [[ "$GLB_ONLY" == 1 ]]; then+ echo "All workers finished (--glb_only: no summaries to merge)."+ else+ echo "All workers finished. Merging summary JSONs..."+ python -m data_process.tools.merge_summaries --output_dir="$OUTPUT_DIR"+ fielseif [[ "$NUM_WORKERS" -gt 1 ]]; thenecho "WARNING: --multi-worker is ignored for a single-file input" \"($INPUT); running one exporter process." >&2fi- blender -b -P data_process/motion_export/export_general.py -- \+ blender -b --python-exit-code 1 -P data_process/motion_export/export_general.py -- \--input="$INPUT" --output_dir="$OUTPUT_DIR" "${EXTRA_ARGS[@]}"fidiff --git a/data_process/scripts/run_extract_features.sh b/data_process/scripts/run_extract_features.shindex d6abfd7..e2c7f01 100755--- a/data_process/scripts/run_extract_features.sh+++ b/data_process/scripts/run_extract_features.sh@@ -7,6 +7,7 @@# bash data_process/scripts/run_extract_features.sh mixamo# APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones# NUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse+# bash data_process/scripts/run_extract_features.sh general # extra assets, objaverse rules## Env overrides:# DATA_DIR Stage-1 export directory (default: dataset/export/<dataset>)@@ -15,6 +16,12 @@# clips; otherwise (default) keep only the first max_clip_len frames.# NUM_WORKERS Parallel worker processes over object types (default: 1)# NO_VIS If 1/true, skip the per-clip MP4 previews+# SAVE_GLB If 1/true, also bake the export's rigs/<asset>.glb into the+# canonical frame: canonical_assets/<dataset>/<object_type>.glb+# beside the features root (dataset/canonical_assets/<dataset>),+# driven by the feature clips with no cond.npy (canonical_assets.py)+# CANONICAL_ASSETS_DIR where SAVE_GLB writes instead (default above; a+# SAVE_DIR outside a features root gives <SAVE_DIR>_canonical_assets)# Extra arguments are passed through to extract_features.py.set -euo pipefail@@ -22,7 +29,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_extract_features.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_extract_features.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"@@ -32,9 +39,9 @@ SAVE_DIR=${SAVE_DIR:-$(features_dir "$DATASET")}EXTRA_ARGS=()# Per-dataset joint-count range (skeletons outside it are skipped here, not# at export โ the exports stay complete). Defaults in extract_features.py-# are 8..150; objaverse uses a wider range.+# are 8..150; objaverse (and general, built like it) uses a wider range.case "$DATASET" in- objaverse) EXTRA_ARGS+=(--min_joints 4 --max_joints 180) ;;+ objaverse|general) EXTRA_ARGS+=(--min_joints 4 --max_joints 180) ;;esaccase "${APPLY_CLIP:-0}" in1|true|TRUE) EXTRA_ARGS+=(--apply_clip) ;;@@ -42,6 +49,10 @@ esaccase "${NO_VIS:-0}" in1|true|TRUE) EXTRA_ARGS+=(--no-vis) ;;esac+case "${SAVE_GLB:-0}" in+ 1|true|TRUE) EXTRA_ARGS+=(--save_glb) ;;+esac+[[ -n "${CANONICAL_ASSETS_DIR:-}" ]] && EXTRA_ARGS+=(--canonical_assets_dir="$CANONICAL_ASSETS_DIR")[[ "${NUM_WORKERS:-1}" -gt 1 ]] && EXTRA_ARGS+=(--num_workers="$NUM_WORKERS")python -m data_process.feature_extraction.extract_features \diff --git a/data_process/scripts/run_joints_face_correct_llm.sh b/data_process/scripts/run_joints_face_correct_llm.shindex ac01492..4338003 100755--- a/data_process/scripts/run_joints_face_correct_llm.sh+++ b/data_process/scripts/run_joints_face_correct_llm.sh@@ -18,7 +18,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_face_correct_llm.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_joints_face_correct_llm.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_joints_face_select_llm.sh b/data_process/scripts/run_joints_face_select_llm.shindex 63950dc..b40a45a 100755--- a/data_process/scripts/run_joints_face_select_llm.sh+++ b/data_process/scripts/run_joints_face_select_llm.sh@@ -13,7 +13,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_face_select_llm.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_joints_face_select_llm.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_joints_face_select_rule.sh b/data_process/scripts/run_joints_face_select_rule.shindex 0056a82..3ee6419 100755--- a/data_process/scripts/run_joints_face_select_rule.sh+++ b/data_process/scripts/run_joints_face_select_rule.sh@@ -3,7 +3,7 @@# Reads joint_names.json + clean_joint_names.json from the export directory.## Usage:-# bash data_process/scripts/run_joints_face_select_rule.sh <truebones|mixamo|objaverse>+# bash data_process/scripts/run_joints_face_select_rule.sh <truebones|mixamo|objaverse|general>## Env overrides: INPUT_DIR, OUTPUT@@ -12,7 +12,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_face_select_rule.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_joints_face_select_rule.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_joints_names_clean_llm.sh b/data_process/scripts/run_joints_names_clean_llm.shindex bf7b086..10b24ce 100755--- a/data_process/scripts/run_joints_names_clean_llm.sh+++ b/data_process/scripts/run_joints_names_clean_llm.sh@@ -16,7 +16,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_names_clean_llm.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_joints_names_clean_llm.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_joints_names_clean_rule.sh b/data_process/scripts/run_joints_names_clean_rule.shindex 07d9966..01ee536 100755--- a/data_process/scripts/run_joints_names_clean_rule.sh+++ b/data_process/scripts/run_joints_names_clean_rule.sh@@ -2,7 +2,7 @@# Stage 3 โ rule-based joint-name cleaning: joint_names.json -> clean_joint_names.json.## Usage:-# bash data_process/scripts/run_joints_names_clean_rule.sh <truebones|mixamo|objaverse> [--report]+# bash data_process/scripts/run_joints_names_clean_rule.sh <truebones|mixamo|objaverse|general> [--report]## Env overrides: INPUT, OUTPUT (default: inside dataset/export/<dataset>)@@ -11,7 +11,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_names_clean_rule.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_joints_names_clean_rule.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_joints_names_correct_llm.sh b/data_process/scripts/run_joints_names_correct_llm.shindex 4472d62..6e2ffdd 100755--- a/data_process/scripts/run_joints_names_correct_llm.sh+++ b/data_process/scripts/run_joints_names_correct_llm.sh@@ -18,7 +18,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_names_correct_llm.sh <truebones|mixamo|objaverse> [extra args...]}+DATASET=${1:?Usage: run_joints_names_correct_llm.sh <truebones|mixamo|objaverse|general> [extra args...]}shiftrequire_dataset "$DATASET"diff --git a/data_process/scripts/run_joints_vis_facing.sh b/data_process/scripts/run_joints_vis_facing.shindex 7a8a849..825caff 100755--- a/data_process/scripts/run_joints_vis_facing.sh+++ b/data_process/scripts/run_joints_vis_facing.sh@@ -26,7 +26,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-DATASET=${1:?Usage: run_joints_vis_facing.sh <truebones|mixamo|objaverse>}+DATASET=${1:?Usage: run_joints_vis_facing.sh <truebones|mixamo|objaverse|general>}require_dataset "$DATASET"EXPORT_DIR=$(export_dir "$DATASET")[[ -d "$EXPORT_DIR/motions" ]] || { echo "export motions not found: $EXPORT_DIR/motions" >&2; exit 1; }diff --git a/data_process/scripts/run_joints_vis_tpose.sh b/data_process/scripts/run_joints_vis_tpose.shindex e27fe1c..b7139e6 100755--- a/data_process/scripts/run_joints_vis_tpose.sh+++ b/data_process/scripts/run_joints_vis_tpose.sh@@ -31,7 +31,7 @@ if [[ -n "${NPZ_PATH:-}" ]]; thencmd+=(--npz_path "$NPZ_PATH")[[ -n "${OUTPUT_PATH:-}" ]] && cmd+=(--output "$OUTPUT_PATH")else- DATASET=${1:?Usage: run_joints_vis_tpose.sh <truebones|mixamo|objaverse> (or set NPZ_PATH)}+ DATASET=${1:?Usage: run_joints_vis_tpose.sh <truebones|mixamo|objaverse|general> (or set NPZ_PATH)}require_dataset "$DATASET"MOTIONS_DIR=${MOTIONS_DIR:-$(export_dir "$DATASET")/motions}OUTPUT_DIR=${OUTPUT_DIR:-$(export_dir "$DATASET")/face_joints_vis}diff --git a/data_process/scripts/run_preprocess_char.sh b/data_process/scripts/run_preprocess_char.shdeleted file mode 100755index cea44c0..0000000--- a/data_process/scripts/run_preprocess_char.sh+++ /dev/null@@ -1,45 +0,0 @@-#!/bin/bash-# Stage 5 โ preprocess one rigged, animated asset through the real export +-# feature pipeline. Leaves exactly three deliverables in OUTPUT_DIR:-# <name>_canonical.{glb,fbx} canonical rest-pose asset-# cond.npy model-side topology conditioning-# motions/<clip>.npz motion features-# Feature NPZs then drive the canonical asset cond-free via run_animate_lbs.sh.-# (KEEP_INTERMEDIATE=1 keeps the export-stage NPZs and visuals.)-#-# Usage:-# CHAR_PATH=asset.glb OUTPUT_DIR=outputs/asset \-# FACE_R=R_Thigh FACE_L=L_Thigh FORMATS=glb,fbx \-# bash data_process/scripts/run_preprocess_char.sh-#-# Env overrides:-# CHAR_PATH (required โ rigged, ANIMATED asset GLB/FBX)-# OUTPUT_DIR (required โ export/, motions/, cond.npy, <name>_canonical.*)-# FACE_R/FACE_L raw face-joint names for facing canonicalization-# BODY_AXIS=1 face pair is a head/tail body axis (serpentine rigs)-# FORMATS comma-separated canonical formats (default: glb)-# SAVE_VIS=1 also render per-clip preview MP4s-# APPLY_CLIP=1 crop motions into training-length clips-# KEEP_INTERMEDIATE=1 keep export-stage NPZs and T-pose/preview visuals--set -euo pipefail-source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"--handle_help "$@"--CHAR_PATH=${CHAR_PATH:?Set CHAR_PATH to a rigged, animated asset GLB/FBX}-OUTPUT_DIR=${OUTPUT_DIR:?Set OUTPUT_DIR for the preprocessed outputs}--EXTRA_ARGS=()-[[ -n "${FACE_R:-}" ]] && EXTRA_ARGS+=(--face_r="$FACE_R")-[[ -n "${FACE_L:-}" ]] && EXTRA_ARGS+=(--face_l="$FACE_L")-[[ -n "${BODY_AXIS:-}" ]] && EXTRA_ARGS+=(--body_axis)-[[ -n "${FORMATS:-}" ]] && EXTRA_ARGS+=(--formats="$FORMATS")-[[ -n "${SAVE_VIS:-}" ]] && EXTRA_ARGS+=(--save_vis)-[[ -n "${APPLY_CLIP:-}" ]] && EXTRA_ARGS+=(--apply_clip)-[[ -n "${KEEP_INTERMEDIATE:-}" ]] && EXTRA_ARGS+=(--keep_intermediate)--blender -b -P data_process/mesh_animation/preprocess_char.py -- \- --char_path="$CHAR_PATH" \- --output_dir="$OUTPUT_DIR" \- "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_render_motion.sh b/data_process/scripts/run_render_motion.shindex bb81a40..ba88add 100755--- a/data_process/scripts/run_render_motion.sh+++ b/data_process/scripts/run_render_motion.sh@@ -6,11 +6,13 @@# bash data_process/scripts/run_render_motion.sh truebones# bash data_process/scripts/run_render_motion.sh objaverse --multi-worker 8# bash data_process/scripts/run_render_motion.sh objaverse --missing-only+# bash data_process/scripts/run_render_motion.sh general --multi-worker 4# DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_motion.sh mixamo#-# Inputs per dataset: truebones/objaverse render the raw FBX/GLB assets;-# mixamo renders the animated character files produced by-# run_animate_mixamo.sh (raw Mixamo FBXs carry no mesh).+# Inputs per dataset: truebones/objaverse/general render the raw FBX/GLB+# assets (general: dataset/raw/general/animation, with the same file list, asset names+# and actions as its exporter); mixamo renders the animated character files+# produced by run_animate_mixamo.sh (raw Mixamo FBXs carry no mesh).## Runs with plain python + the pip `bpy` module: EEVEE needs the module's# GPU context (`blender -b` has no display surface). --multi-worker N@@ -43,10 +45,11 @@ while [[ $# -gt 0 ]]; docase "$1" in--multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;--missing-only)- # Only render_objaverse.py implements the flag; the other two- # renderers would reject it from argparse after loading bpy.- if [[ "$DATASET" != "objaverse" ]]; then- echo "ERROR: --missing-only is only supported for objaverse (the other renderers resume per asset anyway)" >&2+ # Only render_objaverse.py (and render_general.py, built on it)+ # implements the flag; the other renderers would reject it from+ # argparse after loading bpy.+ if [[ "$DATASET" != "objaverse" && "$DATASET" != "general" ]]; then+ echo "ERROR: --missing-only is only supported for objaverse and general (the other renderers resume per asset anyway)" >&2exit 2fiEXTRA_ARGS+=("$1"); shift ;;@@ -58,6 +61,7 @@ case "$DATASET" intruebones) DATA_DIR=${DATA_DIR:-dataset/raw/truebones/animation} ;;mixamo) DATA_DIR=${DATA_DIR:-outputs/mixamo_characters} ;;objaverse) DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb} ;;+ general) DATA_DIR=${DATA_DIR:-$GENERAL_RAW_DIR} ;;esacOUTPUT_DIR=${OUTPUT_DIR:-$(render_dir "$DATASET")}@@ -78,9 +82,9 @@ if [[ "$NUM_WORKERS" -gt 1 ]]; thenecho "Launching $NUM_WORKERS render workers (per-worker logs: $LOG_DIR)..."pids=()for worker_id in $(seq 0 $((NUM_WORKERS - 1))); do- # Pin each worker to one GPU before the process starts: EEVEE picks- # its device when the bpy module builds its GL context, so the- # restriction has to be in the environment, not set from Python.+ # Round-robin CUDA_VISIBLE_DEVICES per worker. It must be set before+ # the process starts, but EEVEE's EGL context ignores it (see the+ # KNOWN LIMITATION above): every worker still renders on one GPU.gpu=$((worker_id % NUM_GPUS))CUDA_VISIBLE_DEVICES="$gpu" \python -m "$RENDERER" "${COMMON_ARGS[@]}" \diff --git a/data_process/scripts/run_render_tpose.sh b/data_process/scripts/run_render_tpose.shindex bd16345..92cbee8 100755--- a/data_process/scripts/run_render_tpose.sh+++ b/data_process/scripts/run_render_tpose.sh@@ -5,10 +5,11 @@# Usage:# bash data_process/scripts/run_render_tpose.sh truebones# bash data_process/scripts/run_render_tpose.sh objaverse --multi-worker 8+# bash data_process/scripts/run_render_tpose.sh general# DATA_DIR=outputs/mixamo_characters bash data_process/scripts/run_render_tpose.sh mixamo#-# Grids are written flat as <tpose_dir>/<name>.png (one per GLB stem, or-# per Truebones object type). Runs with plain python + the pip `bpy`+# Grids are written flat as <tpose_dir>/<name>.png (one per asset file, named+# like its exported object type, or per Truebones object type). Runs with plain python + the pip `bpy`# module: EEVEE needs the module's GPU context.## Env overrides: DATA_DIR, OUTPUT_DIR, RESOLUTION, SAMPLES, CAMERA_DIST@@ -37,6 +38,7 @@ case "$DATASET" inEXTRA_ARGS+=(--species_grids) ;; # one grid per speciesmixamo) DATA_DIR=${DATA_DIR:-dataset/raw/mixamo/character_refined} ;;objaverse) DATA_DIR=${DATA_DIR:-dataset/raw/objaverse/glb} ;;+ general) DATA_DIR=${DATA_DIR:-$GENERAL_RAW_DIR} ;;esacOUTPUT_DIR=${OUTPUT_DIR:-$(tpose_dir "$DATASET")}diff --git a/data_process/scripts/run_rig_preprocess.sh b/data_process/scripts/run_rig_preprocess.shnew file mode 100755index 0000000..4a98a11--- /dev/null+++ b/data_process/scripts/run_rig_preprocess.sh@@ -0,0 +1,86 @@+#!/bin/bash+# Process one rigged 3D asset (GLB/GLTF/FBX, with or without animation) into+# the model's inputs: <OUTPUT_DIR>/cond.npy (its topology condition),+# <OUTPUT_DIR>/<name>.glb (its canonical asset) and <OUTPUT_DIR>/preview.png+# (facing and joint labels to check), through the same export,+# joint-annotation and feature code as the dataset pipeline+# (data_process/rig_preprocess, see its AGENTS.md). The output directory is+# then an asset for sampling: python -m unimate.inference.sample --asset <OUTPUT_DIR>.+#+# Usage:+# INPUT=my_robot.glb OUTPUT_DIR=outputs/my_robot bash data_process/scripts/run_rig_preprocess.sh+# INPUT=rig.fbx OUTPUT_DIR=out FACE_R=R_Thigh FACE_L=L_Thigh bash data_process/scripts/run_rig_preprocess.sh+# INPUT="dataset/raw/truebones/animation/Dog-*.fbx" PROFILE=truebones ANNOTATE=dataset \+# OUTPUT_DIR=out/Dog bash data_process/scripts/run_rig_preprocess.sh -- --keep_intermediate+# VERIFY=1 ... # then compare the output with the dataset's copy (training assets)+# # by default a run stops before the cond: check out/annotation.json (see out/REVIEW.md), then+# ANNOTATION=out/annotation.json INPUT=rig.glb OUTPUT_DIR=out bash data_process/scripts/run_rig_preprocess.sh+# REVIEW=0 INPUT=rig.glb OUTPUT_DIR=out bash data_process/scripts/run_rig_preprocess.sh # build directly+#+# Env overrides:+# INPUT (required) the asset file; a glob or space-separated list for+# PROFILE=truebones (one species' clip FBXs); no spaces in paths+# OUTPUT_DIR (required)+# PROFILE auto (default: truebones for several {Species}-{Action}.fbx clips;+# with ANNOTATE=dataset, the dataset of --dataset_export_dir, passed after --;+# else general) | general | objaverse | truebones: the dataset whose+# processing to reproduce+# ANNOTATE llm | rule | dataset: joint-name / facing source (default: llm when+# DEEPSEEK_API_KEY / OPENAI_API_KEY is set for the model, else rule;+# rule when ANNOTATION, ANNOTATE_NAMES or ANNOTATE_FACE is given)+# ANNOTATE_NAMES, ANNOTATE_FACE rule | llm: one source differing from ANNOTATE+# (not with ANNOTATE=dataset), e.g. ANNOTATE_FACE=llm for rule names+# with an LLM-chosen face pair+# NAME object type for the asset (default: from its file name)+# FACE_R, FACE_L raw names of the facing pair (overrides the annotation);+# BODY_AXIS=1 when it is a head / tail axis+# EXP_DIR training run whose joint width (config.json) the asset is checked+# against (default: the released UniML3D models' width, 71)+# FORMATS canonical asset files: glb (default) or glb,fbx+# SAVE_CLIPS 1: keep the feature clips in <OUTPUT_DIR>/motions/ (ground truth+# for in-betweening / motion editing)+# REST_ROTATION stand up a rest pose authored lying down / upside down, e.g. x180,+# x90, x-90 (preview.png and summary.json notes show when it is needed)+# REVIEW 1 / 0: stop after the annotation, before the cond, and write+# annotation.json, annotation_preview.png and REVIEW.md to check the+# joint labels and the facing pair (by hand, or with an LLM / agent) /+# build directly. Default: review, except with ANNOTATION or ANNOTATE=dataset+# ANNOTATION a reviewed annotation.json: its labels and facing pair replace the+# annotation, and the run continues to the cond+# VERIFY 1: run `verify` afterwards when the asset was built (exit 1 on a difference)+# CONDA_ENV conda environment to activate (default: unimate)+# Arguments after `--` go to `python -m data_process.rig_preprocess run`+# (--keep_intermediate, --overwrite, LLM options, ...).++set -euo pipefail+source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"++handle_help "$@"+[[ "${1:-}" == "--" ]] && shift++INPUT=${INPUT:?Set INPUT to the asset file}+OUTPUT_DIR=${OUTPUT_DIR:?Set OUTPUT_DIR}+read -r -a INPUTS <<< "$INPUT"+# shellcheck disable=SC2206+INPUTS=(${INPUTS[@]}) # expand globs++ARGS=(--input "${INPUTS[@]}" --output_dir "$OUTPUT_DIR"+ --profile "${PROFILE:-auto}" ${ANNOTATE:+--annotate "$ANNOTATE"})+[[ -n "${FORMATS:-}" ]] && ARGS+=(--formats "$FORMATS")+[[ "${SAVE_CLIPS:-0}" == 1 ]] && ARGS+=(--save_clips)+[[ -n "${REST_ROTATION:-}" ]] && ARGS+=(--rest_rotation "$REST_ROTATION")+[[ -n "${ANNOTATE_NAMES:-}" ]] && ARGS+=(--annotate_names "$ANNOTATE_NAMES")+[[ -n "${ANNOTATE_FACE:-}" ]] && ARGS+=(--annotate_face "$ANNOTATE_FACE")+[[ -n "${NAME:-}" ]] && ARGS+=(--name "$NAME")+[[ -n "${EXP_DIR:-}" ]] && ARGS+=(--exp_dir "$EXP_DIR")+[[ -n "${FACE_R:-}" ]] && ARGS+=(--face_r "$FACE_R")+[[ -n "${FACE_L:-}" ]] && ARGS+=(--face_l "$FACE_L")+[[ "${BODY_AXIS:-0}" == 1 ]] && ARGS+=(--body_axis)+[[ "${REVIEW:-}" == 1 ]] && ARGS+=(--review)+[[ "${REVIEW:-}" == 0 ]] && ARGS+=(--no_review)+[[ -n "${ANNOTATION:-}" ]] && ARGS+=(--annotation "$ANNOTATION")++python -m data_process.rig_preprocess run "${ARGS[@]}" "$@"+if [[ "${VERIFY:-0}" == 1 && -f "$OUTPUT_DIR/cond.npy" ]]; then+ python -m data_process.rig_preprocess verify --output_dir "$OUTPUT_DIR"+fidiff --git a/data_process/tools/merge_summaries.py b/data_process/tools/merge_summaries.pyindex 1c65ce5..6486ca5 100644--- a/data_process/tools/merge_summaries.py+++ b/data_process/tools/merge_summaries.py@@ -85,7 +85,7 @@ def merge_summaries(output_dir, fps=30, keep_shards=False):af_path = os.path.join(output_dir, "clip_frames.json")frames = {}if os.path.isfile(af_path):- # Ignore a "_summary" block embedded by older exporter versions.+ # A "_summary" block in the file is not a clip.frames = {k: v for k, v in _load(af_path).items() if k != "_summary"}af_shards = sorted(glob.glob(os.path.join(output_dir, "clip_frames_worker*.json")))for shard in af_shards:diff --git a/data_process/tools/patch_annotations.py b/data_process/tools/patch_annotations.pyindex 3078cfb..5e22fb6 100644--- a/data_process/tools/patch_annotations.py+++ b/data_process/tools/patch_annotations.py@@ -22,7 +22,7 @@ place. Two kinds of fixes:orientation and where one sided label sits on both sides of the body(a duplicated chain) โ there only that chain is relabelled;- the ``clean`` fields of face_joint_names.json are re-synced;- - objaverse rigs are flagged in ``rig_flags.json``: automatically when the+ - objaverse (and general) rigs are flagged in ``rig_flags.json``: automatically when thefacing pair cannot be trusted (``source: empty``, a Bone/Bone pair, a bodyaxis through unnamed bones), plus the hand-reviewed categories. Only the``FILTER_CATEGORIES`` rigs (``tpose_wrong``, ``not_in_legacy_raw``) are also@@ -36,11 +36,20 @@ place. Two kinds of fixes:facing axis (|cos| > 0.8 averaged over the clip), otherwise the qualifieris dropped (thresholds in body heights, see ``--still`` / ``--travel``).- manual overrides (JSON files in ``--patch_dir``)+ manual overrides (JSON files in ``--patch_dir``; a step whose file is absent+ leaves its earlier output, e.g. a downloaded export's, as it is)<ds>_joint_labels.json {rig: {raw_joint_name: clean_label}}<ds>_face_pairs.json {rig: {"r_hip": raw, "l_hip": raw, "source": s,["body_axis": true]}}<ds>_captions.json {clip: caption} (applied before the rules)+ <ds>_captions_llm.json {clip: {"old", "new", "ok", "tries"}} (vlm_caption/caption_rewrite_llm.py+ output: an entry with ok true replaces the caption while it still equals+ "old"; <ds>_captions.json wins for the same clip)+ <ds>_captions_generic.json {clip: caption} (the second, generic caption of every clip; the+ patch is its only source. Same rules as above ->+ export/<ds>/motion_captions_generic.json)+ <ds>_captions_detail.json {clip: caption} (the third, detail caption, 7-19 words; the same+ way -> export/<ds>/motion_captions_detail.json)<ds>_categories.json {rig: category} (moves the rig in category_groups.json)<ds>_filtered_clips.txt <clip> # <reason> (any dataset: individual clips ->export/<ds>/filtered_clips.txt, skipped by stage 4)@@ -53,9 +62,19 @@ place. Two kinds of fixes:the animal is tilted; export/<ds>/motions/<clip>.npz is REWRITTEN withanim_local_rot[:, 0] = anim_local_rot[:, 0] * quat on every frame. Idempotent,and undone when the entry is removed; see apply_root_offsets)- <ds>_rig_flags.txt <rig> # <category>: <reason> (objaverse: hand-reviewed rig flags ->- export/objaverse/rig_flags.json; only tpose_wrong / not_in_legacy_raw- also go to export/objaverse/filtered_objects.txt, which stage 4 skips)+ <ds>_rest_orientation.json {rig: {"quat": [w, x, y, z], "offset": [x, y, z], "reason": s}} (any+ dataset: the rig's rest pose lies, leans or is upside down; every+ export/<ds>/motions/<rig>-*.npz is REWRITTEN with its rest pose moved by+ p -> quat * p + offset in export space (offset optional); the+ processed GLB applies the same rotation. Idempotent, undone when the entry+ is removed; see apply_rest_orientations)+ <ds>_rig_flags.txt <rig> # <category>: <reason> (objaverse / general: hand-reviewed rig+ flags -> export/<ds>/rig_flags.json; only tpose_wrong / not_in_legacy_raw+ also go to export/<ds>/filtered_objects.txt, which stage 4 skips)++ ``general`` (extra assets of your own in dataset/raw/general/animation, objaverse layout) gets the+ objaverse treatment: automatic side-from-x on duplicated chains and rig flags. Its patch+ files, if you write any, are named general_*.Usage (from the repo root; ``dataset/export/<ds>`` may be symlinks):python data_process/tools/patch_annotations.py [--dry_run] [--datasets truebones mixamo]@@ -82,10 +101,13 @@ import numpy as npsys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))from data_process.utils.kinematics import ( # noqa: E402- fk, frame_global, qmul, qrot, rest_global,+ frame_global, qmul, qrot, rest_global,)-DATASETS = ('truebones', 'mixamo', 'objaverse')+DATASETS = ('truebones', 'mixamo', 'objaverse', 'general')+# Multi-object layouts (one rigged asset per object type, arbitrary rigs): automatic+# side-from-x on duplicated chains, rig_flags.json and filtered_objects.txt.+OBJECT_DATASETS = ('objaverse', 'general')# One subject for the whole corpus. The three datasets are trained as one# mixture, so a per-dataset subject ('An animal' / 'A person') would hand the# text encoder a free dataset label and defeat the topology-agnostic@@ -318,7 +340,7 @@ def patch_joint_labels(ds, root, clean, names, overrides, stats, log):log.append(f'[{ds}] {rig}: no NPZ, side-from-x skipped')continueside_from_x(clean[rig], np.load(f, allow_pickle=True), stats)- if ds == 'objaverse': # auto: duplicated-side chains on rigs that face +Z per their names+ if ds in OBJECT_DATASETS: # auto: duplicated-side chains on rigs that face +Z per their namesfor rig in clean:if rig in explicit:continue@@ -386,6 +408,17 @@ def patch_face_pairs(ds, face, clean, names, overrides, stats, log):if entry[key].get('clean') != new:entry[key]['clean'] = newstats['face-clean-resync'] += 1+ # re-sync source: it names what the pair is, so it follows the labels+ # ('manual' overrides and relabelled Bone pairs otherwise keep a stale tag)+ if entry.get('body_axis'):+ src = 'body_axis'+ else:+ parts = {re.sub(r'^(Left|Right)\s+|\s+End$', '', entry[k]['clean']).lower()+ for k in ('r_hip', 'l_hip')}+ src = parts.pop() if len(parts) == 1 else entry.get('source')+ if src and entry.get('source') != src:+ entry['source'] = src+ stats['face-source-resync'] += 1# Rig flag categories. Only FILTER_CATEGORIES end up in filtered_objects.txt (stage 4 skips@@ -402,7 +435,7 @@ FILTER_CATEGORY_NOTES = {}-def load_rig_flags(path, names, log):+def load_rig_flags(ds, path, names, log):"""``patches/<ds>_rig_flags.txt``: one ``<rig> # <category>: <reason>`` per line โ hand-reviewedrigs (``tpose_wrong`` = rest pose lying / rotated, ``not_in_legacy_raw`` = raw GLB absent from the legacyraw set and held out, ``facing_wrong`` = the facing pair does not give the real front, ``object_no_front``@@ -423,13 +456,13 @@ def load_rig_flags(path, names, log):if not rig:continueif rig not in names:- log.append(f'[objaverse] rig flag for unknown rig {rig!r} ignored')+ log.append(f'[{ds}] rig flag for unknown rig {rig!r} ignored')continuecat, _, reason = comment.strip().partition(':')cat = cat.strip() or 'flagged'if rig in out:prev = out[rig][0]- log.append(f'[objaverse] rig {rig!r} is listed twice in {os.path.basename(path)}: '+ log.append(f'[{ds}] rig {rig!r} is listed twice in {os.path.basename(path)}: 'f'{prev!r} then {cat!r}; ' +(f'{cat!r} wins and the {prev!r} entry is dropped'if prev != cat else 'the duplicate line is redundant'))@@ -441,7 +474,7 @@ def write_filtered_clips(ds, root, patch_dir, frames, dry_run, log):"""export/<ds>/filtered_clips.txt โ hand-reviewed individual clips that stage 4 skips.Every dataset may have one: stage 4 applies the clip list before it groups- clips into object types, so it is not objaverse-only the way+ clips into object types, so it is not limited to objaverse / general the wayfiltered_objects.txt is. patches/<ds>_filtered_clips.txt holds``<clip stem> # <reason>`` lines (motion discontinuity, skeletons thatdisagree with the object's reference rig, and anything else reviewed by hand).@@ -598,7 +631,10 @@ def apply_root_offsets(ds, root, patch_dir, dry_run, log):and the next run of this script fixes them again. Returns the clips whose NPZchanged (their cached root motion is stale)."""- want = load_overrides(os.path.join(patch_dir, f'{ds}_root_offsets.json'))+ src = os.path.join(patch_dir, f'{ds}_root_offsets.json')+ if not os.path.isfile(src):+ return [] # no patch file: leave earlier fixes in place+ want = load_overrides(src)mdir = os.path.join(root, 'motions')changed, n_ok = [], 0targets = set(want)@@ -644,7 +680,100 @@ def apply_root_offsets(ds, root, patch_dir, dry_run, log):return changed-def write_rig_flags(root, face, frames, manual, dry_run, log):+REST_ORIENTATION_KEY = 'rest_orientation_applied' # the quat currently baked into the rest pose+++def apply_rest_orientations(ds, root, patch_dir, dry_run, log):+ """Stand a rig's rest pose up, in place, in every export NPZ of the rig.++ A rest pose can lie flat, lean or stand upside down while the clips play+ upright: the bind pose was authored in another orientation. Since the rest is+ one rigid pose, a single rotation fixes it. patches/<ds>_rest_orientation.json+ maps ``rig -> {"quat": [w, x, y, z], "offset": [x, y, z], "reason": ...}``, a+ rotation and an optional translation in export space (the offset puts the+ turned pose back on the ground); ``p -> quat * p + offset`` is applied to the+ root's rest transform (``rest_local_rot[0]``, ``rest_local_pos[0]``,+ ``offsets[0]``), moving the whole rest pose rigidly. Clip frames are untouched,+ so every clip keeps its motion and only the reference pose changes.+ The export GLB applies the same rotation to its armature+ (``processed_assets.export_asset_glb``), so the two stay aligned.++ Idempotent like apply_root_offsets: the applied quat is stored in each NPZ under+ ``rest_orientation_applied`` (quat then offset, 7 values); a changed entry undoes+ the old transform first, and an+ NPZ that carries the key but whose rig is no longer listed is restored. Returns+ the rigs whose NPZs changed (their processed GLBs must be rebuilt).+ """+ src = os.path.join(patch_dir, f'{ds}_rest_orientation.json')+ if not os.path.isfile(src):+ return [] # no patch file: leave earlier fixes in place+ want = load_overrides(src)+ mdir = os.path.join(root, 'motions')+ by_rig = {}+ for f in glob.glob(os.path.join(mdir, '*.npz')):+ # Mixamo clips share one rig and carry no object-type prefix.+ rig = 'mixamo' if ds == 'mixamo' else os.path.basename(f).split('-')[0]+ by_rig.setdefault(rig, []).append(f)+ targets = set(want)+ for rig, paths in by_rig.items(): # previously fixed rigs no longer listed+ if rig not in targets:+ with np.load(paths[0], allow_pickle=True) as d:+ if REST_ORIENTATION_KEY in d.files:+ targets.add(rig)+ changed, n_ok = [], 0+ for rig in sorted(targets):+ paths = sorted(by_rig.get(rig, []))+ if not paths:+ log.append(f'[{ds}] rest orientation for {rig!r}: no export NPZ; ignored')+ continue+ new = want.get(rig)+ fix_new = None+ if new is not None:+ q = np.asarray(new['quat'], dtype=np.float64)+ fix_new = np.concatenate([q / np.linalg.norm(q),+ np.asarray(new.get('offset', [0, 0, 0]), dtype=np.float64)])+ rig_changed = False+ for path in paths:+ with np.load(path, allow_pickle=True) as d:+ data = {k: d[k] for k in d.files}+ have = data.pop(REST_ORIENTATION_KEY, None)+ if have is not None and fix_new is not None and have.shape == fix_new.shape \+ and np.allclose(have, fix_new, rtol=1e-6, atol=1e-6):+ n_ok += 1+ continue+ rot = data['rest_local_rot'].astype(np.float64)+ q0, p0 = rot[0], data['rest_local_pos'][0].astype(np.float64)+ if have is not None: # undo what is baked in+ have = np.asarray(have, dtype=np.float64)+ inv = have[:4] * np.array([1, -1, -1, -1])+ t = have[4:7] if len(have) >= 7 else np.zeros(3)+ q0, p0 = qmul(inv, q0), qrot(inv, p0 - t)+ if fix_new is not None:+ q0, p0 = qmul(fix_new[:4], q0), qrot(fix_new[:4], p0) + fix_new[4:]+ data[REST_ORIENTATION_KEY] = fix_new+ rot[0] = q0 / np.linalg.norm(q0)+ data['rest_local_rot'] = rot.astype(data['rest_local_rot'].dtype)+ for key in ('rest_local_pos', 'offsets'):+ if key in data:+ arr = data[key].copy()+ arr[0] = p0+ data[key] = arr+ if not dry_run:+ _save_npz_atomic(path, data)+ rig_changed = True+ if rig_changed:+ what = 'restored (entry removed)' if fix_new is None else \+ f'{np.degrees(2 * np.arccos(min(1.0, abs(fix_new[0])))):.1f} deg'+ log.append(f'[{ds}] rest orientation {rig}: {what}, {len(paths)} clip(s)'+ + (' (dry run)' if dry_run else ''))+ changed.append(rig)+ if want or changed:+ log.append(f'[{ds}] rest_orientation: {len(want)} listed, {len(changed)} rig(s) rewritten, '+ f'{n_ok} clip(s) already applied')+ return changed+++def write_rig_flags(ds, root, face, frames, manual, dry_run, log):"""rig_flags.json (every flagged rig with its category) + filtered_objects.txt (FILTER_CATEGORIES only)."""flags = OrderedDict()for rig, entry in face.items():@@ -662,7 +791,7 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):for rig, fl in sorted(flags.items()):cats.setdefault(fl['category'], []).append(rig)bad = OrderedDict((rig, fl) for rig, fl in flags.items() if fl['category'] in FILTER_CATEGORIES)- clips = [c for c in frames if object_type_of('objaverse', c) in bad]+ clips = [c for c in frames if object_type_of(ds, c) in bad]out = OrderedDict([('_comment', 'Per-rig flags. Automatic facing checks (empty_pair: no bilateral joint pair; ''bone_pair: the pair consists of unnamed Bone joints; body_axis_unnamed) plus the '@@ -680,7 +809,7 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):('verified_ok', verified), # automatic flags cleared by hand review])lines = ['# Object types skipped by stage 4 (extract_features.py reads this file automatically):',- '# rigs flagged ' + ' / '.join(FILTER_CATEGORIES) + ' in patches/objaverse_rig_flags.txt.']+ '# rigs flagged ' + ' / '.join(FILTER_CATEGORIES) + f' in patches/{ds}_rig_flags.txt.']lines += [f'# {c}: {FILTER_CATEGORY_NOTES[c]}' for c in FILTER_CATEGORIES if c in FILTER_CATEGORY_NOTES]lines += ['# Other flags (empty_pair, bone_pair, facing_wrong, object_no_front, ...) are recorded in','# rig_flags.json only and are NOT filtered.',@@ -693,7 +822,7 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):json.dump(out, f, indent=1)with open(os.path.join(root, 'filtered_objects.txt'), 'w') as f:f.write('\n'.join(lines) + '\n')- log.append(f'[objaverse] rig_flags.json: {dict(out["counts"])} (+{len(verified)} verified_ok); filtered_objects.txt: '+ log.append(f'[{ds}] rig_flags.json: {dict(out["counts"])} (+{len(verified)} verified_ok); filtered_objects.txt: 'f'{len(bad)} rigs / {len(clips)} clips ({", ".join(FILTER_CATEGORIES)})')return bad@@ -728,7 +857,7 @@ def root_motion(root, ds, clips, cache_path, face, log):"""Per clip: net XZ root displacement in body heights and the traveldirection relative to the facing pair at frame 0."""cache = load_json(cache_path, OrderedDict()) if cache_path else OrderedDict()- key = lambda c: f'{ds}/{c}@{CAPTION_FRAMES}' # window-scoped: pre-2026-09-21 full-clip rows are ignored+ key = lambda c: f'{ds}/{c}@{CAPTION_FRAMES}' # window-scoped: rows of other windows are ignoredtodo = [c for c in clips if key(c) not in cache]if todo:log.append(f'[{ds}] computing root motion for {len(todo)} clips')@@ -805,6 +934,28 @@ def normalize_locomotion(caption, motion, still, travel, stats):return LOCO_RE.sub(repl, caption)+def load_caption_rewrites(ds, path, caps, log):+ """``{clip: new caption}`` from a ``caption_rewrite_llm.py`` patch: the+ entries that passed its validation (``ok``) and whose ``old`` caption is+ still the current one (a rewrite of a caption that changed since is stale)."""+ out, stale = {}, 0+ for clip, entry in load_overrides(path).items():+ if (not isinstance(entry, dict) or not entry.get('ok') or not entry.get('new')+ or not entry.get('old') or clip not in caps):+ continue+ current = caps[clip].strip().rstrip('.')+ if current == entry['new'].strip().rstrip('.'):+ continue # applied by an earlier run+ if current != entry['old'].strip().rstrip('.'):+ stale += 1+ continue+ out[clip] = entry['new']+ if stale:+ log.append(f'[{ds}] {stale} caption rewrites in {os.path.basename(path)} skipped: '+ f'the caption changed since')+ return out++def patch_captions(ds, caps, overrides, motion, args, stats, log):subject = SUBJECT[ds]for clip, text in (overrides or {}).items():@@ -833,10 +984,45 @@ def patch_captions(ds, caps, overrides, motion, args, stats, log):stats['caption-changed'] += 1+# Extra caption versions beside motion_captions.json (HumanML3D-style multiple+# captions): the short generic one and the longer detail one.+EXTRA_CAPTION_VERSIONS = ('generic', 'detail')+++def build_extra_captions(ds, version, caps, patch_dir, motion, args, stats, log):+ """One extra caption version per clip (``generic`` or ``detail``).++ The source is the reviewed patch ``<ds>_captions_<version>.json`` itself:+ no pipeline stage generates these captions. Only clips present in the+ export captions are kept, in their key order; the same subject / grammar /+ locomotion rules as the normal captions run on the result. Returns None+ when there is no patch; ``run_dataset`` then keeps an existing+ ``motion_captions_<version>.json``.+ """+ src = load_overrides(os.path.join(patch_dir, f'{ds}_captions_{version}.json'))+ if not src:+ return None+ unknown = [c for c in src if c not in caps]+ if unknown:+ log.append(f'[{ds}] {len(unknown)} {version} captions for unknown clips ignored, e.g. {unknown[:3]}')+ out = {c: src[c] for c in caps if c in src}+ vstats = Counter()+ patch_captions(ds, out, None, motion, args, vstats, log)+ for k, v in vstats.items():+ stats[f'{version}-{k}'] += v+ stats[f'{version}-captions'] += len(out)+ missing = len(caps) - len(out)+ if missing:+ stats[f'{version}-missing'] += missing+ return out++# ----------------------------------------------------------------------------def write_report(ds, report_dir, before, clean, face, caps, names, motion):- """TSV of every change: kind, key, joint/-, old, new, note."""+def write_report(ds, report_dir, before, clean, face, caps, names, motion, extra=None):+ """TSV of every change: kind, key, joint/-, old, new, note. Extra caption+ versions (``extra``: {version: {clip: caption} or None}) are compared with+ their export files as they were before this run (kind ``caption_<v>``)."""os.makedirs(report_dir, exist_ok=True)rows = []for rig, labels in clean.items():@@ -855,6 +1041,14 @@ def write_report(ds, report_dir, before, clean, face, caps, names, motion):m = motion.get(clip, {})note = f"net={m.get('net')} dir={m.get('dir')}" if m else ''rows.append(('caption', clip, '-', o, text, note))+ for version, new in (extra or {}).items():+ if new is None: # no patch: the file is kept, nothing changes+ continue+ old = before.get('extra', {}).get(version) or {}+ new = new or {}+ for clip in list(new) + [c for c in old if c not in new]:+ if old.get(clip) != new.get(clip):+ rows.append((f'caption_{version}', clip, '-', old.get(clip), new.get(clip), ''))path = os.path.join(report_dir, f'{ds}_changes.tsv')with open(path, 'w') as f:f.write('kind\tkey\tjoint\told\tnew\tnote\n')@@ -877,15 +1071,21 @@ def run_dataset(ds, args, log):pd = args.patch_dirstats = Counter()before = {'clean': json.loads(json.dumps(clean)), 'face': json.loads(json.dumps(face)) if face else {},- 'caps': dict(caps) if caps else {}}+ 'caps': dict(caps) if caps else {},+ 'extra': {v: load_json(os.path.join(root, f'motion_captions_{v}.json'))+ for v in EXTRA_CAPTION_VERSIONS}}patch_joint_labels(ds, root, clean, names, load_overrides(os.path.join(pd, f'{ds}_joint_labels.json')), stats, log)if face is not None:patch_face_pairs(ds, face, clean, names, load_overrides(os.path.join(pd, f'{ds}_face_pairs.json')), stats, log)- if ds == 'objaverse':- manual = load_rig_flags(os.path.join(pd, f'{ds}_rig_flags.txt'), names, log)- write_rig_flags(root, face, frames, manual, args.dry_run, log)+ flags_src = os.path.join(pd, f'{ds}_rig_flags.txt')+ if ds in OBJECT_DATASETS and os.path.isfile(flags_src):+ manual = load_rig_flags(ds, flags_src, names, log)+ write_rig_flags(ds, root, face, frames, manual, args.dry_run, log)stats['manual-rig-flag'] += len(manual)+ elif ds in OBJECT_DATASETS:+ log.append(f'[{ds}] no {ds}_rig_flags.txt in {pd}; rig_flags.json / '+ f'filtered_objects.txt left as they are')stats['manual-filtered-clip'] += len(write_filtered_clips(ds, root, pd, frames, args.dry_run, log))stats['manual-clip-trim'] += len(@@ -894,6 +1094,8 @@ def run_dataset(ds, args, log):write_activity_keep(ds, root, pd, frames, args.dry_run, log))rewritten = apply_root_offsets(ds, root, pd, args.dry_run, log)stats['root-offset-rewritten'] += len(rewritten)+ stats['rest-orientation-rewritten'] += len(+ apply_rest_orientations(ds, root, pd, args.dry_run, log))if rewritten and not args.dry_run:# the cached travel direction was measured against the old (tilted) facingcache_path = os.path.join(pd, '.root_motion_cache.json')@@ -905,21 +1107,35 @@ def run_dataset(ds, args, log):save_json(cache_path, cache, False)if groups is not None:patch_categories(ds, groups, load_overrides(os.path.join(pd, f'{ds}_categories.json')), names, stats, log)+ extra = {}if caps is not None:motion = root_motion(root, ds, list(caps), os.path.join(pd, '.root_motion_cache.json'),face or {}, log) if not args.skip_locomotion else {}- patch_captions(ds, caps, load_overrides(os.path.join(pd, f'{ds}_captions.json')), motion, args, stats, log)+ overrides = load_caption_rewrites(ds, os.path.join(pd, f'{ds}_captions_llm.json'), caps, log)+ overrides.update(load_overrides(os.path.join(pd, f'{ds}_captions.json')))+ patch_captions(ds, caps, overrides, motion, args, stats, log)+ extra = {v: build_extra_captions(ds, v, caps, pd, motion, args, stats, log)+ for v in EXTRA_CAPTION_VERSIONS}save_json(os.path.join(root, 'clean_joint_names.json'), clean, args.dry_run)if face is not None:save_json(os.path.join(root, 'face_joint_names.json'), face, args.dry_run)if caps is not None:save_json(os.path.join(root, 'motion_captions.json'), caps, args.dry_run)+ for version in EXTRA_CAPTION_VERSIONS:+ path = os.path.join(root, f'motion_captions_{version}.json')+ if extra.get(version) is not None:+ save_json(path, extra[version], args.dry_run)+ elif caps is not None and os.path.isfile(path):+ # No patch file: keep the captions an earlier run (or the released+ # export) wrote; deleting them is a manual decision.+ log.append(f'[{ds}] no {ds}_captions_{version}.json in {pd}; kept {path}')if groups is not None:save_json(os.path.join(root, 'category_groups.json'), groups, args.dry_run)if args.report_dir:write_report(ds, args.report_dir, before, clean, face or {}, caps or {}, names,- motion if caps is not None and not args.skip_locomotion else {})+ motion if caps is not None and not args.skip_locomotion else {},+ extra=extra)log.append(f'[{ds}] ' + ', '.join(f'{k}={v}' for k, v in sorted(stats.items())))@@ -937,7 +1153,8 @@ def main():ap.add_argument('--skip_locomotion', action='store_true', help='do not touch forward/in place')ap.add_argument('--report_dir', default=None,help='write <ds>_changes.tsv (every label / face / caption change) here')- ap.add_argument('--dry_run', action='store_true', help='report only, write nothing')+ ap.add_argument('--dry_run', action='store_true', help='report only; writes nothing but the root-motion cache '+ '(<patch_dir>/.root_motion_cache.json)')args = ap.parse_args()os.makedirs(args.patch_dir, exist_ok=True)log = []@@ -945,8 +1162,9 @@ def main():run_dataset(ds, args, log)print('\n'.join(log))if args.dry_run:- print('(dry run โ nothing written)')+ print('(dry run โ no annotation written; only the root-motion cache)')+ return 0if __name__ == '__main__':- main()+ sys.exit(main())diff --git a/data_process/tools/sync_captions.py b/data_process/tools/sync_captions.pynew file mode 100644index 0000000..e378e20--- /dev/null+++ b/data_process/tools/sync_captions.py@@ -0,0 +1,95 @@+#!/usr/bin/env python3+"""Refresh the caption files of existing stage-4 feature directories.++Stage 4 writes ``captions.json`` and ``captions_<v>.json`` (v = generic,+detail) on every run,+but captions can change without anything else changing (a patch applied by+``tools/patch_annotations.py``), and a stage-4 rerun is expensive. This copies+the export captions into ``features/<ds>/``, keyed like that directory's+``captions.json`` (``<clip>-<NNN>``) with the same helper stage 4 uses:++ * ``captions.json`` values refreshed from ``motion_captions.json``;+ the key set is never changed (which clips exist+ is stage 4's decision, not this tool's)+ * ``captions_generic.json`` rebuilt from ``motion_captions_generic.json``+ * ``captions_detail.json`` rebuilt from ``motion_captions_detail.json``++After it changes anything, refresh the text-embedding cache+(``python -m unimate.tools.precompute_text_emb``); stale cache entries are+detected by their text and re-encoded. ``cond.npy`` keeps the old captions per+object, which nothing reads; stage 4's resume cache sees the caption digest+change and re-processes those objects on its next run.++Usage (from the repo root):+ python data_process/tools/sync_captions.py [--datasets truebones mixamo] [--dry_run]+"""++import argparse+import os+import sys++sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))++from data_process.feature_extraction.metadata import ( # noqa: E402+ EXTRA_CAPTION_VERSIONS, by_feature_key, load_extra_captions, load_json,+ load_motion_captions, save_json,+)++DATASETS = ('truebones', 'mixamo', 'objaverse', 'general')+++def main():+ here = os.path.dirname(os.path.abspath(__file__))+ repo = os.path.abspath(os.path.join(here, '..', '..'))+ ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])+ ap.add_argument('--export_root', default=os.path.join(repo, 'dataset', 'export'))+ ap.add_argument('--features_root', default=os.path.join(repo, 'dataset', 'features'))+ ap.add_argument('--datasets', nargs='+', default=list(DATASETS))+ ap.add_argument('--dry_run', action='store_true', help='report only, write nothing')+ args = ap.parse_args()++ for ds in args.datasets:+ export_dir = os.path.join(args.export_root, ds)+ feat_dir = os.path.join(args.features_root, ds)+ captions_path = os.path.join(feat_dir, 'captions.json')+ captions = load_json(captions_path)+ if captions is None:+ print(f'[{ds}] skipped: no {captions_path}')+ continue++ normal = load_motion_captions(export_dir) or {}+ fresh = by_feature_key(captions, normal)+ changed = {k: (captions[k], fresh[k]) for k in fresh if fresh[k] != captions[k]}+ orphans = sorted(set(captions) - set(fresh))+ print(f'[{ds}] captions.json: {len(changed)} of {len(captions)} captions change'+ + (f'; {len(orphans)} feature clips have no export caption (kept), e.g. {orphans[:3]}'+ if orphans else ''))+ for k, (old, new) in sorted(changed.items())[:5]:+ print(f' {k}: {old!r} -> {new!r}')+ if changed and not args.dry_run:+ save_json(captions_path, {k: fresh.get(k, v) for k, v in captions.items()})++ # Same rule as stage 4 (metadata.save_outputs): captions_<v>.json exists+ # exactly when some saved clip has a <v> caption; otherwise a stale one goes.+ for version in EXTRA_CAPTION_VERSIONS:+ name = f'captions_{version}.json'+ path = os.path.join(feat_dir, name)+ out = by_feature_key(captions, load_extra_captions(export_dir, version) or {})+ if out:+ old = load_json(path) or {}+ n_changed = sum(old.get(k) != v for k, v in out.items()) + len(set(old) - set(out))+ missing = sorted(set(captions) - set(out))+ print(f'[{ds}] {name}: {len(out)}/{len(captions)} feature clips, '+ f'{n_changed} entries change'+ + (f'; {len(missing)} without, e.g. {missing[:3]}' if missing else ''))+ if n_changed and not args.dry_run:+ save_json(path, out)+ else:+ print(f'[{ds}] no {version} captions for any feature clip in {export_dir}'+ + (f'; removing stale {name}' if os.path.isfile(path) else ''))+ if os.path.isfile(path) and not args.dry_run:+ os.remove(path)+++if __name__ == '__main__':+ main()diff --git a/data_process/tools/vis_caption.py b/data_process/tools/vis_caption.pyindex 689fe43..f74cf0f 100644--- a/data_process/tools/vis_caption.py+++ b/data_process/tools/vis_caption.py@@ -60,7 +60,7 @@ from data_process.utils.motion_features import ( # noqa: E402PREVIEW_DPI, PREVIEW_FIGSIZE)from data_process.utils.plotting import render_skeleton_motion_ground # noqa: E402-DATASETS = ('truebones', 'mixamo', 'objaverse')+DATASETS = ('truebones', 'mixamo', 'objaverse', 'general')MAX_RENDER_FRAMES = 200 # keep in sync with blender_render.MAX_RENDER_FRAMESDEFAULT_MAX_FRAMES = 1000 # render cost is linear in frames; 1000 cuts almost nothingdiff --git a/data_process/utils/asset_files.py b/data_process/utils/asset_files.pynew file mode 100644index 0000000..dbbf53b--- /dev/null+++ b/data_process/utils/asset_files.py@@ -0,0 +1,290 @@+"""Files of the processed assets: paths, notes, GLB JSON (bpy-free).++Layout:++ <export>/rigs/<asset>.glb rest-pose asset on the export's pruned+ skeleton (export stage, --save_glb /+ --glb_only; built by processed_assets.py)+ <export>/rigs/texture_issues/<asset>.txt exactly while that GLB has a texture problem+ <export>/rigs/glb_errors/<asset>.txt exactly while building it last failed+ <export>/replaced/{motions,videos}/ an asset's clips from before it was+ exported again (set_aside_old_clips)+ <root>/canonical_assets/<ds>/<name>.glb the same in the stage-4 canonical frame+ (feature_extraction/canonical_assets.py),+ with its own glb_errors/ and texture_issues/++One note file per asset, so parallel workers never share one.+"""++import hashlib+import json+import os+import struct++import numpy as np+from loguru import logger++# Sibling of motions/ in an export directory: rigs/<asset>.glb, the asset+# that motions/<asset>-*.npz animate.+PROCESSED_GLB_DIR = 'rigs'+TEXTURE_ISSUES_DIR = 'texture_issues'+GLB_ERRORS_DIR = 'glb_errors'+# Sibling of a features root (<root>/features/<ds>): <root>/canonical_assets/<ds>.+CANONICAL_ASSETS_DIR = 'canonical_assets'+# Mixamo character whose canonical GLB drives the shared Mixamo rig by default.+DEFAULT_MIXAMO_CHARACTER = 'Michelle'+# Sibling of motions/ in an export directory: clips of an earlier export of an+# asset that is exported again (replaced/motions/, replaced/videos/).+REPLACED_DIR = 'replaced'+# The canonical rest-pose GLB of a cond entry carries its joint order under+# this glTF extras key (feature_extraction/canonical_assets.py).+CANONICAL_ORDER_KEY = 'canonical_joint_order'+++def default_assets_dir(features_dir):+ """Where a features directory's canonical assets go by default: beside the+ features root, never inside it. ``<root>/features<suffix>/<ds>`` (e.g.+ ``dataset/features/truebones``, ``dataset/features_v2/truebones``) gives+ ``<root>/canonical_assets<suffix>/<ds>``; any other directory ``<dir>``+ (e.g. a temporary stage-4 ``SAVE_DIR``) gives ``<dir>_canonical_assets``.+ The path is made absolute without resolving symlinks."""+ path = os.path.abspath(features_dir.rstrip(os.sep))+ parent = os.path.dirname(path)+ name = os.path.basename(parent)+ if name.startswith('features'):+ return os.path.join(os.path.dirname(parent),+ CANONICAL_ASSETS_DIR + name[len('features'):],+ os.path.basename(path))+ return path + '_' + CANONICAL_ASSETS_DIR+++# ---------------------------------------------------------------------------+# Export-stage assets+# ---------------------------------------------------------------------------++def processed_glb_path(output_dir, asset):+ """``<output_dir>/rigs/<asset>.glb``."""+ return os.path.join(output_dir, PROCESSED_GLB_DIR, f'{asset}.glb')+++def asset_clip_npzs(output_dir, asset):+ """Sorted ``motions/<asset>-*.npz`` of one exported asset."""+ motion_dir = os.path.join(output_dir, 'motions')+ if not os.path.isdir(motion_dir):+ return []+ prefix = f'{asset}-'+ return sorted(os.path.join(motion_dir, f) for f in os.listdir(motion_dir)+ if f.startswith(prefix) and f.endswith('.npz'))+++def asset_glb_missing(output_dir, asset):+ """True when the asset has clip NPZs but no processed GLB yet."""+ return (not os.path.isfile(processed_glb_path(output_dir, asset))+ and bool(asset_clip_npzs(output_dir, asset)))+++def main_skeleton_clips(output_dir, asset, want=None):+ """``(main, others)``: the asset's clip NPZs on its main skeleton, and+ those on any other one.++ An objaverse / general asset has one skeleton (pruning is shared by its+ clips); a Truebones species can ship rig variants, pruned separately. The+ main skeleton is *want* (joint names) when given, else the asset's entry+ in ``joint_names.json`` (the export's record: the dominant variant), else+ the one most clips use.+ """+ groups = {}+ for path in asset_clip_npzs(output_dir, asset):+ with np.load(path, allow_pickle=True) as npz:+ groups.setdefault(tuple(str(n) for n in npz['names']), []).append(path)+ if not groups:+ return [], []+ if want is None and len(groups) > 1:+ joint_names_path = os.path.join(output_dir, 'joint_names.json')+ if os.path.isfile(joint_names_path):+ with open(joint_names_path) as f:+ want = json.load(f).get(asset)+ key = tuple(want) if want is not None and tuple(want) in groups \+ else max(groups, key=lambda k: len(groups[k]))+ others = sorted(p for k, paths in groups.items() if k != key for p in paths)+ return groups[key], others+++def write_note(notes_dir, kind, name, lines):+ """``<notes_dir>/<kind>/<name>.txt`` holding *lines*, or removed when there+ are none (*kind*: ``TEXTURE_ISSUES_DIR`` or ``GLB_ERRORS_DIR``)."""+ path = os.path.join(notes_dir, kind, f'{name}.txt')+ if lines:+ os.makedirs(os.path.dirname(path), exist_ok=True)+ with open(path, 'w') as f:+ f.write('\n'.join(lines) + '\n')+ elif os.path.isfile(path):+ os.remove(path)+++def set_aside_old_clips(output_dir, asset):+ """Move ``motions/<asset>-*.npz`` and ``videos/<asset>-*.mp4`` of an+ earlier export to ``replaced/``. A fresh export rewrites the asset whole,+ and a clip it no longer produces (renamed, now too short) must not stay+ among the new ones. Returns the number of clips moved."""+ moved = 0+ for sub, ext in (('motions', '.npz'), ('videos', '.mp4')):+ src_dir = os.path.join(output_dir, sub)+ if not os.path.isdir(src_dir):+ continue+ names = [f for f in os.listdir(src_dir)+ if f.startswith(f'{asset}-') and f.endswith(ext)]+ if not names:+ continue+ dst_dir = os.path.join(output_dir, REPLACED_DIR, sub)+ os.makedirs(dst_dir, exist_ok=True)+ for name in names:+ os.replace(os.path.join(src_dir, name), os.path.join(dst_dir, name))+ if sub == 'motions':+ moved = len(names)+ return moved+++def remove_asset_glb(output_dir, asset):+ """Delete the asset's processed GLB and its notes; called before an asset+ is (re-)exported, since the GLB was built from the skeleton being replaced."""+ paths = [processed_glb_path(output_dir, asset)] + [+ os.path.join(output_dir, PROCESSED_GLB_DIR, kind, f'{asset}.txt')+ for kind in (TEXTURE_ISSUES_DIR, GLB_ERRORS_DIR)]+ removed = [p for p in paths if os.path.isfile(p)]+ for p in removed:+ os.remove(p)+ if removed and removed[0].endswith('.glb'):+ logger.info(f"Removed the stale processed GLB of '{asset}'")+++def record_glb_error(output_dir, key, message):+ """``rigs/glb_errors/<key>.txt`` holding *message*, or removed when it is+ None: an asset's GLB failure, or one with no single asset to retry."""+ if message:+ logger.error(message)+ write_note(os.path.join(output_dir, PROCESSED_GLB_DIR), GLB_ERRORS_DIR, key,+ [message] if message else [])+++def log_other_skeletons(asset, others):+ """Warn about clips the asset's processed GLB cannot drive (another rig variant)."""+ if others:+ logger.warning(f"'{asset}': {len(others)} clip(s) use another skeleton than its "+ f"processed GLB (drive them from the source file): "+ f"{[os.path.basename(p)[:-4] for p in others]}")+++def report_glb_errors(output_dir):+ """Log the recorded GLB failures of an export directory; returns their count."""+ errors_dir = os.path.join(output_dir, PROCESSED_GLB_DIR, GLB_ERRORS_DIR)+ n = len(os.listdir(errors_dir)) if os.path.isdir(errors_dir) else 0+ if n:+ logger.error(f"{n} asset(s) have no processed GLB; see {errors_dir} "+ f"(rerun with --glb_only to retry). Their NPZs are complete.")+ return n+++# ---------------------------------------------------------------------------+# GLB contents+# ---------------------------------------------------------------------------++def read_glb_json(path):+ """The JSON chunk of a GLB file (ValueError when it is not one)."""+ with open(path, 'rb') as f:+ magic, _version, _length = struct.unpack('<4sII', f.read(12))+ if magic != b'glTF':+ raise ValueError(f"{path} is not a GLB file")+ chunk_len, chunk_type = struct.unpack('<I4s', f.read(8))+ if chunk_type != b'JSON':+ raise ValueError(f"{path}: first GLB chunk is not JSON")+ return json.loads(f.read(chunk_len))+++def glb_images(path):+ """``[(name, n_bytes), ...]`` for the images embedded in a GLB file; an+ image stored by URI (not embedded) counts with 0 bytes."""+ gltf = read_glb_json(path)+ views = gltf.get('bufferViews', [])+ return [(img.get('name', ''),+ views[img['bufferView']].get('byteLength', 0) if 'bufferView' in img else 0)+ for img in gltf.get('images', [])]+++def glb_image_digests(path):+ """``[(name, sha1 of the embedded bytes), ...]`` for the images of a GLB+ file (an image stored by URI, not embedded, has None)."""+ with open(path, 'rb') as f:+ data = f.read()+ json_len = struct.unpack_from('<I', data, 12)[0]+ gltf = json.loads(data[20:20 + json_len])+ bin_start = 20 + json_len + 8 # BIN chunk header follows the JSON chunk+ views = gltf.get('bufferViews', [])+ out = []+ for img in gltf.get('images', []):+ if 'bufferView' not in img:+ out.append((img.get('name', ''), None))+ continue+ view = views[img['bufferView']]+ off = bin_start + view.get('byteOffset', 0)+ out.append((img.get('name', ''), hashlib.sha1(data[off:off + view['byteLength']]).hexdigest()))+ return out+++def glb_node_extra(path, key):+ """The first value of glTF extras *key* on any node of a GLB, or None+ (also for a missing or unreadable file)."""+ try:+ gltf = read_glb_json(path)+ except (OSError, ValueError, struct.error):+ return None+ for node in gltf.get('nodes', []):+ value = (node.get('extras') or {}).get(key)+ if value:+ return value+ return None+++def glb_joint_order(path):+ """The joint order a canonical GLB stores (``CANONICAL_ORDER_KEY``), as a+ list of names, or None (no such GLB, or not a canonical asset)."""+ value = glb_node_extra(path, CANONICAL_ORDER_KEY)+ if isinstance(value, str):+ value = json.loads(value)+ return None if value is None else [str(n) for n in value]+++def check_glb_textures(glb_path, expected, unresolved):+ """Compare the textures a GLB embeds with the ones the scene used.++ Args:+ expected: ``{image name: names it may be embedded under}`` for the+ images the exported materials use that have pixel data (the+ exporter names an image after its datablock or its file stem).+ unresolved: names of the material images whose files were not found.++ Returns:+ A list of problems (empty when every texture made it in). An image the+ exporter merged into another (diffuse + opacity into one RGBA image,+ metallic + roughness) is embedded under the sources' names joined by+ ``-``, and counts as present. Images still missing by name are+ reported, not failed: the exporter may also skip an image wired+ through a node it cannot translate.+ """+ embedded = glb_images(glb_path)+ problems = []+ if unresolved:+ problems.append(f"texture files not found: {sorted(unresolved)}")+ empty = [n for n, size in embedded if size == 0]+ if empty:+ problems.append(f"images without embedded data: {empty}")+ if expected and not any(size for _, size in embedded):+ problems.append(f"no texture embedded although the materials use {len(expected)}")+ else:+ names = {n for n, _ in embedded}+ names |= {part for n in names for part in n.split('-')} # merged 'A-B'+ absent = sorted(n for n, aliases in expected.items() if not aliases & names)+ if absent:+ problems.append(f"{len(absent)} of {len(expected)} material images not embedded, "+ f"alone or merged (dropped by the glTF exporter?): {absent[:5]}")+ return problemsdiff --git a/data_process/utils/blender_export.py b/data_process/utils/blender_export.pyindex d03dc93..923de90 100644--- a/data_process/utils/blender_export.py+++ b/data_process/utils/blender_export.py@@ -40,11 +40,16 @@ def import_fbx(path: str):bpy.ops.import_scene.fbx(filepath=path, use_anim=True)+# Float noise on an action's frame range (60.0000038 for 60) must not add a+# frame: an end within this of an integer rounds to it.+FRAME_RANGE_EPS = 1e-3++def action_frame_range(action):"""Return (start, end) integer frame range for an action."""fr = action.frame_range- start = int(floor(fr[0]))- end = int(ceil(fr[1]))+ start = int(floor(fr[0] + FRAME_RANGE_EPS))+ end = int(ceil(fr[1] - FRAME_RANGE_EPS))return start, max(start, end)@@ -101,8 +106,8 @@ def load_excluded_stems(input_dir: Path) -> set:return stems-def list_gltf_files(input_dir: Path):- exts = {".glb", ".gltf"}+def _list_files(input_dir: Path, exts):+ input_dir = Path(input_dir)paths = sorted(p for p in input_dir.glob("*") if p.is_file() and p.suffix.lower() in exts)excluded = load_excluded_stems(input_dir)if not excluded:@@ -113,6 +118,73 @@ def list_gltf_files(input_dir: Path):return kept+def list_gltf_files(input_dir: Path):+ """``.glb``/``.gltf`` files directly under *input_dir* (objaverse layout),+ minus the stems listed in ``excluded.csv``."""+ return _list_files(input_dir, {".glb", ".gltf"})+++# Formats of the ``general`` dataset (and of any export_general.py input):+# the importer is picked per file by extension.+GENERAL_ASSET_EXTS = frozenset({".glb", ".gltf", ".fbx"})+++def list_asset_files(input_dir: Path):+ """``.glb``/``.gltf``/``.fbx`` files directly under *input_dir*, minus the+ stems listed in ``excluded.csv``. The asset list of the ``general``+ dataset: its exporter and renderer must see the same files."""+ return _list_files(input_dir, GENERAL_ASSET_EXTS)+++def strip_gltf_action_suffix(armature):+ """Rename the armature's actions back to their glTF animation names.++ Blender's glTF importer names the action of animation ``walk`` on node+ ``Armature`` ``walk_Armature``. A multi-animation GLB (one asset, its+ clips as named animations) would otherwise export ``{asset}-walk_Armature``.+ An action keeps its name when the stripped name is empty or already taken.+ """+ if armature is None:+ return+ suffix = f"_{armature.name}"+ owned = set()+ anim_data = armature.animation_data+ if anim_data is not None:+ if anim_data.action is not None:+ owned.add(anim_data.action)+ owned.update(s.action for t in anim_data.nla_tracks for s in t.strips+ if s.action is not None)+ for action in sorted(owned, key=lambda a: a.name):+ base = action.name[:-len(suffix)] if action.name.endswith(suffix) else ''+ if base and base not in bpy.data.actions:+ action.name = base+++def discover_clip_actions(armature):+ """Pose actions of the loaded scene (:func:`discover_pose_actions`), or,+ when the pose filter finds none, the armature's bound action (single-action+ files such as Mixamo animation-only FBX). The armature's actions are first+ renamed to their glTF animation names (:func:`strip_gltf_action_suffix`).++ The ``general`` exporter and renderer both call this, so they derive the+ same clip names; the objaverse pair has neither the renaming nor the+ fallback.++ Returns:+ ``(action_names, frame_ranges)`` like :func:`discover_pose_actions`.+ """+ strip_gltf_action_suffix(armature)+ action_names, frame_ranges = discover_pose_actions()+ if action_names or armature is None:+ return action_names, frame_ranges+ anim_data = armature.animation_data+ if anim_data is None or anim_data.action is None:+ return action_names, frame_ranges+ action = anim_data.action+ logger.info(f"No pose actions discovered; falling back to bound action '{action.name}'.")+ return [action.name], {action.name: action_frame_range(action)}++def action_is_relevant_pose(action) -> bool:"""Return True if the action has fcurves that animate pose bones."""if action is None or len(action.fcurves) == 0:@@ -172,6 +244,41 @@ def sanitize_object_type(obj_type):return obj_type.replace('-', '_').replace(' ', '_')+def asset_name(path):+ """Object-type name of one asset file: its stem with ``-`` and every+ whitespace character replaced by ``_``.++ Clips are named ``{asset}-{action}`` and every later stage takes the+ object type as the text before the first ``-``, so a stem like+ ``my-robot`` must become ``my_robot``. Objaverse stems (hex ids) and the+ Mixamo character names are unchanged by it.+ """+ return re.sub(r'[-\s]', '_', Path(path).stem)+++def asset_names(paths):+ """``{path: asset_name(path)}`` for a list of asset files.++ Raises:+ ValueError: when two files map to the same name (``a-b.glb`` and+ ``a_b.glb``, or ``x.glb`` and ``x.fbx``); they would export and+ render into one object type, the second silently skipped as+ already done.+ """+ names = {}+ by_name = {}+ for p in paths:+ name = asset_name(p)+ by_name.setdefault(name, []).append(Path(p).name)+ names[p] = name+ clashes = {n: files for n, files in by_name.items() if len(files) > 1}+ if clashes:+ detail = '; '.join(f"{n}: {', '.join(files)}" for n, files in sorted(clashes.items()))+ raise ValueError(f"{len(clashes)} asset name(s) are shared by several files "+ f"(rename them): {detail}")+ return names++def sanitize_action_name(raw_name):"""Extract a clean action name from Blender action names.@@ -234,7 +341,8 @@ def unique_clip_names(action_names):the render under one clip name end up being different animations and thecaption describes the wrong motion.- Colliding names get a ``_00``/``_01``โฆ suffix in list order. Callers on+ Colliding names get a ``_00``/``_01``โฆ suffix in list order, skipping any+ that another action's name already takes. Callers onboth sides must pass the SAME ordered action list (the full set of poseactions, before any frame-count filtering) or the suffixes diverge.@@ -250,6 +358,10 @@ def unique_clip_names(action_names):for base in bases:total[base] = total.get(base, 0) + 1+ # A suffixed name must not land on a name another action already has+ # (``Take_001_00`` from ``Take 001|A`` next to an action named+ # ``Take_001_00``): skip to the next free index.+ used = {base for base in bases if total[base] == 1}seen = {}mapping = {}for action_name, base in zip(action_names, bases):@@ -257,8 +369,11 @@ def unique_clip_names(action_names):mapping[action_name] = baseelse:idx = seen.get(base, 0)+ while f"{base}_{idx:02d}" in used:+ idx += 1seen[base] = idx + 1mapping[action_name] = f"{base}_{idx:02d}"+ used.add(mapping[action_name])return mapping@@ -268,6 +383,13 @@ def unique_clip_names(action_names):_SKINNING_WEIGHT_EPS = 1e-5++def _max_weight(skin_matrix, j):+ """Largest weight of bone *j* over the vertices; 0 without a mesh+ (a ``(0, nbones)`` skin matrix)."""+ return float(skin_matrix[:, j].max()) if len(skin_matrix) else 0.0++_CONTROL_PREFIXES = ('MCH-', 'ORG-', 'VIS-')_CONTROL_SUFFIXES_LOWER = ('_ik', '.ik', '_pole', '.pole',@@ -311,6 +433,14 @@ def choose_main_mesh(meshes):return sorted(meshes, key=lambda m: len(m.data.vertices), reverse=True)[0]+def find_skinned_meshes(armature):+ """Meshes deformed by *armature* (an Armature modifier targeting it), in+ scene order. A rig built from one mesh per part (robots) skins each part+ to its own bone, so the main mesh alone leaves most bones unskinned."""+ return [m for m in find_meshes()+ if any(md.type == 'ARMATURE' and md.object == armature for md in m.modifiers)]++# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ# Animation state management# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ@@ -519,14 +649,16 @@ def collapse_unused_root(anim: Animation):def _is_prunable(j, skin_matrix, anims_list, rest_anim, names,offset_eps, pos_var_eps, rot_eps,mean_bone_len=1.0, is_leaf=True,- consider_parent_rotate=True):+ consider_parent_rotate=True, end_effector_suffixes=()):"""Return True if bone j should be removed from the skeleton.Works with one or more animation clips for aggregated decisions.Conditions (checked in order, first match wins):- (0) Name convention โ Blender control/mechanism prefix or suffix.+ (0) Name convention โ Blender control/mechanism prefix or suffix,+ except a leaf ending in one of+ *end_effector_suffixes*, which (4) decides.(1) Near-zero offset โ bone head coincident with parent in rest pose.(2) Near-zero positionโ bone stays at parent origin in ALL clips.(3) Non-rigid (leaf) โ bone length varies from rest in ANY clip;@@ -540,10 +672,11 @@ def _is_prunable(j, skin_matrix, anims_list, rest_anim, names,Distance conditions are normalized by mean_bone_len (scale-agnostic)."""# Skinned bones are never prunable- if skin_matrix[:, j].max() >= _SKINNING_WEIGHT_EPS:+ if _max_weight(skin_matrix, j) >= _SKINNING_WEIGHT_EPS:return False- if name_is_control_bone(names[j]): # (0)+ if name_is_control_bone(names[j]) and not ( # (0)+ is_leaf and names[j].lower().endswith(tuple(end_effector_suffixes))):return Truenorm = max(mean_bone_len, 1e-9)@@ -661,7 +794,7 @@ def _find_root(parents):def _pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,offset_eps, pos_var_eps, rot_eps, min_joints,- consider_parent_rotate=True):+ consider_parent_rotate=True, end_effector_suffixes=()):"""Pass 1: iteratively remove prunable leaf bones."""count = 0mbl = _compute_mean_bone_len(rest_anim)@@ -673,7 +806,8 @@ def _pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,and _is_prunable(j, skin_matrix, anims_list, rest_anim, names,offset_eps, pos_var_eps, rot_eps,mean_bone_len=mbl, is_leaf=True,- consider_parent_rotate=consider_parent_rotate)]+ consider_parent_rotate=consider_parent_rotate,+ end_effector_suffixes=end_effector_suffixes)]if not candidates:breakkeep = [j for j in range(n_joints) if j not in set(candidates)]@@ -695,7 +829,7 @@ def _pass_promote_root(anims_list, rest_anim, names, skin_matrix):root_idx = _find_root(anims_list[0].parents)children = children_from_parents(anims_list[0].parents)[root_idx]if not (len(children) == 1- and skin_matrix[:, root_idx].max() < _SKINNING_WEIGHT_EPS):+ and _max_weight(skin_matrix, root_idx) < _SKINNING_WEIGHT_EPS):breaklogger.info(f"Promoting root child (removing '{names[root_idx]}')")new_anims = []@@ -747,7 +881,7 @@ def _pass_merge_passthrough(anims_list, rest_anim, names, skin_matrix,def prune_skeleton_shared(anims_list, rest_anim, names, skin_matrix,offset_eps=1e-3, pos_var_eps=1e-3,rot_eps=np.deg2rad(2.0), min_joints=4,- consider_parent_rotate=True):+ consider_parent_rotate=True, end_effector_suffixes=()):"""Iterative skeleton pruning across multiple animation clips.Guarantees identical topology across all clips. Three passes per@@ -762,6 +896,9 @@ def prune_skeleton_shared(anims_list, rest_anim, names, skin_matrix,rotate are preserved when their parent rotates โ treating themas meaningful end-effectors (e.g. spider toe tips). If False,such leaves are pruned regardless of parent motion.+ end_effector_suffixes: Lowercase name suffixes (e.g. ``('_tip',)``)+ that mark a leaf as an end effector rather than a control bone,+ so the name rule does not prune it."""stats = {"leaf": 0, "root_promoted": 0, "passthrough": 0}@@ -769,7 +906,8 @@ def prune_skeleton_shared(anims_list, rest_anim, names, skin_matrix,anims_list, rest_anim, names, skin_matrix, n_leaf = \_pass_prune_leaves(anims_list, rest_anim, names, skin_matrix,offset_eps, pos_var_eps, rot_eps, min_joints,- consider_parent_rotate=consider_parent_rotate)+ consider_parent_rotate=consider_parent_rotate,+ end_effector_suffixes=end_effector_suffixes)anims_list, rest_anim, names, skin_matrix, n_root = \_pass_promote_root(anims_list, rest_anim, names, skin_matrix)@@ -911,20 +1049,25 @@ def extract_animation_frames(armature, start_frame, end_frame, nbones, dtype=np.# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโdef compute_bone_weight_sums(mesh, armature):- """Sum skin weights per bone across all vertices."""- vg_to_bone = build_vgroup_to_bone_mapping(mesh, armature)+ """Sum skin weights per bone across all vertices of *mesh* (one mesh or a+ list of meshes)."""weight_sums = {bname: 0.0 for bname in armature.data.bones.keys()}- for v in mesh.data.vertices:- for g in v.groups:- bname = vg_to_bone.get(g.group)- if bname:- weight_sums[bname] += g.weight+ for m in (mesh if isinstance(mesh, (list, tuple)) else [mesh]):+ vg_to_bone = build_vgroup_to_bone_mapping(m, armature)+ for v in m.data.vertices:+ for g in v.groups:+ bname = vg_to_bone.get(g.group)+ if bname:+ weight_sums[bname] += g.weightreturn weight_sumsdef build_skeleton_arrays(armature, mesh, dtype=np.float64, apply_world=True):"""Extract skeleton topology, rest pose, and skin matrix from Blender objects.+ *mesh* is one mesh, a list of meshes (their skin matrices are stacked,+ rows in list order) or None.+Returns:bone_names, parents_array, rest_local_pos, rest_local_rot, skin_matrix"""@@ -932,11 +1075,14 @@ def build_skeleton_arrays(armature, mesh, dtype=np.float64, apply_world=True):bone_names = list(bones.keys())bone_name_to_index = {b.name: i for i, b in enumerate(bones)}- if mesh is None:+ meshes = [] if mesh is None else (list(mesh) if isinstance(mesh, (list, tuple)) else [mesh])+ if not meshes:skin_matrix = np.zeros((0, len(bone_names)), dtype=dtype)else:- vg_to_bone = build_vgroup_to_bone_mapping(mesh, armature)- _, skin_matrix = build_vertex_groups_sparse(mesh, vg_to_bone, bone_name_to_index)+ skin_matrix = np.concatenate([+ build_vertex_groups_sparse(m, build_vgroup_to_bone_mapping(m, armature),+ bone_name_to_index)[1]+ for m in meshes], axis=0)parents_array = np.array([-1 if bone.parent is None else bones.find(bone.parent.name)@@ -1014,6 +1160,8 @@ def prepare_skeleton(armature, mesh=None, dtype=np.float64, apply_world=True):When *mesh* is None (e.g. Mixamo clips without a character mesh), thesecondary-root prune step is skipped and skin_matrix is returned empty.+ A list of meshes gives the skin weights of all of them+ (:func:`build_skeleton_arrays`).Returns a dict with keys: nbones, bone_names, parents_array,rest_local_pos, rest_local_rot, skin_matrix, rest_anim, rest_anim_shared.@@ -1113,19 +1261,29 @@ def save_motion(motion_path, vis_path, anim, rest_anim, names, skin_matrix, fps,)logger.info(f"Saved visualization: {vis_path}")- np.savez(- motion_path,- rest_local_pos=rest_anim.positions[0],- rest_local_rot=rest_anim.rotations[0].qs,- anim_local_pos=anim.positions,- anim_local_rot=anim.rotations.qs,- offsets=anim.offsets,- names=np.array(names),- skin_matrix=skin_matrix,- fps=fps,- parents=anim.parents,- action_name=action_name,- )+ # Written beside the target and renamed into place, so an interrupted run+ # never leaves a truncated NPZ that a resumed run would take as done.+ tmp_path = f"{motion_path}.{os.getpid()}.tmp"+ try:+ with open(tmp_path, 'wb') as f:+ np.savez(+ f,+ rest_local_pos=rest_anim.positions[0],+ rest_local_rot=rest_anim.rotations[0].qs,+ anim_local_pos=anim.positions,+ anim_local_rot=anim.rotations.qs,+ offsets=anim.offsets,+ names=np.array(names),+ skin_matrix=skin_matrix,+ fps=fps,+ parents=anim.parents,+ action_name=action_name,+ )+ os.replace(tmp_path, motion_path)+ except BaseException:+ if os.path.exists(tmp_path):+ os.remove(tmp_path)+ raiselogger.info(f"Saved motion data: {motion_path}")diff --git a/data_process/utils/blender_rig.py b/data_process/utils/blender_rig.pyindex 2255667..6f319c0 100644--- a/data_process/utils/blender_rig.py+++ b/data_process/utils/blender_rig.py@@ -12,7 +12,9 @@ Organized as five sections:5. Character export โ export the animated mesh + armature to GLB/FBX.The counterpart for the *export* direction (Blender scene โ NPZ) lives in-:mod:`data_process.utils.blender_export`.+:mod:`data_process.utils.blender_export`; the export stage's rest-pose GLB per+asset in :mod:`data_process.utils.processed_assets` (bpy) and+:mod:`data_process.utils.asset_files` (paths and notes, bpy-free)."""import os@@ -62,13 +64,14 @@ def select_objs(obj_list=None, deselect_first=False):def load_file(filepath, *args, **kwargs):"""Import a 3D file and return the list of newly-added scene objects."""old_objs = set(bpy.context.scene.objects)- if filepath.endswith(".glb"):+ ext = os.path.splitext(filepath)[1].lower()+ if ext in (".glb", ".gltf"):bpy.ops.import_scene.gltf(filepath=filepath, *args, **kwargs)- elif filepath.endswith(".fbx"):+ elif ext == ".fbx":bpy.ops.import_scene.fbx(filepath=filepath, *args, **kwargs)- elif filepath.endswith(".obj"):+ elif ext == ".obj":bpy.ops.wm.obj_import(filepath=filepath, *args, **kwargs)- elif filepath.endswith(".ply"):+ elif ext == ".ply":bpy.ops.wm.ply_import(filepath=filepath, *args, **kwargs)else:raise RuntimeError(f"Invalid input file: {filepath}")@@ -101,10 +104,16 @@ def repair_and_pack_textures(source_path):(their texture files could not be found on disk)."""src_dir = os.path.dirname(os.path.abspath(source_path))- try:- bpy.ops.file.find_missing_files(directory=src_dir)- except RuntimeError as exc:- logger.warning(f"find_missing_files failed: {exc}")+ # A symlinked asset keeps its textures beside the link target.+ search_dirs = [src_dir]+ real_dir = os.path.dirname(os.path.realpath(source_path))+ if real_dir != src_dir:+ search_dirs.append(real_dir)+ for directory in search_dirs:+ try:+ bpy.ops.file.find_missing_files(directory=directory)+ except RuntimeError as exc:+ logger.warning(f"find_missing_files failed in {directory}: {exc}")broken = []for img in bpy.data.images:@@ -352,6 +361,64 @@ def rebuild_action_from_data(armature, anim_data_dict):EXTRA_BONES_STRATEGIES = ('keep', 'merge', 'remove')+def skin_rigid_parts(armature):+ """Bind the meshes hanging under *armature* without an Armature modifier+ (rigid parts parented to a bone or to the armature object) to one bone+ each, weight 1, keeping their rest-pose world transform: the bone they (or+ their nearest ancestor) are parented to, else the main root bone (the one+ with the most bones below it). They then move with that bone like before,+ and count as skinned for code that keeps only skinned meshes. Every part's+ bone is chosen before any part is re-parented (a part under another part+ would lose its bone otherwise), the armature is put in its rest pose+ meanwhile, and a mesh shared with other objects is copied first (vertex+ groups belong to the mesh). Returns the names of the meshes converted."""+ bones = armature.data.bones+ roots = [b for b in bones if b.parent is None]+ if not roots:+ return []+ main_root = max(roots, key=lambda b: len(b.children_recursive)).name+ plan = []+ for obj in armature.children_recursive:+ if obj.type != 'MESH' or any(m.type == 'ARMATURE' for m in obj.modifiers):+ continue+ bone, node = None, obj+ while node is not None and node != armature:+ if node.parent == armature and node.parent_type == 'BONE' \+ and node.parent_bone in bones:+ bone = node.parent_bone+ break+ node = node.parent+ plan.append((obj, bone or main_root))+ if not plan:+ return []+ pose_position = armature.data.pose_position+ armature.data.pose_position = 'REST'+ update_scene()+ worlds = [obj.matrix_world.copy() for obj, _ in plan]+ for obj, bone in plan:+ if obj.data.users > 1:+ obj.data = obj.data.copy()+ for group in list(obj.vertex_groups): # unbound groups named like+ if group.name in bones and group.name != bone: # bones would deform now+ obj.vertex_groups.remove(group)+ group = obj.vertex_groups.get(bone) or obj.vertex_groups.new(name=bone)+ group.add(list(range(len(obj.data.vertices))), 1.0, 'REPLACE')+ modifier = obj.modifiers.new(name='Armature', type='ARMATURE')+ modifier.object = armature+ for obj, _ in plan:+ obj.parent = armature+ obj.parent_type = 'OBJECT'+ update_scene()+ for (obj, _), world in zip(plan, worlds):+ obj.matrix_world = world+ update_scene()+ armature.data.pose_position = pose_position+ update_scene()+ logger.info(f"Bound {len(plan)} rigid part(s) to their bones: "+ f"{[(o.name, b) for o, b in plan][:5]}")+ return [obj.name for obj, _ in plan]++def find_skinned_meshes(armature):"""Return all mesh objects whose Armature modifier targets *armature*."""meshes = []@@ -365,20 +432,43 @@ def find_skinned_meshes(armature):return meshes-def build_ancestor_map(armature, kept_bone_set):- """Map each armature bone NOT in *kept_bone_set* to its nearest kept ancestor.+def _topmost_kept(bone, kept_bone_set):+ """The kept bone nearest *bone* in its own subtree (breadth first), or None."""+ queue = list(bone.children)+ while queue:+ b = queue.pop(0)+ if b.name in kept_bone_set:+ return b+ queue.extend(b.children)+ return None+- Returns ``{dropped_name: ancestor_name_or_None}``. ``None`` indicates- the chain to the root is entirely dropped (orphan bone).+def build_ancestor_map(armature, kept_bone_set):+ """Map each armature bone NOT in *kept_bone_set* to the kept bone that+ takes its weights: its nearest kept ancestor. A bone with none (a dropped+ root, such as the pass-through root the export promotes away, or a separate+ root) maps to the topmost kept bone of its own subtree, else to the main+ kept root (the one with the most kept bones below it), so its vertices+ follow the body instead of losing their weights.++ Returns ``{dropped_name: kept_name_or_None}``; ``None`` only when no+ bone is kept."""+ bones = armature.data.bones+ kept_roots = [b for b in bones if b.name in kept_bone_set+ and not any(a.name in kept_bone_set for a in b.parent_recursive)]+ main_root = max(kept_roots, key=lambda b: sum(c.name in kept_bone_set+ for c in b.children_recursive),+ default=None)mapping = {}- for bone in armature.data.bones:+ for bone in bones:if bone.name in kept_bone_set:continueparent = bone.parentwhile parent is not None and parent.name not in kept_bone_set:parent = parent.parent- mapping[bone.name] = parent.name if parent is not None else None+ target = parent or _topmost_kept(bone, kept_bone_set) or main_root+ mapping[bone.name] = target.name if target is not None else Nonereturn mapping@@ -414,7 +504,9 @@ def transfer_vertex_group(mesh_obj, src_name, dst_name):def merge_extra_bone_weights(armature, kept_bone_set):- """Transfer weights of dropped bones to their nearest kept ancestor.+ """Transfer weights of dropped bones to the kept bone that takes them+ (:func:`build_ancestor_map`: the nearest kept ancestor; for a dropped+ root, the kept bone below it).Returns the ancestor map used (for logging)."""@@ -537,6 +629,41 @@ def remove_armature_bones(armature, bone_names):bpy.ops.object.mode_set(mode='OBJECT')+_DEDUP_SUFFIX = re.compile(r'^(.*)\.(\d{3})$')+++def rename_truncated_bones(armature, motion_names):+ """Give bones the motion's name where the two differ only in how a long+ name was cut. Blender caps bone names at 63 bytes and de-duplicates with+ ``.NNN``, and Blender versions differ in how many characters they keep+ before the suffix (``<base>.001`` in one, ``<base>h.001`` in another). A+ motion name absent from+ the armature matches the one bone, itself not a motion name and cut at the+ limit (63 bytes), with the same ``.NNN`` suffix whose base starts with the+ motion name's base (itself cut: at least 59 bytes); renaming the bone+ renames its vertex groups too. Returns ``{old: new}``."""+ wanted = [str(n) for n in motion_names]+ wanted_set = set(wanted)+ bone_names = set(armature.data.bones.keys())+ free = [b for b in bone_names if b not in wanted_set and len(b.encode()) == 63]+ renames = {}+ for name in wanted:+ m = _DEDUP_SUFFIX.match(name)+ if name in bone_names or m is None or len(name.encode()) < 59:+ continue+ cands = [b for b in free if b not in renames+ and (mb := _DEDUP_SUFFIX.match(b)) is not None+ and mb.group(2) == m.group(2) and mb.group(1).startswith(m.group(1))]+ if len(cands) == 1:+ renames[cands[0]] = name+ for old, new in renames.items():+ armature.data.bones[old].name = new+ if renames:+ logger.info(f"Renamed {len(renames)} bone(s) cut differently from the motion's "+ f"names: {list(renames.items())[:3]}")+ return renames++def sync_armature_bones(armature, anim_bone_names, extra_bones_strategy='merge'):"""Reconcile armature bones with the animation bone list.@@ -544,8 +671,9 @@ def sync_armature_bones(armature, anim_bone_names, extra_bones_strategy='merge')NOT present in *anim_bone_names*:- ``'merge'`` (default): transfer each extra bone's vertex weights to- its nearest kept ancestor on every skinned mesh, then delete the- bone. Mesh stays attached; articulation in the merged region is lost.+ its nearest kept ancestor on every skinned mesh (a dropped root's to+ the kept bone below it, see :func:`build_ancestor_map`), then delete+ the bone. Mesh stays attached; articulation in the merged region is lost.- ``'remove'``: delete the bones AND every mesh vertex whose dominant(max-weight) bone is one of them. Vertices with mixed influenceswhose primary bone survives are kept, so we don't punch holes near@@ -562,6 +690,7 @@ def sync_armature_bones(armature, anim_bone_names, extra_bones_strategy='merge')f"extra_bones_strategy must be one of {EXTRA_BONES_STRATEGIES}, "f"got {extra_bones_strategy!r}")+ rename_truncated_bones(armature, anim_bone_names)armature_bone_set = {bone.name for bone in armature.data.bones}anim_bone_set = {str(name) for name in anim_bone_names}@@ -580,8 +709,8 @@ def sync_armature_bones(armature, anim_bone_names, extra_bones_strategy='merge')for src, dst in sorted(ancestor_map.items()))logger.warning(- f"Merged {len(extra)} extra armature bones into nearest kept "- f"ancestor and removed them: {summary}"+ f"Merged {len(extra)} extra armature bones into the kept bone that "+ f"takes their weights and removed them: {summary}")remove_armature_bones(armature, extra)else: # 'remove'@@ -638,8 +767,8 @@ def export_selected_to_file(filepath, char_anim_type='glb', custom_props=False):"""Export currently-selected scene objects to GLB or FBX.``custom_props`` also exports object-level custom properties (glTF- extras / FBX user properties) โ used by ``preprocess_char`` to carry the- canonical joint order inside the asset.+ extras / FBX user properties) โ used by the canonical bake to carry+ the canonical joint order inside the asset."""if char_anim_type == 'glb':bpy.ops.export_scene.gltf(@@ -675,5 +804,3 @@ def export_selected_to_file(filepath, char_anim_type='glb', custom_props=False):else:raise RuntimeError(f"Invalid char_anim_type: {char_anim_type}")logger.info(f"Saved animated character: {filepath}")--diff --git a/data_process/utils/motion_features.py b/data_process/utils/motion_features.pyindex daf42ba..fa38d95 100644--- a/data_process/utils/motion_features.py+++ b/data_process/utils/motion_features.py@@ -48,6 +48,7 @@ from data_process.utils.topology import (compute_laplacian_eigenvectors,)from data_process.utils.skeleton import (+ DegenerateSkeletonError,reorder_anim_bfs,get_root_facing_quat,canonicalize_anim,@@ -82,14 +83,6 @@ PREVIEW_FIGSIZE = (6.4, 6.4)PREVIEW_DPI = 150-class DegenerateSkeletonError(ValueError):- """Raised when a T-pose cannot be canonicalized (e.g. zero-length bones).-- Callers treat this as a skip-with-reason for the whole object type rather- than a hard failure.- """--# ===========================================================================# Joint-name resolution# ===========================================================================@@ -309,10 +302,11 @@ def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,body_axis=False, raw_names=None, head_trim=0):"""Load a motion NPZ and canonicalize facing / XZ / ground + scale.- ``scale_factor`` and ``ground_height`` are reused from T-pose- calibration so every motion of this object type shares a- diameter-consistent scale and a consistent floor height (matching the- npz pipeline); ``root_pose_init_xz`` is recomputed fresh per+ ``scale_factor`` is reused from the T-pose calibration so every motion of+ this object type shares a diameter-consistent scale. ``ground_height`` is+ the T-pose's floor when the caller reuses it (``use_tpos_ground_height``),+ else None: the motion is grounded on its own minimum Y (the default).+ ``root_pose_init_xz`` is recomputed fresh permotion so each motion starts centered at the origin. Clips are thensimple slices of the canonicalized motion.@@ -326,11 +320,14 @@ def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,if motion_anim is None:return None, skip_reason- motion_anim, _, _, _ = canonicalize_anim(- motion_anim,- face_joint_idxs=face_joint_idxs, body_axis=body_axis,- ground_height=ground_height,- scale_factor=scale_factor)+ try:+ motion_anim, _, _, _ = canonicalize_anim(+ motion_anim,+ face_joint_idxs=face_joint_idxs, body_axis=body_axis,+ ground_height=ground_height,+ scale_factor=scale_factor)+ except DegenerateSkeletonError as e:+ return None, f'no facing in any frame: {e}'return (motion_anim, motion_fps), None@@ -504,7 +501,8 @@ def _activity_below_threshold(global_positions, threshold):def _discontinuity_metrics(global_positions):- """Per-frame mean joint displacement in body units, plus its median / max.+ """Per-frame mean joint displacement in clip extents (the largest side of+ the box the whole clip sweeps, root travel included), plus its median / max.A clip stitched together from several actions (or one with a teleportingroot) shows a single frame whose displacement dwarfs the rest of the clip.@@ -524,7 +522,7 @@ def _discontinuity_above_threshold(global_positions, step_threshold, ratio_thres"""Return (is_discontinuous, metrics).Flags a clip when one frame moves the skeleton at least *step_threshold*- body lengths AND that step is at least *ratio_threshold* times the clip's+ clip extents (see :func:`_discontinuity_metrics`) AND that step is at least *ratio_threshold* times the clip'smedian step โ the signature of concatenated actions, not of fast motion,which raises the median too. Non-finite input counts as discontinuous.``step_threshold <= 0`` disables the filter (non-finite input is still@@ -674,7 +672,7 @@ def _process_motion_file(object_npz_path, ctx):motion_anim, _, _ = trim_static_ends(motion_anim, static_threshold=ctx['static_threshold'])if len(motion_anim) < min_frames:- reason = (f'too few frames after static trim'+ reason = ('too few frames after static trim'+ (f' (after a {head_trim}-frame head trim)' if head_trim else '')+ f': {len(motion_anim)} < {min_frames} (orig {orig_nframes})')return {'n_saved': 0, 'frames': 0, 'clip_captions': {},@@ -688,12 +686,17 @@ def _process_motion_file(object_npz_path, ctx):clip_captions, filtered, n_saved, frames = {}, [], 0, 0for clip_idx, (clip_start, clip_end) in enumerate(clips):- (clip_local_rotations, clip_global_positions,- clip_root_facing_quat) = extract_clip_feats(- motion_anim, clip_start, clip_end,- ctx['face_joint_idxs'],- body_axis=ctx['body_axis'],- )+ try:+ (clip_local_rotations, clip_global_positions,+ clip_root_facing_quat) = extract_clip_feats(+ motion_anim, clip_start, clip_end,+ ctx['face_joint_idxs'],+ body_axis=ctx['body_axis'],+ )+ except DegenerateSkeletonError as e:+ filtered.append({'name': f'{base_name}-{clip_idx:03d}',+ 'reason': f'no facing in any frame: {e}'})+ continueis_low, clip_metrics = _activity_below_threshold(clip_global_positions, ctx['activity_threshold'])diff --git a/data_process/utils/processed_assets.py b/data_process/utils/processed_assets.pynew file mode 100644index 0000000..00770d0--- /dev/null+++ b/data_process/utils/processed_assets.py@@ -0,0 +1,773 @@+"""Build the export stage's processed assets (bpy).++``export_asset_glb`` writes ``<export>/rigs/<asset>.glb``: the asset in its+rest pose on the export's pruned skeleton, without animation, textures+embedded and checked (layout and notes: :mod:`data_process.utils.asset_files`).+The exporters call it through :func:`build_asset_glb` / :func:`add_asset_glb`,+which record a failure instead of raising.+"""++import glob+import os++import bpy+import numpy as np+from loguru import logger+from mathutils import Matrix++from data_process.utils.asset_files import (+ GLB_ERRORS_DIR,+ PROCESSED_GLB_DIR,+ TEXTURE_ISSUES_DIR,+ check_glb_textures,+ log_other_skeletons,+ main_skeleton_clips,+ processed_glb_path,+ write_note,+)+from data_process.utils.blender_rig import (+ build_ancestor_map,+ export_selected_to_file,+ find_skinned_meshes,+ rename_truncated_bones,+ repair_and_pack_textures,+ select_objs,+ sync_armature_bones,+ update_scene,+)+from data_process.utils.kinematics import clip_global, rest_global+from data_process.utils.skeleton import similarity_fit+++# ---------------------------------------------------------------------------+# Materials+# ---------------------------------------------------------------------------++def _node_tree_images(node_tree, seen_groups=None):+ """Images of every Image Texture node in *node_tree*, through node groups."""+ seen_groups = set() if seen_groups is None else seen_groups+ images = set()+ for node in node_tree.nodes:+ if node.type == 'TEX_IMAGE' and node.image is not None:+ images.add(node.image)+ elif node.type == 'GROUP' and node.node_tree is not None \+ and node.node_tree.name not in seen_groups:+ seen_groups.add(node.node_tree.name)+ images |= _node_tree_images(node.node_tree, seen_groups)+ return images+++def material_images(objs):+ """Images the materials of *objs* use (any node depth)."""+ images = set()+ for obj in objs:+ for slot in getattr(obj, 'material_slots', ()):+ mat = slot.material+ if mat is not None and mat.use_nodes and mat.node_tree is not None:+ images |= _node_tree_images(mat.node_tree)+ return images+++def _upstream_nodes(socket):+ """Every node feeding *socket*, transitively."""+ seen, stack = set(), [link.from_node for link in socket.links]+ while stack:+ node = stack.pop()+ if node in seen:+ continue+ seen.add(node)+ stack.extend(link.from_node for inp in node.inputs for link in inp.links)+ return seen+++def _material_output(mat):+ """The active Material Output node of *mat*, or None."""+ if not mat.use_nodes or mat.node_tree is None:+ return None+ outputs = [n for n in mat.node_tree.nodes if n.type == 'OUTPUT_MATERIAL']+ return next((n for n in outputs if n.is_active_output), outputs[0] if outputs else None)+++def _surface_images(mat):+ """Names of the images feeding the surface of *mat*'s active output."""+ out = _material_output(mat)+ if out is None:+ return set()+ return {n.image.name for n in _upstream_nodes(out.inputs['Surface'])+ if n.type == 'TEX_IMAGE' and n.image is not None}+++def principled_from_specgloss(objects):+ """Rebuild glTF spec-gloss materials as a Principled BSDF the exporter reads.++ Blender's glTF importer turns a ``KHR_materials_pbrSpecularGlossiness``+ material into Diffuse + Glossy (+ Emission, + a Transparent mix), which its+ exporter cannot translate, so it would write the material without its+ diffuse image. A material whose+ output is fed by a Diffuse BSDF and no Principled BSDF gets a Principled+ BSDF wired to the output: Base Color from the Diffuse color, Normal from+ the Diffuse normal, Emission from the Emission node, Alpha from the+ Transparent mix factor, metallic 0, and the Glossy roughness when it is a+ value. The old nodes stay, unconnected. The specular-glossiness map has no+ slot in a metallic-roughness material (it would have to be baked into a+ new image), so it is not carried, and a roughness that came from it falls+ back to the Principled default (0.5).++ Returns ``(converted material names, names of the images no material+ surface uses any more)``.+ """+ mats = {slot.material for obj in objects if obj.type == 'MESH'+ for slot in obj.material_slots if slot.material is not None}+ converted, before = [], set()+ for mat in mats:+ out = _material_output(mat)+ if out is None or not out.inputs['Surface'].is_linked:+ continue+ upstream = _upstream_nodes(out.inputs['Surface'])+ types = {n.type for n in upstream}+ if 'BSDF_PRINCIPLED' in types or 'BSDF_DIFFUSE' not in types:+ continue+ before |= _surface_images(mat)+ tree = mat.node_tree+ first = lambda kind: next((n for n in upstream if n.type == kind), None) # noqa: E731+ diffuse, glossy, emission = first('BSDF_DIFFUSE'), first('BSDF_GLOSSY'), first('EMISSION')+ bsdf = tree.nodes.new('ShaderNodeBsdfPrincipled')+ bsdf.location = (out.location.x - 300, out.location.y - 300)++ def carry(src, dst):+ if src.is_linked:+ tree.links.new(src.links[0].from_socket, dst)+ else:+ dst.default_value = src.default_value++ carry(diffuse.inputs['Color'], bsdf.inputs['Base Color'])+ if diffuse.inputs['Normal'].is_linked:+ carry(diffuse.inputs['Normal'], bsdf.inputs['Normal'])+ bsdf.inputs['Metallic'].default_value = 0.0+ if glossy is not None and not glossy.inputs['Roughness'].is_linked:+ bsdf.inputs['Roughness'].default_value = glossy.inputs['Roughness'].default_value+ if emission is not None:+ carry(emission.inputs['Color'], bsdf.inputs.get('Emission Color') or bsdf.inputs['Emission'])+ bsdf.inputs['Emission Strength'].default_value = emission.inputs['Strength'].default_value+ for mix in (n for n in upstream if n.type == 'MIX_SHADER'):+ # result = (1 - Fac) * Shader_1 + Fac * Shader_2: opacity is Fac+ # when the first shader is the Transparent BSDF.+ first_shader = mix.inputs[1].links[0].from_node if mix.inputs[1].is_linked else None+ if first_shader is not None and first_shader.type == 'BSDF_TRANSPARENT':+ carry(mix.inputs['Fac'], bsdf.inputs['Alpha'])+ break+ tree.links.new(bsdf.outputs['BSDF'], out.inputs['Surface'])+ converted.append(mat.name)+ # An image is dropped only if no material surface uses it any more.+ dropped = before - set().union(*(_surface_images(m) for m in mats)) if converted else set()+ if converted:+ logger.info(f"Rebuilt {len(converted)} spec-gloss material(s) as Principled BSDF: "+ f"{sorted(converted)[:4]}"+ + (f"; spec-gloss map(s) not carried: {sorted(dropped)}" if dropped else ''))+ return converted, dropped+++# Weights Blender uses to read a colour linked into a float socket.+_LUMINANCE = np.array([0.2126, 0.7152, 0.0722], dtype=np.float32)+++def _roughness_from_gloss(gloss):+ """A packed, non-colour image holding ``1 - gloss`` (*gloss* read as+ luminance), or None when *gloss* has no pixel data."""+ w, h = gloss.size+ if not (w and h and gloss.has_data):+ logger.warning(f"Gloss map '{gloss.name}' has no pixel data; left as roughness")+ return None+ px = np.empty(w * h * gloss.channels, dtype=np.float32)+ gloss.pixels.foreach_get(px)+ px = px.reshape(-1, gloss.channels)+ rough = 1.0 - (px[:, :3] @ _LUMINANCE if gloss.channels >= 3 else px[:, 0])+ out = np.ones((w * h, 4), dtype=np.float32)+ out[:, :3] = np.clip(rough, 0.0, 1.0)[:, None]+ # Named after the map's file (FBX image datablocks are often 'file6').+ stem = os.path.splitext(bpy.path.basename(gloss.filepath) or gloss.name)[0]+ image = bpy.data.images.new(f"{stem}_roughness", w, h)+ image.colorspace_settings.name = 'Non-Color'+ image.pixels.foreach_set(out.ravel())+ image.file_format = 'PNG'+ image.pack()+ return image+++def pbr_from_fbx_phong(objects):+ """Convert the Phong values Blender's FBX importer writes into a Principled+ BSDF to the metallic-roughness values they stand for.++ - Metallic: the importer copies FBX ``ReflectionFactor`` (or a reflection+ map), an environment-reflection strength rather than metalness, which+ the glTF exporter would write as metal. It becomes 0.+ - Roughness: a ``ShininessExponent`` texture is a gloss map (bright =+ shiny) that the importer links to Roughness unchanged. It is replaced by+ a packed image of ``1 - gloss``, gloss read as Blender reads a colour in+ a float socket (luminance).++ Specular keeps the importer's mapping (``SpecularFactor * 2``). Returns ``(materials whose metallic was reset,+ {gloss image name: roughness image name})``.+ """+ mats = {slot.material for obj in objects if obj.type == 'MESH'+ for slot in obj.material_slots if slot.material is not None}+ reset, inverted = [], {}+ for mat in mats:+ if not mat.use_nodes or mat.node_tree is None:+ continue+ tree = mat.node_tree+ for bsdf in [n for n in tree.nodes if n.type == 'BSDF_PRINCIPLED']:+ metallic = bsdf.inputs['Metallic']+ if metallic.is_linked or metallic.default_value != 0.0:+ for link in list(metallic.links):+ tree.links.remove(link)+ metallic.default_value = 0.0+ reset.append(mat.name)+ roughness = bsdf.inputs['Roughness']+ if not roughness.is_linked:+ continue+ link = roughness.links[0]+ node = link.from_node+ if node.type != 'TEX_IMAGE' or node.image is None or link.from_socket.name != 'Color':+ continue+ if sum(len(out.links) for out in node.outputs) > 1:+ logger.warning(f"'{mat.name}': gloss texture node also feeds other inputs; "+ f"left as roughness")+ continue+ gloss = node.image+ if gloss.name not in inverted:+ inverted[gloss.name] = _roughness_from_gloss(gloss)+ if inverted[gloss.name] is not None:+ node.image = inverted[gloss.name]+ inverted = {k: v.name for k, v in inverted.items() if v is not None}+ if reset or inverted:+ logger.info(f"FBX materials: metallic reset to 0 on {len(set(reset))}, "+ f"gloss map(s) inverted to roughness: {sorted(inverted)[:4]}")+ return sorted(set(reset)), inverted+++# ---------------------------------------------------------------------------+# Scene preparation+# ---------------------------------------------------------------------------++def _strip_to_rig(armature, kept_bones):+ """Reduce the scene to *armature* and the meshes that belong to it.++ Belonging = skinned to it (Armature modifier) or parented under it, the+ rule the renderer uses (``blender_render.keep_armature_meshes``); other+ meshes and armatures are deleted, among them armatures nested under this+ one with everything below them and the meshes they deform, except+ ancestors of what is kept (their transforms place the rig; empties are+ never deleted). A mesh parented to a bone that is about to be pruned moves+ to the bone that takes that bone's weights (:func:`build_ancestor_map`:+ the nearest kept ancestor, for a dropped root the kept bone below it),+ keeping its rest-pose world transform. Bone constraints and drivers go,+ and every bone gets the standard inheritance (rotation, full scale, local+ location): an NPZ replayed onto the asset is keyed as ``rest^-1 @ local``+ (``compute_bone_keyframes``), which reproduces the exported pose under+ exactly those rules, as glTF assumes.++ Raises:+ ValueError: no mesh belongs to the armature.+ """+ def deformed_by_other(obj):+ return any(mod.type == 'ARMATURE' and mod.object is not None and mod.object != armature+ for mod in getattr(obj, 'modifiers', ()))++ def under_other_armature(obj):+ node = obj.parent+ while node is not None and node != armature:+ if node.type == 'ARMATURE':+ return True+ node = node.parent+ return False++ # Other armatures under this one, what hangs below them, and meshes they+ # deform are another rig: a second armature would confuse character loading.+ rig = {armature} | set(find_skinned_meshes(armature)) | {+ o for o in armature.children_recursive+ if o.type != 'ARMATURE' and not deformed_by_other(o) and not under_other_armature(o)}+ keep = set(rig)+ for obj in rig:+ parent = obj.parent+ while parent is not None:+ keep.add(parent)+ parent = parent.parent+ for obj in list(bpy.context.scene.objects):+ if obj.type in ('MESH', 'ARMATURE') and obj not in keep:+ bpy.data.objects.remove(obj, do_unlink=True)+ if not any(o.type == 'MESH' for o in rig):+ raise ValueError(f"no mesh is skinned to or parented under armature '{armature.name}'")++ # Re-parent in the rest pose, so the result does not depend on the frame+ # the scene was left at (last extracted frame vs. freshly imported).+ pose_position = armature.data.pose_position+ armature.data.pose_position = 'REST'+ update_scene()+ target_of = build_ancestor_map(armature, kept_bones)+ for obj in list(armature.children_recursive):+ if obj.parent != armature or obj.parent_type != 'BONE' \+ or obj.parent_bone in kept_bones:+ continue+ target = target_of.get(obj.parent_bone)+ world = obj.matrix_world.copy()+ if target is None:+ obj.parent_type = 'OBJECT'+ obj.parent_bone = ''+ else:+ obj.parent_bone = target+ update_scene()+ obj.matrix_world = world+ logger.info(f"Re-parented '{obj.name}' to "+ f"{'the armature' if target is None else repr(target)} (bone pruned)")+ armature.data.pose_position = pose_position++ for bone in armature.data.bones:+ bone.use_inherit_rotation = True+ bone.inherit_scale = 'FULL'+ bone.use_local_location = True+ for pose_bone in armature.pose.bones:+ for constraint in list(pose_bone.constraints):+ pose_bone.constraints.remove(constraint)+ if armature.animation_data is not None:+ for driver in list(armature.animation_data.drivers):+ armature.animation_data.drivers.remove(driver)+ update_scene()+++def _set_armature_world(armature, new):+ """Give *armature* the world matrix *new*; meshes outside its hierarchy keep+ their armature-relative pose explicitly, its descendants follow it."""+ old = np.array(armature.matrix_world)+ inside = set(armature.children_recursive)+ others = [(o, np.linalg.inv(old) @ np.array(o.matrix_world))+ for o in bpy.context.scene.objects+ if o.type == 'MESH' and o not in inside]+ armature.matrix_world = Matrix(np.asarray(new).tolist())+ update_scene()+ for obj, rel in others:+ obj.matrix_world = Matrix((np.asarray(new) @ rel).tolist())+ update_scene()+++# Export space is Blender's world turned -90 degrees about X (Y-up).+_EXPORT_FROM_BLENDER = np.array([[1., 0, 0, 0], [0, 0, 1, 0], [0, -1, 0, 0], [0, 0, 0, 1]])+++def _rotate_rest(armature, fix, names, rest_pos):+ """Move the whole rig by the rest-orientation fix an export NPZ records+ (``rest_orientation_applied``: quat w, x, y, z then an optional offset, export+ space; the NPZ's rest is already ``p -> quat * p + offset`` of the original).++ The rig's rest joints match the original NPZ rest up to a similarity (scale,+ and possibly a constant offset that replay cancels); the fix is applied in that+ frame, so the rig ends up related to the fixed NPZ exactly as it was to the+ original one. *rest_pos* is the fixed NPZ's rest (export space)."""+ fix = np.asarray(fix, dtype=np.float64)+ w, x, y, z = fix[:4] / np.linalg.norm(fix[:4])+ R = np.array([[1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y)],+ [2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x)],+ [2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)]])+ t = fix[4:7] if len(fix) >= 7 else np.zeros(3)+ C = _EXPORT_FROM_BLENDER+ arm_world = np.array(armature.matrix_world)+ bones = armature.data.bones+ keep = [j for j, n in enumerate(names) if n in bones]+ heads = np.stack([(C @ arm_world @ np.array(bones[names[j]].matrix_local))[:3, 3] for j in keep])+ original = (np.asarray(rest_pos)[keep] - t) @ R # R^T (p - t), row vectors+ Rf, scale, delta, _ = similarity_fit(original, heads) # heads ~ scale Rf original + delta+ A = Rf @ R @ Rf.T+ X = np.eye(4)+ X[:3, :3] = A+ X[:3, 3] = delta - A @ delta + scale * Rf @ t+ _set_armature_world(armature, np.linalg.inv(C) @ X @ C @ arm_world)+++def _similarity_armature_frame(armature, tol=1e-3):+ """Give *armature* the nearest proper similarity of its world transform.++ Export NPZs store the skeleton in armature space, and a rotation-only+ skeleton cannot carry a non-uniform or mirroring object scale. When the+ armature's world matrix has one (singular values differing by more than+ *tol*, or a negative determinant), it is replaced by its polar rotation+ (made proper) times the mean scale, keeping its translation; meshes keep+ their placement relative to the armature, so the asset becomes its+ armature-space shape, the one its clips animate. Returns True when changed.+ """+ M = np.array(armature.matrix_world)+ U, sv, Vt = np.linalg.svd(M[:3, :3])+ det = np.linalg.det(M[:3, :3])+ if sv.max() / max(sv.min(), 1e-12) - 1 <= tol and det > 0:+ return False+ R = U @ np.diag([1.0, 1.0, np.sign(np.linalg.det(U @ Vt))]) @ Vt+ new = np.eye(4)+ new[:3, :3] = R * np.cbrt(abs(det))+ new[:3, 3] = M[:3, 3]+ _set_armature_world(armature, new)+ logger.info(f"Armature '{armature.name}' had a non-uniform or mirroring scale "+ f"(singular values {np.round(sv, 4).tolist()}, det {det:.3g}); "+ f"exported in armature space")+ return True+++def _reset_to_rest(armature):+ """Drop every action and animation binding, and put every bone at rest."""+ for action in list(bpy.data.actions):+ bpy.data.actions.remove(action)+ for obj in bpy.context.scene.objects:+ if obj.animation_data is not None:+ obj.animation_data_clear()+ shape_keys = getattr(getattr(obj, 'data', None), 'shape_keys', None)+ if shape_keys is not None and shape_keys.animation_data is not None:+ shape_keys.animation_data_clear()+ for pose_bone in armature.pose.bones:+ pose_bone.location = (0.0, 0.0, 0.0)+ pose_bone.rotation_quaternion = (1.0, 0.0, 0.0, 0.0)+ pose_bone.rotation_euler = (0.0, 0.0, 0.0)+ pose_bone.rotation_axis_angle = (0.0, 0.0, 1.0, 0.0)+ pose_bone.scale = (1.0, 1.0, 1.0)+ update_scene()+++def _check_hierarchy(armature, names, parents):+ """Every NPZ joint the rig has must hang under the bone of its nearest+ NPZ ancestor the rig has (ValueError otherwise)."""+ bones = armature.data.bones+ for j, name in enumerate(names):+ if name not in bones:+ continue+ p = int(parents[j])+ while p >= 0 and names[p] not in bones:+ p = int(parents[p])+ want = names[p] if p >= 0 else None+ got = bones[name].parent.name if bones[name].parent is not None else None+ if got != want:+ raise ValueError(f"bone '{name}' hangs under {got!r}, the NPZ puts it under "+ f"{want!r}; wrong asset or rig?")+++# Largest difference (relative to its extent) between the rig's rest skeleton+# and the NPZ's that a processed GLB may have; the same tolerance as the+# canonical bake (canonical_rig.FIT_TOL) and skeleton.motion_axis_alignment.+REST_TOL = 5e-3+# Written into export NPZs by tools/patch_annotations.py (apply_rest_orientations);+# keep the name in sync.+REST_ORIENTATION_KEY = 'rest_orientation_applied'+++def _check_rest(armature, names, rest_pos, tol):+ """The rig's rest joints must be the NPZ's (*rest_pos*, export space) up+ to a similarity, within *tol* of the extent; ValueError otherwise."""+ arm_world = np.array(armature.matrix_world)+ bones = armature.data.bones+ keep = [j for j, n in enumerate(names) if n in bones]+ rig = np.stack([(arm_world @ np.array(bones[names[j]].matrix_local))[:3, 3] for j in keep])+ resid = similarity_fit(rest_pos[keep], rig)[3]+ if resid > tol:+ raise ValueError(+ f"rest skeleton is {resid:.1e} of its extent from the NPZ's (tolerance {tol:g}); "+ f"the clips were extracted from a different reading of this file")+++# A rig whose rest skeleton is further from the origin than this many times its+# own extent is moved next to it (:func:`_recentre_far_rig`).+FAR_ORIGIN = 10.0+++def _translation(t):+ T = np.eye(4)+ T[:3, 3] = t+ return T+++def _map_rig(armature, world_map, arm_translation=None):+ """Apply *world_map* (a 4x4 uniform scale plus translation, world space) to+ the whole rig in its rest pose, in the data rather than the object+ transforms: the bones in armature space, each mesh's vertices around its own+ centre. *arm_translation* moves the armature object too (its rotation and+ scale stay); the bones absorb the difference. The asset ends up+ ``world_map`` of what it was, with no large values left in the node chain+ or vertex data for an exporter writing float32 to lose precision on."""+ W = np.asarray(world_map, dtype=np.float64)+ A = np.array(armature.matrix_world)+ A_new = A.copy()+ if arm_translation is not None:+ A_new[:3, 3] = arm_translation+ meshes = [o for o in bpy.context.scene.objects if o.type == 'MESH']+ old = {o: np.array(o.matrix_world) for o in meshes}++ def depth(obj):+ n = 0+ while obj.parent is not None:+ obj, n = obj.parent, n + 1+ return n++ X = np.linalg.inv(A_new) @ W @ A # armature space: old -> new+ view_layer = bpy.context.view_layer+ previous = view_layer.objects.active+ view_layer.objects.active = armature+ bpy.ops.object.mode_set(mode='EDIT')+ # Read everything before writing: moving a bone's tail moves the head of a+ # child connected to it.+ edit_bones = list(armature.data.edit_bones)+ ends = [(np.array(eb.head), np.array(eb.tail), eb.roll) for eb in edit_bones]+ for eb, (head, tail, roll) in zip(edit_bones, ends):+ eb.head = (X[:3, :3] @ head + X[:3, 3]).tolist()+ eb.tail = (X[:3, :3] @ tail + X[:3, 3]).tolist()+ eb.roll = roll+ bpy.ops.object.mode_set(mode='OBJECT')+ view_layer.objects.active = previous+ armature.matrix_world = Matrix(A_new.tolist())+ update_scene()++ for obj in meshes: # each mesh maps its own data+ if obj.data.users > 1:+ obj.data = obj.data.copy()+ for obj in sorted(meshes, key=depth): # parents first: children are placed by world+ target = W @ old[obj] # the object's frame, mapped+ new = old[obj].copy()+ new[:3, 3] = target[:3, 3]+ data_map = np.linalg.inv(new) @ target # carries the scale into the data+ if len(obj.data.vertices):+ co = np.empty(3 * len(obj.data.vertices))+ obj.data.vertices.foreach_get('co', co)+ co = co.reshape(-1, 3) @ data_map[:3, :3].T + data_map[:3, 3]+ centre = (co.min(0) + co.max(0)) / 2+ data_map = _translation(-centre) @ data_map+ new = new @ _translation(centre)+ obj.data.transform(Matrix(data_map.tolist()), shape_keys=True)+ obj.matrix_world = Matrix(new.tolist())+ update_scene()+++def _recentre_far_rig(armature, names, rest_pos, clip_paths, far=FAR_ORIGIN):+ """Move a rig lying more than *far* times its skeleton's extent from the+ origin next to it, when its clips then replay closer to the origin too:+ skeleton centred over the origin in X/Y, lowest joint at Z = 0 (Blender+ world), armature object at the origin. Far away, the float32 node+ transforms and inverse bind matrices of the GLB lose the pose's precision.++ Replay keys are relative to the NPZ rest pose, so a clip replays where it+ was exported, displaced by the move. Some assets were exported with their+ rest pose far away but their clips at the origin (the offset is between the+ rest pose and the animation in the source); moving those would carry every+ replay away, so they stay. *rest_pos* (export space, joints *names*) maps+ the clips (*clip_paths*) into the rig's frame. Returns True when moved."""+ A = np.array(armature.matrix_world)+ bones = armature.data.bones+ heads = np.stack([(A @ np.array(b.matrix_local))[:3, 3] for b in bones])+ lo, hi = heads.min(0), heads.max(0)+ extent = float((hi - lo).max())+ centre = (lo + hi) / 2+ if extent <= 0 or np.linalg.norm(centre) <= far * extent:+ return False+ shift = -np.array([centre[0], centre[1], lo[2]])+ keep = [j for j, n in enumerate(names) if n in bones]+ rig = np.stack([(A @ np.array(bones[names[j]].matrix_local))[:3, 3] for j in keep])+ R, s, t, _ = similarity_fit(np.asarray(rest_pos)[keep], rig) # export space -> world+ replay = []+ for path in clip_paths:+ with np.load(path, allow_pickle=True) as npz:+ replay.append(s * clip_global(npz).reshape(-1, 3).mean(0) @ R.T + t)+ replay = np.array(replay).reshape(-1, 3)+ before = float(np.linalg.norm(replay, axis=1).max()) if len(replay) else np.inf+ after = float(np.linalg.norm(replay + shift, axis=1).max()) if len(replay) else 0.0+ if after >= before:+ logger.info(f"Rig '{armature.name}' is {np.linalg.norm(centre) / extent:.0f}x its extent "+ f"from the origin but its clips replay {before / extent:.0f}x from it; "+ f"not moved")+ return False+ _map_rig(armature, _translation(shift), arm_translation=np.zeros(3))+ logger.info(f"Rig '{armature.name}' was {np.linalg.norm(centre) / extent:.0f}x its extent "+ f"from the origin; moved by {np.round(shift, 3).tolist()} (clips replay "+ f"{after / extent:.1f}x its extent from it, was {before / extent:.0f}x)")+ return True+++def _rescale_to_npz(armature, names, rest_pos, tol):+ """Scale a rig whose skeleton stands more than *tol* times taller or shorter+ than the NPZ's (*rest_pos*, export space) to the NPZ's size, about the+ armature's origin. Heights are taken over the joints both have, the rig's+ along world Z in armature units, the NPZ's along Y; a similarity fit or a+ bounding box would confuse a character's own proportions (short legs, wide+ arms) with a unit error. Returns the factor applied, or None."""+ A = np.array(armature.matrix_world)+ bones = armature.data.bones+ keep = [j for j, n in enumerate(names) if n in bones]+ heads = np.stack([(A @ np.array(bones[names[j]].matrix_local))[:3, 3] for j in keep])+ arm_scale = np.cbrt(abs(np.linalg.det(A[:3, :3])))+ npz_height = float(np.ptp(np.asarray(rest_pos)[keep, 1]))+ ratio = float(np.ptp(heads[:, 2]) / arm_scale / npz_height) if npz_height > 0 else np.nan+ if not np.isfinite(ratio) or ratio <= 0:+ logger.warning(f"Rig '{armature.name}': no height to compare with the export "+ f"skeleton's; not rescaled")+ return None+ if 1 / tol <= ratio <= tol:+ return None+ S = np.diag([1 / ratio] * 3 + [1.0])+ _map_rig(armature, A @ S @ np.linalg.inv(A))+ logger.info(f"Rig '{armature.name}' stood {ratio:.3g}x the height of the export skeleton; "+ f"scaled by {1 / ratio:.4g}")+ return 1 / ratio+++# ---------------------------------------------------------------------------+# Building+# ---------------------------------------------------------------------------++def export_asset_glb(armature, npz_path, output_dir, asset, source_path=None, required=None,+ rest_tol=REST_TOL, rescale_tol=None):+ """Write ``<output_dir>/rigs/<asset>.glb``: the asset on the export's+ pruned skeleton, in its rest pose, without animation.++ The skeleton is that of *npz_path* (one of the asset's clips). Bones whose+ long names were cut differently than in the NPZ are renamed to it+ (``rename_truncated_bones``); bones the export pruned are removed and+ their (sub-threshold) weights merged into the bone that takes them (the+ nearest kept ancestor; for a dropped root, the kept bone below it), the+ reconciliation the mesh-animation entry points apply+ (``sync_armature_bones``). Bone names and hierarchy then equal the NPZ's+ ``names`` / ``parents`` (checked), and so does the rest pose (checked up+ to a similarity, *rest_tol*; the root's Z-up -> Y-up turn is cancelled by+ keyframing); a Mixamo character keeps its own rest pose and proportions. Any clip NPZ of that skeleton drives the file+ directly (``animate_npz``), with no extra bones to reconcile.+ With *rescale_tol*, a rig whose skeleton is more than that factor larger+ or smaller than the NPZ's is scaled to its size (a character authored in+ other units). A rig further than :data:`FAR_ORIGIN` times its extent from+ the origin is moved next to it when its clips (``motions/<asset>-*.npz``)+ then replay closer to the origin too (:func:`_recentre_far_rig`).+ Spec-gloss materials are rebuilt as Principled BSDF+ (:func:`principled_from_specgloss`); textures are re-linked next to the+ source, packed, embedded and checked (``rigs/texture_issues/<asset>.txt``);+ an FBX source's materials get metallic 0 and their gloss maps inverted+ into roughness (:func:`pbr_from_fbx_phong`).++ The scene is modified in place (objects deleted, bones removed, actions+ dropped): call it after the asset's NPZs are extracted, or on a freshly+ imported asset. The file is written under a temporary name and renamed,+ so an interrupted run never leaves a truncated GLB behind.++ Args:+ required: NPZ joints the rig must have (default: all of them). Others+ may be absent (e.g. the finger bones of a Mixamo character whose+ mesh has none); the GLB then lacks them and replay skips them.+ rest_tol: largest rest-skeleton difference from the NPZ's, relative to+ its extent (default :data:`REST_TOL`); None skips the check (a+ Mixamo character, whose proportions are its own).+ rescale_tol: size factor beyond which the rig is scaled to the NPZ's+ (:func:`_rescale_to_npz`); None (default) never rescales.++ Returns:+ The GLB path.+ """+ with np.load(npz_path, allow_pickle=True) as npz:+ names = [str(n) for n in npz['names']]+ parents = np.asarray(npz['parents'])+ rest_pos = rest_global(npz)+ rest_fix = np.asarray(npz[REST_ORIENTATION_KEY]) if REST_ORIENTATION_KEY in npz.files else None+ rename_truncated_bones(armature, names)+ _strip_to_rig(armature, set(names))+ sync_armature_bones(armature, names, extra_bones_strategy='merge')+ missing = [n for n in names if n not in armature.data.bones]+ must = set(names) if required is None else set(required)+ lost = [n for n in missing if n in must]+ if lost:+ raise ValueError(f"{len(lost)} required NPZ joints are not bones of armature "+ f"'{armature.name}' (e.g. {lost[:3]}); wrong asset or rig?")+ if missing:+ logger.warning(f"'{asset}': {len(missing)} of {len(names)} export joints are not in "+ f"the rig (e.g. {missing[:3]}); the GLB has the other "+ f"{len(names) - len(missing)}")+ _check_hierarchy(armature, names, parents)+ # Actions go first: an animated object transform would override the frame.+ _reset_to_rest(armature)+ _similarity_armature_frame(armature)+ if rest_fix is not None:+ _rotate_rest(armature, rest_fix, names, rest_pos)+ if rest_tol is not None:+ _check_rest(armature, names, rest_pos, rest_tol)+ if rescale_tol is not None:+ _rescale_to_npz(armature, names, rest_pos, rescale_tol)+ clip_paths = sorted(glob.glob(os.path.join(+ output_dir, 'motions', f'{glob.escape(asset)}-*.npz'))) or [npz_path]+ _recentre_far_rig(armature, names, rest_pos, clip_paths)++ export_objs = [o for o in bpy.context.scene.objects if o.type in ('MESH', 'ARMATURE')]+ _, spec_maps = principled_from_specgloss(export_objs)+ unresolved = set()+ if source_path is not None:+ unresolved = set(repair_and_pack_textures(source_path))+ # After the repair: inverting a gloss map needs its pixels.+ if os.fspath(source_path).lower().endswith('.fbx'):+ pbr_from_fbx_phong(export_objs)+ used = [img for img in material_images(export_objs) if img.name not in spec_maps]+ unresolved &= {img.name for img in used}+ expected = {+ img.name: {img.name, os.path.splitext(img.name)[0],+ os.path.splitext(bpy.path.basename(img.filepath))[0]} - {''}+ for img in used+ if img.name not in unresolved+ and (img.packed_file is not None or img.source == 'GENERATED')}++ final = processed_glb_path(output_dir, asset)+ os.makedirs(os.path.dirname(final), exist_ok=True)+ partial = os.path.join(os.path.dirname(final), f'.{asset}.partial.glb')+ select_objs(export_objs, deselect_first=True)+ try:+ export_selected_to_file(partial, 'glb')+ os.replace(partial, final)+ finally:+ if os.path.isfile(partial):+ os.remove(partial)++ try:+ problems = check_glb_textures(final, expected, unresolved)+ except Exception as exc: # noqa: BLE001 โ the GLB is written; keep it+ problems = [f"texture check failed: {type(exc).__name__}: {exc}"]+ write_note(os.path.join(output_dir, PROCESSED_GLB_DIR), TEXTURE_ISSUES_DIR, asset, problems)+ if problems:+ logger.warning(f"Textures of '{asset}': " + '; '.join(problems))+ logger.info(f"Saved processed GLB {final} ({len(names) - len(missing)} joints, "+ f"{len(expected) + len(unresolved)} material image(s)"+ + (f", {len(unresolved)} not found" if unresolved else '') + ")")+ return final+++def build_asset_glb(asset, npz_path, output_dir, get_armature, source_path=None,+ required=None, rest_tol=REST_TOL, rescale_tol=None):+ """:func:`export_asset_glb`, with a failure recorded instead of raised.++ The GLB comes after the asset's NPZs are complete, so a GLB that cannot be+ built must not fail the export (the wrapper would refuse to merge the+ shards, on every rerun). The error goes to ``rigs/glb_errors/<asset>.txt``,+ removed again once a rerun with ``--save_glb`` / ``--glb_only`` succeeds.++ Args:+ get_armature: callable returning the asset's armature in the current+ scene (importing the asset when needed), called inside the guard.++ Returns:+ True when the GLB was written.+ """+ notes = os.path.join(output_dir, PROCESSED_GLB_DIR)+ try:+ export_asset_glb(get_armature(), npz_path, output_dir, asset,+ source_path=source_path, required=required, rest_tol=rest_tol,+ rescale_tol=rescale_tol)+ except Exception as exc: # noqa: BLE001 โ the NPZs are done; keep the batch going+ logger.exception(f"Processed GLB of '{asset}' failed (NPZs unaffected)")+ write_note(notes, GLB_ERRORS_DIR, asset, [f'{type(exc).__name__}: {exc}'])+ return False+ write_note(notes, GLB_ERRORS_DIR, asset, [])+ return True+++def add_asset_glb(asset, output_dir, get_armature, source_path=None):+ """Build the missing processed GLB of an already exported asset from one of+ its main-skeleton clips (``main_skeleton_clips``), guarded like+ :func:`build_asset_glb`. Returns True when the GLB was written."""+ main, others = main_skeleton_clips(output_dir, asset)+ if not main:+ return False+ log_other_skeletons(asset, others)+ return build_asset_glb(asset, main[0], output_dir, get_armature, source_path=source_path)diff --git a/data_process/utils/skeleton.py b/data_process/utils/skeleton.pyindex ac1ca5a..721e9c5 100644--- a/data_process/utils/skeleton.py+++ b/data_process/utils/skeleton.py@@ -2,7 +2,9 @@Topology-agnostic helpers that operate on ``Animation`` objects: BFS jointreordering, geometric measurements, uniform scaling, root alignment,-facing-axis alignment and motion activity metrics.+facing-axis alignment and motion activity metrics; plus rest-skeleton fitting+(similarity fit, collinearity, bone-axis alignment between two rigs of one+skeleton) on plain arrays.Naming conventions:get_* โ pure computations that return a value (no mutation).@@ -15,6 +17,7 @@ import statisticsfrom collections import dequeimport numpy as np+from loguru import loggerfrom Quaternions import Quaternionsfrom Animation import Animation, positions_global, offset_lengths@@ -287,6 +290,20 @@ def align_root_to_ground(anim, ground_height=None):# Root orientation alignment (facing axis)# ---------------------------------------------------------------------------+class DegenerateSkeletonError(ValueError):+ """Raised when a T-pose cannot be canonicalized (e.g. zero-length bones,+ a face pair without horizontal separation).++ Callers treat this as a skip-with-reason for the whole object type rather+ than a hard failure.+ """+++# Smallest horizontal component of the unit face-pair vector that still+# defines a facing direction.+FACING_EPS = 1e-6++def _normalize(v, eps=1e-8):"""Return ``v`` normalized along its last axis with a numerical floor."""norm = np.sqrt((v ** 2).sum(axis=-1, keepdims=True))@@ -323,6 +340,9 @@ def get_root_facing_quat(joints, face_joint_idxs, body_axis=False):Returns:Quaternions of shape (F,) aligning the skeleton to face Z+.++ Raises:+ DegenerateSkeletonError: no frame's pair has horizontal separation."""if face_joint_idxs is None or (len(face_joint_idxs) >= 2@@ -345,7 +365,24 @@ def get_root_facing_quat(joints, face_joint_idxs, body_axis=False):)across = _normalize(across)- forward = _normalize(np.cross(np.array([[0, 1, 0]]), across, axis=-1))+ forward = np.cross(np.array([[0, 1, 0]]), across, axis=-1)+ # A pair with no horizontal separation (coincident or vertical joints)+ # has no facing: such a frame takes the nearest valid frame's, and a+ # sequence without any valid frame (e.g. the single T-pose frame) cannot+ # be aligned at all.+ valid = np.sqrt((forward ** 2).sum(axis=-1)) > FACING_EPS+ if not valid.all():+ if not valid.any():+ raise DegenerateSkeletonError(+ f"face joints {list(face_joint_idxs)} have no horizontal "+ f"separation; the facing direction is undefined")+ good = np.flatnonzero(valid)+ frames = np.arange(len(forward))+ after = good[np.minimum(np.searchsorted(good, frames), len(good) - 1)]+ before = good[np.maximum(np.searchsorted(good, frames, side='right') - 1, 0)]+ nearest = np.where(np.abs(frames - before) <= np.abs(after - frames), before, after)+ forward = forward[nearest]+ forward = _normalize(forward)if body_axis:rot_correction = Quaternions.from_euler(np.array([0, -np.pi / 2, 0]), "xyz")@@ -353,6 +390,11 @@ def get_root_facing_quat(joints, face_joint_idxs, body_axis=False):target = np.array([[0, 0, 1]]).repeat(len(forward), axis=0)root_quat = Quaternions.between(forward, target)+ # ``between`` degenerates to the zero quaternion when forward is exactly -Z+ # (antiparallel to the target); the rotation there is a half turn about Y.+ flipped = (forward * target).sum(axis=-1) < -1.0 + 1e-9+ if flipped.any():+ root_quat.qs[flipped] = np.array([0.0, 0.0, 1.0, 0.0])return root_quat@@ -531,3 +573,108 @@ def get_motion_activity_metrics(positions, top_k_ratio=0.25, top_k_min=2):"joint_activity_topk": joint_activity_topk,"root_activity": root_activity,}+++# ---------------------------------------------------------------------------+# Rest-skeleton fitting (plain arrays)+# ---------------------------------------------------------------------------++def line_ratio(points):+ """Second over first singular value of the centred *points*: 0 when they+ lie on one line, where a rotation about that line cannot be measured."""+ points = np.asarray(points, dtype=np.float64)+ sv = np.linalg.svd(points - points.mean(0), compute_uv=False)+ return float(sv[1] / max(sv[0], 1e-12)) if len(sv) > 1 else 0.0+++def similarity_fit(X, Y):+ """Least-squares similarity ``Y ~= s R X + t`` between matched point sets+ (Umeyama, no reflection). Returns ``(R, s, t, residual)``, the residual+ being the largest point error over the extent of *Y*."""+ X = np.asarray(X, dtype=np.float64)+ Y = np.asarray(Y, dtype=np.float64)+ mx, my = X.mean(0), Y.mean(0)+ Xc, Yc = X - mx, Y - my+ U, S, Vt = np.linalg.svd(Yc.T @ Xc)+ D = np.eye(3)+ D[2, 2] = np.sign(np.linalg.det(U @ Vt))+ R = U @ D @ Vt+ s = float((S * D.diagonal()).sum() / max((Xc ** 2).sum(), 1e-30))+ t = my - s * R @ mx+ extent = max(float(np.ptp(Y, axis=0).max()), 1e-12)+ residual = float(np.abs(s * Xc @ R.T + my - Y).max()) / extent+ return R, s, t, residual+++def motion_axis_alignment(motion_rest_global, motion_names, rig_rest_global, rig_names,+ conj_rot=None, rel_tol=5e-3, min_angle_deg=0.01,+ min_line_ratio=5e-2, min_coverage=0.8):+ """Per-bone rotation from a motion's bone axes to a rig's, or None.++ Keys are ``restโปยน @ anim`` in each bone's own axes, so they replay+ correctly only on a rig whose bones have the motion's axes. One skeleton+ can be given different axes by different importers (Blender's glTF+ importer orients bones by a heuristic that changed between versions), and+ replaying a clip made with one onto an import made with the other plays+ it rotated and drifting. When the two rest skeletons are the same up to a+ similarity (at least *min_coverage* of the motion's joints found in the+ rig by name, joint positions within *rel_tol* of the extent; a different+ character of the same rig family is well outside it), each motion bone+ gets ``C_j = E_jแต Rแต T_j`` (*E*: its rest+ rotation in the motion's world frame, times *conj_rot* when given; *T*:+ the rig bone's rest rotation; *R*: the similarity's rotation), and+ conjugating the bone's keys by it plays the motion's world-space change+ from rest on the rig's own axes (exactly when the two skeletons have one+ scale; location keys are not rescaled). A skeleton whose joints lie+ (nearly) on one line is left alone: its roll about the line cannot be+ measured, and a fitted one would twist the motion.++ Args:+ motion_rest_global: ``(J, 4, 4)`` rest transforms of the motion (FK).+ motion_names: its ``J`` bone names.+ rig_rest_global: ``(N, 4, 4)`` armature-space rest matrices of the rig.+ rig_names: its ``N`` bone names.+ conj_rot: optional ``(J, 3, 3)`` rotations the keys are already+ conjugated by (the T-pose frames of the ``.npy`` path).++ Returns:+ ``(J, 3, 3)`` rotations to conjugate the keys by (*conj_rot* composed+ with the alignment; *conj_rot* alone, or identity, for bones the rig+ lacks), or None when the skeletons differ (another character: the+ local keys are the retargeting rule), cannot be fitted, or the axes+ already agree.+ """+ rig_index = {str(n): i for i, n in enumerate(rig_names)}+ common = [j for j, n in enumerate(motion_names) if str(n) in rig_index]+ if len(common) < max(3, min_coverage * len(motion_names)):+ return None+ X = motion_rest_global[common, :3, 3]+ Y = rig_rest_global[[rig_index[str(motion_names[j])] for j in common], :3, 3]+ R, s, _t, residual = similarity_fit(X, Y)+ if residual > rel_tol:+ return None+ E = motion_rest_global[:, :3, :3].copy()+ E /= np.linalg.norm(E, axis=1, keepdims=True)+ if conj_rot is not None:+ E = E @ conj_rot+ C = np.broadcast_to(np.eye(3), E.shape).copy()+ for j in common:+ T = rig_rest_global[rig_index[str(motion_names[j])], :3, :3]+ T = T / np.linalg.norm(T, axis=0, keepdims=True)+ C[j] = E[j].T @ R.T @ T+ angles = np.degrees(np.arccos(np.clip((np.trace(C, axis1=1, axis2=2) - 1) / 2, -1, 1)))+ if angles.max() < min_angle_deg:+ return None+ if line_ratio(X) < min_line_ratio:+ logger.warning("Rig bone axes may differ from the motion's, but the joints lie "+ "(nearly) on one line, so the difference cannot be measured; keys "+ "replayed unchanged")+ return None+ changed = [str(motion_names[j]) for j in common if angles[j] >= min_angle_deg]+ logger.info(f"Rig bone axes differ from the motion's on {len(changed)} bone(s) "+ f"({changed[:4]}{'...' if len(changed) > 4 else ''}): keys re-expressed "+ f"in the rig's axes (rest skeletons agree within {residual:.1e})")+ if abs(s - 1) > rel_tol:+ logger.warning(f"The rig's rest skeleton is {s:.4g}x the motion's: location keys "+ f"are not rescaled")+ return C if conj_rot is None else conj_rot @ Cdiff --git a/data_process/utils/vlm.py b/data_process/utils/vlm.pyindex 7bf1905..d9002cf 100644--- a/data_process/utils/vlm.py+++ b/data_process/utils/vlm.py@@ -428,8 +428,8 @@ def find_shard_paths(output_json):"""Every ``<base>.shard<N><ext>`` file sitting next to ``output_json``.Globbing (rather than trusting a ``num_gpus`` argument) means a merge- run launched with fewer GPUs than the run that produced the shards can- no longer silently drop the extra shards.+ run launched with fewer GPUs than the run that produced the shards does+ not drop the extra shards."""base, ext = os.path.splitext(output_json)pattern = "{}.shard*{}".format(os.path.basename(base), ext)@@ -449,9 +449,10 @@ def merge_shards(output_json, num_gpus=None):"""Merge per-GPU shard files into a single output JSON.Loads every ``output_json.shard*`` file found next to ``output_json``,- merges them on top of the existing ``output_json`` contents, and writes- the combined result back. ``num_gpus`` is only used to warn when fewer- shards than expected are present.+ merges them on top of the existing ``output_json`` contents, writes+ the combined result back and renames the merged shards ``*.merged``.+ ``num_gpus`` is only used to warn when fewer shards than expected are+ present.Returns:The merged dict.@@ -505,6 +506,14 @@ def merge_shards(output_json, num_gpus=None):f.write("\n".join(sorted(failed)) + ("\n" if failed else ""))logger.info("Merged failure lists -> {} pending clips in {}".format(len(failed), failed_out))++ # Merged shards step aside (renamed ``*.merged``, out of the glob): left+ # in place they would win over ``output_json`` again at the next merge+ # and resume, undoing later edits such as patched or deleted captions.+ for spath in shard_paths:+ for path in (spath, os.path.splitext(spath)[0] + "_failed.txt"):+ if os.path.exists(path):+ os.replace(path, path + ".merged")return mergeddiff --git a/data_process/vlm_caption/caption_motion.py b/data_process/vlm_caption/caption_motion.pyindex 9beb41f..fe383ad 100644--- a/data_process/vlm_caption/caption_motion.py+++ b/data_process/vlm_caption/caption_motion.py@@ -7,8 +7,8 @@ Supports three backends (see :mod:`data_process.vlm_caption.backends`):The backend is auto-detected from the model name, or set explicitly via``--backend``. ``--task`` selects the dataset system prompt (see-:mod:`data_process.vlm_caption.prompts`): ``mixamo``, ``objaverse`` or-``truebones``.+:mod:`data_process.vlm_caption.prompts`): ``mixamo``, ``objaverse``,+``truebones``, or ``general`` (your extra assets; the objaverse prompt).Input layout: ``<render_root>/<motion_name>/v00{0..3}/*.png`` โ onedirectory per motion with one sub-directory of frame PNGs per camera view.@@ -39,6 +39,7 @@ Usage:import argparseimport os+import sysimport reimport threadingfrom concurrent.futures import ThreadPoolExecutor, as_completed@@ -187,10 +188,10 @@ def caption_batch(captions = load_json(save_path)- # Resume from the merged output as well: shard files are not cleaned up- # after a merge, so a tidied-up directory would otherwise re-caption the- # whole dataset. The shard wins on conflict; only the shard's own entries- # are written back to save_path.+ # Resume from the merged output as well: a shard holds only this run's+ # clips (merged shards are renamed away), so without it every clip would+ # be captioned again. The shard wins on conflict; only the shard's own+ # entries are written back to save_path.already_done = dict(captions)if save_path != output_json:for name, caption in load_json(output_json).items():@@ -371,13 +372,18 @@ def caption_batch(n_ok, n_err, len(captions), save_path))if n_err:logger.info("Failed motion names appended to {}".format(failed_path))- return captions+ return captions, n_err# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ# CLI# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# Exit status of a run that finished with clips left uncaptioned (listed in+# the *_failed.txt file); run_caption_motion.sh merges the shards anyway.+EXIT_CLIPS_FAILED = 3++def parse_args():parser = argparse.ArgumentParser(description="Caption multi-view motion renders using a VLM "@@ -436,8 +442,8 @@ def parse_args():parser.add_argument('--base_url', type=str, default=None,help='[openai] Override API base URL.')parser.add_argument('--max_retries', type=int, default=MAX_RETRIES,- help='[openai/gemini] Max retry attempts for '- 'empty responses.')+ help='Max retry attempts for empty or unusable '+ 'responses (every backend).')# โโ Gemini 3-specific โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโparser.add_argument('--thinking_level', type=str, default='low',@@ -535,7 +541,7 @@ def main():)# โโ Run โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ- caption_batch(+ _, n_err = caption_batch(render_root=args.render_root,backend=backend,system_prompt=system_prompt,@@ -558,6 +564,10 @@ def main():media_resolution=args.media_resolution,thinking_budget=args.thinking_budget,)+ if n_err:+ # Finished, but some clips have no caption: distinct from a crash (1),+ # so the multi-GPU wrapper still merges the shards.+ sys.exit(EXIT_CLIPS_FAILED)if __name__ == "__main__":diff --git a/data_process/vlm_caption/caption_rewrite_llm.py b/data_process/vlm_caption/caption_rewrite_llm.pyindex 75d3931..36c9cec 100644--- a/data_process/vlm_caption/caption_rewrite_llm.py+++ b/data_process/vlm_caption/caption_rewrite_llm.py@@ -11,8 +11,9 @@ one sentence) and retries with sampling until it passes.Nothing is modified in place: the result is a patch JSON``{clip: {"old": ..., "new": ..., "ok": bool, "tries": n}}`` that-``data_process/tools/patch_annotations.py`` applies (after review) to-``motion_captions.json``.+``data_process/tools/patch_annotations.py`` applies to ``motion_captions.json``+when it is saved as ``<patch_dir>/<dataset>_captions_llm.json``: entries+with ``ok`` true, while the caption still equals ``old``. Review it first.Usage:python data_process/vlm_caption/caption_rewrite_llm.py \\diff --git a/data_process/vlm_caption/prompts.py b/data_process/vlm_caption/prompts.pyindex 38a19f2..bb49af7 100644--- a/data_process/vlm_caption/prompts.py+++ b/data_process/vlm_caption/prompts.py@@ -2,7 +2,8 @@The three dataset prompts (``mixamo``, ``objaverse``, ``truebones``) produce<12-word motion captions and are selected via ``--task`` in-:mod:`data_process.vlm_caption.caption_motion`. ``CATEGORY_PROMPT`` is the+:mod:`data_process.vlm_caption.caption_motion`; ``general`` (extra assets of+your own, objaverse layout) uses the objaverse prompt. ``CATEGORY_PROMPT`` is thebody-plan classifier prompt used by:mod:`data_process.vlm_caption.classify_category`.@@ -420,6 +421,9 @@ TASK_PROMPTS = {"mixamo": MIXAMO_PROMPT,"objaverse": OBJAVERSE_PROMPT,"truebones": TRUEBONES_PROMPT,+ # Arbitrary rigged objects, like objaverse; the same prompt keeps the+ # caption style of the extra data identical to the corpus it joins.+ "general": OBJAVERSE_PROMPT,}def get_prompt(task):diff --git a/scripts/_common.sh b/scripts/_common.shindex 916ea53..f4502ba 100644--- a/scripts/_common.sh+++ b/scripts/_common.sh@@ -39,6 +39,25 @@ handle_help() {esac}+# add_asset_args : append the sampler's asset flags to the wrapper's CMD array:+# --asset for each of $ASSETS, --cond_path for each of $COND_PATH (both+# space-separated), --cond_dataset_type $COND_DATASET_TYPE. (ASSETS, plural:+# ASSET is the stage-5 character kind of the animate wrappers.)+add_asset_args() {+ local _items+ if [[ -n "${ASSETS:-}" ]]; then read -r -a _items <<< "$ASSETS"; CMD+=(--asset "${_items[@]}"); fi+ if [[ -n "${COND_PATH:-}" ]]; then read -r -a _items <<< "$COND_PATH"; CMD+=(--cond_path "${_items[@]}"); fi+ if [[ -n "${COND_DATASET_TYPE:-}" ]]; then CMD+=(--cond_dataset_type "$COND_DATASET_TYPE"); fi+ return 0+}++# add_prompt_args : append --prompt with the '|'-separated prompts of $PROMPT to CMD.+add_prompt_args() {+ local _prompts+ if [[ -n "${PROMPT:-}" ]]; then IFS='|' read -r -a _prompts <<< "$PROMPT"; CMD+=(--prompt "${_prompts[@]}"); fi+ return 0+}+# select_gpu : unless CUDA_VISIBLE_DEVICES is already set, pin the run to the# GPU with the most free memory. No-op when nvidia-smi is unavailable.select_gpu() {diff --git a/scripts/run_animate_motion.sh b/scripts/run_animate_motion.shindex 4cb7197..839bbc2 100755--- a/scripts/run_animate_motion.sh+++ b/scripts/run_animate_motion.sh@@ -3,17 +3,35 @@# and export GLB + FBX (batch front-end for stage 5 of data_process/).## Usage:+# bash scripts/run_animate_motion.sh <SAMPLES_DIR> [OUTPUT_DIR]# bash scripts/run_animate_motion.sh <DATASET_TYPE> <INPUT...> [OUTPUT_DIR]# bash scripts/run_animate_motion.sh -h | --help#-# DATASET_TYPE: truebones | objaverse | mixamo+# SAMPLES_DIR: a sampler output directory (holding manifest.json, written by+# unimate.inference.sample): every motion drives the canonical GLB of+# the asset it was generated for, with that asset's cond; the joint+# order of motion, cond and GLB is checked first (data_process/+# mesh_animation/sample_manifest.py). OUTPUT_DIR defaults to+# <SAMPLES_DIR>/animated. INCLUDE_GT=1 also drives the saved ground+# truth of in-betweening / motion editing; CHAR_PATH overrides the+# character of every motion; SKIP_INVALID=1 drives the motions that+# pass the check and reports the others. A directory whose manifest+# is in its one mode subdirectory (inbetween/, motion_edit/,+# motion_expand/) works too.+#+# DATASET_TYPE: truebones | objaverse | mixamo | general# INPUT...: one or more .npy / .npz files, or a single directory containing them# OUTPUT_DIR: directory to save .glb / .fbx outputs (default: outputs/animated)# Detected as the last arg when it does not end in .npy / .npz.## Env overrides:-# ANIM_MODE (default: fk), CHAR_PATH (default: auto; mixamo falls back to-# dataset/raw/mixamo/character_refined/Y_Bot.fbx),+# ANIM_MODE (default: fk), CHAR_PATH (default: auto, by ASSET),+# ASSET (canonical, the default: the canonical asset+# dataset/canonical_assets/<DATASET_TYPE>/<object_type>.glb at canonical scale,+# falling back to export when missing; export: the processed asset+# dataset/export/<DATASET_TYPE>/rigs/<object_type>.glb, else the raw one),+# CANONICAL (1: canonical required, no fallback; 0: same as ASSET=export),+# CHARACTER (mixamo character, default Michelle; also in manifest mode),# COND_PATH (default: dataset/features/<DATASET_TYPE>/cond.npy),# EXTRA_BONES_STRATEGY (merge|remove|keep, default: merge)#@@ -34,6 +52,47 @@ if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; thenexit 0fi+if [ "$#" -ge 1 ] && { [ -f "$1/manifest.json" ] || [[ "$1" == *.json && -f "$1" ]] \+ || compgen -G "$1/*/manifest.json" >/dev/null; }; then+ MANIFEST_SRC="$1"+ if [ -d "$MANIFEST_SRC" ]; then SAMPLES_DIR="$MANIFEST_SRC"; else SAMPLES_DIR="$(dirname "$MANIFEST_SRC")"; fi+ OUTPUT_DIR="${2:-$SAMPLES_DIR/animated}"+ LIST_ARGS=("$MANIFEST_SRC")+ [ "${INCLUDE_GT:-0}" = 1 ] && LIST_ARGS+=(--include_gt)+ [ -n "${CHAR_PATH:-}" ] && LIST_ARGS+=(--char_path "$CHAR_PATH")+ [ -n "${CHARACTER:-}" ] && LIST_ARGS+=(--character "$CHARACTER")+ [ "${SKIP_INVALID:-0}" = 1 ] && LIST_ARGS+=(--skip_invalid)+ if command -v conda >/dev/null 2>&1; then+ eval "$(conda shell.bash hook 2>/dev/null)"+ conda activate "${CONDA_ENV:-unimate}" 2>/dev/null || true+ fi+ JOBS=$(python -m data_process.mesh_animation.sample_manifest "${LIST_ARGS[@]}")+ mkdir -p "$OUTPUT_DIR"+ TOTAL=$(printf '%s\n' "$JOBS" | grep -c .)+ echo "============================================================"+ echo "MANIFEST = $MANIFEST_SRC (${TOTAL} motion(s))"+ echo "OUTPUT_DIR = $OUTPUT_DIR"+ echo "============================================================"+ i=0+ while IFS=$'\t' read -r anim char cond dtype; do+ i=$((i + 1))+ stem=$(basename "${anim%.*}")+ echo "# [$i/$TOTAL] $stem (char=$char, cond=$cond, dataset_type=$dtype)"+ started=$(date +%s)+ # ASSET here is the stage-5 character kind, not the sampler's asset list+ ANIM_PATH="$anim" CHAR_PATH="$char" COND_PATH="$cond" OUTPUT_DIR="$OUTPUT_DIR" \+ ANIM_MODE="${ANIM_MODE:-fk}" EXTRA_BONES_STRATEGY="${EXTRA_BONES_STRATEGY:-merge}" \+ ASSET=canonical CANONICAL= CHARACTER= bash "$STAGE1_WRAPPER" "$dtype" </dev/null+ out="$OUTPUT_DIR/$stem.fbx"+ if [ ! -f "$out" ] || [ "$(stat -c %Y "$out" 2>/dev/null || stat -f %m "$out")" -lt "$started" ]; then+ echo "ERROR: did not produce expected FBX at '$out'" >&2+ exit 1+ fi+ done <<< "$JOBS"+ echo "BATCH DONE โ ${TOTAL} motion(s) animated into $OUTPUT_DIR"+ exit 0+fi+if [ "$#" -lt 2 ]; thenecho "ERROR: expected at least 2 positional args: <DATASET_TYPE> <INPUT...> [OUTPUT_DIR]" >&2echo "Usage: bash $(basename "$0") <DATASET_TYPE> <INPUT...> [OUTPUT_DIR]" >&2@@ -44,9 +103,9 @@ DATASET_TYPE="$1"shiftcase "$DATASET_TYPE" in- truebones|objaverse|mixamo) ;;+ truebones|objaverse|mixamo|general) ;;*)- echo "ERROR: unknown DATASET_TYPE='$DATASET_TYPE' (expected: truebones | objaverse | mixamo)" >&2+ echo "ERROR: unknown DATASET_TYPE='$DATASET_TYPE' (expected: truebones | objaverse | mixamo | general)" >&2exit 1;;esac@@ -119,8 +178,11 @@ process_one_motion() {echo "# DATASET_TYPE = $DATASET_TYPE"echo "# OUTPUT_DIR = $OUTPUT_DIR"echo "# ANIM_MODE = $ANIM_MODE"- echo "# CHAR_PATH = ${CHAR_PATH:-<auto>}"+ echo "# CHAR_PATH = ${CHAR_PATH:-<auto, ASSET=${ASSET:-canonical}>}${CHARACTER:+ (CHARACTER=$CHARACTER)}${CANONICAL:+ (CANONICAL=$CANONICAL)}"echo "############################################################"+ # An FBX left by an earlier run must not count as this run's output.+ local started+ started=$(date +%s)(export OUTPUT_DIR ANIM_MODEexport ANIM_PATH="$anim_path"@@ -133,10 +195,19 @@ process_one_motion() {if [ -n "${EXTRA_BONES_STRATEGY:-}" ]; thenexport EXTRA_BONES_STRATEGYfi+ if [ -n "${CHARACTER:-}" ]; then+ export CHARACTER+ fi+ if [ -n "${CANONICAL:-}" ]; then+ export CANONICAL+ fi+ if [ -n "${ASSET:-}" ]; then+ export ASSET+ fibash "$STAGE1_WRAPPER" "$DATASET_TYPE")- if [ ! -f "$exported_fbx" ]; then+ if [ ! -f "$exported_fbx" ] || [ "$(stat -c %Y "$exported_fbx" 2>/dev/null || stat -f %m "$exported_fbx")" -lt "$started" ]; thenecho "ERROR: did not produce expected FBX at '$exported_fbx'" >&2return 1fidiff --git a/scripts/run_sample_motion_edit.sh b/scripts/run_sample_motion_edit.shindex 575a134..9edd11a 100755--- a/scripts/run_sample_motion_edit.sh+++ b/scripts/run_sample_motion_edit.sh@@ -9,9 +9,11 @@## Arguments:# exp_dir training output directory (config.json, dataset_stats.npy, checkpoints/)-# test_cases_json {"<object_type>-<clip_id>": "edited prompt"} โ keys must name-# dataset clips (their motion is the clamped ground truth); omit+# test_cases_json {"<asset>-<clip_id>": "edited prompt"} โ keys must name clips+# (their motion is the clamped ground truth): dataset clips, or+# those of a rig_preprocess output made with --save_clips (ASSETS); omit# to use every clip of the eval split with its original caption+# (with no eval split, every training clip, one per caption)# cfg_scale classifier-free guidance scale (default: value saved in the run's config)## Env overrides:@@ -21,6 +23,8 @@# (default: random window)# REPLICATE edits per clip (default: 3); each draws a new window and noise# SEED integer seed for deterministic cropping and noise (default: unset)+# ASSETS, COND_PATH, COND_DATASET_TYPE assets the test cases may name, as in+# run_sample_motion_text.sh# OUTPUT_DIR (default: <exp_dir>/samples[_<stem>]_edit_<first joint>_n<count>)# CONDA_ENV conda environment to activate (default: unimate)# CUDA_VISIBLE_DEVICES GPU to use (default: the one with the most free memory)@@ -61,5 +65,6 @@ CMD=(python -m unimate.inference.sample[[ -n "$CFG_SCALE" ]] && CMD+=(--cfg_scale "$CFG_SCALE")[[ -n "$GT_START_FRAME" ]] && CMD+=(--gt_start_frame "$GT_START_FRAME")[[ -n "$SEED" ]] && CMD+=(--seed "$SEED")+add_asset_args"${CMD[@]}"diff --git a/scripts/run_sample_motion_expand.sh b/scripts/run_sample_motion_expand.shindex f21cd75..8a1e6bd 100644--- a/scripts/run_sample_motion_expand.sh+++ b/scripts/run_sample_motion_expand.sh@@ -6,11 +6,13 @@## Usage:# bash scripts/run_sample_motion_expand.sh <exp_dir> <test_cases_json> [cfg_scale]+# ASSETS=outputs/rig/robot PROMPT="An object walks forward.|An object turns left.|An object sits down." \+# bash scripts/run_sample_motion_expand.sh <exp_dir>## Arguments:# exp_dir training output directory (config.json, dataset_stats.npy, checkpoints/)-# test_cases_json {"<object_type>-<label>": ["prompt 1", "prompt 2", ...]} โ one-# prompt per segment, in order+# test_cases_json {"<asset>-<label>": ["prompt 1", "prompt 2", ...]} โ one+# prompt per segment, in order; omit with ASSETS + PROMPT# cfg_scale classifier-free guidance scale (default: value saved in the run's config)## Env overrides:@@ -18,6 +20,13 @@# max_motion_length (default: 10)# REPLICATE independent chains per case (default: 3)# SEED integer seed for deterministic noise (default: unset)+# ASSETS space-separated asset references (--asset), as in+# run_sample_motion_text.sh+# PROMPT the segments of one chain per ASSETS entry, separated by '|'+# COND_PATH space-separated cond.npy files whose entries the test cases+# may name (--cond_path)+# COND_DATASET_TYPE dataset whose stats normalize assets that are not dataset+# object types (--cond_dataset_type)# OUTPUT_DIR (default: <exp_dir>/samples_<stem>_expand_o<overlap>)# CONDA_ENV conda environment to activate (default: unimate)# CUDA_VISIBLE_DEVICES GPU to use (default: the one with the most free memory)@@ -29,18 +38,23 @@ source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"EXP_DIR=${1:?Usage: run_sample_motion_expand.sh <exp_dir> <test_cases_json> [cfg_scale]}-TEST_CASES=${2:?Usage: run_sample_motion_expand.sh <exp_dir> <test_cases_json> [cfg_scale]}+TEST_CASES=${2:-}+if [[ -z "$TEST_CASES" && -z "${PROMPT:-}" ]]; then+ echo "Usage: run_sample_motion_expand.sh <exp_dir> <test_cases_json> [cfg_scale] (or ASSETS + PROMPT)" >&2+ exit 1+fiCFG_SCALE=${3:-}EXPAND_OVERLAP=${EXPAND_OVERLAP:-10}REPLICATE=${REPLICATE:-3}SEED=${SEED:-}-TEST_NAME=$(basename "${TEST_CASES%.json}")+TEST_NAME=${TEST_CASES:+$(basename "${TEST_CASES%.json}")}+TEST_NAME=${TEST_NAME:-prompt}OUTPUT_DIR=${OUTPUT_DIR:-$EXP_DIR/samples_${TEST_NAME}_expand_o${EXPAND_OVERLAP}}select_gpuecho "Exp dir: $EXP_DIR"-echo "Test cases: $TEST_CASES"+echo "Test cases: ${TEST_CASES:-<ASSETS + PROMPT>}"echo "Cfg scale: ${CFG_SCALE:-<config default>}"echo "Overlap: $EXPAND_OVERLAP"echo "Replicate: $REPLICATE"@@ -48,8 +62,12 @@ echo "Output dir: $OUTPUT_DIR/motion_expand"CMD=(python -m unimate.inference.sample--exp_dir "$EXP_DIR" --output_dir "$OUTPUT_DIR" --num_repetitions "$REPLICATE"- --motion_expand --expand_overlap "$EXPAND_OVERLAP" --test_cases_json "$TEST_CASES")+ --motion_expand --expand_overlap "$EXPAND_OVERLAP")+[[ -n "$TEST_CASES" ]] && CMD+=(--test_cases_json "$TEST_CASES")[[ -n "$CFG_SCALE" ]] && CMD+=(--cfg_scale "$CFG_SCALE")[[ -n "$SEED" ]] && CMD+=(--seed "$SEED")+add_asset_args+add_prompt_args+"${CMD[@]}"diff --git a/scripts/run_sample_motion_inbetween.sh b/scripts/run_sample_motion_inbetween.shindex b0af8e4..aa4a112 100755--- a/scripts/run_sample_motion_inbetween.sh+++ b/scripts/run_sample_motion_inbetween.sh@@ -8,9 +8,11 @@## Arguments:# exp_dir training output directory (config.json, dataset_stats.npy, checkpoints/)-# test_cases_json {"<object_type>-<clip_id>": "prompt"} โ keys must name dataset-# clips (their motion is the clamped ground truth); omit to use+# test_cases_json {"<asset>-<clip_id>": "prompt"} โ keys must name clips (their+# motion is the clamped ground truth): dataset clips, or those of+# a rig_preprocess output made with --save_clips (ASSETS); omit to use# every clip of the eval split with its original caption+# (with no eval split, every training clip, one per caption)# cfg_scale classifier-free guidance scale (default: value saved in the run's config)## Env overrides:@@ -20,6 +22,8 @@# (default: random window)# REPLICATE samples per clip (default: 3); each draws a new window and noise# SEED integer seed for deterministic cropping and noise (default: unset)+# ASSETS, COND_PATH, COND_DATASET_TYPE assets the test cases may name, as in+# run_sample_motion_text.sh# OUTPUT_DIR (default: <exp_dir>/samples[_<stem>]_inbetween_<keep tag>)# CONDA_ENV conda environment to activate (default: unimate)# CUDA_VISIBLE_DEVICES GPU to use (default: the one with the most free memory)@@ -58,5 +62,6 @@ CMD=(python -m unimate.inference.sample[[ -n "$CFG_SCALE" ]] && CMD+=(--cfg_scale "$CFG_SCALE")[[ -n "$GT_START_FRAME" ]] && CMD+=(--gt_start_frame "$GT_START_FRAME")[[ -n "$SEED" ]] && CMD+=(--seed "$SEED")+add_asset_args"${CMD[@]}"diff --git a/scripts/run_sample_motion_text.sh b/scripts/run_sample_motion_text.shindex f8c9ca4..425e653 100644--- a/scripts/run_sample_motion_text.sh+++ b/scripts/run_sample_motion_text.sh@@ -3,16 +3,27 @@## Usage:# bash scripts/run_sample_motion_text.sh <exp_dir> [test_cases_json] [cfg_scale]+# ASSETS=outputs/rig/robot PROMPT="An object walks forward." bash scripts/run_sample_motion_text.sh <exp_dir>## Arguments:# exp_dir training output directory (config.json, dataset_stats.npy, checkpoints/)-# test_cases_json {"<object_type>-<case_id>": "prompt"}; omit to enumerate the-# dataset's eval split (or its train prompts when there is none)+# test_cases_json {"<asset>-<case_id>": "prompt"} or a list of {"asset", "prompt",+# "id"} entries; omit (or pass "") with ASSETS + PROMPT, or to+# enumerate the dataset's eval split (its train prompts when+# there is none)# cfg_scale classifier-free guidance scale (default: value saved in the run's config)## Env overrides:# REPLICATE samples per test case (default: 3)# SEED integer seed for deterministic noise (default: unset)+# ASSETS space-separated asset references (--asset): rig_preprocess+# output directories, one-entry cond.npy files,+# <dataset>:<object_type> or names+# PROMPT prompts sampled on every ASSETS entry, separated by '|' (--prompt)+# COND_PATH space-separated cond.npy files whose entries the test cases+# may name (--cond_path)+# COND_DATASET_TYPE dataset whose stats normalize assets that are not dataset+# object types (--cond_dataset_type)# OUTPUT_DIR (default: <exp_dir>/samples[_<test_cases stem>])# CONDA_ENV conda environment to activate (default: unimate)# CUDA_VISIBLE_DEVICES GPU to use (default: the one with the most free memory)@@ -32,7 +43,11 @@ OUTPUT_DIR=${OUTPUT_DIR:-$EXP_DIR/samples${TEST_NAME:+_$TEST_NAME}}select_gpuecho "Exp dir: $EXP_DIR"-echo "Test cases: ${TEST_CASES:-<dataset split>}"+if [[ -n "$TEST_CASES" ]]; then CASES_DESC=$TEST_CASES+elif [[ -n "${PROMPT:-}" ]]; then CASES_DESC="<ASSETS + PROMPT>"+elif [[ -n "${ASSETS:-}" ]]; then CASES_DESC="<ASSETS, unconditional: needs cfg_scale 1 (pass PROMPT otherwise)>"+else CASES_DESC="<dataset split>"; fi+echo "Test cases: $CASES_DESC"echo "Cfg scale: ${CFG_SCALE:-<config default>}"echo "Replicate: $REPLICATE"echo "Output dir: $OUTPUT_DIR"@@ -43,4 +58,7 @@ CMD=(python -m unimate.inference.sample[[ -n "$CFG_SCALE" ]] && CMD+=(--cfg_scale "$CFG_SCALE")[[ -n "$SEED" ]] && CMD+=(--seed "$SEED")+add_asset_args+add_prompt_args+"${CMD[@]}"diff --git a/scripts/run_train.sh b/scripts/run_train.shindex c86e9d3..6fc5fcb 100755--- a/scripts/run_train.sh+++ b/scripts/run_train.sh@@ -14,14 +14,14 @@# CONDA_ENV conda environment to activate (default: unimate)# CUDA_VISIBLE_DEVICES GPU to use (default: the one with the most free memory)#-# For multi-GPU runs use scripts/slurm/slurm_train.sh instead.+# Multi-GPU: accelerate launch --num_processes <N> -m unimate.training.train --config <config>set -euo pipefailsource "$(dirname "${BASH_SOURCE[0]}")/_common.sh"handle_help "$@"-CONFIG=${1:-configs/uniml3d_60frames_graph_adaln.json}-shift || true+CONFIG=configs/uniml3d_60frames_graph_adaln.json+if [[ $# -gt 0 && "$1" != "--" ]]; then CONFIG=$1; shift; fi[[ "${1:-}" == "--" ]] && shift[[ -f "$CONFIG" ]] || { echo "No such config: $CONFIG" >&2; exit 1; }@@ -30,7 +30,7 @@ select_gpuecho "Config: $CONFIG"echo "Output dir: ${OUTPUT_DIR:-<config default>}"-CMD=(accelerate launch -m unimate.training.train --config "$CONFIG")+CMD=(accelerate launch --num_processes 1 -m unimate.training.train --config "$CONFIG")[[ -n "${OUTPUT_DIR:-}" ]] && CMD+=(--output_dir "$OUTPUT_DIR")"${CMD[@]}" "$@"diff --git a/unimate/configs/schema.py b/unimate/configs/schema.pyindex 423946b..c656d25 100644--- a/unimate/configs/schema.py+++ b/unimate/configs/schema.py@@ -54,6 +54,18 @@ class ObjaverseConfig:filter_object: bool = False # If True, exclude object types listed in {path}/filtered_objects.txt+@dataclass+class GeneralConfig:+ """Optional dataset slot ``general`` for your own data, in the objaverse+ layout; used when ``dataset.dataset_list`` names it. Same fields and filters as+ :class:`ObjaverseConfig`.+ """+ type: str = "general"+ path: str = "dataset/features/general"+ objects_num: int = -1 # Number of object types to use (-1 = all)+ filter_object: bool = False # If True, exclude object types listed in {path}/filtered_objects.txt++# ---------------------------------------------------------------------------# Dataset config (shared settings + mixture routing)# ---------------------------------------------------------------------------@@ -65,6 +77,7 @@ class DatasetConfig:All datasets are loaded via the unified Mixture pipeline.Use ``dataset_list`` to specify which sub-datasets to include(e.g. ``["truebones"]`` for single-dataset, ``["truebones", "mixamo"]`` for combined).+ ``"general"`` adds your own data (:class:`GeneralConfig`); it is optional."""dataset_list: List[str] = field(default_factory=lambda: ["truebones"])@@ -72,6 +85,7 @@ class DatasetConfig:truebones: Optional[TruebonesConfig] = Nonemixamo: Optional[MixamoConfig] = Noneobjaverse: Optional[ObjaverseConfig] = None+ general: Optional[GeneralConfig] = None# Topology conditioningtopology_condition_type: str = "tpos" # 'tpos' or 'first_frame'@@ -88,18 +102,21 @@ class DatasetConfig:# Normalizationuse_dataset_stats: bool = True # True: stats keyed per dataset_type; False: one global pool across all motionsbalanced_stats: bool = False # True: equal-weight per object type; False: frame-weighted pooling- tie_std: bool = False # True: average std within position(0:3) and rotation(3:9) groups+ tie_std: bool = False # True: average std within position(0:3), rotation(3:9) and velocity(9:12)# Samplingsampler_alpha: float = 0.5 # Power-law exponent: 0=uniform-per-sample, 0.5=sqrt-balanced, 1=uniform-per-type# Two-level sampling: dataset d gets total probability โ N_d^(1-sampler_dataset_alpha)# (N_d = its clips; 0 = its natural clip share, 1 = equal per dataset), then sampler_alpha# balances object types inside it. None = legacy single level, object types pooled across- # datasets (lets one-rig Mixamo fall to <1% of a UniML3D mixture). Default None keeps- # configs and runs written before 2026-09-27 bit-identical; the *_v2 configs set 0.25.+ # datasets (lets one-rig Mixamo fall to <1% of a UniML3D mixture).sampler_dataset_alpha: Optional[float] = None+ # Explicit first level instead: {dataset_type: weight}, dataset d gets total probability+ # w_d / sum(w), whatever its clip count; sampler_alpha still balances object types inside+ # it. Keys must be exactly dataset_list; exclusive with sampler_dataset_alpha.+ sampler_dataset_weights: Optional[Dict[str, float]] = None- # Train/eval split (clip-level, stratified per object_type).+ # Train/eval split: whole object types (truebones, objaverse, general), clips (mixamo).# test_split_ratio=0 disables splitting (all clips โ train).test_split_ratio: float = 0.0split_seed: int = 42@@ -116,8 +133,19 @@ class DatasetConfig:# Re-align cropped clips so frame-0 facing is identity (only applies when start_idx > 0)realign_feature: bool = True+ # Up to three captions per clip (HumanML3D-style): the normal one (captions.json), a short+ # generic one (features/<ds>/captions_generic.json, ~6 words) and a longer detail one+ # (captions_detail.json, 7-19 words, the choreography in order). Each training sample+ # uses the generic caption with probability generic_caption_prob, the detail caption with+ # probability detail_caption_prob, else the normal one; the two must sum to <= 1+ # (1/3 + 1/3 = uniform over the three). 0 = that file is not even read. A clip without+ # the drawn version uses its normal caption.+ # Eval / sampling keep the normal caption.+ generic_caption_prob: float = 0.0+ detail_caption_prob: float = 0.0+# Runtime field โ populated in __post_init__, not serialized- data_configs: Optional[Dict[str, Union[TruebonesConfig, MixamoConfig, ObjaverseConfig]]] = field(default_factory=dict)+ data_configs: Optional[Dict[str, Union[TruebonesConfig, MixamoConfig, ObjaverseConfig, GeneralConfig]]] = field(default_factory=dict)def __post_init__(self):if not self.dataset_list:@@ -135,10 +163,31 @@ class DatasetConfig:"must be 'tpos' or 'first_frame'")+ for name in ('generic_caption_prob', 'detail_caption_prob'):+ if not 0.0 <= getattr(self, name) <= 1.0:+ raise ValueError(f"{name} must be in [0, 1], got {getattr(self, name)}")+ if self.generic_caption_prob + self.detail_caption_prob > 1.0 + 1e-9:+ raise ValueError(+ "generic_caption_prob + detail_caption_prob must be <= 1, got "+ f"{self.generic_caption_prob} + {self.detail_caption_prob}"+ )++ if self.sampler_dataset_weights is not None:+ if self.sampler_dataset_alpha is not None:+ raise ValueError("Set sampler_dataset_weights or sampler_dataset_alpha, not both")+ if set(self.sampler_dataset_weights) != set(self.dataset_list):+ raise ValueError(+ f"sampler_dataset_weights keys {sorted(self.sampler_dataset_weights)} must be "+ f"exactly dataset_list {sorted(self.dataset_list)}")+ bad = {k: w for k, w in self.sampler_dataset_weights.items() if not w > 0}+ if bad:+ raise ValueError(f"sampler_dataset_weights must be > 0, got {bad}")+_config_map = {'truebones': lambda: self.truebones or TruebonesConfig(),'mixamo': lambda: self.mixamo or MixamoConfig(),'objaverse': lambda: self.objaverse or ObjaverseConfig(),+ 'general': lambda: self.general or GeneralConfig(),}for ds_type in self.dataset_list:factory = _config_map.get(ds_type)@@ -151,6 +200,14 @@ class DatasetConfig:# Model config# ---------------------------------------------------------------------------+# Version used when a config gives only ``text_encoder_type``.+TEXT_ENCODER_DEFAULTS = {+ 'bert': 'distilbert/distilbert-base-uncased',+ 'clip': 'ViT-B/32',+ 't5': 'google/flan-t5-base',+}++@dataclassclass ModelConfig:"""Model configuration."""@@ -216,22 +273,17 @@ class ModelConfig:def __post_init__(self):# Fail on the config, not minutes later inside create_model once the# dataset has already been loaded.- for field, value, choices in (+ for name, value, choices in (('attention', self.attention, self.ATTENTION_CHOICES),('text_cond', self.text_cond, self.TEXT_COND_CHOICES)):if value not in choices:raise ValueError(- f"model.{field}={value!r} is not one of {list(choices)}")+ f"model.{name}={value!r} is not one of {list(choices)}")- _defaults = {- 'bert': 'distilbert/distilbert-base-uncased',- 'clip': 'ViT-B/32',- 't5': 'google/flan-t5-base',- }if self.text_encoder_version is None:- if self.text_encoder_type not in _defaults:+ if self.text_encoder_type not in TEXT_ENCODER_DEFAULTS:raise ValueError(f"Unknown text_encoder_type: {self.text_encoder_type}")- self.text_encoder_version = _defaults[self.text_encoder_type]+ self.text_encoder_version = TEXT_ENCODER_DEFAULTS[self.text_encoder_type]# ---------------------------------------------------------------------------@@ -288,6 +340,11 @@ class TrainingConfig:lambda_geo: float = 0.5lambda_smooth: float = 0.1+ # Flow-matching time distribution during training (flow only)+ t_sampling: str = "uniform" # 'uniform', 'logit_normal'+ t_logit_mean: float = 0.0 # logit_normal: t = sigmoid(mean + std * z)+ t_logit_std: float = 1.0+# EMAuse_ema: bool = Trueema_decay: float = 0.9999@@ -335,7 +392,7 @@ class MainConfig:"""Top-level config loaded from JSON.JSON top-level keys: experiment, truebones, mixamo, objaverse,- dataset, model, scheduler, training, sampling.+ general (optional), dataset, model, scheduler, training, sampling."""experiment: ExperimentConfigtruebones: TruebonesConfig@@ -346,6 +403,8 @@ class MainConfig:scheduler: SchedulerConfigtraining: TrainingConfigsampling: SamplingArgs+ # Last and defaulted, so configs without a "general" key still load.+ general: GeneralConfig = field(default_factory=GeneralConfig)@classmethoddef from_json(cls, config_path: Union[str, Path]) -> 'MainConfig':@@ -359,12 +418,14 @@ class MainConfig:truebones = TruebonesConfig(**data.get('truebones', {}))mixamo = MixamoConfig(**data.get('mixamo', {}))objaverse = ObjaverseConfig(**data.get('objaverse', {}))+ general = GeneralConfig(**data.get('general', {}))# Inject per-dataset configs into dataset block for __post_init__dataset_data = data['dataset']dataset_data['truebones'] = truebonesdataset_data['mixamo'] = mixamodataset_data['objaverse'] = objaverse+ dataset_data['general'] = generaldataset = DatasetConfig(**dataset_data)model_data = dict(data['model'])@@ -406,6 +467,7 @@ class MainConfig:truebones=truebones,mixamo=mixamo,objaverse=objaverse,+ general=general,dataset=dataset,model=model,scheduler=scheduler,@@ -419,7 +481,7 @@ class MainConfig:# Remove runtime-only fields that are reconstructed in __post_init__data['dataset'].pop('data_configs', None)# Remove nested sub-configs from dataset (they live at top level)- for key in ('truebones', 'mixamo', 'objaverse'):+ for key in ('truebones', 'mixamo', 'objaverse', 'general'):data['dataset'].pop(key, None)with open(config_path, 'w') as f:json.dump(data, f, indent=4)diff --git a/unimate/dataset/conditioning.py b/unimate/dataset/conditioning.pyindex b8e3942..06f357a 100644--- a/unimate/dataset/conditioning.py+++ b/unimate/dataset/conditioning.py@@ -181,6 +181,9 @@ def create_sample_condition(raw_motion = motion_data['motion'] # (F, J, D)parents = motion_data['parents'] # (J,)tpos = np.array(condition['tpos_first_frame'])+ # Grounded like the training T-pose (Mixture._precompute_object_type_meta).+ if config.dataset.ground_motion_height:+ tpos[..., 1] -= tpos[..., 1].min()max_motion_length = config.dataset.max_motion_lengthraw_motion, start_idx = apply_cropping(@@ -190,7 +193,7 @@ def create_sample_condition(# Re-align so frame-0 facing is identity (matches training-time# invariant assumed by the feature pipeline).- if start_idx > 0:+ if config.dataset.realign_feature and start_idx > 0:raw_motion = realign_unimate_clip(raw_motion, parents)# Pad tpos (J, 3) to motion feature dim (J, 12): identity 6Ddiff --git a/unimate/dataset/factory.py b/unimate/dataset/factory.pyindex bb9359b..51b2041 100644--- a/unimate/dataset/factory.py+++ b/unimate/dataset/factory.py@@ -4,7 +4,7 @@ from typing import Optional, Setfrom torch.utils.data import DataLoader-from unimate.dataset.mixture.dataset import Mixture, MixtureSampler+from unimate.dataset.mixture.dataset import Mixture, MixtureSampler, SkeletonDatasetfrom unimate.dataset.mixture.collate import mixture_batch_collate@@ -72,27 +72,84 @@ def create_dataset(target_object_types=target_object_types,target_clip_stems=target_clip_stems,stats_path=stats_path,+ # Sampling always conditions on the normal caption; generic / detail+ # ones are a training-time augmentation, so inference never loads them.+ generic_caption_prob=0.0 if inference else dataset_config.generic_caption_prob,+ detail_caption_prob=0.0 if inference else dataset_config.detail_caption_prob,)# Propagate auto-computed max_joints and max_depth back to config so model uses themdataset_config.max_joints = dataset.max_jointsdataset_config.max_depth = dataset.max_depth+ # Stats loaded from ``stats_path`` may override the stats settings; keep+ # the saved config.json truthful about the normalization in use.+ md = dataset.motion_dataset+ dataset_config.use_dataset_stats = md.use_dataset_stats+ dataset_config.balanced_stats = md.balanced_stats+ dataset_config.tie_std = md.tie_stdreturn dataset+def create_skeleton_dataset(dataset_config, model_config, stats_path: str):+ """A dataset for sampling on skeletons given by their cond entries alone+ (:class:`SkeletonDataset`): the run's padded widths (``config.json``) and+ normalization stats (``dataset_stats.npy``), no feature directory read.+ Add skeletons with ``.motion_dataset.add_cond_object``."""+ if dataset_config.max_joints <= 0 or dataset_config.max_depth <= 0:+ raise ValueError(+ f"Sampling needs the run's saved max_joints/max_depth in its config, got "+ f"max_joints={dataset_config.max_joints}, max_depth={dataset_config.max_depth}.")+ dataset = SkeletonDataset(+ stats_path=stats_path,+ max_motion_length=dataset_config.max_motion_length,+ max_joints=dataset_config.max_joints,+ max_depth=dataset_config.max_depth,+ topology_condition_type=dataset_config.topology_condition_type,+ feature_len=dataset_config.feature_len,+ text_encoder_type=model_config.text_encoder_type,+ text_encoder_version=model_config.text_encoder_version,+ use_dataset_stats=dataset_config.use_dataset_stats,+ balanced_stats=dataset_config.balanced_stats,+ tie_std=dataset_config.tie_std,+ max_freqs=model_config.max_freqs,+ ground_motion_height=dataset_config.ground_motion_height,+ realign_feature=dataset_config.realign_feature,+ )+ md = dataset.motion_dataset+ dataset_config.use_dataset_stats = md.use_dataset_stats+ dataset_config.balanced_stats = md.balanced_stats+ dataset_config.tie_std = md.tie_std+ return dataset++def create_train_dataloader(dataset_config,model_config,balanced: bool = False,batch_size: int = 16,num_workers: int = 8,+ stats_path: Optional[str] = None,+ seed: int = 0,):- """Build the training DataLoader over the train split of the Mixture dataset."""- dataset = create_dataset(dataset_config, model_config)+ """Build the training DataLoader over the train split of the Mixture dataset.++ ``seed`` (``training.seed``) seeds the balanced sampler's draw order.++ ``stats_path`` loads normalization stats from disk instead of computing+ them from the data (resume / fine-tune keep the normalization the+ checkpoint was trained with).+ """+ dataset = create_dataset(dataset_config, model_config, stats_path=stats_path)+ if not balanced and dataset_config.sampler_dataset_weights is not None:+ raise ValueError("dataset.sampler_dataset_weights needs training.balanced=true "+ "(MixtureSampler); without it clips are drawn uniformly")sampler = MixtureSampler(dataset, alpha=dataset_config.sampler_alpha,- dataset_alpha=dataset_config.sampler_dataset_alpha) if balanced else None+ dataset_alpha=dataset_config.sampler_dataset_alpha,+ dataset_weights=dataset_config.sampler_dataset_weights,+ seed=seed,+ ) if balanced else Nonereturn DataLoader(dataset,diff --git a/unimate/dataset/mixture/augmentations.py b/unimate/dataset/mixture/augmentations.pyindex a8b11d6..05e149d 100644--- a/unimate/dataset/mixture/augmentations.py+++ b/unimate/dataset/mixture/augmentations.py@@ -23,13 +23,17 @@ from unimate.utils.topology_utils import (compute_edge_indexs,compute_laplacian_eigenvectors,)-from unimate.utils.rotation_conversions import rotation_6d_to_matrix_np+from unimate.utils.rotation_conversions import matrix_to_quaternion_np, rotation_6d_to_matrix_npfrom unimate.utils.motion_utils import (compute_rifke,+ compute_local_velocity,hml_rotations_to_bvh_quaternions,fk_global_positions,)+# Identity rotation in the 6-D layout the features use.+_IDENTITY_6D = Quaternions.id(1).rotation_matrix(cont6d=True)[0]+# ---------------------------------------------------------------------------# Low-level skeleton helpers@@ -74,14 +78,25 @@ def collapse_single_child_joint(aug, j, c, p):"""Collapse a single-child interior joint *j* into its child *c*.Merges offsets and rotation channels, then re-parents *c* to grandparent- *p*. In the HML layout, joint k's slot stores ``bvh[parents[k]]``: after- collapse, c's parent becomes p, so ``motion[c, 3:9]`` must hold- ``bvh[p]`` โ which is exactly what ``motion[j, 3:9]`` stored pre-collapse.- Grandchildren's slots hold ``bvh[c]`` which must remain ``bvh[c]_old``- for FK below c to be preserved, so they are left untouched. Offsets are- summed in the rest frame (identity rest rotations).+ *p*. In the HML layout, joint k's slot stores the LOCAL rotation+ ``bvh[parents[k]]``: after collapse, c's parent becomes p, so+ ``motion[c, 3:9]`` must hold ``bvh[p]``, which is what ``motion[j, 3:9]``+ stored pre-collapse. j's own local rotation ``L_j`` (stored in c's slot)+ is folded into c's (stored in the slots of c's children) as+ ``L_j @ L_c``, so every global rotation from c down is preserved.+ Offsets are summed in the rest frame (identity rest rotations).++ Positions are preserved only where ``L_j`` is identity: c lands at+ ``p_world + G_pยท(offset_j + offset_c)`` instead of+ ``p_world + G_pยทoffset_j + G_pยทL_jยทoffset_c``, and its subtree shifts+ rigidly with it, because one straight bone cannot bend."""- aug['motion'][:, c, 3:9] = aug['motion'][:, j, 3:9]+ motion = aug['motion']+ L_j = rotation_6d_to_matrix_np(motion[:, c, 3:9]) # (T, 3, 3)+ for k in np.flatnonzero(aug['parents'] == c):+ L_c = rotation_6d_to_matrix_np(motion[:, k, 3:9])+ motion[:, k, 3:9] = Quaternions(matrix_to_quaternion_np(L_j @ L_c)).rotation_matrix(cont6d=True)+ motion[:, c, 3:9] = motion[:, j, 3:9]aug['offsets'][c] = aug['offsets'][j] + aug['offsets'][c]aug['parents'][c] = p@@ -110,19 +125,46 @@ def _augmented_global_positions(aug):def recompute_position_features(aug):- """Recompute RIFKE position features (0:3) for non-root joints via FK.+ """Recompute the position-derived features of non-root joints via FK:+ RIFKE positions (0:3) and local velocities (9:12).Re-runs the same FK pipeline used at data-processing time- (``positions_global`` + ``compute_rifke``) so the augmented clip is- consistent with the feature-extraction convention.+ (``positions_global`` + ``compute_rifke`` / ``compute_local_velocity``)+ so the augmented clip is consistent with the feature-extraction+ convention. The root's channels are untouched: every op keeps the root."""global_positions = _augmented_global_positions(aug)- root_facing_quat = Quaternions.from_transforms(+ root_facing_quat = Quaternions(matrix_to_quaternion_np(rotation_6d_to_matrix_np(aug['motion'][:, 0, 3:9])- )+ ))rifke = compute_rifke(global_positions, root_facing_quat)# Root RIFKE channels [0, Y, 0] are invariant to offset changes.aug['motion'][:, 1:, :3] = rifke[:, 1:]+ _recompute_velocity_features(aug, global_positions, root_facing_quat)+++def _recompute_velocity_features(aug, global_positions, root_facing_quat):+ """Rewrite the local velocities (9:12) of non-root joints in place.++ ``global_positions`` come from FK with the root pinned at XZ = 0, which+ drops the root's planar translation. Velocity is linear in position and+ every joint shares that translation, so its contribution is one vector+ per frame: the root's stored velocity minus the root's pinned velocity.++ Feature frame t holds the step t -> t+1, and the last feature frame's+ successor is not stored. That frame keeps its old value plus the change+ measured one frame earlier, which leaves joints the op did not move+ unchanged up to rounding and extrapolates the others (and a new joint's+ all-zero placeholder) by one frame.+ """+ motion = aug['motion']+ if motion.shape[0] < 2:+ return+ vel = compute_local_velocity(global_positions, root_facing_quat) # (T-1, J, 3)+ vel = vel + (motion[:-1, 0, 9:12] - vel[:, 0])[:, None]+ old = motion[:, 1:, 9:12].copy()+ motion[:-1, 1:, 9:12] = vel[:, 1:]+ motion[-1, 1:, 9:12] = old[-1] + (vel[-1, 1:] - old[-2])def recompute_tpos_positions(aug):@@ -230,17 +272,18 @@ def apply_joint_addition_linear(aug, max_freqs, max_path_len=5.):offset_new = offsets[ji] * alphaoffset_ji_rem = offsets[ji] * (1 - alpha)- # HML: the new joint's slot stores bvh[parents[new]] = bvh[pi],- # which is exactly what old motion[ji, 3:9] already holds. With- # identity local rotation at the new joint, bvh[new] = bvh[pi],- # so after concat motion[ji+1, 3:9] (old-ji's shifted slot, which- # must store bvh[new]) is already correct โ leave it unchanged.+ # HML: slot k stores the LOCAL rotation of parents[k]. The new joint's+ # slot therefore stores bvh[pi], which old motion[ji, 3:9] already holds.+ # Old ji (now at ji+1) must store bvh[new], the new joint's local+ # rotation, which is identity; keeping bvh[pi] there would apply pi's+ # rotation twice and rotate ji's whole subtree.new_motion = np.zeros((n_frames, 1, D), dtype=motion.dtype)new_motion[:, 0, 3:9] = motion[:, ji, 3:9]aug['motion'] = np.concatenate([motion[:, :ji], new_motion, motion[:, ji:]], axis=1,)+ aug['motion'][:, ji + 1, 3:9] = _IDENTITY_6Daug['offsets'] = np.concatenate([offsets[:ji], offset_new[None], offsets[ji:]], axis=0,)@@ -320,16 +363,14 @@ def apply_joint_addition_ellipsoid(aug, max_freqs,sample inside a bone-aligned ellipsoid with Gaussian falloff.FK preservation: the rest skeleton shape is the running sum of- ``offsets`` (identity rest rotations), so all offsets stay in the rest- frame (``offset_ji_rem = d_rem_parent``) and the rest-pose position of- every node in ji's subtree is preserved. At every frame,- ``pi_world + bvh[pi]ยทoffsets[ji]- = pi_world + bvh[pi]ยทoffset_new + bvh[new]ยทoffset_ji_rem`` requires- ``bvh[new]ยทd_rem_parent = bvh[pi]ยทd_rem_parent``; choosing identity- local rotation at the new joint satisfies this exactly and leaves every- global rotation in the subtree unchanged โ so neither ji's HML slot- (now at ji+1, storing ``bvh[new] = bvh[pi]``, already the pre-existing- value) nor any descendant slot needs to be modified.+ ``offsets`` (identity rest rotations), so ``offset_new + offset_ji_rem =+ offsets[ji]`` keeps the rest pose of ji's subtree. ``bvh`` rotations are+ local, so with ``G`` the global rotation of pi, ji sits at+ ``pi_world + Gยทoffset_new + GยทL_newยทoffset_ji_rem``, which equals the+ original ``pi_world + Gยทoffsets[ji]`` exactly when the new joint's local+ rotation ``L_new`` is identity. That identity goes into old ji's slot+ (now ji+1, which stores ``bvh[new]``); the new joint's own slot takes+ ``bvh[pi]`` from old ji's slot, and every other slot is unchanged."""motion = aug['motion']parents = aug['parents']@@ -361,6 +402,7 @@ def apply_joint_addition_ellipsoid(aug, max_freqs,aug['motion'] = np.concatenate([motion[:, :ji], new_motion, motion[:, ji:]], axis=1,)+ aug['motion'][:, ji + 1, 3:9] = _IDENTITY_6D # bvh[new]: identity localaug['offsets'] = np.concatenate([offsets[:ji], offset_new[None], offsets[ji:]], axis=0,)@@ -396,7 +438,9 @@ def apply_skeleton_pooling(aug, pool_rate, max_freqs):Teaches the model that skeletons differing only in linear-chain lengthare topologically equivalent. Rotation slots are re-wired into the- child and offsets are merged so the FK chain is preserved.+ child and offsets are merged; global rotations below each collapsed+ joint are preserved, positions only where that joint does not rotate+ (see :func:`collapse_single_child_joint`)."""parents = aug['parents']n_joints = len(parents)diff --git a/unimate/dataset/mixture/dataset.py b/unimate/dataset/mixture/dataset.pyindex 346e3a2..b84472c 100644--- a/unimate/dataset/mixture/dataset.py+++ b/unimate/dataset/mixture/dataset.py@@ -50,7 +50,7 @@ logger = get_logger(file_name=__file__, debug="mixture_dataset")# Per-dataset train/eval split mode:# - 'object_type' โ hold out whole object types as eval. Default for-# multi-topology datasets (truebones, objaverse): training+# multi-topology datasets (truebones, objaverse, general): training# never sees the held-out skeletons, so eval measures# generalization to unseen topologies.# - 'clip' โ stratified per-object-type at the clip level (train/eval@@ -66,6 +66,7 @@ _DATASET_SPLIT_MODE = {'mixamo': 'clip','truebones': 'object_type','objaverse': 'object_type',+ 'general': 'object_type',}# The UniMate representation stores per-frame deltas (root trajectory and@@ -189,6 +190,27 @@ def model_joint_names(cond_entry, object_type):# MotionDataset โ stores all motion clips in memory# ---------------------------------------------------------------------------++# Datasets normalized with another one's stats when a loaded per-dataset stats+# file lacks them (``_check_loaded_stats``).+STATS_FALLBACK = {'general': 'objaverse'}+# add_cond_object: fewest joints a skeleton can have (a lone root leaves no+# key for the final layer's attention), and how far from the origin (relative+# to the T-pose extent) its T-pose root may sit.+MIN_COND_JOINTS = 2+ROOT_XZ_TOL = 1e-6+++def _check_new_clip_key(motion_dict, key, dataset_type):+ """Clip keys share one namespace across datasets (Mixamo's are+ ``mixamo_<clip>``, the others' the file stem); a key loaded twice would+ silently replace the first clip."""+ if key in motion_dict:+ raise ValueError(+ f"clip key {key!r} of {dataset_type} is already taken by "+ f"{motion_dict[key].get('dataset_type', 'another dataset')}; rename the asset")++class MotionDataset(data.Dataset):"""In-memory dataset of motion clips across heterogeneous skeletons.@@ -226,6 +248,8 @@ class MotionDataset(data.Dataset):target_object_types: Optional[Set[str]] = None,target_clip_stems: Optional[Set[str]] = None,stats_path: Optional[str] = None,+ generic_caption_prob: float = 0.0,+ detail_caption_prob: float = 0.0,):self._init_config(topology_condition_type, max_motion_length, config_max_depth,@@ -237,6 +261,11 @@ class MotionDataset(data.Dataset):self._target_object_types = target_object_typesself._target_clip_stems = target_clip_stemsself._stats_path = stats_path+ self.generic_caption_prob = generic_caption_prob+ self.detail_caption_prob = detail_caption_prob+ # Extra caption versions this run samples -> probability, in draw order.+ self.caption_probs = {v: p for v, p in (('generic', generic_caption_prob),+ ('detail', detail_caption_prob)) if p > 0}self._init_data_state(data_dict)if self._target_object_types is not None:# Sampling uses both train and eval maps; honoring test_objects.txt@@ -304,6 +333,11 @@ class MotionDataset(data.Dataset):"""Initialize per-dataset state derived from ``data_dict``."""self.dataset_stats = {}self.cond_dict = {}+ # Object types registered from a cond entry alone (add_cond_object),+ # and the clips each was given there (ground truth for in-betweening /+ # motion editing; empty for a skeleton-only asset).+ self.cond_only_object_types = set()+ self.cond_object_clips = {}self.dataset_object_count = {}self.split_mode_by_dataset = dict(_DATASET_SPLIT_MODE)# ``explicit_eval_objects[dt]`` (from ``test_objects.txt``) overrides@@ -324,6 +358,9 @@ class MotionDataset(data.Dataset):"""self._encoder_spec = (text_encoder_type, text_encoder_version)self.text_encoder = None+ # Whether self.text_encoder was built here (and so is freed here): a+ # caller may lend its own encoder for add_cond_object.+ self._owns_text_encoder = Falseroots = [info.get('root_dir') for info in data_dict.values()]# Joint names are shared vocabulary โ the same string encodes# identically everywhere, so merging them by name across datasets is@@ -333,24 +370,40 @@ class MotionDataset(data.Dataset):# rather than merely unlikely.joint_cache = load_caches(roots, 'joint', text_encoder_type,text_encoder_version)- text_cache = {- (ds_type, clip): tokens- for ds_type, info in data_dict.items()- for clip, tokens in load_cache(info.get('root_dir'), 'caption',- text_encoder_type,- text_encoder_version).items()- }self._preencode_joint_names(data_dict, joint_cache)- self._preencode_captions(data_dict, text_cache)+ # Normal captions (captions.json) and, when a run samples them, the+ # extra versions (captions_generic.json, captions_detail.json): same+ # keys, separate caches. Passing the texts drops any cached entry+ # built from an older caption, so it is re-encoded instead of+ # silently reused.+ self._caption_tokens, self._caption_emb = self._preencode_captions(+ data_dict, 'captions', 'caption', 'captions')+ self._extra_caption_tokens, self._extra_caption_emb = {}, {}+ self._extra_captions = {}+ for v, p in self.caption_probs.items():+ self._extra_caption_tokens[v], self._extra_caption_emb[v] = (+ self._preencode_captions(data_dict, f'captions_{v}',+ f'caption_{v}', f'{v} captions'))+ self._extra_captions[v] = {+ (ds_type, clip): cap+ for ds_type, info in data_dict.items()+ for clip, cap in info.get(f'captions_{v}', {}).items()+ }+ logger.info(+ f'{v.capitalize()} captions for {len(self._extra_caption_emb[v])}/'+ f'{len(self._caption_emb)} captioned clips; each sample uses '+ f'one with p={p}')if self.text_encoder is not None:del self.text_encoderself.text_encoder = None+ self._owns_text_encoder = Falseif torch.cuda.is_available():torch.cuda.empty_cache()def _ensure_text_encoder(self):"""Build the text encoder on first use (i.e. on the first cache miss)."""if self.text_encoder is None:+ self._owns_text_encoder = Trueencoder_type, encoder_version = self._encoder_speclogger.info(f'Text cache incomplete; loading {encoder_type} encoder')self.text_encoder = create_text_encoder(@@ -371,6 +424,7 @@ class MotionDataset(data.Dataset):'truebones': self._load_multi_topology,'mixamo': self._load_single_topology,'objaverse': self._load_multi_topology,+ 'general': self._load_multi_topology,}for dataset_type, dataset_info in data_dict.items():loader = loaders.get(dataset_type)@@ -491,37 +545,43 @@ class MotionDataset(data.Dataset):all_names, 'joint names', cache=cache).items()}- def _preencode_captions(self, data_dict, cache=None):- """Encode each clip's caption โ ``self._caption_tokens`` (+ pooled).+ def _preencode_captions(self, data_dict, field, cache_kind, label):+ """Encode each clip's ``data_dict[ds][field]`` caption.- Skipped when ``target_object_types`` is set: captions come from the- per-test-case spec and are encoded inline at sample time.+ Returns ``(tokens, pooled)`` dicts keyed by ``(dataset_type, clip)``.+ Skipped (empty dicts) when ``target_object_types`` is set: captions+ come from the per-test-case spec and are encoded inline at sample time."""if self._target_object_types is not None:- self._caption_emb = {}- self._caption_tokens = {}- logger.info('Skipping caption pre-encoding (inference mode).')- return+ logger.info(f'Skipping {label} pre-encoding (inference mode).')+ return {}, {}caption_by_clip = {(ds_type, clip): capfor ds_type, dataset_info in data_dict.items()- for clip, cap in dataset_info.get('captions', {}).items()+ for clip, cap in dataset_info.get(field, {}).items()if cap}+ enc_type, enc_version = self._encoder_spec+ cache = {+ (ds_type, clip): tokens+ for ds_type, info in data_dict.items()+ for clip, tokens in load_cache(+ info.get('root_dir'), cache_kind, enc_type, enc_version,+ texts=info.get(field, {})).items()+ }# Keyed by (dataset_type, clip): a clip's# features are one lookup, with no caption-string round trip. Repeated# captions are still encoded once โ _batch_encode_unique dedupes the# strings before the forward pass.- cache = cache or {}todo = {k: c for k, c in caption_by_clip.items() if k not in cache}- by_text = self._batch_encode_unique(list(todo.values()), 'captions')+ by_text = self._batch_encode_unique(list(todo.values()), label)# A cache hit carries both views: the token sequence text_cond=# 'cross_attn' attends, and the pooled vector text_cond='adaln' reads# โ the latter exactly as the encoder produced it, so adaLN gets the# same number the pooled encoder always returned. Only a cache MISS# falls back to pooling the sequence here.- self._caption_tokens, self._caption_emb = {}, {}+ tokens_by_key, pooled_by_key = {}, {}for k in caption_by_clip:if k in cache:tokens, pooled = cache[k]@@ -530,8 +590,9 @@ class MotionDataset(data.Dataset):if tokens is None:continuepooled = pool_tokens(tokens)- self._caption_tokens[k] = tokens- self._caption_emb[k] = pooled+ tokens_by_key[k] = tokens+ pooled_by_key[k] = pooled+ return tokens_by_key, pooled_by_keydef _precompute_object_type_meta(self, object_type, cond_entry):"""Per-object-type meta reused across all clips of that type."""@@ -624,14 +685,18 @@ class MotionDataset(data.Dataset):if cap_key in self._caption_emb:entry['caption_emb'] = self._caption_emb[cap_key] # (D,)entry['caption_tokens'] = self._caption_tokens[cap_key] # (T, D)+ for v, emb in self._extra_caption_emb.items():+ if cap_key in emb:+ entry[f'caption_{v}'] = self._extra_captions[v][cap_key]+ entry[f'caption_{v}_emb'] = emb[cap_key]+ entry[f'caption_{v}_tokens'] = self._extra_caption_tokens[v][cap_key]return entrydef _check_clip_fps(self, motion_dir, motion_name, npz):"""Reject clips whose frame rate is not the one the features assume.- Pre-``fps`` feature sets predate the field; they were all built at 30- fps, so a missing field is accepted with one warning per directory- rather than failing the run.+ A clip without an ``fps`` field is taken as 30 fps, with one warning+ per directory, rather than failing the run."""if 'fps' not in npz:if motion_dir not in self._fps_unset_dirs:@@ -657,9 +722,158 @@ class MotionDataset(data.Dataset):f'across {n_types} object type{"s" if n_types != 1 else ""}')+ def _check_object_type_unique(self, object_type, dataset_type):+ """Object types and clip names key one shared namespace across datasets+ (``cond_dict``, ``motion_dict``), so a name two datasets share would+ silently replace one skeleton with the other. Only the ``general``+ dataset (assets named by their file stem) can realistically hit this."""+ for other, counts in self.dataset_object_count.items():+ if other != dataset_type and object_type in counts:+ raise ValueError(+ f"Object type {object_type!r} exists in both {other!r} and "+ f"{dataset_type!r}; rename the asset (object types must be "+ f"unique across dataset_list).")++ # ------------------------------------------------------------------+ # Assets given by their cond entry alone (inference)+ # ------------------------------------------------------------------++ def resolve_stats_key(self, dataset_type):+ """The ``dataset_stats`` key that normalizes an asset of *dataset_type*:+ itself, its ``STATS_FALLBACK``, or (one global pool) any key."""+ stats = self.dataset_stats+ if not stats:+ raise ValueError("No normalization stats loaded; sampling an asset from a "+ "cond entry needs the run's dataset_stats.npy.")+ if dataset_type in stats:+ return dataset_type+ like = STATS_FALLBACK.get(dataset_type)+ if like in stats:+ return like+ entries = list(stats.values())+ arrays = ('mean_root', 'std_root', 'mean_local', 'std_local')+ if all(all(np.array_equal(e[k], entries[0][k]) for k in arrays) for e in entries):+ return next(iter(stats))+ raise ValueError(f"No normalization stats for {dataset_type!r}; the run has "+ f"per-dataset stats for {sorted(stats)}.")++ def merge_joint_name_cache(self, root_dirs):+ """Add the joint-name embeddings cached in these feature directories+ (``unimate/tools/precompute_text_emb.py``) to the ones already encoded,+ so :meth:`add_cond_object` encodes only names none of them holds."""+ enc_type, enc_version = self._encoder_spec+ cached = load_caches([d for d in root_dirs if d], 'joint', enc_type, enc_version)+ for name, emb in cached.items():+ if name not in self._joint_name_emb:+ self._joint_name_emb[name] = emb if emb.ndim == 1 else pool_tokens(emb)++ def add_cond_object(self, object_type, cond_entry, dataset_type, motion_dir=None,+ clip_files=(), clip_key_prefix=''):+ """Register an object type from its stage-4 cond entry, for sampling.++ The skeleton is taken from *cond_entry* (an entry of a feature+ directory's ``cond.npy``, or of ``data_process.rig_preprocess``'s); the+ motion features are normalized with the stats of *dataset_type*+ (see :meth:`resolve_stats_key`). It always gets one reference clip,+ its rest pose (``<object_type>-rest_pose.npz``), which supplies the+ skeleton to ``create_sample_condition`` like any clip and is no ground+ truth. *clip_files* (file names in *motion_dir*, stage-4 feature NPZs+ of this skeleton) are loaded as its real clips, keyed+ ``<clip_key_prefix><file>``: the ground truth in-betweening and+ motion editing hold. Returns the stats key used.+ """+ if object_type in self.cond_dict:+ raise ValueError(f"Object type {object_type!r} is already loaded.")+ if '-' in object_type:+ raise ValueError(f"Object type {object_type!r} must not contain '-'.")+ n_joints = len(cond_entry['parents'])+ if n_joints > self.max_joints:+ raise ValueError(+ f"{object_type!r} has {n_joints} joints; the model is padded to "+ f"max_joints={self.max_joints}.")+ if n_joints < MIN_COND_JOINTS:+ raise ValueError(f"{object_type!r} has {n_joints} joint(s); a skeleton needs "+ f"at least {MIN_COND_JOINTS}.")+ tpos = np.asarray(cond_entry['tpos_first_frame'], dtype=np.float64)+ extent = float(np.ptp(tpos, axis=0).max()) or 1.0+ root_xz = float(np.abs(tpos[0, [0, 2]]).max())+ if root_xz > ROOT_XZ_TOL * extent:+ # The root stats of the X / Z channels sit at the std floor+ # (every training T-pose has its root at the origin), so an+ # off-centre root would reach the model hugely amplified.+ raise ValueError(+ f"{object_type!r}: the T-pose root is at XZ "+ f"{tpos[0, [0, 2]].round(6).tolist()}, not the origin: not a stage-4 "+ f"(canonical) cond entry.")+ depth = int(np.max(cond_entry['joint_depths']))+ if depth > self.max_depth:+ logger.warning(f"{object_type!r}: joint depth {depth} exceeds the model's "+ f"max_depth={self.max_depth}; deeper joints share its embedding.")+ stats_key = self.resolve_stats_key(dataset_type)+ if root_xz > 0:+ cond_entry = dict(cond_entry)+ cond_entry['tpos_first_frame'] = tpos.copy()+ cond_entry['tpos_first_frame'][0, [0, 2]] = 0.0++ names = [n for n in model_joint_names(cond_entry, object_type)+ if n not in self._joint_name_emb]+ if names:+ encoded = self._batch_encode_unique(names, f'{object_type} joint names')+ self._joint_name_emb.update(+ {n: e if e.ndim == 1 else pool_tokens(e) for n, e in encoded.items()})+ if self.text_encoder is not None and self._owns_text_encoder:+ del self.text_encoder+ self.text_encoder = None+ self._owns_text_encoder = False+ if torch.cuda.is_available():+ torch.cuda.empty_cache()+ meta = self._precompute_object_type_meta(object_type, cond_entry)++ # The rest pose as a two-frame still clip: T-pose positions, T-pose+ # local rotations (identity after the rebase), canonical facing.+ parents = meta['condition']['parents']+ positions = np.repeat(meta['condition']['tpos'][None], 2, axis=0)+ local_rotations = np.repeat(meta['tpos_local_rotations'][None], 2, axis=0)+ rebased = compute_rots_from_tpos(+ Quaternions(local_rotations.copy()), Quaternions(local_rotations), parents)+ facing = Quaternions(np.tile([1.0, 0.0, 0.0, 0.0], (2, 1)))+ motion_feats = compute_unimate_motion_feats(positions, rebased, parents, facing)+ valid, reason = _validate_motion(motion_feats, parents=parents,+ offsets=meta['offsets'],+ gt_global_positions=positions)+ if not valid:+ raise ValueError(f"{object_type!r}: rest-pose features invalid ({reason}).")++ entry = {'motion': motion_feats, 'offsets': meta['offsets'],+ 'object_type': object_type, 'joint_names_emb': meta['joint_names_emb'],+ 'dataset_type': stats_key, 'caption': ''}+ entry.update(meta['condition'])+ rest_clip = f'{object_type}-rest_pose.npz'+ _check_new_clip_key(self.train_motion_dict, rest_clip, dataset_type)+ clips = {}+ for fname in clip_files:+ key = f'{clip_key_prefix}{fname}'+ clip = self._load_motion_file(motion_dir, fname, stats_key, meta, {})+ if clip is None:+ continue+ _check_new_clip_key(self.train_motion_dict, key, dataset_type)+ clips[key] = clip+ self.cond_dict[object_type] = cond_entry+ self.cond_only_object_types.add(object_type)+ self.cond_object_clips[object_type] = sorted(clips)+ self.train_motion_dict[rest_clip] = entry+ self.train_motion_dict.update(clips)+ self.train_object_motions_map[object_type] = [rest_clip] + sorted(clips)+ self.selected_object_types = sorted(set(self.selected_object_types) | {object_type})+ logger.info(f"Registered {object_type!r} from its cond entry ({n_joints} joints, "+ f"{dataset_type!r} -> stats {stats_key!r}"+ + (f", {len(clips)}/{len(clip_files)} clip(s)" if clip_files else '')+ + ").")+ return stats_key+def _load_multi_topology(self, dataset_info, dataset_type, motion_dict,selected_object_types, name_list):- """Loader for multi-topology datasets (truebones, objaverse)."""+ """Loader for multi-topology datasets (truebones, objaverse, general)."""cond_dict = dataset_info['cond_dict']motion_dir = dataset_info['motion_dir']captions = dataset_info.get('captions', {})@@ -711,6 +925,7 @@ class MotionDataset(data.Dataset):# create_sample_condition picks one at random anyway.object_motions = object_motions[:3]+ self._check_object_type_unique(object_type, dataset_type)self.cond_dict[object_type] = cond_dict[object_type]meta = self._precompute_object_type_meta(object_type, cond_dict[object_type],@@ -724,6 +939,7 @@ class MotionDataset(data.Dataset):)if entry is None:continue+ _check_new_clip_key(motion_dict, motion_name, dataset_type)motion_dict[motion_name] = entryselected_object_types.add(object_type)name_list.append(motion_name)@@ -795,6 +1011,7 @@ class MotionDataset(data.Dataset):# Register only after confirming clips exist, so empty types don't# leak into ``self.cond_dict`` / ``_compute_max_joints_and_depth``.+ self._check_object_type_unique(object_type, dataset_type)self.cond_dict[object_type] = cond_datameta = self._precompute_object_type_meta(object_type, cond_data)@@ -809,6 +1026,7 @@ class MotionDataset(data.Dataset):if entry is None:continueprefixed_name = f'{object_type}_{motion_name}'+ _check_new_clip_key(motion_dict, prefixed_name, dataset_type)motion_dict[prefixed_name] = entryselected_object_types.add(object_type)name_list.append(prefixed_name)@@ -906,9 +1124,21 @@ class MotionDataset(data.Dataset):'mean': mean,'std': std,}- results['caption'] = data['caption']- results['caption_emb'] = data['caption_emb']- results['caption_tokens'] = data['caption_tokens']+ # Up to three captions per clip: version v with probability+ # caption_probs[v], else the normal one; a clip without the drawn+ # version keeps its normal caption.+ key = 'caption'+ if self.caption_probs:+ u, acc = random.random(), 0.0+ for v, p in self.caption_probs.items():+ acc += p+ if u < acc:+ if f'caption_{v}_emb' in data:+ key = f'caption_{v}'+ break+ results['caption'] = data[key]+ results['caption_emb'] = data[f'{key}_emb']+ results['caption_tokens'] = data[f'{key}_tokens']return results@@ -999,9 +1229,11 @@ class MotionDataset(data.Dataset):f'max_depths={max_depths_by_dataset[dt]}')logger.info(f'Data max_joints={data_max_joints} | max_depths={data_max_depths}')- # Headroom: joint-addition aug can add 1 joint.+ # Headroom: joint-addition aug inserts 1 joint between a joint and its+ # parent, which also pushes that joint's subtree one level deeper.if self.use_addition_aug:data_max_joints += 1+ data_max_depths += 1if self.config_max_depth > 0 and data_max_depths > self.config_max_depth:logger.info(f'Clamping max_depths from {data_max_depths} to {self.config_max_depth}')@@ -1178,6 +1410,7 @@ class MotionDataset(data.Dataset):if self._stats_path is not None:logger.info(f"Loading saved dataset stats from {self._stats_path}")self.dataset_stats = np.load(self._stats_path, allow_pickle=True).item()+ self._check_loaded_stats()returnmode = "object-type balanced" if self.balanced_stats else "frame-weighted"@@ -1211,9 +1444,15 @@ class MotionDataset(data.Dataset):assert np.all(std_root > 0) and np.all(std_local > 0)+ # ``pool`` and the two flags record how the arrays were made, so a+ # run that loads this file can tell a global pool from a+ # per-dataset one (see ``_check_loaded_stats``).stats = {'mean_root': mean_root, 'std_root': std_root,'mean_local': mean_local, 'std_local': std_local,+ 'pool': label,+ 'balanced_stats': bool(self.balanced_stats),+ 'tie_std': bool(self.tie_std),}# Global mode broadcasts the single result across all dataset_type# keys so downstream lookups remain uniform.@@ -1234,6 +1473,93 @@ class MotionDataset(data.Dataset):logger.info("Dataset statistics calculation finished.")+ def _check_loaded_stats(self):+ """Validate stats loaded from disk and reconcile them with the+ datasets loaded now.++ Each entry records its ``pool`` (``'<all>'`` for the global pool, else+ its dataset type). A single entry is one pool either way, so it counts+ as global when tagged ``'<all>'`` or when the config sets+ ``use_dataset_stats=false``. In an untagged file, identical entries+ count as global and differing ones as per-dataset. A global file is applied to dataset types it has+ no key for (a fine-tune that adds a dataset normalizes it like the+ rest); a per-dataset file missing a dataset that has clips raises here+ rather than as a ``KeyError`` mid-training.+ """+ path = self._stats_path+ stats = self.dataset_stats+ arrays = ('mean_root', 'std_root', 'mean_local', 'std_local')+ if not isinstance(stats, dict) or not stats:+ raise ValueError(f"{path} holds no normalization stats.")+ for d, e in stats.items():+ if not isinstance(e, dict) or any(k not in e for k in arrays):+ raise ValueError(f"{path}: entry {d!r} lacks one of {arrays}.")+ bad = [k for k in arrays if np.shape(e[k]) != (self.feature_len,)]+ if bad:+ raise ValueError(+ f"{path}: entry {d!r} has {bad} of shape "+ f"{np.shape(e[bad[0]])}, expected ({self.feature_len},)."+ )++ entries = list(stats.values())+ if len(entries) == 1:+ is_global = (entries[0].get('pool') == '<all>'+ or not self.use_dataset_stats)+ elif all('pool' in e for e in entries):+ is_global = all(e['pool'] == '<all>' for e in entries)+ else:+ is_global = all(+ all(np.array_equal(e[k], entries[0][k]) for k in arrays)+ for e in entries[1:]+ )++ if not self.use_dataset_stats and not is_global:+ raise ValueError(+ f"{path} holds per-dataset stats ({sorted(stats)}), but the "+ f"config sets use_dataset_stats=false."+ )+ # The file wins over the config; adopting its settings lets+ # create_dataset write them back, so the run's config.json describes+ # the normalization it actually used.+ if self.use_dataset_stats and is_global:+ logger.warning(+ f"{path} holds one global stats pool, but the config sets "+ f"use_dataset_stats=true; using the file's global pool."+ )+ self.use_dataset_stats = False+ for flag in ('balanced_stats', 'tie_std'):+ saved = {bool(e[flag]) for e in entries if flag in e}+ if len(saved) == 1 and saved != {bool(getattr(self, flag))}:+ logger.warning(+ f"{path} was computed with {flag}={next(iter(saved))}, the "+ f"config sets {flag}={getattr(self, flag)}; using the file."+ )+ setattr(self, flag, next(iter(saved)))++ missing = sorted(set(self.dataset_object_count) - set(stats))+ if not missing:+ return+ if is_global:+ for d in missing:+ stats[d] = entries[0]+ logger.info(f"Applied the global stats of {path} to {missing}.")+ return+ # The general dataset is processed like objaverse: a per-dataset file+ # without a general entry normalizes it with the objaverse entry, so+ # general data can be added for a fine-tune.+ for d, like in STATS_FALLBACK.items():+ if d in missing and like in stats:+ stats[d] = stats[like]+ missing.remove(d)+ logger.info(f"{path} has no {d!r} stats; using its {like!r} entry for them.")+ with_clips = [d for d in missing+ if sum(self.dataset_object_count[d].values()) > 0]+ if with_clips:+ raise ValueError(+ f"{path} holds per-dataset stats for {sorted(stats)} and none "+ f"for {with_clips}, whose clips could not be normalized."+ )+def _stats_frame_weighted(self, d_type=None):"""Pool all frames equally. ``d_type=None`` pools across all dataset types."""sum_root = np.zeros(self.feature_len)@@ -1351,13 +1677,67 @@ class MotionDataset(data.Dataset):def save_dataset_stats(self, path):"""Persist ``self.dataset_stats`` as a pickled .npy. Inferencereloads with ``np.load(path, allow_pickle=True).item()``.++ Written to a temp file and renamed: a resumed run rewrites the file it+ loaded its stats from while other ranks may still be reading it."""if os.path.dirname(path):os.makedirs(os.path.dirname(path), exist_ok=True)- np.save(path, self.dataset_stats, allow_pickle=True)+ tmp = f"{path}.tmp"+ with open(tmp, 'wb') as f: # a file object keeps np.save from appending .npy+ np.save(f, self.dataset_stats, allow_pickle=True)+ os.replace(tmp, path)logger.info(f"Saved dataset stats to {path}")+class SkeletonDataset(data.Dataset):+ """A :class:`MotionDataset` that loads no feature directory, for sampling.++ It holds a run's normalization stats (``stats_path``, required) and padded+ widths, and nothing else until skeletons are added with+ ``motion_dataset.add_cond_object`` (each with its rest pose and, for+ in-betweening / motion editing, the clips it is given). Sampling then needs+ no dataset on disk beyond what the skeletons themselves come from.+ ``create_sample_condition`` takes it like a :class:`Mixture`.+ """++ def __init__(self, stats_path: str, max_motion_length: int, max_joints: int,+ max_depth: int, topology_condition_type: str = 'tpos',+ feature_len: int = 12, text_encoder_type: str = "t5",+ text_encoder_version: str = "t5-base", use_dataset_stats: bool = True,+ balanced_stats: bool = False, tie_std: bool = False, max_freqs: int = 8,+ ground_motion_height: bool = True, realign_feature: bool = True):+ if not stats_path or not os.path.isfile(stats_path):+ raise FileNotFoundError(f"Normalization stats {stats_path!r} not found.")+ self.motion_dataset = MotionDataset(+ data_dict={},+ topology_condition_type=topology_condition_type,+ max_motion_length=max_motion_length,+ config_max_depth=max_depth,+ feature_len=feature_len,+ text_encoder_type=text_encoder_type,+ text_encoder_version=text_encoder_version,+ use_dataset_stats=use_dataset_stats,+ balanced_stats=balanced_stats,+ tie_std=tie_std,+ max_freqs=max_freqs,+ ground_motion_height=ground_motion_height,+ realign_feature=realign_feature,+ prebuilt_max_joints=max_joints,+ prebuilt_max_depth=max_depth,+ target_object_types=set(),+ stats_path=stats_path,+ )+ self.max_joints = self.motion_dataset.max_joints+ self.max_depth = self.motion_dataset.max_depth++ def __getitem__(self, item):+ return self.motion_dataset[item]++ def __len__(self):+ return len(self.motion_dataset)++# ---------------------------------------------------------------------------# MixtureSampler โ balanced sampling across skeleton types# ---------------------------------------------------------------------------@@ -1374,18 +1754,24 @@ class MixtureSampler(data.Sampler):alpha=1.0 : uniform per type (ignores motion count entirely)dataset_alpha=0.0 : each dataset keeps its natural share of the clipsdataset_alpha=1.0 : equal probability per dataset- dataset_alpha=None: single level (default, the behaviour before 2026-09-27) โ+ dataset_weights={d: w}: explicit first level instead โ dataset d gets+ total probability w_d / sum(w) whatever its size (a dataset with no+ training clips drops out and the rest are renormalized).+ dataset_alpha=None: single level (default) โobject types pooled across all datasets. A dataset of one rig (Mixamo)- then counts as one type and falls below 1% of a UniML3D mixture; the- *_v2 configs set 0.25.+ then counts as one type and falls below 1% of a UniML3D mixture.With one dataset both levels coincide with the legacy behaviour.This is a plain Sampler (not DistributedSampler) that yields all indices.Accelerate's BatchSamplerShard handles per-rank sharding automatically.+ Each epoch draws with a generator seeded from ``seed`` and the epoch โ the+ same on every rank, so the shards stay disjoint โ so runs with different+ seeds see different draw orders (seed 0 gives the epoch-only order)."""- def __init__(self, data_source, alpha: float = 0.5, dataset_alpha=None):+ def __init__(self, data_source, alpha: float = 0.5, dataset_alpha=None,+ dataset_weights=None, seed: int = 0):super().__init__(data_source)name_list = data_source.motion_dataset.train_name_list@@ -1403,7 +1789,17 @@ class MixtureSampler(data.Sampler):objects_by_dataset[key[0]].append(key)weights = np.zeros(total_samples)- if dataset_alpha is None:+ if dataset_alpha is not None and dataset_weights is not None:+ raise ValueError("Pass dataset_alpha or dataset_weights, not both")+ if dataset_weights is not None:+ missing = sorted(set(objects_by_dataset) - set(dataset_weights))+ if missing:+ raise ValueError(f"dataset_weights has no weight for {missing}")+ absent = sorted(set(dataset_weights) - set(objects_by_dataset))+ if absent:+ logger.warning(f'MixtureSampler: no training clips in {absent}; '+ f'their weight is dropped and the rest renormalized')+ if dataset_alpha is None and dataset_weights is None:# w_i = n_k^(-alpha) for sample i in type k, giving type k total# probability โ n_k * n_k^(-alpha) = n_k^(1-alpha).for indices in indices_by_object.values():@@ -1411,7 +1807,8 @@ class MixtureSampler(data.Sampler):else:for d_type, keys in objects_by_dataset.items():n_d = sum(len(indices_by_object[k]) for k in keys)- d_share = n_d ** (1 - dataset_alpha)+ d_share = (float(dataset_weights[d_type]) if dataset_weights is not None+ else n_d ** (1 - dataset_alpha))type_mass = {k: len(indices_by_object[k]) ** (1 - alpha) for k in keys}z = sum(type_mass.values())for k in keys:@@ -1421,7 +1818,9 @@ class MixtureSampler(data.Sampler):weights /= weights.sum()dataset_probs = {d: sum(weights[indices_by_object[k]].sum() for k in keys)for d, keys in objects_by_dataset.items()}- logger.info(f'MixtureSampler (alpha={alpha}, dataset_alpha={dataset_alpha}): '+ level1 = (f'dataset_weights={dataset_weights}' if dataset_weights is not None+ else f'dataset_alpha={dataset_alpha}')+ logger.info(f'MixtureSampler (alpha={alpha}, {level1}): '+ ', '.join(f'{d} n={sum(len(indices_by_object[k]) for k in keys)} 'f'p={dataset_probs[d]:.3f}'for d, keys in objects_by_dataset.items()))@@ -1440,6 +1839,7 @@ class MixtureSampler(data.Sampler):self.weights = torch.as_tensor(weights, dtype=torch.double)self.total_samples = total_samples+ self.seed = seedself.epoch = 0def set_epoch(self, epoch):@@ -1447,7 +1847,9 @@ class MixtureSampler(data.Sampler):def __iter__(self):g = torch.Generator()- g.manual_seed(self.epoch)+ # Spread seeds apart so seed s at epoch e never replays seed s + 1;+ # wrapped to 63 bits, which manual_seed accepts for any seed.+ g.manual_seed((self.seed * 1_000_003 + self.epoch) % (1 << 63))indices = torch.multinomial(self.weights, self.total_samples, replacement=True, generator=g).tolist()@@ -1493,11 +1895,22 @@ class Mixture(data.Dataset):target_object_types: Optional[Set[str]] = None,target_clip_stems: Optional[Set[str]] = None,stats_path: Optional[str] = None,+ generic_caption_prob: float = 0.0,+ detail_caption_prob: float = 0.0,):logger.info(f'Initializing Mixture Dataset with datasets: {list(data_configs.keys())}')+ # Extra caption versions are only read when a training run samples them.+ training = target_object_types is None+ probs = {'generic': generic_caption_prob if training else 0.0,+ 'detail': detail_caption_prob if training else 0.0}+ # A saved (padded) width carries one joint of joint-addition headroom;+ # the skeleton filter is the width the training data had.+ filter_max_joints = (prebuilt_max_joints - 1 if prebuilt_max_joints > 0 and use_addition_aug+ else max_joints)data_dict = {dataset_config.type: self._build_dataset_entry(- dataset_config, min_joints, max_joints,+ dataset_config, min_joints, filter_max_joints,+ tuple(v for v, p in probs.items() if p > 0),)for _, dataset_config in data_configs.items()}@@ -1528,6 +1941,8 @@ class Mixture(data.Dataset):target_object_types=target_object_types,target_clip_stems=target_clip_stems,stats_path=stats_path,+ generic_caption_prob=probs['generic'],+ detail_caption_prob=probs['detail'],)if target_object_types is None:# Training only: inference can produce a tiny (even empty) split.@@ -1541,9 +1956,12 @@ class Mixture(data.Dataset):# ------------------------------------------------------------------@staticmethod- def _build_dataset_entry(dataset_config, min_joints, max_joints):+ def _build_dataset_entry(dataset_config, min_joints, max_joints,+ caption_versions=()):"""Build the ``data_dict`` entry for one dataset: load cond / captions- / test_objects, apply dataset-specific and joint-count filters.+ (+ ``captions_<v>.json`` for each extra version in+ ``caption_versions``) / test_objects, apply dataset-specific and+ joint-count filters."""root_dir = dataset_config.pathdataset_type = dataset_config.type@@ -1552,6 +1970,9 @@ class Mixture(data.Dataset):cond_dict = Mixture._load_cond_dict(root_dir)captions = Mixture._load_captions(root_dir, dataset_type)+ extra = {f'captions_{v}': Mixture._load_captions(root_dir, dataset_type,+ f'captions_{v}.json')+ for v in caption_versions}test_objects = Mixture._load_test_objects(root_dir, dataset_type)cond_dict = Mixture._apply_dataset_filters(cond_dict, dataset_config, root_dir)@@ -1562,6 +1983,7 @@ class Mixture(data.Dataset):'motion_dir': motion_dir,'cond_dict': cond_dict,'captions': captions,+ **extra,'test_objects': test_objects,}@@ -1572,16 +1994,20 @@ class Mixture(data.Dataset):return np.load(cond_file, allow_pickle=True).item()@staticmethod- def _load_captions(root_dir, dataset_type):- """Load the flat ``{clip_stem: caption}`` dict from ``captions.json``.-- The feature-extraction stage writes it whenever any clip has a- caption; a missing file means the dataset is caption-free.+ def _load_captions(root_dir, dataset_type, filename='captions.json'):+ """Load a flat ``{clip_stem: caption}`` dict (``captions.json``,+ ``captions_generic.json`` or ``captions_detail.json``).++ The feature-extraction stage writes them whenever any clip has a+ caption; a missing ``captions.json`` means the dataset is caption-free,+ a missing ``captions_<v>.json`` that its clips have only the normal+ caption (and the other versions present)."""- captions_file = pjoin(root_dir, 'captions.json')+ captions_file = pjoin(root_dir, filename)if not os.path.exists(captions_file):- logger.info(f'{dataset_type.capitalize()}: no captions.json in '- f'{root_dir} (caption-free dataset)')+ logger.info(f'{dataset_type.capitalize()}: no {filename} in '+ f'{root_dir}' + (' (caption-free dataset)'+ if filename == 'captions.json' else ''))return {}with open(captions_file) as f:captions = json.load(f)@@ -1614,7 +2040,7 @@ class Mixture(data.Dataset):"""Dispatch to per-dataset filter (subset / object filter / count cap)."""if dataset_config.type == 'truebones':return Mixture._filter_truebones(cond_dict, dataset_config, root_dir)- if dataset_config.type == 'objaverse':+ if dataset_config.type in ('objaverse', 'general'):return Mixture._filter_objaverse(cond_dict, dataset_config, root_dir)return cond_dict@@ -1641,7 +2067,8 @@ class Mixture(data.Dataset):@staticmethoddef _filter_objaverse(cond_dict, dataset_config, root_dir):- """Apply objaverse-specific filters: filter_object + objects_num cap."""+ """Apply the objaverse / general filters: filter_object + objects_num cap."""+ label = dataset_config.type.capitalize()if dataset_config.filter_object:filter_path = pjoin(root_dir, 'filtered_objects.txt')if os.path.exists(filter_path):@@ -1650,19 +2077,19 @@ class Mixture(data.Dataset):before = len(cond_dict)cond_dict = {k: v for k, v in cond_dict.items() if k not in excluded}logger.info(- f'Objaverse: filter_object removed {before - len(cond_dict)} '+ f'{label}: filter_object removed {before - len(cond_dict)} 'f'of {before} object types via {filter_path}')else:logger.info(- f'Objaverse: filter_object=True but no filtered_objects.txt '+ f'{label}: filter_object=True but no filtered_objects.txt 'f'at {filter_path}; skipping filtering.')objects_num = dataset_config.objects_numif objects_num > 0:keys = sorted(cond_dict.keys())[:objects_num]cond_dict = {k: cond_dict[k] for k in keys}- logger.info(f'Objaverse Dataset has {len(cond_dict)} object types')+ logger.info(f'{label} Dataset has {len(cond_dict)} object types')return cond_dict@staticmethoddiff --git a/unimate/inference/assets.py b/unimate/inference/assets.pynew file mode 100644index 0000000..5c855eb--- /dev/null+++ b/unimate/inference/assets.py@@ -0,0 +1,287 @@+"""Skeletons to sample on, named by reference.++An *asset* is what sampling and mesh driving need about one skeleton: its+stage-4 cond entry (topology, T-pose, joint names), the dataset whose+normalization stats apply to it, its canonical rest-pose GLB when there is+one, and where its feature clips live. A dataset object type and a custom rig+processed by ``data_process.rig_preprocess`` are the same kind of asset; only+where the files sit differs.++A reference is one of:++- a directory holding ``cond.npy`` with one entry: a ``rig_preprocess``+ output (``summary.json`` beside it names the stats dataset; ``<name>.glb``+ is its canonical GLB; ``motions/`` its clips, written by ``--save_clips``);+- a ``.npy`` cond file with exactly one entry;+- ``<dataset>:<object_type>``: an object type of a dataset (the run's feature+ directory for it, else ``dataset/features/<dataset>``);+- a bare ``<object_type>``: the assets given explicitly (``--asset``) first,+ then the entries of the ``--cond_path`` files, then the run's datasets in+ ``dataset_list`` order.+"""++import json+import os+from dataclasses import dataclass, field+from typing import Dict, Iterable, List, Optional++import numpy as np++from data_process.utils.asset_files import DEFAULT_MIXAMO_CHARACTER, default_assets_dir+from unimate.utils.logger import get_logger++logger = get_logger(file_name=__file__)++# Datasets a feature directory or a stats entry can be named after.+DATASET_TYPES = ('truebones', 'mixamo', 'objaverse', 'general')+# Normalization of an asset whose dataset nothing tells.+DEFAULT_STATS_DATASET = 'objaverse'+# The Mixamo object type drives DEFAULT_MIXAMO_CHARACTER's canonical GLB by default.+MIXAMO_OBJECT_TYPE = 'mixamo'+++@dataclass+class Asset:+ """One skeleton: see the module docstring."""+ name: str+ cond: dict = field(repr=False)+ stats_dataset: str+ cond_path: str+ source: str+ dataset: Optional[str] = None+ canonical_glb: Optional[str] = None+ motion_dir: Optional[str] = None+ joint_cache_dir: Optional[str] = None++ @property+ def joint_names(self) -> List[str]:+ return [str(n) for n in self.cond['joint_names']]++ def clip_files(self, clip_ids: Optional[Iterable[str]] = None) -> List[str]:+ """Feature-clip file names of this asset in ``motion_dir`` (all, or+ those whose stem matches one of *clip_ids*: the clip id, or+ ``<name>-<id>`` / ``<name>_<id>``)."""+ if not self.motion_dir or not os.path.isdir(self.motion_dir):+ return []+ single = self.dataset == MIXAMO_OBJECT_TYPE+ names = sorted(f for f in os.listdir(self.motion_dir) if f.endswith('.npz')+ and (single or f.startswith(f'{self.name}-')))+ if clip_ids is None:+ return names+ wanted = set()+ for cid in clip_ids:+ if single and cid.startswith(self.clip_key_prefix):+ cid = cid[len(self.clip_key_prefix):] # the loader's key spelling+ wanted |= {cid, f'{self.name}-{cid}', f'{self.name}_{cid}'}+ return [f for f in names if os.path.splitext(f)[0] in wanted]++ @property+ def clip_key_prefix(self) -> str:+ """Mixamo clips carry no object-type prefix on disk; the loader keys+ them ``mixamo_<file>``."""+ return f'{MIXAMO_OBJECT_TYPE}_' if self.dataset == MIXAMO_OBJECT_TYPE else ''++ def manifest_entry(self) -> dict:+ return {+ 'source': self.source,+ 'dataset': self.dataset,+ 'cond_path': _display_path(self.cond_path),+ 'cond_key': self.name,+ 'canonical_glb': _display_path(self.canonical_glb) if self.canonical_glb else None,+ 'stats_dataset': self.stats_dataset,+ 'joint_names': self.joint_names,+ }+++def _display_path(path: str) -> str:+ """*path* relative to the working directory when it lies under it (runs+ start from the repo root), else absolute."""+ absolute = os.path.abspath(path)+ rel = os.path.relpath(absolute)+ return absolute if rel.startswith('..') else rel+++def _load_cond_file(path: str) -> Dict[str, dict]:+ conds = np.load(path, allow_pickle=True).item()+ if not isinstance(conds, dict):+ raise ValueError(f"{path} is not a {{object_type: cond}} dict.")+ return conds+++def _summary(folder: str) -> Optional[dict]:+ """A ``rig_preprocess`` output's ``summary.json`` in *folder*, or None."""+ path = os.path.join(folder, 'summary.json')+ if not os.path.isfile(path):+ return None+ with open(path) as f:+ return json.load(f)+++def feature_dir_dataset(path: str) -> Optional[str]:+ """The dataset a cond file's directory is the feature directory of+ (``.../<dataset>/cond.npy`` with no ``rig_preprocess`` summary beside it),+ or None."""+ folder = os.path.dirname(os.path.abspath(path))+ name = os.path.basename(os.path.realpath(folder))+ if name in DATASET_TYPES and _summary(folder) is None:+ return name+ return None+++def cond_file_dataset_type(path: str) -> str:+ """Dataset whose stats normalize the entries of a cond file: the+ ``stats_dataset`` / ``reference`` / ``profile`` of a ``rig_preprocess``+ ``summary.json`` beside it, else its feature directory's name, else+ ``DEFAULT_STATS_DATASET``."""+ info = _summary(os.path.dirname(os.path.abspath(path)))+ if info is not None:+ for key in ('stats_dataset', 'reference', 'profile'):+ if info.get(key) in DATASET_TYPES:+ return info[key]+ return DEFAULT_STATS_DATASET+ return feature_dir_dataset(path) or DEFAULT_STATS_DATASET+++class AssetResolver:+ """Resolve references (module docstring) against a run's datasets.++ *dataset_config* is the run's ``config.dataset``; *assets* are references+ given explicitly (resolved now, addressable by name afterwards);+ *cond_paths* are cond files whose entries are addressable by name;+ *cond_dataset_type* overrides the stats dataset of every asset that is not+ a dataset object type.+ """++ def __init__(self, dataset_config, assets: Iterable[str] = (),+ cond_paths: Iterable[str] = (), cond_dataset_type: Optional[str] = None):+ self._dataset_dirs = {cfg.type: cfg.path+ for cfg in (dataset_config.data_configs or {}).values()}+ self._cond_dataset_type = cond_dataset_type+ self._conds: Dict[str, Dict[str, dict]] = {}+ self._cond_paths = list(cond_paths)+ for path in self._cond_paths:+ if not os.path.isfile(path):+ raise FileNotFoundError(f"--cond_path {path} not found.")+ self.explicit: Dict[str, Asset] = {}+ for ref in assets:+ asset = self.resolve(ref)+ other = self.explicit.get(asset.name)+ if other is not None and other.source != asset.source:+ raise ValueError(+ f"Two assets are named {asset.name!r} ({other.source}, {asset.source}); "+ f"sample them in separate runs (or rename a custom one with "+ f"rig_preprocess --name).")+ self.explicit[asset.name] = asset++ # -- files ---------------------------------------------------------------++ def _cond(self, path: str) -> Dict[str, dict]:+ key = os.path.abspath(path)+ if key not in self._conds:+ self._conds[key] = _load_cond_file(path)+ return self._conds[key]++ def features_dir(self, dataset: str) -> str:+ return self._dataset_dirs.get(dataset) or os.path.join('dataset', 'features', dataset)++ # -- references ----------------------------------------------------------++ def resolve(self, ref: str) -> Asset:+ """The asset *ref* names; ``KeyError`` when nothing does, ``ValueError``+ when the reference is malformed."""+ ref = str(ref).strip()+ if not ref:+ raise ValueError("Empty asset reference.")+ if os.path.isdir(ref):+ return self._bundle(ref)+ if ref.endswith('.npy') and os.path.isfile(ref):+ return self._single_cond_file(ref)+ dataset, sep, name = ref.partition(':')+ if sep and dataset in DATASET_TYPES and not os.path.exists(ref):+ return self.dataset_asset(dataset, name)+ if os.sep in ref or ref.endswith('.npy'):+ raise KeyError(f"Asset {ref!r}: no such directory or cond file.")+ return self._by_name(ref)++ def dataset_asset(self, dataset: str, name: str, cond: Optional[dict] = None,+ root: Optional[str] = None) -> Asset:+ """Object type *name* of *dataset*'s feature directory (*root*, default+ :meth:`features_dir`; *cond*: its entry when already loaded)."""+ root = root or self.features_dir(dataset)+ cond_path = os.path.join(root, 'cond.npy')+ if cond is None:+ if not os.path.isfile(cond_path):+ raise KeyError(f"Asset '{dataset}:{name}': no {cond_path}.")+ conds = self._cond(cond_path)+ if name not in conds:+ raise KeyError(f"Asset '{dataset}:{name}': not an object type of {cond_path}.")+ cond = conds[name]+ glb_name = DEFAULT_MIXAMO_CHARACTER if dataset == MIXAMO_OBJECT_TYPE else name+ glb = os.path.join(default_assets_dir(root), f'{glb_name}.glb')+ return Asset(name=name, cond=cond, stats_dataset=dataset, cond_path=cond_path,+ source=f'{dataset}:{name}', dataset=dataset,+ canonical_glb=glb if os.path.isfile(glb) else None,+ motion_dir=os.path.join(root, 'motions'), joint_cache_dir=root)++ def _bundle(self, path: str) -> Asset:+ cond_path = os.path.join(path, 'cond.npy')+ if not os.path.isfile(cond_path):+ raise KeyError(f"Asset {path!r}: a directory without cond.npy (expected a "+ f"data_process.rig_preprocess output).")+ asset = self._single_cond_file(cond_path)+ asset.source = _display_path(path)+ motions = os.path.join(path, 'motions')+ asset.motion_dir = motions if os.path.isdir(motions) else None+ return asset++ def _single_cond_file(self, path: str) -> Asset:+ conds = self._cond(path)+ if len(conds) != 1:+ raise ValueError(+ f"{path} holds {len(conds)} entries; pass it with --cond_path and name the "+ f"object type in the test cases, or reference '<dataset>:<object_type>'.")+ (name, cond), = conds.items()+ dataset = feature_dir_dataset(path)+ if dataset:+ return self.dataset_asset(dataset, name, cond, root=os.path.dirname(path) or '.')+ glb = os.path.join(os.path.dirname(os.path.abspath(path)), f'{name}.glb')+ return Asset(name=name, cond=cond,+ stats_dataset=self._cond_dataset_type or cond_file_dataset_type(path),+ cond_path=path, source=_display_path(path),+ canonical_glb=glb if os.path.isfile(glb) else None)++ def _by_name(self, name: str) -> Asset:+ if name in self.explicit:+ shadowed = self._dataset_with(name)+ if shadowed and shadowed != self.explicit[name].dataset:+ logger.warning(+ f"{name!r} is both the given asset {self.explicit[name].source} and an "+ f"object type of {shadowed}; using the given asset "+ f"('{shadowed}:{name}' names the dataset's).")+ return self.explicit[name]+ for path in self._cond_paths:+ conds = self._cond(path)+ if name in conds:+ ds = feature_dir_dataset(path)+ if ds:+ return self.dataset_asset(ds, name, conds[name],+ root=os.path.dirname(path) or '.')+ glb = os.path.join(os.path.dirname(os.path.abspath(path)), f'{name}.glb')+ return Asset(name=name, cond=conds[name],+ stats_dataset=self._cond_dataset_type or cond_file_dataset_type(path),+ cond_path=path, source=_display_path(path),+ canonical_glb=glb if os.path.isfile(glb) else None)+ dataset = self._dataset_with(name)+ if dataset:+ return self.dataset_asset(dataset, name)+ raise KeyError(+ f"Asset {name!r} is none of the given assets, not in the --cond_path files and "+ f"not an object type of the run's datasets ({', '.join(self._dataset_dirs) or 'none'}).")++ def _dataset_with(self, name: str) -> Optional[str]:+ """The first of the run's datasets whose cond.npy has object type *name*."""+ for dataset, root in self._dataset_dirs.items():+ cond_path = os.path.join(root, 'cond.npy')+ if os.path.isfile(cond_path) and name in self._cond(cond_path):+ return dataset+ return Nonediff --git a/unimate/inference/motion_editing.py b/unimate/inference/motion_editing.pyindex ec42b2d..229151c 100644--- a/unimate/inference/motion_editing.py+++ b/unimate/inference/motion_editing.py@@ -114,7 +114,8 @@ def build_joint_keep_mask(Logs a warning per sample for any ``keep_names`` entries that don'tmatch a joint in that skeleton โ a single keep list is typically appliedacross heterogeneous skeletons (different topology, different naming),- so partial matches are expected and shouldn't fail the run.+ so partial matches are expected and shouldn't fail the run; a batch in+ which nothing matches gets an all-False mask (sampled unconstrained).Args:joint_names_per_sample: per-sample lists of joint name strings,@@ -152,9 +153,10 @@ def build_joint_keep_mask()if total_hits == 0:- raise ValueError(+ logger.warning(f"build_joint_keep_mask: no keep_joints matched any skeleton in "- f"the batch (keep_names={keep_names}). Check spelling against the "- f"clip's joint_names or clean_joint_names."+ f"this batch (keep_names={keep_names}); its samples are generated "+ f"unconstrained. Check spelling against the clip's joint_names or "+ f"clean_joint_names.")return maskdiff --git a/unimate/inference/motion_expansion.py b/unimate/inference/motion_expansion.pyindex 1e27d41..59792d7 100644--- a/unimate/inference/motion_expansion.py+++ b/unimate/inference/motion_expansion.py@@ -15,14 +15,26 @@ so the seam frames appear once and motion continuity is enforced by theflow ODE's clamping (same machinery as in-betweening โ only the maskconstruction differs).+The model only ever saw windows whose frame 0 faces +Z (stage-4+canonicalization, ``realign_unimate_clip`` after a crop). The tail of a+segment that turned does not, so each seed is re-aligned to its own frame 0+before sampling, and the new segment is rotated back into the chain's frame+afterwards. RIFKE positions and local velocities live in the per-frame+facing frame, so only the rotation channels change and the trajectory stays+continuous across the seam.+Total length: ``max_T + (max_T - overlap) * (N - 1)``."""from typing import List+import numpy as npimport torch+from Quaternions import Quaternionsfrom unimate.utils.logger import get_logger+from unimate.utils.motion_utils import rotate_unimate_facing+from unimate.utils.rotation_conversions import matrix_to_quaternion_np, rotation_6d_to_matrix_npfrom unimate.inference.generate import generate_sampleslogger = get_logger(file_name=__file__)@@ -61,6 +73,46 @@ def seed_expansion_x1(return x1+# Normalized channels whose std is below this are constant by construction+# (root X/Z and the yaw-only facing's fixed 6-D entries; their std is floored+# at 1e-8). Facing rotations leave them unchanged, and renormalizing them+# would only amplify float error, so they keep their normalized values.+_CONST_STD = 1e-6+++def rotate_segment_facing(seq, cond, q_per_sample):+ """Apply :func:`rotate_unimate_facing` to every sample of a batch.++ Args:+ seq: normalized motion ``(B, J, D, T)``.+ cond: conditioning dict with ``mean`` / ``std`` ``(B, J, D)``,+ ``n_joints`` ``(B,)`` and ``parents`` (list of ``(J_i,)``).+ q_per_sample: one single-element ``Quaternions`` per sample.++ Returns:+ Normalized ``(B, J, D, T)`` motion with the rotated facing; padded+ joints are left untouched.+ """+ out = seq.clone()+ for b, q in enumerate(q_per_sample):+ nj = int(cond['n_joints'][b])+ mean = cond['mean'][b, :nj].double().cpu().numpy()[None] # (1, J, D)+ std = cond['std'][b, :nj].double().cpu().numpy()[None]+ norm = seq[b, :nj].double().cpu().numpy().transpose(2, 0, 1) # (T, J, D)+ rotated = rotate_unimate_facing(norm * std + mean, cond['parents'][b], q)+ renorm = np.where(std < _CONST_STD, norm, (rotated - mean) / std)+ out[b, :nj] = torch.from_numpy(renorm.transpose(1, 2, 0)).to(seq)+ return out+++def first_frame_facing(seq, cond):+ """Frame-0 root facing of each sample of a normalized ``(B, J, D, T)`` batch."""+ root_6d = seq[:, 0, 3:9, 0].double() * cond['std'][:, 0, 3:9].double() \+ + cond['mean'][:, 0, 3:9].double() # (B, 6)+ mats = rotation_6d_to_matrix_np(root_6d.cpu().numpy())+ return [Quaternions(matrix_to_quaternion_np(m[None]))[0] for m in mats]++# ---------------------------------------------------------------------------# Chain generation# ---------------------------------------------------------------------------@@ -111,8 +163,14 @@ def expand_motion_chain(if prev_seq is None:x1_known = Nonekeep_mask = None+ q_seed = Noneelse:- x1_known = seed_expansion_x1(prev_seq, overlap, motion_shape, device)+ # Seed in the frame the model was trained on: the tail's own+ # frame 0 faces +Z.+ tail = prev_seq[:, :, :, -overlap:]+ q_seed = first_frame_facing(tail, cond)+ tail = rotate_segment_facing(tail, cond, q_seed)+ x1_known = seed_expansion_x1(tail, overlap, motion_shape, device)keep_mask = build_overlap_keep_mask(motion_shape[0], overlap, max_T, device,)@@ -129,6 +187,10 @@ def expand_motion_chain(x1_known=x1_known,keep_mask=keep_mask,)+ if q_seed is not None:+ # Back into the chain's frame; the pinned overlap then reproduces+ # the previous segment's tail.+ sample = rotate_segment_facing(sample, cond, [-q for q in q_seed])segments.append(sample)prev_seq = samplelogger.info(diff --git a/unimate/inference/sample.py b/unimate/inference/sample.pyindex 4345c73..fc76754 100644--- a/unimate/inference/sample.py+++ b/unimate/inference/sample.py@@ -1,4 +1,12 @@-"""Inference entry point: generate motion samples from a trained model."""+"""Inference entry point: generate motion samples from a trained model.++The skeletons are named by reference (``unimate.inference.assets``): a+``data_process.rig_preprocess`` output, a cond file, or a dataset object type.+With test cases (``--test_cases_json``, ``--test_cases_txt``, or ``--asset``+with ``--prompt``) only those assets are read, so a run needs no dataset on+disk beyond them; without any, the dataset's own split is enumerated.+Next to the motions it writes ``manifest.json`` (see :class:`SampleManifest`).+"""import dataclassesimport gcimport glob@@ -19,13 +27,14 @@ warnings.filterwarnings("ignore")from unimate.configs.schema import MainConfigfrom unimate.dataset.conditioning import create_sample_condition-from unimate.dataset.factory import create_dataset+from unimate.dataset.factory import create_dataset, create_skeleton_datasetfrom unimate.inference.motion_inbetweening import build_keep_mask, parse_keep_framesfrom unimate.inference.motion_editing import (build_joint_keep_mask,get_joint_names,parse_keep_joints,)+from unimate.inference.assets import Asset, AssetResolverfrom unimate.inference.motion_expansion import expand_motion_chainfrom unimate.models.flow.transport import Samplerfrom unimate.models.factory import create_diffusion, create_model, create_transport@@ -58,12 +67,25 @@ class InferenceArgs:output_dir: Optional[str] = Nonenum_repetitions: int = 1cfg_scale: Optional[float] = None- # JSON of {"<object_type>-<case_id>": caption}; used when cfg_scale > 1.0.- # When None, every clip in the dataset is enumerated as a test case.+ # Test cases at cfg_scale > 1.0: a JSON map {"<asset>-<case_id>": prompt}+ # (<asset> a name or '<dataset>:<object_type>'; prompt lists for+ # --motion_expand; <case_id> is the pinned clip id for --inbetween /+ # --motion_edit), or a JSON list of {"asset": ref, "prompt": ...,+ # "id": case_id, "clip": clip_id} (ref also a rig_preprocess output+ # directory or a cond file). When neither this nor --prompt is given,+ # every clip of the dataset's eval split is enumerated as a test case.test_cases_json: Optional[str] = None- # Plain-text object_types (one per line) for unconditional sampling at- # cfg_scale == 1.0. '#' starts a comment.+ # Assets for unconditional sampling at cfg_scale == 1.0, one reference+ # per line. '#' starts a comment.test_cases_txt: Optional[str] = None+ # Assets by reference (unimate.inference.assets): a rig_preprocess output+ # directory, a one-entry cond.npy, '<dataset>:<object_type>' or a name.+ # With --prompt (or alone at cfg_scale 1.0) they are the test cases;+ # with --test_cases_json the test cases can name them.+ asset: Tuple[str, ...] = ()+ # Prompts sampled on every --asset, one test case each (for+ # --motion_expand: the segments of one chain per asset).+ prompt: Tuple[str, ...] = ()# Per-chunk inference batch size โ caps GPU memory regardless of total count.batch_size: int = 64# Skip mp4/PNG renders; write only the .npy motion features.@@ -99,7 +121,8 @@ class InferenceArgs:gt_start_frame: Optional[int] = None# Motion expansion: chain multiple text-conditioned generations into one# long motion. Requires --test_cases_json whose values are *lists* of- # prompts (one segment per prompt). Each segment after the first pins+ # prompts (one segment per prompt), or --asset with --prompt (the+ # segments, one chain per asset). Each segment after the first pins# its first `expand_overlap` frames to the previous segment's last# `expand_overlap` frames via replacement-style sampling. Mutually# exclusive with --inbetween / --motion_edit.@@ -107,6 +130,15 @@ class InferenceArgs:# Number of frames overlapped (clamped) between consecutive expansion# segments. Must satisfy 0 < expand_overlap < max_motion_length.expand_overlap: int = 10+ # cond.npy files of {object_type: cond} (a feature directory's, or a+ # rig_preprocess output's) whose entries the test cases can name: after+ # the --asset assets and before the run's datasets.+ cond_path: Tuple[str, ...] = ()+ # Dataset whose normalization stats apply to every asset that is not a+ # dataset object type. Default: the cond file's feature directory, else+ # the rig_preprocess summary.json's stats_dataset / reference / profile,+ # else objaverse. Irrelevant for a run with one global stats pool.+ cond_dataset_type: Optional[str] = Noneclass _UnconditionalWrapper(nn.Module):@@ -184,21 +216,6 @@ def _load_checkpoint(model, model_path: str, config: MainConfig):logger.info("Using standard model weights (no EMA).")-# ----------------------------------------------------------------------------# Test-case loaders-# ------------------------------------------------------------------------------def _known_object_types(dataset) -> set:- """Object types with at least one train or eval clip โ only these can- realize a test case (otherwise no reference clip is available)."""- md = dataset.motion_dataset- return {- ot for ot in md.cond_dict.keys()- if md.train_object_motions_map.get(ot) or md.eval_object_motions_map.get(ot)- }--def _make_text_encoder(config: MainConfig, device: torch.device):# pool=False so a prompt yields its token sequence; ``_encode_prompt``# derives the pooled vector from it, the same rule the data loader uses.@@ -232,68 +249,6 @@ def _release_gpu():torch.cuda.empty_cache()-def _resolve_inference_scope(args: InferenceArgs) -> Tuple[Optional[set], Optional[set]]:- """Return ``(target_object_types, target_clip_stems)`` extracted from the- test-case spec, or ``(None, None)`` when neither file is supplied.-- ``target_clip_stems`` is populated only for in-betweening / motion- editing โ they pin specific GT clips that must actually be on disk.- """- object_types: Optional[set] = None- clip_stems: Optional[set] = None-- if args.test_cases_json is not None:- with open(args.test_cases_json) as f:- raw_cases = json.load(f)- object_types = set()- if args.inbetween or args.motion_edit:- clip_stems = set()- for case_key in raw_cases:- obj_type, _, clip_id = case_key.partition('-')- if obj_type:- object_types.add(obj_type)- if clip_stems is not None and clip_id:- # On-disk file stem is "{obj_type}-{clip_id}" for- # Truebones/Objaverse and "{obj_type}_{clip_id}" for Mixamo- # (prefixed at load). Also keep the literal clip_id for- # users who already supply the full stem in case_key.- clip_stems.add(clip_id)- clip_stems.add(f"{obj_type}-{clip_id}")- clip_stems.add(f"{obj_type}_{clip_id}")- logger.info(- f"Inference scope from {args.test_cases_json}: "- f"{len(object_types)} object types"- + (f", {len(clip_stems)} pinned clip stems" if clip_stems else '')- )-- if args.test_cases_txt is not None:- if object_types is None:- object_types = set()- with open(args.test_cases_txt) as f:- for line in f:- s = line.strip()- if s and not s.startswith('#'):- object_types.add(s)- logger.info(- f"Inference scope augmented from {args.test_cases_txt}: "- f"{len(object_types)} object types total."- )-- return object_types, clip_stems---def _resolve_stats_path(exp_dir: str) -> Optional[str]:- """Return path to ``dataset_stats.npy`` in ``exp_dir``, or ``None``."""- candidate = os.path.join(exp_dir, "dataset_stats.npy")- if os.path.isfile(candidate):- return candidate- logger.warning(- f"No dataset_stats.npy at {candidate} โ stats will be recomputed from "- f"the loaded clips, which may not match training-time normalization."- )- return None--def _resolve_clip_name(dataset, obj_type: str, clip_id: str) -> Optional[str]:"""Find the dataset key whose stem matches ``clip_id`` for ``obj_type``.@@ -316,114 +271,6 @@ def _resolve_clip_name(dataset, obj_type: str, clip_id: str) -> Optional[str]:return None-def _load_test_cases_json(- json_path: str,- config: MainConfig,- dataset,- device: torch.device,- resolve_clip: bool = False,-) -> List[TestCase]:- """Caption-conditioned cases from {"<object_type>-<id>": caption} JSON.-- When ``resolve_clip`` is True (in-betweening), the ``<id>`` portion is- treated as a dataset clip stem and must resolve to an actual clip in- train or eval motion_dict โ otherwise the case is dropped, since- in-betweening has nothing to clamp against. When False, ``<id>`` is a- free-form tag and only ``<object_type>`` needs to exist.- """- with open(json_path) as f:- raw_cases: Dict[str, str] = json.load(f)-- known = _known_object_types(dataset)- cases: List[Tuple[str, str, str, Optional[str]]] = []- for case_key, caption in raw_cases.items():- obj_type, _, clip_id = case_key.partition('-')- if obj_type not in known:- logger.warning(- f"Test case {case_key!r}: object_type={obj_type!r} not in dataset; skipping."- )- continue- # An empty caption at cfg>1 collapses to ~unconditional output and- # is almost always a data-entry mistake; skip rather than silently- # generating garbage.- if not caption or not caption.strip():- logger.warning(f"Test case {case_key!r}: empty caption; skipping.")- continue-- resolved_clip: Optional[str] = None- if resolve_clip:- if not clip_id:- logger.warning(- f"Test case {case_key!r}: in-betweening requires "- f"'<object_type>-<clip_id>' format with a non-empty id; skipping."- )- continue- resolved_clip = _resolve_clip_name(dataset, obj_type, clip_id)- if resolved_clip is None:- logger.warning(- f"Test case {case_key!r}: clip_id={clip_id!r} not found in "- f"train or eval motion_dict for object_type={obj_type!r}; skipping."- )- continue-- cases.append((case_key, obj_type, caption, resolved_clip))-- if not cases:- raise ValueError(- f"No usable test cases in {json_path}: object_types not present in dataset."- )- logger.info(f"Loaded {len(cases)} test cases from {json_path}")-- encoder = _make_text_encoder(config, device)- encoded: List[TestCase] = []- for case_key, obj_type, caption, clip_name in cases:- encoded.append((case_key, obj_type, caption,- _encode_prompt(encoder, caption), clip_name))- del encoder- _release_gpu()- return encoded---def _load_test_cases_txt(- txt_path: str,- config: MainConfig,- dataset,- device: torch.device,-) -> List[TestCase]:- """Unconditional cases โ one object_type per line.-- The empty-string embedding is a shape-only placeholder; ``_UnconditionalWrapper``- zeroes it inside the model at sample time.- """- with open(txt_path) as f:- object_types = [- line.strip() for line in f- if line.strip() and not line.lstrip().startswith('#')- ]-- known = _known_object_types(dataset)- cases = []- for ot in object_types:- if ot not in known:- logger.warning(f"object_type={ot!r} not in dataset; skipping.")- continue- cases.append(ot)-- if not cases:- raise ValueError(- f"No usable test cases in {txt_path}: object_types not present in dataset."- )- logger.info(f"Loaded {len(cases)} unconditional test cases from {txt_path}")-- encoder = _make_text_encoder(config, device)- null_emb = _encode_prompt(encoder, "")- del encoder- _release_gpu()-- # case_id == object_type since txt rows have no per-case suffix.- return [(ot, ot, "", null_emb, None) for ot in cases]--def _load_test_cases_from_dataset(dataset) -> List[TestCase]:"""Enumerate test cases from the dataset, prioritised by data-loader source.@@ -492,104 +339,302 @@ def _load_test_cases_from_dataset(dataset) -> List[TestCase]:return encoded-def _load_expand_test_cases(- json_path: str,- config: MainConfig,- dataset,- device: torch.device,-) -> List[Tuple[str, str, List[str], List[np.ndarray]]]:- """Motion-expand cases from {"<object_type>-<label>": [prompt, ...]} JSON.-- Each value must be a non-empty list of non-empty strings. The list- order is the segment order: prompt[0] drives the free first segment,- prompt[i>0] drives a follow-up segment whose first ``expand_overlap``- frames are clamped to the previous segment's last ``expand_overlap``- frames.-- Returns ``[(case_id, object_type, [prompts], [caption_embs])]``.- Unlike motion-edit / in-betweening, no GT clip is pinned โ only the- skeleton from ``<object_type>`` is needed.+# ---------------------------------------------------------------------------+# Test cases: what to sample, on which asset+# ---------------------------------------------------------------------------+++class CaseSpec:+ """One test case before its asset is resolved.++ ``ref`` names the asset (``unimate.inference.assets``); ``tag`` is the+ free part of the case id (output files are ``<asset>-<tag>-rep_<r>-<i>.npy``,+ or ``<asset>-rep_...`` without one); ``prompt`` a string, a list of+ strings (``--motion_expand``) or ``''`` (unconditional); ``clip`` the+ clip id whose ground truth in-betweening / motion editing hold."""- with open(json_path) as f:- raw_cases = json.load(f)-- known = _known_object_types(dataset)- cases: List[Tuple[str, str, List[str]]] = []- for case_key, prompts in raw_cases.items():- obj_type, _, _ = case_key.partition('-')- if obj_type not in known:- logger.warning(- f"Test case {case_key!r}: object_type={obj_type!r} not in dataset; skipping."- )- continue- if not isinstance(prompts, list):- logger.warning(- f"Test case {case_key!r}: value must be a list of prompts for "- f"--motion_expand (got {type(prompts).__name__}); skipping."- )+ ref: str+ tag: Optional[str]+ prompt: object+ clip: Optional[str] = None+ asset: Optional[Asset] = None++ @property+ def case_id(self) -> str:+ return f'{self.asset.name}-{self.tag}' if self.tag else self.asset.name+++def _slug(text: str, max_len: int = 48) -> str:+ """A file-name tag from a prompt: its words, lower-case, joined by '_'."""+ words = re.findall(r'[a-z0-9]+', str(text).lower())+ slug = ''+ for w in words:+ if len(slug) + len(w) + 1 > max_len:+ break+ slug = f'{slug}_{w}' if slug else w+ return slug or 'case'+++def _cases_from_json(path: str, needs_gt: bool) -> List[CaseSpec]:+ """Cases of a test-case file: a ``{"<asset>-<tag>": prompt}`` map (the tag+ is also the pinned clip id of in-betweening / motion editing), or a list of+ ``{"asset": ref, "prompt": ..., "id": tag, "clip": clip id}`` entries+ (``id`` and ``clip`` optional; the tag defaults to the clip id, else a slug+ of the prompt)."""+ with open(path) as f:+ raw = json.load(f)+ specs = []+ if isinstance(raw, dict):+ for key, prompt in raw.items():+ ref, _, tag = key.partition('-')+ specs.append(CaseSpec(ref=ref, tag=tag or None, prompt=prompt,+ clip=(tag or None) if needs_gt else None))+ elif isinstance(raw, list):+ for i, entry in enumerate(raw):+ if not isinstance(entry, dict) or not entry.get('asset'):+ raise ValueError(f"{path} entry {i}: expected an object with an 'asset' key.")+ prompt = entry.get('prompt', '')+ clip = entry.get('clip') or (entry.get('id') if needs_gt else None)+ first = prompt[0] if isinstance(prompt, list) and prompt else prompt+ tag = entry.get('id') or clip or (_slug(first) if first else None)+ specs.append(CaseSpec(ref=str(entry['asset']), tag=tag, prompt=prompt,+ clip=clip))+ else:+ raise ValueError(f"{path}: expected a JSON object or list of test cases.")+ if not specs:+ raise ValueError(f"No test cases in {path}.")+ return specs+++def _parse_cases(args: InferenceArgs, cfg_scale: float) -> Optional[List[CaseSpec]]:+ """The test cases the arguments name, or None when the dataset's own split+ is enumerated instead (cfg > 1 without a test-case file or prompts)."""+ if cfg_scale > 1.0:+ if args.test_cases_json is not None:+ return _cases_from_json(args.test_cases_json, args.inbetween or args.motion_edit)+ if args.prompt:+ if args.motion_expand:+ return [CaseSpec(ref=ref, tag=_slug(args.prompt[0]), prompt=list(args.prompt))+ for ref in args.asset]+ return [CaseSpec(ref=ref, tag=_slug(p), prompt=p)+ for ref in args.asset for p in args.prompt]+ return None+ if args.test_cases_txt is not None:+ with open(args.test_cases_txt) as f:+ refs = [line.strip() for line in f+ if line.strip() and not line.lstrip().startswith('#')]+ if not refs:+ raise ValueError(f"No object types in {args.test_cases_txt}.")+ else:+ refs = list(args.asset)+ return [CaseSpec(ref=ref, tag=None, prompt='') for ref in refs]+++def _resolve_cases(specs: List[CaseSpec], resolver: AssetResolver+ ) -> Tuple[List[CaseSpec], Dict[str, Asset]]:+ """Resolve every case's asset. A case whose asset cannot be found is+ skipped with a warning; two different assets sharing a name are an error+ (object types and output files are keyed by it)."""+ assets: Dict[str, Asset] = {}+ resolved: Dict[str, Asset] = {}+ kept, seen = [], set()+ for spec in specs:+ if spec.ref not in resolved:+ try:+ resolved[spec.ref] = resolver.resolve(spec.ref)+ except (KeyError, ValueError) as exc:+ logger.warning(f"Test case asset {spec.ref!r} skipped: {exc}")+ resolved[spec.ref] = None+ asset = resolved[spec.ref]+ if asset is None:continue- valid = [p.strip() for p in prompts if isinstance(p, str) and p.strip()]- if len(valid) != len(prompts):- logger.warning(- f"Test case {case_key!r}: {len(prompts) - len(valid)} prompt(s) "- f"empty or non-string; keeping {len(valid)}/{len(prompts)}."- )- if not valid:- logger.warning(f"Test case {case_key!r}: no usable prompts; skipping.")+ other = assets.get(asset.name)+ if other is not None and other.source != asset.source:+ raise ValueError(+ f"Two assets named {asset.name!r} in one run ({other.source}, "+ f"{asset.source}); sample them in separate runs (or rename a custom one "+ f"with rig_preprocess --name).")+ assets[asset.name] = asset+ spec.asset = asset+ if spec.case_id in seen:+ base, n = spec.tag or 'case', 2+ while f'{asset.name}-{base}_{n}' in seen:+ n += 1+ logger.warning(f"Test case id {spec.case_id!r} repeats; this one becomes "+ f"'{asset.name}-{base}_{n}'.")+ spec.tag = f'{base}_{n}'+ seen.add(spec.case_id)+ kept.append(spec)+ if not kept:+ raise ValueError("No test case names a usable asset (see the warnings above).")+ return kept, assets+++def _register_assets(dataset, assets: Dict[str, Asset], specs: List[CaseSpec],+ needs_gt: bool, config: MainConfig,+ cache_dirs: Tuple[str, ...] = ()) -> Dict[str, Asset]:+ """Add the assets to a :class:`SkeletonDataset` (with the clips the cases+ pin when *needs_gt*). Joint names come from the feature directories'+ caches (*cache_dirs*: the run's, where present) before any is encoded.+ Returns the assets that registered; one the model cannot take (too many+ joints, not a canonical cond) is dropped with a warning."""+ md = dataset.motion_dataset+ md.merge_joint_name_cache(sorted({a.joint_cache_dir for a in assets.values()+ if a.joint_cache_dir} | set(cache_dirs)))+ clip_ids: Dict[str, set] = {}+ for spec in specs:+ if spec.clip:+ clip_ids.setdefault(spec.asset.name, set()).add(spec.clip)+ lo = config.dataset.min_joints+ usable = {}+ for name, asset in assets.items():+ n_joints = len(asset.cond['parents'])+ if n_joints < lo:+ logger.warning(f"{name!r} has {n_joints} joints, fewer than the {lo} the run "+ f"trained on; expect weaker motion.")+ clip_files = asset.clip_files(clip_ids.get(name, ())) if needs_gt else ()+ try:+ md.add_cond_object(name, asset.cond, asset.stats_dataset,+ motion_dir=asset.motion_dir, clip_files=clip_files,+ clip_key_prefix=asset.clip_key_prefix)+ except ValueError as exc:+ logger.warning(f"Asset {asset.source} not usable: {exc}")continue- cases.append((case_key, obj_type, valid))+ usable[name] = asset+ if not usable:+ raise ValueError("None of the test cases' assets can be sampled (see the warnings above).")+ return usable++def _encode_cases(specs: List[CaseSpec], dataset, encoder, args: InferenceArgs,+ cfg_scale: float):+ """``TestCase`` tuples (or, for ``--motion_expand``, ``(case_id,+ object_type, prompts, encodings)``) for the cases whose asset registered."""+ md = dataset.motion_dataset+ needs_gt = args.inbetween or args.motion_edit+ cases = []+ null_enc = None+ for spec in specs:+ name = spec.asset.name+ if name not in md.cond_dict:+ continue+ if cfg_scale == 1.0:+ if null_enc is None:+ null_enc = _encode_prompt(encoder, "")+ cases.append((spec.case_id, name, "", null_enc, None))+ continue+ if args.motion_expand:+ prompts = spec.prompt+ if not isinstance(prompts, list):+ logger.warning(f"Test case {spec.case_id!r}: --motion_expand needs a list "+ f"of prompts, got {type(prompts).__name__}; skipping.")+ continue+ valid = [p.strip() for p in prompts if isinstance(p, str) and p.strip()]+ if len(valid) != len(prompts):+ logger.warning(f"Test case {spec.case_id!r}: {len(prompts) - len(valid)} "+ f"prompt(s) empty or non-string; keeping {len(valid)}.")+ if not valid:+ continue+ cases.append((spec.case_id, name, valid,+ [_encode_prompt(encoder, p) for p in valid]))+ continue+ prompt = spec.prompt+ if not isinstance(prompt, str):+ logger.warning(f"Test case {spec.case_id!r}: expected a prompt string, got "+ f"{type(prompt).__name__} (prompt lists are for "+ f"--motion_expand); skipping.")+ continue+ if not prompt.strip():+ # At cfg > 1 an empty prompt collapses to unconditional output,+ # almost always a data-entry mistake.+ logger.warning(f"Test case {spec.case_id!r}: empty prompt; skipping.")+ continue+ clip_name = None+ if needs_gt:+ if not spec.clip:+ logger.warning(f"Test case {spec.case_id!r}: in-betweening / motion editing "+ f"need a clip id; skipping.")+ continue+ clip_name = _resolve_clip_name(dataset, name, spec.clip)+ if clip_name not in md.cond_object_clips.get(name, ()):+ clip_name = None # the rest-pose reference is no ground truth+ if clip_name is None:+ where = spec.asset.motion_dir or 'none (a rig_preprocess output keeps its ' \+ 'clips with --save_clips)'+ logger.warning(f"Test case {spec.case_id!r}: no clip file of {name!r} matches "+ f"{spec.clip!r} (clip directory: {where}); skipping.")+ continue+ cases.append((spec.case_id, name, prompt, _encode_prompt(encoder, prompt), clip_name))if not cases:- raise ValueError(- f"No usable motion-expand test cases in {json_path}."- )- logger.info(f"Loaded {len(cases)} motion-expand test cases from {json_path}.")+ raise ValueError("No usable test cases (see the warnings above).")+ logger.info(f"Prepared {len(cases)} test case(s) on {len({c[1] for c in cases})} asset(s).")+ return cases+++class SampleManifest:+ """``<output_dir>/manifest.json``: for every saved motion, the asset it+ was generated for (cond file and key, joint order, canonical GLB, stats+ dataset), its prompt and the run that made it. Mesh driving+ (``scripts/run_animate_motion.sh <output_dir>``) reads it instead of being+ told the dataset, cond and character again.++ A run into a directory that already holds a manifest adds to it: earlier+ motions stay drivable. An earlier asset of the same name that differs+ (another cond, another joint order) loses its motions from the manifest,+ since their files may be overwritten by this run's. Rewritten after every+ chunk.+ """- encoder = _make_text_encoder(config, device)- encoded: List[Tuple[str, str, List[str], List[np.ndarray]]] = []- for case_key, obj_type, prompts in cases:- embs = [_encode_prompt(encoder, p) for p in prompts]- encoded.append((case_key, obj_type, prompts, embs))- del encoder- _release_gpu()- return encoded+ FILE = 'manifest.json'+ FORMAT = 'unimate-samples/1'+ # What makes two manifest assets of one name the same skeleton.+ IDENTITY = ('cond_path', 'cond_key', 'joint_names')++ def __init__(self, output_dir: str, args: InferenceArgs, model_path: str,+ cfg_scale: float, mode: str, assets: Dict[str, Asset]):+ self.path = os.path.join(output_dir, self.FILE)+ self.data = {'format': self.FORMAT, 'motions_dir': 'motions', 'runs': [],+ 'assets': {}, 'samples': {}}+ if os.path.isfile(self.path):+ try:+ with open(self.path) as f:+ old = json.load(f)+ except (OSError, ValueError):+ old = {}+ if old.get('format') == self.FORMAT:+ self.data.update(runs=old.get('runs', []), assets=old.get('assets', {}),+ samples=old.get('samples', {}))+ for name, asset in sorted(assets.items()):+ entry = asset.manifest_entry()+ prev = self.data['assets'].get(name)+ if prev is not None and any(prev.get(k) != entry[k] for k in self.IDENTITY):+ dropped = [k for k, v in self.data['samples'].items() if v.get('asset') == name]+ for k in dropped:+ del self.data['samples'][k]+ logger.warning(f"{self.path}: asset {name!r} differs from the one earlier runs "+ f"sampled ({prev.get('source')}); their {len(dropped)} motion(s) "+ f"are no longer listed.")+ self.data['assets'][name] = entry+ self.run = len(self.data['runs'])+ self.data['runs'].append({'exp_dir': args.exp_dir, 'checkpoint': model_path,+ 'cfg_scale': cfg_scale, 'seed': args.seed, 'mode': mode})++ def add(self, npy_name: str, case_id: str, asset: str, prompt: str, kind: str = 'sample'):+ self.data['samples'][npy_name] = {'asset': asset, 'case_id': case_id, 'prompt': prompt,+ 'kind': kind, 'run': self.run}++ def save(self):+ tmp = f'{self.path}.tmp'+ with open(tmp, 'w') as f:+ json.dump(self.data, f, indent=2, ensure_ascii=False)+ os.replace(tmp, self.path)-def _select_test_cases(- args: InferenceArgs,- cfg_scale: float,- config: MainConfig,- dataset,- device: torch.device,- resolve_clip: bool = False,-) -> List[TestCase]:- """Test-case selection at cfg > 1.0, in priority order:-- 1. ``--test_cases_json`` provided- โ use the JSON's ``{object_type-id: prompt}`` map verbatim- (curated prompts).- 2. dataset's eval split populated by ``test_objects.txt``- โ per-clip enumeration over the listed object_types (eval mode).- 3. dataset's eval split populated by ``test_split_ratio > 0``- โ per-clip enumeration over the random eval clips (eval mode).- 4. no JSON, no eval split- โ dedup'd unique ``(object_type, caption)`` prompts from the train- split (visualization sweep).-- Cases 2/3/4 are all handled by ``_load_test_cases_from_dataset``, which- introspects the dataloader's split state and logs which source it used.-- At cfg == 1.0 the function always returns unconditional test cases from- ``--test_cases_txt`` (one ``object_type`` per line, null caption emb).- """- if cfg_scale > 1.0:- if args.test_cases_json is not None:- return _load_test_cases_json(- args.test_cases_json, config, dataset, device,- resolve_clip=resolve_clip,- )- return _load_test_cases_from_dataset(dataset)- return _load_test_cases_txt(args.test_cases_txt, config, dataset, device)+def _resolve_stats_path(exp_dir: str) -> Optional[str]:+ """Return path to ``dataset_stats.npy`` in ``exp_dir``, or ``None``."""+ candidate = os.path.join(exp_dir, "dataset_stats.npy")+ return candidate if os.path.isfile(candidate) else None# ---------------------------------------------------------------------------@@ -610,28 +655,39 @@ def _validate_cfg_inputs(cfg_scale: float, args: InferenceArgs):"--motion_expand is mutually exclusive with --inbetween / --motion_edit.")if (args.inbetween or args.motion_edit) and cfg_scale <= 1.0:- # cfg==1.0 uses --test_cases_txt (no clip pinning), so there's no- # GT motion to clamp known frames/joints against.+ # cfg==1.0 samples unconditionally from --test_cases_txt / --asset+ # (no clip pinning), so there's no GT motion to clamp against.mode = '--inbetween' if args.inbetween else '--motion_edit'raise ValueError(f"{mode} requires cfg_scale > 1.0 so test cases come from "f"--test_cases_json or the dataset eval split (both pin a clip).")+ if args.prompt and args.test_cases_json is not None:+ raise ValueError("Give --prompt or --test_cases_json, not both.")+ if args.prompt and not args.asset:+ raise ValueError("--prompt is sampled on the --asset assets; give at least one.")+ if args.prompt and (args.inbetween or args.motion_edit):+ raise ValueError("--inbetween / --motion_edit pin clips by id: give them in "+ "--test_cases_json.")if args.motion_expand and cfg_scale <= 1.0:raise ValueError(- "--motion_expand requires cfg_scale > 1.0 and a --test_cases_json "- "whose values are lists of prompts (one per segment)."+ "--motion_expand requires cfg_scale > 1.0 and prompt lists (--test_cases_json) "+ "or --asset with --prompt.")- if args.motion_expand and args.test_cases_json is None:+ if args.motion_expand and args.test_cases_json is None and not args.prompt:raise ValueError(- "--motion_expand requires --test_cases_json with per-case prompt lists."+ "--motion_expand requires --test_cases_json with per-case prompt lists, or "+ "--asset with --prompt (the segments).")if args.motion_edit and not args.keep_joints.strip():raise ValueError("--motion_edit requires --keep_joints (comma-separated joint names).")if cfg_scale > 1.0:- if args.test_cases_json is None:+ if (args.cond_path or args.asset) and args.test_cases_json is None and not args.prompt:+ raise ValueError("--asset / --cond_path at cfg_scale > 1.0 need --prompt or "+ "--test_cases_json naming what to sample.")+ if args.test_cases_json is None and not args.prompt:logger.info(f"cfg_scale={cfg_scale} > 1.0 with no --test_cases_json: "f"enumerating the dataset's test split (per-clip if eval split "@@ -643,10 +699,25 @@ def _validate_cfg_inputs(cfg_scale: float, args: InferenceArgs):"test_objects.txt drives the eval split at data-loading time.")else: # cfg_scale == 1.0- if args.test_cases_txt is None:- raise ValueError("cfg_scale=1.0 requires --test_cases_txt.")+ if args.test_cases_txt is None and not args.asset:+ raise ValueError("cfg_scale=1.0 requires --test_cases_txt or --asset.")if args.test_cases_json is not None:logger.warning("--test_cases_json ignored because cfg_scale == 1.0.")+ if args.prompt:+ logger.warning("--prompt ignored because cfg_scale == 1.0 (unconditional).")+++def _load_captions_map(path: str) -> Dict[str, str]:+ """An earlier run's ``captions.json`` in the same output directory, so it+ keeps listing that run's motions (as ``manifest.json`` does)."""+ if not os.path.isfile(path):+ return {}+ try:+ with open(path) as f:+ data = json.load(f)+ except (OSError, ValueError):+ return {}+ return data if isinstance(data, dict) else {}def _wrap_for_cfg(model, cfg_scale: float):@@ -690,13 +761,14 @@ def _run_sampling(config: MainConfig,sample_model,dataset,- test_cases: List[TestCase],+ cases: List[TestCase],diffusion,gen_diffusion,cfg_scale: float,device: torch.device,args: InferenceArgs,output_dir: str,+ manifest: Optional['SampleManifest'] = None,):"""Generate ``num_repetitions`` samples per test case, in chunks of``args.batch_size``. ``no_grad`` keeps the ODE rollout from accumulating@@ -707,13 +779,30 @@ def _run_sampling(pinned clip_name). That GT plus a keep_mask are passed to``generate_samples`` which routes to the replacement-style sampler.Each case's GT is also saved as ``<case_id>-gt.npy`` for side-by-side- rendering.+ rendering. *manifest* records every saved motion."""+ test_cases = cases# Only pin the reference clip when in-betweening or motion-editing (the# clip's GT motion is what the sampler clamps against). Other runs let# create_sample_condition pick a random clip of the same object_type,# since only the skeleton is needed.needs_gt = args.inbetween or args.motion_edit+ if needs_gt and args.gt_start_frame:+ # A pinned clip that ends before the start frame has no GT window.+ md = dataset.motion_dataset+ kept = []+ for case in test_cases:+ clip = case[4]+ n = (md.train_motion_dict.get(clip) or md.eval_motion_dict[clip])['motion'].shape[0]+ if n > args.gt_start_frame:+ kept.append(case)+ else:+ logger.warning(f"Test case {case[0]!r}: clip {clip!r} has {n} frames, "+ f"none from --gt_start_frame={args.gt_start_frame}; skipping.")+ if not kept:+ raise ValueError(f"No test case has a clip longer than "+ f"--gt_start_frame={args.gt_start_frame}.")+ test_cases = keptif needs_gt:test_case_captions = [(ot, cap, emb, clip) for _, ot, cap, emb, clip in test_cases@@ -740,7 +829,7 @@ def _run_sampling(# values are the prompt used to condition each sample. Rewritten after# every chunk so a partial run still leaves a valid index on disk.captions_path = os.path.join(output_dir, 'captions.json')- captions_map: Dict[str, str] = {}+ captions_map: Dict[str, str] = _load_captions_map(captions_path)logger.info(f"Starting sampling: {total} test cases ร {args.num_repetitions} reps "@@ -759,7 +848,9 @@ def _run_sampling(config=config,data=dataset,test_case_captions=test_case_captions[chunk_start:chunk_end],- gt_start_frame=args.gt_start_frame,+ # Only a pinned GT clip is cropped; a plain run's random+ # reference clip just supplies the skeleton.+ gt_start_frame=args.gt_start_frame if needs_gt else None,)cond = {k: v.to(device) if torch.is_tensor(v) else v@@ -925,9 +1016,18 @@ def _run_sampling(for object_idx, case_id in enumerate(chunk_case_ids):npy_name = f'{case_id}-rep_{rep_i}-{object_idx}.npy'- captions_map[npy_name] = captions_text[chunk_start + object_idx]+ caption = captions_text[chunk_start + object_idx]+ captions_map[npy_name] = caption+ if manifest is not None:+ obj_type = test_cases[chunk_start + object_idx][1]+ manifest.add(npy_name, case_id, obj_type, caption)+ if needs_gt:+ manifest.add(f'{case_id}-gt_rep_{rep_i}-{object_idx}.npy',+ case_id, obj_type, caption, kind='ground_truth')with open(captions_path, 'w') as f:json.dump(captions_map, f, indent=2, ensure_ascii=False)+ if manifest is not None:+ manifest.save()# Drop chunk-scoped tensors before the next batch so peak GPU# memory tracks per-batch, not per-run, usage.@@ -941,13 +1041,14 @@ def _run_expansion_sampling(config: MainConfig,sample_model,dataset,- expand_cases: List[Tuple[str, str, List[str], List[np.ndarray]]],+ cases: List[Tuple[str, str, List[str], List[CaptionEnc]]],diffusion,gen_diffusion,cfg_scale: float,device: torch.device,args: InferenceArgs,output_dir: str,+ manifest: Optional['SampleManifest'] = None,):"""Per-case chain generation: each case produces one concatenated motionof length ``max_T + (max_T - overlap) * (N - 1)``.@@ -957,8 +1058,10 @@ def _run_expansion_sampling(once via ``create_sample_condition`` and the per-segment cond is builtby swapping in the segment's pre-encoded ``caption_emb`` โ ``caption_emb``is a fixed-shape ``(B, text_dim)`` tensor (see ``mixture_batch_collate``)- so the swap is a single tensor assignment.+ so the swap is a single tensor assignment. *manifest* records every+ saved motion."""+ expand_cases = casesoverlap = args.expand_overlapmax_T = config.dataset.max_motion_lengthif overlap <= 0 or overlap >= max_T:@@ -968,7 +1071,7 @@ def _run_expansion_sampling()captions_path = os.path.join(output_dir, 'captions.json')- captions_map: Dict[str, str] = {}+ captions_map: Dict[str, str] = _load_captions_map(captions_path)logger.info(f"Starting motion-expand sampling: {len(expand_cases)} case(s) ร "@@ -1062,6 +1165,9 @@ def _run_expansion_sampling(captions_map[npy_name] = joined_captionwith open(captions_path, 'w') as f:json.dump(captions_map, f, indent=2, ensure_ascii=False)+ if manifest is not None:+ manifest.add(npy_name, case_id, obj_type, joined_caption)+ manifest.save()del cond, cond_per_segment, chain_release_gpu()@@ -1072,6 +1178,65 @@ def _run_expansion_sampling(# ---------------------------------------------------------------------------+def _mode(args: InferenceArgs, cfg_scale: float) -> str:+ if args.inbetween:+ return 'inbetween'+ if args.motion_edit:+ return 'motion_edit'+ if args.motion_expand:+ return 'motion_expand'+ return 'text' if cfg_scale > 1.0 else 'unconditional'+++def _prepare_cases(args: InferenceArgs, config: MainConfig, cfg_scale: float,+ stats_path: Optional[str], device: torch.device):+ """``(dataset, cases, assets)``: the dataset sampling conditions on, the+ encoded test cases (``TestCase`` tuples, or expansion cases) and the+ assets they use (name -> :class:`Asset`)."""+ needs_gt = args.inbetween or args.motion_edit+ specs = _parse_cases(args, cfg_scale)+ if specs is None:+ # No test cases: enumerate the dataset's own split (all its clips).+ if stats_path is None:+ logger.warning(+ f"No dataset_stats.npy in {args.exp_dir}: stats are recomputed from the "+ f"loaded clips, which may not match training-time normalization.")+ dataset = create_dataset(dataset_config=config.dataset, model_config=config.model,+ inference=True, stats_path=stats_path)+ cases = _load_test_cases_from_dataset(dataset)+ md = dataset.motion_dataset+ resolver = AssetResolver(config.dataset)+ assets = {}+ for _, name, _, _, clip in cases:+ if name not in assets:+ entry = md.train_motion_dict.get(clip) or md.eval_motion_dict[clip]+ assets[name] = resolver.dataset_asset(entry['dataset_type'], name,+ md.cond_dict[name])+ return dataset, cases, assets++ if stats_path is None:+ raise FileNotFoundError(+ f"No dataset_stats.npy in {args.exp_dir}: sampling needs the run's "+ f"normalization statistics.")+ resolver = AssetResolver(config.dataset, assets=args.asset, cond_paths=args.cond_path,+ cond_dataset_type=args.cond_dataset_type)+ specs, assets = _resolve_cases(specs, resolver)+ dataset = create_skeleton_dataset(config.dataset, config.model, stats_path)+ encoder = _make_text_encoder(config, device)+ md = dataset.motion_dataset+ md.text_encoder = encoder # joint names not in a cache use it too+ try:+ run_dirs = tuple(d for d in (resolver.features_dir(cfg.type) for cfg in+ config.dataset.data_configs.values()) if os.path.isdir(d))+ assets = _register_assets(dataset, assets, specs, needs_gt, config, run_dirs)+ cases = _encode_cases(specs, dataset, encoder, args, cfg_scale)+ finally:+ md.text_encoder = None+ del encoder+ _release_gpu()+ return dataset, cases, {n: a for n, a in assets.items() if any(c[1] == n for c in cases)}++def main(args: InferenceArgs):# Read config first so CLI-omitted fields can fall back to the saved# ``sampling`` block (mirrors the cfg_scale fallback below).@@ -1123,17 +1288,13 @@ def main(args: InferenceArgs):set_seed(args.seed)logger.info(f"Set random seed to [{args.seed}]")- target_object_types, target_clip_stems = _resolve_inference_scope(args)+ device = torch.device(config.sampling.device)stats_path = _resolve_stats_path(args.exp_dir)-- dataset = create_dataset(- dataset_config=config.dataset,- model_config=config.model,- inference=True,- target_object_types=target_object_types,- target_clip_stems=target_clip_stems,- stats_path=stats_path,- )+ # Test cases are encoded before the diffusion model moves to the device,+ # so the text encoder and the model don't coexist on the GPU.+ dataset, cases, assets = _prepare_cases(args, config, cfg_scale, stats_path, device)+ manifest = SampleManifest(output_dir, args, model_path, cfg_scale,+ _mode(args, cfg_scale), assets)logger.info("Creating model and diffusion...")model = create_model(@@ -1145,56 +1306,27 @@ def main(args: InferenceArgs):logger.info(f"Loading checkpoints from [{model_path}]...")_load_checkpoint(model, model_path, config)- device = torch.device(config.sampling.device)-- # Encode test cases before moving the diffusion model to device, so the- # text encoder and diffusion model don't coexist on GPU. Motion-expand- # has a distinct test-case structure (per-case prompt lists), so it- # uses its own loader + runner.- if args.motion_expand:- expand_cases = _load_expand_test_cases(- args.test_cases_json, config, dataset, device,- )- test_cases = None- else:- expand_cases = None- test_cases = _select_test_cases(- args, cfg_scale, config, dataset, device,- resolve_clip=args.inbetween or args.motion_edit,- )-model.to(device)model.eval()logger.info(f"Using cfg_scale={cfg_scale}")sample_model = _wrap_for_cfg(model, cfg_scale)- if args.motion_expand:- _run_expansion_sampling(- config=config,- sample_model=sample_model,- dataset=dataset,- expand_cases=expand_cases,- diffusion=diffusion,- gen_diffusion=gen_diffusion,- cfg_scale=cfg_scale,- device=device,- args=args,- output_dir=output_dir,- )- else:- _run_sampling(- config=config,- sample_model=sample_model,- dataset=dataset,- test_cases=test_cases,- diffusion=diffusion,- gen_diffusion=gen_diffusion,- cfg_scale=cfg_scale,- device=device,- args=args,- output_dir=output_dir,- )+ run = _run_expansion_sampling if args.motion_expand else _run_sampling+ run(+ config=config,+ sample_model=sample_model,+ dataset=dataset,+ cases=cases,+ diffusion=diffusion,+ gen_diffusion=gen_diffusion,+ cfg_scale=cfg_scale,+ device=device,+ args=args,+ output_dir=output_dir,+ manifest=manifest,+ )+ logger.info(f"Samples and {SampleManifest.FILE} in {output_dir}")if __name__ == "__main__":diff --git a/unimate/models/denoiser/base.py b/unimate/models/denoiser/base.pyindex ef538ea..32dc6ae 100644--- a/unimate/models/denoiser/base.py+++ b/unimate/models/denoiser/base.py@@ -221,6 +221,13 @@ class UniMateDenoiserBase(nn.Module):nn.init.normal_(self.cond_embedder.weight, std=0.02)nn.init.constant_(self.cond_embedder.bias, 0)+ # Depth embedding: _basic_init skips nn.Embedding, whose N(0, 1)+ # default makes this per-joint constant ~10x louder (RMS 1.0) than the+ # motion tokens (~0.1) it is added to, so after the block norms nearly+ # all of each token's energy at step 0 is the depth code.+ if self.use_depth_emb:+ nn.init.normal_(self.depth_embedding.weight, std=0.02)+# Zero-out adaLN modulation layers in UniMate blocks:for block in self.transformer_blocks:nn.init.constant_(block.adaLN_modulation[-1].weight, 0)@@ -305,7 +312,9 @@ class UniMateDenoiserBase(nn.Module):# masked row (which returns NaN).if force_mask:drop = torch.ones(tokens.shape[0], dtype=torch.bool, device=tokens.device)- elif self.training and self.cond_mask_prob > 0.:+ elif self.training:+ # An all-False draw at cond_mask_prob == 0 still routes+ # null_caption through torch.where below, so DDP sees it used.drop = torch.bernoulli(torch.ones(tokens.shape[0], device=tokens.device) * self.cond_mask_prob).to(torch.bool)diff --git a/unimate/models/denoiser/rope.py b/unimate/models/denoiser/rope.pyindex aa3995f..3dc55d9 100644--- a/unimate/models/denoiser/rope.py+++ b/unimate/models/denoiser/rope.py@@ -133,15 +133,16 @@ class RopeND:def _get_pos_embs(self, position_ids: torch.Tensor, device: torch.device,- ) -> Tuple[torch.Tensor, torch.Tensor]:- """Look up (or lazily create) per-axis embeddings and concatenate.+ ) -> Tuple[List[torch.Tensor], List[torch.Tensor]]:+ """Look up (or lazily create) the per-axis embeddings.Args:position_ids: ``(nd, L)`` integer position indices per axis.device: Target device for the frequency tables.Returns:- ``(cos, sin)`` each of shape ``(L, head_dim)``.+ ``(cos, sin)``: per-axis lists, entry ``i`` of shape+ ``(L, split_dims[i])``."""if position_ids.dim() == 1:position_ids = position_ids.unsqueeze(0)@@ -161,7 +162,7 @@ class RopeND:cos_parts.append(getattr(self, cache_key_cos)[position_ids[i, :], :])sin_parts.append(getattr(self, cache_key_sin)[position_ids[i, :], :])- return torch.cat(cos_parts, dim=-1), torch.cat(sin_parts, dim=-1)+ return cos_parts, sin_parts# ------------------------------------------------------------------# Forward@@ -183,11 +184,24 @@ class RopeND:Returns:Rotated ``(q, k)`` with the same shapes."""- cos_emb, sin_emb = self._get_pos_embs(position_ids, device=q.device)- # Shape (L, C) โ (1, 1, L, C) to broadcast over (B, H, L, C)- cos_emb = cos_emb.unsqueeze(0).unsqueeze(0)- sin_emb = sin_emb.unsqueeze(0).unsqueeze(0)- return _apply_rotary_pos_emb(q, k, cos_emb, sin_emb)+ cos_parts, sin_parts = self._get_pos_embs(position_ids, device=q.device)+ if self.nd == 1:+ # Shape (L, C) โ (1, 1, L, C) to broadcast over (B, H, L, C)+ return _apply_rotary_pos_emb(+ q, k, cos_parts[0][None, None], sin_parts[0][None, None])+ # Each axis rotates its own block of channels: rotate_half pairs+ # channels within a block, so a pair always shares one angle.+ q_out, k_out, start = [], [], 0+ for cos_emb, sin_emb, dim in zip(cos_parts, sin_parts, self.split_dims):+ # Shape (L, d) โ (1, 1, L, d) to broadcast over (B, H, L, d)+ q_i, k_i = _apply_rotary_pos_emb(+ q[..., start:start + dim], k[..., start:start + dim],+ cos_emb[None, None], sin_emb[None, None],+ )+ q_out.append(q_i)+ k_out.append(k_i)+ start += dim+ return torch.cat(q_out, dim=-1), torch.cat(k_out, dim=-1)# ---------------------------------------------------------------------------diff --git a/unimate/models/factory.py b/unimate/models/factory.pyindex 7d131d5..31e1bf1 100644--- a/unimate/models/factory.py+++ b/unimate/models/factory.py@@ -149,7 +149,8 @@ def create_transport(path-dependent default (0 for the stable velocity + Linear/GVPcombination used by all current configs).training_config: supplies ``lambda_geo`` / ``lambda_smooth`` for the- auxiliary losses.+ auxiliary losses and ``t_sampling`` / ``t_logit_mean`` /+ ``t_logit_std`` for the training-time distribution."""if prediction == "noise":@@ -192,6 +193,9 @@ def create_transport(sample_eps=sample_eps,lambda_geo=training_config.lambda_geo,lambda_smooth=training_config.lambda_smooth,+ t_sampling=training_config.t_sampling,+ t_logit_mean=training_config.t_logit_mean,+ t_logit_std=training_config.t_logit_std,)return state\ No newline at end of filediff --git a/unimate/models/flow/transport.py b/unimate/models/flow/transport.pyindex afe9318..90180b4 100644--- a/unimate/models/flow/transport.py+++ b/unimate/models/flow/transport.py@@ -59,6 +59,11 @@ class Transport:lambda_geo: weight for the auxiliary geodesic rotation loss.lambda_smooth: weight for the auxiliary velocity smoothness loss(penalizes ฮ on the local-velocity channels of the predicted x1).+ t_sampling: training-time distribution โ 'uniform' (U[t0, t1]) or+ 'logit_normal' (t = sigmoid(m + sยทz), z ~ N(0, 1), Esser et al.+ 2024), rescaled to [t0, t1].+ t_logit_mean / t_logit_std: m and s of the logit-normal. With t = 0+ noise and t = 1 data, m > 0 shifts samples toward the data end."""def __init__(@@ -71,6 +76,9 @@ class Transport:sample_eps,lambda_geo=0.0,lambda_smooth=0.0,+ t_sampling="uniform",+ t_logit_mean=0.0,+ t_logit_std=1.0,):path_options = {PathType.LINEAR: path.ICPlan,@@ -85,6 +93,11 @@ class Transport:self.sample_eps = sample_epsself.lambda_geo = lambda_geoself.lambda_smooth = lambda_smooth+ if t_sampling not in ("uniform", "logit_normal"):+ raise ValueError(f"Unknown t_sampling: {t_sampling!r}")+ self.t_sampling = t_sampling+ self.t_logit_mean = t_logit_mean+ self.t_logit_std = t_logit_stdself.l2_loss = lambda a, b: (a - b) ** 2@@ -203,11 +216,16 @@ class Transport:Args:x1: data point, shape (B, *dim).Returns:- (t, x0, x1) where x0 ~ N(0, I), t ~ U[t0, t1].+ (t, x0, x1) where x0 ~ N(0, I) and t follows ``t_sampling`` on+ [t0, t1]."""x0 = th.randn_like(x1)t0, t1 = self.check_interval(self.train_eps, self.sample_eps)- t = th.rand((x1.shape[0],)) * (t1 - t0) + t0+ if self.t_sampling == "logit_normal":+ u = th.sigmoid(self.t_logit_mean + self.t_logit_std * th.randn((x1.shape[0],)))+ else:+ u = th.rand((x1.shape[0],))+ t = u * (t1 - t0) + t0t = t.to(x1)return t, x0, x1diff --git a/unimate/tools/precompute_text_emb.py b/unimate/tools/precompute_text_emb.pyindex 7fca738..b3c33fe 100644--- a/unimate/tools/precompute_text_emb.py+++ b/unimate/tools/precompute_text_emb.py@@ -12,6 +12,10 @@ One file per source, each keyed the way the dataset looks it up:keys of captions.json), so a clip's features are adirect lookup; two clips sharing a caption each gettheir own entry+ * ``captions_generic.json`` -> ``caption_generic_emb_cache.npz``, the same for+ the second, generic caption of each clip+ * ``captions_detail.json`` -> ``caption_detail_emb_cache.npz``, the same for+ the third, detail caption of each clip* ``cond.npy`` -> ``joint_emb_cache.npz`` one entry per UNIQUE jointname, taken from the cleaned vocabulary``clean_joint_names`` (see@@ -33,7 +37,7 @@ using another.Usage (from the repo root)::- # every dataset/features/* with the config's encoder+ # the config's datasets (their feature paths) with the config's encoderpython -m unimate.tools.precompute_text_emb --config configs/uniml3d_60frames_graph_adaln.json# one input folder (positional or --input_dir), encoder given explicitly@@ -43,9 +47,11 @@ Usage (from the repo root)::# re-encode even if a cache is already therepython -m unimate.tools.precompute_text_emb --config <cfg> --overwrite-Re-run it after regenerating captions or joint names: entries the cache does-not cover are encoded at load time anyway, so a stale cache costs speed rather-than correctness, but a fresh one keeps the encoder out of the run entirely.+Re-run it after regenerating captions or joint names. Caption caches store+the text of every entry, so an entry whose caption has since changed is+re-encoded here; until then the data loader treats it as a cache miss and+encodes it live at startup (correct, but it loads the text encoder). A cache+without stored texts is rebuilt in full."""import argparse@@ -58,13 +64,13 @@ import numpy as npimport torchfrom tqdm import tqdm-from unimate.configs.schema import MainConfig+from unimate.configs.schema import TEXT_ENCODER_DEFAULTS, MainConfigfrom unimate.dataset.mixture.dataset import model_joint_namesfrom unimate.models.text_encoder.factory import create_text_encoderfrom unimate.utils.logger import get_logger-from unimate.utils.text_emb_cache import (KINDS, cache_path, load_cache,- pooled_from_hidden, save_cache,- sequences_from_hidden)+from unimate.utils.text_emb_cache import (KINDS, cache_path, cache_texts,+ load_cache, pooled_from_hidden,+ save_cache, sequences_from_hidden)logger = get_logger(file_name=__file__)@@ -78,16 +84,20 @@ DEFAULT_GLOB = 'dataset/features/*'def collect_texts(root_dir):"""What each source needs encoded -> ``{kind: {key: text}}``.- ``'caption'`` maps every clip key to its caption; ``'joint'`` maps each+ ``'caption'`` / ``'caption_generic'`` / ``'caption_detail'`` map every+ clip key to its normal / generic / detail caption; ``'joint'`` maps eachunique joint name to itself. A source with no file simply does not appear."""out = {}- captions_path = pjoin(root_dir, 'captions.json')- if os.path.isfile(captions_path):- with open(captions_path) as f:- captions = json.load(f)- out['caption'] = {clip: cap for clip, cap in captions.items() if cap}+ for kind, name in (('caption', 'captions.json'),+ ('caption_generic', 'captions_generic.json'),+ ('caption_detail', 'captions_detail.json')):+ captions_path = pjoin(root_dir, name)+ if os.path.isfile(captions_path):+ with open(captions_path) as f:+ captions = json.load(f)+ out[kind] = {clip: cap for clip, cap in captions.items() if cap}cond_path = pjoin(root_dir, 'cond.npy')if os.path.isfile(cond_path):@@ -159,10 +169,12 @@ def parse_args():'dataset/features/truebones. Same as --input_dir.')ap.add_argument('--input_dir', action='append', default=[], metavar='DIR',help='Feature directory to process; repeatable. With '- f'neither form given, every {DEFAULT_GLOB} is done.')+ 'neither form given: the --config datasets\' feature '+ f'paths, else every {DEFAULT_GLOB}.')ap.add_argument('--config', default=None,help='Training config to take the encoder type / version '- 'from; --encoder_type / --encoder_version override it')+ 'and the feature directories from; --encoder_type / '+ '--encoder_version override it')ap.add_argument('--encoder_type', default=None, help="'t5', 'clip', 'bert'")ap.add_argument('--encoder_version', default=None,help="e.g. 'google/flan-t5-base' (None = the type's default)")@@ -178,14 +190,29 @@ def main():args = parse_args()encoder_type, encoder_version = args.encoder_type, args.encoder_version+ config_roots = []if args.config:cfg = MainConfig.from_json(args.config)+ # Without explicit folders: the feature directories the config trains on.+ paths = [getattr(cfg, ds).path for ds in cfg.dataset.dataset_list]+ config_roots = [p for p in paths if os.path.isdir(p)]+ for p in paths:+ if p not in config_roots:+ logger.warning(f'{args.config}: feature directory {p} not found; skipped')+ if not config_roots and not (args.inputs or args.input_dir):+ raise SystemExit(f'None of the feature directories of {args.config} exist: {paths}')+ # The config's version belongs to the config's encoder type only.+ if encoder_type is None or encoder_type == cfg.model.text_encoder_type:+ encoder_version = encoder_version or cfg.model.text_encoder_versionencoder_type = encoder_type or cfg.model.text_encoder_type- encoder_version = encoder_version or cfg.model.text_encoder_versionif not encoder_type:raise SystemExit('Give --encoder_type or --config')+ if encoder_version is None:+ encoder_version = TEXT_ENCODER_DEFAULTS.get(encoder_type)+ if encoder_version is None:+ raise SystemExit(f"Unknown --encoder_type {encoder_type!r}")- roots = resolve_roots(list(args.inputs) + list(args.input_dir))+ roots = resolve_roots(list(args.inputs) + list(args.input_dir) or config_roots)logger.info(f'Encoder: {encoder_type} / {encoder_version} on {args.device}')# The encoder is built on the first directory that actually needs it, so a@@ -198,18 +225,30 @@ def main():continuefor kind, text_by_key in by_kind.items():- hit = ({} if args.overwrite- else load_cache(root, kind, encoder_type, encoder_version))+ sequence = not KINDS[kind][1]+ # A caption cache without stored texts cannot be checked for+ # captions edited since it was built: rebuild it in full.+ rebuild = args.overwrite or (sequence and cache_texts(root, kind) is None)+ loaded = ({} if rebuild+ else load_cache(root, kind, encoder_type, encoder_version,+ texts=text_by_key if sequence else None))+ # Entries for clips no longer listed are dropped (the file is+ # rewritten below whenever there are any).+ orphans = [k for k in loaded if k not in text_by_key]+ hit = {k: v for k, v in loaded.items() if k in text_by_key}# A sequence kind loads as (tokens, pooled) pairs; split them so- # both views survive an incremental re-run.+ # both views survive an incremental re-run. A pooled kind loads as+ # (D,) vectors: keep them as the pooled values and give save_cache+ # a (1, D) sequence so they sit beside newly encoded (T, D) ones.if KINDS[kind][1]:- cached, cached_pooled = dict(hit), {}+ cached = {k: v[None] for k, v in hit.items()}+ cached_pooled = dict(hit)else:cached = {k: v[0] for k, v in hit.items()}cached_pooled = {k: v[1] for k, v in hit.items()}todo_keys = [k for k in text_by_key if k not in cached]path = cache_path(root, kind)- if not todo_keys:+ if not todo_keys and not orphans:logger.info(f'[{root}] {kind}: {len(text_by_key)} keys already 'f'cached in {os.path.basename(path)}; nothing to do')continue@@ -219,21 +258,24 @@ def main():unique = sorted({text_by_key[k] for k in todo_keys})logger.info(f'[{root}] {kind}: {len(text_by_key)} keys, 'f'{len(todo_keys)} to encode over {len(unique)} distinct '- f'strings, {len(cached)} reused')- if encoder is None:+ f'strings, {len(cached)} reused, {len(orphans)} orphans dropped')+ if unique and encoder is None:# pool=False: the cache stores token sequences; pooling for the# adaLN path happens in the data loader.encoder = create_text_encoder(encoder_type=encoder_type,encoder_version=encoder_version,device=args.device, pool=False)- by_text, pooled_by_text = encode_texts(- unique, encoder, args.chunk_size,- f'{os.path.basename(root.rstrip("/"))}/{kind}')- cached.update({k: by_text[text_by_key[k]] for k in todo_keys})- cached_pooled.update({k: pooled_by_text[text_by_key[k]]- for k in todo_keys})+ if unique:+ by_text, pooled_by_text = encode_texts(+ unique, encoder, args.chunk_size,+ f'{os.path.basename(root.rstrip("/"))}/{kind}')+ cached.update({k: by_text[text_by_key[k]] for k in todo_keys})+ cached_pooled.update({k: pooled_by_text[text_by_key[k]]+ for k in todo_keys})n = save_cache(path, cached, kind, encoder_type, encoder_version,- pooled_by_key=cached_pooled)+ pooled_by_key=cached_pooled,+ texts_by_key=({k: text_by_key[k] for k in cached}+ if sequence else None))what = 'pooled vectors' if KINDS[kind][1] else 'token sequences'logger.info(f'[{root}] wrote {n} {what} -> {path}')diff --git a/unimate/training/ema.py b/unimate/training/ema.pyindex 5064752..d034954 100644--- a/unimate/training/ema.py+++ b/unimate/training/ema.py@@ -66,6 +66,7 @@ class EMAModel:self.cur_decay_value = decayone_minus_decay = 1 - decay+ _check_pairing(self.shadow_params, parameters)for s_param, param in zip(self.shadow_params, parameters):if param.dtype != s_param.dtype:param = param.to(s_param.dtype)@@ -78,6 +79,7 @@ class EMAModel:def copy_to(self, parameters: Iterator[torch.nn.Parameter]) -> None:"""Copy shadow (averaged) params into *parameters*."""parameters = [p for p in parameters if p.requires_grad]+ _check_pairing(self.shadow_params, parameters)for s_param, param in zip(self.shadow_params, parameters):param.data.copy_(s_param.data)@@ -88,6 +90,7 @@ class EMAModel:def restore(self, parameters: Iterator[torch.nn.Parameter]) -> None:"""Restore params saved by :meth:`store`."""parameters = [p for p in parameters if p.requires_grad]+ _check_pairing(self.collected_params, parameters)for c_param, param in zip(self.collected_params, parameters):param.data.copy_(c_param.data)@@ -115,4 +118,17 @@ class EMAModel:def to(self, device=None, dtype=None) -> None:"""Move shadow params to the given device / dtype."""- self.shadow_params = [p.to(device=device, dtype=dtype) for p in self.shadow_params]\ No newline at end of file+ self.shadow_params = [p.to(device=device, dtype=dtype) for p in self.shadow_params]+++def _check_pairing(saved, parameters) -> None:+ """Saved tensors pair with the model's trainable parameters by position;+ a count or shape mismatch (another model, a changed parameter order) would+ otherwise be truncated by ``zip`` or broadcast by ``copy_`` silently."""+ if len(saved) != len(parameters):+ raise ValueError(f"EMA holds {len(saved)} tensors for {len(parameters)} "+ f"trainable parameters")+ for i, (a, b) in enumerate(zip(saved, parameters)):+ if a.shape != b.shape:+ raise ValueError(f"EMA tensor {i} has shape {tuple(a.shape)}, the "+ f"parameter {tuple(b.shape)}")diff --git a/unimate/training/tracker.py b/unimate/training/tracker.pyindex b5a000e..3ac6655 100644--- a/unimate/training/tracker.py+++ b/unimate/training/tracker.py@@ -22,7 +22,8 @@ class TrainingTracker:Args:max_epochs: Stop after this many full passes through the dataset.max_steps: Stop after this many optimizer steps.- steps_per_epoch: Number of batches per epoch (``len(dataloader)``).+ steps_per_epoch: Optimizer steps per epoch (this rank's batches over+ the gradient accumulation).Used to estimate epochs from steps (or vice versa)."""diff --git a/unimate/training/train.py b/unimate/training/train.pyindex 414ecd1..64d11bf 100644--- a/unimate/training/train.py+++ b/unimate/training/train.py@@ -5,9 +5,11 @@ Entry point (usually via ``accelerate launch``, see scripts/)::python -m unimate.training.train --config configs/<config>.jsonCLI overrides (via tyro): ``--output_dir``, ``--batch_size``, ``--num_workers``,-``--resume``.+``--resume``, ``--stats_path``."""+import json+import mathimport osimport warningsfrom dataclasses import dataclass@@ -44,6 +46,10 @@ class TrainingArgs:batch_size: Optional[int] = Nonenum_workers: Optional[int] = Noneresume: Optional[str] = None # path to checkpoint .pt file to resume from+ # Normalization stats to load instead of computing them from the data+ # (e.g. the dataset_stats.npy of the run being fine-tuned). Default: the+ # resumed run's own file with --resume, otherwise computed.+ stats_path: Optional[str] = None# ===================================================================@@ -78,14 +84,25 @@ def train_diffusion(args: TrainingArgs, config: MainConfig,# ---- Dataset ----logger.info("Creating dataset loader...")- batch_size = args.batch_size or config.training.batch_size- num_workers = args.num_workers or config.training.num_workers+ # `is not None`, not `or`: --num_workers 0 (load in the main process) is+ # a valid override that `or` would silently replace with the config value.+ batch_size = (args.batch_size if args.batch_size is not None+ else config.training.batch_size)+ num_workers = (args.num_workers if args.num_workers is not None+ else config.training.num_workers)+ # Write the overrides back so the run's config.json records what ran.+ config.training.batch_size = batch_size+ config.training.num_workers = num_workers+ stats_src = _resolve_train_stats_path(args)+ _pin_resumed_max_depth(args, config)dataloader = create_train_dataloader(dataset_config=config.dataset,model_config=config.model,balanced=config.training.balanced,batch_size=batch_size,num_workers=num_workers,+ stats_path=stats_src,+ seed=config.training.seed or 0,)# ---- Persist config (after dataset so auto-computed max_joints /@@ -115,20 +132,13 @@ def train_diffusion(args: TrainingArgs, config: MainConfig,checkpoint_dir=checkpoint_dir,)- # ---- Optimizer & LR schedule ----+ # ---- Optimizer (the LR schedule follows the tracker, below) ----optimizer = torch.optim.AdamW(model.parameters(),lr=config.training.learning_rate,weight_decay=config.training.weight_decay,betas=(config.training.adam_beta1, config.training.adam_beta2),)- lr_scheduler = get_cosine_with_min_lr_schedule_with_warmup(- optimizer,- num_warmup_steps=int(config.training.warmup_ratio * config.training.num_steps),- num_training_steps=config.training.num_steps,- min_lr_rate=config.training.min_lr_ratio,- num_cycles=0.5,- )# ---- Accelerate: wrap model, optimizer, dataloader for distributed ----# NOTE: lr_scheduler is NOT wrapped by accelerator. AcceleratedScheduler@@ -145,10 +155,20 @@ def train_diffusion(args: TrainingArgs, config: MainConfig,writer = SummaryWriter(log_dir) if is_main else None# ---- Training tracker ----+ # Optimizer steps per epoch: this rank's batches over the accumulation+ # (accelerate also steps on an epoch's last, partial group).tracker = TrainingTracker(max_epochs=config.training.num_epochs,max_steps=config.training.num_steps,- steps_per_epoch=len(dataloader),+ steps_per_epoch=math.ceil(+ len(dataloader) / config.training.gradient_accumulation_steps),+ )+ lr_scheduler = get_cosine_with_min_lr_schedule_with_warmup(+ optimizer,+ num_warmup_steps=int(config.training.warmup_ratio * tracker.total_steps),+ num_training_steps=tracker.total_steps,+ min_lr_rate=config.training.min_lr_ratio,+ num_cycles=0.5,)# ---- Resume from checkpoint (if requested) ----@@ -157,6 +177,11 @@ def train_diffusion(args: TrainingArgs, config: MainConfig,args.resume, accelerator, model, optimizer, lr_scheduler,trainer, tracker,)+ # The checkpoint holds no RNG state; reseeding with the resumed step+ # keeps the flow-matching t, noise and caption-dropout draws from+ # repeating the stream the run already used from step 0.+ if config.training.seed is not None:+ set_seed(config.training.seed + tracker.current_step, device_specific=True)if is_main:logger.info(f"Training for {tracker.get_duration_str()}")@@ -292,22 +317,26 @@ def train_diffusion(args: TrainingArgs, config: MainConfig,# ===================================================================def _set_sampler_epoch(dataloader, epoch: int):- """Call set_epoch on the underlying sampler if it supports it (for DDP).+ """Set the sampler's epoch so each epoch draws different indices (DDP-safe).++ An Accelerate-prepared loader (``DataLoaderShard``) re-sets its sampler+ to its own ``iteration`` counter at the start of every ``__iter__``; that+ counter starts at 0 in a new process, so it must be set through the+ loader's ``set_epoch`` or a resumed run replays the epoch order from 0.- After accelerator.prepare(), the chain is:- dataloader.batch_sampler โ BatchSamplerShard- .batch_sampler โ original BatchSampler- .sampler โ MixtureSampler (or DistributedSampler)- We walk the chain until we find a sampler with set_epoch.+ A plain loader: walk the batch_sampler chain (Accelerate's+ BatchSamplerShard stores the original as ``.batch_sampler``, whose+ ``.sampler`` is the MixtureSampler or DistributedSampler)."""- # Walk batch_sampler chain (Accelerate's BatchSamplerShard wraps the original)+ if hasattr(dataloader, 'set_epoch'):+ dataloader.set_epoch(epoch)+ returnbs = getattr(dataloader, 'batch_sampler', None)while bs is not None:sampler = getattr(bs, 'sampler', None)if sampler is not None and hasattr(sampler, 'set_epoch'):sampler.set_epoch(epoch)return- # Accelerate's BatchSamplerShard stores the original as .batch_samplerbs = getattr(bs, 'batch_sampler', None)# Fallback: direct sampler attribute@@ -337,15 +366,12 @@ def _log_metrics(writer, trainer, lr_scheduler, loss_dict, grad_norm,def _save_checkpoint_and_visualize(trainer, accelerator, model, optimizer,lr_scheduler, tracker, config,checkpoint_dir, debug_dir):- """Generate sample visualizations and persist a checkpoint to disk."""- trainer.visualize_samples(- save_dir=debug_dir,- prefix=f"step_{tracker.current_step}",- num_samples=config.sampling.num_samples,- cfg_scale=config.sampling.cfg_scale,- )- torch.cuda.empty_cache()+ """Persist a checkpoint to disk, then render sample visualizations.+ The checkpoint is written first and a visualization failure is only+ logged, so a crash in the debug ODE solve or the renderer can neither+ lose the checkpoint nor end the run.+ """checkpoint_dict = {"model_state_dict": accelerator.get_state_dict(model),"optimizer_state_dict": optimizer.state_dict(),@@ -361,6 +387,72 @@ def _save_checkpoint_and_visualize(trainer, accelerator, model, optimizer,)accelerator.save(checkpoint_dict, checkpoint_path)logger.info(f"Saved checkpoint to {checkpoint_path}")+ del checkpoint_dict++ try:+ trainer.visualize_samples(+ save_dir=debug_dir,+ prefix=f"step_{tracker.current_step}",+ num_samples=config.sampling.num_samples,+ cfg_scale=config.sampling.cfg_scale,+ )+ except Exception:+ logger.exception(+ f"Sample visualization failed at step {tracker.current_step}; "+ f"the checkpoint is saved and training continues.")+ torch.cuda.empty_cache()+++def _resumed_run_dir(resume_path):+ """``<run>`` for ``<run>/checkpoints/<ckpt>.pt`` (the layout training+ writes), else the checkpoint's own directory."""+ ckpt_dir = os.path.dirname(os.path.abspath(resume_path))+ return (os.path.dirname(ckpt_dir)+ if os.path.basename(ckpt_dir) == "checkpoints" else ckpt_dir)+++def _pin_resumed_max_depth(args, config):+ """With ``--resume`` and an auto ``dataset.max_depth`` (0), cap it at the+ resumed run's saved value, so the depth embedding keeps the checkpoint's+ shape even if the auto value computed from the data has grown since.+ A config that sets ``max_depth`` wins."""+ if not args.resume or config.dataset.max_depth > 0:+ return+ run_config = os.path.join(_resumed_run_dir(args.resume), "config.json")+ if not os.path.isfile(run_config):+ logger.warning(f"No config.json at {run_config}; max_depth is recomputed "+ f"from the data and must match the checkpoint's")+ return+ with open(run_config) as f:+ saved = json.load(f).get("dataset", {}).get("max_depth", 0)+ if saved > 0:+ config.dataset.max_depth = saved+ logger.info(f"Resuming: dataset.max_depth capped at the run's saved {saved}")+++def _resolve_train_stats_path(args):+ """Stats file the run must reuse, or ``None`` to compute from the data.++ ``--stats_path`` wins. With ``--resume``, the checkpoint's run directory+ supplies the stats the weights were trained with: ``<run>`` for+ ``<run>/checkpoints/<ckpt>.pt`` (the layout training writes), else the+ checkpoint's own directory. Recomputing them would shift the+ normalization whenever the data changed in between.+ """+ if args.stats_path:+ if not os.path.isfile(args.stats_path):+ raise FileNotFoundError(f"--stats_path {args.stats_path} does not exist")+ return args.stats_path+ if args.resume:+ candidate = os.path.join(_resumed_run_dir(args.resume), "dataset_stats.npy")+ if os.path.isfile(candidate):+ return candidate+ logger.warning(+ f"No dataset_stats.npy at {candidate}; recomputing stats from the "+ f"current data, which only matches the checkpoint if the data and "+ f"the stats settings are unchanged. Pass --stats_path to pin them."+ )+ return Nonedef _resume_from_checkpoint(diff --git a/unimate/utils/motion_utils.py b/unimate/utils/motion_utils.pyindex bd8d937..deef6de 100644--- a/unimate/utils/motion_utils.py+++ b/unimate/utils/motion_utils.py@@ -12,7 +12,7 @@ import numpy as npfrom Quaternions import Quaternions-from unimate.utils.rotation_conversions import rotation_6d_to_matrix_np+from unimate.utils.rotation_conversions import matrix_to_quaternion_np, rotation_6d_to_matrix_np# ===========================================================================@@ -204,23 +204,42 @@ def realign_unimate_clip(clip_feats, parents):Returns:Re-aligned clip features (T, J, 12)."""- clip = clip_feats.copy()+ q0 = Quaternions(matrix_to_quaternion_np(+ rotation_6d_to_matrix_np(clip_feats[0, 0, 3:9][None])))[0]+ return rotate_unimate_facing(clip_feats, parents, q0)+++def rotate_unimate_facing(clip_feats, parents, q):+ """Re-express a clip in a world frame rotated by the facing quaternion ``q``.++ Root facing (joint 0) becomes ``q_t * q^{-1}`` and the slots of the+ root's direct children (which hold the root's world rotation, HML+ reordering) become ``q * q_child``. RIFKE positions and local velocities+ are expressed in the per-frame facing frame and do not change.- # Decode first-frame root facing quaternion- q0 = Quaternions.from_transforms(- rotation_6d_to_matrix_np(clip[0, 0, 3:9][None]))[0]+ ``rotate_unimate_facing(clip, parents, q0)`` with ``q0`` the frame-0+ facing is :func:`realign_unimate_clip`; applying ``-q0`` afterwards+ restores the original frame.++ Args:+ clip_feats: Clip motion features (T, J, 12).+ parents: Parent index list (J,).+ q: Facing rotation as a single-element ``Quaternions``.++ Returns:+ Rotated clip features (T, J, 12).+ """+ clip = clip_feats.copy()- # Root facing (joint 0): make relative to first frameroot_6d = clip[:, 0, 3:9]- q_root = Quaternions.from_transforms(rotation_6d_to_matrix_np(root_6d))- clip[:, 0, 3:9] = (q_root * -q0).rotation_matrix(cont6d=True)+ q_root = Quaternions(matrix_to_quaternion_np(rotation_6d_to_matrix_np(root_6d)))+ clip[:, 0, 3:9] = (q_root * -q).rotation_matrix(cont6d=True)- # Direct children of root: apply facing rotation to root's local rotationfor j, p in enumerate(parents):if p == 0:child_6d = clip[:, j, 3:9]- q_child = Quaternions.from_transforms(rotation_6d_to_matrix_np(child_6d))- clip[:, j, 3:9] = (q0 * q_child).rotation_matrix(cont6d=True)+ q_child = Quaternions(matrix_to_quaternion_np(rotation_6d_to_matrix_np(child_6d)))+ clip[:, j, 3:9] = (q * q_child).rotation_matrix(cont6d=True)return clip@@ -235,7 +254,13 @@ def hml_rotations_to_bvh_quaternions(motion_rot6d_hml, parents):HML convention (see :func:`compute_cont6d_params`):``hml[j] = bvh[parents[j]]`` for ``j >= 1``. Slot 0 holds the rootfacing quaternion โ it's ignored here, since the root's global rotation- is recovered from any child-of-root slot (which stores ``bvh[0]``).+ is recovered from the child-of-root slots (which store ``bvh[0]``).++ A joint with several children has one copy of its rotation per child.+ They are identical in real data but differ slightly in generated motion,+ and a motion edit may pin only some of them, so the decoded rotation is+ their chordal mean (the sum of the matrices projected back onto SO(3))+ rather than any single copy. Leaf joints keep the identity.Args:motion_rot6d_hml: 6D rotations in HML order (T, J, 6).@@ -247,9 +272,17 @@ def hml_rotations_to_bvh_quaternions(motion_rot6d_hml, parents):T, J, _ = motion_rot6d_hml.shapebvh_rot_mat = np.tile(np.eye(3), (T, J, 1, 1)) # identity defaultshml_mat = rotation_6d_to_matrix_np(motion_rot6d_hml)+ copy_sum = np.zeros((T, J, 3, 3))+ has_child = np.zeros(J, dtype=bool)for j, p in enumerate(parents[1:], 1):- bvh_rot_mat[:, p] = hml_mat[:, j]- return Quaternions.from_transforms(bvh_rot_mat)+ copy_sum[:, p] += hml_mat[:, j]+ has_child[p] = True+ U, _, Vt = np.linalg.svd(copy_sum[:, has_child])+ # Flip the last singular direction where needed so the result is a+ # rotation (det +1), not a reflection.+ U[..., :, -1] *= np.sign(np.linalg.det(U @ Vt))[..., None]+ bvh_rot_mat[:, has_child] = U @ Vt+ return Quaternions(matrix_to_quaternion_np(bvh_rot_mat))def fk_global_positions(rotations_bvh, parents, offsets, root_positions):@@ -301,7 +334,7 @@ def recover_unimate_root_quat_and_pos(data):Returns:(r_rot_quat (..., T), r_pos (..., T, 3))."""- r_rot_quat = Quaternions.from_transforms(rotation_6d_to_matrix_np(data[:, 3:9]))+ r_rot_quat = Quaternions(matrix_to_quaternion_np(rotation_6d_to_matrix_np(data[:, 3:9])))r_pos = np.zeros(data.shape[:-1] + (3,)) # (T, 3)r_pos[..., 1:, [0, 2]] = data[..., :-1, [9, 11]] # XZ velocitydiff --git a/unimate/utils/rotation_conversions.py b/unimate/utils/rotation_conversions.pyindex 5eeff7f..2b98492 100644--- a/unimate/utils/rotation_conversions.py+++ b/unimate/utils/rotation_conversions.py@@ -51,3 +51,34 @@ def rotation_6d_to_matrix_np(cont6d: np.ndarray) -> np.ndarray:z = z / np.linalg.norm(z, axis=-1, keepdims=True)y = np.cross(z, x, axis=-1)return np.concatenate([x[..., None], y[..., None], z[..., None]], axis=-1)+++def matrix_to_quaternion_np(mats: np.ndarray) -> np.ndarray:+ """Convert rotation matrices to unit quaternions (numpy version).++ Shepperd's method with one branch per matrix, chosen by the largest of+ ``|w|, |x|, |y|, |z|``. ``Quaternions.from_transforms`` (Motion library)+ applies every branch whose candidate ties for the largest, so a rotation+ with two equal components (e.g. exactly 90 deg about one axis) comes back+ as a different rotation; use this instead.++ Args:+ mats: Rotation matrices (*, 3, 3).++ Returns:+ Quaternions (*, 4) as ``(w, x, y, z)``, ``w >= 0``.+ """+ m = np.asarray(mats, dtype=np.float64)+ d0, d1, d2 = m[..., 0, 0], m[..., 1, 1], m[..., 2, 2]+ cand = np.stack([1 + d0 + d1 + d2, 1 + d0 - d1 - d2,+ 1 - d0 + d1 - d2, 1 - d0 - d1 + d2], axis=-1) # 4 * component^2+ branch = np.argmax(cand, axis=-1)+ s = 2.0 * np.sqrt(np.maximum(np.take_along_axis(cand, branch[..., None], -1)[..., 0], 1e-12))+ m21_m12, m02_m20, m10_m01 = m[..., 2, 1] - m[..., 1, 2], m[..., 0, 2] - m[..., 2, 0], m[..., 1, 0] - m[..., 0, 1]+ m21p12, m02p20, m10p01 = m[..., 2, 1] + m[..., 1, 2], m[..., 0, 2] + m[..., 2, 0], m[..., 1, 0] + m[..., 0, 1]+ q = np.where(branch[..., None] == 0, np.stack([s / 4, m21_m12 / s, m02_m20 / s, m10_m01 / s], -1),+ np.where(branch[..., None] == 1, np.stack([m21_m12 / s, s / 4, m10p01 / s, m02p20 / s], -1),+ np.where(branch[..., None] == 2, np.stack([m02_m20 / s, m10p01 / s, s / 4, m21p12 / s], -1),+ np.stack([m10_m01 / s, m02p20 / s, m21p12 / s, s / 4], -1))))+ q = q / np.linalg.norm(q, axis=-1, keepdims=True)+ return np.where(q[..., :1] < 0, -q, q)diff --git a/unimate/utils/text_emb_cache.py b/unimate/utils/text_emb_cache.pyindex be348f9..7eb71aa 100644--- a/unimate/utils/text_emb_cache.py+++ b/unimate/utils/text_emb_cache.py@@ -9,9 +9,10 @@ both: with a complete cache the encoder is never instantiated at all.**Per-token, not pooled.** Each text is stored as its ``(T_i, D)`` sequence ofnon-padding token features, because ``text_cond='cross_attn'`` needs thesequence โ a pooled vector would leave its cross-attention attending a single-token, i.e. a more expensive adaLN. The pooled vector ``adaLN`` wants is the-mean over those same rows, so it is derived on load rather than stored: one-source of truth, and it cannot drift from the tokens.+token, i.e. a more expensive adaLN. The pooled vector ``adaLN`` wants (the+mask-weighted mean the encoder computes) is stored beside the tokens as+``pooled``, the encoder's own number: a mean recomputed from the stored float32+rows would differ from it in the last bits.A caption key carries an embedding and a mask: ``emb`` is padded to the file'slongest sequence and ``mask`` marks its real tokens. ``load_cache`` returns@@ -21,16 +22,25 @@ which is where it would actually cost: truebones captions are 42% full, so aresident padded array would be 107 MB against 45 MB, and objaverse's 1.2 GBagainst 431 MB.-Two files per feature directory, flat beside ``captions.json`` โ captions and-joint names are separate sources with very different shapes (a caption is ~13-tokens, a joint name ~2), and only one of them changes when captions are-re-annotated::+Up to three files per feature directory, flat beside ``captions.json`` โ+captions and joint names are separate sources with very different shapes (a+caption is ~13 tokens, a joint name ~2), and only one of them changes when+captions are re-annotated::dataset/features/<ds>/caption_emb_cache.npz one entry per CLIP KEY, i.e.the keys of captions.json- keys : (N,) unicode the clip key to look up by- emb : (N, T_max, D) f32 per-token features, right-padded- mask : (N, T_max) bool True on this key's real tokens+ keys : (N,) unicode the clip key to look up by+ emb : (N, T_max, D) f32 per-token features, right-padded+ mask : (N, T_max) bool True on this key's real tokens+ texts : (N,) unicode the caption each entry encodes (optional)+ pooled: (N, D) f32 the encoder's pooled vector (required)++ dataset/features/<ds>/caption_generic_emb_cache.npz the same, for the+ generic captions of+ captions_generic.json+ dataset/features/<ds>/caption_detail_emb_cache.npz the same, for the+ detail captions of+ captions_detail.jsondataset/features/<ds>/joint_emb_cache.npz one entry per UNIQUE joint namekeys : (N,) unicode the joint name@@ -42,13 +52,20 @@ re-annotated::Joint names are stored pooled because that is all the model ever sees of them:``joint_names_emb`` is one vector per joint. Captions keep their sequence-because ``text_cond='cross_attn'`` attends it; the pooled vector ``adaLN`` uses-is the mean over those rows, derived by the data loader.+because ``text_cond='cross_attn'`` attends it; ``adaLN`` reads their stored+``pooled`` vector.-The encoder is no longer part of the filename, so ``meta`` is the only thing-standing between a t5 cache and a clip run: ``load_cache`` refuses a file whose-recorded encoder is not the one being asked for, alongside the format-version-and dimension checks.+**Stale entries.** Keys are clips, so a caption edited after the cache was+built would otherwise keep its old embedding. A file that stores ``texts`` is+checked on load against the current captions (``load_cache(..., texts=...)``):+an entry whose text differs is dropped, i.e. treated as a cache miss and+re-encoded. Files without ``texts`` cannot be checked; they are used with a+warning, and ``precompute_text_emb`` rebuilds them in full.++The encoder is not part of the filename, so ``meta`` is the only thing standing+between a t5 cache and a clip run: ``load_cache`` refuses a file whose recorded+encoder is not the one being asked for, alongside the format-version and+dimension checks.Written by ``unimate/tools/precompute_text_emb.py``; read by the mixturedataset. A missing or partial cache is never fatal โ the dataset encodes@@ -71,14 +88,16 @@ logger = get_logger(file_name=__file__)CACHE_FORMAT = 4-# The two text sources: the file each is cached in, and whether it is stored+# The text sources: the file each is cached in, and whether it is stored# pooled (one vector) or as a token sequence with a mask.-KINDS = {'caption': ('caption_emb_cache.npz', False), # captions.json- 'joint': ('joint_emb_cache.npz', True)} # cond.npy joint names+KINDS = {'caption': ('caption_emb_cache.npz', False), # captions.json+ 'caption_generic': ('caption_generic_emb_cache.npz', False), # captions_generic.json+ 'caption_detail': ('caption_detail_emb_cache.npz', False), # captions_detail.json+ 'joint': ('joint_emb_cache.npz', True)} # cond.npy joint namesdef cache_path(root_dir: str, kind: str) -> str:- """Path of one feature directory's cache for *kind* ('text' | 'joint').+ """Path of one feature directory's cache for *kind* (a key of ``KINDS``).Flat, beside ``captions.json`` and ``cond.npy`` โ the files it is derivedfrom โ rather than in a subdirectory of its own.@@ -137,7 +156,8 @@ def pool(tokens: np.ndarray) -> np.ndarray:def save_cache(path: str, tokens_by_key: Dict[str, np.ndarray], kind: str,encoder_type: str, encoder_version: Optional[str],- pooled_by_key: Optional[Dict[str, np.ndarray]] = None) -> int:+ pooled_by_key: Optional[Dict[str, np.ndarray]] = None,+ texts_by_key: Optional[Dict[str, str]] = None) -> int:"""Write ``{key: (T_i, D) tokens}`` to *path*. Returns the entry count.A pooled *kind* ('joint') stores one vector per key as ``(N, D)`` and no@@ -148,6 +168,8 @@ def save_cache(path: str, tokens_by_key: Dict[str, np.ndarray], kind: str,``pooled_by_key`` should come from :func:`pooled_from_hidden`; without itthe pooled vectors fall back to :func:`pool` over the sequences.+ ``texts_by_key`` (every key's source text) is stored so a later load can+ tell an entry built from an older caption; see :func:`load_cache`."""if not tokens_by_key:raise ValueError('refusing to write an empty text-embedding cache')@@ -183,6 +205,12 @@ def save_cache(path: str, tokens_by_key: Dict[str, np.ndarray], kind: str,arrays['pooled'] = np.stack([_pooled(k, x)for k, x in zip(keys, seqs)]) # (N, D)+ if texts_by_key is not None:+ missing = [k for k in keys if k not in texts_by_key]+ if missing:+ raise ValueError(f'texts_by_key lacks {len(missing)} keys, e.g. {missing[:3]}')+ arrays['texts'] = np.asarray([texts_by_key[k] for k in keys], dtype=object)+arrays['meta'] = np.asarray({'encoder_type': encoder_type,'encoder_version': encoder_version,'emb_dim': int(dim),@@ -201,14 +229,36 @@ def save_cache(path: str, tokens_by_key: Dict[str, np.ndarray], kind: str,return len(keys)+def cache_texts(root_dir: str, kind: str) -> Optional[Dict[str, str]]:+ """``{key: text}`` stored in a cache file, or None (no file / no texts)."""+ path = cache_path(root_dir, kind)+ if not os.path.isfile(path):+ return None+ try:+ data = np.load(path, allow_pickle=True)+ if 'texts' not in data.files:+ return None+ return {str(k): str(t) for k, t in zip(data['keys'], data['texts'])}+ except Exception: # noqa: BLE001 โ unreadable means unknown+ return None++def load_cache(root_dir: str, kind: str, encoder_type: str,encoder_version: Optional[str],- expected_dim: Optional[int] = None) -> Dict[str, np.ndarray]:+ expected_dim: Optional[int] = None,+ texts: Optional[Dict[str, str]] = None) -> Dict[str, np.ndarray]:"""Load one feature directory's *kind* cache.``{key: (D,)}`` for a pooled kind ('joint'); for a sequence kind- ('caption') ``{key: ((T_i, D) tokens, (D,) pooled)}`` โ the tokens trimmed- by their mask, the pooled vector as the encoder produced it.+ ('caption', 'caption_generic', 'caption_detail')+ ``{key: ((T_i, D) tokens, (D,) pooled)}`` โ+ the tokens trimmed by their mask, the pooled vector as the encoder+ produced it.++ With ``texts`` (the current ``{key: text}``), an entry whose stored text+ differs from it is dropped โ a cache miss the caller re-encodes โ so an+ edited caption never reuses its old embedding. A file without stored texts+ cannot be checked; it is used as is, with a warning.``{}`` when absent or unusable. Never raises on a bad file: a corrupt,outdated, wrong-encoder or dimension-mismatched cache is logged and@@ -244,6 +294,8 @@ def load_cache(root_dir: str, kind: str, encoder_type: str,mask = None if KINDS[kind][1] else np.asarray(data['mask'], dtype=bool)stored_pooled = (np.asarray(data['pooled'], dtype=np.float32)if 'pooled' in data.files else None)+ stored_texts = ([str(t) for t in data['texts']]+ if 'texts' in data.files else None)except Exception as e: # noqa: BLE001 โ a bad cache is a cache misslogger.warning(f'Ignoring unreadable text-embedding cache {path}: {e}')return {}@@ -258,9 +310,23 @@ def load_cache(root_dir: str, kind: str, encoder_type: str,f'features but the model expects {expected_dim}-d โ 'f'regenerate it with unimate/tools/precompute_text_emb.py')return {}+ keep = range(len(keys))+ if texts is not None:+ if stored_texts is None:+ logger.warning(+ f'{path} stores no caption texts, so entries built from an older '+ f'caption cannot be detected โ rebuild it with '+ f'unimate/tools/precompute_text_emb.py')+ else:+ keep = [i for i, k in enumerate(keys)+ if k not in texts or texts[k] == stored_texts[i]]+ stale = len(keys) - len(keep)+ if stale:+ logger.warning(f'{path}: {stale} entries encode an older text; '+ f'they will be re-encoded')if mask is None:- logger.info(f'Loaded {len(keys)} pooled text embeddings from {path}')- return {k: emb[i] for i, k in enumerate(keys)}+ logger.info(f'Loaded {len(keep)} pooled text embeddings from {path}')+ return {keys[i]: emb[i] for i in keep}if stored_pooled is None or len(stored_pooled) != len(keys):logger.warning(f'Ignoring text-embedding cache {path}: no stored pooled 'f'vectors โ regenerate it with '@@ -268,9 +334,9 @@ def load_cache(root_dir: str, kind: str, encoder_type: str,return {}# Boolean indexing copies, so the padded (N, T_max, D) block is released# once these per-key sequences are built โ the padding stays on disk.- logger.info(f'Loaded {len(keys)} cached text sequences '+ logger.info(f'Loaded {len(keep)} cached text sequences 'f'({int(mask.sum())} tokens) from {path}')- return {k: (emb[i][mask[i]], stored_pooled[i]) for i, k in enumerate(keys)}+ return {keys[i]: (emb[i][mask[i]], stored_pooled[i]) for i in keep}def load_caches(root_dirs: List[str], kind: str, encoder_type: str,