[SIGGRAPH Asia 2026] UniMate: One Unified Model to Animate Diverse Skeletons
Python
Announce the preview checkpoints; add two-level dataset sampling
Friedrich-Madbf9928c9ff
diff --git a/README.md b/README.mdindex 662561d..f65d870 100644--- a/README.md+++ b/README.md@@ -30,12 +30,13 @@## ๐ฅ News+- **[2026-09-27]** **Preview checkpoints** are released on [Hugging Face](https://huggingface.co/Linzhan/UniMate). ๐- **[2026-09-06]** The **training and inference code** is released. ๐- **[2026-08-30]** The raw **UniML3D dataset** and its [data-processing pipeline](data_process/) are released. ๐- **[2026-08-01]** Our [Interactive Demo](https://linzhanmou.com/unimate/interactive.html) is live โ browse our animation results in 3D. ๐ฎ- **[2026-07-18]** UniMate is accepted to SIGGRAPH Asia 2026! ๐-> **[TODO]** Pretrained checkpoints, the exact training configuration and data manifest used to produce them, evaluation scripts, and demo prompts will be released soon.+> **[Update]** Preview checkpoints are released at [Linzhan/UniMate](https://huggingface.co/Linzhan/UniMate); new checkpoints will be synced there as they are released.## ๐ ๏ธ Environment Setup@@ -87,7 +88,7 @@ Runs are configured by the JSON files in [`configs/`](configs/).The two axes are independent and all four combinations are implemented, so `full` x `adaln` and `graph` x `cross_attn` also run if you set them in a config; the two shipped pairings are the ones the paper compares.-Not shared across configs: `training.batch_size`, `training.num_steps` and `dataset.max_joints` are tuned per data combination (GPU memory tracks batch x frames x joints). `dataset.max_joints` is `100` for `truebones_*` / `mixamo_*` and `60` for `objaverse_*` / `uniml3d_*`; `dataset.min_joints` is `5` everywhere. Together they bound the skeleton sizes a run admits (object types outside the range are dropped) and, through what survives, the joint-axis padding width. Every other setting is identical.+Not shared across configs: `training.batch_size`, `training.num_steps`, `model.num_layers` and `dataset.max_joints` are tuned per data combination (GPU memory tracks batch x frames x joints; depth grows with the data: 6 layers for a single Truebones / Mixamo source, 8 for Objaverse, 10 for the full mixture). `dataset.max_joints` is `100` for `truebones_*` / `mixamo_*` and `60` for `objaverse_*` / `uniml3d_*`; `dataset.min_joints` is `5` everywhere. Together they bound the skeleton sizes a run admits (object types outside the range are dropped) and, through what survives, the joint-axis padding width. Mixamo is a single skeleton, so its configs also turn off the object-type balancing (a no-op with one type) and, by choice, the topology augmentations. Every other setting is identical.</details>@@ -122,7 +123,7 @@ Each run writes to `outputs/<experiment name>/`:Clips are drawn by a power-law-balanced sampler when `training.balanced` is set โ a type with `n` clips is sampled in proportion to `n^(1-sampler_alpha)`, so at the default `sampler_alpha = 0.5` a species with 100 clips is seen ten times as often as one with a single clip rather than a hundred times โ then augmented on the fly โ joint addition, leaf removal, chain pooling and per-bone length perturbation (`dataset.use_*_aug`) โ so the model sees more topologies than the data literally contains. Every clip is padded to `max_joints` on the joint axis and `max_motion_length` on the time axis, with masks carried alongside; nothing padded ever contributes to attention or to the loss.-Training is **flow matching** (`training.diff_model = "flow"`): the network predicts the velocity of a linear interpolant between noise and data, under a masked L2 loss plus an optional geodesic rotation term (`training.lambda_geo`). Conditioning is dropped with probability `model.cond_mask_prob` so the same weights serve the conditional and unconditional branches that classifier-free guidance interpolates at sampling time. AdamW with a cosine schedule and warmup, gradient clipping at `training.max_grad_norm`, and an EMA copy of the weights (`training.use_ema`) โ the copy inference loads by default.+Training is **flow matching** (`training.diff_model = "flow"`): the network predicts the velocity of a linear interpolant between noise and data, under a masked L2 loss plus two auxiliary terms computed on the reconstructed clean motion โ a geodesic rotation loss (`training.lambda_geo`) and a velocity-smoothness loss (`training.lambda_smooth`). Conditioning is dropped with probability `model.cond_mask_prob` so the same weights serve the conditional and unconditional branches that classifier-free guidance interpolates at sampling time. AdamW with a cosine schedule and warmup, gradient clipping at `training.max_grad_norm`, and an EMA copy of the weights (`training.use_ema`) โ the copy inference loads by default.</details>@@ -156,7 +157,7 @@ Given a rigged 3D asset and a text prompt, UniMate generates articulated motion<img src="assets/qualitative.png" alt="Qualitative results" width="100%"></div>-Sampling starts from the output directory of a training run (`config.json`, `dataset_stats.npy`, `checkpoints/`) โ pretrained checkpoints in the same layout are coming soon. The target skeleton โ T-pose and topology conditioning โ is taken from the dataset, so the `dataset/features/<dataset>/` directory the model was trained on must be present.+Sampling starts from the output directory of a training run (`config.json`, `dataset_stats.npy`, `checkpoints/`) โ the [released checkpoints](https://huggingface.co/Linzhan/UniMate) use the same layout. The target skeleton โ T-pose and topology conditioning โ is taken from the dataset, so the `dataset/features/<dataset>/` directory the model was trained on must be present.```bashpython -m unimate.inference.sample \diff --git a/configs/mixamo_60frames_full_cross_attn.json b/configs/mixamo_60frames_full_cross_attn.jsonindex 2c0bd91..078119e 100644--- a/configs/mixamo_60frames_full_cross_attn.json+++ b/configs/mixamo_60frames_full_cross_attn.json@@ -28,6 +28,7 @@"use_dataset_stats": true,"balanced_stats": false,"sampler_alpha": 0.5,+ "sampler_dataset_alpha": 0.0,"use_addition_aug": false,"use_removal_aug": false,"use_pooling_aug": false,@@ -70,6 +71,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/mixamo_60frames_graph_adaln.json b/configs/mixamo_60frames_graph_adaln.jsonindex 00cf6e9..ea8e42e 100644--- a/configs/mixamo_60frames_graph_adaln.json+++ b/configs/mixamo_60frames_graph_adaln.json@@ -28,6 +28,7 @@"use_dataset_stats": true,"balanced_stats": false,"sampler_alpha": 0.5,+ "sampler_dataset_alpha": 0.0,"use_addition_aug": false,"use_removal_aug": false,"use_pooling_aug": false,@@ -70,6 +71,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/objaverse_60frames_full_cross_attn.json b/configs/objaverse_60frames_full_cross_attn.jsonindex 896391e..6e37361 100644--- a/configs/objaverse_60frames_full_cross_attn.json+++ b/configs/objaverse_60frames_full_cross_attn.json@@ -28,6 +28,7 @@"use_dataset_stats": true,"balanced_stats": true,"sampler_alpha": 0.5,+ "sampler_dataset_alpha": 0.0,"use_addition_aug": true,"use_removal_aug": true,"use_pooling_aug": true,@@ -70,6 +71,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/objaverse_60frames_graph_adaln.json b/configs/objaverse_60frames_graph_adaln.jsonindex 3d11251..a6dfad6 100644--- a/configs/objaverse_60frames_graph_adaln.json+++ b/configs/objaverse_60frames_graph_adaln.json@@ -28,6 +28,7 @@"use_dataset_stats": true,"balanced_stats": true,"sampler_alpha": 0.5,+ "sampler_dataset_alpha": 0.0,"use_addition_aug": true,"use_removal_aug": true,"use_pooling_aug": true,@@ -70,6 +71,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/truebones_60frames_full_cross_attn.json b/configs/truebones_60frames_full_cross_attn.jsonindex 062d82e..51cc07f 100644--- a/configs/truebones_60frames_full_cross_attn.json+++ b/configs/truebones_60frames_full_cross_attn.json@@ -28,6 +28,7 @@"use_dataset_stats": true,"balanced_stats": true,"sampler_alpha": 0.5,+ "sampler_dataset_alpha": 0.0,"use_addition_aug": true,"use_removal_aug": true,"use_pooling_aug": true,@@ -70,6 +71,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/truebones_60frames_graph_adaln.json b/configs/truebones_60frames_graph_adaln.jsonindex 9533615..c6726ed 100644--- a/configs/truebones_60frames_graph_adaln.json+++ b/configs/truebones_60frames_graph_adaln.json@@ -28,6 +28,7 @@"use_dataset_stats": true,"balanced_stats": true,"sampler_alpha": 0.5,+ "sampler_dataset_alpha": 0.0,"use_addition_aug": true,"use_removal_aug": true,"use_pooling_aug": true,@@ -70,6 +71,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/uniml3d_60frames_full_cross_attn.json b/configs/uniml3d_60frames_full_cross_attn.jsonindex f2feae0..403a626 100644--- a/configs/uniml3d_60frames_full_cross_attn.json+++ b/configs/uniml3d_60frames_full_cross_attn.json@@ -72,6 +72,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/configs/uniml3d_60frames_graph_adaln.json b/configs/uniml3d_60frames_graph_adaln.jsonindex 1c14a38..9dfdbd4 100644--- a/configs/uniml3d_60frames_graph_adaln.json+++ b/configs/uniml3d_60frames_graph_adaln.json@@ -72,6 +72,7 @@"log_interval": 50,"save_interval": 10000,"lambda_geo": 0.5,+ "lambda_smooth": 0.1,"use_ema": true,"ema_decay": 0.9999,"max_grad_norm": 1.0,diff --git a/data_process/README.md b/data_process/README.mdindex 516ce88..c0492c6 100644--- a/data_process/README.md+++ b/data_process/README.md@@ -236,7 +236,9 @@ Each export directory receives:| `joint_count.json`, `clip_frames.json`, `summary.json` | joints per skeleton, frames per clip, dataset-level summary || `.completed/{asset}.json` | per-asset completion markers; delete one to force a re-export |-`--multi-worker N` shards a *directory* input over N Blender processes (objaverse, mixamo, auto mode). Each worker writes its own `*_worker{i}.json` summary shards, merged afterwards by `tools/merge_summaries.py`. Truebones is exported by a single-process exporter and ignores `--multi-worker`.+`--multi-worker N` shards a *directory* input over N Blender processes (objaverse, mixamo, auto mode). Each worker writes its own `*_worker{i}.json` summary shards, merged afterwards by `tools/merge_summaries.py` โ only when every worker exited cleanly; a worker that had failed assets exits non-zero and the merge is left for the next run. Truebones is exported by a single-process exporter and ignores `--multi-worker`.++An optional `excluded.csv` (column `object_id` or `file`) in the input directory or its parent lists asset stems the objaverse exporter skips entirely (`run_export_general.sh` does not read it).Skeleton pruning is joint across all of an asset's clips, so an asset only completes atomically โ the marker is written once every clip of that asset is on disk. Markers also carry the pruned joint names, which is what makes `joint_names.json` rebuildable from disk after an interrupted run.@@ -261,7 +263,7 @@ Each clip becomes `<clip>/v00{0..3}/*.png` (the per-frame PNGs the caption stage- **Frame cap.** Rendering stops at the first 200 frames per clip (`MAX_RENDER_FRAMES`, ~6.7 s at 30 fps) โ enough context for captioning without paying for full-length renders. Clips shorter than `MIN_ACTION_FRAMES` (default 5) are skipped.- **`--fps` must match stage 1.** glTF stores keyframe times in seconds, so the importer resamples them at the scene frame rate; a mismatch makes the renders cover a different time window than the NPZs. Default 30 in both stages.-- **Multi-GPU.** `--multi-worker N` shards assets over N processes. `CUDA_VISIBLE_DEVICES` pinning does **not** bind EEVEE's EGL context, so on a shared multi-GPU machine every worker still renders on the first GPU โ give each render job a cgroup / scheduler allocation that exposes exactly one GPU instead.+- **Multi-GPU.** `--multi-worker N` shards assets over N processes, and `NUM_GPUS=k` assigns worker `i` to `CUDA_VISIBLE_DEVICES=i % k`. That pinning does **not** bind EEVEE's EGL context, so on a shared multi-GPU machine every worker still renders on the first GPU โ give each render job a cgroup / scheduler allocation that exposes exactly one GPU instead. `--missing-only` exists for objaverse only; the wrapper rejects it for the other datasets, which resume per asset anyway.- Quality knobs (`RESOLUTION`, `SAMPLES`, `CAMERA_DIST`) are environment overrides on both wrappers; run either with `-h`.</details>@@ -318,7 +320,7 @@ python data_process/vlm_caption/caption_rewrite_llm.py \### Stage 3 โ Joint annotation: name cleanup & facing direction-Two tasks, each with a rule-based and an LLM variant: joint-name cleaning (`names_*`) maps raw bone names onto a canonical anatomical vocabulary, and facing-direction selection (`face_*`) picks the bilateral joint pair โ or head/tail body axis, for serpentine rigs โ that defines each rig's heading. The LLM variants default to `deepseek-v4-flash` (`MODEL=` switches to any `gpt-*` model or a local HF model id) and fall back to the rules on failure.+Two tasks, each with a rule-based and an LLM variant: joint-name cleaning (`names_*`) maps raw bone names onto a canonical anatomical vocabulary, and facing-direction selection (`face_*`) picks the bilateral joint pair โ or head/tail body axis, for serpentine rigs โ that defines each rig's heading. The LLM variants default to `deepseek-v4-flash` (`MODEL=` switches to any `gpt-*` model or a local HF model id). The two required passes fall back to the rules when a rig fails or exceeds its per-rig budget (`RIG_TIMEOUT`); the two refinement passes below instead keep the entry they were asked to correct.**Required** โ these two produce the artifacts stage 4 consumes:@@ -382,10 +384,12 @@ Applies the corrections found in a manual audit of the stage-2/3 outputs, so the| Output | Scope ||--------|-------|| `export/<ds>/filtered_clips.txt` | individual clips, any dataset |-| `export/objaverse/filtered_objects.txt` | whole rigs โ only the `tpose_wrong` flag (rest pose lies flat / rotated / upside-down) |+| `export/<ds>/activity_keep.txt` | `<clip>`: exempt from stage 4's low-activity filter โ a real motion confined to a few joints (a head turn, a clap, a wave) that the joint-activity gate cannot tell from a held pose; every other filter still applies |+| `export/<ds>/clip_trims.txt` | `<clip> <N>`: the clip's export NPZ starts N frames late โ stage 4 drops `motions/<clip>.npz` frames `0..N-1` on load โ for a bind-pose or foreign opening frame that snaps into the motion, so the clip is kept instead of filtered |+| `export/objaverse/filtered_objects.txt` | whole rigs โ only the `tpose_wrong` flag (rest pose lies flat / rotated / upside-down) and `not_in_legacy_raw` (raw GLB absent from the legacy raw set, held out of training) || `export/objaverse/rig_flags.json` | informational: `empty_pair`, `bone_pair`, `body_axis_unnamed`, `facing_wrong`, `object_no_front` |-Everything except `tpose_wrong` is informational โ those rigs are kept and trained with the facing they have. Per-rig evidence (rest-pose metrics, mesh T-pose grids and corrected-skeleton panels, all checked by eye) lives under `export/objaverse/tpose_abnormal_vis/`.+Every other flag is informational โ those rigs are kept and trained with the facing they have. Per-rig evidence (rest-pose metrics, mesh T-pose grids and corrected-skeleton panels, all checked by eye) lives under `export/objaverse/tpose_abnormal_vis/`.Truebones uses `filtered_clips.txt` for the three clips whose rig disagrees with their object's reference skeleton; objaverse uses it for the motion-discontinuity review.@@ -403,7 +407,7 @@ APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebonesNUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse```-Every object type is canonicalized against its T-pose (facing โ XZ-centering โ diameter scaling โ grounding), static pre/post-roll is trimmed, low-activity and motion-discontinuous clips are filtered, and the per-object topology condition is written to `cond.npy`. Output goes to `dataset/features/<dataset>/`: `motions/{object}-{motion}-{clip_idx}.npz`, `videos/`, `tpose/`, plus `cond.npy`, `captions.json`, `category_groups.json`, `filtered_clips.json`, and a `metadata.txt` report.+Every object type is canonicalized against its T-pose (facing โ XZ-centering โ diameter scaling โ grounding), static pre/post-roll is trimmed, low-activity and motion-discontinuous clips are filtered, and the per-object topology condition is written to `cond.npy`. Output goes to `dataset/features/<dataset>/`: `motions/{object}-{motion}-{clip_idx}.npz`, `videos/`, `tpose/`, plus `cond.npy` and a `metadata.txt` report, and โ whenever they have content โ `captions.json`, `category_groups.json` and `filtered_clips.json` (runtime-filtered clips and skipped object types; clips pre-filtered through `filtered_clips.txt` are dropped before processing and not listed).<details><summary><b>Details โ</b> filters and thresholds</summary>@@ -423,7 +427,7 @@ Every object type is canonicalized against its T-pose (facing โ XZ-centeringClips dropped at runtime and skipped object types are recorded in `filtered_clips.json`; `metadata.txt` reports the dataset totals.-Two hand-reviewed skip lists are read from the export directory before any processing, both **optional**: `filtered_clips.txt` (individual clips, every dataset) and `filtered_objects.txt` (whole rigs, objaverse only), both written by `tools/patch_annotations.py`. The matching flags take a path, `"auto"` (the default) or an empty string to disable. `--category_groups` follows the same convention for `category_groups.json`, which this stage only copies through.+Two hand-reviewed skip lists are read from the export directory before any processing, both **optional**: `filtered_clips.txt` (individual clips, every dataset) and `filtered_objects.txt` (whole rigs, objaverse only), both written by `tools/patch_annotations.py`, plus `clip_trims.txt` (`<clip> <N>`: drop the first N frames of a clip's export NPZ; `--clip_trims`) and `activity_keep.txt` (clips exempt from the low-activity filter; `--activity_keep`). The matching flags take a path, `"auto"` (the default) or an empty string to disable. `--category_groups` follows the same convention for `category_groups.json`, which this stage only copies through.</details>@@ -443,14 +447,16 @@ Used at inference time to render generated motions onto their rigged meshes:```bashANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse # stage-4 / generated clip + cond.npy-ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ rig+CHAR_PATH=char.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ rigbash data_process/scripts/run_animate_fbx.sh # raw animation FBXs โ any characterbash data_process/scripts/run_animate_mixamo.sh # batch over Mixamo export NPZs (Y Bot)+CHAR_PATH=asset.glb OUTPUT_DIR=outputs/asset bash data_process/scripts/run_preprocess_char.sh # one asset โ cond.npy + canonical rig+CHAR_PATH=outputs/asset/asset_canonical.glb ANIM_PATH=motion.npz bash data_process/scripts/run_animate_lbs.sh # NumPy FK + LBS, no cond needed```-`run_animate_motion.sh` accepts both the stage-4 feature NPZ and the `.npy` motion features written by the model's sampler. The character is auto-resolved for truebones / objaverse from the raw asset directories; mixamo needs `CHAR_PATH` (default `character_refined/Y_Bot.fbx`, the rig the animation FBXs are authored on).+`run_animate_motion.sh` accepts both the stage-4 feature NPZ and the `.npy` motion features written by the model's sampler. The character is auto-resolved for truebones / objaverse from the raw asset directories; mixamo defaults to `CHAR_PATH=dataset/raw/mixamo/character_refined/Y_Bot.fbx` (the rig the animation FBXs are authored on) when that file has been downloaded, and requires `CHAR_PATH` otherwise. The last two wrappers are described under [Custom Assets](#custom-assets).-Armature bones absent from the motion are handled by `--extra_bones_strategy`: `merge` (default) transfers their vertex weights to the nearest kept ancestor and removes them, `remove` deletes the bones and their vertices, `keep` leaves them un-keyed.+Armature bones absent from the motion are handled by `--extra_bones_strategy` on `animate_motion.py` / `animate_npz.py` (`EXTRA_BONES_STRATEGY=` on their wrappers and on `scripts/run_animate_motion.sh`): `merge` (default) transfers their vertex weights to the nearest kept ancestor and removes them, `remove` deletes the bones and their vertices, `keep` leaves them un-keyed. The FBX / raw-clip paths (`animate_fbx`, `animate_mixamo`, `animate_lbs`) leave undriven bones at rest.`run_animate_fbx.sh` needs no export stage at all: it bakes a raw animation clip โ or a whole directory of them โ onto any character that shares the clips' bone names (the Mixamo convention):@@ -500,7 +506,7 @@ CHAR_PATH=outputs/asset/<name>_canonical.glb ANIM_PATH=<motion.npz-or-directory>bash data_process/scripts/run_animate_lbs.sh # one animated GLB per action```-`animate_lbs` computes FK and Linear Blend Skinning manually in NumPy (Blender only parses the asset; the math is numerically identical to Blender's armature modifier); `SAVE=npz,obj` additionally dumps raw deformed vertex sequences / per-frame OBJs.+`animate_lbs` computes FK and Linear Blend Skinning manually in NumPy (Blender only parses the asset; the math is numerically identical to Blender's armature modifier). `SAVE` is the full list of outputs: `SAVE=glb,npz,obj` writes the rigged GLB plus raw deformed vertex sequences and per-frame OBJs (`SAVE=npz,obj` alone drops the GLB). A feature NPZ that is not from a canonical asset can still be driven by passing `DATASET_TYPE=` and/or `COND_PATH=` (a single-entry `cond.npy` is accepted whatever the clip is named).## Data Formats@@ -541,19 +547,19 @@ The motion representation stores per-frame deltas without dividing by the frame| Key | Shape | Description ||-----|-------|-------------|| `object_type` | str | object-type name (clip-name prefix) |-| `parents` | `(J,)` | parent indices in canonical BFS joint order |-| `offsets`, `tpos_offsets` | `(J, 3)` | bind-pose and canonical T-pose bone offsets |-| `joint_names`, `clean_joint_names` | `(J,)` | raw and cleaned bone names (stage 3) |-| `tpos_first_frame` | `(J, 3)` | canonical T-pose global joint positions |-| `tpos_local_rotations`, `tpos_global_rotations` | `(J, 4)` | T-pose local / global rotations (quaternions) |-| `joint_relations`, `joint_graph_dists` | `(J, J)` | pairwise edge-relation types and clamped topology distances |-| `joint_depths` | `(J,)` | depth in the kinematic tree |-| `edge_indexs` | `(2, 2(Jโ1))` | undirected edge list |-| `spectral_feats` | `(J, K)` | Laplacian eigenvectors (`K = --max_freqs`) |+| `parents` | list, len `J` | parent indices in canonical BFS joint order (`-1` = root) |+| `offsets`, `tpos_offsets` | `(J, 3)` float64 | local bone offsets of the canonicalized T-pose (scaled, root moved by centering / grounding) and offsets recomputed from `tpos_first_frame` |+| `joint_names`, `clean_joint_names` | list, len `J` | raw and cleaned bone names (stage 3) |+| `tpos_first_frame` | `(J, 3)` float64 | canonical T-pose global joint positions |+| `tpos_local_rotations`, `tpos_global_rotations` | `(J, 4)` float64 | T-pose local / global rotations (quaternions, `w, x, y, z`) |+| `joint_relations`, `joint_graph_dists` | `(J, J)` int16 | pairwise edge-relation types and topology distances clamped at `--max_path_len` |+| `joint_depths` | `(J,)` int64 | depth in the kinematic tree |+| `edge_indexs` | `(2, 2(Jโ1))` int64 | undirected edge list |+| `spectral_feats` | `(J, K)` float32 | Laplacian eigenvectors (`K = --max_freqs`) || `kinematic_chains` | list | root-to-leaf joint chains || `scale_factor`, `ground_height`, `ground_height_mode` | scalar | calibration applied to every clip of the type |-| `face_joint_idxs` | dict | `{r_hip, l_hip, body_axis}` indices of the facing pair (when annotated) |-| `captions` | dict | `{clip_stem: caption}` for the type's clips (also flattened into `captions.json`) |+| `face_joint_idxs` | dict | `{r_hip, l_hip, body_axis}` indices of the facing pair; `-1, -1` when the rig has no resolved pair (identity facing) |+| `captions` | dict | `{clip_stem: caption}` for the type's saved clips (also flattened into `captions.json`) |</details>@@ -576,13 +582,15 @@ The export directory is where every stage before feature extraction accumulates| `face_joint_names.json` | 3 | facing pair (or body axis) per rig || `failed_clean_names.txt`, `failed_face_joints.txt` | 3 | rigs that fell back to the rules || `filtered_clips.txt` | QA patch | clips stage 4 skips |+| `clip_trims.txt` | QA patch | clips whose first N frames stage 4 drops |+| `activity_keep.txt` | QA patch | clips stage 4's low-activity filter keeps || `filtered_objects.txt`, `rig_flags.json` | QA patch | objaverse only: rigs stage 4 skips, and the full flag list |</details>## Tools-Standalone utilities in `data_process/tools/`. The QA visualizers have wrappers (listed under [Stage 3](#stage-3--joint-annotation-name-cleanup--facing-direction)); the rest are run directly.+Standalone utilities in `data_process/tools/`. `vis_tpose.py` and `vis_tpose_facing.py` have wrappers (listed under [Stage 3](#stage-3--joint-annotation-name-cleanup--facing-direction)); the rest are run directly.| Tool | Purpose ||------|---------|diff --git a/data_process/feature_extraction/extract_features.py b/data_process/feature_extraction/extract_features.pyindex 39a6003..d909a3b 100644--- a/data_process/feature_extraction/extract_features.py+++ b/data_process/feature_extraction/extract_features.py@@ -12,7 +12,11 @@ Supports three dataset layouts via ``--dataset_type``:bare motion stems.An optional ``filtered_clips.txt`` listing individual clips to skip is read-for every layout.+for every layout, and an optional ``clip_trims.txt`` (``<clip stem> <N>``)+drops the first N frames of a clip's export NPZ on load, and an optional+``activity_keep.txt`` exempts listed clips from the low-activity filter. All+three are generated from the hand-reviewed patches by+``tools/patch_annotations.py``.Stage-2 (captions, category groups) and stage-3 (clean / face joint names)metadata JSONs are read from ``data_dir`` โ see@@ -41,6 +45,7 @@ import hashlibimport jsonimport multiprocessing as mpimport os+import zipfilefrom os.path import join as pjoinimport numpy as np@@ -148,13 +153,31 @@ def parse_args():"with optional '#' comments. Default 'auto' uses ""<data_dir>/filtered_clips.txt if present. Pass an ""empty string to disable. Complements the on-the-fly "- "quality filters; entries land in filtered_clips.json.")+ "quality filters. Listed clips are dropped before "+ "any object is processed, so they do NOT appear in "+ "filtered_clips.json (which records runtime filters).")parser.add_argument("--filtered_objects", type=str, default="auto",help="(objaverse only) Path to a txt file listing object ""types to skip (one per line; blanks and '#' comments ""ignored). Default 'auto' uses ""<data_dir>/filtered_objects.txt if present. Pass an ""empty string to disable.")+ parser.add_argument("--clip_trims", type=str, default="auto",+ help="Path to a txt file of hand-reviewed head trims, one "+ "'<clip stem> <N>' per line with optional '#' comments: "+ "the first N frames of that clip's export NPZ are "+ "dropped on load, before downsampling and the static "+ "trim (a bind-pose or foreign first frame that snaps "+ "into the motion). Default 'auto' uses "+ "<data_dir>/clip_trims.txt if present. Pass an empty "+ "string to disable.")+ parser.add_argument("--activity_keep", type=str, default="auto",+ help="Path to a txt file of clip stems (one per line, '#' "+ "comments) exempt from the low-activity filter: "+ "hand-reviewed small but real motions such as a head "+ "turn or a wave. Default 'auto' uses "+ "<data_dir>/activity_keep.txt if present. Pass an empty "+ "string to disable.")parser.add_argument("--category_groups", type=str, default="auto",help="Path to the body-plan category JSON copied into ""the feature dir (the training loader reads it for "@@ -208,6 +231,35 @@ def resolve_filtered_clips_path(path, data_dir):return _resolve_skip_list_path(path, data_dir, 'filtered_clips.txt')+def resolve_clip_trims_path(path, data_dir):+ return _resolve_skip_list_path(path, data_dir, 'clip_trims.txt')+++def resolve_activity_keep_path(path, data_dir):+ return _resolve_skip_list_path(path, data_dir, 'activity_keep.txt')+++def load_clip_trims(path):+ """Read ``<clip stem> <N>`` head trims; a falsy path yields an empty dict.++ Malformed lines raise instead of being skipped: a trim that silently does+ nothing would leave the snap it was written for in the training data.+ """+ trims = {}+ if not path:+ return trims+ with open(path, 'r') as f:+ for lineno, line in enumerate(f, 1):+ body = line.split('#', 1)[0].split()+ if not body:+ continue+ if len(body) != 2 or not body[1].isdigit() or int(body[1]) < 1:+ raise ValueError(f'{path}:{lineno}: expected "<clip stem> <N>" with '+ f'N >= 1, got {line.strip()!r}')+ trims[body[0]] = int(body[1])+ return trims++def resolve_category_groups(path, metadata):"""Return the category groups to copy into the feature dir, or None.@@ -215,7 +267,9 @@ def resolve_category_groups(path, metadata):the data dir; an empty string drops it; any other value is loaded as anexplicit path. Nothing in this stage consumes the groups โ they arepassed through to the feature dir โ so dropping them only means the- copy is made later, by hand, once the classifier has finished.+ copy is made later, by hand, once the classifier has finished. Like the+ two skip lists, a missing explicit path is skipped with a warning rather+ than aborting the run."""if not path:if metadata.pop('category_groups', None) is not None:@@ -224,7 +278,10 @@ def resolve_category_groups(path, metadata):if path != 'auto':groups = load_json(path)if groups is None:- raise ValueError(f'--category_groups {path!r} not found')+ logger.warning(f'--category_groups {path!r} not found; the feature '+ f'dir will get no category_groups.json')+ metadata.pop('category_groups', None)+ return Nonemetadata['category_groups'] = groupsreturn metadata.get('category_groups')@@ -349,9 +406,43 @@ def _cache_params(task):# export dirs are symlinks) does not invalidate every entry.params['clips_digest'] = _digest(sorted(os.path.basename(p) for p in task['object_npzs']))+ # Only objects with a head trim carry the key, so adding clip_trims.txt+ # re-processes exactly those object types and leaves every other cached+ # result valid.+ if task.get('head_trims'):+ params['head_trims_digest'] = _digest(task['head_trims'])+ # Same for activity_keep.txt: only objects owning a listed clip carry it.+ if task.get('activity_keep'):+ params['activity_keep_digest'] = _digest(sorted(task['activity_keep']))+ # Root offsets are baked into the export NPZs in place (same names), so+ # neither clips_digest nor any list above sees them; the quats recorded in+ # the NPZs are the only trace. Adding, changing or removing one therefore+ # re-processes exactly the owning object type.+ if task.get('root_offsets'):+ params['root_offsets_digest'] = _digest(task['root_offsets'])return params+# Written into an export NPZ by tools/patch_annotations.py (apply_root_offsets)+# when it bakes a root orientation fix in; keep the name in sync.+ROOT_OFFSET_KEY = 'root_offset_applied'+++def read_root_offsets(object_npzs):+ """``{clip stem: [w, x, y, z]}`` for the export NPZs carrying a baked-in+ root offset. Only the zip directory is read for the others, so this stays+ cheap over a whole dataset."""+ offsets = {}+ for path in object_npzs:+ with zipfile.ZipFile(path) as zf:+ if ROOT_OFFSET_KEY + '.npy' not in zf.namelist():+ continue+ with np.load(path) as d:+ offsets[os.path.basename(path)[:-4]] = [+ round(float(x), 6) for x in d[ROOT_OFFSET_KEY]]+ return offsets++def _params_hash(params):"""Stable hash of the cache-invalidating settings."""blob = json.dumps(params, sort_keys=True, default=str)@@ -436,6 +527,7 @@ def process_object_task(task):clip_prefix = task.pop('clip_prefix')params = _cache_params(task)params_hash = _params_hash(params)+ task.pop('root_offsets', None) # cache key only; process_object reads the NPZscached = _load_cached_result(part_path, object_type, params, params_hash)if cached is not None:@@ -482,8 +574,32 @@ def build_object_tasks(args, clip_stride, motion_dir, metadata):object_types, all_motions = discover_object_types(motion_dir, args.dataset_type, args)+ trims_path = resolve_clip_trims_path(args.clip_trims, args.data_dir)+ clip_trims = load_clip_trims(trims_path)+ if clip_trims:+ stems = {m[:-4] for m in all_motions}+ missing = sorted(set(clip_trims) - stems)+ logger.info(f'Loaded {len(clip_trims)} head trims from {trims_path}'+ + (f'; {len(missing)} name no motion being processed '+ f'(filtered or absent): {missing[:5]}' if missing else ''))++ keep_path = resolve_activity_keep_path(args.activity_keep, args.data_dir)+ activity_keep = load_filtered_objects(keep_path)+ if activity_keep:+ stems = {m[:-4] for m in all_motions}+ missing = sorted(activity_keep - stems)+ logger.info(f'Loaded {len(activity_keep)} low-activity exemptions from {keep_path}'+ + (f'; {len(missing)} name no motion being processed '+ f'(filtered or absent): {missing[:5]}' if missing else ''))+tasks = []for object_type in object_types:+ object_npzs = collect_object_npzs(object_type, args.dataset_type,+ motion_dir, all_motions)+ head_trims = {s: clip_trims[s] for s in+ (os.path.basename(p)[:-4] for p in object_npzs) if s in clip_trims}+ keep = sorted(s for s in (os.path.basename(p)[:-4] for p in object_npzs)+ if s in activity_keep)face_joints = get_object_face_joints(object_type, face_joint_names)if not face_joints:if args.dataset_type == 'mixamo':@@ -494,8 +610,10 @@ def build_object_tasks(args, clip_stride, motion_dir, metadata):tasks.append(dict(object_type=object_type,part_path=_object_part_path(args.save_dir, object_type),- object_npzs=collect_object_npzs(object_type, args.dataset_type,- motion_dir, all_motions),+ object_npzs=object_npzs,+ head_trims=head_trims,+ activity_keep=keep,+ root_offsets=read_root_offsets(object_npzs),# Mixamo clip files carry no "{object_type}-" prefix; its single# object type owns the whole directory.clip_prefix=('' if args.dataset_type == 'mixamo'diff --git a/data_process/joint_annotation/names_clean_llm.py b/data_process/joint_annotation/names_clean_llm.pyindex 389c3cd..bbc4614 100644--- a/data_process/joint_annotation/names_clean_llm.py+++ b/data_process/joint_annotation/names_clean_llm.py@@ -57,12 +57,15 @@ from data_process.joint_annotation.llm import ( # noqa: E402FatalLLMError,LLMClient,MAX_RETRIES,+ RigTimeout,add_llm_args,append_id_list,+ install_rig_alarm,load_json,read_id_list,save_json,strip_llm_noise,+ uninstall_rig_alarm,)@@ -290,46 +293,59 @@ def rule_based_clean(rig_id, raw_names):def clean_rig(rig_id, raw_names, client, system_prompt=SYSTEM_PROMPT,- max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True):+ max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True,+ timeout_seconds=0):# type: (...) -> Tuple[List[str], bool]"""Clean one rig's joint list via the LLM. Returns ``(names, used_fallback)``.Structural errors (length mismatch, bad JSON, ...) are fed back to the- LLM on the next attempt so it can self-correct.+ LLM on the next attempt so it can self-correct. ``timeout_seconds`` arms+ a per-rig SIGALRM (0 disables) so a hung backend cannot stall the batch;+ a timed-out rig falls back to the rules like an exhausted retry loop."""if not raw_names:return [], Falselast_err = Nonecorrection = None- for attempt in range(1, max_retries + 1):- try:- raw_text = client.generate(- system_prompt, build_user_prompt(rig_id, raw_names, correction=correction),- max_tokens)- return parse_response(raw_text, len(raw_names)), False- except FatalLLMError:- raise- except Exception as err: # noqa: BLE001 โ retry on any backend/parse error- last_err = err- if isinstance(err, ValueError):- correction = (- "Your previous response failed validation ({}). Return a "- "JSON array of EXACTLY {} strings โ one cleaned label per "- "input joint, in the same order. Do not add, merge, split, "- "or skip any entries. Output only the JSON array, no prose "- "or code fences."- ).format(err, len(raw_names))- logger.warning("[{}] attempt {}/{} failed: {}".format(- rig_id, attempt, max_retries, err))- if attempt < max_retries:- time.sleep(1.5)-- if not fallback_on_error:+ timed_out = False+ prev_handler = install_rig_alarm(timeout_seconds)+ try:+ for attempt in range(1, max_retries + 1):+ try:+ raw_text = client.generate(+ system_prompt, build_user_prompt(rig_id, raw_names, correction=correction),+ max_tokens)+ return parse_response(raw_text, len(raw_names)), False+ except (RigTimeout, FatalLLMError):+ raise+ except Exception as err: # noqa: BLE001 โ retry on any backend/parse error+ last_err = err+ if isinstance(err, ValueError):+ correction = (+ "Your previous response failed validation ({}). Return a "+ "JSON array of EXACTLY {} strings โ one cleaned label per "+ "input joint, in the same order. Do not add, merge, split, "+ "or skip any entries. Output only the JSON array, no prose "+ "or code fences."+ ).format(err, len(raw_names))+ logger.warning("[{}] attempt {}/{} failed: {}".format(+ rig_id, attempt, max_retries, err))+ if attempt < max_retries:+ time.sleep(1.5)+ except RigTimeout:+ timed_out = True+ last_err = "rig timeout after {}s".format(timeout_seconds)+ finally:+ uninstall_rig_alarm(prev_handler)++ if not fallback_on_error and not timed_out:raise RuntimeError("LLM cleaning failed for {}: {}".format(rig_id, last_err))- logger.error("[{}] all {} attempts failed ({}); falling back to rule-based "- "cleaner".format(rig_id, max_retries, last_err))+ logger.error("[{}] {} ({}); falling back to rule-based cleaner".format(+ rig_id,+ "timed out" if timed_out else "all {} attempts failed".format(max_retries),+ last_err))return rule_based_clean(rig_id, raw_names), True@@ -339,7 +355,8 @@ def clean_rig(rig_id, raw_names, client, system_prompt=SYSTEM_PROMPT,def clean_all(input_path, output_path, client, system_prompt=SYSTEM_PROMPT,max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True,- failed_path=None, overwrite=False, redo_failed=False):+ failed_path=None, overwrite=False, redo_failed=False,+ rig_timeout=0):# type: (...) -> Dict[str, List[str]]"""Clean every rig in ``input_path`` and write to ``output_path``.@@ -387,6 +404,7 @@ def clean_all(input_path, output_path, client, system_prompt=SYSTEM_PROMPT,rig_id, raw_names, client,system_prompt=system_prompt, max_tokens=max_tokens,max_retries=max_retries, fallback_on_error=fallback_on_error,+ timeout_seconds=rig_timeout,)fallback_count += int(used_fallback)cleaned[rig_id] = names_out@@ -439,6 +457,11 @@ def main():help="Also re-process the rig ids listed in --failed_log ""(they always have a full-length entry from the ""rule-based fallback, so they are otherwise skipped).")+ parser.add_argument("--rig_timeout", type=int, default=0,+ help="Per-rig wall-clock budget in seconds (SIGALRM); a "+ "rig that exceeds it falls back to the rule-based "+ "cleaner and is listed in --failed_log. 0 disables "+ "(default). Thinking backends need a generous value.")add_llm_args(parser, default_max_tokens=2048)args = parser.parse_args()@@ -457,6 +480,7 @@ def main():failed_path=args.failed_log,overwrite=args.overwrite,redo_failed=args.redo_failed,+ rig_timeout=args.rig_timeout,)with open(args.input, "r") as f:diff --git a/data_process/joint_annotation/names_clean_rule.py b/data_process/joint_annotation/names_clean_rule.pyindex 9c00973..00660c9 100644--- a/data_process/joint_annotation/names_clean_rule.py+++ b/data_process/joint_annotation/names_clean_rule.py@@ -15,6 +15,7 @@ import jsonimport osimport reimport sys+import tempfile_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))if _REPO_ROOT not in sys.path:@@ -583,8 +584,20 @@ def main():for animal, joints in data.items():clean_data[animal] = [post_process(clean_joint_name(j, animal)) for j in joints]- with open(output_path, "w") as f:- json.dump(clean_data, f, indent=2)+ # Atomic write (temp file + os.replace), like every other annotation+ # writer: a crash mid-dump must not leave a truncated+ # clean_joint_names.json that the LLM pass would then "resume" from.+ fd, tmp = tempfile.mkstemp(prefix=os.path.basename(output_path) + ".",+ suffix=".tmp",+ dir=os.path.dirname(output_path) or ".")+ try:+ with os.fdopen(fd, "w") as f:+ json.dump(clean_data, f, indent=2)+ os.replace(tmp, output_path)+ except BaseException:+ if os.path.exists(tmp):+ os.unlink(tmp)+ raiseprint("Written {} entries to {}".format(len(clean_data), output_path))diff --git a/data_process/mesh_animation/animate_lbs.py b/data_process/mesh_animation/animate_lbs.pyindex c971005..9093d2a 100644--- a/data_process/mesh_animation/animate_lbs.py+++ b/data_process/mesh_animation/animate_lbs.py@@ -178,7 +178,14 @@ def load_motion_local_mats(anim_path, dataset_type=None, cond_path=None):build_anim_from_npz,load_cond_data,)- assert dataset_type, "--dataset_type is required for feature-format motion"+ # Either flag resolves the cond: --dataset_type picks the dataset's+ # default cond.npy (and the 'mixamo' key), --cond_path names the file+ # directly (object type from the clip's filename prefix, or the file's+ # single entry โ a preprocess_char cond has exactly one).+ if not dataset_type and not cond_path:+ raise RuntimeError(+ "feature-format motion needs --dataset_type and/or --cond_path "+ "(omit both only for a canonical asset from preprocess_char)")cond_path = cond_path or COND_PATH_TEMPLATE.format(dataset_type=dataset_type)cond_data = load_cond_data(cond_path, anim_path, dataset_type)anim, rest_anim, _tpos, raw = build_anim_from_npz(anim_path, cond_data)diff --git a/data_process/mesh_animation/animate_motion.py b/data_process/mesh_animation/animate_motion.pyindex 6613b03..60b23c4 100644--- a/data_process/mesh_animation/animate_motion.py+++ b/data_process/mesh_animation/animate_motion.py@@ -222,9 +222,22 @@ def resolve_char_path(anim_path, dataset_type):def load_cond_data(cond_path, anim_path, dataset_type):+ """Pick the clip's cond entry out of ``cond_path``.++ The key is the object type resolved from the filename (or ``'mixamo'``).+ A cond file holding exactly one entry (``preprocess_char.py`` output, or+ a custom single-asset export) is accepted whatever the clip is called,+ since there is nothing to disambiguate.+ """object_type = resolve_object_type(anim_path, dataset_type)cond_dict = np.load(cond_path, allow_pickle=True).item()if object_type not in cond_dict:+ if len(cond_dict) == 1:+ (only_key,) = cond_dict.keys()+ logger.warning(+ f"object_type '{object_type}' not in {cond_path}; using its single "+ f"entry '{only_key}'")+ return cond_dict[only_key]raise KeyError(f"object_type '{object_type}' (resolved from '{anim_path}' under "f"dataset_type='{dataset_type}') not found in cond file {cond_path}; "diff --git a/data_process/mesh_animation/preprocess_char.py b/data_process/mesh_animation/preprocess_char.pyindex 8f44487..88bacaf 100644--- a/data_process/mesh_animation/preprocess_char.py+++ b/data_process/mesh_animation/preprocess_char.py@@ -274,9 +274,15 @@ def bake_canonical_asset(char_path, cond, out_base, formats=('glb',)):def preprocess_asset(char_path, output_dir, face_r=None, face_l=None,body_axis=False, target_diameter=2.0, formats=('glb',),- save_vis=False, apply_clip=False, max_clip_len=10 ** 8,- keep_intermediate=False):- """Export + extract + bake one asset; see the module docstring."""+ save_vis=False, apply_clip=False, max_clip_len=200,+ clip_stride=110, keep_intermediate=False):+ """Export + extract + bake one asset; see the module docstring.++ ``max_clip_len`` / ``clip_stride`` mirror stage 4's ``--max_clip_len`` and+ ``max_clip_len - diffusion_max_len``: without ``apply_clip`` a motion is+ truncated to ``max_clip_len`` frames, with it the motion is cropped into+ overlapping windows of that length.+ """assert os.path.exists(char_path), f"Asset not found: {char_path}"name = os.path.splitext(os.path.basename(char_path))[0]os.makedirs(output_dir, exist_ok=True)@@ -305,6 +311,7 @@ def preprocess_asset(char_path, output_dir, face_r=None, face_l=None,face_joints=face_joints, clean_names=clean_names,expected_names=raw_names,apply_clip=apply_clip, max_clip_len=max_clip_len,+ clip_stride=clip_stride,target_diameter=target_diameter, save_vis=save_vis)assert cond is not None, ("feature extraction skipped this asset ""(degenerate skeleton or no usable clips โ see log)")@@ -363,8 +370,17 @@ def parse_args():parser.add_argument("--save_vis", action='store_true',help="Also render per-clip preview MP4s (slow).")parser.add_argument("--apply_clip", action='store_true',- help="Crop motions into overlapping training-length clips "- "instead of one full-length feature NPZ per motion.")+ help="Crop motions into overlapping windows of "+ "--max_clip_len frames (stride max_clip_len - "+ "diffusion_max_len) instead of truncating each "+ "motion to --max_clip_len frames.")+ parser.add_argument("--max_clip_len", type=int, default=200,+ help="Frames kept per motion / window length "+ "(stage-4 default: 200).")+ parser.add_argument("--diffusion_max_len", type=int, default=90,+ help="Training crop length used to derive the window "+ "stride max_clip_len - diffusion_max_len "+ "(stage-4 default: 90).")parser.add_argument("--keep_intermediate", action='store_true',help="Keep export/ NPZs and T-pose/preview visuals (default: ""only canonical asset + cond.npy + motions/ remain).")@@ -373,6 +389,10 @@ def parse_args():if __name__ == "__main__":args = parse_args()+ if args.apply_clip and args.max_clip_len <= args.diffusion_max_len:+ raise SystemExit(+ f"--apply_clip needs max_clip_len ({args.max_clip_len}) > "+ f"diffusion_max_len ({args.diffusion_max_len}) for a positive stride")preprocess_asset(char_path=args.char_path,output_dir=args.output_dir,@@ -383,5 +403,7 @@ if __name__ == "__main__":formats=tuple(f.strip().lower() for f in args.formats.split(',') if f.strip()),save_vis=args.save_vis,apply_clip=args.apply_clip,+ max_clip_len=args.max_clip_len,+ clip_stride=args.max_clip_len - args.diffusion_max_len,keep_intermediate=args.keep_intermediate,)diff --git a/data_process/motion_export/export_general.py b/data_process/motion_export/export_general.pyindex 9b5d427..55d2224 100644--- a/data_process/motion_export/export_general.py+++ b/data_process/motion_export/export_general.py@@ -257,23 +257,27 @@ def main():if saved:all_joint_names.update(saved)except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1logger.exception(f"Failed to export {asset_path}")with open(error_log, 'a') as log_file:log_file.write(f"Failed to export {asset_path}: {e}\n")worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""- write_export_summary(args.output_dir, all_joint_names,+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,worker_suffix=worker_suffix)- else:- saved = export_asset(args.input, args.output_dir, save_name=args.name,- **export_kwargs)- if saved:- write_export_summary(args.output_dir, saved)+ # Summaries are written first so a partial batch still leaves the+ # canonical JSONs consistent; the non-zero exit then tells the+ # wrapper not to merge shards and Slurm afterok chains to stop.if n_failed:logger.error(f"{n_failed}/{len(asset_paths)} assets failed; "f"see {error_log}")sys.exit(1)+ else:+ saved = export_asset(args.input, args.output_dir, save_name=args.name,+ **export_kwargs)+ if saved:+ write_export_summary(args.output_dir, saved, fps=args.fps)if __name__ == "__main__":diff --git a/data_process/motion_export/export_mixamo.py b/data_process/motion_export/export_mixamo.pyindex 0cc849a..dbc67a6 100644--- a/data_process/motion_export/export_mixamo.py+++ b/data_process/motion_export/export_mixamo.py@@ -217,7 +217,8 @@ def main(args):ref_bone_names = json.load(f).get("mixamo")all_joint_names = {"mixamo": ref_bone_names} if ref_bone_names else {}- write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,+ worker_suffix=worker_suffix)if n_failed:logger.error(f"{n_failed}/{len(anim_paths)} clips failed; see {error_log}")diff --git a/data_process/motion_export/export_objaverse.py b/data_process/motion_export/export_objaverse.pyindex c2b0818..7493cc2 100644--- a/data_process/motion_export/export_objaverse.py+++ b/data_process/motion_export/export_objaverse.py@@ -188,7 +188,8 @@ def main():log_file.write(f"Failed to export {glb_path}: {e}\n")worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""- write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,+ worker_suffix=worker_suffix)if n_failed:logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")diff --git a/data_process/motion_export/export_truebones.py b/data_process/motion_export/export_truebones.pyindex 5edb21c..f578146 100644--- a/data_process/motion_export/export_truebones.py+++ b/data_process/motion_export/export_truebones.py@@ -253,7 +253,7 @@ def main():with open(error_log, 'a') as log_file:log_file.write(f"Failed to export species {species}: {e}\n")- write_export_summary(args.output_dir, all_joint_names)+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps)if n_failed:logger.error(f"{n_failed}/{len(by_species)} species failed; see {error_log}")sys.exit(1)diff --git a/data_process/motion_rendering/render_tpose.py b/data_process/motion_rendering/render_tpose.pyindex f4414a8..8de3e2e 100644--- a/data_process/motion_rendering/render_tpose.py+++ b/data_process/motion_rendering/render_tpose.py@@ -15,6 +15,7 @@ Usage (plain python โ EEVEE needs the pip ``bpy`` module's GPU context):"""import os+import sysimport argparsefrom pathlib import Path@@ -154,14 +155,27 @@ def main():error_log = os.path.join(args.output_dir,f'tpose_errors_worker{args.worker_id}.log')+ n_failed = 0for name, path in assets:try:render_asset_tpose(name, path, args.output_dir, args)- except Exception as e:- logger.error(f"Failed to render {path}: {e}")+ except Exception as e: # noqa: BLE001 โ keep the batch going+ n_failed += 1+ logger.exception(f"Failed to render {path}")with open(error_log, 'a') as f:f.write(f"{path}\t{e}\n")+ if n_failed:+ logger.error(f"{n_failed}/{len(assets)} assets failed; see {error_log}")+ return 1 if n_failed else 0+if __name__ == "__main__":- main()+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures+ # from `set -e` and from Slurm's afterok dependencies (same guard as the+ # motion renderers).+ try:+ sys.exit(main())+ except Exception:+ logger.exception("render_tpose failed")+ sys.exit(1)diff --git a/data_process/scripts/run_animate_lbs.sh b/data_process/scripts/run_animate_lbs.shold mode 100644new mode 100755diff --git a/data_process/scripts/run_animate_motion.sh b/data_process/scripts/run_animate_motion.shindex d42f44d..7c9ab7b 100755--- a/data_process/scripts/run_animate_motion.sh+++ b/data_process/scripts/run_animate_motion.sh@@ -11,10 +11,13 @@## Env overrides:# ANIM_PATH (required) motion .npz, or a .npy of model motion features-# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; required for mixamo)+# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; mixamo+# defaults to dataset/raw/mixamo/character_refined/Y_Bot.fbx)# COND_PATH cond.npy (default: dataset/features/<dataset>/cond.npy)# OUTPUT_DIR (default: outputs/animated)# ANIM_MODE fk|ik, .npy input only (default: fk)+# EXTRA_BONES_STRATEGY merge|remove|keep for armature bones absent from the+# motion (default: merge, see animate_motion.py)set -euo pipefailsource "$(dirname "${BASH_SOURCE[0]}")/_common.sh"@@ -29,13 +32,23 @@ ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion .npz/.npy for dataset=$DATASET}OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}ANIM_MODE=${ANIM_MODE:-fk}+# Mixamo has no per-object mesh to auto-resolve; the animation FBXs are+# authored on Y Bot, so that character is the default when it is downloaded.+MIXAMO_DEFAULT_CHAR=dataset/raw/mixamo/character_refined/Y_Bot.fbx+if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then+ if [[ -f "$MIXAMO_DEFAULT_CHAR" ]]; then+ CHAR_PATH=$MIXAMO_DEFAULT_CHAR+ else+ echo "dataset=mixamo requires CHAR_PATH (default $MIXAMO_DEFAULT_CHAR not found;" \+ "download it with run_download.sh mixamo)" >&2+ exit 1+ fi+fi+EXTRA_ARGS=()[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")[[ -n "${COND_PATH:-}" ]] && EXTRA_ARGS+=(--cond_path="$COND_PATH")-if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then- echo "dataset=mixamo requires CHAR_PATH (no auto-resolved character mesh)" >&2- exit 1-fi+[[ -n "${EXTRA_BONES_STRATEGY:-}" ]] && EXTRA_ARGS+=(--extra_bones_strategy="$EXTRA_BONES_STRATEGY")blender -b -P data_process/mesh_animation/animate_motion.py -- \--dataset_type="$DATASET" \diff --git a/data_process/scripts/run_animate_npz.sh b/data_process/scripts/run_animate_npz.shindex 6fe9f9b..c30055b 100755--- a/data_process/scripts/run_animate_npz.sh+++ b/data_process/scripts/run_animate_npz.sh@@ -10,6 +10,8 @@# ANIM_PATH (required โ NPZ whose skeleton matches the rig)# OUTPUT_DIR (default: outputs/animated)# CHAR_ANIM_TYPE glb|fbx (default: glb)+# EXTRA_BONES_STRATEGY merge|remove|keep for armature bones absent from the+# NPZ (default: merge, see animate_npz.py)set -euo pipefailsource "$(dirname "${BASH_SOURCE[0]}")/_common.sh"@@ -21,8 +23,11 @@ ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion NPZ matching the rig}OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-glb}+EXTRA_ARGS=()+[[ -n "${EXTRA_BONES_STRATEGY:-}" ]] && EXTRA_ARGS+=(--extra_bones_strategy="$EXTRA_BONES_STRATEGY")+blender -b -P data_process/mesh_animation/animate_npz.py -- \--char_path="$CHAR_PATH" \--anim_path="$ANIM_PATH" \--output_dir="$OUTPUT_DIR" \- --char_anim_type="$CHAR_ANIM_TYPE" "$@"+ --char_anim_type="$CHAR_ANIM_TYPE" "${EXTRA_ARGS[@]}" "$@"diff --git a/data_process/scripts/run_caption_motion.sh b/data_process/scripts/run_caption_motion.shindex 6e4b5ec..de38b24 100755--- a/data_process/scripts/run_caption_motion.sh+++ b/data_process/scripts/run_caption_motion.sh@@ -48,7 +48,7 @@ doneMODEL=${MODEL:-Qwen/Qwen3.5-9B}if [[ "${MODEL,,}" == *qwen* ]]; thenbackend=local # backends.py: BACKEND_QWEN-elif [[ "$MODEL" == gemini* ]]; then+elif [[ "${MODEL,,}" == *gemini* ]]; thenbackend=geminielsebackend=openai@@ -80,7 +80,8 @@ MAX_RETRIES=${MAX_RETRIES:-4}EXTRA=()case "$backend" inlocal)- MAX_TOKENS=${MAX_TOKENS:-800} ;;+ MAX_TOKENS=${MAX_TOKENS:-800}+ EXTRA+=(--max_retries="$MAX_RETRIES") ;;gemini)# Reasoning/answer tokens share one budget -> generous default.# Default 2 workers โ token/minute quotas are easy to exhaust with@@ -131,7 +132,7 @@ if [[ "$MODE" == "multi-gpu" ]]; thenpids=()for gpu_id in $(seq 0 $((num_gpus - 1))); do- python -m data_process.vlm_caption.caption_motion "${COMMON[@]}" \+ python -m data_process.vlm_caption.caption_motion "${COMMON[@]}" "${EXTRA[@]}" \--num_gpus="$num_gpus" --gpu_id="$gpu_id" \"${PASSTHRU[@]}" > >(tee "$LOG_DIR/gpu${gpu_id}.log") 2>&1 &pids+=($!)diff --git a/data_process/scripts/run_export.sh b/data_process/scripts/run_export.shindex 352a167..685b924 100755--- a/data_process/scripts/run_export.sh+++ b/data_process/scripts/run_export.sh@@ -30,6 +30,13 @@ handle_help "$@"MODE=autocase "${1:-}" intruebones|mixamo|objaverse|auto) MODE="$1"; shift ;;+ ""|--*) ;; # no dataset given: auto mode (DATA_DIR required), flags follow+ *)+ # A bare word that is not a dataset name is almost always a typo;+ # silently treating it as an exporter flag would land in auto mode+ # and fail later on a missing DATA_DIR.+ echo "ERROR: unknown dataset '$1' (expected truebones | mixamo | objaverse | auto)" >&2+ exit 2 ;;esacNUM_WORKERS=1diff --git a/data_process/scripts/run_export_general.sh b/data_process/scripts/run_export_general.shold mode 100644new mode 100755diff --git a/data_process/scripts/run_joints_face_correct_llm.sh b/data_process/scripts/run_joints_face_correct_llm.shindex 7cca9b0..ac01492 100755--- a/data_process/scripts/run_joints_face_correct_llm.sh+++ b/data_process/scripts/run_joints_face_correct_llm.sh@@ -26,7 +26,10 @@ RAW=${RAW:-$(export_dir "$DATASET")/joint_names.json}CLEANED=${CLEANED:-$(export_dir "$DATASET")/clean_joint_names.json}FACE=${FACE:-$(export_dir "$DATASET")/face_joint_names.json}MODEL=${MODEL:-deepseek-v4-flash}-MAX_TOKENS=${MAX_TOKENS:-512}+# Matches the script's own default: this pass forces reasoning_effort=medium,+# and on gpt-5* models the reasoning tokens share this budget (512 hit+# finish_reason='length').+MAX_TOKENS=${MAX_TOKENS:-2048}# Thinking backends need a much larger per-rig budget (see names_correct wrapper).if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fidiff --git a/data_process/scripts/run_joints_names_clean_llm.sh b/data_process/scripts/run_joints_names_clean_llm.shindex 53b688d..bf7b086 100755--- a/data_process/scripts/run_joints_names_clean_llm.sh+++ b/data_process/scripts/run_joints_names_clean_llm.sh@@ -6,8 +6,10 @@# MODEL=deepseek-v4-flash bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # DeepSeek API# MODEL=Qwen/Qwen3-8B bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # local HF model#-# Env overrides: INPUT, OUTPUT, MODEL, MAX_TOKENS. Extra args are passed through-# (e.g. --no_fallback). API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.+# Env overrides: INPUT, OUTPUT, MODEL, MAX_TOKENS, RIG_TIMEOUT (per-rig budget+# in seconds; default 300 for deepseek*, else 60). Extra args are passed+# through (e.g. --no_fallback, --overwrite, --redo_failed). API backends read+# OPENAI_API_KEY / DEEPSEEK_API_KEY.set -euo pipefailsource "$(dirname "${BASH_SOURCE[0]}")/_common.sh"@@ -22,8 +24,11 @@ INPUT=${INPUT:-$(export_dir "$DATASET")/joint_names.json}OUTPUT=${OUTPUT:-$(export_dir "$DATASET")/clean_joint_names.json}MODEL=${MODEL:-deepseek-v4-flash}MAX_TOKENS=${MAX_TOKENS:-2048}+# One batched call per rig; thinking backends need a much larger budget (same+# split as the other stage-3 wrappers).+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-60}; fipython -m data_process.joint_annotation.names_clean_llm \--input="$INPUT" --output="$OUTPUT" \- --model="$MODEL" --max_tokens="$MAX_TOKENS" \+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \--report "$@"diff --git a/data_process/scripts/run_joints_vis_facing.sh b/data_process/scripts/run_joints_vis_facing.shold mode 100644new mode 100755diff --git a/data_process/scripts/run_preprocess_char.sh b/data_process/scripts/run_preprocess_char.shold mode 100644new mode 100755diff --git a/data_process/scripts/run_render_motion.sh b/data_process/scripts/run_render_motion.shindex 3056677..bb81a40 100755--- a/data_process/scripts/run_render_motion.sh+++ b/data_process/scripts/run_render_motion.sh@@ -42,6 +42,14 @@ EXTRA_ARGS=()while [[ $# -gt 0 ]]; docase "$1" in--multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;+ --missing-only)+ # Only render_objaverse.py implements the flag; the other two+ # renderers would reject it from argparse after loading bpy.+ if [[ "$DATASET" != "objaverse" ]]; then+ echo "ERROR: --missing-only is only supported for objaverse (the other renderers resume per asset anyway)" >&2+ exit 2+ fi+ EXTRA_ARGS+=("$1"); shift ;;*) EXTRA_ARGS+=("$1"); shift ;;esacdonediff --git a/data_process/tools/patch_annotations.py b/data_process/tools/patch_annotations.pyindex fa7bab3..3078cfb 100644--- a/data_process/tools/patch_annotations.py+++ b/data_process/tools/patch_annotations.py@@ -24,11 +24,12 @@ place. Two kinds of fixes:- the ``clean`` fields of face_joint_names.json are re-synced;- objaverse rigs are flagged in ``rig_flags.json``: automatically when thefacing pair cannot be trusted (``source: empty``, a Bone/Bone pair, a body- axis through unnamed bones), plus the hand-reviewed categories. Only- ``tpose_wrong`` rigs are also written to ``filtered_objects.txt``, which- stage 4 skips; every other flag is informational and the rig is kept;- - caption grammar (``breaksdance`` -> ``breakdances``) and the per-dataset- subject phrase;+ axis through unnamed bones), plus the hand-reviewed categories. Only the+ ``FILTER_CATEGORIES`` rigs (``tpose_wrong``, ``not_in_legacy_raw``) are also+ written to ``filtered_objects.txt``, which stage 4 skips; every other flag is+ informational and the rig is kept;+ - caption grammar (``breaksdance`` -> ``breakdances``) and the one corpus+ subject phrase (``An object``, the same in all three datasets);- locomotion qualifiers grounded in the root trajectory: ``<verb> forward``with no root travel -> ``<verb> in place``; ``<verb> in place`` with cleartravel -> ``<verb> forward|backward`` when the travel is clearly along the@@ -43,9 +44,18 @@ place. Two kinds of fixes:<ds>_categories.json {rig: category} (moves the rig in category_groups.json)<ds>_filtered_clips.txt <clip> # <reason> (any dataset: individual clips ->export/<ds>/filtered_clips.txt, skipped by stage 4)+ <ds>_clip_trims.txt <clip> <N> # <reason> (any dataset: drop the first N frames of the clip's+ export NPZ -> export/<ds>/clip_trims.txt, applied by stage 4 on load)+ <ds>_activity_keep.txt <clip> # <reason> (any dataset: small but real motions exempt from+ stage 4's low-activity filter -> export/<ds>/activity_keep.txt)+ <ds>_root_offsets.json {clip: {"quat": [w, x, y, z], "reason": s}} (any dataset: the source+ file keyed the root joint with a constant extra rotation for the whole clip, so+ the animal is tilted; export/<ds>/motions/<clip>.npz is REWRITTEN with+ anim_local_rot[:, 0] = anim_local_rot[:, 0] * quat on every frame. Idempotent,+ and undone when the entry is removed; see apply_root_offsets)<ds>_rig_flags.txt <rig> # <category>: <reason> (objaverse: hand-reviewed rig flags ->- export/objaverse/rig_flags.json; only tpose_wrong also goes to- export/objaverse/filtered_objects.txt, which stage 4 skips)+ export/objaverse/rig_flags.json; only tpose_wrong / not_in_legacy_raw+ also go to export/objaverse/filtered_objects.txt, which stage 4 skips)Usage (from the repo root; ``dataset/export/<ds>`` may be symlinks):python data_process/tools/patch_annotations.py [--dry_run] [--datasets truebones mixamo]@@ -61,12 +71,27 @@ import globimport jsonimport osimport re+import sys+import tempfilefrom collections import Counter, OrderedDictimport numpy as np+# Usually run as a script (`python data_process/tools/patch_annotations.py`),+# so the repo root is not on sys.path and `data_process` is not importable.+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))++from data_process.utils.kinematics import ( # noqa: E402+ fk, frame_global, qmul, qrot, rest_global,+)+DATASETS = ('truebones', 'mixamo', 'objaverse')-SUBJECT = {'truebones': 'An animal', 'mixamo': 'A person', 'objaverse': 'An object'}+# One subject for the whole corpus. The three datasets are trained as one+# mixture, so a per-dataset subject ('An animal' / 'A person') would hand the+# text encoder a free dataset label and defeat the topology-agnostic+# conditioning. Keep this in sync with the `subject=` fields in+# data_process/vlm_caption/prompts.py.+SUBJECT = {ds: 'An object' for ds in DATASETS}# Rigs whose multi-segment legs / claw arms are labelled by chain position.CHAIN_RIGS = {@@ -113,50 +138,11 @@ LOCO_VERBS = ('walks|runs|trots|jogs|sprints|marches|gallops|crawls|slithers|swi'bounds|scurries|waddles|limps|sneaks|strides|paddles|glides|dashes|skips|struts|''creeps|prowls|stalks|shuffles|scampers|canters|flutters|soars|hovers|strolls|''wanders|races')+CAPTION_FRAMES = 200 # blender_render.MAX_RENDER_FRAMESFACING_COS = 0.8 # |cos(travel, facing)| needed to call travel forward / backwardLOCO_RE = re.compile(r'\b(%s)((?: \w+ly)?) (forward|in place)\b' % LOCO_VERBS)-# ----------------------------------------------------------------------------# Kinematics (numpy only; quaternions are w,x,y,z as written by the exporter)-# -----------------------------------------------------------------------------def qmul(a, b):- w1, x1, y1, z1 = a[..., 0], a[..., 1], a[..., 2], a[..., 3]- w2, x2, y2, z2 = b[..., 0], b[..., 1], b[..., 2], b[..., 3]- return np.stack([w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2,- w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,- w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2,- w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2], -1)---def qrot(q, v):- qv = np.concatenate([np.zeros(v.shape[:-1] + (1,)), v], -1)- return qmul(qmul(q, qv), q * np.array([1, -1, -1, -1]))[..., 1:]---def fk(local_pos, local_rot, parents):- n = len(parents)- g = np.zeros((n, 3))- gr = np.zeros((n, 4))- for i in range(n):- p = parents[i]- if p < 0:- g[i], gr[i] = local_pos[i], local_rot[i]- else:- g[i] = g[p] + qrot(gr[p], local_pos[i])- gr[i] = qmul(gr[p], local_rot[i])- return g---def rest_global(d):- return fk(d['rest_local_pos'], d['rest_local_rot'], d['parents'])---def frame_global(d, t=0):- return fk(d['anim_local_pos'][t], d['anim_local_rot'][t], d['parents'])--# ---------------------------------------------------------------------------# Helpers# ---------------------------------------------------------------------------@@ -175,12 +161,21 @@ def load_overrides(path):def save_json(path, obj, dry_run):+ """Atomic write (temp file in the target dir + ``os.replace``). The temp+ name is unique per process so two concurrent runs cannot clobber each+ other's half-written file."""if dry_run:return- tmp = path + '.tmp'- with open(tmp, 'w') as f:- json.dump(obj, f, indent=2, ensure_ascii=False)- os.replace(tmp, path)+ fd, tmp = tempfile.mkstemp(prefix=os.path.basename(path) + '.',+ suffix='.tmp', dir=os.path.dirname(path) or '.')+ try:+ with os.fdopen(fd, 'w') as f:+ json.dump(obj, f, indent=2, ensure_ascii=False)+ os.replace(tmp, path)+ except BaseException:+ if os.path.exists(tmp):+ os.unlink(tmp)+ raisedef object_type_of(ds, clip):@@ -397,14 +392,27 @@ def patch_face_pairs(ds, face, clean, names, overrides, stats, log):# them); every other category is just recorded in rig_flags.json.AUTO_FLAG = {'empty': 'empty_pair', 'bone/bone pair': 'bone_pair','body axis through unnamed bones': 'body_axis_unnamed'}-FILTER_CATEGORIES = ('tpose_wrong',)+FILTER_CATEGORIES = ('tpose_wrong', 'not_in_legacy_raw')++# One line per filtering category for the filtered_objects.txt header, so adding a+# category to FILTER_CATEGORIES does not leave the header describing only the first one.+FILTER_CATEGORY_NOTES = {+ 'tpose_wrong': 'the exported rest pose lies flat / is rotated / upside-down, so the stage-4 T-pose is unusable',+ 'not_in_legacy_raw': 'the raw GLB is absent from the legacy 6,974-asset raw set and is held out of training',+}def load_rig_flags(path, names, log):"""``patches/<ds>_rig_flags.txt``: one ``<rig> # <category>: <reason>`` per line โ hand-reviewed- rigs (``tpose_wrong`` = rest pose lying / rotated, ``facing_wrong`` = the facing pair does not give the- real front, ``object_no_front`` = prop without a front, ``verified_ok`` = hand-checked, clears an automatic- flag). Unknown rigs are reported and ignored."""+ rigs (``tpose_wrong`` = rest pose lying / rotated, ``not_in_legacy_raw`` = raw GLB absent from the legacy+ raw set and held out, ``facing_wrong`` = the facing pair does not give the real front, ``object_no_front``+ = prop without a front, ``verified_ok`` = hand-checked, clears an automatic flag). Unknown rigs are+ reported and ignored.++ A rig listed twice keeps the LAST line, which is how a later decision overrides an earlier+ one, but the override is reported: two lines with different categories mean the earlier+ verdict (often a ``verified_ok`` review) silently disappears from ``rig_flags.json``, and+ which one wins depends on the order of the file rather than on anything meaningful."""out = OrderedDict()if not os.path.isfile(path):return out@@ -418,7 +426,14 @@ def load_rig_flags(path, names, log):log.append(f'[objaverse] rig flag for unknown rig {rig!r} ignored')continuecat, _, reason = comment.strip().partition(':')- out[rig] = (cat.strip() or 'flagged', reason.strip())+ cat = cat.strip() or 'flagged'+ if rig in out:+ prev = out[rig][0]+ log.append(f'[objaverse] rig {rig!r} is listed twice in {os.path.basename(path)}: '+ f'{prev!r} then {cat!r}; ' ++ (f'{cat!r} wins and the {prev!r} entry is dropped'+ if prev != cat else 'the duplicate line is redundant'))+ out[rig] = (cat, reason.strip())return out@@ -444,6 +459,9 @@ def write_filtered_clips(ds, root, patch_dir, frames, dry_run, log):continueif frames and clip not in frames:log.append(f'[{ds}] filtered clip {clip!r} is not in clip_frames.json; kept anyway')+ if clip in clips:+ log.append(f'[{ds}] filtered clip {clip!r} is listed twice in '+ f'{os.path.basename(src)}; the last reason wins')clips[clip] = comment.strip()lines = ['# Individual export clips skipped by stage 4 (extract_features.py reads this file','# automatically; --filtered_clips auto). The clip-level twin of filtered_objects.txt.',@@ -460,6 +478,172 @@ def write_filtered_clips(ds, root, patch_dir, frames, dry_run, log):return clips+def write_clip_trims(ds, root, patch_dir, frames, dry_run, log):+ """export/<ds>/clip_trims.txt โ hand-reviewed head trims that stage 4 applies.++ patches/<ds>_clip_trims.txt holds ``<clip stem> <N> # <reason>`` lines: the+ first N frames of that clip are a bind pose or a foreign pose that snaps into+ the motion. N counts frames of the export NPZ itself (its own fps), so any+ reader applies ``anim[N:]``; stage 4 does so on load, before downsampling and+ its static trim. The NPZs are not rewritten: a stage-1 rerun would undo that.+ """+ src = os.path.join(patch_dir, f'{ds}_clip_trims.txt')+ path = os.path.join(root, 'clip_trims.txt')+ if not os.path.isfile(src):+ return {}+ trims = OrderedDict()+ with open(src) as f:+ for lineno, line in enumerate(f, 1):+ body, _, comment = line.partition('#')+ parts = body.split()+ if not parts:+ continue+ if len(parts) != 2 or not parts[1].isdigit() or int(parts[1]) < 1:+ raise ValueError(f'{src}:{lineno}: expected "<clip> <N>" with N >= 1, got {line.strip()!r}')+ clip, n = parts[0], int(parts[1])+ if frames and clip not in frames:+ log.append(f'[{ds}] clip trim {clip!r} is not in clip_frames.json; kept anyway')+ elif frames and n >= frames[clip]:+ raise ValueError(f'{src}:{lineno}: trimming {n} frames removes all {frames[clip]} of {clip}')+ if clip in trims:+ log.append(f'[{ds}] clip trim {clip!r} is listed twice in {os.path.basename(src)}; the last wins')+ trims[clip] = (n, comment.strip())+ lines = ['# Hand-reviewed head trims applied by stage 4 (extract_features.py reads this file',+ '# automatically; --clip_trims auto): "<clip> <N>" drops the first N frames of the',+ "# clip's export NPZ (motions/<clip>.npz, its own fps): anim_local_rot/pos[N:].",+ f'# Generated by data_process/tools/patch_annotations.py from '+ f'patches/{ds}_clip_trims.txt โ {len(trims)} clips.']+ for clip, (n, why) in trims.items():+ lines.append(f'{clip} {n} # {why}' if why else f'{clip} {n}')+ if not dry_run:+ with open(path, 'w') as f:+ f.write('\n'.join(lines) + '\n')+ log.append(f'[{ds}] clip_trims.txt: {len(trims)} clips, {sum(n for n, _ in trims.values())} frames dropped')+ return trims+++def write_activity_keep(ds, root, patch_dir, frames, dry_run, log):+ """export/<ds>/activity_keep.txt โ clips exempt from stage 4's low-activity filter.++ patches/<ds>_activity_keep.txt holds ``<clip stem> # <reason>`` lines: a real+ motion confined to a few joints (a head turn, a clap, a wave) that the joint-activity+ gate reads as a held pose. Stage 4 still applies every other filter to them.+ """+ src = os.path.join(patch_dir, f'{ds}_activity_keep.txt')+ path = os.path.join(root, 'activity_keep.txt')+ if not os.path.isfile(src):+ return {}+ keep = OrderedDict()+ with open(src) as f:+ for lineno, line in enumerate(f, 1):+ body, _, comment = line.partition('#')+ parts = body.split()+ if not parts:+ continue+ if len(parts) != 1:+ raise ValueError(f'{src}:{lineno}: expected "<clip>", got {line.strip()!r}')+ clip = parts[0]+ if frames and clip not in frames:+ log.append(f'[{ds}] activity keep {clip!r} is not in clip_frames.json; kept anyway')+ if clip in keep:+ log.append(f'[{ds}] activity keep {clip!r} is listed twice in {os.path.basename(src)}')+ keep[clip] = comment.strip()+ lines = ['# Hand-reviewed exemptions from the low-activity filter, applied by stage 4',+ '# (extract_features.py reads this file automatically; --activity_keep auto).',+ f'# Generated by data_process/tools/patch_annotations.py from '+ f'patches/{ds}_activity_keep.txt โ {len(keep)} clips.']+ for clip, why in keep.items():+ lines.append(f'{clip} # {why}' if why else clip)+ if not dry_run:+ with open(path, 'w') as f:+ f.write('\n'.join(lines) + '\n')+ log.append(f'[{ds}] activity_keep.txt: {len(keep)} clips')+ return keep+++ROOT_OFFSET_KEY = 'root_offset_applied' # the quat currently baked into this NPZ (w, x, y, z)+++def _save_npz_atomic(path, data):+ fd, tmp = tempfile.mkstemp(prefix=os.path.basename(path) + '.', suffix='.tmp',+ dir=os.path.dirname(path) or '.')+ try:+ with os.fdopen(fd, 'wb') as f:+ np.savez(f, **data)+ os.replace(tmp, path)+ except BaseException:+ if os.path.exists(tmp):+ os.unlink(tmp)+ raise+++def apply_root_offsets(ds, root, patch_dir, dry_run, log):+ """Remove a constant whole-body orientation error from export clips, in place.++ Some vendor FBX files key the root joint with a fixed extra rotation for the whole+ clip (Truebones: nine Dog clips rolled 44.6 deg about the pelvis's local X axis,+ Dog-Stare 134.6 deg, Flamingo-One_Leg_Bent 27.5 deg), so the animal floats tilted+ while every child joint animates correctly. patches/<ds>_root_offsets.json maps+ ``clip -> {"quat": [w, x, y, z], "reason": ...}``; the fix post-multiplies the+ root's local rotation on every frame, ``q0[t] <- q0[t] * quat``, which turns the+ whole body rigidly about the root in body coordinates (so it stays right when the+ clip turns). The quats were estimated by hand review and checked against Dog2, which+ carries the same animations upright.++ Unlike clip_trims this rewrites the NPZ, so every reader (stage 4, previews, mesh+ driving, audits) sees the fixed motion. To stay reproducible it is idempotent: the+ applied quat is stored in the NPZ under ``root_offset_applied``. A matching entry is+ skipped; a changed entry undoes the old quat first; an NPZ that carries the key but+ is no longer listed is restored. A stage-1 rerun writes fresh NPZs without the key,+ and the next run of this script fixes them again. Returns the clips whose NPZ+ changed (their cached root motion is stale).+ """+ want = load_overrides(os.path.join(patch_dir, f'{ds}_root_offsets.json'))+ mdir = os.path.join(root, 'motions')+ changed, n_ok = [], 0+ targets = set(want)+ for f in glob.glob(os.path.join(mdir, '*.npz')): # previously fixed clips no longer listed+ clip = os.path.basename(f)[:-4]+ if clip not in targets:+ with np.load(f, allow_pickle=True) as d:+ if ROOT_OFFSET_KEY in d.files:+ targets.add(clip)+ for clip in sorted(targets):+ path = os.path.join(mdir, clip + '.npz')+ if not os.path.isfile(path):+ log.append(f'[{ds}] root offset for {clip!r}: no export NPZ; ignored')+ continue+ with np.load(path, allow_pickle=True) as d:+ data = {k: d[k] for k in d.files}+ have = data.pop(ROOT_OFFSET_KEY, None)+ new = want.get(clip)+ q_new = None+ if new is not None:+ q_new = np.asarray(new['quat'], dtype=np.float64)+ q_new = q_new / np.linalg.norm(q_new)+ if have is not None and q_new is not None and np.allclose(have, q_new, atol=1e-6):+ n_ok += 1+ continue+ rot = data['anim_local_rot'].astype(np.float64)+ q0 = rot[:, 0]+ if have is not None: # undo what is baked in+ q0 = qmul(q0, np.asarray(have, dtype=np.float64) * np.array([1, -1, -1, -1]))+ if q_new is not None:+ q0 = qmul(q0, np.broadcast_to(q_new, q0.shape))+ data[ROOT_OFFSET_KEY] = q_new.astype(np.float32)+ rot[:, 0] = q0 / np.linalg.norm(q0, axis=-1, keepdims=True)+ data['anim_local_rot'] = rot.astype(data['anim_local_rot'].dtype)+ what = 'restored (entry removed)' if q_new is None else \+ f'{np.degrees(2 * np.arccos(min(1.0, abs(q_new[0])))):.1f} deg'+ log.append(f'[{ds}] root offset {clip}: {what}' + (' (dry run)' if dry_run else ''))+ if not dry_run:+ _save_npz_atomic(path, data)+ changed.append(clip)+ if want or changed:+ log.append(f'[{ds}] root_offsets: {len(want)} listed, {len(changed)} rewritten, {n_ok} already applied')+ return changed++def write_rig_flags(root, face, frames, manual, dry_run, log):"""rig_flags.json (every flagged rig with its category) + filtered_objects.txt (FILTER_CATEGORIES only)."""flags = OrderedDict()@@ -482,8 +666,10 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):out = OrderedDict([('_comment', 'Per-rig flags. Automatic facing checks (empty_pair: no bilateral joint pair; ''bone_pair: the pair consists of unnamed Bone joints; body_axis_unnamed) plus the '- 'hand-reviewed categories (tpose_wrong: the exported rest pose is unusable; facing_wrong: '- 'the pair does not give the real front). Only ' + ', '.join(FILTER_CATEGORIES) ++ 'hand-reviewed categories (tpose_wrong: the exported rest pose is unusable; '+ 'not_in_legacy_raw: the raw GLB is absent from the legacy raw set and is held out of '+ 'training; facing_wrong: the pair does not give the real front). Only '+ + ', '.join(FILTER_CATEGORIES) +' is written to filtered_objects.txt, which stage 4 skips; every other flag is ''informational and those rigs are kept. verified_ok lists rigs whose automatic flag was ''cleared after review.'),@@ -494,9 +680,9 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):('verified_ok', verified), # automatic flags cleared by hand review])lines = ['# Object types skipped by stage 4 (extract_features.py reads this file automatically):',- '# rigs flagged ' + ' / '.join(FILTER_CATEGORIES) + ' in patches/objaverse_rig_flags.txt โ the exported',- '# rest pose lies flat / is rotated / upside-down, so the stage-4 T-pose is unusable.',- '# Other flags (empty_pair, bone_pair, facing_wrong, object_no_front, ...) are recorded in',+ '# rigs flagged ' + ' / '.join(FILTER_CATEGORIES) + ' in patches/objaverse_rig_flags.txt.']+ lines += [f'# {c}: {FILTER_CATEGORY_NOTES[c]}' for c in FILTER_CATEGORIES if c in FILTER_CATEGORY_NOTES]+ lines += ['# Other flags (empty_pair, bone_pair, facing_wrong, object_no_front, ...) are recorded in','# rig_flags.json only and are NOT filtered.',f'# Generated by data_process/tools/patch_annotations.py โ {len(bad)} rigs, {len(clips)} clips.','# Delete this file to keep them (they then get the rest-pose facing as exported).']@@ -542,7 +728,7 @@ def root_motion(root, ds, clips, cache_path, face, log):"""Per clip: net XZ root displacement in body heights and the traveldirection relative to the facing pair at frame 0."""cache = load_json(cache_path, OrderedDict()) if cache_path else OrderedDict()- key = lambda c: f'{ds}/{c}'+ key = lambda c: f'{ds}/{c}@{CAPTION_FRAMES}' # window-scoped: pre-2026-09-21 full-clip rows are ignoredtodo = [c for c in clips if key(c) not in cache]if todo:log.append(f'[{ds}] computing root motion for {len(todo)} clips')@@ -553,7 +739,14 @@ def root_motion(root, ds, clips, cache_path, face, log):d = np.load(f, allow_pickle=True)g0 = rest_global(d)height = float(np.ptp(g0, axis=0).max()) or 1.0- r = d['anim_local_pos'][:, 0, :]+ # Only the first MAX_RENDER_FRAMES frames count. That is the window the+ # captioner was shown, the window the released video covers, and the only+ # window stage 4 keeps - so it is the window the caption describes.+ # Measuring endpoint-to-endpoint over the WHOLE clip silently breaks every+ # looping animation: the root returns to its start, net comes out 0.000,+ # and a clip that visibly crosses six body lengths gets pinned at+ # "in place" (and the reverse for clips that only travel after frame 200).+ r = d['anim_local_pos'][:CAPTION_FRAMES, 0, :]disp = r[-1] - r[0]net = float(np.hypot(disp[0], disp[2]) / height)direction = ''@@ -695,6 +888,21 @@ def run_dataset(ds, args, log):stats['manual-rig-flag'] += len(manual)stats['manual-filtered-clip'] += len(write_filtered_clips(ds, root, pd, frames, args.dry_run, log))+ stats['manual-clip-trim'] += len(+ write_clip_trims(ds, root, pd, frames, args.dry_run, log))+ stats['manual-activity-keep'] += len(+ write_activity_keep(ds, root, pd, frames, args.dry_run, log))+ rewritten = apply_root_offsets(ds, root, pd, args.dry_run, log)+ stats['root-offset-rewritten'] += len(rewritten)+ if rewritten and not args.dry_run:+ # the cached travel direction was measured against the old (tilted) facing+ cache_path = os.path.join(pd, '.root_motion_cache.json')+ cache = load_json(cache_path, OrderedDict())+ stale = [k for k in cache if k.split('@')[0] in {f'{ds}/{c}' for c in rewritten}]+ for k in stale:+ del cache[k]+ if stale:+ save_json(cache_path, cache, False)if groups is not None:patch_categories(ds, groups, load_overrides(os.path.join(pd, f'{ds}_categories.json')), names, stats, log)if caps is not None:diff --git a/data_process/tools/vis_caption.py b/data_process/tools/vis_caption.pynew file mode 100644index 0000000..689fe43--- /dev/null+++ b/data_process/tools/vis_caption.py@@ -0,0 +1,262 @@+#!/usr/bin/env python3+"""Render an export-stage skeleton as video with its caption burned in.++QA tool for the caption layer: it puts the text and the motion it is supposed+to describe in one frame, so a wrong direction, a missing second phase or a+body part that never moves is visible without cross-referencing anything.++Input is the export layer (``dataset/export/<ds>/motions/<clip>.npz`` plus+``motion_captions.json``), not the feature layer, so what you see is the same+data the renderer showed the captioner. Frames are truncated to+``--max_frames`` (default 1000), which is long enough that only a handful of+clips are cut at all. Note that the captioner itself only ever saw the first+``blender_render.MAX_RENDER_FRAMES`` (200) frames, so a mismatch beyond frame+200 is a gap in the caption's coverage, not necessarily a wrong caption; pass+``--max_frames 200`` to review exactly what it saw.++The skeleton is drawn on a world-anchored checkerboard floor with a contact+shadow and a root trail, so translation reads off the tiles scrolling+underfoot โ which is what makes "walks forward" vs "walks in place" legible.++The look follows the stage-4 clip previews in ``dataset/features/<ds>/videos``+(``motion_features.save_clip`` with ``vis_ground=True``): same renderer, same+``PREVIEW_FIGSIZE`` / ``PREVIEW_DPI``, and the caption alone as the title. Two+differences: the source is the export clip the captioner actually saw, at its+full joint count and in the asset's own frame, rather than the canonicalized+stage-4 clip; and the joints keep the renderer's plain red-root / blue-joint+scheme instead of stage 4's spectral palette, which recolours every skeleton+differently and makes body parts harder to follow across clips.++Usage (from the repo root):+ # one clip+ python -m data_process.tools.vis_caption truebones --clips Lion-Die++ # a random sample, each as its own MP4 plus one tiled grid video+ python -m data_process.tools.vis_caption mixamo --random 6 --grid++ # every captioned clip of a dataset, rendered in parallel and resumable+ python -m data_process.tools.vis_caption mixamo --all --workers 16++ # a contact sheet PNG as well (for pasting into a review)+ python -m data_process.tools.vis_caption truebones --random 4 --grid --sheet++Outputs land in ``outputs/caption_vis/<dataset>/``.+"""++import argparse+import json+import os+import random+import sys++import imageio+import numpy as np+from concurrent.futures import ProcessPoolExecutor++sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))++from data_process.utils.kinematics import clip_global # noqa: E402+from data_process.utils.motion_features import ( # noqa: E402+ PREVIEW_DPI, PREVIEW_FIGSIZE)+from data_process.utils.plotting import render_skeleton_motion_ground # noqa: E402++DATASETS = ('truebones', 'mixamo', 'objaverse')+MAX_RENDER_FRAMES = 200 # keep in sync with blender_render.MAX_RENDER_FRAMES+DEFAULT_MAX_FRAMES = 1000 # render cost is linear in frames; 1000 cuts almost nothing+++def export_dir(root, ds):+ return os.path.join(root, 'dataset', 'export', ds)+++def load_captions(root, ds):+ path = os.path.join(export_dir(root, ds), 'motion_captions.json')+ if not os.path.isfile(path):+ raise SystemExit('no captions for %s at %s' % (ds, path))+ with open(path) as f:+ return json.load(f)+++def load_clip(root, ds, clip, max_frames):+ """(parents, positions (T, J, 3)) for one export clip, Y-up as exported."""+ path = os.path.join(export_dir(root, ds), 'motions', clip + '.npz')+ if not os.path.isfile(path):+ raise SystemExit('no such clip: %s' % path)+ d = np.load(path, allow_pickle=True)+ return np.asarray(d['parents']), clip_global(d, max_frames=max_frames)+++def tile(frame_stacks, labels=None):+ """Tile per-clip frame stacks into one video, holding each clip's last frame."""+ n = len(frame_stacks)+ cols = int(np.ceil(np.sqrt(n)))+ rows = int(np.ceil(n / cols))+ T = max(len(f) for f in frame_stacks)+ H, W = frame_stacks[0].shape[1:3]+ out = np.full((T, rows * H, cols * W, 3), 255, dtype=np.uint8)+ for i, fr in enumerate(frame_stacks):+ held = np.concatenate([fr, np.repeat(fr[-1:], T - len(fr), axis=0)]) if len(fr) < T else fr+ r, c = divmod(i, cols)+ out[:, r * H:(r + 1) * H, c * W:(c + 1) * W] = held[:T, :H, :W]+ return out+++def _title(clip, caption, with_clip_name):+ return '%s\n%s' % (clip, caption) if with_clip_name else caption+++def _render_one(job):+ """Render one clip to <out_dir>/<clip>.mp4. Runs in a worker process."""+ (root, ds, clip, caption, out_dir, max_frames, elev, azim,+ trail, figsize, dpi, fps, with_clip_name) = job+ try:+ parents, pos = load_clip(root, ds, clip, max_frames)+ frames = render_skeleton_motion_ground(+ parents, pos, elev=elev, azim=azim, rotate_root=True,+ title=_title(clip, caption, with_clip_name), trail=trail,+ figsize=tuple(figsize), dpi=dpi)+ # Write to a sibling temp file and rename: a run that is killed+ # mid-encode must not leave a half-written MP4 behind, because+ # --skip_existing would then treat it as done on the next pass.+ final = os.path.join(out_dir, clip + '.mp4')+ tmp = os.path.join(out_dir, '.%s.tmp.mp4' % clip) # keep .mp4: imageio picks the plugin by extension+ imageio.mimwrite(tmp, frames, fps=fps)+ os.replace(tmp, final)+ return clip, len(pos), None+ except Exception as e: # one bad clip must not stop the batch+ return clip, 0, '%s: %s' % (type(e).__name__, e)+++def main():+ root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))+ ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])+ ap.add_argument('dataset', choices=DATASETS)+ ap.add_argument('--clips', nargs='+', default=None, help='clip names (export stems)')+ ap.add_argument('--random', type=int, default=0, help='sample N random clips instead')+ ap.add_argument('--clips_file', default=None,+ help='file with one clip name per line (# comments and blanks ignored); '+ 'for a subset too long for a command line, e.g. only the clips that '+ 'survive filtered_clips.txt / filtered_objects.txt')+ ap.add_argument('--all', action='store_true',+ help='every captioned clip of the dataset; implies --skip_existing '+ 'so an interrupted run resumes. Not compatible with --grid / '+ '--sheet, which hold every frame of every clip in memory.')+ ap.add_argument('--workers', type=int, default=1,+ help='render this many clips in parallel (one process each)')+ ap.add_argument('--skip_existing', action='store_true',+ help='leave clips that already have an MP4 in the output dir alone')+ ap.add_argument('--seed', type=int, default=0)+ ap.add_argument('--output_dir', default=None,+ help='default outputs/caption_vis/<dataset>')+ ap.add_argument('--fps', type=int, default=30)+ ap.add_argument('--max_frames', type=int, default=DEFAULT_MAX_FRAMES,+ help='0 = whole clip; default %d. Pass %d to see exactly the '+ 'range the captioner saw' % (DEFAULT_MAX_FRAMES, MAX_RENDER_FRAMES))+ ap.add_argument('--grid', action='store_true', help='also write one tiled MP4')+ ap.add_argument('--sheet', action='store_true', help='also write a contact-sheet PNG')+ ap.add_argument('--sheet_cols', type=int, default=6, help='frames per row in the sheet')+ # the stage-4 previews take render_skeleton_motion_ground's own defaults+ # (save_clip passes no camera), so match them here rather than inventing a view+ ap.add_argument('--elev', type=float, default=20.0)+ ap.add_argument('--azim', type=float, default=-60.0)+ ap.add_argument('--figsize', type=float, nargs=2, default=PREVIEW_FIGSIZE,+ help='default matches the stage-4 previews')+ ap.add_argument('--dpi', type=int, default=PREVIEW_DPI,+ help='default matches the stage-4 previews')+ ap.add_argument('--with_clip_name', action='store_true',+ help='put the clip name above the caption; off by default so the '+ 'frame carries the caption alone, like the stage-4 previews')+ ap.add_argument('--no_trail', action='store_true')+ args = ap.parse_args()++ caps = load_captions(root, args.dataset)+ if args.all:+ clips = sorted(caps)+ elif args.random:+ random.seed(args.seed)+ clips = random.sample(sorted(caps), min(args.random, len(caps)))+ elif args.clips_file:+ clips = [l.split('#')[0].strip() for l in open(args.clips_file)]+ clips = [c for c in clips if c]+ elif args.clips:+ clips = args.clips+ else:+ raise SystemExit('give --clips, --clips_file, --random N or --all')+ if args.all and (args.grid or args.sheet):+ raise SystemExit('--grid / --sheet keep every frame in memory; use --random N for those')++ out_dir = args.output_dir or os.path.join(root, 'outputs', 'caption_vis', args.dataset)+ os.makedirs(out_dir, exist_ok=True)++ if args.skip_existing or args.all:+ before = len(clips)+ clips = [c for c in clips if not os.path.isfile(os.path.join(out_dir, c + '.mp4'))]+ if before != len(clips):+ print('skipping %d clip(s) already rendered' % (before - len(clips)))++ if args.workers > 1:+ jobs = [(root, args.dataset, c, caps[c], out_dir, args.max_frames, args.elev,+ args.azim, not args.no_trail, args.figsize, args.dpi, args.fps,+ args.with_clip_name)+ for c in clips if c in caps]+ print('rendering %d clip(s) with %d workers -> %s' % (len(jobs), args.workers, out_dir))+ done = failed = 0+ with ProcessPoolExecutor(args.workers) as ex:+ for clip, n, err in ex.map(_render_one, jobs, chunksize=4):+ if err:+ failed += 1+ print(' FAILED %-38s %s' % (clip, err))+ else:+ done += 1+ if done % 100 == 0 or done == len(jobs):+ print(' %d/%d rendered' % (done, len(jobs)), flush=True)+ print('wrote %d MP4(s) to %s%s' % (done, out_dir,+ ', %d failed' % failed if failed else ''))+ return++ stacks, kept = [], []+ for clip in clips:+ if clip not in caps:+ print(' skip %s (no caption)' % clip)+ continue+ parents, pos = load_clip(root, args.dataset, clip, args.max_frames)+ caption = caps[clip]+ print(' %-38s %3d frames | %s' % (clip, len(pos), caption))+ frames = render_skeleton_motion_ground(+ parents, pos, elev=args.elev, azim=args.azim,+ # The export layer is Y-up; _maybe_rotate_y_up maps input Y onto+ # matplotlib's vertical Z axis, so Y-up data wants rotate_root=True.+ rotate_root=True,+ title=_title(clip, caption, args.with_clip_name),+ trail=not args.no_trail, figsize=tuple(args.figsize), dpi=args.dpi)+ final = os.path.join(out_dir, clip + '.mp4')+ tmp = os.path.join(out_dir, '.%s.tmp.mp4' % clip)+ imageio.mimwrite(tmp, frames, fps=args.fps)+ os.replace(tmp, final)+ stacks.append(frames)+ kept.append(clip)++ if not kept:+ raise SystemExit('nothing rendered')+ print('wrote %d MP4(s) to %s' % (len(kept), out_dir))++ if args.grid and len(kept) > 1:+ grid = tile(stacks)+ p = os.path.join(out_dir, 'grid_%s.mp4' % '_'.join(kept[:3])[:60])+ imageio.mimwrite(p, grid, fps=args.fps)+ print('wrote grid %s (%d frames, %dx%d)' % (p, len(grid), grid.shape[2], grid.shape[1]))++ if args.sheet:+ cols = args.sheet_cols+ rowsimg = []+ for clip, fr in zip(kept, stacks):+ idx = [round(i * (len(fr) - 1) / max(cols - 1, 1)) for i in range(cols)]+ rowsimg.append(np.concatenate([fr[i] for i in idx], axis=1))+ sheet = np.concatenate(rowsimg, axis=0)+ p = os.path.join(out_dir, 'sheet_%s.png' % '_'.join(kept[:3])[:60])+ imageio.imwrite(p, sheet)+ print('wrote sheet %s (%dx%d)' % (p, sheet.shape[1], sheet.shape[0]))+++if __name__ == '__main__':+ main()diff --git a/data_process/tools/vis_tpose_facing.py b/data_process/tools/vis_tpose_facing.pyindex 897f587..1b4036b 100644--- a/data_process/tools/vis_tpose_facing.py+++ b/data_process/tools/vis_tpose_facing.py@@ -100,10 +100,18 @@ def index_motion_npzs(motions_dir):across un-prefixed clips, so its ``mixamo`` key maps to the first NPZ."""index = {}- for p in sorted(Path(motions_dir).glob('*.npz')):+ npzs = sorted(Path(motions_dir).glob('*.npz'))+ for p in npzs:key = p.stem.split('-', 1)[0] if '-' in p.stem else p.stemindex.setdefault(key, p)- index.setdefault('mixamo', p)+ # Mixamo clips carry no object prefix, so the per-prefix keys above are+ # action names; the rig itself is addressed as 'mixamo'. Only add that+ # alias when un-prefixed clips exist, otherwise every prefixed export dir+ # (truebones, objaverse) would gain a spurious 'mixamo' rig pointing at+ # its first clip.+ unprefixed = [p for p in npzs if '-' not in p.stem]+ if unprefixed:+ index.setdefault('mixamo', unprefixed[0])return indexdiff --git a/data_process/utils/kinematics.py b/data_process/utils/kinematics.pynew file mode 100644index 0000000..951fa16--- /dev/null+++ b/data_process/utils/kinematics.py@@ -0,0 +1,75 @@+"""Forward kinematics for export-stage NPZs (numpy only, no bpy, no Motion).++Quaternions are ``w, x, y, z`` โ the order the exporter writes (Blender's+``Matrix.to_quaternion``). Export space is Blender rotated -90 deg about X+(a proper rotation, so handedness is preserved): **Y-up**, ground at the+minimum Y.++Batched over any number of leading axes, so the same call serves one rest+pose ``(J, .)`` and a whole clip ``(T, J, .)``.+"""++from typing import Sequence++import numpy as np+++def qmul(a, b):+ """Hamilton product of two ``w, x, y, z`` quaternion arrays."""+ w1, x1, y1, z1 = a[..., 0], a[..., 1], a[..., 2], a[..., 3]+ w2, x2, y2, z2 = b[..., 0], b[..., 1], b[..., 2], b[..., 3]+ return np.stack([w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2,+ w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,+ w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2,+ w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2], -1)+++def qrot(q, v):+ """Rotate vectors *v* ``(..., 3)`` by quaternions *q* ``(..., 4)``."""+ qv = np.concatenate([np.zeros(v.shape[:-1] + (1,)), v], -1)+ return qmul(qmul(q, qv), q * np.array([1, -1, -1, -1]))[..., 1:]+++def fk(local_pos, local_rot, parents: Sequence[int]) -> np.ndarray:+ """Global joint positions from local positions / rotations and a parent list.++ ``local_pos`` and ``local_rot`` are ``(..., J, 3)`` and ``(..., J, 4)``;+ the result is ``(..., J, 3)``. ``parents[i] < 0`` marks the root.+ """+ local_pos = np.asarray(local_pos, dtype=np.float64)+ local_rot = np.asarray(local_rot, dtype=np.float64)+ parents = np.asarray(parents)+ lead = local_pos.shape[:-2]+ g = np.zeros(lead + local_pos.shape[-2:])+ gr = np.zeros(lead + local_rot.shape[-2:])+ for i in range(len(parents)):+ p = int(parents[i])+ if p < 0:+ g[..., i, :] = local_pos[..., i, :]+ gr[..., i, :] = local_rot[..., i, :]+ else:+ g[..., i, :] = g[..., p, :] + qrot(gr[..., p, :], local_pos[..., i, :])+ gr[..., i, :] = qmul(gr[..., p, :], local_rot[..., i, :])+ return g+++def rest_global(d) -> np.ndarray:+ """Global rest-pose joint positions ``(J, 3)`` from a loaded export NPZ."""+ return fk(d['rest_local_pos'], d['rest_local_rot'], d['parents'])+++def clip_global(d, max_frames: int = 0) -> np.ndarray:+ """Global joint positions ``(T, J, 3)`` for a whole export clip.++ ``max_frames`` > 0 truncates, which is how callers match+ ``blender_render.MAX_RENDER_FRAMES`` โ the range the captioner actually saw.+ """+ pos, rot = d['anim_local_pos'], d['anim_local_rot']+ if max_frames and len(pos) > max_frames:+ pos, rot = pos[:max_frames], rot[:max_frames]+ return fk(pos, rot, d['parents'])+++def frame_global(d, t: int = 0) -> np.ndarray:+ """Global joint positions ``(J, 3)`` at a single frame of an export clip."""+ return fk(d['anim_local_pos'][t], d['anim_local_rot'][t], d['parents'])diff --git a/data_process/utils/motion_features.py b/data_process/utils/motion_features.pyindex e35e1c5..daf42ba 100644--- a/data_process/utils/motion_features.py+++ b/data_process/utils/motion_features.py@@ -193,9 +193,14 @@ def _names_permutation(ref_names, ref_parents, names, parents):def _load_npz_anim(object_npz_path, raw_parents, raw_njoints,- corps_idxs, bfs_order, fps, raw_names=None):+ corps_idxs, bfs_order, fps, raw_names=None, head_trim=0):"""Load a motion NPZ, apply joint subset/reorder, and downsample.+ ``head_trim`` drops the first N frames of the export NPZ itself (its own+ frame rate, before downsampling and the static trim), so a+ ``clip_trims.txt`` entry means the same frames to every reader of the+ export, not only to this stage.+Returns:Tuple of (motion_anim, motion_fps, None) on success, or(None, None, reason_string) when the motion should be skipped.@@ -211,6 +216,14 @@ def _load_npz_anim(object_npz_path, raw_parents, raw_njoints,anim_local_rot = anim_data['anim_local_rot']anim_local_pos = anim_data['anim_local_pos']rest_local_pos = anim_data['rest_local_pos']+ if head_trim:+ if head_trim >= nframes:+ reason = f'head trim of {head_trim} frames removes all {nframes}'+ logger.warning(f'Skipping clip ({reason}): {object_npz_path}')+ return None, None, reason+ anim_local_rot = anim_local_rot[head_trim:]+ anim_local_pos = anim_local_pos[head_trim:]+ nframes -= head_trim# corps_idxs/bfs_order are positional indices into the T-pose clip's# joint order; a source file enumerating the same skeleton in a@@ -293,7 +306,7 @@ def _load_tpos_npz(tpos_data, corps_idxs=None, corps_names=None):def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,corps_idxs, bfs_order, fps, scale_factor,ground_height, face_joint_idxs,- body_axis=False, raw_names=None):+ body_axis=False, raw_names=None, head_trim=0):"""Load a motion NPZ and canonicalize facing / XZ / ground + scale.``scale_factor`` and ``ground_height`` are reused from T-pose@@ -309,7 +322,7 @@ def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,"""motion_anim, motion_fps, skip_reason = _load_npz_anim(object_npz_path, raw_parents, raw_njoints, corps_idxs, bfs_order, fps,- raw_names=raw_names)+ raw_names=raw_names, head_trim=head_trim)if motion_anim is None:return None, skip_reason@@ -514,12 +527,16 @@ def _discontinuity_above_threshold(global_positions, step_threshold, ratio_thresbody lengths AND that step is at least *ratio_threshold* times the clip'smedian step โ the signature of concatenated actions, not of fast motion,which raises the median too. Non-finite input counts as discontinuous.+ ``step_threshold <= 0`` disables the filter (non-finite input is still+ rejected); without this guard ``step_max >= 0`` always holds and the+ ratio test alone would decide."""if not np.isfinite(global_positions).all():return True, {'step_med': 0.0, 'step_max': float('inf'),'jump_ratio': float('inf'), 'jump_frames': len(global_positions)}metrics = _discontinuity_metrics(global_positions)- is_bad = (metrics['step_max'] >= step_threshold+ is_bad = (step_threshold > 0+ and metrics['step_max'] >= step_thresholdand metrics['jump_ratio'] >= ratio_threshold)return is_bad, metrics@@ -636,13 +653,17 @@ def _process_motion_file(object_npz_path, ctx):"""base_name = str(Path(object_npz_path).stem)min_frames = ctx['min_frames']+ head_trim = ctx.get('head_trims', {}).get(base_name, 0)+ if head_trim:+ logger.info(f'[{base_name}] dropping the first {head_trim} export frames (clip_trims.txt)')# --- Load + motion-level canonicalize ----------------------------------result, skip_reason = load_and_canonicalize_motion(object_npz_path, ctx['raw_parents'], ctx['raw_njoints'],ctx['corps_idxs'], ctx['bfs_order'], ctx['fps'], ctx['scale_factor'],ctx['ground_height'], ctx['face_joint_idxs'],- body_axis=ctx['body_axis'], raw_names=ctx.get('raw_names'))+ body_axis=ctx['body_axis'], raw_names=ctx.get('raw_names'),+ head_trim=head_trim)if result is None:return {'n_saved': 0, 'frames': 0, 'clip_captions': {},'filtered': [{'name': base_name, 'reason': skip_reason}]}@@ -653,8 +674,9 @@ def _process_motion_file(object_npz_path, ctx):motion_anim, _, _ = trim_static_ends(motion_anim, static_threshold=ctx['static_threshold'])if len(motion_anim) < min_frames:- reason = (f'too few frames after static trim: {len(motion_anim)} '- f'< {min_frames} (orig {orig_nframes})')+ reason = (f'too few frames after static trim'+ + (f' (after a {head_trim}-frame head trim)' if head_trim else '')+ + f': {len(motion_anim)} < {min_frames} (orig {orig_nframes})')return {'n_saved': 0, 'frames': 0, 'clip_captions': {},'filtered': [{'name': base_name, 'reason': reason}]}@@ -676,6 +698,10 @@ def _process_motion_file(object_npz_path, ctx):is_low, clip_metrics = _activity_below_threshold(clip_global_positions, ctx['activity_threshold'])clip_name = f'{base_name}-{clip_idx:03d}'+ if is_low and base_name in ctx['activity_keep']:+ logger.info(f'[{clip_name}] low activity ({_format_activity(clip_metrics)}) '+ f'kept: listed in activity_keep.txt')+ is_low = Falseif is_low:filtered.append({'name': clip_name,@@ -726,7 +752,7 @@ def process_object(object_type, object_npzs, save_dir,jump_step_threshold=0.2, jump_ratio_threshold=8.0,min_joints=None, max_joints=None,use_tpos_ground_height=False, save_vis=True,- vis_ground=False):+ vis_ground=False, head_trims=None, activity_keep=None):"""Process all motion clips for one object type.Emits per-clip NPZs holding raw canonicalized arrays โ global@@ -743,6 +769,16 @@ def process_object(object_type, object_npzs, save_dir,default). Whichever is used is recorded in the cond as ``ground_height``/ ``ground_height_mode``.+ ``head_trims`` maps a motion stem to N: the first N frames of that+ motion's export NPZ are dropped on load, before downsampling and the+ static trim (hand-reviewed bind-pose / foreign first frames, from+ ``clip_trims.txt``).++ ``activity_keep`` is a set of motion stems exempt from the low-activity+ filter: hand-reviewed small but real motions (a head turn, a wave) that the+ joint-activity gate cannot tell from a held pose (``activity_keep.txt``).+ Every other filter still applies to them.+Object types with no motion NPZs, a joint count outside``[min_joints, max_joints]`` or a degenerate (zero-extent) skeleton areskipped with a reason instead of raising.@@ -848,6 +884,8 @@ def process_object(object_type, object_npzs, save_dir,'min_frames': min_frames,'jump_step_threshold': jump_step_threshold,'jump_ratio_threshold': jump_ratio_threshold,+ 'head_trims': head_trims or {},+ 'activity_keep': set(activity_keep or ()),# I/O'motions_save_dir': motions_save_dir,'animations_save_dir': animations_save_dir,diff --git a/data_process/utils/plotting.py b/data_process/utils/plotting.pyindex e4ebebe..9a59157 100644--- a/data_process/utils/plotting.py+++ b/data_process/utils/plotting.py@@ -1003,10 +1003,18 @@ def _wrap_title(text: str, fig_width_in: float, font_pt: float,"""Hard-wrap a caption to the figure width (suptitle's own wrapping isunreliable and long captions ran off both edges).+ Line breaks already in *text* are kept: each one is wrapped on its own, so a+ caller that writes "<clip>\n<caption>" gets two blocks rather than one run-on+ paragraph (``textwrap`` treats a whole string as one paragraph and would fold+ the newline into a space).+``fill`` is the share of the figure width a full line may occupy."""chars = max(24, int(fig_width_in * 72.0 * fill / _mean_advance_pt(font_pt, family)))- return '\n'.join(textwrap.wrap(text, chars)) or text+ out = []+ for para in text.split('\n'):+ out.extend(textwrap.wrap(para, chars) or [para])+ return '\n'.join(out) or textdef _draw_caption(fig, text: str, figsize, font_pt: float = CAPTION_FONT_SIZE,diff --git a/data_process/utils/vlm.py b/data_process/utils/vlm.pyindex 0ccb63f..7bf1905 100644--- a/data_process/utils/vlm.py+++ b/data_process/utils/vlm.py@@ -481,6 +481,30 @@ def merge_shards(output_json, num_gpus=None):save_json(merged, output_json)logger.info("Merged {} shards -> {} total entries in {}".format(len(shard_paths), len(merged), output_json))++ # Each shard also records the clips it could not caption in+ # ``<shard stem>_failed.txt``. Fold them into the canonical+ # ``<output stem>_failed.txt`` so the next (single-process or sharded)+ # run retries them; drop clips that another shard or a later pass+ # captioned in the meantime.+ base = os.path.splitext(output_json)[0]+ failed_out = base + "_failed.txt"+ failed = set()+ if os.path.exists(failed_out):+ with open(failed_out) as f:+ failed.update(line.strip() for line in f if line.strip())+ n_before = len(failed)+ for spath in shard_paths:+ shard_failed = os.path.splitext(spath)[0] + "_failed.txt"+ if os.path.exists(shard_failed):+ with open(shard_failed) as f:+ failed.update(line.strip() for line in f if line.strip())+ failed = {name for name in failed if not is_completed(merged.get(name))}+ if failed or n_before:+ with open(failed_out, "w") as f:+ f.write("\n".join(sorted(failed)) + ("\n" if failed else ""))+ logger.info("Merged failure lists -> {} pending clips in {}".format(+ len(failed), failed_out))return mergeddiff --git a/data_process/vlm_caption/backends.py b/data_process/vlm_caption/backends.pyindex a51e834..322d0de 100644--- a/data_process/vlm_caption/backends.py+++ b/data_process/vlm_caption/backends.py@@ -53,12 +53,14 @@ def detect_backend(model_name):"""Auto-detect backend from model name.- Contains 'qwen' (case-insensitive) -> qwen- - Starts with 'gemini' -> gemini+ - Contains 'gemini' (case-insensitive, so ``google/gemini-...`` and+ ``Gemini-3-...`` route correctly) -> gemini- Everything else -> openai"""- if "qwen" in model_name.lower():+ lowered = model_name.lower()+ if "qwen" in lowered:return BACKEND_QWEN- elif model_name.startswith("gemini"):+ elif "gemini" in lowered:return BACKEND_GEMINIelse:return BACKEND_OPENAIdiff --git a/data_process/vlm_caption/prompts.py b/data_process/vlm_caption/prompts.pyindex 32da6e3..38a19f2 100644--- a/data_process/vlm_caption/prompts.py+++ b/data_process/vlm_caption/prompts.py@@ -18,6 +18,12 @@ vocabulary, and style examples.# Shared caption-prompt scaffold# โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ+# The subject every caption opens with, in every dataset. The three+# datasets train as one mixture, so a per-dataset subject would hand the+# text encoder a free dataset label and defeat the topology-agnostic+# conditioning. Keep in sync with patch_annotations.py's SUBJECT.+CORPUS_SUBJECT = 'An object'+_INPUT_CAMERAS = ("# Input\n""Four synchronized cameras 90 degrees apart at fixed elevation, labeled "@@ -151,7 +157,7 @@ MIXAMO_PROMPT = _caption_prompt(articulation="limbs move, pelvis fixed",in_place_example="'runs in place', 'marches in place'",vocab_block="",- subject="A person",+ subject=CORPUS_SUBJECT,uncertain_fallback=("If heading is uncertain, say 'one arm' / 'one leg' instead of ""guessing left/right."@@ -164,12 +170,12 @@ MIXAMO_PROMPT = _caption_prompt(no_appearance_extra=", no clothing",still_clause="subtle sway/breathing still counts as motion",examples=(- "- A person walks forward.\n"- "- A person runs in place.\n"- "- A person punches with the right fist.\n"- "- A person raises one arm while seated.\n"- "- A person turns around as if carrying something.\n"- "- A person crouches and then rises back up.\n"+ "- An object walks forward.\n"+ "- An object runs in place.\n"+ "- An object punches with the right fist.\n"+ "- An object raises one arm while seated.\n"+ "- An object turns around as if carrying something.\n"+ "- An object crouches and then rises back up.\n"),)@@ -259,7 +265,7 @@ TRUEBONES_PROMPT = _caption_prompt("them. Do not copy the label ""verbatim: rewrite it in the required format and never carry a ""species word into the caption โ the example label above becomes "- "'An animal runs forward and then falls to the right.'\n\n"+ "'An object runs forward and then falls to the right.'\n\n"),step1=("1. ESTABLISH HEADING. Cues, in priority order: (a) "@@ -272,7 +278,7 @@ TRUEBONES_PROMPT = _caption_prompt(articulation="limbs/wings/tail/jaw move, torso fixed",in_place_example="'trots in place'",vocab_block="",- subject="An animal",+ subject=CORPUS_SUBJECT,uncertain_fallback=("If heading is uncertain, say 'one foreleg' / 'a wing' instead of ""guessing left/right."@@ -289,12 +295,12 @@ TRUEBONES_PROMPT = _caption_prompt(no_appearance_extra="/color/texture",still_clause="subtle sway, breathing, tail flicks, ear twitches still count as motion",examples=(- "- An animal gallops forward.\n"- "- An animal turns to the right.\n"- "- An animal rears upward on its hind legs.\n"- "- An animal flaps its wings and rises.\n"- "- An animal swims forward with its tail.\n"- "- An animal strikes forward with its tail curled.\n"+ "- An object gallops forward.\n"+ "- An object turns to the right.\n"+ "- An object rears upward on its hind legs.\n"+ "- An object flaps its wings and rises.\n"+ "- An object swims forward with its tail.\n"+ "- An object strikes forward with its tail curled.\n"),)diff --git a/scripts/run_animate_motion.sh b/scripts/run_animate_motion.shindex ec864dc..4cb7197 100755--- a/scripts/run_animate_motion.sh+++ b/scripts/run_animate_motion.sh@@ -12,8 +12,10 @@# Detected as the last arg when it does not end in .npy / .npz.## Env overrides:-# ANIM_MODE (default: fk), CHAR_PATH (default: auto),-# COND_PATH (default: dataset/features/<DATASET_TYPE>/cond.npy)+# ANIM_MODE (default: fk), CHAR_PATH (default: auto; mixamo falls back to+# dataset/raw/mixamo/character_refined/Y_Bot.fbx),+# COND_PATH (default: dataset/features/<DATASET_TYPE>/cond.npy),+# EXTRA_BONES_STRATEGY (merge|remove|keep, default: merge)## Output: $OUTPUT_DIR/<anim_stem>.{glb,fbx}@@ -128,6 +130,9 @@ process_one_motion() {if [ -n "${CHAR_PATH:-}" ]; thenexport CHAR_PATHfi+ if [ -n "${EXTRA_BONES_STRATEGY:-}" ]; then+ export EXTRA_BONES_STRATEGY+ fibash "$STAGE1_WRAPPER" "$DATASET_TYPE")diff --git a/unimate/configs/schema.py b/unimate/configs/schema.pyindex 9c2d967..423946b 100644--- a/unimate/configs/schema.py+++ b/unimate/configs/schema.py@@ -92,6 +92,12 @@ class DatasetConfig:# Samplingsampler_alpha: float = 0.5 # Power-law exponent: 0=uniform-per-sample, 0.5=sqrt-balanced, 1=uniform-per-type+ # Two-level sampling: dataset d gets total probability โ N_d^(1-sampler_dataset_alpha)+ # (N_d = its clips; 0 = its natural clip share, 1 = equal per dataset), then sampler_alpha+ # balances object types inside it. None = legacy single level, object types pooled across+ # datasets (lets one-rig Mixamo fall to <1% of a UniML3D mixture). Default None keeps+ # configs and runs written before 2026-09-27 bit-identical; the *_v2 configs set 0.25.+ sampler_dataset_alpha: Optional[float] = None# Train/eval split (clip-level, stratified per object_type).# test_split_ratio=0 disables splitting (all clips โ train).@@ -296,15 +302,24 @@ class TrainingConfig:@dataclassclass SamplingArgs:- """Sampling arguments."""+ """Sampling block: read by the periodic debug visualization during+ training (``num_samples``, ``cfg_scale``) and as fallbacks by+ ``unimate.inference.sample`` (``model_path``, ``cfg_scale``, ``device``).++ ``output_dir``, ``seed``, ``object_type`` and ``num_repetitions`` are no+ longer consumed by any entry point โ inference has its own ``--output_dir``+ / ``--seed`` / ``--num_repetitions`` flags and test-case files. They stay+ here only so every saved ``outputs/*/config.json`` (written with+ ``dataclasses.asdict``) keeps loading; do not add new users.+ """model_path: Optional[str] = None- output_dir: Optional[str] = None- seed: Optional[int] = 10+ output_dir: Optional[str] = None # deprecated, unused+ seed: Optional[int] = 10 # deprecated, unused (inference: --seed)# Sampling hyperparameters- object_type: Optional[List[str]] = None- num_samples: int = 4- num_repetitions: int = 3+ object_type: Optional[List[str]] = None # deprecated, unused+ num_samples: int = 4 # debug visualization: object types per dataset per split+ num_repetitions: int = 3 # deprecated, unused (inference: --num_repetitions)device: str = "cuda"# CFGdiff --git a/unimate/dataset/factory.py b/unimate/dataset/factory.pyindex 69ddfec..bb9359b 100644--- a/unimate/dataset/factory.py+++ b/unimate/dataset/factory.py@@ -91,7 +91,8 @@ def create_train_dataloader("""Build the training DataLoader over the train split of the Mixture dataset."""dataset = create_dataset(dataset_config, model_config)- sampler = MixtureSampler(dataset, alpha=dataset_config.sampler_alpha) if balanced else None+ sampler = MixtureSampler(dataset, alpha=dataset_config.sampler_alpha,+ dataset_alpha=dataset_config.sampler_dataset_alpha) if balanced else Nonereturn DataLoader(dataset,diff --git a/unimate/dataset/mixture/dataset.py b/unimate/dataset/mixture/dataset.pyindex b3a55a7..346e3a2 100644--- a/unimate/dataset/mixture/dataset.py+++ b/unimate/dataset/mixture/dataset.py@@ -1363,50 +1363,80 @@ class MotionDataset(data.Dataset):# ---------------------------------------------------------------------------class MixtureSampler(data.Sampler):- """Power-law balanced sampling across skeleton types.+ """Power-law balanced sampling across datasets and skeleton types.- Per object type k with n_k motions, total sampling probability โ n_k^(1-alpha).- Per sample within type k, weight โ n_k^(-alpha).+ Two levels. Dataset d with N_d motions gets total probability+ โ N_d^(1-dataset_alpha); inside it, object type k with n_k motions gets a+ share โ n_k^(1-alpha), split evenly over its motions.- alpha=0.0 : uniform per sample (data-rich types dominate)+ alpha=0.0 : uniform per sample within the dataset (data-rich types dominate)alpha=0.5 : square-root balanced (default โ good compromise)alpha=1.0 : uniform per type (ignores motion count entirely)+ dataset_alpha=0.0 : each dataset keeps its natural share of the clips+ dataset_alpha=1.0 : equal probability per dataset+ dataset_alpha=None: single level (default, the behaviour before 2026-09-27) โ+ object types pooled across all datasets. A dataset of one rig (Mixamo)+ then counts as one type and falls below 1% of a UniML3D mixture; the+ *_v2 configs set 0.25.++ With one dataset both levels coincide with the legacy behaviour.This is a plain Sampler (not DistributedSampler) that yields all indices.Accelerate's BatchSamplerShard handles per-rank sharding automatically."""- def __init__(self, data_source, alpha: float = 0.5):+ def __init__(self, data_source, alpha: float = 0.5, dataset_alpha=None):super().__init__(data_source)name_list = data_source.motion_dataset.train_name_listmotion_dict = data_source.motion_dataset.train_motion_dicttotal_samples = len(name_list)+ # Keyed by (dataset_type, object_type): nothing stops two datasets from+ # using the same object-type name.indices_by_object = defaultdict(list)for i, name in enumerate(name_list):- indices_by_object[motion_dict[name]['object_type']].append(i)+ m = motion_dict[name]+ indices_by_object[(m['dataset_type'], m['object_type'])].append(i)+ objects_by_dataset = defaultdict(list)+ for key in indices_by_object:+ objects_by_dataset[key[0]].append(key)- # w_i = n_k^(-alpha) for sample i in type k, giving type k total- # probability โ n_k * n_k^(-alpha) = n_k^(1-alpha).weights = np.zeros(total_samples)- for indices in indices_by_object.values():- n_k = len(indices)- weights[indices] = n_k ** (-alpha)+ if dataset_alpha is None:+ # w_i = n_k^(-alpha) for sample i in type k, giving type k total+ # probability โ n_k * n_k^(-alpha) = n_k^(1-alpha).+ for indices in indices_by_object.values():+ weights[indices] = len(indices) ** (-alpha)+ else:+ for d_type, keys in objects_by_dataset.items():+ n_d = sum(len(indices_by_object[k]) for k in keys)+ d_share = n_d ** (1 - dataset_alpha)+ type_mass = {k: len(indices_by_object[k]) ** (1 - alpha) for k in keys}+ z = sum(type_mass.values())+ for k in keys:+ indices = indices_by_object[k]+ weights[indices] = d_share * type_mass[k] / z / len(indices)weights /= weights.sum()+ dataset_probs = {d: sum(weights[indices_by_object[k]].sum() for k in keys)+ for d, keys in objects_by_dataset.items()}+ logger.info(f'MixtureSampler (alpha={alpha}, dataset_alpha={dataset_alpha}): '+ + ', '.join(f'{d} n={sum(len(indices_by_object[k]) for k in keys)} '+ f'p={dataset_probs[d]:.3f}'+ for d, keys in objects_by_dataset.items()))type_probs = {object_type: weights[indices].sum()for object_type, indices in indices_by_object.items()}sorted_types = sorted(type_probs.items(), key=lambda x: x[1], reverse=True)- logger.info(f'MixtureSampler (alpha={alpha}): {len(sorted_types)} object types')+ logger.info(f'MixtureSampler: {len(sorted_types)} object types')for name, prob in sorted_types[:5]:- logger.info(f' top: {name} โ n={len(indices_by_object[name])}, p={prob:.4f}')+ logger.info(f' top: {name[1]} ({name[0]}) โ n={len(indices_by_object[name])}, p={prob:.4f}')if len(sorted_types) > 10:logger.info(' ...')for name, prob in sorted_types[-5:]:- logger.info(f' bot: {name} โ n={len(indices_by_object[name])}, p={prob:.4f}')+ logger.info(f' bot: {name[1]} ({name[0]}) โ n={len(indices_by_object[name])}, p={prob:.4f}')self.weights = torch.as_tensor(weights, dtype=torch.double)self.total_samples = total_samplesdiff --git a/unimate/inference/sample.py b/unimate/inference/sample.pyindex 4b76965..4345c73 100644--- a/unimate/inference/sample.py+++ b/unimate/inference/sample.py@@ -77,7 +77,9 @@ class InferenceArgs:# train or eval motion_dict) or the dataset's eval split.inbetween: bool = False# Comma-separated signed temporal indices to hold clean. Negatives count- # from the per-clip valid length. Default keeps first + last frame.+ # from the end of the generation window (max_motion_length), so -1 is the+ # last slot even when the GT clip is shorter (see build_keep_mask).+ # Default keeps first + last slot.keep_frames: str = "0,-1"# Text-guided motion editing: clamp the listed joints to GT for all# frames and let the model denoise the rest under a new caption (e.g.,