Skip to content

[SIGGRAPH Asia 2026] UniMate: One Unified Model to Animate Diverse Skeletons

Python

Announce the preview checkpoints; add two-level dataset sampling

Friedrich-Madbf9928c9ff
diff --git a/README.md b/README.md
index 662561d..f65d870 100644
--- a/README.md
+++ b/README.md
@@ -30,12 +30,13 @@
## ๐Ÿ”ฅ News
+- **[2026-09-27]** **Preview checkpoints** are released on [Hugging Face](https://huggingface.co/Linzhan/UniMate). ๐Ÿš€
- **[2026-09-06]** The **training and inference code** is released. ๐Ÿš€
- **[2026-08-30]** The raw **UniML3D dataset** and its [data-processing pipeline](data_process/) are released. ๐Ÿš€
- **[2026-08-01]** Our [Interactive Demo](https://linzhanmou.com/unimate/interactive.html) is live โ€” browse our animation results in 3D. ๐ŸŽฎ
- **[2026-07-18]** UniMate is accepted to SIGGRAPH Asia 2026! ๐ŸŽ‰
-> **[TODO]** Pretrained checkpoints, the exact training configuration and data manifest used to produce them, evaluation scripts, and demo prompts will be released soon.
+> **[Update]** Preview checkpoints are released at [Linzhan/UniMate](https://huggingface.co/Linzhan/UniMate); new checkpoints will be synced there as they are released.
## ๐Ÿ› ๏ธ Environment Setup
@@ -87,7 +88,7 @@ Runs are configured by the JSON files in [`configs/`](configs/).
The two axes are independent and all four combinations are implemented, so `full` x `adaln` and `graph` x `cross_attn` also run if you set them in a config; the two shipped pairings are the ones the paper compares.
-Not shared across configs: `training.batch_size`, `training.num_steps` and `dataset.max_joints` are tuned per data combination (GPU memory tracks batch x frames x joints). `dataset.max_joints` is `100` for `truebones_*` / `mixamo_*` and `60` for `objaverse_*` / `uniml3d_*`; `dataset.min_joints` is `5` everywhere. Together they bound the skeleton sizes a run admits (object types outside the range are dropped) and, through what survives, the joint-axis padding width. Every other setting is identical.
+Not shared across configs: `training.batch_size`, `training.num_steps`, `model.num_layers` and `dataset.max_joints` are tuned per data combination (GPU memory tracks batch x frames x joints; depth grows with the data: 6 layers for a single Truebones / Mixamo source, 8 for Objaverse, 10 for the full mixture). `dataset.max_joints` is `100` for `truebones_*` / `mixamo_*` and `60` for `objaverse_*` / `uniml3d_*`; `dataset.min_joints` is `5` everywhere. Together they bound the skeleton sizes a run admits (object types outside the range are dropped) and, through what survives, the joint-axis padding width. Mixamo is a single skeleton, so its configs also turn off the object-type balancing (a no-op with one type) and, by choice, the topology augmentations. Every other setting is identical.
</details>
@@ -122,7 +123,7 @@ Each run writes to `outputs/<experiment name>/`:
Clips are drawn by a power-law-balanced sampler when `training.balanced` is set โ€” a type with `n` clips is sampled in proportion to `n^(1-sampler_alpha)`, so at the default `sampler_alpha = 0.5` a species with 100 clips is seen ten times as often as one with a single clip rather than a hundred times โ€” then augmented on the fly โ€” joint addition, leaf removal, chain pooling and per-bone length perturbation (`dataset.use_*_aug`) โ€” so the model sees more topologies than the data literally contains. Every clip is padded to `max_joints` on the joint axis and `max_motion_length` on the time axis, with masks carried alongside; nothing padded ever contributes to attention or to the loss.
-Training is **flow matching** (`training.diff_model = "flow"`): the network predicts the velocity of a linear interpolant between noise and data, under a masked L2 loss plus an optional geodesic rotation term (`training.lambda_geo`). Conditioning is dropped with probability `model.cond_mask_prob` so the same weights serve the conditional and unconditional branches that classifier-free guidance interpolates at sampling time. AdamW with a cosine schedule and warmup, gradient clipping at `training.max_grad_norm`, and an EMA copy of the weights (`training.use_ema`) โ€” the copy inference loads by default.
+Training is **flow matching** (`training.diff_model = "flow"`): the network predicts the velocity of a linear interpolant between noise and data, under a masked L2 loss plus two auxiliary terms computed on the reconstructed clean motion โ€” a geodesic rotation loss (`training.lambda_geo`) and a velocity-smoothness loss (`training.lambda_smooth`). Conditioning is dropped with probability `model.cond_mask_prob` so the same weights serve the conditional and unconditional branches that classifier-free guidance interpolates at sampling time. AdamW with a cosine schedule and warmup, gradient clipping at `training.max_grad_norm`, and an EMA copy of the weights (`training.use_ema`) โ€” the copy inference loads by default.
</details>
@@ -156,7 +157,7 @@ Given a rigged 3D asset and a text prompt, UniMate generates articulated motion
<img src="assets/qualitative.png" alt="Qualitative results" width="100%">
</div>
-Sampling starts from the output directory of a training run (`config.json`, `dataset_stats.npy`, `checkpoints/`) โ€” pretrained checkpoints in the same layout are coming soon. The target skeleton โ€” T-pose and topology conditioning โ€” is taken from the dataset, so the `dataset/features/<dataset>/` directory the model was trained on must be present.
+Sampling starts from the output directory of a training run (`config.json`, `dataset_stats.npy`, `checkpoints/`) โ€” the [released checkpoints](https://huggingface.co/Linzhan/UniMate) use the same layout. The target skeleton โ€” T-pose and topology conditioning โ€” is taken from the dataset, so the `dataset/features/<dataset>/` directory the model was trained on must be present.
```bash
python -m unimate.inference.sample \
diff --git a/configs/mixamo_60frames_full_cross_attn.json b/configs/mixamo_60frames_full_cross_attn.json
index 2c0bd91..078119e 100644
--- a/configs/mixamo_60frames_full_cross_attn.json
+++ b/configs/mixamo_60frames_full_cross_attn.json
@@ -28,6 +28,7 @@
"use_dataset_stats": true,
"balanced_stats": false,
"sampler_alpha": 0.5,
+ "sampler_dataset_alpha": 0.0,
"use_addition_aug": false,
"use_removal_aug": false,
"use_pooling_aug": false,
@@ -70,6 +71,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/mixamo_60frames_graph_adaln.json b/configs/mixamo_60frames_graph_adaln.json
index 00cf6e9..ea8e42e 100644
--- a/configs/mixamo_60frames_graph_adaln.json
+++ b/configs/mixamo_60frames_graph_adaln.json
@@ -28,6 +28,7 @@
"use_dataset_stats": true,
"balanced_stats": false,
"sampler_alpha": 0.5,
+ "sampler_dataset_alpha": 0.0,
"use_addition_aug": false,
"use_removal_aug": false,
"use_pooling_aug": false,
@@ -70,6 +71,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/objaverse_60frames_full_cross_attn.json b/configs/objaverse_60frames_full_cross_attn.json
index 896391e..6e37361 100644
--- a/configs/objaverse_60frames_full_cross_attn.json
+++ b/configs/objaverse_60frames_full_cross_attn.json
@@ -28,6 +28,7 @@
"use_dataset_stats": true,
"balanced_stats": true,
"sampler_alpha": 0.5,
+ "sampler_dataset_alpha": 0.0,
"use_addition_aug": true,
"use_removal_aug": true,
"use_pooling_aug": true,
@@ -70,6 +71,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/objaverse_60frames_graph_adaln.json b/configs/objaverse_60frames_graph_adaln.json
index 3d11251..a6dfad6 100644
--- a/configs/objaverse_60frames_graph_adaln.json
+++ b/configs/objaverse_60frames_graph_adaln.json
@@ -28,6 +28,7 @@
"use_dataset_stats": true,
"balanced_stats": true,
"sampler_alpha": 0.5,
+ "sampler_dataset_alpha": 0.0,
"use_addition_aug": true,
"use_removal_aug": true,
"use_pooling_aug": true,
@@ -70,6 +71,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/truebones_60frames_full_cross_attn.json b/configs/truebones_60frames_full_cross_attn.json
index 062d82e..51cc07f 100644
--- a/configs/truebones_60frames_full_cross_attn.json
+++ b/configs/truebones_60frames_full_cross_attn.json
@@ -28,6 +28,7 @@
"use_dataset_stats": true,
"balanced_stats": true,
"sampler_alpha": 0.5,
+ "sampler_dataset_alpha": 0.0,
"use_addition_aug": true,
"use_removal_aug": true,
"use_pooling_aug": true,
@@ -70,6 +71,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/truebones_60frames_graph_adaln.json b/configs/truebones_60frames_graph_adaln.json
index 9533615..c6726ed 100644
--- a/configs/truebones_60frames_graph_adaln.json
+++ b/configs/truebones_60frames_graph_adaln.json
@@ -28,6 +28,7 @@
"use_dataset_stats": true,
"balanced_stats": true,
"sampler_alpha": 0.5,
+ "sampler_dataset_alpha": 0.0,
"use_addition_aug": true,
"use_removal_aug": true,
"use_pooling_aug": true,
@@ -70,6 +71,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/uniml3d_60frames_full_cross_attn.json b/configs/uniml3d_60frames_full_cross_attn.json
index f2feae0..403a626 100644
--- a/configs/uniml3d_60frames_full_cross_attn.json
+++ b/configs/uniml3d_60frames_full_cross_attn.json
@@ -72,6 +72,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/configs/uniml3d_60frames_graph_adaln.json b/configs/uniml3d_60frames_graph_adaln.json
index 1c14a38..9dfdbd4 100644
--- a/configs/uniml3d_60frames_graph_adaln.json
+++ b/configs/uniml3d_60frames_graph_adaln.json
@@ -72,6 +72,7 @@
"log_interval": 50,
"save_interval": 10000,
"lambda_geo": 0.5,
+ "lambda_smooth": 0.1,
"use_ema": true,
"ema_decay": 0.9999,
"max_grad_norm": 1.0,
diff --git a/data_process/README.md b/data_process/README.md
index 516ce88..c0492c6 100644
--- a/data_process/README.md
+++ b/data_process/README.md
@@ -236,7 +236,9 @@ Each export directory receives:
| `joint_count.json`, `clip_frames.json`, `summary.json` | joints per skeleton, frames per clip, dataset-level summary |
| `.completed/{asset}.json` | per-asset completion markers; delete one to force a re-export |
-`--multi-worker N` shards a *directory* input over N Blender processes (objaverse, mixamo, auto mode). Each worker writes its own `*_worker{i}.json` summary shards, merged afterwards by `tools/merge_summaries.py`. Truebones is exported by a single-process exporter and ignores `--multi-worker`.
+`--multi-worker N` shards a *directory* input over N Blender processes (objaverse, mixamo, auto mode). Each worker writes its own `*_worker{i}.json` summary shards, merged afterwards by `tools/merge_summaries.py` โ€” only when every worker exited cleanly; a worker that had failed assets exits non-zero and the merge is left for the next run. Truebones is exported by a single-process exporter and ignores `--multi-worker`.
+
+An optional `excluded.csv` (column `object_id` or `file`) in the input directory or its parent lists asset stems the objaverse exporter skips entirely (`run_export_general.sh` does not read it).
Skeleton pruning is joint across all of an asset's clips, so an asset only completes atomically โ€” the marker is written once every clip of that asset is on disk. Markers also carry the pruned joint names, which is what makes `joint_names.json` rebuildable from disk after an interrupted run.
@@ -261,7 +263,7 @@ Each clip becomes `<clip>/v00{0..3}/*.png` (the per-frame PNGs the caption stage
- **Frame cap.** Rendering stops at the first 200 frames per clip (`MAX_RENDER_FRAMES`, ~6.7 s at 30 fps) โ€” enough context for captioning without paying for full-length renders. Clips shorter than `MIN_ACTION_FRAMES` (default 5) are skipped.
- **`--fps` must match stage 1.** glTF stores keyframe times in seconds, so the importer resamples them at the scene frame rate; a mismatch makes the renders cover a different time window than the NPZs. Default 30 in both stages.
-- **Multi-GPU.** `--multi-worker N` shards assets over N processes. `CUDA_VISIBLE_DEVICES` pinning does **not** bind EEVEE's EGL context, so on a shared multi-GPU machine every worker still renders on the first GPU โ€” give each render job a cgroup / scheduler allocation that exposes exactly one GPU instead.
+- **Multi-GPU.** `--multi-worker N` shards assets over N processes, and `NUM_GPUS=k` assigns worker `i` to `CUDA_VISIBLE_DEVICES=i % k`. That pinning does **not** bind EEVEE's EGL context, so on a shared multi-GPU machine every worker still renders on the first GPU โ€” give each render job a cgroup / scheduler allocation that exposes exactly one GPU instead. `--missing-only` exists for objaverse only; the wrapper rejects it for the other datasets, which resume per asset anyway.
- Quality knobs (`RESOLUTION`, `SAMPLES`, `CAMERA_DIST`) are environment overrides on both wrappers; run either with `-h`.
</details>
@@ -318,7 +320,7 @@ python data_process/vlm_caption/caption_rewrite_llm.py \
### Stage 3 โ€” Joint annotation: name cleanup & facing direction
-Two tasks, each with a rule-based and an LLM variant: joint-name cleaning (`names_*`) maps raw bone names onto a canonical anatomical vocabulary, and facing-direction selection (`face_*`) picks the bilateral joint pair โ€” or head/tail body axis, for serpentine rigs โ€” that defines each rig's heading. The LLM variants default to `deepseek-v4-flash` (`MODEL=` switches to any `gpt-*` model or a local HF model id) and fall back to the rules on failure.
+Two tasks, each with a rule-based and an LLM variant: joint-name cleaning (`names_*`) maps raw bone names onto a canonical anatomical vocabulary, and facing-direction selection (`face_*`) picks the bilateral joint pair โ€” or head/tail body axis, for serpentine rigs โ€” that defines each rig's heading. The LLM variants default to `deepseek-v4-flash` (`MODEL=` switches to any `gpt-*` model or a local HF model id). The two required passes fall back to the rules when a rig fails or exceeds its per-rig budget (`RIG_TIMEOUT`); the two refinement passes below instead keep the entry they were asked to correct.
**Required** โ€” these two produce the artifacts stage 4 consumes:
@@ -382,10 +384,12 @@ Applies the corrections found in a manual audit of the stage-2/3 outputs, so the
| Output | Scope |
|--------|-------|
| `export/<ds>/filtered_clips.txt` | individual clips, any dataset |
-| `export/objaverse/filtered_objects.txt` | whole rigs โ€” only the `tpose_wrong` flag (rest pose lies flat / rotated / upside-down) |
+| `export/<ds>/activity_keep.txt` | `<clip>`: exempt from stage 4's low-activity filter โ€” a real motion confined to a few joints (a head turn, a clap, a wave) that the joint-activity gate cannot tell from a held pose; every other filter still applies |
+| `export/<ds>/clip_trims.txt` | `<clip> <N>`: the clip's export NPZ starts N frames late โ€” stage 4 drops `motions/<clip>.npz` frames `0..N-1` on load โ€” for a bind-pose or foreign opening frame that snaps into the motion, so the clip is kept instead of filtered |
+| `export/objaverse/filtered_objects.txt` | whole rigs โ€” only the `tpose_wrong` flag (rest pose lies flat / rotated / upside-down) and `not_in_legacy_raw` (raw GLB absent from the legacy raw set, held out of training) |
| `export/objaverse/rig_flags.json` | informational: `empty_pair`, `bone_pair`, `body_axis_unnamed`, `facing_wrong`, `object_no_front` |
-Everything except `tpose_wrong` is informational โ€” those rigs are kept and trained with the facing they have. Per-rig evidence (rest-pose metrics, mesh T-pose grids and corrected-skeleton panels, all checked by eye) lives under `export/objaverse/tpose_abnormal_vis/`.
+Every other flag is informational โ€” those rigs are kept and trained with the facing they have. Per-rig evidence (rest-pose metrics, mesh T-pose grids and corrected-skeleton panels, all checked by eye) lives under `export/objaverse/tpose_abnormal_vis/`.
Truebones uses `filtered_clips.txt` for the three clips whose rig disagrees with their object's reference skeleton; objaverse uses it for the motion-discontinuity review.
@@ -403,7 +407,7 @@ APPLY_CLIP=1 bash data_process/scripts/run_extract_features.sh truebones
NUM_WORKERS=8 NO_VIS=1 bash data_process/scripts/run_extract_features.sh objaverse
```
-Every object type is canonicalized against its T-pose (facing โ†’ XZ-centering โ†’ diameter scaling โ†’ grounding), static pre/post-roll is trimmed, low-activity and motion-discontinuous clips are filtered, and the per-object topology condition is written to `cond.npy`. Output goes to `dataset/features/<dataset>/`: `motions/{object}-{motion}-{clip_idx}.npz`, `videos/`, `tpose/`, plus `cond.npy`, `captions.json`, `category_groups.json`, `filtered_clips.json`, and a `metadata.txt` report.
+Every object type is canonicalized against its T-pose (facing โ†’ XZ-centering โ†’ diameter scaling โ†’ grounding), static pre/post-roll is trimmed, low-activity and motion-discontinuous clips are filtered, and the per-object topology condition is written to `cond.npy`. Output goes to `dataset/features/<dataset>/`: `motions/{object}-{motion}-{clip_idx}.npz`, `videos/`, `tpose/`, plus `cond.npy` and a `metadata.txt` report, and โ€” whenever they have content โ€” `captions.json`, `category_groups.json` and `filtered_clips.json` (runtime-filtered clips and skipped object types; clips pre-filtered through `filtered_clips.txt` are dropped before processing and not listed).
<details>
<summary><b>Details โ€”</b> filters and thresholds</summary>
@@ -423,7 +427,7 @@ Every object type is canonicalized against its T-pose (facing โ†’ XZ-centering
Clips dropped at runtime and skipped object types are recorded in `filtered_clips.json`; `metadata.txt` reports the dataset totals.
-Two hand-reviewed skip lists are read from the export directory before any processing, both **optional**: `filtered_clips.txt` (individual clips, every dataset) and `filtered_objects.txt` (whole rigs, objaverse only), both written by `tools/patch_annotations.py`. The matching flags take a path, `"auto"` (the default) or an empty string to disable. `--category_groups` follows the same convention for `category_groups.json`, which this stage only copies through.
+Two hand-reviewed skip lists are read from the export directory before any processing, both **optional**: `filtered_clips.txt` (individual clips, every dataset) and `filtered_objects.txt` (whole rigs, objaverse only), both written by `tools/patch_annotations.py`, plus `clip_trims.txt` (`<clip> <N>`: drop the first N frames of a clip's export NPZ; `--clip_trims`) and `activity_keep.txt` (clips exempt from the low-activity filter; `--activity_keep`). The matching flags take a path, `"auto"` (the default) or an empty string to disable. `--category_groups` follows the same convention for `category_groups.json`, which this stage only copies through.
</details>
@@ -443,14 +447,16 @@ Used at inference time to render generated motions onto their rigged meshes:
```bash
ANIM_PATH=clip.npz bash data_process/scripts/run_animate_motion.sh objaverse # stage-4 / generated clip + cond.npy
-ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ†’ rig
+CHAR_PATH=char.glb ANIM_PATH=clip.npz bash data_process/scripts/run_animate_npz.sh # stage-1 export NPZ โ†’ rig
bash data_process/scripts/run_animate_fbx.sh # raw animation FBXs โ†’ any character
bash data_process/scripts/run_animate_mixamo.sh # batch over Mixamo export NPZs (Y Bot)
+CHAR_PATH=asset.glb OUTPUT_DIR=outputs/asset bash data_process/scripts/run_preprocess_char.sh # one asset โ†’ cond.npy + canonical rig
+CHAR_PATH=outputs/asset/asset_canonical.glb ANIM_PATH=motion.npz bash data_process/scripts/run_animate_lbs.sh # NumPy FK + LBS, no cond needed
```
-`run_animate_motion.sh` accepts both the stage-4 feature NPZ and the `.npy` motion features written by the model's sampler. The character is auto-resolved for truebones / objaverse from the raw asset directories; mixamo needs `CHAR_PATH` (default `character_refined/Y_Bot.fbx`, the rig the animation FBXs are authored on).
+`run_animate_motion.sh` accepts both the stage-4 feature NPZ and the `.npy` motion features written by the model's sampler. The character is auto-resolved for truebones / objaverse from the raw asset directories; mixamo defaults to `CHAR_PATH=dataset/raw/mixamo/character_refined/Y_Bot.fbx` (the rig the animation FBXs are authored on) when that file has been downloaded, and requires `CHAR_PATH` otherwise. The last two wrappers are described under [Custom Assets](#custom-assets).
-Armature bones absent from the motion are handled by `--extra_bones_strategy`: `merge` (default) transfers their vertex weights to the nearest kept ancestor and removes them, `remove` deletes the bones and their vertices, `keep` leaves them un-keyed.
+Armature bones absent from the motion are handled by `--extra_bones_strategy` on `animate_motion.py` / `animate_npz.py` (`EXTRA_BONES_STRATEGY=` on their wrappers and on `scripts/run_animate_motion.sh`): `merge` (default) transfers their vertex weights to the nearest kept ancestor and removes them, `remove` deletes the bones and their vertices, `keep` leaves them un-keyed. The FBX / raw-clip paths (`animate_fbx`, `animate_mixamo`, `animate_lbs`) leave undriven bones at rest.
`run_animate_fbx.sh` needs no export stage at all: it bakes a raw animation clip โ€” or a whole directory of them โ€” onto any character that shares the clips' bone names (the Mixamo convention):
@@ -500,7 +506,7 @@ CHAR_PATH=outputs/asset/<name>_canonical.glb ANIM_PATH=<motion.npz-or-directory>
bash data_process/scripts/run_animate_lbs.sh # one animated GLB per action
```
-`animate_lbs` computes FK and Linear Blend Skinning manually in NumPy (Blender only parses the asset; the math is numerically identical to Blender's armature modifier); `SAVE=npz,obj` additionally dumps raw deformed vertex sequences / per-frame OBJs.
+`animate_lbs` computes FK and Linear Blend Skinning manually in NumPy (Blender only parses the asset; the math is numerically identical to Blender's armature modifier). `SAVE` is the full list of outputs: `SAVE=glb,npz,obj` writes the rigged GLB plus raw deformed vertex sequences and per-frame OBJs (`SAVE=npz,obj` alone drops the GLB). A feature NPZ that is not from a canonical asset can still be driven by passing `DATASET_TYPE=` and/or `COND_PATH=` (a single-entry `cond.npy` is accepted whatever the clip is named).
## Data Formats
@@ -541,19 +547,19 @@ The motion representation stores per-frame deltas without dividing by the frame
| Key | Shape | Description |
|-----|-------|-------------|
| `object_type` | str | object-type name (clip-name prefix) |
-| `parents` | `(J,)` | parent indices in canonical BFS joint order |
-| `offsets`, `tpos_offsets` | `(J, 3)` | bind-pose and canonical T-pose bone offsets |
-| `joint_names`, `clean_joint_names` | `(J,)` | raw and cleaned bone names (stage 3) |
-| `tpos_first_frame` | `(J, 3)` | canonical T-pose global joint positions |
-| `tpos_local_rotations`, `tpos_global_rotations` | `(J, 4)` | T-pose local / global rotations (quaternions) |
-| `joint_relations`, `joint_graph_dists` | `(J, J)` | pairwise edge-relation types and clamped topology distances |
-| `joint_depths` | `(J,)` | depth in the kinematic tree |
-| `edge_indexs` | `(2, 2(Jโˆ’1))` | undirected edge list |
-| `spectral_feats` | `(J, K)` | Laplacian eigenvectors (`K = --max_freqs`) |
+| `parents` | list, len `J` | parent indices in canonical BFS joint order (`-1` = root) |
+| `offsets`, `tpos_offsets` | `(J, 3)` float64 | local bone offsets of the canonicalized T-pose (scaled, root moved by centering / grounding) and offsets recomputed from `tpos_first_frame` |
+| `joint_names`, `clean_joint_names` | list, len `J` | raw and cleaned bone names (stage 3) |
+| `tpos_first_frame` | `(J, 3)` float64 | canonical T-pose global joint positions |
+| `tpos_local_rotations`, `tpos_global_rotations` | `(J, 4)` float64 | T-pose local / global rotations (quaternions, `w, x, y, z`) |
+| `joint_relations`, `joint_graph_dists` | `(J, J)` int16 | pairwise edge-relation types and topology distances clamped at `--max_path_len` |
+| `joint_depths` | `(J,)` int64 | depth in the kinematic tree |
+| `edge_indexs` | `(2, 2(Jโˆ’1))` int64 | undirected edge list |
+| `spectral_feats` | `(J, K)` float32 | Laplacian eigenvectors (`K = --max_freqs`) |
| `kinematic_chains` | list | root-to-leaf joint chains |
| `scale_factor`, `ground_height`, `ground_height_mode` | scalar | calibration applied to every clip of the type |
-| `face_joint_idxs` | dict | `{r_hip, l_hip, body_axis}` indices of the facing pair (when annotated) |
-| `captions` | dict | `{clip_stem: caption}` for the type's clips (also flattened into `captions.json`) |
+| `face_joint_idxs` | dict | `{r_hip, l_hip, body_axis}` indices of the facing pair; `-1, -1` when the rig has no resolved pair (identity facing) |
+| `captions` | dict | `{clip_stem: caption}` for the type's saved clips (also flattened into `captions.json`) |
</details>
@@ -576,13 +582,15 @@ The export directory is where every stage before feature extraction accumulates
| `face_joint_names.json` | 3 | facing pair (or body axis) per rig |
| `failed_clean_names.txt`, `failed_face_joints.txt` | 3 | rigs that fell back to the rules |
| `filtered_clips.txt` | QA patch | clips stage 4 skips |
+| `clip_trims.txt` | QA patch | clips whose first N frames stage 4 drops |
+| `activity_keep.txt` | QA patch | clips stage 4's low-activity filter keeps |
| `filtered_objects.txt`, `rig_flags.json` | QA patch | objaverse only: rigs stage 4 skips, and the full flag list |
</details>
## Tools
-Standalone utilities in `data_process/tools/`. The QA visualizers have wrappers (listed under [Stage 3](#stage-3--joint-annotation-name-cleanup--facing-direction)); the rest are run directly.
+Standalone utilities in `data_process/tools/`. `vis_tpose.py` and `vis_tpose_facing.py` have wrappers (listed under [Stage 3](#stage-3--joint-annotation-name-cleanup--facing-direction)); the rest are run directly.
| Tool | Purpose |
|------|---------|
diff --git a/data_process/feature_extraction/extract_features.py b/data_process/feature_extraction/extract_features.py
index 39a6003..d909a3b 100644
--- a/data_process/feature_extraction/extract_features.py
+++ b/data_process/feature_extraction/extract_features.py
@@ -12,7 +12,11 @@ Supports three dataset layouts via ``--dataset_type``:
bare motion stems.
An optional ``filtered_clips.txt`` listing individual clips to skip is read
-for every layout.
+for every layout, and an optional ``clip_trims.txt`` (``<clip stem> <N>``)
+drops the first N frames of a clip's export NPZ on load, and an optional
+``activity_keep.txt`` exempts listed clips from the low-activity filter. All
+three are generated from the hand-reviewed patches by
+``tools/patch_annotations.py``.
Stage-2 (captions, category groups) and stage-3 (clean / face joint names)
metadata JSONs are read from ``data_dir`` โ€” see
@@ -41,6 +45,7 @@ import hashlib
import json
import multiprocessing as mp
import os
+import zipfile
from os.path import join as pjoin
import numpy as np
@@ -148,13 +153,31 @@ def parse_args():
"with optional '#' comments. Default 'auto' uses "
"<data_dir>/filtered_clips.txt if present. Pass an "
"empty string to disable. Complements the on-the-fly "
- "quality filters; entries land in filtered_clips.json.")
+ "quality filters. Listed clips are dropped before "
+ "any object is processed, so they do NOT appear in "
+ "filtered_clips.json (which records runtime filters).")
parser.add_argument("--filtered_objects", type=str, default="auto",
help="(objaverse only) Path to a txt file listing object "
"types to skip (one per line; blanks and '#' comments "
"ignored). Default 'auto' uses "
"<data_dir>/filtered_objects.txt if present. Pass an "
"empty string to disable.")
+ parser.add_argument("--clip_trims", type=str, default="auto",
+ help="Path to a txt file of hand-reviewed head trims, one "
+ "'<clip stem> <N>' per line with optional '#' comments: "
+ "the first N frames of that clip's export NPZ are "
+ "dropped on load, before downsampling and the static "
+ "trim (a bind-pose or foreign first frame that snaps "
+ "into the motion). Default 'auto' uses "
+ "<data_dir>/clip_trims.txt if present. Pass an empty "
+ "string to disable.")
+ parser.add_argument("--activity_keep", type=str, default="auto",
+ help="Path to a txt file of clip stems (one per line, '#' "
+ "comments) exempt from the low-activity filter: "
+ "hand-reviewed small but real motions such as a head "
+ "turn or a wave. Default 'auto' uses "
+ "<data_dir>/activity_keep.txt if present. Pass an empty "
+ "string to disable.")
parser.add_argument("--category_groups", type=str, default="auto",
help="Path to the body-plan category JSON copied into "
"the feature dir (the training loader reads it for "
@@ -208,6 +231,35 @@ def resolve_filtered_clips_path(path, data_dir):
return _resolve_skip_list_path(path, data_dir, 'filtered_clips.txt')
+def resolve_clip_trims_path(path, data_dir):
+ return _resolve_skip_list_path(path, data_dir, 'clip_trims.txt')
+
+
+def resolve_activity_keep_path(path, data_dir):
+ return _resolve_skip_list_path(path, data_dir, 'activity_keep.txt')
+
+
+def load_clip_trims(path):
+ """Read ``<clip stem> <N>`` head trims; a falsy path yields an empty dict.
+
+ Malformed lines raise instead of being skipped: a trim that silently does
+ nothing would leave the snap it was written for in the training data.
+ """
+ trims = {}
+ if not path:
+ return trims
+ with open(path, 'r') as f:
+ for lineno, line in enumerate(f, 1):
+ body = line.split('#', 1)[0].split()
+ if not body:
+ continue
+ if len(body) != 2 or not body[1].isdigit() or int(body[1]) < 1:
+ raise ValueError(f'{path}:{lineno}: expected "<clip stem> <N>" with '
+ f'N >= 1, got {line.strip()!r}')
+ trims[body[0]] = int(body[1])
+ return trims
+
+
def resolve_category_groups(path, metadata):
"""Return the category groups to copy into the feature dir, or None.
@@ -215,7 +267,9 @@ def resolve_category_groups(path, metadata):
the data dir; an empty string drops it; any other value is loaded as an
explicit path. Nothing in this stage consumes the groups โ€” they are
passed through to the feature dir โ€” so dropping them only means the
- copy is made later, by hand, once the classifier has finished.
+ copy is made later, by hand, once the classifier has finished. Like the
+ two skip lists, a missing explicit path is skipped with a warning rather
+ than aborting the run.
"""
if not path:
if metadata.pop('category_groups', None) is not None:
@@ -224,7 +278,10 @@ def resolve_category_groups(path, metadata):
if path != 'auto':
groups = load_json(path)
if groups is None:
- raise ValueError(f'--category_groups {path!r} not found')
+ logger.warning(f'--category_groups {path!r} not found; the feature '
+ f'dir will get no category_groups.json')
+ metadata.pop('category_groups', None)
+ return None
metadata['category_groups'] = groups
return metadata.get('category_groups')
@@ -349,9 +406,43 @@ def _cache_params(task):
# export dirs are symlinks) does not invalidate every entry.
params['clips_digest'] = _digest(
sorted(os.path.basename(p) for p in task['object_npzs']))
+ # Only objects with a head trim carry the key, so adding clip_trims.txt
+ # re-processes exactly those object types and leaves every other cached
+ # result valid.
+ if task.get('head_trims'):
+ params['head_trims_digest'] = _digest(task['head_trims'])
+ # Same for activity_keep.txt: only objects owning a listed clip carry it.
+ if task.get('activity_keep'):
+ params['activity_keep_digest'] = _digest(sorted(task['activity_keep']))
+ # Root offsets are baked into the export NPZs in place (same names), so
+ # neither clips_digest nor any list above sees them; the quats recorded in
+ # the NPZs are the only trace. Adding, changing or removing one therefore
+ # re-processes exactly the owning object type.
+ if task.get('root_offsets'):
+ params['root_offsets_digest'] = _digest(task['root_offsets'])
return params
+# Written into an export NPZ by tools/patch_annotations.py (apply_root_offsets)
+# when it bakes a root orientation fix in; keep the name in sync.
+ROOT_OFFSET_KEY = 'root_offset_applied'
+
+
+def read_root_offsets(object_npzs):
+ """``{clip stem: [w, x, y, z]}`` for the export NPZs carrying a baked-in
+ root offset. Only the zip directory is read for the others, so this stays
+ cheap over a whole dataset."""
+ offsets = {}
+ for path in object_npzs:
+ with zipfile.ZipFile(path) as zf:
+ if ROOT_OFFSET_KEY + '.npy' not in zf.namelist():
+ continue
+ with np.load(path) as d:
+ offsets[os.path.basename(path)[:-4]] = [
+ round(float(x), 6) for x in d[ROOT_OFFSET_KEY]]
+ return offsets
+
+
def _params_hash(params):
"""Stable hash of the cache-invalidating settings."""
blob = json.dumps(params, sort_keys=True, default=str)
@@ -436,6 +527,7 @@ def process_object_task(task):
clip_prefix = task.pop('clip_prefix')
params = _cache_params(task)
params_hash = _params_hash(params)
+ task.pop('root_offsets', None) # cache key only; process_object reads the NPZs
cached = _load_cached_result(part_path, object_type, params, params_hash)
if cached is not None:
@@ -482,8 +574,32 @@ def build_object_tasks(args, clip_stride, motion_dir, metadata):
object_types, all_motions = discover_object_types(motion_dir, args.dataset_type, args)
+ trims_path = resolve_clip_trims_path(args.clip_trims, args.data_dir)
+ clip_trims = load_clip_trims(trims_path)
+ if clip_trims:
+ stems = {m[:-4] for m in all_motions}
+ missing = sorted(set(clip_trims) - stems)
+ logger.info(f'Loaded {len(clip_trims)} head trims from {trims_path}'
+ + (f'; {len(missing)} name no motion being processed '
+ f'(filtered or absent): {missing[:5]}' if missing else ''))
+
+ keep_path = resolve_activity_keep_path(args.activity_keep, args.data_dir)
+ activity_keep = load_filtered_objects(keep_path)
+ if activity_keep:
+ stems = {m[:-4] for m in all_motions}
+ missing = sorted(activity_keep - stems)
+ logger.info(f'Loaded {len(activity_keep)} low-activity exemptions from {keep_path}'
+ + (f'; {len(missing)} name no motion being processed '
+ f'(filtered or absent): {missing[:5]}' if missing else ''))
+
tasks = []
for object_type in object_types:
+ object_npzs = collect_object_npzs(object_type, args.dataset_type,
+ motion_dir, all_motions)
+ head_trims = {s: clip_trims[s] for s in
+ (os.path.basename(p)[:-4] for p in object_npzs) if s in clip_trims}
+ keep = sorted(s for s in (os.path.basename(p)[:-4] for p in object_npzs)
+ if s in activity_keep)
face_joints = get_object_face_joints(object_type, face_joint_names)
if not face_joints:
if args.dataset_type == 'mixamo':
@@ -494,8 +610,10 @@ def build_object_tasks(args, clip_stride, motion_dir, metadata):
tasks.append(dict(
object_type=object_type,
part_path=_object_part_path(args.save_dir, object_type),
- object_npzs=collect_object_npzs(object_type, args.dataset_type,
- motion_dir, all_motions),
+ object_npzs=object_npzs,
+ head_trims=head_trims,
+ activity_keep=keep,
+ root_offsets=read_root_offsets(object_npzs),
# Mixamo clip files carry no "{object_type}-" prefix; its single
# object type owns the whole directory.
clip_prefix=('' if args.dataset_type == 'mixamo'
diff --git a/data_process/joint_annotation/names_clean_llm.py b/data_process/joint_annotation/names_clean_llm.py
index 389c3cd..bbc4614 100644
--- a/data_process/joint_annotation/names_clean_llm.py
+++ b/data_process/joint_annotation/names_clean_llm.py
@@ -57,12 +57,15 @@ from data_process.joint_annotation.llm import ( # noqa: E402
FatalLLMError,
LLMClient,
MAX_RETRIES,
+ RigTimeout,
add_llm_args,
append_id_list,
+ install_rig_alarm,
load_json,
read_id_list,
save_json,
strip_llm_noise,
+ uninstall_rig_alarm,
)
@@ -290,46 +293,59 @@ def rule_based_clean(rig_id, raw_names):
def clean_rig(rig_id, raw_names, client, system_prompt=SYSTEM_PROMPT,
- max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True):
+ max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True,
+ timeout_seconds=0):
# type: (...) -> Tuple[List[str], bool]
"""Clean one rig's joint list via the LLM. Returns ``(names, used_fallback)``.
Structural errors (length mismatch, bad JSON, ...) are fed back to the
- LLM on the next attempt so it can self-correct.
+ LLM on the next attempt so it can self-correct. ``timeout_seconds`` arms
+ a per-rig SIGALRM (0 disables) so a hung backend cannot stall the batch;
+ a timed-out rig falls back to the rules like an exhausted retry loop.
"""
if not raw_names:
return [], False
last_err = None
correction = None
- for attempt in range(1, max_retries + 1):
- try:
- raw_text = client.generate(
- system_prompt, build_user_prompt(rig_id, raw_names, correction=correction),
- max_tokens)
- return parse_response(raw_text, len(raw_names)), False
- except FatalLLMError:
- raise
- except Exception as err: # noqa: BLE001 โ€” retry on any backend/parse error
- last_err = err
- if isinstance(err, ValueError):
- correction = (
- "Your previous response failed validation ({}). Return a "
- "JSON array of EXACTLY {} strings โ€” one cleaned label per "
- "input joint, in the same order. Do not add, merge, split, "
- "or skip any entries. Output only the JSON array, no prose "
- "or code fences."
- ).format(err, len(raw_names))
- logger.warning("[{}] attempt {}/{} failed: {}".format(
- rig_id, attempt, max_retries, err))
- if attempt < max_retries:
- time.sleep(1.5)
-
- if not fallback_on_error:
+ timed_out = False
+ prev_handler = install_rig_alarm(timeout_seconds)
+ try:
+ for attempt in range(1, max_retries + 1):
+ try:
+ raw_text = client.generate(
+ system_prompt, build_user_prompt(rig_id, raw_names, correction=correction),
+ max_tokens)
+ return parse_response(raw_text, len(raw_names)), False
+ except (RigTimeout, FatalLLMError):
+ raise
+ except Exception as err: # noqa: BLE001 โ€” retry on any backend/parse error
+ last_err = err
+ if isinstance(err, ValueError):
+ correction = (
+ "Your previous response failed validation ({}). Return a "
+ "JSON array of EXACTLY {} strings โ€” one cleaned label per "
+ "input joint, in the same order. Do not add, merge, split, "
+ "or skip any entries. Output only the JSON array, no prose "
+ "or code fences."
+ ).format(err, len(raw_names))
+ logger.warning("[{}] attempt {}/{} failed: {}".format(
+ rig_id, attempt, max_retries, err))
+ if attempt < max_retries:
+ time.sleep(1.5)
+ except RigTimeout:
+ timed_out = True
+ last_err = "rig timeout after {}s".format(timeout_seconds)
+ finally:
+ uninstall_rig_alarm(prev_handler)
+
+ if not fallback_on_error and not timed_out:
raise RuntimeError("LLM cleaning failed for {}: {}".format(rig_id, last_err))
- logger.error("[{}] all {} attempts failed ({}); falling back to rule-based "
- "cleaner".format(rig_id, max_retries, last_err))
+ logger.error("[{}] {} ({}); falling back to rule-based cleaner".format(
+ rig_id,
+ "timed out" if timed_out else "all {} attempts failed".format(max_retries),
+ last_err))
return rule_based_clean(rig_id, raw_names), True
@@ -339,7 +355,8 @@ def clean_rig(rig_id, raw_names, client, system_prompt=SYSTEM_PROMPT,
def clean_all(input_path, output_path, client, system_prompt=SYSTEM_PROMPT,
max_tokens=2048, max_retries=MAX_RETRIES, fallback_on_error=True,
- failed_path=None, overwrite=False, redo_failed=False):
+ failed_path=None, overwrite=False, redo_failed=False,
+ rig_timeout=0):
# type: (...) -> Dict[str, List[str]]
"""Clean every rig in ``input_path`` and write to ``output_path``.
@@ -387,6 +404,7 @@ def clean_all(input_path, output_path, client, system_prompt=SYSTEM_PROMPT,
rig_id, raw_names, client,
system_prompt=system_prompt, max_tokens=max_tokens,
max_retries=max_retries, fallback_on_error=fallback_on_error,
+ timeout_seconds=rig_timeout,
)
fallback_count += int(used_fallback)
cleaned[rig_id] = names_out
@@ -439,6 +457,11 @@ def main():
help="Also re-process the rig ids listed in --failed_log "
"(they always have a full-length entry from the "
"rule-based fallback, so they are otherwise skipped).")
+ parser.add_argument("--rig_timeout", type=int, default=0,
+ help="Per-rig wall-clock budget in seconds (SIGALRM); a "
+ "rig that exceeds it falls back to the rule-based "
+ "cleaner and is listed in --failed_log. 0 disables "
+ "(default). Thinking backends need a generous value.")
add_llm_args(parser, default_max_tokens=2048)
args = parser.parse_args()
@@ -457,6 +480,7 @@ def main():
failed_path=args.failed_log,
overwrite=args.overwrite,
redo_failed=args.redo_failed,
+ rig_timeout=args.rig_timeout,
)
with open(args.input, "r") as f:
diff --git a/data_process/joint_annotation/names_clean_rule.py b/data_process/joint_annotation/names_clean_rule.py
index 9c00973..00660c9 100644
--- a/data_process/joint_annotation/names_clean_rule.py
+++ b/data_process/joint_annotation/names_clean_rule.py
@@ -15,6 +15,7 @@ import json
import os
import re
import sys
+import tempfile
_REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
if _REPO_ROOT not in sys.path:
@@ -583,8 +584,20 @@ def main():
for animal, joints in data.items():
clean_data[animal] = [post_process(clean_joint_name(j, animal)) for j in joints]
- with open(output_path, "w") as f:
- json.dump(clean_data, f, indent=2)
+ # Atomic write (temp file + os.replace), like every other annotation
+ # writer: a crash mid-dump must not leave a truncated
+ # clean_joint_names.json that the LLM pass would then "resume" from.
+ fd, tmp = tempfile.mkstemp(prefix=os.path.basename(output_path) + ".",
+ suffix=".tmp",
+ dir=os.path.dirname(output_path) or ".")
+ try:
+ with os.fdopen(fd, "w") as f:
+ json.dump(clean_data, f, indent=2)
+ os.replace(tmp, output_path)
+ except BaseException:
+ if os.path.exists(tmp):
+ os.unlink(tmp)
+ raise
print("Written {} entries to {}".format(len(clean_data), output_path))
diff --git a/data_process/mesh_animation/animate_lbs.py b/data_process/mesh_animation/animate_lbs.py
index c971005..9093d2a 100644
--- a/data_process/mesh_animation/animate_lbs.py
+++ b/data_process/mesh_animation/animate_lbs.py
@@ -178,7 +178,14 @@ def load_motion_local_mats(anim_path, dataset_type=None, cond_path=None):
build_anim_from_npz,
load_cond_data,
)
- assert dataset_type, "--dataset_type is required for feature-format motion"
+ # Either flag resolves the cond: --dataset_type picks the dataset's
+ # default cond.npy (and the 'mixamo' key), --cond_path names the file
+ # directly (object type from the clip's filename prefix, or the file's
+ # single entry โ€” a preprocess_char cond has exactly one).
+ if not dataset_type and not cond_path:
+ raise RuntimeError(
+ "feature-format motion needs --dataset_type and/or --cond_path "
+ "(omit both only for a canonical asset from preprocess_char)")
cond_path = cond_path or COND_PATH_TEMPLATE.format(dataset_type=dataset_type)
cond_data = load_cond_data(cond_path, anim_path, dataset_type)
anim, rest_anim, _tpos, raw = build_anim_from_npz(anim_path, cond_data)
diff --git a/data_process/mesh_animation/animate_motion.py b/data_process/mesh_animation/animate_motion.py
index 6613b03..60b23c4 100644
--- a/data_process/mesh_animation/animate_motion.py
+++ b/data_process/mesh_animation/animate_motion.py
@@ -222,9 +222,22 @@ def resolve_char_path(anim_path, dataset_type):
def load_cond_data(cond_path, anim_path, dataset_type):
+ """Pick the clip's cond entry out of ``cond_path``.
+
+ The key is the object type resolved from the filename (or ``'mixamo'``).
+ A cond file holding exactly one entry (``preprocess_char.py`` output, or
+ a custom single-asset export) is accepted whatever the clip is called,
+ since there is nothing to disambiguate.
+ """
object_type = resolve_object_type(anim_path, dataset_type)
cond_dict = np.load(cond_path, allow_pickle=True).item()
if object_type not in cond_dict:
+ if len(cond_dict) == 1:
+ (only_key,) = cond_dict.keys()
+ logger.warning(
+ f"object_type '{object_type}' not in {cond_path}; using its single "
+ f"entry '{only_key}'")
+ return cond_dict[only_key]
raise KeyError(
f"object_type '{object_type}' (resolved from '{anim_path}' under "
f"dataset_type='{dataset_type}') not found in cond file {cond_path}; "
diff --git a/data_process/mesh_animation/preprocess_char.py b/data_process/mesh_animation/preprocess_char.py
index 8f44487..88bacaf 100644
--- a/data_process/mesh_animation/preprocess_char.py
+++ b/data_process/mesh_animation/preprocess_char.py
@@ -274,9 +274,15 @@ def bake_canonical_asset(char_path, cond, out_base, formats=('glb',)):
def preprocess_asset(char_path, output_dir, face_r=None, face_l=None,
body_axis=False, target_diameter=2.0, formats=('glb',),
- save_vis=False, apply_clip=False, max_clip_len=10 ** 8,
- keep_intermediate=False):
- """Export + extract + bake one asset; see the module docstring."""
+ save_vis=False, apply_clip=False, max_clip_len=200,
+ clip_stride=110, keep_intermediate=False):
+ """Export + extract + bake one asset; see the module docstring.
+
+ ``max_clip_len`` / ``clip_stride`` mirror stage 4's ``--max_clip_len`` and
+ ``max_clip_len - diffusion_max_len``: without ``apply_clip`` a motion is
+ truncated to ``max_clip_len`` frames, with it the motion is cropped into
+ overlapping windows of that length.
+ """
assert os.path.exists(char_path), f"Asset not found: {char_path}"
name = os.path.splitext(os.path.basename(char_path))[0]
os.makedirs(output_dir, exist_ok=True)
@@ -305,6 +311,7 @@ def preprocess_asset(char_path, output_dir, face_r=None, face_l=None,
face_joints=face_joints, clean_names=clean_names,
expected_names=raw_names,
apply_clip=apply_clip, max_clip_len=max_clip_len,
+ clip_stride=clip_stride,
target_diameter=target_diameter, save_vis=save_vis)
assert cond is not None, ("feature extraction skipped this asset "
"(degenerate skeleton or no usable clips โ€” see log)")
@@ -363,8 +370,17 @@ def parse_args():
parser.add_argument("--save_vis", action='store_true',
help="Also render per-clip preview MP4s (slow).")
parser.add_argument("--apply_clip", action='store_true',
- help="Crop motions into overlapping training-length clips "
- "instead of one full-length feature NPZ per motion.")
+ help="Crop motions into overlapping windows of "
+ "--max_clip_len frames (stride max_clip_len - "
+ "diffusion_max_len) instead of truncating each "
+ "motion to --max_clip_len frames.")
+ parser.add_argument("--max_clip_len", type=int, default=200,
+ help="Frames kept per motion / window length "
+ "(stage-4 default: 200).")
+ parser.add_argument("--diffusion_max_len", type=int, default=90,
+ help="Training crop length used to derive the window "
+ "stride max_clip_len - diffusion_max_len "
+ "(stage-4 default: 90).")
parser.add_argument("--keep_intermediate", action='store_true',
help="Keep export/ NPZs and T-pose/preview visuals (default: "
"only canonical asset + cond.npy + motions/ remain).")
@@ -373,6 +389,10 @@ def parse_args():
if __name__ == "__main__":
args = parse_args()
+ if args.apply_clip and args.max_clip_len <= args.diffusion_max_len:
+ raise SystemExit(
+ f"--apply_clip needs max_clip_len ({args.max_clip_len}) > "
+ f"diffusion_max_len ({args.diffusion_max_len}) for a positive stride")
preprocess_asset(
char_path=args.char_path,
output_dir=args.output_dir,
@@ -383,5 +403,7 @@ if __name__ == "__main__":
formats=tuple(f.strip().lower() for f in args.formats.split(',') if f.strip()),
save_vis=args.save_vis,
apply_clip=args.apply_clip,
+ max_clip_len=args.max_clip_len,
+ clip_stride=args.max_clip_len - args.diffusion_max_len,
keep_intermediate=args.keep_intermediate,
)
diff --git a/data_process/motion_export/export_general.py b/data_process/motion_export/export_general.py
index 9b5d427..55d2224 100644
--- a/data_process/motion_export/export_general.py
+++ b/data_process/motion_export/export_general.py
@@ -257,23 +257,27 @@ def main():
if saved:
all_joint_names.update(saved)
except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
logger.exception(f"Failed to export {asset_path}")
with open(error_log, 'a') as log_file:
log_file.write(f"Failed to export {asset_path}: {e}\n")
worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""
- write_export_summary(args.output_dir, all_joint_names,
+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,
worker_suffix=worker_suffix)
- else:
- saved = export_asset(args.input, args.output_dir, save_name=args.name,
- **export_kwargs)
- if saved:
- write_export_summary(args.output_dir, saved)
+ # Summaries are written first so a partial batch still leaves the
+ # canonical JSONs consistent; the non-zero exit then tells the
+ # wrapper not to merge shards and Slurm afterok chains to stop.
if n_failed:
logger.error(f"{n_failed}/{len(asset_paths)} assets failed; "
f"see {error_log}")
sys.exit(1)
+ else:
+ saved = export_asset(args.input, args.output_dir, save_name=args.name,
+ **export_kwargs)
+ if saved:
+ write_export_summary(args.output_dir, saved, fps=args.fps)
if __name__ == "__main__":
diff --git a/data_process/motion_export/export_mixamo.py b/data_process/motion_export/export_mixamo.py
index 0cc849a..dbc67a6 100644
--- a/data_process/motion_export/export_mixamo.py
+++ b/data_process/motion_export/export_mixamo.py
@@ -217,7 +217,8 @@ def main(args):
ref_bone_names = json.load(f).get("mixamo")
all_joint_names = {"mixamo": ref_bone_names} if ref_bone_names else {}
- write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)
+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,
+ worker_suffix=worker_suffix)
if n_failed:
logger.error(f"{n_failed}/{len(anim_paths)} clips failed; see {error_log}")
diff --git a/data_process/motion_export/export_objaverse.py b/data_process/motion_export/export_objaverse.py
index c2b0818..7493cc2 100644
--- a/data_process/motion_export/export_objaverse.py
+++ b/data_process/motion_export/export_objaverse.py
@@ -188,7 +188,8 @@ def main():
log_file.write(f"Failed to export {glb_path}: {e}\n")
worker_suffix = f"_worker{args.worker_id}" if args.num_workers > 1 else ""
- write_export_summary(args.output_dir, all_joint_names, worker_suffix=worker_suffix)
+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps,
+ worker_suffix=worker_suffix)
if n_failed:
logger.error(f"{n_failed}/{len(gltf_paths)} assets failed; see {error_log}")
diff --git a/data_process/motion_export/export_truebones.py b/data_process/motion_export/export_truebones.py
index 5edb21c..f578146 100644
--- a/data_process/motion_export/export_truebones.py
+++ b/data_process/motion_export/export_truebones.py
@@ -253,7 +253,7 @@ def main():
with open(error_log, 'a') as log_file:
log_file.write(f"Failed to export species {species}: {e}\n")
- write_export_summary(args.output_dir, all_joint_names)
+ write_export_summary(args.output_dir, all_joint_names, fps=args.fps)
if n_failed:
logger.error(f"{n_failed}/{len(by_species)} species failed; see {error_log}")
sys.exit(1)
diff --git a/data_process/motion_rendering/render_tpose.py b/data_process/motion_rendering/render_tpose.py
index f4414a8..8de3e2e 100644
--- a/data_process/motion_rendering/render_tpose.py
+++ b/data_process/motion_rendering/render_tpose.py
@@ -15,6 +15,7 @@ Usage (plain python โ€” EEVEE needs the pip ``bpy`` module's GPU context):
"""
import os
+import sys
import argparse
from pathlib import Path
@@ -154,14 +155,27 @@ def main():
error_log = os.path.join(args.output_dir,
f'tpose_errors_worker{args.worker_id}.log')
+ n_failed = 0
for name, path in assets:
try:
render_asset_tpose(name, path, args.output_dir, args)
- except Exception as e:
- logger.error(f"Failed to render {path}: {e}")
+ except Exception as e: # noqa: BLE001 โ€” keep the batch going
+ n_failed += 1
+ logger.exception(f"Failed to render {path}")
with open(error_log, 'a') as f:
f.write(f"{path}\t{e}\n")
+ if n_failed:
+ logger.error(f"{n_failed}/{len(assets)} assets failed; see {error_log}")
+ return 1 if n_failed else 0
+
if __name__ == "__main__":
- main()
+ # bpy/Blender exits 0 even on an uncaught exception, which hides failures
+ # from `set -e` and from Slurm's afterok dependencies (same guard as the
+ # motion renderers).
+ try:
+ sys.exit(main())
+ except Exception:
+ logger.exception("render_tpose failed")
+ sys.exit(1)
diff --git a/data_process/scripts/run_animate_lbs.sh b/data_process/scripts/run_animate_lbs.sh
old mode 100644
new mode 100755
diff --git a/data_process/scripts/run_animate_motion.sh b/data_process/scripts/run_animate_motion.sh
index d42f44d..7c9ab7b 100755
--- a/data_process/scripts/run_animate_motion.sh
+++ b/data_process/scripts/run_animate_motion.sh
@@ -11,10 +11,13 @@
#
# Env overrides:
# ANIM_PATH (required) motion .npz, or a .npy of model motion features
-# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; required for mixamo)
+# CHAR_PATH character mesh (auto-resolved for truebones/objaverse; mixamo
+# defaults to dataset/raw/mixamo/character_refined/Y_Bot.fbx)
# COND_PATH cond.npy (default: dataset/features/<dataset>/cond.npy)
# OUTPUT_DIR (default: outputs/animated)
# ANIM_MODE fk|ik, .npy input only (default: fk)
+# EXTRA_BONES_STRATEGY merge|remove|keep for armature bones absent from the
+# motion (default: merge, see animate_motion.py)
set -euo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
@@ -29,13 +32,23 @@ ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion .npz/.npy for dataset=$DATASET}
OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}
ANIM_MODE=${ANIM_MODE:-fk}
+# Mixamo has no per-object mesh to auto-resolve; the animation FBXs are
+# authored on Y Bot, so that character is the default when it is downloaded.
+MIXAMO_DEFAULT_CHAR=dataset/raw/mixamo/character_refined/Y_Bot.fbx
+if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then
+ if [[ -f "$MIXAMO_DEFAULT_CHAR" ]]; then
+ CHAR_PATH=$MIXAMO_DEFAULT_CHAR
+ else
+ echo "dataset=mixamo requires CHAR_PATH (default $MIXAMO_DEFAULT_CHAR not found;" \
+ "download it with run_download.sh mixamo)" >&2
+ exit 1
+ fi
+fi
+
EXTRA_ARGS=()
[[ -n "${CHAR_PATH:-}" ]] && EXTRA_ARGS+=(--char_path="$CHAR_PATH")
[[ -n "${COND_PATH:-}" ]] && EXTRA_ARGS+=(--cond_path="$COND_PATH")
-if [[ "$DATASET" == "mixamo" && -z "${CHAR_PATH:-}" ]]; then
- echo "dataset=mixamo requires CHAR_PATH (no auto-resolved character mesh)" >&2
- exit 1
-fi
+[[ -n "${EXTRA_BONES_STRATEGY:-}" ]] && EXTRA_ARGS+=(--extra_bones_strategy="$EXTRA_BONES_STRATEGY")
blender -b -P data_process/mesh_animation/animate_motion.py -- \
--dataset_type="$DATASET" \
diff --git a/data_process/scripts/run_animate_npz.sh b/data_process/scripts/run_animate_npz.sh
index 6fe9f9b..c30055b 100755
--- a/data_process/scripts/run_animate_npz.sh
+++ b/data_process/scripts/run_animate_npz.sh
@@ -10,6 +10,8 @@
# ANIM_PATH (required โ€” NPZ whose skeleton matches the rig)
# OUTPUT_DIR (default: outputs/animated)
# CHAR_ANIM_TYPE glb|fbx (default: glb)
+# EXTRA_BONES_STRATEGY merge|remove|keep for armature bones absent from the
+# NPZ (default: merge, see animate_npz.py)
set -euo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
@@ -21,8 +23,11 @@ ANIM_PATH=${ANIM_PATH:?Set ANIM_PATH to a motion NPZ matching the rig}
OUTPUT_DIR=${OUTPUT_DIR:-outputs/animated}
CHAR_ANIM_TYPE=${CHAR_ANIM_TYPE:-glb}
+EXTRA_ARGS=()
+[[ -n "${EXTRA_BONES_STRATEGY:-}" ]] && EXTRA_ARGS+=(--extra_bones_strategy="$EXTRA_BONES_STRATEGY")
+
blender -b -P data_process/mesh_animation/animate_npz.py -- \
--char_path="$CHAR_PATH" \
--anim_path="$ANIM_PATH" \
--output_dir="$OUTPUT_DIR" \
- --char_anim_type="$CHAR_ANIM_TYPE" "$@"
+ --char_anim_type="$CHAR_ANIM_TYPE" "${EXTRA_ARGS[@]}" "$@"
diff --git a/data_process/scripts/run_caption_motion.sh b/data_process/scripts/run_caption_motion.sh
index 6e4b5ec..de38b24 100755
--- a/data_process/scripts/run_caption_motion.sh
+++ b/data_process/scripts/run_caption_motion.sh
@@ -48,7 +48,7 @@ done
MODEL=${MODEL:-Qwen/Qwen3.5-9B}
if [[ "${MODEL,,}" == *qwen* ]]; then
backend=local # backends.py: BACKEND_QWEN
-elif [[ "$MODEL" == gemini* ]]; then
+elif [[ "${MODEL,,}" == *gemini* ]]; then
backend=gemini
else
backend=openai
@@ -80,7 +80,8 @@ MAX_RETRIES=${MAX_RETRIES:-4}
EXTRA=()
case "$backend" in
local)
- MAX_TOKENS=${MAX_TOKENS:-800} ;;
+ MAX_TOKENS=${MAX_TOKENS:-800}
+ EXTRA+=(--max_retries="$MAX_RETRIES") ;;
gemini)
# Reasoning/answer tokens share one budget -> generous default.
# Default 2 workers โ€” token/minute quotas are easy to exhaust with
@@ -131,7 +132,7 @@ if [[ "$MODE" == "multi-gpu" ]]; then
pids=()
for gpu_id in $(seq 0 $((num_gpus - 1))); do
- python -m data_process.vlm_caption.caption_motion "${COMMON[@]}" \
+ python -m data_process.vlm_caption.caption_motion "${COMMON[@]}" "${EXTRA[@]}" \
--num_gpus="$num_gpus" --gpu_id="$gpu_id" \
"${PASSTHRU[@]}" > >(tee "$LOG_DIR/gpu${gpu_id}.log") 2>&1 &
pids+=($!)
diff --git a/data_process/scripts/run_export.sh b/data_process/scripts/run_export.sh
index 352a167..685b924 100755
--- a/data_process/scripts/run_export.sh
+++ b/data_process/scripts/run_export.sh
@@ -30,6 +30,13 @@ handle_help "$@"
MODE=auto
case "${1:-}" in
truebones|mixamo|objaverse|auto) MODE="$1"; shift ;;
+ ""|--*) ;; # no dataset given: auto mode (DATA_DIR required), flags follow
+ *)
+ # A bare word that is not a dataset name is almost always a typo;
+ # silently treating it as an exporter flag would land in auto mode
+ # and fail later on a missing DATA_DIR.
+ echo "ERROR: unknown dataset '$1' (expected truebones | mixamo | objaverse | auto)" >&2
+ exit 2 ;;
esac
NUM_WORKERS=1
diff --git a/data_process/scripts/run_export_general.sh b/data_process/scripts/run_export_general.sh
old mode 100644
new mode 100755
diff --git a/data_process/scripts/run_joints_face_correct_llm.sh b/data_process/scripts/run_joints_face_correct_llm.sh
index 7cca9b0..ac01492 100755
--- a/data_process/scripts/run_joints_face_correct_llm.sh
+++ b/data_process/scripts/run_joints_face_correct_llm.sh
@@ -26,7 +26,10 @@ RAW=${RAW:-$(export_dir "$DATASET")/joint_names.json}
CLEANED=${CLEANED:-$(export_dir "$DATASET")/clean_joint_names.json}
FACE=${FACE:-$(export_dir "$DATASET")/face_joint_names.json}
MODEL=${MODEL:-deepseek-v4-flash}
-MAX_TOKENS=${MAX_TOKENS:-512}
+# Matches the script's own default: this pass forces reasoning_effort=medium,
+# and on gpt-5* models the reasoning tokens share this budget (512 hit
+# finish_reason='length').
+MAX_TOKENS=${MAX_TOKENS:-2048}
# Thinking backends need a much larger per-rig budget (see names_correct wrapper).
if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-15}; fi
diff --git a/data_process/scripts/run_joints_names_clean_llm.sh b/data_process/scripts/run_joints_names_clean_llm.sh
index 53b688d..bf7b086 100755
--- a/data_process/scripts/run_joints_names_clean_llm.sh
+++ b/data_process/scripts/run_joints_names_clean_llm.sh
@@ -6,8 +6,10 @@
# MODEL=deepseek-v4-flash bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # DeepSeek API
# MODEL=Qwen/Qwen3-8B bash data_process/scripts/run_joints_names_clean_llm.sh objaverse # local HF model
#
-# Env overrides: INPUT, OUTPUT, MODEL, MAX_TOKENS. Extra args are passed through
-# (e.g. --no_fallback). API backends read OPENAI_API_KEY / DEEPSEEK_API_KEY.
+# Env overrides: INPUT, OUTPUT, MODEL, MAX_TOKENS, RIG_TIMEOUT (per-rig budget
+# in seconds; default 300 for deepseek*, else 60). Extra args are passed
+# through (e.g. --no_fallback, --overwrite, --redo_failed). API backends read
+# OPENAI_API_KEY / DEEPSEEK_API_KEY.
set -euo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
@@ -22,8 +24,11 @@ INPUT=${INPUT:-$(export_dir "$DATASET")/joint_names.json}
OUTPUT=${OUTPUT:-$(export_dir "$DATASET")/clean_joint_names.json}
MODEL=${MODEL:-deepseek-v4-flash}
MAX_TOKENS=${MAX_TOKENS:-2048}
+# One batched call per rig; thinking backends need a much larger budget (same
+# split as the other stage-3 wrappers).
+if [[ "$MODEL" == deepseek* ]]; then RIG_TIMEOUT=${RIG_TIMEOUT:-300}; else RIG_TIMEOUT=${RIG_TIMEOUT:-60}; fi
python -m data_process.joint_annotation.names_clean_llm \
--input="$INPUT" --output="$OUTPUT" \
- --model="$MODEL" --max_tokens="$MAX_TOKENS" \
+ --model="$MODEL" --max_tokens="$MAX_TOKENS" --rig_timeout="$RIG_TIMEOUT" \
--report "$@"
diff --git a/data_process/scripts/run_joints_vis_facing.sh b/data_process/scripts/run_joints_vis_facing.sh
old mode 100644
new mode 100755
diff --git a/data_process/scripts/run_preprocess_char.sh b/data_process/scripts/run_preprocess_char.sh
old mode 100644
new mode 100755
diff --git a/data_process/scripts/run_render_motion.sh b/data_process/scripts/run_render_motion.sh
index 3056677..bb81a40 100755
--- a/data_process/scripts/run_render_motion.sh
+++ b/data_process/scripts/run_render_motion.sh
@@ -42,6 +42,14 @@ EXTRA_ARGS=()
while [[ $# -gt 0 ]]; do
case "$1" in
--multi-worker) NUM_WORKERS="${2:?--multi-worker requires a value}"; shift 2 ;;
+ --missing-only)
+ # Only render_objaverse.py implements the flag; the other two
+ # renderers would reject it from argparse after loading bpy.
+ if [[ "$DATASET" != "objaverse" ]]; then
+ echo "ERROR: --missing-only is only supported for objaverse (the other renderers resume per asset anyway)" >&2
+ exit 2
+ fi
+ EXTRA_ARGS+=("$1"); shift ;;
*) EXTRA_ARGS+=("$1"); shift ;;
esac
done
diff --git a/data_process/tools/patch_annotations.py b/data_process/tools/patch_annotations.py
index fa7bab3..3078cfb 100644
--- a/data_process/tools/patch_annotations.py
+++ b/data_process/tools/patch_annotations.py
@@ -24,11 +24,12 @@ place. Two kinds of fixes:
- the ``clean`` fields of face_joint_names.json are re-synced;
- objaverse rigs are flagged in ``rig_flags.json``: automatically when the
facing pair cannot be trusted (``source: empty``, a Bone/Bone pair, a body
- axis through unnamed bones), plus the hand-reviewed categories. Only
- ``tpose_wrong`` rigs are also written to ``filtered_objects.txt``, which
- stage 4 skips; every other flag is informational and the rig is kept;
- - caption grammar (``breaksdance`` -> ``breakdances``) and the per-dataset
- subject phrase;
+ axis through unnamed bones), plus the hand-reviewed categories. Only the
+ ``FILTER_CATEGORIES`` rigs (``tpose_wrong``, ``not_in_legacy_raw``) are also
+ written to ``filtered_objects.txt``, which stage 4 skips; every other flag is
+ informational and the rig is kept;
+ - caption grammar (``breaksdance`` -> ``breakdances``) and the one corpus
+ subject phrase (``An object``, the same in all three datasets);
- locomotion qualifiers grounded in the root trajectory: ``<verb> forward``
with no root travel -> ``<verb> in place``; ``<verb> in place`` with clear
travel -> ``<verb> forward|backward`` when the travel is clearly along the
@@ -43,9 +44,18 @@ place. Two kinds of fixes:
<ds>_categories.json {rig: category} (moves the rig in category_groups.json)
<ds>_filtered_clips.txt <clip> # <reason> (any dataset: individual clips ->
export/<ds>/filtered_clips.txt, skipped by stage 4)
+ <ds>_clip_trims.txt <clip> <N> # <reason> (any dataset: drop the first N frames of the clip's
+ export NPZ -> export/<ds>/clip_trims.txt, applied by stage 4 on load)
+ <ds>_activity_keep.txt <clip> # <reason> (any dataset: small but real motions exempt from
+ stage 4's low-activity filter -> export/<ds>/activity_keep.txt)
+ <ds>_root_offsets.json {clip: {"quat": [w, x, y, z], "reason": s}} (any dataset: the source
+ file keyed the root joint with a constant extra rotation for the whole clip, so
+ the animal is tilted; export/<ds>/motions/<clip>.npz is REWRITTEN with
+ anim_local_rot[:, 0] = anim_local_rot[:, 0] * quat on every frame. Idempotent,
+ and undone when the entry is removed; see apply_root_offsets)
<ds>_rig_flags.txt <rig> # <category>: <reason> (objaverse: hand-reviewed rig flags ->
- export/objaverse/rig_flags.json; only tpose_wrong also goes to
- export/objaverse/filtered_objects.txt, which stage 4 skips)
+ export/objaverse/rig_flags.json; only tpose_wrong / not_in_legacy_raw
+ also go to export/objaverse/filtered_objects.txt, which stage 4 skips)
Usage (from the repo root; ``dataset/export/<ds>`` may be symlinks):
python data_process/tools/patch_annotations.py [--dry_run] [--datasets truebones mixamo]
@@ -61,12 +71,27 @@ import glob
import json
import os
import re
+import sys
+import tempfile
from collections import Counter, OrderedDict
import numpy as np
+# Usually run as a script (`python data_process/tools/patch_annotations.py`),
+# so the repo root is not on sys.path and `data_process` is not importable.
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))
+
+from data_process.utils.kinematics import ( # noqa: E402
+ fk, frame_global, qmul, qrot, rest_global,
+)
+
DATASETS = ('truebones', 'mixamo', 'objaverse')
-SUBJECT = {'truebones': 'An animal', 'mixamo': 'A person', 'objaverse': 'An object'}
+# One subject for the whole corpus. The three datasets are trained as one
+# mixture, so a per-dataset subject ('An animal' / 'A person') would hand the
+# text encoder a free dataset label and defeat the topology-agnostic
+# conditioning. Keep this in sync with the `subject=` fields in
+# data_process/vlm_caption/prompts.py.
+SUBJECT = {ds: 'An object' for ds in DATASETS}
# Rigs whose multi-segment legs / claw arms are labelled by chain position.
CHAIN_RIGS = {
@@ -113,50 +138,11 @@ LOCO_VERBS = ('walks|runs|trots|jogs|sprints|marches|gallops|crawls|slithers|swi
'bounds|scurries|waddles|limps|sneaks|strides|paddles|glides|dashes|skips|struts|'
'creeps|prowls|stalks|shuffles|scampers|canters|flutters|soars|hovers|strolls|'
'wanders|races')
+CAPTION_FRAMES = 200 # blender_render.MAX_RENDER_FRAMES
FACING_COS = 0.8 # |cos(travel, facing)| needed to call travel forward / backward
LOCO_RE = re.compile(r'\b(%s)((?: \w+ly)?) (forward|in place)\b' % LOCO_VERBS)
-# ---------------------------------------------------------------------------
-# Kinematics (numpy only; quaternions are w,x,y,z as written by the exporter)
-# ---------------------------------------------------------------------------
-
-def qmul(a, b):
- w1, x1, y1, z1 = a[..., 0], a[..., 1], a[..., 2], a[..., 3]
- w2, x2, y2, z2 = b[..., 0], b[..., 1], b[..., 2], b[..., 3]
- return np.stack([w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2,
- w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,
- w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2,
- w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2], -1)
-
-
-def qrot(q, v):
- qv = np.concatenate([np.zeros(v.shape[:-1] + (1,)), v], -1)
- return qmul(qmul(q, qv), q * np.array([1, -1, -1, -1]))[..., 1:]
-
-
-def fk(local_pos, local_rot, parents):
- n = len(parents)
- g = np.zeros((n, 3))
- gr = np.zeros((n, 4))
- for i in range(n):
- p = parents[i]
- if p < 0:
- g[i], gr[i] = local_pos[i], local_rot[i]
- else:
- g[i] = g[p] + qrot(gr[p], local_pos[i])
- gr[i] = qmul(gr[p], local_rot[i])
- return g
-
-
-def rest_global(d):
- return fk(d['rest_local_pos'], d['rest_local_rot'], d['parents'])
-
-
-def frame_global(d, t=0):
- return fk(d['anim_local_pos'][t], d['anim_local_rot'][t], d['parents'])
-
-
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
@@ -175,12 +161,21 @@ def load_overrides(path):
def save_json(path, obj, dry_run):
+ """Atomic write (temp file in the target dir + ``os.replace``). The temp
+ name is unique per process so two concurrent runs cannot clobber each
+ other's half-written file."""
if dry_run:
return
- tmp = path + '.tmp'
- with open(tmp, 'w') as f:
- json.dump(obj, f, indent=2, ensure_ascii=False)
- os.replace(tmp, path)
+ fd, tmp = tempfile.mkstemp(prefix=os.path.basename(path) + '.',
+ suffix='.tmp', dir=os.path.dirname(path) or '.')
+ try:
+ with os.fdopen(fd, 'w') as f:
+ json.dump(obj, f, indent=2, ensure_ascii=False)
+ os.replace(tmp, path)
+ except BaseException:
+ if os.path.exists(tmp):
+ os.unlink(tmp)
+ raise
def object_type_of(ds, clip):
@@ -397,14 +392,27 @@ def patch_face_pairs(ds, face, clean, names, overrides, stats, log):
# them); every other category is just recorded in rig_flags.json.
AUTO_FLAG = {'empty': 'empty_pair', 'bone/bone pair': 'bone_pair',
'body axis through unnamed bones': 'body_axis_unnamed'}
-FILTER_CATEGORIES = ('tpose_wrong',)
+FILTER_CATEGORIES = ('tpose_wrong', 'not_in_legacy_raw')
+
+# One line per filtering category for the filtered_objects.txt header, so adding a
+# category to FILTER_CATEGORIES does not leave the header describing only the first one.
+FILTER_CATEGORY_NOTES = {
+ 'tpose_wrong': 'the exported rest pose lies flat / is rotated / upside-down, so the stage-4 T-pose is unusable',
+ 'not_in_legacy_raw': 'the raw GLB is absent from the legacy 6,974-asset raw set and is held out of training',
+}
def load_rig_flags(path, names, log):
"""``patches/<ds>_rig_flags.txt``: one ``<rig> # <category>: <reason>`` per line โ€” hand-reviewed
- rigs (``tpose_wrong`` = rest pose lying / rotated, ``facing_wrong`` = the facing pair does not give the
- real front, ``object_no_front`` = prop without a front, ``verified_ok`` = hand-checked, clears an automatic
- flag). Unknown rigs are reported and ignored."""
+ rigs (``tpose_wrong`` = rest pose lying / rotated, ``not_in_legacy_raw`` = raw GLB absent from the legacy
+ raw set and held out, ``facing_wrong`` = the facing pair does not give the real front, ``object_no_front``
+ = prop without a front, ``verified_ok`` = hand-checked, clears an automatic flag). Unknown rigs are
+ reported and ignored.
+
+ A rig listed twice keeps the LAST line, which is how a later decision overrides an earlier
+ one, but the override is reported: two lines with different categories mean the earlier
+ verdict (often a ``verified_ok`` review) silently disappears from ``rig_flags.json``, and
+ which one wins depends on the order of the file rather than on anything meaningful."""
out = OrderedDict()
if not os.path.isfile(path):
return out
@@ -418,7 +426,14 @@ def load_rig_flags(path, names, log):
log.append(f'[objaverse] rig flag for unknown rig {rig!r} ignored')
continue
cat, _, reason = comment.strip().partition(':')
- out[rig] = (cat.strip() or 'flagged', reason.strip())
+ cat = cat.strip() or 'flagged'
+ if rig in out:
+ prev = out[rig][0]
+ log.append(f'[objaverse] rig {rig!r} is listed twice in {os.path.basename(path)}: '
+ f'{prev!r} then {cat!r}; ' +
+ (f'{cat!r} wins and the {prev!r} entry is dropped'
+ if prev != cat else 'the duplicate line is redundant'))
+ out[rig] = (cat, reason.strip())
return out
@@ -444,6 +459,9 @@ def write_filtered_clips(ds, root, patch_dir, frames, dry_run, log):
continue
if frames and clip not in frames:
log.append(f'[{ds}] filtered clip {clip!r} is not in clip_frames.json; kept anyway')
+ if clip in clips:
+ log.append(f'[{ds}] filtered clip {clip!r} is listed twice in '
+ f'{os.path.basename(src)}; the last reason wins')
clips[clip] = comment.strip()
lines = ['# Individual export clips skipped by stage 4 (extract_features.py reads this file',
'# automatically; --filtered_clips auto). The clip-level twin of filtered_objects.txt.',
@@ -460,6 +478,172 @@ def write_filtered_clips(ds, root, patch_dir, frames, dry_run, log):
return clips
+def write_clip_trims(ds, root, patch_dir, frames, dry_run, log):
+ """export/<ds>/clip_trims.txt โ€” hand-reviewed head trims that stage 4 applies.
+
+ patches/<ds>_clip_trims.txt holds ``<clip stem> <N> # <reason>`` lines: the
+ first N frames of that clip are a bind pose or a foreign pose that snaps into
+ the motion. N counts frames of the export NPZ itself (its own fps), so any
+ reader applies ``anim[N:]``; stage 4 does so on load, before downsampling and
+ its static trim. The NPZs are not rewritten: a stage-1 rerun would undo that.
+ """
+ src = os.path.join(patch_dir, f'{ds}_clip_trims.txt')
+ path = os.path.join(root, 'clip_trims.txt')
+ if not os.path.isfile(src):
+ return {}
+ trims = OrderedDict()
+ with open(src) as f:
+ for lineno, line in enumerate(f, 1):
+ body, _, comment = line.partition('#')
+ parts = body.split()
+ if not parts:
+ continue
+ if len(parts) != 2 or not parts[1].isdigit() or int(parts[1]) < 1:
+ raise ValueError(f'{src}:{lineno}: expected "<clip> <N>" with N >= 1, got {line.strip()!r}')
+ clip, n = parts[0], int(parts[1])
+ if frames and clip not in frames:
+ log.append(f'[{ds}] clip trim {clip!r} is not in clip_frames.json; kept anyway')
+ elif frames and n >= frames[clip]:
+ raise ValueError(f'{src}:{lineno}: trimming {n} frames removes all {frames[clip]} of {clip}')
+ if clip in trims:
+ log.append(f'[{ds}] clip trim {clip!r} is listed twice in {os.path.basename(src)}; the last wins')
+ trims[clip] = (n, comment.strip())
+ lines = ['# Hand-reviewed head trims applied by stage 4 (extract_features.py reads this file',
+ '# automatically; --clip_trims auto): "<clip> <N>" drops the first N frames of the',
+ "# clip's export NPZ (motions/<clip>.npz, its own fps): anim_local_rot/pos[N:].",
+ f'# Generated by data_process/tools/patch_annotations.py from '
+ f'patches/{ds}_clip_trims.txt โ€” {len(trims)} clips.']
+ for clip, (n, why) in trims.items():
+ lines.append(f'{clip} {n} # {why}' if why else f'{clip} {n}')
+ if not dry_run:
+ with open(path, 'w') as f:
+ f.write('\n'.join(lines) + '\n')
+ log.append(f'[{ds}] clip_trims.txt: {len(trims)} clips, {sum(n for n, _ in trims.values())} frames dropped')
+ return trims
+
+
+def write_activity_keep(ds, root, patch_dir, frames, dry_run, log):
+ """export/<ds>/activity_keep.txt โ€” clips exempt from stage 4's low-activity filter.
+
+ patches/<ds>_activity_keep.txt holds ``<clip stem> # <reason>`` lines: a real
+ motion confined to a few joints (a head turn, a clap, a wave) that the joint-activity
+ gate reads as a held pose. Stage 4 still applies every other filter to them.
+ """
+ src = os.path.join(patch_dir, f'{ds}_activity_keep.txt')
+ path = os.path.join(root, 'activity_keep.txt')
+ if not os.path.isfile(src):
+ return {}
+ keep = OrderedDict()
+ with open(src) as f:
+ for lineno, line in enumerate(f, 1):
+ body, _, comment = line.partition('#')
+ parts = body.split()
+ if not parts:
+ continue
+ if len(parts) != 1:
+ raise ValueError(f'{src}:{lineno}: expected "<clip>", got {line.strip()!r}')
+ clip = parts[0]
+ if frames and clip not in frames:
+ log.append(f'[{ds}] activity keep {clip!r} is not in clip_frames.json; kept anyway')
+ if clip in keep:
+ log.append(f'[{ds}] activity keep {clip!r} is listed twice in {os.path.basename(src)}')
+ keep[clip] = comment.strip()
+ lines = ['# Hand-reviewed exemptions from the low-activity filter, applied by stage 4',
+ '# (extract_features.py reads this file automatically; --activity_keep auto).',
+ f'# Generated by data_process/tools/patch_annotations.py from '
+ f'patches/{ds}_activity_keep.txt โ€” {len(keep)} clips.']
+ for clip, why in keep.items():
+ lines.append(f'{clip} # {why}' if why else clip)
+ if not dry_run:
+ with open(path, 'w') as f:
+ f.write('\n'.join(lines) + '\n')
+ log.append(f'[{ds}] activity_keep.txt: {len(keep)} clips')
+ return keep
+
+
+ROOT_OFFSET_KEY = 'root_offset_applied' # the quat currently baked into this NPZ (w, x, y, z)
+
+
+def _save_npz_atomic(path, data):
+ fd, tmp = tempfile.mkstemp(prefix=os.path.basename(path) + '.', suffix='.tmp',
+ dir=os.path.dirname(path) or '.')
+ try:
+ with os.fdopen(fd, 'wb') as f:
+ np.savez(f, **data)
+ os.replace(tmp, path)
+ except BaseException:
+ if os.path.exists(tmp):
+ os.unlink(tmp)
+ raise
+
+
+def apply_root_offsets(ds, root, patch_dir, dry_run, log):
+ """Remove a constant whole-body orientation error from export clips, in place.
+
+ Some vendor FBX files key the root joint with a fixed extra rotation for the whole
+ clip (Truebones: nine Dog clips rolled 44.6 deg about the pelvis's local X axis,
+ Dog-Stare 134.6 deg, Flamingo-One_Leg_Bent 27.5 deg), so the animal floats tilted
+ while every child joint animates correctly. patches/<ds>_root_offsets.json maps
+ ``clip -> {"quat": [w, x, y, z], "reason": ...}``; the fix post-multiplies the
+ root's local rotation on every frame, ``q0[t] <- q0[t] * quat``, which turns the
+ whole body rigidly about the root in body coordinates (so it stays right when the
+ clip turns). The quats were estimated by hand review and checked against Dog2, which
+ carries the same animations upright.
+
+ Unlike clip_trims this rewrites the NPZ, so every reader (stage 4, previews, mesh
+ driving, audits) sees the fixed motion. To stay reproducible it is idempotent: the
+ applied quat is stored in the NPZ under ``root_offset_applied``. A matching entry is
+ skipped; a changed entry undoes the old quat first; an NPZ that carries the key but
+ is no longer listed is restored. A stage-1 rerun writes fresh NPZs without the key,
+ and the next run of this script fixes them again. Returns the clips whose NPZ
+ changed (their cached root motion is stale).
+ """
+ want = load_overrides(os.path.join(patch_dir, f'{ds}_root_offsets.json'))
+ mdir = os.path.join(root, 'motions')
+ changed, n_ok = [], 0
+ targets = set(want)
+ for f in glob.glob(os.path.join(mdir, '*.npz')): # previously fixed clips no longer listed
+ clip = os.path.basename(f)[:-4]
+ if clip not in targets:
+ with np.load(f, allow_pickle=True) as d:
+ if ROOT_OFFSET_KEY in d.files:
+ targets.add(clip)
+ for clip in sorted(targets):
+ path = os.path.join(mdir, clip + '.npz')
+ if not os.path.isfile(path):
+ log.append(f'[{ds}] root offset for {clip!r}: no export NPZ; ignored')
+ continue
+ with np.load(path, allow_pickle=True) as d:
+ data = {k: d[k] for k in d.files}
+ have = data.pop(ROOT_OFFSET_KEY, None)
+ new = want.get(clip)
+ q_new = None
+ if new is not None:
+ q_new = np.asarray(new['quat'], dtype=np.float64)
+ q_new = q_new / np.linalg.norm(q_new)
+ if have is not None and q_new is not None and np.allclose(have, q_new, atol=1e-6):
+ n_ok += 1
+ continue
+ rot = data['anim_local_rot'].astype(np.float64)
+ q0 = rot[:, 0]
+ if have is not None: # undo what is baked in
+ q0 = qmul(q0, np.asarray(have, dtype=np.float64) * np.array([1, -1, -1, -1]))
+ if q_new is not None:
+ q0 = qmul(q0, np.broadcast_to(q_new, q0.shape))
+ data[ROOT_OFFSET_KEY] = q_new.astype(np.float32)
+ rot[:, 0] = q0 / np.linalg.norm(q0, axis=-1, keepdims=True)
+ data['anim_local_rot'] = rot.astype(data['anim_local_rot'].dtype)
+ what = 'restored (entry removed)' if q_new is None else \
+ f'{np.degrees(2 * np.arccos(min(1.0, abs(q_new[0])))):.1f} deg'
+ log.append(f'[{ds}] root offset {clip}: {what}' + (' (dry run)' if dry_run else ''))
+ if not dry_run:
+ _save_npz_atomic(path, data)
+ changed.append(clip)
+ if want or changed:
+ log.append(f'[{ds}] root_offsets: {len(want)} listed, {len(changed)} rewritten, {n_ok} already applied')
+ return changed
+
+
def write_rig_flags(root, face, frames, manual, dry_run, log):
"""rig_flags.json (every flagged rig with its category) + filtered_objects.txt (FILTER_CATEGORIES only)."""
flags = OrderedDict()
@@ -482,8 +666,10 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):
out = OrderedDict([
('_comment', 'Per-rig flags. Automatic facing checks (empty_pair: no bilateral joint pair; '
'bone_pair: the pair consists of unnamed Bone joints; body_axis_unnamed) plus the '
- 'hand-reviewed categories (tpose_wrong: the exported rest pose is unusable; facing_wrong: '
- 'the pair does not give the real front). Only ' + ', '.join(FILTER_CATEGORIES) +
+ 'hand-reviewed categories (tpose_wrong: the exported rest pose is unusable; '
+ 'not_in_legacy_raw: the raw GLB is absent from the legacy raw set and is held out of '
+ 'training; facing_wrong: the pair does not give the real front). Only '
+ + ', '.join(FILTER_CATEGORIES) +
' is written to filtered_objects.txt, which stage 4 skips; every other flag is '
'informational and those rigs are kept. verified_ok lists rigs whose automatic flag was '
'cleared after review.'),
@@ -494,9 +680,9 @@ def write_rig_flags(root, face, frames, manual, dry_run, log):
('verified_ok', verified), # automatic flags cleared by hand review
])
lines = ['# Object types skipped by stage 4 (extract_features.py reads this file automatically):',
- '# rigs flagged ' + ' / '.join(FILTER_CATEGORIES) + ' in patches/objaverse_rig_flags.txt โ€” the exported',
- '# rest pose lies flat / is rotated / upside-down, so the stage-4 T-pose is unusable.',
- '# Other flags (empty_pair, bone_pair, facing_wrong, object_no_front, ...) are recorded in',
+ '# rigs flagged ' + ' / '.join(FILTER_CATEGORIES) + ' in patches/objaverse_rig_flags.txt.']
+ lines += [f'# {c}: {FILTER_CATEGORY_NOTES[c]}' for c in FILTER_CATEGORIES if c in FILTER_CATEGORY_NOTES]
+ lines += ['# Other flags (empty_pair, bone_pair, facing_wrong, object_no_front, ...) are recorded in',
'# rig_flags.json only and are NOT filtered.',
f'# Generated by data_process/tools/patch_annotations.py โ€” {len(bad)} rigs, {len(clips)} clips.',
'# Delete this file to keep them (they then get the rest-pose facing as exported).']
@@ -542,7 +728,7 @@ def root_motion(root, ds, clips, cache_path, face, log):
"""Per clip: net XZ root displacement in body heights and the travel
direction relative to the facing pair at frame 0."""
cache = load_json(cache_path, OrderedDict()) if cache_path else OrderedDict()
- key = lambda c: f'{ds}/{c}'
+ key = lambda c: f'{ds}/{c}@{CAPTION_FRAMES}' # window-scoped: pre-2026-09-21 full-clip rows are ignored
todo = [c for c in clips if key(c) not in cache]
if todo:
log.append(f'[{ds}] computing root motion for {len(todo)} clips')
@@ -553,7 +739,14 @@ def root_motion(root, ds, clips, cache_path, face, log):
d = np.load(f, allow_pickle=True)
g0 = rest_global(d)
height = float(np.ptp(g0, axis=0).max()) or 1.0
- r = d['anim_local_pos'][:, 0, :]
+ # Only the first MAX_RENDER_FRAMES frames count. That is the window the
+ # captioner was shown, the window the released video covers, and the only
+ # window stage 4 keeps - so it is the window the caption describes.
+ # Measuring endpoint-to-endpoint over the WHOLE clip silently breaks every
+ # looping animation: the root returns to its start, net comes out 0.000,
+ # and a clip that visibly crosses six body lengths gets pinned at
+ # "in place" (and the reverse for clips that only travel after frame 200).
+ r = d['anim_local_pos'][:CAPTION_FRAMES, 0, :]
disp = r[-1] - r[0]
net = float(np.hypot(disp[0], disp[2]) / height)
direction = ''
@@ -695,6 +888,21 @@ def run_dataset(ds, args, log):
stats['manual-rig-flag'] += len(manual)
stats['manual-filtered-clip'] += len(
write_filtered_clips(ds, root, pd, frames, args.dry_run, log))
+ stats['manual-clip-trim'] += len(
+ write_clip_trims(ds, root, pd, frames, args.dry_run, log))
+ stats['manual-activity-keep'] += len(
+ write_activity_keep(ds, root, pd, frames, args.dry_run, log))
+ rewritten = apply_root_offsets(ds, root, pd, args.dry_run, log)
+ stats['root-offset-rewritten'] += len(rewritten)
+ if rewritten and not args.dry_run:
+ # the cached travel direction was measured against the old (tilted) facing
+ cache_path = os.path.join(pd, '.root_motion_cache.json')
+ cache = load_json(cache_path, OrderedDict())
+ stale = [k for k in cache if k.split('@')[0] in {f'{ds}/{c}' for c in rewritten}]
+ for k in stale:
+ del cache[k]
+ if stale:
+ save_json(cache_path, cache, False)
if groups is not None:
patch_categories(ds, groups, load_overrides(os.path.join(pd, f'{ds}_categories.json')), names, stats, log)
if caps is not None:
diff --git a/data_process/tools/vis_caption.py b/data_process/tools/vis_caption.py
new file mode 100644
index 0000000..689fe43
--- /dev/null
+++ b/data_process/tools/vis_caption.py
@@ -0,0 +1,262 @@
+#!/usr/bin/env python3
+"""Render an export-stage skeleton as video with its caption burned in.
+
+QA tool for the caption layer: it puts the text and the motion it is supposed
+to describe in one frame, so a wrong direction, a missing second phase or a
+body part that never moves is visible without cross-referencing anything.
+
+Input is the export layer (``dataset/export/<ds>/motions/<clip>.npz`` plus
+``motion_captions.json``), not the feature layer, so what you see is the same
+data the renderer showed the captioner. Frames are truncated to
+``--max_frames`` (default 1000), which is long enough that only a handful of
+clips are cut at all. Note that the captioner itself only ever saw the first
+``blender_render.MAX_RENDER_FRAMES`` (200) frames, so a mismatch beyond frame
+200 is a gap in the caption's coverage, not necessarily a wrong caption; pass
+``--max_frames 200`` to review exactly what it saw.
+
+The skeleton is drawn on a world-anchored checkerboard floor with a contact
+shadow and a root trail, so translation reads off the tiles scrolling
+underfoot โ€” which is what makes "walks forward" vs "walks in place" legible.
+
+The look follows the stage-4 clip previews in ``dataset/features/<ds>/videos``
+(``motion_features.save_clip`` with ``vis_ground=True``): same renderer, same
+``PREVIEW_FIGSIZE`` / ``PREVIEW_DPI``, and the caption alone as the title. Two
+differences: the source is the export clip the captioner actually saw, at its
+full joint count and in the asset's own frame, rather than the canonicalized
+stage-4 clip; and the joints keep the renderer's plain red-root / blue-joint
+scheme instead of stage 4's spectral palette, which recolours every skeleton
+differently and makes body parts harder to follow across clips.
+
+Usage (from the repo root):
+ # one clip
+ python -m data_process.tools.vis_caption truebones --clips Lion-Die
+
+ # a random sample, each as its own MP4 plus one tiled grid video
+ python -m data_process.tools.vis_caption mixamo --random 6 --grid
+
+ # every captioned clip of a dataset, rendered in parallel and resumable
+ python -m data_process.tools.vis_caption mixamo --all --workers 16
+
+ # a contact sheet PNG as well (for pasting into a review)
+ python -m data_process.tools.vis_caption truebones --random 4 --grid --sheet
+
+Outputs land in ``outputs/caption_vis/<dataset>/``.
+"""
+
+import argparse
+import json
+import os
+import random
+import sys
+
+import imageio
+import numpy as np
+from concurrent.futures import ProcessPoolExecutor
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))
+
+from data_process.utils.kinematics import clip_global # noqa: E402
+from data_process.utils.motion_features import ( # noqa: E402
+ PREVIEW_DPI, PREVIEW_FIGSIZE)
+from data_process.utils.plotting import render_skeleton_motion_ground # noqa: E402
+
+DATASETS = ('truebones', 'mixamo', 'objaverse')
+MAX_RENDER_FRAMES = 200 # keep in sync with blender_render.MAX_RENDER_FRAMES
+DEFAULT_MAX_FRAMES = 1000 # render cost is linear in frames; 1000 cuts almost nothing
+
+
+def export_dir(root, ds):
+ return os.path.join(root, 'dataset', 'export', ds)
+
+
+def load_captions(root, ds):
+ path = os.path.join(export_dir(root, ds), 'motion_captions.json')
+ if not os.path.isfile(path):
+ raise SystemExit('no captions for %s at %s' % (ds, path))
+ with open(path) as f:
+ return json.load(f)
+
+
+def load_clip(root, ds, clip, max_frames):
+ """(parents, positions (T, J, 3)) for one export clip, Y-up as exported."""
+ path = os.path.join(export_dir(root, ds), 'motions', clip + '.npz')
+ if not os.path.isfile(path):
+ raise SystemExit('no such clip: %s' % path)
+ d = np.load(path, allow_pickle=True)
+ return np.asarray(d['parents']), clip_global(d, max_frames=max_frames)
+
+
+def tile(frame_stacks, labels=None):
+ """Tile per-clip frame stacks into one video, holding each clip's last frame."""
+ n = len(frame_stacks)
+ cols = int(np.ceil(np.sqrt(n)))
+ rows = int(np.ceil(n / cols))
+ T = max(len(f) for f in frame_stacks)
+ H, W = frame_stacks[0].shape[1:3]
+ out = np.full((T, rows * H, cols * W, 3), 255, dtype=np.uint8)
+ for i, fr in enumerate(frame_stacks):
+ held = np.concatenate([fr, np.repeat(fr[-1:], T - len(fr), axis=0)]) if len(fr) < T else fr
+ r, c = divmod(i, cols)
+ out[:, r * H:(r + 1) * H, c * W:(c + 1) * W] = held[:T, :H, :W]
+ return out
+
+
+def _title(clip, caption, with_clip_name):
+ return '%s\n%s' % (clip, caption) if with_clip_name else caption
+
+
+def _render_one(job):
+ """Render one clip to <out_dir>/<clip>.mp4. Runs in a worker process."""
+ (root, ds, clip, caption, out_dir, max_frames, elev, azim,
+ trail, figsize, dpi, fps, with_clip_name) = job
+ try:
+ parents, pos = load_clip(root, ds, clip, max_frames)
+ frames = render_skeleton_motion_ground(
+ parents, pos, elev=elev, azim=azim, rotate_root=True,
+ title=_title(clip, caption, with_clip_name), trail=trail,
+ figsize=tuple(figsize), dpi=dpi)
+ # Write to a sibling temp file and rename: a run that is killed
+ # mid-encode must not leave a half-written MP4 behind, because
+ # --skip_existing would then treat it as done on the next pass.
+ final = os.path.join(out_dir, clip + '.mp4')
+ tmp = os.path.join(out_dir, '.%s.tmp.mp4' % clip) # keep .mp4: imageio picks the plugin by extension
+ imageio.mimwrite(tmp, frames, fps=fps)
+ os.replace(tmp, final)
+ return clip, len(pos), None
+ except Exception as e: # one bad clip must not stop the batch
+ return clip, 0, '%s: %s' % (type(e).__name__, e)
+
+
+def main():
+ root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
+ ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])
+ ap.add_argument('dataset', choices=DATASETS)
+ ap.add_argument('--clips', nargs='+', default=None, help='clip names (export stems)')
+ ap.add_argument('--random', type=int, default=0, help='sample N random clips instead')
+ ap.add_argument('--clips_file', default=None,
+ help='file with one clip name per line (# comments and blanks ignored); '
+ 'for a subset too long for a command line, e.g. only the clips that '
+ 'survive filtered_clips.txt / filtered_objects.txt')
+ ap.add_argument('--all', action='store_true',
+ help='every captioned clip of the dataset; implies --skip_existing '
+ 'so an interrupted run resumes. Not compatible with --grid / '
+ '--sheet, which hold every frame of every clip in memory.')
+ ap.add_argument('--workers', type=int, default=1,
+ help='render this many clips in parallel (one process each)')
+ ap.add_argument('--skip_existing', action='store_true',
+ help='leave clips that already have an MP4 in the output dir alone')
+ ap.add_argument('--seed', type=int, default=0)
+ ap.add_argument('--output_dir', default=None,
+ help='default outputs/caption_vis/<dataset>')
+ ap.add_argument('--fps', type=int, default=30)
+ ap.add_argument('--max_frames', type=int, default=DEFAULT_MAX_FRAMES,
+ help='0 = whole clip; default %d. Pass %d to see exactly the '
+ 'range the captioner saw' % (DEFAULT_MAX_FRAMES, MAX_RENDER_FRAMES))
+ ap.add_argument('--grid', action='store_true', help='also write one tiled MP4')
+ ap.add_argument('--sheet', action='store_true', help='also write a contact-sheet PNG')
+ ap.add_argument('--sheet_cols', type=int, default=6, help='frames per row in the sheet')
+ # the stage-4 previews take render_skeleton_motion_ground's own defaults
+ # (save_clip passes no camera), so match them here rather than inventing a view
+ ap.add_argument('--elev', type=float, default=20.0)
+ ap.add_argument('--azim', type=float, default=-60.0)
+ ap.add_argument('--figsize', type=float, nargs=2, default=PREVIEW_FIGSIZE,
+ help='default matches the stage-4 previews')
+ ap.add_argument('--dpi', type=int, default=PREVIEW_DPI,
+ help='default matches the stage-4 previews')
+ ap.add_argument('--with_clip_name', action='store_true',
+ help='put the clip name above the caption; off by default so the '
+ 'frame carries the caption alone, like the stage-4 previews')
+ ap.add_argument('--no_trail', action='store_true')
+ args = ap.parse_args()
+
+ caps = load_captions(root, args.dataset)
+ if args.all:
+ clips = sorted(caps)
+ elif args.random:
+ random.seed(args.seed)
+ clips = random.sample(sorted(caps), min(args.random, len(caps)))
+ elif args.clips_file:
+ clips = [l.split('#')[0].strip() for l in open(args.clips_file)]
+ clips = [c for c in clips if c]
+ elif args.clips:
+ clips = args.clips
+ else:
+ raise SystemExit('give --clips, --clips_file, --random N or --all')
+ if args.all and (args.grid or args.sheet):
+ raise SystemExit('--grid / --sheet keep every frame in memory; use --random N for those')
+
+ out_dir = args.output_dir or os.path.join(root, 'outputs', 'caption_vis', args.dataset)
+ os.makedirs(out_dir, exist_ok=True)
+
+ if args.skip_existing or args.all:
+ before = len(clips)
+ clips = [c for c in clips if not os.path.isfile(os.path.join(out_dir, c + '.mp4'))]
+ if before != len(clips):
+ print('skipping %d clip(s) already rendered' % (before - len(clips)))
+
+ if args.workers > 1:
+ jobs = [(root, args.dataset, c, caps[c], out_dir, args.max_frames, args.elev,
+ args.azim, not args.no_trail, args.figsize, args.dpi, args.fps,
+ args.with_clip_name)
+ for c in clips if c in caps]
+ print('rendering %d clip(s) with %d workers -> %s' % (len(jobs), args.workers, out_dir))
+ done = failed = 0
+ with ProcessPoolExecutor(args.workers) as ex:
+ for clip, n, err in ex.map(_render_one, jobs, chunksize=4):
+ if err:
+ failed += 1
+ print(' FAILED %-38s %s' % (clip, err))
+ else:
+ done += 1
+ if done % 100 == 0 or done == len(jobs):
+ print(' %d/%d rendered' % (done, len(jobs)), flush=True)
+ print('wrote %d MP4(s) to %s%s' % (done, out_dir,
+ ', %d failed' % failed if failed else ''))
+ return
+
+ stacks, kept = [], []
+ for clip in clips:
+ if clip not in caps:
+ print(' skip %s (no caption)' % clip)
+ continue
+ parents, pos = load_clip(root, args.dataset, clip, args.max_frames)
+ caption = caps[clip]
+ print(' %-38s %3d frames | %s' % (clip, len(pos), caption))
+ frames = render_skeleton_motion_ground(
+ parents, pos, elev=args.elev, azim=args.azim,
+ # The export layer is Y-up; _maybe_rotate_y_up maps input Y onto
+ # matplotlib's vertical Z axis, so Y-up data wants rotate_root=True.
+ rotate_root=True,
+ title=_title(clip, caption, args.with_clip_name),
+ trail=not args.no_trail, figsize=tuple(args.figsize), dpi=args.dpi)
+ final = os.path.join(out_dir, clip + '.mp4')
+ tmp = os.path.join(out_dir, '.%s.tmp.mp4' % clip)
+ imageio.mimwrite(tmp, frames, fps=args.fps)
+ os.replace(tmp, final)
+ stacks.append(frames)
+ kept.append(clip)
+
+ if not kept:
+ raise SystemExit('nothing rendered')
+ print('wrote %d MP4(s) to %s' % (len(kept), out_dir))
+
+ if args.grid and len(kept) > 1:
+ grid = tile(stacks)
+ p = os.path.join(out_dir, 'grid_%s.mp4' % '_'.join(kept[:3])[:60])
+ imageio.mimwrite(p, grid, fps=args.fps)
+ print('wrote grid %s (%d frames, %dx%d)' % (p, len(grid), grid.shape[2], grid.shape[1]))
+
+ if args.sheet:
+ cols = args.sheet_cols
+ rowsimg = []
+ for clip, fr in zip(kept, stacks):
+ idx = [round(i * (len(fr) - 1) / max(cols - 1, 1)) for i in range(cols)]
+ rowsimg.append(np.concatenate([fr[i] for i in idx], axis=1))
+ sheet = np.concatenate(rowsimg, axis=0)
+ p = os.path.join(out_dir, 'sheet_%s.png' % '_'.join(kept[:3])[:60])
+ imageio.imwrite(p, sheet)
+ print('wrote sheet %s (%dx%d)' % (p, sheet.shape[1], sheet.shape[0]))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/data_process/tools/vis_tpose_facing.py b/data_process/tools/vis_tpose_facing.py
index 897f587..1b4036b 100644
--- a/data_process/tools/vis_tpose_facing.py
+++ b/data_process/tools/vis_tpose_facing.py
@@ -100,10 +100,18 @@ def index_motion_npzs(motions_dir):
across un-prefixed clips, so its ``mixamo`` key maps to the first NPZ.
"""
index = {}
- for p in sorted(Path(motions_dir).glob('*.npz')):
+ npzs = sorted(Path(motions_dir).glob('*.npz'))
+ for p in npzs:
key = p.stem.split('-', 1)[0] if '-' in p.stem else p.stem
index.setdefault(key, p)
- index.setdefault('mixamo', p)
+ # Mixamo clips carry no object prefix, so the per-prefix keys above are
+ # action names; the rig itself is addressed as 'mixamo'. Only add that
+ # alias when un-prefixed clips exist, otherwise every prefixed export dir
+ # (truebones, objaverse) would gain a spurious 'mixamo' rig pointing at
+ # its first clip.
+ unprefixed = [p for p in npzs if '-' not in p.stem]
+ if unprefixed:
+ index.setdefault('mixamo', unprefixed[0])
return index
diff --git a/data_process/utils/kinematics.py b/data_process/utils/kinematics.py
new file mode 100644
index 0000000..951fa16
--- /dev/null
+++ b/data_process/utils/kinematics.py
@@ -0,0 +1,75 @@
+"""Forward kinematics for export-stage NPZs (numpy only, no bpy, no Motion).
+
+Quaternions are ``w, x, y, z`` โ€” the order the exporter writes (Blender's
+``Matrix.to_quaternion``). Export space is Blender rotated -90 deg about X
+(a proper rotation, so handedness is preserved): **Y-up**, ground at the
+minimum Y.
+
+Batched over any number of leading axes, so the same call serves one rest
+pose ``(J, .)`` and a whole clip ``(T, J, .)``.
+"""
+
+from typing import Sequence
+
+import numpy as np
+
+
+def qmul(a, b):
+ """Hamilton product of two ``w, x, y, z`` quaternion arrays."""
+ w1, x1, y1, z1 = a[..., 0], a[..., 1], a[..., 2], a[..., 3]
+ w2, x2, y2, z2 = b[..., 0], b[..., 1], b[..., 2], b[..., 3]
+ return np.stack([w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2,
+ w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,
+ w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2,
+ w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2], -1)
+
+
+def qrot(q, v):
+ """Rotate vectors *v* ``(..., 3)`` by quaternions *q* ``(..., 4)``."""
+ qv = np.concatenate([np.zeros(v.shape[:-1] + (1,)), v], -1)
+ return qmul(qmul(q, qv), q * np.array([1, -1, -1, -1]))[..., 1:]
+
+
+def fk(local_pos, local_rot, parents: Sequence[int]) -> np.ndarray:
+ """Global joint positions from local positions / rotations and a parent list.
+
+ ``local_pos`` and ``local_rot`` are ``(..., J, 3)`` and ``(..., J, 4)``;
+ the result is ``(..., J, 3)``. ``parents[i] < 0`` marks the root.
+ """
+ local_pos = np.asarray(local_pos, dtype=np.float64)
+ local_rot = np.asarray(local_rot, dtype=np.float64)
+ parents = np.asarray(parents)
+ lead = local_pos.shape[:-2]
+ g = np.zeros(lead + local_pos.shape[-2:])
+ gr = np.zeros(lead + local_rot.shape[-2:])
+ for i in range(len(parents)):
+ p = int(parents[i])
+ if p < 0:
+ g[..., i, :] = local_pos[..., i, :]
+ gr[..., i, :] = local_rot[..., i, :]
+ else:
+ g[..., i, :] = g[..., p, :] + qrot(gr[..., p, :], local_pos[..., i, :])
+ gr[..., i, :] = qmul(gr[..., p, :], local_rot[..., i, :])
+ return g
+
+
+def rest_global(d) -> np.ndarray:
+ """Global rest-pose joint positions ``(J, 3)`` from a loaded export NPZ."""
+ return fk(d['rest_local_pos'], d['rest_local_rot'], d['parents'])
+
+
+def clip_global(d, max_frames: int = 0) -> np.ndarray:
+ """Global joint positions ``(T, J, 3)`` for a whole export clip.
+
+ ``max_frames`` > 0 truncates, which is how callers match
+ ``blender_render.MAX_RENDER_FRAMES`` โ€” the range the captioner actually saw.
+ """
+ pos, rot = d['anim_local_pos'], d['anim_local_rot']
+ if max_frames and len(pos) > max_frames:
+ pos, rot = pos[:max_frames], rot[:max_frames]
+ return fk(pos, rot, d['parents'])
+
+
+def frame_global(d, t: int = 0) -> np.ndarray:
+ """Global joint positions ``(J, 3)`` at a single frame of an export clip."""
+ return fk(d['anim_local_pos'][t], d['anim_local_rot'][t], d['parents'])
diff --git a/data_process/utils/motion_features.py b/data_process/utils/motion_features.py
index e35e1c5..daf42ba 100644
--- a/data_process/utils/motion_features.py
+++ b/data_process/utils/motion_features.py
@@ -193,9 +193,14 @@ def _names_permutation(ref_names, ref_parents, names, parents):
def _load_npz_anim(object_npz_path, raw_parents, raw_njoints,
- corps_idxs, bfs_order, fps, raw_names=None):
+ corps_idxs, bfs_order, fps, raw_names=None, head_trim=0):
"""Load a motion NPZ, apply joint subset/reorder, and downsample.
+ ``head_trim`` drops the first N frames of the export NPZ itself (its own
+ frame rate, before downsampling and the static trim), so a
+ ``clip_trims.txt`` entry means the same frames to every reader of the
+ export, not only to this stage.
+
Returns:
Tuple of (motion_anim, motion_fps, None) on success, or
(None, None, reason_string) when the motion should be skipped.
@@ -211,6 +216,14 @@ def _load_npz_anim(object_npz_path, raw_parents, raw_njoints,
anim_local_rot = anim_data['anim_local_rot']
anim_local_pos = anim_data['anim_local_pos']
rest_local_pos = anim_data['rest_local_pos']
+ if head_trim:
+ if head_trim >= nframes:
+ reason = f'head trim of {head_trim} frames removes all {nframes}'
+ logger.warning(f'Skipping clip ({reason}): {object_npz_path}')
+ return None, None, reason
+ anim_local_rot = anim_local_rot[head_trim:]
+ anim_local_pos = anim_local_pos[head_trim:]
+ nframes -= head_trim
# corps_idxs/bfs_order are positional indices into the T-pose clip's
# joint order; a source file enumerating the same skeleton in a
@@ -293,7 +306,7 @@ def _load_tpos_npz(tpos_data, corps_idxs=None, corps_names=None):
def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,
corps_idxs, bfs_order, fps, scale_factor,
ground_height, face_joint_idxs,
- body_axis=False, raw_names=None):
+ body_axis=False, raw_names=None, head_trim=0):
"""Load a motion NPZ and canonicalize facing / XZ / ground + scale.
``scale_factor`` and ``ground_height`` are reused from T-pose
@@ -309,7 +322,7 @@ def load_and_canonicalize_motion(object_npz_path, raw_parents, raw_njoints,
"""
motion_anim, motion_fps, skip_reason = _load_npz_anim(
object_npz_path, raw_parents, raw_njoints, corps_idxs, bfs_order, fps,
- raw_names=raw_names)
+ raw_names=raw_names, head_trim=head_trim)
if motion_anim is None:
return None, skip_reason
@@ -514,12 +527,16 @@ def _discontinuity_above_threshold(global_positions, step_threshold, ratio_thres
body lengths AND that step is at least *ratio_threshold* times the clip's
median step โ€” the signature of concatenated actions, not of fast motion,
which raises the median too. Non-finite input counts as discontinuous.
+ ``step_threshold <= 0`` disables the filter (non-finite input is still
+ rejected); without this guard ``step_max >= 0`` always holds and the
+ ratio test alone would decide.
"""
if not np.isfinite(global_positions).all():
return True, {'step_med': 0.0, 'step_max': float('inf'),
'jump_ratio': float('inf'), 'jump_frames': len(global_positions)}
metrics = _discontinuity_metrics(global_positions)
- is_bad = (metrics['step_max'] >= step_threshold
+ is_bad = (step_threshold > 0
+ and metrics['step_max'] >= step_threshold
and metrics['jump_ratio'] >= ratio_threshold)
return is_bad, metrics
@@ -636,13 +653,17 @@ def _process_motion_file(object_npz_path, ctx):
"""
base_name = str(Path(object_npz_path).stem)
min_frames = ctx['min_frames']
+ head_trim = ctx.get('head_trims', {}).get(base_name, 0)
+ if head_trim:
+ logger.info(f'[{base_name}] dropping the first {head_trim} export frames (clip_trims.txt)')
# --- Load + motion-level canonicalize ----------------------------------
result, skip_reason = load_and_canonicalize_motion(
object_npz_path, ctx['raw_parents'], ctx['raw_njoints'],
ctx['corps_idxs'], ctx['bfs_order'], ctx['fps'], ctx['scale_factor'],
ctx['ground_height'], ctx['face_joint_idxs'],
- body_axis=ctx['body_axis'], raw_names=ctx.get('raw_names'))
+ body_axis=ctx['body_axis'], raw_names=ctx.get('raw_names'),
+ head_trim=head_trim)
if result is None:
return {'n_saved': 0, 'frames': 0, 'clip_captions': {},
'filtered': [{'name': base_name, 'reason': skip_reason}]}
@@ -653,8 +674,9 @@ def _process_motion_file(object_npz_path, ctx):
motion_anim, _, _ = trim_static_ends(
motion_anim, static_threshold=ctx['static_threshold'])
if len(motion_anim) < min_frames:
- reason = (f'too few frames after static trim: {len(motion_anim)} '
- f'< {min_frames} (orig {orig_nframes})')
+ reason = (f'too few frames after static trim'
+ + (f' (after a {head_trim}-frame head trim)' if head_trim else '')
+ + f': {len(motion_anim)} < {min_frames} (orig {orig_nframes})')
return {'n_saved': 0, 'frames': 0, 'clip_captions': {},
'filtered': [{'name': base_name, 'reason': reason}]}
@@ -676,6 +698,10 @@ def _process_motion_file(object_npz_path, ctx):
is_low, clip_metrics = _activity_below_threshold(
clip_global_positions, ctx['activity_threshold'])
clip_name = f'{base_name}-{clip_idx:03d}'
+ if is_low and base_name in ctx['activity_keep']:
+ logger.info(f'[{clip_name}] low activity ({_format_activity(clip_metrics)}) '
+ f'kept: listed in activity_keep.txt')
+ is_low = False
if is_low:
filtered.append({
'name': clip_name,
@@ -726,7 +752,7 @@ def process_object(object_type, object_npzs, save_dir,
jump_step_threshold=0.2, jump_ratio_threshold=8.0,
min_joints=None, max_joints=None,
use_tpos_ground_height=False, save_vis=True,
- vis_ground=False):
+ vis_ground=False, head_trims=None, activity_keep=None):
"""Process all motion clips for one object type.
Emits per-clip NPZs holding raw canonicalized arrays โ€” global
@@ -743,6 +769,16 @@ def process_object(object_type, object_npzs, save_dir,
default). Whichever is used is recorded in the cond as ``ground_height``
/ ``ground_height_mode``.
+ ``head_trims`` maps a motion stem to N: the first N frames of that
+ motion's export NPZ are dropped on load, before downsampling and the
+ static trim (hand-reviewed bind-pose / foreign first frames, from
+ ``clip_trims.txt``).
+
+ ``activity_keep`` is a set of motion stems exempt from the low-activity
+ filter: hand-reviewed small but real motions (a head turn, a wave) that the
+ joint-activity gate cannot tell from a held pose (``activity_keep.txt``).
+ Every other filter still applies to them.
+
Object types with no motion NPZs, a joint count outside
``[min_joints, max_joints]`` or a degenerate (zero-extent) skeleton are
skipped with a reason instead of raising.
@@ -848,6 +884,8 @@ def process_object(object_type, object_npzs, save_dir,
'min_frames': min_frames,
'jump_step_threshold': jump_step_threshold,
'jump_ratio_threshold': jump_ratio_threshold,
+ 'head_trims': head_trims or {},
+ 'activity_keep': set(activity_keep or ()),
# I/O
'motions_save_dir': motions_save_dir,
'animations_save_dir': animations_save_dir,
diff --git a/data_process/utils/plotting.py b/data_process/utils/plotting.py
index e4ebebe..9a59157 100644
--- a/data_process/utils/plotting.py
+++ b/data_process/utils/plotting.py
@@ -1003,10 +1003,18 @@ def _wrap_title(text: str, fig_width_in: float, font_pt: float,
"""Hard-wrap a caption to the figure width (suptitle's own wrapping is
unreliable and long captions ran off both edges).
+ Line breaks already in *text* are kept: each one is wrapped on its own, so a
+ caller that writes "<clip>\n<caption>" gets two blocks rather than one run-on
+ paragraph (``textwrap`` treats a whole string as one paragraph and would fold
+ the newline into a space).
+
``fill`` is the share of the figure width a full line may occupy.
"""
chars = max(24, int(fig_width_in * 72.0 * fill / _mean_advance_pt(font_pt, family)))
- return '\n'.join(textwrap.wrap(text, chars)) or text
+ out = []
+ for para in text.split('\n'):
+ out.extend(textwrap.wrap(para, chars) or [para])
+ return '\n'.join(out) or text
def _draw_caption(fig, text: str, figsize, font_pt: float = CAPTION_FONT_SIZE,
diff --git a/data_process/utils/vlm.py b/data_process/utils/vlm.py
index 0ccb63f..7bf1905 100644
--- a/data_process/utils/vlm.py
+++ b/data_process/utils/vlm.py
@@ -481,6 +481,30 @@ def merge_shards(output_json, num_gpus=None):
save_json(merged, output_json)
logger.info("Merged {} shards -> {} total entries in {}".format(
len(shard_paths), len(merged), output_json))
+
+ # Each shard also records the clips it could not caption in
+ # ``<shard stem>_failed.txt``. Fold them into the canonical
+ # ``<output stem>_failed.txt`` so the next (single-process or sharded)
+ # run retries them; drop clips that another shard or a later pass
+ # captioned in the meantime.
+ base = os.path.splitext(output_json)[0]
+ failed_out = base + "_failed.txt"
+ failed = set()
+ if os.path.exists(failed_out):
+ with open(failed_out) as f:
+ failed.update(line.strip() for line in f if line.strip())
+ n_before = len(failed)
+ for spath in shard_paths:
+ shard_failed = os.path.splitext(spath)[0] + "_failed.txt"
+ if os.path.exists(shard_failed):
+ with open(shard_failed) as f:
+ failed.update(line.strip() for line in f if line.strip())
+ failed = {name for name in failed if not is_completed(merged.get(name))}
+ if failed or n_before:
+ with open(failed_out, "w") as f:
+ f.write("\n".join(sorted(failed)) + ("\n" if failed else ""))
+ logger.info("Merged failure lists -> {} pending clips in {}".format(
+ len(failed), failed_out))
return merged
diff --git a/data_process/vlm_caption/backends.py b/data_process/vlm_caption/backends.py
index a51e834..322d0de 100644
--- a/data_process/vlm_caption/backends.py
+++ b/data_process/vlm_caption/backends.py
@@ -53,12 +53,14 @@ def detect_backend(model_name):
"""Auto-detect backend from model name.
- Contains 'qwen' (case-insensitive) -> qwen
- - Starts with 'gemini' -> gemini
+ - Contains 'gemini' (case-insensitive, so ``google/gemini-...`` and
+ ``Gemini-3-...`` route correctly) -> gemini
- Everything else -> openai
"""
- if "qwen" in model_name.lower():
+ lowered = model_name.lower()
+ if "qwen" in lowered:
return BACKEND_QWEN
- elif model_name.startswith("gemini"):
+ elif "gemini" in lowered:
return BACKEND_GEMINI
else:
return BACKEND_OPENAI
diff --git a/data_process/vlm_caption/prompts.py b/data_process/vlm_caption/prompts.py
index 32da6e3..38a19f2 100644
--- a/data_process/vlm_caption/prompts.py
+++ b/data_process/vlm_caption/prompts.py
@@ -18,6 +18,12 @@ vocabulary, and style examples.
# Shared caption-prompt scaffold
# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
+# The subject every caption opens with, in every dataset. The three
+# datasets train as one mixture, so a per-dataset subject would hand the
+# text encoder a free dataset label and defeat the topology-agnostic
+# conditioning. Keep in sync with patch_annotations.py's SUBJECT.
+CORPUS_SUBJECT = 'An object'
+
_INPUT_CAMERAS = (
"# Input\n"
"Four synchronized cameras 90 degrees apart at fixed elevation, labeled "
@@ -151,7 +157,7 @@ MIXAMO_PROMPT = _caption_prompt(
articulation="limbs move, pelvis fixed",
in_place_example="'runs in place', 'marches in place'",
vocab_block="",
- subject="A person",
+ subject=CORPUS_SUBJECT,
uncertain_fallback=(
"If heading is uncertain, say 'one arm' / 'one leg' instead of "
"guessing left/right."
@@ -164,12 +170,12 @@ MIXAMO_PROMPT = _caption_prompt(
no_appearance_extra=", no clothing",
still_clause="subtle sway/breathing still counts as motion",
examples=(
- "- A person walks forward.\n"
- "- A person runs in place.\n"
- "- A person punches with the right fist.\n"
- "- A person raises one arm while seated.\n"
- "- A person turns around as if carrying something.\n"
- "- A person crouches and then rises back up.\n"
+ "- An object walks forward.\n"
+ "- An object runs in place.\n"
+ "- An object punches with the right fist.\n"
+ "- An object raises one arm while seated.\n"
+ "- An object turns around as if carrying something.\n"
+ "- An object crouches and then rises back up.\n"
),
)
@@ -259,7 +265,7 @@ TRUEBONES_PROMPT = _caption_prompt(
"them. Do not copy the label "
"verbatim: rewrite it in the required format and never carry a "
"species word into the caption โ€” the example label above becomes "
- "'An animal runs forward and then falls to the right.'\n\n"
+ "'An object runs forward and then falls to the right.'\n\n"
),
step1=(
"1. ESTABLISH HEADING. Cues, in priority order: (a) "
@@ -272,7 +278,7 @@ TRUEBONES_PROMPT = _caption_prompt(
articulation="limbs/wings/tail/jaw move, torso fixed",
in_place_example="'trots in place'",
vocab_block="",
- subject="An animal",
+ subject=CORPUS_SUBJECT,
uncertain_fallback=(
"If heading is uncertain, say 'one foreleg' / 'a wing' instead of "
"guessing left/right."
@@ -289,12 +295,12 @@ TRUEBONES_PROMPT = _caption_prompt(
no_appearance_extra="/color/texture",
still_clause="subtle sway, breathing, tail flicks, ear twitches still count as motion",
examples=(
- "- An animal gallops forward.\n"
- "- An animal turns to the right.\n"
- "- An animal rears upward on its hind legs.\n"
- "- An animal flaps its wings and rises.\n"
- "- An animal swims forward with its tail.\n"
- "- An animal strikes forward with its tail curled.\n"
+ "- An object gallops forward.\n"
+ "- An object turns to the right.\n"
+ "- An object rears upward on its hind legs.\n"
+ "- An object flaps its wings and rises.\n"
+ "- An object swims forward with its tail.\n"
+ "- An object strikes forward with its tail curled.\n"
),
)
diff --git a/scripts/run_animate_motion.sh b/scripts/run_animate_motion.sh
index ec864dc..4cb7197 100755
--- a/scripts/run_animate_motion.sh
+++ b/scripts/run_animate_motion.sh
@@ -12,8 +12,10 @@
# Detected as the last arg when it does not end in .npy / .npz.
#
# Env overrides:
-# ANIM_MODE (default: fk), CHAR_PATH (default: auto),
-# COND_PATH (default: dataset/features/<DATASET_TYPE>/cond.npy)
+# ANIM_MODE (default: fk), CHAR_PATH (default: auto; mixamo falls back to
+# dataset/raw/mixamo/character_refined/Y_Bot.fbx),
+# COND_PATH (default: dataset/features/<DATASET_TYPE>/cond.npy),
+# EXTRA_BONES_STRATEGY (merge|remove|keep, default: merge)
#
# Output: $OUTPUT_DIR/<anim_stem>.{glb,fbx}
@@ -128,6 +130,9 @@ process_one_motion() {
if [ -n "${CHAR_PATH:-}" ]; then
export CHAR_PATH
fi
+ if [ -n "${EXTRA_BONES_STRATEGY:-}" ]; then
+ export EXTRA_BONES_STRATEGY
+ fi
bash "$STAGE1_WRAPPER" "$DATASET_TYPE"
)
diff --git a/unimate/configs/schema.py b/unimate/configs/schema.py
index 9c2d967..423946b 100644
--- a/unimate/configs/schema.py
+++ b/unimate/configs/schema.py
@@ -92,6 +92,12 @@ class DatasetConfig:
# Sampling
sampler_alpha: float = 0.5 # Power-law exponent: 0=uniform-per-sample, 0.5=sqrt-balanced, 1=uniform-per-type
+ # Two-level sampling: dataset d gets total probability โˆ N_d^(1-sampler_dataset_alpha)
+ # (N_d = its clips; 0 = its natural clip share, 1 = equal per dataset), then sampler_alpha
+ # balances object types inside it. None = legacy single level, object types pooled across
+ # datasets (lets one-rig Mixamo fall to <1% of a UniML3D mixture). Default None keeps
+ # configs and runs written before 2026-09-27 bit-identical; the *_v2 configs set 0.25.
+ sampler_dataset_alpha: Optional[float] = None
# Train/eval split (clip-level, stratified per object_type).
# test_split_ratio=0 disables splitting (all clips โ†’ train).
@@ -296,15 +302,24 @@ class TrainingConfig:
@dataclass
class SamplingArgs:
- """Sampling arguments."""
+ """Sampling block: read by the periodic debug visualization during
+ training (``num_samples``, ``cfg_scale``) and as fallbacks by
+ ``unimate.inference.sample`` (``model_path``, ``cfg_scale``, ``device``).
+
+ ``output_dir``, ``seed``, ``object_type`` and ``num_repetitions`` are no
+ longer consumed by any entry point โ€” inference has its own ``--output_dir``
+ / ``--seed`` / ``--num_repetitions`` flags and test-case files. They stay
+ here only so every saved ``outputs/*/config.json`` (written with
+ ``dataclasses.asdict``) keeps loading; do not add new users.
+ """
model_path: Optional[str] = None
- output_dir: Optional[str] = None
- seed: Optional[int] = 10
+ output_dir: Optional[str] = None # deprecated, unused
+ seed: Optional[int] = 10 # deprecated, unused (inference: --seed)
# Sampling hyperparameters
- object_type: Optional[List[str]] = None
- num_samples: int = 4
- num_repetitions: int = 3
+ object_type: Optional[List[str]] = None # deprecated, unused
+ num_samples: int = 4 # debug visualization: object types per dataset per split
+ num_repetitions: int = 3 # deprecated, unused (inference: --num_repetitions)
device: str = "cuda"
# CFG
diff --git a/unimate/dataset/factory.py b/unimate/dataset/factory.py
index 69ddfec..bb9359b 100644
--- a/unimate/dataset/factory.py
+++ b/unimate/dataset/factory.py
@@ -91,7 +91,8 @@ def create_train_dataloader(
"""Build the training DataLoader over the train split of the Mixture dataset."""
dataset = create_dataset(dataset_config, model_config)
- sampler = MixtureSampler(dataset, alpha=dataset_config.sampler_alpha) if balanced else None
+ sampler = MixtureSampler(dataset, alpha=dataset_config.sampler_alpha,
+ dataset_alpha=dataset_config.sampler_dataset_alpha) if balanced else None
return DataLoader(
dataset,
diff --git a/unimate/dataset/mixture/dataset.py b/unimate/dataset/mixture/dataset.py
index b3a55a7..346e3a2 100644
--- a/unimate/dataset/mixture/dataset.py
+++ b/unimate/dataset/mixture/dataset.py
@@ -1363,50 +1363,80 @@ class MotionDataset(data.Dataset):
# ---------------------------------------------------------------------------
class MixtureSampler(data.Sampler):
- """Power-law balanced sampling across skeleton types.
+ """Power-law balanced sampling across datasets and skeleton types.
- Per object type k with n_k motions, total sampling probability โˆ n_k^(1-alpha).
- Per sample within type k, weight โˆ n_k^(-alpha).
+ Two levels. Dataset d with N_d motions gets total probability
+ โˆ N_d^(1-dataset_alpha); inside it, object type k with n_k motions gets a
+ share โˆ n_k^(1-alpha), split evenly over its motions.
- alpha=0.0 : uniform per sample (data-rich types dominate)
+ alpha=0.0 : uniform per sample within the dataset (data-rich types dominate)
alpha=0.5 : square-root balanced (default โ€” good compromise)
alpha=1.0 : uniform per type (ignores motion count entirely)
+ dataset_alpha=0.0 : each dataset keeps its natural share of the clips
+ dataset_alpha=1.0 : equal probability per dataset
+ dataset_alpha=None: single level (default, the behaviour before 2026-09-27) โ€”
+ object types pooled across all datasets. A dataset of one rig (Mixamo)
+ then counts as one type and falls below 1% of a UniML3D mixture; the
+ *_v2 configs set 0.25.
+
+ With one dataset both levels coincide with the legacy behaviour.
This is a plain Sampler (not DistributedSampler) that yields all indices.
Accelerate's BatchSamplerShard handles per-rank sharding automatically.
"""
- def __init__(self, data_source, alpha: float = 0.5):
+ def __init__(self, data_source, alpha: float = 0.5, dataset_alpha=None):
super().__init__(data_source)
name_list = data_source.motion_dataset.train_name_list
motion_dict = data_source.motion_dataset.train_motion_dict
total_samples = len(name_list)
+ # Keyed by (dataset_type, object_type): nothing stops two datasets from
+ # using the same object-type name.
indices_by_object = defaultdict(list)
for i, name in enumerate(name_list):
- indices_by_object[motion_dict[name]['object_type']].append(i)
+ m = motion_dict[name]
+ indices_by_object[(m['dataset_type'], m['object_type'])].append(i)
+ objects_by_dataset = defaultdict(list)
+ for key in indices_by_object:
+ objects_by_dataset[key[0]].append(key)
- # w_i = n_k^(-alpha) for sample i in type k, giving type k total
- # probability โˆ n_k * n_k^(-alpha) = n_k^(1-alpha).
weights = np.zeros(total_samples)
- for indices in indices_by_object.values():
- n_k = len(indices)
- weights[indices] = n_k ** (-alpha)
+ if dataset_alpha is None:
+ # w_i = n_k^(-alpha) for sample i in type k, giving type k total
+ # probability โˆ n_k * n_k^(-alpha) = n_k^(1-alpha).
+ for indices in indices_by_object.values():
+ weights[indices] = len(indices) ** (-alpha)
+ else:
+ for d_type, keys in objects_by_dataset.items():
+ n_d = sum(len(indices_by_object[k]) for k in keys)
+ d_share = n_d ** (1 - dataset_alpha)
+ type_mass = {k: len(indices_by_object[k]) ** (1 - alpha) for k in keys}
+ z = sum(type_mass.values())
+ for k in keys:
+ indices = indices_by_object[k]
+ weights[indices] = d_share * type_mass[k] / z / len(indices)
weights /= weights.sum()
+ dataset_probs = {d: sum(weights[indices_by_object[k]].sum() for k in keys)
+ for d, keys in objects_by_dataset.items()}
+ logger.info(f'MixtureSampler (alpha={alpha}, dataset_alpha={dataset_alpha}): '
+ + ', '.join(f'{d} n={sum(len(indices_by_object[k]) for k in keys)} '
+ f'p={dataset_probs[d]:.3f}'
+ for d, keys in objects_by_dataset.items()))
type_probs = {
object_type: weights[indices].sum()
for object_type, indices in indices_by_object.items()
}
sorted_types = sorted(type_probs.items(), key=lambda x: x[1], reverse=True)
- logger.info(f'MixtureSampler (alpha={alpha}): {len(sorted_types)} object types')
+ logger.info(f'MixtureSampler: {len(sorted_types)} object types')
for name, prob in sorted_types[:5]:
- logger.info(f' top: {name} โ€” n={len(indices_by_object[name])}, p={prob:.4f}')
+ logger.info(f' top: {name[1]} ({name[0]}) โ€” n={len(indices_by_object[name])}, p={prob:.4f}')
if len(sorted_types) > 10:
logger.info(' ...')
for name, prob in sorted_types[-5:]:
- logger.info(f' bot: {name} โ€” n={len(indices_by_object[name])}, p={prob:.4f}')
+ logger.info(f' bot: {name[1]} ({name[0]}) โ€” n={len(indices_by_object[name])}, p={prob:.4f}')
self.weights = torch.as_tensor(weights, dtype=torch.double)
self.total_samples = total_samples
diff --git a/unimate/inference/sample.py b/unimate/inference/sample.py
index 4b76965..4345c73 100644
--- a/unimate/inference/sample.py
+++ b/unimate/inference/sample.py
@@ -77,7 +77,9 @@ class InferenceArgs:
# train or eval motion_dict) or the dataset's eval split.
inbetween: bool = False
# Comma-separated signed temporal indices to hold clean. Negatives count
- # from the per-clip valid length. Default keeps first + last frame.
+ # from the end of the generation window (max_motion_length), so -1 is the
+ # last slot even when the GT clip is shorter (see build_keep_mask).
+ # Default keeps first + last slot.
keep_frames: str = "0,-1"
# Text-guided motion editing: clamp the listed joints to GT for all
# frames and let the model denoise the rest under a new caption (e.g.,