Python
Self-contained ONNX release with the MHR body graph, model download, TensorRT, public scoring script and rewritten docs.
mohamdevc1672a4f6534
diff --git a/.gitignore b/.gitignoreindex d2cf080..5653a03 100644--- a/.gitignore+++ b/.gitignore@@ -1,58 +1,26 @@-# Byte-compiled / cache__pycache__/*.py[cod]-*$py.class.pytest_cache/-.mypy_cache/-.ruff_cache/--# Editor / OS.vscode/.idea/.DS_Store--# Virtual environments.venv/venv/-env/-# RF-DETR auto-downloaded weights-rf-detr*.pth-*.pth.lock+# weights and outputs*.onnx+*.pt*.pth-*.mp4--# Rerun recordings*.rrd--# Annotation outputs-data/-annotations/-images/-checkpoints/--# Model weights — too large for git, use Git LFS or release artifacts-models/*.onnx-models/*.pt-models/*.pth-models/mhr_assets/-commands.md-!models/README.md--# Transfer archives (dataset tarballs)+*.mp4*.tar*.tar.gz--# Training run outputs (checkpoints / ONNX exports / logs)+models/runs/+checkpoints/-# Notebook checkpoints-.ipynb_checkpoints/-/competitors/-*.png--# Saved teacher runs for `demo.py --show-reference` (renders + packed .npz).-# Local and regenerable: tools/pack_reference.py rebuilds the .npz from a-# video_to_pose_pipeline --save_meshes run. 247 MB here.-reference/+# training data+data/+data_*/+annotations/+images/diff --git a/README.md b/README.mdindex fe77402..b4b5578 100644--- a/README.md+++ b/README.md@@ -1,449 +1,141 @@-# InstantHMR+# InstantHMR: Real-Time Human Mesh Recovery on Your Phone-+InstantHMR recovers the 3D body of every person in an image: pose, body shape, 70 keypoints in 2D+and 3D, and a full body mesh in the [Momentum Human Rig](https://github.com/facebookresearch/MHR)+(MHR). It runs in about 5 ms per person on a phone NPU and in about 1 ms on a desktop GPU.-A lightweight, ONNX-exportable MHR body regressor for **3D human pose estimation-and mesh recovery**: a RepViT-M1.5 backbone + a 9-token cross-attention decoder +-CLIFF camera conditioning.+<p align="center">+ <img src="assets/demo_desktop.gif" height="300" alt="InstantHMR on a desktop GPU, shown in the Rerun viewer">+ <img src="assets/demo_phone.gif" height="300" alt="InstantHMR on a Samsung Galaxy S23">+</p>-It is trained on the **released ground-truth annotations** of-[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)-— the human MHR fits used to build SAM 3D Body — predicting per-person MHR-parameters and 70 keypoints from a single 224×224 crop. Distilling from-[`facebook/sam-3d-body-dinov3`](https://huggingface.co/facebook/sam-3d-body-dinov3)-by running the model over your own images is supported as an **option** (see-[Training](#training)), but it is not how the shipped models were trained.+## Models-InstantHMR ships as a single `.onnx` file. The demo pipeline pairs it with-**RF-DETR** for person detection — both stages are timed independently so-you always know where the latency is going. Optionally, the demo decodes-the raw `mhr_params` / `shape_params` outputs through Meta's-**[MHR body model](https://github.com/facebookresearch/MHR)** to render a-full dense body mesh in the Rerun viewer.+Four sizes, trained on the released annotations of the+[SAM 3D Body dataset](https://huggingface.co/datasets/facebook/sam-3d-body-dataset).+Accuracy is PA-MPJPE (lower is better) under the protocol of the paper; latency is per person,+for the network as measured in the paper (the desktop column with TensorRT directly).-- **Inputs:** `image (N, 3, 224, 224)`, `cliff_cond (N, 3)`.-- **Outputs:** `mhr_params (204)`, `shape_params (45)`, `cam_trans (3)`,- `joints_2d (70, 2)` (normalised crop coords),- `joints_3d (70, 3)` (body-centred metres, Y-down).-- **Speed (InstantHMR ONNX alone):** ~5 ms / frame (~200 FPS) on a single- RTX 4070 with the fp16 ONNX (CUDA EP); CPU works too (~25 FPS, depending- on hardware). On Apple Silicon, pass `--device coreml` to use- `CoreMLExecutionProvider`.-- **Speed (full demo, end-to-end):** the demo also runs **RF-DETR** every- frame, and on most hardware the detector — not InstantHMR — is the- bottleneck. Use `--detector-stride N` to run RF-DETR only every Nth- frame and reuse the previous bbox in between (see *Performance tuning*- below).-- **Note for RTX 50-series (Blackwell, sm_120):** stock Torch with- CUDA 12.4 wheels does not ship sm_120 kernels and falls back to slow- paths. Use `python install.py` which automatically pulls Torch- cu128 (>=2.7) on Blackwell GPUs.+| Model | Parameters | EMDB-1 (mm) | 3DPW (mm) | RTX 4070, TensorRT (ms) | Galaxy S23 NPU (ms) | Licence |+|---|---|---|---|---|---|---|+| `nano` | 20.0 M | 61.7 | 46.6 | 0.70 | 4.60 | CC BY-NC 4.0 |+| `tiny` | 20.3 M | 58.0 | 44.0 | 0.73 | 4.84 | CC BY-NC 4.0 |+| `small` | 24.4 M | 53.3 | 41.7 | 0.84 | 5.38 | CC BY-NC 4.0 |+| `medium` (default) | 32.2 M | 48.8 | 40.0 | 0.98 | 5.49 | CC BY-NC 4.0 |+| `medium-commercial` | 32.2 M | 72.2 | 50.7 | 0.98 | 5.49 | Apache-2.0 |-## Install--```bash-git clone <this-repo> instanthmr-cd instanthmr-```+`medium-commercial` is the medium model trained only on images that allow commercial use+(Harmony4D and part of COCO). The weights download automatically from+[Hugging Face](https://huggingface.co/momolesang/InstantHMR) on first use.-Choose the path that matches your needs:+## Install-| Goal | Python | Command |-|------|--------|---------|-| Pose estimation only (joints + skeleton) | 3.11 or 3.12 | `python install.py` |-| + MHR body mesh rendering | **3.12** | `python install.py` then see [MHR body mesh](#mhr-body-mesh-rendering) |+Python 3.10 to 3.12, and on Linux or Windows an NVIDIA driver 525 or newer.```bash-# Pose estimation only (Python 3.11+):-conda create -n instanthmr python=3.11 -y && conda activate instanthmr--# Picks the right torch + onnxruntime wheels for your machine:-# - Linux + NVIDIA GPU → cu128 (Blackwell/RTX 50) or cu124 (Ada/Ampere/Hopper)-# + onnxruntime-gpu + bundled CUDA / cuDNN runtime libs-# - macOS → stock torch + onnxruntime (CoreML EP included)-# - Linux without GPU → CPU torch + onnxruntime-python install.py+git clone https://github.com/Naturalpad-studio/InstantHMR.git+cd InstantHMR+python -m venv .venv && source .venv/bin/activate+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126+pip install -r requirements.txt```-`python install.py --dry-run` prints the pip commands without running them.-`python install.py --force-cpu` skips GPU detection on Linux.-If you'd rather manage wheels yourself, `pip install -r requirements.txt`-still works — it installs the CPU fallback path.+The first `pip` line installs PyTorch for CUDA 12, which the person detector uses and onnxruntime+shares, so a single set of CUDA libraries is installed. On a Mac, or without an NVIDIA GPU, skip+it.-The first run downloads the RF-DETR (medium) checkpoint automatically.-The InstantHMR ONNX weights are on HuggingFace — see-[Model weights](#model-weights) below.--## Run the demo+## Demo```bash-# Single image-python demo.py --image path/to/photo.jpg--# Video file-python demo.py --video path/to/clip.mp4--# Live webcam (index 0)+python demo.py --image photo.jpg+python demo.py --video clip.mp4 --smoothpython demo.py --camera 0```-`python demo.py --help` lists every flag (model path, device, detector-variant, confidence, max persons, frame skip, `.rrd` recording, …).--The demo opens a Rerun viewer with the source image + 2D skeleton, a live-RF-DETR / InstantHMR / total-latency plot, and the 3D scene with the-predicted camera frustum.--## MHR body mesh rendering--The demo can render a full dense body mesh for each detected person by-running a forward pass through Meta's **MHR (Momentum Human Rig)** body-model, using the `mhr_params (204,)` and `shape_params (45,)` outputs that-InstantHMR already produces. Render time is displayed separately in the-Rerun latency plot and in the console.--### Prerequisites--| Requirement | Notes |-|-------------|-------|-| **Python ≥ 3.12** | `pymomentum` has no Python 3.11 wheels; recreate your conda env if needed |-| NVIDIA GPU / Apple Silicon | CPU works but is much slower |-| CUDA toolkit | Must match your PyTorch wheel (Linux/Windows) |--### Step 1 — Install in one pip call--Always pass **both** requirement files to a **single** `pip install` call.-Running them separately causes pip to first choose a CUDA-13 torch for-`rfdetr`, then downgrade to CUDA-12 torch for `pymomentum`, leaving-`torchvision` broken.--```bash-conda create -n instanthmr python=3.12 -y && conda activate instanthmr-pip install -r requirements.txt -r requirements-mhr.txt-```--`requirements-mhr.txt` auto-selects the right package for your platform-(`pymomentum-gpu` on Linux/Windows, `pymomentum-cpu` on macOS).-CPU-only Linux users: see the comment at the top of `requirements-mhr.txt`.--> **⚠ Wrong `pymomentum` on PyPI**-> `pip install pymomentum` installs an **unrelated** legacy SMS library-> (pyMomentum v0.1.x by MomentumAS) — not the Meta geometry library.-> `requirements-mhr.txt` pulls in the correct package automatically.-> If you accidentally installed the wrong one first:-> ```bash-> pip uninstall pymomentum-> pip install -r requirements.txt -r requirements-mhr.txt-> ```--> **Torch/torchvision version mismatch?**-> ```bash-> python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__)"-> # If they differ, wipe both and reinstall together:-> pip uninstall -y torch torchvision-> pip install -r requirements.txt -r requirements-mhr.txt-> ```+The [Rerun](https://rerun.io) viewer opens with the frame and its 2D skeletons, the 3D scene, and+the latency of each stage. People are found by the [RF-DETR](https://github.com/roboflow/rf-detr)+detector, then InstantHMR runs on all of them in one batch.-### Step 2 — Download the body-model assets--```bash-curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.1/assets.zip-unzip assets.zip -d models/mhr_assets-```--This unpacks the body template, blend shapes, skinning weights, and pose-correctives into `models/mhr_assets/`.--### Step 3 — Run the demo--```bash-# Single image — LOD 0 (73 639 vertices, highest quality)-python demo.py --image photo.jpg --mhr-assets models/mhr_assets --mhr-lod 0--# Video file — LOD 1 (18 439 vertices, good quality)-python demo.py --video clip.mp4 --mhr-assets models/mhr_assets --mhr-lod 1--# Live camera — LOD 3 (4 899 vertices, real-time default)-python demo.py --camera 0 --mhr-assets models/mhr_assets-```--### LOD reference--| `--mhr-lod` | Vertices | Faces | Recommended use |-|:-----------:|----------|-------|-----------------|-| 0 | 73 639 | — | offline images, max quality |-| 1 | 18 439 | — | video, powerful GPU |-| 2 | 10 661 | — | video, mid-range GPU |-| **3** | **4 899** | — | **live camera (default)** |-| 4 | 2 461 | — | weak GPU / many persons |-| 5 | 971 | — | borderline real-time |-| 6 | 595 | — | debugging / stress test |--### How it works--```-InstantHMR ONNX- └─ mhr_params (204,) ──┐- └─ shape_params (45,) ──┤─→ MHR.forward() → vertices (V, 3)- └─ cam_trans (3,) ──┘ + cam_trans → camera-space mesh-```--The 204-dim `mhr_params` are MHR **model parameters**, not a 6-D rotation-encoding. Read off `character_torch.parameter_transform.parameter_names`, the-layout is:--| slice | contents |+| Option | ||---|---|-| `0:3` | `root_tx/ty/tz` — always 0 in the corpus; 1 unit = 10 cm, applied at joint 1 |-| `3:6` | `root_rx/ry/rz` — extrinsic-XYZ Euler, driving joint-parameter slots 10/11/12 and nothing else |-| `6:136` | 130 local joint angles (`spine0_rx_flexible`, `l_knee_ry`, …) |-| `136:204` | 68 bone-scale parameters |--Because the root rotation is exclusive to slots 10/11/12, `6:136` is exactly-invariant to a camera roll — which is what `pose_split` in the trainer relies-on. `shape_params (45,)` encode identity blend shapes (20 body + 20 head + 5-hand components) and drive the **mesh only**: a ±2σ probe moves the 127-joint-skeleton by `0.00e+00 cm`. Facial expression is set to neutral (zeros) since-InstantHMR does not regress it.--## Performance tuning--On every machine we've measured, RF-DETR (the detector) costs ~5–10× more-per frame than InstantHMR. The two flags below target that bottleneck.--| Flag | Effect |-| --- | --- |-| `--detector-stride N` | Run RF-DETR every Nth frame; reuse the previous bbox (slightly expanded) on the in-between frames. Stride 2–3 is the single biggest knob — typically 2–3× end-to-end FPS for slow movement, with negligible quality loss. |-| `--detector-variant nano` | Use the smallest RF-DETR; biggest win on CPU / Apple Silicon where the detector is doing all the work. |-| `--device coreml` | On macOS, route the InstantHMR ONNX through `CoreMLExecutionProvider`. |-| `--max-persons N` | Maximum number of persons processed per frame (default: **2**). See below. |-| `--no-batch-persons` | Disable batched multi-person HMR (one ONNX call per person). The default is batched. |-| `--focal PX` | Your camera's focal length **in pixels**, when you know it. Models exported from a `--cliff-focal` checkpoint are conditioned on *angles*, so they need a focal; without one the demo falls back to `1.05 x` the image diagonal, which is a reasonable phone/webcam stand-in but mis-places the person in depth on any camera whose true focal differs. Older pixel-conditioned graphs ignore it and are unaffected. |--### `--max-persons` and why it matters--`--max-persons` does two things at once:--1. **Caps detections** — RF-DETR may find more people than you need. Setting `N`- keeps only the `N` highest-confidence detections per frame.--2. **Fixes the ONNX batch size** — CUDA / ONNX Runtime compiles GPU kernels the- first time a model runs at a new input shape. Because the batch dimension- counts as part of the shape, switching from 1 person to 2 persons mid-demo- would normally trigger a multi-second stall. The pipeline avoids this by- **always** padding the input to exactly `N` crops (zero-filling unused slots)- so the session always sees `batch = N`, regardless of how many people are- actually in frame.--**Startup:** during `PosePipeline.__init__` the pipeline runs two warm-up passes-— `batch = N` and `batch = 1` (the single-person path) — so all CUDA kernels-are compiled before the first real frame. Startup cost is roughly `2 ×`-one-inference time and does **not** scale with `N`.--**Choosing `N`:**--| Scene | Recommended `--max-persons` |-|-------|----------------------------|-| Single person / portrait | 1 |-| Two people, couples, sparring | **2 (default)** |-| Small group, team sport | 4–6 |-| Crowd / many people visible | raise `N`; also consider `--detector-stride` |--Raising `N` does not slow down single-person frames at inference time (the-padded slots run on GPU in parallel), but it does proportionally increase-the memory footprint of the batch buffer and may slightly increase latency-when `N` is large and the GPU can no longer fully parallelise the extra crops.--Measured on RTX 4070 + torch 2.5 cu121 + ORT 1.25 (1080p video, 1 person, 150 frames after warm-up):--| `--detector-stride` | RF-DETR ms | HMR ms | total ms | FPS |-| ---: | ---: | ---: | ---: | ---: |-| 1 | 27.1 | 5.3 | 32.4 | **30.9** |-| 2 | 13.3 | 5.3 | 18.6 | **53.8** |-| 3 | 8.7 | 5.5 | 14.2 | **70.4** |-| 4 | 6.6 | 5.3 | 11.9 | **83.9** |--You can re-run the benchmark on your own hardware with:--```bash-python tools/bench.py --video vid1.mp4 --max-frames 150 --detector-stride 3-```+| `--model` | `nano`, `tiny`, `small`, `medium` (default), `medium-commercial`, or a path to an `.onnx` file |+| `--device` | `cuda` (default), `tensorrt`, `cpu` or `coreml` |+| `--smooth` | temporal smoothing for videos and cameras (1 Euro filter, no visible lag) |+| `--focal` | the camera's focal length in pixels, if you know it; it places people at the right depth |+| `--detector` | RF-DETR size: `nano`, `small`, `medium` (default), `base`, `large` |+| `--max-persons` | most people per frame (default 4) |+| `--no-mesh` | skeleton only |+| `--save-rrd` | also save the recording, to replay with `rerun file.rrd` |++The detector is the slow stage (about 27 ms per frame for RF-DETR medium on an RTX 4070, against+5 ms for InstantHMR with TensorRT). For a faster live demo, use `--detector nano` or+`--detector-stride 2`.## Use it from Python```python+import cv2from instanthmr import PosePipeline-pipeline = PosePipeline(- onnx_path="models/instanthmr.onnx",- device="cuda",- detector_variant="medium",-)--result = pipeline.predict(image_rgb)-for r in result.persons:- print(r.joints_3d_cam.shape) # (70, 3)- print(r.joints_2d.shape) # (70, 2)- print(r.mhr_params.shape) # (204,) — MHR pose parameters- print(r.shape_params.shape) # (45,) — MHR identity shape parameters-```--The 70 keypoints follow the MHR70 ordering — see-[`instanthmr/skeleton.py`](instanthmr/skeleton.py) for joint names and the-canonical skeleton edge list.--To run the MHR mesh decoder yourself:--```python-from instanthmr.mhr_renderer import MHRRenderer--renderer = MHRRenderer(assets_folder="models/mhr_assets", device="cuda", lod=3)--for r in result.persons:- verts_local = renderer.forward(r.mhr_params, r.shape_params) # (V, 3)- verts_cam = verts_local + r.cam_trans # camera space- faces = renderer.faces # (F, 3) int32+pipeline = PosePipeline("medium", device="cuda")+image = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB)+for person in pipeline(image).persons:+ person.joints_2d # (70, 2) keypoints, pixels+ person.joints_3d_cam # (70, 3) keypoints, metres, camera frame (x right, y down, z forward)+ person.vertices_cam # (18439, 3) mesh, metres, camera frame; faces: pipeline.hmr.faces+ person.mhr_params # (204,) MHR pose and skeleton parameters+ person.shape_params # (45,) MHR body shape```-## Model weights--The InstantHMR ONNX weights are hosted on HuggingFace:--> **<https://huggingface.co/momolesang/InstantHMR>**--Download `instanthmr.onnx` and place it under `models/`, or pass any path-via `--model`.--## Training+With your own person boxes, use `InstantHMR` directly:+`InstantHMR(resolve_model("medium")).predict(image, boxes)`. The 70 keypoints are listed in+`instanthmr.JOINT_NAMES`.-InstantHMR can be trained from two different label sources. **Both produce the-exact same `images/*.png` + `annotations/*.npz` layout** that-[`training/`](training/README.md) consumes, so training is identical either way:+## TensorRT-1. **Released ground-truth annotations — this is what every shipped model uses.**- Train directly on- [`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)- — the human MHR fits used to build SAM 3D Body — converted to the `.npz`- schema by [`tools/parquet_to_npz.py`](tools/parquet_to_npz.py) and laid out- by [`datasets_pipeline/build_split.py`](datasets_pipeline/build_split.py)- into `data/sam3d_gt_<dataset>/`. The released models were trained only on- this kind (sa1b / aic / harmony4d / coco / mpii).-2. **Teacher distillation — an option, not the default.** Run- `facebook/sam-3d-body-dinov3` over your own images with- [`tools/annotate_dataset.py`](tools/annotate_dataset.py) to generate- per-person pseudo-labels — see [`docs/annotation.md`](docs/annotation.md).- This is the only thing that writes `data/sam3d_distill_mix/`. Use it to label- imagery the released dataset does not cover; note that it caps the student at- the teacher's own accuracy, which training on the ground truth does not.--The `data/sam3d_distill_mix` directory keeps "distill" in its name for-historical reasons. The name does **not** mean the labels are teacher inference —-only `tools/annotate_dataset.py` produces those.--### Train on the original annotations (COCO example)--The dataset is gated: first **request access** on the-[dataset page](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)-and `huggingface-cli login`. The annotation parquets ship **without images**, so-you download the source images once per sub-dataset (COCO and MPII are the-easiest — a single `wget`; others may need extra access / undistortion, see the-[dataset setup guides](https://github.com/facebookresearch/sam-3d-body/tree/main/data)).+TensorRT runs InstantHMR in fp16 on NVIDIA GPUs. Install it next to onnxruntime and pick the+device:```bash-# 1. Annotations — download the coco_train split (uses Meta's sam-3d-body repo)-git clone https://github.com/facebookresearch/sam-3d-body-python sam-3d-body/data/scripts/download.py \- --save_dir $ANN_DIR --splits coco_train--# 2. Images — COCO 2014 train (the split the parquet filenames reference)-mkdir -p $COCO_IMG_DIR && cd $COCO_IMG_DIR-wget http://images.cocodataset.org/zips/train2014.zip && unzip -q train2014.zip-cd ---# 3. Convert parquet -> per-crop images/ + annotations/ (.npz)-python tools/parquet_to_npz.py \- --annotation_dir $ANN_DIR/coco_train \- --image_dir $COCO_IMG_DIR \- --output_dir data/sam3d_gt_coco \- --validate--# 4. Train on it (see training/README.md for the released models' settings):-python training/train_ddp.py --data_root data --output_dir runs/coco+pip install tensorrt-cu12==10.15.*+python demo.py --camera 0 --device tensorrt```-The converter recovers `cam_trans` from the MHR params, derives the 1.2× square-crop, skips low-quality fits (`mhr_valid`), and writes the same keys-`tools/annotate_dataset.py` does. `--validate` asserts that each person's-`joints_3d + cam_trans` reprojects onto the stored `joints_2d` to < 1 px.--### Other splits (MPII, AI Challenger, Harmony4D, EgoHumans)--Every split uses the **same converter** — the annotation schema and the MHR math-are dataset-independent (verified: 0 px reprojection and an image-centred-principal point on all of them). Only the **image preparation** differs. Download-the annotations the same way (`download.py --splits <split>`), prepare the images-as below, then run the converter into its own `data/sam3d_gt_<name>/` folder.--| Split | Image source | Prep | `--image_dir` |-|-------|--------------|------|---------------|-| `mpii_train` | `wget` from MPI servers (no registration) | unpack `mpii_human_pose_v1.tar.gz` | `$MPII_IMG_DIR` (contains `images/`) |-| `aic_train` | AI Challenger 2017 keypoint images (original site defunct — use a mirror) | arrange as `train/images/` | `$AIC_IMG_DIR` |-| `harmony4d_train` | HF `Jyun-Ting/Harmony4D` | **undistort** with Meta's `data/scripts/harmony4d/` script → `$HARMONY4D_IMG_DIR` | `$HARMONY4D_IMG_DIR` |-| `egohumans_train` | Google Drive (see setup guide) | **undistort** with Meta's `data/scripts/egohumans/` script → `$EGOHUMANS_IMG_DIR` | `$EGOHUMANS_IMG_DIR` |--```bash-# --- MPII --------------------------------------------------------------------python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits mpii_train-mkdir -p $MPII_IMG_DIR && cd $MPII_IMG_DIR-wget https://datasets.d2.mpi-inf.mpg.de/andriluka14cvpr/mpii_human_pose_v1.tar.gz-tar xzf mpii_human_pose_v1.tar.gz && cd - # -> $MPII_IMG_DIR/images/-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/mpii_train \- --image_dir $MPII_IMG_DIR --output_dir data/sam3d_gt_mpii --validate--# --- AI Challenger ------------------------------------------------------------python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits aic_train-# place the keypoint images so that $AIC_IMG_DIR/train/images/<hash>.jpg exists-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/aic_train \- --image_dir $AIC_IMG_DIR --output_dir data/sam3d_gt_aic --validate+The first run builds an engine for your GPU, about two minutes, and caches it in+`models/trt_cache/`; later runs start in under a second. The engine covers batches of up to+`--max-persons` people. On an RTX 4070, the `medium` network takes 1.2 ms per person with+TensorRT against 3.5 ms with CUDA; the MHR body (keypoints and mesh) runs on CUDA in 1.2 ms+either way.-# --- Harmony4D (needs undistortion) ------------------------------------------python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits harmony4d_train-# 1) download Jyun-Ting/Harmony4D into $HARMONY4D_DATA_DIR-# 2) undistort -> $HARMONY4D_IMG_DIR (Meta's data/scripts/harmony4d/, see setup guide)-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/harmony4d_train \- --image_dir $HARMONY4D_IMG_DIR --output_dir data/sam3d_gt_harmony4d --validate+## Training and evaluation-# --- EgoHumans (needs undistortion) ------------------------------------------python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits egohumans_train-# 1) download EgoHumans into $EGOHUMANS_DATA_DIR-# 2) undistort -> $EGOHUMANS_IMG_DIR (Meta's data/scripts/egohumans/, see setup guide)-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/egohumans_train \- --image_dir $EGOHUMANS_IMG_DIR --output_dir data/sam3d_gt_egohumans --validate-```+- [`training/`](training/README.md): train the released models, or your own, from the SAM 3D Body+ annotations, and export them to ONNX.+- [`datasets_pipeline/`](datasets_pipeline/README.md): download the annotations and images and+ build the training data.+- [`benchmark/`](benchmark/README.md): reproduce the paper's EMDB-1 and 3DPW numbers.-> **Harmony4D / EgoHumans must be undistorted first.** Their annotations-> (`keypoints_2d`, `cam_int`) correspond to the **undistorted** frames produced-> by Meta's per-dataset scripts, so point `--image_dir` at the undistorted output,-> not the raw download. The image-prep details and exact undistortion commands-> live in the [dataset setup guides](https://github.com/facebookresearch/sam-3d-body/tree/main/data).+## Licence-To train on several datasets at once, convert each into its own-`data/sam3d_gt_<name>/` under the same `--data_root`; `train_ddp.py --mix` sets-how often each one is sampled.+The code is under the [Apache-2.0](LICENSE) licence. The `nano`, `tiny`, `small` and `medium`+weights are under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), for+non-commercial use only, because some of their training images (SA-1B, AI Challenger) are; `medium-commercial` and the MHR body graph are under Apache-2.0. The body model is Meta's+[MHR](https://github.com/facebookresearch/MHR) (Apache-2.0).-## Documentation+## Acknowledgements-- **[`training/README.md`](training/README.md)** — training and ONNX export.-- **[`docs/annotation.md`](docs/annotation.md)** — training data generation- with the SAM3D teacher.+InstantHMR is trained on the annotations Meta released with+[SAM 3D Body](https://github.com/facebookresearch/sam-3d-body) and uses Meta's+[MHR](https://github.com/facebookresearch/MHR) body model. People are detected with+[RF-DETR](https://github.com/roboflow/rf-detr) and shown with [Rerun](https://rerun.io). Training+was granted access to the HPC resources of IDRIS under the allocation AD011017116 made by GENCI.-## License+## Citation-The code in this repository is released under the **Apache License 2.0**-(see [LICENSE](LICENSE)).--The model weights distributed at-<https://huggingface.co/momolesang/InstantHMR> are released under the-[SAM license](https://github.com/facebookresearch/sam-3d-body), since-InstantHMR is trained on `facebook/sam-3d-body-dataset` labels. Please-review the SAM and RF-DETR licenses before downstream use.+<!-- ticket 20: add the authors and the arXiv identifier. -->+```bibtex+@article{instanthmr2026,+ title = {InstantHMR: Real-Time Human Mesh Recovery on Your Phone},+ year = {2026}+}+```diff --git a/assets/demo_desktop.gif b/assets/demo_desktop.gifnew file mode 100644index 0000000..0a4c55dBinary files /dev/null and b/assets/demo_desktop.gif differdiff --git a/assets/demo_phone.gif b/assets/demo_phone.gifnew file mode 100644index 0000000..ad3743bBinary files /dev/null and b/assets/demo_phone.gif differdiff --git a/benchmark/README.md b/benchmark/README.mdnew file mode 100644index 0000000..ef5a965--- /dev/null+++ b/benchmark/README.md@@ -0,0 +1,55 @@+# Evaluation++`evaluate.py` scores an InstantHMR model on EMDB-1 and 3DPW test under the protocol of the paper:+every 10th frame, ground-truth person boxes, the calibrated focal length, and Meta's official MHR+to SMPL fitting. EMDB-1 is scored on the 24 SMPL joints and 3DPW on the 14 joints regressed with+the Human3.6M regressor, as in each benchmark's literature; both joint sets are printed.++## Setup++```bash+pip install -r requirements.txt -r benchmark/requirements.txt+git clone https://github.com/facebookresearch/MHR.git ../MHR+curl -L -o mhr_assets.zip https://github.com/facebookresearch/MHR/releases/latest/download/assets.zip+mkdir -p training/checkpoints && unzip -j mhr_assets.zip assets/mhr_model.pt -d training/checkpoints+```++Then put these files in one folder, the `--smpl-dir`:++| File | Where from |+|---|---|+| `SMPL_MALE.pkl`, `SMPL_FEMALE.pkl` | [SMPL](https://smpl.is.tue.mpg.de), "SMPL for Python", renamed |+| `SMPL_NEUTRAL.pkl` | [SMPLify](https://smplify.is.tue.mpg.de), `basicModel_neutral_lbs_10_207_0_v1.0.0.pkl`, renamed |+| `J_regressor_h36m.npy` | the data of [SPIN](https://github.com/nkolot/SPIN) (`fetch_data.sh`) |++And the benchmarks:++- **EMDB**: request access at [emdb.ait.ethz.ch](https://eth-ait.github.io/emdb/); `--data` is the+ folder that holds `P0/`, `P1/`, ...+- **3DPW**: from [virtualhumans.mpi-inf.mpg.de/3DPW](https://virtualhumans.mpi-inf.mpg.de/3DPW/);+ `--data` is the folder that holds `sequenceFiles/` and `imageFiles/`.++## Run++```bash+python benchmark/evaluate.py --dataset emdb --data <EMDB> --model medium \+ --smpl-dir <SMPL files> --mhr-repo ../MHR+python benchmark/evaluate.py --dataset 3dpw --data <3DPW> --model medium \+ --smpl-dir <SMPL files> --mhr-repo ../MHR+```++`--limit 50` scores the first 50 frames as a quick check. The fitting runs on the GPU; a full+benchmark takes a few minutes.++## Expected results++`medium` reproduces Table 2 of the paper:++| | PA-MPJPE (mm) | MPJPE (mm) | PVE (mm) |+|---|---|---|---|+| EMDB-1, 24 joints | 48.8 | 74.9 | 89.1 |+| 3DPW, 14 joints | 40.0 | 64.0 | 76.8 |++The paper ran the PyTorch checkpoints in fp16 and the released ONNX graphs run in fp32, so single+frames differ slightly; the means above agree to 0.003 mm. Meta's fitting is not bitwise+deterministic on a GPU, so a rerun can move them by a few hundredths of a millimetre.diff --git a/benchmark/evaluate.py b/benchmark/evaluate.pynew file mode 100644index 0000000..b4f8755--- /dev/null+++ b/benchmark/evaluate.py@@ -0,0 +1,240 @@+#!/usr/bin/env python3+"""Score an InstantHMR model on EMDB-1 or 3DPW test under the paper's protocol.++Protocol: every 10th frame; ground-truth person boxes (EMDB's own boxes; for 3DPW, the extent of+the 2D keypoints with confidence > 0.1, padded by 20%); the calibrated focal length; the MHR mesh+(with Meta's pose correctives) converted to SMPL by Meta's official fitting+(facebookresearch/MHR, tools/mhr_smpl_conversion); EMDB-1 scored on the 24 SMPL joints, 3DPW on+the 14 joints regressed from the mesh with the H36M regressor. Both joint sets are printed.++ python benchmark/evaluate.py --dataset emdb --data <EMDB root> --model medium \\+ --smpl-dir <SMPL files> --mhr-repo <MHR clone> --mhr-model training/checkpoints/mhr_model.pt+ python benchmark/evaluate.py --dataset 3dpw --data <3DPW root> ...++See benchmark/README.md for the downloads.+"""+from __future__ import annotations++import argparse+import json+import os+import pickle+import sys+import types+from pathlib import Path++import cv2+import numpy as np+import torch+from tqdm import tqdm++HERE = Path(__file__).resolve().parent+sys.path.insert(0, str(HERE.parent))+sys.path.insert(0, str(HERE))++from benchlib import joints as J # noqa: E402+from benchlib import metrics as M # noqa: E402+from instanthmr import InstantHMR, resolve_model # noqa: E402+from make_3dpw_gt import _install_chumpy_stub, _to_numpy, load_smpl, smpl_forward # noqa: E402++MM = 1000.0+++class _FacesOnlyRig:+ """What Meta's MHR -> SMPL conversion reads from an MHR model: its faces."""++ def __init__(self, faces):+ self.character = types.SimpleNamespace(mesh=types.SimpleNamespace(faces=faces))++ def to(self, _device):+ return self+++class _AnyName(types.ModuleType):+ def __getattr__(self, name):+ if name.startswith("__"):+ raise AttributeError(name)+ return type(name, (), {})+++def emdb_frames(root: Path, stride: int) -> list[dict]:+ """EMDB-1: frames with `good_frames_mask` set and not in `bboxes.invalid_idxs`, EMDB's boxes."""+ out = []+ for pkl in sorted(root.glob("P*/*/*_data.pkl")):+ with open(pkl, "rb") as f:+ d = pickle.load(f)+ if not d["emdb1"]:+ continue+ good, bad = np.asarray(d["good_frames_mask"], bool), set(np.ravel(d["bboxes"]["invalid_idxs"]).tolist())+ s = d["smpl"]+ for f in range(0, int(d["n_frames"]), stride):+ if good[f] and f not in bad:+ out.append(dict(image=pkl.parent / "images" / f"{f:05d}.jpg", bbox=np.float32(d["bboxes"]["bboxes"][f]),+ K=np.asarray(d["camera"]["intrinsics"]), seq=str(d["name"]),+ gender=str(d["gender"]), betas=np.ravel(s["betas"])[:10],+ pose=np.concatenate([s["poses_root"][f], s["poses_body"][f]]), trans=s["trans"][f],+ extrinsics=np.asarray(d["camera"]["extrinsics"][f])))+ return out+++def threedpw_frames(root: Path, stride: int) -> list[dict]:+ """3DPW test: `campose_valid` people with >= 4 confident 2D keypoints, box = their extent + 20%."""+ out = []+ for pkl in sorted((root / "sequenceFiles" / "test").glob("*.pkl")):+ with open(pkl, "rb") as f:+ seq = pickle.load(f, encoding="latin1")+ for f in range(0, len(seq["poses"][0]), stride):+ for p in range(len(seq["poses"])):+ kp = np.asarray(seq["poses2d"][p][f])+ ok = kp[2] > 0.1+ if not seq["campose_valid"][p][f] or ok.sum() < 4:+ continue+ (x1, y1), (x2, y2) = kp[:2, ok].min(1), kp[:2, ok].max(1)+ w, h = x2 - x1, y2 - y1+ out.append(dict(image=root / "imageFiles" / seq["sequence"] / f"image_{f:05d}.jpg",+ bbox=np.float32([x1 - 0.2 * w, y1 - 0.2 * h, x2 + 0.2 * w, y2 + 0.2 * h]),+ K=np.asarray(seq["cam_intrinsics"]), seq=seq["sequence"],+ gender=str(seq["genders"][p]), betas=np.ravel(seq["betas"][p])[:10],+ pose=seq["poses"][p][f], trans=seq["trans"][p][f], extrinsics=seq["cam_poses"][f]))+ return out+++def ground_truth(samples: list[dict], smpl_dir: Path, device) -> tuple[np.ndarray, np.ndarray]:+ """(N, 24, 3) SMPL joints and (N, 6890, 3) vertices, camera space, metres."""+ models, joints, verts = {}, [], []+ for s in samples:+ g = "MALE" if s["gender"].lower().startswith("m") else "FEMALE"+ if g not in models:+ models[g] = load_smpl(smpl_dir / f"SMPL_{g}.pkl", device)+ m = models[g]+ betas = torch.zeros(m["shapedirs"].shape[-1], device=device)+ betas[:10] = torch.as_tensor(np.float32(s["betas"]), device=device)+ v, j = smpl_forward(m, torch.as_tensor(np.float32(s["pose"])[None], device=device), betas,+ torch.as_tensor(np.float32(s["trans"])[None], device=device))+ E = torch.as_tensor(np.float32(s["extrinsics"]), device=device)+ verts.append((v[0] @ E[:3, :3].T + E[:3, 3]).cpu().numpy())+ joints.append((j[0] @ E[:3, :3].T + E[:3, 3]).cpu().numpy())+ return np.stack(joints), np.stack(verts)+++def predict(samples: list[dict], model: Path, device: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:+ hmr = InstantHMR(model, device=device, max_batch=1)+ params, shapes, trans = [], [], []+ for s in tqdm(samples, desc="InstantHMR", leave=False):+ rgb = cv2.cvtColor(cv2.imread(str(s["image"])), cv2.COLOR_BGR2RGB)+ p = hmr.predict(rgb, [s["bbox"]], focal=float(s["K"][0, 0]))[0]+ params.append(p.mhr_params), shapes.append(p.shape_params), trans.append(p.cam_trans)+ return np.stack(params), np.stack(shapes), np.stack(trans)+++def mhr_vertices(rig, params: np.ndarray, shapes: np.ndarray, device) -> np.ndarray:+ """The full rig's mesh (pose correctives included), body-centred, metres, camera axes."""+ out = []+ with torch.no_grad():+ for i in range(0, len(params), 256):+ p, s = (torch.as_tensor(a[i:i + 256], device=device) for a in (params, shapes))+ v, _ = rig(s, p, torch.zeros(len(p), 72, device=device), True)+ out.append((v * torch.tensor([0.01, -0.01, -0.01], device=device)).cpu().numpy())+ return np.concatenate(out)+++def official_smpl_fit(verts: np.ndarray, cam_trans: np.ndarray, faces: np.ndarray, mhr_repo: Path, smpl_dir: Path,+ device) -> tuple[np.ndarray, np.ndarray]:+ """Meta's MHR -> SMPL conversion, called unchanged: (N, 24, 3) joints and (N, 6890, 3) vertices.++ It only reads the rig's faces from the `mhr` package and imports `pymomentum.solver` without using+ it for this direction, so both are replaced by stand-ins and neither needs installing.+ """+ import smplx+ from smplx.utils import Struct++ _install_chumpy_stub()+ with open(smpl_dir / "SMPL_NEUTRAL.pkl", "rb") as f:+ neutral = pickle.load(f, encoding="latin1")+ neutral = Struct(**{k: _to_numpy(neutral[k]) for k in ("v_template", "shapedirs", "posedirs", "J_regressor",+ "weights", "f", "kintree_table")})+ neutral.f, neutral.kintree_table = neutral.f.astype(np.int64), neutral.kintree_table.astype(np.int64)+ rig = _FacesOnlyRig(faces)+ mhr_module = types.ModuleType("mhr.mhr")+ mhr_module.MHR = _FacesOnlyRig+ sys.modules.update({"mhr": types.ModuleType("mhr"), "mhr.mhr": mhr_module,+ "pymomentum": types.ModuleType("pymomentum"),+ "pymomentum.solver": _AnyName("pymomentum.solver")})+ tool = mhr_repo / "tools" / "mhr_smpl_conversion"+ sys.path.insert(0, str(tool))+ cwd = os.getcwd()+ os.chdir(tool)+ try:+ from conversion import Conversion++ smpl = smplx.SMPL(model_path=str(smpl_dir), data_struct=neutral, gender="neutral").to(device)+ conversion = Conversion(mhr_model=rig, smpl_model=smpl, method="pytorch")+ joints, vertices = [], []+ for i in range(0, len(verts), 512):+ items = [dict(mhr_model_params=np.zeros(1, np.float32), shape_params=np.zeros(1, np.float32),+ expr_params=np.zeros(1, np.float32), pred_vertices=verts[k], pred_cam_t=cam_trans[k])+ for k in range(i, min(i + 512, len(verts)))]+ res = conversion.convert_sam3d_output_to_smpl(items, return_smpl_parameters=True)+ p = {k: torch.as_tensor(v, device=device).float() for k, v in res.result_parameters.items()}+ with torch.no_grad():+ o = smpl(betas=p["betas"], body_pose=p["body_pose"], global_orient=p["global_orient"],+ transl=p["transl"])+ joints.append(o.joints[:, :24].cpu().numpy()), vertices.append(o.vertices.cpu().numpy())+ finally:+ os.chdir(cwd)+ return np.concatenate(joints), np.concatenate(vertices)+++def score(joint_set: str, jp, vp, jg, vg, h36m_regressor) -> dict:+ """PA-MPJPE, MPJPE and PVE in mm. "24": SMPL joints rooted at the mid-hip (joints 1, 2);+ "14": the H36M-regressed joints rooted at their pelvis. PVE uses the same root as MPJPE."""+ vp, vg = vp.astype(np.float64) * MM, vg.astype(np.float64) * MM+ if joint_set == "24":+ jp, jg = jp.astype(np.float64) * MM, jg.astype(np.float64) * MM+ rp, rg = jp[:, 1:3].mean(1), jg[:, 1:3].mean(1)+ else:+ jp = np.einsum("jv,nvc->njc", h36m_regressor, vp)[:, J.J14_FROM_H36M17]+ jg = np.einsum("jv,nvc->njc", h36m_regressor, vg)[:, J.J14_FROM_H36M17]+ rp, rg = J.pelvis(jp), J.pelvis(jg)+ return dict(PA_MPJPE=float(M.pa_mpjpe(jp, jg).mean()), MPJPE=float(M.mpjpe(jp, jg, root=(rp, rg)).mean()),+ PVE=float(M.mpjpe(vp, vg, root=(rp, rg)).mean()))+++def main() -> None:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--dataset", required=True, choices=("emdb", "3dpw"))+ ap.add_argument("--data", type=Path, required=True, help="EMDB root (P0/, P1/, ...) or 3DPW root "+ "(sequenceFiles/, imageFiles/)")+ ap.add_argument("--model", default="medium", help="model name or .onnx path")+ ap.add_argument("--smpl-dir", type=Path, required=True,+ help="SMPL_MALE.pkl, SMPL_FEMALE.pkl, SMPL_NEUTRAL.pkl, J_regressor_h36m.npy")+ ap.add_argument("--mhr-repo", type=Path, required=True, help="clone of github.com/facebookresearch/MHR")+ ap.add_argument("--mhr-model", type=Path, default=HERE.parent / "training" / "checkpoints" / "mhr_model.pt")+ ap.add_argument("--stride", type=int, default=10)+ ap.add_argument("--limit", type=int, help="score only the first N frames (a quick check)")+ ap.add_argument("--device", default="cuda", choices=("cuda", "cpu"))+ ap.add_argument("--save", type=Path, help="also save the SMPL predictions (.npz)")+ args = ap.parse_args()++ samples = (emdb_frames if args.dataset == "emdb" else threedpw_frames)(args.data, args.stride)[:args.limit]+ print(f"{args.dataset}: {len(samples)} frames")+ params, shapes, cam_trans = predict(samples, resolve_model(args.model), args.device)+ rig = torch.jit.load(str(args.mhr_model), map_location=args.device).eval()+ verts = mhr_vertices(rig, params, shapes, args.device)+ faces = rig.character_torch.mesh.faces.cpu().numpy()+ jp, vp = official_smpl_fit(verts, cam_trans, faces, args.mhr_repo, args.smpl_dir, args.device)+ jg, vg = ground_truth(samples, args.smpl_dir, args.device)+ h36m = np.load(args.smpl_dir / "J_regressor_h36m.npy").astype(np.float64)++ published = "24" if args.dataset == "emdb" else "14"+ results = {js: score(js, jp, vp, jg, vg, h36m) for js in ("24", "14")}+ for js, r in results.items():+ mark = " <- paper" if js == published else ""+ print(f"J{js}: PA-MPJPE {r['PA_MPJPE']:.1f} MPJPE {r['MPJPE']:.1f} PVE {r['PVE']:.1f} mm{mark}")+ if args.save:+ np.savez(args.save, joints=jp, vertices=vp, mhr_params=params, shape_params=shapes, cam_trans=cam_trans,+ results=json.dumps(results))+++if __name__ == "__main__":+ main()diff --git a/benchmark/requirements.txt b/benchmark/requirements.txtnew file mode 100644index 0000000..ee277ca--- /dev/null+++ b/benchmark/requirements.txt@@ -0,0 +1,5 @@+torch+smplx+trimesh+scikit-learn+tqdmdiff --git a/datasets_pipeline/README.md b/datasets_pipeline/README.mdindex 400dc4b..e0bd172 100644--- a/datasets_pipeline/README.md+++ b/datasets_pipeline/README.md@@ -1,62 +1,44 @@# datasets_pipeline-Rebuild any split of the SAM-3D-Body corpus from scratch, keeping the original-frames this time, into the layout the training code already understands.+Builds the training data from the released annotations of+[SAM 3D Body](https://huggingface.co/datasets/facebook/sam-3d-body-dataset) and the original+images of each dataset. One folder per dataset:```-data/sam3d_gt_<dataset>/-├── original_images/ referenced source frames, native relative paths-├── annotations/ <name>.npz-├── body_crops/ <name>.png 224 square person crop-├── hands_crops/ <name>_l.png <name>_r.png-└── images -> body_crops compat symlink+<data root>/sam3d_gt_<dataset>/+├── annotations/ <name>.npz MHR parameters, camera, 2D and 3D keypoints of one person+└── images/ <name>.png the person crop```-`body_crops` is **byte-identical** to what `tools/parquet_to_npz.py` writes into-`images/` — verified md5-for-md5 on 120 AIC crops against the existing-`data/sam3d_gt_aic/`. `build_split.py` imports that module's crop and annotation-builders rather than re-implementing them, so the two can never drift. The-`images` symlink means `SAM3DStudentDataset` finds these folders with **no code-change**; that path is tested, not assumed.--## The scripts-| script | does ||---|---|-| `fetch_annotations.py` | parquet shards from the gated HF repo |-| `probe_split.py` | **is this split worth its download?** — from annotations alone |-| `fetch_images.py` | source frames, per dataset |-| `build_split.py` | parquet + frames → the folder above |-| `verify_split.py` | integrity, reprojection, and how much hand signal a split carries |--Plus `plan_disk.py` (cost a split before downloading it), `stream_archives.py`-(build a dataset larger than your disk), `registry.py` (every URL and gotcha in-one place), and `undistort.md`.+| `fetch_annotations.py` | downloads the annotation shards (gated: accept the terms on Hugging Face first) |+| `probe_split.py` | tells you what a split contains, from its annotations alone |+| `plan_disk.py` | how much disk a split will take |+| `fetch_images.py` | downloads the source images, or prints how to get them |+| `build_split.py` | annotations + images -> the folder above |+| `verify_split.py` | checks a built split |-Annotations are ~1 GB per split; frames are tens to thousands of GB. So the-order is always: fetch annotations → `probe_split.py` → decide → fetch frames.+`registry.py` holds every URL and per-dataset detail; `stream_archives.py` builds a dataset larger+than your disk; `undistort.md` covers the datasets whose frames must be undistorted first.## Quick start-```bash-export ANN_DIR=/scratch/sam3d_ann-export IMG_ROOT=/scratch/sam3d_img--python datasets_pipeline/fetch_annotations.py --list-python datasets_pipeline/fetch_annotations.py --splits coco_train --out $ANN_DIR-python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/coco_train--python datasets_pipeline/fetch_images.py --dataset coco --dest $IMG_ROOT/coco-python datasets_pipeline/build_split.py \- --split coco_train \- --annotation-dir $ANN_DIR/coco_train \- --image-dir $IMG_ROOT/coco \- --data-root data --validate+The released models were trained on a **window corpus**: for each person, four square crops cut+from the full image (the standard 1.2x box crop and three other framings), stored side by side at+288 px. COCO as an example:-python datasets_pipeline/verify_split.py --dir data/sam3d_gt_coco+```bash+python datasets_pipeline/fetch_annotations.py --splits coco_train --out annotations+python datasets_pipeline/fetch_images.py --dataset coco --dest images/coco+python datasets_pipeline/build_split.py --split coco_train \+ --annotation-dir annotations/coco_train --image-dir images/coco \+ --windows 4 --window-sizes 288 --no-hands --output-dir 'data_win{size}/sam3d_gt_coco'```-Everything is resume-safe: re-running skips crops already on disk.+Then train with `--data_root data_win288` (see [`training/README.md`](../training/README.md)).+Without `--windows`, `build_split.py` writes single crops into `data/sam3d_gt_<dataset>/`, which+the trainer also reads. Every script resumes where it stopped.## What each dataset costs, and how to get itdiff --git a/demo.py b/demo.pyindex 5c8fd4a..0b94ae4 100755--- a/demo.py+++ b/demo.py@@ -1,666 +1,100 @@#!/usr/bin/env python3-"""InstantHMR — demo runner.+"""InstantHMR demo: people in an image, a video or a live camera, shown in the Rerun viewer.-Detects persons in a video / image / live camera stream with RF-DETR, runs-InstantHMR on each, and visualises the results in Rerun: 3D joints, camera-pose as a pinhole frustum, the source image with the projected 2D skeleton-drawn on top, and live latency plots for RF-DETR, InstantHMR, MHR mesh-rendering, and total pipeline cost.--Optionally renders a full MHR body mesh per person by decoding the-``mhr_params`` and ``shape_params`` outputs through Meta's MHR model, either-from the TorchScript rig used by the distillation (``--mhr-model``) or from the-unpacked asset folder (``--mhr-assets``). See ``instanthmr/mhr_renderer.py``.--MHR-only checkpoints----------------------A model exported from ``train.py`` has FOUR ONNX outputs — it-has no 3D coordinate head, so its 70 keypoints are derived from the MHR forward-pass, exactly like the SAM3D teacher does. Such a model therefore *requires* an-MHR rig; when none is requested explicitly the demo falls back to-``checkpoints/mhr_model.pt``.--Usage------- # Single image (skeleton only; needs a 5-output model)- python demo.py --image path/to/photo.jpg-- # Single image with the MHR mesh, from the TorchScript rig- python demo.py --image path/to/photo.jpg --mhr-model-- # MHR-only student: the rig is mandatory and auto-selected- python demo.py --video vid1.mp4 --model models/instanthmr_mhr_only.onnx-- # Same, but decoding the mesh from the asset folder at a chosen LOD- python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets --mhr-lod 3-- # Video file- python demo.py --video path/to/clip.mp4-- # Video file, with the saved SAM 3D Body teacher overlaid for comparison- # (bare flag reads reference/<video stem>.npz)- python demo.py --video vid1.mp4 --mhr-model --show-reference-- # Live camera (index 0)- python demo.py --camera 0-- # Foot-contact model (--prev-token checkpoint): each person is fed their own- # prediction from 5 frames earlier, and planted feet are drawn red, with a- # per-foot contact-probability plot. --no-prev withholds the previous pose.- python demo.py --video clip.mp4 --model models/instanthmr_g11Ap_b4_ep119.onnx+ python demo.py --image photo.jpg+ python demo.py --video clip.mp4 --smooth+ python demo.py --camera 0 --model small --device tensorrt"""-from __future__ import annotationsimport argparse-from collections import dequeimport sysimport time-from pathlib import Pathimport cv2import numpy as np-from instanthmr import PosePipeline, onnx_output_names-from instanthmr.mhr_renderer import DEFAULT_MHR_SCRIPT-from instanthmr.reference import ReferenceTrack, resolve_reference+from instanthmr import DEFAULT_MODEL, MODELS, PosePipelinefrom instanthmr.smoothing import TemporalSmootherfrom instanthmr.visualizer import RerunVisualizer-DEFAULT_MODEL = "models/instanthmr.onnx"--def parse_args() -> argparse.Namespace:- p = argparse.ArgumentParser(- description="Run InstantHMR (RF-DETR + ONNX HMR) on a video, image, or camera.",- formatter_class=argparse.RawDescriptionHelpFormatter,- )-+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)src = p.add_mutually_exclusive_group(required=True)- src.add_argument("--video", type=str, help="Path to an input video file.")- src.add_argument("--image", type=str, help="Path to an input image.")- src.add_argument(- "--camera", type=int, metavar="IDX",- help="Live camera index (0, 1, ...).",- )-- p.add_argument(- "--model", type=str, default=DEFAULT_MODEL,- help=f"Path to the InstantHMR ONNX file (default: {DEFAULT_MODEL}).",- )- p.add_argument(- "--device", type=str, default="cuda",- help="Inference device for InstantHMR: 'cuda', 'coreml', or 'cpu' (default: cuda).",- )- p.add_argument(- "--detector-variant", type=str, default="medium",- choices=["nano", "small", "medium", "base", "large"],- help="RF-DETR size variant (default: medium).",- )- p.add_argument(- "--det-confidence", type=float, default=0.5,- help="RF-DETR detection confidence threshold (default: 0.5).",- )- p.add_argument(- "--max-persons", type=int, default=2,- help="Max persons processed per frame (default: 2).",- )- p.add_argument(- "--frame-skip", type=int, default=1,- help="Process every Nth video frame (default: 1).",- )- p.add_argument(- "--detector-stride", type=int, default=1,- help="Run RF-DETR every Nth processed frame; reuse the previous bbox in "- "between (default: 1). The detector is the dominant cost — stride "- "2-3 typically doubles end-to-end FPS with negligible quality loss.",- )- p.add_argument(- "--no-batch-persons", action="store_true",- help="Disable batched multi-person HMR (one ONNX call per person).",- )- p.add_argument(- "--focal", type=float, default=None,- help="Camera focal length in PIXELS, when it is known. A graph exported "- "from a --cliff-focal checkpoint is conditioned on angles, so it "- "needs one; without it the demo falls back to 1.05x the image "- "diagonal, which mis-places the person in depth on any camera "- "whose true focal differs.",- )- p.add_argument(- "--no-prev", action="store_true",- help="Prev-token models only: withhold the previous pose on every frame "- "(the single-frame model). By default each tracked person is fed "- "their own prediction from --prev-lag frames earlier.",- )- p.add_argument(- "--prev-lag", type=int, default=5, metavar="N",- help="Processed frames between a prediction and the frame it feeds "- "(default: 5, as in the benchmarks; the token was trained on pairs "- "5-20 frames apart).",- )- p.add_argument(- "--contact-threshold", type=float, default=0.5, metavar="P",- help="Prev-token models only: a foot is drawn planted when its contact "- "probability exceeds P (default: 0.5). On held-out jumps, "- "0.8 gets both feet off in 92%% of flight frames instead of 83%%, "- "and keeps 98%% (instead of 99%%) of both-planted frames.",- )- p.add_argument(- "--smoothing-filter", action="store_true",- help="Temporally smooth per-person predictions with a moving-average "- "filter (see instanthmr/smoothing.py). Non-learned and free; NLF "- "reports a 5-frame filter of this kind giving 3DPW 59.0 -> 57.2 mm.",- )- p.add_argument(- "--smoothing-window", type=int, default=5, metavar="N",- help="Frames averaged by --smoothing-filter (default: 5). Rounded up to "- "the next odd number in centered mode.",- )- p.add_argument(- "--smoothing-mode", choices=("centered", "causal", "1euro"),- default="centered",- help="'centered' = moving average, no phase lag, delays output by "- "(N-1)//2 frames — the filter NLF measured, so use it for benchmark "- "numbers. 'causal' = trailing average, zero latency but real phase "- "lag. '1euro' = adaptive cutoff (Casiez et al. 2012), zero latency "- "and no lag on fast motion — use it for demos. --camera always uses "- "1euro (default: centered).",- )- p.add_argument(- "--smoothing-beta", type=float, default=1.0, metavar="S",- help="Scales every per-field beta in --smoothing-mode 1euro (default: "- "1.0). The jitter-versus-lag knob: raise it if motion still lags, "- "lower it if the pose looks noisy while you hold still.",- )- p.add_argument(- "--smoothing-mincutoff", type=float, default=1.0, metavar="S",- help="Scales every per-field mincutoff in --smoothing-mode 1euro "- "(default: 1.0). Governs how hard the pose is smoothed at rest.",- )- p.add_argument(- "--save-rrd", type=str, default=None,- help="Optional path to save the Rerun recording as a .rrd file.",- )- p.add_argument(- "--no-spawn", action="store_true",- help="Don't spawn the Rerun viewer GUI (useful when only saving .rrd).",- )- p.add_argument(- "--mhr-model", type=str, nargs="?", default=None, const=DEFAULT_MHR_SCRIPT,- metavar="PT",- help=(- "Decode the body mesh with the TorchScript MHR rig used by the "- "distillation (bare flag = checkpoints/mhr_model.pt). Needs only "- "PyTorch, so it works regardless of the pymomentum build. Takes "- "precedence over --mhr-assets."- ),- )- p.add_argument(- "--mhr-assets", type=str, default=None, metavar="DIR",- help=(- "Path to the unpacked MHR assets folder. When provided, runs a "- "full MHR forward pass per person and renders the body mesh in "- "the 3D scene. See instanthmr/mhr_renderer.py for setup."- ),- )- p.add_argument(- "--mhr-lod", type=int, default=3, metavar="N",- choices=range(7),- help=(- "MHR level-of-detail for --mhr-assets (0=73 639 verts … 6=595 "- "verts). Default: 3 (4 899 verts). The TorchScript rig has a fixed "- "LOD (18 439 verts) and ignores this."- ),- )- p.add_argument(- "--show-reference", type=str, nargs="?", default=None, const="",- metavar="NPZ",- help=(- "Overlay a saved SAM 3D Body (teacher) prediction for this clip, "- "so you can compare it against the student in the same 3D scene. "- "Bare flag = reference/<video stem>.npz. The reference stores MHR "- "parameters and is decoded by the same rig as the student, so any "- "difference you see is the model, not the renderer. Build one with "- "tools/pack_reference.py — see instanthmr/reference.py."- ),- )- p.add_argument(- "--mesh-alpha", type=float, default=0.35, metavar="A",- help=(- "Body-mesh opacity in [0, 1] (default: 0.35). Applied as a Rerun "- "albedo factor so the 70 keypoints stay visible inside the body; "- "raise it towards 1.0 for a solid mesh."- ),- )-+ src.add_argument("--image", help="image file")+ src.add_argument("--video", help="video file")+ src.add_argument("--camera", type=int, metavar="INDEX", help="live camera, e.g. 0")+ p.add_argument("--model", default=DEFAULT_MODEL,+ help=f"{', '.join(MODELS)}, or a path to an .onnx file (default: {DEFAULT_MODEL})")+ p.add_argument("--device", default="cuda", choices=("cuda", "tensorrt", "cpu", "coreml"))+ p.add_argument("--max-persons", type=int, default=4)+ p.add_argument("--detector", default="medium", choices=("nano", "small", "medium", "base", "large"),+ help="RF-DETR size (default: medium)")+ p.add_argument("--det-confidence", type=float, default=0.5)+ p.add_argument("--detector-stride", type=int, default=1, metavar="N",+ help="run the detector every N frames and reuse its boxes in between")+ p.add_argument("--focal", type=float, help="camera focal length in pixels, if known")+ p.add_argument("--smooth", action="store_true", help="temporal smoothing (1 Euro filter)")+ p.add_argument("--no-mesh", action="store_true", help="show the skeleton only")+ p.add_argument("--mesh-alpha", type=float, default=0.35, help="mesh opacity (default: 0.35)")+ p.add_argument("--save-rrd", metavar="PATH", help="also save the Rerun recording")+ p.add_argument("--no-viewer", action="store_true", help="do not open the Rerun viewer")return p.parse_args()-def resolve_model_path(args: argparse.Namespace) -> Path:- model_path = Path(args.model)- if not model_path.exists():- sys.stderr.write(- f"\n[error] InstantHMR ONNX not found: {model_path}\n"- " Place 'instanthmr.onnx' under models/, or pass\n"- " --model /path/to/instanthmr.onnx\n\n"- )- sys.exit(1)- return model_path---def build_pipeline(args: argparse.Namespace, mhr=None) -> PosePipeline:- return PosePipeline(- onnx_path=resolve_model_path(args),- device=args.device,- detector_variant=args.detector_variant,- det_confidence=args.det_confidence,- max_persons=args.max_persons,- detector_stride=args.detector_stride,- batch_persons=not args.no_batch_persons,- mhr=mhr,- focal=args.focal,- use_prev=not args.no_prev,- prev_lag=args.prev_lag,- )---def _model_needs_mhr(model_path: Path) -> bool:- """True when the graph has no ``joints_3d`` output (MHR-only student).-- Cheap protobuf peek, so the decision is made before any session or detector- is loaded. An unreadable graph is treated as "does not need MHR" — the- pipeline then raises a precise error of its own.- """- names = onnx_output_names(model_path)- return names is not None and "joints_3d" not in names---def build_mhr(args: argparse.Namespace, required: bool = False):- """Build the MHR rig requested on the command line.-- ``--mhr-model`` (TorchScript) wins over ``--mhr-assets`` (pip package).- When *required* — i.e. the ONNX has no 3D head — and neither flag was given,- fall back to the TorchScript rig that shipped with the training code.- """- script_path = args.mhr_model- assets_folder = args.mhr_assets-- if script_path is None and assets_folder is None:- if not required:- return None- script_path = DEFAULT_MHR_SCRIPT- print(- f"[info] {Path(args.model).name} has no 'joints_3d' output "- "(MHR-only student): its 70 keypoints are derived from the MHR "- f"forward pass.\n Using {script_path}; override with "- "--mhr-model / --mhr-assets."- )-- if script_path is not None:- path = Path(script_path)- if not path.exists():- sys.stderr.write(- f"\n[error] MHR TorchScript rig not found: {path}\n"- " This is the ~700 MB mhr_model.pt the distillation loads with\n"- " torch.jit.load; see training/checkpoints/README.md.\n"- " Alternatively use --mhr-assets with the unpacked MHR asset folder.\n\n"- )- sys.exit(1)- from instanthmr.mhr_renderer import MHRScriptRenderer-- print(f"Loading MHR body model (TorchScript) from {path} …")- return MHRScriptRenderer(path, device=args.device)-- assets_path = Path(assets_folder)- if not assets_path.exists():- sys.stderr.write(- f"\n[error] MHR assets folder not found: {assets_path}\n"- " See instanthmr/mhr_renderer.py for download instructions.\n\n"- )- sys.exit(1)-- try:- from instanthmr.mhr_renderer import MHRRenderer- except ImportError as e:- sys.stderr.write(- f"\n[error] Cannot import MHRRenderer: {e}\n"- "\n"- " MHR mesh rendering requires Python >= 3.12 and:\n"- " pip install -r requirements-mhr.txt\n"- "\n"- " If you see 'No module named pymomentum', make sure you installed\n"- " the CORRECT package (NOT the legacy pyMomentum SMS library):\n"- " pip uninstall pymomentum # remove wrong package\n"- " pip install pymomentum-gpu # NVIDIA GPU\n"- " pip install pymomentum-cpu # macOS / CPU-only\n"- "\n"- " Or skip it entirely and use --mhr-model, which needs only torch.\n"- "\n"- " See README.md §'MHR body mesh' for full setup instructions.\n\n"- )- sys.exit(1)-- print(f"Loading MHR body model (LOD {args.mhr_lod}) from {assets_path} …")- return MHRRenderer(- assets_folder=assets_path,- device=args.device,- lod=args.mhr_lod,- )---def build_all(args: argparse.Namespace):- """Build the MHR rig, the pipeline and the visualizer, in that order.-- The rig comes first because an MHR-only graph cannot produce 3D keypoints- without it — the pipeline takes it as a constructor argument rather than- the visualizer alone. It is returned as well so the temporal filter can- re-derive keypoints from filtered pose parameters.- """- model_path = resolve_model_path(args)- mhr = build_mhr(args, required=_model_needs_mhr(model_path))- pipeline = build_pipeline(args, mhr=mhr)- if pipeline.hmr.has_prev_input:- print(f"Prev-token model: previous pose "- f"{f'fed back from {args.prev_lag} frames earlier' if pipeline.feeds_prev else 'withheld (--no-prev)'}; "- f"foot contact shown in red.", flush=True)- viz = RerunVisualizer(- application_id="instanthmr_demo",- spawn_viewer=not args.no_spawn,- save_path=args.save_rrd,- mhr_renderer=mhr,- mesh_alpha=args.mesh_alpha,- reference=build_reference(args),- contact_threshold=args.contact_threshold,- )- return pipeline, viz, mhr---def build_reference(args: argparse.Namespace):- """Load the saved teacher track for ``--show-reference``, or ``None``."""- path = resolve_reference(args.show_reference, getattr(args, "video", None))- if path is None:- return None- if not path.exists():- sys.exit(- f"[error] reference not found: {path}\n"- " Build it by running the teacher once, then packing it:\n"- " MOMENTUM_ENABLED=0 python main.py --video_path <clip> \\\n"- " --output_path reference/<stem> --save_meshes\n"- " python tools/pack_reference.py reference/<stem>/meshes "- "reference/<stem>.npz"- )- track = ReferenceTrack(path)- print(f"reference: {path} — {len(track)} frames, {track.num_persons} persons")- return track---def _print_timings(- prefix: str,- detector_ms: float,- hmr_ms: float,- render_ms: float,- total_ms: float,- n_persons: int,-) -> None:- fps = 1000.0 / total_ms if total_ms > 0 else 0.0- render_part = f" MHR={render_ms:6.1f} ms" if render_ms > 0.0 else ""- print(- f"{prefix} "- f"persons={n_persons:>2d} "- f"RF-DETR={detector_ms:6.1f} ms "- f"InstantHMR={hmr_ms:6.1f} ms"- f"{render_part} "- f"total={total_ms:6.1f} ms ({fps:5.1f} fps)"- )---# ----------------------------------------------------------------------------# Source-specific runners-# ------------------------------------------------------------------------------def build_smoother(args: argparse.Namespace, live: bool = False, mhr=None):- """Construct the temporal filter, or ``None`` when --smoothing-filter is off.-- ``live`` forces the adaptive filter: a camera cannot buffer future frames,- and a trailing average would show visible phase lag on fast motion.-- Handing it the MHR backend lets it re-derive the 3D keypoints from the- filtered pose parameters, so they stay welded to the mesh instead of- drifting off it during fast motion.- """- if not args.smoothing_filter:- return None- mode = "1euro" if live else args.smoothing_mode- kp_fn = mhr.keypoints if getattr(mhr, "has_keypoints", False) else None- sm = TemporalSmoother(- window=args.smoothing_window,- mode=mode,- beta_scale=args.smoothing_beta,- mincutoff_scale=args.smoothing_mincutoff,- keypoint_fn=kp_fn,- )- if kp_fn is None:- print(" note: no MHR keypoint backend — keypoints are filtered on their "- "own, so they may drift off the mesh during fast motion.")- if sm.mode == "1euro":- print(f"temporal smoothing: 1euro, beta x{sm.beta_scale:g}, "- f"mincutoff x{sm.mincutoff_scale:g}, no latency")- else:- lag = f", {sm.lag} frame lag" if sm.lag else ""- print(f"temporal smoothing: window={sm.window}, mode={sm.mode}{lag}")- return sm---def run_image(args: argparse.Namespace) -> None:- image_path = Path(args.image)- if not image_path.exists():- sys.exit(f"[error] image not found: {image_path}")-- bgr = cv2.imread(str(image_path))- if bgr is None:- sys.exit(f"[error] cv2.imread failed for {image_path}")- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)-- pipeline, viz, _ = build_all(args)-- result = pipeline.predict(rgb)-- render_ms = viz.log_frame(- rgb, result.persons,- frame_idx=0, timestamp=0.0,- detector_ms=result.detector_ms,- hmr_ms=result.hmr_ms,- total_ms=result.total_ms,- )- _print_timings(- "image:",- result.detector_ms, result.hmr_ms, render_ms, result.total_ms,- len(result.persons),- )---def run_video(args: argparse.Namespace) -> None:- video_path = Path(args.video)- if not video_path.exists():- sys.exit(f"[error] video not found: {video_path}")-- cap = cv2.VideoCapture(str(video_path))+def frames(args):+ """(index, timestamp in s, RGB frame) from the chosen source."""+ if args.image:+ bgr = cv2.imread(args.image)+ if bgr is None:+ sys.exit(f"cannot read image {args.image}")+ yield 0, 0.0, cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)+ return+ cap = cv2.VideoCapture(args.video if args.video else args.camera)if not cap.isOpened():- sys.exit(f"[error] cv2.VideoCapture failed for {video_path}")-+ sys.exit(f"cannot open {args.video or f'camera {args.camera}'}")fps = cap.get(cv2.CAP_PROP_FPS) or 30.0- total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))- print(f"video: {video_path.name}, {total} frames @ {fps:.1f} fps")-- pipeline, viz, mhr = build_all(args)- smoother = build_smoother(args, mhr=mhr)-- frame_idx = 0- sent = 0- det_times: list[float] = []- hmr_times: list[float] = []- render_times: list[float] = []- tot_times: list[float] = []-- # When smoothing is centred the filter emits frame t only after it has seen- # t + (window-1)//2, so the frames still in flight wait here with the data- # the visualizer will need for them.- pending: deque[tuple[int, np.ndarray, float, float, float]] = deque()-- def emit(persons) -> None:- f_idx, f_rgb, d_ms, h_ms, t_ms = pending.popleft()- render_times.append(viz.log_frame(- f_rgb, persons,- frame_idx=f_idx,- timestamp=f_idx / fps,- detector_ms=d_ms,- hmr_ms=h_ms,- total_ms=t_ms,- ))-+ start, i = time.perf_counter(), 0try:while True:- ret, bgr = cap.read()- if not ret:- break-- if frame_idx % args.frame_skip != 0:- frame_idx += 1- continue-- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)-- result = pipeline.predict(rgb)- det_times.append(result.detector_ms)- hmr_times.append(result.hmr_ms)- tot_times.append(result.total_ms)-- pending.append((frame_idx, rgb, result.detector_ms,- result.hmr_ms, result.total_ms))- if smoother is None:- emit(result.persons)- else:- smoothed = smoother.push(result.persons, timestamp=frame_idx / fps)- if smoothed is not None:- emit(smoothed)-- sent += 1- frame_idx += 1-- if sent % 30 == 0:- _print_timings(- f" frame {frame_idx:>5d}/{total:<5d}",- float(np.mean(det_times[-30:])),- float(np.mean(hmr_times[-30:])),- float(np.mean(render_times[-30:])),- float(np.mean(tot_times[-30:])),- len(result.persons),- )- if smoother is not None:- for smoothed in smoother.flush():- emit(smoothed)- except KeyboardInterrupt:- print("\ninterrupted")- finally:- cap.release()-- if smoother is not None:- report = smoother.jitter_report()- if report:- print("\n" + report)-- if tot_times:- # Skip the first frame from the average — it bears the warm-up cost.- warm = slice(1, None) if len(tot_times) > 1 else slice(None)- render_avg = float(np.mean(render_times[warm])) if render_times else 0.0- render_part = f" MHR={render_avg:.1f} ms" if render_avg > 0 else ""- print(- "\nsummary "- f"frames={sent:d} "- f"RF-DETR={float(np.mean(det_times[warm])):.1f} ms "- f"InstantHMR={float(np.mean(hmr_times[warm])):.1f} ms"- f"{render_part} "- f"total={float(np.mean(tot_times[warm])):.1f} ms "- f"({1000.0 / float(np.mean(tot_times[warm])):.1f} fps)"- )---def run_camera(args: argparse.Namespace) -> None:- cap = cv2.VideoCapture(args.camera)- if not cap.isOpened():- sys.exit(f"[error] cannot open camera index {args.camera}")-- width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))- height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))- fps = cap.get(cv2.CAP_PROP_FPS) or 30.0- print(f"camera {args.camera}: {width}x{height} @ {fps:.1f} fps — Ctrl+C to stop")-- pipeline, viz, mhr = build_all(args)- smoother = build_smoother(args, live=True, mhr=mhr)-- frame_idx = 0- det_times: list[float] = []- hmr_times: list[float] = []- render_times: list[float] = []- tot_times: list[float] = []- try:- while True:- ret, bgr = cap.read()- if not ret:- print("[warn] camera read failed")- break-- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)- result = pipeline.predict(rgb)- det_times.append(result.detector_ms)- hmr_times.append(result.hmr_ms)- tot_times.append(result.total_ms)-- persons = result.persons- if smoother is not None:- # Wall clock, not frame_idx/fps: a webcam's real frame interval- # drifts and drops, and the adaptive cutoff is a function of- # speed, so it needs the interval that actually elapsed.- persons = smoother.push(- persons, timestamp=time.perf_counter()) or persons-- render_ms = viz.log_frame(- rgb, persons,- frame_idx=frame_idx,- timestamp=frame_idx / fps,- detector_ms=result.detector_ms,- hmr_ms=result.hmr_ms,- total_ms=result.total_ms,- )- render_times.append(render_ms)-- if frame_idx and frame_idx % 30 == 0:- _print_timings(- f" frame {frame_idx:>5d}",- float(np.mean(det_times[-30:])),- float(np.mean(hmr_times[-30:])),- float(np.mean(render_times[-30:])),- float(np.mean(tot_times[-30:])),- len(result.persons),- )- frame_idx += 1- except KeyboardInterrupt:- print("\nstopped")+ ok, bgr = cap.read()+ if not ok:+ return+ t = i / fps if args.video else time.perf_counter() - start+ yield i, t, cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)+ i += 1finally:cap.release()def main() -> None:args = parse_args()- if args.image is not None:- run_image(args)- elif args.video is not None:- run_video(args)- else:- run_camera(args)+ print(f"loading InstantHMR ({args.model}) on {args.device} and RF-DETR {args.detector}...")+ pipeline = PosePipeline(args.model, device=args.device, detector=args.detector,+ det_confidence=args.det_confidence, max_persons=args.max_persons,+ detector_stride=args.detector_stride, focal=args.focal)+ print(f"InstantHMR runs on {pipeline.hmr.provider}")+ viewer = RerunVisualizer(None if args.no_mesh else pipeline.hmr.faces, args.mesh_alpha,+ spawn=not args.no_viewer, save_path=args.save_rrd)+ smoother = TemporalSmoother() if args.smooth else None++ history = []+ try:+ for i, t, rgb in frames(args):+ result = pipeline(rgb)+ persons = smoother(result.persons, t) if smoother else result.persons+ timings = {"detector": result.detector_ms, "InstantHMR": result.hmr_ms}+ viewer.log(i, rgb, persons, timings)+ history.append((result.detector_ms, result.hmr_ms))+ if i % 30 == 0:+ print(f"frame {i:5d} {len(persons)} person(s) detector {result.detector_ms:5.1f} ms "+ f"InstantHMR {result.hmr_ms:5.1f} ms")+ except KeyboardInterrupt:+ pass+ if len(history) > 1:+ det, hmr = np.median(np.array(history[1:]), axis=0)+ print(f"median over {len(history) - 1} frames: detector {det:.1f} ms, InstantHMR {hmr:.1f} ms")if __name__ == "__main__":diff --git a/docs/annotation.md b/docs/annotation.mddeleted file mode 100644index 60eae44..0000000--- a/docs/annotation.md+++ /dev/null@@ -1,120 +0,0 @@-# Building a training dataset by teacher annotation (optional path)--> **This page describes the OPTIONAL distillation path.** The default — and what-> every shipped model and published number in this repo actually used — is the-> released ground-truth annotations of-> [`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset),-> laid out by `datasets_pipeline/build_split.py` into `data/sam3d_gt_<dataset>/`.-> See `datasets_pipeline/README.md`.->-> Use this page when you need labels for imagery the released dataset does not-> cover. The cost is a ceiling: a model trained on teacher output cannot exceed-> the teacher, while one trained on the ground truth can.--On this path InstantHMR is supervised by the **SAM3D teacher**-(`facebook/sam-3d-body-dinov3`): for every detector-found person we run-the teacher and store its outputs alongside a tight 224×224 crop. The-training loop then learns to mimic those teacher outputs from the crop alone.-This is the only thing that writes `data/sam3d_distill_mix/`.--The bundled `tools/annotate_dataset.py` uses YOLO for the-detector during annotation. That choice is independent of the runtime-detector — InstantHMR sees the same 1.2× square crop either way, and the-demo uses RF-DETR at inference time for better recall. If you re-run the-annotation with a different detector, just preserve the crop geometry.--The annotation pipeline is `tools/annotate_dataset.py`.--## What it does--```-root_dir/ output_dir/-├── COCO/ ─┐ ├── images/-├── Harmony4D/ │ YOLO + SAM3D │ COCO_<stem>_p0.png (224×224)-├── Synthetic/ │ ──────────────► │ COCO_<stem>_p0.npz-└── … ─┘ │ …- └── annotations/- COCO_<stem>_p0.npz- …-```--For each detected person (capped at `--max_persons` per image):--1. **YOLO 11n** detects persons (`conf ≥ --confidence`, default 0.7).-2. The **full-resolution** image is passed to SAM3D with the raw bbox.- SAM3D's CLIFF camera head sees the full frame, so `pred_cam_t` and- `focal_length` are produced in real perspective coordinates — no manual- camera math.-3. A 1.2× **square** crop around the bbox is resized to 224×224 and stored- as a PNG.-4. The teacher outputs are stored as a `.npz` next to the image:-- | Key | Shape | Description |- |-----|-------|-------------|- | `orig_shape` | `(2,)` | `[H, W]` of the original frame |- | `bbox` | `(4,)` | raw YOLO `[x1, y1, x2, y2]` |- | `bbox_square` | `(4,)` | the 1.2× expanded square crop bounds |- | `cam_focal_length` | `(2,)` | `[fx, fy]` (≈ √(H²+W²)) |- | `cam_trans` | `(3,)` | SAM3D translation in camera frame |- | `mhr_model_params` | `(204,)` | MHR pose + scale |- | `shape_params` | `(45,)` | MHR identity blendshapes |- | `joints_3d` | `(70, 3)` | SAM3D `pred_keypoints_3d` |- | `joints_2d` | `(70, 2)` | SAM3D `pred_keypoints_2d`, in **full-frame** pixels |--The script is **resume-safe**: it scans existing `.npz` filenames at startup-and skips any image whose first crop is already on disk.--## Running it--```bash-# Setup: clone Meta's sam-3d-body and download the dinov3 teacher checkpoint.-git clone https://github.com/facebookresearch/sam-3d-body-huggingface-cli download facebook/sam-3d-body-dinov3 \- --local-dir checkpoints/sam-3d-body-dinov3--# Annotate a tree of images-python tools/annotate_dataset.py \- --root_dir /data/training_images \- --output_dir /data/cliff_annotations \- --confidence 0.7--# Quick balanced subset (50k crops, shuffled across datasets)-python tools/annotate_dataset.py \- --root_dir /data/training_images \- --output_dir /data/cliff_annotations \- --shuffle --max_samples 50000--# Visually inspect what's getting saved-python tools/annotate_dataset.py \- --root_dir /data/training_images \- --output_dir /data/cliff_annotations \- --use_rerun-```--Useful options:--- `--shuffle` — process images in random order so a partial run still spans- the full dataset diversity.-- `--max_samples N` — stop after N **crops** are saved (not images), counting- whatever was already on disk.-- `--timeout 5.0` — per-image SIGALRM; rare GPU hangs don't kill the run.-- `--gc_interval 200` — periodic CUDA cache + GC reset to keep VRAM stable- across long runs.--## Why a separate cloud script?--This is the same per-person preprocessing the demo uses at inference time-(`InstantHMR._preprocess`). Aligning the training data with the runtime-preprocessing was the single biggest factor in getting InstantHMR to match-SAM3D's accuracy: any drift between annotation crops and inference crops-shows up directly as test-time error.--## Datasets we used--InstantHMR was trained on a mix of:--- **COCO** (in-the-wild humans),-- **Harmony4D** (multi-person studio captures),-- a **synthetic dataset** of rendered humans, and-- domain-specific footage (rugby, aikido) that the model is meant to handle.-diff --git a/install.py b/install.pydeleted file mode 100644index 01f8e38..0000000--- a/install.py+++ /dev/null@@ -1,159 +0,0 @@-#!/usr/bin/env python3-"""Platform-aware installer for InstantHMR.--Picks the right Torch + ONNX Runtime wheels for the current machine:-- - macOS (Apple Silicon / Intel) — stock ``torch`` + ``onnxruntime``- (which bundles the CoreML execution provider).- - Linux + NVIDIA GPU — Torch built for the matching CUDA toolkit- (cu128 for Blackwell sm_120+, cu124 for Ada / Ampere / Turing) plus- ``onnxruntime-gpu`` and the ``nvidia-*-cu12`` runtime wheels so a- system CUDA install isn't required.- - Linux without an NVIDIA GPU — Torch CPU wheel + ``onnxruntime``.--Run *after* you've created and activated a fresh Python environment::-- python install.py--For a dry run that just prints the commands::-- python install.py --dry-run-"""--from __future__ import annotations--import argparse-import platform-import subprocess-import sys-from typing import Optional---def run_pip(args: list[str], dry_run: bool = False) -> None:- cmd = [sys.executable, "-m", "pip", "install"] + args- print(">>", " ".join(cmd))- if not dry_run:- subprocess.check_call(cmd)---def detect_cuda_compute_cap() -> Optional[float]:- """Return the highest CUDA compute capability across visible GPUs.-- Returns ``None`` if ``nvidia-smi`` is missing, fails, or reports no- GPU.- """- try:- out = subprocess.check_output(- ["nvidia-smi", "--query-gpu=compute_cap", "--format=csv,noheader"],- text=True, stderr=subprocess.DEVNULL,- )- except (FileNotFoundError, subprocess.CalledProcessError):- return None- caps: list[float] = []- for line in out.splitlines():- line = line.strip()- if not line:- continue- try:- caps.append(float(line))- except ValueError:- pass- return max(caps) if caps else None---def install_mac(dry_run: bool) -> None:- print("[install] platform: macOS — using stock torch + onnxruntime (CoreML EP)")- run_pip(["torch>=2.2", "torchvision"], dry_run)- run_pip(["onnxruntime>=1.19"], dry_run)---def install_linux_cpu(dry_run: bool) -> None:- print("[install] platform: Linux without NVIDIA GPU — using CPU wheels")- run_pip([- "--index-url", "https://download.pytorch.org/whl/cpu",- "torch>=2.2", "torchvision",- ], dry_run)- run_pip(["onnxruntime>=1.19"], dry_run)---def install_linux_cuda(compute_cap: float, dry_run: bool) -> None:- print(f"[install] platform: Linux + NVIDIA (compute capability {compute_cap})")- if compute_cap >= 12.0:- # Blackwell (RTX 50-series, B100, B200): needs sm_120 kernels which- # only land natively in the cu128 wheels (Torch 2.7+).- print("[install] → Blackwell detected, installing torch cu128 (>=2.7)")- run_pip([- "--index-url", "https://download.pytorch.org/whl/cu128",- "torch>=2.7", "torchvision",- ], dry_run)- elif compute_cap >= 8.0:- # Ampere (sm_80/86), Ada (sm_89), Hopper (sm_90). cu124 wheels cover- # all of these with native kernels.- print("[install] → Ampere/Ada/Hopper, installing torch cu124 (>=2.4)")- run_pip([- "--index-url", "https://download.pytorch.org/whl/cu124",- "torch>=2.4", "torchvision",- ], dry_run)- else:- # Turing (sm_75) and older — cu121 still has stable wheels here.- print("[install] → pre-Ampere, installing torch cu121 (>=2.2)")- run_pip([- "--index-url", "https://download.pytorch.org/whl/cu121",- "torch>=2.2", "torchvision",- ], dry_run)-- run_pip(["onnxruntime-gpu>=1.19"], dry_run)- # Bundled CUDA runtime libs so users without a system CUDA toolkit- # can still load CUDA EP. ORT's preload_dlls() picks them up.- run_pip([- "nvidia-cudnn-cu12>=9.0,<10",- "nvidia-cuda-runtime-cu12",- "nvidia-cublas-cu12",- ], dry_run)---def install_common(dry_run: bool) -> None:- print("[install] common deps")- run_pip([- "numpy>=1.24",- "opencv-python>=4.8",- "pillow>=10",- "rfdetr>=1.0",- "rerun-sdk>=0.21",- ], dry_run)---def main() -> None:- parser = argparse.ArgumentParser(description=__doc__,- formatter_class=argparse.RawDescriptionHelpFormatter)- parser.add_argument("--dry-run", action="store_true",- help="Print pip commands but don't run them.")- parser.add_argument("--force-cpu", action="store_true",- help="Force the CPU-only path on Linux even if an NVIDIA GPU is detected.")- args = parser.parse_args()-- sysname = platform.system()- machine = platform.machine()- print(f"[install] python={platform.python_version()} system={sysname} machine={machine}")-- if sysname == "Darwin":- install_mac(args.dry_run)- elif sysname == "Linux":- if args.force_cpu:- install_linux_cpu(args.dry_run)- else:- cap = detect_cuda_compute_cap()- if cap is None:- install_linux_cpu(args.dry_run)- else:- install_linux_cuda(cap, args.dry_run)- else:- sys.exit(f"[install] unsupported platform: {sysname}. "- f"Install torch + onnxruntime manually, then `pip install -r requirements.txt`.")-- install_common(args.dry_run)- print("\n[install] done. verify with: python -c 'from instanthmr import PosePipeline; print(\"ok\")'")---if __name__ == "__main__":- main()diff --git a/instanthmr/__init__.py b/instanthmr/__init__.pyindex 1fce43e..4cb11df 100644--- a/instanthmr/__init__.py+++ b/instanthmr/__init__.py@@ -1,21 +1,9 @@-"""InstantHMR — standalone 3D human pose inference + Rerun visualization."""+"""InstantHMR: real-time 3D human mesh recovery from a single image."""+from .inference import HMRPrediction, InstantHMR+from .models import DEFAULT_MODEL, MODELS, resolve_model+from .pipeline import FrameResult, PosePipeline+from .skeleton import JOINT_NAMES, SKELETON_EDGES-from .inference import InstantHMR, HMRPrediction, onnx_output_names-from .detector import RFDETRDetector-from .pipeline import PosePipeline, FrameResult-from .skeleton import JOINT_NAMES, SKELETON_EDGES, NUM_JOINTS, edges_for--__all__ = [- "InstantHMR",- "HMRPrediction",- "onnx_output_names",- "RFDETRDetector",- "PosePipeline",- "FrameResult",- "JOINT_NAMES",- "SKELETON_EDGES",- "NUM_JOINTS",- "edges_for",-]--__version__ = "0.1.0"+__all__ = ["InstantHMR", "HMRPrediction", "PosePipeline", "FrameResult", "MODELS", "DEFAULT_MODEL",+ "resolve_model", "JOINT_NAMES", "SKELETON_EDGES"]+__version__ = "1.0.0"diff --git a/instanthmr/detector.py b/instanthmr/detector.pyindex 7d16364..062a0cf 100644--- a/instanthmr/detector.py+++ b/instanthmr/detector.py@@ -1,104 +1,37 @@-"""RF-DETR person detector.--Wraps Roboflow's RF-DETR (medium variant by default) with a minimal API:-``detect(image_rgb)`` returns a list of-``{"bbox": [x1, y1, x2, y2], "confidence": float}`` dicts sorted by-descending confidence.--RF-DETR is a transformer-based detector with strong COCO recall on small-and partially-occluded persons — a good match for upstream pose-estimation, which is sensitive to bbox localisation.-"""-+"""RF-DETR person detector."""from __future__ import annotations-import numpy as np---COCO_PERSON_CLASS_ID = 1 # RF-DETR exposes COCO 1-indexed class ids+import logging+import warnings+import numpy as np-class RFDETRDetector:- """Person detector backed by RF-DETR.+VARIANTS = {"nano": "RFDETRNano", "small": "RFDETRSmall", "medium": "RFDETRMedium",+ "base": "RFDETRBase", "large": "RFDETRLarge"}+COCO_PERSON = 1- Args:- variant: Which RF-DETR variant to load. One of- ``"nano"``, ``"small"``, ``"medium"``, ``"base"``, ``"large"``.- Default ``"medium"`` balances speed and accuracy.- confidence: Minimum detection score.- max_persons: Maximum number of persons returned per frame.- optimize_for_inference: Call RF-DETR's TorchScript / fused-op- optimisation pass on construction. Adds a few seconds of- startup but speeds up subsequent ``detect`` calls noticeably.- """- _VARIANTS = {- "nano": "RFDETRNano",- "small": "RFDETRSmall",- "medium": "RFDETRMedium",- "base": "RFDETRBase",- "large": "RFDETRLarge",- }+class PersonDetector:+ """Returns `(boxes (N, 4) x1 y1 x2 y2, scores (N,))` for the people in an RGB frame, best first."""- def __init__(- self,- variant: str = "medium",- confidence: float = 0.5,- max_persons: int = 5,- optimize_for_inference: bool = True,- ):+ def __init__(self, variant: str = "medium", confidence: float = 0.5, max_persons: int = 4):import rfdetr- if variant not in self._VARIANTS:- raise ValueError(- f"Unknown RF-DETR variant {variant!r}. "- f"Choose one of {list(self._VARIANTS)}."- )-- cls = getattr(rfdetr, self._VARIANTS[variant])- self._model = cls()- if optimize_for_inference:- self._model.optimize_for_inference()-- self._confidence = confidence- self._max_persons = max_persons- self._variant = variant-- def warmup(self) -> None:- """Run one silent forward pass to pre-compile PyTorch/CUDA kernels."""- dummy = np.zeros((224, 224, 3), dtype=np.uint8)- self.detect(dummy)-- @property- def variant(self) -> str:- return self._variant-- def detect(self, image_rgb: np.ndarray) -> list[dict]:- """Detect persons in an RGB image.-- Args:- image_rgb: ``(H, W, 3)`` uint8 RGB array.-- Returns:- List of detection dicts sorted by descending confidence,- truncated to ``max_persons``.- """- from PIL import Image as PILImage-- pil_image = PILImage.fromarray(image_rgb)- sv_detections = self._model.predict(pil_image, threshold=self._confidence)-- detections: list[dict] = []- if len(sv_detections) > 0 and sv_detections.class_id is not None:- mask = sv_detections.class_id == COCO_PERSON_CLASS_ID- for bbox, conf in zip(- sv_detections.xyxy[mask],- sv_detections.confidence[mask],- ):- detections.append({- "bbox": np.asarray(bbox, dtype=np.float32),- "confidence": float(conf),- })-- detections.sort(key=lambda d: d["confidence"], reverse=True)- return detections[: self._max_persons]+ logging.getLogger("rf-detr").setLevel(logging.ERROR)+ if variant not in VARIANTS:+ raise ValueError(f"unknown RF-DETR variant {variant!r}: choose one of {', '.join(VARIANTS)}")+ self.model = getattr(rfdetr, VARIANTS[variant])()+ if hasattr(self.model, "optimize_for_inference"): # rfdetr < 1.10+ with warnings.catch_warnings():+ warnings.simplefilter("ignore")+ self.model.optimize_for_inference()+ self.confidence, self.max_persons = confidence, max_persons++ def __call__(self, image_rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray]:+ from PIL import Image++ det = self.model.predict(Image.fromarray(image_rgb), threshold=self.confidence)+ keep = det.class_id == COCO_PERSON if det.class_id is not None else np.zeros(len(det), bool)+ boxes, scores = det.xyxy[keep].astype(np.float32), det.confidence[keep].astype(np.float32)+ order = np.argsort(-scores)[: self.max_persons]+ return boxes[order], scores[order]diff --git a/instanthmr/inference.py b/instanthmr/inference.pyindex 493cea4..673096f 100644--- a/instanthmr/inference.py+++ b/instanthmr/inference.py@@ -1,581 +1,183 @@-"""ONNX inference for InstantHMR.--InstantHMR takes a square person crop (224 px unless the ONNX metadata-says otherwise) and a 3-vector CLIFF condition-(bbox center / scale in full-frame coords) and returns mhr_params, shape,-camera translation, and 70 joints in 2D (crop space) and 3D (camera coords,-rig-local, metres, Y-down).--This module wraps the ONNX session and the per-person preprocessing — square-crop, CLIFF cond, ImageNet normalisation, and re-projection of joints back-into full-frame pixels.--Two graph layouts are supported, and the session tells them apart by itself:-- * **5 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d, joints_3d)``- — older students with their own 3D coordinate head (earlier trainers,- not in this repository).- * **4 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d)`` — the- MHR-only student (``train.py``) has no 3D head, exactly- like the SAM3D teacher. Its 70 keypoints are derived here by running the- MHR skeleton forward on the predicted parameters, so such a model needs an- MHR backend (see :mod:`instanthmr.mhr_renderer`) passed as ``mhr=``.-"""-+"""InstantHMR inference with onnxruntime: person crops in, MHR pose, 3D keypoints and mesh out."""from __future__ import annotations+import base64import mathfrom dataclasses import dataclassfrom pathlib import Path-from typing import Optional+from typing import Sequenceimport cv2import numpy as np-IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)-INPUT_SIZE = 224-CROP_EXPAND = 1.2 # square crop around the detector bbox, matching training-# Focal used when the camera is uncalibrated, as a multiple of the image-# diagonal. A focal-aware checkpoint needs a number here whatever happens ---# the conditioning is an ANGLE, so there is no "unknown" to pass. 1.05x diag-# (~50 degrees horizontal FOV on 16:9) is the usual stand-in for a phone or-# webcam; pass the real value through `focal=` whenever it is known.-#-# It applies to focal-aware graphs ONLY. A pixel-conditioned checkpoint was-# trained on a corpus whose annotations set f = sqrt(H^2+W^2) exactly (see-# tools/annotate_dataset.py), so 1.0x diag IS that model's camera and using-# anything else would put a 5% error into its rendered depth.+CROP_EXPAND = 1.2+# Focal used when the camera is not calibrated, as a multiple of the image diagonal+# (about 50 degrees of horizontal field of view on a 16:9 frame).FOCAL_FALLBACK_DIAG = 1.05---def onnx_output_names(onnx_path: str | Path) -> Optional[list[str]]:- """Read a graph's output names without creating an inference session.-- Lets a caller know whether it must supply an MHR backend *before* paying for- session creation and detector loading. Returns ``None`` when the ``onnx``- package is unavailable or the file cannot be parsed — callers should then- fall back to letting :class:`InstantHMR` decide.- """- try:- import onnx-- model = onnx.load(str(onnx_path), load_external_data=False)- return [o.name for o in model.graph.output]- except Exception: # noqa: BLE001 — purely advisory- return None+DEVICES = ("cuda", "tensorrt", "cpu", "coreml")@dataclassclass HMRPrediction:- """Per-person outputs from one InstantHMR forward pass.-- All numpy, all in the FULL-frame coordinate system.-- Attributes:- bbox: (4,) raw detector bbox [x1, y1, x2, y2].- confidence: detection confidence in [0, 1].- joints_3d_local: (70, 3) rig-local joints, metres, Y-down. Either read- straight from the model's 3D head, or regressed from the MHR- skeleton when the graph has no such head.- joints_3d_cam: (70, 3) joints in camera space (= local + cam_trans).- joints_2d: (70, 2) joints projected into full-frame pixel coords.- cam_trans: (3,) camera translation [tx, ty, tz], metres.- focal_length: (2,) [fx, fy] full-frame focal in pixels -- the camera- this prediction is expressed in. The real focal when one was- supplied, else the diagonal stand-in ``_focal_px`` picks.- principal_point: (2,) [cx, cy] = full-frame centre.- image_shape: (H, W) of the source frame.- mhr_params: (204,) MHR pose parameters (34 joints × 6-D rotation).- Pass to ``instanthmr.mhr_renderer.MHRRenderer.forward()``- together with ``shape_params`` to obtain a full body mesh.- shape_params: (45,) MHR identity blend-shape coefficients- (20 body + 20 head + 5 hand). Pair with ``mhr_params`` and- feed both into ``MHRRenderer.forward()``.- """+ """One person. Pixel coordinates are in the full frame; 3D is in metres, camera axes+ (x right, y down, z forward), with `*_local` centred on the body and `*_cam` in camera space."""- bbox: np.ndarray+ bbox: np.ndarray # (4,) x1, y1, x2, y2confidence: float- joints_3d_local: np.ndarray- joints_3d_cam: np.ndarray- joints_2d: np.ndarray- cam_trans: np.ndarray- focal_length: np.ndarray- principal_point: np.ndarray- image_shape: tuple[int, int]- mhr_params: np.ndarray- shape_params: np.ndarray- contact_prob: Optional[np.ndarray] = None # (2,) [left, right] foot, prev-token graphs only+ joints_2d: np.ndarray # (70, 2) pixels+ joints_3d_local: np.ndarray # (70, 3)+ cam_trans: np.ndarray # (3,)+ mhr_params: np.ndarray # (204,) MHR model parameters+ shape_params: np.ndarray # (45,) MHR identity coefficients+ focal_length: float # pixels, the camera the 3D is expressed in+ image_shape: tuple[int, int] # (H, W)+ vertices_local: np.ndarray # (18439, 3)- def prev_pose(self) -> np.ndarray:- """This prediction encoded as a prev-token graph's ``prev_pose`` input:- the 70 rig-local keypoints, then ``cam_trans`` as (x/z, y/z, log z)."""- t = self.cam_trans- z = max(float(t[2]), 0.1)- return np.concatenate([self.joints_3d_local.reshape(-1),- [t[0] / z, t[1] / z, math.log(z)]]).astype(np.float32)+ @property+ def joints_3d_cam(self) -> np.ndarray:+ return self.joints_3d_local + self.cam_trans++ @property+ def vertices_cam(self) -> np.ndarray:+ return self.vertices_local + self.cam_trans+++def providers_for(device: str, max_batch: int = 1, image_size: int = 288,+ cache_dir: str | Path = "models/trt_cache") -> list:+ """onnxruntime execution providers for a device name, always ending on the CPU."""+ import onnxruntime as ort++ available = set(ort.get_available_providers())+ if device == "tensorrt":+ try:+ import tensorrt_libs # noqa: F401 (loads libnvinfer for onnxruntime)+ except ImportError as e:+ raise RuntimeError("TensorRT is not installed: pip install tensorrt-cu12==10.15.*") from e+ if "TensorrtExecutionProvider" not in available:+ raise RuntimeError("this onnxruntime build has no TensorRT support: install onnxruntime-gpu")+ Path(cache_dir).mkdir(parents=True, exist_ok=True)+ shape = lambda n: f"image:{n}x3x{image_size}x{image_size},cliff_cond:{n}x3" # noqa: E731+ trt = {"trt_fp16_enable": True, "trt_engine_cache_enable": True, "trt_engine_cache_path": str(cache_dir),+ "trt_timing_cache_enable": True, "trt_profile_min_shapes": shape(1),+ "trt_profile_opt_shapes": shape(max_batch), "trt_profile_max_shapes": shape(max_batch)}+ return [("TensorrtExecutionProvider", trt), "CUDAExecutionProvider", "CPUExecutionProvider"]+ if device == "cuda":+ if "CUDAExecutionProvider" not in available:+ raise RuntimeError("CUDA is not available in this onnxruntime build: install onnxruntime-gpu")+ return ["CUDAExecutionProvider", "CPUExecutionProvider"]+ if device == "coreml" and "CoreMLExecutionProvider" in available:+ return ["CoreMLExecutionProvider", "CPUExecutionProvider"]+ return ["CPUExecutionProvider"]class InstantHMR:- """ONNX wrapper for the InstantHMR model.-- Args:- onnx_path: path to the exported graph.- device: ``"cuda"``, ``"coreml"`` or ``"cpu"``.- providers: explicit onnxruntime execution providers (overrides *device*).- mhr: MHR backend used to derive ``joints_3d`` when the graph has no 3D- head. Any object exposing ``keypoints(mhr_params, shape_params)``- works; see :func:`instanthmr.mhr_renderer.build_mhr`. Ignored for- 5-output graphs.- focal: the camera's focal length in pixels, when it is known. Used by- the focal-aware conditioning and reported back in- ``HMRPrediction.focal_length``. Left unset, an uncalibrated- stand-in of ``FOCAL_FALLBACK_DIAG * sqrt(H^2+W^2)`` is used.- cliff_focal: which conditioning form the checkpoint was trained with.- Read from the ONNX metadata that ``tools/pth_to_onnx.py`` stamps,- so it normally needs no setting; pass it explicitly for a graph- exported before that existed.+ """An exported InstantHMR network and the MHR body graph that turns its output into 3D.- Example- -------- hmr = InstantHMR("models/instanthmr.onnx")- out = hmr.predict(image_rgb, bbox=[x1, y1, x2, y2])- print(out.joints_3d_cam.shape) # (70, 3)+ `device` runs the network ("cuda", "tensorrt", "cpu" or "coreml"); the body graph runs on+ CUDA when the network does, else on the CPU. `focal` is the camera focal length in pixels+ when it is known; otherwise 1.05 x the image diagonal is assumed. `max_batch` is the most+ people predicted in one call (TensorRT builds its engine for batches up to it)."""- def __init__(- self,- onnx_path: str | Path,- device: str = "cuda",- providers: Optional[list[str]] = None,- mhr: object | None = None,- focal: float | None = None,- cliff_focal: bool | None = None,- ):+ def __init__(self, model_path: str | Path, body_path: str | Path | None = None, device: str = "cuda",+ focal: float | None = None, max_batch: int = 4):import onnxruntime as ort- onnx_path = Path(onnx_path)- if not onnx_path.exists():- raise FileNotFoundError(f"ONNX model not found: {onnx_path}")+ from .models import resolve_body- # ORT >= 1.19 ships preload_dlls() which discovers CUDA / cuDNN- # bundled by the nvidia-* pip wheels (or by torch) and adds them to- # the loader's search path. Calling it before session creation is- # the simplest fix for "libcudnn.so.9: cannot open shared object- # file" on machines that don't ship system CUDA.+ if device not in DEVICES:+ raise ValueError(f"device must be one of {DEVICES}")if hasattr(ort, "preload_dlls"):- try:- ort.preload_dlls()- except Exception:- # preload_dlls is best-effort — never let it block init.- pass-- if providers is None:- providers = self._default_providers(device)-- sess_options = ort.SessionOptions()- sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL- sess_options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL- # 0 == let ORT pick. We don't override threading on CUDA (compute- # is on-device); on CPU/CoreML, ORT's defaults are saner than ours.-- self.session = ort.InferenceSession(- str(onnx_path), sess_options=sess_options, providers=providers,- )- self.active_provider = self.session.get_providers()[0]-- in_names = [i.name for i in self.session.get_inputs()]- out_names = [o.name for o in self.session.get_outputs()]- self._in_image = in_names[0] # "image"- self._in_cliff = in_names[1] # "cliff_cond"- # Output order from export: mhr_params, shape_params, cam_trans,- # joints_2d[, joints_3d]. The MHR-only student stops after joints_2d.- self._out_names = out_names- # A --prev-token graph (tools/pth_to_onnx.py) also takes prev_pose and- # has_prev, and returns contact_logits instead of a joints_3d head.- self.has_prev_input = "prev_pose" in in_names- self._prev_dim = self.session.get_inputs()[2].shape[1] if self.has_prev_input else 0- self.has_joints_3d_head = "joints_3d" in out_names or (- len(out_names) >= 5 and "contact_logits" not in out_names)- self._mhr = mhr-- # Two independent things, and conflating them is what made this wrong:- #- # cliff_focal -- WHICH conditioning form the checkpoint was trained- # with. Angles off the optical axis, or image-normalised pixels.- # The vector is 3 floats either way, so feeding a model the wrong- # one raises nothing; it silently mis-places the person in depth.- # tools/pth_to_onnx.py stamps it into the graph metadata, so a- # freshly exported model configures itself.- # focal -- WHAT the camera's focal length is, in pixels. Only used by- # the angular form. Unknown is not an option there, so an- # uncalibrated camera falls back to FOCAL_FALLBACK_DIAG * diagonal.- #- # Passing `focal=` alone still implies the angular form, which is how- # this argument behaved before `cliff_focal` existed.- meta = self.session.get_modelmeta().custom_metadata_map or {}- # The crop side the graph was trained at: stamped by tools/pth_to_onnx.py,- # else the graph's static input height, else the historical 224.- dim = self.session.get_inputs()[0].shape[2]- self.input_size = int(meta.get("image_size")- or (dim if isinstance(dim, int) else INPUT_SIZE))- if cliff_focal is None:- cliff_focal = (str(meta.get("cliff_focal", "")).lower() == "true"- or focal is not None)- self.cliff_focal = bool(cliff_focal)+ ort.preload_dlls()+ ort.set_default_logger_severity(3)+ options = ort.SessionOptions()+ options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL+ options.log_severity_level = 3++ size = 288+ if device == "tensorrt": # the engine profile needs the crop size before the session exists+ probe = ort.InferenceSession(str(model_path), providers=["CPUExecutionProvider"])+ size = int(probe.get_modelmeta().custom_metadata_map.get("image_size", size))+ del probe+ self.network = ort.InferenceSession(str(model_path), sess_options=options,+ providers=providers_for(device, max_batch, size))+ self.provider = self.network.get_providers()[0]+ meta = self.network.get_modelmeta().custom_metadata_map+ self.input_size = int(meta.get("image_size", size))+ self.cliff_focal = meta.get("cliff_focal", "true") == "true"self.focal = focal+ self.max_batch = max_batch- def _focal_px(self, h: int, w: int, focal: float | None = None) -> float:- """The camera focal in pixels: this call's, the session's, or a- stand-in -- in that order."""+ body_path = Path(body_path) if body_path else resolve_body(Path(model_path))+ self.body = ort.InferenceSession(str(body_path), sess_options=options,+ providers=providers_for("cuda" if device in ("cuda", "tensorrt") else "cpu"))+ self.faces = np.frombuffer(base64.b64decode(self.body.get_modelmeta().custom_metadata_map["mesh_faces"]),+ "<i4").reshape(-1, 3)++ def focal_px(self, h: int, w: int, focal: float | None = None) -> float:f = focal if focal is not None else self.focalif f is not None:return float(f)- diag = math.sqrt(h * h + w * w)- return (FOCAL_FALLBACK_DIAG * diag) if self.cliff_focal else diag-- def _feeds(self, crops: np.ndarray, cliffs: np.ndarray,- prev: Optional[list] = None) -> dict:- """Session inputs; for a prev-token graph, ``prev[i]`` is person i's- ``HMRPrediction.prev_pose()`` from an earlier frame, or None = withheld."""- feeds = {self._in_image: crops, self._in_cliff: cliffs}- if self.has_prev_input:- n = crops.shape[0]- pp = np.zeros((n, self._prev_dim), dtype=np.float32)- hp = np.zeros((n,), dtype=np.float32)- for i, v in enumerate((prev or [])[:n]):- if v is not None:- pp[i], hp[i] = v, 1.0- feeds["prev_pose"], feeds["has_prev"] = pp, hp- return feeds-- def _contact(self, outs: list, i: int) -> Optional[np.ndarray]:- if "contact_logits" not in self._out_names:- return None- z = outs[self._out_names.index("contact_logits")][i].astype(np.float32)- return 1.0 / (1.0 + np.exp(-z))-- @property- def derives_joints_3d(self) -> bool:- """True when the 70 keypoints must come from an MHR forward pass."""- return not self.has_joints_3d_head-- def set_mhr(self, mhr: object | None) -> None:- """Attach (or replace) the MHR backend used to derive ``joints_3d``."""- self._mhr = mhr-- def _joints_3d(self, mhr_params: np.ndarray, shape_params: np.ndarray) -> np.ndarray:- """Regress the 70 keypoints from predicted MHR parameters.-- Mirrors ``train.MHRForwardPass.get_native_keypoints``:- skeleton forward, cm -> m, Y/Z flip, then the (70, 127) affine map. The- student was trained through exactly this operator, so any other- keypoint definition would silently shift the predictions.- """- if self._mhr is None:- raise RuntimeError(- "this ONNX graph has no 'joints_3d' output — it comes from "- "train.py, where the 3D keypoints are derived from "- "the MHR forward pass.\n"- "Pass an MHR backend, e.g.\n"- " from instanthmr.mhr_renderer import build_mhr\n"- " InstantHMR(onnx, mhr=build_mhr(script_path='checkpoints/mhr_model.pt'))\n"- "or run demo.py with --mhr-model / --mhr-assets."- )- return np.asarray(- self._mhr.keypoints(mhr_params, shape_params), dtype=np.float32,- )-- # ------------------------------------------------------------------- # Public API- # -------------------------------------------------------------------- def predict(- self,- image_rgb: np.ndarray,- bbox: np.ndarray | list[float],- confidence: float = 1.0,- focal: float | None = None,- prev: Optional[np.ndarray] = None,- ) -> HMRPrediction:- """Run InstantHMR on a single person.-- Args:- image_rgb: (H, W, 3) uint8 RGB full-frame image.- bbox: tight person bbox [x1, y1, x2, y2] in pixel coords.- confidence: detection confidence to attach to the output.- focal: this frame's focal length in pixels, overriding the one- given to the constructor. Use it when the focal varies per- image, as it does across calibrated benchmark sequences.- prev: prev-token graphs only, this person's earlier- ``HMRPrediction.prev_pose()``; None withholds it.-- Returns:- ``HMRPrediction`` with joints in full-frame coordinates.- """- image_rgb = np.ascontiguousarray(image_rgb)- h, w = image_rgb.shape[:2]- bbox_arr = np.asarray(bbox, dtype=np.float32).reshape(4)+ return (FOCAL_FALLBACK_DIAG if self.cliff_focal else 1.0) * math.hypot(h, w)- crop, sq_x1, sq_y1, sq_size, cliff_cond = self._preprocess(- image_rgb, bbox_arr, h, w, focal- )-- outs = self.session.run(- None,- # (1, 3, S, S) with S = self.input_size, and (1, 3)- self._feeds(crop[np.newaxis], cliff_cond[np.newaxis], [prev]),- )- # Order: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]- mhr_params = outs[0][0].astype(np.float32)- shape_params = outs[1][0].astype(np.float32)- cam_trans = outs[2][0].astype(np.float32)- joints_2d_norm = outs[3][0].astype(np.float32) # (70, 2) in [-1, 1]- if self.has_joints_3d_head:- joints_3d_local = outs[4][0].astype(np.float32) # (70, 3) rig-local- else:- joints_3d_local = self._joints_3d(mhr_params, shape_params)-- # Re-project the 2D head from normalised crop space → full-frame pixels.- crop_px = (joints_2d_norm + 1.0) * 0.5 * self.input_size- scale = sq_size / self.input_size- joints_2d = np.stack(- [crop_px[:, 0] * scale + sq_x1, crop_px[:, 1] * scale + sq_y1],- axis=-1,- ).astype(np.float32)-- joints_3d_cam = joints_3d_local + cam_trans-- # Report the focal actually used, so anything that unprojects or- # renders these joints uses the same camera the model was conditioned on.- f = self._focal_px(h, w, focal)- focal_length = np.array([f, f], dtype=np.float32)- principal_point = np.array([w / 2.0, h / 2.0], dtype=np.float32)-- return HMRPrediction(- bbox=bbox_arr,- confidence=float(confidence),- joints_3d_local=joints_3d_local,- joints_3d_cam=joints_3d_cam,- joints_2d=joints_2d,- cam_trans=cam_trans,- focal_length=focal_length,- principal_point=principal_point,- image_shape=(h, w),- mhr_params=mhr_params,- shape_params=shape_params,- contact_prob=self._contact(outs, 0),- )-- def predict_batch(- self,- image_rgb: np.ndarray,- detections: list[dict],- padded_to: int | None = None,- focal: float | None = None,- prev: Optional[list] = None,- ) -> list[HMRPrediction]:- """Run InstantHMR on multiple persons in a single ONNX call.-- Args:- image_rgb: (H, W, 3) uint8 RGB full-frame image.- detections: list of ``{"bbox": [x1,y1,x2,y2], "confidence": f}``.- padded_to: when set, the ONNX batch dimension is always this size- (zero-padded if fewer detections). Keeping a constant batch- size prevents CUDA from re-compiling kernels each time the- person count changes.- focal: this frame's focal length in pixels, overriding the one- given to the constructor.- prev: prev-token graphs only, one earlier- ``HMRPrediction.prev_pose()`` (or None) per detection.-- Returns:- One ``HMRPrediction`` per input detection, in the same order.- """- if not detections:+ def predict(self, image_rgb: np.ndarray, bboxes: Sequence, confidences: Sequence[float] | None = None,+ focal: float | None = None) -> list[HMRPrediction]:+ """All people of one frame in a single call. `bboxes` are tight person boxes, x1 y1 x2 y2."""+ if len(bboxes) == 0:return []-- image_rgb = np.ascontiguousarray(image_rgb)h, w = image_rgb.shape[:2]- n = len(detections)- batch = max(padded_to or n, n) # always >= n; zero-pad the rest-- # Use zeros so padded slots produce deterministic (discarded) outputs.- crops = np.zeros((batch, 3, self.input_size, self.input_size), dtype=np.float32)- cliffs = np.zeros((batch, 3), dtype=np.float32)- sq_meta = [] # per-person (sq_x1, sq_y1, sq_size, bbox)- for i, det in enumerate(detections):- bbox_arr = np.asarray(det["bbox"], dtype=np.float32).reshape(4)- crop, sq_x1, sq_y1, sq_size, cliff = self._preprocess(- image_rgb, bbox_arr, h, w, focal,- )- crops[i] = crop- cliffs[i] = cliff- sq_meta.append((sq_x1, sq_y1, sq_size, bbox_arr))-- outs = self.session.run(None, self._feeds(crops, cliffs, prev))- mhr_params_b = outs[0].astype(np.float32, copy=False)- shape_params_b = outs[1].astype(np.float32, copy=False)- cam_trans_b = outs[2].astype(np.float32, copy=False)- joints_2d_norm_b = outs[3].astype(np.float32, copy=False)- if self.has_joints_3d_head:- joints_3d_local_b = outs[4].astype(np.float32, copy=False)- else:- # One batched MHR skeleton forward for the whole frame (~0.9 ms),- # including the zero-padded slots — they are discarded below.- joints_3d_local_b = self._joints_3d(mhr_params_b, shape_params_b)-- f = self._focal_px(h, w, focal)- focal_length = np.array([f, f], dtype=np.float32)- principal_point = np.array([w / 2.0, h / 2.0], dtype=np.float32)-- results: list[HMRPrediction] = []- for i, det in enumerate(detections):- sq_x1, sq_y1, sq_size, bbox_arr = sq_meta[i]- joints_2d_norm = joints_2d_norm_b[i]- joints_3d_local = joints_3d_local_b[i]- cam_trans = cam_trans_b[i]-- crop_px = (joints_2d_norm + 1.0) * 0.5 * self.input_size- scale = sq_size / self.input_size- joints_2d = np.stack(- [crop_px[:, 0] * scale + sq_x1, crop_px[:, 1] * scale + sq_y1],- axis=-1,- ).astype(np.float32)-+ f = self.focal_px(h, w, focal)+ crops, cliffs, squares = zip(*(self._preprocess(image_rgb, np.asarray(b, np.float32), f) for b in bboxes))+ crops, cliffs = np.stack(crops), np.stack(cliffs)+ chunks = [self._run(crops[i:i + self.max_batch], cliffs[i:i + self.max_batch])+ for i in range(0, len(crops), self.max_batch)]+ outs = {k: np.concatenate([c[k] for c in chunks]) for k in chunks[0]}+ confidences = confidences if confidences is not None else [1.0] * len(bboxes)++ results = []+ for i, (x0, y0, side) in enumerate(squares):+ j2d = (outs["joints_2d"][i] + 1.0) * 0.5 * side + np.array([x0, y0], np.float32)results.append(HMRPrediction(- bbox=bbox_arr,- confidence=float(det.get("confidence", 1.0)),- joints_3d_local=joints_3d_local,- joints_3d_cam=joints_3d_local + cam_trans,- joints_2d=joints_2d,- cam_trans=cam_trans,- focal_length=focal_length,- principal_point=principal_point,- image_shape=(h, w),- mhr_params=mhr_params_b[i],- shape_params=shape_params_b[i],- contact_prob=self._contact(outs, i),- ))+ bbox=np.asarray(bboxes[i], np.float32), confidence=float(confidences[i]),+ joints_2d=j2d.astype(np.float32), joints_3d_local=outs["joints_3d"][i],+ cam_trans=outs["cam_trans"][i], mhr_params=outs["mhr_params"][i],+ shape_params=outs["shape_params"][i], focal_length=f, image_shape=(h, w),+ vertices_local=outs["vertices"][i]))return results- def warmup(self, max_batch_size: int = 1) -> None:- """Pre-warm the ONNX session for the two batch sizes used at runtime.-- With the padded-batch strategy the pipeline only ever calls the session- with batch=1 (single person) or batch=max_batch_size (multi-person,- zero-padded to a fixed size). Two runs here compile all CUDA kernels- upfront so no mid-demo stall occurs when persons first appear.- """- dummy_image = np.zeros(- (max_batch_size, 3, self.input_size, self.input_size), dtype=np.float32- )- dummy_cliff = np.zeros((max_batch_size, 3), dtype=np.float32)- # Large batch first — allocates the most memory; batch=1 reuses the plan.- self.session.run(None, self._feeds(dummy_image, dummy_cliff))- if max_batch_size > 1:- self.session.run(None, self._feeds(dummy_image[:1], dummy_cliff[:1]))-- # ------------------------------------------------------------------- # Preprocessing- # -------------------------------------------------------------------- def _preprocess(- self,- image_rgb: np.ndarray,- bbox: np.ndarray,- h: int,- w: int,- focal: float | None = None,- ) -> tuple[np.ndarray, float, float, float, np.ndarray]:- """Square 1.2x crop, ImageNet normalise, and CLIFF conditioning."""- x1, y1, x2, y2 = bbox.astype(float)- bw = x2 - x1- bh = y2 - y1- cx = (x1 + x2) / 2.0- cy = (y1 + y2) / 2.0-- # CLIFF conditioning vector (full-frame coords). Must match the variant- # the checkpoint was TRAINED with -- see `cliff_focal` in- # training/train.py. A model trained- # with the focal-aware form and run with the pixel form (or the other- # way round) does not error; it silently mis-places the person in depth.+ def _run(self, crops: np.ndarray, cliffs: np.ndarray) -> dict[str, np.ndarray]:+ names = ["mhr_params", "shape_params", "cam_trans", "joints_2d"]+ outs = dict(zip(names, self.network.run(names, {"image": crops, "cliff_cond": cliffs})))+ outs.update(zip(["joints_3d", "vertices"], self.body.run(+ ["joints_3d", "vertices"], {"mhr_params": outs["mhr_params"], "shape_params": outs["shape_params"]})))+ return outs++ def warmup(self, max_batch: int = 1) -> None:+ for n in sorted({1, max_batch}):+ self.network.run(None, {"image": np.zeros((n, 3, self.input_size, self.input_size), np.float32),+ "cliff_cond": np.zeros((n, 3), np.float32)})+ self.body.run(None, {"mhr_params": np.zeros((n, 204), np.float32),+ "shape_params": np.zeros((n, 45), np.float32)})++ def _preprocess(self, image_rgb: np.ndarray, bbox: np.ndarray, focal: float):+ """The square 1.2x crop the model was trained on, and its CLIFF conditioning vector."""+ h, w = image_rgb.shape[:2]+ x1, y1, x2, y2 = bbox.astype(np.float64)+ cx, cy, size = (x1 + x2) / 2, (y1 + y2) / 2, max(x2 - x1, y2 - y1)if self.cliff_focal:- f = self._focal_px(h, w, focal)- cliff_cond = np.array([math.atan((cx - w / 2.0) / f),- math.atan((cy - h / 2.0) / f),- max(bw, bh) / f], dtype=np.float32)+ cliff = [math.atan((cx - w / 2) / focal), math.atan((cy - h / 2) / focal), size / focal]else:- cx_norm = 2.0 * (cx / w) - 1.0- cy_norm = 2.0 * (cy / h) - 1.0- b_scale = max(bw, bh) / max(w, h)- cliff_cond = np.array([cx_norm, cy_norm, b_scale], dtype=np.float32)-- sq_size = max(bw, bh) * CROP_EXPAND- half = sq_size / 2.0- sq_x1 = cx - half- sq_y1 = cy - half-- ix1 = int(math.floor(sq_x1))- iy1 = int(math.floor(sq_y1))- ix2 = int(math.ceil(sq_x1 + sq_size))- iy2 = int(math.ceil(sq_y1 + sq_size))-- pad_left = max(0, -ix1)- pad_top = max(0, -iy1)- pad_right = max(0, ix2 - w)- pad_bottom = max(0, iy2 - h)-- src_x1 = max(0, ix1)- src_y1 = max(0, iy1)- src_x2 = min(w, ix2)- src_y2 = min(h, iy2)-- patch = image_rgb[src_y1:src_y2, src_x1:src_x2]- if pad_left or pad_top or pad_right or pad_bottom:- patch = cv2.copyMakeBorder(- patch, pad_top, pad_bottom, pad_left, pad_right,- cv2.BORDER_CONSTANT, value=(0, 0, 0),- )-- crop_sq = cv2.resize(patch, (self.input_size, self.input_size), interpolation=cv2.INTER_LINEAR)- crop = crop_sq.astype(np.float32) / 255.0- crop = (crop - IMAGENET_MEAN) / IMAGENET_STD- crop = np.transpose(crop, (2, 0, 1)).astype(np.float32) # (3, S, S)-- return crop, float(sq_x1), float(sq_y1), float(sq_size), cliff_cond-- # ------------------------------------------------------------------- # Provider selection- # -------------------------------------------------------------------- @staticmethod- def _default_providers(device: str) -> list[str]:- """Pick ORT execution providers based on the requested *device*.-- Recognised values:- ``"cuda"`` → ``CUDAExecutionProvider`` (NVIDIA).- ``"coreml"`` → ``CoreMLExecutionProvider`` (Apple Silicon / macOS).- ``"cpu"`` / anything else → CPU only.-- ``CPUExecutionProvider`` is always appended as a fallback so a- partially-supported model still loads.-- TensorRT is intentionally **not** requested by default: the wheel- distinction between ``onnxruntime-gpu`` and the ``tensorrt`` system- libraries is fiddly, and asking for an EP whose runtime libraries- aren't installed produces a verbose error before falling back.- Pass ``providers=`` explicitly if you want TensorRT.- """- import onnxruntime as ort-- available = set(ort.get_available_providers())- wanted: list[str] = []- device_l = device.lower()- if "cuda" in device_l and "CUDAExecutionProvider" in available:- wanted.append("CUDAExecutionProvider")- elif "coreml" in device_l and "CoreMLExecutionProvider" in available:- wanted.append("CoreMLExecutionProvider")- wanted.append("CPUExecutionProvider")- return wanted+ cliff = [2 * cx / w - 1, 2 * cy / h - 1, size / max(w, h)]++ side = size * CROP_EXPAND+ sx, sy = cx - side / 2, cy - side / 2+ ix1, iy1 = math.floor(sx), math.floor(sy)+ ix2, iy2 = math.ceil(sx + side), math.ceil(sy + side)+ patch = image_rgb[max(0, iy1):min(h, iy2), max(0, ix1):min(w, ix2)]+ patch = cv2.copyMakeBorder(patch, max(0, -iy1), max(0, iy2 - h), max(0, -ix1), max(0, ix2 - w),+ cv2.BORDER_CONSTANT, value=(0, 0, 0))+ crop = cv2.resize(patch, (self.input_size, self.input_size), interpolation=cv2.INTER_LINEAR)+ crop = ((crop.astype(np.float32) / 255.0 - IMAGENET_MEAN) / IMAGENET_STD).transpose(2, 0, 1)+ return np.ascontiguousarray(crop), np.asarray(cliff, np.float32), (float(sx), float(sy), float(side))diff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.pydeleted file mode 100644index 97b3bb4..0000000--- a/instanthmr/mhr_renderer.py+++ /dev/null@@ -1,407 +0,0 @@-"""MHR (Momentum Human Rig) decoder for InstantHMR.--Takes the 204-dim pose parameters and 45-dim shape parameters produced by-InstantHMR and runs a forward pass through Meta's MHR body model to obtain-- * the full body **mesh** (skinned vertices), and- * the 70 native **keypoints**, regressed from the 127 skeleton joints.--The keypoints matter because ``train.py`` deletes the-student's 3D coordinate head: its ONNX graph has four outputs instead of five-and the 3D pose must be derived here, exactly the way the teacher derives it-from the mesh. The (70, 127) affine map lives in-``training/assets/mhr_j127_to_kp70.npy`` and was fitted against-the teacher's own labels (held-out residual: 1.36 mm mean).--Two interchangeable backends------------------------------``MHRScriptRenderer`` (recommended)- Loads the self-contained TorchScript rig at ``checkpoints/mhr_model.pt`` —- the same artefact the distillation used. Needs nothing but PyTorch, so it- is immune to the ``pymomentum`` / ``torch`` ABI pairing below.--``MHRRenderer``- Loads the unpacked MHR asset folder through the ``mhr`` pip package. This- lets you pick a mesh LOD, but ``pymomentum-gpu`` is a torch C++ extension- and **must** match the installed torch (0.1.113 wants ``torch>=2.8,<2.9``);- a mismatch segfaults the process rather than raising.--Use :func:`build_mhr` to get whichever one the caller configured.--Quick setup for the asset backend (requires Python >= 3.12)::-- pip install -r requirements-mhr.txt- curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip- unzip assets.zip -d models/mhr_assets- python demo.py --image photo.jpg --mhr-assets models/mhr_assets/assets--WARNING — wrong pymomentum package on PyPI--------------------------------------------``pip install pymomentum`` installs an **unrelated** legacy SMS library-(pyMomentum v0.1.x by MomentumAS). The correct Meta package is::-- pip install pymomentum-gpu # NVIDIA GPU- pip install pymomentum-cpu # macOS / CPU-only--See README.md §'MHR body mesh' for the full installation walkthrough.-"""--from __future__ import annotations--from pathlib import Path-from typing import Optional--import numpy as np---_REPO_ROOT = Path(__file__).resolve().parent.parent--#: (70, 127) affine map from MHR skeleton joints to the 70 native keypoints.-DEFAULT_KP_REGRESSOR = _REPO_ROOT / "training/assets/mhr_j127_to_kp70.npy"-#: TorchScript rig shipped with the training code.-DEFAULT_MHR_SCRIPT = _REPO_ROOT / "checkpoints/mhr_model.pt"--NUM_MODEL_PARAMS = 204 # 136 pose + 68 scale-NUM_IDENTITY = 45 # identity blend-shape coefficients-NUM_EXPRESSION = 72 # facial expression coefficients (InstantHMR predicts none)-NUM_SKELETON_JOINTS = 127-NUM_KEYPOINTS = 70---class _MHRBase:- """Parameter marshalling, coordinate conversion, rig evaluation and keypoints.-- Both backends expose the same three objects, so everything below the loader- is shared. A subclass' ``__init__`` must set:-- * ``self._model`` — callable ``(identity, pose, expr) -> (verts, skel)``- * ``self._character`` — exposes ``model_parameters_to_joint_parameters``- and ``joint_parameters_to_skeleton_state``- * ``self._n_params`` — width its parameter transform expects, i.e. 204- model parameters plus that rig's blend-shape slots-- and it must implement the ``faces`` property.- """-- def __init__(self, device: str = "cuda", kp_regressor: str | Path | None = None):- import torch-- self._torch = torch- self._device = self._resolve_device(torch, device)- self._kp_regressor = None-- path = Path(kp_regressor) if kp_regressor is not None else DEFAULT_KP_REGRESSOR- if path.exists():- W = np.load(path).astype(np.float32)- if W.shape != (NUM_KEYPOINTS, NUM_SKELETON_JOINTS):- raise ValueError(- f"expected a ({NUM_KEYPOINTS}, {NUM_SKELETON_JOINTS}) keypoint "- f"regressor in {path}, got {W.shape}"- )- self._kp_regressor = torch.from_numpy(W).to(self._device)- elif kp_regressor is not None:- raise FileNotFoundError(f"keypoint regressor not found: {path}")-- @staticmethod- def _resolve_device(torch, device):- """Map the demo's device string onto a torch device.-- ``--device coreml`` selects an onnxruntime execution provider, not a- torch backend, so anything torch cannot run on falls back to CPU.- """- name = str(device).lower()- if "cuda" in name and torch.cuda.is_available():- return torch.device(device)- if "cuda" in name:- return torch.device("cpu")- try:- return torch.device(device)- except (RuntimeError, TypeError):- return torch.device("cpu")-- # ------------------------------------------------------------------- # Capabilities- # -------------------------------------------------------------------- @property- def has_keypoints(self) -> bool:- """True when the 70-keypoint regressor was found and loaded."""- return self._kp_regressor is not None-- @property- def faces(self) -> np.ndarray:- raise NotImplementedError-- # ------------------------------------------------------------------- # Coordinate conventions- # -------------------------------------------------------------------- @staticmethod- def _to_vision(xyz):- """Raw MHR (centimetres, Y-up / Z-back) -> InstantHMR (metres, Y-down / Z-forward).-- Negating both Y and Z keeps the frame right-handed after the Y flip.- This is the exact transform ``train.MHRForwardPass``- applies before the keypoint regression, so student and demo agree.- """- out = xyz / 100.0- out[..., 1] *= -1.0- out[..., 2] *= -1.0- return out-- # ------------------------------------------------------------------- # Public API- # -------------------------------------------------------------------- def keypoints(- self,- mhr_params: np.ndarray,- shape_params: np.ndarray,- ) -> np.ndarray:- """Regress the 70 native keypoints from the MHR skeleton.-- Skeleton branch only — no blend shapes, no skinning — so this is ~3x- cheaper than a full mesh forward (≈0.9 ms vs ≈2.6 ms at batch 2 on an- RTX 4070).-- Args:- mhr_params: ``(204,)`` or ``(B, 204)`` float32 pose parameters.- shape_params: ``(45,)`` or ``(B, 45)`` — accepted for symmetry with- :meth:`forward`; identity coefficients do not move the skeleton- (they only displace the rest-pose vertices), which is why the- distillation passes zeros here too.-- Returns:- ``(70, 3)`` or ``(B, 70, 3)`` float32 keypoints in the **rig-local**- frame (metres, Y-down) — the frame ``cam_trans`` is relative to.- Add ``cam_trans`` for camera space.- """- if self._kp_regressor is None:- raise RuntimeError(- f"no MHR keypoint regressor loaded (looked for {DEFAULT_KP_REGRESSOR}).\n"- "It ships with the training code; regenerate it with\n"- " python training/train.py --fit-regressor"- )- torch = self._torch- pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)-- with torch.no_grad():- skel = self._skeleton_state(pose)- kps = self._regress(skel)-- out = kps.cpu().numpy().astype(np.float32)- return out[0] if squeeze else out-- def forward(- self,- mhr_params: np.ndarray,- shape_params: np.ndarray,- ) -> np.ndarray:- """Decode one person's parameters into body-space mesh vertices.-- Args:- mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.- shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.-- Returns:- ``(V, 3)`` float32 vertices in the **rig-local** frame (metres,- Y-down). Add the person's ``cam_trans`` to move to camera space.- """- verts, _ = self.forward_full(mhr_params, shape_params)- return verts-- def forward_full(- self,- mhr_params: np.ndarray,- shape_params: np.ndarray,- ) -> tuple[np.ndarray, Optional[np.ndarray]]:- """Mesh vertices **and** the 70 keypoints from a single MHR forward pass.-- Returns ``(verts, keypoints)`` with the same leading shape convention as- :meth:`forward` / :meth:`keypoints`. ``keypoints`` is ``None`` when no- regressor is available.- """- torch = self._torch- pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)- identity, _ = self._as_batch(shape_params, NUM_IDENTITY)- if identity.shape[0] != pose.shape[0]:- identity = identity.expand(pose.shape[0], -1)-- with torch.no_grad():- verts, skel = self._skin(identity, pose)- verts = self._to_vision(verts.float())- kps = self._regress(skel) if self._kp_regressor is not None else None-- v = verts.cpu().numpy().astype(np.float32)- k = kps.cpu().numpy().astype(np.float32) if kps is not None else None- if squeeze:- return v[0], (None if k is None else k[0])- return v, k-- # ------------------------------------------------------------------- # Internals- # -------------------------------------------------------------------- def _as_batch(self, arr: np.ndarray, dim: int):- """numpy ``(dim,)`` / ``(B, dim)`` -> torch ``(B, dim)`` on the rig device."""- torch = self._torch- t = torch.as_tensor(np.ascontiguousarray(arr), dtype=torch.float32)- squeeze = t.dim() == 1- if squeeze:- t = t.unsqueeze(0)- if t.shape[-1] != dim:- raise ValueError(f"expected trailing dimension {dim}, got {tuple(t.shape)}")- return t.to(self._device), squeeze-- def _regress(self, skel):- """``(B, 127, 8)`` skeleton state -> ``(B, 70, 3)`` keypoints (vision frame)."""- joints = self._to_vision(skel[..., :3].float())- return self._torch.einsum("kj,bjc->bkc", self._kp_regressor, joints)-- def _pad(self, pose, total: int):- """Right-pad the 204 model parameters with zero blend-shape coefficients."""- torch = self._torch- extra = total - pose.shape[-1]- if extra <= 0:- return pose- return torch.cat(- [pose, torch.zeros(pose.shape[0], extra, dtype=pose.dtype, device=pose.device)],- dim=1,- )-- def _skeleton_state(self, pose):- """``(B, 204)`` model parameters -> ``(B, 127, 8)`` raw skeleton state.-- Skips blend shapes and skinning entirely: only the two rigid-transform- stages the keypoint regression needs.- """- ch = self._character- joint_params = ch.model_parameters_to_joint_parameters(self._pad(pose, self._n_params))- return ch.joint_parameters_to_skeleton_state(joint_params)-- def _skin(self, identity, pose):- """Full rig evaluation -> ``((B, V, 3) verts, (B, 127, 8) skeleton state)``.-- InstantHMR predicts no facial expression, so those coefficients are zero.- """- torch = self._torch- expr = torch.zeros(- pose.shape[0], NUM_EXPRESSION, dtype=pose.dtype, device=pose.device,- )- verts, skel = self._model(identity, pose, expr)- return verts, skel---class MHRScriptRenderer(_MHRBase):- """MHR rig loaded from the TorchScript artefact used by the distillation.-- Args:- script_path: path to ``mhr_model.pt`` (default: ``checkpoints/mhr_model.pt``).- device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.- kp_regressor: optional override for the (70, 127) keypoint map.-- The mesh LOD is baked into the artefact (18 439 vertices / 36 874 faces);- use :class:`MHRRenderer` if you need to choose one.- """-- def __init__(- self,- script_path: str | Path = DEFAULT_MHR_SCRIPT,- device: str = "cuda",- kp_regressor: str | Path | None = None,- ) -> None:- super().__init__(device=device, kp_regressor=kp_regressor)- torch = self._torch-- script_path = Path(script_path)- if not script_path.exists():- raise FileNotFoundError(f"MHR TorchScript rig not found: {script_path}")-- self._model = torch.jit.load(str(script_path), map_location=self._device).eval()- for p in self._model.parameters():- p.requires_grad_(False)-- self._character = self._model.character_torch- # 204 model parameters + 45 identity blend-shape slots in this rig's- # parameter transform; read it off rather than assuming.- self._n_params = len(self._character.parameter_transform.parameter_names)-- faces = self._character.mesh.faces- self._faces = faces.cpu().numpy().astype(np.int32)-- @property- def faces(self) -> np.ndarray:- """Triangle face indices ``(F, 3)`` int32, shared across all frames."""- return self._faces----class MHRRenderer(_MHRBase):- """MHR rig loaded from the unpacked asset folder via the ``mhr`` pip package.-- Args:- assets_folder: path to the folder that holds ``lod*.fbx`` and- ``compact_v6_1.model`` (i.e. the ``assets/`` directory inside the- unpacked ``assets.zip``).- device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.- lod: level-of-detail mesh resolution:- 0 = 73 639 verts (highest), 3 = 4 899, 6 = 595 (lowest).- LOD 3 is a good default for live demos.- kp_regressor: optional override for the (70, 127) keypoint map.-- Note:- ``pymomentum-gpu`` is a torch C++ extension. If its build does not match- the installed torch (0.1.113 requires ``torch>=2.8,<2.9``) the process- segfaults inside the native library instead of raising — prefer- :class:`MHRScriptRenderer` when in doubt.- """-- def __init__(- self,- assets_folder: str | Path = "models/mhr_assets",- device: str = "cuda",- lod: int = 3,- kp_regressor: str | Path | None = None,- ) -> None:- super().__init__(device=device, kp_regressor=kp_regressor)- from mhr.mhr import MHR-- folder = Path(assets_folder)- # assets.zip unpacks to <dir>/assets/ — accept either level.- if not (folder / f"lod{lod}.fbx").exists() and (folder / "assets" / f"lod{lod}.fbx").exists():- folder = folder / "assets"-- self._model = MHR.from_files(folder=folder, device=self._device, lod=lod)- self._character = self._model.character_torch- self._n_params = self._model.character.parameter_transform.size-- faces_raw = self._model.character.mesh.faces- if hasattr(faces_raw, "cpu"):- self._faces: np.ndarray = faces_raw.cpu().numpy().astype(np.int32)- else:- self._faces = np.asarray(faces_raw, dtype=np.int32)-- @property- def faces(self) -> np.ndarray:- """Triangle face indices ``(F, 3)`` int32, shared across all frames."""- return self._faces----def build_mhr(- script_path: str | Path | None = None,- assets_folder: str | Path | None = None,- device: str = "cuda",- lod: int = 3,- kp_regressor: str | Path | None = None,-) -> _MHRBase | None:- """Build whichever MHR backend the caller configured.-- ``script_path`` wins when both are given. Returns ``None`` when neither is- set, so callers can treat "no MHR" as a plain absence.- """- if script_path is not None:- return MHRScriptRenderer(script_path, device=device, kp_regressor=kp_regressor)- if assets_folder is not None:- return MHRRenderer(assets_folder, device=device, lod=lod, kp_regressor=kp_regressor)- return Nonediff --git a/instanthmr/models.py b/instanthmr/models.pynew file mode 100644index 0000000..95bccb3--- /dev/null+++ b/instanthmr/models.py@@ -0,0 +1,40 @@+"""The released InstantHMR models and their download from the Hugging Face Hub."""+from __future__ import annotations++from pathlib import Path++REPO_ID = "momolesang/InstantHMR"++MODELS = {+ "nano": "instanthmr-nano.onnx",+ "tiny": "instanthmr-tiny.onnx",+ "small": "instanthmr-small.onnx",+ "medium": "instanthmr-medium.onnx",+ "medium-commercial": "instanthmr-medium-commercial.onnx",+}+DEFAULT_MODEL = "medium"+BODY = "mhr-body.onnx"+++def _download(filename: str) -> Path:+ from huggingface_hub import hf_hub_download++ return Path(hf_hub_download(REPO_ID, filename))+++def resolve_model(model: str | Path) -> Path:+ """A local `.onnx` path, or a model name downloaded once into the Hugging Face cache."""+ path = Path(model)+ if path.suffix == ".onnx":+ if not path.exists():+ raise FileNotFoundError(f"model not found: {path}")+ return path+ if str(model) not in MODELS:+ raise ValueError(f"unknown model {model!r}: choose one of {', '.join(MODELS)} or pass an .onnx path")+ return _download(MODELS[str(model)])+++def resolve_body(model_path: Path) -> Path:+ """The MHR body graph: next to a local model if there is one, else downloaded."""+ local = Path(model_path).with_name(BODY)+ return local if local.exists() else _download(BODY)diff --git a/instanthmr/pipeline.py b/instanthmr/pipeline.pyindex e2393f3..f2775a0 100644--- a/instanthmr/pipeline.py+++ b/instanthmr/pipeline.py@@ -1,260 +1,49 @@-"""End-to-end pose estimation pipeline.--Combines the RF-DETR person detector with InstantHMR to produce per-person-3D + 2D joints from a single RGB frame, while reporting **separate**-detector and HMR timings so the demo can surface them.-"""-+"""Person detection followed by InstantHMR, with per-stage timings."""from __future__ import annotationsimport time-from collections import dequefrom dataclasses import dataclassfrom pathlib import Path-from typing import Optionalimport numpy as np-from .detector import RFDETRDetector-from .inference import InstantHMR, HMRPrediction-from .smoothing import _Tracker+from .detector import PersonDetector+from .inference import HMRPrediction, InstantHMR+from .models import DEFAULT_MODEL, resolve_model@dataclassclass FrameResult:- """Output of one ``PosePipeline.predict`` call.-- Attributes:- persons: Per-person ``HMRPrediction``s (possibly empty).- detector_ms: Wall-clock time for the detector forward pass (0 on- frames where the detector was skipped via ``detector_stride``).- hmr_ms: Wall-clock time for InstantHMR across all persons in the- frame (sum, not average — or one batched call when- ``batch_persons=True``).- total_ms: ``detector_ms + hmr_ms`` plus pipeline overhead.- """-persons: list[HMRPrediction]detector_ms: floathmr_ms: float- total_ms: float---def _expand_bbox(bbox: np.ndarray, w: int, h: int, expand: float) -> np.ndarray:- bw = bbox[2] - bbox[0]- bh = bbox[3] - bbox[1]- return np.array([- max(0.0, bbox[0] - bw * expand),- max(0.0, bbox[1] - bh * expand),- min(float(w), bbox[2] + bw * expand),- min(float(h), bbox[3] + bh * expand),- ], dtype=np.float32)---@dataclass-class _Box:- """What :class:`instanthmr.smoothing._Tracker` reads off a person."""- bbox: np.ndarrayclass PosePipeline:- """Full RF-DETR → InstantHMR pipeline.-- Args:- onnx_path: Path to ``instanthmr.onnx``.- device: ``"cuda"``, ``"coreml"`` or ``"cpu"`` for InstantHMR.- detector_variant: ``"nano" | "small" | "medium" | "base" | "large"``.- det_confidence: RF-DETR confidence threshold.- max_persons: Maximum number of persons returned per frame.- detector_stride: Run the detector every ``N`` frames; on the- in-between frames, reuse the previous frame's detections- (slightly expanded) to drive InstantHMR. ``1`` reproduces the- original "detect every frame" behaviour. The detector is by- far the dominant cost on most hardware, so stride 2–3 is the- single biggest knob for end-to-end FPS.- batch_persons: When ``True`` (default), all detected persons in a- frame are batched into a single InstantHMR ONNX call instead- of looping. This is a meaningful win for multi-person frames- and free for single-person frames.- mhr: Optional MHR backend (see :mod:`instanthmr.mhr_renderer`).- Required for MHR-only graphs, whose 3D keypoints are derived- from the predicted MHR parameters instead of a 3D head; ignored- by graphs that carry a ``joints_3d`` output.- focal: The camera's focal length in pixels, when known. A graph- exported from a ``--cliff-focal`` checkpoint is conditioned on- angles rather than pixels, so it needs a focal; without one it- falls back to a diagonal-derived stand-in, which mis-places the- person in depth on any camera whose true focal differs.- cliff_focal: Override the conditioning form; normally read from the- ONNX metadata. See :class:`instanthmr.inference.InstantHMR`.- use_prev: For a prev-token graph, feed each tracked person their own- prediction from ``prev_lag`` frames earlier (IoU tracking on the- detector boxes); the previous pose is withheld on a track's first- ``prev_lag`` frames. ``False`` always withholds it. Ignored by- other graphs.- prev_lag: Processed frames between a prediction and the frame it- feeds. 5 (167 ms at 30 fps) is what the benchmarks use, inside- the 5-20 frame pairs the token was trained on.+ """RF-DETR, then InstantHMR on every detected person in one batched call.- Example- -------- pipeline = PosePipeline("models/instanthmr.onnx", detector_stride=2)- result = pipeline.predict(image_rgb)- for p in result.persons:- print(p.joints_3d_cam.shape) # (70, 3)+ `model` is a released model name (nano, tiny, small, medium, medium-commercial) or an .onnx+ path. `detector_stride` > 1 runs the detector every N frames and reuses its boxes in between."""- def __init__(- self,- onnx_path: str | Path,- device: str = "cuda",- detector_variant: str = "medium",- det_confidence: float = 0.5,- max_persons: int = 2,- detector_stride: int = 1,- batch_persons: bool = True,- mhr: object | None = None,- focal: float | None = None,- cliff_focal: bool | None = None,- use_prev: bool = True,- prev_lag: int = 5,- ):- if detector_stride < 1:- raise ValueError("detector_stride must be >= 1")-- self.hmr = InstantHMR(onnx_path, device=device, mhr=mhr,- focal=focal, cliff_focal=cliff_focal)- if self.hmr.derives_joints_3d and mhr is None:- raise ValueError(- f"{Path(onnx_path).name} has no 'joints_3d' output (MHR-only student): "- "its 3D keypoints come from an MHR forward pass, so an MHR backend is "- "required. Pass mhr=build_mhr(...), or run demo.py with --mhr-model / "- "--mhr-assets."- )- self.detector = RFDETRDetector(- variant=detector_variant,- confidence=det_confidence,- max_persons=max_persons,- )-- # Warm up both models before the first real frame.- # - RF-DETR (PyTorch): first call triggers CUDA JIT for transformer ops.- # - InstantHMR (ONNX/CUDA EP): first call at a new batch size triggers- # kernel compilation and GPU memory allocation.- # With padded-batch inference the session only ever sees two shapes:- # batch=1 and batch=max_persons — two warm-up calls covers everything.- print(f"Warming up models (max_persons={max_persons})…", flush=True)- self.detector.warmup()- self.hmr.warmup(max_batch_size=max_persons)- if mhr is not None:- self._warmup_mhr(mhr, max_persons)- print("Ready.", flush=True)-- self._max_persons = max_persons- self._detector_stride = detector_stride- self._batch_persons = batch_persons- self._frame_idx = 0- self._last_detections: list[dict] = []- self._last_bbox: Optional[np.ndarray] = None- self.feeds_prev = use_prev and self.hmr.has_prev_input- self._prev_lag = max(1, prev_lag)- self._tracker = _Tracker()- self._history: dict[int, deque] = {} # track id -> last prev_lag prev_pose()s-- @staticmethod- def _warmup_mhr(mhr, max_persons: int) -> None:- """Run the MHR rig once per batch shape the demo will actually use.-- TorchScript specialises on input shapes over the first few calls, so an- un-warmed rig costs hundreds of ms on frame 1 instead of ~1 ms. Three- passes is what it takes for the profiling executor to settle: the first- records shapes, the second triggers the fusion compile, the third runs- the compiled kernels.- """- pose = np.zeros((max_persons, 204), dtype=np.float32)- shape = np.zeros((max_persons, 45), dtype=np.float32)- for _ in range(3):- mhr.forward(pose[0], shape[0]) # mesh: always single-person- if getattr(mhr, "has_keypoints", False):- for n in sorted({1, max_persons}): # keypoints: 1 and the padded batch- mhr.keypoints(pose[:n], shape[:n])-- def predict(self, image_rgb: np.ndarray) -> FrameResult:- """Detect all persons in *image_rgb* and run InstantHMR on each."""- t_total_start = time.perf_counter()- h, w = image_rgb.shape[:2]-- # ---- Detector (with optional stride) ----- run_detector = (self._frame_idx % self._detector_stride) == 0- if run_detector:- t0 = time.perf_counter()- detections = self.detector.detect(image_rgb)- detector_ms = (time.perf_counter() - t0) * 1000.0- if detections:- self._last_detections = detections- else:- detections = []- detector_ms = 0.0-- # If the detector skipped or missed this frame, reuse the previous- # detections (slightly expanded to absorb small motion). This is a- # cheap form of tracking that keeps HMR fed with a plausible bbox.- if not detections and self._last_detections:- detections = [- {- "bbox": _expand_bbox(d["bbox"], w, h, expand=0.1),- "confidence": float(d["confidence"]) * 0.9,- }- for d in self._last_detections- ]- elif not detections and self._last_bbox is not None:- # Backwards-compatible single-bbox fallback (used when no prior- # multi-person detections exist).- fb = _expand_bbox(self._last_bbox, w, h, expand=0.1)- detections = [{"bbox": fb, "confidence": 0.0}]-- # ---- InstantHMR ----- # Hard-cap so we never exceed the pre-warmed batch size.- detections = detections[: self._max_persons]- outputs: list[HMRPrediction] = []+ def __init__(self, model: str | Path = DEFAULT_MODEL, device: str = "cuda", detector: str = "medium",+ det_confidence: float = 0.5, max_persons: int = 4, detector_stride: int = 1,+ focal: float | None = None):+ self.hmr = InstantHMR(resolve_model(model), device=device, focal=focal, max_batch=max_persons)+ self.detector = PersonDetector(detector, det_confidence, max_persons)+ self.detector_stride = max(1, detector_stride)+ self._frame = 0+ self._boxes = np.zeros((0, 4), np.float32)+ self._scores = np.zeros(0, np.float32)+ self.detector(np.zeros((288, 288, 3), np.uint8))+ self.hmr.warmup(max_persons)++ def __call__(self, image_rgb: np.ndarray) -> FrameResult:+ t0 = time.perf_counter()+ if self._frame % self.detector_stride == 0:+ self._boxes, self._scores = self.detector(image_rgb)t1 = time.perf_counter()- ids, prev = [], [None] * len(detections)- if self.feeds_prev:- ids = self._tracker.assign([_Box(d["bbox"]) for d in detections])- prev = [h[0] if len(h) == self._prev_lag else None- for h in (self._history.get(t, ()) for t in ids)]- if detections:- if self._batch_persons and len(detections) > 1:- # Always pad to _max_persons so ONNX sees a constant batch- # shape and never triggers a mid-demo recompilation.- outputs = self.hmr.predict_batch(- image_rgb, detections, padded_to=self._max_persons, prev=prev,- )- else:- for det, pp in zip(detections, prev):- outputs.append(self.hmr.predict(- image_rgb,- bbox=det["bbox"],- confidence=det["confidence"],- prev=pp,- ))- if self.feeds_prev:- for t, o in zip(ids, outputs):- self._history.setdefault(t, deque(maxlen=self._prev_lag)).append(o.prev_pose())- for t in set(self._history) - self._tracker.active_ids:- del self._history[t]- hmr_ms = (time.perf_counter() - t1) * 1000.0-- if outputs:- best = max(outputs, key=lambda o: o.confidence)- self._last_bbox = best.bbox.copy()-- total_ms = (time.perf_counter() - t_total_start) * 1000.0- self._frame_idx += 1-- return FrameResult(- persons=outputs,- detector_ms=detector_ms,- hmr_ms=hmr_ms,- total_ms=total_ms,- )+ persons = self.hmr.predict(image_rgb, self._boxes, self._scores)+ t2 = time.perf_counter()+ self._frame += 1+ return FrameResult(persons, (t1 - t0) * 1000, (t2 - t1) * 1000)diff --git a/instanthmr/reference.py b/instanthmr/reference.pydeleted file mode 100644index 1401797..0000000--- a/instanthmr/reference.py+++ /dev/null@@ -1,104 +0,0 @@-"""Saved SAM 3D Body predictions, for side-by-side comparison in the demo.--The teacher's own output on a clip is the most useful thing to hold a student-against visually: it shares the rig, so the two meshes are directly comparable-rather than needing a conversion.--A reference file is produced by running the teacher once and packing the result:-- MOMENTUM_ENABLED=0 python main.py --video_path vid1.mp4 \\- --output_path reference/vid1 --save_meshes # in video_to_pose_pipeline- python tools/pack_reference.py reference/vid1/meshes reference/vid1.npz--It stores MHR **parameters**, not vertices, and ``demo.py`` decodes them with-the same rig it uses for the student. That is what makes the comparison honest:-any difference you see in the viewer is the model, never the renderer.--``MOMENTUM_ENABLED=0`` matters -- it makes the teacher's MHR head load the-TorchScript rig instead of ``mhr.mhr.MHR``, whose pymomentum extension needs-``torch>=2.8`` and segfaults (not raises) against an older torch.-"""--from __future__ import annotations--from dataclasses import dataclass-from pathlib import Path--import numpy as np---@dataclass(frozen=True)-class ReferencePerson:- """One person in one reference frame, in the demo's own conventions."""-- mhr_params: np.ndarray # (204,) float32- shape_params: np.ndarray # (45,) float32- cam_trans: np.ndarray # (3,) float32, rig-local -> camera space- joints_3d_cam: np.ndarray # (70, 3) float32, camera space- joints_2d: np.ndarray # (70, 2) float32, pixels- bbox: np.ndarray # (4,) float32, xyxy---class ReferenceTrack:- """Random access by frame index into a packed reference run.-- Frames where the teacher detected nobody were never written, so a lookup- for them returns an empty list rather than raising.- """-- def __init__(self, path: str | Path):- self.path = Path(path)- z = np.load(self.path)- self.image_shape = z["image_shape"]- counts = z["counts"].astype(int)- starts = np.concatenate([[0], np.cumsum(counts)[:-1]])- # frame index -> (offset, count) into the flat per-person arrays- self._span = {int(f): (int(s), int(c))- for f, s, c in zip(z["frame_idx"], starts, counts)}- self._model = z["model_params"]- self._shape = z["shape_params"]- self._trans = z["cam_trans"]- self._j3d = z["joints_3d"]- self._j2d = z["joints_2d"]- self._bbox = z["bbox"]- self.num_frames = len(counts)- self.num_persons = int(counts.sum())-- def __len__(self) -> int:- return self.num_frames-- def persons_at(self, frame_idx: int) -> list[ReferencePerson]:- span = self._span.get(int(frame_idx))- if span is None:- return []- start, count = span- out = []- for i in range(start, start + count):- # joints_3d as saved by the teacher are rig-local, the same frame- # its vertices are in; the demo works in camera space.- out.append(ReferencePerson(- mhr_params=self._model[i],- shape_params=self._shape[i],- cam_trans=self._trans[i],- joints_3d_cam=self._j3d[i] + self._trans[i],- joints_2d=self._j2d[i],- bbox=self._bbox[i],- ))- return out---def resolve_reference(arg: str | None, video: str | Path | None) -> Path | None:- """Turn ``--show-reference`` into a path.-- A bare flag means "the reference for this clip": ``reference/<stem>.npz``- beside the repo root, named after the input video.- """- if arg is None:- return None- if arg:- return Path(arg)- if video is None:- raise SystemExit(- "[error] --show-reference with no path needs --video to name the "- "reference file (reference/<video stem>.npz)")- return Path("reference") / f"{Path(video).stem}.npz"diff --git a/instanthmr/smoothing.py b/instanthmr/smoothing.pyindex 3ac56eb..581f913 100644--- a/instanthmr/smoothing.py+++ b/instanthmr/smoothing.py@@ -1,445 +1,101 @@-"""Temporal smoothing of per-person HMR predictions.+"""Temporal smoothing of per-person predictions with the 1 Euro filter.-InstantHMR is a frame-by-frame model: every frame is estimated independently,-so its outputs carry per-frame noise that reads as jitter in a video — most-visibly in ``cam_trans``, which is a free 3-vector from a linear head with-almost nothing tying it to the observed pixels.--A non-learned moving-average filter over a short window removes much of that.-NLF (arXiv:2407.07532, Tables 2-3) reports a 5-frame filter of this kind-improving 3DPW MPJPE 59.0 -> 57.2 mm and EMDB 68.4 -> 66.7 mm, at zero-training cost. This module is that filter.--Two things the filter needs that the pipeline does not provide:--* **Identity across frames.** ``PosePipeline.predict`` returns an unordered- list of detections per frame, so a person's index is not stable. We attach a- minimal greedy IoU tracker (:class:`_Tracker`) purely to decide which- predictions belong to the same person before averaging them.-* **Lookahead**, for ``mode="centered"``. A centred window has no phase lag- but needs ``(window - 1) // 2`` future frames, so :meth:`TemporalSmoother.push`- returns results delayed by that many frames and :meth:`flush` drains the- tail. ``mode="causal"`` trades phase lag for zero latency.--The moving average has one structural weakness: it is a *fixed*-cutoff filter,-so it cannot tell jitter from fast intentional motion and lags during both. On-a waving hand that is plainly visible. ``mode="1euro"`` fixes it with the-1 Euro filter (Casiez, Roussel & Vogel, CHI 2012): an exponential smoother-whose cutoff frequency rises with the signal's own speed. At rest the cutoff is-low and jitter is crushed; during fast motion the cutoff opens up and the-output tracks. It is causal, needs no lookahead, and is the right choice for-demos and live camera. The centred moving average remains the right choice for-benchmark numbers, since that is the filter NLF measured.--Caveat on ``mhr_params``: these are MHR joint parameters (angles and scales),-and this filter averages them linearly. That is well-behaved for continuous-motion but is *wrong* across an angle wraparound — a global yaw crossing +/-pi-will be averaged toward zero instead of through the discontinuity, producing a-brief flip. Short windows on real footage rarely hit it; if you see one, drop-``mhr_params`` from ``SMOOTHED_FIELDS`` and accept a mesh that jitters while-the keypoints do not.+InstantHMR estimates every frame on its own, so a video shows per-frame jitter. The 1 Euro filter+(Casiez, Roussel and Vogel, CHI 2012) smooths each coordinate with a cutoff that rises with its+own speed: still joints are smoothed hard and fast ones barely, so it adds no visible lag. It is+causal, so it works on a live camera. People are matched across frames by box overlap."""-from __future__ import annotations-from collections import dequefrom dataclasses import replace-from typing import Iterator, Sequence+from typing import Sequenceimport numpy as npfrom .inference import HMRPrediction-#: Fields averaged across the window. All are Euclidean quantities for which a-#: linear mean is the right operation, except ``mhr_params`` — see the module-#: docstring. ``joints_3d_cam`` is deliberately absent: it is recomputed from-#: the smoothed ``joints_3d_local`` and ``cam_trans`` so the three stay-#: consistent with each other. ``joints_3d_local`` is filtered only when no-#: ``keypoint_fn`` is available; with one, it is re-derived from the filtered-#: ``mhr_params`` so the keypoints and the mesh cannot disagree.-SMOOTHED_FIELDS = (- "joints_3d_local",- "cam_trans",- "joints_2d",- "mhr_params",- "shape_params",-)---#: Per-field ``(mincutoff_hz, beta)`` for ``mode="1euro"``.-#:-#: ``beta`` scales a *speed*, so its right value depends on the field's units —-#: this is the classic tuning trap with the 1 Euro filter. ``joints_2d`` is in-#: full-frame pixels (a fast hand runs ~500-1500 px/s), the 3D fields are in-#: metres (~1-3 m/s), ``mhr_params`` in radians and scale units. The values-#: below put all of them at a comparable cutoff (~9 Hz) during fast motion-#: while holding ~1 Hz at rest. ``shape_params`` gets beta 0 on purpose:-#: identity does not change, so it should be filtered as hard as possible —-#: the per-track limit of that idea is NLF's shared-beta-per-track fitting.-ONE_EURO_DEFAULTS: dict[str, tuple[float, float]] = {- "joints_2d": (1.0, 0.01),+# (mincutoff in Hz, beta) per field, in the field's own units: pixels for joints_2d, metres for 3D.+FIELDS = {+ "joints_2d": (1.0, 0.01),"joints_3d_local": (1.0, 4.0),- "cam_trans": (0.6, 2.0),- "mhr_params": (1.0, 2.0),- "shape_params": (0.3, 0.0),+ "vertices_local": (1.0, 4.0),+ "cam_trans": (0.6, 2.0),}class _OneEuro:- """1 Euro filter (Casiez, Roussel & Vogel, CHI 2012), vectorised.-- Operates elementwise on an array of any shape, so every joint — indeed- every coordinate — adapts on its own speed. That is the property that- matters here: a still torso stays heavily smoothed in the same frame where- a waving hand is barely smoothed at all.- """-def __init__(self, mincutoff: float, beta: float, dcutoff: float = 1.0):- self.mincutoff = float(mincutoff)- self.beta = float(beta)- self.dcutoff = float(dcutoff)- self._x: np.ndarray | None = None- self._dx: np.ndarray | None = None- self._t: float | None = None+ self.mincutoff, self.beta, self.dcutoff = mincutoff, beta, dcutoff+ self.x = self.dx = self.t = None@staticmethod- def _alpha(cutoff, dt: float):- tau = 1.0 / (2.0 * np.pi * np.maximum(cutoff, 1e-6))- return 1.0 / (1.0 + tau / dt)+ def _alpha(cutoff, dt):+ return 1.0 / (1.0 + 1.0 / (2 * np.pi * np.maximum(cutoff, 1e-6) * dt))def __call__(self, x: np.ndarray, t: float) -> np.ndarray:- x = np.asarray(x, dtype=np.float64)- if self._x is None:- self._x, self._dx, self._t = x, np.zeros_like(x), t+ if self.x is None:+ self.x, self.dx, self.t = x.astype(np.float64), np.zeros(x.shape), treturn x- dt = t - self._t- if not (dt > 0.0):- # Repeated or non-monotonic timestamps would blow up the derivative;- # fall back to a nominal 30 fps step rather than dividing by zero.- dt = 1.0 / 30.0- self._t = t-+ dt = t - self.t if t > self.t else 1 / 30+ self.t = ta_d = self._alpha(self.dcutoff, dt)- self._dx = a_d * ((x - self._x) / dt) + (1.0 - a_d) * self._dx- a = self._alpha(self.mincutoff + self.beta * np.abs(self._dx), dt)- self._x = a * x + (1.0 - a) * self._x- return self._x+ self.dx = a_d * (x - self.x) / dt + (1 - a_d) * self.dx+ a = self._alpha(self.mincutoff + self.beta * np.abs(self.dx), dt)+ self.x = a * x + (1 - a) * self.x+ return self.x.astype(np.float32)def _iou(a: np.ndarray, b: np.ndarray) -> float:- """Intersection-over-union of two ``[x1, y1, x2, y2]`` boxes."""- x1 = max(float(a[0]), float(b[0]))- y1 = max(float(a[1]), float(b[1]))- x2 = min(float(a[2]), float(b[2]))- y2 = min(float(a[3]), float(b[3]))- iw = max(0.0, x2 - x1)- ih = max(0.0, y2 - y1)+ iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))+ ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))inter = iw * ih- if inter <= 0.0:- return 0.0- area_a = max(0.0, float(a[2] - a[0])) * max(0.0, float(a[3] - a[1]))- area_b = max(0.0, float(b[2] - b[0])) * max(0.0, float(b[3] - b[1]))- union = area_a + area_b - inter- return inter / union if union > 0.0 else 0.0+ union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter+ return float(inter / union) if union > 0 else 0.0-class _Tracker:- """Greedy IoU association of detections to persistent track ids.+class TemporalSmoother:+ """`smoother(persons, timestamp)` returns the same people with smoothed fields.- Deliberately minimal — it exists only to group predictions of the same- person before averaging, not to be a general-purpose tracker. Tracks- survive ``max_age`` frames without a match so a brief miss does not restart- the filter (and reset the smoothing) for that person.+ `beta_scale` trades jitter for lag: raise it if fast motion lags, lower it if a still person+ shakes. A track survives `max_age` frames without a match before its filter is reset."""- def __init__(self, iou_threshold: float = 0.3, max_age: int = 5):- self.iou_threshold = iou_threshold- self.max_age = max_age- self._boxes: dict[int, np.ndarray] = {}- self._age: dict[int, int] = {}+ def __init__(self, beta_scale: float = 1.0, iou_threshold: float = 0.3, max_age: int = 5):+ self.beta_scale, self.iou_threshold, self.max_age = beta_scale, iou_threshold, max_age+ self._tracks: dict[int, dict] = {}self._next_id = 0- def assign(self, persons: Sequence[HMRPrediction]) -> list[int]:- """Return one track id per person, in the order given."""- ids: list[int] = [-1] * len(persons)- free = set(self._boxes)+ def __call__(self, persons: Sequence[HMRPrediction], timestamp: float) -> list[HMRPrediction]:+ ids = self._assign(persons)+ out = []+ for track_id, person in zip(ids, persons):+ filters = self._tracks[track_id]["filters"]+ fields = {}+ for name, (mincutoff, beta) in FIELDS.items():+ value = getattr(person, name)+ if value is None:+ continue+ f = filters.setdefault(name, _OneEuro(mincutoff, beta * self.beta_scale))+ fields[name] = f(value, timestamp)+ out.append(replace(person, **fields))+ return out- # Score every (detection, track) pair, then take them best-first. With- # the handful of people this pipeline handles, an O(n*m) sort is far- # cheaper than pulling in a Hungarian solver.- pairs = [- (_iou(p.bbox, self._boxes[t]), i, t)- for i, p in enumerate(persons)- for t in free- ]- pairs.sort(reverse=True)- taken: set[int] = set()+ def _assign(self, persons: Sequence[HMRPrediction]) -> list[int]:+ pairs = sorted(((_iou(p.bbox, tr["box"]), i, t) for i, p in enumerate(persons)+ for t, tr in self._tracks.items()), reverse=True)+ ids, used = [-1] * len(persons), set()for score, i, t in pairs:- if score < self.iou_threshold:- break- if ids[i] != -1 or t not in free or i in taken:- continue- ids[i] = t- free.discard(t)- taken.add(i)-+ if score >= self.iou_threshold and ids[i] == -1 and t not in used:+ ids[i] = t+ used.add(t)for i, p in enumerate(persons):if ids[i] == -1:ids[i] = self._next_id+ self._tracks[ids[i]] = {"filters": {}}self._next_id += 1- self._boxes[ids[i]] = np.asarray(p.bbox, dtype=np.float64)- self._age[ids[i]] = 0-- for t in list(free):- self._age[t] = self._age.get(t, 0) + 1- if self._age[t] > self.max_age:- self._boxes.pop(t, None)- self._age.pop(t, None)+ self._tracks[ids[i]].update(box=p.bbox, age=0)+ for t in set(self._tracks) - set(ids):+ self._tracks[t]["age"] += 1+ if self._tracks[t]["age"] > self.max_age:+ del self._tracks[t]return ids-- @property- def active_ids(self) -> set[int]:- return set(self._boxes)---class TemporalSmoother:- """Temporal filter over per-person predictions.-- Args:- window: frames averaged by the moving-average modes. Forced odd in- ``"centered"``. Ignored by ``"1euro"``.- mode: ``"centered"`` (moving average, no phase lag, ``(window-1)//2``- frames of latency — the filter NLF measured, so the one to use for- benchmark numbers), ``"causal"`` (trailing average, zero latency,- real phase lag) or ``"1euro"`` (adaptive, zero latency, no phase- lag during fast motion — the one to use for demos and live camera).- beta_scale: multiplies every per-field ``beta`` in ``"1euro"`` mode.- This is *the* jitter-versus-lag knob: raise it if motion still lags,- lower it if the output looks noisy while you hold still.- mincutoff_scale: multiplies every per-field ``mincutoff``. Governs how- hard the signal is smoothed at rest.- keypoint_fn: optional ``f(mhr_params, shape_params) -> (70, 3)``. When- given, ``joints_3d_local`` is *re-derived* from the filtered pose- parameters instead of being filtered on its own. This is what keeps- the keypoints welded to the mesh — see :meth:`_finalize`.- iou_threshold: minimum IoU to associate a detection with a track.- max_age: frames a track survives without a detection.-- Usage mirrors a delay line — in the zero-latency modes ``push`` simply- always returns a frame and ``flush`` yields nothing::-- for persons in stream:- out = smoother.push(persons, timestamp=t)- if out is not None:- render(out)- for out in smoother.flush():- render(out)- """-- def __init__(- self,- window: int = 5,- mode: str = "centered",- beta_scale: float = 1.0,- mincutoff_scale: float = 1.0,- keypoint_fn=None,- iou_threshold: float = 0.3,- max_age: int = 5,- ):- if mode not in ("centered", "causal", "1euro"):- raise ValueError(- f"mode must be 'centered', 'causal' or '1euro', got {mode!r}")- if window < 1:- raise ValueError(f"window must be >= 1, got {window}")- if mode == "centered" and window % 2 == 0:- window += 1- self.window = window- self.mode = mode- self.lag = (window - 1) // 2 if mode == "centered" else 0- self.beta_scale = float(beta_scale)- self.mincutoff_scale = float(mincutoff_scale)- self.keypoint_fn = keypoint_fn-- self._tracker = _Tracker(iou_threshold=iou_threshold, max_age=max_age)- self._buf: deque[dict[int, HMRPrediction]] = deque(maxlen=window)- # mode="1euro" only: one filter bank per track, built on first sight and- # dropped when the tracker retires the track.- self._banks: dict[int, dict[str, _OneEuro]] = {}- self._clock = 0.0- # Frames in and frames out. Every pushed frame is emitted exactly once,- # so a caller holding its own delay line can pop one entry per returned- # frame and stay in step with the video.- self._n_pushed = 0- self._n_emitted = 0-- # Jitter accounting, so the effect is measurable rather than only- # visible. Both are mean frame-to-frame deltas over the same tracks.- self._prev_raw: dict[int, HMRPrediction] = {}- self._prev_out: dict[int, HMRPrediction] = {}- self._raw_cam: list[float] = []- self._raw_j3d: list[float] = []- self._out_cam: list[float] = []- self._out_j3d: list[float] = []-- # ------------------------------------------------------------------- def push(- self,- persons: Sequence[HMRPrediction],- timestamp: float | None = None,- ) -> list[HMRPrediction] | None:- """Feed one frame. Returns a smoothed frame, or ``None`` while filling.-- ``timestamp`` (seconds) is used only by ``mode="1euro"``, which needs a- real time base to turn deltas into speeds — pass wall-clock for a live- camera and ``frame_idx / fps`` for a file, so ``--frame-skip`` is- accounted for. It defaults to a nominal 30 fps clock.- """- ids = self._tracker.assign(persons)- frame = {t: p for t, p in zip(ids, persons)}- self._accumulate(frame, self._prev_raw, self._raw_cam, self._raw_j3d)- self._prev_raw = frame-- if timestamp is None:- self._clock += 1.0 / 30.0- timestamp = self._clock-- if self.mode == "1euro":- self._n_pushed += 1- out = self._one_euro(frame, timestamp)- self._n_emitted += 1- return out-- self._buf.append(frame)- self._n_pushed += 1-- # Emit frame ``_n_emitted`` as soon as ``lag`` frames after it have- # arrived. Near the start of the stream that window is one-sided rather- # than centred — the usual boundary treatment for a moving average, and- # what keeps frames-in equal to frames-out.- if self._n_pushed - 1 < self._n_emitted + self.lag:- return None- return self._emit_next()-- def flush(self) -> Iterator[list[HMRPrediction]]:- """Drain the frames still held in the delay line at end of stream."""- while self._n_emitted < self._n_pushed:- yield self._emit_next()-- def _one_euro(self, frame: dict[int, HMRPrediction], t: float) -> list[HMRPrediction]:- """Filter each track's fields independently, elementwise, on its own speed."""- alive = self._tracker.active_ids- for dead in [k for k in self._banks if k not in alive]:- del self._banks[dead]-- out: list[HMRPrediction] = []- emitted: dict[int, HMRPrediction] = {}- for track_id, pred in frame.items():- bank = self._banks.get(track_id)- if bank is None:- bank = {- name: _OneEuro(- mincutoff=ONE_EURO_DEFAULTS[name][0] * self.mincutoff_scale,- beta=ONE_EURO_DEFAULTS[name][1] * self.beta_scale,- )- for name in SMOOTHED_FIELDS- }- self._banks[track_id] = bank- fields = {- name: bank[name](getattr(pred, name), t).astype(np.float32)- for name in SMOOTHED_FIELDS- }- smoothed = self._finalize(pred, fields)- out.append(smoothed)- emitted[track_id] = smoothed-- self._accumulate(emitted, self._prev_out, self._out_cam, self._out_j3d)- self._prev_out = emitted- return out-- def _emit_next(self) -> list[HMRPrediction]:- buf_start = self._n_pushed - len(self._buf)- out = self._emit(self._n_emitted - buf_start)- self._n_emitted += 1- return out-- # ------------------------------------------------------------------- def _finalize(self, anchor: HMRPrediction, fields: dict) -> HMRPrediction:- """Assemble one smoothed prediction, keeping its parts self-consistent.-- Filtering ``mhr_params`` and ``joints_3d_local`` independently makes the- green keypoints drift off the skin during fast motion, because the mesh- is decoded from the former while the keypoints come from the latter and- forward kinematics is *nonlinear* — ``filter(FK(p)) != FK(filter(p))``.- Adaptive filtering makes it worse still, since each field then gets its- own cutoff driven by its own speed in its own units.-- So when a keypoint backend is available we do not filter the keypoints- at all: we filter the pose parameters and re-derive the keypoints from- them, which is exactly how ``inference.py`` produces them for an- MHR-only graph. The two agree by construction, in every mode.- """- if self.keypoint_fn is not None:- fields["joints_3d_local"] = np.asarray(- self.keypoint_fn(fields["mhr_params"], fields["shape_params"]),- dtype=np.float32,- ).reshape(anchor.joints_3d_local.shape)- fields["joints_3d_cam"] = (- fields["joints_3d_local"] + fields["cam_trans"][None, :]- ).astype(np.float32)- return replace(anchor, **fields)-- def _emit(self, centre: int) -> list[HMRPrediction]:- """Average every track present in frame ``centre`` over the buffer."""- out: list[HMRPrediction] = []- emitted: dict[int, HMRPrediction] = {}- for track_id, anchor in self._buf[centre].items():- samples = [f[track_id] for f in self._buf if track_id in f]- fields = {- name: np.mean(- np.stack([np.asarray(getattr(s, name), dtype=np.float64)- for s in samples]),- axis=0,- ).astype(np.float32)- for name in SMOOTHED_FIELDS- }- smoothed = self._finalize(anchor, fields)- out.append(smoothed)- emitted[track_id] = smoothed-- self._accumulate(emitted, self._prev_out, self._out_cam, self._out_j3d)- self._prev_out = emitted- return out-- @staticmethod- def _accumulate(cur, prev, cam_acc: list[float], j3d_acc: list[float]) -> None:- """Record mean frame-to-frame movement for tracks seen in both frames."""- for track_id, p in cur.items():- q = prev.get(track_id)- if q is None:- continue- cam_acc.append(float(np.linalg.norm(p.cam_trans - q.cam_trans)))- j3d_acc.append(- float(np.linalg.norm(p.joints_3d_local - q.joints_3d_local, axis=-1).mean())- )-- # ------------------------------------------------------------------- def jitter_report(self) -> str | None:- """One-line before/after summary, or ``None`` if nothing was measured.-- These are mean per-frame *movement* magnitudes, not errors: real motion- contributes to both columns. The filter is doing its job when the- numbers drop without the pose visibly lagging the video.- """- if not self._raw_cam or not self._out_cam:- return None- rc, oc = float(np.mean(self._raw_cam)), float(np.mean(self._out_cam))- rj, oj = float(np.mean(self._raw_j3d)), float(np.mean(self._out_j3d))- how = (f"1euro, beta x{self.beta_scale:g}" if self.mode == "1euro"- else f"window={self.window}, {self.mode}")- return (- f"smoothing ({how}) — mean frame-to-frame delta\n"- f" cam_trans {rc * 1000:7.2f} -> {oc * 1000:7.2f} mm "- f"({100.0 * (1.0 - oc / rc) if rc > 0 else 0.0:+.0f}%)\n"- f" joints_3d_local {rj * 1000:7.2f} -> {oj * 1000:7.2f} mm "- f"({100.0 * (1.0 - oj / rj) if rj > 0 else 0.0:+.0f}%)"- )diff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.pyindex 41c7562..b1d1505 100644--- a/instanthmr/visualizer.py+++ b/instanthmr/visualizer.py@@ -1,476 +1,103 @@-"""Rerun-based 3D visualization for the InstantHMR demo.--Logs, per frame:- - ``camera/image`` : annotated RGB frame (2D skeleton + bbox + a- text panel showing detector / HMR / render / total ms).- - ``camera`` : pinhole intrinsics so the image becomes a- proper frustum in the 3D scene.- - ``world/persons/...`` : the 70 3D keypoints + skeleton lines per person.- For an MHR-only student these come from the MHR- forward pass itself (see ``instanthmr.inference``).- - ``world/persons/.../mesh``: full MHR body mesh when an MHR backend is- provided (vertices from a true MHR forward pass).- Rendered semi-transparent via ``albedo_factor``- so the keypoints inside the body stay visible.- - ``timing/detector_ms`` : scalar plot of RF-DETR latency.- - ``timing/hmr_ms`` : scalar plot of InstantHMR latency.- - ``timing/render_ms`` : scalar plot of MHR mesh rendering latency.- - ``timing/total_ms`` : scalar plot of full pipeline latency.- - ``timing/fps`` : scalar plot of effective frames-per-second.- - ``contact/person_N/{left,right}``: per-foot contact probability, and the- feet drawn red while planted (> contact_threshold), for a- prev-token graph that has a contact head.--``rr.init(..., spawn=True)`` is called automatically so the viewer opens-on construction (pass ``spawn_viewer=False`` to suppress).-"""-+"""Rerun viewer: the frame with 2D skeletons and timings, the 3D scene, and a latency plot."""from __future__ import annotations-import time-from typing import TYPE_CHECKING, Optional, Sequence+from typing import Optional, Sequenceimport cv2import numpy as npfrom .inference import HMRPrediction-from .skeleton import edges_for--if TYPE_CHECKING:- from .mhr_renderer import MHRRenderer+from .skeleton import SKELETON_EDGES-#: Base skin tone of the body mesh, before the transparency factor is applied.MESH_COLOR = (200, 160, 130)-#: Default mesh opacity. Low enough that the 70 keypoints read through the-#: body, high enough that the silhouette is still legible.-MESH_ALPHA = 0.35--#: Reference (teacher) overlay colours — a cool palette against the student's-#: warm skin tone and green/yellow skeleton, so the two read apart at a glance-#: even where the bodies overlap.-REF_MESH_COLOR = (110, 170, 255)-REF_JOINT_COLOR = (0, 200, 255)-REF_SKELETON_COLOR = (255, 110, 255)--#: MHR70 keypoints of each foot (ankle, big toe, small toe, heel), in the-#: [left, right] order of ``HMRPrediction.contact_prob``.-FOOT_KP = ((13, 15, 16, 17), (14, 18, 19, 20))-FOOT_NAMES = ("left", "right")-PLANTED_COLOR = (255, 40, 40)-FREE_COLOR = (230, 230, 230)+JOINT_COLOR = (0, 230, 0)+BONE_COLOR = (255, 230, 0)+BOX_COLOR = (0, 200, 255)class RerunVisualizer:- """Rerun logger for the demo.-- Args:- application_id: Rerun application id.- spawn_viewer: open the viewer GUI on construction.- save_path: optional ``.rrd`` recording path.- mhr_renderer: MHR backend used to skin the body mesh. Any object with- ``.forward(mhr_params, shape_params)`` and ``.faces`` works; see- :mod:`instanthmr.mhr_renderer`.- mesh_alpha: body-mesh opacity in ``[0, 1]``. Applied through Rerun's- ``albedo_factor`` rather than per-vertex alpha, so one scalar- controls the whole surface.- reference: optional :class:`instanthmr.reference.ReferenceTrack` whose- saved teacher prediction is logged alongside the student's under- ``world/reference/``, in the cool palette above. It is decoded by- the SAME ``mhr_renderer``, so a visible difference is the model and- not the renderer.- """-- def __init__(- self,- application_id: str = "instanthmr_demo",- spawn_viewer: bool = True,- save_path: str | None = None,- mhr_renderer: "MHRRenderer | None" = None,- mesh_alpha: float = MESH_ALPHA,- reference=None,- contact_threshold: float = 0.5,- ):+ def __init__(self, faces: Optional[np.ndarray] = None, mesh_alpha: float = 0.35, spawn: bool = True,+ save_path: Optional[str] = None):import rerun as rr- self._rr = rr- self._mhr_renderer = mhr_renderer- self._mesh_albedo = (- *MESH_COLOR,- int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),- )- self._reference = reference- self._ref_albedo = (- *REF_MESH_COLOR,- int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),- )- self._prev_num_ref = 0- self.contact_threshold = contact_threshold- rr.init(application_id, spawn=spawn_viewer)-- # SAM3D / MHR convention: right-handed, Y-down camera frame.- rr.log("world", rr.ViewCoordinates.RIGHT_HAND_Y_DOWN, static=True)-+ self.rr = rr+ self.faces = faces+ self.albedo = (*MESH_COLOR, int(round(255 * min(max(mesh_alpha, 0.0), 1.0))))+ rr.init("instanthmr", spawn=spawn)if save_path:rr.save(save_path)-- self._prev_num_persons = 0- self._blueprint_sent = False-- # ------------------------------------------------------------------- # Per-frame logging- # -------------------------------------------------------------------- def log_frame(- self,- image_rgb: np.ndarray,- persons: Sequence[HMRPrediction],- frame_idx: int,- timestamp: float | None = None,- detector_ms: Optional[float] = None,- hmr_ms: Optional[float] = None,- total_ms: Optional[float] = None,- ) -> float:- """Log one frame of the pipeline to Rerun.-- Args:- image_rgb: full-frame RGB image, ``(H, W, 3)`` uint8.- persons: per-person HMR predictions for this frame.- frame_idx: monotonic frame counter (sets the Rerun timeline).- timestamp: optional wall-clock timestamp (seconds).- detector_ms: RF-DETR latency for this frame (ms).- hmr_ms: InstantHMR latency for this frame, summed across- persons (ms).- total_ms: full pipeline latency for this frame (ms).-- Returns:- render_ms: time spent running MHR forward passes + Rerun mesh- logging for this frame (0.0 when no MHRRenderer is set).- """- rr = self._rr- rr.set_time("frame", sequence=frame_idx)- if timestamp is not None:- rr.set_time("timestamp", duration=timestamp)-- # Per-stage scalar plots- if detector_ms is not None:- rr.log("timing/detector_ms", rr.Scalars(float(detector_ms)))- if hmr_ms is not None:- rr.log("timing/hmr_ms", rr.Scalars(float(hmr_ms)))- if total_ms is not None:- rr.log("timing/total_ms", rr.Scalars(float(total_ms)))- if total_ms > 0:- rr.log("timing/fps", rr.Scalars(1000.0 / float(total_ms)))-- # Clear stale person entities from previous frames- n = len(persons)- for stale in range(n, self._prev_num_persons):- rr.log(f"world/persons/person_{stale}", rr.Clear(recursive=True))- self._prev_num_persons = n+ rr.log("world", rr.ViewCoordinates.RIGHT_HAND_Y_DOWN, static=True)+ self._n_logged = 0+ self._layout_sent = False++ def log(self, frame: int, image_rgb: np.ndarray, persons: Sequence[HMRPrediction],+ timings: dict[str, float]) -> None:+ rr = self.rr+ rr.set_time("frame", sequence=frame)+ for name, ms in timings.items():+ rr.log(f"timing/{name}", rr.Scalars(float(ms)))+ for stale in range(len(persons), self._n_logged):+ rr.log(f"world/person_{stale}", rr.Clear(recursive=True))+ self._n_logged = len(persons)++ for i, p in enumerate(persons):+ joints = p.joints_3d_cam+ rr.log(f"world/person_{i}/joints", rr.Points3D(joints, radii=0.012, colors=JOINT_COLOR))+ rr.log(f"world/person_{i}/skeleton",+ rr.LineStrips3D(joints[np.array(SKELETON_EDGES)], colors=BONE_COLOR, radii=0.004))+ if self.faces is not None and p.vertices_local is not None:+ rr.log(f"world/person_{i}/mesh", rr.Mesh3D(vertex_positions=p.vertices_cam,+ triangle_indices=self.faces, albedo_factor=self.albedo))h, w = image_rgb.shape[:2]-- # Before the no-detection early-out below: the reference is independent- # of whether the student found anybody, and seeing it alone on a frame- # the student missed is exactly the failure worth looking at.- if self._reference is not None:- self._log_reference(frame_idx)-- # Always log the image — even when no person was detected — so the- # viewer keeps showing the camera feed and the timing HUD.- render_ms = 0.0- if not persons:- blank = self._draw_overlay(image_rgb, [], detector_ms, hmr_ms, render_ms, total_ms)- rr.log("camera/image", rr.Image(blank))- rr.log("timing/render_ms", rr.Scalars(0.0))- return render_ms-- # --- MHR mesh rendering (timed) ----------------------------------- if self._mhr_renderer is not None:- t0 = time.perf_counter()- for idx, person in enumerate(persons):- self._log_person_mesh(idx, person)- render_ms = (time.perf_counter() - t0) * 1000.0-- rr.log("timing/render_ms", rr.Scalars(render_ms))-- for idx, person in enumerate(persons):- self._log_person(idx, person)-- annotated = self._draw_overlay(- image_rgb, persons, detector_ms, hmr_ms, render_ms, total_ms,- self.contact_threshold,- )- rr.log("camera/image", rr.Image(annotated))-- # Pinhole camera in the 3D scene — full-frame focal so the image- # plane sits in front of the joints, not on top of them.- first = persons[0]- fx, fy = float(first.focal_length[0]), float(first.focal_length[1])- cx, cy = w / 2.0, h / 2.0- depth = float(first.joints_3d_cam[:, 2].mean())- plane_dist = max(depth * 0.25, 0.2)-- rr.log(- "camera",- rr.Pinhole(- width=w,- height=h,- focal_length=[fx, fy],- principal_point=[cx, cy],- image_plane_distance=plane_dist,- ),- )-- if not self._blueprint_sent:- self._send_blueprint(first)- self._blueprint_sent = True-- return render_ms-- # ------------------------------------------------------------------- # Helpers- # -------------------------------------------------------------------- def _log_person(self, idx: int, person: HMRPrediction) -> None:- """Log the 70 3D keypoints and the skeleton wiring for one person.-- For an MHR-only student ``joints_3d_cam`` was regressed from the MHR- skeleton that also produced the mesh, so the points sit exactly inside- the surface logged by :meth:`_log_person_mesh`.- """- rr = self._rr- path = f"world/persons/person_{idx}"-- joints = person.joints_3d_cam- rr.log(- f"{path}/joints",- rr.Points3D(positions=joints, radii=0.012, colors=[0, 230, 0]),- )-- edges = edges_for(joints.shape[0])- if edges:- lines = [- [joints[i].tolist(), joints[j].tolist()]- for i, j in edges- ]- rr.log(- f"{path}/skeleton",- rr.LineStrips3D(lines, colors=[255, 230, 0], radii=0.004),- )-- if person.contact_prob is not None:- for name, kp, prob in zip(FOOT_NAMES, FOOT_KP, person.contact_prob):- planted = prob > self.contact_threshold- rr.log(f"{path}/feet/{name}", rr.Points3D(- positions=joints[list(kp)], radii=0.03 if planted else 0.015,- colors=[PLANTED_COLOR if planted else FREE_COLOR]))- rr.log(f"contact/person_{idx}/{name}", rr.Scalars(float(prob)))-- def _log_reference(self, frame_idx: int) -> None:- """Log the saved teacher prediction for this frame under ``world/reference``.-- Kept in a sibling entity tree rather than mixed into- ``world/persons/`` so the viewer can toggle either side on its own.- """- rr = self._rr- persons = self._reference.persons_at(frame_idx)-- for stale in range(len(persons), self._prev_num_ref):- rr.log(f"world/reference/person_{stale}", rr.Clear(recursive=True))- self._prev_num_ref = len(persons)-- for idx, ref in enumerate(persons):- path = f"world/reference/person_{idx}"- joints = ref.joints_3d_cam- rr.log(- f"{path}/joints",- rr.Points3D(positions=joints, radii=0.012,- colors=list(REF_JOINT_COLOR)),- )- edges = edges_for(joints.shape[0])- if edges:- rr.log(- f"{path}/skeleton",- rr.LineStrips3D(- [[joints[i].tolist(), joints[j].tolist()] for i, j in edges],- colors=list(REF_SKELETON_COLOR), radii=0.004,- ),- )- # The teacher stored parameters, not vertices; decoding them with- # the student's own rig is what keeps the two meshes comparable.- if self._mhr_renderer is not None:- verts = self._mhr_renderer.forward(- ref.mhr_params, ref.shape_params,- ) + ref.cam_trans- rr.log(- f"{path}/mesh",- rr.Mesh3D(- vertex_positions=verts,- triangle_indices=self._mhr_renderer.faces,- albedo_factor=self._ref_albedo,- ),- )-- def _log_person_mesh(self, idx: int, person: HMRPrediction) -> None:- """Run MHR forward pass and log the resulting mesh vertices via Rerun."""- assert self._mhr_renderer is not None-- # Decode mhr_params + shape_params → body-space vertices.- verts_local = self._mhr_renderer.forward(- person.mhr_params, person.shape_params,- ) # (V, 3), body-centred-- # Shift from body space to camera space using InstantHMR's cam_trans.- verts_cam = verts_local + person.cam_trans # (V, 3) broadcast-- faces = self._mhr_renderer.faces # (F, 3) int32, shared across frames-- # Tint + transparency come from `albedo_factor`, a single RGBA multiplier- # for the whole surface. Per-vertex colours would work too, but the- # viewer multiplies the two, so keeping the alpha in one place makes the- # `mesh_alpha` knob mean exactly what it says.- self._rr.log(- f"world/persons/person_{idx}/mesh",- self._rr.Mesh3D(- vertex_positions=verts_cam,- triangle_indices=faces,- albedo_factor=self._mesh_albedo,- ),- )-- @staticmethod- def _draw_overlay(- image_rgb: np.ndarray,- persons: Sequence[HMRPrediction],- detector_ms: Optional[float],- hmr_ms: Optional[float],- render_ms: float,- total_ms: Optional[float],- contact_threshold: float = 0.5,- ) -> np.ndarray:- """Draw per-person bbox + 2D skeleton + a timing HUD onto the frame."""- out = image_rgb.copy()- h, w = out.shape[:2]-- for person in persons:- x1, y1, x2, y2 = person.bbox.astype(int)- cv2.rectangle(out, (x1, y1), (x2, y2), (0, 200, 255), 2)- label = f"{person.confidence:.2f}"- (tw, th), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)- cv2.rectangle(out, (x1, y1 - th - 6), (x1 + tw + 4, y1), (0, 200, 255), -1)- cv2.putText(- out, label, (x1 + 2, y1 - 4),- cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1,- )-- j2d = person.joints_2d- valid = (- (j2d[:, 0] >= 0) & (j2d[:, 0] < w)- & (j2d[:, 1] >= 0) & (j2d[:, 1] < h)- )-- for i, j in edges_for(j2d.shape[0]):- if valid[i] and valid[j]:- pt1 = tuple(j2d[i].astype(int))- pt2 = tuple(j2d[j].astype(int))- cv2.line(out, pt1, pt2, (255, 230, 0), 2)-- for k, (x, y) in enumerate(j2d):- if valid[k]:- cv2.circle(out, (int(x), int(y)), 3, (0, 230, 0), -1)-- if person.contact_prob is not None:- r = max(4, int(round(0.012 * max(h, w))))- for name, kp, prob in zip(FOOT_NAMES, FOOT_KP, person.contact_prob):- planted = prob > contact_threshold- for k in kp:- if valid[k]:- cv2.circle(out, tuple(j2d[k].astype(int)), r,- PLANTED_COLOR if planted else FREE_COLOR,- -1 if planted else 2)- k = kp[0]- if valid[k]:- cv2.putText(out, f"{name[0].upper()} {prob:.2f}",- (int(j2d[k, 0]) + r + 2, int(j2d[k, 1]) - r),- cv2.FONT_HERSHEY_SIMPLEX, max(0.5, r / 12),- PLANTED_COLOR if planted else FREE_COLOR, 2, cv2.LINE_AA)-- # ---- Timing HUD (top-left, semi-transparent black panel) ----- lines: list[str] = []- if detector_ms is not None:- lines.append(f"RF-DETR : {detector_ms:6.1f} ms")- if hmr_ms is not None:- lines.append(f"InstantHMR: {hmr_ms:6.1f} ms")- if render_ms > 0.0:- lines.append(f"MHR mesh : {render_ms:6.1f} ms")- if total_ms is not None:- fps = 1000.0 / total_ms if total_ms > 0 else 0.0- lines.append(f"Total : {total_ms:6.1f} ms ({fps:5.1f} fps)")- if lines:- font = cv2.FONT_HERSHEY_SIMPLEX- scale = max(0.5, min(w, h) / 1500.0)- thick = max(1, int(round(scale * 1.5)))- sizes = [- cv2.getTextSize(s, font, scale, thick)[0] for s in lines- ]- line_h = max(s[1] for s in sizes) + 8- box_w = max(s[0] for s in sizes) + 16- box_h = line_h * len(lines) + 8-- overlay = out.copy()- cv2.rectangle(overlay, (8, 8), (8 + box_w, 8 + box_h), (0, 0, 0), -1)- cv2.addWeighted(overlay, 0.55, out, 0.45, 0, out)-- y = 8 + line_h- for s in lines:- cv2.putText(- out, s, (16, y),- font, scale, (255, 255, 255), thick, cv2.LINE_AA,- )- y += line_h-- return out-- def _send_blueprint(self, first: HMRPrediction) -> None:- """Send the blueprint: camera image | 3D scene | timings (+ contact)."""+ rr.log("camera/image", rr.Image(draw_overlay(image_rgb, persons, timings)))+ if persons:+ depth = float(np.median([p.cam_trans[2] for p in persons]))+ rr.log("camera", rr.Pinhole(width=w, height=h, focal_length=persons[0].focal_length,+ principal_point=[w / 2, h / 2], image_plane_distance=max(0.25 * depth, 0.2)))+ if not self._layout_sent:+ self._send_layout(persons[0])+ self._layout_sent = True++ def _send_layout(self, first: HMRPrediction) -> None:import rerun.blueprint as rrb- rr = self._rr-- joints = first.joints_3d_cam- center = joints.mean(axis=0)- extent = float(np.abs(joints - center).max())- cam_distance = max(extent * 4.0, 1.5)- # Y-down convention: place the orbit camera "in front" along -Z.- eye_pos = [- float(center[0]),- float(center[1]),- float(center[2] - cam_distance),- ]- plots = [rrb.TimeSeriesView(origin="timing", name="Latency (ms)")]- if first.contact_prob is not None:- plots.append(rrb.TimeSeriesView(origin="contact",- name="Foot contact probability"))- blueprint = rrb.Blueprint(- rrb.Horizontal(- rrb.Vertical(- rrb.Spatial2DView(origin="camera/image", name="Camera"),- *plots,- row_shares=[3] + [1] * len(plots),- ),- rrb.Spatial3DView(- origin="/",- name="3D Scene",- eye_controls=rrb.EyeControls3D(- position=eye_pos,- look_target=[float(center[0]), float(center[1]), float(center[2])],- eye_up=[0.0, -1.0, 0.0],- kind=rrb.Eye3DKind.Orbital,- tracking_entity="world/persons/person_0",- ),- background=rrb.Background(color=[25, 25, 25]),- ),- column_shares=[2, 3],- ),- )- rr.send_blueprint(blueprint)+ centre = first.joints_3d_cam.mean(0)+ distance = max(4 * float(np.abs(first.joints_3d_cam - centre).max()), 1.5)+ self.rr.send_blueprint(rrb.Blueprint(rrb.Horizontal(+ rrb.Vertical(rrb.Spatial2DView(origin="camera/image", name="Camera"),+ rrb.TimeSeriesView(origin="timing", name="Latency (ms)"), row_shares=[3, 1]),+ rrb.Spatial3DView(origin="/", name="3D", background=rrb.Background(color=[25, 25, 25]),+ eye_controls=rrb.EyeControls3D(+ position=[float(centre[0]), float(centre[1]), float(centre[2] - distance)],+ look_target=centre.tolist(), eye_up=[0.0, -1.0, 0.0], kind=rrb.Eye3DKind.Orbital)),+ column_shares=[2, 3])))+++def draw_overlay(image_rgb: np.ndarray, persons: Sequence[HMRPrediction], timings: dict[str, float]) -> np.ndarray:+ """The frame with each person's box and 2D skeleton, and the stage timings top left."""+ out = image_rgb.copy()+ h, w = out.shape[:2]+ thick = max(2, round(min(h, w) / 400))+ for p in persons:+ x1, y1, x2, y2 = p.bbox.astype(int)+ cv2.rectangle(out, (x1, y1), (x2, y2), BOX_COLOR, thick)+ pts = p.joints_2d.astype(int)+ for a, b in SKELETON_EDGES:+ cv2.line(out, tuple(pts[a]), tuple(pts[b]), BONE_COLOR, thick, cv2.LINE_AA)+ for pt in pts:+ cv2.circle(out, tuple(pt), thick + 1, JOINT_COLOR, -1, cv2.LINE_AA)++ lines = [f"{name:<9s}{ms:6.1f} ms" for name, ms in timings.items()]+ total = sum(timings.values())+ if total > 0:+ lines.append(f"{'total':<9s}{total:6.1f} ms ({1000 / total:.0f} fps)")+ scale = max(0.5, min(h, w) / 1200)+ line_h = int(28 * scale)+ panel = out.copy()+ cv2.rectangle(panel, (8, 8), (int(300 * scale) + 16, 16 + line_h * len(lines)), (0, 0, 0), -1)+ cv2.addWeighted(panel, 0.55, out, 0.45, 0, out)+ for k, line in enumerate(lines):+ cv2.putText(out, line, (16, 8 + line_h * (k + 1)), cv2.FONT_HERSHEY_SIMPLEX, scale * 0.7,+ (255, 255, 255), max(1, round(scale * 1.5)), cv2.LINE_AA)+ return outdiff --git a/models/README.md b/models/README.mddeleted file mode 100644index e9541ce..0000000--- a/models/README.md+++ /dev/null@@ -1,140 +0,0 @@-----license: other-license_name: sam-license-license_link: https://github.com/facebookresearch/sam-3d-body/blob/main/LICENSE-library_name: onnx-pipeline_tag: keypoint-detection-tags:- - human-pose-estimation- - human-mesh-recovery- - 3d-pose-estimation- - onnx- - mhr- - sam3d- - repvit- - cliff-base_model: facebook/sam-3d-body-dinov3------# InstantHMR — model weights----A lightweight, ONNX-exportable MHR body regressor for **3D human pose-estimation and mesh recovery**: a RepViT-M1.5 backbone + a 9-token-cross-attention decoder + CLIFF camera conditioning.--Trained on the released ground-truth annotations of-[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)-— the human MHR fits used to build SAM 3D Body — predicting per-person MHR-parameters and 70 keypoints from a single 224×224 crop.--Demo, training, and inference code live at the main InstantHMR-repository: <https://github.com/mohamdev/InstantHMR>.--## Files--| File | Size | Purpose |-|---|---|---|-| `instanthmr.onnx` | ~77 MB | fp16 ONNX export — what the demo and `InstantHMR(...)` wrapper load. |-| `instanthmr.pth` | ~461 MB | PyTorch checkpoint for fine-tuning. |--Most users only need `instanthmr.onnx`. The `.pth` is included for people-who want to continue training, swap the head, or re-export with different-options — see [`training/README.md`](../training/README.md).--## ONNX I/O--- **Inputs**- - `image` — `(N, 3, 224, 224)` float32, ImageNet-normalised RGB.- - `cliff_cond` — `(N, 3)` float32, `[cx_norm, cy_norm, b_scale]` of the- detector bbox in the **full** frame (`cx_norm, cy_norm ∈ [-1, 1]`,- `b_scale = max(bw, bh) / max(W, H)`).-- **Outputs**- - `mhr_params` — `(N, 204)` MHR pose + scale parameters.- - `shape_params` — `(N, 45)` MHR identity blendshapes.- - `cam_trans` — `(N, 3)` body translation in camera frame (metres).- - `joints_2d` — `(N, 70, 2)` keypoints in normalised crop coords `[-1, 1]`.- - `joints_3d` — `(N, 70, 3)` body-centred 3D keypoints (metres, Y-down).--The 70 keypoints follow the MHR70 ordering — see-[`instanthmr/skeleton.py`](https://github.com/mohamdev/InstantHMR/blob/main/instanthmr/skeleton.py)-for joint names and skeleton edges.--## Quick start (ONNX)--Minimal end-to-end inference with `onnxruntime`, no extra dependencies:--```python-import cv2-import numpy as np-import onnxruntime as ort--IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)-IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)--# 1. Load the ONNX session (CUDA if available, otherwise CPU).-sess = ort.InferenceSession(- "instanthmr.onnx",- providers=["CUDAExecutionProvider", "CPUExecutionProvider"],-)--# 2. Load an RGB image and a person bbox [x1, y1, x2, y2].-image = cv2.cvtColor(cv2.imread("person.jpg"), cv2.COLOR_BGR2RGB)-H, W = image.shape[:2]-x1, y1, x2, y2 = 120, 40, 380, 600 # replace with your detector output--# 3. Square 1.2x crop around the bbox, resized to 224x224.-cx, cy = (x1 + x2) / 2, (y1 + y2) / 2-size = max(x2 - x1, y2 - y1) * 1.2-sx1, sy1 = int(cx - size / 2), int(cy - size / 2)-sx2, sy2 = int(cx + size / 2), int(cy + size / 2)-pad = [max(0, -sy1), max(0, sy2 - H), max(0, -sx1), max(0, sx2 - W)]-patch = image[max(0, sy1):min(H, sy2), max(0, sx1):min(W, sx2)]-patch = cv2.copyMakeBorder(patch, *pad, cv2.BORDER_CONSTANT, value=0)-crop = cv2.resize(patch, (224, 224), interpolation=cv2.INTER_LINEAR)--# 4. Normalise + CHW.-img = (crop.astype(np.float32) / 255.0 - IMAGENET_MEAN) / IMAGENET_STD-img = np.transpose(img, (2, 0, 1))[None] # (1, 3, 224, 224)--# 5. CLIFF conditioning vector (full-frame coords).-cliff = np.array([- 2.0 * cx / W - 1.0,- 2.0 * cy / H - 1.0,- max(x2 - x1, y2 - y1) / max(W, H),-], dtype=np.float32)[None] # (1, 3)--# 6. Run the model.-mhr_params, shape_params, cam_trans, joints_2d, joints_3d = sess.run(- None, {"image": img, "cliff_cond": cliff}-)--print(joints_3d.shape) # (1, 70, 3) — body-centred metres, Y-down-print(joints_2d.shape) # (1, 70, 2) — normalised crop coords [-1, 1]-print(mhr_params.shape) # (1, 204) — feed into Meta's MHR module for a mesh-```--For a full pipeline (RF-DETR person detector + InstantHMR + Rerun-visualiser), use the [`PosePipeline`](https://github.com/mohamdev/InstantHMR/blob/main/instanthmr/pipeline.py)-helper from the main repository — `pip install -r requirements.txt` then-`python demo.py --image path/to/photo.jpg`.--## Body mesh--InstantHMR does **not** run the MHR mesh decoder — rendering the full-mesh is resource-heavy and the default pipeline stays lightweight. The-raw `mhr_params` and `shape_params` are exposed so you can feed them-into Meta's MHR TorchScript module from the original-[SAM 3D Body release](https://github.com/facebookresearch/sam-3d-body)-when you need vertices.--## License--These weights are released under the **SAM license**, since InstantHMR-is trained on `facebook/sam-3d-body-dataset` labels. Refer to-<https://github.com/facebookresearch/sam-3d-body> for the full license-text and applicable use restrictions.--The accompanying source code in the InstantHMR repository is released-separately under the Apache License 2.0.diff --git a/models/instanthmr.gif b/models/instanthmr.gifdeleted file mode 100644index 564b817..0000000Binary files a/models/instanthmr.gif and /dev/null differdiff --git a/requirements-mhr.txt b/requirements-mhr.txtdeleted file mode 100644index 8951b7e..0000000--- a/requirements-mhr.txt+++ /dev/null@@ -1,43 +0,0 @@-# MHR body mesh rendering — install TOGETHER with requirements.txt-#-# IMPORTANT: always install both files in a single pip call so the resolver-# can pick a torch version compatible with rfdetr, torchvision, AND-# pymomentum-gpu at once. Installing them separately causes pip to first-# choose a torch version for rfdetr and then downgrade it for pymomentum,-# leaving torchvision broken.-#-# pip install -r requirements.txt -r requirements-mhr.txt-#-# Prerequisites-# --------------# * Python >= 3.12 (pymomentum has no Python 3.11 wheels; recreate the-# conda env if needed: conda create -n instanthmr python=3.12 -y)-# * CUDA toolkit on the system path (recommended for real-time use)-#-# After installing, also download the MHR body-model assets:-#-# curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip-# unzip assets.zip -d models/mhr_assets-#-# Then run:-#-# python demo.py --image photo.jpg --mhr-assets models/mhr_assets--# pymomentum — Meta's momentum geometry library (C++ Python bindings).-# This package brings its own torch constraint; do NOT add a separate-# torch>=X line here or pip will resolve torch twice and downgrade it.-#-# WARNING: "pip install pymomentum" installs an UNRELATED legacy SMS library-# (pyMomentum v0.1.x by MomentumAS). The correct package names are below.-#-# Platform markers handle the selection automatically:-# Linux / Windows → pymomentum-gpu (NVIDIA GPU)-# macOS → pymomentum-cpu (Apple Silicon / Intel)-pymomentum-gpu ; platform_system != "Darwin"-pymomentum-cpu ; platform_system == "Darwin"-#-# CPU-only Linux / Windows: comment out pymomentum-gpu above and uncomment:-# pymomentum-cpu ; platform_system != "Darwin"--# MHR body model (Meta's Momentum Human Rig)-mhr @ git+https://github.com/facebookresearch/MHR.gitdiff --git a/requirements.txt b/requirements.txtindex 1644ca1..1821d1c 100644--- a/requirements.txt+++ b/requirements.txt@@ -1,32 +1,7 @@-# Recommended install path is `python install.py`, which picks the right-# torch + onnxruntime wheels for your hardware (Linux + CUDA, Mac, or CPU).-# This file is a portable fallback that uses the CPU wheel of torch and the-# stock CPU build of onnxruntime — it works everywhere but leaves a lot of-# performance on the table on NVIDIA GPUs and Apple Silicon.--# ----- inference + viz (required by demo.py) ------numpy>=1.24-opencv-python>=4.8-pillow>=10-rfdetr>=1.0-rerun-sdk>=0.21--# Pulled in transitively by rfdetr; on Linux + CUDA you almost certainly-# want a CUDA wheel from the pytorch index instead — see install.py.-torch>=2.2-torchvision--# Mac ships CoreML EP inside plain onnxruntime; Linux GPU wants-# onnxruntime-gpu (which install.py picks). The CPU wheel below is the-# safe fallback.-onnxruntime>=1.19--# ----- MHR body mesh rendering ------# Optional. Requires Python >= 3.12 and a separate setup step.-# See requirements-mhr.txt and the "MHR body mesh" section of README.md.--# ----- training / annotation (optional, for reproduction) ------# timm>=1.0-# tqdm-# loguru-# (For annotation: install Meta's sam-3d-body separately — see docs/annotation.md)+numpy+opencv-python+huggingface_hub+rerun-sdk==0.31.*+rfdetr>=1.6+onnxruntime-gpu[cuda,cudnn]==1.26.* ; platform_system != "Darwin"+onnxruntime==1.26.* ; platform_system == "Darwin"diff --git a/tools/annotate_dataset.py b/tools/annotate_dataset.pydeleted file mode 100755index 7298948..0000000--- a/tools/annotate_dataset.py+++ /dev/null@@ -1,876 +0,0 @@-#!/usr/bin/env python3-"""-Cloud-grade annotation pipeline for CLIFF-style student distillation.--Scans a root directory tree for images, runs YOLO person detection + SAM3D-teacher inference, and saves tight 224×224 crops alongside full-frame-aware-annotations (.npz) ready for CLIFF-style training.--Output layout:- <output_root>/- images/ <datasetName>_<imageStem>_p<personID>.jpg (224×224 crop)- annotations/ <datasetName>_<imageStem>_p<personID>.npz--NPZ payload (per crop):- orig_shape (2,) [H, W] of the full uncropped image- bbox (4,) [x1, y1, x2, y2] raw YOLO detection in original image coords- (used by CLIFF student for bbox_center / bbox_scale conditioning)- bbox_square (4,) [x1, y1, x2, y2] expanded square crop (1.2×) used for 224×224 resize- cam_focal_length (2,) [fx, fy] where f = sqrt(H² + W²)- cam_trans (3,) [tx, ty, tz] from SAM3D (computed with raw bbox, full-frame perspective)- mhr_model_params (204,) pose + scale params- shape_params (45,) identity blendshapes- joints_3d (70, 3) regressed 3D joints (Y-down camera convention, meters)- joints_2d (70, 2) projected into ORIGINAL full-frame pixel space--Resume-safe: existing annotations are skipped automatically.--Usage:- python tools/annotate_dataset_sam3d_cloud.py \\- --root_dir /data/training_images \\- --output_dir /data/cliff_annotations \\- --confidence 0.7-- # Quick diverse subset (50k crops sampled across all datasets):- python tools/annotate_dataset_sam3d_cloud.py \\- --root_dir /data/training_images \\- --output_dir /data/cliff_annotations \\- --shuffle --max_samples 50000-- # Structure of --root_dir:- # /data/training_images/- # Harmony4D/ *.jpg *.png ...- # COCO/ *.jpg ...- # Synthetic/ *.png ...-"""--from __future__ import annotations--import argparse-import gc-import math-import os-import random-import signal-import sys-import time-from contextlib import contextmanager-from pathlib import Path--# Ensure project root is importable when running from repo root.-_PROJECT_ROOT = Path(__file__).resolve().parent.parent-if str(_PROJECT_ROOT) not in sys.path:- sys.path.insert(0, str(_PROJECT_ROOT))--import cv2-import numpy as np-import torch-from loguru import logger-from tqdm import tqdm---# ----------------------------------------------------------------------------# Constants-# ----------------------------------------------------------------------------IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".bmp", ".webp"}-CROP_SIZE = 224---# ----------------------------------------------------------------------------# Rerun helpers-# ----------------------------------------------------------------------------def setup_rerun_blueprint():- """Send a 3-panel Rerun blueprint: raw image | overlay | 3D mesh."""- import rerun as rr- import rerun.blueprint as rrb-- blueprint = rrb.Blueprint(- rrb.Horizontal(- rrb.Spatial2DView(- origin="raw_image",- name="Raw Image",- ),- rrb.Spatial2DView(- origin="overlay/image",- name="Overlay",- ),- rrb.Spatial3DView(- origin="scene_3d",- name="3D Mesh",- background=rrb.Background(color=[25, 25, 25]),- ),- ),- )- rr.send_blueprint(blueprint)---_MAX_RERUN_PERSONS = 10 # upper bound for clearing stale entities---def log_to_rerun(- image_rgb: np.ndarray,- detections: list[dict],- sam3d_outputs: list[dict | None],- faces: np.ndarray | None,-) -> None:- """Log one image with all its person detections to the Rerun viewer.-- All entities are logged as static so Rerun only ever holds one frame in- memory — previous data is overwritten, not accumulated.-- Args:- image_rgb: Full uncropped image (H, W, 3), uint8 RGB.- detections: YOLO detection dicts with ``"bbox"`` keys.- sam3d_outputs: Parallel list of raw SAM3D output dicts (may contain None).- faces: Mesh triangle indices from the SAM3D teacher.- """- import rerun as rr-- h, w = image_rgb.shape[:2]-- # Clear all person entities from the previous frame to avoid stale meshes- for i in range(_MAX_RERUN_PERSONS):- rr.log(f"scene_3d/person_{i}", rr.Clear(recursive=True), static=True)- rr.log("scene_3d/camera", rr.Clear(recursive=True), static=True)-- # Panel 1: raw image- rr.log("raw_image", rr.Image(image_rgb), static=True)-- overlay = image_rgb.copy()- first_cam_t = None- first_fx, first_fy = None, None- first_joints_cam = None-- for person_id, (det, sam3d_out) in enumerate(zip(detections, sam3d_outputs)):- if sam3d_out is None:- continue-- # Extract outputs- cam_t = sam3d_out.get("pred_cam_t")- if cam_t is not None and hasattr(cam_t, "cpu"):- cam_t = cam_t.cpu().numpy()- cam_t = np.asarray(cam_t, dtype=np.float32).ravel()[:3] if cam_t is not None else np.zeros(3, dtype=np.float32)-- focal = sam3d_out.get("focal_length")- if focal is not None and hasattr(focal, "cpu"):- focal = focal.cpu().numpy()- if focal is not None:- focal = np.asarray(focal, dtype=np.float32).ravel()- fx = float(focal[0])- fy = float(focal[-1]) # handles both (1,) and (2,)- else:- f = math.sqrt(h * h + w * w)- fx, fy = f, f- cx, cy = w / 2.0, h / 2.0-- if first_cam_t is None:- first_cam_t = cam_t- first_fx, first_fy = fx, fy-- # Vertices- verts = sam3d_out.get("pred_vertices", sam3d_out.get("vertices", sam3d_out.get("verts")))- if verts is not None:- if hasattr(verts, "cpu"):- verts = verts.cpu().numpy()- verts = np.asarray(verts, dtype=np.float32)- if verts.ndim == 3 and verts.shape[0] == 1:- verts = verts[0]-- verts_cam = verts + cam_t-- # Panel 2: project mesh onto overlay- z = verts_cam[:, 2]- valid = z > 0.01- u = (fx * verts_cam[valid, 0] / z[valid] + cx).astype(int)- v = (fy * verts_cam[valid, 1] / z[valid] + cy).astype(int)- mask = (u >= 0) & (u < w) & (v >= 0) & (v < h)- for px, py in zip(u[mask], v[mask]):- cv2.circle(overlay, (px, py), 1, (0, 255, 128), -1)-- # Panel 3: 3D mesh- person_path = f"scene_3d/person_{person_id}"- if faces is not None:- rr.log(- f"{person_path}/mesh",- rr.Mesh3D(- vertex_positions=verts_cam,- triangle_indices=faces,- albedo_factor=[180, 200, 220, 200],- ),- static=True,- )- else:- rr.log(- f"{person_path}/mesh",- rr.Points3D(positions=verts_cam, radii=0.003, colors=[180, 200, 220]),- static=True,- )-- # 3D joints- joints_3d = sam3d_out.get("pred_keypoints_3d")- if joints_3d is not None:- if hasattr(joints_3d, "cpu"):- joints_3d = joints_3d.cpu().numpy()- joints_3d = np.asarray(joints_3d, dtype=np.float32)- if joints_3d.ndim == 3 and joints_3d.shape[0] == 1:- joints_3d = joints_3d[0]- joints_cam = joints_3d + cam_t- if first_joints_cam is None:- first_joints_cam = joints_cam- rr.log(- f"scene_3d/person_{person_id}/joints",- rr.Points3D(positions=joints_cam, radii=0.008, colors=[0, 255, 0]),- static=True,- )-- rr.log("overlay/image", rr.Image(overlay), static=True)-- # Log Pinhole camera in the 3D scene so the image appears as a properly- # sized frustum. Resolution must match the full frame (not the 224 crop)- # because cam_focal_length is computed from sqrt(H² + W²).- if first_cam_t is not None:- mesh_depth = float(first_joints_cam[:, 2].mean()) if first_joints_cam is not None else None- image_plane_dist = max(mesh_depth * 0.25, 0.2) if mesh_depth else 0.5-- rr.log(- "scene_3d/camera",- rr.Pinhole(- resolution=[w, h],- focal_length=[first_fx, first_fy],- principal_point=[w / 2.0, h / 2.0],- image_plane_distance=image_plane_dist,- ),- static=True,- )- rr.log("scene_3d/camera/image", rr.Image(image_rgb), static=True)---# ----------------------------------------------------------------------------# Timeout helper (POSIX — uses SIGALRM)-# ----------------------------------------------------------------------------class InferenceTimeout(Exception):- pass---@contextmanager-def time_limit(seconds: float):- """Raise InferenceTimeout if the block takes longer than *seconds*."""- def _handler(signum, frame):- raise InferenceTimeout(f"Inference exceeded {seconds}s timeout")-- old = signal.signal(signal.SIGALRM, _handler)- signal.setitimer(signal.ITIMER_REAL, seconds)- try:- yield- finally:- signal.setitimer(signal.ITIMER_REAL, 0)- signal.signal(signal.SIGALRM, old)---# ----------------------------------------------------------------------------# Image discovery-# ----------------------------------------------------------------------------def discover_images(root_dir: Path) -> list[tuple[str, Path]]:- """Walk *root_dir* and return ``(dataset_name, image_path)`` pairs.-- ``dataset_name`` is the name of the first-level subdirectory under- *root_dir*. Images directly in *root_dir* get dataset_name ``"root"``.- """- entries: list[tuple[str, Path]] = []- root_dir = root_dir.resolve()-- for path in sorted(root_dir.rglob("*")):- if not path.is_file():- continue- if path.suffix.lower() not in IMAGE_EXTENSIONS:- continue-- # Derive dataset name from the first child directory- try:- relative = path.relative_to(root_dir)- parts = relative.parts- dataset_name = parts[0] if len(parts) > 1 else "root"- except ValueError:- dataset_name = "root"-- entries.append((dataset_name, path))-- return entries---# ----------------------------------------------------------------------------# YOLO detector (standalone, no pipeline dependency)-# ----------------------------------------------------------------------------class YOLOPersonDetector:- """Thin wrapper around ultralytics YOLO for person detection."""-- def __init__(- self,- model_name: str = "yolo11n.pt",- device: str = "cuda:0",- confidence: float = 0.7,- ):- from ultralytics import YOLO-- self._model = YOLO(model_name)- self._device = device- self._confidence = confidence-- def detect(self, image_rgb: np.ndarray) -> list[dict]:- """Return list of ``{"bbox": [x1,y1,x2,y2], "confidence": float}``."""- results = self._model(- image_rgb,- device=self._device,- classes=[0], # COCO person- conf=self._confidence,- verbose=False,- )- detections = []- for r in results:- if r.boxes is None:- continue- for box, conf in zip(- r.boxes.xyxy.cpu().numpy(),- r.boxes.conf.cpu().numpy(),- ):- detections.append({- "bbox": box.astype(np.float32),- "confidence": float(conf),- })- detections.sort(key=lambda d: d["confidence"], reverse=True)- return detections---# ----------------------------------------------------------------------------# SAM3D teacher (uses the estimator directly for max control)-# ----------------------------------------------------------------------------class SAM3DTeacher:- """Loads the SAM3D teacher and exposes a simple per-crop inference API.-- We call the ``SAM3DBodyEstimator`` directly (not via the pipeline wrapper)- so that we have access to the raw output dict with ``pred_cam_t``,- ``focal_length``, ``mhr_model_params``, etc.- """-- def __init__(- self,- checkpoint_path: str,- mhr_path: str,- hf_repo_id: str = "facebook/sam-3d-body-dinov3",- device: str = "cuda:0",- ):- self.device = device-- # Add sam-3d-body to path- for candidate in [- Path.cwd() / "sam-3d-body",- Path.cwd().parent / "sam-3d-body",- Path.home() / "sam-3d-body",- ]:- if candidate.exists() and (candidate / "sam_3d_body").exists():- if str(candidate) not in sys.path:- sys.path.insert(0, str(candidate))- break-- from sam_3d_body import SAM3DBodyEstimator-- ckpt = Path(checkpoint_path)- if ckpt.exists():- from sam_3d_body import load_sam_3d_body- logger.info(f"Loading SAM3D from local checkpoint: {ckpt}")- model, cfg = load_sam_3d_body(- checkpoint_path=str(ckpt),- mhr_path=mhr_path,- )- else:- from sam_3d_body import load_sam_3d_body_hf- logger.info(f"Loading SAM3D from HuggingFace: {hf_repo_id}")- model, cfg = load_sam_3d_body_hf(hf_repo_id=hf_repo_id)-- model = model.to(device).eval()- self._estimator = SAM3DBodyEstimator(model, cfg)- self._faces = self._estimator.faces-- logger.info(f"SAM3D teacher ready on {device}")-- @property- def faces(self) -> np.ndarray:- return self._faces-- @torch.inference_mode()- def predict_person(- self,- full_image_rgb: np.ndarray,- bbox: np.ndarray,- ) -> dict | None:- """Run SAM3D on one person in *full_image_rgb*.-- We pass the **full image** with the bbox in full-frame coordinates.- SAM3D's CLIFF camera head then uses the bbox position, bbox scale,- and full-image dimensions to compute ``pred_cam_t`` and- ``focal_length`` natively in full-frame perspective space — no- manual coordinate conversion needed.-- Args:- full_image_rgb: The original uncropped image (H, W, 3), uint8 RGB.- bbox: ``[x1, y1, x2, y2]`` in full-image pixel coords.-- Returns:- Raw SAM3D output dict for the single person, or ``None`` on- failure.- """- bbox_arr = np.array(bbox, dtype=np.float32).reshape(1, 4)-- output_list = self._estimator.process_one_image(- full_image_rgb,- bboxes=bbox_arr,- inference_type="body",- )-- if not output_list:- return None-- return output_list[0]---# ----------------------------------------------------------------------------# Crop & annotation writer-# ----------------------------------------------------------------------------def make_annotation_name(- dataset_name: str,- image_stem: str,- person_id: int,-) -> str:- """Deterministic filename: ``<dataset>_<stem>_p<id>``."""- # Sanitise: replace path separators / spaces that may leak from subfolders- safe_stem = image_stem.replace(os.sep, "_").replace(" ", "_")- return f"{dataset_name}_{safe_stem}_p{person_id}"---def get_square_crop_padded(image: np.ndarray, bbox: np.ndarray, expand: float = 1.2) -> tuple[np.ndarray, np.ndarray]:- """- Extracts a perfectly square crop from the image, padding with black pixels- if the crop goes outside the image boundaries.- """- h, w = image.shape[:2]- x1, y1, x2, y2 = bbox.astype(float)-- cx, cy = (x1 + x2) / 2.0, (y1 + y2) / 2.0- size = max(x2 - x1, y2 - y1) * expand- half = size / 2.0-- # The perfect square coordinates (may fall outside the image)- sq_x1, sq_y1 = int(cx - half), int(cy - half)- sq_x2, sq_y2 = int(cx + half), int(cy + half)-- # Calculate padding needed if the square goes out of bounds- pad_top = max(0, -sq_y1)- pad_bottom = max(0, sq_y2 - h)- pad_left = max(0, -sq_x1)- pad_right = max(0, sq_x2 - w)-- # Extract the valid portion of the image- valid_x1, valid_y1 = max(0, sq_x1), max(0, sq_y1)- valid_x2, valid_y2 = min(w, sq_x2), min(h, sq_y2)- crop_valid = image[valid_y1:valid_y2, valid_x1:valid_x2]-- # Pad the valid crop to make it perfectly square- crop_square = cv2.copyMakeBorder(- crop_valid, pad_top, pad_bottom, pad_left, pad_right,- cv2.BORDER_CONSTANT, value=(0, 0, 0)- )-- sq_bbox = np.array([sq_x1, sq_y1, sq_x2, sq_y2], dtype=np.float32)- return crop_square, sq_bbox---def save_crop_and_annotation(- full_image_rgb: np.ndarray,- bbox: np.ndarray,- sam3d_output: dict,- name: str,- images_dir: Path,- annotations_dir: Path,-) -> None:- """Resize the person crop to 224x224, save as JPEG, and write the .npz."""- h, w = full_image_rgb.shape[:2]-- # Extract perfect square crop (padded if out of bounds)- crop_square, sq_bbox = get_square_crop_padded(full_image_rgb, bbox, expand=1.2)-- # Resize to 224x224 and save as lossless PNG (RGB -> BGR for cv2)- crop_224 = cv2.resize(crop_square, (CROP_SIZE, CROP_SIZE), interpolation=cv2.INTER_LINEAR)- crop_bgr = cv2.cvtColor(crop_224, cv2.COLOR_RGB2BGR)- cv2.imwrite(str(images_dir / f"{name}.png"), crop_bgr)-- # ... (The rest of Claude's function below this line stays exactly the same!) ...- f = math.sqrt(h * h + w * w)-- # Extract SAM3D outputs- cam_trans = sam3d_output.get("pred_cam_t")- if cam_trans is not None and hasattr(cam_trans, "numpy"):- cam_trans = cam_trans.numpy()- cam_trans = np.asarray(cam_trans, dtype=np.float32).ravel()[:3]-- focal = sam3d_output.get("focal_length")- if focal is not None and hasattr(focal, "numpy"):- focal = focal.numpy()- if focal is not None:- focal = np.asarray(focal, dtype=np.float32).ravel()- if focal.size == 1:- focal = np.array([focal[0], focal[0]], dtype=np.float32)- else:- focal = np.array([f, f], dtype=np.float32)-- joints_3d = _to_np(sam3d_output.get("pred_keypoints_3d")) # (70, 3)- joints_2d = _to_np(sam3d_output.get("pred_keypoints_2d")) # (70, 2)- mhr_model_params = _to_np(sam3d_output.get("mhr_model_params")) # (204,)- shape_params = _to_np(sam3d_output.get("shape_params")) # (45,)-- # Build the annotation payload- data: dict[str, np.ndarray] = {- "orig_shape": np.array([h, w], dtype=np.int32),- "bbox": bbox.astype(np.float32), # raw YOLO detection — CLIFF student needs this- "bbox_square": sq_bbox, # expanded square crop used for the 224×224 resize- "cam_focal_length": focal.astype(np.float32),- "cam_trans": cam_trans,- }- if mhr_model_params is not None:- data["mhr_model_params"] = mhr_model_params.astype(np.float32)- if shape_params is not None:- data["shape_params"] = shape_params.astype(np.float32)- if joints_3d is not None:- data["joints_3d"] = joints_3d.astype(np.float32)- if joints_2d is not None:- data["joints_2d"] = joints_2d.astype(np.float32)-- np.savez(annotations_dir / f"{name}.npz", **data)---def _to_np(x) -> np.ndarray | None:- """Convert torch.Tensor / np.ndarray to a squeezed numpy array."""- if x is None:- return None- if hasattr(x, "cpu"):- x = x.cpu().numpy()- x = np.asarray(x, dtype=np.float32)- # Squeeze leading batch dim if present: (1, ...) → (...)- if x.ndim >= 2 and x.shape[0] == 1:- x = x[0]- return x---# ----------------------------------------------------------------------------# CLI-# ----------------------------------------------------------------------------def parse_args() -> argparse.Namespace:- p = argparse.ArgumentParser(- description="Cloud-grade CLIFF annotation pipeline (YOLO + SAM3D)",- formatter_class=argparse.RawDescriptionHelpFormatter,- epilog=__doc__,- )-- # I/O- p.add_argument(- "--root_dir", type=str, required=True,- help="Root directory containing dataset subdirectories of images.",- )- p.add_argument(- "--output_dir", type=str, required=True,- help="Output directory (images/ and annotations/ created inside).",- )-- # Detection- p.add_argument(- "--yolo_model", type=str, default="yolo11n.pt",- help="YOLO model name or path (default: yolo11n.pt).",- )- p.add_argument(- "--confidence", type=float, default=0.7,- help="YOLO person detection confidence threshold (default: 0.7).",- )- p.add_argument(- "--max_persons", type=int, default=10,- help="Max persons to annotate per image (default: 10).",- )-- # SAM3D teacher- p.add_argument(- "--checkpoint_path", type=str,- default="checkpoints/sam-3d-body-dinov3/model.ckpt",- )- p.add_argument(- "--mhr_path", type=str,- default="checkpoints/sam-3d-body-dinov3/assets/mhr_model.pt",- )- p.add_argument(- "--hf_repo", type=str, default="facebook/sam-3d-body-dinov3",- )-- # Performance- p.add_argument(- "--timeout", type=float, default=5.0,- help="Per-image timeout in seconds (default: 5.0).",- )- p.add_argument(- "--gc_interval", type=int, default=200,- help="Force GC + CUDA cache clear every N images (default: 200).",- )- p.add_argument(- "--max_samples", type=int, default=None,- help="Stop after producing N annotation crops (not N images).",- )-- # Misc- p.add_argument("--device", type=str, default="cuda:0")- p.add_argument("--debug", action="store_true")- p.add_argument(- "--shuffle", action="store_true",- help="Shuffle image order for diverse sampling across datasets.",- )- p.add_argument(- "--use_rerun", action="store_true",- help="Open Rerun viewer with 3 panels: raw image, overlay, 3D mesh.",- )-- return p.parse_args()---# ----------------------------------------------------------------------------# Main-# ----------------------------------------------------------------------------def main() -> None:- args = parse_args()-- # Logging- logger.remove()- fmt = (- "<green>{time:HH:mm:ss}</green> | <level>{level:<8}</level> | "- "<cyan>{function}</cyan> - <level>{message}</level>"- )- logger.add(sys.stderr, level="DEBUG" if args.debug else "INFO", format=fmt)-- root_dir = Path(args.root_dir)- output_dir = Path(args.output_dir)- images_dir = output_dir / "images"- annotations_dir = output_dir / "annotations"- images_dir.mkdir(parents=True, exist_ok=True)- annotations_dir.mkdir(parents=True, exist_ok=True)-- # ------------------------------------------------------------------- # 1. Discover images- # ------------------------------------------------------------------- logger.info(f"Scanning {root_dir} for images ...")- all_images = discover_images(root_dir)- if not all_images:- logger.error(f"No images found under {root_dir}")- return-- if args.shuffle:- random.shuffle(all_images)- logger.info("Image order shuffled for diverse cross-dataset sampling")-- logger.info(f"Found {len(all_images)} images across {len(set(d for d, _ in all_images))} dataset(s)")- if args.max_samples:- logger.info(f"Will stop after {args.max_samples} annotation crops")-- # ------------------------------------------------------------------- # 2. Build resume set — collect existing annotation stems- # ------------------------------------------------------------------- existing_stems: set[str] = set()- for p in annotations_dir.glob("*.npz"):- existing_stems.add(p.stem)-- if existing_stems:- logger.info(f"Resume: {len(existing_stems)} annotations already on disk")-- # Count existing annotations toward the budget so resumed runs stop correctly- total_crops = len(existing_stems)-- # ------------------------------------------------------------------- # 3. Load models- # ------------------------------------------------------------------- logger.info("Loading YOLO detector ...")- detector = YOLOPersonDetector(- model_name=args.yolo_model,- device=args.device,- confidence=args.confidence,- )-- logger.info("Loading SAM3D teacher ...")- teacher = SAM3DTeacher(- checkpoint_path=args.checkpoint_path,- mhr_path=args.mhr_path,- hf_repo_id=args.hf_repo,- device=args.device,- )-- # ------------------------------------------------------------------- # 3b. Rerun setup- # ------------------------------------------------------------------- use_rerun = args.use_rerun- if use_rerun:- try:- import rerun as rr- rr.init("sam3d_cloud_annotation", spawn=True)- rr.disable_timeline("frame")- rr.log("scene_3d", rr.ViewCoordinates.RIGHT_HAND_Y_DOWN, static=True)- setup_rerun_blueprint()- logger.info("Rerun viewer launched (3-panel layout, no history)")- except ImportError:- logger.warning("rerun-sdk not installed, disabling --use_rerun")- use_rerun = False-- # ------------------------------------------------------------------- # 4. Processing loop- # ------------------------------------------------------------------- stats = {"processed": 0, "skipped": 0, "failed": 0, "timed_out": 0, "crops": 0}- t0 = time.perf_counter()-- pbar = tqdm(all_images, desc="Annotating", unit="img", dynamic_ncols=True)- for dataset_name, img_path in pbar:- # Use relative path from root so subfolder structure prevents collisions- # e.g. COCO/train/000001.jpg → "train_000001" (dataset_name = "COCO")- rel_path = img_path.relative_to(root_dir)- image_stem = str(rel_path.with_suffix("")).replace(os.sep, "_").replace(" ", "_")- # Strip the dataset prefix since make_annotation_name re-adds it- if image_stem.startswith(dataset_name + "_"):- image_stem = image_stem[len(dataset_name) + 1:]-- # Quick check: can we skip the entire image?- # We check for person 0; if it exists the image was processed before.- first_name = make_annotation_name(dataset_name, image_stem, 0)- if first_name in existing_stems:- stats["skipped"] += 1- continue-- # Load image- img_bgr = cv2.imread(str(img_path))- if img_bgr is None:- logger.warning(f"Cannot read {img_path}")- stats["failed"] += 1- continue- img_rgb = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB)- H, W = img_rgb.shape[:2]-- # Detect persons- try:- with time_limit(args.timeout):- detections = detector.detect(img_rgb)- except InferenceTimeout:- logger.warning(f"YOLO timeout on {img_path.name}")- stats["timed_out"] += 1- continue- except Exception as e:- logger.warning(f"YOLO failed on {img_path.name}: {e}")- stats["failed"] += 1- continue-- if not detections:- stats["processed"] += 1- continue-- detections = detections[: args.max_persons]-- # Process each detected person — pass the FULL image so SAM3D's- # CLIFF camera head uses the correct bbox_center, bbox_size, and- # img_size to produce cam_trans in full-frame perspective space.- rerun_sam3d_outputs: list[dict | None] = []- for person_id, det in enumerate(detections):- name = make_annotation_name(dataset_name, image_stem, person_id)-- # Skip if this specific crop already exists- if name in existing_stems:- rerun_sam3d_outputs.append(None)- continue-- bbox = det["bbox"]-- try:- with time_limit(args.timeout):- sam3d_out = teacher.predict_person(img_rgb, bbox)- except InferenceTimeout:- logger.warning(f"SAM3D timeout on {name}")- stats["timed_out"] += 1- rerun_sam3d_outputs.append(None)- continue- except Exception as e:- logger.warning(f"SAM3D failed on {name}: {e}")- torch.cuda.empty_cache()- if args.debug:- raise- stats["failed"] += 1- rerun_sam3d_outputs.append(None)- continue-- if sam3d_out is None:- stats["failed"] += 1- rerun_sam3d_outputs.append(None)- continue-- # Validate essential outputs- if sam3d_out.get("mhr_model_params") is None:- logger.warning(f"No mhr_model_params for {name} (keys: {list(sam3d_out.keys())})")- stats["failed"] += 1- rerun_sam3d_outputs.append(None)- continue-- try:- save_crop_and_annotation(- img_rgb, bbox, sam3d_out, name,- images_dir, annotations_dir,- )- stats["crops"] += 1- total_crops += 1- except Exception as e:- logger.warning(f"Failed to save {name}: {e}")- if args.debug:- raise- stats["failed"] += 1- rerun_sam3d_outputs.append(None)- continue-- rerun_sam3d_outputs.append(sam3d_out)-- # Rerun visualization (after all persons for this image)- if use_rerun and any(o is not None for o in rerun_sam3d_outputs):- log_to_rerun(img_rgb, detections, rerun_sam3d_outputs, teacher.faces)-- stats["processed"] += 1-- # Stop once we have enough crops- if args.max_samples and total_crops >= args.max_samples:- logger.info(f"Reached {total_crops} total crops (--max_samples {args.max_samples}), stopping")- break-- # Periodic cleanup- if stats["processed"] % args.gc_interval == 0:- torch.cuda.empty_cache()- gc.collect()-- # Update progress bar- pbar.set_postfix(- crops=stats["crops"],- fail=stats["failed"],- skip=stats["skipped"],- )-- elapsed = time.perf_counter() - t0- hours = elapsed / 3600-- logger.info("=" * 60)- logger.info("ANNOTATION COMPLETE")- logger.info("=" * 60)- logger.info(f" Images scanned: {len(all_images)}")- logger.info(f" Images processed: {stats['processed']}")- logger.info(f" Images skipped: {stats['skipped']} (already annotated)")- logger.info(f" Crops saved: {stats['crops']} (total on disk: {total_crops})")- logger.info(f" Failures: {stats['failed']}")- logger.info(f" Timeouts: {stats['timed_out']}")- logger.info(f" Wall time: {hours:.2f}h ({elapsed / max(stats['processed'], 1):.2f}s/img)")- logger.info(f" Output: {output_dir}")- logger.info("=" * 60)---if __name__ == "__main__":- main()diff --git a/tools/bench.py b/tools/bench.pydeleted file mode 100644index 115ee0b..0000000--- a/tools/bench.py+++ /dev/null@@ -1,138 +0,0 @@-#!/usr/bin/env python3-"""Headless benchmark for the InstantHMR pipeline.--Runs ``PosePipeline`` on a video and prints per-stage latency stats with no-Rerun, no GUI, no .rrd writing — so the numbers reflect inference cost and-not visualisation overhead.--Usage:- python tools/bench.py --video vid1.mp4 [--device cuda] [--detector-variant medium]- [--max-frames 200] [--warmup 10]-"""--from __future__ import annotations--import argparse-import statistics-import sys-import time-from pathlib import Path--import cv2-import numpy as np--# Make the package importable when run from anywhere.-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))--from instanthmr import PosePipeline---def parse_args() -> argparse.Namespace:- p = argparse.ArgumentParser()- p.add_argument("--video", type=str, required=True)- p.add_argument("--model", type=str, default="models/instanthmr.onnx")- p.add_argument("--device", type=str, default="cuda")- p.add_argument("--detector-variant", type=str, default="medium",- choices=["nano", "small", "medium", "base", "large"])- p.add_argument("--det-confidence", type=float, default=0.5)- p.add_argument("--max-persons", type=int, default=5)- p.add_argument("--detector-stride", type=int, default=1,- help="Run RF-DETR every N frames (default 1 = every frame).")- p.add_argument("--no-batch-persons", action="store_true",- help="Disable batched multi-person HMR (one ORT call per person).")- p.add_argument("--max-frames", type=int, default=200,- help="Cap measured frames (after warm-up).")- p.add_argument("--warmup", type=int, default=10,- help="Frames discarded before timing starts.")- return p.parse_args()---def _summary(name: str, samples: list[float]) -> str:- if not samples:- return f"{name}: no samples"- arr = np.asarray(samples, dtype=np.float64)- return (- f"{name:>14s} "- f"mean={arr.mean():6.2f} ms "- f"median={np.median(arr):6.2f} ms "- f"p95={np.percentile(arr, 95):6.2f} ms "- f"min={arr.min():6.2f} ms "- f"max={arr.max():6.2f} ms"- )---def main() -> None:- args = parse_args()-- cap = cv2.VideoCapture(args.video)- if not cap.isOpened():- sys.exit(f"[error] cannot open video: {args.video}")-- src_fps = cap.get(cv2.CAP_PROP_FPS) or 30.0- total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))- width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))- height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))- print(f"video: {args.video} {width}x{height} {total} frames @ {src_fps:.1f} fps")-- pipeline = PosePipeline(- onnx_path=args.model,- device=args.device,- detector_variant=args.detector_variant,- det_confidence=args.det_confidence,- max_persons=args.max_persons,- detector_stride=args.detector_stride,- batch_persons=not args.no_batch_persons,- )- print(f"InstantHMR EP: {pipeline.hmr.active_provider}")- print(f"detector variant: {args.detector_variant}")-- det: list[float] = []- hmr: list[float] = []- tot: list[float] = []- wall: list[float] = []- n_persons: list[int] = []-- idx = 0- measured = 0- last_wall = None- while True:- ret, bgr = cap.read()- if not ret:- break-- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)- t_wall = time.perf_counter()- result = pipeline.predict(rgb)-- if idx >= args.warmup:- det.append(result.detector_ms)- hmr.append(result.hmr_ms)- tot.append(result.total_ms)- n_persons.append(len(result.persons))- if last_wall is not None:- wall.append((t_wall - last_wall) * 1000.0)- measured += 1- last_wall = t_wall- idx += 1- if measured >= args.max_frames:- break-- cap.release()-- if not tot:- sys.exit("[error] no measured frames — increase --max-frames or shorten --warmup")-- print()- print(f"measured frames: {len(tot)} (warmup={args.warmup})")- print(f"avg persons / frame: {statistics.mean(n_persons):.2f}")- print(_summary("detector_ms", det))- print(_summary("hmr_ms", hmr))- print(_summary("total_ms", tot))- if wall:- print(_summary("wall_ms", wall))- mean_total = float(np.mean(tot))- print(f"\nthroughput (1 / mean total): {1000.0 / mean_total:5.1f} FPS")---if __name__ == "__main__":- main()diff --git a/tools/export_onnx.py b/tools/export_onnx.pynew file mode 100644index 0000000..0e169c9--- /dev/null+++ b/tools/export_onnx.py@@ -0,0 +1,108 @@+#!/usr/bin/env python3+"""Export InstantHMR to ONNX: one network graph per checkpoint, and one shared MHR body graph.++network (--ckpt): image (B, 3, S, S) ImageNet-normalised RGB crop, cliff_cond (B, 3)+ -> mhr_params (B, 204), shape_params (B, 45), cam_trans (B, 3),+ joints_2d (B, 70, 2) in [-1, 1] over the crop+body (--body): mhr_params, shape_params -> joints_3d (B, 70, 3), vertices (B, 18439, 3),+ metres, body-centred (add cam_trans for camera space)++The network graph stores its crop size and conditioning form in the model metadata, the body+graph its mesh faces, so instanthmr.InstantHMR configures itself from the files.++ python tools/export_onnx.py --ckpt runs/medium/best_student_model_v3.pth --output models/instanthmr-medium.onnx+ python tools/export_onnx.py --body --output models/mhr-body.onnx+"""+import argparse+import base64+import sys+from pathlib import Path++import numpy as np+import onnx+import onnxruntime as ort+import torch++REPO = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(REPO / "training"))++import train as T # noqa: E402+from mhr_body import MHRBody # noqa: E402++NETWORK_OUTPUTS = ["mhr_params", "shape_params", "cam_trans", "joints_2d"]+BODY_OUTPUTS = ["joints_3d", "vertices"]+++class Network(torch.nn.Module):+ def __init__(self, net):+ super().__init__()+ self.net = net++ def forward(self, image, cliff_cond):+ out = self.net(image, cliff_cond)+ return tuple(out[k] for k in NETWORK_OUTPUTS)+++def load_network(ckpt: Path):+ state = torch.load(ckpt, map_location="cpu", weights_only=False)+ state = {k.replace("_orig_mod.", ""): v for k, v in state.get("model_state_dict", state).items()}+ if "head_2d_logits.weight" not in state or any(k.startswith("head_3d.") for k in state):+ sys.exit(f"[error] {ckpt} was not written by training/train.py")+ cfg, _ = T.config_from_checkpoint(state, ckpt)+ cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2+ cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])+ net = T.InstantHMRStudent(cfg, pretrained=False).eval()+ missing, unexpected = net.load_state_dict(state, strict=False)+ if missing or unexpected:+ sys.exit(f"[error] checkpoint and model disagree: missing {missing[:5]}, unexpected {unexpected[:5]}")+ return net, cfg+++def export(module, inputs: dict, outputs: list, path: Path, metadata: dict) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ torch.onnx.export(module, tuple(inputs.values()), str(path), input_names=list(inputs), output_names=outputs,+ dynamic_axes={n: {0: "batch"} for n in [*inputs, *outputs]}, opset_version=17, dynamo=False)+ model = onnx.load(str(path))+ for key, value in metadata.items():+ entry = model.metadata_props.add()+ entry.key, entry.value = key, value+ onnx.save(model, str(path))++ session = ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])+ got = session.run(None, {k: v.numpy() for k, v in inputs.items()})+ with torch.no_grad():+ want = module(*inputs.values())+ worst = {n: float(np.abs(g - w.numpy()).max()) for n, g, w in zip(outputs, got, want)}+ print(f"wrote {path} ({path.stat().st_size / 1e6:.1f} MB); onnxruntime vs PyTorch, max abs diff: "+ + ", ".join(f"{n} {v:.1e}" for n, v in worst.items()))+ if max(worst.values()) > 1e-3:+ sys.exit("[error] the ONNX graph disagrees with the PyTorch model")+++def main() -> None:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ what = ap.add_mutually_exclusive_group(required=True)+ what.add_argument("--ckpt", type=Path, help="checkpoint written by training/train.py or train_ddp.py")+ what.add_argument("--body", action="store_true", help="export the MHR body graph instead")+ ap.add_argument("--output", type=Path, required=True)+ ap.add_argument("--mhr-model", type=Path, default=REPO / "training" / "checkpoints" / "mhr_model.pt",+ help="MHR TorchScript rig the body is read from")+ args = ap.parse_args()++ if args.body:+ body = MHRBody.from_rig(args.mhr_model).eval()+ faces = torch.jit.load(str(args.mhr_model), map_location="cpu").character_torch.mesh.faces.numpy()+ inputs = {"mhr_params": torch.randn(2, 204) * 0.3, "shape_params": torch.randn(2, 45)}+ export(body, inputs, BODY_OUTPUTS, args.output,+ {"mesh_faces": base64.b64encode(faces.astype("<i4").tobytes()).decode()})+ return++ net, cfg = load_network(args.ckpt)+ size = int(cfg.image_size)+ inputs = {"image": torch.randn(2, 3, size, size), "cliff_cond": torch.randn(2, 3) * 0.3}+ export(Network(net).eval(), inputs, NETWORK_OUTPUTS, args.output,+ {"image_size": str(size), "cliff_focal": str(bool(cfg.cliff_focal)).lower()})+++if __name__ == "__main__":+ main()diff --git a/tools/pack_reference.py b/tools/pack_reference.pydeleted file mode 100644index 2768fb6..0000000--- a/tools/pack_reference.py+++ /dev/null@@ -1,84 +0,0 @@-#!/usr/bin/env python3-"""Pack a SAM 3D Body reference run into one small .npz for ``demo.py --show-reference``.--The reference comes from ``video_to_pose_pipeline/main.py --save_meshes``, which-writes one ``frame_%06d.npz`` per frame carrying the full 18,439-vertex mesh AND-a copy of the constant 36,874-face index array -- ~360 kB per frame, so a few-minutes of video costs a gigabyte.--None of that needs storing. The saved ``person_i_vertices`` are **exactly** what-``instanthmr.mhr_renderer`` reproduces from the saved ``mhr_model_params`` and-``mhr_shape_params`` -- verified at 0.0005 mm on vid4, same rig, same rig-local-frame, faces from the rig itself. So this keeps the parameters and drops the-mesh, which is ~1 kB per frame and has a useful side effect: the reference and-the student are then decoded by the SAME rig, so anything you see differing in-the viewer is the model, never the renderer.-- python tools/pack_reference.py reference/vid4/meshes reference/vid4.npz--Frames with no detection produce no input file and are simply absent from the-output; ``frame_idx`` records which frames are present.-"""--from __future__ import annotations--import argparse-from pathlib import Path--import numpy as np--# Per-person fields copied straight through, with the output key they take.-PERSON_FIELDS = {- "mhr_model_params": "model_params",- "mhr_shape_params": "shape_params",- "cam_translation": "cam_trans",- "joints_3d": "joints_3d",- "joints_2d": "joints_2d",- "bbox": "bbox",- "cam_focal_length": "focal_length",- "cam_principal_point": "principal_point",-}---def main() -> None:- ap = argparse.ArgumentParser(description=__doc__,- formatter_class=argparse.RawDescriptionHelpFormatter)- ap.add_argument("mesh_dir", help="the pipeline's meshes/ directory")- ap.add_argument("out", help="output .npz")- a = ap.parse_args()-- files = sorted(Path(a.mesh_dir).glob("frame_*.npz"))- if not files:- raise SystemExit(f"no frame_*.npz under {a.mesh_dir}")-- frame_idx, counts = [], []- cols: dict[str, list] = {k: [] for k in PERSON_FIELDS.values()}- image_shape = None-- for f in files:- d = np.load(f)- n = int(d["num_persons"])- frame_idx.append(int(d["frame_idx"]))- counts.append(n)- if image_shape is None:- image_shape = d["image_shape"]- for i in range(n):- for src, dst in PERSON_FIELDS.items():- cols[dst].append(d[f"person_{i}_{src}"])-- out = {- "frame_idx": np.asarray(frame_idx, np.int32),- "counts": np.asarray(counts, np.int32),- "image_shape": np.asarray(image_shape, np.int32),- }- for dst, vals in cols.items():- out[dst] = np.stack(vals).astype(np.float32)-- Path(a.out).parent.mkdir(parents=True, exist_ok=True)- np.savez_compressed(a.out, **out)- mb = Path(a.out).stat().st_size / 1e6- print(f"{len(files)} frames, {sum(counts)} persons -> {a.out} ({mb:.2f} MB)")---if __name__ == "__main__":- main()diff --git a/tools/parquet_to_npz.py b/tools/parquet_to_npz.pyindex a7e5c09..16d20a9 100644--- a/tools/parquet_to_npz.py+++ b/tools/parquet_to_npz.py@@ -4,18 +4,13 @@ Convert the official ``facebook/sam-3d-body-dataset`` parquet annotations into tper-crop ``images/*.png`` + ``annotations/*.npz`` layout consumed by``training/train.py``.-This lets you train InstantHMR directly on the original SAM 3D Body ground-truth-MHR fits, instead of (or alongside) the teacher-distilled pseudo-labels produced-by ``tools/annotate_dataset.py``. The output ``.npz`` schema is exactly the one-that script writes, so the training notebook needs **no changes**.--Key facts established by auditing the parquet (see tools/README or docs):+These are the ground-truth MHR fits SAM 3D Body was built from. Facts about the parquet:* The parquet has no ``cam_trans`` column, but the SAM3D translation is fullyrecoverable. The global translation is the first three MHR ``model_params``:cam_trans = [ model_params[0], -model_params[1], -model_params[2] ] / 10- and the body-centred joints (distillation convention, vision Y-down) are+ and the body-centred joints (vision frame, Y down) arejoints_3d = keypoints_3d[:, :3] * [1, -1, -1] # negate Y and Z@@ -27,7 +22,7 @@ Key facts established by auditing the parquet (see tools/README or docs):* Persons with ``mhr_valid == False`` have unreliable fits and are skipped bydefault (the 2D in-the-wild splits — coco/mpii/aic — rely on this flag).-NPZ payload (per crop) — identical to tools/annotate_dataset.py:+NPZ payload (per crop):orig_shape (2,) [H, W] of the full uncropped imagebbox (4,) [x1, y1, x2, y2] in original image coords (xyxy)bbox_square (4,) [x1, y1, x2, y2] expanded square crop (1.2x)@@ -84,17 +79,12 @@ from tqdm import tqdmCROP_SIZE = 224# Negate Y and Z to map MHR-native (Y-up) coordinates into the vision camera-# frame (Y-down, Z-forward) used everywhere in the distillation pipeline.+# frame (Y-down, Z-forward) the trainer uses.NEG_YZ = np.array([1.0, -1.0, -1.0], dtype=np.float64)# MHR root translation (model_params[:3]) is stored in decimetres.TRANS_SCALE = 10.0-# ----------------------------------------------------------------------------# Crop geometry — kept byte-for-byte in sync with-# tools/annotate_dataset.py:get_square_crop_padded so the 224 crops produced-# here are pixel-identical to the distillation crops.-# ---------------------------------------------------------------------------def get_square_crop_padded(image: np.ndarray, bbox: np.ndarray, expand: float = 1.2):"""Square crop around *bbox*, black-padded where it leaves the image."""h, w = image.shape[:2]@@ -169,8 +159,8 @@ def reproj_error(joints_3d: np.ndarray, cam_trans: np.ndarray,orig_shape: np.ndarray) -> float:"""RMS px error of (joints_3d + cam_trans) projected onto joints_2d.- Uses the exact camera model the notebook loss assumes: focal from cam_int,- principal point at the image centre.+ Camera model of the training losses: focal from cam_int, principal point at+ the image centre."""h, w = float(orig_shape[0]), float(orig_shape[1])cam = joints_3d + cam_trans[None, :]@@ -197,13 +187,9 @@ def build_annotation(row, img_h: int, img_w: int) -> dict:cam_trans = cam_trans_from_params(model_params) # from model_params[:3] BEFORE zeroingfocal = np.array([cam_int[0, 0], cam_int[1, 1]], dtype=np.float32)- # The dataset bakes the global translation into model_params[:3], so the MHR- # rig decodes an already-camera-positioned mesh. The distillation pipeline- # (notebook Cells 2 & 8) instead expects a *body-centred* mesh and adds- # cam_trans separately. Zero the translation here so the MHR mesh comes out- # body-centred — verified to match `joints_3d` to ~1e-7. Without this, the- # notebook double-counts cam_trans (loss_structural / loss_feet /- # loss_reproj_mhr and the MHR-param visualiser all go wrong).+ # The dataset bakes the global translation into model_params[:3]; the trainer+ # expects a body-centred mesh and carries the translation in cam_trans, so it+ # is zeroed here (the mesh then matches `joints_3d` to ~1e-7).model_params = model_params.copy()model_params[:3] = 0.0@@ -235,7 +221,7 @@ def save_person(full_img_rgb, ann, name, images_dir, annotations_dir):# ---------------------------------------------------------------------------def parse_args() -> argparse.Namespace:p = argparse.ArgumentParser(- description="Convert sam-3d-body-dataset parquet -> distillation .npz crops.",+ description="Convert sam-3d-body-dataset parquet -> per-person .npz crops.",formatter_class=argparse.RawDescriptionHelpFormatter,epilog=__doc__,)diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.pydeleted file mode 100644index e6ad1f4..0000000--- a/tools/pth_to_onnx.py+++ /dev/null@@ -1,243 +0,0 @@-#!/usr/bin/env python3-"""-Convert a trained InstantHMR student checkpoint (.pth) into the ONNX file the-demo expects at ``models/instanthmr.onnx``.--It reuses the exact model architecture and deploy wrapper from the training-script that produced the checkpoint, plus the FP32 export recipe from notebook-Cell 15, so the graph I/O matches ``instanthmr/inference.py``:-- inputs : image (N, 3, S, S), cliff_cond (N, 3); S = the checkpoint's input- size (224 unless trained with --image-size), stamped as image_size- outputs: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]--Architecture--------------Only checkpoints of ``training/train.py`` are supported: a SimCC 2D head-(``head_2d_logits``) and no 3D head, since the 70 keypoints come from the MHR-forward pass, so the graph has FOUR outputs. Checkpoints of the earlier-trainers (a ``head_3d`` or a ``head_2d_xy`` head) are rejected with an error:-exporting them through this architecture would silently leave a head at its-initial weights. A missing-key check makes any other mismatch a hard error.--Usage------- python tools/pth_to_onnx.py \- --ckpt training/runs/<run>/best_student_model_v3.pth \- --output models/instanthmr_<run>.onnx-"""-import argparse-import importlib-import sys-from pathlib import Path--import torch--REPO = Path(__file__).resolve().parent.parent-TRAIN_DIR = REPO / "training"--# The model class + deploy wrapper live in the standalone training scripts.-sys.path.insert(0, str(TRAIN_DIR))--# arch name -> training module that defines the matching InstantHMRStudent.-ARCH_MODULES = {- "mhr_only": "train", # SimCC 2D head, NO 3D head-}--# Canonical ONNX output order. The MHR-only graph stops after `joints_2d`.-ALL_OUTPUT_KEYS = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]---def detect_arch(state: dict) -> str:- """Pick the architecture that matches the checkpoint's prediction heads."""- has_3d = any(k.startswith("head_3d.") for k in state)- has_simcc = "head_2d_logits.weight" in state-- if has_simcc and not has_3d:- # train.py: the 3D coordinate head was removed and the- # decoder shrank from 141 to 71 queries.- return "mhr_only"- if has_3d or "head_2d_xy.0.weight" in state:- raise SystemExit(- "[error] this checkpoint comes from an earlier trainer (it has a 'head_3d' or "- "'head_2d_xy' head), which this release does not include.")- raise SystemExit(- "[error] cannot infer architecture: checkpoint has no 'head_2d_logits.weight'.")---def main() -> None:- ap = argparse.ArgumentParser(description=__doc__,- formatter_class=argparse.RawDescriptionHelpFormatter)- ap.add_argument("--ckpt", type=Path, required=True,- help="Path to the .pth checkpoint written by training/train.py or train_ddp.py.")- ap.add_argument("--output", type=Path, default=REPO / "models" / "instanthmr.onnx",- help="Where to write the ONNX (default: models/instanthmr.onnx).")- ap.add_argument("--allow-missing", action="store_true",- help="Export even if checkpoint weights are missing (NOT recommended).")- ap.add_argument("--opset", type=int, default=17)- ap.add_argument("--cliff-focal", dest="cliff_focal", action="store_true",- default=None,- help="Force the focal-aware CLIFF conditioning flag stamped "- "into the ONNX metadata. Normally read from the "- "run_config.json beside the checkpoint.")- ap.add_argument("--no-cliff-focal", dest="cliff_focal", action="store_false",- help="Force the pixel-normalised CLIFF conditioning flag.")- args = ap.parse_args()-- if not args.ckpt.exists():- sys.exit(f"[error] checkpoint not found: {args.ckpt}")-- device = torch.device("cpu") # export on CPU — portable, no CUDA needed-- ckpt = torch.load(args.ckpt, map_location=device, weights_only=False)- state = ckpt.get("model_state_dict", ckpt)- state = {k.replace("_orig_mod.", ""): v for k, v in state.items()} # strip torch.compile prefix-- arch = detect_arch(state)- T = importlib.import_module(ARCH_MODULES[arch])- print(f"Architecture: {arch} (from {ARCH_MODULES[arch]}.py)")-- # Restore the configuration the checkpoint was TRAINED with, not the- # defaults. `--bound-scales` puts a tanh remap of the 77 body-size- # parameters inside forward(); exporting without it leaves the head's raw- # pre-tanh numbers to be read as bone scales, which is 21.0 mm MPJPE /- # 14.4 mm PA-MPJPE of silent corruption. See T.config_from_checkpoint.- cfg, prov = T.config_from_checkpoint(state, args.ckpt, cliff_focal=args.cliff_focal)- print(f" bound_scales : {getattr(cfg, 'bound_scales', False)} "- f"({prov.get('bound_scales', 'default')})")- print(f" cliff_focal : {getattr(cfg, 'cliff_focal', False)} "- f"({prov.get('cliff_focal', 'default')})")- print(f" cont_head : {getattr(cfg, 'cont_head', False)} "- f"({prov.get('cont_head', 'default')})")-- # SimCC head geometry lives in the checkpoint — mirror it so the graph and- # the bin centres agree with what was actually trained.- if "head_2d_logits.bias" in state:- cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2- if "kp2d_bin_centers" in state:- cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])- print(f" SimCC head: {cfg.kp2d_bins} bins over [-{cfg.kp2d_range}, {cfg.kp2d_range}]")-- # --- Build the model and load weights ---- model = T.InstantHMRStudent(cfg, pretrained=False).to(device).eval()- missing, unexpected = model.load_state_dict(state, strict=False)- print(f"Loaded {args.ckpt.name} | epoch {ckpt.get('epoch', '?')} "- f"| PA-MPJPE {ckpt.get('val_pa_mpjpe', '?')} | source {ckpt.get('source', '?')}")- if unexpected:- # Fatal, not a note. An unexpected key means the checkpoint carries- # state this architecture has no slot for, i.e. the config above is- # wrong -- exactly how `scale_lo`/`scale_hi` used to be dropped.- sys.exit(- f"\n[error] {len(unexpected)} key(s) in the checkpoint have no "- f"home in the model built here:\n {unexpected[:8]}\n"- f" The exported graph would silently differ from the "- f"trained one. Fix the config, do not ignore this."- )- if missing:- print(f" missing keys : {len(missing)} -> {missing[:8]}")- if not args.allow_missing:- sys.exit(- f"\n[error] {len(missing)} parameter(s) had no weights in the checkpoint, so they "- f"would be exported at random init.\n"- f" This usually means the checkpoint does not match training/train.py's\n"- f" model. --allow-missing forces the export."- )-- deploy = T.HMRDeployWrapper(model).eval()-- # --- Export FP32 ONNX (Cell 15 recipe) ---- dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)- dummy_cliff = torch.randn(1, 3, device=device)- inputs, input_names = (dummy_img, dummy_cliff), ["image", "cliff_cond"]- prev_token = bool(getattr(cfg, "prev_token", False))- if prev_token:- # --prev-token: the previous pose (T.PREV_DIM floats) and its 0/1 flag- # are graph inputs, the per-foot contact logits a fifth output.- inputs += (torch.randn(1, T.PREV_DIM, device=device), torch.ones(1, device=device))- input_names += ["prev_pose", "has_prev"]- print(f" prev_token : {prev_token}")-- # Ask the wrapper how many tensors it returns instead of hard-coding five:- # train.py drops `joints_3d` from the tuple.- with torch.no_grad():- n_out = len(deploy(*inputs))- if n_out not in (4, 5):- sys.exit(f"[error] unexpected deploy wrapper output count: {n_out} (expected 4 or 5)")- output_keys = (ALL_OUTPUT_KEYS[:4] + ["contact_logits"] if prev_token- else ALL_OUTPUT_KEYS[:n_out])- print(f" graph outputs: {n_out} -> {output_keys}")- if "joints_3d" not in output_keys:- print(" note: no 'joints_3d' output — the 70 keypoints are derived from the\n"- " MHR forward pass at inference time (needs --mhr-model / --mhr-assets\n"- " in demo.py).")-- dynamic_axes = {k: {0: "batch"} for k in input_names + output_keys}-- args.output.parent.mkdir(parents=True, exist_ok=True)- torch.onnx.export(- deploy, inputs, str(args.output),- input_names=input_names, output_names=output_keys,- dynamic_axes=dynamic_axes, opset_version=args.opset,- )- # Stamp which CLIFF conditioning this graph expects. The vector has the- # same shape either way, so a consumer that guesses wrong gets no error --- # only a person placed at the wrong depth. instanthmr.inference reads this.- try:- import onnx- m = onnx.load(str(args.output))- for k, v in (("cliff_focal", str(bool(getattr(cfg, "cliff_focal", False))).lower()),- ("bound_scales", str(bool(getattr(cfg, "bound_scales", False))).lower()),- ("arch", arch),- ("image_size", str(int(cfg.image_size))),- ("prev_token", str(prev_token).lower())):- e = m.metadata_props.add()- e.key, e.value = k, v- onnx.save(m, str(args.output))- print(f" metadata: cliff_focal={getattr(cfg, 'cliff_focal', False)} "- f"bound_scales={getattr(cfg, 'bound_scales', False)} arch={arch} "- f"image_size={cfg.image_size}")- except Exception as e: # noqa: BLE001- print(f" [warning] could not stamp ONNX metadata: {e}")-- print(f"\n✅ Wrote {args.output} ({args.output.stat().st_size / 1e6:.1f} MB)")-- # --- Sanity check: run once through onnxruntime and verify I/O ---- try:- import numpy as np- import onnxruntime as ort- sess = ort.InferenceSession(str(args.output), providers=["CPUExecutionProvider"])- in_names = [i.name for i in sess.get_inputs()]- outs = sess.run(None, {n: x.numpy() for n, x in zip(in_names, inputs)})- print(f" onnxruntime OK | inputs {in_names}")- for o, v in zip(sess.get_outputs(), outs):- v = np.asarray(v)- print(f" {o.name:14s} {list(v.shape)} std={v.std():.4f}")-- # Parity against the configured PyTorch model on the SAME input. This- # is the check that catches a config-level export defect: identical- # weights, different forward. A dropped tanh remap shows up here as a- # large mhr_params delta while every tensor keeps its right shape.- with torch.no_grad():- ref = deploy(*inputs)- worst = max(float(np.abs(np.asarray(o) - r.numpy()).max())- for o, r in zip(outs, ref))- print(f" torch<->onnx max abs diff: {worst:.3e}")- if worst > 1e-3:- sys.exit(- f"\n[error] the ONNX graph disagrees with the PyTorch model it "- f"was built from by {worst:.3e}.\n Identical weights, "- f"different forward -- the exported configuration is wrong."- )- # A dead 2D head (wrong arch, weights left at init) predicts every joint- # at the crop centre; real predictions spread over normalised [-1, 1].- j2d = np.asarray(outs[output_keys.index("joints_2d")])- if j2d.std() < 0.01:- print(f"\n[warning] joints_2d std is {j2d.std():.5f} — the 2D keypoints are collapsed "- f"onto the crop centre.\n The exported 2D head is almost certainly "- f"untrained.")- except Exception as e: # noqa: BLE001- print(f" (onnxruntime sanity check skipped: {e})")---if __name__ == "__main__":- main()diff --git a/training/README.md b/training/README.mdindex a8b21bf..d75850f 100644--- a/training/README.md+++ b/training/README.md@@ -21,9 +21,13 @@ A change to the model or a loss goes in `train.py`, so both entry points get it.pip install -r training/requirements.txt```-The MHR rig is not in git. Copy `assets/mhr_model.pt` from the-[SAM 3D Body checkpoint](https://huggingface.co/facebook/sam-3d-body-dinov3) to-`training/checkpoints/mhr_model.pt`, or pass `--mhr_model_path`.+The trainer needs Meta's MHR rig, `mhr_model.pt` (Apache-2.0, 700 MB), from the+[MHR release](https://github.com/facebookresearch/MHR/releases):++```bash+curl -L -o mhr_assets.zip https://github.com/facebookresearch/MHR/releases/latest/download/assets.zip+mkdir -p training/checkpoints && unzip -j mhr_assets.zip assets/mhr_model.pt -d training/checkpoints+```## Data@@ -113,15 +117,17 @@ python tools/ddp_smoke.py --backbone hgnetv2_b4 --image-size 288 \# The model must be able to fit 8 images.python training/train.py --overfit-test --data_root <data>-python -m pytest -q training/test_train_ddp.py+python -m pytest -q training```## Export to ONNX```bash-python tools/pth_to_onnx.py --ckpt runs/medium/best_student_model_v3.pth \- --output models/instanthmr_medium.onnx+python tools/export_onnx.py --ckpt runs/medium/best_student_model_v3.pth --output models/my-model.onnx+python tools/export_onnx.py --body --output models/mhr-body.onnx # once, shared by every model+python demo.py --image photo.jpg --model models/my-model.onnx```-The exporter reads the training settings from the `run_config.json` beside the-checkpoint, so keep the two together.+The first graph is the network; the second turns its MHR parameters into the 70 3D keypoints and+the mesh, and the demo looks for it next to the model. The exporter reads the training settings+from the `run_config.json` beside the checkpoint, so keep the two together.diff --git a/training/mhr_body.py b/training/mhr_body.pynew file mode 100644index 0000000..6cd419c--- /dev/null+++ b/training/mhr_body.py@@ -0,0 +1,161 @@+"""The MHR body as an exportable PyTorch module: model parameters -> 70 keypoints + mesh.++`MHRBody` rebuilds, from about 12 MB of constants read once from the TorchScript rig, what the+network's outputs mean geometrically: forward kinematics of the 127 joints, the identity blend+shapes and linear blend skinning of the 18,439-vertex mesh, and the 70 keypoints as the exact+landmark readout the network was trained against (`W_joint @ joints + W_vert @ vertices`).+Meta's pose correctives are left out: they are a 663 MB network, and the network was trained on+the mesh without them (mean 1.9 mm from the full rig).++Every op is static and exports to ONNX, so `tools/export_onnx.py` appends this module to the+network and the released graphs need no MHR package at run time. Outputs are body-centred, in+metres, in the camera axes (x right, y down, z forward); add `cam_trans` for camera space.+"""+from __future__ import annotations++import math+from pathlib import Path++import numpy as np+import torch++N_MODEL_PARAMS = 204+LANDMARKS = Path(__file__).resolve().parent / "assets" / "mhr_landmarks70.npz"+++def _fk_levels(parents: np.ndarray) -> tuple[np.ndarray, np.ndarray]:+ """Prefix-multiplication schedule: at level l, every active joint j becomes partner(j) o j."""+ chains = []+ for j in range(len(parents)):+ chain = [j]+ while parents[chain[-1]] >= 0:+ chain.append(int(parents[chain[-1]]))+ chains.append(chain[::-1])+ partners, actives = [], []+ level = 0+ while True:+ partner, active = np.arange(len(parents)), np.zeros(len(parents), bool)+ for chain in chains:+ i = len(chain) - 1+ if (i >> level) & 1:+ partner[chain[i]] = chain[((i >> level) << level) - 1]+ active[chain[i]] = True+ if not active.any():+ return np.stack(partners), np.stack(actives)+ partners.append(partner)+ actives.append(active)+ level += 1+++def _normalize(q):+ return q / torch.sqrt((q * q).sum(-1, keepdim=True)).clamp_min(1e-12)+++def _qmul(a, b):+ x1, y1, z1, w1 = a.unbind(-1)+ x2, y2, z2, w2 = b.unbind(-1)+ return torch.stack((w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,+ w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2,+ w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2,+ w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2), -1)+++def _cross(a, b):+ ax, ay, az = a.unbind(-1)+ bx, by, bz = b.unbind(-1)+ return torch.stack((ay * bz - az * by, az * bx - ax * bz, ax * by - ay * bx), -1)+++def _rotate(q, v):+ axis, w = q[..., :3], q[..., 3:]+ av = _cross(axis, v)+ return v + 2 * (av * w + _cross(axis, av))+++def _euler_xyz_to_quat(e):+ cr, sr = torch.cos(e[..., 0] / 2), torch.sin(e[..., 0] / 2)+ cp, sp = torch.cos(e[..., 1] / 2), torch.sin(e[..., 1] / 2)+ cy, sy = torch.cos(e[..., 2] / 2), torch.sin(e[..., 2] / 2)+ return torch.stack((sr * cp * cy - cr * sp * sy, cr * sp * cy + sr * cp * sy,+ cr * cp * sy - sr * sp * cy, cr * cp * cy + sr * sp * sy), -1)+++def _compose(a, b):+ """a o b for skeleton states [t(3), q(4) xyzw, s(1)]."""+ ta, qa, sa = a[..., :3], _normalize(a[..., 3:7]), a[..., 7:]+ tb, qb, sb = b[..., :3], _normalize(b[..., 3:7]), b[..., 7:]+ return torch.cat((ta + sa * _rotate(qa, tb), _qmul(qa, qb), sa * sb), -1)+++def _to_camera_axes(xyz):+ """MHR (cm, y up, z back) -> metres, y down, z forward."""+ return xyz * torch.tensor([0.01, -0.01, -0.01], dtype=xyz.dtype)+++class MHRBody(torch.nn.Module):+ """`(B, 204)` mhr_params + `(B, 45)` shape_params -> `(B, 70, 3)` keypoints, `(B, 18439, 3)` vertices."""++ def __init__(self, constants: dict[str, np.ndarray]):+ super().__init__()+ for name, value in constants.items():+ self.register_buffer(name, torch.as_tensor(value), persistent=False)++ @classmethod+ def from_rig(cls, rig_path: str | Path, landmarks: str | Path = LANDMARKS) -> "MHRBody":+ rig = torch.jit.load(str(rig_path), map_location="cpu").eval()+ ct = rig.character_torch+ sk, lbs, bs = ct.skeleton, ct.linear_blend_skinning, ct.blend_shape+ transform = ct.parameter_transform.parameter_transform+ if transform[:, N_MODEL_PARAMS:].any():+ raise ValueError(f"{rig_path}: the parameter transform uses the identity columns")++ n_verts = bs.base_shape.shape[0]+ vert = lbs.vert_indices_flattened.long()+ counts = torch.bincount(vert, minlength=n_verts)+ order = torch.argsort(vert, stable=True)+ vert = vert[order]+ slot = torch.arange(len(vert)) - (torch.cumsum(counts, 0) - counts)[vert]+ skin_joint = torch.zeros(n_verts, int(counts.max()), dtype=torch.long)+ skin_weight = torch.zeros(n_verts, int(counts.max()))+ skin_joint[vert, slot] = lbs.skin_indices_flattened.long()[order]+ skin_weight[vert, slot] = lbs.skin_weights_flattened.float()[order]++ partner, active = _fk_levels(sk.joint_parents.numpy())+ lm = np.load(landmarks)+ return cls(dict(+ parameter_transform=transform[:, :N_MODEL_PARAMS].numpy(),+ joint_offsets=sk.joint_translation_offsets.numpy(),+ joint_prerotations=sk.joint_prerotations.numpy(),+ fk_partner=partner, fk_active=active,+ inverse_bind_pose=lbs.inverse_bind_pose.numpy(),+ base_shape=bs.base_shape.numpy(), shape_vectors=bs.shape_vectors.numpy(),+ skin_joint=skin_joint.numpy(), skin_weight=skin_weight.numpy(),+ landmark_vertices=lm["vert_idx"].astype(np.int64),+ W_joint=lm["W_joint"].astype(np.float32), W_vert=lm["W_vert"].astype(np.float32),+ ))++ def skeleton(self, mhr_params):+ """`(B, 127, 8)` global joint states, MHR units."""+ jp = (mhr_params @ self.parameter_transform.T).unflatten(-1, (-1, 7))+ q = _qmul(self.joint_prerotations, _euler_xyz_to_quat(jp[..., 3:6]))+ state = torch.cat((jp[..., :3] + self.joint_offsets, q, torch.exp(math.log(2.0) * jp[..., 6:])), -1)+ for level in range(self.fk_partner.shape[0]):+ composed = _compose(state.index_select(-2, self.fk_partner[level]), state)+ state = torch.where(self.fk_active[level, :, None], composed, state)+ return state++ def skin(self, state, shape_params):+ """`(B, 18439, 3)` posed vertices, MHR units."""+ n_verts, n_inf = self.skin_joint.shape+ rest = self.base_shape + (shape_params @ self.shape_vectors.flatten(1)).unflatten(-1, (n_verts, 3))+ bone = _compose(state, self.inverse_bind_pose)+ bone = bone.index_select(-2, self.skin_joint.flatten()).unflatten(-2, (n_verts, n_inf))+ t, q, s = bone[..., :3], _normalize(bone[..., 3:7]), bone[..., 7:]+ return ((t + _rotate(q, s * rest.unsqueeze(-2))) * self.skin_weight[..., None]).sum(-2)++ def forward(self, mhr_params, shape_params):+ state = self.skeleton(mhr_params)+ vertices = _to_camera_axes(self.skin(state, shape_params))+ joints = _to_camera_axes(state[..., :3])+ keypoints = self.W_joint @ joints + self.W_vert @ vertices.index_select(-2, self.landmark_vertices)+ return keypoints, verticesdiff --git a/training/test_mhr_body.py b/training/test_mhr_body.pynew file mode 100644index 0000000..36173b4--- /dev/null+++ b/training/test_mhr_body.py@@ -0,0 +1,59 @@+"""The in-graph MHR body against the rig it replaces.++ python -m pytest -q training/test_mhr_body.py+"""+import glob+import os+import sys+from pathlib import Path++import numpy as np+import pytest+import torch++HERE = Path(__file__).resolve().parent+sys.path.insert(0, str(HERE))+RIG = HERE / "checkpoints" / "mhr_model.pt"+pytestmark = pytest.mark.skipif(not RIG.exists(), reason=f"needs the MHR rig at {RIG}")++[email protected](scope="module")+def poses():+ """Real poses from a corpus ($INSTANTHMR_DATA, default data/), else random parameters."""+ root = Path(os.environ.get("INSTANTHMR_DATA", HERE.parent / "data"))+ files = sorted(glob.glob(str(root / "*" / "annotations" / "*.npz")))[:16]+ if files:+ p = np.stack([np.load(f)["mhr_model_params"] for f in files]).astype(np.float32)+ s = np.stack([np.load(f)["shape_params"] for f in files]).astype(np.float32)+ else:+ g = np.random.default_rng(0)+ p = (g.standard_normal((16, 204)) * 0.3).astype(np.float32)+ s = g.standard_normal((16, 45)).astype(np.float32)+ p[:, :3] = 0+ return torch.from_numpy(p), torch.from_numpy(s)++[email protected](scope="module")+def body():+ from mhr_body import MHRBody+ return MHRBody.from_rig(RIG).eval()+++def test_keypoints_match_the_training_readout(body, poses):+ import train as T+ ref = T.MHRForwardPass(str(RIG), "cpu", landmark_assets=str(HERE / "assets" / "mhr_landmarks70.npz"))+ with torch.no_grad():+ want = ref.get_native_keypoints(*poses, exact=True)+ got, _ = body(*poses)+ assert float((got - want).abs().max()) < 1e-5 # metres: 0.01 mm+++def test_vertices_match_the_rig_without_pose_correctives(body, poses):+ rig = torch.jit.load(str(RIG), map_location="cpu").eval()+ p, s = poses+ with torch.no_grad():+ v, _ = rig(s, p, torch.zeros(len(p), 72), False)+ want = torch.stack([v[..., 0], -v[..., 1], -v[..., 2]], -1) / 100.0+ _, got = body(p, s)+ assert got.shape == want.shape == (len(p), 18439, 3)+ assert float((got - want).abs().max()) < 1e-5diff --git a/training/train.py b/training/train.pyindex f68d9b8..100fe6f 100644--- a/training/train.py+++ b/training/train.py@@ -1,43 +1,14 @@#!/usr/bin/env python3-# ============================================================-# MHR-ONLY distillation — no concurrent 3D coordinate head.-# -------------------------------------------------------------# SAM3D-Body (the teacher) has NO 3D and NO 2D pose head. Its only-# predictive modules are head_pose.proj (519 outputs = MHR params) and-# head_camera.proj (3 outputs = s, tx, ty). The 70 keypoints are-# obtained as keypoint_mapping @ [verts ; joints] AFTER the MHR-# forward, and the 2D ones by perspective re-projection of those 3D-# points. Everything is derived from the mesh.-#-# The student mirrors that: the 3D head is GONE and every 3D quantity-# is derived from the MHR forward pass. Concretely-#-# mhr_params, shape_params ──► MHR skeleton (127 joints)-# ──► W @ joints = 70 native keypoints-#-# where W is a (70, 127) affine regressor fitted offline against the-# teacher's own labels (see fit_keypoint_regressor / --fit-regressor).-# Held-out residual of that fit: 1.36 mm mean / 0.005 mm median, which-# is two orders of magnitude below the target PA-MPJPE. Because W's-# rows sum to exactly 1 it commutes with rigid transforms, so the-# augmentation counter-rotations keep working unchanged.-#-# The 2D SimCC head is KEPT. It is training-only (deployment re-projects-# the MHR keypoints) but it is the only supervision that survives a-# horizontal flip un-masked, so removing it would starve the flipped-# half of the augmented samples.-#-# EXPORT CONTRACT CHANGED: the ONNX now has FOUR outputs-# (mhr_params, shape_params, cam_trans, joints_2d)-# instead of five — joints_3d is gone. instanthmr/inference.py must be-# patched before this model can be used there; see the note on-# HMRDeployWrapper below.-#-# Sanity: python3 train.py --self-test --data_root ../data/sam3d_gt_mpii-# python3 train.py --overfit-test --data_root ../data/sam3d_gt_mpii-# Refit W: python3 train.py --fit-regressor --data_root ../data-# Full: python3 -u train.py --data_root /datasets/instanthmr_data --num_workers 8-# ============================================================+"""InstantHMR trainer: dataset, augmentation, model, MHR forward pass, losses, and a single-GPU loop.++The network predicts the MHR parameters (pose, skeleton, body shape), the camera translation, and+the 70 keypoints in 2D (SimCC). Every 3D quantity is derived from the MHR forward pass, as in+SAM 3D Body. `train_ddp.py` imports this file for multi-GPU runs; see training/README.md.++ python training/train.py --self-test --data_root data+ python training/train.py --overfit-test --data_root data+ python training/train.py --data_root data --output_dir runs/my_run+"""import os# Must be set before the first CUDA allocation (read lazily by the allocator).@@ -82,9 +53,6 @@ import matplotlibmatplotlib.use("Agg")import matplotlib.pyplot as pltimport cv2-import onnxruntime as ort--print(ort.get_available_providers())device = torch.device("cuda" if torch.cuda.is_available() else "cpu")print(f"Using device: {device}")@@ -1268,10 +1236,9 @@ class SAM3DStudentDataset(Dataset):# Where the box sits in the FIELD OF VIEW, and how large it is# ANGULARLY -- not in image-normalised pixels.#- # Under a fixed focal the two are interchangeable, and the corpus- # this trainer was written for had exactly that: every crop from- # tools/annotate_dataset.py carries the synthetic f = sqrt(H^2+W^2),- # i.e. f/diag = 1.000 with zero spread. The rebuilt corpus carries+ # Under a fixed focal the two are interchangeable, and the first+ # corpus had exactly that: every crop carried the synthetic+ # f = sqrt(H^2+W^2), i.e. f/diag = 1.000 with zero spread. The corpus# the datasets' own focals instead, and those span f/diag 0.6-1.8 on# coco/aic/mpii (sd ~0.35) with harmony4d off at 0.285. There the# pixel version is ambiguous: two crops with identical conditioning@@ -2820,212 +2787,6 @@ def train_instant_hmr():print(f" best RAW: {best_raw_pa:.1f} mm -> {best_raw_path}")print(f" best EMA: {best_ema_pa:.1f} mm -> {best_ema_path}")-# ============================================================-# Cell 15 — Export & Static Quantization-# ============================================================-class HMRDeployWrapper(nn.Module):- """FOUR outputs: (mhr_params, shape_params, cam_trans, joints_2d).-- BREAKING CHANGE vs the earlier trainer, which exported a- fifth tensor `joints_3d` straight from the (now deleted) 3D head.-- The MHR skeleton block cannot be folded into the ONNX graph: the TorchScript- artefact exposes `character_torch` as a `pymomentum...Character` object, not- an `nn.Module`, and torch.onnx.export refuses to trace through it- ("Tried to trace <...Character> but it is not part of the active trace").- So the 3D keypoints have to be produced by the consumer.-- Required patch in instanthmr/inference.py (NOT applied here — it would add a- hard torch/mhr dependency to a module that is currently numpy + onnxruntime- only, which is not a risk-free change to make blind):-- * __init__ (~l.119): the comment "Output order from export: ...,- joints_3d" is now wrong; assert len(out_names) == 4 and load the MHR- TorchScript model + training/assets/mhr_j127_to_kp70.npy.- * predict (~l.157-162): `joints_3d_local = outs[4][0]` must become-- joints_3d_local = self._mhr_keypoints(mhr_params[None])[0]-- where `_mhr_keypoints` is MHRForwardPass.get_native_keypoints from this- file (skeleton forward -> /100 -> negate Y,Z -> W @ joints).- * predict_batch (~l.239): same for `joints_3d_local_b = outs[4]`, batched.- * The docstrings of `InstantHMR` and `HMRPrediction` that advertise a- 70x3 joints_3d output straight from the ONNX session.-- Measured cost of that extra forward: ~0.6 ms at batch 1 and ~0.7 ms at- batch 8 on an RTX 4070 (no_grad, skeleton branch only, no skinning).- """-- def __init__(self, model):- super().__init__()- self.model = model-- def forward(self, image, cliff_cond, prev_pose=None, has_prev=None):- # A --prev-token graph takes the previous pose as two extra inputs and- # returns the per-foot contact logits as a fifth output.- out = self.model(image, cliff_cond, prev_pose, has_prev)- base = (out["mhr_params"], out["shape_params"], out["cam_trans"],- out["joints_2d"])- return base + (out["contact_logits"],) if "contact_logits" in out else base--def topological_sort_onnx(model):- available = set()- for inp in model.graph.input:- available.add(inp.name)- for init in model.graph.initializer:- available.add(init.name)-- remaining = list(model.graph.node)- sorted_nodes = []-- while remaining:- progress = False- for node in remaining:- if all(inp in available or inp == '' for inp in node.input):- sorted_nodes.append(node)- remaining.remove(node)- for out in node.output:- available.add(out)- progress = True- break- if not progress:- sorted_nodes.extend(remaining)- break-- del model.graph.node[:]- model.graph.node.extend(sorted_nodes)- return model--def export_and_quantize():- print("--- Exporting and Quantizing Full Model for Deployment ---")- try:- from onnxruntime.quantization import (- CalibrationDataReader, quantize_static,- QuantType, QuantFormat, CalibrationMethod,- )-- class HMRCalibrationDataReader(CalibrationDataReader):- def __init__(self, dataloader, max_samples=100):- self.data_list = []- print(f"Extracting {max_samples} individual images for INT8 calibration...")- for batch in dataloader:- images = batch["image"].numpy().astype(np.float32)- cliffs = batch["cliff_cond"].numpy().astype(np.float32)- for b_idx in range(images.shape[0]):- if len(self.data_list) >= max_samples:- break- self.data_list.append({- "image": images[b_idx:b_idx + 1],- "cliff_cond": cliffs[b_idx:b_idx + 1]- })- if len(self.data_list) >= max_samples:- break- self.enum_data = iter(self.data_list)-- def get_next(self):- return next(self.enum_data, None)-- except ImportError:- print("⚠️ 'onnxruntime' is not installed. INT8 quantization will fail.")- HMRCalibrationDataReader = None-- export_dir = Path(cfg.log_dir) / "export"- export_dir.mkdir(parents=True, exist_ok=True)-- onnx_fp32_path = export_dir / "sam3d_full_fp32.onnx"- onnx_fp16_path = export_dir / "sam3d_full_fp16.onnx"- onnx_int8_path = export_dir / "sam3d_full_int8.onnx"-- output_keys = ["mhr_params", "shape_params", "cam_trans", "joints_2d"]- print("⚠️ Export contract: 4 outputs (joints_3d dropped). "- "instanthmr/inference.py needs the patch described on HMRDeployWrapper.")- export_target_model = InstantHMRStudent(cfg, pretrained=False).to(device)- best_ckpt_path = os.path.join(cfg.log_dir, "best_student_model_v3.pth")-- if os.path.exists(best_ckpt_path):- ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=True)- clean_state_dict = {k.replace('_orig_mod.', ''): v for k, v in ckpt['model_state_dict'].items()}- export_target_model.load_state_dict(clean_state_dict, strict=False)- print(f"✅ Loaded best model from epoch {ckpt.get('epoch', 0)+1}.\n")-- deploy_model = HMRDeployWrapper(export_target_model)- deploy_model.eval()-- dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)- dummy_cliff = torch.randn(1, 3, device=device)-- dynamic_axes = {"image": {0: "batch"}, "cliff_cond": {0: "batch"}}- for k in output_keys:- dynamic_axes[k] = {0: "batch"}-- try:- import onnx- torch.onnx.export(- deploy_model, (dummy_img, dummy_cliff), str(onnx_fp32_path),- input_names=["image", "cliff_cond"], output_names=output_keys,- dynamic_axes=dynamic_axes, opset_version=17- )- print(f"✅ FP32 saved: {onnx_fp32_path.name} | Size: {onnx_fp32_path.stat().st_size / 1e6:.1f} MB")- except Exception as e:- print(f"❌ FP32 export failed: {e}")-- try:- from onnxruntime.transformers.float16 import convert_float_to_float16- import warnings- with warnings.catch_warnings():- warnings.simplefilter("ignore")- model_fp32 = onnx.load(str(onnx_fp32_path))- model_fp16 = convert_float_to_float16(model_fp32, keep_io_types=True)- model_fp16 = topological_sort_onnx(model_fp16)- onnx.save(model_fp16, str(onnx_fp16_path))- onnx.checker.check_model(onnx.load(str(onnx_fp16_path)), full_check=True)- print(f"✅ FP16 saved: {onnx_fp16_path.name} | Size: {onnx_fp16_path.stat().st_size / 1e6:.1f} MB")- except ImportError:- print("⚠️ FP16 skipped (onnxruntime.transformers not available)")- except Exception as e:- print(f"❌ FP16 conversion failed: {e}")-- try:- calib_reader = HMRCalibrationDataReader(train_loader, max_samples=100)- calib_reader.enum_data = iter(calib_reader.data_list)- fp32_model = onnx.load(str(onnx_fp32_path))- nodes_to_exclude = []- for node in fp32_model.graph.node:- name = node.name or ''- if any(kw in name for kw in ['/model/transformer/', '/model/head_',- '/model/feat_proj', '/model/cond_proj']):- nodes_to_exclude.append(name)- del fp32_model- print(f" Excluding {len(nodes_to_exclude)} transformer/head nodes from quantization")-- quantize_static(- model_input=str(onnx_fp32_path),- model_output=str(onnx_int8_path),- calibration_data_reader=calib_reader,- quant_format=QuantFormat.QDQ,- op_types_to_quantize=['Conv'],- per_channel=True,- reduce_range=False,- activation_type=QuantType.QInt8,- weight_type=QuantType.QInt8,- nodes_to_exclude=nodes_to_exclude,- calibrate_method=CalibrationMethod.Percentile,- extra_options={'CalibTensorRangeSymmetric': True},- )- print(f"✅ INT8 saved: {onnx_int8_path.name} | Size: {onnx_int8_path.stat().st_size / 1e6:.1f} MB")- except NameError:- print("⚠️ INT8 skipped (train_loader not available)")- except Exception as e:- import traceback- print(f"❌ INT8 conversion failed: {e}")- traceback.print_exc()-- sd_path = export_dir / "sam3d_student_state_dict.pth"- torch.save(export_target_model.state_dict(), sd_path)- print(f"\n✅ PyTorch state dict saved: {sd_path.name}")- else:- print("⚠️ No checkpoint found at:", best_ckpt_path)-# ============================================================# Self-tests# ============================================================@@ -3390,8 +3151,6 @@ def parse_args():help="Refit the (70,127) skeleton->keypoint regressor from the dataset and exit.")p.add_argument("--overfit-steps", dest="overfit_steps", type=int, default=3000,help="Number of steps for --overfit-test (default 3000; see run_overfit_test).")- p.add_argument("--no-export", dest="no_export", action="store_true",- help="Skip the ONNX export / quantization step after training.")p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",help="Read the 70 keypoints out of the skinned mesh with the "@@ -3558,9 +3317,7 @@ def main():if torch.cuda.is_available():torch.cuda.empty_cache()train_instant_hmr()-- if not args.no_export:- export_and_quantize()+ print(f"Export the best checkpoint with: python tools/export_onnx.py --ckpt {cfg.log_dir}/best_student_model_v3.pth --output model.onnx")if __name__ == "__main__":main()\ No newline at end of file