Skip to content
Python

Self-contained ONNX release with the MHR body graph, model download, TensorRT, public scoring script and rewritten docs.

mohamdevc1672a4f6534
diff --git a/.gitignore b/.gitignore
index d2cf080..5653a03 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,58 +1,26 @@
-# Byte-compiled / cache
__pycache__/
*.py[cod]
-*$py.class
.pytest_cache/
-.mypy_cache/
-.ruff_cache/
-
-# Editor / OS
.vscode/
.idea/
.DS_Store
-
-# Virtual environments
.venv/
venv/
-env/
-# RF-DETR auto-downloaded weights
-rf-detr*.pth
-*.pth.lock
+# weights and outputs
*.onnx
+*.pt
*.pth
-*.mp4
-
-# Rerun recordings
*.rrd
-
-# Annotation outputs
-data/
-annotations/
-images/
-checkpoints/
-
-# Model weights — too large for git, use Git LFS or release artifacts
-models/*.onnx
-models/*.pt
-models/*.pth
-models/mhr_assets/
-commands.md
-!models/README.md
-
-# Transfer archives (dataset tarballs)
+*.mp4
*.tar
*.tar.gz
-
-# Training run outputs (checkpoints / ONNX exports / logs)
+models/
runs/
+checkpoints/
-# Notebook checkpoints
-.ipynb_checkpoints/
-/competitors/
-*.png
-
-# Saved teacher runs for `demo.py --show-reference` (renders + packed .npz).
-# Local and regenerable: tools/pack_reference.py rebuilds the .npz from a
-# video_to_pose_pipeline --save_meshes run. 247 MB here.
-reference/
+# training data
+data/
+data_*/
+annotations/
+images/
diff --git a/README.md b/README.md
index fe77402..b4b5578 100644
--- a/README.md
+++ b/README.md
@@ -1,449 +1,141 @@
-# InstantHMR
+# InstantHMR: Real-Time Human Mesh Recovery on Your Phone
-![InstantHMR demo](models/instanthmr.gif)
+InstantHMR recovers the 3D body of every person in an image: pose, body shape, 70 keypoints in 2D
+and 3D, and a full body mesh in the [Momentum Human Rig](https://github.com/facebookresearch/MHR)
+(MHR). It runs in about 5 ms per person on a phone NPU and in about 1 ms on a desktop GPU.
-A lightweight, ONNX-exportable MHR body regressor for **3D human pose estimation
-and mesh recovery**: a RepViT-M1.5 backbone + a 9-token cross-attention decoder +
-CLIFF camera conditioning.
+<p align="center">
+ <img src="assets/demo_desktop.gif" height="300" alt="InstantHMR on a desktop GPU, shown in the Rerun viewer">
+ <img src="assets/demo_phone.gif" height="300" alt="InstantHMR on a Samsung Galaxy S23">
+</p>
-It is trained on the **released ground-truth annotations** of
-[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)
-— the human MHR fits used to build SAM 3D Body — predicting per-person MHR
-parameters and 70 keypoints from a single 224×224 crop. Distilling from
-[`facebook/sam-3d-body-dinov3`](https://huggingface.co/facebook/sam-3d-body-dinov3)
-by running the model over your own images is supported as an **option** (see
-[Training](#training)), but it is not how the shipped models were trained.
+## Models
-InstantHMR ships as a single `.onnx` file. The demo pipeline pairs it with
-**RF-DETR** for person detection — both stages are timed independently so
-you always know where the latency is going. Optionally, the demo decodes
-the raw `mhr_params` / `shape_params` outputs through Meta's
-**[MHR body model](https://github.com/facebookresearch/MHR)** to render a
-full dense body mesh in the Rerun viewer.
+Four sizes, trained on the released annotations of the
+[SAM 3D Body dataset](https://huggingface.co/datasets/facebook/sam-3d-body-dataset).
+Accuracy is PA-MPJPE (lower is better) under the protocol of the paper; latency is per person,
+for the network as measured in the paper (the desktop column with TensorRT directly).
-- **Inputs:** `image (N, 3, 224, 224)`, `cliff_cond (N, 3)`.
-- **Outputs:** `mhr_params (204)`, `shape_params (45)`, `cam_trans (3)`,
- `joints_2d (70, 2)` (normalised crop coords),
- `joints_3d (70, 3)` (body-centred metres, Y-down).
-- **Speed (InstantHMR ONNX alone):** ~5 ms / frame (~200 FPS) on a single
- RTX 4070 with the fp16 ONNX (CUDA EP); CPU works too (~25 FPS, depending
- on hardware). On Apple Silicon, pass `--device coreml` to use
- `CoreMLExecutionProvider`.
-- **Speed (full demo, end-to-end):** the demo also runs **RF-DETR** every
- frame, and on most hardware the detector — not InstantHMR — is the
- bottleneck. Use `--detector-stride N` to run RF-DETR only every Nth
- frame and reuse the previous bbox in between (see *Performance tuning*
- below).
-- **Note for RTX 50-series (Blackwell, sm_120):** stock Torch with
- CUDA 12.4 wheels does not ship sm_120 kernels and falls back to slow
- paths. Use `python install.py` which automatically pulls Torch
- cu128 (>=2.7) on Blackwell GPUs.
+| Model | Parameters | EMDB-1 (mm) | 3DPW (mm) | RTX 4070, TensorRT (ms) | Galaxy S23 NPU (ms) | Licence |
+|---|---|---|---|---|---|---|
+| `nano` | 20.0 M | 61.7 | 46.6 | 0.70 | 4.60 | CC BY-NC 4.0 |
+| `tiny` | 20.3 M | 58.0 | 44.0 | 0.73 | 4.84 | CC BY-NC 4.0 |
+| `small` | 24.4 M | 53.3 | 41.7 | 0.84 | 5.38 | CC BY-NC 4.0 |
+| `medium` (default) | 32.2 M | 48.8 | 40.0 | 0.98 | 5.49 | CC BY-NC 4.0 |
+| `medium-commercial` | 32.2 M | 72.2 | 50.7 | 0.98 | 5.49 | Apache-2.0 |
-## Install
-
-```bash
-git clone <this-repo> instanthmr
-cd instanthmr
-```
+`medium-commercial` is the medium model trained only on images that allow commercial use
+(Harmony4D and part of COCO). The weights download automatically from
+[Hugging Face](https://huggingface.co/momolesang/InstantHMR) on first use.
-Choose the path that matches your needs:
+## Install
-| Goal | Python | Command |
-|------|--------|---------|
-| Pose estimation only (joints + skeleton) | 3.11 or 3.12 | `python install.py` |
-| + MHR body mesh rendering | **3.12** | `python install.py` then see [MHR body mesh](#mhr-body-mesh-rendering) |
+Python 3.10 to 3.12, and on Linux or Windows an NVIDIA driver 525 or newer.
```bash
-# Pose estimation only (Python 3.11+):
-conda create -n instanthmr python=3.11 -y && conda activate instanthmr
-
-# Picks the right torch + onnxruntime wheels for your machine:
-# - Linux + NVIDIA GPU → cu128 (Blackwell/RTX 50) or cu124 (Ada/Ampere/Hopper)
-# + onnxruntime-gpu + bundled CUDA / cuDNN runtime libs
-# - macOS → stock torch + onnxruntime (CoreML EP included)
-# - Linux without GPU → CPU torch + onnxruntime
-python install.py
+git clone https://github.com/Naturalpad-studio/InstantHMR.git
+cd InstantHMR
+python -m venv .venv && source .venv/bin/activate
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
+pip install -r requirements.txt
```
-`python install.py --dry-run` prints the pip commands without running them.
-`python install.py --force-cpu` skips GPU detection on Linux.
-If you'd rather manage wheels yourself, `pip install -r requirements.txt`
-still works — it installs the CPU fallback path.
+The first `pip` line installs PyTorch for CUDA 12, which the person detector uses and onnxruntime
+shares, so a single set of CUDA libraries is installed. On a Mac, or without an NVIDIA GPU, skip
+it.
-The first run downloads the RF-DETR (medium) checkpoint automatically.
-The InstantHMR ONNX weights are on HuggingFace — see
-[Model weights](#model-weights) below.
-
-## Run the demo
+## Demo
```bash
-# Single image
-python demo.py --image path/to/photo.jpg
-
-# Video file
-python demo.py --video path/to/clip.mp4
-
-# Live webcam (index 0)
+python demo.py --image photo.jpg
+python demo.py --video clip.mp4 --smooth
python demo.py --camera 0
```
-`python demo.py --help` lists every flag (model path, device, detector
-variant, confidence, max persons, frame skip, `.rrd` recording, …).
-
-The demo opens a Rerun viewer with the source image + 2D skeleton, a live
-RF-DETR / InstantHMR / total-latency plot, and the 3D scene with the
-predicted camera frustum.
-
-## MHR body mesh rendering
-
-The demo can render a full dense body mesh for each detected person by
-running a forward pass through Meta's **MHR (Momentum Human Rig)** body
-model, using the `mhr_params (204,)` and `shape_params (45,)` outputs that
-InstantHMR already produces. Render time is displayed separately in the
-Rerun latency plot and in the console.
-
-### Prerequisites
-
-| Requirement | Notes |
-|-------------|-------|
-| **Python ≥ 3.12** | `pymomentum` has no Python 3.11 wheels; recreate your conda env if needed |
-| NVIDIA GPU / Apple Silicon | CPU works but is much slower |
-| CUDA toolkit | Must match your PyTorch wheel (Linux/Windows) |
-
-### Step 1 — Install in one pip call
-
-Always pass **both** requirement files to a **single** `pip install` call.
-Running them separately causes pip to first choose a CUDA-13 torch for
-`rfdetr`, then downgrade to CUDA-12 torch for `pymomentum`, leaving
-`torchvision` broken.
-
-```bash
-conda create -n instanthmr python=3.12 -y && conda activate instanthmr
-pip install -r requirements.txt -r requirements-mhr.txt
-```
-
-`requirements-mhr.txt` auto-selects the right package for your platform
-(`pymomentum-gpu` on Linux/Windows, `pymomentum-cpu` on macOS).
-CPU-only Linux users: see the comment at the top of `requirements-mhr.txt`.
-
-> **⚠ Wrong `pymomentum` on PyPI**
-> `pip install pymomentum` installs an **unrelated** legacy SMS library
-> (pyMomentum v0.1.x by MomentumAS) — not the Meta geometry library.
-> `requirements-mhr.txt` pulls in the correct package automatically.
-> If you accidentally installed the wrong one first:
-> ```bash
-> pip uninstall pymomentum
-> pip install -r requirements.txt -r requirements-mhr.txt
-> ```
-
-> **Torch/torchvision version mismatch?**
-> ```bash
-> python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__)"
-> # If they differ, wipe both and reinstall together:
-> pip uninstall -y torch torchvision
-> pip install -r requirements.txt -r requirements-mhr.txt
-> ```
+The [Rerun](https://rerun.io) viewer opens with the frame and its 2D skeletons, the 3D scene, and
+the latency of each stage. People are found by the [RF-DETR](https://github.com/roboflow/rf-detr)
+detector, then InstantHMR runs on all of them in one batch.
-### Step 2 — Download the body-model assets
-
-```bash
-curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.1/assets.zip
-unzip assets.zip -d models/mhr_assets
-```
-
-This unpacks the body template, blend shapes, skinning weights, and pose
-correctives into `models/mhr_assets/`.
-
-### Step 3 — Run the demo
-
-```bash
-# Single image — LOD 0 (73 639 vertices, highest quality)
-python demo.py --image photo.jpg --mhr-assets models/mhr_assets --mhr-lod 0
-
-# Video file — LOD 1 (18 439 vertices, good quality)
-python demo.py --video clip.mp4 --mhr-assets models/mhr_assets --mhr-lod 1
-
-# Live camera — LOD 3 (4 899 vertices, real-time default)
-python demo.py --camera 0 --mhr-assets models/mhr_assets
-```
-
-### LOD reference
-
-| `--mhr-lod` | Vertices | Faces | Recommended use |
-|:-----------:|----------|-------|-----------------|
-| 0 | 73 639 | — | offline images, max quality |
-| 1 | 18 439 | — | video, powerful GPU |
-| 2 | 10 661 | — | video, mid-range GPU |
-| **3** | **4 899** | — | **live camera (default)** |
-| 4 | 2 461 | — | weak GPU / many persons |
-| 5 | 971 | — | borderline real-time |
-| 6 | 595 | — | debugging / stress test |
-
-### How it works
-
-```
-InstantHMR ONNX
- └─ mhr_params (204,) ──┐
- └─ shape_params (45,) ──┤─→ MHR.forward() → vertices (V, 3)
- └─ cam_trans (3,) ──┘ + cam_trans → camera-space mesh
-```
-
-The 204-dim `mhr_params` are MHR **model parameters**, not a 6-D rotation
-encoding. Read off `character_torch.parameter_transform.parameter_names`, the
-layout is:
-
-| slice | contents |
+| Option | |
|---|---|
-| `0:3` | `root_tx/ty/tz` — always 0 in the corpus; 1 unit = 10 cm, applied at joint 1 |
-| `3:6` | `root_rx/ry/rz` — extrinsic-XYZ Euler, driving joint-parameter slots 10/11/12 and nothing else |
-| `6:136` | 130 local joint angles (`spine0_rx_flexible`, `l_knee_ry`, …) |
-| `136:204` | 68 bone-scale parameters |
-
-Because the root rotation is exclusive to slots 10/11/12, `6:136` is exactly
-invariant to a camera roll — which is what `pose_split` in the trainer relies
-on. `shape_params (45,)` encode identity blend shapes (20 body + 20 head + 5
-hand components) and drive the **mesh only**: a ±2σ probe moves the 127-joint
-skeleton by `0.00e+00 cm`. Facial expression is set to neutral (zeros) since
-InstantHMR does not regress it.
-
-## Performance tuning
-
-On every machine we've measured, RF-DETR (the detector) costs ~5–10× more
-per frame than InstantHMR. The two flags below target that bottleneck.
-
-| Flag | Effect |
-| --- | --- |
-| `--detector-stride N` | Run RF-DETR every Nth frame; reuse the previous bbox (slightly expanded) on the in-between frames. Stride 2–3 is the single biggest knob — typically 2–3× end-to-end FPS for slow movement, with negligible quality loss. |
-| `--detector-variant nano` | Use the smallest RF-DETR; biggest win on CPU / Apple Silicon where the detector is doing all the work. |
-| `--device coreml` | On macOS, route the InstantHMR ONNX through `CoreMLExecutionProvider`. |
-| `--max-persons N` | Maximum number of persons processed per frame (default: **2**). See below. |
-| `--no-batch-persons` | Disable batched multi-person HMR (one ONNX call per person). The default is batched. |
-| `--focal PX` | Your camera's focal length **in pixels**, when you know it. Models exported from a `--cliff-focal` checkpoint are conditioned on *angles*, so they need a focal; without one the demo falls back to `1.05 x` the image diagonal, which is a reasonable phone/webcam stand-in but mis-places the person in depth on any camera whose true focal differs. Older pixel-conditioned graphs ignore it and are unaffected. |
-
-### `--max-persons` and why it matters
-
-`--max-persons` does two things at once:
-
-1. **Caps detections** — RF-DETR may find more people than you need. Setting `N`
- keeps only the `N` highest-confidence detections per frame.
-
-2. **Fixes the ONNX batch size** — CUDA / ONNX Runtime compiles GPU kernels the
- first time a model runs at a new input shape. Because the batch dimension
- counts as part of the shape, switching from 1 person to 2 persons mid-demo
- would normally trigger a multi-second stall. The pipeline avoids this by
- **always** padding the input to exactly `N` crops (zero-filling unused slots)
- so the session always sees `batch = N`, regardless of how many people are
- actually in frame.
-
-**Startup:** during `PosePipeline.__init__` the pipeline runs two warm-up passes
-— `batch = N` and `batch = 1` (the single-person path) — so all CUDA kernels
-are compiled before the first real frame. Startup cost is roughly `2 ×`
-one-inference time and does **not** scale with `N`.
-
-**Choosing `N`:**
-
-| Scene | Recommended `--max-persons` |
-|-------|----------------------------|
-| Single person / portrait | 1 |
-| Two people, couples, sparring | **2 (default)** |
-| Small group, team sport | 4–6 |
-| Crowd / many people visible | raise `N`; also consider `--detector-stride` |
-
-Raising `N` does not slow down single-person frames at inference time (the
-padded slots run on GPU in parallel), but it does proportionally increase
-the memory footprint of the batch buffer and may slightly increase latency
-when `N` is large and the GPU can no longer fully parallelise the extra crops.
-
-Measured on RTX 4070 + torch 2.5 cu121 + ORT 1.25 (1080p video, 1 person, 150 frames after warm-up):
-
-| `--detector-stride` | RF-DETR ms | HMR ms | total ms | FPS |
-| ---: | ---: | ---: | ---: | ---: |
-| 1 | 27.1 | 5.3 | 32.4 | **30.9** |
-| 2 | 13.3 | 5.3 | 18.6 | **53.8** |
-| 3 | 8.7 | 5.5 | 14.2 | **70.4** |
-| 4 | 6.6 | 5.3 | 11.9 | **83.9** |
-
-You can re-run the benchmark on your own hardware with:
-
-```bash
-python tools/bench.py --video vid1.mp4 --max-frames 150 --detector-stride 3
-```
+| `--model` | `nano`, `tiny`, `small`, `medium` (default), `medium-commercial`, or a path to an `.onnx` file |
+| `--device` | `cuda` (default), `tensorrt`, `cpu` or `coreml` |
+| `--smooth` | temporal smoothing for videos and cameras (1 Euro filter, no visible lag) |
+| `--focal` | the camera's focal length in pixels, if you know it; it places people at the right depth |
+| `--detector` | RF-DETR size: `nano`, `small`, `medium` (default), `base`, `large` |
+| `--max-persons` | most people per frame (default 4) |
+| `--no-mesh` | skeleton only |
+| `--save-rrd` | also save the recording, to replay with `rerun file.rrd` |
+
+The detector is the slow stage (about 27 ms per frame for RF-DETR medium on an RTX 4070, against
+5 ms for InstantHMR with TensorRT). For a faster live demo, use `--detector nano` or
+`--detector-stride 2`.
## Use it from Python
```python
+import cv2
from instanthmr import PosePipeline
-pipeline = PosePipeline(
- onnx_path="models/instanthmr.onnx",
- device="cuda",
- detector_variant="medium",
-)
-
-result = pipeline.predict(image_rgb)
-for r in result.persons:
- print(r.joints_3d_cam.shape) # (70, 3)
- print(r.joints_2d.shape) # (70, 2)
- print(r.mhr_params.shape) # (204,) — MHR pose parameters
- print(r.shape_params.shape) # (45,) — MHR identity shape parameters
-```
-
-The 70 keypoints follow the MHR70 ordering — see
-[`instanthmr/skeleton.py`](instanthmr/skeleton.py) for joint names and the
-canonical skeleton edge list.
-
-To run the MHR mesh decoder yourself:
-
-```python
-from instanthmr.mhr_renderer import MHRRenderer
-
-renderer = MHRRenderer(assets_folder="models/mhr_assets", device="cuda", lod=3)
-
-for r in result.persons:
- verts_local = renderer.forward(r.mhr_params, r.shape_params) # (V, 3)
- verts_cam = verts_local + r.cam_trans # camera space
- faces = renderer.faces # (F, 3) int32
+pipeline = PosePipeline("medium", device="cuda")
+image = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB)
+for person in pipeline(image).persons:
+ person.joints_2d # (70, 2) keypoints, pixels
+ person.joints_3d_cam # (70, 3) keypoints, metres, camera frame (x right, y down, z forward)
+ person.vertices_cam # (18439, 3) mesh, metres, camera frame; faces: pipeline.hmr.faces
+ person.mhr_params # (204,) MHR pose and skeleton parameters
+ person.shape_params # (45,) MHR body shape
```
-## Model weights
-
-The InstantHMR ONNX weights are hosted on HuggingFace:
-
-> **<https://huggingface.co/momolesang/InstantHMR>**
-
-Download `instanthmr.onnx` and place it under `models/`, or pass any path
-via `--model`.
-
-## Training
+With your own person boxes, use `InstantHMR` directly:
+`InstantHMR(resolve_model("medium")).predict(image, boxes)`. The 70 keypoints are listed in
+`instanthmr.JOINT_NAMES`.
-InstantHMR can be trained from two different label sources. **Both produce the
-exact same `images/*.png` + `annotations/*.npz` layout** that
-[`training/`](training/README.md) consumes, so training is identical either way:
+## TensorRT
-1. **Released ground-truth annotations — this is what every shipped model uses.**
- Train directly on
- [`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)
- — the human MHR fits used to build SAM 3D Body — converted to the `.npz`
- schema by [`tools/parquet_to_npz.py`](tools/parquet_to_npz.py) and laid out
- by [`datasets_pipeline/build_split.py`](datasets_pipeline/build_split.py)
- into `data/sam3d_gt_<dataset>/`. The released models were trained only on
- this kind (sa1b / aic / harmony4d / coco / mpii).
-2. **Teacher distillation — an option, not the default.** Run
- `facebook/sam-3d-body-dinov3` over your own images with
- [`tools/annotate_dataset.py`](tools/annotate_dataset.py) to generate
- per-person pseudo-labels — see [`docs/annotation.md`](docs/annotation.md).
- This is the only thing that writes `data/sam3d_distill_mix/`. Use it to label
- imagery the released dataset does not cover; note that it caps the student at
- the teacher's own accuracy, which training on the ground truth does not.
-
-The `data/sam3d_distill_mix` directory keeps "distill" in its name for
-historical reasons. The name does **not** mean the labels are teacher inference —
-only `tools/annotate_dataset.py` produces those.
-
-### Train on the original annotations (COCO example)
-
-The dataset is gated: first **request access** on the
-[dataset page](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)
-and `huggingface-cli login`. The annotation parquets ship **without images**, so
-you download the source images once per sub-dataset (COCO and MPII are the
-easiest — a single `wget`; others may need extra access / undistortion, see the
-[dataset setup guides](https://github.com/facebookresearch/sam-3d-body/tree/main/data)).
+TensorRT runs InstantHMR in fp16 on NVIDIA GPUs. Install it next to onnxruntime and pick the
+device:
```bash
-# 1. Annotations — download the coco_train split (uses Meta's sam-3d-body repo)
-git clone https://github.com/facebookresearch/sam-3d-body
-python sam-3d-body/data/scripts/download.py \
- --save_dir $ANN_DIR --splits coco_train
-
-# 2. Images — COCO 2014 train (the split the parquet filenames reference)
-mkdir -p $COCO_IMG_DIR && cd $COCO_IMG_DIR
-wget http://images.cocodataset.org/zips/train2014.zip && unzip -q train2014.zip
-cd -
-
-# 3. Convert parquet -> per-crop images/ + annotations/ (.npz)
-python tools/parquet_to_npz.py \
- --annotation_dir $ANN_DIR/coco_train \
- --image_dir $COCO_IMG_DIR \
- --output_dir data/sam3d_gt_coco \
- --validate
-
-# 4. Train on it (see training/README.md for the released models' settings):
-python training/train_ddp.py --data_root data --output_dir runs/coco
+pip install tensorrt-cu12==10.15.*
+python demo.py --camera 0 --device tensorrt
```
-The converter recovers `cam_trans` from the MHR params, derives the 1.2× square
-crop, skips low-quality fits (`mhr_valid`), and writes the same keys
-`tools/annotate_dataset.py` does. `--validate` asserts that each person's
-`joints_3d + cam_trans` reprojects onto the stored `joints_2d` to < 1 px.
-
-### Other splits (MPII, AI Challenger, Harmony4D, EgoHumans)
-
-Every split uses the **same converter** — the annotation schema and the MHR math
-are dataset-independent (verified: 0 px reprojection and an image-centred
-principal point on all of them). Only the **image preparation** differs. Download
-the annotations the same way (`download.py --splits <split>`), prepare the images
-as below, then run the converter into its own `data/sam3d_gt_<name>/` folder.
-
-| Split | Image source | Prep | `--image_dir` |
-|-------|--------------|------|---------------|
-| `mpii_train` | `wget` from MPI servers (no registration) | unpack `mpii_human_pose_v1.tar.gz` | `$MPII_IMG_DIR` (contains `images/`) |
-| `aic_train` | AI Challenger 2017 keypoint images (original site defunct — use a mirror) | arrange as `train/images/` | `$AIC_IMG_DIR` |
-| `harmony4d_train` | HF `Jyun-Ting/Harmony4D` | **undistort** with Meta's `data/scripts/harmony4d/` script → `$HARMONY4D_IMG_DIR` | `$HARMONY4D_IMG_DIR` |
-| `egohumans_train` | Google Drive (see setup guide) | **undistort** with Meta's `data/scripts/egohumans/` script → `$EGOHUMANS_IMG_DIR` | `$EGOHUMANS_IMG_DIR` |
-
-```bash
-# --- MPII -------------------------------------------------------------------
-python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits mpii_train
-mkdir -p $MPII_IMG_DIR && cd $MPII_IMG_DIR
-wget https://datasets.d2.mpi-inf.mpg.de/andriluka14cvpr/mpii_human_pose_v1.tar.gz
-tar xzf mpii_human_pose_v1.tar.gz && cd - # -> $MPII_IMG_DIR/images/
-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/mpii_train \
- --image_dir $MPII_IMG_DIR --output_dir data/sam3d_gt_mpii --validate
-
-# --- AI Challenger -----------------------------------------------------------
-python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits aic_train
-# place the keypoint images so that $AIC_IMG_DIR/train/images/<hash>.jpg exists
-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/aic_train \
- --image_dir $AIC_IMG_DIR --output_dir data/sam3d_gt_aic --validate
+The first run builds an engine for your GPU, about two minutes, and caches it in
+`models/trt_cache/`; later runs start in under a second. The engine covers batches of up to
+`--max-persons` people. On an RTX 4070, the `medium` network takes 1.2 ms per person with
+TensorRT against 3.5 ms with CUDA; the MHR body (keypoints and mesh) runs on CUDA in 1.2 ms
+either way.
-# --- Harmony4D (needs undistortion) -----------------------------------------
-python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits harmony4d_train
-# 1) download Jyun-Ting/Harmony4D into $HARMONY4D_DATA_DIR
-# 2) undistort -> $HARMONY4D_IMG_DIR (Meta's data/scripts/harmony4d/, see setup guide)
-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/harmony4d_train \
- --image_dir $HARMONY4D_IMG_DIR --output_dir data/sam3d_gt_harmony4d --validate
+## Training and evaluation
-# --- EgoHumans (needs undistortion) -----------------------------------------
-python sam-3d-body/data/scripts/download.py --save_dir $ANN_DIR --splits egohumans_train
-# 1) download EgoHumans into $EGOHUMANS_DATA_DIR
-# 2) undistort -> $EGOHUMANS_IMG_DIR (Meta's data/scripts/egohumans/, see setup guide)
-python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/egohumans_train \
- --image_dir $EGOHUMANS_IMG_DIR --output_dir data/sam3d_gt_egohumans --validate
-```
+- [`training/`](training/README.md): train the released models, or your own, from the SAM 3D Body
+ annotations, and export them to ONNX.
+- [`datasets_pipeline/`](datasets_pipeline/README.md): download the annotations and images and
+ build the training data.
+- [`benchmark/`](benchmark/README.md): reproduce the paper's EMDB-1 and 3DPW numbers.
-> **Harmony4D / EgoHumans must be undistorted first.** Their annotations
-> (`keypoints_2d`, `cam_int`) correspond to the **undistorted** frames produced
-> by Meta's per-dataset scripts, so point `--image_dir` at the undistorted output,
-> not the raw download. The image-prep details and exact undistortion commands
-> live in the [dataset setup guides](https://github.com/facebookresearch/sam-3d-body/tree/main/data).
+## Licence
-To train on several datasets at once, convert each into its own
-`data/sam3d_gt_<name>/` under the same `--data_root`; `train_ddp.py --mix` sets
-how often each one is sampled.
+The code is under the [Apache-2.0](LICENSE) licence. The `nano`, `tiny`, `small` and `medium`
+weights are under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), for
+non-commercial use only, because some of their training images (SA-1B, AI Challenger) are; `medium-commercial` and the MHR body graph are under Apache-2.0. The body model is Meta's
+[MHR](https://github.com/facebookresearch/MHR) (Apache-2.0).
-## Documentation
+## Acknowledgements
-- **[`training/README.md`](training/README.md)** — training and ONNX export.
-- **[`docs/annotation.md`](docs/annotation.md)** — training data generation
- with the SAM3D teacher.
+InstantHMR is trained on the annotations Meta released with
+[SAM 3D Body](https://github.com/facebookresearch/sam-3d-body) and uses Meta's
+[MHR](https://github.com/facebookresearch/MHR) body model. People are detected with
+[RF-DETR](https://github.com/roboflow/rf-detr) and shown with [Rerun](https://rerun.io). Training
+was granted access to the HPC resources of IDRIS under the allocation AD011017116 made by GENCI.
-## License
+## Citation
-The code in this repository is released under the **Apache License 2.0**
-(see [LICENSE](LICENSE)).
-
-The model weights distributed at
-<https://huggingface.co/momolesang/InstantHMR> are released under the
-[SAM license](https://github.com/facebookresearch/sam-3d-body), since
-InstantHMR is trained on `facebook/sam-3d-body-dataset` labels. Please
-review the SAM and RF-DETR licenses before downstream use.
+<!-- ticket 20: add the authors and the arXiv identifier. -->
+```bibtex
+@article{instanthmr2026,
+ title = {InstantHMR: Real-Time Human Mesh Recovery on Your Phone},
+ year = {2026}
+}
+```
diff --git a/assets/demo_desktop.gif b/assets/demo_desktop.gif
new file mode 100644
index 0000000..0a4c55d
Binary files /dev/null and b/assets/demo_desktop.gif differ
diff --git a/assets/demo_phone.gif b/assets/demo_phone.gif
new file mode 100644
index 0000000..ad3743b
Binary files /dev/null and b/assets/demo_phone.gif differ
diff --git a/benchmark/README.md b/benchmark/README.md
new file mode 100644
index 0000000..ef5a965
--- /dev/null
+++ b/benchmark/README.md
@@ -0,0 +1,55 @@
+# Evaluation
+
+`evaluate.py` scores an InstantHMR model on EMDB-1 and 3DPW test under the protocol of the paper:
+every 10th frame, ground-truth person boxes, the calibrated focal length, and Meta's official MHR
+to SMPL fitting. EMDB-1 is scored on the 24 SMPL joints and 3DPW on the 14 joints regressed with
+the Human3.6M regressor, as in each benchmark's literature; both joint sets are printed.
+
+## Setup
+
+```bash
+pip install -r requirements.txt -r benchmark/requirements.txt
+git clone https://github.com/facebookresearch/MHR.git ../MHR
+curl -L -o mhr_assets.zip https://github.com/facebookresearch/MHR/releases/latest/download/assets.zip
+mkdir -p training/checkpoints && unzip -j mhr_assets.zip assets/mhr_model.pt -d training/checkpoints
+```
+
+Then put these files in one folder, the `--smpl-dir`:
+
+| File | Where from |
+|---|---|
+| `SMPL_MALE.pkl`, `SMPL_FEMALE.pkl` | [SMPL](https://smpl.is.tue.mpg.de), "SMPL for Python", renamed |
+| `SMPL_NEUTRAL.pkl` | [SMPLify](https://smplify.is.tue.mpg.de), `basicModel_neutral_lbs_10_207_0_v1.0.0.pkl`, renamed |
+| `J_regressor_h36m.npy` | the data of [SPIN](https://github.com/nkolot/SPIN) (`fetch_data.sh`) |
+
+And the benchmarks:
+
+- **EMDB**: request access at [emdb.ait.ethz.ch](https://eth-ait.github.io/emdb/); `--data` is the
+ folder that holds `P0/`, `P1/`, ...
+- **3DPW**: from [virtualhumans.mpi-inf.mpg.de/3DPW](https://virtualhumans.mpi-inf.mpg.de/3DPW/);
+ `--data` is the folder that holds `sequenceFiles/` and `imageFiles/`.
+
+## Run
+
+```bash
+python benchmark/evaluate.py --dataset emdb --data <EMDB> --model medium \
+ --smpl-dir <SMPL files> --mhr-repo ../MHR
+python benchmark/evaluate.py --dataset 3dpw --data <3DPW> --model medium \
+ --smpl-dir <SMPL files> --mhr-repo ../MHR
+```
+
+`--limit 50` scores the first 50 frames as a quick check. The fitting runs on the GPU; a full
+benchmark takes a few minutes.
+
+## Expected results
+
+`medium` reproduces Table 2 of the paper:
+
+| | PA-MPJPE (mm) | MPJPE (mm) | PVE (mm) |
+|---|---|---|---|
+| EMDB-1, 24 joints | 48.8 | 74.9 | 89.1 |
+| 3DPW, 14 joints | 40.0 | 64.0 | 76.8 |
+
+The paper ran the PyTorch checkpoints in fp16 and the released ONNX graphs run in fp32, so single
+frames differ slightly; the means above agree to 0.003 mm. Meta's fitting is not bitwise
+deterministic on a GPU, so a rerun can move them by a few hundredths of a millimetre.
diff --git a/benchmark/evaluate.py b/benchmark/evaluate.py
new file mode 100644
index 0000000..b4f8755
--- /dev/null
+++ b/benchmark/evaluate.py
@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""Score an InstantHMR model on EMDB-1 or 3DPW test under the paper's protocol.
+
+Protocol: every 10th frame; ground-truth person boxes (EMDB's own boxes; for 3DPW, the extent of
+the 2D keypoints with confidence > 0.1, padded by 20%); the calibrated focal length; the MHR mesh
+(with Meta's pose correctives) converted to SMPL by Meta's official fitting
+(facebookresearch/MHR, tools/mhr_smpl_conversion); EMDB-1 scored on the 24 SMPL joints, 3DPW on
+the 14 joints regressed from the mesh with the H36M regressor. Both joint sets are printed.
+
+ python benchmark/evaluate.py --dataset emdb --data <EMDB root> --model medium \\
+ --smpl-dir <SMPL files> --mhr-repo <MHR clone> --mhr-model training/checkpoints/mhr_model.pt
+ python benchmark/evaluate.py --dataset 3dpw --data <3DPW root> ...
+
+See benchmark/README.md for the downloads.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import pickle
+import sys
+import types
+from pathlib import Path
+
+import cv2
+import numpy as np
+import torch
+from tqdm import tqdm
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent))
+sys.path.insert(0, str(HERE))
+
+from benchlib import joints as J # noqa: E402
+from benchlib import metrics as M # noqa: E402
+from instanthmr import InstantHMR, resolve_model # noqa: E402
+from make_3dpw_gt import _install_chumpy_stub, _to_numpy, load_smpl, smpl_forward # noqa: E402
+
+MM = 1000.0
+
+
+class _FacesOnlyRig:
+ """What Meta's MHR -> SMPL conversion reads from an MHR model: its faces."""
+
+ def __init__(self, faces):
+ self.character = types.SimpleNamespace(mesh=types.SimpleNamespace(faces=faces))
+
+ def to(self, _device):
+ return self
+
+
+class _AnyName(types.ModuleType):
+ def __getattr__(self, name):
+ if name.startswith("__"):
+ raise AttributeError(name)
+ return type(name, (), {})
+
+
+def emdb_frames(root: Path, stride: int) -> list[dict]:
+ """EMDB-1: frames with `good_frames_mask` set and not in `bboxes.invalid_idxs`, EMDB's boxes."""
+ out = []
+ for pkl in sorted(root.glob("P*/*/*_data.pkl")):
+ with open(pkl, "rb") as f:
+ d = pickle.load(f)
+ if not d["emdb1"]:
+ continue
+ good, bad = np.asarray(d["good_frames_mask"], bool), set(np.ravel(d["bboxes"]["invalid_idxs"]).tolist())
+ s = d["smpl"]
+ for f in range(0, int(d["n_frames"]), stride):
+ if good[f] and f not in bad:
+ out.append(dict(image=pkl.parent / "images" / f"{f:05d}.jpg", bbox=np.float32(d["bboxes"]["bboxes"][f]),
+ K=np.asarray(d["camera"]["intrinsics"]), seq=str(d["name"]),
+ gender=str(d["gender"]), betas=np.ravel(s["betas"])[:10],
+ pose=np.concatenate([s["poses_root"][f], s["poses_body"][f]]), trans=s["trans"][f],
+ extrinsics=np.asarray(d["camera"]["extrinsics"][f])))
+ return out
+
+
+def threedpw_frames(root: Path, stride: int) -> list[dict]:
+ """3DPW test: `campose_valid` people with >= 4 confident 2D keypoints, box = their extent + 20%."""
+ out = []
+ for pkl in sorted((root / "sequenceFiles" / "test").glob("*.pkl")):
+ with open(pkl, "rb") as f:
+ seq = pickle.load(f, encoding="latin1")
+ for f in range(0, len(seq["poses"][0]), stride):
+ for p in range(len(seq["poses"])):
+ kp = np.asarray(seq["poses2d"][p][f])
+ ok = kp[2] > 0.1
+ if not seq["campose_valid"][p][f] or ok.sum() < 4:
+ continue
+ (x1, y1), (x2, y2) = kp[:2, ok].min(1), kp[:2, ok].max(1)
+ w, h = x2 - x1, y2 - y1
+ out.append(dict(image=root / "imageFiles" / seq["sequence"] / f"image_{f:05d}.jpg",
+ bbox=np.float32([x1 - 0.2 * w, y1 - 0.2 * h, x2 + 0.2 * w, y2 + 0.2 * h]),
+ K=np.asarray(seq["cam_intrinsics"]), seq=seq["sequence"],
+ gender=str(seq["genders"][p]), betas=np.ravel(seq["betas"][p])[:10],
+ pose=seq["poses"][p][f], trans=seq["trans"][p][f], extrinsics=seq["cam_poses"][f]))
+ return out
+
+
+def ground_truth(samples: list[dict], smpl_dir: Path, device) -> tuple[np.ndarray, np.ndarray]:
+ """(N, 24, 3) SMPL joints and (N, 6890, 3) vertices, camera space, metres."""
+ models, joints, verts = {}, [], []
+ for s in samples:
+ g = "MALE" if s["gender"].lower().startswith("m") else "FEMALE"
+ if g not in models:
+ models[g] = load_smpl(smpl_dir / f"SMPL_{g}.pkl", device)
+ m = models[g]
+ betas = torch.zeros(m["shapedirs"].shape[-1], device=device)
+ betas[:10] = torch.as_tensor(np.float32(s["betas"]), device=device)
+ v, j = smpl_forward(m, torch.as_tensor(np.float32(s["pose"])[None], device=device), betas,
+ torch.as_tensor(np.float32(s["trans"])[None], device=device))
+ E = torch.as_tensor(np.float32(s["extrinsics"]), device=device)
+ verts.append((v[0] @ E[:3, :3].T + E[:3, 3]).cpu().numpy())
+ joints.append((j[0] @ E[:3, :3].T + E[:3, 3]).cpu().numpy())
+ return np.stack(joints), np.stack(verts)
+
+
+def predict(samples: list[dict], model: Path, device: str) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ hmr = InstantHMR(model, device=device, max_batch=1)
+ params, shapes, trans = [], [], []
+ for s in tqdm(samples, desc="InstantHMR", leave=False):
+ rgb = cv2.cvtColor(cv2.imread(str(s["image"])), cv2.COLOR_BGR2RGB)
+ p = hmr.predict(rgb, [s["bbox"]], focal=float(s["K"][0, 0]))[0]
+ params.append(p.mhr_params), shapes.append(p.shape_params), trans.append(p.cam_trans)
+ return np.stack(params), np.stack(shapes), np.stack(trans)
+
+
+def mhr_vertices(rig, params: np.ndarray, shapes: np.ndarray, device) -> np.ndarray:
+ """The full rig's mesh (pose correctives included), body-centred, metres, camera axes."""
+ out = []
+ with torch.no_grad():
+ for i in range(0, len(params), 256):
+ p, s = (torch.as_tensor(a[i:i + 256], device=device) for a in (params, shapes))
+ v, _ = rig(s, p, torch.zeros(len(p), 72, device=device), True)
+ out.append((v * torch.tensor([0.01, -0.01, -0.01], device=device)).cpu().numpy())
+ return np.concatenate(out)
+
+
+def official_smpl_fit(verts: np.ndarray, cam_trans: np.ndarray, faces: np.ndarray, mhr_repo: Path, smpl_dir: Path,
+ device) -> tuple[np.ndarray, np.ndarray]:
+ """Meta's MHR -> SMPL conversion, called unchanged: (N, 24, 3) joints and (N, 6890, 3) vertices.
+
+ It only reads the rig's faces from the `mhr` package and imports `pymomentum.solver` without using
+ it for this direction, so both are replaced by stand-ins and neither needs installing.
+ """
+ import smplx
+ from smplx.utils import Struct
+
+ _install_chumpy_stub()
+ with open(smpl_dir / "SMPL_NEUTRAL.pkl", "rb") as f:
+ neutral = pickle.load(f, encoding="latin1")
+ neutral = Struct(**{k: _to_numpy(neutral[k]) for k in ("v_template", "shapedirs", "posedirs", "J_regressor",
+ "weights", "f", "kintree_table")})
+ neutral.f, neutral.kintree_table = neutral.f.astype(np.int64), neutral.kintree_table.astype(np.int64)
+ rig = _FacesOnlyRig(faces)
+ mhr_module = types.ModuleType("mhr.mhr")
+ mhr_module.MHR = _FacesOnlyRig
+ sys.modules.update({"mhr": types.ModuleType("mhr"), "mhr.mhr": mhr_module,
+ "pymomentum": types.ModuleType("pymomentum"),
+ "pymomentum.solver": _AnyName("pymomentum.solver")})
+ tool = mhr_repo / "tools" / "mhr_smpl_conversion"
+ sys.path.insert(0, str(tool))
+ cwd = os.getcwd()
+ os.chdir(tool)
+ try:
+ from conversion import Conversion
+
+ smpl = smplx.SMPL(model_path=str(smpl_dir), data_struct=neutral, gender="neutral").to(device)
+ conversion = Conversion(mhr_model=rig, smpl_model=smpl, method="pytorch")
+ joints, vertices = [], []
+ for i in range(0, len(verts), 512):
+ items = [dict(mhr_model_params=np.zeros(1, np.float32), shape_params=np.zeros(1, np.float32),
+ expr_params=np.zeros(1, np.float32), pred_vertices=verts[k], pred_cam_t=cam_trans[k])
+ for k in range(i, min(i + 512, len(verts)))]
+ res = conversion.convert_sam3d_output_to_smpl(items, return_smpl_parameters=True)
+ p = {k: torch.as_tensor(v, device=device).float() for k, v in res.result_parameters.items()}
+ with torch.no_grad():
+ o = smpl(betas=p["betas"], body_pose=p["body_pose"], global_orient=p["global_orient"],
+ transl=p["transl"])
+ joints.append(o.joints[:, :24].cpu().numpy()), vertices.append(o.vertices.cpu().numpy())
+ finally:
+ os.chdir(cwd)
+ return np.concatenate(joints), np.concatenate(vertices)
+
+
+def score(joint_set: str, jp, vp, jg, vg, h36m_regressor) -> dict:
+ """PA-MPJPE, MPJPE and PVE in mm. "24": SMPL joints rooted at the mid-hip (joints 1, 2);
+ "14": the H36M-regressed joints rooted at their pelvis. PVE uses the same root as MPJPE."""
+ vp, vg = vp.astype(np.float64) * MM, vg.astype(np.float64) * MM
+ if joint_set == "24":
+ jp, jg = jp.astype(np.float64) * MM, jg.astype(np.float64) * MM
+ rp, rg = jp[:, 1:3].mean(1), jg[:, 1:3].mean(1)
+ else:
+ jp = np.einsum("jv,nvc->njc", h36m_regressor, vp)[:, J.J14_FROM_H36M17]
+ jg = np.einsum("jv,nvc->njc", h36m_regressor, vg)[:, J.J14_FROM_H36M17]
+ rp, rg = J.pelvis(jp), J.pelvis(jg)
+ return dict(PA_MPJPE=float(M.pa_mpjpe(jp, jg).mean()), MPJPE=float(M.mpjpe(jp, jg, root=(rp, rg)).mean()),
+ PVE=float(M.mpjpe(vp, vg, root=(rp, rg)).mean()))
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--dataset", required=True, choices=("emdb", "3dpw"))
+ ap.add_argument("--data", type=Path, required=True, help="EMDB root (P0/, P1/, ...) or 3DPW root "
+ "(sequenceFiles/, imageFiles/)")
+ ap.add_argument("--model", default="medium", help="model name or .onnx path")
+ ap.add_argument("--smpl-dir", type=Path, required=True,
+ help="SMPL_MALE.pkl, SMPL_FEMALE.pkl, SMPL_NEUTRAL.pkl, J_regressor_h36m.npy")
+ ap.add_argument("--mhr-repo", type=Path, required=True, help="clone of github.com/facebookresearch/MHR")
+ ap.add_argument("--mhr-model", type=Path, default=HERE.parent / "training" / "checkpoints" / "mhr_model.pt")
+ ap.add_argument("--stride", type=int, default=10)
+ ap.add_argument("--limit", type=int, help="score only the first N frames (a quick check)")
+ ap.add_argument("--device", default="cuda", choices=("cuda", "cpu"))
+ ap.add_argument("--save", type=Path, help="also save the SMPL predictions (.npz)")
+ args = ap.parse_args()
+
+ samples = (emdb_frames if args.dataset == "emdb" else threedpw_frames)(args.data, args.stride)[:args.limit]
+ print(f"{args.dataset}: {len(samples)} frames")
+ params, shapes, cam_trans = predict(samples, resolve_model(args.model), args.device)
+ rig = torch.jit.load(str(args.mhr_model), map_location=args.device).eval()
+ verts = mhr_vertices(rig, params, shapes, args.device)
+ faces = rig.character_torch.mesh.faces.cpu().numpy()
+ jp, vp = official_smpl_fit(verts, cam_trans, faces, args.mhr_repo, args.smpl_dir, args.device)
+ jg, vg = ground_truth(samples, args.smpl_dir, args.device)
+ h36m = np.load(args.smpl_dir / "J_regressor_h36m.npy").astype(np.float64)
+
+ published = "24" if args.dataset == "emdb" else "14"
+ results = {js: score(js, jp, vp, jg, vg, h36m) for js in ("24", "14")}
+ for js, r in results.items():
+ mark = " <- paper" if js == published else ""
+ print(f"J{js}: PA-MPJPE {r['PA_MPJPE']:.1f} MPJPE {r['MPJPE']:.1f} PVE {r['PVE']:.1f} mm{mark}")
+ if args.save:
+ np.savez(args.save, joints=jp, vertices=vp, mhr_params=params, shape_params=shapes, cam_trans=cam_trans,
+ results=json.dumps(results))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/requirements.txt b/benchmark/requirements.txt
new file mode 100644
index 0000000..ee277ca
--- /dev/null
+++ b/benchmark/requirements.txt
@@ -0,0 +1,5 @@
+torch
+smplx
+trimesh
+scikit-learn
+tqdm
diff --git a/datasets_pipeline/README.md b/datasets_pipeline/README.md
index 400dc4b..e0bd172 100644
--- a/datasets_pipeline/README.md
+++ b/datasets_pipeline/README.md
@@ -1,62 +1,44 @@
# datasets_pipeline
-Rebuild any split of the SAM-3D-Body corpus from scratch, keeping the original
-frames this time, into the layout the training code already understands.
+Builds the training data from the released annotations of
+[SAM 3D Body](https://huggingface.co/datasets/facebook/sam-3d-body-dataset) and the original
+images of each dataset. One folder per dataset:
```
-data/sam3d_gt_<dataset>/
-├── original_images/ referenced source frames, native relative paths
-├── annotations/ <name>.npz
-├── body_crops/ <name>.png 224 square person crop
-├── hands_crops/ <name>_l.png <name>_r.png
-└── images -> body_crops compat symlink
+<data root>/sam3d_gt_<dataset>/
+├── annotations/ <name>.npz MHR parameters, camera, 2D and 3D keypoints of one person
+└── images/ <name>.png the person crop
```
-`body_crops` is **byte-identical** to what `tools/parquet_to_npz.py` writes into
-`images/` — verified md5-for-md5 on 120 AIC crops against the existing
-`data/sam3d_gt_aic/`. `build_split.py` imports that module's crop and annotation
-builders rather than re-implementing them, so the two can never drift. The
-`images` symlink means `SAM3DStudentDataset` finds these folders with **no code
-change**; that path is tested, not assumed.
-
-## The scripts
-
| script | does |
|---|---|
-| `fetch_annotations.py` | parquet shards from the gated HF repo |
-| `probe_split.py` | **is this split worth its download?** — from annotations alone |
-| `fetch_images.py` | source frames, per dataset |
-| `build_split.py` | parquet + frames → the folder above |
-| `verify_split.py` | integrity, reprojection, and how much hand signal a split carries |
-
-Plus `plan_disk.py` (cost a split before downloading it), `stream_archives.py`
-(build a dataset larger than your disk), `registry.py` (every URL and gotcha in
-one place), and `undistort.md`.
+| `fetch_annotations.py` | downloads the annotation shards (gated: accept the terms on Hugging Face first) |
+| `probe_split.py` | tells you what a split contains, from its annotations alone |
+| `plan_disk.py` | how much disk a split will take |
+| `fetch_images.py` | downloads the source images, or prints how to get them |
+| `build_split.py` | annotations + images -> the folder above |
+| `verify_split.py` | checks a built split |
-Annotations are ~1 GB per split; frames are tens to thousands of GB. So the
-order is always: fetch annotations → `probe_split.py` → decide → fetch frames.
+`registry.py` holds every URL and per-dataset detail; `stream_archives.py` builds a dataset larger
+than your disk; `undistort.md` covers the datasets whose frames must be undistorted first.
## Quick start
-```bash
-export ANN_DIR=/scratch/sam3d_ann
-export IMG_ROOT=/scratch/sam3d_img
-
-python datasets_pipeline/fetch_annotations.py --list
-python datasets_pipeline/fetch_annotations.py --splits coco_train --out $ANN_DIR
-python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/coco_train
-
-python datasets_pipeline/fetch_images.py --dataset coco --dest $IMG_ROOT/coco
-python datasets_pipeline/build_split.py \
- --split coco_train \
- --annotation-dir $ANN_DIR/coco_train \
- --image-dir $IMG_ROOT/coco \
- --data-root data --validate
+The released models were trained on a **window corpus**: for each person, four square crops cut
+from the full image (the standard 1.2x box crop and three other framings), stored side by side at
+288 px. COCO as an example:
-python datasets_pipeline/verify_split.py --dir data/sam3d_gt_coco
+```bash
+python datasets_pipeline/fetch_annotations.py --splits coco_train --out annotations
+python datasets_pipeline/fetch_images.py --dataset coco --dest images/coco
+python datasets_pipeline/build_split.py --split coco_train \
+ --annotation-dir annotations/coco_train --image-dir images/coco \
+ --windows 4 --window-sizes 288 --no-hands --output-dir 'data_win{size}/sam3d_gt_coco'
```
-Everything is resume-safe: re-running skips crops already on disk.
+Then train with `--data_root data_win288` (see [`training/README.md`](../training/README.md)).
+Without `--windows`, `build_split.py` writes single crops into `data/sam3d_gt_<dataset>/`, which
+the trainer also reads. Every script resumes where it stopped.
## What each dataset costs, and how to get it
diff --git a/demo.py b/demo.py
index 5c8fd4a..0b94ae4 100755
--- a/demo.py
+++ b/demo.py
@@ -1,666 +1,100 @@
#!/usr/bin/env python3
-"""InstantHMR — demo runner.
+"""InstantHMR demo: people in an image, a video or a live camera, shown in the Rerun viewer.
-Detects persons in a video / image / live camera stream with RF-DETR, runs
-InstantHMR on each, and visualises the results in Rerun: 3D joints, camera
-pose as a pinhole frustum, the source image with the projected 2D skeleton
-drawn on top, and live latency plots for RF-DETR, InstantHMR, MHR mesh
-rendering, and total pipeline cost.
-
-Optionally renders a full MHR body mesh per person by decoding the
-``mhr_params`` and ``shape_params`` outputs through Meta's MHR model, either
-from the TorchScript rig used by the distillation (``--mhr-model``) or from the
-unpacked asset folder (``--mhr-assets``). See ``instanthmr/mhr_renderer.py``.
-
-MHR-only checkpoints
---------------------
-A model exported from ``train.py`` has FOUR ONNX outputs — it
-has no 3D coordinate head, so its 70 keypoints are derived from the MHR forward
-pass, exactly like the SAM3D teacher does. Such a model therefore *requires* an
-MHR rig; when none is requested explicitly the demo falls back to
-``checkpoints/mhr_model.pt``.
-
-Usage
------
- # Single image (skeleton only; needs a 5-output model)
- python demo.py --image path/to/photo.jpg
-
- # Single image with the MHR mesh, from the TorchScript rig
- python demo.py --image path/to/photo.jpg --mhr-model
-
- # MHR-only student: the rig is mandatory and auto-selected
- python demo.py --video vid1.mp4 --model models/instanthmr_mhr_only.onnx
-
- # Same, but decoding the mesh from the asset folder at a chosen LOD
- python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets --mhr-lod 3
-
- # Video file
- python demo.py --video path/to/clip.mp4
-
- # Video file, with the saved SAM 3D Body teacher overlaid for comparison
- # (bare flag reads reference/<video stem>.npz)
- python demo.py --video vid1.mp4 --mhr-model --show-reference
-
- # Live camera (index 0)
- python demo.py --camera 0
-
- # Foot-contact model (--prev-token checkpoint): each person is fed their own
- # prediction from 5 frames earlier, and planted feet are drawn red, with a
- # per-foot contact-probability plot. --no-prev withholds the previous pose.
- python demo.py --video clip.mp4 --model models/instanthmr_g11Ap_b4_ep119.onnx
+ python demo.py --image photo.jpg
+ python demo.py --video clip.mp4 --smooth
+ python demo.py --camera 0 --model small --device tensorrt
"""
-
from __future__ import annotations
import argparse
-from collections import deque
import sys
import time
-from pathlib import Path
import cv2
import numpy as np
-from instanthmr import PosePipeline, onnx_output_names
-from instanthmr.mhr_renderer import DEFAULT_MHR_SCRIPT
-from instanthmr.reference import ReferenceTrack, resolve_reference
+from instanthmr import DEFAULT_MODEL, MODELS, PosePipeline
from instanthmr.smoothing import TemporalSmoother
from instanthmr.visualizer import RerunVisualizer
-DEFAULT_MODEL = "models/instanthmr.onnx"
-
-
def parse_args() -> argparse.Namespace:
- p = argparse.ArgumentParser(
- description="Run InstantHMR (RF-DETR + ONNX HMR) on a video, image, or camera.",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- )
-
+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
src = p.add_mutually_exclusive_group(required=True)
- src.add_argument("--video", type=str, help="Path to an input video file.")
- src.add_argument("--image", type=str, help="Path to an input image.")
- src.add_argument(
- "--camera", type=int, metavar="IDX",
- help="Live camera index (0, 1, ...).",
- )
-
- p.add_argument(
- "--model", type=str, default=DEFAULT_MODEL,
- help=f"Path to the InstantHMR ONNX file (default: {DEFAULT_MODEL}).",
- )
- p.add_argument(
- "--device", type=str, default="cuda",
- help="Inference device for InstantHMR: 'cuda', 'coreml', or 'cpu' (default: cuda).",
- )
- p.add_argument(
- "--detector-variant", type=str, default="medium",
- choices=["nano", "small", "medium", "base", "large"],
- help="RF-DETR size variant (default: medium).",
- )
- p.add_argument(
- "--det-confidence", type=float, default=0.5,
- help="RF-DETR detection confidence threshold (default: 0.5).",
- )
- p.add_argument(
- "--max-persons", type=int, default=2,
- help="Max persons processed per frame (default: 2).",
- )
- p.add_argument(
- "--frame-skip", type=int, default=1,
- help="Process every Nth video frame (default: 1).",
- )
- p.add_argument(
- "--detector-stride", type=int, default=1,
- help="Run RF-DETR every Nth processed frame; reuse the previous bbox in "
- "between (default: 1). The detector is the dominant cost — stride "
- "2-3 typically doubles end-to-end FPS with negligible quality loss.",
- )
- p.add_argument(
- "--no-batch-persons", action="store_true",
- help="Disable batched multi-person HMR (one ONNX call per person).",
- )
- p.add_argument(
- "--focal", type=float, default=None,
- help="Camera focal length in PIXELS, when it is known. A graph exported "
- "from a --cliff-focal checkpoint is conditioned on angles, so it "
- "needs one; without it the demo falls back to 1.05x the image "
- "diagonal, which mis-places the person in depth on any camera "
- "whose true focal differs.",
- )
- p.add_argument(
- "--no-prev", action="store_true",
- help="Prev-token models only: withhold the previous pose on every frame "
- "(the single-frame model). By default each tracked person is fed "
- "their own prediction from --prev-lag frames earlier.",
- )
- p.add_argument(
- "--prev-lag", type=int, default=5, metavar="N",
- help="Processed frames between a prediction and the frame it feeds "
- "(default: 5, as in the benchmarks; the token was trained on pairs "
- "5-20 frames apart).",
- )
- p.add_argument(
- "--contact-threshold", type=float, default=0.5, metavar="P",
- help="Prev-token models only: a foot is drawn planted when its contact "
- "probability exceeds P (default: 0.5). On held-out jumps, "
- "0.8 gets both feet off in 92%% of flight frames instead of 83%%, "
- "and keeps 98%% (instead of 99%%) of both-planted frames.",
- )
- p.add_argument(
- "--smoothing-filter", action="store_true",
- help="Temporally smooth per-person predictions with a moving-average "
- "filter (see instanthmr/smoothing.py). Non-learned and free; NLF "
- "reports a 5-frame filter of this kind giving 3DPW 59.0 -> 57.2 mm.",
- )
- p.add_argument(
- "--smoothing-window", type=int, default=5, metavar="N",
- help="Frames averaged by --smoothing-filter (default: 5). Rounded up to "
- "the next odd number in centered mode.",
- )
- p.add_argument(
- "--smoothing-mode", choices=("centered", "causal", "1euro"),
- default="centered",
- help="'centered' = moving average, no phase lag, delays output by "
- "(N-1)//2 frames — the filter NLF measured, so use it for benchmark "
- "numbers. 'causal' = trailing average, zero latency but real phase "
- "lag. '1euro' = adaptive cutoff (Casiez et al. 2012), zero latency "
- "and no lag on fast motion — use it for demos. --camera always uses "
- "1euro (default: centered).",
- )
- p.add_argument(
- "--smoothing-beta", type=float, default=1.0, metavar="S",
- help="Scales every per-field beta in --smoothing-mode 1euro (default: "
- "1.0). The jitter-versus-lag knob: raise it if motion still lags, "
- "lower it if the pose looks noisy while you hold still.",
- )
- p.add_argument(
- "--smoothing-mincutoff", type=float, default=1.0, metavar="S",
- help="Scales every per-field mincutoff in --smoothing-mode 1euro "
- "(default: 1.0). Governs how hard the pose is smoothed at rest.",
- )
- p.add_argument(
- "--save-rrd", type=str, default=None,
- help="Optional path to save the Rerun recording as a .rrd file.",
- )
- p.add_argument(
- "--no-spawn", action="store_true",
- help="Don't spawn the Rerun viewer GUI (useful when only saving .rrd).",
- )
- p.add_argument(
- "--mhr-model", type=str, nargs="?", default=None, const=DEFAULT_MHR_SCRIPT,
- metavar="PT",
- help=(
- "Decode the body mesh with the TorchScript MHR rig used by the "
- "distillation (bare flag = checkpoints/mhr_model.pt). Needs only "
- "PyTorch, so it works regardless of the pymomentum build. Takes "
- "precedence over --mhr-assets."
- ),
- )
- p.add_argument(
- "--mhr-assets", type=str, default=None, metavar="DIR",
- help=(
- "Path to the unpacked MHR assets folder. When provided, runs a "
- "full MHR forward pass per person and renders the body mesh in "
- "the 3D scene. See instanthmr/mhr_renderer.py for setup."
- ),
- )
- p.add_argument(
- "--mhr-lod", type=int, default=3, metavar="N",
- choices=range(7),
- help=(
- "MHR level-of-detail for --mhr-assets (0=73 639 verts … 6=595 "
- "verts). Default: 3 (4 899 verts). The TorchScript rig has a fixed "
- "LOD (18 439 verts) and ignores this."
- ),
- )
- p.add_argument(
- "--show-reference", type=str, nargs="?", default=None, const="",
- metavar="NPZ",
- help=(
- "Overlay a saved SAM 3D Body (teacher) prediction for this clip, "
- "so you can compare it against the student in the same 3D scene. "
- "Bare flag = reference/<video stem>.npz. The reference stores MHR "
- "parameters and is decoded by the same rig as the student, so any "
- "difference you see is the model, not the renderer. Build one with "
- "tools/pack_reference.py — see instanthmr/reference.py."
- ),
- )
- p.add_argument(
- "--mesh-alpha", type=float, default=0.35, metavar="A",
- help=(
- "Body-mesh opacity in [0, 1] (default: 0.35). Applied as a Rerun "
- "albedo factor so the 70 keypoints stay visible inside the body; "
- "raise it towards 1.0 for a solid mesh."
- ),
- )
-
+ src.add_argument("--image", help="image file")
+ src.add_argument("--video", help="video file")
+ src.add_argument("--camera", type=int, metavar="INDEX", help="live camera, e.g. 0")
+ p.add_argument("--model", default=DEFAULT_MODEL,
+ help=f"{', '.join(MODELS)}, or a path to an .onnx file (default: {DEFAULT_MODEL})")
+ p.add_argument("--device", default="cuda", choices=("cuda", "tensorrt", "cpu", "coreml"))
+ p.add_argument("--max-persons", type=int, default=4)
+ p.add_argument("--detector", default="medium", choices=("nano", "small", "medium", "base", "large"),
+ help="RF-DETR size (default: medium)")
+ p.add_argument("--det-confidence", type=float, default=0.5)
+ p.add_argument("--detector-stride", type=int, default=1, metavar="N",
+ help="run the detector every N frames and reuse its boxes in between")
+ p.add_argument("--focal", type=float, help="camera focal length in pixels, if known")
+ p.add_argument("--smooth", action="store_true", help="temporal smoothing (1 Euro filter)")
+ p.add_argument("--no-mesh", action="store_true", help="show the skeleton only")
+ p.add_argument("--mesh-alpha", type=float, default=0.35, help="mesh opacity (default: 0.35)")
+ p.add_argument("--save-rrd", metavar="PATH", help="also save the Rerun recording")
+ p.add_argument("--no-viewer", action="store_true", help="do not open the Rerun viewer")
return p.parse_args()
-def resolve_model_path(args: argparse.Namespace) -> Path:
- model_path = Path(args.model)
- if not model_path.exists():
- sys.stderr.write(
- f"\n[error] InstantHMR ONNX not found: {model_path}\n"
- " Place 'instanthmr.onnx' under models/, or pass\n"
- " --model /path/to/instanthmr.onnx\n\n"
- )
- sys.exit(1)
- return model_path
-
-
-def build_pipeline(args: argparse.Namespace, mhr=None) -> PosePipeline:
- return PosePipeline(
- onnx_path=resolve_model_path(args),
- device=args.device,
- detector_variant=args.detector_variant,
- det_confidence=args.det_confidence,
- max_persons=args.max_persons,
- detector_stride=args.detector_stride,
- batch_persons=not args.no_batch_persons,
- mhr=mhr,
- focal=args.focal,
- use_prev=not args.no_prev,
- prev_lag=args.prev_lag,
- )
-
-
-def _model_needs_mhr(model_path: Path) -> bool:
- """True when the graph has no ``joints_3d`` output (MHR-only student).
-
- Cheap protobuf peek, so the decision is made before any session or detector
- is loaded. An unreadable graph is treated as "does not need MHR" — the
- pipeline then raises a precise error of its own.
- """
- names = onnx_output_names(model_path)
- return names is not None and "joints_3d" not in names
-
-
-def build_mhr(args: argparse.Namespace, required: bool = False):
- """Build the MHR rig requested on the command line.
-
- ``--mhr-model`` (TorchScript) wins over ``--mhr-assets`` (pip package).
- When *required* — i.e. the ONNX has no 3D head — and neither flag was given,
- fall back to the TorchScript rig that shipped with the training code.
- """
- script_path = args.mhr_model
- assets_folder = args.mhr_assets
-
- if script_path is None and assets_folder is None:
- if not required:
- return None
- script_path = DEFAULT_MHR_SCRIPT
- print(
- f"[info] {Path(args.model).name} has no 'joints_3d' output "
- "(MHR-only student): its 70 keypoints are derived from the MHR "
- f"forward pass.\n Using {script_path}; override with "
- "--mhr-model / --mhr-assets."
- )
-
- if script_path is not None:
- path = Path(script_path)
- if not path.exists():
- sys.stderr.write(
- f"\n[error] MHR TorchScript rig not found: {path}\n"
- " This is the ~700 MB mhr_model.pt the distillation loads with\n"
- " torch.jit.load; see training/checkpoints/README.md.\n"
- " Alternatively use --mhr-assets with the unpacked MHR asset folder.\n\n"
- )
- sys.exit(1)
- from instanthmr.mhr_renderer import MHRScriptRenderer
-
- print(f"Loading MHR body model (TorchScript) from {path} …")
- return MHRScriptRenderer(path, device=args.device)
-
- assets_path = Path(assets_folder)
- if not assets_path.exists():
- sys.stderr.write(
- f"\n[error] MHR assets folder not found: {assets_path}\n"
- " See instanthmr/mhr_renderer.py for download instructions.\n\n"
- )
- sys.exit(1)
-
- try:
- from instanthmr.mhr_renderer import MHRRenderer
- except ImportError as e:
- sys.stderr.write(
- f"\n[error] Cannot import MHRRenderer: {e}\n"
- "\n"
- " MHR mesh rendering requires Python >= 3.12 and:\n"
- " pip install -r requirements-mhr.txt\n"
- "\n"
- " If you see 'No module named pymomentum', make sure you installed\n"
- " the CORRECT package (NOT the legacy pyMomentum SMS library):\n"
- " pip uninstall pymomentum # remove wrong package\n"
- " pip install pymomentum-gpu # NVIDIA GPU\n"
- " pip install pymomentum-cpu # macOS / CPU-only\n"
- "\n"
- " Or skip it entirely and use --mhr-model, which needs only torch.\n"
- "\n"
- " See README.md §'MHR body mesh' for full setup instructions.\n\n"
- )
- sys.exit(1)
-
- print(f"Loading MHR body model (LOD {args.mhr_lod}) from {assets_path} …")
- return MHRRenderer(
- assets_folder=assets_path,
- device=args.device,
- lod=args.mhr_lod,
- )
-
-
-def build_all(args: argparse.Namespace):
- """Build the MHR rig, the pipeline and the visualizer, in that order.
-
- The rig comes first because an MHR-only graph cannot produce 3D keypoints
- without it — the pipeline takes it as a constructor argument rather than
- the visualizer alone. It is returned as well so the temporal filter can
- re-derive keypoints from filtered pose parameters.
- """
- model_path = resolve_model_path(args)
- mhr = build_mhr(args, required=_model_needs_mhr(model_path))
- pipeline = build_pipeline(args, mhr=mhr)
- if pipeline.hmr.has_prev_input:
- print(f"Prev-token model: previous pose "
- f"{f'fed back from {args.prev_lag} frames earlier' if pipeline.feeds_prev else 'withheld (--no-prev)'}; "
- f"foot contact shown in red.", flush=True)
- viz = RerunVisualizer(
- application_id="instanthmr_demo",
- spawn_viewer=not args.no_spawn,
- save_path=args.save_rrd,
- mhr_renderer=mhr,
- mesh_alpha=args.mesh_alpha,
- reference=build_reference(args),
- contact_threshold=args.contact_threshold,
- )
- return pipeline, viz, mhr
-
-
-def build_reference(args: argparse.Namespace):
- """Load the saved teacher track for ``--show-reference``, or ``None``."""
- path = resolve_reference(args.show_reference, getattr(args, "video", None))
- if path is None:
- return None
- if not path.exists():
- sys.exit(
- f"[error] reference not found: {path}\n"
- " Build it by running the teacher once, then packing it:\n"
- " MOMENTUM_ENABLED=0 python main.py --video_path <clip> \\\n"
- " --output_path reference/<stem> --save_meshes\n"
- " python tools/pack_reference.py reference/<stem>/meshes "
- "reference/<stem>.npz"
- )
- track = ReferenceTrack(path)
- print(f"reference: {path} — {len(track)} frames, {track.num_persons} persons")
- return track
-
-
-def _print_timings(
- prefix: str,
- detector_ms: float,
- hmr_ms: float,
- render_ms: float,
- total_ms: float,
- n_persons: int,
-) -> None:
- fps = 1000.0 / total_ms if total_ms > 0 else 0.0
- render_part = f" MHR={render_ms:6.1f} ms" if render_ms > 0.0 else ""
- print(
- f"{prefix} "
- f"persons={n_persons:>2d} "
- f"RF-DETR={detector_ms:6.1f} ms "
- f"InstantHMR={hmr_ms:6.1f} ms"
- f"{render_part} "
- f"total={total_ms:6.1f} ms ({fps:5.1f} fps)"
- )
-
-
-# ---------------------------------------------------------------------------
-# Source-specific runners
-# ---------------------------------------------------------------------------
-
-
-def build_smoother(args: argparse.Namespace, live: bool = False, mhr=None):
- """Construct the temporal filter, or ``None`` when --smoothing-filter is off.
-
- ``live`` forces the adaptive filter: a camera cannot buffer future frames,
- and a trailing average would show visible phase lag on fast motion.
-
- Handing it the MHR backend lets it re-derive the 3D keypoints from the
- filtered pose parameters, so they stay welded to the mesh instead of
- drifting off it during fast motion.
- """
- if not args.smoothing_filter:
- return None
- mode = "1euro" if live else args.smoothing_mode
- kp_fn = mhr.keypoints if getattr(mhr, "has_keypoints", False) else None
- sm = TemporalSmoother(
- window=args.smoothing_window,
- mode=mode,
- beta_scale=args.smoothing_beta,
- mincutoff_scale=args.smoothing_mincutoff,
- keypoint_fn=kp_fn,
- )
- if kp_fn is None:
- print(" note: no MHR keypoint backend — keypoints are filtered on their "
- "own, so they may drift off the mesh during fast motion.")
- if sm.mode == "1euro":
- print(f"temporal smoothing: 1euro, beta x{sm.beta_scale:g}, "
- f"mincutoff x{sm.mincutoff_scale:g}, no latency")
- else:
- lag = f", {sm.lag} frame lag" if sm.lag else ""
- print(f"temporal smoothing: window={sm.window}, mode={sm.mode}{lag}")
- return sm
-
-
-def run_image(args: argparse.Namespace) -> None:
- image_path = Path(args.image)
- if not image_path.exists():
- sys.exit(f"[error] image not found: {image_path}")
-
- bgr = cv2.imread(str(image_path))
- if bgr is None:
- sys.exit(f"[error] cv2.imread failed for {image_path}")
- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
-
- pipeline, viz, _ = build_all(args)
-
- result = pipeline.predict(rgb)
-
- render_ms = viz.log_frame(
- rgb, result.persons,
- frame_idx=0, timestamp=0.0,
- detector_ms=result.detector_ms,
- hmr_ms=result.hmr_ms,
- total_ms=result.total_ms,
- )
- _print_timings(
- "image:",
- result.detector_ms, result.hmr_ms, render_ms, result.total_ms,
- len(result.persons),
- )
-
-
-def run_video(args: argparse.Namespace) -> None:
- video_path = Path(args.video)
- if not video_path.exists():
- sys.exit(f"[error] video not found: {video_path}")
-
- cap = cv2.VideoCapture(str(video_path))
+def frames(args):
+ """(index, timestamp in s, RGB frame) from the chosen source."""
+ if args.image:
+ bgr = cv2.imread(args.image)
+ if bgr is None:
+ sys.exit(f"cannot read image {args.image}")
+ yield 0, 0.0, cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
+ return
+ cap = cv2.VideoCapture(args.video if args.video else args.camera)
if not cap.isOpened():
- sys.exit(f"[error] cv2.VideoCapture failed for {video_path}")
-
+ sys.exit(f"cannot open {args.video or f'camera {args.camera}'}")
fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
- total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
- print(f"video: {video_path.name}, {total} frames @ {fps:.1f} fps")
-
- pipeline, viz, mhr = build_all(args)
- smoother = build_smoother(args, mhr=mhr)
-
- frame_idx = 0
- sent = 0
- det_times: list[float] = []
- hmr_times: list[float] = []
- render_times: list[float] = []
- tot_times: list[float] = []
-
- # When smoothing is centred the filter emits frame t only after it has seen
- # t + (window-1)//2, so the frames still in flight wait here with the data
- # the visualizer will need for them.
- pending: deque[tuple[int, np.ndarray, float, float, float]] = deque()
-
- def emit(persons) -> None:
- f_idx, f_rgb, d_ms, h_ms, t_ms = pending.popleft()
- render_times.append(viz.log_frame(
- f_rgb, persons,
- frame_idx=f_idx,
- timestamp=f_idx / fps,
- detector_ms=d_ms,
- hmr_ms=h_ms,
- total_ms=t_ms,
- ))
-
+ start, i = time.perf_counter(), 0
try:
while True:
- ret, bgr = cap.read()
- if not ret:
- break
-
- if frame_idx % args.frame_skip != 0:
- frame_idx += 1
- continue
-
- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
-
- result = pipeline.predict(rgb)
- det_times.append(result.detector_ms)
- hmr_times.append(result.hmr_ms)
- tot_times.append(result.total_ms)
-
- pending.append((frame_idx, rgb, result.detector_ms,
- result.hmr_ms, result.total_ms))
- if smoother is None:
- emit(result.persons)
- else:
- smoothed = smoother.push(result.persons, timestamp=frame_idx / fps)
- if smoothed is not None:
- emit(smoothed)
-
- sent += 1
- frame_idx += 1
-
- if sent % 30 == 0:
- _print_timings(
- f" frame {frame_idx:>5d}/{total:<5d}",
- float(np.mean(det_times[-30:])),
- float(np.mean(hmr_times[-30:])),
- float(np.mean(render_times[-30:])),
- float(np.mean(tot_times[-30:])),
- len(result.persons),
- )
- if smoother is not None:
- for smoothed in smoother.flush():
- emit(smoothed)
- except KeyboardInterrupt:
- print("\ninterrupted")
- finally:
- cap.release()
-
- if smoother is not None:
- report = smoother.jitter_report()
- if report:
- print("\n" + report)
-
- if tot_times:
- # Skip the first frame from the average — it bears the warm-up cost.
- warm = slice(1, None) if len(tot_times) > 1 else slice(None)
- render_avg = float(np.mean(render_times[warm])) if render_times else 0.0
- render_part = f" MHR={render_avg:.1f} ms" if render_avg > 0 else ""
- print(
- "\nsummary "
- f"frames={sent:d} "
- f"RF-DETR={float(np.mean(det_times[warm])):.1f} ms "
- f"InstantHMR={float(np.mean(hmr_times[warm])):.1f} ms"
- f"{render_part} "
- f"total={float(np.mean(tot_times[warm])):.1f} ms "
- f"({1000.0 / float(np.mean(tot_times[warm])):.1f} fps)"
- )
-
-
-def run_camera(args: argparse.Namespace) -> None:
- cap = cv2.VideoCapture(args.camera)
- if not cap.isOpened():
- sys.exit(f"[error] cannot open camera index {args.camera}")
-
- width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
- height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
- fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
- print(f"camera {args.camera}: {width}x{height} @ {fps:.1f} fps — Ctrl+C to stop")
-
- pipeline, viz, mhr = build_all(args)
- smoother = build_smoother(args, live=True, mhr=mhr)
-
- frame_idx = 0
- det_times: list[float] = []
- hmr_times: list[float] = []
- render_times: list[float] = []
- tot_times: list[float] = []
- try:
- while True:
- ret, bgr = cap.read()
- if not ret:
- print("[warn] camera read failed")
- break
-
- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
- result = pipeline.predict(rgb)
- det_times.append(result.detector_ms)
- hmr_times.append(result.hmr_ms)
- tot_times.append(result.total_ms)
-
- persons = result.persons
- if smoother is not None:
- # Wall clock, not frame_idx/fps: a webcam's real frame interval
- # drifts and drops, and the adaptive cutoff is a function of
- # speed, so it needs the interval that actually elapsed.
- persons = smoother.push(
- persons, timestamp=time.perf_counter()) or persons
-
- render_ms = viz.log_frame(
- rgb, persons,
- frame_idx=frame_idx,
- timestamp=frame_idx / fps,
- detector_ms=result.detector_ms,
- hmr_ms=result.hmr_ms,
- total_ms=result.total_ms,
- )
- render_times.append(render_ms)
-
- if frame_idx and frame_idx % 30 == 0:
- _print_timings(
- f" frame {frame_idx:>5d}",
- float(np.mean(det_times[-30:])),
- float(np.mean(hmr_times[-30:])),
- float(np.mean(render_times[-30:])),
- float(np.mean(tot_times[-30:])),
- len(result.persons),
- )
- frame_idx += 1
- except KeyboardInterrupt:
- print("\nstopped")
+ ok, bgr = cap.read()
+ if not ok:
+ return
+ t = i / fps if args.video else time.perf_counter() - start
+ yield i, t, cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
+ i += 1
finally:
cap.release()
def main() -> None:
args = parse_args()
- if args.image is not None:
- run_image(args)
- elif args.video is not None:
- run_video(args)
- else:
- run_camera(args)
+ print(f"loading InstantHMR ({args.model}) on {args.device} and RF-DETR {args.detector}...")
+ pipeline = PosePipeline(args.model, device=args.device, detector=args.detector,
+ det_confidence=args.det_confidence, max_persons=args.max_persons,
+ detector_stride=args.detector_stride, focal=args.focal)
+ print(f"InstantHMR runs on {pipeline.hmr.provider}")
+ viewer = RerunVisualizer(None if args.no_mesh else pipeline.hmr.faces, args.mesh_alpha,
+ spawn=not args.no_viewer, save_path=args.save_rrd)
+ smoother = TemporalSmoother() if args.smooth else None
+
+ history = []
+ try:
+ for i, t, rgb in frames(args):
+ result = pipeline(rgb)
+ persons = smoother(result.persons, t) if smoother else result.persons
+ timings = {"detector": result.detector_ms, "InstantHMR": result.hmr_ms}
+ viewer.log(i, rgb, persons, timings)
+ history.append((result.detector_ms, result.hmr_ms))
+ if i % 30 == 0:
+ print(f"frame {i:5d} {len(persons)} person(s) detector {result.detector_ms:5.1f} ms "
+ f"InstantHMR {result.hmr_ms:5.1f} ms")
+ except KeyboardInterrupt:
+ pass
+ if len(history) > 1:
+ det, hmr = np.median(np.array(history[1:]), axis=0)
+ print(f"median over {len(history) - 1} frames: detector {det:.1f} ms, InstantHMR {hmr:.1f} ms")
if __name__ == "__main__":
diff --git a/docs/annotation.md b/docs/annotation.md
deleted file mode 100644
index 60eae44..0000000
--- a/docs/annotation.md
+++ /dev/null
@@ -1,120 +0,0 @@
-# Building a training dataset by teacher annotation (optional path)
-
-> **This page describes the OPTIONAL distillation path.** The default — and what
-> every shipped model and published number in this repo actually used — is the
-> released ground-truth annotations of
-> [`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset),
-> laid out by `datasets_pipeline/build_split.py` into `data/sam3d_gt_<dataset>/`.
-> See `datasets_pipeline/README.md`.
->
-> Use this page when you need labels for imagery the released dataset does not
-> cover. The cost is a ceiling: a model trained on teacher output cannot exceed
-> the teacher, while one trained on the ground truth can.
-
-On this path InstantHMR is supervised by the **SAM3D teacher**
-(`facebook/sam-3d-body-dinov3`): for every detector-found person we run
-the teacher and store its outputs alongside a tight 224×224 crop. The
-training loop then learns to mimic those teacher outputs from the crop alone.
-This is the only thing that writes `data/sam3d_distill_mix/`.
-
-The bundled `tools/annotate_dataset.py` uses YOLO for the
-detector during annotation. That choice is independent of the runtime
-detector — InstantHMR sees the same 1.2× square crop either way, and the
-demo uses RF-DETR at inference time for better recall. If you re-run the
-annotation with a different detector, just preserve the crop geometry.
-
-The annotation pipeline is `tools/annotate_dataset.py`.
-
-## What it does
-
-```
-root_dir/ output_dir/
-├── COCO/ ─┐ ├── images/
-├── Harmony4D/ │ YOLO + SAM3D │ COCO_<stem>_p0.png (224×224)
-├── Synthetic/ │ ──────────────► │ COCO_<stem>_p0.npz
-└── … ─┘ │ …
- └── annotations/
- COCO_<stem>_p0.npz
- …
-```
-
-For each detected person (capped at `--max_persons` per image):
-
-1. **YOLO 11n** detects persons (`conf ≥ --confidence`, default 0.7).
-2. The **full-resolution** image is passed to SAM3D with the raw bbox.
- SAM3D's CLIFF camera head sees the full frame, so `pred_cam_t` and
- `focal_length` are produced in real perspective coordinates — no manual
- camera math.
-3. A 1.2× **square** crop around the bbox is resized to 224×224 and stored
- as a PNG.
-4. The teacher outputs are stored as a `.npz` next to the image:
-
- | Key | Shape | Description |
- |-----|-------|-------------|
- | `orig_shape` | `(2,)` | `[H, W]` of the original frame |
- | `bbox` | `(4,)` | raw YOLO `[x1, y1, x2, y2]` |
- | `bbox_square` | `(4,)` | the 1.2× expanded square crop bounds |
- | `cam_focal_length` | `(2,)` | `[fx, fy]` (≈ √(H²+W²)) |
- | `cam_trans` | `(3,)` | SAM3D translation in camera frame |
- | `mhr_model_params` | `(204,)` | MHR pose + scale |
- | `shape_params` | `(45,)` | MHR identity blendshapes |
- | `joints_3d` | `(70, 3)` | SAM3D `pred_keypoints_3d` |
- | `joints_2d` | `(70, 2)` | SAM3D `pred_keypoints_2d`, in **full-frame** pixels |
-
-The script is **resume-safe**: it scans existing `.npz` filenames at startup
-and skips any image whose first crop is already on disk.
-
-## Running it
-
-```bash
-# Setup: clone Meta's sam-3d-body and download the dinov3 teacher checkpoint.
-git clone https://github.com/facebookresearch/sam-3d-body
-huggingface-cli download facebook/sam-3d-body-dinov3 \
- --local-dir checkpoints/sam-3d-body-dinov3
-
-# Annotate a tree of images
-python tools/annotate_dataset.py \
- --root_dir /data/training_images \
- --output_dir /data/cliff_annotations \
- --confidence 0.7
-
-# Quick balanced subset (50k crops, shuffled across datasets)
-python tools/annotate_dataset.py \
- --root_dir /data/training_images \
- --output_dir /data/cliff_annotations \
- --shuffle --max_samples 50000
-
-# Visually inspect what's getting saved
-python tools/annotate_dataset.py \
- --root_dir /data/training_images \
- --output_dir /data/cliff_annotations \
- --use_rerun
-```
-
-Useful options:
-
-- `--shuffle` — process images in random order so a partial run still spans
- the full dataset diversity.
-- `--max_samples N` — stop after N **crops** are saved (not images), counting
- whatever was already on disk.
-- `--timeout 5.0` — per-image SIGALRM; rare GPU hangs don't kill the run.
-- `--gc_interval 200` — periodic CUDA cache + GC reset to keep VRAM stable
- across long runs.
-
-## Why a separate cloud script?
-
-This is the same per-person preprocessing the demo uses at inference time
-(`InstantHMR._preprocess`). Aligning the training data with the runtime
-preprocessing was the single biggest factor in getting InstantHMR to match
-SAM3D's accuracy: any drift between annotation crops and inference crops
-shows up directly as test-time error.
-
-## Datasets we used
-
-InstantHMR was trained on a mix of:
-
-- **COCO** (in-the-wild humans),
-- **Harmony4D** (multi-person studio captures),
-- a **synthetic dataset** of rendered humans, and
-- domain-specific footage (rugby, aikido) that the model is meant to handle.
-
diff --git a/install.py b/install.py
deleted file mode 100644
index 01f8e38..0000000
--- a/install.py
+++ /dev/null
@@ -1,159 +0,0 @@
-#!/usr/bin/env python3
-"""Platform-aware installer for InstantHMR.
-
-Picks the right Torch + ONNX Runtime wheels for the current machine:
-
- - macOS (Apple Silicon / Intel) — stock ``torch`` + ``onnxruntime``
- (which bundles the CoreML execution provider).
- - Linux + NVIDIA GPU — Torch built for the matching CUDA toolkit
- (cu128 for Blackwell sm_120+, cu124 for Ada / Ampere / Turing) plus
- ``onnxruntime-gpu`` and the ``nvidia-*-cu12`` runtime wheels so a
- system CUDA install isn't required.
- - Linux without an NVIDIA GPU — Torch CPU wheel + ``onnxruntime``.
-
-Run *after* you've created and activated a fresh Python environment::
-
- python install.py
-
-For a dry run that just prints the commands::
-
- python install.py --dry-run
-"""
-
-from __future__ import annotations
-
-import argparse
-import platform
-import subprocess
-import sys
-from typing import Optional
-
-
-def run_pip(args: list[str], dry_run: bool = False) -> None:
- cmd = [sys.executable, "-m", "pip", "install"] + args
- print(">>", " ".join(cmd))
- if not dry_run:
- subprocess.check_call(cmd)
-
-
-def detect_cuda_compute_cap() -> Optional[float]:
- """Return the highest CUDA compute capability across visible GPUs.
-
- Returns ``None`` if ``nvidia-smi`` is missing, fails, or reports no
- GPU.
- """
- try:
- out = subprocess.check_output(
- ["nvidia-smi", "--query-gpu=compute_cap", "--format=csv,noheader"],
- text=True, stderr=subprocess.DEVNULL,
- )
- except (FileNotFoundError, subprocess.CalledProcessError):
- return None
- caps: list[float] = []
- for line in out.splitlines():
- line = line.strip()
- if not line:
- continue
- try:
- caps.append(float(line))
- except ValueError:
- pass
- return max(caps) if caps else None
-
-
-def install_mac(dry_run: bool) -> None:
- print("[install] platform: macOS — using stock torch + onnxruntime (CoreML EP)")
- run_pip(["torch>=2.2", "torchvision"], dry_run)
- run_pip(["onnxruntime>=1.19"], dry_run)
-
-
-def install_linux_cpu(dry_run: bool) -> None:
- print("[install] platform: Linux without NVIDIA GPU — using CPU wheels")
- run_pip([
- "--index-url", "https://download.pytorch.org/whl/cpu",
- "torch>=2.2", "torchvision",
- ], dry_run)
- run_pip(["onnxruntime>=1.19"], dry_run)
-
-
-def install_linux_cuda(compute_cap: float, dry_run: bool) -> None:
- print(f"[install] platform: Linux + NVIDIA (compute capability {compute_cap})")
- if compute_cap >= 12.0:
- # Blackwell (RTX 50-series, B100, B200): needs sm_120 kernels which
- # only land natively in the cu128 wheels (Torch 2.7+).
- print("[install] → Blackwell detected, installing torch cu128 (>=2.7)")
- run_pip([
- "--index-url", "https://download.pytorch.org/whl/cu128",
- "torch>=2.7", "torchvision",
- ], dry_run)
- elif compute_cap >= 8.0:
- # Ampere (sm_80/86), Ada (sm_89), Hopper (sm_90). cu124 wheels cover
- # all of these with native kernels.
- print("[install] → Ampere/Ada/Hopper, installing torch cu124 (>=2.4)")
- run_pip([
- "--index-url", "https://download.pytorch.org/whl/cu124",
- "torch>=2.4", "torchvision",
- ], dry_run)
- else:
- # Turing (sm_75) and older — cu121 still has stable wheels here.
- print("[install] → pre-Ampere, installing torch cu121 (>=2.2)")
- run_pip([
- "--index-url", "https://download.pytorch.org/whl/cu121",
- "torch>=2.2", "torchvision",
- ], dry_run)
-
- run_pip(["onnxruntime-gpu>=1.19"], dry_run)
- # Bundled CUDA runtime libs so users without a system CUDA toolkit
- # can still load CUDA EP. ORT's preload_dlls() picks them up.
- run_pip([
- "nvidia-cudnn-cu12>=9.0,<10",
- "nvidia-cuda-runtime-cu12",
- "nvidia-cublas-cu12",
- ], dry_run)
-
-
-def install_common(dry_run: bool) -> None:
- print("[install] common deps")
- run_pip([
- "numpy>=1.24",
- "opencv-python>=4.8",
- "pillow>=10",
- "rfdetr>=1.0",
- "rerun-sdk>=0.21",
- ], dry_run)
-
-
-def main() -> None:
- parser = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- parser.add_argument("--dry-run", action="store_true",
- help="Print pip commands but don't run them.")
- parser.add_argument("--force-cpu", action="store_true",
- help="Force the CPU-only path on Linux even if an NVIDIA GPU is detected.")
- args = parser.parse_args()
-
- sysname = platform.system()
- machine = platform.machine()
- print(f"[install] python={platform.python_version()} system={sysname} machine={machine}")
-
- if sysname == "Darwin":
- install_mac(args.dry_run)
- elif sysname == "Linux":
- if args.force_cpu:
- install_linux_cpu(args.dry_run)
- else:
- cap = detect_cuda_compute_cap()
- if cap is None:
- install_linux_cpu(args.dry_run)
- else:
- install_linux_cuda(cap, args.dry_run)
- else:
- sys.exit(f"[install] unsupported platform: {sysname}. "
- f"Install torch + onnxruntime manually, then `pip install -r requirements.txt`.")
-
- install_common(args.dry_run)
- print("\n[install] done. verify with: python -c 'from instanthmr import PosePipeline; print(\"ok\")'")
-
-
-if __name__ == "__main__":
- main()
diff --git a/instanthmr/__init__.py b/instanthmr/__init__.py
index 1fce43e..4cb11df 100644
--- a/instanthmr/__init__.py
+++ b/instanthmr/__init__.py
@@ -1,21 +1,9 @@
-"""InstantHMR — standalone 3D human pose inference + Rerun visualization."""
+"""InstantHMR: real-time 3D human mesh recovery from a single image."""
+from .inference import HMRPrediction, InstantHMR
+from .models import DEFAULT_MODEL, MODELS, resolve_model
+from .pipeline import FrameResult, PosePipeline
+from .skeleton import JOINT_NAMES, SKELETON_EDGES
-from .inference import InstantHMR, HMRPrediction, onnx_output_names
-from .detector import RFDETRDetector
-from .pipeline import PosePipeline, FrameResult
-from .skeleton import JOINT_NAMES, SKELETON_EDGES, NUM_JOINTS, edges_for
-
-__all__ = [
- "InstantHMR",
- "HMRPrediction",
- "onnx_output_names",
- "RFDETRDetector",
- "PosePipeline",
- "FrameResult",
- "JOINT_NAMES",
- "SKELETON_EDGES",
- "NUM_JOINTS",
- "edges_for",
-]
-
-__version__ = "0.1.0"
+__all__ = ["InstantHMR", "HMRPrediction", "PosePipeline", "FrameResult", "MODELS", "DEFAULT_MODEL",
+ "resolve_model", "JOINT_NAMES", "SKELETON_EDGES"]
+__version__ = "1.0.0"
diff --git a/instanthmr/detector.py b/instanthmr/detector.py
index 7d16364..062a0cf 100644
--- a/instanthmr/detector.py
+++ b/instanthmr/detector.py
@@ -1,104 +1,37 @@
-"""RF-DETR person detector.
-
-Wraps Roboflow's RF-DETR (medium variant by default) with a minimal API:
-``detect(image_rgb)`` returns a list of
-``{"bbox": [x1, y1, x2, y2], "confidence": float}`` dicts sorted by
-descending confidence.
-
-RF-DETR is a transformer-based detector with strong COCO recall on small
-and partially-occluded persons — a good match for upstream pose
-estimation, which is sensitive to bbox localisation.
-"""
-
+"""RF-DETR person detector."""
from __future__ import annotations
-import numpy as np
-
-
-COCO_PERSON_CLASS_ID = 1 # RF-DETR exposes COCO 1-indexed class ids
+import logging
+import warnings
+import numpy as np
-class RFDETRDetector:
- """Person detector backed by RF-DETR.
+VARIANTS = {"nano": "RFDETRNano", "small": "RFDETRSmall", "medium": "RFDETRMedium",
+ "base": "RFDETRBase", "large": "RFDETRLarge"}
+COCO_PERSON = 1
- Args:
- variant: Which RF-DETR variant to load. One of
- ``"nano"``, ``"small"``, ``"medium"``, ``"base"``, ``"large"``.
- Default ``"medium"`` balances speed and accuracy.
- confidence: Minimum detection score.
- max_persons: Maximum number of persons returned per frame.
- optimize_for_inference: Call RF-DETR's TorchScript / fused-op
- optimisation pass on construction. Adds a few seconds of
- startup but speeds up subsequent ``detect`` calls noticeably.
- """
- _VARIANTS = {
- "nano": "RFDETRNano",
- "small": "RFDETRSmall",
- "medium": "RFDETRMedium",
- "base": "RFDETRBase",
- "large": "RFDETRLarge",
- }
+class PersonDetector:
+ """Returns `(boxes (N, 4) x1 y1 x2 y2, scores (N,))` for the people in an RGB frame, best first."""
- def __init__(
- self,
- variant: str = "medium",
- confidence: float = 0.5,
- max_persons: int = 5,
- optimize_for_inference: bool = True,
- ):
+ def __init__(self, variant: str = "medium", confidence: float = 0.5, max_persons: int = 4):
import rfdetr
- if variant not in self._VARIANTS:
- raise ValueError(
- f"Unknown RF-DETR variant {variant!r}. "
- f"Choose one of {list(self._VARIANTS)}."
- )
-
- cls = getattr(rfdetr, self._VARIANTS[variant])
- self._model = cls()
- if optimize_for_inference:
- self._model.optimize_for_inference()
-
- self._confidence = confidence
- self._max_persons = max_persons
- self._variant = variant
-
- def warmup(self) -> None:
- """Run one silent forward pass to pre-compile PyTorch/CUDA kernels."""
- dummy = np.zeros((224, 224, 3), dtype=np.uint8)
- self.detect(dummy)
-
- @property
- def variant(self) -> str:
- return self._variant
-
- def detect(self, image_rgb: np.ndarray) -> list[dict]:
- """Detect persons in an RGB image.
-
- Args:
- image_rgb: ``(H, W, 3)`` uint8 RGB array.
-
- Returns:
- List of detection dicts sorted by descending confidence,
- truncated to ``max_persons``.
- """
- from PIL import Image as PILImage
-
- pil_image = PILImage.fromarray(image_rgb)
- sv_detections = self._model.predict(pil_image, threshold=self._confidence)
-
- detections: list[dict] = []
- if len(sv_detections) > 0 and sv_detections.class_id is not None:
- mask = sv_detections.class_id == COCO_PERSON_CLASS_ID
- for bbox, conf in zip(
- sv_detections.xyxy[mask],
- sv_detections.confidence[mask],
- ):
- detections.append({
- "bbox": np.asarray(bbox, dtype=np.float32),
- "confidence": float(conf),
- })
-
- detections.sort(key=lambda d: d["confidence"], reverse=True)
- return detections[: self._max_persons]
+ logging.getLogger("rf-detr").setLevel(logging.ERROR)
+ if variant not in VARIANTS:
+ raise ValueError(f"unknown RF-DETR variant {variant!r}: choose one of {', '.join(VARIANTS)}")
+ self.model = getattr(rfdetr, VARIANTS[variant])()
+ if hasattr(self.model, "optimize_for_inference"): # rfdetr < 1.10
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore")
+ self.model.optimize_for_inference()
+ self.confidence, self.max_persons = confidence, max_persons
+
+ def __call__(self, image_rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ from PIL import Image
+
+ det = self.model.predict(Image.fromarray(image_rgb), threshold=self.confidence)
+ keep = det.class_id == COCO_PERSON if det.class_id is not None else np.zeros(len(det), bool)
+ boxes, scores = det.xyxy[keep].astype(np.float32), det.confidence[keep].astype(np.float32)
+ order = np.argsort(-scores)[: self.max_persons]
+ return boxes[order], scores[order]
diff --git a/instanthmr/inference.py b/instanthmr/inference.py
index 493cea4..673096f 100644
--- a/instanthmr/inference.py
+++ b/instanthmr/inference.py
@@ -1,581 +1,183 @@
-"""ONNX inference for InstantHMR.
-
-InstantHMR takes a square person crop (224 px unless the ONNX metadata
-says otherwise) and a 3-vector CLIFF condition
-(bbox center / scale in full-frame coords) and returns mhr_params, shape,
-camera translation, and 70 joints in 2D (crop space) and 3D (camera coords,
-rig-local, metres, Y-down).
-
-This module wraps the ONNX session and the per-person preprocessing — square
-crop, CLIFF cond, ImageNet normalisation, and re-projection of joints back
-into full-frame pixels.
-
-Two graph layouts are supported, and the session tells them apart by itself:
-
- * **5 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d, joints_3d)``
- — older students with their own 3D coordinate head (earlier trainers,
- not in this repository).
- * **4 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d)`` — the
- MHR-only student (``train.py``) has no 3D head, exactly
- like the SAM3D teacher. Its 70 keypoints are derived here by running the
- MHR skeleton forward on the predicted parameters, so such a model needs an
- MHR backend (see :mod:`instanthmr.mhr_renderer`) passed as ``mhr=``.
-"""
-
+"""InstantHMR inference with onnxruntime: person crops in, MHR pose, 3D keypoints and mesh out."""
from __future__ import annotations
+import base64
import math
from dataclasses import dataclass
from pathlib import Path
-from typing import Optional
+from typing import Sequence
import cv2
import numpy as np
-
IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
-INPUT_SIZE = 224
-CROP_EXPAND = 1.2 # square crop around the detector bbox, matching training
-# Focal used when the camera is uncalibrated, as a multiple of the image
-# diagonal. A focal-aware checkpoint needs a number here whatever happens --
-# the conditioning is an ANGLE, so there is no "unknown" to pass. 1.05x diag
-# (~50 degrees horizontal FOV on 16:9) is the usual stand-in for a phone or
-# webcam; pass the real value through `focal=` whenever it is known.
-#
-# It applies to focal-aware graphs ONLY. A pixel-conditioned checkpoint was
-# trained on a corpus whose annotations set f = sqrt(H^2+W^2) exactly (see
-# tools/annotate_dataset.py), so 1.0x diag IS that model's camera and using
-# anything else would put a 5% error into its rendered depth.
+CROP_EXPAND = 1.2
+# Focal used when the camera is not calibrated, as a multiple of the image diagonal
+# (about 50 degrees of horizontal field of view on a 16:9 frame).
FOCAL_FALLBACK_DIAG = 1.05
-
-
-def onnx_output_names(onnx_path: str | Path) -> Optional[list[str]]:
- """Read a graph's output names without creating an inference session.
-
- Lets a caller know whether it must supply an MHR backend *before* paying for
- session creation and detector loading. Returns ``None`` when the ``onnx``
- package is unavailable or the file cannot be parsed — callers should then
- fall back to letting :class:`InstantHMR` decide.
- """
- try:
- import onnx
-
- model = onnx.load(str(onnx_path), load_external_data=False)
- return [o.name for o in model.graph.output]
- except Exception: # noqa: BLE001 — purely advisory
- return None
+DEVICES = ("cuda", "tensorrt", "cpu", "coreml")
@dataclass
class HMRPrediction:
- """Per-person outputs from one InstantHMR forward pass.
-
- All numpy, all in the FULL-frame coordinate system.
-
- Attributes:
- bbox: (4,) raw detector bbox [x1, y1, x2, y2].
- confidence: detection confidence in [0, 1].
- joints_3d_local: (70, 3) rig-local joints, metres, Y-down. Either read
- straight from the model's 3D head, or regressed from the MHR
- skeleton when the graph has no such head.
- joints_3d_cam: (70, 3) joints in camera space (= local + cam_trans).
- joints_2d: (70, 2) joints projected into full-frame pixel coords.
- cam_trans: (3,) camera translation [tx, ty, tz], metres.
- focal_length: (2,) [fx, fy] full-frame focal in pixels -- the camera
- this prediction is expressed in. The real focal when one was
- supplied, else the diagonal stand-in ``_focal_px`` picks.
- principal_point: (2,) [cx, cy] = full-frame centre.
- image_shape: (H, W) of the source frame.
- mhr_params: (204,) MHR pose parameters (34 joints × 6-D rotation).
- Pass to ``instanthmr.mhr_renderer.MHRRenderer.forward()``
- together with ``shape_params`` to obtain a full body mesh.
- shape_params: (45,) MHR identity blend-shape coefficients
- (20 body + 20 head + 5 hand). Pair with ``mhr_params`` and
- feed both into ``MHRRenderer.forward()``.
- """
+ """One person. Pixel coordinates are in the full frame; 3D is in metres, camera axes
+ (x right, y down, z forward), with `*_local` centred on the body and `*_cam` in camera space."""
- bbox: np.ndarray
+ bbox: np.ndarray # (4,) x1, y1, x2, y2
confidence: float
- joints_3d_local: np.ndarray
- joints_3d_cam: np.ndarray
- joints_2d: np.ndarray
- cam_trans: np.ndarray
- focal_length: np.ndarray
- principal_point: np.ndarray
- image_shape: tuple[int, int]
- mhr_params: np.ndarray
- shape_params: np.ndarray
- contact_prob: Optional[np.ndarray] = None # (2,) [left, right] foot, prev-token graphs only
+ joints_2d: np.ndarray # (70, 2) pixels
+ joints_3d_local: np.ndarray # (70, 3)
+ cam_trans: np.ndarray # (3,)
+ mhr_params: np.ndarray # (204,) MHR model parameters
+ shape_params: np.ndarray # (45,) MHR identity coefficients
+ focal_length: float # pixels, the camera the 3D is expressed in
+ image_shape: tuple[int, int] # (H, W)
+ vertices_local: np.ndarray # (18439, 3)
- def prev_pose(self) -> np.ndarray:
- """This prediction encoded as a prev-token graph's ``prev_pose`` input:
- the 70 rig-local keypoints, then ``cam_trans`` as (x/z, y/z, log z)."""
- t = self.cam_trans
- z = max(float(t[2]), 0.1)
- return np.concatenate([self.joints_3d_local.reshape(-1),
- [t[0] / z, t[1] / z, math.log(z)]]).astype(np.float32)
+ @property
+ def joints_3d_cam(self) -> np.ndarray:
+ return self.joints_3d_local + self.cam_trans
+
+ @property
+ def vertices_cam(self) -> np.ndarray:
+ return self.vertices_local + self.cam_trans
+
+
+def providers_for(device: str, max_batch: int = 1, image_size: int = 288,
+ cache_dir: str | Path = "models/trt_cache") -> list:
+ """onnxruntime execution providers for a device name, always ending on the CPU."""
+ import onnxruntime as ort
+
+ available = set(ort.get_available_providers())
+ if device == "tensorrt":
+ try:
+ import tensorrt_libs # noqa: F401 (loads libnvinfer for onnxruntime)
+ except ImportError as e:
+ raise RuntimeError("TensorRT is not installed: pip install tensorrt-cu12==10.15.*") from e
+ if "TensorrtExecutionProvider" not in available:
+ raise RuntimeError("this onnxruntime build has no TensorRT support: install onnxruntime-gpu")
+ Path(cache_dir).mkdir(parents=True, exist_ok=True)
+ shape = lambda n: f"image:{n}x3x{image_size}x{image_size},cliff_cond:{n}x3" # noqa: E731
+ trt = {"trt_fp16_enable": True, "trt_engine_cache_enable": True, "trt_engine_cache_path": str(cache_dir),
+ "trt_timing_cache_enable": True, "trt_profile_min_shapes": shape(1),
+ "trt_profile_opt_shapes": shape(max_batch), "trt_profile_max_shapes": shape(max_batch)}
+ return [("TensorrtExecutionProvider", trt), "CUDAExecutionProvider", "CPUExecutionProvider"]
+ if device == "cuda":
+ if "CUDAExecutionProvider" not in available:
+ raise RuntimeError("CUDA is not available in this onnxruntime build: install onnxruntime-gpu")
+ return ["CUDAExecutionProvider", "CPUExecutionProvider"]
+ if device == "coreml" and "CoreMLExecutionProvider" in available:
+ return ["CoreMLExecutionProvider", "CPUExecutionProvider"]
+ return ["CPUExecutionProvider"]
class InstantHMR:
- """ONNX wrapper for the InstantHMR model.
-
- Args:
- onnx_path: path to the exported graph.
- device: ``"cuda"``, ``"coreml"`` or ``"cpu"``.
- providers: explicit onnxruntime execution providers (overrides *device*).
- mhr: MHR backend used to derive ``joints_3d`` when the graph has no 3D
- head. Any object exposing ``keypoints(mhr_params, shape_params)``
- works; see :func:`instanthmr.mhr_renderer.build_mhr`. Ignored for
- 5-output graphs.
- focal: the camera's focal length in pixels, when it is known. Used by
- the focal-aware conditioning and reported back in
- ``HMRPrediction.focal_length``. Left unset, an uncalibrated
- stand-in of ``FOCAL_FALLBACK_DIAG * sqrt(H^2+W^2)`` is used.
- cliff_focal: which conditioning form the checkpoint was trained with.
- Read from the ONNX metadata that ``tools/pth_to_onnx.py`` stamps,
- so it normally needs no setting; pass it explicitly for a graph
- exported before that existed.
+ """An exported InstantHMR network and the MHR body graph that turns its output into 3D.
- Example
- -------
- hmr = InstantHMR("models/instanthmr.onnx")
- out = hmr.predict(image_rgb, bbox=[x1, y1, x2, y2])
- print(out.joints_3d_cam.shape) # (70, 3)
+ `device` runs the network ("cuda", "tensorrt", "cpu" or "coreml"); the body graph runs on
+ CUDA when the network does, else on the CPU. `focal` is the camera focal length in pixels
+ when it is known; otherwise 1.05 x the image diagonal is assumed. `max_batch` is the most
+ people predicted in one call (TensorRT builds its engine for batches up to it).
"""
- def __init__(
- self,
- onnx_path: str | Path,
- device: str = "cuda",
- providers: Optional[list[str]] = None,
- mhr: object | None = None,
- focal: float | None = None,
- cliff_focal: bool | None = None,
- ):
+ def __init__(self, model_path: str | Path, body_path: str | Path | None = None, device: str = "cuda",
+ focal: float | None = None, max_batch: int = 4):
import onnxruntime as ort
- onnx_path = Path(onnx_path)
- if not onnx_path.exists():
- raise FileNotFoundError(f"ONNX model not found: {onnx_path}")
+ from .models import resolve_body
- # ORT >= 1.19 ships preload_dlls() which discovers CUDA / cuDNN
- # bundled by the nvidia-* pip wheels (or by torch) and adds them to
- # the loader's search path. Calling it before session creation is
- # the simplest fix for "libcudnn.so.9: cannot open shared object
- # file" on machines that don't ship system CUDA.
+ if device not in DEVICES:
+ raise ValueError(f"device must be one of {DEVICES}")
if hasattr(ort, "preload_dlls"):
- try:
- ort.preload_dlls()
- except Exception:
- # preload_dlls is best-effort — never let it block init.
- pass
-
- if providers is None:
- providers = self._default_providers(device)
-
- sess_options = ort.SessionOptions()
- sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
- sess_options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
- # 0 == let ORT pick. We don't override threading on CUDA (compute
- # is on-device); on CPU/CoreML, ORT's defaults are saner than ours.
-
- self.session = ort.InferenceSession(
- str(onnx_path), sess_options=sess_options, providers=providers,
- )
- self.active_provider = self.session.get_providers()[0]
-
- in_names = [i.name for i in self.session.get_inputs()]
- out_names = [o.name for o in self.session.get_outputs()]
- self._in_image = in_names[0] # "image"
- self._in_cliff = in_names[1] # "cliff_cond"
- # Output order from export: mhr_params, shape_params, cam_trans,
- # joints_2d[, joints_3d]. The MHR-only student stops after joints_2d.
- self._out_names = out_names
- # A --prev-token graph (tools/pth_to_onnx.py) also takes prev_pose and
- # has_prev, and returns contact_logits instead of a joints_3d head.
- self.has_prev_input = "prev_pose" in in_names
- self._prev_dim = self.session.get_inputs()[2].shape[1] if self.has_prev_input else 0
- self.has_joints_3d_head = "joints_3d" in out_names or (
- len(out_names) >= 5 and "contact_logits" not in out_names)
- self._mhr = mhr
-
- # Two independent things, and conflating them is what made this wrong:
- #
- # cliff_focal -- WHICH conditioning form the checkpoint was trained
- # with. Angles off the optical axis, or image-normalised pixels.
- # The vector is 3 floats either way, so feeding a model the wrong
- # one raises nothing; it silently mis-places the person in depth.
- # tools/pth_to_onnx.py stamps it into the graph metadata, so a
- # freshly exported model configures itself.
- # focal -- WHAT the camera's focal length is, in pixels. Only used by
- # the angular form. Unknown is not an option there, so an
- # uncalibrated camera falls back to FOCAL_FALLBACK_DIAG * diagonal.
- #
- # Passing `focal=` alone still implies the angular form, which is how
- # this argument behaved before `cliff_focal` existed.
- meta = self.session.get_modelmeta().custom_metadata_map or {}
- # The crop side the graph was trained at: stamped by tools/pth_to_onnx.py,
- # else the graph's static input height, else the historical 224.
- dim = self.session.get_inputs()[0].shape[2]
- self.input_size = int(meta.get("image_size")
- or (dim if isinstance(dim, int) else INPUT_SIZE))
- if cliff_focal is None:
- cliff_focal = (str(meta.get("cliff_focal", "")).lower() == "true"
- or focal is not None)
- self.cliff_focal = bool(cliff_focal)
+ ort.preload_dlls()
+ ort.set_default_logger_severity(3)
+ options = ort.SessionOptions()
+ options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
+ options.log_severity_level = 3
+
+ size = 288
+ if device == "tensorrt": # the engine profile needs the crop size before the session exists
+ probe = ort.InferenceSession(str(model_path), providers=["CPUExecutionProvider"])
+ size = int(probe.get_modelmeta().custom_metadata_map.get("image_size", size))
+ del probe
+ self.network = ort.InferenceSession(str(model_path), sess_options=options,
+ providers=providers_for(device, max_batch, size))
+ self.provider = self.network.get_providers()[0]
+ meta = self.network.get_modelmeta().custom_metadata_map
+ self.input_size = int(meta.get("image_size", size))
+ self.cliff_focal = meta.get("cliff_focal", "true") == "true"
self.focal = focal
+ self.max_batch = max_batch
- def _focal_px(self, h: int, w: int, focal: float | None = None) -> float:
- """The camera focal in pixels: this call's, the session's, or a
- stand-in -- in that order."""
+ body_path = Path(body_path) if body_path else resolve_body(Path(model_path))
+ self.body = ort.InferenceSession(str(body_path), sess_options=options,
+ providers=providers_for("cuda" if device in ("cuda", "tensorrt") else "cpu"))
+ self.faces = np.frombuffer(base64.b64decode(self.body.get_modelmeta().custom_metadata_map["mesh_faces"]),
+ "<i4").reshape(-1, 3)
+
+ def focal_px(self, h: int, w: int, focal: float | None = None) -> float:
f = focal if focal is not None else self.focal
if f is not None:
return float(f)
- diag = math.sqrt(h * h + w * w)
- return (FOCAL_FALLBACK_DIAG * diag) if self.cliff_focal else diag
-
- def _feeds(self, crops: np.ndarray, cliffs: np.ndarray,
- prev: Optional[list] = None) -> dict:
- """Session inputs; for a prev-token graph, ``prev[i]`` is person i's
- ``HMRPrediction.prev_pose()`` from an earlier frame, or None = withheld."""
- feeds = {self._in_image: crops, self._in_cliff: cliffs}
- if self.has_prev_input:
- n = crops.shape[0]
- pp = np.zeros((n, self._prev_dim), dtype=np.float32)
- hp = np.zeros((n,), dtype=np.float32)
- for i, v in enumerate((prev or [])[:n]):
- if v is not None:
- pp[i], hp[i] = v, 1.0
- feeds["prev_pose"], feeds["has_prev"] = pp, hp
- return feeds
-
- def _contact(self, outs: list, i: int) -> Optional[np.ndarray]:
- if "contact_logits" not in self._out_names:
- return None
- z = outs[self._out_names.index("contact_logits")][i].astype(np.float32)
- return 1.0 / (1.0 + np.exp(-z))
-
- @property
- def derives_joints_3d(self) -> bool:
- """True when the 70 keypoints must come from an MHR forward pass."""
- return not self.has_joints_3d_head
-
- def set_mhr(self, mhr: object | None) -> None:
- """Attach (or replace) the MHR backend used to derive ``joints_3d``."""
- self._mhr = mhr
-
- def _joints_3d(self, mhr_params: np.ndarray, shape_params: np.ndarray) -> np.ndarray:
- """Regress the 70 keypoints from predicted MHR parameters.
-
- Mirrors ``train.MHRForwardPass.get_native_keypoints``:
- skeleton forward, cm -> m, Y/Z flip, then the (70, 127) affine map. The
- student was trained through exactly this operator, so any other
- keypoint definition would silently shift the predictions.
- """
- if self._mhr is None:
- raise RuntimeError(
- "this ONNX graph has no 'joints_3d' output — it comes from "
- "train.py, where the 3D keypoints are derived from "
- "the MHR forward pass.\n"
- "Pass an MHR backend, e.g.\n"
- " from instanthmr.mhr_renderer import build_mhr\n"
- " InstantHMR(onnx, mhr=build_mhr(script_path='checkpoints/mhr_model.pt'))\n"
- "or run demo.py with --mhr-model / --mhr-assets."
- )
- return np.asarray(
- self._mhr.keypoints(mhr_params, shape_params), dtype=np.float32,
- )
-
- # ------------------------------------------------------------------
- # Public API
- # ------------------------------------------------------------------
-
- def predict(
- self,
- image_rgb: np.ndarray,
- bbox: np.ndarray | list[float],
- confidence: float = 1.0,
- focal: float | None = None,
- prev: Optional[np.ndarray] = None,
- ) -> HMRPrediction:
- """Run InstantHMR on a single person.
-
- Args:
- image_rgb: (H, W, 3) uint8 RGB full-frame image.
- bbox: tight person bbox [x1, y1, x2, y2] in pixel coords.
- confidence: detection confidence to attach to the output.
- focal: this frame's focal length in pixels, overriding the one
- given to the constructor. Use it when the focal varies per
- image, as it does across calibrated benchmark sequences.
- prev: prev-token graphs only, this person's earlier
- ``HMRPrediction.prev_pose()``; None withholds it.
-
- Returns:
- ``HMRPrediction`` with joints in full-frame coordinates.
- """
- image_rgb = np.ascontiguousarray(image_rgb)
- h, w = image_rgb.shape[:2]
- bbox_arr = np.asarray(bbox, dtype=np.float32).reshape(4)
+ return (FOCAL_FALLBACK_DIAG if self.cliff_focal else 1.0) * math.hypot(h, w)
- crop, sq_x1, sq_y1, sq_size, cliff_cond = self._preprocess(
- image_rgb, bbox_arr, h, w, focal
- )
-
- outs = self.session.run(
- None,
- # (1, 3, S, S) with S = self.input_size, and (1, 3)
- self._feeds(crop[np.newaxis], cliff_cond[np.newaxis], [prev]),
- )
- # Order: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]
- mhr_params = outs[0][0].astype(np.float32)
- shape_params = outs[1][0].astype(np.float32)
- cam_trans = outs[2][0].astype(np.float32)
- joints_2d_norm = outs[3][0].astype(np.float32) # (70, 2) in [-1, 1]
- if self.has_joints_3d_head:
- joints_3d_local = outs[4][0].astype(np.float32) # (70, 3) rig-local
- else:
- joints_3d_local = self._joints_3d(mhr_params, shape_params)
-
- # Re-project the 2D head from normalised crop space → full-frame pixels.
- crop_px = (joints_2d_norm + 1.0) * 0.5 * self.input_size
- scale = sq_size / self.input_size
- joints_2d = np.stack(
- [crop_px[:, 0] * scale + sq_x1, crop_px[:, 1] * scale + sq_y1],
- axis=-1,
- ).astype(np.float32)
-
- joints_3d_cam = joints_3d_local + cam_trans
-
- # Report the focal actually used, so anything that unprojects or
- # renders these joints uses the same camera the model was conditioned on.
- f = self._focal_px(h, w, focal)
- focal_length = np.array([f, f], dtype=np.float32)
- principal_point = np.array([w / 2.0, h / 2.0], dtype=np.float32)
-
- return HMRPrediction(
- bbox=bbox_arr,
- confidence=float(confidence),
- joints_3d_local=joints_3d_local,
- joints_3d_cam=joints_3d_cam,
- joints_2d=joints_2d,
- cam_trans=cam_trans,
- focal_length=focal_length,
- principal_point=principal_point,
- image_shape=(h, w),
- mhr_params=mhr_params,
- shape_params=shape_params,
- contact_prob=self._contact(outs, 0),
- )
-
- def predict_batch(
- self,
- image_rgb: np.ndarray,
- detections: list[dict],
- padded_to: int | None = None,
- focal: float | None = None,
- prev: Optional[list] = None,
- ) -> list[HMRPrediction]:
- """Run InstantHMR on multiple persons in a single ONNX call.
-
- Args:
- image_rgb: (H, W, 3) uint8 RGB full-frame image.
- detections: list of ``{"bbox": [x1,y1,x2,y2], "confidence": f}``.
- padded_to: when set, the ONNX batch dimension is always this size
- (zero-padded if fewer detections). Keeping a constant batch
- size prevents CUDA from re-compiling kernels each time the
- person count changes.
- focal: this frame's focal length in pixels, overriding the one
- given to the constructor.
- prev: prev-token graphs only, one earlier
- ``HMRPrediction.prev_pose()`` (or None) per detection.
-
- Returns:
- One ``HMRPrediction`` per input detection, in the same order.
- """
- if not detections:
+ def predict(self, image_rgb: np.ndarray, bboxes: Sequence, confidences: Sequence[float] | None = None,
+ focal: float | None = None) -> list[HMRPrediction]:
+ """All people of one frame in a single call. `bboxes` are tight person boxes, x1 y1 x2 y2."""
+ if len(bboxes) == 0:
return []
-
- image_rgb = np.ascontiguousarray(image_rgb)
h, w = image_rgb.shape[:2]
- n = len(detections)
- batch = max(padded_to or n, n) # always >= n; zero-pad the rest
-
- # Use zeros so padded slots produce deterministic (discarded) outputs.
- crops = np.zeros((batch, 3, self.input_size, self.input_size), dtype=np.float32)
- cliffs = np.zeros((batch, 3), dtype=np.float32)
- sq_meta = [] # per-person (sq_x1, sq_y1, sq_size, bbox)
- for i, det in enumerate(detections):
- bbox_arr = np.asarray(det["bbox"], dtype=np.float32).reshape(4)
- crop, sq_x1, sq_y1, sq_size, cliff = self._preprocess(
- image_rgb, bbox_arr, h, w, focal,
- )
- crops[i] = crop
- cliffs[i] = cliff
- sq_meta.append((sq_x1, sq_y1, sq_size, bbox_arr))
-
- outs = self.session.run(None, self._feeds(crops, cliffs, prev))
- mhr_params_b = outs[0].astype(np.float32, copy=False)
- shape_params_b = outs[1].astype(np.float32, copy=False)
- cam_trans_b = outs[2].astype(np.float32, copy=False)
- joints_2d_norm_b = outs[3].astype(np.float32, copy=False)
- if self.has_joints_3d_head:
- joints_3d_local_b = outs[4].astype(np.float32, copy=False)
- else:
- # One batched MHR skeleton forward for the whole frame (~0.9 ms),
- # including the zero-padded slots — they are discarded below.
- joints_3d_local_b = self._joints_3d(mhr_params_b, shape_params_b)
-
- f = self._focal_px(h, w, focal)
- focal_length = np.array([f, f], dtype=np.float32)
- principal_point = np.array([w / 2.0, h / 2.0], dtype=np.float32)
-
- results: list[HMRPrediction] = []
- for i, det in enumerate(detections):
- sq_x1, sq_y1, sq_size, bbox_arr = sq_meta[i]
- joints_2d_norm = joints_2d_norm_b[i]
- joints_3d_local = joints_3d_local_b[i]
- cam_trans = cam_trans_b[i]
-
- crop_px = (joints_2d_norm + 1.0) * 0.5 * self.input_size
- scale = sq_size / self.input_size
- joints_2d = np.stack(
- [crop_px[:, 0] * scale + sq_x1, crop_px[:, 1] * scale + sq_y1],
- axis=-1,
- ).astype(np.float32)
-
+ f = self.focal_px(h, w, focal)
+ crops, cliffs, squares = zip(*(self._preprocess(image_rgb, np.asarray(b, np.float32), f) for b in bboxes))
+ crops, cliffs = np.stack(crops), np.stack(cliffs)
+ chunks = [self._run(crops[i:i + self.max_batch], cliffs[i:i + self.max_batch])
+ for i in range(0, len(crops), self.max_batch)]
+ outs = {k: np.concatenate([c[k] for c in chunks]) for k in chunks[0]}
+ confidences = confidences if confidences is not None else [1.0] * len(bboxes)
+
+ results = []
+ for i, (x0, y0, side) in enumerate(squares):
+ j2d = (outs["joints_2d"][i] + 1.0) * 0.5 * side + np.array([x0, y0], np.float32)
results.append(HMRPrediction(
- bbox=bbox_arr,
- confidence=float(det.get("confidence", 1.0)),
- joints_3d_local=joints_3d_local,
- joints_3d_cam=joints_3d_local + cam_trans,
- joints_2d=joints_2d,
- cam_trans=cam_trans,
- focal_length=focal_length,
- principal_point=principal_point,
- image_shape=(h, w),
- mhr_params=mhr_params_b[i],
- shape_params=shape_params_b[i],
- contact_prob=self._contact(outs, i),
- ))
+ bbox=np.asarray(bboxes[i], np.float32), confidence=float(confidences[i]),
+ joints_2d=j2d.astype(np.float32), joints_3d_local=outs["joints_3d"][i],
+ cam_trans=outs["cam_trans"][i], mhr_params=outs["mhr_params"][i],
+ shape_params=outs["shape_params"][i], focal_length=f, image_shape=(h, w),
+ vertices_local=outs["vertices"][i]))
return results
- def warmup(self, max_batch_size: int = 1) -> None:
- """Pre-warm the ONNX session for the two batch sizes used at runtime.
-
- With the padded-batch strategy the pipeline only ever calls the session
- with batch=1 (single person) or batch=max_batch_size (multi-person,
- zero-padded to a fixed size). Two runs here compile all CUDA kernels
- upfront so no mid-demo stall occurs when persons first appear.
- """
- dummy_image = np.zeros(
- (max_batch_size, 3, self.input_size, self.input_size), dtype=np.float32
- )
- dummy_cliff = np.zeros((max_batch_size, 3), dtype=np.float32)
- # Large batch first — allocates the most memory; batch=1 reuses the plan.
- self.session.run(None, self._feeds(dummy_image, dummy_cliff))
- if max_batch_size > 1:
- self.session.run(None, self._feeds(dummy_image[:1], dummy_cliff[:1]))
-
- # ------------------------------------------------------------------
- # Preprocessing
- # ------------------------------------------------------------------
-
- def _preprocess(
- self,
- image_rgb: np.ndarray,
- bbox: np.ndarray,
- h: int,
- w: int,
- focal: float | None = None,
- ) -> tuple[np.ndarray, float, float, float, np.ndarray]:
- """Square 1.2x crop, ImageNet normalise, and CLIFF conditioning."""
- x1, y1, x2, y2 = bbox.astype(float)
- bw = x2 - x1
- bh = y2 - y1
- cx = (x1 + x2) / 2.0
- cy = (y1 + y2) / 2.0
-
- # CLIFF conditioning vector (full-frame coords). Must match the variant
- # the checkpoint was TRAINED with -- see `cliff_focal` in
- # training/train.py. A model trained
- # with the focal-aware form and run with the pixel form (or the other
- # way round) does not error; it silently mis-places the person in depth.
+ def _run(self, crops: np.ndarray, cliffs: np.ndarray) -> dict[str, np.ndarray]:
+ names = ["mhr_params", "shape_params", "cam_trans", "joints_2d"]
+ outs = dict(zip(names, self.network.run(names, {"image": crops, "cliff_cond": cliffs})))
+ outs.update(zip(["joints_3d", "vertices"], self.body.run(
+ ["joints_3d", "vertices"], {"mhr_params": outs["mhr_params"], "shape_params": outs["shape_params"]})))
+ return outs
+
+ def warmup(self, max_batch: int = 1) -> None:
+ for n in sorted({1, max_batch}):
+ self.network.run(None, {"image": np.zeros((n, 3, self.input_size, self.input_size), np.float32),
+ "cliff_cond": np.zeros((n, 3), np.float32)})
+ self.body.run(None, {"mhr_params": np.zeros((n, 204), np.float32),
+ "shape_params": np.zeros((n, 45), np.float32)})
+
+ def _preprocess(self, image_rgb: np.ndarray, bbox: np.ndarray, focal: float):
+ """The square 1.2x crop the model was trained on, and its CLIFF conditioning vector."""
+ h, w = image_rgb.shape[:2]
+ x1, y1, x2, y2 = bbox.astype(np.float64)
+ cx, cy, size = (x1 + x2) / 2, (y1 + y2) / 2, max(x2 - x1, y2 - y1)
if self.cliff_focal:
- f = self._focal_px(h, w, focal)
- cliff_cond = np.array([math.atan((cx - w / 2.0) / f),
- math.atan((cy - h / 2.0) / f),
- max(bw, bh) / f], dtype=np.float32)
+ cliff = [math.atan((cx - w / 2) / focal), math.atan((cy - h / 2) / focal), size / focal]
else:
- cx_norm = 2.0 * (cx / w) - 1.0
- cy_norm = 2.0 * (cy / h) - 1.0
- b_scale = max(bw, bh) / max(w, h)
- cliff_cond = np.array([cx_norm, cy_norm, b_scale], dtype=np.float32)
-
- sq_size = max(bw, bh) * CROP_EXPAND
- half = sq_size / 2.0
- sq_x1 = cx - half
- sq_y1 = cy - half
-
- ix1 = int(math.floor(sq_x1))
- iy1 = int(math.floor(sq_y1))
- ix2 = int(math.ceil(sq_x1 + sq_size))
- iy2 = int(math.ceil(sq_y1 + sq_size))
-
- pad_left = max(0, -ix1)
- pad_top = max(0, -iy1)
- pad_right = max(0, ix2 - w)
- pad_bottom = max(0, iy2 - h)
-
- src_x1 = max(0, ix1)
- src_y1 = max(0, iy1)
- src_x2 = min(w, ix2)
- src_y2 = min(h, iy2)
-
- patch = image_rgb[src_y1:src_y2, src_x1:src_x2]
- if pad_left or pad_top or pad_right or pad_bottom:
- patch = cv2.copyMakeBorder(
- patch, pad_top, pad_bottom, pad_left, pad_right,
- cv2.BORDER_CONSTANT, value=(0, 0, 0),
- )
-
- crop_sq = cv2.resize(patch, (self.input_size, self.input_size), interpolation=cv2.INTER_LINEAR)
- crop = crop_sq.astype(np.float32) / 255.0
- crop = (crop - IMAGENET_MEAN) / IMAGENET_STD
- crop = np.transpose(crop, (2, 0, 1)).astype(np.float32) # (3, S, S)
-
- return crop, float(sq_x1), float(sq_y1), float(sq_size), cliff_cond
-
- # ------------------------------------------------------------------
- # Provider selection
- # ------------------------------------------------------------------
-
- @staticmethod
- def _default_providers(device: str) -> list[str]:
- """Pick ORT execution providers based on the requested *device*.
-
- Recognised values:
- ``"cuda"`` → ``CUDAExecutionProvider`` (NVIDIA).
- ``"coreml"`` → ``CoreMLExecutionProvider`` (Apple Silicon / macOS).
- ``"cpu"`` / anything else → CPU only.
-
- ``CPUExecutionProvider`` is always appended as a fallback so a
- partially-supported model still loads.
-
- TensorRT is intentionally **not** requested by default: the wheel
- distinction between ``onnxruntime-gpu`` and the ``tensorrt`` system
- libraries is fiddly, and asking for an EP whose runtime libraries
- aren't installed produces a verbose error before falling back.
- Pass ``providers=`` explicitly if you want TensorRT.
- """
- import onnxruntime as ort
-
- available = set(ort.get_available_providers())
- wanted: list[str] = []
- device_l = device.lower()
- if "cuda" in device_l and "CUDAExecutionProvider" in available:
- wanted.append("CUDAExecutionProvider")
- elif "coreml" in device_l and "CoreMLExecutionProvider" in available:
- wanted.append("CoreMLExecutionProvider")
- wanted.append("CPUExecutionProvider")
- return wanted
+ cliff = [2 * cx / w - 1, 2 * cy / h - 1, size / max(w, h)]
+
+ side = size * CROP_EXPAND
+ sx, sy = cx - side / 2, cy - side / 2
+ ix1, iy1 = math.floor(sx), math.floor(sy)
+ ix2, iy2 = math.ceil(sx + side), math.ceil(sy + side)
+ patch = image_rgb[max(0, iy1):min(h, iy2), max(0, ix1):min(w, ix2)]
+ patch = cv2.copyMakeBorder(patch, max(0, -iy1), max(0, iy2 - h), max(0, -ix1), max(0, ix2 - w),
+ cv2.BORDER_CONSTANT, value=(0, 0, 0))
+ crop = cv2.resize(patch, (self.input_size, self.input_size), interpolation=cv2.INTER_LINEAR)
+ crop = ((crop.astype(np.float32) / 255.0 - IMAGENET_MEAN) / IMAGENET_STD).transpose(2, 0, 1)
+ return np.ascontiguousarray(crop), np.asarray(cliff, np.float32), (float(sx), float(sy), float(side))
diff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.py
deleted file mode 100644
index 97b3bb4..0000000
--- a/instanthmr/mhr_renderer.py
+++ /dev/null
@@ -1,407 +0,0 @@
-"""MHR (Momentum Human Rig) decoder for InstantHMR.
-
-Takes the 204-dim pose parameters and 45-dim shape parameters produced by
-InstantHMR and runs a forward pass through Meta's MHR body model to obtain
-
- * the full body **mesh** (skinned vertices), and
- * the 70 native **keypoints**, regressed from the 127 skeleton joints.
-
-The keypoints matter because ``train.py`` deletes the
-student's 3D coordinate head: its ONNX graph has four outputs instead of five
-and the 3D pose must be derived here, exactly the way the teacher derives it
-from the mesh. The (70, 127) affine map lives in
-``training/assets/mhr_j127_to_kp70.npy`` and was fitted against
-the teacher's own labels (held-out residual: 1.36 mm mean).
-
-Two interchangeable backends
-----------------------------
-``MHRScriptRenderer`` (recommended)
- Loads the self-contained TorchScript rig at ``checkpoints/mhr_model.pt`` —
- the same artefact the distillation used. Needs nothing but PyTorch, so it
- is immune to the ``pymomentum`` / ``torch`` ABI pairing below.
-
-``MHRRenderer``
- Loads the unpacked MHR asset folder through the ``mhr`` pip package. This
- lets you pick a mesh LOD, but ``pymomentum-gpu`` is a torch C++ extension
- and **must** match the installed torch (0.1.113 wants ``torch>=2.8,<2.9``);
- a mismatch segfaults the process rather than raising.
-
-Use :func:`build_mhr` to get whichever one the caller configured.
-
-Quick setup for the asset backend (requires Python >= 3.12)::
-
- pip install -r requirements-mhr.txt
- curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
- unzip assets.zip -d models/mhr_assets
- python demo.py --image photo.jpg --mhr-assets models/mhr_assets/assets
-
-WARNING — wrong pymomentum package on PyPI
-------------------------------------------
-``pip install pymomentum`` installs an **unrelated** legacy SMS library
-(pyMomentum v0.1.x by MomentumAS). The correct Meta package is::
-
- pip install pymomentum-gpu # NVIDIA GPU
- pip install pymomentum-cpu # macOS / CPU-only
-
-See README.md §'MHR body mesh' for the full installation walkthrough.
-"""
-
-from __future__ import annotations
-
-from pathlib import Path
-from typing import Optional
-
-import numpy as np
-
-
-_REPO_ROOT = Path(__file__).resolve().parent.parent
-
-#: (70, 127) affine map from MHR skeleton joints to the 70 native keypoints.
-DEFAULT_KP_REGRESSOR = _REPO_ROOT / "training/assets/mhr_j127_to_kp70.npy"
-#: TorchScript rig shipped with the training code.
-DEFAULT_MHR_SCRIPT = _REPO_ROOT / "checkpoints/mhr_model.pt"
-
-NUM_MODEL_PARAMS = 204 # 136 pose + 68 scale
-NUM_IDENTITY = 45 # identity blend-shape coefficients
-NUM_EXPRESSION = 72 # facial expression coefficients (InstantHMR predicts none)
-NUM_SKELETON_JOINTS = 127
-NUM_KEYPOINTS = 70
-
-
-class _MHRBase:
- """Parameter marshalling, coordinate conversion, rig evaluation and keypoints.
-
- Both backends expose the same three objects, so everything below the loader
- is shared. A subclass' ``__init__`` must set:
-
- * ``self._model`` — callable ``(identity, pose, expr) -> (verts, skel)``
- * ``self._character`` — exposes ``model_parameters_to_joint_parameters``
- and ``joint_parameters_to_skeleton_state``
- * ``self._n_params`` — width its parameter transform expects, i.e. 204
- model parameters plus that rig's blend-shape slots
-
- and it must implement the ``faces`` property.
- """
-
- def __init__(self, device: str = "cuda", kp_regressor: str | Path | None = None):
- import torch
-
- self._torch = torch
- self._device = self._resolve_device(torch, device)
- self._kp_regressor = None
-
- path = Path(kp_regressor) if kp_regressor is not None else DEFAULT_KP_REGRESSOR
- if path.exists():
- W = np.load(path).astype(np.float32)
- if W.shape != (NUM_KEYPOINTS, NUM_SKELETON_JOINTS):
- raise ValueError(
- f"expected a ({NUM_KEYPOINTS}, {NUM_SKELETON_JOINTS}) keypoint "
- f"regressor in {path}, got {W.shape}"
- )
- self._kp_regressor = torch.from_numpy(W).to(self._device)
- elif kp_regressor is not None:
- raise FileNotFoundError(f"keypoint regressor not found: {path}")
-
- @staticmethod
- def _resolve_device(torch, device):
- """Map the demo's device string onto a torch device.
-
- ``--device coreml`` selects an onnxruntime execution provider, not a
- torch backend, so anything torch cannot run on falls back to CPU.
- """
- name = str(device).lower()
- if "cuda" in name and torch.cuda.is_available():
- return torch.device(device)
- if "cuda" in name:
- return torch.device("cpu")
- try:
- return torch.device(device)
- except (RuntimeError, TypeError):
- return torch.device("cpu")
-
- # ------------------------------------------------------------------
- # Capabilities
- # ------------------------------------------------------------------
-
- @property
- def has_keypoints(self) -> bool:
- """True when the 70-keypoint regressor was found and loaded."""
- return self._kp_regressor is not None
-
- @property
- def faces(self) -> np.ndarray:
- raise NotImplementedError
-
- # ------------------------------------------------------------------
- # Coordinate conventions
- # ------------------------------------------------------------------
-
- @staticmethod
- def _to_vision(xyz):
- """Raw MHR (centimetres, Y-up / Z-back) -> InstantHMR (metres, Y-down / Z-forward).
-
- Negating both Y and Z keeps the frame right-handed after the Y flip.
- This is the exact transform ``train.MHRForwardPass``
- applies before the keypoint regression, so student and demo agree.
- """
- out = xyz / 100.0
- out[..., 1] *= -1.0
- out[..., 2] *= -1.0
- return out
-
- # ------------------------------------------------------------------
- # Public API
- # ------------------------------------------------------------------
-
- def keypoints(
- self,
- mhr_params: np.ndarray,
- shape_params: np.ndarray,
- ) -> np.ndarray:
- """Regress the 70 native keypoints from the MHR skeleton.
-
- Skeleton branch only — no blend shapes, no skinning — so this is ~3x
- cheaper than a full mesh forward (≈0.9 ms vs ≈2.6 ms at batch 2 on an
- RTX 4070).
-
- Args:
- mhr_params: ``(204,)`` or ``(B, 204)`` float32 pose parameters.
- shape_params: ``(45,)`` or ``(B, 45)`` — accepted for symmetry with
- :meth:`forward`; identity coefficients do not move the skeleton
- (they only displace the rest-pose vertices), which is why the
- distillation passes zeros here too.
-
- Returns:
- ``(70, 3)`` or ``(B, 70, 3)`` float32 keypoints in the **rig-local**
- frame (metres, Y-down) — the frame ``cam_trans`` is relative to.
- Add ``cam_trans`` for camera space.
- """
- if self._kp_regressor is None:
- raise RuntimeError(
- f"no MHR keypoint regressor loaded (looked for {DEFAULT_KP_REGRESSOR}).\n"
- "It ships with the training code; regenerate it with\n"
- " python training/train.py --fit-regressor"
- )
- torch = self._torch
- pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)
-
- with torch.no_grad():
- skel = self._skeleton_state(pose)
- kps = self._regress(skel)
-
- out = kps.cpu().numpy().astype(np.float32)
- return out[0] if squeeze else out
-
- def forward(
- self,
- mhr_params: np.ndarray,
- shape_params: np.ndarray,
- ) -> np.ndarray:
- """Decode one person's parameters into body-space mesh vertices.
-
- Args:
- mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.
- shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.
-
- Returns:
- ``(V, 3)`` float32 vertices in the **rig-local** frame (metres,
- Y-down). Add the person's ``cam_trans`` to move to camera space.
- """
- verts, _ = self.forward_full(mhr_params, shape_params)
- return verts
-
- def forward_full(
- self,
- mhr_params: np.ndarray,
- shape_params: np.ndarray,
- ) -> tuple[np.ndarray, Optional[np.ndarray]]:
- """Mesh vertices **and** the 70 keypoints from a single MHR forward pass.
-
- Returns ``(verts, keypoints)`` with the same leading shape convention as
- :meth:`forward` / :meth:`keypoints`. ``keypoints`` is ``None`` when no
- regressor is available.
- """
- torch = self._torch
- pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)
- identity, _ = self._as_batch(shape_params, NUM_IDENTITY)
- if identity.shape[0] != pose.shape[0]:
- identity = identity.expand(pose.shape[0], -1)
-
- with torch.no_grad():
- verts, skel = self._skin(identity, pose)
- verts = self._to_vision(verts.float())
- kps = self._regress(skel) if self._kp_regressor is not None else None
-
- v = verts.cpu().numpy().astype(np.float32)
- k = kps.cpu().numpy().astype(np.float32) if kps is not None else None
- if squeeze:
- return v[0], (None if k is None else k[0])
- return v, k
-
- # ------------------------------------------------------------------
- # Internals
- # ------------------------------------------------------------------
-
- def _as_batch(self, arr: np.ndarray, dim: int):
- """numpy ``(dim,)`` / ``(B, dim)`` -> torch ``(B, dim)`` on the rig device."""
- torch = self._torch
- t = torch.as_tensor(np.ascontiguousarray(arr), dtype=torch.float32)
- squeeze = t.dim() == 1
- if squeeze:
- t = t.unsqueeze(0)
- if t.shape[-1] != dim:
- raise ValueError(f"expected trailing dimension {dim}, got {tuple(t.shape)}")
- return t.to(self._device), squeeze
-
- def _regress(self, skel):
- """``(B, 127, 8)`` skeleton state -> ``(B, 70, 3)`` keypoints (vision frame)."""
- joints = self._to_vision(skel[..., :3].float())
- return self._torch.einsum("kj,bjc->bkc", self._kp_regressor, joints)
-
- def _pad(self, pose, total: int):
- """Right-pad the 204 model parameters with zero blend-shape coefficients."""
- torch = self._torch
- extra = total - pose.shape[-1]
- if extra <= 0:
- return pose
- return torch.cat(
- [pose, torch.zeros(pose.shape[0], extra, dtype=pose.dtype, device=pose.device)],
- dim=1,
- )
-
- def _skeleton_state(self, pose):
- """``(B, 204)`` model parameters -> ``(B, 127, 8)`` raw skeleton state.
-
- Skips blend shapes and skinning entirely: only the two rigid-transform
- stages the keypoint regression needs.
- """
- ch = self._character
- joint_params = ch.model_parameters_to_joint_parameters(self._pad(pose, self._n_params))
- return ch.joint_parameters_to_skeleton_state(joint_params)
-
- def _skin(self, identity, pose):
- """Full rig evaluation -> ``((B, V, 3) verts, (B, 127, 8) skeleton state)``.
-
- InstantHMR predicts no facial expression, so those coefficients are zero.
- """
- torch = self._torch
- expr = torch.zeros(
- pose.shape[0], NUM_EXPRESSION, dtype=pose.dtype, device=pose.device,
- )
- verts, skel = self._model(identity, pose, expr)
- return verts, skel
-
-
-class MHRScriptRenderer(_MHRBase):
- """MHR rig loaded from the TorchScript artefact used by the distillation.
-
- Args:
- script_path: path to ``mhr_model.pt`` (default: ``checkpoints/mhr_model.pt``).
- device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.
- kp_regressor: optional override for the (70, 127) keypoint map.
-
- The mesh LOD is baked into the artefact (18 439 vertices / 36 874 faces);
- use :class:`MHRRenderer` if you need to choose one.
- """
-
- def __init__(
- self,
- script_path: str | Path = DEFAULT_MHR_SCRIPT,
- device: str = "cuda",
- kp_regressor: str | Path | None = None,
- ) -> None:
- super().__init__(device=device, kp_regressor=kp_regressor)
- torch = self._torch
-
- script_path = Path(script_path)
- if not script_path.exists():
- raise FileNotFoundError(f"MHR TorchScript rig not found: {script_path}")
-
- self._model = torch.jit.load(str(script_path), map_location=self._device).eval()
- for p in self._model.parameters():
- p.requires_grad_(False)
-
- self._character = self._model.character_torch
- # 204 model parameters + 45 identity blend-shape slots in this rig's
- # parameter transform; read it off rather than assuming.
- self._n_params = len(self._character.parameter_transform.parameter_names)
-
- faces = self._character.mesh.faces
- self._faces = faces.cpu().numpy().astype(np.int32)
-
- @property
- def faces(self) -> np.ndarray:
- """Triangle face indices ``(F, 3)`` int32, shared across all frames."""
- return self._faces
-
-
-
-class MHRRenderer(_MHRBase):
- """MHR rig loaded from the unpacked asset folder via the ``mhr`` pip package.
-
- Args:
- assets_folder: path to the folder that holds ``lod*.fbx`` and
- ``compact_v6_1.model`` (i.e. the ``assets/`` directory inside the
- unpacked ``assets.zip``).
- device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.
- lod: level-of-detail mesh resolution:
- 0 = 73 639 verts (highest), 3 = 4 899, 6 = 595 (lowest).
- LOD 3 is a good default for live demos.
- kp_regressor: optional override for the (70, 127) keypoint map.
-
- Note:
- ``pymomentum-gpu`` is a torch C++ extension. If its build does not match
- the installed torch (0.1.113 requires ``torch>=2.8,<2.9``) the process
- segfaults inside the native library instead of raising — prefer
- :class:`MHRScriptRenderer` when in doubt.
- """
-
- def __init__(
- self,
- assets_folder: str | Path = "models/mhr_assets",
- device: str = "cuda",
- lod: int = 3,
- kp_regressor: str | Path | None = None,
- ) -> None:
- super().__init__(device=device, kp_regressor=kp_regressor)
- from mhr.mhr import MHR
-
- folder = Path(assets_folder)
- # assets.zip unpacks to <dir>/assets/ — accept either level.
- if not (folder / f"lod{lod}.fbx").exists() and (folder / "assets" / f"lod{lod}.fbx").exists():
- folder = folder / "assets"
-
- self._model = MHR.from_files(folder=folder, device=self._device, lod=lod)
- self._character = self._model.character_torch
- self._n_params = self._model.character.parameter_transform.size
-
- faces_raw = self._model.character.mesh.faces
- if hasattr(faces_raw, "cpu"):
- self._faces: np.ndarray = faces_raw.cpu().numpy().astype(np.int32)
- else:
- self._faces = np.asarray(faces_raw, dtype=np.int32)
-
- @property
- def faces(self) -> np.ndarray:
- """Triangle face indices ``(F, 3)`` int32, shared across all frames."""
- return self._faces
-
-
-
-def build_mhr(
- script_path: str | Path | None = None,
- assets_folder: str | Path | None = None,
- device: str = "cuda",
- lod: int = 3,
- kp_regressor: str | Path | None = None,
-) -> _MHRBase | None:
- """Build whichever MHR backend the caller configured.
-
- ``script_path`` wins when both are given. Returns ``None`` when neither is
- set, so callers can treat "no MHR" as a plain absence.
- """
- if script_path is not None:
- return MHRScriptRenderer(script_path, device=device, kp_regressor=kp_regressor)
- if assets_folder is not None:
- return MHRRenderer(assets_folder, device=device, lod=lod, kp_regressor=kp_regressor)
- return None
diff --git a/instanthmr/models.py b/instanthmr/models.py
new file mode 100644
index 0000000..95bccb3
--- /dev/null
+++ b/instanthmr/models.py
@@ -0,0 +1,40 @@
+"""The released InstantHMR models and their download from the Hugging Face Hub."""
+from __future__ import annotations
+
+from pathlib import Path
+
+REPO_ID = "momolesang/InstantHMR"
+
+MODELS = {
+ "nano": "instanthmr-nano.onnx",
+ "tiny": "instanthmr-tiny.onnx",
+ "small": "instanthmr-small.onnx",
+ "medium": "instanthmr-medium.onnx",
+ "medium-commercial": "instanthmr-medium-commercial.onnx",
+}
+DEFAULT_MODEL = "medium"
+BODY = "mhr-body.onnx"
+
+
+def _download(filename: str) -> Path:
+ from huggingface_hub import hf_hub_download
+
+ return Path(hf_hub_download(REPO_ID, filename))
+
+
+def resolve_model(model: str | Path) -> Path:
+ """A local `.onnx` path, or a model name downloaded once into the Hugging Face cache."""
+ path = Path(model)
+ if path.suffix == ".onnx":
+ if not path.exists():
+ raise FileNotFoundError(f"model not found: {path}")
+ return path
+ if str(model) not in MODELS:
+ raise ValueError(f"unknown model {model!r}: choose one of {', '.join(MODELS)} or pass an .onnx path")
+ return _download(MODELS[str(model)])
+
+
+def resolve_body(model_path: Path) -> Path:
+ """The MHR body graph: next to a local model if there is one, else downloaded."""
+ local = Path(model_path).with_name(BODY)
+ return local if local.exists() else _download(BODY)
diff --git a/instanthmr/pipeline.py b/instanthmr/pipeline.py
index e2393f3..f2775a0 100644
--- a/instanthmr/pipeline.py
+++ b/instanthmr/pipeline.py
@@ -1,260 +1,49 @@
-"""End-to-end pose estimation pipeline.
-
-Combines the RF-DETR person detector with InstantHMR to produce per-person
-3D + 2D joints from a single RGB frame, while reporting **separate**
-detector and HMR timings so the demo can surface them.
-"""
-
+"""Person detection followed by InstantHMR, with per-stage timings."""
from __future__ import annotations
import time
-from collections import deque
from dataclasses import dataclass
from pathlib import Path
-from typing import Optional
import numpy as np
-from .detector import RFDETRDetector
-from .inference import InstantHMR, HMRPrediction
-from .smoothing import _Tracker
+from .detector import PersonDetector
+from .inference import HMRPrediction, InstantHMR
+from .models import DEFAULT_MODEL, resolve_model
@dataclass
class FrameResult:
- """Output of one ``PosePipeline.predict`` call.
-
- Attributes:
- persons: Per-person ``HMRPrediction``s (possibly empty).
- detector_ms: Wall-clock time for the detector forward pass (0 on
- frames where the detector was skipped via ``detector_stride``).
- hmr_ms: Wall-clock time for InstantHMR across all persons in the
- frame (sum, not average — or one batched call when
- ``batch_persons=True``).
- total_ms: ``detector_ms + hmr_ms`` plus pipeline overhead.
- """
-
persons: list[HMRPrediction]
detector_ms: float
hmr_ms: float
- total_ms: float
-
-
-def _expand_bbox(bbox: np.ndarray, w: int, h: int, expand: float) -> np.ndarray:
- bw = bbox[2] - bbox[0]
- bh = bbox[3] - bbox[1]
- return np.array([
- max(0.0, bbox[0] - bw * expand),
- max(0.0, bbox[1] - bh * expand),
- min(float(w), bbox[2] + bw * expand),
- min(float(h), bbox[3] + bh * expand),
- ], dtype=np.float32)
-
-
-@dataclass
-class _Box:
- """What :class:`instanthmr.smoothing._Tracker` reads off a person."""
- bbox: np.ndarray
class PosePipeline:
- """Full RF-DETR → InstantHMR pipeline.
-
- Args:
- onnx_path: Path to ``instanthmr.onnx``.
- device: ``"cuda"``, ``"coreml"`` or ``"cpu"`` for InstantHMR.
- detector_variant: ``"nano" | "small" | "medium" | "base" | "large"``.
- det_confidence: RF-DETR confidence threshold.
- max_persons: Maximum number of persons returned per frame.
- detector_stride: Run the detector every ``N`` frames; on the
- in-between frames, reuse the previous frame's detections
- (slightly expanded) to drive InstantHMR. ``1`` reproduces the
- original "detect every frame" behaviour. The detector is by
- far the dominant cost on most hardware, so stride 2–3 is the
- single biggest knob for end-to-end FPS.
- batch_persons: When ``True`` (default), all detected persons in a
- frame are batched into a single InstantHMR ONNX call instead
- of looping. This is a meaningful win for multi-person frames
- and free for single-person frames.
- mhr: Optional MHR backend (see :mod:`instanthmr.mhr_renderer`).
- Required for MHR-only graphs, whose 3D keypoints are derived
- from the predicted MHR parameters instead of a 3D head; ignored
- by graphs that carry a ``joints_3d`` output.
- focal: The camera's focal length in pixels, when known. A graph
- exported from a ``--cliff-focal`` checkpoint is conditioned on
- angles rather than pixels, so it needs a focal; without one it
- falls back to a diagonal-derived stand-in, which mis-places the
- person in depth on any camera whose true focal differs.
- cliff_focal: Override the conditioning form; normally read from the
- ONNX metadata. See :class:`instanthmr.inference.InstantHMR`.
- use_prev: For a prev-token graph, feed each tracked person their own
- prediction from ``prev_lag`` frames earlier (IoU tracking on the
- detector boxes); the previous pose is withheld on a track's first
- ``prev_lag`` frames. ``False`` always withholds it. Ignored by
- other graphs.
- prev_lag: Processed frames between a prediction and the frame it
- feeds. 5 (167 ms at 30 fps) is what the benchmarks use, inside
- the 5-20 frame pairs the token was trained on.
+ """RF-DETR, then InstantHMR on every detected person in one batched call.
- Example
- -------
- pipeline = PosePipeline("models/instanthmr.onnx", detector_stride=2)
- result = pipeline.predict(image_rgb)
- for p in result.persons:
- print(p.joints_3d_cam.shape) # (70, 3)
+ `model` is a released model name (nano, tiny, small, medium, medium-commercial) or an .onnx
+ path. `detector_stride` > 1 runs the detector every N frames and reuses its boxes in between.
"""
- def __init__(
- self,
- onnx_path: str | Path,
- device: str = "cuda",
- detector_variant: str = "medium",
- det_confidence: float = 0.5,
- max_persons: int = 2,
- detector_stride: int = 1,
- batch_persons: bool = True,
- mhr: object | None = None,
- focal: float | None = None,
- cliff_focal: bool | None = None,
- use_prev: bool = True,
- prev_lag: int = 5,
- ):
- if detector_stride < 1:
- raise ValueError("detector_stride must be >= 1")
-
- self.hmr = InstantHMR(onnx_path, device=device, mhr=mhr,
- focal=focal, cliff_focal=cliff_focal)
- if self.hmr.derives_joints_3d and mhr is None:
- raise ValueError(
- f"{Path(onnx_path).name} has no 'joints_3d' output (MHR-only student): "
- "its 3D keypoints come from an MHR forward pass, so an MHR backend is "
- "required. Pass mhr=build_mhr(...), or run demo.py with --mhr-model / "
- "--mhr-assets."
- )
- self.detector = RFDETRDetector(
- variant=detector_variant,
- confidence=det_confidence,
- max_persons=max_persons,
- )
-
- # Warm up both models before the first real frame.
- # - RF-DETR (PyTorch): first call triggers CUDA JIT for transformer ops.
- # - InstantHMR (ONNX/CUDA EP): first call at a new batch size triggers
- # kernel compilation and GPU memory allocation.
- # With padded-batch inference the session only ever sees two shapes:
- # batch=1 and batch=max_persons — two warm-up calls covers everything.
- print(f"Warming up models (max_persons={max_persons})…", flush=True)
- self.detector.warmup()
- self.hmr.warmup(max_batch_size=max_persons)
- if mhr is not None:
- self._warmup_mhr(mhr, max_persons)
- print("Ready.", flush=True)
-
- self._max_persons = max_persons
- self._detector_stride = detector_stride
- self._batch_persons = batch_persons
- self._frame_idx = 0
- self._last_detections: list[dict] = []
- self._last_bbox: Optional[np.ndarray] = None
- self.feeds_prev = use_prev and self.hmr.has_prev_input
- self._prev_lag = max(1, prev_lag)
- self._tracker = _Tracker()
- self._history: dict[int, deque] = {} # track id -> last prev_lag prev_pose()s
-
- @staticmethod
- def _warmup_mhr(mhr, max_persons: int) -> None:
- """Run the MHR rig once per batch shape the demo will actually use.
-
- TorchScript specialises on input shapes over the first few calls, so an
- un-warmed rig costs hundreds of ms on frame 1 instead of ~1 ms. Three
- passes is what it takes for the profiling executor to settle: the first
- records shapes, the second triggers the fusion compile, the third runs
- the compiled kernels.
- """
- pose = np.zeros((max_persons, 204), dtype=np.float32)
- shape = np.zeros((max_persons, 45), dtype=np.float32)
- for _ in range(3):
- mhr.forward(pose[0], shape[0]) # mesh: always single-person
- if getattr(mhr, "has_keypoints", False):
- for n in sorted({1, max_persons}): # keypoints: 1 and the padded batch
- mhr.keypoints(pose[:n], shape[:n])
-
- def predict(self, image_rgb: np.ndarray) -> FrameResult:
- """Detect all persons in *image_rgb* and run InstantHMR on each."""
- t_total_start = time.perf_counter()
- h, w = image_rgb.shape[:2]
-
- # ---- Detector (with optional stride) ----
- run_detector = (self._frame_idx % self._detector_stride) == 0
- if run_detector:
- t0 = time.perf_counter()
- detections = self.detector.detect(image_rgb)
- detector_ms = (time.perf_counter() - t0) * 1000.0
- if detections:
- self._last_detections = detections
- else:
- detections = []
- detector_ms = 0.0
-
- # If the detector skipped or missed this frame, reuse the previous
- # detections (slightly expanded to absorb small motion). This is a
- # cheap form of tracking that keeps HMR fed with a plausible bbox.
- if not detections and self._last_detections:
- detections = [
- {
- "bbox": _expand_bbox(d["bbox"], w, h, expand=0.1),
- "confidence": float(d["confidence"]) * 0.9,
- }
- for d in self._last_detections
- ]
- elif not detections and self._last_bbox is not None:
- # Backwards-compatible single-bbox fallback (used when no prior
- # multi-person detections exist).
- fb = _expand_bbox(self._last_bbox, w, h, expand=0.1)
- detections = [{"bbox": fb, "confidence": 0.0}]
-
- # ---- InstantHMR ----
- # Hard-cap so we never exceed the pre-warmed batch size.
- detections = detections[: self._max_persons]
- outputs: list[HMRPrediction] = []
+ def __init__(self, model: str | Path = DEFAULT_MODEL, device: str = "cuda", detector: str = "medium",
+ det_confidence: float = 0.5, max_persons: int = 4, detector_stride: int = 1,
+ focal: float | None = None):
+ self.hmr = InstantHMR(resolve_model(model), device=device, focal=focal, max_batch=max_persons)
+ self.detector = PersonDetector(detector, det_confidence, max_persons)
+ self.detector_stride = max(1, detector_stride)
+ self._frame = 0
+ self._boxes = np.zeros((0, 4), np.float32)
+ self._scores = np.zeros(0, np.float32)
+ self.detector(np.zeros((288, 288, 3), np.uint8))
+ self.hmr.warmup(max_persons)
+
+ def __call__(self, image_rgb: np.ndarray) -> FrameResult:
+ t0 = time.perf_counter()
+ if self._frame % self.detector_stride == 0:
+ self._boxes, self._scores = self.detector(image_rgb)
t1 = time.perf_counter()
- ids, prev = [], [None] * len(detections)
- if self.feeds_prev:
- ids = self._tracker.assign([_Box(d["bbox"]) for d in detections])
- prev = [h[0] if len(h) == self._prev_lag else None
- for h in (self._history.get(t, ()) for t in ids)]
- if detections:
- if self._batch_persons and len(detections) > 1:
- # Always pad to _max_persons so ONNX sees a constant batch
- # shape and never triggers a mid-demo recompilation.
- outputs = self.hmr.predict_batch(
- image_rgb, detections, padded_to=self._max_persons, prev=prev,
- )
- else:
- for det, pp in zip(detections, prev):
- outputs.append(self.hmr.predict(
- image_rgb,
- bbox=det["bbox"],
- confidence=det["confidence"],
- prev=pp,
- ))
- if self.feeds_prev:
- for t, o in zip(ids, outputs):
- self._history.setdefault(t, deque(maxlen=self._prev_lag)).append(o.prev_pose())
- for t in set(self._history) - self._tracker.active_ids:
- del self._history[t]
- hmr_ms = (time.perf_counter() - t1) * 1000.0
-
- if outputs:
- best = max(outputs, key=lambda o: o.confidence)
- self._last_bbox = best.bbox.copy()
-
- total_ms = (time.perf_counter() - t_total_start) * 1000.0
- self._frame_idx += 1
-
- return FrameResult(
- persons=outputs,
- detector_ms=detector_ms,
- hmr_ms=hmr_ms,
- total_ms=total_ms,
- )
+ persons = self.hmr.predict(image_rgb, self._boxes, self._scores)
+ t2 = time.perf_counter()
+ self._frame += 1
+ return FrameResult(persons, (t1 - t0) * 1000, (t2 - t1) * 1000)
diff --git a/instanthmr/reference.py b/instanthmr/reference.py
deleted file mode 100644
index 1401797..0000000
--- a/instanthmr/reference.py
+++ /dev/null
@@ -1,104 +0,0 @@
-"""Saved SAM 3D Body predictions, for side-by-side comparison in the demo.
-
-The teacher's own output on a clip is the most useful thing to hold a student
-against visually: it shares the rig, so the two meshes are directly comparable
-rather than needing a conversion.
-
-A reference file is produced by running the teacher once and packing the result:
-
- MOMENTUM_ENABLED=0 python main.py --video_path vid1.mp4 \\
- --output_path reference/vid1 --save_meshes # in video_to_pose_pipeline
- python tools/pack_reference.py reference/vid1/meshes reference/vid1.npz
-
-It stores MHR **parameters**, not vertices, and ``demo.py`` decodes them with
-the same rig it uses for the student. That is what makes the comparison honest:
-any difference you see in the viewer is the model, never the renderer.
-
-``MOMENTUM_ENABLED=0`` matters -- it makes the teacher's MHR head load the
-TorchScript rig instead of ``mhr.mhr.MHR``, whose pymomentum extension needs
-``torch>=2.8`` and segfaults (not raises) against an older torch.
-"""
-
-from __future__ import annotations
-
-from dataclasses import dataclass
-from pathlib import Path
-
-import numpy as np
-
-
-@dataclass(frozen=True)
-class ReferencePerson:
- """One person in one reference frame, in the demo's own conventions."""
-
- mhr_params: np.ndarray # (204,) float32
- shape_params: np.ndarray # (45,) float32
- cam_trans: np.ndarray # (3,) float32, rig-local -> camera space
- joints_3d_cam: np.ndarray # (70, 3) float32, camera space
- joints_2d: np.ndarray # (70, 2) float32, pixels
- bbox: np.ndarray # (4,) float32, xyxy
-
-
-class ReferenceTrack:
- """Random access by frame index into a packed reference run.
-
- Frames where the teacher detected nobody were never written, so a lookup
- for them returns an empty list rather than raising.
- """
-
- def __init__(self, path: str | Path):
- self.path = Path(path)
- z = np.load(self.path)
- self.image_shape = z["image_shape"]
- counts = z["counts"].astype(int)
- starts = np.concatenate([[0], np.cumsum(counts)[:-1]])
- # frame index -> (offset, count) into the flat per-person arrays
- self._span = {int(f): (int(s), int(c))
- for f, s, c in zip(z["frame_idx"], starts, counts)}
- self._model = z["model_params"]
- self._shape = z["shape_params"]
- self._trans = z["cam_trans"]
- self._j3d = z["joints_3d"]
- self._j2d = z["joints_2d"]
- self._bbox = z["bbox"]
- self.num_frames = len(counts)
- self.num_persons = int(counts.sum())
-
- def __len__(self) -> int:
- return self.num_frames
-
- def persons_at(self, frame_idx: int) -> list[ReferencePerson]:
- span = self._span.get(int(frame_idx))
- if span is None:
- return []
- start, count = span
- out = []
- for i in range(start, start + count):
- # joints_3d as saved by the teacher are rig-local, the same frame
- # its vertices are in; the demo works in camera space.
- out.append(ReferencePerson(
- mhr_params=self._model[i],
- shape_params=self._shape[i],
- cam_trans=self._trans[i],
- joints_3d_cam=self._j3d[i] + self._trans[i],
- joints_2d=self._j2d[i],
- bbox=self._bbox[i],
- ))
- return out
-
-
-def resolve_reference(arg: str | None, video: str | Path | None) -> Path | None:
- """Turn ``--show-reference`` into a path.
-
- A bare flag means "the reference for this clip": ``reference/<stem>.npz``
- beside the repo root, named after the input video.
- """
- if arg is None:
- return None
- if arg:
- return Path(arg)
- if video is None:
- raise SystemExit(
- "[error] --show-reference with no path needs --video to name the "
- "reference file (reference/<video stem>.npz)")
- return Path("reference") / f"{Path(video).stem}.npz"
diff --git a/instanthmr/smoothing.py b/instanthmr/smoothing.py
index 3ac56eb..581f913 100644
--- a/instanthmr/smoothing.py
+++ b/instanthmr/smoothing.py
@@ -1,445 +1,101 @@
-"""Temporal smoothing of per-person HMR predictions.
+"""Temporal smoothing of per-person predictions with the 1 Euro filter.
-InstantHMR is a frame-by-frame model: every frame is estimated independently,
-so its outputs carry per-frame noise that reads as jitter in a video — most
-visibly in ``cam_trans``, which is a free 3-vector from a linear head with
-almost nothing tying it to the observed pixels.
-
-A non-learned moving-average filter over a short window removes much of that.
-NLF (arXiv:2407.07532, Tables 2-3) reports a 5-frame filter of this kind
-improving 3DPW MPJPE 59.0 -> 57.2 mm and EMDB 68.4 -> 66.7 mm, at zero
-training cost. This module is that filter.
-
-Two things the filter needs that the pipeline does not provide:
-
-* **Identity across frames.** ``PosePipeline.predict`` returns an unordered
- list of detections per frame, so a person's index is not stable. We attach a
- minimal greedy IoU tracker (:class:`_Tracker`) purely to decide which
- predictions belong to the same person before averaging them.
-* **Lookahead**, for ``mode="centered"``. A centred window has no phase lag
- but needs ``(window - 1) // 2`` future frames, so :meth:`TemporalSmoother.push`
- returns results delayed by that many frames and :meth:`flush` drains the
- tail. ``mode="causal"`` trades phase lag for zero latency.
-
-The moving average has one structural weakness: it is a *fixed*-cutoff filter,
-so it cannot tell jitter from fast intentional motion and lags during both. On
-a waving hand that is plainly visible. ``mode="1euro"`` fixes it with the
-1 Euro filter (Casiez, Roussel & Vogel, CHI 2012): an exponential smoother
-whose cutoff frequency rises with the signal's own speed. At rest the cutoff is
-low and jitter is crushed; during fast motion the cutoff opens up and the
-output tracks. It is causal, needs no lookahead, and is the right choice for
-demos and live camera. The centred moving average remains the right choice for
-benchmark numbers, since that is the filter NLF measured.
-
-Caveat on ``mhr_params``: these are MHR joint parameters (angles and scales),
-and this filter averages them linearly. That is well-behaved for continuous
-motion but is *wrong* across an angle wraparound — a global yaw crossing +/-pi
-will be averaged toward zero instead of through the discontinuity, producing a
-brief flip. Short windows on real footage rarely hit it; if you see one, drop
-``mhr_params`` from ``SMOOTHED_FIELDS`` and accept a mesh that jitters while
-the keypoints do not.
+InstantHMR estimates every frame on its own, so a video shows per-frame jitter. The 1 Euro filter
+(Casiez, Roussel and Vogel, CHI 2012) smooths each coordinate with a cutoff that rises with its
+own speed: still joints are smoothed hard and fast ones barely, so it adds no visible lag. It is
+causal, so it works on a live camera. People are matched across frames by box overlap.
"""
-
from __future__ import annotations
-from collections import deque
from dataclasses import replace
-from typing import Iterator, Sequence
+from typing import Sequence
import numpy as np
from .inference import HMRPrediction
-#: Fields averaged across the window. All are Euclidean quantities for which a
-#: linear mean is the right operation, except ``mhr_params`` — see the module
-#: docstring. ``joints_3d_cam`` is deliberately absent: it is recomputed from
-#: the smoothed ``joints_3d_local`` and ``cam_trans`` so the three stay
-#: consistent with each other. ``joints_3d_local`` is filtered only when no
-#: ``keypoint_fn`` is available; with one, it is re-derived from the filtered
-#: ``mhr_params`` so the keypoints and the mesh cannot disagree.
-SMOOTHED_FIELDS = (
- "joints_3d_local",
- "cam_trans",
- "joints_2d",
- "mhr_params",
- "shape_params",
-)
-
-
-#: Per-field ``(mincutoff_hz, beta)`` for ``mode="1euro"``.
-#:
-#: ``beta`` scales a *speed*, so its right value depends on the field's units —
-#: this is the classic tuning trap with the 1 Euro filter. ``joints_2d`` is in
-#: full-frame pixels (a fast hand runs ~500-1500 px/s), the 3D fields are in
-#: metres (~1-3 m/s), ``mhr_params`` in radians and scale units. The values
-#: below put all of them at a comparable cutoff (~9 Hz) during fast motion
-#: while holding ~1 Hz at rest. ``shape_params`` gets beta 0 on purpose:
-#: identity does not change, so it should be filtered as hard as possible —
-#: the per-track limit of that idea is NLF's shared-beta-per-track fitting.
-ONE_EURO_DEFAULTS: dict[str, tuple[float, float]] = {
- "joints_2d": (1.0, 0.01),
+# (mincutoff in Hz, beta) per field, in the field's own units: pixels for joints_2d, metres for 3D.
+FIELDS = {
+ "joints_2d": (1.0, 0.01),
"joints_3d_local": (1.0, 4.0),
- "cam_trans": (0.6, 2.0),
- "mhr_params": (1.0, 2.0),
- "shape_params": (0.3, 0.0),
+ "vertices_local": (1.0, 4.0),
+ "cam_trans": (0.6, 2.0),
}
class _OneEuro:
- """1 Euro filter (Casiez, Roussel & Vogel, CHI 2012), vectorised.
-
- Operates elementwise on an array of any shape, so every joint — indeed
- every coordinate — adapts on its own speed. That is the property that
- matters here: a still torso stays heavily smoothed in the same frame where
- a waving hand is barely smoothed at all.
- """
-
def __init__(self, mincutoff: float, beta: float, dcutoff: float = 1.0):
- self.mincutoff = float(mincutoff)
- self.beta = float(beta)
- self.dcutoff = float(dcutoff)
- self._x: np.ndarray | None = None
- self._dx: np.ndarray | None = None
- self._t: float | None = None
+ self.mincutoff, self.beta, self.dcutoff = mincutoff, beta, dcutoff
+ self.x = self.dx = self.t = None
@staticmethod
- def _alpha(cutoff, dt: float):
- tau = 1.0 / (2.0 * np.pi * np.maximum(cutoff, 1e-6))
- return 1.0 / (1.0 + tau / dt)
+ def _alpha(cutoff, dt):
+ return 1.0 / (1.0 + 1.0 / (2 * np.pi * np.maximum(cutoff, 1e-6) * dt))
def __call__(self, x: np.ndarray, t: float) -> np.ndarray:
- x = np.asarray(x, dtype=np.float64)
- if self._x is None:
- self._x, self._dx, self._t = x, np.zeros_like(x), t
+ if self.x is None:
+ self.x, self.dx, self.t = x.astype(np.float64), np.zeros(x.shape), t
return x
- dt = t - self._t
- if not (dt > 0.0):
- # Repeated or non-monotonic timestamps would blow up the derivative;
- # fall back to a nominal 30 fps step rather than dividing by zero.
- dt = 1.0 / 30.0
- self._t = t
-
+ dt = t - self.t if t > self.t else 1 / 30
+ self.t = t
a_d = self._alpha(self.dcutoff, dt)
- self._dx = a_d * ((x - self._x) / dt) + (1.0 - a_d) * self._dx
- a = self._alpha(self.mincutoff + self.beta * np.abs(self._dx), dt)
- self._x = a * x + (1.0 - a) * self._x
- return self._x
+ self.dx = a_d * (x - self.x) / dt + (1 - a_d) * self.dx
+ a = self._alpha(self.mincutoff + self.beta * np.abs(self.dx), dt)
+ self.x = a * x + (1 - a) * self.x
+ return self.x.astype(np.float32)
def _iou(a: np.ndarray, b: np.ndarray) -> float:
- """Intersection-over-union of two ``[x1, y1, x2, y2]`` boxes."""
- x1 = max(float(a[0]), float(b[0]))
- y1 = max(float(a[1]), float(b[1]))
- x2 = min(float(a[2]), float(b[2]))
- y2 = min(float(a[3]), float(b[3]))
- iw = max(0.0, x2 - x1)
- ih = max(0.0, y2 - y1)
+ iw = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))
+ ih = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
inter = iw * ih
- if inter <= 0.0:
- return 0.0
- area_a = max(0.0, float(a[2] - a[0])) * max(0.0, float(a[3] - a[1]))
- area_b = max(0.0, float(b[2] - b[0])) * max(0.0, float(b[3] - b[1]))
- union = area_a + area_b - inter
- return inter / union if union > 0.0 else 0.0
+ union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter
+ return float(inter / union) if union > 0 else 0.0
-class _Tracker:
- """Greedy IoU association of detections to persistent track ids.
+class TemporalSmoother:
+ """`smoother(persons, timestamp)` returns the same people with smoothed fields.
- Deliberately minimal — it exists only to group predictions of the same
- person before averaging, not to be a general-purpose tracker. Tracks
- survive ``max_age`` frames without a match so a brief miss does not restart
- the filter (and reset the smoothing) for that person.
+ `beta_scale` trades jitter for lag: raise it if fast motion lags, lower it if a still person
+ shakes. A track survives `max_age` frames without a match before its filter is reset.
"""
- def __init__(self, iou_threshold: float = 0.3, max_age: int = 5):
- self.iou_threshold = iou_threshold
- self.max_age = max_age
- self._boxes: dict[int, np.ndarray] = {}
- self._age: dict[int, int] = {}
+ def __init__(self, beta_scale: float = 1.0, iou_threshold: float = 0.3, max_age: int = 5):
+ self.beta_scale, self.iou_threshold, self.max_age = beta_scale, iou_threshold, max_age
+ self._tracks: dict[int, dict] = {}
self._next_id = 0
- def assign(self, persons: Sequence[HMRPrediction]) -> list[int]:
- """Return one track id per person, in the order given."""
- ids: list[int] = [-1] * len(persons)
- free = set(self._boxes)
+ def __call__(self, persons: Sequence[HMRPrediction], timestamp: float) -> list[HMRPrediction]:
+ ids = self._assign(persons)
+ out = []
+ for track_id, person in zip(ids, persons):
+ filters = self._tracks[track_id]["filters"]
+ fields = {}
+ for name, (mincutoff, beta) in FIELDS.items():
+ value = getattr(person, name)
+ if value is None:
+ continue
+ f = filters.setdefault(name, _OneEuro(mincutoff, beta * self.beta_scale))
+ fields[name] = f(value, timestamp)
+ out.append(replace(person, **fields))
+ return out
- # Score every (detection, track) pair, then take them best-first. With
- # the handful of people this pipeline handles, an O(n*m) sort is far
- # cheaper than pulling in a Hungarian solver.
- pairs = [
- (_iou(p.bbox, self._boxes[t]), i, t)
- for i, p in enumerate(persons)
- for t in free
- ]
- pairs.sort(reverse=True)
- taken: set[int] = set()
+ def _assign(self, persons: Sequence[HMRPrediction]) -> list[int]:
+ pairs = sorted(((_iou(p.bbox, tr["box"]), i, t) for i, p in enumerate(persons)
+ for t, tr in self._tracks.items()), reverse=True)
+ ids, used = [-1] * len(persons), set()
for score, i, t in pairs:
- if score < self.iou_threshold:
- break
- if ids[i] != -1 or t not in free or i in taken:
- continue
- ids[i] = t
- free.discard(t)
- taken.add(i)
-
+ if score >= self.iou_threshold and ids[i] == -1 and t not in used:
+ ids[i] = t
+ used.add(t)
for i, p in enumerate(persons):
if ids[i] == -1:
ids[i] = self._next_id
+ self._tracks[ids[i]] = {"filters": {}}
self._next_id += 1
- self._boxes[ids[i]] = np.asarray(p.bbox, dtype=np.float64)
- self._age[ids[i]] = 0
-
- for t in list(free):
- self._age[t] = self._age.get(t, 0) + 1
- if self._age[t] > self.max_age:
- self._boxes.pop(t, None)
- self._age.pop(t, None)
+ self._tracks[ids[i]].update(box=p.bbox, age=0)
+ for t in set(self._tracks) - set(ids):
+ self._tracks[t]["age"] += 1
+ if self._tracks[t]["age"] > self.max_age:
+ del self._tracks[t]
return ids
-
- @property
- def active_ids(self) -> set[int]:
- return set(self._boxes)
-
-
-class TemporalSmoother:
- """Temporal filter over per-person predictions.
-
- Args:
- window: frames averaged by the moving-average modes. Forced odd in
- ``"centered"``. Ignored by ``"1euro"``.
- mode: ``"centered"`` (moving average, no phase lag, ``(window-1)//2``
- frames of latency — the filter NLF measured, so the one to use for
- benchmark numbers), ``"causal"`` (trailing average, zero latency,
- real phase lag) or ``"1euro"`` (adaptive, zero latency, no phase
- lag during fast motion — the one to use for demos and live camera).
- beta_scale: multiplies every per-field ``beta`` in ``"1euro"`` mode.
- This is *the* jitter-versus-lag knob: raise it if motion still lags,
- lower it if the output looks noisy while you hold still.
- mincutoff_scale: multiplies every per-field ``mincutoff``. Governs how
- hard the signal is smoothed at rest.
- keypoint_fn: optional ``f(mhr_params, shape_params) -> (70, 3)``. When
- given, ``joints_3d_local`` is *re-derived* from the filtered pose
- parameters instead of being filtered on its own. This is what keeps
- the keypoints welded to the mesh — see :meth:`_finalize`.
- iou_threshold: minimum IoU to associate a detection with a track.
- max_age: frames a track survives without a detection.
-
- Usage mirrors a delay line — in the zero-latency modes ``push`` simply
- always returns a frame and ``flush`` yields nothing::
-
- for persons in stream:
- out = smoother.push(persons, timestamp=t)
- if out is not None:
- render(out)
- for out in smoother.flush():
- render(out)
- """
-
- def __init__(
- self,
- window: int = 5,
- mode: str = "centered",
- beta_scale: float = 1.0,
- mincutoff_scale: float = 1.0,
- keypoint_fn=None,
- iou_threshold: float = 0.3,
- max_age: int = 5,
- ):
- if mode not in ("centered", "causal", "1euro"):
- raise ValueError(
- f"mode must be 'centered', 'causal' or '1euro', got {mode!r}")
- if window < 1:
- raise ValueError(f"window must be >= 1, got {window}")
- if mode == "centered" and window % 2 == 0:
- window += 1
- self.window = window
- self.mode = mode
- self.lag = (window - 1) // 2 if mode == "centered" else 0
- self.beta_scale = float(beta_scale)
- self.mincutoff_scale = float(mincutoff_scale)
- self.keypoint_fn = keypoint_fn
-
- self._tracker = _Tracker(iou_threshold=iou_threshold, max_age=max_age)
- self._buf: deque[dict[int, HMRPrediction]] = deque(maxlen=window)
- # mode="1euro" only: one filter bank per track, built on first sight and
- # dropped when the tracker retires the track.
- self._banks: dict[int, dict[str, _OneEuro]] = {}
- self._clock = 0.0
- # Frames in and frames out. Every pushed frame is emitted exactly once,
- # so a caller holding its own delay line can pop one entry per returned
- # frame and stay in step with the video.
- self._n_pushed = 0
- self._n_emitted = 0
-
- # Jitter accounting, so the effect is measurable rather than only
- # visible. Both are mean frame-to-frame deltas over the same tracks.
- self._prev_raw: dict[int, HMRPrediction] = {}
- self._prev_out: dict[int, HMRPrediction] = {}
- self._raw_cam: list[float] = []
- self._raw_j3d: list[float] = []
- self._out_cam: list[float] = []
- self._out_j3d: list[float] = []
-
- # ------------------------------------------------------------------
- def push(
- self,
- persons: Sequence[HMRPrediction],
- timestamp: float | None = None,
- ) -> list[HMRPrediction] | None:
- """Feed one frame. Returns a smoothed frame, or ``None`` while filling.
-
- ``timestamp`` (seconds) is used only by ``mode="1euro"``, which needs a
- real time base to turn deltas into speeds — pass wall-clock for a live
- camera and ``frame_idx / fps`` for a file, so ``--frame-skip`` is
- accounted for. It defaults to a nominal 30 fps clock.
- """
- ids = self._tracker.assign(persons)
- frame = {t: p for t, p in zip(ids, persons)}
- self._accumulate(frame, self._prev_raw, self._raw_cam, self._raw_j3d)
- self._prev_raw = frame
-
- if timestamp is None:
- self._clock += 1.0 / 30.0
- timestamp = self._clock
-
- if self.mode == "1euro":
- self._n_pushed += 1
- out = self._one_euro(frame, timestamp)
- self._n_emitted += 1
- return out
-
- self._buf.append(frame)
- self._n_pushed += 1
-
- # Emit frame ``_n_emitted`` as soon as ``lag`` frames after it have
- # arrived. Near the start of the stream that window is one-sided rather
- # than centred — the usual boundary treatment for a moving average, and
- # what keeps frames-in equal to frames-out.
- if self._n_pushed - 1 < self._n_emitted + self.lag:
- return None
- return self._emit_next()
-
- def flush(self) -> Iterator[list[HMRPrediction]]:
- """Drain the frames still held in the delay line at end of stream."""
- while self._n_emitted < self._n_pushed:
- yield self._emit_next()
-
- def _one_euro(self, frame: dict[int, HMRPrediction], t: float) -> list[HMRPrediction]:
- """Filter each track's fields independently, elementwise, on its own speed."""
- alive = self._tracker.active_ids
- for dead in [k for k in self._banks if k not in alive]:
- del self._banks[dead]
-
- out: list[HMRPrediction] = []
- emitted: dict[int, HMRPrediction] = {}
- for track_id, pred in frame.items():
- bank = self._banks.get(track_id)
- if bank is None:
- bank = {
- name: _OneEuro(
- mincutoff=ONE_EURO_DEFAULTS[name][0] * self.mincutoff_scale,
- beta=ONE_EURO_DEFAULTS[name][1] * self.beta_scale,
- )
- for name in SMOOTHED_FIELDS
- }
- self._banks[track_id] = bank
- fields = {
- name: bank[name](getattr(pred, name), t).astype(np.float32)
- for name in SMOOTHED_FIELDS
- }
- smoothed = self._finalize(pred, fields)
- out.append(smoothed)
- emitted[track_id] = smoothed
-
- self._accumulate(emitted, self._prev_out, self._out_cam, self._out_j3d)
- self._prev_out = emitted
- return out
-
- def _emit_next(self) -> list[HMRPrediction]:
- buf_start = self._n_pushed - len(self._buf)
- out = self._emit(self._n_emitted - buf_start)
- self._n_emitted += 1
- return out
-
- # ------------------------------------------------------------------
- def _finalize(self, anchor: HMRPrediction, fields: dict) -> HMRPrediction:
- """Assemble one smoothed prediction, keeping its parts self-consistent.
-
- Filtering ``mhr_params`` and ``joints_3d_local`` independently makes the
- green keypoints drift off the skin during fast motion, because the mesh
- is decoded from the former while the keypoints come from the latter and
- forward kinematics is *nonlinear* — ``filter(FK(p)) != FK(filter(p))``.
- Adaptive filtering makes it worse still, since each field then gets its
- own cutoff driven by its own speed in its own units.
-
- So when a keypoint backend is available we do not filter the keypoints
- at all: we filter the pose parameters and re-derive the keypoints from
- them, which is exactly how ``inference.py`` produces them for an
- MHR-only graph. The two agree by construction, in every mode.
- """
- if self.keypoint_fn is not None:
- fields["joints_3d_local"] = np.asarray(
- self.keypoint_fn(fields["mhr_params"], fields["shape_params"]),
- dtype=np.float32,
- ).reshape(anchor.joints_3d_local.shape)
- fields["joints_3d_cam"] = (
- fields["joints_3d_local"] + fields["cam_trans"][None, :]
- ).astype(np.float32)
- return replace(anchor, **fields)
-
- def _emit(self, centre: int) -> list[HMRPrediction]:
- """Average every track present in frame ``centre`` over the buffer."""
- out: list[HMRPrediction] = []
- emitted: dict[int, HMRPrediction] = {}
- for track_id, anchor in self._buf[centre].items():
- samples = [f[track_id] for f in self._buf if track_id in f]
- fields = {
- name: np.mean(
- np.stack([np.asarray(getattr(s, name), dtype=np.float64)
- for s in samples]),
- axis=0,
- ).astype(np.float32)
- for name in SMOOTHED_FIELDS
- }
- smoothed = self._finalize(anchor, fields)
- out.append(smoothed)
- emitted[track_id] = smoothed
-
- self._accumulate(emitted, self._prev_out, self._out_cam, self._out_j3d)
- self._prev_out = emitted
- return out
-
- @staticmethod
- def _accumulate(cur, prev, cam_acc: list[float], j3d_acc: list[float]) -> None:
- """Record mean frame-to-frame movement for tracks seen in both frames."""
- for track_id, p in cur.items():
- q = prev.get(track_id)
- if q is None:
- continue
- cam_acc.append(float(np.linalg.norm(p.cam_trans - q.cam_trans)))
- j3d_acc.append(
- float(np.linalg.norm(p.joints_3d_local - q.joints_3d_local, axis=-1).mean())
- )
-
- # ------------------------------------------------------------------
- def jitter_report(self) -> str | None:
- """One-line before/after summary, or ``None`` if nothing was measured.
-
- These are mean per-frame *movement* magnitudes, not errors: real motion
- contributes to both columns. The filter is doing its job when the
- numbers drop without the pose visibly lagging the video.
- """
- if not self._raw_cam or not self._out_cam:
- return None
- rc, oc = float(np.mean(self._raw_cam)), float(np.mean(self._out_cam))
- rj, oj = float(np.mean(self._raw_j3d)), float(np.mean(self._out_j3d))
- how = (f"1euro, beta x{self.beta_scale:g}" if self.mode == "1euro"
- else f"window={self.window}, {self.mode}")
- return (
- f"smoothing ({how}) — mean frame-to-frame delta\n"
- f" cam_trans {rc * 1000:7.2f} -> {oc * 1000:7.2f} mm "
- f"({100.0 * (1.0 - oc / rc) if rc > 0 else 0.0:+.0f}%)\n"
- f" joints_3d_local {rj * 1000:7.2f} -> {oj * 1000:7.2f} mm "
- f"({100.0 * (1.0 - oj / rj) if rj > 0 else 0.0:+.0f}%)"
- )
diff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.py
index 41c7562..b1d1505 100644
--- a/instanthmr/visualizer.py
+++ b/instanthmr/visualizer.py
@@ -1,476 +1,103 @@
-"""Rerun-based 3D visualization for the InstantHMR demo.
-
-Logs, per frame:
- - ``camera/image`` : annotated RGB frame (2D skeleton + bbox + a
- text panel showing detector / HMR / render / total ms).
- - ``camera`` : pinhole intrinsics so the image becomes a
- proper frustum in the 3D scene.
- - ``world/persons/...`` : the 70 3D keypoints + skeleton lines per person.
- For an MHR-only student these come from the MHR
- forward pass itself (see ``instanthmr.inference``).
- - ``world/persons/.../mesh``: full MHR body mesh when an MHR backend is
- provided (vertices from a true MHR forward pass).
- Rendered semi-transparent via ``albedo_factor``
- so the keypoints inside the body stay visible.
- - ``timing/detector_ms`` : scalar plot of RF-DETR latency.
- - ``timing/hmr_ms`` : scalar plot of InstantHMR latency.
- - ``timing/render_ms`` : scalar plot of MHR mesh rendering latency.
- - ``timing/total_ms`` : scalar plot of full pipeline latency.
- - ``timing/fps`` : scalar plot of effective frames-per-second.
- - ``contact/person_N/{left,right}``: per-foot contact probability, and the
- feet drawn red while planted (> contact_threshold), for a
- prev-token graph that has a contact head.
-
-``rr.init(..., spawn=True)`` is called automatically so the viewer opens
-on construction (pass ``spawn_viewer=False`` to suppress).
-"""
-
+"""Rerun viewer: the frame with 2D skeletons and timings, the 3D scene, and a latency plot."""
from __future__ import annotations
-import time
-from typing import TYPE_CHECKING, Optional, Sequence
+from typing import Optional, Sequence
import cv2
import numpy as np
from .inference import HMRPrediction
-from .skeleton import edges_for
-
-if TYPE_CHECKING:
- from .mhr_renderer import MHRRenderer
+from .skeleton import SKELETON_EDGES
-#: Base skin tone of the body mesh, before the transparency factor is applied.
MESH_COLOR = (200, 160, 130)
-#: Default mesh opacity. Low enough that the 70 keypoints read through the
-#: body, high enough that the silhouette is still legible.
-MESH_ALPHA = 0.35
-
-#: Reference (teacher) overlay colours — a cool palette against the student's
-#: warm skin tone and green/yellow skeleton, so the two read apart at a glance
-#: even where the bodies overlap.
-REF_MESH_COLOR = (110, 170, 255)
-REF_JOINT_COLOR = (0, 200, 255)
-REF_SKELETON_COLOR = (255, 110, 255)
-
-#: MHR70 keypoints of each foot (ankle, big toe, small toe, heel), in the
-#: [left, right] order of ``HMRPrediction.contact_prob``.
-FOOT_KP = ((13, 15, 16, 17), (14, 18, 19, 20))
-FOOT_NAMES = ("left", "right")
-PLANTED_COLOR = (255, 40, 40)
-FREE_COLOR = (230, 230, 230)
+JOINT_COLOR = (0, 230, 0)
+BONE_COLOR = (255, 230, 0)
+BOX_COLOR = (0, 200, 255)
class RerunVisualizer:
- """Rerun logger for the demo.
-
- Args:
- application_id: Rerun application id.
- spawn_viewer: open the viewer GUI on construction.
- save_path: optional ``.rrd`` recording path.
- mhr_renderer: MHR backend used to skin the body mesh. Any object with
- ``.forward(mhr_params, shape_params)`` and ``.faces`` works; see
- :mod:`instanthmr.mhr_renderer`.
- mesh_alpha: body-mesh opacity in ``[0, 1]``. Applied through Rerun's
- ``albedo_factor`` rather than per-vertex alpha, so one scalar
- controls the whole surface.
- reference: optional :class:`instanthmr.reference.ReferenceTrack` whose
- saved teacher prediction is logged alongside the student's under
- ``world/reference/``, in the cool palette above. It is decoded by
- the SAME ``mhr_renderer``, so a visible difference is the model and
- not the renderer.
- """
-
- def __init__(
- self,
- application_id: str = "instanthmr_demo",
- spawn_viewer: bool = True,
- save_path: str | None = None,
- mhr_renderer: "MHRRenderer | None" = None,
- mesh_alpha: float = MESH_ALPHA,
- reference=None,
- contact_threshold: float = 0.5,
- ):
+ def __init__(self, faces: Optional[np.ndarray] = None, mesh_alpha: float = 0.35, spawn: bool = True,
+ save_path: Optional[str] = None):
import rerun as rr
- self._rr = rr
- self._mhr_renderer = mhr_renderer
- self._mesh_albedo = (
- *MESH_COLOR,
- int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),
- )
- self._reference = reference
- self._ref_albedo = (
- *REF_MESH_COLOR,
- int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),
- )
- self._prev_num_ref = 0
- self.contact_threshold = contact_threshold
- rr.init(application_id, spawn=spawn_viewer)
-
- # SAM3D / MHR convention: right-handed, Y-down camera frame.
- rr.log("world", rr.ViewCoordinates.RIGHT_HAND_Y_DOWN, static=True)
-
+ self.rr = rr
+ self.faces = faces
+ self.albedo = (*MESH_COLOR, int(round(255 * min(max(mesh_alpha, 0.0), 1.0))))
+ rr.init("instanthmr", spawn=spawn)
if save_path:
rr.save(save_path)
-
- self._prev_num_persons = 0
- self._blueprint_sent = False
-
- # ------------------------------------------------------------------
- # Per-frame logging
- # ------------------------------------------------------------------
-
- def log_frame(
- self,
- image_rgb: np.ndarray,
- persons: Sequence[HMRPrediction],
- frame_idx: int,
- timestamp: float | None = None,
- detector_ms: Optional[float] = None,
- hmr_ms: Optional[float] = None,
- total_ms: Optional[float] = None,
- ) -> float:
- """Log one frame of the pipeline to Rerun.
-
- Args:
- image_rgb: full-frame RGB image, ``(H, W, 3)`` uint8.
- persons: per-person HMR predictions for this frame.
- frame_idx: monotonic frame counter (sets the Rerun timeline).
- timestamp: optional wall-clock timestamp (seconds).
- detector_ms: RF-DETR latency for this frame (ms).
- hmr_ms: InstantHMR latency for this frame, summed across
- persons (ms).
- total_ms: full pipeline latency for this frame (ms).
-
- Returns:
- render_ms: time spent running MHR forward passes + Rerun mesh
- logging for this frame (0.0 when no MHRRenderer is set).
- """
- rr = self._rr
- rr.set_time("frame", sequence=frame_idx)
- if timestamp is not None:
- rr.set_time("timestamp", duration=timestamp)
-
- # Per-stage scalar plots
- if detector_ms is not None:
- rr.log("timing/detector_ms", rr.Scalars(float(detector_ms)))
- if hmr_ms is not None:
- rr.log("timing/hmr_ms", rr.Scalars(float(hmr_ms)))
- if total_ms is not None:
- rr.log("timing/total_ms", rr.Scalars(float(total_ms)))
- if total_ms > 0:
- rr.log("timing/fps", rr.Scalars(1000.0 / float(total_ms)))
-
- # Clear stale person entities from previous frames
- n = len(persons)
- for stale in range(n, self._prev_num_persons):
- rr.log(f"world/persons/person_{stale}", rr.Clear(recursive=True))
- self._prev_num_persons = n
+ rr.log("world", rr.ViewCoordinates.RIGHT_HAND_Y_DOWN, static=True)
+ self._n_logged = 0
+ self._layout_sent = False
+
+ def log(self, frame: int, image_rgb: np.ndarray, persons: Sequence[HMRPrediction],
+ timings: dict[str, float]) -> None:
+ rr = self.rr
+ rr.set_time("frame", sequence=frame)
+ for name, ms in timings.items():
+ rr.log(f"timing/{name}", rr.Scalars(float(ms)))
+ for stale in range(len(persons), self._n_logged):
+ rr.log(f"world/person_{stale}", rr.Clear(recursive=True))
+ self._n_logged = len(persons)
+
+ for i, p in enumerate(persons):
+ joints = p.joints_3d_cam
+ rr.log(f"world/person_{i}/joints", rr.Points3D(joints, radii=0.012, colors=JOINT_COLOR))
+ rr.log(f"world/person_{i}/skeleton",
+ rr.LineStrips3D(joints[np.array(SKELETON_EDGES)], colors=BONE_COLOR, radii=0.004))
+ if self.faces is not None and p.vertices_local is not None:
+ rr.log(f"world/person_{i}/mesh", rr.Mesh3D(vertex_positions=p.vertices_cam,
+ triangle_indices=self.faces, albedo_factor=self.albedo))
h, w = image_rgb.shape[:2]
-
- # Before the no-detection early-out below: the reference is independent
- # of whether the student found anybody, and seeing it alone on a frame
- # the student missed is exactly the failure worth looking at.
- if self._reference is not None:
- self._log_reference(frame_idx)
-
- # Always log the image — even when no person was detected — so the
- # viewer keeps showing the camera feed and the timing HUD.
- render_ms = 0.0
- if not persons:
- blank = self._draw_overlay(image_rgb, [], detector_ms, hmr_ms, render_ms, total_ms)
- rr.log("camera/image", rr.Image(blank))
- rr.log("timing/render_ms", rr.Scalars(0.0))
- return render_ms
-
- # --- MHR mesh rendering (timed) ----------------------------------
- if self._mhr_renderer is not None:
- t0 = time.perf_counter()
- for idx, person in enumerate(persons):
- self._log_person_mesh(idx, person)
- render_ms = (time.perf_counter() - t0) * 1000.0
-
- rr.log("timing/render_ms", rr.Scalars(render_ms))
-
- for idx, person in enumerate(persons):
- self._log_person(idx, person)
-
- annotated = self._draw_overlay(
- image_rgb, persons, detector_ms, hmr_ms, render_ms, total_ms,
- self.contact_threshold,
- )
- rr.log("camera/image", rr.Image(annotated))
-
- # Pinhole camera in the 3D scene — full-frame focal so the image
- # plane sits in front of the joints, not on top of them.
- first = persons[0]
- fx, fy = float(first.focal_length[0]), float(first.focal_length[1])
- cx, cy = w / 2.0, h / 2.0
- depth = float(first.joints_3d_cam[:, 2].mean())
- plane_dist = max(depth * 0.25, 0.2)
-
- rr.log(
- "camera",
- rr.Pinhole(
- width=w,
- height=h,
- focal_length=[fx, fy],
- principal_point=[cx, cy],
- image_plane_distance=plane_dist,
- ),
- )
-
- if not self._blueprint_sent:
- self._send_blueprint(first)
- self._blueprint_sent = True
-
- return render_ms
-
- # ------------------------------------------------------------------
- # Helpers
- # ------------------------------------------------------------------
-
- def _log_person(self, idx: int, person: HMRPrediction) -> None:
- """Log the 70 3D keypoints and the skeleton wiring for one person.
-
- For an MHR-only student ``joints_3d_cam`` was regressed from the MHR
- skeleton that also produced the mesh, so the points sit exactly inside
- the surface logged by :meth:`_log_person_mesh`.
- """
- rr = self._rr
- path = f"world/persons/person_{idx}"
-
- joints = person.joints_3d_cam
- rr.log(
- f"{path}/joints",
- rr.Points3D(positions=joints, radii=0.012, colors=[0, 230, 0]),
- )
-
- edges = edges_for(joints.shape[0])
- if edges:
- lines = [
- [joints[i].tolist(), joints[j].tolist()]
- for i, j in edges
- ]
- rr.log(
- f"{path}/skeleton",
- rr.LineStrips3D(lines, colors=[255, 230, 0], radii=0.004),
- )
-
- if person.contact_prob is not None:
- for name, kp, prob in zip(FOOT_NAMES, FOOT_KP, person.contact_prob):
- planted = prob > self.contact_threshold
- rr.log(f"{path}/feet/{name}", rr.Points3D(
- positions=joints[list(kp)], radii=0.03 if planted else 0.015,
- colors=[PLANTED_COLOR if planted else FREE_COLOR]))
- rr.log(f"contact/person_{idx}/{name}", rr.Scalars(float(prob)))
-
- def _log_reference(self, frame_idx: int) -> None:
- """Log the saved teacher prediction for this frame under ``world/reference``.
-
- Kept in a sibling entity tree rather than mixed into
- ``world/persons/`` so the viewer can toggle either side on its own.
- """
- rr = self._rr
- persons = self._reference.persons_at(frame_idx)
-
- for stale in range(len(persons), self._prev_num_ref):
- rr.log(f"world/reference/person_{stale}", rr.Clear(recursive=True))
- self._prev_num_ref = len(persons)
-
- for idx, ref in enumerate(persons):
- path = f"world/reference/person_{idx}"
- joints = ref.joints_3d_cam
- rr.log(
- f"{path}/joints",
- rr.Points3D(positions=joints, radii=0.012,
- colors=list(REF_JOINT_COLOR)),
- )
- edges = edges_for(joints.shape[0])
- if edges:
- rr.log(
- f"{path}/skeleton",
- rr.LineStrips3D(
- [[joints[i].tolist(), joints[j].tolist()] for i, j in edges],
- colors=list(REF_SKELETON_COLOR), radii=0.004,
- ),
- )
- # The teacher stored parameters, not vertices; decoding them with
- # the student's own rig is what keeps the two meshes comparable.
- if self._mhr_renderer is not None:
- verts = self._mhr_renderer.forward(
- ref.mhr_params, ref.shape_params,
- ) + ref.cam_trans
- rr.log(
- f"{path}/mesh",
- rr.Mesh3D(
- vertex_positions=verts,
- triangle_indices=self._mhr_renderer.faces,
- albedo_factor=self._ref_albedo,
- ),
- )
-
- def _log_person_mesh(self, idx: int, person: HMRPrediction) -> None:
- """Run MHR forward pass and log the resulting mesh vertices via Rerun."""
- assert self._mhr_renderer is not None
-
- # Decode mhr_params + shape_params → body-space vertices.
- verts_local = self._mhr_renderer.forward(
- person.mhr_params, person.shape_params,
- ) # (V, 3), body-centred
-
- # Shift from body space to camera space using InstantHMR's cam_trans.
- verts_cam = verts_local + person.cam_trans # (V, 3) broadcast
-
- faces = self._mhr_renderer.faces # (F, 3) int32, shared across frames
-
- # Tint + transparency come from `albedo_factor`, a single RGBA multiplier
- # for the whole surface. Per-vertex colours would work too, but the
- # viewer multiplies the two, so keeping the alpha in one place makes the
- # `mesh_alpha` knob mean exactly what it says.
- self._rr.log(
- f"world/persons/person_{idx}/mesh",
- self._rr.Mesh3D(
- vertex_positions=verts_cam,
- triangle_indices=faces,
- albedo_factor=self._mesh_albedo,
- ),
- )
-
- @staticmethod
- def _draw_overlay(
- image_rgb: np.ndarray,
- persons: Sequence[HMRPrediction],
- detector_ms: Optional[float],
- hmr_ms: Optional[float],
- render_ms: float,
- total_ms: Optional[float],
- contact_threshold: float = 0.5,
- ) -> np.ndarray:
- """Draw per-person bbox + 2D skeleton + a timing HUD onto the frame."""
- out = image_rgb.copy()
- h, w = out.shape[:2]
-
- for person in persons:
- x1, y1, x2, y2 = person.bbox.astype(int)
- cv2.rectangle(out, (x1, y1), (x2, y2), (0, 200, 255), 2)
- label = f"{person.confidence:.2f}"
- (tw, th), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
- cv2.rectangle(out, (x1, y1 - th - 6), (x1 + tw + 4, y1), (0, 200, 255), -1)
- cv2.putText(
- out, label, (x1 + 2, y1 - 4),
- cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1,
- )
-
- j2d = person.joints_2d
- valid = (
- (j2d[:, 0] >= 0) & (j2d[:, 0] < w)
- & (j2d[:, 1] >= 0) & (j2d[:, 1] < h)
- )
-
- for i, j in edges_for(j2d.shape[0]):
- if valid[i] and valid[j]:
- pt1 = tuple(j2d[i].astype(int))
- pt2 = tuple(j2d[j].astype(int))
- cv2.line(out, pt1, pt2, (255, 230, 0), 2)
-
- for k, (x, y) in enumerate(j2d):
- if valid[k]:
- cv2.circle(out, (int(x), int(y)), 3, (0, 230, 0), -1)
-
- if person.contact_prob is not None:
- r = max(4, int(round(0.012 * max(h, w))))
- for name, kp, prob in zip(FOOT_NAMES, FOOT_KP, person.contact_prob):
- planted = prob > contact_threshold
- for k in kp:
- if valid[k]:
- cv2.circle(out, tuple(j2d[k].astype(int)), r,
- PLANTED_COLOR if planted else FREE_COLOR,
- -1 if planted else 2)
- k = kp[0]
- if valid[k]:
- cv2.putText(out, f"{name[0].upper()} {prob:.2f}",
- (int(j2d[k, 0]) + r + 2, int(j2d[k, 1]) - r),
- cv2.FONT_HERSHEY_SIMPLEX, max(0.5, r / 12),
- PLANTED_COLOR if planted else FREE_COLOR, 2, cv2.LINE_AA)
-
- # ---- Timing HUD (top-left, semi-transparent black panel) ----
- lines: list[str] = []
- if detector_ms is not None:
- lines.append(f"RF-DETR : {detector_ms:6.1f} ms")
- if hmr_ms is not None:
- lines.append(f"InstantHMR: {hmr_ms:6.1f} ms")
- if render_ms > 0.0:
- lines.append(f"MHR mesh : {render_ms:6.1f} ms")
- if total_ms is not None:
- fps = 1000.0 / total_ms if total_ms > 0 else 0.0
- lines.append(f"Total : {total_ms:6.1f} ms ({fps:5.1f} fps)")
- if lines:
- font = cv2.FONT_HERSHEY_SIMPLEX
- scale = max(0.5, min(w, h) / 1500.0)
- thick = max(1, int(round(scale * 1.5)))
- sizes = [
- cv2.getTextSize(s, font, scale, thick)[0] for s in lines
- ]
- line_h = max(s[1] for s in sizes) + 8
- box_w = max(s[0] for s in sizes) + 16
- box_h = line_h * len(lines) + 8
-
- overlay = out.copy()
- cv2.rectangle(overlay, (8, 8), (8 + box_w, 8 + box_h), (0, 0, 0), -1)
- cv2.addWeighted(overlay, 0.55, out, 0.45, 0, out)
-
- y = 8 + line_h
- for s in lines:
- cv2.putText(
- out, s, (16, y),
- font, scale, (255, 255, 255), thick, cv2.LINE_AA,
- )
- y += line_h
-
- return out
-
- def _send_blueprint(self, first: HMRPrediction) -> None:
- """Send the blueprint: camera image | 3D scene | timings (+ contact)."""
+ rr.log("camera/image", rr.Image(draw_overlay(image_rgb, persons, timings)))
+ if persons:
+ depth = float(np.median([p.cam_trans[2] for p in persons]))
+ rr.log("camera", rr.Pinhole(width=w, height=h, focal_length=persons[0].focal_length,
+ principal_point=[w / 2, h / 2], image_plane_distance=max(0.25 * depth, 0.2)))
+ if not self._layout_sent:
+ self._send_layout(persons[0])
+ self._layout_sent = True
+
+ def _send_layout(self, first: HMRPrediction) -> None:
import rerun.blueprint as rrb
- rr = self._rr
-
- joints = first.joints_3d_cam
- center = joints.mean(axis=0)
- extent = float(np.abs(joints - center).max())
- cam_distance = max(extent * 4.0, 1.5)
- # Y-down convention: place the orbit camera "in front" along -Z.
- eye_pos = [
- float(center[0]),
- float(center[1]),
- float(center[2] - cam_distance),
- ]
- plots = [rrb.TimeSeriesView(origin="timing", name="Latency (ms)")]
- if first.contact_prob is not None:
- plots.append(rrb.TimeSeriesView(origin="contact",
- name="Foot contact probability"))
- blueprint = rrb.Blueprint(
- rrb.Horizontal(
- rrb.Vertical(
- rrb.Spatial2DView(origin="camera/image", name="Camera"),
- *plots,
- row_shares=[3] + [1] * len(plots),
- ),
- rrb.Spatial3DView(
- origin="/",
- name="3D Scene",
- eye_controls=rrb.EyeControls3D(
- position=eye_pos,
- look_target=[float(center[0]), float(center[1]), float(center[2])],
- eye_up=[0.0, -1.0, 0.0],
- kind=rrb.Eye3DKind.Orbital,
- tracking_entity="world/persons/person_0",
- ),
- background=rrb.Background(color=[25, 25, 25]),
- ),
- column_shares=[2, 3],
- ),
- )
- rr.send_blueprint(blueprint)
+ centre = first.joints_3d_cam.mean(0)
+ distance = max(4 * float(np.abs(first.joints_3d_cam - centre).max()), 1.5)
+ self.rr.send_blueprint(rrb.Blueprint(rrb.Horizontal(
+ rrb.Vertical(rrb.Spatial2DView(origin="camera/image", name="Camera"),
+ rrb.TimeSeriesView(origin="timing", name="Latency (ms)"), row_shares=[3, 1]),
+ rrb.Spatial3DView(origin="/", name="3D", background=rrb.Background(color=[25, 25, 25]),
+ eye_controls=rrb.EyeControls3D(
+ position=[float(centre[0]), float(centre[1]), float(centre[2] - distance)],
+ look_target=centre.tolist(), eye_up=[0.0, -1.0, 0.0], kind=rrb.Eye3DKind.Orbital)),
+ column_shares=[2, 3])))
+
+
+def draw_overlay(image_rgb: np.ndarray, persons: Sequence[HMRPrediction], timings: dict[str, float]) -> np.ndarray:
+ """The frame with each person's box and 2D skeleton, and the stage timings top left."""
+ out = image_rgb.copy()
+ h, w = out.shape[:2]
+ thick = max(2, round(min(h, w) / 400))
+ for p in persons:
+ x1, y1, x2, y2 = p.bbox.astype(int)
+ cv2.rectangle(out, (x1, y1), (x2, y2), BOX_COLOR, thick)
+ pts = p.joints_2d.astype(int)
+ for a, b in SKELETON_EDGES:
+ cv2.line(out, tuple(pts[a]), tuple(pts[b]), BONE_COLOR, thick, cv2.LINE_AA)
+ for pt in pts:
+ cv2.circle(out, tuple(pt), thick + 1, JOINT_COLOR, -1, cv2.LINE_AA)
+
+ lines = [f"{name:<9s}{ms:6.1f} ms" for name, ms in timings.items()]
+ total = sum(timings.values())
+ if total > 0:
+ lines.append(f"{'total':<9s}{total:6.1f} ms ({1000 / total:.0f} fps)")
+ scale = max(0.5, min(h, w) / 1200)
+ line_h = int(28 * scale)
+ panel = out.copy()
+ cv2.rectangle(panel, (8, 8), (int(300 * scale) + 16, 16 + line_h * len(lines)), (0, 0, 0), -1)
+ cv2.addWeighted(panel, 0.55, out, 0.45, 0, out)
+ for k, line in enumerate(lines):
+ cv2.putText(out, line, (16, 8 + line_h * (k + 1)), cv2.FONT_HERSHEY_SIMPLEX, scale * 0.7,
+ (255, 255, 255), max(1, round(scale * 1.5)), cv2.LINE_AA)
+ return out
diff --git a/models/README.md b/models/README.md
deleted file mode 100644
index e9541ce..0000000
--- a/models/README.md
+++ /dev/null
@@ -1,140 +0,0 @@
----
-license: other
-license_name: sam-license
-license_link: https://github.com/facebookresearch/sam-3d-body/blob/main/LICENSE
-library_name: onnx
-pipeline_tag: keypoint-detection
-tags:
- - human-pose-estimation
- - human-mesh-recovery
- - 3d-pose-estimation
- - onnx
- - mhr
- - sam3d
- - repvit
- - cliff
-base_model: facebook/sam-3d-body-dinov3
----
-
-# InstantHMR — model weights
-
-![InstantHMR demo](instanthmr.gif)
-
-A lightweight, ONNX-exportable MHR body regressor for **3D human pose
-estimation and mesh recovery**: a RepViT-M1.5 backbone + a 9-token
-cross-attention decoder + CLIFF camera conditioning.
-
-Trained on the released ground-truth annotations of
-[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)
-— the human MHR fits used to build SAM 3D Body — predicting per-person MHR
-parameters and 70 keypoints from a single 224×224 crop.
-
-Demo, training, and inference code live at the main InstantHMR
-repository: <https://github.com/mohamdev/InstantHMR>.
-
-## Files
-
-| File | Size | Purpose |
-|---|---|---|
-| `instanthmr.onnx` | ~77 MB | fp16 ONNX export — what the demo and `InstantHMR(...)` wrapper load. |
-| `instanthmr.pth` | ~461 MB | PyTorch checkpoint for fine-tuning. |
-
-Most users only need `instanthmr.onnx`. The `.pth` is included for people
-who want to continue training, swap the head, or re-export with different
-options — see [`training/README.md`](../training/README.md).
-
-## ONNX I/O
-
-- **Inputs**
- - `image` — `(N, 3, 224, 224)` float32, ImageNet-normalised RGB.
- - `cliff_cond` — `(N, 3)` float32, `[cx_norm, cy_norm, b_scale]` of the
- detector bbox in the **full** frame (`cx_norm, cy_norm ∈ [-1, 1]`,
- `b_scale = max(bw, bh) / max(W, H)`).
-- **Outputs**
- - `mhr_params` — `(N, 204)` MHR pose + scale parameters.
- - `shape_params` — `(N, 45)` MHR identity blendshapes.
- - `cam_trans` — `(N, 3)` body translation in camera frame (metres).
- - `joints_2d` — `(N, 70, 2)` keypoints in normalised crop coords `[-1, 1]`.
- - `joints_3d` — `(N, 70, 3)` body-centred 3D keypoints (metres, Y-down).
-
-The 70 keypoints follow the MHR70 ordering — see
-[`instanthmr/skeleton.py`](https://github.com/mohamdev/InstantHMR/blob/main/instanthmr/skeleton.py)
-for joint names and skeleton edges.
-
-## Quick start (ONNX)
-
-Minimal end-to-end inference with `onnxruntime`, no extra dependencies:
-
-```python
-import cv2
-import numpy as np
-import onnxruntime as ort
-
-IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
-IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
-
-# 1. Load the ONNX session (CUDA if available, otherwise CPU).
-sess = ort.InferenceSession(
- "instanthmr.onnx",
- providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
-)
-
-# 2. Load an RGB image and a person bbox [x1, y1, x2, y2].
-image = cv2.cvtColor(cv2.imread("person.jpg"), cv2.COLOR_BGR2RGB)
-H, W = image.shape[:2]
-x1, y1, x2, y2 = 120, 40, 380, 600 # replace with your detector output
-
-# 3. Square 1.2x crop around the bbox, resized to 224x224.
-cx, cy = (x1 + x2) / 2, (y1 + y2) / 2
-size = max(x2 - x1, y2 - y1) * 1.2
-sx1, sy1 = int(cx - size / 2), int(cy - size / 2)
-sx2, sy2 = int(cx + size / 2), int(cy + size / 2)
-pad = [max(0, -sy1), max(0, sy2 - H), max(0, -sx1), max(0, sx2 - W)]
-patch = image[max(0, sy1):min(H, sy2), max(0, sx1):min(W, sx2)]
-patch = cv2.copyMakeBorder(patch, *pad, cv2.BORDER_CONSTANT, value=0)
-crop = cv2.resize(patch, (224, 224), interpolation=cv2.INTER_LINEAR)
-
-# 4. Normalise + CHW.
-img = (crop.astype(np.float32) / 255.0 - IMAGENET_MEAN) / IMAGENET_STD
-img = np.transpose(img, (2, 0, 1))[None] # (1, 3, 224, 224)
-
-# 5. CLIFF conditioning vector (full-frame coords).
-cliff = np.array([
- 2.0 * cx / W - 1.0,
- 2.0 * cy / H - 1.0,
- max(x2 - x1, y2 - y1) / max(W, H),
-], dtype=np.float32)[None] # (1, 3)
-
-# 6. Run the model.
-mhr_params, shape_params, cam_trans, joints_2d, joints_3d = sess.run(
- None, {"image": img, "cliff_cond": cliff}
-)
-
-print(joints_3d.shape) # (1, 70, 3) — body-centred metres, Y-down
-print(joints_2d.shape) # (1, 70, 2) — normalised crop coords [-1, 1]
-print(mhr_params.shape) # (1, 204) — feed into Meta's MHR module for a mesh
-```
-
-For a full pipeline (RF-DETR person detector + InstantHMR + Rerun
-visualiser), use the [`PosePipeline`](https://github.com/mohamdev/InstantHMR/blob/main/instanthmr/pipeline.py)
-helper from the main repository — `pip install -r requirements.txt` then
-`python demo.py --image path/to/photo.jpg`.
-
-## Body mesh
-
-InstantHMR does **not** run the MHR mesh decoder — rendering the full
-mesh is resource-heavy and the default pipeline stays lightweight. The
-raw `mhr_params` and `shape_params` are exposed so you can feed them
-into Meta's MHR TorchScript module from the original
-[SAM 3D Body release](https://github.com/facebookresearch/sam-3d-body)
-when you need vertices.
-
-## License
-
-These weights are released under the **SAM license**, since InstantHMR
-is trained on `facebook/sam-3d-body-dataset` labels. Refer to
-<https://github.com/facebookresearch/sam-3d-body> for the full license
-text and applicable use restrictions.
-
-The accompanying source code in the InstantHMR repository is released
-separately under the Apache License 2.0.
diff --git a/models/instanthmr.gif b/models/instanthmr.gif
deleted file mode 100644
index 564b817..0000000
Binary files a/models/instanthmr.gif and /dev/null differ
diff --git a/requirements-mhr.txt b/requirements-mhr.txt
deleted file mode 100644
index 8951b7e..0000000
--- a/requirements-mhr.txt
+++ /dev/null
@@ -1,43 +0,0 @@
-# MHR body mesh rendering — install TOGETHER with requirements.txt
-#
-# IMPORTANT: always install both files in a single pip call so the resolver
-# can pick a torch version compatible with rfdetr, torchvision, AND
-# pymomentum-gpu at once. Installing them separately causes pip to first
-# choose a torch version for rfdetr and then downgrade it for pymomentum,
-# leaving torchvision broken.
-#
-# pip install -r requirements.txt -r requirements-mhr.txt
-#
-# Prerequisites
-# -------------
-# * Python >= 3.12 (pymomentum has no Python 3.11 wheels; recreate the
-# conda env if needed: conda create -n instanthmr python=3.12 -y)
-# * CUDA toolkit on the system path (recommended for real-time use)
-#
-# After installing, also download the MHR body-model assets:
-#
-# curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
-# unzip assets.zip -d models/mhr_assets
-#
-# Then run:
-#
-# python demo.py --image photo.jpg --mhr-assets models/mhr_assets
-
-# pymomentum — Meta's momentum geometry library (C++ Python bindings).
-# This package brings its own torch constraint; do NOT add a separate
-# torch>=X line here or pip will resolve torch twice and downgrade it.
-#
-# WARNING: "pip install pymomentum" installs an UNRELATED legacy SMS library
-# (pyMomentum v0.1.x by MomentumAS). The correct package names are below.
-#
-# Platform markers handle the selection automatically:
-# Linux / Windows → pymomentum-gpu (NVIDIA GPU)
-# macOS → pymomentum-cpu (Apple Silicon / Intel)
-pymomentum-gpu ; platform_system != "Darwin"
-pymomentum-cpu ; platform_system == "Darwin"
-#
-# CPU-only Linux / Windows: comment out pymomentum-gpu above and uncomment:
-# pymomentum-cpu ; platform_system != "Darwin"
-
-# MHR body model (Meta's Momentum Human Rig)
-mhr @ git+https://github.com/facebookresearch/MHR.git
diff --git a/requirements.txt b/requirements.txt
index 1644ca1..1821d1c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,32 +1,7 @@
-# Recommended install path is `python install.py`, which picks the right
-# torch + onnxruntime wheels for your hardware (Linux + CUDA, Mac, or CPU).
-# This file is a portable fallback that uses the CPU wheel of torch and the
-# stock CPU build of onnxruntime — it works everywhere but leaves a lot of
-# performance on the table on NVIDIA GPUs and Apple Silicon.
-
-# ----- inference + viz (required by demo.py) -----
-numpy>=1.24
-opencv-python>=4.8
-pillow>=10
-rfdetr>=1.0
-rerun-sdk>=0.21
-
-# Pulled in transitively by rfdetr; on Linux + CUDA you almost certainly
-# want a CUDA wheel from the pytorch index instead — see install.py.
-torch>=2.2
-torchvision
-
-# Mac ships CoreML EP inside plain onnxruntime; Linux GPU wants
-# onnxruntime-gpu (which install.py picks). The CPU wheel below is the
-# safe fallback.
-onnxruntime>=1.19
-
-# ----- MHR body mesh rendering -----
-# Optional. Requires Python >= 3.12 and a separate setup step.
-# See requirements-mhr.txt and the "MHR body mesh" section of README.md.
-
-# ----- training / annotation (optional, for reproduction) -----
-# timm>=1.0
-# tqdm
-# loguru
-# (For annotation: install Meta's sam-3d-body separately — see docs/annotation.md)
+numpy
+opencv-python
+huggingface_hub
+rerun-sdk==0.31.*
+rfdetr>=1.6
+onnxruntime-gpu[cuda,cudnn]==1.26.* ; platform_system != "Darwin"
+onnxruntime==1.26.* ; platform_system == "Darwin"
diff --git a/tools/annotate_dataset.py b/tools/annotate_dataset.py
deleted file mode 100755
index 7298948..0000000
--- a/tools/annotate_dataset.py
+++ /dev/null
@@ -1,876 +0,0 @@
-#!/usr/bin/env python3
-"""
-Cloud-grade annotation pipeline for CLIFF-style student distillation.
-
-Scans a root directory tree for images, runs YOLO person detection + SAM3D
-teacher inference, and saves tight 224×224 crops alongside full-frame-aware
-annotations (.npz) ready for CLIFF-style training.
-
-Output layout:
- <output_root>/
- images/ <datasetName>_<imageStem>_p<personID>.jpg (224×224 crop)
- annotations/ <datasetName>_<imageStem>_p<personID>.npz
-
-NPZ payload (per crop):
- orig_shape (2,) [H, W] of the full uncropped image
- bbox (4,) [x1, y1, x2, y2] raw YOLO detection in original image coords
- (used by CLIFF student for bbox_center / bbox_scale conditioning)
- bbox_square (4,) [x1, y1, x2, y2] expanded square crop (1.2×) used for 224×224 resize
- cam_focal_length (2,) [fx, fy] where f = sqrt(H² + W²)
- cam_trans (3,) [tx, ty, tz] from SAM3D (computed with raw bbox, full-frame perspective)
- mhr_model_params (204,) pose + scale params
- shape_params (45,) identity blendshapes
- joints_3d (70, 3) regressed 3D joints (Y-down camera convention, meters)
- joints_2d (70, 2) projected into ORIGINAL full-frame pixel space
-
-Resume-safe: existing annotations are skipped automatically.
-
-Usage:
- python tools/annotate_dataset_sam3d_cloud.py \\
- --root_dir /data/training_images \\
- --output_dir /data/cliff_annotations \\
- --confidence 0.7
-
- # Quick diverse subset (50k crops sampled across all datasets):
- python tools/annotate_dataset_sam3d_cloud.py \\
- --root_dir /data/training_images \\
- --output_dir /data/cliff_annotations \\
- --shuffle --max_samples 50000
-
- # Structure of --root_dir:
- # /data/training_images/
- # Harmony4D/ *.jpg *.png ...
- # COCO/ *.jpg ...
- # Synthetic/ *.png ...
-"""
-
-from __future__ import annotations
-
-import argparse
-import gc
-import math
-import os
-import random
-import signal
-import sys
-import time
-from contextlib import contextmanager
-from pathlib import Path
-
-# Ensure project root is importable when running from repo root.
-_PROJECT_ROOT = Path(__file__).resolve().parent.parent
-if str(_PROJECT_ROOT) not in sys.path:
- sys.path.insert(0, str(_PROJECT_ROOT))
-
-import cv2
-import numpy as np
-import torch
-from loguru import logger
-from tqdm import tqdm
-
-
-# ---------------------------------------------------------------------------
-# Constants
-# ---------------------------------------------------------------------------
-IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".bmp", ".webp"}
-CROP_SIZE = 224
-
-
-# ---------------------------------------------------------------------------
-# Rerun helpers
-# ---------------------------------------------------------------------------
-def setup_rerun_blueprint():
- """Send a 3-panel Rerun blueprint: raw image | overlay | 3D mesh."""
- import rerun as rr
- import rerun.blueprint as rrb
-
- blueprint = rrb.Blueprint(
- rrb.Horizontal(
- rrb.Spatial2DView(
- origin="raw_image",
- name="Raw Image",
- ),
- rrb.Spatial2DView(
- origin="overlay/image",
- name="Overlay",
- ),
- rrb.Spatial3DView(
- origin="scene_3d",
- name="3D Mesh",
- background=rrb.Background(color=[25, 25, 25]),
- ),
- ),
- )
- rr.send_blueprint(blueprint)
-
-
-_MAX_RERUN_PERSONS = 10 # upper bound for clearing stale entities
-
-
-def log_to_rerun(
- image_rgb: np.ndarray,
- detections: list[dict],
- sam3d_outputs: list[dict | None],
- faces: np.ndarray | None,
-) -> None:
- """Log one image with all its person detections to the Rerun viewer.
-
- All entities are logged as static so Rerun only ever holds one frame in
- memory — previous data is overwritten, not accumulated.
-
- Args:
- image_rgb: Full uncropped image (H, W, 3), uint8 RGB.
- detections: YOLO detection dicts with ``"bbox"`` keys.
- sam3d_outputs: Parallel list of raw SAM3D output dicts (may contain None).
- faces: Mesh triangle indices from the SAM3D teacher.
- """
- import rerun as rr
-
- h, w = image_rgb.shape[:2]
-
- # Clear all person entities from the previous frame to avoid stale meshes
- for i in range(_MAX_RERUN_PERSONS):
- rr.log(f"scene_3d/person_{i}", rr.Clear(recursive=True), static=True)
- rr.log("scene_3d/camera", rr.Clear(recursive=True), static=True)
-
- # Panel 1: raw image
- rr.log("raw_image", rr.Image(image_rgb), static=True)
-
- overlay = image_rgb.copy()
- first_cam_t = None
- first_fx, first_fy = None, None
- first_joints_cam = None
-
- for person_id, (det, sam3d_out) in enumerate(zip(detections, sam3d_outputs)):
- if sam3d_out is None:
- continue
-
- # Extract outputs
- cam_t = sam3d_out.get("pred_cam_t")
- if cam_t is not None and hasattr(cam_t, "cpu"):
- cam_t = cam_t.cpu().numpy()
- cam_t = np.asarray(cam_t, dtype=np.float32).ravel()[:3] if cam_t is not None else np.zeros(3, dtype=np.float32)
-
- focal = sam3d_out.get("focal_length")
- if focal is not None and hasattr(focal, "cpu"):
- focal = focal.cpu().numpy()
- if focal is not None:
- focal = np.asarray(focal, dtype=np.float32).ravel()
- fx = float(focal[0])
- fy = float(focal[-1]) # handles both (1,) and (2,)
- else:
- f = math.sqrt(h * h + w * w)
- fx, fy = f, f
- cx, cy = w / 2.0, h / 2.0
-
- if first_cam_t is None:
- first_cam_t = cam_t
- first_fx, first_fy = fx, fy
-
- # Vertices
- verts = sam3d_out.get("pred_vertices", sam3d_out.get("vertices", sam3d_out.get("verts")))
- if verts is not None:
- if hasattr(verts, "cpu"):
- verts = verts.cpu().numpy()
- verts = np.asarray(verts, dtype=np.float32)
- if verts.ndim == 3 and verts.shape[0] == 1:
- verts = verts[0]
-
- verts_cam = verts + cam_t
-
- # Panel 2: project mesh onto overlay
- z = verts_cam[:, 2]
- valid = z > 0.01
- u = (fx * verts_cam[valid, 0] / z[valid] + cx).astype(int)
- v = (fy * verts_cam[valid, 1] / z[valid] + cy).astype(int)
- mask = (u >= 0) & (u < w) & (v >= 0) & (v < h)
- for px, py in zip(u[mask], v[mask]):
- cv2.circle(overlay, (px, py), 1, (0, 255, 128), -1)
-
- # Panel 3: 3D mesh
- person_path = f"scene_3d/person_{person_id}"
- if faces is not None:
- rr.log(
- f"{person_path}/mesh",
- rr.Mesh3D(
- vertex_positions=verts_cam,
- triangle_indices=faces,
- albedo_factor=[180, 200, 220, 200],
- ),
- static=True,
- )
- else:
- rr.log(
- f"{person_path}/mesh",
- rr.Points3D(positions=verts_cam, radii=0.003, colors=[180, 200, 220]),
- static=True,
- )
-
- # 3D joints
- joints_3d = sam3d_out.get("pred_keypoints_3d")
- if joints_3d is not None:
- if hasattr(joints_3d, "cpu"):
- joints_3d = joints_3d.cpu().numpy()
- joints_3d = np.asarray(joints_3d, dtype=np.float32)
- if joints_3d.ndim == 3 and joints_3d.shape[0] == 1:
- joints_3d = joints_3d[0]
- joints_cam = joints_3d + cam_t
- if first_joints_cam is None:
- first_joints_cam = joints_cam
- rr.log(
- f"scene_3d/person_{person_id}/joints",
- rr.Points3D(positions=joints_cam, radii=0.008, colors=[0, 255, 0]),
- static=True,
- )
-
- rr.log("overlay/image", rr.Image(overlay), static=True)
-
- # Log Pinhole camera in the 3D scene so the image appears as a properly
- # sized frustum. Resolution must match the full frame (not the 224 crop)
- # because cam_focal_length is computed from sqrt(H² + W²).
- if first_cam_t is not None:
- mesh_depth = float(first_joints_cam[:, 2].mean()) if first_joints_cam is not None else None
- image_plane_dist = max(mesh_depth * 0.25, 0.2) if mesh_depth else 0.5
-
- rr.log(
- "scene_3d/camera",
- rr.Pinhole(
- resolution=[w, h],
- focal_length=[first_fx, first_fy],
- principal_point=[w / 2.0, h / 2.0],
- image_plane_distance=image_plane_dist,
- ),
- static=True,
- )
- rr.log("scene_3d/camera/image", rr.Image(image_rgb), static=True)
-
-
-# ---------------------------------------------------------------------------
-# Timeout helper (POSIX — uses SIGALRM)
-# ---------------------------------------------------------------------------
-class InferenceTimeout(Exception):
- pass
-
-
-@contextmanager
-def time_limit(seconds: float):
- """Raise InferenceTimeout if the block takes longer than *seconds*."""
- def _handler(signum, frame):
- raise InferenceTimeout(f"Inference exceeded {seconds}s timeout")
-
- old = signal.signal(signal.SIGALRM, _handler)
- signal.setitimer(signal.ITIMER_REAL, seconds)
- try:
- yield
- finally:
- signal.setitimer(signal.ITIMER_REAL, 0)
- signal.signal(signal.SIGALRM, old)
-
-
-# ---------------------------------------------------------------------------
-# Image discovery
-# ---------------------------------------------------------------------------
-def discover_images(root_dir: Path) -> list[tuple[str, Path]]:
- """Walk *root_dir* and return ``(dataset_name, image_path)`` pairs.
-
- ``dataset_name`` is the name of the first-level subdirectory under
- *root_dir*. Images directly in *root_dir* get dataset_name ``"root"``.
- """
- entries: list[tuple[str, Path]] = []
- root_dir = root_dir.resolve()
-
- for path in sorted(root_dir.rglob("*")):
- if not path.is_file():
- continue
- if path.suffix.lower() not in IMAGE_EXTENSIONS:
- continue
-
- # Derive dataset name from the first child directory
- try:
- relative = path.relative_to(root_dir)
- parts = relative.parts
- dataset_name = parts[0] if len(parts) > 1 else "root"
- except ValueError:
- dataset_name = "root"
-
- entries.append((dataset_name, path))
-
- return entries
-
-
-# ---------------------------------------------------------------------------
-# YOLO detector (standalone, no pipeline dependency)
-# ---------------------------------------------------------------------------
-class YOLOPersonDetector:
- """Thin wrapper around ultralytics YOLO for person detection."""
-
- def __init__(
- self,
- model_name: str = "yolo11n.pt",
- device: str = "cuda:0",
- confidence: float = 0.7,
- ):
- from ultralytics import YOLO
-
- self._model = YOLO(model_name)
- self._device = device
- self._confidence = confidence
-
- def detect(self, image_rgb: np.ndarray) -> list[dict]:
- """Return list of ``{"bbox": [x1,y1,x2,y2], "confidence": float}``."""
- results = self._model(
- image_rgb,
- device=self._device,
- classes=[0], # COCO person
- conf=self._confidence,
- verbose=False,
- )
- detections = []
- for r in results:
- if r.boxes is None:
- continue
- for box, conf in zip(
- r.boxes.xyxy.cpu().numpy(),
- r.boxes.conf.cpu().numpy(),
- ):
- detections.append({
- "bbox": box.astype(np.float32),
- "confidence": float(conf),
- })
- detections.sort(key=lambda d: d["confidence"], reverse=True)
- return detections
-
-
-# ---------------------------------------------------------------------------
-# SAM3D teacher (uses the estimator directly for max control)
-# ---------------------------------------------------------------------------
-class SAM3DTeacher:
- """Loads the SAM3D teacher and exposes a simple per-crop inference API.
-
- We call the ``SAM3DBodyEstimator`` directly (not via the pipeline wrapper)
- so that we have access to the raw output dict with ``pred_cam_t``,
- ``focal_length``, ``mhr_model_params``, etc.
- """
-
- def __init__(
- self,
- checkpoint_path: str,
- mhr_path: str,
- hf_repo_id: str = "facebook/sam-3d-body-dinov3",
- device: str = "cuda:0",
- ):
- self.device = device
-
- # Add sam-3d-body to path
- for candidate in [
- Path.cwd() / "sam-3d-body",
- Path.cwd().parent / "sam-3d-body",
- Path.home() / "sam-3d-body",
- ]:
- if candidate.exists() and (candidate / "sam_3d_body").exists():
- if str(candidate) not in sys.path:
- sys.path.insert(0, str(candidate))
- break
-
- from sam_3d_body import SAM3DBodyEstimator
-
- ckpt = Path(checkpoint_path)
- if ckpt.exists():
- from sam_3d_body import load_sam_3d_body
- logger.info(f"Loading SAM3D from local checkpoint: {ckpt}")
- model, cfg = load_sam_3d_body(
- checkpoint_path=str(ckpt),
- mhr_path=mhr_path,
- )
- else:
- from sam_3d_body import load_sam_3d_body_hf
- logger.info(f"Loading SAM3D from HuggingFace: {hf_repo_id}")
- model, cfg = load_sam_3d_body_hf(hf_repo_id=hf_repo_id)
-
- model = model.to(device).eval()
- self._estimator = SAM3DBodyEstimator(model, cfg)
- self._faces = self._estimator.faces
-
- logger.info(f"SAM3D teacher ready on {device}")
-
- @property
- def faces(self) -> np.ndarray:
- return self._faces
-
- @torch.inference_mode()
- def predict_person(
- self,
- full_image_rgb: np.ndarray,
- bbox: np.ndarray,
- ) -> dict | None:
- """Run SAM3D on one person in *full_image_rgb*.
-
- We pass the **full image** with the bbox in full-frame coordinates.
- SAM3D's CLIFF camera head then uses the bbox position, bbox scale,
- and full-image dimensions to compute ``pred_cam_t`` and
- ``focal_length`` natively in full-frame perspective space — no
- manual coordinate conversion needed.
-
- Args:
- full_image_rgb: The original uncropped image (H, W, 3), uint8 RGB.
- bbox: ``[x1, y1, x2, y2]`` in full-image pixel coords.
-
- Returns:
- Raw SAM3D output dict for the single person, or ``None`` on
- failure.
- """
- bbox_arr = np.array(bbox, dtype=np.float32).reshape(1, 4)
-
- output_list = self._estimator.process_one_image(
- full_image_rgb,
- bboxes=bbox_arr,
- inference_type="body",
- )
-
- if not output_list:
- return None
-
- return output_list[0]
-
-
-# ---------------------------------------------------------------------------
-# Crop & annotation writer
-# ---------------------------------------------------------------------------
-def make_annotation_name(
- dataset_name: str,
- image_stem: str,
- person_id: int,
-) -> str:
- """Deterministic filename: ``<dataset>_<stem>_p<id>``."""
- # Sanitise: replace path separators / spaces that may leak from subfolders
- safe_stem = image_stem.replace(os.sep, "_").replace(" ", "_")
- return f"{dataset_name}_{safe_stem}_p{person_id}"
-
-
-def get_square_crop_padded(image: np.ndarray, bbox: np.ndarray, expand: float = 1.2) -> tuple[np.ndarray, np.ndarray]:
- """
- Extracts a perfectly square crop from the image, padding with black pixels
- if the crop goes outside the image boundaries.
- """
- h, w = image.shape[:2]
- x1, y1, x2, y2 = bbox.astype(float)
-
- cx, cy = (x1 + x2) / 2.0, (y1 + y2) / 2.0
- size = max(x2 - x1, y2 - y1) * expand
- half = size / 2.0
-
- # The perfect square coordinates (may fall outside the image)
- sq_x1, sq_y1 = int(cx - half), int(cy - half)
- sq_x2, sq_y2 = int(cx + half), int(cy + half)
-
- # Calculate padding needed if the square goes out of bounds
- pad_top = max(0, -sq_y1)
- pad_bottom = max(0, sq_y2 - h)
- pad_left = max(0, -sq_x1)
- pad_right = max(0, sq_x2 - w)
-
- # Extract the valid portion of the image
- valid_x1, valid_y1 = max(0, sq_x1), max(0, sq_y1)
- valid_x2, valid_y2 = min(w, sq_x2), min(h, sq_y2)
- crop_valid = image[valid_y1:valid_y2, valid_x1:valid_x2]
-
- # Pad the valid crop to make it perfectly square
- crop_square = cv2.copyMakeBorder(
- crop_valid, pad_top, pad_bottom, pad_left, pad_right,
- cv2.BORDER_CONSTANT, value=(0, 0, 0)
- )
-
- sq_bbox = np.array([sq_x1, sq_y1, sq_x2, sq_y2], dtype=np.float32)
- return crop_square, sq_bbox
-
-
-def save_crop_and_annotation(
- full_image_rgb: np.ndarray,
- bbox: np.ndarray,
- sam3d_output: dict,
- name: str,
- images_dir: Path,
- annotations_dir: Path,
-) -> None:
- """Resize the person crop to 224x224, save as JPEG, and write the .npz."""
- h, w = full_image_rgb.shape[:2]
-
- # Extract perfect square crop (padded if out of bounds)
- crop_square, sq_bbox = get_square_crop_padded(full_image_rgb, bbox, expand=1.2)
-
- # Resize to 224x224 and save as lossless PNG (RGB -> BGR for cv2)
- crop_224 = cv2.resize(crop_square, (CROP_SIZE, CROP_SIZE), interpolation=cv2.INTER_LINEAR)
- crop_bgr = cv2.cvtColor(crop_224, cv2.COLOR_RGB2BGR)
- cv2.imwrite(str(images_dir / f"{name}.png"), crop_bgr)
-
- # ... (The rest of Claude's function below this line stays exactly the same!) ...
- f = math.sqrt(h * h + w * w)
-
- # Extract SAM3D outputs
- cam_trans = sam3d_output.get("pred_cam_t")
- if cam_trans is not None and hasattr(cam_trans, "numpy"):
- cam_trans = cam_trans.numpy()
- cam_trans = np.asarray(cam_trans, dtype=np.float32).ravel()[:3]
-
- focal = sam3d_output.get("focal_length")
- if focal is not None and hasattr(focal, "numpy"):
- focal = focal.numpy()
- if focal is not None:
- focal = np.asarray(focal, dtype=np.float32).ravel()
- if focal.size == 1:
- focal = np.array([focal[0], focal[0]], dtype=np.float32)
- else:
- focal = np.array([f, f], dtype=np.float32)
-
- joints_3d = _to_np(sam3d_output.get("pred_keypoints_3d")) # (70, 3)
- joints_2d = _to_np(sam3d_output.get("pred_keypoints_2d")) # (70, 2)
- mhr_model_params = _to_np(sam3d_output.get("mhr_model_params")) # (204,)
- shape_params = _to_np(sam3d_output.get("shape_params")) # (45,)
-
- # Build the annotation payload
- data: dict[str, np.ndarray] = {
- "orig_shape": np.array([h, w], dtype=np.int32),
- "bbox": bbox.astype(np.float32), # raw YOLO detection — CLIFF student needs this
- "bbox_square": sq_bbox, # expanded square crop used for the 224×224 resize
- "cam_focal_length": focal.astype(np.float32),
- "cam_trans": cam_trans,
- }
- if mhr_model_params is not None:
- data["mhr_model_params"] = mhr_model_params.astype(np.float32)
- if shape_params is not None:
- data["shape_params"] = shape_params.astype(np.float32)
- if joints_3d is not None:
- data["joints_3d"] = joints_3d.astype(np.float32)
- if joints_2d is not None:
- data["joints_2d"] = joints_2d.astype(np.float32)
-
- np.savez(annotations_dir / f"{name}.npz", **data)
-
-
-def _to_np(x) -> np.ndarray | None:
- """Convert torch.Tensor / np.ndarray to a squeezed numpy array."""
- if x is None:
- return None
- if hasattr(x, "cpu"):
- x = x.cpu().numpy()
- x = np.asarray(x, dtype=np.float32)
- # Squeeze leading batch dim if present: (1, ...) → (...)
- if x.ndim >= 2 and x.shape[0] == 1:
- x = x[0]
- return x
-
-
-# ---------------------------------------------------------------------------
-# CLI
-# ---------------------------------------------------------------------------
-def parse_args() -> argparse.Namespace:
- p = argparse.ArgumentParser(
- description="Cloud-grade CLIFF annotation pipeline (YOLO + SAM3D)",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog=__doc__,
- )
-
- # I/O
- p.add_argument(
- "--root_dir", type=str, required=True,
- help="Root directory containing dataset subdirectories of images.",
- )
- p.add_argument(
- "--output_dir", type=str, required=True,
- help="Output directory (images/ and annotations/ created inside).",
- )
-
- # Detection
- p.add_argument(
- "--yolo_model", type=str, default="yolo11n.pt",
- help="YOLO model name or path (default: yolo11n.pt).",
- )
- p.add_argument(
- "--confidence", type=float, default=0.7,
- help="YOLO person detection confidence threshold (default: 0.7).",
- )
- p.add_argument(
- "--max_persons", type=int, default=10,
- help="Max persons to annotate per image (default: 10).",
- )
-
- # SAM3D teacher
- p.add_argument(
- "--checkpoint_path", type=str,
- default="checkpoints/sam-3d-body-dinov3/model.ckpt",
- )
- p.add_argument(
- "--mhr_path", type=str,
- default="checkpoints/sam-3d-body-dinov3/assets/mhr_model.pt",
- )
- p.add_argument(
- "--hf_repo", type=str, default="facebook/sam-3d-body-dinov3",
- )
-
- # Performance
- p.add_argument(
- "--timeout", type=float, default=5.0,
- help="Per-image timeout in seconds (default: 5.0).",
- )
- p.add_argument(
- "--gc_interval", type=int, default=200,
- help="Force GC + CUDA cache clear every N images (default: 200).",
- )
- p.add_argument(
- "--max_samples", type=int, default=None,
- help="Stop after producing N annotation crops (not N images).",
- )
-
- # Misc
- p.add_argument("--device", type=str, default="cuda:0")
- p.add_argument("--debug", action="store_true")
- p.add_argument(
- "--shuffle", action="store_true",
- help="Shuffle image order for diverse sampling across datasets.",
- )
- p.add_argument(
- "--use_rerun", action="store_true",
- help="Open Rerun viewer with 3 panels: raw image, overlay, 3D mesh.",
- )
-
- return p.parse_args()
-
-
-# ---------------------------------------------------------------------------
-# Main
-# ---------------------------------------------------------------------------
-def main() -> None:
- args = parse_args()
-
- # Logging
- logger.remove()
- fmt = (
- "<green>{time:HH:mm:ss}</green> | <level>{level:<8}</level> | "
- "<cyan>{function}</cyan> - <level>{message}</level>"
- )
- logger.add(sys.stderr, level="DEBUG" if args.debug else "INFO", format=fmt)
-
- root_dir = Path(args.root_dir)
- output_dir = Path(args.output_dir)
- images_dir = output_dir / "images"
- annotations_dir = output_dir / "annotations"
- images_dir.mkdir(parents=True, exist_ok=True)
- annotations_dir.mkdir(parents=True, exist_ok=True)
-
- # ------------------------------------------------------------------
- # 1. Discover images
- # ------------------------------------------------------------------
- logger.info(f"Scanning {root_dir} for images ...")
- all_images = discover_images(root_dir)
- if not all_images:
- logger.error(f"No images found under {root_dir}")
- return
-
- if args.shuffle:
- random.shuffle(all_images)
- logger.info("Image order shuffled for diverse cross-dataset sampling")
-
- logger.info(f"Found {len(all_images)} images across {len(set(d for d, _ in all_images))} dataset(s)")
- if args.max_samples:
- logger.info(f"Will stop after {args.max_samples} annotation crops")
-
- # ------------------------------------------------------------------
- # 2. Build resume set — collect existing annotation stems
- # ------------------------------------------------------------------
- existing_stems: set[str] = set()
- for p in annotations_dir.glob("*.npz"):
- existing_stems.add(p.stem)
-
- if existing_stems:
- logger.info(f"Resume: {len(existing_stems)} annotations already on disk")
-
- # Count existing annotations toward the budget so resumed runs stop correctly
- total_crops = len(existing_stems)
-
- # ------------------------------------------------------------------
- # 3. Load models
- # ------------------------------------------------------------------
- logger.info("Loading YOLO detector ...")
- detector = YOLOPersonDetector(
- model_name=args.yolo_model,
- device=args.device,
- confidence=args.confidence,
- )
-
- logger.info("Loading SAM3D teacher ...")
- teacher = SAM3DTeacher(
- checkpoint_path=args.checkpoint_path,
- mhr_path=args.mhr_path,
- hf_repo_id=args.hf_repo,
- device=args.device,
- )
-
- # ------------------------------------------------------------------
- # 3b. Rerun setup
- # ------------------------------------------------------------------
- use_rerun = args.use_rerun
- if use_rerun:
- try:
- import rerun as rr
- rr.init("sam3d_cloud_annotation", spawn=True)
- rr.disable_timeline("frame")
- rr.log("scene_3d", rr.ViewCoordinates.RIGHT_HAND_Y_DOWN, static=True)
- setup_rerun_blueprint()
- logger.info("Rerun viewer launched (3-panel layout, no history)")
- except ImportError:
- logger.warning("rerun-sdk not installed, disabling --use_rerun")
- use_rerun = False
-
- # ------------------------------------------------------------------
- # 4. Processing loop
- # ------------------------------------------------------------------
- stats = {"processed": 0, "skipped": 0, "failed": 0, "timed_out": 0, "crops": 0}
- t0 = time.perf_counter()
-
- pbar = tqdm(all_images, desc="Annotating", unit="img", dynamic_ncols=True)
- for dataset_name, img_path in pbar:
- # Use relative path from root so subfolder structure prevents collisions
- # e.g. COCO/train/000001.jpg → "train_000001" (dataset_name = "COCO")
- rel_path = img_path.relative_to(root_dir)
- image_stem = str(rel_path.with_suffix("")).replace(os.sep, "_").replace(" ", "_")
- # Strip the dataset prefix since make_annotation_name re-adds it
- if image_stem.startswith(dataset_name + "_"):
- image_stem = image_stem[len(dataset_name) + 1:]
-
- # Quick check: can we skip the entire image?
- # We check for person 0; if it exists the image was processed before.
- first_name = make_annotation_name(dataset_name, image_stem, 0)
- if first_name in existing_stems:
- stats["skipped"] += 1
- continue
-
- # Load image
- img_bgr = cv2.imread(str(img_path))
- if img_bgr is None:
- logger.warning(f"Cannot read {img_path}")
- stats["failed"] += 1
- continue
- img_rgb = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB)
- H, W = img_rgb.shape[:2]
-
- # Detect persons
- try:
- with time_limit(args.timeout):
- detections = detector.detect(img_rgb)
- except InferenceTimeout:
- logger.warning(f"YOLO timeout on {img_path.name}")
- stats["timed_out"] += 1
- continue
- except Exception as e:
- logger.warning(f"YOLO failed on {img_path.name}: {e}")
- stats["failed"] += 1
- continue
-
- if not detections:
- stats["processed"] += 1
- continue
-
- detections = detections[: args.max_persons]
-
- # Process each detected person — pass the FULL image so SAM3D's
- # CLIFF camera head uses the correct bbox_center, bbox_size, and
- # img_size to produce cam_trans in full-frame perspective space.
- rerun_sam3d_outputs: list[dict | None] = []
- for person_id, det in enumerate(detections):
- name = make_annotation_name(dataset_name, image_stem, person_id)
-
- # Skip if this specific crop already exists
- if name in existing_stems:
- rerun_sam3d_outputs.append(None)
- continue
-
- bbox = det["bbox"]
-
- try:
- with time_limit(args.timeout):
- sam3d_out = teacher.predict_person(img_rgb, bbox)
- except InferenceTimeout:
- logger.warning(f"SAM3D timeout on {name}")
- stats["timed_out"] += 1
- rerun_sam3d_outputs.append(None)
- continue
- except Exception as e:
- logger.warning(f"SAM3D failed on {name}: {e}")
- torch.cuda.empty_cache()
- if args.debug:
- raise
- stats["failed"] += 1
- rerun_sam3d_outputs.append(None)
- continue
-
- if sam3d_out is None:
- stats["failed"] += 1
- rerun_sam3d_outputs.append(None)
- continue
-
- # Validate essential outputs
- if sam3d_out.get("mhr_model_params") is None:
- logger.warning(f"No mhr_model_params for {name} (keys: {list(sam3d_out.keys())})")
- stats["failed"] += 1
- rerun_sam3d_outputs.append(None)
- continue
-
- try:
- save_crop_and_annotation(
- img_rgb, bbox, sam3d_out, name,
- images_dir, annotations_dir,
- )
- stats["crops"] += 1
- total_crops += 1
- except Exception as e:
- logger.warning(f"Failed to save {name}: {e}")
- if args.debug:
- raise
- stats["failed"] += 1
- rerun_sam3d_outputs.append(None)
- continue
-
- rerun_sam3d_outputs.append(sam3d_out)
-
- # Rerun visualization (after all persons for this image)
- if use_rerun and any(o is not None for o in rerun_sam3d_outputs):
- log_to_rerun(img_rgb, detections, rerun_sam3d_outputs, teacher.faces)
-
- stats["processed"] += 1
-
- # Stop once we have enough crops
- if args.max_samples and total_crops >= args.max_samples:
- logger.info(f"Reached {total_crops} total crops (--max_samples {args.max_samples}), stopping")
- break
-
- # Periodic cleanup
- if stats["processed"] % args.gc_interval == 0:
- torch.cuda.empty_cache()
- gc.collect()
-
- # Update progress bar
- pbar.set_postfix(
- crops=stats["crops"],
- fail=stats["failed"],
- skip=stats["skipped"],
- )
-
- elapsed = time.perf_counter() - t0
- hours = elapsed / 3600
-
- logger.info("=" * 60)
- logger.info("ANNOTATION COMPLETE")
- logger.info("=" * 60)
- logger.info(f" Images scanned: {len(all_images)}")
- logger.info(f" Images processed: {stats['processed']}")
- logger.info(f" Images skipped: {stats['skipped']} (already annotated)")
- logger.info(f" Crops saved: {stats['crops']} (total on disk: {total_crops})")
- logger.info(f" Failures: {stats['failed']}")
- logger.info(f" Timeouts: {stats['timed_out']}")
- logger.info(f" Wall time: {hours:.2f}h ({elapsed / max(stats['processed'], 1):.2f}s/img)")
- logger.info(f" Output: {output_dir}")
- logger.info("=" * 60)
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/bench.py b/tools/bench.py
deleted file mode 100644
index 115ee0b..0000000
--- a/tools/bench.py
+++ /dev/null
@@ -1,138 +0,0 @@
-#!/usr/bin/env python3
-"""Headless benchmark for the InstantHMR pipeline.
-
-Runs ``PosePipeline`` on a video and prints per-stage latency stats with no
-Rerun, no GUI, no .rrd writing — so the numbers reflect inference cost and
-not visualisation overhead.
-
-Usage:
- python tools/bench.py --video vid1.mp4 [--device cuda] [--detector-variant medium]
- [--max-frames 200] [--warmup 10]
-"""
-
-from __future__ import annotations
-
-import argparse
-import statistics
-import sys
-import time
-from pathlib import Path
-
-import cv2
-import numpy as np
-
-# Make the package importable when run from anywhere.
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from instanthmr import PosePipeline
-
-
-def parse_args() -> argparse.Namespace:
- p = argparse.ArgumentParser()
- p.add_argument("--video", type=str, required=True)
- p.add_argument("--model", type=str, default="models/instanthmr.onnx")
- p.add_argument("--device", type=str, default="cuda")
- p.add_argument("--detector-variant", type=str, default="medium",
- choices=["nano", "small", "medium", "base", "large"])
- p.add_argument("--det-confidence", type=float, default=0.5)
- p.add_argument("--max-persons", type=int, default=5)
- p.add_argument("--detector-stride", type=int, default=1,
- help="Run RF-DETR every N frames (default 1 = every frame).")
- p.add_argument("--no-batch-persons", action="store_true",
- help="Disable batched multi-person HMR (one ORT call per person).")
- p.add_argument("--max-frames", type=int, default=200,
- help="Cap measured frames (after warm-up).")
- p.add_argument("--warmup", type=int, default=10,
- help="Frames discarded before timing starts.")
- return p.parse_args()
-
-
-def _summary(name: str, samples: list[float]) -> str:
- if not samples:
- return f"{name}: no samples"
- arr = np.asarray(samples, dtype=np.float64)
- return (
- f"{name:>14s} "
- f"mean={arr.mean():6.2f} ms "
- f"median={np.median(arr):6.2f} ms "
- f"p95={np.percentile(arr, 95):6.2f} ms "
- f"min={arr.min():6.2f} ms "
- f"max={arr.max():6.2f} ms"
- )
-
-
-def main() -> None:
- args = parse_args()
-
- cap = cv2.VideoCapture(args.video)
- if not cap.isOpened():
- sys.exit(f"[error] cannot open video: {args.video}")
-
- src_fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
- total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
- width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
- height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
- print(f"video: {args.video} {width}x{height} {total} frames @ {src_fps:.1f} fps")
-
- pipeline = PosePipeline(
- onnx_path=args.model,
- device=args.device,
- detector_variant=args.detector_variant,
- det_confidence=args.det_confidence,
- max_persons=args.max_persons,
- detector_stride=args.detector_stride,
- batch_persons=not args.no_batch_persons,
- )
- print(f"InstantHMR EP: {pipeline.hmr.active_provider}")
- print(f"detector variant: {args.detector_variant}")
-
- det: list[float] = []
- hmr: list[float] = []
- tot: list[float] = []
- wall: list[float] = []
- n_persons: list[int] = []
-
- idx = 0
- measured = 0
- last_wall = None
- while True:
- ret, bgr = cap.read()
- if not ret:
- break
-
- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
- t_wall = time.perf_counter()
- result = pipeline.predict(rgb)
-
- if idx >= args.warmup:
- det.append(result.detector_ms)
- hmr.append(result.hmr_ms)
- tot.append(result.total_ms)
- n_persons.append(len(result.persons))
- if last_wall is not None:
- wall.append((t_wall - last_wall) * 1000.0)
- measured += 1
- last_wall = t_wall
- idx += 1
- if measured >= args.max_frames:
- break
-
- cap.release()
-
- if not tot:
- sys.exit("[error] no measured frames — increase --max-frames or shorten --warmup")
-
- print()
- print(f"measured frames: {len(tot)} (warmup={args.warmup})")
- print(f"avg persons / frame: {statistics.mean(n_persons):.2f}")
- print(_summary("detector_ms", det))
- print(_summary("hmr_ms", hmr))
- print(_summary("total_ms", tot))
- if wall:
- print(_summary("wall_ms", wall))
- mean_total = float(np.mean(tot))
- print(f"\nthroughput (1 / mean total): {1000.0 / mean_total:5.1f} FPS")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/export_onnx.py b/tools/export_onnx.py
new file mode 100644
index 0000000..0e169c9
--- /dev/null
+++ b/tools/export_onnx.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Export InstantHMR to ONNX: one network graph per checkpoint, and one shared MHR body graph.
+
+network (--ckpt): image (B, 3, S, S) ImageNet-normalised RGB crop, cliff_cond (B, 3)
+ -> mhr_params (B, 204), shape_params (B, 45), cam_trans (B, 3),
+ joints_2d (B, 70, 2) in [-1, 1] over the crop
+body (--body): mhr_params, shape_params -> joints_3d (B, 70, 3), vertices (B, 18439, 3),
+ metres, body-centred (add cam_trans for camera space)
+
+The network graph stores its crop size and conditioning form in the model metadata, the body
+graph its mesh faces, so instanthmr.InstantHMR configures itself from the files.
+
+ python tools/export_onnx.py --ckpt runs/medium/best_student_model_v3.pth --output models/instanthmr-medium.onnx
+ python tools/export_onnx.py --body --output models/mhr-body.onnx
+"""
+import argparse
+import base64
+import sys
+from pathlib import Path
+
+import numpy as np
+import onnx
+import onnxruntime as ort
+import torch
+
+REPO = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO / "training"))
+
+import train as T # noqa: E402
+from mhr_body import MHRBody # noqa: E402
+
+NETWORK_OUTPUTS = ["mhr_params", "shape_params", "cam_trans", "joints_2d"]
+BODY_OUTPUTS = ["joints_3d", "vertices"]
+
+
+class Network(torch.nn.Module):
+ def __init__(self, net):
+ super().__init__()
+ self.net = net
+
+ def forward(self, image, cliff_cond):
+ out = self.net(image, cliff_cond)
+ return tuple(out[k] for k in NETWORK_OUTPUTS)
+
+
+def load_network(ckpt: Path):
+ state = torch.load(ckpt, map_location="cpu", weights_only=False)
+ state = {k.replace("_orig_mod.", ""): v for k, v in state.get("model_state_dict", state).items()}
+ if "head_2d_logits.weight" not in state or any(k.startswith("head_3d.") for k in state):
+ sys.exit(f"[error] {ckpt} was not written by training/train.py")
+ cfg, _ = T.config_from_checkpoint(state, ckpt)
+ cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2
+ cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])
+ net = T.InstantHMRStudent(cfg, pretrained=False).eval()
+ missing, unexpected = net.load_state_dict(state, strict=False)
+ if missing or unexpected:
+ sys.exit(f"[error] checkpoint and model disagree: missing {missing[:5]}, unexpected {unexpected[:5]}")
+ return net, cfg
+
+
+def export(module, inputs: dict, outputs: list, path: Path, metadata: dict) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ torch.onnx.export(module, tuple(inputs.values()), str(path), input_names=list(inputs), output_names=outputs,
+ dynamic_axes={n: {0: "batch"} for n in [*inputs, *outputs]}, opset_version=17, dynamo=False)
+ model = onnx.load(str(path))
+ for key, value in metadata.items():
+ entry = model.metadata_props.add()
+ entry.key, entry.value = key, value
+ onnx.save(model, str(path))
+
+ session = ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])
+ got = session.run(None, {k: v.numpy() for k, v in inputs.items()})
+ with torch.no_grad():
+ want = module(*inputs.values())
+ worst = {n: float(np.abs(g - w.numpy()).max()) for n, g, w in zip(outputs, got, want)}
+ print(f"wrote {path} ({path.stat().st_size / 1e6:.1f} MB); onnxruntime vs PyTorch, max abs diff: "
+ + ", ".join(f"{n} {v:.1e}" for n, v in worst.items()))
+ if max(worst.values()) > 1e-3:
+ sys.exit("[error] the ONNX graph disagrees with the PyTorch model")
+
+
+def main() -> None:
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ what = ap.add_mutually_exclusive_group(required=True)
+ what.add_argument("--ckpt", type=Path, help="checkpoint written by training/train.py or train_ddp.py")
+ what.add_argument("--body", action="store_true", help="export the MHR body graph instead")
+ ap.add_argument("--output", type=Path, required=True)
+ ap.add_argument("--mhr-model", type=Path, default=REPO / "training" / "checkpoints" / "mhr_model.pt",
+ help="MHR TorchScript rig the body is read from")
+ args = ap.parse_args()
+
+ if args.body:
+ body = MHRBody.from_rig(args.mhr_model).eval()
+ faces = torch.jit.load(str(args.mhr_model), map_location="cpu").character_torch.mesh.faces.numpy()
+ inputs = {"mhr_params": torch.randn(2, 204) * 0.3, "shape_params": torch.randn(2, 45)}
+ export(body, inputs, BODY_OUTPUTS, args.output,
+ {"mesh_faces": base64.b64encode(faces.astype("<i4").tobytes()).decode()})
+ return
+
+ net, cfg = load_network(args.ckpt)
+ size = int(cfg.image_size)
+ inputs = {"image": torch.randn(2, 3, size, size), "cliff_cond": torch.randn(2, 3) * 0.3}
+ export(Network(net).eval(), inputs, NETWORK_OUTPUTS, args.output,
+ {"image_size": str(size), "cliff_focal": str(bool(cfg.cliff_focal)).lower()})
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/pack_reference.py b/tools/pack_reference.py
deleted file mode 100644
index 2768fb6..0000000
--- a/tools/pack_reference.py
+++ /dev/null
@@ -1,84 +0,0 @@
-#!/usr/bin/env python3
-"""Pack a SAM 3D Body reference run into one small .npz for ``demo.py --show-reference``.
-
-The reference comes from ``video_to_pose_pipeline/main.py --save_meshes``, which
-writes one ``frame_%06d.npz`` per frame carrying the full 18,439-vertex mesh AND
-a copy of the constant 36,874-face index array -- ~360 kB per frame, so a few
-minutes of video costs a gigabyte.
-
-None of that needs storing. The saved ``person_i_vertices`` are **exactly** what
-``instanthmr.mhr_renderer`` reproduces from the saved ``mhr_model_params`` and
-``mhr_shape_params`` -- verified at 0.0005 mm on vid4, same rig, same rig-local
-frame, faces from the rig itself. So this keeps the parameters and drops the
-mesh, which is ~1 kB per frame and has a useful side effect: the reference and
-the student are then decoded by the SAME rig, so anything you see differing in
-the viewer is the model, never the renderer.
-
- python tools/pack_reference.py reference/vid4/meshes reference/vid4.npz
-
-Frames with no detection produce no input file and are simply absent from the
-output; ``frame_idx`` records which frames are present.
-"""
-
-from __future__ import annotations
-
-import argparse
-from pathlib import Path
-
-import numpy as np
-
-# Per-person fields copied straight through, with the output key they take.
-PERSON_FIELDS = {
- "mhr_model_params": "model_params",
- "mhr_shape_params": "shape_params",
- "cam_translation": "cam_trans",
- "joints_3d": "joints_3d",
- "joints_2d": "joints_2d",
- "bbox": "bbox",
- "cam_focal_length": "focal_length",
- "cam_principal_point": "principal_point",
-}
-
-
-def main() -> None:
- ap = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("mesh_dir", help="the pipeline's meshes/ directory")
- ap.add_argument("out", help="output .npz")
- a = ap.parse_args()
-
- files = sorted(Path(a.mesh_dir).glob("frame_*.npz"))
- if not files:
- raise SystemExit(f"no frame_*.npz under {a.mesh_dir}")
-
- frame_idx, counts = [], []
- cols: dict[str, list] = {k: [] for k in PERSON_FIELDS.values()}
- image_shape = None
-
- for f in files:
- d = np.load(f)
- n = int(d["num_persons"])
- frame_idx.append(int(d["frame_idx"]))
- counts.append(n)
- if image_shape is None:
- image_shape = d["image_shape"]
- for i in range(n):
- for src, dst in PERSON_FIELDS.items():
- cols[dst].append(d[f"person_{i}_{src}"])
-
- out = {
- "frame_idx": np.asarray(frame_idx, np.int32),
- "counts": np.asarray(counts, np.int32),
- "image_shape": np.asarray(image_shape, np.int32),
- }
- for dst, vals in cols.items():
- out[dst] = np.stack(vals).astype(np.float32)
-
- Path(a.out).parent.mkdir(parents=True, exist_ok=True)
- np.savez_compressed(a.out, **out)
- mb = Path(a.out).stat().st_size / 1e6
- print(f"{len(files)} frames, {sum(counts)} persons -> {a.out} ({mb:.2f} MB)")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/parquet_to_npz.py b/tools/parquet_to_npz.py
index a7e5c09..16d20a9 100644
--- a/tools/parquet_to_npz.py
+++ b/tools/parquet_to_npz.py
@@ -4,18 +4,13 @@ Convert the official ``facebook/sam-3d-body-dataset`` parquet annotations into t
per-crop ``images/*.png`` + ``annotations/*.npz`` layout consumed by
``training/train.py``.
-This lets you train InstantHMR directly on the original SAM 3D Body ground-truth
-MHR fits, instead of (or alongside) the teacher-distilled pseudo-labels produced
-by ``tools/annotate_dataset.py``. The output ``.npz`` schema is exactly the one
-that script writes, so the training notebook needs **no changes**.
-
-Key facts established by auditing the parquet (see tools/README or docs):
+These are the ground-truth MHR fits SAM 3D Body was built from. Facts about the parquet:
* The parquet has no ``cam_trans`` column, but the SAM3D translation is fully
recoverable. The global translation is the first three MHR ``model_params``:
cam_trans = [ model_params[0], -model_params[1], -model_params[2] ] / 10
- and the body-centred joints (distillation convention, vision Y-down) are
+ and the body-centred joints (vision frame, Y down) are
joints_3d = keypoints_3d[:, :3] * [1, -1, -1] # negate Y and Z
@@ -27,7 +22,7 @@ Key facts established by auditing the parquet (see tools/README or docs):
* Persons with ``mhr_valid == False`` have unreliable fits and are skipped by
default (the 2D in-the-wild splits — coco/mpii/aic — rely on this flag).
-NPZ payload (per crop) — identical to tools/annotate_dataset.py:
+NPZ payload (per crop):
orig_shape (2,) [H, W] of the full uncropped image
bbox (4,) [x1, y1, x2, y2] in original image coords (xyxy)
bbox_square (4,) [x1, y1, x2, y2] expanded square crop (1.2x)
@@ -84,17 +79,12 @@ from tqdm import tqdm
CROP_SIZE = 224
# Negate Y and Z to map MHR-native (Y-up) coordinates into the vision camera
-# frame (Y-down, Z-forward) used everywhere in the distillation pipeline.
+# frame (Y-down, Z-forward) the trainer uses.
NEG_YZ = np.array([1.0, -1.0, -1.0], dtype=np.float64)
# MHR root translation (model_params[:3]) is stored in decimetres.
TRANS_SCALE = 10.0
-# ---------------------------------------------------------------------------
-# Crop geometry — kept byte-for-byte in sync with
-# tools/annotate_dataset.py:get_square_crop_padded so the 224 crops produced
-# here are pixel-identical to the distillation crops.
-# ---------------------------------------------------------------------------
def get_square_crop_padded(image: np.ndarray, bbox: np.ndarray, expand: float = 1.2):
"""Square crop around *bbox*, black-padded where it leaves the image."""
h, w = image.shape[:2]
@@ -169,8 +159,8 @@ def reproj_error(joints_3d: np.ndarray, cam_trans: np.ndarray,
orig_shape: np.ndarray) -> float:
"""RMS px error of (joints_3d + cam_trans) projected onto joints_2d.
- Uses the exact camera model the notebook loss assumes: focal from cam_int,
- principal point at the image centre.
+ Camera model of the training losses: focal from cam_int, principal point at
+ the image centre.
"""
h, w = float(orig_shape[0]), float(orig_shape[1])
cam = joints_3d + cam_trans[None, :]
@@ -197,13 +187,9 @@ def build_annotation(row, img_h: int, img_w: int) -> dict:
cam_trans = cam_trans_from_params(model_params) # from model_params[:3] BEFORE zeroing
focal = np.array([cam_int[0, 0], cam_int[1, 1]], dtype=np.float32)
- # The dataset bakes the global translation into model_params[:3], so the MHR
- # rig decodes an already-camera-positioned mesh. The distillation pipeline
- # (notebook Cells 2 & 8) instead expects a *body-centred* mesh and adds
- # cam_trans separately. Zero the translation here so the MHR mesh comes out
- # body-centred — verified to match `joints_3d` to ~1e-7. Without this, the
- # notebook double-counts cam_trans (loss_structural / loss_feet /
- # loss_reproj_mhr and the MHR-param visualiser all go wrong).
+ # The dataset bakes the global translation into model_params[:3]; the trainer
+ # expects a body-centred mesh and carries the translation in cam_trans, so it
+ # is zeroed here (the mesh then matches `joints_3d` to ~1e-7).
model_params = model_params.copy()
model_params[:3] = 0.0
@@ -235,7 +221,7 @@ def save_person(full_img_rgb, ann, name, images_dir, annotations_dir):
# ---------------------------------------------------------------------------
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(
- description="Convert sam-3d-body-dataset parquet -> distillation .npz crops.",
+ description="Convert sam-3d-body-dataset parquet -> per-person .npz crops.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.py
deleted file mode 100644
index e6ad1f4..0000000
--- a/tools/pth_to_onnx.py
+++ /dev/null
@@ -1,243 +0,0 @@
-#!/usr/bin/env python3
-"""
-Convert a trained InstantHMR student checkpoint (.pth) into the ONNX file the
-demo expects at ``models/instanthmr.onnx``.
-
-It reuses the exact model architecture and deploy wrapper from the training
-script that produced the checkpoint, plus the FP32 export recipe from notebook
-Cell 15, so the graph I/O matches ``instanthmr/inference.py``:
-
- inputs : image (N, 3, S, S), cliff_cond (N, 3); S = the checkpoint's input
- size (224 unless trained with --image-size), stamped as image_size
- outputs: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]
-
-Architecture
-------------
-Only checkpoints of ``training/train.py`` are supported: a SimCC 2D head
-(``head_2d_logits``) and no 3D head, since the 70 keypoints come from the MHR
-forward pass, so the graph has FOUR outputs. Checkpoints of the earlier
-trainers (a ``head_3d`` or a ``head_2d_xy`` head) are rejected with an error:
-exporting them through this architecture would silently leave a head at its
-initial weights. A missing-key check makes any other mismatch a hard error.
-
-Usage
------
- python tools/pth_to_onnx.py \
- --ckpt training/runs/<run>/best_student_model_v3.pth \
- --output models/instanthmr_<run>.onnx
-"""
-import argparse
-import importlib
-import sys
-from pathlib import Path
-
-import torch
-
-REPO = Path(__file__).resolve().parent.parent
-TRAIN_DIR = REPO / "training"
-
-# The model class + deploy wrapper live in the standalone training scripts.
-sys.path.insert(0, str(TRAIN_DIR))
-
-# arch name -> training module that defines the matching InstantHMRStudent.
-ARCH_MODULES = {
- "mhr_only": "train", # SimCC 2D head, NO 3D head
-}
-
-# Canonical ONNX output order. The MHR-only graph stops after `joints_2d`.
-ALL_OUTPUT_KEYS = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]
-
-
-def detect_arch(state: dict) -> str:
- """Pick the architecture that matches the checkpoint's prediction heads."""
- has_3d = any(k.startswith("head_3d.") for k in state)
- has_simcc = "head_2d_logits.weight" in state
-
- if has_simcc and not has_3d:
- # train.py: the 3D coordinate head was removed and the
- # decoder shrank from 141 to 71 queries.
- return "mhr_only"
- if has_3d or "head_2d_xy.0.weight" in state:
- raise SystemExit(
- "[error] this checkpoint comes from an earlier trainer (it has a 'head_3d' or "
- "'head_2d_xy' head), which this release does not include.")
- raise SystemExit(
- "[error] cannot infer architecture: checkpoint has no 'head_2d_logits.weight'.")
-
-
-def main() -> None:
- ap = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("--ckpt", type=Path, required=True,
- help="Path to the .pth checkpoint written by training/train.py or train_ddp.py.")
- ap.add_argument("--output", type=Path, default=REPO / "models" / "instanthmr.onnx",
- help="Where to write the ONNX (default: models/instanthmr.onnx).")
- ap.add_argument("--allow-missing", action="store_true",
- help="Export even if checkpoint weights are missing (NOT recommended).")
- ap.add_argument("--opset", type=int, default=17)
- ap.add_argument("--cliff-focal", dest="cliff_focal", action="store_true",
- default=None,
- help="Force the focal-aware CLIFF conditioning flag stamped "
- "into the ONNX metadata. Normally read from the "
- "run_config.json beside the checkpoint.")
- ap.add_argument("--no-cliff-focal", dest="cliff_focal", action="store_false",
- help="Force the pixel-normalised CLIFF conditioning flag.")
- args = ap.parse_args()
-
- if not args.ckpt.exists():
- sys.exit(f"[error] checkpoint not found: {args.ckpt}")
-
- device = torch.device("cpu") # export on CPU — portable, no CUDA needed
-
- ckpt = torch.load(args.ckpt, map_location=device, weights_only=False)
- state = ckpt.get("model_state_dict", ckpt)
- state = {k.replace("_orig_mod.", ""): v for k, v in state.items()} # strip torch.compile prefix
-
- arch = detect_arch(state)
- T = importlib.import_module(ARCH_MODULES[arch])
- print(f"Architecture: {arch} (from {ARCH_MODULES[arch]}.py)")
-
- # Restore the configuration the checkpoint was TRAINED with, not the
- # defaults. `--bound-scales` puts a tanh remap of the 77 body-size
- # parameters inside forward(); exporting without it leaves the head's raw
- # pre-tanh numbers to be read as bone scales, which is 21.0 mm MPJPE /
- # 14.4 mm PA-MPJPE of silent corruption. See T.config_from_checkpoint.
- cfg, prov = T.config_from_checkpoint(state, args.ckpt, cliff_focal=args.cliff_focal)
- print(f" bound_scales : {getattr(cfg, 'bound_scales', False)} "
- f"({prov.get('bound_scales', 'default')})")
- print(f" cliff_focal : {getattr(cfg, 'cliff_focal', False)} "
- f"({prov.get('cliff_focal', 'default')})")
- print(f" cont_head : {getattr(cfg, 'cont_head', False)} "
- f"({prov.get('cont_head', 'default')})")
-
- # SimCC head geometry lives in the checkpoint — mirror it so the graph and
- # the bin centres agree with what was actually trained.
- if "head_2d_logits.bias" in state:
- cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2
- if "kp2d_bin_centers" in state:
- cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])
- print(f" SimCC head: {cfg.kp2d_bins} bins over [-{cfg.kp2d_range}, {cfg.kp2d_range}]")
-
- # --- Build the model and load weights ---
- model = T.InstantHMRStudent(cfg, pretrained=False).to(device).eval()
- missing, unexpected = model.load_state_dict(state, strict=False)
- print(f"Loaded {args.ckpt.name} | epoch {ckpt.get('epoch', '?')} "
- f"| PA-MPJPE {ckpt.get('val_pa_mpjpe', '?')} | source {ckpt.get('source', '?')}")
- if unexpected:
- # Fatal, not a note. An unexpected key means the checkpoint carries
- # state this architecture has no slot for, i.e. the config above is
- # wrong -- exactly how `scale_lo`/`scale_hi` used to be dropped.
- sys.exit(
- f"\n[error] {len(unexpected)} key(s) in the checkpoint have no "
- f"home in the model built here:\n {unexpected[:8]}\n"
- f" The exported graph would silently differ from the "
- f"trained one. Fix the config, do not ignore this."
- )
- if missing:
- print(f" missing keys : {len(missing)} -> {missing[:8]}")
- if not args.allow_missing:
- sys.exit(
- f"\n[error] {len(missing)} parameter(s) had no weights in the checkpoint, so they "
- f"would be exported at random init.\n"
- f" This usually means the checkpoint does not match training/train.py's\n"
- f" model. --allow-missing forces the export."
- )
-
- deploy = T.HMRDeployWrapper(model).eval()
-
- # --- Export FP32 ONNX (Cell 15 recipe) ---
- dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)
- dummy_cliff = torch.randn(1, 3, device=device)
- inputs, input_names = (dummy_img, dummy_cliff), ["image", "cliff_cond"]
- prev_token = bool(getattr(cfg, "prev_token", False))
- if prev_token:
- # --prev-token: the previous pose (T.PREV_DIM floats) and its 0/1 flag
- # are graph inputs, the per-foot contact logits a fifth output.
- inputs += (torch.randn(1, T.PREV_DIM, device=device), torch.ones(1, device=device))
- input_names += ["prev_pose", "has_prev"]
- print(f" prev_token : {prev_token}")
-
- # Ask the wrapper how many tensors it returns instead of hard-coding five:
- # train.py drops `joints_3d` from the tuple.
- with torch.no_grad():
- n_out = len(deploy(*inputs))
- if n_out not in (4, 5):
- sys.exit(f"[error] unexpected deploy wrapper output count: {n_out} (expected 4 or 5)")
- output_keys = (ALL_OUTPUT_KEYS[:4] + ["contact_logits"] if prev_token
- else ALL_OUTPUT_KEYS[:n_out])
- print(f" graph outputs: {n_out} -> {output_keys}")
- if "joints_3d" not in output_keys:
- print(" note: no 'joints_3d' output — the 70 keypoints are derived from the\n"
- " MHR forward pass at inference time (needs --mhr-model / --mhr-assets\n"
- " in demo.py).")
-
- dynamic_axes = {k: {0: "batch"} for k in input_names + output_keys}
-
- args.output.parent.mkdir(parents=True, exist_ok=True)
- torch.onnx.export(
- deploy, inputs, str(args.output),
- input_names=input_names, output_names=output_keys,
- dynamic_axes=dynamic_axes, opset_version=args.opset,
- )
- # Stamp which CLIFF conditioning this graph expects. The vector has the
- # same shape either way, so a consumer that guesses wrong gets no error --
- # only a person placed at the wrong depth. instanthmr.inference reads this.
- try:
- import onnx
- m = onnx.load(str(args.output))
- for k, v in (("cliff_focal", str(bool(getattr(cfg, "cliff_focal", False))).lower()),
- ("bound_scales", str(bool(getattr(cfg, "bound_scales", False))).lower()),
- ("arch", arch),
- ("image_size", str(int(cfg.image_size))),
- ("prev_token", str(prev_token).lower())):
- e = m.metadata_props.add()
- e.key, e.value = k, v
- onnx.save(m, str(args.output))
- print(f" metadata: cliff_focal={getattr(cfg, 'cliff_focal', False)} "
- f"bound_scales={getattr(cfg, 'bound_scales', False)} arch={arch} "
- f"image_size={cfg.image_size}")
- except Exception as e: # noqa: BLE001
- print(f" [warning] could not stamp ONNX metadata: {e}")
-
- print(f"\n✅ Wrote {args.output} ({args.output.stat().st_size / 1e6:.1f} MB)")
-
- # --- Sanity check: run once through onnxruntime and verify I/O ---
- try:
- import numpy as np
- import onnxruntime as ort
- sess = ort.InferenceSession(str(args.output), providers=["CPUExecutionProvider"])
- in_names = [i.name for i in sess.get_inputs()]
- outs = sess.run(None, {n: x.numpy() for n, x in zip(in_names, inputs)})
- print(f" onnxruntime OK | inputs {in_names}")
- for o, v in zip(sess.get_outputs(), outs):
- v = np.asarray(v)
- print(f" {o.name:14s} {list(v.shape)} std={v.std():.4f}")
-
- # Parity against the configured PyTorch model on the SAME input. This
- # is the check that catches a config-level export defect: identical
- # weights, different forward. A dropped tanh remap shows up here as a
- # large mhr_params delta while every tensor keeps its right shape.
- with torch.no_grad():
- ref = deploy(*inputs)
- worst = max(float(np.abs(np.asarray(o) - r.numpy()).max())
- for o, r in zip(outs, ref))
- print(f" torch<->onnx max abs diff: {worst:.3e}")
- if worst > 1e-3:
- sys.exit(
- f"\n[error] the ONNX graph disagrees with the PyTorch model it "
- f"was built from by {worst:.3e}.\n Identical weights, "
- f"different forward -- the exported configuration is wrong."
- )
- # A dead 2D head (wrong arch, weights left at init) predicts every joint
- # at the crop centre; real predictions spread over normalised [-1, 1].
- j2d = np.asarray(outs[output_keys.index("joints_2d")])
- if j2d.std() < 0.01:
- print(f"\n[warning] joints_2d std is {j2d.std():.5f} — the 2D keypoints are collapsed "
- f"onto the crop centre.\n The exported 2D head is almost certainly "
- f"untrained.")
- except Exception as e: # noqa: BLE001
- print(f" (onnxruntime sanity check skipped: {e})")
-
-
-if __name__ == "__main__":
- main()
diff --git a/training/README.md b/training/README.md
index a8b21bf..d75850f 100644
--- a/training/README.md
+++ b/training/README.md
@@ -21,9 +21,13 @@ A change to the model or a loss goes in `train.py`, so both entry points get it.
pip install -r training/requirements.txt
```
-The MHR rig is not in git. Copy `assets/mhr_model.pt` from the
-[SAM 3D Body checkpoint](https://huggingface.co/facebook/sam-3d-body-dinov3) to
-`training/checkpoints/mhr_model.pt`, or pass `--mhr_model_path`.
+The trainer needs Meta's MHR rig, `mhr_model.pt` (Apache-2.0, 700 MB), from the
+[MHR release](https://github.com/facebookresearch/MHR/releases):
+
+```bash
+curl -L -o mhr_assets.zip https://github.com/facebookresearch/MHR/releases/latest/download/assets.zip
+mkdir -p training/checkpoints && unzip -j mhr_assets.zip assets/mhr_model.pt -d training/checkpoints
+```
## Data
@@ -113,15 +117,17 @@ python tools/ddp_smoke.py --backbone hgnetv2_b4 --image-size 288 \
# The model must be able to fit 8 images.
python training/train.py --overfit-test --data_root <data>
-python -m pytest -q training/test_train_ddp.py
+python -m pytest -q training
```
## Export to ONNX
```bash
-python tools/pth_to_onnx.py --ckpt runs/medium/best_student_model_v3.pth \
- --output models/instanthmr_medium.onnx
+python tools/export_onnx.py --ckpt runs/medium/best_student_model_v3.pth --output models/my-model.onnx
+python tools/export_onnx.py --body --output models/mhr-body.onnx # once, shared by every model
+python demo.py --image photo.jpg --model models/my-model.onnx
```
-The exporter reads the training settings from the `run_config.json` beside the
-checkpoint, so keep the two together.
+The first graph is the network; the second turns its MHR parameters into the 70 3D keypoints and
+the mesh, and the demo looks for it next to the model. The exporter reads the training settings
+from the `run_config.json` beside the checkpoint, so keep the two together.
diff --git a/training/mhr_body.py b/training/mhr_body.py
new file mode 100644
index 0000000..6cd419c
--- /dev/null
+++ b/training/mhr_body.py
@@ -0,0 +1,161 @@
+"""The MHR body as an exportable PyTorch module: model parameters -> 70 keypoints + mesh.
+
+`MHRBody` rebuilds, from about 12 MB of constants read once from the TorchScript rig, what the
+network's outputs mean geometrically: forward kinematics of the 127 joints, the identity blend
+shapes and linear blend skinning of the 18,439-vertex mesh, and the 70 keypoints as the exact
+landmark readout the network was trained against (`W_joint @ joints + W_vert @ vertices`).
+Meta's pose correctives are left out: they are a 663 MB network, and the network was trained on
+the mesh without them (mean 1.9 mm from the full rig).
+
+Every op is static and exports to ONNX, so `tools/export_onnx.py` appends this module to the
+network and the released graphs need no MHR package at run time. Outputs are body-centred, in
+metres, in the camera axes (x right, y down, z forward); add `cam_trans` for camera space.
+"""
+from __future__ import annotations
+
+import math
+from pathlib import Path
+
+import numpy as np
+import torch
+
+N_MODEL_PARAMS = 204
+LANDMARKS = Path(__file__).resolve().parent / "assets" / "mhr_landmarks70.npz"
+
+
+def _fk_levels(parents: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ """Prefix-multiplication schedule: at level l, every active joint j becomes partner(j) o j."""
+ chains = []
+ for j in range(len(parents)):
+ chain = [j]
+ while parents[chain[-1]] >= 0:
+ chain.append(int(parents[chain[-1]]))
+ chains.append(chain[::-1])
+ partners, actives = [], []
+ level = 0
+ while True:
+ partner, active = np.arange(len(parents)), np.zeros(len(parents), bool)
+ for chain in chains:
+ i = len(chain) - 1
+ if (i >> level) & 1:
+ partner[chain[i]] = chain[((i >> level) << level) - 1]
+ active[chain[i]] = True
+ if not active.any():
+ return np.stack(partners), np.stack(actives)
+ partners.append(partner)
+ actives.append(active)
+ level += 1
+
+
+def _normalize(q):
+ return q / torch.sqrt((q * q).sum(-1, keepdim=True)).clamp_min(1e-12)
+
+
+def _qmul(a, b):
+ x1, y1, z1, w1 = a.unbind(-1)
+ x2, y2, z2, w2 = b.unbind(-1)
+ return torch.stack((w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,
+ w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2,
+ w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2,
+ w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2), -1)
+
+
+def _cross(a, b):
+ ax, ay, az = a.unbind(-1)
+ bx, by, bz = b.unbind(-1)
+ return torch.stack((ay * bz - az * by, az * bx - ax * bz, ax * by - ay * bx), -1)
+
+
+def _rotate(q, v):
+ axis, w = q[..., :3], q[..., 3:]
+ av = _cross(axis, v)
+ return v + 2 * (av * w + _cross(axis, av))
+
+
+def _euler_xyz_to_quat(e):
+ cr, sr = torch.cos(e[..., 0] / 2), torch.sin(e[..., 0] / 2)
+ cp, sp = torch.cos(e[..., 1] / 2), torch.sin(e[..., 1] / 2)
+ cy, sy = torch.cos(e[..., 2] / 2), torch.sin(e[..., 2] / 2)
+ return torch.stack((sr * cp * cy - cr * sp * sy, cr * sp * cy + sr * cp * sy,
+ cr * cp * sy - sr * sp * cy, cr * cp * cy + sr * sp * sy), -1)
+
+
+def _compose(a, b):
+ """a o b for skeleton states [t(3), q(4) xyzw, s(1)]."""
+ ta, qa, sa = a[..., :3], _normalize(a[..., 3:7]), a[..., 7:]
+ tb, qb, sb = b[..., :3], _normalize(b[..., 3:7]), b[..., 7:]
+ return torch.cat((ta + sa * _rotate(qa, tb), _qmul(qa, qb), sa * sb), -1)
+
+
+def _to_camera_axes(xyz):
+ """MHR (cm, y up, z back) -> metres, y down, z forward."""
+ return xyz * torch.tensor([0.01, -0.01, -0.01], dtype=xyz.dtype)
+
+
+class MHRBody(torch.nn.Module):
+ """`(B, 204)` mhr_params + `(B, 45)` shape_params -> `(B, 70, 3)` keypoints, `(B, 18439, 3)` vertices."""
+
+ def __init__(self, constants: dict[str, np.ndarray]):
+ super().__init__()
+ for name, value in constants.items():
+ self.register_buffer(name, torch.as_tensor(value), persistent=False)
+
+ @classmethod
+ def from_rig(cls, rig_path: str | Path, landmarks: str | Path = LANDMARKS) -> "MHRBody":
+ rig = torch.jit.load(str(rig_path), map_location="cpu").eval()
+ ct = rig.character_torch
+ sk, lbs, bs = ct.skeleton, ct.linear_blend_skinning, ct.blend_shape
+ transform = ct.parameter_transform.parameter_transform
+ if transform[:, N_MODEL_PARAMS:].any():
+ raise ValueError(f"{rig_path}: the parameter transform uses the identity columns")
+
+ n_verts = bs.base_shape.shape[0]
+ vert = lbs.vert_indices_flattened.long()
+ counts = torch.bincount(vert, minlength=n_verts)
+ order = torch.argsort(vert, stable=True)
+ vert = vert[order]
+ slot = torch.arange(len(vert)) - (torch.cumsum(counts, 0) - counts)[vert]
+ skin_joint = torch.zeros(n_verts, int(counts.max()), dtype=torch.long)
+ skin_weight = torch.zeros(n_verts, int(counts.max()))
+ skin_joint[vert, slot] = lbs.skin_indices_flattened.long()[order]
+ skin_weight[vert, slot] = lbs.skin_weights_flattened.float()[order]
+
+ partner, active = _fk_levels(sk.joint_parents.numpy())
+ lm = np.load(landmarks)
+ return cls(dict(
+ parameter_transform=transform[:, :N_MODEL_PARAMS].numpy(),
+ joint_offsets=sk.joint_translation_offsets.numpy(),
+ joint_prerotations=sk.joint_prerotations.numpy(),
+ fk_partner=partner, fk_active=active,
+ inverse_bind_pose=lbs.inverse_bind_pose.numpy(),
+ base_shape=bs.base_shape.numpy(), shape_vectors=bs.shape_vectors.numpy(),
+ skin_joint=skin_joint.numpy(), skin_weight=skin_weight.numpy(),
+ landmark_vertices=lm["vert_idx"].astype(np.int64),
+ W_joint=lm["W_joint"].astype(np.float32), W_vert=lm["W_vert"].astype(np.float32),
+ ))
+
+ def skeleton(self, mhr_params):
+ """`(B, 127, 8)` global joint states, MHR units."""
+ jp = (mhr_params @ self.parameter_transform.T).unflatten(-1, (-1, 7))
+ q = _qmul(self.joint_prerotations, _euler_xyz_to_quat(jp[..., 3:6]))
+ state = torch.cat((jp[..., :3] + self.joint_offsets, q, torch.exp(math.log(2.0) * jp[..., 6:])), -1)
+ for level in range(self.fk_partner.shape[0]):
+ composed = _compose(state.index_select(-2, self.fk_partner[level]), state)
+ state = torch.where(self.fk_active[level, :, None], composed, state)
+ return state
+
+ def skin(self, state, shape_params):
+ """`(B, 18439, 3)` posed vertices, MHR units."""
+ n_verts, n_inf = self.skin_joint.shape
+ rest = self.base_shape + (shape_params @ self.shape_vectors.flatten(1)).unflatten(-1, (n_verts, 3))
+ bone = _compose(state, self.inverse_bind_pose)
+ bone = bone.index_select(-2, self.skin_joint.flatten()).unflatten(-2, (n_verts, n_inf))
+ t, q, s = bone[..., :3], _normalize(bone[..., 3:7]), bone[..., 7:]
+ return ((t + _rotate(q, s * rest.unsqueeze(-2))) * self.skin_weight[..., None]).sum(-2)
+
+ def forward(self, mhr_params, shape_params):
+ state = self.skeleton(mhr_params)
+ vertices = _to_camera_axes(self.skin(state, shape_params))
+ joints = _to_camera_axes(state[..., :3])
+ keypoints = self.W_joint @ joints + self.W_vert @ vertices.index_select(-2, self.landmark_vertices)
+ return keypoints, vertices
diff --git a/training/test_mhr_body.py b/training/test_mhr_body.py
new file mode 100644
index 0000000..36173b4
--- /dev/null
+++ b/training/test_mhr_body.py
@@ -0,0 +1,59 @@
+"""The in-graph MHR body against the rig it replaces.
+
+ python -m pytest -q training/test_mhr_body.py
+"""
+import glob
+import os
+import sys
+from pathlib import Path
+
+import numpy as np
+import pytest
+import torch
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+RIG = HERE / "checkpoints" / "mhr_model.pt"
+pytestmark = pytest.mark.skipif(not RIG.exists(), reason=f"needs the MHR rig at {RIG}")
+
+
[email protected](scope="module")
+def poses():
+ """Real poses from a corpus ($INSTANTHMR_DATA, default data/), else random parameters."""
+ root = Path(os.environ.get("INSTANTHMR_DATA", HERE.parent / "data"))
+ files = sorted(glob.glob(str(root / "*" / "annotations" / "*.npz")))[:16]
+ if files:
+ p = np.stack([np.load(f)["mhr_model_params"] for f in files]).astype(np.float32)
+ s = np.stack([np.load(f)["shape_params"] for f in files]).astype(np.float32)
+ else:
+ g = np.random.default_rng(0)
+ p = (g.standard_normal((16, 204)) * 0.3).astype(np.float32)
+ s = g.standard_normal((16, 45)).astype(np.float32)
+ p[:, :3] = 0
+ return torch.from_numpy(p), torch.from_numpy(s)
+
+
[email protected](scope="module")
+def body():
+ from mhr_body import MHRBody
+ return MHRBody.from_rig(RIG).eval()
+
+
+def test_keypoints_match_the_training_readout(body, poses):
+ import train as T
+ ref = T.MHRForwardPass(str(RIG), "cpu", landmark_assets=str(HERE / "assets" / "mhr_landmarks70.npz"))
+ with torch.no_grad():
+ want = ref.get_native_keypoints(*poses, exact=True)
+ got, _ = body(*poses)
+ assert float((got - want).abs().max()) < 1e-5 # metres: 0.01 mm
+
+
+def test_vertices_match_the_rig_without_pose_correctives(body, poses):
+ rig = torch.jit.load(str(RIG), map_location="cpu").eval()
+ p, s = poses
+ with torch.no_grad():
+ v, _ = rig(s, p, torch.zeros(len(p), 72), False)
+ want = torch.stack([v[..., 0], -v[..., 1], -v[..., 2]], -1) / 100.0
+ _, got = body(p, s)
+ assert got.shape == want.shape == (len(p), 18439, 3)
+ assert float((got - want).abs().max()) < 1e-5
diff --git a/training/train.py b/training/train.py
index f68d9b8..100fe6f 100644
--- a/training/train.py
+++ b/training/train.py
@@ -1,43 +1,14 @@
#!/usr/bin/env python3
-# ============================================================
-# MHR-ONLY distillation — no concurrent 3D coordinate head.
-# ------------------------------------------------------------
-# SAM3D-Body (the teacher) has NO 3D and NO 2D pose head. Its only
-# predictive modules are head_pose.proj (519 outputs = MHR params) and
-# head_camera.proj (3 outputs = s, tx, ty). The 70 keypoints are
-# obtained as keypoint_mapping @ [verts ; joints] AFTER the MHR
-# forward, and the 2D ones by perspective re-projection of those 3D
-# points. Everything is derived from the mesh.
-#
-# The student mirrors that: the 3D head is GONE and every 3D quantity
-# is derived from the MHR forward pass. Concretely
-#
-# mhr_params, shape_params ──► MHR skeleton (127 joints)
-# ──► W @ joints = 70 native keypoints
-#
-# where W is a (70, 127) affine regressor fitted offline against the
-# teacher's own labels (see fit_keypoint_regressor / --fit-regressor).
-# Held-out residual of that fit: 1.36 mm mean / 0.005 mm median, which
-# is two orders of magnitude below the target PA-MPJPE. Because W's
-# rows sum to exactly 1 it commutes with rigid transforms, so the
-# augmentation counter-rotations keep working unchanged.
-#
-# The 2D SimCC head is KEPT. It is training-only (deployment re-projects
-# the MHR keypoints) but it is the only supervision that survives a
-# horizontal flip un-masked, so removing it would starve the flipped
-# half of the augmented samples.
-#
-# EXPORT CONTRACT CHANGED: the ONNX now has FOUR outputs
-# (mhr_params, shape_params, cam_trans, joints_2d)
-# instead of five — joints_3d is gone. instanthmr/inference.py must be
-# patched before this model can be used there; see the note on
-# HMRDeployWrapper below.
-#
-# Sanity: python3 train.py --self-test --data_root ../data/sam3d_gt_mpii
-# python3 train.py --overfit-test --data_root ../data/sam3d_gt_mpii
-# Refit W: python3 train.py --fit-regressor --data_root ../data
-# Full: python3 -u train.py --data_root /datasets/instanthmr_data --num_workers 8
-# ============================================================
+"""InstantHMR trainer: dataset, augmentation, model, MHR forward pass, losses, and a single-GPU loop.
+
+The network predicts the MHR parameters (pose, skeleton, body shape), the camera translation, and
+the 70 keypoints in 2D (SimCC). Every 3D quantity is derived from the MHR forward pass, as in
+SAM 3D Body. `train_ddp.py` imports this file for multi-GPU runs; see training/README.md.
+
+ python training/train.py --self-test --data_root data
+ python training/train.py --overfit-test --data_root data
+ python training/train.py --data_root data --output_dir runs/my_run
+"""
import os
# Must be set before the first CUDA allocation (read lazily by the allocator).
@@ -82,9 +53,6 @@ import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import cv2
-import onnxruntime as ort
-
-print(ort.get_available_providers())
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
@@ -1268,10 +1236,9 @@ class SAM3DStudentDataset(Dataset):
# Where the box sits in the FIELD OF VIEW, and how large it is
# ANGULARLY -- not in image-normalised pixels.
#
- # Under a fixed focal the two are interchangeable, and the corpus
- # this trainer was written for had exactly that: every crop from
- # tools/annotate_dataset.py carries the synthetic f = sqrt(H^2+W^2),
- # i.e. f/diag = 1.000 with zero spread. The rebuilt corpus carries
+ # Under a fixed focal the two are interchangeable, and the first
+ # corpus had exactly that: every crop carried the synthetic
+ # f = sqrt(H^2+W^2), i.e. f/diag = 1.000 with zero spread. The corpus
# the datasets' own focals instead, and those span f/diag 0.6-1.8 on
# coco/aic/mpii (sd ~0.35) with harmony4d off at 0.285. There the
# pixel version is ambiguous: two crops with identical conditioning
@@ -2820,212 +2787,6 @@ def train_instant_hmr():
print(f" best RAW: {best_raw_pa:.1f} mm -> {best_raw_path}")
print(f" best EMA: {best_ema_pa:.1f} mm -> {best_ema_path}")
-# ============================================================
-# Cell 15 — Export & Static Quantization
-# ============================================================
-class HMRDeployWrapper(nn.Module):
- """FOUR outputs: (mhr_params, shape_params, cam_trans, joints_2d).
-
- BREAKING CHANGE vs the earlier trainer, which exported a
- fifth tensor `joints_3d` straight from the (now deleted) 3D head.
-
- The MHR skeleton block cannot be folded into the ONNX graph: the TorchScript
- artefact exposes `character_torch` as a `pymomentum...Character` object, not
- an `nn.Module`, and torch.onnx.export refuses to trace through it
- ("Tried to trace <...Character> but it is not part of the active trace").
- So the 3D keypoints have to be produced by the consumer.
-
- Required patch in instanthmr/inference.py (NOT applied here — it would add a
- hard torch/mhr dependency to a module that is currently numpy + onnxruntime
- only, which is not a risk-free change to make blind):
-
- * __init__ (~l.119): the comment "Output order from export: ...,
- joints_3d" is now wrong; assert len(out_names) == 4 and load the MHR
- TorchScript model + training/assets/mhr_j127_to_kp70.npy.
- * predict (~l.157-162): `joints_3d_local = outs[4][0]` must become
-
- joints_3d_local = self._mhr_keypoints(mhr_params[None])[0]
-
- where `_mhr_keypoints` is MHRForwardPass.get_native_keypoints from this
- file (skeleton forward -> /100 -> negate Y,Z -> W @ joints).
- * predict_batch (~l.239): same for `joints_3d_local_b = outs[4]`, batched.
- * The docstrings of `InstantHMR` and `HMRPrediction` that advertise a
- 70x3 joints_3d output straight from the ONNX session.
-
- Measured cost of that extra forward: ~0.6 ms at batch 1 and ~0.7 ms at
- batch 8 on an RTX 4070 (no_grad, skeleton branch only, no skinning).
- """
-
- def __init__(self, model):
- super().__init__()
- self.model = model
-
- def forward(self, image, cliff_cond, prev_pose=None, has_prev=None):
- # A --prev-token graph takes the previous pose as two extra inputs and
- # returns the per-foot contact logits as a fifth output.
- out = self.model(image, cliff_cond, prev_pose, has_prev)
- base = (out["mhr_params"], out["shape_params"], out["cam_trans"],
- out["joints_2d"])
- return base + (out["contact_logits"],) if "contact_logits" in out else base
-
-def topological_sort_onnx(model):
- available = set()
- for inp in model.graph.input:
- available.add(inp.name)
- for init in model.graph.initializer:
- available.add(init.name)
-
- remaining = list(model.graph.node)
- sorted_nodes = []
-
- while remaining:
- progress = False
- for node in remaining:
- if all(inp in available or inp == '' for inp in node.input):
- sorted_nodes.append(node)
- remaining.remove(node)
- for out in node.output:
- available.add(out)
- progress = True
- break
- if not progress:
- sorted_nodes.extend(remaining)
- break
-
- del model.graph.node[:]
- model.graph.node.extend(sorted_nodes)
- return model
-
-def export_and_quantize():
- print("--- Exporting and Quantizing Full Model for Deployment ---")
- try:
- from onnxruntime.quantization import (
- CalibrationDataReader, quantize_static,
- QuantType, QuantFormat, CalibrationMethod,
- )
-
- class HMRCalibrationDataReader(CalibrationDataReader):
- def __init__(self, dataloader, max_samples=100):
- self.data_list = []
- print(f"Extracting {max_samples} individual images for INT8 calibration...")
- for batch in dataloader:
- images = batch["image"].numpy().astype(np.float32)
- cliffs = batch["cliff_cond"].numpy().astype(np.float32)
- for b_idx in range(images.shape[0]):
- if len(self.data_list) >= max_samples:
- break
- self.data_list.append({
- "image": images[b_idx:b_idx + 1],
- "cliff_cond": cliffs[b_idx:b_idx + 1]
- })
- if len(self.data_list) >= max_samples:
- break
- self.enum_data = iter(self.data_list)
-
- def get_next(self):
- return next(self.enum_data, None)
-
- except ImportError:
- print("⚠️ 'onnxruntime' is not installed. INT8 quantization will fail.")
- HMRCalibrationDataReader = None
-
- export_dir = Path(cfg.log_dir) / "export"
- export_dir.mkdir(parents=True, exist_ok=True)
-
- onnx_fp32_path = export_dir / "sam3d_full_fp32.onnx"
- onnx_fp16_path = export_dir / "sam3d_full_fp16.onnx"
- onnx_int8_path = export_dir / "sam3d_full_int8.onnx"
-
- output_keys = ["mhr_params", "shape_params", "cam_trans", "joints_2d"]
- print("⚠️ Export contract: 4 outputs (joints_3d dropped). "
- "instanthmr/inference.py needs the patch described on HMRDeployWrapper.")
- export_target_model = InstantHMRStudent(cfg, pretrained=False).to(device)
- best_ckpt_path = os.path.join(cfg.log_dir, "best_student_model_v3.pth")
-
- if os.path.exists(best_ckpt_path):
- ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=True)
- clean_state_dict = {k.replace('_orig_mod.', ''): v for k, v in ckpt['model_state_dict'].items()}
- export_target_model.load_state_dict(clean_state_dict, strict=False)
- print(f"✅ Loaded best model from epoch {ckpt.get('epoch', 0)+1}.\n")
-
- deploy_model = HMRDeployWrapper(export_target_model)
- deploy_model.eval()
-
- dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)
- dummy_cliff = torch.randn(1, 3, device=device)
-
- dynamic_axes = {"image": {0: "batch"}, "cliff_cond": {0: "batch"}}
- for k in output_keys:
- dynamic_axes[k] = {0: "batch"}
-
- try:
- import onnx
- torch.onnx.export(
- deploy_model, (dummy_img, dummy_cliff), str(onnx_fp32_path),
- input_names=["image", "cliff_cond"], output_names=output_keys,
- dynamic_axes=dynamic_axes, opset_version=17
- )
- print(f"✅ FP32 saved: {onnx_fp32_path.name} | Size: {onnx_fp32_path.stat().st_size / 1e6:.1f} MB")
- except Exception as e:
- print(f"❌ FP32 export failed: {e}")
-
- try:
- from onnxruntime.transformers.float16 import convert_float_to_float16
- import warnings
- with warnings.catch_warnings():
- warnings.simplefilter("ignore")
- model_fp32 = onnx.load(str(onnx_fp32_path))
- model_fp16 = convert_float_to_float16(model_fp32, keep_io_types=True)
- model_fp16 = topological_sort_onnx(model_fp16)
- onnx.save(model_fp16, str(onnx_fp16_path))
- onnx.checker.check_model(onnx.load(str(onnx_fp16_path)), full_check=True)
- print(f"✅ FP16 saved: {onnx_fp16_path.name} | Size: {onnx_fp16_path.stat().st_size / 1e6:.1f} MB")
- except ImportError:
- print("⚠️ FP16 skipped (onnxruntime.transformers not available)")
- except Exception as e:
- print(f"❌ FP16 conversion failed: {e}")
-
- try:
- calib_reader = HMRCalibrationDataReader(train_loader, max_samples=100)
- calib_reader.enum_data = iter(calib_reader.data_list)
- fp32_model = onnx.load(str(onnx_fp32_path))
- nodes_to_exclude = []
- for node in fp32_model.graph.node:
- name = node.name or ''
- if any(kw in name for kw in ['/model/transformer/', '/model/head_',
- '/model/feat_proj', '/model/cond_proj']):
- nodes_to_exclude.append(name)
- del fp32_model
- print(f" Excluding {len(nodes_to_exclude)} transformer/head nodes from quantization")
-
- quantize_static(
- model_input=str(onnx_fp32_path),
- model_output=str(onnx_int8_path),
- calibration_data_reader=calib_reader,
- quant_format=QuantFormat.QDQ,
- op_types_to_quantize=['Conv'],
- per_channel=True,
- reduce_range=False,
- activation_type=QuantType.QInt8,
- weight_type=QuantType.QInt8,
- nodes_to_exclude=nodes_to_exclude,
- calibrate_method=CalibrationMethod.Percentile,
- extra_options={'CalibTensorRangeSymmetric': True},
- )
- print(f"✅ INT8 saved: {onnx_int8_path.name} | Size: {onnx_int8_path.stat().st_size / 1e6:.1f} MB")
- except NameError:
- print("⚠️ INT8 skipped (train_loader not available)")
- except Exception as e:
- import traceback
- print(f"❌ INT8 conversion failed: {e}")
- traceback.print_exc()
-
- sd_path = export_dir / "sam3d_student_state_dict.pth"
- torch.save(export_target_model.state_dict(), sd_path)
- print(f"\n✅ PyTorch state dict saved: {sd_path.name}")
- else:
- print("⚠️ No checkpoint found at:", best_ckpt_path)
-
# ============================================================
# Self-tests
# ============================================================
@@ -3390,8 +3151,6 @@ def parse_args():
help="Refit the (70,127) skeleton->keypoint regressor from the dataset and exit.")
p.add_argument("--overfit-steps", dest="overfit_steps", type=int, default=3000,
help="Number of steps for --overfit-test (default 3000; see run_overfit_test).")
- p.add_argument("--no-export", dest="no_export", action="store_true",
- help="Skip the ONNX export / quantization step after training.")
p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")
p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",
help="Read the 70 keypoints out of the skinned mesh with the "
@@ -3558,9 +3317,7 @@ def main():
if torch.cuda.is_available():
torch.cuda.empty_cache()
train_instant_hmr()
-
- if not args.no_export:
- export_and_quantize()
+ print(f"Export the best checkpoint with: python tools/export_onnx.py --ckpt {cfg.log_dir}/best_student_model_v3.pth --output model.onnx")
if __name__ == "__main__":
main()
\ No newline at end of file