Skip to content
Python

feat: demo now runs with mhr_only model, added benchmarking scripts for 3dpw and coco

mohamdevb29e7d740291
diff --git a/benchmark/README.md b/benchmark/README.md
new file mode 100644
index 0000000..f22f02e
--- /dev/null
+++ b/benchmark/README.md
@@ -0,0 +1,142 @@
+# InstantHMR benchmarks
+
+Evaluation harness for publishing numbers on `models/instanthmr_mhr_only_ckpt90.onnx`.
+
+Two benchmarks are implemented, chosen because they are the two cheapest paths
+to a credible table — one 3D, one 2D:
+
+| Benchmark | What it measures | Data cost | Status |
+|---|---|---|---|
+| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT, free licence form | ready — **but the test split is contaminated, see below** |
+| **COCO** val2017 | OKS AP / PCK, 2D keypoint localisation in the wild | 820 MB, direct download, no registration | ready — clean |
+
+## Why these two
+
+**3DPW** is the benchmark essentially every monocular HMR paper reports, so it
+is the one that makes your numbers comparable. It is also unusually cheap here:
+the GT pickles store `jointPositions` (24 SMPL joints in world coordinates)
+directly, so **no SMPL model download or body-model forward pass is needed** —
+which is what makes most 3DPW harnesses painful to set up.
+
+**COCO val2017** needs no registration at all and the joint mapping is free:
+MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.
+
+### The ones deliberately skipped
+
+- **Human3.6M** — the licence forbids redistribution, the download is manual and
+ slow, and protocol variants (P1 vs P2, 17 vs 14 joints) differ per paper. High
+ effort, high chance of an unfair comparison.
+- **EMDB** — a genuinely good modern benchmark (EMDB-1 for pose), but needs
+ registration *and* the SMPL body model *and* their toolkit to unpack. Worth
+ adding as a second 3D benchmark after 3DPW is in place; `benchlib/threedpw.py`
+ is the template to copy.
+- **RICH** — SMPL-X GT, large download, mostly used for contact estimation.
+- **MPII** — 2D only, so it adds little once COCO is in.
+
+## Setup
+
+```bash
+# COCO — fully automatic (~820 MB)
+python benchmark/download.py --coco
+
+# 3DPW — accept the licence at
+# https://virtualhumans.mpi-inf.mpg.de/3DPW/licence.html
+# then unzip sequenceFiles.zip (and imageFiles.zip if you don't have it) and check:
+python benchmark/download.py --check-3dpw \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --image-root /path/to/3DPW/imageFiles
+```
+
+## Running
+
+```bash
+# COCO val2017 (~10 min on an RTX 4070)
+python benchmark/eval_coco.py \
+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \
+ --coco-root benchmark/data/coco
+
+# 3DPW test — add --stride 10 for a 10x faster smoke run
+python benchmark/eval_3dpw.py \
+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --image-root /path/to/3DPW/imageFiles \
+ --exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations
+```
+
+Both write a JSON report to `benchmark/results/`.
+
+## Protocols, stated precisely
+
+Write these into the paper — they are the details a reviewer will ask about.
+
+**3DPW test.** All 24 test sequences, every frame where `campose_valid` is set
+and the person is at least 60 % inside the image. Person boxes come from the
+projected GT joints padded by `--bbox-scale` (default 1.2), so no detector is in
+the loop. MPJPE is mid-hip-aligned; PA-MPJPE is Procrustes (scale + rotation +
+translation). Predictions use `joints_3d_local`, the rig-local pose, so the
+camera translation never enters the metric.
+
+**COCO val2017.** Top-down with **ground-truth boxes**, one crop per annotated
+person with `num_keypoints ≥ 1` and `iscrowd = 0`. These AP numbers are *not*
+comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits no
+per-keypoint confidence, so every keypoint is scored 1.0 — read AP next to the
+confidence-free `mean_OKS` and `PCK` numbers.
+
+### The joint-convention caveat — do not skip this
+
+InstantHMR predicts MHR70 joints; 3DPW's GT is SMPL. The two rigs place several
+landmarks differently, which puts a floor under the error that has nothing to do
+with model quality. The harness therefore reports three rows:
+
+- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` are the
+ two joints where the rigs genuinely disagree (MHR has no head-top; `nose` is
+ substituted), so this row carries a systematic penalty.
+- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles), where
+ both rigs mean the same anatomical point. **This is the cleanest measure of
+ the model.**
+- **J14+adapter** *(optional)* — a linear MHR70 → SMPL-J14 joint regressor,
+ fitted on 3DPW **train**, that absorbs the rig offset. Standard practice when
+ a method's joint convention differs from the benchmark's. Fit it once, then
+ apply to test:
+
+ ```bash
+ python benchmark/eval_3dpw.py --onnx ... --split train --stride 10 \
+ --fit-adapter benchmark/results/adapter_j14.npz
+ python benchmark/eval_3dpw.py --onnx ... --split test \
+ --adapter benchmark/results/adapter_j14.npz
+ ```
+
+ Fit on train, report on test — never fit on the split you report.
+
+### Train/test contamination — READ THIS BEFORE PUBLISHING
+
+**COCO val2017 is clean.** All 25,594 COCO images in `data/` are
+`COCO_train2014_*`. COCO's 2017 split re-partitions the 2014 data such that
+val2017 is a 5k subset of val2014, which is disjoint from train2014. Verified by
+image-id intersection: **0 of 5000** val2017 images appear in training.
+
+**3DPW test is contaminated.** `data/sam3d_gt_3dpw` holds only train sequences,
+but `data/sam3d_distill_mix` contains teacher-distilled crops from **all 60**
+3DPW sequences — including all 24 test sequences (9,060 person-crops, ~28.5k
+distinct frames across the dataset). `--exclude-seen` quantifies this: it splits
+the report into frames the model saw during training and frames it did not.
+
+Note that even the "not in training data" subset is only *frame*-level held out.
+Neighbouring frames of the same person in the same scene were seen, so it still
+overstates true generalisation. **To publish a 3DPW number, retrain with the 24
+test sequences removed from `sam3d_distill_mix`.** The contamination-split
+report tells you how large the effect is in the meantime.
+
+## Layout
+
+```
+benchmark/
+ download.py COCO fetcher + 3DPW presence check
+ eval_3dpw.py 3D: MPJPE, PA-MPJPE, PCK3D, AUC, per-sequence breakdown
+ eval_coco.py 2D: OKS AP/AR, mean OKS, PCK, NME, per-keypoint breakdown
+ benchlib/
+ joints.py MHR70 <-> COCO17 / SMPL24 index maps, J14 & J12 sets
+ metrics.py Procrustes alignment and the metric definitions
+ threedpw.py 3DPW pickle loading, projection, box derivation
+ runner.py batched ONNX inference with overlapped image decoding
+```
diff --git a/benchmark/benchlib/__init__.py b/benchmark/benchlib/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/benchmark/benchlib/joints.py b/benchmark/benchlib/joints.py
new file mode 100644
index 0000000..f4e08dd
--- /dev/null
+++ b/benchmark/benchlib/joints.py
@@ -0,0 +1,98 @@
+"""Joint-convention bookkeeping for the benchmarks.
+
+InstantHMR regresses 70 keypoints in the MHR70 ordering (see
+``instanthmr.skeleton.JOINT_NAMES``). Public benchmarks use other conventions:
+COCO's 17 keypoints for 2D, and SMPL's 24 body joints for 3DPW. This module
+holds the index maps between them, plus the two evaluation joint sets used for
+3D metrics.
+
+A note that matters for the paper: MHR and SMPL are different rigs, so even a
+perfect prediction carries a systematic offset at joints whose *definition*
+differs (notably ``neck`` and ``head``). The J12 set below contains only the
+limb joints, where both rigs agree on the anatomy; J14 adds neck + head to match
+the joint count published by most 3DPW papers. Report both.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+# --------------------------------------------------------------------------
+# MHR70 (the model's own ordering)
+# --------------------------------------------------------------------------
+# Indices we rely on. Body landmarks 0-14 happen to follow COCO's ordering,
+# but the wrists live out in the hand block at 41 / 62.
+MHR = {
+ "nose": 0, "left_eye": 1, "right_eye": 2, "left_ear": 3, "right_ear": 4,
+ "left_shoulder": 5, "right_shoulder": 6,
+ "left_elbow": 7, "right_elbow": 8,
+ "left_hip": 9, "right_hip": 10,
+ "left_knee": 11, "right_knee": 12,
+ "left_ankle": 13, "right_ankle": 14,
+ "right_wrist": 41, "left_wrist": 62,
+ "neck": 69,
+}
+NUM_MHR_JOINTS = 70
+
+# --------------------------------------------------------------------------
+# COCO 17 keypoints
+# --------------------------------------------------------------------------
+COCO17_NAMES = [
+ "nose", "left_eye", "right_eye", "left_ear", "right_ear",
+ "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
+ "left_wrist", "right_wrist", "left_hip", "right_hip",
+ "left_knee", "right_knee", "left_ankle", "right_ankle",
+]
+# MHR70 index for each COCO keypoint, in COCO order.
+COCO17_FROM_MHR70 = np.array(
+ [MHR[n] for n in COCO17_NAMES], dtype=np.int64
+)
+
+# --------------------------------------------------------------------------
+# SMPL 24 body joints (the ordering of 3DPW's ``jointPositions``)
+# --------------------------------------------------------------------------
+SMPL24_NAMES = [
+ "pelvis", "left_hip", "right_hip", "spine1", "left_knee", "right_knee",
+ "spine2", "left_ankle", "right_ankle", "spine3", "left_foot", "right_foot",
+ "neck", "left_collar", "right_collar", "head", "left_shoulder",
+ "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist",
+ "left_hand", "right_hand",
+]
+SMPL = {n: i for i, n in enumerate(SMPL24_NAMES)}
+
+# --------------------------------------------------------------------------
+# 3D evaluation joint sets
+# --------------------------------------------------------------------------
+# LSP-14 ordering, the joint set 3DPW papers report on.
+J14_NAMES = [
+ "right_ankle", "right_knee", "right_hip",
+ "left_hip", "left_knee", "left_ankle",
+ "right_wrist", "right_elbow", "right_shoulder",
+ "left_shoulder", "left_elbow", "left_wrist",
+ "neck", "head",
+]
+# The first 12 are limbs — anatomically the same landmark in both rigs.
+J12_NAMES = J14_NAMES[:12]
+
+# MHR70 has no head-top joint; ``nose`` is the closest available landmark and is
+# only used for the J14 numbers (flagged as approximate in the report).
+_MHR_FOR_J14 = dict(MHR, head=MHR["nose"])
+
+J14_FROM_MHR70 = np.array([_MHR_FOR_J14[n] for n in J14_NAMES], dtype=np.int64)
+J14_FROM_SMPL24 = np.array([SMPL[n] for n in J14_NAMES], dtype=np.int64)
+J12_FROM_MHR70 = J14_FROM_MHR70[:12]
+J12_FROM_SMPL24 = J14_FROM_SMPL24[:12]
+
+# Pelvis for root alignment: the mid-hip, computed the same way on both sides.
+J14_HIP_IDX = (J14_NAMES.index("left_hip"), J14_NAMES.index("right_hip"))
+
+JOINT_SETS = {
+ "J14": dict(names=J14_NAMES, mhr=J14_FROM_MHR70, smpl=J14_FROM_SMPL24),
+ "J12": dict(names=J12_NAMES, mhr=J12_FROM_MHR70, smpl=J12_FROM_SMPL24),
+}
+
+
+def pelvis(joints: np.ndarray) -> np.ndarray:
+ """Mid-hip root for a (..., J, 3) array indexed in a J14/J12 joint set."""
+ li, ri = J14_HIP_IDX
+ return 0.5 * (joints[..., li, :] + joints[..., ri, :])
diff --git a/benchmark/benchlib/metrics.py b/benchmark/benchlib/metrics.py
new file mode 100644
index 0000000..1867e39
--- /dev/null
+++ b/benchmark/benchlib/metrics.py
@@ -0,0 +1,108 @@
+"""3D and 2D pose metrics.
+
+All 3D functions take arrays in millimetres and return millimetres.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+# --------------------------------------------------------------------------
+# 3D
+# --------------------------------------------------------------------------
+def similarity_transform(pred: np.ndarray, gt: np.ndarray) -> np.ndarray:
+ """Procrustes-align ``pred`` onto ``gt`` (scale + rotation + translation).
+
+ Args:
+ pred: (N, J, 3) predicted joints.
+ gt: (N, J, 3) ground-truth joints.
+
+ Returns:
+ (N, J, 3) the aligned prediction.
+ """
+ pred = np.asarray(pred, dtype=np.float64)
+ gt = np.asarray(gt, dtype=np.float64)
+
+ mu_p = pred.mean(axis=1, keepdims=True)
+ mu_g = gt.mean(axis=1, keepdims=True)
+ X, Y = pred - mu_p, gt - mu_g
+
+ # Per-sample covariance of the centred point sets.
+ K = np.einsum("nji,njk->nik", X, Y)
+ U, S, Vt = np.linalg.svd(K)
+
+ # Reflection guard: force det(R) = +1 so we never mirror the skeleton.
+ Z = np.zeros_like(U) + np.eye(3)
+ Z[:, 2, 2] = np.sign(np.linalg.det(np.einsum("nij,njk->nik", U, Vt)))
+ R = np.einsum("nij,njk,nlk->nil", Vt.transpose(0, 2, 1), Z, U)
+
+ var_x = (X ** 2).sum(axis=(1, 2))
+ trace = (S * np.diagonal(Z, axis1=1, axis2=2)).sum(axis=1)
+ scale = np.where(var_x > 0, trace / np.maximum(var_x, 1e-12), 1.0)
+
+ aligned = scale[:, None, None] * np.einsum("nij,nkj->nki", R, X) + mu_g
+ return aligned
+
+
+def per_joint_error(pred: np.ndarray, gt: np.ndarray) -> np.ndarray:
+ """(N, J) Euclidean distance per joint."""
+ return np.linalg.norm(np.asarray(pred) - np.asarray(gt), axis=-1)
+
+
+def mpjpe(pred: np.ndarray, gt: np.ndarray, root: np.ndarray | None = None) -> np.ndarray:
+ """Root-aligned MPJPE, (N, J) per-joint errors.
+
+ ``root`` is an (N, 3) origin subtracted from both sets. Pass ``None`` to
+ compare without any alignment (i.e. absolute-position error).
+ """
+ if root is not None:
+ pred = pred - root[0][:, None, :]
+ gt = gt - root[1][:, None, :]
+ return per_joint_error(pred, gt)
+
+
+def pa_mpjpe(pred: np.ndarray, gt: np.ndarray) -> np.ndarray:
+ """Procrustes-aligned MPJPE, (N, J) per-joint errors."""
+ return per_joint_error(similarity_transform(pred, gt), gt)
+
+
+def pck3d(errors: np.ndarray, threshold: float) -> float:
+ """Fraction of joints under ``threshold`` mm."""
+ return float((errors < threshold).mean())
+
+
+def auc3d(errors: np.ndarray, lo: float = 0.0, hi: float = 150.0,
+ steps: int = 31) -> float:
+ """Area under the PCK-vs-threshold curve over [lo, hi] mm."""
+ ts = np.linspace(lo, hi, steps)
+ return float(np.mean([pck3d(errors, t) for t in ts]))
+
+
+# --------------------------------------------------------------------------
+# 2D
+# --------------------------------------------------------------------------
+def pck2d(pred: np.ndarray, gt: np.ndarray, valid: np.ndarray,
+ norm: np.ndarray, threshold: float) -> float:
+ """2D PCK at ``threshold`` * ``norm``.
+
+ Args:
+ pred: (N, J, 2) predicted pixel coords.
+ gt: (N, J, 2) ground-truth pixel coords.
+ valid: (N, J) bool, which joints are labelled.
+ norm: (N,) per-sample normaliser in pixels (e.g. bbox diagonal).
+ threshold: fraction of ``norm`` counted as correct.
+ """
+ d = np.linalg.norm(pred - gt, axis=-1) / np.maximum(norm[:, None], 1e-6)
+ if not valid.any():
+ return float("nan")
+ return float((d[valid] < threshold).mean())
+
+
+def nme2d(pred: np.ndarray, gt: np.ndarray, valid: np.ndarray,
+ norm: np.ndarray) -> float:
+ """Normalised mean error: mean pixel distance / ``norm``."""
+ d = np.linalg.norm(pred - gt, axis=-1) / np.maximum(norm[:, None], 1e-6)
+ if not valid.any():
+ return float("nan")
+ return float(d[valid].mean())
diff --git a/benchmark/benchlib/runner.py b/benchmark/benchlib/runner.py
new file mode 100644
index 0000000..cf88fbb
--- /dev/null
+++ b/benchmark/benchlib/runner.py
@@ -0,0 +1,119 @@
+"""Batched InstantHMR inference over a flat list of (image, bbox) samples.
+
+The public API is :func:`run_samples`. It groups samples by source image so
+each frame is decoded once and all its people go through the ONNX graph in a
+single call, and it overlaps JPEG decoding with GPU work via a small prefetch
+pool. Predictions come back in the input order.
+"""
+
+from __future__ import annotations
+
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from pathlib import Path
+
+import cv2
+import numpy as np
+from tqdm import tqdm
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+if str(REPO_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPO_ROOT))
+
+from instanthmr import InstantHMR # noqa: E402
+
+
+@dataclass
+class Sample:
+ """One person to evaluate."""
+ image_path: str
+ bbox: np.ndarray # (4,) xyxy in original image pixels
+ meta: dict # carried through untouched
+
+
+def build_model(onnx_path: str, device: str = "cuda",
+ mhr_script: str | None = None) -> InstantHMR:
+ """Create the ONNX session, attaching the MHR rig when the graph needs it.
+
+ The ``mhr_only`` graphs have no 3D head: their joints are derived by running
+ the predicted MHR parameters through the rig, so a backend is mandatory.
+ """
+ from instanthmr.inference import onnx_output_names
+
+ names = onnx_output_names(onnx_path) or []
+ mhr = None
+ if "joints_3d" not in names:
+ from instanthmr.mhr_renderer import build_mhr
+ script = mhr_script or str(REPO_ROOT / "checkpoints" / "mhr_model.pt")
+ if not Path(script).exists():
+ raise FileNotFoundError(
+ f"this graph has no joints_3d output and needs the MHR rig, "
+ f"but {script} is missing"
+ )
+ mhr = build_mhr(script_path=script, device=device)
+ return InstantHMR(onnx_path, device=device, mhr=mhr)
+
+
+def _group_by_image(samples: list[Sample]) -> list[tuple[str, list[int]]]:
+ """Group sample indices by image path, preserving first-seen order."""
+ order: list[str] = []
+ groups: dict[str, list[int]] = {}
+ for i, s in enumerate(samples):
+ if s.image_path not in groups:
+ groups[s.image_path] = []
+ order.append(s.image_path)
+ groups[s.image_path].append(i)
+ return [(p, groups[p]) for p in order]
+
+
+def _load_rgb(path: str) -> np.ndarray | None:
+ img = cv2.imread(path, cv2.IMREAD_COLOR)
+ if img is None:
+ return None
+ return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+
+
+def run_samples(model: InstantHMR, samples: list[Sample],
+ num_workers: int = 6, desc: str = "inference"):
+ """Run the model over every sample.
+
+ Returns:
+ ``(predictions, ok)`` where ``predictions`` is a list the same length as
+ ``samples`` (``None`` where the image could not be read) and ``ok`` is a
+ boolean mask of the usable entries.
+ """
+ groups = _group_by_image(samples)
+ preds: list[object | None] = [None] * len(samples)
+
+ with ThreadPoolExecutor(max_workers=num_workers) as pool:
+ # Keep a bounded window of decoded frames in flight ahead of the GPU.
+ window = max(2 * num_workers, 8)
+ pending: dict[int, object] = {}
+ next_submit = 0
+
+ def submit_until(limit: int) -> None:
+ nonlocal next_submit
+ while next_submit < min(limit, len(groups)):
+ pending[next_submit] = pool.submit(_load_rgb, groups[next_submit][0])
+ next_submit += 1
+
+ submit_until(window)
+ for gi in tqdm(range(len(groups)), desc=desc, unit="img"):
+ path, idxs = groups[gi]
+ image = pending.pop(gi).result()
+ submit_until(gi + 1 + window)
+ if image is None:
+ continue
+ dets = [{"bbox": samples[i].bbox, "confidence": 1.0} for i in idxs]
+ out = model.predict_batch(image, dets)
+ for i, p in zip(idxs, out):
+ preds[i] = p
+
+ ok = np.array([p is not None for p in preds], dtype=bool)
+ return preds, ok
+
+
+def stack_joints(preds, ok, attr: str) -> np.ndarray:
+ """Stack one joint array (e.g. ``joints_3d_local``) over the valid preds."""
+ return np.stack([getattr(p, attr) for p, k in zip(preds, ok) if k]).astype(np.float64)
diff --git a/benchmark/benchlib/threedpw.py b/benchmark/benchlib/threedpw.py
new file mode 100644
index 0000000..d5a4417
--- /dev/null
+++ b/benchmark/benchlib/threedpw.py
@@ -0,0 +1,180 @@
+"""3DPW ground truth loading.
+
+3DPW ships per-sequence pickles under ``sequenceFiles/<split>/<seq>.pkl``. The
+useful fields for us:
+
+ jointPositions list[P] of (F, 72) 24 SMPL joints, **world** coords, metres
+ cam_poses (F, 4, 4) world -> camera extrinsics (OpenCV frame)
+ cam_intrinsics (3, 3) shared camera matrix
+ campose_valid list[P] of (F,) 1 where the camera pose is trustworthy
+ genders list[P]
+
+Because ``jointPositions`` is stored directly, evaluating 3D joint error needs
+**no SMPL model download** — we never have to re-run the body model.
+"""
+
+from __future__ import annotations
+
+import pickle
+from pathlib import Path
+
+import numpy as np
+
+TEST_SEQUENCES = 24 # sanity check: the official test split has 24 sequences
+
+
+def load_sequence(pkl_path: str | Path) -> dict:
+ with open(pkl_path, "rb") as f:
+ return pickle.load(f, encoding="latin1")
+
+
+def project(points_cam: np.ndarray, K: np.ndarray) -> np.ndarray:
+ """Pinhole-project (..., 3) camera-space points to (..., 2) pixels."""
+ z = np.maximum(points_cam[..., 2:3], 1e-6)
+ uv = points_cam[..., :2] / z
+ return uv * np.array([K[0, 0], K[1, 1]]) + np.array([K[0, 2], K[1, 2]])
+
+
+def bbox_from_points(uv: np.ndarray, scale: float) -> np.ndarray:
+ """Square-ish xyxy box around 2D points, expanded by ``scale``.
+
+ The GT joints stop at the wrists/ankles and the head joint sits inside the
+ skull, so the raw joint extent is tighter than a person box. ``scale``
+ (default 1.2 at the call site) pads it out to roughly a detector box.
+ """
+ x1, y1 = uv.min(axis=0)
+ x2, y2 = uv.max(axis=0)
+ cx, cy = (x1 + x2) / 2.0, (y1 + y2) / 2.0
+ w, h = (x2 - x1) * scale, (y2 - y1) * scale
+ return np.array([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2],
+ dtype=np.float32)
+
+
+def build_samples(sequence_dir: str | Path, image_root: str | Path,
+ split: str = "test", stride: int = 1,
+ bbox_scale: float = 1.2,
+ min_visible_frac: float = 0.6):
+ """Enumerate every evaluatable (frame, person) in a 3DPW split.
+
+ Args:
+ sequence_dir: ``sequenceFiles`` directory.
+ image_root: ``imageFiles`` directory.
+ split: ``test`` / ``train`` / ``validation``.
+ stride: keep every Nth frame (1 = all frames, the standard protocol).
+ bbox_scale: padding applied to the projected-joint box.
+ min_visible_frac: skip a person when fewer than this fraction of their
+ projected joints fall inside the image — they are mostly out of
+ frame, and a crop around them is meaningless.
+
+ Returns:
+ ``(samples, gt_joints)`` — a list of dicts and an (N, 24, 3) array of
+ camera-space GT joints in metres, index-aligned.
+ """
+ seq_dir = Path(sequence_dir) / split
+ if not seq_dir.is_dir():
+ raise FileNotFoundError(f"no such 3DPW split directory: {seq_dir}")
+ image_root = Path(image_root)
+
+ samples: list[dict] = []
+ gts: list[np.ndarray] = []
+
+ for pkl in sorted(seq_dir.glob("*.pkl")):
+ seq = load_sequence(pkl)
+ name = seq["sequence"]
+ K = np.asarray(seq["cam_intrinsics"], dtype=np.float64)
+ cam_poses = np.asarray(seq["cam_poses"], dtype=np.float64)
+ num_frames = cam_poses.shape[0]
+
+ for pid, jp in enumerate(seq["jointPositions"]):
+ joints_w = np.asarray(jp, dtype=np.float64).reshape(-1, 24, 3)
+ valid = np.asarray(seq["campose_valid"][pid]).astype(bool)
+
+ for f in range(0, min(num_frames, joints_w.shape[0]), stride):
+ if not valid[f]:
+ continue
+ T = cam_poses[f]
+ joints_cam = joints_w[f] @ T[:3, :3].T + T[:3, 3]
+ if (joints_cam[:, 2] <= 0).any(): # behind the camera
+ continue
+
+ uv = project(joints_cam, K)
+ img_path = image_root / name / f"image_{f:05d}.jpg"
+ samples.append(dict(
+ image_path=str(img_path),
+ bbox=bbox_from_points(uv, bbox_scale),
+ uv=uv.astype(np.float32),
+ sequence=name, person=pid, frame=f,
+ gender=str(seq["genders"][pid]),
+ ))
+ gts.append(joints_cam)
+
+ if not samples:
+ raise RuntimeError(f"no valid samples found in {seq_dir}")
+
+ # Drop people who are mostly outside the frame; needs image size, which we
+ # take from the first image of each sequence (3DPW is constant-resolution).
+ return _filter_offscreen(samples, np.stack(gts), image_root, min_visible_frac)
+
+
+def _filter_offscreen(samples, gts, image_root: Path, min_visible_frac: float):
+ import cv2
+
+ sizes: dict[str, tuple[int, int]] = {}
+ keep = np.ones(len(samples), dtype=bool)
+ for i, s in enumerate(samples):
+ seq = s["sequence"]
+ if seq not in sizes:
+ img = cv2.imread(s["image_path"], cv2.IMREAD_COLOR)
+ if img is None:
+ sizes[seq] = (0, 0)
+ else:
+ sizes[seq] = img.shape[:2]
+ h, w = sizes[seq]
+ if h == 0:
+ keep[i] = False
+ continue
+ uv = s["uv"]
+ inside = ((uv[:, 0] >= 0) & (uv[:, 0] < w)
+ & (uv[:, 1] >= 0) & (uv[:, 1] < h))
+ if inside.mean() < min_visible_frac:
+ keep[i] = False
+
+ samples = [s for s, k in zip(samples, keep) if k]
+ return samples, gts[keep]
+
+
+# --------------------------------------------------------------------------
+# Train/test contamination
+# --------------------------------------------------------------------------
+# The distillation folders store one npz per person-crop, named after the source
+# frame. Two layouts appear in this repo:
+# data/sam3d_distill_mix : imageFiles_<sequence>_image_<frame>_p<n>.npz
+# data/sam3d_gt_3dpw : 3dpw_<sequence>_image_<frame>_p<n>.npz
+_SEEN_PATTERNS = (
+ r"imageFiles_(?P<seq>.+?)_image_(?P<frame>\d+)_p\d+\.npz$",
+ r"3dpw_(?P<seq>.+?)_image_(?P<frame>\d+)_p\d+\.npz$",
+)
+
+
+def seen_frames(training_dirs) -> set[tuple[str, int]]:
+ """Collect every (sequence, frame) that appears in the training data.
+
+ Used to quantify — and optionally exclude — 3DPW frames the model was
+ trained on. Returns an empty set when no directory exists.
+ """
+ import os
+ import re
+
+ compiled = [re.compile(p) for p in _SEEN_PATTERNS]
+ seen: set[tuple[str, int]] = set()
+ for d in training_dirs:
+ d = Path(d)
+ if not d.is_dir():
+ continue
+ for fname in os.listdir(d):
+ for rx in compiled:
+ m = rx.match(fname)
+ if m:
+ seen.add((m.group("seq"), int(m.group("frame"))))
+ break
+ return seen
diff --git a/benchmark/download.py b/benchmark/download.py
new file mode 100644
index 0000000..32eb140
--- /dev/null
+++ b/benchmark/download.py
@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""Fetch the benchmark data.
+
+COCO val2017 is a direct public download and is handled fully automatically.
+3DPW requires accepting a licence on the MPI page, so this script cannot fetch
+it for you — it verifies what you already have and tells you what is missing.
+
+ python benchmark/download.py --coco
+ python benchmark/download.py --check-3dpw --sequence-dir ... --image-root ...
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+import urllib.request
+import zipfile
+from pathlib import Path
+
+COCO_FILES = [
+ ("http://images.cocodataset.org/annotations/annotations_trainval2017.zip",
+ "annotations_trainval2017.zip", "annotations/person_keypoints_val2017.json"),
+ ("http://images.cocodataset.org/zips/val2017.zip",
+ "val2017.zip", "val2017"),
+]
+
+THREEDPW_HELP = """
+3DPW must be downloaded from the official page (free, licence form):
+
+ https://virtualhumans.mpi-inf.mpg.de/3DPW/licence.html
+
+Fill the form; the confirmation mail carries the download links. You need:
+
+ sequenceFiles.zip ~ 30 MB <-- the ground truth (required)
+ imageFiles.zip ~ 4.6 GB <-- the RGB frames
+
+Unzip so the layout is:
+
+ <3dpw_root>/sequenceFiles/{train,validation,test}/*.pkl
+ <3dpw_root>/imageFiles/<sequence>/image_XXXXX.jpg
+"""
+
+
+def _progress(block_num, block_size, total):
+ if total <= 0:
+ return
+ done = min(block_num * block_size, total)
+ pct = 100.0 * done / total
+ sys.stdout.write(f"\r {pct:5.1f}% ({done/1e6:.0f} / {total/1e6:.0f} MB)")
+ sys.stdout.flush()
+
+
+def download_coco(root: Path) -> None:
+ root.mkdir(parents=True, exist_ok=True)
+ for url, zip_name, marker in COCO_FILES:
+ if (root / marker).exists():
+ print(f"[coco] {marker} already present — skipping")
+ continue
+ zip_path = root / zip_name
+ if not zip_path.exists():
+ print(f"[coco] downloading {url}")
+ urllib.request.urlretrieve(url, zip_path, _progress)
+ print()
+ print(f"[coco] extracting {zip_name}")
+ with zipfile.ZipFile(zip_path) as z:
+ # The annotations zip carries train2017 too; we only need val.
+ members = [m for m in z.namelist()
+ if "train2017" not in m or "val" in m]
+ z.extractall(root, members=members)
+ zip_path.unlink()
+ print(f"[coco] ok -> {root / marker}")
+ print(f"\n[coco] ready. Use --coco-root {root}")
+
+
+def check_3dpw(sequence_dir: Path | None, image_root: Path | None) -> bool:
+ ok = True
+ if sequence_dir is None or not sequence_dir.is_dir():
+ print(f"[3dpw] MISSING sequenceFiles: {sequence_dir}")
+ ok = False
+ else:
+ for split, expected in (("test", 24), ("validation", 12), ("train", 24)):
+ pkls = list((sequence_dir / split).glob("*.pkl"))
+ flag = "ok " if pkls else "MISSING"
+ print(f"[3dpw] {flag} sequenceFiles/{split:<11s} "
+ f"{len(pkls)} pkl (official: {expected})")
+ if split == "test" and not pkls:
+ ok = False
+ if image_root is None or not image_root.is_dir():
+ print(f"[3dpw] MISSING imageFiles: {image_root}")
+ ok = False
+ else:
+ seqs = [d for d in image_root.iterdir() if d.is_dir()]
+ print(f"[3dpw] ok imageFiles {len(seqs)} sequences "
+ f"(official: 60)")
+ if not ok:
+ print(THREEDPW_HELP)
+ else:
+ print("\n[3dpw] ready.")
+ return ok
+
+
+def main():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--coco", action="store_true", help="download COCO val2017")
+ p.add_argument("--coco-root", default="benchmark/data/coco")
+ p.add_argument("--check-3dpw", action="store_true")
+ p.add_argument("--sequence-dir", default=None)
+ p.add_argument("--image-root", default=None)
+ a = p.parse_args()
+
+ if a.coco:
+ download_coco(Path(a.coco_root))
+ if a.check_3dpw:
+ check_3dpw(
+ Path(a.sequence_dir) if a.sequence_dir else None,
+ Path(a.image_root) if a.image_root else None,
+ )
+ if not (a.coco or a.check_3dpw):
+ p.print_help()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.py
new file mode 100644
index 0000000..33f770f
--- /dev/null
+++ b/benchmark/eval_3dpw.py
@@ -0,0 +1,234 @@
+#!/usr/bin/env python3
+"""3DPW test-set evaluation for InstantHMR (MPJPE / PA-MPJPE / PCK / AUC).
+
+The GT comes straight from the 3DPW pickles' ``jointPositions`` field, so no
+SMPL model download is needed. Predicted MHR70 joints are mapped onto the
+SMPL joint convention by name; see ``benchlib/joints.py`` for the caveat about
+``neck`` and ``head``, which are the only two joints where the two rigs disagree
+on the underlying anatomy.
+
+Usage:
+ python benchmark/eval_3dpw.py \
+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --image-root /path/to/3DPW/imageFiles
+
+Add ``--stride 10`` for a fast smoke run over 1/10th of the frames.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from benchlib import joints as J
+from benchlib import metrics as M
+from benchlib import threedpw as P
+from benchlib.runner import Sample, build_model, run_samples, stack_joints
+
+MM = 1000.0 # metres -> millimetres
+
+
+def parse_args():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--onnx", required=True, help="path to the exported graph")
+ p.add_argument("--sequence-dir", required=True, help="3DPW sequenceFiles/")
+ p.add_argument("--image-root", required=True, help="3DPW imageFiles/")
+ p.add_argument("--split", default="test",
+ choices=["test", "train", "validation"])
+ p.add_argument("--stride", type=int, default=1,
+ help="evaluate every Nth frame (1 = full protocol)")
+ p.add_argument("--bbox-scale", type=float, default=1.2,
+ help="padding on the projected-GT-joint person box")
+ p.add_argument("--device", default="cuda")
+ p.add_argument("--mhr-script", default=None)
+ p.add_argument("--num-workers", type=int, default=6)
+ p.add_argument("--limit", type=int, default=0, help="cap sample count (debug)")
+ p.add_argument("--out", default="benchmark/results/3dpw.json")
+ p.add_argument("--adapter", default=None,
+ help="npz with a fitted MHR70->J14 joint regressor to also report")
+ p.add_argument("--exclude-seen", nargs="*", default=None,
+ metavar="DIR",
+ help="training annotation dirs; frames found there are "
+ "reported separately as a held-out subset")
+ p.add_argument("--fit-adapter", default=None,
+ help="fit that regressor on THIS split and write it here "
+ "(run on train/validation, never on test)")
+ return p.parse_args()
+
+
+def evaluate(pred_mhr: np.ndarray, gt_smpl: np.ndarray, set_name: str) -> dict:
+ """Metrics for one joint set. Inputs are (N, 70, 3) and (N, 24, 3) in mm."""
+ spec = J.JOINT_SETS[set_name]
+ pred = pred_mhr[:, spec["mhr"], :]
+ gt = gt_smpl[:, spec["smpl"], :]
+
+ root_p, root_g = J.pelvis(pred), J.pelvis(gt)
+ e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
+ e_pa = M.pa_mpjpe(pred, gt)
+
+ return dict(
+ joint_set=set_name,
+ num_joints=len(spec["names"]),
+ num_samples=int(pred.shape[0]),
+ MPJPE_mm=float(e_mpjpe.mean()),
+ PA_MPJPE_mm=float(e_pa.mean()),
+ PCK3D_50mm=M.pck3d(e_pa, 50.0),
+ PCK3D_100mm=M.pck3d(e_pa, 100.0),
+ AUC_0_150mm=M.auc3d(e_pa),
+ per_joint_PA_MPJPE_mm={
+ n: float(v) for n, v in zip(spec["names"], e_pa.mean(axis=0))
+ },
+ )
+
+
+def fit_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray) -> np.ndarray:
+ """Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.
+
+ Fitted on centred poses and used without a bias, so the mapping stays
+ equivariant to rotation and translation — it only removes the systematic
+ difference in where the two rigs place each landmark.
+ """
+ gt = gt_smpl[:, J.J14_FROM_SMPL24, :]
+ X = pred_mhr - pred_mhr.mean(axis=1, keepdims=True) # (N, 70, 3)
+ Y = gt - gt.mean(axis=1, keepdims=True) # (N, 14, 3)
+ # Flatten samples and coordinates into the row dimension.
+ A = X.transpose(0, 2, 1).reshape(-1, X.shape[1]) # (N*3, 70)
+ B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1]) # (N*3, 14)
+ W, *_ = np.linalg.lstsq(A, B, rcond=None)
+ return W.T # (14, 70)
+
+
+def evaluate_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray,
+ W: np.ndarray) -> dict:
+ pred = np.einsum("jk,nkc->njc", W, pred_mhr)
+ gt = gt_smpl[:, J.J14_FROM_SMPL24, :]
+ root_p, root_g = J.pelvis(pred), J.pelvis(gt)
+ e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
+ e_pa = M.pa_mpjpe(pred, gt)
+ return dict(
+ joint_set="J14+adapter", num_joints=14,
+ num_samples=int(pred.shape[0]),
+ MPJPE_mm=float(e_mpjpe.mean()),
+ PA_MPJPE_mm=float(e_pa.mean()),
+ PCK3D_50mm=M.pck3d(e_pa, 50.0),
+ PCK3D_100mm=M.pck3d(e_pa, 100.0),
+ AUC_0_150mm=M.auc3d(e_pa),
+ )
+
+
+def main():
+ args = parse_args()
+
+ print(f"[3DPW] loading GT from {args.sequence_dir} ({args.split} split)")
+ samples, gt_joints = P.build_samples(
+ args.sequence_dir, args.image_root, split=args.split,
+ stride=args.stride, bbox_scale=args.bbox_scale,
+ )
+ if args.limit:
+ samples, gt_joints = samples[:args.limit], gt_joints[:args.limit]
+ seqs = sorted({s["sequence"] for s in samples})
+ print(f"[3DPW] {len(samples)} person-frames across {len(seqs)} sequences")
+
+ model = build_model(args.onnx, args.device, args.mhr_script)
+ preds, ok = run_samples(
+ model,
+ [Sample(s["image_path"], s["bbox"], s) for s in samples],
+ num_workers=args.num_workers, desc="3DPW",
+ )
+ if not ok.all():
+ print(f"[3DPW] WARNING: {int((~ok).sum())} samples had unreadable images")
+
+ # joints_3d_local is the body-centred rig pose; root alignment and
+ # Procrustes both make the camera translation irrelevant.
+ pred_mhr = stack_joints(preds, ok, "joints_3d_local") * MM
+ gt_smpl = gt_joints[ok] * MM
+
+ kept_meta = [s for s, k in zip(samples, ok) if k]
+ W_adapt = np.load(args.adapter)["W"] if args.adapter else None
+
+ def _rows(pm, gm):
+ rows = [evaluate(pm, gm, s) for s in ("J14", "J12")]
+ if W_adapt is not None:
+ rows.append(evaluate_adapter(pm, gm, W_adapt))
+ return rows
+ report = dict(
+ model=str(args.onnx), dataset="3DPW", split=args.split,
+ stride=args.stride, bbox_scale=args.bbox_scale,
+ num_sequences=len(seqs), sequences=seqs,
+ num_samples=int(pred_mhr.shape[0]),
+ results=_rows(pred_mhr, gt_smpl),
+ )
+
+ # Frames present in the training data are leaked; score them separately so
+ # the honest number is visible next to the optimistic one.
+ if args.exclude_seen:
+ seen = P.seen_frames(args.exclude_seen)
+ is_seen = np.array([(m["sequence"], m["frame"]) in seen for m in kept_meta])
+ report["contamination"] = dict(
+ training_dirs=list(args.exclude_seen),
+ seen_frames=int(is_seen.sum()),
+ unseen_frames=int((~is_seen).sum()),
+ seen_fraction=float(is_seen.mean()),
+ seen_sequences=sorted({m["sequence"] for m, f in zip(kept_meta, is_seen) if f}),
+ )
+ if (~is_seen).any():
+ report["results_unseen_frames"] = _rows(pred_mhr[~is_seen], gt_smpl[~is_seen])
+ if is_seen.any():
+ report["results_seen_frames"] = _rows(pred_mhr[is_seen], gt_smpl[is_seen])
+
+ if args.fit_adapter:
+ W = fit_adapter(pred_mhr, gt_smpl)
+ Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)
+ np.savez(args.fit_adapter, W=W, split=args.split)
+ print(f"[3DPW] fitted joint adapter on '{args.split}' -> {args.fit_adapter}")
+
+ # Per-sequence PA-MPJPE on the headline joint set.
+ spec = J.JOINT_SETS["J14"]
+ e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :], gt_smpl[:, spec["smpl"], :]).mean(axis=1)
+ kept = [s for s, k in zip(samples, ok) if k]
+ per_seq = {}
+ for name in seqs:
+ m = np.array([s["sequence"] == name for s in kept])
+ if m.any():
+ per_seq[name] = dict(n=int(m.sum()), PA_MPJPE_mm=float(e[m].mean()))
+ report["per_sequence_J14"] = per_seq
+
+ out = Path(args.out)
+ out.parent.mkdir(parents=True, exist_ok=True)
+ out.write_text(json.dumps(report, indent=2))
+
+ def _table(title, rows):
+ print(f"\n{title}")
+ print(f" {'joint set':<14s} {'N':>7s} {'MPJPE':>10s} {'PA-MPJPE':>10s}")
+ for r in rows:
+ print(f" {r['joint_set']:<14s} {r['num_samples']:7d} "
+ f"{r['MPJPE_mm']:9.1f} {r['PA_MPJPE_mm']:9.1f}")
+
+ print("\n" + "=" * 60)
+ print(f"3DPW {args.split} | {report['num_samples']} person-frames "
+ f"| {len(seqs)} sequences (mm)")
+ print("=" * 60)
+ _table("ALL frames:", report["results"])
+ if "results_unseen_frames" in report:
+ c = report["contamination"]
+ _table(f"Frames NOT in training data "
+ f"({c['unseen_frames']}/{report['num_samples']}, "
+ f"{100*(1-c['seen_fraction']):.1f}%):",
+ report["results_unseen_frames"])
+ if "results_seen_frames" in report:
+ _table("Frames SEEN in training (leaked — for reference only):",
+ report["results_seen_frames"])
+ print("=" * 60)
+ print(f"written to {out}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/eval_coco.py b/benchmark/eval_coco.py
new file mode 100644
index 0000000..1111763
--- /dev/null
+++ b/benchmark/eval_coco.py
@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+"""COCO val2017 keypoint evaluation for InstantHMR (OKS AP/AR, PCK, NME).
+
+This is a *top-down with ground-truth boxes* protocol: one crop per annotated
+person, so the numbers measure pose accuracy alone and are not comparable to
+detector-in-the-loop AP from the COCO leaderboard. It is the standard ablation
+protocol for single-person pose regressors, and it is the honest one for a
+model that never sees a detector at eval time.
+
+InstantHMR regresses joint *positions* with no per-keypoint confidence, so
+every predicted keypoint is emitted with score 1.0. That is fine for OKS AP
+here (exactly one prediction per GT person) but it means AP cannot be improved
+by confidence ranking the way heatmap methods do — read AP alongside the mean
+OKS and PCK numbers, which are confidence-free.
+
+Usage:
+ python benchmark/eval_coco.py \
+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \
+ --coco-root benchmark/data/coco
+"""
+
+from __future__ import annotations
+
+import argparse
+import contextlib
+import io
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from benchlib import joints as J
+from benchlib import metrics as M
+from benchlib.runner import Sample, build_model, run_samples
+
+# Official COCO per-keypoint OKS tolerances.
+OKS_SIGMAS = np.array(
+ [.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62,
+ 1.07, 1.07, .87, .87, .89, .89]
+) / 10.0
+
+
+def parse_args():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--onnx", required=True)
+ p.add_argument("--coco-root", default="benchmark/data/coco",
+ help="dir holding val2017/ and annotations/")
+ p.add_argument("--ann", default=None,
+ help="override path to person_keypoints_val2017.json")
+ p.add_argument("--images", default=None, help="override path to val2017/")
+ p.add_argument("--min-keypoints", type=int, default=1,
+ help="skip GT persons with fewer labelled keypoints")
+ p.add_argument("--device", default="cuda")
+ p.add_argument("--mhr-script", default=None)
+ p.add_argument("--num-workers", type=int, default=6)
+ p.add_argument("--limit", type=int, default=0)
+ p.add_argument("--out", default="benchmark/results/coco.json")
+ return p.parse_args()
+
+
+def oks(pred: np.ndarray, gt: np.ndarray, vis: np.ndarray,
+ area: np.ndarray) -> np.ndarray:
+ """Per-sample OKS. pred/gt are (N, 17, 2), vis (N, 17), area (N,)."""
+ d2 = ((pred - gt) ** 2).sum(axis=-1)
+ var = (2 * OKS_SIGMAS) ** 2
+ e = d2 / var[None, :] / (area[:, None] + np.spacing(1)) / 2.0
+ m = vis > 0
+ out = np.full(len(pred), np.nan)
+ has = m.any(axis=1)
+ out[has] = np.array([np.exp(-e[i][m[i]]).mean() for i in np.where(has)[0]])
+ return out
+
+
+def main():
+ args = parse_args()
+ from pycocotools.coco import COCO
+ from pycocotools.cocoeval import COCOeval
+
+ root = Path(args.coco_root)
+ ann_file = Path(args.ann) if args.ann else root / "annotations" / "person_keypoints_val2017.json"
+ img_dir = Path(args.images) if args.images else root / "val2017"
+ for pth in (ann_file, img_dir):
+ if not pth.exists():
+ raise FileNotFoundError(
+ f"{pth} not found — run: python benchmark/download.py --coco"
+ )
+
+ print(f"[COCO] loading {ann_file}")
+ with contextlib.redirect_stdout(io.StringIO()):
+ coco = COCO(str(ann_file))
+
+ samples: list[Sample] = []
+ for ann_id in sorted(coco.getAnnIds(catIds=[1], iscrowd=False)):
+ a = coco.anns[ann_id]
+ if a["num_keypoints"] < args.min_keypoints or a["area"] <= 0:
+ continue
+ x, y, w, h = a["bbox"]
+ if w <= 1 or h <= 1:
+ continue
+ info = coco.imgs[a["image_id"]]
+ samples.append(Sample(
+ image_path=str(img_dir / info["file_name"]),
+ bbox=np.array([x, y, x + w, y + h], dtype=np.float32),
+ meta=dict(ann_id=ann_id, image_id=a["image_id"],
+ keypoints=np.array(a["keypoints"], dtype=np.float64).reshape(17, 3),
+ area=float(a["area"]), bbox_wh=(float(w), float(h))),
+ ))
+ # Group people from the same image together so each JPEG is decoded once.
+ samples.sort(key=lambda s: s.image_path)
+ if args.limit:
+ samples = samples[:args.limit]
+ print(f"[COCO] {len(samples)} annotated persons")
+
+ model = build_model(args.onnx, args.device, args.mhr_script)
+ preds, ok = run_samples(model, samples, num_workers=args.num_workers, desc="COCO")
+ if not ok.all():
+ print(f"[COCO] WARNING: {int((~ok).sum())} samples had unreadable images")
+
+ kept = [s for s, k in zip(samples, ok) if k]
+ pred_kp = np.stack([
+ p.joints_2d[J.COCO17_FROM_MHR70] for p, k in zip(preds, ok) if k
+ ]).astype(np.float64)
+ gt_kp = np.stack([s.meta["keypoints"] for s in kept])
+ gt_xy, gt_vis = gt_kp[:, :, :2], gt_kp[:, :, 2]
+ areas = np.array([s.meta["area"] for s in kept])
+ diag = np.array([np.hypot(*s.meta["bbox_wh"]) for s in kept])
+
+ # --- confidence-free metrics -----------------------------------------
+ valid = gt_vis > 0
+ sample_oks = oks(pred_kp, gt_xy, gt_vis, areas)
+ summary = {
+ "num_samples": len(kept),
+ "mean_OKS": float(np.nanmean(sample_oks)),
+ "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.1),
+ "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.2),
+ "NME_bboxdiag": M.nme2d(pred_kp, gt_xy, valid, diag),
+ }
+ per_kp = {}
+ for i, name in enumerate(J.COCO17_NAMES):
+ v = valid[:, i]
+ per_kp[name] = {
+ "n": int(v.sum()),
+ "[email protected]": float("nan") if not v.any() else
+ float((np.linalg.norm(pred_kp[v, i] - gt_xy[v, i], axis=-1)
+ / diag[v] < 0.1).mean()),
+ }
+
+ # --- OKS AP through the official evaluator ---------------------------
+ results = [dict(
+ image_id=int(s.meta["image_id"]), category_id=1, score=1.0,
+ keypoints=np.concatenate(
+ [pred_kp[i], np.ones((17, 1))], axis=1).reshape(-1).tolist(),
+ ) for i, s in enumerate(kept)]
+
+ with contextlib.redirect_stdout(io.StringIO()):
+ dt = coco.loadRes(results)
+ ev = COCOeval(coco, dt, "keypoints")
+ ev.params.imgIds = sorted({r["image_id"] for r in results})
+ ev.evaluate(); ev.accumulate()
+ ev.summarize()
+ ap_keys = ["AP", "AP50", "AP75", "AP_medium", "AP_large",
+ "AR", "AR50", "AR75", "AR_medium", "AR_large"]
+ summary["oks_ap"] = {k: float(v) for k, v in zip(ap_keys, ev.stats)}
+
+ report = dict(model=str(args.onnx), dataset="COCO val2017",
+ protocol="top-down, ground-truth boxes, score=1.0",
+ summary=summary, per_keypoint=per_kp)
+ out = Path(args.out)
+ out.parent.mkdir(parents=True, exist_ok=True)
+ out.write_text(json.dumps(report, indent=2))
+
+ print("\n" + "=" * 68)
+ print(f"COCO val2017 (GT boxes) | {summary['num_samples']} persons")
+ print("=" * 68)
+ print(f" OKS AP {summary['oks_ap']['AP']*100:6.2f} "
+ f"AP50 {summary['oks_ap']['AP50']*100:6.2f} "
+ f"AP75 {summary['oks_ap']['AP75']*100:6.2f}")
+ print(f" mean OKS {summary['mean_OKS']:6.4f}")
+ print(f" [email protected] {summary['[email protected]_bboxdiag']*100:6.2f} % "
+ f"[email protected] {summary['[email protected]_bboxdiag']*100:6.2f} %")
+ print(f" NME {summary['NME_bboxdiag']:6.4f} (bbox-diagonal units)")
+ print("=" * 68)
+ print(f"written to {out}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/results/3dpw_test.json b/benchmark/results/3dpw_test.json
new file mode 100644
index 0000000..8b2912f
--- /dev/null
+++ b/benchmark/results/3dpw_test.json
@@ -0,0 +1,354 @@
+{
+ "model": "models/instanthmr_mhr_only_ckpt90.onnx",
+ "dataset": "3DPW",
+ "split": "test",
+ "stride": 1,
+ "bbox_scale": 1.2,
+ "num_sequences": 24,
+ "sequences": [
+ "downtown_arguing_00",
+ "downtown_bar_00",
+ "downtown_bus_00",
+ "downtown_cafe_00",
+ "downtown_car_00",
+ "downtown_crossStreets_00",
+ "downtown_downstairs_00",
+ "downtown_enterShop_00",
+ "downtown_rampAndStairs_00",
+ "downtown_runForBus_00",
+ "downtown_runForBus_01",
+ "downtown_sitOnStairs_00",
+ "downtown_stairs_00",
+ "downtown_upstairs_00",
+ "downtown_walkBridge_01",
+ "downtown_walkUphill_00",
+ "downtown_walking_00",
+ "downtown_warmWelcome_00",
+ "downtown_weeklyMarket_00",
+ "downtown_windowShopping_00",
+ "flat_guitar_01",
+ "flat_packBags_00",
+ "office_phoneCall_00",
+ "outdoors_fencing_01"
+ ],
+ "num_samples": 35463,
+ "results": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 77.25832087421568,
+ "PA_MPJPE_mm": 54.8939122400647,
+ "PCK3D_50mm": 0.5790421405005619,
+ "PCK3D_100mm": 0.8924653864591264,
+ "AUC_0_150mm": 0.6371804272928843,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 58.99963246267586,
+ "right_knee": 42.38728858483827,
+ "right_hip": 50.39103164429582,
+ "left_hip": 45.39265522544049,
+ "left_knee": 46.095110047060146,
+ "left_ankle": 62.30361585517269,
+ "right_wrist": 63.81340497982142,
+ "right_elbow": 41.90725242230058,
+ "right_shoulder": 43.96807667630508,
+ "left_shoulder": 50.08607425327331,
+ "left_elbow": 48.06366572018603,
+ "left_wrist": 66.96362486775135,
+ "neck": 32.920260588579204,
+ "head": 115.22307803320544
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 35463,
+ "MPJPE_mm": 71.98718459643236,
+ "PA_MPJPE_mm": 48.292409136297024,
+ "PCK3D_50mm": 0.6549784282209626,
+ "PCK3D_100mm": 0.9406353100414516,
+ "AUC_0_150mm": 0.6786479562676105,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 56.99815377789105,
+ "right_knee": 43.630305291279456,
+ "right_hip": 46.529599094848884,
+ "left_hip": 42.83447178924256,
+ "left_knee": 47.52243530675108,
+ "left_ankle": 57.855173112259216,
+ "right_wrist": 62.29407156666568,
+ "right_elbow": 36.678818675428985,
+ "right_shoulder": 38.21018561144652,
+ "left_shoulder": 40.09340440903251,
+ "left_elbow": 40.734850438026946,
+ "left_wrist": 66.12744056269153
+ }
+ },
+ {
+ "joint_set": "J14+adapter",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 66.85722222445656,
+ "PA_MPJPE_mm": 44.083164610596384,
+ "PCK3D_50mm": 0.7056892294181863,
+ "PCK3D_100mm": 0.9425034543044863,
+ "AUC_0_150mm": 0.705719636913712
+ }
+ ],
+ "contamination": {
+ "training_dirs": [
+ "data/sam3d_distill_mix/annotations",
+ "data/sam3d_gt_3dpw/annotations"
+ ],
+ "seen_frames": 10353,
+ "unseen_frames": 25110,
+ "seen_fraction": 0.29193807630488117,
+ "seen_sequences": [
+ "downtown_arguing_00",
+ "downtown_bar_00",
+ "downtown_bus_00",
+ "downtown_cafe_00",
+ "downtown_car_00",
+ "downtown_crossStreets_00",
+ "downtown_downstairs_00",
+ "downtown_enterShop_00",
+ "downtown_rampAndStairs_00",
+ "downtown_runForBus_00",
+ "downtown_runForBus_01",
+ "downtown_sitOnStairs_00",
+ "downtown_stairs_00",
+ "downtown_upstairs_00",
+ "downtown_walkBridge_01",
+ "downtown_walkUphill_00",
+ "downtown_walking_00",
+ "downtown_warmWelcome_00",
+ "downtown_weeklyMarket_00",
+ "downtown_windowShopping_00",
+ "flat_guitar_01",
+ "flat_packBags_00",
+ "office_phoneCall_00",
+ "outdoors_fencing_01"
+ ]
+ },
+ "results_unseen_frames": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 25110,
+ "MPJPE_mm": 80.50931989672257,
+ "PA_MPJPE_mm": 57.05366140421865,
+ "PCK3D_50mm": 0.5548216419184161,
+ "PCK3D_100mm": 0.881657848324515,
+ "AUC_0_150mm": 0.6249301231264464,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 61.47931140891157,
+ "right_knee": 43.66784597368266,
+ "right_hip": 51.590643380821774,
+ "left_hip": 45.99633112852736,
+ "left_knee": 47.61141586722965,
+ "left_ankle": 65.61625292500608,
+ "right_wrist": 69.17219482531468,
+ "right_elbow": 43.22191976400349,
+ "right_shoulder": 44.582150790065214,
+ "left_shoulder": 50.70150039417648,
+ "left_elbow": 51.159805717594764,
+ "left_wrist": 72.72767706391078,
+ "neck": 34.115691861368255,
+ "head": 117.10851855844838
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 25110,
+ "MPJPE_mm": 75.3534859518377,
+ "PA_MPJPE_mm": 50.59458176134388,
+ "PCK3D_50mm": 0.6247710075667064,
+ "PCK3D_100mm": 0.9298320722155847,
+ "AUC_0_150mm": 0.6652752619656309,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 59.2944636488828,
+ "right_knee": 44.98211013553142,
+ "right_hip": 47.548593595632475,
+ "left_hip": 43.394665628452856,
+ "left_knee": 49.07030713900323,
+ "left_ankle": 60.92885186640498,
+ "right_wrist": 67.01462392288674,
+ "right_elbow": 38.323823357280524,
+ "right_shoulder": 39.95733150137268,
+ "left_shoulder": 41.6934507272976,
+ "left_elbow": 43.797015072444125,
+ "left_wrist": 71.12974454093712
+ }
+ },
+ {
+ "joint_set": "J14+adapter",
+ "num_joints": 14,
+ "num_samples": 25110,
+ "MPJPE_mm": 70.43668183981028,
+ "PA_MPJPE_mm": 46.514347723018886,
+ "PCK3D_50mm": 0.6766484610570632,
+ "PCK3D_100mm": 0.9324372759856631,
+ "AUC_0_150mm": 0.6915008065892562
+ }
+ ],
+ "results_seen_frames": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 10353,
+ "MPJPE_mm": 69.3734000343482,
+ "PA_MPJPE_mm": 49.655691288465555,
+ "PCK3D_50mm": 0.6377861489423355,
+ "PCK3D_100mm": 0.9186778159539678,
+ "AUC_0_150mm": 0.6668921183601361,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 52.98545895355014,
+ "right_knee": 39.2814452511299,
+ "right_hip": 47.48151259627433,
+ "left_hip": 43.92850938109478,
+ "left_knee": 42.41748625256036,
+ "left_ankle": 54.26919908481471,
+ "right_wrist": 50.81628211491889,
+ "right_elbow": 38.71867935650709,
+ "right_shoulder": 42.478711178718214,
+ "left_shoulder": 48.593429570758246,
+ "left_elbow": 40.554337473790454,
+ "left_wrist": 52.98358520334844,
+ "neck": 30.020880770194864,
+ "head": 110.65016085085732
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 10353,
+ "MPJPE_mm": 63.82261132933802,
+ "PA_MPJPE_mm": 42.70875660901737,
+ "PCK3D_50mm": 0.7282430213464697,
+ "PCK3D_100mm": 0.9668373096364983,
+ "AUC_0_150mm": 0.7110818743515204,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 51.42872068017997,
+ "right_knee": 40.351659522983596,
+ "right_hip": 44.05814619089098,
+ "left_hip": 41.47578664459172,
+ "left_knee": 43.768251813285275,
+ "left_ankle": 50.400322004696164,
+ "right_wrist": 50.84491966241465,
+ "right_elbow": 32.689050727849335,
+ "right_shoulder": 33.97268601750797,
+ "left_shoulder": 36.21267775476452,
+ "left_elbow": 33.30792549161378,
+ "left_wrist": 53.99493279743053
+ }
+ },
+ {
+ "joint_set": "J14+adapter",
+ "num_joints": 14,
+ "num_samples": 10353,
+ "MPJPE_mm": 58.175658335580685,
+ "PA_MPJPE_mm": 38.18661211828219,
+ "PCK3D_50mm": 0.7761242428005685,
+ "PCK3D_100mm": 0.9669178016034,
+ "AUC_0_150mm": 0.7402057597232443
+ }
+ ],
+ "per_sequence_J14": {
+ "downtown_arguing_00": {
+ "n": 1793,
+ "PA_MPJPE_mm": 53.86209217665553
+ },
+ "downtown_bar_00": {
+ "n": 2458,
+ "PA_MPJPE_mm": 67.11104059483931
+ },
+ "downtown_bus_00": {
+ "n": 2534,
+ "PA_MPJPE_mm": 57.3890391264324
+ },
+ "downtown_cafe_00": {
+ "n": 2076,
+ "PA_MPJPE_mm": 52.69031252948052
+ },
+ "downtown_car_00": {
+ "n": 1546,
+ "PA_MPJPE_mm": 60.43672446675064
+ },
+ "downtown_crossStreets_00": {
+ "n": 1042,
+ "PA_MPJPE_mm": 55.938241587076654
+ },
+ "downtown_downstairs_00": {
+ "n": 638,
+ "PA_MPJPE_mm": 47.09329939736882
+ },
+ "downtown_enterShop_00": {
+ "n": 1385,
+ "PA_MPJPE_mm": 46.63087615806622
+ },
+ "downtown_rampAndStairs_00": {
+ "n": 1954,
+ "PA_MPJPE_mm": 56.34401250060689
+ },
+ "downtown_runForBus_00": {
+ "n": 1302,
+ "PA_MPJPE_mm": 52.232966837233924
+ },
+ "downtown_runForBus_01": {
+ "n": 1437,
+ "PA_MPJPE_mm": 66.2344342328675
+ },
+ "downtown_sitOnStairs_00": {
+ "n": 2631,
+ "PA_MPJPE_mm": 52.626225175849925
+ },
+ "downtown_stairs_00": {
+ "n": 1197,
+ "PA_MPJPE_mm": 63.31683556710035
+ },
+ "downtown_upstairs_00": {
+ "n": 825,
+ "PA_MPJPE_mm": 47.17526695041395
+ },
+ "downtown_walkBridge_01": {
+ "n": 1182,
+ "PA_MPJPE_mm": 44.298063167659926
+ },
+ "downtown_walkUphill_00": {
+ "n": 387,
+ "PA_MPJPE_mm": 44.89834428674509
+ },
+ "downtown_walking_00": {
+ "n": 2594,
+ "PA_MPJPE_mm": 52.14282318214196
+ },
+ "downtown_warmWelcome_00": {
+ "n": 1122,
+ "PA_MPJPE_mm": 64.06683409393634
+ },
+ "downtown_weeklyMarket_00": {
+ "n": 1055,
+ "PA_MPJPE_mm": 47.945786223105564
+ },
+ "downtown_windowShopping_00": {
+ "n": 1821,
+ "PA_MPJPE_mm": 44.496857077869684
+ },
+ "flat_guitar_01": {
+ "n": 748,
+ "PA_MPJPE_mm": 60.36081947705571
+ },
+ "flat_packBags_00": {
+ "n": 1273,
+ "PA_MPJPE_mm": 64.67982670780314
+ },
+ "office_phoneCall_00": {
+ "n": 1524,
+ "PA_MPJPE_mm": 45.326043123588434
+ },
+ "outdoors_fencing_01": {
+ "n": 939,
+ "PA_MPJPE_mm": 55.91377958820054
+ }
+ }
+}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_adapter.json b/benchmark/results/3dpw_test_adapter.json
new file mode 100644
index 0000000..8f64214
--- /dev/null
+++ b/benchmark/results/3dpw_test_adapter.json
@@ -0,0 +1,334 @@
+{
+ "model": "models/instanthmr_mhr_only_ckpt90.onnx",
+ "dataset": "3DPW",
+ "split": "test",
+ "stride": 1,
+ "bbox_scale": 1.2,
+ "num_sequences": 24,
+ "sequences": [
+ "downtown_arguing_00",
+ "downtown_bar_00",
+ "downtown_bus_00",
+ "downtown_cafe_00",
+ "downtown_car_00",
+ "downtown_crossStreets_00",
+ "downtown_downstairs_00",
+ "downtown_enterShop_00",
+ "downtown_rampAndStairs_00",
+ "downtown_runForBus_00",
+ "downtown_runForBus_01",
+ "downtown_sitOnStairs_00",
+ "downtown_stairs_00",
+ "downtown_upstairs_00",
+ "downtown_walkBridge_01",
+ "downtown_walkUphill_00",
+ "downtown_walking_00",
+ "downtown_warmWelcome_00",
+ "downtown_weeklyMarket_00",
+ "downtown_windowShopping_00",
+ "flat_guitar_01",
+ "flat_packBags_00",
+ "office_phoneCall_00",
+ "outdoors_fencing_01"
+ ],
+ "num_samples": 35463,
+ "results": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 77.25832087421568,
+ "PA_MPJPE_mm": 54.8939122400647,
+ "PCK3D_50mm": 0.5790421405005619,
+ "PCK3D_100mm": 0.8924653864591264,
+ "AUC_0_150mm": 0.6371804272928843,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 58.99963246267586,
+ "right_knee": 42.38728858483827,
+ "right_hip": 50.39103164429582,
+ "left_hip": 45.39265522544049,
+ "left_knee": 46.095110047060146,
+ "left_ankle": 62.30361585517269,
+ "right_wrist": 63.81340497982142,
+ "right_elbow": 41.90725242230058,
+ "right_shoulder": 43.96807667630508,
+ "left_shoulder": 50.08607425327331,
+ "left_elbow": 48.06366572018603,
+ "left_wrist": 66.96362486775135,
+ "neck": 32.920260588579204,
+ "head": 115.22307803320544
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 35463,
+ "MPJPE_mm": 71.98718459643236,
+ "PA_MPJPE_mm": 48.292409136297024,
+ "PCK3D_50mm": 0.6549784282209626,
+ "PCK3D_100mm": 0.9406353100414516,
+ "AUC_0_150mm": 0.6786479562676105,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 56.99815377789105,
+ "right_knee": 43.630305291279456,
+ "right_hip": 46.529599094848884,
+ "left_hip": 42.83447178924256,
+ "left_knee": 47.52243530675108,
+ "left_ankle": 57.855173112259216,
+ "right_wrist": 62.29407156666568,
+ "right_elbow": 36.678818675428985,
+ "right_shoulder": 38.21018561144652,
+ "left_shoulder": 40.09340440903251,
+ "left_elbow": 40.734850438026946,
+ "left_wrist": 66.12744056269153
+ }
+ },
+ {
+ "joint_set": "J14+adapter",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 66.85722222445656,
+ "PA_MPJPE_mm": 44.083164610596384,
+ "PCK3D_50mm": 0.7056892294181863,
+ "PCK3D_100mm": 0.9425034543044863,
+ "AUC_0_150mm": 0.705719636913712
+ }
+ ],
+ "contamination": {
+ "training_dirs": [
+ "data/sam3d_distill_mix/annotations",
+ "data/sam3d_gt_3dpw/annotations"
+ ],
+ "seen_frames": 10353,
+ "unseen_frames": 25110,
+ "seen_fraction": 0.29193807630488117,
+ "seen_sequences": [
+ "downtown_arguing_00",
+ "downtown_bar_00",
+ "downtown_bus_00",
+ "downtown_cafe_00",
+ "downtown_car_00",
+ "downtown_crossStreets_00",
+ "downtown_downstairs_00",
+ "downtown_enterShop_00",
+ "downtown_rampAndStairs_00",
+ "downtown_runForBus_00",
+ "downtown_runForBus_01",
+ "downtown_sitOnStairs_00",
+ "downtown_stairs_00",
+ "downtown_upstairs_00",
+ "downtown_walkBridge_01",
+ "downtown_walkUphill_00",
+ "downtown_walking_00",
+ "downtown_warmWelcome_00",
+ "downtown_weeklyMarket_00",
+ "downtown_windowShopping_00",
+ "flat_guitar_01",
+ "flat_packBags_00",
+ "office_phoneCall_00",
+ "outdoors_fencing_01"
+ ]
+ },
+ "results_unseen_frames": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 25110,
+ "MPJPE_mm": 80.50931989672257,
+ "PA_MPJPE_mm": 57.05366140421865,
+ "PCK3D_50mm": 0.5548216419184161,
+ "PCK3D_100mm": 0.881657848324515,
+ "AUC_0_150mm": 0.6249301231264464,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 61.47931140891157,
+ "right_knee": 43.66784597368266,
+ "right_hip": 51.590643380821774,
+ "left_hip": 45.99633112852736,
+ "left_knee": 47.61141586722965,
+ "left_ankle": 65.61625292500608,
+ "right_wrist": 69.17219482531468,
+ "right_elbow": 43.22191976400349,
+ "right_shoulder": 44.582150790065214,
+ "left_shoulder": 50.70150039417648,
+ "left_elbow": 51.159805717594764,
+ "left_wrist": 72.72767706391078,
+ "neck": 34.115691861368255,
+ "head": 117.10851855844838
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 25110,
+ "MPJPE_mm": 75.3534859518377,
+ "PA_MPJPE_mm": 50.59458176134388,
+ "PCK3D_50mm": 0.6247710075667064,
+ "PCK3D_100mm": 0.9298320722155847,
+ "AUC_0_150mm": 0.6652752619656309,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 59.2944636488828,
+ "right_knee": 44.98211013553142,
+ "right_hip": 47.548593595632475,
+ "left_hip": 43.394665628452856,
+ "left_knee": 49.07030713900323,
+ "left_ankle": 60.92885186640498,
+ "right_wrist": 67.01462392288674,
+ "right_elbow": 38.323823357280524,
+ "right_shoulder": 39.95733150137268,
+ "left_shoulder": 41.6934507272976,
+ "left_elbow": 43.797015072444125,
+ "left_wrist": 71.12974454093712
+ }
+ }
+ ],
+ "results_seen_frames": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 10353,
+ "MPJPE_mm": 69.3734000343482,
+ "PA_MPJPE_mm": 49.655691288465555,
+ "PCK3D_50mm": 0.6377861489423355,
+ "PCK3D_100mm": 0.9186778159539678,
+ "AUC_0_150mm": 0.6668921183601361,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 52.98545895355014,
+ "right_knee": 39.2814452511299,
+ "right_hip": 47.48151259627433,
+ "left_hip": 43.92850938109478,
+ "left_knee": 42.41748625256036,
+ "left_ankle": 54.26919908481471,
+ "right_wrist": 50.81628211491889,
+ "right_elbow": 38.71867935650709,
+ "right_shoulder": 42.478711178718214,
+ "left_shoulder": 48.593429570758246,
+ "left_elbow": 40.554337473790454,
+ "left_wrist": 52.98358520334844,
+ "neck": 30.020880770194864,
+ "head": 110.65016085085732
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 10353,
+ "MPJPE_mm": 63.82261132933802,
+ "PA_MPJPE_mm": 42.70875660901737,
+ "PCK3D_50mm": 0.7282430213464697,
+ "PCK3D_100mm": 0.9668373096364983,
+ "AUC_0_150mm": 0.7110818743515204,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 51.42872068017997,
+ "right_knee": 40.351659522983596,
+ "right_hip": 44.05814619089098,
+ "left_hip": 41.47578664459172,
+ "left_knee": 43.768251813285275,
+ "left_ankle": 50.400322004696164,
+ "right_wrist": 50.84491966241465,
+ "right_elbow": 32.689050727849335,
+ "right_shoulder": 33.97268601750797,
+ "left_shoulder": 36.21267775476452,
+ "left_elbow": 33.30792549161378,
+ "left_wrist": 53.99493279743053
+ }
+ }
+ ],
+ "per_sequence_J14": {
+ "downtown_arguing_00": {
+ "n": 1793,
+ "PA_MPJPE_mm": 53.86209217665553
+ },
+ "downtown_bar_00": {
+ "n": 2458,
+ "PA_MPJPE_mm": 67.11104059483931
+ },
+ "downtown_bus_00": {
+ "n": 2534,
+ "PA_MPJPE_mm": 57.3890391264324
+ },
+ "downtown_cafe_00": {
+ "n": 2076,
+ "PA_MPJPE_mm": 52.69031252948052
+ },
+ "downtown_car_00": {
+ "n": 1546,
+ "PA_MPJPE_mm": 60.43672446675064
+ },
+ "downtown_crossStreets_00": {
+ "n": 1042,
+ "PA_MPJPE_mm": 55.938241587076654
+ },
+ "downtown_downstairs_00": {
+ "n": 638,
+ "PA_MPJPE_mm": 47.09329939736882
+ },
+ "downtown_enterShop_00": {
+ "n": 1385,
+ "PA_MPJPE_mm": 46.63087615806622
+ },
+ "downtown_rampAndStairs_00": {
+ "n": 1954,
+ "PA_MPJPE_mm": 56.34401250060689
+ },
+ "downtown_runForBus_00": {
+ "n": 1302,
+ "PA_MPJPE_mm": 52.232966837233924
+ },
+ "downtown_runForBus_01": {
+ "n": 1437,
+ "PA_MPJPE_mm": 66.2344342328675
+ },
+ "downtown_sitOnStairs_00": {
+ "n": 2631,
+ "PA_MPJPE_mm": 52.626225175849925
+ },
+ "downtown_stairs_00": {
+ "n": 1197,
+ "PA_MPJPE_mm": 63.31683556710035
+ },
+ "downtown_upstairs_00": {
+ "n": 825,
+ "PA_MPJPE_mm": 47.17526695041395
+ },
+ "downtown_walkBridge_01": {
+ "n": 1182,
+ "PA_MPJPE_mm": 44.298063167659926
+ },
+ "downtown_walkUphill_00": {
+ "n": 387,
+ "PA_MPJPE_mm": 44.89834428674509
+ },
+ "downtown_walking_00": {
+ "n": 2594,
+ "PA_MPJPE_mm": 52.14282318214196
+ },
+ "downtown_warmWelcome_00": {
+ "n": 1122,
+ "PA_MPJPE_mm": 64.06683409393634
+ },
+ "downtown_weeklyMarket_00": {
+ "n": 1055,
+ "PA_MPJPE_mm": 47.945786223105564
+ },
+ "downtown_windowShopping_00": {
+ "n": 1821,
+ "PA_MPJPE_mm": 44.496857077869684
+ },
+ "flat_guitar_01": {
+ "n": 748,
+ "PA_MPJPE_mm": 60.36081947705571
+ },
+ "flat_packBags_00": {
+ "n": 1273,
+ "PA_MPJPE_mm": 64.67982670780314
+ },
+ "office_phoneCall_00": {
+ "n": 1524,
+ "PA_MPJPE_mm": 45.326043123588434
+ },
+ "outdoors_fencing_01": {
+ "n": 939,
+ "PA_MPJPE_mm": 55.91377958820054
+ }
+ }
+}
\ No newline at end of file
diff --git a/benchmark/results/adapter_j14.npz b/benchmark/results/adapter_j14.npz
new file mode 100644
index 0000000..82c864c
Binary files /dev/null and b/benchmark/results/adapter_j14.npz differ
diff --git a/benchmark/results/coco_val2017.json b/benchmark/results/coco_val2017.json
new file mode 100644
index 0000000..6472b0d
--- /dev/null
+++ b/benchmark/results/coco_val2017.json
@@ -0,0 +1,94 @@
+{
+ "model": "models/instanthmr_mhr_only_ckpt90.onnx",
+ "dataset": "COCO val2017",
+ "protocol": "top-down, ground-truth boxes, score=1.0",
+ "summary": {
+ "num_samples": 6352,
+ "mean_OKS": 0.715087947740202,
+ "[email protected]_bboxdiag": 0.8967235945173349,
+ "[email protected]_bboxdiag": 0.9517847980649418,
+ "NME_bboxdiag": 0.04808343393139624,
+ "oks_ap": {
+ "AP": 0.44039902588970287,
+ "AP50": 0.697060689022555,
+ "AP75": 0.47301161863312285,
+ "AP_medium": 0.43565038188009775,
+ "AP_large": 0.46188555986388313,
+ "AR": 0.5974968513853904,
+ "AR50": 0.8148614609571788,
+ "AR75": 0.6572732997481109,
+ "AR_medium": 0.5542201584266594,
+ "AR_large": 0.6594574507617986
+ }
+ },
+ "per_keypoint": {
+ "nose": {
+ "n": 4268,
+ "[email protected]": 0.9395501405810684
+ },
+ "left_eye": {
+ "n": 3703,
+ "[email protected]": 0.9343775317310289
+ },
+ "right_eye": {
+ "n": 3685,
+ "[email protected]": 0.9383989145183175
+ },
+ "left_ear": {
+ "n": 3185,
+ "[email protected]": 0.930298273155416
+ },
+ "right_ear": {
+ "n": 3256,
+ "[email protected]": 0.9302825552825553
+ },
+ "left_shoulder": {
+ "n": 5501,
+ "[email protected]": 0.9014724595528085
+ },
+ "right_shoulder": {
+ "n": 5495,
+ "[email protected]": 0.902820746132848
+ },
+ "left_elbow": {
+ "n": 4336,
+ "[email protected]": 0.8996771217712177
+ },
+ "right_elbow": {
+ "n": 4378,
+ "[email protected]": 0.901553220648698
+ },
+ "left_wrist": {
+ "n": 3945,
+ "[email protected]": 0.8869455006337136
+ },
+ "right_wrist": {
+ "n": 4012,
+ "[email protected]": 0.8898305084745762
+ },
+ "left_hip": {
+ "n": 4755,
+ "[email protected]": 0.8441640378548896
+ },
+ "right_hip": {
+ "n": 4784,
+ "[email protected]": 0.844690635451505
+ },
+ "left_knee": {
+ "n": 3513,
+ "[email protected]": 0.8778821520068317
+ },
+ "right_knee": {
+ "n": 3500,
+ "[email protected]": 0.8805714285714286
+ },
+ "left_ankle": {
+ "n": 2957,
+ "[email protected]": 0.8782549881636794
+ },
+ "right_ankle": {
+ "n": 2942,
+ "[email protected]": 0.8820530251529571
+ }
+ }
+}
\ No newline at end of file
diff --git a/demo.py b/demo.py
index 3a511bb..82a9995 100755
--- a/demo.py
+++ b/demo.py
@@ -8,21 +8,31 @@ drawn on top, and live latency plots for RF-DETR, InstantHMR, MHR mesh
rendering, and total pipeline cost.
Optionally renders a full MHR body mesh per person by decoding the
-``mhr_params`` and ``shape_params`` outputs through Meta's MHR model:
-
- python demo.py --image photo.jpg \\
- --mhr-assets models/mhr_assets
-
-See ``instanthmr/mhr_renderer.py`` for setup instructions (assets download,
-MHR package installation).
+``mhr_params`` and ``shape_params`` outputs through Meta's MHR model, either
+from the TorchScript rig used by the distillation (``--mhr-model``) or from the
+unpacked asset folder (``--mhr-assets``). See ``instanthmr/mhr_renderer.py``.
+
+MHR-only checkpoints
+--------------------
+A model exported from ``train_distill_mhr_only.py`` has FOUR ONNX outputs — it
+has no 3D coordinate head, so its 70 keypoints are derived from the MHR forward
+pass, exactly like the SAM3D teacher does. Such a model therefore *requires* an
+MHR rig; when none is requested explicitly the demo falls back to
+``checkpoints/mhr_model.pt``.
Usage
-----
- # Single image (skeleton only)
+ # Single image (skeleton only; needs a 5-output model)
python demo.py --image path/to/photo.jpg
- # Single image with MHR mesh
- python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets
+ # Single image with the MHR mesh, from the TorchScript rig
+ python demo.py --image path/to/photo.jpg --mhr-model
+
+ # MHR-only student: the rig is mandatory and auto-selected
+ python demo.py --image path/to/photo.jpg --model models/instanthmr_mhr_only.onnx
+
+ # Same, but decoding the mesh from the asset folder at a chosen LOD
+ python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets --mhr-lod 3
# Video file
python demo.py --video path/to/clip.mp4
@@ -40,7 +50,8 @@ from pathlib import Path
import cv2
import numpy as np
-from instanthmr import PosePipeline
+from instanthmr import PosePipeline, onnx_output_names
+from instanthmr.mhr_renderer import DEFAULT_MHR_SCRIPT
from instanthmr.visualizer import RerunVisualizer
@@ -104,6 +115,16 @@ def parse_args() -> argparse.Namespace:
"--no-spawn", action="store_true",
help="Don't spawn the Rerun viewer GUI (useful when only saving .rrd).",
)
+ p.add_argument(
+ "--mhr-model", type=str, nargs="?", default=None, const=DEFAULT_MHR_SCRIPT,
+ metavar="PT",
+ help=(
+ "Decode the body mesh with the TorchScript MHR rig used by the "
+ "distillation (bare flag = checkpoints/mhr_model.pt). Needs only "
+ "PyTorch, so it works regardless of the pymomentum build. Takes "
+ "precedence over --mhr-assets."
+ ),
+ )
p.add_argument(
"--mhr-assets", type=str, default=None, metavar="DIR",
help=(
@@ -116,15 +137,24 @@ def parse_args() -> argparse.Namespace:
"--mhr-lod", type=int, default=3, metavar="N",
choices=range(7),
help=(
- "MHR level-of-detail (0=73 639 verts … 6=595 verts). "
- "Default: 3 (4 899 verts)."
+ "MHR level-of-detail for --mhr-assets (0=73 639 verts … 6=595 "
+ "verts). Default: 3 (4 899 verts). The TorchScript rig has a fixed "
+ "LOD (18 439 verts) and ignores this."
+ ),
+ )
+ p.add_argument(
+ "--mesh-alpha", type=float, default=0.35, metavar="A",
+ help=(
+ "Body-mesh opacity in [0, 1] (default: 0.35). Applied as a Rerun "
+ "albedo factor so the 70 keypoints stay visible inside the body; "
+ "raise it towards 1.0 for a solid mesh."
),
)
return p.parse_args()
-def build_pipeline(args: argparse.Namespace) -> PosePipeline:
+def resolve_model_path(args: argparse.Namespace) -> Path:
model_path = Path(args.model)
if not model_path.exists():
sys.stderr.write(
@@ -133,23 +163,70 @@ def build_pipeline(args: argparse.Namespace) -> PosePipeline:
" --model /path/to/instanthmr.onnx\n\n"
)
sys.exit(1)
+ return model_path
+
+def build_pipeline(args: argparse.Namespace, mhr=None) -> PosePipeline:
return PosePipeline(
- onnx_path=model_path,
+ onnx_path=resolve_model_path(args),
device=args.device,
detector_variant=args.detector_variant,
det_confidence=args.det_confidence,
max_persons=args.max_persons,
detector_stride=args.detector_stride,
batch_persons=not args.no_batch_persons,
+ mhr=mhr,
)
-def build_mhr_renderer(args: argparse.Namespace):
- if args.mhr_assets is None:
- return None
+def _model_needs_mhr(model_path: Path) -> bool:
+ """True when the graph has no ``joints_3d`` output (MHR-only student).
- assets_path = Path(args.mhr_assets)
+ Cheap protobuf peek, so the decision is made before any session or detector
+ is loaded. An unreadable graph is treated as "does not need MHR" — the
+ pipeline then raises a precise error of its own.
+ """
+ names = onnx_output_names(model_path)
+ return names is not None and "joints_3d" not in names
+
+
+def build_mhr(args: argparse.Namespace, required: bool = False):
+ """Build the MHR rig requested on the command line.
+
+ ``--mhr-model`` (TorchScript) wins over ``--mhr-assets`` (pip package).
+ When *required* — i.e. the ONNX has no 3D head — and neither flag was given,
+ fall back to the TorchScript rig that shipped with the training code.
+ """
+ script_path = args.mhr_model
+ assets_folder = args.mhr_assets
+
+ if script_path is None and assets_folder is None:
+ if not required:
+ return None
+ script_path = DEFAULT_MHR_SCRIPT
+ print(
+ f"[info] {Path(args.model).name} has no 'joints_3d' output "
+ "(MHR-only student): its 70 keypoints are derived from the MHR "
+ f"forward pass.\n Using {script_path}; override with "
+ "--mhr-model / --mhr-assets."
+ )
+
+ if script_path is not None:
+ path = Path(script_path)
+ if not path.exists():
+ sys.stderr.write(
+ f"\n[error] MHR TorchScript rig not found: {path}\n"
+ " This is the ~700 MB mhr_model.pt the distillation loads with\n"
+ " torch.jit.load; see instanthmr_distill_train/checkpoints/README.md.\n"
+ " Alternatively use --mhr-assets with the unpacked MHR asset folder.\n\n"
+ )
+ sys.exit(1)
+ from instanthmr.mhr_renderer import MHRScriptRenderer
+
+ print(f"Loading MHR body model (TorchScript) from {path} …")
+ return MHRScriptRenderer(path, device=args.device)
+
+ assets_path = Path(assets_folder)
if not assets_path.exists():
sys.stderr.write(
f"\n[error] MHR assets folder not found: {assets_path}\n"
@@ -172,6 +249,8 @@ def build_mhr_renderer(args: argparse.Namespace):
" pip install pymomentum-gpu # NVIDIA GPU\n"
" pip install pymomentum-cpu # macOS / CPU-only\n"
"\n"
+ " Or skip it entirely and use --mhr-model, which needs only torch.\n"
+ "\n"
" See README.md §'MHR body mesh' for full setup instructions.\n\n"
)
sys.exit(1)
@@ -184,13 +263,24 @@ def build_mhr_renderer(args: argparse.Namespace):
)
-def build_visualizer(args: argparse.Namespace, mhr_renderer=None) -> RerunVisualizer:
- return RerunVisualizer(
+def build_all(args: argparse.Namespace):
+ """Build the MHR rig, the pipeline and the visualizer, in that order.
+
+ The rig comes first because an MHR-only graph cannot produce 3D keypoints
+ without it — the pipeline takes it as a constructor argument rather than
+ the visualizer alone.
+ """
+ model_path = resolve_model_path(args)
+ mhr = build_mhr(args, required=_model_needs_mhr(model_path))
+ pipeline = build_pipeline(args, mhr=mhr)
+ viz = RerunVisualizer(
application_id="instanthmr_demo",
spawn_viewer=not args.no_spawn,
save_path=args.save_rrd,
- mhr_renderer=mhr_renderer,
+ mhr_renderer=mhr,
+ mesh_alpha=args.mesh_alpha,
)
+ return pipeline, viz
def _print_timings(
@@ -228,9 +318,7 @@ def run_image(args: argparse.Namespace) -> None:
sys.exit(f"[error] cv2.imread failed for {image_path}")
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
- pipeline = build_pipeline(args)
- mhr = build_mhr_renderer(args)
- viz = build_visualizer(args, mhr)
+ pipeline, viz = build_all(args)
result = pipeline.predict(rgb)
@@ -261,9 +349,7 @@ def run_video(args: argparse.Namespace) -> None:
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
print(f"video: {video_path.name}, {total} frames @ {fps:.1f} fps")
- pipeline = build_pipeline(args)
- mhr = build_mhr_renderer(args)
- viz = build_visualizer(args, mhr)
+ pipeline, viz = build_all(args)
frame_idx = 0
sent = 0
@@ -341,9 +427,7 @@ def run_camera(args: argparse.Namespace) -> None:
fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
print(f"camera {args.camera}: {width}x{height} @ {fps:.1f} fps — Ctrl+C to stop")
- pipeline = build_pipeline(args)
- mhr = build_mhr_renderer(args)
- viz = build_visualizer(args, mhr)
+ pipeline, viz = build_all(args)
frame_idx = 0
det_times: list[float] = []
diff --git a/instanthmr/__init__.py b/instanthmr/__init__.py
index 1194ccc..1fce43e 100644
--- a/instanthmr/__init__.py
+++ b/instanthmr/__init__.py
@@ -1,6 +1,6 @@
"""InstantHMR — standalone 3D human pose inference + Rerun visualization."""
-from .inference import InstantHMR, HMRPrediction
+from .inference import InstantHMR, HMRPrediction, onnx_output_names
from .detector import RFDETRDetector
from .pipeline import PosePipeline, FrameResult
from .skeleton import JOINT_NAMES, SKELETON_EDGES, NUM_JOINTS, edges_for
@@ -8,6 +8,7 @@ from .skeleton import JOINT_NAMES, SKELETON_EDGES, NUM_JOINTS, edges_for
__all__ = [
"InstantHMR",
"HMRPrediction",
+ "onnx_output_names",
"RFDETRDetector",
"PosePipeline",
"FrameResult",
diff --git a/instanthmr/inference.py b/instanthmr/inference.py
index 0dca31b..0f6fdf2 100644
--- a/instanthmr/inference.py
+++ b/instanthmr/inference.py
@@ -3,11 +3,22 @@
InstantHMR takes a 224x224 person crop and a 3-vector CLIFF condition
(bbox center / scale in full-frame coords) and returns mhr_params, shape,
camera translation, and 70 joints in 2D (crop space) and 3D (camera coords,
-body-centred, metres, Y-down).
+rig-local, metres, Y-down).
This module wraps the ONNX session and the per-person preprocessing — square
crop, CLIFF cond, ImageNet normalisation, and re-projection of joints back
into full-frame pixels.
+
+Two graph layouts are supported, and the session tells them apart by itself:
+
+ * **5 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d, joints_3d)``
+ — the student has its own 3D coordinate head (``train_distill.py`` and the
+ optimized / correctives scripts).
+ * **4 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d)`` — the
+ MHR-only student (``train_distill_mhr_only.py``) has no 3D head, exactly
+ like the SAM3D teacher. Its 70 keypoints are derived here by running the
+ MHR skeleton forward on the predicted parameters, so such a model needs an
+ MHR backend (see :mod:`instanthmr.mhr_renderer`) passed as ``mhr=``.
"""
from __future__ import annotations
@@ -27,6 +38,23 @@ INPUT_SIZE = 224
CROP_EXPAND = 1.2 # square crop around the detector bbox, matching training
+def onnx_output_names(onnx_path: str | Path) -> Optional[list[str]]:
+ """Read a graph's output names without creating an inference session.
+
+ Lets a caller know whether it must supply an MHR backend *before* paying for
+ session creation and detector loading. Returns ``None`` when the ``onnx``
+ package is unavailable or the file cannot be parsed — callers should then
+ fall back to letting :class:`InstantHMR` decide.
+ """
+ try:
+ import onnx
+
+ model = onnx.load(str(onnx_path), load_external_data=False)
+ return [o.name for o in model.graph.output]
+ except Exception: # noqa: BLE001 — purely advisory
+ return None
+
+
@dataclass
class HMRPrediction:
"""Per-person outputs from one InstantHMR forward pass.
@@ -36,7 +64,9 @@ class HMRPrediction:
Attributes:
bbox: (4,) raw detector bbox [x1, y1, x2, y2].
confidence: detection confidence in [0, 1].
- joints_3d_local: (70, 3) body-centred joints, metres, Y-down.
+ joints_3d_local: (70, 3) rig-local joints, metres, Y-down. Either read
+ straight from the model's 3D head, or regressed from the MHR
+ skeleton when the graph has no such head.
joints_3d_cam: (70, 3) joints in camera space (= local + cam_trans).
joints_2d: (70, 2) joints projected into full-frame pixel coords.
cam_trans: (3,) camera translation [tx, ty, tz], metres.
@@ -67,6 +97,15 @@ class HMRPrediction:
class InstantHMR:
"""ONNX wrapper for the InstantHMR model.
+ Args:
+ onnx_path: path to the exported graph.
+ device: ``"cuda"``, ``"coreml"`` or ``"cpu"``.
+ providers: explicit onnxruntime execution providers (overrides *device*).
+ mhr: MHR backend used to derive ``joints_3d`` when the graph has no 3D
+ head. Any object exposing ``keypoints(mhr_params, shape_params)``
+ works; see :func:`instanthmr.mhr_renderer.build_mhr`. Ignored for
+ 5-output graphs.
+
Example
-------
hmr = InstantHMR("models/instanthmr.onnx")
@@ -79,6 +118,7 @@ class InstantHMR:
onnx_path: str | Path,
device: str = "cuda",
providers: Optional[list[str]] = None,
+ mhr: object | None = None,
):
import onnxruntime as ort
@@ -116,8 +156,42 @@ class InstantHMR:
out_names = [o.name for o in self.session.get_outputs()]
self._in_image = in_names[0] # "image"
self._in_cliff = in_names[1] # "cliff_cond"
- # Output order from export: mhr_params, shape_params, cam_trans, joints_2d, joints_3d
+ # Output order from export: mhr_params, shape_params, cam_trans,
+ # joints_2d[, joints_3d]. The MHR-only student stops after joints_2d.
self._out_names = out_names
+ self.has_joints_3d_head = "joints_3d" in out_names or len(out_names) >= 5
+ self._mhr = mhr
+
+ @property
+ def derives_joints_3d(self) -> bool:
+ """True when the 70 keypoints must come from an MHR forward pass."""
+ return not self.has_joints_3d_head
+
+ def set_mhr(self, mhr: object | None) -> None:
+ """Attach (or replace) the MHR backend used to derive ``joints_3d``."""
+ self._mhr = mhr
+
+ def _joints_3d(self, mhr_params: np.ndarray, shape_params: np.ndarray) -> np.ndarray:
+ """Regress the 70 keypoints from predicted MHR parameters.
+
+ Mirrors ``train_distill_mhr_only.MHRForwardPass.get_native_keypoints``:
+ skeleton forward, cm -> m, Y/Z flip, then the (70, 127) affine map. The
+ student was trained through exactly this operator, so any other
+ keypoint definition would silently shift the predictions.
+ """
+ if self._mhr is None:
+ raise RuntimeError(
+ "this ONNX graph has no 'joints_3d' output — it comes from "
+ "train_distill_mhr_only.py, where the 3D keypoints are derived from "
+ "the MHR forward pass.\n"
+ "Pass an MHR backend, e.g.\n"
+ " from instanthmr.mhr_renderer import build_mhr\n"
+ " InstantHMR(onnx, mhr=build_mhr(script_path='checkpoints/mhr_model.pt'))\n"
+ "or run demo.py with --mhr-model / --mhr-assets."
+ )
+ return np.asarray(
+ self._mhr.keypoints(mhr_params, shape_params), dtype=np.float32,
+ )
# ------------------------------------------------------------------
# Public API
@@ -154,12 +228,15 @@ class InstantHMR:
self._in_cliff: cliff_cond[np.newaxis], # (1, 3)
},
)
- # Order: mhr_params, shape_params, cam_trans, joints_2d, joints_3d
+ # Order: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]
mhr_params = outs[0][0].astype(np.float32)
shape_params = outs[1][0].astype(np.float32)
cam_trans = outs[2][0].astype(np.float32)
joints_2d_norm = outs[3][0].astype(np.float32) # (70, 2) in [-1, 1]
- joints_3d_local = outs[4][0].astype(np.float32) # (70, 3) body-centred
+ if self.has_joints_3d_head:
+ joints_3d_local = outs[4][0].astype(np.float32) # (70, 3) rig-local
+ else:
+ joints_3d_local = self._joints_3d(mhr_params, shape_params)
# Re-project the 2D head from normalised crop space → full-frame pixels.
crop_px = (joints_2d_norm + 1.0) * 0.5 * INPUT_SIZE
@@ -238,7 +315,12 @@ class InstantHMR:
shape_params_b = outs[1].astype(np.float32, copy=False)
cam_trans_b = outs[2].astype(np.float32, copy=False)
joints_2d_norm_b = outs[3].astype(np.float32, copy=False)
- joints_3d_local_b = outs[4].astype(np.float32, copy=False)
+ if self.has_joints_3d_head:
+ joints_3d_local_b = outs[4].astype(np.float32, copy=False)
+ else:
+ # One batched MHR skeleton forward for the whole frame (~0.9 ms),
+ # including the zero-padded slots — they are discarded below.
+ joints_3d_local_b = self._joints_3d(mhr_params_b, shape_params_b)
f = math.sqrt(h * h + w * w)
focal_length = np.array([f, f], dtype=np.float32)
diff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.py
index fc6b79d..b08a5e9 100644
--- a/instanthmr/mhr_renderer.py
+++ b/instanthmr/mhr_renderer.py
@@ -1,15 +1,39 @@
-"""MHR (Momentum Human Rig) mesh decoder for InstantHMR.
+"""MHR (Momentum Human Rig) decoder for InstantHMR.
Takes the 204-dim pose parameters and 45-dim shape parameters produced by
-InstantHMR and decodes them into full body mesh vertices by running a forward
-pass through Meta's MHR body model.
+InstantHMR and runs a forward pass through Meta's MHR body model to obtain
-Quick setup (requires Python >= 3.12)::
+ * the full body **mesh** (skinned vertices), and
+ * the 70 native **keypoints**, regressed from the 127 skeleton joints.
+
+The keypoints matter because ``train_distill_mhr_only.py`` deletes the
+student's 3D coordinate head: its ONNX graph has four outputs instead of five
+and the 3D pose must be derived here, exactly the way the teacher derives it
+from the mesh. The (70, 127) affine map lives in
+``instanthmr_distill_train/assets/mhr_j127_to_kp70.npy`` and was fitted against
+the teacher's own labels (held-out residual: 1.36 mm mean).
+
+Two interchangeable backends
+----------------------------
+``MHRScriptRenderer`` (recommended)
+ Loads the self-contained TorchScript rig at ``checkpoints/mhr_model.pt`` —
+ the same artefact the distillation used. Needs nothing but PyTorch, so it
+ is immune to the ``pymomentum`` / ``torch`` ABI pairing below.
+
+``MHRRenderer``
+ Loads the unpacked MHR asset folder through the ``mhr`` pip package. This
+ lets you pick a mesh LOD, but ``pymomentum-gpu`` is a torch C++ extension
+ and **must** match the installed torch (0.1.113 wants ``torch>=2.8,<2.9``);
+ a mismatch segfaults the process rather than raising.
+
+Use :func:`build_mhr` to get whichever one the caller configured.
+
+Quick setup for the asset backend (requires Python >= 3.12)::
pip install -r requirements-mhr.txt
curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
unzip assets.zip -d models/mhr_assets
- python demo.py --image photo.jpg --mhr-assets models/mhr_assets
+ python demo.py --image photo.jpg --mhr-assets models/mhr_assets/assets
WARNING — wrong pymomentum package on PyPI
------------------------------------------
@@ -25,20 +49,311 @@ See README.md §'MHR body mesh' for the full installation walkthrough.
from __future__ import annotations
from pathlib import Path
+from typing import Optional
import numpy as np
-class MHRRenderer:
- """Loads and caches the MHR body model, runs per-person forward passes.
+_REPO_ROOT = Path(__file__).resolve().parent.parent
+
+#: (70, 127) affine map from MHR skeleton joints to the 70 native keypoints.
+DEFAULT_KP_REGRESSOR = _REPO_ROOT / "instanthmr_distill_train/assets/mhr_j127_to_kp70.npy"
+#: TorchScript rig shipped with the training code.
+DEFAULT_MHR_SCRIPT = _REPO_ROOT / "checkpoints/mhr_model.pt"
+
+NUM_MODEL_PARAMS = 204 # 136 pose + 68 scale
+NUM_IDENTITY = 45 # identity blend-shape coefficients
+NUM_EXPRESSION = 72 # facial expression coefficients (InstantHMR predicts none)
+NUM_SKELETON_JOINTS = 127
+NUM_KEYPOINTS = 70
+
+
+class _MHRBase:
+ """Parameter marshalling, coordinate conversion, rig evaluation and keypoints.
+
+ Both backends expose the same three objects, so everything below the loader
+ is shared. A subclass' ``__init__`` must set:
+
+ * ``self._model`` — callable ``(identity, pose, expr) -> (verts, skel)``
+ * ``self._character`` — exposes ``model_parameters_to_joint_parameters``
+ and ``joint_parameters_to_skeleton_state``
+ * ``self._n_params`` — width its parameter transform expects, i.e. 204
+ model parameters plus that rig's blend-shape slots
+
+ and it must implement the ``faces`` property.
+ """
+
+ def __init__(self, device: str = "cuda", kp_regressor: str | Path | None = None):
+ import torch
+
+ self._torch = torch
+ self._device = self._resolve_device(torch, device)
+ self._kp_regressor = None
+
+ path = Path(kp_regressor) if kp_regressor is not None else DEFAULT_KP_REGRESSOR
+ if path.exists():
+ W = np.load(path).astype(np.float32)
+ if W.shape != (NUM_KEYPOINTS, NUM_SKELETON_JOINTS):
+ raise ValueError(
+ f"expected a ({NUM_KEYPOINTS}, {NUM_SKELETON_JOINTS}) keypoint "
+ f"regressor in {path}, got {W.shape}"
+ )
+ self._kp_regressor = torch.from_numpy(W).to(self._device)
+ elif kp_regressor is not None:
+ raise FileNotFoundError(f"keypoint regressor not found: {path}")
+
+ @staticmethod
+ def _resolve_device(torch, device):
+ """Map the demo's device string onto a torch device.
+
+ ``--device coreml`` selects an onnxruntime execution provider, not a
+ torch backend, so anything torch cannot run on falls back to CPU.
+ """
+ name = str(device).lower()
+ if "cuda" in name and torch.cuda.is_available():
+ return torch.device(device)
+ if "cuda" in name:
+ return torch.device("cpu")
+ try:
+ return torch.device(device)
+ except (RuntimeError, TypeError):
+ return torch.device("cpu")
+
+ # ------------------------------------------------------------------
+ # Capabilities
+ # ------------------------------------------------------------------
+
+ @property
+ def has_keypoints(self) -> bool:
+ """True when the 70-keypoint regressor was found and loaded."""
+ return self._kp_regressor is not None
+
+ @property
+ def faces(self) -> np.ndarray:
+ raise NotImplementedError
+
+ # ------------------------------------------------------------------
+ # Coordinate conventions
+ # ------------------------------------------------------------------
+
+ @staticmethod
+ def _to_vision(xyz):
+ """Raw MHR (centimetres, Y-up / Z-back) -> InstantHMR (metres, Y-down / Z-forward).
+
+ Negating both Y and Z keeps the frame right-handed after the Y flip.
+ This is the exact transform ``train_distill_mhr_only.MHRForwardPass``
+ applies before the keypoint regression, so student and demo agree.
+ """
+ out = xyz / 100.0
+ out[..., 1] *= -1.0
+ out[..., 2] *= -1.0
+ return out
+
+ # ------------------------------------------------------------------
+ # Public API
+ # ------------------------------------------------------------------
+
+ def keypoints(
+ self,
+ mhr_params: np.ndarray,
+ shape_params: np.ndarray,
+ ) -> np.ndarray:
+ """Regress the 70 native keypoints from the MHR skeleton.
+
+ Skeleton branch only — no blend shapes, no skinning — so this is ~3x
+ cheaper than a full mesh forward (≈0.9 ms vs ≈2.6 ms at batch 2 on an
+ RTX 4070).
+
+ Args:
+ mhr_params: ``(204,)`` or ``(B, 204)`` float32 pose parameters.
+ shape_params: ``(45,)`` or ``(B, 45)`` — accepted for symmetry with
+ :meth:`forward`; identity coefficients do not move the skeleton
+ (they only displace the rest-pose vertices), which is why the
+ distillation passes zeros here too.
+
+ Returns:
+ ``(70, 3)`` or ``(B, 70, 3)`` float32 keypoints in the **rig-local**
+ frame (metres, Y-down) — the frame ``cam_trans`` is relative to.
+ Add ``cam_trans`` for camera space.
+ """
+ if self._kp_regressor is None:
+ raise RuntimeError(
+ f"no MHR keypoint regressor loaded (looked for {DEFAULT_KP_REGRESSOR}).\n"
+ "It ships with the training code; regenerate it with\n"
+ " python instanthmr_distill_train/train_distill_mhr_only.py --fit-regressor"
+ )
+ torch = self._torch
+ pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)
+
+ with torch.no_grad():
+ skel = self._skeleton_state(pose)
+ kps = self._regress(skel)
+
+ out = kps.cpu().numpy().astype(np.float32)
+ return out[0] if squeeze else out
+
+ def forward(
+ self,
+ mhr_params: np.ndarray,
+ shape_params: np.ndarray,
+ ) -> np.ndarray:
+ """Decode one person's parameters into body-space mesh vertices.
+
+ Args:
+ mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.
+ shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.
+
+ Returns:
+ ``(V, 3)`` float32 vertices in the **rig-local** frame (metres,
+ Y-down). Add the person's ``cam_trans`` to move to camera space.
+ """
+ verts, _ = self.forward_full(mhr_params, shape_params)
+ return verts
+
+ def forward_full(
+ self,
+ mhr_params: np.ndarray,
+ shape_params: np.ndarray,
+ ) -> tuple[np.ndarray, Optional[np.ndarray]]:
+ """Mesh vertices **and** the 70 keypoints from a single MHR forward pass.
+
+ Returns ``(verts, keypoints)`` with the same leading shape convention as
+ :meth:`forward` / :meth:`keypoints`. ``keypoints`` is ``None`` when no
+ regressor is available.
+ """
+ torch = self._torch
+ pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)
+ identity, _ = self._as_batch(shape_params, NUM_IDENTITY)
+ if identity.shape[0] != pose.shape[0]:
+ identity = identity.expand(pose.shape[0], -1)
+
+ with torch.no_grad():
+ verts, skel = self._skin(identity, pose)
+ verts = self._to_vision(verts.float())
+ kps = self._regress(skel) if self._kp_regressor is not None else None
+
+ v = verts.cpu().numpy().astype(np.float32)
+ k = kps.cpu().numpy().astype(np.float32) if kps is not None else None
+ if squeeze:
+ return v[0], (None if k is None else k[0])
+ return v, k
+
+ # ------------------------------------------------------------------
+ # Internals
+ # ------------------------------------------------------------------
+
+ def _as_batch(self, arr: np.ndarray, dim: int):
+ """numpy ``(dim,)`` / ``(B, dim)`` -> torch ``(B, dim)`` on the rig device."""
+ torch = self._torch
+ t = torch.as_tensor(np.ascontiguousarray(arr), dtype=torch.float32)
+ squeeze = t.dim() == 1
+ if squeeze:
+ t = t.unsqueeze(0)
+ if t.shape[-1] != dim:
+ raise ValueError(f"expected trailing dimension {dim}, got {tuple(t.shape)}")
+ return t.to(self._device), squeeze
+
+ def _regress(self, skel):
+ """``(B, 127, 8)`` skeleton state -> ``(B, 70, 3)`` keypoints (vision frame)."""
+ joints = self._to_vision(skel[..., :3].float())
+ return self._torch.einsum("kj,bjc->bkc", self._kp_regressor, joints)
+
+ def _pad(self, pose, total: int):
+ """Right-pad the 204 model parameters with zero blend-shape coefficients."""
+ torch = self._torch
+ extra = total - pose.shape[-1]
+ if extra <= 0:
+ return pose
+ return torch.cat(
+ [pose, torch.zeros(pose.shape[0], extra, dtype=pose.dtype, device=pose.device)],
+ dim=1,
+ )
+
+ def _skeleton_state(self, pose):
+ """``(B, 204)`` model parameters -> ``(B, 127, 8)`` raw skeleton state.
+
+ Skips blend shapes and skinning entirely: only the two rigid-transform
+ stages the keypoint regression needs.
+ """
+ ch = self._character
+ joint_params = ch.model_parameters_to_joint_parameters(self._pad(pose, self._n_params))
+ return ch.joint_parameters_to_skeleton_state(joint_params)
+
+ def _skin(self, identity, pose):
+ """Full rig evaluation -> ``((B, V, 3) verts, (B, 127, 8) skeleton state)``.
+
+ InstantHMR predicts no facial expression, so those coefficients are zero.
+ """
+ torch = self._torch
+ expr = torch.zeros(
+ pose.shape[0], NUM_EXPRESSION, dtype=pose.dtype, device=pose.device,
+ )
+ verts, skel = self._model(identity, pose, expr)
+ return verts, skel
+
+
+class MHRScriptRenderer(_MHRBase):
+ """MHR rig loaded from the TorchScript artefact used by the distillation.
+
+ Args:
+ script_path: path to ``mhr_model.pt`` (default: ``checkpoints/mhr_model.pt``).
+ device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.
+ kp_regressor: optional override for the (70, 127) keypoint map.
+
+ The mesh LOD is baked into the artefact (18 439 vertices / 36 874 faces);
+ use :class:`MHRRenderer` if you need to choose one.
+ """
+
+ def __init__(
+ self,
+ script_path: str | Path = DEFAULT_MHR_SCRIPT,
+ device: str = "cuda",
+ kp_regressor: str | Path | None = None,
+ ) -> None:
+ super().__init__(device=device, kp_regressor=kp_regressor)
+ torch = self._torch
+
+ script_path = Path(script_path)
+ if not script_path.exists():
+ raise FileNotFoundError(f"MHR TorchScript rig not found: {script_path}")
+
+ self._model = torch.jit.load(str(script_path), map_location=self._device).eval()
+ for p in self._model.parameters():
+ p.requires_grad_(False)
+
+ self._character = self._model.character_torch
+ # 204 model parameters + 45 identity blend-shape slots in this rig's
+ # parameter transform; read it off rather than assuming.
+ self._n_params = len(self._character.parameter_transform.parameter_names)
+
+ faces = self._character.mesh.faces
+ self._faces = faces.cpu().numpy().astype(np.int32)
+
+ @property
+ def faces(self) -> np.ndarray:
+ """Triangle face indices ``(F, 3)`` int32, shared across all frames."""
+ return self._faces
+
+
+
+class MHRRenderer(_MHRBase):
+ """MHR rig loaded from the unpacked asset folder via the ``mhr`` pip package.
Args:
- assets_folder: path to the unpacked MHR assets folder (the directory
- that contains the blend-shape files, template mesh, etc.).
+ assets_folder: path to the folder that holds ``lod*.fbx`` and
+ ``compact_v6_1.model`` (i.e. the ``assets/`` directory inside the
+ unpacked ``assets.zip``).
device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.
lod: level-of-detail mesh resolution:
0 = 73 639 verts (highest), 3 = 4 899, 6 = 595 (lowest).
LOD 3 is a good default for live demos.
+ kp_regressor: optional override for the (70, 127) keypoint map.
+
+ Note:
+ ``pymomentum-gpu`` is a torch C++ extension. If its build does not match
+ the installed torch (0.1.113 requires ``torch>=2.8,<2.9``) the process
+ segfaults inside the native library instead of raising — prefer
+ :class:`MHRScriptRenderer` when in doubt.
"""
def __init__(
@@ -46,20 +361,22 @@ class MHRRenderer:
assets_folder: str | Path = "models/mhr_assets",
device: str = "cuda",
lod: int = 3,
+ kp_regressor: str | Path | None = None,
) -> None:
- import torch
+ super().__init__(device=device, kp_regressor=kp_regressor)
from mhr.mhr import MHR
- self._torch = torch
- self._device = torch.device(device)
- self._model = MHR.from_files(
- folder=Path(assets_folder),
- device=self._device,
- lod=lod,
- )
+ folder = Path(assets_folder)
+ # assets.zip unpacks to <dir>/assets/ — accept either level.
+ if not (folder / f"lod{lod}.fbx").exists() and (folder / "assets" / f"lod{lod}.fbx").exists():
+ folder = folder / "assets"
+
+ self._model = MHR.from_files(folder=folder, device=self._device, lod=lod)
+ self._character = self._model.character_torch
+ self._n_params = self._model.character.parameter_transform.size
faces_raw = self._model.character.mesh.faces
- if isinstance(faces_raw, torch.Tensor):
+ if hasattr(faces_raw, "cpu"):
self._faces: np.ndarray = faces_raw.cpu().numpy().astype(np.int32)
else:
self._faces = np.asarray(faces_raw, dtype=np.int32)
@@ -69,42 +386,22 @@ class MHRRenderer:
"""Triangle face indices ``(F, 3)`` int32, shared across all frames."""
return self._faces
- def forward(
- self,
- mhr_params: np.ndarray,
- shape_params: np.ndarray,
- ) -> np.ndarray:
- """Decode one person's parameters into body-space mesh vertices.
- Args:
- mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.
- shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.
- Returns:
- ``(V, 3)`` float32 vertices in **body-centred** space (metres,
- Y-down). Add the person's ``cam_trans`` to move to camera space.
- """
- torch = self._torch
- identity = torch.from_numpy(shape_params).unsqueeze(0).to(self._device) # (1, 45)
- pose = torch.from_numpy(mhr_params).unsqueeze(0).to(self._device) # (1, 204)
- # MHR also accepts facial expression coefficients (72-D); InstantHMR
- # does not predict them, so we pass zeros (neutral expression).
- expr = torch.zeros(1, 72, dtype=torch.float32, device=self._device)
+def build_mhr(
+ script_path: str | Path | None = None,
+ assets_folder: str | Path | None = None,
+ device: str = "cuda",
+ lod: int = 3,
+ kp_regressor: str | Path | None = None,
+) -> _MHRBase | None:
+ """Build whichever MHR backend the caller configured.
- with torch.no_grad():
- verts, _ = self._model(identity, pose, expr)
-
- v = verts[0].cpu().numpy().astype(np.float32) # (V, 3)
-
- # --- Coordinate-space alignment -----------------------------------
- # MHR body template (FBX) is Y-up right-handed and in centimetres.
- # InstantHMR / SAM3D uses Y-down right-handed and metres.
- #
- # 1. cm → m
- v /= 100.0
- # 2. Y-up → Y-down: negate Y.
- # Negate Z as well to preserve right-handedness after the Y flip.
- v[:, 1] *= -1.0
- v[:, 2] *= -1.0
-
- return v
+ ``script_path`` wins when both are given. Returns ``None`` when neither is
+ set, so callers can treat "no MHR" as a plain absence.
+ """
+ if script_path is not None:
+ return MHRScriptRenderer(script_path, device=device, kp_regressor=kp_regressor)
+ if assets_folder is not None:
+ return MHRRenderer(assets_folder, device=device, lod=lod, kp_regressor=kp_regressor)
+ return None
diff --git a/instanthmr/pipeline.py b/instanthmr/pipeline.py
index c3e8ddd..ff5df10 100644
--- a/instanthmr/pipeline.py
+++ b/instanthmr/pipeline.py
@@ -68,6 +68,10 @@ class PosePipeline:
frame are batched into a single InstantHMR ONNX call instead
of looping. This is a meaningful win for multi-person frames
and free for single-person frames.
+ mhr: Optional MHR backend (see :mod:`instanthmr.mhr_renderer`).
+ Required for MHR-only graphs, whose 3D keypoints are derived
+ from the predicted MHR parameters instead of a 3D head; ignored
+ by graphs that carry a ``joints_3d`` output.
Example
-------
@@ -86,11 +90,19 @@ class PosePipeline:
max_persons: int = 2,
detector_stride: int = 1,
batch_persons: bool = True,
+ mhr: object | None = None,
):
if detector_stride < 1:
raise ValueError("detector_stride must be >= 1")
- self.hmr = InstantHMR(onnx_path, device=device)
+ self.hmr = InstantHMR(onnx_path, device=device, mhr=mhr)
+ if self.hmr.derives_joints_3d and mhr is None:
+ raise ValueError(
+ f"{Path(onnx_path).name} has no 'joints_3d' output (MHR-only student): "
+ "its 3D keypoints come from an MHR forward pass, so an MHR backend is "
+ "required. Pass mhr=build_mhr(...), or run demo.py with --mhr-model / "
+ "--mhr-assets."
+ )
self.detector = RFDETRDetector(
variant=detector_variant,
confidence=det_confidence,
@@ -106,6 +118,8 @@ class PosePipeline:
print(f"Warming up models (max_persons={max_persons})…", flush=True)
self.detector.warmup()
self.hmr.warmup(max_batch_size=max_persons)
+ if mhr is not None:
+ self._warmup_mhr(mhr, max_persons)
print("Ready.", flush=True)
self._max_persons = max_persons
@@ -115,6 +129,24 @@ class PosePipeline:
self._last_detections: list[dict] = []
self._last_bbox: Optional[np.ndarray] = None
+ @staticmethod
+ def _warmup_mhr(mhr, max_persons: int) -> None:
+ """Run the MHR rig once per batch shape the demo will actually use.
+
+ TorchScript specialises on input shapes over the first few calls, so an
+ un-warmed rig costs hundreds of ms on frame 1 instead of ~1 ms. Three
+ passes is what it takes for the profiling executor to settle: the first
+ records shapes, the second triggers the fusion compile, the third runs
+ the compiled kernels.
+ """
+ pose = np.zeros((max_persons, 204), dtype=np.float32)
+ shape = np.zeros((max_persons, 45), dtype=np.float32)
+ for _ in range(3):
+ mhr.forward(pose[0], shape[0]) # mesh: always single-person
+ if getattr(mhr, "has_keypoints", False):
+ for n in sorted({1, max_persons}): # keypoints: 1 and the padded batch
+ mhr.keypoints(pose[:n], shape[:n])
+
def predict(self, image_rgb: np.ndarray) -> FrameResult:
"""Detect all persons in *image_rgb* and run InstantHMR on each."""
t_total_start = time.perf_counter()
diff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.py
index e51587e..bbf9f3d 100644
--- a/instanthmr/visualizer.py
+++ b/instanthmr/visualizer.py
@@ -5,9 +5,13 @@ Logs, per frame:
text panel showing detector / HMR / render / total ms).
- ``camera`` : pinhole intrinsics so the image becomes a
proper frustum in the 3D scene.
- - ``world/persons/...`` : 3D joints + skeleton lines per person.
- - ``world/persons/.../mesh``: full MHR body mesh when an MHRRenderer is
+ - ``world/persons/...`` : the 70 3D keypoints + skeleton lines per person.
+ For an MHR-only student these come from the MHR
+ forward pass itself (see ``instanthmr.inference``).
+ - ``world/persons/.../mesh``: full MHR body mesh when an MHR backend is
provided (vertices from a true MHR forward pass).
+ Rendered semi-transparent via ``albedo_factor``
+ so the keypoints inside the body stay visible.
- ``timing/detector_ms`` : scalar plot of RF-DETR latency.
- ``timing/hmr_ms`` : scalar plot of InstantHMR latency.
- ``timing/render_ms`` : scalar plot of MHR mesh rendering latency.
@@ -32,19 +36,44 @@ from .skeleton import edges_for
if TYPE_CHECKING:
from .mhr_renderer import MHRRenderer
+#: Base skin tone of the body mesh, before the transparency factor is applied.
+MESH_COLOR = (200, 160, 130)
+#: Default mesh opacity. Low enough that the 70 keypoints read through the
+#: body, high enough that the silhouette is still legible.
+MESH_ALPHA = 0.35
+
class RerunVisualizer:
+ """Rerun logger for the demo.
+
+ Args:
+ application_id: Rerun application id.
+ spawn_viewer: open the viewer GUI on construction.
+ save_path: optional ``.rrd`` recording path.
+ mhr_renderer: MHR backend used to skin the body mesh. Any object with
+ ``.forward(mhr_params, shape_params)`` and ``.faces`` works; see
+ :mod:`instanthmr.mhr_renderer`.
+ mesh_alpha: body-mesh opacity in ``[0, 1]``. Applied through Rerun's
+ ``albedo_factor`` rather than per-vertex alpha, so one scalar
+ controls the whole surface.
+ """
+
def __init__(
self,
application_id: str = "instanthmr_demo",
spawn_viewer: bool = True,
save_path: str | None = None,
mhr_renderer: "MHRRenderer | None" = None,
+ mesh_alpha: float = MESH_ALPHA,
):
import rerun as rr
self._rr = rr
self._mhr_renderer = mhr_renderer
+ self._mesh_albedo = (
+ *MESH_COLOR,
+ int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),
+ )
rr.init(application_id, spawn=spawn_viewer)
# SAM3D / MHR convention: right-handed, Y-down camera frame.
@@ -165,6 +194,12 @@ class RerunVisualizer:
# ------------------------------------------------------------------
def _log_person(self, idx: int, person: HMRPrediction) -> None:
+ """Log the 70 3D keypoints and the skeleton wiring for one person.
+
+ For an MHR-only student ``joints_3d_cam`` was regressed from the MHR
+ skeleton that also produced the mesh, so the points sit exactly inside
+ the surface logged by :meth:`_log_person_mesh`.
+ """
rr = self._rr
path = f"world/persons/person_{idx}"
@@ -199,15 +234,16 @@ class RerunVisualizer:
faces = self._mhr_renderer.faces # (F, 3) int32, shared across frames
- n = len(verts_cam)
- skin = np.tile(np.array([[200, 160, 130, 140]], dtype=np.uint8), (n, 1))
-
+ # Tint + transparency come from `albedo_factor`, a single RGBA multiplier
+ # for the whole surface. Per-vertex colours would work too, but the
+ # viewer multiplies the two, so keeping the alpha in one place makes the
+ # `mesh_alpha` knob mean exactly what it says.
self._rr.log(
f"world/persons/person_{idx}/mesh",
self._rr.Mesh3D(
vertex_positions=verts_cam,
triangle_indices=faces,
- vertex_colors=skin,
+ albedo_factor=self._mesh_albedo,
),
)
diff --git a/instanthmr_distill_train/requirements.txt b/instanthmr_distill_train/requirements.txt
index 0dc2ea7..0d83e24 100644
--- a/instanthmr_distill_train/requirements.txt
+++ b/instanthmr_distill_train/requirements.txt
@@ -17,6 +17,15 @@ numpy>=1.24
pillow>=10
opencv-python>=4.8
+# --- LeVJEPA encoder (train_levjepa_*.py) ---
+# THIS is what was missing when the perelha job died with
+# "ModuleNotFoundError: No module named 'transformers'".
+# The weights ship their own modeling code, hence trust_remote_code=True, which
+# needs `transformers` itself plus safetensors to read the checkpoint.
+transformers>=4.44
+safetensors>=0.4
+huggingface_hub>=0.24
+
# --- utilities / logging ---
tqdm>=4.65
matplotlib>=3.7
diff --git a/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh b/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh
index 0677d92..fc57210 100755
--- a/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh
+++ b/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh
@@ -12,7 +12,7 @@ GPU_REQUIRED=true
# --data_root points at the dataset synced by datasynch_perso (see below).
# It is synced next to this project folder, i.e. ../instanthmr_data once we
# cd into instanthmr_distill_train.
-EXEC="python3 -u train_distill_mhr_only.py --data_root /datasets/instanthmr_data --num_workers 8 --batch_size 128 $useropt"
+EXEC="python3 -u train_distill_mhr_only.py --data_root /datasets/instanthmr_data --num_workers 8 --batch_size 128 --w_keypoints3d 1.0 $useropt"
# SBATCH --output/--error paths are relative to the submission directory, so
# create the log folder before submitting.
@@ -28,10 +28,9 @@ sbatch ${sbatchopt} << eof
#SBATCH --mail-type=ALL
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
-#SBATCH --output=instanthmr_distill_mhr_only/stdout_dolcino.txt
-#SBATCH --error=instanthmr_distill_mhr_only/stderr_dolcino.txt
-#SBATCH --account=dept_rob
-#SBATCH --partition=robgpu
+#SBATCH --output=instanthmr_distill_mhr_only/stdout_perelha.txt
+#SBATCH --error=instanthmr_distill_mhr_only/stderr_perelha.txt
+#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --mem-per-gpu=50000
#SBATCH --time=4-00:00:00
diff --git a/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh b/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh
new file mode 100755
index 0000000..538d592
--- /dev/null
+++ b/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh
@@ -0,0 +1,77 @@
+#!/bin/sh -e
+
+echo "example: $0 --gpu 1"
+
+# if `requirements.txt` is present, virtual environment is installed and used
+EXEC="python3 train_levjepa_cloud.py"
+
+GPU_REQUIRED=true
+
+. /pfcalcul/tools/sbatchHelpers5.sh
+
+# ---------------------------------------------------------------------------
+# BEFORE THE FIRST SUBMIT, stage the encoder ONCE from a machine with internet:
+#
+# python3 -c "from huggingface_hub import snapshot_download; \
+# print(snapshot_download('galilai-group/LeVJEPA-VideoMix-Large'))"
+# rsync -avPL <printed path>/ <cluster>:<ENCODER_DIR>/
+#
+# -L (--copy-links) is MANDATORY. A HuggingFace snapshot dir is nothing but
+# symlinks into ../../blobs/, so plain `rsync -av` copies six dangling links
+# (~650 bytes) instead of the 1.2 GB of weights, and the job then fails to load
+# the model. The trailing slash on the source flattens the snapshot into
+# ENCODER_DIR rather than nesting it under a commit-hash folder.
+#
+# Compute nodes generally cannot reach huggingface.co, so --encoder_path points
+# at that staged copy and the loader runs with local_files_only=True. Without
+# it the job dies on the first AutoModel.from_pretrained call.
+#
+# requirements.txt must contain `transformers` + `safetensors` (the weights ship
+# their own modeling code, hence trust_remote_code=True). A venv built before
+# that line was added will NOT have it -- delete venv/ so it is rebuilt, or the
+# job fails with "ModuleNotFoundError: No module named 'transformers'".
+# ---------------------------------------------------------------------------
+# NOTE the LEADING SLASH: without it this is a relative path and resolves
+# against the submission directory, so from_pretrained gets a path that does
+# not exist. Point it at the flattened snapshot (config.json + *.py +
+# model.safetensors directly inside), not at a nested commit-hash folder.
+ENCODER_DIR="/pfcalcul/work/kchalabi/envs/lstm/instanthmr_distill_train/levjepa_encoder"
+
+# batch_size is in CLIPS (16 frames each). Stills are encoded as T=1 -- 197
+# tokens instead of 3137 -- so they run at static_batch_mult x that batch.
+# The encoder is FROZEN: no backward through 305 M params, no LoRA state.
+# --max_hours 92 checkpoints and stops cleanly before SLURM's 4-day SIGKILL.
+EXEC="python3 -u train_levjepa_cloud.py --data_root /datasets/instanthmr_data --encoder_path $ENCODER_DIR --num_workers 8 --batch_size 8 --static_batch_mult 8 --epochs 30 --max_hours 92 $useropt"
+
+# SBATCH --output/--error paths are relative to the submission directory, so
+# create the log folder before submitting.
+mkdir -p instanthmr_levjepa_cloud
+
+jobMessage=$(
+######################### SBATCH launcher
+######################### #SBATCH --option # <= this is an enabled paramater
+######################### ##SBATCH --option # <= this is a disabled parameter
+sbatch ${sbatchopt} << eof
+#!/bin/bash
+#SBATCH --job-name="instanthmr_levjepa_cloud"
+#SBATCH --mail-type=ALL
+#SBATCH --ntasks=1
+#SBATCH --cpus-per-task=8
+#SBATCH --output=instanthmr_levjepa_cloud/stdout_perelha.txt
+#SBATCH --error=instanthmr_levjepa_cloud/stderr_perelha.txt
+#SBATCH --partition=gpu
+#SBATCH --gres=gpu:1
+#SBATCH --mem-per-gpu=50000
+#SBATCH --time=4-00:00:00
+
+. /pfcalcul/tools/sbatchHelpers5.sh
+
+/pfcalcul/tools/datasync /datasets/instanthmr_data
+
+
+time $EXEC
+
+eof
+)
+
+showSubmitted
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 1e00486..341f64f 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -181,6 +181,16 @@ class DistillConfig:
warmup_epochs: int = 3
grad_clip: float = 1.0
anomaly_loss_threshold: float = 100.0
+ # A skipped step updates NOTHING — not the weights, not the EMA, not the LR
+ # schedule (they all live behind the same `continue`). So a model that is
+ # anomalous on every batch is frozen for good: the loss can never come back
+ # under the threshold and the run idles until early stopping, which took
+ # ~139 wasted epochs the one time it happened. After this many CONSECUTIVE
+ # skips, roll the raw weights back to the EMA copy, scale the LR down and
+ # clear the optimiser moments; give up after `max_ema_rollbacks` attempts.
+ anomaly_skip_patience: int = 50
+ anomaly_rollback_lr_decay: float = 0.5
+ max_ema_rollbacks: int = 3
use_amp: bool = True
ema_decay: float = 0.9998
early_stop_patience: int = 150
@@ -837,6 +847,42 @@ class DistillationLoss(nn.Module):
losses['total_loss'] = sum(losses.values())
return losses
+def format_loss_terms(losses, top_k: int = 5) -> str:
+ """The largest individual loss terms, for diagnosing an anomalous batch.
+
+ The anomaly guard only ever printed the total, which says nothing about
+ which head blew up. The 2D terms are bounded (`joints_2d` is a SimCC
+ expectation over bins in [-kp2d_range, kp2d_range], and the SimCC CE by
+ log(kp2d_bins)); the parameter regressions and the FK-derived keypoint loss
+ are not, so this line is what tells the two cases apart in the job log.
+ """
+ terms = [(k.replace("loss_", ""), float(v))
+ for k, v in losses.items() if k != "total_loss"]
+ terms.sort(key=lambda kv: kv[1], reverse=True)
+ return " ".join(f"{k}={v:.1f}" for k, v in terms[:top_k])
+
+
+def scale_lr(optimizer, factor: float, scheduler=None) -> None:
+ """Multiply every learning-rate anchor in place.
+
+ OneCycleLR does not read a stored LR — it interpolates between `initial_lr`,
+ `max_lr` and `min_lr` on the param groups, and those all scale linearly with
+ the peak. Scaling the anchors therefore rescales the whole remaining
+ schedule while leaving its *position* (``last_epoch``) untouched, which is
+ what both the resume path and the rollback want.
+ """
+ for group in optimizer.param_groups:
+ for key in ("lr", "max_lr", "initial_lr", "min_lr"):
+ if key in group:
+ group[key] *= factor
+ if scheduler is not None:
+ # LRScheduler stashes these at construction and restores them from a
+ # checkpoint, so they have to move with the param groups.
+ for attr in ("base_lrs", "_last_lr"):
+ if hasattr(scheduler, attr):
+ setattr(scheduler, attr, [lr * factor for lr in getattr(scheduler, attr)])
+
+
# ============================================================
# Cell 10 — HMR Evaluation Metrics (MPJPE & PA-MPJPE)
# ============================================================
@@ -1010,6 +1056,17 @@ def train_instant_hmr():
scaler.load_state_dict(ckpt["scaler_state_dict"])
if "scheduler_state_dict" in ckpt:
scheduler.load_state_dict(ckpt["scheduler_state_dict"])
+ # Both loads above put the checkpoint's LR state back: `param_groups`
+ # (max_lr / initial_lr / min_lr) come from the optimizer state_dict and
+ # `base_lrs` from the scheduler's, which overwrites everything the
+ # freshly-built OneCycleLR just installed. Without this, --lr is
+ # SILENTLY IGNORED on every resume. Re-apply the requested peak by
+ # rescaling the anchors, keeping the schedule's shape and position.
+ ckpt_max_lr = float(optimizer.param_groups[0].get("max_lr", cfg.lr))
+ if ckpt_max_lr > 0.0 and abs(ckpt_max_lr - cfg.lr) > 1e-12:
+ scale_lr(optimizer, cfg.lr / ckpt_max_lr, scheduler)
+ print(f" LR rescaled: checkpoint peak {ckpt_max_lr:.2e} -> requested "
+ f"{cfg.lr:.2e} (current {optimizer.param_groups[0]['lr']:.2e})")
start_epoch = ckpt.get("epoch", -1) + 1
best_overall_pa = ckpt.get("val_pa_mpjpe", float('inf'))
best_raw_pa = best_overall_pa
@@ -1052,6 +1109,11 @@ def train_instant_hmr():
global_step = start_epoch * steps_per_epoch
criterion.set_step(global_step)
+ # Skips are counted ACROSS epochs — a stalled run skips whole epochs at a
+ # time, so a per-epoch counter would never trip.
+ consecutive_skips = 0
+ ema_rollbacks = 0
+ training_aborted = False
for epoch in range(start_epoch, cfg.epochs):
model.train()
tm = {'total': 0.0, '2d': 0.0, 'simcc': 0.0, '3d': 0.0, 'pose': 0.0, 'scale': 0.0, 'shape': 0.0, 'mhr': 0.0, 'reproj': 0.0}
@@ -1066,12 +1128,43 @@ def train_instant_hmr():
losses = criterion(preds_fp32, batch)
loss = losses['total_loss']
loss_val = loss.item()
+
+ skip_reason = None
if math.isnan(loss_val) or math.isinf(loss_val):
- print("⚠️ WARNING: NaN/Inf loss detected! Skipping step.")
- continue
- if loss_val > cfg.anomaly_loss_threshold:
- print(f"⚠️ WARNING: anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold} — skipping step.")
+ skip_reason = "NaN/Inf loss detected!"
+ elif loss_val > cfg.anomaly_loss_threshold:
+ skip_reason = f"anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold}"
+
+ if skip_reason is not None:
+ consecutive_skips += 1
+ print(f"⚠️ WARNING: {skip_reason} — skipping step. "
+ f"[{consecutive_skips} in a row] {format_loss_terms(losses)}")
+ if consecutive_skips >= cfg.anomaly_skip_patience:
+ if ema_rollbacks >= cfg.max_ema_rollbacks:
+ print(f"\n🛑 Aborting: still stuck after {cfg.max_ema_rollbacks} EMA "
+ f"rollback(s). The best RAW / EMA checkpoints on disk are "
+ f"unaffected, and the ONNX export still runs off "
+ f"best_student_model_v3.pth — inspect the loss terms above "
+ f"before relaunching.")
+ training_aborted = True
+ break
+ ema_rollbacks += 1
+ print(f"\n🚑 {consecutive_skips} consecutive skipped steps: the raw weights "
+ f"are outside the valid MHR region and CANNOT recover on their own, "
+ f"because a skipped step updates neither the weights nor the LR.\n"
+ f" Rolling back to the EMA weights "
+ f"(attempt {ema_rollbacks}/{cfg.max_ema_rollbacks}), scaling the LR by "
+ f"{cfg.anomaly_rollback_lr_decay} and clearing the optimiser moments.")
+ model.load_state_dict(ema_model.module.state_dict())
+ # The Adam moments are what drove the model out of the valid
+ # region; carrying them over would walk straight back out.
+ optimizer.state.clear()
+ scale_lr(optimizer, cfg.anomaly_rollback_lr_decay, scheduler)
+ print(f" LR is now {optimizer.param_groups[0]['lr']:.2e}", flush=True)
+ consecutive_skips = 0
continue
+
+ consecutive_skips = 0
scaler.scale(loss).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
@@ -1096,6 +1189,8 @@ def train_instant_hmr():
pbar.set_postfix({'Tot': f"{loss.item():.3f}",
'MHR': f"{losses.get('loss_mhr_joints', torch.tensor(0)).item():.3f}"})
pbar.close()
+ if training_aborted:
+ break
avg_train = {k: v / max(nb, 1) for k, v in tm.items()}
raw_val, raw_hmr, n_raw = run_validation(model, "raw", epoch)
diff --git a/instanthmr_distill_train/train_levjepa_cloud.py b/instanthmr_distill_train/train_levjepa_cloud.py
new file mode 100644
index 0000000..be00e98
--- /dev/null
+++ b/instanthmr_distill_train/train_levjepa_cloud.py
@@ -0,0 +1,457 @@
+#!/usr/bin/env python3
+"""Frozen-LeVJEPA MHR regression on the FULL corpus, 90/10 train/val. No LoRA.
+
+Differences from train_levjepa_lora.py (which this replaces for cluster runs):
+
+ * NO LoRA. The encoder is frozen end to end, so there is no backward pass
+ through 305 M parameters -- ~7x cheaper per clip and no adapter state.
+ * ALL SIX datasets, not just the video ones. 90/10 random split over the
+ combined pool with a fixed seed, so val is in-domain (use your own held-out
+ samples for the real generalisation check).
+ * Still images are encoded as T=1 clips, NOT by repeating the frame 16x.
+ RoPE + tubelet_size=1 make the frame count free, so a still costs 197 tokens
+ instead of 3137 for the same single label -- 16x less compute. Video and
+ still batches alternate; both go through the same decoder.
+ * OFFLINE-SAFE. --encoder_path loads a pre-downloaded snapshot directory, so
+ the compute node never needs to reach huggingface.co. See the submit script
+ for how to stage it.
+
+WHY NO LoRA
+-----------
+The local frozen probe (10k Harmony4D clips, 20 epochs) reached 101.8 mm train
+but plateaued at ~181 mm val on 3DPW with the val loss RISING after epoch 10 --
+classic overfitting. The bottleneck was training-domain diversity, not encoder
+capacity, so adding trainable encoder parameters was the wrong lever. Feeding
+the model all 961 k crops instead of 10 k Harmony4D clips is the right one.
+
+ATTENTION MODE -- DO NOT CHANGE
+-------------------------------
+attn_mode stays "block_causal" as shipped. Full attention raises no error and
+silently returns worse features.
+"""
+from __future__ import annotations
+
+import argparse
+import contextlib
+import io
+import math
+import os
+import random
+import sys
+import time
+from collections import defaultdict
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+
+import numpy as np
+import torch
+import torch.nn as nn
+from torch.utils.data import DataLoader, Dataset, Subset
+from tqdm import tqdm
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+with contextlib.redirect_stdout(io.StringIO()):
+ import train_distill_mhr_only as base
+ import train_levjepa_probe as probe
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+
+@dataclass
+class CloudConfig:
+ data_root: str = str(base.DEFAULT_DATA_ROOT)
+ mhr_model_path: str = str(base.PROJECT_ROOT / "checkpoints/mhr_model.pt")
+ kp_regressor_path: str = str(base.PROJECT_ROOT / "assets/mhr_j127_to_kp70.npy")
+ log_dir: str = str(base.PROJECT_ROOT / "runs/levjepa_cloud")
+
+ encoder_id: str = probe.ENCODER_ID
+ encoder_path: str | None = None # local snapshot dir -> no network needed
+ image_size: int = 224
+ clip_len: int = 16
+ clip_stride: int = 4
+ frame_stride: dict = field(default_factory=lambda: {
+ "sam3d_gt_harmony4d": 1, "sam3d_gt_3dpw": 4, "sam3d_distill_mix": 1})
+ # every dataset with usable temporal structure
+ video_datasets: tuple = ("sam3d_gt_harmony4d", "sam3d_distill_mix", "sam3d_gt_3dpw")
+ # every dataset without it -> encoded as T=1
+ static_datasets: tuple = ("sam3d_gt_coco", "sam3d_gt_aic", "sam3d_gt_mpii")
+ use_static: bool = True
+ val_split: float = 0.1
+ split_seed: int = 42
+ max_video_clips: int | None = None
+ max_static: int | None = None
+
+ d_model: int = 512
+ n_heads: int = 8
+ n_decoder_layers: int = 4
+ dropout: float = 0.1
+ decoder_mode: str = "perframe"
+ pose_dim: int = 136
+ scale_dim: int = 68
+ shape_dim: int = 45
+ cam_dim: int = 3
+ cliff_dim: int = 3
+ num_joints: int = 70
+ kp2d_bins: int = 96
+ kp2d_range: float = 1.5
+
+ @property
+ def model_params_dim(self) -> int:
+ return self.pose_dim + self.scale_dim
+
+ batch_size: int = 8 # video clips per batch
+ static_batch_mult: int = 8 # still batches are 16x cheaper -> bigger
+ lr: float = 3e-4
+ weight_decay: float = 1e-4
+ epochs: int = 30
+ warmup_pct: float = 0.05
+ grad_clip: float = 1.0
+ use_amp: bool = True
+ num_workers: int = 8
+ anomaly_loss_threshold: float = 100.0
+ early_stop_patience: int = 8
+ resume: bool = True
+ max_hours: float = 92.0
+ save_every_steps: int = 2000
+ # Print a progress line every N batches. tqdm is disabled when stderr is not
+ # a tty (i.e. every SLURM job) and an epoch here is HOURS long, so without
+ # this the log stays empty until the first epoch ends and the run looks hung.
+ log_every: int = 200
+
+ w_pose: float = 1.0
+ w_scale: float = 0.1
+ w_shape: float = 1.0
+ w_cam: float = 0.1
+ w_keypoints3d: float = 2.0
+ w_keypoints2d: float = 10.0
+ w_simcc: float = 1.0
+ w_3d_joints: float = 1e-3
+ w_reproj: float = 0.01
+ kp3d_warmup_steps: int = 2000
+
+ native_mapping_ids: list = field(default_factory=lambda: base.DistillConfig().native_mapping_ids)
+ mhr_mapping_ids: list = field(default_factory=lambda: base.DistillConfig().mhr_mapping_ids)
+
+
+class FrozenEncoder(nn.Module):
+ def __init__(self, cfg: CloudConfig):
+ super().__init__()
+ from transformers import AutoModel
+ src = cfg.encoder_path or cfg.encoder_id
+ if cfg.encoder_path and not Path(cfg.encoder_path).exists():
+ raise FileNotFoundError(
+ f"--encoder_path '{cfg.encoder_path}' does not exist. Stage the "
+ f"snapshot next to the data (see submit_train_levjepa_cloud_*.sh); "
+ f"compute nodes usually cannot reach huggingface.co.")
+ print(f" loading encoder from: {src}")
+ self.net = AutoModel.from_pretrained(
+ src, trust_remote_code=True, local_files_only=bool(cfg.encoder_path)
+ ).eval().to(torch.float16).to(device)
+ for p in self.net.parameters():
+ p.requires_grad_(False)
+ self.embed_dim = int(self.net.config.embed_dim)
+ self.grid = int(self.net.config.img_size // self.net.config.patch_size)
+
+ @torch.no_grad()
+ def forward(self, images):
+ """(B,T,3,H,W) -> (B,T,grid^2,C). T is free: 16 for clips, 1 for stills."""
+ B, T = images.shape[:2]
+ out = self.net(images.permute(0, 2, 1, 3, 4).to(torch.float16))
+ tok = out[0] if isinstance(out, (tuple, list)) else getattr(out, "last_hidden_state", out)
+ return tok[:, 1:].reshape(B, T, self.grid ** 2, -1).float()
+
+
+
+class CompactPairs:
+ """List-like view over two numpy byte buffers, replacing a list of Paths.
+
+ 961 k (Path, Path) tuples cost ~1.6 GB resident. Worse, CPython refcounts
+ every one of those objects, so a forked DataLoader worker dirties the pages
+ holding them and copy-on-write quietly turns into copy-everything -- with
+ several workers that is tens of GB and a host-RAM OOM kill, which is exactly
+ how job 23815712 died after 117 minutes.
+
+ Two numpy arrays are single allocations with no per-element refcounts, so
+ forks genuinely share them.
+ """
+
+ __slots__ = ("a", "b")
+
+ def __init__(self, pairs):
+ self.a = np.array([str(x[0]) for x in pairs], dtype="S")
+ self.b = np.array([str(x[1]) for x in pairs], dtype="S")
+
+ def __len__(self):
+ return len(self.a)
+
+ def __getitem__(self, i):
+ return Path(self.a[i].decode()), Path(self.b[i].decode())
+
+
+def make_base_dataset(cfg):
+ """One scan of the corpus, with the path list compacted, shared by everyone."""
+ with contextlib.redirect_stdout(io.StringIO()):
+ ds = base.SAM3DStudentDataset(cfg.data_root, augment=False,
+ image_size=cfg.image_size,
+ max_images=None, per_dataset_caps=None)
+ n = len(ds.pairs)
+ ds.pairs = CompactPairs(ds.pairs)
+ print(f" corpus: {n:,} crops (path list compacted for fork-safety)")
+ return ds
+
+
+class StaticFrameDataset(Dataset):
+ """Still images as T=1 clips -- 197 tokens, not 3137. Same labels, same losses."""
+
+ def __init__(self, cfg, datasets, max_items=None, base_ds=None):
+ self.base = base_ds if base_ds is not None else make_base_dataset(cfg)
+ keep = set(datasets)
+ self.idx = [i for i, (_, n) in enumerate(self.base.pairs)
+ if n.parent.parent.name in keep]
+ if max_items and len(self.idx) > max_items:
+ rng = np.random.RandomState(0)
+ self.idx = [self.idx[i] for i in sorted(rng.choice(len(self.idx), max_items, False))]
+ print(f" static {'/'.join(datasets)}: {len(self.idx):,} images as T=1 clips")
+
+ def __len__(self):
+ return len(self.idx)
+
+ def __getitem__(self, i):
+ f = self.base[self.idx[i]]
+ out = {}
+ for k, v in f.items():
+ v = v if torch.is_tensor(v) else torch.as_tensor(np.asarray(v))
+ out[k] = v.unsqueeze(0).float() # (1, ...) -> T=1
+ return out
+
+
+def split_90_10(ds, val_frac, seed):
+ n = len(ds)
+ g = np.random.RandomState(seed)
+ perm = g.permutation(n)
+ nv = int(n * val_frac)
+ return Subset(ds, perm[nv:].tolist()), Subset(ds, perm[:nv].tolist())
+
+
+def run_split(enc, dec, crit, cfg, mhr, loaders, opt=None, scaler=None, sched=None,
+ gstep=0, tag="train", t_start=None):
+ train = opt is not None
+ dec.train(train)
+ m, h, nb = defaultdict(float), defaultdict(float), 0
+ stop = False
+ for name, loader in loaders:
+ if loader is None:
+ continue
+ bar = tqdm(loader, desc=f"{tag}:{name}", disable=not sys.stderr.isatty(),
+ mininterval=60.0)
+ nl = 0 # per-loader counter; nb runs across loaders
+ for b in bar:
+ b = {k: v.to(device, non_blocking=True) for k, v in b.items()}
+ flat = probe.flatten_clip(b)
+ patch = enc(b["image"])
+ ctx = contextlib.nullcontext() if train else torch.no_grad()
+ with ctx:
+ with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
+ enabled=cfg.use_amp):
+ preds16 = dec(patch, b["cliff_cond"])
+ preds = {k: v.float() for k, v in preds16.items()}
+ losses = crit(preds, flat)
+ total = losses["total_loss"]
+ if train:
+ if not torch.isfinite(total) or total.item() > cfg.anomaly_loss_threshold:
+ m["skipped"] += 1
+ opt.zero_grad(set_to_none=True)
+ continue
+ opt.zero_grad(set_to_none=True)
+ scaler.scale(total).backward()
+ scaler.unscale_(opt)
+ torch.nn.utils.clip_grad_norm_(dec.parameters(), cfg.grad_clip)
+ scaler.step(opt); scaler.update()
+ if sched.last_epoch < sched.total_steps - 1:
+ sched.step()
+ gstep += 1
+ crit.set_step(gstep)
+ for k, v in losses.items():
+ m[k] += float(v.detach())
+ for k, v in base.evaluate_hmr_batch(preds, flat, cfg, mhr).items():
+ h[k] += v
+ nb += 1
+ nl += 1
+ if train and cfg.log_every and nl % cfg.log_every == 0:
+ el = (time.time() - t_start) / 60.0 if t_start else 0.0
+ print(f" [{tag}:{name}] batch {nl}/{len(loader)} | {el:6.1f} min "
+ f"| tot {m['total_loss']/nb:.4f} "
+ f"| Mesh PA {h['Mesh_PA_MPJPE']/nb:.1f} mm "
+ f"| lr {sched.get_last_lr()[0]:.2e}", flush=True)
+ if train and t_start and (time.time() - t_start) / 3600.0 > cfg.max_hours:
+ stop = True
+ break
+ bar.close()
+ if stop:
+ break
+ return ({k: v / max(nb, 1) for k, v in m.items()},
+ {k: v / max(nb, 1) for k, v in h.items()}, gstep, stop)
+
+
+def main():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--data_root", default=None)
+ p.add_argument("--output_dir", default=None)
+ p.add_argument("--mhr_model_path", default=None)
+ p.add_argument("--kp_regressor_path", default=None)
+ p.add_argument("--encoder_path", default=None,
+ help="Local HF snapshot dir for the encoder (offline nodes).")
+ p.add_argument("--clip_len", type=int, default=None)
+ p.add_argument("--clip_stride", type=int, default=None)
+ p.add_argument("--batch_size", type=int, default=None, help="video CLIPS per batch")
+ p.add_argument("--static_batch_mult", type=int, default=None)
+ p.add_argument("--epochs", type=int, default=None)
+ p.add_argument("--lr", type=float, default=None)
+ p.add_argument("--num_workers", type=int, default=None)
+ p.add_argument("--max_video_clips", type=int, default=None)
+ p.add_argument("--max_static", type=int, default=None)
+ p.add_argument("--no-static", dest="no_static", action="store_true")
+ p.add_argument("--val_split", type=float, default=None)
+ p.add_argument("--max_hours", type=float, default=None)
+ p.add_argument("--log_every", type=int, default=None,
+ help="Batches between progress lines (0 disables).")
+ p.add_argument("--no-resume", dest="no_resume", action="store_true")
+ p.add_argument("--seed", type=int, default=0)
+ p.add_argument("--self-test", dest="self_test", action="store_true")
+ args = p.parse_args()
+
+ torch.manual_seed(args.seed); np.random.seed(args.seed); random.seed(args.seed)
+ cfg = CloudConfig()
+ for k in ("data_root", "mhr_model_path", "kp_regressor_path", "encoder_path",
+ "clip_len", "clip_stride", "batch_size", "static_batch_mult", "epochs",
+ "lr", "num_workers", "max_video_clips", "max_static", "val_split",
+ "max_hours", "log_every"):
+ v = getattr(args, k)
+ if v is not None:
+ setattr(cfg, k, v)
+ if args.output_dir: cfg.log_dir = args.output_dir
+ if args.no_static: cfg.use_static = False
+ if args.no_resume: cfg.resume = False
+
+ print("=" * 74)
+ print(f"encoder : {cfg.encoder_path or cfg.encoder_id} FROZEN (no LoRA)")
+ print(f"data : video {cfg.video_datasets}")
+ print(f" static {cfg.static_datasets if cfg.use_static else '(disabled)'} as T=1")
+ print(f"split : {100*(1-cfg.val_split):.0f}/{100*cfg.val_split:.0f} random, seed {cfg.split_seed}")
+ print(f"batch : {cfg.batch_size} clips (T={cfg.clip_len}) | "
+ f"{cfg.batch_size*cfg.static_batch_mult} stills (T=1)")
+ print(f"budget : {cfg.epochs} epochs, stop at {cfg.max_hours} h, resume={cfg.resume}")
+ if torch.cuda.is_available():
+ pr = torch.cuda.get_device_properties(0)
+ print(f"GPU : {pr.name} {pr.total_memory/2**30:.1f} GiB")
+ if pr.total_memory / 2**30 < 10: # 6 GB MIG slice, not a real card
+ print("WARNING : that looks like a small MIG slice. perelha advertises "
+ "gpu:nvidia_a30_1g.6gb:4 next to the A6000 / 6000-Ada, and a bare "
+ "--gres=gpu:1 can land on one. Request the card explicitly, "
+ "e.g. --gres=gpu:rtx_a6000:1")
+ print("=" * 74)
+
+ for pth, what in ((cfg.kp_regressor_path, "(70,127) keypoint regressor"),
+ (cfg.mhr_model_path, "MHR TorchScript model")):
+ if not Path(pth).exists():
+ raise FileNotFoundError(f"{what} not found at '{pth}'. It must be rsynced "
+ f"to the cluster alongside the training scripts.")
+ W = np.load(cfg.kp_regressor_path)
+ mhr = base.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W)
+ enc = FrozenEncoder(cfg)
+ dec = probe.ClipPoseDecoder(cfg, enc.embed_dim, enc.grid).to(device)
+ crit = base.DistillationLoss(cfg, mhr).to(device)
+ print(f" encoder {sum(q.numel() for q in enc.parameters())/1e6:.1f} M frozen | "
+ f"decoder {sum(q.numel() for q in dec.parameters())/1e6:.2f} M trainable")
+
+ shared = make_base_dataset(cfg)
+ vid = probe.ClipDataset(cfg, cfg.video_datasets, max_clips=cfg.max_video_clips,
+ base_ds=shared)
+ vtr, vva = split_90_10(vid, cfg.val_split, cfg.split_seed)
+ stat = (StaticFrameDataset(cfg, cfg.static_datasets, cfg.max_static, base_ds=shared)
+ if cfg.use_static else None)
+ if stat is not None:
+ str_, sva = split_90_10(stat, cfg.val_split, cfg.split_seed)
+ else:
+ str_ = sva = None
+ print(f" train: {len(vtr):,} clips + {len(str_) if str_ else 0:,} stills | "
+ f"val: {len(vva):,} clips + {len(sva) if sva else 0:,} stills")
+
+ # persistent_workers=False on purpose: this script runs FOUR loaders
+ # (train/val x clips/stills). Persistent pools keep all four sets of workers
+ # alive simultaneously, so the resident dataset is duplicated ~4x more than
+ # necessary. Non-persistent workers exist only while their loader iterates.
+ kw = dict(num_workers=cfg.num_workers, pin_memory=True)
+ if cfg.num_workers > 0:
+ kw.update(persistent_workers=False, prefetch_factor=2)
+ mk = lambda d, bs, sh: (DataLoader(d, batch_size=bs, shuffle=sh, drop_last=sh, **kw)
+ if d is not None and len(d) else None)
+ sbs = cfg.batch_size * cfg.static_batch_mult
+ train_loaders = [("clips", mk(vtr, cfg.batch_size, True)), ("stills", mk(str_, sbs, True))]
+ val_loaders = [("clips", mk(vva, cfg.batch_size, False)), ("stills", mk(sva, sbs, False))]
+
+ if args.self_test:
+ for nm, l in train_loaders:
+ if l is None: continue
+ b = next(iter(l)); b = {k: v.to(device) for k, v in b.items()}
+ f = enc(b["image"])
+ o = dec(f, b["cliff_cond"])
+ ls = crit({k: v.float() for k, v in o.items()}, probe.flatten_clip(b))
+ print(f" {nm}: image{tuple(b['image'].shape)} -> feat{tuple(f.shape)} "
+ f"-> mhr{tuple(o['mhr_params'].shape)} | loss {float(ls['total_loss']):.4f}")
+ assert torch.isfinite(ls["total_loss"])
+ print(" SELF-TEST PASSED"); return
+
+ opt = torch.optim.AdamW(dec.parameters(), lr=cfg.lr, weight_decay=cfg.weight_decay)
+ scaler = torch.amp.GradScaler(device="cuda", enabled=cfg.use_amp)
+ spe = sum(len(l) for _, l in train_loaders if l is not None)
+ sched = torch.optim.lr_scheduler.OneCycleLR(
+ opt, max_lr=cfg.lr, epochs=cfg.epochs, steps_per_epoch=max(1, spe),
+ pct_start=cfg.warmup_pct)
+
+ os.makedirs(cfg.log_dir, exist_ok=True)
+ last_p = os.path.join(cfg.log_dir, "levjepa_cloud_last.pth")
+ best_p = os.path.join(cfg.log_dir, "levjepa_cloud_best.pth")
+ start_ep, best, no_imp, gstep = 0, float("inf"), 0, 0
+ if cfg.resume and os.path.exists(last_p):
+ ck = torch.load(last_p, map_location=device, weights_only=False)
+ dec.load_state_dict(ck["decoder"]); opt.load_state_dict(ck["optimizer"])
+ scaler.load_state_dict(ck["scaler"]); sched.load_state_dict(ck["scheduler"])
+ start_ep, best, gstep = ck["epoch"] + 1, ck.get("best", float("inf")), ck.get("gstep", 0)
+ print(f" resumed from epoch {start_ep} (best val PA {best:.1f} mm)")
+
+ def save(path, ep):
+ torch.save({"decoder": dec.state_dict(), "optimizer": opt.state_dict(),
+ "scaler": scaler.state_dict(), "scheduler": sched.state_dict(),
+ "epoch": ep, "best": best, "gstep": gstep,
+ "cfg": asdict(cfg)}, path)
+
+ t0 = time.time()
+ for ep in range(start_ep, cfg.epochs):
+ tm, th, gstep, stop = run_split(enc, dec, crit, cfg, mhr, train_loaders,
+ opt, scaler, sched, gstep, f"ep{ep+1}", t0)
+ vm, vh, _, _ = run_split(enc, dec, crit, cfg, mhr, val_loaders, tag=f"ep{ep+1}val")
+ el = (time.time() - t0) / 3600.0
+ print(f"\nEpoch {ep+1}/{cfg.epochs} | {el:.2f} h | lr {sched.get_last_lr()[0]:.2e} "
+ f"| skipped {int(tm.get('skipped',0))}")
+ print(f" [train] tot {tm['total_loss']:.4f} | 3d {tm.get('loss_3d_native',0):.4f} "
+ f"| 2d {tm.get('loss_2d_native',0):.4f} | Mesh PA {th['Mesh_PA_MPJPE']:.1f} mm")
+ print(f" [val ] tot {vm['total_loss']:.4f} | Mesh PA {vh['Mesh_PA_MPJPE']:.1f} "
+ f"| Mesh MPJPE {vh['Mesh_MPJPE']:.1f} | Mesh52 PA {vh['Mesh52_PA_MPJPE']:.1f} mm")
+ save(last_p, ep)
+ if vh["Mesh_PA_MPJPE"] < best:
+ best, no_imp = vh["Mesh_PA_MPJPE"], 0
+ save(best_p, ep); print(f" new best {best:.1f} mm")
+ else:
+ no_imp += 1
+ if no_imp >= cfg.early_stop_patience:
+ print(f" early stop after {no_imp} epochs without improvement"); break
+ if stop:
+ print(f" time budget {cfg.max_hours} h reached -- stopped cleanly"); break
+ print(f"\nBest val Mesh PA-MPJPE: {best:.1f} mm ({(time.time()-t0)/3600:.2f} h)")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/instanthmr_distill_train/train_levjepa_probe.py b/instanthmr_distill_train/train_levjepa_probe.py
index c201982..99b4e32 100644
--- a/instanthmr_distill_train/train_levjepa_probe.py
+++ b/instanthmr_distill_train/train_levjepa_probe.py
@@ -162,7 +162,17 @@ class ProbeConfig:
# ============================================================
# Clip index + dataset
# ============================================================
-TRACK_RE = re.compile(r"^(.*)_(\d{5})_p(\d+)$")
+# Non-greedy prefix + open-ended index: hardcoding \d{5} silently dropped every
+# family numbering frames with 6 digits (pilates/workout/dance inside
+# sam3d_distill_mix -- ~25k frames of real video).
+TRACK_RE = re.compile(r"^(.+?)_(\d+)_p(\d+)$")
+
+# Families that sit inside a "video" folder but are STILL IMAGES. With the
+# looser regex above they would otherwise chain consecutive COCO image ids into
+# fake tracks, feeding the model 16 unrelated photos as one clip. They are also
+# already in sam3d_gt_coco (8,514 of 10,656 crop ids identical), so skipping
+# them here loses no data.
+STATIC_IN_VIDEO_PREFIXES = ("train2014_COCO",)
def build_clip_index(pairs, frame_stride, clip_len, clip_stride, datasets=None):
@@ -175,6 +185,8 @@ def build_clip_index(pairs, frame_stride, clip_len, clip_stride, datasets=None):
sub = npz_path.parent.parent.name
if datasets is not None and sub not in datasets:
continue
+ if npz_path.stem.startswith(STATIC_IN_VIDEO_PREFIXES):
+ continue
m = TRACK_RE.match(npz_path.stem)
if not m:
continue
@@ -200,12 +212,19 @@ class ClipDataset(Dataset):
here is byte-identical to a sample in the single-image pipeline.
"""
- def __init__(self, cfg: ProbeConfig, datasets, max_clips=None, augment=False):
+ def __init__(self, cfg: ProbeConfig, datasets, max_clips=None, augment=False,
+ base_ds=None):
self.cfg = cfg
- with contextlib.redirect_stdout(io.StringIO()):
- self.base = base.SAM3DStudentDataset(
- cfg.data_root, augment=augment, image_size=cfg.image_size,
- max_images=None, per_dataset_caps=None)
+ # base_ds lets callers share ONE scan of the corpus. Building a second
+ # SAM3DStudentDataset costs another ~1.6 GB of resident Path objects,
+ # which every forked DataLoader worker then copies.
+ if base_ds is not None:
+ self.base = base_ds
+ else:
+ with contextlib.redirect_stdout(io.StringIO()):
+ self.base = base.SAM3DStudentDataset(
+ cfg.data_root, augment=augment, image_size=cfg.image_size,
+ max_images=None, per_dataset_caps=None)
self.clips = build_clip_index(
self.base.pairs, cfg.frame_stride, cfg.clip_len, cfg.clip_stride,
datasets=set(datasets))
@@ -424,7 +443,10 @@ class ClipPoseDecoder(nn.Module):
mem = self.feat_proj(patch_tokens) # (B,T,N,d)
q = self.query_embed.expand(B * T, -1, -1)
- q = q + self.temporal_embed.expand(B, -1, -1, -1).reshape(B * T, 1, -1)
+ # slice, don't assume T == cfg.clip_len: still images are encoded as
+ # T=1 clips, which costs 197 tokens instead of 3137 for the same single
+ # label -- 16x cheaper than repeating the frame to fill the clip.
+ q = q + self.temporal_embed[:, :T].expand(B, -1, -1, -1).reshape(B * T, 1, -1)
q = q + self.cond_proj(cliff_cond).reshape(B * T, 1, -1)
if self.mode == "perframe":
diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.py
index aaf5b53..c4365a8 100644
--- a/tools/pth_to_onnx.py
+++ b/tools/pth_to_onnx.py
@@ -8,18 +8,25 @@ script that produced the checkpoint, plus the FP32 export recipe from notebook
Cell 15, so the graph I/O matches ``instanthmr/inference.py``:
inputs : image (N, 3, 224, 224), cliff_cond (N, 3)
- outputs: mhr_params, shape_params, cam_trans, joints_2d, joints_3d (in this order)
+ outputs: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]
Architecture selection
----------------------
-The student head changed between training scripts: ``train_distill.py``
-regresses 2D joints directly (``head_2d_xy``), while the optimized/correctives
-scripts predict them with a SimCC classification head (``head_2d_logits`` +
-``kp2d_bin_centers``). Exporting a SimCC checkpoint through the old architecture
-silently leaves ``head_2d_xy`` at its near-zero init, which collapses every 2D
-keypoint onto the centre of the bbox. ``--arch auto`` (the default) picks the
-right module by inspecting the checkpoint, and a missing-key check makes any
-future mismatch a hard error instead of a silent one.
+The student head changed twice across training scripts:
+
+ * ``train_distill.py`` regresses 2D joints directly (``head_2d_xy``);
+ * the optimized / correctives scripts predict them with a SimCC
+ classification head (``head_2d_logits`` + ``kp2d_bin_centers``);
+ * ``train_distill_mhr_only.py`` additionally **deletes the 3D head**
+ (``head_3d``) — the 70 keypoints are derived from the MHR forward pass by
+ the consumer, so its graph has FOUR outputs instead of five.
+
+Exporting a SimCC checkpoint through the old architecture silently leaves
+``head_2d_xy`` at its near-zero init, which collapses every 2D keypoint onto
+the centre of the bbox. ``--arch auto`` (the default) picks the right module by
+inspecting the checkpoint, and a missing-key check makes any future mismatch a
+hard error instead of a silent one. The output list is read off the deploy
+wrapper itself, so a 4- vs 5-output graph never has to be configured by hand.
Usage
-----
@@ -28,8 +35,8 @@ Usage
# or point at a specific checkpoint / output
python tools/pth_to_onnx.py \
- --ckpt instanthmr_distill_train/runs/distill_optimized_v2/best_student_model_ema.pth \
- --output models/instanthmr.onnx
+ --ckpt instanthmr_distill_train/runs/distill_mhr_only/distill_mhr_only/best_student_model_ema.pth \
+ --output models/instanthmr_mhr_only.onnx
"""
import argparse
import importlib
@@ -46,15 +53,26 @@ sys.path.insert(0, str(TRAIN_DIR))
# arch name -> training module that defines the matching InstantHMRStudent.
ARCH_MODULES = {
+ "mhr_only": "train_distill_mhr_only", # SimCC 2D head, NO 3D head
"correctives": "train_distill_optimized_correctives", # SimCC 2D head (head_2d_logits)
"optimized": "train_distill_optimized", # SimCC 2D head, identical weights
"legacy": "train_distill", # direct 2D regression (head_2d_xy)
}
+# Canonical ONNX output order. The MHR-only graph stops after `joints_2d`.
+ALL_OUTPUT_KEYS = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]
+
def detect_arch(state: dict) -> str:
- """Pick the architecture that matches the checkpoint's 2D head."""
- if "head_2d_logits.weight" in state:
+ """Pick the architecture that matches the checkpoint's prediction heads."""
+ has_3d = any(k.startswith("head_3d.") for k in state)
+ has_simcc = "head_2d_logits.weight" in state
+
+ if has_simcc and not has_3d:
+ # train_distill_mhr_only.py: the 3D coordinate head was removed and the
+ # decoder shrank from 141 to 71 queries.
+ return "mhr_only"
+ if has_simcc:
# `optimized` and `correctives` share identical parameters (they differ
# only by an fp32 softmax cast), so either module loads such a
# checkpoint; prefer the newer one.
@@ -126,9 +144,22 @@ def main() -> None:
deploy = T.HMRDeployWrapper(model).eval()
# --- Export FP32 ONNX (Cell 15 recipe) ---
- output_keys = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]
dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)
dummy_cliff = torch.randn(1, 3, device=device)
+
+ # Ask the wrapper how many tensors it returns instead of hard-coding five:
+ # train_distill_mhr_only.py drops `joints_3d` from the tuple.
+ with torch.no_grad():
+ n_out = len(deploy(dummy_img, dummy_cliff))
+ if n_out not in (4, 5):
+ sys.exit(f"[error] unexpected deploy wrapper output count: {n_out} (expected 4 or 5)")
+ output_keys = ALL_OUTPUT_KEYS[:n_out]
+ print(f" graph outputs: {n_out} -> {output_keys}")
+ if n_out == 4:
+ print(" note: no 'joints_3d' output — the 70 keypoints are derived from the\n"
+ " MHR forward pass at inference time (needs --mhr-model / --mhr-assets\n"
+ " in demo.py).")
+
dynamic_axes = {"image": {0: "batch"}, "cliff_cond": {0: "batch"}}
for k in output_keys:
dynamic_axes[k] = {0: "batch"}