Python
feat: demo now runs with mhr_only model, added benchmarking scripts for 3dpw and coco
mohamdevb29e7d740291
diff --git a/benchmark/README.md b/benchmark/README.mdnew file mode 100644index 0000000..f22f02e--- /dev/null+++ b/benchmark/README.md@@ -0,0 +1,142 @@+# InstantHMR benchmarks++Evaluation harness for publishing numbers on `models/instanthmr_mhr_only_ckpt90.onnx`.++Two benchmarks are implemented, chosen because they are the two cheapest paths+to a credible table — one 3D, one 2D:++| Benchmark | What it measures | Data cost | Status |+|---|---|---|---|+| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT, free licence form | ready — **but the test split is contaminated, see below** |+| **COCO** val2017 | OKS AP / PCK, 2D keypoint localisation in the wild | 820 MB, direct download, no registration | ready — clean |++## Why these two++**3DPW** is the benchmark essentially every monocular HMR paper reports, so it+is the one that makes your numbers comparable. It is also unusually cheap here:+the GT pickles store `jointPositions` (24 SMPL joints in world coordinates)+directly, so **no SMPL model download or body-model forward pass is needed** —+which is what makes most 3DPW harnesses painful to set up.++**COCO val2017** needs no registration at all and the joint mapping is free:+MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.++### The ones deliberately skipped++- **Human3.6M** — the licence forbids redistribution, the download is manual and+ slow, and protocol variants (P1 vs P2, 17 vs 14 joints) differ per paper. High+ effort, high chance of an unfair comparison.+- **EMDB** — a genuinely good modern benchmark (EMDB-1 for pose), but needs+ registration *and* the SMPL body model *and* their toolkit to unpack. Worth+ adding as a second 3D benchmark after 3DPW is in place; `benchlib/threedpw.py`+ is the template to copy.+- **RICH** — SMPL-X GT, large download, mostly used for contact estimation.+- **MPII** — 2D only, so it adds little once COCO is in.++## Setup++```bash+# COCO — fully automatic (~820 MB)+python benchmark/download.py --coco++# 3DPW — accept the licence at+# https://virtualhumans.mpi-inf.mpg.de/3DPW/licence.html+# then unzip sequenceFiles.zip (and imageFiles.zip if you don't have it) and check:+python benchmark/download.py --check-3dpw \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --image-root /path/to/3DPW/imageFiles+```++## Running++```bash+# COCO val2017 (~10 min on an RTX 4070)+python benchmark/eval_coco.py \+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \+ --coco-root benchmark/data/coco++# 3DPW test — add --stride 10 for a 10x faster smoke run+python benchmark/eval_3dpw.py \+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --image-root /path/to/3DPW/imageFiles \+ --exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations+```++Both write a JSON report to `benchmark/results/`.++## Protocols, stated precisely++Write these into the paper — they are the details a reviewer will ask about.++**3DPW test.** All 24 test sequences, every frame where `campose_valid` is set+and the person is at least 60 % inside the image. Person boxes come from the+projected GT joints padded by `--bbox-scale` (default 1.2), so no detector is in+the loop. MPJPE is mid-hip-aligned; PA-MPJPE is Procrustes (scale + rotation ++translation). Predictions use `joints_3d_local`, the rig-local pose, so the+camera translation never enters the metric.++**COCO val2017.** Top-down with **ground-truth boxes**, one crop per annotated+person with `num_keypoints ≥ 1` and `iscrowd = 0`. These AP numbers are *not*+comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits no+per-keypoint confidence, so every keypoint is scored 1.0 — read AP next to the+confidence-free `mean_OKS` and `PCK` numbers.++### The joint-convention caveat — do not skip this++InstantHMR predicts MHR70 joints; 3DPW's GT is SMPL. The two rigs place several+landmarks differently, which puts a floor under the error that has nothing to do+with model quality. The harness therefore reports three rows:++- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` are the+ two joints where the rigs genuinely disagree (MHR has no head-top; `nose` is+ substituted), so this row carries a systematic penalty.+- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles), where+ both rigs mean the same anatomical point. **This is the cleanest measure of+ the model.**+- **J14+adapter** *(optional)* — a linear MHR70 → SMPL-J14 joint regressor,+ fitted on 3DPW **train**, that absorbs the rig offset. Standard practice when+ a method's joint convention differs from the benchmark's. Fit it once, then+ apply to test:++ ```bash+ python benchmark/eval_3dpw.py --onnx ... --split train --stride 10 \+ --fit-adapter benchmark/results/adapter_j14.npz+ python benchmark/eval_3dpw.py --onnx ... --split test \+ --adapter benchmark/results/adapter_j14.npz+ ```++ Fit on train, report on test — never fit on the split you report.++### Train/test contamination — READ THIS BEFORE PUBLISHING++**COCO val2017 is clean.** All 25,594 COCO images in `data/` are+`COCO_train2014_*`. COCO's 2017 split re-partitions the 2014 data such that+val2017 is a 5k subset of val2014, which is disjoint from train2014. Verified by+image-id intersection: **0 of 5000** val2017 images appear in training.++**3DPW test is contaminated.** `data/sam3d_gt_3dpw` holds only train sequences,+but `data/sam3d_distill_mix` contains teacher-distilled crops from **all 60**+3DPW sequences — including all 24 test sequences (9,060 person-crops, ~28.5k+distinct frames across the dataset). `--exclude-seen` quantifies this: it splits+the report into frames the model saw during training and frames it did not.++Note that even the "not in training data" subset is only *frame*-level held out.+Neighbouring frames of the same person in the same scene were seen, so it still+overstates true generalisation. **To publish a 3DPW number, retrain with the 24+test sequences removed from `sam3d_distill_mix`.** The contamination-split+report tells you how large the effect is in the meantime.++## Layout++```+benchmark/+ download.py COCO fetcher + 3DPW presence check+ eval_3dpw.py 3D: MPJPE, PA-MPJPE, PCK3D, AUC, per-sequence breakdown+ eval_coco.py 2D: OKS AP/AR, mean OKS, PCK, NME, per-keypoint breakdown+ benchlib/+ joints.py MHR70 <-> COCO17 / SMPL24 index maps, J14 & J12 sets+ metrics.py Procrustes alignment and the metric definitions+ threedpw.py 3DPW pickle loading, projection, box derivation+ runner.py batched ONNX inference with overlapped image decoding+```diff --git a/benchmark/benchlib/__init__.py b/benchmark/benchlib/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/benchmark/benchlib/joints.py b/benchmark/benchlib/joints.pynew file mode 100644index 0000000..f4e08dd--- /dev/null+++ b/benchmark/benchlib/joints.py@@ -0,0 +1,98 @@+"""Joint-convention bookkeeping for the benchmarks.++InstantHMR regresses 70 keypoints in the MHR70 ordering (see+``instanthmr.skeleton.JOINT_NAMES``). Public benchmarks use other conventions:+COCO's 17 keypoints for 2D, and SMPL's 24 body joints for 3DPW. This module+holds the index maps between them, plus the two evaluation joint sets used for+3D metrics.++A note that matters for the paper: MHR and SMPL are different rigs, so even a+perfect prediction carries a systematic offset at joints whose *definition*+differs (notably ``neck`` and ``head``). The J12 set below contains only the+limb joints, where both rigs agree on the anatomy; J14 adds neck + head to match+the joint count published by most 3DPW papers. Report both.+"""++from __future__ import annotations++import numpy as np++# --------------------------------------------------------------------------+# MHR70 (the model's own ordering)+# --------------------------------------------------------------------------+# Indices we rely on. Body landmarks 0-14 happen to follow COCO's ordering,+# but the wrists live out in the hand block at 41 / 62.+MHR = {+ "nose": 0, "left_eye": 1, "right_eye": 2, "left_ear": 3, "right_ear": 4,+ "left_shoulder": 5, "right_shoulder": 6,+ "left_elbow": 7, "right_elbow": 8,+ "left_hip": 9, "right_hip": 10,+ "left_knee": 11, "right_knee": 12,+ "left_ankle": 13, "right_ankle": 14,+ "right_wrist": 41, "left_wrist": 62,+ "neck": 69,+}+NUM_MHR_JOINTS = 70++# --------------------------------------------------------------------------+# COCO 17 keypoints+# --------------------------------------------------------------------------+COCO17_NAMES = [+ "nose", "left_eye", "right_eye", "left_ear", "right_ear",+ "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",+ "left_wrist", "right_wrist", "left_hip", "right_hip",+ "left_knee", "right_knee", "left_ankle", "right_ankle",+]+# MHR70 index for each COCO keypoint, in COCO order.+COCO17_FROM_MHR70 = np.array(+ [MHR[n] for n in COCO17_NAMES], dtype=np.int64+)++# --------------------------------------------------------------------------+# SMPL 24 body joints (the ordering of 3DPW's ``jointPositions``)+# --------------------------------------------------------------------------+SMPL24_NAMES = [+ "pelvis", "left_hip", "right_hip", "spine1", "left_knee", "right_knee",+ "spine2", "left_ankle", "right_ankle", "spine3", "left_foot", "right_foot",+ "neck", "left_collar", "right_collar", "head", "left_shoulder",+ "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist",+ "left_hand", "right_hand",+]+SMPL = {n: i for i, n in enumerate(SMPL24_NAMES)}++# --------------------------------------------------------------------------+# 3D evaluation joint sets+# --------------------------------------------------------------------------+# LSP-14 ordering, the joint set 3DPW papers report on.+J14_NAMES = [+ "right_ankle", "right_knee", "right_hip",+ "left_hip", "left_knee", "left_ankle",+ "right_wrist", "right_elbow", "right_shoulder",+ "left_shoulder", "left_elbow", "left_wrist",+ "neck", "head",+]+# The first 12 are limbs — anatomically the same landmark in both rigs.+J12_NAMES = J14_NAMES[:12]++# MHR70 has no head-top joint; ``nose`` is the closest available landmark and is+# only used for the J14 numbers (flagged as approximate in the report).+_MHR_FOR_J14 = dict(MHR, head=MHR["nose"])++J14_FROM_MHR70 = np.array([_MHR_FOR_J14[n] for n in J14_NAMES], dtype=np.int64)+J14_FROM_SMPL24 = np.array([SMPL[n] for n in J14_NAMES], dtype=np.int64)+J12_FROM_MHR70 = J14_FROM_MHR70[:12]+J12_FROM_SMPL24 = J14_FROM_SMPL24[:12]++# Pelvis for root alignment: the mid-hip, computed the same way on both sides.+J14_HIP_IDX = (J14_NAMES.index("left_hip"), J14_NAMES.index("right_hip"))++JOINT_SETS = {+ "J14": dict(names=J14_NAMES, mhr=J14_FROM_MHR70, smpl=J14_FROM_SMPL24),+ "J12": dict(names=J12_NAMES, mhr=J12_FROM_MHR70, smpl=J12_FROM_SMPL24),+}+++def pelvis(joints: np.ndarray) -> np.ndarray:+ """Mid-hip root for a (..., J, 3) array indexed in a J14/J12 joint set."""+ li, ri = J14_HIP_IDX+ return 0.5 * (joints[..., li, :] + joints[..., ri, :])diff --git a/benchmark/benchlib/metrics.py b/benchmark/benchlib/metrics.pynew file mode 100644index 0000000..1867e39--- /dev/null+++ b/benchmark/benchlib/metrics.py@@ -0,0 +1,108 @@+"""3D and 2D pose metrics.++All 3D functions take arrays in millimetres and return millimetres.+"""++from __future__ import annotations++import numpy as np+++# --------------------------------------------------------------------------+# 3D+# --------------------------------------------------------------------------+def similarity_transform(pred: np.ndarray, gt: np.ndarray) -> np.ndarray:+ """Procrustes-align ``pred`` onto ``gt`` (scale + rotation + translation).++ Args:+ pred: (N, J, 3) predicted joints.+ gt: (N, J, 3) ground-truth joints.++ Returns:+ (N, J, 3) the aligned prediction.+ """+ pred = np.asarray(pred, dtype=np.float64)+ gt = np.asarray(gt, dtype=np.float64)++ mu_p = pred.mean(axis=1, keepdims=True)+ mu_g = gt.mean(axis=1, keepdims=True)+ X, Y = pred - mu_p, gt - mu_g++ # Per-sample covariance of the centred point sets.+ K = np.einsum("nji,njk->nik", X, Y)+ U, S, Vt = np.linalg.svd(K)++ # Reflection guard: force det(R) = +1 so we never mirror the skeleton.+ Z = np.zeros_like(U) + np.eye(3)+ Z[:, 2, 2] = np.sign(np.linalg.det(np.einsum("nij,njk->nik", U, Vt)))+ R = np.einsum("nij,njk,nlk->nil", Vt.transpose(0, 2, 1), Z, U)++ var_x = (X ** 2).sum(axis=(1, 2))+ trace = (S * np.diagonal(Z, axis1=1, axis2=2)).sum(axis=1)+ scale = np.where(var_x > 0, trace / np.maximum(var_x, 1e-12), 1.0)++ aligned = scale[:, None, None] * np.einsum("nij,nkj->nki", R, X) + mu_g+ return aligned+++def per_joint_error(pred: np.ndarray, gt: np.ndarray) -> np.ndarray:+ """(N, J) Euclidean distance per joint."""+ return np.linalg.norm(np.asarray(pred) - np.asarray(gt), axis=-1)+++def mpjpe(pred: np.ndarray, gt: np.ndarray, root: np.ndarray | None = None) -> np.ndarray:+ """Root-aligned MPJPE, (N, J) per-joint errors.++ ``root`` is an (N, 3) origin subtracted from both sets. Pass ``None`` to+ compare without any alignment (i.e. absolute-position error).+ """+ if root is not None:+ pred = pred - root[0][:, None, :]+ gt = gt - root[1][:, None, :]+ return per_joint_error(pred, gt)+++def pa_mpjpe(pred: np.ndarray, gt: np.ndarray) -> np.ndarray:+ """Procrustes-aligned MPJPE, (N, J) per-joint errors."""+ return per_joint_error(similarity_transform(pred, gt), gt)+++def pck3d(errors: np.ndarray, threshold: float) -> float:+ """Fraction of joints under ``threshold`` mm."""+ return float((errors < threshold).mean())+++def auc3d(errors: np.ndarray, lo: float = 0.0, hi: float = 150.0,+ steps: int = 31) -> float:+ """Area under the PCK-vs-threshold curve over [lo, hi] mm."""+ ts = np.linspace(lo, hi, steps)+ return float(np.mean([pck3d(errors, t) for t in ts]))+++# --------------------------------------------------------------------------+# 2D+# --------------------------------------------------------------------------+def pck2d(pred: np.ndarray, gt: np.ndarray, valid: np.ndarray,+ norm: np.ndarray, threshold: float) -> float:+ """2D PCK at ``threshold`` * ``norm``.++ Args:+ pred: (N, J, 2) predicted pixel coords.+ gt: (N, J, 2) ground-truth pixel coords.+ valid: (N, J) bool, which joints are labelled.+ norm: (N,) per-sample normaliser in pixels (e.g. bbox diagonal).+ threshold: fraction of ``norm`` counted as correct.+ """+ d = np.linalg.norm(pred - gt, axis=-1) / np.maximum(norm[:, None], 1e-6)+ if not valid.any():+ return float("nan")+ return float((d[valid] < threshold).mean())+++def nme2d(pred: np.ndarray, gt: np.ndarray, valid: np.ndarray,+ norm: np.ndarray) -> float:+ """Normalised mean error: mean pixel distance / ``norm``."""+ d = np.linalg.norm(pred - gt, axis=-1) / np.maximum(norm[:, None], 1e-6)+ if not valid.any():+ return float("nan")+ return float(d[valid].mean())diff --git a/benchmark/benchlib/runner.py b/benchmark/benchlib/runner.pynew file mode 100644index 0000000..cf88fbb--- /dev/null+++ b/benchmark/benchlib/runner.py@@ -0,0 +1,119 @@+"""Batched InstantHMR inference over a flat list of (image, bbox) samples.++The public API is :func:`run_samples`. It groups samples by source image so+each frame is decoded once and all its people go through the ONNX graph in a+single call, and it overlaps JPEG decoding with GPU work via a small prefetch+pool. Predictions come back in the input order.+"""++from __future__ import annotations++import sys+from concurrent.futures import ThreadPoolExecutor+from dataclasses import dataclass+from pathlib import Path++import cv2+import numpy as np+from tqdm import tqdm++REPO_ROOT = Path(__file__).resolve().parents[2]+if str(REPO_ROOT) not in sys.path:+ sys.path.insert(0, str(REPO_ROOT))++from instanthmr import InstantHMR # noqa: E402+++@dataclass+class Sample:+ """One person to evaluate."""+ image_path: str+ bbox: np.ndarray # (4,) xyxy in original image pixels+ meta: dict # carried through untouched+++def build_model(onnx_path: str, device: str = "cuda",+ mhr_script: str | None = None) -> InstantHMR:+ """Create the ONNX session, attaching the MHR rig when the graph needs it.++ The ``mhr_only`` graphs have no 3D head: their joints are derived by running+ the predicted MHR parameters through the rig, so a backend is mandatory.+ """+ from instanthmr.inference import onnx_output_names++ names = onnx_output_names(onnx_path) or []+ mhr = None+ if "joints_3d" not in names:+ from instanthmr.mhr_renderer import build_mhr+ script = mhr_script or str(REPO_ROOT / "checkpoints" / "mhr_model.pt")+ if not Path(script).exists():+ raise FileNotFoundError(+ f"this graph has no joints_3d output and needs the MHR rig, "+ f"but {script} is missing"+ )+ mhr = build_mhr(script_path=script, device=device)+ return InstantHMR(onnx_path, device=device, mhr=mhr)+++def _group_by_image(samples: list[Sample]) -> list[tuple[str, list[int]]]:+ """Group sample indices by image path, preserving first-seen order."""+ order: list[str] = []+ groups: dict[str, list[int]] = {}+ for i, s in enumerate(samples):+ if s.image_path not in groups:+ groups[s.image_path] = []+ order.append(s.image_path)+ groups[s.image_path].append(i)+ return [(p, groups[p]) for p in order]+++def _load_rgb(path: str) -> np.ndarray | None:+ img = cv2.imread(path, cv2.IMREAD_COLOR)+ if img is None:+ return None+ return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)+++def run_samples(model: InstantHMR, samples: list[Sample],+ num_workers: int = 6, desc: str = "inference"):+ """Run the model over every sample.++ Returns:+ ``(predictions, ok)`` where ``predictions`` is a list the same length as+ ``samples`` (``None`` where the image could not be read) and ``ok`` is a+ boolean mask of the usable entries.+ """+ groups = _group_by_image(samples)+ preds: list[object | None] = [None] * len(samples)++ with ThreadPoolExecutor(max_workers=num_workers) as pool:+ # Keep a bounded window of decoded frames in flight ahead of the GPU.+ window = max(2 * num_workers, 8)+ pending: dict[int, object] = {}+ next_submit = 0++ def submit_until(limit: int) -> None:+ nonlocal next_submit+ while next_submit < min(limit, len(groups)):+ pending[next_submit] = pool.submit(_load_rgb, groups[next_submit][0])+ next_submit += 1++ submit_until(window)+ for gi in tqdm(range(len(groups)), desc=desc, unit="img"):+ path, idxs = groups[gi]+ image = pending.pop(gi).result()+ submit_until(gi + 1 + window)+ if image is None:+ continue+ dets = [{"bbox": samples[i].bbox, "confidence": 1.0} for i in idxs]+ out = model.predict_batch(image, dets)+ for i, p in zip(idxs, out):+ preds[i] = p++ ok = np.array([p is not None for p in preds], dtype=bool)+ return preds, ok+++def stack_joints(preds, ok, attr: str) -> np.ndarray:+ """Stack one joint array (e.g. ``joints_3d_local``) over the valid preds."""+ return np.stack([getattr(p, attr) for p, k in zip(preds, ok) if k]).astype(np.float64)diff --git a/benchmark/benchlib/threedpw.py b/benchmark/benchlib/threedpw.pynew file mode 100644index 0000000..d5a4417--- /dev/null+++ b/benchmark/benchlib/threedpw.py@@ -0,0 +1,180 @@+"""3DPW ground truth loading.++3DPW ships per-sequence pickles under ``sequenceFiles/<split>/<seq>.pkl``. The+useful fields for us:++ jointPositions list[P] of (F, 72) 24 SMPL joints, **world** coords, metres+ cam_poses (F, 4, 4) world -> camera extrinsics (OpenCV frame)+ cam_intrinsics (3, 3) shared camera matrix+ campose_valid list[P] of (F,) 1 where the camera pose is trustworthy+ genders list[P]++Because ``jointPositions`` is stored directly, evaluating 3D joint error needs+**no SMPL model download** — we never have to re-run the body model.+"""++from __future__ import annotations++import pickle+from pathlib import Path++import numpy as np++TEST_SEQUENCES = 24 # sanity check: the official test split has 24 sequences+++def load_sequence(pkl_path: str | Path) -> dict:+ with open(pkl_path, "rb") as f:+ return pickle.load(f, encoding="latin1")+++def project(points_cam: np.ndarray, K: np.ndarray) -> np.ndarray:+ """Pinhole-project (..., 3) camera-space points to (..., 2) pixels."""+ z = np.maximum(points_cam[..., 2:3], 1e-6)+ uv = points_cam[..., :2] / z+ return uv * np.array([K[0, 0], K[1, 1]]) + np.array([K[0, 2], K[1, 2]])+++def bbox_from_points(uv: np.ndarray, scale: float) -> np.ndarray:+ """Square-ish xyxy box around 2D points, expanded by ``scale``.++ The GT joints stop at the wrists/ankles and the head joint sits inside the+ skull, so the raw joint extent is tighter than a person box. ``scale``+ (default 1.2 at the call site) pads it out to roughly a detector box.+ """+ x1, y1 = uv.min(axis=0)+ x2, y2 = uv.max(axis=0)+ cx, cy = (x1 + x2) / 2.0, (y1 + y2) / 2.0+ w, h = (x2 - x1) * scale, (y2 - y1) * scale+ return np.array([cx - w / 2, cy - h / 2, cx + w / 2, cy + h / 2],+ dtype=np.float32)+++def build_samples(sequence_dir: str | Path, image_root: str | Path,+ split: str = "test", stride: int = 1,+ bbox_scale: float = 1.2,+ min_visible_frac: float = 0.6):+ """Enumerate every evaluatable (frame, person) in a 3DPW split.++ Args:+ sequence_dir: ``sequenceFiles`` directory.+ image_root: ``imageFiles`` directory.+ split: ``test`` / ``train`` / ``validation``.+ stride: keep every Nth frame (1 = all frames, the standard protocol).+ bbox_scale: padding applied to the projected-joint box.+ min_visible_frac: skip a person when fewer than this fraction of their+ projected joints fall inside the image — they are mostly out of+ frame, and a crop around them is meaningless.++ Returns:+ ``(samples, gt_joints)`` — a list of dicts and an (N, 24, 3) array of+ camera-space GT joints in metres, index-aligned.+ """+ seq_dir = Path(sequence_dir) / split+ if not seq_dir.is_dir():+ raise FileNotFoundError(f"no such 3DPW split directory: {seq_dir}")+ image_root = Path(image_root)++ samples: list[dict] = []+ gts: list[np.ndarray] = []++ for pkl in sorted(seq_dir.glob("*.pkl")):+ seq = load_sequence(pkl)+ name = seq["sequence"]+ K = np.asarray(seq["cam_intrinsics"], dtype=np.float64)+ cam_poses = np.asarray(seq["cam_poses"], dtype=np.float64)+ num_frames = cam_poses.shape[0]++ for pid, jp in enumerate(seq["jointPositions"]):+ joints_w = np.asarray(jp, dtype=np.float64).reshape(-1, 24, 3)+ valid = np.asarray(seq["campose_valid"][pid]).astype(bool)++ for f in range(0, min(num_frames, joints_w.shape[0]), stride):+ if not valid[f]:+ continue+ T = cam_poses[f]+ joints_cam = joints_w[f] @ T[:3, :3].T + T[:3, 3]+ if (joints_cam[:, 2] <= 0).any(): # behind the camera+ continue++ uv = project(joints_cam, K)+ img_path = image_root / name / f"image_{f:05d}.jpg"+ samples.append(dict(+ image_path=str(img_path),+ bbox=bbox_from_points(uv, bbox_scale),+ uv=uv.astype(np.float32),+ sequence=name, person=pid, frame=f,+ gender=str(seq["genders"][pid]),+ ))+ gts.append(joints_cam)++ if not samples:+ raise RuntimeError(f"no valid samples found in {seq_dir}")++ # Drop people who are mostly outside the frame; needs image size, which we+ # take from the first image of each sequence (3DPW is constant-resolution).+ return _filter_offscreen(samples, np.stack(gts), image_root, min_visible_frac)+++def _filter_offscreen(samples, gts, image_root: Path, min_visible_frac: float):+ import cv2++ sizes: dict[str, tuple[int, int]] = {}+ keep = np.ones(len(samples), dtype=bool)+ for i, s in enumerate(samples):+ seq = s["sequence"]+ if seq not in sizes:+ img = cv2.imread(s["image_path"], cv2.IMREAD_COLOR)+ if img is None:+ sizes[seq] = (0, 0)+ else:+ sizes[seq] = img.shape[:2]+ h, w = sizes[seq]+ if h == 0:+ keep[i] = False+ continue+ uv = s["uv"]+ inside = ((uv[:, 0] >= 0) & (uv[:, 0] < w)+ & (uv[:, 1] >= 0) & (uv[:, 1] < h))+ if inside.mean() < min_visible_frac:+ keep[i] = False++ samples = [s for s, k in zip(samples, keep) if k]+ return samples, gts[keep]+++# --------------------------------------------------------------------------+# Train/test contamination+# --------------------------------------------------------------------------+# The distillation folders store one npz per person-crop, named after the source+# frame. Two layouts appear in this repo:+# data/sam3d_distill_mix : imageFiles_<sequence>_image_<frame>_p<n>.npz+# data/sam3d_gt_3dpw : 3dpw_<sequence>_image_<frame>_p<n>.npz+_SEEN_PATTERNS = (+ r"imageFiles_(?P<seq>.+?)_image_(?P<frame>\d+)_p\d+\.npz$",+ r"3dpw_(?P<seq>.+?)_image_(?P<frame>\d+)_p\d+\.npz$",+)+++def seen_frames(training_dirs) -> set[tuple[str, int]]:+ """Collect every (sequence, frame) that appears in the training data.++ Used to quantify — and optionally exclude — 3DPW frames the model was+ trained on. Returns an empty set when no directory exists.+ """+ import os+ import re++ compiled = [re.compile(p) for p in _SEEN_PATTERNS]+ seen: set[tuple[str, int]] = set()+ for d in training_dirs:+ d = Path(d)+ if not d.is_dir():+ continue+ for fname in os.listdir(d):+ for rx in compiled:+ m = rx.match(fname)+ if m:+ seen.add((m.group("seq"), int(m.group("frame"))))+ break+ return seendiff --git a/benchmark/download.py b/benchmark/download.pynew file mode 100644index 0000000..32eb140--- /dev/null+++ b/benchmark/download.py@@ -0,0 +1,124 @@+#!/usr/bin/env python3+"""Fetch the benchmark data.++COCO val2017 is a direct public download and is handled fully automatically.+3DPW requires accepting a licence on the MPI page, so this script cannot fetch+it for you — it verifies what you already have and tells you what is missing.++ python benchmark/download.py --coco+ python benchmark/download.py --check-3dpw --sequence-dir ... --image-root ...+"""++from __future__ import annotations++import argparse+import sys+import urllib.request+import zipfile+from pathlib import Path++COCO_FILES = [+ ("http://images.cocodataset.org/annotations/annotations_trainval2017.zip",+ "annotations_trainval2017.zip", "annotations/person_keypoints_val2017.json"),+ ("http://images.cocodataset.org/zips/val2017.zip",+ "val2017.zip", "val2017"),+]++THREEDPW_HELP = """+3DPW must be downloaded from the official page (free, licence form):++ https://virtualhumans.mpi-inf.mpg.de/3DPW/licence.html++Fill the form; the confirmation mail carries the download links. You need:++ sequenceFiles.zip ~ 30 MB <-- the ground truth (required)+ imageFiles.zip ~ 4.6 GB <-- the RGB frames++Unzip so the layout is:++ <3dpw_root>/sequenceFiles/{train,validation,test}/*.pkl+ <3dpw_root>/imageFiles/<sequence>/image_XXXXX.jpg+"""+++def _progress(block_num, block_size, total):+ if total <= 0:+ return+ done = min(block_num * block_size, total)+ pct = 100.0 * done / total+ sys.stdout.write(f"\r {pct:5.1f}% ({done/1e6:.0f} / {total/1e6:.0f} MB)")+ sys.stdout.flush()+++def download_coco(root: Path) -> None:+ root.mkdir(parents=True, exist_ok=True)+ for url, zip_name, marker in COCO_FILES:+ if (root / marker).exists():+ print(f"[coco] {marker} already present — skipping")+ continue+ zip_path = root / zip_name+ if not zip_path.exists():+ print(f"[coco] downloading {url}")+ urllib.request.urlretrieve(url, zip_path, _progress)+ print()+ print(f"[coco] extracting {zip_name}")+ with zipfile.ZipFile(zip_path) as z:+ # The annotations zip carries train2017 too; we only need val.+ members = [m for m in z.namelist()+ if "train2017" not in m or "val" in m]+ z.extractall(root, members=members)+ zip_path.unlink()+ print(f"[coco] ok -> {root / marker}")+ print(f"\n[coco] ready. Use --coco-root {root}")+++def check_3dpw(sequence_dir: Path | None, image_root: Path | None) -> bool:+ ok = True+ if sequence_dir is None or not sequence_dir.is_dir():+ print(f"[3dpw] MISSING sequenceFiles: {sequence_dir}")+ ok = False+ else:+ for split, expected in (("test", 24), ("validation", 12), ("train", 24)):+ pkls = list((sequence_dir / split).glob("*.pkl"))+ flag = "ok " if pkls else "MISSING"+ print(f"[3dpw] {flag} sequenceFiles/{split:<11s} "+ f"{len(pkls)} pkl (official: {expected})")+ if split == "test" and not pkls:+ ok = False+ if image_root is None or not image_root.is_dir():+ print(f"[3dpw] MISSING imageFiles: {image_root}")+ ok = False+ else:+ seqs = [d for d in image_root.iterdir() if d.is_dir()]+ print(f"[3dpw] ok imageFiles {len(seqs)} sequences "+ f"(official: 60)")+ if not ok:+ print(THREEDPW_HELP)+ else:+ print("\n[3dpw] ready.")+ return ok+++def main():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--coco", action="store_true", help="download COCO val2017")+ p.add_argument("--coco-root", default="benchmark/data/coco")+ p.add_argument("--check-3dpw", action="store_true")+ p.add_argument("--sequence-dir", default=None)+ p.add_argument("--image-root", default=None)+ a = p.parse_args()++ if a.coco:+ download_coco(Path(a.coco_root))+ if a.check_3dpw:+ check_3dpw(+ Path(a.sequence_dir) if a.sequence_dir else None,+ Path(a.image_root) if a.image_root else None,+ )+ if not (a.coco or a.check_3dpw):+ p.print_help()+++if __name__ == "__main__":+ main()diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.pynew file mode 100644index 0000000..33f770f--- /dev/null+++ b/benchmark/eval_3dpw.py@@ -0,0 +1,234 @@+#!/usr/bin/env python3+"""3DPW test-set evaluation for InstantHMR (MPJPE / PA-MPJPE / PCK / AUC).++The GT comes straight from the 3DPW pickles' ``jointPositions`` field, so no+SMPL model download is needed. Predicted MHR70 joints are mapped onto the+SMPL joint convention by name; see ``benchlib/joints.py`` for the caveat about+``neck`` and ``head``, which are the only two joints where the two rigs disagree+on the underlying anatomy.++Usage:+ python benchmark/eval_3dpw.py \+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --image-root /path/to/3DPW/imageFiles++Add ``--stride 10`` for a fast smoke run over 1/10th of the frames.+"""++from __future__ import annotations++import argparse+import json+import sys+from pathlib import Path++import numpy as np++sys.path.insert(0, str(Path(__file__).resolve().parent))++from benchlib import joints as J+from benchlib import metrics as M+from benchlib import threedpw as P+from benchlib.runner import Sample, build_model, run_samples, stack_joints++MM = 1000.0 # metres -> millimetres+++def parse_args():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--onnx", required=True, help="path to the exported graph")+ p.add_argument("--sequence-dir", required=True, help="3DPW sequenceFiles/")+ p.add_argument("--image-root", required=True, help="3DPW imageFiles/")+ p.add_argument("--split", default="test",+ choices=["test", "train", "validation"])+ p.add_argument("--stride", type=int, default=1,+ help="evaluate every Nth frame (1 = full protocol)")+ p.add_argument("--bbox-scale", type=float, default=1.2,+ help="padding on the projected-GT-joint person box")+ p.add_argument("--device", default="cuda")+ p.add_argument("--mhr-script", default=None)+ p.add_argument("--num-workers", type=int, default=6)+ p.add_argument("--limit", type=int, default=0, help="cap sample count (debug)")+ p.add_argument("--out", default="benchmark/results/3dpw.json")+ p.add_argument("--adapter", default=None,+ help="npz with a fitted MHR70->J14 joint regressor to also report")+ p.add_argument("--exclude-seen", nargs="*", default=None,+ metavar="DIR",+ help="training annotation dirs; frames found there are "+ "reported separately as a held-out subset")+ p.add_argument("--fit-adapter", default=None,+ help="fit that regressor on THIS split and write it here "+ "(run on train/validation, never on test)")+ return p.parse_args()+++def evaluate(pred_mhr: np.ndarray, gt_smpl: np.ndarray, set_name: str) -> dict:+ """Metrics for one joint set. Inputs are (N, 70, 3) and (N, 24, 3) in mm."""+ spec = J.JOINT_SETS[set_name]+ pred = pred_mhr[:, spec["mhr"], :]+ gt = gt_smpl[:, spec["smpl"], :]++ root_p, root_g = J.pelvis(pred), J.pelvis(gt)+ e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))+ e_pa = M.pa_mpjpe(pred, gt)++ return dict(+ joint_set=set_name,+ num_joints=len(spec["names"]),+ num_samples=int(pred.shape[0]),+ MPJPE_mm=float(e_mpjpe.mean()),+ PA_MPJPE_mm=float(e_pa.mean()),+ PCK3D_50mm=M.pck3d(e_pa, 50.0),+ PCK3D_100mm=M.pck3d(e_pa, 100.0),+ AUC_0_150mm=M.auc3d(e_pa),+ per_joint_PA_MPJPE_mm={+ n: float(v) for n, v in zip(spec["names"], e_pa.mean(axis=0))+ },+ )+++def fit_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray) -> np.ndarray:+ """Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.++ Fitted on centred poses and used without a bias, so the mapping stays+ equivariant to rotation and translation — it only removes the systematic+ difference in where the two rigs place each landmark.+ """+ gt = gt_smpl[:, J.J14_FROM_SMPL24, :]+ X = pred_mhr - pred_mhr.mean(axis=1, keepdims=True) # (N, 70, 3)+ Y = gt - gt.mean(axis=1, keepdims=True) # (N, 14, 3)+ # Flatten samples and coordinates into the row dimension.+ A = X.transpose(0, 2, 1).reshape(-1, X.shape[1]) # (N*3, 70)+ B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1]) # (N*3, 14)+ W, *_ = np.linalg.lstsq(A, B, rcond=None)+ return W.T # (14, 70)+++def evaluate_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray,+ W: np.ndarray) -> dict:+ pred = np.einsum("jk,nkc->njc", W, pred_mhr)+ gt = gt_smpl[:, J.J14_FROM_SMPL24, :]+ root_p, root_g = J.pelvis(pred), J.pelvis(gt)+ e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))+ e_pa = M.pa_mpjpe(pred, gt)+ return dict(+ joint_set="J14+adapter", num_joints=14,+ num_samples=int(pred.shape[0]),+ MPJPE_mm=float(e_mpjpe.mean()),+ PA_MPJPE_mm=float(e_pa.mean()),+ PCK3D_50mm=M.pck3d(e_pa, 50.0),+ PCK3D_100mm=M.pck3d(e_pa, 100.0),+ AUC_0_150mm=M.auc3d(e_pa),+ )+++def main():+ args = parse_args()++ print(f"[3DPW] loading GT from {args.sequence_dir} ({args.split} split)")+ samples, gt_joints = P.build_samples(+ args.sequence_dir, args.image_root, split=args.split,+ stride=args.stride, bbox_scale=args.bbox_scale,+ )+ if args.limit:+ samples, gt_joints = samples[:args.limit], gt_joints[:args.limit]+ seqs = sorted({s["sequence"] for s in samples})+ print(f"[3DPW] {len(samples)} person-frames across {len(seqs)} sequences")++ model = build_model(args.onnx, args.device, args.mhr_script)+ preds, ok = run_samples(+ model,+ [Sample(s["image_path"], s["bbox"], s) for s in samples],+ num_workers=args.num_workers, desc="3DPW",+ )+ if not ok.all():+ print(f"[3DPW] WARNING: {int((~ok).sum())} samples had unreadable images")++ # joints_3d_local is the body-centred rig pose; root alignment and+ # Procrustes both make the camera translation irrelevant.+ pred_mhr = stack_joints(preds, ok, "joints_3d_local") * MM+ gt_smpl = gt_joints[ok] * MM++ kept_meta = [s for s, k in zip(samples, ok) if k]+ W_adapt = np.load(args.adapter)["W"] if args.adapter else None++ def _rows(pm, gm):+ rows = [evaluate(pm, gm, s) for s in ("J14", "J12")]+ if W_adapt is not None:+ rows.append(evaluate_adapter(pm, gm, W_adapt))+ return rows+ report = dict(+ model=str(args.onnx), dataset="3DPW", split=args.split,+ stride=args.stride, bbox_scale=args.bbox_scale,+ num_sequences=len(seqs), sequences=seqs,+ num_samples=int(pred_mhr.shape[0]),+ results=_rows(pred_mhr, gt_smpl),+ )++ # Frames present in the training data are leaked; score them separately so+ # the honest number is visible next to the optimistic one.+ if args.exclude_seen:+ seen = P.seen_frames(args.exclude_seen)+ is_seen = np.array([(m["sequence"], m["frame"]) in seen for m in kept_meta])+ report["contamination"] = dict(+ training_dirs=list(args.exclude_seen),+ seen_frames=int(is_seen.sum()),+ unseen_frames=int((~is_seen).sum()),+ seen_fraction=float(is_seen.mean()),+ seen_sequences=sorted({m["sequence"] for m, f in zip(kept_meta, is_seen) if f}),+ )+ if (~is_seen).any():+ report["results_unseen_frames"] = _rows(pred_mhr[~is_seen], gt_smpl[~is_seen])+ if is_seen.any():+ report["results_seen_frames"] = _rows(pred_mhr[is_seen], gt_smpl[is_seen])++ if args.fit_adapter:+ W = fit_adapter(pred_mhr, gt_smpl)+ Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)+ np.savez(args.fit_adapter, W=W, split=args.split)+ print(f"[3DPW] fitted joint adapter on '{args.split}' -> {args.fit_adapter}")++ # Per-sequence PA-MPJPE on the headline joint set.+ spec = J.JOINT_SETS["J14"]+ e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :], gt_smpl[:, spec["smpl"], :]).mean(axis=1)+ kept = [s for s, k in zip(samples, ok) if k]+ per_seq = {}+ for name in seqs:+ m = np.array([s["sequence"] == name for s in kept])+ if m.any():+ per_seq[name] = dict(n=int(m.sum()), PA_MPJPE_mm=float(e[m].mean()))+ report["per_sequence_J14"] = per_seq++ out = Path(args.out)+ out.parent.mkdir(parents=True, exist_ok=True)+ out.write_text(json.dumps(report, indent=2))++ def _table(title, rows):+ print(f"\n{title}")+ print(f" {'joint set':<14s} {'N':>7s} {'MPJPE':>10s} {'PA-MPJPE':>10s}")+ for r in rows:+ print(f" {r['joint_set']:<14s} {r['num_samples']:7d} "+ f"{r['MPJPE_mm']:9.1f} {r['PA_MPJPE_mm']:9.1f}")++ print("\n" + "=" * 60)+ print(f"3DPW {args.split} | {report['num_samples']} person-frames "+ f"| {len(seqs)} sequences (mm)")+ print("=" * 60)+ _table("ALL frames:", report["results"])+ if "results_unseen_frames" in report:+ c = report["contamination"]+ _table(f"Frames NOT in training data "+ f"({c['unseen_frames']}/{report['num_samples']}, "+ f"{100*(1-c['seen_fraction']):.1f}%):",+ report["results_unseen_frames"])+ if "results_seen_frames" in report:+ _table("Frames SEEN in training (leaked — for reference only):",+ report["results_seen_frames"])+ print("=" * 60)+ print(f"written to {out}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/eval_coco.py b/benchmark/eval_coco.pynew file mode 100644index 0000000..1111763--- /dev/null+++ b/benchmark/eval_coco.py@@ -0,0 +1,191 @@+#!/usr/bin/env python3+"""COCO val2017 keypoint evaluation for InstantHMR (OKS AP/AR, PCK, NME).++This is a *top-down with ground-truth boxes* protocol: one crop per annotated+person, so the numbers measure pose accuracy alone and are not comparable to+detector-in-the-loop AP from the COCO leaderboard. It is the standard ablation+protocol for single-person pose regressors, and it is the honest one for a+model that never sees a detector at eval time.++InstantHMR regresses joint *positions* with no per-keypoint confidence, so+every predicted keypoint is emitted with score 1.0. That is fine for OKS AP+here (exactly one prediction per GT person) but it means AP cannot be improved+by confidence ranking the way heatmap methods do — read AP alongside the mean+OKS and PCK numbers, which are confidence-free.++Usage:+ python benchmark/eval_coco.py \+ --onnx models/instanthmr_mhr_only_ckpt90.onnx \+ --coco-root benchmark/data/coco+"""++from __future__ import annotations++import argparse+import contextlib+import io+import json+import sys+from pathlib import Path++import numpy as np++sys.path.insert(0, str(Path(__file__).resolve().parent))++from benchlib import joints as J+from benchlib import metrics as M+from benchlib.runner import Sample, build_model, run_samples++# Official COCO per-keypoint OKS tolerances.+OKS_SIGMAS = np.array(+ [.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62,+ 1.07, 1.07, .87, .87, .89, .89]+) / 10.0+++def parse_args():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--onnx", required=True)+ p.add_argument("--coco-root", default="benchmark/data/coco",+ help="dir holding val2017/ and annotations/")+ p.add_argument("--ann", default=None,+ help="override path to person_keypoints_val2017.json")+ p.add_argument("--images", default=None, help="override path to val2017/")+ p.add_argument("--min-keypoints", type=int, default=1,+ help="skip GT persons with fewer labelled keypoints")+ p.add_argument("--device", default="cuda")+ p.add_argument("--mhr-script", default=None)+ p.add_argument("--num-workers", type=int, default=6)+ p.add_argument("--limit", type=int, default=0)+ p.add_argument("--out", default="benchmark/results/coco.json")+ return p.parse_args()+++def oks(pred: np.ndarray, gt: np.ndarray, vis: np.ndarray,+ area: np.ndarray) -> np.ndarray:+ """Per-sample OKS. pred/gt are (N, 17, 2), vis (N, 17), area (N,)."""+ d2 = ((pred - gt) ** 2).sum(axis=-1)+ var = (2 * OKS_SIGMAS) ** 2+ e = d2 / var[None, :] / (area[:, None] + np.spacing(1)) / 2.0+ m = vis > 0+ out = np.full(len(pred), np.nan)+ has = m.any(axis=1)+ out[has] = np.array([np.exp(-e[i][m[i]]).mean() for i in np.where(has)[0]])+ return out+++def main():+ args = parse_args()+ from pycocotools.coco import COCO+ from pycocotools.cocoeval import COCOeval++ root = Path(args.coco_root)+ ann_file = Path(args.ann) if args.ann else root / "annotations" / "person_keypoints_val2017.json"+ img_dir = Path(args.images) if args.images else root / "val2017"+ for pth in (ann_file, img_dir):+ if not pth.exists():+ raise FileNotFoundError(+ f"{pth} not found — run: python benchmark/download.py --coco"+ )++ print(f"[COCO] loading {ann_file}")+ with contextlib.redirect_stdout(io.StringIO()):+ coco = COCO(str(ann_file))++ samples: list[Sample] = []+ for ann_id in sorted(coco.getAnnIds(catIds=[1], iscrowd=False)):+ a = coco.anns[ann_id]+ if a["num_keypoints"] < args.min_keypoints or a["area"] <= 0:+ continue+ x, y, w, h = a["bbox"]+ if w <= 1 or h <= 1:+ continue+ info = coco.imgs[a["image_id"]]+ samples.append(Sample(+ image_path=str(img_dir / info["file_name"]),+ bbox=np.array([x, y, x + w, y + h], dtype=np.float32),+ meta=dict(ann_id=ann_id, image_id=a["image_id"],+ keypoints=np.array(a["keypoints"], dtype=np.float64).reshape(17, 3),+ area=float(a["area"]), bbox_wh=(float(w), float(h))),+ ))+ # Group people from the same image together so each JPEG is decoded once.+ samples.sort(key=lambda s: s.image_path)+ if args.limit:+ samples = samples[:args.limit]+ print(f"[COCO] {len(samples)} annotated persons")++ model = build_model(args.onnx, args.device, args.mhr_script)+ preds, ok = run_samples(model, samples, num_workers=args.num_workers, desc="COCO")+ if not ok.all():+ print(f"[COCO] WARNING: {int((~ok).sum())} samples had unreadable images")++ kept = [s for s, k in zip(samples, ok) if k]+ pred_kp = np.stack([+ p.joints_2d[J.COCO17_FROM_MHR70] for p, k in zip(preds, ok) if k+ ]).astype(np.float64)+ gt_kp = np.stack([s.meta["keypoints"] for s in kept])+ gt_xy, gt_vis = gt_kp[:, :, :2], gt_kp[:, :, 2]+ areas = np.array([s.meta["area"] for s in kept])+ diag = np.array([np.hypot(*s.meta["bbox_wh"]) for s in kept])++ # --- confidence-free metrics -----------------------------------------+ valid = gt_vis > 0+ sample_oks = oks(pred_kp, gt_xy, gt_vis, areas)+ summary = {+ "num_samples": len(kept),+ "mean_OKS": float(np.nanmean(sample_oks)),+ "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.1),+ "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.2),+ "NME_bboxdiag": M.nme2d(pred_kp, gt_xy, valid, diag),+ }+ per_kp = {}+ for i, name in enumerate(J.COCO17_NAMES):+ v = valid[:, i]+ per_kp[name] = {+ "n": int(v.sum()),+ "[email protected]": float("nan") if not v.any() else+ float((np.linalg.norm(pred_kp[v, i] - gt_xy[v, i], axis=-1)+ / diag[v] < 0.1).mean()),+ }++ # --- OKS AP through the official evaluator ---------------------------+ results = [dict(+ image_id=int(s.meta["image_id"]), category_id=1, score=1.0,+ keypoints=np.concatenate(+ [pred_kp[i], np.ones((17, 1))], axis=1).reshape(-1).tolist(),+ ) for i, s in enumerate(kept)]++ with contextlib.redirect_stdout(io.StringIO()):+ dt = coco.loadRes(results)+ ev = COCOeval(coco, dt, "keypoints")+ ev.params.imgIds = sorted({r["image_id"] for r in results})+ ev.evaluate(); ev.accumulate()+ ev.summarize()+ ap_keys = ["AP", "AP50", "AP75", "AP_medium", "AP_large",+ "AR", "AR50", "AR75", "AR_medium", "AR_large"]+ summary["oks_ap"] = {k: float(v) for k, v in zip(ap_keys, ev.stats)}++ report = dict(model=str(args.onnx), dataset="COCO val2017",+ protocol="top-down, ground-truth boxes, score=1.0",+ summary=summary, per_keypoint=per_kp)+ out = Path(args.out)+ out.parent.mkdir(parents=True, exist_ok=True)+ out.write_text(json.dumps(report, indent=2))++ print("\n" + "=" * 68)+ print(f"COCO val2017 (GT boxes) | {summary['num_samples']} persons")+ print("=" * 68)+ print(f" OKS AP {summary['oks_ap']['AP']*100:6.2f} "+ f"AP50 {summary['oks_ap']['AP50']*100:6.2f} "+ f"AP75 {summary['oks_ap']['AP75']*100:6.2f}")+ print(f" mean OKS {summary['mean_OKS']:6.4f}")+ print(f" [email protected] {summary['[email protected]_bboxdiag']*100:6.2f} % "+ f"[email protected] {summary['[email protected]_bboxdiag']*100:6.2f} %")+ print(f" NME {summary['NME_bboxdiag']:6.4f} (bbox-diagonal units)")+ print("=" * 68)+ print(f"written to {out}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/results/3dpw_test.json b/benchmark/results/3dpw_test.jsonnew file mode 100644index 0000000..8b2912f--- /dev/null+++ b/benchmark/results/3dpw_test.json@@ -0,0 +1,354 @@+{+ "model": "models/instanthmr_mhr_only_ckpt90.onnx",+ "dataset": "3DPW",+ "split": "test",+ "stride": 1,+ "bbox_scale": 1.2,+ "num_sequences": 24,+ "sequences": [+ "downtown_arguing_00",+ "downtown_bar_00",+ "downtown_bus_00",+ "downtown_cafe_00",+ "downtown_car_00",+ "downtown_crossStreets_00",+ "downtown_downstairs_00",+ "downtown_enterShop_00",+ "downtown_rampAndStairs_00",+ "downtown_runForBus_00",+ "downtown_runForBus_01",+ "downtown_sitOnStairs_00",+ "downtown_stairs_00",+ "downtown_upstairs_00",+ "downtown_walkBridge_01",+ "downtown_walkUphill_00",+ "downtown_walking_00",+ "downtown_warmWelcome_00",+ "downtown_weeklyMarket_00",+ "downtown_windowShopping_00",+ "flat_guitar_01",+ "flat_packBags_00",+ "office_phoneCall_00",+ "outdoors_fencing_01"+ ],+ "num_samples": 35463,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 77.25832087421568,+ "PA_MPJPE_mm": 54.8939122400647,+ "PCK3D_50mm": 0.5790421405005619,+ "PCK3D_100mm": 0.8924653864591264,+ "AUC_0_150mm": 0.6371804272928843,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 58.99963246267586,+ "right_knee": 42.38728858483827,+ "right_hip": 50.39103164429582,+ "left_hip": 45.39265522544049,+ "left_knee": 46.095110047060146,+ "left_ankle": 62.30361585517269,+ "right_wrist": 63.81340497982142,+ "right_elbow": 41.90725242230058,+ "right_shoulder": 43.96807667630508,+ "left_shoulder": 50.08607425327331,+ "left_elbow": 48.06366572018603,+ "left_wrist": 66.96362486775135,+ "neck": 32.920260588579204,+ "head": 115.22307803320544+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 71.98718459643236,+ "PA_MPJPE_mm": 48.292409136297024,+ "PCK3D_50mm": 0.6549784282209626,+ "PCK3D_100mm": 0.9406353100414516,+ "AUC_0_150mm": 0.6786479562676105,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 56.99815377789105,+ "right_knee": 43.630305291279456,+ "right_hip": 46.529599094848884,+ "left_hip": 42.83447178924256,+ "left_knee": 47.52243530675108,+ "left_ankle": 57.855173112259216,+ "right_wrist": 62.29407156666568,+ "right_elbow": 36.678818675428985,+ "right_shoulder": 38.21018561144652,+ "left_shoulder": 40.09340440903251,+ "left_elbow": 40.734850438026946,+ "left_wrist": 66.12744056269153+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 66.85722222445656,+ "PA_MPJPE_mm": 44.083164610596384,+ "PCK3D_50mm": 0.7056892294181863,+ "PCK3D_100mm": 0.9425034543044863,+ "AUC_0_150mm": 0.705719636913712+ }+ ],+ "contamination": {+ "training_dirs": [+ "data/sam3d_distill_mix/annotations",+ "data/sam3d_gt_3dpw/annotations"+ ],+ "seen_frames": 10353,+ "unseen_frames": 25110,+ "seen_fraction": 0.29193807630488117,+ "seen_sequences": [+ "downtown_arguing_00",+ "downtown_bar_00",+ "downtown_bus_00",+ "downtown_cafe_00",+ "downtown_car_00",+ "downtown_crossStreets_00",+ "downtown_downstairs_00",+ "downtown_enterShop_00",+ "downtown_rampAndStairs_00",+ "downtown_runForBus_00",+ "downtown_runForBus_01",+ "downtown_sitOnStairs_00",+ "downtown_stairs_00",+ "downtown_upstairs_00",+ "downtown_walkBridge_01",+ "downtown_walkUphill_00",+ "downtown_walking_00",+ "downtown_warmWelcome_00",+ "downtown_weeklyMarket_00",+ "downtown_windowShopping_00",+ "flat_guitar_01",+ "flat_packBags_00",+ "office_phoneCall_00",+ "outdoors_fencing_01"+ ]+ },+ "results_unseen_frames": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 25110,+ "MPJPE_mm": 80.50931989672257,+ "PA_MPJPE_mm": 57.05366140421865,+ "PCK3D_50mm": 0.5548216419184161,+ "PCK3D_100mm": 0.881657848324515,+ "AUC_0_150mm": 0.6249301231264464,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 61.47931140891157,+ "right_knee": 43.66784597368266,+ "right_hip": 51.590643380821774,+ "left_hip": 45.99633112852736,+ "left_knee": 47.61141586722965,+ "left_ankle": 65.61625292500608,+ "right_wrist": 69.17219482531468,+ "right_elbow": 43.22191976400349,+ "right_shoulder": 44.582150790065214,+ "left_shoulder": 50.70150039417648,+ "left_elbow": 51.159805717594764,+ "left_wrist": 72.72767706391078,+ "neck": 34.115691861368255,+ "head": 117.10851855844838+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 25110,+ "MPJPE_mm": 75.3534859518377,+ "PA_MPJPE_mm": 50.59458176134388,+ "PCK3D_50mm": 0.6247710075667064,+ "PCK3D_100mm": 0.9298320722155847,+ "AUC_0_150mm": 0.6652752619656309,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 59.2944636488828,+ "right_knee": 44.98211013553142,+ "right_hip": 47.548593595632475,+ "left_hip": 43.394665628452856,+ "left_knee": 49.07030713900323,+ "left_ankle": 60.92885186640498,+ "right_wrist": 67.01462392288674,+ "right_elbow": 38.323823357280524,+ "right_shoulder": 39.95733150137268,+ "left_shoulder": 41.6934507272976,+ "left_elbow": 43.797015072444125,+ "left_wrist": 71.12974454093712+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 25110,+ "MPJPE_mm": 70.43668183981028,+ "PA_MPJPE_mm": 46.514347723018886,+ "PCK3D_50mm": 0.6766484610570632,+ "PCK3D_100mm": 0.9324372759856631,+ "AUC_0_150mm": 0.6915008065892562+ }+ ],+ "results_seen_frames": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 10353,+ "MPJPE_mm": 69.3734000343482,+ "PA_MPJPE_mm": 49.655691288465555,+ "PCK3D_50mm": 0.6377861489423355,+ "PCK3D_100mm": 0.9186778159539678,+ "AUC_0_150mm": 0.6668921183601361,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 52.98545895355014,+ "right_knee": 39.2814452511299,+ "right_hip": 47.48151259627433,+ "left_hip": 43.92850938109478,+ "left_knee": 42.41748625256036,+ "left_ankle": 54.26919908481471,+ "right_wrist": 50.81628211491889,+ "right_elbow": 38.71867935650709,+ "right_shoulder": 42.478711178718214,+ "left_shoulder": 48.593429570758246,+ "left_elbow": 40.554337473790454,+ "left_wrist": 52.98358520334844,+ "neck": 30.020880770194864,+ "head": 110.65016085085732+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 10353,+ "MPJPE_mm": 63.82261132933802,+ "PA_MPJPE_mm": 42.70875660901737,+ "PCK3D_50mm": 0.7282430213464697,+ "PCK3D_100mm": 0.9668373096364983,+ "AUC_0_150mm": 0.7110818743515204,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 51.42872068017997,+ "right_knee": 40.351659522983596,+ "right_hip": 44.05814619089098,+ "left_hip": 41.47578664459172,+ "left_knee": 43.768251813285275,+ "left_ankle": 50.400322004696164,+ "right_wrist": 50.84491966241465,+ "right_elbow": 32.689050727849335,+ "right_shoulder": 33.97268601750797,+ "left_shoulder": 36.21267775476452,+ "left_elbow": 33.30792549161378,+ "left_wrist": 53.99493279743053+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 10353,+ "MPJPE_mm": 58.175658335580685,+ "PA_MPJPE_mm": 38.18661211828219,+ "PCK3D_50mm": 0.7761242428005685,+ "PCK3D_100mm": 0.9669178016034,+ "AUC_0_150mm": 0.7402057597232443+ }+ ],+ "per_sequence_J14": {+ "downtown_arguing_00": {+ "n": 1793,+ "PA_MPJPE_mm": 53.86209217665553+ },+ "downtown_bar_00": {+ "n": 2458,+ "PA_MPJPE_mm": 67.11104059483931+ },+ "downtown_bus_00": {+ "n": 2534,+ "PA_MPJPE_mm": 57.3890391264324+ },+ "downtown_cafe_00": {+ "n": 2076,+ "PA_MPJPE_mm": 52.69031252948052+ },+ "downtown_car_00": {+ "n": 1546,+ "PA_MPJPE_mm": 60.43672446675064+ },+ "downtown_crossStreets_00": {+ "n": 1042,+ "PA_MPJPE_mm": 55.938241587076654+ },+ "downtown_downstairs_00": {+ "n": 638,+ "PA_MPJPE_mm": 47.09329939736882+ },+ "downtown_enterShop_00": {+ "n": 1385,+ "PA_MPJPE_mm": 46.63087615806622+ },+ "downtown_rampAndStairs_00": {+ "n": 1954,+ "PA_MPJPE_mm": 56.34401250060689+ },+ "downtown_runForBus_00": {+ "n": 1302,+ "PA_MPJPE_mm": 52.232966837233924+ },+ "downtown_runForBus_01": {+ "n": 1437,+ "PA_MPJPE_mm": 66.2344342328675+ },+ "downtown_sitOnStairs_00": {+ "n": 2631,+ "PA_MPJPE_mm": 52.626225175849925+ },+ "downtown_stairs_00": {+ "n": 1197,+ "PA_MPJPE_mm": 63.31683556710035+ },+ "downtown_upstairs_00": {+ "n": 825,+ "PA_MPJPE_mm": 47.17526695041395+ },+ "downtown_walkBridge_01": {+ "n": 1182,+ "PA_MPJPE_mm": 44.298063167659926+ },+ "downtown_walkUphill_00": {+ "n": 387,+ "PA_MPJPE_mm": 44.89834428674509+ },+ "downtown_walking_00": {+ "n": 2594,+ "PA_MPJPE_mm": 52.14282318214196+ },+ "downtown_warmWelcome_00": {+ "n": 1122,+ "PA_MPJPE_mm": 64.06683409393634+ },+ "downtown_weeklyMarket_00": {+ "n": 1055,+ "PA_MPJPE_mm": 47.945786223105564+ },+ "downtown_windowShopping_00": {+ "n": 1821,+ "PA_MPJPE_mm": 44.496857077869684+ },+ "flat_guitar_01": {+ "n": 748,+ "PA_MPJPE_mm": 60.36081947705571+ },+ "flat_packBags_00": {+ "n": 1273,+ "PA_MPJPE_mm": 64.67982670780314+ },+ "office_phoneCall_00": {+ "n": 1524,+ "PA_MPJPE_mm": 45.326043123588434+ },+ "outdoors_fencing_01": {+ "n": 939,+ "PA_MPJPE_mm": 55.91377958820054+ }+ }+}\ No newline at end of filediff --git a/benchmark/results/3dpw_test_adapter.json b/benchmark/results/3dpw_test_adapter.jsonnew file mode 100644index 0000000..8f64214--- /dev/null+++ b/benchmark/results/3dpw_test_adapter.json@@ -0,0 +1,334 @@+{+ "model": "models/instanthmr_mhr_only_ckpt90.onnx",+ "dataset": "3DPW",+ "split": "test",+ "stride": 1,+ "bbox_scale": 1.2,+ "num_sequences": 24,+ "sequences": [+ "downtown_arguing_00",+ "downtown_bar_00",+ "downtown_bus_00",+ "downtown_cafe_00",+ "downtown_car_00",+ "downtown_crossStreets_00",+ "downtown_downstairs_00",+ "downtown_enterShop_00",+ "downtown_rampAndStairs_00",+ "downtown_runForBus_00",+ "downtown_runForBus_01",+ "downtown_sitOnStairs_00",+ "downtown_stairs_00",+ "downtown_upstairs_00",+ "downtown_walkBridge_01",+ "downtown_walkUphill_00",+ "downtown_walking_00",+ "downtown_warmWelcome_00",+ "downtown_weeklyMarket_00",+ "downtown_windowShopping_00",+ "flat_guitar_01",+ "flat_packBags_00",+ "office_phoneCall_00",+ "outdoors_fencing_01"+ ],+ "num_samples": 35463,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 77.25832087421568,+ "PA_MPJPE_mm": 54.8939122400647,+ "PCK3D_50mm": 0.5790421405005619,+ "PCK3D_100mm": 0.8924653864591264,+ "AUC_0_150mm": 0.6371804272928843,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 58.99963246267586,+ "right_knee": 42.38728858483827,+ "right_hip": 50.39103164429582,+ "left_hip": 45.39265522544049,+ "left_knee": 46.095110047060146,+ "left_ankle": 62.30361585517269,+ "right_wrist": 63.81340497982142,+ "right_elbow": 41.90725242230058,+ "right_shoulder": 43.96807667630508,+ "left_shoulder": 50.08607425327331,+ "left_elbow": 48.06366572018603,+ "left_wrist": 66.96362486775135,+ "neck": 32.920260588579204,+ "head": 115.22307803320544+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 71.98718459643236,+ "PA_MPJPE_mm": 48.292409136297024,+ "PCK3D_50mm": 0.6549784282209626,+ "PCK3D_100mm": 0.9406353100414516,+ "AUC_0_150mm": 0.6786479562676105,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 56.99815377789105,+ "right_knee": 43.630305291279456,+ "right_hip": 46.529599094848884,+ "left_hip": 42.83447178924256,+ "left_knee": 47.52243530675108,+ "left_ankle": 57.855173112259216,+ "right_wrist": 62.29407156666568,+ "right_elbow": 36.678818675428985,+ "right_shoulder": 38.21018561144652,+ "left_shoulder": 40.09340440903251,+ "left_elbow": 40.734850438026946,+ "left_wrist": 66.12744056269153+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 66.85722222445656,+ "PA_MPJPE_mm": 44.083164610596384,+ "PCK3D_50mm": 0.7056892294181863,+ "PCK3D_100mm": 0.9425034543044863,+ "AUC_0_150mm": 0.705719636913712+ }+ ],+ "contamination": {+ "training_dirs": [+ "data/sam3d_distill_mix/annotations",+ "data/sam3d_gt_3dpw/annotations"+ ],+ "seen_frames": 10353,+ "unseen_frames": 25110,+ "seen_fraction": 0.29193807630488117,+ "seen_sequences": [+ "downtown_arguing_00",+ "downtown_bar_00",+ "downtown_bus_00",+ "downtown_cafe_00",+ "downtown_car_00",+ "downtown_crossStreets_00",+ "downtown_downstairs_00",+ "downtown_enterShop_00",+ "downtown_rampAndStairs_00",+ "downtown_runForBus_00",+ "downtown_runForBus_01",+ "downtown_sitOnStairs_00",+ "downtown_stairs_00",+ "downtown_upstairs_00",+ "downtown_walkBridge_01",+ "downtown_walkUphill_00",+ "downtown_walking_00",+ "downtown_warmWelcome_00",+ "downtown_weeklyMarket_00",+ "downtown_windowShopping_00",+ "flat_guitar_01",+ "flat_packBags_00",+ "office_phoneCall_00",+ "outdoors_fencing_01"+ ]+ },+ "results_unseen_frames": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 25110,+ "MPJPE_mm": 80.50931989672257,+ "PA_MPJPE_mm": 57.05366140421865,+ "PCK3D_50mm": 0.5548216419184161,+ "PCK3D_100mm": 0.881657848324515,+ "AUC_0_150mm": 0.6249301231264464,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 61.47931140891157,+ "right_knee": 43.66784597368266,+ "right_hip": 51.590643380821774,+ "left_hip": 45.99633112852736,+ "left_knee": 47.61141586722965,+ "left_ankle": 65.61625292500608,+ "right_wrist": 69.17219482531468,+ "right_elbow": 43.22191976400349,+ "right_shoulder": 44.582150790065214,+ "left_shoulder": 50.70150039417648,+ "left_elbow": 51.159805717594764,+ "left_wrist": 72.72767706391078,+ "neck": 34.115691861368255,+ "head": 117.10851855844838+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 25110,+ "MPJPE_mm": 75.3534859518377,+ "PA_MPJPE_mm": 50.59458176134388,+ "PCK3D_50mm": 0.6247710075667064,+ "PCK3D_100mm": 0.9298320722155847,+ "AUC_0_150mm": 0.6652752619656309,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 59.2944636488828,+ "right_knee": 44.98211013553142,+ "right_hip": 47.548593595632475,+ "left_hip": 43.394665628452856,+ "left_knee": 49.07030713900323,+ "left_ankle": 60.92885186640498,+ "right_wrist": 67.01462392288674,+ "right_elbow": 38.323823357280524,+ "right_shoulder": 39.95733150137268,+ "left_shoulder": 41.6934507272976,+ "left_elbow": 43.797015072444125,+ "left_wrist": 71.12974454093712+ }+ }+ ],+ "results_seen_frames": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 10353,+ "MPJPE_mm": 69.3734000343482,+ "PA_MPJPE_mm": 49.655691288465555,+ "PCK3D_50mm": 0.6377861489423355,+ "PCK3D_100mm": 0.9186778159539678,+ "AUC_0_150mm": 0.6668921183601361,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 52.98545895355014,+ "right_knee": 39.2814452511299,+ "right_hip": 47.48151259627433,+ "left_hip": 43.92850938109478,+ "left_knee": 42.41748625256036,+ "left_ankle": 54.26919908481471,+ "right_wrist": 50.81628211491889,+ "right_elbow": 38.71867935650709,+ "right_shoulder": 42.478711178718214,+ "left_shoulder": 48.593429570758246,+ "left_elbow": 40.554337473790454,+ "left_wrist": 52.98358520334844,+ "neck": 30.020880770194864,+ "head": 110.65016085085732+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 10353,+ "MPJPE_mm": 63.82261132933802,+ "PA_MPJPE_mm": 42.70875660901737,+ "PCK3D_50mm": 0.7282430213464697,+ "PCK3D_100mm": 0.9668373096364983,+ "AUC_0_150mm": 0.7110818743515204,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 51.42872068017997,+ "right_knee": 40.351659522983596,+ "right_hip": 44.05814619089098,+ "left_hip": 41.47578664459172,+ "left_knee": 43.768251813285275,+ "left_ankle": 50.400322004696164,+ "right_wrist": 50.84491966241465,+ "right_elbow": 32.689050727849335,+ "right_shoulder": 33.97268601750797,+ "left_shoulder": 36.21267775476452,+ "left_elbow": 33.30792549161378,+ "left_wrist": 53.99493279743053+ }+ }+ ],+ "per_sequence_J14": {+ "downtown_arguing_00": {+ "n": 1793,+ "PA_MPJPE_mm": 53.86209217665553+ },+ "downtown_bar_00": {+ "n": 2458,+ "PA_MPJPE_mm": 67.11104059483931+ },+ "downtown_bus_00": {+ "n": 2534,+ "PA_MPJPE_mm": 57.3890391264324+ },+ "downtown_cafe_00": {+ "n": 2076,+ "PA_MPJPE_mm": 52.69031252948052+ },+ "downtown_car_00": {+ "n": 1546,+ "PA_MPJPE_mm": 60.43672446675064+ },+ "downtown_crossStreets_00": {+ "n": 1042,+ "PA_MPJPE_mm": 55.938241587076654+ },+ "downtown_downstairs_00": {+ "n": 638,+ "PA_MPJPE_mm": 47.09329939736882+ },+ "downtown_enterShop_00": {+ "n": 1385,+ "PA_MPJPE_mm": 46.63087615806622+ },+ "downtown_rampAndStairs_00": {+ "n": 1954,+ "PA_MPJPE_mm": 56.34401250060689+ },+ "downtown_runForBus_00": {+ "n": 1302,+ "PA_MPJPE_mm": 52.232966837233924+ },+ "downtown_runForBus_01": {+ "n": 1437,+ "PA_MPJPE_mm": 66.2344342328675+ },+ "downtown_sitOnStairs_00": {+ "n": 2631,+ "PA_MPJPE_mm": 52.626225175849925+ },+ "downtown_stairs_00": {+ "n": 1197,+ "PA_MPJPE_mm": 63.31683556710035+ },+ "downtown_upstairs_00": {+ "n": 825,+ "PA_MPJPE_mm": 47.17526695041395+ },+ "downtown_walkBridge_01": {+ "n": 1182,+ "PA_MPJPE_mm": 44.298063167659926+ },+ "downtown_walkUphill_00": {+ "n": 387,+ "PA_MPJPE_mm": 44.89834428674509+ },+ "downtown_walking_00": {+ "n": 2594,+ "PA_MPJPE_mm": 52.14282318214196+ },+ "downtown_warmWelcome_00": {+ "n": 1122,+ "PA_MPJPE_mm": 64.06683409393634+ },+ "downtown_weeklyMarket_00": {+ "n": 1055,+ "PA_MPJPE_mm": 47.945786223105564+ },+ "downtown_windowShopping_00": {+ "n": 1821,+ "PA_MPJPE_mm": 44.496857077869684+ },+ "flat_guitar_01": {+ "n": 748,+ "PA_MPJPE_mm": 60.36081947705571+ },+ "flat_packBags_00": {+ "n": 1273,+ "PA_MPJPE_mm": 64.67982670780314+ },+ "office_phoneCall_00": {+ "n": 1524,+ "PA_MPJPE_mm": 45.326043123588434+ },+ "outdoors_fencing_01": {+ "n": 939,+ "PA_MPJPE_mm": 55.91377958820054+ }+ }+}\ No newline at end of filediff --git a/benchmark/results/adapter_j14.npz b/benchmark/results/adapter_j14.npznew file mode 100644index 0000000..82c864cBinary files /dev/null and b/benchmark/results/adapter_j14.npz differdiff --git a/benchmark/results/coco_val2017.json b/benchmark/results/coco_val2017.jsonnew file mode 100644index 0000000..6472b0d--- /dev/null+++ b/benchmark/results/coco_val2017.json@@ -0,0 +1,94 @@+{+ "model": "models/instanthmr_mhr_only_ckpt90.onnx",+ "dataset": "COCO val2017",+ "protocol": "top-down, ground-truth boxes, score=1.0",+ "summary": {+ "num_samples": 6352,+ "mean_OKS": 0.715087947740202,+ "[email protected]_bboxdiag": 0.8967235945173349,+ "[email protected]_bboxdiag": 0.9517847980649418,+ "NME_bboxdiag": 0.04808343393139624,+ "oks_ap": {+ "AP": 0.44039902588970287,+ "AP50": 0.697060689022555,+ "AP75": 0.47301161863312285,+ "AP_medium": 0.43565038188009775,+ "AP_large": 0.46188555986388313,+ "AR": 0.5974968513853904,+ "AR50": 0.8148614609571788,+ "AR75": 0.6572732997481109,+ "AR_medium": 0.5542201584266594,+ "AR_large": 0.6594574507617986+ }+ },+ "per_keypoint": {+ "nose": {+ "n": 4268,+ "[email protected]": 0.9395501405810684+ },+ "left_eye": {+ "n": 3703,+ "[email protected]": 0.9343775317310289+ },+ "right_eye": {+ "n": 3685,+ "[email protected]": 0.9383989145183175+ },+ "left_ear": {+ "n": 3185,+ "[email protected]": 0.930298273155416+ },+ "right_ear": {+ "n": 3256,+ "[email protected]": 0.9302825552825553+ },+ "left_shoulder": {+ "n": 5501,+ "[email protected]": 0.9014724595528085+ },+ "right_shoulder": {+ "n": 5495,+ "[email protected]": 0.902820746132848+ },+ "left_elbow": {+ "n": 4336,+ "[email protected]": 0.8996771217712177+ },+ "right_elbow": {+ "n": 4378,+ "[email protected]": 0.901553220648698+ },+ "left_wrist": {+ "n": 3945,+ "[email protected]": 0.8869455006337136+ },+ "right_wrist": {+ "n": 4012,+ "[email protected]": 0.8898305084745762+ },+ "left_hip": {+ "n": 4755,+ "[email protected]": 0.8441640378548896+ },+ "right_hip": {+ "n": 4784,+ "[email protected]": 0.844690635451505+ },+ "left_knee": {+ "n": 3513,+ "[email protected]": 0.8778821520068317+ },+ "right_knee": {+ "n": 3500,+ "[email protected]": 0.8805714285714286+ },+ "left_ankle": {+ "n": 2957,+ "[email protected]": 0.8782549881636794+ },+ "right_ankle": {+ "n": 2942,+ "[email protected]": 0.8820530251529571+ }+ }+}\ No newline at end of filediff --git a/demo.py b/demo.pyindex 3a511bb..82a9995 100755--- a/demo.py+++ b/demo.py@@ -8,21 +8,31 @@ drawn on top, and live latency plots for RF-DETR, InstantHMR, MHR meshrendering, and total pipeline cost.Optionally renders a full MHR body mesh per person by decoding the-``mhr_params`` and ``shape_params`` outputs through Meta's MHR model:-- python demo.py --image photo.jpg \\- --mhr-assets models/mhr_assets--See ``instanthmr/mhr_renderer.py`` for setup instructions (assets download,-MHR package installation).+``mhr_params`` and ``shape_params`` outputs through Meta's MHR model, either+from the TorchScript rig used by the distillation (``--mhr-model``) or from the+unpacked asset folder (``--mhr-assets``). See ``instanthmr/mhr_renderer.py``.++MHR-only checkpoints+--------------------+A model exported from ``train_distill_mhr_only.py`` has FOUR ONNX outputs — it+has no 3D coordinate head, so its 70 keypoints are derived from the MHR forward+pass, exactly like the SAM3D teacher does. Such a model therefore *requires* an+MHR rig; when none is requested explicitly the demo falls back to+``checkpoints/mhr_model.pt``.Usage------ # Single image (skeleton only)+ # Single image (skeleton only; needs a 5-output model)python demo.py --image path/to/photo.jpg- # Single image with MHR mesh- python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets+ # Single image with the MHR mesh, from the TorchScript rig+ python demo.py --image path/to/photo.jpg --mhr-model++ # MHR-only student: the rig is mandatory and auto-selected+ python demo.py --image path/to/photo.jpg --model models/instanthmr_mhr_only.onnx++ # Same, but decoding the mesh from the asset folder at a chosen LOD+ python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets --mhr-lod 3# Video filepython demo.py --video path/to/clip.mp4@@ -40,7 +50,8 @@ from pathlib import Pathimport cv2import numpy as np-from instanthmr import PosePipeline+from instanthmr import PosePipeline, onnx_output_names+from instanthmr.mhr_renderer import DEFAULT_MHR_SCRIPTfrom instanthmr.visualizer import RerunVisualizer@@ -104,6 +115,16 @@ def parse_args() -> argparse.Namespace:"--no-spawn", action="store_true",help="Don't spawn the Rerun viewer GUI (useful when only saving .rrd).",)+ p.add_argument(+ "--mhr-model", type=str, nargs="?", default=None, const=DEFAULT_MHR_SCRIPT,+ metavar="PT",+ help=(+ "Decode the body mesh with the TorchScript MHR rig used by the "+ "distillation (bare flag = checkpoints/mhr_model.pt). Needs only "+ "PyTorch, so it works regardless of the pymomentum build. Takes "+ "precedence over --mhr-assets."+ ),+ )p.add_argument("--mhr-assets", type=str, default=None, metavar="DIR",help=(@@ -116,15 +137,24 @@ def parse_args() -> argparse.Namespace:"--mhr-lod", type=int, default=3, metavar="N",choices=range(7),help=(- "MHR level-of-detail (0=73 639 verts … 6=595 verts). "- "Default: 3 (4 899 verts)."+ "MHR level-of-detail for --mhr-assets (0=73 639 verts … 6=595 "+ "verts). Default: 3 (4 899 verts). The TorchScript rig has a fixed "+ "LOD (18 439 verts) and ignores this."+ ),+ )+ p.add_argument(+ "--mesh-alpha", type=float, default=0.35, metavar="A",+ help=(+ "Body-mesh opacity in [0, 1] (default: 0.35). Applied as a Rerun "+ "albedo factor so the 70 keypoints stay visible inside the body; "+ "raise it towards 1.0 for a solid mesh."),)return p.parse_args()-def build_pipeline(args: argparse.Namespace) -> PosePipeline:+def resolve_model_path(args: argparse.Namespace) -> Path:model_path = Path(args.model)if not model_path.exists():sys.stderr.write(@@ -133,23 +163,70 @@ def build_pipeline(args: argparse.Namespace) -> PosePipeline:" --model /path/to/instanthmr.onnx\n\n")sys.exit(1)+ return model_path++def build_pipeline(args: argparse.Namespace, mhr=None) -> PosePipeline:return PosePipeline(- onnx_path=model_path,+ onnx_path=resolve_model_path(args),device=args.device,detector_variant=args.detector_variant,det_confidence=args.det_confidence,max_persons=args.max_persons,detector_stride=args.detector_stride,batch_persons=not args.no_batch_persons,+ mhr=mhr,)-def build_mhr_renderer(args: argparse.Namespace):- if args.mhr_assets is None:- return None+def _model_needs_mhr(model_path: Path) -> bool:+ """True when the graph has no ``joints_3d`` output (MHR-only student).- assets_path = Path(args.mhr_assets)+ Cheap protobuf peek, so the decision is made before any session or detector+ is loaded. An unreadable graph is treated as "does not need MHR" — the+ pipeline then raises a precise error of its own.+ """+ names = onnx_output_names(model_path)+ return names is not None and "joints_3d" not in names+++def build_mhr(args: argparse.Namespace, required: bool = False):+ """Build the MHR rig requested on the command line.++ ``--mhr-model`` (TorchScript) wins over ``--mhr-assets`` (pip package).+ When *required* — i.e. the ONNX has no 3D head — and neither flag was given,+ fall back to the TorchScript rig that shipped with the training code.+ """+ script_path = args.mhr_model+ assets_folder = args.mhr_assets++ if script_path is None and assets_folder is None:+ if not required:+ return None+ script_path = DEFAULT_MHR_SCRIPT+ print(+ f"[info] {Path(args.model).name} has no 'joints_3d' output "+ "(MHR-only student): its 70 keypoints are derived from the MHR "+ f"forward pass.\n Using {script_path}; override with "+ "--mhr-model / --mhr-assets."+ )++ if script_path is not None:+ path = Path(script_path)+ if not path.exists():+ sys.stderr.write(+ f"\n[error] MHR TorchScript rig not found: {path}\n"+ " This is the ~700 MB mhr_model.pt the distillation loads with\n"+ " torch.jit.load; see instanthmr_distill_train/checkpoints/README.md.\n"+ " Alternatively use --mhr-assets with the unpacked MHR asset folder.\n\n"+ )+ sys.exit(1)+ from instanthmr.mhr_renderer import MHRScriptRenderer++ print(f"Loading MHR body model (TorchScript) from {path} …")+ return MHRScriptRenderer(path, device=args.device)++ assets_path = Path(assets_folder)if not assets_path.exists():sys.stderr.write(f"\n[error] MHR assets folder not found: {assets_path}\n"@@ -172,6 +249,8 @@ def build_mhr_renderer(args: argparse.Namespace):" pip install pymomentum-gpu # NVIDIA GPU\n"" pip install pymomentum-cpu # macOS / CPU-only\n""\n"+ " Or skip it entirely and use --mhr-model, which needs only torch.\n"+ "\n"" See README.md §'MHR body mesh' for full setup instructions.\n\n")sys.exit(1)@@ -184,13 +263,24 @@ def build_mhr_renderer(args: argparse.Namespace):)-def build_visualizer(args: argparse.Namespace, mhr_renderer=None) -> RerunVisualizer:- return RerunVisualizer(+def build_all(args: argparse.Namespace):+ """Build the MHR rig, the pipeline and the visualizer, in that order.++ The rig comes first because an MHR-only graph cannot produce 3D keypoints+ without it — the pipeline takes it as a constructor argument rather than+ the visualizer alone.+ """+ model_path = resolve_model_path(args)+ mhr = build_mhr(args, required=_model_needs_mhr(model_path))+ pipeline = build_pipeline(args, mhr=mhr)+ viz = RerunVisualizer(application_id="instanthmr_demo",spawn_viewer=not args.no_spawn,save_path=args.save_rrd,- mhr_renderer=mhr_renderer,+ mhr_renderer=mhr,+ mesh_alpha=args.mesh_alpha,)+ return pipeline, vizdef _print_timings(@@ -228,9 +318,7 @@ def run_image(args: argparse.Namespace) -> None:sys.exit(f"[error] cv2.imread failed for {image_path}")rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)- pipeline = build_pipeline(args)- mhr = build_mhr_renderer(args)- viz = build_visualizer(args, mhr)+ pipeline, viz = build_all(args)result = pipeline.predict(rgb)@@ -261,9 +349,7 @@ def run_video(args: argparse.Namespace) -> None:total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))print(f"video: {video_path.name}, {total} frames @ {fps:.1f} fps")- pipeline = build_pipeline(args)- mhr = build_mhr_renderer(args)- viz = build_visualizer(args, mhr)+ pipeline, viz = build_all(args)frame_idx = 0sent = 0@@ -341,9 +427,7 @@ def run_camera(args: argparse.Namespace) -> None:fps = cap.get(cv2.CAP_PROP_FPS) or 30.0print(f"camera {args.camera}: {width}x{height} @ {fps:.1f} fps — Ctrl+C to stop")- pipeline = build_pipeline(args)- mhr = build_mhr_renderer(args)- viz = build_visualizer(args, mhr)+ pipeline, viz = build_all(args)frame_idx = 0det_times: list[float] = []diff --git a/instanthmr/__init__.py b/instanthmr/__init__.pyindex 1194ccc..1fce43e 100644--- a/instanthmr/__init__.py+++ b/instanthmr/__init__.py@@ -1,6 +1,6 @@"""InstantHMR — standalone 3D human pose inference + Rerun visualization."""-from .inference import InstantHMR, HMRPrediction+from .inference import InstantHMR, HMRPrediction, onnx_output_namesfrom .detector import RFDETRDetectorfrom .pipeline import PosePipeline, FrameResultfrom .skeleton import JOINT_NAMES, SKELETON_EDGES, NUM_JOINTS, edges_for@@ -8,6 +8,7 @@ from .skeleton import JOINT_NAMES, SKELETON_EDGES, NUM_JOINTS, edges_for__all__ = ["InstantHMR","HMRPrediction",+ "onnx_output_names","RFDETRDetector","PosePipeline","FrameResult",diff --git a/instanthmr/inference.py b/instanthmr/inference.pyindex 0dca31b..0f6fdf2 100644--- a/instanthmr/inference.py+++ b/instanthmr/inference.py@@ -3,11 +3,22 @@InstantHMR takes a 224x224 person crop and a 3-vector CLIFF condition(bbox center / scale in full-frame coords) and returns mhr_params, shape,camera translation, and 70 joints in 2D (crop space) and 3D (camera coords,-body-centred, metres, Y-down).+rig-local, metres, Y-down).This module wraps the ONNX session and the per-person preprocessing — squarecrop, CLIFF cond, ImageNet normalisation, and re-projection of joints backinto full-frame pixels.++Two graph layouts are supported, and the session tells them apart by itself:++ * **5 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d, joints_3d)``+ — the student has its own 3D coordinate head (``train_distill.py`` and the+ optimized / correctives scripts).+ * **4 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d)`` — the+ MHR-only student (``train_distill_mhr_only.py``) has no 3D head, exactly+ like the SAM3D teacher. Its 70 keypoints are derived here by running the+ MHR skeleton forward on the predicted parameters, so such a model needs an+ MHR backend (see :mod:`instanthmr.mhr_renderer`) passed as ``mhr=``."""from __future__ import annotations@@ -27,6 +38,23 @@ INPUT_SIZE = 224CROP_EXPAND = 1.2 # square crop around the detector bbox, matching training+def onnx_output_names(onnx_path: str | Path) -> Optional[list[str]]:+ """Read a graph's output names without creating an inference session.++ Lets a caller know whether it must supply an MHR backend *before* paying for+ session creation and detector loading. Returns ``None`` when the ``onnx``+ package is unavailable or the file cannot be parsed — callers should then+ fall back to letting :class:`InstantHMR` decide.+ """+ try:+ import onnx++ model = onnx.load(str(onnx_path), load_external_data=False)+ return [o.name for o in model.graph.output]+ except Exception: # noqa: BLE001 — purely advisory+ return None++@dataclassclass HMRPrediction:"""Per-person outputs from one InstantHMR forward pass.@@ -36,7 +64,9 @@ class HMRPrediction:Attributes:bbox: (4,) raw detector bbox [x1, y1, x2, y2].confidence: detection confidence in [0, 1].- joints_3d_local: (70, 3) body-centred joints, metres, Y-down.+ joints_3d_local: (70, 3) rig-local joints, metres, Y-down. Either read+ straight from the model's 3D head, or regressed from the MHR+ skeleton when the graph has no such head.joints_3d_cam: (70, 3) joints in camera space (= local + cam_trans).joints_2d: (70, 2) joints projected into full-frame pixel coords.cam_trans: (3,) camera translation [tx, ty, tz], metres.@@ -67,6 +97,15 @@ class HMRPrediction:class InstantHMR:"""ONNX wrapper for the InstantHMR model.+ Args:+ onnx_path: path to the exported graph.+ device: ``"cuda"``, ``"coreml"`` or ``"cpu"``.+ providers: explicit onnxruntime execution providers (overrides *device*).+ mhr: MHR backend used to derive ``joints_3d`` when the graph has no 3D+ head. Any object exposing ``keypoints(mhr_params, shape_params)``+ works; see :func:`instanthmr.mhr_renderer.build_mhr`. Ignored for+ 5-output graphs.+Example-------hmr = InstantHMR("models/instanthmr.onnx")@@ -79,6 +118,7 @@ class InstantHMR:onnx_path: str | Path,device: str = "cuda",providers: Optional[list[str]] = None,+ mhr: object | None = None,):import onnxruntime as ort@@ -116,8 +156,42 @@ class InstantHMR:out_names = [o.name for o in self.session.get_outputs()]self._in_image = in_names[0] # "image"self._in_cliff = in_names[1] # "cliff_cond"- # Output order from export: mhr_params, shape_params, cam_trans, joints_2d, joints_3d+ # Output order from export: mhr_params, shape_params, cam_trans,+ # joints_2d[, joints_3d]. The MHR-only student stops after joints_2d.self._out_names = out_names+ self.has_joints_3d_head = "joints_3d" in out_names or len(out_names) >= 5+ self._mhr = mhr++ @property+ def derives_joints_3d(self) -> bool:+ """True when the 70 keypoints must come from an MHR forward pass."""+ return not self.has_joints_3d_head++ def set_mhr(self, mhr: object | None) -> None:+ """Attach (or replace) the MHR backend used to derive ``joints_3d``."""+ self._mhr = mhr++ def _joints_3d(self, mhr_params: np.ndarray, shape_params: np.ndarray) -> np.ndarray:+ """Regress the 70 keypoints from predicted MHR parameters.++ Mirrors ``train_distill_mhr_only.MHRForwardPass.get_native_keypoints``:+ skeleton forward, cm -> m, Y/Z flip, then the (70, 127) affine map. The+ student was trained through exactly this operator, so any other+ keypoint definition would silently shift the predictions.+ """+ if self._mhr is None:+ raise RuntimeError(+ "this ONNX graph has no 'joints_3d' output — it comes from "+ "train_distill_mhr_only.py, where the 3D keypoints are derived from "+ "the MHR forward pass.\n"+ "Pass an MHR backend, e.g.\n"+ " from instanthmr.mhr_renderer import build_mhr\n"+ " InstantHMR(onnx, mhr=build_mhr(script_path='checkpoints/mhr_model.pt'))\n"+ "or run demo.py with --mhr-model / --mhr-assets."+ )+ return np.asarray(+ self._mhr.keypoints(mhr_params, shape_params), dtype=np.float32,+ )# ------------------------------------------------------------------# Public API@@ -154,12 +228,15 @@ class InstantHMR:self._in_cliff: cliff_cond[np.newaxis], # (1, 3)},)- # Order: mhr_params, shape_params, cam_trans, joints_2d, joints_3d+ # Order: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]mhr_params = outs[0][0].astype(np.float32)shape_params = outs[1][0].astype(np.float32)cam_trans = outs[2][0].astype(np.float32)joints_2d_norm = outs[3][0].astype(np.float32) # (70, 2) in [-1, 1]- joints_3d_local = outs[4][0].astype(np.float32) # (70, 3) body-centred+ if self.has_joints_3d_head:+ joints_3d_local = outs[4][0].astype(np.float32) # (70, 3) rig-local+ else:+ joints_3d_local = self._joints_3d(mhr_params, shape_params)# Re-project the 2D head from normalised crop space → full-frame pixels.crop_px = (joints_2d_norm + 1.0) * 0.5 * INPUT_SIZE@@ -238,7 +315,12 @@ class InstantHMR:shape_params_b = outs[1].astype(np.float32, copy=False)cam_trans_b = outs[2].astype(np.float32, copy=False)joints_2d_norm_b = outs[3].astype(np.float32, copy=False)- joints_3d_local_b = outs[4].astype(np.float32, copy=False)+ if self.has_joints_3d_head:+ joints_3d_local_b = outs[4].astype(np.float32, copy=False)+ else:+ # One batched MHR skeleton forward for the whole frame (~0.9 ms),+ # including the zero-padded slots — they are discarded below.+ joints_3d_local_b = self._joints_3d(mhr_params_b, shape_params_b)f = math.sqrt(h * h + w * w)focal_length = np.array([f, f], dtype=np.float32)diff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.pyindex fc6b79d..b08a5e9 100644--- a/instanthmr/mhr_renderer.py+++ b/instanthmr/mhr_renderer.py@@ -1,15 +1,39 @@-"""MHR (Momentum Human Rig) mesh decoder for InstantHMR.+"""MHR (Momentum Human Rig) decoder for InstantHMR.Takes the 204-dim pose parameters and 45-dim shape parameters produced by-InstantHMR and decodes them into full body mesh vertices by running a forward-pass through Meta's MHR body model.+InstantHMR and runs a forward pass through Meta's MHR body model to obtain-Quick setup (requires Python >= 3.12)::+ * the full body **mesh** (skinned vertices), and+ * the 70 native **keypoints**, regressed from the 127 skeleton joints.++The keypoints matter because ``train_distill_mhr_only.py`` deletes the+student's 3D coordinate head: its ONNX graph has four outputs instead of five+and the 3D pose must be derived here, exactly the way the teacher derives it+from the mesh. The (70, 127) affine map lives in+``instanthmr_distill_train/assets/mhr_j127_to_kp70.npy`` and was fitted against+the teacher's own labels (held-out residual: 1.36 mm mean).++Two interchangeable backends+----------------------------+``MHRScriptRenderer`` (recommended)+ Loads the self-contained TorchScript rig at ``checkpoints/mhr_model.pt`` —+ the same artefact the distillation used. Needs nothing but PyTorch, so it+ is immune to the ``pymomentum`` / ``torch`` ABI pairing below.++``MHRRenderer``+ Loads the unpacked MHR asset folder through the ``mhr`` pip package. This+ lets you pick a mesh LOD, but ``pymomentum-gpu`` is a torch C++ extension+ and **must** match the installed torch (0.1.113 wants ``torch>=2.8,<2.9``);+ a mismatch segfaults the process rather than raising.++Use :func:`build_mhr` to get whichever one the caller configured.++Quick setup for the asset backend (requires Python >= 3.12)::pip install -r requirements-mhr.txtcurl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zipunzip assets.zip -d models/mhr_assets- python demo.py --image photo.jpg --mhr-assets models/mhr_assets+ python demo.py --image photo.jpg --mhr-assets models/mhr_assets/assetsWARNING — wrong pymomentum package on PyPI------------------------------------------@@ -25,20 +49,311 @@ See README.md §'MHR body mesh' for the full installation walkthrough.from __future__ import annotationsfrom pathlib import Path+from typing import Optionalimport numpy as np-class MHRRenderer:- """Loads and caches the MHR body model, runs per-person forward passes.+_REPO_ROOT = Path(__file__).resolve().parent.parent++#: (70, 127) affine map from MHR skeleton joints to the 70 native keypoints.+DEFAULT_KP_REGRESSOR = _REPO_ROOT / "instanthmr_distill_train/assets/mhr_j127_to_kp70.npy"+#: TorchScript rig shipped with the training code.+DEFAULT_MHR_SCRIPT = _REPO_ROOT / "checkpoints/mhr_model.pt"++NUM_MODEL_PARAMS = 204 # 136 pose + 68 scale+NUM_IDENTITY = 45 # identity blend-shape coefficients+NUM_EXPRESSION = 72 # facial expression coefficients (InstantHMR predicts none)+NUM_SKELETON_JOINTS = 127+NUM_KEYPOINTS = 70+++class _MHRBase:+ """Parameter marshalling, coordinate conversion, rig evaluation and keypoints.++ Both backends expose the same three objects, so everything below the loader+ is shared. A subclass' ``__init__`` must set:++ * ``self._model`` — callable ``(identity, pose, expr) -> (verts, skel)``+ * ``self._character`` — exposes ``model_parameters_to_joint_parameters``+ and ``joint_parameters_to_skeleton_state``+ * ``self._n_params`` — width its parameter transform expects, i.e. 204+ model parameters plus that rig's blend-shape slots++ and it must implement the ``faces`` property.+ """++ def __init__(self, device: str = "cuda", kp_regressor: str | Path | None = None):+ import torch++ self._torch = torch+ self._device = self._resolve_device(torch, device)+ self._kp_regressor = None++ path = Path(kp_regressor) if kp_regressor is not None else DEFAULT_KP_REGRESSOR+ if path.exists():+ W = np.load(path).astype(np.float32)+ if W.shape != (NUM_KEYPOINTS, NUM_SKELETON_JOINTS):+ raise ValueError(+ f"expected a ({NUM_KEYPOINTS}, {NUM_SKELETON_JOINTS}) keypoint "+ f"regressor in {path}, got {W.shape}"+ )+ self._kp_regressor = torch.from_numpy(W).to(self._device)+ elif kp_regressor is not None:+ raise FileNotFoundError(f"keypoint regressor not found: {path}")++ @staticmethod+ def _resolve_device(torch, device):+ """Map the demo's device string onto a torch device.++ ``--device coreml`` selects an onnxruntime execution provider, not a+ torch backend, so anything torch cannot run on falls back to CPU.+ """+ name = str(device).lower()+ if "cuda" in name and torch.cuda.is_available():+ return torch.device(device)+ if "cuda" in name:+ return torch.device("cpu")+ try:+ return torch.device(device)+ except (RuntimeError, TypeError):+ return torch.device("cpu")++ # ------------------------------------------------------------------+ # Capabilities+ # ------------------------------------------------------------------++ @property+ def has_keypoints(self) -> bool:+ """True when the 70-keypoint regressor was found and loaded."""+ return self._kp_regressor is not None++ @property+ def faces(self) -> np.ndarray:+ raise NotImplementedError++ # ------------------------------------------------------------------+ # Coordinate conventions+ # ------------------------------------------------------------------++ @staticmethod+ def _to_vision(xyz):+ """Raw MHR (centimetres, Y-up / Z-back) -> InstantHMR (metres, Y-down / Z-forward).++ Negating both Y and Z keeps the frame right-handed after the Y flip.+ This is the exact transform ``train_distill_mhr_only.MHRForwardPass``+ applies before the keypoint regression, so student and demo agree.+ """+ out = xyz / 100.0+ out[..., 1] *= -1.0+ out[..., 2] *= -1.0+ return out++ # ------------------------------------------------------------------+ # Public API+ # ------------------------------------------------------------------++ def keypoints(+ self,+ mhr_params: np.ndarray,+ shape_params: np.ndarray,+ ) -> np.ndarray:+ """Regress the 70 native keypoints from the MHR skeleton.++ Skeleton branch only — no blend shapes, no skinning — so this is ~3x+ cheaper than a full mesh forward (≈0.9 ms vs ≈2.6 ms at batch 2 on an+ RTX 4070).++ Args:+ mhr_params: ``(204,)`` or ``(B, 204)`` float32 pose parameters.+ shape_params: ``(45,)`` or ``(B, 45)`` — accepted for symmetry with+ :meth:`forward`; identity coefficients do not move the skeleton+ (they only displace the rest-pose vertices), which is why the+ distillation passes zeros here too.++ Returns:+ ``(70, 3)`` or ``(B, 70, 3)`` float32 keypoints in the **rig-local**+ frame (metres, Y-down) — the frame ``cam_trans`` is relative to.+ Add ``cam_trans`` for camera space.+ """+ if self._kp_regressor is None:+ raise RuntimeError(+ f"no MHR keypoint regressor loaded (looked for {DEFAULT_KP_REGRESSOR}).\n"+ "It ships with the training code; regenerate it with\n"+ " python instanthmr_distill_train/train_distill_mhr_only.py --fit-regressor"+ )+ torch = self._torch+ pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)++ with torch.no_grad():+ skel = self._skeleton_state(pose)+ kps = self._regress(skel)++ out = kps.cpu().numpy().astype(np.float32)+ return out[0] if squeeze else out++ def forward(+ self,+ mhr_params: np.ndarray,+ shape_params: np.ndarray,+ ) -> np.ndarray:+ """Decode one person's parameters into body-space mesh vertices.++ Args:+ mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.+ shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.++ Returns:+ ``(V, 3)`` float32 vertices in the **rig-local** frame (metres,+ Y-down). Add the person's ``cam_trans`` to move to camera space.+ """+ verts, _ = self.forward_full(mhr_params, shape_params)+ return verts++ def forward_full(+ self,+ mhr_params: np.ndarray,+ shape_params: np.ndarray,+ ) -> tuple[np.ndarray, Optional[np.ndarray]]:+ """Mesh vertices **and** the 70 keypoints from a single MHR forward pass.++ Returns ``(verts, keypoints)`` with the same leading shape convention as+ :meth:`forward` / :meth:`keypoints`. ``keypoints`` is ``None`` when no+ regressor is available.+ """+ torch = self._torch+ pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)+ identity, _ = self._as_batch(shape_params, NUM_IDENTITY)+ if identity.shape[0] != pose.shape[0]:+ identity = identity.expand(pose.shape[0], -1)++ with torch.no_grad():+ verts, skel = self._skin(identity, pose)+ verts = self._to_vision(verts.float())+ kps = self._regress(skel) if self._kp_regressor is not None else None++ v = verts.cpu().numpy().astype(np.float32)+ k = kps.cpu().numpy().astype(np.float32) if kps is not None else None+ if squeeze:+ return v[0], (None if k is None else k[0])+ return v, k++ # ------------------------------------------------------------------+ # Internals+ # ------------------------------------------------------------------++ def _as_batch(self, arr: np.ndarray, dim: int):+ """numpy ``(dim,)`` / ``(B, dim)`` -> torch ``(B, dim)`` on the rig device."""+ torch = self._torch+ t = torch.as_tensor(np.ascontiguousarray(arr), dtype=torch.float32)+ squeeze = t.dim() == 1+ if squeeze:+ t = t.unsqueeze(0)+ if t.shape[-1] != dim:+ raise ValueError(f"expected trailing dimension {dim}, got {tuple(t.shape)}")+ return t.to(self._device), squeeze++ def _regress(self, skel):+ """``(B, 127, 8)`` skeleton state -> ``(B, 70, 3)`` keypoints (vision frame)."""+ joints = self._to_vision(skel[..., :3].float())+ return self._torch.einsum("kj,bjc->bkc", self._kp_regressor, joints)++ def _pad(self, pose, total: int):+ """Right-pad the 204 model parameters with zero blend-shape coefficients."""+ torch = self._torch+ extra = total - pose.shape[-1]+ if extra <= 0:+ return pose+ return torch.cat(+ [pose, torch.zeros(pose.shape[0], extra, dtype=pose.dtype, device=pose.device)],+ dim=1,+ )++ def _skeleton_state(self, pose):+ """``(B, 204)`` model parameters -> ``(B, 127, 8)`` raw skeleton state.++ Skips blend shapes and skinning entirely: only the two rigid-transform+ stages the keypoint regression needs.+ """+ ch = self._character+ joint_params = ch.model_parameters_to_joint_parameters(self._pad(pose, self._n_params))+ return ch.joint_parameters_to_skeleton_state(joint_params)++ def _skin(self, identity, pose):+ """Full rig evaluation -> ``((B, V, 3) verts, (B, 127, 8) skeleton state)``.++ InstantHMR predicts no facial expression, so those coefficients are zero.+ """+ torch = self._torch+ expr = torch.zeros(+ pose.shape[0], NUM_EXPRESSION, dtype=pose.dtype, device=pose.device,+ )+ verts, skel = self._model(identity, pose, expr)+ return verts, skel+++class MHRScriptRenderer(_MHRBase):+ """MHR rig loaded from the TorchScript artefact used by the distillation.++ Args:+ script_path: path to ``mhr_model.pt`` (default: ``checkpoints/mhr_model.pt``).+ device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.+ kp_regressor: optional override for the (70, 127) keypoint map.++ The mesh LOD is baked into the artefact (18 439 vertices / 36 874 faces);+ use :class:`MHRRenderer` if you need to choose one.+ """++ def __init__(+ self,+ script_path: str | Path = DEFAULT_MHR_SCRIPT,+ device: str = "cuda",+ kp_regressor: str | Path | None = None,+ ) -> None:+ super().__init__(device=device, kp_regressor=kp_regressor)+ torch = self._torch++ script_path = Path(script_path)+ if not script_path.exists():+ raise FileNotFoundError(f"MHR TorchScript rig not found: {script_path}")++ self._model = torch.jit.load(str(script_path), map_location=self._device).eval()+ for p in self._model.parameters():+ p.requires_grad_(False)++ self._character = self._model.character_torch+ # 204 model parameters + 45 identity blend-shape slots in this rig's+ # parameter transform; read it off rather than assuming.+ self._n_params = len(self._character.parameter_transform.parameter_names)++ faces = self._character.mesh.faces+ self._faces = faces.cpu().numpy().astype(np.int32)++ @property+ def faces(self) -> np.ndarray:+ """Triangle face indices ``(F, 3)`` int32, shared across all frames."""+ return self._faces++++class MHRRenderer(_MHRBase):+ """MHR rig loaded from the unpacked asset folder via the ``mhr`` pip package.Args:- assets_folder: path to the unpacked MHR assets folder (the directory- that contains the blend-shape files, template mesh, etc.).+ assets_folder: path to the folder that holds ``lod*.fbx`` and+ ``compact_v6_1.model`` (i.e. the ``assets/`` directory inside the+ unpacked ``assets.zip``).device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.lod: level-of-detail mesh resolution:0 = 73 639 verts (highest), 3 = 4 899, 6 = 595 (lowest).LOD 3 is a good default for live demos.+ kp_regressor: optional override for the (70, 127) keypoint map.++ Note:+ ``pymomentum-gpu`` is a torch C++ extension. If its build does not match+ the installed torch (0.1.113 requires ``torch>=2.8,<2.9``) the process+ segfaults inside the native library instead of raising — prefer+ :class:`MHRScriptRenderer` when in doubt."""def __init__(@@ -46,20 +361,22 @@ class MHRRenderer:assets_folder: str | Path = "models/mhr_assets",device: str = "cuda",lod: int = 3,+ kp_regressor: str | Path | None = None,) -> None:- import torch+ super().__init__(device=device, kp_regressor=kp_regressor)from mhr.mhr import MHR- self._torch = torch- self._device = torch.device(device)- self._model = MHR.from_files(- folder=Path(assets_folder),- device=self._device,- lod=lod,- )+ folder = Path(assets_folder)+ # assets.zip unpacks to <dir>/assets/ — accept either level.+ if not (folder / f"lod{lod}.fbx").exists() and (folder / "assets" / f"lod{lod}.fbx").exists():+ folder = folder / "assets"++ self._model = MHR.from_files(folder=folder, device=self._device, lod=lod)+ self._character = self._model.character_torch+ self._n_params = self._model.character.parameter_transform.sizefaces_raw = self._model.character.mesh.faces- if isinstance(faces_raw, torch.Tensor):+ if hasattr(faces_raw, "cpu"):self._faces: np.ndarray = faces_raw.cpu().numpy().astype(np.int32)else:self._faces = np.asarray(faces_raw, dtype=np.int32)@@ -69,42 +386,22 @@ class MHRRenderer:"""Triangle face indices ``(F, 3)`` int32, shared across all frames."""return self._faces- def forward(- self,- mhr_params: np.ndarray,- shape_params: np.ndarray,- ) -> np.ndarray:- """Decode one person's parameters into body-space mesh vertices.- Args:- mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.- shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.- Returns:- ``(V, 3)`` float32 vertices in **body-centred** space (metres,- Y-down). Add the person's ``cam_trans`` to move to camera space.- """- torch = self._torch- identity = torch.from_numpy(shape_params).unsqueeze(0).to(self._device) # (1, 45)- pose = torch.from_numpy(mhr_params).unsqueeze(0).to(self._device) # (1, 204)- # MHR also accepts facial expression coefficients (72-D); InstantHMR- # does not predict them, so we pass zeros (neutral expression).- expr = torch.zeros(1, 72, dtype=torch.float32, device=self._device)+def build_mhr(+ script_path: str | Path | None = None,+ assets_folder: str | Path | None = None,+ device: str = "cuda",+ lod: int = 3,+ kp_regressor: str | Path | None = None,+) -> _MHRBase | None:+ """Build whichever MHR backend the caller configured.- with torch.no_grad():- verts, _ = self._model(identity, pose, expr)-- v = verts[0].cpu().numpy().astype(np.float32) # (V, 3)-- # --- Coordinate-space alignment ------------------------------------ # MHR body template (FBX) is Y-up right-handed and in centimetres.- # InstantHMR / SAM3D uses Y-down right-handed and metres.- #- # 1. cm → m- v /= 100.0- # 2. Y-up → Y-down: negate Y.- # Negate Z as well to preserve right-handedness after the Y flip.- v[:, 1] *= -1.0- v[:, 2] *= -1.0-- return v+ ``script_path`` wins when both are given. Returns ``None`` when neither is+ set, so callers can treat "no MHR" as a plain absence.+ """+ if script_path is not None:+ return MHRScriptRenderer(script_path, device=device, kp_regressor=kp_regressor)+ if assets_folder is not None:+ return MHRRenderer(assets_folder, device=device, lod=lod, kp_regressor=kp_regressor)+ return Nonediff --git a/instanthmr/pipeline.py b/instanthmr/pipeline.pyindex c3e8ddd..ff5df10 100644--- a/instanthmr/pipeline.py+++ b/instanthmr/pipeline.py@@ -68,6 +68,10 @@ class PosePipeline:frame are batched into a single InstantHMR ONNX call insteadof looping. This is a meaningful win for multi-person framesand free for single-person frames.+ mhr: Optional MHR backend (see :mod:`instanthmr.mhr_renderer`).+ Required for MHR-only graphs, whose 3D keypoints are derived+ from the predicted MHR parameters instead of a 3D head; ignored+ by graphs that carry a ``joints_3d`` output.Example-------@@ -86,11 +90,19 @@ class PosePipeline:max_persons: int = 2,detector_stride: int = 1,batch_persons: bool = True,+ mhr: object | None = None,):if detector_stride < 1:raise ValueError("detector_stride must be >= 1")- self.hmr = InstantHMR(onnx_path, device=device)+ self.hmr = InstantHMR(onnx_path, device=device, mhr=mhr)+ if self.hmr.derives_joints_3d and mhr is None:+ raise ValueError(+ f"{Path(onnx_path).name} has no 'joints_3d' output (MHR-only student): "+ "its 3D keypoints come from an MHR forward pass, so an MHR backend is "+ "required. Pass mhr=build_mhr(...), or run demo.py with --mhr-model / "+ "--mhr-assets."+ )self.detector = RFDETRDetector(variant=detector_variant,confidence=det_confidence,@@ -106,6 +118,8 @@ class PosePipeline:print(f"Warming up models (max_persons={max_persons})…", flush=True)self.detector.warmup()self.hmr.warmup(max_batch_size=max_persons)+ if mhr is not None:+ self._warmup_mhr(mhr, max_persons)print("Ready.", flush=True)self._max_persons = max_persons@@ -115,6 +129,24 @@ class PosePipeline:self._last_detections: list[dict] = []self._last_bbox: Optional[np.ndarray] = None+ @staticmethod+ def _warmup_mhr(mhr, max_persons: int) -> None:+ """Run the MHR rig once per batch shape the demo will actually use.++ TorchScript specialises on input shapes over the first few calls, so an+ un-warmed rig costs hundreds of ms on frame 1 instead of ~1 ms. Three+ passes is what it takes for the profiling executor to settle: the first+ records shapes, the second triggers the fusion compile, the third runs+ the compiled kernels.+ """+ pose = np.zeros((max_persons, 204), dtype=np.float32)+ shape = np.zeros((max_persons, 45), dtype=np.float32)+ for _ in range(3):+ mhr.forward(pose[0], shape[0]) # mesh: always single-person+ if getattr(mhr, "has_keypoints", False):+ for n in sorted({1, max_persons}): # keypoints: 1 and the padded batch+ mhr.keypoints(pose[:n], shape[:n])+def predict(self, image_rgb: np.ndarray) -> FrameResult:"""Detect all persons in *image_rgb* and run InstantHMR on each."""t_total_start = time.perf_counter()diff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.pyindex e51587e..bbf9f3d 100644--- a/instanthmr/visualizer.py+++ b/instanthmr/visualizer.py@@ -5,9 +5,13 @@ Logs, per frame:text panel showing detector / HMR / render / total ms).- ``camera`` : pinhole intrinsics so the image becomes aproper frustum in the 3D scene.- - ``world/persons/...`` : 3D joints + skeleton lines per person.- - ``world/persons/.../mesh``: full MHR body mesh when an MHRRenderer is+ - ``world/persons/...`` : the 70 3D keypoints + skeleton lines per person.+ For an MHR-only student these come from the MHR+ forward pass itself (see ``instanthmr.inference``).+ - ``world/persons/.../mesh``: full MHR body mesh when an MHR backend isprovided (vertices from a true MHR forward pass).+ Rendered semi-transparent via ``albedo_factor``+ so the keypoints inside the body stay visible.- ``timing/detector_ms`` : scalar plot of RF-DETR latency.- ``timing/hmr_ms`` : scalar plot of InstantHMR latency.- ``timing/render_ms`` : scalar plot of MHR mesh rendering latency.@@ -32,19 +36,44 @@ from .skeleton import edges_forif TYPE_CHECKING:from .mhr_renderer import MHRRenderer+#: Base skin tone of the body mesh, before the transparency factor is applied.+MESH_COLOR = (200, 160, 130)+#: Default mesh opacity. Low enough that the 70 keypoints read through the+#: body, high enough that the silhouette is still legible.+MESH_ALPHA = 0.35+class RerunVisualizer:+ """Rerun logger for the demo.++ Args:+ application_id: Rerun application id.+ spawn_viewer: open the viewer GUI on construction.+ save_path: optional ``.rrd`` recording path.+ mhr_renderer: MHR backend used to skin the body mesh. Any object with+ ``.forward(mhr_params, shape_params)`` and ``.faces`` works; see+ :mod:`instanthmr.mhr_renderer`.+ mesh_alpha: body-mesh opacity in ``[0, 1]``. Applied through Rerun's+ ``albedo_factor`` rather than per-vertex alpha, so one scalar+ controls the whole surface.+ """+def __init__(self,application_id: str = "instanthmr_demo",spawn_viewer: bool = True,save_path: str | None = None,mhr_renderer: "MHRRenderer | None" = None,+ mesh_alpha: float = MESH_ALPHA,):import rerun as rrself._rr = rrself._mhr_renderer = mhr_renderer+ self._mesh_albedo = (+ *MESH_COLOR,+ int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),+ )rr.init(application_id, spawn=spawn_viewer)# SAM3D / MHR convention: right-handed, Y-down camera frame.@@ -165,6 +194,12 @@ class RerunVisualizer:# ------------------------------------------------------------------def _log_person(self, idx: int, person: HMRPrediction) -> None:+ """Log the 70 3D keypoints and the skeleton wiring for one person.++ For an MHR-only student ``joints_3d_cam`` was regressed from the MHR+ skeleton that also produced the mesh, so the points sit exactly inside+ the surface logged by :meth:`_log_person_mesh`.+ """rr = self._rrpath = f"world/persons/person_{idx}"@@ -199,15 +234,16 @@ class RerunVisualizer:faces = self._mhr_renderer.faces # (F, 3) int32, shared across frames- n = len(verts_cam)- skin = np.tile(np.array([[200, 160, 130, 140]], dtype=np.uint8), (n, 1))-+ # Tint + transparency come from `albedo_factor`, a single RGBA multiplier+ # for the whole surface. Per-vertex colours would work too, but the+ # viewer multiplies the two, so keeping the alpha in one place makes the+ # `mesh_alpha` knob mean exactly what it says.self._rr.log(f"world/persons/person_{idx}/mesh",self._rr.Mesh3D(vertex_positions=verts_cam,triangle_indices=faces,- vertex_colors=skin,+ albedo_factor=self._mesh_albedo,),)diff --git a/instanthmr_distill_train/requirements.txt b/instanthmr_distill_train/requirements.txtindex 0dc2ea7..0d83e24 100644--- a/instanthmr_distill_train/requirements.txt+++ b/instanthmr_distill_train/requirements.txt@@ -17,6 +17,15 @@ numpy>=1.24pillow>=10opencv-python>=4.8+# --- LeVJEPA encoder (train_levjepa_*.py) ---+# THIS is what was missing when the perelha job died with+# "ModuleNotFoundError: No module named 'transformers'".+# The weights ship their own modeling code, hence trust_remote_code=True, which+# needs `transformers` itself plus safetensors to read the checkpoint.+transformers>=4.44+safetensors>=0.4+huggingface_hub>=0.24+# --- utilities / logging ---tqdm>=4.65matplotlib>=3.7diff --git a/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh b/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.shindex 0677d92..fc57210 100755--- a/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh+++ b/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh@@ -12,7 +12,7 @@ GPU_REQUIRED=true# --data_root points at the dataset synced by datasynch_perso (see below).# It is synced next to this project folder, i.e. ../instanthmr_data once we# cd into instanthmr_distill_train.-EXEC="python3 -u train_distill_mhr_only.py --data_root /datasets/instanthmr_data --num_workers 8 --batch_size 128 $useropt"+EXEC="python3 -u train_distill_mhr_only.py --data_root /datasets/instanthmr_data --num_workers 8 --batch_size 128 --w_keypoints3d 1.0 $useropt"# SBATCH --output/--error paths are relative to the submission directory, so# create the log folder before submitting.@@ -28,10 +28,9 @@ sbatch ${sbatchopt} << eof#SBATCH --mail-type=ALL#SBATCH --ntasks=1#SBATCH --cpus-per-task=8-#SBATCH --output=instanthmr_distill_mhr_only/stdout_dolcino.txt-#SBATCH --error=instanthmr_distill_mhr_only/stderr_dolcino.txt-#SBATCH --account=dept_rob-#SBATCH --partition=robgpu+#SBATCH --output=instanthmr_distill_mhr_only/stdout_perelha.txt+#SBATCH --error=instanthmr_distill_mhr_only/stderr_perelha.txt+#SBATCH --partition=gpu#SBATCH --gres=gpu:1#SBATCH --mem-per-gpu=50000#SBATCH --time=4-00:00:00diff --git a/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh b/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.shnew file mode 100755index 0000000..538d592--- /dev/null+++ b/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh@@ -0,0 +1,77 @@+#!/bin/sh -e++echo "example: $0 --gpu 1"++# if `requirements.txt` is present, virtual environment is installed and used+EXEC="python3 train_levjepa_cloud.py"++GPU_REQUIRED=true++. /pfcalcul/tools/sbatchHelpers5.sh++# ---------------------------------------------------------------------------+# BEFORE THE FIRST SUBMIT, stage the encoder ONCE from a machine with internet:+#+# python3 -c "from huggingface_hub import snapshot_download; \+# print(snapshot_download('galilai-group/LeVJEPA-VideoMix-Large'))"+# rsync -avPL <printed path>/ <cluster>:<ENCODER_DIR>/+#+# -L (--copy-links) is MANDATORY. A HuggingFace snapshot dir is nothing but+# symlinks into ../../blobs/, so plain `rsync -av` copies six dangling links+# (~650 bytes) instead of the 1.2 GB of weights, and the job then fails to load+# the model. The trailing slash on the source flattens the snapshot into+# ENCODER_DIR rather than nesting it under a commit-hash folder.+#+# Compute nodes generally cannot reach huggingface.co, so --encoder_path points+# at that staged copy and the loader runs with local_files_only=True. Without+# it the job dies on the first AutoModel.from_pretrained call.+#+# requirements.txt must contain `transformers` + `safetensors` (the weights ship+# their own modeling code, hence trust_remote_code=True). A venv built before+# that line was added will NOT have it -- delete venv/ so it is rebuilt, or the+# job fails with "ModuleNotFoundError: No module named 'transformers'".+# ---------------------------------------------------------------------------+# NOTE the LEADING SLASH: without it this is a relative path and resolves+# against the submission directory, so from_pretrained gets a path that does+# not exist. Point it at the flattened snapshot (config.json + *.py ++# model.safetensors directly inside), not at a nested commit-hash folder.+ENCODER_DIR="/pfcalcul/work/kchalabi/envs/lstm/instanthmr_distill_train/levjepa_encoder"++# batch_size is in CLIPS (16 frames each). Stills are encoded as T=1 -- 197+# tokens instead of 3137 -- so they run at static_batch_mult x that batch.+# The encoder is FROZEN: no backward through 305 M params, no LoRA state.+# --max_hours 92 checkpoints and stops cleanly before SLURM's 4-day SIGKILL.+EXEC="python3 -u train_levjepa_cloud.py --data_root /datasets/instanthmr_data --encoder_path $ENCODER_DIR --num_workers 8 --batch_size 8 --static_batch_mult 8 --epochs 30 --max_hours 92 $useropt"++# SBATCH --output/--error paths are relative to the submission directory, so+# create the log folder before submitting.+mkdir -p instanthmr_levjepa_cloud++jobMessage=$(+######################### SBATCH launcher+######################### #SBATCH --option # <= this is an enabled paramater+######################### ##SBATCH --option # <= this is a disabled parameter+sbatch ${sbatchopt} << eof+#!/bin/bash+#SBATCH --job-name="instanthmr_levjepa_cloud"+#SBATCH --mail-type=ALL+#SBATCH --ntasks=1+#SBATCH --cpus-per-task=8+#SBATCH --output=instanthmr_levjepa_cloud/stdout_perelha.txt+#SBATCH --error=instanthmr_levjepa_cloud/stderr_perelha.txt+#SBATCH --partition=gpu+#SBATCH --gres=gpu:1+#SBATCH --mem-per-gpu=50000+#SBATCH --time=4-00:00:00++. /pfcalcul/tools/sbatchHelpers5.sh++/pfcalcul/tools/datasync /datasets/instanthmr_data+++time $EXEC++eof+)++showSubmitteddiff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex 1e00486..341f64f 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -181,6 +181,16 @@ class DistillConfig:warmup_epochs: int = 3grad_clip: float = 1.0anomaly_loss_threshold: float = 100.0+ # A skipped step updates NOTHING — not the weights, not the EMA, not the LR+ # schedule (they all live behind the same `continue`). So a model that is+ # anomalous on every batch is frozen for good: the loss can never come back+ # under the threshold and the run idles until early stopping, which took+ # ~139 wasted epochs the one time it happened. After this many CONSECUTIVE+ # skips, roll the raw weights back to the EMA copy, scale the LR down and+ # clear the optimiser moments; give up after `max_ema_rollbacks` attempts.+ anomaly_skip_patience: int = 50+ anomaly_rollback_lr_decay: float = 0.5+ max_ema_rollbacks: int = 3use_amp: bool = Trueema_decay: float = 0.9998early_stop_patience: int = 150@@ -837,6 +847,42 @@ class DistillationLoss(nn.Module):losses['total_loss'] = sum(losses.values())return losses+def format_loss_terms(losses, top_k: int = 5) -> str:+ """The largest individual loss terms, for diagnosing an anomalous batch.++ The anomaly guard only ever printed the total, which says nothing about+ which head blew up. The 2D terms are bounded (`joints_2d` is a SimCC+ expectation over bins in [-kp2d_range, kp2d_range], and the SimCC CE by+ log(kp2d_bins)); the parameter regressions and the FK-derived keypoint loss+ are not, so this line is what tells the two cases apart in the job log.+ """+ terms = [(k.replace("loss_", ""), float(v))+ for k, v in losses.items() if k != "total_loss"]+ terms.sort(key=lambda kv: kv[1], reverse=True)+ return " ".join(f"{k}={v:.1f}" for k, v in terms[:top_k])+++def scale_lr(optimizer, factor: float, scheduler=None) -> None:+ """Multiply every learning-rate anchor in place.++ OneCycleLR does not read a stored LR — it interpolates between `initial_lr`,+ `max_lr` and `min_lr` on the param groups, and those all scale linearly with+ the peak. Scaling the anchors therefore rescales the whole remaining+ schedule while leaving its *position* (``last_epoch``) untouched, which is+ what both the resume path and the rollback want.+ """+ for group in optimizer.param_groups:+ for key in ("lr", "max_lr", "initial_lr", "min_lr"):+ if key in group:+ group[key] *= factor+ if scheduler is not None:+ # LRScheduler stashes these at construction and restores them from a+ # checkpoint, so they have to move with the param groups.+ for attr in ("base_lrs", "_last_lr"):+ if hasattr(scheduler, attr):+ setattr(scheduler, attr, [lr * factor for lr in getattr(scheduler, attr)])++# ============================================================# Cell 10 — HMR Evaluation Metrics (MPJPE & PA-MPJPE)# ============================================================@@ -1010,6 +1056,17 @@ def train_instant_hmr():scaler.load_state_dict(ckpt["scaler_state_dict"])if "scheduler_state_dict" in ckpt:scheduler.load_state_dict(ckpt["scheduler_state_dict"])+ # Both loads above put the checkpoint's LR state back: `param_groups`+ # (max_lr / initial_lr / min_lr) come from the optimizer state_dict and+ # `base_lrs` from the scheduler's, which overwrites everything the+ # freshly-built OneCycleLR just installed. Without this, --lr is+ # SILENTLY IGNORED on every resume. Re-apply the requested peak by+ # rescaling the anchors, keeping the schedule's shape and position.+ ckpt_max_lr = float(optimizer.param_groups[0].get("max_lr", cfg.lr))+ if ckpt_max_lr > 0.0 and abs(ckpt_max_lr - cfg.lr) > 1e-12:+ scale_lr(optimizer, cfg.lr / ckpt_max_lr, scheduler)+ print(f" LR rescaled: checkpoint peak {ckpt_max_lr:.2e} -> requested "+ f"{cfg.lr:.2e} (current {optimizer.param_groups[0]['lr']:.2e})")start_epoch = ckpt.get("epoch", -1) + 1best_overall_pa = ckpt.get("val_pa_mpjpe", float('inf'))best_raw_pa = best_overall_pa@@ -1052,6 +1109,11 @@ def train_instant_hmr():global_step = start_epoch * steps_per_epochcriterion.set_step(global_step)+ # Skips are counted ACROSS epochs — a stalled run skips whole epochs at a+ # time, so a per-epoch counter would never trip.+ consecutive_skips = 0+ ema_rollbacks = 0+ training_aborted = Falsefor epoch in range(start_epoch, cfg.epochs):model.train()tm = {'total': 0.0, '2d': 0.0, 'simcc': 0.0, '3d': 0.0, 'pose': 0.0, 'scale': 0.0, 'shape': 0.0, 'mhr': 0.0, 'reproj': 0.0}@@ -1066,12 +1128,43 @@ def train_instant_hmr():losses = criterion(preds_fp32, batch)loss = losses['total_loss']loss_val = loss.item()++ skip_reason = Noneif math.isnan(loss_val) or math.isinf(loss_val):- print("⚠️ WARNING: NaN/Inf loss detected! Skipping step.")- continue- if loss_val > cfg.anomaly_loss_threshold:- print(f"⚠️ WARNING: anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold} — skipping step.")+ skip_reason = "NaN/Inf loss detected!"+ elif loss_val > cfg.anomaly_loss_threshold:+ skip_reason = f"anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold}"++ if skip_reason is not None:+ consecutive_skips += 1+ print(f"⚠️ WARNING: {skip_reason} — skipping step. "+ f"[{consecutive_skips} in a row] {format_loss_terms(losses)}")+ if consecutive_skips >= cfg.anomaly_skip_patience:+ if ema_rollbacks >= cfg.max_ema_rollbacks:+ print(f"\n🛑 Aborting: still stuck after {cfg.max_ema_rollbacks} EMA "+ f"rollback(s). The best RAW / EMA checkpoints on disk are "+ f"unaffected, and the ONNX export still runs off "+ f"best_student_model_v3.pth — inspect the loss terms above "+ f"before relaunching.")+ training_aborted = True+ break+ ema_rollbacks += 1+ print(f"\n🚑 {consecutive_skips} consecutive skipped steps: the raw weights "+ f"are outside the valid MHR region and CANNOT recover on their own, "+ f"because a skipped step updates neither the weights nor the LR.\n"+ f" Rolling back to the EMA weights "+ f"(attempt {ema_rollbacks}/{cfg.max_ema_rollbacks}), scaling the LR by "+ f"{cfg.anomaly_rollback_lr_decay} and clearing the optimiser moments.")+ model.load_state_dict(ema_model.module.state_dict())+ # The Adam moments are what drove the model out of the valid+ # region; carrying them over would walk straight back out.+ optimizer.state.clear()+ scale_lr(optimizer, cfg.anomaly_rollback_lr_decay, scheduler)+ print(f" LR is now {optimizer.param_groups[0]['lr']:.2e}", flush=True)+ consecutive_skips = 0continue++ consecutive_skips = 0scaler.scale(loss).backward()scaler.unscale_(optimizer)torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)@@ -1096,6 +1189,8 @@ def train_instant_hmr():pbar.set_postfix({'Tot': f"{loss.item():.3f}",'MHR': f"{losses.get('loss_mhr_joints', torch.tensor(0)).item():.3f}"})pbar.close()+ if training_aborted:+ breakavg_train = {k: v / max(nb, 1) for k, v in tm.items()}raw_val, raw_hmr, n_raw = run_validation(model, "raw", epoch)diff --git a/instanthmr_distill_train/train_levjepa_cloud.py b/instanthmr_distill_train/train_levjepa_cloud.pynew file mode 100644index 0000000..be00e98--- /dev/null+++ b/instanthmr_distill_train/train_levjepa_cloud.py@@ -0,0 +1,457 @@+#!/usr/bin/env python3+"""Frozen-LeVJEPA MHR regression on the FULL corpus, 90/10 train/val. No LoRA.++Differences from train_levjepa_lora.py (which this replaces for cluster runs):++ * NO LoRA. The encoder is frozen end to end, so there is no backward pass+ through 305 M parameters -- ~7x cheaper per clip and no adapter state.+ * ALL SIX datasets, not just the video ones. 90/10 random split over the+ combined pool with a fixed seed, so val is in-domain (use your own held-out+ samples for the real generalisation check).+ * Still images are encoded as T=1 clips, NOT by repeating the frame 16x.+ RoPE + tubelet_size=1 make the frame count free, so a still costs 197 tokens+ instead of 3137 for the same single label -- 16x less compute. Video and+ still batches alternate; both go through the same decoder.+ * OFFLINE-SAFE. --encoder_path loads a pre-downloaded snapshot directory, so+ the compute node never needs to reach huggingface.co. See the submit script+ for how to stage it.++WHY NO LoRA+-----------+The local frozen probe (10k Harmony4D clips, 20 epochs) reached 101.8 mm train+but plateaued at ~181 mm val on 3DPW with the val loss RISING after epoch 10 --+classic overfitting. The bottleneck was training-domain diversity, not encoder+capacity, so adding trainable encoder parameters was the wrong lever. Feeding+the model all 961 k crops instead of 10 k Harmony4D clips is the right one.++ATTENTION MODE -- DO NOT CHANGE+-------------------------------+attn_mode stays "block_causal" as shipped. Full attention raises no error and+silently returns worse features.+"""+from __future__ import annotations++import argparse+import contextlib+import io+import math+import os+import random+import sys+import time+from collections import defaultdict+from dataclasses import asdict, dataclass, field+from pathlib import Path++import numpy as np+import torch+import torch.nn as nn+from torch.utils.data import DataLoader, Dataset, Subset+from tqdm import tqdm++HERE = Path(__file__).resolve().parent+sys.path.insert(0, str(HERE))+with contextlib.redirect_stdout(io.StringIO()):+ import train_distill_mhr_only as base+ import train_levjepa_probe as probe++device = torch.device("cuda" if torch.cuda.is_available() else "cpu")+++@dataclass+class CloudConfig:+ data_root: str = str(base.DEFAULT_DATA_ROOT)+ mhr_model_path: str = str(base.PROJECT_ROOT / "checkpoints/mhr_model.pt")+ kp_regressor_path: str = str(base.PROJECT_ROOT / "assets/mhr_j127_to_kp70.npy")+ log_dir: str = str(base.PROJECT_ROOT / "runs/levjepa_cloud")++ encoder_id: str = probe.ENCODER_ID+ encoder_path: str | None = None # local snapshot dir -> no network needed+ image_size: int = 224+ clip_len: int = 16+ clip_stride: int = 4+ frame_stride: dict = field(default_factory=lambda: {+ "sam3d_gt_harmony4d": 1, "sam3d_gt_3dpw": 4, "sam3d_distill_mix": 1})+ # every dataset with usable temporal structure+ video_datasets: tuple = ("sam3d_gt_harmony4d", "sam3d_distill_mix", "sam3d_gt_3dpw")+ # every dataset without it -> encoded as T=1+ static_datasets: tuple = ("sam3d_gt_coco", "sam3d_gt_aic", "sam3d_gt_mpii")+ use_static: bool = True+ val_split: float = 0.1+ split_seed: int = 42+ max_video_clips: int | None = None+ max_static: int | None = None++ d_model: int = 512+ n_heads: int = 8+ n_decoder_layers: int = 4+ dropout: float = 0.1+ decoder_mode: str = "perframe"+ pose_dim: int = 136+ scale_dim: int = 68+ shape_dim: int = 45+ cam_dim: int = 3+ cliff_dim: int = 3+ num_joints: int = 70+ kp2d_bins: int = 96+ kp2d_range: float = 1.5++ @property+ def model_params_dim(self) -> int:+ return self.pose_dim + self.scale_dim++ batch_size: int = 8 # video clips per batch+ static_batch_mult: int = 8 # still batches are 16x cheaper -> bigger+ lr: float = 3e-4+ weight_decay: float = 1e-4+ epochs: int = 30+ warmup_pct: float = 0.05+ grad_clip: float = 1.0+ use_amp: bool = True+ num_workers: int = 8+ anomaly_loss_threshold: float = 100.0+ early_stop_patience: int = 8+ resume: bool = True+ max_hours: float = 92.0+ save_every_steps: int = 2000+ # Print a progress line every N batches. tqdm is disabled when stderr is not+ # a tty (i.e. every SLURM job) and an epoch here is HOURS long, so without+ # this the log stays empty until the first epoch ends and the run looks hung.+ log_every: int = 200++ w_pose: float = 1.0+ w_scale: float = 0.1+ w_shape: float = 1.0+ w_cam: float = 0.1+ w_keypoints3d: float = 2.0+ w_keypoints2d: float = 10.0+ w_simcc: float = 1.0+ w_3d_joints: float = 1e-3+ w_reproj: float = 0.01+ kp3d_warmup_steps: int = 2000++ native_mapping_ids: list = field(default_factory=lambda: base.DistillConfig().native_mapping_ids)+ mhr_mapping_ids: list = field(default_factory=lambda: base.DistillConfig().mhr_mapping_ids)+++class FrozenEncoder(nn.Module):+ def __init__(self, cfg: CloudConfig):+ super().__init__()+ from transformers import AutoModel+ src = cfg.encoder_path or cfg.encoder_id+ if cfg.encoder_path and not Path(cfg.encoder_path).exists():+ raise FileNotFoundError(+ f"--encoder_path '{cfg.encoder_path}' does not exist. Stage the "+ f"snapshot next to the data (see submit_train_levjepa_cloud_*.sh); "+ f"compute nodes usually cannot reach huggingface.co.")+ print(f" loading encoder from: {src}")+ self.net = AutoModel.from_pretrained(+ src, trust_remote_code=True, local_files_only=bool(cfg.encoder_path)+ ).eval().to(torch.float16).to(device)+ for p in self.net.parameters():+ p.requires_grad_(False)+ self.embed_dim = int(self.net.config.embed_dim)+ self.grid = int(self.net.config.img_size // self.net.config.patch_size)++ @torch.no_grad()+ def forward(self, images):+ """(B,T,3,H,W) -> (B,T,grid^2,C). T is free: 16 for clips, 1 for stills."""+ B, T = images.shape[:2]+ out = self.net(images.permute(0, 2, 1, 3, 4).to(torch.float16))+ tok = out[0] if isinstance(out, (tuple, list)) else getattr(out, "last_hidden_state", out)+ return tok[:, 1:].reshape(B, T, self.grid ** 2, -1).float()++++class CompactPairs:+ """List-like view over two numpy byte buffers, replacing a list of Paths.++ 961 k (Path, Path) tuples cost ~1.6 GB resident. Worse, CPython refcounts+ every one of those objects, so a forked DataLoader worker dirties the pages+ holding them and copy-on-write quietly turns into copy-everything -- with+ several workers that is tens of GB and a host-RAM OOM kill, which is exactly+ how job 23815712 died after 117 minutes.++ Two numpy arrays are single allocations with no per-element refcounts, so+ forks genuinely share them.+ """++ __slots__ = ("a", "b")++ def __init__(self, pairs):+ self.a = np.array([str(x[0]) for x in pairs], dtype="S")+ self.b = np.array([str(x[1]) for x in pairs], dtype="S")++ def __len__(self):+ return len(self.a)++ def __getitem__(self, i):+ return Path(self.a[i].decode()), Path(self.b[i].decode())+++def make_base_dataset(cfg):+ """One scan of the corpus, with the path list compacted, shared by everyone."""+ with contextlib.redirect_stdout(io.StringIO()):+ ds = base.SAM3DStudentDataset(cfg.data_root, augment=False,+ image_size=cfg.image_size,+ max_images=None, per_dataset_caps=None)+ n = len(ds.pairs)+ ds.pairs = CompactPairs(ds.pairs)+ print(f" corpus: {n:,} crops (path list compacted for fork-safety)")+ return ds+++class StaticFrameDataset(Dataset):+ """Still images as T=1 clips -- 197 tokens, not 3137. Same labels, same losses."""++ def __init__(self, cfg, datasets, max_items=None, base_ds=None):+ self.base = base_ds if base_ds is not None else make_base_dataset(cfg)+ keep = set(datasets)+ self.idx = [i for i, (_, n) in enumerate(self.base.pairs)+ if n.parent.parent.name in keep]+ if max_items and len(self.idx) > max_items:+ rng = np.random.RandomState(0)+ self.idx = [self.idx[i] for i in sorted(rng.choice(len(self.idx), max_items, False))]+ print(f" static {'/'.join(datasets)}: {len(self.idx):,} images as T=1 clips")++ def __len__(self):+ return len(self.idx)++ def __getitem__(self, i):+ f = self.base[self.idx[i]]+ out = {}+ for k, v in f.items():+ v = v if torch.is_tensor(v) else torch.as_tensor(np.asarray(v))+ out[k] = v.unsqueeze(0).float() # (1, ...) -> T=1+ return out+++def split_90_10(ds, val_frac, seed):+ n = len(ds)+ g = np.random.RandomState(seed)+ perm = g.permutation(n)+ nv = int(n * val_frac)+ return Subset(ds, perm[nv:].tolist()), Subset(ds, perm[:nv].tolist())+++def run_split(enc, dec, crit, cfg, mhr, loaders, opt=None, scaler=None, sched=None,+ gstep=0, tag="train", t_start=None):+ train = opt is not None+ dec.train(train)+ m, h, nb = defaultdict(float), defaultdict(float), 0+ stop = False+ for name, loader in loaders:+ if loader is None:+ continue+ bar = tqdm(loader, desc=f"{tag}:{name}", disable=not sys.stderr.isatty(),+ mininterval=60.0)+ nl = 0 # per-loader counter; nb runs across loaders+ for b in bar:+ b = {k: v.to(device, non_blocking=True) for k, v in b.items()}+ flat = probe.flatten_clip(b)+ patch = enc(b["image"])+ ctx = contextlib.nullcontext() if train else torch.no_grad()+ with ctx:+ with torch.amp.autocast(device_type="cuda", dtype=torch.float16,+ enabled=cfg.use_amp):+ preds16 = dec(patch, b["cliff_cond"])+ preds = {k: v.float() for k, v in preds16.items()}+ losses = crit(preds, flat)+ total = losses["total_loss"]+ if train:+ if not torch.isfinite(total) or total.item() > cfg.anomaly_loss_threshold:+ m["skipped"] += 1+ opt.zero_grad(set_to_none=True)+ continue+ opt.zero_grad(set_to_none=True)+ scaler.scale(total).backward()+ scaler.unscale_(opt)+ torch.nn.utils.clip_grad_norm_(dec.parameters(), cfg.grad_clip)+ scaler.step(opt); scaler.update()+ if sched.last_epoch < sched.total_steps - 1:+ sched.step()+ gstep += 1+ crit.set_step(gstep)+ for k, v in losses.items():+ m[k] += float(v.detach())+ for k, v in base.evaluate_hmr_batch(preds, flat, cfg, mhr).items():+ h[k] += v+ nb += 1+ nl += 1+ if train and cfg.log_every and nl % cfg.log_every == 0:+ el = (time.time() - t_start) / 60.0 if t_start else 0.0+ print(f" [{tag}:{name}] batch {nl}/{len(loader)} | {el:6.1f} min "+ f"| tot {m['total_loss']/nb:.4f} "+ f"| Mesh PA {h['Mesh_PA_MPJPE']/nb:.1f} mm "+ f"| lr {sched.get_last_lr()[0]:.2e}", flush=True)+ if train and t_start and (time.time() - t_start) / 3600.0 > cfg.max_hours:+ stop = True+ break+ bar.close()+ if stop:+ break+ return ({k: v / max(nb, 1) for k, v in m.items()},+ {k: v / max(nb, 1) for k, v in h.items()}, gstep, stop)+++def main():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--data_root", default=None)+ p.add_argument("--output_dir", default=None)+ p.add_argument("--mhr_model_path", default=None)+ p.add_argument("--kp_regressor_path", default=None)+ p.add_argument("--encoder_path", default=None,+ help="Local HF snapshot dir for the encoder (offline nodes).")+ p.add_argument("--clip_len", type=int, default=None)+ p.add_argument("--clip_stride", type=int, default=None)+ p.add_argument("--batch_size", type=int, default=None, help="video CLIPS per batch")+ p.add_argument("--static_batch_mult", type=int, default=None)+ p.add_argument("--epochs", type=int, default=None)+ p.add_argument("--lr", type=float, default=None)+ p.add_argument("--num_workers", type=int, default=None)+ p.add_argument("--max_video_clips", type=int, default=None)+ p.add_argument("--max_static", type=int, default=None)+ p.add_argument("--no-static", dest="no_static", action="store_true")+ p.add_argument("--val_split", type=float, default=None)+ p.add_argument("--max_hours", type=float, default=None)+ p.add_argument("--log_every", type=int, default=None,+ help="Batches between progress lines (0 disables).")+ p.add_argument("--no-resume", dest="no_resume", action="store_true")+ p.add_argument("--seed", type=int, default=0)+ p.add_argument("--self-test", dest="self_test", action="store_true")+ args = p.parse_args()++ torch.manual_seed(args.seed); np.random.seed(args.seed); random.seed(args.seed)+ cfg = CloudConfig()+ for k in ("data_root", "mhr_model_path", "kp_regressor_path", "encoder_path",+ "clip_len", "clip_stride", "batch_size", "static_batch_mult", "epochs",+ "lr", "num_workers", "max_video_clips", "max_static", "val_split",+ "max_hours", "log_every"):+ v = getattr(args, k)+ if v is not None:+ setattr(cfg, k, v)+ if args.output_dir: cfg.log_dir = args.output_dir+ if args.no_static: cfg.use_static = False+ if args.no_resume: cfg.resume = False++ print("=" * 74)+ print(f"encoder : {cfg.encoder_path or cfg.encoder_id} FROZEN (no LoRA)")+ print(f"data : video {cfg.video_datasets}")+ print(f" static {cfg.static_datasets if cfg.use_static else '(disabled)'} as T=1")+ print(f"split : {100*(1-cfg.val_split):.0f}/{100*cfg.val_split:.0f} random, seed {cfg.split_seed}")+ print(f"batch : {cfg.batch_size} clips (T={cfg.clip_len}) | "+ f"{cfg.batch_size*cfg.static_batch_mult} stills (T=1)")+ print(f"budget : {cfg.epochs} epochs, stop at {cfg.max_hours} h, resume={cfg.resume}")+ if torch.cuda.is_available():+ pr = torch.cuda.get_device_properties(0)+ print(f"GPU : {pr.name} {pr.total_memory/2**30:.1f} GiB")+ if pr.total_memory / 2**30 < 10: # 6 GB MIG slice, not a real card+ print("WARNING : that looks like a small MIG slice. perelha advertises "+ "gpu:nvidia_a30_1g.6gb:4 next to the A6000 / 6000-Ada, and a bare "+ "--gres=gpu:1 can land on one. Request the card explicitly, "+ "e.g. --gres=gpu:rtx_a6000:1")+ print("=" * 74)++ for pth, what in ((cfg.kp_regressor_path, "(70,127) keypoint regressor"),+ (cfg.mhr_model_path, "MHR TorchScript model")):+ if not Path(pth).exists():+ raise FileNotFoundError(f"{what} not found at '{pth}'. It must be rsynced "+ f"to the cluster alongside the training scripts.")+ W = np.load(cfg.kp_regressor_path)+ mhr = base.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W)+ enc = FrozenEncoder(cfg)+ dec = probe.ClipPoseDecoder(cfg, enc.embed_dim, enc.grid).to(device)+ crit = base.DistillationLoss(cfg, mhr).to(device)+ print(f" encoder {sum(q.numel() for q in enc.parameters())/1e6:.1f} M frozen | "+ f"decoder {sum(q.numel() for q in dec.parameters())/1e6:.2f} M trainable")++ shared = make_base_dataset(cfg)+ vid = probe.ClipDataset(cfg, cfg.video_datasets, max_clips=cfg.max_video_clips,+ base_ds=shared)+ vtr, vva = split_90_10(vid, cfg.val_split, cfg.split_seed)+ stat = (StaticFrameDataset(cfg, cfg.static_datasets, cfg.max_static, base_ds=shared)+ if cfg.use_static else None)+ if stat is not None:+ str_, sva = split_90_10(stat, cfg.val_split, cfg.split_seed)+ else:+ str_ = sva = None+ print(f" train: {len(vtr):,} clips + {len(str_) if str_ else 0:,} stills | "+ f"val: {len(vva):,} clips + {len(sva) if sva else 0:,} stills")++ # persistent_workers=False on purpose: this script runs FOUR loaders+ # (train/val x clips/stills). Persistent pools keep all four sets of workers+ # alive simultaneously, so the resident dataset is duplicated ~4x more than+ # necessary. Non-persistent workers exist only while their loader iterates.+ kw = dict(num_workers=cfg.num_workers, pin_memory=True)+ if cfg.num_workers > 0:+ kw.update(persistent_workers=False, prefetch_factor=2)+ mk = lambda d, bs, sh: (DataLoader(d, batch_size=bs, shuffle=sh, drop_last=sh, **kw)+ if d is not None and len(d) else None)+ sbs = cfg.batch_size * cfg.static_batch_mult+ train_loaders = [("clips", mk(vtr, cfg.batch_size, True)), ("stills", mk(str_, sbs, True))]+ val_loaders = [("clips", mk(vva, cfg.batch_size, False)), ("stills", mk(sva, sbs, False))]++ if args.self_test:+ for nm, l in train_loaders:+ if l is None: continue+ b = next(iter(l)); b = {k: v.to(device) for k, v in b.items()}+ f = enc(b["image"])+ o = dec(f, b["cliff_cond"])+ ls = crit({k: v.float() for k, v in o.items()}, probe.flatten_clip(b))+ print(f" {nm}: image{tuple(b['image'].shape)} -> feat{tuple(f.shape)} "+ f"-> mhr{tuple(o['mhr_params'].shape)} | loss {float(ls['total_loss']):.4f}")+ assert torch.isfinite(ls["total_loss"])+ print(" SELF-TEST PASSED"); return++ opt = torch.optim.AdamW(dec.parameters(), lr=cfg.lr, weight_decay=cfg.weight_decay)+ scaler = torch.amp.GradScaler(device="cuda", enabled=cfg.use_amp)+ spe = sum(len(l) for _, l in train_loaders if l is not None)+ sched = torch.optim.lr_scheduler.OneCycleLR(+ opt, max_lr=cfg.lr, epochs=cfg.epochs, steps_per_epoch=max(1, spe),+ pct_start=cfg.warmup_pct)++ os.makedirs(cfg.log_dir, exist_ok=True)+ last_p = os.path.join(cfg.log_dir, "levjepa_cloud_last.pth")+ best_p = os.path.join(cfg.log_dir, "levjepa_cloud_best.pth")+ start_ep, best, no_imp, gstep = 0, float("inf"), 0, 0+ if cfg.resume and os.path.exists(last_p):+ ck = torch.load(last_p, map_location=device, weights_only=False)+ dec.load_state_dict(ck["decoder"]); opt.load_state_dict(ck["optimizer"])+ scaler.load_state_dict(ck["scaler"]); sched.load_state_dict(ck["scheduler"])+ start_ep, best, gstep = ck["epoch"] + 1, ck.get("best", float("inf")), ck.get("gstep", 0)+ print(f" resumed from epoch {start_ep} (best val PA {best:.1f} mm)")++ def save(path, ep):+ torch.save({"decoder": dec.state_dict(), "optimizer": opt.state_dict(),+ "scaler": scaler.state_dict(), "scheduler": sched.state_dict(),+ "epoch": ep, "best": best, "gstep": gstep,+ "cfg": asdict(cfg)}, path)++ t0 = time.time()+ for ep in range(start_ep, cfg.epochs):+ tm, th, gstep, stop = run_split(enc, dec, crit, cfg, mhr, train_loaders,+ opt, scaler, sched, gstep, f"ep{ep+1}", t0)+ vm, vh, _, _ = run_split(enc, dec, crit, cfg, mhr, val_loaders, tag=f"ep{ep+1}val")+ el = (time.time() - t0) / 3600.0+ print(f"\nEpoch {ep+1}/{cfg.epochs} | {el:.2f} h | lr {sched.get_last_lr()[0]:.2e} "+ f"| skipped {int(tm.get('skipped',0))}")+ print(f" [train] tot {tm['total_loss']:.4f} | 3d {tm.get('loss_3d_native',0):.4f} "+ f"| 2d {tm.get('loss_2d_native',0):.4f} | Mesh PA {th['Mesh_PA_MPJPE']:.1f} mm")+ print(f" [val ] tot {vm['total_loss']:.4f} | Mesh PA {vh['Mesh_PA_MPJPE']:.1f} "+ f"| Mesh MPJPE {vh['Mesh_MPJPE']:.1f} | Mesh52 PA {vh['Mesh52_PA_MPJPE']:.1f} mm")+ save(last_p, ep)+ if vh["Mesh_PA_MPJPE"] < best:+ best, no_imp = vh["Mesh_PA_MPJPE"], 0+ save(best_p, ep); print(f" new best {best:.1f} mm")+ else:+ no_imp += 1+ if no_imp >= cfg.early_stop_patience:+ print(f" early stop after {no_imp} epochs without improvement"); break+ if stop:+ print(f" time budget {cfg.max_hours} h reached -- stopped cleanly"); break+ print(f"\nBest val Mesh PA-MPJPE: {best:.1f} mm ({(time.time()-t0)/3600:.2f} h)")+++if __name__ == "__main__":+ main()diff --git a/instanthmr_distill_train/train_levjepa_probe.py b/instanthmr_distill_train/train_levjepa_probe.pyindex c201982..99b4e32 100644--- a/instanthmr_distill_train/train_levjepa_probe.py+++ b/instanthmr_distill_train/train_levjepa_probe.py@@ -162,7 +162,17 @@ class ProbeConfig:# ============================================================# Clip index + dataset# ============================================================-TRACK_RE = re.compile(r"^(.*)_(\d{5})_p(\d+)$")+# Non-greedy prefix + open-ended index: hardcoding \d{5} silently dropped every+# family numbering frames with 6 digits (pilates/workout/dance inside+# sam3d_distill_mix -- ~25k frames of real video).+TRACK_RE = re.compile(r"^(.+?)_(\d+)_p(\d+)$")++# Families that sit inside a "video" folder but are STILL IMAGES. With the+# looser regex above they would otherwise chain consecutive COCO image ids into+# fake tracks, feeding the model 16 unrelated photos as one clip. They are also+# already in sam3d_gt_coco (8,514 of 10,656 crop ids identical), so skipping+# them here loses no data.+STATIC_IN_VIDEO_PREFIXES = ("train2014_COCO",)def build_clip_index(pairs, frame_stride, clip_len, clip_stride, datasets=None):@@ -175,6 +185,8 @@ def build_clip_index(pairs, frame_stride, clip_len, clip_stride, datasets=None):sub = npz_path.parent.parent.nameif datasets is not None and sub not in datasets:continue+ if npz_path.stem.startswith(STATIC_IN_VIDEO_PREFIXES):+ continuem = TRACK_RE.match(npz_path.stem)if not m:continue@@ -200,12 +212,19 @@ class ClipDataset(Dataset):here is byte-identical to a sample in the single-image pipeline."""- def __init__(self, cfg: ProbeConfig, datasets, max_clips=None, augment=False):+ def __init__(self, cfg: ProbeConfig, datasets, max_clips=None, augment=False,+ base_ds=None):self.cfg = cfg- with contextlib.redirect_stdout(io.StringIO()):- self.base = base.SAM3DStudentDataset(- cfg.data_root, augment=augment, image_size=cfg.image_size,- max_images=None, per_dataset_caps=None)+ # base_ds lets callers share ONE scan of the corpus. Building a second+ # SAM3DStudentDataset costs another ~1.6 GB of resident Path objects,+ # which every forked DataLoader worker then copies.+ if base_ds is not None:+ self.base = base_ds+ else:+ with contextlib.redirect_stdout(io.StringIO()):+ self.base = base.SAM3DStudentDataset(+ cfg.data_root, augment=augment, image_size=cfg.image_size,+ max_images=None, per_dataset_caps=None)self.clips = build_clip_index(self.base.pairs, cfg.frame_stride, cfg.clip_len, cfg.clip_stride,datasets=set(datasets))@@ -424,7 +443,10 @@ class ClipPoseDecoder(nn.Module):mem = self.feat_proj(patch_tokens) # (B,T,N,d)q = self.query_embed.expand(B * T, -1, -1)- q = q + self.temporal_embed.expand(B, -1, -1, -1).reshape(B * T, 1, -1)+ # slice, don't assume T == cfg.clip_len: still images are encoded as+ # T=1 clips, which costs 197 tokens instead of 3137 for the same single+ # label -- 16x cheaper than repeating the frame to fill the clip.+ q = q + self.temporal_embed[:, :T].expand(B, -1, -1, -1).reshape(B * T, 1, -1)q = q + self.cond_proj(cliff_cond).reshape(B * T, 1, -1)if self.mode == "perframe":diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.pyindex aaf5b53..c4365a8 100644--- a/tools/pth_to_onnx.py+++ b/tools/pth_to_onnx.py@@ -8,18 +8,25 @@ script that produced the checkpoint, plus the FP32 export recipe from notebookCell 15, so the graph I/O matches ``instanthmr/inference.py``:inputs : image (N, 3, 224, 224), cliff_cond (N, 3)- outputs: mhr_params, shape_params, cam_trans, joints_2d, joints_3d (in this order)+ outputs: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]Architecture selection-----------------------The student head changed between training scripts: ``train_distill.py``-regresses 2D joints directly (``head_2d_xy``), while the optimized/correctives-scripts predict them with a SimCC classification head (``head_2d_logits`` +-``kp2d_bin_centers``). Exporting a SimCC checkpoint through the old architecture-silently leaves ``head_2d_xy`` at its near-zero init, which collapses every 2D-keypoint onto the centre of the bbox. ``--arch auto`` (the default) picks the-right module by inspecting the checkpoint, and a missing-key check makes any-future mismatch a hard error instead of a silent one.+The student head changed twice across training scripts:++ * ``train_distill.py`` regresses 2D joints directly (``head_2d_xy``);+ * the optimized / correctives scripts predict them with a SimCC+ classification head (``head_2d_logits`` + ``kp2d_bin_centers``);+ * ``train_distill_mhr_only.py`` additionally **deletes the 3D head**+ (``head_3d``) — the 70 keypoints are derived from the MHR forward pass by+ the consumer, so its graph has FOUR outputs instead of five.++Exporting a SimCC checkpoint through the old architecture silently leaves+``head_2d_xy`` at its near-zero init, which collapses every 2D keypoint onto+the centre of the bbox. ``--arch auto`` (the default) picks the right module by+inspecting the checkpoint, and a missing-key check makes any future mismatch a+hard error instead of a silent one. The output list is read off the deploy+wrapper itself, so a 4- vs 5-output graph never has to be configured by hand.Usage-----@@ -28,8 +35,8 @@ Usage# or point at a specific checkpoint / outputpython tools/pth_to_onnx.py \- --ckpt instanthmr_distill_train/runs/distill_optimized_v2/best_student_model_ema.pth \- --output models/instanthmr.onnx+ --ckpt instanthmr_distill_train/runs/distill_mhr_only/distill_mhr_only/best_student_model_ema.pth \+ --output models/instanthmr_mhr_only.onnx"""import argparseimport importlib@@ -46,15 +53,26 @@ sys.path.insert(0, str(TRAIN_DIR))# arch name -> training module that defines the matching InstantHMRStudent.ARCH_MODULES = {+ "mhr_only": "train_distill_mhr_only", # SimCC 2D head, NO 3D head"correctives": "train_distill_optimized_correctives", # SimCC 2D head (head_2d_logits)"optimized": "train_distill_optimized", # SimCC 2D head, identical weights"legacy": "train_distill", # direct 2D regression (head_2d_xy)}+# Canonical ONNX output order. The MHR-only graph stops after `joints_2d`.+ALL_OUTPUT_KEYS = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]+def detect_arch(state: dict) -> str:- """Pick the architecture that matches the checkpoint's 2D head."""- if "head_2d_logits.weight" in state:+ """Pick the architecture that matches the checkpoint's prediction heads."""+ has_3d = any(k.startswith("head_3d.") for k in state)+ has_simcc = "head_2d_logits.weight" in state++ if has_simcc and not has_3d:+ # train_distill_mhr_only.py: the 3D coordinate head was removed and the+ # decoder shrank from 141 to 71 queries.+ return "mhr_only"+ if has_simcc:# `optimized` and `correctives` share identical parameters (they differ# only by an fp32 softmax cast), so either module loads such a# checkpoint; prefer the newer one.@@ -126,9 +144,22 @@ def main() -> None:deploy = T.HMRDeployWrapper(model).eval()# --- Export FP32 ONNX (Cell 15 recipe) ---- output_keys = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)dummy_cliff = torch.randn(1, 3, device=device)++ # Ask the wrapper how many tensors it returns instead of hard-coding five:+ # train_distill_mhr_only.py drops `joints_3d` from the tuple.+ with torch.no_grad():+ n_out = len(deploy(dummy_img, dummy_cliff))+ if n_out not in (4, 5):+ sys.exit(f"[error] unexpected deploy wrapper output count: {n_out} (expected 4 or 5)")+ output_keys = ALL_OUTPUT_KEYS[:n_out]+ print(f" graph outputs: {n_out} -> {output_keys}")+ if n_out == 4:+ print(" note: no 'joints_3d' output — the 70 keypoints are derived from the\n"+ " MHR forward pass at inference time (needs --mhr-model / --mhr-assets\n"+ " in demo.py).")+dynamic_axes = {"image": {0: "batch"}, "cliff_cond": {0: "batch"}}for k in output_keys:dynamic_axes[k] = {0: "batch"}