Python
3DPW GT: SMPL forward + H36M regressor, replacing raw jointPositions
mohamdev7f51828b467c
diff --git a/benchmark/README.md b/benchmark/README.mdindex bd340f9..b79cce3 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -7,16 +7,18 @@ to a credible table — one 3D, one 2D:| Benchmark | What it measures | Data cost | Status ||---|---|---|---|-| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT, free licence form | ready — **but the test split is contaminated, see below** |+| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT + 500 MB SMPL | ready — published-protocol GT; **check contamination below** || **COCO** val2017 | OKS AP / PCK, 2D keypoint localisation in the wild | 820 MB, direct download, no registration | ready — clean |## Why these two**3DPW** is the benchmark essentially every monocular HMR paper reports, so it-is the one that makes your numbers comparable. It is also unusually cheap here:-the GT pickles store `jointPositions` (24 SMPL joints in world coordinates)-directly, so **no SMPL model download or body-model forward pass is needed** —-which is what makes most 3DPW harnesses painful to set up.+is the one that makes your numbers comparable. The GT pickles store+`jointPositions` (24 SMPL joints in world coordinates) directly, which is+tempting and wrong — published numbers use SMPL run *forward* on the GT+`poses`/`betas` with the Human3.6M regressor applied to the vertices. The two+references are **46 mm apart in J12 PA-MPJPE**, so the harness builds the real+one once with `make_3dpw_gt.py` and scores against that.**COCO val2017** needs no registration at all and the joint mapping is free:MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.@@ -47,6 +49,27 @@ python benchmark/download.py --check-3dpw \--image-root /path/to/3DPW/imageFiles```+Then build the ground truth. This needs two files that cannot be redistributed+here — put both in `benchmark/data/smpl/` (gitignored):++| file | where |+|---|---|+| `SMPL_{MALE,FEMALE}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so the gendered pair is required. |+| `J_regressor_h36m.npy` | `visiondata.cis.upenn.edu/spin/data.tar.gz` (SPIN); also ships inside HMR2.0 / 4D-Humans |++```bash+python benchmark/make_3dpw_gt.py \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --smpl-dir benchmark/data/smpl --check+```++That writes `<3DPW root>/gt_h36m/{test,validation,train}.npz` — 17 H36M joints+per person-frame in the same world frame as `jointPositions`, 14 MB for all+three splits. `--check` additionally regresses the SMPL kinematic joints and+compares them to `jointPositions`: **0.001 mm max over 74,620 person-frames**,+which is what certifies the forward pass, the gendered model choice and the+world frame all the way through.+## Running```bash@@ -61,6 +84,12 @@ python benchmark/eval_3dpw.py \--sequence-dir /path/to/3DPW/sequenceFiles \--image-root /path/to/3DPW/imageFiles \--exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations++# same thing for a training checkpoint instead of an exported graph+python benchmark/eval_3dpw_ckpt.py \+ --ckpt instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --image-root /path/to/3DPW/imageFiles --split test```Both write a JSON report to `benchmark/results/`.@@ -82,51 +111,66 @@ comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits noper-keypoint confidence, so every keypoint is scored 1.0 — read AP next to theconfidence-free `mean_OKS` and `PCK` numbers.-### The GT-joint caveat — READ THIS BEFORE QUOTING A 3DPW NUMBER+### The ground truth — what it is, and what it used to be-**The ground truth this harness uses is not the one 3DPW papers report on.**-`benchlib/threedpw.py` reads `jointPositions` straight from the sequence-pickles: the 24 **SMPL kinematic-tree** joints. Every published 3DPW number —+The harness scores against the **published-protocol GT**: gendered SMPL run+forward on each frame's `poses`/`betas`/`trans`, then `J_regressor_h36m.npy`+applied to the 6890 vertices. This is the reference every 3DPW paper uses —NLF's included ("obtained through the same Human3.6M-style joint regressor that-all prior works use") — instead runs SMPL *forward* on the GT `poses` / `betas`-and applies the H36M joint regressor to the resulting 6890 vertices. Those are-different landmarks, most visibly at the shoulders and hips, which is 4 of the-12 joints in J12.--That is what makes this harness cheap to set up (see "Why these two" above) and-it is fine for *tracking a run*. It is **not** comparable to a published table.--Fixing it needs two files that are not in this repo:+all prior works use"). `benchmark/make_3dpw_gt.py` precomputes it; `--gt+jointpositions` restores the pickles' raw `jointPositions` field, which is what+this harness used before and is **not** comparable to any published table.-| file | where |-|---|---|-| `SMPL_{MALE,FEMALE,NEUTRAL}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so get the gendered pair. |-| `J_regressor_h36m.npy` | ships inside SPIN / HMR2.0 / 4D-Humans |+The difference is not cosmetic. Over 1,437 test person-frames, the same+landmarks under the two references:-Then: `V = SMPL(poses, betas)` per frame, `J14 = J_regressor @ V`, and fit the-existing `--fit-adapter` linear MHR70 -> J14 map on 3DPW **train** against that-GT before reporting on test. A forward pass, not a fit.--`v_template_clothed` in the pickles cannot substitute — it is a static T-pose-surface with no blend weights, no pose blendshapes and no joint regressor, so it-cannot be posed.--This does **not** affect `instanthmr_distill_train/val3dpw.py`, which uses the-same GT for checkpoint selection: selection only needs a consistent metric, not-a comparable one.+| | raw | root-relative | after Procrustes |+|---|---|---|---|+| J14 mean | 64.2 mm | 86.3 mm | 47.2 mm |+| J12 mean | 63.6 mm | 88.1 mm | 46.0 mm |+| right hip | 154.5 mm | 86.8 mm | 114.5 mm |+| head | 116.3 mm | 42.0 mm | 62.2 mm |++The hips dominate, because the two conventions disagree about where a hip is:+H36M's are marker-derived and **292 mm apart**, SMPL's kinematic ones sit near+the femoral heads, **120 mm apart**. Shoulder width goes the other way (303 vs+387 mm), and H36M's `head` is the top of the skull where SMPL's is inside it+(neck→head 192 vs 100 mm).++Two details worth keeping straight:++- **`jointPositions` is the kinematic joint, not the regressed one.** Comparing+ `J_regressor @ posed vertices` to it gives a 4.5 mm residual and looks like a+ bug in the forward pass. The joints that match are the ones carried by the+ global transforms. `make_3dpw_gt.py --check` compares those, and gets 0.001 mm.+- **`betas` is 300-dim for 48 of the 87 subject tracks and 10-dim for the other+ 39, and entries 10: are all exactly zero.** So `--num-betas 10`, the published+ protocol, is also lossless here.++`v_template_clothed` in the pickles is not a substitute for any of this — it is+a static T-pose surface with no blend weights and no pose blendshapes, so it+cannot be posed. Using it makes the residual worse (17.9 mm), not better.++Selection inside `instanthmr_distill_train/val3dpw.py` follows the same default,+so **3DPW numbers from before and after this change are not comparable** — the+new GT is several mm harsher. The cache has to be rsynced next to the cluster's+`sequenceFiles/` or the trainer raises on startup.### The joint-convention caveat — do not skip this-InstantHMR predicts MHR70 joints; 3DPW's GT is SMPL. The two rigs place several-landmarks differently, which puts a floor under the error that has nothing to do-with model quality. The harness therefore reports three rows:--- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` are the- two joints where the rigs genuinely disagree (MHR has no head-top; `nose` is- substituted), so this row carries a systematic penalty.-- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles), where- both rigs mean the same anatomical point. **This is the cleanest measure of- the model.**+InstantHMR predicts MHR70 joints; the GT is now in the H36M convention. The two+place several landmarks differently, which puts a floor under the error that has+nothing to do with model quality. The harness therefore reports three rows:++- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` carry+ the largest systematic penalty (MHR has no head-top; `nose` is substituted).+- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles). Against+ `jointpositions` this was the clean row, because MHR and SMPL agree on those+ landmarks. Against the H36M GT it is **not** clean: the hips alone move 292 vs+ 120 mm apart between the conventions. Read it as a consistent tracking metric,+ not as a rig-free measure of the model.+- **J14+adapter** is therefore the row to compare against a published table,+ not an optional extra.- **J14+adapter** *(optional)* — a linear MHR70 → SMPL-J14 joint regressor,fitted on 3DPW **train**, that absorbs the rig offset. Standard practice whena method's joint convention differs from the benchmark's. Fit it once, thendiff --git a/benchmark/benchlib/joints.py b/benchmark/benchlib/joints.pyindex f4e08dd..331d355 100644--- a/benchmark/benchlib/joints.py+++ b/benchmark/benchlib/joints.py@@ -60,6 +60,25 @@ SMPL24_NAMES = []SMPL = {n: i for i, n in enumerate(SMPL24_NAMES)}+# --------------------------------------------------------------------------+# Human3.6M 17 joints — the rows of ``J_regressor_h36m.npy``+# --------------------------------------------------------------------------+# This is the GT every published 3DPW number is measured against: SMPL run+# forward on the sequence's poses/betas, then this regressor applied to the+# 6890 vertices. The row order below was read off the data, not a table — each+# row was matched to its nearest SMPL kinematic joint on a real sequence — and+# it reproduces SPIN's ``H36M_TO_J14 = [6,5,4,1,2,3,16,15,14,11,12,13,8,10]``+# exactly. Note it is *left-first* in the legs and arms, which is the opposite+# of how H36M's own raw skeleton is usually tabulated.+H36M17_NAMES = [+ "pelvis", "left_hip", "left_knee", "left_ankle",+ "right_hip", "right_knee", "right_ankle",+ "spine", "neck", "nose", "head",+ "left_shoulder", "left_elbow", "left_wrist",+ "right_shoulder", "right_elbow", "right_wrist",+]+H36M = {n: i for i, n in enumerate(H36M17_NAMES)}+# --------------------------------------------------------------------------# 3D evaluation joint sets# --------------------------------------------------------------------------@@ -80,17 +99,29 @@ _MHR_FOR_J14 = dict(MHR, head=MHR["nose"])J14_FROM_MHR70 = np.array([_MHR_FOR_J14[n] for n in J14_NAMES], dtype=np.int64)J14_FROM_SMPL24 = np.array([SMPL[n] for n in J14_NAMES], dtype=np.int64)+J14_FROM_H36M17 = np.array([H36M[n] for n in J14_NAMES], dtype=np.int64)J12_FROM_MHR70 = J14_FROM_MHR70[:12]J12_FROM_SMPL24 = J14_FROM_SMPL24[:12]+J12_FROM_H36M17 = J14_FROM_H36M17[:12]# Pelvis for root alignment: the mid-hip, computed the same way on both sides.J14_HIP_IDX = (J14_NAMES.index("left_hip"), J14_NAMES.index("right_hip"))JOINT_SETS = {- "J14": dict(names=J14_NAMES, mhr=J14_FROM_MHR70, smpl=J14_FROM_SMPL24),- "J12": dict(names=J12_NAMES, mhr=J12_FROM_MHR70, smpl=J12_FROM_SMPL24),+ "J14": dict(names=J14_NAMES, mhr=J14_FROM_MHR70,+ smpl=J14_FROM_SMPL24, h36m=J14_FROM_H36M17),+ "J12": dict(names=J12_NAMES, mhr=J12_FROM_MHR70,+ smpl=J12_FROM_SMPL24, h36m=J12_FROM_H36M17),}+# ``benchlib.threedpw.build_samples`` returns GT in one of two layouts; this+# picks the matching index array out of a joint set.+GT_LAYOUTS = {"h36m": "h36m", "jointpositions": "smpl"}+++def gt_index(spec: dict, gt: str) -> np.ndarray:+ return spec[GT_LAYOUTS[gt]]+def pelvis(joints: np.ndarray) -> np.ndarray:"""Mid-hip root for a (..., J, 3) array indexed in a J14/J12 joint set."""diff --git a/benchmark/benchlib/threedpw.py b/benchmark/benchlib/threedpw.pyindex d5a4417..8ecd1c1 100644--- a/benchmark/benchlib/threedpw.py+++ b/benchmark/benchlib/threedpw.py@@ -9,8 +9,17 @@ useful fields for us:campose_valid list[P] of (F,) 1 where the camera pose is trustworthygenders list[P]-Because ``jointPositions`` is stored directly, evaluating 3D joint error needs-**no SMPL model download** — we never have to re-run the body model.+``jointPositions`` holds the 24 SMPL *kinematic-tree* joints, which is not the+ground truth 3DPW papers report on: those run SMPL forward on ``poses``/+``betas`` and apply the Human3.6M regressor to the resulting vertices. That+reference is precomputed once by ``benchmark/make_3dpw_gt.py`` into+``<3DPW root>/gt_h36m/<split>.npz`` — 17 joints per person-frame in the same+world frame — and is what ``build_samples`` returns by default. Pass+``gt="jointpositions"`` to get the old, non-comparable GT back.++Person boxes and the off-screen filter keep using ``jointPositions`` either+way, so switching the GT changes only the numbers being compared, never which+frames are evaluated or how they are cropped."""from __future__ import annotations@@ -50,10 +59,32 @@ def bbox_from_points(uv: np.ndarray, scale: float) -> np.ndarray:dtype=np.float32)+def gt_h36m_path(sequence_dir: str | Path, split: str,+ gt_dir: str | Path | None = None) -> Path:+ """Where the regressed GT lives: next to ``sequenceFiles``, not in the repo."""+ base = Path(gt_dir) if gt_dir else Path(sequence_dir).parent / "gt_h36m"+ return base / f"{split}.npz"+++def load_gt_h36m(sequence_dir: str | Path, split: str,+ gt_dir: str | Path | None = None) -> dict:+ path = gt_h36m_path(sequence_dir, split, gt_dir)+ if not path.is_file():+ raise FileNotFoundError(+ f"regressed 3DPW ground truth not found: {path}\n"+ "Build it with benchmark/make_3dpw_gt.py, or pass "+ "gt='jointpositions' to fall back to the raw (non-comparable) "+ "SMPL kinematic joints.")+ with np.load(path) as z:+ return {k: z[k] for k in z.files if "|" in k}++def build_samples(sequence_dir: str | Path, image_root: str | Path,split: str = "test", stride: int = 1,bbox_scale: float = 1.2,- min_visible_frac: float = 0.6):+ min_visible_frac: float = 0.6,+ gt: str = "h36m",+ gt_dir: str | Path | None = None):"""Enumerate every evaluatable (frame, person) in a 3DPW split.Args:@@ -66,10 +97,19 @@ def build_samples(sequence_dir: str | Path, image_root: str | Path,projected joints fall inside the image — they are mostly out offrame, and a crop around them is meaningless.+ gt: ``h36m`` for the published-protocol GT (17 regressed joints,+ requires the cache from ``make_3dpw_gt.py``); ``jointpositions``+ for the raw 24 SMPL kinematic joints in the pickles.+ gt_dir: override the cache location.+Returns:- ``(samples, gt_joints)`` — a list of dicts and an (N, 24, 3) array of- camera-space GT joints in metres, index-aligned.+ ``(samples, gt_joints)`` — a list of dicts and an (N, 17, 3) array+ (``h36m``) or (N, 24, 3) array (``jointpositions``) of camera-space GT+ joints in metres, index-aligned."""+ if gt not in ("h36m", "jointpositions"):+ raise ValueError(f"unknown gt source: {gt}")+ table = load_gt_h36m(sequence_dir, split, gt_dir) if gt == "h36m" else Noneseq_dir = Path(sequence_dir) / splitif not seq_dir.is_dir():raise FileNotFoundError(f"no such 3DPW split directory: {seq_dir}")@@ -87,6 +127,10 @@ def build_samples(sequence_dir: str | Path, image_root: str | Path,for pid, jp in enumerate(seq["jointPositions"]):joints_w = np.asarray(jp, dtype=np.float64).reshape(-1, 24, 3)+ # The box always comes from the SMPL joints; only the scored GT+ # changes, so the two sources are evaluated on identical crops.+ gt_w = (np.asarray(table[f"{name}|{pid}"], dtype=np.float64)+ if table is not None else joints_w)valid = np.asarray(seq["campose_valid"][pid]).astype(bool)for f in range(0, min(num_frames, joints_w.shape[0]), stride):@@ -106,7 +150,7 @@ def build_samples(sequence_dir: str | Path, image_root: str | Path,sequence=name, person=pid, frame=f,gender=str(seq["genders"][pid]),))- gts.append(joints_cam)+ gts.append(gt_w[f] @ T[:3, :3].T + T[:3, 3])if not samples:raise RuntimeError(f"no valid samples found in {seq_dir}")diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.pyindex 33f770f..71d37dc 100644--- a/benchmark/eval_3dpw.py+++ b/benchmark/eval_3dpw.py@@ -1,11 +1,14 @@#!/usr/bin/env python3"""3DPW test-set evaluation for InstantHMR (MPJPE / PA-MPJPE / PCK / AUC).-The GT comes straight from the 3DPW pickles' ``jointPositions`` field, so no-SMPL model download is needed. Predicted MHR70 joints are mapped onto the-SMPL joint convention by name; see ``benchlib/joints.py`` for the caveat about-``neck`` and ``head``, which are the only two joints where the two rigs disagree-on the underlying anatomy.+By default the GT is the published protocol: SMPL run forward on the sequence's+poses/betas and the Human3.6M regressor applied to the vertices, precomputed by+``benchmark/make_3dpw_gt.py``. ``--gt jointpositions`` falls back to the raw+SMPL kinematic joints in the pickles, which is what this harness used before+and is not comparable to any published table. Predicted MHR70 joints are mapped+onto the joint set by name; see ``benchlib/joints.py`` for the caveat about+``neck`` and ``head``, which are the only two joints where the rigs disagree on+the underlying anatomy.Usage:python benchmark/eval_3dpw.py \@@ -47,6 +50,12 @@ def parse_args():help="evaluate every Nth frame (1 = full protocol)")p.add_argument("--bbox-scale", type=float, default=1.2,help="padding on the projected-GT-joint person box")+ p.add_argument("--gt", default="h36m", choices=["h36m", "jointpositions"],+ help="h36m: SMPL forward + H36M regressor, the published "+ "protocol. jointpositions: the raw pickle field.")+ p.add_argument("--gt-dir", default=None,+ help="where make_3dpw_gt.py wrote its npz "+ "(default: <sequence-dir>/../gt_h36m)")p.add_argument("--device", default="cuda")p.add_argument("--mhr-script", default=None)p.add_argument("--num-workers", type=int, default=6)@@ -64,11 +73,12 @@ def parse_args():return p.parse_args()-def evaluate(pred_mhr: np.ndarray, gt_smpl: np.ndarray, set_name: str) -> dict:- """Metrics for one joint set. Inputs are (N, 70, 3) and (N, 24, 3) in mm."""+def evaluate(pred_mhr: np.ndarray, gt_raw: np.ndarray, set_name: str,+ gt_src: str) -> dict:+ """Metrics for one joint set. Inputs are (N, 70, 3) and (N, J, 3) in mm."""spec = J.JOINT_SETS[set_name]pred = pred_mhr[:, spec["mhr"], :]- gt = gt_smpl[:, spec["smpl"], :]+ gt = gt_raw[:, J.gt_index(spec, gt_src), :]root_p, root_g = J.pelvis(pred), J.pelvis(gt)e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))@@ -89,14 +99,23 @@ def evaluate(pred_mhr: np.ndarray, gt_smpl: np.ndarray, set_name: str) -> dict:)-def fit_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray) -> np.ndarray:+def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,+ gt_src: str) -> np.ndarray:"""Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.Fitted on centred poses and used without a bias, so the mapping staysequivariant to rotation and translation — it only removes the systematicdifference in where the two rigs place each landmark.++ Centring must happen in float64. Doing it in float32 leaves a ~0.04 mm+ residual in the all-ones direction, which is exactly the direction the+ centring annihilates; ``lstsq`` then sees a singular value of 3e-7 relative+ instead of zero, keeps it, and loads that null direction with weights of+ +-2500. The fitted map still scores ~37 mm in sample in float64 and blows up+ to 430 mm in float32."""- gt = gt_smpl[:, J.J14_FROM_SMPL24, :]+ gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :].astype(np.float64)+ pred_mhr = pred_mhr.astype(np.float64)X = pred_mhr - pred_mhr.mean(axis=1, keepdims=True) # (N, 70, 3)Y = gt - gt.mean(axis=1, keepdims=True) # (N, 14, 3)# Flatten samples and coordinates into the row dimension.@@ -106,10 +125,14 @@ def fit_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray) -> np.ndarray:return W.T # (14, 70)-def evaluate_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray,- W: np.ndarray) -> dict:- pred = np.einsum("jk,nkc->njc", W, pred_mhr)- gt = gt_smpl[:, J.J14_FROM_SMPL24, :]+def evaluate_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,+ W: np.ndarray, gt_src: str) -> dict:+ # W was fitted on centred keypoints, so apply it to centred keypoints. Both+ # metrics below are translation-invariant, so this costs nothing and makes+ # the result independent of whatever W does to the all-ones direction.+ centred = pred_mhr - pred_mhr.mean(axis=1, keepdims=True)+ pred = np.einsum("jk,nkc->njc", W, centred)+ gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :]root_p, root_g = J.pelvis(pred), J.pelvis(gt)e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))e_pa = M.pa_mpjpe(pred, gt)@@ -131,6 +154,7 @@ def main():samples, gt_joints = P.build_samples(args.sequence_dir, args.image_root, split=args.split,stride=args.stride, bbox_scale=args.bbox_scale,+ gt=args.gt, gt_dir=args.gt_dir,)if args.limit:samples, gt_joints = samples[:args.limit], gt_joints[:args.limit]@@ -155,12 +179,12 @@ def main():W_adapt = np.load(args.adapter)["W"] if args.adapter else Nonedef _rows(pm, gm):- rows = [evaluate(pm, gm, s) for s in ("J14", "J12")]+ rows = [evaluate(pm, gm, s, args.gt) for s in ("J14", "J12")]if W_adapt is not None:- rows.append(evaluate_adapter(pm, gm, W_adapt))+ rows.append(evaluate_adapter(pm, gm, W_adapt, args.gt))return rowsreport = dict(- model=str(args.onnx), dataset="3DPW", split=args.split,+ model=str(args.onnx), dataset="3DPW", split=args.split, gt=args.gt,stride=args.stride, bbox_scale=args.bbox_scale,num_sequences=len(seqs), sequences=seqs,num_samples=int(pred_mhr.shape[0]),@@ -185,14 +209,15 @@ def main():report["results_seen_frames"] = _rows(pred_mhr[is_seen], gt_smpl[is_seen])if args.fit_adapter:- W = fit_adapter(pred_mhr, gt_smpl)+ W = fit_adapter(pred_mhr, gt_smpl, args.gt)Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)- np.savez(args.fit_adapter, W=W, split=args.split)+ np.savez(args.fit_adapter, W=W, split=args.split, gt=args.gt)print(f"[3DPW] fitted joint adapter on '{args.split}' -> {args.fit_adapter}")# Per-sequence PA-MPJPE on the headline joint set.spec = J.JOINT_SETS["J14"]- e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :], gt_smpl[:, spec["smpl"], :]).mean(axis=1)+ e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :],+ gt_smpl[:, J.gt_index(spec, args.gt), :]).mean(axis=1)kept = [s for s, k in zip(samples, ok) if k]per_seq = {}for name in seqs:@@ -214,7 +239,7 @@ def main():print("\n" + "=" * 60)print(f"3DPW {args.split} | {report['num_samples']} person-frames "- f"| {len(seqs)} sequences (mm)")+ f"| {len(seqs)} sequences | GT: {args.gt} (mm)")print("=" * 60)_table("ALL frames:", report["results"])if "results_unseen_frames" in report:diff --git a/benchmark/eval_3dpw_ckpt.py b/benchmark/eval_3dpw_ckpt.pynew file mode 100644index 0000000..37f6749--- /dev/null+++ b/benchmark/eval_3dpw_ckpt.py@@ -0,0 +1,173 @@+#!/usr/bin/env python3+"""3DPW evaluation for a training checkpoint (.pth), not an ONNX export.++``eval_3dpw.py`` scores exported graphs; this scores checkpoints straight out of+``runs/``. The crop and the MHR forward pass come from ``val3dpw.py`` (which was+cross-validated against ``eval_3dpw.py``) and the metrics are ``eval_3dpw.py``'s+own functions, so the two scripts report the same numbers for the same weights.++ # headline: published-protocol GT on the test split+ python benchmark/eval_3dpw_ckpt.py --ckpt runs/*/best_student_model_v3.pth \+ --sequence-dir .../sequenceFiles --image-root .../imageFiles \+ --split test++ # the MHR->J14 adapter, fitted on train and applied to test+ python benchmark/eval_3dpw_ckpt.py --ckpt <ckpt> ... --split train \+ --stride 10 --fit-adapter benchmark/results/adapter_j14_h36m.npz+"""++from __future__ import annotations++import argparse+import json+import sys+from pathlib import Path++import numpy as np+import torch++ROOT = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))+sys.path.insert(0, str(ROOT / "benchmark"))++import eval_3dpw as E # noqa: E402+import train_distill_mhr_only as T # noqa: E402+import val3dpw # noqa: E402+from benchlib import joints as J # noqa: E402++MM = 1000.0+++def parse_args():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--ckpt", nargs="+", required=True)+ p.add_argument("--sequence-dir", required=True)+ p.add_argument("--image-root", required=True)+ p.add_argument("--split", default="test")+ p.add_argument("--stride", type=int, default=1)+ p.add_argument("--gt", nargs="+", default=["h36m"],+ choices=["h36m", "jointpositions"])+ p.add_argument("--gt-dir", default=None)+ p.add_argument("--adapter", default=None,+ help="npz with a fitted MHR70->J14 regressor to also report")+ p.add_argument("--fit-adapter", default=None,+ help="fit that regressor on THIS split and write it here "+ "(run on train, never on test)")+ p.add_argument("--backbone", default="repvit_m2_3")+ p.add_argument("--batch-size", type=int, default=64)+ p.add_argument("--num-workers", type=int, default=6)+ p.add_argument("--device", default="cuda")+ p.add_argument("--out", default=None)+ return p.parse_args()++[email protected]_grad()+def predict(model, loader, mhr_module, device, use_amp: bool = True):+ """(N, 70, 3) predicted keypoints in mm, plus the indices they came from.++ The indices let the same forward pass be scored against more than one GT+ source: the sample list is identical whichever GT ``build_samples`` returns,+ only the joint values differ.+ """+ model.eval()+ preds, keeps, n_dropped, base = [], [], 0, 0+ for b in loader:+ img = b["image"].to(device, non_blocking=True)+ cc = b["cliff_cond"].to(device, non_blocking=True)+ with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):+ out = model(img, cc)+ kp = mhr_module.get_native_keypoints(out["mhr_params"].float(),+ out["shape_params"].float())+ # Same guard as val3dpw.evaluate: a non-finite prediction makes the+ # Procrustes SVD raise rather than return a large error.+ finite = torch.isfinite(kp).flatten(1).all(dim=1).cpu()+ keep = b["ok"].bool() & finite+ n_dropped += int((b["ok"].bool() & ~finite).sum())+ if bool(keep.any()):+ preds.append(kp[keep.to(kp.device)].float().cpu().numpy())+ keeps.append(base + np.nonzero(keep.numpy())[0])+ base += int(keep.shape[0])+ return (np.concatenate(preds) * MM, np.concatenate(keeps), n_dropped)+++def main():+ args = parse_args()+ device = torch.device(args.device)+ cfg = T.DistillConfig(backbone=args.backbone)+ mhr = T.MHRForwardPass(cfg.mhr_model_path, device,+ kp_regressor=np.load(cfg.kp_regressor_path))+ W_adapt = W_adapt_gt = None+ if args.adapter:+ ad = np.load(args.adapter)+ W_adapt, W_adapt_gt = ad["W"], str(ad["gt"]) if "gt" in ad else "unknown"++ # One dataset per GT source: same samples and same crops, different joint+ # values, so a single forward pass scores against all of them.+ sets = {}+ for gt in args.gt:+ ds = val3dpw.ThreeDPWValSet(args.sequence_dir, args.image_root,+ split=args.split, stride=args.stride,+ gt=gt, gt_dir=args.gt_dir)+ sets[gt] = ds+ print(f"[3DPW] {args.split} / GT={gt}: {len(ds):,} person-frames "+ f"(stride {args.stride})", flush=True)+ first = sets[args.gt[0]]+ loader = torch.utils.data.DataLoader(+ first, batch_size=args.batch_size, shuffle=False,+ num_workers=args.num_workers, pin_memory=True)++ report = {}+ hdr = (f"\n {'run':<8s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "+ f"{'J12 MPJPE':>10s} {'J14 PA':>8s} {'J14 MPJPE':>10s}"+ + (f" {'J14+ad PA':>10s}" if W_adapt is not None else ""))+ print(hdr)++ for ck_path in args.ckpt:+ model = T.InstantHMRStudent(cfg, pretrained=False).to(device)+ ck = torch.load(ck_path, map_location="cpu", weights_only=False)+ model.load_state_dict(ck["model_state_dict"])+ pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp)+ del model+ torch.cuda.empty_cache()++ for gt, ds in sets.items():+ gt_arr = np.asarray(ds.gt, dtype=np.float64)[idx] * MM+ rows = {s: E.evaluate(pred, gt_arr, s, gt) for s in ("J14", "J12")}+ # An adapter absorbs the offset between the MHR rig and one specific+ # GT convention, so scoring it against the other one is meaningless.+ if W_adapt is not None and W_adapt_gt == gt:+ rows["J14+adapter"] = E.evaluate_adapter(pred, gt_arr, W_adapt, gt)+ report[f"{ck_path}|{gt}"] = dict(+ ckpt=ck_path, gt=gt, split=args.split, stride=args.stride,+ epoch=ck.get("epoch"), source=ck.get("source"),+ n=int(pred.shape[0]), n_dropped=n_dropped,+ results=list(rows.values()))++ run = Path(ck_path).parts[-3]+ name = Path(ck_path).stem.replace("best_student_model_", "")+ line = (f" {run:<8s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "+ f"{rows['J12']['PA_MPJPE_mm']:8.2f} {rows['J12']['MPJPE_mm']:10.2f} "+ f"{rows['J14']['PA_MPJPE_mm']:8.2f} {rows['J14']['MPJPE_mm']:10.2f}")+ if W_adapt is not None:+ line += (f" {rows['J14+adapter']['PA_MPJPE_mm']:10.2f}"+ if "J14+adapter" in rows+ else f" {'n/a':>10s}")+ print(line + (f" ({n_dropped} dropped)" if n_dropped else ""), flush=True)++ if args.fit_adapter:+ if args.split == "test":+ raise SystemExit("refusing to fit the adapter on the test split")+ Wf = E.fit_adapter(pred, gt_arr, gt)+ Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)+ np.savez(args.fit_adapter, W=Wf, split=args.split, gt=gt, ckpt=ck_path)+ print(f" fitted adapter on '{args.split}' -> {args.fit_adapter}")++ if args.out:+ Path(args.out).parent.mkdir(parents=True, exist_ok=True)+ Path(args.out).write_text(json.dumps(report, indent=2))+ print(f"\nwritten to {args.out}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/make_3dpw_gt.py b/benchmark/make_3dpw_gt.pynew file mode 100644index 0000000..2da8176--- /dev/null+++ b/benchmark/make_3dpw_gt.py@@ -0,0 +1,237 @@+#!/usr/bin/env python3+"""Build 3DPW ground truth the way published papers do: SMPL forward + H36M regressor.++The sequence pickles ship ``jointPositions`` — the 24 SMPL *kinematic-tree*+joints — and using them directly is what makes `benchlib/threedpw.py` cheap to+set up. It is also not the ground truth any 3DPW paper reports on. Every+published number (SPIN, CLIFF, HMR2.0, NLF, ...) instead runs the gendered SMPL+model forward on the GT ``poses``/``betas``/``trans`` and applies the Human3.6M+joint regressor to the 6890 resulting vertices. The two differ most at the+shoulders and hips, i.e. 4 of the 12 joints in J12.++This script produces that reference once, as ``<out>/<split>.npz`` holding the+17 H36M joints per (sequence, person) in the **same world frame** as+``jointPositions`` — so everything downstream keeps using ``cam_poses`` exactly+as before and only the joint values change.++ python benchmark/make_3dpw_gt.py \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --smpl-dir benchmark/data/smpl \+ --out benchmark/data/3dpw_gt_h36m++Needs, once:+ SMPL_{MALE,FEMALE}.pkl smpl.is.tue.mpg.de (free research licence)+ J_regressor_h36m.npy SPIN's data bundle+"""++from __future__ import annotations++import argparse+import pickle+import sys+import types+from pathlib import Path++import numpy as np+import torch++# --------------------------------------------------------------------------+# SMPL+# --------------------------------------------------------------------------+# The official pkls store several fields as chumpy arrays. chumpy does not+# import on numpy >= 1.24 (np.bool/np.float aliases) nor on Python >= 3.11+# (inspect.getargspec), and we only need the numeric payload, so unpickle+# against a stub whose instances keep whatever state the pickle sets.+++class _ChStub:+ def __array__(self, dtype=None):+ a = np.asarray(self.__dict__["x"])+ return a.astype(dtype) if dtype is not None else a+++def _install_chumpy_stub() -> None:+ if "chumpy" in sys.modules:+ return+ ch = types.ModuleType("chumpy.ch")+ ch.Ch = _ChStub+ root = types.ModuleType("chumpy")+ root.ch = ch+ root.Ch = _ChStub+ sys.modules["chumpy"] = root+ sys.modules["chumpy.ch"] = ch+++def _to_numpy(v):+ if isinstance(v, _ChStub):+ return np.asarray(v.__dict__["x"], dtype=np.float64)+ if hasattr(v, "toarray"): # scipy sparse J_regressor+ return np.asarray(v.toarray(), dtype=np.float64)+ return np.asarray(v, dtype=np.float64)+++def load_smpl(path: Path, device) -> dict:+ _install_chumpy_stub()+ with open(path, "rb") as f:+ d = pickle.load(f, encoding="latin1")+ out = dict(+ v_template=_to_numpy(d["v_template"]),+ shapedirs=_to_numpy(d["shapedirs"]),+ posedirs=_to_numpy(d["posedirs"]),+ J_regressor=_to_numpy(d["J_regressor"]),+ weights=_to_numpy(d["weights"]),+ parents=np.asarray(d["kintree_table"], dtype=np.int64)[0].copy(),+ )+ out["parents"][0] = -1+ return {k: (torch.as_tensor(v, dtype=torch.float32, device=device)+ if k != "parents" else torch.as_tensor(v, device=device))+ for k, v in out.items()}+++def rodrigues(rvec: torch.Tensor) -> torch.Tensor:+ """(..., 3) axis-angle -> (..., 3, 3) rotation matrices."""+ theta = rvec.norm(dim=-1, keepdim=True).clamp_min(1e-12)+ axis = rvec / theta+ x, y, z = axis.unbind(-1)+ zero = torch.zeros_like(x)+ K = torch.stack([zero, -z, y, z, zero, -x, -y, x, zero], dim=-1)+ K = K.reshape(*rvec.shape[:-1], 3, 3)+ eye = torch.eye(3, device=rvec.device, dtype=rvec.dtype).expand_as(K)+ s, c = torch.sin(theta)[..., None], torch.cos(theta)[..., None]+ return eye + s * K + (1 - c) * (K @ K)+++def smpl_forward(m: dict, poses: torch.Tensor, betas: torch.Tensor,+ trans: torch.Tensor):+ """Standard SMPL LBS.++ poses (N, 72), betas (B,), trans (N, 3) -> vertices (N, 6890, 3) and the+ 24 kinematic-tree joints (N, 24, 3). The latter are the joint *centres*+ carried by the global transforms, which is what 3DPW's ``jointPositions``+ stores — not ``J_regressor @ posed vertices``, which differs by ~4.5 mm+ because pose blendshapes and skinning move the surface.+ """+ N = poses.shape[0]+ v_shaped = m["v_template"] + torch.einsum("vck,k->vc", m["shapedirs"], betas)+ J = m["J_regressor"] @ v_shaped # (24, 3)++ R = rodrigues(poses.reshape(N, 24, 3)) # (N, 24, 3, 3)+ eye = torch.eye(3, device=R.device).expand(N, 23, 3, 3)+ pose_feature = (R[:, 1:] - eye).reshape(N, 207)+ v_posed = v_shaped + torch.einsum("vck,nk->nvc", m["posedirs"], pose_feature)++ # Global joint transforms down the kinematic tree.+ parents = m["parents"]+ G = torch.zeros(N, 24, 4, 4, device=R.device)+ G[:, :, 3, 3] = 1.0+ G[:, 0, :3, :3], G[:, 0, :3, 3] = R[:, 0], J[0]+ for i in range(1, 24):+ loc = torch.zeros(N, 4, 4, device=R.device)+ loc[:, 3, 3] = 1.0+ loc[:, :3, :3] = R[:, i]+ loc[:, :3, 3] = J[i] - J[parents[i]]+ G[:, i] = G[:, parents[i]] @ loc++ J_posed = G[:, :, :3, 3] + trans[:, None, :]++ # Remove the rest-pose offset so the transform maps rest -> posed.+ J_h = torch.cat([J, torch.zeros(24, 1, device=R.device)], dim=-1) # (24, 4)+ G = G - torch.nn.functional.pad(G @ J_h[None, :, :, None], (3, 0, 0, 0))++ T = torch.einsum("vj,njab->nvab", m["weights"], G) # (N, 6890, 4, 4)+ v_h = torch.cat([v_posed, torch.ones(N, 6890, 1, device=R.device)], dim=-1)+ verts = torch.einsum("nvab,nvb->nva", T, v_h)[..., :3]+ return verts + trans[:, None, :], J_posed+++# --------------------------------------------------------------------------+# Driver+# --------------------------------------------------------------------------+def build_split(seq_dir: Path, smpl_dir: Path, regressor: np.ndarray,+ split: str, num_betas: int, device, chunk: int,+ check: bool) -> dict:+ models = {g: load_smpl(smpl_dir / f"SMPL_{g.upper()}.pkl", device)+ for g in ("male", "female")}+ n_dirs = models["male"]["shapedirs"].shape[-1]+ Jreg = torch.as_tensor(regressor, dtype=torch.float32, device=device)++ out: dict[str, np.ndarray] = {}+ resid = []+ for pkl in sorted((seq_dir / split).glob("*.pkl")):+ with open(pkl, "rb") as f:+ seq = pickle.load(f, encoding="latin1")+ name = str(seq["sequence"])+ for pid, poses in enumerate(seq["poses"]):+ gender = str(seq["genders"][pid])+ gender = "male" if gender.lower().startswith("m") else "female"+ m = models[gender]+ p = torch.as_tensor(np.asarray(poses, np.float32), device=device)+ t = torch.as_tensor(np.asarray(seq["trans"][pid], np.float32), device=device)+ # 3DPW stores 300 shape coefficients for some subjects and 10 for+ # others; zero-padding to the model's 300 dirs covers both, and+ # ``num_betas`` then truncates uniformly. The published protocol is+ # the first 10 (SPIN's pw3d preprocessing).+ raw = np.asarray(seq["betas"][pid], np.float32).ravel()+ b = np.zeros(n_dirs, np.float32)+ k = min(num_betas, raw.shape[0])+ b[:k] = raw[:k]+ b = torch.as_tensor(b, device=device)++ j17, j24 = [], []+ for s in range(0, p.shape[0], chunk):+ V, Jk = smpl_forward(m, p[s:s + chunk], b, t[s:s + chunk])+ j17.append((Jreg @ V).cpu().numpy())+ if check:+ j24.append(Jk.cpu().numpy())+ out[f"{name}|{pid}"] = np.concatenate(j17).astype(np.float32)++ if check:+ gt24 = np.asarray(seq["jointPositions"][pid], np.float32).reshape(-1, 24, 3)+ d = np.linalg.norm(np.concatenate(j24) - gt24, axis=-1)+ resid.append(d)+ print(f" {name:<32s} {len(seq['poses'])} person(s)", flush=True)++ if check and resid:+ d = np.concatenate(resid) * 1000.0+ print(f"[check] SMPL-24 forward vs jointPositions: mean {d.mean():.3f} mm, "+ f"p99 {np.percentile(d, 99):.3f} mm, max {d.max():.3f} mm")+ return out+++def main():+ ap = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--sequence-dir", required=True)+ ap.add_argument("--smpl-dir", required=True,+ help="holds SMPL_MALE.pkl / SMPL_FEMALE.pkl")+ ap.add_argument("--regressor", default="benchmark/data/smpl/J_regressor_h36m.npy")+ ap.add_argument("--out", default="benchmark/data/3dpw_gt_h36m")+ ap.add_argument("--splits", nargs="+", default=["test", "validation", "train"])+ ap.add_argument("--num-betas", type=int, default=10,+ help="10 reproduces the published protocol; 300 uses the "+ "full shape vector 3DPW actually stores")+ ap.add_argument("--device", default="cuda")+ ap.add_argument("--chunk", type=int, default=64)+ ap.add_argument("--check", action="store_true",+ help="also regress the SMPL-24 joints and report their "+ "distance to jointPositions (should be ~0)")+ args = ap.parse_args()++ device = torch.device(args.device if torch.cuda.is_available() else "cpu")+ regressor = np.load(args.regressor)+ seq_dir, out_dir = Path(args.sequence_dir), Path(args.out)+ out_dir.mkdir(parents=True, exist_ok=True)++ for split in args.splits:+ print(f"[3DPW] {split}")+ data = build_split(seq_dir, Path(args.smpl_dir), regressor, split,+ args.num_betas, device, args.chunk, args.check)+ dst = out_dir / f"{split}.npz"+ np.savez_compressed(dst, layout="h36m17", num_betas=args.num_betas,+ regressor=Path(args.regressor).name, **data)+ n = sum(v.shape[0] for v in data.values())+ print(f"[3DPW] {split}: {len(data)} person-tracks, {n} frames -> {dst}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/results/3dpw_test_b3_s1.json b/benchmark/results/3dpw_test_b3_s1.jsonnew file mode 100644index 0000000..d5b306d--- /dev/null+++ b/benchmark/results/3dpw_test_b3_s1.json@@ -0,0 +1,136 @@+{+ "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|h36m": {+ "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",+ "gt": "h36m",+ "split": "test",+ "stride": 1,+ "epoch": 137,+ "source": "ema",+ "n": 35463,+ "n_dropped": 0,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 104.63785012091508,+ "PA_MPJPE_mm": 67.89201715239723,+ "PCK3D_50mm": 0.386628316837267,+ "PCK3D_100mm": 0.8279313248013019,+ "AUC_0_150mm": 0.555249314824265,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 66.1026158925688,+ "right_knee": 48.77531527426408,+ "right_hip": 98.43968262009881,+ "left_hip": 86.54123107665853,+ "left_knee": 46.110874847033735,+ "left_ankle": 67.11189987887853,+ "right_wrist": 70.71902986402077,+ "right_elbow": 57.68874380634787,+ "right_shoulder": 51.452820373726844,+ "left_shoulder": 53.46686079217727,+ "left_elbow": 66.61272199119497,+ "left_wrist": 74.65868335746043,+ "neck": 31.322680447479044,+ "head": 131.48507991165152+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 102.43934163084393,+ "PA_MPJPE_mm": 62.492379121774945,+ "PCK3D_50mm": 0.43160947090394686,+ "PCK3D_100mm": 0.8874202220154339,+ "AUC_0_150mm": 0.5877625294150285,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 62.453416312484485,+ "right_knee": 50.619841347293885,+ "right_hip": 98.35461927230007,+ "left_hip": 85.97593910675288,+ "left_knee": 47.09601993678759,+ "left_ankle": 62.49923408065495,+ "right_wrist": 66.505798652139,+ "right_elbow": 48.842995632922765,+ "right_shoulder": 51.30888543605562,+ "left_shoulder": 51.65398595718133,+ "left_elbow": 54.28141719363774,+ "left_wrist": 70.316396533089+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 72.82834498471384,+ "PA_MPJPE_mm": 42.94566703536861,+ "PCK3D_50mm": 0.7122816134321083,+ "PCK3D_100mm": 0.946737243243461,+ "AUC_0_150mm": 0.7131403003143016+ }+ ]+ },+ "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|jointpositions": {+ "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",+ "gt": "jointpositions",+ "split": "test",+ "stride": 1,+ "epoch": 137,+ "source": "ema",+ "n": 35463,+ "n_dropped": 0,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 78.92239927507839,+ "PA_MPJPE_mm": 55.88134236358216,+ "PCK3D_50mm": 0.5672270092369914,+ "PCK3D_100mm": 0.889079563810974,+ "AUC_0_150mm": 0.6319885423517287,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 57.2149217889093,+ "right_knee": 44.6731540800345,+ "right_hip": 51.30738310230049,+ "left_hip": 47.680922082194506,+ "left_knee": 47.11408900032607,+ "left_ankle": 58.746754378530234,+ "right_wrist": 68.48622774964359,+ "right_elbow": 41.41727540449767,+ "right_shoulder": 45.64858842957503,+ "left_shoulder": 50.333023561680356,+ "left_elbow": 47.20158941256118,+ "left_wrist": 71.92360887402488,+ "neck": 34.406638478724766,+ "head": 116.18461674714762+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 72.61393647860733,+ "PA_MPJPE_mm": 49.30383775900926,+ "PCK3D_50mm": 0.6433066388442414,+ "PCK3D_100mm": 0.938478132137721,+ "AUC_0_150mm": 0.6735733805853685,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 54.539697485801305,+ "right_knee": 46.066113533422204,+ "right_hip": 47.155299592078904,+ "left_hip": 44.40314628895565,+ "left_knee": 49.06421851012075,+ "left_ankle": 54.51289182848444,+ "right_wrist": 66.35472929929244,+ "right_elbow": 36.84928335165509,+ "right_shoulder": 39.98208452636371,+ "left_shoulder": 41.037157437753955,+ "left_elbow": 40.987603019551045,+ "left_wrist": 70.69382823463162+ }+ }+ ]+ }+}\ No newline at end of filediff --git a/benchmark/results/3dpw_test_v3_s1.json b/benchmark/results/3dpw_test_v3_s1.jsonnew file mode 100644index 0000000..1486c7d--- /dev/null+++ b/benchmark/results/3dpw_test_v3_s1.json@@ -0,0 +1,136 @@+{+ "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|h36m": {+ "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",+ "gt": "h36m",+ "split": "test",+ "stride": 1,+ "epoch": 126,+ "source": "ema",+ "n": 35463,+ "n_dropped": 0,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 105.70162783403123,+ "PA_MPJPE_mm": 69.05148473277063,+ "PCK3D_50mm": 0.37404175780793664,+ "PCK3D_100mm": 0.8228153286523983,+ "AUC_0_150mm": 0.5488736167026034,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 68.26131519537937,+ "right_knee": 48.22895082504128,+ "right_hip": 98.35751407606348,+ "left_hip": 86.86613781457758,+ "left_knee": 46.25701446183899,+ "left_ankle": 69.39201526188755,+ "right_wrist": 71.39134970517831,+ "right_elbow": 57.63600754408093,+ "right_shoulder": 52.59923939395279,+ "left_shoulder": 55.207572449651074,+ "left_elbow": 67.88209503589113,+ "left_wrist": 76.36662877587462,+ "neck": 32.38007584338846,+ "head": 135.8948698759832+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 102.98084160960175,+ "PA_MPJPE_mm": 63.198364128279884,+ "PCK3D_50mm": 0.42087997819323425,+ "PCK3D_100mm": 0.8823257103647933,+ "AUC_0_150mm": 0.5837227290354721,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 64.60270339441506,+ "right_knee": 49.996919602186196,+ "right_hip": 98.25229959444044,+ "left_hip": 86.3963348166251,+ "left_knee": 47.03163228047883,+ "left_ankle": 64.51319472082247,+ "right_wrist": 67.54483867159513,+ "right_elbow": 48.887356763210875,+ "right_shoulder": 52.06354180217583,+ "left_shoulder": 52.42872324128663,+ "left_elbow": 54.99294113405385,+ "left_wrist": 71.66988351806822+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 73.81513519657514,+ "PA_MPJPE_mm": 44.1324115732964,+ "PCK3D_50mm": 0.6958459722608272,+ "PCK3D_100mm": 0.9417461257407116,+ "AUC_0_150mm": 0.7059962931443702+ }+ ]+ },+ "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|jointpositions": {+ "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",+ "gt": "jointpositions",+ "split": "test",+ "stride": 1,+ "epoch": 126,+ "source": "ema",+ "n": 35463,+ "n_dropped": 0,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 80.3476390328507,+ "PA_MPJPE_mm": 56.98931886430932,+ "PCK3D_50mm": 0.5518246381540519,+ "PCK3D_100mm": 0.8840864321365125,+ "AUC_0_150mm": 0.6254987511485651,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 60.29150986805832,+ "right_knee": 44.09355165407435,+ "right_hip": 52.21253960531078,+ "left_hip": 48.437115641833024,+ "left_knee": 47.35052736862379,+ "left_ankle": 61.33749022699511,+ "right_wrist": 69.4740236792569,+ "right_elbow": 41.995470135750026,+ "right_shoulder": 46.26280446156064,+ "left_shoulder": 52.228924129906055,+ "left_elbow": 48.42258559131994,+ "left_wrist": 73.93419187162651,+ "neck": 33.73966925436235,+ "head": 118.07006061165272+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 73.30164497963386,+ "PA_MPJPE_mm": 50.3841473765136,+ "PCK3D_50mm": 0.6284108319469118,+ "PCK3D_100mm": 0.9329653441615204,+ "AUC_0_150mm": 0.6672313927676856,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 57.91085514413626,+ "right_knee": 45.33655452203096,+ "right_hip": 47.530686446692584,+ "left_hip": 45.00132633545716,+ "left_knee": 49.146387447390694,+ "left_ankle": 56.970314712065104,+ "right_wrist": 67.99991243571321,+ "right_elbow": 38.08431350645294,+ "right_shoulder": 40.287118217560185,+ "left_shoulder": 41.70494390891554,+ "left_elbow": 42.05522689574795,+ "left_wrist": 72.5821289460007+ }+ }+ ]+ }+}\ No newline at end of filediff --git a/benchmark/results/adapter_j14_h36m_b3_s1.npz b/benchmark/results/adapter_j14_h36m_b3_s1.npznew file mode 100644index 0000000..69b8a7bBinary files /dev/null and b/benchmark/results/adapter_j14_h36m_b3_s1.npz differdiff --git a/benchmark/results/adapter_j14_h36m_v3_s1.npz b/benchmark/results/adapter_j14_h36m_v3_s1.npznew file mode 100644index 0000000..da708aeBinary files /dev/null and b/benchmark/results/adapter_j14_h36m_v3_s1.npz differdiff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.pyindex ba40515..244940f 100755--- a/instanthmr_distill_train/train_distill_jz.py+++ b/instanthmr_distill_train/train_distill_jz.py@@ -491,14 +491,16 @@ def train(args, cfg, rank, local_rank, world):raise SystemExit("--val-3dpw needs --val-3dpw-images")from val3dpw import ThreeDPWValSetdpw = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,- split=args.val_3dpw_split, stride=args.val_3dpw_stride)+ split=args.val_3dpw_split, stride=args.val_3dpw_stride,+ gt=args.val_3dpw_gt)# Stride-shard across ranks; the metric is all-reduced afterwards.dpw_loader = torch.utils.data.DataLoader(torch.utils.data.Subset(dpw, list(range(rank, len(dpw), world))),batch_size=cfg.batch_size, shuffle=False, drop_last=False,num_workers=max(1, cfg.num_workers // 2), pin_memory=True)log(f"3DPW {args.val_3dpw_split}: {len(dpw):,} person-frames "- f"(stride {args.val_3dpw_stride}) -> selection metric is J12 PA-MPJPE")+ f"(stride {args.val_3dpw_stride}, GT {args.val_3dpw_gt}) "+ f"-> selection metric is J12 PA-MPJPE")# Optional read-only monitor on the split the paper reports. Selection# NEVER touches it: picking the best of ~100 epochs on the test split@@ -507,7 +509,8 @@ def train(args, cfg, rank, local_rank, world):# not, so it is logged and never compared to `best`.if args.val_3dpw_test:dpw_t = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,- split="test", stride=args.val_3dpw_test_stride)+ split="test", stride=args.val_3dpw_test_stride,+ gt=args.val_3dpw_gt)dpw_test_loader = torch.utils.data.DataLoader(torch.utils.data.Subset(dpw_t, list(range(rank, len(dpw_t), world))),batch_size=cfg.batch_size, shuffle=False, drop_last=False,@@ -627,6 +630,7 @@ def train(args, cfg, rank, local_rank, world):"w_keypoints3d": cfg.w_keypoints3d, "kp3d_loss": cfg.kp3d_loss,"pose_split": cfg.pose_split, "pose_beta": cfg.pose_beta,"finger_weight": cfg.finger_weight,+ "val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,"started": time.strftime("%Y-%m-%d %H:%M")})for epoch in range(start_epoch, args.epochs):@@ -724,16 +728,20 @@ def train(args, cfg, rank, local_rank, world):if dpw_loader is not None:import val3dpwdpw_raw = all_reduce_mean(- val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device, cfg.use_amp),+ val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device,+ cfg.use_amp, gt=args.val_3dpw_gt),device)dpw_em = all_reduce_mean(- val3dpw.evaluate(ema, dpw_loader, mhr_module, device, cfg.use_amp),+ val3dpw.evaluate(ema, dpw_loader, mhr_module, device,+ cfg.use_amp, gt=args.val_3dpw_gt),device)if dpw_test_loader is not None:dpw_test_raw = all_reduce_mean(val3dpw.evaluate(- ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp), device)+ ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp,+ gt=args.val_3dpw_gt), device)dpw_test_em = all_reduce_mean(val3dpw.evaluate(- ema, dpw_test_loader, mhr_module, device, cfg.use_amp), device)+ ema, dpw_test_loader, mhr_module, device, cfg.use_amp,+ gt=args.val_3dpw_gt), device)raw["Mesh_PA_MPJPE"], em["Mesh_PA_MPJPE"] = (dpw_raw["J12_PA_MPJPE"], dpw_em["J12_PA_MPJPE"])raw["Mesh_MPJPE"], em["Mesh_MPJPE"] = (@@ -872,6 +880,13 @@ def parse_args():p.add_argument("--val-3dpw-test-stride", type=int, default=5)p.add_argument("--val-3dpw-stride", type=int, default=5,help="3DPW is 30 fps; neighbouring frames are near-duplicates.")+ p.add_argument("--val-3dpw-gt", default="h36m",+ choices=("h36m", "jointpositions"),+ help="h36m (default) is the published protocol and needs "+ "gt_h36m/ next to sequenceFiles/ (see "+ "benchmark/make_3dpw_gt.py). jointpositions is the raw "+ "pickle field the runs before this flag selected on — "+ "several mm apart, so a resumed run must keep its own.")p.add_argument("--mix", default="sqrt",help="Dataset mixture: sqrt | uniform | equal | ""'sa1b=0.4,aic=0.35,coco=0.15,mpii=0.1'.")diff --git a/instanthmr_distill_train/val3dpw.py b/instanthmr_distill_train/val3dpw.pyindex 4966e24..08eb332 100644--- a/instanthmr_distill_train/val3dpw.py+++ b/instanthmr_distill_train/val3dpw.py@@ -21,6 +21,12 @@ all 60 sequences (see ``benchmark/README.md``). On Jean Zay it is not: the builtcorpus is coco/mpii/aic/sa1b only, so 3DPW there is genuinely unseen — which iswhat makes this worth doing on the cluster and not on the laptop.+GT is the published protocol by default — SMPL forward + the H36M regressor,+precomputed by ``benchmark/make_3dpw_gt.py`` into ``<3DPW root>/gt_h36m/`` and+rsynced next to ``sequenceFiles``. ``gt="jointpositions"`` restores the raw+pickle field the harness used before; the two are several mm apart, so numbers+from the two sources must never be compared to each other.+Selection metric is **PA-MPJPE on J12**. J12 is limbs only, where MHR and SMPLmean the same anatomical points; J14 adds neck and head, which the two rigsplace differently and which therefore carry a systematic penalty unrelated to@@ -99,11 +105,13 @@ class ThreeDPWValSet(Dataset):def __init__(self, sequence_dir: str, image_root: str,split: str = "validation", stride: int = 5,- bbox_scale: float = 1.2):+ bbox_scale: float = 1.2, gt: str = "h36m",+ gt_dir: str | None = None):self.samples, self.gt = P.build_samples(sequence_dir, image_root, split=split,- stride=stride, bbox_scale=bbox_scale)+ stride=stride, bbox_scale=bbox_scale, gt=gt, gt_dir=gt_dir)self.gt = np.asarray(self.gt, dtype=np.float32)+ self.gt_source = gtdef __len__(self) -> int:return len(self.samples)@@ -128,7 +136,8 @@ class ThreeDPWValSet(Dataset):@torch.no_grad()-def evaluate(model, loader, mhr_module, device, use_amp: bool = True) -> dict:+def evaluate(model, loader, mhr_module, device, use_amp: bool = True,+ gt: str = "h36m") -> dict:"""PA-MPJPE and MPJPE in mm on J12 and J14, from the MHR forward pass.Predictions come from ``mhr_params`` through the same FK + (70, 127)@@ -176,12 +185,13 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True) -> dict:return respred = np.concatenate(preds) * MM # (N, 70, 3) rig-local, mm- gt = np.concatenate(gts) * MM # (N, 24, 3) camera space, mm+ gt_all = np.concatenate(gts) * MM # (N, J, 3) camera space, mmres["n"] = float(len(pred))for name in ("J12", "J14"):spec = J.JOINT_SETS[name]- p, g = pred[:, spec["mhr"], :], gt[:, spec["smpl"], :]+ p = pred[:, spec["mhr"], :]+ g = gt_all[:, J.gt_index(spec, gt), :]rp, rg = J.pelvis(p), J.pelvis(g)res[f"{name}_MPJPE"] = float(M.mpjpe(p, g, root=(rp, rg)).mean())res[f"{name}_PA_MPJPE"] = float(M.pa_mpjpe(p, g).mean())