Python
feat: now using MHR gt regressor
mohamdeve1805d89364d
diff --git a/benchmark/README.md b/benchmark/README.mdindex b79cce3..ece42b7 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -169,12 +169,10 @@ nothing to do with model quality. The harness therefore reports three rows:landmarks. Against the H36M GT it is **not** clean: the hips alone move 292 vs120 mm apart between the conventions. Read it as a consistent tracking metric,not as a rig-free measure of the model.-- **J14+adapter** is therefore the row to compare against a published table,- not an optional extra.-- **J14+adapter** *(optional)* — a linear MHR70 → SMPL-J14 joint regressor,- fitted on 3DPW **train**, that absorbs the rig offset. Standard practice when- a method's joint convention differs from the benchmark's. Fit it once, then- apply to test:+- **J14+adapter** — a linear MHR70 → J14 regressor (14x70, no bias, applied to+ centred keypoints, so it is equivariant to rotation and translation and can+ only move landmarks, never fix a pose). It absorbs the rig offset and is the+ row to compare against a published table, not an optional extra.```bashpython benchmark/eval_3dpw.py --onnx ... --split train --stride 10 \@@ -183,7 +181,37 @@ nothing to do with model quality. The harness therefore reports three rows:--adapter benchmark/results/adapter_j14.npz```- Fit on train, report on test — never fit on the split you report.+ Fit on train, report on test — never fit on the split you report. The two+ splits share no sequences, so this is a real holdout, and an adapter is+ specific to one model *and* one GT convention: applying b3_s1's to another+ checkpoint, or an `h36m` one to `jointpositions`, is meaningless.++ **Fit it from the teacher, not from the student.** `--fit-adapter` uses one+ checkpoint's own predictions as the input side, so the map can absorb that+ checkpoint's systematic error along with the rig offset — a different adapter+ per checkpoint, and a fair reader will call it tuning.+ `benchmark/fit_adapter_mhr.py` instead pairs the **teacher's** MHR labels in+ `data/sam3d_gt_3dpw` with the H36M GT on the same 3DPW train frames, giving+ one adapter for every checkpoint that never saw a student:++ ```bash+ python benchmark/fit_adapter_mhr.py --sequence-dir /path/to/3DPW/sequenceFiles \+ --out benchmark/results/adapter_j14_h36m_teacher.npz+ ```++ That it is a rig conversion rather than an error sponge is visible two ways.+ The teacher's own J14 error against the GT drops from **44.90 mm to 14.83 mm**+ on a held-out half — near the teacher's accuracy floor, which absorbing error+ could not reach. And every row reads anatomically:+ `right_hip = +1.16 right_hip - 0.32 left_hip` (widening SMPL's 120 mm hips+ toward H36M's 292 mm), `head = +0.57 left_ear +0.55 right_ear -0.47 neck`+ (extrapolating to the top of the skull), `right_elbow` from the cubital fossa,+ `right_shoulder` from the acromion, `right_knee = +0.84 right_knee` — nearly+ identity, because the knee is the same point in both conventions.++ The fit must also be truncated: see `RCOND` in `fit_adapter`, without which+ the map reaches the hips through +-2000 weights on near-collinear finger+ joints and stops being a joint regressor in any readable sense.### Train/test contamination — READ THIS BEFORE PUBLISHINGdiff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.pyindex 71d37dc..4ec8bf5 100644--- a/benchmark/eval_3dpw.py+++ b/benchmark/eval_3dpw.py@@ -36,6 +36,7 @@ from benchlib import threedpw as Pfrom benchlib.runner import Sample, build_model, run_samples, stack_jointsMM = 1000.0 # metres -> millimetres+RCOND = 3e-3 # singular-value cutoff for the joint adapter; see fit_adapterdef parse_args():@@ -107,6 +108,15 @@ def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,equivariant to rotation and translation — it only removes the systematicdifference in where the two rigs place each landmark.+ The 70 MHR keypoints are 40 finger joints millimetres apart, so the design+ matrix is badly conditioned and the unregularised solution reaches those+ landmarks through +-2000 weights on fingers that nearly cancel. It scores+ well and is not a joint regressor. Truncating the small singular values+ (``RCOND``) gives a rank-24 map with weights under 0.9, where the H36M hip+ reads as ``0.94 * MHR hip + 0.99 * neck``-style combinations of nearby+ landmarks, and costs nothing measurable: on a sequence-level holdout inside+ 3DPW train, 45.74 mm against the unregularised fit's 45.76 mm.+Centring must happen in float64. Doing it in float32 leaves a ~0.04 mmresidual in the all-ones direction, which is exactly the direction thecentring annihilates; ``lstsq`` then sees a singular value of 3e-7 relative@@ -121,7 +131,7 @@ def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,# Flatten samples and coordinates into the row dimension.A = X.transpose(0, 2, 1).reshape(-1, X.shape[1]) # (N*3, 70)B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1]) # (N*3, 14)- W, *_ = np.linalg.lstsq(A, B, rcond=None)+ W, *_ = np.linalg.lstsq(A, B, rcond=RCOND)return W.T # (14, 70)diff --git a/benchmark/fit_adapter_mhr.py b/benchmark/fit_adapter_mhr.pynew file mode 100644index 0000000..3618372--- /dev/null+++ b/benchmark/fit_adapter_mhr.py@@ -0,0 +1,113 @@+#!/usr/bin/env python3+"""Fit the MHR70 -> J14 adapter from teacher labels, independent of any student.++``eval_3dpw_ckpt.py --fit-adapter`` fits the map against one checkpoint's own+predictions, which lets it absorb that checkpoint's systematic error along with+the skeleton difference. This fits the same map against the **teacher's** MHR+labels in ``data/sam3d_gt_3dpw`` instead, so the result is one adapter for every+checkpoint and describes only the rig conversion.++Both sides come from the same 3DPW frame: the teacher's 70 MHR keypoints, and+the H36M joints regressed from GT SMPL by ``make_3dpw_gt.py``. The annotation+filename carries the person index, and it matches 3DPW's own (verified: the+correct pairing sits at 0.5-2.4 deg of residual rotation, the wrong one at+7-108 deg).++ python benchmark/fit_adapter_mhr.py \\+ --annotations data/sam3d_gt_3dpw/annotations \\+ --sequence-dir /path/to/3DPW/sequenceFiles \\+ --out benchmark/results/adapter_j14_h36m_teacher.npz+"""++from __future__ import annotations++import argparse+import pickle+import re+import sys+from pathlib import Path++import numpy as np++sys.path.insert(0, str(Path(__file__).resolve().parent))++import eval_3dpw as E # noqa: E402+from benchlib import joints as J # noqa: E402+from benchlib import metrics as M # noqa: E402+from benchlib import threedpw as P # noqa: E402++NAME_RX = re.compile(r"3dpw_(?P<seq>.+?)_image_(?P<frame>\d+)_p(?P<pid>\d+)\.npz$")+++def build_pairs(ann_dir: Path, sequence_dir: Path, split: str, stride: int):+ """(N, 70, 3) teacher MHR keypoints and (N, 17, 3) H36M GT, camera space, mm."""+ table = P.load_gt_h36m(sequence_dir, split)+ seqs: dict[str, dict] = {}+ mhr, gt = [], []+ skipped = 0++ files = sorted(f for f in ann_dir.iterdir() if NAME_RX.match(f.name))+ for f in files[::stride]:+ m = NAME_RX.match(f.name)+ seq, frame, pid = m["seq"], int(m["frame"]), int(m["pid"])+ key = f"{seq}|{pid}"+ if key not in table:+ skipped += 1+ continue+ if seq not in seqs:+ with open(sequence_dir / split / f"{seq}.pkl", "rb") as fh:+ s = pickle.load(fh, encoding="latin1")+ seqs[seq] = dict(cam=np.asarray(s["cam_poses"]),+ valid=[np.asarray(v).astype(bool)+ for v in s["campose_valid"]])+ s = seqs[seq]+ if frame >= len(s["cam"]) or not s["valid"][pid][frame]:+ skipped += 1+ continue++ z = np.load(f)+ T = s["cam"][frame]+ g = table[key][frame] @ T[:3, :3].T + T[:3, 3] # world -> camera+ mhr.append(z["joints_3d"])+ gt.append(g)++ return (np.stack(mhr) * 1000.0, np.stack(gt) * 1000.0, len(files), skipped)+++def main():+ ap = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--annotations", default="data/sam3d_gt_3dpw/annotations")+ ap.add_argument("--sequence-dir", required=True)+ ap.add_argument("--split", default="train")+ ap.add_argument("--stride", type=int, default=1)+ ap.add_argument("--out", default="benchmark/results/adapter_j14_h36m_teacher.npz")+ args = ap.parse_args()++ mhr, gt, n_files, skipped = build_pairs(+ Path(args.annotations), Path(args.sequence_dir), args.split, args.stride)+ print(f"[adapter] {len(mhr)} teacher/GT pairs from {n_files} annotations "+ f"({skipped} skipped: no GT track or campose_valid=0)")++ # How far apart the two skeletons are before any conversion. This is the+ # teacher's own J14 error plus the rig offset, and the adapter's job is the+ # second part.+ before = M.pa_mpjpe(mhr[:, J.J14_FROM_MHR70, :], gt[:, J.J14_FROM_H36M17, :]).mean()++ half = len(mhr) // 2+ W_half = E.fit_adapter(mhr[:half], gt[:half], "h36m")+ held = M.pa_mpjpe(+ np.einsum("jk,nkc->njc", W_half,+ mhr[half:] - mhr[half:].mean(axis=1, keepdims=True)),+ gt[half:, J.J14_FROM_H36M17, :]).mean()+ print(f"[adapter] teacher vs GT J14 PA-MPJPE: {before:.2f} mm raw, "+ f"{held:.2f} mm with a half-fit adapter on the held-out half")++ W = E.fit_adapter(mhr, gt, "h36m")+ Path(args.out).parent.mkdir(parents=True, exist_ok=True)+ np.savez(args.out, W=W, split=args.split, gt="h36m", source="teacher_mhr")+ print(f"[adapter] |W|max {np.abs(W).max():.2f} -> {args.out}")+++if __name__ == "__main__":+ main()diff --git a/benchmark/results/3dpw_test_b3_s1.json b/benchmark/results/3dpw_test_b3_s1.jsonindex d5b306d..06c5dea 100644--- a/benchmark/results/3dpw_test_b3_s1.json+++ b/benchmark/results/3dpw_test_b3_s1.json@@ -63,73 +63,11 @@"joint_set": "J14+adapter","num_joints": 14,"num_samples": 35463,- "MPJPE_mm": 72.82834498471384,- "PA_MPJPE_mm": 42.94566703536861,- "PCK3D_50mm": 0.7122816134321083,- "PCK3D_100mm": 0.946737243243461,- "AUC_0_150mm": 0.7131403003143016- }- ]- },- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|jointpositions": {- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",- "gt": "jointpositions",- "split": "test",- "stride": 1,- "epoch": 137,- "source": "ema",- "n": 35463,- "n_dropped": 0,- "results": [- {- "joint_set": "J14",- "num_joints": 14,- "num_samples": 35463,- "MPJPE_mm": 78.92239927507839,- "PA_MPJPE_mm": 55.88134236358216,- "PCK3D_50mm": 0.5672270092369914,- "PCK3D_100mm": 0.889079563810974,- "AUC_0_150mm": 0.6319885423517287,- "per_joint_PA_MPJPE_mm": {- "right_ankle": 57.2149217889093,- "right_knee": 44.6731540800345,- "right_hip": 51.30738310230049,- "left_hip": 47.680922082194506,- "left_knee": 47.11408900032607,- "left_ankle": 58.746754378530234,- "right_wrist": 68.48622774964359,- "right_elbow": 41.41727540449767,- "right_shoulder": 45.64858842957503,- "left_shoulder": 50.333023561680356,- "left_elbow": 47.20158941256118,- "left_wrist": 71.92360887402488,- "neck": 34.406638478724766,- "head": 116.18461674714762- }- },- {- "joint_set": "J12",- "num_joints": 12,- "num_samples": 35463,- "MPJPE_mm": 72.61393647860733,- "PA_MPJPE_mm": 49.30383775900926,- "PCK3D_50mm": 0.6433066388442414,- "PCK3D_100mm": 0.938478132137721,- "AUC_0_150mm": 0.6735733805853685,- "per_joint_PA_MPJPE_mm": {- "right_ankle": 54.539697485801305,- "right_knee": 46.066113533422204,- "right_hip": 47.155299592078904,- "left_hip": 44.40314628895565,- "left_knee": 49.06421851012075,- "left_ankle": 54.51289182848444,- "right_wrist": 66.35472929929244,- "right_elbow": 36.84928335165509,- "right_shoulder": 39.98208452636371,- "left_shoulder": 41.037157437753955,- "left_elbow": 40.987603019551045,- "left_wrist": 70.69382823463162- }+ "MPJPE_mm": 73.44449613823136,+ "PA_MPJPE_mm": 42.380049729446,+ "PCK3D_50mm": 0.7164247646440355,+ "PCK3D_100mm": 0.9470897232930902,+ "AUC_0_150mm": 0.7166858272872446}]}diff --git a/benchmark/results/3dpw_test_b3_s1_teacher_adapter.json b/benchmark/results/3dpw_test_b3_s1_teacher_adapter.jsonnew file mode 100644index 0000000..a372e83--- /dev/null+++ b/benchmark/results/3dpw_test_b3_s1_teacher_adapter.json@@ -0,0 +1,74 @@+{+ "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|h36m": {+ "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",+ "gt": "h36m",+ "split": "test",+ "stride": 1,+ "epoch": 137,+ "source": "ema",+ "n": 35463,+ "n_dropped": 0,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 104.63785012091508,+ "PA_MPJPE_mm": 67.89201715239723,+ "PCK3D_50mm": 0.386628316837267,+ "PCK3D_100mm": 0.8279313248013019,+ "AUC_0_150mm": 0.555249314824265,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 66.1026158925688,+ "right_knee": 48.77531527426408,+ "right_hip": 98.43968262009881,+ "left_hip": 86.54123107665853,+ "left_knee": 46.110874847033735,+ "left_ankle": 67.11189987887853,+ "right_wrist": 70.71902986402077,+ "right_elbow": 57.68874380634787,+ "right_shoulder": 51.452820373726844,+ "left_shoulder": 53.46686079217727,+ "left_elbow": 66.61272199119497,+ "left_wrist": 74.65868335746043,+ "neck": 31.322680447479044,+ "head": 131.48507991165152+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 102.43934163084393,+ "PA_MPJPE_mm": 62.492379121774945,+ "PCK3D_50mm": 0.43160947090394686,+ "PCK3D_100mm": 0.8874202220154339,+ "AUC_0_150mm": 0.5877625294150285,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 62.453416312484485,+ "right_knee": 50.619841347293885,+ "right_hip": 98.35461927230007,+ "left_hip": 85.97593910675288,+ "left_knee": 47.09601993678759,+ "left_ankle": 62.49923408065495,+ "right_wrist": 66.505798652139,+ "right_elbow": 48.842995632922765,+ "right_shoulder": 51.30888543605562,+ "left_shoulder": 51.65398595718133,+ "left_elbow": 54.28141719363774,+ "left_wrist": 70.316396533089+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 69.02998274054096,+ "PA_MPJPE_mm": 41.82524856095757,+ "PCK3D_50mm": 0.7230534037487764,+ "PCK3D_100mm": 0.9491320934092273,+ "AUC_0_150mm": 0.7205023578153955+ }+ ]+ }+}\ No newline at end of filediff --git a/benchmark/results/3dpw_test_v3_s1.json b/benchmark/results/3dpw_test_v3_s1.jsonindex 1486c7d..b4b2be9 100644--- a/benchmark/results/3dpw_test_v3_s1.json+++ b/benchmark/results/3dpw_test_v3_s1.json@@ -63,73 +63,11 @@"joint_set": "J14+adapter","num_joints": 14,"num_samples": 35463,- "MPJPE_mm": 73.81513519657514,- "PA_MPJPE_mm": 44.1324115732964,- "PCK3D_50mm": 0.6958459722608272,- "PCK3D_100mm": 0.9417461257407116,- "AUC_0_150mm": 0.7059962931443702- }- ]- },- "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|jointpositions": {- "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",- "gt": "jointpositions",- "split": "test",- "stride": 1,- "epoch": 126,- "source": "ema",- "n": 35463,- "n_dropped": 0,- "results": [- {- "joint_set": "J14",- "num_joints": 14,- "num_samples": 35463,- "MPJPE_mm": 80.3476390328507,- "PA_MPJPE_mm": 56.98931886430932,- "PCK3D_50mm": 0.5518246381540519,- "PCK3D_100mm": 0.8840864321365125,- "AUC_0_150mm": 0.6254987511485651,- "per_joint_PA_MPJPE_mm": {- "right_ankle": 60.29150986805832,- "right_knee": 44.09355165407435,- "right_hip": 52.21253960531078,- "left_hip": 48.437115641833024,- "left_knee": 47.35052736862379,- "left_ankle": 61.33749022699511,- "right_wrist": 69.4740236792569,- "right_elbow": 41.995470135750026,- "right_shoulder": 46.26280446156064,- "left_shoulder": 52.228924129906055,- "left_elbow": 48.42258559131994,- "left_wrist": 73.93419187162651,- "neck": 33.73966925436235,- "head": 118.07006061165272- }- },- {- "joint_set": "J12",- "num_joints": 12,- "num_samples": 35463,- "MPJPE_mm": 73.30164497963386,- "PA_MPJPE_mm": 50.3841473765136,- "PCK3D_50mm": 0.6284108319469118,- "PCK3D_100mm": 0.9329653441615204,- "AUC_0_150mm": 0.6672313927676856,- "per_joint_PA_MPJPE_mm": {- "right_ankle": 57.91085514413626,- "right_knee": 45.33655452203096,- "right_hip": 47.530686446692584,- "left_hip": 45.00132633545716,- "left_knee": 49.146387447390694,- "left_ankle": 56.970314712065104,- "right_wrist": 67.99991243571321,- "right_elbow": 38.08431350645294,- "right_shoulder": 40.287118217560185,- "left_shoulder": 41.70494390891554,- "left_elbow": 42.05522689574795,- "left_wrist": 72.5821289460007- }+ "MPJPE_mm": 74.66172176671088,+ "PA_MPJPE_mm": 43.63754202578393,+ "PCK3D_50mm": 0.7020536494777253,+ "PCK3D_100mm": 0.9413050221357471,+ "AUC_0_150mm": 0.70911897400432}]}diff --git a/benchmark/results/3dpw_test_v3_s1_teacher_adapter.json b/benchmark/results/3dpw_test_v3_s1_teacher_adapter.jsonnew file mode 100644index 0000000..aef8a8e--- /dev/null+++ b/benchmark/results/3dpw_test_v3_s1_teacher_adapter.json@@ -0,0 +1,74 @@+{+ "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|h36m": {+ "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",+ "gt": "h36m",+ "split": "test",+ "stride": 1,+ "epoch": 126,+ "source": "ema",+ "n": 35463,+ "n_dropped": 0,+ "results": [+ {+ "joint_set": "J14",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 105.70162783403123,+ "PA_MPJPE_mm": 69.05148473277063,+ "PCK3D_50mm": 0.37404175780793664,+ "PCK3D_100mm": 0.8228153286523983,+ "AUC_0_150mm": 0.5488736167026034,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 68.26131519537937,+ "right_knee": 48.22895082504128,+ "right_hip": 98.35751407606348,+ "left_hip": 86.86613781457758,+ "left_knee": 46.25701446183899,+ "left_ankle": 69.39201526188755,+ "right_wrist": 71.39134970517831,+ "right_elbow": 57.63600754408093,+ "right_shoulder": 52.59923939395279,+ "left_shoulder": 55.207572449651074,+ "left_elbow": 67.88209503589113,+ "left_wrist": 76.36662877587462,+ "neck": 32.38007584338846,+ "head": 135.8948698759832+ }+ },+ {+ "joint_set": "J12",+ "num_joints": 12,+ "num_samples": 35463,+ "MPJPE_mm": 102.98084160960175,+ "PA_MPJPE_mm": 63.198364128279884,+ "PCK3D_50mm": 0.42087997819323425,+ "PCK3D_100mm": 0.8823257103647933,+ "AUC_0_150mm": 0.5837227290354721,+ "per_joint_PA_MPJPE_mm": {+ "right_ankle": 64.60270339441506,+ "right_knee": 49.996919602186196,+ "right_hip": 98.25229959444044,+ "left_hip": 86.3963348166251,+ "left_knee": 47.03163228047883,+ "left_ankle": 64.51319472082247,+ "right_wrist": 67.54483867159513,+ "right_elbow": 48.887356763210875,+ "right_shoulder": 52.06354180217583,+ "left_shoulder": 52.42872324128663,+ "left_elbow": 54.99294113405385,+ "left_wrist": 71.66988351806822+ }+ },+ {+ "joint_set": "J14+adapter",+ "num_joints": 14,+ "num_samples": 35463,+ "MPJPE_mm": 70.59166700598287,+ "PA_MPJPE_mm": 43.23008138157838,+ "PCK3D_50mm": 0.7050406661268687,+ "PCK3D_100mm": 0.9443907331987866,+ "AUC_0_150mm": 0.7119192574437612+ }+ ]+ }+}\ No newline at end of filediff --git a/benchmark/results/adapter_j14_h36m_b3_s1.npz b/benchmark/results/adapter_j14_h36m_b3_s1.npzindex 69b8a7b..8e7ba78 100644Binary files a/benchmark/results/adapter_j14_h36m_b3_s1.npz and b/benchmark/results/adapter_j14_h36m_b3_s1.npz differdiff --git a/benchmark/results/adapter_j14_h36m_teacher.npz b/benchmark/results/adapter_j14_h36m_teacher.npznew file mode 100644index 0000000..99c0139Binary files /dev/null and b/benchmark/results/adapter_j14_h36m_teacher.npz differdiff --git a/benchmark/results/adapter_j14_h36m_v3_s1.npz b/benchmark/results/adapter_j14_h36m_v3_s1.npzindex da708ae..a0d8647 100644Binary files a/benchmark/results/adapter_j14_h36m_v3_s1.npz and b/benchmark/results/adapter_j14_h36m_v3_s1.npz differ