Python
3DPW adapter truncation + teacher-fitted adapter; document the GT change and the EMDB plan
mohamdev52a69b5ee414
diff --git a/CLAUDE.md b/CLAUDE.mdindex 0aeccc6..888c073 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -76,7 +76,7 @@ Docs are the source of truth; the code is bigger than any context window.| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` and `--losses rebalanced` change, the pair-index cache || 2 | `datasets_pipeline/jeanzay/STATUS.md` | Jean Zay cluster + data state, what is built, job templates *(git-excluded, local only)* || 3 | `datasets_pipeline/JEAN_ZAY.md` | site runbook: env, quotas, proxy quirks *(git-excluded, local only)* |-| 4 | `benchmark/README.md` | how published numbers are produced; J12 vs J14, and **why the current 3DPW GT is not yet the published protocol** |+| 4 | `benchmark/README.md` | how published numbers are produced; the 3DPW GT (SMPL forward + H36M regressor), the MHR->J14 adapter, and why **J14+adapter** is the row to quote || 5 | `datasets_pipeline/README.md` | how the corpus is built, including the SA-1B visibility mask || 6 | `docs/architecture.md` | the student model |@@ -118,6 +118,24 @@ loss and metric in the trainer is blind to them.it bare, so any buffer you register lands on the CPU and raises on step 1. Buildtensors on `mhr_module.device` instead.+**3DPW GT is the published protocol, and it is a cache on disk.** The pickles'+`jointPositions` field is the 24 SMPL *kinematic* joints, which is not what any+paper reports; `benchmark/make_3dpw_gt.py` precomputes the real reference (SMPL+forward + `J_regressor_h36m`) into `<3DPW root>/gt_h36m/`, and both the harness+and `val3dpw.py` default to it. Two consequences: a job whose 3DPW directory+lacks that cache raises at startup, and **numbers from before 2026-09-05 are ~13+mm apart from later ones** — `--val-3dpw-gt jointpositions` keeps a resumed run+on its original metric.++**Anything fitted over the 70 MHR keypoints needs float64 and a truncated+`lstsq`.** 40 of the 70 are finger joints millimetres apart, so the design+matrix is near-singular. Centring in float32 leaves a 0.04 mm residual in the+exact direction centring annihilates, `lstsq` keeps it as signal, and the J14+adapter went from 37 mm to 430 mm in-sample looking like a broken GT. Even in+float64 the unregularised fit reaches landmarks through ±2000 weights on+fingers; `RCOND` in `eval_3dpw.fit_adapter` truncates it to a rank-24 map with+weights under 1 at no measurable cost.+**Seed spread is wide.** Observed 70-keypoint PA-MPJPE varies 34–49 mm acrossseeds, which is larger than several of the changes on the roadmap. A single-runcomparison proves nothing; use 2–3 seeds.diff --git a/benchmark/README.md b/benchmark/README.mdindex ece42b7..4161df4 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -28,13 +28,62 @@ MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.- **Human3.6M** — the licence forbids redistribution, the download is manual andslow, and protocol variants (P1 vs P2, 17 vs 14 joints) differ per paper. Higheffort, high chance of an unfair comparison.-- **EMDB** — a genuinely good modern benchmark (EMDB-1 for pose), but needs- registration *and* the SMPL body model *and* their toolkit to unpack. Worth- adding as a second 3D benchmark after 3DPW is in place; `benchlib/threedpw.py`- is the template to copy.-- **RICH** — SMPL-X GT, large download, mostly used for contact estimation.+- **RICH** — the reason to skip it is not the size alone. Its GT is **SMPL-X**,+ so it needs the SMPL-X models *and* a second forward pass (different joint+ count, hand and face parameters) that none of our SMPL code covers; the test+ split alone is 125K images at 4K, against EMDB-1's few thousand; and NLF+ *trains* on RICH (the ∗ in SAM 3D Body's Table 2), so the column is not+ out-of-domain for every method in it.- **MPII** — 2D only, so it adds little once COCO is in.+### Next: EMDB — waiting on registration (asked 2026-09-05)++**Status: blocked on the application at `emdb.ait.ethz.ch`** (a form, approved+by email). Nothing else is missing — the two blockers this file used to list,+the SMPL body model and their toolkit, are gone: `benchmark/data/smpl/` holds+the gendered models and the toolkit is only a viewer, not a format we need.++Use **EMDB-1** (17 sequences, 13.5 min), the camera-frame pose split every paper+reports. EMDB-2 (25 sequences) is global-trajectory and not what this model does.++It is structurally a simpler 3DPW, one subject per sequence:++| EMDB pkl | 3DPW equivalent |+|---|---|+| `smpl{poses_root (N,3), poses_body (N,69), betas (10,), trans}` | `poses` (N,72), `betas`, `trans` |+| `camera{intrinsics, extrinsics (N,4,4), width, height}` | `cam_intrinsics`, `cam_poses` |+| `good_frames_mask` | `campose_valid` |+| `bboxes` (N,4) xyxy — **provided** | derived from projected joints |++`betas` is 10-dim, exactly the count the published protocol uses, so+`make_3dpw_gt.py`'s forward pass drops in unchanged.++**The protocol is the 24 SMPL joints, mid-hip aligned — not the H36M+regressor.** NLF's Table 3 reads "Results on EMDB1 (24j)" and its metrics+section states that "on 3DPW and EMDB the reference point is the midpoint+between the two hip joints"; SAM 3D Body's Table 2 column header is "EMDB (24)".+Those 24 joints are what `smpl_forward` already returns as its second output.++Two things already done, so the integration is a loader and nothing else:++- `benchmark/results/adapter_smpl24_teacher.npz` — the MHR70 → SMPL-24+ conversion, fitted the same student-free way as the J14 one, from 22,209+ teacher/GT pairs on 3DPW train (where `jointPositions` **is** the SMPL-24 set,+ verified to 0.001 mm). Reads anatomically: `pelvis = 0.38 right_hip + 0.34+ left_hip`, `left_knee = 0.93 left_knee`, `left_ankle` from heel + ankle + toe.+- **The conversion floor: 25.11 mm MPJPE / 18.77 mm PA-MPJPE** on a held-out+ half. That is what an MHR-rigged model carries on a 24-joint benchmark before+ it makes a single mistake — an *upper* bound, since it also contains the+ teacher's own 3DPW error, which cannot be separated without perfect MHR GT.+ Quote it next to the result: the table it joins runs 61.7 (SAM 3D Body) to+ 118.5 (HMR2.0b) MPJPE, so ~25 mm of rig overhead is material.++What is left is `benchlib/emdb.py`, ~80 lines mirroring `threedpw.py`. It was+deliberately **not** written ahead of the data: `dataset.md` does not say+whether `extrinsics` is world→camera or camera→world, and guessing wrong gives+quietly wrong numbers rather than a crash. Thirty seconds with a real pkl+settles it.+## Setup```bash@@ -70,6 +119,15 @@ compares them to `jointPositions`: **0.001 mm max over 74,620 person-frames**,which is what certifies the forward pass, the gendered model choice and theworld frame all the way through.+Then fit the MHR70 → J14 adapter once. It is student-free and dataset-wide, so+this is a one-time step, not something to repeat per checkpoint:++```bash+python benchmark/fit_adapter_mhr.py \+ --sequence-dir /path/to/3DPW/sequenceFiles \+ --out benchmark/results/adapter_j14_h36m_teacher.npz+```+## Running```bash@@ -85,15 +143,40 @@ python benchmark/eval_3dpw.py \--image-root /path/to/3DPW/imageFiles \--exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations-# same thing for a training checkpoint instead of an exported graph+# same thing for a training checkpoint instead of an exported graph. --adapter+# adds the J14+adapter row, which is the one to quote.python benchmark/eval_3dpw_ckpt.py \- --ckpt instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth \+ --ckpt instanthmr_distill_train/runs/*/*/best_student_model_v3.pth \--sequence-dir /path/to/3DPW/sequenceFiles \- --image-root /path/to/3DPW/imageFiles --split test+ --image-root /path/to/3DPW/imageFiles --split test \+ --adapter benchmark/results/adapter_j14_h36m_teacher.npz```Both write a JSON report to `benchmark/results/`.+## Current numbers (2026-09-05)++3DPW **test**, all 35,463 person-frames, published-protocol GT, the two Jean Zay+checkpoints pulled at ~epoch 130. Full reports in+`benchmark/results/3dpw_test_{b3_s1,v3_s1}_teacher_adapter.json`.++| run | preset | J12 PA | J14 PA | **J14+adapter PA** | J14+adapter MPJPE |+|---|---|---|---|---|---|+| b3_s1 | baseline | 62.49 | 67.89 | **41.83** | 69.03 |+| v3_s1 | v2 | 63.20 | 69.05 | **43.23** | 70.59 |++**Quote the adapter row.** Raw J14 is dominated by convention, not quality:+per-joint it is 131 mm at `head` and 98/87 mm at the hips against ~47 mm at the+knees.++Two caveats before this goes in a table. The 1.4 mm gap between the two runs is+**not** a result — measured seed spread in this repo is far wider, so+baseline-vs-v2 needs 2-3 seeds per arm through the same path. And these are the+only numbers here that are genuinely held out: both checkpoints were trained on+Jean Zay, whose corpus is coco/mpii/aic/sa1b/harmony4d with `3dpw_train`+annotations-only, so all 24 test sequences are unseen. The local `data/` corpus+is **not** clean — see the contamination section.+## Protocols, stated preciselyWrite these into the paper — they are the details a reviewer will ask about.@@ -237,11 +320,14 @@ report tells you how large the effect is in the meantime.```benchmark/download.py COCO fetcher + 3DPW presence check- eval_3dpw.py 3D: MPJPE, PA-MPJPE, PCK3D, AUC, per-sequence breakdown+ make_3dpw_gt.py SMPL forward + H36M regressor -> <3DPW root>/gt_h36m/+ fit_adapter_mhr.py MHR70 -> J14 adapter from TEACHER labels (student-free)+ eval_3dpw.py 3D from an ONNX graph: MPJPE, PA-MPJPE, PCK3D, AUC+ eval_3dpw_ckpt.py the same, for a .pth straight out of runs/eval_coco.py 2D: OKS AP/AR, mean OKS, PCK, NME, per-keypoint breakdownbenchlib/- joints.py MHR70 <-> COCO17 / SMPL24 index maps, J14 & J12 sets+ joints.py MHR70 <-> COCO17 / SMPL24 / H36M17 maps, J14 & J12 setsmetrics.py Procrustes alignment and the metric definitions- threedpw.py 3DPW pickle loading, projection, box derivation+ threedpw.py 3DPW pickle loading, the GT switch, projection, boxesrunner.py batched ONNX inference with overlapped image decoding```diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.pyindex 4ec8bf5..31450c8 100644--- a/benchmark/eval_3dpw.py+++ b/benchmark/eval_3dpw.py@@ -100,6 +100,23 @@ def evaluate(pred_mhr: np.ndarray, gt_raw: np.ndarray, set_name: str,)+def fit_linear_adapter(X_mhr: np.ndarray, Y_target: np.ndarray) -> np.ndarray:+ """Least-squares map (J, 70) from MHR70 keypoints onto any target joint set.++ Both sides are centred per sample, so the map is bias-free and equivariant+ to rotation and translation; see ``fit_adapter`` for why the centring is in+ float64 and why the singular values are truncated.+ """+ X = X_mhr.astype(np.float64)+ Y = Y_target.astype(np.float64)+ X = X - X.mean(axis=1, keepdims=True)+ Y = Y - Y.mean(axis=1, keepdims=True)+ A = X.transpose(0, 2, 1).reshape(-1, X.shape[1])+ B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1])+ W, *_ = np.linalg.lstsq(A, B, rcond=RCOND)+ return W.T++def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,gt_src: str) -> np.ndarray:"""Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.@@ -124,15 +141,8 @@ def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,+-2500. The fitted map still scores ~37 mm in sample in float64 and blows upto 430 mm in float32."""- gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :].astype(np.float64)- pred_mhr = pred_mhr.astype(np.float64)- X = pred_mhr - pred_mhr.mean(axis=1, keepdims=True) # (N, 70, 3)- Y = gt - gt.mean(axis=1, keepdims=True) # (N, 14, 3)- # Flatten samples and coordinates into the row dimension.- A = X.transpose(0, 2, 1).reshape(-1, X.shape[1]) # (N*3, 70)- B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1]) # (N*3, 14)- W, *_ = np.linalg.lstsq(A, B, rcond=RCOND)- return W.T # (14, 70)+ gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :]+ return fit_linear_adapter(pred_mhr, gt) # (14, 70)def evaluate_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,diff --git a/benchmark/eval_3dpw_ckpt.py b/benchmark/eval_3dpw_ckpt.pyindex 37f6749..9a5fc33 100644--- a/benchmark/eval_3dpw_ckpt.py+++ b/benchmark/eval_3dpw_ckpt.py@@ -118,7 +118,7 @@ def main():num_workers=args.num_workers, pin_memory=True)report = {}- hdr = (f"\n {'run':<8s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "+ hdr = (f"\n {'run':<24s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "f"{'J12 MPJPE':>10s} {'J14 PA':>8s} {'J14 MPJPE':>10s}"+ (f" {'J14+ad PA':>10s}" if W_adapt is not None else ""))print(hdr)@@ -146,7 +146,7 @@ def main():run = Path(ck_path).parts[-3]name = Path(ck_path).stem.replace("best_student_model_", "")- line = (f" {run:<8s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "+ line = (f" {run:<24s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "f"{rows['J12']['PA_MPJPE_mm']:8.2f} {rows['J12']['MPJPE_mm']:10.2f} "f"{rows['J14']['PA_MPJPE_mm']:8.2f} {rows['J14']['MPJPE_mm']:10.2f}")if W_adapt is not None:diff --git a/benchmark/results/adapter_smpl24_teacher.npz b/benchmark/results/adapter_smpl24_teacher.npznew file mode 100644index 0000000..84b2041Binary files /dev/null and b/benchmark/results/adapter_smpl24_teacher.npz differdiff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.mdindex 6e646cc..2781d64 100644--- a/instanthmr_distill_train/README.md+++ b/instanthmr_distill_train/README.md@@ -95,6 +95,23 @@ non-finger keypoints, reported next to the existing metrics. **Selection isunchanged** — and with `--val-3dpw` on, both are replaced by the 3DPW J12numbers anyway.+## `--val-3dpw-gt` (added 2026-09-05)++3DPW validation now scores against the published-protocol ground truth — SMPL+run forward on the sequence's `poses`/`betas`, then the Human3.6M regressor —+instead of the pickles' raw `jointPositions`. Two things follow.++`ThreeDPWValSet` needs the precomputed reference next to `sequenceFiles/`, built+once by `benchmark/make_3dpw_gt.py` and rsynced to the cluster as+`$SCRATCH/instanthmr/3dpw/gt_h36m/`. It is 14 MB for all three splits. Without+it the job raises at startup rather than silently falling back.++The two references are ~13 mm apart on J12 PA-MPJPE, so **a selection metric+from before this change cannot be compared with one from after**. A resumed run+must keep the metric it started on: `--val-3dpw-gt jointpositions`, or+`DPW_GT=jointpositions` for `52_train_ddp.slurm`. `run_config.json` records+which was used.+## The pair index`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` per