Skip to content
Python

3DPW adapter truncation + teacher-fitted adapter; document the GT change and the EMDB plan

mohamdev52a69b5ee414
diff --git a/CLAUDE.md b/CLAUDE.md
index 0aeccc6..888c073 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -76,7 +76,7 @@ Docs are the source of truth; the code is bigger than any context window.
| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` and `--losses rebalanced` change, the pair-index cache |
| 2 | `datasets_pipeline/jeanzay/STATUS.md` | Jean Zay cluster + data state, what is built, job templates *(git-excluded, local only)* |
| 3 | `datasets_pipeline/JEAN_ZAY.md` | site runbook: env, quotas, proxy quirks *(git-excluded, local only)* |
-| 4 | `benchmark/README.md` | how published numbers are produced; J12 vs J14, and **why the current 3DPW GT is not yet the published protocol** |
+| 4 | `benchmark/README.md` | how published numbers are produced; the 3DPW GT (SMPL forward + H36M regressor), the MHR->J14 adapter, and why **J14+adapter** is the row to quote |
| 5 | `datasets_pipeline/README.md` | how the corpus is built, including the SA-1B visibility mask |
| 6 | `docs/architecture.md` | the student model |
@@ -118,6 +118,24 @@ loss and metric in the trainer is blind to them.
it bare, so any buffer you register lands on the CPU and raises on step 1. Build
tensors on `mhr_module.device` instead.
+**3DPW GT is the published protocol, and it is a cache on disk.** The pickles'
+`jointPositions` field is the 24 SMPL *kinematic* joints, which is not what any
+paper reports; `benchmark/make_3dpw_gt.py` precomputes the real reference (SMPL
+forward + `J_regressor_h36m`) into `<3DPW root>/gt_h36m/`, and both the harness
+and `val3dpw.py` default to it. Two consequences: a job whose 3DPW directory
+lacks that cache raises at startup, and **numbers from before 2026-09-05 are ~13
+mm apart from later ones** — `--val-3dpw-gt jointpositions` keeps a resumed run
+on its original metric.
+
+**Anything fitted over the 70 MHR keypoints needs float64 and a truncated
+`lstsq`.** 40 of the 70 are finger joints millimetres apart, so the design
+matrix is near-singular. Centring in float32 leaves a 0.04 mm residual in the
+exact direction centring annihilates, `lstsq` keeps it as signal, and the J14
+adapter went from 37 mm to 430 mm in-sample looking like a broken GT. Even in
+float64 the unregularised fit reaches landmarks through ±2000 weights on
+fingers; `RCOND` in `eval_3dpw.fit_adapter` truncates it to a rank-24 map with
+weights under 1 at no measurable cost.
+
**Seed spread is wide.** Observed 70-keypoint PA-MPJPE varies 34–49 mm across
seeds, which is larger than several of the changes on the roadmap. A single-run
comparison proves nothing; use 2–3 seeds.
diff --git a/benchmark/README.md b/benchmark/README.md
index ece42b7..4161df4 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -28,13 +28,62 @@ MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.
- **Human3.6M** — the licence forbids redistribution, the download is manual and
slow, and protocol variants (P1 vs P2, 17 vs 14 joints) differ per paper. High
effort, high chance of an unfair comparison.
-- **EMDB** — a genuinely good modern benchmark (EMDB-1 for pose), but needs
- registration *and* the SMPL body model *and* their toolkit to unpack. Worth
- adding as a second 3D benchmark after 3DPW is in place; `benchlib/threedpw.py`
- is the template to copy.
-- **RICH** — SMPL-X GT, large download, mostly used for contact estimation.
+- **RICH** — the reason to skip it is not the size alone. Its GT is **SMPL-X**,
+ so it needs the SMPL-X models *and* a second forward pass (different joint
+ count, hand and face parameters) that none of our SMPL code covers; the test
+ split alone is 125K images at 4K, against EMDB-1's few thousand; and NLF
+ *trains* on RICH (the ∗ in SAM 3D Body's Table 2), so the column is not
+ out-of-domain for every method in it.
- **MPII** — 2D only, so it adds little once COCO is in.
+### Next: EMDB — waiting on registration (asked 2026-09-05)
+
+**Status: blocked on the application at `emdb.ait.ethz.ch`** (a form, approved
+by email). Nothing else is missing — the two blockers this file used to list,
+the SMPL body model and their toolkit, are gone: `benchmark/data/smpl/` holds
+the gendered models and the toolkit is only a viewer, not a format we need.
+
+Use **EMDB-1** (17 sequences, 13.5 min), the camera-frame pose split every paper
+reports. EMDB-2 (25 sequences) is global-trajectory and not what this model does.
+
+It is structurally a simpler 3DPW, one subject per sequence:
+
+| EMDB pkl | 3DPW equivalent |
+|---|---|
+| `smpl{poses_root (N,3), poses_body (N,69), betas (10,), trans}` | `poses` (N,72), `betas`, `trans` |
+| `camera{intrinsics, extrinsics (N,4,4), width, height}` | `cam_intrinsics`, `cam_poses` |
+| `good_frames_mask` | `campose_valid` |
+| `bboxes` (N,4) xyxy — **provided** | derived from projected joints |
+
+`betas` is 10-dim, exactly the count the published protocol uses, so
+`make_3dpw_gt.py`'s forward pass drops in unchanged.
+
+**The protocol is the 24 SMPL joints, mid-hip aligned — not the H36M
+regressor.** NLF's Table 3 reads "Results on EMDB1 (24j)" and its metrics
+section states that "on 3DPW and EMDB the reference point is the midpoint
+between the two hip joints"; SAM 3D Body's Table 2 column header is "EMDB (24)".
+Those 24 joints are what `smpl_forward` already returns as its second output.
+
+Two things already done, so the integration is a loader and nothing else:
+
+- `benchmark/results/adapter_smpl24_teacher.npz` — the MHR70 → SMPL-24
+ conversion, fitted the same student-free way as the J14 one, from 22,209
+ teacher/GT pairs on 3DPW train (where `jointPositions` **is** the SMPL-24 set,
+ verified to 0.001 mm). Reads anatomically: `pelvis = 0.38 right_hip + 0.34
+ left_hip`, `left_knee = 0.93 left_knee`, `left_ankle` from heel + ankle + toe.
+- **The conversion floor: 25.11 mm MPJPE / 18.77 mm PA-MPJPE** on a held-out
+ half. That is what an MHR-rigged model carries on a 24-joint benchmark before
+ it makes a single mistake — an *upper* bound, since it also contains the
+ teacher's own 3DPW error, which cannot be separated without perfect MHR GT.
+ Quote it next to the result: the table it joins runs 61.7 (SAM 3D Body) to
+ 118.5 (HMR2.0b) MPJPE, so ~25 mm of rig overhead is material.
+
+What is left is `benchlib/emdb.py`, ~80 lines mirroring `threedpw.py`. It was
+deliberately **not** written ahead of the data: `dataset.md` does not say
+whether `extrinsics` is world→camera or camera→world, and guessing wrong gives
+quietly wrong numbers rather than a crash. Thirty seconds with a real pkl
+settles it.
+
## Setup
```bash
@@ -70,6 +119,15 @@ compares them to `jointPositions`: **0.001 mm max over 74,620 person-frames**,
which is what certifies the forward pass, the gendered model choice and the
world frame all the way through.
+Then fit the MHR70 → J14 adapter once. It is student-free and dataset-wide, so
+this is a one-time step, not something to repeat per checkpoint:
+
+```bash
+python benchmark/fit_adapter_mhr.py \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --out benchmark/results/adapter_j14_h36m_teacher.npz
+```
+
## Running
```bash
@@ -85,15 +143,40 @@ python benchmark/eval_3dpw.py \
--image-root /path/to/3DPW/imageFiles \
--exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations
-# same thing for a training checkpoint instead of an exported graph
+# same thing for a training checkpoint instead of an exported graph. --adapter
+# adds the J14+adapter row, which is the one to quote.
python benchmark/eval_3dpw_ckpt.py \
- --ckpt instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth \
+ --ckpt instanthmr_distill_train/runs/*/*/best_student_model_v3.pth \
--sequence-dir /path/to/3DPW/sequenceFiles \
- --image-root /path/to/3DPW/imageFiles --split test
+ --image-root /path/to/3DPW/imageFiles --split test \
+ --adapter benchmark/results/adapter_j14_h36m_teacher.npz
```
Both write a JSON report to `benchmark/results/`.
+## Current numbers (2026-09-05)
+
+3DPW **test**, all 35,463 person-frames, published-protocol GT, the two Jean Zay
+checkpoints pulled at ~epoch 130. Full reports in
+`benchmark/results/3dpw_test_{b3_s1,v3_s1}_teacher_adapter.json`.
+
+| run | preset | J12 PA | J14 PA | **J14+adapter PA** | J14+adapter MPJPE |
+|---|---|---|---|---|---|
+| b3_s1 | baseline | 62.49 | 67.89 | **41.83** | 69.03 |
+| v3_s1 | v2 | 63.20 | 69.05 | **43.23** | 70.59 |
+
+**Quote the adapter row.** Raw J14 is dominated by convention, not quality:
+per-joint it is 131 mm at `head` and 98/87 mm at the hips against ~47 mm at the
+knees.
+
+Two caveats before this goes in a table. The 1.4 mm gap between the two runs is
+**not** a result — measured seed spread in this repo is far wider, so
+baseline-vs-v2 needs 2-3 seeds per arm through the same path. And these are the
+only numbers here that are genuinely held out: both checkpoints were trained on
+Jean Zay, whose corpus is coco/mpii/aic/sa1b/harmony4d with `3dpw_train`
+annotations-only, so all 24 test sequences are unseen. The local `data/` corpus
+is **not** clean — see the contamination section.
+
## Protocols, stated precisely
Write these into the paper — they are the details a reviewer will ask about.
@@ -237,11 +320,14 @@ report tells you how large the effect is in the meantime.
```
benchmark/
download.py COCO fetcher + 3DPW presence check
- eval_3dpw.py 3D: MPJPE, PA-MPJPE, PCK3D, AUC, per-sequence breakdown
+ make_3dpw_gt.py SMPL forward + H36M regressor -> <3DPW root>/gt_h36m/
+ fit_adapter_mhr.py MHR70 -> J14 adapter from TEACHER labels (student-free)
+ eval_3dpw.py 3D from an ONNX graph: MPJPE, PA-MPJPE, PCK3D, AUC
+ eval_3dpw_ckpt.py the same, for a .pth straight out of runs/
eval_coco.py 2D: OKS AP/AR, mean OKS, PCK, NME, per-keypoint breakdown
benchlib/
- joints.py MHR70 <-> COCO17 / SMPL24 index maps, J14 & J12 sets
+ joints.py MHR70 <-> COCO17 / SMPL24 / H36M17 maps, J14 & J12 sets
metrics.py Procrustes alignment and the metric definitions
- threedpw.py 3DPW pickle loading, projection, box derivation
+ threedpw.py 3DPW pickle loading, the GT switch, projection, boxes
runner.py batched ONNX inference with overlapped image decoding
```
diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.py
index 4ec8bf5..31450c8 100644
--- a/benchmark/eval_3dpw.py
+++ b/benchmark/eval_3dpw.py
@@ -100,6 +100,23 @@ def evaluate(pred_mhr: np.ndarray, gt_raw: np.ndarray, set_name: str,
)
+def fit_linear_adapter(X_mhr: np.ndarray, Y_target: np.ndarray) -> np.ndarray:
+ """Least-squares map (J, 70) from MHR70 keypoints onto any target joint set.
+
+ Both sides are centred per sample, so the map is bias-free and equivariant
+ to rotation and translation; see ``fit_adapter`` for why the centring is in
+ float64 and why the singular values are truncated.
+ """
+ X = X_mhr.astype(np.float64)
+ Y = Y_target.astype(np.float64)
+ X = X - X.mean(axis=1, keepdims=True)
+ Y = Y - Y.mean(axis=1, keepdims=True)
+ A = X.transpose(0, 2, 1).reshape(-1, X.shape[1])
+ B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1])
+ W, *_ = np.linalg.lstsq(A, B, rcond=RCOND)
+ return W.T
+
+
def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
gt_src: str) -> np.ndarray:
"""Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.
@@ -124,15 +141,8 @@ def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
+-2500. The fitted map still scores ~37 mm in sample in float64 and blows up
to 430 mm in float32.
"""
- gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :].astype(np.float64)
- pred_mhr = pred_mhr.astype(np.float64)
- X = pred_mhr - pred_mhr.mean(axis=1, keepdims=True) # (N, 70, 3)
- Y = gt - gt.mean(axis=1, keepdims=True) # (N, 14, 3)
- # Flatten samples and coordinates into the row dimension.
- A = X.transpose(0, 2, 1).reshape(-1, X.shape[1]) # (N*3, 70)
- B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1]) # (N*3, 14)
- W, *_ = np.linalg.lstsq(A, B, rcond=RCOND)
- return W.T # (14, 70)
+ gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :]
+ return fit_linear_adapter(pred_mhr, gt) # (14, 70)
def evaluate_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
diff --git a/benchmark/eval_3dpw_ckpt.py b/benchmark/eval_3dpw_ckpt.py
index 37f6749..9a5fc33 100644
--- a/benchmark/eval_3dpw_ckpt.py
+++ b/benchmark/eval_3dpw_ckpt.py
@@ -118,7 +118,7 @@ def main():
num_workers=args.num_workers, pin_memory=True)
report = {}
- hdr = (f"\n {'run':<8s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "
+ hdr = (f"\n {'run':<24s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "
f"{'J12 MPJPE':>10s} {'J14 PA':>8s} {'J14 MPJPE':>10s}"
+ (f" {'J14+ad PA':>10s}" if W_adapt is not None else ""))
print(hdr)
@@ -146,7 +146,7 @@ def main():
run = Path(ck_path).parts[-3]
name = Path(ck_path).stem.replace("best_student_model_", "")
- line = (f" {run:<8s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "
+ line = (f" {run:<24s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "
f"{rows['J12']['PA_MPJPE_mm']:8.2f} {rows['J12']['MPJPE_mm']:10.2f} "
f"{rows['J14']['PA_MPJPE_mm']:8.2f} {rows['J14']['MPJPE_mm']:10.2f}")
if W_adapt is not None:
diff --git a/benchmark/results/adapter_smpl24_teacher.npz b/benchmark/results/adapter_smpl24_teacher.npz
new file mode 100644
index 0000000..84b2041
Binary files /dev/null and b/benchmark/results/adapter_smpl24_teacher.npz differ
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index 6e646cc..2781d64 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -95,6 +95,23 @@ non-finger keypoints, reported next to the existing metrics. **Selection is
unchanged** — and with `--val-3dpw` on, both are replaced by the 3DPW J12
numbers anyway.
+## `--val-3dpw-gt` (added 2026-09-05)
+
+3DPW validation now scores against the published-protocol ground truth — SMPL
+run forward on the sequence's `poses`/`betas`, then the Human3.6M regressor —
+instead of the pickles' raw `jointPositions`. Two things follow.
+
+`ThreeDPWValSet` needs the precomputed reference next to `sequenceFiles/`, built
+once by `benchmark/make_3dpw_gt.py` and rsynced to the cluster as
+`$SCRATCH/instanthmr/3dpw/gt_h36m/`. It is 14 MB for all three splits. Without
+it the job raises at startup rather than silently falling back.
+
+The two references are ~13 mm apart on J12 PA-MPJPE, so **a selection metric
+from before this change cannot be compared with one from after**. A resumed run
+must keep the metric it started on: `--val-3dpw-gt jointpositions`, or
+`DPW_GT=jointpositions` for `52_train_ddp.slurm`. `run_config.json` records
+which was used.
+
## The pair index
`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` per