Skip to content
Python

3DPW GT: SMPL forward + H36M regressor, replacing raw jointPositions

mohamdev7f51828b467c
diff --git a/benchmark/README.md b/benchmark/README.md
index bd340f9..b79cce3 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -7,16 +7,18 @@ to a credible table — one 3D, one 2D:
| Benchmark | What it measures | Data cost | Status |
|---|---|---|---|
-| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT, free licence form | ready — **but the test split is contaminated, see below** |
+| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT + 500 MB SMPL | ready — published-protocol GT; **check contamination below** |
| **COCO** val2017 | OKS AP / PCK, 2D keypoint localisation in the wild | 820 MB, direct download, no registration | ready — clean |
## Why these two
**3DPW** is the benchmark essentially every monocular HMR paper reports, so it
-is the one that makes your numbers comparable. It is also unusually cheap here:
-the GT pickles store `jointPositions` (24 SMPL joints in world coordinates)
-directly, so **no SMPL model download or body-model forward pass is needed** —
-which is what makes most 3DPW harnesses painful to set up.
+is the one that makes your numbers comparable. The GT pickles store
+`jointPositions` (24 SMPL joints in world coordinates) directly, which is
+tempting and wrong — published numbers use SMPL run *forward* on the GT
+`poses`/`betas` with the Human3.6M regressor applied to the vertices. The two
+references are **46 mm apart in J12 PA-MPJPE**, so the harness builds the real
+one once with `make_3dpw_gt.py` and scores against that.
**COCO val2017** needs no registration at all and the joint mapping is free:
MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.
@@ -47,6 +49,27 @@ python benchmark/download.py --check-3dpw \
--image-root /path/to/3DPW/imageFiles
```
+Then build the ground truth. This needs two files that cannot be redistributed
+here — put both in `benchmark/data/smpl/` (gitignored):
+
+| file | where |
+|---|---|
+| `SMPL_{MALE,FEMALE}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so the gendered pair is required. |
+| `J_regressor_h36m.npy` | `visiondata.cis.upenn.edu/spin/data.tar.gz` (SPIN); also ships inside HMR2.0 / 4D-Humans |
+
+```bash
+python benchmark/make_3dpw_gt.py \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --smpl-dir benchmark/data/smpl --check
+```
+
+That writes `<3DPW root>/gt_h36m/{test,validation,train}.npz` — 17 H36M joints
+per person-frame in the same world frame as `jointPositions`, 14 MB for all
+three splits. `--check` additionally regresses the SMPL kinematic joints and
+compares them to `jointPositions`: **0.001 mm max over 74,620 person-frames**,
+which is what certifies the forward pass, the gendered model choice and the
+world frame all the way through.
+
## Running
```bash
@@ -61,6 +84,12 @@ python benchmark/eval_3dpw.py \
--sequence-dir /path/to/3DPW/sequenceFiles \
--image-root /path/to/3DPW/imageFiles \
--exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations
+
+# same thing for a training checkpoint instead of an exported graph
+python benchmark/eval_3dpw_ckpt.py \
+ --ckpt instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --image-root /path/to/3DPW/imageFiles --split test
```
Both write a JSON report to `benchmark/results/`.
@@ -82,51 +111,66 @@ comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits no
per-keypoint confidence, so every keypoint is scored 1.0 — read AP next to the
confidence-free `mean_OKS` and `PCK` numbers.
-### The GT-joint caveat — READ THIS BEFORE QUOTING A 3DPW NUMBER
+### The ground truth — what it is, and what it used to be
-**The ground truth this harness uses is not the one 3DPW papers report on.**
-`benchlib/threedpw.py` reads `jointPositions` straight from the sequence
-pickles: the 24 **SMPL kinematic-tree** joints. Every published 3DPW number —
+The harness scores against the **published-protocol GT**: gendered SMPL run
+forward on each frame's `poses`/`betas`/`trans`, then `J_regressor_h36m.npy`
+applied to the 6890 vertices. This is the reference every 3DPW paper uses —
NLF's included ("obtained through the same Human3.6M-style joint regressor that
-all prior works use") — instead runs SMPL *forward* on the GT `poses` / `betas`
-and applies the H36M joint regressor to the resulting 6890 vertices. Those are
-different landmarks, most visibly at the shoulders and hips, which is 4 of the
-12 joints in J12.
-
-That is what makes this harness cheap to set up (see "Why these two" above) and
-it is fine for *tracking a run*. It is **not** comparable to a published table.
-
-Fixing it needs two files that are not in this repo:
+all prior works use"). `benchmark/make_3dpw_gt.py` precomputes it; `--gt
+jointpositions` restores the pickles' raw `jointPositions` field, which is what
+this harness used before and is **not** comparable to any published table.
-| file | where |
-|---|---|
-| `SMPL_{MALE,FEMALE,NEUTRAL}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so get the gendered pair. |
-| `J_regressor_h36m.npy` | ships inside SPIN / HMR2.0 / 4D-Humans |
+The difference is not cosmetic. Over 1,437 test person-frames, the same
+landmarks under the two references:
-Then: `V = SMPL(poses, betas)` per frame, `J14 = J_regressor @ V`, and fit the
-existing `--fit-adapter` linear MHR70 -> J14 map on 3DPW **train** against that
-GT before reporting on test. A forward pass, not a fit.
-
-`v_template_clothed` in the pickles cannot substitute — it is a static T-pose
-surface with no blend weights, no pose blendshapes and no joint regressor, so it
-cannot be posed.
-
-This does **not** affect `instanthmr_distill_train/val3dpw.py`, which uses the
-same GT for checkpoint selection: selection only needs a consistent metric, not
-a comparable one.
+| | raw | root-relative | after Procrustes |
+|---|---|---|---|
+| J14 mean | 64.2 mm | 86.3 mm | 47.2 mm |
+| J12 mean | 63.6 mm | 88.1 mm | 46.0 mm |
+| right hip | 154.5 mm | 86.8 mm | 114.5 mm |
+| head | 116.3 mm | 42.0 mm | 62.2 mm |
+
+The hips dominate, because the two conventions disagree about where a hip is:
+H36M's are marker-derived and **292 mm apart**, SMPL's kinematic ones sit near
+the femoral heads, **120 mm apart**. Shoulder width goes the other way (303 vs
+387 mm), and H36M's `head` is the top of the skull where SMPL's is inside it
+(neck→head 192 vs 100 mm).
+
+Two details worth keeping straight:
+
+- **`jointPositions` is the kinematic joint, not the regressed one.** Comparing
+ `J_regressor @ posed vertices` to it gives a 4.5 mm residual and looks like a
+ bug in the forward pass. The joints that match are the ones carried by the
+ global transforms. `make_3dpw_gt.py --check` compares those, and gets 0.001 mm.
+- **`betas` is 300-dim for 48 of the 87 subject tracks and 10-dim for the other
+ 39, and entries 10: are all exactly zero.** So `--num-betas 10`, the published
+ protocol, is also lossless here.
+
+`v_template_clothed` in the pickles is not a substitute for any of this — it is
+a static T-pose surface with no blend weights and no pose blendshapes, so it
+cannot be posed. Using it makes the residual worse (17.9 mm), not better.
+
+Selection inside `instanthmr_distill_train/val3dpw.py` follows the same default,
+so **3DPW numbers from before and after this change are not comparable** — the
+new GT is several mm harsher. The cache has to be rsynced next to the cluster's
+`sequenceFiles/` or the trainer raises on startup.
### The joint-convention caveat — do not skip this
-InstantHMR predicts MHR70 joints; 3DPW's GT is SMPL. The two rigs place several
-landmarks differently, which puts a floor under the error that has nothing to do
-with model quality. The harness therefore reports three rows:
-
-- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` are the
- two joints where the rigs genuinely disagree (MHR has no head-top; `nose` is
- substituted), so this row carries a systematic penalty.
-- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles), where
- both rigs mean the same anatomical point. **This is the cleanest measure of
- the model.**
+InstantHMR predicts MHR70 joints; the GT is now in the H36M convention. The two
+place several landmarks differently, which puts a floor under the error that has
+nothing to do with model quality. The harness therefore reports three rows:
+
+- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` carry
+ the largest systematic penalty (MHR has no head-top; `nose` is substituted).
+- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles). Against
+ `jointpositions` this was the clean row, because MHR and SMPL agree on those
+ landmarks. Against the H36M GT it is **not** clean: the hips alone move 292 vs
+ 120 mm apart between the conventions. Read it as a consistent tracking metric,
+ not as a rig-free measure of the model.
+- **J14+adapter** is therefore the row to compare against a published table,
+ not an optional extra.
- **J14+adapter** *(optional)* — a linear MHR70 → SMPL-J14 joint regressor,
fitted on 3DPW **train**, that absorbs the rig offset. Standard practice when
a method's joint convention differs from the benchmark's. Fit it once, then
diff --git a/benchmark/benchlib/joints.py b/benchmark/benchlib/joints.py
index f4e08dd..331d355 100644
--- a/benchmark/benchlib/joints.py
+++ b/benchmark/benchlib/joints.py
@@ -60,6 +60,25 @@ SMPL24_NAMES = [
]
SMPL = {n: i for i, n in enumerate(SMPL24_NAMES)}
+# --------------------------------------------------------------------------
+# Human3.6M 17 joints — the rows of ``J_regressor_h36m.npy``
+# --------------------------------------------------------------------------
+# This is the GT every published 3DPW number is measured against: SMPL run
+# forward on the sequence's poses/betas, then this regressor applied to the
+# 6890 vertices. The row order below was read off the data, not a table — each
+# row was matched to its nearest SMPL kinematic joint on a real sequence — and
+# it reproduces SPIN's ``H36M_TO_J14 = [6,5,4,1,2,3,16,15,14,11,12,13,8,10]``
+# exactly. Note it is *left-first* in the legs and arms, which is the opposite
+# of how H36M's own raw skeleton is usually tabulated.
+H36M17_NAMES = [
+ "pelvis", "left_hip", "left_knee", "left_ankle",
+ "right_hip", "right_knee", "right_ankle",
+ "spine", "neck", "nose", "head",
+ "left_shoulder", "left_elbow", "left_wrist",
+ "right_shoulder", "right_elbow", "right_wrist",
+]
+H36M = {n: i for i, n in enumerate(H36M17_NAMES)}
+
# --------------------------------------------------------------------------
# 3D evaluation joint sets
# --------------------------------------------------------------------------
@@ -80,17 +99,29 @@ _MHR_FOR_J14 = dict(MHR, head=MHR["nose"])
J14_FROM_MHR70 = np.array([_MHR_FOR_J14[n] for n in J14_NAMES], dtype=np.int64)
J14_FROM_SMPL24 = np.array([SMPL[n] for n in J14_NAMES], dtype=np.int64)
+J14_FROM_H36M17 = np.array([H36M[n] for n in J14_NAMES], dtype=np.int64)
J12_FROM_MHR70 = J14_FROM_MHR70[:12]
J12_FROM_SMPL24 = J14_FROM_SMPL24[:12]
+J12_FROM_H36M17 = J14_FROM_H36M17[:12]
# Pelvis for root alignment: the mid-hip, computed the same way on both sides.
J14_HIP_IDX = (J14_NAMES.index("left_hip"), J14_NAMES.index("right_hip"))
JOINT_SETS = {
- "J14": dict(names=J14_NAMES, mhr=J14_FROM_MHR70, smpl=J14_FROM_SMPL24),
- "J12": dict(names=J12_NAMES, mhr=J12_FROM_MHR70, smpl=J12_FROM_SMPL24),
+ "J14": dict(names=J14_NAMES, mhr=J14_FROM_MHR70,
+ smpl=J14_FROM_SMPL24, h36m=J14_FROM_H36M17),
+ "J12": dict(names=J12_NAMES, mhr=J12_FROM_MHR70,
+ smpl=J12_FROM_SMPL24, h36m=J12_FROM_H36M17),
}
+# ``benchlib.threedpw.build_samples`` returns GT in one of two layouts; this
+# picks the matching index array out of a joint set.
+GT_LAYOUTS = {"h36m": "h36m", "jointpositions": "smpl"}
+
+
+def gt_index(spec: dict, gt: str) -> np.ndarray:
+ return spec[GT_LAYOUTS[gt]]
+
def pelvis(joints: np.ndarray) -> np.ndarray:
"""Mid-hip root for a (..., J, 3) array indexed in a J14/J12 joint set."""
diff --git a/benchmark/benchlib/threedpw.py b/benchmark/benchlib/threedpw.py
index d5a4417..8ecd1c1 100644
--- a/benchmark/benchlib/threedpw.py
+++ b/benchmark/benchlib/threedpw.py
@@ -9,8 +9,17 @@ useful fields for us:
campose_valid list[P] of (F,) 1 where the camera pose is trustworthy
genders list[P]
-Because ``jointPositions`` is stored directly, evaluating 3D joint error needs
-**no SMPL model download** — we never have to re-run the body model.
+``jointPositions`` holds the 24 SMPL *kinematic-tree* joints, which is not the
+ground truth 3DPW papers report on: those run SMPL forward on ``poses``/
+``betas`` and apply the Human3.6M regressor to the resulting vertices. That
+reference is precomputed once by ``benchmark/make_3dpw_gt.py`` into
+``<3DPW root>/gt_h36m/<split>.npz`` — 17 joints per person-frame in the same
+world frame — and is what ``build_samples`` returns by default. Pass
+``gt="jointpositions"`` to get the old, non-comparable GT back.
+
+Person boxes and the off-screen filter keep using ``jointPositions`` either
+way, so switching the GT changes only the numbers being compared, never which
+frames are evaluated or how they are cropped.
"""
from __future__ import annotations
@@ -50,10 +59,32 @@ def bbox_from_points(uv: np.ndarray, scale: float) -> np.ndarray:
dtype=np.float32)
+def gt_h36m_path(sequence_dir: str | Path, split: str,
+ gt_dir: str | Path | None = None) -> Path:
+ """Where the regressed GT lives: next to ``sequenceFiles``, not in the repo."""
+ base = Path(gt_dir) if gt_dir else Path(sequence_dir).parent / "gt_h36m"
+ return base / f"{split}.npz"
+
+
+def load_gt_h36m(sequence_dir: str | Path, split: str,
+ gt_dir: str | Path | None = None) -> dict:
+ path = gt_h36m_path(sequence_dir, split, gt_dir)
+ if not path.is_file():
+ raise FileNotFoundError(
+ f"regressed 3DPW ground truth not found: {path}\n"
+ "Build it with benchmark/make_3dpw_gt.py, or pass "
+ "gt='jointpositions' to fall back to the raw (non-comparable) "
+ "SMPL kinematic joints.")
+ with np.load(path) as z:
+ return {k: z[k] for k in z.files if "|" in k}
+
+
def build_samples(sequence_dir: str | Path, image_root: str | Path,
split: str = "test", stride: int = 1,
bbox_scale: float = 1.2,
- min_visible_frac: float = 0.6):
+ min_visible_frac: float = 0.6,
+ gt: str = "h36m",
+ gt_dir: str | Path | None = None):
"""Enumerate every evaluatable (frame, person) in a 3DPW split.
Args:
@@ -66,10 +97,19 @@ def build_samples(sequence_dir: str | Path, image_root: str | Path,
projected joints fall inside the image — they are mostly out of
frame, and a crop around them is meaningless.
+ gt: ``h36m`` for the published-protocol GT (17 regressed joints,
+ requires the cache from ``make_3dpw_gt.py``); ``jointpositions``
+ for the raw 24 SMPL kinematic joints in the pickles.
+ gt_dir: override the cache location.
+
Returns:
- ``(samples, gt_joints)`` — a list of dicts and an (N, 24, 3) array of
- camera-space GT joints in metres, index-aligned.
+ ``(samples, gt_joints)`` — a list of dicts and an (N, 17, 3) array
+ (``h36m``) or (N, 24, 3) array (``jointpositions``) of camera-space GT
+ joints in metres, index-aligned.
"""
+ if gt not in ("h36m", "jointpositions"):
+ raise ValueError(f"unknown gt source: {gt}")
+ table = load_gt_h36m(sequence_dir, split, gt_dir) if gt == "h36m" else None
seq_dir = Path(sequence_dir) / split
if not seq_dir.is_dir():
raise FileNotFoundError(f"no such 3DPW split directory: {seq_dir}")
@@ -87,6 +127,10 @@ def build_samples(sequence_dir: str | Path, image_root: str | Path,
for pid, jp in enumerate(seq["jointPositions"]):
joints_w = np.asarray(jp, dtype=np.float64).reshape(-1, 24, 3)
+ # The box always comes from the SMPL joints; only the scored GT
+ # changes, so the two sources are evaluated on identical crops.
+ gt_w = (np.asarray(table[f"{name}|{pid}"], dtype=np.float64)
+ if table is not None else joints_w)
valid = np.asarray(seq["campose_valid"][pid]).astype(bool)
for f in range(0, min(num_frames, joints_w.shape[0]), stride):
@@ -106,7 +150,7 @@ def build_samples(sequence_dir: str | Path, image_root: str | Path,
sequence=name, person=pid, frame=f,
gender=str(seq["genders"][pid]),
))
- gts.append(joints_cam)
+ gts.append(gt_w[f] @ T[:3, :3].T + T[:3, 3])
if not samples:
raise RuntimeError(f"no valid samples found in {seq_dir}")
diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.py
index 33f770f..71d37dc 100644
--- a/benchmark/eval_3dpw.py
+++ b/benchmark/eval_3dpw.py
@@ -1,11 +1,14 @@
#!/usr/bin/env python3
"""3DPW test-set evaluation for InstantHMR (MPJPE / PA-MPJPE / PCK / AUC).
-The GT comes straight from the 3DPW pickles' ``jointPositions`` field, so no
-SMPL model download is needed. Predicted MHR70 joints are mapped onto the
-SMPL joint convention by name; see ``benchlib/joints.py`` for the caveat about
-``neck`` and ``head``, which are the only two joints where the two rigs disagree
-on the underlying anatomy.
+By default the GT is the published protocol: SMPL run forward on the sequence's
+poses/betas and the Human3.6M regressor applied to the vertices, precomputed by
+``benchmark/make_3dpw_gt.py``. ``--gt jointpositions`` falls back to the raw
+SMPL kinematic joints in the pickles, which is what this harness used before
+and is not comparable to any published table. Predicted MHR70 joints are mapped
+onto the joint set by name; see ``benchlib/joints.py`` for the caveat about
+``neck`` and ``head``, which are the only two joints where the rigs disagree on
+the underlying anatomy.
Usage:
python benchmark/eval_3dpw.py \
@@ -47,6 +50,12 @@ def parse_args():
help="evaluate every Nth frame (1 = full protocol)")
p.add_argument("--bbox-scale", type=float, default=1.2,
help="padding on the projected-GT-joint person box")
+ p.add_argument("--gt", default="h36m", choices=["h36m", "jointpositions"],
+ help="h36m: SMPL forward + H36M regressor, the published "
+ "protocol. jointpositions: the raw pickle field.")
+ p.add_argument("--gt-dir", default=None,
+ help="where make_3dpw_gt.py wrote its npz "
+ "(default: <sequence-dir>/../gt_h36m)")
p.add_argument("--device", default="cuda")
p.add_argument("--mhr-script", default=None)
p.add_argument("--num-workers", type=int, default=6)
@@ -64,11 +73,12 @@ def parse_args():
return p.parse_args()
-def evaluate(pred_mhr: np.ndarray, gt_smpl: np.ndarray, set_name: str) -> dict:
- """Metrics for one joint set. Inputs are (N, 70, 3) and (N, 24, 3) in mm."""
+def evaluate(pred_mhr: np.ndarray, gt_raw: np.ndarray, set_name: str,
+ gt_src: str) -> dict:
+ """Metrics for one joint set. Inputs are (N, 70, 3) and (N, J, 3) in mm."""
spec = J.JOINT_SETS[set_name]
pred = pred_mhr[:, spec["mhr"], :]
- gt = gt_smpl[:, spec["smpl"], :]
+ gt = gt_raw[:, J.gt_index(spec, gt_src), :]
root_p, root_g = J.pelvis(pred), J.pelvis(gt)
e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
@@ -89,14 +99,23 @@ def evaluate(pred_mhr: np.ndarray, gt_smpl: np.ndarray, set_name: str) -> dict:
)
-def fit_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray) -> np.ndarray:
+def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
+ gt_src: str) -> np.ndarray:
"""Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.
Fitted on centred poses and used without a bias, so the mapping stays
equivariant to rotation and translation — it only removes the systematic
difference in where the two rigs place each landmark.
+
+ Centring must happen in float64. Doing it in float32 leaves a ~0.04 mm
+ residual in the all-ones direction, which is exactly the direction the
+ centring annihilates; ``lstsq`` then sees a singular value of 3e-7 relative
+ instead of zero, keeps it, and loads that null direction with weights of
+ +-2500. The fitted map still scores ~37 mm in sample in float64 and blows up
+ to 430 mm in float32.
"""
- gt = gt_smpl[:, J.J14_FROM_SMPL24, :]
+ gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :].astype(np.float64)
+ pred_mhr = pred_mhr.astype(np.float64)
X = pred_mhr - pred_mhr.mean(axis=1, keepdims=True) # (N, 70, 3)
Y = gt - gt.mean(axis=1, keepdims=True) # (N, 14, 3)
# Flatten samples and coordinates into the row dimension.
@@ -106,10 +125,14 @@ def fit_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray) -> np.ndarray:
return W.T # (14, 70)
-def evaluate_adapter(pred_mhr: np.ndarray, gt_smpl: np.ndarray,
- W: np.ndarray) -> dict:
- pred = np.einsum("jk,nkc->njc", W, pred_mhr)
- gt = gt_smpl[:, J.J14_FROM_SMPL24, :]
+def evaluate_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
+ W: np.ndarray, gt_src: str) -> dict:
+ # W was fitted on centred keypoints, so apply it to centred keypoints. Both
+ # metrics below are translation-invariant, so this costs nothing and makes
+ # the result independent of whatever W does to the all-ones direction.
+ centred = pred_mhr - pred_mhr.mean(axis=1, keepdims=True)
+ pred = np.einsum("jk,nkc->njc", W, centred)
+ gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :]
root_p, root_g = J.pelvis(pred), J.pelvis(gt)
e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
e_pa = M.pa_mpjpe(pred, gt)
@@ -131,6 +154,7 @@ def main():
samples, gt_joints = P.build_samples(
args.sequence_dir, args.image_root, split=args.split,
stride=args.stride, bbox_scale=args.bbox_scale,
+ gt=args.gt, gt_dir=args.gt_dir,
)
if args.limit:
samples, gt_joints = samples[:args.limit], gt_joints[:args.limit]
@@ -155,12 +179,12 @@ def main():
W_adapt = np.load(args.adapter)["W"] if args.adapter else None
def _rows(pm, gm):
- rows = [evaluate(pm, gm, s) for s in ("J14", "J12")]
+ rows = [evaluate(pm, gm, s, args.gt) for s in ("J14", "J12")]
if W_adapt is not None:
- rows.append(evaluate_adapter(pm, gm, W_adapt))
+ rows.append(evaluate_adapter(pm, gm, W_adapt, args.gt))
return rows
report = dict(
- model=str(args.onnx), dataset="3DPW", split=args.split,
+ model=str(args.onnx), dataset="3DPW", split=args.split, gt=args.gt,
stride=args.stride, bbox_scale=args.bbox_scale,
num_sequences=len(seqs), sequences=seqs,
num_samples=int(pred_mhr.shape[0]),
@@ -185,14 +209,15 @@ def main():
report["results_seen_frames"] = _rows(pred_mhr[is_seen], gt_smpl[is_seen])
if args.fit_adapter:
- W = fit_adapter(pred_mhr, gt_smpl)
+ W = fit_adapter(pred_mhr, gt_smpl, args.gt)
Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)
- np.savez(args.fit_adapter, W=W, split=args.split)
+ np.savez(args.fit_adapter, W=W, split=args.split, gt=args.gt)
print(f"[3DPW] fitted joint adapter on '{args.split}' -> {args.fit_adapter}")
# Per-sequence PA-MPJPE on the headline joint set.
spec = J.JOINT_SETS["J14"]
- e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :], gt_smpl[:, spec["smpl"], :]).mean(axis=1)
+ e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :],
+ gt_smpl[:, J.gt_index(spec, args.gt), :]).mean(axis=1)
kept = [s for s, k in zip(samples, ok) if k]
per_seq = {}
for name in seqs:
@@ -214,7 +239,7 @@ def main():
print("\n" + "=" * 60)
print(f"3DPW {args.split} | {report['num_samples']} person-frames "
- f"| {len(seqs)} sequences (mm)")
+ f"| {len(seqs)} sequences | GT: {args.gt} (mm)")
print("=" * 60)
_table("ALL frames:", report["results"])
if "results_unseen_frames" in report:
diff --git a/benchmark/eval_3dpw_ckpt.py b/benchmark/eval_3dpw_ckpt.py
new file mode 100644
index 0000000..37f6749
--- /dev/null
+++ b/benchmark/eval_3dpw_ckpt.py
@@ -0,0 +1,173 @@
+#!/usr/bin/env python3
+"""3DPW evaluation for a training checkpoint (.pth), not an ONNX export.
+
+``eval_3dpw.py`` scores exported graphs; this scores checkpoints straight out of
+``runs/``. The crop and the MHR forward pass come from ``val3dpw.py`` (which was
+cross-validated against ``eval_3dpw.py``) and the metrics are ``eval_3dpw.py``'s
+own functions, so the two scripts report the same numbers for the same weights.
+
+ # headline: published-protocol GT on the test split
+ python benchmark/eval_3dpw_ckpt.py --ckpt runs/*/best_student_model_v3.pth \
+ --sequence-dir .../sequenceFiles --image-root .../imageFiles \
+ --split test
+
+ # the MHR->J14 adapter, fitted on train and applied to test
+ python benchmark/eval_3dpw_ckpt.py --ckpt <ckpt> ... --split train \
+ --stride 10 --fit-adapter benchmark/results/adapter_j14_h36m.npz
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
+sys.path.insert(0, str(ROOT / "benchmark"))
+
+import eval_3dpw as E # noqa: E402
+import train_distill_mhr_only as T # noqa: E402
+import val3dpw # noqa: E402
+from benchlib import joints as J # noqa: E402
+
+MM = 1000.0
+
+
+def parse_args():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--ckpt", nargs="+", required=True)
+ p.add_argument("--sequence-dir", required=True)
+ p.add_argument("--image-root", required=True)
+ p.add_argument("--split", default="test")
+ p.add_argument("--stride", type=int, default=1)
+ p.add_argument("--gt", nargs="+", default=["h36m"],
+ choices=["h36m", "jointpositions"])
+ p.add_argument("--gt-dir", default=None)
+ p.add_argument("--adapter", default=None,
+ help="npz with a fitted MHR70->J14 regressor to also report")
+ p.add_argument("--fit-adapter", default=None,
+ help="fit that regressor on THIS split and write it here "
+ "(run on train, never on test)")
+ p.add_argument("--backbone", default="repvit_m2_3")
+ p.add_argument("--batch-size", type=int, default=64)
+ p.add_argument("--num-workers", type=int, default=6)
+ p.add_argument("--device", default="cuda")
+ p.add_argument("--out", default=None)
+ return p.parse_args()
+
+
+def predict(model, loader, mhr_module, device, use_amp: bool = True):
+ """(N, 70, 3) predicted keypoints in mm, plus the indices they came from.
+
+ The indices let the same forward pass be scored against more than one GT
+ source: the sample list is identical whichever GT ``build_samples`` returns,
+ only the joint values differ.
+ """
+ model.eval()
+ preds, keeps, n_dropped, base = [], [], 0, 0
+ for b in loader:
+ img = b["image"].to(device, non_blocking=True)
+ cc = b["cliff_cond"].to(device, non_blocking=True)
+ with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):
+ out = model(img, cc)
+ kp = mhr_module.get_native_keypoints(out["mhr_params"].float(),
+ out["shape_params"].float())
+ # Same guard as val3dpw.evaluate: a non-finite prediction makes the
+ # Procrustes SVD raise rather than return a large error.
+ finite = torch.isfinite(kp).flatten(1).all(dim=1).cpu()
+ keep = b["ok"].bool() & finite
+ n_dropped += int((b["ok"].bool() & ~finite).sum())
+ if bool(keep.any()):
+ preds.append(kp[keep.to(kp.device)].float().cpu().numpy())
+ keeps.append(base + np.nonzero(keep.numpy())[0])
+ base += int(keep.shape[0])
+ return (np.concatenate(preds) * MM, np.concatenate(keeps), n_dropped)
+
+
+def main():
+ args = parse_args()
+ device = torch.device(args.device)
+ cfg = T.DistillConfig(backbone=args.backbone)
+ mhr = T.MHRForwardPass(cfg.mhr_model_path, device,
+ kp_regressor=np.load(cfg.kp_regressor_path))
+ W_adapt = W_adapt_gt = None
+ if args.adapter:
+ ad = np.load(args.adapter)
+ W_adapt, W_adapt_gt = ad["W"], str(ad["gt"]) if "gt" in ad else "unknown"
+
+ # One dataset per GT source: same samples and same crops, different joint
+ # values, so a single forward pass scores against all of them.
+ sets = {}
+ for gt in args.gt:
+ ds = val3dpw.ThreeDPWValSet(args.sequence_dir, args.image_root,
+ split=args.split, stride=args.stride,
+ gt=gt, gt_dir=args.gt_dir)
+ sets[gt] = ds
+ print(f"[3DPW] {args.split} / GT={gt}: {len(ds):,} person-frames "
+ f"(stride {args.stride})", flush=True)
+ first = sets[args.gt[0]]
+ loader = torch.utils.data.DataLoader(
+ first, batch_size=args.batch_size, shuffle=False,
+ num_workers=args.num_workers, pin_memory=True)
+
+ report = {}
+ hdr = (f"\n {'run':<8s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "
+ f"{'J12 MPJPE':>10s} {'J14 PA':>8s} {'J14 MPJPE':>10s}"
+ + (f" {'J14+ad PA':>10s}" if W_adapt is not None else ""))
+ print(hdr)
+
+ for ck_path in args.ckpt:
+ model = T.InstantHMRStudent(cfg, pretrained=False).to(device)
+ ck = torch.load(ck_path, map_location="cpu", weights_only=False)
+ model.load_state_dict(ck["model_state_dict"])
+ pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp)
+ del model
+ torch.cuda.empty_cache()
+
+ for gt, ds in sets.items():
+ gt_arr = np.asarray(ds.gt, dtype=np.float64)[idx] * MM
+ rows = {s: E.evaluate(pred, gt_arr, s, gt) for s in ("J14", "J12")}
+ # An adapter absorbs the offset between the MHR rig and one specific
+ # GT convention, so scoring it against the other one is meaningless.
+ if W_adapt is not None and W_adapt_gt == gt:
+ rows["J14+adapter"] = E.evaluate_adapter(pred, gt_arr, W_adapt, gt)
+ report[f"{ck_path}|{gt}"] = dict(
+ ckpt=ck_path, gt=gt, split=args.split, stride=args.stride,
+ epoch=ck.get("epoch"), source=ck.get("source"),
+ n=int(pred.shape[0]), n_dropped=n_dropped,
+ results=list(rows.values()))
+
+ run = Path(ck_path).parts[-3]
+ name = Path(ck_path).stem.replace("best_student_model_", "")
+ line = (f" {run:<8s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "
+ f"{rows['J12']['PA_MPJPE_mm']:8.2f} {rows['J12']['MPJPE_mm']:10.2f} "
+ f"{rows['J14']['PA_MPJPE_mm']:8.2f} {rows['J14']['MPJPE_mm']:10.2f}")
+ if W_adapt is not None:
+ line += (f" {rows['J14+adapter']['PA_MPJPE_mm']:10.2f}"
+ if "J14+adapter" in rows
+ else f" {'n/a':>10s}")
+ print(line + (f" ({n_dropped} dropped)" if n_dropped else ""), flush=True)
+
+ if args.fit_adapter:
+ if args.split == "test":
+ raise SystemExit("refusing to fit the adapter on the test split")
+ Wf = E.fit_adapter(pred, gt_arr, gt)
+ Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)
+ np.savez(args.fit_adapter, W=Wf, split=args.split, gt=gt, ckpt=ck_path)
+ print(f" fitted adapter on '{args.split}' -> {args.fit_adapter}")
+
+ if args.out:
+ Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+ Path(args.out).write_text(json.dumps(report, indent=2))
+ print(f"\nwritten to {args.out}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/make_3dpw_gt.py b/benchmark/make_3dpw_gt.py
new file mode 100644
index 0000000..2da8176
--- /dev/null
+++ b/benchmark/make_3dpw_gt.py
@@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+"""Build 3DPW ground truth the way published papers do: SMPL forward + H36M regressor.
+
+The sequence pickles ship ``jointPositions`` — the 24 SMPL *kinematic-tree*
+joints — and using them directly is what makes `benchlib/threedpw.py` cheap to
+set up. It is also not the ground truth any 3DPW paper reports on. Every
+published number (SPIN, CLIFF, HMR2.0, NLF, ...) instead runs the gendered SMPL
+model forward on the GT ``poses``/``betas``/``trans`` and applies the Human3.6M
+joint regressor to the 6890 resulting vertices. The two differ most at the
+shoulders and hips, i.e. 4 of the 12 joints in J12.
+
+This script produces that reference once, as ``<out>/<split>.npz`` holding the
+17 H36M joints per (sequence, person) in the **same world frame** as
+``jointPositions`` — so everything downstream keeps using ``cam_poses`` exactly
+as before and only the joint values change.
+
+ python benchmark/make_3dpw_gt.py \
+ --sequence-dir /path/to/3DPW/sequenceFiles \
+ --smpl-dir benchmark/data/smpl \
+ --out benchmark/data/3dpw_gt_h36m
+
+Needs, once:
+ SMPL_{MALE,FEMALE}.pkl smpl.is.tue.mpg.de (free research licence)
+ J_regressor_h36m.npy SPIN's data bundle
+"""
+
+from __future__ import annotations
+
+import argparse
+import pickle
+import sys
+import types
+from pathlib import Path
+
+import numpy as np
+import torch
+
+# --------------------------------------------------------------------------
+# SMPL
+# --------------------------------------------------------------------------
+# The official pkls store several fields as chumpy arrays. chumpy does not
+# import on numpy >= 1.24 (np.bool/np.float aliases) nor on Python >= 3.11
+# (inspect.getargspec), and we only need the numeric payload, so unpickle
+# against a stub whose instances keep whatever state the pickle sets.
+
+
+class _ChStub:
+ def __array__(self, dtype=None):
+ a = np.asarray(self.__dict__["x"])
+ return a.astype(dtype) if dtype is not None else a
+
+
+def _install_chumpy_stub() -> None:
+ if "chumpy" in sys.modules:
+ return
+ ch = types.ModuleType("chumpy.ch")
+ ch.Ch = _ChStub
+ root = types.ModuleType("chumpy")
+ root.ch = ch
+ root.Ch = _ChStub
+ sys.modules["chumpy"] = root
+ sys.modules["chumpy.ch"] = ch
+
+
+def _to_numpy(v):
+ if isinstance(v, _ChStub):
+ return np.asarray(v.__dict__["x"], dtype=np.float64)
+ if hasattr(v, "toarray"): # scipy sparse J_regressor
+ return np.asarray(v.toarray(), dtype=np.float64)
+ return np.asarray(v, dtype=np.float64)
+
+
+def load_smpl(path: Path, device) -> dict:
+ _install_chumpy_stub()
+ with open(path, "rb") as f:
+ d = pickle.load(f, encoding="latin1")
+ out = dict(
+ v_template=_to_numpy(d["v_template"]),
+ shapedirs=_to_numpy(d["shapedirs"]),
+ posedirs=_to_numpy(d["posedirs"]),
+ J_regressor=_to_numpy(d["J_regressor"]),
+ weights=_to_numpy(d["weights"]),
+ parents=np.asarray(d["kintree_table"], dtype=np.int64)[0].copy(),
+ )
+ out["parents"][0] = -1
+ return {k: (torch.as_tensor(v, dtype=torch.float32, device=device)
+ if k != "parents" else torch.as_tensor(v, device=device))
+ for k, v in out.items()}
+
+
+def rodrigues(rvec: torch.Tensor) -> torch.Tensor:
+ """(..., 3) axis-angle -> (..., 3, 3) rotation matrices."""
+ theta = rvec.norm(dim=-1, keepdim=True).clamp_min(1e-12)
+ axis = rvec / theta
+ x, y, z = axis.unbind(-1)
+ zero = torch.zeros_like(x)
+ K = torch.stack([zero, -z, y, z, zero, -x, -y, x, zero], dim=-1)
+ K = K.reshape(*rvec.shape[:-1], 3, 3)
+ eye = torch.eye(3, device=rvec.device, dtype=rvec.dtype).expand_as(K)
+ s, c = torch.sin(theta)[..., None], torch.cos(theta)[..., None]
+ return eye + s * K + (1 - c) * (K @ K)
+
+
+def smpl_forward(m: dict, poses: torch.Tensor, betas: torch.Tensor,
+ trans: torch.Tensor):
+ """Standard SMPL LBS.
+
+ poses (N, 72), betas (B,), trans (N, 3) -> vertices (N, 6890, 3) and the
+ 24 kinematic-tree joints (N, 24, 3). The latter are the joint *centres*
+ carried by the global transforms, which is what 3DPW's ``jointPositions``
+ stores — not ``J_regressor @ posed vertices``, which differs by ~4.5 mm
+ because pose blendshapes and skinning move the surface.
+ """
+ N = poses.shape[0]
+ v_shaped = m["v_template"] + torch.einsum("vck,k->vc", m["shapedirs"], betas)
+ J = m["J_regressor"] @ v_shaped # (24, 3)
+
+ R = rodrigues(poses.reshape(N, 24, 3)) # (N, 24, 3, 3)
+ eye = torch.eye(3, device=R.device).expand(N, 23, 3, 3)
+ pose_feature = (R[:, 1:] - eye).reshape(N, 207)
+ v_posed = v_shaped + torch.einsum("vck,nk->nvc", m["posedirs"], pose_feature)
+
+ # Global joint transforms down the kinematic tree.
+ parents = m["parents"]
+ G = torch.zeros(N, 24, 4, 4, device=R.device)
+ G[:, :, 3, 3] = 1.0
+ G[:, 0, :3, :3], G[:, 0, :3, 3] = R[:, 0], J[0]
+ for i in range(1, 24):
+ loc = torch.zeros(N, 4, 4, device=R.device)
+ loc[:, 3, 3] = 1.0
+ loc[:, :3, :3] = R[:, i]
+ loc[:, :3, 3] = J[i] - J[parents[i]]
+ G[:, i] = G[:, parents[i]] @ loc
+
+ J_posed = G[:, :, :3, 3] + trans[:, None, :]
+
+ # Remove the rest-pose offset so the transform maps rest -> posed.
+ J_h = torch.cat([J, torch.zeros(24, 1, device=R.device)], dim=-1) # (24, 4)
+ G = G - torch.nn.functional.pad(G @ J_h[None, :, :, None], (3, 0, 0, 0))
+
+ T = torch.einsum("vj,njab->nvab", m["weights"], G) # (N, 6890, 4, 4)
+ v_h = torch.cat([v_posed, torch.ones(N, 6890, 1, device=R.device)], dim=-1)
+ verts = torch.einsum("nvab,nvb->nva", T, v_h)[..., :3]
+ return verts + trans[:, None, :], J_posed
+
+
+# --------------------------------------------------------------------------
+# Driver
+# --------------------------------------------------------------------------
+def build_split(seq_dir: Path, smpl_dir: Path, regressor: np.ndarray,
+ split: str, num_betas: int, device, chunk: int,
+ check: bool) -> dict:
+ models = {g: load_smpl(smpl_dir / f"SMPL_{g.upper()}.pkl", device)
+ for g in ("male", "female")}
+ n_dirs = models["male"]["shapedirs"].shape[-1]
+ Jreg = torch.as_tensor(regressor, dtype=torch.float32, device=device)
+
+ out: dict[str, np.ndarray] = {}
+ resid = []
+ for pkl in sorted((seq_dir / split).glob("*.pkl")):
+ with open(pkl, "rb") as f:
+ seq = pickle.load(f, encoding="latin1")
+ name = str(seq["sequence"])
+ for pid, poses in enumerate(seq["poses"]):
+ gender = str(seq["genders"][pid])
+ gender = "male" if gender.lower().startswith("m") else "female"
+ m = models[gender]
+ p = torch.as_tensor(np.asarray(poses, np.float32), device=device)
+ t = torch.as_tensor(np.asarray(seq["trans"][pid], np.float32), device=device)
+ # 3DPW stores 300 shape coefficients for some subjects and 10 for
+ # others; zero-padding to the model's 300 dirs covers both, and
+ # ``num_betas`` then truncates uniformly. The published protocol is
+ # the first 10 (SPIN's pw3d preprocessing).
+ raw = np.asarray(seq["betas"][pid], np.float32).ravel()
+ b = np.zeros(n_dirs, np.float32)
+ k = min(num_betas, raw.shape[0])
+ b[:k] = raw[:k]
+ b = torch.as_tensor(b, device=device)
+
+ j17, j24 = [], []
+ for s in range(0, p.shape[0], chunk):
+ V, Jk = smpl_forward(m, p[s:s + chunk], b, t[s:s + chunk])
+ j17.append((Jreg @ V).cpu().numpy())
+ if check:
+ j24.append(Jk.cpu().numpy())
+ out[f"{name}|{pid}"] = np.concatenate(j17).astype(np.float32)
+
+ if check:
+ gt24 = np.asarray(seq["jointPositions"][pid], np.float32).reshape(-1, 24, 3)
+ d = np.linalg.norm(np.concatenate(j24) - gt24, axis=-1)
+ resid.append(d)
+ print(f" {name:<32s} {len(seq['poses'])} person(s)", flush=True)
+
+ if check and resid:
+ d = np.concatenate(resid) * 1000.0
+ print(f"[check] SMPL-24 forward vs jointPositions: mean {d.mean():.3f} mm, "
+ f"p99 {np.percentile(d, 99):.3f} mm, max {d.max():.3f} mm")
+ return out
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--sequence-dir", required=True)
+ ap.add_argument("--smpl-dir", required=True,
+ help="holds SMPL_MALE.pkl / SMPL_FEMALE.pkl")
+ ap.add_argument("--regressor", default="benchmark/data/smpl/J_regressor_h36m.npy")
+ ap.add_argument("--out", default="benchmark/data/3dpw_gt_h36m")
+ ap.add_argument("--splits", nargs="+", default=["test", "validation", "train"])
+ ap.add_argument("--num-betas", type=int, default=10,
+ help="10 reproduces the published protocol; 300 uses the "
+ "full shape vector 3DPW actually stores")
+ ap.add_argument("--device", default="cuda")
+ ap.add_argument("--chunk", type=int, default=64)
+ ap.add_argument("--check", action="store_true",
+ help="also regress the SMPL-24 joints and report their "
+ "distance to jointPositions (should be ~0)")
+ args = ap.parse_args()
+
+ device = torch.device(args.device if torch.cuda.is_available() else "cpu")
+ regressor = np.load(args.regressor)
+ seq_dir, out_dir = Path(args.sequence_dir), Path(args.out)
+ out_dir.mkdir(parents=True, exist_ok=True)
+
+ for split in args.splits:
+ print(f"[3DPW] {split}")
+ data = build_split(seq_dir, Path(args.smpl_dir), regressor, split,
+ args.num_betas, device, args.chunk, args.check)
+ dst = out_dir / f"{split}.npz"
+ np.savez_compressed(dst, layout="h36m17", num_betas=args.num_betas,
+ regressor=Path(args.regressor).name, **data)
+ n = sum(v.shape[0] for v in data.values())
+ print(f"[3DPW] {split}: {len(data)} person-tracks, {n} frames -> {dst}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmark/results/3dpw_test_b3_s1.json b/benchmark/results/3dpw_test_b3_s1.json
new file mode 100644
index 0000000..d5b306d
--- /dev/null
+++ b/benchmark/results/3dpw_test_b3_s1.json
@@ -0,0 +1,136 @@
+{
+ "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|h36m": {
+ "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
+ "gt": "h36m",
+ "split": "test",
+ "stride": 1,
+ "epoch": 137,
+ "source": "ema",
+ "n": 35463,
+ "n_dropped": 0,
+ "results": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 104.63785012091508,
+ "PA_MPJPE_mm": 67.89201715239723,
+ "PCK3D_50mm": 0.386628316837267,
+ "PCK3D_100mm": 0.8279313248013019,
+ "AUC_0_150mm": 0.555249314824265,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 66.1026158925688,
+ "right_knee": 48.77531527426408,
+ "right_hip": 98.43968262009881,
+ "left_hip": 86.54123107665853,
+ "left_knee": 46.110874847033735,
+ "left_ankle": 67.11189987887853,
+ "right_wrist": 70.71902986402077,
+ "right_elbow": 57.68874380634787,
+ "right_shoulder": 51.452820373726844,
+ "left_shoulder": 53.46686079217727,
+ "left_elbow": 66.61272199119497,
+ "left_wrist": 74.65868335746043,
+ "neck": 31.322680447479044,
+ "head": 131.48507991165152
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 35463,
+ "MPJPE_mm": 102.43934163084393,
+ "PA_MPJPE_mm": 62.492379121774945,
+ "PCK3D_50mm": 0.43160947090394686,
+ "PCK3D_100mm": 0.8874202220154339,
+ "AUC_0_150mm": 0.5877625294150285,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 62.453416312484485,
+ "right_knee": 50.619841347293885,
+ "right_hip": 98.35461927230007,
+ "left_hip": 85.97593910675288,
+ "left_knee": 47.09601993678759,
+ "left_ankle": 62.49923408065495,
+ "right_wrist": 66.505798652139,
+ "right_elbow": 48.842995632922765,
+ "right_shoulder": 51.30888543605562,
+ "left_shoulder": 51.65398595718133,
+ "left_elbow": 54.28141719363774,
+ "left_wrist": 70.316396533089
+ }
+ },
+ {
+ "joint_set": "J14+adapter",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 72.82834498471384,
+ "PA_MPJPE_mm": 42.94566703536861,
+ "PCK3D_50mm": 0.7122816134321083,
+ "PCK3D_100mm": 0.946737243243461,
+ "AUC_0_150mm": 0.7131403003143016
+ }
+ ]
+ },
+ "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|jointpositions": {
+ "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
+ "gt": "jointpositions",
+ "split": "test",
+ "stride": 1,
+ "epoch": 137,
+ "source": "ema",
+ "n": 35463,
+ "n_dropped": 0,
+ "results": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 78.92239927507839,
+ "PA_MPJPE_mm": 55.88134236358216,
+ "PCK3D_50mm": 0.5672270092369914,
+ "PCK3D_100mm": 0.889079563810974,
+ "AUC_0_150mm": 0.6319885423517287,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 57.2149217889093,
+ "right_knee": 44.6731540800345,
+ "right_hip": 51.30738310230049,
+ "left_hip": 47.680922082194506,
+ "left_knee": 47.11408900032607,
+ "left_ankle": 58.746754378530234,
+ "right_wrist": 68.48622774964359,
+ "right_elbow": 41.41727540449767,
+ "right_shoulder": 45.64858842957503,
+ "left_shoulder": 50.333023561680356,
+ "left_elbow": 47.20158941256118,
+ "left_wrist": 71.92360887402488,
+ "neck": 34.406638478724766,
+ "head": 116.18461674714762
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 35463,
+ "MPJPE_mm": 72.61393647860733,
+ "PA_MPJPE_mm": 49.30383775900926,
+ "PCK3D_50mm": 0.6433066388442414,
+ "PCK3D_100mm": 0.938478132137721,
+ "AUC_0_150mm": 0.6735733805853685,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 54.539697485801305,
+ "right_knee": 46.066113533422204,
+ "right_hip": 47.155299592078904,
+ "left_hip": 44.40314628895565,
+ "left_knee": 49.06421851012075,
+ "left_ankle": 54.51289182848444,
+ "right_wrist": 66.35472929929244,
+ "right_elbow": 36.84928335165509,
+ "right_shoulder": 39.98208452636371,
+ "left_shoulder": 41.037157437753955,
+ "left_elbow": 40.987603019551045,
+ "left_wrist": 70.69382823463162
+ }
+ }
+ ]
+ }
+}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_v3_s1.json b/benchmark/results/3dpw_test_v3_s1.json
new file mode 100644
index 0000000..1486c7d
--- /dev/null
+++ b/benchmark/results/3dpw_test_v3_s1.json
@@ -0,0 +1,136 @@
+{
+ "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|h36m": {
+ "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",
+ "gt": "h36m",
+ "split": "test",
+ "stride": 1,
+ "epoch": 126,
+ "source": "ema",
+ "n": 35463,
+ "n_dropped": 0,
+ "results": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 105.70162783403123,
+ "PA_MPJPE_mm": 69.05148473277063,
+ "PCK3D_50mm": 0.37404175780793664,
+ "PCK3D_100mm": 0.8228153286523983,
+ "AUC_0_150mm": 0.5488736167026034,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 68.26131519537937,
+ "right_knee": 48.22895082504128,
+ "right_hip": 98.35751407606348,
+ "left_hip": 86.86613781457758,
+ "left_knee": 46.25701446183899,
+ "left_ankle": 69.39201526188755,
+ "right_wrist": 71.39134970517831,
+ "right_elbow": 57.63600754408093,
+ "right_shoulder": 52.59923939395279,
+ "left_shoulder": 55.207572449651074,
+ "left_elbow": 67.88209503589113,
+ "left_wrist": 76.36662877587462,
+ "neck": 32.38007584338846,
+ "head": 135.8948698759832
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 35463,
+ "MPJPE_mm": 102.98084160960175,
+ "PA_MPJPE_mm": 63.198364128279884,
+ "PCK3D_50mm": 0.42087997819323425,
+ "PCK3D_100mm": 0.8823257103647933,
+ "AUC_0_150mm": 0.5837227290354721,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 64.60270339441506,
+ "right_knee": 49.996919602186196,
+ "right_hip": 98.25229959444044,
+ "left_hip": 86.3963348166251,
+ "left_knee": 47.03163228047883,
+ "left_ankle": 64.51319472082247,
+ "right_wrist": 67.54483867159513,
+ "right_elbow": 48.887356763210875,
+ "right_shoulder": 52.06354180217583,
+ "left_shoulder": 52.42872324128663,
+ "left_elbow": 54.99294113405385,
+ "left_wrist": 71.66988351806822
+ }
+ },
+ {
+ "joint_set": "J14+adapter",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 73.81513519657514,
+ "PA_MPJPE_mm": 44.1324115732964,
+ "PCK3D_50mm": 0.6958459722608272,
+ "PCK3D_100mm": 0.9417461257407116,
+ "AUC_0_150mm": 0.7059962931443702
+ }
+ ]
+ },
+ "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|jointpositions": {
+ "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",
+ "gt": "jointpositions",
+ "split": "test",
+ "stride": 1,
+ "epoch": 126,
+ "source": "ema",
+ "n": 35463,
+ "n_dropped": 0,
+ "results": [
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 35463,
+ "MPJPE_mm": 80.3476390328507,
+ "PA_MPJPE_mm": 56.98931886430932,
+ "PCK3D_50mm": 0.5518246381540519,
+ "PCK3D_100mm": 0.8840864321365125,
+ "AUC_0_150mm": 0.6254987511485651,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 60.29150986805832,
+ "right_knee": 44.09355165407435,
+ "right_hip": 52.21253960531078,
+ "left_hip": 48.437115641833024,
+ "left_knee": 47.35052736862379,
+ "left_ankle": 61.33749022699511,
+ "right_wrist": 69.4740236792569,
+ "right_elbow": 41.995470135750026,
+ "right_shoulder": 46.26280446156064,
+ "left_shoulder": 52.228924129906055,
+ "left_elbow": 48.42258559131994,
+ "left_wrist": 73.93419187162651,
+ "neck": 33.73966925436235,
+ "head": 118.07006061165272
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 35463,
+ "MPJPE_mm": 73.30164497963386,
+ "PA_MPJPE_mm": 50.3841473765136,
+ "PCK3D_50mm": 0.6284108319469118,
+ "PCK3D_100mm": 0.9329653441615204,
+ "AUC_0_150mm": 0.6672313927676856,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 57.91085514413626,
+ "right_knee": 45.33655452203096,
+ "right_hip": 47.530686446692584,
+ "left_hip": 45.00132633545716,
+ "left_knee": 49.146387447390694,
+ "left_ankle": 56.970314712065104,
+ "right_wrist": 67.99991243571321,
+ "right_elbow": 38.08431350645294,
+ "right_shoulder": 40.287118217560185,
+ "left_shoulder": 41.70494390891554,
+ "left_elbow": 42.05522689574795,
+ "left_wrist": 72.5821289460007
+ }
+ }
+ ]
+ }
+}
\ No newline at end of file
diff --git a/benchmark/results/adapter_j14_h36m_b3_s1.npz b/benchmark/results/adapter_j14_h36m_b3_s1.npz
new file mode 100644
index 0000000..69b8a7b
Binary files /dev/null and b/benchmark/results/adapter_j14_h36m_b3_s1.npz differ
diff --git a/benchmark/results/adapter_j14_h36m_v3_s1.npz b/benchmark/results/adapter_j14_h36m_v3_s1.npz
new file mode 100644
index 0000000..da708ae
Binary files /dev/null and b/benchmark/results/adapter_j14_h36m_v3_s1.npz differ
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index ba40515..244940f 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -491,14 +491,16 @@ def train(args, cfg, rank, local_rank, world):
raise SystemExit("--val-3dpw needs --val-3dpw-images")
from val3dpw import ThreeDPWValSet
dpw = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,
- split=args.val_3dpw_split, stride=args.val_3dpw_stride)
+ split=args.val_3dpw_split, stride=args.val_3dpw_stride,
+ gt=args.val_3dpw_gt)
# Stride-shard across ranks; the metric is all-reduced afterwards.
dpw_loader = torch.utils.data.DataLoader(
torch.utils.data.Subset(dpw, list(range(rank, len(dpw), world))),
batch_size=cfg.batch_size, shuffle=False, drop_last=False,
num_workers=max(1, cfg.num_workers // 2), pin_memory=True)
log(f"3DPW {args.val_3dpw_split}: {len(dpw):,} person-frames "
- f"(stride {args.val_3dpw_stride}) -> selection metric is J12 PA-MPJPE")
+ f"(stride {args.val_3dpw_stride}, GT {args.val_3dpw_gt}) "
+ f"-> selection metric is J12 PA-MPJPE")
# Optional read-only monitor on the split the paper reports. Selection
# NEVER touches it: picking the best of ~100 epochs on the test split
@@ -507,7 +509,8 @@ def train(args, cfg, rank, local_rank, world):
# not, so it is logged and never compared to `best`.
if args.val_3dpw_test:
dpw_t = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,
- split="test", stride=args.val_3dpw_test_stride)
+ split="test", stride=args.val_3dpw_test_stride,
+ gt=args.val_3dpw_gt)
dpw_test_loader = torch.utils.data.DataLoader(
torch.utils.data.Subset(dpw_t, list(range(rank, len(dpw_t), world))),
batch_size=cfg.batch_size, shuffle=False, drop_last=False,
@@ -627,6 +630,7 @@ def train(args, cfg, rank, local_rank, world):
"w_keypoints3d": cfg.w_keypoints3d, "kp3d_loss": cfg.kp3d_loss,
"pose_split": cfg.pose_split, "pose_beta": cfg.pose_beta,
"finger_weight": cfg.finger_weight,
+ "val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,
"started": time.strftime("%Y-%m-%d %H:%M")})
for epoch in range(start_epoch, args.epochs):
@@ -724,16 +728,20 @@ def train(args, cfg, rank, local_rank, world):
if dpw_loader is not None:
import val3dpw
dpw_raw = all_reduce_mean(
- val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device, cfg.use_amp),
+ val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device,
+ cfg.use_amp, gt=args.val_3dpw_gt),
device)
dpw_em = all_reduce_mean(
- val3dpw.evaluate(ema, dpw_loader, mhr_module, device, cfg.use_amp),
+ val3dpw.evaluate(ema, dpw_loader, mhr_module, device,
+ cfg.use_amp, gt=args.val_3dpw_gt),
device)
if dpw_test_loader is not None:
dpw_test_raw = all_reduce_mean(val3dpw.evaluate(
- ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp), device)
+ ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp,
+ gt=args.val_3dpw_gt), device)
dpw_test_em = all_reduce_mean(val3dpw.evaluate(
- ema, dpw_test_loader, mhr_module, device, cfg.use_amp), device)
+ ema, dpw_test_loader, mhr_module, device, cfg.use_amp,
+ gt=args.val_3dpw_gt), device)
raw["Mesh_PA_MPJPE"], em["Mesh_PA_MPJPE"] = (
dpw_raw["J12_PA_MPJPE"], dpw_em["J12_PA_MPJPE"])
raw["Mesh_MPJPE"], em["Mesh_MPJPE"] = (
@@ -872,6 +880,13 @@ def parse_args():
p.add_argument("--val-3dpw-test-stride", type=int, default=5)
p.add_argument("--val-3dpw-stride", type=int, default=5,
help="3DPW is 30 fps; neighbouring frames are near-duplicates.")
+ p.add_argument("--val-3dpw-gt", default="h36m",
+ choices=("h36m", "jointpositions"),
+ help="h36m (default) is the published protocol and needs "
+ "gt_h36m/ next to sequenceFiles/ (see "
+ "benchmark/make_3dpw_gt.py). jointpositions is the raw "
+ "pickle field the runs before this flag selected on — "
+ "several mm apart, so a resumed run must keep its own.")
p.add_argument("--mix", default="sqrt",
help="Dataset mixture: sqrt | uniform | equal | "
"'sa1b=0.4,aic=0.35,coco=0.15,mpii=0.1'.")
diff --git a/instanthmr_distill_train/val3dpw.py b/instanthmr_distill_train/val3dpw.py
index 4966e24..08eb332 100644
--- a/instanthmr_distill_train/val3dpw.py
+++ b/instanthmr_distill_train/val3dpw.py
@@ -21,6 +21,12 @@ all 60 sequences (see ``benchmark/README.md``). On Jean Zay it is not: the built
corpus is coco/mpii/aic/sa1b only, so 3DPW there is genuinely unseen — which is
what makes this worth doing on the cluster and not on the laptop.
+GT is the published protocol by default — SMPL forward + the H36M regressor,
+precomputed by ``benchmark/make_3dpw_gt.py`` into ``<3DPW root>/gt_h36m/`` and
+rsynced next to ``sequenceFiles``. ``gt="jointpositions"`` restores the raw
+pickle field the harness used before; the two are several mm apart, so numbers
+from the two sources must never be compared to each other.
+
Selection metric is **PA-MPJPE on J12**. J12 is limbs only, where MHR and SMPL
mean the same anatomical points; J14 adds neck and head, which the two rigs
place differently and which therefore carry a systematic penalty unrelated to
@@ -99,11 +105,13 @@ class ThreeDPWValSet(Dataset):
def __init__(self, sequence_dir: str, image_root: str,
split: str = "validation", stride: int = 5,
- bbox_scale: float = 1.2):
+ bbox_scale: float = 1.2, gt: str = "h36m",
+ gt_dir: str | None = None):
self.samples, self.gt = P.build_samples(
sequence_dir, image_root, split=split,
- stride=stride, bbox_scale=bbox_scale)
+ stride=stride, bbox_scale=bbox_scale, gt=gt, gt_dir=gt_dir)
self.gt = np.asarray(self.gt, dtype=np.float32)
+ self.gt_source = gt
def __len__(self) -> int:
return len(self.samples)
@@ -128,7 +136,8 @@ class ThreeDPWValSet(Dataset):
@torch.no_grad()
-def evaluate(model, loader, mhr_module, device, use_amp: bool = True) -> dict:
+def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
+ gt: str = "h36m") -> dict:
"""PA-MPJPE and MPJPE in mm on J12 and J14, from the MHR forward pass.
Predictions come from ``mhr_params`` through the same FK + (70, 127)
@@ -176,12 +185,13 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True) -> dict:
return res
pred = np.concatenate(preds) * MM # (N, 70, 3) rig-local, mm
- gt = np.concatenate(gts) * MM # (N, 24, 3) camera space, mm
+ gt_all = np.concatenate(gts) * MM # (N, J, 3) camera space, mm
res["n"] = float(len(pred))
for name in ("J12", "J14"):
spec = J.JOINT_SETS[name]
- p, g = pred[:, spec["mhr"], :], gt[:, spec["smpl"], :]
+ p = pred[:, spec["mhr"], :]
+ g = gt_all[:, J.gt_index(spec, gt), :]
rp, rg = J.pelvis(p), J.pelvis(g)
res[f"{name}_MPJPE"] = float(M.mpjpe(p, g, root=(rp, rg)).mean())
res[f"{name}_PA_MPJPE"] = float(M.pa_mpjpe(p, g).mean())