Skip to content
Python

feat: --image-size as a checkpoint setting, context crops in build_split, CPU-tier backbone latencies

mohamdev36ae38720c00
diff --git a/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md b/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md
index 445bc7b..90d24be 100644
--- a/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md
+++ b/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md
@@ -4,9 +4,25 @@
**Blocked by:** None (can start immediately).
-**Status:** ready-for-agent
+**Status:** done (2026-09-25), uncommitted
-- [ ] The B4 random-weight export's latency matches the trained g8h's within noise on the S23 CPU (63 ms) — the check that random weights are a valid latency proxy
-- [ ] CPU-only latency and peak memory for B0, B1 and B4 on Galaxy S23, A73, A53, A14 and Pixel 8; S23 NPU latency and placement for each
-- [ ] Every export places 100% of layers on the NPU on the S23 and passes fp16 parity
-- [ ] A table recorded in the status doc and a recommended CPU-tier size against the 33 ms budget on mid-range phones (A73 / Pixel 8)
+- [x] The B4 random-weight export's latency matches the trained g8h's within noise on the S23 CPU (63 ms) — the check that random weights are a valid latency proxy
+- [x] CPU-only latency and peak memory for B0, B1 and B4 on Galaxy S23, A73, A53, A14 and Pixel 8; S23 NPU latency and placement for each
+- [~] Every export places 100% of layers on the NPU on the S23 (yes, all four); fp16 parity NOT measured: it is meaningless on random weights, so it moves to ticket 11 on the trained members
+- [x] A table recorded in the status doc and a recommended CPU-tier size against the 33 ms budget on mid-range phones (A73 / Pixel 8)
+
+## Answer
+
+Exports: g8h's exact architecture and config (continuous head, bounded scales, CLIFF focal), backbone swapped, network only (no FK/mesh), 224 px, opset 17, onnxsim. AI Hub, ORT runtime compiled for the S23. Raw rows: `benchmark/results/family_latency_aihub.json`; scripts: `tools/profiling/`.
+
+| model | params (backbone) | S23 NPU | S23 CPU | A73 CPU | Pixel 8 CPU | A53 CPU | A14 CPU |
+|---|---|---|---|---|---|---|---|
+| g8h trained (B4) | 32.2 M (13.6 M) | 2.44 ms | 61.7 | 84.9 | 83.2 | 187.9 | 195.2 |
+| random B4 | 32.2 M (13.6 M) | 2.47 | 60.0 | 81.7 | 78.7 | 194.2 | 199.4 |
+| random B1 | 20.3 M (2.2 M) | 1.88 | 26.9 | 36.9 | 33.2 | 77.6 | 76.8 |
+| random B0 | 20.0 M (1.9 M) | 1.60 | 23.9 | 32.7 | 29.9 | 79.7 | 68.6 |
+
+- Random weights are a valid latency proxy: random B4 is within 2-6% of the trained g8h on every phone.
+- All four place 100% of layers on the S23 NPU.
+- **Recommendation: HGNetV2-B0 is the CPU-tier member** (under 33 ms on mid-range: A73 32.7, Pixel 8 29.9). B1 is 3.3 ms slower and just over budget on the A73 (36.9); worth training as the middle point of the accuracy-vs-latency curve.
+- **Budget phones (A53, A14) stay at 70-80 ms with either backbone.** B0 and B1 differ by <2 ms there: below B4 the backbone is no longer the cost. ~18 M of the 20 M parameters are the decoder and heads, unchanged by the backbone swap. Reaching 30 FPS on budget phones needs a slimmer decoder (width / depth), which is a new work item, not a backbone choice.
diff --git a/.scratch/v3-framing-robustness/issues/05-input-size-setting.md b/.scratch/v3-framing-robustness/issues/05-input-size-setting.md
index 8996d7c..845d46c 100644
--- a/.scratch/v3-framing-robustness/issues/05-input-size-setting.md
+++ b/.scratch/v3-framing-robustness/issues/05-input-size-setting.md
@@ -4,9 +4,22 @@
**Blocked by:** None (can start immediately).
-**Status:** ready-for-agent
+**Status:** done (2026-09-25), uncommitted
-- [ ] baseline and v2 augmented samples are bit-identical against `git show HEAD`, compared on alternating calls
-- [ ] g8h_s0 evaluated through the harness gives the same 3DPW test J14 and EMDB-1 numbers as recorded, to the decimal
-- [ ] A random-weight model configured for 288 runs end to end through the 3DPW and EMDB harnesses, and its ONNX metadata reads 288
-- [ ] `ddp_smoke` passes at 224 and at 288
+- [x] baseline and v2 augmented samples are bit-identical against `git show HEAD`, compared on alternating calls
+- [x] g8h_s0 evaluated through the harness gives the same 3DPW test J14 and EMDB-1 numbers as recorded, to the decimal
+- [x] A random-weight model configured for 288 runs end to end through the 3DPW and EMDB harnesses, and its ONNX metadata reads 288
+- [x] `ddp_smoke` passes at 224 and at 288
+
+## Answer
+
+`--image-size` on the DDP trainer (multiple of 32), recorded in run_config.json. `config_from_checkpoint` reads the size from the weights (`mem_pos_embed` grid, 7x7 = 224, 9x9 = 288) and refuses a run_config.json that contradicts it. The dataset (both trainers and the self/overfit tests), `val3dpw`, the 3DPW / EMDB / SMPL-fit harnesses, `eval_all.py` grouping, `pth_to_onnx.py` metadata (`image_size`), `instanthmr.inference` and `ddp_smoke.py` all follow it.
+
+Evidence: `tools/verify_input_size.py` PASS (baseline and v2 0/24 samples differ from the pre-change trainer; labels and CLIFF identical at 224 and 288; 288 checkpoint round-trips through config, export metadata, inference on a real frame and both harnesses). g8h_s0 full regression identical to 4 decimals (3DPW test J14+adapter PA 40.7983, EMDB SMPL24+adapter 51.9492). `ddp_smoke` passes at 224 and 288; `--self-test` passes.
+
+Open, outside this repo (story 21 is NOT satisfied end to end yet):
+- neurocomp-app's own exporter (`tools/modelexport/pose.py`) hard-codes `INPUT_SHAPES` 224 and its metadata stamp clears all props and never writes `image_size`.
+- the Android engine hard-codes `POSE_INPUT_SIZE = 224` in `Preprocess.kt`.
+Both must read `image_size` before a non-224 model can ship. File this in the phone-app tracker.
+
+Note for ticket 10: `RandomPixelate` uses absolute resolutions (32-112 px), so at 288 it degrades relatively harder than at 224, which slightly confounds a 224-vs-288 comparison.
diff --git a/benchmark/eval_3dpw_ckpt.py b/benchmark/eval_3dpw_ckpt.py
index 125fc79..307d4fc 100644
--- a/benchmark/eval_3dpw_ckpt.py
+++ b/benchmark/eval_3dpw_ckpt.py
@@ -54,6 +54,16 @@ def parse_args():
p.add_argument("--fit-adapter", default=None,
help="fit that regressor on THIS split and write it here "
"(run on train, never on test)")
+ p.add_argument("--landmarks", default="fitted", choices=["fitted", "exact"],
+ help="how the 70 MHR keypoints are read off the prediction. "
+ "fitted (default): the (70, 127) skeleton matrix, which "
+ "is what the exported graph's consumer uses and what "
+ "every recorded number in benchmark/results was scored "
+ "with. exact: the teacher's mesh+joint mapping, the "
+ "operator the stored annotations -- and so the adapter "
+ "-- came out of. Worth 0.13 mm of J14+adapter PA-MPJPE "
+ "on 3DPW test, with or without --exact-landmarks "
+ "training.")
p.add_argument("--backbone", default=None,
help="override the backbone; normally read from the "
"run_config.json beside the checkpoint")
@@ -71,7 +81,8 @@ def parse_args():
@torch.no_grad()
-def predict(model, loader, mhr_module, device, use_amp: bool = True):
+def predict(model, loader, mhr_module, device, use_amp: bool = True,
+ exact_landmarks: bool = False):
"""(N, 70, 3) predicted keypoints in mm, plus the indices they came from.
The indices let the same forward pass be scored against more than one GT
@@ -86,7 +97,8 @@ def predict(model, loader, mhr_module, device, use_amp: bool = True):
with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):
out = model(img, cc)
kp = mhr_module.get_native_keypoints(out["mhr_params"].float(),
- out["shape_params"].float())
+ out["shape_params"].float(),
+ exact=exact_landmarks)
# Same guard as val3dpw.evaluate: a non-finite prediction makes the
# Procrustes SVD raise rather than return a large error.
finite = torch.isfinite(kp).flatten(1).all(dim=1).cpu()
@@ -129,10 +141,21 @@ def main():
"conditioning is built once for all of them.\n"
"Evaluate the two groups in separate invocations.")
cliff_focal = focals.pop()
+ sizes = {c.image_size for c in cfgs.values()}
+ if len(sizes) > 1:
+ raise SystemExit(
+ f"the checkpoints disagree on the input size ({sorted(sizes)}), and the "
+ "crops are built once for all of them.\n"
+ "Evaluate the groups in separate invocations.")
+ input_size = sizes.pop()
cfg = next(iter(cfgs.values()))
- mhr = T.MHRForwardPass(cfg.mhr_model_path, device,
- kp_regressor=np.load(cfg.kp_regressor_path))
+ exact_lm = args.landmarks == "exact"
+ mhr = T.MHRForwardPass(
+ cfg.mhr_model_path, device, kp_regressor=np.load(cfg.kp_regressor_path),
+ landmark_assets=cfg.landmark_assets_path if exact_lm else None)
+ print(f"[readout] 70 keypoints via the "
+ f"{'teacher exact mesh+joint mapping' if exact_lm else 'fitted (70, 127) skeleton matrix'}")
W_adapt = W_adapt_gt = None
if args.adapter:
ad = np.load(args.adapter)
@@ -145,7 +168,8 @@ def main():
ds = val3dpw.ThreeDPWValSet(args.sequence_dir, args.image_root,
split=args.split, stride=args.stride,
gt=gt, gt_dir=args.gt_dir,
- cliff_focal=cliff_focal)
+ cliff_focal=cliff_focal,
+ input_size=input_size)
sets[gt] = ds
print(f"[3DPW] {args.split} / GT={gt}: {len(ds):,} person-frames "
f"(stride {args.stride}), CLIFF conditioning "
@@ -169,7 +193,8 @@ def main():
# Strict: an unexpected key means the config restored above is wrong
# and the model being scored is not the model that was trained.
model.load_state_dict(ck["model_state_dict"])
- pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp)
+ pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp,
+ exact_landmarks=exact_lm)
del model
torch.cuda.empty_cache()
@@ -183,6 +208,7 @@ def main():
report[f"{ck_path}|{gt}"] = dict(
ckpt=ck_path, gt=gt, split=args.split, stride=args.stride,
epoch=ck.get("epoch"), source=ck.get("source"),
+ landmarks=args.landmarks,
n=int(pred.shape[0]), n_dropped=n_dropped,
results=list(rows.values()))
diff --git a/benchmark/eval_all.py b/benchmark/eval_all.py
index db036a5..c33ffd4 100644
--- a/benchmark/eval_all.py
+++ b/benchmark/eval_all.py
@@ -65,11 +65,12 @@ def group_by_conditioning(ckpts: list[str]) -> list[list[str]]:
import torch
sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
import train_distill_mhr_only as T
- groups: dict[bool, list[str]] = {}
+ groups: dict[tuple[bool, int], list[str]] = {}
for c in ckpts:
st = torch.load(c, map_location="cpu", weights_only=False)
cfg, _ = T.config_from_checkpoint(st.get("model_state_dict", st), c)
- groups.setdefault(bool(cfg.cliff_focal), []).append(c)
+ # the input size too: the harnesses build one crop set per invocation
+ groups.setdefault((bool(cfg.cliff_focal), int(cfg.image_size)), []).append(c)
del st
return list(groups.values())
diff --git a/benchmark/eval_emdb_ckpt.py b/benchmark/eval_emdb_ckpt.py
index b696aee..2f46010 100644
--- a/benchmark/eval_emdb_ckpt.py
+++ b/benchmark/eval_emdb_ckpt.py
@@ -68,6 +68,10 @@ def parse_args():
p.add_argument("--gt-dir", default=None)
p.add_argument("--adapter", default="benchmark/results/adapter_smpl24_teacher.npz",
help="npz with the MHR70 -> SMPL24 regressor")
+ p.add_argument("--landmarks", default="fitted", choices=["fitted", "exact"],
+ help="how the 70 MHR keypoints are read off the prediction; "
+ "see eval_3dpw_ckpt.py. fitted is the deployed readout "
+ "and what every recorded number was scored with.")
p.add_argument("--backbone", default=None)
p.add_argument("--cliff-focal", dest="cliff_focal", action="store_true",
default=None)
@@ -89,12 +93,13 @@ class EMDBValSet(torch.utils.data.Dataset):
"""
def __init__(self, emdb_root, stride=1, bbox="gt-joints", bbox_scale=1.2,
- gt_dir=None, cliff_focal=False):
+ gt_dir=None, cliff_focal=False, input_size=val3dpw.INPUT_SIZE):
self.samples, gt = D.build_samples(
emdb_root, stride=stride, bbox=bbox, bbox_scale=bbox_scale,
gt_dir=gt_dir)
self.gt = np.asarray(gt, dtype=np.float32)
self.cliff_focal = cliff_focal
+ self.input_size = input_size
def __len__(self):
return len(self.samples)
@@ -103,12 +108,13 @@ class EMDBValSet(torch.utils.data.Dataset):
s = self.samples[i]
bgr = cv2.imread(s["image_path"], cv2.IMREAD_COLOR)
if bgr is None:
- return {"image": torch.zeros(3, val3dpw.INPUT_SIZE, val3dpw.INPUT_SIZE),
+ return {"image": torch.zeros(3, self.input_size, self.input_size),
"cliff_cond": torch.zeros(3), "ok": torch.tensor(0.0)}
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
h, w = rgb.shape[:2]
crop, cliff = val3dpw._preprocess(
- rgb, s["bbox"], h, w, s["focal"] if self.cliff_focal else None)
+ rgb, s["bbox"], h, w, s["focal"] if self.cliff_focal else None,
+ size=self.input_size)
return {"image": torch.from_numpy(crop),
"cliff_cond": torch.from_numpy(cliff),
"ok": torch.tensor(1.0)}
@@ -168,10 +174,21 @@ def main():
"conditioning is built once for all of them.\n"
"Evaluate the two groups in separate invocations.")
cliff_focal = focals.pop()
+ sizes = {c.image_size for c in cfgs.values()}
+ if len(sizes) > 1:
+ raise SystemExit(
+ f"the checkpoints disagree on the input size ({sorted(sizes)}), and the "
+ "crops are built once for all of them.\n"
+ "Evaluate the groups in separate invocations.")
+ input_size = sizes.pop()
cfg = next(iter(cfgs.values()))
- mhr = T.MHRForwardPass(cfg.mhr_model_path, device,
- kp_regressor=np.load(cfg.kp_regressor_path))
+ exact_lm = args.landmarks == "exact"
+ mhr = T.MHRForwardPass(
+ cfg.mhr_model_path, device, kp_regressor=np.load(cfg.kp_regressor_path),
+ landmark_assets=cfg.landmark_assets_path if exact_lm else None)
+ print(f"[readout] 70 keypoints via the "
+ f"{'teacher exact mesh+joint mapping' if exact_lm else 'fitted (70, 127) skeleton matrix'}")
ad = np.load(args.adapter)
if str(ad["gt"]) != "smpl24":
raise SystemExit(f"{args.adapter} targets '{ad['gt']}', not smpl24; "
@@ -180,7 +197,7 @@ def main():
ds = EMDBValSet(args.emdb_root, stride=args.stride, bbox=args.bbox,
bbox_scale=args.bbox_scale, gt_dir=args.gt_dir,
- cliff_focal=cliff_focal)
+ cliff_focal=cliff_focal, input_size=input_size)
seqs = sorted({s["sequence"] for s in ds.samples})
print(f"[EMDB1] {len(ds):,} frames across {len(seqs)} sequences "
f"(stride {args.stride}, boxes: {args.bbox}), CLIFF conditioning "
@@ -199,7 +216,8 @@ def main():
model = T.InstantHMRStudent(cfgs[ck_path], pretrained=False).to(device)
ck = torch.load(ck_path, map_location="cpu", weights_only=False)
model.load_state_dict(ck["model_state_dict"])
- pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp)
+ pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp,
+ exact_landmarks=exact_lm)
del model
torch.cuda.empty_cache()
@@ -219,7 +237,7 @@ def main():
report[ck_path] = dict(
ckpt=ck_path, dataset="EMDB1", split="emdb1", stride=args.stride,
- bbox=args.bbox, adapter=str(args.adapter),
+ bbox=args.bbox, adapter=str(args.adapter), landmarks=args.landmarks,
epoch=ck.get("epoch"), source=ck.get("source"),
n=int(pred.shape[0]), n_dropped=n_dropped,
num_sequences=len(seqs), results=rows,
diff --git a/benchmark/eval_smpl_fit.py b/benchmark/eval_smpl_fit.py
index f10c46e..3e621d5 100644
--- a/benchmark/eval_smpl_fit.py
+++ b/benchmark/eval_smpl_fit.py
@@ -92,10 +92,17 @@ def parse_args():
p.add_argument("--bbox", default="gt-joints", choices=["gt-joints", "annotated"],
help="EMDB only; 3DPW always uses projected-GT-joint boxes")
p.add_argument("--fit-iters", type=int, default=400,
- help="USE >= 1500. 400 converges on the round trip but NOT "
- "on real predictions: 16.1 mm residual on 3DPW vs 8.0 "
- "at 4000, worth 3.6 mm of J14 PA-MPJPE, and the sign "
- "differs per dataset. See benchmark/README.md.")
+ help="0 skips the SMPL parameter fit entirely and reports "
+ "the fit-free rows only (PVE, PA-PVE, J14/H36M) -- the "
+ "settled protocol, no optimiser and no free parameter, "
+ "and the only way to reach PVE without a "
+ "hyperparameter that has a 3-4 mm blast radius. "
+ "Otherwise USE >= 1500: 400 converges on the round trip "
+ "but NOT on real predictions -- 16.1 mm residual on "
+ "3DPW vs 8.0 at 4000, worth 3.6 mm of J14 PA-MPJPE, "
+ "and the sign differs per dataset. The fit buys the 24 "
+ "SMPL KINEMATIC joints and nothing else. See "
+ "benchmark/README.md.")
p.add_argument("--fit-batch", type=int, default=64)
p.add_argument("--batch-size", type=int, default=64)
p.add_argument("--num-workers", type=int, default=10)
@@ -241,12 +248,14 @@ def threedpw_gt_smpl(sequence_dir, split, samples, smpl_dir, device):
class ThreeDPWSet(torch.utils.data.Dataset):
"""3DPW crops, built by the same ``val3dpw._preprocess`` EMDB uses."""
- def __init__(self, sequence_dir, image_root, split, stride, cliff_focal):
+ def __init__(self, sequence_dir, image_root, split, stride, cliff_focal,
+ input_size=val3dpw.INPUT_SIZE):
from benchlib import threedpw as P3
self.samples, _ = P3.build_samples(
sequence_dir, image_root, split=split, stride=stride,
gt="jointpositions")
self.cliff_focal = cliff_focal
+ self.input_size = input_size
def __len__(self):
return len(self.samples)
@@ -255,12 +264,13 @@ class ThreeDPWSet(torch.utils.data.Dataset):
s = self.samples[i]
bgr = cv2.imread(s["image_path"], cv2.IMREAD_COLOR)
if bgr is None:
- return {"image": torch.zeros(3, val3dpw.INPUT_SIZE, val3dpw.INPUT_SIZE),
+ return {"image": torch.zeros(3, self.input_size, self.input_size),
"cliff_cond": torch.zeros(3), "ok": torch.tensor(0.0)}
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
h, w = rgb.shape[:2]
crop, cliff = val3dpw._preprocess(
- rgb, s["bbox"], h, w, s["focal"] if self.cliff_focal else None)
+ rgb, s["bbox"], h, w, s["focal"] if self.cliff_focal else None,
+ size=self.input_size)
return {"image": torch.from_numpy(crop),
"cliff_cond": torch.from_numpy(cliff),
"ok": torch.tensor(1.0)}
@@ -281,10 +291,10 @@ def main():
if args.dataset == "emdb":
ds = EMDBValSet(args.emdb_root, stride=args.stride, bbox=args.bbox,
- cliff_focal=cfg.cliff_focal)
+ cliff_focal=cfg.cliff_focal, input_size=cfg.image_size)
else:
ds = ThreeDPWSet(args.sequence_dir, args.image_root, args.split,
- args.stride, cfg.cliff_focal)
+ args.stride, cfg.cliff_focal, cfg.image_size)
loader = torch.utils.data.DataLoader(
ds, batch_size=args.batch_size, shuffle=False,
num_workers=args.num_workers, pin_memory=True)
@@ -306,11 +316,25 @@ def main():
Jgt, Vgt, genders = threedpw_gt_smpl(args.sequence_dir, args.split,
kept, args.smpl_dir, device)
- # Fit per gender group, so one SMPL model serves each batch.
+ # `barycentric_transfer` already returns the prediction in SMPL
+ # topology, vertex i <-> SMPL vertex i, so this surface IS the answer
+ # for every vertex-space metric. Keeping it costs one more (N, 6890, 3)
+ # and removes the optimiser from PVE entirely.
+ Vd = torch.zeros_like(Vgt)
+ fitting = args.fit_iters > 0
+ # Fit per gender group, so one SMPL model serves each batch. The fit
+ # exists only to recover the 24 SMPL KINEMATIC joints, which come off
+ # the chain and not off the surface.
Jp = torch.zeros_like(Jgt)
- Vp = torch.zeros_like(Vgt)
+ # 2.9 GB at 35,463 frames, so it is allocated only when a fit fills it.
+ Vp = torch.zeros_like(Vgt) if args.fit_iters > 0 else None
resid = torch.zeros(len(kept))
- for g in ("male", "female"):
+ if not fitting:
+ for k in range(0, len(kept), args.fit_batch):
+ sel = np.arange(k, min(k + args.fit_batch, len(kept)))
+ verts = rig.vertices(mp[sel].to(device), sh[sel].to(device))
+ Vd[sel] = barycentric_transfer(verts, rig.faces, tri, bary).cpu()
+ for g in ("male", "female") if fitting else ():
gi = np.array([i for i, x in enumerate(genders) if x == g])
if not len(gi):
continue
@@ -319,6 +343,7 @@ def main():
sel = gi[k:k + args.fit_batch]
verts = rig.vertices(mp[sel].to(device), sh[sel].to(device))
tgt = barycentric_transfer(verts, rig.faces, tri, bary)
+ Vd[sel] = tgt.cpu()
th, be, tr, r = fit_smpl_to_targets(sm, tgt, iters=args.fit_iters)
v, j = sm.forward(th, be, tr)
Jp[sel] = j.cpu()
@@ -344,30 +369,55 @@ def main():
sel = J.J14_FROM_H36M17
CH = 256
pve_sum = pve_n = pa_pve_sum = 0.0
- hp_parts, hg_parts = [], []
- for i in range(0, Vp.shape[0], CH):
- a = Vp[i:i + CH].numpy() * MM
+ pve_d_sum = pa_pve_d_sum = 0.0
+ hp_parts, hg_parts, hpd_parts = [], [], []
+ for i in range(0, Vd.shape[0], CH):
b = Vgt[i:i + CH].numpy() * MM
- n = a.shape[0]
- e = M.mpjpe(a, b, root=(rp[i:i + n], rg[i:i + n]))
- pve_sum += float(e.sum()); pve_n += e.size
- pa_pve_sum += float(M.pa_mpjpe(a, b).sum())
- hp_parts.append(np.einsum("jv,nvc->njc", Jreg, a)[:, sel])
- hg_parts.append(np.einsum("jv,nvc->njc", Jreg, b)[:, sel])
- h_p = np.concatenate(hp_parts) # already mm
+ n = b.shape[0]
+ # Fit-free: the prediction's own surface against the GT surface,
+ # rooted at the H36M pelvis regressed from EACH side by the same
+ # operator, so the regressor's bias cancels instead of being
+ # compared against a kinematic joint it does not equal.
+ a_d = Vd[i:i + CH].numpy() * MM
+ h_pd = np.einsum("jv,nvc->njc", Jreg, a_d)[:, sel]
+ h_gd = np.einsum("jv,nvc->njc", Jreg, b)[:, sel]
+ pve_d_sum += float(M.mpjpe(a_d, b,
+ root=(J.pelvis(h_pd), J.pelvis(h_gd))).sum())
+ pa_pve_d_sum += float(M.pa_mpjpe(a_d, b).sum())
+ pve_n += a_d.shape[0] * a_d.shape[1]
+ hpd_parts.append(h_pd)
+ hg_parts.append(h_gd)
+ if fitting:
+ a = Vp[i:i + CH].numpy() * MM
+ pve_sum += float(M.mpjpe(a, b, root=(rp[i:i + n], rg[i:i + n])).sum())
+ pa_pve_sum += float(M.pa_mpjpe(a, b).sum())
+ hp_parts.append(np.einsum("jv,nvc->njc", Jreg, a)[:, sel])
+ h_pd = np.concatenate(hpd_parts) # already mm
h_g = np.concatenate(hg_parts)
- del hp_parts, hg_parts
- hp_r, hg_r = J.pelvis(h_p), J.pelvis(h_g)
+ h_p = np.concatenate(hp_parts) if fitting else None
+ del hp_parts, hg_parts, hpd_parts
+ hpd_r, hg_r = J.pelvis(h_pd), J.pelvis(h_g)
+ hp_r = J.pelvis(h_p) if fitting else None
res = dict(
ckpt=ck_path, dataset=tag, bbox=args.bbox, stride=args.stride,
n=int(len(kept)), fit_iters=args.fit_iters,
epoch=st.get("epoch"), source=st.get("source"),
- fit_residual_mm=float(resid.mean() * MM),
- MPJPE_mm=float(M.mpjpe(jp, jg, root=(rp, rg)).mean()),
- PA_MPJPE_mm=float(M.pa_mpjpe(jp, jg).mean()),
- PVE_mm=pve_sum / pve_n,
- PA_PVE_mm=pa_pve_sum / pve_n,
+ fit_residual_mm=float(resid.mean() * MM) if fitting else None,
+ # The fit-free rows: the prediction's own surface, in SMPL
+ # topology, against the benchmark's own GT surface. No optimiser,
+ # no free parameter, so these are the PVE numbers to quote. The
+ # *_mm keys below keep their old meaning -- the FITTED mesh -- so
+ # rows recorded before 2026-09-22 stay comparable.
+ PVE_direct_mm=pve_d_sum / pve_n,
+ PA_PVE_direct_mm=pa_pve_d_sum / pve_n,
+ J14_h36m_direct_MPJPE_mm=float(
+ M.mpjpe(h_pd, h_g, root=(hpd_r, hg_r)).mean()),
+ J14_h36m_direct_PA_MPJPE_mm=float(M.pa_mpjpe(h_pd, h_g).mean()),
+ MPJPE_mm=float(M.mpjpe(jp, jg, root=(rp, rg)).mean()) if fitting else None,
+ PA_MPJPE_mm=float(M.pa_mpjpe(jp, jg).mean()) if fitting else None,
+ PVE_mm=pve_sum / pve_n if fitting else None,
+ PA_PVE_mm=pa_pve_sum / pve_n if fitting else None,
# The published 3DPW protocol is 14 joints in the H36M convention,
# reached here THROUGH the mesh conversion: MHR mesh -> barycentric
# map -> fitted SMPL -> J_regressor_h36m, against the same
@@ -380,21 +430,29 @@ def main():
# space. Both are legitimate, and note that J_regressor_h36m is
# itself a fitted linear map (SMPL vertices -> H36M joints) -- SMPL-
# native methods use it for exactly the same reason we need ours.
- J14_h36m_viamesh_MPJPE_mm=float(M.mpjpe(h_p, h_g, root=(hp_r, hg_r)).mean()),
- J14_h36m_viamesh_PA_MPJPE_mm=float(M.pa_mpjpe(h_p, h_g).mean()),
+ J14_h36m_viamesh_MPJPE_mm=float(
+ M.mpjpe(h_p, h_g, root=(hp_r, hg_r)).mean()) if fitting else None,
+ J14_h36m_viamesh_PA_MPJPE_mm=float(
+ M.pa_mpjpe(h_p, h_g).mean()) if fitting else None,
)
- e = M.pa_mpjpe(jp, jg).mean(1)
+ e = (M.pa_mpjpe(jp, jg) if fitting else M.pa_mpjpe(h_pd, h_g)).mean(1)
res["per_sequence_PA_MPJPE_mm"] = {
n: float(e[[i for i, s in enumerate(kept) if s["sequence"] == n]].mean())
for n in sorted({s["sequence"] for s in kept})}
report[ck_path] = res
- print(f"\n {Path(ck_path).parts[-3]:<10s} SMPL24: MPJPE {res['MPJPE_mm']:6.2f} "
- f"PA {res['PA_MPJPE_mm']:6.2f} PVE {res['PVE_mm']:6.2f} "
- f"PA-PVE {res['PA_PVE_mm']:6.2f} | J14/H36M via mesh: MPJPE "
- f"{res['J14_h36m_viamesh_MPJPE_mm']:6.2f} "
- f"PA {res['J14_h36m_viamesh_PA_MPJPE_mm']:6.2f}"
- f" (fit residual {res['fit_residual_mm']:.2f} mm)\n", flush=True)
+ print(f"\n {Path(ck_path).parts[-3]:<10s} fit-free: PVE "
+ f"{res['PVE_direct_mm']:6.2f} PA-PVE {res['PA_PVE_direct_mm']:6.2f}"
+ f" | J14/H36M: MPJPE {res['J14_h36m_direct_MPJPE_mm']:6.2f} "
+ f"PA {res['J14_h36m_direct_PA_MPJPE_mm']:6.2f}", flush=True)
+ if fitting:
+ print(f" {Path(ck_path).parts[-3]:<10s} fitted: SMPL24 MPJPE "
+ f"{res['MPJPE_mm']:6.2f} PA {res['PA_MPJPE_mm']:6.2f} PVE "
+ f"{res['PVE_mm']:6.2f} PA-PVE {res['PA_PVE_mm']:6.2f} | "
+ f"J14/H36M via mesh: MPJPE {res['J14_h36m_viamesh_MPJPE_mm']:6.2f}"
+ f" PA {res['J14_h36m_viamesh_PA_MPJPE_mm']:6.2f}"
+ f" (fit residual {res['fit_residual_mm']:.2f} mm)", flush=True)
+ print(flush=True)
# Written after every checkpoint, not at the end: a full sweep is ~20
# minutes per checkpoint and losing all of it to a crash in the last
# one has already happened once.
diff --git a/benchmark/results/family_latency_aihub.json b/benchmark/results/family_latency_aihub.json
new file mode 100644
index 0000000..4100378
--- /dev/null
+++ b/benchmark/results/family_latency_aihub.json
@@ -0,0 +1,266 @@
+[
+ {
+ "model": "g8h_trained_b4",
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp0moor2g",
+ "ms": 2.44,
+ "peak_mb": 160.2,
+ "layers": {
+ "NPU": 918
+ }
+ },
+ {
+ "model": "g8h_trained_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp8ejj7zp",
+ "ms": 61.74,
+ "peak_mb": 199.1,
+ "layers": {
+ "CPU": 791
+ }
+ },
+ {
+ "model": "g8h_trained_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jgk266yyg",
+ "ms": 84.89,
+ "peak_mb": 169.4,
+ "layers": {
+ "CPU": 791
+ }
+ },
+ {
+ "model": "g8h_trained_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "j5ql4427p",
+ "ms": 187.85,
+ "peak_mb": 177.8,
+ "layers": {
+ "CPU": 791
+ }
+ },
+ {
+ "model": "g8h_trained_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "jglywwke5",
+ "ms": 195.18,
+ "peak_mb": 176.4,
+ "layers": {
+ "CPU": 791
+ }
+ },
+ {
+ "model": "g8h_trained_b4",
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "j568oo1vg",
+ "ms": 83.2,
+ "peak_mb": 233.4,
+ "layers": {
+ "CPU": 791
+ }
+ },
+ {
+ "model": "random_b4",
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp3zoomx5",
+ "ms": 2.47,
+ "peak_mb": 80.9,
+ "layers": {
+ "NPU": 890
+ }
+ },
+ {
+ "model": "random_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jgolddv4g",
+ "ms": 60.04,
+ "peak_mb": 149.3,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jpvl22w75",
+ "ms": 81.72,
+ "peak_mb": 131.9,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "jgjr33l7p",
+ "ms": 194.15,
+ "peak_mb": 134.0,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "jpe766v75",
+ "ms": 199.35,
+ "peak_mb": 134.5,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4",
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "jgzlzz7z5",
+ "ms": 78.69,
+ "peak_mb": 187.4,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b1",
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "j5wlyy9zp",
+ "ms": 1.88,
+ "peak_mb": 97.8,
+ "layers": {
+ "NPU": 827
+ }
+ },
+ {
+ "model": "random_b1",
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jg9zoo4qp",
+ "ms": 26.94,
+ "peak_mb": 136.7,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b1",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jp1noo8kg",
+ "ms": 36.88,
+ "peak_mb": 123.1,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b1",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "jgdd66vkg",
+ "ms": 77.58,
+ "peak_mb": 129.8,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b1",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "j57eoodqp",
+ "ms": 76.84,
+ "peak_mb": 122.1,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b1",
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "jp4yeewqp",
+ "ms": 33.22,
+ "peak_mb": 171.4,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0",
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jpxl001jp",
+ "ms": 1.6,
+ "peak_mb": 81.8,
+ "layers": {
+ "NPU": 827
+ }
+ },
+ {
+ "model": "random_b0",
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "j5m099zyg",
+ "ms": 23.86,
+ "peak_mb": 134.3,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jgnz119vg",
+ "ms": 32.68,
+ "peak_mb": 120.4,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "jprlxx4vp",
+ "ms": 79.72,
+ "peak_mb": 126.1,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0",
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "jp2roo7xg",
+ "ms": 68.58,
+ "peak_mb": 120.6,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0",
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "jpyo884r5",
+ "ms": 29.89,
+ "peak_mb": 166.4,
+ "layers": {
+ "CPU": 722
+ }
+ }
+]
\ No newline at end of file
diff --git a/benchmark/results/gen8_emdb1_annotated.json b/benchmark/results/gen8_emdb1_annotated.json
new file mode 100644
index 0000000..42b0f14
--- /dev/null
+++ b/benchmark/results/gen8_emdb1_annotated.json
@@ -0,0 +1,346 @@
+{
+ "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
+ "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
+ "dataset": "EMDB1",
+ "split": "emdb1",
+ "stride": 1,
+ "bbox": "annotated",
+ "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
+ "landmarks": "fitted",
+ "epoch": 295,
+ "source": "raw",
+ "n": 24103,
+ "n_dropped": 0,
+ "num_sequences": 17,
+ "results": [
+ {
+ "joint_set": "SMPL24+adapter",
+ "num_joints": 24,
+ "num_samples": 24103,
+ "MPJPE_mm": 88.76262010632146,
+ "PA_MPJPE_mm": 64.32690564747114,
+ "PCK3D_50mm": 0.5623694837433791,
+ "PCK3D_100mm": 0.8426821004300986,
+ "AUC_0_150mm": 0.6134404029481004,
+ "per_joint_PA_MPJPE_mm": {
+ "pelvis": 42.72739616523092,
+ "left_hip": 43.53664928490318,
+ "right_hip": 42.47946649645917,
+ "spine1": 51.80011363644271,
+ "left_knee": 58.04927382682903,
+ "right_knee": 56.29593421755156,
+ "spine2": 51.50166121350724,
+ "left_ankle": 70.10281185148675,
+ "right_ankle": 70.07250431760954,
+ "spine3": 50.67987416584745,
+ "left_foot": 77.79060701035993,
+ "right_foot": 78.21471337079649,
+ "neck": 49.27518692448127,
+ "left_collar": 43.24291462481559,
+ "right_collar": 42.80192669461389,
+ "head": 55.14169371785151,
+ "left_shoulder": 47.904915709508444,
+ "right_shoulder": 47.244650493662036,
+ "left_elbow": 61.798039706379,
+ "right_elbow": 57.81949942904837,
+ "left_wrist": 99.87869570494013,
+ "right_wrist": 101.27008016526156,
+ "left_hand": 119.02842811021682,
+ "right_hand": 125.18869870150422
+ }
+ },
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 24103,
+ "MPJPE_mm": 100.20960661675163,
+ "PA_MPJPE_mm": 67.70112248806943,
+ "PCK3D_50mm": 0.48429359712187575,
+ "PCK3D_100mm": 0.8266783625037785,
+ "AUC_0_150mm": 0.5808690468383478,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 70.23891257552113,
+ "right_knee": 53.370854089752505,
+ "right_hip": 51.9298041156822,
+ "left_hip": 49.25334293941451,
+ "left_knee": 56.7825543131599,
+ "left_ankle": 72.06322892772938,
+ "right_wrist": 102.62766164388755,
+ "right_elbow": 53.70033232898691,
+ "right_shoulder": 58.22250209407417,
+ "left_shoulder": 49.761324857441736,
+ "left_elbow": 58.86439891553489,
+ "left_wrist": 103.59978537156461,
+ "neck": 59.81198585359963,
+ "head": 107.58902680662273
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 24103,
+ "MPJPE_mm": 95.63595481944421,
+ "PA_MPJPE_mm": 64.06419658736174,
+ "PCK3D_50mm": 0.53339141738926,
+ "PCK3D_100mm": 0.8486426309311428,
+ "AUC_0_150mm": 0.6047484831005286,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 69.03874380342945,
+ "right_knee": 54.265726018154375,
+ "right_hip": 51.31842966016909,
+ "left_hip": 49.38816719105823,
+ "left_knee": 57.51416548085523,
+ "left_ankle": 70.59377879986292,
+ "right_wrist": 98.00290890921097,
+ "right_elbow": 50.425163158690815,
+ "right_shoulder": 57.98955549907401,
+ "left_shoulder": 54.372273287172675,
+ "left_elbow": 56.59655723988543,
+ "left_wrist": 99.26489000077778
+ }
+ }
+ ],
+ "per_sequence_SMPL24_adapter": {
+ "P1_14_outdoor_climb": {
+ "n": 1270,
+ "PA_MPJPE_mm": 75.1569086897259
+ },
+ "P2_23_outdoor_hug_tree": {
+ "n": 1086,
+ "PA_MPJPE_mm": 70.37429717601877
+ },
+ "P3_31_outdoor_workout": {
+ "n": 1216,
+ "PA_MPJPE_mm": 58.15779649961338
+ },
+ "P3_32_outdoor_soccer_warmup_a": {
+ "n": 1084,
+ "PA_MPJPE_mm": 55.763399060412645
+ },
+ "P3_33_outdoor_soccer_warmup_b": {
+ "n": 1433,
+ "PA_MPJPE_mm": 64.57149990299354
+ },
+ "P5_42_indoor_dancing": {
+ "n": 1291,
+ "PA_MPJPE_mm": 83.09306734173849
+ },
+ "P5_44_indoor_rom": {
+ "n": 1381,
+ "PA_MPJPE_mm": 55.185263572409305
+ },
+ "P6_49_outdoor_big_stairs_down": {
+ "n": 1559,
+ "PA_MPJPE_mm": 77.3131934192975
+ },
+ "P6_50_outdoor_workout": {
+ "n": 1532,
+ "PA_MPJPE_mm": 55.07497626438046
+ },
+ "P6_51_outdoor_dancing": {
+ "n": 1427,
+ "PA_MPJPE_mm": 63.936823334609144
+ },
+ "P7_57_outdoor_rock_chair": {
+ "n": 1558,
+ "PA_MPJPE_mm": 56.42255440924773
+ },
+ "P7_59_outdoor_rom": {
+ "n": 1839,
+ "PA_MPJPE_mm": 59.816166687025365
+ },
+ "P7_60_outdoor_workout": {
+ "n": 1693,
+ "PA_MPJPE_mm": 55.52638191914596
+ },
+ "P8_64_outdoor_skateboard": {
+ "n": 1704,
+ "PA_MPJPE_mm": 62.05609838052144
+ },
+ "P8_68_outdoor_handstand": {
+ "n": 1606,
+ "PA_MPJPE_mm": 83.5236655338729
+ },
+ "P8_69_outdoor_cartwheel": {
+ "n": 656,
+ "PA_MPJPE_mm": 58.87959907752564
+ },
+ "P9_76_outdoor_sitting": {
+ "n": 1768,
+ "PA_MPJPE_mm": 59.30274501084284
+ }
+ }
+ },
+ "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth": {
+ "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
+ "dataset": "EMDB1",
+ "split": "emdb1",
+ "stride": 1,
+ "bbox": "annotated",
+ "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
+ "landmarks": "fitted",
+ "epoch": 247,
+ "source": "raw",
+ "n": 24103,
+ "n_dropped": 0,
+ "num_sequences": 17,
+ "results": [
+ {
+ "joint_set": "SMPL24+adapter",
+ "num_joints": 24,
+ "num_samples": 24103,
+ "MPJPE_mm": 89.24478111621829,
+ "PA_MPJPE_mm": 66.89051058318199,
+ "PCK3D_50mm": 0.5427246262567592,
+ "PCK3D_100mm": 0.8293072093377035,
+ "AUC_0_150mm": 0.6015779501860072,
+ "per_joint_PA_MPJPE_mm": {
+ "pelvis": 43.810182621115835,
+ "left_hip": 44.13638723000304,
+ "right_hip": 42.90311175608087,
+ "spine1": 53.19471820477527,
+ "left_knee": 58.01026433731803,
+ "right_knee": 56.6788951694521,
+ "spine2": 53.05014683883006,
+ "left_ankle": 70.70555056990528,
+ "right_ankle": 71.9338142391788,
+ "spine3": 52.454466492137556,
+ "left_foot": 78.779893834194,
+ "right_foot": 80.9591141900794,
+ "neck": 52.30428934621626,
+ "left_collar": 46.13822355677994,
+ "right_collar": 45.72129105536187,
+ "head": 59.05837199161462,
+ "left_shoulder": 49.52196280870598,
+ "right_shoulder": 49.190675699586116,
+ "left_elbow": 63.988331123564954,
+ "right_elbow": 60.81275012273367,
+ "left_wrist": 106.9344451042545,
+ "right_wrist": 106.81214625580598,
+ "left_hand": 127.6084289102973,
+ "right_hand": 130.6647925383788
+ }
+ },
+ {
+ "joint_set": "J14",
+ "num_joints": 14,
+ "num_samples": 24103,
+ "MPJPE_mm": 100.28093707341523,
+ "PA_MPJPE_mm": 69.73662325205572,
+ "PCK3D_50mm": 0.46777816632191604,
+ "PCK3D_100mm": 0.814531089787282,
+ "AUC_0_150mm": 0.5708741153318393,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 72.38978289942779,
+ "right_knee": 54.03149026043775,
+ "right_hip": 51.356108506491154,
+ "left_hip": 49.811121151005466,
+ "left_knee": 56.808665864142974,
+ "left_ankle": 72.92868528434977,
+ "right_wrist": 108.19024936320407,
+ "right_elbow": 56.45063342066514,
+ "right_shoulder": 60.36568877055438,
+ "left_shoulder": 50.575689764376556,
+ "left_elbow": 60.60541881006109,
+ "left_wrist": 110.6946994511135,
+ "neck": 62.07527690734759,
+ "head": 110.02921507560288
+ }
+ },
+ {
+ "joint_set": "J12",
+ "num_joints": 12,
+ "num_samples": 24103,
+ "MPJPE_mm": 96.16395031415158,
+ "PA_MPJPE_mm": 66.1919422644989,
+ "PCK3D_50mm": 0.5132832704089394,
+ "PCK3D_100mm": 0.837381930326792,
+ "AUC_0_150mm": 0.5940371720113368,
+ "per_joint_PA_MPJPE_mm": {
+ "right_ankle": 71.03626620506343,
+ "right_knee": 54.86410905938645,
+ "right_hip": 51.47818801727088,
+ "left_hip": 50.6134082786491,
+ "left_knee": 57.6991870489433,
+ "left_ankle": 71.41209076510692,
+ "right_wrist": 103.01994285775166,
+ "right_elbow": 53.49245806856022,
+ "right_shoulder": 60.90576162520488,
+ "left_shoulder": 56.61177561619242,
+ "left_elbow": 57.93463650794004,
+ "left_wrist": 105.23548312391765
+ }
+ }
+ ],
+ "per_sequence_SMPL24_adapter": {
+ "P1_14_outdoor_climb": {
+ "n": 1270,
+ "PA_MPJPE_mm": 83.788047040943
+ },
+ "P2_23_outdoor_hug_tree": {
+ "n": 1086,
+ "PA_MPJPE_mm": 74.10063803076984
+ },
+ "P3_31_outdoor_workout": {
+ "n": 1216,
+ "PA_MPJPE_mm": 61.92140648476573
+ },
+ "P3_32_outdoor_soccer_warmup_a": {
+ "n": 1084,
+ "PA_MPJPE_mm": 60.40650615697361
+ },
+ "P3_33_outdoor_soccer_warmup_b": {
+ "n": 1433,
+ "PA_MPJPE_mm": 70.6191221994521
+ },
+ "P5_42_indoor_dancing": {
+ "n": 1291,
+ "PA_MPJPE_mm": 81.43654095908283
+ },
+ "P5_44_indoor_rom": {
+ "n": 1381,
+ "PA_MPJPE_mm": 53.06881540316626
+ },
+ "P6_49_outdoor_big_stairs_down": {
+ "n": 1559,
+ "PA_MPJPE_mm": 81.41758018488571
+ },
+ "P6_50_outdoor_workout": {
+ "n": 1532,
+ "PA_MPJPE_mm": 57.0616796141908
+ },
+ "P6_51_outdoor_dancing": {
+ "n": 1427,
+ "PA_MPJPE_mm": 65.66184766547114
+ },
+ "P7_57_outdoor_rock_chair": {
+ "n": 1558,
+ "PA_MPJPE_mm": 59.24821940321826
+ },
+ "P7_59_outdoor_rom": {
+ "n": 1839,
+ "PA_MPJPE_mm": 62.59178945483681
+ },
+ "P7_60_outdoor_workout": {
+ "n": 1693,
+ "PA_MPJPE_mm": 64.53617334097041
+ },
+ "P8_64_outdoor_skateboard": {
+ "n": 1704,
+ "PA_MPJPE_mm": 64.85601161182977
+ },
+ "P8_68_outdoor_handstand": {
+ "n": 1606,
+ "PA_MPJPE_mm": 78.86868543325276
+ },
+ "P8_69_outdoor_cartwheel": {
+ "n": 656,
+ "PA_MPJPE_mm": 58.5489513986187
+ },
+ "P9_76_outdoor_sitting": {
+ "n": 1768,
+ "PA_MPJPE_mm": 59.203835018123364
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/instanthmr/inference.py b/instanthmr/inference.py
index 27b20ed..d08c109 100644
--- a/instanthmr/inference.py
+++ b/instanthmr/inference.py
@@ -1,6 +1,7 @@
"""ONNX inference for InstantHMR.
-InstantHMR takes a 224x224 person crop and a 3-vector CLIFF condition
+InstantHMR takes a square person crop (224 px unless the ONNX metadata
+says otherwise) and a 3-vector CLIFF condition
(bbox center / scale in full-frame coords) and returns mhr_params, shape,
camera translation, and 70 joints in 2D (crop space) and 3D (camera coords,
rig-local, metres, Y-down).
@@ -200,6 +201,11 @@ class InstantHMR:
# Passing `focal=` alone still implies the angular form, which is how
# this argument behaved before `cliff_focal` existed.
meta = self.session.get_modelmeta().custom_metadata_map or {}
+ # The crop side the graph was trained at: stamped by tools/pth_to_onnx.py,
+ # else the graph's static input height, else the historical 224.
+ dim = self.session.get_inputs()[0].shape[2]
+ self.input_size = int(meta.get("image_size")
+ or (dim if isinstance(dim, int) else INPUT_SIZE))
if cliff_focal is None:
cliff_focal = (str(meta.get("cliff_focal", "")).lower() == "true"
or focal is not None)
@@ -281,7 +287,7 @@ class InstantHMR:
outs = self.session.run(
None,
{
- self._in_image: crop[np.newaxis], # (1, 3, 224, 224)
+ self._in_image: crop[np.newaxis], # (1, 3, S, S), S = self.input_size
self._in_cliff: cliff_cond[np.newaxis], # (1, 3)
},
)
@@ -296,8 +302,8 @@ class InstantHMR:
joints_3d_local = self._joints_3d(mhr_params, shape_params)
# Re-project the 2D head from normalised crop space → full-frame pixels.
- crop_px = (joints_2d_norm + 1.0) * 0.5 * INPUT_SIZE
- scale = sq_size / INPUT_SIZE
+ crop_px = (joints_2d_norm + 1.0) * 0.5 * self.input_size
+ scale = sq_size / self.input_size
joints_2d = np.stack(
[crop_px[:, 0] * scale + sq_x1, crop_px[:, 1] * scale + sq_y1],
axis=-1,
@@ -356,7 +362,7 @@ class InstantHMR:
batch = max(padded_to or n, n) # always >= n; zero-pad the rest
# Use zeros so padded slots produce deterministic (discarded) outputs.
- crops = np.zeros((batch, 3, INPUT_SIZE, INPUT_SIZE), dtype=np.float32)
+ crops = np.zeros((batch, 3, self.input_size, self.input_size), dtype=np.float32)
cliffs = np.zeros((batch, 3), dtype=np.float32)
sq_meta = [] # per-person (sq_x1, sq_y1, sq_size, bbox)
for i, det in enumerate(detections):
@@ -394,8 +400,8 @@ class InstantHMR:
joints_3d_local = joints_3d_local_b[i]
cam_trans = cam_trans_b[i]
- crop_px = (joints_2d_norm + 1.0) * 0.5 * INPUT_SIZE
- scale = sq_size / INPUT_SIZE
+ crop_px = (joints_2d_norm + 1.0) * 0.5 * self.input_size
+ scale = sq_size / self.input_size
joints_2d = np.stack(
[crop_px[:, 0] * scale + sq_x1, crop_px[:, 1] * scale + sq_y1],
axis=-1,
@@ -425,7 +431,7 @@ class InstantHMR:
upfront so no mid-demo stall occurs when persons first appear.
"""
dummy_image = np.zeros(
- (max_batch_size, 3, INPUT_SIZE, INPUT_SIZE), dtype=np.float32
+ (max_batch_size, 3, self.input_size, self.input_size), dtype=np.float32
)
dummy_cliff = np.zeros((max_batch_size, 3), dtype=np.float32)
# Large batch first — allocates the most memory; batch=1 reuses the plan.
@@ -500,10 +506,10 @@ class InstantHMR:
cv2.BORDER_CONSTANT, value=(0, 0, 0),
)
- crop_224 = cv2.resize(patch, (INPUT_SIZE, INPUT_SIZE), interpolation=cv2.INTER_LINEAR)
- crop = crop_224.astype(np.float32) / 255.0
+ crop_sq = cv2.resize(patch, (self.input_size, self.input_size), interpolation=cv2.INTER_LINEAR)
+ crop = crop_sq.astype(np.float32) / 255.0
crop = (crop - IMAGENET_MEAN) / IMAGENET_STD
- crop = np.transpose(crop, (2, 0, 1)).astype(np.float32) # (3, 224, 224)
+ crop = np.transpose(crop, (2, 0, 1)).astype(np.float32) # (3, S, S)
return crop, float(sq_x1), float(sq_y1), float(sq_size), cliff_cond
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index f239791..c522325 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -570,21 +570,69 @@ resampled identity; gradients agree to 2e-07 relative in `model_params` and
so the readout is translation-equivariant and commutes with `to_vision` — which
is why the mapping is applied after the unit/axis conversion.
-**Scoring is unchanged, on purpose.** `val3dpw.py` and
-`benchmark/eval_3dpw_ckpt.py` both go through `get_native_keypoints`, i.e. the
-fitted readout, and neither was touched. The flag changes what the model is
-trained to match, not what it is measured with, so the reported J14 PA-MPJPE
-stays comparable with earlier generations and keeps measuring the path that
-actually ships. **Deployment is unchanged**: the exported graph never emitted
-these 3D keypoints — the consumer derives them — so no operator is added to the
-phone runtime. Making the deployed readout exact is a separate decision with its
-own latency measurement.
+**Scoring stays on the fitted readout by default, and it is now a switch.**
+`val3dpw.evaluate(..., exact_landmarks=)`, `--val-landmarks {fitted,exact}` on
+the trainer, and `--landmarks {fitted,exact}` on `benchmark/eval_3dpw_ckpt.py`
+and `benchmark/eval_emdb_ckpt.py` all default to `fitted`, which reproduces
+every recorded number to the decimal.
+
+Measured 2026-09-22, 3DPW test stride 1, J14+adapter PA-MPJPE:
+
+| run | trained with `--exact-landmarks` | fitted | exact |
+|---|---|---|---|
+| g8h_s0 | yes | 40.80 | **40.66** |
+| g8h_s1 | yes | 40.89 | **40.77** |
+| g8r_s0 | yes | 41.27 | **41.13** |
+| g7h_s0 | **no** | 42.90 | **42.77** |
+
+0.13 mm, the same on the run that never saw the flag — the switch is a property
+of the operator, not of the training. EMDB-1 SMPL24+adapter moves 0.09-0.11 mm
+the same way. It is an offset, not a re-ranking, and it is far inside the
+34-49 mm seed spread.
+
+What it removes is nonetheless a genuine mismatch: the MHR70 -> J14 and
+MHR70 -> SMPL24 adapters are fitted by `benchmark/fit_adapter_mhr.py` from the
+stored annotation `joints_3d`, which is the teacher's exact mesh+joint readout,
+so `fitted` hands the adapter keypoints from a different operator than the one
+it was fitted on. The GT side needs nothing: 3DPW's H36M joints and EMDB's SMPL
+joints are SMPL-derived and independent of any MHR readout.
+
+`fitted` remains the default because it is what the exported graph's consumer
+computes, so the published number keeps describing the path that ships, and
+because flipping it would shift every recorded row by 0.13 mm. **Deployment is
+unchanged**: the exported graph never emitted these 3D keypoints — the consumer
+derives them — so no operator is added to the phone runtime. Making the deployed
+readout exact is a separate decision with its own latency measurement.
`instanthmr_distill_train/assets/mhr_landmarks70.npz` (167 KB) is **a new file
and must be rsynced**, or the job dies at startup. Default off; with it off
every loss term is bit-identical, verified across `legacy`, `rebalanced`,
`w_verts` on/off and `cont_head` on/off.
+## `--image-size` — the network input side (added 2026-09-25)
+
+Default 224, i.e. unchanged. Must be a multiple of 32, because the decoder's
+positional grid is `image_size // 32` (7x7 at 224, 9x9 at 288). One setting
+drives every place that builds the input: the training dataset, `val3dpw`
+(and so the 3DPW / EMDB / SMPL-fit harnesses), `tools/pth_to_onnx.py` and
+`instanthmr.inference`.
+
+**It is recoverable from the weights.** `mem_pos_embed` is a saved buffer with
+one row per patch, so `config_from_checkpoint` reads the grid off it; the exact
+value comes from `run_config.json`, and a `run_config.json` that contradicts
+the weights raises instead of silently resizing. The export stamps
+`image_size` into the ONNX metadata, and `instanthmr.inference` reads it back
+(falling back to the graph's static input shape, then 224). The harnesses
+refuse to score checkpoints of different sizes in one invocation, like
+`--cliff-focal`.
+
+On today's 224 corpus a larger size only upsamples the stored crop; the real
+test of resolution needs the context-crop corpus (`build_split.py --context`).
+
+Verified by `tools/verify_input_size.py`: baseline and v2 samples bit-identical
+at 224; a 288 checkpoint round-trips through the config, the export metadata,
+the inference package and both harnesses.
+
## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)
**Do not retry this without reading the failure mode.** The flag is gone from
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 412e904..4e38c99 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -389,7 +389,8 @@ def build_jz_loaders(cfg, args, rank: int, world: int):
crop_centre_fix=cfg.crop_centre_fix,
cliff_focal=cfg.cliff_focal,
occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
- jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality)
+ jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality,
+ image_size=cfg.image_size)
# Rank 0 builds the index (or reads the cache) while the others wait, then
# everyone loads the same cache. Building on all ranks at once would put
@@ -538,7 +539,9 @@ def train(args, cfg, rank, local_rank, world):
mhr_module = T.MHRForwardPass(
cfg.mhr_model_path, device, kp_regressor=W,
n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,
- landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)
+ landmark_assets=(cfg.landmark_assets_path
+ if cfg.exact_landmarks or args.val_landmarks == "exact"
+ else None))
criterion = T.DistillationLoss(cfg, mhr_module)
train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)
@@ -563,7 +566,8 @@ def train(args, cfg, rank, local_rank, world):
f"-> J14 numbers match benchmark/eval_3dpw_ckpt.py")
dpw = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,
split=args.val_3dpw_split, stride=args.val_3dpw_stride,
- gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal)
+ gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal,
+ input_size=cfg.image_size)
# Stride-shard across ranks; the metric is all-reduced afterwards.
dpw_loader = torch.utils.data.DataLoader(
torch.utils.data.Subset(dpw, list(range(rank, len(dpw), world))),
@@ -571,7 +575,8 @@ def train(args, cfg, rank, local_rank, world):
num_workers=max(1, cfg.num_workers // 2), pin_memory=True)
log(f"3DPW {args.val_3dpw_split}: {len(dpw):,} person-frames "
f"(stride {args.val_3dpw_stride}, GT {args.val_3dpw_gt}) "
- f"-> selection metric is J14 PA-MPJPE (adapter-applied)")
+ f"-> selection metric is J14 PA-MPJPE (adapter-applied, "
+ f"{args.val_landmarks} 70-keypoint readout)")
# Optional read-only monitor on the split the paper reports. Selection
# NEVER touches it: picking the best of ~100 epochs on the test split
@@ -581,7 +586,8 @@ def train(args, cfg, rank, local_rank, world):
if args.val_3dpw_test:
dpw_t = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,
split="test", stride=args.val_3dpw_test_stride,
- gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal)
+ gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal,
+ input_size=cfg.image_size)
dpw_test_loader = torch.utils.data.DataLoader(
torch.utils.data.Subset(dpw_t, list(range(rank, len(dpw_t), world))),
batch_size=cfg.batch_size, shuffle=False, drop_last=False,
@@ -695,6 +701,7 @@ def train(args, cfg, rank, local_rank, world):
"lr": cfg.lr, "lr_arg": args.lr, "lr_scaling": args.lr_scaling,
"mix": args.mix, "steps_per_epoch": steps_per_epoch,
"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,
+ "image_size": cfg.image_size,
"cont_head": cfg.cont_head, "root_rot_loss": cfg.root_rot_loss,
"exact_landmarks": cfg.exact_landmarks,
"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,
@@ -704,6 +711,7 @@ def train(args, cfg, rank, local_rank, world):
"pose_split": cfg.pose_split, "pose_beta": cfg.pose_beta,
"finger_weight": cfg.finger_weight,
"val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,
+ "val_landmarks": args.val_landmarks if args.val_3dpw else None,
"cliff_focal": cfg.cliff_focal,
"w_verts": cfg.w_verts, "n_verts": cfg.n_verts,
"crop_centre_fix": cfg.crop_centre_fix,
@@ -884,20 +892,22 @@ def train(args, cfg, rank, local_rank, world):
dpw_raw = dpw_em = dpw_test_raw = dpw_test_em = None
if dpw_loader is not None:
import val3dpw
+ exact_lm = args.val_landmarks == "exact"
dpw_raw = all_reduce_mean(
val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device,
cfg.use_amp, gt=args.val_3dpw_gt,
- adapter=dpw_adapter),
+ adapter=dpw_adapter, exact_landmarks=exact_lm),
device)
dpw_em = all_reduce_mean(
val3dpw.evaluate(ema, dpw_loader, mhr_module, device,
cfg.use_amp, gt=args.val_3dpw_gt,
- adapter=dpw_adapter),
+ adapter=dpw_adapter, exact_landmarks=exact_lm),
device)
if dpw_test_loader is not None:
dpw_test_raw = all_reduce_mean(val3dpw.evaluate(
ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp,
- gt=args.val_3dpw_gt, adapter=dpw_adapter), device)
+ gt=args.val_3dpw_gt, adapter=dpw_adapter,
+ exact_landmarks=exact_lm), device)
dpw_test_em = all_reduce_mean(val3dpw.evaluate(
ema, dpw_test_loader, mhr_module, device, cfg.use_amp,
gt=args.val_3dpw_gt, adapter=dpw_adapter), device)
@@ -1103,6 +1113,19 @@ def parse_args():
"benchmark/make_3dpw_gt.py). jointpositions is the raw "
"pickle field the runs before this flag selected on — "
"several mm apart, so a resumed run must keep its own.")
+ p.add_argument("--val-landmarks", dest="val_landmarks", default="fitted",
+ choices=("fitted", "exact"),
+ help="Which readout the 3DPW selection metric scores. "
+ "fitted (default) is the (70, 127) skeleton matrix the "
+ "exported graph's consumer uses, and what every "
+ "recorded run was selected on. exact is the teacher's "
+ "mesh+joint mapping -- what --exact-landmarks trains "
+ "against, and the operator the adapter was fitted "
+ "from. Measured 0.13 mm of J14+adapter PA-MPJPE on "
+ "3DPW test and 0.05 mm on validation, with or without "
+ "--exact-landmarks training, so it does not move "
+ "selection; it is an offset, and mixing the two in one "
+ "table is what to avoid.")
p.add_argument("--mix", default="sqrt",
help="Dataset mixture: sqrt | uniform | equal | "
"'sa1b=0.4,aic=0.35,coco=0.15,mpii=0.1'.")
@@ -1113,6 +1136,11 @@ def parse_args():
p.add_argument("--w_simcc", type=float, default=None)
p.add_argument("--weight_decay", type=float, default=None)
p.add_argument("--backbone", default=None)
+ p.add_argument("--image-size", dest="image_size", type=int, default=None,
+ help="Network input side in px (default 224). A multiple of "
+ "32: the decoder's positional grid is image_size // 32. "
+ "Recorded in run_config.json and read back by "
+ "config_from_checkpoint, the harnesses and the export.")
p.add_argument("--ema-decay", type=float, default=None)
p.add_argument("--sync-bn", action="store_true")
@@ -1151,6 +1179,15 @@ def parse_args():
"farthest-point subset of the rig's own 18,439-vertex "
"mesh (default 595, lod6's count). Costs 0.6 ms per "
"batch-64 step against 16.7 ms for the full mesh.")
+ p.add_argument("--early-stop-patience", dest="early_stop_patience", type=int, default=None,
+ help="Stop after this many epochs with no improvement in the "
+ "selection metric (config default 150). The counter is "
+ "carried in the checkpoint, so it accumulates across "
+ "chained links. Raise it to disable: with OneCycleLR, "
+ "which anneals to ~0 only in the last 10%% of the planned "
+ "epochs, a plateau at high LR can expire the patience "
+ "just before the refinement phase that would have ended "
+ "it -- g8r_s0 stopped at 243 of 300 that way.")
p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",
help="Supervise the 70 keypoints with the teacher's exact "
"mesh+joint mapping over the 468 vertices it references, "
@@ -1208,11 +1245,17 @@ def build_cfg(args):
cfg.cont_head = True
cfg.root_rot_loss = True
if args.exact_landmarks: cfg.exact_landmarks = True
+ if args.early_stop_patience is not None:
+ cfg.early_stop_patience = args.early_stop_patience
if args.w_verts is not None: cfg.w_verts = args.w_verts
if args.n_verts is not None: cfg.n_verts = args.n_verts
if args.w_simcc is not None: cfg.w_simcc = args.w_simcc
if args.weight_decay is not None: cfg.weight_decay = args.weight_decay
if args.ema_decay is not None: cfg.ema_decay = args.ema_decay
+ if args.image_size is not None:
+ if args.image_size % 32:
+ raise SystemExit(f"--image-size {args.image_size} is not a multiple of 32")
+ cfg.image_size = args.image_size
if args.preset == "v2":
apply_v2_preset(cfg)
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 8f2dc3a..75395d1 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -434,7 +434,9 @@ def apply_rebalanced_losses(cfg) -> None:
def config_from_checkpoint(state, ckpt_path=None, **overrides):
"""Rebuild the DistillConfig a checkpoint was actually trained with.
- Two sources, because the checkpoint dict carries neither.
+ Two sources, because the checkpoint dict carries neither. (``image_size``
+ sits in both: the positional-embedding grid in the weights and the value
+ in run_config.json, which must agree.)
* The ARCHITECTURE flags are readable from the weights. ``--bound-scales``
registers ``scale_lo`` / ``scale_hi`` buffers, so their presence in the
@@ -484,6 +486,22 @@ def config_from_checkpoint(state, ckpt_path=None, **overrides):
setattr(cfg, k, run_cfg[k])
prov[k] = "run_config.json"
+ # The input size IS in the weights: the decoder's positional embedding is a
+ # saved buffer with one row per 32-px patch, so its grid fixes the size up
+ # to the stride. run_config.json carries the exact value; a contradiction
+ # between the two means the wrong run_config.json sits beside the weights.
+ if "mem_pos_embed" in state:
+ grid = math.isqrt(state["mem_pos_embed"].shape[1])
+ cfg.image_size = grid * 32
+ prov["image_size"] = "checkpoint weights"
+ if "image_size" in run_cfg and int(run_cfg["image_size"]) != grid * 32:
+ raise ValueError(
+ f"run_config.json says image_size={run_cfg['image_size']} but "
+ f"the weights have a {grid}x{grid} patch grid ({grid * 32} px)")
+ elif "image_size" in run_cfg:
+ cfg.image_size = int(run_cfg["image_size"])
+ prov["image_size"] = "run_config.json"
+
for k, v in overrides.items():
if v is not None:
setattr(cfg, k, v)
@@ -866,7 +884,8 @@ def build_dataloaders(cfg):
crop_centre_fix=cfg.crop_centre_fix,
cliff_focal=cfg.cliff_focal,
occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
- jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality)
+ jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality,
+ image_size=cfg.image_size)
full_dataset = SAM3DStudentDataset(
cfg.data_root,
augment=cfg.augment,
@@ -1421,10 +1440,21 @@ class MHRForwardPass:
return (torch.einsum('kj,bjc->bkc', self.lm_W_joint, joints_vision)
+ torch.einsum('kv,bvc->bkc', self.lm_W_vert, v))
- def get_native_keypoints(self, model_params, shape_params):
- """(B, 70, 3) native keypoints in the annotation frame."""
- j = self.get_joints(model_params, shape_params)[..., :3]
- return self.regress_keypoints(self.to_vision(j))
+ def get_native_keypoints(self, model_params, shape_params, exact: bool = False):
+ """(B, 70, 3) native keypoints in the annotation frame.
+
+ `exact=True` switches from the fitted (70, 127) skeleton matrix to the
+ teacher's own `Wj @ J + Wv @ V` readout -- the operator the stored
+ annotations came out of, and therefore the one the MHR70 -> J14 / SMPL24
+ adapters were fitted against. It needs a module built with
+ `landmark_assets=`. Default off: the fitted readout is what the exported
+ graph's consumer uses, so it is what the published metric measures.
+ """
+ if not exact:
+ j = self.get_joints(model_params, shape_params)[..., :3]
+ return self.regress_keypoints(self.to_vision(j))
+ j, v = self.get_joints_and_vertices(model_params, shape_params)
+ return self.regress_keypoints_exact(self.to_vision(j), self.to_vision(v))
class DistillationLoss(nn.Module):
def __init__(self, cfg, mhr_module):
@@ -2477,7 +2507,7 @@ def _mock_preds_from(batch):
def run_self_tests():
print("--- Architecture Output Shapes ---")
test_model = InstantHMRStudent(cfg, pretrained=False)
- out = test_model(torch.randn(2, 3, 224, 224), torch.randn(2, 3))
+ out = test_model(torch.randn(2, 3, cfg.image_size, cfg.image_size), torch.randn(2, 3))
for k, v in out.items():
print(f" {k}: {tuple(v.shape)}")
n_par = sum(p.numel() for p in test_model.parameters())
@@ -2496,7 +2526,8 @@ def run_self_tests():
print("\n--- Perfect Student (identity batch) ---")
clean_ds = SAM3DStudentDataset(cfg.data_root, augment=False,
- max_images=256, per_dataset_caps=cfg.per_dataset_caps)
+ max_images=256, per_dataset_caps=cfg.per_dataset_caps,
+ image_size=cfg.image_size)
clean_loader = DataLoader(clean_ds, batch_size=32, shuffle=False, num_workers=0)
cb = {k: v.to(device) for k, v in next(iter(clean_loader)).items()}
l_clean = criterion(_mock_preds_from(cb), cb)
@@ -2514,7 +2545,8 @@ def run_self_tests():
max_images=256, per_dataset_caps=cfg.per_dataset_caps,
geom_p=1.0, geom_rot_deg=cfg.geom_rot_deg,
geom_scale_range=cfg.geom_scale_range,
- geom_trans=cfg.geom_trans, geom_flip_p=0.0)
+ geom_trans=cfg.geom_trans, geom_flip_p=0.0,
+ image_size=cfg.image_size)
aug_loader = DataLoader(aug_ds, batch_size=32, shuffle=False, num_workers=0)
ab = {k: v.to(device) for k, v in next(iter(aug_loader)).items()}
@@ -2549,7 +2581,8 @@ def run_self_tests():
flip_ds = SAM3DStudentDataset(cfg.data_root, augment=True,
max_images=256, per_dataset_caps=cfg.per_dataset_caps,
geom_p=1.0, geom_rot_deg=0.0, geom_scale_range=0.0,
- geom_trans=0.0, geom_flip_p=1.0)
+ geom_trans=0.0, geom_flip_p=1.0,
+ image_size=cfg.image_size)
flip_loader = DataLoader(flip_ds, batch_size=32, shuffle=False, num_workers=0)
fb = {k: v.to(device) for k, v in next(iter(flip_loader)).items()}
with torch.no_grad():
@@ -2581,7 +2614,8 @@ def run_self_tests():
max_images=256, per_dataset_caps=cfg.per_dataset_caps,
geom_p=1.0, geom_rot_deg=0.0,
geom_scale_range=cfg.geom_scale_range,
- geom_trans=cfg.geom_trans, geom_flip_p=0.0)
+ geom_trans=cfg.geom_trans, geom_flip_p=0.0,
+ image_size=cfg.image_size)
st_loader = DataLoader(st_ds, batch_size=32, shuffle=False, num_workers=0)
sb = {k: v.to(device) for k, v in next(iter(st_loader)).items()}
l_aug = criterion(_mock_preds_from(sb), sb)
@@ -2632,7 +2666,8 @@ def run_overfit_test(steps=3000, subset=8, lr=5e-4):
max_images=subset,
per_dataset_caps=cfg.per_dataset_caps,
geom_p=0.0, # STRICTLY DISABLE SPATIAL AUGS
- geom_flip_p=0.0 # STRICTLY DISABLE FLIPS
+ geom_flip_p=0.0, # STRICTLY DISABLE FLIPS
+ image_size=cfg.image_size
)
overfit_loader = DataLoader(overfit_dataset, batch_size=subset, shuffle=False, num_workers=0)
batch = next(iter(overfit_loader))
diff --git a/instanthmr_distill_train/val3dpw.py b/instanthmr_distill_train/val3dpw.py
index 8700837..222305d 100644
--- a/instanthmr_distill_train/val3dpw.py
+++ b/instanthmr_distill_train/val3dpw.py
@@ -62,7 +62,7 @@ MM = 1000.0
def _preprocess(image_rgb: np.ndarray, bbox: np.ndarray, h: int, w: int,
- focal: float | None = None):
+ focal: float | None = None, size: int = INPUT_SIZE):
"""Square 1.2x crop + ImageNet normalise + CLIFF conditioning.
A transcription of ``instanthmr.inference.InstantHMR._preprocess``. It is
@@ -98,7 +98,7 @@ def _preprocess(image_rgb: np.ndarray, bbox: np.ndarray, h: int, w: int,
if pl or pt or pr or pb:
patch = cv2.copyMakeBorder(patch, pt, pb, pl, pr,
cv2.BORDER_CONSTANT, value=(0, 0, 0))
- crop = cv2.resize(patch, (INPUT_SIZE, INPUT_SIZE), interpolation=cv2.INTER_LINEAR)
+ crop = cv2.resize(patch, (size, size), interpolation=cv2.INTER_LINEAR)
crop = (crop.astype(np.float32) / 255.0 - IMAGENET_MEAN) / IMAGENET_STD
return np.ascontiguousarray(crop.transpose(2, 0, 1)), cliff
@@ -115,13 +115,15 @@ class ThreeDPWValSet(Dataset):
def __init__(self, sequence_dir: str, image_root: str,
split: str = "validation", stride: int = 5,
bbox_scale: float = 1.2, gt: str = "h36m",
- gt_dir: str | None = None, cliff_focal: bool = False):
+ gt_dir: str | None = None, cliff_focal: bool = False,
+ input_size: int = INPUT_SIZE):
self.samples, self.gt = P.build_samples(
sequence_dir, image_root, split=split,
stride=stride, bbox_scale=bbox_scale, gt=gt, gt_dir=gt_dir)
self.gt = np.asarray(self.gt, dtype=np.float32)
self.gt_source = gt
self.cliff_focal = cliff_focal
+ self.input_size = input_size
def __len__(self) -> int:
return len(self.samples)
@@ -132,14 +134,15 @@ class ThreeDPWValSet(Dataset):
if bgr is None:
# A missing frame must not kill a 20-hour job; return a zero crop
# and flag it so the metric can drop it.
- return {"image": torch.zeros(3, INPUT_SIZE, INPUT_SIZE),
+ return {"image": torch.zeros(3, self.input_size, self.input_size),
"cliff_cond": torch.zeros(3),
"gt": torch.from_numpy(self.gt[i]),
"ok": torch.tensor(0.0)}
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
h, w = rgb.shape[:2]
crop, cliff = _preprocess(rgb, s["bbox"], h, w,
- s["focal"] if self.cliff_focal else None)
+ s["focal"] if self.cliff_focal else None,
+ size=self.input_size)
return {"image": torch.from_numpy(crop),
"cliff_cond": torch.from_numpy(cliff),
"gt": torch.from_numpy(self.gt[i]),
@@ -148,12 +151,20 @@ class ThreeDPWValSet(Dataset):
@torch.no_grad()
def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
- gt: str = "h36m", adapter: np.ndarray | None = None) -> dict:
+ gt: str = "h36m", adapter: np.ndarray | None = None,
+ exact_landmarks: bool = False) -> dict:
"""J14 PA-MPJPE and MPJPE in mm, adapter-applied — the paper number.
- Predictions come from ``mhr_params`` through the same FK + (70, 127)
- regressor the training loss uses, so this measures exactly the geometry the
- model is being trained to produce.
+ Predictions come from ``mhr_params`` through FK and the fitted (70, 127)
+ regressor, which is the readout the exported graph's consumer uses.
+
+ ``exact_landmarks=True`` uses the teacher's mesh+joint mapping instead —
+ the readout ``--exact-landmarks`` trains against and the one the adapter
+ was fitted from. Measured on 3DPW test stride 5 it is worth 0.13 mm of
+ J14+adapter PA-MPJPE, the same amount on a run trained WITHOUT
+ ``--exact-landmarks``, so it is a property of the operator and not of the
+ training flag. Off by default: flipping it shifts every recorded number by
+ that much and stops the metric describing the deployed path.
"""
model.eval()
preds, gts = [], []
@@ -166,7 +177,8 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):
out = model(img, cc)
kp = mhr_module.get_native_keypoints(out["mhr_params"].float(),
- out["shape_params"].float())
+ out["shape_params"].float(),
+ exact=exact_landmarks)
# Drop non-finite predictions BEFORE they reach the metric. The forward
# runs under fp16 autocast, so an early-training head output above
# 65504 is already inf by the time .float() sees it, and the MHR forward
diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.py
index 7906ea2..5e4c52d 100644
--- a/tools/ddp_smoke.py
+++ b/tools/ddp_smoke.py
@@ -3,6 +3,7 @@
python tools/ddp_smoke.py --backbone hgnetv2_b4 --w-verts 0.35
python tools/ddp_smoke.py --backbone repvit_m2_3 # the control
+ python tools/ddp_smoke.py --image-size 288 # a non-224 input
WHY THIS EXISTS. A plain single-process smoke run -- `train_distill_jz.py` on a
laptop, `--self-test`, `--overfit-test` -- builds no DDP Reducer at all, so it
@@ -51,6 +52,8 @@ def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--backbone", default=None, help="timm name; default is the config's")
+ p.add_argument("--image-size", dest="image_size", type=int, default=None,
+ help="network input side; the positional grid and every crop follow it")
p.add_argument("--data_root", default="data")
p.add_argument("--batch_size", type=int, default=2,
help="Small on purpose. Which parameters receive a gradient "
@@ -98,10 +101,15 @@ def main() -> int:
cfg.exact_landmarks = True
if args.backbone:
cfg.backbone = args.backbone
+ if args.image_size is not None:
+ if args.image_size % 32:
+ raise SystemExit(f"--image-size {args.image_size} is not a multiple of 32")
+ cfg.image_size = args.image_size
ds = T.SAM3DStudentDataset(cfg.data_root, augment=True, max_images=args.batch_size,
per_dataset_caps={}, cliff_focal=cfg.cliff_focal,
- crop_centre_fix=cfg.crop_centre_fix)
+ crop_centre_fix=cfg.crop_centre_fix,
+ image_size=cfg.image_size)
loader = torch.utils.data.DataLoader(ds, batch_size=args.batch_size, num_workers=0)
batch = {k: (v.to(T.device) if torch.is_tensor(v) else v)
for k, v in next(iter(loader)).items()}
diff --git a/tools/profiling/export_family.py b/tools/profiling/export_family.py
new file mode 100644
index 0000000..04f0bf7
--- /dev/null
+++ b/tools/profiling/export_family.py
@@ -0,0 +1,47 @@
+"""Usage: python tools/profiling/export_family.py <out_dir> then python tools/profiling/profile_family.py <out_dir>
+Ticket 03: ONNX exports of the g8h architecture with the backbone swapped.
+Random weights for B0/B1/B4, plus the trained g8h, all through one path."""
+import io, sys, copy, json
+from pathlib import Path
+import torch, onnx, onnxsim
+REPO = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(REPO / "instanthmr_distill_train"))
+import train_distill_mhr_only as T
+
+CKPT = REPO / "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_ema.pth"
+OUT = Path(sys.argv[1])
+blob = torch.load(CKPT, map_location="cpu", weights_only=False)
+state = {k.replace("_orig_mod.", ""): v for k, v in blob["model_state_dict"].items()}
+cfg, _ = T.config_from_checkpoint(state, CKPT)
+cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2
+cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])
+
+class Deploy(torch.nn.Module):
+ def __init__(self, m): super().__init__(); self.m = m
+ def forward(self, image, cliff_cond):
+ o = self.m(image, cliff_cond)
+ return o["mhr_params"], o["shape_params"], o["cam_trans"], o["joints_2d"]
+
+def export(model, name):
+ model.eval()
+ buf = io.BytesIO()
+ torch.onnx.export(Deploy(model), (torch.zeros(1, 3, 224, 224), torch.zeros(1, 3)), buf,
+ input_names=["image", "cliff_cond"],
+ output_names=["mhr_params", "shape_params", "cam_trans", "joints_2d"],
+ opset_version=17)
+ m, ok = onnxsim.simplify(onnx.load_from_string(buf.getvalue()))
+ assert ok
+ p = OUT / f"{name}.onnx"; onnx.save(m, p)
+ n = sum(x.numel() for x in model.parameters())
+ nb = sum(x.numel() for x in model.backbone.parameters())
+ print(json.dumps(dict(name=name, params_M=round(n/1e6, 2), backbone_M=round(nb/1e6, 2),
+ mb=round(p.stat().st_size/2**20, 1))), flush=True)
+
+trained = T.InstantHMRStudent(copy.deepcopy(cfg), pretrained=False)
+missing, unexpected = trained.load_state_dict(state, strict=False)
+assert not missing and not unexpected, (missing[:5], unexpected[:5])
+export(trained, "g8h_trained_b4")
+for bb in ("hgnetv2_b4", "hgnetv2_b1", "hgnetv2_b0"):
+ c = copy.deepcopy(cfg); c.backbone = bb
+ torch.manual_seed(0)
+ export(T.InstantHMRStudent(c, pretrained=False), f"random_{bb.split('_')[1]}")
diff --git a/tools/profiling/profile_family.py b/tools/profiling/profile_family.py
new file mode 100644
index 0000000..85bca8c
--- /dev/null
+++ b/tools/profiling/profile_family.py
@@ -0,0 +1,35 @@
+import json, sys
+from pathlib import Path
+import qai_hub as hub
+D = Path(sys.argv[1])
+CPU = ["Samsung Galaxy S23", "Samsung Galaxy A73 5G", "Samsung Galaxy A53 5G",
+ "Samsung Galaxy A14 5G", "Google Pixel 8"]
+specs = dict(image=(1, 3, 224, 224), cliff_cond=(1, 3))
+names = ["g8h_trained_b4", "random_b4", "random_b1", "random_b0"]
+comp = {n: hub.submit_compile_job(model=str(D / f"{n}.onnx"), device=hub.Device("Samsung Galaxy S23"),
+ input_specs=specs, options="--target_runtime onnx", name=f"t03-{n}")
+ for n in names}
+jobs = []
+for n, cj in comp.items():
+ m = cj.get_target_model()
+ if m is None:
+ print(json.dumps(dict(model=n, compile_failed=cj.get_status().message)), flush=True); continue
+ jobs.append((n, "npu", "Samsung Galaxy S23",
+ hub.submit_profile_job(model=m, device=hub.Device("Samsung Galaxy S23"), name=f"t03-{n}-npu")))
+ for d in CPU:
+ jobs.append((n, "cpu", d, hub.submit_profile_job(model=m, device=hub.Device(d),
+ name=f"t03-{n}-cpu", options="--compute_unit cpu")))
+out = []
+for n, u, d, j in jobs:
+ row = dict(model=n, unit=u, device=d, job=j.job_id)
+ if j.wait().success:
+ p = j.download_profile(); s = p["execution_summary"]
+ units = {}
+ for l in p.get("execution_detail", []):
+ units[l.get("compute_unit")] = units.get(l.get("compute_unit"), 0) + 1
+ row.update(ms=round(s["estimated_inference_time"] / 1000, 2),
+ peak_mb=round(s.get("estimated_inference_peak_memory", 0) / 2**20, 1), layers=units)
+ else:
+ row["error"] = j.get_status().message
+ print(json.dumps(row), flush=True); out.append(row)
+json.dump(out, open(D / "profile.json", "w"), indent=1)
diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.py
index 104dea0..9479237 100644
--- a/tools/pth_to_onnx.py
+++ b/tools/pth_to_onnx.py
@@ -7,7 +7,8 @@ It reuses the exact model architecture and deploy wrapper from the training
script that produced the checkpoint, plus the FP32 export recipe from notebook
Cell 15, so the graph I/O matches ``instanthmr/inference.py``:
- inputs : image (N, 3, 224, 224), cliff_cond (N, 3)
+ inputs : image (N, 3, S, S), cliff_cond (N, 3); S = the checkpoint's input
+ size (224 unless trained with --image-size), stamped as image_size
outputs: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]
Architecture selection
@@ -35,7 +36,7 @@ Usage
# or point at a specific checkpoint / output
python ./tools/pth_to_onnx.py \
- --ckpt ./instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth \
+ --ckpt ./instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth \
--output ./models/instanthmr.onnx
"""
import argparse
@@ -208,12 +209,14 @@ def main() -> None:
m = onnx.load(str(args.output))
for k, v in (("cliff_focal", str(bool(getattr(cfg, "cliff_focal", False))).lower()),
("bound_scales", str(bool(getattr(cfg, "bound_scales", False))).lower()),
- ("arch", arch)):
+ ("arch", arch),
+ ("image_size", str(int(cfg.image_size)))):
e = m.metadata_props.add()
e.key, e.value = k, v
onnx.save(m, str(args.output))
print(f" metadata: cliff_focal={getattr(cfg, 'cliff_focal', False)} "
- f"bound_scales={getattr(cfg, 'bound_scales', False)} arch={arch}")
+ f"bound_scales={getattr(cfg, 'bound_scales', False)} arch={arch} "
+ f"image_size={cfg.image_size}")
except Exception as e: # noqa: BLE001
print(f" [warning] could not stamp ONNX metadata: {e}")
diff --git a/tools/verify_input_size.py b/tools/verify_input_size.py
new file mode 100644
index 0000000..88fed6b
--- /dev/null
+++ b/tools/verify_input_size.py
@@ -0,0 +1,234 @@
+#!/usr/bin/env python3
+"""Verify that the network input size is one checkpoint setting, end to end.
+
+ python tools/verify_input_size.py --ref <reference train_distill_mhr_only.py> \
+ --data_root data/sam3d_gt_coco --emdb-root ~/Downloads/EMDB_root \
+ --dpw-seq ~/Downloads/sequenceFiles/sequenceFiles --dpw-img ~/Downloads/imageFiles
+
+--ref is the trainer as it was before the change (default: HEAD). Checks:
+
+1. baseline and v2 augmented samples are bit-identical to --ref at 224,
+ compared on alternating calls (the rig's first call differs by ~5e-10).
+2. A 288 checkpoint round-trips: config_from_checkpoint reads 288 from the
+ weights (the positional-embedding grid) with or without run_config.json,
+ and refuses a run_config.json that contradicts the weights.
+3. The 3DPW / EMDB crop builder emits the configured size, and at 224 is
+ byte-identical to --ref's.
+4. The ONNX export of a 288 checkpoint carries image_size=288 in its metadata,
+ and the inference package builds 288 crops from it on a real frame.
+5. The EMDB and 3DPW harnesses run a 288 checkpoint end to end.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import random
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import cv2
+import numpy as np
+import torch
+
+REPO = Path(__file__).resolve().parent.parent
+TRAIN = REPO / "instanthmr_distill_train"
+sys.path.insert(0, str(TRAIN))
+sys.path.insert(0, str(REPO))
+import train_distill_mhr_only as T # noqa: E402
+import val3dpw # noqa: E402
+
+G8H = TRAIN / "runs/g8h_s0/g8h_s0/best_student_model_v3.pth"
+fails: list[str] = []
+
+
+def check(ok: bool, msg: str) -> None:
+ print(f" {'OK ' if ok else 'BAD'} {msg}")
+ if not ok:
+ fails.append(msg)
+
+
+def load_module(path: Path, name: str):
+ spec = importlib.util.spec_from_file_location(name, path)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def dataset(mod, root: str, v2: bool, augment: bool = True, **size):
+ cfg = mod.DistillConfig()
+ cfg.cliff_focal = True
+ cfg.crop_centre_fix = True
+ if v2:
+ jz = load_module(TRAIN / "train_distill_jz.py", "jz_for_preset")
+ jz.apply_v2_preset(cfg)
+ return mod.SAM3DStudentDataset(
+ root, augment=augment, max_images=64, geom_p=cfg.geom_p,
+ geom_rot_deg=cfg.geom_rot_deg, geom_scale_range=cfg.geom_scale_range,
+ geom_trans=cfg.geom_trans, geom_flip_p=cfg.geom_flip_p,
+ geom_scale_max=cfg.geom_scale_max, cliff_follows_aug=cfg.cliff_follows_aug,
+ crop_centre_fix=cfg.crop_centre_fix, cliff_focal=cfg.cliff_focal,
+ occl_p=cfg.occl_p, occl_scale=cfg.occl_scale, jpeg_p=cfg.jpeg_p,
+ jpeg_quality=cfg.jpeg_quality, **size)
+
+
+def sample(ds, i: int, seed: int) -> dict:
+ random.seed(seed)
+ np.random.seed(seed)
+ torch.manual_seed(seed)
+ return ds[i]
+
+
+def same(a, b) -> bool:
+ if isinstance(a, torch.Tensor):
+ return torch.equal(a, b)
+ if isinstance(a, np.ndarray):
+ return np.array_equal(a, b, equal_nan=a.dtype.kind == "f")
+ if isinstance(a, dict):
+ return a.keys() == b.keys() and all(same(a[k], b[k]) for k in a)
+ return a == b
+
+
+def g8h_cfg():
+ st = torch.load(G8H, map_location="cpu", weights_only=False)["model_state_dict"]
+ cfg, _ = T.config_from_checkpoint(st, G8H)
+ return cfg
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--ref", type=Path, default=None)
+ p.add_argument("--data_root", default=str(REPO / "data/sam3d_gt_coco"))
+ p.add_argument("--emdb-root", type=Path, default=Path.home() / "Downloads/EMDB_root")
+ p.add_argument("--dpw-seq", type=Path,
+ default=Path.home() / "Downloads/sequenceFiles/sequenceFiles")
+ p.add_argument("--dpw-img", type=Path, default=Path.home() / "Downloads/imageFiles")
+ p.add_argument("--n", type=int, default=24)
+ args = p.parse_args()
+ tmp = Path(tempfile.mkdtemp(prefix="verify_input_size_"))
+
+ ref_path = args.ref
+ if ref_path is None:
+ ref_path = tmp / "train_distill_mhr_only_ref.py"
+ ref_path.write_text(subprocess.run(
+ ["git", "-C", str(REPO), "show",
+ "HEAD:instanthmr_distill_train/train_distill_mhr_only.py"],
+ check=True, capture_output=True, text=True).stdout)
+ R = load_module(ref_path, "train_distill_mhr_only_ref")
+
+ print("[1] augmented samples vs reference, 224")
+ for v2 in (False, True):
+ # the new code as both trainers now call it (image_size passed), the
+ # reference as it was called before (no such argument)
+ new = dataset(T, args.data_root, v2, image_size=224)
+ old = dataset(R, args.data_root, v2)
+ diff = 0
+ for k in range(args.n):
+ i = k % len(new)
+ a = sample(new, i, 1000 + k) # alternate: new, old, new, old ...
+ b = sample(old, i, 1000 + k)
+ diff += not same(a, b)
+ check(diff == 0, f"{'v2' if v2 else 'baseline'}: {diff} of {args.n} samples differ")
+
+ print("[1b] labels do not depend on the input size (no augmentation)")
+ d224 = dataset(T, args.data_root, False, augment=False, image_size=224)
+ d288 = dataset(T, args.data_root, False, augment=False, image_size=288)
+ bad = 0
+ for i in range(min(args.n, len(d224))):
+ a, b = d224[i], d288[i]
+ bad += not (a["image"].shape[-1] == 224 and b["image"].shape[-1] == 288
+ and torch.allclose(a["joints_2d"], b["joints_2d"], atol=1e-6)
+ and torch.equal(a["cliff_cond"], b["cliff_cond"]))
+ check(bad == 0, f"224 vs 288: {bad} samples with a different label or CLIFF vector")
+
+ print("[2] a 288 checkpoint round-trips through config_from_checkpoint")
+ cfg288 = g8h_cfg()
+ cfg288.image_size = 288
+ torch.manual_seed(0)
+ m288 = T.InstantHMRStudent(cfg288, pretrained=False).eval()
+ ck_dir = tmp / "r288"
+ ck_dir.mkdir()
+ ck = ck_dir / "best_student_model_v3.pth"
+ torch.save({"model_state_dict": m288.state_dict(), "epoch": 0}, ck)
+ st = m288.state_dict()
+ c, prov = T.config_from_checkpoint(st, ck)
+ check(c.image_size == 288, f"no run_config.json: image_size={c.image_size} "
+ f"[{prov.get('image_size')}]")
+ rc = {"image_size": 288, "cliff_focal": True, "backbone": cfg288.backbone}
+ (ck_dir / "run_config.json").write_text(json.dumps(rc))
+ c, prov = T.config_from_checkpoint(st, ck)
+ check(c.image_size == 288, f"with run_config.json: image_size={c.image_size}")
+ (ck_dir / "bad").mkdir()
+ (ck_dir / "bad" / "run_config.json").write_text(json.dumps({**rc, "image_size": 224}))
+ try:
+ T.config_from_checkpoint(st, ck_dir / "bad" / "x.pth")
+ check(False, "contradicting run_config.json was accepted")
+ except ValueError as e:
+ check(True, f"contradicting run_config.json refused: {e}")
+ c224 = g8h_cfg()
+ check(c224.image_size == 224, f"g8h_s0 reads back as {c224.image_size}")
+
+ print("[3] crop builder (3DPW / EMDB validation)")
+ frame = next(iter(sorted(args.dpw_img.rglob("*.jpg"))))
+ rgb = cv2.cvtColor(cv2.imread(str(frame)), cv2.COLOR_BGR2RGB)
+ h, w = rgb.shape[:2]
+ box = np.array([w * 0.3, h * 0.2, w * 0.6, h * 0.9])
+ Rv = load_module(args.ref.parent / "val3dpw.py", "val3dpw_ref") if args.ref else None
+ a, ca = val3dpw._preprocess(rgb, box, h, w, 1500.0)
+ if Rv is not None:
+ b, cb = Rv._preprocess(rgb, box, h, w, 1500.0)
+ check(np.array_equal(a, b) and np.array_equal(ca, cb), "224 crop byte-identical to reference")
+ else:
+ print(" SKIP 224 crop vs reference (pass --ref <dir>/train_distill_mhr_only.py with val3dpw.py beside it)")
+ check(a.shape == (3, 224, 224), f"default crop {a.shape}")
+ a288, _ = val3dpw._preprocess(rgb, box, h, w, 1500.0, size=288)
+ check(a288.shape == (3, 288, 288), f"size=288 crop {a288.shape}")
+
+ print("[4] ONNX export and the inference package")
+ onnx_path = tmp / "r288.onnx"
+ r = subprocess.run([sys.executable, str(REPO / "tools/pth_to_onnx.py"), "--ckpt", str(ck),
+ "--output", str(onnx_path)], capture_output=True, text=True)
+ check(r.returncode == 0, f"pth_to_onnx exit {r.returncode}"
+ + ("" if r.returncode == 0 else f": {r.stderr[-400:]}"))
+ if onnx_path.exists():
+ import onnx
+ meta = {q.key: q.value for q in onnx.load(str(onnx_path)).metadata_props}
+ check(meta.get("image_size") == "288", f"ONNX metadata image_size={meta.get('image_size')}")
+ from instanthmr.inference import InstantHMR
+ from instanthmr.mhr_renderer import build_mhr
+ net = InstantHMR(onnx_path, device="cpu",
+ mhr=build_mhr(script_path=str(REPO / "checkpoints/mhr_model.pt")))
+ check(net.input_size == 288, f"InstantHMR input_size={net.input_size}")
+ out = net.predict(rgb, box)
+ j2d = np.asarray(out.joints_2d)
+ check(np.isfinite(j2d).all() and j2d.shape[-1] == 2, f"predict on a real frame: joints_2d {j2d.shape}")
+
+ print("[5] harnesses run a 288 checkpoint")
+ for script, extra in (
+ ("benchmark/eval_emdb_ckpt.py",
+ ["--emdb-root", str(args.emdb_root), "--stride", "400",
+ "--adapter", str(REPO / "benchmark/results/adapter_smpl24_teacher.npz")]),
+ ("benchmark/eval_3dpw_ckpt.py",
+ ["--sequence-dir", str(args.dpw_seq), "--image-root", str(args.dpw_img),
+ "--split", "test", "--stride", "400"])):
+ out_json = tmp / (Path(script).stem + ".json")
+ r = subprocess.run([sys.executable, str(REPO / script), "--ckpt", str(ck),
+ *extra, "--num-workers", "2", "--out", str(out_json)],
+ capture_output=True, text=True, cwd=REPO)
+ check(r.returncode == 0 and out_json.exists(),
+ f"{Path(script).name}: exit {r.returncode}"
+ + ("" if r.returncode == 0 else f": {r.stderr[-400:]}"))
+
+ print(f"scratch: {tmp}")
+ if fails:
+ print(f"FAIL ({len(fails)})")
+ sys.exit(1)
+ print("PASS")
+
+
+if __name__ == "__main__":
+ main()