Python
feat: supervise the 70 landmarks with SAM 3D Body's exact mesh mapping behind --exact-landmarks
mohamdev4f626fdf6218
diff --git a/CLAUDE.md b/CLAUDE.mdindex 934b9db..ee83451 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -183,6 +183,23 @@ decode agree, so a wrong choice trains fine and only shows up as a rotation lossmeasuring distance in a permuted space. `instanthmr_distill_train/mhr_cont.py`uses the rig's and says so; do not "fix" it to match the teacher's line.+**The vertex subset now has two independent consumers, and mixing them breaks+a loss silently.** `--w-verts` skins 595 farthest-point samples; the teacher's+exact 70-landmark mapping (`--exact-landmarks`) needs the 468 vertices its first+70 rows reference. **Those two sets overlap in exactly 10 vertices** -- the 468+are not a subset of the 595, and raising `n_verts` does not reach them, because+farthest-point sampling optimises coverage, not landmark support. So+`_build_vertex_subset` skins their UNION (1053) and keeps `verts_loss_pos` /+`lm_vert_pos` to index the two consumers separately. If `loss_verts` ever+averages over the union instead of `verts_loss_pos`, it silently changes value+and stops being comparable with generations 6-8 -- no error, just a different+number. The union is free (181.2 ms for 595 alone vs 180.9 ms for 1053 at batch+64), because the cost is the forward-kinematics pass, not the vertex count.+`instanthmr_distill_train/assets/mhr_landmarks70.npz` is a new file and must be+rsynced. Scoring deliberately stays on the fitted readout: `val3dpw.py` and+`benchmark/eval_3dpw_ckpt.py` both call `get_native_keypoints`, so the metric+keeps measuring the deployed path and stays comparable across generations.+**`--cont-head` adds two files that must be rsynced.**`instanthmr_distill_train/mhr_cont.py` and`instanthmr_distill_train/assets/mhr_cont_head.npz` (30 KB, built bydiff --git a/docs/todo.md b/docs/todo.mdindex b61773e..fd8eb95 100644--- a/docs/todo.md+++ b/docs/todo.md@@ -417,7 +417,39 @@ latency has been measured for this change.# 5. Exact 70-landmark readout using a small vertex subset-**Status:** open, scoped to extending existing subset skinning. Preserve MHR's+**Status: implemented 2026-09-13** behind `--exact-landmarks`, default off, as+a training-loss change only. Asset built by `tools/build_landmark_assets.py`+(the first 70 rows of `head_pose.keypoint_mapping`, vertex columns restricted to+their 468-vertex support); `MHRForwardPass._build_vertex_subset` now takes the+union of those 468 with the vertex-loss subset; `loss_3d_native` and+`loss_reproj` read through `regress_keypoints_exact`. Gated by+`tools/verify_exact_landmarks.py`. Measurements in+`datasets_pipeline/jeanzay/STATUS.md`, "Generation 8b".++Verified: **2.4e-04 mm** against full 18,439-vertex skinning with the full+teacher mapping, on GT, on pose perturbed by N(0, 0.25) rad, and under resampled+identity; gradients agree to 2e-07 relative in `model_params` and 2e-09 absolute+in `shape_params`. The 468 and the 595 vertex-loss samples **overlap in only+10**, confirming the warning below. Union skinning is 1053 vertices and costs+nothing measurable (181.2 ms for 595 alone vs 180.9 ms for the union, batch 64,+forward+backward). `loss_verts` still averages over exactly its own 595 with+uniform weighting.++The disagreement this removes, measured against the teacher's readout on+identical GT geometry: the fitted `(70, 127)` matrix is **3.037 mm mean over the+30 non-finger landmarks** (1.306 mm over all 70, 18.05 mm worst), in the+2.93-3.75 mm band recorded below. The exact readout is 0.000 mm by construction.+It also gives the 70 landmarks a gradient into `shape_params` -- they move+0.821 mm mean when identity is resampled, against exactly 0.000 mm for the+fitted matrix, which is blind to identity by construction.++**Scoring is deliberately unchanged.** `val3dpw.py` and+`benchmark/eval_3dpw_ckpt.py` both call `get_native_keypoints`, i.e. the fitted+readout, and neither was touched. So the flag changes what the model is trained+to match, not what it is measured with, and the reported J14 PA-MPJPE stays+comparable with generations 6-8 and keeps measuring the deployed path.++Original scoping, kept for context: preserve MHR'sdirect pose-parameter prediction and rig/mesh separation. First evaluate as atraining-loss change; a deployed landmark-readout change is a separate decisionsubject to the smartphone speed constraint.diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.mdindex 11c20a4..f239791 100644--- a/instanthmr_distill_train/README.md+++ b/instanthmr_distill_train/README.md@@ -518,6 +518,73 @@ previous commit — verified: 1544 state_dict keys, same sha256, forward and all12 loss terms 0.000e+00 apart on 48 real augmented samples, under both`--losses legacy` and `--losses rebalanced`.+## `--exact-landmarks` — the teacher's own 70-landmark readout (added 2026-09-13)++**What it changes.** The student derives its 70 annotation keypoints from the+127 skeleton joints alone, through a fitted `(70, 127)` matrix. SAM 3D Body does+not — it reads them off the *skinned mesh* and the joints together, with a fixed+matrix from its checkpoint:++```+teacher: K = W_joint @ J + W_vertex @ V+student: K ≈ W_fitted @ J+```++**21 of the 70 landmarks have no joint contribution at all** — nose, elbows, toe+tips, acromion and other surface points — so a skeleton-only matrix cannot+represent them, and none of the 70 can move with body shape. Measured against+the teacher's readout on identical GT geometry:++| readout | all 70 | 30 non-finger | worst |+|---|---|---|---|+| fitted `(70, 127)` | 1.306 mm mean | **3.037 mm mean** | 18.05 mm |+| exact subset readout | 0.000 mm | 0.000 mm | 0.000 mm |++Those targets came out of the teacher's mapping, so this removes an operator+mismatch between the prediction and its own label. It affects `loss_3d_native`+and `loss_reproj` only.++**Why it is affordable.** The first 70 mapping rows reference just **468+distinct vertices**. `MHRForwardPass` already skins an arbitrary subset exactly+(not approximately — it rewrites the rig's three flattened influence tables and+leaves the op untouched), so this takes the **union** of those 468 with the+`--w-verts` subset. Do not assume the 595 already contain the 468: measured,+they overlap in **10**. Union is 1053 vertices, and the cost is unmeasurable —+181.2 ms for 595 alone against 180.9 ms for the union, batch 64, forward+backward.+The skinning is dominated by the forward-kinematics pass, not the vertex count.++`loss_verts` keeps averaging over exactly its own 595 farthest-point samples+with uniform weighting, via `verts_loss_pos`; averaging over the union instead+would silently reweight the term and break comparability with generations 6-8.++**It gives identity a second geometric gradient.** Resample `shape_params` and+the exact landmarks move **0.821 mm mean**; the fitted ones move exactly+**0.000 mm**. Before this, `--w-verts` was the only route by which the 45+identity coefficients saw any geometry at all.++**Verified** (`python tools/verify_exact_landmarks.py --data_root data`) against+full 18,439-vertex skinning with the full `(70, 18566)` teacher mapping:+**2.4e-04 mm** on real GT, on pose perturbed by N(0, 0.25) rad, and under+resampled identity; gradients agree to 2e-07 relative in `model_params` and+2e-09 absolute in `shape_params`. Every row of the mapping sums to exactly 1.0,+so the readout is translation-equivariant and commutes with `to_vision` — which+is why the mapping is applied after the unit/axis conversion.++**Scoring is unchanged, on purpose.** `val3dpw.py` and+`benchmark/eval_3dpw_ckpt.py` both go through `get_native_keypoints`, i.e. the+fitted readout, and neither was touched. The flag changes what the model is+trained to match, not what it is measured with, so the reported J14 PA-MPJPE+stays comparable with earlier generations and keeps measuring the path that+actually ships. **Deployment is unchanged**: the exported graph never emitted+these 3D keypoints — the consumer derives them — so no operator is added to the+phone runtime. Making the deployed readout exact is a separate decision with its+own latency measurement.++`instanthmr_distill_train/assets/mhr_landmarks70.npz` (167 KB) is **a new file+and must be rsynced**, or the job dies at startup. Default off; with it off+every loss term is bit-identical, verified across `legacy`, `rebalanced`,+`w_verts` on/off and `cont_head` on/off.+## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)**Do not retry this without reading the failure mode.** The flag is gone fromdiff --git a/instanthmr_distill_train/assets/mhr_landmarks70.npz b/instanthmr_distill_train/assets/mhr_landmarks70.npznew file mode 100644index 0000000..c89da78Binary files /dev/null and b/instanthmr_distill_train/assets/mhr_landmarks70.npz differdiff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.pyindex 1572751..412e904 100755--- a/instanthmr_distill_train/train_distill_jz.py+++ b/instanthmr_distill_train/train_distill_jz.py@@ -535,8 +535,10 @@ def train(args, cfg, rank, local_rank, world):out_dir.mkdir(parents=True, exist_ok=True)W = np.load(cfg.kp_regressor_path)- mhr_module = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W,- n_verts=cfg.n_verts if cfg.w_verts > 0 else 0)+ mhr_module = T.MHRForwardPass(+ cfg.mhr_model_path, device, kp_regressor=W,+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,+ landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)criterion = T.DistillationLoss(cfg, mhr_module)train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)@@ -694,6 +696,7 @@ def train(args, cfg, rank, local_rank, world):"mix": args.mix, "steps_per_epoch": steps_per_epoch,"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,"cont_head": cfg.cont_head, "root_rot_loss": cfg.root_rot_loss,+ "exact_landmarks": cfg.exact_landmarks,"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,"jpeg_p": cfg.jpeg_p, "cam_loss": cfg.cam_loss,"losses": args.losses, "w_shape": cfg.w_shape,@@ -1148,6 +1151,12 @@ def parse_args():"farthest-point subset of the rig's own 18,439-vertex ""mesh (default 595, lod6's count). Costs 0.6 ms per ""batch-64 step against 16.7 ms for the full mesh.")+ p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",+ help="Supervise the 70 keypoints with the teacher's exact "+ "mesh+joint mapping over the 468 vertices it references, "+ "instead of the fitted skeleton-only matrix. Changes "+ "loss_3d_native and loss_reproj; export is unchanged. "+ "Needs assets/mhr_landmarks70.npz.")p.add_argument("--cont-head", dest="cont_head", action="store_true",help="Reproduce the teacher's regression pathway: the head ""emits 447 numbers in its continuous space (6D root, "@@ -1198,6 +1207,7 @@ def build_cfg(args):if args.cont_head:cfg.cont_head = Truecfg.root_rot_loss = True+ if args.exact_landmarks: cfg.exact_landmarks = Trueif args.w_verts is not None: cfg.w_verts = args.w_vertsif args.n_verts is not None: cfg.n_verts = args.n_vertsif args.w_simcc is not None: cfg.w_simcc = args.w_simcc@@ -1308,6 +1318,7 @@ def main():if args.losses != "legacy" else ""))log(f"head {'continuous (teacher pathway, 447 dims)' if cfg.cont_head else 'linear 204-vector'}"+ (f" | root loss = rotation matrix" if cfg.root_rot_loss else ""))+ log(f"landmarks {'teacher exact mapping (468 mesh vertices + 50 joints)' if cfg.exact_landmarks else 'fitted (70, 127) skeleton regressor'}")log(f"surface w_verts={cfg.w_verts}"+ (f" over {cfg.n_verts} mesh vertices" if cfg.w_verts > 0 else " (off)")+ f" | crop_centre_fix={cfg.crop_centre_fix}")diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex 4e5b58d..8f2dc3a 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -322,6 +322,18 @@ class DistillConfig:# rotation, and 13.59% of corpus root angles sit beyond 2.8 rad.root_rot_loss: bool = Falsew_root_rot: float = 1.0++ # --- the teacher's exact 70-landmark readout (--exact-landmarks) --------+ # Off by default. On, loss_3d_native and loss_reproj read the 70 keypoints+ # out of the skinned mesh with the teacher's own fixed mapping+ # (K = Wj @ J + Wv @ V) instead of the fitted (70, 127) skeleton-only+ # matrix. 21 of the 70 are pure mesh-surface points with no joint+ # contribution at all, so the fitted matrix cannot represent them; measured+ # 2.93-3.75 mm mean disagreement over the 30 non-finger landmarks on+ # identical GT geometry. Training-only: the exported graph never emitted+ # these keypoints, so deployment still uses the approximate readout.+ exact_landmarks: bool = False+ landmark_assets_path: str = str(PROJECT_ROOT / "assets/mhr_landmarks70.npz")use_amp: bool = Trueema_decay: float = 0.9998early_stop_patience: int = 150@@ -456,6 +468,12 @@ def config_from_checkpoint(state, ckpt_path=None, **overrides):cfg.root_rot_loss = cfg.cont_headprov["cont_head"] = "checkpoint weights"+ # --exact-landmarks leaves NO trace in the weights: it changes which+ # operator two training losses use, not the graph. So it can only come from+ # run_config.json, and its absence there is indistinguishable from "off".+ # That is acceptable because nothing downstream of a checkpoint depends on+ # it -- export and inference use the fitted readout either way.+run_cfg = {}if ckpt_path is not None:rc = Path(ckpt_path).resolve().parent / "run_config.json"@@ -1256,7 +1274,8 @@ class MHRForwardPass:per training step and makes the mesh LOD irrelevant to training cost."""- def __init__(self, mhr_path, device, kp_regressor=None, n_verts: int = 0):+ def __init__(self, mhr_path, device, kp_regressor=None, n_verts: int = 0,+ landmark_assets: str | None = None):self.device = deviceself.mhr = torch.jit.load(mhr_path, map_location=device).eval()for p in self.mhr.parameters():@@ -1266,13 +1285,33 @@ class MHRForwardPass:W = torch.as_tensor(kp_regressor, dtype=torch.float32, device=device)assert W.shape == (70, 127), f"expected a (70, 127) regressor, got {tuple(W.shape)}"self.kp_regressor = W++ # The teacher's exact landmark readout, when asked for: K = Wj @ J + Wv @ V+ # over the 468 mesh vertices its first 70 mapping rows actually touch.+ self.lm_W_vert = self.lm_W_joint = None+ lm_verts = None+ if landmark_assets is not None:+ a = np.load(landmark_assets)+ lm_verts = torch.as_tensor(a["vert_idx"], dtype=torch.int64, device=device)+ self.lm_W_vert = torch.as_tensor(a["W_vert"], dtype=torch.float32, device=device)+ self.lm_W_joint = torch.as_tensor(a["W_joint"], dtype=torch.float32, device=device)+self.vert_idx = None- if n_verts:- self._build_vertex_subset(int(n_verts))+ self.verts_loss_pos = None # where the w_verts samples sit in the subset+ self.lm_vert_pos = None # where the landmark vertices sit, in W_vert order+ if n_verts or lm_verts is not None:+ self._build_vertex_subset(int(n_verts), lm_verts)- def _build_vertex_subset(self, n: int):+ def _build_vertex_subset(self, n: int, extra: torch.Tensor | None = None):"""A farthest-point subset of the rig mesh, plus a skinning op for it.+ `extra` is an additional index set to include -- the 468 vertices the+ teacher's landmark mapping references. The subset becomes the UNION, so+ the two consumers are independent: `loss_verts` still averages over+ exactly the `n` farthest-point samples it always did, with the same+ uniform weighting, via `verts_loss_pos`. The 468 are not a subset of the+ 595 and a larger `n` does not contain them -- they have to be named.+The rig's `linear_blend_skinning` asserts it is handed all 18,439vertices, so the subset is installed by rewriting its three flattenedinfluence tables -- (vertex, bone, weight) triples -- to reference only@@ -1289,24 +1328,37 @@ class MHRForwardPass:# Farthest-point sampling in float64: the mesh spans ~180 cm and# neighbouring vertices are sub-millimetre apart, so float32 ties would# make the traversal order depend on accumulation noise.- P = rest.double()- idx = [0]- d = (P - P[0]).norm(dim=-1)- for _ in range(n - 1):- i = int(d.argmax())- idx.append(i)- d = torch.minimum(d, (P - P[i]).norm(dim=-1))- sub = torch.tensor(sorted(idx), device=self.device)+ fps = None+ if n:+ P = rest.double()+ idx = [0]+ d = (P - P[0]).norm(dim=-1)+ for _ in range(n - 1):+ i = int(d.argmax())+ idx.append(i)+ d = torch.minimum(d, (P - P[i]).norm(dim=-1))+ fps = torch.tensor(sorted(idx), device=self.device)++ parts = [t for t in (fps, extra.to(self.device) if extra is not None else None)+ if t is not None]+ assert parts, "_build_vertex_subset called with neither n_verts nor extra"+ sub = torch.unique(torch.cat(parts))+ n_sub = int(sub.numel())+ # Positions, not vertex ids: both consumers index the skinned output.+ # searchsorted is exact here because `sub` is sorted and contains both.+ self.verts_loss_pos = torch.searchsorted(sub, fps) if fps is not None else None+ self.lm_vert_pos = (torch.searchsorted(sub, extra.to(self.device))+ if extra is not None else None)lbs = copy.deepcopy(ct.linear_blend_skinning)vi = ct.linear_blend_skinning.vert_indices_flattenedkeep = torch.isin(vi, sub)remap = torch.full((int(vi.max()) + 1,), -1, dtype=torch.int64, device=self.device)- remap[sub] = torch.arange(n, device=self.device)+ remap[sub] = torch.arange(n_sub, device=self.device)lbs.vert_indices_flattened = remap[vi[keep]]lbs.skin_indices_flattened = ct.linear_blend_skinning.skin_indices_flattened[keep]lbs.skin_weights_flattened = ct.linear_blend_skinning.skin_weights_flattened[keep]- lbs.num_vertices = n+ lbs.num_vertices = n_subself.vert_idx, self.vert_lbs = sub, lbsdef get_joints(self, model_params, shape_params):@@ -1348,6 +1400,27 @@ class MHRForwardPass:"no keypoint regressor loaded — run --fit-regressor or pass kp_regressor_path")return torch.einsum('kj,bjc->bkc', self.kp_regressor, joints_vision)+ def regress_keypoints_exact(self, joints_vision, verts_vision):+ """The teacher's own landmark readout: `Wj @ J + Wv @ V`.++ `joints_vision` is (B, 127, 3) and `verts_vision` the (B, n_sub, 3)+ skinned subset, both already through `to_vision`. That is legitimate+ rather than a shortcut: every row of the mapping sums to exactly 1.0, so+ the readout is translation-equivariant, and `to_vision` is a diagonal+ scale-and-flip, so the two commute. Verified against the full+ (70, 18566) mapping in `tools/verify_exact_landmarks.py`.++ 21 of the 70 landmarks have NO joint contribution -- nose, elbows, toe+ tips, acromion and other surface points -- which is why a fitted+ skeleton-only matrix cannot reproduce them and why these carry a+ gradient into `shape_params`, unlike `regress_keypoints`.+ """+ assert self.lm_W_vert is not None, (+ "MHRForwardPass was built without landmark_assets")+ v = verts_vision.index_select(1, self.lm_vert_pos)+ return (torch.einsum('kj,bjc->bkc', self.lm_W_joint, joints_vision)+ + torch.einsum('kv,bvc->bkc', self.lm_W_vert, v))+def get_native_keypoints(self, model_params, shape_params):"""(B, 70, 3) native keypoints in the annotation frame."""j = self.get_joints(model_params, shape_params)[..., :3]@@ -1551,7 +1624,12 @@ class DistillationLoss(nn.Module):losses['loss_cam'] = self.mse(preds["cam_trans"], targets["cam_trans"]) * self.cfg.w_camwant_verts = float(getattr(self.cfg, "w_verts", 0.0)) > 0.0- if want_verts:+ exact_lm = bool(getattr(self.cfg, "exact_landmarks", False))+ # The prediction needs skinned vertices for either consumer. The TARGET+ # needs them only for loss_verts: the landmark losses compare against+ # the stored annotation, which the teacher already produced with this+ # mapping, so --exact-landmarks adds no target-side cost at all.+ if want_verts or exact_lm:pred_mhr_joints, pred_verts = self.mhr_module.get_joints_and_vertices(preds["mhr_params"], preds["shape_params"])else:@@ -1590,7 +1668,15 @@ class DistillationLoss(nn.Module):# logged value reads as mean PVE / 1000. Masked to m_noflip for the# same reason loss_mhr_joints is: under a horizontal flip the target# PARAMETERS are not mirrored, only the 2D and 3D labels are.- e_verts = (pred_verts - tgt_verts.detach()).norm(dim=-1).mean(1) / 100.0+ # Restricted to the farthest-point samples this term has always+ # used. --exact-landmarks widens the skinned subset to the union+ # with the mapping's 468 vertices; averaging over the union instead+ # would silently reweight the term and break comparability with+ # generations 6-8.+ vp = self.mhr_module.verts_loss_pos+ pv = pred_verts if vp is None else pred_verts.index_select(1, vp)+ tv = tgt_verts if vp is None else tgt_verts.index_select(1, vp)+ e_verts = (pv - tv.detach()).norm(dim=-1).mean(1) / 100.0losses['loss_verts'] = (self._mmean(e_verts, m_noflip)* self.cfg.w_verts * self.fk_scale)@@ -1598,7 +1684,16 @@ class DistillationLoss(nn.Module):# 127 skeleton joints, so they are differentiable w.r.t. mhr_params and# carry no extra forward pass (pred_mhr_joints is already computed).pred_mhr_vision = self.mhr_module.to_vision(pred_mhr_joints)- pred_kp3d = self.mhr_module.regress_keypoints(pred_mhr_vision)+ if exact_lm:+ # The teacher's readout, off the skinned mesh, instead of the fitted+ # skeleton-only matrix. Feeds loss_3d_native and loss_reproj below,+ # which is the whole change: `tgt_3d` already came out of this same+ # mapping, so this removes an operator mismatch between the+ # prediction and its own target.+ pred_kp3d = self.mhr_module.regress_keypoints_exact(+ pred_mhr_vision, self.mhr_module.to_vision(pred_verts))+ else:+ pred_kp3d = self.mhr_module.regress_keypoints(pred_mhr_vision)pred_2d = preds["joints_2d"]tgt_2d = targets["joints_2d"]@@ -2730,6 +2825,12 @@ def parse_args():p.add_argument("--no-export", dest="no_export", action="store_true",help="Skip the ONNX export / quantization step after training.")p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")+ p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",+ help="Read the 70 keypoints out of the skinned mesh with the "+ "teacher's own fixed mapping, instead of the fitted "+ "(70, 127) skeleton-only matrix. Affects loss_3d_native "+ "and loss_reproj only; the exported graph is unchanged. "+ "Needs assets/mhr_landmarks70.npz.")p.add_argument("--cont-head", dest="cont_head", action="store_true",help="Regress the teacher's continuous space (6D root, 260-dim ""body, 28 scale PCA coefficients, 2x54 hand blocks; 447 "@@ -2809,6 +2910,8 @@ def main():if getattr(args, "cont_head", False):cfg.cont_head = Truecfg.root_rot_loss = True+ if getattr(args, "exact_landmarks", False):+ cfg.exact_landmarks = Trueif args.losses == "rebalanced": apply_rebalanced_losses(cfg)# An explicit --w_keypoints3d still wins over the preset.if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3d@@ -2871,7 +2974,10 @@ def main():cfg.kp_regressor_path)train_loader, val_loader, full_dataset = build_dataloaders(cfg)- mhr_module = MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W)+ mhr_module = MHRForwardPass(+ cfg.mhr_model_path, device, kp_regressor=W,+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,+ landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)if args.self_test:run_self_tests()diff --git a/tools/build_landmark_assets.py b/tools/build_landmark_assets.pynew file mode 100644index 0000000..9d5236d--- /dev/null+++ b/tools/build_landmark_assets.py@@ -0,0 +1,100 @@+#!/usr/bin/env python3+"""Build `instanthmr_distill_train/assets/mhr_landmarks70.npz` for `--exact-landmarks`.++ python tools/build_landmark_assets.py++The student derives its 70 annotation keypoints from the 127 skeleton joints+alone, through a fitted `(70, 127)` matrix. SAM 3D Body does not: it reads them+out of the SKINNED MESH and the joints together, with a fixed matrix that ships+in its checkpoint --++ K = W_joint @ J + W_vertex @ V++-- so 21 of the 70 (nose, elbows, toe tips, acromion and other surface points)+have no joint contribution at all and move with body shape, which a+skeleton-only fit cannot represent. Measured elsewhere at 2.93-3.75 mm mean over+the 30 non-finger landmarks on identical GT geometry.++This extracts the first 70 rows of `head_pose.keypoint_mapping` (308 x 18566:+18,439 mesh vertices then 127 joints) and, critically, restricts the vertex+block to the **468 vertices it actually references**, remapped to a compact+index. That is what makes the exact readout affordable: `MHRForwardPass` already+skins an arbitrary vertex subset, so the union of these 468 with the vertex+loss's own subset costs one skinning pass over ~1000 vertices instead of 18,439.++`head_pose` and `head_pose_hand` carry byte-identical copies of the mapping.+Every row sums to exactly 1.0, so the readout is translation-equivariant and+commutes with `MHRForwardPass.to_vision` (a diagonal scale-and-flip) -- which is+why the mapping may be applied after the unit/axis conversion rather than+before.+"""++from __future__ import annotations++import argparse+import glob+from pathlib import Path++import numpy as np+import torch++N_VERTS_FULL = 18439+N_JOINTS = 127+N_LANDMARKS = 70+++def main():+ ap = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--teacher-ckpt", type=Path, default=None)+ ap.add_argument("--out", type=Path,+ default=Path("instanthmr_distill_train/assets/mhr_landmarks70.npz"))+ args = ap.parse_args()++ ckpt = args.teacher_ckpt+ if ckpt is None:+ hits = glob.glob(str(Path.home() / ".cache/huggingface/hub/"+ "models--facebook--sam-3d-body-dinov3/snapshots/*/model.ckpt"))+ if not hits:+ raise SystemExit("teacher model.ckpt not found — pass --teacher-ckpt")+ ckpt = Path(hits[0])+ sd = torch.load(ckpt, map_location="cpu", weights_only=False)+ sd = sd.get("state_dict", sd)+ K = sd["head_pose.keypoint_mapping"]+ assert torch.equal(K, sd["head_pose_hand.keypoint_mapping"]), \+ "the two heads' mappings differ — decide which one is canonical first"+ assert K.shape == (308, N_VERTS_FULL + N_JOINTS), tuple(K.shape)+ print(f"teacher mapping from {ckpt}: {tuple(K.shape)}")++ K70 = K[:N_LANDMARKS].double()+ Wv_full, Wj = K70[:, :N_VERTS_FULL], K70[:, N_VERTS_FULL:]++ vert_idx = (Wv_full.abs().sum(0) > 0).nonzero().flatten()+ Wv = Wv_full[:, vert_idx] # (70, 468), compact columns+ print(f" {len(vert_idx)} distinct vertices referenced, "+ f"{int((Wj.abs().sum(0) > 0).sum())} distinct joints")++ rs = K70.sum(1)+ assert (rs - 1.0).abs().max() < 1e-5, f"rows do not sum to 1 (max {rs.max()})"+ print(f" row sums 1.0 to {float((rs - 1.0).abs().max()):.2e} — "+ f"readout is translation-equivariant")++ # Dropping the zero columns must be exactly lossless, not nearly.+ dropped = Wv_full.sum() - Wv.sum()+ assert abs(float(dropped)) < 1e-9, dropped+ nv = (Wv.abs() > 0).sum(1)+ nj = (Wj.abs() > 0).sum(1)+ print(f" {int(((nv > 0) & (nj == 0)).sum())} landmarks are pure mesh surface, "+ f"{int(((nj > 0) & (nv == 0)).sum())} pure joint, "+ f"{int(((nv > 0) & (nj > 0)).sum())} mixed")++ args.out.parent.mkdir(parents=True, exist_ok=True)+ np.savez(args.out,+ vert_idx=vert_idx.numpy().astype(np.int64),+ W_vert=Wv.numpy().astype(np.float32),+ W_joint=Wj.numpy().astype(np.float32))+ print(f"wrote {args.out}")+++if __name__ == "__main__":+ main()diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.pyindex 2ce5382..7906ea2 100644--- a/tools/ddp_smoke.py+++ b/tools/ddp_smoke.py@@ -64,6 +64,9 @@ def main() -> int:p.add_argument("--bound-scales", dest="bound_scales", action="store_true", default=True)p.add_argument("--cliff-focal", dest="cliff_focal", action="store_true", default=True)p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true", default=True)+ p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",+ help="Use the teacher's exact landmark readout, which widens "+ "the skinned vertex subset.")p.add_argument("--cont-head", dest="cont_head", action="store_true",help="Build the teacher's continuous regression head instead ""of the linear 204-vector head. A changed head shape is "@@ -91,6 +94,8 @@ def main() -> int:if args.cont_head:cfg.cont_head = Truecfg.root_rot_loss = True+ if args.exact_landmarks:+ cfg.exact_landmarks = Trueif args.backbone:cfg.backbone = args.backbone@@ -101,15 +106,17 @@ def main() -> int:batch = {k: (v.to(T.device) if torch.is_tensor(v) else v)for k, v in next(iter(loader)).items()}- mhr = T.MHRForwardPass(cfg.mhr_model_path, T.device,- kp_regressor=np.load(cfg.kp_regressor_path),- n_verts=cfg.n_verts if cfg.w_verts > 0 else 0)+ mhr = T.MHRForwardPass(+ cfg.mhr_model_path, T.device, kp_regressor=np.load(cfg.kp_regressor_path),+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,+ landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)T.mhr_module = mhrmodel = T.InstantHMRStudent(cfg, pretrained=args.pretrained).to(T.device)n = sum(q.numel() for q in model.parameters())print(f"{cfg.backbone}: {n/1e6:.2f} M parameters, "f"head={'continuous' if cfg.cont_head else 'linear'}, "+ f"landmarks={'exact' if cfg.exact_landmarks else 'fitted'}, "f"w_verts={cfg.w_verts}, batch {args.batch_size}")ddp = torch.nn.parallel.DistributedDataParallel(model) # find_unused_parameters=Falsediff --git a/tools/verify_exact_landmarks.py b/tools/verify_exact_landmarks.pynew file mode 100644index 0000000..4ebfbd9--- /dev/null+++ b/tools/verify_exact_landmarks.py@@ -0,0 +1,209 @@+#!/usr/bin/env python3+"""Gate `--exact-landmarks` before it costs cluster time.++ python tools/verify_exact_landmarks.py --data_root data++The claim under test is narrow and checkable: reading the 70 landmarks off a+468-vertex SUBSET of the skinned mesh gives the same answer, and the same+gradients, as reading them off all 18,439 vertices with the teacher's full+mapping. If that holds, the subset is an exact restriction rather than an+approximation, exactly as `--w-verts` is.++Checks:++1. **Landmark equivalence** against the full-mesh reference, on real GT, on+ perturbed pose parameters, and under changed identity coefficients -- the+ last because 21 of the 70 landmarks are pure mesh-surface points that move+ with shape, which is the whole reason for doing this.+2. **Gradient equivalence** w.r.t. both `model_params` and `shape_params`.+3. **Shape sensitivity**, reported rather than asserted: how far the fitted+ skeleton-only readout is from the teacher's, and how much of that the+ exact one recovers.+4. **Subset integrity**: the union contains all 468 mapping vertices and all+ 595 vertex-loss samples, and `verts_loss_pos` recovers the latter exactly.+5. **Cost**: skinned-vertex count and wall time at a real batch size.++Exit status is 0 only if every check passes.+"""++from __future__ import annotations++import argparse+import glob+import sys+import time+from pathlib import Path++import numpy as np+import torch++sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "instanthmr_distill_train"))+import train_distill_mhr_only as T # noqa: E402++OK, BAD = " ok ", "*FAIL*"+_fails = 0+++def check(cond, label, detail=""):+ global _fails+ if not cond:+ _fails += 1+ print(f"[{OK if cond else BAD}] {label}{(' ' + detail) if detail else ''}")+++def full_mapping(asset, device):+ """The (70, 18566) reference. From the teacher when present; otherwise the+ asset's 468 columns scattered back, which `build_landmark_assets.py` has+ already asserted is the same matrix."""+ hits = glob.glob(str(Path.home() / ".cache/huggingface/hub/"+ "models--facebook--sam-3d-body-dinov3/snapshots/*/model.ckpt"))+ if hits:+ sd = torch.load(hits[0], map_location="cpu", weights_only=False)+ sd = sd.get("state_dict", sd)+ print(f"reference mapping: teacher checkpoint {Path(hits[0]).parent.name[:12]}")+ return sd["head_pose.keypoint_mapping"][:70].to(device)+ K = torch.zeros(70, 18439 + 127, device=device)+ K[:, torch.as_tensor(asset["vert_idx"], device=device)] = \+ torch.as_tensor(asset["W_vert"], device=device)+ K[:, 18439:] = torch.as_tensor(asset["W_joint"], device=device)+ print("reference mapping: rebuilt from the asset (teacher ckpt not found)")+ return K+++def reference_landmarks(mhr, K_full, model_params, shape_params):+ """Full 18,439-vertex skinning, then the full mapping, in the vision frame."""+ ct = mhr.mhr.character_torch+ cat = torch.cat([model_params, shape_params], dim=1)+ skel = ct.joint_parameters_to_skeleton_state(+ ct.model_parameters_to_joint_parameters(cat))+ verts = ct.linear_blend_skinning(skel, ct.blend_shape(shape_params))+ vj = torch.cat([mhr.to_vision(verts), mhr.to_vision(skel[..., :3])], dim=1)+ return torch.einsum('kn,bnc->bkc', K_full, vj)+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("--data_root", default="data")+ ap.add_argument("--n", type=int, default=64)+ ap.add_argument("--batch", type=int, default=64, help="batch for the cost measurement")+ ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")+ args = ap.parse_args()+ dev = args.device++ cfg = T.DistillConfig()+ T.cfg = cfg+ T.apply_rebalanced_losses(cfg)+ cfg.w_verts = 0.35+ cfg.exact_landmarks = True++ asset = np.load(cfg.landmark_assets_path)+ mhr = T.MHRForwardPass(cfg.mhr_model_path, dev,+ kp_regressor=np.load(cfg.kp_regressor_path),+ n_verts=cfg.n_verts, landmark_assets=cfg.landmark_assets_path)+ K_full = full_mapping(asset, dev)++ files = sorted(glob.glob(f"{args.data_root}/*/annotations/*.npz"))[:args.n]+ if not files:+ files = sorted(glob.glob(f"{args.data_root}/annotations/*.npz"))[:args.n]+ gt = torch.from_numpy(np.stack([np.load(f)["mhr_model_params"] for f in files])).float().to(dev)+ sh = torch.from_numpy(np.stack([np.load(f)["shape_params"] for f in files])).float().to(dev)+ print(f"{len(gt)} real GT annotations from {args.data_root}, device {dev}\n")++ # --- 4. subset integrity (first: everything else depends on it) ---------+ sub = mhr.vert_idx+ lm_v = torch.as_tensor(asset["vert_idx"], device=dev)+ check(bool(torch.isin(lm_v, sub).all()), "all 468 mapping vertices are in the subset")+ check(int(sub.numel()) == int(torch.unique(torch.cat([sub])).numel()),+ "subset has no duplicates")+ check(bool((sub[mhr.lm_vert_pos] == lm_v).all()),+ "lm_vert_pos indexes them in the mapping's column order")+ check(mhr.verts_loss_pos.numel() == cfg.n_verts,+ "verts_loss_pos still names exactly the vertex-loss samples",+ f"{mhr.verts_loss_pos.numel()} of {int(sub.numel())} skinned")+ overlap = int(torch.isin(lm_v, sub[mhr.verts_loss_pos]).sum())+ print(f" (union is {int(sub.numel())} vertices: {cfg.n_verts} farthest-point "+ f"+ 468 mapping, overlapping in {overlap})")++ # --- 1 & 2. equivalence, on GT / perturbed pose / changed shape ---------+ cases = [("real GT", gt, sh)]+ torch.manual_seed(0)+ p = gt.clone(); p[:, 6:136] += torch.randn_like(p[:, 6:136]) * 0.25+ cases.append(("pose perturbed by N(0, 0.25) rad", p, sh))+ cases.append(("identity coefficients resampled", gt, sh + torch.randn_like(sh) * 1.0))++ for tag, mp, sp in cases:+ mp_a = mp.clone().requires_grad_(True); sp_a = sp.clone().requires_grad_(True)+ j, v = mhr.get_joints_and_vertices(mp_a, sp_a)+ got = mhr.regress_keypoints_exact(mhr.to_vision(j), mhr.to_vision(v))++ mp_b = mp.clone().requires_grad_(True); sp_b = sp.clone().requires_grad_(True)+ ref = reference_landmarks(mhr, K_full, mp_b, sp_b)++ d = (got - ref).abs().max() * 1000.0 # vision frame is metres+ check(d < 1e-3, f"landmarks vs full mesh [{tag}]", f"max {d:.3e} mm")++ torch.manual_seed(1)+ w = torch.randn_like(got)+ (got * w).sum().backward(); (ref * w).sum().backward()+ gm = (mp_a.grad - mp_b.grad).abs().max()+ gs = (sp_a.grad - sp_b.grad).abs().max()+ rel_m = gm / mp_b.grad.abs().max().clamp_min(1e-12)+ check(rel_m < 1e-4, f"d/d(model_params) vs full mesh [{tag}]",+ f"max {gm:.3e} (rel {rel_m:.1e})")+ check(gs / sp_b.grad.abs().max().clamp_min(1e-12) < 1e-4,+ f"d/d(shape_params) vs full mesh [{tag}]", f"max {gs:.3e}")++ # --- 3. what the change is actually worth -------------------------------+ with torch.no_grad():+ j, v = mhr.get_joints_and_vertices(gt, sh)+ jv = mhr.to_vision(j)+ exact = mhr.regress_keypoints_exact(jv, mhr.to_vision(v))+ fitted = mhr.regress_keypoints(jv)+ ref = reference_landmarks(mhr, K_full, gt, sh)+ nonfinger = torch.from_numpy(+ np.setdiff1d(np.arange(70), T.FINGER_KP)).to(dev)+ print(f"\ndisagreement with the teacher's readout on identical GT geometry:")+ for name, kp in (("fitted (70,127) skeleton matrix", fitted), ("exact subset readout", exact)):+ e = (kp - ref).norm(dim=-1) * 1000.0+ print(f" {name:34s} all 70: {e.mean():7.3f} mm mean / {e.max():8.3f} max | "+ f"{len(nonfinger)} non-finger: {e[:, nonfinger].mean():6.3f} mm")++ # shape sensitivity: the property a skeleton-only readout cannot have+ with torch.no_grad():+ sh2 = sh + torch.randn_like(sh) * 1.0+ j2, v2 = mhr.get_joints_and_vertices(gt, sh2)+ e_ex = (mhr.regress_keypoints_exact(mhr.to_vision(j2), mhr.to_vision(v2)) - exact)+ e_fi = (mhr.regress_keypoints(mhr.to_vision(j2)) - fitted)+ print(f" landmark motion when identity is resampled:")+ print(f" exact {e_ex.norm(dim=-1).mean()*1000:7.3f} mm mean <- carries a shape gradient")+ print(f" fitted {e_fi.norm(dim=-1).mean()*1000:7.3f} mm mean <- blind to identity by construction")++ # --- 5. cost ------------------------------------------------------------+ print(f"\ncost at batch {args.batch}:")+ for tag, nv, la in (("w_verts only (595 skinned)", cfg.n_verts, None),+ ("exact landmarks only (468)", 0, cfg.landmark_assets_path),+ ("both (union)", cfg.n_verts, cfg.landmark_assets_path)):+ m = T.MHRForwardPass(cfg.mhr_model_path, dev,+ kp_regressor=np.load(cfg.kp_regressor_path),+ n_verts=nv, landmark_assets=la)+ mp = gt[:1].repeat(args.batch, 1).clone().requires_grad_(True)+ sp = sh[:1].repeat(args.batch, 1).clone().requires_grad_(True)+ for i in range(12):+ if i == 2:+ torch.cuda.synchronize() if dev == "cuda" else None+ t0 = time.perf_counter()+ jj, vv = m.get_joints_and_vertices(mp, sp)+ (jj.sum() + vv.sum()).backward()+ mp.grad = sp.grad = None+ torch.cuda.synchronize() if dev == "cuda" else None+ ms = (time.perf_counter() - t0) / 10 * 1000+ print(f" {tag:30s} {int(m.vert_idx.numel()):5d} vertices {ms:6.2f} ms fwd+bwd")++ print()+ print(f"{_fails} check(s) FAILED" if _fails else "all checks passed")+ return 1 if _fails else 0+++if __name__ == "__main__":+ raise SystemExit(main())