Skip to content
Python

feat: supervise the 70 landmarks with SAM 3D Body's exact mesh mapping behind --exact-landmarks

mohamdev4f626fdf6218
diff --git a/CLAUDE.md b/CLAUDE.md
index 934b9db..ee83451 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -183,6 +183,23 @@ decode agree, so a wrong choice trains fine and only shows up as a rotation loss
measuring distance in a permuted space. `instanthmr_distill_train/mhr_cont.py`
uses the rig's and says so; do not "fix" it to match the teacher's line.
+**The vertex subset now has two independent consumers, and mixing them breaks
+a loss silently.** `--w-verts` skins 595 farthest-point samples; the teacher's
+exact 70-landmark mapping (`--exact-landmarks`) needs the 468 vertices its first
+70 rows reference. **Those two sets overlap in exactly 10 vertices** -- the 468
+are not a subset of the 595, and raising `n_verts` does not reach them, because
+farthest-point sampling optimises coverage, not landmark support. So
+`_build_vertex_subset` skins their UNION (1053) and keeps `verts_loss_pos` /
+`lm_vert_pos` to index the two consumers separately. If `loss_verts` ever
+averages over the union instead of `verts_loss_pos`, it silently changes value
+and stops being comparable with generations 6-8 -- no error, just a different
+number. The union is free (181.2 ms for 595 alone vs 180.9 ms for 1053 at batch
+64), because the cost is the forward-kinematics pass, not the vertex count.
+`instanthmr_distill_train/assets/mhr_landmarks70.npz` is a new file and must be
+rsynced. Scoring deliberately stays on the fitted readout: `val3dpw.py` and
+`benchmark/eval_3dpw_ckpt.py` both call `get_native_keypoints`, so the metric
+keeps measuring the deployed path and stays comparable across generations.
+
**`--cont-head` adds two files that must be rsynced.**
`instanthmr_distill_train/mhr_cont.py` and
`instanthmr_distill_train/assets/mhr_cont_head.npz` (30 KB, built by
diff --git a/docs/todo.md b/docs/todo.md
index b61773e..fd8eb95 100644
--- a/docs/todo.md
+++ b/docs/todo.md
@@ -417,7 +417,39 @@ latency has been measured for this change.
# 5. Exact 70-landmark readout using a small vertex subset
-**Status:** open, scoped to extending existing subset skinning. Preserve MHR's
+**Status: implemented 2026-09-13** behind `--exact-landmarks`, default off, as
+a training-loss change only. Asset built by `tools/build_landmark_assets.py`
+(the first 70 rows of `head_pose.keypoint_mapping`, vertex columns restricted to
+their 468-vertex support); `MHRForwardPass._build_vertex_subset` now takes the
+union of those 468 with the vertex-loss subset; `loss_3d_native` and
+`loss_reproj` read through `regress_keypoints_exact`. Gated by
+`tools/verify_exact_landmarks.py`. Measurements in
+`datasets_pipeline/jeanzay/STATUS.md`, "Generation 8b".
+
+Verified: **2.4e-04 mm** against full 18,439-vertex skinning with the full
+teacher mapping, on GT, on pose perturbed by N(0, 0.25) rad, and under resampled
+identity; gradients agree to 2e-07 relative in `model_params` and 2e-09 absolute
+in `shape_params`. The 468 and the 595 vertex-loss samples **overlap in only
+10**, confirming the warning below. Union skinning is 1053 vertices and costs
+nothing measurable (181.2 ms for 595 alone vs 180.9 ms for the union, batch 64,
+forward+backward). `loss_verts` still averages over exactly its own 595 with
+uniform weighting.
+
+The disagreement this removes, measured against the teacher's readout on
+identical GT geometry: the fitted `(70, 127)` matrix is **3.037 mm mean over the
+30 non-finger landmarks** (1.306 mm over all 70, 18.05 mm worst), in the
+2.93-3.75 mm band recorded below. The exact readout is 0.000 mm by construction.
+It also gives the 70 landmarks a gradient into `shape_params` -- they move
+0.821 mm mean when identity is resampled, against exactly 0.000 mm for the
+fitted matrix, which is blind to identity by construction.
+
+**Scoring is deliberately unchanged.** `val3dpw.py` and
+`benchmark/eval_3dpw_ckpt.py` both call `get_native_keypoints`, i.e. the fitted
+readout, and neither was touched. So the flag changes what the model is trained
+to match, not what it is measured with, and the reported J14 PA-MPJPE stays
+comparable with generations 6-8 and keeps measuring the deployed path.
+
+Original scoping, kept for context: preserve MHR's
direct pose-parameter prediction and rig/mesh separation. First evaluate as a
training-loss change; a deployed landmark-readout change is a separate decision
subject to the smartphone speed constraint.
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index 11c20a4..f239791 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -518,6 +518,73 @@ previous commit — verified: 1544 state_dict keys, same sha256, forward and all
12 loss terms 0.000e+00 apart on 48 real augmented samples, under both
`--losses legacy` and `--losses rebalanced`.
+## `--exact-landmarks` — the teacher's own 70-landmark readout (added 2026-09-13)
+
+**What it changes.** The student derives its 70 annotation keypoints from the
+127 skeleton joints alone, through a fitted `(70, 127)` matrix. SAM 3D Body does
+not — it reads them off the *skinned mesh* and the joints together, with a fixed
+matrix from its checkpoint:
+
+```
+teacher: K = W_joint @ J + W_vertex @ V
+student: K ≈ W_fitted @ J
+```
+
+**21 of the 70 landmarks have no joint contribution at all** — nose, elbows, toe
+tips, acromion and other surface points — so a skeleton-only matrix cannot
+represent them, and none of the 70 can move with body shape. Measured against
+the teacher's readout on identical GT geometry:
+
+| readout | all 70 | 30 non-finger | worst |
+|---|---|---|---|
+| fitted `(70, 127)` | 1.306 mm mean | **3.037 mm mean** | 18.05 mm |
+| exact subset readout | 0.000 mm | 0.000 mm | 0.000 mm |
+
+Those targets came out of the teacher's mapping, so this removes an operator
+mismatch between the prediction and its own label. It affects `loss_3d_native`
+and `loss_reproj` only.
+
+**Why it is affordable.** The first 70 mapping rows reference just **468
+distinct vertices**. `MHRForwardPass` already skins an arbitrary subset exactly
+(not approximately — it rewrites the rig's three flattened influence tables and
+leaves the op untouched), so this takes the **union** of those 468 with the
+`--w-verts` subset. Do not assume the 595 already contain the 468: measured,
+they overlap in **10**. Union is 1053 vertices, and the cost is unmeasurable —
+181.2 ms for 595 alone against 180.9 ms for the union, batch 64, forward+backward.
+The skinning is dominated by the forward-kinematics pass, not the vertex count.
+
+`loss_verts` keeps averaging over exactly its own 595 farthest-point samples
+with uniform weighting, via `verts_loss_pos`; averaging over the union instead
+would silently reweight the term and break comparability with generations 6-8.
+
+**It gives identity a second geometric gradient.** Resample `shape_params` and
+the exact landmarks move **0.821 mm mean**; the fitted ones move exactly
+**0.000 mm**. Before this, `--w-verts` was the only route by which the 45
+identity coefficients saw any geometry at all.
+
+**Verified** (`python tools/verify_exact_landmarks.py --data_root data`) against
+full 18,439-vertex skinning with the full `(70, 18566)` teacher mapping:
+**2.4e-04 mm** on real GT, on pose perturbed by N(0, 0.25) rad, and under
+resampled identity; gradients agree to 2e-07 relative in `model_params` and
+2e-09 absolute in `shape_params`. Every row of the mapping sums to exactly 1.0,
+so the readout is translation-equivariant and commutes with `to_vision` — which
+is why the mapping is applied after the unit/axis conversion.
+
+**Scoring is unchanged, on purpose.** `val3dpw.py` and
+`benchmark/eval_3dpw_ckpt.py` both go through `get_native_keypoints`, i.e. the
+fitted readout, and neither was touched. The flag changes what the model is
+trained to match, not what it is measured with, so the reported J14 PA-MPJPE
+stays comparable with earlier generations and keeps measuring the path that
+actually ships. **Deployment is unchanged**: the exported graph never emitted
+these 3D keypoints — the consumer derives them — so no operator is added to the
+phone runtime. Making the deployed readout exact is a separate decision with its
+own latency measurement.
+
+`instanthmr_distill_train/assets/mhr_landmarks70.npz` (167 KB) is **a new file
+and must be rsynced**, or the job dies at startup. Default off; with it off
+every loss term is bit-identical, verified across `legacy`, `rebalanced`,
+`w_verts` on/off and `cont_head` on/off.
+
## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)
**Do not retry this without reading the failure mode.** The flag is gone from
diff --git a/instanthmr_distill_train/assets/mhr_landmarks70.npz b/instanthmr_distill_train/assets/mhr_landmarks70.npz
new file mode 100644
index 0000000..c89da78
Binary files /dev/null and b/instanthmr_distill_train/assets/mhr_landmarks70.npz differ
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 1572751..412e904 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -535,8 +535,10 @@ def train(args, cfg, rank, local_rank, world):
out_dir.mkdir(parents=True, exist_ok=True)
W = np.load(cfg.kp_regressor_path)
- mhr_module = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W,
- n_verts=cfg.n_verts if cfg.w_verts > 0 else 0)
+ mhr_module = T.MHRForwardPass(
+ cfg.mhr_model_path, device, kp_regressor=W,
+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,
+ landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)
criterion = T.DistillationLoss(cfg, mhr_module)
train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)
@@ -694,6 +696,7 @@ def train(args, cfg, rank, local_rank, world):
"mix": args.mix, "steps_per_epoch": steps_per_epoch,
"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,
"cont_head": cfg.cont_head, "root_rot_loss": cfg.root_rot_loss,
+ "exact_landmarks": cfg.exact_landmarks,
"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,
"jpeg_p": cfg.jpeg_p, "cam_loss": cfg.cam_loss,
"losses": args.losses, "w_shape": cfg.w_shape,
@@ -1148,6 +1151,12 @@ def parse_args():
"farthest-point subset of the rig's own 18,439-vertex "
"mesh (default 595, lod6's count). Costs 0.6 ms per "
"batch-64 step against 16.7 ms for the full mesh.")
+ p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",
+ help="Supervise the 70 keypoints with the teacher's exact "
+ "mesh+joint mapping over the 468 vertices it references, "
+ "instead of the fitted skeleton-only matrix. Changes "
+ "loss_3d_native and loss_reproj; export is unchanged. "
+ "Needs assets/mhr_landmarks70.npz.")
p.add_argument("--cont-head", dest="cont_head", action="store_true",
help="Reproduce the teacher's regression pathway: the head "
"emits 447 numbers in its continuous space (6D root, "
@@ -1198,6 +1207,7 @@ def build_cfg(args):
if args.cont_head:
cfg.cont_head = True
cfg.root_rot_loss = True
+ if args.exact_landmarks: cfg.exact_landmarks = True
if args.w_verts is not None: cfg.w_verts = args.w_verts
if args.n_verts is not None: cfg.n_verts = args.n_verts
if args.w_simcc is not None: cfg.w_simcc = args.w_simcc
@@ -1308,6 +1318,7 @@ def main():
if args.losses != "legacy" else ""))
log(f"head {'continuous (teacher pathway, 447 dims)' if cfg.cont_head else 'linear 204-vector'}"
+ (f" | root loss = rotation matrix" if cfg.root_rot_loss else ""))
+ log(f"landmarks {'teacher exact mapping (468 mesh vertices + 50 joints)' if cfg.exact_landmarks else 'fitted (70, 127) skeleton regressor'}")
log(f"surface w_verts={cfg.w_verts}"
+ (f" over {cfg.n_verts} mesh vertices" if cfg.w_verts > 0 else " (off)")
+ f" | crop_centre_fix={cfg.crop_centre_fix}")
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 4e5b58d..8f2dc3a 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -322,6 +322,18 @@ class DistillConfig:
# rotation, and 13.59% of corpus root angles sit beyond 2.8 rad.
root_rot_loss: bool = False
w_root_rot: float = 1.0
+
+ # --- the teacher's exact 70-landmark readout (--exact-landmarks) --------
+ # Off by default. On, loss_3d_native and loss_reproj read the 70 keypoints
+ # out of the skinned mesh with the teacher's own fixed mapping
+ # (K = Wj @ J + Wv @ V) instead of the fitted (70, 127) skeleton-only
+ # matrix. 21 of the 70 are pure mesh-surface points with no joint
+ # contribution at all, so the fitted matrix cannot represent them; measured
+ # 2.93-3.75 mm mean disagreement over the 30 non-finger landmarks on
+ # identical GT geometry. Training-only: the exported graph never emitted
+ # these keypoints, so deployment still uses the approximate readout.
+ exact_landmarks: bool = False
+ landmark_assets_path: str = str(PROJECT_ROOT / "assets/mhr_landmarks70.npz")
use_amp: bool = True
ema_decay: float = 0.9998
early_stop_patience: int = 150
@@ -456,6 +468,12 @@ def config_from_checkpoint(state, ckpt_path=None, **overrides):
cfg.root_rot_loss = cfg.cont_head
prov["cont_head"] = "checkpoint weights"
+ # --exact-landmarks leaves NO trace in the weights: it changes which
+ # operator two training losses use, not the graph. So it can only come from
+ # run_config.json, and its absence there is indistinguishable from "off".
+ # That is acceptable because nothing downstream of a checkpoint depends on
+ # it -- export and inference use the fitted readout either way.
+
run_cfg = {}
if ckpt_path is not None:
rc = Path(ckpt_path).resolve().parent / "run_config.json"
@@ -1256,7 +1274,8 @@ class MHRForwardPass:
per training step and makes the mesh LOD irrelevant to training cost.
"""
- def __init__(self, mhr_path, device, kp_regressor=None, n_verts: int = 0):
+ def __init__(self, mhr_path, device, kp_regressor=None, n_verts: int = 0,
+ landmark_assets: str | None = None):
self.device = device
self.mhr = torch.jit.load(mhr_path, map_location=device).eval()
for p in self.mhr.parameters():
@@ -1266,13 +1285,33 @@ class MHRForwardPass:
W = torch.as_tensor(kp_regressor, dtype=torch.float32, device=device)
assert W.shape == (70, 127), f"expected a (70, 127) regressor, got {tuple(W.shape)}"
self.kp_regressor = W
+
+ # The teacher's exact landmark readout, when asked for: K = Wj @ J + Wv @ V
+ # over the 468 mesh vertices its first 70 mapping rows actually touch.
+ self.lm_W_vert = self.lm_W_joint = None
+ lm_verts = None
+ if landmark_assets is not None:
+ a = np.load(landmark_assets)
+ lm_verts = torch.as_tensor(a["vert_idx"], dtype=torch.int64, device=device)
+ self.lm_W_vert = torch.as_tensor(a["W_vert"], dtype=torch.float32, device=device)
+ self.lm_W_joint = torch.as_tensor(a["W_joint"], dtype=torch.float32, device=device)
+
self.vert_idx = None
- if n_verts:
- self._build_vertex_subset(int(n_verts))
+ self.verts_loss_pos = None # where the w_verts samples sit in the subset
+ self.lm_vert_pos = None # where the landmark vertices sit, in W_vert order
+ if n_verts or lm_verts is not None:
+ self._build_vertex_subset(int(n_verts), lm_verts)
- def _build_vertex_subset(self, n: int):
+ def _build_vertex_subset(self, n: int, extra: torch.Tensor | None = None):
"""A farthest-point subset of the rig mesh, plus a skinning op for it.
+ `extra` is an additional index set to include -- the 468 vertices the
+ teacher's landmark mapping references. The subset becomes the UNION, so
+ the two consumers are independent: `loss_verts` still averages over
+ exactly the `n` farthest-point samples it always did, with the same
+ uniform weighting, via `verts_loss_pos`. The 468 are not a subset of the
+ 595 and a larger `n` does not contain them -- they have to be named.
+
The rig's `linear_blend_skinning` asserts it is handed all 18,439
vertices, so the subset is installed by rewriting its three flattened
influence tables -- (vertex, bone, weight) triples -- to reference only
@@ -1289,24 +1328,37 @@ class MHRForwardPass:
# Farthest-point sampling in float64: the mesh spans ~180 cm and
# neighbouring vertices are sub-millimetre apart, so float32 ties would
# make the traversal order depend on accumulation noise.
- P = rest.double()
- idx = [0]
- d = (P - P[0]).norm(dim=-1)
- for _ in range(n - 1):
- i = int(d.argmax())
- idx.append(i)
- d = torch.minimum(d, (P - P[i]).norm(dim=-1))
- sub = torch.tensor(sorted(idx), device=self.device)
+ fps = None
+ if n:
+ P = rest.double()
+ idx = [0]
+ d = (P - P[0]).norm(dim=-1)
+ for _ in range(n - 1):
+ i = int(d.argmax())
+ idx.append(i)
+ d = torch.minimum(d, (P - P[i]).norm(dim=-1))
+ fps = torch.tensor(sorted(idx), device=self.device)
+
+ parts = [t for t in (fps, extra.to(self.device) if extra is not None else None)
+ if t is not None]
+ assert parts, "_build_vertex_subset called with neither n_verts nor extra"
+ sub = torch.unique(torch.cat(parts))
+ n_sub = int(sub.numel())
+ # Positions, not vertex ids: both consumers index the skinned output.
+ # searchsorted is exact here because `sub` is sorted and contains both.
+ self.verts_loss_pos = torch.searchsorted(sub, fps) if fps is not None else None
+ self.lm_vert_pos = (torch.searchsorted(sub, extra.to(self.device))
+ if extra is not None else None)
lbs = copy.deepcopy(ct.linear_blend_skinning)
vi = ct.linear_blend_skinning.vert_indices_flattened
keep = torch.isin(vi, sub)
remap = torch.full((int(vi.max()) + 1,), -1, dtype=torch.int64, device=self.device)
- remap[sub] = torch.arange(n, device=self.device)
+ remap[sub] = torch.arange(n_sub, device=self.device)
lbs.vert_indices_flattened = remap[vi[keep]]
lbs.skin_indices_flattened = ct.linear_blend_skinning.skin_indices_flattened[keep]
lbs.skin_weights_flattened = ct.linear_blend_skinning.skin_weights_flattened[keep]
- lbs.num_vertices = n
+ lbs.num_vertices = n_sub
self.vert_idx, self.vert_lbs = sub, lbs
def get_joints(self, model_params, shape_params):
@@ -1348,6 +1400,27 @@ class MHRForwardPass:
"no keypoint regressor loaded — run --fit-regressor or pass kp_regressor_path")
return torch.einsum('kj,bjc->bkc', self.kp_regressor, joints_vision)
+ def regress_keypoints_exact(self, joints_vision, verts_vision):
+ """The teacher's own landmark readout: `Wj @ J + Wv @ V`.
+
+ `joints_vision` is (B, 127, 3) and `verts_vision` the (B, n_sub, 3)
+ skinned subset, both already through `to_vision`. That is legitimate
+ rather than a shortcut: every row of the mapping sums to exactly 1.0, so
+ the readout is translation-equivariant, and `to_vision` is a diagonal
+ scale-and-flip, so the two commute. Verified against the full
+ (70, 18566) mapping in `tools/verify_exact_landmarks.py`.
+
+ 21 of the 70 landmarks have NO joint contribution -- nose, elbows, toe
+ tips, acromion and other surface points -- which is why a fitted
+ skeleton-only matrix cannot reproduce them and why these carry a
+ gradient into `shape_params`, unlike `regress_keypoints`.
+ """
+ assert self.lm_W_vert is not None, (
+ "MHRForwardPass was built without landmark_assets")
+ v = verts_vision.index_select(1, self.lm_vert_pos)
+ return (torch.einsum('kj,bjc->bkc', self.lm_W_joint, joints_vision)
+ + torch.einsum('kv,bvc->bkc', self.lm_W_vert, v))
+
def get_native_keypoints(self, model_params, shape_params):
"""(B, 70, 3) native keypoints in the annotation frame."""
j = self.get_joints(model_params, shape_params)[..., :3]
@@ -1551,7 +1624,12 @@ class DistillationLoss(nn.Module):
losses['loss_cam'] = self.mse(preds["cam_trans"], targets["cam_trans"]) * self.cfg.w_cam
want_verts = float(getattr(self.cfg, "w_verts", 0.0)) > 0.0
- if want_verts:
+ exact_lm = bool(getattr(self.cfg, "exact_landmarks", False))
+ # The prediction needs skinned vertices for either consumer. The TARGET
+ # needs them only for loss_verts: the landmark losses compare against
+ # the stored annotation, which the teacher already produced with this
+ # mapping, so --exact-landmarks adds no target-side cost at all.
+ if want_verts or exact_lm:
pred_mhr_joints, pred_verts = self.mhr_module.get_joints_and_vertices(
preds["mhr_params"], preds["shape_params"])
else:
@@ -1590,7 +1668,15 @@ class DistillationLoss(nn.Module):
# logged value reads as mean PVE / 1000. Masked to m_noflip for the
# same reason loss_mhr_joints is: under a horizontal flip the target
# PARAMETERS are not mirrored, only the 2D and 3D labels are.
- e_verts = (pred_verts - tgt_verts.detach()).norm(dim=-1).mean(1) / 100.0
+ # Restricted to the farthest-point samples this term has always
+ # used. --exact-landmarks widens the skinned subset to the union
+ # with the mapping's 468 vertices; averaging over the union instead
+ # would silently reweight the term and break comparability with
+ # generations 6-8.
+ vp = self.mhr_module.verts_loss_pos
+ pv = pred_verts if vp is None else pred_verts.index_select(1, vp)
+ tv = tgt_verts if vp is None else tgt_verts.index_select(1, vp)
+ e_verts = (pv - tv.detach()).norm(dim=-1).mean(1) / 100.0
losses['loss_verts'] = (self._mmean(e_verts, m_noflip)
* self.cfg.w_verts * self.fk_scale)
@@ -1598,7 +1684,16 @@ class DistillationLoss(nn.Module):
# 127 skeleton joints, so they are differentiable w.r.t. mhr_params and
# carry no extra forward pass (pred_mhr_joints is already computed).
pred_mhr_vision = self.mhr_module.to_vision(pred_mhr_joints)
- pred_kp3d = self.mhr_module.regress_keypoints(pred_mhr_vision)
+ if exact_lm:
+ # The teacher's readout, off the skinned mesh, instead of the fitted
+ # skeleton-only matrix. Feeds loss_3d_native and loss_reproj below,
+ # which is the whole change: `tgt_3d` already came out of this same
+ # mapping, so this removes an operator mismatch between the
+ # prediction and its own target.
+ pred_kp3d = self.mhr_module.regress_keypoints_exact(
+ pred_mhr_vision, self.mhr_module.to_vision(pred_verts))
+ else:
+ pred_kp3d = self.mhr_module.regress_keypoints(pred_mhr_vision)
pred_2d = preds["joints_2d"]
tgt_2d = targets["joints_2d"]
@@ -2730,6 +2825,12 @@ def parse_args():
p.add_argument("--no-export", dest="no_export", action="store_true",
help="Skip the ONNX export / quantization step after training.")
p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")
+ p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",
+ help="Read the 70 keypoints out of the skinned mesh with the "
+ "teacher's own fixed mapping, instead of the fitted "
+ "(70, 127) skeleton-only matrix. Affects loss_3d_native "
+ "and loss_reproj only; the exported graph is unchanged. "
+ "Needs assets/mhr_landmarks70.npz.")
p.add_argument("--cont-head", dest="cont_head", action="store_true",
help="Regress the teacher's continuous space (6D root, 260-dim "
"body, 28 scale PCA coefficients, 2x54 hand blocks; 447 "
@@ -2809,6 +2910,8 @@ def main():
if getattr(args, "cont_head", False):
cfg.cont_head = True
cfg.root_rot_loss = True
+ if getattr(args, "exact_landmarks", False):
+ cfg.exact_landmarks = True
if args.losses == "rebalanced": apply_rebalanced_losses(cfg)
# An explicit --w_keypoints3d still wins over the preset.
if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3d
@@ -2871,7 +2974,10 @@ def main():
cfg.kp_regressor_path)
train_loader, val_loader, full_dataset = build_dataloaders(cfg)
- mhr_module = MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W)
+ mhr_module = MHRForwardPass(
+ cfg.mhr_model_path, device, kp_regressor=W,
+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,
+ landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)
if args.self_test:
run_self_tests()
diff --git a/tools/build_landmark_assets.py b/tools/build_landmark_assets.py
new file mode 100644
index 0000000..9d5236d
--- /dev/null
+++ b/tools/build_landmark_assets.py
@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""Build `instanthmr_distill_train/assets/mhr_landmarks70.npz` for `--exact-landmarks`.
+
+ python tools/build_landmark_assets.py
+
+The student derives its 70 annotation keypoints from the 127 skeleton joints
+alone, through a fitted `(70, 127)` matrix. SAM 3D Body does not: it reads them
+out of the SKINNED MESH and the joints together, with a fixed matrix that ships
+in its checkpoint --
+
+ K = W_joint @ J + W_vertex @ V
+
+-- so 21 of the 70 (nose, elbows, toe tips, acromion and other surface points)
+have no joint contribution at all and move with body shape, which a
+skeleton-only fit cannot represent. Measured elsewhere at 2.93-3.75 mm mean over
+the 30 non-finger landmarks on identical GT geometry.
+
+This extracts the first 70 rows of `head_pose.keypoint_mapping` (308 x 18566:
+18,439 mesh vertices then 127 joints) and, critically, restricts the vertex
+block to the **468 vertices it actually references**, remapped to a compact
+index. That is what makes the exact readout affordable: `MHRForwardPass` already
+skins an arbitrary vertex subset, so the union of these 468 with the vertex
+loss's own subset costs one skinning pass over ~1000 vertices instead of 18,439.
+
+`head_pose` and `head_pose_hand` carry byte-identical copies of the mapping.
+Every row sums to exactly 1.0, so the readout is translation-equivariant and
+commutes with `MHRForwardPass.to_vision` (a diagonal scale-and-flip) -- which is
+why the mapping may be applied after the unit/axis conversion rather than
+before.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+from pathlib import Path
+
+import numpy as np
+import torch
+
+N_VERTS_FULL = 18439
+N_JOINTS = 127
+N_LANDMARKS = 70
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--teacher-ckpt", type=Path, default=None)
+ ap.add_argument("--out", type=Path,
+ default=Path("instanthmr_distill_train/assets/mhr_landmarks70.npz"))
+ args = ap.parse_args()
+
+ ckpt = args.teacher_ckpt
+ if ckpt is None:
+ hits = glob.glob(str(Path.home() / ".cache/huggingface/hub/"
+ "models--facebook--sam-3d-body-dinov3/snapshots/*/model.ckpt"))
+ if not hits:
+ raise SystemExit("teacher model.ckpt not found — pass --teacher-ckpt")
+ ckpt = Path(hits[0])
+ sd = torch.load(ckpt, map_location="cpu", weights_only=False)
+ sd = sd.get("state_dict", sd)
+ K = sd["head_pose.keypoint_mapping"]
+ assert torch.equal(K, sd["head_pose_hand.keypoint_mapping"]), \
+ "the two heads' mappings differ — decide which one is canonical first"
+ assert K.shape == (308, N_VERTS_FULL + N_JOINTS), tuple(K.shape)
+ print(f"teacher mapping from {ckpt}: {tuple(K.shape)}")
+
+ K70 = K[:N_LANDMARKS].double()
+ Wv_full, Wj = K70[:, :N_VERTS_FULL], K70[:, N_VERTS_FULL:]
+
+ vert_idx = (Wv_full.abs().sum(0) > 0).nonzero().flatten()
+ Wv = Wv_full[:, vert_idx] # (70, 468), compact columns
+ print(f" {len(vert_idx)} distinct vertices referenced, "
+ f"{int((Wj.abs().sum(0) > 0).sum())} distinct joints")
+
+ rs = K70.sum(1)
+ assert (rs - 1.0).abs().max() < 1e-5, f"rows do not sum to 1 (max {rs.max()})"
+ print(f" row sums 1.0 to {float((rs - 1.0).abs().max()):.2e} — "
+ f"readout is translation-equivariant")
+
+ # Dropping the zero columns must be exactly lossless, not nearly.
+ dropped = Wv_full.sum() - Wv.sum()
+ assert abs(float(dropped)) < 1e-9, dropped
+ nv = (Wv.abs() > 0).sum(1)
+ nj = (Wj.abs() > 0).sum(1)
+ print(f" {int(((nv > 0) & (nj == 0)).sum())} landmarks are pure mesh surface, "
+ f"{int(((nj > 0) & (nv == 0)).sum())} pure joint, "
+ f"{int(((nv > 0) & (nj > 0)).sum())} mixed")
+
+ args.out.parent.mkdir(parents=True, exist_ok=True)
+ np.savez(args.out,
+ vert_idx=vert_idx.numpy().astype(np.int64),
+ W_vert=Wv.numpy().astype(np.float32),
+ W_joint=Wj.numpy().astype(np.float32))
+ print(f"wrote {args.out}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.py
index 2ce5382..7906ea2 100644
--- a/tools/ddp_smoke.py
+++ b/tools/ddp_smoke.py
@@ -64,6 +64,9 @@ def main() -> int:
p.add_argument("--bound-scales", dest="bound_scales", action="store_true", default=True)
p.add_argument("--cliff-focal", dest="cliff_focal", action="store_true", default=True)
p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true", default=True)
+ p.add_argument("--exact-landmarks", dest="exact_landmarks", action="store_true",
+ help="Use the teacher's exact landmark readout, which widens "
+ "the skinned vertex subset.")
p.add_argument("--cont-head", dest="cont_head", action="store_true",
help="Build the teacher's continuous regression head instead "
"of the linear 204-vector head. A changed head shape is "
@@ -91,6 +94,8 @@ def main() -> int:
if args.cont_head:
cfg.cont_head = True
cfg.root_rot_loss = True
+ if args.exact_landmarks:
+ cfg.exact_landmarks = True
if args.backbone:
cfg.backbone = args.backbone
@@ -101,15 +106,17 @@ def main() -> int:
batch = {k: (v.to(T.device) if torch.is_tensor(v) else v)
for k, v in next(iter(loader)).items()}
- mhr = T.MHRForwardPass(cfg.mhr_model_path, T.device,
- kp_regressor=np.load(cfg.kp_regressor_path),
- n_verts=cfg.n_verts if cfg.w_verts > 0 else 0)
+ mhr = T.MHRForwardPass(
+ cfg.mhr_model_path, T.device, kp_regressor=np.load(cfg.kp_regressor_path),
+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0,
+ landmark_assets=cfg.landmark_assets_path if cfg.exact_landmarks else None)
T.mhr_module = mhr
model = T.InstantHMRStudent(cfg, pretrained=args.pretrained).to(T.device)
n = sum(q.numel() for q in model.parameters())
print(f"{cfg.backbone}: {n/1e6:.2f} M parameters, "
f"head={'continuous' if cfg.cont_head else 'linear'}, "
+ f"landmarks={'exact' if cfg.exact_landmarks else 'fitted'}, "
f"w_verts={cfg.w_verts}, batch {args.batch_size}")
ddp = torch.nn.parallel.DistributedDataParallel(model) # find_unused_parameters=False
diff --git a/tools/verify_exact_landmarks.py b/tools/verify_exact_landmarks.py
new file mode 100644
index 0000000..4ebfbd9
--- /dev/null
+++ b/tools/verify_exact_landmarks.py
@@ -0,0 +1,209 @@
+#!/usr/bin/env python3
+"""Gate `--exact-landmarks` before it costs cluster time.
+
+ python tools/verify_exact_landmarks.py --data_root data
+
+The claim under test is narrow and checkable: reading the 70 landmarks off a
+468-vertex SUBSET of the skinned mesh gives the same answer, and the same
+gradients, as reading them off all 18,439 vertices with the teacher's full
+mapping. If that holds, the subset is an exact restriction rather than an
+approximation, exactly as `--w-verts` is.
+
+Checks:
+
+1. **Landmark equivalence** against the full-mesh reference, on real GT, on
+ perturbed pose parameters, and under changed identity coefficients -- the
+ last because 21 of the 70 landmarks are pure mesh-surface points that move
+ with shape, which is the whole reason for doing this.
+2. **Gradient equivalence** w.r.t. both `model_params` and `shape_params`.
+3. **Shape sensitivity**, reported rather than asserted: how far the fitted
+ skeleton-only readout is from the teacher's, and how much of that the
+ exact one recovers.
+4. **Subset integrity**: the union contains all 468 mapping vertices and all
+ 595 vertex-loss samples, and `verts_loss_pos` recovers the latter exactly.
+5. **Cost**: skinned-vertex count and wall time at a real batch size.
+
+Exit status is 0 only if every check passes.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import torch
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "instanthmr_distill_train"))
+import train_distill_mhr_only as T # noqa: E402
+
+OK, BAD = " ok ", "*FAIL*"
+_fails = 0
+
+
+def check(cond, label, detail=""):
+ global _fails
+ if not cond:
+ _fails += 1
+ print(f"[{OK if cond else BAD}] {label}{(' ' + detail) if detail else ''}")
+
+
+def full_mapping(asset, device):
+ """The (70, 18566) reference. From the teacher when present; otherwise the
+ asset's 468 columns scattered back, which `build_landmark_assets.py` has
+ already asserted is the same matrix."""
+ hits = glob.glob(str(Path.home() / ".cache/huggingface/hub/"
+ "models--facebook--sam-3d-body-dinov3/snapshots/*/model.ckpt"))
+ if hits:
+ sd = torch.load(hits[0], map_location="cpu", weights_only=False)
+ sd = sd.get("state_dict", sd)
+ print(f"reference mapping: teacher checkpoint {Path(hits[0]).parent.name[:12]}")
+ return sd["head_pose.keypoint_mapping"][:70].to(device)
+ K = torch.zeros(70, 18439 + 127, device=device)
+ K[:, torch.as_tensor(asset["vert_idx"], device=device)] = \
+ torch.as_tensor(asset["W_vert"], device=device)
+ K[:, 18439:] = torch.as_tensor(asset["W_joint"], device=device)
+ print("reference mapping: rebuilt from the asset (teacher ckpt not found)")
+ return K
+
+
+def reference_landmarks(mhr, K_full, model_params, shape_params):
+ """Full 18,439-vertex skinning, then the full mapping, in the vision frame."""
+ ct = mhr.mhr.character_torch
+ cat = torch.cat([model_params, shape_params], dim=1)
+ skel = ct.joint_parameters_to_skeleton_state(
+ ct.model_parameters_to_joint_parameters(cat))
+ verts = ct.linear_blend_skinning(skel, ct.blend_shape(shape_params))
+ vj = torch.cat([mhr.to_vision(verts), mhr.to_vision(skel[..., :3])], dim=1)
+ return torch.einsum('kn,bnc->bkc', K_full, vj)
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--data_root", default="data")
+ ap.add_argument("--n", type=int, default=64)
+ ap.add_argument("--batch", type=int, default=64, help="batch for the cost measurement")
+ ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+ args = ap.parse_args()
+ dev = args.device
+
+ cfg = T.DistillConfig()
+ T.cfg = cfg
+ T.apply_rebalanced_losses(cfg)
+ cfg.w_verts = 0.35
+ cfg.exact_landmarks = True
+
+ asset = np.load(cfg.landmark_assets_path)
+ mhr = T.MHRForwardPass(cfg.mhr_model_path, dev,
+ kp_regressor=np.load(cfg.kp_regressor_path),
+ n_verts=cfg.n_verts, landmark_assets=cfg.landmark_assets_path)
+ K_full = full_mapping(asset, dev)
+
+ files = sorted(glob.glob(f"{args.data_root}/*/annotations/*.npz"))[:args.n]
+ if not files:
+ files = sorted(glob.glob(f"{args.data_root}/annotations/*.npz"))[:args.n]
+ gt = torch.from_numpy(np.stack([np.load(f)["mhr_model_params"] for f in files])).float().to(dev)
+ sh = torch.from_numpy(np.stack([np.load(f)["shape_params"] for f in files])).float().to(dev)
+ print(f"{len(gt)} real GT annotations from {args.data_root}, device {dev}\n")
+
+ # --- 4. subset integrity (first: everything else depends on it) ---------
+ sub = mhr.vert_idx
+ lm_v = torch.as_tensor(asset["vert_idx"], device=dev)
+ check(bool(torch.isin(lm_v, sub).all()), "all 468 mapping vertices are in the subset")
+ check(int(sub.numel()) == int(torch.unique(torch.cat([sub])).numel()),
+ "subset has no duplicates")
+ check(bool((sub[mhr.lm_vert_pos] == lm_v).all()),
+ "lm_vert_pos indexes them in the mapping's column order")
+ check(mhr.verts_loss_pos.numel() == cfg.n_verts,
+ "verts_loss_pos still names exactly the vertex-loss samples",
+ f"{mhr.verts_loss_pos.numel()} of {int(sub.numel())} skinned")
+ overlap = int(torch.isin(lm_v, sub[mhr.verts_loss_pos]).sum())
+ print(f" (union is {int(sub.numel())} vertices: {cfg.n_verts} farthest-point "
+ f"+ 468 mapping, overlapping in {overlap})")
+
+ # --- 1 & 2. equivalence, on GT / perturbed pose / changed shape ---------
+ cases = [("real GT", gt, sh)]
+ torch.manual_seed(0)
+ p = gt.clone(); p[:, 6:136] += torch.randn_like(p[:, 6:136]) * 0.25
+ cases.append(("pose perturbed by N(0, 0.25) rad", p, sh))
+ cases.append(("identity coefficients resampled", gt, sh + torch.randn_like(sh) * 1.0))
+
+ for tag, mp, sp in cases:
+ mp_a = mp.clone().requires_grad_(True); sp_a = sp.clone().requires_grad_(True)
+ j, v = mhr.get_joints_and_vertices(mp_a, sp_a)
+ got = mhr.regress_keypoints_exact(mhr.to_vision(j), mhr.to_vision(v))
+
+ mp_b = mp.clone().requires_grad_(True); sp_b = sp.clone().requires_grad_(True)
+ ref = reference_landmarks(mhr, K_full, mp_b, sp_b)
+
+ d = (got - ref).abs().max() * 1000.0 # vision frame is metres
+ check(d < 1e-3, f"landmarks vs full mesh [{tag}]", f"max {d:.3e} mm")
+
+ torch.manual_seed(1)
+ w = torch.randn_like(got)
+ (got * w).sum().backward(); (ref * w).sum().backward()
+ gm = (mp_a.grad - mp_b.grad).abs().max()
+ gs = (sp_a.grad - sp_b.grad).abs().max()
+ rel_m = gm / mp_b.grad.abs().max().clamp_min(1e-12)
+ check(rel_m < 1e-4, f"d/d(model_params) vs full mesh [{tag}]",
+ f"max {gm:.3e} (rel {rel_m:.1e})")
+ check(gs / sp_b.grad.abs().max().clamp_min(1e-12) < 1e-4,
+ f"d/d(shape_params) vs full mesh [{tag}]", f"max {gs:.3e}")
+
+ # --- 3. what the change is actually worth -------------------------------
+ with torch.no_grad():
+ j, v = mhr.get_joints_and_vertices(gt, sh)
+ jv = mhr.to_vision(j)
+ exact = mhr.regress_keypoints_exact(jv, mhr.to_vision(v))
+ fitted = mhr.regress_keypoints(jv)
+ ref = reference_landmarks(mhr, K_full, gt, sh)
+ nonfinger = torch.from_numpy(
+ np.setdiff1d(np.arange(70), T.FINGER_KP)).to(dev)
+ print(f"\ndisagreement with the teacher's readout on identical GT geometry:")
+ for name, kp in (("fitted (70,127) skeleton matrix", fitted), ("exact subset readout", exact)):
+ e = (kp - ref).norm(dim=-1) * 1000.0
+ print(f" {name:34s} all 70: {e.mean():7.3f} mm mean / {e.max():8.3f} max | "
+ f"{len(nonfinger)} non-finger: {e[:, nonfinger].mean():6.3f} mm")
+
+ # shape sensitivity: the property a skeleton-only readout cannot have
+ with torch.no_grad():
+ sh2 = sh + torch.randn_like(sh) * 1.0
+ j2, v2 = mhr.get_joints_and_vertices(gt, sh2)
+ e_ex = (mhr.regress_keypoints_exact(mhr.to_vision(j2), mhr.to_vision(v2)) - exact)
+ e_fi = (mhr.regress_keypoints(mhr.to_vision(j2)) - fitted)
+ print(f" landmark motion when identity is resampled:")
+ print(f" exact {e_ex.norm(dim=-1).mean()*1000:7.3f} mm mean <- carries a shape gradient")
+ print(f" fitted {e_fi.norm(dim=-1).mean()*1000:7.3f} mm mean <- blind to identity by construction")
+
+ # --- 5. cost ------------------------------------------------------------
+ print(f"\ncost at batch {args.batch}:")
+ for tag, nv, la in (("w_verts only (595 skinned)", cfg.n_verts, None),
+ ("exact landmarks only (468)", 0, cfg.landmark_assets_path),
+ ("both (union)", cfg.n_verts, cfg.landmark_assets_path)):
+ m = T.MHRForwardPass(cfg.mhr_model_path, dev,
+ kp_regressor=np.load(cfg.kp_regressor_path),
+ n_verts=nv, landmark_assets=la)
+ mp = gt[:1].repeat(args.batch, 1).clone().requires_grad_(True)
+ sp = sh[:1].repeat(args.batch, 1).clone().requires_grad_(True)
+ for i in range(12):
+ if i == 2:
+ torch.cuda.synchronize() if dev == "cuda" else None
+ t0 = time.perf_counter()
+ jj, vv = m.get_joints_and_vertices(mp, sp)
+ (jj.sum() + vv.sum()).backward()
+ mp.grad = sp.grad = None
+ torch.cuda.synchronize() if dev == "cuda" else None
+ ms = (time.perf_counter() - t0) / 10 * 1000
+ print(f" {tag:30s} {int(m.vert_idx.numel()):5d} vertices {ms:6.2f} ms fwd+bwd")
+
+ print()
+ print(f"{_fails} check(s) FAILED" if _fails else "all checks passed")
+ return 1 if _fails else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())