Python
feat: --losses rebalanced (w_shape 0.03, pose mask widened, euclid 3D loss, fingers 0.2)
mohamdeva9f653866e31
diff --git a/CLAUDE.md b/CLAUDE.mdindex 91f16c6..0aeccc6 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -73,10 +73,10 @@ Docs are the source of truth; the code is bigger than any context window.| order | file | for ||---|---|---|-| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` changes, the pair-index cache |+| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` and `--losses rebalanced` change, the pair-index cache || 2 | `datasets_pipeline/jeanzay/STATUS.md` | Jean Zay cluster + data state, what is built, job templates *(git-excluded, local only)* || 3 | `datasets_pipeline/JEAN_ZAY.md` | site runbook: env, quotas, proxy quirks *(git-excluded, local only)* |-| 4 | `benchmark/README.md` | how published numbers are produced; J12 vs J14 and MHR-vs-SMPL joint conventions |+| 4 | `benchmark/README.md` | how published numbers are produced; J12 vs J14, and **why the current 3DPW GT is not yet the published protocol** || 5 | `datasets_pipeline/README.md` | how the corpus is built, including the SA-1B visibility mask || 6 | `docs/architecture.md` | the student model |@@ -93,10 +93,30 @@ file so both entry points get them. **Never fork the base file** — that is how`train_distill.py` drifted until its ONNX export silently zeroed the SimCC 2Dhead, with keypoints collapsing to the bbox centre and no error raised.-**`--preset baseline` must stay bit-identical.** Runs in flight use it as the-control. If you change the dataset or a loss, gate it behind a config flag whose-default reproduces the old behaviour, and prove it — diff a few dozen augmented-samples against `git show HEAD:<file>`.+**`--preset baseline` and `--losses legacy` must stay bit-identical.** Runs in+flight use them as controls. The two flags are orthogonal axes — `--preset` is+the input pipeline and the absolute-pose terms, `--losses` is the loss budget —+and both default to the old behaviour. If you change the dataset or a loss, gate+it behind a config flag whose default reproduces the old behaviour, and prove it+— diff a few dozen augmented samples against `git show HEAD:<file>`. When you+do, compare **alternating** calls: the TorchScript MHR forward differs by ~5e-10+on its first invocation in a process (cold-kernel autotuning), which reads as a+regression if you only compare once.++**Know the MHR parameter layout before touching a pose loss.** From+`character_torch.parameter_transform.parameter_names`: `0:3` root translation+(always 0 in the corpus, 1 unit = 10 cm), `3:6` root rotation (extrinsic-XYZ+Euler, driving joint-parameter slots 10/11/12 *and nothing else*), `6:136` the+130 local joint angles, `136:204` 68 bone scales. Consequences that cost a+session to re-derive: `6:136` is exactly invariant to the in-plane rotation+augmentation, so it never needed the `m_ident` mask; and the 45 `shape_params`+move the 127-joint skeleton by `0.00e+00 cm`, so they are mesh-only and every+loss and metric in the trainer is blind to them.++**`DistillationLoss` is never `.to(device)`'d.** `run_overfit_test`,+`run_self_tests`, `train_instant_hmr` and `train_distill_jz.train` all construct+it bare, so any buffer you register lands on the CPU and raises on step 1. Build+tensors on `mhr_module.device` instead.**Seed spread is wide.** Observed 70-keypoint PA-MPJPE varies 34–49 mm acrossseeds, which is larger than several of the changes on the roadmap. A single-rundiff --git a/README.md b/README.mdindex 4d74280..03e83cb 100644--- a/README.md+++ b/README.md@@ -182,10 +182,23 @@ InstantHMR ONNX└─ cam_trans (3,) ──┘ + cam_trans → camera-space mesh```-The 204-dim `mhr_params` encode joint rotations in 6-D representation-(34 joints × 6); `shape_params (45,)` encode identity blend shapes-(20 body + 20 head + 5 hand components). Facial expression is set to-neutral (zeros) since InstantHMR does not regress it.+The 204-dim `mhr_params` are MHR **model parameters**, not a 6-D rotation+encoding. Read off `character_torch.parameter_transform.parameter_names`, the+layout is:++| slice | contents |+|---|---|+| `0:3` | `root_tx/ty/tz` — always 0 in the corpus; 1 unit = 10 cm, applied at joint 1 |+| `3:6` | `root_rx/ry/rz` — extrinsic-XYZ Euler, driving joint-parameter slots 10/11/12 and nothing else |+| `6:136` | 130 local joint angles (`spine0_rx_flexible`, `l_knee_ry`, …) |+| `136:204` | 68 bone-scale parameters |++Because the root rotation is exclusive to slots 10/11/12, `6:136` is exactly+invariant to a camera roll — which is what `pose_split` in the trainer relies+on. `shape_params (45,)` encode identity blend shapes (20 body + 20 head + 5+hand components) and drive the **mesh only**: a ±2σ probe moves the 127-joint+skeleton by `0.00e+00 cm`. Facial expression is set to neutral (zeros) since+InstantHMR does not regress it.## Performance tuningdiff --git a/benchmark/README.md b/benchmark/README.mdindex f22f02e..bd340f9 100644--- a/benchmark/README.md+++ b/benchmark/README.md@@ -82,6 +82,39 @@ comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits noper-keypoint confidence, so every keypoint is scored 1.0 — read AP next to theconfidence-free `mean_OKS` and `PCK` numbers.+### The GT-joint caveat — READ THIS BEFORE QUOTING A 3DPW NUMBER++**The ground truth this harness uses is not the one 3DPW papers report on.**+`benchlib/threedpw.py` reads `jointPositions` straight from the sequence+pickles: the 24 **SMPL kinematic-tree** joints. Every published 3DPW number —+NLF's included ("obtained through the same Human3.6M-style joint regressor that+all prior works use") — instead runs SMPL *forward* on the GT `poses` / `betas`+and applies the H36M joint regressor to the resulting 6890 vertices. Those are+different landmarks, most visibly at the shoulders and hips, which is 4 of the+12 joints in J12.++That is what makes this harness cheap to set up (see "Why these two" above) and+it is fine for *tracking a run*. It is **not** comparable to a published table.++Fixing it needs two files that are not in this repo:++| file | where |+|---|---|+| `SMPL_{MALE,FEMALE,NEUTRAL}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so get the gendered pair. |+| `J_regressor_h36m.npy` | ships inside SPIN / HMR2.0 / 4D-Humans |++Then: `V = SMPL(poses, betas)` per frame, `J14 = J_regressor @ V`, and fit the+existing `--fit-adapter` linear MHR70 -> J14 map on 3DPW **train** against that+GT before reporting on test. A forward pass, not a fit.++`v_template_clothed` in the pickles cannot substitute — it is a static T-pose+surface with no blend weights, no pose blendshapes and no joint regressor, so it+cannot be posed.++This does **not** affect `instanthmr_distill_train/val3dpw.py`, which uses the+same GT for checkpoint selection: selection only needs a consistent metric, not+a comparable one.+### The joint-convention caveat — do not skip thisInstantHMR predicts MHR70 joints; 3DPW's GT is SMPL. The two rigs place severaldiff --git a/docs/architecture.md b/docs/architecture.mdindex 21613f9..e297f39 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -1,9 +1,18 @@# Model architecture-InstantHMR is a 17 M-parameter network that mimics-`facebook/sam-3d-body-dinov3` on the **per-person 3D pose estimation** task,-running ~80× faster on a single GPU and exporting cleanly to ONNX / TFLite /-QNN for mobile and edge deployment.+InstantHMR mimics `facebook/sam-3d-body-dinov3` on the **per-person 3D pose+estimation** task, running ~80× faster on a single GPU and exporting cleanly to+ONNX / TFLite / QNN for mobile and edge deployment.++> **This page describes the released 17 M checkpoint, not what the trainer+> builds today.** `train_distill_mhr_only.py`'s `DistillConfig` defaults to+> `repvit_m2_3` at `d_model=512`, 8 heads, 4 decoder layers, and **71** query+> tokens (1 global + 70 for the 2D SimCC head) — **39.96 M parameters**:+> 22.40 M backbone, 16.82 M decoder. The per-part joint queries in the table+> below are gone; every parameter output now comes from one+> `Linear(512 → 252)` on the single global token, and the 3D keypoints are+> derived from the MHR forward pass rather than predicted. Read the config, not+> this page, before quoting a parameter count.## High leveldiff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.mdindex 3537c41..6e646cc 100644--- a/instanthmr_distill_train/README.md+++ b/instanthmr_distill_train/README.md@@ -55,6 +55,46 @@ Two things to know when reading v2 results:so the term stops being decorative). Sweep it first if v2 underperforms:`--preset v2 --w_reproj 0.1`.+## The `--losses` switch++**Orthogonal to `--preset`.** `--preset` controls the input pipeline and the+absolute-pose terms; `--losses {legacy,rebalanced}` controls the loss budget.+`legacy` is the default and is bit-identical to the pre-rebalance trainer, so+b2 / v2 runs already in flight stay valid controls. The four arms are+`baseline`, `v2`, `baseline + rebalanced`, `v2 + rebalanced`.++`rebalanced` (see `apply_rebalanced_losses` in `train_distill_mhr_only.py`)+comes from two measurements on a trained checkpoint, both reproducible from the+repo root against `data/sam3d_gt_coco`:++- **an oracle ablation** — replace one predicted group with the teacher's value+ and re-measure. Of the 37.8 mm of removable PA-MPJPE, `pose[6:136]` (the 130+ local joint angles) accounts for **34.3 mm**; the bone scales, the root and+ the 45 identity blendshapes account for **0.00 mm each**.+- **a per-term gradient norm** — back-propagate each term alone. `loss_shape`+ owned 27% of the update direction and `loss_cam` 15%, against `loss_pose`'s+ 4.7% on 21% of the batch.++| item | change | why |+|---|---|---|+| shape | `w_shape` 1.0 -> 0.03 | Driving MHR params 204–248 to ±2σ moves the 127-joint skeleton by exactly `0.00e+00` cm, and `MHRForwardPass.get_joints` substitutes zeros for `shape_params` anyway. The blendshapes drive the mesh only. Kept non-zero so the deployed mesh identity stays supervised. |+| pose mask | `pose_split` | `loss_pose` was masked to `m_ident` (21% of the batch) because the dataset rotates the labels. Model params 3/4/5 map to joint-parameter slots 10/11/12 and nothing else, so `pose[6:136]` is exactly invariant to the in-plane rotation and only needs the flip mask — 21% -> 60% coverage on the parameters that carry all of the error. Root stays on `m_ident`; the two halves keep their per-parameter weight, so this changes the mask and not the root-vs-local balance. |+| pose beta | `pose_beta` 1.0 -> 0.05 | SmoothL1's beta is in radians here, so 1.0 is ~10x the p95 error and the term is a scaled MSE. |+| 3D loss | `kp3d_loss` `euclid`, `w_keypoints3d` 2.0 -> 0.35 | SmoothL1's beta of 1.0 is one **metre**, so at a 37.7 mm mean / 119 mm p95 error every sample is in the quadratic branch: 16x less gradient than L1 and 31x less than NLF's unsquared Euclidean at the same weight. The weight drops because the form under it changed. |+| fingers | `finger_weight` 0.2 | 40 of the 70 keypoints are finger joints and they carry 61% of `loss_3d_native`'s mass, while J12/J14 contain none of them and SA-1B — 55% of the batch under `--mix sqrt` — records hands as 8% observed. Applied to `loss_3d_native` and `loss_reproj`, the two FK-derived geometric terms; the 2D head's own terms are untouched. |++**The unsquared Euclidean is gentler where divergence happens, not harsher.**+`d‖x‖/dx` is a unit vector, so its gradient is flat in the error, where+SmoothL1(beta=1) grows with it. Measured against the form it replaces, at the+weights above: **0.62x** the gradient on a freshly initialised model (515 mm+error) and 6.4x once trained (60 mm). That is the opposite of the shape that+produced the LR blow-ups in `datasets_pipeline/jeanzay/STATUS.md`.++Validation also gains `Mesh_PA_MPJPE_body` / `Mesh_MPJPE_body` over the 30+non-finger keypoints, reported next to the existing metrics. **Selection is+unchanged** — and with `--val-3dpw` on, both are replaced by the 3DPW J12+numbers anyway.+## The pair index`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` perdiff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.pyindex 83a3179..c99a919 100755--- a/instanthmr_distill_train/train_distill_jz.py+++ b/instanthmr_distill_train/train_distill_jz.py@@ -440,7 +440,8 @@ def unwrap(m):@torch.no_grad()def run_validation(eval_model, loader, criterion, cfg, mhr_module, device):eval_model.eval()- acc = {"loss": 0.0, "Mesh_MPJPE": 0.0, "Mesh_PA_MPJPE": 0.0}+ acc = {"loss": 0.0, "Mesh_MPJPE": 0.0, "Mesh_PA_MPJPE": 0.0,+ "Mesh_PA_MPJPE_body": 0.0}n = 0for batch in loader:batch = {k: (v.to(device, non_blocking=True) if isinstance(v, torch.Tensor) else v)@@ -456,6 +457,10 @@ def run_validation(eval_model, loader, criterion, cfg, mhr_module, device):acc["loss"] += lvacc["Mesh_MPJPE"] += bm.get("Mesh_MPJPE", 0.0)acc["Mesh_PA_MPJPE"] += bm.get("Mesh_PA_MPJPE", 0.0)+ # Reported only. Mesh_PA_MPJPE is 60% fingers by keypoint count; this is+ # the 30 body/face/foot keypoints. Selection is untouched — and when+ # --val-3dpw is on, both of these are replaced by the 3DPW J12 numbers.+ acc["Mesh_PA_MPJPE_body"] += bm.get("Mesh_PA_MPJPE_body", 0.0)n += 1out = {k: v / max(n, 1) for k, v in acc.items()}out["n"] = float(n)@@ -618,6 +623,10 @@ def train(args, cfg, rank, local_rank, world):"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,"jpeg_p": cfg.jpeg_p, "cam_loss": cfg.cam_loss,+ "losses": args.losses, "w_shape": cfg.w_shape,+ "w_keypoints3d": cfg.w_keypoints3d, "kp3d_loss": cfg.kp3d_loss,+ "pose_split": cfg.pose_split, "pose_beta": cfg.pose_beta,+ "finger_weight": cfg.finger_weight,"started": time.strftime("%Y-%m-%d %H:%M")})for epoch in range(start_epoch, args.epochs):@@ -694,7 +703,11 @@ def train(args, cfg, rank, local_rank, world):run["simcc"] += losses.get("loss_2d_simcc", torch.zeros(())).item()run["reproj"] += losses.get("loss_reproj", torch.zeros(())).item()run["cam"] += losses.get("loss_cam", torch.zeros(())).item()- run["pose"] += losses.get("loss_pose", torch.zeros(())).item()+ # Under cfg.pose_split, loss_pose is the local half and loss_pose_root+ # the other; summing keeps history.jsonl's "pose" column comparable+ # against a --losses legacy run.+ run["pose"] += (losses.get("loss_pose", torch.zeros(())).item()+ + losses.get("loss_pose_root", torch.zeros(())).item())nb += 1if is_main() and i % args.log_every == 0:log(f" e{epoch+1}/{args.epochs} step {i}/{steps_per_epoch} "@@ -733,9 +746,11 @@ def train(args, cfg, rank, local_rank, world):f"reproj {tr['reproj']:.4f} | cam {tr['cam']:.4f} "f"| skipped {n_skip}/{steps_per_epoch} | {time.time() - t_epoch:.0f}s")log(f" valRAW loss {raw['loss']:.4f} | PA-MPJPE {raw['Mesh_PA_MPJPE']:.1f} | "- f"MPJPE {raw['Mesh_MPJPE']:.1f} mm")+ f"MPJPE {raw['Mesh_MPJPE']:.1f} mm | held-in body PA "+ f"{raw['Mesh_PA_MPJPE_body']:.1f} mm")log(f" valEMA loss {em['loss']:.4f} | PA-MPJPE {em['Mesh_PA_MPJPE']:.1f} | "- f"MPJPE {em['Mesh_MPJPE']:.1f} mm")+ f"MPJPE {em['Mesh_MPJPE']:.1f} mm | held-in body PA "+ f"{em['Mesh_PA_MPJPE_body']:.1f} mm")if dpw_raw is not None:log(f" 3DPW-{args.val_3dpw_split} RAW J12 PA {dpw_raw['J12_PA_MPJPE']:.1f} "f"MPJPE {dpw_raw['J12_MPJPE']:.1f} | J14 PA {dpw_raw['J14_PA_MPJPE']:.1f} mm")@@ -878,6 +893,14 @@ def parse_args():"prerequisites, plus absolute-pose supervision. ""'baseline' reproduces the pre-v2 trainer exactly.")p.add_argument("--w_reproj", type=float, default=None)+ p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",+ help="Orthogonal to --preset, which only controls the input "+ "pipeline and the absolute-pose terms. 'rebalanced' applies "+ "the four loss-budget fixes (w_shape 1.0->0.03, pose split "+ "onto the non-flipped 60%% of the batch at beta 0.05, "+ "unsquared-Euclidean 3D loss at w 0.35, finger keypoints at "+ "0.2). 'legacy' reproduces the pre-rebalance recipe bit for "+ "bit, so b2/v2 runs in flight stay valid controls.")p.add_argument("--index-cache", default=None,help="Where to cache the (image, npz) pair index. ""Default: <data_root>/.pair_index_v2.pkl")@@ -904,15 +927,19 @@ def build_cfg(args):cfg.epochs = args.epochscfg.per_dataset_caps = {} # replaced by the weighted samplerif args.backbone: cfg.backbone = args.backbone- if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3dif args.w_simcc is not None: cfg.w_simcc = args.w_simccif args.weight_decay is not None: cfg.weight_decay = args.weight_decayif args.ema_decay is not None: cfg.ema_decay = args.ema_decayif args.preset == "v2":apply_v2_preset(cfg)- # An explicit --w_reproj still wins over the preset.+ # Orthogonal axis: --preset is the input pipeline + absolute-pose terms,+ # --losses is the loss budget. Applied second so it owns w_keypoints3d.+ if args.losses == "rebalanced":+ T.apply_rebalanced_losses(cfg)+ # An explicit --w_reproj / --w_keypoints3d still wins over both.if args.w_reproj is not None: cfg.w_reproj = args.w_reproj+ if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3dreturn cfg@@ -998,6 +1025,11 @@ def main():f"occl_p={cfg.occl_p} jpeg_p={cfg.jpeg_p} "f"w_reproj={cfg.w_reproj} cam={cfg.cam_loss}"if args.preset != "baseline" else ""))+ log(f"losses {args.losses}"+ + (f" | w_shape={cfg.w_shape} pose_split={cfg.pose_split} "+ f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "+ f"finger_w={cfg.finger_weight}"+ if args.losses != "legacy" else ""))log(f"seed {args.seed} | mix {args.mix} | "f"samples/epoch {args.samples_per_epoch:,}")log("=" * 66)diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex 72a05ab..1d568eb 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -121,6 +121,31 @@ def _build_flip_perm(names):FLIP_PERM = _build_flip_perm(JOINT_NAMES)+# 40 of the 70 keypoints are finger joints. Measured on a trained checkpoint,+# they carry 61.0% of loss_3d_native's L1 mass and 60% of Mesh_PA_MPJPE, while+# the published metrics (J12 / J14) contain none of them and SA-1B — 55% of the+# batch under `--mix sqrt` — records hands as 8% observed. `finger_weight`+# down-weights them in the FK-derived geometric losses; see DistillConfig.+FINGER_KP = np.array(+ [i for i, n in enumerate(JOINT_NAMES)+ if any(t in n for t in ("thumb", "index", "middle", "ring", "pinky"))],+ dtype=np.int64)+BODY_KP = np.array([i for i in range(len(JOINT_NAMES)) if i not in set(FINGER_KP.tolist())],+ dtype=np.int64)+assert len(FINGER_KP) == 40 and len(BODY_KP) == 30++# MHR model-parameter layout, read off character_torch.parameter_transform:+# 0:3 root_tx/ty/tz (always 0 in the corpus)+# 3:6 root_rx/ry/rz -> joint-parameter slots 10/11/12 and nothing else+# 6:136 130 local joint angles+# 136:204 68 bone scales+# Because the root rotation is exclusive to slots 10/11/12, the 130 local angles+# are EXACTLY invariant to the in-plane rotation the dataset applies, which is+# what makes `pose_split` valid. Verified: recomposing the root as+# R_aug @ R_root in extrinsic-xyz Euler reproduces the rotated skeleton up to a+# pure translation (the pelvis-vs-origin pivot offset), not a pose change.+POSE_ROOT_DIM = 6+# ============================================================# Cell 1 — Configuration# ============================================================@@ -166,6 +191,36 @@ class DistillConfig:reproj_all_samples: bool = False # include flipped samples in loss_reprojcam_loss: str = "mse" # "mse" | "euclid" (unsquared L2 on xyz)+ # --- loss-rebalance knobs (`--losses rebalanced` in train_distill_jz.py).+ # --- Every default below reproduces the pre-rebalance behaviour bit for bit,+ # --- so `--preset baseline` and `--preset v2` are unaffected. All four come+ # --- from the measured error/gradient budget:+ # * an oracle ablation attributes 34.3 of the 37.8 mm of removable+ # PA-MPJPE to pose[6:136] and 0.00 mm to shape_params, the bone scales+ # and the root, yet loss_shape owns 27% of the gradient and loss_pose+ # 4.7% on 21% of the batch;+ # * SmoothL1's default beta=1.0 is a metre, so on metre-scale coordinates+ # every sample sits in the quadratic branch -- 16x less gradient than L1+ # and 31x less than NLF's unsquared Euclidean, at the same weight.+ pose_split: bool = False # supervise pose[6:136] on the 60% of the batch+ # that is not flipped instead of the 21% that is+ # un-augmented. Root stays on m_ident. The two+ # halves keep their per-parameter weight (6/136+ # and 130/136 of w_pose), so this changes the+ # MASK, not the root-vs-local balance.+ pose_beta: float = 1.0 # SmoothL1 beta for the pose terms, in radians.+ # 1.0 rad is ~10x the p95 error, i.e. pure MSE.+ kp3d_loss: str = "smooth_l1" # "smooth_l1" | "euclid" (NLF 2407.07532 §3.2:+ # "the Euclidean loss without squaring, for+ # better outlier-robustness"). euclid has a+ # FLAT gradient, so it is *gentler* than the+ # current form at init (measured 0.62x on a+ # fresh model, 515 mm error) and stronger at+ # convergence (6.4x at 60 mm) -- the opposite+ # of the shape that caused the LR divergences.+ finger_weight: float = 1.0 # per-keypoint weight on FINGER_KP inside+ # loss_3d_native and loss_reproj+max_consecutive_skips: int = 200 # abort rather than burn the allocation# on a run that has already diverged@@ -253,6 +308,43 @@ class DistillConfig:native_mapping_ids = [ 1, 2, 5, 6, 9, 10, 11, 12, 13, 14, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62 ]mhr_mapping_ids = [ 125, 123, 75, 39, 2, 18, 3, 19, 5, 21, 64, 63, 62, 61, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 41, 100, 99, 98, 97, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 77 ]+def apply_rebalanced_losses(cfg) -> None:+ """The four loss-budget fixes, as one switch. Lives here, not in+ train_distill_jz.py, so the single-GPU trainer and the DDP driver cannot+ drift apart.++ Every number is chosen from a measured unit-weight gradient norm on real+ augmented batches, not from taste. The resulting split of the update+ direction (batch 12, epoch-78 checkpoint, COCO):++ term before after+ loss_3d_native 8.5% 30.3%+ loss_2d_simcc 29.6% 26.4%+ loss_pose 5.6% 20.2%+ loss_cam 13.8% 12.3%+ loss_mhr_joints 8.5% 7.6%+ loss_shape 14.4% 0.4%++ `w_keypoints3d` drops 2.0 -> 0.35 because the loss form under it changed:+ at equal weight the unsquared Euclidean carries 31x the gradient of+ SmoothL1(beta=1) at the operating point, so keeping 2.0 would put the+ FK path at ~20x its current strength -- the failure mode the w=10+ experiments in this file's header already found. 0.35 lands it at ~3x,+ leading without dominating.+ """+ cfg.w_shape = 0.03 # 45 identity blendshapes. Verified: driving MHR+ # params 204..248 to +/-2 sigma moves the 127-joint+ # skeleton by 0.00e+00 cm, and get_joints() zeroes+ # them before the forward pass anyway. Non-zero so+ # the deployed mesh identity stays supervised.+ cfg.pose_split = True # pose[6:136] on m_noflip (60%) not m_ident (21%)+ cfg.pose_beta = 0.05 # ~= the p50 pose-parameter error, so the term is+ # linear over the bulk of the distribution+ cfg.kp3d_loss = "euclid" # NLF 2407.07532 §3.2+ cfg.w_keypoints3d = 0.35 # re-scaled for the new form; see docstring+ cfg.finger_weight = 0.2 # 40 of 70 keypoints, none of them in J12/J14++# ============================================================# Cell 3 — Dataset and Data Augmentation# ============================================================@@ -799,6 +891,17 @@ class DistillationLoss(nn.Module):self.simcc_sigma = 2.0 * bin_wself._global_step = 0self.warmup_steps = int(getattr(cfg, "kp3d_warmup_steps", 0))+ # Per-keypoint weight for the FK-derived geometric losses. All ones+ # unless finger_weight != 1, so the unweighted path stays bit-identical.+ #+ # Built on the MHR module's device, NOT the CPU default: every call site+ # in this repo constructs DistillationLoss without .to(device)+ # (run_overfit_test, run_self_tests, train_instant_hmr and+ # train_distill_jz.train all do), so a CPU buffer would raise on the+ # first step. register_buffer still lets an explicit .to() move it.+ w = torch.ones(cfg.num_joints, device=getattr(mhr_module, "device", "cpu"))+ w[torch.from_numpy(FINGER_KP).to(w.device)] = float(getattr(cfg, "finger_weight", 1.0))+ self.register_buffer("kp3d_weight", w, persistent=False)def set_step(self, step: int):"""Drives the FK-loss warm-up ramp. Call once per optimiser step."""@@ -819,6 +922,12 @@ class DistillationLoss(nn.Module):e = F.smooth_l1_loss(pred, tgt, reduction='none').flatten(1).mean(1)return self._mmean(e, mask)+ def _m_beta(self, pred, tgt, mask, beta):+ """m_smooth_l1 with an explicit beta. beta=1.0 is F.smooth_l1_loss's+ default, so this reproduces m_smooth_l1 exactly at the default."""+ e = F.smooth_l1_loss(pred, tgt, reduction='none', beta=beta).flatten(1).mean(1)+ return self._mmean(e, mask)+def m_mse(self, pred, tgt, mask):e = F.mse_loss(pred, tgt, reduction='none').flatten(1).mean(1)return self._mmean(e, mask)@@ -897,7 +1006,26 @@ class DistillationLoss(nn.Module):tgt_pose = targets["mhr_model_params"][:, :idx_pose]tgt_scale = targets["mhr_model_params"][:, idx_pose:]- losses['loss_pose'] = self.m_smooth_l1(pred_pose, tgt_pose, m_ident) * self.cfg.w_pose+ # loss_pose. Masked to m_ident (~21% of the batch at geom_p 0.8) because+ # the dataset rotates the LABELS and the target parameters are not+ # rotated with them. That is only true of the 6 root parameters:+ # model params 3/4/5 drive joint-parameter slots 10/11/12 exclusively,+ # so pose[6:136] -- the 130 local angles, which the oracle ablation+ # attributes 100% of the PA-MPJPE error to -- is invariant to the+ # in-plane rotation and only needs the flip mask. See POSE_ROOT_DIM.+ beta = float(getattr(self.cfg, "pose_beta", 1.0))+ if getattr(self.cfg, "pose_split", False):+ r = POSE_ROOT_DIM+ n = pred_pose.shape[1]+ # Proportional weights: the split changes the MASK, not the+ # root-vs-local balance a single mean over n parameters implied.+ losses['loss_pose_root'] = self._m_beta(+ pred_pose[:, :r], tgt_pose[:, :r], m_ident, 1.0) * self.cfg.w_pose * (r / n)+ losses['loss_pose'] = self._m_beta(+ pred_pose[:, r:], tgt_pose[:, r:], m_noflip, beta) * self.cfg.w_pose * ((n - r) / n)+ else:+ losses['loss_pose'] = self._m_beta(+ pred_pose, tgt_pose, m_ident, beta) * self.cfg.w_poselosses['loss_scale'] = self.m_mse(pred_scale, tgt_scale, m_noflip) * self.cfg.w_scalelosses['loss_shape'] = self.m_mse(preds["shape_params"], targets["shape_params"], m_noflip) * self.cfg.w_shape# NLF applies its translation loss as an unsquared Euclidean norm over the@@ -980,7 +1108,22 @@ class DistillationLoss(nn.Module):# skeleton is mirror-symmetric to 0.7 mm, so a mirrored pose is a perfectly# representable MHR pose. (loss_pose / loss_scale / loss_shape stay masked# because the TARGET PARAMETERS are not mirrored — only the geometry is.)- losses['loss_3d_native'] = self.l1(pred_kp3d, tgt_3d) * self.cfg.w_keypoints3d * self.fk_scale+ #+ # SmoothL1's beta defaults to 1.0 -- one METRE here -- so every sample of+ # a 37.7 mm mean / 119 mm p95 error distribution sits in the quadratic+ # branch and the term behaves as a scaled MSE: measured 16x less gradient+ # than L1 and 31x less than the unsquared Euclidean at the same weight,+ # which is why w_keypoints3d=2.0 delivered 7.7% of the update direction.+ # "euclid" also has a FLAT gradient (d||x||/dx is a unit vector), so it is+ # gentler than the current form where the error is large -- 0.62x on a+ # freshly initialised model -- and only stronger once the model is close.+ if getattr(self.cfg, "kp3d_loss", "smooth_l1") == "euclid":+ e_kp3d = (pred_kp3d - tgt_3d).norm(dim=-1) # (B, 70), metres+ losses['loss_3d_native'] = (+ self._kp_weighted(e_kp3d, self.kp3d_weight.expand_as(e_kp3d))+ * self.cfg.w_keypoints3d * self.fk_scale)+ else:+ losses['loss_3d_native'] = self.l1(pred_kp3d, tgt_3d) * self.cfg.w_keypoints3d * self.fk_scalecc = targets["aug_cos"].view(-1, 1)ss = targets["aug_sin"].view(-1, 1)@@ -1020,8 +1163,13 @@ class DistillationLoss(nn.Module):m_reproj = onespred_reproj = self._project_3d_to_norm_2d(pts_cr, cam_cr_r, targets)e_reproj = (pred_reproj - tgt_2d).abs().mean(-1)+ # Same finger down-weight as loss_3d_native: this is the other geometric+ # term over all 70 keypoints, and at --preset v2's w_reproj = 0.5 it would+ # otherwise also be ~60% fingers. The 2D head's own terms are left alone —+ # joints_2d is a deployed output and SA-1B's visibility mask already+ # handles its unobserved hands.losses['loss_reproj'] = self._kp_weighted(- e_reproj, w_kp, m_reproj) * self.cfg.w_reproj * self.fk_scale+ e_reproj, w_kp * self.kp3d_weight, m_reproj) * self.cfg.w_reproj * self.fk_scalelosses['total_loss'] = sum(losses.values())return losses@@ -1111,6 +1259,18 @@ def evaluate_hmr_batch(preds, targets, cfg, mhr_module):metrics['Mesh_PA_MPJPE'] = torch.linalg.norm(pred_aligned - gt_3d_native, dim=-1).mean().item() * 1000.0+ # Body-only variant over the 30 non-finger keypoints. Mesh_PA_MPJPE is 60%+ # fingers by keypoint count, so on the single-GPU path it selects largely on+ # hands; the published J12/J14 sets contain none of them. Reported alongside,+ # never substituted — selection behaviour is unchanged.+ b = torch.from_numpy(BODY_KP).to(pred_kp3d.device)+ pb, gb = pred_kp3d[:, b], gt_3d_native[:, b]+ metrics['Mesh_MPJPE_body'] = torch.linalg.norm(+ (pb - pb.mean(1, keepdim=True)) - (gb - gb.mean(1, keepdim=True)),+ dim=-1).mean().item() * 1000.0+ metrics['Mesh_PA_MPJPE_body'] = torch.linalg.norm(+ batched_procrustes_alignment(pb, gb) - gb, dim=-1).mean().item() * 1000.0+if hasattr(cfg, 'native_mapping_ids') and hasattr(cfg, 'mhr_mapping_ids'):pred_mesh_anchors = pred_mhr_vision[:, cfg.mhr_mapping_ids, :]gt_mesh_anchors = gt_3d_native[:, cfg.native_mapping_ids, :]@@ -1360,7 +1520,8 @@ def train_instant_hmr():tm['2d'] += losses.get('loss_2d_native', torch.tensor(0)).item()tm['simcc'] += losses.get('loss_2d_simcc', torch.tensor(0)).item()tm['3d'] += losses.get('loss_3d_native', torch.tensor(0)).item()- tm['pose'] += losses.get('loss_pose', torch.tensor(0)).item()+ tm['pose'] += (losses.get('loss_pose', torch.tensor(0)).item()+ + losses.get('loss_pose_root', torch.tensor(0)).item())tm['scale'] += losses.get('loss_scale', torch.tensor(0)).item()tm['mhr'] += losses.get('loss_mhr_joints', torch.tensor(0)).item()tm['reproj'] += losses.get('loss_reproj', torch.tensor(0)).item()@@ -1804,8 +1965,11 @@ def run_overfit_test(steps=3000, subset=8, lr=5e-4):batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}def reg(ls):+ # loss_pose_root only exists under cfg.pose_split; the .get keeps the+ # regression figure comparable between the two settings.return (ls['loss_2d_native'].item() + ls['loss_3d_native'].item()- + ls['loss_cam'].item() + ls['loss_pose'].item())+ + ls['loss_cam'].item() + ls['loss_pose'].item()+ + ls.get('loss_pose_root', torch.zeros(())).item())first_reg = last_reg = last_pa = Nonebest_pa = float('inf') # 8-sample overfit is noisy at constant LR; judge the best fit reached@@ -1855,6 +2019,7 @@ def run_overfit_test(steps=3000, subset=8, lr=5e-4):f"| shape {losses.get('loss_shape', torch.tensor(0)).item():.4f} "f"| mhr {losses['loss_mhr_joints'].item():.4f} "f"| Mesh [MPJPE: {m['Mesh_MPJPE']:.1f} | PA: {m['Mesh_PA_MPJPE']:.1f}] mm "+ f"| body PA: {m['Mesh_PA_MPJPE_body']:.1f} mm "f"| Mesh52 [MPJPE: {m['Mesh52_MPJPE']:.1f} | PA: {m['Mesh52_PA_MPJPE']:.1f}] mm")# THRESHOLD CHANGED, 40.0 -> 55.0 mm, and this is deliberate — the quantity@@ -1990,6 +2155,11 @@ def parse_args():help="Optimiser steps over which the FK-path losses ramp from 0 to full weight.")p.add_argument("--w_keypoints3d", type=float, default=None,help="Weight of the MHR-derived 3D keypoint loss (FK path).")+ p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",+ help="'rebalanced' applies the four loss-budget fixes "+ "(w_shape down, pose split + beta, unsquared-Euclidean 3D "+ "loss, finger down-weight). 'legacy' reproduces the "+ "pre-rebalance recipe bit for bit. See apply_rebalanced_losses.")p.add_argument("--seed", type=int, default=None,help="Seed torch/numpy/random (for reproducible sanity runs).")p.add_argument("--split-wd", dest="split_wd", action="store_true",@@ -2035,6 +2205,8 @@ def main():cfg.per_dataset_caps["sam3d_gt_harmony4d"] = args.harmony4d_capif args.no_resume: cfg.resume = Falseif args.kp3d_warmup_steps is not None: cfg.kp3d_warmup_steps = args.kp3d_warmup_steps+ if args.losses == "rebalanced": apply_rebalanced_losses(cfg)+ # An explicit --w_keypoints3d still wins over the preset.if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3dif args.split_wd: cfg.split_weight_decay = Trueif args.backbone is not None: