Skip to content
Python

feat: --losses rebalanced (w_shape 0.03, pose mask widened, euclid 3D loss, fingers 0.2)

mohamdeva9f653866e31
diff --git a/CLAUDE.md b/CLAUDE.md
index 91f16c6..0aeccc6 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -73,10 +73,10 @@ Docs are the source of truth; the code is bigger than any context window.
| order | file | for |
|---|---|---|
-| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` changes, the pair-index cache |
+| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` and `--losses rebalanced` change, the pair-index cache |
| 2 | `datasets_pipeline/jeanzay/STATUS.md` | Jean Zay cluster + data state, what is built, job templates *(git-excluded, local only)* |
| 3 | `datasets_pipeline/JEAN_ZAY.md` | site runbook: env, quotas, proxy quirks *(git-excluded, local only)* |
-| 4 | `benchmark/README.md` | how published numbers are produced; J12 vs J14 and MHR-vs-SMPL joint conventions |
+| 4 | `benchmark/README.md` | how published numbers are produced; J12 vs J14, and **why the current 3DPW GT is not yet the published protocol** |
| 5 | `datasets_pipeline/README.md` | how the corpus is built, including the SA-1B visibility mask |
| 6 | `docs/architecture.md` | the student model |
@@ -93,10 +93,30 @@ file so both entry points get them. **Never fork the base file** — that is how
`train_distill.py` drifted until its ONNX export silently zeroed the SimCC 2D
head, with keypoints collapsing to the bbox centre and no error raised.
-**`--preset baseline` must stay bit-identical.** Runs in flight use it as the
-control. If you change the dataset or a loss, gate it behind a config flag whose
-default reproduces the old behaviour, and prove it — diff a few dozen augmented
-samples against `git show HEAD:<file>`.
+**`--preset baseline` and `--losses legacy` must stay bit-identical.** Runs in
+flight use them as controls. The two flags are orthogonal axes — `--preset` is
+the input pipeline and the absolute-pose terms, `--losses` is the loss budget —
+and both default to the old behaviour. If you change the dataset or a loss, gate
+it behind a config flag whose default reproduces the old behaviour, and prove it
+— diff a few dozen augmented samples against `git show HEAD:<file>`. When you
+do, compare **alternating** calls: the TorchScript MHR forward differs by ~5e-10
+on its first invocation in a process (cold-kernel autotuning), which reads as a
+regression if you only compare once.
+
+**Know the MHR parameter layout before touching a pose loss.** From
+`character_torch.parameter_transform.parameter_names`: `0:3` root translation
+(always 0 in the corpus, 1 unit = 10 cm), `3:6` root rotation (extrinsic-XYZ
+Euler, driving joint-parameter slots 10/11/12 *and nothing else*), `6:136` the
+130 local joint angles, `136:204` 68 bone scales. Consequences that cost a
+session to re-derive: `6:136` is exactly invariant to the in-plane rotation
+augmentation, so it never needed the `m_ident` mask; and the 45 `shape_params`
+move the 127-joint skeleton by `0.00e+00 cm`, so they are mesh-only and every
+loss and metric in the trainer is blind to them.
+
+**`DistillationLoss` is never `.to(device)`'d.** `run_overfit_test`,
+`run_self_tests`, `train_instant_hmr` and `train_distill_jz.train` all construct
+it bare, so any buffer you register lands on the CPU and raises on step 1. Build
+tensors on `mhr_module.device` instead.
**Seed spread is wide.** Observed 70-keypoint PA-MPJPE varies 34–49 mm across
seeds, which is larger than several of the changes on the roadmap. A single-run
diff --git a/README.md b/README.md
index 4d74280..03e83cb 100644
--- a/README.md
+++ b/README.md
@@ -182,10 +182,23 @@ InstantHMR ONNX
└─ cam_trans (3,) ──┘ + cam_trans → camera-space mesh
```
-The 204-dim `mhr_params` encode joint rotations in 6-D representation
-(34 joints × 6); `shape_params (45,)` encode identity blend shapes
-(20 body + 20 head + 5 hand components). Facial expression is set to
-neutral (zeros) since InstantHMR does not regress it.
+The 204-dim `mhr_params` are MHR **model parameters**, not a 6-D rotation
+encoding. Read off `character_torch.parameter_transform.parameter_names`, the
+layout is:
+
+| slice | contents |
+|---|---|
+| `0:3` | `root_tx/ty/tz` — always 0 in the corpus; 1 unit = 10 cm, applied at joint 1 |
+| `3:6` | `root_rx/ry/rz` — extrinsic-XYZ Euler, driving joint-parameter slots 10/11/12 and nothing else |
+| `6:136` | 130 local joint angles (`spine0_rx_flexible`, `l_knee_ry`, …) |
+| `136:204` | 68 bone-scale parameters |
+
+Because the root rotation is exclusive to slots 10/11/12, `6:136` is exactly
+invariant to a camera roll — which is what `pose_split` in the trainer relies
+on. `shape_params (45,)` encode identity blend shapes (20 body + 20 head + 5
+hand components) and drive the **mesh only**: a ±2σ probe moves the 127-joint
+skeleton by `0.00e+00 cm`. Facial expression is set to neutral (zeros) since
+InstantHMR does not regress it.
## Performance tuning
diff --git a/benchmark/README.md b/benchmark/README.md
index f22f02e..bd340f9 100644
--- a/benchmark/README.md
+++ b/benchmark/README.md
@@ -82,6 +82,39 @@ comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits no
per-keypoint confidence, so every keypoint is scored 1.0 — read AP next to the
confidence-free `mean_OKS` and `PCK` numbers.
+### The GT-joint caveat — READ THIS BEFORE QUOTING A 3DPW NUMBER
+
+**The ground truth this harness uses is not the one 3DPW papers report on.**
+`benchlib/threedpw.py` reads `jointPositions` straight from the sequence
+pickles: the 24 **SMPL kinematic-tree** joints. Every published 3DPW number —
+NLF's included ("obtained through the same Human3.6M-style joint regressor that
+all prior works use") — instead runs SMPL *forward* on the GT `poses` / `betas`
+and applies the H36M joint regressor to the resulting 6890 vertices. Those are
+different landmarks, most visibly at the shoulders and hips, which is 4 of the
+12 joints in J12.
+
+That is what makes this harness cheap to set up (see "Why these two" above) and
+it is fine for *tracking a run*. It is **not** comparable to a published table.
+
+Fixing it needs two files that are not in this repo:
+
+| file | where |
+|---|---|
+| `SMPL_{MALE,FEMALE,NEUTRAL}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so get the gendered pair. |
+| `J_regressor_h36m.npy` | ships inside SPIN / HMR2.0 / 4D-Humans |
+
+Then: `V = SMPL(poses, betas)` per frame, `J14 = J_regressor @ V`, and fit the
+existing `--fit-adapter` linear MHR70 -> J14 map on 3DPW **train** against that
+GT before reporting on test. A forward pass, not a fit.
+
+`v_template_clothed` in the pickles cannot substitute — it is a static T-pose
+surface with no blend weights, no pose blendshapes and no joint regressor, so it
+cannot be posed.
+
+This does **not** affect `instanthmr_distill_train/val3dpw.py`, which uses the
+same GT for checkpoint selection: selection only needs a consistent metric, not
+a comparable one.
+
### The joint-convention caveat — do not skip this
InstantHMR predicts MHR70 joints; 3DPW's GT is SMPL. The two rigs place several
diff --git a/docs/architecture.md b/docs/architecture.md
index 21613f9..e297f39 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -1,9 +1,18 @@
# Model architecture
-InstantHMR is a 17 M-parameter network that mimics
-`facebook/sam-3d-body-dinov3` on the **per-person 3D pose estimation** task,
-running ~80× faster on a single GPU and exporting cleanly to ONNX / TFLite /
-QNN for mobile and edge deployment.
+InstantHMR mimics `facebook/sam-3d-body-dinov3` on the **per-person 3D pose
+estimation** task, running ~80× faster on a single GPU and exporting cleanly to
+ONNX / TFLite / QNN for mobile and edge deployment.
+
+> **This page describes the released 17 M checkpoint, not what the trainer
+> builds today.** `train_distill_mhr_only.py`'s `DistillConfig` defaults to
+> `repvit_m2_3` at `d_model=512`, 8 heads, 4 decoder layers, and **71** query
+> tokens (1 global + 70 for the 2D SimCC head) — **39.96 M parameters**:
+> 22.40 M backbone, 16.82 M decoder. The per-part joint queries in the table
+> below are gone; every parameter output now comes from one
+> `Linear(512 → 252)` on the single global token, and the 3D keypoints are
+> derived from the MHR forward pass rather than predicted. Read the config, not
+> this page, before quoting a parameter count.
## High level
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index 3537c41..6e646cc 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -55,6 +55,46 @@ Two things to know when reading v2 results:
so the term stops being decorative). Sweep it first if v2 underperforms:
`--preset v2 --w_reproj 0.1`.
+## The `--losses` switch
+
+**Orthogonal to `--preset`.** `--preset` controls the input pipeline and the
+absolute-pose terms; `--losses {legacy,rebalanced}` controls the loss budget.
+`legacy` is the default and is bit-identical to the pre-rebalance trainer, so
+b2 / v2 runs already in flight stay valid controls. The four arms are
+`baseline`, `v2`, `baseline + rebalanced`, `v2 + rebalanced`.
+
+`rebalanced` (see `apply_rebalanced_losses` in `train_distill_mhr_only.py`)
+comes from two measurements on a trained checkpoint, both reproducible from the
+repo root against `data/sam3d_gt_coco`:
+
+- **an oracle ablation** — replace one predicted group with the teacher's value
+ and re-measure. Of the 37.8 mm of removable PA-MPJPE, `pose[6:136]` (the 130
+ local joint angles) accounts for **34.3 mm**; the bone scales, the root and
+ the 45 identity blendshapes account for **0.00 mm each**.
+- **a per-term gradient norm** — back-propagate each term alone. `loss_shape`
+ owned 27% of the update direction and `loss_cam` 15%, against `loss_pose`'s
+ 4.7% on 21% of the batch.
+
+| item | change | why |
+|---|---|---|
+| shape | `w_shape` 1.0 -> 0.03 | Driving MHR params 204–248 to ±2σ moves the 127-joint skeleton by exactly `0.00e+00` cm, and `MHRForwardPass.get_joints` substitutes zeros for `shape_params` anyway. The blendshapes drive the mesh only. Kept non-zero so the deployed mesh identity stays supervised. |
+| pose mask | `pose_split` | `loss_pose` was masked to `m_ident` (21% of the batch) because the dataset rotates the labels. Model params 3/4/5 map to joint-parameter slots 10/11/12 and nothing else, so `pose[6:136]` is exactly invariant to the in-plane rotation and only needs the flip mask — 21% -> 60% coverage on the parameters that carry all of the error. Root stays on `m_ident`; the two halves keep their per-parameter weight, so this changes the mask and not the root-vs-local balance. |
+| pose beta | `pose_beta` 1.0 -> 0.05 | SmoothL1's beta is in radians here, so 1.0 is ~10x the p95 error and the term is a scaled MSE. |
+| 3D loss | `kp3d_loss` `euclid`, `w_keypoints3d` 2.0 -> 0.35 | SmoothL1's beta of 1.0 is one **metre**, so at a 37.7 mm mean / 119 mm p95 error every sample is in the quadratic branch: 16x less gradient than L1 and 31x less than NLF's unsquared Euclidean at the same weight. The weight drops because the form under it changed. |
+| fingers | `finger_weight` 0.2 | 40 of the 70 keypoints are finger joints and they carry 61% of `loss_3d_native`'s mass, while J12/J14 contain none of them and SA-1B — 55% of the batch under `--mix sqrt` — records hands as 8% observed. Applied to `loss_3d_native` and `loss_reproj`, the two FK-derived geometric terms; the 2D head's own terms are untouched. |
+
+**The unsquared Euclidean is gentler where divergence happens, not harsher.**
+`d‖x‖/dx` is a unit vector, so its gradient is flat in the error, where
+SmoothL1(beta=1) grows with it. Measured against the form it replaces, at the
+weights above: **0.62x** the gradient on a freshly initialised model (515 mm
+error) and 6.4x once trained (60 mm). That is the opposite of the shape that
+produced the LR blow-ups in `datasets_pipeline/jeanzay/STATUS.md`.
+
+Validation also gains `Mesh_PA_MPJPE_body` / `Mesh_MPJPE_body` over the 30
+non-finger keypoints, reported next to the existing metrics. **Selection is
+unchanged** — and with `--val-3dpw` on, both are replaced by the 3DPW J12
+numbers anyway.
+
## The pair index
`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` per
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 83a3179..c99a919 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -440,7 +440,8 @@ def unwrap(m):
@torch.no_grad()
def run_validation(eval_model, loader, criterion, cfg, mhr_module, device):
eval_model.eval()
- acc = {"loss": 0.0, "Mesh_MPJPE": 0.0, "Mesh_PA_MPJPE": 0.0}
+ acc = {"loss": 0.0, "Mesh_MPJPE": 0.0, "Mesh_PA_MPJPE": 0.0,
+ "Mesh_PA_MPJPE_body": 0.0}
n = 0
for batch in loader:
batch = {k: (v.to(device, non_blocking=True) if isinstance(v, torch.Tensor) else v)
@@ -456,6 +457,10 @@ def run_validation(eval_model, loader, criterion, cfg, mhr_module, device):
acc["loss"] += lv
acc["Mesh_MPJPE"] += bm.get("Mesh_MPJPE", 0.0)
acc["Mesh_PA_MPJPE"] += bm.get("Mesh_PA_MPJPE", 0.0)
+ # Reported only. Mesh_PA_MPJPE is 60% fingers by keypoint count; this is
+ # the 30 body/face/foot keypoints. Selection is untouched — and when
+ # --val-3dpw is on, both of these are replaced by the 3DPW J12 numbers.
+ acc["Mesh_PA_MPJPE_body"] += bm.get("Mesh_PA_MPJPE_body", 0.0)
n += 1
out = {k: v / max(n, 1) for k, v in acc.items()}
out["n"] = float(n)
@@ -618,6 +623,10 @@ def train(args, cfg, rank, local_rank, world):
"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,
"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,
"jpeg_p": cfg.jpeg_p, "cam_loss": cfg.cam_loss,
+ "losses": args.losses, "w_shape": cfg.w_shape,
+ "w_keypoints3d": cfg.w_keypoints3d, "kp3d_loss": cfg.kp3d_loss,
+ "pose_split": cfg.pose_split, "pose_beta": cfg.pose_beta,
+ "finger_weight": cfg.finger_weight,
"started": time.strftime("%Y-%m-%d %H:%M")})
for epoch in range(start_epoch, args.epochs):
@@ -694,7 +703,11 @@ def train(args, cfg, rank, local_rank, world):
run["simcc"] += losses.get("loss_2d_simcc", torch.zeros(())).item()
run["reproj"] += losses.get("loss_reproj", torch.zeros(())).item()
run["cam"] += losses.get("loss_cam", torch.zeros(())).item()
- run["pose"] += losses.get("loss_pose", torch.zeros(())).item()
+ # Under cfg.pose_split, loss_pose is the local half and loss_pose_root
+ # the other; summing keeps history.jsonl's "pose" column comparable
+ # against a --losses legacy run.
+ run["pose"] += (losses.get("loss_pose", torch.zeros(())).item()
+ + losses.get("loss_pose_root", torch.zeros(())).item())
nb += 1
if is_main() and i % args.log_every == 0:
log(f" e{epoch+1}/{args.epochs} step {i}/{steps_per_epoch} "
@@ -733,9 +746,11 @@ def train(args, cfg, rank, local_rank, world):
f"reproj {tr['reproj']:.4f} | cam {tr['cam']:.4f} "
f"| skipped {n_skip}/{steps_per_epoch} | {time.time() - t_epoch:.0f}s")
log(f" valRAW loss {raw['loss']:.4f} | PA-MPJPE {raw['Mesh_PA_MPJPE']:.1f} | "
- f"MPJPE {raw['Mesh_MPJPE']:.1f} mm")
+ f"MPJPE {raw['Mesh_MPJPE']:.1f} mm | held-in body PA "
+ f"{raw['Mesh_PA_MPJPE_body']:.1f} mm")
log(f" valEMA loss {em['loss']:.4f} | PA-MPJPE {em['Mesh_PA_MPJPE']:.1f} | "
- f"MPJPE {em['Mesh_MPJPE']:.1f} mm")
+ f"MPJPE {em['Mesh_MPJPE']:.1f} mm | held-in body PA "
+ f"{em['Mesh_PA_MPJPE_body']:.1f} mm")
if dpw_raw is not None:
log(f" 3DPW-{args.val_3dpw_split} RAW J12 PA {dpw_raw['J12_PA_MPJPE']:.1f} "
f"MPJPE {dpw_raw['J12_MPJPE']:.1f} | J14 PA {dpw_raw['J14_PA_MPJPE']:.1f} mm")
@@ -878,6 +893,14 @@ def parse_args():
"prerequisites, plus absolute-pose supervision. "
"'baseline' reproduces the pre-v2 trainer exactly.")
p.add_argument("--w_reproj", type=float, default=None)
+ p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",
+ help="Orthogonal to --preset, which only controls the input "
+ "pipeline and the absolute-pose terms. 'rebalanced' applies "
+ "the four loss-budget fixes (w_shape 1.0->0.03, pose split "
+ "onto the non-flipped 60%% of the batch at beta 0.05, "
+ "unsquared-Euclidean 3D loss at w 0.35, finger keypoints at "
+ "0.2). 'legacy' reproduces the pre-rebalance recipe bit for "
+ "bit, so b2/v2 runs in flight stay valid controls.")
p.add_argument("--index-cache", default=None,
help="Where to cache the (image, npz) pair index. "
"Default: <data_root>/.pair_index_v2.pkl")
@@ -904,15 +927,19 @@ def build_cfg(args):
cfg.epochs = args.epochs
cfg.per_dataset_caps = {} # replaced by the weighted sampler
if args.backbone: cfg.backbone = args.backbone
- if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3d
if args.w_simcc is not None: cfg.w_simcc = args.w_simcc
if args.weight_decay is not None: cfg.weight_decay = args.weight_decay
if args.ema_decay is not None: cfg.ema_decay = args.ema_decay
if args.preset == "v2":
apply_v2_preset(cfg)
- # An explicit --w_reproj still wins over the preset.
+ # Orthogonal axis: --preset is the input pipeline + absolute-pose terms,
+ # --losses is the loss budget. Applied second so it owns w_keypoints3d.
+ if args.losses == "rebalanced":
+ T.apply_rebalanced_losses(cfg)
+ # An explicit --w_reproj / --w_keypoints3d still wins over both.
if args.w_reproj is not None: cfg.w_reproj = args.w_reproj
+ if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3d
return cfg
@@ -998,6 +1025,11 @@ def main():
f"occl_p={cfg.occl_p} jpeg_p={cfg.jpeg_p} "
f"w_reproj={cfg.w_reproj} cam={cfg.cam_loss}"
if args.preset != "baseline" else ""))
+ log(f"losses {args.losses}"
+ + (f" | w_shape={cfg.w_shape} pose_split={cfg.pose_split} "
+ f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "
+ f"finger_w={cfg.finger_weight}"
+ if args.losses != "legacy" else ""))
log(f"seed {args.seed} | mix {args.mix} | "
f"samples/epoch {args.samples_per_epoch:,}")
log("=" * 66)
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 72a05ab..1d568eb 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -121,6 +121,31 @@ def _build_flip_perm(names):
FLIP_PERM = _build_flip_perm(JOINT_NAMES)
+# 40 of the 70 keypoints are finger joints. Measured on a trained checkpoint,
+# they carry 61.0% of loss_3d_native's L1 mass and 60% of Mesh_PA_MPJPE, while
+# the published metrics (J12 / J14) contain none of them and SA-1B — 55% of the
+# batch under `--mix sqrt` — records hands as 8% observed. `finger_weight`
+# down-weights them in the FK-derived geometric losses; see DistillConfig.
+FINGER_KP = np.array(
+ [i for i, n in enumerate(JOINT_NAMES)
+ if any(t in n for t in ("thumb", "index", "middle", "ring", "pinky"))],
+ dtype=np.int64)
+BODY_KP = np.array([i for i in range(len(JOINT_NAMES)) if i not in set(FINGER_KP.tolist())],
+ dtype=np.int64)
+assert len(FINGER_KP) == 40 and len(BODY_KP) == 30
+
+# MHR model-parameter layout, read off character_torch.parameter_transform:
+# 0:3 root_tx/ty/tz (always 0 in the corpus)
+# 3:6 root_rx/ry/rz -> joint-parameter slots 10/11/12 and nothing else
+# 6:136 130 local joint angles
+# 136:204 68 bone scales
+# Because the root rotation is exclusive to slots 10/11/12, the 130 local angles
+# are EXACTLY invariant to the in-plane rotation the dataset applies, which is
+# what makes `pose_split` valid. Verified: recomposing the root as
+# R_aug @ R_root in extrinsic-xyz Euler reproduces the rotated skeleton up to a
+# pure translation (the pelvis-vs-origin pivot offset), not a pose change.
+POSE_ROOT_DIM = 6
+
# ============================================================
# Cell 1 — Configuration
# ============================================================
@@ -166,6 +191,36 @@ class DistillConfig:
reproj_all_samples: bool = False # include flipped samples in loss_reproj
cam_loss: str = "mse" # "mse" | "euclid" (unsquared L2 on xyz)
+ # --- loss-rebalance knobs (`--losses rebalanced` in train_distill_jz.py).
+ # --- Every default below reproduces the pre-rebalance behaviour bit for bit,
+ # --- so `--preset baseline` and `--preset v2` are unaffected. All four come
+ # --- from the measured error/gradient budget:
+ # * an oracle ablation attributes 34.3 of the 37.8 mm of removable
+ # PA-MPJPE to pose[6:136] and 0.00 mm to shape_params, the bone scales
+ # and the root, yet loss_shape owns 27% of the gradient and loss_pose
+ # 4.7% on 21% of the batch;
+ # * SmoothL1's default beta=1.0 is a metre, so on metre-scale coordinates
+ # every sample sits in the quadratic branch -- 16x less gradient than L1
+ # and 31x less than NLF's unsquared Euclidean, at the same weight.
+ pose_split: bool = False # supervise pose[6:136] on the 60% of the batch
+ # that is not flipped instead of the 21% that is
+ # un-augmented. Root stays on m_ident. The two
+ # halves keep their per-parameter weight (6/136
+ # and 130/136 of w_pose), so this changes the
+ # MASK, not the root-vs-local balance.
+ pose_beta: float = 1.0 # SmoothL1 beta for the pose terms, in radians.
+ # 1.0 rad is ~10x the p95 error, i.e. pure MSE.
+ kp3d_loss: str = "smooth_l1" # "smooth_l1" | "euclid" (NLF 2407.07532 §3.2:
+ # "the Euclidean loss without squaring, for
+ # better outlier-robustness"). euclid has a
+ # FLAT gradient, so it is *gentler* than the
+ # current form at init (measured 0.62x on a
+ # fresh model, 515 mm error) and stronger at
+ # convergence (6.4x at 60 mm) -- the opposite
+ # of the shape that caused the LR divergences.
+ finger_weight: float = 1.0 # per-keypoint weight on FINGER_KP inside
+ # loss_3d_native and loss_reproj
+
max_consecutive_skips: int = 200 # abort rather than burn the allocation
# on a run that has already diverged
@@ -253,6 +308,43 @@ class DistillConfig:
native_mapping_ids = [ 1, 2, 5, 6, 9, 10, 11, 12, 13, 14, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62 ]
mhr_mapping_ids = [ 125, 123, 75, 39, 2, 18, 3, 19, 5, 21, 64, 63, 62, 61, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 41, 100, 99, 98, 97, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 77 ]
+def apply_rebalanced_losses(cfg) -> None:
+ """The four loss-budget fixes, as one switch. Lives here, not in
+ train_distill_jz.py, so the single-GPU trainer and the DDP driver cannot
+ drift apart.
+
+ Every number is chosen from a measured unit-weight gradient norm on real
+ augmented batches, not from taste. The resulting split of the update
+ direction (batch 12, epoch-78 checkpoint, COCO):
+
+ term before after
+ loss_3d_native 8.5% 30.3%
+ loss_2d_simcc 29.6% 26.4%
+ loss_pose 5.6% 20.2%
+ loss_cam 13.8% 12.3%
+ loss_mhr_joints 8.5% 7.6%
+ loss_shape 14.4% 0.4%
+
+ `w_keypoints3d` drops 2.0 -> 0.35 because the loss form under it changed:
+ at equal weight the unsquared Euclidean carries 31x the gradient of
+ SmoothL1(beta=1) at the operating point, so keeping 2.0 would put the
+ FK path at ~20x its current strength -- the failure mode the w=10
+ experiments in this file's header already found. 0.35 lands it at ~3x,
+ leading without dominating.
+ """
+ cfg.w_shape = 0.03 # 45 identity blendshapes. Verified: driving MHR
+ # params 204..248 to +/-2 sigma moves the 127-joint
+ # skeleton by 0.00e+00 cm, and get_joints() zeroes
+ # them before the forward pass anyway. Non-zero so
+ # the deployed mesh identity stays supervised.
+ cfg.pose_split = True # pose[6:136] on m_noflip (60%) not m_ident (21%)
+ cfg.pose_beta = 0.05 # ~= the p50 pose-parameter error, so the term is
+ # linear over the bulk of the distribution
+ cfg.kp3d_loss = "euclid" # NLF 2407.07532 §3.2
+ cfg.w_keypoints3d = 0.35 # re-scaled for the new form; see docstring
+ cfg.finger_weight = 0.2 # 40 of 70 keypoints, none of them in J12/J14
+
+
# ============================================================
# Cell 3 — Dataset and Data Augmentation
# ============================================================
@@ -799,6 +891,17 @@ class DistillationLoss(nn.Module):
self.simcc_sigma = 2.0 * bin_w
self._global_step = 0
self.warmup_steps = int(getattr(cfg, "kp3d_warmup_steps", 0))
+ # Per-keypoint weight for the FK-derived geometric losses. All ones
+ # unless finger_weight != 1, so the unweighted path stays bit-identical.
+ #
+ # Built on the MHR module's device, NOT the CPU default: every call site
+ # in this repo constructs DistillationLoss without .to(device)
+ # (run_overfit_test, run_self_tests, train_instant_hmr and
+ # train_distill_jz.train all do), so a CPU buffer would raise on the
+ # first step. register_buffer still lets an explicit .to() move it.
+ w = torch.ones(cfg.num_joints, device=getattr(mhr_module, "device", "cpu"))
+ w[torch.from_numpy(FINGER_KP).to(w.device)] = float(getattr(cfg, "finger_weight", 1.0))
+ self.register_buffer("kp3d_weight", w, persistent=False)
def set_step(self, step: int):
"""Drives the FK-loss warm-up ramp. Call once per optimiser step."""
@@ -819,6 +922,12 @@ class DistillationLoss(nn.Module):
e = F.smooth_l1_loss(pred, tgt, reduction='none').flatten(1).mean(1)
return self._mmean(e, mask)
+ def _m_beta(self, pred, tgt, mask, beta):
+ """m_smooth_l1 with an explicit beta. beta=1.0 is F.smooth_l1_loss's
+ default, so this reproduces m_smooth_l1 exactly at the default."""
+ e = F.smooth_l1_loss(pred, tgt, reduction='none', beta=beta).flatten(1).mean(1)
+ return self._mmean(e, mask)
+
def m_mse(self, pred, tgt, mask):
e = F.mse_loss(pred, tgt, reduction='none').flatten(1).mean(1)
return self._mmean(e, mask)
@@ -897,7 +1006,26 @@ class DistillationLoss(nn.Module):
tgt_pose = targets["mhr_model_params"][:, :idx_pose]
tgt_scale = targets["mhr_model_params"][:, idx_pose:]
- losses['loss_pose'] = self.m_smooth_l1(pred_pose, tgt_pose, m_ident) * self.cfg.w_pose
+ # loss_pose. Masked to m_ident (~21% of the batch at geom_p 0.8) because
+ # the dataset rotates the LABELS and the target parameters are not
+ # rotated with them. That is only true of the 6 root parameters:
+ # model params 3/4/5 drive joint-parameter slots 10/11/12 exclusively,
+ # so pose[6:136] -- the 130 local angles, which the oracle ablation
+ # attributes 100% of the PA-MPJPE error to -- is invariant to the
+ # in-plane rotation and only needs the flip mask. See POSE_ROOT_DIM.
+ beta = float(getattr(self.cfg, "pose_beta", 1.0))
+ if getattr(self.cfg, "pose_split", False):
+ r = POSE_ROOT_DIM
+ n = pred_pose.shape[1]
+ # Proportional weights: the split changes the MASK, not the
+ # root-vs-local balance a single mean over n parameters implied.
+ losses['loss_pose_root'] = self._m_beta(
+ pred_pose[:, :r], tgt_pose[:, :r], m_ident, 1.0) * self.cfg.w_pose * (r / n)
+ losses['loss_pose'] = self._m_beta(
+ pred_pose[:, r:], tgt_pose[:, r:], m_noflip, beta) * self.cfg.w_pose * ((n - r) / n)
+ else:
+ losses['loss_pose'] = self._m_beta(
+ pred_pose, tgt_pose, m_ident, beta) * self.cfg.w_pose
losses['loss_scale'] = self.m_mse(pred_scale, tgt_scale, m_noflip) * self.cfg.w_scale
losses['loss_shape'] = self.m_mse(preds["shape_params"], targets["shape_params"], m_noflip) * self.cfg.w_shape
# NLF applies its translation loss as an unsquared Euclidean norm over the
@@ -980,7 +1108,22 @@ class DistillationLoss(nn.Module):
# skeleton is mirror-symmetric to 0.7 mm, so a mirrored pose is a perfectly
# representable MHR pose. (loss_pose / loss_scale / loss_shape stay masked
# because the TARGET PARAMETERS are not mirrored — only the geometry is.)
- losses['loss_3d_native'] = self.l1(pred_kp3d, tgt_3d) * self.cfg.w_keypoints3d * self.fk_scale
+ #
+ # SmoothL1's beta defaults to 1.0 -- one METRE here -- so every sample of
+ # a 37.7 mm mean / 119 mm p95 error distribution sits in the quadratic
+ # branch and the term behaves as a scaled MSE: measured 16x less gradient
+ # than L1 and 31x less than the unsquared Euclidean at the same weight,
+ # which is why w_keypoints3d=2.0 delivered 7.7% of the update direction.
+ # "euclid" also has a FLAT gradient (d||x||/dx is a unit vector), so it is
+ # gentler than the current form where the error is large -- 0.62x on a
+ # freshly initialised model -- and only stronger once the model is close.
+ if getattr(self.cfg, "kp3d_loss", "smooth_l1") == "euclid":
+ e_kp3d = (pred_kp3d - tgt_3d).norm(dim=-1) # (B, 70), metres
+ losses['loss_3d_native'] = (
+ self._kp_weighted(e_kp3d, self.kp3d_weight.expand_as(e_kp3d))
+ * self.cfg.w_keypoints3d * self.fk_scale)
+ else:
+ losses['loss_3d_native'] = self.l1(pred_kp3d, tgt_3d) * self.cfg.w_keypoints3d * self.fk_scale
cc = targets["aug_cos"].view(-1, 1)
ss = targets["aug_sin"].view(-1, 1)
@@ -1020,8 +1163,13 @@ class DistillationLoss(nn.Module):
m_reproj = ones
pred_reproj = self._project_3d_to_norm_2d(pts_cr, cam_cr_r, targets)
e_reproj = (pred_reproj - tgt_2d).abs().mean(-1)
+ # Same finger down-weight as loss_3d_native: this is the other geometric
+ # term over all 70 keypoints, and at --preset v2's w_reproj = 0.5 it would
+ # otherwise also be ~60% fingers. The 2D head's own terms are left alone —
+ # joints_2d is a deployed output and SA-1B's visibility mask already
+ # handles its unobserved hands.
losses['loss_reproj'] = self._kp_weighted(
- e_reproj, w_kp, m_reproj) * self.cfg.w_reproj * self.fk_scale
+ e_reproj, w_kp * self.kp3d_weight, m_reproj) * self.cfg.w_reproj * self.fk_scale
losses['total_loss'] = sum(losses.values())
return losses
@@ -1111,6 +1259,18 @@ def evaluate_hmr_batch(preds, targets, cfg, mhr_module):
metrics['Mesh_PA_MPJPE'] = torch.linalg.norm(
pred_aligned - gt_3d_native, dim=-1).mean().item() * 1000.0
+ # Body-only variant over the 30 non-finger keypoints. Mesh_PA_MPJPE is 60%
+ # fingers by keypoint count, so on the single-GPU path it selects largely on
+ # hands; the published J12/J14 sets contain none of them. Reported alongside,
+ # never substituted — selection behaviour is unchanged.
+ b = torch.from_numpy(BODY_KP).to(pred_kp3d.device)
+ pb, gb = pred_kp3d[:, b], gt_3d_native[:, b]
+ metrics['Mesh_MPJPE_body'] = torch.linalg.norm(
+ (pb - pb.mean(1, keepdim=True)) - (gb - gb.mean(1, keepdim=True)),
+ dim=-1).mean().item() * 1000.0
+ metrics['Mesh_PA_MPJPE_body'] = torch.linalg.norm(
+ batched_procrustes_alignment(pb, gb) - gb, dim=-1).mean().item() * 1000.0
+
if hasattr(cfg, 'native_mapping_ids') and hasattr(cfg, 'mhr_mapping_ids'):
pred_mesh_anchors = pred_mhr_vision[:, cfg.mhr_mapping_ids, :]
gt_mesh_anchors = gt_3d_native[:, cfg.native_mapping_ids, :]
@@ -1360,7 +1520,8 @@ def train_instant_hmr():
tm['2d'] += losses.get('loss_2d_native', torch.tensor(0)).item()
tm['simcc'] += losses.get('loss_2d_simcc', torch.tensor(0)).item()
tm['3d'] += losses.get('loss_3d_native', torch.tensor(0)).item()
- tm['pose'] += losses.get('loss_pose', torch.tensor(0)).item()
+ tm['pose'] += (losses.get('loss_pose', torch.tensor(0)).item()
+ + losses.get('loss_pose_root', torch.tensor(0)).item())
tm['scale'] += losses.get('loss_scale', torch.tensor(0)).item()
tm['mhr'] += losses.get('loss_mhr_joints', torch.tensor(0)).item()
tm['reproj'] += losses.get('loss_reproj', torch.tensor(0)).item()
@@ -1804,8 +1965,11 @@ def run_overfit_test(steps=3000, subset=8, lr=5e-4):
batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}
def reg(ls):
+ # loss_pose_root only exists under cfg.pose_split; the .get keeps the
+ # regression figure comparable between the two settings.
return (ls['loss_2d_native'].item() + ls['loss_3d_native'].item()
- + ls['loss_cam'].item() + ls['loss_pose'].item())
+ + ls['loss_cam'].item() + ls['loss_pose'].item()
+ + ls.get('loss_pose_root', torch.zeros(())).item())
first_reg = last_reg = last_pa = None
best_pa = float('inf') # 8-sample overfit is noisy at constant LR; judge the best fit reached
@@ -1855,6 +2019,7 @@ def run_overfit_test(steps=3000, subset=8, lr=5e-4):
f"| shape {losses.get('loss_shape', torch.tensor(0)).item():.4f} "
f"| mhr {losses['loss_mhr_joints'].item():.4f} "
f"| Mesh [MPJPE: {m['Mesh_MPJPE']:.1f} | PA: {m['Mesh_PA_MPJPE']:.1f}] mm "
+ f"| body PA: {m['Mesh_PA_MPJPE_body']:.1f} mm "
f"| Mesh52 [MPJPE: {m['Mesh52_MPJPE']:.1f} | PA: {m['Mesh52_PA_MPJPE']:.1f}] mm")
# THRESHOLD CHANGED, 40.0 -> 55.0 mm, and this is deliberate — the quantity
@@ -1990,6 +2155,11 @@ def parse_args():
help="Optimiser steps over which the FK-path losses ramp from 0 to full weight.")
p.add_argument("--w_keypoints3d", type=float, default=None,
help="Weight of the MHR-derived 3D keypoint loss (FK path).")
+ p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",
+ help="'rebalanced' applies the four loss-budget fixes "
+ "(w_shape down, pose split + beta, unsquared-Euclidean 3D "
+ "loss, finger down-weight). 'legacy' reproduces the "
+ "pre-rebalance recipe bit for bit. See apply_rebalanced_losses.")
p.add_argument("--seed", type=int, default=None,
help="Seed torch/numpy/random (for reproducible sanity runs).")
p.add_argument("--split-wd", dest="split_wd", action="store_true",
@@ -2035,6 +2205,8 @@ def main():
cfg.per_dataset_caps["sam3d_gt_harmony4d"] = args.harmony4d_cap
if args.no_resume: cfg.resume = False
if args.kp3d_warmup_steps is not None: cfg.kp3d_warmup_steps = args.kp3d_warmup_steps
+ if args.losses == "rebalanced": apply_rebalanced_losses(cfg)
+ # An explicit --w_keypoints3d still wins over the preset.
if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3d
if args.split_wd: cfg.split_weight_decay = True
if args.backbone is not None: