Python
fix: bound MHR body-size outputs and repair anomalous batches instead of discarding them
mohamdeve01030293fc3
diff --git a/CLAUDE.md b/CLAUDE.mdindex 767b58a..497a53a 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -16,6 +16,35 @@ Keep commit messages to one or two short sentences.---+## 0.5 Write for a human, not a compiler++**The user is a person reading a terminal, not a log parser.** Answers that are+technically correct and unreadable are failures. This has come up more than+once — take it seriously.++- **Define every name before you use it.** `scale_lo`, `m_noflip`, `fk_scale`,+ `n_skip_run` mean nothing to someone who has not just read that function. If+ a variable, flag or file has to appear, say what it *is* in plain words+ first, once.+- **Give commands in the form the user actually runs them.** This repo launches+ through `datasets_pipeline/jeanzay/52_train_ddp.slurm` with `RUN_NAME=... MIX=...+ EXTRA_TRAIN_ARGS=... sbatch`. Read the launcher and match it. A bare `srun+ python ...` line the user has never typed is not an answer.+- **Say what to run, in what order, and what happens next.** If a command is+ optional, say so. If you offer a check, say what a good result looks like.+- **Lead with the answer.** The finding first, the evidence after. Do not make+ the user read a derivation to learn whether it worked.+- **Explain the mechanism, don't just name it.** "The bone scales enter the+ kinematics multiplicatively, so +10 makes a 336 m skeleton" teaches+ something; "unbounded scale parameters caused divergence" does not.+- **Cut the ceremony.** No restating the request, no listing what you are about+ to do, no summary of the summary.++If the user says they cannot follow an answer, that is a defect to fix in the+next answer, not a preference to note.++---+## 1. Think before codingDon't assume. Don't hide confusion. Surface tradeoffs.@@ -113,6 +142,39 @@ augmentation, so it never needed the `m_ident` mask; and the 45 `shape_params`move the 127-joint skeleton by `0.00e+00 cm`, so they are mesh-only and everyloss and metric in the trainer is blind to them.+**The student's MHR outputs are unbounded and 77 of them are body SIZE.** The+head is a plain `nn.Linear`. `0:3` root translation (exactly `0.000e+00` in all+24,000 sampled annotations — the person is placed by `cam_trans`), `130:136`+the `*_length/_width_flexible` parameters, and `136:204` the 68 bone scales all+scale the skeleton, the last **multiplicatively along the kinematic chain**:+uniform +10 gives a 336 m skeleton, +25 gives 28,300 km. That, not the learning+rate, killed four 300-epoch runs at epochs 24-32 on 2026-09-06. Watch for the+two traps: `130:136` *look* like pose parameters (they sit inside `6:136`) but+drive joint **translation** channels, so bounding only `136:204` still reaches+375 m; and `6:130` are genuine rotations that stay at ~1.5 m at any magnitude,+so leave them alone. `--bound-scales` fixes it from+`assets/mhr_size_bounds.npz`; that file is **new and must be rsynced** or the+job dies at startup.++**The anomaly guard deletes the gradient that would fix the anomaly.** A batch+over `anomaly_loss_threshold` was skipped whole — but `loss_scale`/`loss_pose`+in that same batch are the only force pulling the sizes back, so the sample was+quarantined for good. Worse, `bad` is all-reduced with `MAX`: one bad sample in+4x64 = 256 vetoes the step for every rank, so 2% runaway samples stop 99.4% of+steps. Use `--anomaly-safe-fallback`, which steps on the non-FK terms instead+(`safe_loss_subset`, cosine +1.0000 with `teacher - prediction`). Never "fix" a+divergence here by lowering the LR alone: `OneCycleLR(pct_start=0.1)` peaks at+10% of the run, so a lower peak only moves *when* it dies.++**`parameter_transform[:, :204]` has rank 192, not 204.** Twelve directions+move the parameters and leave the skeleton *exactly* unchanged — each+`*_flexible` parameter against its `scale_*` partner, plus the spine rotations.+Along them every geometric loss has zero gradient, so the model is regressing+12 dimensions of solver-arbitrary teacher noise that no image can determine.+Do **not** "fix" this by zeroing the `*_flexible` parameters even though the rig+pins them to `[0,0]`: the teacher genuinely uses them (`leg_length_flexible`+mean 0.32, max 1.64), so zeroing shortens every leg.+**`DistillationLoss` is never `.to(device)`'d.** `run_overfit_test`,`run_self_tests`, `train_instant_hmr` and `train_distill_jz.train` all constructit bare, so any buffer you register lands on the CPU and raises on step 1. Build@@ -146,6 +208,13 @@ mandatory for new runs, because the rebuilt corpus carries real per-imagefocals spanning `f/diag` 0.6-1.8 where the pixel form is ambiguous; the oldcorpus was constant at 1.000. See `instanthmr_distill_train/README.md`.+**One reported number: J14 PA-MPJPE, adapter-applied, on the H36M GT.** The+training log, `summarize_runs.py` and `benchmark/eval_3dpw_ckpt.py` all print+it and they agree to the decimal. J12 was removed on 2026-09-06 — it was the+rig-clean row against the old GT and is not against this one. Runs started+before that date recorded J14 *without* the adapter, ~26 mm higher; do not put+old and new rows in one table.+**PA-MPJPE is blind to half of what a demo shows.** Procrustes removestranslation, rotation and scale, so `cam_trans` drift, body-size error andtemporal jitter are all invisible to it — and to root-relative MPJPE for thediff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.mdindex 73002d9..cffe86e 100644--- a/instanthmr_distill_train/README.md+++ b/instanthmr_distill_train/README.md@@ -161,6 +161,134 @@ nothing; it silently mis-places the person in depth. `run_config.json` recordsThe default is `False` and is bit-identical to the pre-flag trainer, verifiedover augmented samples tensor by tensor.+## `--bound-scales` / `--anomaly-safe-fallback` (added 2026-09-06, mandatory)++**Use both on every new run.** They fix the divergence that killed all four+runs of the 2026-09-06 sweep at epochs 24-32 (`fno_s*`, `fsa_s*`), and every+earlier sweep that was "fixed" by lowering the peak LR.++### What was actually wrong++`head_global` is a plain `nn.Linear`. 77 of its outputs are **body-size**+parameters that the MHR forward kinematics multiplies along the kinematic+chain, and nothing bounded them:++| block | what it is | unbounded behaviour (measured on `checkpoints/mhr_model.pt`) |+|---|---|---|+| `0:3` | root translation | linear; +100 -> 22 m skeleton. Teacher value is **exactly 0.000e+00** in all 24,000 sampled annotations across all six splits — the person is placed by `cam_trans`. |+| `130:136` | `*_length/_width_flexible` | linear, transform weights up to 10.0; +100 -> 32 m |+| `136:204` | the 68 bone scales | **multiplicative**: +10 -> 336 m, +20 -> 703 km, +25 -> 28,300 km |++Note `130:136`. Those six sit at the end of the *pose* block but they drive+joint **translation** channels, not rotations, so they are size parameters.+Bounding only `136:204` still let a blown-up head reach a 375 m skeleton.+`6:130` are genuine rotations and stay at ~1.5 m at any magnitude — leave them+alone.++Loss on real teacher targets, with the head drifted by `d` on the size block:++| drift | total loss, unbounded | total loss, `--bound-scales` |+|---|---|---|+| 0 | 6.5 | 6.5 |+| 10 | **140** (over `anomaly_loss_threshold=100`) | 15.6 |+| 20 | **2.2e5** | 25.1 |++That is where the `4.06e7` / `8.76e13` losses in the dead logs come from.++### Why the guard made it permanent rather than catching it++An anomalous batch was discarded whole. But `loss_scale` / `loss_pose` in that+same batch are small and point straight back at the teacher — they are the+**only** force pulling the sizes out of the runaway region — so skipping+removed the fix along with the fault and quarantined those samples for good.+Two amplifiers turned that into a dead job:++* `bad` is all-reduced with `MAX`, so **one** bad sample out of 4x64 = 256+ vetoes the step for every rank. At 1% runaway samples 92% of steps are+ vetoed; at 2%, 99.4%. The run aborts when 2% of the data is bad, not the model.+* The EMA rollback (`anomaly_skip_patience`, `max_ema_rollbacks`) existed only+ in `train_distill_mhr_only`'s single-GPU loop. The DDP path had the abort and+ **no recovery at all**.++`--anomaly-safe-fallback` steps on `safe_loss_subset(losses)` — everything+except `FK_LOSS_KEYS` — instead of discarding the batch. Verified: with the+scales sitting +20 off the teacher, that gradient has cosine **+1.0000** with+`teacher - prediction` and moves **100%** of the 68 scales the right way, while+the full loss is 2.9e5. The rollback is now in the DDP loop too.++### Why lowering the LR never worked++`OneCycleLR(pct_start=0.1)` puts the peak at 10% of the run — epoch 30 of 300.+The four dead runs died at epochs 24-32. Lowering `max_lr` does not remove the+peak, it just slows the random walk toward the runaway region.++### The bound++`assets/mhr_size_bounds.npz` holds, per parameter, the union of the rig's own+`character_torch.parameter_limits` and the observed teacher range; the model+widens each by `--scale-bound-margin` (0.5) and squashes with `tanh`. The rig+bounds every one of these to `[-1.1, 1.1]`; the teacher respects that for the+scales but **not** for `130:136`, which the rig pins to `[0,0]` while the+teacher uses them up to 1.64 (`leg_length_flexible`) — hence the union.++Measured over 24,000 annotations from all six splits: **0.000000%** of teacher+targets fall outside the bound and the worst sits at `|atanh| = 0.80`, well+inside tanh's linear region, so the bound costs nothing. With the flag on, a+head deliberately blown up to `bias ~ N(0, 5000)` still yields a 1.58 m+skeleton. Exports to ONNX (matches torch to 1.2e-7).++Regenerate the asset from the rig plus the corpus if either changes; the file+also stores `rig_lo` / `rig_hi` unmodified for reference.++Both flags default to **off** and are bit-identical to the pre-flag trainer,+verified tensor by tensor. `run_config.json` records `bound_scales`,+`scale_bound_margin` and `anomaly_safe_fallback`.++### The 12-dimensional null space (not fixed, know it exists)++`parameter_transform[:, :204]` has **rank 192**. Twelve directions move the+parameters and leave the skeleton *exactly* unchanged — they pair each+`*_flexible` parameter against its `scale_*` partner (`scale_spine_length` vs+`spine_length_flexible`, `scale_hip_width` vs `hip_width_flexible`, and the+spine rotations). Along those, every geometric loss has zero gradient and only+`loss_pose` / `loss_scale` act, so the model is being asked to regress 12+dimensions of solver-arbitrary teacher noise that no image can determine.+Confirmed directly: pushing `scale_spine_length` to 20.3 along a null direction+moves the skeleton by 1.2e-7 m. The bound now keeps that wandering finite;+collapsing the redundancy properly would be a separate change.++## The reported number++One number everywhere: **J14 PA-MPJPE, adapter-applied, against the+published-protocol 3DPW ground truth.** The training log, `summarize_runs.py`,+`benchmark/eval_3dpw_ckpt.py` and a paper results table all print the same+quantity — cross-checked to the decimal (41.71 mm for `b3_s1` on 3DPW test).+J14 MPJPE rides alongside it. J12 is gone: it was the rig-clean row against the+old `jointPositions` GT, and against the H36M one it is not (those hips sit+292 mm apart against the MHR rig's ~120).++The adapter is `benchmark/results/adapter_j14_h36m_teacher.npz`, a fixed 14x70+map fitted from the teacher's labels and independent of any student, so it does+not move between checkpoints. `--val-3dpw-adapter none` scores raw MHR joints+instead; a missing file is a hard error, not a silent fallback.++The epoch log prints two lines because there are **two different datasets**:++```+held-in (teacher agreement) RAW loss 5.09 body PA 153.5 | EMA ...+3DPW-validation RAW J14 PA 126.0 MPJPE 283.0 | EMA J14 PA 125.7 MPJPE 284.0 mm+```++`held-in` is a 2% slice of the **training corpus** scored against the+**teacher's** labels — agreement, not generalisation, and only a training-health+signal. `3DPW-validation` is 12 held-out sequences against real **MoCap**.+Before 2026-09-06 both appeared on one line under a single "val" label, because+`Mesh_PA_MPJPE` is overwritten with the 3DPW number when `--val-3dpw` is on.++**Runs started before 2026-09-06 are not comparable**: their `J14_PA_MPJPE` in+`history.jsonl` is the raw MHR joint set, ~26 mm higher than the adapter-applied+one. `summarize_runs.py` still renders them, in the same column.+## The pair index`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` perdiff --git a/instanthmr_distill_train/assets/mhr_size_bounds.npz b/instanthmr_distill_train/assets/mhr_size_bounds.npznew file mode 100644index 0000000..3a3e87cBinary files /dev/null and b/instanthmr_distill_train/assets/mhr_size_bounds.npz differdiff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.pyindex 66555c7..82af302 100755--- a/instanthmr_distill_train/train_distill_jz.py+++ b/instanthmr_distill_train/train_distill_jz.py@@ -460,7 +460,7 @@ def run_validation(eval_model, loader, criterion, cfg, mhr_module, device):acc["Mesh_PA_MPJPE"] += bm.get("Mesh_PA_MPJPE", 0.0)# Reported only. Mesh_PA_MPJPE is 60% fingers by keypoint count; this is# the 30 body/face/foot keypoints. Selection is untouched — and when- # --val-3dpw is on, both of these are replaced by the 3DPW J12 numbers.+ # --val-3dpw is on, both of these are replaced by the 3DPW J14 numbers.acc["Mesh_PA_MPJPE_body"] += bm.get("Mesh_PA_MPJPE_body", 0.0)n += 1out = {k: v / max(n, 1) for k, v in acc.items()}@@ -486,11 +486,23 @@ def train(args, cfg, rank, local_rank, world):train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)steps_per_epoch = len(train_loader)- dpw_loader = dpw_test_loader = None+ dpw_loader = dpw_test_loader = dpw_adapter = Noneif args.val_3dpw:if not args.val_3dpw_images:raise SystemExit("--val-3dpw needs --val-3dpw-images")from val3dpw import ThreeDPWValSet+ if str(args.val_3dpw_adapter).lower() != "none":+ ap = Path(args.val_3dpw_adapter)+ if not ap.is_absolute():+ ap = Path(__file__).resolve().parent.parent / ap+ if not ap.is_file():+ raise SystemExit(+ f"3DPW adapter not found: {ap}\nIt ships in the repo as "+ "benchmark/results/adapter_j14_h36m_teacher.npz; rsync it, "+ "or pass --val-3dpw-adapter none to score raw MHR joints.")+ dpw_adapter = np.load(ap)["W"]+ log(f"3DPW adapter: {ap.name} {tuple(dpw_adapter.shape)} "+ f"-> J14 numbers match benchmark/eval_3dpw_ckpt.py")dpw = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,split=args.val_3dpw_split, stride=args.val_3dpw_stride,gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal)@@ -501,7 +513,7 @@ def train(args, cfg, rank, local_rank, world):num_workers=max(1, cfg.num_workers // 2), pin_memory=True)log(f"3DPW {args.val_3dpw_split}: {len(dpw):,} person-frames "f"(stride {args.val_3dpw_stride}, GT {args.val_3dpw_gt}) "- f"-> selection metric is J12 PA-MPJPE")+ f"-> selection metric is J14 PA-MPJPE (adapter-applied)")# Optional read-only monitor on the split the paper reports. Selection# NEVER touches it: picking the best of ~100 epochs on the test split@@ -633,8 +645,12 @@ def train(args, cfg, rank, local_rank, world):"finger_weight": cfg.finger_weight,"val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,"cliff_focal": cfg.cliff_focal,+ "bound_scales": cfg.bound_scales,+ "scale_bound_margin": cfg.scale_bound_margin,+ "anomaly_safe_fallback": cfg.anomaly_safe_fallback,"started": time.strftime("%Y-%m-%d %H:%M")})+ ema_rollbacks = 0for epoch in range(start_epoch, args.epochs):sampler.set_epoch(epoch)ddp_model.train()@@ -644,7 +660,7 @@ def train(args, cfg, rank, local_rank, world):run = {"total": 0.0, "3d": 0.0, "2d": 0.0, "simcc": 0.0,"reproj": 0.0, "cam": 0.0, "pose": 0.0}nb = 0- n_skip = n_skip_run = 0+ n_skip = n_skip_run = n_repair = 0t_epoch = time.time()criterion.set_step(epoch * steps_per_epoch)for i, batch in enumerate(train_loader):@@ -667,6 +683,45 @@ def train(args, cfg, rank, local_rank, world):dist.all_reduce(flag, op=dist.ReduceOp.MAX)bad = int(flag.item())if bad:+ # An anomalous batch is anomalous because the FK terms blew up.+ # `loss_scale` / `loss_pose` in the SAME batch are small and+ # point straight back at the teacher — they are the only force+ # that pulls the bone scales out of the runaway region — so+ # discarding the batch wholesale removes the fix along with the+ # fault and quarantines those samples permanently. Step on the+ # non-FK subset instead when it is finite. Every rank must take+ # the same branch or the all-reduce deadlocks, so vote again.+ safe_ok, safe = 0, None+ if cfg.anomaly_safe_fallback:+ safe = T.safe_loss_subset(losses)+ sv = safe.item()+ safe_ok = int(math.isfinite(sv)+ and sv <= cfg.anomaly_loss_threshold)+ if world > 1:+ f2 = torch.tensor([safe_ok], device=device)+ dist.all_reduce(f2, op=dist.ReduceOp.MIN)+ safe_ok = int(f2.item())+ if safe_ok:+ scaler.scale(safe).backward()+ scaler.unscale_(optimizer)+ torch.nn.utils.clip_grad_norm_(ddp_model.parameters(),+ cfg.grad_clip)+ scaler.step(optimizer)+ scaler.update()+ scheduler.step()+ ema.update_parameters(unwrap(ddp_model))+ n_repair += 1+ if n_repair == 1 or n_repair % 200 == 0:+ log(f"🩹 repaired step {i}: FK terms dropped "+ f"(total {lv:.1f}), stepped on the bounded terms "+ f"({sv:.3f}) [{n_repair} this epoch]")+ # A repaired step DID update the weights in the corrective+ # direction, so it is progress, not a skip: the consecutive+ # counter that drives rollback/abort resets.+ n_skip_run = 0+ del preds, losses, loss, safe+ continue+n_skip += 1n_skip_run += 1# Step the schedule ANYWAY. Skipping it freezes the LR, and a@@ -684,6 +739,38 @@ def train(args, cfg, rank, local_rank, world):if n_skip_run == 1 or n_skip_run % 100 == 0:log(f"⚠️ skipping step {i} (loss {lv:.1f}) "f"[{n_skip_run} consecutive]")++ # EMA rollback. This existed only in train_distill_mhr_only's+ # single-GPU loop; the DDP path had the abort and no recovery,+ # so every Jean Zay divergence burned the whole allocation.+ # Both branches are driven by the already-all-reduced skip+ # count, so all ranks roll back on the same step, and the EMA+ # is bit-identical across ranks (DDP keeps the weights in sync+ # and every rank averages the same thing).+ if n_skip_run >= cfg.anomaly_skip_patience:+ if ema_rollbacks >= cfg.max_ema_rollbacks:+ raise RuntimeError(+ f"still stuck after {cfg.max_ema_rollbacks} EMA "+ f"rollback(s) at epoch {epoch + 1}. The best RAW / "+ f"EMA checkpoints on disk are unaffected. Exiting "+ f"rather than burning the remaining "+ f"{args.epochs - epoch} epochs.")+ ema_rollbacks += 1+ log(f"🚑 {n_skip_run} consecutive anomalous steps: rolling "+ f"the raw weights back to the EMA copy "+ f"(attempt {ema_rollbacks}/{cfg.max_ema_rollbacks}), "+ f"scaling the LR by {cfg.anomaly_rollback_lr_decay} "+ f"and clearing the optimiser moments.")+ unwrap(ddp_model).load_state_dict(ema.module.state_dict())+ # The Adam moments are what drove the model out of the+ # valid region; carrying them over walks straight back out.+ optimizer.state.clear()+ T.scale_lr(optimizer, cfg.anomaly_rollback_lr_decay,+ scheduler)+ log(f" LR is now {optimizer.param_groups[0]['lr']:.2e}")+ n_skip_run = 0+ continue+if n_skip_run >= cfg.max_consecutive_skips:raise RuntimeError(f"{n_skip_run} consecutive anomalous steps at epoch "@@ -731,41 +818,49 @@ def train(args, cfg, rank, local_rank, world):import val3dpwdpw_raw = all_reduce_mean(val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device,- cfg.use_amp, gt=args.val_3dpw_gt),+ cfg.use_amp, gt=args.val_3dpw_gt,+ adapter=dpw_adapter),device)dpw_em = all_reduce_mean(val3dpw.evaluate(ema, dpw_loader, mhr_module, device,- cfg.use_amp, gt=args.val_3dpw_gt),+ cfg.use_amp, gt=args.val_3dpw_gt,+ adapter=dpw_adapter),device)if dpw_test_loader is not None:dpw_test_raw = all_reduce_mean(val3dpw.evaluate(ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp,- gt=args.val_3dpw_gt), device)+ gt=args.val_3dpw_gt, adapter=dpw_adapter), device)dpw_test_em = all_reduce_mean(val3dpw.evaluate(ema, dpw_test_loader, mhr_module, device, cfg.use_amp,- gt=args.val_3dpw_gt), device)+ gt=args.val_3dpw_gt, adapter=dpw_adapter), device)raw["Mesh_PA_MPJPE"], em["Mesh_PA_MPJPE"] = (- dpw_raw["J12_PA_MPJPE"], dpw_em["J12_PA_MPJPE"])+ dpw_raw["J14_PA_MPJPE"], dpw_em["J14_PA_MPJPE"])raw["Mesh_MPJPE"], em["Mesh_MPJPE"] = (- dpw_raw["J12_MPJPE"], dpw_em["J12_MPJPE"])+ dpw_raw["J14_MPJPE"], dpw_em["J14_MPJPE"])log(f"\n📈 epoch {epoch+1}/{args.epochs} | lr {scheduler.get_last_lr()[0]:.2e}"f" | peak GPU {torch.cuda.max_memory_allocated() / 2**30:.1f} GiB")log(f" train total {tr['total']:.4f} | 3d {tr['3d']:.4f} | "f"2d {tr['2d']:.4f} | simcc {tr['simcc']:.3f} | "f"reproj {tr['reproj']:.4f} | cam {tr['cam']:.4f} "- f"| skipped {n_skip}/{steps_per_epoch} | {time.time() - t_epoch:.0f}s")- log(f" valRAW loss {raw['loss']:.4f} | PA-MPJPE {raw['Mesh_PA_MPJPE']:.1f} | "- f"MPJPE {raw['Mesh_MPJPE']:.1f} mm | held-in body PA "- f"{raw['Mesh_PA_MPJPE_body']:.1f} mm")- log(f" valEMA loss {em['loss']:.4f} | PA-MPJPE {em['Mesh_PA_MPJPE']:.1f} | "- f"MPJPE {em['Mesh_MPJPE']:.1f} mm | held-in body PA "- f"{em['Mesh_PA_MPJPE_body']:.1f} mm")+ f"| skipped {n_skip}/{steps_per_epoch}"+ f"{f' | repaired {n_repair}' if n_repair else ''}"+ f" | {time.time() - t_epoch:.0f}s")+ # Two different datasets, so two lines. The held-in slice scores+ # against the TEACHER's labels on training-corpus data (agreement);+ # 3DPW scores against real MoCap on sequences never seen+ # (generalisation). With --val-3dpw on, Mesh_PA_MPJPE / Mesh_MPJPE are+ # overwritten above with the 3DPW J14 numbers, so they are not printed+ # under a held-in label there.if dpw_raw is not None:- log(f" 3DPW-{args.val_3dpw_split} RAW J12 PA {dpw_raw['J12_PA_MPJPE']:.1f} "- f"MPJPE {dpw_raw['J12_MPJPE']:.1f} | J14 PA {dpw_raw['J14_PA_MPJPE']:.1f} mm")- log(f" 3DPW-{args.val_3dpw_split} EMA J12 PA {dpw_em['J12_PA_MPJPE']:.1f} "- f"MPJPE {dpw_em['J12_MPJPE']:.1f} | J14 PA {dpw_em['J14_PA_MPJPE']:.1f} mm")+ log(f" held-in (teacher agreement) RAW loss {raw['loss']:.4f} "+ f"body PA {raw['Mesh_PA_MPJPE_body']:.1f} | EMA loss "+ f"{em['loss']:.4f} body PA {em['Mesh_PA_MPJPE_body']:.1f} mm")+ log(f" 3DPW-{args.val_3dpw_split} "+ f"RAW J14 PA {dpw_raw['J14_PA_MPJPE']:.1f} "+ f"MPJPE {dpw_raw['J14_MPJPE']:.1f} | "+ f"EMA J14 PA {dpw_em['J14_PA_MPJPE']:.1f} "+ f"MPJPE {dpw_em['J14_MPJPE']:.1f} mm")# Non-finite predictions are dropped rather than crashing the metric# (see val3dpw.evaluate). Say so — a handful at epoch 1 is the raw# branch still settling, a rising count is a real instability.@@ -774,6 +869,13 @@ def train(args, cfg, rank, local_rank, world):log(f" ⚠️ 3DPW: {nd:.0f} non-finite prediction(s) dropped "f"(raw {dpw_raw.get('n_dropped', 0.0):.0f} / "f"ema {dpw_em.get('n_dropped', 0.0):.0f})")+ else:+ log(f" held-in RAW loss {raw['loss']:.4f} | PA-MPJPE "+ f"{raw['Mesh_PA_MPJPE']:.1f} | MPJPE {raw['Mesh_MPJPE']:.1f} mm | "+ f"body PA {raw['Mesh_PA_MPJPE_body']:.1f} mm")+ log(f" held-in EMA loss {em['loss']:.4f} | PA-MPJPE "+ f"{em['Mesh_PA_MPJPE']:.1f} | MPJPE {em['Mesh_MPJPE']:.1f} mm | "+ f"body PA {em['Mesh_PA_MPJPE_body']:.1f} mm")if dpw_test_raw is not None:log(f" 3DPW-TEST (report only, not used for selection)")log(f" RAW J14 MPJPE {dpw_test_raw['J14_MPJPE']:.1f} | "@@ -845,6 +947,18 @@ def parse_args():p.add_argument("--data_root", default=os.environ.get("DATA_ROOT"))p.add_argument("--output_dir", required=False,default=os.environ.get("OUTPUT_DIR"))+ p.add_argument("--bound-scales", dest="bound_scales", action="store_true",+ help="squash the 68 bone scales into the MHR rig's own "+ "parameter_limits (widened by --scale-bound-margin). "+ "Off by default so --preset baseline stays "+ "bit-identical; ON for every new run.")+ p.add_argument("--scale-bound-margin", dest="scale_bound_margin",+ type=float, default=None,+ help="how far to widen each rig limit (default 0.5)")+ p.add_argument("--anomaly-safe-fallback", dest="anomaly_safe_fallback",+ action="store_true",+ help="on an anomalous batch, step on the non-FK terms "+ "instead of discarding the batch")p.add_argument("--mhr_model_path", default=None)p.add_argument("--kp_regressor_path", default=None)@@ -868,7 +982,7 @@ def parse_args():"not a generalisation measure. 0 disables it.")p.add_argument("--val-3dpw", default=os.environ.get("THREEDPW_SEQ"),help="3DPW sequenceFiles/ directory. When given, PA-MPJPE on "- "J12 becomes the checkpoint-selection metric.")+ "J14 (adapter-applied) becomes the checkpoint-selection metric.")p.add_argument("--val-3dpw-images", default=os.environ.get("THREEDPW_IMG"),help="3DPW imageFiles/ directory.")p.add_argument("--val-3dpw-split", default="validation",@@ -882,6 +996,12 @@ def parse_args():p.add_argument("--val-3dpw-test-stride", type=int, default=5)p.add_argument("--val-3dpw-stride", type=int, default=5,help="3DPW is 30 fps; neighbouring frames are near-duplicates.")+ p.add_argument("--val-3dpw-adapter",+ default="benchmark/results/adapter_j14_h36m_teacher.npz",+ help="Fixed MHR70->J14 map applied before scoring, so the "+ "training log reports the same number as "+ "benchmark/eval_3dpw_ckpt.py and the paper table. "+ "'none' scores the raw MHR joints instead.")p.add_argument("--cliff-focal", action="store_true",help="Perspective-correct CLIFF conditioning: angles off the ""optical axis and angular box size instead of "@@ -952,6 +1072,10 @@ def build_cfg(args):cfg = T.DistillConfig()if args.data_root: cfg.data_root = args.data_rootif args.output_dir: cfg.log_dir = args.output_dir+ if args.bound_scales: cfg.bound_scales = True+ if args.anomaly_safe_fallback: cfg.anomaly_safe_fallback = True+ if args.scale_bound_margin is not None:+ cfg.scale_bound_margin = args.scale_bound_marginif args.mhr_model_path: cfg.mhr_model_path = args.mhr_model_pathif args.kp_regressor_path: cfg.kp_regressor_path = args.kp_regressor_pathcfg.batch_size = args.batch_size@@ -1066,6 +1190,10 @@ def main():f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "f"finger_w={cfg.finger_weight}"if args.losses != "legacy" else ""))+ log(f"stability bound_scales={cfg.bound_scales}"+ + (f" (margin {cfg.scale_bound_margin})" if cfg.bound_scales else "")+ + f" | safe_fallback={cfg.anomaly_safe_fallback}"+ + f" | rollback@{cfg.anomaly_skip_patience} x{cfg.max_ema_rollbacks}")log(f"seed {args.seed} | mix {args.mix} | "f"samples/epoch {args.samples_per_epoch:,}")log("=" * 66)diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex 3fff58e..4516586 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -272,6 +272,27 @@ class DistillConfig:anomaly_skip_patience: int = 50anomaly_rollback_lr_decay: float = 0.5max_ema_rollbacks: int = 3+ # On an anomalous batch, back-propagate the terms that do NOT go through the+ # MHR forward kinematics instead of discarding the batch. A skipped step+ # throws away `loss_scale` / `loss_pose` -- the ONLY force pulling the bone+ # scales back toward the teacher -- so the guard removes exactly the gradient+ # that would fix the anomaly and the sample is quarantined for good. Those+ # parameter-space terms are bounded by the target magnitudes, so stepping on+ # them is safe even when the FK terms are 1e13. Default off: `--preset+ # baseline` and `--losses legacy` must stay bit-identical.+ anomaly_safe_fallback: bool = False+ # Squash the 68 bone scales into the rig's OWN parameter_limits (widened by+ # scale_bound_margin). mhr_params[136:204] are raw linear outputs today and+ # they enter the forward kinematics multiplicatively along the kinematic+ # chain: measured on checkpoints/mhr_model.pt, a uniform +10 turns a 1.5 m+ # skeleton into 336 m and a +25 into 28,300 km, which is where the 4e7 and+ # 8.8e13 losses in the diverged runs come from. The rig bounds every one of+ # them to [-1.1, 1.1]; measured over 9,000 teacher annotations, 0.0000% of+ # targets fall outside those limits at margin 0.5 and the worst target sits+ # at |atanh| = 0.97, so the bound is free. Default off for bit-identity.+ bound_scales: bool = False+ scale_bound_margin: float = 0.5+ scale_bounds_path: str = str(PROJECT_ROOT / "assets/mhr_size_bounds.npz")use_amp: bool = Trueema_decay: float = 0.9998early_stop_patience: int = 150@@ -825,6 +846,31 @@ class InstantHMRStudent(nn.Module):nn.init.normal_(self.head_2d_logits.weight, mean=0.0, std=1e-4)nn.init.constant_(self.head_2d_logits.bias, 0.0)+ # Bone-scale bound. Buffers, not constants, so they survive+ # state_dict round-trips and bake into the ONNX graph; tanh exports.+ self.bound_scales = bool(getattr(cfg, "bound_scales", False))+ if self.bound_scales:+ b = np.load(cfg.scale_bounds_path)+ # 130, not pose_dim=136: params 130:136 are the six+ # *_length/*_width_flexible parameters. They sit at the end of the+ # pose block but they drive joint TRANSLATION channels with weights+ # up to 10.0, not rotations, so they are body-size parameters and+ # are exactly as unbounded as the 68 scales. Leaving them free+ # still let a blown-up head reach a 59 m skeleton with the scales+ # already clamped -- measured.+ # Two slices, not one: 0:3 is root translation (no rig limit, but+ # exactly 0.000e+00 in all 24,000 sampled annotations across all six+ # splits -- the person is placed by cam_trans, not by these) and+ # 130:204 is the size block. Measured: with the size block clamped+ # but root translation free, a blown-up head still reached a 375 m+ # skeleton.+ self.bound_slices = list(zip(b["starts"].tolist(), b["stops"].tolist()))+ lo = torch.from_numpy(b["lo"]).float()+ hi = torch.from_numpy(b["hi"]).float()+ m = torch.clamp(0.25 * (hi - lo), min=float(cfg.scale_bound_margin))+ self.register_buffer("scale_lo", lo - m)+ self.register_buffer("scale_hi", hi + m)+def forward(self, images, cliff_cond):B = images.shape[0]img_feats = self.backbone.forward_features(images)@@ -851,6 +897,25 @@ class InstantHMRStudent(nn.Module):pred_shape_params = global_preds[:, idx_mhr : idx_shape]pred_cam_trans = global_preds[:, idx_shape :]+ if self.bound_scales:+ # tanh(0) = 0 maps a zero-init head to the middle of each interval,+ # which is ~0 because they are near-symmetric, so this does not move+ # the starting point of a fresh run. Static python loop over two+ # slices: it unrolls under tracing, so it exports to ONNX.+ parts, prev, off = [], 0, 0+ for a, b_ in self.bound_slices:+ if a > prev:+ parts.append(pred_mhr_params[:, prev:a])+ n = b_ - a+ t = torch.tanh(pred_mhr_params[:, a:b_])+ lo = self.scale_lo[off:off + n]+ hi = self.scale_hi[off:off + n]+ parts.append(lo + (hi - lo) * 0.5 * (t + 1.0))+ prev, off = b_, off + n+ if prev < pred_mhr_params.shape[1]:+ parts.append(pred_mhr_params[:, prev:])+ pred_mhr_params = torch.cat(parts, dim=1)+feat_2d_processed = self.head_2d_feat(feat_2d)logits_2d = self.head_2d_logits(feat_2d_processed)logits_2d = logits_2d.unflatten(-1, (2, self.kp2d_bins))@@ -1210,6 +1275,26 @@ class DistillationLoss(nn.Module):losses['total_loss'] = sum(losses.values())return losses+# The three terms that back-propagate through the MHR forward kinematics. They+# are the only unbounded ones: the parameter regressions are bounded by the+# target magnitudes and the 2D terms are bounded by construction (SimCC spans+# +/-kp2d_range and its CE by log(kp2d_bins)).+FK_LOSS_KEYS = ("loss_3d_native", "loss_mhr_joints", "loss_reproj")+++def safe_loss_subset(losses):+ """Total over the terms that do NOT pass through the forward kinematics.++ This is what an anomalous batch should still be allowed to contribute. A+ blown-up bone scale makes `loss_3d_native` / `loss_reproj` enormous while+ `loss_scale` and `loss_pose` stay small and point straight back at the+ teacher value, so dropping only the FK terms keeps the restoring gradient+ and discards the exploding one.+ """+ return sum(v for k, v in losses.items()+ if k != "total_loss" and k not in FK_LOSS_KEYS)++def format_loss_terms(losses, top_k: int = 5) -> str:"""The largest individual loss terms, for diagnosing an anomalous batch.@@ -1511,6 +1596,27 @@ def train_instant_hmr():skip_reason = f"anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold}"if skip_reason is not None:+ # Same policy as train_distill_jz: an anomalous batch still+ # carries the restoring gradient in its non-FK terms, so step+ # on those rather than discarding it. Kept in both loops on+ # purpose -- the DDP path having a recovery the single-GPU path+ # lacked (and vice versa) is how this file and its fork drifted.+ if cfg.anomaly_safe_fallback:+ safe = safe_loss_subset(losses)+ sv = safe.item()+ if math.isfinite(sv) and sv <= cfg.anomaly_loss_threshold:+ scaler.scale(safe).backward()+ scaler.unscale_(optimizer)+ torch.nn.utils.clip_grad_norm_(model.parameters(),+ cfg.grad_clip)+ scaler.step(optimizer)+ scaler.update()+ scheduler.step()+ ema_model.update_parameters(model)+ global_step += 1+ criterion.set_step(global_step)+ consecutive_skips = 0+ continueconsecutive_skips += 1print(f"⚠️ WARNING: {skip_reason} — skipping step. "f"[{consecutive_skips} in a row] {format_loss_terms(losses)}")@@ -2187,6 +2293,16 @@ def parse_args():p.add_argument("--no-export", dest="no_export", action="store_true",help="Skip the ONNX export / quantization step after training.")p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")+ p.add_argument("--bound-scales", dest="bound_scales", action="store_true",+ help="Squash the body-size parameters (root translation and "+ "130:204) into the MHR rig's own parameter_limits. "+ "Off by default for bit-identity; ON for new runs.")+ p.add_argument("--scale-bound-margin", dest="scale_bound_margin", type=float,+ default=None, help="How far to widen each limit (default 0.5).")+ p.add_argument("--anomaly-safe-fallback", dest="anomaly_safe_fallback",+ action="store_true",+ help="On an anomalous batch, step on the non-FK terms "+ "instead of discarding the batch.")p.add_argument("--kp3d-warmup-steps", dest="kp3d_warmup_steps", type=int, default=None,help="Optimiser steps over which the FK-path losses ramp from 0 to full weight.")p.add_argument("--w_keypoints3d", type=float, default=None,@@ -2240,6 +2356,9 @@ def main():else:cfg.per_dataset_caps["sam3d_gt_harmony4d"] = args.harmony4d_capif args.no_resume: cfg.resume = False+ if args.bound_scales: cfg.bound_scales = True+ if args.anomaly_safe_fallback: cfg.anomaly_safe_fallback = True+ if args.scale_bound_margin is not None: cfg.scale_bound_margin = args.scale_bound_marginif args.kp3d_warmup_steps is not None: cfg.kp3d_warmup_steps = args.kp3d_warmup_stepsif args.losses == "rebalanced": apply_rebalanced_losses(cfg)# An explicit --w_keypoints3d still wins over the preset.diff --git a/instanthmr_distill_train/val3dpw.py b/instanthmr_distill_train/val3dpw.pyindex ebaed1a..8700837 100644--- a/instanthmr_distill_train/val3dpw.py+++ b/instanthmr_distill_train/val3dpw.py@@ -27,12 +27,12 @@ rsynced next to ``sequenceFiles``. ``gt="jointpositions"`` restores the rawpickle field the harness used before; the two are several mm apart, so numbersfrom the two sources must never be compared to each other.-Selection metric is **PA-MPJPE on J12**. J12 is limbs only, where MHR and SMPL-mean the same anatomical points; J14 adds neck and head, which the two rigs-place differently and which therefore carry a systematic penalty unrelated to-model quality. Procrustes alignment additionally removes any residual-coordinate-frame convention mismatch between the rig-local prediction and 3DPW's-camera space, so the number is comparable across runs without further care.+Selection metric is **J14 PA-MPJPE with the MHR70 -> J14 adapter applied** —+the same quantity `benchmark/eval_3dpw_ckpt.py` reports and a paper quotes, so+the training log, the run summary and the results table all carry one number.+The adapter (`benchmark/results/adapter_j14_h36m_teacher.npz`) is a fixed 14x70+map fitted from the teacher's MHR labels, independent of any student, so it+does not move between checkpoints or between runs."""from __future__ import annotations@@ -148,8 +148,8 @@ class ThreeDPWValSet(Dataset):@torch.no_grad()def evaluate(model, loader, mhr_module, device, use_amp: bool = True,- gt: str = "h36m") -> dict:- """PA-MPJPE and MPJPE in mm on J12 and J14, from the MHR forward pass.+ gt: str = "h36m", adapter: np.ndarray | None = None) -> dict:+ """J14 PA-MPJPE and MPJPE in mm, adapter-applied — the paper number.Predictions come from ``mhr_params`` through the same FK + (70, 127)regressor the training loss uses, so this measures exactly the geometry the@@ -187,10 +187,9 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True,# Always return the SAME key set. train_distill_jz.all_reduce_mean builds a# tensor from sorted(keys) and all-reduces it, so a rank returning a short# dict while the others return a full one is a size mismatch -> NCCL hang.- res: dict[str, float] = {"n": 0.0, "n_dropped": float(n_dropped)}- for name in ("J12", "J14"):- res[f"{name}_MPJPE"] = float("inf")- res[f"{name}_PA_MPJPE"] = float("inf")+ res: dict[str, float] = {"n": 0.0, "n_dropped": float(n_dropped),+ "J14_MPJPE": float("inf"),+ "J14_PA_MPJPE": float("inf")}if not preds:# inf, not 0: this epoch's 3DPW number must never look like a new best.return res@@ -199,11 +198,16 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True,gt_all = np.concatenate(gts) * MM # (N, J, 3) camera space, mmres["n"] = float(len(pred))- for name in ("J12", "J14"):- spec = J.JOINT_SETS[name]+ spec = J.JOINT_SETS["J14"]+ g = gt_all[:, J.gt_index(spec, gt), :]+ if adapter is not None:+ # The adapter is fitted on centred keypoints, so apply it to centred+ # keypoints; both metrics below are translation-invariant anyway.+ p = np.einsum("jk,nkc->njc", adapter,+ pred - pred.mean(axis=1, keepdims=True))+ else:p = pred[:, spec["mhr"], :]- g = gt_all[:, J.gt_index(spec, gt), :]- rp, rg = J.pelvis(p), J.pelvis(g)- res[f"{name}_MPJPE"] = float(M.mpjpe(p, g, root=(rp, rg)).mean())- res[f"{name}_PA_MPJPE"] = float(M.pa_mpjpe(p, g).mean())+ rp, rg = J.pelvis(p), J.pelvis(g)+ res["J14_MPJPE"] = float(M.mpjpe(p, g, root=(rp, rg)).mean())+ res["J14_PA_MPJPE"] = float(M.pa_mpjpe(p, g).mean())return res