Skip to content
Python

fix: bound MHR body-size outputs and repair anomalous batches instead of discarding them

mohamdeve01030293fc3
diff --git a/CLAUDE.md b/CLAUDE.md
index 767b58a..497a53a 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -16,6 +16,35 @@ Keep commit messages to one or two short sentences.
---
+## 0.5 Write for a human, not a compiler
+
+**The user is a person reading a terminal, not a log parser.** Answers that are
+technically correct and unreadable are failures. This has come up more than
+once — take it seriously.
+
+- **Define every name before you use it.** `scale_lo`, `m_noflip`, `fk_scale`,
+ `n_skip_run` mean nothing to someone who has not just read that function. If
+ a variable, flag or file has to appear, say what it *is* in plain words
+ first, once.
+- **Give commands in the form the user actually runs them.** This repo launches
+ through `datasets_pipeline/jeanzay/52_train_ddp.slurm` with `RUN_NAME=... MIX=...
+ EXTRA_TRAIN_ARGS=... sbatch`. Read the launcher and match it. A bare `srun
+ python ...` line the user has never typed is not an answer.
+- **Say what to run, in what order, and what happens next.** If a command is
+ optional, say so. If you offer a check, say what a good result looks like.
+- **Lead with the answer.** The finding first, the evidence after. Do not make
+ the user read a derivation to learn whether it worked.
+- **Explain the mechanism, don't just name it.** "The bone scales enter the
+ kinematics multiplicatively, so +10 makes a 336 m skeleton" teaches
+ something; "unbounded scale parameters caused divergence" does not.
+- **Cut the ceremony.** No restating the request, no listing what you are about
+ to do, no summary of the summary.
+
+If the user says they cannot follow an answer, that is a defect to fix in the
+next answer, not a preference to note.
+
+---
+
## 1. Think before coding
Don't assume. Don't hide confusion. Surface tradeoffs.
@@ -113,6 +142,39 @@ augmentation, so it never needed the `m_ident` mask; and the 45 `shape_params`
move the 127-joint skeleton by `0.00e+00 cm`, so they are mesh-only and every
loss and metric in the trainer is blind to them.
+**The student's MHR outputs are unbounded and 77 of them are body SIZE.** The
+head is a plain `nn.Linear`. `0:3` root translation (exactly `0.000e+00` in all
+24,000 sampled annotations — the person is placed by `cam_trans`), `130:136`
+the `*_length/_width_flexible` parameters, and `136:204` the 68 bone scales all
+scale the skeleton, the last **multiplicatively along the kinematic chain**:
+uniform +10 gives a 336 m skeleton, +25 gives 28,300 km. That, not the learning
+rate, killed four 300-epoch runs at epochs 24-32 on 2026-09-06. Watch for the
+two traps: `130:136` *look* like pose parameters (they sit inside `6:136`) but
+drive joint **translation** channels, so bounding only `136:204` still reaches
+375 m; and `6:130` are genuine rotations that stay at ~1.5 m at any magnitude,
+so leave them alone. `--bound-scales` fixes it from
+`assets/mhr_size_bounds.npz`; that file is **new and must be rsynced** or the
+job dies at startup.
+
+**The anomaly guard deletes the gradient that would fix the anomaly.** A batch
+over `anomaly_loss_threshold` was skipped whole — but `loss_scale`/`loss_pose`
+in that same batch are the only force pulling the sizes back, so the sample was
+quarantined for good. Worse, `bad` is all-reduced with `MAX`: one bad sample in
+4x64 = 256 vetoes the step for every rank, so 2% runaway samples stop 99.4% of
+steps. Use `--anomaly-safe-fallback`, which steps on the non-FK terms instead
+(`safe_loss_subset`, cosine +1.0000 with `teacher - prediction`). Never "fix" a
+divergence here by lowering the LR alone: `OneCycleLR(pct_start=0.1)` peaks at
+10% of the run, so a lower peak only moves *when* it dies.
+
+**`parameter_transform[:, :204]` has rank 192, not 204.** Twelve directions
+move the parameters and leave the skeleton *exactly* unchanged — each
+`*_flexible` parameter against its `scale_*` partner, plus the spine rotations.
+Along them every geometric loss has zero gradient, so the model is regressing
+12 dimensions of solver-arbitrary teacher noise that no image can determine.
+Do **not** "fix" this by zeroing the `*_flexible` parameters even though the rig
+pins them to `[0,0]`: the teacher genuinely uses them (`leg_length_flexible`
+mean 0.32, max 1.64), so zeroing shortens every leg.
+
**`DistillationLoss` is never `.to(device)`'d.** `run_overfit_test`,
`run_self_tests`, `train_instant_hmr` and `train_distill_jz.train` all construct
it bare, so any buffer you register lands on the CPU and raises on step 1. Build
@@ -146,6 +208,13 @@ mandatory for new runs, because the rebuilt corpus carries real per-image
focals spanning `f/diag` 0.6-1.8 where the pixel form is ambiguous; the old
corpus was constant at 1.000. See `instanthmr_distill_train/README.md`.
+**One reported number: J14 PA-MPJPE, adapter-applied, on the H36M GT.** The
+training log, `summarize_runs.py` and `benchmark/eval_3dpw_ckpt.py` all print
+it and they agree to the decimal. J12 was removed on 2026-09-06 — it was the
+rig-clean row against the old GT and is not against this one. Runs started
+before that date recorded J14 *without* the adapter, ~26 mm higher; do not put
+old and new rows in one table.
+
**PA-MPJPE is blind to half of what a demo shows.** Procrustes removes
translation, rotation and scale, so `cam_trans` drift, body-size error and
temporal jitter are all invisible to it — and to root-relative MPJPE for the
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index 73002d9..cffe86e 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -161,6 +161,134 @@ nothing; it silently mis-places the person in depth. `run_config.json` records
The default is `False` and is bit-identical to the pre-flag trainer, verified
over augmented samples tensor by tensor.
+## `--bound-scales` / `--anomaly-safe-fallback` (added 2026-09-06, mandatory)
+
+**Use both on every new run.** They fix the divergence that killed all four
+runs of the 2026-09-06 sweep at epochs 24-32 (`fno_s*`, `fsa_s*`), and every
+earlier sweep that was "fixed" by lowering the peak LR.
+
+### What was actually wrong
+
+`head_global` is a plain `nn.Linear`. 77 of its outputs are **body-size**
+parameters that the MHR forward kinematics multiplies along the kinematic
+chain, and nothing bounded them:
+
+| block | what it is | unbounded behaviour (measured on `checkpoints/mhr_model.pt`) |
+|---|---|---|
+| `0:3` | root translation | linear; +100 -> 22 m skeleton. Teacher value is **exactly 0.000e+00** in all 24,000 sampled annotations across all six splits — the person is placed by `cam_trans`. |
+| `130:136` | `*_length/_width_flexible` | linear, transform weights up to 10.0; +100 -> 32 m |
+| `136:204` | the 68 bone scales | **multiplicative**: +10 -> 336 m, +20 -> 703 km, +25 -> 28,300 km |
+
+Note `130:136`. Those six sit at the end of the *pose* block but they drive
+joint **translation** channels, not rotations, so they are size parameters.
+Bounding only `136:204` still let a blown-up head reach a 375 m skeleton.
+`6:130` are genuine rotations and stay at ~1.5 m at any magnitude — leave them
+alone.
+
+Loss on real teacher targets, with the head drifted by `d` on the size block:
+
+| drift | total loss, unbounded | total loss, `--bound-scales` |
+|---|---|---|
+| 0 | 6.5 | 6.5 |
+| 10 | **140** (over `anomaly_loss_threshold=100`) | 15.6 |
+| 20 | **2.2e5** | 25.1 |
+
+That is where the `4.06e7` / `8.76e13` losses in the dead logs come from.
+
+### Why the guard made it permanent rather than catching it
+
+An anomalous batch was discarded whole. But `loss_scale` / `loss_pose` in that
+same batch are small and point straight back at the teacher — they are the
+**only** force pulling the sizes out of the runaway region — so skipping
+removed the fix along with the fault and quarantined those samples for good.
+Two amplifiers turned that into a dead job:
+
+* `bad` is all-reduced with `MAX`, so **one** bad sample out of 4x64 = 256
+ vetoes the step for every rank. At 1% runaway samples 92% of steps are
+ vetoed; at 2%, 99.4%. The run aborts when 2% of the data is bad, not the model.
+* The EMA rollback (`anomaly_skip_patience`, `max_ema_rollbacks`) existed only
+ in `train_distill_mhr_only`'s single-GPU loop. The DDP path had the abort and
+ **no recovery at all**.
+
+`--anomaly-safe-fallback` steps on `safe_loss_subset(losses)` — everything
+except `FK_LOSS_KEYS` — instead of discarding the batch. Verified: with the
+scales sitting +20 off the teacher, that gradient has cosine **+1.0000** with
+`teacher - prediction` and moves **100%** of the 68 scales the right way, while
+the full loss is 2.9e5. The rollback is now in the DDP loop too.
+
+### Why lowering the LR never worked
+
+`OneCycleLR(pct_start=0.1)` puts the peak at 10% of the run — epoch 30 of 300.
+The four dead runs died at epochs 24-32. Lowering `max_lr` does not remove the
+peak, it just slows the random walk toward the runaway region.
+
+### The bound
+
+`assets/mhr_size_bounds.npz` holds, per parameter, the union of the rig's own
+`character_torch.parameter_limits` and the observed teacher range; the model
+widens each by `--scale-bound-margin` (0.5) and squashes with `tanh`. The rig
+bounds every one of these to `[-1.1, 1.1]`; the teacher respects that for the
+scales but **not** for `130:136`, which the rig pins to `[0,0]` while the
+teacher uses them up to 1.64 (`leg_length_flexible`) — hence the union.
+
+Measured over 24,000 annotations from all six splits: **0.000000%** of teacher
+targets fall outside the bound and the worst sits at `|atanh| = 0.80`, well
+inside tanh's linear region, so the bound costs nothing. With the flag on, a
+head deliberately blown up to `bias ~ N(0, 5000)` still yields a 1.58 m
+skeleton. Exports to ONNX (matches torch to 1.2e-7).
+
+Regenerate the asset from the rig plus the corpus if either changes; the file
+also stores `rig_lo` / `rig_hi` unmodified for reference.
+
+Both flags default to **off** and are bit-identical to the pre-flag trainer,
+verified tensor by tensor. `run_config.json` records `bound_scales`,
+`scale_bound_margin` and `anomaly_safe_fallback`.
+
+### The 12-dimensional null space (not fixed, know it exists)
+
+`parameter_transform[:, :204]` has **rank 192**. Twelve directions move the
+parameters and leave the skeleton *exactly* unchanged — they pair each
+`*_flexible` parameter against its `scale_*` partner (`scale_spine_length` vs
+`spine_length_flexible`, `scale_hip_width` vs `hip_width_flexible`, and the
+spine rotations). Along those, every geometric loss has zero gradient and only
+`loss_pose` / `loss_scale` act, so the model is being asked to regress 12
+dimensions of solver-arbitrary teacher noise that no image can determine.
+Confirmed directly: pushing `scale_spine_length` to 20.3 along a null direction
+moves the skeleton by 1.2e-7 m. The bound now keeps that wandering finite;
+collapsing the redundancy properly would be a separate change.
+
+## The reported number
+
+One number everywhere: **J14 PA-MPJPE, adapter-applied, against the
+published-protocol 3DPW ground truth.** The training log, `summarize_runs.py`,
+`benchmark/eval_3dpw_ckpt.py` and a paper results table all print the same
+quantity — cross-checked to the decimal (41.71 mm for `b3_s1` on 3DPW test).
+J14 MPJPE rides alongside it. J12 is gone: it was the rig-clean row against the
+old `jointPositions` GT, and against the H36M one it is not (those hips sit
+292 mm apart against the MHR rig's ~120).
+
+The adapter is `benchmark/results/adapter_j14_h36m_teacher.npz`, a fixed 14x70
+map fitted from the teacher's labels and independent of any student, so it does
+not move between checkpoints. `--val-3dpw-adapter none` scores raw MHR joints
+instead; a missing file is a hard error, not a silent fallback.
+
+The epoch log prints two lines because there are **two different datasets**:
+
+```
+held-in (teacher agreement) RAW loss 5.09 body PA 153.5 | EMA ...
+3DPW-validation RAW J14 PA 126.0 MPJPE 283.0 | EMA J14 PA 125.7 MPJPE 284.0 mm
+```
+
+`held-in` is a 2% slice of the **training corpus** scored against the
+**teacher's** labels — agreement, not generalisation, and only a training-health
+signal. `3DPW-validation` is 12 held-out sequences against real **MoCap**.
+Before 2026-09-06 both appeared on one line under a single "val" label, because
+`Mesh_PA_MPJPE` is overwritten with the 3DPW number when `--val-3dpw` is on.
+
+**Runs started before 2026-09-06 are not comparable**: their `J14_PA_MPJPE` in
+`history.jsonl` is the raw MHR joint set, ~26 mm higher than the adapter-applied
+one. `summarize_runs.py` still renders them, in the same column.
+
## The pair index
`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` per
diff --git a/instanthmr_distill_train/assets/mhr_size_bounds.npz b/instanthmr_distill_train/assets/mhr_size_bounds.npz
new file mode 100644
index 0000000..3a3e87c
Binary files /dev/null and b/instanthmr_distill_train/assets/mhr_size_bounds.npz differ
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 66555c7..82af302 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -460,7 +460,7 @@ def run_validation(eval_model, loader, criterion, cfg, mhr_module, device):
acc["Mesh_PA_MPJPE"] += bm.get("Mesh_PA_MPJPE", 0.0)
# Reported only. Mesh_PA_MPJPE is 60% fingers by keypoint count; this is
# the 30 body/face/foot keypoints. Selection is untouched — and when
- # --val-3dpw is on, both of these are replaced by the 3DPW J12 numbers.
+ # --val-3dpw is on, both of these are replaced by the 3DPW J14 numbers.
acc["Mesh_PA_MPJPE_body"] += bm.get("Mesh_PA_MPJPE_body", 0.0)
n += 1
out = {k: v / max(n, 1) for k, v in acc.items()}
@@ -486,11 +486,23 @@ def train(args, cfg, rank, local_rank, world):
train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)
steps_per_epoch = len(train_loader)
- dpw_loader = dpw_test_loader = None
+ dpw_loader = dpw_test_loader = dpw_adapter = None
if args.val_3dpw:
if not args.val_3dpw_images:
raise SystemExit("--val-3dpw needs --val-3dpw-images")
from val3dpw import ThreeDPWValSet
+ if str(args.val_3dpw_adapter).lower() != "none":
+ ap = Path(args.val_3dpw_adapter)
+ if not ap.is_absolute():
+ ap = Path(__file__).resolve().parent.parent / ap
+ if not ap.is_file():
+ raise SystemExit(
+ f"3DPW adapter not found: {ap}\nIt ships in the repo as "
+ "benchmark/results/adapter_j14_h36m_teacher.npz; rsync it, "
+ "or pass --val-3dpw-adapter none to score raw MHR joints.")
+ dpw_adapter = np.load(ap)["W"]
+ log(f"3DPW adapter: {ap.name} {tuple(dpw_adapter.shape)} "
+ f"-> J14 numbers match benchmark/eval_3dpw_ckpt.py")
dpw = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,
split=args.val_3dpw_split, stride=args.val_3dpw_stride,
gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal)
@@ -501,7 +513,7 @@ def train(args, cfg, rank, local_rank, world):
num_workers=max(1, cfg.num_workers // 2), pin_memory=True)
log(f"3DPW {args.val_3dpw_split}: {len(dpw):,} person-frames "
f"(stride {args.val_3dpw_stride}, GT {args.val_3dpw_gt}) "
- f"-> selection metric is J12 PA-MPJPE")
+ f"-> selection metric is J14 PA-MPJPE (adapter-applied)")
# Optional read-only monitor on the split the paper reports. Selection
# NEVER touches it: picking the best of ~100 epochs on the test split
@@ -633,8 +645,12 @@ def train(args, cfg, rank, local_rank, world):
"finger_weight": cfg.finger_weight,
"val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,
"cliff_focal": cfg.cliff_focal,
+ "bound_scales": cfg.bound_scales,
+ "scale_bound_margin": cfg.scale_bound_margin,
+ "anomaly_safe_fallback": cfg.anomaly_safe_fallback,
"started": time.strftime("%Y-%m-%d %H:%M")})
+ ema_rollbacks = 0
for epoch in range(start_epoch, args.epochs):
sampler.set_epoch(epoch)
ddp_model.train()
@@ -644,7 +660,7 @@ def train(args, cfg, rank, local_rank, world):
run = {"total": 0.0, "3d": 0.0, "2d": 0.0, "simcc": 0.0,
"reproj": 0.0, "cam": 0.0, "pose": 0.0}
nb = 0
- n_skip = n_skip_run = 0
+ n_skip = n_skip_run = n_repair = 0
t_epoch = time.time()
criterion.set_step(epoch * steps_per_epoch)
for i, batch in enumerate(train_loader):
@@ -667,6 +683,45 @@ def train(args, cfg, rank, local_rank, world):
dist.all_reduce(flag, op=dist.ReduceOp.MAX)
bad = int(flag.item())
if bad:
+ # An anomalous batch is anomalous because the FK terms blew up.
+ # `loss_scale` / `loss_pose` in the SAME batch are small and
+ # point straight back at the teacher — they are the only force
+ # that pulls the bone scales out of the runaway region — so
+ # discarding the batch wholesale removes the fix along with the
+ # fault and quarantines those samples permanently. Step on the
+ # non-FK subset instead when it is finite. Every rank must take
+ # the same branch or the all-reduce deadlocks, so vote again.
+ safe_ok, safe = 0, None
+ if cfg.anomaly_safe_fallback:
+ safe = T.safe_loss_subset(losses)
+ sv = safe.item()
+ safe_ok = int(math.isfinite(sv)
+ and sv <= cfg.anomaly_loss_threshold)
+ if world > 1:
+ f2 = torch.tensor([safe_ok], device=device)
+ dist.all_reduce(f2, op=dist.ReduceOp.MIN)
+ safe_ok = int(f2.item())
+ if safe_ok:
+ scaler.scale(safe).backward()
+ scaler.unscale_(optimizer)
+ torch.nn.utils.clip_grad_norm_(ddp_model.parameters(),
+ cfg.grad_clip)
+ scaler.step(optimizer)
+ scaler.update()
+ scheduler.step()
+ ema.update_parameters(unwrap(ddp_model))
+ n_repair += 1
+ if n_repair == 1 or n_repair % 200 == 0:
+ log(f"🩹 repaired step {i}: FK terms dropped "
+ f"(total {lv:.1f}), stepped on the bounded terms "
+ f"({sv:.3f}) [{n_repair} this epoch]")
+ # A repaired step DID update the weights in the corrective
+ # direction, so it is progress, not a skip: the consecutive
+ # counter that drives rollback/abort resets.
+ n_skip_run = 0
+ del preds, losses, loss, safe
+ continue
+
n_skip += 1
n_skip_run += 1
# Step the schedule ANYWAY. Skipping it freezes the LR, and a
@@ -684,6 +739,38 @@ def train(args, cfg, rank, local_rank, world):
if n_skip_run == 1 or n_skip_run % 100 == 0:
log(f"⚠️ skipping step {i} (loss {lv:.1f}) "
f"[{n_skip_run} consecutive]")
+
+ # EMA rollback. This existed only in train_distill_mhr_only's
+ # single-GPU loop; the DDP path had the abort and no recovery,
+ # so every Jean Zay divergence burned the whole allocation.
+ # Both branches are driven by the already-all-reduced skip
+ # count, so all ranks roll back on the same step, and the EMA
+ # is bit-identical across ranks (DDP keeps the weights in sync
+ # and every rank averages the same thing).
+ if n_skip_run >= cfg.anomaly_skip_patience:
+ if ema_rollbacks >= cfg.max_ema_rollbacks:
+ raise RuntimeError(
+ f"still stuck after {cfg.max_ema_rollbacks} EMA "
+ f"rollback(s) at epoch {epoch + 1}. The best RAW / "
+ f"EMA checkpoints on disk are unaffected. Exiting "
+ f"rather than burning the remaining "
+ f"{args.epochs - epoch} epochs.")
+ ema_rollbacks += 1
+ log(f"🚑 {n_skip_run} consecutive anomalous steps: rolling "
+ f"the raw weights back to the EMA copy "
+ f"(attempt {ema_rollbacks}/{cfg.max_ema_rollbacks}), "
+ f"scaling the LR by {cfg.anomaly_rollback_lr_decay} "
+ f"and clearing the optimiser moments.")
+ unwrap(ddp_model).load_state_dict(ema.module.state_dict())
+ # The Adam moments are what drove the model out of the
+ # valid region; carrying them over walks straight back out.
+ optimizer.state.clear()
+ T.scale_lr(optimizer, cfg.anomaly_rollback_lr_decay,
+ scheduler)
+ log(f" LR is now {optimizer.param_groups[0]['lr']:.2e}")
+ n_skip_run = 0
+ continue
+
if n_skip_run >= cfg.max_consecutive_skips:
raise RuntimeError(
f"{n_skip_run} consecutive anomalous steps at epoch "
@@ -731,41 +818,49 @@ def train(args, cfg, rank, local_rank, world):
import val3dpw
dpw_raw = all_reduce_mean(
val3dpw.evaluate(ddp_model, dpw_loader, mhr_module, device,
- cfg.use_amp, gt=args.val_3dpw_gt),
+ cfg.use_amp, gt=args.val_3dpw_gt,
+ adapter=dpw_adapter),
device)
dpw_em = all_reduce_mean(
val3dpw.evaluate(ema, dpw_loader, mhr_module, device,
- cfg.use_amp, gt=args.val_3dpw_gt),
+ cfg.use_amp, gt=args.val_3dpw_gt,
+ adapter=dpw_adapter),
device)
if dpw_test_loader is not None:
dpw_test_raw = all_reduce_mean(val3dpw.evaluate(
ddp_model, dpw_test_loader, mhr_module, device, cfg.use_amp,
- gt=args.val_3dpw_gt), device)
+ gt=args.val_3dpw_gt, adapter=dpw_adapter), device)
dpw_test_em = all_reduce_mean(val3dpw.evaluate(
ema, dpw_test_loader, mhr_module, device, cfg.use_amp,
- gt=args.val_3dpw_gt), device)
+ gt=args.val_3dpw_gt, adapter=dpw_adapter), device)
raw["Mesh_PA_MPJPE"], em["Mesh_PA_MPJPE"] = (
- dpw_raw["J12_PA_MPJPE"], dpw_em["J12_PA_MPJPE"])
+ dpw_raw["J14_PA_MPJPE"], dpw_em["J14_PA_MPJPE"])
raw["Mesh_MPJPE"], em["Mesh_MPJPE"] = (
- dpw_raw["J12_MPJPE"], dpw_em["J12_MPJPE"])
+ dpw_raw["J14_MPJPE"], dpw_em["J14_MPJPE"])
log(f"\n📈 epoch {epoch+1}/{args.epochs} | lr {scheduler.get_last_lr()[0]:.2e}"
f" | peak GPU {torch.cuda.max_memory_allocated() / 2**30:.1f} GiB")
log(f" train total {tr['total']:.4f} | 3d {tr['3d']:.4f} | "
f"2d {tr['2d']:.4f} | simcc {tr['simcc']:.3f} | "
f"reproj {tr['reproj']:.4f} | cam {tr['cam']:.4f} "
- f"| skipped {n_skip}/{steps_per_epoch} | {time.time() - t_epoch:.0f}s")
- log(f" valRAW loss {raw['loss']:.4f} | PA-MPJPE {raw['Mesh_PA_MPJPE']:.1f} | "
- f"MPJPE {raw['Mesh_MPJPE']:.1f} mm | held-in body PA "
- f"{raw['Mesh_PA_MPJPE_body']:.1f} mm")
- log(f" valEMA loss {em['loss']:.4f} | PA-MPJPE {em['Mesh_PA_MPJPE']:.1f} | "
- f"MPJPE {em['Mesh_MPJPE']:.1f} mm | held-in body PA "
- f"{em['Mesh_PA_MPJPE_body']:.1f} mm")
+ f"| skipped {n_skip}/{steps_per_epoch}"
+ f"{f' | repaired {n_repair}' if n_repair else ''}"
+ f" | {time.time() - t_epoch:.0f}s")
+ # Two different datasets, so two lines. The held-in slice scores
+ # against the TEACHER's labels on training-corpus data (agreement);
+ # 3DPW scores against real MoCap on sequences never seen
+ # (generalisation). With --val-3dpw on, Mesh_PA_MPJPE / Mesh_MPJPE are
+ # overwritten above with the 3DPW J14 numbers, so they are not printed
+ # under a held-in label there.
if dpw_raw is not None:
- log(f" 3DPW-{args.val_3dpw_split} RAW J12 PA {dpw_raw['J12_PA_MPJPE']:.1f} "
- f"MPJPE {dpw_raw['J12_MPJPE']:.1f} | J14 PA {dpw_raw['J14_PA_MPJPE']:.1f} mm")
- log(f" 3DPW-{args.val_3dpw_split} EMA J12 PA {dpw_em['J12_PA_MPJPE']:.1f} "
- f"MPJPE {dpw_em['J12_MPJPE']:.1f} | J14 PA {dpw_em['J14_PA_MPJPE']:.1f} mm")
+ log(f" held-in (teacher agreement) RAW loss {raw['loss']:.4f} "
+ f"body PA {raw['Mesh_PA_MPJPE_body']:.1f} | EMA loss "
+ f"{em['loss']:.4f} body PA {em['Mesh_PA_MPJPE_body']:.1f} mm")
+ log(f" 3DPW-{args.val_3dpw_split} "
+ f"RAW J14 PA {dpw_raw['J14_PA_MPJPE']:.1f} "
+ f"MPJPE {dpw_raw['J14_MPJPE']:.1f} | "
+ f"EMA J14 PA {dpw_em['J14_PA_MPJPE']:.1f} "
+ f"MPJPE {dpw_em['J14_MPJPE']:.1f} mm")
# Non-finite predictions are dropped rather than crashing the metric
# (see val3dpw.evaluate). Say so — a handful at epoch 1 is the raw
# branch still settling, a rising count is a real instability.
@@ -774,6 +869,13 @@ def train(args, cfg, rank, local_rank, world):
log(f" ⚠️ 3DPW: {nd:.0f} non-finite prediction(s) dropped "
f"(raw {dpw_raw.get('n_dropped', 0.0):.0f} / "
f"ema {dpw_em.get('n_dropped', 0.0):.0f})")
+ else:
+ log(f" held-in RAW loss {raw['loss']:.4f} | PA-MPJPE "
+ f"{raw['Mesh_PA_MPJPE']:.1f} | MPJPE {raw['Mesh_MPJPE']:.1f} mm | "
+ f"body PA {raw['Mesh_PA_MPJPE_body']:.1f} mm")
+ log(f" held-in EMA loss {em['loss']:.4f} | PA-MPJPE "
+ f"{em['Mesh_PA_MPJPE']:.1f} | MPJPE {em['Mesh_MPJPE']:.1f} mm | "
+ f"body PA {em['Mesh_PA_MPJPE_body']:.1f} mm")
if dpw_test_raw is not None:
log(f" 3DPW-TEST (report only, not used for selection)")
log(f" RAW J14 MPJPE {dpw_test_raw['J14_MPJPE']:.1f} | "
@@ -845,6 +947,18 @@ def parse_args():
p.add_argument("--data_root", default=os.environ.get("DATA_ROOT"))
p.add_argument("--output_dir", required=False,
default=os.environ.get("OUTPUT_DIR"))
+ p.add_argument("--bound-scales", dest="bound_scales", action="store_true",
+ help="squash the 68 bone scales into the MHR rig's own "
+ "parameter_limits (widened by --scale-bound-margin). "
+ "Off by default so --preset baseline stays "
+ "bit-identical; ON for every new run.")
+ p.add_argument("--scale-bound-margin", dest="scale_bound_margin",
+ type=float, default=None,
+ help="how far to widen each rig limit (default 0.5)")
+ p.add_argument("--anomaly-safe-fallback", dest="anomaly_safe_fallback",
+ action="store_true",
+ help="on an anomalous batch, step on the non-FK terms "
+ "instead of discarding the batch")
p.add_argument("--mhr_model_path", default=None)
p.add_argument("--kp_regressor_path", default=None)
@@ -868,7 +982,7 @@ def parse_args():
"not a generalisation measure. 0 disables it.")
p.add_argument("--val-3dpw", default=os.environ.get("THREEDPW_SEQ"),
help="3DPW sequenceFiles/ directory. When given, PA-MPJPE on "
- "J12 becomes the checkpoint-selection metric.")
+ "J14 (adapter-applied) becomes the checkpoint-selection metric.")
p.add_argument("--val-3dpw-images", default=os.environ.get("THREEDPW_IMG"),
help="3DPW imageFiles/ directory.")
p.add_argument("--val-3dpw-split", default="validation",
@@ -882,6 +996,12 @@ def parse_args():
p.add_argument("--val-3dpw-test-stride", type=int, default=5)
p.add_argument("--val-3dpw-stride", type=int, default=5,
help="3DPW is 30 fps; neighbouring frames are near-duplicates.")
+ p.add_argument("--val-3dpw-adapter",
+ default="benchmark/results/adapter_j14_h36m_teacher.npz",
+ help="Fixed MHR70->J14 map applied before scoring, so the "
+ "training log reports the same number as "
+ "benchmark/eval_3dpw_ckpt.py and the paper table. "
+ "'none' scores the raw MHR joints instead.")
p.add_argument("--cliff-focal", action="store_true",
help="Perspective-correct CLIFF conditioning: angles off the "
"optical axis and angular box size instead of "
@@ -952,6 +1072,10 @@ def build_cfg(args):
cfg = T.DistillConfig()
if args.data_root: cfg.data_root = args.data_root
if args.output_dir: cfg.log_dir = args.output_dir
+ if args.bound_scales: cfg.bound_scales = True
+ if args.anomaly_safe_fallback: cfg.anomaly_safe_fallback = True
+ if args.scale_bound_margin is not None:
+ cfg.scale_bound_margin = args.scale_bound_margin
if args.mhr_model_path: cfg.mhr_model_path = args.mhr_model_path
if args.kp_regressor_path: cfg.kp_regressor_path = args.kp_regressor_path
cfg.batch_size = args.batch_size
@@ -1066,6 +1190,10 @@ def main():
f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "
f"finger_w={cfg.finger_weight}"
if args.losses != "legacy" else ""))
+ log(f"stability bound_scales={cfg.bound_scales}"
+ + (f" (margin {cfg.scale_bound_margin})" if cfg.bound_scales else "")
+ + f" | safe_fallback={cfg.anomaly_safe_fallback}"
+ + f" | rollback@{cfg.anomaly_skip_patience} x{cfg.max_ema_rollbacks}")
log(f"seed {args.seed} | mix {args.mix} | "
f"samples/epoch {args.samples_per_epoch:,}")
log("=" * 66)
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 3fff58e..4516586 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -272,6 +272,27 @@ class DistillConfig:
anomaly_skip_patience: int = 50
anomaly_rollback_lr_decay: float = 0.5
max_ema_rollbacks: int = 3
+ # On an anomalous batch, back-propagate the terms that do NOT go through the
+ # MHR forward kinematics instead of discarding the batch. A skipped step
+ # throws away `loss_scale` / `loss_pose` -- the ONLY force pulling the bone
+ # scales back toward the teacher -- so the guard removes exactly the gradient
+ # that would fix the anomaly and the sample is quarantined for good. Those
+ # parameter-space terms are bounded by the target magnitudes, so stepping on
+ # them is safe even when the FK terms are 1e13. Default off: `--preset
+ # baseline` and `--losses legacy` must stay bit-identical.
+ anomaly_safe_fallback: bool = False
+ # Squash the 68 bone scales into the rig's OWN parameter_limits (widened by
+ # scale_bound_margin). mhr_params[136:204] are raw linear outputs today and
+ # they enter the forward kinematics multiplicatively along the kinematic
+ # chain: measured on checkpoints/mhr_model.pt, a uniform +10 turns a 1.5 m
+ # skeleton into 336 m and a +25 into 28,300 km, which is where the 4e7 and
+ # 8.8e13 losses in the diverged runs come from. The rig bounds every one of
+ # them to [-1.1, 1.1]; measured over 9,000 teacher annotations, 0.0000% of
+ # targets fall outside those limits at margin 0.5 and the worst target sits
+ # at |atanh| = 0.97, so the bound is free. Default off for bit-identity.
+ bound_scales: bool = False
+ scale_bound_margin: float = 0.5
+ scale_bounds_path: str = str(PROJECT_ROOT / "assets/mhr_size_bounds.npz")
use_amp: bool = True
ema_decay: float = 0.9998
early_stop_patience: int = 150
@@ -825,6 +846,31 @@ class InstantHMRStudent(nn.Module):
nn.init.normal_(self.head_2d_logits.weight, mean=0.0, std=1e-4)
nn.init.constant_(self.head_2d_logits.bias, 0.0)
+ # Bone-scale bound. Buffers, not constants, so they survive
+ # state_dict round-trips and bake into the ONNX graph; tanh exports.
+ self.bound_scales = bool(getattr(cfg, "bound_scales", False))
+ if self.bound_scales:
+ b = np.load(cfg.scale_bounds_path)
+ # 130, not pose_dim=136: params 130:136 are the six
+ # *_length/*_width_flexible parameters. They sit at the end of the
+ # pose block but they drive joint TRANSLATION channels with weights
+ # up to 10.0, not rotations, so they are body-size parameters and
+ # are exactly as unbounded as the 68 scales. Leaving them free
+ # still let a blown-up head reach a 59 m skeleton with the scales
+ # already clamped -- measured.
+ # Two slices, not one: 0:3 is root translation (no rig limit, but
+ # exactly 0.000e+00 in all 24,000 sampled annotations across all six
+ # splits -- the person is placed by cam_trans, not by these) and
+ # 130:204 is the size block. Measured: with the size block clamped
+ # but root translation free, a blown-up head still reached a 375 m
+ # skeleton.
+ self.bound_slices = list(zip(b["starts"].tolist(), b["stops"].tolist()))
+ lo = torch.from_numpy(b["lo"]).float()
+ hi = torch.from_numpy(b["hi"]).float()
+ m = torch.clamp(0.25 * (hi - lo), min=float(cfg.scale_bound_margin))
+ self.register_buffer("scale_lo", lo - m)
+ self.register_buffer("scale_hi", hi + m)
+
def forward(self, images, cliff_cond):
B = images.shape[0]
img_feats = self.backbone.forward_features(images)
@@ -851,6 +897,25 @@ class InstantHMRStudent(nn.Module):
pred_shape_params = global_preds[:, idx_mhr : idx_shape]
pred_cam_trans = global_preds[:, idx_shape :]
+ if self.bound_scales:
+ # tanh(0) = 0 maps a zero-init head to the middle of each interval,
+ # which is ~0 because they are near-symmetric, so this does not move
+ # the starting point of a fresh run. Static python loop over two
+ # slices: it unrolls under tracing, so it exports to ONNX.
+ parts, prev, off = [], 0, 0
+ for a, b_ in self.bound_slices:
+ if a > prev:
+ parts.append(pred_mhr_params[:, prev:a])
+ n = b_ - a
+ t = torch.tanh(pred_mhr_params[:, a:b_])
+ lo = self.scale_lo[off:off + n]
+ hi = self.scale_hi[off:off + n]
+ parts.append(lo + (hi - lo) * 0.5 * (t + 1.0))
+ prev, off = b_, off + n
+ if prev < pred_mhr_params.shape[1]:
+ parts.append(pred_mhr_params[:, prev:])
+ pred_mhr_params = torch.cat(parts, dim=1)
+
feat_2d_processed = self.head_2d_feat(feat_2d)
logits_2d = self.head_2d_logits(feat_2d_processed)
logits_2d = logits_2d.unflatten(-1, (2, self.kp2d_bins))
@@ -1210,6 +1275,26 @@ class DistillationLoss(nn.Module):
losses['total_loss'] = sum(losses.values())
return losses
+# The three terms that back-propagate through the MHR forward kinematics. They
+# are the only unbounded ones: the parameter regressions are bounded by the
+# target magnitudes and the 2D terms are bounded by construction (SimCC spans
+# +/-kp2d_range and its CE by log(kp2d_bins)).
+FK_LOSS_KEYS = ("loss_3d_native", "loss_mhr_joints", "loss_reproj")
+
+
+def safe_loss_subset(losses):
+ """Total over the terms that do NOT pass through the forward kinematics.
+
+ This is what an anomalous batch should still be allowed to contribute. A
+ blown-up bone scale makes `loss_3d_native` / `loss_reproj` enormous while
+ `loss_scale` and `loss_pose` stay small and point straight back at the
+ teacher value, so dropping only the FK terms keeps the restoring gradient
+ and discards the exploding one.
+ """
+ return sum(v for k, v in losses.items()
+ if k != "total_loss" and k not in FK_LOSS_KEYS)
+
+
def format_loss_terms(losses, top_k: int = 5) -> str:
"""The largest individual loss terms, for diagnosing an anomalous batch.
@@ -1511,6 +1596,27 @@ def train_instant_hmr():
skip_reason = f"anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold}"
if skip_reason is not None:
+ # Same policy as train_distill_jz: an anomalous batch still
+ # carries the restoring gradient in its non-FK terms, so step
+ # on those rather than discarding it. Kept in both loops on
+ # purpose -- the DDP path having a recovery the single-GPU path
+ # lacked (and vice versa) is how this file and its fork drifted.
+ if cfg.anomaly_safe_fallback:
+ safe = safe_loss_subset(losses)
+ sv = safe.item()
+ if math.isfinite(sv) and sv <= cfg.anomaly_loss_threshold:
+ scaler.scale(safe).backward()
+ scaler.unscale_(optimizer)
+ torch.nn.utils.clip_grad_norm_(model.parameters(),
+ cfg.grad_clip)
+ scaler.step(optimizer)
+ scaler.update()
+ scheduler.step()
+ ema_model.update_parameters(model)
+ global_step += 1
+ criterion.set_step(global_step)
+ consecutive_skips = 0
+ continue
consecutive_skips += 1
print(f"⚠️ WARNING: {skip_reason} — skipping step. "
f"[{consecutive_skips} in a row] {format_loss_terms(losses)}")
@@ -2187,6 +2293,16 @@ def parse_args():
p.add_argument("--no-export", dest="no_export", action="store_true",
help="Skip the ONNX export / quantization step after training.")
p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")
+ p.add_argument("--bound-scales", dest="bound_scales", action="store_true",
+ help="Squash the body-size parameters (root translation and "
+ "130:204) into the MHR rig's own parameter_limits. "
+ "Off by default for bit-identity; ON for new runs.")
+ p.add_argument("--scale-bound-margin", dest="scale_bound_margin", type=float,
+ default=None, help="How far to widen each limit (default 0.5).")
+ p.add_argument("--anomaly-safe-fallback", dest="anomaly_safe_fallback",
+ action="store_true",
+ help="On an anomalous batch, step on the non-FK terms "
+ "instead of discarding the batch.")
p.add_argument("--kp3d-warmup-steps", dest="kp3d_warmup_steps", type=int, default=None,
help="Optimiser steps over which the FK-path losses ramp from 0 to full weight.")
p.add_argument("--w_keypoints3d", type=float, default=None,
@@ -2240,6 +2356,9 @@ def main():
else:
cfg.per_dataset_caps["sam3d_gt_harmony4d"] = args.harmony4d_cap
if args.no_resume: cfg.resume = False
+ if args.bound_scales: cfg.bound_scales = True
+ if args.anomaly_safe_fallback: cfg.anomaly_safe_fallback = True
+ if args.scale_bound_margin is not None: cfg.scale_bound_margin = args.scale_bound_margin
if args.kp3d_warmup_steps is not None: cfg.kp3d_warmup_steps = args.kp3d_warmup_steps
if args.losses == "rebalanced": apply_rebalanced_losses(cfg)
# An explicit --w_keypoints3d still wins over the preset.
diff --git a/instanthmr_distill_train/val3dpw.py b/instanthmr_distill_train/val3dpw.py
index ebaed1a..8700837 100644
--- a/instanthmr_distill_train/val3dpw.py
+++ b/instanthmr_distill_train/val3dpw.py
@@ -27,12 +27,12 @@ rsynced next to ``sequenceFiles``. ``gt="jointpositions"`` restores the raw
pickle field the harness used before; the two are several mm apart, so numbers
from the two sources must never be compared to each other.
-Selection metric is **PA-MPJPE on J12**. J12 is limbs only, where MHR and SMPL
-mean the same anatomical points; J14 adds neck and head, which the two rigs
-place differently and which therefore carry a systematic penalty unrelated to
-model quality. Procrustes alignment additionally removes any residual
-coordinate-frame convention mismatch between the rig-local prediction and 3DPW's
-camera space, so the number is comparable across runs without further care.
+Selection metric is **J14 PA-MPJPE with the MHR70 -> J14 adapter applied** —
+the same quantity `benchmark/eval_3dpw_ckpt.py` reports and a paper quotes, so
+the training log, the run summary and the results table all carry one number.
+The adapter (`benchmark/results/adapter_j14_h36m_teacher.npz`) is a fixed 14x70
+map fitted from the teacher's MHR labels, independent of any student, so it
+does not move between checkpoints or between runs.
"""
from __future__ import annotations
@@ -148,8 +148,8 @@ class ThreeDPWValSet(Dataset):
@torch.no_grad()
def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
- gt: str = "h36m") -> dict:
- """PA-MPJPE and MPJPE in mm on J12 and J14, from the MHR forward pass.
+ gt: str = "h36m", adapter: np.ndarray | None = None) -> dict:
+ """J14 PA-MPJPE and MPJPE in mm, adapter-applied — the paper number.
Predictions come from ``mhr_params`` through the same FK + (70, 127)
regressor the training loss uses, so this measures exactly the geometry the
@@ -187,10 +187,9 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
# Always return the SAME key set. train_distill_jz.all_reduce_mean builds a
# tensor from sorted(keys) and all-reduces it, so a rank returning a short
# dict while the others return a full one is a size mismatch -> NCCL hang.
- res: dict[str, float] = {"n": 0.0, "n_dropped": float(n_dropped)}
- for name in ("J12", "J14"):
- res[f"{name}_MPJPE"] = float("inf")
- res[f"{name}_PA_MPJPE"] = float("inf")
+ res: dict[str, float] = {"n": 0.0, "n_dropped": float(n_dropped),
+ "J14_MPJPE": float("inf"),
+ "J14_PA_MPJPE": float("inf")}
if not preds:
# inf, not 0: this epoch's 3DPW number must never look like a new best.
return res
@@ -199,11 +198,16 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
gt_all = np.concatenate(gts) * MM # (N, J, 3) camera space, mm
res["n"] = float(len(pred))
- for name in ("J12", "J14"):
- spec = J.JOINT_SETS[name]
+ spec = J.JOINT_SETS["J14"]
+ g = gt_all[:, J.gt_index(spec, gt), :]
+ if adapter is not None:
+ # The adapter is fitted on centred keypoints, so apply it to centred
+ # keypoints; both metrics below are translation-invariant anyway.
+ p = np.einsum("jk,nkc->njc", adapter,
+ pred - pred.mean(axis=1, keepdims=True))
+ else:
p = pred[:, spec["mhr"], :]
- g = gt_all[:, J.gt_index(spec, gt), :]
- rp, rg = J.pelvis(p), J.pelvis(g)
- res[f"{name}_MPJPE"] = float(M.mpjpe(p, g, root=(rp, rg)).mean())
- res[f"{name}_PA_MPJPE"] = float(M.pa_mpjpe(p, g).mean())
+ rp, rg = J.pelvis(p), J.pelvis(g)
+ res["J14_MPJPE"] = float(M.mpjpe(p, g, root=(rp, rg)).mean())
+ res["J14_PA_MPJPE"] = float(M.pa_mpjpe(p, g).mean())
return res