Skip to content
Python

feat: surface (PVE) loss on a bit-exact rig-mesh vertex subset, gated v2 crop-centre fix, and the five-split harmony4d mixture

mohamdevdfa7a2daebca
diff --git a/CLAUDE.md b/CLAUDE.md
index 8bcf048..85fc5e9 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -167,6 +167,22 @@ steps. Use `--anomaly-safe-fallback`, which steps on the non-FK terms instead
divergence here by lowering the LR alone: `OneCycleLR(pct_start=0.1)` peaks at
10% of the run, so a lower peak only moves *when* it dies.
+**Body shape is invisible to every geometric loss, and the fix is a vertex
+term.** `MHRForwardPass.get_joints` zeroes the 45 `shape_params`, and that is
+not a shortcut: they move the 127-joint skeleton by *exactly* `0.000e+00 cm`.
+Measured `d(loss)/d(shape_params)` is `8.9e-03` through vertices and
+`0.000e+00` through joints, so `--w-verts` is the ONLY route by which shape gets
+a geometric gradient; ignoring identity costs 5.95 mm mean / 22.68 mm max PVE.
+Do **not** reach for `models/mhr_assets/assets/lod6.fbx` to make it cheap — it
+needs `pymomentum` (which segfaults here) and it is a different topology, so the
+teacher correspondence would have to be fitted. `n_verts` instead takes a
+farthest-point subset of the rig's OWN 18,439-vertex mesh and rewrites the three
+flattened influence tables of the rig's own `linear_blend_skinning`; the op is
+untouched, so the answer is bit-exact against the full mesh (4.6e-05 cm) and the
+vertex ids match the teacher's. 595 vertices cost +0.6 ms per batch-64 step
+against 16.7 ms for all 18,439. `loss_verts` belongs in `FK_LOSS_KEYS` — a
+blown-up bone scale explodes vertices exactly as it explodes joints.
+
**`parameter_transform[:, :204]` has rank 192, not 204.** Twelve directions
move the parameters and leave the skeleton *exactly* unchanged — each
`*_flexible` parameter against its `scale_*` partner, plus the spine rotations.
diff --git a/docs/todo.md b/docs/todo.md
index 32a3832..3847459 100644
--- a/docs/todo.md
+++ b/docs/todo.md
@@ -9,7 +9,7 @@ a number appears, the command that produced it is named.
---
-## Done (2026-09-06)
+## Done (2026-09-06 / 09-07)
### 1. The ONNX exporter dropped the scale bounds — 21 mm, silently
@@ -63,51 +63,46 @@ this corrupts. The `bound_scales` half is where the 14–21 mm lives.
---
+### 3. The v2 crop-centre update forgot to divide by the zoom — fixed 2026-09-07
+
+`train_distill_mhr_only.py` wrote `cx -= dx * crop_w / 2.0`. The augmentation
+warps the crop with `u' = s*R*(u - c) + c + t`, so inverting it puts the visible
+window's centre at `-t/s`: the shift has to be divided by the zoom. It
+over-corrected when zooming in and under-corrected when zooming out, and it was
+live in every `--preset v2` run (`geom_trans=0.20`, `geom_scale_max=2.0`).
+
+**Fixed behind `--crop-centre-fix`** (config `crop_centre_fix`), default off so
+generation 5 and `--preset baseline` stay bit-identical — verified: every loss
+term matches `git show HEAD` to the last printed digit on 48 real augmented
+samples, under both `baseline/legacy` and `v2/rebalanced`.
+
+**Verified on the fix itself.** Reference is the exact preimage of the crop
+centre under the real `M_total` that `cv2.warpAffine` was called with, over 400
+augmented COCO crops at v2 strength with rotation disabled so the zoom is the
+only variable:
+
+| | box-centre error, mean | p95 | max |
+|---|---|---|---|
+| `crop_centre_fix=False` | 7.73 px | 21.00 px | 34.93 px |
+| `crop_centre_fix=True` | **0.68 px** | 1.12 px | 1.50 px |
+
+The 0.68 px residual is the bbox-rounding floor.
+
+**With rotation on (±30°) a ~31 px residual remains against that reference, and
+it is NOT this bug** — the fix moves it only 33.40 → 30.93 px. The code models
+the in-plane rotation as a camera roll (it rotates `joints_3d`, `cam_trans` and
+the box centre about the image centre together), for which "preimage of the crop
+centre" is the wrong target, so most of that 31 px is the reference, not a
+defect. Expressing the `dx`/`dy` correction in the rotated axes instead changes
+the distance from the box centre to the actual body centre by 63.75 → 63.15 px,
+i.e. nothing. **No second defect demonstrated; see item 11 for what is still
+open.**
+
## Queue
Ordered by (cost of being wrong) x (cheapness of the fix). Items 3, 4 and 6 are
defects; 5 and 7 onward are opportunities the audit raised that are not bugs.
-### 3. The v2 crop-centre update forgets to divide by the zoom — **in flight**
-
-`train_distill_mhr_only.py:678` writes `cx -= dx * crop_w / 2.0`. The
-augmentation warps the crop with `u' = s*R*(u - c) + c + t`; inverting it puts
-the visible window's centre at `-t/s`, so the correction needs the `/s`. It
-over-corrects when zooming in and under-corrects when zooming out. Live in
-every `--preset v2` run (`geom_trans=0.20`, `geom_scale_max=2.0`).
-
-**Measured** over 972 augmented COCO/AIC/MPII samples drawn from the real v2
-distribution:
-
-| | px err mean | px p95 | deg mean | deg p95 | deg max |
-|---|---|---|---|---|---|
-| all | 11.7 | 33.6 | 0.61 | 1.73 | 3.21 |
-
-0.6° of error in the bearing the network is told, on average — 4% of the crop's
-own angular half-width, 13% at worst.
-
-**It is a wrong input, not a wrong label.** Verified: feeding the teacher's own
-3D joints and `cam_trans` through the reprojection loss's full geometry
-(counter-rotate, un-mirror, project in the original frame, apply `aug_M`)
-against the augmented 2D targets, over 64 samples with 34 flipped, gives a
-residual of **0.000 px**. The augmented geometry chain is exact. And the third
-conditioning channel — angular size, `b_scale / scale`, the one that sets depth
-and motivated `--cliff-focal` — is already correct. The damage is confined to
-lateral placement, which PA-MPJPE cannot see.
-
-**Fix** (one character per line):
-
-```python
-cx -= dx * crop_w / (2.0 * scale)
-cy -= dy * crop_w / (2.0 * scale)
-```
-
-**Do not patch this on the cluster while runs are chaining.**
-`52_train_ddp.slurm` resubmits itself on walltime, so an rsync'd trainer changes
-the augmentation partway through a 300-epoch run. Land it for the next
-generation, and note in `STATUS.md` that generation 5 carried the uncorrected
-version.
-
### 4. Validation drops the frames the model failed on
`val3dpw.py:179` removes non-finite predictions before scoring — necessary,
@@ -166,6 +161,93 @@ that they *help* the global token via self-attention. Sequence: finish the
runs, score both routes, then a separate ablation with the 70 queries removed.
(`--detach-2d-head` is gone; it was tried and rejected, see the trainer README.)
+### 5b. Late divergence the loss guard cannot see (`bno_s1`, job 1813482)
+
+**Open. This killed a healthy run at epoch 92 of 300 and nothing caught it.**
+
+`bno_s1` reached 45.77 mm J14 PA at epoch 84 and was dead by 92, aborting on
+`still stuck after 3 EMA rollback(s)`. Its twin `bno_s0`, identical config and
+a different seed, skipped exactly **one** step in 100 epochs.
+
+This is **not** the generation-4 failure. `--bound-scales` held: the anomalous
+losses are 50-175, not the 4e7 / 8.8e13 of the size-parameter blowup, and the
+skeleton stayed metre-scale. PA-MPJPE removes scale, so 341 mm means the
+**pose** was destroyed — the 124 rotations at `mhr_params[6:130]`, which
+`--bound-scales` deliberately leaves free.
+
+It is also not the learning rate. The collapse ran at `lr 6.8e-5`, a quarter of
+nominal, well past OneCycle's epoch-30 peak.
+
+**The precursor, nine epochs early, that nobody was watching.** Epochs 75-84 ran
+with `skipped 0/1562` every single epoch — no NaN, no loss over the threshold —
+while the two validation branches disagreed wildly:
+
+| epoch | RAW J14 PA | EMA J14 PA | skipped |
+|---|---|---|---|
+| 75 | 53.0 | 47.7 | 0 |
+| 76 | 51.7 | **188.9** | 0 |
+| 77 | 48.4 | 123.4 | 0 |
+| 78 | 49.3 | 69.5 | 0 |
+| 79 | 47.4 | 50.8 | 0 |
+| 80-84 | 47-48 | 45.8-48.7 | 0 |
+| **85** | **268.7** | 47.1 | 67 |
+| 86 | 302.3 | 321.0 | 0 |
+
+RAW is sampled once per epoch; the EMA integrates every step. So the epoch-76
+spike says the weights took a large excursion *inside* an epoch that the
+end-of-epoch snapshot missed entirely and the loss guard never saw. The EMA
+decayed it out over ~3 epochs (half-life `ln2/(1-0.9998)` = 3,466 steps ≈ 2.2
+epochs, which matches), everything looked fine again, and nine epochs later the
+run died for real.
+
+**`EMA / RAW` on the validation metric is a free excursion detector and is not
+monitored.** Add it to the epoch line and to `summarize_runs.py`; a ratio above
+~1.5 with `skipped 0` means the weights are wandering between snapshots.
+
+**Each rollback made things immediately worse.** Both visible rollbacks are
+followed by a markedly worse epoch:
+
+| rollback | epoch after | train total | skipped |
+|---|---|---|---|
+| attempt 2 (end of ep 73) | 74 | 3.19 (was 2.64) | 74 |
+| attempt 3 (end of ep 84) | 85 | **7.99** (was 2.60) | 67 |
+
+Attempt 3 restored a **45.8 mm** EMA at `lr 6.8e-5` and the model was at 268 mm
+one epoch later. Restoring good weights and then destroying them within 1,562
+steps is not a learning-rate problem — the recovery mechanism is a prime
+suspect, not just the failure.
+
+**Ranked hypotheses, and how to test each.**
+
+1. **The Adam restart in the rollback.** `optimizer.state.clear()` zeroes both
+ moments, so the very next step is `lr * m̂/(√v̂+ε) = lr * sign(g)` for **all**
+ 40M parameters at once — a coordinated jump of norm `lr * √(4e7)` ≈ 0.43 at
+ this LR, against the heavily damped updates steady-state Adam produces.
+ *Test:* keep the moments, or re-warm the LR over ~200 steps after a
+ rollback, and see whether the post-rollback epoch still degrades.
+2. **The rotations are genuinely unbounded.** `--bound-scales` covers `0:3` and
+ `130:204` only. `6:130` cannot blow up the skeleton's *size* but can produce
+ arbitrary pose. *Test:* log `mhr_params[6:130]` percentiles per epoch; if
+ they drift monotonically through epochs 75-84 the wandering is there.
+3. **The guard's trigger is the wrong quantity.** It fires on
+ `loss > 100 or non-finite`. Epochs 75-84 satisfied neither while the model
+ visited bad regions. *Test:* trigger on gradient norm or on a weight-delta
+ norm instead of, or as well as, the loss.
+
+**Two log defects found while diagnosing this, both worth fixing:**
+
+* `bad` is all-reduced with `ReduceOp.MAX` (`train_distill_jz.py:682`) but the
+ message prints **rank 0's own** loss. That is why the log shows
+ `skipping step 57 (loss 10.8)` against a threshold of 100 — a healthy rank
+ vetoed by a sick one. Print the reduced maximum, or the vetoing rank.
+* `📈 epoch` reports RAW and EMA side by side but nothing flags their
+ divergence, which was the only visible symptom for nine epochs.
+
+**Ruled out.** The safe-fallback path was *not* masking the collapse: `repaired`
+fired **once** in the entire run, and only one epoch shows a `repaired` count.
+The concern filed as item 6 is real bookkeeping sloppiness but had nothing to do
+with this failure.
+
### 6. `n_repair` overstates successful updates
`GradScaler.step` silently skips the optimiser when gradients overflow, but
@@ -213,8 +295,12 @@ body shape.
**Consequence for claims, today**: PA-MPJPE can improve while the mesh gets
worse. Do not say "better mesh" on the strength of a pose number.
-**Consequence for work, later**: a surface term and a surface metric, with a
-valid cross-topology correspondence.
+
+**Half-addressed 2026-09-07 by `--w-verts`** (see item 13): the surface *term* now
+exists and is the only loss whose gradient reaches `shape_params` — measured
+`d(loss)/d(shape)` is `8.9e-03` through vertices and **exactly 0.000e+00**
+through joints. What is still missing is the surface *metric*: nothing in
+`benchmark/` reports PVE, so the claim side of this item is unchanged.
### 10. `fy` for both axes — checked, not a problem here
@@ -231,6 +317,124 @@ The `bno` / `dt` mix is `aic/coco/mpii` only — perfectly isotropic — and 3DP
differs by 0.4%. Revisit only if a new split with anisotropic pixels enters the
mix.
+
+### 11. The rotation half of the CLIFF crop-centre correction is unverified
+
+Item 3 fixed the zoom. The rotation is a separate question and it was **not**
+resolved: with `geom_rot_deg=30` a ~31 px box-centre residual remains against a
+preimage-of-crop-centre reference, and the zoom fix moves it by only 2.5 px.
+
+Two readings, and the measurement does not separate them. Either the code is
+right — it treats the in-plane rotation as a camera roll, rotating `joints_3d`,
+`cam_trans` and the box centre about the image centre together, and under that
+model the box is *supposed* to move, so the reference is simply wrong — or there
+is a real error hiding under a reference too loose to see it. What was ruled
+out: expressing the `dx`/`dy` correction in the rotated axes rather than the
+unrotated ones, which changes the box-centre-to-body-centre distance from 63.75
+to 63.15 px, i.e. nothing.
+
+**To settle it** the reference has to come from the camera-roll model itself,
+not from the warp: construct a synthetic camera, roll it by the augmentation
+angle, and check that the conditioning the dataset emits equals the conditioning
+that rolled camera would produce for the same box. Until then, do not "fix" the
+rotation — the current form is not shown to be wrong.
+
+### 12. Benchmark HGNetv2 against RepViT — after the baseline lands
+
+Not to be run now. `repvit_m2_3` is the incumbent and generation 6 exists to
+give it a solid, uncontaminated baseline on the five-split corpus; swapping the
+trunk at the same time would confound both questions.
+
+When it is time, the comparison has to cover all four axes, because a backbone
+that wins on one can lose the deployment:
+
+* **Model sizes** — which HGNetv2 variants exist and their parameter counts
+ against `repvit_m2_3`'s. `--backbone` already takes any timm name, and
+ `config_from_checkpoint` reads it back out of `run_config.json`, so no code
+ change is needed to try one.
+* **Accuracy** — J14 PA-MPJPE with the adapter, 2 to 3 seeds. The documented
+ seed spread is 34–49 mm on the 70-keypoint metric, so a single run proves
+ nothing.
+* **Speed** — ONNX latency on the deployment target, not PyTorch training
+ throughput. `tools/bench_student_arch.py` is the existing harness.
+* **Capabilities** — HGNetv2 is a plain conv stack where RepViT is
+ reparameterisable. Check that the export path survives (`tools/pth_to_onnx.py`
+ has a torch↔ONNX parity gate that exits if the worst output differs by more
+ than 1e-3) and that `backbone.forward_features` returns something the
+ `feat_proj` reshape accepts — the trainer handles both 4-D and 3-D feature
+ maps, but the channel count feeds `backbone_feat_dim` and that is a config
+ field, not inferred.
+* **On our data, locally** — the local corpus plus `checkpoints/mhr_model.pt` is
+ enough to run the 1-batch overfit test and a short run per backbone before
+ spending cluster time. Judge the overfit at ~1k steps.
+
+### 13. PVE has no metric yet — only a loss
+
+`--w-verts` supervises the surface (it is the fix half of item 9), but nothing
+*reports* it. Competitors quote PVE, so a submission needs it.
+
+The pieces are in place and the awkward part is already solved: the vertex
+subset is drawn from the rig's **own** 18,439-vertex mesh, not from a separate
+LOD asset, so predicted and teacher vertices share topology and vertex ids and
+the correspondence is exact — no cross-topology fitting, which is what makes PVE
+comparisons between rigs contentious. `MHRForwardPass.get_joints_and_vertices`
+returns them.
+
+**What is missing**: 3DPW ships SMPL, and MHR is not SMPL. A published PVE
+number needs an MHR→SMPL surface correspondence with the same care the J14
+adapter needed — float64 and a truncated `lstsq`, fitted on train sequences and
+never refitted on the reported split. Until that exists, PVE can be reported
+against the *teacher* on held-out crops, which measures distillation fidelity
+and is honest as long as it is labelled as such.
+
+**Do not raise `n_verts` for the metric.** 595 vertices is a sampling choice for
+the loss; a reported PVE should use all 18,439, which costs 21.6 ms per batch-64
+forward and is irrelevant at evaluation time.
+
+### 14. Do the tanh size bounds block extreme poses? — measured, they do not
+
+Raised because `vid4.mp4` looked poorly fitted after `--bound-scales` landed.
+**The bound cannot be the cause, and here is why.**
+
+`--bound-scales` squashes two slices through a `tanh`: `mhr_params[0:3]` (root
+translation) and `[130:204]` (the six `*_flexible` size parameters and the 68
+bone scales) — 77 parameters, all of them body SIZE. The 124 local joint
+**rotations** at `[6:130]`, which is what a yoga pose actually is, and the root
+rotation at `[3:6]` are **not bounded at all**.
+
+Measured over 4,000 teacher annotations from each of the six local splits —
+1,848,000 bounded targets in total:
+
+| split | targets outside the bound | worst \|atanh\| | max \|rotation param\| |
+|---|---|---|---|
+| 3dpw | 0.0000% | 0.63 | 2.64 |
+| aic | 0.0000% | 0.90 | 2.94 |
+| coco | 0.0000% | 0.97 | 2.94 |
+| harmony4d | 0.0000% | 0.59 | 2.56 |
+| mpii | 0.0000% | 0.80 | 2.94 |
+| distill_mix | 0.0000% | 1.63 | 2.84 |
+| **all** | **0 / 1,848,000** | **1.63** | |
+
+Not one target is clipped, and the worst sits at `|atanh| = 1.63`, i.e.
+`tanh = 0.926` — a value a linear head reaches without saturating. The one real
+cost is gradient damping, `1 - tanh²`, and on the teacher distribution that is
+0.994x at the median, 0.875x at p99 and 0.733x at worst. Harmless.
+
+**So look elsewhere for the `vid4.mp4` regression.** Two candidates, in order:
+
+1. **A stale ONNX.** `models/instanthmr_mhr_only_ckpt90.onnx` and
+ `models/instanthmr.onnx` carry **no `metadata_props` at all**, so
+ `instanthmr.inference` cannot know how they were conditioned and falls back
+ to the pixel-CLIFF form with a `1.00 x diag` focal. Feeding a focal-aware
+ model that vector raises nothing and silently mis-places the person in depth.
+ Every 2026-09-07 export (`models/instanthmr_mhr_only.onnx`, `models/gen5/*`)
+ does carry `cliff_focal=true, bound_scales=true`. **Re-render `vid4.mp4` with
+ a stamped export before treating this as a model regression.**
+2. **The deployment focal fallback.** With no annotation to read a focal from,
+ `demo.py` uses `1.05 x diag` for focal-aware graphs. If the real camera is
+ far from that, depth is wrong — `--focal PX` overrides it.
+
+
---
## Evaluation hygiene the audit raised
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index 5a349df..1867bb5 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -291,6 +291,117 @@ Confirmed directly: pushing `scale_spine_length` to 20.3 along a null direction
moves the skeleton by 1.2e-7 m. The bound now keeps that wandering finite;
collapsing the redundancy properly would be a separate change.
+## `--w-verts` — the surface (PVE) term (added 2026-09-07)
+
+Every geometric loss in this trainer is blind to body shape. `get_joints` zeroes
+the 45 `shape_params` before the forward pass, and that is not a shortcut — the
+identity coefficients move the 127-joint skeleton by **exactly** `0.000e+00 cm`.
+They are mesh-only. So `w_shape` in parameter space was the only thing
+supervising them, and at `--losses rebalanced` that weight is 0.03, carrying
+0.2% of the trunk gradient.
+
+`--w-verts W` adds a term that skins the rig and compares vertices, which is the
+only route by which a geometric gradient can reach shape. Measured
+`d(loss)/d(shape_params)`: **8.9e-03** through vertices, **exactly 0.000e+00**
+through joints.
+
+**What it costs to ignore shape.** Running the rig on real teacher parameters
+with the identity zeroed against the true identity: **5.95 mm mean, 22.68 mm max
+per-vertex error**. That is a floor a shape-blind model cannot perceive, let
+alone fix.
+
+### It uses the rig's own mesh, not a separate LOD
+
+The obvious way to make this cheap is to load `lod6.fbx` (595 vertices against
+lod0's 18,439). Don't — it needs `pymomentum` to parse, which segfaults against
+the torch in this environment, and lod6 is a *different topology*, so the
+teacher correspondence would have to be fitted rather than being exact.
+
+Instead `n_verts` picks a farthest-point subset of the rig's **own** 18,439
+vertices and rewrites the three flattened influence tables of the rig's own
+`linear_blend_skinning` — `(vertex, bone, weight)` triples — to reference only
+those, under a compacted index. The skinning op is untouched, so the result is
+the full mesh's answer restricted to those vertices, **verified bit-exact to
+4.6e-05 cm**. Same topology, same vertex ids as the teacher: the correspondence
+is exact, not fitted. The default `n_verts=595` is lod6's vertex count, chosen
+so the sampling density matches what the LOD would have given.
+
+Measured at batch 64 on an RTX 4070:
+
+| path | fwd+bwd | vs skeleton-only |
+|---|---|---|
+| skeleton FK only (what the trainer already did) | 4.85 ms | — |
+| + 595-vertex subset | 5.44 ms | **+0.6 ms** |
+| + all 18,439 vertices | 21.6 ms | +16.7 ms |
+
+Against a 207 ms step that is **+0.3%** for the subset and +8.1% for the full
+mesh. On a real training step, end to end: **+0.9% wall time and +0 MiB peak
+GPU** — the backbone activations dominate and the vertex tensors fit in slack.
+
+### Choosing the weight
+
+The term is an unsquared Euclidean distance in **metres**, the same form as
+`loss_3d_native` under `--losses rebalanced`, so the weights are directly
+comparable and `loss_verts / w_verts * 1000` reads as mean PVE in millimetres.
+Per-term gradient norm on the trunk, pretrained backbone, `v2 + rebalanced`:
+
+| term | \|grad\| | share |
+|---|---|---|
+| `loss_reproj` | 33.97 | 61.3% |
+| `loss_verts` (at w=1.0) | 7.21 | 13.0% |
+| `loss_cam` | 4.51 | 8.1% |
+| `loss_3d_native` | 3.74 | 6.7% |
+| `loss_pose` | 2.93 | 5.3% |
+| `loss_shape` | 0.12 | **0.2%** |
+
+**`--w-verts 0.35` is the recommended setting**: `|grad| = 2.52`, 5.0% of the
+update, just under `loss_3d_native` and nowhere near `loss_reproj`. At 1.0 it
+would be the second-largest term in the budget, which is not what a first trial
+should do.
+
+`loss_verts` is in `FK_LOSS_KEYS`, so `--anomaly-safe-fallback` drops it along
+with the other forward-kinematics terms on an anomalous batch — a blown-up bone
+scale explodes the vertices exactly as it explodes the joints. It is also
+multiplied by `fk_scale`, so it ramps over `kp3d_warmup_steps` (2000) like
+`loss_3d_native` and `loss_reproj`; a PVE read from the first epoch is
+suppressed by that ramp, not genuinely small.
+
+`w_verts = 0.0` is the default and disables the term **and every code path it
+touches** — `MHRForwardPass` is then constructed with `n_verts=0` and builds no
+subset at all. Verified: every loss term matches `git show HEAD` to the last
+printed digit on 48 real augmented samples, under both `baseline/legacy` and
+`v2/rebalanced`.
+
+**There is no PVE metric yet, only the loss.** See `docs/todo.md` item 13.
+
+## `--crop-centre-fix` (added 2026-09-07)
+
+`--preset v2` sets `cliff_follows_aug`, which moves the CLIFF conditioning to
+describe the *augmented* box. The crop-centre half of that correction was
+missing a division by the zoom: the warp is `u' = s*R*(u - c) + c + t`, so
+inverting it puts the visible window's centre at `-t/s`, not `-t`.
+
+Verified against the exact preimage of the crop centre under the real
+`M_total` that `cv2.warpAffine` was called with, over 400 augmented COCO crops
+at v2 strength, rotation disabled so the zoom is the only variable:
+
+| | mean | p95 | max |
+|---|---|---|---|
+| off (the gen-5 behaviour) | 7.73 px | 21.00 px | 34.93 px |
+| **on** | **0.68 px** | 1.12 px | 1.50 px |
+
+The residual is the bbox-rounding floor. **Default off** so generation 5 and
+`--preset baseline` reproduce bit-for-bit; it is a no-op under `--preset
+baseline`, which does not set `cliff_follows_aug` at all. Pass it for every new
+`v2` run.
+
+This is a wrong **input**, not a wrong label — the reprojection chain is exact
+to 0.000 px — and it only affects lateral placement, which PA-MPJPE cannot see.
+Do not expect the benchmark to move; expect the conditioning to stop lying.
+
+The rotation half of the same correction is **not** settled: see `docs/todo.md`
+item 11.
+
## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)
**Do not retry this without reading the failure mode.** The flag is gone from
@@ -485,5 +596,15 @@ python3 train_distill.py \
--no-export # skip ONNX export/quantization after training
```
+On the cluster, flags go through `EXTRA_TRAIN_ARGS` on
+`datasets_pipeline/jeanzay/52_train_ddp.slurm`, not on a bare command line. The
+generation-6 set is:
+
+```
+--preset {baseline|v2} --losses rebalanced \
+--cliff-focal --bound-scales --anomaly-safe-fallback --crop-centre-fix \
+--w-verts {0|0.35}
+```
+
Training auto-resumes from `runs/<name>/best_student_model_v3.pth` if present.
Outputs (best checkpoints + `export/*.onnx`) land under `--output_dir`.
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 82af302..9d3d380 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -332,6 +332,7 @@ def build_jz_loaders(cfg, args, rank: int, world: int):
geom_trans=cfg.geom_trans, geom_flip_p=cfg.geom_flip_p,
geom_scale_max=cfg.geom_scale_max,
cliff_follows_aug=cfg.cliff_follows_aug,
+ crop_centre_fix=cfg.crop_centre_fix,
cliff_focal=cfg.cliff_focal,
occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality)
@@ -480,7 +481,8 @@ def train(args, cfg, rank, local_rank, world):
out_dir.mkdir(parents=True, exist_ok=True)
W = np.load(cfg.kp_regressor_path)
- mhr_module = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W)
+ mhr_module = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W,
+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0)
criterion = T.DistillationLoss(cfg, mhr_module)
train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)
@@ -645,6 +647,8 @@ def train(args, cfg, rank, local_rank, world):
"finger_weight": cfg.finger_weight,
"val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,
"cliff_focal": cfg.cliff_focal,
+ "w_verts": cfg.w_verts, "n_verts": cfg.n_verts,
+ "crop_centre_fix": cfg.crop_centre_fix,
"bound_scales": cfg.bound_scales,
"scale_bound_margin": cfg.scale_bound_margin,
"anomaly_safe_fallback": cfg.anomaly_safe_fallback,
@@ -659,6 +663,10 @@ def train(args, cfg, rank, local_rank, world):
# history.jsonl and you would have to guess from the total.
run = {"total": 0.0, "3d": 0.0, "2d": 0.0, "simcc": 0.0,
"reproj": 0.0, "cam": 0.0, "pose": 0.0}
+ # The surface term is worth its own column: at w_verts=1 it reads
+ # directly as mean PVE in metres, so verts/w_verts*1000 is millimetres.
+ if cfg.w_verts > 0:
+ run["verts"] = 0.0
nb = 0
n_skip = n_skip_run = n_repair = 0
t_epoch = time.time()
@@ -795,6 +803,8 @@ def train(args, cfg, rank, local_rank, world):
run["2d"] += losses.get("loss_2d_native", torch.zeros(())).item()
run["simcc"] += losses.get("loss_2d_simcc", torch.zeros(())).item()
run["reproj"] += losses.get("loss_reproj", torch.zeros(())).item()
+ if "verts" in run:
+ run["verts"] += losses.get("loss_verts", torch.zeros(())).item()
run["cam"] += losses.get("loss_cam", torch.zeros(())).item()
# Under cfg.pose_split, loss_pose is the local half and loss_pose_root
# the other; summing keeps history.jsonl's "pose" column comparable
@@ -1046,6 +1056,25 @@ def parse_args():
"prerequisites, plus absolute-pose supervision. "
"'baseline' reproduces the pre-v2 trainer exactly.")
p.add_argument("--w_reproj", type=float, default=None)
+ p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true",
+ help="Divide the CLIFF crop-centre correction by the zoom. "
+ "Only affects --preset v2 (cliff_follows_aug). Without "
+ "it the box centre the network is told is off by 7.73 px "
+ "mean / 21.0 px p95 at v2 zoom strength. Default "
+ "off so generation 5 reproduces bit-for-bit.")
+ p.add_argument("--w-verts", dest="w_verts", type=float, default=None,
+ help="Weight on the surface (PVE) term: unsquared Euclidean "
+ "distance, in metres, between predicted and teacher mesh "
+ "vertices. 0 (the default) disables it and every code "
+ "path it touches. This is the only geometric loss that "
+ "sees shape_params, which move the skeleton by exactly "
+ "0 cm. Try 0.35, the same scale as --preset v2's "
+ "w_keypoints3d.")
+ p.add_argument("--n-verts", dest="n_verts", type=int, default=None,
+ help="How many mesh vertices the surface term uses: a "
+ "farthest-point subset of the rig's own 18,439-vertex "
+ "mesh (default 595, lod6's count). Costs 0.6 ms per "
+ "batch-64 step against 16.7 ms for the full mesh.")
p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",
help="Orthogonal to --preset, which only controls the input "
"pipeline and the absolute-pose terms. 'rebalanced' applies "
@@ -1084,6 +1113,9 @@ def build_cfg(args):
cfg.epochs = args.epochs
cfg.per_dataset_caps = {} # replaced by the weighted sampler
if args.backbone: cfg.backbone = args.backbone
+ if args.crop_centre_fix: cfg.crop_centre_fix = True
+ if args.w_verts is not None: cfg.w_verts = args.w_verts
+ if args.n_verts is not None: cfg.n_verts = args.n_verts
if args.w_simcc is not None: cfg.w_simcc = args.w_simcc
if args.weight_decay is not None: cfg.weight_decay = args.weight_decay
if args.ema_decay is not None: cfg.ema_decay = args.ema_decay
@@ -1190,6 +1222,9 @@ def main():
f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "
f"finger_w={cfg.finger_weight}"
if args.losses != "legacy" else ""))
+ log(f"surface w_verts={cfg.w_verts}"
+ + (f" over {cfg.n_verts} mesh vertices" if cfg.w_verts > 0 else " (off)")
+ + f" | crop_centre_fix={cfg.crop_centre_fix}")
log(f"stability bound_scales={cfg.bound_scales}"
+ (f" (margin {cfg.scale_bound_margin})" if cfg.bound_scales else "")
+ f" | safe_fallback={cfg.anomaly_safe_fallback}"
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index f91ac95..34649ae 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -45,6 +45,7 @@ os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
import argparse
import io
+import copy
import json
import math
import random
@@ -181,6 +182,9 @@ class DistillConfig:
geom_scale_max: float | None = None # zoom-in ceiling; None => symmetric
# U(1-range, 1+range) as before. 2.0 is
# MeTRAbs's quarter-area truncation bound.
+ # Divide the CLIFF crop-centre correction by the zoom (see the derivation at
+ # the call site). Default off so gen-5 and `--preset baseline` reproduce.
+ crop_centre_fix: bool = False
cliff_follows_aug: bool = False # CLIFF cx/cy/b_scale track the augmented
# crop box, as a real detector box would
cliff_focal: bool = False # perspective-correct CLIFF conditioning:
@@ -318,6 +322,24 @@ class DistillConfig:
w_3d_joints: float = 1e-3 # on the raw 127 skeleton joints (see below)
w_reproj: float = 0.01 # single re-projection of the MHR keypoints
w_simcc: float = 1.0
+ # Surface (PVE) term. `shape_params` move the 127-joint skeleton by exactly
+ # 0.000e+00 cm -- measured -- so every other geometric loss here is blind to
+ # body shape and the mesh is supervised only by w_shape in parameter space.
+ # This term skins the rig and compares vertices, which is the only way a
+ # geometric gradient reaches the 45 identity coefficients.
+ #
+ # n_verts is a farthest-point subset of the rig's OWN 18,439-vertex mesh,
+ # not a separate LOD asset: same topology and same vertex ids as the
+ # teacher, so the correspondence is exact rather than approximate, and the
+ # rig's own skinning op is driven on the subset (verified bit-exact against
+ # the full mesh to 4.6e-05 cm). Measured at batch 64 on an RTX 4070:
+ # skeleton-only FK 4.85 ms, +595 verts 5.44 ms, +all 18,439 verts 21.6 ms --
+ # i.e. the subset costs 0.6 ms (0.3% of a 207 ms step) where the full mesh
+ # costs 16.7 ms (8.1%). 595 is lod6's vertex count.
+ # w_verts = 0.0 disables the term and every code path it touches, so
+ # `--preset baseline` / `--losses legacy` stay bit-identical by default.
+ w_verts: float = 0.0
+ n_verts: int = 595
# Linear ramp (in optimiser steps) for the two losses that back-propagate
# through the MHR forward kinematics, so that loss_pose can pull the
# parameters into a valid region before the geometric terms engage.
@@ -475,6 +497,7 @@ class SAM3DStudentDataset(Dataset):
geom_flip_p: float = 0.5,
geom_scale_max: float | None = None,
cliff_follows_aug: bool = False,
+ crop_centre_fix: bool = False,
cliff_focal: bool = False,
occl_p: float = 0.2, occl_scale: tuple = (0.02, 0.15),
jpeg_p: float = 0.0, jpeg_quality: tuple = (30, 90)):
@@ -489,6 +512,7 @@ class SAM3DStudentDataset(Dataset):
self.geom_flip_p = geom_flip_p
self.geom_scale_max = geom_scale_max
self.cliff_follows_aug = cliff_follows_aug
+ self.crop_centre_fix = crop_centre_fix
self.cliff_focal = cliff_focal
tfm_list = [transforms.Resize((image_size, image_size))]
@@ -706,8 +730,22 @@ class SAM3DStudentDataset(Dataset):
# sized detector box looks like at deployment.
if self.cliff_follows_aug:
crop_w = float(sq_bbox[2] - sq_bbox[0])
- cx -= dx * crop_w / 2.0
- cy -= dy * crop_w / 2.0
+ # The warp is u' = s*R*(u - c) + c + t, so inverting it puts the
+ # visible window's centre at -t/s, not at -t: the shift has to be
+ # divided by the zoom. Without the /s the correction is too big
+ # when zooming in and too small when zooming out -- measured over
+ # 972 augmented samples at v2 strength (geom_trans 0.20,
+ # geom_scale_max 2.0): 11.7 px mean / 33.6 px p95 of error in the
+ # box centre, i.e. 0.61 deg mean / 1.73 deg p95 in the bearing the
+ # network is conditioned on. It is a wrong INPUT, not a wrong
+ # label -- the reprojection chain is exact to 0.000 px -- and it
+ # only affects lateral placement, which PA-MPJPE cannot see.
+ #
+ # Default off so `--preset baseline` and every gen-5 run stay
+ # bit-identical; `--crop-centre-fix` turns it on.
+ z = scale if getattr(self, "crop_centre_fix", False) else 1.0
+ cx -= dx * crop_w / (2.0 * z)
+ cy -= dy * crop_w / (2.0 * z)
b_scale = b_scale / scale
image = self.transform(img_pil)
@@ -768,6 +806,7 @@ def build_dataloaders(cfg):
geom_trans=cfg.geom_trans, geom_flip_p=cfg.geom_flip_p,
geom_scale_max=cfg.geom_scale_max,
cliff_follows_aug=cfg.cliff_follows_aug,
+ crop_centre_fix=cfg.crop_centre_fix,
cliff_focal=cfg.cliff_focal,
occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality)
@@ -997,7 +1036,7 @@ class MHRForwardPass:
per training step and makes the mesh LOD irrelevant to training cost.
"""
- def __init__(self, mhr_path, device, kp_regressor=None):
+ def __init__(self, mhr_path, device, kp_regressor=None, n_verts: int = 0):
self.device = device
self.mhr = torch.jit.load(mhr_path, map_location=device).eval()
for p in self.mhr.parameters():
@@ -1007,6 +1046,48 @@ class MHRForwardPass:
W = torch.as_tensor(kp_regressor, dtype=torch.float32, device=device)
assert W.shape == (70, 127), f"expected a (70, 127) regressor, got {tuple(W.shape)}"
self.kp_regressor = W
+ self.vert_idx = None
+ if n_verts:
+ self._build_vertex_subset(int(n_verts))
+
+ def _build_vertex_subset(self, n: int):
+ """A farthest-point subset of the rig mesh, plus a skinning op for it.
+
+ The rig's `linear_blend_skinning` asserts it is handed all 18,439
+ vertices, so the subset is installed by rewriting its three flattened
+ influence tables -- (vertex, bone, weight) triples -- to reference only
+ the chosen vertices under a compacted index. The op itself is
+ untouched, which is why the result is the full mesh's answer restricted
+ to those vertices rather than an approximation of it.
+
+ Deterministic: the seed vertex is fixed and the input mesh is fixed, so
+ every DDP rank builds the identical subset without communicating.
+ """
+ ct = self.mhr.character_torch
+ rest = ct.mesh.rest_vertices.to(self.device)
+ assert n <= rest.shape[0], f"n_verts={n} exceeds the {rest.shape[0]}-vertex mesh"
+ # Farthest-point sampling in float64: the mesh spans ~180 cm and
+ # neighbouring vertices are sub-millimetre apart, so float32 ties would
+ # make the traversal order depend on accumulation noise.
+ P = rest.double()
+ idx = [0]
+ d = (P - P[0]).norm(dim=-1)
+ for _ in range(n - 1):
+ i = int(d.argmax())
+ idx.append(i)
+ d = torch.minimum(d, (P - P[i]).norm(dim=-1))
+ sub = torch.tensor(sorted(idx), device=self.device)
+
+ lbs = copy.deepcopy(ct.linear_blend_skinning)
+ vi = ct.linear_blend_skinning.vert_indices_flattened
+ keep = torch.isin(vi, sub)
+ remap = torch.full((int(vi.max()) + 1,), -1, dtype=torch.int64, device=self.device)
+ remap[sub] = torch.arange(n, device=self.device)
+ lbs.vert_indices_flattened = remap[vi[keep]]
+ lbs.skin_indices_flattened = ct.linear_blend_skinning.skin_indices_flattened[keep]
+ lbs.skin_weights_flattened = ct.linear_blend_skinning.skin_weights_flattened[keep]
+ lbs.num_vertices = n
+ self.vert_idx, self.vert_lbs = sub, lbs
def get_joints(self, model_params, shape_params):
"""(B, 127, 8) skeleton state in raw MHR units (cm, Y-up)."""
@@ -1017,6 +1098,24 @@ class MHRForwardPass:
skel_state = self.mhr.character_torch.joint_parameters_to_skeleton_state(joint_params)
return skel_state.to(model_params.device)
+ def get_joints_and_vertices(self, model_params, shape_params):
+ """(B, 127, 3) joints and (B, n_verts, 3) skin, both raw MHR cm.
+
+ One forward kinematics pass feeds both. Unlike `get_joints` this passes
+ the REAL identity coefficients rather than zeros -- which changes the
+ skeleton by exactly nothing (verified bit-identical, `torch.equal`),
+ because the 45 identity coefficients are mesh-only -- while the mesh
+ needs them, since they are the only thing this term exists to supervise.
+ """
+ assert self.vert_idx is not None, "MHRForwardPass was built with n_verts=0"
+ ct = self.mhr.character_torch
+ cat = torch.cat([model_params, shape_params], dim=1).to(self.device)
+ skel = ct.joint_parameters_to_skeleton_state(
+ ct.model_parameters_to_joint_parameters(cat))
+ rest = ct.blend_shape(shape_params.to(self.device))[:, self.vert_idx]
+ verts = self.vert_lbs(skel, rest)
+ return skel[..., :3].to(model_params.device), verts.to(model_params.device)
+
@staticmethod
def to_vision(joints):
"""Raw MHR (cm, Y-up/Z-back) -> annotation frame (metres, Y-down/Z-fwd)."""
@@ -1192,14 +1291,30 @@ class DistillationLoss(nn.Module):
else:
losses['loss_cam'] = self.mse(preds["cam_trans"], targets["cam_trans"]) * self.cfg.w_cam
- pred_mhr_joints = self.mhr_module.get_joints(preds["mhr_params"], preds["shape_params"])[..., :3]
+ want_verts = float(getattr(self.cfg, "w_verts", 0.0)) > 0.0
+ if want_verts:
+ pred_mhr_joints, pred_verts = self.mhr_module.get_joints_and_vertices(
+ preds["mhr_params"], preds["shape_params"])
+ else:
+ pred_mhr_joints = self.mhr_module.get_joints(preds["mhr_params"], preds["shape_params"])[..., :3]
with torch.no_grad():
- tgt_mhr_joints = self.mhr_module.get_joints(targets["mhr_model_params"], targets["shape_params"])[..., :3]
+ if want_verts:
+ tgt_mhr_joints, tgt_verts = self.mhr_module.get_joints_and_vertices(
+ targets["mhr_model_params"], targets["shape_params"])
+ else:
+ tgt_mhr_joints = self.mhr_module.get_joints(targets["mhr_model_params"], targets["shape_params"])[..., :3]
c = targets["aug_cos"].view(-1, 1) if "aug_cos" in targets else torch.ones(B, 1, device=ones.device)
s = targets["aug_sin"].view(-1, 1) if "aug_sin" in targets else torch.zeros(B, 1, device=ones.device)
xm, ym = tgt_mhr_joints[..., 0], tgt_mhr_joints[..., 1]
tgt_mhr_joints = torch.stack(
[c * xm - s * ym, s * xm + c * ym, tgt_mhr_joints[..., 2]], dim=-1)
+ if want_verts:
+ # The same in-plane rotation the joints get: the dataset rotates
+ # the LABELS, not the target parameters, so the FK'd target has
+ # to be brought into the augmented frame.
+ xv, yv = tgt_verts[..., 0], tgt_verts[..., 1]
+ tgt_verts = torch.stack(
+ [c * xv - s * yv, s * xv + c * yv, tgt_verts[..., 2]], dim=-1)
# KEPT (w=1e-3) even though loss_3d_native now covers the same parameters.
# It is not redundant: the best linear reconstruction of the 127 skeleton
@@ -1210,6 +1325,16 @@ class DistillationLoss(nn.Module):
losses['loss_mhr_joints'] = self.m_smooth_l1(
pred_mhr_joints, tgt_mhr_joints.detach(), m_noflip) * self.cfg.w_3d_joints
+ if want_verts:
+ # Per-vertex unsquared Euclidean distance in METRES, so the weight is
+ # on the same scale as w_keypoints3d under kp3d_loss="euclid" and the
+ # logged value reads as mean PVE / 1000. Masked to m_noflip for the
+ # same reason loss_mhr_joints is: under a horizontal flip the target
+ # PARAMETERS are not mirrored, only the 2D and 3D labels are.
+ e_verts = (pred_verts - tgt_verts.detach()).norm(dim=-1).mean(1) / 100.0
+ losses['loss_verts'] = (self._mmean(e_verts, m_noflip)
+ * self.cfg.w_verts * self.fk_scale)
+
# The 70 native keypoints ARE the mesh now: an affine function of the
# 127 skeleton joints, so they are differentiable w.r.t. mhr_params and
# carry no extra forward pass (pred_mhr_joints is already computed).
@@ -1332,7 +1457,7 @@ class DistillationLoss(nn.Module):
# are the only unbounded ones: the parameter regressions are bounded by the
# target magnitudes and the 2D terms are bounded by construction (SimCC spans
# +/-kp2d_range and its CE by log(kp2d_bins)).
-FK_LOSS_KEYS = ("loss_3d_native", "loss_mhr_joints", "loss_reproj")
+FK_LOSS_KEYS = ("loss_3d_native", "loss_mhr_joints", "loss_reproj", "loss_verts")
def safe_loss_subset(losses):
diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.py
index c2a3a4d..1fee455 100644
--- a/tools/pth_to_onnx.py
+++ b/tools/pth_to_onnx.py
@@ -35,7 +35,7 @@ Usage
# or point at a specific checkpoint / output
python ./tools/pth_to_onnx.py \
- --ckpt ./instanthmr_distill_train/runs/b2_s0/b2_s0/best_student_model_ema.pth \
+ --ckpt ./instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth \
--output ./models/instanthmr_mhr_only.onnx
"""
import argparse