Python
feat: surface (PVE) loss on a bit-exact rig-mesh vertex subset, gated v2 crop-centre fix, and the five-split harmony4d mixture
mohamdevdfa7a2daebca
diff --git a/CLAUDE.md b/CLAUDE.mdindex 8bcf048..85fc5e9 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -167,6 +167,22 @@ steps. Use `--anomaly-safe-fallback`, which steps on the non-FK terms insteaddivergence here by lowering the LR alone: `OneCycleLR(pct_start=0.1)` peaks at10% of the run, so a lower peak only moves *when* it dies.+**Body shape is invisible to every geometric loss, and the fix is a vertex+term.** `MHRForwardPass.get_joints` zeroes the 45 `shape_params`, and that is+not a shortcut: they move the 127-joint skeleton by *exactly* `0.000e+00 cm`.+Measured `d(loss)/d(shape_params)` is `8.9e-03` through vertices and+`0.000e+00` through joints, so `--w-verts` is the ONLY route by which shape gets+a geometric gradient; ignoring identity costs 5.95 mm mean / 22.68 mm max PVE.+Do **not** reach for `models/mhr_assets/assets/lod6.fbx` to make it cheap — it+needs `pymomentum` (which segfaults here) and it is a different topology, so the+teacher correspondence would have to be fitted. `n_verts` instead takes a+farthest-point subset of the rig's OWN 18,439-vertex mesh and rewrites the three+flattened influence tables of the rig's own `linear_blend_skinning`; the op is+untouched, so the answer is bit-exact against the full mesh (4.6e-05 cm) and the+vertex ids match the teacher's. 595 vertices cost +0.6 ms per batch-64 step+against 16.7 ms for all 18,439. `loss_verts` belongs in `FK_LOSS_KEYS` — a+blown-up bone scale explodes vertices exactly as it explodes joints.+**`parameter_transform[:, :204]` has rank 192, not 204.** Twelve directionsmove the parameters and leave the skeleton *exactly* unchanged — each`*_flexible` parameter against its `scale_*` partner, plus the spine rotations.diff --git a/docs/todo.md b/docs/todo.mdindex 32a3832..3847459 100644--- a/docs/todo.md+++ b/docs/todo.md@@ -9,7 +9,7 @@ a number appears, the command that produced it is named.----## Done (2026-09-06)+## Done (2026-09-06 / 09-07)### 1. The ONNX exporter dropped the scale bounds — 21 mm, silently@@ -63,51 +63,46 @@ this corrupts. The `bound_scales` half is where the 14–21 mm lives.---+### 3. The v2 crop-centre update forgot to divide by the zoom — fixed 2026-09-07++`train_distill_mhr_only.py` wrote `cx -= dx * crop_w / 2.0`. The augmentation+warps the crop with `u' = s*R*(u - c) + c + t`, so inverting it puts the visible+window's centre at `-t/s`: the shift has to be divided by the zoom. It+over-corrected when zooming in and under-corrected when zooming out, and it was+live in every `--preset v2` run (`geom_trans=0.20`, `geom_scale_max=2.0`).++**Fixed behind `--crop-centre-fix`** (config `crop_centre_fix`), default off so+generation 5 and `--preset baseline` stay bit-identical — verified: every loss+term matches `git show HEAD` to the last printed digit on 48 real augmented+samples, under both `baseline/legacy` and `v2/rebalanced`.++**Verified on the fix itself.** Reference is the exact preimage of the crop+centre under the real `M_total` that `cv2.warpAffine` was called with, over 400+augmented COCO crops at v2 strength with rotation disabled so the zoom is the+only variable:++| | box-centre error, mean | p95 | max |+|---|---|---|---|+| `crop_centre_fix=False` | 7.73 px | 21.00 px | 34.93 px |+| `crop_centre_fix=True` | **0.68 px** | 1.12 px | 1.50 px |++The 0.68 px residual is the bbox-rounding floor.++**With rotation on (±30°) a ~31 px residual remains against that reference, and+it is NOT this bug** — the fix moves it only 33.40 → 30.93 px. The code models+the in-plane rotation as a camera roll (it rotates `joints_3d`, `cam_trans` and+the box centre about the image centre together), for which "preimage of the crop+centre" is the wrong target, so most of that 31 px is the reference, not a+defect. Expressing the `dx`/`dy` correction in the rotated axes instead changes+the distance from the box centre to the actual body centre by 63.75 → 63.15 px,+i.e. nothing. **No second defect demonstrated; see item 11 for what is still+open.**+## QueueOrdered by (cost of being wrong) x (cheapness of the fix). Items 3, 4 and 6 aredefects; 5 and 7 onward are opportunities the audit raised that are not bugs.-### 3. The v2 crop-centre update forgets to divide by the zoom — **in flight**--`train_distill_mhr_only.py:678` writes `cx -= dx * crop_w / 2.0`. The-augmentation warps the crop with `u' = s*R*(u - c) + c + t`; inverting it puts-the visible window's centre at `-t/s`, so the correction needs the `/s`. It-over-corrects when zooming in and under-corrects when zooming out. Live in-every `--preset v2` run (`geom_trans=0.20`, `geom_scale_max=2.0`).--**Measured** over 972 augmented COCO/AIC/MPII samples drawn from the real v2-distribution:--| | px err mean | px p95 | deg mean | deg p95 | deg max |-|---|---|---|---|---|---|-| all | 11.7 | 33.6 | 0.61 | 1.73 | 3.21 |--0.6° of error in the bearing the network is told, on average — 4% of the crop's-own angular half-width, 13% at worst.--**It is a wrong input, not a wrong label.** Verified: feeding the teacher's own-3D joints and `cam_trans` through the reprojection loss's full geometry-(counter-rotate, un-mirror, project in the original frame, apply `aug_M`)-against the augmented 2D targets, over 64 samples with 34 flipped, gives a-residual of **0.000 px**. The augmented geometry chain is exact. And the third-conditioning channel — angular size, `b_scale / scale`, the one that sets depth-and motivated `--cliff-focal` — is already correct. The damage is confined to-lateral placement, which PA-MPJPE cannot see.--**Fix** (one character per line):--```python-cx -= dx * crop_w / (2.0 * scale)-cy -= dy * crop_w / (2.0 * scale)-```--**Do not patch this on the cluster while runs are chaining.**-`52_train_ddp.slurm` resubmits itself on walltime, so an rsync'd trainer changes-the augmentation partway through a 300-epoch run. Land it for the next-generation, and note in `STATUS.md` that generation 5 carried the uncorrected-version.-### 4. Validation drops the frames the model failed on`val3dpw.py:179` removes non-finite predictions before scoring — necessary,@@ -166,6 +161,93 @@ that they *help* the global token via self-attention. Sequence: finish theruns, score both routes, then a separate ablation with the 70 queries removed.(`--detach-2d-head` is gone; it was tried and rejected, see the trainer README.)+### 5b. Late divergence the loss guard cannot see (`bno_s1`, job 1813482)++**Open. This killed a healthy run at epoch 92 of 300 and nothing caught it.**++`bno_s1` reached 45.77 mm J14 PA at epoch 84 and was dead by 92, aborting on+`still stuck after 3 EMA rollback(s)`. Its twin `bno_s0`, identical config and+a different seed, skipped exactly **one** step in 100 epochs.++This is **not** the generation-4 failure. `--bound-scales` held: the anomalous+losses are 50-175, not the 4e7 / 8.8e13 of the size-parameter blowup, and the+skeleton stayed metre-scale. PA-MPJPE removes scale, so 341 mm means the+**pose** was destroyed — the 124 rotations at `mhr_params[6:130]`, which+`--bound-scales` deliberately leaves free.++It is also not the learning rate. The collapse ran at `lr 6.8e-5`, a quarter of+nominal, well past OneCycle's epoch-30 peak.++**The precursor, nine epochs early, that nobody was watching.** Epochs 75-84 ran+with `skipped 0/1562` every single epoch — no NaN, no loss over the threshold —+while the two validation branches disagreed wildly:++| epoch | RAW J14 PA | EMA J14 PA | skipped |+|---|---|---|---|+| 75 | 53.0 | 47.7 | 0 |+| 76 | 51.7 | **188.9** | 0 |+| 77 | 48.4 | 123.4 | 0 |+| 78 | 49.3 | 69.5 | 0 |+| 79 | 47.4 | 50.8 | 0 |+| 80-84 | 47-48 | 45.8-48.7 | 0 |+| **85** | **268.7** | 47.1 | 67 |+| 86 | 302.3 | 321.0 | 0 |++RAW is sampled once per epoch; the EMA integrates every step. So the epoch-76+spike says the weights took a large excursion *inside* an epoch that the+end-of-epoch snapshot missed entirely and the loss guard never saw. The EMA+decayed it out over ~3 epochs (half-life `ln2/(1-0.9998)` = 3,466 steps ≈ 2.2+epochs, which matches), everything looked fine again, and nine epochs later the+run died for real.++**`EMA / RAW` on the validation metric is a free excursion detector and is not+monitored.** Add it to the epoch line and to `summarize_runs.py`; a ratio above+~1.5 with `skipped 0` means the weights are wandering between snapshots.++**Each rollback made things immediately worse.** Both visible rollbacks are+followed by a markedly worse epoch:++| rollback | epoch after | train total | skipped |+|---|---|---|---|+| attempt 2 (end of ep 73) | 74 | 3.19 (was 2.64) | 74 |+| attempt 3 (end of ep 84) | 85 | **7.99** (was 2.60) | 67 |++Attempt 3 restored a **45.8 mm** EMA at `lr 6.8e-5` and the model was at 268 mm+one epoch later. Restoring good weights and then destroying them within 1,562+steps is not a learning-rate problem — the recovery mechanism is a prime+suspect, not just the failure.++**Ranked hypotheses, and how to test each.**++1. **The Adam restart in the rollback.** `optimizer.state.clear()` zeroes both+ moments, so the very next step is `lr * m̂/(√v̂+ε) = lr * sign(g)` for **all**+ 40M parameters at once — a coordinated jump of norm `lr * √(4e7)` ≈ 0.43 at+ this LR, against the heavily damped updates steady-state Adam produces.+ *Test:* keep the moments, or re-warm the LR over ~200 steps after a+ rollback, and see whether the post-rollback epoch still degrades.+2. **The rotations are genuinely unbounded.** `--bound-scales` covers `0:3` and+ `130:204` only. `6:130` cannot blow up the skeleton's *size* but can produce+ arbitrary pose. *Test:* log `mhr_params[6:130]` percentiles per epoch; if+ they drift monotonically through epochs 75-84 the wandering is there.+3. **The guard's trigger is the wrong quantity.** It fires on+ `loss > 100 or non-finite`. Epochs 75-84 satisfied neither while the model+ visited bad regions. *Test:* trigger on gradient norm or on a weight-delta+ norm instead of, or as well as, the loss.++**Two log defects found while diagnosing this, both worth fixing:**++* `bad` is all-reduced with `ReduceOp.MAX` (`train_distill_jz.py:682`) but the+ message prints **rank 0's own** loss. That is why the log shows+ `skipping step 57 (loss 10.8)` against a threshold of 100 — a healthy rank+ vetoed by a sick one. Print the reduced maximum, or the vetoing rank.+* `📈 epoch` reports RAW and EMA side by side but nothing flags their+ divergence, which was the only visible symptom for nine epochs.++**Ruled out.** The safe-fallback path was *not* masking the collapse: `repaired`+fired **once** in the entire run, and only one epoch shows a `repaired` count.+The concern filed as item 6 is real bookkeeping sloppiness but had nothing to do+with this failure.+### 6. `n_repair` overstates successful updates`GradScaler.step` silently skips the optimiser when gradients overflow, but@@ -213,8 +295,12 @@ body shape.**Consequence for claims, today**: PA-MPJPE can improve while the mesh getsworse. Do not say "better mesh" on the strength of a pose number.-**Consequence for work, later**: a surface term and a surface metric, with a-valid cross-topology correspondence.++**Half-addressed 2026-09-07 by `--w-verts`** (see item 13): the surface *term* now+exists and is the only loss whose gradient reaches `shape_params` — measured+`d(loss)/d(shape)` is `8.9e-03` through vertices and **exactly 0.000e+00**+through joints. What is still missing is the surface *metric*: nothing in+`benchmark/` reports PVE, so the claim side of this item is unchanged.### 10. `fy` for both axes — checked, not a problem here@@ -231,6 +317,124 @@ The `bno` / `dt` mix is `aic/coco/mpii` only — perfectly isotropic — and 3DPdiffers by 0.4%. Revisit only if a new split with anisotropic pixels enters themix.++### 11. The rotation half of the CLIFF crop-centre correction is unverified++Item 3 fixed the zoom. The rotation is a separate question and it was **not**+resolved: with `geom_rot_deg=30` a ~31 px box-centre residual remains against a+preimage-of-crop-centre reference, and the zoom fix moves it by only 2.5 px.++Two readings, and the measurement does not separate them. Either the code is+right — it treats the in-plane rotation as a camera roll, rotating `joints_3d`,+`cam_trans` and the box centre about the image centre together, and under that+model the box is *supposed* to move, so the reference is simply wrong — or there+is a real error hiding under a reference too loose to see it. What was ruled+out: expressing the `dx`/`dy` correction in the rotated axes rather than the+unrotated ones, which changes the box-centre-to-body-centre distance from 63.75+to 63.15 px, i.e. nothing.++**To settle it** the reference has to come from the camera-roll model itself,+not from the warp: construct a synthetic camera, roll it by the augmentation+angle, and check that the conditioning the dataset emits equals the conditioning+that rolled camera would produce for the same box. Until then, do not "fix" the+rotation — the current form is not shown to be wrong.++### 12. Benchmark HGNetv2 against RepViT — after the baseline lands++Not to be run now. `repvit_m2_3` is the incumbent and generation 6 exists to+give it a solid, uncontaminated baseline on the five-split corpus; swapping the+trunk at the same time would confound both questions.++When it is time, the comparison has to cover all four axes, because a backbone+that wins on one can lose the deployment:++* **Model sizes** — which HGNetv2 variants exist and their parameter counts+ against `repvit_m2_3`'s. `--backbone` already takes any timm name, and+ `config_from_checkpoint` reads it back out of `run_config.json`, so no code+ change is needed to try one.+* **Accuracy** — J14 PA-MPJPE with the adapter, 2 to 3 seeds. The documented+ seed spread is 34–49 mm on the 70-keypoint metric, so a single run proves+ nothing.+* **Speed** — ONNX latency on the deployment target, not PyTorch training+ throughput. `tools/bench_student_arch.py` is the existing harness.+* **Capabilities** — HGNetv2 is a plain conv stack where RepViT is+ reparameterisable. Check that the export path survives (`tools/pth_to_onnx.py`+ has a torch↔ONNX parity gate that exits if the worst output differs by more+ than 1e-3) and that `backbone.forward_features` returns something the+ `feat_proj` reshape accepts — the trainer handles both 4-D and 3-D feature+ maps, but the channel count feeds `backbone_feat_dim` and that is a config+ field, not inferred.+* **On our data, locally** — the local corpus plus `checkpoints/mhr_model.pt` is+ enough to run the 1-batch overfit test and a short run per backbone before+ spending cluster time. Judge the overfit at ~1k steps.++### 13. PVE has no metric yet — only a loss++`--w-verts` supervises the surface (it is the fix half of item 9), but nothing+*reports* it. Competitors quote PVE, so a submission needs it.++The pieces are in place and the awkward part is already solved: the vertex+subset is drawn from the rig's **own** 18,439-vertex mesh, not from a separate+LOD asset, so predicted and teacher vertices share topology and vertex ids and+the correspondence is exact — no cross-topology fitting, which is what makes PVE+comparisons between rigs contentious. `MHRForwardPass.get_joints_and_vertices`+returns them.++**What is missing**: 3DPW ships SMPL, and MHR is not SMPL. A published PVE+number needs an MHR→SMPL surface correspondence with the same care the J14+adapter needed — float64 and a truncated `lstsq`, fitted on train sequences and+never refitted on the reported split. Until that exists, PVE can be reported+against the *teacher* on held-out crops, which measures distillation fidelity+and is honest as long as it is labelled as such.++**Do not raise `n_verts` for the metric.** 595 vertices is a sampling choice for+the loss; a reported PVE should use all 18,439, which costs 21.6 ms per batch-64+forward and is irrelevant at evaluation time.++### 14. Do the tanh size bounds block extreme poses? — measured, they do not++Raised because `vid4.mp4` looked poorly fitted after `--bound-scales` landed.+**The bound cannot be the cause, and here is why.**++`--bound-scales` squashes two slices through a `tanh`: `mhr_params[0:3]` (root+translation) and `[130:204]` (the six `*_flexible` size parameters and the 68+bone scales) — 77 parameters, all of them body SIZE. The 124 local joint+**rotations** at `[6:130]`, which is what a yoga pose actually is, and the root+rotation at `[3:6]` are **not bounded at all**.++Measured over 4,000 teacher annotations from each of the six local splits —+1,848,000 bounded targets in total:++| split | targets outside the bound | worst \|atanh\| | max \|rotation param\| |+|---|---|---|---|+| 3dpw | 0.0000% | 0.63 | 2.64 |+| aic | 0.0000% | 0.90 | 2.94 |+| coco | 0.0000% | 0.97 | 2.94 |+| harmony4d | 0.0000% | 0.59 | 2.56 |+| mpii | 0.0000% | 0.80 | 2.94 |+| distill_mix | 0.0000% | 1.63 | 2.84 |+| **all** | **0 / 1,848,000** | **1.63** | |++Not one target is clipped, and the worst sits at `|atanh| = 1.63`, i.e.+`tanh = 0.926` — a value a linear head reaches without saturating. The one real+cost is gradient damping, `1 - tanh²`, and on the teacher distribution that is+0.994x at the median, 0.875x at p99 and 0.733x at worst. Harmless.++**So look elsewhere for the `vid4.mp4` regression.** Two candidates, in order:++1. **A stale ONNX.** `models/instanthmr_mhr_only_ckpt90.onnx` and+ `models/instanthmr.onnx` carry **no `metadata_props` at all**, so+ `instanthmr.inference` cannot know how they were conditioned and falls back+ to the pixel-CLIFF form with a `1.00 x diag` focal. Feeding a focal-aware+ model that vector raises nothing and silently mis-places the person in depth.+ Every 2026-09-07 export (`models/instanthmr_mhr_only.onnx`, `models/gen5/*`)+ does carry `cliff_focal=true, bound_scales=true`. **Re-render `vid4.mp4` with+ a stamped export before treating this as a model regression.**+2. **The deployment focal fallback.** With no annotation to read a focal from,+ `demo.py` uses `1.05 x diag` for focal-aware graphs. If the real camera is+ far from that, depth is wrong — `--focal PX` overrides it.++---## Evaluation hygiene the audit raiseddiff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.mdindex 5a349df..1867bb5 100644--- a/instanthmr_distill_train/README.md+++ b/instanthmr_distill_train/README.md@@ -291,6 +291,117 @@ Confirmed directly: pushing `scale_spine_length` to 20.3 along a null directionmoves the skeleton by 1.2e-7 m. The bound now keeps that wandering finite;collapsing the redundancy properly would be a separate change.+## `--w-verts` — the surface (PVE) term (added 2026-09-07)++Every geometric loss in this trainer is blind to body shape. `get_joints` zeroes+the 45 `shape_params` before the forward pass, and that is not a shortcut — the+identity coefficients move the 127-joint skeleton by **exactly** `0.000e+00 cm`.+They are mesh-only. So `w_shape` in parameter space was the only thing+supervising them, and at `--losses rebalanced` that weight is 0.03, carrying+0.2% of the trunk gradient.++`--w-verts W` adds a term that skins the rig and compares vertices, which is the+only route by which a geometric gradient can reach shape. Measured+`d(loss)/d(shape_params)`: **8.9e-03** through vertices, **exactly 0.000e+00**+through joints.++**What it costs to ignore shape.** Running the rig on real teacher parameters+with the identity zeroed against the true identity: **5.95 mm mean, 22.68 mm max+per-vertex error**. That is a floor a shape-blind model cannot perceive, let+alone fix.++### It uses the rig's own mesh, not a separate LOD++The obvious way to make this cheap is to load `lod6.fbx` (595 vertices against+lod0's 18,439). Don't — it needs `pymomentum` to parse, which segfaults against+the torch in this environment, and lod6 is a *different topology*, so the+teacher correspondence would have to be fitted rather than being exact.++Instead `n_verts` picks a farthest-point subset of the rig's **own** 18,439+vertices and rewrites the three flattened influence tables of the rig's own+`linear_blend_skinning` — `(vertex, bone, weight)` triples — to reference only+those, under a compacted index. The skinning op is untouched, so the result is+the full mesh's answer restricted to those vertices, **verified bit-exact to+4.6e-05 cm**. Same topology, same vertex ids as the teacher: the correspondence+is exact, not fitted. The default `n_verts=595` is lod6's vertex count, chosen+so the sampling density matches what the LOD would have given.++Measured at batch 64 on an RTX 4070:++| path | fwd+bwd | vs skeleton-only |+|---|---|---|+| skeleton FK only (what the trainer already did) | 4.85 ms | — |+| + 595-vertex subset | 5.44 ms | **+0.6 ms** |+| + all 18,439 vertices | 21.6 ms | +16.7 ms |++Against a 207 ms step that is **+0.3%** for the subset and +8.1% for the full+mesh. On a real training step, end to end: **+0.9% wall time and +0 MiB peak+GPU** — the backbone activations dominate and the vertex tensors fit in slack.++### Choosing the weight++The term is an unsquared Euclidean distance in **metres**, the same form as+`loss_3d_native` under `--losses rebalanced`, so the weights are directly+comparable and `loss_verts / w_verts * 1000` reads as mean PVE in millimetres.+Per-term gradient norm on the trunk, pretrained backbone, `v2 + rebalanced`:++| term | \|grad\| | share |+|---|---|---|+| `loss_reproj` | 33.97 | 61.3% |+| `loss_verts` (at w=1.0) | 7.21 | 13.0% |+| `loss_cam` | 4.51 | 8.1% |+| `loss_3d_native` | 3.74 | 6.7% |+| `loss_pose` | 2.93 | 5.3% |+| `loss_shape` | 0.12 | **0.2%** |++**`--w-verts 0.35` is the recommended setting**: `|grad| = 2.52`, 5.0% of the+update, just under `loss_3d_native` and nowhere near `loss_reproj`. At 1.0 it+would be the second-largest term in the budget, which is not what a first trial+should do.++`loss_verts` is in `FK_LOSS_KEYS`, so `--anomaly-safe-fallback` drops it along+with the other forward-kinematics terms on an anomalous batch — a blown-up bone+scale explodes the vertices exactly as it explodes the joints. It is also+multiplied by `fk_scale`, so it ramps over `kp3d_warmup_steps` (2000) like+`loss_3d_native` and `loss_reproj`; a PVE read from the first epoch is+suppressed by that ramp, not genuinely small.++`w_verts = 0.0` is the default and disables the term **and every code path it+touches** — `MHRForwardPass` is then constructed with `n_verts=0` and builds no+subset at all. Verified: every loss term matches `git show HEAD` to the last+printed digit on 48 real augmented samples, under both `baseline/legacy` and+`v2/rebalanced`.++**There is no PVE metric yet, only the loss.** See `docs/todo.md` item 13.++## `--crop-centre-fix` (added 2026-09-07)++`--preset v2` sets `cliff_follows_aug`, which moves the CLIFF conditioning to+describe the *augmented* box. The crop-centre half of that correction was+missing a division by the zoom: the warp is `u' = s*R*(u - c) + c + t`, so+inverting it puts the visible window's centre at `-t/s`, not `-t`.++Verified against the exact preimage of the crop centre under the real+`M_total` that `cv2.warpAffine` was called with, over 400 augmented COCO crops+at v2 strength, rotation disabled so the zoom is the only variable:++| | mean | p95 | max |+|---|---|---|---|+| off (the gen-5 behaviour) | 7.73 px | 21.00 px | 34.93 px |+| **on** | **0.68 px** | 1.12 px | 1.50 px |++The residual is the bbox-rounding floor. **Default off** so generation 5 and+`--preset baseline` reproduce bit-for-bit; it is a no-op under `--preset+baseline`, which does not set `cliff_follows_aug` at all. Pass it for every new+`v2` run.++This is a wrong **input**, not a wrong label — the reprojection chain is exact+to 0.000 px — and it only affects lateral placement, which PA-MPJPE cannot see.+Do not expect the benchmark to move; expect the conditioning to stop lying.++The rotation half of the same correction is **not** settled: see `docs/todo.md`+item 11.+## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)**Do not retry this without reading the failure mode.** The flag is gone from@@ -485,5 +596,15 @@ python3 train_distill.py \--no-export # skip ONNX export/quantization after training```+On the cluster, flags go through `EXTRA_TRAIN_ARGS` on+`datasets_pipeline/jeanzay/52_train_ddp.slurm`, not on a bare command line. The+generation-6 set is:++```+--preset {baseline|v2} --losses rebalanced \+--cliff-focal --bound-scales --anomaly-safe-fallback --crop-centre-fix \+--w-verts {0|0.35}+```+Training auto-resumes from `runs/<name>/best_student_model_v3.pth` if present.Outputs (best checkpoints + `export/*.onnx`) land under `--output_dir`.diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.pyindex 82af302..9d3d380 100755--- a/instanthmr_distill_train/train_distill_jz.py+++ b/instanthmr_distill_train/train_distill_jz.py@@ -332,6 +332,7 @@ def build_jz_loaders(cfg, args, rank: int, world: int):geom_trans=cfg.geom_trans, geom_flip_p=cfg.geom_flip_p,geom_scale_max=cfg.geom_scale_max,cliff_follows_aug=cfg.cliff_follows_aug,+ crop_centre_fix=cfg.crop_centre_fix,cliff_focal=cfg.cliff_focal,occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality)@@ -480,7 +481,8 @@ def train(args, cfg, rank, local_rank, world):out_dir.mkdir(parents=True, exist_ok=True)W = np.load(cfg.kp_regressor_path)- mhr_module = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W)+ mhr_module = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=W,+ n_verts=cfg.n_verts if cfg.w_verts > 0 else 0)criterion = T.DistillationLoss(cfg, mhr_module)train_loader, val_loader, sampler, _ = build_jz_loaders(cfg, args, rank, world)@@ -645,6 +647,8 @@ def train(args, cfg, rank, local_rank, world):"finger_weight": cfg.finger_weight,"val_3dpw_gt": args.val_3dpw_gt if args.val_3dpw else None,"cliff_focal": cfg.cliff_focal,+ "w_verts": cfg.w_verts, "n_verts": cfg.n_verts,+ "crop_centre_fix": cfg.crop_centre_fix,"bound_scales": cfg.bound_scales,"scale_bound_margin": cfg.scale_bound_margin,"anomaly_safe_fallback": cfg.anomaly_safe_fallback,@@ -659,6 +663,10 @@ def train(args, cfg, rank, local_rank, world):# history.jsonl and you would have to guess from the total.run = {"total": 0.0, "3d": 0.0, "2d": 0.0, "simcc": 0.0,"reproj": 0.0, "cam": 0.0, "pose": 0.0}+ # The surface term is worth its own column: at w_verts=1 it reads+ # directly as mean PVE in metres, so verts/w_verts*1000 is millimetres.+ if cfg.w_verts > 0:+ run["verts"] = 0.0nb = 0n_skip = n_skip_run = n_repair = 0t_epoch = time.time()@@ -795,6 +803,8 @@ def train(args, cfg, rank, local_rank, world):run["2d"] += losses.get("loss_2d_native", torch.zeros(())).item()run["simcc"] += losses.get("loss_2d_simcc", torch.zeros(())).item()run["reproj"] += losses.get("loss_reproj", torch.zeros(())).item()+ if "verts" in run:+ run["verts"] += losses.get("loss_verts", torch.zeros(())).item()run["cam"] += losses.get("loss_cam", torch.zeros(())).item()# Under cfg.pose_split, loss_pose is the local half and loss_pose_root# the other; summing keeps history.jsonl's "pose" column comparable@@ -1046,6 +1056,25 @@ def parse_args():"prerequisites, plus absolute-pose supervision. ""'baseline' reproduces the pre-v2 trainer exactly.")p.add_argument("--w_reproj", type=float, default=None)+ p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true",+ help="Divide the CLIFF crop-centre correction by the zoom. "+ "Only affects --preset v2 (cliff_follows_aug). Without "+ "it the box centre the network is told is off by 7.73 px "+ "mean / 21.0 px p95 at v2 zoom strength. Default "+ "off so generation 5 reproduces bit-for-bit.")+ p.add_argument("--w-verts", dest="w_verts", type=float, default=None,+ help="Weight on the surface (PVE) term: unsquared Euclidean "+ "distance, in metres, between predicted and teacher mesh "+ "vertices. 0 (the default) disables it and every code "+ "path it touches. This is the only geometric loss that "+ "sees shape_params, which move the skeleton by exactly "+ "0 cm. Try 0.35, the same scale as --preset v2's "+ "w_keypoints3d.")+ p.add_argument("--n-verts", dest="n_verts", type=int, default=None,+ help="How many mesh vertices the surface term uses: a "+ "farthest-point subset of the rig's own 18,439-vertex "+ "mesh (default 595, lod6's count). Costs 0.6 ms per "+ "batch-64 step against 16.7 ms for the full mesh.")p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",help="Orthogonal to --preset, which only controls the input ""pipeline and the absolute-pose terms. 'rebalanced' applies "@@ -1084,6 +1113,9 @@ def build_cfg(args):cfg.epochs = args.epochscfg.per_dataset_caps = {} # replaced by the weighted samplerif args.backbone: cfg.backbone = args.backbone+ if args.crop_centre_fix: cfg.crop_centre_fix = True+ if args.w_verts is not None: cfg.w_verts = args.w_verts+ if args.n_verts is not None: cfg.n_verts = args.n_vertsif args.w_simcc is not None: cfg.w_simcc = args.w_simccif args.weight_decay is not None: cfg.weight_decay = args.weight_decayif args.ema_decay is not None: cfg.ema_decay = args.ema_decay@@ -1190,6 +1222,9 @@ def main():f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "f"finger_w={cfg.finger_weight}"if args.losses != "legacy" else ""))+ log(f"surface w_verts={cfg.w_verts}"+ + (f" over {cfg.n_verts} mesh vertices" if cfg.w_verts > 0 else " (off)")+ + f" | crop_centre_fix={cfg.crop_centre_fix}")log(f"stability bound_scales={cfg.bound_scales}"+ (f" (margin {cfg.scale_bound_margin})" if cfg.bound_scales else "")+ f" | safe_fallback={cfg.anomaly_safe_fallback}"diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex f91ac95..34649ae 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -45,6 +45,7 @@ os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")import argparseimport io+import copyimport jsonimport mathimport random@@ -181,6 +182,9 @@ class DistillConfig:geom_scale_max: float | None = None # zoom-in ceiling; None => symmetric# U(1-range, 1+range) as before. 2.0 is# MeTRAbs's quarter-area truncation bound.+ # Divide the CLIFF crop-centre correction by the zoom (see the derivation at+ # the call site). Default off so gen-5 and `--preset baseline` reproduce.+ crop_centre_fix: bool = Falsecliff_follows_aug: bool = False # CLIFF cx/cy/b_scale track the augmented# crop box, as a real detector box wouldcliff_focal: bool = False # perspective-correct CLIFF conditioning:@@ -318,6 +322,24 @@ class DistillConfig:w_3d_joints: float = 1e-3 # on the raw 127 skeleton joints (see below)w_reproj: float = 0.01 # single re-projection of the MHR keypointsw_simcc: float = 1.0+ # Surface (PVE) term. `shape_params` move the 127-joint skeleton by exactly+ # 0.000e+00 cm -- measured -- so every other geometric loss here is blind to+ # body shape and the mesh is supervised only by w_shape in parameter space.+ # This term skins the rig and compares vertices, which is the only way a+ # geometric gradient reaches the 45 identity coefficients.+ #+ # n_verts is a farthest-point subset of the rig's OWN 18,439-vertex mesh,+ # not a separate LOD asset: same topology and same vertex ids as the+ # teacher, so the correspondence is exact rather than approximate, and the+ # rig's own skinning op is driven on the subset (verified bit-exact against+ # the full mesh to 4.6e-05 cm). Measured at batch 64 on an RTX 4070:+ # skeleton-only FK 4.85 ms, +595 verts 5.44 ms, +all 18,439 verts 21.6 ms --+ # i.e. the subset costs 0.6 ms (0.3% of a 207 ms step) where the full mesh+ # costs 16.7 ms (8.1%). 595 is lod6's vertex count.+ # w_verts = 0.0 disables the term and every code path it touches, so+ # `--preset baseline` / `--losses legacy` stay bit-identical by default.+ w_verts: float = 0.0+ n_verts: int = 595# Linear ramp (in optimiser steps) for the two losses that back-propagate# through the MHR forward kinematics, so that loss_pose can pull the# parameters into a valid region before the geometric terms engage.@@ -475,6 +497,7 @@ class SAM3DStudentDataset(Dataset):geom_flip_p: float = 0.5,geom_scale_max: float | None = None,cliff_follows_aug: bool = False,+ crop_centre_fix: bool = False,cliff_focal: bool = False,occl_p: float = 0.2, occl_scale: tuple = (0.02, 0.15),jpeg_p: float = 0.0, jpeg_quality: tuple = (30, 90)):@@ -489,6 +512,7 @@ class SAM3DStudentDataset(Dataset):self.geom_flip_p = geom_flip_pself.geom_scale_max = geom_scale_maxself.cliff_follows_aug = cliff_follows_aug+ self.crop_centre_fix = crop_centre_fixself.cliff_focal = cliff_focaltfm_list = [transforms.Resize((image_size, image_size))]@@ -706,8 +730,22 @@ class SAM3DStudentDataset(Dataset):# sized detector box looks like at deployment.if self.cliff_follows_aug:crop_w = float(sq_bbox[2] - sq_bbox[0])- cx -= dx * crop_w / 2.0- cy -= dy * crop_w / 2.0+ # The warp is u' = s*R*(u - c) + c + t, so inverting it puts the+ # visible window's centre at -t/s, not at -t: the shift has to be+ # divided by the zoom. Without the /s the correction is too big+ # when zooming in and too small when zooming out -- measured over+ # 972 augmented samples at v2 strength (geom_trans 0.20,+ # geom_scale_max 2.0): 11.7 px mean / 33.6 px p95 of error in the+ # box centre, i.e. 0.61 deg mean / 1.73 deg p95 in the bearing the+ # network is conditioned on. It is a wrong INPUT, not a wrong+ # label -- the reprojection chain is exact to 0.000 px -- and it+ # only affects lateral placement, which PA-MPJPE cannot see.+ #+ # Default off so `--preset baseline` and every gen-5 run stay+ # bit-identical; `--crop-centre-fix` turns it on.+ z = scale if getattr(self, "crop_centre_fix", False) else 1.0+ cx -= dx * crop_w / (2.0 * z)+ cy -= dy * crop_w / (2.0 * z)b_scale = b_scale / scaleimage = self.transform(img_pil)@@ -768,6 +806,7 @@ def build_dataloaders(cfg):geom_trans=cfg.geom_trans, geom_flip_p=cfg.geom_flip_p,geom_scale_max=cfg.geom_scale_max,cliff_follows_aug=cfg.cliff_follows_aug,+ crop_centre_fix=cfg.crop_centre_fix,cliff_focal=cfg.cliff_focal,occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality)@@ -997,7 +1036,7 @@ class MHRForwardPass:per training step and makes the mesh LOD irrelevant to training cost."""- def __init__(self, mhr_path, device, kp_regressor=None):+ def __init__(self, mhr_path, device, kp_regressor=None, n_verts: int = 0):self.device = deviceself.mhr = torch.jit.load(mhr_path, map_location=device).eval()for p in self.mhr.parameters():@@ -1007,6 +1046,48 @@ class MHRForwardPass:W = torch.as_tensor(kp_regressor, dtype=torch.float32, device=device)assert W.shape == (70, 127), f"expected a (70, 127) regressor, got {tuple(W.shape)}"self.kp_regressor = W+ self.vert_idx = None+ if n_verts:+ self._build_vertex_subset(int(n_verts))++ def _build_vertex_subset(self, n: int):+ """A farthest-point subset of the rig mesh, plus a skinning op for it.++ The rig's `linear_blend_skinning` asserts it is handed all 18,439+ vertices, so the subset is installed by rewriting its three flattened+ influence tables -- (vertex, bone, weight) triples -- to reference only+ the chosen vertices under a compacted index. The op itself is+ untouched, which is why the result is the full mesh's answer restricted+ to those vertices rather than an approximation of it.++ Deterministic: the seed vertex is fixed and the input mesh is fixed, so+ every DDP rank builds the identical subset without communicating.+ """+ ct = self.mhr.character_torch+ rest = ct.mesh.rest_vertices.to(self.device)+ assert n <= rest.shape[0], f"n_verts={n} exceeds the {rest.shape[0]}-vertex mesh"+ # Farthest-point sampling in float64: the mesh spans ~180 cm and+ # neighbouring vertices are sub-millimetre apart, so float32 ties would+ # make the traversal order depend on accumulation noise.+ P = rest.double()+ idx = [0]+ d = (P - P[0]).norm(dim=-1)+ for _ in range(n - 1):+ i = int(d.argmax())+ idx.append(i)+ d = torch.minimum(d, (P - P[i]).norm(dim=-1))+ sub = torch.tensor(sorted(idx), device=self.device)++ lbs = copy.deepcopy(ct.linear_blend_skinning)+ vi = ct.linear_blend_skinning.vert_indices_flattened+ keep = torch.isin(vi, sub)+ remap = torch.full((int(vi.max()) + 1,), -1, dtype=torch.int64, device=self.device)+ remap[sub] = torch.arange(n, device=self.device)+ lbs.vert_indices_flattened = remap[vi[keep]]+ lbs.skin_indices_flattened = ct.linear_blend_skinning.skin_indices_flattened[keep]+ lbs.skin_weights_flattened = ct.linear_blend_skinning.skin_weights_flattened[keep]+ lbs.num_vertices = n+ self.vert_idx, self.vert_lbs = sub, lbsdef get_joints(self, model_params, shape_params):"""(B, 127, 8) skeleton state in raw MHR units (cm, Y-up)."""@@ -1017,6 +1098,24 @@ class MHRForwardPass:skel_state = self.mhr.character_torch.joint_parameters_to_skeleton_state(joint_params)return skel_state.to(model_params.device)+ def get_joints_and_vertices(self, model_params, shape_params):+ """(B, 127, 3) joints and (B, n_verts, 3) skin, both raw MHR cm.++ One forward kinematics pass feeds both. Unlike `get_joints` this passes+ the REAL identity coefficients rather than zeros -- which changes the+ skeleton by exactly nothing (verified bit-identical, `torch.equal`),+ because the 45 identity coefficients are mesh-only -- while the mesh+ needs them, since they are the only thing this term exists to supervise.+ """+ assert self.vert_idx is not None, "MHRForwardPass was built with n_verts=0"+ ct = self.mhr.character_torch+ cat = torch.cat([model_params, shape_params], dim=1).to(self.device)+ skel = ct.joint_parameters_to_skeleton_state(+ ct.model_parameters_to_joint_parameters(cat))+ rest = ct.blend_shape(shape_params.to(self.device))[:, self.vert_idx]+ verts = self.vert_lbs(skel, rest)+ return skel[..., :3].to(model_params.device), verts.to(model_params.device)+@staticmethoddef to_vision(joints):"""Raw MHR (cm, Y-up/Z-back) -> annotation frame (metres, Y-down/Z-fwd)."""@@ -1192,14 +1291,30 @@ class DistillationLoss(nn.Module):else:losses['loss_cam'] = self.mse(preds["cam_trans"], targets["cam_trans"]) * self.cfg.w_cam- pred_mhr_joints = self.mhr_module.get_joints(preds["mhr_params"], preds["shape_params"])[..., :3]+ want_verts = float(getattr(self.cfg, "w_verts", 0.0)) > 0.0+ if want_verts:+ pred_mhr_joints, pred_verts = self.mhr_module.get_joints_and_vertices(+ preds["mhr_params"], preds["shape_params"])+ else:+ pred_mhr_joints = self.mhr_module.get_joints(preds["mhr_params"], preds["shape_params"])[..., :3]with torch.no_grad():- tgt_mhr_joints = self.mhr_module.get_joints(targets["mhr_model_params"], targets["shape_params"])[..., :3]+ if want_verts:+ tgt_mhr_joints, tgt_verts = self.mhr_module.get_joints_and_vertices(+ targets["mhr_model_params"], targets["shape_params"])+ else:+ tgt_mhr_joints = self.mhr_module.get_joints(targets["mhr_model_params"], targets["shape_params"])[..., :3]c = targets["aug_cos"].view(-1, 1) if "aug_cos" in targets else torch.ones(B, 1, device=ones.device)s = targets["aug_sin"].view(-1, 1) if "aug_sin" in targets else torch.zeros(B, 1, device=ones.device)xm, ym = tgt_mhr_joints[..., 0], tgt_mhr_joints[..., 1]tgt_mhr_joints = torch.stack([c * xm - s * ym, s * xm + c * ym, tgt_mhr_joints[..., 2]], dim=-1)+ if want_verts:+ # The same in-plane rotation the joints get: the dataset rotates+ # the LABELS, not the target parameters, so the FK'd target has+ # to be brought into the augmented frame.+ xv, yv = tgt_verts[..., 0], tgt_verts[..., 1]+ tgt_verts = torch.stack(+ [c * xv - s * yv, s * xv + c * yv, tgt_verts[..., 2]], dim=-1)# KEPT (w=1e-3) even though loss_3d_native now covers the same parameters.# It is not redundant: the best linear reconstruction of the 127 skeleton@@ -1210,6 +1325,16 @@ class DistillationLoss(nn.Module):losses['loss_mhr_joints'] = self.m_smooth_l1(pred_mhr_joints, tgt_mhr_joints.detach(), m_noflip) * self.cfg.w_3d_joints+ if want_verts:+ # Per-vertex unsquared Euclidean distance in METRES, so the weight is+ # on the same scale as w_keypoints3d under kp3d_loss="euclid" and the+ # logged value reads as mean PVE / 1000. Masked to m_noflip for the+ # same reason loss_mhr_joints is: under a horizontal flip the target+ # PARAMETERS are not mirrored, only the 2D and 3D labels are.+ e_verts = (pred_verts - tgt_verts.detach()).norm(dim=-1).mean(1) / 100.0+ losses['loss_verts'] = (self._mmean(e_verts, m_noflip)+ * self.cfg.w_verts * self.fk_scale)+# The 70 native keypoints ARE the mesh now: an affine function of the# 127 skeleton joints, so they are differentiable w.r.t. mhr_params and# carry no extra forward pass (pred_mhr_joints is already computed).@@ -1332,7 +1457,7 @@ class DistillationLoss(nn.Module):# are the only unbounded ones: the parameter regressions are bounded by the# target magnitudes and the 2D terms are bounded by construction (SimCC spans# +/-kp2d_range and its CE by log(kp2d_bins)).-FK_LOSS_KEYS = ("loss_3d_native", "loss_mhr_joints", "loss_reproj")+FK_LOSS_KEYS = ("loss_3d_native", "loss_mhr_joints", "loss_reproj", "loss_verts")def safe_loss_subset(losses):diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.pyindex c2a3a4d..1fee455 100644--- a/tools/pth_to_onnx.py+++ b/tools/pth_to_onnx.py@@ -35,7 +35,7 @@ Usage# or point at a specific checkpoint / outputpython ./tools/pth_to_onnx.py \- --ckpt ./instanthmr_distill_train/runs/b2_s0/b2_s0/best_student_model_ema.pth \+ --ckpt ./instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth \--output ./models/instanthmr_mhr_only.onnx"""import argparse