Skip to content
Python

feat: --preset v3 samples crops from context crops over a wide framing range; 288 px phone latencies

mohamdev2cf4b455805f
diff --git a/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md b/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md
index 6bcc3f6..dd691d3 100644
--- a/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md
+++ b/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md
@@ -4,8 +4,12 @@
**Blocked by:** None (can start immediately; already queued as job 189727).
-**Status:** ready-for-human
+**Status:** done (2026-09-26)
-- [ ] `hmr_archive_<jobid>.out` ends with an `ls -lh` listing five tars: aic, coco, mpii, sa1b, harmony4d
-- [ ] The aic and mpii tars hold real frames (hard links/copies), not links: `tar tvf` shows regular files under `original_images/`
-- [ ] If the job timed out on sa1b, the partial sa1b tar was deleted before resubmitting (the script skips any existing tar, including a half-written one)
+- [x] `hmr_archive_<jobid>.out` ends with an `ls -lh` listing five tars: aic, coco, mpii, sa1b, harmony4d
+- [x] The aic and mpii tars hold real frames (hard links/copies), not links: `tar tvf` shows regular files under `original_images/`
+- [x] If the job timed out on sa1b, the partial sa1b tar was deleted before resubmitting (the script skips any existing tar, including a half-written one)
+
+## Answer
+
+Job 189727 wrote five split tars, but `40_archive.slurm`'s unanchored `--exclude=images` also dropped every nested `images/` folder -- i.e. all AIC and MPII original photos (0 in either tar; mpii.tar 228 MB). Fixed in the template (exclude only a top-level `images` symlink, anchored; tested on a fake tree) and rsynced. The photos were archived separately by job 226766: `sam3d_gt_aic_originals.tar` 13 GB, `sam3d_gt_mpii_originals.tar` 2.6 GB. `sam3d_gt_harmony4d.tar` verified to hold all 496,036 crops. STORE total 155 GB.
diff --git a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
index 49872c1..9ffc66c 100644
--- a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
+++ b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
@@ -4,8 +4,22 @@
**Blocked by:** 05
-**Status:** ready-for-agent
+**Status:** done (2026-09-25), uncommitted
-- [ ] NPU latency on Galaxy S22, S23, S24 and S25; CPU latency on S23, A73, A53, A14 and Pixel 8
-- [ ] 100% NPU placement and fp16 parity on the S23
-- [ ] Result recorded next to ticket 03's table, with the 224-vs-288 latency ratio per phone
+- [x] NPU latency on Galaxy S22, S23, S24 and S25; CPU latency on S23, A73, A53, A14 and Pixel 8
+- [~] 100% NPU placement on all four Snapdragon phones (yes); fp16 parity deferred to trained weights (ticket 11), as in ticket 03
+- [x] Result recorded next to ticket 03's table, with the 224-vs-288 latency ratio per phone
+
+## Answer
+
+Random-weight exports at 288 px (`tools/profiling/export_family.py --image-size 288`), same method as ticket 03. Raw rows: `benchmark/results/family_latency_aihub_288.json`.
+
+| model @ 288 | S22 NPU | S23 NPU | S24 NPU | S25 NPU | S23 CPU | A73 CPU | Pixel 8 CPU | A53 CPU | A14 CPU |
+|---|---|---|---|---|---|---|---|---|---|
+| B4 | 5.5 | 3.27 | 2.31 | 1.88 | 93.3 | 119.0 | 115.0 | 288.7 | 294.2 |
+| B0 | 3.7 | 2.05 | 1.31 | 1.00 | 29.3 | 38.2 | 32.1 | 93.5 | 79.0 |
+
+Cost of 288 over 224 (same backbone): B4 x1.32 on the S23 NPU, x1.46-1.55 on CPUs; B0 x1.28 NPU, x1.07-1.23 CPU. All exports 100% on the NPU.
+
+- **The 288 arm is deployable on the NPU tier:** B4 at 288 is 1.9-5.5 ms across Snapdragon 8 Gen 1-Elite, far inside 33 ms. Ticket 10 may train it.
+- **Not for the CPU tier with B4** (93-294 ms). B0 at 288 stays near budget on mid-range CPUs (Pixel 8 32.1, A73 38.2) because below B4 the decoder, not the image, dominates cost -- a 288 CPU-tier member is plausible if 288 proves worth it.
diff --git a/.scratch/v3-framing-robustness/issues/07-v3-preset.md b/.scratch/v3-framing-robustness/issues/07-v3-preset.md
index f1f6590..72da005 100644
--- a/.scratch/v3-framing-robustness/issues/07-v3-preset.md
+++ b/.scratch/v3-framing-robustness/issues/07-v3-preset.md
@@ -4,11 +4,24 @@
**Blocked by:** 04, 05
-**Status:** ready-for-agent
-
-- [ ] A small local split rebuilt with ticket 04's builder (context 2.0 / 448) trains under v3 at 224 and at 288
-- [ ] Verification on real samples: 2D labels land on the right pixels; CLIFF crop-centre error at the rounding floor (mean / p95 / max px, as in the crop-centre-fix check); no black pixel inside the stored context; box-scale draws cover 0.6–2.0x; output size equals the configured input size
-- [ ] baseline and v2 samples bit-identical against HEAD (alternating calls)
-- [ ] The v3 dataset refuses a split without context geometry, with a message naming the split
-- [ ] `ddp_smoke` passes for v3 at 224 and at 288
-- [ ] Files that must be rsynced to the cluster listed in the status doc
+**Status:** done (2026-09-25), uncommitted
+
+- [x] A small local split rebuilt with ticket 04's builder (context 2.0 / 448) trains under v3 at 224 and at 288
+- [x] Verification on real samples: 2D labels land on the right pixels; CLIFF crop-centre error at the rounding floor (mean / p95 / max px, as in the crop-centre-fix check); no black pixel inside the stored context; box-scale draws cover 0.6–2.0x; output size equals the configured input size
+- [x] baseline and v2 samples bit-identical against HEAD (alternating calls)
+- [x] The v3 dataset refuses a split without context geometry, with a message naming the split
+- [x] `ddp_smoke` passes for v3 at 224 and at 288
+- [x] Files that must be rsynced to the cluster listed in the status doc
+
+## Answer
+
+Implemented as specified, with two decisions taken with the user (2026-09-25):
+- **Clamp (Q22, option b):** keep the rotated window inside the stored square only on sides where the square ends inside the photo. Tail restored: P(>1.6x) 17.0% (unclamped 18.5%, all-sides clamp 10.5%), P(>1.8x) 6.6% (8.8% / 1.9%), loose windows keep their shift.
+- **geom_p (Q23):** 80/20 as v2; the 20% identity windows keep root-rotation supervision. Revisit only if v3 helps.
+
+Evidence: `tools/verify_v3_preset.py` PASS (see README section `--preset v3`). `ddp_smoke --context-crops` passes at 224 and 288. A 1-epoch `train_distill_jz.py --preset v3 --image-size 288` run on the local context split completed and recorded preset / image_size / v3 fields in run_config.json. Rsync list and cluster checks: STATUS.md, "Code ready, not launched".
+
+Findings:
+- 95% of 2.0x context squares (AIC, COCO) run past the frame; windows >1.5x are ~38% black on average (frame edge, representative of deployment). Harmony4D / SA-1B may differ; measure after the rebuild.
+- Naming: `--preset v3` is unrelated to the old `v3_s{0,1,2}` runs (v2 + rebalance). Noted in STATUS.
+- `apply_v3_preset` forces `cliff_focal`.
diff --git a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
index def5f0e..22b9f6d 100644
--- a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
+++ b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
@@ -2,10 +2,14 @@
**What to build:** Harmony4D rejoins the corpus as context crops. Its original frames exist nowhere, so its zips are streamed scene by scene: download, undistort, build context crops, delete. The route depends on the `prepost` download test: HTTP 206 means the streaming job runs on Jean Zay (keeping no 4K originals); otherwise the same script runs on the laptop and the crops are rsynced as one tar.
-**Blocked by:** 04, and the `prepost` curl result (job 190182)
+**Blocked by:** None (04 done; route decided)
-**Status:** ready-for-human
+**Status:** ready-for-human -- route 2 (laptop)
-- [ ] Route chosen from the `prepost` result and recorded
+- [x] Route chosen from the `prepost` result and recorded
- [ ] Undistortion blockers in the undistortion notes handled before the first full scene
- [ ] 496,036 train crops (± source-missing), verified and archived to `$STORE`
+
+## Route decided (2026-09-27): laptop
+
+`prepost` job 226661 returned `HTTP 302, 0 bytes`: Hugging Face's redirect to its file host `us.aws.cdn.hf.co` cannot be followed from Jean Zay compute nodes (the login node could, 2026-09-25; same failure as STATUS 7.1 on 2026-09-03). So stream on the laptop, scene by scene (65 GB free: download one zip, undistort, build context crops with `--context 2.0 --body-size 448 --output-dir`, delete), then rsync one tar of crops to Jean Zay's context data root.
diff --git a/benchmark/results/family_latency_aihub_288.json b/benchmark/results/family_latency_aihub_288.json
new file mode 100644
index 0000000..f9fe6ec
--- /dev/null
+++ b/benchmark/results/family_latency_aihub_288.json
@@ -0,0 +1,218 @@
+[
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S22 5G",
+ "job": "jg9zoorwp",
+ "ms": 3.7,
+ "peak_mb": 150.2,
+ "layers": {
+ "NPU": 827
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp1noo98g",
+ "ms": 2.05,
+ "peak_mb": 90.1,
+ "layers": {
+ "NPU": 827
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S24",
+ "job": "jgdd66krg",
+ "ms": 1.31,
+ "peak_mb": 169.7,
+ "layers": {
+ "NPU": 827
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S25",
+ "job": "j57eoomvp",
+ "ms": 1.0,
+ "peak_mb": 167.3,
+ "layers": {
+ "NPU": 827
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp4yee78p",
+ "ms": 29.25,
+ "peak_mb": 139.7,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jpxl00q3p",
+ "ms": 38.2,
+ "peak_mb": 125.0,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "j5m0997dg",
+ "ms": 93.51,
+ "peak_mb": 125.8,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "jgnz114kg",
+ "ms": 78.97,
+ "peak_mb": 182.6,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b0_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "jprlxxr0p",
+ "ms": 32.06,
+ "peak_mb": 142.9,
+ "layers": {
+ "CPU": 722
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S22 5G",
+ "job": "jp2roo1rg",
+ "ms": 5.5,
+ "peak_mb": 174.3,
+ "layers": {
+ "NPU": 890
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jpyo88l85",
+ "ms": 3.27,
+ "peak_mb": 159.2,
+ "layers": {
+ "NPU": 890
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S24",
+ "job": "jp0moow9g",
+ "ms": 2.31,
+ "peak_mb": 232.4,
+ "layers": {
+ "NPU": 890
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "npu",
+ "device": "Samsung Galaxy S25",
+ "job": "jp8ejjnkp",
+ "ms": 1.88,
+ "peak_mb": 141.0,
+ "layers": {
+ "NPU": 890
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jgk2661wg",
+ "ms": 93.32,
+ "peak_mb": 173.5,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "j5ql44nnp",
+ "ms": 119.02,
+ "peak_mb": 160.3,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "jglywwdj5",
+ "ms": 288.74,
+ "peak_mb": 146.3,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "j568oox6g",
+ "ms": 294.21,
+ "peak_mb": 147.4,
+ "layers": {
+ "CPU": 762
+ }
+ },
+ {
+ "model": "random_b4_288",
+ "image_size": 288,
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "jp3zood35",
+ "ms": 115.04,
+ "peak_mb": 198.5,
+ "layers": {
+ "CPU": 762
+ }
+ }
+]
\ No newline at end of file
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index c522325..9251cf8 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -609,6 +609,44 @@ and must be rsynced**, or the job dies at startup. Default off; with it off
every loss term is bit-identical, verified across `legacy`, `rebalanced`,
`w_verts` on/off and `cont_head` on/off.
+## `--preset v3` — framing robustness from context crops (added 2026-09-25)
+
+v2, plus: every crop is cut from a **context crop** (`build_split.py --context
+2.0 --body-size 448`, its own `--data_root`) through a window drawn in frame
+pixels -- side log-uniform in 0.6-2.0x the tight box (1.2 = today's crop, 0.6 =
+v2's deepest truncation), centre shifted up to +/-15% of the side, v2's
+rotation and flip. It is one resampling step from the stored 448 px crop, so
+zoom-in no longer upsamples a 224 crop. 20% of samples (`geom_p` 0.8, as v2)
+keep the identity window, i.e. today's crop, which also keeps root-rotation
+supervision on the un-augmented subset. Validation uses the identity window.
+
+**Why.** g8h loses 12-15 mm on EMDB under EMDB's own boxes, and 16 mm once our
+own boxes are enlarged to 1.6x -- the loosest framing a 1.2x-cropped corpus can
+show, and then only with black margins. v2 cannot help: it only zooms in.
+
+**The window is an ordinary `aug_M`** relative to `bbox_square`, so labels, the
+reprojection loss and the model are untouched; the CLIFF vector is computed
+from the exact window (centre, side / 1.2) and then flipped and rotated with
+the labels. `apply_v3_preset` forces `cliff_focal`.
+
+**The clamp.** The rotated window is kept inside the stored square only on
+sides where that square ends inside the photo. Where it already runs past the
+photo's edge, further out is outside the photo too, so the black read there is
+the real frame edge -- what a deployment crop shows when its box leaves the
+frame. Clamping every side instead thinned windows looser than 1.6x from 18.5%
+to 10.5% and centred them; this rule keeps 17.0%, with their shift intact.
+
+**Expect black.** In AIC / COCO the person usually nearly fills the photo: 95%
+of 2.0x squares run past the frame, and windows looser than 1.5x are on average
+38% black (frame edge). That is representative of deployment, but it caps how
+much loose-and-real-scene framing this corpus can teach.
+
+Verified by `tools/verify_v3_preset.py` on real AIC rows built both ways:
+baseline / v2 bit-identical; identity window = today's labels and CLIFF
+exactly; reprojection chain vs labels 2e-4 px; CLIFF vs the recovered window
+0.000 px; no read past an inside-photo stored edge; standard or mixed roots
+refused (also through `PairIndex`).
+
## `--image-size` — the network input side (added 2026-09-25)
Default 224, i.e. unchanged. Must be a multiple of 32, because the decoder's
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 4e38c99..49f55f2 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -171,6 +171,11 @@ class PairIndex:
def __len__(self):
return len(self._stem)
+ def first_per_subfolder(self):
+ """Index of the first pair in each sub-folder (one np.unique over the
+ byte array; no Path objects)."""
+ return [int(k) for k in np.unique(self._sub, return_index=True)[1]]
+
def __getitem__(self, i):
if isinstance(i, slice):
# Stay compact through --max_images too, or the one configuration
@@ -390,7 +395,9 @@ def build_jz_loaders(cfg, args, rank: int, world: int):
cliff_focal=cfg.cliff_focal,
occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality,
- image_size=cfg.image_size)
+ image_size=cfg.image_size,
+ context_crops=cfg.context_crops, v3_box_min=cfg.v3_box_min,
+ v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)
# Rank 0 builds the index (or reads the cache) while the others wait, then
# everyone loads the same cache. Building on all ranks at once would put
@@ -701,7 +708,8 @@ def train(args, cfg, rank, local_rank, world):
"lr": cfg.lr, "lr_arg": args.lr, "lr_scaling": args.lr_scaling,
"mix": args.mix, "steps_per_epoch": steps_per_epoch,
"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,
- "image_size": cfg.image_size,
+ "image_size": cfg.image_size, "context_crops": cfg.context_crops,
+ "v3_box": [cfg.v3_box_min, cfg.v3_box_max], "v3_shift": cfg.v3_shift,
"cont_head": cfg.cont_head, "root_rot_loss": cfg.root_rot_loss,
"exact_landmarks": cfg.exact_landmarks,
"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,
@@ -1154,10 +1162,13 @@ def parse_args():
help="Skip the timm download. Compute nodes have no internet, "
"so run --warm-cache once on a login node instead.")
p.add_argument("--log-every", type=int, default=100)
- p.add_argument("--preset", choices=["baseline", "v2"], default="baseline",
+ p.add_argument("--preset", choices=["baseline", "v2", "v3"], default="baseline",
help="'v2' enables the wave-1 bundle: truncation, occlusion "
"and JPEG augmentation with the SimCC-mask and CLIFF "
"prerequisites, plus absolute-pose supervision. "
+ "'v3' is v2 with every crop sampled from a context-crop "
+ "split over a wide range of framings (needs --data_root "
+ "of build_split.py --context splits). "
"'baseline' reproduces the pre-v2 trainer exactly.")
p.add_argument("--w_reproj", type=float, default=None)
p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true",
@@ -1257,8 +1268,10 @@ def build_cfg(args):
raise SystemExit(f"--image-size {args.image_size} is not a multiple of 32")
cfg.image_size = args.image_size
- if args.preset == "v2":
+ if args.preset in ("v2", "v3"):
apply_v2_preset(cfg)
+ if args.preset == "v3":
+ apply_v3_preset(cfg)
# Orthogonal axis: --preset is the input pipeline + absolute-pose terms,
# --losses is the loss budget. Applied second so it owns w_keypoints3d.
if args.losses == "rebalanced":
@@ -1300,6 +1313,26 @@ def apply_v2_preset(cfg) -> None:
cfg.cam_loss = "euclid"
+def apply_v3_preset(cfg) -> None:
+ """v2 plus framing robustness: every crop is sampled from a context crop
+ (build_split.py --context 2.0 --body-size 448) over a wide range of
+ framings -- loose and off-centre like EMDB's and a phone detector's boxes,
+ tight and truncated like a person close to the camera -- always surrounded
+ by real scene. Applied on top of apply_v2_preset: v2's occlusion, JPEG,
+ rotation, flip and absolute-pose terms are kept; its zoom/shift (which can
+ only zoom INTO a 1.2x crop) are replaced by the context window.
+
+ Measured motivation (2026-09-25): g8h loses 12-15 mm on EMDB under EMDB's
+ own boxes, and 16 mm when our boxes are enlarged to 1.6x -- the loosest
+ framing the 1.2x corpus can show, and only with black margins."""
+ cfg.context_crops = True
+ cfg.v3_box_min, cfg.v3_box_max = 0.6, 2.0
+ cfg.v3_shift = 0.15
+ # The window's CLIFF vector is built in the perspective-correct form; the
+ # pixel form is ambiguous on this corpus's real focals. Mandatory, not a flag.
+ cfg.cliff_focal = True
+
+
def main():
args = parse_args()
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 75395d1..f13ffce 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -170,6 +170,14 @@ class DistillConfig:
log_dir: str = str(PROJECT_ROOT / "runs/distill_mhr_only")
image_size: int = 224
+ # --preset v3: sample each crop from a context crop (build_split.py
+ # --context) over a wide range of framings. The box multiple is drawn
+ # log-uniformly in [v3_box_min, v3_box_max] x the tight box (1.2 = today's
+ # crop), the centre shifted by up to +/- v3_shift of the window side.
+ context_crops: bool = False
+ v3_box_min: float = 0.6
+ v3_box_max: float = 2.0
+ v3_shift: float = 0.15
max_images: int | None = None
per_dataset_caps: dict = field(default_factory=lambda: {"sam3d_gt_harmony4d": 300_000})
val_split: float = 0.1
@@ -556,6 +564,43 @@ class RandomPixelate:
img = F_t.resize(img, [orig_h, orig_w], interpolation=InterpolationMode.NEAREST)
return img
+def _apply_label_geometry(M_total, aug_flip, angle, joints_2d, joints_2d_vis,
+ joints_3d, cam_trans, cx, cy, orig_w, orig_h, W, H):
+ """Move every label with an augmentation: joints_2d through M_total (crop
+ px -> augmented px), the left/right swap for a flip, the in-plane rotation
+ of joints_3d / cam_trans, and the CLIFF box centre with both. Shared by the
+ standard and the v3 path so the two can never disagree."""
+ px = (joints_2d[:, 0] + 1.0) * 0.5 * W
+ py = (joints_2d[:, 1] + 1.0) * 0.5 * H
+ px2 = M_total[0, 0] * px + M_total[0, 1] * py + M_total[0, 2]
+ py2 = M_total[1, 0] * px + M_total[1, 1] * py + M_total[1, 2]
+ joints_2d[:, 0] = (px2 / (0.5 * W) - 1.0).astype(np.float32)
+ joints_2d[:, 1] = (py2 / (0.5 * H) - 1.0).astype(np.float32)
+
+ if aug_flip > 0.5:
+ joints_3d[:, 0] = -joints_3d[:, 0]
+ joints_3d = joints_3d[FLIP_PERM]
+ # joints_2d was already mirrored geometrically by M_total, but a
+ # mirrored image LOOKS like the opposite-handed person, so the
+ # left/right labels must be swapped too - exactly as for joints_3d.
+ # Without this the 2D head gets contradictory left/right supervision
+ # and collapses symmetric joints toward the body midline.
+ joints_2d = joints_2d[FLIP_PERM]
+ joints_2d_vis = joints_2d_vis[FLIP_PERM]
+ cam_trans[0] = -cam_trans[0]
+ cx = orig_w - cx
+ rad = math.radians(angle)
+ cos_t, sin_t = math.cos(rad), math.sin(rad)
+ R_v = np.array([[cos_t, sin_t], [-sin_t, cos_t]], dtype=np.float32)
+ joints_3d[:, :2] = joints_3d[:, :2] @ R_v.T
+ cam_trans[:2] = R_v @ cam_trans[:2]
+
+ dxy = np.array([cx - orig_w / 2.0, cy - orig_h / 2.0], dtype=np.float64)
+ dxy = R_v.astype(np.float64) @ dxy
+ cx, cy = orig_w / 2.0 + dxy[0], orig_h / 2.0 + dxy[1]
+ return joints_2d, joints_2d_vis, joints_3d, cam_trans, cx, cy, cos_t, sin_t
+
+
class SAM3DStudentDataset(Dataset):
def __init__(self, data_root: str,
image_size: int = 224, max_images: int | None = None,
@@ -569,9 +614,13 @@ class SAM3DStudentDataset(Dataset):
crop_centre_fix: bool = False,
cliff_focal: bool = False,
occl_p: float = 0.2, occl_scale: tuple = (0.02, 0.15),
- jpeg_p: float = 0.0, jpeg_quality: tuple = (30, 90)):
+ jpeg_p: float = 0.0, jpeg_quality: tuple = (30, 90),
+ context_crops: bool = False, v3_box_min: float = 0.6,
+ v3_box_max: float = 2.0, v3_shift: float = 0.15):
super().__init__()
self.image_size = image_size
+ self.context_crops = context_crops
+ self.v3_box_min, self.v3_box_max, self.v3_shift = v3_box_min, v3_box_max, v3_shift
self.data_root = Path(data_root)
self.augment = augment
self.geom_p = geom_p
@@ -623,6 +672,7 @@ class SAM3DStudentDataset(Dataset):
self.pairs = list(pairs) if isinstance(pairs, (list, tuple)) else pairs
print(f" Prebuilt index: {len(self.pairs)} (image, npz) pairs "
f"(augment={augment}).")
+ self._check_context_split()
return
self.pairs = []
@@ -666,6 +716,32 @@ class SAM3DStudentDataset(Dataset):
print(f" Found {len(dataset_dirs)} sub-folder(s) under {self.data_root}; "
f"{len(self.pairs)} valid (image, npz) pairs (augment={augment}).")
+ self._check_context_split()
+
+ def _check_context_split(self):
+ """v3 samples from context crops. A standard 224 split would still
+ 'work' -- with the 1.2x crop read as if it were the context, i.e. every
+ box silently 1.67x too large -- so refuse it up front."""
+ if not self.context_crops or len(self.pairs) == 0:
+ return
+ # One probe per sub-folder, so a --data_root mixing context and
+ # standard splits fails here and not hours in, inside a worker.
+ firsts = getattr(self.pairs, "first_per_subfolder", None)
+ if firsts is not None:
+ probes = firsts()
+ else:
+ seen = {}
+ for k, (_, npz) in enumerate(self.pairs):
+ seen.setdefault(Path(npz).parent, k)
+ probes = list(seen.values())
+ for k in probes:
+ npz = Path(self.pairs[k][1])
+ with np.load(npz) as z:
+ if "bbox_context" not in z.files:
+ raise ValueError(
+ f"{npz.parent.parent.name}: {npz.name} has no bbox_context. "
+ f"--preset v3 needs a context-crop split, built with "
+ f"build_split.py --context 2.0 --body-size 448 --output-dir ...")
@staticmethod
def _find_dataset_dirs(root: Path):
@@ -682,6 +758,88 @@ class SAM3DStudentDataset(Dataset):
dirs.append((ann, img))
return dirs
+ def _context_window(self, img_pil, ann, W, H, b_size, cx, cy):
+ """v3: cut the network input out of a context crop in ONE resampling.
+
+ The window is drawn in full-frame px: side = m x the tight box, m
+ log-uniform in [v3_box_min, v3_box_max]; centre = box centre + a shift of
+ up to +/- v3_shift x side; plus v2's flip and rotation. It is clamped so
+ the ROTATED window stays inside the stored context square, shrinking
+ the side first if it cannot fit -- so no synthetic black margin ever
+ appears, only the real frame edge the builder padded.
+
+ The window is expressed as an ordinary aug_M relative to the standard
+ 1.2x crop (bbox_square at W px), so labels, the reprojection loss and
+ everything downstream are unchanged. The image is warped from the
+ context crop with aug_M composed with the context -> standard-crop map.
+ Un-augmented samples use the identity window, i.e. today's crop.
+
+ Returns (image, M_total, aug_active, flip, angle, window centre (frame
+ px, before flip/rotation), window side / 1.2 in frame px).
+ """
+ sq = ann["bbox_square"].astype(np.float64)
+ ctx = ann["bbox_context"].astype(np.float64)
+ kx, ky = W / max(sq[2] - sq[0], 1.0), H / max(sq[3] - sq[1], 1.0)
+ M_total = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]], dtype=np.float64)
+ aug_active, aug_flip, angle, centre, bpx = 0.0, 0.0, 0.0, None, None
+
+ if self.augment and random.random() < self.geom_p:
+ aug_active = 1.0
+ aug_flip = 1.0 if random.random() < self.geom_flip_p else 0.0
+ angle = random.uniform(-self.geom_rot_deg, self.geom_rot_deg)
+ m = math.exp(random.uniform(math.log(self.v3_box_min), math.log(self.v3_box_max)))
+ side = m * b_size
+ wx = cx + random.uniform(-self.v3_shift, self.v3_shift) * side
+ wy = cy + random.uniform(-self.v3_shift, self.v3_shift) * side
+ # Keep the rotated window inside the stored square -- but only on
+ # sides where that square ends INSIDE the photo. Where it already
+ # runs past the photo's edge, anything further out is outside the
+ # photo too, so the black it reads IS the real frame edge (what a
+ # deployment crop shows when its box leaves the frame); clamping
+ # there would only thin out the loose, off-centre framings v3 is for.
+ rad = math.radians(angle)
+ reach = abs(math.cos(rad)) + abs(math.sin(rad)) # rotated square's extent / side
+ oh, ow = (float(v) for v in ann["orig_shape"][:2])
+ lo_x = ctx[0] if ctx[0] >= 0 else -math.inf
+ hi_x = ctx[2] if ctx[2] <= ow else math.inf
+ lo_y = ctx[1] if ctx[1] >= 0 else -math.inf
+ hi_y = ctx[3] if ctx[3] <= oh else math.inf
+ side = min(side, min(hi_x - lo_x, hi_y - lo_y) / reach)
+ e = side * reach / 2.0
+ wx = min(max(wx, lo_x + e), hi_x - e)
+ wy = min(max(wy, lo_y + e), hi_y - e)
+ centre, bpx = (wx, wy), side / 1.2
+
+ # standard-crop px: flip, then zoom and rotate about the window
+ # centre and move it to the output centre -- v2's M_total form. The
+ # zoom is per axis: bbox_square's int() edges can make it 198x200
+ # px, and only per-axis zoom keeps the window SQUARE in the frame,
+ # which the clamp above and the 3D label rotation both assume.
+ c = np.array([(wx - sq[0]) * kx, (wy - sq[1]) * ky])
+ M_flip = (np.array([[-1.0, 0.0, float(W)], [0.0, 1.0, 0.0]]) if aug_flip
+ else np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]))
+ fc = M_flip[:, :2] @ c + M_flip[:, 2]
+ rot = cv2.getRotationMatrix2D((0.0, 0.0), angle, 1.0)[:, :2]
+ lin = rot @ np.diag([(sq[2] - sq[0]) / side, (sq[3] - sq[1]) / side])
+ M_rst = np.hstack([lin, (np.array([W / 2.0, H / 2.0]) - lin @ fc)[:, None]])
+ A2, t2 = M_rst[:, :2], M_rst[:, 2]
+ A1, t1 = M_flip[:, :2], M_flip[:, 2]
+ M_total = np.hstack([A2 @ A1, (A2 @ t1 + t2)[:, None]])
+
+ # context-crop pixel index -> standard-crop pixel index (pixel centres,
+ # as cv2.resize maps them on both sides; verified against today's crops
+ # in tools/verify_context_crops.py)
+ ctx_np = np.asarray(img_pil)
+ sc_y, sc_x = ctx_np.shape[:2]
+ ax, ay = (ctx[2] - ctx[0]) / sc_x, (ctx[3] - ctx[1]) / sc_y
+ C = np.array([[ax * kx, 0.0, (0.5 * ax + ctx[0] - sq[0]) * kx - 0.5],
+ [0.0, ay * ky, (0.5 * ay + ctx[1] - sq[1]) * ky - 0.5]])
+ A, t = M_total[:, :2], M_total[:, 2]
+ M_img = np.hstack([A @ C[:, :2], (A @ C[:, 2] + t)[:, None]])
+ img_np = cv2.warpAffine(ctx_np, M_img, (W, H), flags=cv2.INTER_LINEAR,
+ borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0))
+ return Image.fromarray(img_np), M_total, aug_active, aug_flip, angle, centre, bpx
+
def __len__(self):
return len(self.pairs)
@@ -732,7 +890,21 @@ class SAM3DStudentDataset(Dataset):
aug_flip, cos_t, sin_t = 0.0, 1.0, 0.0
M_total = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]], dtype=np.float64)
aug_active = 0.0
- if self.augment and random.random() < self.geom_p:
+ if self.context_crops:
+ (img_pil, M_total, aug_active, aug_flip, angle,
+ v3_centre, v3_bpx) = self._context_window(img_pil, ann, W, H, b_size, cx, cy)
+ if aug_active:
+ # CLIFF describes the window actually cut: its centre, and the
+ # tight box that would give it at the standard 1.2x expansion.
+ # Both are exact (no crop-space approximation), then flipped and
+ # rotated with the labels below.
+ cx, cy = v3_centre
+ b_scale = v3_bpx / max(orig_w, orig_h)
+ (joints_2d, joints_2d_vis, joints_3d, cam_trans, cx, cy,
+ cos_t, sin_t) = _apply_label_geometry(
+ M_total, aug_flip, angle, joints_2d, joints_2d_vis, joints_3d,
+ cam_trans, cx, cy, orig_w, orig_h, W, H)
+ elif self.augment and random.random() < self.geom_p:
aug_active = 1.0
img_np = np.asarray(img_pil.resize((W, H)))
@@ -767,34 +939,10 @@ class SAM3DStudentDataset(Dataset):
borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0))
img_pil = Image.fromarray(img_np)
- px = (joints_2d[:, 0] + 1.0) * 0.5 * W
- py = (joints_2d[:, 1] + 1.0) * 0.5 * H
- px2 = M_total[0, 0] * px + M_total[0, 1] * py + M_total[0, 2]
- py2 = M_total[1, 0] * px + M_total[1, 1] * py + M_total[1, 2]
- joints_2d[:, 0] = (px2 / (0.5 * W) - 1.0).astype(np.float32)
- joints_2d[:, 1] = (py2 / (0.5 * H) - 1.0).astype(np.float32)
-
- if aug_flip > 0.5:
- joints_3d[:, 0] = -joints_3d[:, 0]
- joints_3d = joints_3d[FLIP_PERM]
- # joints_2d was already mirrored geometrically by M_total, but a
- # mirrored image LOOKS like the opposite-handed person, so the
- # left/right labels must be swapped too - exactly as for joints_3d.
- # Without this the 2D head gets contradictory left/right supervision
- # and collapses symmetric joints toward the body midline.
- joints_2d = joints_2d[FLIP_PERM]
- joints_2d_vis = joints_2d_vis[FLIP_PERM]
- cam_trans[0] = -cam_trans[0]
- cx = orig_w - cx
- rad = math.radians(angle)
- cos_t, sin_t = math.cos(rad), math.sin(rad)
- R_v = np.array([[cos_t, sin_t], [-sin_t, cos_t]], dtype=np.float32)
- joints_3d[:, :2] = joints_3d[:, :2] @ R_v.T
- cam_trans[:2] = R_v @ cam_trans[:2]
-
- dxy = np.array([cx - orig_w / 2.0, cy - orig_h / 2.0], dtype=np.float64)
- dxy = R_v.astype(np.float64) @ dxy
- cx, cy = orig_w / 2.0 + dxy[0], orig_h / 2.0 + dxy[1]
+ (joints_2d, joints_2d_vis, joints_3d, cam_trans, cx, cy,
+ cos_t, sin_t) = _apply_label_geometry(
+ M_total, aug_flip, angle, joints_2d, joints_2d_vis, joints_3d,
+ cam_trans, cx, cy, orig_w, orig_h, W, H)
# CLIFF conditioning describes the box this crop came from. The flip
# and rotation above already move it; dx/dy/scale did not, which was
@@ -885,7 +1033,9 @@ def build_dataloaders(cfg):
cliff_focal=cfg.cliff_focal,
occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality,
- image_size=cfg.image_size)
+ image_size=cfg.image_size,
+ context_crops=cfg.context_crops, v3_box_min=cfg.v3_box_min,
+ v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)
full_dataset = SAM3DStudentDataset(
cfg.data_root,
augment=cfg.augment,
diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.py
index 5e4c52d..acd8b2f 100644
--- a/tools/ddp_smoke.py
+++ b/tools/ddp_smoke.py
@@ -52,6 +52,9 @@ def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--backbone", default=None, help="timm name; default is the config's")
+ p.add_argument("--context-crops", dest="context_crops", action="store_true",
+ help="--preset v3's dataset; --data_root must be a context-crop "
+ "split (build_split.py --context 2.0 --body-size 448)")
p.add_argument("--image-size", dest="image_size", type=int, default=None,
help="network input side; the positional grid and every crop follow it")
p.add_argument("--data_root", default="data")
@@ -109,7 +112,8 @@ def main() -> int:
ds = T.SAM3DStudentDataset(cfg.data_root, augment=True, max_images=args.batch_size,
per_dataset_caps={}, cliff_focal=cfg.cliff_focal,
crop_centre_fix=cfg.crop_centre_fix,
- image_size=cfg.image_size)
+ image_size=cfg.image_size,
+ context_crops=args.context_crops)
loader = torch.utils.data.DataLoader(ds, batch_size=args.batch_size, num_workers=0)
batch = {k: (v.to(T.device) if torch.is_tensor(v) else v)
for k, v in next(iter(loader)).items()}
diff --git a/tools/profiling/export_family.py b/tools/profiling/export_family.py
index 04f0bf7..ebd9edb 100644
--- a/tools/profiling/export_family.py
+++ b/tools/profiling/export_family.py
@@ -1,7 +1,15 @@
-"""Usage: python tools/profiling/export_family.py <out_dir> then python tools/profiling/profile_family.py <out_dir>
-Ticket 03: ONNX exports of the g8h architecture with the backbone swapped.
-Random weights for B0/B1/B4, plus the trained g8h, all through one path."""
-import io, sys, copy, json
+"""ONNX exports of the g8h architecture with the backbone and input size swapped.
+
+ python tools/profiling/export_family.py --out /tmp/fam # ticket 03
+ python tools/profiling/export_family.py --out /tmp/fam288 --image-size 288 \
+ --backbones b4 --no-trained # ticket 06
+
+Random weights (latency does not depend on them; ticket 03 measured random B4
+within 2-6% of the trained g8h on every phone), plus optionally the trained
+g8h. Network only, no FK / mesh: that is what differs between members.
+Then profile with tools/profiling/profile_family.py.
+"""
+import argparse, copy, io, json, sys
from pathlib import Path
import torch, onnx, onnxsim
REPO = Path(__file__).resolve().parents[2]
@@ -9,12 +17,7 @@ sys.path.insert(0, str(REPO / "instanthmr_distill_train"))
import train_distill_mhr_only as T
CKPT = REPO / "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_ema.pth"
-OUT = Path(sys.argv[1])
-blob = torch.load(CKPT, map_location="cpu", weights_only=False)
-state = {k.replace("_orig_mod.", ""): v for k, v in blob["model_state_dict"].items()}
-cfg, _ = T.config_from_checkpoint(state, CKPT)
-cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2
-cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])
+
class Deploy(torch.nn.Module):
def __init__(self, m): super().__init__(); self.m = m
@@ -22,26 +25,53 @@ class Deploy(torch.nn.Module):
o = self.m(image, cliff_cond)
return o["mhr_params"], o["shape_params"], o["cam_trans"], o["joints_2d"]
-def export(model, name):
+
+def export(model, path: Path, size: int):
model.eval()
buf = io.BytesIO()
- torch.onnx.export(Deploy(model), (torch.zeros(1, 3, 224, 224), torch.zeros(1, 3)), buf,
+ torch.onnx.export(Deploy(model), (torch.zeros(1, 3, size, size), torch.zeros(1, 3)), buf,
input_names=["image", "cliff_cond"],
output_names=["mhr_params", "shape_params", "cam_trans", "joints_2d"],
opset_version=17)
m, ok = onnxsim.simplify(onnx.load_from_string(buf.getvalue()))
assert ok
- p = OUT / f"{name}.onnx"; onnx.save(m, p)
+ onnx.save(m, path)
n = sum(x.numel() for x in model.parameters())
nb = sum(x.numel() for x in model.backbone.parameters())
- print(json.dumps(dict(name=name, params_M=round(n/1e6, 2), backbone_M=round(nb/1e6, 2),
- mb=round(p.stat().st_size/2**20, 1))), flush=True)
-
-trained = T.InstantHMRStudent(copy.deepcopy(cfg), pretrained=False)
-missing, unexpected = trained.load_state_dict(state, strict=False)
-assert not missing and not unexpected, (missing[:5], unexpected[:5])
-export(trained, "g8h_trained_b4")
-for bb in ("hgnetv2_b4", "hgnetv2_b1", "hgnetv2_b0"):
- c = copy.deepcopy(cfg); c.backbone = bb
- torch.manual_seed(0)
- export(T.InstantHMRStudent(c, pretrained=False), f"random_{bb.split('_')[1]}")
+ print(json.dumps(dict(name=path.stem, image_size=size, params_M=round(n / 1e6, 2),
+ backbone_M=round(nb / 1e6, 2))), flush=True)
+
+
+def main():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--out", type=Path, required=True)
+ p.add_argument("--image-size", type=int, default=224)
+ p.add_argument("--backbones", default="b4,b1,b0", help="hgnetv2 sizes, comma-separated")
+ p.add_argument("--no-trained", action="store_true", help="skip the trained g8h export")
+ args = p.parse_args()
+ args.out.mkdir(parents=True, exist_ok=True)
+
+ blob = torch.load(CKPT, map_location="cpu", weights_only=False)
+ state = {k.replace("_orig_mod.", ""): v for k, v in blob["model_state_dict"].items()}
+ cfg, _ = T.config_from_checkpoint(state, CKPT)
+ cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2
+ cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])
+
+ if not args.no_trained:
+ if args.image_size != cfg.image_size:
+ sys.exit("the trained g8h is a 224 model; pass --no-trained with --image-size")
+ trained = T.InstantHMRStudent(copy.deepcopy(cfg), pretrained=False)
+ missing, unexpected = trained.load_state_dict(state, strict=False)
+ assert not missing and not unexpected, (missing[:5], unexpected[:5])
+ export(trained, args.out / "g8h_trained_b4.onnx", args.image_size)
+ for b in args.backbones.split(","):
+ c = copy.deepcopy(cfg)
+ c.backbone, c.image_size = f"hgnetv2_{b}", args.image_size
+ torch.manual_seed(0)
+ export(T.InstantHMRStudent(c, pretrained=False),
+ args.out / f"random_{b}_{args.image_size}.onnx", args.image_size)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/profiling/profile_family.py b/tools/profiling/profile_family.py
index 85bca8c..c6b166a 100644
--- a/tools/profiling/profile_family.py
+++ b/tools/profiling/profile_family.py
@@ -1,35 +1,64 @@
-import json, sys
+"""Compile every .onnx in a directory for the S23 (ORT runtime) and profile it on
+AI Hub: NPU on the Snapdragon 8-series phones, CPU-only on the reference phones.
+
+ python tools/profiling/profile_family.py --dir /tmp/fam288 --image-size 288
+
+Writes <dir>/profile.json (one row per model x device x unit) and prints rows.
+"""
+import argparse, json
from pathlib import Path
import qai_hub as hub
-D = Path(sys.argv[1])
+
CPU = ["Samsung Galaxy S23", "Samsung Galaxy A73 5G", "Samsung Galaxy A53 5G",
"Samsung Galaxy A14 5G", "Google Pixel 8"]
-specs = dict(image=(1, 3, 224, 224), cliff_cond=(1, 3))
-names = ["g8h_trained_b4", "random_b4", "random_b1", "random_b0"]
-comp = {n: hub.submit_compile_job(model=str(D / f"{n}.onnx"), device=hub.Device("Samsung Galaxy S23"),
- input_specs=specs, options="--target_runtime onnx", name=f"t03-{n}")
- for n in names}
-jobs = []
-for n, cj in comp.items():
- m = cj.get_target_model()
- if m is None:
- print(json.dumps(dict(model=n, compile_failed=cj.get_status().message)), flush=True); continue
- jobs.append((n, "npu", "Samsung Galaxy S23",
- hub.submit_profile_job(model=m, device=hub.Device("Samsung Galaxy S23"), name=f"t03-{n}-npu")))
- for d in CPU:
- jobs.append((n, "cpu", d, hub.submit_profile_job(model=m, device=hub.Device(d),
- name=f"t03-{n}-cpu", options="--compute_unit cpu")))
-out = []
-for n, u, d, j in jobs:
- row = dict(model=n, unit=u, device=d, job=j.job_id)
- if j.wait().success:
- p = j.download_profile(); s = p["execution_summary"]
- units = {}
- for l in p.get("execution_detail", []):
- units[l.get("compute_unit")] = units.get(l.get("compute_unit"), 0) + 1
- row.update(ms=round(s["estimated_inference_time"] / 1000, 2),
- peak_mb=round(s.get("estimated_inference_peak_memory", 0) / 2**20, 1), layers=units)
- else:
- row["error"] = j.get_status().message
- print(json.dumps(row), flush=True); out.append(row)
-json.dump(out, open(D / "profile.json", "w"), indent=1)
+NPU = ["Samsung Galaxy S22 5G", "Samsung Galaxy S23", "Samsung Galaxy S24", "Samsung Galaxy S25"]
+
+
+def main():
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--dir", type=Path, required=True)
+ p.add_argument("--image-size", type=int, default=224)
+ p.add_argument("--npu", default="S23", help="'S23' or 'all' Snapdragon 8-series phones")
+ args = p.parse_args()
+ s = args.image_size
+ specs = dict(image=(1, 3, s, s), cliff_cond=(1, 3))
+ npu = NPU if args.npu == "all" else ["Samsung Galaxy S23"]
+
+ comp = {f.stem: hub.submit_compile_job(model=str(f), device=hub.Device("Samsung Galaxy S23"),
+ input_specs=specs, options="--target_runtime onnx",
+ name=f"fam-{f.stem}")
+ for f in sorted(args.dir.glob("*.onnx"))}
+ jobs = []
+ for n, cj in comp.items():
+ m = cj.get_target_model()
+ if m is None:
+ print(json.dumps(dict(model=n, compile_failed=cj.get_status().message)), flush=True)
+ continue
+ for d in npu:
+ jobs.append((n, "npu", d, hub.submit_profile_job(model=m, device=hub.Device(d),
+ name=f"fam-{n}-npu")))
+ for d in CPU:
+ jobs.append((n, "cpu", d, hub.submit_profile_job(model=m, device=hub.Device(d),
+ name=f"fam-{n}-cpu", options="--compute_unit cpu")))
+ out = []
+ for n, u, d, j in jobs:
+ row = dict(model=n, image_size=s, unit=u, device=d, job=j.job_id)
+ if j.wait().success:
+ prof = j.download_profile()
+ summ = prof["execution_summary"]
+ units = {}
+ for layer in prof.get("execution_detail", []):
+ units[layer.get("compute_unit")] = units.get(layer.get("compute_unit"), 0) + 1
+ row.update(ms=round(summ["estimated_inference_time"] / 1000, 2),
+ peak_mb=round(summ.get("estimated_inference_peak_memory", 0) / 2**20, 1),
+ layers=units)
+ else:
+ row["error"] = j.get_status().message
+ print(json.dumps(row), flush=True)
+ out.append(row)
+ json.dump(out, open(args.dir / "profile.json", "w"), indent=1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/verify_v3_preset.py b/tools/verify_v3_preset.py
new file mode 100644
index 0000000..81b71b3
--- /dev/null
+++ b/tools/verify_v3_preset.py
@@ -0,0 +1,290 @@
+#!/usr/bin/env python3
+"""Verify the v3 preset on real context crops.
+
+Needs the same rows built twice with datasets_pipeline/build_split.py:
+standard (default flags) and context (--context 2.0 --body-size 448).
+
+ python tools/verify_v3_preset.py --std <root with sam3d_gt_aic> --ctx <root> \
+ --ref <train_distill_mhr_only.py before the change>
+
+Checks, all from the dataset's outputs plus the npz geometry:
+
+1. baseline and v2 samples are bit-identical to --ref (alternating calls).
+2. Un-augmented v3 (the identity window) reproduces today's crop: labels and
+ CLIFF equal the standard split's, image within resampling noise.
+3. Reprojection chain: joints_3d + cam_trans projected through bbox_square and
+ aug_M land on the joints_2d labels, exactly as the loss computes them.
+4. CLIFF describes the window actually cut: the preimage of the crop centre
+ under aug_M, flipped and rotated like the labels, is the CLIFF centre, and
+ the CLIFF size is the window side / 1.2.
+5. No pixel is read from beyond a stored-square edge that lies INSIDE the
+ photo (that would be a synthetic black margin). Past an edge that is
+ already outside the photo, black is the real frame edge and allowed.
+ Also reports the loose tail the clamp leaves: P(>1.6x), P(>1.8x) and the
+ mean shift of loose windows.
+6. Box-scale draws cover the configured range (log-uniform 0.6-2.0x).
+7. Output size follows image_size (224 and 288).
+8. A standard split is refused by the v3 dataset, naming the split.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import math
+import random
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+from PIL import Image
+
+REPO = Path(__file__).resolve().parent.parent
+TRAIN = REPO / "instanthmr_distill_train"
+sys.path.insert(0, str(TRAIN))
+import train_distill_mhr_only as T # noqa: E402
+import train_distill_jz as JZ # noqa: E402
+
+MEAN = np.array([0.485, 0.456, 0.406])
+STD = np.array([0.229, 0.224, 0.225])
+fails: list[str] = []
+
+
+def check(ok: bool, msg: str) -> None:
+ print(f" {'OK ' if ok else 'BAD'} {msg}")
+ if not ok:
+ fails.append(msg)
+
+
+def load_module(path: Path, name: str):
+ spec = importlib.util.spec_from_file_location(name, path)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+def cfg_for(preset: str, image_size: int = 224):
+ cfg = T.DistillConfig()
+ cfg.cliff_focal = True
+ cfg.crop_centre_fix = True
+ cfg.image_size = image_size
+ if preset in ("v2", "v3"):
+ JZ.apply_v2_preset(cfg)
+ if preset == "v3":
+ JZ.apply_v3_preset(cfg)
+ return cfg
+
+
+def dataset(mod, root, cfg, augment=True, v3=False):
+ kw = dict(geom_p=cfg.geom_p, geom_rot_deg=cfg.geom_rot_deg,
+ geom_scale_range=cfg.geom_scale_range, geom_trans=cfg.geom_trans,
+ geom_flip_p=cfg.geom_flip_p, geom_scale_max=cfg.geom_scale_max,
+ cliff_follows_aug=cfg.cliff_follows_aug, crop_centre_fix=cfg.crop_centre_fix,
+ cliff_focal=cfg.cliff_focal, occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
+ jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality, image_size=cfg.image_size)
+ if v3:
+ kw.update(context_crops=True, v3_box_min=cfg.v3_box_min,
+ v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)
+ return mod.SAM3DStudentDataset(str(root), augment=augment, **kw)
+
+
+def seeded(ds, i, seed):
+ random.seed(seed)
+ np.random.seed(seed)
+ torch.manual_seed(seed)
+ return ds[i]
+
+
+def same(a, b) -> bool:
+ if isinstance(a, torch.Tensor):
+ return torch.equal(a, b)
+ if isinstance(a, np.ndarray):
+ return np.array_equal(a, b, equal_nan=a.dtype.kind == "f")
+ return a == b
+
+
+def to_uint8(img: torch.Tensor) -> np.ndarray:
+ x = img.numpy().transpose(1, 2, 0) * STD + MEAN
+ return np.clip(x * 255.0, 0, 255)
+
+
+def reproject(s: dict, size: int) -> np.ndarray:
+ """The loss's own chain: undo the in-plane rotation and the flip on the 3D
+ labels (the dataset applied flip then rotation), project in the original
+ camera, map into the 1.2x crop, then let aug_M re-apply both."""
+ c, sn = float(s["aug_cos"]), float(s["aug_sin"])
+ R = np.array([[c, sn], [-sn, c]])
+ j3 = s["joints_3d"].numpy().astype(np.float64).copy()
+ cam = s["cam_trans"].numpy().astype(np.float64).copy()
+ j3[:, :2] = j3[:, :2] @ R # inverse of @ R.T
+ cam[:2] = R.T @ cam[:2] # inverse of R @
+ if float(s["aug_flip"]) > 0.5:
+ j3[:, 0], cam[0] = -j3[:, 0], -cam[0]
+ j = j3 + cam
+ fx, fy = s["cam_focal"].numpy()
+ h, w = s["orig_shape"].numpy()
+ u = fx * j[:, 0] / j[:, 2] + w / 2.0
+ v = fy * j[:, 1] / j[:, 2] + h / 2.0
+ sq = np.asarray(s["bbox_square"], dtype=np.float64)
+ k = size / max(sq[2] - sq[0], 1.0)
+ uc, vc = (u - sq[0]) * k, (v - sq[1]) * k
+ M = s["aug_M"].numpy().astype(np.float64).reshape(2, 3)
+ ua = M[0, 0] * uc + M[0, 1] * vc + M[0, 2]
+ va = M[1, 0] * uc + M[1, 1] * vc + M[1, 2]
+ return np.stack([ua / size * 2 - 1, va / size * 2 - 1], axis=1)
+
+
+def window_from_sample(s: dict, ann: dict, size: int, sc: int): # noqa: C901
+ """Recover the window from aug_M + npz geometry, independently of the
+ dataset's own bookkeeping. Returns (centre in frame px, side in frame px,
+ context-crop px of the 4 output corners)."""
+ M = s["aug_M"].numpy().astype(np.float64).reshape(2, 3)
+ A, t = M[:, :2], M[:, 2]
+ Ainv = np.linalg.inv(A)
+ sq = ann["bbox_square"].astype(np.float64)
+ ks = (sq[2:] - sq[:2]) / size # std px -> frame px
+ to_frame = lambda o: sq[:2] + (Ainv @ (np.asarray(o, float) - t)) * ks # noqa: E731
+ centre = to_frame([size / 2.0, size / 2.0])
+ side = size / abs(np.linalg.det(A)) ** 0.5 * (ks[0] * ks[1]) ** 0.5
+ ctx = ann["bbox_context"].astype(np.float64)
+ corners = [to_frame(c) for c in ([0, 0], [size, 0], [0, size], [size, size])]
+ return centre, side, corners
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--std", type=Path, required=True)
+ p.add_argument("--ctx", type=Path, required=True)
+ p.add_argument("--ref", type=Path, required=True)
+ p.add_argument("--n", type=int, default=400)
+ args = p.parse_args()
+ R = load_module(args.ref, "train_distill_mhr_only_ref")
+
+ print("[1] baseline and v2 vs reference")
+ for preset in ("baseline", "v2"):
+ cfg = cfg_for(preset)
+ new, old = dataset(T, args.std, cfg), dataset(R, args.std, cfg)
+ diff = 0
+ for k in range(24):
+ a, b = seeded(new, k, 7 + k), seeded(old, k, 7 + k)
+ diff += not all(same(a[x], b[x]) for x in a)
+ check(diff == 0, f"{preset}: {diff} of 24 samples differ")
+
+ cfg = cfg_for("v3")
+ std0 = dataset(T, args.std, cfg_for("baseline"), augment=False)
+ ctx0 = dataset(T, args.ctx, cfg, augment=False, v3=True)
+ print("[2] identity window reproduces today's crop")
+ lab, cl, ps = 0.0, 0.0, []
+ for i in range(min(64, len(std0))):
+ a, b = std0[i], ctx0[i]
+ lab = max(lab, float((a["joints_2d"] - b["joints_2d"]).abs().max()))
+ cl = max(cl, float((a["cliff_cond"] - b["cliff_cond"]).abs().max()))
+ ia, ib = to_uint8(a["image"]), to_uint8(b["image"])
+ mse = np.mean((ia[8:-8, 8:-8] - ib[8:-8, 8:-8]) ** 2)
+ ps.append(10 * math.log10(255.0 ** 2 / max(mse, 1e-9)))
+ check(lab < 1e-5, f"joints_2d max diff {lab:.2e}")
+ check(cl < 1e-6, f"cliff_cond max diff {cl:.2e}")
+ check(min(ps) > 25.0, f"image PSNR mean {np.mean(ps):.1f} dB, min {min(ps):.1f} dB "
+ f"(448 context crop vs 224 crop, resampling noise)")
+
+ for size in (224, 288):
+ cfg = cfg_for("v3", size)
+ ds = dataset(T, args.ctx, cfg, augment=True, v3=True)
+ print(f"[3-7] augmented v3 at {size} px, {args.n} samples")
+ rep, cc, cs, outside, shapes, ms, shifts = 0.0, 0.0, 0.0, 0, set(), [], []
+ for k in range(args.n):
+ i = k % len(ds)
+ s = seeded(ds, i, 100 + k)
+ ann = dict(np.load(ds.pairs[i][1]))
+ shapes.add(tuple(s["image"].shape))
+ # 3: reprojection chain vs labels (visible keypoints, output px)
+ vis = s["joints_2d_vis"].numpy() > 0
+ err = np.abs(reproject(s, size) - s["joints_2d"].numpy())[vis] * size / 2
+ rep = max(rep, float(err.max()))
+ # 4: CLIFF vs the window recovered from aug_M. Augmented samples
+ # only: an un-augmented one keeps today's CLIFF (the tight-box
+ # centre), which is 0.5 px off the int()-rounded 1.2x square.
+ sc = Image.open(ds.pairs[i][0]).size[0]
+ centre, side, corners = window_from_sample(s, ann, size, sc)
+ h, w = ann["orig_shape"][:2]
+ flip, cos_t, sin_t = float(s["aug_flip"]), float(s["aug_cos"]), float(s["aug_sin"])
+ cx, cy = centre
+ if flip > 0.5:
+ cx = w - cx
+ d = np.array([[cos_t, sin_t], [-sin_t, cos_t]]) @ np.array([cx - w / 2, cy - h / 2])
+ f = float(ann["cam_focal_length"][1])
+ want = np.array([math.atan(d[0] / f), math.atan(d[1] / f), side / 1.2 / f])
+ got = s["cliff_cond"].numpy().astype(np.float64)
+ if float(s["aug_active"]) > 0.5:
+ cc = max(cc, float(np.abs(np.tan(got[:2]) - np.tan(want[:2])).max() * f))
+ cs = max(cs, abs(got[2] - want[2]) / want[2])
+ # 5: no corner past a stored-square edge that is inside the photo
+ ctx = ann["bbox_context"].astype(np.float64)
+ tol = 0.5 * (ctx[2] - ctx[0]) / sc # half a stored pixel
+ for c in corners:
+ outside += int((ctx[0] >= 0 and c[0] < ctx[0] - tol) or
+ (ctx[2] <= w and c[0] > ctx[2] + tol) or
+ (ctx[1] >= 0 and c[1] < ctx[1] - tol) or
+ (ctx[3] <= h and c[1] > ctx[3] + tol)) > 0
+ tight = ann["bbox"].astype(np.float64)
+ ts = max(tight[2] - tight[0], tight[3] - tight[1])
+ ms.append(side / ts)
+ tc = (tight[:2] + tight[2:]) / 2
+ shifts.append(float(np.abs(np.asarray(centre) - tc).max() / side))
+ ms, shifts = np.array(ms), np.array(shifts)
+ on = np.abs(ms - 1.2) > 1e-3
+ active = ms[on]
+ loose = on & (ms > 1.6)
+ p16, p18 = np.mean(active > 1.6), np.mean(active > 1.8)
+ ideal16 = math.log(2.0 / 1.6) / math.log(2.0 / 0.6)
+ ideal18 = math.log(2.0 / 1.8) / math.log(2.0 / 0.6)
+ check(p16 > 0.8 * ideal16,
+ f"loose tail: P(>1.6x) {p16:.3f} (unclamped {ideal16:.3f}), P(>1.8x) {p18:.3f} "
+ f"(unclamped {ideal18:.3f}); mean shift of >1.6x windows "
+ f"{shifts[loose].mean():.3f} of the side (unclamped 0.100)")
+ check(rep < 1e-3, f"reprojection chain vs labels: max {rep:.2e} px")
+ check(cc < 0.05 and cs < 1e-4, f"CLIFF vs recovered window: centre {cc:.3f} px, "
+ f"size rel {cs:.1e}")
+ check(outside == 0, f"{outside} samples read past a stored-square edge inside the photo")
+ lo, hi = active.min(), active.max()
+ frac = np.mean(active < 1.1)
+ expect = math.log(1.1 / 0.6) / math.log(2.0 / 0.6)
+ check(lo < 0.65 and hi > 1.8 and abs(frac - expect) < 0.08,
+ f"box multiple drawn {lo:.2f}-{hi:.2f}x; P(<1.1x) {frac:.2f} vs "
+ f"log-uniform {expect:.2f} ({len(active)} active draws)")
+ check(shapes == {(3, size, size)}, f"output shapes {shapes}")
+
+ print("[8] a standard split is refused")
+ try:
+ dataset(T, args.std, cfg_for("v3"), v3=True)
+ check(False, "standard split accepted by the v3 dataset")
+ except ValueError as e:
+ check("sam3d_gt_aic" in str(e), f"refused: {e}")
+
+ # a root mixing a context split (sorted first) with a standard one
+ import tempfile
+ mixed = Path(tempfile.mkdtemp(prefix="v3_mixed_"))
+ (mixed / "sam3d_gt_aic").symlink_to((args.ctx / "sam3d_gt_aic").resolve())
+ (mixed / "sam3d_gt_zstd").symlink_to((args.std / "sam3d_gt_aic").resolve())
+ try:
+ dataset(T, mixed, cfg_for("v3"), v3=True)
+ check(False, "mixed context + standard root accepted")
+ except ValueError as e:
+ check("sam3d_gt_zstd" in str(e), f"mixed root refused: {str(e)[:60]}...")
+ idx = JZ.build_pair_index(str(mixed), mixed / "idx.npz")
+ try:
+ T.SAM3DStudentDataset(str(mixed), augment=True, pairs=idx, context_crops=True)
+ check(False, "mixed root accepted through a PairIndex")
+ except ValueError as e:
+ check("sam3d_gt_zstd" in str(e), "mixed root refused through a PairIndex")
+
+ if fails:
+ print(f"FAIL ({len(fails)})")
+ sys.exit(1)
+ print("PASS")
+
+
+if __name__ == "__main__":
+ main()