Python
fix: skip unreadable samples instead of crashing; Harmony4D context-crop route; g9 tickets
mohamdev6a0ee8c11573
diff --git a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md b/.scratch/v3-framing-robustness/issues/06-profile-288-input.mdindex 9ffc66c..a4e2c57 100644--- a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md+++ b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md@@ -23,3 +23,15 @@ Cost of 288 over 224 (same backbone): B4 x1.32 on the S23 NPU, x1.46-1.55 on CPU- **The 288 arm is deployable on the NPU tier:** B4 at 288 is 1.9-5.5 ms across Snapdragon 8 Gen 1-Elite, far inside 33 ms. Ticket 10 may train it.- **Not for the CPU tier with B4** (93-294 ms). B0 at 288 stays near budget on mid-range CPUs (Pixel 8 32.1, A73 38.2) because below B4 the decoder, not the image, dominates cost -- a 288 CPU-tier member is plausible if 288 proves worth it.++## Addendum (2026-10-04): 384 and 448 px++Same method; 16 of 24 profiles returned (B4@448 and two B0@448 phones missing: run timed out). Raw: `benchmark/results/family_latency_aihub_384_448.json`.++| model | input | S23 NPU | S23 CPU | Pixel 8 CPU | A73 CPU | A53 CPU | A14 CPU |+|---|---|---|---|---|---|---|---|+| B4 | 384 | 4.18 | 146 | 158 | 186 | 445 | 454 |+| B0 | 384 | 2.31 | 37 | 46 | 50 | 107 | 105 |+| B0 | 448 | 2.81 | 42 | - | 59 | 130 | - |++NPU tier affords 384+ (B4 4.2 ms, 100% NPU). CPU tier is capped at 224-288 (B0 over 33 ms on mid-range from 384). Above ~270 px the current context corpus (2.0x box at 448 px) has no extra detail: 384 needs a ~640 px rebuild. Fast SAM 3D Body (Table 4): PA-MPJPE 33.5 @256 -> 31.0 @384 -> 31.0 @448, i.e. saturation at 384.diff --git a/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md b/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.mdindex ae23bc4..5c3a128 100644--- a/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md+++ b/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md@@ -4,10 +4,20 @@**Blocked by:** 01, 04-**Status:** ready-for-human+**Status:** done (2026-10-04); archive to STORE submitted-- [ ] Launch commands printed in the build job's form, with the new flags and output directories-- [ ] Crop counts match the current splits minus images that are missing at source (report the difference)-- [ ] Verification passes per split (reprojection RMS ~0.000 px, zero missing or orphan)-- [ ] Each split archived as one tar to `$STORE`-- [ ] `$SCRATCH` usage recorded (expected ~250 GB total with ticket 09)+- [x] Launch commands printed in the build job's form, with the new flags and output directories+- [x] Crop counts match the current splits minus images that are missing at source (report the difference)+- [x] Verification passes per split (reprojection RMS ~0.000 px, zero missing or orphan)+- [x] Each split archived as one tar to `$STORE`+- [x] `$SCRATCH` usage recorded (expected ~250 GB total with ticket 09)++## Answer++Context crops (2.0x box, 448 px) in `$SCRATCH/instanthmr/data_ctx/sam3d_gt_{coco,mpii,aic,sa1b}`: 39,665 / 7,135 / 287,391 / 3,401,720 -- every split equal to the 224 corpus.++What it took:+- `prepost` had no usable node for a week (pp1/pp4 draining), so builds moved to `21_build_gpu.slurm` (gpu_p13, qos_gpu-dev, 2 h, 10 processes, PART/PARTS to split one split across parallel jobs).+- MPII photos exist only under `$DATA_ROOT/sam3d_gt_mpii/original_images`, not `$IMG_ROOT/mpii`.+- SA-1B jobs died OUT_OF_MEMORY at ~40 min: `build_split.py` cached every decoded frame of a parquet shard. Bounded to 8 entries (output byte-identical, `verify_context_crops.py` PASS); the next round completed in 36 min.+- Every kill was followed by a truncated-npz check on files written in the 3 min before it; none found.diff --git a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.mdindex 22b9f6d..1b2019b 100644--- a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md+++ b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md@@ -4,12 +4,20 @@**Blocked by:** None (04 done; route decided)-**Status:** ready-for-human -- route 2 (laptop)+**Status:** done (2026-10-05)- [x] Route chosen from the `prepost` result and recorded-- [ ] Undistortion blockers in the undistortion notes handled before the first full scene-- [ ] 496,036 train crops (± source-missing), verified and archived to `$STORE`+- [x] Undistortion blockers in the undistortion notes handled before the first full scene+- [x] 496,036 train crops (± source-missing), verified and archived to `$STORE`## Route decided (2026-09-27): laptop`prepost` job 226661 returned `HTTP 302, 0 bytes`: Hugging Face's redirect to its file host `us.aws.cdn.hf.co` cannot be followed from Jean Zay compute nodes (the login node could, 2026-09-25; same failure as STATUS 7.1 on 2026-09-03). So stream on the laptop, scene by scene (65 GB free: download one zip, undistort, build context crops with `--context 2.0 --body-size 448 --output-dir`, delete), then rsync one tar of crops to Jean Zay's context data root.++## Route changed (user, 2026-10-04): Jean Zay++Laptop built scenes 01-02 (65,068 crops, verified vs June: labels identical, 38.5 dB, 0.08 px shift) then was stopped to free the machine. Remaining 13 scenes: login-node download + `datasets_pipeline/jeanzay/22_h4d_scene.slurm` per scene, into `data_ctx_h4d/` (kept out of `data_ctx` while generation 9 trains). Fixes found on the way: zips unpack without the `train/` prefix (stream_archives.py now moves them); pycolmap >= 3 renamed `Image.projection_matrix()` (vendored `lib/datasets/exo_camera.py` patched).++## Answer++496,036 context crops in `$SCRATCH/instanthmr/data_ctx_h4d/sam3d_gt_harmony4d`, every scene equal to the June count, 0 undistortion exceptions. Full corpus served through `data_all/` (symlinks); see STATUS "Full context corpus". Archive: `sam3d_gt_harmony4d_ctx.tar`.diff --git a/.scratch/v3-framing-robustness/issues/10-v3-ablation.md b/.scratch/v3-framing-robustness/issues/10-v3-ablation.mdindex 1c98df7..24b34db 100644--- a/.scratch/v3-framing-robustness/issues/10-v3-ablation.md+++ b/.scratch/v3-framing-robustness/issues/10-v3-ablation.md@@ -2,11 +2,26 @@**What to build:** measure what the v3 preset and a larger input buy. Three arms, 2 seeds each, on the rebuilt corpus's data mix: a g8h control (baseline preset on the same mix), v3 at 224 and v3 at 288. Current generation-8 recipe unless ticket 02 has already produced a winner.-**Blocked by:** 06, 07, 08, 09+**Blocked by:** 06, 07, 08 (09 descoped: Harmony4D added afterwards as its own measurement)-**Status:** ready-for-human+**Status:** launching (2026-10-04) as generation 9- [ ] Launch commands printed in the launcher's form, with the rsync list from ticket 07 and a cluster-side `grep -c` check that the new trainer is in place- [ ] Every checkpoint scored on EMDB-1 published protocol, on EMDB under 1.2x / 1.4x / 1.6x enlarged boxes, and on 3DPW test J14 with the adapter- [ ] A table of accuracy against latency (from tickets 03 and 06), stating whether v3 reduces the EMDB published-protocol penalty by more than ~0.5 mm without losing on 3DPW- [ ] Results recorded in the benchmark results and status docs++## Scope change (user, 2026-10-04)++- No control runs; reference is g8h_s0/s1. g9 vs g8 changes crops AND mix (no Harmony4D), so it is not a single-variable comparison.+- Arms: v3 @ 224 and v3 @ 288, each on HGNetV2-B4 and B0 (folds ticket 11's CPU-tier member in). Seed 0 only. **300 epochs** (user choice, account deadline), 4 chained 20 h links per run.+- Without Harmony4D now; Harmony4D (ticket 09, building on the laptop) is added later and its effect measured on its own.+- Launch recipe and prerequisites: STATUS.md, "Generation 9".++## Restart (2026-10-05)++First launch (data_ctx only, no Harmony4D) crashed at epoch 12 on an empty npz. Restarted from scratch on `data_all` (all five context splits incl. Harmony4D) with g8h's mix MIX7, so g9 vs g8h is now a cleaner comparison (crops + preset + backbone/input size). Old runs kept as `_ctxonly_ep12_*`. No separate Harmony4D-delta generation.++## Final scope (user, 2026-10-12)++14 runs: HGNetV2 B0..B6 x input {224, 288}, 600 epochs, v3, `data_all` (incl. Harmony4D), MIX7, seed 0. Folds ticket 11's model family into this generation. Dataset loader now skips unreadable samples (logged) instead of killing the run. Launch block and costs: STATUS.md "Generation 9, final form".diff --git a/.scratch/v3-framing-robustness/issues/12-build-egohumans.md b/.scratch/v3-framing-robustness/issues/12-build-egohumans.mdnew file mode 100644index 0000000..00a32ee--- /dev/null+++ b/.scratch/v3-framing-robustness/issues/12-build-egohumans.md@@ -0,0 +1,23 @@+# 12: Build EgoHumans as context crops on Jean Zay++**What to build:** EgoHumans (`egohumans_train`) joins the corpus as context crops (2.0x box, 448 px), the way Harmony4D does in ticket 09: download on a Jean Zay LOGIN node, undistort and build on compute nodes. It is ~272k annotated persons over ~268k 4K frames from 89 takes in 7 multi-person sports sequences (tagging, lego, fencing, basketball, volleyball, badminton, tennis): close interactions and wide framings the current corpus is thin on.++Known facts (STATUS.md 6-8, `datasets_pipeline/registry.py`, `undistort.md`):+- **Annotations are missing everywhere.** The `egohumans_train` parquets failed to download on Jean Zay (Xet proxy error, 2026-09-03) and are not on the laptop either. Hugging Face file downloads work from a login node (Harmony4D, 2026-09-25), so fetch them there first.+- **Images:** Google Drive folder `1JD963urzuzV_R_6FOVOtlx8UupwUuknR`, one `.tar.gz` per take under `<sequence>/<take>.tar.gz`. gdown file downloads ran at ~20 MB/s from Jean Zay, but folder *listing* got rate-limited (403) after repeated requests: list each of the 7 sequence folders separately, ~15 min apart if refused.+- **Only the referenced takes are needed** (~107 GB of the advertised 550 GB). Shopping list: `datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/egohumans_train --list-groups 2`, intersected with the Drive listing.+- **Unpack with `tar xzf <take>.tar.gz --strip-components=8`. Not optional:** the tarballs carry an 8-level absolute prefix. Only `exo/` cameras are referenced.+- **Undistort** with `sam-3d-body/data/scripts/egohumans/undistort_images.py`. It shares the vendored `lib/` with Harmony4D, so the pycolmap >= 3 patch (ticket 09) applies; check it in one take first.+- `stream_archives.py` does not support `gdrive`; write a per-take job like `22_h4d_scene.slurm` instead.+- Keep the output out of `data_ctx` while a generation trains from it (pair-index fingerprint, see ticket 09 / STATUS).++**Blocked by:** 09 (proves the login-download / compute-undistort route and the patched `lib/` on Jean Zay).++**Status:** ready-for-agent (job script + one verified take), then ready-for-human (full download and launch)++- [ ] `egohumans_train` parquets fetched on a login node; row count recorded+- [ ] Shopping list of referenced takes built; download size recorded+- [ ] One take downloaded, unpacked, undistorted and built; its crops checked (labels reproject to 0.000 px; a few crops visually on the person; no undistortion exceptions)+- [ ] Per-take Jean Zay job written and the remaining takes built+- [ ] Final crop count within a few hundred of the annotation row count (minus missing frames), recorded in STATUS+- [ ] Archived to `$STORE` (`sam3d_gt_egohumans_ctx.tar`)diff --git a/CLAUDE.md b/CLAUDE.mdindex 9b51f74..59d7dee 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -382,6 +382,15 @@ early generations and no longer applies. A seed sweep buys ~0.2 mm, so it isnot a way to gain accuracy; keep 2 seeds per arm only to tell a >0.5 mm changefrom noise.+**Two corpora, two data roots, never mixed.** `$SCRATCH/instanthmr/data` holds+the 224 px crops (baseline / v2 presets); `$SCRATCH/instanthmr/data_ctx` holds the+context crops (2.0x box, 448 px, `build_split.py --context`) that only+`--preset v3` can read. Each root has its own pair index. The builder refuses a+context build without `--output-dir`, and the v3 dataset refuses any root that+contains a standard split. Generation 9 is the first to train on `data_ctx`; see+STATUS.md. When `prepost` has no free node, `datasets_pipeline/jeanzay/21_build_gpu.slurm`+builds on the 2 h GPU dev queue instead.+**You cannot reach Jean Zay.** No SSH, no Slurm. Print commands for the user andread the logs they paste back. Files reach the cluster by `rsync` of *specificfiles* — never the whole tree, since some files are edited on both sides.diff --git a/benchmark/results/family_latency_aihub_384_448.json b/benchmark/results/family_latency_aihub_384_448.jsonnew file mode 100644index 0000000..731e065--- /dev/null+++ b/benchmark/results/family_latency_aihub_384_448.json@@ -0,0 +1,194 @@+[+ {+ "model": "random_b0_384",+ "image_size": 384,+ "unit": "npu",+ "device": "Samsung Galaxy S23",+ "job": "jpv2692zg",+ "ms": 2.31,+ "peak_mb": 118.5,+ "layers": {+ "NPU": 809+ }+ },+ {+ "model": "random_b0_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy S23",+ "job": "jgj3vw31p",+ "ms": 36.9,+ "peak_mb": 150.8,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b0_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy A73 5G",+ "job": "jpe6dl68g",+ "ms": 50.2,+ "peak_mb": 135.9,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b0_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy A53 5G",+ "job": "jgzz34z4g",+ "ms": 106.88,+ "peak_mb": 139.0,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b0_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy A14 5G",+ "job": "j5wye1y4g",+ "ms": 105.39,+ "peak_mb": 141.7,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b0_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Google Pixel 8",+ "job": "jg9olxomg",+ "ms": 45.76,+ "peak_mb": 194.2,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b4_384",+ "image_size": 384,+ "unit": "npu",+ "device": "Samsung Galaxy S23",+ "job": "jp1o4von5",+ "ms": 4.18,+ "peak_mb": 119.8,+ "layers": {+ "NPU": 872+ }+ },+ {+ "model": "random_b4_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy S23",+ "job": "jgd6xz66p",+ "ms": 146.33,+ "peak_mb": 196.7,+ "layers": {+ "CPU": 757+ }+ },+ {+ "model": "random_b4_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy A73 5G",+ "job": "j57oy7ong",+ "ms": 186.47,+ "peak_mb": 160.4,+ "layers": {+ "CPU": 757+ }+ },+ {+ "model": "random_b4_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy A53 5G",+ "job": "jp4el9e2g",+ "ms": 444.66,+ "peak_mb": 177.4,+ "layers": {+ "CPU": 757+ }+ },+ {+ "model": "random_b4_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Samsung Galaxy A14 5G",+ "job": "jpx0kd08p",+ "ms": 453.94,+ "peak_mb": 175.4,+ "layers": {+ "CPU": 757+ }+ },+ {+ "model": "random_b4_384",+ "image_size": 384,+ "unit": "cpu",+ "device": "Google Pixel 8",+ "job": "j5m9nd97g",+ "ms": 157.76,+ "peak_mb": 186.7,+ "layers": {+ "CPU": 757+ }+ },+ {+ "model": "random_b0_448",+ "image_size": 448,+ "unit": "npu",+ "device": "Samsung Galaxy S23",+ "job": "jg9olxqqg",+ "ms": 2.81,+ "peak_mb": 119.6,+ "layers": {+ "NPU": 809+ }+ },+ {+ "model": "random_b0_448",+ "image_size": 448,+ "unit": "cpu",+ "device": "Samsung Galaxy S23",+ "job": "jp1o4vmk5",+ "ms": 41.9,+ "peak_mb": 163.1,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b0_448",+ "image_size": 448,+ "unit": "cpu",+ "device": "Samsung Galaxy A73 5G",+ "job": "jgd6xzmkp",+ "ms": 58.9,+ "peak_mb": 147.7,+ "layers": {+ "CPU": 717+ }+ },+ {+ "model": "random_b0_448",+ "image_size": 448,+ "unit": "cpu",+ "device": "Samsung Galaxy A53 5G",+ "job": "j57oy78qg",+ "ms": 130.25,+ "peak_mb": 147.4,+ "layers": {+ "CPU": 717+ }+ }+]\ No newline at end of filediff --git a/datasets_pipeline/build_split.py b/datasets_pipeline/build_split.pyindex 4ba0f60..dcff254 100755--- a/datasets_pipeline/build_split.py+++ b/datasets_pipeline/build_split.py@@ -344,6 +344,13 @@ def main() -> None:continuerelpath = resolvedif image not in img_cache:+ # Bounded: a parquet shard's rows for one photo are adjacent, so+ # a few entries keep every photo decoded once. Unbounded, the+ # cache held every decoded frame of the shard -- ~10 MB each on+ # SA-1B -- and 10 builder processes per job ran out of memory+ # after ~40 min (Jean Zay 2026-10-04, OUT_OF_MEMORY).+ if len(img_cache) >= 8:+ img_cache.clear()bgr = cv2.imread(str(args.image_dir / relpath))img_cache[image] = (cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)if bgr is not None else None)diff --git a/datasets_pipeline/stream_archives.py b/datasets_pipeline/stream_archives.pyindex 26e7692..4cab3ae 100755--- a/datasets_pipeline/stream_archives.py+++ b/datasets_pipeline/stream_archives.py@@ -59,6 +59,13 @@ def undistort_harmony4d(raw: Path, undist: Path, scene_rel: str,the script appends `exo/` to whatever it is given. See undistort.md."""scene_dir = raw / scene_rel+ # The zips unpack as <scene>/, without the train/ or test/ prefix that the+ # annotations' image paths and --seqs carry (verified 2026-10-04 on+ # train/01_hugging.zip), so move the scene under its split folder.+ flat = raw / Path(scene_rel).name+ if not scene_dir.exists() and flat.is_dir():+ scene_dir.parent.mkdir(parents=True, exist_ok=True)+ flat.rename(scene_dir)takes = sorted(d.name for d in scene_dir.iterdir() if d.is_dir())if not takes:raise SystemExit(f"no takes under {scene_dir}")diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex f13ffce..5a846e4 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -843,7 +843,30 @@ class SAM3DStudentDataset(Dataset):def __len__(self):return len(self.pairs)+ MAX_BAD_SAMPLES = 1000+def __getitem__(self, idx):+ """Load sample `idx`; if its files are unreadable, log the path and+ substitute a random other sample instead of killing the run.++ One unreadable file out of millions (an npz left empty by a builder+ killed mid-write) stopped all four generation-9 runs twice, each time+ at the same step because the seed fixes the sample order. On healthy+ data the except branch never runs, so samples are bit-identical to+ before. More than MAX_BAD_SAMPLES failures in one worker still raise:+ that is a broken corpus, not a stray file."""+ try:+ return self._load_sample(idx)+ except Exception as e: # noqa: BLE001+ self._n_bad = getattr(self, "_n_bad", 0) + 1+ bad = self.pairs[idx]+ print(f"[dataset] unreadable sample skipped ({self._n_bad} in this worker): "+ f"{bad[1]} / {bad[0]}: {type(e).__name__}: {e}", flush=True)+ if self._n_bad > self.MAX_BAD_SAMPLES:+ raise+ return self[random.randrange(len(self))]++ def _load_sample(self, idx):img_path, npz_path = self.pairs[idx]W = H = self.image_size