Skip to content
Python

fix: skip unreadable samples instead of crashing; Harmony4D context-crop route; g9 tickets

mohamdev6a0ee8c11573
diff --git a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
index 9ffc66c..a4e2c57 100644
--- a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
+++ b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
@@ -23,3 +23,15 @@ Cost of 288 over 224 (same backbone): B4 x1.32 on the S23 NPU, x1.46-1.55 on CPU
- **The 288 arm is deployable on the NPU tier:** B4 at 288 is 1.9-5.5 ms across Snapdragon 8 Gen 1-Elite, far inside 33 ms. Ticket 10 may train it.
- **Not for the CPU tier with B4** (93-294 ms). B0 at 288 stays near budget on mid-range CPUs (Pixel 8 32.1, A73 38.2) because below B4 the decoder, not the image, dominates cost -- a 288 CPU-tier member is plausible if 288 proves worth it.
+
+## Addendum (2026-10-04): 384 and 448 px
+
+Same method; 16 of 24 profiles returned (B4@448 and two B0@448 phones missing: run timed out). Raw: `benchmark/results/family_latency_aihub_384_448.json`.
+
+| model | input | S23 NPU | S23 CPU | Pixel 8 CPU | A73 CPU | A53 CPU | A14 CPU |
+|---|---|---|---|---|---|---|---|
+| B4 | 384 | 4.18 | 146 | 158 | 186 | 445 | 454 |
+| B0 | 384 | 2.31 | 37 | 46 | 50 | 107 | 105 |
+| B0 | 448 | 2.81 | 42 | - | 59 | 130 | - |
+
+NPU tier affords 384+ (B4 4.2 ms, 100% NPU). CPU tier is capped at 224-288 (B0 over 33 ms on mid-range from 384). Above ~270 px the current context corpus (2.0x box at 448 px) has no extra detail: 384 needs a ~640 px rebuild. Fast SAM 3D Body (Table 4): PA-MPJPE 33.5 @256 -> 31.0 @384 -> 31.0 @448, i.e. saturation at 384.
diff --git a/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md b/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md
index ae23bc4..5c3a128 100644
--- a/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md
+++ b/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md
@@ -4,10 +4,20 @@
**Blocked by:** 01, 04
-**Status:** ready-for-human
+**Status:** done (2026-10-04); archive to STORE submitted
-- [ ] Launch commands printed in the build job's form, with the new flags and output directories
-- [ ] Crop counts match the current splits minus images that are missing at source (report the difference)
-- [ ] Verification passes per split (reprojection RMS ~0.000 px, zero missing or orphan)
-- [ ] Each split archived as one tar to `$STORE`
-- [ ] `$SCRATCH` usage recorded (expected ~250 GB total with ticket 09)
+- [x] Launch commands printed in the build job's form, with the new flags and output directories
+- [x] Crop counts match the current splits minus images that are missing at source (report the difference)
+- [x] Verification passes per split (reprojection RMS ~0.000 px, zero missing or orphan)
+- [x] Each split archived as one tar to `$STORE`
+- [x] `$SCRATCH` usage recorded (expected ~250 GB total with ticket 09)
+
+## Answer
+
+Context crops (2.0x box, 448 px) in `$SCRATCH/instanthmr/data_ctx/sam3d_gt_{coco,mpii,aic,sa1b}`: 39,665 / 7,135 / 287,391 / 3,401,720 -- every split equal to the 224 corpus.
+
+What it took:
+- `prepost` had no usable node for a week (pp1/pp4 draining), so builds moved to `21_build_gpu.slurm` (gpu_p13, qos_gpu-dev, 2 h, 10 processes, PART/PARTS to split one split across parallel jobs).
+- MPII photos exist only under `$DATA_ROOT/sam3d_gt_mpii/original_images`, not `$IMG_ROOT/mpii`.
+- SA-1B jobs died OUT_OF_MEMORY at ~40 min: `build_split.py` cached every decoded frame of a parquet shard. Bounded to 8 entries (output byte-identical, `verify_context_crops.py` PASS); the next round completed in 36 min.
+- Every kill was followed by a truncated-npz check on files written in the 3 min before it; none found.
diff --git a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
index 22b9f6d..1b2019b 100644
--- a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
+++ b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
@@ -4,12 +4,20 @@
**Blocked by:** None (04 done; route decided)
-**Status:** ready-for-human -- route 2 (laptop)
+**Status:** done (2026-10-05)
- [x] Route chosen from the `prepost` result and recorded
-- [ ] Undistortion blockers in the undistortion notes handled before the first full scene
-- [ ] 496,036 train crops (± source-missing), verified and archived to `$STORE`
+- [x] Undistortion blockers in the undistortion notes handled before the first full scene
+- [x] 496,036 train crops (± source-missing), verified and archived to `$STORE`
## Route decided (2026-09-27): laptop
`prepost` job 226661 returned `HTTP 302, 0 bytes`: Hugging Face's redirect to its file host `us.aws.cdn.hf.co` cannot be followed from Jean Zay compute nodes (the login node could, 2026-09-25; same failure as STATUS 7.1 on 2026-09-03). So stream on the laptop, scene by scene (65 GB free: download one zip, undistort, build context crops with `--context 2.0 --body-size 448 --output-dir`, delete), then rsync one tar of crops to Jean Zay's context data root.
+
+## Route changed (user, 2026-10-04): Jean Zay
+
+Laptop built scenes 01-02 (65,068 crops, verified vs June: labels identical, 38.5 dB, 0.08 px shift) then was stopped to free the machine. Remaining 13 scenes: login-node download + `datasets_pipeline/jeanzay/22_h4d_scene.slurm` per scene, into `data_ctx_h4d/` (kept out of `data_ctx` while generation 9 trains). Fixes found on the way: zips unpack without the `train/` prefix (stream_archives.py now moves them); pycolmap >= 3 renamed `Image.projection_matrix()` (vendored `lib/datasets/exo_camera.py` patched).
+
+## Answer
+
+496,036 context crops in `$SCRATCH/instanthmr/data_ctx_h4d/sam3d_gt_harmony4d`, every scene equal to the June count, 0 undistortion exceptions. Full corpus served through `data_all/` (symlinks); see STATUS "Full context corpus". Archive: `sam3d_gt_harmony4d_ctx.tar`.
diff --git a/.scratch/v3-framing-robustness/issues/10-v3-ablation.md b/.scratch/v3-framing-robustness/issues/10-v3-ablation.md
index 1c98df7..24b34db 100644
--- a/.scratch/v3-framing-robustness/issues/10-v3-ablation.md
+++ b/.scratch/v3-framing-robustness/issues/10-v3-ablation.md
@@ -2,11 +2,26 @@
**What to build:** measure what the v3 preset and a larger input buy. Three arms, 2 seeds each, on the rebuilt corpus's data mix: a g8h control (baseline preset on the same mix), v3 at 224 and v3 at 288. Current generation-8 recipe unless ticket 02 has already produced a winner.
-**Blocked by:** 06, 07, 08, 09
+**Blocked by:** 06, 07, 08 (09 descoped: Harmony4D added afterwards as its own measurement)
-**Status:** ready-for-human
+**Status:** launching (2026-10-04) as generation 9
- [ ] Launch commands printed in the launcher's form, with the rsync list from ticket 07 and a cluster-side `grep -c` check that the new trainer is in place
- [ ] Every checkpoint scored on EMDB-1 published protocol, on EMDB under 1.2x / 1.4x / 1.6x enlarged boxes, and on 3DPW test J14 with the adapter
- [ ] A table of accuracy against latency (from tickets 03 and 06), stating whether v3 reduces the EMDB published-protocol penalty by more than ~0.5 mm without losing on 3DPW
- [ ] Results recorded in the benchmark results and status docs
+
+## Scope change (user, 2026-10-04)
+
+- No control runs; reference is g8h_s0/s1. g9 vs g8 changes crops AND mix (no Harmony4D), so it is not a single-variable comparison.
+- Arms: v3 @ 224 and v3 @ 288, each on HGNetV2-B4 and B0 (folds ticket 11's CPU-tier member in). Seed 0 only. **300 epochs** (user choice, account deadline), 4 chained 20 h links per run.
+- Without Harmony4D now; Harmony4D (ticket 09, building on the laptop) is added later and its effect measured on its own.
+- Launch recipe and prerequisites: STATUS.md, "Generation 9".
+
+## Restart (2026-10-05)
+
+First launch (data_ctx only, no Harmony4D) crashed at epoch 12 on an empty npz. Restarted from scratch on `data_all` (all five context splits incl. Harmony4D) with g8h's mix MIX7, so g9 vs g8h is now a cleaner comparison (crops + preset + backbone/input size). Old runs kept as `_ctxonly_ep12_*`. No separate Harmony4D-delta generation.
+
+## Final scope (user, 2026-10-12)
+
+14 runs: HGNetV2 B0..B6 x input {224, 288}, 600 epochs, v3, `data_all` (incl. Harmony4D), MIX7, seed 0. Folds ticket 11's model family into this generation. Dataset loader now skips unreadable samples (logged) instead of killing the run. Launch block and costs: STATUS.md "Generation 9, final form".
diff --git a/.scratch/v3-framing-robustness/issues/12-build-egohumans.md b/.scratch/v3-framing-robustness/issues/12-build-egohumans.md
new file mode 100644
index 0000000..00a32ee
--- /dev/null
+++ b/.scratch/v3-framing-robustness/issues/12-build-egohumans.md
@@ -0,0 +1,23 @@
+# 12: Build EgoHumans as context crops on Jean Zay
+
+**What to build:** EgoHumans (`egohumans_train`) joins the corpus as context crops (2.0x box, 448 px), the way Harmony4D does in ticket 09: download on a Jean Zay LOGIN node, undistort and build on compute nodes. It is ~272k annotated persons over ~268k 4K frames from 89 takes in 7 multi-person sports sequences (tagging, lego, fencing, basketball, volleyball, badminton, tennis): close interactions and wide framings the current corpus is thin on.
+
+Known facts (STATUS.md 6-8, `datasets_pipeline/registry.py`, `undistort.md`):
+- **Annotations are missing everywhere.** The `egohumans_train` parquets failed to download on Jean Zay (Xet proxy error, 2026-09-03) and are not on the laptop either. Hugging Face file downloads work from a login node (Harmony4D, 2026-09-25), so fetch them there first.
+- **Images:** Google Drive folder `1JD963urzuzV_R_6FOVOtlx8UupwUuknR`, one `.tar.gz` per take under `<sequence>/<take>.tar.gz`. gdown file downloads ran at ~20 MB/s from Jean Zay, but folder *listing* got rate-limited (403) after repeated requests: list each of the 7 sequence folders separately, ~15 min apart if refused.
+- **Only the referenced takes are needed** (~107 GB of the advertised 550 GB). Shopping list: `datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/egohumans_train --list-groups 2`, intersected with the Drive listing.
+- **Unpack with `tar xzf <take>.tar.gz --strip-components=8`. Not optional:** the tarballs carry an 8-level absolute prefix. Only `exo/` cameras are referenced.
+- **Undistort** with `sam-3d-body/data/scripts/egohumans/undistort_images.py`. It shares the vendored `lib/` with Harmony4D, so the pycolmap >= 3 patch (ticket 09) applies; check it in one take first.
+- `stream_archives.py` does not support `gdrive`; write a per-take job like `22_h4d_scene.slurm` instead.
+- Keep the output out of `data_ctx` while a generation trains from it (pair-index fingerprint, see ticket 09 / STATUS).
+
+**Blocked by:** 09 (proves the login-download / compute-undistort route and the patched `lib/` on Jean Zay).
+
+**Status:** ready-for-agent (job script + one verified take), then ready-for-human (full download and launch)
+
+- [ ] `egohumans_train` parquets fetched on a login node; row count recorded
+- [ ] Shopping list of referenced takes built; download size recorded
+- [ ] One take downloaded, unpacked, undistorted and built; its crops checked (labels reproject to 0.000 px; a few crops visually on the person; no undistortion exceptions)
+- [ ] Per-take Jean Zay job written and the remaining takes built
+- [ ] Final crop count within a few hundred of the annotation row count (minus missing frames), recorded in STATUS
+- [ ] Archived to `$STORE` (`sam3d_gt_egohumans_ctx.tar`)
diff --git a/CLAUDE.md b/CLAUDE.md
index 9b51f74..59d7dee 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -382,6 +382,15 @@ early generations and no longer applies. A seed sweep buys ~0.2 mm, so it is
not a way to gain accuracy; keep 2 seeds per arm only to tell a >0.5 mm change
from noise.
+**Two corpora, two data roots, never mixed.** `$SCRATCH/instanthmr/data` holds
+the 224 px crops (baseline / v2 presets); `$SCRATCH/instanthmr/data_ctx` holds the
+context crops (2.0x box, 448 px, `build_split.py --context`) that only
+`--preset v3` can read. Each root has its own pair index. The builder refuses a
+context build without `--output-dir`, and the v3 dataset refuses any root that
+contains a standard split. Generation 9 is the first to train on `data_ctx`; see
+STATUS.md. When `prepost` has no free node, `datasets_pipeline/jeanzay/21_build_gpu.slurm`
+builds on the 2 h GPU dev queue instead.
+
**You cannot reach Jean Zay.** No SSH, no Slurm. Print commands for the user and
read the logs they paste back. Files reach the cluster by `rsync` of *specific
files* — never the whole tree, since some files are edited on both sides.
diff --git a/benchmark/results/family_latency_aihub_384_448.json b/benchmark/results/family_latency_aihub_384_448.json
new file mode 100644
index 0000000..731e065
--- /dev/null
+++ b/benchmark/results/family_latency_aihub_384_448.json
@@ -0,0 +1,194 @@
+[
+ {
+ "model": "random_b0_384",
+ "image_size": 384,
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jpv2692zg",
+ "ms": 2.31,
+ "peak_mb": 118.5,
+ "layers": {
+ "NPU": 809
+ }
+ },
+ {
+ "model": "random_b0_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jgj3vw31p",
+ "ms": 36.9,
+ "peak_mb": 150.8,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b0_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jpe6dl68g",
+ "ms": 50.2,
+ "peak_mb": 135.9,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b0_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "jgzz34z4g",
+ "ms": 106.88,
+ "peak_mb": 139.0,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b0_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "j5wye1y4g",
+ "ms": 105.39,
+ "peak_mb": 141.7,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b0_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "jg9olxomg",
+ "ms": 45.76,
+ "peak_mb": 194.2,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b4_384",
+ "image_size": 384,
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp1o4von5",
+ "ms": 4.18,
+ "peak_mb": 119.8,
+ "layers": {
+ "NPU": 872
+ }
+ },
+ {
+ "model": "random_b4_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jgd6xz66p",
+ "ms": 146.33,
+ "peak_mb": 196.7,
+ "layers": {
+ "CPU": 757
+ }
+ },
+ {
+ "model": "random_b4_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "j57oy7ong",
+ "ms": 186.47,
+ "peak_mb": 160.4,
+ "layers": {
+ "CPU": 757
+ }
+ },
+ {
+ "model": "random_b4_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "jp4el9e2g",
+ "ms": 444.66,
+ "peak_mb": 177.4,
+ "layers": {
+ "CPU": 757
+ }
+ },
+ {
+ "model": "random_b4_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A14 5G",
+ "job": "jpx0kd08p",
+ "ms": 453.94,
+ "peak_mb": 175.4,
+ "layers": {
+ "CPU": 757
+ }
+ },
+ {
+ "model": "random_b4_384",
+ "image_size": 384,
+ "unit": "cpu",
+ "device": "Google Pixel 8",
+ "job": "j5m9nd97g",
+ "ms": 157.76,
+ "peak_mb": 186.7,
+ "layers": {
+ "CPU": 757
+ }
+ },
+ {
+ "model": "random_b0_448",
+ "image_size": 448,
+ "unit": "npu",
+ "device": "Samsung Galaxy S23",
+ "job": "jg9olxqqg",
+ "ms": 2.81,
+ "peak_mb": 119.6,
+ "layers": {
+ "NPU": 809
+ }
+ },
+ {
+ "model": "random_b0_448",
+ "image_size": 448,
+ "unit": "cpu",
+ "device": "Samsung Galaxy S23",
+ "job": "jp1o4vmk5",
+ "ms": 41.9,
+ "peak_mb": 163.1,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b0_448",
+ "image_size": 448,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A73 5G",
+ "job": "jgd6xzmkp",
+ "ms": 58.9,
+ "peak_mb": 147.7,
+ "layers": {
+ "CPU": 717
+ }
+ },
+ {
+ "model": "random_b0_448",
+ "image_size": 448,
+ "unit": "cpu",
+ "device": "Samsung Galaxy A53 5G",
+ "job": "j57oy78qg",
+ "ms": 130.25,
+ "peak_mb": 147.4,
+ "layers": {
+ "CPU": 717
+ }
+ }
+]
\ No newline at end of file
diff --git a/datasets_pipeline/build_split.py b/datasets_pipeline/build_split.py
index 4ba0f60..dcff254 100755
--- a/datasets_pipeline/build_split.py
+++ b/datasets_pipeline/build_split.py
@@ -344,6 +344,13 @@ def main() -> None:
continue
relpath = resolved
if image not in img_cache:
+ # Bounded: a parquet shard's rows for one photo are adjacent, so
+ # a few entries keep every photo decoded once. Unbounded, the
+ # cache held every decoded frame of the shard -- ~10 MB each on
+ # SA-1B -- and 10 builder processes per job ran out of memory
+ # after ~40 min (Jean Zay 2026-10-04, OUT_OF_MEMORY).
+ if len(img_cache) >= 8:
+ img_cache.clear()
bgr = cv2.imread(str(args.image_dir / relpath))
img_cache[image] = (cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
if bgr is not None else None)
diff --git a/datasets_pipeline/stream_archives.py b/datasets_pipeline/stream_archives.py
index 26e7692..4cab3ae 100755
--- a/datasets_pipeline/stream_archives.py
+++ b/datasets_pipeline/stream_archives.py
@@ -59,6 +59,13 @@ def undistort_harmony4d(raw: Path, undist: Path, scene_rel: str,
the script appends `exo/` to whatever it is given. See undistort.md.
"""
scene_dir = raw / scene_rel
+ # The zips unpack as <scene>/, without the train/ or test/ prefix that the
+ # annotations' image paths and --seqs carry (verified 2026-10-04 on
+ # train/01_hugging.zip), so move the scene under its split folder.
+ flat = raw / Path(scene_rel).name
+ if not scene_dir.exists() and flat.is_dir():
+ scene_dir.parent.mkdir(parents=True, exist_ok=True)
+ flat.rename(scene_dir)
takes = sorted(d.name for d in scene_dir.iterdir() if d.is_dir())
if not takes:
raise SystemExit(f"no takes under {scene_dir}")
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index f13ffce..5a846e4 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -843,7 +843,30 @@ class SAM3DStudentDataset(Dataset):
def __len__(self):
return len(self.pairs)
+ MAX_BAD_SAMPLES = 1000
+
def __getitem__(self, idx):
+ """Load sample `idx`; if its files are unreadable, log the path and
+ substitute a random other sample instead of killing the run.
+
+ One unreadable file out of millions (an npz left empty by a builder
+ killed mid-write) stopped all four generation-9 runs twice, each time
+ at the same step because the seed fixes the sample order. On healthy
+ data the except branch never runs, so samples are bit-identical to
+ before. More than MAX_BAD_SAMPLES failures in one worker still raise:
+ that is a broken corpus, not a stray file."""
+ try:
+ return self._load_sample(idx)
+ except Exception as e: # noqa: BLE001
+ self._n_bad = getattr(self, "_n_bad", 0) + 1
+ bad = self.pairs[idx]
+ print(f"[dataset] unreadable sample skipped ({self._n_bad} in this worker): "
+ f"{bad[1]} / {bad[0]}: {type(e).__name__}: {e}", flush=True)
+ if self._n_bad > self.MAX_BAD_SAMPLES:
+ raise
+ return self[random.randrange(len(self))]
+
+ def _load_sample(self, idx):
img_path, npz_path = self.pairs[idx]
W = H = self.image_size