Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
v4.13.2 — Stop means stop
mrbizarroe1d500e36d2f
diff --git a/README.md b/README.mdindex 17aab86..fc3acb9 100644--- a/README.md+++ b/README.md@@ -19,7 +19,7 @@</p>-> **Current release: v4.13.1.** **Hailuo H3 that looks right**: High clips run twice the sampler steps, so faces and motion come out sharp (a 5 s High clip now takes about 34 min instead of 19); every clip keeps its last frame; Turbo applies its whole adapter; Upscale ×2 no longer flashes white at the end; twelve more H3 fixes. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).+> **Current release: v4.13.2.** **Stop means stop**: pressing Stop while Turbo is still preparing no longer lets the render start afterwards; H3 LoRAs imported before 4.13.1 are applied at their trained strength again; very large Upscale ×2 canvases decode safely (or are refused up front); a failed Turbo companion download is not retried by every job; a silent clip stays silent after Upscale ×2. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).## Overviewdiff --git a/VERSION b/VERSIONindex 56d2fb5..650edfe 100644--- a/VERSION+++ b/VERSION@@ -1 +1 @@-4.13.1+4.13.2diff --git a/docs/STATE.md b/docs/STATE.mdindex fd8feaa..e838b55 100644--- a/docs/STATE.md+++ b/docs/STATE.md@@ -1,6 +1,66 @@# Phosphene — project state, history, open work-> **🚀 2026-09-16 — v4.13.1 released: the H3 + ×2 fix wave (owner's order: "fix it and ship it").**+> **🛑 2026-09-17 — v4.13.2 released (public, tag `v4.13.2`): "Stop means stop" — five fixes from an independent review of 4.13.1.**+> **Why:** Codex re-reviewed the shipped 4.13.1 (PM hub `notes/ship-review-0917/REVIEW.md`): NOT ship-safe, 1 P1 + 4 P2.+> Each was reproduced on `efe5fcf` with CPU-only probes before fixing (`scratchpad/hotfix/repro_shipped.py`).+> **Fixes (dev `0e4ec45` `f39222c` `d44d36c`; public = one release commit cherry-picked onto `efe5fcf`, same patch-id):**+> - **P1 Stop.** `run_tracked_subprocess` (own group, registered, killed by Stop, timeout kills the group);+> `_register_job_pgid` publishes a pgid and reads the flag in ONE `LOCK` section, and `stop_current_job` sets the flag+> inside the section that reads the pgids. Used by the H3 renderer, both trainers, post-process mux, the Turbo+> companion fetch and the ×2 hold-tail/finish. Dispatch re-checks Stop after the fetch and before the spawn; the warm+> helper refuses a stopped job. Cancellation OWNER = the job the worker thread runs (`_JOB_CTX`, set around+> `run_job_inner`): children (One Shot parts `<take>-p1`, retakes) obey their take's Stop; off-queue work (editor+> proxies, film assembly) never inherits a render's Stop. `JobCancelled` lets best-effort steps re-raise.+> - **P2 LoRA strength.** Files whose per-module alphas are (or will be) folded list at 1.0; `_h3_lora_prepare` moves the+> sidecar to 1.0 and keeps `recommended_strength_before_fold`; a job without `h3_lora_scale_v` (queued by an older+> build) at that old number renders at 1.0 and the migrated value is written back to ITS `params.loras` entry (so Load+> Params / Finish replay it). `make_job` stamps `h3_lora_scale_v = 2`. Other strengths are kept.+> - **P3 int32 decode guard.** `_decode_tiling_peak_elements` checks the tiles the pinned tiler really cuts (causal extra+> frame, per-axis spatial size, last tile, 3×frames×canvas blend buffer). Temporal-only when its tiles are ≥4 latent+> frames; otherwise spatial + temporal (blended seams); refuse when nothing fits. ×2 admission refuses such canvases+> up front (`upscale_canvas_decodable`, `_decode_min_group_frames`). **Behaviour change:** 4096×2304 now decodes in+> 2048 px spatial + 64-frame tiles instead of 16-frame unblended temporal tiles. CPU check, real LTX-2.5 VAE, 384×512+> latent: spatial 256 px + 32-frame tiles 40.5 dB vs single pass; 16-frame temporal 31.5 dB.+> - **P4 companion fetch.** One fetch at a time (non-blocking lock, all callers), uuid temp file, failure remembered+> 6 h per target (in memory; Install Turbo / the managed download retry at once), Stop is not a failure,+> `/status` `turbo.adaln_fetch`.+> - **P5 ×2 sound.** The result always carries the source's sound or none (silent 121-frame sources kept the model's+> invented track before); `_probe_audio_state` tells "none" from "unknown" (unknown → optional-map mux + re-probe);+> the `.mux.mp4` temp is removed on every exit.+> **Codex review of the exact diff (3 rounds, gpt-6-astra xhigh):** R1 NOT SHIP-SAFE (recipe kept 0.5; temp file on+> Stop; editor proxy inherited a render's Stop; 17-frame admission too strict) → fixed. R2 NOT SHIP-SAFE (take parts+> ignored Stop; repeated-path LoRA writeback) → fixed. **R3: "SHIP-SAFE … No new serious defect or remaining release+> blocker was found in this diff."** Reports: PM hub `notes/ship-review-0917/HOTFIX_REVIEW{_R1,_R2,}.md`.+> Known, not fixed: `_join_take_parts` still uses plain `subprocess.run` (CPU join, pre-existing); failure memory resets+> on panel restart; a render meeting an in-flight companion fetch renders without it.+> **Gates:** new `test_h3_stop_and_companion` 22, `test_h3_lora_strength_migration` 14, `test_upscale_x2_tail` 31.+> Full `release_gates.sh` 95 PASS / 0 FAIL / 0 SKIP in the release worktree, MLX forced to the CPU (8199 rendering).+> **Clean room:** fresh clone of the release commit, fresh GitHub clone of the engine, `ltx_checkout.sh` + `ltx_venv.sh`+> + install's uv pip steps, no weights, empty state/outputs/HF; booted on :8398 → `/version` 4.13.2, UI 200, integrity+> OK, no tracebacks; stopped by PID, removed. Runner unchanged (`codex/h3-engine-v2` tip `11b90a0`).+> **Not run:** any render, update-path gate. **Pinokio post: NOT posted** — draft `launch/09_pinokio_4132_stop_fixes.md`+> (covers 4.13.1 + 4.13.2; 4.13.1 was never posted). **Next promote:** parent on the v4.13.2 release commit.++> **🧹 2026-09-17 — GitHub triage: PR #82 merged; #80 #81 closed (for the next release).**+> **PR #82** (Viktorminator): the CivitAI modal's Search button used `.pill-btn` (`width:100%`), so the query field+> shrank to 43 px and the button took 823 px of the row. Reproduced on 4.13.1 and verified fixed with static+> servers on :8471/:8472/:8473: at 1017 px the field is 799 px and the button 67 px; at 375 px, 233/69; the+> Hugging Face source is fine too; Enter and the button both call `/civitai/search`. `release_gates.sh --fast`+> in the PR worktree: 87 PASS / 0 FAIL / 2 SKIP (MLX on the CPU). **Squash-merged on public main as `efe5fcf`**,+> on top of the v4.13.1 release commit. VERSION is unchanged, so Pinokio users who press Update get this CSS fix+> under the 4.13.1 label. Cherry-picked to dev as `13bfebd`. **Next promote:** the snapshot commit's parent is+> `efe5fcf` (origin/main), and the dev tree already carries the change. **Release notes:** include "the CivitAI+> search field is full width again (thanks @Viktorminator, #82)".+> **Issues:** #80 and #81 are closed (fixed in 4.12.4; the reporter was silent after we asked). #62: Piotr's+> package is reviewed. The spec is the High recipe, all 5200 steps ran, all 52 captions carry `cjhtrn`, and the+> zip's `training_data/` is empty. Next step, promised on the thread: retrain his dataset here and measure+> the adapter's identity-family delta_rms (his is 7.8e-04; working characters are ~1.7e-03). This needs about 6 h of+> GPU time. Minor finding: `lora_lab/train_character.py` ignores the spec's `checkpoint_interval` and uses+> steps//5 instead; this does not change what gets learned. #63: cross-linked to #83. The staged runner still+> hardcodes `anchors = ("first",)`, so a last-frame anchor is not wired. #24: corrected the old Avoid description.+> Left alone: #78 and #48 (nudged 09-16 and 09-08), #61 (asked 09-13), #83 (answered 09-16).++> **🚀 2026-09-16 — v4.13.1 released (public `ba83d51`, tag `v4.13.1`): the H3 + ×2 fix wave (owner's order: "fix it and ship it").**> **Shipped (cherry-picked onto public 4.13.0, NOT a dev read-tree):** `3a5c5cc` autoplay only on click,> `85a7098` H3 reference image validated with PIL, `745dc9c` Turbo adaLN + provenance, `8eafdc5` High = 16> sigma points, `1e1ec51` ×2 int32 tail + full frame count, `3a4d9ad`, `a1c9808` the twelve round-2 fixes,@@ -12,8 +72,15 @@> `minimax-h3-mlx codex/h3-engine-v2`, which is `11b90a0` (mux keeps every frame, LoRA accounting). So H3> users get the runner fix by pressing Update; no reinstall.> **Gates:** full `release_gates.sh` 93 PASS / 0 FAIL / 0 SKIP in the release worktree, MLX forced to the CPU-> device via a `sitecustomize` shim (the GPU was rendering the owner's jobs on 8199). Clean-room boot of a-> fresh clone on a spare port: see the release report in the PM hub.+> device via a `sitecustomize` shim (the GPU was rendering the owner's jobs on 8199). **Clean room:** fresh clone of `ba83d51` in a temp dir; `ltx_checkout.sh` + `ltx_venv.sh` + the install's+> uv pip steps (ltx25.7 packages) with no weights, empty `HF_HOME`/state/outputs; panel booted on :8397 — `/version`+> 4.13.1 / `ba83d51`, UI 200 with the TEST PANEL badge, every High cell `steps 16 / forwards 15` (High 5 s ~34 min),+> model integrity OK, no tracebacks; `install_h3.js` branch clone + `h3_checkout.sh` → `11b90a0`. Stopped by PID,+> clone removed. **Not run:** update-path gate on a previous install and any render (no GPU tonight).+> **Pinokio post: NOT posted** — draft in the PM hub `launch/08_pinokio_4131_h3_fixes.md`, waiting for the owner's go.+> **Next promote:** public main is again a cherry-pick of dev; a later read-tree of dev is fine (it carries these+> commits) but will ship the held Editor audio work — only on the owner's word after he has used it.+> **🔎 2026-09-16 — H3 round-2 review: High runs 15 forwards, twelve bugs fixed (UNRELEASED, `dev`).**> **High = 16 points (`8eafdc5`).** The owner ran a matched gym A/B: High 1024×576 was clearly better at 15diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.pyindex cac47a3..64fdb45 100644--- a/mlx_ltx_panel.py+++ b/mlx_ltx_panel.py@@ -2961,7 +2961,31 @@ def _h3_turbo_asset(key: str | None = None) -> dict:"sha256": H3_TURBO_ASSET_SHA256, "bytes": H3_TURBO_ASSET_BYTES}-def _h3_turbo_fetch_embedder(target_dir, push_log, runner=None) -> bool:+# ONE companion fetch at a time, from any caller (the install button, the+# managed download, the first Turbo render): they share this lock instead of+# racing each other onto the same file. A failed attempt is remembered per+# target so a queue of Turbo jobs does not pay the same network timeout once+# per job; the Turbo install button (force=True) retries at any time.+_h3_turbo_embedder_lock = threading.Lock()+_h3_turbo_embedder_failures: dict[str, dict] = {}+H3_TURBO_EMBEDDER_RETRY_SEC = 6 * 3600+++def _h3_turbo_embedder_retry_state(target_dir=None) -> dict | None:+ """The remembered failure for the companion under `target_dir`, while its+ cooldown lasts: {"failed_at", "retry_at", "error"}; else None."""+ try:+ target = Path(target_dir if target_dir is not None else _h3_turbo_dir())+ except Exception: # noqa: BLE001+ return None+ rec = _h3_turbo_embedder_failures.get(str(target / H3_TURBO_EMBEDDER_FILE))+ if not rec or time.time() >= rec["retry_at"]:+ return None+ return dict(rec)+++def _h3_turbo_fetch_embedder(target_dir, push_log, runner=None, *,+ force: bool = False) -> bool:"""Fetch Turbo's adaLN companion (H3_TURBO_EMBEDDER_FILE, ~63 MB) with theH3 pack's own `scripts/fetch_time_embedder.py`, when the resolved adaptercarries adaLN pairs and the file is not already there.@@ -2969,7 +2993,13 @@ def _h3_turbo_fetch_embedder(target_dir, push_log, runner=None) -> bool:Best-effort by design: the adapter works without it (208 of 259 pairs),so a failure here is logged loudly and the render log repeats it — itnever fails the adapter download. Returns True when the file is present- afterwards (or was never needed)."""+ afterwards (or was never needed).++ Coordinated: one fetch at a time (a second caller returns False at once+ rather than waiting or writing the same file), each attempt writes its+ own temp file, and a failure is not retried for+ H3_TURBO_EMBEDDER_RETRY_SEC unless `force` (the install button). A+ JobCancelled from the runner — Stop during the fetch — is re-raised."""import subprocesstarget_dir = Path(target_dir)target = target_dir / H3_TURBO_EMBEDDER_FILE@@ -2978,6 +3008,16 @@ def _h3_turbo_fetch_embedder(target_dir, push_log, runner=None) -> bool:return Trueif _h3_real_file(target, H3_TURBO_EMBEDDER_MIN_BYTES):return True+ key = str(target)+ if not force:+ rec = _h3_turbo_embedder_retry_state(target_dir)+ if rec is not None:+ mins = max(1, int((rec["retry_at"] - time.time()) // 60))+ push_log(f"[h3:turbo] not fetching {H3_TURBO_EMBEDDER_FILE} again yet — "+ f"the last attempt failed ({rec['error']}); next automatic try "+ f"in ~{mins} min, or press Install Turbo to retry now. "+ f"Turbo's {resolved['adaln_pairs']} adaLN pairs will be skipped")+ return Falsepython = _h3_python()fetcher = H3_ROOT / "scripts" / "fetch_time_embedder.py"if python is None or not fetcher.is_file():@@ -2985,40 +3025,63 @@ def _h3_turbo_fetch_embedder(target_dir, push_log, runner=None) -> bool:f"(H3 pack venv or {fetcher.name} missing) — Turbo's "f"{resolved['adaln_pairs']} adaLN pairs will be skipped")return False- env = dict(os.environ)- hf_token = _active_hf_token()- if hf_token:- env["HF_TOKEN"] = hf_token- env["HUGGING_FACE_HUB_TOKEN"] = hf_token- # One small index JSON goes through the hub cache; keep it inside the H3- # models tree instead of ~/.cache/huggingface.- env.setdefault("HF_HOME", str(H3_MODELS / "hf_home"))- env["PYTHONUNBUFFERED"] = "1"- tmp = target.with_name(target.name + ".partial")- push_log(f"[h3:turbo] fetching {H3_TURBO_EMBEDDER_FILE} (~60 MB, four "- f"tensors range-read from MiniMaxAI/MiniMax-H3) for the "- f"adapter's {resolved['adaln_pairs']} adaLN pairs…")+ if not _h3_turbo_embedder_lock.acquire(blocking=False):+ push_log(f"[h3:turbo] {H3_TURBO_EMBEDDER_FILE} is already being fetched — "+ f"not starting a second download; Turbo's "+ f"{resolved['adaln_pairs']} adaLN pairs are skipped until it lands")+ return False+ import uuid as _uuid+ tmp = target.with_name(f"{target.name}.{_uuid.uuid4().hex}.partial")try:- target_dir.mkdir(parents=True, exist_ok=True)- proc = (runner or subprocess.run)(- [str(python), str(fetcher), "--out", str(tmp)],- capture_output=True, text=True, env=env, timeout=1800)- if proc.returncode != 0:- raise RuntimeError((proc.stderr or proc.stdout or "")[-300:].strip()- or f"exit {proc.returncode}")- if not _h3_real_file(tmp, H3_TURBO_EMBEDDER_MIN_BYTES):- raise RuntimeError("fetch wrote no usable file")- tmp.replace(target)- except Exception as e: # noqa: BLE001+ if _h3_real_file(target, H3_TURBO_EMBEDDER_MIN_BYTES):+ return True # landed while we waited for the lock+ env = dict(os.environ)+ hf_token = _active_hf_token()+ if hf_token:+ env["HF_TOKEN"] = hf_token+ env["HUGGING_FACE_HUB_TOKEN"] = hf_token+ # One small index JSON goes through the hub cache; keep it inside the+ # H3 models tree instead of ~/.cache/huggingface.+ env.setdefault("HF_HOME", str(H3_MODELS / "hf_home"))+ env["PYTHONUNBUFFERED"] = "1"+ push_log(f"[h3:turbo] fetching {H3_TURBO_EMBEDDER_FILE} (~60 MB, four "+ f"tensors range-read from MiniMaxAI/MiniMax-H3) for the "+ f"adapter's {resolved['adaln_pairs']} adaLN pairs…")+ try:+ target_dir.mkdir(parents=True, exist_ok=True)+ proc = (runner or subprocess.run)(+ [str(python), str(fetcher), "--out", str(tmp)],+ capture_output=True, text=True, env=env, timeout=1800)+ if proc.returncode != 0:+ raise RuntimeError((proc.stderr or proc.stdout or "")[-300:].strip()+ or f"exit {proc.returncode}")+ if not _h3_real_file(tmp, H3_TURBO_EMBEDDER_MIN_BYTES):+ raise RuntimeError("fetch wrote no usable file")+ tmp.replace(target)+ except JobCancelled:+ # The user stopped the job; that is not a failed download.+ push_log(f"[h3:turbo] {H3_TURBO_EMBEDDER_FILE} fetch stopped")+ raise+ except Exception as e: # noqa: BLE001+ now = time.time()+ _h3_turbo_embedder_failures[key] = {+ "failed_at": now, "retry_at": now + H3_TURBO_EMBEDDER_RETRY_SEC,+ "error": (str(e) or type(e).__name__)[:200]}+ push_log(f"[h3:turbo] WARNING: {H3_TURBO_EMBEDDER_FILE} fetch failed "+ f"({e}) — Turbo still works, but its adaLN pairs are "+ f"skipped; next automatic try in "+ f"{H3_TURBO_EMBEDDER_RETRY_SEC // 3600} h, or press "+ f"Install Turbo to retry now")+ return False+ _h3_turbo_embedder_failures.pop(key, None)+ push_log(f"[h3:turbo] adaLN companion installed → {target}")+ return True+ finally:try:tmp.unlink()except OSError:pass- push_log(f"[h3:turbo] WARNING: {H3_TURBO_EMBEDDER_FILE} fetch failed "- f"({e}) — Turbo still works, but its adaLN pairs are skipped")- return False- push_log(f"[h3:turbo] adaLN companion installed → {target}")- return True+ _h3_turbo_embedder_lock.release()def _h3_turbo_download_bg(target_dir, push_log, asset: dict | None = None) -> None:@@ -3032,7 +3095,7 @@ def _h3_turbo_download_bg(target_dir, push_log, asset: dict | None = None) -> Notarget_dir.mkdir(parents=True, exist_ok=True)target = target_dir / asset['file']if target.is_file():- _h3_turbo_fetch_embedder(target_dir, push_log)+ _h3_turbo_fetch_embedder(target_dir, push_log, force=True)_set_h3_turbo_dl(status="done", mb=total_mb, total_mb=total_mb,error=None)return@@ -3070,7 +3133,7 @@ def _h3_turbo_download_bg(target_dir, push_log, asset: dict | None = None) -> No"please retry")tmp.replace(target)push_log(f"[h3:turbo] adapter installed → {target}")- _h3_turbo_fetch_embedder(target_dir, push_log)+ _h3_turbo_fetch_embedder(target_dir, push_log, force=True)_set_h3_turbo_dl(status="done", mb=total_mb, total_mb=total_mb,error=None)except Exception as e: # noqa: BLE001@@ -3119,7 +3182,11 @@ def _h3_install_turbo(push_log, download_fn=None, embedder_fn=None) -> dict:total_mb=60, error=None)def _embedder_only():- ok = (embedder_fn or _h3_turbo_fetch_embedder)(target, push_log)+ if embedder_fn is not None:+ ok = embedder_fn(target, push_log)+ else:+ # The button is the explicit retry: past any cooldown.+ ok = _h3_turbo_fetch_embedder(target, push_log, force=True)_set_h3_turbo_dl(status="done" if ok else "error", mb=60,total_mb=60,error=None if ok else@@ -10533,6 +10600,9 @@ def h3_turbo_status() -> dict:"missing": paths["missing"],"note": h3_turbo_note(paths),"label": "Turbo",+ # A remembered companion-fetch failure (and when Turbo retries it by+ # itself); None when there is none. The install button retries now.+ "adaln_fetch": _h3_turbo_embedder_retry_state(paths.get("dir")),"adaln": {"pairs": paths.get("adaln_pairs", 0),"embedder": (str(paths["embedder"])if paths.get("embedder") else None),@@ -11456,6 +11526,101 @@ def _h3_lora_convert_kohya(path: Path) -> dict:"trigger_words": [trigger] if trigger else []}+# Bumped when the meaning of a picked H3 LoRA strength changes. 2 = per-module+# alphas are folded into lora_B (4.13.1), so a folded file reads as trained at+# 1.0. make_job stamps it on every H3 job; a job without it was queued by an+# older build, with the strength that build recommended.+H3_LORA_SCALE_VERSION = 2+++def _h3_lora_scale_state(path: Path) -> str:+ """How this file's PEFT scale is carried, from its header alone:+ "folded" the scale is inside lora_B (our conversions, exporters that say so)+ "mixed" per-module alphas that disagree — `_h3_lora_prepare` folds them+ "plain" anything else: the sidecar's strength applies as written+ """+ try:+ header, buf_start = _safetensors_header(path)+ except Exception: # noqa: BLE001+ return "plain"+ try:+ if _h3_lora_scale_report(path, header, buf_start)["source"] == "folded":+ return "folded"+ ratios = _h3_lora_alpha_ratios(path, header, buf_start)+ except Exception: # noqa: BLE001+ return "plain"+ if ratios and (max(ratios) - min(ratios)) >= 1e-6:+ return "mixed"+ return "plain"+++def _h3_lora_sidecar_json(path: Path) -> dict | None:+ try:+ data = json.loads(path.with_suffix(".json").read_text(encoding="utf-8"))+ except (OSError, ValueError):+ return None+ return data if isinstance(data, dict) else None+++def _h3_lora_migrate_sidecar(path: Path) -> float | None:+ """Move a folded file's sidecar to strength 1.0, keeping what it said.++ An adapter imported before its scale was folded into lora_B carries the+ strength that USED to be right (alpha/rank, e.g. 0.5) next to a file that+ now already contains it — applied again, the adapter is scaled twice. The+ old number is kept as `recommended_strength_before_fold`, which is what+ lets a job queued with it be recognised (see `_h3_lora_migrate_strength`).+ Returns that pre-fold strength (None when there is none)."""+ data = _h3_lora_sidecar_json(path)+ if data is None:+ return None+ before = data.get("recommended_strength_before_fold")+ if before is not None:+ try:+ return float(before)+ except (TypeError, ValueError):+ return None+ try:+ current = float(data.get("recommended_strength") or 1.0)+ except (TypeError, ValueError):+ return None+ if math.isclose(current, 1.0, abs_tol=1e-6):+ return None+ if _h3_lora_scale_state(path) != "folded":+ return None+ data["recommended_strength_before_fold"] = current+ data["recommended_strength"] = 1.0+ data["strength_migrated_at"] = iso_now()+ try:+ atomic_write_text(path.with_suffix(".json"), json.dumps(data, indent=2))+ except Exception as exc: # noqa: BLE001+ push(f"[h3:lora] WARN: could not update {path.with_suffix('.json').name} ({exc})")+ push(f"[h3:lora] {path.name}: its scale is now inside the file, so its "+ f"recommended strength is 1.0 (was {current:g})")+ return current+++def _h3_lora_migrate_strength(path: Path, strength: float, params: dict) -> float:+ """The strength to render a picked, already-prepared H3 LoRA at.++ A job queued by a build before the fold (no `h3_lora_scale_v`) that still+ carries the file's old automatic recommendation gets 1.0 — the same+ adapter as trained, instead of that scale applied a second time. Any+ other strength is the user's own choice and is kept."""+ before = _h3_lora_migrate_sidecar(path)+ if before is None or not math.isclose(strength, before, abs_tol=1e-4):+ return strength+ if params.get("h3_lora_scale_v"):+ push(f"[h3:lora] note: {path.name} is picked at {strength:g}, its old "+ f"recommended strength. The file now carries that scale itself and "+ f"reads as trained at 1.0 — {strength:g} renders it weaker.")+ return strength+ push(f"[h3:lora] {path.name}: this job was queued at the file's old "+ f"recommended strength {strength:g}; that scale is now inside the file, "+ f"so it renders at 1.0 (the same adapter, not scaled twice)")+ return 1.0++def _h3_lora_prepare(path: Path) -> dict:"""Make `path` loadable by the H3 runner, or raise saying why it can't be.@@ -11464,6 +11629,17 @@ def _h3_lora_prepare(path: Path) -> dict:before a render (so a file dropped in by hand, or a pre-existing download,gets the same treatment instead of failing inside the runner). Idempotent:a bare-layout file is inspected and returned untouched."""+ try:+ return _h3_lora_prepare_file(path)+ finally:+ # A file whose scale is (now) inside lora_B keeps no stale strength.+ try:+ _h3_lora_migrate_sidecar(path)+ except Exception: # noqa: BLE001+ pass+++def _h3_lora_prepare_file(path: Path) -> dict:info = _h3_lora_layout(path)if info["ok"]:folded = _h3_lora_fold_mixed_alphas(path)@@ -11598,6 +11774,7 @@ def _h3_lora_layout_cached(path: Path) -> dict:except Exception as exc:info = {"layout": "not_a_lora", "prefix": "", "pairs": 0, "ok": False,"convertible": False, "reason": str(exc)}+ info["scale_state"] = _h3_lora_scale_state(path)# Drop stale entries for the same path (an edited file leaves one behind).for stale in [k for k in _H3_LORA_LAYOUT_CACHE if k.split("|")[0] == str(path)]:_H3_LORA_LAYOUT_CACHE.pop(stale, None)@@ -11648,12 +11825,16 @@ def list_h3_user_loras() -> list[dict]:"size_bytes": size_bytes,"lane": "h3","trigger_words": list(meta.get("trigger_words") or []),- # A kohya file is always converted with alpha/rank folded into- # lora_B, so 1.0 is right. Imports before 2026-09-16 saved the- # trainer's alpha/rank here as well — a double scale — and are- # corrected on read rather than trusted.- "recommended_strength": (1.0 if meta.get("lora_layout") == "kohya"- else float(meta.get("recommended_strength") or 1.0)),+ # A file whose scale is inside lora_B (a kohya conversion, a+ # folded mixed-alpha file) or is about to be (mixed per-module+ # alphas, folded at the next render) reads as trained at 1.0.+ # Imports before 2026-09-16 saved alpha/rank here as well — a+ # double scale — so the sidecar is corrected on read, not trusted.+ "recommended_strength": (+ 1.0 if (meta.get("lora_layout") == "kohya"+ or info.get("layout") == "kohya"+ or info.get("scale_state") in ("folded", "mixed"))+ else float(meta.get("recommended_strength") or 1.0)),"preview_url": preview_url,"preview_type": preview_type,"base_model": meta.get("base_model") or "MiniMax H3",@@ -14259,16 +14440,27 @@ def _probe_video_frames(path: str) -> int:return 0-def _video_has_audio(path: str) -> bool:+def _probe_audio_state(path: str) -> str:+ """"audio", "none", or "unknown" (ffprobe failed or could not read it).++ The difference between the last two matters wherever the answer decides+ whether a soundtrack is dropped: "none" is a fact about the file,+ "unknown" is a fact about the probe."""try:- out = subprocess.run(- [str(FFPROBE), "-v", "error", "-select_streams", "a:0",+ r = subprocess.run(+ [str(FFPROBE), "-v", "error", "-select_streams", "a","-show_entries", "stream=codec_type", "-of", "csv=p=0", path],- capture_output=True, text=True, timeout=10,- ).stdout.strip()- return "audio" in out- except Exception:- return False+ capture_output=True, text=True, timeout=30,+ )+ except Exception: # noqa: BLE001+ return "unknown"+ if r.returncode != 0:+ return "unknown"+ return "audio" if "audio" in (r.stdout or "") else "none"+++def _video_has_audio(path: str) -> bool:+ return _probe_audio_state(path) == "audio"def ltx_grid_frames_up(frames: int) -> int:@@ -14302,6 +14494,38 @@ def upscale_frame_plan(src_frames: int, form_frames=None) -> tuple[int, int]:return out_frames, grid+# mlx_warm_helper._DECODE_MAX_ELEMENTS: the decoder's int32 limit, with headroom.+UPSCALE_DECODE_MAX_ELEMENTS = int(0.9 * 2 ** 31)+++def _decode_min_group_frames(frames: int) -> int:+ """Output frames of the longest temporal group when the decode is cut into+ the SMALLEST temporal tiles the pinned tiler allows (2 latent frames, no+ overlap): split_with_symmetric_overlaps + split_temporal_latents' causal+ shift, in plain arithmetic. 17 for long clips, 9 for a 17-frame clip."""+ f = max(1, (max(1, int(frames)) - 1) // 8 + 1)+ size = 2+ if f <= size:+ return 1 + (f - 1) * 8+ amount = (f + size - 1) // size+ starts = [i * size for i in range(amount)]+ ends = [s + size for s in starts]+ ends[-1] = f+ starts = starts[:1] + [s - 1 for s in starts[1:]]+ return max(1 + (e - 1) * 8 - s * 8 for s, e in zip(starts, ends))+++def upscale_canvas_decodable(width: int, height: int, frames: int) -> bool:+ """Whether ANY decode tiling keeps this canvas under the decoder's int32+ limit. Tiling cannot shrink the accumulation buffer the tiler keeps per+ temporal group — 3 × its frames × the full canvas — below what the+ shortest temporal tiles give (`_decode_min_group_frames`); every other+ tensor a spatial tile can shrink. Mirrors the refusal in the helper's+ `_int32_safe_decode_tiling`."""+ return (3 * _decode_min_group_frames(frames) * int(width) * int(height)+ <= UPSCALE_DECODE_MAX_ELEMENTS)++def _upscale_hold_tail_cmd(src: str, out: str, frames: int, grid: int) -> list[str]:"""ffmpeg argv: `src` with its last frame held until it is `grid` frames.@@ -14347,36 +14571,66 @@ def _upscale_finish_cmd(video: str, audio_source: str | None, out: str, *,def _upscale_finish(video_path: Path, audio_source: str, *, frames: int,- fps: float, trim: bool) -> bool:+ fps: float, trim: bool, job: dict | None = None) -> bool:"""Bring the ×2 render back to the source clip's length and soundtrack.Returns whether the source's audio is on the result. The whole point ofthe H3 → LTX ×2 lane: H3 wrote the dialogue and the sound; the upscaler- must not throw them away, nor a single frame of the picture."""- has_audio = _video_has_audio(audio_source)- if not (has_audio or trim):- return False+ must not throw them away, nor a single frame of the picture.++ THE SOUND POLICY IS UNCONDITIONAL: the result carries the source's sound+ or no sound. The IC pipeline writes a soundtrack of its own, and it used+ to survive whenever the source was silent AND no trim was needed — so a+ silent 121-frame clip came back with invented audio while a silent+ 124-frame clip came back silent. A source whose audio could not be+ PROBED is not treated as silent: its sound is muxed in if it has any."""+ state = _probe_audio_state(audio_source)+ want_source = state in ("audio", "unknown")tmp = video_path.with_name(video_path.stem + ".mux.mp4")- cmd = _upscale_finish_cmd(str(video_path), audio_source if has_audio else None,- str(tmp), frames=frames, fps=fps, trim=trim,- codec=output_codec_settings())- try:- r = subprocess.run(cmd, capture_output=True, text=True, timeout=900)+ codec = output_codec_settings()++ def _run(with_source: bool) -> tuple[bool, str]:+ cmd = _upscale_finish_cmd(str(video_path),+ audio_source if with_source else None,+ str(tmp), frames=frames, fps=fps, trim=trim,+ codec=codec)+ if with_source and state == "unknown":+ # Optional map: a source that really has no audio stream yields a+ # silent result instead of an ffmpeg error.+ cmd[cmd.index("1:a:0")] = "1:a:0?"+ try:+ r = run_tracked_subprocess(cmd, pgid_key="mux_pgid",+ label="the ×2 finish", job=job,+ timeout=900)+ except subprocess.TimeoutExpired:+ tmp.unlink(missing_ok=True)+ return False, "timed out"if r.returncode != 0 or not tmp.is_file():tmp.unlink(missing_ok=True)- if trim:- raise RuntimeError("Upscale ×2: could not cut the render back to "- f"{frames} frames: {(r.stderr or '')[-200:].strip()}")- push(f"Upscale: audio mux failed ({(r.stderr or '')[-160:].strip()}); clip kept silent")- return False+ return False, (r.stderr or "")[-200:].strip()+ return True, ""++ try:+ ok, err = _run(want_source)+ kept = want_source and ok+ if ok and state == "unknown":+ kept = _probe_audio_state(str(tmp)) == "audio"+ if not ok and want_source:+ push(f"Upscale ×2: could not carry the source sound over ({err}) — "+ "delivering the clip without sound")+ ok, err = _run(False)+ if not ok:+ raise RuntimeError(+ (f"Upscale ×2: could not cut the render back to {frames} frames: {err}"+ if trim else+ f"Upscale ×2: could not finish the clip (removing the model's own "+ f"soundtrack failed): {err}"))tmp.replace(video_path)- return has_audio- except subprocess.TimeoutExpired:+ return kept+ finally:+ # Stop, a spawn error, anything: the half-written temp sits NEXT TO+ # the output, where the gallery would list it.tmp.unlink(missing_ok=True)- if trim:- raise RuntimeError("Upscale ×2: cutting the render back to length timed out")- push("Upscale: audio mux timed out; clip kept silent")- return Falsedef _native_render_for(src: Path) -> Path:@@ -15564,6 +15818,23 @@ class WarmHelper:self._ensure()with self.lock:assert self.proc is not None and self.proc.stdin is not None+ # A Stop for THIS job that landed before the job reached the+ # helper (stop_current_job kills the helper, and _ensure just+ # respawned it) must not start the render anyway. Stop sets+ # the flag before it kills, so a kill that lands after this+ # check still ends the job the ordinary way.+ with LOCK:+ _cur = STATE.get("current")+ _stopped = bool(_cur is not None and job_spec.get("id")+ and _cur.get("id") == job_spec.get("id")+ and _cur.get("cancel_requested"))+ # A child of the worker's queue job (a take part) carries+ # its own id; its owner is the job this thread runs.+ _owner = _thread_job()+ _stopped = _stopped or bool(_owner is not None+ and _owner.get("cancel_requested"))+ if _stopped:+ raise JobCancelled("Stopped before the render started.")try:self.proc.stdin.write(json.dumps(job_spec) + "\n")self.proc.stdin.flush()@@ -16099,23 +16370,128 @@ def compute_upscale_plan(w: int, h: int, mode: str | None,}+class JobCancelled(RuntimeError):+ """Stop reached the job before (or while) one of its subprocesses ran.++ The worker already files any exception raised under `cancel_requested` as+ `cancelled`; the type exists so a best-effort step (the Turbo companion+ fetch) can tell "the user stopped this" apart from "the step failed" and+ re-raise instead of carrying on to the GPU."""+++# The queue job the CURRENT THREAD is running, set by the worker around+# run_job_inner. Cancellation belongs to that job only: editor proxies, film+# assembly and other HTTP-thread work run outside the queue, and must not+# inherit a Stop that was meant for whatever render happens to be current.+_JOB_CTX = threading.local()+++def _thread_job() -> dict | None:+ return getattr(_JOB_CTX, "job", None)+++def _cancel_requested(job: dict | None = None) -> bool:+ """Whether Stop was pressed for `job` — or for the queue job this thread+ is running, which OWNS every child it dispatches synchronously (a One+ Shot take's parts and retakes are their own dicts with their own ids,+ and Stop only marks the queued parent). No job = no cancellation owner."""+ if job is None:+ return False+ owner = _thread_job()+ with LOCK:+ return bool(job.get("cancel_requested")+ or (owner is not None and owner is not job+ and owner.get("cancel_requested")))+++def _raise_if_cancelled(job: dict | None, before: str) -> None:+ if _cancel_requested(job):+ raise JobCancelled(f"Stopped before {before}.")+++def _register_job_pgid(key: str, pgid: int, job: dict | None = None) -> None:+ """Publish a freshly spawned process group where Stop looks for it.++ THE ORDERING IS THE FIX. `stop_current_job` sets `cancel_requested` and+ reads every pgid inside one `LOCK` section; this registers the pgid and+ reads the flag inside another. Whichever section runs first, the other+ sees its effect: either Stop finds this pgid and kills it, or this finds+ the flag and kills the group itself. Registering first and checking a+ moment later outside the lock (the old shape) left a window where Stop+ saw neither, and the render it had just cancelled started anyway."""+ with LOCK:+ STATE[key] = pgid+ cancelled = _cancel_requested(job)+ if cancelled:+ try:+ os.killpg(pgid, signal.SIGKILL)+ except (ProcessLookupError, PermissionError, OSError):+ pass+ raise JobCancelled("Stopped before this step could start.")+++def _clear_job_pgid(key: str, pgid: int | None) -> None:+ with LOCK:+ if pgid is None or STATE.get(key) == pgid:+ STATE[key] = None+++def run_tracked_subprocess(cmd: list[str], *, pgid_key: str, label: str,+ job: dict | None = None, timeout: float | None = None,+ **popen_kw) -> subprocess.CompletedProcess:+ """`subprocess.run(capture_output=True, text=True)`, but Stop can end it.++ Refuses to start after Stop, runs in its own process group registered+ under `pgid_key` (which `stop_current_job` kills), and kills the whole+ group on timeout (then re-raises TimeoutExpired, like `subprocess.run`).+ A process that exits after Stop is returned as-is; callers that go on to+ more work check `_raise_if_cancelled` themselves. Raises JobCancelled+ when Stop ended the process."""+ _raise_if_cancelled(job, label)+ popen_kw.pop("capture_output", None)+ popen_kw.pop("text", None)+ proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,+ text=True, start_new_session=True, **popen_kw)+ # start_new_session: the child is its own group leader, pgid == pid (and+ # os.getpgid would race a child that has already exited).+ pgid = proc.pid+ try:+ _register_job_pgid(pgid_key, pgid, job)+ try:+ stdout, stderr = proc.communicate(timeout=timeout)+ except subprocess.TimeoutExpired:+ try:+ os.killpg(pgid, signal.SIGKILL)+ except (ProcessLookupError, PermissionError, OSError):+ pass+ proc.communicate()+ raise+ except BaseException:+ if proc.poll() is None:+ try:+ os.killpg(pgid, signal.SIGKILL)+ except (ProcessLookupError, PermissionError, OSError):+ pass+ try:+ proc.wait(timeout=5)+ except Exception: # noqa: BLE001+ pass+ raise+ finally:+ _clear_job_pgid(pgid_key, pgid)+ if proc.returncode != 0 and _cancel_requested(job):+ raise JobCancelled(f"Stopped during {label}.")+ return subprocess.CompletedProcess(cmd, proc.returncode, stdout or "", stderr or "")++def run_postprocess_tracked(cmd: list[str], label: str) -> tuple[str, str]:"""Run a post-process in its own process group so /stop can kill it."""push(f"{label}: " + " ".join(shlex.quote(c) for c in cmd))env = os.environ.copy()env["PATH"] = f"{FFMPEG_BIN}:{env.get('PATH', '')}"- proc = subprocess.Popen(- cmd, env=env, text=True,- stdout=subprocess.PIPE, stderr=subprocess.PIPE,- start_new_session=True,- )- with LOCK:- STATE["mux_pgid"] = os.getpgid(proc.pid)- try:- stdout, stderr = proc.communicate()- finally:- with LOCK:- STATE["mux_pgid"] = None+ proc = run_tracked_subprocess(cmd, pgid_key="mux_pgid", label=label,+ job=_thread_job(), env=env)+ stdout, stderr = proc.stdout, proc.stderrif proc.returncode != 0:push((stderr or stdout or "").strip())raise RuntimeError(f"{label.lower()} exited with code {proc.returncode}")@@ -16168,11 +16544,15 @@ def stop_current_job(timeout: float = 5.0) -> None:subprocess). Worker advances."""with LOCK:cur = STATE["current"]+ # Set BEFORE the pgids are read, in the same section: a worker that+ # spawns right now either registers its pgid first (read below and+ # killed) or sees this flag when it registers (and kills itself) —+ # see _register_job_pgid.+ if cur is not None:+ cur["cancel_requested"] = Truemux_pgid = STATE.get("mux_pgid")train_pgid = STATE.get("train_pgid")h3_pgid = STATE.get("h3_pgid")- if cur is not None:- cur["cancel_requested"] = Truepush("Stop requested — killing helper + ffmpeg post-process + training subprocess to abort current job.")HELPER.kill()# Image-engine subprocesses (HiDream's BF16 helper, mflux's per-family@@ -22280,6 +22660,8 @@ def make_job(form: dict[str, list[str]] | dict[str, str], *,"to spend it on your LoRA. Rendering at this shape's own "f"{_h3_steps or H3_TIERS[_h3_tier]['steps']} sigma points.")_tier_cfg = H3_TIERS[_h3_tier]+ # The picked LoRA strengths mean what this build's picker shows.+ job["params"]["h3_lora_scale_v"] = H3_LORA_SCALE_VERSION# Portrait = the cell's own canvas, rotated. Identical pixel count and# packed-row count, so the tier's measured estimate stays honest.if _h3_orientation == "portrait":@@ -23456,7 +23838,8 @@ def run_train_job_inner(job: dict) -> None:)with LOCK:STATE["pid"] = proc.pid- STATE["train_pgid"] = os.getpgid(proc.pid)+ # Registered and checked against Stop in one lock section.+ _register_job_pgid("train_pgid", proc.pid, job)# Stream stdout. Each line is either JSON (a structured progress# event) or plain text (forwarded as-is to the log).assert proc.stdout is not None@@ -23845,7 +24228,7 @@ def run_train_job_inner(job: dict) -> None:)with LOCK:STATE["pid"] = audio_proc.pid- STATE["train_pgid"] = os.getpgid(audio_proc.pid)+ _register_job_pgid("train_pgid", audio_proc.pid, job)assert audio_proc.stdout is not Nonefor raw in audio_proc.stdout:line = raw.rstrip("\n")@@ -23918,6 +24301,8 @@ def run_train_job_inner(job: dict) -> None:f"{audio_rc}")except FileNotFoundError as e:audio_failed_reason = f"audio trainer not available: {e}"+ except JobCancelled:+ raiseexcept Exception as e:audio_failed_reason = f"audio trainer raised: {e}"finally:@@ -24255,6 +24640,7 @@ def run_h3_job_inner(job: dict) -> None:user_lora_stack: list[tuple[Path, float]] = []_h3_lora_root = _h3_loras_dir()_picked: list[tuple[Path, float]] = []+ _picked_entries: list[dict] = [] # the recipe entry each pick came from_foreign: list[str] = []for _entry in (p.get("loras") or []):_raw = str((_entry or {}).get("path") or "").strip()@@ -24278,6 +24664,7 @@ def run_h3_job_inner(job: dict) -> None:except (TypeError, ValueError):_st = 1.0_picked.append((_rp, max(-2.0, min(2.0, _st))))+ _picked_entries.append(_entry)if _foreign:# Never silent: an LTX LoRA handed to H3's loader matches zero modules# and renders as though no adapter were attached at all.@@ -24312,8 +24699,19 @@ def run_h3_job_inner(job: dict) -> None:# files never went through the download path); a diffusers-namespace# file raises with the sentence naming the alpha problem._stack_layouts = []- for _pp, _ps in _picked:+ for _i, (_pp, _ps) in enumerate(_picked):_stack_layouts.append(_h3_lora_prepare(_pp).get("layout"))+ # After prepare, so a file folded just now is recognised too.+ _new = _h3_lora_migrate_strength(_pp, _ps, p)+ if _new != _ps and isinstance(_picked_entries[_i], dict):+ # Into the recipe as well — THIS entry only (a stack may list+ # the same file twice at different strengths): the sidecar+ # copies `loras`, and Load Params / Finish replay it with this+ # build's stamp, so a stale 0.5 there would read as deliberate.+ _picked_entries[_i]["strength"] = _new+ _picked[_i] = (_pp, _new)+ # The recipe now means what this build means by a strength.+ p["h3_lora_scale_v"] = H3_LORA_SCALE_VERSIONuser_lora, user_lora_strength = _picked[0]user_lora_stack = list(_picked)p["h3_lora_path"] = str(user_lora)@@ -24375,11 +24773,17 @@ def run_h3_job_inner(job: dict) -> None:f"folded v0.1 fallback) under {turbo_paths['dir']}.")if h3_turbo_adaln_state(turbo_paths) == "no_embedder":# An adapter downloaded before the adaLN companion came back.- # ~60 MB, once; a failure leaves the render running without the- # pairs, and the dispatch log below says so.+ # ~60 MB, once: a failure is remembered (retried after a cooldown+ # or from the Turbo install button), and the render runs without+ # the pairs, which the dispatch log below says out loud. The fetch+ # runs as a tracked process group, so Stop ends it — and a Stop+ # that lands during it ends the JOB, not just the download._h3_turbo_fetch_embedder(turbo_paths["dir"], push,- runner=lambda *a, **kw: subprocess.run(*a, **{**kw, "timeout": 300}))+ runner=lambda cmd, **kw: run_tracked_subprocess(+ cmd, pgid_key="h3_pgid", label="the Turbo companion download",+ job=job, **{**kw, "timeout": 300}))+ _raise_if_cancelled(job, "the H3 render")turbo_paths = h3_turbo_paths()steps = h3_turbo_steps(turbo_paths)@@ -24698,6 +25102,9 @@ def run_h3_job_inner(job: dict) -> None:_decode_est = (H3_DECODE_SEC_PER_PX_FRAME * int(width) * int(height)* int(window_frames) * _hw_speed_factor("h3"))last_step, total_steps = 0, max(1, steps - 1)+ # Last look before the GPU: a Stop pressed during any of the preparation+ # above (the companion fetch, the reference crop) ends the job here.+ _raise_if_cancelled(job, "the H3 render")try:proc = subprocess.Popen(cmd,@@ -24713,7 +25120,9 @@ def run_h3_job_inner(job: dict) -> None:)with LOCK:STATE["pid"] = proc.pid- STATE["h3_pgid"] = os.getpgid(proc.pid)+ # Registered and checked against Stop in ONE lock section, so a Stop+ # racing this spawn either finds the pgid or is found by it.+ _register_job_pgid("h3_pgid", proc.pid, job)# Crash guard — see reap_orphan_subprocesses(). start_new_session means# a SIGKILLed panel leaves this 40 GiB render running; the next boot# reaps it from this file before the queue resumes the same job.@@ -26126,6 +26535,13 @@ def run_job_inner(job: dict) -> None:# The form's generation length is not an upscale control; it only# widens the ceiling past the length this lane is measured at.out_frames, frames = upscale_frame_plan(src_frames, p.get("frames"))+ if not upscale_canvas_decodable(up_w, up_h, frames):+ # The helper refuses this too, but only after the whole render.+ raise RenderRefused(+ "hardware_tier",+ f"Upscale ×2 to {up_w}×{up_h} is past what the video decoder can "+ f"write safely, even in tiles (the GPU kernels' 2^31-element "+ f"limit). Use a smaller source clip.")if src_frames and out_frames < src_frames:push(f"Upscale ×2: {source.name} is {src_frames} frames; the ×2 covers "f"its first {out_frames}")@@ -26178,8 +26594,14 @@ def run_job_inner(job: dict) -> None:model_src = srcif frames > (src_frames or frames):model_src = str(work_dir / f"{source.stem}_hold{frames}.mp4")- _r = subprocess.run(_upscale_hold_tail_cmd(src, model_src, src_frames, frames),- capture_output=True, text=True, timeout=600)+ try:+ _r = run_tracked_subprocess(+ _upscale_hold_tail_cmd(src, model_src, src_frames, frames),+ pgid_key="mux_pgid", label="the ×2 preparation", job=job,+ timeout=600)+ except BaseException:+ shutil.rmtree(work_dir, ignore_errors=True)+ raiseif _r.returncode != 0 or _probe_video_frames(model_src) != frames:shutil.rmtree(work_dir, ignore_errors=True)raise RuntimeError(@@ -26223,12 +26645,13 @@ def run_job_inner(job: dict) -> None:+ (f"from the clip's own latent, {refine_steps}-step refine + Pixel Spatial Upscaler"if start_from == "source" else "full re-render from noise + Pixel Spatial Upscaler"))try:+ _raise_if_cancelled(job, "the ×2 render")result = HELPER.run(job_spec)if "seed_used" in result:push(f"seed used: {result['seed_used']}")p["seed_used"] = result["seed_used"]muxed = _upscale_finish(final_out, src, frames=out_frames, fps=src_fps,- trim=frames != out_frames)+ trim=frames != out_frames, job=job)finally:shutil.rmtree(work_dir, ignore_errors=True)delivered = _probe_video_frames(str(final_out))@@ -27669,7 +28092,11 @@ def worker_loop() -> None:# concurrently. Blocking acquire is correct here — the worker is# already serialized; it only waits if an inline image is mid-flight.with _GPU_LOCK:- run_job_inner(job)+ _JOB_CTX.job = job+ try:+ run_job_inner(job)+ finally:+ _JOB_CTX.job = Nonejob["status"] = "done"_CONSEC_FAIL.update(sig="", n=0)except JobStopped as stop:diff --git a/mlx_warm_helper.py b/mlx_warm_helper.pyindex f5b66a5..da5bdab 100644--- a/mlx_warm_helper.py+++ b/mlx_warm_helper.py@@ -797,36 +797,130 @@ def _vae_decode_max_latent_frames(h_lat: int, w_lat: int) -> int:return n-def _int32_safe_decode_tiling(orig, latent_shape, frame_rate=24.0):+def _decode_tiling_peak_elements(tiling, latent_shape, T=None) -> int:+ """The largest tensor a decode with `tiling` builds, from the tile+ extents the pinned tiler will ACTUALLY cut — the causal extra latent frame+ on every temporal tile after the first, the per-axis spatial tile size+ `prepare_tiles_for_decoding` derives, and the last tile running to the+ edge — plus the float32 accumulation buffer `tiled_decode` allocates per+ temporal group (3 × frames × the FULL canvas). Pure arithmetic over the+ tiler's own split functions; no tensor is built. `T` is the tiling module."""+ _, _, f_lat, h_lat, w_lat = (int(v) for v in latent_shape)+ if tiling is None:+ return _vae_decode_peak_elements(f_lat, h_lat, w_lat)+ if T is None:+ from ltx_core_mlx.model.video_vae import tiling as T+ t_starts, t_ends = [0], [f_lat]+ h_max, w_max = h_lat, w_lat+ sc = getattr(tiling, "spatial_config", None)+ if sc is not None:+ long_side = max(h_lat, w_lat)+ size_lat = sc.tile_size_in_pixels // 32+ overlap = sc.tile_overlap_in_pixels // 32+ lower = max(2, overlap + 1)+ maxes = []+ for n in (h_lat, w_lat):+ adjusted = max(lower, round(size_lat * n / long_side))+ iv = T.split_with_symmetric_overlaps(adjusted, overlap)(n)+ maxes.append(max(e - s for s, e in zip(iv.starts, iv.ends)))+ h_max, w_max = maxes+ tc = getattr(tiling, "temporal_config", None)+ if tc is not None:+ iv = T.split_temporal_latents(tc.tile_size_in_frames // 8,+ tc.tile_overlap_in_frames // 8)(f_lat)+ t_starts, t_ends = list(iv.starts), list(iv.ends)+ elif sc is not None:+ # tiled_decode needs a temporal slice with integer bounds; spatial+ # tiling without temporal tiling cannot run at all.+ return 1 << 62+ t_max = max(e - s for s, e in zip(t_starts, t_ends))+ peak = _vae_decode_peak_elements(t_max, h_max, w_max)+ out_frames = max(1 + (e - 1) * 8 - s * 8 for s, e in zip(t_starts, t_ends))+ return max(peak, 3 * out_frames * (32 * h_lat) * (32 * w_lat))+++def _decode_temporal_config(T, latent_frames: int, frame_rate: float):+ tile_frames = int(latent_frames) * 8+ one_second = max(8, (int(frame_rate) // 8) * 8)+ overlap = min(one_second, (tile_frames // 32) * 8)+ if overlap >= tile_frames:+ overlap = tile_frames - 8+ return T.TemporalTilingConfig(tile_size_in_frames=tile_frames,+ tile_overlap_in_frames=overlap)+++def _int32_safe_decode_tiling(orig, latent_shape, frame_rate=24.0, T=None):"""`_compute_decode_tiling`, plus an element cap next to its byte budget.Upstream only tiles when a decode would exceed a MEMORY budget (8 GB bydefault), so a 2048×1152 × 121-frame clip — 2.3 GB of block-3 activation —decodes in one pass and runs straight into the int32 wrap above. When the- pass is too big, this returns the same temporal tiling upstream uses for- long clips, with tiles small enough to stay under the cap. Everything+ pass is too big, this returns upstream's own temporal tiling with tiles+ small enough to stay under the cap; when no temporal tiling can (a canvas+ so large that two latent frames plus the causal third already wrap), it+ ALSO tiles in space. Every candidate is checked against the extents the+ tiler will really cut (`_decode_tiling_peak_elements`), and a canvas no+ tiling can fit is refused rather than decoded into garbage. Everythingthat already fits keeps upstream's answer untouched."""cfg = orig(latent_shape, frame_rate=frame_rate)+ if T is None:+ from ltx_core_mlx.model.video_vae import tiling as T+ cap = _DECODE_MAX_ELEMENTS++ def fits(c) -> bool:+ return _decode_tiling_peak_elements(c, latent_shape, T) <= cap++ if fits(cfg):+ return cfg_, _, f_lat, h_lat, w_lat = (int(v) for v in latent_shape)+ spatial = getattr(cfg, "spatial_config", None) if cfg is not None else None+ # 1. Temporal only (the 4.13.1 answer, now verified): split_temporal_latents+ # moves every tile after the first one latent frame earlier (causal+ # continuity), so those tiles are size + 1 long. Taken as-is when its+ # tiles are 4+ latent frames: those are 32+ frames with an 8-frame blend+ # between them. Shorter tiles have NO blend (overlap rounds to 0), and on+ # the real decoder that seam costs more than a blended spatial one+ # (CPU, LTX-2.5 VAE, 384×512: 16-frame temporal tiles 31.5 dB vs the+ # single pass; 256 px spatial + 32-frame temporal tiles 40.5 dB).max_lat = _vae_decode_max_latent_frames(h_lat, w_lat)- if f_lat <= max_lat:- return cfg- # split_temporal_latents moves every tile after the first one latent- # frame earlier (causal continuity), so those tiles are size + 1 long.size = max(2, max_lat - 1)- tc = getattr(cfg, "temporal_config", None) if cfg is not None else None- if tc is not None and tc.tile_size_in_frames // 8 <= size:- return cfg- from ltx_core_mlx.model.video_vae.tiling import TemporalTilingConfig, TilingConfig- tile_frames = size * 8- one_second = max(8, (int(frame_rate) // 8) * 8)- overlap = min(one_second, (tile_frames // 32) * 8)- if overlap >= tile_frames:- overlap = tile_frames - 8- return TilingConfig(- spatial_config=getattr(cfg, "spatial_config", None) if cfg is not None else None,- temporal_config=TemporalTilingConfig(- tile_size_in_frames=tile_frames, tile_overlap_in_frames=overlap))+ temporal_only = T.TilingConfig(+ spatial_config=spatial,+ temporal_config=_decode_temporal_config(T, size, frame_rate))+ if size >= 4 and fits(temporal_only):+ return temporal_only+ # 2. Space as well. Largest spatial tile first (fewest seams), each with+ # the longest temporal tile that fits; prefer temporal tiles of 4+.+ long_px = max(h_lat, w_lat) * 32+ overlap_px = 64+ spatial_sizes = []+ for k in range(2, 129):+ per_tile = -(-long_px // k) # ceil(long / k)+ px = -(-per_tile // 32) * 32 # up to a multiple of 32+ if px <= overlap_px + 32:+ break+ if px < long_px and px not in spatial_sizes:+ spatial_sizes.append(px)+ fallback = None+ for px in spatial_sizes:+ sc = T.SpatialTilingConfig(tile_size_in_pixels=px, tile_overlap_in_pixels=overlap_px)+ for t in range(max(2, f_lat), 1, -1):+ c = T.TilingConfig(spatial_config=sc,+ temporal_config=_decode_temporal_config(T, t, frame_rate))+ if fits(c):+ if t >= min(4, max(2, f_lat)):+ return c+ fallback = fallback or c+ break+ if fits(temporal_only):+ return temporal_only+ if fallback is not None:+ return fallback+ raise RuntimeError(+ f"This canvas ({w_lat * 32}×{h_lat * 32}) is too large for the video "+ f"decoder to write safely, even in tiles: its tensors would pass the "+ f"2^31-element limit of the GPU kernels and the frames would come out "+ f"corrupted. Use a smaller size.")def _install_decode_int32_guard() -> None:@@ -845,16 +939,21 @@ def _install_decode_int32_guard() -> None:returndef _guarded(latent_shape, frame_rate: float = 24.0):- cfg = _int32_safe_decode_tiling(orig, latent_shape, frame_rate=frame_rate)- tc =getattr(cfg, "temporal_config", None) if cfg is not None else None- if tc is not None:+ up = orig(latent_shape, frame_rate=frame_rate)+ cfg = _int32_safe_decode_tiling(lambda *_a, **_k: up, latent_shape,+ frame_rate=frame_rate)+ if cfg is not up:_, _, f_lat, h_lat, w_lat = (int(v) for v in latent_shape)- if f_lat > _vae_decode_max_latent_frames(h_lat, w_lat):- emit({"event": "log",- "line": f"vae-decode int32 guard: {f_lat} latent frames at "- f"{w_lat * 32}×{h_lat * 32} decode in tiles of "- f"{tc.tile_size_in_frames} frames "- f"(overlap {tc.tile_overlap_in_frames})"})+ tc = getattr(cfg, "temporal_config", None)+ sc = getattr(cfg, "spatial_config", None)+ emit({"event": "log",+ "line": f"vae-decode int32 guard: {f_lat} latent frames at "+ f"{w_lat * 32}×{h_lat * 32} decode in tiles of "+ f"{tc.tile_size_in_frames} frames "+ f"(overlap {tc.tile_overlap_in_frames})"+ + (f" × {sc.tile_size_in_pixels}px squares "+ f"(overlap {sc.tile_overlap_in_pixels}px)"+ if sc is not None else "")})return cfg_guarded._phosphene_int32_guard = Truediff --git a/test_h3_lora_strength_migration.py b/test_h3_lora_strength_migration.pynew file mode 100644index 0000000..c70712a--- /dev/null+++ b/test_h3_lora_strength_migration.py@@ -0,0 +1,331 @@+#!/usr/bin/env python3+"""An H3 LoRA imported before its scale was folded keeps its strength (ship review 2026-09-17).++Before 4.13.1, importing an adapter whose per-module alpha/rank were 0.5 and+1.0 saved a recommended strength of 0.5. 4.13.1 folds those ratios into each+lora_B at render time, so the file itself now carries them — but the sidecar+(and every job already queued with that recommendation) still said 0.5, and+the modules rendered at 0.25 and 0.5: the adapter, weakened. The listing+only corrected kohya imports.++Now: a folded (or about-to-be-folded) file lists at 1.0; its sidecar is moved+to 1.0 with the old number kept; a job queued by an older build at that old+number renders at 1.0; a strength anyone chose deliberately is kept.++MLX runs on the CPU (tiny tensors); no model, no GPU."""+from __future__ import annotations++import json+import os+import struct+import sys+import tempfile+import unittest+import unittest.mock+from contextlib import ExitStack+from pathlib import Path++ROOT = Path(__file__).resolve().parent+_TMP = Path(tempfile.mkdtemp(prefix="phos-h3-lora-mig-"))+for _k, _d in (("LTX_STATE_DIR", "state"), ("LTX_OUTPUT_DIR", "out"),+ ("LTX_UPLOADS_DIR", "uploads")):+ (_TMP / _d).mkdir(parents=True, exist_ok=True)+ os.environ[_k] = str(_TMP / _d)+os.environ["PHOSPHENE_ANALYTICS_DISABLED"] = "1"+os.environ["PHOSPHENE_DISABLE_VERSION_CHECK"] = "1"+os.environ["LTX_H3_FORCE_CAPABLE"] = "1"+os.environ.setdefault("LTX_PORT", "8299")+sys.path.insert(0, str(ROOT))++import mlx_ltx_panel as P # noqa: E402+++def _safetensors(path: Path, tensors: dict, meta: dict | None = None) -> None:+ header, blobs, off = {}, [], 0+ for name, (shape, vals) in tensors.items():+ raw = struct.pack(f"<{len(vals)}f", *vals)+ header[name] = {"dtype": "F32", "shape": list(shape),+ "data_offsets": [off, off + len(raw)]}+ blobs.append(raw)+ off += len(raw)+ if meta:+ header["__metadata__"] = {k: str(v) for k, v in meta.items()}+ hb = json.dumps(header).encode()+ hb += b" " * (-len(hb) % 8)+ path.write_bytes(struct.pack("<Q", len(hb)) + hb + b"".join(blobs))+++def _module(name: str, rank: int, alpha: float | None) -> dict:+ t = {f"{name}.lora_A.weight": ((rank, 3), [1.0] * (rank * 3)),+ f"{name}.lora_B.weight": ((4, rank), [1.0] * (4 * rank))}+ if alpha is not None:+ t[f"{name}.alpha"] = ((), [alpha])+ return t+++M1, M2 = "blocks.1.attn.qkv_proj", "blocks.2.attn.qkv_proj"+++class _LoraDir(unittest.TestCase):+ def setUp(self):+ import mlx.core as mx+ mx.set_default_device(mx.cpu)+ self.dir = Path(tempfile.mkdtemp(prefix="h3loras-", dir=_TMP))+ self.st = ExitStack()+ self.st.enter_context(unittest.mock.patch.object(P, "_h3_loras_dir", lambda: self.dir))+ self.logs = []+ self.st.enter_context(unittest.mock.patch.object(P, "push", self.logs.append))+ P._H3_LORA_LAYOUT_CACHE.clear()++ def tearDown(self):+ self.st.close()+ P._H3_LORA_LAYOUT_CACHE.clear()++ def imported(self, name, tensors, strength, meta=None, layout="bare"):+ """A file + the sidecar a pre-4.13.1 import wrote for it."""+ f = self.dir / f"{name}.safetensors"+ _safetensors(f, tensors, meta)+ f.with_suffix(".json").write_text(json.dumps({+ "name": name, "kind": "import", "lora_layout": layout,+ "recommended_strength": strength}))+ return f++ def mixed(self, name="mixed"):+ # alpha/rank 0.5 and 1.0; the old import picked 0.5 (farthest from 1).+ return self.imported(name, {**_module(M1, 4, 2.0), **_module(M2, 4, 4.0)}, 0.5)++ def listed(self, f):+ P._H3_LORA_LAYOUT_CACHE.clear()+ return next(e for e in P.list_h3_user_loras()+ if e["filename"] == f.name)["recommended_strength"]++ def sidecar(self, f):+ return json.loads(f.with_suffix(".json").read_text())++ def effective(self, f, strength):+ import mlx.core as mx+ w = mx.load(str(f))+ return [round(float(w[f"{m}.lora_B.weight"][0, 0].item()) * strength, 6)+ for m in (M1, M2)]++ def dispatch(self, f, strength, params):+ P._h3_lora_prepare(f)+ return P._h3_lora_migrate_strength(f, strength, params)+++class AFoldedFileReadsAsTrained(_LoraDir):+ def test_the_old_import_lists_at_one_before_any_render(self):+ f = self.mixed()+ self.assertEqual(P._h3_lora_scale_state(f), "mixed")+ self.assertEqual(self.listed(f), 1.0)+ self.assertEqual(self.sidecar(f)["recommended_strength"], 0.5) # untouched by a read++ def test_a_job_queued_before_the_update_renders_as_trained(self):+ f = self.mixed()+ st = self.dispatch(f, 0.5, {}) # no h3_lora_scale_v: old build+ self.assertEqual(st, 1.0)+ self.assertEqual(self.effective(f, st), [0.5, 1.0]) # as trained, not [0.25, 0.5]+ sc = self.sidecar(f)+ self.assertEqual((sc["recommended_strength"], sc["recommended_strength_before_fold"]),+ (1.0, 0.5))+ self.assertTrue(sc.get("strength_migrated_at"))+ self.assertEqual(self.listed(f), 1.0)+ self.assertTrue(any("renders at 1.0" in m for m in self.logs))++ def test_a_file_4_13_1_already_folded_is_corrected_too(self):+ f = self.mixed()+ P._h3_lora_fold_mixed_alphas(f) # what 4.13.1 did, sidecar untouched+ self.assertEqual(P._h3_lora_scale_state(f), "folded")+ self.assertEqual(self.sidecar(f)["recommended_strength"], 0.5)+ self.assertEqual(self.listed(f), 1.0)+ self.assertEqual(self.dispatch(f, 0.5, {}), 1.0)+ # and a second job from the same old queue, after the sidecar moved+ self.assertEqual(self.dispatch(f, 0.5, {}), 1.0)++ def test_a_deliberate_strength_is_kept(self):+ f = self.mixed()+ self.assertEqual(self.dispatch(f, 0.8, {}), 0.8)+ self.assertEqual(self.dispatch(f, 1.0, {}), 1.0)+ self.assertEqual(self.dispatch(f, -0.5, {}), -0.5)++ def test_a_job_queued_by_this_build_keeps_its_strength_and_says_so(self):+ f = self.mixed()+ self.logs.clear()+ v = {"h3_lora_scale_v": P.H3_LORA_SCALE_VERSION}+ self.assertEqual(self.dispatch(f, 0.5, v), 0.5)+ self.assertTrue(any("renders it weaker" in m for m in self.logs))+ self.assertEqual(self.dispatch(f, 1.0, v), 1.0)++ def test_a_uniform_alpha_keeps_its_strength(self):+ # Not folded: the strength IS where alpha/rank gets applied.+ f = self.imported("uniform", {**_module(M1, 4, 2.0), **_module(M2, 4, 2.0)}, 0.5)+ before = f.read_bytes()+ self.assertEqual(P._h3_lora_scale_state(f), "plain")+ self.assertEqual(self.listed(f), 0.5)+ self.assertEqual(self.dispatch(f, 0.5, {}), 0.5)+ self.assertEqual(f.read_bytes(), before)+ self.assertNotIn("recommended_strength_before_fold", self.sidecar(f))+ self.assertEqual(self.effective(f, 0.5), [0.5, 0.5])++ def test_a_kohya_conversion_saved_at_its_alpha_is_corrected(self):+ f = self.imported("kohya", {**_module(M1, 4, None), **_module(M2, 4, None)}, 0.25,+ meta={"converted_by": "phosphene", "converted_from": "kohya",+ "ss_network_alpha": "1"},+ layout="kohya")+ self.assertEqual(self.listed(f), 1.0)+ self.assertEqual(self.dispatch(f, 0.25, {}), 1.0)+ self.assertEqual(self.sidecar(f)["recommended_strength_before_fold"], 0.25)++ def test_an_exporter_folded_file_at_one_is_left_alone(self):+ f = self.imported("baked", {**_module(M1, 4, None)}, 1.0,+ meta={"baked_scale": "0.0625"})+ before = f.with_suffix(".json").read_text()+ self.assertEqual(self.dispatch(f, 1.0, {}), 1.0)+ self.assertEqual(self.dispatch(f, 0.3, {}), 0.3)+ self.assertEqual(f.with_suffix(".json").read_text(), before)++ def test_no_sidecar_is_fine(self):+ f = self.dir / "bare.safetensors"+ _safetensors(f, {**_module(M1, 4, 2.0), **_module(M2, 4, 4.0)})+ self.assertEqual(self.listed(f), 1.0)+ self.assertEqual(self.dispatch(f, 0.5, {}), 0.5)+ self.assertFalse(f.with_suffix(".json").exists())+++class TheRenderPathUsesIt(_LoraDir):+ def test_the_runner_gets_one_for_an_old_queued_job(self):+ f = self.mixed()+ job = {"id": "lora-old", "params": {+ "engine": "h3", "mode": "t2v", "h3_tier": "standard_5s",+ "prompt": "a dancer", "seed": "7", "h3_turbo": False,+ "loras": [{"path": str(f), "strength": 0.5}]}}+ seen = {}++ class Intercept(Exception):+ pass++ def popen(cmd, **kw):+ seen["cmd"] = cmd+ raise Intercept()++ paths = dict(missing=[], repairable=False, dit=_TMP / "dit", python=sys.executable,+ runner=_TMP / "gen.py", compact_root=_TMP / "c", text_config=_TMP / "t")+ ff = _TMP / "ffmpeg"+ ff.write_text("")+ with ExitStack() as st:+ for name, val in {+ "h3_paths": lambda: paths, "h3_capable": lambda: True,+ "h3_supports_lora": lambda: True, "h3_supports_lora_stack": lambda: False,+ "h3_lora_max_stack": lambda: 1, "h3_dit_choice": lambda: ("bf16", None),+ "output_codec_settings": lambda: {"crf": "18", "pix_fmt": "yuv420p"},+ "_h3_runner_has_flag": lambda flag: False, "FFMPEG": ff}.items():+ st.enter_context(unittest.mock.patch.object(P, name, val))+ for name in ("h3_supports_memory_limits", "h3_supports_prompt_cache",+ "h3_live_preview_ready", "h3_supports_stage_a",+ "h3_supports_tae_draft"):+ st.enter_context(unittest.mock.patch.object(P, name, lambda: False))+ st.enter_context(unittest.mock.patch.object(P.HELPER, "is_alive", lambda: False))+ st.enter_context(unittest.mock.patch.object(P.subprocess, "Popen", popen))+ with self.assertRaises(Intercept):+ P.run_h3_job_inner(job)+ cmd = seen["cmd"]+ spec = cmd[cmd.index("--lora") + 1]+ self.assertEqual(spec, f"{f.resolve()}:1")+ self.assertEqual(job["params"]["h3_lora_strength"], 1.0)+ self.assertEqual(self.effective(f, 1.0), [0.5, 1.0])++ def _dispatch_argv(self, job, stack=False):+ seen = {}++ class Intercept(Exception):+ pass++ def popen(cmd, **kw):+ seen["cmd"] = cmd+ raise Intercept()++ paths = dict(missing=[], repairable=False, dit=_TMP / "dit", python=sys.executable,+ runner=_TMP / "gen.py", compact_root=_TMP / "c", text_config=_TMP / "t")+ ff = _TMP / "ffmpeg"+ ff.write_text("")+ with ExitStack() as st:+ for name, val in {+ "h3_paths": lambda: paths, "h3_capable": lambda: True,+ "h3_supports_lora": lambda: True,+ "h3_supports_lora_stack": lambda: stack,+ "h3_lora_max_stack": lambda: 4 if stack else 1,+ "h3_dit_choice": lambda: ("bf16", None),+ "output_codec_settings": lambda: {"crf": "18", "pix_fmt": "yuv420p"},+ "_h3_runner_has_flag": lambda flag: False, "FFMPEG": ff}.items():+ st.enter_context(unittest.mock.patch.object(P, name, val))+ for name in ("h3_supports_memory_limits", "h3_supports_prompt_cache",+ "h3_live_preview_ready", "h3_supports_stage_a",+ "h3_supports_tae_draft"):+ st.enter_context(unittest.mock.patch.object(P, name, lambda: False))+ st.enter_context(unittest.mock.patch.object(P.HELPER, "is_alive", lambda: False))+ st.enter_context(unittest.mock.patch.object(P.subprocess, "Popen", popen))+ with self.assertRaises(Intercept):+ P.run_h3_job_inner(job)+ cmd = seen["cmd"]+ got = [float(cmd[i + 1].rsplit(":", 1)[1]) for i, a in enumerate(cmd) if a == "--lora"]+ return got if stack else got[0]++ def test_a_repeated_file_in_a_stack_migrates_only_its_own_entry(self):+ # Codex hotfix review round 2: the same file twice, once at the old+ # recommendation and once deliberately.+ f = self.mixed()+ job = {"id": "lora-dup", "params": {+ "engine": "h3", "mode": "t2v", "h3_tier": "standard_5s",+ "prompt": "a dancer", "seed": "7", "h3_turbo": False,+ "loras": [{"path": str(f), "strength": 0.5},+ {"path": str(f), "strength": 0.7}]}}+ self.assertEqual(self._dispatch_argv(job, stack=True), [1.0, 0.7])+ self.assertEqual([l["strength"] for l in job["params"]["loras"]], [1.0, 0.7])+ replay = P.make_job({k: [v] for k, v in {+ "mode": "t2v", "engine": "h3", "prompt": "a dancer", "h3_turbo": "0",+ "loras": json.dumps(job["params"]["loras"])}.items()})+ replay["id"] = "lora-dup2"+ self.assertEqual(self._dispatch_argv(replay, stack=True), [1.0, 0.7])++ def test_the_corrected_strength_survives_load_params_and_finish(self):+ # Codex hotfix review #1: the recipe the sidecar copies (and Load+ # Params / Finish replay) must carry the migrated strength.+ f = self.mixed()+ job = {"id": "lora-rt", "params": {+ "engine": "h3", "mode": "t2v", "h3_tier": "standard_5s",+ "prompt": "a dancer", "seed": "7", "h3_turbo": False,+ "loras": [{"path": str(f), "strength": 0.5, "name": "mixed"}]}}+ self.assertEqual(self._dispatch_argv(job), 1.0)+ rec = job["params"]+ self.assertEqual(rec["loras"][0]["strength"], 1.0)+ self.assertEqual(rec["loras"][0]["name"], "mixed") # entry kept+ self.assertEqual(rec["h3_lora_scale_v"], P.H3_LORA_SCALE_VERSION)+ # What queue.js restores from the sidecar, through a real make_job.+ replay = P.make_job({k: [v] for k, v in {+ "mode": "t2v", "engine": "h3", "prompt": "a dancer", "h3_turbo": "0",+ "loras": json.dumps([{"path": l["path"], "strength": l["strength"]}+ for l in rec["loras"]])}.items()})+ self.assertEqual(replay["params"]["loras"][0]["strength"], 1.0)+ replay["id"] = "lora-rt2"+ self.assertEqual(self._dispatch_argv(replay), 1.0)++ def test_a_deliberate_strength_is_not_rewritten_in_the_recipe(self):+ f = self.mixed()+ job = {"id": "lora-keep", "params": {+ "engine": "h3", "mode": "t2v", "h3_tier": "standard_5s",+ "prompt": "a dancer", "seed": "7", "h3_turbo": False,+ "loras": [{"path": str(f), "strength": 0.7}]}}+ self.assertEqual(self._dispatch_argv(job), 0.7)+ self.assertEqual(job["params"]["loras"][0]["strength"], 0.7)++ def test_make_job_stamps_the_scale_version(self):+ j = P.make_job({k: [v] for k, v in {+ "mode": "t2v", "engine": "h3", "prompt": "a dancer", "h3_turbo": "0"}.items()})+ self.assertEqual(j["params"].get("h3_lora_scale_v"), P.H3_LORA_SCALE_VERSION)+ ltx = P.make_job({k: [v] for k, v in {+ "mode": "t2v", "engine": "ltx", "prompt": "a dancer"}.items()})+ self.assertNotIn("h3_lora_scale_v", ltx["params"])+++if __name__ == "__main__":+ unittest.main()diff --git a/test_h3_stop_and_companion.py b/test_h3_stop_and_companion.pynew file mode 100644index 0000000..be185ec--- /dev/null+++ b/test_h3_stop_and_companion.py@@ -0,0 +1,542 @@+#!/usr/bin/env python3+"""Stop means stop, and the Turbo companion is fetched once (ship review 2026-09-17).++P1: an H3 Turbo job whose adapter lacks its adaLN companion fetched it with a+plain subprocess, outside anything Stop knew about, and then spawned the+renderer without looking at `cancel_requested`. A Stop pressed during the+fetch was followed by a full GPU render. Registration of every tracked+process group also raced Stop: the pgid was published, and the flag read,+in separate moments.++P4: a failed "once" fetch was retried by every Turbo job (up to 300 s each,+holding the GPU gate), outside the installer's lock, on a shared .partial.++No GPU, no network, no model: the renderer's Popen is replaced by a short+`sleep` (or refused outright), and the fetch runner is a fake."""+from __future__ import annotations++import os+import subprocess+import sys+import tempfile+import threading+import time+import unittest+import unittest.mock+from contextlib import ExitStack+from pathlib import Path++ROOT = Path(__file__).resolve().parent+_TMP = Path(tempfile.mkdtemp(prefix="phos-h3-stop-"))+for _k, _d in (("LTX_STATE_DIR", "state"), ("LTX_OUTPUT_DIR", "out"),+ ("LTX_UPLOADS_DIR", "uploads")):+ (_TMP / _d).mkdir(parents=True, exist_ok=True)+ os.environ[_k] = str(_TMP / _d)+os.environ["PHOSPHENE_ANALYTICS_DISABLED"] = "1"+os.environ["PHOSPHENE_DISABLE_VERSION_CHECK"] = "1"+os.environ.setdefault("LTX_PORT", "8298")+sys.path.insert(0, str(ROOT))++import mlx_ltx_panel as P # noqa: E402+++def _alive(pid: int) -> bool:+ try:+ os.kill(pid, 0)+ except ProcessLookupError:+ return False+ # A zombie still answers kill(0); ask ps for its state.+ out = subprocess.run(["ps", "-o", "stat=", "-p", str(pid)],+ capture_output=True, text=True).stdout.strip()+ return bool(out) and not out.startswith("Z")+++class _CurrentJob:+ """Make `job` the worker's current job for the duration."""++ def __init__(self, job):+ self.job = job++ def __enter__(self):+ with P.LOCK:+ self.saved = (P.STATE.get("current"), P.STATE.get("h3_pgid"),+ P.STATE.get("mux_pgid"))+ P.STATE["current"] = self.job+ return self.job++ def __exit__(self, *exc):+ with P.LOCK:+ P.STATE["current"], P.STATE["h3_pgid"], P.STATE["mux_pgid"] = self.saved+ return False+++class TrackedProcessesObeyStop(unittest.TestCase):+ def test_a_stopped_job_never_spawns(self):+ job = {"id": "t-pre", "cancel_requested": True}+ with _CurrentJob(job), unittest.mock.patch.object(+ P.subprocess, "Popen", side_effect=AssertionError("spawned")):+ with self.assertRaises(P.JobCancelled):+ P.run_tracked_subprocess(["sleep", "30"], pgid_key="h3_pgid",+ label="x", job=job)++ def test_stop_between_spawn_and_registration_kills_the_group(self):+ # The race the review named: the process exists, Stop has already+ # read the (empty) pgid slot, and only then is the pgid published.+ job = {"id": "t-race"}+ spawned = []+ real_popen = subprocess.Popen++ def popen_then_stop(*a, **kw):+ proc = real_popen(*a, **kw)+ spawned.append(proc)+ job["cancel_requested"] = True # Stop, right here+ return proc++ with _CurrentJob(job), unittest.mock.patch.object(+ P.subprocess, "Popen", side_effect=popen_then_stop):+ t0 = time.time()+ with self.assertRaises(P.JobCancelled):+ P.run_tracked_subprocess(["sleep", "30"], pgid_key="mux_pgid",+ label="x", job=job)+ self.assertLess(time.time() - t0, 5)+ self.assertIsNotNone(spawned[0].poll())+ self.assertFalse(_alive(spawned[0].pid))+ self.assertIsNone(P.STATE.get("mux_pgid"))++ def test_stop_current_job_ends_a_running_tracked_process(self):+ job = {"id": "t-live"}+ result = {}++ def run():+ try:+ P.run_tracked_subprocess(["sleep", "30"], pgid_key="mux_pgid",+ label="the fetch", job=job)+ result["r"] = "finished"+ except P.JobCancelled as e:+ result["r"] = e++ with _CurrentJob(job):+ th = threading.Thread(target=run)+ t0 = time.time()+ th.start()+ deadline = time.time() + 5+ while P.STATE.get("mux_pgid") is None and time.time() < deadline:+ time.sleep(0.02)+ pgid = P.STATE.get("mux_pgid")+ self.assertIsNotNone(pgid)+ P.stop_current_job()+ th.join(timeout=10)+ self.assertFalse(th.is_alive())+ self.assertIsInstance(result["r"], P.JobCancelled)+ self.assertLess(time.time() - t0, 8)+ self.assertFalse(_alive(pgid))++ def test_a_finished_process_is_returned_like_subprocess_run(self):+ r = P.run_tracked_subprocess([sys.executable, "-c",+ "import sys; print('out'); print('err', file=sys.stderr); sys.exit(3)"],+ pgid_key="mux_pgid", label="x", job={"id": "t-ok"})+ self.assertEqual((r.returncode, r.stdout.strip(), r.stderr.strip()), (3, "out", "err"))++ def test_timeout_kills_the_group_and_raises(self):+ t0 = time.time()+ with self.assertRaises(subprocess.TimeoutExpired):+ P.run_tracked_subprocess(["sleep", "30"], pgid_key="mux_pgid",+ label="x", job={"id": "t-to"}, timeout=0.5)+ self.assertLess(time.time() - t0, 5)++ def test_stop_sets_the_flag_before_it_reads_the_pgids(self):+ src = (ROOT / "mlx_ltx_panel.py").read_text(encoding="utf-8")+ body = src[src.index("def stop_current_job("):]+ body = body[:body.index('push("Stop requested')]+ self.assertLess(body.index('cur["cancel_requested"] = True'),+ body.index('h3_pgid = STATE.get("h3_pgid")'))+ self.assertIn("with LOCK:", body[:body.index('cur["cancel_requested"] = True')])++ def test_work_outside_the_queue_ignores_a_stop_meant_for_the_render(self):+ # Codex hotfix review #3: an editor proxy built on an HTTP thread+ # while a stopped render unwinds must still run.+ stopped = {"id": "t-render", "cancel_requested": True}+ with _CurrentJob(stopped):+ out, _err = P.run_postprocess_tracked(+ [sys.executable, "-c", "print('proxy built')"], "Timeline proxy")+ self.assertIn("proxy built", out)++ def test_the_worker_threads_own_job_is_the_owner(self):+ job = {"id": "t-owner", "cancel_requested": True}+ P._JOB_CTX.job = job+ try:+ with unittest.mock.patch.object(P.subprocess, "Popen",+ side_effect=AssertionError("spawned")):+ with self.assertRaises(P.JobCancelled):+ P.run_postprocess_tracked(["true"], "Mux")+ finally:+ P._JOB_CTX.job = None+ # and another thread's view is unaffected+ seen = []+ th = threading.Thread(target=lambda: seen.append(P._thread_job()))+ P._JOB_CTX.job = job+ try:+ th.start(); th.join()+ finally:+ P._JOB_CTX.job = None+ self.assertEqual(seen, [None])+ src = (ROOT / "mlx_ltx_panel.py").read_text(encoding="utf-8")+ loop = src[src.index("with _GPU_LOCK:\n _JOB_CTX.job = job"):]+ self.assertLess(loop.index("run_job_inner(job)"), loop.index("_JOB_CTX.job = None"))++ def test_every_pgid_is_published_through_the_race_free_helper(self):+ src = (ROOT / "mlx_ltx_panel.py").read_text(encoding="utf-8")+ for key in ("h3_pgid", "train_pgid", "mux_pgid"):+ self.assertNotIn(f'STATE["{key}"] = os.getpgid', src, key)+++def _h3_dispatch_patches(stack, fetch, popen):+ tmp = _TMP / "pack"+ paths = dict(missing=[], repairable=False, dit=tmp / "dit", python=sys.executable,+ runner=tmp / "generate_staged.py", compact_root=tmp / "compact",+ text_config=tmp / "text")+ turbo = dict(files_ok=True, adaln_pairs=51, embedder=None, lora=tmp / "turbo.safetensors",+ dir=tmp / "turbo", version="v4-600-EMA", fallback=False, missing=[])+ for name, val in {+ "h3_paths": lambda: paths, "h3_capable": lambda: True,+ "h3_supports_lora": lambda: True, "h3_turbo_paths": lambda: dict(turbo),+ "_h3_turbo_fetch_embedder": fetch, "h3_dit_choice": lambda: ("bf16", None),+ "output_codec_settings": lambda: {"crf": "18", "pix_fmt": "yuv420p"},+ "push": lambda msg: None, "_h3_runner_has_flag": lambda flag: False,+ "h3_supports_lora_adaln": lambda: True}.items():+ stack.enter_context(unittest.mock.patch.object(P, name, val))+ for name in ("h3_supports_memory_limits", "h3_supports_prompt_cache",+ "h3_live_preview_ready", "h3_supports_stage_a", "h3_supports_tae_draft"):+ stack.enter_context(unittest.mock.patch.object(P, name, lambda: False))+ stack.enter_context(unittest.mock.patch.object(P.HELPER, "is_alive", lambda: False))+ stack.enter_context(unittest.mock.patch.object(P.subprocess, "Popen", popen))+ ff = _TMP / "ffmpeg"+ ff.write_text("")+ stack.enter_context(unittest.mock.patch.object(P, "FFMPEG", ff))+++def _turbo_job(jid):+ return {"id": jid, "params": {"engine": "h3", "mode": "t2v", "h3_tier": "standard_5s",+ "prompt": "a lighthouse at dusk", "seed": "42",+ "h3_turbo": True, "loras": []}}+++class TheTurboDispatchObeysStop(unittest.TestCase):+ def test_stop_during_the_companion_fetch_never_reaches_the_renderer(self):+ job = _turbo_job("t-fetch-stop")+ fetches = []++ def fetch(target_dir, push_log, runner=None, **kw):+ fetches.append(runner)+ job["cancel_requested"] = True # Stop, mid-fetch+ return False++ with ExitStack() as st, _CurrentJob(job):+ _h3_dispatch_patches(st, fetch, unittest.mock.Mock(+ side_effect=AssertionError("renderer spawned after Stop")))+ with self.assertRaises(P.JobCancelled):+ P.run_h3_job_inner(job)+ self.assertEqual(len(fetches), 1)+ self.assertIsNotNone(fetches[0], "the fetch must get the tracked runner")++ def test_the_fetch_runs_as_a_tracked_group_stop_can_kill(self):+ job = _turbo_job("t-fetch-kill")+ seen = {}++ def fetch(target_dir, push_log, runner=None, **kw):+ def stop_soon():+ deadline = time.time() + 5+ while P.STATE.get("h3_pgid") is None and time.time() < deadline:+ time.sleep(0.02)+ seen["pgid"] = P.STATE.get("h3_pgid")+ P.stop_current_job()+ th = threading.Thread(target=stop_soon)+ th.start()+ try:+ runner(["sleep", "30"], capture_output=True, text=True, env=None,+ timeout=1800)+ finally:+ th.join()++ real_popen = subprocess.Popen+ spawned = []++ def popen(cmd, **kw):+ if cmd and cmd[0] == "caffeinate":+ raise AssertionError("renderer spawned after Stop")+ p = real_popen(cmd, **kw)+ spawned.append(p)+ return p++ with ExitStack() as st, _CurrentJob(job):+ _h3_dispatch_patches(st, fetch, popen)+ t0 = time.time()+ with self.assertRaises(P.JobCancelled):+ P.run_h3_job_inner(job)+ self.assertLess(time.time() - t0, 8)+ self.assertEqual(seen["pgid"], spawned[0].pid)+ self.assertFalse(_alive(spawned[0].pid))++ def test_stop_racing_the_renderer_spawn_kills_it(self):+ job = _turbo_job("t-spawn-race")+ real_popen = subprocess.Popen+ spawned = []++ def popen(cmd, **kw):+ self.assertEqual(cmd[0], "caffeinate")+ kw.pop("cwd", None)+ p = real_popen(["sleep", "30"], **{k: v for k, v in kw.items()+ if k in ("stdout", "stderr", "text",+ "bufsize", "start_new_session",+ "env")})+ spawned.append(p)+ job["cancel_requested"] = True # Stop lands mid-spawn+ return p++ with ExitStack() as st, _CurrentJob(job):+ _h3_dispatch_patches(st, lambda *a, **k: True, popen)+ st.enter_context(unittest.mock.patch.object(+ P, "h3_turbo_paths", lambda: dict(+ files_ok=True, adaln_pairs=51, embedder=_TMP / "emb",+ lora=_TMP / "turbo.safetensors", dir=_TMP / "turbo",+ version="v4-600-EMA", fallback=False, missing=[])))+ with self.assertRaises(P.JobCancelled):+ P.run_h3_job_inner(job)+ self.assertEqual(len(spawned), 1)+ spawned[0].wait(timeout=5)+ self.assertFalse(_alive(spawned[0].pid))+ self.assertIsNone(P.STATE.get("h3_pgid"))++ def test_the_x2_lane_checks_stop_before_the_helper(self):+ src = (ROOT / "mlx_ltx_panel.py").read_text(encoding="utf-8")+ i = src.index(' if mode == "upscale":')+ body = src[i:src.index(' if mode == "ingredients":', i)]+ self.assertIn('_raise_if_cancelled(job, "the ×2 render")', body)+ self.assertLess(body.index('_raise_if_cancelled(job, "the ×2 render")'),+ body.index("HELPER.run(job_spec)"))+ self.assertIn('pgid_key="mux_pgid", label="the ×2 preparation"', body)+ self.assertNotIn("subprocess.run(_upscale_hold_tail_cmd", body)++ def test_the_helper_refuses_a_job_stopped_before_it_arrived(self):+ job = {"id": "t-helper", "cancel_requested": True}+ writes = []++ class FakeStdin:+ def write(self, s):+ writes.append(s)++ def flush(self):+ pass++ class FakeProc:+ stdin = FakeStdin()++ def poll(self):+ return None++ h = P.WarmHelper()+ h.proc = FakeProc()+ with _CurrentJob(job), unittest.mock.patch.object(h, "_ensure", lambda: None):+ with self.assertRaises(P.JobCancelled):+ h._run_once({"action": "generate", "id": "t-helper"})+ self.assertEqual(writes, [])+ h.proc = None+++class AOneShotPartBelongsToItsTake(unittest.TestCase):+ """Codex hotfix review round 2: a take's parts are their own dicts+ (`<take>-p1`) and Stop marks only the queued take."""++ def setUp(self):+ self.parent = {"id": "take-x", "params": {}}+ P._JOB_CTX.job = self.parent+ P._h3_turbo_embedder_failures.clear()++ def tearDown(self):+ P._JOB_CTX.job = None+ P._h3_turbo_embedder_failures.clear()++ def test_the_owner_cancels_its_children(self):+ child = _turbo_job("take-x-p1")+ self.assertFalse(P._cancel_requested(child))+ self.parent["cancel_requested"] = True+ self.assertTrue(P._cancel_requested(child))+ self.assertFalse(child.get("cancel_requested"))+ with self.assertRaises(P.JobCancelled):+ P._raise_if_cancelled(child, "the part")++ def test_the_helper_refuses_a_part_of_a_stopped_take(self):+ self.parent["cancel_requested"] = True+ writes = []++ class FakeProc:+ class stdin:+ @staticmethod+ def write(s):+ writes.append(s)++ @staticmethod+ def flush():+ pass++ def poll(self):+ return None++ h = P.WarmHelper()+ h.proc = FakeProc()+ with _CurrentJob(self.parent), unittest.mock.patch.object(h, "_ensure", lambda: None):+ with self.assertRaises(P.JobCancelled):+ h._run_once({"action": "generate", "id": "take-x-p1"})+ self.assertEqual(writes, [])+ h.proc = None++ def test_stop_during_a_parts_companion_fetch_ends_the_take_and_is_no_failure(self):+ child = _turbo_job("take-x-p1")+ root = _TMP / "pack-take"+ (root / "scripts").mkdir(parents=True, exist_ok=True)+ fetcher = root / "scripts" / "fetch_time_embedder.py"+ fetcher.write_text("import time\ntime.sleep(30)\n")+ turbo_dir = _TMP / "turbo-take"+ real_fetch = P._h3_turbo_fetch_embedder+ spawned = []+ real_popen = subprocess.Popen++ def popen(cmd, **kw):+ if cmd and cmd[0] == "caffeinate":+ raise AssertionError("renderer spawned after Stop")+ proc = real_popen(cmd, **kw)+ spawned.append(proc)+ return proc++ def stop_soon():+ deadline = time.time() + 10+ while P.STATE.get("h3_pgid") is None and time.time() < deadline:+ time.sleep(0.02)+ P.stop_current_job()++ with ExitStack() as st, _CurrentJob(self.parent):+ _h3_dispatch_patches(st, real_fetch, popen)+ st.enter_context(unittest.mock.patch.object(P, "H3_ROOT", root))+ st.enter_context(unittest.mock.patch.object(P, "_h3_python", lambda: Path(sys.executable)))+ st.enter_context(unittest.mock.patch.object(+ P, "h3_turbo_paths", lambda: dict(+ files_ok=True, adaln_pairs=51, embedder=None, lora=turbo_dir / "l.safetensors",+ dir=turbo_dir, version="v4-600-EMA", fallback=False, missing=[])))+ th = threading.Thread(target=stop_soon)+ th.start()+ t0 = time.time()+ with self.assertRaises(P.JobCancelled):+ P.run_h3_job_inner(child)+ th.join()+ self.assertLess(time.time() - t0, 10)+ self.assertTrue(self.parent.get("cancel_requested"))+ self.assertEqual(len(spawned), 1)+ self.assertFalse(_alive(spawned[0].pid))+ self.assertEqual(P._h3_turbo_embedder_failures, {})+ self.assertFalse(list(turbo_dir.glob("*.partial")) if turbo_dir.exists() else [])+++class TheCompanionIsFetchedOnce(unittest.TestCase):+ def setUp(self):+ self.dir = Path(tempfile.mkdtemp(prefix="phos-emb-"))+ root = self.dir / "pack"+ (root / "scripts").mkdir(parents=True)+ (root / "scripts" / "fetch_time_embedder.py").write_text("# fixture")+ self.st = ExitStack()+ for name, val in {"H3_ROOT": root, "_h3_python": lambda: Path(sys.executable),+ "H3_TURBO_EMBEDDER_MIN_BYTES": 1,+ "h3_turbo_paths": lambda: {"adaln_pairs": 51}}.items():+ self.st.enter_context(unittest.mock.patch.object(P, name, val))+ P._h3_turbo_embedder_failures.clear()+ self.logs = []+ self.calls = []++ def tearDown(self):+ self.st.close()+ P._h3_turbo_embedder_failures.clear()++ def failing(self, cmd, **kw):+ self.calls.append(cmd[cmd.index("--out") + 1])+ Path(cmd[cmd.index("--out") + 1]).write_bytes(b"half")++ class R:+ returncode, stdout, stderr = 1, "", "network is unreachable"+ return R()++ def ok(self, cmd, **kw):+ self.calls.append(cmd[cmd.index("--out") + 1])+ Path(cmd[cmd.index("--out") + 1]).write_bytes(b"tensors")++ class R:+ returncode, stdout, stderr = 0, "", ""+ return R()++ def test_a_failure_is_not_retried_by_the_next_job(self):+ f = P._h3_turbo_fetch_embedder+ self.assertFalse(f(self.dir, self.logs.append, runner=self.failing))+ self.assertFalse(f(self.dir, self.logs.append, runner=self.failing))+ self.assertFalse(f(self.dir, self.logs.append, runner=self.failing))+ self.assertEqual(len(self.calls), 1)+ self.assertTrue(any("not fetching" in m and "Install Turbo" in m for m in self.logs))+ self.assertFalse(list(self.dir.glob("*.partial")))+ state = P._h3_turbo_embedder_retry_state(self.dir)+ self.assertIn("network is unreachable", state["error"])+ # The cooldown ends by itself...+ P._h3_turbo_embedder_failures[str(self.dir / P.H3_TURBO_EMBEDDER_FILE)]["retry_at"] = 0+ self.assertIsNone(P._h3_turbo_embedder_retry_state(self.dir))+ self.assertTrue(f(self.dir, self.logs.append, runner=self.ok))+ self.assertEqual(len(self.calls), 2)+ self.assertIsNone(P._h3_turbo_embedder_retry_state(self.dir))++ def test_the_install_button_retries_at_once(self):+ f = P._h3_turbo_fetch_embedder+ self.assertFalse(f(self.dir, self.logs.append, runner=self.failing))+ self.assertTrue(f(self.dir, self.logs.append, runner=self.ok, force=True))+ self.assertEqual(len(self.calls), 2)+ self.assertTrue((self.dir / P.H3_TURBO_EMBEDDER_FILE).is_file())+ src = (ROOT / "mlx_ltx_panel.py").read_text(encoding="utf-8")+ install = src[src.index("def _h3_install_turbo("):]+ install = install[:install.index("\ndef ")]+ self.assertIn("_h3_turbo_fetch_embedder(target, push_log, force=True)", install)+ bg = src[src.index("def _h3_turbo_download_bg("):src.index("def _h3_install_turbo(")]+ self.assertEqual(bg.count("_h3_turbo_fetch_embedder(target_dir, push_log, force=True)"), 2)++ def test_one_fetch_at_a_time_and_each_writes_its_own_file(self):+ f = P._h3_turbo_fetch_embedder+ with P._h3_turbo_embedder_lock:+ self.assertFalse(f(self.dir, self.logs.append, runner=self.ok, force=True))+ self.assertEqual(self.calls, [])+ self.assertTrue(any("already being fetched" in m for m in self.logs))+ self.assertIsNone(P._h3_turbo_embedder_retry_state(self.dir)) # not a failure+ self.assertFalse(f(self.dir, self.logs.append, runner=self.failing))+ self.assertTrue(f(self.dir, self.logs.append, runner=self.ok, force=True))+ self.assertEqual(len(set(self.calls)), 2)+ for c in self.calls:+ self.assertTrue(c.endswith(".partial"))+ self.assertNotEqual(Path(c).name, P.H3_TURBO_EMBEDDER_FILE + ".partial")+ self.assertFalse(list(self.dir.glob("*.partial")))++ def test_stop_during_the_fetch_is_not_a_failed_download(self):+ def stopped(cmd, **kw):+ raise P.JobCancelled("Stopped during the fetch.")+ with self.assertRaises(P.JobCancelled):+ P._h3_turbo_fetch_embedder(self.dir, self.logs.append, runner=stopped)+ self.assertIsNone(P._h3_turbo_embedder_retry_state(self.dir))+ self.assertFalse(P._h3_turbo_embedder_lock.locked())++ def test_status_names_a_remembered_failure(self):+ self.assertIsNone(P._h3_turbo_embedder_retry_state(self.dir))+ P._h3_turbo_fetch_embedder(self.dir, self.logs.append, runner=self.failing)+ with unittest.mock.patch.object(P, "h3_turbo_paths", lambda: {+ "adaln_pairs": 51, "dir": self.dir, "files_ok": True, "lora": None,+ "version": "v4-600-EMA", "fallback": False, "missing": [],+ "embedder": None}), \+ unittest.mock.patch.object(P, "h3_supports_lora", lambda: True), \+ unittest.mock.patch.object(P, "_h3_retune_turbo_estimates", lambda: None), \+ unittest.mock.patch.object(P, "h3_turbo_note", lambda paths: ""):+ status = P.h3_turbo_status()+ self.assertIn("network is unreachable", status["adaln_fetch"]["error"])+++if __name__ == "__main__":+ unittest.main()diff --git a/test_upscale_x2_tail.py b/test_upscale_x2_tail.pyindex 4632678..c02b519 100644--- a/test_upscale_x2_tail.py+++ b/test_upscale_x2_tail.py@@ -48,7 +48,8 @@ except Exception: # noqa: BLE001def _helper_ns() -> dict:tree = ast.parse(HELPER_SRC)names = {"_vae_decode_peak_elements", "_vae_decode_max_latent_frames",- "_int32_safe_decode_tiling"}+ "_int32_safe_decode_tiling", "_decode_tiling_peak_elements",+ "_decode_temporal_config"}picked = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name in names]assert {n.name for n in picked} == namesconsts = [n for n in tree.body if isinstance(n, ast.Assign)@@ -183,6 +184,207 @@ class TheGuardIsWired(unittest.TestCase):self.assertIn("_compute_decode_tiling(", src)+def _pure_tiling():+ """The vendored tiler's config classes and split functions, exec'd from+ source without its MLX import — so the tile-extent checks below run on any+ machine, against the exact interval code the decoder uses."""+ import dataclasses+ import types+ src_dir = None+ for base in sys.path + [str(ROOT / "ltx-2-mlx" / "env" / "lib" / "python3.11" / "site-packages")]:+ cand = Path(base) / "ltx_core_mlx" / "model" / "video_vae"+ if (cand / "tiling.py").is_file():+ src_dir = cand+ break+ if src_dir is None:+ return None, None+ names = {"SpatialTilingConfig", "TemporalTilingConfig", "TilingConfig",+ "DimensionIntervals", "default_split_operation",+ "split_with_symmetric_overlaps", "split_temporal_latents"}+ tree = ast.parse((src_dir / "tiling.py").read_text(encoding="utf-8"))+ body = [ast.ImportFrom(module="__future__", names=[ast.alias(name="annotations")], level=0)]+ body += [n for n in tree.body if isinstance(n, (ast.ClassDef, ast.FunctionDef))+ and n.name in names]+ ns = {"dataclass": dataclasses.dataclass, "replace": dataclasses.replace,+ "field": dataclasses.field}+ exec(compile(ast.fix_missing_locations(ast.Module(body=body, type_ignores=[])),+ "tiling", "exec"), ns)+ ns["DEFAULT_SPLIT_OPERATION"] = ns["default_split_operation"]+ orig = [n for n in ast.parse((src_dir / "video_vae.py").read_text(encoding="utf-8")).body+ if isinstance(n, ast.FunctionDef) and n.name == "_compute_decode_tiling"]+ ons = {"os": os, "TilingConfig": ns["TilingConfig"],+ "TemporalTilingConfig": ns["TemporalTilingConfig"]}+ exec(compile(ast.Module(body=orig, type_ignores=[]), "orig", "exec"), ons)+ return types.SimpleNamespace(**ns), ons["_compute_decode_tiling"]+++# (width, height, latent frames): the review's matrix plus the canvases the+# 4.13.1 temporal-only guard got wrong (2 latent frames + the causal third).+GUARD_CASES = [(2048, 1152, 17), (2688, 1536, 17), (4096, 2304, 17),+ (4096, 4096, 4), (4096, 4096, 17), (4096, 4096, 31),+ (4480, 4480, 17), (6144, 3456, 17), (1280, 768, 16),+ (2048, 1152, 16), (4096, 4096, 1), (4096, 4096, 2), (4096, 4096, 3)]+++class TheGuardChecksTheTilesItReturns(unittest.TestCase):+ """Ship review 2026-09-17, P2: `size = max(2, max_lat - 1)` returned+ [2, 3]-latent tiles for a 4096×4096 decode — 2.56e9 elements, past the cap+ the guard exists to keep."""++ def setUp(self):+ self.T, self.orig = _pure_tiling()+ if self.T is None:+ self.skipTest("vendored ltx_core_mlx tiling.py not found")+ os.environ.pop("LTX2_VAE_DECODE_BUDGET_GB", None)+ self.ns = _helper_ns()++ def guard(self, shape):+ return self.ns["_int32_safe_decode_tiling"](self.orig, shape, 24.0, T=self.T)++ def extents(self, cfg, shape):+ """Tile lengths per axis, straight from the tiler's split functions,+ the way prepare_tiles_for_decoding derives them."""+ T = self.T+ _, _, f, h, w = shape+ out = {"t": [(0, f)], "h": [(0, h)], "w": [(0, w)]}+ sc = cfg.spatial_config+ if sc is not None:+ long_side, size, ov = max(h, w), sc.tile_size_in_pixels // 32, sc.tile_overlap_in_pixels // 32+ for axis, n in (("h", h), ("w", w)):+ iv = T.split_with_symmetric_overlaps(+ max(max(2, ov + 1), round(size * n / long_side)), ov)(n)+ out[axis] = list(zip(iv.starts, iv.ends))+ tc = cfg.temporal_config+ iv = T.split_temporal_latents(tc.tile_size_in_frames // 8, tc.tile_overlap_in_frames // 8)(f)+ out["t"] = list(zip(iv.starts, iv.ends))+ return out++ def test_the_4_13_1_answer_overflowed_on_a_square_4k_decode(self):+ shape = (1, 128, 4, 128, 128)+ max_lat = self.ns["_vae_decode_max_latent_frames"](128, 128)+ self.assertEqual(max_lat, 2)+ old = self.T.TilingConfig(temporal_config=self.T.TemporalTilingConfig(16, 0))+ lengths = [e - s for s, e in self.extents(old, shape)["t"]]+ self.assertEqual(lengths, [2, 3])+ self.assertGreater(self.ns["_vae_decode_peak_elements"](3, 128, 128), INT32)+ self.assertGreater(self.ns["_decode_tiling_peak_elements"](old, shape, self.T), INT32)++ def test_every_returned_tiling_is_under_the_cap_on_every_tile(self):+ cap = self.ns["_DECODE_MAX_ELEMENTS"]+ peak = self.ns["_vae_decode_peak_elements"]+ for w, h, f in GUARD_CASES:+ shape = (1, 128, f, h // 32, w // 32)+ cfg = self.guard(shape)+ if cfg is None:+ self.assertLessEqual(peak(f, h // 32, w // 32), cap, (w, h, f))+ continue+ ext = self.extents(cfg, shape)+ for ts, te in ext["t"]:+ out_frames = 1 + (te - 1) * 8 - ts * 8+ self.assertLessEqual(3 * out_frames * h * w, cap, (w, h, f)) # the blend buffer+ for hs, he in ext["h"]:+ for ws, we in ext["w"]:+ self.assertLessEqual(peak(te - ts, he - hs, we - ws), cap,+ (w, h, f, (ts, te), (hs, he), (ws, we)))+ # The whole clip and the whole canvas are covered.+ self.assertEqual(ext["t"][0][0], 0)+ self.assertEqual(ext["t"][-1][1], f)+ for axis, n in (("h", h // 32), ("w", w // 32)):+ covered = set()+ for a, b in ext[axis]:+ covered.update(range(a, b))+ self.assertEqual(covered, set(range(n)), (w, h, f, axis))++ def test_the_canvases_4_13_1_got_right_keep_their_tiles(self):+ # Same answers as the shipped guard where its tiles were blended.+ for (w, h, f), frames in [((2048, 1152, 17), 96), ((2688, 1536, 17), 48)]:+ cfg = self.guard((1, 128, f, h // 32, w // 32))+ self.assertIsNone(cfg.spatial_config)+ self.assertEqual(cfg.temporal_config.tile_size_in_frames, frames)++ def test_unblended_temporal_tiles_give_way_to_blended_spatial_ones(self):+ # 4096×2304: 4.13.1 cut 16-frame temporal tiles with no blend at all.+ cfg = self.guard((1, 128, 17, 72, 128))+ self.assertIsNotNone(cfg.spatial_config)+ self.assertGreaterEqual(cfg.temporal_config.tile_size_in_frames, 32)+ self.assertGreater(cfg.temporal_config.tile_overlap_in_frames, 0)+ self.assertGreater(cfg.spatial_config.tile_overlap_in_pixels, 0)++ def test_only_a_canvas_temporal_tiles_cannot_fit_is_tiled_in_space(self):+ cfg = self.guard((1, 128, 17, 128, 128))+ self.assertIsNotNone(cfg.spatial_config)+ self.assertLess(cfg.spatial_config.tile_size_in_pixels, 4096)+ self.assertGreaterEqual(cfg.temporal_config.tile_size_in_frames, 32) # blended seams++ def test_a_canvas_no_tiling_can_fit_is_refused(self):+ with self.assertRaises(RuntimeError) as ctx:+ self.guard((1, 128, 17, 256, 256)) # 8192×8192+ self.assertIn("too large", str(ctx.exception))+ # The panel refuses the same canvas BEFORE the render, and admits the+ # ones the helper can tile.+ self.assertFalse(panel.upscale_canvas_decodable(8192, 8192, 129))+ for w, h, f in GUARD_CASES:+ self.assertTrue(panel.upscale_canvas_decodable(w, h, 8 * f - 7), (w, h, f))+ for w, h in [(6144, 6144), (8192, 4608), (7680, 4320)]:+ refused = not panel.upscale_canvas_decodable(w, h, 129)+ try:+ self.guard((1, 128, 17, h // 32, w // 32))+ helper_refused = False+ except RuntimeError:+ helper_refused = True+ if refused:+ self.assertTrue(helper_refused, (w, h)) # never admit what the helper refuses+ if not refused:+ self.assertFalse(helper_refused, (w, h))++ def test_admission_and_the_helper_agree_on_every_length(self):+ # Codex hotfix review #4: a 17-frame clip's shortest groups are 9+ # frames, not 17 — 8192×8192 × 17 tiles fine and must be admitted.+ self.assertEqual(panel._decode_min_group_frames(17), 9)+ self.assertEqual(panel._decode_min_group_frames(129), 17)+ self.assertEqual(panel._decode_min_group_frames(9), 9)+ self.assertEqual(panel._decode_min_group_frames(1), 1)+ self.assertTrue(panel.upscale_canvas_decodable(8192, 8192, 17))+ for w, h in [(8192, 8192), (6144, 6144), (8192, 4608), (9600, 9600),+ (12800, 7200), (16384, 16384)]:+ for frames in (1, 9, 17, 25, 33, 41, 121, 129, 241):+ f = (frames - 1) // 8 + 1+ try:+ self.guard((1, 128, f, h // 32, w // 32))+ helper_ok = True+ except RuntimeError:+ helper_ok = False+ self.assertEqual(panel.upscale_canvas_decodable(w, h, frames), helper_ok,+ (w, h, frames))++ @unittest.skipUnless(HAVE_LTX, "ltx_core_mlx not importable (run with the repo venv)")+ def test_the_real_tiler_cuts_what_the_check_measured(self):+ cap = self.ns["_DECODE_MAX_ELEMENTS"]+ peak = self.ns["_vae_decode_peak_elements"]+ from ltx_core_mlx.model.video_vae import tiling as RT+ for w, h, f in GUARD_CASES:+ shape = (1, 128, f, h // 32, w // 32)+ cfg = self.ns["_int32_safe_decode_tiling"](_vv._compute_decode_tiling, shape, 24.0)+ if cfg is None:+ continue+ real = RT.TilingConfig(+ spatial_config=(RT.SpatialTilingConfig(cfg.spatial_config.tile_size_in_pixels,+ cfg.spatial_config.tile_overlap_in_pixels)+ if cfg.spatial_config else None),+ temporal_config=RT.TemporalTilingConfig(cfg.temporal_config.tile_size_in_frames,+ cfg.temporal_config.tile_overlap_in_frames))+ worst = 0+ for t in prepare_tiles_for_decoding(shape, real):+ n = [t.in_coords[i].stop - t.in_coords[i].start for i in (2, 3, 4)]+ worst = max(worst, peak(*n))+ o = t.out_coords[2]+ self.assertIsInstance(o.stop, int) # tiled_decode needs a bounded slice+ self.assertLessEqual(3 * (o.stop - o.start) * h * w, cap)+ self.assertLessEqual(worst, cap, (w, h, f))+ # The pure check never under-counts what the real tiler cuts.+ self.assertLessEqual(worst, self.ns["_decode_tiling_peak_elements"](real, shape, RT))++import mlx_ltx_panel as panel # noqa: E402@@ -322,5 +524,133 @@ class TheClipGoesInAndComesOutTheSameLength(unittest.TestCase):+def _tiny(path, frames, tone):+ cmd = [str(panel.FFMPEG), "-y", "-v", "error", "-f", "lavfi", "-i", "testsrc2=size=32x32:rate=24"]+ if tone:+ cmd += ["-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000",+ "-map", "0:v", "-map", "1:a", "-af", f"atrim=end={frames / 24 + 0.3:.4f}",+ "-c:a", "aac"]+ cmd += ["-frames:v", str(frames), "-c:v", "libx264", "-threads", "1",+ "-pix_fmt", "yuv420p", str(path)]+ subprocess.run(cmd, check=True, capture_output=True)+ return path++[email protected](Path(str(panel.FFMPEG)).is_file(), "ffmpeg not installed")+class TheSoundIsTheSourcesOrNone(unittest.TestCase):+ """Ship review 2026-09-17, P2: the ×2 model writes a soundtrack of its own.+ A SILENT source kept it when no trim was needed (9/17/73/121/129 frames)+ and lost it when one was (124) — the result's sound depended on the+ frame count. And a failed probe read as "silent"."""++ def setUp(self):+ self.tmp = Path(tempfile.mkdtemp(prefix="x2sound_"))+ self.codec = unittest.mock.patch.object(+ panel, "output_codec_settings",+ return_value={"pix_fmt": "yuv420p", "crf": "18", "preset": "medium"})+ self.codec.start()++ def tearDown(self):+ import shutil+ self.codec.stop()+ shutil.rmtree(self.tmp, ignore_errors=True)++ def test_a_silent_source_comes_back_silent_at_every_length(self):+ for n in (1, 8, 9, 10, 17, 72, 73, 121, 124, 129, 130, 243):+ src = _tiny(self.tmp / f"src{n}.mp4", n, tone=False)+ deliver, grid = panel.upscale_frame_plan(n, 121)+ model = _tiny(self.tmp / f"model{n}.mp4", grid, tone=True) # the model's own sound+ self.assertTrue(panel._video_has_audio(str(model)))+ kept = panel._upscale_finish(model, str(src), frames=deliver, fps=24.0,+ trim=grid != deliver)+ self.assertFalse(kept, n)+ self.assertFalse(panel._video_has_audio(str(model)), n)+ self.assertEqual(panel._probe_video_frames(str(model)), deliver, n)+ self.assertFalse(list(self.tmp.glob("*.mux.mp4")))++ def test_a_source_with_sound_keeps_it_at_every_length(self):+ for n in (9, 121, 124, 129):+ src = _tiny(self.tmp / f"s{n}.mp4", n, tone=True)+ deliver, grid = panel.upscale_frame_plan(n, 121)+ model = _tiny(self.tmp / f"m{n}.mp4", grid, tone=False)+ self.assertTrue(panel._upscale_finish(model, str(src), frames=deliver, fps=24.0,+ trim=grid != deliver), n)+ self.assertTrue(panel._video_has_audio(str(model)), n)+ self.assertEqual(panel._probe_video_frames(str(model)), deliver, n)++ def test_a_failed_probe_does_not_throw_the_sound_away(self):+ src = _tiny(self.tmp / "s.mp4", 121, tone=True)+ model = _tiny(self.tmp / "m.mp4", 121, tone=True)+ real = panel._probe_audio_state+ with unittest.mock.patch.object(+ panel, "_probe_audio_state",+ side_effect=lambda p: "unknown" if p == str(src) else real(p)):+ self.assertTrue(panel._upscale_finish(model, str(src), frames=121, fps=24.0,+ trim=False))+ self.assertTrue(panel._video_has_audio(str(model)))+ # ...and a silent source behind a failed probe still ends silent.+ silent = _tiny(self.tmp / "q.mp4", 121, tone=False)+ model2 = _tiny(self.tmp / "m2.mp4", 121, tone=True)+ with unittest.mock.patch.object(+ panel, "_probe_audio_state",+ side_effect=lambda p: "unknown" if p == str(silent) else real(p)):+ self.assertFalse(panel._upscale_finish(model2, str(silent), frames=121,+ fps=24.0, trim=False))+ self.assertFalse(panel._video_has_audio(str(model2)))++ def test_the_probe_tells_none_from_unknown(self):+ self.assertEqual(panel._probe_audio_state(str(_tiny(self.tmp / "a.mp4", 9, True))), "audio")+ self.assertEqual(panel._probe_audio_state(str(_tiny(self.tmp / "b.mp4", 9, False))), "none")+ self.assertEqual(panel._probe_audio_state(str(self.tmp / "missing.mp4")), "unknown")+ junk = self.tmp / "junk.mp4"+ junk.write_bytes(b"not a video")+ self.assertEqual(panel._probe_audio_state(str(junk)), "unknown")++ def test_an_unmuxable_source_still_drops_the_models_sound(self):+ src = _tiny(self.tmp / "s.mp4", 121, tone=True)+ model = _tiny(self.tmp / "m.mp4", 121, tone=True)+ broken = self.tmp / "broken.mp4"+ broken.write_bytes(b"x")+ with unittest.mock.patch.object(panel, "_probe_audio_state",+ side_effect=lambda p: "audio" if p == str(broken)+ else "none"):+ self.assertFalse(panel._upscale_finish(model, str(broken), frames=121,+ fps=24.0, trim=False))+ self.assertFalse(panel._video_has_audio(str(model)))+ self.assertEqual(panel._probe_video_frames(str(model)), 121)++ def test_a_stopped_finish_leaves_no_temp_file_for_the_gallery(self):+ # Codex hotfix review #2: ffmpeg had opened <out>.mux.mp4 when Stop hit.+ src = _tiny(self.tmp / "s.mp4", 9, tone=False)+ model = _tiny(self.tmp / "m.mp4", 9, tone=True)++ def stopped_mid_write(cmd, **kw):+ Path(cmd[-1]).write_bytes(b"half an mp4")+ raise panel.JobCancelled("Stopped during the ×2 finish.")++ with unittest.mock.patch.object(panel, "run_tracked_subprocess", stopped_mid_write):+ with self.assertRaises(panel.JobCancelled):+ panel._upscale_finish(model, str(src), frames=9, fps=24.0, trim=False)+ self.assertEqual(sorted(p.name for p in self.tmp.iterdir()), ["m.mp4", "s.mp4"])++ def boom(cmd, **kw):+ Path(cmd[-1]).write_bytes(b"x")+ raise OSError("spawn failed")++ with unittest.mock.patch.object(panel, "run_tracked_subprocess", boom):+ with self.assertRaises(OSError):+ panel._upscale_finish(model, str(src), frames=9, fps=24.0, trim=False)+ self.assertFalse(list(self.tmp.glob("*.mux.mp4")))++ def test_stop_ends_the_finish(self):+ src = _tiny(self.tmp / "s.mp4", 9, tone=False)+ model = _tiny(self.tmp / "m.mp4", 9, tone=True)+ before = model.read_bytes()+ with self.assertRaises(panel.JobCancelled):+ panel._upscale_finish(model, str(src), frames=9, fps=24.0, trim=False,+ job={"id": "x", "cancel_requested": True})+ self.assertEqual(model.read_bytes(), before)++if __name__ == "__main__":unittest.main()