Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

release(v4.9.5): character training actually re-trains (#62 cache), a finished H3 clip survives a shutdown abort (#76)

mrbizarro0984b65c7804
diff --git a/README.md b/README.md
index e2eda1c..d6689ba 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@
</p>
-> **Current release: v4.9.4.** Four small fixes from the first day of fleet data on 4.9.3. A queue that fails three times in a row on the same problem now pauses itself and says what to fix, instead of burning through every waiting job. H3 failures name the actual reason ("last line: …") instead of "see the log above". Step counts that exceed what the model's schedule holds are capped instead of crashing the render after the models load. A stale HiDream setting in Image Studio falls back to an installed engine. Full notes on the [releases page](https://github.com/mrbizarro/phosphene/releases).
+> **Current release: v4.9.5.** Two fixes. Character training re-trains for real: the preprocessing cache used to be keyed by position, so re-training from the same folder with a new trigger word silently reused the old encoded captions (and dropping a photo paired later photos with the wrong captions) — the likely cause behind "training completes but the character never shows up" (#62). And a Hailuo H3 render that finishes its clip and then crashes during shutdown is kept as a finished render instead of a failure (#76, engine fix included on Update). Full notes on the [releases page](https://github.com/mrbizarro/phosphene/releases).
## Overview
diff --git a/VERSION b/VERSION
index fa42341..2391fa0 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-4.9.4
+4.9.5
diff --git a/docs/STATE.md b/docs/STATE.md
index 416c600..e29ecfa 100644
--- a/docs/STATE.md
+++ b/docs/STATE.md
@@ -1,5 +1,32 @@
# Phosphene — project state, history, open work
+> **🩹 2026-09-05 — v4.9.5: character training actually re-trains (#62 cache), H3 shutdown abort (#76).**
+> Both validated end-to-end before promote: real Gemma preprocess twice on
+> a 3-image set — one changed caption re-encoded exactly that one file,
+> unchanged inputs reused; stub H3 engine that writes the clip then aborts
+> → job done + log line, stub that aborts before writing → still failed.
+> Full gates green. Owner rule from today: bug fixes ship same day, validated
+> (memory feedback_ship_bugfixes_validated).
+
+> **🩹 2026-09-05 — after v4.9.4: two more on dev/beta (fe31e41), UNRELEASED.**
+> **#76 (PhantombrainM):** H3 helper aborted at interpreter shutdown AFTER
+> the MP4 was written (mlx#4248 stream teardown → PyThreadState_Get /
+> SIGABRT); panel called finished renders failures. Engine fix pushed to
+> minimax-h3-mlx `codex/h3-engine-v2` 79c252b (+ cherry-picked to the
+> owner's `codex/live-preview` 7cdcf99): guarded `atexit.register(
+> mx.clear_streams)`. Panel guard `_h3_clip_is_complete` keeps a whole clip
+> (mtime/size/ffprobe) when the helper exits non-zero. **#62 root cause
+> candidate FOUND on our side:** the vendored preprocess skips cached
+> conditions/latents by INDEX FILENAME only — a re-train from the same
+> folder with a new trigger trained on the OLD captions, a dropped photo
+> misaligned every later latent. `lora_lab/preprocess_images.py` now
+> writes `.precomputed/manifest.json` and invalidates exactly what changed
+> (`_reconcile_precomputed`, 5 tests). Asked PiotrAstroCamp for one
+> fresh-folder run to confirm (his sample-character test proved his
+> render path is fine). Fleet 12 h after 4.9.4: 35 installs on it, no new
+> failure class; the "[Errno 2]" venv_broken signature is pre-existing.
+> Pinokio post for 4.9.4 still NOT made (Chrome logged out of pinokio.co).
+
> **🩹 2026-09-04 — v4.9.4: the four follow-ups from 4.9.3's first day, shipped.**
> Queue circuit breaker (3 identical failures → pause + why), H3 failures
> carry their last engine line, stage step counts capped at the checkpoint's
diff --git a/lora_lab/preprocess_images.py b/lora_lab/preprocess_images.py
index 327332a..258cea7 100644
--- a/lora_lab/preprocess_images.py
+++ b/lora_lab/preprocess_images.py
@@ -96,6 +96,73 @@ def _apply_gemma_max_length_override() -> int | None:
return n
+def _reconcile_precomputed(precomputed: Path, image_files: list, captions: list,
+ canvas: tuple) -> str:
+ """Drop cached tensors that no longer match the inputs, and say what was done.
+
+ The vendored encoders skip work by INDEX FILENAME — "condition_0007 exists,
+ skipping" — with no idea what produced it. Re-training from the same folder
+ with a new trigger therefore trained on the OLD captions; dropping a photo
+ shifted every later index so latents paired with the wrong captions; a new
+ canvas kept the old latents. Issue #62: three triggers, one folder, one
+ result. A manifest of (image name/size/mtime, caption hash, canvas, encode
+ length) is written BEFORE encoding, so a crash mid-way resumes correctly
+ and any change to the inputs invalidates exactly what it must.
+ """
+ import hashlib
+ import json
+ import shutil
+
+ def _h(text: str) -> str:
+ return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
+
+ cur = {
+ "images": [[f.name, f.stat().st_size, int(f.stat().st_mtime)] for f in image_files],
+ "captions": [_h(c) for c in captions],
+ "canvas": list(canvas),
+ "gemma_max_length": os.environ.get("LTX2_GEMMA_MAX_LENGTH", ""),
+ }
+ latents_dir = precomputed / "latents"
+ conditions_dir = precomputed / "conditions"
+ manifest = precomputed / "manifest.json"
+ try:
+ prev = json.loads(manifest.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ prev = None
+
+ def _has_files(d: Path) -> bool:
+ return d.is_dir() and any(d.iterdir())
+
+ if prev is None:
+ if _has_files(latents_dir) or _has_files(conditions_dir):
+ shutil.rmtree(latents_dir, ignore_errors=True)
+ shutil.rmtree(conditions_dir, ignore_errors=True)
+ action = "cache predates manifests, cannot prove it matches these inputs - re-encoding everything"
+ else:
+ action = "fresh"
+ elif prev.get("images") != cur["images"] or prev.get("canvas") != cur["canvas"]:
+ shutil.rmtree(latents_dir, ignore_errors=True)
+ shutil.rmtree(conditions_dir, ignore_errors=True)
+ action = "images or canvas changed - re-encoding everything"
+ elif (prev.get("gemma_max_length") != cur["gemma_max_length"]
+ or len(prev.get("captions") or []) != len(cur["captions"])):
+ shutil.rmtree(conditions_dir, ignore_errors=True)
+ action = "caption encode settings changed - re-encoding captions"
+ elif prev.get("captions") != cur["captions"]:
+ n = 0
+ for i, (a, b) in enumerate(zip(prev["captions"], cur["captions"])):
+ if a != b:
+ (conditions_dir / f"condition_{i:04d}.safetensors").unlink(missing_ok=True)
+ n += 1
+ action = f"{n} caption(s) changed - re-encoding those"
+ else:
+ action = "inputs unchanged - reusing cached tensors"
+
+ precomputed.mkdir(parents=True, exist_ok=True)
+ manifest.write_text(json.dumps(cur), encoding="utf-8")
+ return action
+
+
def _resolve_captions(image_files: list[Path], captions_dir: Path | None, caption_ext: str) -> list[str]:
captions: list[str] = []
if captions_dir is None:
@@ -267,6 +334,12 @@ def preprocess_images(
caption_ext,
)
+ print("cache: " + _reconcile_precomputed(
+ precomputed, image_files, captions,
+ (int(target_width), int(target_height), str(crop_anchor))))
+ latents_dir.mkdir(parents=True, exist_ok=True)
+ conditions_dir.mkdir(parents=True, exist_ok=True)
+
print("Phase 1: encoding text captions (Gemma)...")
_encode_all_captions(
captions=captions,
diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.py
index 1b60cbe..64efc88 100644
--- a/mlx_ltx_panel.py
+++ b/mlx_ltx_panel.py
@@ -20946,6 +20946,29 @@ def _h3_fit_first_frame(src: Path, width: int, height: int, job_id: str) -> Path
return src
+def _h3_clip_is_complete(path: Path, started_at: float) -> bool:
+ """True when the H3 output at `path` is a whole, playable clip written by
+ THIS run: exists, newer than the process start, non-trivial size, and
+ ffprobe (when present) reads a duration from it."""
+ try:
+ st = path.stat()
+ except OSError:
+ return False
+ if st.st_mtime < started_at - 1 or st.st_size < 200_000:
+ return False
+ ffprobe = shutil.which("ffprobe")
+ if not ffprobe:
+ return True
+ try:
+ out = subprocess.run(
+ [ffprobe, "-v", "error", "-show_entries", "format=duration",
+ "-of", "default=nw=1:nk=1", str(path)],
+ capture_output=True, text=True, timeout=20)
+ return out.returncode == 0 and float((out.stdout or "0").strip() or 0) > 0
+ except Exception: # noqa: BLE001
+ return False
+
+
def run_h3_job_inner(job: dict) -> None:
"""Render one job on the Hailuo H3 engine (optional subprocess pack).
@@ -21590,6 +21613,17 @@ def run_h3_job_inner(job: dict) -> None:
# LTX. It is a viewer decision, not a crash; worker_loop maps this
# exception to the neutral `stopped` history state.
raise JobStopped("H3 render stopped early at the next forward boundary")
+ if rc != 0 and _h3_clip_is_complete(out_path, t0):
+ # #76 (@PhantombrainM): the engine finished the clip, then aborted
+ # during interpreter shutdown (mlx stream teardown after the
+ # runtime was gone → PyThreadState_Get / SIGABRT). The file is
+ # whole and plays; calling that a failure threw away a finished
+ # render. The engine fix is upstream; this keeps older engine
+ # checkouts honest too.
+ push(f"[h3] the engine exited with {rc} AFTER the clip was fully "
+ f"written (a shutdown-time abort, not a render fault) — "
+ f"keeping the file.")
+ rc = 0
if rc != 0:
# Negative rc means the child died on a signal, and NAMING it is
# the whole diagnosis: a jetsam kill during joint denoise (issue
diff --git a/test_train_precompute_manifest.py b/test_train_precompute_manifest.py
new file mode 100644
index 0000000..9b4c9b9
--- /dev/null
+++ b/test_train_precompute_manifest.py
@@ -0,0 +1,81 @@
+"""Issue #62: the trainer's preprocessed cache was keyed by index only, so a
+re-train from the same folder with a new trigger trained on the OLD captions
+(and a dropped photo shifted every latent under the wrong caption). The
+manifest reconcile invalidates exactly what changed."""
+import json
+import os
+import tempfile
+import unittest
+from pathlib import Path
+
+import lora_lab.preprocess_images as PI
+
+
+def _mk(tmp, names, captions, canvas=(768, 768, "center")):
+ imgs = []
+ for n in names:
+ f = Path(tmp) / "images" / n
+ f.parent.mkdir(parents=True, exist_ok=True)
+ f.write_bytes(b"x" * 10)
+ imgs.append(f)
+ pre = Path(tmp) / "out" / ".precomputed"
+ return pre, imgs, list(captions), canvas
+
+
+def _seed_cache(pre, n):
+ for i in range(n):
+ (pre / "latents").mkdir(parents=True, exist_ok=True)
+ (pre / "conditions").mkdir(parents=True, exist_ok=True)
+ (pre / "latents" / f"latent_{i:04d}.safetensors").write_bytes(b"l")
+ (pre / "conditions" / f"condition_{i:04d}.safetensors").write_bytes(b"c")
+
+
+class PrecomputeManifest(unittest.TestCase):
+ def test_fresh_then_unchanged_reuses(self):
+ with tempfile.TemporaryDirectory() as t:
+ pre, imgs, caps, cv = _mk(t, ["a.png", "b.png"], ["trn a", "trn b"])
+ self.assertEqual(PI._reconcile_precomputed(pre, imgs, caps, cv), "fresh")
+ _seed_cache(pre, 2)
+ self.assertIn("unchanged", PI._reconcile_precomputed(pre, imgs, caps, cv))
+ self.assertTrue((pre / "conditions" / "condition_0001.safetensors").exists())
+
+ def test_new_trigger_reencodes_only_changed_captions(self):
+ with tempfile.TemporaryDirectory() as t:
+ pre, imgs, caps, cv = _mk(t, ["a.png", "b.png", "c.png"], ["mmx26 a", "mmx26 b", "same"])
+ PI._reconcile_precomputed(pre, imgs, caps, cv); _seed_cache(pre, 3)
+ act = PI._reconcile_precomputed(pre, imgs, ["mrztrn a", "mrztrn b", "same"], cv)
+ self.assertIn("2 caption(s) changed", act)
+ self.assertFalse((pre / "conditions" / "condition_0000.safetensors").exists())
+ self.assertFalse((pre / "conditions" / "condition_0001.safetensors").exists())
+ self.assertTrue((pre / "conditions" / "condition_0002.safetensors").exists())
+ self.assertTrue((pre / "latents" / "latent_0000.safetensors").exists())
+
+ def test_dropped_photo_wipes_everything(self):
+ with tempfile.TemporaryDirectory() as t:
+ pre, imgs, caps, cv = _mk(t, ["a.png", "b.png", "c.png"], ["x", "y", "z"])
+ PI._reconcile_precomputed(pre, imgs, caps, cv); _seed_cache(pre, 3)
+ act = PI._reconcile_precomputed(pre, imgs[:1] + imgs[2:], ["x", "z"], cv)
+ self.assertIn("images or canvas changed", act)
+ self.assertFalse((pre / "latents").exists())
+ self.assertFalse((pre / "conditions").exists())
+
+ def test_cache_without_manifest_is_not_trusted(self):
+ with tempfile.TemporaryDirectory() as t:
+ pre, imgs, caps, cv = _mk(t, ["a.png"], ["x"])
+ _seed_cache(pre, 1)
+ act = PI._reconcile_precomputed(pre, imgs, caps, cv)
+ self.assertIn("predates manifests", act)
+ self.assertFalse((pre / "latents").exists())
+ self.assertTrue(json.loads((pre / "manifest.json").read_text())["captions"])
+
+ def test_canvas_change_wipes_latents(self):
+ with tempfile.TemporaryDirectory() as t:
+ pre, imgs, caps, cv = _mk(t, ["a.png"], ["x"])
+ PI._reconcile_precomputed(pre, imgs, caps, cv); _seed_cache(pre, 1)
+ act = PI._reconcile_precomputed(pre, imgs, caps, (1024, 576, "center"))
+ self.assertIn("images or canvas changed", act)
+ self.assertFalse((pre / "latents").exists())
+
+
+if __name__ == "__main__":
+ unittest.main()