Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
release(v4.19.1): voices train correctly again, and a failed training says why
mrbizarro7afda3c0b429
diff --git a/README.md b/README.mdindex 7c202d1..163e584 100644--- a/README.md+++ b/README.md@@ -19,7 +19,7 @@</p>-> **Current release: v4.19.0.** **Hailuo H3 grows up, and trained characters look like themselves again.** On Hailuo H3: **Keyframes** (an end frame, first + last, or stills at exact moments), **Extend** any clip, **Lip-sync** a face to your own voice or song, **Add sound** to a silent clip, smoother 10-15 s shots (each window carries the end of the last one, picture and sound), and **Fast HD**. Characters and voices trained in Phosphene since June were saved with their numbers in the wrong order and carried no likeness; new trainings save correctly and **Update repairs the ones you already have** (found by @tanis2000, #62). **Extend no longer wears your clip down** round after round: the part you already had is delivered untouched and only the new part is encoded (#48). 4.18.0: Play all, Use this audio for..., LoRA-aware Enhance. Press Update in Pinokio, then restart. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).+> **Current release: v4.19.1.** Voices trained in Phosphene save correctly again (4.19.0 still saved them scrambled; Update repairs them), and when a training run stops, the Logs now say why. **4.19.0: Hailuo H3 grows up, and trained characters look like themselves again.** On Hailuo H3: **Keyframes** (an end frame, first + last, or stills at exact moments), **Extend** any clip, **Lip-sync** a face to your own voice or song, **Add sound** to a silent clip, smoother 10-15 s shots (each window carries the end of the last one, picture and sound), and **Fast HD**. Characters and voices trained in Phosphene since June were saved with their numbers in the wrong order and carried no likeness; new trainings save correctly and **Update repairs the ones you already have** (found by @tanis2000, #62). **Extend no longer wears your clip down** round after round: the part you already had is delivered untouched and only the new part is encoded (#48). 4.18.0: Play all, Use this audio for..., LoRA-aware Enhance. Press Update in Pinokio, then restart. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).## Overviewdiff --git a/VERSION b/VERSIONindex a69aa5a..1fc0e81 100644--- a/VERSION+++ b/VERSION@@ -1 +1 @@-4.19.0+4.19.1diff --git a/docs/ANALYTICS.md b/docs/ANALYTICS.mdindex c8387b1..51bd4e9 100644--- a/docs/ANALYTICS.md+++ b/docs/ANALYTICS.md@@ -238,6 +238,7 @@ Same as `render_completed` (minus `first_render`), plus:| `error_class` | string | `"metal_watchdog"` | **A closed 19-value taxonomy** (`refused`, `oom_jetsam`, `metal_watchdog`, `metal_oom`, `native_crash`, `helper_start_timeout`, `helper_exit`, `model_missing`, `model_corrupt`, `download_failed`, `venv_broken`, `bad_params`, `input_missing`, `file_missing`, `disk_full`, `export_failed`, `timeout`, `cancelled_race`, `other`). Classification runs on the original error text locally; **only the class leaves the machine.** `refused` is in the taxonomy but never rides on a `render_failed` event — it is the value that routes the event to `render_refused` instead, below || `error_fingerprint` | string | `"a3f09c21e7b4"` | Only when `error_class` is `other`: 12 hex chars of the SHA-256 of the already-scrubbed first line. Lets "the same unknown error, 17 times, all on M1 Max" be counted without transmitting the text — the readable line stays in your own `state/usage-log.jsonl` || `error_signature` | string | `"RuntimeError: helper exited before first frame"` | **The only free-text field the panel sends**, see the scrubbing rules below. Kept for ONE transition release alongside `error_class`, then removed |+| `train_error` | string | `"oom"` | 4.19.1 — only on a failed **training** job (`mode: train`): why it stopped. **Closed vocabulary**: `oom` (MLX/Metal could not allocate, or macOS killed the trainer with SIGKILL), `missing_file` (incl. the LTX-2.3 training weights not downloaded: the trainer now checks them first and stops within seconds), `caption_check` (the captions carry another trigger), `disk_full`, `import_error` (the venv is missing a package), `nan_loss` (the loss stopped being a number for 10 steps in a row; the trainer now stops there), `other`. The trainer classifies its own exception by TYPE first and then by MLX's fixed allocation-failure fragments (`lora_lab/failure.py`), sends the word in its `error` event, and the panel forwards it only if it is on this list. Never a path, caption, trigger or message: those stay in the Logs tab and `<dataset>/train.log`. Before 4.19.1 every one of these reached the fleet as `helper_exit` + "training exited with code 1"; that sentence and its `error_class` are unchanged, so the old series continues and `train_error` splits it |Cancelled jobs are **not** reported — a user cancelling is not a signal aboutthe software.diff --git a/lora_lab/failure.py b/lora_lab/failure.pynew file mode 100644index 0000000..c910e3d--- /dev/null+++ b/lora_lab/failure.py@@ -0,0 +1,98 @@+"""Why a training run stopped, as one word from a closed list (4.19.1).++The fleet only ever saw "training exited with code 1 — see Logs for stack+trace": 4.19.0 had nine of those on two Macs and nothing to tell a memory+failure from a missing file from a full disk. The trainer knows — it holds the+exception — so it names the class here and sends it with its `error` event;+the panel checks it against the same list and sends only the word.++Classified from the exception's TYPE first and only then from MLX's own+fixed message fragments (MLX raises every Metal allocation failure as a plain+RuntimeError). Never from a path, a caption, a trigger or anything else the+user typed: the class is all that may leave the machine.+"""+from __future__ import annotations++import errno+import math++#: The closed vocabulary. docs/ANALYTICS.md, `render_failed.train_error`.+TRAIN_ERROR_CLASSES = ("oom", "missing_file", "caption_check", "disk_full",+ "import_error", "nan_loss", "other")++# MLX / Metal allocation failures. All of these arrive as RuntimeError text.+_OOM_FRAGMENTS = (+ "[metal::malloc]",+ "insufficient memory",+ "outofmemory", # kIOGPUCommandBufferCallbackErrorOutOfMemory+ "out of memory",+ "resource limit", # "[metal::malloc] Resource limit (499000) exceeded"+ "maximum allowed buffer size",+ "unable to allocate",+)+_DISK_FRAGMENTS = ("no space left on device", "disk quota exceeded")++#: Consecutive non-finite losses before a run is stopped as `nan_loss`. One+#: NaN step poisons every LoRA parameter through the optimizer, so every step+#: after it is NaN too; a short streak is enough to be sure and saves hours.+NAN_LOSS_ABORT_STEPS = 10+++class NonFiniteLoss(RuntimeError):+ """The loss stopped being a number; the adapter is no longer trainable."""+++class CaptionCheckError(RuntimeError):+ """The captions do not carry the trigger the job was started with."""+++def _chain(exc: BaseException):+ seen = set()+ while exc is not None and id(exc) not in seen:+ seen.add(id(exc))+ yield exc+ exc = exc.__cause__ or exc.__context__+++def classify_exception(exc: BaseException | None) -> str:+ """One of TRAIN_ERROR_CLASSES for the exception that stopped a run.++ The whole cause chain is read (the dataset loader wraps a missing file in+ a RuntimeError), most specific signal first."""+ if exc is None:+ return "other"+ links = list(_chain(exc))+ for e in links:+ if isinstance(e, NonFiniteLoss):+ return "nan_loss"+ if isinstance(e, CaptionCheckError):+ return "caption_check"+ for e in links:+ if isinstance(e, MemoryError):+ return "oom"+ if isinstance(e, OSError) and e.errno in (errno.ENOSPC, errno.EDQUOT):+ return "disk_full"+ for e in links:+ text = str(e).lower()+ if any(f in text for f in _DISK_FRAGMENTS):+ return "disk_full"+ if any(f in text for f in _OOM_FRAGMENTS):+ return "oom"+ for e in links:+ if isinstance(e, ImportError):+ return "import_error"+ if isinstance(e, FileNotFoundError):+ return "missing_file"+ # huggingface_hub reads a local model folder that does not exist as a+ # malformed repo id: the shape of a missing LTX-2.3 pack before 4.19.1.+ if (type(e).__name__ == "HFValidationError"+ and "repo id must be in the form" in str(e).lower()):+ return "missing_file"+ return "other"+++def is_finite_loss(value) -> bool:+ try:+ return math.isfinite(float(value))+ except (TypeError, ValueError):+ return Falsediff --git a/lora_lab/train.py b/lora_lab/train.pyindex 93c840f..e89576c 100644--- a/lora_lab/train.py+++ b/lora_lab/train.py@@ -160,6 +160,32 @@ def _patch_loader_prefer_dev_transformer() -> None:ml_module.load_transformer = patched+def training_weights_problem(model_dir: str | Path) -> str | None:+ """What is missing for a training run, in one sentence, or None (4.19.1).++ Checked before the crop and the caption encode. Without it a missing+ LTX-2.3 base pack died ~10 s in inside the preprocessor's Hugging Face+ lookup ("Repo id must be in the form 'repo_name'..."), and a missing+ full-precision dev transformer died only after minutes of caption and+ image encoding - both as a bare "exited with code 1". The Train tab shows+ a download banner for these, but Start was never blocked by it.+ Same locations and size floor as the loader patch above."""+ model_dir = Path(model_dir)+ if not model_dir.is_dir():+ return (f"the LTX-2.3 base pack is not downloaded (no {model_dir.name}/ in the models "+ "folder). Open the Train tab and click Download all, then start again.")+ full_dev_min = 15 * 1000**3+ for cand in (model_dir / "transformer-dev.safetensors",+ model_dir.parent / "ltx-2.3-mlx-q8" / "transformer-dev.safetensors"):+ try:+ if cand.is_file() and cand.stat().st_size >= full_dev_min:+ return None+ except OSError:+ continue+ return ("the full-precision LTX-2.3 dev transformer (~21 GB) is not downloaded, or "+ "only partly. Open the Train tab and click Download all, then start again.")++def _patch_compute_video_positions_fps_kwarg() -> None:"""Translate ``fps=`` → ``frame_rate=`` across the trainer.diff --git a/lora_lab/train_audio.py b/lora_lab/train_audio.pyindex 3db05ff..1eb5cbb 100644--- a/lora_lab/train_audio.py+++ b/lora_lab/train_audio.py@@ -72,6 +72,9 @@ from typing import Anyimport yaml+from lora_lab.failure import (NAN_LOSS_ABORT_STEPS, NonFiniteLoss, classify_exception,+ is_finite_loss)+logger = logging.getLogger(__name__)@@ -117,8 +120,12 @@ def emit(event: str, **kwargs: Any) -> None:_REAL_STDOUT.flush()-def emit_error_and_exit(stage: str, message: str, code: int = 1) -> None:- emit("error", stage=stage, message=message)+def emit_error_and_exit(stage: str, message: str, code: int = 1,+ exc: BaseException | None = None) -> None:+ # `error_class` is the closed word the panel may report (lora_lab.failure);+ # `message` stays on this Mac (the Logs tab and <job>/train.log).+ emit("error", stage=stage, message=message,+ error_class=classify_exception(exc))sys.exit(code)@@ -368,13 +375,7 @@ def run_training(# The audio-attn exclusion filter inside train.py drops audio_* matches —# but here we WANT those targets, so we deliberately re-monkey-patch the# filter AFTER lab_train's patch to restore the audio paths.- lab_train._patch_loader_prefer_dev_transformer()- lab_train._patch_strategy_for_image_only()- lab_train._patch_compute_video_positions_fps_kwarg()- # The voice adapter saves through the same _save_checkpoint, so without- # this it is written scrambled exactly like the character one (#62).- lab_train._patch_contiguous_checkpoint_save()- _restore_audio_lora_targets()+ _apply_trainer_patches(lab_train)total_steps = int(config_dict["optimization"]["steps"])emit("train_start", total_steps=total_steps)@@ -390,12 +391,25 @@ def run_training(_orig_update = tp_mod.TrainingProgress.update_training+ nonfinite_streak = 0+def _sniff_update(self, *, loss, lr, step_time, advance=True): # noqa: ANN001- nonlocal last_loss+ nonlocal last_loss, nonfinite_streaktry:- last_loss = float(loss)+ value = float(loss)except Exception: # noqa: BLE001- pass+ value = None+ if value is not None:+ if is_finite_loss(value):+ last_loss = value+ nonfinite_streak = 0+ else:+ # Same stop as the face phase (lora_lab.failure, 4.19.1).+ nonfinite_streak += 1+ if nonfinite_streak >= NAN_LOSS_ABORT_STEPS:+ raise NonFiniteLoss(+ f"the voice training loss stopped being a number for "+ f"{nonfinite_streak} steps in a row")return _orig_update(self, loss=loss, lr=lr, step_time=step_time, advance=advance)tp_mod.TrainingProgress.update_training = _sniff_update@@ -444,18 +458,42 @@ def _restore_audio_lora_targets() -> None:The face-side patch drops any LoRA target whose path contains``audio_attn`` or ``audio_ff`` — that's correct for face training (randomgradients corrupt the audio path). For AUDIO training those are exactly- the targets we want to keep, so we re-bind ``_find_lora_targets`` to the- underlying upstream implementation.+ the targets we want to keep, so ``_find_lora_targets`` goes back to the+ upstream implementation.++ 4.19.1: this used to ``importlib.reload`` the trainer module. A reload+ re-runs ``from safetensors.numpy import save_file as save_safetensors``,+ which silently threw away the #62 contiguous-save patch applied just+ before it — so every voice trained on 4.19.0 was still written scrambled.+ The face patch keeps the upstream function (``_phos_orig_find_lora_targets``)+ and switching it off rebinds that, touching nothing else."""- import ltx_trainer_mlx.trainer as _trainer+ from lora_lab import train as lab_train- # The original was captured via closure in the face patch; we can't reach- # back into it, so re-import the upstream module fresh.- import importlib- importlib.reload(_trainer)+ lab_train._patch_lora_target_exclude_audio(enabled=False)logger.info("audio LoRA: restored upstream _find_lora_targets (audio paths preserved)")+def _apply_trainer_patches(lab_train: Any) -> None:+ """The patch set train.py / train_character.py apply, for a voice run.++ The audio-target restore runs BEFORE the contiguous save, and the save+ patch is checked afterwards: a voice adapter saved without it is+ scrambled (#62), and nothing downstream would notice."""+ import ltx_trainer_mlx.trainer as _trainer++ lab_train._patch_loader_prefer_dev_transformer()+ lab_train._patch_strategy_for_image_only()+ lab_train._patch_compute_video_positions_fps_kwarg()+ _restore_audio_lora_targets()+ # The voice adapter saves through the same _save_checkpoint, so without+ # this it is written scrambled exactly like the character one (#62).+ lab_train._patch_contiguous_checkpoint_save()+ if getattr(_trainer.save_safetensors, "__name__", "") != "save_contiguous":+ raise RuntimeError("the voice trainer's checkpoint save is not the "+ "contiguous one; refusing to write a scrambled voice (#62)")++# ----------------------------------------------------------------------# Phase 5: discover the final checkpoint + copy to output_path# ----------------------------------------------------------------------@@ -545,7 +583,7 @@ def run_pipeline(spec_path: Path) -> int:)except Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit_error_and_exit("preprocess", str(exc))+ emit_error_and_exit("preprocess", str(exc), exc=exc)# ---- phase 3: generate audio YAML ----audio_train_out_dir = dataset_dir / "audio_train_output"@@ -562,7 +600,7 @@ def run_pipeline(spec_path: Path) -> int:emit("log", msg=f"wrote audio config: {audio_yaml_path}")except Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit_error_and_exit("config", str(exc))+ emit_error_and_exit("config", str(exc), exc=exc)# ---- phase 4: train ----try:@@ -572,7 +610,7 @@ def run_pipeline(spec_path: Path) -> int:)except Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit_error_and_exit("train", str(exc))+ emit_error_and_exit("train", str(exc), exc=exc)# ---- phase 5: publish final checkpoint ----# trainer.train() already returns the last-saved checkpoint path; double-@@ -582,13 +620,13 @@ def run_pipeline(spec_path: Path) -> int:if not checkpoint_path.exists():checkpoint_path = find_final_checkpoint(audio_train_out_dir, audio_steps)except Exception as exc: # noqa: BLE001- emit_error_and_exit("publish", f"failed to locate checkpoint: {exc}")+ emit_error_and_exit("publish", f"failed to locate checkpoint: {exc}", exc=exc)output_path.parent.mkdir(parents=True, exist_ok=True)try:shutil.copy2(str(checkpoint_path), str(output_path))except Exception as exc: # noqa: BLE001- emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}")+ emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}", exc=exc)# Same honesty the image LoRA gets: say how much delta the voice adapter# carries. #62 reported the voice LoRA "equally ignored", and a run that@@ -634,7 +672,8 @@ def main(argv: list[str] | None = None) -> int:raiseexcept Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit("error", stage="unknown", message=str(exc))+ emit("error", stage="unknown", message=str(exc),+ error_class=classify_exception(exc))return 1diff --git a/lora_lab/train_character.py b/lora_lab/train_character.pyindex 33f2a01..7e48673 100644--- a/lora_lab/train_character.py+++ b/lora_lab/train_character.py@@ -107,6 +107,9 @@ from typing import Anyimport yaml+from lora_lab.failure import (NAN_LOSS_ABORT_STEPS, NonFiniteLoss, classify_exception,+ is_finite_loss)+logger = logging.getLogger(__name__)@@ -332,8 +335,12 @@ def emit(event: str, **kwargs: Any) -> None:_REAL_STDOUT.flush()-def emit_error_and_exit(stage: str, message: str, code: int = 1) -> None:- emit("error", stage=stage, message=message)+def emit_error_and_exit(stage: str, message: str, code: int = 1,+ exc: BaseException | None = None) -> None:+ # `error_class` is the closed word the panel may report (lora_lab.failure);+ # `message` stays on this Mac (the Logs tab and <job>/train.log).+ emit("error", stage=stage, message=message,+ error_class=classify_exception(exc))sys.exit(code)@@ -825,14 +832,31 @@ def run_training(_orig_update = tp_mod.TrainingProgress.update_training+ nonfinite_streak = 0+def _sniff_update(self, *, loss, lr, step_time, advance=True): # noqa: ANN001- nonlocal last_loss, loss_sum, loss_n+ nonlocal last_loss, loss_sum, loss_n, nonfinite_streaktry:- last_loss = float(loss)- loss_sum += last_loss- loss_n += 1+ value = float(loss)except Exception: # noqa: BLE001- pass+ value = None+ if value is not None:+ if is_finite_loss(value):+ last_loss = value+ loss_sum += value+ loss_n += 1+ nonfinite_streak = 0+ else:+ # One NaN step poisons every LoRA parameter through the+ # optimizer; every step after it is NaN as well. Stop instead+ # of spending hours writing a file of NaNs (4.19.1: the fleet+ # could not tell this from any other "exit 1").+ nonfinite_streak += 1+ if nonfinite_streak >= NAN_LOSS_ABORT_STEPS:+ raise NonFiniteLoss(+ f"the training loss stopped being a number for "+ f"{nonfinite_streak} steps in a row - the adapter "+ f"cannot learn from here")return _orig_update(self, loss=loss, lr=lr, step_time=step_time, advance=advance)tp_mod.TrainingProgress.update_training = _sniff_update@@ -1026,6 +1050,13 @@ def run_pipeline(spec_path: Path) -> int:emit("start", job_id=job_id)+ # Weights first: a missing pack is a ten-second answer, not a failure+ # minutes into the caption encode (4.19.1, lora_lab.train).+ from lora_lab import train as lab_train+ _missing = lab_train.training_weights_problem(DEFAULT_MODEL_PATH)+ if _missing:+ emit_error_and_exit("models", _missing, exc=FileNotFoundError(_missing))+# Resolve preset + overrides.cfg = resolve_preset(preset, advanced)cfg["preset_name"] = preset@@ -1099,14 +1130,14 @@ def run_pipeline(spec_path: Path) -> int:try:source_files = list_source_images(images_dir)except Exception as exc: # noqa: BLE001- emit_error_and_exit("ingest", str(exc))+ emit_error_and_exit("ingest", str(exc), exc=exc)image_count = len(source_files)try:recipe = resolve_recipe(cfg)except ValueError as exc:- emit_error_and_exit("config", str(exc))+ emit_error_and_exit("config", str(exc), exc=exc)estimated_wall_s = estimate_wall_seconds(image_count, preset, advanced)emit(@@ -1147,7 +1178,7 @@ def run_pipeline(spec_path: Path) -> int:)except Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit_error_and_exit("crop", str(exc))+ emit_error_and_exit("crop", str(exc), exc=exc)try:run_preprocess(@@ -1160,7 +1191,7 @@ def run_pipeline(spec_path: Path) -> int:)except Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit_error_and_exit("preprocess", str(exc))+ emit_error_and_exit("preprocess", str(exc), exc=exc)try:checkpoint_path, training_wall_s = run_training(@@ -1171,14 +1202,14 @@ def run_pipeline(spec_path: Path) -> int:)except Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit_error_and_exit("train", str(exc))+ emit_error_and_exit("train", str(exc), exc=exc)# Move the checkpoint to the spec's output path.output_lora_path.parent.mkdir(parents=True, exist_ok=True)try:shutil.copy2(str(checkpoint_path), str(output_lora_path))except Exception as exc: # noqa: BLE001- emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}")+ emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}", exc=exc)# Measure before the sidecar so the strength verdict is on record even if# sidecar writing fails — the number is the part a stuck user needs.@@ -1194,7 +1225,7 @@ def run_pipeline(spec_path: Path) -> int:strength=strength,)except Exception as exc: # noqa: BLE001- emit_error_and_exit("sidecar", str(exc))+ emit_error_and_exit("sidecar", str(exc), exc=exc)emit("done")return 0@@ -1225,7 +1256,8 @@ def main(argv: list[str] | None = None) -> int:raiseexcept Exception as exc: # noqa: BLE001traceback.print_exc(file=sys.stderr)- emit("error", stage="unknown", message=str(exc))+ emit("error", stage="unknown", message=str(exc),+ error_class=classify_exception(exc))return 1diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.pyindex 1414323..cf006ce 100644--- a/mlx_ltx_panel.py+++ b/mlx_ltx_panel.py@@ -21,6 +21,7 @@ import ioimport jsonimport mathimport collections+import errnoimport osimport randomimport re@@ -18679,6 +18680,11 @@ def _analytics_render_event(job: dict) -> None:props["error_signature"] = scrubbedif props["error_class"] == "other":props["error_fingerprint"] = _analytics_error_fingerprint(scrubbed)+ if mode == "train":+ # 4.19.1: why a TRAINING run failed, from the trainer's+ # own exception (lora_lab.failure) — never its text.+ te = str(job.get("train_error") or "")+ props["train_error"] = te if te in TRAIN_ERROR_CLASSES else "other"_analytics_capture(event, props)except Exception:pass@@ -33136,6 +33142,45 @@ def _train_wipe_caption_conditions(dataset_dir: Path) -> int:return n+#: Why a training job failed, one word (docs/ANALYTICS.md `train_error`).+#: Mirrors lora_lab.failure.TRAIN_ERROR_CLASSES, which the trainer classifies+#: into; the panel never forwards a word outside this list.+TRAIN_ERROR_CLASSES = ("oom", "missing_file", "caption_check", "disk_full",+ "import_error", "nan_loss", "other")+++def train_error_from_event(payload: dict) -> dict:+ """The trainer's `error` event, its class held to the closed list."""+ cls = str(payload.get("error_class") or "other")+ return {+ "stage": str(payload.get("stage") or "unknown")[:40],+ "message": str(payload.get("message") or "")[:2000],+ "error_class": cls if cls in TRAIN_ERROR_CLASSES else "other",+ }+++def train_error_for_exit(rc: int, trainer_error: dict | None) -> str:+ """The class of a trainer that exited non-zero. The trainer's own word+ wins; a process killed by SIGKILL without one is macOS reclaiming memory+ (jetsam), the one signal death that has a single cause."""+ if trainer_error and trainer_error.get("error_class") in TRAIN_ERROR_CLASSES:+ return trainer_error["error_class"]+ if rc == -9:+ return "oom"+ return "other"+++def _train_release_render_memory() -> None:+ """Stop the warm render helper before a training run (4.19.1)."""+ try:+ if HELPER.is_alive():+ push("[train] releasing the render engine's memory for training "+ "(it reloads on the next render)")+ HELPER.kill()+ except Exception as e: # never let housekeeping fail a training run+ push(f"[train] could not release the render engine: {e}")++def run_train_job_inner(job: dict) -> None:"""Run a Train Character job. Reads the dataset that /train/upload builtunder state/train_character/<train_job_id>/, writes a job-spec JSON the@@ -33179,6 +33224,7 @@ def run_train_job_inner(job: dict) -> None:images_dir = dataset_dir / "images"captions_dir = dataset_dir / "captions"if not images_dir.is_dir():+ job["train_error"] = "missing_file"raise RuntimeError(f"dataset images not found: {images_dir} — ""/train/upload should have created this. Re-upload your dataset.")@@ -33299,6 +33345,7 @@ def run_train_job_inner(job: dict) -> None:f"({other[1]}/{len(mismatched)} mismatched files)"if other else "")if len(mismatched) == len(user_cap_files):+ job["train_error"] = "caption_check"raise RuntimeError(f"none of the {len(user_cap_files)} existing caption files "f"contain the trigger {trigger!r} this job was submitted "@@ -33387,9 +33434,12 @@ def run_train_job_inner(job: dict) -> None:try:spec_path.write_text(json.dumps(spec, indent=2), encoding="utf-8")except OSError as e:+ job["train_error"] = ("disk_full" if e.errno in (errno.ENOSPC, errno.EDQUOT)+ else "other")raise RuntimeError(f"could not write spec.json: {e}")if not LORA_LAB_RUN_SH.exists():+ job["train_error"] = "missing_file"raise RuntimeError(f"lora-lab shim not found at {LORA_LAB_RUN_SH}. ""Train Character depends on the sibling lab at "@@ -33427,6 +33477,11 @@ def run_train_job_inner(job: dict) -> None:_train_phase = "start"_train_env = trainer_child_env()_encode_retried = False+ _trainer_error: dict | None = None+ # A warm render helper keeps an LTX model resident (up to tens of GB) for+ # HELPER_IDLE_TIMEOUT after the last render. The trainer needs that memory+ # for the dev transformer and its activations; give it back first.+ _train_release_render_memory()try:while True:proc = subprocess.Popen(@@ -33520,7 +33575,20 @@ def run_train_job_inner(job: dict) -> None:if last_step % 50 == 0 or last_step == total_steps:push(f"[train] step {last_step}/{total_steps} loss={last_loss}")elif evt == "log":- push(f"[train] {payload.get('msg', '')}")+ # The trainer writes its log lines as `line`; `msg` is+ # the older key. Read both (4.19.0 showed an empty+ # "[train] " for the recipe line).+ push(f"[train] {payload.get('msg') or payload.get('line') or ''}")+ elif evt == "error":+ # The trainer's own account of why it stopped: the+ # stage, its message (stays on this Mac) and a closed+ # class (lora_lab.failure) — the only part analytics+ # may carry. Until 4.19.1 this fell through to the+ # generic branch and the class was lost: every one of+ # these reached the fleet as "exited with code 1".+ _trainer_error = train_error_from_event(payload)+ push(f"[train] stopped in {_trainer_error['stage']}: "+ f"{_trainer_error['message']} ({_trainer_error['error_class']})")elif evt == "adapter_strength":# A GREEN TALLY WAS NEVER PROOF THE FILE COULD DO# ANYTHING. A LoRA can train to completion, write a@@ -33605,10 +33673,12 @@ def run_train_job_inner(job: dict) -> None:"trainer process (AGX_RELAX_CDM_CTXSTORE_TIMEOUT=1) — the ""screen may feel less responsive until it finishes.")_train_watchdog_seen = False+ _trainer_error = None_train_phase = "start"continuebreakif rc != 0:+ job["train_error"] = train_error_for_exit(rc, _trainer_error)# -6 is SIGABRT. With a watchdog signature in the log it is macOS# killing the GPU command buffer, not our code faulting — and the# lever is the canvas, not the preset. Reported in #61: 576x576@@ -33653,6 +33723,7 @@ def run_train_job_inner(job: dict) -> None:elapsed = round(time.time() - t0, 2)if not final_lora_path.is_file():+ job["train_error"] = "missing_file"raise RuntimeError(f"training finished but output .safetensors missing at {final_lora_path}")diff --git a/scripts/test_convert_ltx_mlx.py b/scripts/test_convert_ltx_mlx.pyindex 35146e4..bfdc42c 100644--- a/scripts/test_convert_ltx_mlx.py+++ b/scripts/test_convert_ltx_mlx.py@@ -453,7 +453,26 @@ class TestEndToEnd:on_disk = json.loads((out_dir / "config.json").read_text())assert on_disk["transformer"]["ff_bias"] is False- def test_the_vendored_config_reader_accepts_the_result(self, tmp_path):+ @pytest.fixture+ def _own_ltx_core(self):+ """The port checkout goes on sys.path for THIS test only (4.19.1).++ It used to stay there, and whichever later test imported an+ ltx_core_mlx submodule for the first time got the port's package+ instead of the venv's - the full suite failed four trainer tests with+ "cannot import name 'utils' from 'ltx_core_mlx' (.../ltx25-port/...)"+ on any Mac that has the port checkout."""+ saved_path = list(sys.path)+ saved = {k: v for k, v in sys.modules.items()+ if k == "ltx_core_mlx" or k.startswith("ltx_core_mlx.")}+ yield+ sys.path[:] = saved_path+ for k in [k for k in sys.modules if k == "ltx_core_mlx" or k.startswith("ltx_core_mlx.")]:+ if k not in saved:+ del sys.modules[k]+ sys.modules.update(saved)++ def test_the_vendored_config_reader_accepts_the_result(self, tmp_path, _own_ltx_core):"""The real proof: our pack is readable by the code that will load it.Needs an ``ltx_core_mlx`` that carries the 2.5 config work (thediff --git a/test_codex_4170_safety.py b/test_codex_4170_safety.pyindex bae1aac..17eaa6e 100644--- a/test_codex_4170_safety.py+++ b/test_codex_4170_safety.py@@ -338,10 +338,14 @@ def test_safety_4_a_queued_h3_job_restores_its_engine_and_h3_recipe(monkeypatch)assert out["prompt"] == "a hen skates"-def test_safety_4_a_queued_ltx_job_restores_steps_and_export_method():+def test_safety_4_a_queued_ltx_job_restores_steps_and_export_method(monkeypatch):# 4.17.4: on a DISTILLED quality make_job now clamps steps to 8 (the lane# cannot run anything else - clamp_distilled_steps), so a restorable,# non-default count has to live on an HQ quality, where `steps` is kept.+ # 4.19.1: as a Mac WITH the PiperSR upscaler (#90 class) - make_job folds+ # pipersr to lanczos where it is not installed, which is every fresh+ # install, so this read the machine instead of testing the restore.+ monkeypatch.setattr(p, "PIPERSR_UPSCALE_ENABLED", True)ltx = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "a fox", "steps": "12","upscale": "fit_720p", "upscale_method": "pipersr", "quality": "high"})["params"]assert ltx["steps"] == 12diff --git a/test_music_engine.py b/test_music_engine.pyindex 14c8e36..356dcf1 100644--- a/test_music_engine.py+++ b/test_music_engine.py@@ -302,7 +302,7 @@ def test_fetch_stages_moves_and_skips_intact(pack, monkeypatch):def test_runner_help_and_empty_input():- music_python = Path.home() / "AI/projects/yue2-mlx/.venv/bin/python"+ music_python = ROOT / Path(os.environ.get("LTX_MUSIC_ROOT", "yue2-mlx")).expanduser() / ".venv/bin/python"runner = ROOT / "scripts/music/yue2_run.py"result = subprocess.run([sys.executable, str(runner), "--model-dir", "/missing", "--vae-dir", "/missing", "--output", "/tmp/no-song.wav"], capture_output=True, text=True, timeout=30)assert result.returncode == 2 and "Give it something" in result.stdoutdiff --git a/test_review_416_ltx.py b/test_review_416_ltx.pyindex 0f36ec9..49e5da6 100644--- a/test_review_416_ltx.py+++ b/test_review_416_ltx.py@@ -340,10 +340,21 @@ class TheHelperAttachesThem(unittest.TestCase):for name, attrs in fake_modules.items():mod = types.ModuleType(name)mod.__dict__.update(attrs)+ # Put back what was there, not nothing (4.19.1): popping a stand-in+ # for `ltx_core_mlx` dropped the REAL package, and the next+ # `import ltx_core_mlx.utils.positions as m` in the same process+ # (the trainer tests) got a fresh package with no `utils` on it.+ self.addCleanup(self._restore_module, name, sys.modules.get(name))sys.modules[name] = mod- self.addCleanup(sys.modules.pop, name, None)return ns+ @staticmethod+ def _restore_module(name, prev):+ if prev is None:+ sys.modules.pop(name, None)+ else:+ sys.modules[name] = prev+def test_fflf_attaches_and_rebuilds_on_a_new_set(self):built = []diff --git a/test_train_failure_class.py b/test_train_failure_class.pynew file mode 100644index 0000000..a45d6da--- /dev/null+++ b/test_train_failure_class.py@@ -0,0 +1,261 @@+#!/usr/bin/env python3+"""A failed training run says WHY, as one closed word (4.19.1).++4.19.0 had nine failed trainings on two Macs and every one reached the fleet+as "training exited with code 1 — see Logs for stack trace": memory, a+missing file, a full disk and a broken venv all looked the same. The trainer+held the exception and emitted an `error` event with its stage and message,+but the panel dropped that event into the generic log branch.++Pinned here:+ * lora_lab.failure.classify_exception — type first, then MLX's fixed+ allocation-failure fragments, through the cause chain;+ * the trainer's `error` event carries `error_class`;+ * the panel keeps only a word from the closed list (an unknown or+ path-shaped value becomes `other`), a SIGKILL with no word is `oom`,+ and its own pre-trainer refusals name their class;+ * render_failed carries `train_error` for training jobs only;+ * the job's error text is unchanged, so the error_class series is too.+The panel half drives the REAL run_train_job_inner against a stand-in+trainer script; no weights, no GPU.+"""+from __future__ import annotations++import errno+import io+import json+import os+import stat+import sys+import tempfile+import unittest+from pathlib import Path+from unittest import mock++ROOT = Path(__file__).resolve().parent+os.environ["LTX_STATE_DIR"] = tempfile.mkdtemp(prefix="phos-train-fail-")+os.environ["PHOSPHENE_ANALYTICS_DISABLED"] = "1"+os.environ["PHOSPHENE_DISABLE_VERSION_CHECK"] = "1"+os.environ.setdefault("LTX_PORT", "8316")+sys.path.insert(0, str(ROOT))++import mlx_ltx_panel as P # noqa: E402+from lora_lab import failure as F # noqa: E402+++class ClassifyException(unittest.TestCase):+ def test_closed_list_matches_the_panel(self) -> None:+ self.assertEqual(F.TRAIN_ERROR_CLASSES, P.TRAIN_ERROR_CLASSES)++ def test_mlx_allocation_failures_are_oom(self) -> None:+ for text in (+ "[metal::malloc] Attempting to allocate 21474836480 bytes which is greater "+ "than the maximum allowed buffer size of 17179869184 bytes.",+ "[metal::malloc] Resource limit (499000) exceeded.",+ "[METAL] Command buffer execution failed: Insufficient Memory "+ "(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)",+ ):+ self.assertEqual(F.classify_exception(RuntimeError(text)), "oom", text)+ self.assertEqual(F.classify_exception(MemoryError()), "oom")++ def test_disk_full(self) -> None:+ self.assertEqual(F.classify_exception(OSError(errno.ENOSPC, "x")), "disk_full")+ self.assertEqual(F.classify_exception(+ RuntimeError("Error while serializing: IoError(Os { code: 28, kind: "+ "StorageFull, message: \"No space left on device\" })")), "disk_full")++ def test_missing_file_through_the_dataset_loaders_wrapper(self) -> None:+ try:+ try:+ raise FileNotFoundError(2, "No such file or directory")+ except FileNotFoundError as e:+ raise RuntimeError("Failed to load latents from /x: missing") from e+ except RuntimeError as wrapped:+ self.assertEqual(F.classify_exception(wrapped), "missing_file")+ self.assertEqual(F.classify_exception(FileNotFoundError(+ "transformer-dev.safetensors not found in /a or /b")), "missing_file")++ def test_import_nan_caption_other(self) -> None:+ self.assertEqual(F.classify_exception(ModuleNotFoundError("No module named 'x'")),+ "import_error")+ self.assertEqual(F.classify_exception(F.NonFiniteLoss("nan")), "nan_loss")+ self.assertEqual(F.classify_exception(F.CaptionCheckError("x")), "caption_check")+ self.assertEqual(F.classify_exception(ValueError("bad")), "other")+ self.assertEqual(F.classify_exception(None), "other")++ def test_a_path_or_trigger_never_becomes_the_class(self) -> None:+ cls = F.classify_exception(RuntimeError("/Users/someone/zqxtrn_v2.safetensors"))+ self.assertIn(cls, F.TRAIN_ERROR_CLASSES)+ self.assertEqual(cls, "other")+++class MissingTrainingWeights(unittest.TestCase):+ """The panel e2e on a fresh 4.19.1 install with no LTX-2.3 pack: the run+ died 9 s in with HFValidationError ("Repo id must be in the form ...", the+ pack's own path read as a repo id) and the class came out `other`."""++ def test_the_hub_shape_of_a_missing_pack_is_missing_file(self) -> None:+ HFValidationError = type("HFValidationError", (ValueError,), {})+ e = HFValidationError("Repo id must be in the form 'repo_name' or "+ "'namespace/repo_name': '/x/mlx_models/ltx-2.3-mlx-q4'.")+ self.assertEqual(F.classify_exception(e), "missing_file")++ def test_what_is_missing_is_said_before_any_work(self) -> None:+ try:+ from lora_lab import train as lab_train+ except ImportError:+ self.skipTest("needs the trainer venv")+ with tempfile.TemporaryDirectory() as td:+ models = Path(td)+ q4 = models / "ltx-2.3-mlx-q4"+ self.assertIn("base pack", lab_train.training_weights_problem(q4))+ q4.mkdir()+ self.assertIn("dev transformer", lab_train.training_weights_problem(q4))+ small = q4 / "transformer-dev.safetensors" # the quantized 11 GB copy (#35)+ with open(small, "wb") as f:+ f.truncate(11 * 1000**3)+ self.assertIn("dev transformer", lab_train.training_weights_problem(q4))+ (models / "ltx-2.3-mlx-q8").mkdir()+ with open(models / "ltx-2.3-mlx-q8" / "transformer-dev.safetensors", "wb") as f:+ f.truncate(21 * 1000**3) # sparse: no disk used+ self.assertIsNone(lab_train.training_weights_problem(q4))++ def test_the_trainer_stops_at_once_with_the_class(self) -> None:+ try:+ import lora_lab.train_character as tc+ except ImportError:+ self.skipTest("needs the trainer venv")+ with tempfile.TemporaryDirectory() as td:+ spec = Path(td) / "spec.json"+ spec.write_text(json.dumps({"job_id": "j", "trigger": "zqxtrn", "preset": "quick",+ "images_dir": str(Path(td) / "images"),+ "output_path": str(Path(td) / "out.safetensors")}))+ buf = io.StringIO()+ with mock.patch.object(tc, "DEFAULT_MODEL_PATH", str(Path(td) / "ltx-2.3-mlx-q4")), \+ mock.patch.object(tc, "_REAL_STDOUT", buf), self.assertRaises(SystemExit):+ tc.run_pipeline(spec)+ ev = json.loads(buf.getvalue().strip().splitlines()[-1])+ self.assertEqual((ev["event"], ev["stage"], ev["error_class"]),+ ("error", "models", "missing_file"))+++class TrainerErrorEvent(unittest.TestCase):+ def _emitted(self, mod, **kw) -> dict:+ buf = io.StringIO()+ with mock.patch.object(mod, "_REAL_STDOUT", buf), self.assertRaises(SystemExit):+ mod.emit_error_and_exit(**kw)+ return json.loads(buf.getvalue().strip().splitlines()[-1])++ def test_face_and_voice_trainers_send_the_class(self) -> None:+ import lora_lab.train_audio as ta+ import lora_lab.train_character as tc+ for mod in (tc, ta):+ ev = self._emitted(mod, stage="train", message="m",+ exc=RuntimeError("[metal::malloc] Resource limit (499000) exceeded."))+ self.assertEqual((ev["event"], ev["stage"], ev["error_class"]), ("error", "train", "oom"))+ ev = self._emitted(mod, stage="config", message="m")+ self.assertEqual(ev["error_class"], "other")+++def _fake_trainer(body: str) -> Path:+ d = Path(tempfile.mkdtemp(prefix="phos-fake-trainer-"))+ sh = d / "lora_lab_run.sh"+ sh.write_text("#!/bin/bash\n" + body + "\n")+ sh.chmod(sh.stat().st_mode | stat.S_IEXEC)+ return sh+++class PanelKeepsTheWord(unittest.TestCase):+ TRIGGER = "zqxtrn"++ def _job(self, sh: Path) -> dict:+ from PIL import Image+ jid = "tfail" + os.urandom(4).hex()+ images = P._train_dataset_dir(jid) / "images"+ images.mkdir(parents=True, exist_ok=True)+ for i in range(P.TRAIN_MIN_IMAGES):+ Image.new("RGB", (64, 64), (i * 9, 40, 80)).save(images / f"char_{i:03d}.png")+ self._sh = sh+ return {"id": "q-" + jid, "params": {"mode": "train", "train_job_id": jid,+ "trigger": self.TRIGGER, "preset": "quick",+ "steps": 10, "rank": 4}}++ def _run(self, body: str) -> tuple[dict, str]:+ sh = _fake_trainer(body)+ job = self._job(sh)+ with mock.patch.object(P, "LORA_LAB_RUN_SH", sh):+ with self.assertRaises(RuntimeError) as cm:+ P.run_train_job_inner(job)+ return job, str(cm.exception)++ def test_the_trainers_class_reaches_the_job(self) -> None:+ ev = {"event": "error", "stage": "train", "error_class": "oom",+ "message": "[metal::malloc] Resource limit (499000) exceeded."}+ job, err = self._run(f"echo '{json.dumps(ev)}'\nexit 1")+ self.assertEqual(job["train_error"], "oom")+ # Same sentence as 4.19.0: the error_class / error_signature series+ # stay comparable across the release.+ self.assertEqual(err, "training exited with code 1 — see Logs for stack trace")++ def test_an_unknown_word_is_not_forwarded(self) -> None:+ ev = {"event": "error", "stage": "train", "error_class": "/Users/x/zqxtrn", "message": "m"}+ job, _ = self._run(f"echo '{json.dumps(ev)}'\nexit 1")+ self.assertEqual(job["train_error"], "other")++ def test_no_event_exit_1_is_other_and_sigkill_is_oom(self) -> None:+ job, _ = self._run("echo boom\nexit 1")+ self.assertEqual(job["train_error"], "other")+ job, _ = self._run("kill -9 $$")+ self.assertEqual(job["train_error"], "oom")++ def test_the_caption_refusal_names_itself(self) -> None:+ sh = _fake_trainer("exit 0")+ job = self._job(sh)+ caps = P._train_dataset_dir(job["params"]["train_job_id"]) / "captions"+ caps.mkdir(parents=True, exist_ok=True)+ for i in range(P.TRAIN_MIN_IMAGES):+ (caps / f"char_{i:03d}.txt").write_text("[VISUAL]: othertrn, a man\n")+ with mock.patch.object(P, "LORA_LAB_RUN_SH", sh), self.assertRaises(RuntimeError):+ P.run_train_job_inner(job)+ self.assertEqual(job["train_error"], "caption_check")++ def test_the_render_helper_is_released_before_training(self) -> None:+ killed = []+ with mock.patch.object(P.HELPER, "is_alive", lambda: True), \+ mock.patch.object(P.HELPER, "kill", lambda: killed.append(1)):+ self._run("exit 1")+ self.assertEqual(killed, [1])++ def test_the_trainers_log_line_is_shown(self) -> None:+ pushed = []+ ev = {"event": "log", "line": "recipe: adam_bias_correction=False"}+ with mock.patch.object(P, "push", lambda s: pushed.append(s)):+ self._run(f"echo '{json.dumps(ev)}'\nexit 1")+ self.assertIn("[train] recipe: adam_bias_correction=False", pushed)+++class TheEventCarriesIt(unittest.TestCase):+ def _props(self, job: dict) -> dict:+ sent = []+ with mock.patch.object(P, "_analytics_capture", lambda e, p: sent.append((e, p))), \+ mock.patch.object(P, "get_settings", lambda: {"analytics_first_render_reported": True}):+ P._analytics_render_event(job)+ return sent[0][1]++ def test_train_failures_only(self) -> None:+ err = "training exited with code 1 — see Logs for stack trace"+ p = self._props({"status": "failed", "error": err, "train_error": "oom",+ "params": {"mode": "train", "width": 512, "height": 512}})+ self.assertEqual(p["train_error"], "oom")+ self.assertEqual(p["error_class"], "helper_exit")+ p = self._props({"status": "failed", "error": err, "train_error": "nonsense",+ "params": {"mode": "train"}})+ self.assertEqual(p["train_error"], "other")+ p = self._props({"status": "failed", "error": "boom", "params": {"mode": "t2v"}})+ self.assertNotIn("train_error", p)+ p = self._props({"status": "done", "params": {"mode": "train"}})+ self.assertNotIn("train_error", p)+++if __name__ == "__main__":+ unittest.main()diff --git a/test_train_voice_save_and_nan.py b/test_train_voice_save_and_nan.pynew file mode 100644index 0000000..6c4bd30--- /dev/null+++ b/test_train_voice_save_and_nan.py@@ -0,0 +1,171 @@+#!/usr/bin/env python3+"""Two 4.19.1 trainer fixes, through the real trainer code on a tiny model.++1. VOICES WERE STILL SAVED SCRAMBLED ON 4.19.0 (#62). `train_audio` applied+ the contiguous-save patch and then "restored" the audio LoRA targets with+ `importlib.reload(ltx_trainer_mlx.trainer)`. A reload re-runs the module's+ `from safetensors.numpy import save_file as save_safetensors`, so the patch+ applied one line earlier was gone and every voice adapter went to disk in+ the scrambled layout 4.19.0 was released to fix. The restore now rebinds+ only `_find_lora_targets`, and the voice run refuses to start without the+ contiguous save.++2. A NaN loss used to train on for hours and finish (or die) as an anonymous+ "exit 1". After NAN_LOSS_ABORT_STEPS non-finite steps in a row the face+ trainer stops with NonFiniteLoss, classified `nan_loss`.++CPU only, no weights, no GPU lock. The end-to-end case runs+`train_character.run_training` itself — every 4.19 patch, the loss sniffer,+the step callback, the checkpoint save — with only `load_ltx_model` swapped+for a one-block LTXModel and three precomputed samples.+"""+from __future__ import annotations++import io+import sys+import tempfile+import types+import unittest+from pathlib import Path+from unittest import mock++ROOT = Path(__file__).resolve().parent+sys.path.insert(0, str(ROOT))++try:+ import mlx.core as mx+ import numpy as np+ from mlx_lm.tuner.lora import LoRALinear+ from safetensors.numpy import load_file++ import lora_lab.train as lab_train+ import lora_lab.train_audio as ta+ import lora_lab.train_character as tc+ import ltx_trainer_mlx.progress as tp_mod+ import ltx_trainer_mlx.trainer as trainer_mod+ from lora_lab import failure as F+ from ltx_core_mlx.model.transformer.model import LTXModel, LTXModelConfig+ from ltx_trainer_mlx.config import LtxTrainerConfig+ from ltx_trainer_mlx.trainer import LtxvTrainer++ mx.set_default_device(mx.cpu)+ HAVE_MLX = True+except ImportError: # pragma: no cover — the gates run in the trainer venv+ HAVE_MLX = False++TINY = dict(num_layers=1, video_dim=64, audio_dim=32, video_num_heads=2,+ audio_num_heads=2, video_head_dim=32, audio_head_dim=16,+ av_cross_num_heads=2, av_cross_head_dim=16)++[email protected](HAVE_MLX, "needs the trainer venv (mlx + ltx_trainer_mlx)")+class VoiceSaveStaysContiguous(unittest.TestCase):+ def setUp(self) -> None:+ self._save = getattr(trainer_mod, "_phos_orig_save_safetensors", trainer_mod.save_safetensors)+ self._targets = getattr(trainer_mod, "_phos_orig_find_lora_targets",+ trainer_mod._find_lora_targets)+ self._tmp = tempfile.TemporaryDirectory()++ def tearDown(self) -> None:+ trainer_mod.save_safetensors = self._save+ lab_train._patch_contiguous_checkpoint_save()+ trainer_mod._find_lora_targets = self._targets+ self._tmp.cleanup()++ def test_restoring_the_audio_targets_keeps_the_contiguous_save(self) -> None:+ """The 4.19.0 bug in one line: this assertion fails there."""+ lab_train._patch_contiguous_checkpoint_save()+ ta._restore_audio_lora_targets()+ self.assertEqual(trainer_mod.save_safetensors.__name__, "save_contiguous")++ def test_the_voice_patch_set_saves_the_adapter_that_was_trained(self) -> None:+ lab_train._patch_lora_target_exclude_audio(True) # as a face run left it+ ta._apply_trainer_patches(lab_train)+ self.assertIs(trainer_mod._find_lora_targets, self._targets, "audio targets restored")+ mx.random.seed(0)+ t = LtxvTrainer.__new__(LtxvTrainer)+ t._transformer = LTXModel(LTXModelConfig(**TINY))+ cfg = tc.resolve_preset("high", {"rank": 4, "alpha": 4, "steps": 10})+ d = tc.build_trainer_config(cfg=cfg, data_root=Path("/d"), output_dir=Path(self._tmp.name))+ d["model"]["model_path"] = self._tmp.name+ t._config = LtxTrainerConfig.model_validate(d)+ t._setup_lora()+ mem = {}+ for path, m in t._transformer.named_modules():+ if isinstance(m, LoRALinear):+ m.lora_b = mx.random.normal(m.lora_b.shape)+ mem[path] = (np.array(m.lora_a), np.array(m.lora_b))+ self.assertTrue(any("audio_attn" in p for p in mem), "voice runs train the audio paths")+ t._global_step, t._checkpoint_paths = 10, []+ disk = load_file(str(t._save_checkpoint()))+ for path, (a, b) in mem.items():+ np.testing.assert_array_equal(disk[f"diffusion_model.{path}.lora_A.weight"], a.T)+ np.testing.assert_array_equal(disk[f"diffusion_model.{path}.lora_B.weight"], b.T)++ def test_a_voice_run_without_the_contiguous_save_refuses(self) -> None:+ with mock.patch.object(lab_train, "_patch_contiguous_checkpoint_save", lambda: None):+ trainer_mod.save_safetensors = self._save+ with self.assertRaises(RuntimeError):+ ta._apply_trainer_patches(lab_train)+++def _fake_load(model_dir=None, **kw):+ mx.random.seed(0)+ return types.SimpleNamespace(transformer=LTXModel(LTXModelConfig(**TINY)),+ video_vae_decoder=None, video_vae_encoder=None,+ audio_vae_decoder=None, vocoder=None)+++def _data(root: Path, nan: bool) -> None:+ lat = root / ".precomputed" / "latents"+ con = root / ".precomputed" / "conditions"+ lat.mkdir(parents=True)+ con.mkdir(parents=True)+ for i in range(3):+ x = mx.random.normal((128, 1, 2, 2))+ if nan:+ x = x * float("nan")+ mx.save_safetensors(str(lat / f"latent_{i:03d}.safetensors"), {+ "latents": x, "num_frames": mx.array(1), "height": mx.array(2),+ "width": mx.array(2), "fps": mx.array(24.0)})+ mx.save_safetensors(str(con / f"condition_{i:03d}.safetensors"), {+ "video_prompt_embeds": mx.random.normal((8, 64)),+ "audio_prompt_embeds": mx.random.normal((8, 32)),+ "prompt_attention_mask": mx.ones((8,))})++[email protected](HAVE_MLX, "needs the trainer venv (mlx + ltx_trainer_mlx)")+class FaceTrainingEndToEnd(unittest.TestCase):+ def setUp(self) -> None:+ self._tmp = tempfile.TemporaryDirectory()+ self._update = tp_mod.TrainingProgress.update_training+ self._stdout = io.StringIO()++ def tearDown(self) -> None:+ tp_mod.TrainingProgress.update_training = self._update+ self._tmp.cleanup()++ def _run(self, *, nan: bool, steps: int):+ root = Path(self._tmp.name)+ (root / "model").mkdir()+ _data(root / "data", nan)+ cfg = tc.resolve_preset("quick", {"rank": 4, "alpha": 4, "steps": steps, "lr": 1e-3})+ with mock.patch.object(trainer_mod, "load_ltx_model", _fake_load), \+ mock.patch.object(tc, "DEFAULT_MODEL_PATH", str(root / "model")), \+ mock.patch.object(tc, "_REAL_STDOUT", self._stdout):+ return tc.run_training(cfg=cfg, data_root=root / "data",+ output_dir=root / "out", estimated_wall_s=5)++ def test_a_short_run_trains_and_saves(self) -> None:+ ck, _ = self._run(nan=False, steps=6)+ self.assertTrue(Path(ck).is_file())+ self.assertIn('"event":"train_progress"', self._stdout.getvalue())++ def test_a_nan_loss_stops_the_run_as_nan_loss(self) -> None:+ with self.assertRaises(F.NonFiniteLoss) as cm:+ self._run(nan=True, steps=F.NAN_LOSS_ABORT_STEPS + 20)+ self.assertEqual(F.classify_exception(cm.exception), "nan_loss")+++if __name__ == "__main__":+ unittest.main()diff --git a/webapp/style/panel.css b/webapp/style/panel.cssindex 3e34906..aade8c1 100644--- a/webapp/style/panel.css+++ b/webapp/style/panel.css@@ -7664,6 +7664,7 @@.live-stage-stop {pointer-events: auto;align-self: center;+ width: auto; flex: 0 0 auto; /* the panel-wide `button { width:100% }` stretched this over the ETA text */height: 29px; padding: 0 11px;border-radius: var(--r-pill);border: 1px solid rgba(255,255,255,.2);