Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

release(v4.19.1): voices train correctly again, and a failed training says why

mrbizarro7afda3c0b429
diff --git a/README.md b/README.md
index 7c202d1..163e584 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@
</p>
-> **Current release: v4.19.0.** **Hailuo H3 grows up, and trained characters look like themselves again.** On Hailuo H3: **Keyframes** (an end frame, first + last, or stills at exact moments), **Extend** any clip, **Lip-sync** a face to your own voice or song, **Add sound** to a silent clip, smoother 10-15 s shots (each window carries the end of the last one, picture and sound), and **Fast HD**. Characters and voices trained in Phosphene since June were saved with their numbers in the wrong order and carried no likeness; new trainings save correctly and **Update repairs the ones you already have** (found by @tanis2000, #62). **Extend no longer wears your clip down** round after round: the part you already had is delivered untouched and only the new part is encoded (#48). 4.18.0: Play all, Use this audio for..., LoRA-aware Enhance. Press Update in Pinokio, then restart. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).
+> **Current release: v4.19.1.** Voices trained in Phosphene save correctly again (4.19.0 still saved them scrambled; Update repairs them), and when a training run stops, the Logs now say why. **4.19.0: Hailuo H3 grows up, and trained characters look like themselves again.** On Hailuo H3: **Keyframes** (an end frame, first + last, or stills at exact moments), **Extend** any clip, **Lip-sync** a face to your own voice or song, **Add sound** to a silent clip, smoother 10-15 s shots (each window carries the end of the last one, picture and sound), and **Fast HD**. Characters and voices trained in Phosphene since June were saved with their numbers in the wrong order and carried no likeness; new trainings save correctly and **Update repairs the ones you already have** (found by @tanis2000, #62). **Extend no longer wears your clip down** round after round: the part you already had is delivered untouched and only the new part is encoded (#48). 4.18.0: Play all, Use this audio for..., LoRA-aware Enhance. Press Update in Pinokio, then restart. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).
## Overview
diff --git a/VERSION b/VERSION
index a69aa5a..1fc0e81 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-4.19.0
+4.19.1
diff --git a/docs/ANALYTICS.md b/docs/ANALYTICS.md
index c8387b1..51bd4e9 100644
--- a/docs/ANALYTICS.md
+++ b/docs/ANALYTICS.md
@@ -238,6 +238,7 @@ Same as `render_completed` (minus `first_render`), plus:
| `error_class` | string | `"metal_watchdog"` | **A closed 19-value taxonomy** (`refused`, `oom_jetsam`, `metal_watchdog`, `metal_oom`, `native_crash`, `helper_start_timeout`, `helper_exit`, `model_missing`, `model_corrupt`, `download_failed`, `venv_broken`, `bad_params`, `input_missing`, `file_missing`, `disk_full`, `export_failed`, `timeout`, `cancelled_race`, `other`). Classification runs on the original error text locally; **only the class leaves the machine.** `refused` is in the taxonomy but never rides on a `render_failed` event — it is the value that routes the event to `render_refused` instead, below |
| `error_fingerprint` | string | `"a3f09c21e7b4"` | Only when `error_class` is `other`: 12 hex chars of the SHA-256 of the already-scrubbed first line. Lets "the same unknown error, 17 times, all on M1 Max" be counted without transmitting the text — the readable line stays in your own `state/usage-log.jsonl` |
| `error_signature` | string | `"RuntimeError: helper exited before first frame"` | **The only free-text field the panel sends**, see the scrubbing rules below. Kept for ONE transition release alongside `error_class`, then removed |
+| `train_error` | string | `"oom"` | 4.19.1 — only on a failed **training** job (`mode: train`): why it stopped. **Closed vocabulary**: `oom` (MLX/Metal could not allocate, or macOS killed the trainer with SIGKILL), `missing_file` (incl. the LTX-2.3 training weights not downloaded: the trainer now checks them first and stops within seconds), `caption_check` (the captions carry another trigger), `disk_full`, `import_error` (the venv is missing a package), `nan_loss` (the loss stopped being a number for 10 steps in a row; the trainer now stops there), `other`. The trainer classifies its own exception by TYPE first and then by MLX's fixed allocation-failure fragments (`lora_lab/failure.py`), sends the word in its `error` event, and the panel forwards it only if it is on this list. Never a path, caption, trigger or message: those stay in the Logs tab and `<dataset>/train.log`. Before 4.19.1 every one of these reached the fleet as `helper_exit` + "training exited with code 1"; that sentence and its `error_class` are unchanged, so the old series continues and `train_error` splits it |
Cancelled jobs are **not** reported — a user cancelling is not a signal about
the software.
diff --git a/lora_lab/failure.py b/lora_lab/failure.py
new file mode 100644
index 0000000..c910e3d
--- /dev/null
+++ b/lora_lab/failure.py
@@ -0,0 +1,98 @@
+"""Why a training run stopped, as one word from a closed list (4.19.1).
+
+The fleet only ever saw "training exited with code 1 — see Logs for stack
+trace": 4.19.0 had nine of those on two Macs and nothing to tell a memory
+failure from a missing file from a full disk. The trainer knows — it holds the
+exception — so it names the class here and sends it with its `error` event;
+the panel checks it against the same list and sends only the word.
+
+Classified from the exception's TYPE first and only then from MLX's own
+fixed message fragments (MLX raises every Metal allocation failure as a plain
+RuntimeError). Never from a path, a caption, a trigger or anything else the
+user typed: the class is all that may leave the machine.
+"""
+from __future__ import annotations
+
+import errno
+import math
+
+#: The closed vocabulary. docs/ANALYTICS.md, `render_failed.train_error`.
+TRAIN_ERROR_CLASSES = ("oom", "missing_file", "caption_check", "disk_full",
+ "import_error", "nan_loss", "other")
+
+# MLX / Metal allocation failures. All of these arrive as RuntimeError text.
+_OOM_FRAGMENTS = (
+ "[metal::malloc]",
+ "insufficient memory",
+ "outofmemory", # kIOGPUCommandBufferCallbackErrorOutOfMemory
+ "out of memory",
+ "resource limit", # "[metal::malloc] Resource limit (499000) exceeded"
+ "maximum allowed buffer size",
+ "unable to allocate",
+)
+_DISK_FRAGMENTS = ("no space left on device", "disk quota exceeded")
+
+#: Consecutive non-finite losses before a run is stopped as `nan_loss`. One
+#: NaN step poisons every LoRA parameter through the optimizer, so every step
+#: after it is NaN too; a short streak is enough to be sure and saves hours.
+NAN_LOSS_ABORT_STEPS = 10
+
+
+class NonFiniteLoss(RuntimeError):
+ """The loss stopped being a number; the adapter is no longer trainable."""
+
+
+class CaptionCheckError(RuntimeError):
+ """The captions do not carry the trigger the job was started with."""
+
+
+def _chain(exc: BaseException):
+ seen = set()
+ while exc is not None and id(exc) not in seen:
+ seen.add(id(exc))
+ yield exc
+ exc = exc.__cause__ or exc.__context__
+
+
+def classify_exception(exc: BaseException | None) -> str:
+ """One of TRAIN_ERROR_CLASSES for the exception that stopped a run.
+
+ The whole cause chain is read (the dataset loader wraps a missing file in
+ a RuntimeError), most specific signal first."""
+ if exc is None:
+ return "other"
+ links = list(_chain(exc))
+ for e in links:
+ if isinstance(e, NonFiniteLoss):
+ return "nan_loss"
+ if isinstance(e, CaptionCheckError):
+ return "caption_check"
+ for e in links:
+ if isinstance(e, MemoryError):
+ return "oom"
+ if isinstance(e, OSError) and e.errno in (errno.ENOSPC, errno.EDQUOT):
+ return "disk_full"
+ for e in links:
+ text = str(e).lower()
+ if any(f in text for f in _DISK_FRAGMENTS):
+ return "disk_full"
+ if any(f in text for f in _OOM_FRAGMENTS):
+ return "oom"
+ for e in links:
+ if isinstance(e, ImportError):
+ return "import_error"
+ if isinstance(e, FileNotFoundError):
+ return "missing_file"
+ # huggingface_hub reads a local model folder that does not exist as a
+ # malformed repo id: the shape of a missing LTX-2.3 pack before 4.19.1.
+ if (type(e).__name__ == "HFValidationError"
+ and "repo id must be in the form" in str(e).lower()):
+ return "missing_file"
+ return "other"
+
+
+def is_finite_loss(value) -> bool:
+ try:
+ return math.isfinite(float(value))
+ except (TypeError, ValueError):
+ return False
diff --git a/lora_lab/train.py b/lora_lab/train.py
index 93c840f..e89576c 100644
--- a/lora_lab/train.py
+++ b/lora_lab/train.py
@@ -160,6 +160,32 @@ def _patch_loader_prefer_dev_transformer() -> None:
ml_module.load_transformer = patched
+def training_weights_problem(model_dir: str | Path) -> str | None:
+ """What is missing for a training run, in one sentence, or None (4.19.1).
+
+ Checked before the crop and the caption encode. Without it a missing
+ LTX-2.3 base pack died ~10 s in inside the preprocessor's Hugging Face
+ lookup ("Repo id must be in the form 'repo_name'..."), and a missing
+ full-precision dev transformer died only after minutes of caption and
+ image encoding - both as a bare "exited with code 1". The Train tab shows
+ a download banner for these, but Start was never blocked by it.
+ Same locations and size floor as the loader patch above."""
+ model_dir = Path(model_dir)
+ if not model_dir.is_dir():
+ return (f"the LTX-2.3 base pack is not downloaded (no {model_dir.name}/ in the models "
+ "folder). Open the Train tab and click Download all, then start again.")
+ full_dev_min = 15 * 1000**3
+ for cand in (model_dir / "transformer-dev.safetensors",
+ model_dir.parent / "ltx-2.3-mlx-q8" / "transformer-dev.safetensors"):
+ try:
+ if cand.is_file() and cand.stat().st_size >= full_dev_min:
+ return None
+ except OSError:
+ continue
+ return ("the full-precision LTX-2.3 dev transformer (~21 GB) is not downloaded, or "
+ "only partly. Open the Train tab and click Download all, then start again.")
+
+
def _patch_compute_video_positions_fps_kwarg() -> None:
"""Translate ``fps=`` → ``frame_rate=`` across the trainer.
diff --git a/lora_lab/train_audio.py b/lora_lab/train_audio.py
index 3db05ff..1eb5cbb 100644
--- a/lora_lab/train_audio.py
+++ b/lora_lab/train_audio.py
@@ -72,6 +72,9 @@ from typing import Any
import yaml
+from lora_lab.failure import (NAN_LOSS_ABORT_STEPS, NonFiniteLoss, classify_exception,
+ is_finite_loss)
+
logger = logging.getLogger(__name__)
@@ -117,8 +120,12 @@ def emit(event: str, **kwargs: Any) -> None:
_REAL_STDOUT.flush()
-def emit_error_and_exit(stage: str, message: str, code: int = 1) -> None:
- emit("error", stage=stage, message=message)
+def emit_error_and_exit(stage: str, message: str, code: int = 1,
+ exc: BaseException | None = None) -> None:
+ # `error_class` is the closed word the panel may report (lora_lab.failure);
+ # `message` stays on this Mac (the Logs tab and <job>/train.log).
+ emit("error", stage=stage, message=message,
+ error_class=classify_exception(exc))
sys.exit(code)
@@ -368,13 +375,7 @@ def run_training(
# The audio-attn exclusion filter inside train.py drops audio_* matches —
# but here we WANT those targets, so we deliberately re-monkey-patch the
# filter AFTER lab_train's patch to restore the audio paths.
- lab_train._patch_loader_prefer_dev_transformer()
- lab_train._patch_strategy_for_image_only()
- lab_train._patch_compute_video_positions_fps_kwarg()
- # The voice adapter saves through the same _save_checkpoint, so without
- # this it is written scrambled exactly like the character one (#62).
- lab_train._patch_contiguous_checkpoint_save()
- _restore_audio_lora_targets()
+ _apply_trainer_patches(lab_train)
total_steps = int(config_dict["optimization"]["steps"])
emit("train_start", total_steps=total_steps)
@@ -390,12 +391,25 @@ def run_training(
_orig_update = tp_mod.TrainingProgress.update_training
+ nonfinite_streak = 0
+
def _sniff_update(self, *, loss, lr, step_time, advance=True): # noqa: ANN001
- nonlocal last_loss
+ nonlocal last_loss, nonfinite_streak
try:
- last_loss = float(loss)
+ value = float(loss)
except Exception: # noqa: BLE001
- pass
+ value = None
+ if value is not None:
+ if is_finite_loss(value):
+ last_loss = value
+ nonfinite_streak = 0
+ else:
+ # Same stop as the face phase (lora_lab.failure, 4.19.1).
+ nonfinite_streak += 1
+ if nonfinite_streak >= NAN_LOSS_ABORT_STEPS:
+ raise NonFiniteLoss(
+ f"the voice training loss stopped being a number for "
+ f"{nonfinite_streak} steps in a row")
return _orig_update(self, loss=loss, lr=lr, step_time=step_time, advance=advance)
tp_mod.TrainingProgress.update_training = _sniff_update
@@ -444,18 +458,42 @@ def _restore_audio_lora_targets() -> None:
The face-side patch drops any LoRA target whose path contains
``audio_attn`` or ``audio_ff`` — that's correct for face training (random
gradients corrupt the audio path). For AUDIO training those are exactly
- the targets we want to keep, so we re-bind ``_find_lora_targets`` to the
- underlying upstream implementation.
+ the targets we want to keep, so ``_find_lora_targets`` goes back to the
+ upstream implementation.
+
+ 4.19.1: this used to ``importlib.reload`` the trainer module. A reload
+ re-runs ``from safetensors.numpy import save_file as save_safetensors``,
+ which silently threw away the #62 contiguous-save patch applied just
+ before it — so every voice trained on 4.19.0 was still written scrambled.
+ The face patch keeps the upstream function (``_phos_orig_find_lora_targets``)
+ and switching it off rebinds that, touching nothing else.
"""
- import ltx_trainer_mlx.trainer as _trainer
+ from lora_lab import train as lab_train
- # The original was captured via closure in the face patch; we can't reach
- # back into it, so re-import the upstream module fresh.
- import importlib
- importlib.reload(_trainer)
+ lab_train._patch_lora_target_exclude_audio(enabled=False)
logger.info("audio LoRA: restored upstream _find_lora_targets (audio paths preserved)")
+def _apply_trainer_patches(lab_train: Any) -> None:
+ """The patch set train.py / train_character.py apply, for a voice run.
+
+ The audio-target restore runs BEFORE the contiguous save, and the save
+ patch is checked afterwards: a voice adapter saved without it is
+ scrambled (#62), and nothing downstream would notice."""
+ import ltx_trainer_mlx.trainer as _trainer
+
+ lab_train._patch_loader_prefer_dev_transformer()
+ lab_train._patch_strategy_for_image_only()
+ lab_train._patch_compute_video_positions_fps_kwarg()
+ _restore_audio_lora_targets()
+ # The voice adapter saves through the same _save_checkpoint, so without
+ # this it is written scrambled exactly like the character one (#62).
+ lab_train._patch_contiguous_checkpoint_save()
+ if getattr(_trainer.save_safetensors, "__name__", "") != "save_contiguous":
+ raise RuntimeError("the voice trainer's checkpoint save is not the "
+ "contiguous one; refusing to write a scrambled voice (#62)")
+
+
# ----------------------------------------------------------------------
# Phase 5: discover the final checkpoint + copy to output_path
# ----------------------------------------------------------------------
@@ -545,7 +583,7 @@ def run_pipeline(spec_path: Path) -> int:
)
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit_error_and_exit("preprocess", str(exc))
+ emit_error_and_exit("preprocess", str(exc), exc=exc)
# ---- phase 3: generate audio YAML ----
audio_train_out_dir = dataset_dir / "audio_train_output"
@@ -562,7 +600,7 @@ def run_pipeline(spec_path: Path) -> int:
emit("log", msg=f"wrote audio config: {audio_yaml_path}")
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit_error_and_exit("config", str(exc))
+ emit_error_and_exit("config", str(exc), exc=exc)
# ---- phase 4: train ----
try:
@@ -572,7 +610,7 @@ def run_pipeline(spec_path: Path) -> int:
)
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit_error_and_exit("train", str(exc))
+ emit_error_and_exit("train", str(exc), exc=exc)
# ---- phase 5: publish final checkpoint ----
# trainer.train() already returns the last-saved checkpoint path; double-
@@ -582,13 +620,13 @@ def run_pipeline(spec_path: Path) -> int:
if not checkpoint_path.exists():
checkpoint_path = find_final_checkpoint(audio_train_out_dir, audio_steps)
except Exception as exc: # noqa: BLE001
- emit_error_and_exit("publish", f"failed to locate checkpoint: {exc}")
+ emit_error_and_exit("publish", f"failed to locate checkpoint: {exc}", exc=exc)
output_path.parent.mkdir(parents=True, exist_ok=True)
try:
shutil.copy2(str(checkpoint_path), str(output_path))
except Exception as exc: # noqa: BLE001
- emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}")
+ emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}", exc=exc)
# Same honesty the image LoRA gets: say how much delta the voice adapter
# carries. #62 reported the voice LoRA "equally ignored", and a run that
@@ -634,7 +672,8 @@ def main(argv: list[str] | None = None) -> int:
raise
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit("error", stage="unknown", message=str(exc))
+ emit("error", stage="unknown", message=str(exc),
+ error_class=classify_exception(exc))
return 1
diff --git a/lora_lab/train_character.py b/lora_lab/train_character.py
index 33f2a01..7e48673 100644
--- a/lora_lab/train_character.py
+++ b/lora_lab/train_character.py
@@ -107,6 +107,9 @@ from typing import Any
import yaml
+from lora_lab.failure import (NAN_LOSS_ABORT_STEPS, NonFiniteLoss, classify_exception,
+ is_finite_loss)
+
logger = logging.getLogger(__name__)
@@ -332,8 +335,12 @@ def emit(event: str, **kwargs: Any) -> None:
_REAL_STDOUT.flush()
-def emit_error_and_exit(stage: str, message: str, code: int = 1) -> None:
- emit("error", stage=stage, message=message)
+def emit_error_and_exit(stage: str, message: str, code: int = 1,
+ exc: BaseException | None = None) -> None:
+ # `error_class` is the closed word the panel may report (lora_lab.failure);
+ # `message` stays on this Mac (the Logs tab and <job>/train.log).
+ emit("error", stage=stage, message=message,
+ error_class=classify_exception(exc))
sys.exit(code)
@@ -825,14 +832,31 @@ def run_training(
_orig_update = tp_mod.TrainingProgress.update_training
+ nonfinite_streak = 0
+
def _sniff_update(self, *, loss, lr, step_time, advance=True): # noqa: ANN001
- nonlocal last_loss, loss_sum, loss_n
+ nonlocal last_loss, loss_sum, loss_n, nonfinite_streak
try:
- last_loss = float(loss)
- loss_sum += last_loss
- loss_n += 1
+ value = float(loss)
except Exception: # noqa: BLE001
- pass
+ value = None
+ if value is not None:
+ if is_finite_loss(value):
+ last_loss = value
+ loss_sum += value
+ loss_n += 1
+ nonfinite_streak = 0
+ else:
+ # One NaN step poisons every LoRA parameter through the
+ # optimizer; every step after it is NaN as well. Stop instead
+ # of spending hours writing a file of NaNs (4.19.1: the fleet
+ # could not tell this from any other "exit 1").
+ nonfinite_streak += 1
+ if nonfinite_streak >= NAN_LOSS_ABORT_STEPS:
+ raise NonFiniteLoss(
+ f"the training loss stopped being a number for "
+ f"{nonfinite_streak} steps in a row - the adapter "
+ f"cannot learn from here")
return _orig_update(self, loss=loss, lr=lr, step_time=step_time, advance=advance)
tp_mod.TrainingProgress.update_training = _sniff_update
@@ -1026,6 +1050,13 @@ def run_pipeline(spec_path: Path) -> int:
emit("start", job_id=job_id)
+ # Weights first: a missing pack is a ten-second answer, not a failure
+ # minutes into the caption encode (4.19.1, lora_lab.train).
+ from lora_lab import train as lab_train
+ _missing = lab_train.training_weights_problem(DEFAULT_MODEL_PATH)
+ if _missing:
+ emit_error_and_exit("models", _missing, exc=FileNotFoundError(_missing))
+
# Resolve preset + overrides.
cfg = resolve_preset(preset, advanced)
cfg["preset_name"] = preset
@@ -1099,14 +1130,14 @@ def run_pipeline(spec_path: Path) -> int:
try:
source_files = list_source_images(images_dir)
except Exception as exc: # noqa: BLE001
- emit_error_and_exit("ingest", str(exc))
+ emit_error_and_exit("ingest", str(exc), exc=exc)
image_count = len(source_files)
try:
recipe = resolve_recipe(cfg)
except ValueError as exc:
- emit_error_and_exit("config", str(exc))
+ emit_error_and_exit("config", str(exc), exc=exc)
estimated_wall_s = estimate_wall_seconds(image_count, preset, advanced)
emit(
@@ -1147,7 +1178,7 @@ def run_pipeline(spec_path: Path) -> int:
)
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit_error_and_exit("crop", str(exc))
+ emit_error_and_exit("crop", str(exc), exc=exc)
try:
run_preprocess(
@@ -1160,7 +1191,7 @@ def run_pipeline(spec_path: Path) -> int:
)
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit_error_and_exit("preprocess", str(exc))
+ emit_error_and_exit("preprocess", str(exc), exc=exc)
try:
checkpoint_path, training_wall_s = run_training(
@@ -1171,14 +1202,14 @@ def run_pipeline(spec_path: Path) -> int:
)
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit_error_and_exit("train", str(exc))
+ emit_error_and_exit("train", str(exc), exc=exc)
# Move the checkpoint to the spec's output path.
output_lora_path.parent.mkdir(parents=True, exist_ok=True)
try:
shutil.copy2(str(checkpoint_path), str(output_lora_path))
except Exception as exc: # noqa: BLE001
- emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}")
+ emit_error_and_exit("publish", f"failed to copy checkpoint: {exc}", exc=exc)
# Measure before the sidecar so the strength verdict is on record even if
# sidecar writing fails — the number is the part a stuck user needs.
@@ -1194,7 +1225,7 @@ def run_pipeline(spec_path: Path) -> int:
strength=strength,
)
except Exception as exc: # noqa: BLE001
- emit_error_and_exit("sidecar", str(exc))
+ emit_error_and_exit("sidecar", str(exc), exc=exc)
emit("done")
return 0
@@ -1225,7 +1256,8 @@ def main(argv: list[str] | None = None) -> int:
raise
except Exception as exc: # noqa: BLE001
traceback.print_exc(file=sys.stderr)
- emit("error", stage="unknown", message=str(exc))
+ emit("error", stage="unknown", message=str(exc),
+ error_class=classify_exception(exc))
return 1
diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.py
index 1414323..cf006ce 100644
--- a/mlx_ltx_panel.py
+++ b/mlx_ltx_panel.py
@@ -21,6 +21,7 @@ import io
import json
import math
import collections
+import errno
import os
import random
import re
@@ -18679,6 +18680,11 @@ def _analytics_render_event(job: dict) -> None:
props["error_signature"] = scrubbed
if props["error_class"] == "other":
props["error_fingerprint"] = _analytics_error_fingerprint(scrubbed)
+ if mode == "train":
+ # 4.19.1: why a TRAINING run failed, from the trainer's
+ # own exception (lora_lab.failure) — never its text.
+ te = str(job.get("train_error") or "")
+ props["train_error"] = te if te in TRAIN_ERROR_CLASSES else "other"
_analytics_capture(event, props)
except Exception:
pass
@@ -33136,6 +33142,45 @@ def _train_wipe_caption_conditions(dataset_dir: Path) -> int:
return n
+#: Why a training job failed, one word (docs/ANALYTICS.md `train_error`).
+#: Mirrors lora_lab.failure.TRAIN_ERROR_CLASSES, which the trainer classifies
+#: into; the panel never forwards a word outside this list.
+TRAIN_ERROR_CLASSES = ("oom", "missing_file", "caption_check", "disk_full",
+ "import_error", "nan_loss", "other")
+
+
+def train_error_from_event(payload: dict) -> dict:
+ """The trainer's `error` event, its class held to the closed list."""
+ cls = str(payload.get("error_class") or "other")
+ return {
+ "stage": str(payload.get("stage") or "unknown")[:40],
+ "message": str(payload.get("message") or "")[:2000],
+ "error_class": cls if cls in TRAIN_ERROR_CLASSES else "other",
+ }
+
+
+def train_error_for_exit(rc: int, trainer_error: dict | None) -> str:
+ """The class of a trainer that exited non-zero. The trainer's own word
+ wins; a process killed by SIGKILL without one is macOS reclaiming memory
+ (jetsam), the one signal death that has a single cause."""
+ if trainer_error and trainer_error.get("error_class") in TRAIN_ERROR_CLASSES:
+ return trainer_error["error_class"]
+ if rc == -9:
+ return "oom"
+ return "other"
+
+
+def _train_release_render_memory() -> None:
+ """Stop the warm render helper before a training run (4.19.1)."""
+ try:
+ if HELPER.is_alive():
+ push("[train] releasing the render engine's memory for training "
+ "(it reloads on the next render)")
+ HELPER.kill()
+ except Exception as e: # never let housekeeping fail a training run
+ push(f"[train] could not release the render engine: {e}")
+
+
def run_train_job_inner(job: dict) -> None:
"""Run a Train Character job. Reads the dataset that /train/upload built
under state/train_character/<train_job_id>/, writes a job-spec JSON the
@@ -33179,6 +33224,7 @@ def run_train_job_inner(job: dict) -> None:
images_dir = dataset_dir / "images"
captions_dir = dataset_dir / "captions"
if not images_dir.is_dir():
+ job["train_error"] = "missing_file"
raise RuntimeError(
f"dataset images not found: {images_dir} — "
"/train/upload should have created this. Re-upload your dataset.")
@@ -33299,6 +33345,7 @@ def run_train_job_inner(job: dict) -> None:
f"({other[1]}/{len(mismatched)} mismatched files)"
if other else "")
if len(mismatched) == len(user_cap_files):
+ job["train_error"] = "caption_check"
raise RuntimeError(
f"none of the {len(user_cap_files)} existing caption files "
f"contain the trigger {trigger!r} this job was submitted "
@@ -33387,9 +33434,12 @@ def run_train_job_inner(job: dict) -> None:
try:
spec_path.write_text(json.dumps(spec, indent=2), encoding="utf-8")
except OSError as e:
+ job["train_error"] = ("disk_full" if e.errno in (errno.ENOSPC, errno.EDQUOT)
+ else "other")
raise RuntimeError(f"could not write spec.json: {e}")
if not LORA_LAB_RUN_SH.exists():
+ job["train_error"] = "missing_file"
raise RuntimeError(
f"lora-lab shim not found at {LORA_LAB_RUN_SH}. "
"Train Character depends on the sibling lab at "
@@ -33427,6 +33477,11 @@ def run_train_job_inner(job: dict) -> None:
_train_phase = "start"
_train_env = trainer_child_env()
_encode_retried = False
+ _trainer_error: dict | None = None
+ # A warm render helper keeps an LTX model resident (up to tens of GB) for
+ # HELPER_IDLE_TIMEOUT after the last render. The trainer needs that memory
+ # for the dev transformer and its activations; give it back first.
+ _train_release_render_memory()
try:
while True:
proc = subprocess.Popen(
@@ -33520,7 +33575,20 @@ def run_train_job_inner(job: dict) -> None:
if last_step % 50 == 0 or last_step == total_steps:
push(f"[train] step {last_step}/{total_steps} loss={last_loss}")
elif evt == "log":
- push(f"[train] {payload.get('msg', '')}")
+ # The trainer writes its log lines as `line`; `msg` is
+ # the older key. Read both (4.19.0 showed an empty
+ # "[train] " for the recipe line).
+ push(f"[train] {payload.get('msg') or payload.get('line') or ''}")
+ elif evt == "error":
+ # The trainer's own account of why it stopped: the
+ # stage, its message (stays on this Mac) and a closed
+ # class (lora_lab.failure) — the only part analytics
+ # may carry. Until 4.19.1 this fell through to the
+ # generic branch and the class was lost: every one of
+ # these reached the fleet as "exited with code 1".
+ _trainer_error = train_error_from_event(payload)
+ push(f"[train] stopped in {_trainer_error['stage']}: "
+ f"{_trainer_error['message']} ({_trainer_error['error_class']})")
elif evt == "adapter_strength":
# A GREEN TALLY WAS NEVER PROOF THE FILE COULD DO
# ANYTHING. A LoRA can train to completion, write a
@@ -33605,10 +33673,12 @@ def run_train_job_inner(job: dict) -> None:
"trainer process (AGX_RELAX_CDM_CTXSTORE_TIMEOUT=1) — the "
"screen may feel less responsive until it finishes.")
_train_watchdog_seen = False
+ _trainer_error = None
_train_phase = "start"
continue
break
if rc != 0:
+ job["train_error"] = train_error_for_exit(rc, _trainer_error)
# -6 is SIGABRT. With a watchdog signature in the log it is macOS
# killing the GPU command buffer, not our code faulting — and the
# lever is the canvas, not the preset. Reported in #61: 576x576
@@ -33653,6 +33723,7 @@ def run_train_job_inner(job: dict) -> None:
elapsed = round(time.time() - t0, 2)
if not final_lora_path.is_file():
+ job["train_error"] = "missing_file"
raise RuntimeError(
f"training finished but output .safetensors missing at {final_lora_path}")
diff --git a/scripts/test_convert_ltx_mlx.py b/scripts/test_convert_ltx_mlx.py
index 35146e4..bfdc42c 100644
--- a/scripts/test_convert_ltx_mlx.py
+++ b/scripts/test_convert_ltx_mlx.py
@@ -453,7 +453,26 @@ class TestEndToEnd:
on_disk = json.loads((out_dir / "config.json").read_text())
assert on_disk["transformer"]["ff_bias"] is False
- def test_the_vendored_config_reader_accepts_the_result(self, tmp_path):
+ @pytest.fixture
+ def _own_ltx_core(self):
+ """The port checkout goes on sys.path for THIS test only (4.19.1).
+
+ It used to stay there, and whichever later test imported an
+ ltx_core_mlx submodule for the first time got the port's package
+ instead of the venv's - the full suite failed four trainer tests with
+ "cannot import name 'utils' from 'ltx_core_mlx' (.../ltx25-port/...)"
+ on any Mac that has the port checkout."""
+ saved_path = list(sys.path)
+ saved = {k: v for k, v in sys.modules.items()
+ if k == "ltx_core_mlx" or k.startswith("ltx_core_mlx.")}
+ yield
+ sys.path[:] = saved_path
+ for k in [k for k in sys.modules if k == "ltx_core_mlx" or k.startswith("ltx_core_mlx.")]:
+ if k not in saved:
+ del sys.modules[k]
+ sys.modules.update(saved)
+
+ def test_the_vendored_config_reader_accepts_the_result(self, tmp_path, _own_ltx_core):
"""The real proof: our pack is readable by the code that will load it.
Needs an ``ltx_core_mlx`` that carries the 2.5 config work (the
diff --git a/test_codex_4170_safety.py b/test_codex_4170_safety.py
index bae1aac..17eaa6e 100644
--- a/test_codex_4170_safety.py
+++ b/test_codex_4170_safety.py
@@ -338,10 +338,14 @@ def test_safety_4_a_queued_h3_job_restores_its_engine_and_h3_recipe(monkeypatch)
assert out["prompt"] == "a hen skates"
-def test_safety_4_a_queued_ltx_job_restores_steps_and_export_method():
+def test_safety_4_a_queued_ltx_job_restores_steps_and_export_method(monkeypatch):
# 4.17.4: on a DISTILLED quality make_job now clamps steps to 8 (the lane
# cannot run anything else - clamp_distilled_steps), so a restorable,
# non-default count has to live on an HQ quality, where `steps` is kept.
+ # 4.19.1: as a Mac WITH the PiperSR upscaler (#90 class) - make_job folds
+ # pipersr to lanczos where it is not installed, which is every fresh
+ # install, so this read the machine instead of testing the restore.
+ monkeypatch.setattr(p, "PIPERSR_UPSCALE_ENABLED", True)
ltx = p.make_job({"mode": "t2v", "engine": "ltx", "prompt": "a fox", "steps": "12",
"upscale": "fit_720p", "upscale_method": "pipersr", "quality": "high"})["params"]
assert ltx["steps"] == 12
diff --git a/test_music_engine.py b/test_music_engine.py
index 14c8e36..356dcf1 100644
--- a/test_music_engine.py
+++ b/test_music_engine.py
@@ -302,7 +302,7 @@ def test_fetch_stages_moves_and_skips_intact(pack, monkeypatch):
def test_runner_help_and_empty_input():
- music_python = Path.home() / "AI/projects/yue2-mlx/.venv/bin/python"
+ music_python = ROOT / Path(os.environ.get("LTX_MUSIC_ROOT", "yue2-mlx")).expanduser() / ".venv/bin/python"
runner = ROOT / "scripts/music/yue2_run.py"
result = subprocess.run([sys.executable, str(runner), "--model-dir", "/missing", "--vae-dir", "/missing", "--output", "/tmp/no-song.wav"], capture_output=True, text=True, timeout=30)
assert result.returncode == 2 and "Give it something" in result.stdout
diff --git a/test_review_416_ltx.py b/test_review_416_ltx.py
index 0f36ec9..49e5da6 100644
--- a/test_review_416_ltx.py
+++ b/test_review_416_ltx.py
@@ -340,10 +340,21 @@ class TheHelperAttachesThem(unittest.TestCase):
for name, attrs in fake_modules.items():
mod = types.ModuleType(name)
mod.__dict__.update(attrs)
+ # Put back what was there, not nothing (4.19.1): popping a stand-in
+ # for `ltx_core_mlx` dropped the REAL package, and the next
+ # `import ltx_core_mlx.utils.positions as m` in the same process
+ # (the trainer tests) got a fresh package with no `utils` on it.
+ self.addCleanup(self._restore_module, name, sys.modules.get(name))
sys.modules[name] = mod
- self.addCleanup(sys.modules.pop, name, None)
return ns
+ @staticmethod
+ def _restore_module(name, prev):
+ if prev is None:
+ sys.modules.pop(name, None)
+ else:
+ sys.modules[name] = prev
+
def test_fflf_attaches_and_rebuilds_on_a_new_set(self):
built = []
diff --git a/test_train_failure_class.py b/test_train_failure_class.py
new file mode 100644
index 0000000..a45d6da
--- /dev/null
+++ b/test_train_failure_class.py
@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""A failed training run says WHY, as one closed word (4.19.1).
+
+4.19.0 had nine failed trainings on two Macs and every one reached the fleet
+as "training exited with code 1 — see Logs for stack trace": memory, a
+missing file, a full disk and a broken venv all looked the same. The trainer
+held the exception and emitted an `error` event with its stage and message,
+but the panel dropped that event into the generic log branch.
+
+Pinned here:
+ * lora_lab.failure.classify_exception — type first, then MLX's fixed
+ allocation-failure fragments, through the cause chain;
+ * the trainer's `error` event carries `error_class`;
+ * the panel keeps only a word from the closed list (an unknown or
+ path-shaped value becomes `other`), a SIGKILL with no word is `oom`,
+ and its own pre-trainer refusals name their class;
+ * render_failed carries `train_error` for training jobs only;
+ * the job's error text is unchanged, so the error_class series is too.
+The panel half drives the REAL run_train_job_inner against a stand-in
+trainer script; no weights, no GPU.
+"""
+from __future__ import annotations
+
+import errno
+import io
+import json
+import os
+import stat
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+ROOT = Path(__file__).resolve().parent
+os.environ["LTX_STATE_DIR"] = tempfile.mkdtemp(prefix="phos-train-fail-")
+os.environ["PHOSPHENE_ANALYTICS_DISABLED"] = "1"
+os.environ["PHOSPHENE_DISABLE_VERSION_CHECK"] = "1"
+os.environ.setdefault("LTX_PORT", "8316")
+sys.path.insert(0, str(ROOT))
+
+import mlx_ltx_panel as P # noqa: E402
+from lora_lab import failure as F # noqa: E402
+
+
+class ClassifyException(unittest.TestCase):
+ def test_closed_list_matches_the_panel(self) -> None:
+ self.assertEqual(F.TRAIN_ERROR_CLASSES, P.TRAIN_ERROR_CLASSES)
+
+ def test_mlx_allocation_failures_are_oom(self) -> None:
+ for text in (
+ "[metal::malloc] Attempting to allocate 21474836480 bytes which is greater "
+ "than the maximum allowed buffer size of 17179869184 bytes.",
+ "[metal::malloc] Resource limit (499000) exceeded.",
+ "[METAL] Command buffer execution failed: Insufficient Memory "
+ "(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)",
+ ):
+ self.assertEqual(F.classify_exception(RuntimeError(text)), "oom", text)
+ self.assertEqual(F.classify_exception(MemoryError()), "oom")
+
+ def test_disk_full(self) -> None:
+ self.assertEqual(F.classify_exception(OSError(errno.ENOSPC, "x")), "disk_full")
+ self.assertEqual(F.classify_exception(
+ RuntimeError("Error while serializing: IoError(Os { code: 28, kind: "
+ "StorageFull, message: \"No space left on device\" })")), "disk_full")
+
+ def test_missing_file_through_the_dataset_loaders_wrapper(self) -> None:
+ try:
+ try:
+ raise FileNotFoundError(2, "No such file or directory")
+ except FileNotFoundError as e:
+ raise RuntimeError("Failed to load latents from /x: missing") from e
+ except RuntimeError as wrapped:
+ self.assertEqual(F.classify_exception(wrapped), "missing_file")
+ self.assertEqual(F.classify_exception(FileNotFoundError(
+ "transformer-dev.safetensors not found in /a or /b")), "missing_file")
+
+ def test_import_nan_caption_other(self) -> None:
+ self.assertEqual(F.classify_exception(ModuleNotFoundError("No module named 'x'")),
+ "import_error")
+ self.assertEqual(F.classify_exception(F.NonFiniteLoss("nan")), "nan_loss")
+ self.assertEqual(F.classify_exception(F.CaptionCheckError("x")), "caption_check")
+ self.assertEqual(F.classify_exception(ValueError("bad")), "other")
+ self.assertEqual(F.classify_exception(None), "other")
+
+ def test_a_path_or_trigger_never_becomes_the_class(self) -> None:
+ cls = F.classify_exception(RuntimeError("/Users/someone/zqxtrn_v2.safetensors"))
+ self.assertIn(cls, F.TRAIN_ERROR_CLASSES)
+ self.assertEqual(cls, "other")
+
+
+class MissingTrainingWeights(unittest.TestCase):
+ """The panel e2e on a fresh 4.19.1 install with no LTX-2.3 pack: the run
+ died 9 s in with HFValidationError ("Repo id must be in the form ...", the
+ pack's own path read as a repo id) and the class came out `other`."""
+
+ def test_the_hub_shape_of_a_missing_pack_is_missing_file(self) -> None:
+ HFValidationError = type("HFValidationError", (ValueError,), {})
+ e = HFValidationError("Repo id must be in the form 'repo_name' or "
+ "'namespace/repo_name': '/x/mlx_models/ltx-2.3-mlx-q4'.")
+ self.assertEqual(F.classify_exception(e), "missing_file")
+
+ def test_what_is_missing_is_said_before_any_work(self) -> None:
+ try:
+ from lora_lab import train as lab_train
+ except ImportError:
+ self.skipTest("needs the trainer venv")
+ with tempfile.TemporaryDirectory() as td:
+ models = Path(td)
+ q4 = models / "ltx-2.3-mlx-q4"
+ self.assertIn("base pack", lab_train.training_weights_problem(q4))
+ q4.mkdir()
+ self.assertIn("dev transformer", lab_train.training_weights_problem(q4))
+ small = q4 / "transformer-dev.safetensors" # the quantized 11 GB copy (#35)
+ with open(small, "wb") as f:
+ f.truncate(11 * 1000**3)
+ self.assertIn("dev transformer", lab_train.training_weights_problem(q4))
+ (models / "ltx-2.3-mlx-q8").mkdir()
+ with open(models / "ltx-2.3-mlx-q8" / "transformer-dev.safetensors", "wb") as f:
+ f.truncate(21 * 1000**3) # sparse: no disk used
+ self.assertIsNone(lab_train.training_weights_problem(q4))
+
+ def test_the_trainer_stops_at_once_with_the_class(self) -> None:
+ try:
+ import lora_lab.train_character as tc
+ except ImportError:
+ self.skipTest("needs the trainer venv")
+ with tempfile.TemporaryDirectory() as td:
+ spec = Path(td) / "spec.json"
+ spec.write_text(json.dumps({"job_id": "j", "trigger": "zqxtrn", "preset": "quick",
+ "images_dir": str(Path(td) / "images"),
+ "output_path": str(Path(td) / "out.safetensors")}))
+ buf = io.StringIO()
+ with mock.patch.object(tc, "DEFAULT_MODEL_PATH", str(Path(td) / "ltx-2.3-mlx-q4")), \
+ mock.patch.object(tc, "_REAL_STDOUT", buf), self.assertRaises(SystemExit):
+ tc.run_pipeline(spec)
+ ev = json.loads(buf.getvalue().strip().splitlines()[-1])
+ self.assertEqual((ev["event"], ev["stage"], ev["error_class"]),
+ ("error", "models", "missing_file"))
+
+
+class TrainerErrorEvent(unittest.TestCase):
+ def _emitted(self, mod, **kw) -> dict:
+ buf = io.StringIO()
+ with mock.patch.object(mod, "_REAL_STDOUT", buf), self.assertRaises(SystemExit):
+ mod.emit_error_and_exit(**kw)
+ return json.loads(buf.getvalue().strip().splitlines()[-1])
+
+ def test_face_and_voice_trainers_send_the_class(self) -> None:
+ import lora_lab.train_audio as ta
+ import lora_lab.train_character as tc
+ for mod in (tc, ta):
+ ev = self._emitted(mod, stage="train", message="m",
+ exc=RuntimeError("[metal::malloc] Resource limit (499000) exceeded."))
+ self.assertEqual((ev["event"], ev["stage"], ev["error_class"]), ("error", "train", "oom"))
+ ev = self._emitted(mod, stage="config", message="m")
+ self.assertEqual(ev["error_class"], "other")
+
+
+def _fake_trainer(body: str) -> Path:
+ d = Path(tempfile.mkdtemp(prefix="phos-fake-trainer-"))
+ sh = d / "lora_lab_run.sh"
+ sh.write_text("#!/bin/bash\n" + body + "\n")
+ sh.chmod(sh.stat().st_mode | stat.S_IEXEC)
+ return sh
+
+
+class PanelKeepsTheWord(unittest.TestCase):
+ TRIGGER = "zqxtrn"
+
+ def _job(self, sh: Path) -> dict:
+ from PIL import Image
+ jid = "tfail" + os.urandom(4).hex()
+ images = P._train_dataset_dir(jid) / "images"
+ images.mkdir(parents=True, exist_ok=True)
+ for i in range(P.TRAIN_MIN_IMAGES):
+ Image.new("RGB", (64, 64), (i * 9, 40, 80)).save(images / f"char_{i:03d}.png")
+ self._sh = sh
+ return {"id": "q-" + jid, "params": {"mode": "train", "train_job_id": jid,
+ "trigger": self.TRIGGER, "preset": "quick",
+ "steps": 10, "rank": 4}}
+
+ def _run(self, body: str) -> tuple[dict, str]:
+ sh = _fake_trainer(body)
+ job = self._job(sh)
+ with mock.patch.object(P, "LORA_LAB_RUN_SH", sh):
+ with self.assertRaises(RuntimeError) as cm:
+ P.run_train_job_inner(job)
+ return job, str(cm.exception)
+
+ def test_the_trainers_class_reaches_the_job(self) -> None:
+ ev = {"event": "error", "stage": "train", "error_class": "oom",
+ "message": "[metal::malloc] Resource limit (499000) exceeded."}
+ job, err = self._run(f"echo '{json.dumps(ev)}'\nexit 1")
+ self.assertEqual(job["train_error"], "oom")
+ # Same sentence as 4.19.0: the error_class / error_signature series
+ # stay comparable across the release.
+ self.assertEqual(err, "training exited with code 1 — see Logs for stack trace")
+
+ def test_an_unknown_word_is_not_forwarded(self) -> None:
+ ev = {"event": "error", "stage": "train", "error_class": "/Users/x/zqxtrn", "message": "m"}
+ job, _ = self._run(f"echo '{json.dumps(ev)}'\nexit 1")
+ self.assertEqual(job["train_error"], "other")
+
+ def test_no_event_exit_1_is_other_and_sigkill_is_oom(self) -> None:
+ job, _ = self._run("echo boom\nexit 1")
+ self.assertEqual(job["train_error"], "other")
+ job, _ = self._run("kill -9 $$")
+ self.assertEqual(job["train_error"], "oom")
+
+ def test_the_caption_refusal_names_itself(self) -> None:
+ sh = _fake_trainer("exit 0")
+ job = self._job(sh)
+ caps = P._train_dataset_dir(job["params"]["train_job_id"]) / "captions"
+ caps.mkdir(parents=True, exist_ok=True)
+ for i in range(P.TRAIN_MIN_IMAGES):
+ (caps / f"char_{i:03d}.txt").write_text("[VISUAL]: othertrn, a man\n")
+ with mock.patch.object(P, "LORA_LAB_RUN_SH", sh), self.assertRaises(RuntimeError):
+ P.run_train_job_inner(job)
+ self.assertEqual(job["train_error"], "caption_check")
+
+ def test_the_render_helper_is_released_before_training(self) -> None:
+ killed = []
+ with mock.patch.object(P.HELPER, "is_alive", lambda: True), \
+ mock.patch.object(P.HELPER, "kill", lambda: killed.append(1)):
+ self._run("exit 1")
+ self.assertEqual(killed, [1])
+
+ def test_the_trainers_log_line_is_shown(self) -> None:
+ pushed = []
+ ev = {"event": "log", "line": "recipe: adam_bias_correction=False"}
+ with mock.patch.object(P, "push", lambda s: pushed.append(s)):
+ self._run(f"echo '{json.dumps(ev)}'\nexit 1")
+ self.assertIn("[train] recipe: adam_bias_correction=False", pushed)
+
+
+class TheEventCarriesIt(unittest.TestCase):
+ def _props(self, job: dict) -> dict:
+ sent = []
+ with mock.patch.object(P, "_analytics_capture", lambda e, p: sent.append((e, p))), \
+ mock.patch.object(P, "get_settings", lambda: {"analytics_first_render_reported": True}):
+ P._analytics_render_event(job)
+ return sent[0][1]
+
+ def test_train_failures_only(self) -> None:
+ err = "training exited with code 1 — see Logs for stack trace"
+ p = self._props({"status": "failed", "error": err, "train_error": "oom",
+ "params": {"mode": "train", "width": 512, "height": 512}})
+ self.assertEqual(p["train_error"], "oom")
+ self.assertEqual(p["error_class"], "helper_exit")
+ p = self._props({"status": "failed", "error": err, "train_error": "nonsense",
+ "params": {"mode": "train"}})
+ self.assertEqual(p["train_error"], "other")
+ p = self._props({"status": "failed", "error": "boom", "params": {"mode": "t2v"}})
+ self.assertNotIn("train_error", p)
+ p = self._props({"status": "done", "params": {"mode": "train"}})
+ self.assertNotIn("train_error", p)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/test_train_voice_save_and_nan.py b/test_train_voice_save_and_nan.py
new file mode 100644
index 0000000..6c4bd30
--- /dev/null
+++ b/test_train_voice_save_and_nan.py
@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""Two 4.19.1 trainer fixes, through the real trainer code on a tiny model.
+
+1. VOICES WERE STILL SAVED SCRAMBLED ON 4.19.0 (#62). `train_audio` applied
+ the contiguous-save patch and then "restored" the audio LoRA targets with
+ `importlib.reload(ltx_trainer_mlx.trainer)`. A reload re-runs the module's
+ `from safetensors.numpy import save_file as save_safetensors`, so the patch
+ applied one line earlier was gone and every voice adapter went to disk in
+ the scrambled layout 4.19.0 was released to fix. The restore now rebinds
+ only `_find_lora_targets`, and the voice run refuses to start without the
+ contiguous save.
+
+2. A NaN loss used to train on for hours and finish (or die) as an anonymous
+ "exit 1". After NAN_LOSS_ABORT_STEPS non-finite steps in a row the face
+ trainer stops with NonFiniteLoss, classified `nan_loss`.
+
+CPU only, no weights, no GPU lock. The end-to-end case runs
+`train_character.run_training` itself — every 4.19 patch, the loss sniffer,
+the step callback, the checkpoint save — with only `load_ltx_model` swapped
+for a one-block LTXModel and three precomputed samples.
+"""
+from __future__ import annotations
+
+import io
+import sys
+import tempfile
+import types
+import unittest
+from pathlib import Path
+from unittest import mock
+
+ROOT = Path(__file__).resolve().parent
+sys.path.insert(0, str(ROOT))
+
+try:
+ import mlx.core as mx
+ import numpy as np
+ from mlx_lm.tuner.lora import LoRALinear
+ from safetensors.numpy import load_file
+
+ import lora_lab.train as lab_train
+ import lora_lab.train_audio as ta
+ import lora_lab.train_character as tc
+ import ltx_trainer_mlx.progress as tp_mod
+ import ltx_trainer_mlx.trainer as trainer_mod
+ from lora_lab import failure as F
+ from ltx_core_mlx.model.transformer.model import LTXModel, LTXModelConfig
+ from ltx_trainer_mlx.config import LtxTrainerConfig
+ from ltx_trainer_mlx.trainer import LtxvTrainer
+
+ mx.set_default_device(mx.cpu)
+ HAVE_MLX = True
+except ImportError: # pragma: no cover — the gates run in the trainer venv
+ HAVE_MLX = False
+
+TINY = dict(num_layers=1, video_dim=64, audio_dim=32, video_num_heads=2,
+ audio_num_heads=2, video_head_dim=32, audio_head_dim=16,
+ av_cross_num_heads=2, av_cross_head_dim=16)
+
+
[email protected](HAVE_MLX, "needs the trainer venv (mlx + ltx_trainer_mlx)")
+class VoiceSaveStaysContiguous(unittest.TestCase):
+ def setUp(self) -> None:
+ self._save = getattr(trainer_mod, "_phos_orig_save_safetensors", trainer_mod.save_safetensors)
+ self._targets = getattr(trainer_mod, "_phos_orig_find_lora_targets",
+ trainer_mod._find_lora_targets)
+ self._tmp = tempfile.TemporaryDirectory()
+
+ def tearDown(self) -> None:
+ trainer_mod.save_safetensors = self._save
+ lab_train._patch_contiguous_checkpoint_save()
+ trainer_mod._find_lora_targets = self._targets
+ self._tmp.cleanup()
+
+ def test_restoring_the_audio_targets_keeps_the_contiguous_save(self) -> None:
+ """The 4.19.0 bug in one line: this assertion fails there."""
+ lab_train._patch_contiguous_checkpoint_save()
+ ta._restore_audio_lora_targets()
+ self.assertEqual(trainer_mod.save_safetensors.__name__, "save_contiguous")
+
+ def test_the_voice_patch_set_saves_the_adapter_that_was_trained(self) -> None:
+ lab_train._patch_lora_target_exclude_audio(True) # as a face run left it
+ ta._apply_trainer_patches(lab_train)
+ self.assertIs(trainer_mod._find_lora_targets, self._targets, "audio targets restored")
+ mx.random.seed(0)
+ t = LtxvTrainer.__new__(LtxvTrainer)
+ t._transformer = LTXModel(LTXModelConfig(**TINY))
+ cfg = tc.resolve_preset("high", {"rank": 4, "alpha": 4, "steps": 10})
+ d = tc.build_trainer_config(cfg=cfg, data_root=Path("/d"), output_dir=Path(self._tmp.name))
+ d["model"]["model_path"] = self._tmp.name
+ t._config = LtxTrainerConfig.model_validate(d)
+ t._setup_lora()
+ mem = {}
+ for path, m in t._transformer.named_modules():
+ if isinstance(m, LoRALinear):
+ m.lora_b = mx.random.normal(m.lora_b.shape)
+ mem[path] = (np.array(m.lora_a), np.array(m.lora_b))
+ self.assertTrue(any("audio_attn" in p for p in mem), "voice runs train the audio paths")
+ t._global_step, t._checkpoint_paths = 10, []
+ disk = load_file(str(t._save_checkpoint()))
+ for path, (a, b) in mem.items():
+ np.testing.assert_array_equal(disk[f"diffusion_model.{path}.lora_A.weight"], a.T)
+ np.testing.assert_array_equal(disk[f"diffusion_model.{path}.lora_B.weight"], b.T)
+
+ def test_a_voice_run_without_the_contiguous_save_refuses(self) -> None:
+ with mock.patch.object(lab_train, "_patch_contiguous_checkpoint_save", lambda: None):
+ trainer_mod.save_safetensors = self._save
+ with self.assertRaises(RuntimeError):
+ ta._apply_trainer_patches(lab_train)
+
+
+def _fake_load(model_dir=None, **kw):
+ mx.random.seed(0)
+ return types.SimpleNamespace(transformer=LTXModel(LTXModelConfig(**TINY)),
+ video_vae_decoder=None, video_vae_encoder=None,
+ audio_vae_decoder=None, vocoder=None)
+
+
+def _data(root: Path, nan: bool) -> None:
+ lat = root / ".precomputed" / "latents"
+ con = root / ".precomputed" / "conditions"
+ lat.mkdir(parents=True)
+ con.mkdir(parents=True)
+ for i in range(3):
+ x = mx.random.normal((128, 1, 2, 2))
+ if nan:
+ x = x * float("nan")
+ mx.save_safetensors(str(lat / f"latent_{i:03d}.safetensors"), {
+ "latents": x, "num_frames": mx.array(1), "height": mx.array(2),
+ "width": mx.array(2), "fps": mx.array(24.0)})
+ mx.save_safetensors(str(con / f"condition_{i:03d}.safetensors"), {
+ "video_prompt_embeds": mx.random.normal((8, 64)),
+ "audio_prompt_embeds": mx.random.normal((8, 32)),
+ "prompt_attention_mask": mx.ones((8,))})
+
+
[email protected](HAVE_MLX, "needs the trainer venv (mlx + ltx_trainer_mlx)")
+class FaceTrainingEndToEnd(unittest.TestCase):
+ def setUp(self) -> None:
+ self._tmp = tempfile.TemporaryDirectory()
+ self._update = tp_mod.TrainingProgress.update_training
+ self._stdout = io.StringIO()
+
+ def tearDown(self) -> None:
+ tp_mod.TrainingProgress.update_training = self._update
+ self._tmp.cleanup()
+
+ def _run(self, *, nan: bool, steps: int):
+ root = Path(self._tmp.name)
+ (root / "model").mkdir()
+ _data(root / "data", nan)
+ cfg = tc.resolve_preset("quick", {"rank": 4, "alpha": 4, "steps": steps, "lr": 1e-3})
+ with mock.patch.object(trainer_mod, "load_ltx_model", _fake_load), \
+ mock.patch.object(tc, "DEFAULT_MODEL_PATH", str(root / "model")), \
+ mock.patch.object(tc, "_REAL_STDOUT", self._stdout):
+ return tc.run_training(cfg=cfg, data_root=root / "data",
+ output_dir=root / "out", estimated_wall_s=5)
+
+ def test_a_short_run_trains_and_saves(self) -> None:
+ ck, _ = self._run(nan=False, steps=6)
+ self.assertTrue(Path(ck).is_file())
+ self.assertIn('"event":"train_progress"', self._stdout.getvalue())
+
+ def test_a_nan_loss_stops_the_run_as_nan_loss(self) -> None:
+ with self.assertRaises(F.NonFiniteLoss) as cm:
+ self._run(nan=True, steps=F.NAN_LOSS_ABORT_STEPS + 20)
+ self.assertEqual(F.classify_exception(cm.exception), "nan_loss")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/webapp/style/panel.css b/webapp/style/panel.css
index 3e34906..aade8c1 100644
--- a/webapp/style/panel.css
+++ b/webapp/style/panel.css
@@ -7664,6 +7664,7 @@
.live-stage-stop {
pointer-events: auto;
align-self: center;
+ width: auto; flex: 0 0 auto; /* the panel-wide `button { width:100% }` stretched this over the ETA text */
height: 29px; padding: 0 11px;
border-radius: var(--r-pill);
border: 1px solid rgba(255,255,255,.2);