Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

release(v4.9.2): three issue fixes — resumable image-model downloads (#73), training survives the caption-encode watchdog kill (#61), letters-only trigger words (#62)

mrbizarro17e24ea15fcb
diff --git a/README.md b/README.md
index b1b3cb8..1ede320 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@
</p>
-> **Current release: v4.9.1.** Fixes the buttons that v4.9.0's reorganization silently broke — the gallery card actions (Quality / Animate / delete), LoRA management, character training controls, timeline editing and CivitAI downloads — reported and diagnosed by a contributor within a day (thanks @blackest). Also adds an "×" on the Recent-uploads strip to delete an imported image. Full notes on the [releases page](https://github.com/mrbizarro/phosphene/releases).
+> **Current release: v4.9.2.** Three issue fixes: an interrupted image-model download now resumes instead of crashing every generation (#73), character training survives the GPU watchdog killing caption encoding by relaunching once at a shorter text encode (#61), and suggested trigger words are letters-only — the old `mrz07` shape was half digit tokens (#62). Includes v4.9.1, which fixed the buttons v4.9.0's reorganization silently broke (thanks @blackest) and added an "×" on the Recent-uploads strip. Full notes on the [releases page](https://github.com/mrbizarro/phosphene/releases).
## Overview
diff --git a/VERSION b/VERSION
index 5b341fd..dad10c7 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-4.9.1
+4.9.2
diff --git a/docs/API.md b/docs/API.md
index 4fa7528..a649a7c 100644
--- a/docs/API.md
+++ b/docs/API.md
@@ -330,7 +330,7 @@ Kick off a character LoRA training. The panel shells out to `lora_lab.train_char
| Field | Type | Note |
|---|---|---|
| `train_job_id` | string | From `/train/upload`. |
-| `trigger` | string | Trigger token, e.g. `bizarrotrn`. Must be compound and rare so it doesn't collide with normal language. The face LoRA lands at `mlx_models/loras/<trigger>_v2.safetensors`; the optional audio LoRA at `<trigger>.audio.safetensors`. |
+| `trigger` | string | Trigger token, e.g. `bizarrotrn`. Must be compound, rare and **letters-only** so it doesn't collide with normal language — digits tokenize as single, very common tokens (`mrz07` → `m / rz / 0 / 7`), and every trigger that has carried an identity here was letters-only; the panel suggests `<3 consonants>trn` (`mrztrn`) and logs a note when a trigger carries digits (#62). The face LoRA lands at `mlx_models/loras/<trigger>_v2.safetensors`; the optional audio LoRA at `<trigger>.audio.safetensors`. |
| `preset` | `quick` \| `medium` \| `high` | Hyperparameter preset. **Default changed in v4.6.1: omitting it now gives `high` for `train_type=character` and `quick` for `train_type=style`** (`TRAIN_DEFAULT_PRESET`) — it used to give `quick` for both, which is the tier that has never carried an identity (#62). See the table below — **the step count is not a constant, and on a sub-64 GB Mac neither is anything else.** |
| `image_count` | int | Confirmed by the panel from the uploaded images. |
| `train_audio` | `true` \| `false` | Optional. Default `false`. When `true`, a voice clip must already exist for this `train_job_id` (uploaded via `/train/upload-voice`) — otherwise `/train/start` returns 400. |
diff --git a/docs/STATE.md b/docs/STATE.md
index f1587db..6b0d7da 100644
--- a/docs/STATE.md
+++ b/docs/STATE.md
@@ -1,5 +1,40 @@
# Phosphene — project state, history, open work
+> **🩹 2026-09-02 — v4.9.2: three issue fixes (#73, #61, #62), shipped the same day as v4.9.1.**
+> **#73 — an interrupted image-engine download looked ready and crashed every
+> render.** Qwen-Image-Edit-2511 stopped at 38 of 54 GB with 14 blobs still
+> `.incomplete`; `/image/engine_status` said cached; mflux resolves a Hub id
+> cached-first and its completeness test only asks whether every file pattern
+> has SOME match, so it loaded the shards it had and died in its weight loader
+> without ever calling the one function that resumes. Now: `image_engine.
+> hf_repo_partial_download()` reads the one unambiguous sign huggingface_hub
+> leaves (`blobs/*.incomplete`); the status route reports `partial` and stops
+> saying cached (pill "resume · N of ~M GB", button "Generate · resumes the
+> download first"); and a pre-flight before every mflux spawn removes the
+> `snapshots/` symlink tree so mflux's own download rule runs, re-links every
+> finished blob and resumes the rest — nothing downloads twice. The error the
+> panel showed was the HEAD of the traceback (`stderr_tail[:1200]`), cut off
+> right before the exception; both slices now keep the END. `test_image_
+> partial_download.py`. **#61 — training dies when the GPU watchdog kills the
+> caption encode.** The render helper's mitigation (retry once at a shorter
+> Gemma pad) is ported: the panel places the kill from the trainer's own
+> phase banners, relaunches the trainer ONCE for a SIGABRT-with-signature in
+> `text_encode` with `LTX2_GEMMA_MAX_LENGTH=256` (honoured by a wrapper in
+> `lora_lab/preprocess_images.py` around the vendored `encode_all_layers`,
+> default 1024), wipes the half-written caption conditions so every caption
+> is encoded at one length, and leaves the image latents (the expensive
+> half) for the preprocessor to skip. Kills in other phases keep the canvas
+> advice. `test_train_encode_retry.py`. **#62 — the trigger the panel
+> suggested was half digits.** `mrz07` tokenizes through Gemma as `m / rz /
+> 0 / 7`; every trigger that has ever carried a face here is letters-only
+> (`bizarrotrn` → `b / izarro / trn`, `elontrn`, `ariatrn`), and the rank-32
+> High trainings reported as "active but no identity" all used the digit
+> shape (`mmx26`, `sfw25`, `3Mar26`). Unproven as THE cause — it is being
+> A/B'd with the reporter — but suggesting the shape known to work costs
+> nothing: `_suggest_trigger_token()` and its JS mirror now emit
+> `<3 consonants>trn`, the Train tab warns (never blocks) on a digit, the
+> trainer logs the same note, API.md says letters-only.
+
> **🩹 2026-09-02 — v4.9.1: the buttons v4.9.0 broke, fixed the same day they were reported.**
> The module split had one blind spot: a function referenced ONLY from its own
> module's generated markup (`onclick="deleteOutput('…')"` inside a template
diff --git a/image_engine.py b/image_engine.py
index f7cc17e..ccd2fc9 100644
--- a/image_engine.py
+++ b/image_engine.py
@@ -103,6 +103,83 @@ def _clean_subprocess_env() -> dict:
return env
+def _hf_hub_root(env: "dict | None" = None) -> Path:
+ """The Hub cache root the mflux subprocess will resolve, from ITS env —
+ HF_HUB_CACHE, else HF_HOME/hub, else ~/.cache/huggingface/hub, the
+ order huggingface_hub itself uses."""
+ import os as _os
+ src = env if env is not None else _os.environ
+ explicit = src.get("HF_HUB_CACHE") or src.get("HUGGINGFACE_HUB_CACHE")
+ if explicit:
+ return Path(explicit).expanduser()
+ home = src.get("HF_HOME")
+ if home:
+ return Path(home).expanduser() / "hub"
+ return Path.home() / ".cache" / "huggingface" / "hub"
+
+
+def _is_hf_repo_id(value) -> bool:
+ # mflux's own test for "this is a Hub id, not a path": exactly one slash
+ # and no path prefix.
+ s = str(value or "")
+ return ("/" in s and s.count("/") == 1
+ and not s.startswith(("./", "../", "~/", "/")))
+
+
+def hf_repo_partial_download(repo_id: str, env: "dict | None" = None) -> "dict | None":
+ """Report an interrupted download of `repo_id` in the Hub cache, or None.
+
+ huggingface_hub streams each file to `blobs/<sha>.incomplete` and renames
+ it on completion, so a leftover `.incomplete` is the one unambiguous sign
+ that a download stopped partway. #73: 38 of 54 GB had landed, 14 files
+ were still `.incomplete`, the panel reported the engine ready and every
+ image generation died in mflux's weight loader.
+ """
+ repo_dir = _hf_hub_root(env) / ("models--" + repo_id.replace("/", "--"))
+ blobs = repo_dir / "blobs"
+ if not blobs.is_dir():
+ return None
+ try:
+ entries = list(blobs.iterdir())
+ except OSError:
+ return None
+ partial = [p for p in entries if p.name.endswith(".incomplete")]
+ if not partial:
+ return None
+ total = 0
+ for p in entries:
+ try:
+ if p.is_file():
+ total += p.stat().st_size
+ except OSError:
+ pass
+ return {"incomplete": len(partial), "on_disk_gb": round(total / 1e9, 1),
+ "repo_dir": repo_dir}
+
+
+def repair_partial_hf_download(repo_id: str, env: "dict | None" = None) -> "dict | None":
+ """Make an interrupted download resume instead of crash.
+
+ mflux resolves a Hub id cached-first, and its completeness test only
+ asks whether every file pattern has SOME match — a snapshot missing 14
+ of its transformer shards passes, mflux loads the shards it finds and
+ dies in WeightLoader, and snapshot_download (the only thing that would
+ resume the `.incomplete` blobs) is never called. Removing `snapshots/`
+ — a tree of symlinks; every finished blob stays where it is — makes the
+ cached rule fail, so mflux's own download rule runs: it re-links each
+ blob that is already complete and resumes the rest. Returns the partial
+ report when a repair was made, None when nothing was needed.
+ """
+ import shutil as _shutil
+ info = hf_repo_partial_download(repo_id, env)
+ if not info:
+ return None
+ snaps = info["repo_dir"] / "snapshots"
+ if snaps.is_dir():
+ _shutil.rmtree(snaps, ignore_errors=True)
+ return info
+
+
class ImageJobCancelled(RuntimeError):
"""Raised when an image-generation subprocess is killed by /stop.
@@ -1078,6 +1155,18 @@ def _generate_mflux(prompt: str, n: int, width: int, height: int,
# `for raw in iter(readline, "")` which could block forever if mflux
# stopped flushing — the deadline check after each line never tripped
# and the queue worker hung indefinitely.
+ # A download that stopped partway must resume, not crash (#73).
+ if _is_hf_repo_id(getattr(config, "mflux_model", None)):
+ _partial = repair_partial_hf_download(config.mflux_model, env)
+ if _partial and on_log is not None:
+ try:
+ on_log(f"[download] {config.mflux_model} was interrupted at "
+ f"{_partial['on_disk_gb']} GB ({_partial['incomplete']} files "
+ f"unfinished) — resuming it before this render; nothing "
+ f"already downloaded is fetched again.")
+ except Exception: # noqa: BLE001
+ pass
+
import collections
import select
last_lines: collections.deque[str] = collections.deque(maxlen=64)
@@ -1304,13 +1393,15 @@ def _generate_mflux(prompt: str, n: int, width: int, height: int,
f" - Quit other big apps (browsers + IDEs eat 8-15 GB).\n"
f"A crash report was saved to ~/Library/Logs/DiagnosticReports/"
f"python3.11-*.ips — useful for a bug report.\n\n"
- f"Tail of stdout/stderr:\n{stderr_tail[:800]}"
+ f"Tail of stdout/stderr:\n{stderr_tail[-800:]}"
)
# Generic non-zero exit — pass through the tail so the user sees
- # whatever mflux printed last.
+ # whatever mflux printed last. Keep the END of the tail: a Python
+ # traceback puts the exception on its last line, and the head-slice
+ # this used to take cut it off right before the message (#73).
raise RuntimeError(
f"{bin_name} failed (exit {rc}) on batch of {n} seeds. "
- f"Tail of stdout/stderr:\n{stderr_tail[:1200]}"
+ f"Tail of stdout/stderr:\n{stderr_tail[-1200:]}"
)
# Build results from whatever files actually landed. If the
diff --git a/lora_lab/preprocess_images.py b/lora_lab/preprocess_images.py
index 6ee0167..327332a 100644
--- a/lora_lab/preprocess_images.py
+++ b/lora_lab/preprocess_images.py
@@ -58,6 +58,44 @@ def _force_eval(*arrays: mx.array) -> None:
mx.eval(*arrays)
+def _apply_gemma_max_length_override() -> int | None:
+ """Honour ``LTX2_GEMMA_MAX_LENGTH`` for the caption encode, the way the
+ render pipelines do.
+
+ The vendored ``_encode_all_captions`` calls ``encode_all_layers(caption)``
+ with the default 1024-token pad, and on some chips the macOS GPU watchdog
+ kills that command buffer partway through the caption list (#61: SIGABRT
+ on the 1st, 7th or 11th caption of different runs — a race on accumulated
+ GPU pressure, not a bad caption). The render helper survives the same
+ kill by retrying once at a shorter encode; the panel relaunches the
+ trainer the same way and passes the length through this variable. A
+ 256-token pad holds every caption the trainer writes (50-80 words) with
+ room to spare, and the transformer already trains and renders against
+ that length on the render side. Returns the applied length, or None when
+ the variable is unset (the default 1024 stays).
+ """
+ raw = (os.environ.get("LTX2_GEMMA_MAX_LENGTH") or "").strip()
+ if not raw:
+ return None
+ try:
+ n = max(64, int(raw))
+ except ValueError:
+ return None
+ from ltx_core_mlx.text_encoders.gemma.encoders import base_encoder as _be
+ cls = _be.GemmaLanguageModel
+ orig = cls.encode_all_layers
+ if getattr(orig, "_ltx_max_length_override", None) == n:
+ return n
+
+ def encode_all_layers(self, text: str, max_length: int = 1024):
+ return orig(self, text, min(int(max_length), n))
+
+ encode_all_layers._ltx_max_length_override = n # type: ignore[attr-defined]
+ cls.encode_all_layers = encode_all_layers
+ print(f"caption encode: LTX2_GEMMA_MAX_LENGTH={n} (default 1024)")
+ return n
+
+
def _resolve_captions(image_files: list[Path], captions_dir: Path | None, caption_ext: str) -> list[str]:
captions: list[str] = []
if captions_dir is None:
@@ -202,6 +240,7 @@ def preprocess_images(
if gemma_model_id is None:
from lora_lab import resolve_default_text_encoder
gemma_model_id = resolve_default_text_encoder()
+ _apply_gemma_max_length_override()
mx.set_cache_limit(mx.device_info()["memory_size"])
model_dir = _resolve_model_dir(model_dir)
diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.py
index e022c1a..24a25c5 100644
--- a/mlx_ltx_panel.py
+++ b/mlx_ltx_panel.py
@@ -2409,22 +2409,38 @@ def _train_reconcile_captions(dataset_dir: Path) -> tuple[int, int, list[str]]:
def _suggest_trigger_token() -> str:
- """Generate a rare token of the form `<3-letter><2-digit>` (e.g. `mrz07`,
- `lqx42`). Avoids English-word collisions by picking three consonants for
- the leading run — phonotactically rare in English, almost always unique
- across the LTX vocabulary's text encoder. The two-digit tail prevents
- accidental clashes with existing LoRA triggers a user already has
- installed (e.g. an `mrz` LoRA + an `mrz07` LoRA coexist fine)."""
- # Skip vowels + ambiguous letters (l, i, o → 1, 0). Keep the set rare-in-
- # English (no common bigrams like 'th', 'st', 'sh').
+ """Generate a rare, LETTERS-ONLY token of the form `<3 consonants>trn`
+ (e.g. `mrztrn`, `kvdtrn`).
+
+ The previous shape was `<3 consonants><2 digits>` (`mrz07`). Through
+ Gemma's tokenizer that is `m / rz / 0 / 7` — two of the four pieces
+ are single digits, among the most common tokens the model knows, with
+ a prior that has nothing to do with a face. Every trigger that has
+ ever carried an identity here is letters-only and ends in `trn`
+ (`bizarrotrn` -> b / izarro / trn; `elontrn`; `ariatrn`), and the
+ trainings reported in #62 as "active but no identity" all used the
+ digit shape the panel itself suggested (`mmx26`, `sfw25`, `3Mar26`).
+ Whether the digits are THE cause is being A/B'd; suggesting the shape
+ that is known to work costs nothing. `trn` keeps the family readable
+ at a glance in the LoRA list; three consonants keep it phonotactically
+ rare in English (no common bigrams like 'th', 'st', 'sh').
+ """
consonants = "bcdfghjkmnpqrstvwxyz"
while True:
letters = "".join(random.choice(consonants) for _ in range(3))
# Reject anything that happens to spell a common letter pattern.
if letters in ("the", "and", "for", "but"):
continue
- digits = f"{random.randint(0, 99):02d}"
- return letters + digits
+ token = letters + "trn"
+ # Stay clear of an adapter the user already has under this name
+ # (the digit tail used to be what avoided that clash).
+ try:
+ if (LORAS_DIR / f"{token}_v2.safetensors").exists() or \
+ (LORAS_DIR / f"{token}.style.safetensors").exists():
+ continue
+ except Exception: # noqa: BLE001 — LORAS_DIR unset this early is fine
+ pass
+ return token
def _train_estimate_seconds(preset: str, image_count: int, *,
@@ -19939,6 +19955,67 @@ _TRAIN_ABORT_RX = re.compile(
)
+def _train_phase_from_line(line: str, payload: "dict | None", current: str) -> str:
+ """Which phase the trainer is in, from the lines it prints.
+
+ The preprocessor's phase banners are plain text (they reach the panel
+ on the merged stdout/stderr pipe); a training step is a JSON event.
+ Anything else keeps the current phase. Phases: start -> text_encode ->
+ image_encode -> train.
+ """
+ if isinstance(payload, dict):
+ if (payload.get("event") or "step") in ("step", "train_progress"):
+ return "train"
+ return current
+ s = line or ""
+ if "Phase 1: encoding text captions" in s:
+ return "text_encode"
+ if "Phase 2: encoding image latents" in s:
+ return "image_encode"
+ return current
+
+
+def _train_encode_retry_wanted(rc: int, watchdog_seen: bool, phase: str,
+ already_retried: bool) -> bool:
+ """One automatic relaunch, and only for the kill this can fix (#61):
+ SIGABRT with a Metal-watchdog signature in the log, during the caption
+ encode, not already retried. A kill during image encode or training
+ is the canvas-sized problem the existing error names; a crash without
+ the signature is not a GPU timeout at all."""
+ return (rc == -6 and bool(watchdog_seen) and phase == "text_encode"
+ and not already_retried)
+
+
+def _train_gemma_fallback_max_length() -> int:
+ """The shorter Gemma pad the relaunch uses — the same knob and default
+ (256, floor 64) as the render helper's fallback, so one env var tunes
+ both: LTX_GEMMA_FALLBACK_MAX_LENGTH."""
+ try:
+ return max(64, int(os.environ.get("LTX_GEMMA_FALLBACK_MAX_LENGTH", "256") or 256))
+ except ValueError:
+ return 256
+
+
+def _train_wipe_caption_conditions(dataset_dir: Path) -> int:
+ """Discard the caption encodings a killed run wrote, so the relaunch
+ re-encodes every caption at one length — the preprocessor skips a
+ condition file that exists, and a dataset half at 1024 tokens and half
+ at 256 would not batch. Image latents live beside them and are left
+ alone. Returns how many files went."""
+ cond = Path(dataset_dir) / "training_data" / ".precomputed" / "conditions"
+ if not cond.is_dir():
+ return 0
+ n = 0
+ for f in cond.iterdir():
+ if f.is_file() and f.name.startswith("condition_"):
+ try:
+ f.unlink()
+ n += 1
+ except OSError:
+ pass
+ return n
+
+
def run_train_job_inner(job: dict) -> None:
"""Run a Train Character job. Reads the dataset that /train/upload built
under state/train_character/<train_job_id>/, writes a job-spec JSON the
@@ -19994,6 +20071,13 @@ def run_train_job_inner(job: dict) -> None:
preset = p.get("preset", "quick")
trigger = p.get("trigger") or _suggest_trigger_token()
+ if re.search(r"\d", trigger):
+ # Not a refusal — the field is the user's — but the log should carry
+ # the one fact that separates the triggers that have carried a face
+ # from the ones reported in #62 (see _suggest_trigger_token).
+ push(f"[train] note: trigger {trigger!r} contains digits, which Gemma "
+ "splits into single very common tokens. Every trigger that has "
+ "carried an identity here was letters-only (e.g. 'mrztrn').")
caption_strategy = p.get("caption_strategy", "trigger_simple")
preset_table = TRAIN_STYLE_PRESETS if is_style else TRAIN_PRESETS
clamp_notes = p.get("train_clamp_notes") or []
@@ -20179,131 +20263,160 @@ def run_train_job_inner(job: dict) -> None:
last_step = 0
total_steps = spec["steps"]
last_loss = None
+ # Where the trainer is, read from its own lines, so a watchdog kill can
+ # be PLACED (#61): the caption encode is the phase that dies on some
+ # chips, and the only one worth relaunching — see _train_encode_retry.
+ _train_phase = "start"
+ _train_env = {**os.environ}
+ _encode_retried = False
try:
- proc = subprocess.Popen(
- cmd,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT,
- text=True,
- bufsize=1,
- env={**os.environ},
- # Own process group so /stop can take down the entire trainer
- # tree (train_character + its lora_lab + LtxvTrainer threads +
- # any ffmpeg helpers) with a single os.killpg. Without this,
- # /stop only killed the warm helper + mux ffmpeg and left the
- # trainer running for hours, blocking the worker.
- start_new_session=True,
- )
- with LOCK:
- STATE["pid"] = proc.pid
- STATE["train_pgid"] = os.getpgid(proc.pid)
- # Stream stdout. Each line is either JSON (a structured progress
- # event) or plain text (forwarded as-is to the log).
- assert proc.stdout is not None
- for raw in proc.stdout:
- line = raw.rstrip("\n")
- if not line:
- continue
- payload = None
- if line.lstrip().startswith("{"):
- try:
- payload = json.loads(line)
- except json.JSONDecodeError:
- payload = None
- if isinstance(payload, dict):
- # Progress event from the lab side. lora-lab's train_character
- # emits 'train_progress' (legacy upstream name from ltx-trainer),
- # while our newer train_audio emits 'step'. Accept both so the
- # face-phase progress bar actually advances (2026-05-15 fix —
- # before this the bar stuck at 'Loading pipeline' through the
- # entire face run).
- evt = payload.get("event") or "step"
- if evt in ("step", "train_progress"):
- try:
- last_step = int(payload.get("step") or last_step)
- except (TypeError, ValueError):
- pass
+ while True:
+ proc = subprocess.Popen(
+ cmd,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ text=True,
+ bufsize=1,
+ env=_train_env,
+ # Own process group so /stop can take down the entire trainer
+ # tree (train_character + its lora_lab + LtxvTrainer threads +
+ # any ffmpeg helpers) with a single os.killpg. Without this,
+ # /stop only killed the warm helper + mux ffmpeg and left the
+ # trainer running for hours, blocking the worker.
+ start_new_session=True,
+ )
+ with LOCK:
+ STATE["pid"] = proc.pid
+ STATE["train_pgid"] = os.getpgid(proc.pid)
+ # Stream stdout. Each line is either JSON (a structured progress
+ # event) or plain text (forwarded as-is to the log).
+ assert proc.stdout is not None
+ for raw in proc.stdout:
+ line = raw.rstrip("\n")
+ if not line:
+ continue
+ payload = None
+ if line.lstrip().startswith("{"):
try:
- total_steps = int(payload.get("total") or total_steps)
- except (TypeError, ValueError):
- pass
- last_loss = payload.get("loss", last_loss)
- # Write progress fields the existing /status reader can
- # pick up — same shape as _compute_progress would emit.
- with LOCK:
- if STATE.get("current") and STATE["current"].get("id") == job["id"]:
- phase_word = "style" if is_style else "face"
- STATE["current"]["progress"] = {
- "phase": "denoise",
- "phase_label": (
- f"Training {phase_word} · step "
- f"{last_step} / {total_steps}"),
- "pct": (last_step / max(1, total_steps)) * 100.0,
- "elapsed_sec": time.time() - t0,
- "eta_sec": max(0,
- (total_steps - last_step) * (
- (time.time() - t0) / max(1, last_step)
- if last_step > 0
- else preset_table.get(
- preset, preset_table["quick"]
- )["seconds_per_step"]
- )),
- "denoise_step": last_step,
- "denoise_total": total_steps,
- }
- if last_step % 50 == 0 or last_step == total_steps:
- push(f"[train] step {last_step}/{total_steps} loss={last_loss}")
- elif evt == "log":
- push(f"[train] {payload.get('msg', '')}")
- elif evt == "adapter_strength":
- # A GREEN TALLY WAS NEVER PROOF THE FILE COULD DO
- # ANYTHING. A LoRA can train to completion, write a
- # perfectly valid safetensors, load without a warning and
- # change NOTHING in the picture — because the deltas it
- # learned are numerically too small to move the model.
- # Every gate we had said "done"; only the render said
- # otherwise, hours later, on the owner's eye.
- #
- # The trainer measures the adapter's own delta RMS against
- # a floor and says so. Surfacing it here is the difference
- # between a training run that reports success and one that
- # reports the truth.
- med = payload.get("delta_rms_median")
- mx = payload.get("delta_rms_max")
- verdict = str(payload.get("verdict") or "unknown")
- carrying = payload.get("carrying_modules")
- mods = payload.get("modules")
- bits = [f"median={med}", f"max={mx}"]
- if carrying is not None and mods is not None:
- bits.append(f"{carrying}/{mods} modules carrying")
- if payload.get("floor") is not None:
- bits.append(f"floor={payload.get('floor')}")
- push(f"[train] adapter strength: delta_rms "
- f"{' · '.join(bits)} ({verdict})")
- train_verdict = verdict
- if verdict != "ok":
- push("[train] WARNING: this adapter may not visibly "
- "change anything it is applied to. Train longer, "
- "raise the rank, or check the captions — a file "
- "that loads is not a file that works.")
- elif evt == "checkpoint":
- push(f"[train] checkpoint → {payload.get('path', '?')}")
- elif evt == "done":
- push(f"[train] done · {payload.get('path', '?')}")
+ payload = json.loads(line)
+ except json.JSONDecodeError:
+ payload = None
+ _train_phase = _train_phase_from_line(line, payload, _train_phase)
+ if isinstance(payload, dict):
+ # Progress event from the lab side. lora-lab's train_character
+ # emits 'train_progress' (legacy upstream name from ltx-trainer),
+ # while our newer train_audio emits 'step'. Accept both so the
+ # face-phase progress bar actually advances (2026-05-15 fix —
+ # before this the bar stuck at 'Loading pipeline' through the
+ # entire face run).
+ evt = payload.get("event") or "step"
+ if evt in ("step", "train_progress"):
+ try:
+ last_step = int(payload.get("step") or last_step)
+ except (TypeError, ValueError):
+ pass
+ try:
+ total_steps = int(payload.get("total") or total_steps)
+ except (TypeError, ValueError):
+ pass
+ last_loss = payload.get("loss", last_loss)
+ # Write progress fields the existing /status reader can
+ # pick up — same shape as _compute_progress would emit.
+ with LOCK:
+ if STATE.get("current") and STATE["current"].get("id") == job["id"]:
+ phase_word = "style" if is_style else "face"
+ STATE["current"]["progress"] = {
+ "phase": "denoise",
+ "phase_label": (
+ f"Training {phase_word} · step "
+ f"{last_step} / {total_steps}"),
+ "pct": (last_step / max(1, total_steps)) * 100.0,
+ "elapsed_sec": time.time() - t0,
+ "eta_sec": max(0,
+ (total_steps - last_step) * (
+ (time.time() - t0) / max(1, last_step)
+ if last_step > 0
+ else preset_table.get(
+ preset, preset_table["quick"]
+ )["seconds_per_step"]
+ )),
+ "denoise_step": last_step,
+ "denoise_total": total_steps,
+ }
+ if last_step % 50 == 0 or last_step == total_steps:
+ push(f"[train] step {last_step}/{total_steps} loss={last_loss}")
+ elif evt == "log":
+ push(f"[train] {payload.get('msg', '')}")
+ elif evt == "adapter_strength":
+ # A GREEN TALLY WAS NEVER PROOF THE FILE COULD DO
+ # ANYTHING. A LoRA can train to completion, write a
+ # perfectly valid safetensors, load without a warning and
+ # change NOTHING in the picture — because the deltas it
+ # learned are numerically too small to move the model.
+ # Every gate we had said "done"; only the render said
+ # otherwise, hours later, on the owner's eye.
+ #
+ # The trainer measures the adapter's own delta RMS against
+ # a floor and says so. Surfacing it here is the difference
+ # between a training run that reports success and one that
+ # reports the truth.
+ med = payload.get("delta_rms_median")
+ mx = payload.get("delta_rms_max")
+ verdict = str(payload.get("verdict") or "unknown")
+ carrying = payload.get("carrying_modules")
+ mods = payload.get("modules")
+ bits = [f"median={med}", f"max={mx}"]
+ if carrying is not None and mods is not None:
+ bits.append(f"{carrying}/{mods} modules carrying")
+ if payload.get("floor") is not None:
+ bits.append(f"floor={payload.get('floor')}")
+ push(f"[train] adapter strength: delta_rms "
+ f"{' · '.join(bits)} ({verdict})")
+ train_verdict = verdict
+ if verdict != "ok":
+ push("[train] WARNING: this adapter may not visibly "
+ "change anything it is applied to. Train longer, "
+ "raise the rank, or check the captions — a file "
+ "that loads is not a file that works.")
+ elif evt == "checkpoint":
+ push(f"[train] checkpoint → {payload.get('path', '?')}")
+ elif evt == "done":
+ push(f"[train] done · {payload.get('path', '?')}")
+ else:
+ push(f"[train] {line}")
else:
push(f"[train] {line}")
- else:
- push(f"[train] {line}")
- # The trainer dies by SIGABRT when macOS kills its GPU command
- # buffer, and `code -6` on its own tells the user nothing. Watch
- # the same signatures the render helper watches (#61).
- if not _train_watchdog_seen and _TRAIN_ABORT_RX.search(line or ""):
- _train_watchdog_seen = True
- rc = proc.wait()
- proc = None
- with LOCK:
- STATE["pid"] = None
- STATE["train_pgid"] = None
+ # The trainer dies by SIGABRT when macOS kills its GPU command
+ # buffer, and `code -6` on its own tells the user nothing. Watch
+ # the same signatures the render helper watches (#61).
+ if not _train_watchdog_seen and _TRAIN_ABORT_RX.search(line or ""):
+ _train_watchdog_seen = True
+ rc = proc.wait()
+ proc = None
+ with LOCK:
+ STATE["pid"] = None
+ STATE["train_pgid"] = None
+ if _train_encode_retry_wanted(rc, _train_watchdog_seen, _train_phase,
+ _encode_retried):
+ # The render helper survives this exact kill by retrying once
+ # at a shorter Gemma encode; this is the same move for the
+ # trainer (#61). Nothing trained yet, so nothing is lost: the
+ # caption conditions written so far are wiped so every caption
+ # is re-encoded at ONE length (the trainer batches them), and
+ # the image latents — the expensive half — are untouched and
+ # skipped by the preprocessor on the relaunch.
+ _encode_retried = True
+ n = _train_gemma_fallback_max_length()
+ wiped = _train_wipe_caption_conditions(dataset_dir)
+ push(f"[train] the macOS GPU watchdog killed the caption encode "
+ f"(Metal, SIGABRT) — relaunching the trainer once with Gemma "
+ f"encoding at {n} tokens instead of 1024. {wiped} caption "
+ f"encoding(s) discarded; image latents kept.")
+ _train_env["LTX2_GEMMA_MAX_LENGTH"] = str(n)
+ _train_watchdog_seen = False
+ _train_phase = "start"
+ continue
+ break
if rc != 0:
# -6 is SIGABRT. With a watchdog signature in the log it is macOS
# killing the GPU command buffer, not our code faulting — and the
@@ -20311,6 +20424,13 @@ def run_train_job_inner(job: dict) -> None:
# aborted on quick AND medium while 512x512 trained fine, which is
# canvas-shaped, not preset-shaped.
if rc == -6 and _train_watchdog_seen:
+ if _train_phase == "text_encode":
+ raise RuntimeError(
+ "macOS killed the training GPU command buffer (Metal "
+ "watchdog, SIGABRT) during the caption encode — twice: "
+ "the automatic relaunch at a shorter Gemma length died "
+ "the same way. Close other GPU apps and retry; if it "
+ "repeats, attach the Logs tail to an issue.")
raise RuntimeError(
"macOS killed the training GPU command buffer (Metal "
"watchdog, SIGABRT). This is the canvas, not the preset — "
diff --git a/panel/routes_image.py b/panel/routes_image.py
index 39d4b85..9d90c18 100644
--- a/panel/routes_image.py
+++ b/panel/routes_image.py
@@ -304,6 +304,7 @@ def get_image_engine_status(h, parsed) -> None:
out = []
for engine, repo, dl_gb, sec, cold in ENGINES:
family_installed = True
+ partial = None
if engine in ("hidream_inline", "hidream_fast_inline", "hidream_quality_inline"):
# HiDream lives outside the HF cache (lab venv path).
# All three modes share the same Dev-BF16 model dir.
@@ -314,6 +315,13 @@ def get_image_engine_status(h, parsed) -> None:
else:
_snap = P._repo_hf_cache_dir(repo)
cached = _snap is not None
+ # An interrupted download is NOT cached (#73): a snapshot
+ # with `.incomplete` blobs beside it looked ready here,
+ # and the render died in mflux's loader. The pill says
+ # "resume" and the render's pre-flight resumes it.
+ partial = P.agent_image_engine.hf_repo_partial_download(repo)
+ if partial:
+ cached = False
# Ideogram: count EITHER the un-gated mirror OR an existing
# official download, and require a real weight file (a
# gated stub snapshot has only README/LICENSE).
@@ -353,6 +361,8 @@ def get_image_engine_status(h, parsed) -> None:
"engine": engine,
"repo_id": repo or "",
"cached": cached,
+ "partial": bool(partial),
+ "partial_gb": (partial["on_disk_gb"] if partial else 0),
"family_installed": family_installed,
"download_gb": dl_gb,
"sec_per_image": round(sec, 1),
diff --git a/test_image_partial_download.py b/test_image_partial_download.py
new file mode 100644
index 0000000..b562ba6
--- /dev/null
+++ b/test_image_partial_download.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""An interrupted image-engine download must resume, not crash (#73).
+
+The report: Qwen-Image-Edit-2511 stopped at 38 of 54 GB with 14 blobs still
+`.incomplete`; `/image/engine_status` said the engine was ready; every
+generation died in mflux's weight loader; and the error the panel showed was
+the HEAD of the traceback, cut off before the exception. Three assertions:
+the partial state is detected from the one unambiguous sign huggingface_hub
+leaves (`blobs/*.incomplete`), the repair removes only the snapshot symlink
+tree (finished blobs survive, so nothing downloads twice), and the status
+route stops calling a partial download "cached".
+"""
+from __future__ import annotations
+
+import json
+import os
+import sys
+import tempfile
+import threading
+import unittest
+import urllib.request
+from http.server import ThreadingHTTPServer
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+HF_HOME = Path(tempfile.mkdtemp(prefix="phos-hf-"))
+STATE = Path(tempfile.mkdtemp(prefix="phos-partial-state-"))
+os.environ["HF_HOME"] = str(HF_HOME)
+os.environ.pop("HF_HUB_CACHE", None)
+os.environ.pop("HUGGINGFACE_HUB_CACHE", None)
+os.environ["LTX_STATE_DIR"] = str(STATE)
+os.environ["PHOSPHENE_ANALYTICS_DISABLED"] = "1"
+os.environ["PHOSPHENE_DISABLE_VERSION_CHECK"] = "1"
+os.environ.setdefault("LTX_PORT", "8312")
+sys.path.insert(0, str(ROOT))
+
+import image_engine # noqa: E402
+
+REPO = "Qwen/Qwen-Image-Edit-2511"
+
+
+def _fake_repo(root: Path, repo: str, *, partial: bool) -> Path:
+ repo_dir = root / "hub" / ("models--" + repo.replace("/", "--"))
+ blobs = repo_dir / "blobs"
+ snap = repo_dir / "snapshots" / "abc123" / "transformer"
+ blobs.mkdir(parents=True, exist_ok=True)
+ snap.mkdir(parents=True, exist_ok=True)
+ (blobs / "aaaa").write_bytes(b"\0" * 4096)
+ if partial:
+ (blobs / "bbbb.incomplete").write_bytes(b"\0" * 1024)
+ link = snap / "model-00001.safetensors"
+ if not link.exists():
+ os.symlink(os.path.relpath(blobs / "aaaa", snap), link)
+ return repo_dir
+
+
+class PartialDownloadDetection(unittest.TestCase):
+ def test_clean_repo_is_not_partial_and_repair_is_a_noop(self) -> None:
+ root = Path(tempfile.mkdtemp())
+ repo_dir = _fake_repo(root, REPO, partial=False)
+ env = {"HF_HOME": str(root)}
+ self.assertIsNone(image_engine.hf_repo_partial_download(REPO, env))
+ self.assertIsNone(image_engine.repair_partial_hf_download(REPO, env))
+ self.assertTrue((repo_dir / "snapshots").is_dir(), "a complete cache is left alone")
+
+ def test_incomplete_blob_is_detected_and_repair_keeps_finished_blobs(self) -> None:
+ root = Path(tempfile.mkdtemp())
+ repo_dir = _fake_repo(root, REPO, partial=True)
+ env = {"HF_HOME": str(root)}
+ info = image_engine.hf_repo_partial_download(REPO, env)
+ self.assertIsNotNone(info)
+ self.assertEqual(info["incomplete"], 1)
+ self.assertGreaterEqual(info["on_disk_gb"], 0.0)
+ repaired = image_engine.repair_partial_hf_download(REPO, env)
+ self.assertIsNotNone(repaired)
+ self.assertFalse((repo_dir / "snapshots").exists(),
+ "the snapshot symlink tree goes, so mflux's cached-first rule fails and it resumes")
+ self.assertTrue((repo_dir / "blobs" / "aaaa").exists(), "finished blobs stay — nothing downloads twice")
+ self.assertTrue((repo_dir / "blobs" / "bbbb.incomplete").exists(),
+ "the partial blob stays — huggingface_hub resumes it")
+
+ def test_hub_root_follows_the_subprocess_env(self) -> None:
+ self.assertEqual(image_engine._hf_hub_root({"HF_HOME": "/x"}), Path("/x/hub"))
+ self.assertEqual(image_engine._hf_hub_root({"HF_HUB_CACHE": "/y"}), Path("/y"))
+ self.assertTrue(image_engine._is_hf_repo_id(REPO))
+ self.assertFalse(image_engine._is_hf_repo_id("/abs/path"))
+ self.assertFalse(image_engine._is_hf_repo_id("~/local/model"))
+
+
+class EngineStatusRoute(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls) -> None:
+ _fake_repo(HF_HOME, REPO, partial=True)
+ import mlx_ltx_panel as P # noqa: E402 — after HF_HOME is set
+ cls.P = P
+ cls.server = ThreadingHTTPServer(("127.0.0.1", 0), P.Handler)
+ cls.port = cls.server.server_address[1]
+ threading.Thread(target=cls.server.serve_forever, daemon=True).start()
+
+ @classmethod
+ def tearDownClass(cls) -> None:
+ cls.server.shutdown()
+ cls.server.server_close()
+
+ def test_partial_download_is_not_reported_cached(self) -> None:
+ body = urllib.request.urlopen(
+ f"http://127.0.0.1:{self.port}/image/engine_status").read()
+ engines = {e["engine"]: e for e in json.loads(body)["engines"]}
+ qwen = engines["qwen_edit_inline"]
+ self.assertFalse(qwen["cached"], "an interrupted download is not cached")
+ self.assertTrue(qwen["partial"], "the status names the partial state")
+ self.assertGreaterEqual(qwen["partial_gb"], 0.0)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/test_train_encode_retry.py b/test_train_encode_retry.py
new file mode 100644
index 0000000..4a75015
--- /dev/null
+++ b/test_train_encode_retry.py
@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+"""Training survives the caption-encode watchdog kill by relaunching once at a
+shorter Gemma length (#61) — the render helper's mitigation, ported.
+
+The trainer runs in its own process and dies by SIGABRT when macOS kills its
+GPU command buffer partway through the caption list (the 1st, 7th or 11th
+caption on different runs: a race on accumulated GPU pressure, not a bad
+input). What is pinned here: the panel places the kill from the trainer's
+own lines, relaunches ONLY for that phase and only once, discards the
+half-written caption encodings so the relaunch encodes every caption at one
+length, and the preprocessor honours the shorter length the relaunch passes.
+"""
+from __future__ import annotations
+
+import os
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+os.environ["LTX_STATE_DIR"] = tempfile.mkdtemp(prefix="phos-train-retry-")
+os.environ["PHOSPHENE_ANALYTICS_DISABLED"] = "1"
+os.environ["PHOSPHENE_DISABLE_VERSION_CHECK"] = "1"
+os.environ.setdefault("LTX_PORT", "8313")
+sys.path.insert(0, str(ROOT))
+
+import mlx_ltx_panel as P # noqa: E402
+
+
+class PhasePlacement(unittest.TestCase):
+ def test_follows_the_preprocessor_banners_and_step_events(self) -> None:
+ f = P._train_phase_from_line
+ self.assertEqual(f("found 51 images in /x", None, "start"), "start")
+ self.assertEqual(f("Phase 1: encoding text captions (Gemma)...", None, "start"), "text_encode")
+ self.assertEqual(f(" [7/51] Encoding: 'a caption'", None, "text_encode"), "text_encode")
+ self.assertEqual(f("Phase 2: encoding image latents at 512x512 (1 frame each, crop_anchor=center)...",
+ None, "text_encode"), "image_encode")
+ self.assertEqual(f('{"event":"train_progress","step":1}', {"event": "train_progress", "step": 1},
+ "image_encode"), "train")
+ self.assertEqual(f('{"event":"log","msg":"x"}', {"event": "log", "msg": "x"}, "train"), "train")
+
+ def test_watchdog_signature_is_the_same_family_the_helper_watches(self) -> None:
+ self.assertTrue(P._TRAIN_ABORT_RX.search(
+ "libc++abi: terminating due to uncaught exception"))
+ self.assertTrue(P._TRAIN_ABORT_RX.search(
+ "Error: kIOGPUCommandBufferCallbackErrorTimeout"))
+ self.assertFalse(P._TRAIN_ABORT_RX.search("step 100/5000 loss=0.12"))
+
+
+class RetryDecision(unittest.TestCase):
+ def test_only_the_caption_encode_kill_is_relaunched_and_only_once(self) -> None:
+ want = P._train_encode_retry_wanted
+ self.assertTrue(want(-6, True, "text_encode", False))
+ self.assertFalse(want(-6, True, "text_encode", True), "one relaunch, not a loop")
+ self.assertFalse(want(-6, True, "image_encode", False), "canvas-shaped kill: the existing error")
+ self.assertFalse(want(-6, True, "train", False))
+ self.assertFalse(want(-6, False, "text_encode", False), "no signature: a crash, not a timeout")
+ self.assertFalse(want(1, True, "text_encode", False))
+ self.assertFalse(want(0, True, "text_encode", False))
+
+ def test_fallback_length_shares_the_render_helper_knob(self) -> None:
+ old = os.environ.pop("LTX_GEMMA_FALLBACK_MAX_LENGTH", None)
+ try:
+ self.assertEqual(P._train_gemma_fallback_max_length(), 256)
+ os.environ["LTX_GEMMA_FALLBACK_MAX_LENGTH"] = "16"
+ self.assertEqual(P._train_gemma_fallback_max_length(), 64, "floor 64")
+ os.environ["LTX_GEMMA_FALLBACK_MAX_LENGTH"] = "512"
+ self.assertEqual(P._train_gemma_fallback_max_length(), 512)
+ os.environ["LTX_GEMMA_FALLBACK_MAX_LENGTH"] = "junk"
+ self.assertEqual(P._train_gemma_fallback_max_length(), 256)
+ finally:
+ if old is None:
+ os.environ.pop("LTX_GEMMA_FALLBACK_MAX_LENGTH", None)
+ else:
+ os.environ["LTX_GEMMA_FALLBACK_MAX_LENGTH"] = old
+
+
+class ConditionsWipe(unittest.TestCase):
+ def test_discards_caption_encodings_and_keeps_image_latents(self) -> None:
+ ds = Path(tempfile.mkdtemp())
+ pre = ds / "training_data" / ".precomputed"
+ (pre / "conditions").mkdir(parents=True)
+ (pre / "latents").mkdir(parents=True)
+ for i in range(3):
+ (pre / "conditions" / f"condition_{i:04d}.safetensors").write_bytes(b"c")
+ (pre / "latents" / f"latent_{i:04d}.safetensors").write_bytes(b"l")
+ self.assertEqual(P._train_wipe_caption_conditions(ds), 3)
+ self.assertEqual(list((pre / "conditions").iterdir()), [])
+ self.assertEqual(len(list((pre / "latents").iterdir())), 3, "the expensive half stays")
+ self.assertEqual(P._train_wipe_caption_conditions(Path(tempfile.mkdtemp())), 0,
+ "no dataset yet: nothing to do, no error")
+
+
+class PreprocessHonoursTheShorterLength(unittest.TestCase):
+ def test_encode_all_layers_is_capped_when_the_variable_is_set(self) -> None:
+ try:
+ from lora_lab import preprocess_images as pp
+ from ltx_core_mlx.text_encoders.gemma.encoders import base_encoder as be
+ except Exception as e: # noqa: BLE001 — the vendored trainer is an install-time dep
+ self.skipTest(f"trainer deps not importable here: {e}")
+ old = os.environ.pop("LTX2_GEMMA_MAX_LENGTH", None)
+ orig = be.GemmaLanguageModel.encode_all_layers
+ try:
+ self.assertIsNone(pp._apply_gemma_max_length_override(), "unset: the default 1024 stays")
+ self.assertIs(be.GemmaLanguageModel.encode_all_layers, orig)
+ os.environ["LTX2_GEMMA_MAX_LENGTH"] = "256"
+ self.assertEqual(pp._apply_gemma_max_length_override(), 256)
+ seen = {}
+
+ def fake(self, text, max_length=1024):
+ seen["max_length"] = max_length
+ return ([], None)
+
+ # The wrapper closes over the ORIGINAL; stub what it calls through
+ # to and check the length it forwards.
+ wrapped = be.GemmaLanguageModel.encode_all_layers
+ self.assertIsNot(wrapped, orig)
+ be.GemmaLanguageModel.encode_all_layers = orig
+ be.GemmaLanguageModel.encode_all_layers = fake
+ pp._apply_gemma_max_length_override()
+ be.GemmaLanguageModel.encode_all_layers(object(), "a caption")
+ self.assertEqual(seen["max_length"], 256)
+ be.GemmaLanguageModel.encode_all_layers(object(), "a caption", 128)
+ self.assertEqual(seen["max_length"], 128, "an explicit shorter length is kept")
+ self.assertEqual(pp._apply_gemma_max_length_override(), 256, "idempotent")
+ finally:
+ be.GemmaLanguageModel.encode_all_layers = orig
+ if old is None:
+ os.environ.pop("LTX2_GEMMA_MAX_LENGTH", None)
+ else:
+ os.environ["LTX2_GEMMA_MAX_LENGTH"] = old
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/webapp/index.html b/webapp/index.html
index eb7e479..504f333 100644
--- a/webapp/index.html
+++ b/webapp/index.html
@@ -2416,13 +2416,13 @@
<!-- ============== TRIGGER + PRESET ============== -->
<div class="composer-card train-card">
<h2>Trigger word
- <span class="h2-hint" id="trainTriggerHint">a rare token the model will associate with this character</span>
+ <span class="h2-hint" id="trainTriggerHint">a rare, letters-only token the model will associate with this character</span>
</h2>
<div class="train-trigger-row">
<input type="text" id="trainTrigger" maxlength="32" placeholder="auto-generated rare token" autocomplete="off">
<button type="button" class="qchip" onclick="trainSuggestTrigger()" title="Suggest a fresh rare token">↻ Suggest</button>
</div>
- <div class="hint" id="trainTriggerExampleHint">Use the trigger word in your video prompts (e.g. <code><span id="trainTriggerExample">mrz07</span> man walking on the beach</code>).</div>
+ <div class="hint" id="trainTriggerExampleHint">Use the trigger word in your video prompts (e.g. <code><span id="trainTriggerExample">mrztrn</span> man walking on the beach</code>).</div>
<h2 style="margin-top:14px">Quality preset
<span class="h2-hint">trade time for fidelity</span>
diff --git a/webapp/js/characters.js b/webapp/js/characters.js
index 3afc013..a3664ac 100644
--- a/webapp/js/characters.js
+++ b/webapp/js/characters.js
@@ -1243,10 +1243,12 @@ function trainInit() {
const t = document.getElementById('trainTrigger');
if (t && !t.value) t.value = trainGenerateTriggerJS();
document.getElementById('trainTriggerExample').textContent =
- (document.getElementById('trainTrigger').value || 'mrz07');
+ (document.getElementById('trainTrigger').value || 'mrztrn');
+ trainTriggerDigitWarn();
document.getElementById('trainTrigger').addEventListener('input', () => {
document.getElementById('trainTriggerExample').textContent =
- (document.getElementById('trainTrigger').value || 'mrz07');
+ (document.getElementById('trainTrigger').value || 'mrztrn');
+ trainTriggerDigitWarn();
trainUpdateButtonState();
trainUpdateEstimate();
});
@@ -1418,18 +1420,34 @@ async function trainInstall(key) {
// Skip vowels + ambiguous letters (l, i, o → 1, 0). Mirrors the Python
// _suggest_trigger_token; not a security boundary so the algorithms can
// drift slightly — server has its own generator for non-JS callers.
+// Letters-only, `<3 consonants>trn` — the shape of every trigger that has
+// carried a face here (bizarrotrn, elontrn, ariatrn). The old `mrz07` shape
+// tokenizes to m / rz / 0 / 7: two single-digit tokens with a huge prior of
+// their own. Mirrors _suggest_trigger_token() server-side (#62).
function trainGenerateTriggerJS() {
const cons = 'bcdfghjkmnpqrstvwxyz';
let letters = '';
for (let i = 0; i < 3; i++) letters += cons[Math.floor(Math.random() * cons.length)];
- const digits = String(Math.floor(Math.random() * 100)).padStart(2, '0');
- return letters + digits;
+ return letters + 'trn';
+}
+
+// Warn (never block) when a typed trigger carries digits — see above.
+function trainTriggerDigitWarn() {
+ const t = document.getElementById('trainTrigger');
+ const hint = document.getElementById('trainTriggerHint');
+ if (!t || !hint) return;
+ const bad = /\d/.test((t.value || '').trim());
+ hint.textContent = bad
+ ? 'digits split into their own very common tokens — every trigger that has carried a face here was letters-only (e.g. mrztrn)'
+ : 'a rare, letters-only token the model will associate with this character';
+ hint.classList.toggle('warn', bad);
}
function trainSuggestTrigger() {
const t = document.getElementById('trainTrigger');
t.value = trainGenerateTriggerJS();
document.getElementById('trainTriggerExample').textContent = t.value;
+ trainTriggerDigitWarn();
}
function trainWireDropZone() {
diff --git a/webapp/js/stage.js b/webapp/js/stage.js
index a4b2748..a7a0223 100644
--- a/webapp/js/stage.js
+++ b/webapp/js/stage.js
@@ -351,6 +351,14 @@ function imgStudioRenderEnginePill() {
pill.dataset.state = 'ready';
pill.innerHTML = '<svg class="ph" aria-hidden="true" style="margin-right:4px;vertical-align:-2px"><use href="#ph-check-bold"/></svg>ready';
pill.title = 'Weights are cached locally';
+ } else if (info.partial) {
+ // A download that stopped partway (#73). Generate resumes it — say so,
+ // instead of "ready" (the old lie) or a fresh full-size download.
+ pill.dataset.state = 'missing';
+ const done = info.partial_gb || 0;
+ const gb = info.download_gb || 0;
+ pill.innerHTML = '<svg class="ph" aria-hidden="true" style="margin-right:4px;vertical-align:-2px"><use href="#ph-download-simple"/></svg>resume' + (gb > 0 ? ` · ${done.toFixed(0)} of ~${gb.toFixed(0)} GB` : '');
+ pill.title = `The weight download was interrupted at ${done.toFixed(1)} GB — Generate resumes it. Nothing already downloaded is fetched again.`;
} else {
pill.dataset.state = 'missing';
const gb = info.download_gb || 0;
@@ -375,7 +383,9 @@ function imgStudioUpdateEstimate() {
// doesn't know its target ahead of time so we hide.
const info = _IMG_ENGINE_STATUS[eng];
let label = 'Generate';
- if (info && !info.cached && (info.download_gb || 0) > 0) {
+ if (info && !info.cached && info.partial) {
+ label = `Generate · resumes the download first (${(info.partial_gb || 0).toFixed(0)} GB done)`;
+ } else if (info && !info.cached && (info.download_gb || 0) > 0) {
label = `Generate · downloads ~${info.download_gb.toFixed(0)} GB first`;
}
btnLabel.textContent = label;
diff --git a/webapp/style/panel.css b/webapp/style/panel.css
index 35490b2..81ddd6d 100644
--- a/webapp/style/panel.css
+++ b/webapp/style/panel.css
@@ -4322,6 +4322,8 @@
opacity: 0.85;
margin-left: 8px;
}
+ /* A trigger with digits in it (#62) — a warning, not a refusal. */
+ .train-pane h2 .h2-hint.warn { color: var(--warning); opacity: 1; }
/* Preflight banner — only visible when a required model isn't on disk.
Same dark-card chrome as the Composer cards. */
.train-preflight-card {