Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

release(v4.15.1): the install fixes — ffmpeg found where Pinokio puts it, broken engines caught at boot

mrbizarrobd1878a5c08a
diff --git a/README.md b/README.md
index fea591f..f2e8ace 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@
</p>
-> **Current release: v4.15.0.** **Hailuo H3 has a Speed switch.** **Fast** runs a 3-step adapter, **Best** runs the full schedule — with **Upscale & Face Fix** after it, a finished HD clip is about **5 minutes** on an M4 Max (Draft 2.8 min + 2.5 min, 1280×768) instead of half an hour. **Room tone** puts one generated ambience bed under a whole film so cuts stop clicking. **The Editor stops rearranging itself**: Sound is a mode you choose, panels open when you ask, the Source monitor is a drop target again. **The render form is four closed sections** — Shot setup, LoRAs, After the render, Advanced — each printing what it hides (`Draft · 3s · Fast · ~2 min`), and **Batch is a section**: paste many prompts, or ask for N takes of one, each with its own seed. Fixed: a film with music over it no longer comes out with silent shots, and a crashed engine names the line of code. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).
+> **Current release: v4.15.1.** **The install fixes.** ffmpeg is found where Pinokio actually puts it, so a render that finished no longer fails to export. A **half-built engine environment** is caught at boot with a one-click repair instead of failing 30 seconds into every render. **Hailuo H3 checks its MLX before it starts** rather than dying eight minutes in. Captioning tells you when it was pointed at LTX-2.5's text encoder instead of the vision model. Under the hood, the panel now reports **which step of getting started failed** — because 55% of new installs never render and nothing said why. See [what is collected](docs/ANALYTICS.md). Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).
## Overview
diff --git a/VERSION b/VERSION
index 5c517bf..fb05571 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-4.15.0
+4.15.1
diff --git a/caption_with_gemma.py b/caption_with_gemma.py
index 394ee83..9a3c749 100644
--- a/caption_with_gemma.py
+++ b/caption_with_gemma.py
@@ -170,6 +170,32 @@ def main() -> int:
_emit("error", message=f"mlx_vlm not installed: {e}. "
f"Run: pip install --no-deps 'mlx-vlm==0.4.4'")
return 1
+ # WHICH GEMMA IS THIS. Two folders live side by side in mlx_models and
+ # only one of them can see an image:
+ #
+ # gemma-3-12b-it-4bit model_type "gemma3" <- the VLM
+ # gemma4-12b-ltx25-q4 model_type "gemma4_unified" <- LTX-2.5's text
+ # encoder, no
+ # vision tower
+ #
+ # Handing the second one to mlx_vlm produces "Model type gemma4_unified
+ # not supported" — 19 captioning failures across 4 installs and five
+ # releases in the fleet, and nothing in that sentence says which folder
+ # was wrong or which one is right. One read of config.json does.
+ try:
+ _cfg = json.loads((gemma_path / "config.json").read_text())
+ _mtype = str(_cfg.get("model_type") or "")
+ except Exception: # noqa: BLE001
+ _mtype = ""
+ if _mtype and not _mtype.startswith("gemma3"):
+ _emit("error",
+ message=f"{gemma_path.name} is not the captioning model — its "
+ f"config says model_type \"{_mtype}\", which has no "
+ f"vision tower (gemma4-12b-ltx25-q4 is LTX-2.5's text "
+ f"encoder). Captioning needs gemma-3-12b-it-4bit. "
+ f"Run /train/preflight from the panel to fetch it, or "
+ f"point LTX_GEMMA_PATH at that folder.")
+ return 1
try:
model, processor = load(str(gemma_path))
except Exception as e:
diff --git a/docs/ANALYTICS.md b/docs/ANALYTICS.md
index 73b802b..9416939 100644
--- a/docs/ANALYTICS.md
+++ b/docs/ANALYTICS.md
@@ -18,7 +18,7 @@ character names.
| **What identifies you** | One random UUID, generated on your Mac, tied to nothing |
| **How often** | Once ever when the install is new, once per panel start, plus once per finished render. No heartbeats |
| **Where it goes** | PostHog (`us.i.posthog.com`), or a self-hosted endpoint you choose |
-| **Location** | None sent, and every event tells the receiver not to derive one from your IP — [details](#location) |
+| **Location** | None sent. Country and city are *derived* from the address your computer connects from, and kept — [details](#location) |
| **How to turn it off** | Settings → *Anonymous usage analytics* → **Turn off**, or `PHOSPHENE_ANALYTICS_DISABLED=1` |
| **Default** | ON, and the shipped build carries a working key — see below |
| **What you can inspect** | `state/usage-log.jsonl` — a plain-text copy of everything the panel sends |
@@ -89,35 +89,62 @@ Settings.
## Location
The panel sends no location field of any kind — no country, no city, no region,
-no timezone, no coordinates, no locale. It never has.
+no timezone, no coordinates, no locale. It never has, and the guard suite fails
+the build if one ever appears in a payload.
-That on its own is not the whole promise, because a receiver can *derive* a
-location from where a request came from, and PostHog does that by default. So
-every event now also carries two instructions telling it not to:
+**That is not the whole story, and this page states the rest plainly.** A
+receiver can *derive* a location from the address a request arrives from, and
+PostHog does that by default. **Since 2026-09-19 — the first build after
+v4.15.0 — Phosphene lets it.** Every event is therefore stored with:
-| Property | Value | What it does |
-|---|---|---|
-| `$geoip_disable` | `true` | PostHog's GeoIP step returns immediately instead of deriving country, city, subdivision, timezone and the city's coordinates |
-| `$ip` | `"0.0.0.0"` | PostHog copies the connecting address into the event's `$ip` property only when the event didn't bring its own. Bringing one keeps the real address off the stored event |
-
-**What that does not do, said plainly:** the request still arrives over TCP from
-a real address, and nothing inside a request body can change that — that is how
-HTTP works, for this panel and for every other program on your Mac. What the
-two flags control is what the receiver is instructed to *derive* from that
-address and *store* on the event. Discarding it at the edge as well is a
-setting on the receiving project, not something this source tree can promise
-you, so this page does not.
-
-If you want the connection itself gone: turn analytics off, or point
-`PHOSPHENE_ANALYTICS_HOST` at a receiver you run (below).
-
-**A note on `0.0.0.0`, since it looks arbitrary.** The obvious spelling —
-sending `$ip: null` — does nothing at all: PostHog's ingest fills the property
-in when the event's value is *falsy*, so a null is silently replaced by your
-real address. It has to be a non-empty string. `127.0.0.1` would be the natural
-choice and is the wrong one: PostHog rewrites loopback and `192.168.*` to a real
-address in Sweden as a local-development convenience, which would invent a
-location the day the disable flag ever went missing.
+| Derived by the receiver | Example |
+|---|---|
+| `$geoip_country_code`, `$geoip_country_name` | `DE`, `Germany` |
+| `$geoip_city_name`, `$geoip_subdivision_1_name` | `Hamburg`, `Hamburg` |
+| `$geoip_postal_code` | `20095` |
+| `$geoip_latitude`, `$geoip_longitude` | the city's centre, not your building |
+| `$geoip_time_zone` | `Europe/Berlin` |
+
+**Why.** Phosphene is a Mac app that renders video locally, and the fleet view
+exists to answer "which build is broken, and on what hardware". *Where* turns
+out to belong to that question: a failure that is really a regional mirror, a
+Hugging Face route, or a locale-dependent path shows up as noise without it.
+The maintainer chose to keep it. This page's job is to tell you, not to talk
+you into it.
+
+**What is NOT stored: the address itself.** The receiving project has *Discard
+client IP data* turned on, which runs the GeoIP step first and then drops the
+address — so the country and city above are kept and the IP that produced them
+is not. Being honest about the shape of that promise: it is a setting on the
+receiving project, not a field inside a request body, so it is something this
+source tree can *state* and cannot *prove to you*. What the source tree can
+prove is the other half — that the panel itself sends nothing, and reads no
+locale, timezone or coordinate off your Mac to send. That is what the tests
+pin.
+
+**A person is still not built.** `$process_person_profile: false` rides on every
+event, so none of this accumulates into a person record. It is a property of an
+event, next to a random install id.
+
+> **Correction, 2026-09-18 — the history, which is not tidy.** An audit of the
+> receiving project found **1,304 stored events carrying a derived location**
+> across **85 installs**, from builds that shipped *before* the suppression flags
+> existed (chiefly 2026-08-09 → 2026-08-15, plus a reverted 2025-05 build).
+> Between **2026-08-12 and v4.15.0** the panel did suppress derivation, on every
+> event, with two properties: `$geoip_disable: true`, and `$ip: "0.0.0.0"` to
+> occupy the property PostHog would otherwise fill from the socket. Those flags
+> are **gone as of 2026-09-19**, deliberately and with this page rewritten in the
+> same commit — which is the only way a promise like this is worth anything. The
+> older rows were left in place. Three postures in thirteen months is the real
+> record, so it is written down instead of smoothed over.
+
+**If you don't want any of it**, there are three off switches and they are the
+same ones as for every other event on this page: the Settings toggle,
+`PHOSPHENE_ANALYTICS_DISABLED=1`, or `PHOSPHENE_ANALYTICS_HOST` pointed at a
+receiver you run. Turning analytics off means no request is made at all, which
+is the only thing that makes a connecting address a non-question — nothing
+inside a request body ever could, for this panel or for any other program on
+your Mac.
---
@@ -205,7 +232,7 @@ Same as `render_completed` (minus `first_render`), plus:
| Field | Type | Example | Notes |
|---|---|---|---|
-| `error_class` | string | `"metal_watchdog"` | **A closed 17-value taxonomy** (`refused`, `oom_jetsam`, `metal_watchdog`, `native_crash`, `helper_start_timeout`, `helper_exit`, `model_missing`, `model_corrupt`, `download_failed`, `venv_broken`, `bad_params`, `input_missing`, `disk_full`, `export_failed`, `timeout`, `cancelled_race`, `other`). Classification runs on the original error text locally; **only the class leaves the machine.** `refused` is in the taxonomy but never rides on a `render_failed` event — it is the value that routes the event to `render_refused` instead, below |
+| `error_class` | string | `"metal_watchdog"` | **A closed 19-value taxonomy** (`refused`, `oom_jetsam`, `metal_watchdog`, `metal_oom`, `native_crash`, `helper_start_timeout`, `helper_exit`, `model_missing`, `model_corrupt`, `download_failed`, `venv_broken`, `bad_params`, `input_missing`, `file_missing`, `disk_full`, `export_failed`, `timeout`, `cancelled_race`, `other`). Classification runs on the original error text locally; **only the class leaves the machine.** `refused` is in the taxonomy but never rides on a `render_failed` event — it is the value that routes the event to `render_refused` instead, below |
| `error_fingerprint` | string | `"a3f09c21e7b4"` | Only when `error_class` is `other`: 12 hex chars of the SHA-256 of the already-scrubbed first line. Lets "the same unknown error, 17 times, all on M1 Max" be counted without transmitting the text — the readable line stays in your own `state/usage-log.jsonl` |
| `error_signature` | string | `"RuntimeError: helper exited before first frame"` | **The only free-text field the panel sends**, see the scrubbing rules below. Kept for ONE transition release alongside `error_class`, then removed |
@@ -278,6 +305,52 @@ How the update pop-up and banner are answered — the only way to know whether t
| `action` | string | `"later"` | **Closed vocabulary**: `shown`, `update_now`, `later` (the pop-up), `banner_update`, `banner_later` (the banner), `restart_needed` (the "Restart to finish update" pill, once per page load) |
| `version` | string | `"4.9.5"` | The version that was prompted |
+### `install_step` (v4.16.0)
+
+The dark half of the funnel. In the 14 days to 2026-09-19, **577 installs
+reported in and only 44.5% of them ever rendered** — 278 booted once and were
+never seen again. Between `app_installed` and the first `render_*` the panel
+said nothing, so *"they tried and it was broken"* and *"they looked and left"*
+were the same shape in the data. These events tell those apart.
+
+**At most one event per step per install, ever** — with one deliberate
+exception: `engine_env` reports again when its answer *changes*, because a
+broken environment that gets repaired is the most useful transition on this
+event. This is not a heartbeat and the panel still has none.
+
+| prop | type | example | why |
+|---|---|---|---|
+| `step` | string | `"engine_env"` | **Closed vocabulary**: `first_boot` (the panel came up), `engine_env` (its Python environment can run a render), `weights_check` (the base model is on disk), `first_queue` (a job reached the worker — someone pressed Render) |
+| `outcome` | string | `"failed"` | **Closed vocabulary**: `started`, `ok`, `failed`, `skipped` |
+| `error_class` | string | `"venv_broken"` | Only on `failed`, and only from the same closed taxonomy `render_failed` uses |
+| `version` | string | `"4.16.0"` | The build that reported it |
+
+**What this event deliberately does not carry**, and why: no path, no file
+name, no URL, no host, no byte count, and — unlike `render_failed` — **no
+`error_signature` and no `error_fingerprint`**. An install-time error line is
+the likeliest place in the whole product for a username-bearing path to
+survive scrubbing, and the class on its own answers the question the event
+exists to ask.
+
+### `update_outcome` (v4.16.0)
+
+Whether pressing **Update** actually produced a new version. 139 people
+pressed it in 14 days; 116 ran something new afterwards and **23 did not**, and
+nothing in the data could say whether the pull failed or the app never
+restarted. `app_updated` cannot answer it by construction — a failed update
+emits nothing at all.
+
+The panel records the version you pressed Update *on* (one string, in local
+settings) and reads it on the next boot: a different version means it landed,
+the same version means it didn't. The marker is cleared when it is read.
+
+| prop | type | example | why |
+|---|---|---|---|
+| `stage` | string | `"restart_pending"` | **Closed vocabulary**: `running_new` (a new version is running), `restart_pending` (same version came back) |
+| `outcome` | string | `"failed"` | **Closed vocabulary**: `ok`, `failed` |
+| `from_version` | string | `"4.15.0"` | The version Update was pressed on |
+| `version` | string | `"4.15.0"` | The version now running |
+
### `broadcast_seen` (v4.9.7)
One event when a developer broadcast (`BROADCAST.json`) is acknowledged. No text — which message is derivable from the date.
@@ -577,8 +650,10 @@ refusal/fault fork (a refusal is `render_refused` and never a `render_failed`
carrying `error_class: refused`; a real crash is untouched; refusals stay out
of the local error rate), prompt/path
non-leakage (including a prompt quoted inside an exception), forbidden-key
-dropping, the geo-disable and `$ip` flags riding on *every* event type and
-being un-overridable by a call site, bucketing, log rotation and the local
+dropping, the receiver directives riding on *every* event type, no `$ip` and no
+`$geoip_*` key leaving the panel (either would empty the country column or
+invent one) and the whole `$` namespace being unreachable from a call site,
+bucketing, log rotation and the local
aggregates.
To watch it live, tail the local log while you render:
diff --git a/docs/STATE.md b/docs/STATE.md
index 45f7b73..b5c5f10 100644
--- a/docs/STATE.md
+++ b/docs/STATE.md
@@ -1,5 +1,30 @@
# Phosphene — project state, history, open work
+> **🔧 2026-09-20 — v4.15.1: the fleet's own bug list, fixed. Five faults, every one of them found in the data rather than reported.** The owner's instruction was "check fleet, github and pinokio and solve all bugs, ship fixes together with analytics improvements". GitHub and Pinokio turned out to be almost clean — every open bug thread there is already fixed and waiting on a reporter to confirm (the H3 Q8 SIGKILL in 4.12.2, the dead `madebyollin/taeh3` repo in 4.12.x, the YuE2 MPS flags in 4.14.3, the 48 GB bf16 pick in 4.12.4). **The fleet was not clean**, and nobody had filed any of this.
+>
+> - **ffmpeg was never where we looked.** Five installs across 4.5.0 → 4.15.0 finished a render and failed the export with "ffmpeg not found on PATH". Every engine spawn has prepended `FFMPEG_BIN` to the child's PATH for months — the miss was one level up: `_resolve_ffmpeg()` knew `pinokio/bin/ffmpeg-env` and nothing else, while **Pinokio's own bootstrap installs `ffmpeg=8.1.2` into its conda prefix** (`bin/miniforge/bin`) and unpacks a private Homebrew beside it. Proven on this Mac: that binary is sitting at `~/pinokio/bin/miniforge/bin/ffmpeg`, 443 KB, dated August. Now searched, along with `bin/miniconda/bin` and `bin/homebrew/bin`; **ffprobe resolves independently** instead of assuming the sibling path exists (the fleet's bare "ffprobe failed:", 3 installs); and a boot warning names every folder that was searched.
+> - **A half-built venv looked perfectly healthy.** `ModuleNotFoundError: No module named 'ltx_pipelines_mlx'` on 4.13.0 / 4.14.3 / 4.15.0 — **one install hit it 28 renders in a row.** Both the panel and pinokio.js only ever checked the INTERPRETER, which in that shape exists; what is missing is the package pip writes last. So no repair was offered and nothing said what was wrong until half a minute into each render. `engine_env_fault()` probes the package directory, the panel refuses the spawn in one second with the fix in the sentence, and **pinokio.js now offers "Repair Phosphene engine" for this shape too**.
+> - **H3 died eight minutes in on an MLX it cannot use.** `seed(): incompatible function arguments` and `repeat(): …` on 4.14.3 and 4.15.0 — MLX signature drift, neither message naming MLX. H3 needs mlx>=0.32; Phosphene pins mlx==0.31.1 (0.31.2 regresses LTX audio by 22 dB), which is why H3 has its own venv. `h3_mlx_fault()` reads the version off the `.dist-info` directory name — a listing, not a subprocess, because this is on the path to a render — and refuses **before** the LTX helper is killed and before any GPU time is spent.
+> - **The captioner would not say which Gemma was wrong.** "Model type gemma4_unified not supported" — 19 events, 4 installs, five releases. Two folders sit side by side in `mlx_models` and only one has a vision tower: `gemma-3-12b-it-4bit` (`gemma3`) and `gemma4-12b-ltx25-q4` (`gemma4_unified`, LTX-2.5's TEXT encoder). mlx_vlm's message names neither the folder it was handed nor the one that works. One read of `config.json` does.
+> - **The redactor was eating our own prose.** `~` matched as a home directory on its own, so the fleet is full of signatures reading "this engine needs `<path>`32 GB free RAM" — `~32 GB` is how a person writes an approximation, and that line is the most useful one in the report. A home path always continues with a separator; `~(?=/)` now says so.
+>
+> **And the analytics half, which is the part that changes what we can see next time.** Both events were proposed on 2026-09-19 and approved with "ship fixes together with analytics improvements".
+>
+> - **`install_step` — the dark half of the funnel.** 577 installs in 14 days, **44.5% of them ever rendered**, 278 booted once and vanished. Between `app_installed` and the first render the panel said nothing, so "they tried and it was broken" and "they looked and left" were the same shape in the data. Four steps — `first_boot`, `engine_env`, `weights_check`, `first_queue` — each **once per install for all time**, with one deliberate exception: `engine_env` reports again when its answer CHANGES, because a broken environment that gets repaired is the most useful row on the event. `first_queue` is emitted from the worker, the one place eight queue-append call sites converge. No path, no file name, no URL, no host, no byte count, and — unlike `render_failed` — **no signature and no fingerprint**: an install-time error line is the likeliest place in the product for a username-bearing path to survive scrubbing, and the closed class answers the question on its own.
+> - **`update_outcome` — the 23 of 139.** People who pressed Update and never ran a new version. `app_updated` cannot show it by construction: a failed update emits nothing. The UI route records the version Update was pressed ON; the next boot compares and emits `running_new`/ok or `restart_pending`/failed, then clears the marker.
+> - Both are documented on `docs/ANALYTICS.md` in the same commit, and the dry-run suite's parity guard is what forced that — it failed the build the moment the events existed without the page.
+>
+> **Tests: 45 new across four files**, and the two fault messages are pinned against the error taxonomy itself, because wording is load-bearing twice over here: "missing (" belongs to `model_missing`, "download"/"fetch" to `download_failed`, and either one files a venv fault under the wrong heading in the fleet view. Both were caught that way, against the live classifier, not guessed.
+
+> **🌍 2026-09-19 — THE GEOIP KILL, REVERSED ON PURPOSE: country and city are now kept, and the page that promised otherwise changed in the same commit (UNRELEASED, dev).** The owner's ruling, verbatim: *"no need to discar i think countrya nd city is fine i wnat to know actualy."* The 2026-08-12 entry below is therefore superseded — not deleted, because three postures in thirteen months is the real record and this file keeps records.
+>
+> - **Both suppression properties are gone from every event.** `$geoip_disable: True` switched the receiver's GeoIP step off; `$ip: "0.0.0.0"` occupied the property so the connecting address was never copied onto the stored event — and **also killed the lookup**, because the step reads `properties.$ip` and resolves nothing from `0.0.0.0`. That second effect is the one that would have been easy to miss: dropping only the disable flag would have looked like a fix and changed nothing. `_ANALYTICS_RECEIVER_DIRECTIVES` is now one key, `$process_person_profile: False` — counts, still not people.
+> - **The `$` namespace is closed to call sites** (`_analytics_clean_props`). While the directives occupied `$ip` and `$geoip_disable`, spreading them last in `_analytics_post` was sufficient armour. It no longer is: a props dict built from job params in code this module does not own could now put a **`$ip` on the wire and spoof the address the GeoIP step reads**, or a `$geoip_city_name` and invent a location outright. Every `$`-prefixed key from a call site is dropped; ours are spread afterwards.
+> - **The owner's note had the project setting backwards, and the correction matters.** *Discard client IP data* does NOT suppress geo — PostHog runs the GeoIP transformation FIRST and discards the address after, so **on** = country and city stored, address not stored; **off** = the same country and city *plus* a retained IP on every event. It is still the owner's click (project settings are not ours to change) but it now costs him nothing he wants. `plugin-advanced-geoip` is the lever if he later wants postal code and city-centre coordinates dropped too — GeoIP stores seven properties, and he asked for two.
+> - **`docs/ANALYTICS.md` Location section rewritten in the same commit**, which is the only thing that makes a promise like this worth reading: it names all seven stored `$geoip_*` properties with examples, says why they are kept, says the address is discarded at the edge **and that a source tree can state that and cannot prove it**, keeps the 1,304-event / 85-install history (the owner chose to keep those rows), and points at the three off switches. The summary table row changed too — it used to read *"None sent, and every event tells the receiver not to derive one"*.
+> - **Guards rewritten, not relaxed, and mutation-tested 3/3 red:** put `$ip` back → red; put `$geoip_disable` back → red; reopen the `$` namespace → red. The surviving invariant is the one that was always the real one: **the panel sends no location field and reads no locale, timezone or coordinate off the Mac.** A derived country is a fact about a connection; a sent one would be a fact this program went looking for.
+> - **Caught a stale fixture the pytest run cannot see.** `scripts/test_analytics_dryrun.py` runs standalone (56 tests), so last night's `cancelled_race` tightening in `9c66022` left `"cancel requested, landing as failed"` classifying as `other` with nobody watching. Replaced with the three strings the panel actually raises (`"H3 render stopped early at the next forward boundary"`, `"cancelled by /stop (rc=-9)"`, `"stopped before the join"`) plus a negative case pinning that a message merely containing the word is **not** a cancel. 56/56.
+
> **🧹 2026-09-18 — The render form is four closed sections (UNRELEASED, dev; waits for owner use).**
> Owner: "The H3 layout is kind of a bit too much … you have a pill for speed, quality, length, and all that. That
> should be inside one section that you can compress and decompress, like we have LoRAs and Customize, because it is
diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.py
index 1240620..941649a 100644
--- a/mlx_ltx_panel.py
+++ b/mlx_ltx_panel.py
@@ -170,6 +170,59 @@ HELPER_SCRIPT = Path(os.environ.get("LTX_HELPER_SCRIPT", str(ROOT / "mlx_warm_he
# at `env/bin/python3.11`. The warning + remediation line saves a
# round-trip on every install variation that lands in a non-standard
# layout.
+def engine_env_fault() -> str:
+ """"" when the engine environment can run a render, else the reason.
+
+ TWO SHAPES, and only the first one was ever checked. The interpreter can
+ be MISSING (a Reset that took the venv, a moved install) — that one has
+ been caught at boot since issue #5. Or the interpreter can be PRESENT with
+ the engine package never installed into it, which is what a pip step that
+ failed halfway leaves behind. The second shape looks healthy to every file
+ check we had, and it fails 30 seconds into a render instead: three installs
+ in the fleet on 4.13.0 / 4.14.3 / 4.15.0, one of them 28 renders in a row,
+ every one of them dying at helper start with
+
+ ModuleNotFoundError: No module named 'ltx_pipelines_mlx'
+
+ Checking the package DIRECTORY rather than importing keeps this cheap
+ enough to run at boot and before every helper spawn: pip writes that
+ directory last, so its absence is the same evidence an import would give.
+ """
+ if not HELPER_PYTHON.is_file():
+ return (f"the engine venv has no Python at {HELPER_PYTHON} — the "
+ f"ltx-2-mlx environment was never built, or something removed it")
+ root = HELPER_PYTHON.parent.parent
+ # ONLY JUDGE A VENV. `LTX_HELPER_PYTHON` exists so a developer can point
+ # the panel at an interpreter somewhere else entirely, and for
+ # /opt/homebrew/bin/python3.11 this arithmetic lands on /opt/homebrew —
+ # whose site-packages legitimately has no ltx_pipelines_mlx in it. Faulting
+ # there would block renders on a working machine. `pyvenv.cfg` is what
+ # makes a directory a venv, so its absence means "not ours to judge".
+ if not (root / "pyvenv.cfg").is_file():
+ return ""
+ try:
+ pkgs = [d for d in (root / "lib").glob("python*/site-packages") if d.is_dir()]
+ except OSError:
+ pkgs = []
+ if pkgs and not any((d / "ltx_pipelines_mlx").exists() for d in pkgs):
+ return ("the engine venv has a Python but no packages — "
+ "ltx_pipelines_mlx never finished installing into it")
+ return ""
+
+
+# Wording is load-bearing twice over. It must name the exact sidebar entry a
+# user has to click, and it must not contain a word that an EARLIER row of the
+# error taxonomy claims: "missing (" belongs to model_missing and "download" to
+# download_failed, and either one would file this fault under the wrong heading
+# in the fleet view, and "fetch" belongs to download_failed (checked against
+# the live classifier, not guessed). The phrase that has to survive is "venv",
+# which is how venv_broken finds it.
+ENGINE_ENV_REPAIR = (
+ "Fix it from the Pinokio sidebar: \"Repair Phosphene engine (models "
+ "kept)\" rebuilds the venv and keeps every model you have. Update works too. "
+ "Renders cannot run until it is repaired."
+)
+
if not HELPER_PYTHON.is_file():
sys.stderr.write(
f"WARN: helper Python not found at {HELPER_PYTHON}\n"
@@ -177,24 +230,88 @@ if not HELPER_PYTHON.is_file():
f" Renders will fail until either path exists, or LTX_HELPER_PYTHON\n"
f" is exported pointing at a working python3.11 with mlx + ltx-pipelines.\n"
)
-# ffmpeg: env var → PATH → Pinokio bundled → Homebrew → /usr/local. First match wins.
+else:
+ _env_fault = engine_env_fault()
+ if _env_fault:
+ sys.stderr.write(f"WARN: {_env_fault}.\n {ENGINE_ENV_REPAIR}\n")
+def _pinokio_bin_dirs() -> list:
+ """Pinokio's own tool folders, newest layout first.
+
+ Pinokio's bootstrap installs `ffmpeg`, `git`, `node` and friends into a
+ conda prefix under its home (`bin/miniforge/bin`, `bin/miniconda/bin` on
+ older installs) and unpacks a private Homebrew beside it. None of those
+ are guaranteed to be on PATH when it launches an app — which is the whole
+ reason this helper exists."""
+ home = (Path(os.environ["PINOKIO_HOME"]).expanduser()
+ if os.environ.get("PINOKIO_HOME")
+ else ROOT.parent.parent if ROOT.parent.name == "api" else None)
+ dirs = []
+ for base in (home, Path.home() / "pinokio"):
+ if not base:
+ continue
+ dirs += [base / d for d in ("bin/ffmpeg-env/bin", "bin/miniforge/bin",
+ "bin/miniconda/bin", "bin/homebrew/bin")]
+ # PINOKIO_HOME is usually ~/pinokio, so both loops find the same folders.
+ # Deduped, order preserved, because this list is also printed to the user.
+ return list(dict.fromkeys(dirs))
+
+
+# ffmpeg: env var -> PATH -> Pinokio's own tool folders -> Homebrew -> /usr/local.
+# First match wins.
+#
+# THE MISS THIS EXISTS TO CLOSE (fleet, 2026-09-20): five installs across
+# 4.5.0 -> 4.15.0 failed their export with "ffmpeg not found on PATH", and the
+# H3 runner spawn has prepended FFMPEG_BIN to the child's PATH the whole time.
+# The bug was one level up: this resolver only knew `bin/ffmpeg-env`, while
+# Pinokio's bootstrap installs `ffmpeg=8.1.2` into its CONDA prefix
+# (`bin/miniforge/bin`) and unpacks a private Homebrew next to it. On a Mac
+# with no system ffmpeg the resolver fell through to a `/usr/local` path that
+# does not exist, and every engine that shells out to it died — a working
+# binary sitting two directories away the whole time.
+def _resolve_tool(name: str, env_var: str) -> Path:
+ override = os.environ.get(env_var)
+ if override and Path(override).exists():
+ return Path(override)
+ found = shutil.which(name)
+ if found:
+ return Path(found)
+ for d in _pinokio_bin_dirs() + [Path("/opt/homebrew/bin"),
+ Path("/usr/local/bin"),
+ Path("/opt/local/bin")]:
+ cand = d / name
+ if cand.exists():
+ return cand
+ return Path("/usr/local/bin") / name # will fail at runtime if missing
+
+
def _resolve_ffmpeg() -> Path:
- candidates = [
- os.environ.get("LTX_FFMPEG"),
- shutil.which("ffmpeg"),
- str(Path.home() / "pinokio/bin/ffmpeg-env/bin/ffmpeg"),
- "/opt/homebrew/bin/ffmpeg",
- "/usr/local/bin/ffmpeg",
- ]
- for c in candidates:
- if c and Path(c).exists():
- return Path(c)
- return Path("/usr/local/bin/ffmpeg") # last-resort default; will fail at runtime if missing
+ return _resolve_tool("ffmpeg", "LTX_FFMPEG")
+
+
+def _resolve_ffprobe(ffmpeg: Path) -> Path:
+ """ffprobe ships beside ffmpeg in every distribution we support, so that
+ is the first guess — but only if it is actually there. A Mac that got its
+ ffmpeg from one place and its ffprobe from another used to end up with a
+ path that never existed, and the panel's duration probe failed with a bare
+ "ffprobe failed:" (3 installs in the fleet)."""
+ beside = ffmpeg.parent / "ffprobe"
+ return beside if beside.exists() else _resolve_tool("ffprobe", "LTX_FFPROBE")
FFMPEG = _resolve_ffmpeg()
FFMPEG_BIN = FFMPEG.parent
-FFPROBE = FFMPEG.parent / "ffprobe" # ships next to ffmpeg in every distribution we support
+FFPROBE = _resolve_ffprobe(FFMPEG)
+# Say it at boot, the way a missing helper venv is said above: an export that
+# dies 20 minutes into a render is a terrible place to learn this.
+if not FFMPEG.exists():
+ _looked = ", ".join(str(d) for d in _pinokio_bin_dirs())
+ sys.stderr.write(
+ "WARN: ffmpeg not found. Looked on PATH and in: "
+ + _looked + ", /opt/homebrew/bin, /usr/local/bin, /opt/local/bin.\n"
+ " Renders will finish and then fail to export.\n"
+ " Fix: install ffmpeg, or export LTX_FFMPEG=/path/to/ffmpeg.\n"
+ )
+
Q4_LOCAL_PATH = MODELS_DIR / "ltx-2.3-mlx-q4"
MODEL_ID = os.environ.get(
@@ -10167,6 +10284,87 @@ def h3_prune_prompt_cache(keep: int = H3_PROMPT_CACHE_KEEP) -> int:
return dropped
+H3_MLX_MIN = (0, 32)
+
+
+def _venv_dist_version(python: Path, dist: str) -> tuple | None:
+ """Version of an installed distribution, read off its `.dist-info`
+ directory name. `None` when the package isn't there at all.
+
+ A directory listing rather than a subprocess on purpose: this runs on the
+ path to a render, and `python -c "import mlx"` costs a second and loads
+ Metal. pip writes the dist-info name from the wheel's own metadata, so it
+ is the same string `importlib.metadata.version()` would return."""
+ try:
+ for site in (python.parent.parent / "lib").glob("python*/site-packages"):
+ for d in site.glob(f"{dist}-*.dist-info"):
+ raw = d.name[len(dist) + 1:-len(".dist-info")]
+ parts = []
+ for chunk in raw.split(".")[:3]:
+ # LEADING digits only. "0.33.0rc1" is 0.33.0, not 0.33.1 —
+ # sweeping up every digit in the chunk turns a release
+ # candidate into a higher version than the release.
+ m = re.match(r"\d+", chunk)
+ if not m:
+ break
+ parts.append(int(m.group(0)))
+ if parts:
+ return tuple(parts)
+ except Exception: # noqa: BLE001
+ return None
+ return None
+
+
+def h3_mlx_fault() -> str:
+ """"" when the H3 venv's MLX can run the H3 runner, else the reason.
+
+ THE FAILURE THIS CATCHES, from the fleet on 4.14.3 and 4.15.0:
+
+ TypeError: seed(): incompatible function arguments
+ TypeError: repeat(): incompatible function arguments
+
+ Both are MLX signature drift, and both arrive ~8 minutes into a render
+ with nothing in them that names MLX as the problem. H3 needs mlx>=0.32
+ while Phosphene's own engine pins mlx==0.31.1 (0.31.2 regresses LTX audio
+ by 22 dB) — which is exactly why H3 has a separate venv. A venv that ends
+ up holding the older MLX anyway — a stray `pip install` into it, a package
+ that pulled MLX down as a dependency — produces those two lines and
+ nothing else.
+
+ Phrasing note, same as engine_env_fault: the word "venv" is what files
+ this under venv_broken in the fleet view, and "missing (" / "download" /
+ "fetch" would file it somewhere wrong."""
+ py = _h3_python()
+ if py is None:
+ return "" # not installed is h3_paths()'s story, not this one
+ got = _venv_dist_version(py, "mlx")
+ if got is None:
+ # UNKNOWN IS NOT BROKEN. `LTX_H3_PYTHON` can point at an interpreter
+ # in a layout this probe does not understand, and a dist-info we
+ # cannot find is not evidence of a package that is not there — it is
+ # the absence of evidence. Refusing here would block H3 on a machine
+ # where it works. Only say something when the venv is legible and the
+ # package is genuinely absent from it.
+ if not any((py.parent.parent / "lib").glob("python*/site-packages")):
+ return ""
+ return ("the Hailuo H3 venv has no MLX installed — its packages never "
+ "finished landing")
+ if got < H3_MLX_MIN:
+ have = ".".join(str(n) for n in got)
+ want = ".".join(str(n) for n in H3_MLX_MIN)
+ return (f"the Hailuo H3 venv has MLX {have}, and the H3 runner needs "
+ f"{want} or newer — on {have} it dies partway through a render "
+ f"with \"seed(): incompatible function arguments\"")
+ return ""
+
+
+H3_MLX_REPAIR = (
+ "Fix it from the Pinokio sidebar: \"Repair Hailuo H3\" reinstalls that "
+ "venv's packages and keeps every weight you have. LTX renders are "
+ "unaffected — the two engines keep separate venvs for this exact reason."
+)
+
+
def h3_paths() -> dict:
"""Resolve every H3 component. Never raises — reports what's missing so
the UI can render an honest install card instead of a stack trace."""
@@ -13633,15 +13831,27 @@ ANALYTICS_API_HOST_DEFAULT = "https://us.posthog.com"
ANALYTICS_TIMEOUT_SEC = 2.0
ANALYTICS_STR_MAX = 120
-# The value we put in the event's own $ip property so PostHog never writes the
-# connecting address there. It has to be a TRUTHY string: PostHog's ingest fills
-# properties.$ip from the socket only when the event didn't carry one
-# (`if (!properties['$ip'] && event.ip)`), so `None` — the obvious spelling — is
-# falsy and gets silently replaced by the real address. 0.0.0.0 is also chosen
-# over the more natural 127.0.0.1: the GeoIP transformation rewrites loopback
-# and 192.168.* to a real Swedish address for local-dev convenience, which would
-# manufacture a location if the disable flag below were ever dropped.
-ANALYTICS_IP_PLACEHOLDER = "0.0.0.0"
+# Location is DERIVED BY THE RECEIVER, on purpose, since 2026-09-19. The panel
+# still sends no location field of its own and never has — it doesn't have to:
+# PostHog resolves country and city from the connecting address, and the fleet
+# view wants them (which build is failing, and where). Two properties used to
+# suppress that, both shipped 2026-08-12 → 2026-09-19, and both are now gone
+# deliberately:
+#
+# $geoip_disable: True switched the receiver's GeoIP step off outright.
+# $ip: "0.0.0.0" occupied the property so the connecting address was
+# never copied onto the stored event — which ALSO
+# killed the lookup, because the step reads
+# properties.$ip and resolves nothing from 0.0.0.0.
+#
+# Reintroducing either silently empties the country and city columns for every
+# install, so neither may come back without changing docs/ANALYTICS.md in the
+# same commit. What replaces them is a PROJECT-side setting on the receiver
+# (PostHog → Project settings → *Discard client IP data*): PostHog runs the
+# GeoIP transformation first and discards the address afterwards, so country
+# and city are stored and the address is not. That is a setting on the project,
+# not a fact this source tree can assert — same discipline as everywhere else
+# in this module: state it, say who can check it, promise nothing.
# Instructions to the receiver, attached to every event next to our own
# properties. They are spread LAST in _analytics_post so no call site can
@@ -13650,8 +13860,6 @@ ANALYTICS_IP_PLACEHOLDER = "0.0.0.0"
# turning red. See _analytics_post for what each one does and does not buy.
_ANALYTICS_RECEIVER_DIRECTIVES = {
"$process_person_profile": False,
- "$geoip_disable": True,
- "$ip": ANALYTICS_IP_PLACEHOLDER,
}
# Local mirror. Written for every captured event whether or not a PostHog
@@ -13683,8 +13891,17 @@ _ANALYTICS_FORBIDDEN_KEYS = frozenset({
# Absolute-path shapes to redact from error signatures. Applied in order:
# the anchored pass catches the macOS roots a Phosphene error is likely to
# quote, the generic pass catches anything else with 2+ path segments.
+#
+# `~` ONLY COUNTS AS A HOME DIRECTORY WHEN A SLASH FOLLOWS IT. It used to
+# match on its own, and the casualty was our own prose: the fleet is full of
+# signatures reading "this engine needs <path>32 GB free RAM" and "H3 needs
+# about <path>64 GB", because "~32 GB" is how a person writes an approximate
+# number. Both of those are panel copy, both are the most useful line in the
+# report, and the redactor was quietly eating the tilde out of the middle of
+# them. A home path always continues with a separator; an approximation never
+# does.
_ANALYTICS_PATH_RES = (
- re.compile(r"(?:~|/(?:Users|home|private|var|tmp|opt|Volumes|Applications"
+ re.compile(r"(?:~(?=/)|/(?:Users|home|private|var|tmp|opt|Volumes|Applications"
r"|Library|System))(?:/[^\s'\"<>,;)\]]*)*"),
re.compile(r"(?:/[\w.+\-]+){2,}/?"),
)
@@ -13835,6 +14052,15 @@ def _analytics_clean_props(props: dict) -> dict:
k = str(key)
if k.lower() in _ANALYTICS_FORBIDDEN_KEYS:
continue
+ if k.startswith("$"):
+ continue # the receiver's namespace belongs to
+ # _ANALYTICS_RECEIVER_DIRECTIVES, not to a call site.
+ # This mattered less while the directives occupied $ip
+ # and $geoip_disable and could not be overridden; now
+ # that they don't, a stray "$ip" from a params dict
+ # would spoof the address the GeoIP step reads and a
+ # stray "$geoip_city_name" would invent a location.
+ # Ours are spread in _analytics_post, after this pass.
if isinstance(val, bool) or isinstance(val, int) or isinstance(val, float):
out[k] = val
elif isinstance(val, str):
@@ -13995,31 +14221,25 @@ def _usage_log_append(record: dict) -> None:
def _analytics_post(payload: dict) -> None:
"""Single-event POST to the PostHog capture API. Never raises.
- Every event carries _ANALYTICS_RECEIVER_DIRECTIVES — three instructions
- to the receiver, verified against PostHog's ingest source rather than
+ Every event carries _ANALYTICS_RECEIVER_DIRECTIVES — one instruction to
+ the receiver, verified against PostHog's ingest source rather than
assumed:
$process_person_profile: False
Don't build a person record for the install id. We want counts,
not people.
- $geoip_disable: True
- PostHog's GeoIP transformation returns immediately instead of
- deriving country, city, subdivision, timezone and the city's
- coordinates. Its first statement is literally
- `if (event.properties?.$geoip_disable or empty(...$ip))`, and this
- is the same property posthog-python sets for disable_geoip=True.
- $ip: ANALYTICS_IP_PLACEHOLDER
- Ingest copies the connecting address into properties.$ip only when
- the event didn't bring its own, so sending one is the only way to
- keep the real address off the stored event. It must be truthy —
- see ANALYTICS_IP_PLACEHOLDER for why `None` silently does nothing.
-
- What this does NOT do, and what docs/ANALYTICS.md is careful not to
- claim: the request still arrives over TCP from a real address, and no
- field inside a request body can change that. What the panel controls is
- what the receiver is instructed to derive from it and store on the
- event. Discarding it at the edge as well is a project-side setting, not
- something this code can assert."""
+
+ What is deliberately NOT here any more, as of 2026-09-19: `$geoip_disable`
+ and a placeholder `$ip`. Both shipped from 2026-08-12 and both stopped the
+ receiver deriving a location — the first by switching the GeoIP step off,
+ the second because that step reads properties.$ip and resolves nothing from
+ 0.0.0.0. The fleet view wants country and city, so the panel now lets the
+ lookup happen and adds no location field of its own.
+
+ What this does NOT do, and what docs/ANALYTICS.md is careful to say in the
+ same words: the panel cannot promise what the receiver keeps. The address
+ is dropped after enrichment by a project-side setting (*Discard client IP
+ data*), which is a setting on the project, not a field in this body."""
key = _analytics_key()
if not key:
return # only reachable if a fork blanks the shipped key; the guard
@@ -14084,6 +14304,12 @@ _ANALYTICS_EVENTS = (
# v4.9.7
"feature_used", "app_updated", "update_prompt", "broadcast_seen",
"queue_paused_breaker",
+ # v4.16.0 — the dark half of the funnel. 577 installs in 14 days and only
+ # 44.5% of them ever rendered; 278 booted once and were never seen again.
+ # Between `app_installed` and the first `render_*` the panel said nothing
+ # at all, so "they tried and it was broken" and "they looked and left"
+ # were the same shape in the data. These two close that.
+ "install_step", "update_outcome",
)
@@ -14185,10 +14411,106 @@ def _analytics_boot() -> None:
"pack": name, "from": was, "to": packs[name],
})
_settings_set_internal(analytics_last_packs=packs)
+ # ---- the install funnel, evaluated where the answers live ---------
+ # Ordered the way a new user meets them: the panel came up, its engine
+ # environment is usable, its weights are on disk. A person who never
+ # renders now leaves a trail that says WHICH of those three was false.
+ _analytics_install_step("first_boot", "ok")
+ _env_fault = engine_env_fault()
+ _analytics_install_step(
+ "engine_env", "failed" if _env_fault else "ok",
+ _analytics_error_class(_env_fault) if _env_fault else "")
+ _analytics_install_step(
+ "weights_check", "ok" if _analytics_weights_ready() else "failed")
+ # ---- did the last Update actually land ---------------------------
+ _pressed = str(get_settings().get("analytics_update_pressed") or "")
+ if _pressed:
+ _settings_set_internal(analytics_update_pressed="")
+ if _pressed != running_version():
+ _analytics_update_outcome("running_new", "ok", _pressed)
+ else:
+ # Pressed Update, came back on the same build. Either the pull
+ # failed or the app never actually restarted into a new one —
+ # both are the hole this event exists to measure.
+ _analytics_update_outcome("restart_pending", "failed", _pressed)
except Exception:
pass
+# Closed vocabularies. A value outside these never reaches the network: the
+# emitter drops it, and the dry-run suite fails the build if a call site
+# invents one. This is the same discipline as the error taxonomy, for the same
+# reason — a free-text step name is a field nobody can chart and a string
+# nobody reviewed.
+_INSTALL_STEPS = ("first_boot", "engine_env", "weights_check", "first_queue")
+_INSTALL_OUTCOMES = ("started", "ok", "failed", "skipped")
+_UPDATE_STAGES = ("running_new", "restart_pending")
+_UPDATE_OUTCOMES = ("ok", "failed")
+
+
+def _analytics_install_step(step: str, outcome: str, error_class: str = "") -> None:
+ """One step of getting from "installed" to "rendered".
+
+ AT MOST ONCE PER STEP PER INSTALL, and for `engine_env` once more each
+ time the answer CHANGES — a broken venv that gets repaired is the single
+ most useful transition on this event, and re-reporting an unchanged `ok`
+ on every boot would turn a 4-event lifetime into a heartbeat. The panel
+ has no heartbeats and docs/ANALYTICS.md says so.
+
+ No paths, no file names, no URLs, no byte counts, no host names. The only
+ thing carried besides the two closed words is `error_class`, which is the
+ existing closed taxonomy — deliberately NOT `error_signature` and NOT
+ `error_fingerprint`: an install-time error line is the likeliest place in
+ the whole product for a username-bearing path to survive scrubbing, and
+ the class alone answers the question this event exists to ask."""
+ try:
+ if step not in _INSTALL_STEPS or outcome not in _INSTALL_OUTCOMES:
+ return
+ seen = get_settings().get("analytics_install_steps")
+ seen = dict(seen) if isinstance(seen, dict) else {}
+ if seen.get(step) == outcome:
+ return
+ seen[step] = outcome
+ _settings_set_internal(analytics_install_steps=seen)
+ props = {"step": step, "outcome": outcome, "version": running_version()}
+ if error_class:
+ props["error_class"] = error_class
+ _analytics_capture("install_step", props)
+ except Exception: # noqa: BLE001
+ pass
+
+
+def _analytics_update_outcome(stage: str, outcome: str, from_version: str = "") -> None:
+ """Did pressing Update actually produce a new version?
+
+ 139 people pressed it in 14 days, 116 ran something new afterwards, and
+ 23 did not — and the panel could not tell "the pull failed" from "they
+ never restarted". The UI records the press; this is read on the NEXT boot,
+ where the version either moved or it didn't."""
+ try:
+ if stage not in _UPDATE_STAGES or outcome not in _UPDATE_OUTCOMES:
+ return
+ props = {"stage": stage, "outcome": outcome, "version": running_version()}
+ if from_version:
+ props["from_version"] = from_version
+ _analytics_capture("update_outcome", props)
+ except Exception: # noqa: BLE001
+ pass
+
+
+def _analytics_weights_ready() -> bool:
+ """Can this install render at all — are the base weights on disk?
+
+ Deliberately a directory probe and not a load: this runs at boot, and the
+ question is "did the download finish", which the presence of the pack's
+ own files answers."""
+ try:
+ d = Path(base_model_dir())
+ return d.is_dir() and any(d.glob("*.safetensors"))
+ except Exception: # noqa: BLE001
+ return False
+
+
def _analytics_job_secrets(job: dict) -> list:
"""Exact strings from this job that must never survive into an error
signature. Collected from the job itself so the redaction is exact
@@ -14390,7 +14712,23 @@ _ANALYTICS_ERROR_CLASSES = (
("refused", tuple(n for _, ns in _ANALYTICS_REFUSAL_REASONS for n in ns)),
("metal_watchdog", ("kiogpucommandbuffercallbackerrortimeout",
"caused gpu timeout error")),
- ("cancelled_race", ("cancel",)),
+ # THE OTHER Metal command-buffer failure, and it is not the watchdog.
+ # `kIOGPUCommandBufferCallbackErrorOutOfMemory` means the GPU ran out of
+ # memory; the timeout above means one buffer took too long. Both print
+ # "[METAL] Command buffer execution failed", so the fleet had them in one
+ # heap — and 17 events across 6 installs sat in `other` (the largest
+ # people-class we have) rather than saying "this machine needs a smaller
+ # canvas". Different cause, different remedy, different class.
+ ("metal_oom", ("kiogpucommandbuffercallbackerroroutofmemory",
+ "command buffer execution failed: insufficient memory")),
+ # WAS the bare substring `cancel`, which is a loaded gun: it sat third in
+ # this table, above oom_jetsam and native_crash, so ANY future message
+ # containing the word "cancelled" would have been filed as a cancel race.
+ # What the panel actually emits here is one shape — mflux's
+ # "cancelled by /stop (rc=-9)" — and the 12 events in 90 days are all it.
+ # The panel's own cancellations read "Stopped before/during ...".
+ ("cancelled_race", ("cancelled by /stop", "stopped before ",
+ "stopped during ", "stopped early")),
("oom_jetsam", ("sigkill", "helper died mid-job")),
("native_crash", ("sigsegv", "sigbus", "sigabrt")),
("helper_start_timeout", ("helper failed to start", "handshake")),
@@ -14449,7 +14787,23 @@ _ANALYTICS_ERROR_CLASSES = (
"resume install", "hq add-on", "missing (",
"runner is behind this panel")),
("venv_broken", ("venv", "dangling", "runner missing",
- "no module named")),
+ "no module named",
+ # The engine tree itself is absent. The fleet's shape is
+ # `[Errno 2] No such file or directory: 'ltx-2-mlx<path>'`
+ # (67 events, 2 installs) — the scrubber leaves the
+ # relative prefix, and that prefix is the whole diagnosis:
+ # this is a broken install, not a stray missing file, so
+ # it must win over `file_missing` below.
+ "ltx-2-mlx/env", "ltx-2-mlx<path>", "'ltx-2-mlx")),
+ # Python's own `[Errno 2] No such file or directory` — 4 people in 30 days
+ # in `other`, plus 67 events on two installs where the missing path was
+ # `ltx-2-mlx/...` itself (an engine tree that is not there, which
+ # `venv_broken` above already claims by name). Anything still reaching
+ # here is the panel opening a file that has gone: its own class, so it
+ # stops padding `other`, and it is BELOW input_missing so a missing
+ # reference image keeps its more specific class.
+ ("file_missing", ("[errno 2] no such file or directory",
+ "no usable temporary directory")),
("disk_full", ("enospc", "no space left")),
# BELOW model_missing / download_failed / model_corrupt / disk_full ON
# PURPOSE (review 2026-09-02): the subprocess-exit sentence wraps the
@@ -16409,6 +16763,15 @@ class WarmHelper:
with self.lock:
if self.proc is not None and self.proc.poll() is None:
return
+ # A broken environment is not a render fault, and it does not get
+ # better on the next attempt — the fleet's worst case retried 28
+ # times. Say it in one second with the fix in the sentence, rather
+ # than spawning a helper whose only job is to print an ImportError
+ # half a minute later.
+ fault = engine_env_fault()
+ if fault:
+ raise RuntimeError(f"engine venv is not usable: {fault}. "
+ f"{ENGINE_ENV_REPAIR}")
env = os.environ.copy()
env["PATH"] = f"{FFMPEG_BIN}:{env.get('PATH', '')}"
env["LTX_MODEL"] = base_model_dir()
@@ -26809,6 +27172,14 @@ def run_h3_job_inner(job: dict) -> None:
"Couldn't write the per-window prompt list to "
f"{chain_prompts_path}: {exc}") from exc
+ # ---- the venv has to be able to run the runner at all ----------------
+ # Checked here, before the LTX helper is killed and before 8 minutes of
+ # GPU time, because the alternative is what the fleet saw: a TypeError
+ # deep in the sampler that names a function and never names MLX.
+ _mlx_fault = h3_mlx_fault()
+ if _mlx_fault:
+ raise RuntimeError(f"H3 cannot start: {_mlx_fault}. {H3_MLX_REPAIR}")
+
# ---- memory safety: the warm helper cannot coexist with H3 -----------
if HELPER.is_alive():
push("[h3] stopping the LTX warm helper — H3 peaks around 40 GiB and "
@@ -30114,6 +30485,11 @@ def worker_loop() -> None:
STATE["log"] = []
caffeinate_on()
persist_queue()
+ # The funnel's last rung, recorded where every path converges: eight
+ # call sites append to the queue, exactly one worker takes from it.
+ # "They pressed Render at least once" is the difference between an
+ # install that was broken and an install that was never tried.
+ _analytics_install_step("first_queue", "started")
try:
# Hold the process-wide GPU gate for the whole job (v3.0.7 P1) so
diff --git a/panel/routes_meta.py b/panel/routes_meta.py
index 05a780e..6bda4b3 100644
--- a/panel/routes_meta.py
+++ b/panel/routes_meta.py
@@ -343,6 +343,14 @@ def post_analytics_ui(h, path, qs, ctype) -> None:
h._json({"ok": False, "error": "unknown action"}, 400); return
P._analytics_capture("update_prompt", {
"action": action, "version": P.running_version()})
+ if action in ("update_now", "banner_update"):
+ # Remember the build they pressed it ON. The next boot compares:
+ # a different version means the update landed, the same version
+ # means it didn't — which is the 23-of-139 hole that `app_updated`
+ # alone can never show, because a failed update emits nothing.
+ # One local string, overwritten each press, cleared on read.
+ P._settings_set_internal(
+ analytics_update_pressed=P.running_version())
elif event == "broadcast_seen":
P._analytics_capture("broadcast_seen", {"version": P.running_version()})
elif event == "feature_used":
diff --git a/scripts/test_analytics_dryrun.py b/scripts/test_analytics_dryrun.py
index 3efab1b..c619ab8 100644
--- a/scripts/test_analytics_dryrun.py
+++ b/scripts/test_analytics_dryrun.py
@@ -620,7 +620,15 @@ class TestEventSchemas(AnalyticsTestCase):
"ffmpeg exited 1": "export_failed",
"prompt required": "bad_params",
"phase timed out after 300s": "timeout",
- "cancel requested, landing as failed": "cancelled_race",
+ # A cancel race in the words the panel actually raises. This
+ # fixture used to read "cancel requested, landing as failed" and
+ # passed against a bare "cancel" needle — which also claimed every
+ # unrelated message with the word in it (see the negative below).
+ "H3 render stopped early at the next forward boundary":
+ "cancelled_race",
+ "cancelled by /stop (rc=-9)": "cancelled_race",
+ "stopped before the join": "cancelled_race",
+ "torch autograd cancellation hook raised": "other",
"some brand new exploding thing": "other",
# REFUSALS WIN OVER EVERYTHING. Each of these used to land as
# `other` — the first one was the single largest render_failed
@@ -958,7 +966,11 @@ class TestReceiverDirectives(AnalyticsTestCase):
def fire_every_event(self) -> dict[str, dict]:
"""One of each event the panel can emit → {event: properties}."""
packs = dict(P._analytics_pack_state())
- self.configure(analytics_last_packs=dict(packs, h3=not packs["h3"]))
+ # install_step is once-per-step-per-install BY DESIGN, and that memory
+ # lives in settings — which outlives a single test inside one process.
+ # A fresh install is the state this fixture is describing.
+ self.configure(analytics_last_packs=dict(packs, h3=not packs["h3"]),
+ analytics_install_steps={})
P._analytics_boot() # app_installed + app_boot + pack_state_change
P._analytics_render_event({
"status": "done", "elapsed_sec": 120.0,
@@ -980,6 +992,12 @@ class TestReceiverDirectives(AnalyticsTestCase):
P._analytics_capture("queue_paused_breaker", {"n_failed": 3, "queued": 2,
"error_class": "model_missing",
"version": "4.9.7"})
+ # v4.16.0 funnel events. install_step is once-per-step-per-install by
+ # design, and _analytics_boot() above already spent `first_boot`,
+ # `engine_env` and `weights_check` on this fixture install — so the
+ # step fired here is the one the worker owns.
+ P._analytics_install_step("first_queue", "started")
+ P._analytics_update_outcome("restart_pending", "failed", "4.15.0")
# A refusal: the panel declining on purpose. Fired here for the same
# reason as the rest — its payload has to carry the receiver
# directives too, and it is the newest way to get an event out.
@@ -1006,56 +1024,59 @@ class TestReceiverDirectives(AnalyticsTestCase):
{k: props.get(k) for k in P._ANALYTICS_RECEIVER_DIRECTIVES},
dict(P._ANALYTICS_RECEIVER_DIRECTIVES),
f"{event} reached the wire without the directives intact")
- self.assertIs(props["$geoip_disable"], True,
- f"{event} would be geolocated by the receiver")
self.assertIs(props["$process_person_profile"], False,
f"{event} would build a person profile")
- def test_no_location_property_is_sent_or_invited(self):
- """We add no location field, and $geoip_disable is the only $geoip_*
- key that may appear — everything else in that namespace is something
- the receiver would have derived."""
+ def test_the_panel_still_sends_no_location_field_of_its_own(self):
+ """The receiver derives country and city from the connecting address
+ (deliberate, 2026-09-19). The PANEL still names no place: it reads no
+ locale, no timezone, no coordinates off this Mac, and it puts nothing
+ in the $geoip_* namespace. That distinction is the whole Location
+ section of docs/ANALYTICS.md — a derived country is a fact about a
+ connection, a sent one would be a fact this program went looking for."""
for event, props in self.fire_every_event().items():
- geo = sorted(k for k in props
- if k.startswith("$geoip_") and k != "$geoip_disable")
+ geo = sorted(k for k in props if k.startswith("$geoip_"))
self.assertEqual(geo, [], f"{event} carried location data: {geo}")
raw = self.spy.raw().lower()
for word in ("country", "city", "latitude", "longitude", "timezone",
"time_zone", "subdivision", "locale", "continent"):
self.assertNotIn(word, raw, f"a payload mentions {word!r}")
- def test_the_ip_placeholder_is_truthy_and_not_a_spoof_trigger(self):
- """Both ways of writing this so that it does nothing.
-
- PostHog's ingest fills properties.$ip from the socket only when the
- event did not bring one — `if (!properties['$ip'] && event.ip)`. Any
- falsy value (None, "", 0) is therefore not suppression, it is the
- default with extra steps, and the real address lands on the event
- anyway. Separately, the GeoIP transformation rewrites 127.0.0.1 and
- 192.168.* to a real address in Sweden as a local-dev convenience, so
- a loopback placeholder would manufacture a location the day the
- disable flag got dropped. Hence: truthy, and neither of those."""
- ip = P.ANALYTICS_IP_PLACEHOLDER
- self.assertIsInstance(ip, str)
- self.assertTrue(ip, "a falsy $ip is silently replaced by the real one")
- self.assertNotEqual(ip, "127.0.0.1")
- self.assertFalse(ip.startswith("192.168."))
- P._analytics_capture("app_boot", {"version": "3.7.0"})
- drain()
- self.assertEqual(self.spy.bodies[0]["properties"]["$ip"], ip,
- "the placeholder never reached the wire")
-
- def test_a_call_site_cannot_override_a_directive(self):
- """The directives are spread last for this reason. A props dict is
- built from job params in places this module does not own."""
- P._analytics_capture("app_boot", {"$geoip_disable": False,
- "$ip": "203.0.113.7",
+ def test_no_ip_property_is_sent_so_the_receiver_can_resolve_one(self):
+ """The reversal that is easiest to make by accident.
+
+ From 2026-08-12 every event carried `$ip: "0.0.0.0"` to keep the
+ connecting address off the stored event. PostHog's GeoIP step reads
+ `properties.$ip`, so that placeholder did not just hide the address —
+ it emptied the country and city columns, which is what the fleet view
+ needs. Sending ANY `$ip` brings that back: 0.0.0.0 resolves to nothing,
+ 127.0.0.1 and 192.168.* are rewritten to a real address in Sweden as a
+ local-dev convenience and would invent a location outright. So: no $ip
+ at all, and the address is discarded after enrichment by the project's
+ own *Discard client IP data* setting."""
+ self.assertNotIn("$ip", P._ANALYTICS_RECEIVER_DIRECTIVES)
+ self.assertNotIn("$geoip_disable", P._ANALYTICS_RECEIVER_DIRECTIVES)
+ for event, props in self.fire_every_event().items():
+ self.assertNotIn("$ip", props,
+ f"{event} sent an $ip, so it will have no country")
+
+ def test_a_call_site_cannot_reach_the_receiver_namespace(self):
+ """The directives are spread last for this reason, and a props dict is
+ built from job params in places this module does not own. While the
+ directives occupied $ip and $geoip_disable, spreading last was enough;
+ now that they don't, _analytics_clean_props drops the whole $ namespace
+ instead — a stray "$ip" would spoof the address the GeoIP step reads
+ and a stray "$geoip_city_name" would invent a location."""
+ P._analytics_capture("app_boot", {"$ip": "203.0.113.7",
+ "$geoip_disable": True,
+ "$geoip_city_name": "Atlantis",
"$process_person_profile": True})
drain()
props = self.spy.bodies[0]["properties"]
- self.assertIs(props["$geoip_disable"], True)
self.assertIs(props["$process_person_profile"], False)
- self.assertEqual(props["$ip"], P.ANALYTICS_IP_PLACEHOLDER)
+ self.assertNotIn("$ip", props)
+ self.assertNotIn("$geoip_disable", props)
+ self.assertNotIn("$geoip_city_name", props)
def test_the_local_mirror_records_only_our_own_properties(self):
"""The directives are transport, not data: state/usage-log.jsonl is
diff --git a/test_caption_model_guard.py b/test_caption_model_guard.py
new file mode 100644
index 0000000..7cc93d4
--- /dev/null
+++ b/test_caption_model_guard.py
@@ -0,0 +1,61 @@
+"""The captioner must say WHICH Gemma folder is wrong.
+
+Fleet: "Model type gemma4_unified not supported." — 19 events, 4 installs,
+five releases. Two folders sit side by side in mlx_models and only one has a
+vision tower:
+
+ gemma-3-12b-it-4bit model_type "gemma3" the VLM
+ gemma4-12b-ltx25-q4 model_type "gemma4_unified" LTX-2.5's text encoder
+
+mlx_vlm's message names neither the folder that was passed nor the one that
+should have been.
+"""
+import json
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+HERE = Path(__file__).parent
+
+
+def run_captioner(model_type):
+ """Drive the real script far enough to reach the model-type gate."""
+ with TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ gemma = root / "gemma-x"
+ gemma.mkdir()
+ if model_type is not None:
+ (gemma / "config.json").write_text(json.dumps({"model_type": model_type}))
+ dataset = root / "ds"
+ (dataset / "images").mkdir(parents=True)
+ (dataset / "images" / "a.png").write_bytes(b"\x89PNG\r\n\x1a\n")
+ out = subprocess.run(
+ [sys.executable, str(HERE / "caption_with_gemma.py"),
+ "--dataset", str(dataset), "--trigger", "xtrn",
+ "--gemma-path", str(gemma)],
+ capture_output=True, text=True, timeout=120)
+ return out.stdout + out.stderr
+
+
+class TheWrongGemmaIsNamed(unittest.TestCase):
+
+ def test_the_ltx25_text_encoder_is_refused_by_name(self):
+ out = run_captioner("gemma4_unified")
+ self.assertIn("gemma4_unified", out)
+ self.assertIn("no vision tower", out)
+
+ def test_the_message_names_the_folder_that_would_work(self):
+ out = run_captioner("gemma4_unified")
+ self.assertIn("gemma-3-12b-it-4bit", out)
+
+ def test_a_config_we_cannot_read_does_not_block_the_load(self):
+ """Unknown is not wrong. If config.json is missing we let mlx_vlm
+ have its say rather than inventing a refusal."""
+ out = run_captioner(None)
+ self.assertNotIn("no vision tower", out)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/test_engine_env_fault.py b/test_engine_env_fault.py
new file mode 100644
index 0000000..d4b5ff9
--- /dev/null
+++ b/test_engine_env_fault.py
@@ -0,0 +1,113 @@
+"""A half-built engine venv must be caught before a render, not during one.
+
+Fleet, 2026-09-20: three installs on 4.13.0 / 4.14.3 / 4.15.0 died at helper
+start with `ModuleNotFoundError: No module named 'ltx_pipelines_mlx'` — one of
+them 28 renders in a row. Every file check the panel and pinokio.js had looked
+at the INTERPRETER, which in that shape exists; what is missing is the package
+pip writes into the venv last. So nothing offered the repair, and nothing said
+what was wrong until half a minute into each render.
+"""
+import unittest
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import mlx_ltx_panel as P
+
+
+class _Venv:
+ """Build a venv tree with the pieces you ask for."""
+
+ def __init__(self, python=True, package=True):
+ self.python, self.package = python, package
+
+ def __enter__(self):
+ self.tmp = TemporaryDirectory()
+ env = Path(self.tmp.name) / "ltx-2-mlx" / "env"
+ (env / "bin").mkdir(parents=True)
+ (env / "pyvenv.cfg").write_text("home = /usr/bin\n")
+ site = env / "lib" / "python3.11" / "site-packages"
+ site.mkdir(parents=True)
+ if self.python:
+ (env / "bin" / "python3.11").write_text("#!/bin/sh\n")
+ if self.package:
+ (site / "ltx_pipelines_mlx").mkdir()
+ self.prev = P.HELPER_PYTHON
+ P.HELPER_PYTHON = env / "bin" / "python3.11"
+ return self
+
+ def __exit__(self, *a):
+ P.HELPER_PYTHON = self.prev
+ self.tmp.cleanup()
+
+
+class TheProbe(unittest.TestCase):
+
+ def test_a_complete_venv_is_no_fault(self):
+ with _Venv():
+ self.assertEqual(P.engine_env_fault(), "")
+
+ def test_a_missing_interpreter_is_named(self):
+ with _Venv(python=False):
+ self.assertIn("no Python", P.engine_env_fault())
+
+ def test_the_half_built_venv_is_named(self):
+ """THE REGRESSION: interpreter present, packages never installed."""
+ with _Venv(package=False):
+ fault = P.engine_env_fault()
+ self.assertTrue(fault, "a venv with no packages read as healthy")
+ self.assertIn("ltx_pipelines_mlx", fault)
+
+ def test_an_interpreter_that_is_not_a_venv_is_not_judged(self):
+ """LTX_HELPER_PYTHON exists so a developer can point the panel at an
+ interpreter elsewhere. For /opt/homebrew/bin/python3.11 the venv
+ arithmetic lands on /opt/homebrew, whose site-packages legitimately
+ has no ltx_pipelines_mlx — faulting there would block renders on a
+ machine where everything works."""
+ with _Venv(package=False):
+ (P.HELPER_PYTHON.parent.parent / "pyvenv.cfg").unlink()
+ self.assertEqual(P.engine_env_fault(), "")
+
+ def test_the_repair_text_names_the_sidebar_entry(self):
+ """The user's next click. If pinokio.js ever renames that menu item,
+ this string has to move with it."""
+ self.assertIn("Repair Phosphene engine", P.ENGINE_ENV_REPAIR)
+
+ def test_pinokio_offers_the_repair_for_the_half_built_shape(self):
+ """The panel can only describe the fault; pinokio.js is what puts the
+ button on screen. Pin the probe it uses."""
+ js = Path(__file__).with_name("pinokio.js").read_text()
+ self.assertIn("site-packages/ltx_pipelines_mlx", js)
+ self.assertIn("!ltx_python || !ltx_pkg", js)
+
+
+class ItIsFiledUnderTheRightHeading(unittest.TestCase):
+ """Wording that a higher row of the taxonomy would claim sends this fault
+ to the wrong bucket in the fleet view — which is how it stays invisible.
+ Both of these were caught that way: "missing (" reads as model_missing and
+ "fetch"/"download" as download_failed."""
+
+ def messages(self):
+ with _Venv(python=False) as _:
+ no_python = P.engine_env_fault()
+ with _Venv(package=False) as _:
+ no_pkg = P.engine_env_fault()
+ return [f"engine venv is not usable: {f}. {P.ENGINE_ENV_REPAIR}"
+ for f in (no_python, no_pkg)]
+
+ def test_both_shapes_classify_as_venv_broken(self):
+ for msg in self.messages():
+ self.assertEqual(P._analytics_error_class(msg), "venv_broken", msg)
+
+ def test_no_message_says_download_or_fetch(self):
+ for msg in self.messages():
+ low = msg.lower()
+ self.assertNotIn("download", low)
+ self.assertNotIn("fetch", low)
+
+ def test_no_message_spells_the_model_missing_needle(self):
+ for msg in self.messages():
+ self.assertNotIn("missing (", msg.lower())
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/test_h3_mlx_guard.py b/test_h3_mlx_guard.py
new file mode 100644
index 0000000..db702e6
--- /dev/null
+++ b/test_h3_mlx_guard.py
@@ -0,0 +1,120 @@
+"""H3 refuses to start on an MLX its runner cannot use.
+
+Fleet, 4.14.3 and 4.15.0: `TypeError: seed(): incompatible function arguments`
+and `repeat(): incompatible function arguments`, both ~8 minutes into a render,
+neither naming MLX. They are signature drift. H3 needs mlx>=0.32 while the LTX
+engine pins mlx==0.31.1 (0.31.2 regresses LTX audio by 22 dB) — the separate
+venv exists for that reason, and a stray install into it puts the old MLX back.
+"""
+import unittest
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import mlx_ltx_panel as P
+
+
+class _H3Venv:
+ def __init__(self, mlx_version="0.32.2"):
+ self.v = mlx_version
+
+ def __enter__(self):
+ self.tmp = TemporaryDirectory()
+ venv = Path(self.tmp.name) / ".venv"
+ site = venv / "lib" / "python3.11" / "site-packages"
+ site.mkdir(parents=True)
+ (venv / "bin").mkdir(parents=True)
+ self.python = venv / "bin" / "python3.11"
+ self.python.write_text("#!/bin/sh\n")
+ if self.v:
+ (site / f"mlx-{self.v}.dist-info").mkdir()
+ self.orig = P._h3_python
+ P._h3_python = lambda: self.python
+ return self
+
+ def __exit__(self, *a):
+ P._h3_python = self.orig
+ self.tmp.cleanup()
+
+
+class ReadingTheVersion(unittest.TestCase):
+
+ def test_a_normal_version(self):
+ with _H3Venv("0.32.2") as v:
+ self.assertEqual(P._venv_dist_version(v.python, "mlx"), (0, 32, 2))
+
+ def test_a_release_candidate_suffix_does_not_break_it(self):
+ with _H3Venv("0.33.0rc1") as v:
+ self.assertEqual(P._venv_dist_version(v.python, "mlx"), (0, 33, 0))
+
+ def test_an_absent_package_reads_as_none(self):
+ with _H3Venv(None) as v:
+ self.assertIsNone(P._venv_dist_version(v.python, "mlx"))
+
+ def test_a_similarly_named_package_is_not_mistaken_for_mlx(self):
+ """`mlx_metal-0.32.2.dist-info` and `mlx_audio-0.5.1.dist-info` sit in
+ the same folder on a real install — 0.5.1 would read as far too old."""
+ with _H3Venv("0.32.2") as v:
+ site = v.python.parent.parent / "lib/python3.11/site-packages"
+ (site / "mlx_audio-0.5.1.dist-info").mkdir()
+ self.assertEqual(P._venv_dist_version(v.python, "mlx"), (0, 32, 2))
+
+
+class TheGuard(unittest.TestCase):
+
+ def test_a_good_venv_is_no_fault(self):
+ with _H3Venv("0.32.2"):
+ self.assertEqual(P.h3_mlx_fault(), "")
+
+ def test_the_old_mlx_is_caught_and_the_symptom_is_named(self):
+ with _H3Venv("0.31.1"):
+ fault = P.h3_mlx_fault()
+ self.assertIn("0.31.1", fault)
+ self.assertIn("seed()", fault)
+
+ def test_a_venv_with_no_mlx_is_caught(self):
+ with _H3Venv(None):
+ self.assertIn("no MLX", P.h3_mlx_fault())
+
+ def test_h3_not_installed_is_not_this_functions_problem(self):
+ orig = P._h3_python
+ P._h3_python = lambda: None
+ try:
+ self.assertEqual(P.h3_mlx_fault(), "")
+ finally:
+ P._h3_python = orig
+
+ def test_an_unreadable_layout_is_not_called_broken(self):
+ """Absence of evidence. LTX_H3_PYTHON can point somewhere this probe
+ does not understand; refusing there would block H3 on a machine where
+ it runs fine."""
+ from tempfile import TemporaryDirectory as _TD
+ with _TD() as tmp:
+ lonely = Path(tmp) / "bin" / "python3.11"
+ lonely.parent.mkdir(parents=True)
+ lonely.write_text("#!/bin/sh\n")
+ orig = P._h3_python
+ P._h3_python = lambda: lonely
+ try:
+ self.assertEqual(P.h3_mlx_fault(), "")
+ finally:
+ P._h3_python = orig
+
+ def test_the_render_path_checks_before_it_spends_anything(self):
+ """Ordering is the point: the check has to run before the LTX helper
+ is killed and before any GPU time is spent."""
+ src = Path(__file__).with_name("mlx_ltx_panel.py").read_text()
+ guard = src.index("_mlx_fault = h3_mlx_fault()")
+ helper_kill = src.index("stopping the LTX warm helper", guard - 4000)
+ self.assertLess(guard, helper_kill)
+
+ def test_the_message_is_filed_under_venv_broken(self):
+ with _H3Venv("0.31.1"):
+ msg = f"H3 cannot start: {P.h3_mlx_fault()}. {P.H3_MLX_REPAIR}"
+ self.assertEqual(P._analytics_error_class(msg), "venv_broken")
+
+ def test_the_repair_names_the_sidebar_entry(self):
+ self.assertIn("Repair Hailuo H3", P.H3_MLX_REPAIR)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/test_install_funnel.py b/test_install_funnel.py
new file mode 100644
index 0000000..39b6ff3
--- /dev/null
+++ b/test_install_funnel.py
@@ -0,0 +1,146 @@
+"""The install funnel: `install_step` and `update_outcome`.
+
+Why they exist (fleet, 14 days to 2026-09-19): 577 installs reported in and
+only 44.5% ever rendered — 278 booted once and vanished. Between
+`app_installed` and the first render the panel said nothing, so "they tried and
+it was broken" and "they looked and left" were the same shape in the data.
+And 23 of 139 people who pressed Update never ran a new version, which
+`app_updated` cannot show by construction: a failed update emits nothing.
+
+The invariants here are the ones that keep this from becoming a heartbeat or a
+free-text field.
+"""
+import unittest
+
+import mlx_ltx_panel as P
+
+
+class _Capture:
+ """Swap the capture function for a list."""
+
+ def __enter__(self):
+ self.seen = []
+ self.orig = P._analytics_capture
+ P._analytics_capture = lambda ev, props=None: self.seen.append((ev, dict(props or {})))
+ return self
+
+ def __exit__(self, *a):
+ P._analytics_capture = self.orig
+
+ def events(self, name):
+ return [p for e, p in self.seen if e == name]
+
+
+class _FreshInstall:
+ """An install that has taken no funnel steps yet."""
+
+ def __enter__(self):
+ self.orig = P.get_settings
+ self.store = {"analytics_install_steps": {}}
+ P.get_settings = lambda: dict(self.store)
+ self.orig_set = P._settings_set_internal
+ P._settings_set_internal = lambda **kv: self.store.update(kv)
+ return self
+
+ def __exit__(self, *a):
+ P.get_settings = self.orig
+ P._settings_set_internal = self.orig_set
+
+
+class OncePerInstall(unittest.TestCase):
+
+ def test_a_step_reports_once(self):
+ with _FreshInstall(), _Capture() as cap:
+ for _ in range(5):
+ P._analytics_install_step("first_queue", "started")
+ self.assertEqual(len(cap.events("install_step")), 1)
+
+ def test_the_same_step_reports_again_when_the_answer_changes(self):
+ """The deliberate exception, and the most useful row on this event:
+ a broken environment that gets repaired."""
+ with _FreshInstall(), _Capture() as cap:
+ P._analytics_install_step("engine_env", "failed", "venv_broken")
+ P._analytics_install_step("engine_env", "failed", "venv_broken")
+ P._analytics_install_step("engine_env", "ok")
+ got = [(e["outcome"], e.get("error_class")) for e in cap.events("install_step")]
+ self.assertEqual(got, [("failed", "venv_broken"), ("ok", None)])
+
+ def test_every_step_together_is_a_handful_of_events_for_all_time(self):
+ """Volume is the whole reason this is safe to ship. Four steps, and
+ the page promises no heartbeats."""
+ with _FreshInstall(), _Capture() as cap:
+ for step in P._INSTALL_STEPS:
+ for _ in range(10):
+ P._analytics_install_step(step, "ok")
+ self.assertEqual(len(cap.events("install_step")), len(P._INSTALL_STEPS))
+ self.assertLessEqual(len(P._INSTALL_STEPS), 6)
+
+
+class ClosedVocabulary(unittest.TestCase):
+
+ def test_an_unknown_step_never_reaches_the_wire(self):
+ with _FreshInstall(), _Capture() as cap:
+ P._analytics_install_step("rummaging_about", "ok")
+ self.assertEqual(cap.events("install_step"), [])
+
+ def test_an_unknown_outcome_never_reaches_the_wire(self):
+ with _FreshInstall(), _Capture() as cap:
+ P._analytics_install_step("first_boot", "sort of")
+ self.assertEqual(cap.events("install_step"), [])
+
+ def test_an_unknown_update_stage_never_reaches_the_wire(self):
+ with _Capture() as cap:
+ P._analytics_update_outcome("wandered_off", "ok")
+ self.assertEqual(cap.events("update_outcome"), [])
+
+ def test_no_signature_or_fingerprint_rides_along(self):
+ """Stated as a rule in docs/ANALYTICS.md: an install-time error line
+ is the likeliest place in the product for a username-bearing path to
+ survive scrubbing, and the class answers the question on its own."""
+ with _FreshInstall(), _Capture() as cap:
+ P._analytics_install_step("engine_env", "failed", "venv_broken")
+ props = cap.events("install_step")[0]
+ self.assertNotIn("error_signature", props)
+ self.assertNotIn("error_fingerprint", props)
+ self.assertEqual(sorted(props),
+ ["error_class", "outcome", "step", "version"])
+
+
+class DidTheUpdateLand(unittest.TestCase):
+
+ def test_a_new_version_after_the_press_is_ok(self):
+ with _Capture() as cap:
+ P._analytics_update_outcome("running_new", "ok", "4.15.0")
+ self.assertEqual(cap.events("update_outcome")[0]["outcome"], "ok")
+
+ def test_the_same_version_after_the_press_is_the_hole_we_are_measuring(self):
+ with _Capture() as cap:
+ P._analytics_update_outcome("restart_pending", "failed", "4.15.0")
+ ev = cap.events("update_outcome")[0]
+ self.assertEqual(ev["stage"], "restart_pending")
+ self.assertEqual(ev["from_version"], "4.15.0")
+
+ def test_the_press_is_recorded_by_the_ui_route(self):
+ """The marker is written where the click arrives; without it the next
+ boot has nothing to compare against."""
+ from pathlib import Path
+ src = Path(__file__).with_name("panel") / "routes_meta.py"
+ text = src.read_text()
+ self.assertIn("analytics_update_pressed", text)
+ self.assertIn('("update_now", "banner_update")', text)
+
+
+class BothEventsAreDocumented(unittest.TestCase):
+ """The page is the specification, and the dry-run suite enforces parity —
+ this is the same check from the other direction, so a rename here fails
+ close to the code that renamed it."""
+
+ def test_the_vocabularies_appear_on_the_page(self):
+ from pathlib import Path
+ doc = (Path(__file__).with_name("docs") / "ANALYTICS.md").read_text()
+ for word in P._INSTALL_STEPS + P._INSTALL_OUTCOMES + P._UPDATE_STAGES:
+ self.assertIn(f"`{word}`", doc, f"{word} is undocumented")
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/test_pasted_paths.py b/test_pasted_paths.py
index d5deaf3..e5bbdd3 100644
--- a/test_pasted_paths.py
+++ b/test_pasted_paths.py
@@ -234,3 +234,56 @@ def test_no_subprocess_pipe_decodes_strict_utf8():
if "errors=" not in window:
offenders.append(f"{name}:{i + 1}")
assert not offenders, f"strict-utf8 subprocess pipes: {offenders}"
+
+
+# ---- the error taxonomy, against strings the FLEET actually sent -------------
+#
+# Every needle below was read out of PostHog on 2026-09-18, not invented here.
+# The audit that prompted this found three faults: two live failure families
+# sitting in `other` (our largest people-class), and one needle that was a
+# loaded gun.
+
+def test_metal_out_of_memory_is_not_the_watchdog():
+ """17 events / 6 installs in 30 days. Both print the same first line, so
+ they were one heap; they are different faults with different remedies."""
+ oom = ("[METAL] Command buffer execution failed: Insufficient Memory "
+ "(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)")
+ timeout = ("[METAL] Command buffer execution failed: Caused GPU Timeout Error "
+ "(00000002:kIOGPUCommandBufferCallbackErrorTimeout)")
+ assert P._analytics_error_class(oom) == "metal_oom"
+ assert P._analytics_error_class(timeout) == "metal_watchdog"
+
+
+def test_a_missing_file_has_its_own_class():
+ assert P._analytics_error_class(
+ "[Errno 2] No such file or directory: '<path>'") == "file_missing"
+ assert P._analytics_error_class(
+ "[Errno 2] No usable temporary directory found in ['<path>']") == "file_missing"
+ # ...but the engine tree missing is still the install's fault, not a stray file,
+ # and a missing reference image keeps its more specific class.
+ assert P._analytics_error_class(
+ "[Errno 2] No such file or directory: 'ltx-2-mlx/env/bin/python'") == "venv_broken"
+ assert P._analytics_error_class(
+ "ref image not found: <path>") == "input_missing"
+
+
+def test_cancelled_race_matches_a_cancellation_not_the_word_cancel():
+ # What the fleet actually files under this class, all 12 events of it:
+ assert P._analytics_error_class(
+ "mflux-generate-qwen-edit cancelled by /stop (rc=-9)") == "cancelled_race"
+ # ...and the panel's own wording.
+ assert P._analytics_error_class("Stopped before the render started.") == "cancelled_race"
+ assert P._analytics_error_class("Stopped during joint denoise.") == "cancelled_race"
+ # The gun that was pointed at us: a bare "cancel" sat ABOVE oom_jetsam and
+ # native_crash, so any future message containing the word would have been
+ # filed as a cancel race rather than as the crash it is.
+ assert P._analytics_error_class(
+ "helper exited from SIGKILL (out of memory (jetsam) or external kill)") == "oom_jetsam"
+ assert P._analytics_error_class(
+ "the download was cancelled by the server mid-stream") != "cancelled_race"
+
+
+def test_the_documented_taxonomy_matches_the_code():
+ doc = (ROOT / "docs" / "ANALYTICS.md").read_text(encoding="utf-8")
+ for name, _ in P._ANALYTICS_ERROR_CLASSES:
+ assert "`%s`" % name in doc, name + " is classified but undocumented"
diff --git a/test_tool_resolution.py b/test_tool_resolution.py
new file mode 100644
index 0000000..b86ea98
--- /dev/null
+++ b/test_tool_resolution.py
@@ -0,0 +1,125 @@
+"""ffmpeg and ffprobe must be found where PINOKIO actually puts them.
+
+The fleet bug, 2026-09-20: five installs across 4.5.0 -> 4.15.0 failed their
+export with "ffmpeg not found on PATH". Every engine spawn already prepends
+FFMPEG_BIN to the child's PATH, so the miss was one level up — the resolver
+knew `pinokio/bin/ffmpeg-env` and nothing else, while Pinokio's bootstrap
+installs `ffmpeg=8.1.2` into its CONDA prefix (`pinokio/bin/miniforge/bin`)
+and unpacks a private Homebrew beside it. On a Mac with no system ffmpeg the
+resolver returned a /usr/local path that does not exist.
+"""
+import os
+import unittest
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import mlx_ltx_panel as P
+
+
+class _FakePinokio:
+ """A Pinokio home with a binary in one of its tool folders."""
+
+ def __init__(self, rel, name="ffmpeg"):
+ self.rel, self.name = rel, name
+
+ def __enter__(self):
+ self.tmp = TemporaryDirectory()
+ home = Path(self.tmp.name)
+ d = home / self.rel
+ d.mkdir(parents=True)
+ self.bin = d / self.name
+ self.bin.write_text("#!/bin/sh\n")
+ self.bin.chmod(0o755)
+ self.prev = os.environ.get("PINOKIO_HOME")
+ os.environ["PINOKIO_HOME"] = str(home)
+ self.which = P.shutil.which
+ P.shutil.which = lambda _n: None # nothing on PATH
+ return self
+
+ def __exit__(self, *a):
+ P.shutil.which = self.which
+ if self.prev is None:
+ os.environ.pop("PINOKIO_HOME", None)
+ else:
+ os.environ["PINOKIO_HOME"] = self.prev
+ self.tmp.cleanup()
+
+
+class WherePinokioPutsIt(unittest.TestCase):
+
+ def test_the_conda_prefix_is_searched(self):
+ """THE REGRESSION. `bin/miniforge/bin` is where Pinokio's own
+ bootstrap log shows ffmpeg being installed."""
+ with _FakePinokio("bin/miniforge/bin") as fake:
+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)
+
+ def test_the_older_miniconda_prefix_is_searched(self):
+ with _FakePinokio("bin/miniconda/bin") as fake:
+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)
+
+ def test_the_bundled_homebrew_is_searched(self):
+ """Pinokio unpacks its own Homebrew.zip into bin/homebrew."""
+ with _FakePinokio("bin/homebrew/bin") as fake:
+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)
+
+ def test_the_original_ffmpeg_env_still_wins_first(self):
+ """The one folder the old resolver knew must keep working."""
+ with _FakePinokio("bin/ffmpeg-env/bin") as fake:
+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)
+
+ def test_an_explicit_override_beats_every_search(self):
+ with _FakePinokio("bin/miniforge/bin") as fake:
+ other = Path(fake.tmp.name) / "my-own-ffmpeg"
+ other.write_text("#!/bin/sh\n")
+ os.environ["LTX_FFMPEG"] = str(other)
+ try:
+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), other)
+ finally:
+ os.environ.pop("LTX_FFMPEG", None)
+
+ def test_an_override_pointing_nowhere_does_not_win(self):
+ """A stale LTX_FFMPEG in a shell profile must not disable the search."""
+ with _FakePinokio("bin/miniforge/bin") as fake:
+ os.environ["LTX_FFMPEG"] = "/nope/ffmpeg"
+ try:
+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)
+ finally:
+ os.environ.pop("LTX_FFMPEG", None)
+
+ def test_nothing_found_still_returns_a_path(self):
+ """Resolution never raises at import time — a missing binary must
+ fail at the call site with a real message, not kill the panel."""
+ with _FakePinokio("bin/miniforge/bin"):
+ got = P._resolve_tool("definitely-not-a-real-tool", "LTX_NOPE")
+ self.assertEqual(got.name, "definitely-not-a-real-tool")
+ self.assertFalse(got.exists())
+
+ def test_the_folder_list_is_deduped(self):
+ """PINOKIO_HOME is normally ~/pinokio, so both passes find the same
+ folders — and this list is printed to the user in the boot warning."""
+ dirs = [str(d) for d in P._pinokio_bin_dirs()]
+ self.assertEqual(len(dirs), len(set(dirs)))
+
+
+class FfprobeTravelsSeparately(unittest.TestCase):
+
+ def test_ffprobe_is_taken_from_beside_ffmpeg_when_it_is_there(self):
+ with TemporaryDirectory() as tmp:
+ d = Path(tmp)
+ (d / "ffmpeg").write_text("")
+ (d / "ffprobe").write_text("")
+ self.assertEqual(P._resolve_ffprobe(d / "ffmpeg"), d / "ffprobe")
+
+ def test_ffprobe_is_searched_when_it_is_not_beside_ffmpeg(self):
+ """The fleet's bare "ffprobe failed:" (3 installs). The old code
+ assumed the sibling path existed and handed the caller a path that
+ never did."""
+ with _FakePinokio("bin/miniforge/bin", name="ffprobe") as fake:
+ with TemporaryDirectory() as lonely:
+ ffmpeg = Path(lonely) / "ffmpeg"
+ ffmpeg.write_text("")
+ self.assertEqual(P._resolve_ffprobe(ffmpeg), fake.bin)
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)
diff --git a/webapp/index.html b/webapp/index.html
index b566532..64d3ef4 100644
--- a/webapp/index.html
+++ b/webapp/index.html
@@ -5146,6 +5146,12 @@ The last line"></textarea>
plus the first line of the error when one fails.
<b>Never your prompts, filenames, paths, images or video.</b>
You are a random ID, generated on this Mac and tied to nothing.
+ The panel sends no location, but the address your computer
+ connects from resolves to a <b>country and city</b> at the
+ receiver, and those are kept next to the event; the address
+ itself is discarded. That is the one thing on this list the
+ panel doesn't send and can't stop — turning analytics off is
+ what stops it, because then no request is made at all.
Everything the panel would send is also written in plain text to
<code>state/usage-log.jsonl</code> so you can read it yourself;
the full schema is in <code>docs/ANALYTICS.md</code>.