Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
release(v4.15.1): the install fixes — ffmpeg found where Pinokio puts it, broken engines caught at boot
mrbizarrobd1878a5c08a
diff --git a/README.md b/README.mdindex fea591f..f2e8ace 100644--- a/README.md+++ b/README.md@@ -19,7 +19,7 @@</p>-> **Current release: v4.15.0.** **Hailuo H3 has a Speed switch.** **Fast** runs a 3-step adapter, **Best** runs the full schedule — with **Upscale & Face Fix** after it, a finished HD clip is about **5 minutes** on an M4 Max (Draft 2.8 min + 2.5 min, 1280×768) instead of half an hour. **Room tone** puts one generated ambience bed under a whole film so cuts stop clicking. **The Editor stops rearranging itself**: Sound is a mode you choose, panels open when you ask, the Source monitor is a drop target again. **The render form is four closed sections** — Shot setup, LoRAs, After the render, Advanced — each printing what it hides (`Draft · 3s · Fast · ~2 min`), and **Batch is a section**: paste many prompts, or ask for N takes of one, each with its own seed. Fixed: a film with music over it no longer comes out with silent shots, and a crashed engine names the line of code. Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).+> **Current release: v4.15.1.** **The install fixes.** ffmpeg is found where Pinokio actually puts it, so a render that finished no longer fails to export. A **half-built engine environment** is caught at boot with a one-click repair instead of failing 30 seconds into every render. **Hailuo H3 checks its MLX before it starts** rather than dying eight minutes in. Captioning tells you when it was pointed at LTX-2.5's text encoder instead of the vision model. Under the hood, the panel now reports **which step of getting started failed** — because 55% of new installs never render and nothing said why. See [what is collected](docs/ANALYTICS.md). Full notes on the [releases page](https://github.com/mrbizarro/Phosphene/releases).## Overviewdiff --git a/VERSION b/VERSIONindex 5c517bf..fb05571 100644--- a/VERSION+++ b/VERSION@@ -1 +1 @@-4.15.0+4.15.1diff --git a/caption_with_gemma.py b/caption_with_gemma.pyindex 394ee83..9a3c749 100644--- a/caption_with_gemma.py+++ b/caption_with_gemma.py@@ -170,6 +170,32 @@ def main() -> int:_emit("error", message=f"mlx_vlm not installed: {e}. "f"Run: pip install --no-deps 'mlx-vlm==0.4.4'")return 1+ # WHICH GEMMA IS THIS. Two folders live side by side in mlx_models and+ # only one of them can see an image:+ #+ # gemma-3-12b-it-4bit model_type "gemma3" <- the VLM+ # gemma4-12b-ltx25-q4 model_type "gemma4_unified" <- LTX-2.5's text+ # encoder, no+ # vision tower+ #+ # Handing the second one to mlx_vlm produces "Model type gemma4_unified+ # not supported" — 19 captioning failures across 4 installs and five+ # releases in the fleet, and nothing in that sentence says which folder+ # was wrong or which one is right. One read of config.json does.+ try:+ _cfg = json.loads((gemma_path / "config.json").read_text())+ _mtype = str(_cfg.get("model_type") or "")+ except Exception: # noqa: BLE001+ _mtype = ""+ if _mtype and not _mtype.startswith("gemma3"):+ _emit("error",+ message=f"{gemma_path.name} is not the captioning model — its "+ f"config says model_type \"{_mtype}\", which has no "+ f"vision tower (gemma4-12b-ltx25-q4 is LTX-2.5's text "+ f"encoder). Captioning needs gemma-3-12b-it-4bit. "+ f"Run /train/preflight from the panel to fetch it, or "+ f"point LTX_GEMMA_PATH at that folder.")+ return 1try:model, processor = load(str(gemma_path))except Exception as e:diff --git a/docs/ANALYTICS.md b/docs/ANALYTICS.mdindex 73b802b..9416939 100644--- a/docs/ANALYTICS.md+++ b/docs/ANALYTICS.md@@ -18,7 +18,7 @@ character names.| **What identifies you** | One random UUID, generated on your Mac, tied to nothing || **How often** | Once ever when the install is new, once per panel start, plus once per finished render. No heartbeats || **Where it goes** | PostHog (`us.i.posthog.com`), or a self-hosted endpoint you choose |-| **Location** | None sent, and every event tells the receiver not to derive one from your IP — [details](#location) |+| **Location** | None sent. Country and city are *derived* from the address your computer connects from, and kept — [details](#location) || **How to turn it off** | Settings → *Anonymous usage analytics* → **Turn off**, or `PHOSPHENE_ANALYTICS_DISABLED=1` || **Default** | ON, and the shipped build carries a working key — see below || **What you can inspect** | `state/usage-log.jsonl` — a plain-text copy of everything the panel sends |@@ -89,35 +89,62 @@ Settings.## LocationThe panel sends no location field of any kind — no country, no city, no region,-no timezone, no coordinates, no locale. It never has.+no timezone, no coordinates, no locale. It never has, and the guard suite fails+the build if one ever appears in a payload.-That on its own is not the whole promise, because a receiver can *derive* a-location from where a request came from, and PostHog does that by default. So-every event now also carries two instructions telling it not to:+**That is not the whole story, and this page states the rest plainly.** A+receiver can *derive* a location from the address a request arrives from, and+PostHog does that by default. **Since 2026-09-19 — the first build after+v4.15.0 — Phosphene lets it.** Every event is therefore stored with:-| Property | Value | What it does |-|---|---|---|-| `$geoip_disable` | `true` | PostHog's GeoIP step returns immediately instead of deriving country, city, subdivision, timezone and the city's coordinates |-| `$ip` | `"0.0.0.0"` | PostHog copies the connecting address into the event's `$ip` property only when the event didn't bring its own. Bringing one keeps the real address off the stored event |--**What that does not do, said plainly:** the request still arrives over TCP from-a real address, and nothing inside a request body can change that — that is how-HTTP works, for this panel and for every other program on your Mac. What the-two flags control is what the receiver is instructed to *derive* from that-address and *store* on the event. Discarding it at the edge as well is a-setting on the receiving project, not something this source tree can promise-you, so this page does not.--If you want the connection itself gone: turn analytics off, or point-`PHOSPHENE_ANALYTICS_HOST` at a receiver you run (below).--**A note on `0.0.0.0`, since it looks arbitrary.** The obvious spelling —-sending `$ip: null` — does nothing at all: PostHog's ingest fills the property-in when the event's value is *falsy*, so a null is silently replaced by your-real address. It has to be a non-empty string. `127.0.0.1` would be the natural-choice and is the wrong one: PostHog rewrites loopback and `192.168.*` to a real-address in Sweden as a local-development convenience, which would invent a-location the day the disable flag ever went missing.+| Derived by the receiver | Example |+|---|---|+| `$geoip_country_code`, `$geoip_country_name` | `DE`, `Germany` |+| `$geoip_city_name`, `$geoip_subdivision_1_name` | `Hamburg`, `Hamburg` |+| `$geoip_postal_code` | `20095` |+| `$geoip_latitude`, `$geoip_longitude` | the city's centre, not your building |+| `$geoip_time_zone` | `Europe/Berlin` |++**Why.** Phosphene is a Mac app that renders video locally, and the fleet view+exists to answer "which build is broken, and on what hardware". *Where* turns+out to belong to that question: a failure that is really a regional mirror, a+Hugging Face route, or a locale-dependent path shows up as noise without it.+The maintainer chose to keep it. This page's job is to tell you, not to talk+you into it.++**What is NOT stored: the address itself.** The receiving project has *Discard+client IP data* turned on, which runs the GeoIP step first and then drops the+address — so the country and city above are kept and the IP that produced them+is not. Being honest about the shape of that promise: it is a setting on the+receiving project, not a field inside a request body, so it is something this+source tree can *state* and cannot *prove to you*. What the source tree can+prove is the other half — that the panel itself sends nothing, and reads no+locale, timezone or coordinate off your Mac to send. That is what the tests+pin.++**A person is still not built.** `$process_person_profile: false` rides on every+event, so none of this accumulates into a person record. It is a property of an+event, next to a random install id.++> **Correction, 2026-09-18 — the history, which is not tidy.** An audit of the+> receiving project found **1,304 stored events carrying a derived location**+> across **85 installs**, from builds that shipped *before* the suppression flags+> existed (chiefly 2026-08-09 → 2026-08-15, plus a reverted 2025-05 build).+> Between **2026-08-12 and v4.15.0** the panel did suppress derivation, on every+> event, with two properties: `$geoip_disable: true`, and `$ip: "0.0.0.0"` to+> occupy the property PostHog would otherwise fill from the socket. Those flags+> are **gone as of 2026-09-19**, deliberately and with this page rewritten in the+> same commit — which is the only way a promise like this is worth anything. The+> older rows were left in place. Three postures in thirteen months is the real+> record, so it is written down instead of smoothed over.++**If you don't want any of it**, there are three off switches and they are the+same ones as for every other event on this page: the Settings toggle,+`PHOSPHENE_ANALYTICS_DISABLED=1`, or `PHOSPHENE_ANALYTICS_HOST` pointed at a+receiver you run. Turning analytics off means no request is made at all, which+is the only thing that makes a connecting address a non-question — nothing+inside a request body ever could, for this panel or for any other program on+your Mac.---@@ -205,7 +232,7 @@ Same as `render_completed` (minus `first_render`), plus:| Field | Type | Example | Notes ||---|---|---|---|-| `error_class` | string | `"metal_watchdog"` | **A closed 17-value taxonomy** (`refused`, `oom_jetsam`, `metal_watchdog`, `native_crash`, `helper_start_timeout`, `helper_exit`, `model_missing`, `model_corrupt`, `download_failed`, `venv_broken`, `bad_params`, `input_missing`, `disk_full`, `export_failed`, `timeout`, `cancelled_race`, `other`). Classification runs on the original error text locally; **only the class leaves the machine.** `refused` is in the taxonomy but never rides on a `render_failed` event — it is the value that routes the event to `render_refused` instead, below |+| `error_class` | string | `"metal_watchdog"` | **A closed 19-value taxonomy** (`refused`, `oom_jetsam`, `metal_watchdog`, `metal_oom`, `native_crash`, `helper_start_timeout`, `helper_exit`, `model_missing`, `model_corrupt`, `download_failed`, `venv_broken`, `bad_params`, `input_missing`, `file_missing`, `disk_full`, `export_failed`, `timeout`, `cancelled_race`, `other`). Classification runs on the original error text locally; **only the class leaves the machine.** `refused` is in the taxonomy but never rides on a `render_failed` event — it is the value that routes the event to `render_refused` instead, below || `error_fingerprint` | string | `"a3f09c21e7b4"` | Only when `error_class` is `other`: 12 hex chars of the SHA-256 of the already-scrubbed first line. Lets "the same unknown error, 17 times, all on M1 Max" be counted without transmitting the text — the readable line stays in your own `state/usage-log.jsonl` || `error_signature` | string | `"RuntimeError: helper exited before first frame"` | **The only free-text field the panel sends**, see the scrubbing rules below. Kept for ONE transition release alongside `error_class`, then removed |@@ -278,6 +305,52 @@ How the update pop-up and banner are answered — the only way to know whether t| `action` | string | `"later"` | **Closed vocabulary**: `shown`, `update_now`, `later` (the pop-up), `banner_update`, `banner_later` (the banner), `restart_needed` (the "Restart to finish update" pill, once per page load) || `version` | string | `"4.9.5"` | The version that was prompted |+### `install_step` (v4.16.0)++The dark half of the funnel. In the 14 days to 2026-09-19, **577 installs+reported in and only 44.5% of them ever rendered** — 278 booted once and were+never seen again. Between `app_installed` and the first `render_*` the panel+said nothing, so *"they tried and it was broken"* and *"they looked and left"*+were the same shape in the data. These events tell those apart.++**At most one event per step per install, ever** — with one deliberate+exception: `engine_env` reports again when its answer *changes*, because a+broken environment that gets repaired is the most useful transition on this+event. This is not a heartbeat and the panel still has none.++| prop | type | example | why |+|---|---|---|---|+| `step` | string | `"engine_env"` | **Closed vocabulary**: `first_boot` (the panel came up), `engine_env` (its Python environment can run a render), `weights_check` (the base model is on disk), `first_queue` (a job reached the worker — someone pressed Render) |+| `outcome` | string | `"failed"` | **Closed vocabulary**: `started`, `ok`, `failed`, `skipped` |+| `error_class` | string | `"venv_broken"` | Only on `failed`, and only from the same closed taxonomy `render_failed` uses |+| `version` | string | `"4.16.0"` | The build that reported it |++**What this event deliberately does not carry**, and why: no path, no file+name, no URL, no host, no byte count, and — unlike `render_failed` — **no+`error_signature` and no `error_fingerprint`**. An install-time error line is+the likeliest place in the whole product for a username-bearing path to+survive scrubbing, and the class on its own answers the question the event+exists to ask.++### `update_outcome` (v4.16.0)++Whether pressing **Update** actually produced a new version. 139 people+pressed it in 14 days; 116 ran something new afterwards and **23 did not**, and+nothing in the data could say whether the pull failed or the app never+restarted. `app_updated` cannot answer it by construction — a failed update+emits nothing at all.++The panel records the version you pressed Update *on* (one string, in local+settings) and reads it on the next boot: a different version means it landed,+the same version means it didn't. The marker is cleared when it is read.++| prop | type | example | why |+|---|---|---|---|+| `stage` | string | `"restart_pending"` | **Closed vocabulary**: `running_new` (a new version is running), `restart_pending` (same version came back) |+| `outcome` | string | `"failed"` | **Closed vocabulary**: `ok`, `failed` |+| `from_version` | string | `"4.15.0"` | The version Update was pressed on |+| `version` | string | `"4.15.0"` | The version now running |+### `broadcast_seen` (v4.9.7)One event when a developer broadcast (`BROADCAST.json`) is acknowledged. No text — which message is derivable from the date.@@ -577,8 +650,10 @@ refusal/fault fork (a refusal is `render_refused` and never a `render_failed`carrying `error_class: refused`; a real crash is untouched; refusals stay outof the local error rate), prompt/pathnon-leakage (including a prompt quoted inside an exception), forbidden-key-dropping, the geo-disable and `$ip` flags riding on *every* event type and-being un-overridable by a call site, bucketing, log rotation and the local+dropping, the receiver directives riding on *every* event type, no `$ip` and no+`$geoip_*` key leaving the panel (either would empty the country column or+invent one) and the whole `$` namespace being unreachable from a call site,+bucketing, log rotation and the localaggregates.To watch it live, tail the local log while you render:diff --git a/docs/STATE.md b/docs/STATE.mdindex 45f7b73..b5c5f10 100644--- a/docs/STATE.md+++ b/docs/STATE.md@@ -1,5 +1,30 @@# Phosphene — project state, history, open work+> **🔧 2026-09-20 — v4.15.1: the fleet's own bug list, fixed. Five faults, every one of them found in the data rather than reported.** The owner's instruction was "check fleet, github and pinokio and solve all bugs, ship fixes together with analytics improvements". GitHub and Pinokio turned out to be almost clean — every open bug thread there is already fixed and waiting on a reporter to confirm (the H3 Q8 SIGKILL in 4.12.2, the dead `madebyollin/taeh3` repo in 4.12.x, the YuE2 MPS flags in 4.14.3, the 48 GB bf16 pick in 4.12.4). **The fleet was not clean**, and nobody had filed any of this.+>+> - **ffmpeg was never where we looked.** Five installs across 4.5.0 → 4.15.0 finished a render and failed the export with "ffmpeg not found on PATH". Every engine spawn has prepended `FFMPEG_BIN` to the child's PATH for months — the miss was one level up: `_resolve_ffmpeg()` knew `pinokio/bin/ffmpeg-env` and nothing else, while **Pinokio's own bootstrap installs `ffmpeg=8.1.2` into its conda prefix** (`bin/miniforge/bin`) and unpacks a private Homebrew beside it. Proven on this Mac: that binary is sitting at `~/pinokio/bin/miniforge/bin/ffmpeg`, 443 KB, dated August. Now searched, along with `bin/miniconda/bin` and `bin/homebrew/bin`; **ffprobe resolves independently** instead of assuming the sibling path exists (the fleet's bare "ffprobe failed:", 3 installs); and a boot warning names every folder that was searched.+> - **A half-built venv looked perfectly healthy.** `ModuleNotFoundError: No module named 'ltx_pipelines_mlx'` on 4.13.0 / 4.14.3 / 4.15.0 — **one install hit it 28 renders in a row.** Both the panel and pinokio.js only ever checked the INTERPRETER, which in that shape exists; what is missing is the package pip writes last. So no repair was offered and nothing said what was wrong until half a minute into each render. `engine_env_fault()` probes the package directory, the panel refuses the spawn in one second with the fix in the sentence, and **pinokio.js now offers "Repair Phosphene engine" for this shape too**.+> - **H3 died eight minutes in on an MLX it cannot use.** `seed(): incompatible function arguments` and `repeat(): …` on 4.14.3 and 4.15.0 — MLX signature drift, neither message naming MLX. H3 needs mlx>=0.32; Phosphene pins mlx==0.31.1 (0.31.2 regresses LTX audio by 22 dB), which is why H3 has its own venv. `h3_mlx_fault()` reads the version off the `.dist-info` directory name — a listing, not a subprocess, because this is on the path to a render — and refuses **before** the LTX helper is killed and before any GPU time is spent.+> - **The captioner would not say which Gemma was wrong.** "Model type gemma4_unified not supported" — 19 events, 4 installs, five releases. Two folders sit side by side in `mlx_models` and only one has a vision tower: `gemma-3-12b-it-4bit` (`gemma3`) and `gemma4-12b-ltx25-q4` (`gemma4_unified`, LTX-2.5's TEXT encoder). mlx_vlm's message names neither the folder it was handed nor the one that works. One read of `config.json` does.+> - **The redactor was eating our own prose.** `~` matched as a home directory on its own, so the fleet is full of signatures reading "this engine needs `<path>`32 GB free RAM" — `~32 GB` is how a person writes an approximation, and that line is the most useful one in the report. A home path always continues with a separator; `~(?=/)` now says so.+>+> **And the analytics half, which is the part that changes what we can see next time.** Both events were proposed on 2026-09-19 and approved with "ship fixes together with analytics improvements".+>+> - **`install_step` — the dark half of the funnel.** 577 installs in 14 days, **44.5% of them ever rendered**, 278 booted once and vanished. Between `app_installed` and the first render the panel said nothing, so "they tried and it was broken" and "they looked and left" were the same shape in the data. Four steps — `first_boot`, `engine_env`, `weights_check`, `first_queue` — each **once per install for all time**, with one deliberate exception: `engine_env` reports again when its answer CHANGES, because a broken environment that gets repaired is the most useful row on the event. `first_queue` is emitted from the worker, the one place eight queue-append call sites converge. No path, no file name, no URL, no host, no byte count, and — unlike `render_failed` — **no signature and no fingerprint**: an install-time error line is the likeliest place in the product for a username-bearing path to survive scrubbing, and the closed class answers the question on its own.+> - **`update_outcome` — the 23 of 139.** People who pressed Update and never ran a new version. `app_updated` cannot show it by construction: a failed update emits nothing. The UI route records the version Update was pressed ON; the next boot compares and emits `running_new`/ok or `restart_pending`/failed, then clears the marker.+> - Both are documented on `docs/ANALYTICS.md` in the same commit, and the dry-run suite's parity guard is what forced that — it failed the build the moment the events existed without the page.+>+> **Tests: 45 new across four files**, and the two fault messages are pinned against the error taxonomy itself, because wording is load-bearing twice over here: "missing (" belongs to `model_missing`, "download"/"fetch" to `download_failed`, and either one files a venv fault under the wrong heading in the fleet view. Both were caught that way, against the live classifier, not guessed.++> **🌍 2026-09-19 — THE GEOIP KILL, REVERSED ON PURPOSE: country and city are now kept, and the page that promised otherwise changed in the same commit (UNRELEASED, dev).** The owner's ruling, verbatim: *"no need to discar i think countrya nd city is fine i wnat to know actualy."* The 2026-08-12 entry below is therefore superseded — not deleted, because three postures in thirteen months is the real record and this file keeps records.+>+> - **Both suppression properties are gone from every event.** `$geoip_disable: True` switched the receiver's GeoIP step off; `$ip: "0.0.0.0"` occupied the property so the connecting address was never copied onto the stored event — and **also killed the lookup**, because the step reads `properties.$ip` and resolves nothing from `0.0.0.0`. That second effect is the one that would have been easy to miss: dropping only the disable flag would have looked like a fix and changed nothing. `_ANALYTICS_RECEIVER_DIRECTIVES` is now one key, `$process_person_profile: False` — counts, still not people.+> - **The `$` namespace is closed to call sites** (`_analytics_clean_props`). While the directives occupied `$ip` and `$geoip_disable`, spreading them last in `_analytics_post` was sufficient armour. It no longer is: a props dict built from job params in code this module does not own could now put a **`$ip` on the wire and spoof the address the GeoIP step reads**, or a `$geoip_city_name` and invent a location outright. Every `$`-prefixed key from a call site is dropped; ours are spread afterwards.+> - **The owner's note had the project setting backwards, and the correction matters.** *Discard client IP data* does NOT suppress geo — PostHog runs the GeoIP transformation FIRST and discards the address after, so **on** = country and city stored, address not stored; **off** = the same country and city *plus* a retained IP on every event. It is still the owner's click (project settings are not ours to change) but it now costs him nothing he wants. `plugin-advanced-geoip` is the lever if he later wants postal code and city-centre coordinates dropped too — GeoIP stores seven properties, and he asked for two.+> - **`docs/ANALYTICS.md` Location section rewritten in the same commit**, which is the only thing that makes a promise like this worth reading: it names all seven stored `$geoip_*` properties with examples, says why they are kept, says the address is discarded at the edge **and that a source tree can state that and cannot prove it**, keeps the 1,304-event / 85-install history (the owner chose to keep those rows), and points at the three off switches. The summary table row changed too — it used to read *"None sent, and every event tells the receiver not to derive one"*.+> - **Guards rewritten, not relaxed, and mutation-tested 3/3 red:** put `$ip` back → red; put `$geoip_disable` back → red; reopen the `$` namespace → red. The surviving invariant is the one that was always the real one: **the panel sends no location field and reads no locale, timezone or coordinate off the Mac.** A derived country is a fact about a connection; a sent one would be a fact this program went looking for.+> - **Caught a stale fixture the pytest run cannot see.** `scripts/test_analytics_dryrun.py` runs standalone (56 tests), so last night's `cancelled_race` tightening in `9c66022` left `"cancel requested, landing as failed"` classifying as `other` with nobody watching. Replaced with the three strings the panel actually raises (`"H3 render stopped early at the next forward boundary"`, `"cancelled by /stop (rc=-9)"`, `"stopped before the join"`) plus a negative case pinning that a message merely containing the word is **not** a cancel. 56/56.+> **🧹 2026-09-18 — The render form is four closed sections (UNRELEASED, dev; waits for owner use).**> Owner: "The H3 layout is kind of a bit too much … you have a pill for speed, quality, length, and all that. That> should be inside one section that you can compress and decompress, like we have LoRAs and Customize, because it isdiff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.pyindex 1240620..941649a 100644--- a/mlx_ltx_panel.py+++ b/mlx_ltx_panel.py@@ -170,6 +170,59 @@ HELPER_SCRIPT = Path(os.environ.get("LTX_HELPER_SCRIPT", str(ROOT / "mlx_warm_he# at `env/bin/python3.11`. The warning + remediation line saves a# round-trip on every install variation that lands in a non-standard# layout.+def engine_env_fault() -> str:+ """"" when the engine environment can run a render, else the reason.++ TWO SHAPES, and only the first one was ever checked. The interpreter can+ be MISSING (a Reset that took the venv, a moved install) — that one has+ been caught at boot since issue #5. Or the interpreter can be PRESENT with+ the engine package never installed into it, which is what a pip step that+ failed halfway leaves behind. The second shape looks healthy to every file+ check we had, and it fails 30 seconds into a render instead: three installs+ in the fleet on 4.13.0 / 4.14.3 / 4.15.0, one of them 28 renders in a row,+ every one of them dying at helper start with++ ModuleNotFoundError: No module named 'ltx_pipelines_mlx'++ Checking the package DIRECTORY rather than importing keeps this cheap+ enough to run at boot and before every helper spawn: pip writes that+ directory last, so its absence is the same evidence an import would give.+ """+ if not HELPER_PYTHON.is_file():+ return (f"the engine venv has no Python at {HELPER_PYTHON} — the "+ f"ltx-2-mlx environment was never built, or something removed it")+ root = HELPER_PYTHON.parent.parent+ # ONLY JUDGE A VENV. `LTX_HELPER_PYTHON` exists so a developer can point+ # the panel at an interpreter somewhere else entirely, and for+ # /opt/homebrew/bin/python3.11 this arithmetic lands on /opt/homebrew —+ # whose site-packages legitimately has no ltx_pipelines_mlx in it. Faulting+ # there would block renders on a working machine. `pyvenv.cfg` is what+ # makes a directory a venv, so its absence means "not ours to judge".+ if not (root / "pyvenv.cfg").is_file():+ return ""+ try:+ pkgs = [d for d in (root / "lib").glob("python*/site-packages") if d.is_dir()]+ except OSError:+ pkgs = []+ if pkgs and not any((d / "ltx_pipelines_mlx").exists() for d in pkgs):+ return ("the engine venv has a Python but no packages — "+ "ltx_pipelines_mlx never finished installing into it")+ return ""+++# Wording is load-bearing twice over. It must name the exact sidebar entry a+# user has to click, and it must not contain a word that an EARLIER row of the+# error taxonomy claims: "missing (" belongs to model_missing and "download" to+# download_failed, and either one would file this fault under the wrong heading+# in the fleet view, and "fetch" belongs to download_failed (checked against+# the live classifier, not guessed). The phrase that has to survive is "venv",+# which is how venv_broken finds it.+ENGINE_ENV_REPAIR = (+ "Fix it from the Pinokio sidebar: \"Repair Phosphene engine (models "+ "kept)\" rebuilds the venv and keeps every model you have. Update works too. "+ "Renders cannot run until it is repaired."+)+if not HELPER_PYTHON.is_file():sys.stderr.write(f"WARN: helper Python not found at {HELPER_PYTHON}\n"@@ -177,24 +230,88 @@ if not HELPER_PYTHON.is_file():f" Renders will fail until either path exists, or LTX_HELPER_PYTHON\n"f" is exported pointing at a working python3.11 with mlx + ltx-pipelines.\n")-# ffmpeg: env var → PATH → Pinokio bundled → Homebrew → /usr/local. First match wins.+else:+ _env_fault = engine_env_fault()+ if _env_fault:+ sys.stderr.write(f"WARN: {_env_fault}.\n {ENGINE_ENV_REPAIR}\n")+def _pinokio_bin_dirs() -> list:+ """Pinokio's own tool folders, newest layout first.++ Pinokio's bootstrap installs `ffmpeg`, `git`, `node` and friends into a+ conda prefix under its home (`bin/miniforge/bin`, `bin/miniconda/bin` on+ older installs) and unpacks a private Homebrew beside it. None of those+ are guaranteed to be on PATH when it launches an app — which is the whole+ reason this helper exists."""+ home = (Path(os.environ["PINOKIO_HOME"]).expanduser()+ if os.environ.get("PINOKIO_HOME")+ else ROOT.parent.parent if ROOT.parent.name == "api" else None)+ dirs = []+ for base in (home, Path.home() / "pinokio"):+ if not base:+ continue+ dirs += [base / d for d in ("bin/ffmpeg-env/bin", "bin/miniforge/bin",+ "bin/miniconda/bin", "bin/homebrew/bin")]+ # PINOKIO_HOME is usually ~/pinokio, so both loops find the same folders.+ # Deduped, order preserved, because this list is also printed to the user.+ return list(dict.fromkeys(dirs))+++# ffmpeg: env var -> PATH -> Pinokio's own tool folders -> Homebrew -> /usr/local.+# First match wins.+#+# THE MISS THIS EXISTS TO CLOSE (fleet, 2026-09-20): five installs across+# 4.5.0 -> 4.15.0 failed their export with "ffmpeg not found on PATH", and the+# H3 runner spawn has prepended FFMPEG_BIN to the child's PATH the whole time.+# The bug was one level up: this resolver only knew `bin/ffmpeg-env`, while+# Pinokio's bootstrap installs `ffmpeg=8.1.2` into its CONDA prefix+# (`bin/miniforge/bin`) and unpacks a private Homebrew next to it. On a Mac+# with no system ffmpeg the resolver fell through to a `/usr/local` path that+# does not exist, and every engine that shells out to it died — a working+# binary sitting two directories away the whole time.+def _resolve_tool(name: str, env_var: str) -> Path:+ override = os.environ.get(env_var)+ if override and Path(override).exists():+ return Path(override)+ found = shutil.which(name)+ if found:+ return Path(found)+ for d in _pinokio_bin_dirs() + [Path("/opt/homebrew/bin"),+ Path("/usr/local/bin"),+ Path("/opt/local/bin")]:+ cand = d / name+ if cand.exists():+ return cand+ return Path("/usr/local/bin") / name # will fail at runtime if missing++def _resolve_ffmpeg() -> Path:- candidates = [- os.environ.get("LTX_FFMPEG"),- shutil.which("ffmpeg"),- str(Path.home() / "pinokio/bin/ffmpeg-env/bin/ffmpeg"),- "/opt/homebrew/bin/ffmpeg",- "/usr/local/bin/ffmpeg",- ]- for c in candidates:- if c and Path(c).exists():- return Path(c)- return Path("/usr/local/bin/ffmpeg") # last-resort default; will fail at runtime if missing+ return _resolve_tool("ffmpeg", "LTX_FFMPEG")+++def _resolve_ffprobe(ffmpeg: Path) -> Path:+ """ffprobe ships beside ffmpeg in every distribution we support, so that+ is the first guess — but only if it is actually there. A Mac that got its+ ffmpeg from one place and its ffprobe from another used to end up with a+ path that never existed, and the panel's duration probe failed with a bare+ "ffprobe failed:" (3 installs in the fleet)."""+ beside = ffmpeg.parent / "ffprobe"+ return beside if beside.exists() else _resolve_tool("ffprobe", "LTX_FFPROBE")FFMPEG = _resolve_ffmpeg()FFMPEG_BIN = FFMPEG.parent-FFPROBE = FFMPEG.parent / "ffprobe" # ships next to ffmpeg in every distribution we support+FFPROBE = _resolve_ffprobe(FFMPEG)+# Say it at boot, the way a missing helper venv is said above: an export that+# dies 20 minutes into a render is a terrible place to learn this.+if not FFMPEG.exists():+ _looked = ", ".join(str(d) for d in _pinokio_bin_dirs())+ sys.stderr.write(+ "WARN: ffmpeg not found. Looked on PATH and in: "+ + _looked + ", /opt/homebrew/bin, /usr/local/bin, /opt/local/bin.\n"+ " Renders will finish and then fail to export.\n"+ " Fix: install ffmpeg, or export LTX_FFMPEG=/path/to/ffmpeg.\n"+ )+Q4_LOCAL_PATH = MODELS_DIR / "ltx-2.3-mlx-q4"MODEL_ID = os.environ.get(@@ -10167,6 +10284,87 @@ def h3_prune_prompt_cache(keep: int = H3_PROMPT_CACHE_KEEP) -> int:return dropped+H3_MLX_MIN = (0, 32)+++def _venv_dist_version(python: Path, dist: str) -> tuple | None:+ """Version of an installed distribution, read off its `.dist-info`+ directory name. `None` when the package isn't there at all.++ A directory listing rather than a subprocess on purpose: this runs on the+ path to a render, and `python -c "import mlx"` costs a second and loads+ Metal. pip writes the dist-info name from the wheel's own metadata, so it+ is the same string `importlib.metadata.version()` would return."""+ try:+ for site in (python.parent.parent / "lib").glob("python*/site-packages"):+ for d in site.glob(f"{dist}-*.dist-info"):+ raw = d.name[len(dist) + 1:-len(".dist-info")]+ parts = []+ for chunk in raw.split(".")[:3]:+ # LEADING digits only. "0.33.0rc1" is 0.33.0, not 0.33.1 —+ # sweeping up every digit in the chunk turns a release+ # candidate into a higher version than the release.+ m = re.match(r"\d+", chunk)+ if not m:+ break+ parts.append(int(m.group(0)))+ if parts:+ return tuple(parts)+ except Exception: # noqa: BLE001+ return None+ return None+++def h3_mlx_fault() -> str:+ """"" when the H3 venv's MLX can run the H3 runner, else the reason.++ THE FAILURE THIS CATCHES, from the fleet on 4.14.3 and 4.15.0:++ TypeError: seed(): incompatible function arguments+ TypeError: repeat(): incompatible function arguments++ Both are MLX signature drift, and both arrive ~8 minutes into a render+ with nothing in them that names MLX as the problem. H3 needs mlx>=0.32+ while Phosphene's own engine pins mlx==0.31.1 (0.31.2 regresses LTX audio+ by 22 dB) — which is exactly why H3 has a separate venv. A venv that ends+ up holding the older MLX anyway — a stray `pip install` into it, a package+ that pulled MLX down as a dependency — produces those two lines and+ nothing else.++ Phrasing note, same as engine_env_fault: the word "venv" is what files+ this under venv_broken in the fleet view, and "missing (" / "download" /+ "fetch" would file it somewhere wrong."""+ py = _h3_python()+ if py is None:+ return "" # not installed is h3_paths()'s story, not this one+ got = _venv_dist_version(py, "mlx")+ if got is None:+ # UNKNOWN IS NOT BROKEN. `LTX_H3_PYTHON` can point at an interpreter+ # in a layout this probe does not understand, and a dist-info we+ # cannot find is not evidence of a package that is not there — it is+ # the absence of evidence. Refusing here would block H3 on a machine+ # where it works. Only say something when the venv is legible and the+ # package is genuinely absent from it.+ if not any((py.parent.parent / "lib").glob("python*/site-packages")):+ return ""+ return ("the Hailuo H3 venv has no MLX installed — its packages never "+ "finished landing")+ if got < H3_MLX_MIN:+ have = ".".join(str(n) for n in got)+ want = ".".join(str(n) for n in H3_MLX_MIN)+ return (f"the Hailuo H3 venv has MLX {have}, and the H3 runner needs "+ f"{want} or newer — on {have} it dies partway through a render "+ f"with \"seed(): incompatible function arguments\"")+ return ""+++H3_MLX_REPAIR = (+ "Fix it from the Pinokio sidebar: \"Repair Hailuo H3\" reinstalls that "+ "venv's packages and keeps every weight you have. LTX renders are "+ "unaffected — the two engines keep separate venvs for this exact reason."+)++def h3_paths() -> dict:"""Resolve every H3 component. Never raises — reports what's missing sothe UI can render an honest install card instead of a stack trace."""@@ -13633,15 +13831,27 @@ ANALYTICS_API_HOST_DEFAULT = "https://us.posthog.com"ANALYTICS_TIMEOUT_SEC = 2.0ANALYTICS_STR_MAX = 120-# The value we put in the event's own $ip property so PostHog never writes the-# connecting address there. It has to be a TRUTHY string: PostHog's ingest fills-# properties.$ip from the socket only when the event didn't carry one-# (`if (!properties['$ip'] && event.ip)`), so `None` — the obvious spelling — is-# falsy and gets silently replaced by the real address. 0.0.0.0 is also chosen-# over the more natural 127.0.0.1: the GeoIP transformation rewrites loopback-# and 192.168.* to a real Swedish address for local-dev convenience, which would-# manufacture a location if the disable flag below were ever dropped.-ANALYTICS_IP_PLACEHOLDER = "0.0.0.0"+# Location is DERIVED BY THE RECEIVER, on purpose, since 2026-09-19. The panel+# still sends no location field of its own and never has — it doesn't have to:+# PostHog resolves country and city from the connecting address, and the fleet+# view wants them (which build is failing, and where). Two properties used to+# suppress that, both shipped 2026-08-12 → 2026-09-19, and both are now gone+# deliberately:+#+# $geoip_disable: True switched the receiver's GeoIP step off outright.+# $ip: "0.0.0.0" occupied the property so the connecting address was+# never copied onto the stored event — which ALSO+# killed the lookup, because the step reads+# properties.$ip and resolves nothing from 0.0.0.0.+#+# Reintroducing either silently empties the country and city columns for every+# install, so neither may come back without changing docs/ANALYTICS.md in the+# same commit. What replaces them is a PROJECT-side setting on the receiver+# (PostHog → Project settings → *Discard client IP data*): PostHog runs the+# GeoIP transformation first and discards the address afterwards, so country+# and city are stored and the address is not. That is a setting on the project,+# not a fact this source tree can assert — same discipline as everywhere else+# in this module: state it, say who can check it, promise nothing.# Instructions to the receiver, attached to every event next to our own# properties. They are spread LAST in _analytics_post so no call site can@@ -13650,8 +13860,6 @@ ANALYTICS_IP_PLACEHOLDER = "0.0.0.0"# turning red. See _analytics_post for what each one does and does not buy._ANALYTICS_RECEIVER_DIRECTIVES = {"$process_person_profile": False,- "$geoip_disable": True,- "$ip": ANALYTICS_IP_PLACEHOLDER,}# Local mirror. Written for every captured event whether or not a PostHog@@ -13683,8 +13891,17 @@ _ANALYTICS_FORBIDDEN_KEYS = frozenset({# Absolute-path shapes to redact from error signatures. Applied in order:# the anchored pass catches the macOS roots a Phosphene error is likely to# quote, the generic pass catches anything else with 2+ path segments.+#+# `~` ONLY COUNTS AS A HOME DIRECTORY WHEN A SLASH FOLLOWS IT. It used to+# match on its own, and the casualty was our own prose: the fleet is full of+# signatures reading "this engine needs <path>32 GB free RAM" and "H3 needs+# about <path>64 GB", because "~32 GB" is how a person writes an approximate+# number. Both of those are panel copy, both are the most useful line in the+# report, and the redactor was quietly eating the tilde out of the middle of+# them. A home path always continues with a separator; an approximation never+# does._ANALYTICS_PATH_RES = (- re.compile(r"(?:~|/(?:Users|home|private|var|tmp|opt|Volumes|Applications"+ re.compile(r"(?:~(?=/)|/(?:Users|home|private|var|tmp|opt|Volumes|Applications"r"|Library|System))(?:/[^\s'\"<>,;)\]]*)*"),re.compile(r"(?:/[\w.+\-]+){2,}/?"),)@@ -13835,6 +14052,15 @@ def _analytics_clean_props(props: dict) -> dict:k = str(key)if k.lower() in _ANALYTICS_FORBIDDEN_KEYS:continue+ if k.startswith("$"):+ continue # the receiver's namespace belongs to+ # _ANALYTICS_RECEIVER_DIRECTIVES, not to a call site.+ # This mattered less while the directives occupied $ip+ # and $geoip_disable and could not be overridden; now+ # that they don't, a stray "$ip" from a params dict+ # would spoof the address the GeoIP step reads and a+ # stray "$geoip_city_name" would invent a location.+ # Ours are spread in _analytics_post, after this pass.if isinstance(val, bool) or isinstance(val, int) or isinstance(val, float):out[k] = valelif isinstance(val, str):@@ -13995,31 +14221,25 @@ def _usage_log_append(record: dict) -> None:def _analytics_post(payload: dict) -> None:"""Single-event POST to the PostHog capture API. Never raises.- Every event carries _ANALYTICS_RECEIVER_DIRECTIVES — three instructions- to the receiver, verified against PostHog's ingest source rather than+ Every event carries _ANALYTICS_RECEIVER_DIRECTIVES — one instruction to+ the receiver, verified against PostHog's ingest source rather thanassumed:$process_person_profile: FalseDon't build a person record for the install id. We want counts,not people.- $geoip_disable: True- PostHog's GeoIP transformation returns immediately instead of- deriving country, city, subdivision, timezone and the city's- coordinates. Its first statement is literally- `if (event.properties?.$geoip_disable or empty(...$ip))`, and this- is the same property posthog-python sets for disable_geoip=True.- $ip: ANALYTICS_IP_PLACEHOLDER- Ingest copies the connecting address into properties.$ip only when- the event didn't bring its own, so sending one is the only way to- keep the real address off the stored event. It must be truthy —- see ANALYTICS_IP_PLACEHOLDER for why `None` silently does nothing.-- What this does NOT do, and what docs/ANALYTICS.md is careful not to- claim: the request still arrives over TCP from a real address, and no- field inside a request body can change that. What the panel controls is- what the receiver is instructed to derive from it and store on the- event. Discarding it at the edge as well is a project-side setting, not- something this code can assert."""++ What is deliberately NOT here any more, as of 2026-09-19: `$geoip_disable`+ and a placeholder `$ip`. Both shipped from 2026-08-12 and both stopped the+ receiver deriving a location — the first by switching the GeoIP step off,+ the second because that step reads properties.$ip and resolves nothing from+ 0.0.0.0. The fleet view wants country and city, so the panel now lets the+ lookup happen and adds no location field of its own.++ What this does NOT do, and what docs/ANALYTICS.md is careful to say in the+ same words: the panel cannot promise what the receiver keeps. The address+ is dropped after enrichment by a project-side setting (*Discard client IP+ data*), which is a setting on the project, not a field in this body."""key = _analytics_key()if not key:return # only reachable if a fork blanks the shipped key; the guard@@ -14084,6 +14304,12 @@ _ANALYTICS_EVENTS = (# v4.9.7"feature_used", "app_updated", "update_prompt", "broadcast_seen","queue_paused_breaker",+ # v4.16.0 — the dark half of the funnel. 577 installs in 14 days and only+ # 44.5% of them ever rendered; 278 booted once and were never seen again.+ # Between `app_installed` and the first `render_*` the panel said nothing+ # at all, so "they tried and it was broken" and "they looked and left"+ # were the same shape in the data. These two close that.+ "install_step", "update_outcome",)@@ -14185,10 +14411,106 @@ def _analytics_boot() -> None:"pack": name, "from": was, "to": packs[name],})_settings_set_internal(analytics_last_packs=packs)+ # ---- the install funnel, evaluated where the answers live ---------+ # Ordered the way a new user meets them: the panel came up, its engine+ # environment is usable, its weights are on disk. A person who never+ # renders now leaves a trail that says WHICH of those three was false.+ _analytics_install_step("first_boot", "ok")+ _env_fault = engine_env_fault()+ _analytics_install_step(+ "engine_env", "failed" if _env_fault else "ok",+ _analytics_error_class(_env_fault) if _env_fault else "")+ _analytics_install_step(+ "weights_check", "ok" if _analytics_weights_ready() else "failed")+ # ---- did the last Update actually land ---------------------------+ _pressed = str(get_settings().get("analytics_update_pressed") or "")+ if _pressed:+ _settings_set_internal(analytics_update_pressed="")+ if _pressed != running_version():+ _analytics_update_outcome("running_new", "ok", _pressed)+ else:+ # Pressed Update, came back on the same build. Either the pull+ # failed or the app never actually restarted into a new one —+ # both are the hole this event exists to measure.+ _analytics_update_outcome("restart_pending", "failed", _pressed)except Exception:pass+# Closed vocabularies. A value outside these never reaches the network: the+# emitter drops it, and the dry-run suite fails the build if a call site+# invents one. This is the same discipline as the error taxonomy, for the same+# reason — a free-text step name is a field nobody can chart and a string+# nobody reviewed.+_INSTALL_STEPS = ("first_boot", "engine_env", "weights_check", "first_queue")+_INSTALL_OUTCOMES = ("started", "ok", "failed", "skipped")+_UPDATE_STAGES = ("running_new", "restart_pending")+_UPDATE_OUTCOMES = ("ok", "failed")+++def _analytics_install_step(step: str, outcome: str, error_class: str = "") -> None:+ """One step of getting from "installed" to "rendered".++ AT MOST ONCE PER STEP PER INSTALL, and for `engine_env` once more each+ time the answer CHANGES — a broken venv that gets repaired is the single+ most useful transition on this event, and re-reporting an unchanged `ok`+ on every boot would turn a 4-event lifetime into a heartbeat. The panel+ has no heartbeats and docs/ANALYTICS.md says so.++ No paths, no file names, no URLs, no byte counts, no host names. The only+ thing carried besides the two closed words is `error_class`, which is the+ existing closed taxonomy — deliberately NOT `error_signature` and NOT+ `error_fingerprint`: an install-time error line is the likeliest place in+ the whole product for a username-bearing path to survive scrubbing, and+ the class alone answers the question this event exists to ask."""+ try:+ if step not in _INSTALL_STEPS or outcome not in _INSTALL_OUTCOMES:+ return+ seen = get_settings().get("analytics_install_steps")+ seen = dict(seen) if isinstance(seen, dict) else {}+ if seen.get(step) == outcome:+ return+ seen[step] = outcome+ _settings_set_internal(analytics_install_steps=seen)+ props = {"step": step, "outcome": outcome, "version": running_version()}+ if error_class:+ props["error_class"] = error_class+ _analytics_capture("install_step", props)+ except Exception: # noqa: BLE001+ pass+++def _analytics_update_outcome(stage: str, outcome: str, from_version: str = "") -> None:+ """Did pressing Update actually produce a new version?++ 139 people pressed it in 14 days, 116 ran something new afterwards, and+ 23 did not — and the panel could not tell "the pull failed" from "they+ never restarted". The UI records the press; this is read on the NEXT boot,+ where the version either moved or it didn't."""+ try:+ if stage not in _UPDATE_STAGES or outcome not in _UPDATE_OUTCOMES:+ return+ props = {"stage": stage, "outcome": outcome, "version": running_version()}+ if from_version:+ props["from_version"] = from_version+ _analytics_capture("update_outcome", props)+ except Exception: # noqa: BLE001+ pass+++def _analytics_weights_ready() -> bool:+ """Can this install render at all — are the base weights on disk?++ Deliberately a directory probe and not a load: this runs at boot, and the+ question is "did the download finish", which the presence of the pack's+ own files answers."""+ try:+ d = Path(base_model_dir())+ return d.is_dir() and any(d.glob("*.safetensors"))+ except Exception: # noqa: BLE001+ return False++def _analytics_job_secrets(job: dict) -> list:"""Exact strings from this job that must never survive into an errorsignature. Collected from the job itself so the redaction is exact@@ -14390,7 +14712,23 @@ _ANALYTICS_ERROR_CLASSES = (("refused", tuple(n for _, ns in _ANALYTICS_REFUSAL_REASONS for n in ns)),("metal_watchdog", ("kiogpucommandbuffercallbackerrortimeout","caused gpu timeout error")),- ("cancelled_race", ("cancel",)),+ # THE OTHER Metal command-buffer failure, and it is not the watchdog.+ # `kIOGPUCommandBufferCallbackErrorOutOfMemory` means the GPU ran out of+ # memory; the timeout above means one buffer took too long. Both print+ # "[METAL] Command buffer execution failed", so the fleet had them in one+ # heap — and 17 events across 6 installs sat in `other` (the largest+ # people-class we have) rather than saying "this machine needs a smaller+ # canvas". Different cause, different remedy, different class.+ ("metal_oom", ("kiogpucommandbuffercallbackerroroutofmemory",+ "command buffer execution failed: insufficient memory")),+ # WAS the bare substring `cancel`, which is a loaded gun: it sat third in+ # this table, above oom_jetsam and native_crash, so ANY future message+ # containing the word "cancelled" would have been filed as a cancel race.+ # What the panel actually emits here is one shape — mflux's+ # "cancelled by /stop (rc=-9)" — and the 12 events in 90 days are all it.+ # The panel's own cancellations read "Stopped before/during ...".+ ("cancelled_race", ("cancelled by /stop", "stopped before ",+ "stopped during ", "stopped early")),("oom_jetsam", ("sigkill", "helper died mid-job")),("native_crash", ("sigsegv", "sigbus", "sigabrt")),("helper_start_timeout", ("helper failed to start", "handshake")),@@ -14449,7 +14787,23 @@ _ANALYTICS_ERROR_CLASSES = ("resume install", "hq add-on", "missing (","runner is behind this panel")),("venv_broken", ("venv", "dangling", "runner missing",- "no module named")),+ "no module named",+ # The engine tree itself is absent. The fleet's shape is+ # `[Errno 2] No such file or directory: 'ltx-2-mlx<path>'`+ # (67 events, 2 installs) — the scrubber leaves the+ # relative prefix, and that prefix is the whole diagnosis:+ # this is a broken install, not a stray missing file, so+ # it must win over `file_missing` below.+ "ltx-2-mlx/env", "ltx-2-mlx<path>", "'ltx-2-mlx")),+ # Python's own `[Errno 2] No such file or directory` — 4 people in 30 days+ # in `other`, plus 67 events on two installs where the missing path was+ # `ltx-2-mlx/...` itself (an engine tree that is not there, which+ # `venv_broken` above already claims by name). Anything still reaching+ # here is the panel opening a file that has gone: its own class, so it+ # stops padding `other`, and it is BELOW input_missing so a missing+ # reference image keeps its more specific class.+ ("file_missing", ("[errno 2] no such file or directory",+ "no usable temporary directory")),("disk_full", ("enospc", "no space left")),# BELOW model_missing / download_failed / model_corrupt / disk_full ON# PURPOSE (review 2026-09-02): the subprocess-exit sentence wraps the@@ -16409,6 +16763,15 @@ class WarmHelper:with self.lock:if self.proc is not None and self.proc.poll() is None:return+ # A broken environment is not a render fault, and it does not get+ # better on the next attempt — the fleet's worst case retried 28+ # times. Say it in one second with the fix in the sentence, rather+ # than spawning a helper whose only job is to print an ImportError+ # half a minute later.+ fault = engine_env_fault()+ if fault:+ raise RuntimeError(f"engine venv is not usable: {fault}. "+ f"{ENGINE_ENV_REPAIR}")env = os.environ.copy()env["PATH"] = f"{FFMPEG_BIN}:{env.get('PATH', '')}"env["LTX_MODEL"] = base_model_dir()@@ -26809,6 +27172,14 @@ def run_h3_job_inner(job: dict) -> None:"Couldn't write the per-window prompt list to "f"{chain_prompts_path}: {exc}") from exc+ # ---- the venv has to be able to run the runner at all ----------------+ # Checked here, before the LTX helper is killed and before 8 minutes of+ # GPU time, because the alternative is what the fleet saw: a TypeError+ # deep in the sampler that names a function and never names MLX.+ _mlx_fault = h3_mlx_fault()+ if _mlx_fault:+ raise RuntimeError(f"H3 cannot start: {_mlx_fault}. {H3_MLX_REPAIR}")+# ---- memory safety: the warm helper cannot coexist with H3 -----------if HELPER.is_alive():push("[h3] stopping the LTX warm helper — H3 peaks around 40 GiB and "@@ -30114,6 +30485,11 @@ def worker_loop() -> None:STATE["log"] = []caffeinate_on()persist_queue()+ # The funnel's last rung, recorded where every path converges: eight+ # call sites append to the queue, exactly one worker takes from it.+ # "They pressed Render at least once" is the difference between an+ # install that was broken and an install that was never tried.+ _analytics_install_step("first_queue", "started")try:# Hold the process-wide GPU gate for the whole job (v3.0.7 P1) sodiff --git a/panel/routes_meta.py b/panel/routes_meta.pyindex 05a780e..6bda4b3 100644--- a/panel/routes_meta.py+++ b/panel/routes_meta.py@@ -343,6 +343,14 @@ def post_analytics_ui(h, path, qs, ctype) -> None:h._json({"ok": False, "error": "unknown action"}, 400); returnP._analytics_capture("update_prompt", {"action": action, "version": P.running_version()})+ if action in ("update_now", "banner_update"):+ # Remember the build they pressed it ON. The next boot compares:+ # a different version means the update landed, the same version+ # means it didn't — which is the 23-of-139 hole that `app_updated`+ # alone can never show, because a failed update emits nothing.+ # One local string, overwritten each press, cleared on read.+ P._settings_set_internal(+ analytics_update_pressed=P.running_version())elif event == "broadcast_seen":P._analytics_capture("broadcast_seen", {"version": P.running_version()})elif event == "feature_used":diff --git a/scripts/test_analytics_dryrun.py b/scripts/test_analytics_dryrun.pyindex 3efab1b..c619ab8 100644--- a/scripts/test_analytics_dryrun.py+++ b/scripts/test_analytics_dryrun.py@@ -620,7 +620,15 @@ class TestEventSchemas(AnalyticsTestCase):"ffmpeg exited 1": "export_failed","prompt required": "bad_params","phase timed out after 300s": "timeout",- "cancel requested, landing as failed": "cancelled_race",+ # A cancel race in the words the panel actually raises. This+ # fixture used to read "cancel requested, landing as failed" and+ # passed against a bare "cancel" needle — which also claimed every+ # unrelated message with the word in it (see the negative below).+ "H3 render stopped early at the next forward boundary":+ "cancelled_race",+ "cancelled by /stop (rc=-9)": "cancelled_race",+ "stopped before the join": "cancelled_race",+ "torch autograd cancellation hook raised": "other","some brand new exploding thing": "other",# REFUSALS WIN OVER EVERYTHING. Each of these used to land as# `other` — the first one was the single largest render_failed@@ -958,7 +966,11 @@ class TestReceiverDirectives(AnalyticsTestCase):def fire_every_event(self) -> dict[str, dict]:"""One of each event the panel can emit → {event: properties}."""packs = dict(P._analytics_pack_state())- self.configure(analytics_last_packs=dict(packs, h3=not packs["h3"]))+ # install_step is once-per-step-per-install BY DESIGN, and that memory+ # lives in settings — which outlives a single test inside one process.+ # A fresh install is the state this fixture is describing.+ self.configure(analytics_last_packs=dict(packs, h3=not packs["h3"]),+ analytics_install_steps={})P._analytics_boot() # app_installed + app_boot + pack_state_changeP._analytics_render_event({"status": "done", "elapsed_sec": 120.0,@@ -980,6 +992,12 @@ class TestReceiverDirectives(AnalyticsTestCase):P._analytics_capture("queue_paused_breaker", {"n_failed": 3, "queued": 2,"error_class": "model_missing","version": "4.9.7"})+ # v4.16.0 funnel events. install_step is once-per-step-per-install by+ # design, and _analytics_boot() above already spent `first_boot`,+ # `engine_env` and `weights_check` on this fixture install — so the+ # step fired here is the one the worker owns.+ P._analytics_install_step("first_queue", "started")+ P._analytics_update_outcome("restart_pending", "failed", "4.15.0")# A refusal: the panel declining on purpose. Fired here for the same# reason as the rest — its payload has to carry the receiver# directives too, and it is the newest way to get an event out.@@ -1006,56 +1024,59 @@ class TestReceiverDirectives(AnalyticsTestCase):{k: props.get(k) for k in P._ANALYTICS_RECEIVER_DIRECTIVES},dict(P._ANALYTICS_RECEIVER_DIRECTIVES),f"{event} reached the wire without the directives intact")- self.assertIs(props["$geoip_disable"], True,- f"{event} would be geolocated by the receiver")self.assertIs(props["$process_person_profile"], False,f"{event} would build a person profile")- def test_no_location_property_is_sent_or_invited(self):- """We add no location field, and $geoip_disable is the only $geoip_*- key that may appear — everything else in that namespace is something- the receiver would have derived."""+ def test_the_panel_still_sends_no_location_field_of_its_own(self):+ """The receiver derives country and city from the connecting address+ (deliberate, 2026-09-19). The PANEL still names no place: it reads no+ locale, no timezone, no coordinates off this Mac, and it puts nothing+ in the $geoip_* namespace. That distinction is the whole Location+ section of docs/ANALYTICS.md — a derived country is a fact about a+ connection, a sent one would be a fact this program went looking for."""for event, props in self.fire_every_event().items():- geo = sorted(k for k in props- if k.startswith("$geoip_") and k != "$geoip_disable")+ geo = sorted(k for k in props if k.startswith("$geoip_"))self.assertEqual(geo, [], f"{event} carried location data: {geo}")raw = self.spy.raw().lower()for word in ("country", "city", "latitude", "longitude", "timezone","time_zone", "subdivision", "locale", "continent"):self.assertNotIn(word, raw, f"a payload mentions {word!r}")- def test_the_ip_placeholder_is_truthy_and_not_a_spoof_trigger(self):- """Both ways of writing this so that it does nothing.-- PostHog's ingest fills properties.$ip from the socket only when the- event did not bring one — `if (!properties['$ip'] && event.ip)`. Any- falsy value (None, "", 0) is therefore not suppression, it is the- default with extra steps, and the real address lands on the event- anyway. Separately, the GeoIP transformation rewrites 127.0.0.1 and- 192.168.* to a real address in Sweden as a local-dev convenience, so- a loopback placeholder would manufacture a location the day the- disable flag got dropped. Hence: truthy, and neither of those."""- ip = P.ANALYTICS_IP_PLACEHOLDER- self.assertIsInstance(ip, str)- self.assertTrue(ip, "a falsy $ip is silently replaced by the real one")- self.assertNotEqual(ip, "127.0.0.1")- self.assertFalse(ip.startswith("192.168."))- P._analytics_capture("app_boot", {"version": "3.7.0"})- drain()- self.assertEqual(self.spy.bodies[0]["properties"]["$ip"], ip,- "the placeholder never reached the wire")-- def test_a_call_site_cannot_override_a_directive(self):- """The directives are spread last for this reason. A props dict is- built from job params in places this module does not own."""- P._analytics_capture("app_boot", {"$geoip_disable": False,- "$ip": "203.0.113.7",+ def test_no_ip_property_is_sent_so_the_receiver_can_resolve_one(self):+ """The reversal that is easiest to make by accident.++ From 2026-08-12 every event carried `$ip: "0.0.0.0"` to keep the+ connecting address off the stored event. PostHog's GeoIP step reads+ `properties.$ip`, so that placeholder did not just hide the address —+ it emptied the country and city columns, which is what the fleet view+ needs. Sending ANY `$ip` brings that back: 0.0.0.0 resolves to nothing,+ 127.0.0.1 and 192.168.* are rewritten to a real address in Sweden as a+ local-dev convenience and would invent a location outright. So: no $ip+ at all, and the address is discarded after enrichment by the project's+ own *Discard client IP data* setting."""+ self.assertNotIn("$ip", P._ANALYTICS_RECEIVER_DIRECTIVES)+ self.assertNotIn("$geoip_disable", P._ANALYTICS_RECEIVER_DIRECTIVES)+ for event, props in self.fire_every_event().items():+ self.assertNotIn("$ip", props,+ f"{event} sent an $ip, so it will have no country")++ def test_a_call_site_cannot_reach_the_receiver_namespace(self):+ """The directives are spread last for this reason, and a props dict is+ built from job params in places this module does not own. While the+ directives occupied $ip and $geoip_disable, spreading last was enough;+ now that they don't, _analytics_clean_props drops the whole $ namespace+ instead — a stray "$ip" would spoof the address the GeoIP step reads+ and a stray "$geoip_city_name" would invent a location."""+ P._analytics_capture("app_boot", {"$ip": "203.0.113.7",+ "$geoip_disable": True,+ "$geoip_city_name": "Atlantis","$process_person_profile": True})drain()props = self.spy.bodies[0]["properties"]- self.assertIs(props["$geoip_disable"], True)self.assertIs(props["$process_person_profile"], False)- self.assertEqual(props["$ip"], P.ANALYTICS_IP_PLACEHOLDER)+ self.assertNotIn("$ip", props)+ self.assertNotIn("$geoip_disable", props)+ self.assertNotIn("$geoip_city_name", props)def test_the_local_mirror_records_only_our_own_properties(self):"""The directives are transport, not data: state/usage-log.jsonl isdiff --git a/test_caption_model_guard.py b/test_caption_model_guard.pynew file mode 100644index 0000000..7cc93d4--- /dev/null+++ b/test_caption_model_guard.py@@ -0,0 +1,61 @@+"""The captioner must say WHICH Gemma folder is wrong.++Fleet: "Model type gemma4_unified not supported." — 19 events, 4 installs,+five releases. Two folders sit side by side in mlx_models and only one has a+vision tower:++ gemma-3-12b-it-4bit model_type "gemma3" the VLM+ gemma4-12b-ltx25-q4 model_type "gemma4_unified" LTX-2.5's text encoder++mlx_vlm's message names neither the folder that was passed nor the one that+should have been.+"""+import json+import subprocess+import sys+import unittest+from pathlib import Path+from tempfile import TemporaryDirectory++HERE = Path(__file__).parent+++def run_captioner(model_type):+ """Drive the real script far enough to reach the model-type gate."""+ with TemporaryDirectory() as tmp:+ root = Path(tmp)+ gemma = root / "gemma-x"+ gemma.mkdir()+ if model_type is not None:+ (gemma / "config.json").write_text(json.dumps({"model_type": model_type}))+ dataset = root / "ds"+ (dataset / "images").mkdir(parents=True)+ (dataset / "images" / "a.png").write_bytes(b"\x89PNG\r\n\x1a\n")+ out = subprocess.run(+ [sys.executable, str(HERE / "caption_with_gemma.py"),+ "--dataset", str(dataset), "--trigger", "xtrn",+ "--gemma-path", str(gemma)],+ capture_output=True, text=True, timeout=120)+ return out.stdout + out.stderr+++class TheWrongGemmaIsNamed(unittest.TestCase):++ def test_the_ltx25_text_encoder_is_refused_by_name(self):+ out = run_captioner("gemma4_unified")+ self.assertIn("gemma4_unified", out)+ self.assertIn("no vision tower", out)++ def test_the_message_names_the_folder_that_would_work(self):+ out = run_captioner("gemma4_unified")+ self.assertIn("gemma-3-12b-it-4bit", out)++ def test_a_config_we_cannot_read_does_not_block_the_load(self):+ """Unknown is not wrong. If config.json is missing we let mlx_vlm+ have its say rather than inventing a refusal."""+ out = run_captioner(None)+ self.assertNotIn("no vision tower", out)+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/test_engine_env_fault.py b/test_engine_env_fault.pynew file mode 100644index 0000000..d4b5ff9--- /dev/null+++ b/test_engine_env_fault.py@@ -0,0 +1,113 @@+"""A half-built engine venv must be caught before a render, not during one.++Fleet, 2026-09-20: three installs on 4.13.0 / 4.14.3 / 4.15.0 died at helper+start with `ModuleNotFoundError: No module named 'ltx_pipelines_mlx'` — one of+them 28 renders in a row. Every file check the panel and pinokio.js had looked+at the INTERPRETER, which in that shape exists; what is missing is the package+pip writes into the venv last. So nothing offered the repair, and nothing said+what was wrong until half a minute into each render.+"""+import unittest+from pathlib import Path+from tempfile import TemporaryDirectory++import mlx_ltx_panel as P+++class _Venv:+ """Build a venv tree with the pieces you ask for."""++ def __init__(self, python=True, package=True):+ self.python, self.package = python, package++ def __enter__(self):+ self.tmp = TemporaryDirectory()+ env = Path(self.tmp.name) / "ltx-2-mlx" / "env"+ (env / "bin").mkdir(parents=True)+ (env / "pyvenv.cfg").write_text("home = /usr/bin\n")+ site = env / "lib" / "python3.11" / "site-packages"+ site.mkdir(parents=True)+ if self.python:+ (env / "bin" / "python3.11").write_text("#!/bin/sh\n")+ if self.package:+ (site / "ltx_pipelines_mlx").mkdir()+ self.prev = P.HELPER_PYTHON+ P.HELPER_PYTHON = env / "bin" / "python3.11"+ return self++ def __exit__(self, *a):+ P.HELPER_PYTHON = self.prev+ self.tmp.cleanup()+++class TheProbe(unittest.TestCase):++ def test_a_complete_venv_is_no_fault(self):+ with _Venv():+ self.assertEqual(P.engine_env_fault(), "")++ def test_a_missing_interpreter_is_named(self):+ with _Venv(python=False):+ self.assertIn("no Python", P.engine_env_fault())++ def test_the_half_built_venv_is_named(self):+ """THE REGRESSION: interpreter present, packages never installed."""+ with _Venv(package=False):+ fault = P.engine_env_fault()+ self.assertTrue(fault, "a venv with no packages read as healthy")+ self.assertIn("ltx_pipelines_mlx", fault)++ def test_an_interpreter_that_is_not_a_venv_is_not_judged(self):+ """LTX_HELPER_PYTHON exists so a developer can point the panel at an+ interpreter elsewhere. For /opt/homebrew/bin/python3.11 the venv+ arithmetic lands on /opt/homebrew, whose site-packages legitimately+ has no ltx_pipelines_mlx — faulting there would block renders on a+ machine where everything works."""+ with _Venv(package=False):+ (P.HELPER_PYTHON.parent.parent / "pyvenv.cfg").unlink()+ self.assertEqual(P.engine_env_fault(), "")++ def test_the_repair_text_names_the_sidebar_entry(self):+ """The user's next click. If pinokio.js ever renames that menu item,+ this string has to move with it."""+ self.assertIn("Repair Phosphene engine", P.ENGINE_ENV_REPAIR)++ def test_pinokio_offers_the_repair_for_the_half_built_shape(self):+ """The panel can only describe the fault; pinokio.js is what puts the+ button on screen. Pin the probe it uses."""+ js = Path(__file__).with_name("pinokio.js").read_text()+ self.assertIn("site-packages/ltx_pipelines_mlx", js)+ self.assertIn("!ltx_python || !ltx_pkg", js)+++class ItIsFiledUnderTheRightHeading(unittest.TestCase):+ """Wording that a higher row of the taxonomy would claim sends this fault+ to the wrong bucket in the fleet view — which is how it stays invisible.+ Both of these were caught that way: "missing (" reads as model_missing and+ "fetch"/"download" as download_failed."""++ def messages(self):+ with _Venv(python=False) as _:+ no_python = P.engine_env_fault()+ with _Venv(package=False) as _:+ no_pkg = P.engine_env_fault()+ return [f"engine venv is not usable: {f}. {P.ENGINE_ENV_REPAIR}"+ for f in (no_python, no_pkg)]++ def test_both_shapes_classify_as_venv_broken(self):+ for msg in self.messages():+ self.assertEqual(P._analytics_error_class(msg), "venv_broken", msg)++ def test_no_message_says_download_or_fetch(self):+ for msg in self.messages():+ low = msg.lower()+ self.assertNotIn("download", low)+ self.assertNotIn("fetch", low)++ def test_no_message_spells_the_model_missing_needle(self):+ for msg in self.messages():+ self.assertNotIn("missing (", msg.lower())+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/test_h3_mlx_guard.py b/test_h3_mlx_guard.pynew file mode 100644index 0000000..db702e6--- /dev/null+++ b/test_h3_mlx_guard.py@@ -0,0 +1,120 @@+"""H3 refuses to start on an MLX its runner cannot use.++Fleet, 4.14.3 and 4.15.0: `TypeError: seed(): incompatible function arguments`+and `repeat(): incompatible function arguments`, both ~8 minutes into a render,+neither naming MLX. They are signature drift. H3 needs mlx>=0.32 while the LTX+engine pins mlx==0.31.1 (0.31.2 regresses LTX audio by 22 dB) — the separate+venv exists for that reason, and a stray install into it puts the old MLX back.+"""+import unittest+from pathlib import Path+from tempfile import TemporaryDirectory++import mlx_ltx_panel as P+++class _H3Venv:+ def __init__(self, mlx_version="0.32.2"):+ self.v = mlx_version++ def __enter__(self):+ self.tmp = TemporaryDirectory()+ venv = Path(self.tmp.name) / ".venv"+ site = venv / "lib" / "python3.11" / "site-packages"+ site.mkdir(parents=True)+ (venv / "bin").mkdir(parents=True)+ self.python = venv / "bin" / "python3.11"+ self.python.write_text("#!/bin/sh\n")+ if self.v:+ (site / f"mlx-{self.v}.dist-info").mkdir()+ self.orig = P._h3_python+ P._h3_python = lambda: self.python+ return self++ def __exit__(self, *a):+ P._h3_python = self.orig+ self.tmp.cleanup()+++class ReadingTheVersion(unittest.TestCase):++ def test_a_normal_version(self):+ with _H3Venv("0.32.2") as v:+ self.assertEqual(P._venv_dist_version(v.python, "mlx"), (0, 32, 2))++ def test_a_release_candidate_suffix_does_not_break_it(self):+ with _H3Venv("0.33.0rc1") as v:+ self.assertEqual(P._venv_dist_version(v.python, "mlx"), (0, 33, 0))++ def test_an_absent_package_reads_as_none(self):+ with _H3Venv(None) as v:+ self.assertIsNone(P._venv_dist_version(v.python, "mlx"))++ def test_a_similarly_named_package_is_not_mistaken_for_mlx(self):+ """`mlx_metal-0.32.2.dist-info` and `mlx_audio-0.5.1.dist-info` sit in+ the same folder on a real install — 0.5.1 would read as far too old."""+ with _H3Venv("0.32.2") as v:+ site = v.python.parent.parent / "lib/python3.11/site-packages"+ (site / "mlx_audio-0.5.1.dist-info").mkdir()+ self.assertEqual(P._venv_dist_version(v.python, "mlx"), (0, 32, 2))+++class TheGuard(unittest.TestCase):++ def test_a_good_venv_is_no_fault(self):+ with _H3Venv("0.32.2"):+ self.assertEqual(P.h3_mlx_fault(), "")++ def test_the_old_mlx_is_caught_and_the_symptom_is_named(self):+ with _H3Venv("0.31.1"):+ fault = P.h3_mlx_fault()+ self.assertIn("0.31.1", fault)+ self.assertIn("seed()", fault)++ def test_a_venv_with_no_mlx_is_caught(self):+ with _H3Venv(None):+ self.assertIn("no MLX", P.h3_mlx_fault())++ def test_h3_not_installed_is_not_this_functions_problem(self):+ orig = P._h3_python+ P._h3_python = lambda: None+ try:+ self.assertEqual(P.h3_mlx_fault(), "")+ finally:+ P._h3_python = orig++ def test_an_unreadable_layout_is_not_called_broken(self):+ """Absence of evidence. LTX_H3_PYTHON can point somewhere this probe+ does not understand; refusing there would block H3 on a machine where+ it runs fine."""+ from tempfile import TemporaryDirectory as _TD+ with _TD() as tmp:+ lonely = Path(tmp) / "bin" / "python3.11"+ lonely.parent.mkdir(parents=True)+ lonely.write_text("#!/bin/sh\n")+ orig = P._h3_python+ P._h3_python = lambda: lonely+ try:+ self.assertEqual(P.h3_mlx_fault(), "")+ finally:+ P._h3_python = orig++ def test_the_render_path_checks_before_it_spends_anything(self):+ """Ordering is the point: the check has to run before the LTX helper+ is killed and before any GPU time is spent."""+ src = Path(__file__).with_name("mlx_ltx_panel.py").read_text()+ guard = src.index("_mlx_fault = h3_mlx_fault()")+ helper_kill = src.index("stopping the LTX warm helper", guard - 4000)+ self.assertLess(guard, helper_kill)++ def test_the_message_is_filed_under_venv_broken(self):+ with _H3Venv("0.31.1"):+ msg = f"H3 cannot start: {P.h3_mlx_fault()}. {P.H3_MLX_REPAIR}"+ self.assertEqual(P._analytics_error_class(msg), "venv_broken")++ def test_the_repair_names_the_sidebar_entry(self):+ self.assertIn("Repair Hailuo H3", P.H3_MLX_REPAIR)+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/test_install_funnel.py b/test_install_funnel.pynew file mode 100644index 0000000..39b6ff3--- /dev/null+++ b/test_install_funnel.py@@ -0,0 +1,146 @@+"""The install funnel: `install_step` and `update_outcome`.++Why they exist (fleet, 14 days to 2026-09-19): 577 installs reported in and+only 44.5% ever rendered — 278 booted once and vanished. Between+`app_installed` and the first render the panel said nothing, so "they tried and+it was broken" and "they looked and left" were the same shape in the data.+And 23 of 139 people who pressed Update never ran a new version, which+`app_updated` cannot show by construction: a failed update emits nothing.++The invariants here are the ones that keep this from becoming a heartbeat or a+free-text field.+"""+import unittest++import mlx_ltx_panel as P+++class _Capture:+ """Swap the capture function for a list."""++ def __enter__(self):+ self.seen = []+ self.orig = P._analytics_capture+ P._analytics_capture = lambda ev, props=None: self.seen.append((ev, dict(props or {})))+ return self++ def __exit__(self, *a):+ P._analytics_capture = self.orig++ def events(self, name):+ return [p for e, p in self.seen if e == name]+++class _FreshInstall:+ """An install that has taken no funnel steps yet."""++ def __enter__(self):+ self.orig = P.get_settings+ self.store = {"analytics_install_steps": {}}+ P.get_settings = lambda: dict(self.store)+ self.orig_set = P._settings_set_internal+ P._settings_set_internal = lambda **kv: self.store.update(kv)+ return self++ def __exit__(self, *a):+ P.get_settings = self.orig+ P._settings_set_internal = self.orig_set+++class OncePerInstall(unittest.TestCase):++ def test_a_step_reports_once(self):+ with _FreshInstall(), _Capture() as cap:+ for _ in range(5):+ P._analytics_install_step("first_queue", "started")+ self.assertEqual(len(cap.events("install_step")), 1)++ def test_the_same_step_reports_again_when_the_answer_changes(self):+ """The deliberate exception, and the most useful row on this event:+ a broken environment that gets repaired."""+ with _FreshInstall(), _Capture() as cap:+ P._analytics_install_step("engine_env", "failed", "venv_broken")+ P._analytics_install_step("engine_env", "failed", "venv_broken")+ P._analytics_install_step("engine_env", "ok")+ got = [(e["outcome"], e.get("error_class")) for e in cap.events("install_step")]+ self.assertEqual(got, [("failed", "venv_broken"), ("ok", None)])++ def test_every_step_together_is_a_handful_of_events_for_all_time(self):+ """Volume is the whole reason this is safe to ship. Four steps, and+ the page promises no heartbeats."""+ with _FreshInstall(), _Capture() as cap:+ for step in P._INSTALL_STEPS:+ for _ in range(10):+ P._analytics_install_step(step, "ok")+ self.assertEqual(len(cap.events("install_step")), len(P._INSTALL_STEPS))+ self.assertLessEqual(len(P._INSTALL_STEPS), 6)+++class ClosedVocabulary(unittest.TestCase):++ def test_an_unknown_step_never_reaches_the_wire(self):+ with _FreshInstall(), _Capture() as cap:+ P._analytics_install_step("rummaging_about", "ok")+ self.assertEqual(cap.events("install_step"), [])++ def test_an_unknown_outcome_never_reaches_the_wire(self):+ with _FreshInstall(), _Capture() as cap:+ P._analytics_install_step("first_boot", "sort of")+ self.assertEqual(cap.events("install_step"), [])++ def test_an_unknown_update_stage_never_reaches_the_wire(self):+ with _Capture() as cap:+ P._analytics_update_outcome("wandered_off", "ok")+ self.assertEqual(cap.events("update_outcome"), [])++ def test_no_signature_or_fingerprint_rides_along(self):+ """Stated as a rule in docs/ANALYTICS.md: an install-time error line+ is the likeliest place in the product for a username-bearing path to+ survive scrubbing, and the class answers the question on its own."""+ with _FreshInstall(), _Capture() as cap:+ P._analytics_install_step("engine_env", "failed", "venv_broken")+ props = cap.events("install_step")[0]+ self.assertNotIn("error_signature", props)+ self.assertNotIn("error_fingerprint", props)+ self.assertEqual(sorted(props),+ ["error_class", "outcome", "step", "version"])+++class DidTheUpdateLand(unittest.TestCase):++ def test_a_new_version_after_the_press_is_ok(self):+ with _Capture() as cap:+ P._analytics_update_outcome("running_new", "ok", "4.15.0")+ self.assertEqual(cap.events("update_outcome")[0]["outcome"], "ok")++ def test_the_same_version_after_the_press_is_the_hole_we_are_measuring(self):+ with _Capture() as cap:+ P._analytics_update_outcome("restart_pending", "failed", "4.15.0")+ ev = cap.events("update_outcome")[0]+ self.assertEqual(ev["stage"], "restart_pending")+ self.assertEqual(ev["from_version"], "4.15.0")++ def test_the_press_is_recorded_by_the_ui_route(self):+ """The marker is written where the click arrives; without it the next+ boot has nothing to compare against."""+ from pathlib import Path+ src = Path(__file__).with_name("panel") / "routes_meta.py"+ text = src.read_text()+ self.assertIn("analytics_update_pressed", text)+ self.assertIn('("update_now", "banner_update")', text)+++class BothEventsAreDocumented(unittest.TestCase):+ """The page is the specification, and the dry-run suite enforces parity —+ this is the same check from the other direction, so a rename here fails+ close to the code that renamed it."""++ def test_the_vocabularies_appear_on_the_page(self):+ from pathlib import Path+ doc = (Path(__file__).with_name("docs") / "ANALYTICS.md").read_text()+ for word in P._INSTALL_STEPS + P._INSTALL_OUTCOMES + P._UPDATE_STAGES:+ self.assertIn(f"`{word}`", doc, f"{word} is undocumented")+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/test_pasted_paths.py b/test_pasted_paths.pyindex d5deaf3..e5bbdd3 100644--- a/test_pasted_paths.py+++ b/test_pasted_paths.py@@ -234,3 +234,56 @@ def test_no_subprocess_pipe_decodes_strict_utf8():if "errors=" not in window:offenders.append(f"{name}:{i + 1}")assert not offenders, f"strict-utf8 subprocess pipes: {offenders}"+++# ---- the error taxonomy, against strings the FLEET actually sent -------------+#+# Every needle below was read out of PostHog on 2026-09-18, not invented here.+# The audit that prompted this found three faults: two live failure families+# sitting in `other` (our largest people-class), and one needle that was a+# loaded gun.++def test_metal_out_of_memory_is_not_the_watchdog():+ """17 events / 6 installs in 30 days. Both print the same first line, so+ they were one heap; they are different faults with different remedies."""+ oom = ("[METAL] Command buffer execution failed: Insufficient Memory "+ "(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)")+ timeout = ("[METAL] Command buffer execution failed: Caused GPU Timeout Error "+ "(00000002:kIOGPUCommandBufferCallbackErrorTimeout)")+ assert P._analytics_error_class(oom) == "metal_oom"+ assert P._analytics_error_class(timeout) == "metal_watchdog"+++def test_a_missing_file_has_its_own_class():+ assert P._analytics_error_class(+ "[Errno 2] No such file or directory: '<path>'") == "file_missing"+ assert P._analytics_error_class(+ "[Errno 2] No usable temporary directory found in ['<path>']") == "file_missing"+ # ...but the engine tree missing is still the install's fault, not a stray file,+ # and a missing reference image keeps its more specific class.+ assert P._analytics_error_class(+ "[Errno 2] No such file or directory: 'ltx-2-mlx/env/bin/python'") == "venv_broken"+ assert P._analytics_error_class(+ "ref image not found: <path>") == "input_missing"+++def test_cancelled_race_matches_a_cancellation_not_the_word_cancel():+ # What the fleet actually files under this class, all 12 events of it:+ assert P._analytics_error_class(+ "mflux-generate-qwen-edit cancelled by /stop (rc=-9)") == "cancelled_race"+ # ...and the panel's own wording.+ assert P._analytics_error_class("Stopped before the render started.") == "cancelled_race"+ assert P._analytics_error_class("Stopped during joint denoise.") == "cancelled_race"+ # The gun that was pointed at us: a bare "cancel" sat ABOVE oom_jetsam and+ # native_crash, so any future message containing the word would have been+ # filed as a cancel race rather than as the crash it is.+ assert P._analytics_error_class(+ "helper exited from SIGKILL (out of memory (jetsam) or external kill)") == "oom_jetsam"+ assert P._analytics_error_class(+ "the download was cancelled by the server mid-stream") != "cancelled_race"+++def test_the_documented_taxonomy_matches_the_code():+ doc = (ROOT / "docs" / "ANALYTICS.md").read_text(encoding="utf-8")+ for name, _ in P._ANALYTICS_ERROR_CLASSES:+ assert "`%s`" % name in doc, name + " is classified but undocumented"diff --git a/test_tool_resolution.py b/test_tool_resolution.pynew file mode 100644index 0000000..b86ea98--- /dev/null+++ b/test_tool_resolution.py@@ -0,0 +1,125 @@+"""ffmpeg and ffprobe must be found where PINOKIO actually puts them.++The fleet bug, 2026-09-20: five installs across 4.5.0 -> 4.15.0 failed their+export with "ffmpeg not found on PATH". Every engine spawn already prepends+FFMPEG_BIN to the child's PATH, so the miss was one level up — the resolver+knew `pinokio/bin/ffmpeg-env` and nothing else, while Pinokio's bootstrap+installs `ffmpeg=8.1.2` into its CONDA prefix (`pinokio/bin/miniforge/bin`)+and unpacks a private Homebrew beside it. On a Mac with no system ffmpeg the+resolver returned a /usr/local path that does not exist.+"""+import os+import unittest+from pathlib import Path+from tempfile import TemporaryDirectory++import mlx_ltx_panel as P+++class _FakePinokio:+ """A Pinokio home with a binary in one of its tool folders."""++ def __init__(self, rel, name="ffmpeg"):+ self.rel, self.name = rel, name++ def __enter__(self):+ self.tmp = TemporaryDirectory()+ home = Path(self.tmp.name)+ d = home / self.rel+ d.mkdir(parents=True)+ self.bin = d / self.name+ self.bin.write_text("#!/bin/sh\n")+ self.bin.chmod(0o755)+ self.prev = os.environ.get("PINOKIO_HOME")+ os.environ["PINOKIO_HOME"] = str(home)+ self.which = P.shutil.which+ P.shutil.which = lambda _n: None # nothing on PATH+ return self++ def __exit__(self, *a):+ P.shutil.which = self.which+ if self.prev is None:+ os.environ.pop("PINOKIO_HOME", None)+ else:+ os.environ["PINOKIO_HOME"] = self.prev+ self.tmp.cleanup()+++class WherePinokioPutsIt(unittest.TestCase):++ def test_the_conda_prefix_is_searched(self):+ """THE REGRESSION. `bin/miniforge/bin` is where Pinokio's own+ bootstrap log shows ffmpeg being installed."""+ with _FakePinokio("bin/miniforge/bin") as fake:+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)++ def test_the_older_miniconda_prefix_is_searched(self):+ with _FakePinokio("bin/miniconda/bin") as fake:+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)++ def test_the_bundled_homebrew_is_searched(self):+ """Pinokio unpacks its own Homebrew.zip into bin/homebrew."""+ with _FakePinokio("bin/homebrew/bin") as fake:+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)++ def test_the_original_ffmpeg_env_still_wins_first(self):+ """The one folder the old resolver knew must keep working."""+ with _FakePinokio("bin/ffmpeg-env/bin") as fake:+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)++ def test_an_explicit_override_beats_every_search(self):+ with _FakePinokio("bin/miniforge/bin") as fake:+ other = Path(fake.tmp.name) / "my-own-ffmpeg"+ other.write_text("#!/bin/sh\n")+ os.environ["LTX_FFMPEG"] = str(other)+ try:+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), other)+ finally:+ os.environ.pop("LTX_FFMPEG", None)++ def test_an_override_pointing_nowhere_does_not_win(self):+ """A stale LTX_FFMPEG in a shell profile must not disable the search."""+ with _FakePinokio("bin/miniforge/bin") as fake:+ os.environ["LTX_FFMPEG"] = "/nope/ffmpeg"+ try:+ self.assertEqual(P._resolve_tool("ffmpeg", "LTX_FFMPEG"), fake.bin)+ finally:+ os.environ.pop("LTX_FFMPEG", None)++ def test_nothing_found_still_returns_a_path(self):+ """Resolution never raises at import time — a missing binary must+ fail at the call site with a real message, not kill the panel."""+ with _FakePinokio("bin/miniforge/bin"):+ got = P._resolve_tool("definitely-not-a-real-tool", "LTX_NOPE")+ self.assertEqual(got.name, "definitely-not-a-real-tool")+ self.assertFalse(got.exists())++ def test_the_folder_list_is_deduped(self):+ """PINOKIO_HOME is normally ~/pinokio, so both passes find the same+ folders — and this list is printed to the user in the boot warning."""+ dirs = [str(d) for d in P._pinokio_bin_dirs()]+ self.assertEqual(len(dirs), len(set(dirs)))+++class FfprobeTravelsSeparately(unittest.TestCase):++ def test_ffprobe_is_taken_from_beside_ffmpeg_when_it_is_there(self):+ with TemporaryDirectory() as tmp:+ d = Path(tmp)+ (d / "ffmpeg").write_text("")+ (d / "ffprobe").write_text("")+ self.assertEqual(P._resolve_ffprobe(d / "ffmpeg"), d / "ffprobe")++ def test_ffprobe_is_searched_when_it_is_not_beside_ffmpeg(self):+ """The fleet's bare "ffprobe failed:" (3 installs). The old code+ assumed the sibling path existed and handed the caller a path that+ never did."""+ with _FakePinokio("bin/miniforge/bin", name="ffprobe") as fake:+ with TemporaryDirectory() as lonely:+ ffmpeg = Path(lonely) / "ffmpeg"+ ffmpeg.write_text("")+ self.assertEqual(P._resolve_ffprobe(ffmpeg), fake.bin)+++if __name__ == "__main__":+ unittest.main(verbosity=2)diff --git a/webapp/index.html b/webapp/index.htmlindex b566532..64d3ef4 100644--- a/webapp/index.html+++ b/webapp/index.html@@ -5146,6 +5146,12 @@ The last line"></textarea>plus the first line of the error when one fails.<b>Never your prompts, filenames, paths, images or video.</b>You are a random ID, generated on this Mac and tied to nothing.+ The panel sends no location, but the address your computer+ connects from resolves to a <b>country and city</b> at the+ receiver, and those are kept next to the event; the address+ itself is discarded. That is the one thing on this list the+ panel doesn't send and can't stop — turning analytics off is+ what stops it, because then no request is made at all.Everything the panel would send is also written in plain text to<code>state/usage-log.jsonl</code> so you can read it yourself;the full schema is in <code>docs/ANALYTICS.md</code>.