Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
docs: re-anchor all docs to v3.0.6 + post-split git topology (doc audit)
mrbizarro41490efdaa78
diff --git a/AGENTS.md b/AGENTS.mddeleted file mode 100644index af1a878..0000000--- a/AGENTS.md+++ /dev/null@@ -1,1799 +0,0 @@-# Phosphene — agent operating manual--This file is the single source of truth for how an AI assistant should-work on this codebase. Read it first, every session, before touching-any file. The generic Pinokio scaffolding starts at "## Generic Pinokio-Development Guide" further down — that's the upstream template. This top-section overrides it where they conflict.-------## 0. Current state — Phosphene 2.0 (released 2026-05-04)--VERSION file is now `2.0.0`. Git tag `v2.0.0` points at commit `fca78bb`-on `main`. Going forward: `2.0.x` for patches, `2.1.0` for features,-`3.0.0` only for another major story like this one. Pre-2.0 commit-messages and inline comments still reference the `Y1.NNN` sequential-counter (Y1.001 → Y1.039). Treat those as historical; do not introduce-new `Y` labels.--### What ships in 2.0 (capabilities checklist)--- **Four modes**: T2V, I2V, FFLF (keyframe interpolation), Extend.-- **Joint audio + video** in a single LTX 2.3 forward pass — sound is- generated WITH the video. Lip-sync, footsteps, ambient noise all- align at the frame level. mlx 0.31.1 pin is what keeps the vocoder- output at healthy levels (-9 to -15 dB peak on dialogue prompts).-- **Quality tiers**: `quick` / `balanced` / `standard` / `high`. The- earlier `draft` key was renamed; do not reintroduce it. `high` =- Q8 two-stage HQ pipeline + TeaCache, requires Q8 download.-- **Speed modes** for Standard renders: `exact` (full sampler) /- `boost` (skip up to 2 stable middle steps via adaptive caching) /- `turbo` (skip up to 3). 23–34 % wall-time savings on stable scenes.- Disabled for High, Extend, and FFLF where step-skipping breaks- quality contracts.-- **Sharp upscale** = PiperSR (CoreML) on the Apple Neural Engine.- Real x2 detail recovery before the ffmpeg fit/encode tail; ~26 s- overhead on a 5 s × 1024×576 native. The earlier "model" upscale- path (LTX latent x2) was retired in Y1.026; it remains hidden- behind `LTX_ENABLE_MODEL_UPSCALE=1` for lab archaeology only.-- **VAE temporal streaming** — long clips no longer freeze at the- final decode. Patch in `patch_ltx_codec.py` defaults to- `LTX_VAE_STREAMING=auto`: streams when latent frames > 121 (≈ 5 s),- full-decodes shorter clips. Helper-side env-var override keeps- installs that haven't re-applied the patch on the right path too.-- **Filtered hf downloads** (Y1.024+): Q4 ≈ 20 GB, Q8 ≈ 37 GB,- Gemma ≈ 7.5 GB, optional spatial upscaler ≈ 1 GB. Older builds- pulled the full upstream repos and bloated to Q4=56 GB / Q8=82 GB- by including duplicate transformer variants and unused upscalers.- `update.js` deletes the dupes on existing installs.-- **Hardware-tier auto-config** (`base` / `standard` / `high` / `pro`)- with realistic per-tier time estimates surfaced on the Quality- pills. Compact-tier users see honest "~12 min" instead of M-Studio- numbers.-- **Now-card progress bar** (Y1.039) — phase-aware, denoise-step-aware.- Computed server-side from history bucket + helper log signals.- Replaces the old `elapsed / global_avg` ratio that mis-paced both- Quick and High renders.-- **Gallery race-free** (Y1.039) — `list_outputs()` skips files that- ffmpeg is still writing (in-flight skip + 2 s mtime cutoff), and- the `/file` URL carries an `&v=<mtime>` cache-bust so the browser- fetches the freshest bytes when a file is replaced.-- **CivitAI LoRA browser** built into the panel. `mlx_models/loras/`- is scanned at boot; sidecar JSON metadata travels with each safetensors.-- **Phosphene 2.0 release badge** — violet `2.0` chip next to the- wordmark in the panel header. Distinct from the amber `DEV` profile- badge so dev/prod tabs are distinguishable at a glance.--### Empirical benchmarks (M4 Max 64 GB, "Comfortable" tier)--All numbers are real wall-clock from sidecar `elapsed_sec` on Y1.039 /-v2.0.0 runs today. Output mp4 carries audio in every case.--| Config | Frames | Wall |-|---|---|---|-| T2V Quick · exact · 640×480 | 121 (5 s) | ~ 2 m 14 s |-| T2V Standard · exact · 1280×704 | 121 (5 s) | ~ 7 m 40 s |-| T2V Standard · boost · 1280×704 | 121 (5 s) | ~ 6 m 20 s |-| T2V Standard · turbo · 1280×704 | 121 (5 s) | ~ 5 m 26 s |-| T2V High Q8 · 1280×704 | 121 (5 s) | ~ 11 m 51 s |-| T2V Balanced · turbo · 720p Sharp | 121 (5 s) | **~ 3 m 30 s** |-| T2V Balanced · turbo · 720p Sharp | 241 (10 s) | **~ 8 m 07 s** |-| I2V Standard · exact · 1280×704 | 121 (5 s) | ~ 7 m 51 s |-| I2V Balanced · turbo · 720p Sharp | 121 (5 s) | ~ 3 m 37 s |-| I2V Balanced · turbo · 720p Sharp | 241 (10 s) | ~ 8 m 26 s |-| FFLF · clamped to 768 px (Comfortable cap) | 121 (5 s) | ~ 5 m 29 s |-| Extend · 768 px (Comfortable cap), +3 s | source 121 → output 184 | ~ 15 m 50 s |--The marquee number for the launch thread is the-**Balanced + Turbo + 720p Sharp** recipe: 5-sec clip in 3 m 30 s, 10-sec-in 8 m 07 s. That's the recommended default for "fast HD with audio".--### Repository layout (now — both clones documented)--`~/pinokio/api/phosphene.git/` is the **production** clone.-Tracks `main`. Pinokio's daily-driver Phosphene panel runs from here on-port 8198.--`~/pinokio/api/phosphene-dev.git/` is the **dev** clone.-Tracks `dev`. Set up via `scripts/setup_dev_panel.sh`. Runs on port 8199-with the amber `DEV` profile badge in the header. Symlinks-`mlx_models/`, `mlx_outputs/`, `panel_uploads/`, and `state/` from a-shared Pinokio fs.link drive so weights aren't duplicated and outputs-land in one gallery regardless of which panel rendered them.--When a session starts: `git checkout dev` in the dev clone, work, push.-Promote to main only when Mr Bizarro gives the explicit green light. See §7.-------## 1. What Phosphene is--A free desktop panel that wraps Lightricks' **LTX 2.3** model running-natively on Apple's **MLX** framework, exposed as a one-click install-through **Pinokio**. Generates video and audio jointly in a single-forward pass on Apple Silicon Macs. Apple-Silicon-only.--Public artifacts:-- GitHub: `github.com/mrbizarro/phosphene` (public, MIT for the panel)-- Pinokio Discover: searchable as "Phosphene"-- Twitter handle: `@AIBizarrothe`--The panel itself is MIT. Model weights have Lightricks' own license.--## 2. Where to work--**The Pinokio install dir IS the canonical workspace.** There is no-separate dev folder. Old "local dev" copy was deleted to consolidate.--```-~/pinokio/api/phosphene.git/ ← edit here-├── .git/ ← real working copy of mrbizarro/phosphene-├── ltx-2-mlx/ ← upstream package, cloned at install-│ ├── env/ ← Python 3.11 venv-│ └── packages/{ltx-core-mlx,ltx-pipelines-mlx}-├── mlx_models/ ← downloaded weights-│ ├── ltx-2.3-mlx-q4/ ← Q4 base (~20 GB, filtered Y1.024+)-│ ├── ltx-2.3-mlx-q8/ ← optional Q8 (~37 GB, filtered Y1.024+)-│ └── gemma-3-12b-it-4bit/ ← text encoder (~7.5 GB)-├── mlx_outputs/ ← generated mp4s-├── panel_uploads/ ← user-uploaded reference images-├── assets/ ← logos, banner, hero clip-├── launch/ ← launch post drafts (X, Reddit, Pinokio, CivitAI)-├── mlx_ltx_panel.py ← MAIN: HTTP server, HTML UI, queue-├── mlx_warm_helper.py ← subprocess holding MLX pipelines warm-├── pinokio.js / install.js / start.js / update.js / reset.js / download_q8.js-├── patch_ltx_codec.py ← applies patches to upstream package-└── required_files.json ← single source of truth for "installed"-```--To work: `cd ~/pinokio/api/phosphene.git/`, edit, `git commit`,-`git push`. Pinokio's Update button does `git pull` here. No separate-copy to keep in sync.--## 3. Architecture (request → frame)--```-Pinokio app (port 42000)- └── proxies to → mlx_ltx_panel.py (port 8198)- │- ├── serves a single-page HTML/JS UI from one Python process- ├── exposes HTTP API: /run, /status, /upload, /helper/restart, ...- ├── owns a job queue (FIFO) + worker thread- └── speaks JSON-over-stdin/stdout to:- │- └── mlx_warm_helper.py (subprocess in same venv)- ├── holds MLX pipeline objects warm across jobs- ├── pipeline classes from ltx-2-mlx package- ├── reaps itself after LTX_IDLE_TIMEOUT (1800s default)- ├── emits events: log / done / error / exit- └── on death → panel sees stdout EOF-```--Both processes run inside `ltx-2-mlx/env` (the Python 3.11 venv). Helper-restarts cheaply via `POST /helper/restart` to pick up code changes-without restarting the panel.--## 4. Locked dependency versions--These are not arbitrary — every pin is a paid lesson:--| Pin | Why |-|---|---|-| `python>=3.11` (force `uv venv --python 3.11`) | Pinokio's `venv:` shortcut defaults to conda's base 3.10. ltx-core-mlx requires `>=3.11`. |-| `mlx==0.31.1` | mlx 0.31.2 introduced a numerical regression that attenuates LTX 2.3 vocoder output by ~22 dB. Verified: same model + same prompt + same seed → -42 dB peak on 0.31.2 vs -9 dB peak on 0.31.1. |-| `mlx-lm==0.31.1` `mlx-metal==0.31.1` | Same release line, kept consistent with mlx. |-| `huggingface-hub>=1.0` | `hf` CLI replaced `huggingface-cli`; older Pinokio bundles ship < 1.0. |-| `ltx-2-mlx` at `main` (HEAD) | dcd639e (0.1.0) was tried as a pin during audio investigation but breaks the Extend `cfg_scale` API and forces a `transformer.safetensors` symlink hack. HEAD + the mlx pin is the right combination. |--When changing a version, document the test that proved the new pin is OK.--## 5. The patch system (`patch_ltx_codec.py`)--We patch the upstream `ltx-2-mlx` package after `pip install`. Patches-are idempotent and fail loud on upstream drift (don't silently ship a-broken pipeline). Each patch has an `OLD` text it expects to find and a-`NEW` text it replaces with, plus a `marker` to detect "already patched."--Currently shipped patches:--1. **Codec (`yuv444p crf 0` + `+faststart`)** — required.- Patches `ltx_core_mlx/model/video_vae/video_vae.py`. Default ffmpeg- args become lossless H.264 (yuv444p crf 0) and the moov atom moves- to the front of the file so gallery thumbnails render the first- frame without downloading the full clip.--2. **I2V free DiT before VAE decode** — optional (warns if upstream changed).- Patches `ImageToVideoPipeline.generate_and_save` in- `ltx_pipelines_mlx/ti2vid_one_stage.py`. Frees DiT + text encoder +- feature extractor before the VAE decode step.--3. **I2V free vae_encoder + feature_extractor BEFORE denoise** — optional.- Patches `generate_from_image` to null `self.vae_encoder` and- `self.feature_extractor` right after `_encode_text_and_load`- returns. The connector inside feature_extractor is ~5.91 GiB on Q4- and was sitting resident through denoise for no reason.--4. **Base `load()` also clears feature_extractor** — optional.- Patches `TextToVideoPipeline.load`. Drops one extra ~5.91 GiB blob- from the peak when the DiT loads.--5. **VAE temporal streaming with auto-skip on short clips** (Y1.035 +- Y1.037). Patches `VideoDecoder.decode_and_stream` in- `ltx_core_mlx/model/video_vae/video_vae.py`. Upstream's method- *advertised* streaming but actually called `self.decode(latent)` on- the full video volume before iterating frames — caused multi-minute- end-of-render stalls / jetsam on long 720p clips. The patch routes- through `tiled_decode()` with a `TemporalTilingConfig` (default- tile = 64 latent frames, overlap = 24). Default mode is `auto`:- tiling is applied only when latent length > `LTX_VAE_STREAMING_AUTO_MAX_FRAMES`- (default 121 video frames ≈ 5 s). Short clips skip tiling and gain- ~30 s on 5-sec Standard renders. `LTX_VAE_STREAMING=0` forces- full-decode fallback; `=1` forces always-stream. Helper-side- `_apply_vae_streaming_decision()` in `mlx_warm_helper.py` sets the- env var per-job based on `num_frames`, so even pre-2.0 patched- installs that haven't re-run `patch_ltx_codec.py` get the right- behavior.--The "upgrade" path in `apply_patch()` lets old patched installs receive-new patch versions without a venv rebuild — used when we added-`+faststart` to the existing codec patch.--## 6. Critical history (so we don't relitigate)--- **mlx 0.31.2 audio regression** (resolved): symptom was "audio sounds- like static / very quiet." Root cause was an MLX numerical change- attenuating the vocoder output by ~22 dB. Fix: `mlx==0.31.1` pin.- Bisecting which mlx commit caused it is a v1.1 task.-- **dcd639e pin detour** (resolved): briefly pinned `ltx-2-mlx` to- commit `dcd639e` (0.1.0) thinking the audio regression was in- dgrauet's commits. It wasn't. Pin caused Extend API mismatch- (`cfg_scale` kwarg unknown) and required a transformer.safetensors- symlink because the 0.1.0 loader hardcoded that filename. Reverted.-- **Pinokio menu API mismatch** (resolved): `info.path` is a function- on newer Pinokio, a string on older. `pinokio.js` now detects both- shapes via `getInstallRoot(info)`.-- **start.js URL hijack** (resolved): the `errno`/`error:` event- patterns were cargo-culted from comfy.git's start.js. They captured- any "Errno" line from the helper into `input.event[0]` and the panel- URL got set to the literal string "Errno." Removed those patterns.-- **silent helper death** (mitigated, not fixed): on memory-pressured- Macs the helper subprocess is SIGKILL'd by macOS jetsam during I2V- pipeline load. Patches 3 and 4 reduced peak; the panel now names- the exit signal (SIGKILL/SIGSEGV/SIGABRT/SIGBUS) and points users at- `~/Library/Logs/DiagnosticReports/python3.11_*.crash` for diagnosis.-- **HTML5 form validation silent block** (resolved): a hidden Extend- input had `value=2 min=0.4 step=0.5`. Chrome's validator only accepts- values on the (min + N*step) sequence, so 2 was off-grid → invalid- → entire form silently failed to submit. Fixed `min=0.5`.--## 7. Working rules — non-negotiable--### Identity-- **Author/committer:** `mrbizarro <[email protected]>`.- Already set in this repo's local git config. Never override.-- **Never use the user's real first name** in commits, code, file- contents, or any artifact that lands on disk or in git. Same for- any third party (reviewers, collaborators) — refer to them by- technical role only ("upstream review", "external reviewer", "user- feedback") if at all.-- **Never add `Co-Authored-By:` trailers.** Phosphene reads as a- serious project shipped by `mrbizarro`. Co-author trailers showing- "Claude" make it look like a vibe-coded toy.--### Branch model — `main` is production, `dev` is staging (Y1.015+)--**TL;DR for agents — non-negotiable:**--> **Push every commit to `dev`. Never push to `main` unless Mr Bizarro-> explicitly says "push to main" / "ship to production" / "merge to-> main".** "It works on my machine" is not permission. "The tests-> pass" is not permission. Mr Bizarro has a Phosphene Dev panel in his-> Pinokio that pulls from `dev`; he tests there, then tells you when-> to promote.--The why:--- **`main`** is what real users get. Pinokio's Discover URL points- here; every commit pushed to `main` is shipped to every user the- next time they click Update. Treat it as production.-- **`dev`** is the staging branch. Push WIP commits here freely. The- local "Phosphene Dev" Pinokio panel (set up via- `scripts/setup_dev_panel.sh`) tracks this branch on port 8199 and- shows an amber DEV badge in the header.-- **Workflow**:- 1. Default to working on `dev`. `git checkout dev` at the start of- any session that touches Phosphene; if you're on `main`, switch.- 2. Commit + push to `origin/dev` whenever a logical change is done.- 3. **Stop there.** Tell Mr Bizarro it's on dev. Wait for explicit- instruction to promote.- 4. When Mr Bizarro says push to main: `git checkout main && git merge dev- --ff-only && git push origin main`. Then `git checkout dev` so- subsequent work continues on dev.- 5. If a fast-forward isn't possible (rare — usually a hotfix landed- on main directly), say so and ask before doing anything destructive.-- The panel auto-detects which profile it is from `git rev-parse- --abbrev-ref HEAD`: branch == `dev` → port 8199 + DEV badge; anything- else → port 8198 + normal header. No config files to edit per panel.-- `update.js` is branch-aware as of Y1.015 (`git pull --ff-only origin- <current branch>`) so the dev panel's Update pulls dev and the- production panel's Update pulls main, from the same shared code.-- The dev panel symlinks `mlx_models/` from the production install- (saves ~36 GB of duplicate weights). State / outputs / uploads stay- separate per panel.-- Hotfixes bypassing dev — only allowed when Mr Bizarro explicitly authorises- ("hotfix straight to main" or similar). Otherwise, even one-line- fixes go via dev → tested in Phosphene Dev panel → merged.--### Git policy — NEVER FORCE-PUSH `origin/main`-- A history-rewrite + force-push on 2026-05-01 (to scrub identity- leaks from old commits) broke `git pull` for every existing user- whose local clone contained orphaned commits — Pinokio's Update- silently stalled for them. GitHub aggressively GC'd the orphaned- objects within hours, so the pre-rewrite history is unrecoverable.-- **Do not run `git filter-branch`, `git rebase` (on shared history),- `git push --force`, `git push --force-with-lease`, or anything else- that rewrites already-published commits on `origin/main`. Ever.**-- If a commit needs amending and it's already pushed, **make a new- commit that reverts/fixes it.** The history is append-only.-- If the user requests a history rewrite (e.g. "scrub my name"), say- no and explain why: existing users get orphaned, their `git pull`- fails, the entire user base needs out-of-band recovery.-- Recovery from the 2026-05-01 incident — primary path is **Pinokio- Reset → Install** (clean, no shell scripts, models re-download is a- one-time cost). The `recover.sh` script is an escape hatch for users- who want to keep their downloaded models — kept in the repo with a- clear "read before running" header, NOT promoted as a `curl | bash`- command. Don't remove `recover.sh` or weaken `update.js`'s- fetch+reset fallback.--### `fs.link` — orthogonal to the history-divergence story-- `fs.link` is in `install.js` only (NOT `update.js`). It maps the- heavy/personal directories — `mlx_models/`, `mlx_outputs/`,- `panel_uploads/`, `state/` — to a Pinokio virtual drive that lives- outside the panel install dir. After a Pinokio Reset → Install, the- drive is rediscovered and the symlinks reattach, so models +- outputs + settings survive a clean reinstall.-- Why install.js only: future users get the drive on first install;- existing pre-Y1.004 users get it whenever they next reinstall (which- is the recommended recovery path anyway). Adding `fs.link` to- update.js would technically migrate existing real folders into the- drive without a Reset, but it conflates routine code updates with a- one-time 36 GB merge step, so we keep them separate.-- `fs.link` is unrelated to the 2026-05-01 history-rewrite incident- — don't conflate them in commit messages or docs.--### Commit message style-- Subject line in imperative mood, 50–72 chars max, no trailing period.- Use `feat:` / `fix:` / `refactor:` / `docs:` / `chore:` / `perf:` /- `test:` prefixes when they help.-- Body explains the **why**: the symptom, the root cause, the chosen- fix. The diff already shows the what.-- No first-person voice. Third person, neutral, technical.-- No conversation transcript. No "user said X", "after we discussed Y",- "per the review on Tuesday." Just describe the change.-- Cite concrete things: file paths, line numbers, package versions,- upstream commits, signal names. Avoid vague nouns ("the issue").-- Reference upstream context where it helps future maintainers (HF- metadata size, upstream commit SHA, GitHub issue number).-- One commit = one logical change. Don't bundle UI polish with bug- fixes with version pins.--### Examples — bad vs good--❌ Bad:-```-fix the audio thing peanut found--cocktailpeanut said the audio sounded like static so we tried a few-things and pinning mlx fixed it. great catch!-```--✅ Good:-```-fix(audio): pin mlx==0.31.1 to recover LTX 2.3 vocoder amplitude--mlx 0.31.2 introduces a numerical change that attenuates the LTX 2.3-vocoder output by ~22 dB. Verified empirically with the same prompt,-seed, and weights:- mlx 0.31.2 → max_volume -42.8 dB (audible only with significant boost)- mlx 0.31.1 → max_volume -9.2 dB--install.js installs the pinned trio (mlx, mlx-lm, mlx-metal) before-the ltx-* packages so transitive resolution does not pull latest.-update.js force-reinstalls them with --no-deps so existing installs-that already pulled 0.31.2 can recover by clicking Update.-```--### Code style-- Comments explain non-obvious decisions and trade-offs. They earn- their keep by explaining context the diff cannot.-- When fixing a bug, leave a comment with the symptom + cause + fix- near the change so the next reader doesn't reintroduce it.-- Patches in `patch_ltx_codec.py` always include a comment block- above each patch explaining what the upstream code does, why we- patch it, and what the patch achieves.--### Working with the running install-- The Pinokio panel may already be running on port 8198 when you- start a session. **Never spawn your own panel from the command- line** — Pinokio expects to own that port and will fail Start- with `Address already in use` if you do.-- To pick up code changes without restarting Pinokio, hit- `POST /helper/restart` — that reloads the helper subprocess- against the patched site-packages.-- For HTML/JS/CSS changes you do need a panel restart. Either- click Stop + Start in Pinokio, or `pkill -f mlx_ltx_panel.py`- and click Start.--### Disk consolidation-- Old `~/Documents/Codex/...` dev folder was deleted.- Don't recreate. The Pinokio install is canonical.-- `~/.cache/huggingface/hub/models--*` was also deleted. Pinokio- install pulls weights into `mlx_models/` directly via `hf- download --local-dir`, not the cache. Don't reintroduce HF cache- symlinks.--## 8. Test workflow--For HTML/UI/JS changes:-1. Edit `mlx_ltx_panel.py`-2. Restart panel (Pinokio Stop + Start, OR `pkill -f mlx_ltx_panel.py`- then click Start in Pinokio)-3. Hard-refresh browser (Cmd+Shift+R)-4. Verify the served HTML contains the change:- `python3 -c "import urllib.request; print('your-marker' in urllib.request.urlopen('http://127.0.0.1:8198/').read().decode())"`--For helper changes (Python pipeline, patches):-1. Edit `mlx_warm_helper.py` or `patch_ltx_codec.py`-2. If touching patches: re-run `./ltx-2-mlx/env/bin/python3.11 patch_ltx_codec.py`-3. `curl -X POST http://127.0.0.1:8198/helper/restart`-4. Submit a test job via `curl -X POST http://127.0.0.1:8198/run -d "mode=t2v" -d "prompt=..." -d "width=512" -d "height=288" -d "frames=49" -d "steps=8" -d "quality=draft"`-5. Poll `/status` until `current` is null and verify the new- `history[0]` entry is `done`.--For audio testing — do not use generic "wizard in forest" prompts.-Use prompts with explicit dialogue or sound events so artifacts are-audible:--```-A woman in her 30s with auburn hair, sitting at a wooden table in a-softly lit kitchen, speaks directly to the camera with a small smile-and clear, confident voice. She says "I really did not think I would-see you again, but here we are." Audio: clear human voice, slight-room reverb, gentle ambient kitchen sounds.-```--Measure with ffmpeg:-```-ffmpeg -i out.mp4 -af "volumedetect" -f null - 2>&1 | grep -E "max_volume|mean_volume"-```-Healthy levels: `max_volume` between -2 dB and -15 dB on dialogue-prompts. Below -25 dB peak suggests a regression.--## 9. Pinokio lifecycle scripts--| Script | Trigger | What it does |-|---|---|---|-| `install.js` | First Install or Resume Install | Clone `ltx-2-mlx`, force Python 3.11 venv, install MLX trio at pinned versions, install ltx-* packages, run `patch_ltx_codec.py`, `hf download` Q4 + Gemma. Idempotent. |-| `start.js` | Start button | Spawn panel inside the venv with `LTX_*` env vars set. Capture URL via regex with capture group, route to `local.set` via `input.event[1]`. |-| `update.js` | Update button | `git pull` panel + ltx-2-mlx, force-reinstall mlx pin, force-reinstall ltx-* packages, re-run patches. |-| `reset.js` | Reset button | Wipe venv + clones. **Preserves** `mlx_outputs/`, `panel_uploads/`, `mlx_models/` (those are user content / huge weight files). |-| `download_q8.js` | Download Q8 button | `hf download dgrauet/ltx-2.3-mlx-q8` to `mlx_models/ltx-2.3-mlx-q8`. |-| `pinokio.js` | Always | Renders the menu. State driven by `required_files.json` (env marker + per-repo file completeness). |--## 10. Files to never touch unprompted--- Anything inside `mlx_models/` — these are downloaded weights, GB-scale-- Anything inside `mlx_outputs/` — user-generated content-- Anything inside `panel_uploads/` — user uploads-- `panel_queue.json`, `panel_hidden.json` — runtime state-- `logs/`, `cache/`, `__pycache__/` — runtime caches-- The `.clinerules`, `.cursorrules`, `.windsurfrules`, `.geminiignore`,- `GEMINI.md`, `QWEN.md`, `AGENTS.md` files — these are generic Pinokio- scaffolding mirroring this `CLAUDE.md`. Edit `CLAUDE.md` first if a- project-wide change is needed; the rest are autogenerated mirrors.--## 11. HTTP API reference (panel)--The panel listens on `127.0.0.1:8198` and serves a single-page HTML/JS-UI plus a JSON API. Pinokio proxies its own port 42000 to 8198. All-endpoints are unauthenticated (loopback only).--### GET endpoints--| Path | Returns | Notes |-|---|---|---|-| `/` | HTML | The single-page UI. Built from one big template string in `mlx_ltx_panel.py`. |-| `/status` | JSON | Snapshot of `STATE` — running flag, current job, queue, history (top 20), log tail, tier, memory pressure, helper alive flag, comfy PIDs, repo download state. Polled by the UI every ~1s. |-| `/uploads` | JSON | Recent files in `panel_uploads/` for the picker's "click to reuse" strip. |-| `/models` | JSON | Per-repo install completeness — driven by `required_files.json`. |-| `/settings` | JSON | Returns `{settings, presets, default_preset}`. `settings` is the current `panel_settings.json` content. `presets` is the OUTPUT_PRESETS table (label + blurb + pix_fmt + crf per preset) so the UI doesn't duplicate it. |-| `/loras` | JSON | Returns `{user, curated, loras_dir, civitai_auth}`. `user` is the scanned `mlx_models/loras/*.safetensors` list (each with sidecar metadata if present). `curated` is the curated Lightricks repo registry minus the HDR-toggle entry (which is exposed as a plain checkbox, not in the picker). `civitai_auth` is True iff `CIVITAI_API_KEY` is set in the panel's env. |-| `/civitai/search` | JSON | Proxies CivitAI's `/api/v1/models` filtered to `types=LORA&baseModels=LTXV 2.3`. Query params: `query`, `nsfw` (bool), `cursor` (CivitAI uses cursor pagination), `limit`. Returns `{items, next_cursor, has_more}` with each item carrying `{id, version_id, name, creator, description, downloads, nsfw, preview_url, filename, size_kb, download_url, trigger_words, base_model, civitai_url}`. |-| `/file?path=…` | bytes (mp4/png/jpg) | Range-aware video serve so `<video>` tags can seek without re-downloading. |-| `/image?path=…` | bytes (image) | Same idea for image previews. |-| `/sidecar?path=…` | JSON | Per-output sidecar (job params, output stats) if it exists. |-| `/assets/<file>` | bytes | Static files from `assets/`. |--### POST endpoints--| Path | Body | Returns | Effect |-|---|---|---|---|-| `/run` `/queue/add` | urlencoded job spec | `{ok, id}` | Creates a job, appends to queue. Field reference below. |-| `/queue/batch` | urlencoded with `prompts` (newline-separated, `---` chunks) | `{ok, added, ids}` | Splits the prompts on `^\s*---\s*$` boundaries, creates one job per chunk. |-| `/queue/remove` | `id=<jid>` | `{removed: bool}` | Drop a queued job by id. |-| `/queue/clear` | (none) | `{cleared: int}` | Clear all queued jobs (running job continues). |-| `/queue/pause` | (none) | `{paused: true}` | Worker stops popping new jobs after the current one. |-| `/queue/resume` | (none) | `{paused: false}` | Worker resumes. |-| `/output/hide` | `path=<p>` | `{hidden: path}` | Adds path to the gallery's hidden set (persisted). |-| `/output/show` | `path=<p>` | `{shown: path}` | Inverse. |-| `/output/show_all` | (none) | `{unhidden_count: int}` | Clear hidden set. |-| `/upload` | multipart `image=<file>` | `{ok, path}` | Saves to `panel_uploads/<ts>_<safename>`. |-| `/helper/restart` | (none) | `{ok}` | SIGTERM the helper subprocess; the next job auto-respawns it. Useful for picking up site-packages changes. |-| `/settings` | `output_preset` / `output_pix_fmt` / `output_crf` / `civitai_api_key` / `hf_token` (any subset) | `{ok, settings, helper_restarted}` | Update + persist. Public-safe `settings` view in the response (no token values, just `has_*` booleans). Helper killed on codec change OR token change so the next job spawns with fresh env. Form parser uses `keep_blank_values=True` so `civitai_api_key=` (the Clear button) is treated as "remove this token" rather than dropped silently. Empty input on the front-end Apply path is "leave as-is" — explicit clearing goes through the dedicated Clear button. |-| `/loras/refresh` | (none) | `{ok, user, loras_dir}` | Rescan `mlx_models/loras/`. Filesystem is the source of truth; no caching layer to invalidate. |-| `/loras/delete` | `path=<abs path>` | `{ok, removed}` | Delete a user-installed LoRA + its sidecar. Path must resolve inside `mlx_models/loras/` to prevent traversal. |-| `/civitai/download` | `download_url=<civitai url>`, `meta=<json>` | `{ok, name, path, sidecar_path, size_bytes, skipped}` | Stream-download a LoRA from CivitAI into `mlx_models/loras/`, write a sidecar JSON. Refuses non-civitai.com hosts. Writes to a `.partial` then atomic-renames so a kill mid-write leaves nothing the next scan would mistake for a complete file. Surfaces 401 with a remediation hint pointing at `CIVITAI_API_KEY`. |-| `/stop` | (none) | `{ok}` | Cancel the current job (kills helper + mux). Worker advances. |-| `/stop_comfy` | (none) | `{ok, killed}` | SIGTERM any ComfyUI process matching `LTX_COMFY_PATTERN`. |-| `/open_pinokio` | (none) | `{ok}` | macOS-only: focus the Pinokio app. |-| `/models/download` | `key=q4|q8|gemma` | `{ok}` | Spawn `hf download` for the named repo. Streams to log. |-| `/models/cancel` | (none) | `{ok, killed}` | SIGTERM the active download. |-| `/prompt/enhance` | `prompt=<p>` | `{ok, original, enhanced}` | Calls helper's Gemma-rewrite path. |--## 12. Job spec — the contract for `/run` and `/queue/add`--Every form field below maps to `params.<field>` on the resulting job.-`make_job()` in `mlx_ltx_panel.py:1140` is the canonical reference.--| Field | Type | Default | Meaning |-|---|---|---|---|-| `mode` | str | `"t2v"` | One of: `t2v`, `i2v`, `i2v_clean_audio`, `extend`, `keyframe`. |-| `prompt` | str | `"A cinematic atmospheric scene"` | Empty falls back to the default. |-| `width` | int | `1280` | Clamped to `>= 32`. T2V only — image flows derive from aspect+quality. |-| `height` | int | `704` | Same as width. |-| `frames` | int | `121` | Number of video frames. The 8k+1 rule (latents are groups of 8). |-| `steps` | int | `8` | Denoising steps. T2V/I2V Q4 distilled requires `>= 8` (schedule is locked). |-| `seed` | int-as-str | `"-1"` | `"-1"` → random. Stored as string so the form roundtrip is lossless. |-| `image` | path | `REFERENCE` | Only used when `mode != "t2v"`. |-| `audio` | path | `AUDIO_DEFAULT` | Only used when `mode == "i2v_clean_audio"` (mux external audio). |-| `video_path` | path | `""` | **Required for `mode=extend`**. Source mp4 to extend. |-| `extend_frames` | int | `5` | Latent-frame count to add. Each latent = 8 video frames. So 5 latents = 40 frames ≈ 1.67s at 24 fps. |-| `extend_direction` | str | `"after"` | `"after"` or `"before"`. |-| `extend_steps` | int | `12` | Denoising steps for the extend pipeline (uses dev transformer). |-| `extend_cfg` | float | `1.0` | CFG scale. 1.0 = no guidance (faster, less RAM). 3.0 = guided (Quality mode). |-| `start_image` | path | `""` | **Required for `mode=keyframe`**. First frame. |-| `end_image` | path | `""` | **Required for `mode=keyframe`**. Last frame. |-| `enhance` | "on"/"off" | `"off"` | Reserved (panel uses the explicit `/prompt/enhance` button instead). |-| `stop_comfy` | "on"/"off" | `"off"` | Kill ComfyUI before this render to free MPS memory. |-| `open_when_done` | "on"/"off" | `"off"` | macOS `open` the output when finished. |-| `quality` | str | `"balanced"` | `"quick"`, `"balanced"`, `"standard"`, or `"high"`. Drives dimensions, step counts, and pipeline. |-| `accel` | str | `"off"` | `"off"` (alias for exact) / `"boost"` / `"turbo"`. Adaptive-cache denoise; boost skips up to 2 cached steps, turbo up to 3. T2V/I2V Standard only — disabled for High, Extend, FFLF. |-| `upscale` | str | `"off"` | `"off"` / `"fit_720p"` / `"x2"`. Post-render export. Balanced defaults to `fit_720p`. |-| `upscale_method` | str | `"lanczos"` | `"lanczos"` (ffmpeg) or `"pipersr"` (Sharp, ANE-accelerated). PiperSR adds ~26 s on 5-sec 1024×576. |-| `no_music` | "on"/"off" | `"off"` | When on, the panel appends a no-music audio constraint to the prompt at submit time. |-| `preset_label` | str | `""` | Optional human-readable label for the queue/history rows. |--## 13. Mode → pipeline → transformer routing--| Mode | Pipeline class | Transformer | Quality flag | Notes |-|---|---|---|---|---|-| `t2v` | `TextToVideoPipeline` | distilled Q4 | quick → smaller dims; balanced → 1024×576 + 720p export; standard → 1280×704; high → `TwoStageHQPipeline` w/ Q8 dev | Most common path. |-| `i2v` | `ImageToVideoPipeline` | distilled Q4 | same tier ladder as T2V | Adds VAE-encoder pass on the input image. |-| `i2v_clean_audio` | `ImageToVideoPipeline` | distilled Q4 | same | Generates LTX video, then panel-side ffmpeg mux replaces the LTX audio with `audio` param. |-| `extend` | `ExtendPipeline` | **Q8 dev** (Y1.036+) | always uses Q8 `transformer-dev.safetensors`; cfg_scale dial via `extend_cfg` | Heavier (~75–80 s/step on 768×416 cfg=1.0). Pre-Y1.024 a duplicate `transformer-dev.safetensors` shipped in the Q4 dir as download bloat — Extend silently loaded from there. Y1.024 download filter pruned the dupe and exposed that Extend is structurally Q8-class. Panel auto-routes to Q8 dir; UI gates on Q8 download (same shape as FFLF). |-| `keyframe` (FFLF) | `KeyframePipeline` | **Q8 dev + distilled LoRA** | requires Q8 download | Disabled on `base` tier. Comfortable tier clamps to 768 max-side (the stage-1 → stage-2 transition OOMs at 1280×704 on 64 GB). |--The helper `get_pipe(kind)` lazily loads the pipeline class corresponding-to the mode. Switching modes between jobs frees the previous pipeline-(`release_pipelines(keep_kind=...)`) — strict one-pipeline-at-a-time-policy keeps memory bounded.--## 14. Quality tier mapping--`quality` is independent of the hardware tier. Hardware tier *clamps*-the maximum dimensions per mode; `quality` picks the pipeline + step-count *within* that clamp.--| `quality` | Pipeline used | Dimensions | Steps | Approx wall on M4 Max 64 GB |-|---|---|---|---|---|-| `quick` | distilled Q4 | 640×480 (4:3 fast sanity) | 8 | ~ 2 m 14 s @ 121 frames |-| `balanced` | distilled Q4 + ffmpeg or PiperSR fit_720p export | 1024×576 → 1280×720 export | 8 | ~ 4 m 36 s (Lanczos) / ~ 5 m 02 s (Sharp) @ 121 frames |-| `standard` | distilled Q4 | 1280×704 (full target, clamped by tier) | 8 | ~ 7 m 40 s @ 121 frames; ~ 5 m 26 s with `accel=turbo` |-| `high` | `TwoStageHQPipeline` (Q8 dev + res_2s sampler + TeaCache) | 1280×704 | 15 stage-1 + 3 stage-2 | ~ 11 m 51 s @ 121 frames |--`high` and `keyframe` require the Q8 model on disk (~37 GB filtered).-`extend` also requires Q8 since Y1.036 (transformer-dev). Panel surfaces-a "Download Q8" CTA + disables the Generate button when the user picks-a Q8-required mode without Q8 installed.--Independent of `quality`, T2V/I2V Standard takes an `accel` flag-(`exact` / `boost` / `turbo`) that drives an adaptive-cache denoise-loop. Boost can skip up to 2 stable middle steps, Turbo up to 3.-Cached steps near-zero wall; full steps stay ~53 s/it on 1280×704.-Per-step decisions are recorded in the sidecar's `accel_metrics` block.--## 15. Hardware tier system--`SYSTEM_TIER` is detected once at boot from `sysctl hw.memsize`. The-override `LTX_TIER_OVERRIDE=base|standard|high|pro` forces a tier-(useful for testing what other users see). `CAPABILITIES` in-`mlx_ltx_panel.py:584` is the canonical table; abbreviated:--| Tier | RAM range | Friendly | t2v/i2v max dim | Q8 | FFLF | Extend |-|---|---|---|---|---|---|---|-| `base` | < 48 GB | Compact | 768 | ❌ | ❌ | ❌ |-| `standard` | 48–79 GB | Comfortable | 1280 | ✅ | ✅ | ✅ |-| `high` | 80–119 GB | Roomy | 1280 | ✅ | ✅ (more frames) | ✅ |-| `pro` | ≥ 120 GB | Studio | 0 (no clamp) | ✅ | ✅ | ✅ |--The thresholds are intentional. M-Studio 64 GB is the canonical-"standard" hardware. Anything below 48 GB cannot fit Q8 weights-(transformer-dev.safetensors is 19.18 GiB plus working tensors).--## 15a. `panel_settings.json` schema--User-controllable preferences. Persisted at `<repo>/state/panel_settings.json`-(Y1.004+); pre-Y1.004 lived at `<repo>/panel_settings.json` and is migrated-on first boot by `_migrate_state_dir()`. State directory is gitignored.--The whole `state/` dir is symlinked into Pinokio's fs.link drive so it-survives Reset → Reinstall (see "Persistent storage / fs.link" in install.js-+ update.js). All persisted-state files (`panel_settings.json`,-`panel_queue.json`, `panel_hidden.json`) live there.--Read at panel startup; mutated via `POST /settings`. The helper subprocess-inherits codec env vars from this at spawn — when settings change, the-panel kills the helper so the next job picks up new values.--```jsonc-{- "version": 2, // bumped from v1 when secrets added- // Metadata so the UI knows which preset to highlight as active.- // "custom" means pix_fmt + crf were set manually outside any preset.- "output_preset": "standard" | "archival" | "web" | "custom",- "output_pix_fmt": "yuv420p" | "yuv444p" | "yuv422p"- | "yuv420p10le" | "yuv422p10le" | "yuv444p10le",- "output_crf": "0".."30", // string (passed via env vars)-- // Secrets. Stored locally only, never sent over the network except- // as Authorization headers / ?token= URL params to the upstream- // services (civitai.com, huggingface.co). NEVER returned by GET- // /settings — the public response surfaces booleans only.- "civitai_api_key": "", // empty = not configured- "hf_token": "" // empty = not configured-}-```--### `GET /settings` security contract--The full settings dict — including `civitai_api_key` and `hf_token` —-is read by the backend only. The HTTP API exposes a *masked* view via-`get_settings_public()`:--```jsonc-{- "version": 2,- "output_preset": "standard",- "output_pix_fmt": "yuv420p",- "output_crf": "18",- "has_civitai_key": false, // bool, never the value- "has_hf_token": false // bool, never the value-}-```--This applies on every response that includes settings — success path,-validation errors, helper-restart confirmations. A user with the panel-proxied through ngrok / tunneled to the LAN never leaks their saved-secrets through a status poll.--### Token resolution priority--`_active_civitai_key()` and `_active_hf_token()` resolve the active-token in this order:--1. The saved value in `panel_settings.json` (set via the Settings- modal). This is the path normal users take.-2. The corresponding env var (`CIVITAI_API_KEY` / `HF_TOKEN`). Power- users with shell-level config keep working unchanged.--Empty string in #1 falls through to #2. The settings UI's "Clear"-button writes an empty string to #1, so a user who configured a key-in the UI and then wants to revert to env-var-based config can do-so cleanly.--For HF specifically, `huggingface_hub.snapshot_download` *also* reads-`~/.cache/huggingface/token` (the file `hf auth login` writes). We-don't override that file — if neither settings nor `HF_TOKEN` env is-set, the library falls back to that cached file. This means power-users who already ran `hf auth login` get gated LoRAs working without-touching the panel's token settings.--Preset table is the single source of truth, defined in `OUTPUT_PRESETS`-in `mlx_ltx_panel.py`:--| key | pix_fmt | crf | Approx 5s @ 1280×704 | Use case |-|---|---|---|---|---|-| `standard` ⭐ default | yuv420p | 18 | ~7 MB | Visually lossless, plays everywhere (X / IG / Discord) |-| `archival` (UI label "Video production") | yuv444p | 0 | ~50 MB | Mathematically lossless, full 4:4:4 chroma. For color grading, compositing, VFX, anywhere downstream re-encode is expected. Internal key kept as `archival` for backwards compat with settings files saved before the rename. |-| `web` | yuv420p | 23 | ~3 MB | Smallest, mobile / embedding |--`+faststart` is always applied (codec patch) regardless of preset, so-gallery thumbnails always render the first frame instantly.--When adding a new preset: edit `OUTPUT_PRESETS` in `mlx_ltx_panel.py`-and the `order` array in the settings modal JS (`openSettingsModal`).-The blurb shown in the UI is read from the server at modal-open time,-so server is the source of truth — no JS strings to keep in sync.--## 15b. LoRA system--LTX 2.3 supports LoRA adapters via the upstream `apply_loras()` function-in `ltx_core_mlx.loader.fuse_loras`. The fusion is **weight-level**: at-pipeline load time, each LoRA's `(lora_A, lora_B)` matrix pair is-multiplied + added to the corresponding base-model weight, modified in-place, then quantized. This is *not* a runtime adapter — switching-LoRAs requires reloading the pipeline.--### Integration hook--The base `TextToVideoPipeline.load()` checks-`getattr(self, "_pending_loras", None)` and, if present, calls-`_fuse_pending_loras(transformer_weights, lora_paths)` before-quantizing the transformer. Set `pipe._pending_loras = [(path, strength), ...]`-between construction and the first job.--The wrapper in `mlx_warm_helper.py:_attach_loras` does this. It also-resolves HuggingFace repo IDs (`Lightricks/LTX-2.3-22b-IC-LoRA-HDR`)-to local files via `snapshot_download`, picking the largest-`.safetensors` in the repo (LoRA repos sometimes ship auxiliary files-like scene embeddings alongside the main weights).--### Pipeline cache invalidation--`get_pipe(kind, loras=...)` keys the cache on `(kind, frozen LoRA set)`.-Changing the LoRA set forces a pipeline rebuild — the cached pipeline's-weights have the previous LoRA set fused in and there's no clean way-to "unfuse." Cost is the standard ~30s pipeline reload.--The frozen LoRA set is order-insensitive (set hash, not list hash) —-fusion is commutative so `[A, B]` and `[B, A]` produce identical-weights and should hit the same cache entry.--### `mlx_models/loras/` directory--User-installed LoRAs live as `<name>.safetensors` files at the install-root's `mlx_models/loras/` directory. An optional sidecar JSON next to-each `.safetensors` carries display metadata:--```jsonc-{- "name": "Crisp Enhance",- "description": "Sharper detail and edge enhancement.",- "trigger_words": [], // optional, prepended to prompt at submit- "recommended_strength": 0.8, // 0.0..1.5 typical, displayed as default in slider- "preview_url": "https://...", // optional, shown in the picker- "base_model": "LTXV 2.3",- "civitai_id": 2530917, // optional, for attribution- "civitai_version_id": 2844417,- "civitai_url": "https://civitai.com/models/2530917",- "downloaded_at": "2026-05-01T..."-}-```--Sidecar absent → panel falls back to the filename as the display name-and zero metadata. Sidecar shape matches the JSON `_civitai_download`-writes, so a CivitAI install lands a usable picker entry without any-hand-editing.--### Curated LoRA registry--`CURATED_LORAS` in `mlx_ltx_panel.py` lists Lightricks' official LoRAs-that we know about. Currently:--| key | repo | UI surface | Note |-|---|---|---|---|-| `hdr` | `Lightricks/LTX-2.3-22b-IC-LoRA-HDR` | "HDR" toggle in form | Hidden from picker (`is_hdr_toggle: True`); exposed as a plain checkbox |-| `motion-track` | `Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control` | Listed in picker | IC-LoRA, needs video conditioning (not yet wired in panel UI) |-| `union-control` | `Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control` | Listed in picker | Same |--All three are **gated** on Hugging Face — users must accept the license-on the model page AND have a valid HF token (`hf auth login`) before-the helper's `snapshot_download` will work. The helper translates the-upstream `GatedRepoError` into a clear actionable error.--### Job spec extension--The `generate` and `extend` actions accept an optional `loras` param:--```jsonc-{- "action": "generate",- "id": "...",- "params": {- "mode": "t2v",- "prompt": "...",- "loras": [- {"path": "/abs/path/to/style.safetensors", "strength": 0.8},- {"path": "Lightricks/LTX-2.3-22b-IC-LoRA-HDR", "strength": 1.0}- ],- ...- }-}-```--Path may be a local file or an HF repo id. Strength is clamped to-±2.0 by the panel before sending; values outside the typical 0.0..1.5-range are usually nonsense and risk numerical issues during fusion.--### HDR toggle semantics--The `hdr` boolean on the job (separate from `loras`) is a UI shortcut.-Worker-side, `run_job_inner` resolves it to-`{"path": "Lightricks/LTX-2.3-22b-IC-LoRA-HDR", "strength": 1.0}`-and appends to the LoRA list before calling the helper. Users see-"HDR" as a feature; the LoRA implementation is hidden.--### CivitAI bridge--CivitAI's REST API is the de-facto LoRA distribution hub for LTX 2.3.-Two endpoints proxy through the panel:--- `/civitai/search` filtered to `baseModels=LTXV 2.3` (the canonical- string CivitAI uses for LTX-2.3 LoRAs as of 2026-05). Cursor-paginated.-- `/civitai/download` streams a `.safetensors` into `mlx_models/loras/`- + writes a sidecar JSON.--Both endpoints accept an optional `CIVITAI_API_KEY` env var. As of 2025,-CivitAI requires a token for most LoRA downloads (search still works-without). The token goes in `?token=<key>` on the URL — many CDN hops-only honor that form, not the Authorization header.--### Adding a new curated LoRA--1. Edit `CURATED_LORAS` in `mlx_ltx_panel.py` with `id`, `name`,- `description`, `repo_id`, `default_strength`, and either- `is_hdr_toggle: True` (= surfaces as a header pill, hidden from- picker) or `False` (= shows in picker as a usable entry).-2. If `is_hdr_toggle`, also add the HTML toggle pill (next to "No- music"), the JS sync block, and the `make_job` resolution that maps- the boolean to the curated repo id.--## 16. `required_files.json` schema--Single source of truth for "what counts as installed." Read by:-- `pinokio.js` → menu rendering (Install vs Resume Install vs Start)-- `mlx_ltx_panel.py` → `/status` and per-job validation-- (indirectly) `install.js` and `download_q8.js` — their `hf download`- paths must match the `local_dir` declared here.--```jsonc-{- "_comment": "...",- "repos": [- {- "key": "q4" | "gemma" | "q8",- "kind": "base" | "optional", // "base" = blocks Start, "optional" = unlock features- "name": "human-readable label",- "blurb": "what this is for",- "repo_id": "<hf-org/repo>", // for `hf download`- "local_dir": "mlx_models/<repo-name>", // relative to install root- "size_gb": 25, // approximate, displayed in UI- "files": [ // every required file inside local_dir- "transformer-distilled.safetensors",- "connector.safetensors",- ...- ]- }- ],- "env": {- "marker_paths": [ // OR-checked; first existing path wins- "ltx-2-mlx/env/pyvenv.cfg", // canonical Pinokio install- "ltx-2-mlx/.venv/pyvenv.cfg", // alternate manual layouts- ...- ]- },- "min_size_bytes": 1024 // file must be >= this to count as "complete"-}-```--When the upstream model layout changes, **edit this file first**. The-menu, panel, and install scripts auto-pick up the change.--## 17. Helper protocol (panel ↔ helper)--The helper subprocess (`mlx_warm_helper.py`) speaks newline-delimited-JSON over stdin/stdout. The panel's `WarmHelper` class wraps it.--### Actions panel sends--```jsonc-// Generation request (T2V, I2V, I2V-clean-audio)-{ "action": "generate",- "id": "<job-id>",- "params": { "mode": "t2v"|"i2v"|"i2v_clean_audio",- "prompt": "...", "output_path": "...",- "height": 704, "width": 1280, "frames": 121,- "seed": 42, "steps": 8, "image": "<path>" } }--// Extend request-{ "action": "extend",- "id": "<job-id>",- "params": { "video_path": "...", "prompt": "...",- "extend_frames": 5, "extend_direction": "after",- "extend_steps": 12, "cfg_scale": 1.0,- "output_path": "...", "seed": 42 } }--// Keyframe (FFLF)-{ "action": "keyframe", "id": "...", "params": {...} }--// Prompt rewrite (Gemma)-{ "action": "enhance", "prompt": "..." }-```--### Events helper emits--```jsonc-{ "event": "ready", "model": "<path>", "low_memory": true } // on startup-{ "event": "log", "line": "free-form panel log line" } // any informational message-{ "event": "done", "id": "...", "output": "<path>",- "elapsed_sec": 33.6, "seed_used": 42 }-{ "event": "error", "id": "...", "error": "<message>",- "trace": "<python traceback>" }-{ "event": "exit", "reason": "python_normal_exit"- | "sigterm(15)"- | "sigterm(2)"- | "idle" } // graceful exit-{ "event": "enhanced", "original": "...", "enhanced": "..." } // /prompt/enhance result-```--**No exit event + stdout pipe closes = non-graceful death** (SIGKILL,-SIGSEGV, SIGABRT, SIGBUS). The panel inspects `proc.poll()` returncode-and translates the negative signal number to a name in the user-facing-error.--### Step breadcrumb log lines--The helper emits these inside `generate`/`extend` actions so the panel-can localize a silent death to a phase:--```-step:get_pipe kind=t2v-step:get_pipe done-step:generate_and_save mode=i2v 1280x704 121f steps=8-step:generate_and_save done-```--When a job dies silently, the panel log's last `step:*` line tells you-which phase (pipeline init vs actual generation).--## 18. Memory budget model--Concrete weight sizes (HuggingFace repo metadata, Q4 unless noted):--| Component | Size on disk | Resident peak (rough) | Notes |-|---|---|---|---|-| `transformer-distilled.safetensors` (Q4) | 10.54 GiB | ~12 GiB | T2V/I2V "DiT" |-| `transformer-dev.safetensors` (Q4) | 11.3 GiB (rough) | ~13 GiB | Used by Extend |-| `transformer-dev.safetensors` (Q8) | 19.18 GiB | ~22 GiB | High-quality + FFLF |-| `connector.safetensors` (Q4) | 5.91 GiB | ~6 GiB | Lives inside `feature_extractor` |-| `Gemma-3-12B-it-4bit` | 7.48 GiB | ~9 GiB | Text encoder, freed before DiT load |-| `vae_decoder.safetensors` | 814 MB | ~1 GiB | Used at the end of generation |-| `vae_encoder.safetensors` | 638 MB | ~800 MB | I2V only (encodes input image) |-| `audio_vae.safetensors` | 107 MB | ~150 MB | |-| `vocoder.safetensors` | 258 MB | ~350 MB | |--Peak during T2V Standard 1280×704 (with patches applied): roughly-DiT + active activations + decoder ≈ 20–22 GiB resident. Without-the patches it's 26+ GiB and OOMs on memory-pressured 64 GB Macs.--Peak during I2V is +800 MB for the encoder pass + ~6 GiB for the-connector that lingered in `feature_extractor` (Patches 3 + 4 fix-the lingering — see §5).--macOS `jetsam` is the OOM killer. It enforces memory pressure based-on a percentage that varies with paging activity, swap usage, and other-heuristics — there is no fixed "kill at X% pressure" rule. Closing-Chrome, Slack, iOS Simulator before a Standard render is the safest-single thing a user can do.--## 19. Debug recipes--### "Helper died silently — where did it die?"--```bash-# Scan the panel log for the last step:* breadcrumb-curl -s http://127.0.0.1:8198/status | python3 -c \- "import json,sys; [print(l) for l in json.load(sys.stdin)['log'] if 'step:' in l][-3:]"-```--If last step is `step:get_pipe` → death during pipeline init (model-load). Likely OOM or weight-corrupt. If last step is-`step:generate_and_save` → death during gen (denoise / VAE decode).-Likely OOM peak or Metal kernel issue.--For non-OOM crashes, check `~/Library/Logs/DiagnosticReports/python3.11_*.crash`-or `*.ips` from around the crash time.--### "Panel says port 8198 already in use"--```bash-lsof -nP -iTCP:8198 -sTCP:LISTEN # who has it-pkill -f mlx_ltx_panel.py # nuke all panels-```--Common cause: a previous manual `python mlx_ltx_panel.py` is still-running. Pinokio's start.js can't take over a port someone else owns.-**Don't spawn the panel manually unless you're going to kill it before-handing back.**--### "I edited mlx_ltx_panel.py but the UI hasn't changed"--The HTML/JS is baked into the Python file as a template string. The-running panel served the old version. Restart it:-- via Pinokio: Stop → Start-- or via shell: `pkill -f mlx_ltx_panel.py` then click Start in Pinokio--Then **hard-refresh the browser** (Cmd+Shift+R) to bypass JS cache.--### "I edited the helper or a patch but nothing changed"--```bash-# 1. Re-apply patches (idempotent, applies to site-packages)-cd ~/pinokio/api/phosphene.git-./ltx-2-mlx/env/bin/python3.11 patch_ltx_codec.py--# 2. Restart the helper subprocess (panel keeps running)-curl -X POST http://127.0.0.1:8198/helper/restart--# 3. Submit a test job-curl -X POST http://127.0.0.1:8198/run \- -d "mode=t2v" -d "prompt=quick test" \- -d "width=512" -d "height=288" -d "frames=49" \- -d "steps=8" -d "quality=draft"--# 4. Wait for done-until [ "$(curl -s http://127.0.0.1:8198/status | python3 -c 'import json,sys;d=json.load(sys.stdin);print(0 if d.get(\"current\") else 1)')" = "1" ]; do sleep 5; done-```--### "How loud / clean is the audio in this clip?"--```bash-FFPROBE="/Volumes/Topaz Video AI 4.0.6/Topaz Video AI.app/Contents/MacOS/ffprobe"-FFMPEG="/Volumes/Topaz Video AI 4.0.6/Topaz Video AI.app/Contents/MacOS/ffmpeg"-"$FFMPEG" -i out.mp4 -af "volumedetect" -f null - 2>&1 | grep -E "max_volume|mean_volume"-```--Healthy on dialogue: `max_volume` between -2 and -15 dB. Below -25 dB-peak is the regression fingerprint — usually means a wrong mlx version.--### "Re-encode for X / web (yuv420p, faststart, 8 Mbps)"--```bash-"$FFMPEG" -y -i in.mp4 \- -c:v h264_videotoolbox -profile:v high -pix_fmt yuv420p \- -b:v 8M -maxrate 12M -bufsize 16M \- -movflags +faststart \- -c:a aac -b:a 192k \- out_x.mp4-```--X (Twitter) rejects yuv444p with a misleading "aspect ratio" error.-Use this recipe for any social upload.--### "Stuck queue / can't cancel"--```bash-curl -X POST http://127.0.0.1:8198/stop # cancel current job-curl -X POST http://127.0.0.1:8198/queue/clear # empty the queue-curl -X POST http://127.0.0.1:8198/helper/restart # nuke the helper-```--### "Force re-download a corrupt model file"--```bash-# Delete the file in question, then click Install in Pinokio (resumes-# only the missing files; intact ones are skipped via `hf` hash check).-rm ~/pinokio/api/phosphene.git/mlx_models/ltx-2.3-mlx-q4/<bad-file>.safetensors-# Then in Pinokio: Phosphene → Install (or Resume Install)-```--### "Force a different hardware tier for testing"--```bash-LTX_TIER_OVERRIDE=base ./run_panel.sh # pretend this Mac is < 48 GB-```--`run_panel.sh` actively unsets this on every run unless explicitly set-on the command line — there's a hard-learned bug history about leaked-overrides bleeding across panel restarts.--### "What does `/status` actually return?"--```bash-curl -s http://127.0.0.1:8198/status | python3 -m json.tool | head -60-```--Top-level keys: `running`, `paused`, `current`, `queue`, `history`,-`log`, `tier`, `memory`, `helper`, `download`, `repos_ready`,-`repos_total`, `base_available`, `q8_available`, `q8_missing`,-`comfy_pids`, `outputs`, `server_now`, `avg_elapsed_sec`.--## 20. Patch authoring template (`patch_ltx_codec.py`)--When adding a new patch:--```python-# ---- Patch N: <one-line summary> ---------------------------------------------# Why: <symptom + root cause + why this fix>. Reference upstream-# file:line and any external context (commit SHA, GitHub issue, paper).-PATCH_<NAME>_OLD = '''<exact text expected to be in the upstream file>'''-PATCH_<NAME>_NEW = '''<replacement text — include a `# PATCHED (LTX23MLX): ...` comment>'''-```--Then in `main()`:--```python-target = _find("<relative/path/to/upstream.py>")-outcome = apply_patch(- target, PATCH_<NAME>_OLD, PATCH_<NAME>_NEW,- marker="<unique substring that proves NEW is on disk>",- label="<NAME>",- upgrade_marker="<optional: substring that distinguishes a newer NEW from an older NEW>",-)-# If this patch is optional (don't fail install on drift):-if outcome in (OUTCOME_DRIFT, OUTCOME_MISSING):- print(" [<NAME>] note: ...", file=sys.stderr)- outcome = OUTCOME_ALREADY-outcomes.append(("<NAME>", outcome))-```--Conventions:-- `marker` is a substring **inside** `PATCH_<NAME>_NEW` that's unlikely- to appear in the unpatched file. Convention: include- `PATCHED (LTX23MLX...)` somewhere in the new text.-- `upgrade_marker` lets you ship a v2 of a patch without forcing users- to nuke their venv. Example: codec patch's `+faststart` was added- later; `upgrade_marker="+faststart"` distinguishes v1 (no faststart)- from v2 and rewrites the line in place.-- Patches that target an upstream code structure that may legitimately- vanish (e.g. an internal helper that gets refactored away) should be- marked **optional** in main() so install warns instead of fails.--## 21. Glossary--- **LTX 2.3** — Lightricks' latent text-to-video diffusion model with- joint audio synthesis. Trained on a large internal dataset.-- **MLX** — Apple's open-source ML array framework, native to Metal.- Like JAX/PyTorch but built for Apple Silicon. Not a torch shim.-- **Q4 / Q8** — Quantization levels of the LTX 2.3 weights. Q4 is- 4-bit quantized (smaller, faster, slightly lower quality), Q8 is- 8-bit (larger, slower, sharper detail). Phosphene defaults to Q4;- Q8 is an optional upgrade.-- **DiT** — Diffusion Transformer. The main denoising network. The- largest single weight tensor.-- **VAE** — Variational Autoencoder. Encodes pixels ↔ latents. The- encoder is used in I2V (encodes the input image); the decoder is- used at the end of every generation.-- **FFLF** — First-Frame / Last-Frame interpolation. Provide two images,- the model generates the motion between them. Requires Q8.-- **BWE** — Bandwidth Extension. Inside the audio vocoder, upsamples- a low-rate signal to full audio rate (108 sequential BigVGAN convs).-- **Vocoder** — Converts mel-spectrogram features to PCM audio. The- final stage of LTX's audio pipeline.-- **TeaCache** — A diffusion-step caching technique that skips- redundant computation when consecutive sigmas are similar. Used in- the High quality two-stage path.-- **jetsam** — macOS' kernel-level memory pressure killer. Sends SIGKILL- to processes when system memory gets tight. The dominant cause of- silent helper death on Macs at full RAM.-- **moov atom** — The MP4 metadata index. With `+faststart` it lives- at the front of the file (allows progressive playback / instant- thumbnails); without, it lives at the end (browser must download- the whole file before showing the first frame).-- **Pinokio** — One-click installer + runtime for AI apps on Mac/Linux/Windows.- Phosphene ships as a Pinokio app. cocktailpeanut authored Pinokio.-- **mrbizarro** — The GitHub handle that authors this repo. Use this,- always, in commits and metadata.--## 22. Open questions & v1.1 candidates--Tracked here so they don't get lost between sessions.--- **mlx 0.31.1 → 0.31.2 audio bisect** — find the exact MLX commit- that attenuates LTX 2.3's vocoder by 22 dB. File upstream issue.-- **A2V (audio-to-video) mode** — `ltx-2-mlx` upstream has- `a2vid_two_stage.py` and `a2vid_two_stage_hq.py` but the panel- doesn't expose them. UI work + helper plumbing required.-- **In-app HF token field** — currently we recommend `hf auth login`- in Terminal. A panel-side settings field that writes- `~/.cache/huggingface/token` would be friendlier for non-CLI users.-- **Pre-flight RAM advisory** — before submitting a heavy job, warn- if `s.memory.swap_gb > 8` or `pressure_pct > 75`. Suggest Draft.-- **Audio mode dropdown** — supersede the binary "No music" toggle- with: With music / Voice + ambient / Sound effects / Silent. Each- option appends/prepends a different prompt constraint.-- **Deeper helper breadcrumbs** — current `step:*` markers cover the- outer call. Adding `image_encode_start`, `text_encode_done`,- `denoise_start`, `decode_start` inside the pipeline (via wrapper- patches) would localize silent deaths down to the phase.-- **Q8 path validation** — Q8 generation has not been end-to-end- tested on a 64 GB Mac since the codec/I2V patches landed. Schedule- a 25 GB Q8 download + High quality T2V test.-- **FFLF validation** — same as above. Requires Q8.--## 23. External references--- LTX-2 paper / model card: https://huggingface.co/Lightricks-- MLX framework: https://github.com/ml-explore/mlx-- `ltx-2-mlx` (the MLX port we wrap): https://github.com/dgrauet/ltx-2-mlx-- Pinokio runtime: https://pinokio.computer-- Q4 weights: `dgrauet/ltx-2.3-mlx-q4` on HF-- Q8 weights: `dgrauet/ltx-2.3-mlx-q8` on HF-- Gemma text encoder: `mlx-community/gemma-3-12b-it-4bit` on HF-------# Generic Pinokio Development Guide--(Everything below this line is the upstream Pinokio scaffolding shipped-with every new project. Useful as a reference when working with the-launcher scripts but the Phosphene-specific rules above take-precedence on conflict.)--# Development Guide for Pinokio Projects--## Non-Negotiable Execution Workflow--To guarantee every contribution follows this guide precisely, obey this checklist **before any edits** and **again before finalizing**. Do not skip or reorder.-1. **AGENTS Snapshot:** Re-open this file and write down (in your working notes or response draft) the exact sections relevant to the requested task. No work begins until this snapshot exists.-2. **Destination Resolution:** Before creating or editing any Pinokio launcher files, resolve `PINOKIO_HOME` to an absolute path and record the intended destination root. If running outside Pinokio's own managed runtime, resolve in this order: `~/.pinokio/config.json` `home`, then `GET http://127.0.0.1:42000/pinokio/home` and use its `path` value, and if loopback is unreachable but `access` exists in `~/.pinokio/config.json`, retry the same request against `<protocol>://<host>:<port>/pinokio/home`, then the `PINOKIO_HOME` environment variable. If `PINOKIO_HOME` is still unresolved, stop and ask the user. Never silently use the current workspace as the launcher destination.-3. **Example Lock-in:** Identify the closest matching script in `~/pinokio/prototype/system/examples`. Record its path and keep it open while editing. Every launcher change must mirror that reference unless the user explicitly instructs otherwise.-4. **Pre-flight Checklist:** Convert the applicable rules from this document and `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md into a task-specific checklist (install/start/reset/update structure, regex patterns, menu defaults, log checks, destination path, etc.). Confirm each item is ticked **before** making changes.-5. **Mid-task Verification:** Any time you touch a Pinokio script, cross-check the corresponding example line to ensure syntax and structure match. Document the reference (example path + line) in your reasoning.-6. **Exit Checklist:** Before responding to the user, revisit the pre-flight checklist and explicitly confirm every item is satisfied. If anything diverges from the example or these rules, fix it first.--If any step cannot be completed, stop immediately and ask the user how to proceed. These six steps are mandatory for every session.--### Critical Pattern Lock: Capturing Web UI URLs--When writing `start.js` (or any script that needs to surface a web URL for a server):--1. **Always copy the capture block from an example such as `system/examples/mochi/start.js`.**-```javascript-on: [{- event: "/(http:\\/\\/[0-9.:]+)/",- done: true-}]-```--2. **Set the local variable using the captured match exactly as below (The regex capture object is passed in as `input.event`, so need to use the index 1 inside the parenthesis):**-```javascript-{- method: "local.set",- params: {- url: "{{input.event[1]}}"- }-}-```--3. Always try to come up with the most generic regex.-4. During the exit checklist, explicitly confirm that the `url` local variable is set via `local.set` API by using the captured regex object as passed in as `input.event` from the previous `shell.run` step.--Deviation from this pattern requires written approval from the user.--- Make sure to keep this entire document and `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md in memory with high priority before making any decision. Pinokio is a system that makes it easy to write launchers through scripting by providing various cross-platform APIs, so whenever possible you should prioritize using Pinokio API over lower level APIs.-- When writing pinokio scripts, ALWAYS check the examples folder (in ~/pinokio/prototype/system/examples folder) to see if there are existing example scripts you can imitate, instead of assuming syntax.-- When implementing pinokio script APIs and you cannot infer the syntax just based on the examples, always search the API documentation `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md to use the correct syntax instead of assuming the syntax.-- When trying to fix something or figure out what's going on, ALWAYS start by checking the `logs` folder before doing anything else, as mentioned in the "Troubleshooting with Logs" section.-- Finally, make sure to ALWAYS follow all the items in the "best practices" section below.--## Determine User Intent-If the initial prompt is simply a URL and nothing else, check the website content and determine the intent, and ask the user to confirm. For example a URL may point to--1. A Tutorial: the intent may be to implement a demo for the tutorial and build a launcher.-2. A Demo: the intent may be a 1-click launcher for the demo-3. Open source project: the intent may be a 1-click launcher for the project-4. Regular website: the intent may be to clone the website and a launcher.-5. There can be other cases, but try to guess.--## Working With Launchers--Apply this section only when the task is to create, modify, debug, review, or document a Pinokio launcher project.--If the request is not about launcher work, do not force an app-launcher vs plugin-launcher decision.--When the task does involve launcher work, first determine whether the request is for an app launcher or a plugin launcher. These are separate project types and must not be mixed.--### Mandatory Destination Resolution-- Before creating, editing, or moving any launcher files, resolve `PINOKIO_HOME` to an absolute path.-- If running outside Pinokio's own managed runtime, resolve `PINOKIO_HOME` in this order:- 1. `~/.pinokio/config.json` -> `home`- 2. `GET http://127.0.0.1:42000/pinokio/home` -> `path`- 3. If loopback is unreachable but `access` exists in `~/.pinokio/config.json`, retry the same request against `<protocol>://<host>:<port>/pinokio/home`- 4. `PINOKIO_HOME` environment variable-- Normalize the resolved value to an absolute path before using it.-- If neither source yields a valid `PINOKIO_HOME`, stop immediately and ask the user how to proceed. Do not guess. Do not silently fall back to the current workspace.-- If the current workspace is outside the resolved `PINOKIO_HOME/api` and `PINOKIO_HOME/plugin` trees, treat the current workspace only as source material, reference material, or evidence. Do not create the launcher in that workspace.-- Before any file creation, record and verify the exact target path:- - app launcher: `PINOKIO_HOME/api/<unique_name>`- - plugin launcher: `PINOKIO_HOME/plugin/<unique_name>`-- If the unique folder name is not obvious, ask the user before creating the target folder.--### 1. App launchers-- App launchers must live under `PINOKIO_HOME/api/<unique_name>`.-- App launchers are usually project-local launchers that manage one app in its own launcher/app folder.-- If you are already inside the target app launcher folder, build in that folder.-- If you are not already inside an app launcher folder, create a new folder under `PINOKIO_HOME/api/<unique_name>`.-- If the folder name is not obvious from the project or the user has not provided one, ask the user to confirm the folder name before creating it.-- Do not place app launchers under `PINOKIO_HOME/plugin`.--### 2. Plugin launchers-- Plugin launchers must live under `PINOKIO_HOME/plugin/<unique_name>`.-- Plugin launchers are reusable shared tools that are installed once and then used across many different folders.-- Standalone plugin launchers should keep `path: "plugin"` in the root `pinokio.js` so Pinokio installs them into `PINOKIO_HOME/plugin`.-- If you are already inside the target plugin launcher folder, build in that folder.-- If you are not already inside a plugin launcher folder, create a new folder under `PINOKIO_HOME/plugin/<unique_name>`.-- If the folder name is not obvious from the project or the user has not provided one, ask the user to confirm the folder name before creating it.-- Do not place plugin launchers under `PINOKIO_HOME/api`.-- When a plugin is meant to operate on the user's current project, its `run` step should target the caller's folder with `{{args.cwd}}` instead of the plugin folder itself.--### 3. Apply structure rules only after choosing the launcher type-- App launchers and plugin launchers are peers. Do not treat a plugin launcher as a special case of an app launcher, or vice versa.-- Decide the launcher type and destination folder first, then apply the project structure and script rules below.--## Project Structure--Pinokio projects normally follow a standardized structure with app logic separated from launcher scripts:--Pinokio projects follow a standardized structure with app logic separated from launcher scripts:--```-project-root/-├── app/ # Self-contained app logic (can be standalone repo)-│ ├── package.json # Node.js projects-│ ├── requirements.txt # Python projects-│ └── ... # Other language-specific files-├── README.md # Documentation-├── install.js # Installation script-├── start.js # Launch script-├── update.js # Update script (for updating the scripts and app logic to the latest)-├── reset.js # Reset dependencies script-├── pinokio.js # UI generator script-└── pinokio.json # Metadata (title, description, icon)-```--- Keep app code in `/app` folder only (never in root)-- Store all launcher files in project root (never in `/app`)-- `/app` folder should be self-contained and publishable---The only exceptions are serverless web apps---purely frontend only web applications that do NOT have a server component and connect to 3rd party API endpoints--in which case the folder structure looks like the following (No need for launcher scripts since the index.html will automatically launch. The only thing needed is the metadata file named pinokio.json):--```-project-root/-├── index.html # The serverless web app entry point-├── ...-├── README.md # Documentation-└── pinokio.json # Metadata (title, description, icon)-```--IMPORTANT: ALWAYS try to follow the best practices in the examples folder (~/pinokio/prototype/system/examples) instead of trying to come up with your own structure. The examples have been optimized for the best user experience.--## Launcher Project Working Directory--- The project working directory for a script is always the same directory as the script location.-- For example, when you run `shell.run` API inside `pinokio/start.js`, the default path for shell execution is `pinokio`.-- If the launcher files are in the project root path, then the default path for shell execution is the project root.-- Therefore, it is important to specify the correct `path` attribute when running `shell.run` API commands.--Example: in the following project structure:--```-project-root/-├── pinokio/ # Pinokio launcher folder-│ ├── start.js # Launch script-│ ├── pinokio.js # UI generator script-│ └── pinokio.json # Metadata (title, description, icon)-└─── backend/- ├── requirements.txt # App dependencies- └── app.py # App code-```--The `pinokio/start.js` should use the correct path `../backend` as the `path` attribute, as follows:--```-{- run: [{- ...- }, {- method: "shell.run",- params: {- message: "python app.py",- venv: "env",- path: "../backend"- }- }, {- ...- }]-}-```--## Development Workflow--### 1. Understanding the Project-- Check `SPEC.md` in project root. If the file exists, use that to learn about the project details (what and how to build)-- If no `SPEC.md` exists, build based on user requirements-### 2. Modifying Existing Launcher Projects-If we are starting with existing launcher script files, work with the existing files instead of coming up with your own.-- **Preserve existing functionality:** Only modify necessary parts-- **Don't touch working scripts:** Unless adding/updating specific commands-- **Follow existing conventions:** Match the style and structure already present-### 3. Try to adopt from examples as much as possible-- If starting from scratch, first determine what type of project you will be building, and then check the examples folder (~/pinokio/prototype/system/examples) to see if you can adopt them instead of coming up everything from scratch.-- Even if there are no relevant examples, check the examples to get inspiration for how you would structure the script files even if you have to write from scratch.-### 4. Writing from scratch as a last resort-If there are relevant examples to adopt from, write the scripts from scratch, but just make sure to follow the requirements in the next section.-### 5. Debugging-When the user reports something is not working, ALWAYS inspect the logs folder to get all the execution logs. For more info on how this works, check the "Troubleshooting with Logs" section below.--## Script Requirements--### 1. 1-click launchable-- The main purpose of Pinokio is to provide an easy interface to invoke commands, which may include launching servers, installing programs, etc. Make sure the final product provides ways to install, launch, reset, and update whatever is needed.--### 2. Write Documentation-- ALWAYS write a documentation. A documentation must be stored as `README.md` in the project root folder, along with the rest of the pinokio launcher script files. A documentation file must contain:- - What the app does- - How to use the app- - API documentation for programmatically accessing the app's main features (Javascript, Python, and Curl)--## Types of launchers-## 1. Launching servers-- When an app requires launching a server, here are the commonly used scripts:- - `install.js`: a script to install the app- - `start.js`: a script to start the app- - `reset.js`: a script to reset all the dependencies installed in the `install.js` step. used if the user wants to restart from scratch- - `update.js`: a script to update the launcher AND the app in case there are new updates. Involves pulling in the relevant git repositories installed through `install.js` (often it's the script repo and some git repositories cloned through the install steps if any)- - `pinokio.js`: the launcher script that ties all of the above scripts together by providing a UI that links to these scripts.- - `pinokio.json`: For metadata--Here's a basic server launcher script example (`start.js`). Unless there's a special reason you need to use another pattern, this is the most recommended pattern. Use this or adopt it as needed, but NEVER try something else unless there's a good reason you should not take this approach:--```javascript-module.exports = {- // By setting daemon: true, the script keeps running even after all items in the `run` array finishes running. Mandatory for launching servers, since otherwise the shells running the server process will get killed after the scripts finish running.- daemon: true,- run: [- {- // The "shell.run" API for running a shell session- method: "shell.run",- params: {- // Edit 'venv' to customize the venv folder path- venv: "env",- // Edit 'env' to customize environment variables (see documentation)- env: { },- // Edit 'path' to customize the path to start the shell from- path: "app",- // Edit 'message' to customize the commands, or to run multiple commands- message: [- "python app.py",- ],- on: [{- // The regular expression pattern to monitor.- // Whenever each "event" pattern occurs in the shell terminal, the shell will return,- // and the script will go onto the next step.- // The regular expression match object will be passed on to the next step as `input.event`- // Useful for capturing the URL at which the server is running (in case the server prints some message about where the server is running)- "event": "/(http:\/\/\\S+)/",-- // Use "done": true to move to the next step while keeping the shell alive.- // Use "kill": true to move to the next step after killing the shell.- "done": true- }]- }- },- {- // This step sets the local variable 'url'.- // This local variable will be used in pinokio.js to display the "Open WebUI" tab when the value is set.- method: "local.set",- params: {- // the input.event is the regular expression match object from the previous step- // In this example, since the pattern was "/(http:\/\/\\S+)/", input.event[1] will include the exact http url match caputred by the parenthesis.- // Therefore setting the local variable 'url'- url: "{{input.event[1]}}"- }- }- ]-}-```--## 2. Launching serverless web apps--- In case of purely static web apps WITHOUT servers or backends (for example an HTML based app that connects to 3rd party servers--either remote or localhost), we do NOT need the launcher scripts.-- In these cases, simply include `index.html` in the project root folder and everything should automatically work. No need for any of the pinokio launcher scripts. (Do-- You still need to include the metadata file so they show up properly on pinokio:- - `pinokio.json`: For metadata--## 3. Launching quick scripts without web UI--- In many cases, we may not even need a web UI, but instead just a simple way to run scripts.-- This may include TUI (Terminal User Interface) apps, a simple launcher-- In these cases, all we need is the launcher file `pinokio.js`, which may link to multiple scripts. In this case, there are no web apps (no serverless apsp, no servers), but instead just the default pinokio launcher UI that calls a bunch of scripts.-- Here are some examples:- - A pinokio script to toggle the desktop theme between dark and light- - Write some code (python or javascript or whatever)- - Write a `toggle.js` pinokio script that executes the code- - Write a `pinokio.js` launcher script to create a sidebar UI that displays the `toggle.js` so the user can simply click the "toggle" button to toggle back and forth between desktop themes- - A pinokio script to fetch some file- - Write some code (python or javascript or whatever)- - Write a `fetch.js` pinokio script that executes the code- - Write a `pinokio.js` launcher script to create a sidebar UI that displays the `fetch.js` so the user can simply click the "fetch" button to fetch some data.-- You still need to include the metadata file so they show up properly on pinokio:- - `pinokio.json`: For metadata--## API--This section lists all the script APIs available on Pinokio. To learn the details of how they are used, you can:-1. Check the examples in the ~/pinokio/prototype/system/examples folder-2. Read the `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md further documentation on the full syntax--### Script API--These APIs can be used to describe each step in a pinokio script:-- shell.run: run shell commands-- input: accept user input-- filepicker: accept file upload-- fs.write: write to file-- fs.read: read from file-- fs.copy: copy files-- fs.download: download files-- fs.link: create a symbolic link (or junction on windows) for folders-- fs.open: open the system file explorer at a given path-- fs.cat: print file contents-- jump: jump to a specific step-- local.set: set local variables for the currently running script-- json.set: update a json file-- json.rm: remove keys from a json file-- json.get: get values from a json file-- log: print to the web terminal-- net: make network requests-- notify: display a notification-- script.download: download a script from a git uri-- script.start: start a script-- script.stop: stop a script-- script.return: return values if the current script was called by a caller script, so the caller script can utilize the return value as `input`-- web.open: open a url in web browser-- hf.download: huggingfac-cli download API-### Template variables-The following variables are accessible inside template expressions (example `{{args.command}` in scripts, resulting in dynamic behaviors of scripts:-- input: An input is a variable that gets passed from one RPC call to the next-- args: args is the parameter object that gets passed into the script (via pinokio.js `params`). Unlike `input` which takes the value passed in from the immediately previous step, `args` is a global value that is the same through out the entire script execution.-- local: local variable object that can be set with `local.set` API-- self: refers to the script file itself (which is JSON or JavaScript). For example if `start.js` that's currently running has `daemon: true` set, `{{self.daemon}}` will evaluate to true.-- uri: The current script uri-- port: The next available port. Very useful when you need to launch an app at a specific port without port conflicts.-- cwd: The current script execution folder path-- platform: The current operating system. May be one of the following: `darwin`, `win32`, `linux`-- arch: The current system architecture. May be one of the following: x32, x64, arm, arm64, s390, s390x, mipsel, ia32, mips, ppc, ppc64-- gpus: array of available GPUs on the machine (example: `['apple']`, `['nvidia']`)-- gpu: the first available GPU (example: `nvidia`)-- current: The current variable points to the index of the currently executing instruction within the run array.-- next: The next variable points to the index of the next instruction to be executed. (null if the current instruction is the final instruction in the run array)-- envs: You can access the environment variables of the currently running process with envs object.-- which: Check whether a command exists and return its absolute path (example: `{{which('winget')}}`). This is the correct way to resolve command paths inside reproducible Pinokio scripts, including custom shell selection such as `shell: "{{which('bash')}}"`. If you are outside a Pinokio-managed shell and only need to inspect Pinokio's environment manually, use `pterm which <command>`, but do NOT copy that user-specific absolute path into launcher scripts.-- exists: Check whether a file or folder exists at the specified relative path (example: `"when": "{{!exists('app')}}"`). Can be used with the `when` attribute to determine a path's existence and trigger custom logic. Use relative paths and it will resolve automatically to the current execution folder.-- running: Check whether a script file is running (example: `"when": "{{!running('start.js')}}"`). Can be used with the `when` attribute to determine a path's existence and trigger custom logic. Use relative paths and it will resolve automatically to the current execution folder.-- os: Pinokio exposes the node.js os module through the os variable.-- path: Pinokio exposes the node.js path module through the os variable (example: `{{path.resolve(...)}}`--## System Capabilities-### Package Management (Use in Order of Preference)-The following package managers come pre-installed with Pinokio, so whenever you need to install a 3rd party binary, remember that these are available. Also, you can assume these are available and include the following package manager commands in Pinokio scripts:-1. **UV** - For Python packages (preferred over pip)-2. **NPM** - For Node.js packages-3. **Conda** - For cross-platform 3rd party binaries-4. **Brew** - Mac-only fallback when other options unavailable-5. **Git** - Full access to git is available.-6. **Bun** - For managing bun packages-**Important:** Include all install commands in the install script for reproducibility.-### HTTPS Proxy Support-- All HTTP servers automatically get HTTPS endpoints-- Convention: `http://localhost:<PORT>` → `https://<PORT>.localhost`-- Full proxy list available at: `http://localhost:2019/config/`-### Pterm Features:-- **Clipboard Access:** Read from or Write to system clipboard via pinokio Pterm CLI (`pterm clipboard` command.)-- **Notifications:** Send desktop alerts via pinokio pterm CLI (`pterm push` command.)-- **Script Testing:** Run launcher scripts via pinokio pterm CLI (`pterm start` command.)-- **File Selection:** Use built-in filepicker for user file/folder input (`pterm filepicker` command.)-- **Command Path Resolution:** Inspect the absolute path of any command as seen by Pinokio via `pterm which <command>`. Use this for debugging or external local tooling, especially when a helper process did not inherit Pinokio's `PATH`, for example `pterm which bash` on Windows. Do NOT hardcode the returned absolute path into launcher scripts; use `which()` or `kernel.which()` in the script itself instead.-- **Git Operations:** Clone repositories, push to GitHub-- **GitHub Integration:** Full GitHub CLI support (`gh` commands)--## Troubleshooting with Logs-Pinokio stores the logs for everything that happened in terminal at the following locations, so you can make use of them to determine what's going on:--### Log Structure-In case there is a `pinokio` folder in the project root folder, you should be able to find the logs folder here:--```-pinokio/-└── logs/ # Direct user interaction logs- ├── api/ # Launcher script logs (install.js, start.js, etc.)- ├── dev/ # AI coding tool logs (organized by tool)- └── shell/ # Direct user interaction logs-```--Otherwise, the `logs` folder should be found at project root:--```-logs/-├── api/ # Launcher script logs (install.js, start.js, etc.)-├── dev/ # AI coding tool logs (organized by tool)-└── shell/ # Direct user interaction logs-```--### Log File Naming-- Unix timestamps for each session-- Special "latest" file contains most recent session logs-- **Default:** Use "latest" files for current issues-- **Historical:** Use timestamped files for pattern analysis and the full history.--## Best practices-### 0. Always reference the logs when debugging-- When the user asks to fix something, ALWAYS check the logs folder first to check what went wrong. Check the "Troubleshooting with Logs" section.-### 1. Shell commands for launching programs-- Launch flags related- - Try as hard as possible to minimize launch flags and parameters when launching an app. For example, instead of `python app.py --port 8610`, try to do `python app.py` unless really necessary. The only exception is when the only way to launch the app is to specify the flags.-- Launch IP related- - Always try to find a way to launch servers at 127.0.0.1 or localhost, often by specifying launch flags or using environment variables. Some apps launch apps at 0.0.0.0 by default but we do not want this.-- Launch Port related- - In case the app itself automatically launches at the next available port by default (for example Gradio does this), do NOT specify port, since it's taken care of by the app itself. Always try to minimize the amount of code.- - If the install instruction says to launch at a specific port, don't use the hardcoded port they suggest since there's a risk of port conflicts. Instead, use Pinokio's `{{port}}` template expression to automatically get the next available port.- - For example, if the instruction says `python app.py --port 7860`, don't use that hardcoded port since there might be another app running at that port. Instead, automatically assign the next available port like this: `python app.py --port {{port}}`- - Note that the `{{port}}` expression always returns the next immediately available port for each step, so if you have multiple steps in a script and use `{{port}}` in multiple steps, the value will be different. So if you want to launch at the next available port and then later reuse that port, you will need to first use `{{port}}` to get the next available port, and save the value in local variable using `local.set`, and then use the `{{local.<variable_name>}}` expression later.-### 2. shell.run API-- When writing `shell.run` API requests, always use relative paths (no absolute paths) for the `path` field. For example, if you need to run a command from `app` folder, the `path` attribute should simply be `app`, instead of its full absolute path.-- If a launcher needs to use a command that Pinokio already provides, prefer resolving it with `{{which('command')}}` inside the script instead of assuming the command name will always be on `PATH`.-- Do NOT automatically avoid `bash`-based install commands on Windows. Pinokio's Windows environment includes `bash` through its bundled toolchain, so commands such as `curl -fsSL ... | bash` are acceptable when they run inside a Pinokio-managed shell and there is no simpler cross-platform alternative.-- If a Windows launcher needs to run the shell itself in bash instead of the default `cmd.exe`, set `shell: "{{which('bash')}}"` on the `shell.run` step.-- If a separate debugging process or external local tool did not inherit Pinokio's environment, you may use `pterm which <command>` to inspect what Pinokio would resolve. Do NOT turn that result into a hardcoded script path; for launcher scripts, always use `which()` or `kernel.which()` so the script stays reproducible across machines.-### 2. Package managers-- When installing python packages, try best to use `uv` instead of `pip` even if the install instruction says to use pip. Instead of `pip install -r requirements.txt`, you can simply use `uv pip install -r requirements.txt` for example. Even if the project's own README says use pip or poetry, first check if there's a way to use uv instead.-- When you need to install some global package, try to use `conda` as much as possible. Even on macs, `brew` should be only used if there are no `conda` options.-### 3. Minimal Always-- If you are starting with existing script files, before modifying, creating, or removing any script files, first look at `pinokio.js` to understand which script files are actually used in the launcher. The only script files used are the ones mentioned in the `pinokio.js` file. The `pinokio.js` file is the file that constructs the UI dynamically.-- Do not create a redundant script file that does something that already exists. Instead modify the existing script file for the feature. For example, do not create an `install.json` file for installation if `install.js` already exists. Instead, modify the `install.js` file.-- Pinokio accepts both JSON and JS script files, so when determining whether a script for a specific purpose already exists, check both JSON and JS files mentioned in the `pinokio.js` file. Do not create script files for rendundant purpose.-- When building launchers for existing projects cloned from a repository, try to stay away from modifying the project folder (the `~/pinokio/api/phosphene.git` folder), even if installations are failing. Instead, try to work around it by creating additional files in the launcher folder, and using those files IN ADDITION to the default project.- - The only exception when you may need to make changes to the project folder is when the user explicitly wants to modify the existing project. Otherwise if the purpose is to simply write a launcher, the app logic folder should never be touched.-- When running shell commands, take full advantage of the Pinokio `shell.run` API, which provides features like `env`, `venv`, `input`, `path`, `sudo`, `on`, etc. which can greatly reduce the amount of script code.- - Python apps: Always use virtual environments via `venv` attribute. This attribute automatically creates a venv or uses if it already exists.-### 4. Try to support Cross-platform as much as possible-- Use cross-platform shell commands only.-- This means, prefer to use commands that work on all platforms instead of the current platform.-- If there are no cross platform commands, use Pinokio's template expressions to conditionally use commands depending on `platform`, `arch`, etc.-- Also try to utilize Pinokio Pterm APIs for various cross-platform system features.-- If it is impossible to implement a cross platform solution (due to the nature of the project itself), set the `platform`, `arch`, and/or `gpu` attributes of the `pinokio.json` file to declare the limitation.-- Pinokio provides various APIs for cross-platform way of calling commonly used system functions, or lets you selectively run commands depending on `platform`, `arch`, etc.-### 5. Do not make assumptions about Pinokio API-- Do NOT make assumptions about which Pinokio APIs exist. Check the documentation.-- Do NOT make assumptions about the Pinokio API syntax. Follow the documentation.-### 6. Scripts must be able to replicate install and launch steps 100%-- The whole point of the scripts is for others to easily download and invoke them via Pinokio interface with one click. Therefore, do not assume the end user's system state, and make everything self-contained.-- When a 3rd party package needs to be installed, or a 3rd party repository needs to be downloaded, include them in the scripts.-### 7 Dynamic UI rendering-- The `pinokio.js` launcher script can change dynamically depending on the current state of the script execution. Which means, depending on what the file returns, it can determine what the sidebar looks like at any given moment of the script cycle.- - `info.exists(relative_path)`: The `info.exists` can be used to check whether a relative path (relative to the script root path) exists. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `info.running(relative_path)`: The `info.running` can be used to check whether a script at a relative path is currently running (relative to the script root path) exists. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `info.local(relative_path)`: The `info.local` can be used to return all the local variables tied to a script that's currently running. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `default`: set the `default` attribute on any menu item for whichever menu needs to be selected by default at a given step. Some example scenarios:- - during the install process, the `install.js` menu item needs to be set as the `default`, so it automatically executes the script- - when launching the `start.js` menu item needs to be set as the `default`, so it automatically executes the script- - after the app has launched, the `default` needs to be set on the web UI URL, so the user is sent to the actual app automatically.- - Check the examples in the ~/pinokio/prototype/system/examples folder to see how these are being used.-### 8. No need for stop scripts-- `pinokio.js` does NOT need a separate `stop` script. Every script that can be started can also be natively stopped through the Pinokio UI, therefore you do not need a separate stop script for start script-### 9. Writing launchers for existing projects-- When writing or modifying pinokio launcher scripts, figure out the install/launch steps by reading the project folder `app`.-- In most cases, the `README.md` file in the `~/pinokio/api/phosphene.git` folder contains the instructions needed to install and run the app, but if not, figure out by scanning the rest of the project files.-- Install scripts should work for each specific operating system, so ignore Docker related instructions. Instead use install/launch instructions for each platform.-### 10. Retrofitting an already-working setup-- Sometimes the user starts outside Pinokio, gets an app working through ad-hoc commands, and only later asks to turn that work into a Pinokio launcher.-- In this case, treat the current working setup and the successful session context as the highest priority source of truth. Do NOT restart from scratch if the app is already working.-- First capture the exact install and launch steps that already succeeded: cloned repositories, package manager commands, environment variables, model downloads, ports, working directories, helper scripts, and any fixes that were required.-- Then convert that knowledge into reproducible Pinokio scripts (`install.js`, `start.js`, `reset.js`, `update.js`, `pinokio.js`, `pinokio.json`) instead of telling the user to manually repeat the ad-hoc process.-- When the successful setup lives in a non-Pinokio folder, use that folder as evidence only. Resolve `PINOKIO_HOME` first, then produce the final launcher in the proper Pinokio location (`PINOKIO_HOME/api/<unique_name>` or `PINOKIO_HOME/plugin/<unique_name>`) unless the user explicitly asks for another layout.-- Replace machine-specific state with reproducible steps. Never hardcode absolute paths, user-specific cache locations, session-only ports, or one-off manual edits if they can be expressed in the launcher.-- Do not simply encode whatever happened to work on the current machine. Generalize the result into the broadest practical cross-platform, cross-machine launcher, and if limitations are unavoidable, declare them explicitly in `pinokio.json` instead of silently baking in local assumptions.-- If the app is already installed but the exact setup steps are partially missing, inspect the current working tree, generated files, dependency manifests, shell history when available, and logs to reconstruct the smallest reliable install and start flow.-- Verify from as clean a state as practical. A launcher is only done when another user could reproduce the working result without relying on undocumented steps from the original ad-hoc session.-### 11. Don't use Docker unless really necessary-- Some projects suggest docker as installation options. But even in these cases, try to find "development" options to launch the app without relying on Docker, as much as possible. We do not need Docker since we can automatically install and launch apps specifically for the user's platform, since we can write scripts that run cross platform.-### 12. pinokio.json-- Do not touch the `version` field since the version is the script schema version and the one pre-set in `pinokio.js` must be used.-- `icon`: It's best if we have a user friendly icon to represent the app, so try to get an image and link it from `pinokio.json`.- - If the git repository for the `~/pinokio/api/phosphene.git` folder points to GitHub (for example https://github.com/<USERNAME>/<REPO_NAME>`, ask the user if they want to download the icon from GitHub, and if approved, get the `avatar_url` by fetching `https://api.github.com/users/<USERNAME>`, and then download the image to the root folder as `icon.png`, and set `icon.png` as the `icon` field of the `pinokio.json`.-### 13. Gitignore-- When a launcher involves cloning 3rd party repositories, downloading files dynamically, or some files to be generated, these need to be included in the .gitignore file. This may include things like:- - Cloning git repositories- - Downloading files- - Dynamically creating files during installation or running, such as Sqlite Databases, or environment variables, or anything specific to the user.-- Make sure these file paths are included in the .gitignore file, and if not, include them in .gitignore.--## AI Libraries (Pytorch, Xformers, Triton, Sageattention, etc.)-If the launcher is for running AI models locally, the install script must declare the AI bundle so Pinokio can install the machine-level prerequisites before the script runs:--```-// install.js-module.exports = {- requires: {- bundle: "ai"- },- run: [- ...- ]-}-```--This is required even when the script also uses `torch.js`. The AI bundle is what triggers installation of common local AI prerequisites such as CUDA on NVIDIA systems and Hugging Face CLI.--If the launcher has a dedicated built-in script named `torch.js`, it can be used as follows:--```-// install.js-module.exports = {- requires: {- bundle: "ai"- },- run: [- // Edit this step with your custom install commands- {- method: "shell.run",- params: {- venv: "venv", // Edit this to customize the venv folder path- path: "app",- message: [- "uv pip install -r requirements.txt"- ],- }- },- // Delete this step if your project does not use torch- {- method: "script.start",- params: {- uri: "torch.js",- params: {- path: "app",- venv: "venv", // Edit this to customize the venv folder path- // xformers: true // uncomment this line if your project requires xformers- // triton: true // uncomment this line if your project requires triton- // sageattention: true // uncomment this line if your project requires sageattention- // flashattention: true // uncomment this line if your project requires flashattention- }- }- },- ]-}-```--The `torch.js` script also includes ways to install pytorch dependent libraries such as xformers, triton, sagetattention. If any of these libraries need to be installed, use the torch.js to install in order to install them cross platform.---## Quick Reference-### Essential Documentation-- **Pinokio Programming:** See `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md → "Programming Pinokio" section-- **Dynamic Menus:** See `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md → "Dynamic menu rendering" section-- **CLI Commands:** See `PTERM.md` at ~/pinokio/prototype/PTERM.md-### Common Patterns-- **Python Virtual Env:** `shell.run` with `venv` attribute-- **Cross-platform Commands:** Always test on multiple platforms-- **Error Handling:** Check logs/api for launcher issues-- **GitHub Operations:** Use `gh` CLI for advanced GitHub features-## Development Principles-1. **Minimize Shell Usage:** Leverage API parameters instead of raw commands-2. **Maintain Separation:** Keep app logic and launchers separate-3. **Follow Conventions:** Match existing project patterns-4. **Test Thoroughly:** Use CLI to verify launcher functionality-5. **Document Changes:** Update relevant metadata and documentationdiff --git a/AGENTS.md b/AGENTS.mdnew file mode 120000index 0000000..681311e--- /dev/null+++ b/AGENTS.md@@ -0,0 +1 @@+CLAUDE.md\ No newline at end of filediff --git a/CLAUDE.md b/CLAUDE.mdindex 2b0c6c9..208ad33 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -8,14 +8,27 @@ section overrides it where they conflict.----## 0. Current state — Phosphene 2.0 (released 2026-05-04)--VERSION file is now `2.0.0`. Git tag `v2.0.0` points at commit `fca78bb`-on `main`. Going forward: `2.0.x` for patches, `2.1.0` for features,-`3.0.0` only for another major story like this one. Pre-2.0 commit-messages and inline comments still reference the `Y1.NNN` sequential-counter (Y1.001 → Y1.039). Treat those as historical; do not introduce-new `Y` labels.+## 0. Current state — Phosphene v3.0.6 (current public release, 2026-05-31)++Public release is **v3.0.6**. The v3.0 line launched 2026-05-23 and has+shipped six patch releases: v3.0.0 (Characters/Voice/Image Studio/A2V) →+v3.0.1 (FFLF crash fix) → v3.0.2 (Boost/Turbo accel restored after a+2-month silent regression) → v3.0.3 (HiDream hidden, issue #15) → v3.0.4+(CivitAI SSL cert) → v3.0.5 (A2V kwarg signature shim, #5) → v3.0.6+(deep-review hardening: CivitAI token-leak fix, dead-HDR revival,+GPU-contention guard, boot version-gate). Published git tags: `v2.0.0`–+`v2.0.5` then `v3.0.0`–`v3.0.6` — there is **no `v2.0.6` tag** (that was+the dev codename that became v3.0.0). The `VERSION` file should read+`3.0.6`. Versioning: `3.0.x` for patches. Pre-2.0 commit messages /+comments still use the `Y1.NNN` counter (Y1.001 → Y1.039) — historical;+do not introduce new `Y` labels.++The authoritative current engineering snapshot is+`~/AI/projects/phosphene/notes/DEEP_REVIEW_2026-05.md` (verified risk+register + 5-phase stabilization plan). The `## 2.0` capability list+below is retained as historical baseline — everything in it shipped; the+v3.0 additions (Characters, voice LoRA, Image Studio, A2V, Train tab,+capability tiers) are now baseline too.### What ships in 2.0 (capabilities checklist)@@ -285,33 +298,37 @@ new patch versions without a venv rebuild — used when we added**TL;DR for agents — non-negotiable:**-> **Push every commit to `dev`. Never push to `main` unless Mr Bizarro-> explicitly says "push to main" / "ship to production" / "merge to-> main".** "It works on my machine" is not permission. "The tests-> pass" is not permission. Mr Bizarro has a Phosphene Dev panel in his-> Pinokio that pulls from `dev`; he tests there, then tells you when-> to promote.+> **REPO SPLIT (2026-05-22) — read this carefully, the old "push to+> origin/dev" instruction is now WRONG.** There are two repos:+> `mrbizarro/phosphene` (PUBLIC, `main`-only — what users install) and+> `mrbizarro/phosphene-beta` (PRIVATE — daily dev). The public `dev`+> branch was **DELETED**. The local working branch is still named `dev`+> but it **tracks `beta/main`** (the private repo). So: **push daily work+> to `beta` (private). Never push to a public `origin/dev` — it does not+> exist.** Promotion to PUBLIC `main` is the gated step that needs Mr+> Bizarro's explicit "push to main" / "ship to production". "It works on+> my machine" / "tests pass" is not permission.The why:-- **`main`** is what real users get. Pinokio's Discover URL points- here; every commit pushed to `main` is shipped to every user the- next time they click Update. Treat it as production.-- **`dev`** is the staging branch. Push WIP commits here freely. The- local "Phosphene Dev" Pinokio panel (set up via- `scripts/setup_dev_panel.sh`) tracks this branch on port 8199 and- shows an amber DEV badge in the header.+- **public `origin/main`** is what real users get. Pinokio's Discover URL+ points here; every commit promoted to public `main` ships to every user+ on their next Update. Production.+- **private `beta/main`** is the staging line. The local `dev` branch+ tracks it; push WIP here freely (`git push beta dev:main`). The local+ "Phosphene Dev" panel tracks `dev` on port 8199 with an amber DEV badge.- **Workflow**:- 1. Default to working on `dev`. `git checkout dev` at the start of- any session that touches Phosphene; if you're on `main`, switch.- 2. Commit + push to `origin/dev` whenever a logical change is done.- 3. **Stop there.** Tell Mr Bizarro it's on dev. Wait for explicit- instruction to promote.- 4. When Mr Bizarro says push to main: `git checkout main && git merge dev- --ff-only && git push origin main`. Then `git checkout dev` so- subsequent work continues on dev.- 5. If a fast-forward isn't possible (rare — usually a hotfix landed- on main directly), say so and ask before doing anything destructive.+ 1. `git checkout dev` at the start of any Phosphene session. `git fetch+ beta && git fetch origin` to surface drift on both remotes.+ 2. Commit, then `git push beta dev:main` whenever a logical change is+ done (lands on the PRIVATE beta).+ 3. **Stop there.** Tell Mr Bizarro it's on beta. Wait for explicit+ instruction to promote to public.+ 4. When Mr Bizarro says ship/push to main: `git push origin dev:main`,+ then tag + `gh release create` (see the release workflow in memory+ `phosphene_repo_split.md`).+ 5. Never force-push or rewrite public `main`. If histories have+ diverged, say so and ask before anything destructive.- The panel auto-detects which profile it is from `git rev-parse--abbrev-ref HEAD`: branch == `dev` → port 8199 + DEV badge; anythingelse → port 8198 + normal header. No config files to edit per panel.@@ -489,10 +506,13 @@ prompts. Below -25 dB peak suggests a regression.- Anything inside `panel_uploads/` — user uploads- `panel_queue.json`, `panel_hidden.json` — runtime state- `logs/`, `cache/`, `__pycache__/` — runtime caches-- The `.clinerules`, `.cursorrules`, `.windsurfrules`, `.geminiignore`,- `GEMINI.md`, `QWEN.md`, `AGENTS.md` files — these are generic Pinokio- scaffolding mirroring this `CLAUDE.md`. Edit `CLAUDE.md` first if a- project-wide change is needed; the rest are autogenerated mirrors.+- `GEMINI.md`, `QWEN.md`, `AGENTS.md` are **symlinks to this `CLAUDE.md`**+ (consolidated 2026-05-31 — they used to be hand-maintained near-duplicate+ copies that silently drifted). `CLAUDE.md` is the single source of truth;+ the other three exist only so Gemini / Qwen / Codex(AGENTS) load the same+ guide. Edit `CLAUDE.md`; never edit the symlinks. If a tool ever refuses a+ symlinked instruction file, replace that one with a 5-line stub pointing+ here — do not fork the full content back out.## 11. HTTP API reference (panel)diff --git a/GEMINI.md b/GEMINI.mddeleted file mode 100644index af1a878..0000000--- a/GEMINI.md+++ /dev/null@@ -1,1799 +0,0 @@-# Phosphene — agent operating manual--This file is the single source of truth for how an AI assistant should-work on this codebase. Read it first, every session, before touching-any file. The generic Pinokio scaffolding starts at "## Generic Pinokio-Development Guide" further down — that's the upstream template. This top-section overrides it where they conflict.-------## 0. Current state — Phosphene 2.0 (released 2026-05-04)--VERSION file is now `2.0.0`. Git tag `v2.0.0` points at commit `fca78bb`-on `main`. Going forward: `2.0.x` for patches, `2.1.0` for features,-`3.0.0` only for another major story like this one. Pre-2.0 commit-messages and inline comments still reference the `Y1.NNN` sequential-counter (Y1.001 → Y1.039). Treat those as historical; do not introduce-new `Y` labels.--### What ships in 2.0 (capabilities checklist)--- **Four modes**: T2V, I2V, FFLF (keyframe interpolation), Extend.-- **Joint audio + video** in a single LTX 2.3 forward pass — sound is- generated WITH the video. Lip-sync, footsteps, ambient noise all- align at the frame level. mlx 0.31.1 pin is what keeps the vocoder- output at healthy levels (-9 to -15 dB peak on dialogue prompts).-- **Quality tiers**: `quick` / `balanced` / `standard` / `high`. The- earlier `draft` key was renamed; do not reintroduce it. `high` =- Q8 two-stage HQ pipeline + TeaCache, requires Q8 download.-- **Speed modes** for Standard renders: `exact` (full sampler) /- `boost` (skip up to 2 stable middle steps via adaptive caching) /- `turbo` (skip up to 3). 23–34 % wall-time savings on stable scenes.- Disabled for High, Extend, and FFLF where step-skipping breaks- quality contracts.-- **Sharp upscale** = PiperSR (CoreML) on the Apple Neural Engine.- Real x2 detail recovery before the ffmpeg fit/encode tail; ~26 s- overhead on a 5 s × 1024×576 native. The earlier "model" upscale- path (LTX latent x2) was retired in Y1.026; it remains hidden- behind `LTX_ENABLE_MODEL_UPSCALE=1` for lab archaeology only.-- **VAE temporal streaming** — long clips no longer freeze at the- final decode. Patch in `patch_ltx_codec.py` defaults to- `LTX_VAE_STREAMING=auto`: streams when latent frames > 121 (≈ 5 s),- full-decodes shorter clips. Helper-side env-var override keeps- installs that haven't re-applied the patch on the right path too.-- **Filtered hf downloads** (Y1.024+): Q4 ≈ 20 GB, Q8 ≈ 37 GB,- Gemma ≈ 7.5 GB, optional spatial upscaler ≈ 1 GB. Older builds- pulled the full upstream repos and bloated to Q4=56 GB / Q8=82 GB- by including duplicate transformer variants and unused upscalers.- `update.js` deletes the dupes on existing installs.-- **Hardware-tier auto-config** (`base` / `standard` / `high` / `pro`)- with realistic per-tier time estimates surfaced on the Quality- pills. Compact-tier users see honest "~12 min" instead of M-Studio- numbers.-- **Now-card progress bar** (Y1.039) — phase-aware, denoise-step-aware.- Computed server-side from history bucket + helper log signals.- Replaces the old `elapsed / global_avg` ratio that mis-paced both- Quick and High renders.-- **Gallery race-free** (Y1.039) — `list_outputs()` skips files that- ffmpeg is still writing (in-flight skip + 2 s mtime cutoff), and- the `/file` URL carries an `&v=<mtime>` cache-bust so the browser- fetches the freshest bytes when a file is replaced.-- **CivitAI LoRA browser** built into the panel. `mlx_models/loras/`- is scanned at boot; sidecar JSON metadata travels with each safetensors.-- **Phosphene 2.0 release badge** — violet `2.0` chip next to the- wordmark in the panel header. Distinct from the amber `DEV` profile- badge so dev/prod tabs are distinguishable at a glance.--### Empirical benchmarks (M4 Max 64 GB, "Comfortable" tier)--All numbers are real wall-clock from sidecar `elapsed_sec` on Y1.039 /-v2.0.0 runs today. Output mp4 carries audio in every case.--| Config | Frames | Wall |-|---|---|---|-| T2V Quick · exact · 640×480 | 121 (5 s) | ~ 2 m 14 s |-| T2V Standard · exact · 1280×704 | 121 (5 s) | ~ 7 m 40 s |-| T2V Standard · boost · 1280×704 | 121 (5 s) | ~ 6 m 20 s |-| T2V Standard · turbo · 1280×704 | 121 (5 s) | ~ 5 m 26 s |-| T2V High Q8 · 1280×704 | 121 (5 s) | ~ 11 m 51 s |-| T2V Balanced · turbo · 720p Sharp | 121 (5 s) | **~ 3 m 30 s** |-| T2V Balanced · turbo · 720p Sharp | 241 (10 s) | **~ 8 m 07 s** |-| I2V Standard · exact · 1280×704 | 121 (5 s) | ~ 7 m 51 s |-| I2V Balanced · turbo · 720p Sharp | 121 (5 s) | ~ 3 m 37 s |-| I2V Balanced · turbo · 720p Sharp | 241 (10 s) | ~ 8 m 26 s |-| FFLF · clamped to 768 px (Comfortable cap) | 121 (5 s) | ~ 5 m 29 s |-| Extend · 768 px (Comfortable cap), +3 s | source 121 → output 184 | ~ 15 m 50 s |--The marquee number for the launch thread is the-**Balanced + Turbo + 720p Sharp** recipe: 5-sec clip in 3 m 30 s, 10-sec-in 8 m 07 s. That's the recommended default for "fast HD with audio".--### Repository layout (now — both clones documented)--`~/pinokio/api/phosphene.git/` is the **production** clone.-Tracks `main`. Pinokio's daily-driver Phosphene panel runs from here on-port 8198.--`~/pinokio/api/phosphene-dev.git/` is the **dev** clone.-Tracks `dev`. Set up via `scripts/setup_dev_panel.sh`. Runs on port 8199-with the amber `DEV` profile badge in the header. Symlinks-`mlx_models/`, `mlx_outputs/`, `panel_uploads/`, and `state/` from a-shared Pinokio fs.link drive so weights aren't duplicated and outputs-land in one gallery regardless of which panel rendered them.--When a session starts: `git checkout dev` in the dev clone, work, push.-Promote to main only when Mr Bizarro gives the explicit green light. See §7.-------## 1. What Phosphene is--A free desktop panel that wraps Lightricks' **LTX 2.3** model running-natively on Apple's **MLX** framework, exposed as a one-click install-through **Pinokio**. Generates video and audio jointly in a single-forward pass on Apple Silicon Macs. Apple-Silicon-only.--Public artifacts:-- GitHub: `github.com/mrbizarro/phosphene` (public, MIT for the panel)-- Pinokio Discover: searchable as "Phosphene"-- Twitter handle: `@AIBizarrothe`--The panel itself is MIT. Model weights have Lightricks' own license.--## 2. Where to work--**The Pinokio install dir IS the canonical workspace.** There is no-separate dev folder. Old "local dev" copy was deleted to consolidate.--```-~/pinokio/api/phosphene.git/ ← edit here-├── .git/ ← real working copy of mrbizarro/phosphene-├── ltx-2-mlx/ ← upstream package, cloned at install-│ ├── env/ ← Python 3.11 venv-│ └── packages/{ltx-core-mlx,ltx-pipelines-mlx}-├── mlx_models/ ← downloaded weights-│ ├── ltx-2.3-mlx-q4/ ← Q4 base (~20 GB, filtered Y1.024+)-│ ├── ltx-2.3-mlx-q8/ ← optional Q8 (~37 GB, filtered Y1.024+)-│ └── gemma-3-12b-it-4bit/ ← text encoder (~7.5 GB)-├── mlx_outputs/ ← generated mp4s-├── panel_uploads/ ← user-uploaded reference images-├── assets/ ← logos, banner, hero clip-├── launch/ ← launch post drafts (X, Reddit, Pinokio, CivitAI)-├── mlx_ltx_panel.py ← MAIN: HTTP server, HTML UI, queue-├── mlx_warm_helper.py ← subprocess holding MLX pipelines warm-├── pinokio.js / install.js / start.js / update.js / reset.js / download_q8.js-├── patch_ltx_codec.py ← applies patches to upstream package-└── required_files.json ← single source of truth for "installed"-```--To work: `cd ~/pinokio/api/phosphene.git/`, edit, `git commit`,-`git push`. Pinokio's Update button does `git pull` here. No separate-copy to keep in sync.--## 3. Architecture (request → frame)--```-Pinokio app (port 42000)- └── proxies to → mlx_ltx_panel.py (port 8198)- │- ├── serves a single-page HTML/JS UI from one Python process- ├── exposes HTTP API: /run, /status, /upload, /helper/restart, ...- ├── owns a job queue (FIFO) + worker thread- └── speaks JSON-over-stdin/stdout to:- │- └── mlx_warm_helper.py (subprocess in same venv)- ├── holds MLX pipeline objects warm across jobs- ├── pipeline classes from ltx-2-mlx package- ├── reaps itself after LTX_IDLE_TIMEOUT (1800s default)- ├── emits events: log / done / error / exit- └── on death → panel sees stdout EOF-```--Both processes run inside `ltx-2-mlx/env` (the Python 3.11 venv). Helper-restarts cheaply via `POST /helper/restart` to pick up code changes-without restarting the panel.--## 4. Locked dependency versions--These are not arbitrary — every pin is a paid lesson:--| Pin | Why |-|---|---|-| `python>=3.11` (force `uv venv --python 3.11`) | Pinokio's `venv:` shortcut defaults to conda's base 3.10. ltx-core-mlx requires `>=3.11`. |-| `mlx==0.31.1` | mlx 0.31.2 introduced a numerical regression that attenuates LTX 2.3 vocoder output by ~22 dB. Verified: same model + same prompt + same seed → -42 dB peak on 0.31.2 vs -9 dB peak on 0.31.1. |-| `mlx-lm==0.31.1` `mlx-metal==0.31.1` | Same release line, kept consistent with mlx. |-| `huggingface-hub>=1.0` | `hf` CLI replaced `huggingface-cli`; older Pinokio bundles ship < 1.0. |-| `ltx-2-mlx` at `main` (HEAD) | dcd639e (0.1.0) was tried as a pin during audio investigation but breaks the Extend `cfg_scale` API and forces a `transformer.safetensors` symlink hack. HEAD + the mlx pin is the right combination. |--When changing a version, document the test that proved the new pin is OK.--## 5. The patch system (`patch_ltx_codec.py`)--We patch the upstream `ltx-2-mlx` package after `pip install`. Patches-are idempotent and fail loud on upstream drift (don't silently ship a-broken pipeline). Each patch has an `OLD` text it expects to find and a-`NEW` text it replaces with, plus a `marker` to detect "already patched."--Currently shipped patches:--1. **Codec (`yuv444p crf 0` + `+faststart`)** — required.- Patches `ltx_core_mlx/model/video_vae/video_vae.py`. Default ffmpeg- args become lossless H.264 (yuv444p crf 0) and the moov atom moves- to the front of the file so gallery thumbnails render the first- frame without downloading the full clip.--2. **I2V free DiT before VAE decode** — optional (warns if upstream changed).- Patches `ImageToVideoPipeline.generate_and_save` in- `ltx_pipelines_mlx/ti2vid_one_stage.py`. Frees DiT + text encoder +- feature extractor before the VAE decode step.--3. **I2V free vae_encoder + feature_extractor BEFORE denoise** — optional.- Patches `generate_from_image` to null `self.vae_encoder` and- `self.feature_extractor` right after `_encode_text_and_load`- returns. The connector inside feature_extractor is ~5.91 GiB on Q4- and was sitting resident through denoise for no reason.--4. **Base `load()` also clears feature_extractor** — optional.- Patches `TextToVideoPipeline.load`. Drops one extra ~5.91 GiB blob- from the peak when the DiT loads.--5. **VAE temporal streaming with auto-skip on short clips** (Y1.035 +- Y1.037). Patches `VideoDecoder.decode_and_stream` in- `ltx_core_mlx/model/video_vae/video_vae.py`. Upstream's method- *advertised* streaming but actually called `self.decode(latent)` on- the full video volume before iterating frames — caused multi-minute- end-of-render stalls / jetsam on long 720p clips. The patch routes- through `tiled_decode()` with a `TemporalTilingConfig` (default- tile = 64 latent frames, overlap = 24). Default mode is `auto`:- tiling is applied only when latent length > `LTX_VAE_STREAMING_AUTO_MAX_FRAMES`- (default 121 video frames ≈ 5 s). Short clips skip tiling and gain- ~30 s on 5-sec Standard renders. `LTX_VAE_STREAMING=0` forces- full-decode fallback; `=1` forces always-stream. Helper-side- `_apply_vae_streaming_decision()` in `mlx_warm_helper.py` sets the- env var per-job based on `num_frames`, so even pre-2.0 patched- installs that haven't re-run `patch_ltx_codec.py` get the right- behavior.--The "upgrade" path in `apply_patch()` lets old patched installs receive-new patch versions without a venv rebuild — used when we added-`+faststart` to the existing codec patch.--## 6. Critical history (so we don't relitigate)--- **mlx 0.31.2 audio regression** (resolved): symptom was "audio sounds- like static / very quiet." Root cause was an MLX numerical change- attenuating the vocoder output by ~22 dB. Fix: `mlx==0.31.1` pin.- Bisecting which mlx commit caused it is a v1.1 task.-- **dcd639e pin detour** (resolved): briefly pinned `ltx-2-mlx` to- commit `dcd639e` (0.1.0) thinking the audio regression was in- dgrauet's commits. It wasn't. Pin caused Extend API mismatch- (`cfg_scale` kwarg unknown) and required a transformer.safetensors- symlink because the 0.1.0 loader hardcoded that filename. Reverted.-- **Pinokio menu API mismatch** (resolved): `info.path` is a function- on newer Pinokio, a string on older. `pinokio.js` now detects both- shapes via `getInstallRoot(info)`.-- **start.js URL hijack** (resolved): the `errno`/`error:` event- patterns were cargo-culted from comfy.git's start.js. They captured- any "Errno" line from the helper into `input.event[0]` and the panel- URL got set to the literal string "Errno." Removed those patterns.-- **silent helper death** (mitigated, not fixed): on memory-pressured- Macs the helper subprocess is SIGKILL'd by macOS jetsam during I2V- pipeline load. Patches 3 and 4 reduced peak; the panel now names- the exit signal (SIGKILL/SIGSEGV/SIGABRT/SIGBUS) and points users at- `~/Library/Logs/DiagnosticReports/python3.11_*.crash` for diagnosis.-- **HTML5 form validation silent block** (resolved): a hidden Extend- input had `value=2 min=0.4 step=0.5`. Chrome's validator only accepts- values on the (min + N*step) sequence, so 2 was off-grid → invalid- → entire form silently failed to submit. Fixed `min=0.5`.--## 7. Working rules — non-negotiable--### Identity-- **Author/committer:** `mrbizarro <[email protected]>`.- Already set in this repo's local git config. Never override.-- **Never use the user's real first name** in commits, code, file- contents, or any artifact that lands on disk or in git. Same for- any third party (reviewers, collaborators) — refer to them by- technical role only ("upstream review", "external reviewer", "user- feedback") if at all.-- **Never add `Co-Authored-By:` trailers.** Phosphene reads as a- serious project shipped by `mrbizarro`. Co-author trailers showing- "Claude" make it look like a vibe-coded toy.--### Branch model — `main` is production, `dev` is staging (Y1.015+)--**TL;DR for agents — non-negotiable:**--> **Push every commit to `dev`. Never push to `main` unless Mr Bizarro-> explicitly says "push to main" / "ship to production" / "merge to-> main".** "It works on my machine" is not permission. "The tests-> pass" is not permission. Mr Bizarro has a Phosphene Dev panel in his-> Pinokio that pulls from `dev`; he tests there, then tells you when-> to promote.--The why:--- **`main`** is what real users get. Pinokio's Discover URL points- here; every commit pushed to `main` is shipped to every user the- next time they click Update. Treat it as production.-- **`dev`** is the staging branch. Push WIP commits here freely. The- local "Phosphene Dev" Pinokio panel (set up via- `scripts/setup_dev_panel.sh`) tracks this branch on port 8199 and- shows an amber DEV badge in the header.-- **Workflow**:- 1. Default to working on `dev`. `git checkout dev` at the start of- any session that touches Phosphene; if you're on `main`, switch.- 2. Commit + push to `origin/dev` whenever a logical change is done.- 3. **Stop there.** Tell Mr Bizarro it's on dev. Wait for explicit- instruction to promote.- 4. When Mr Bizarro says push to main: `git checkout main && git merge dev- --ff-only && git push origin main`. Then `git checkout dev` so- subsequent work continues on dev.- 5. If a fast-forward isn't possible (rare — usually a hotfix landed- on main directly), say so and ask before doing anything destructive.-- The panel auto-detects which profile it is from `git rev-parse- --abbrev-ref HEAD`: branch == `dev` → port 8199 + DEV badge; anything- else → port 8198 + normal header. No config files to edit per panel.-- `update.js` is branch-aware as of Y1.015 (`git pull --ff-only origin- <current branch>`) so the dev panel's Update pulls dev and the- production panel's Update pulls main, from the same shared code.-- The dev panel symlinks `mlx_models/` from the production install- (saves ~36 GB of duplicate weights). State / outputs / uploads stay- separate per panel.-- Hotfixes bypassing dev — only allowed when Mr Bizarro explicitly authorises- ("hotfix straight to main" or similar). Otherwise, even one-line- fixes go via dev → tested in Phosphene Dev panel → merged.--### Git policy — NEVER FORCE-PUSH `origin/main`-- A history-rewrite + force-push on 2026-05-01 (to scrub identity- leaks from old commits) broke `git pull` for every existing user- whose local clone contained orphaned commits — Pinokio's Update- silently stalled for them. GitHub aggressively GC'd the orphaned- objects within hours, so the pre-rewrite history is unrecoverable.-- **Do not run `git filter-branch`, `git rebase` (on shared history),- `git push --force`, `git push --force-with-lease`, or anything else- that rewrites already-published commits on `origin/main`. Ever.**-- If a commit needs amending and it's already pushed, **make a new- commit that reverts/fixes it.** The history is append-only.-- If the user requests a history rewrite (e.g. "scrub my name"), say- no and explain why: existing users get orphaned, their `git pull`- fails, the entire user base needs out-of-band recovery.-- Recovery from the 2026-05-01 incident — primary path is **Pinokio- Reset → Install** (clean, no shell scripts, models re-download is a- one-time cost). The `recover.sh` script is an escape hatch for users- who want to keep their downloaded models — kept in the repo with a- clear "read before running" header, NOT promoted as a `curl | bash`- command. Don't remove `recover.sh` or weaken `update.js`'s- fetch+reset fallback.--### `fs.link` — orthogonal to the history-divergence story-- `fs.link` is in `install.js` only (NOT `update.js`). It maps the- heavy/personal directories — `mlx_models/`, `mlx_outputs/`,- `panel_uploads/`, `state/` — to a Pinokio virtual drive that lives- outside the panel install dir. After a Pinokio Reset → Install, the- drive is rediscovered and the symlinks reattach, so models +- outputs + settings survive a clean reinstall.-- Why install.js only: future users get the drive on first install;- existing pre-Y1.004 users get it whenever they next reinstall (which- is the recommended recovery path anyway). Adding `fs.link` to- update.js would technically migrate existing real folders into the- drive without a Reset, but it conflates routine code updates with a- one-time 36 GB merge step, so we keep them separate.-- `fs.link` is unrelated to the 2026-05-01 history-rewrite incident- — don't conflate them in commit messages or docs.--### Commit message style-- Subject line in imperative mood, 50–72 chars max, no trailing period.- Use `feat:` / `fix:` / `refactor:` / `docs:` / `chore:` / `perf:` /- `test:` prefixes when they help.-- Body explains the **why**: the symptom, the root cause, the chosen- fix. The diff already shows the what.-- No first-person voice. Third person, neutral, technical.-- No conversation transcript. No "user said X", "after we discussed Y",- "per the review on Tuesday." Just describe the change.-- Cite concrete things: file paths, line numbers, package versions,- upstream commits, signal names. Avoid vague nouns ("the issue").-- Reference upstream context where it helps future maintainers (HF- metadata size, upstream commit SHA, GitHub issue number).-- One commit = one logical change. Don't bundle UI polish with bug- fixes with version pins.--### Examples — bad vs good--❌ Bad:-```-fix the audio thing peanut found--cocktailpeanut said the audio sounded like static so we tried a few-things and pinning mlx fixed it. great catch!-```--✅ Good:-```-fix(audio): pin mlx==0.31.1 to recover LTX 2.3 vocoder amplitude--mlx 0.31.2 introduces a numerical change that attenuates the LTX 2.3-vocoder output by ~22 dB. Verified empirically with the same prompt,-seed, and weights:- mlx 0.31.2 → max_volume -42.8 dB (audible only with significant boost)- mlx 0.31.1 → max_volume -9.2 dB--install.js installs the pinned trio (mlx, mlx-lm, mlx-metal) before-the ltx-* packages so transitive resolution does not pull latest.-update.js force-reinstalls them with --no-deps so existing installs-that already pulled 0.31.2 can recover by clicking Update.-```--### Code style-- Comments explain non-obvious decisions and trade-offs. They earn- their keep by explaining context the diff cannot.-- When fixing a bug, leave a comment with the symptom + cause + fix- near the change so the next reader doesn't reintroduce it.-- Patches in `patch_ltx_codec.py` always include a comment block- above each patch explaining what the upstream code does, why we- patch it, and what the patch achieves.--### Working with the running install-- The Pinokio panel may already be running on port 8198 when you- start a session. **Never spawn your own panel from the command- line** — Pinokio expects to own that port and will fail Start- with `Address already in use` if you do.-- To pick up code changes without restarting Pinokio, hit- `POST /helper/restart` — that reloads the helper subprocess- against the patched site-packages.-- For HTML/JS/CSS changes you do need a panel restart. Either- click Stop + Start in Pinokio, or `pkill -f mlx_ltx_panel.py`- and click Start.--### Disk consolidation-- Old `~/Documents/Codex/...` dev folder was deleted.- Don't recreate. The Pinokio install is canonical.-- `~/.cache/huggingface/hub/models--*` was also deleted. Pinokio- install pulls weights into `mlx_models/` directly via `hf- download --local-dir`, not the cache. Don't reintroduce HF cache- symlinks.--## 8. Test workflow--For HTML/UI/JS changes:-1. Edit `mlx_ltx_panel.py`-2. Restart panel (Pinokio Stop + Start, OR `pkill -f mlx_ltx_panel.py`- then click Start in Pinokio)-3. Hard-refresh browser (Cmd+Shift+R)-4. Verify the served HTML contains the change:- `python3 -c "import urllib.request; print('your-marker' in urllib.request.urlopen('http://127.0.0.1:8198/').read().decode())"`--For helper changes (Python pipeline, patches):-1. Edit `mlx_warm_helper.py` or `patch_ltx_codec.py`-2. If touching patches: re-run `./ltx-2-mlx/env/bin/python3.11 patch_ltx_codec.py`-3. `curl -X POST http://127.0.0.1:8198/helper/restart`-4. Submit a test job via `curl -X POST http://127.0.0.1:8198/run -d "mode=t2v" -d "prompt=..." -d "width=512" -d "height=288" -d "frames=49" -d "steps=8" -d "quality=draft"`-5. Poll `/status` until `current` is null and verify the new- `history[0]` entry is `done`.--For audio testing — do not use generic "wizard in forest" prompts.-Use prompts with explicit dialogue or sound events so artifacts are-audible:--```-A woman in her 30s with auburn hair, sitting at a wooden table in a-softly lit kitchen, speaks directly to the camera with a small smile-and clear, confident voice. She says "I really did not think I would-see you again, but here we are." Audio: clear human voice, slight-room reverb, gentle ambient kitchen sounds.-```--Measure with ffmpeg:-```-ffmpeg -i out.mp4 -af "volumedetect" -f null - 2>&1 | grep -E "max_volume|mean_volume"-```-Healthy levels: `max_volume` between -2 dB and -15 dB on dialogue-prompts. Below -25 dB peak suggests a regression.--## 9. Pinokio lifecycle scripts--| Script | Trigger | What it does |-|---|---|---|-| `install.js` | First Install or Resume Install | Clone `ltx-2-mlx`, force Python 3.11 venv, install MLX trio at pinned versions, install ltx-* packages, run `patch_ltx_codec.py`, `hf download` Q4 + Gemma. Idempotent. |-| `start.js` | Start button | Spawn panel inside the venv with `LTX_*` env vars set. Capture URL via regex with capture group, route to `local.set` via `input.event[1]`. |-| `update.js` | Update button | `git pull` panel + ltx-2-mlx, force-reinstall mlx pin, force-reinstall ltx-* packages, re-run patches. |-| `reset.js` | Reset button | Wipe venv + clones. **Preserves** `mlx_outputs/`, `panel_uploads/`, `mlx_models/` (those are user content / huge weight files). |-| `download_q8.js` | Download Q8 button | `hf download dgrauet/ltx-2.3-mlx-q8` to `mlx_models/ltx-2.3-mlx-q8`. |-| `pinokio.js` | Always | Renders the menu. State driven by `required_files.json` (env marker + per-repo file completeness). |--## 10. Files to never touch unprompted--- Anything inside `mlx_models/` — these are downloaded weights, GB-scale-- Anything inside `mlx_outputs/` — user-generated content-- Anything inside `panel_uploads/` — user uploads-- `panel_queue.json`, `panel_hidden.json` — runtime state-- `logs/`, `cache/`, `__pycache__/` — runtime caches-- The `.clinerules`, `.cursorrules`, `.windsurfrules`, `.geminiignore`,- `GEMINI.md`, `QWEN.md`, `AGENTS.md` files — these are generic Pinokio- scaffolding mirroring this `CLAUDE.md`. Edit `CLAUDE.md` first if a- project-wide change is needed; the rest are autogenerated mirrors.--## 11. HTTP API reference (panel)--The panel listens on `127.0.0.1:8198` and serves a single-page HTML/JS-UI plus a JSON API. Pinokio proxies its own port 42000 to 8198. All-endpoints are unauthenticated (loopback only).--### GET endpoints--| Path | Returns | Notes |-|---|---|---|-| `/` | HTML | The single-page UI. Built from one big template string in `mlx_ltx_panel.py`. |-| `/status` | JSON | Snapshot of `STATE` — running flag, current job, queue, history (top 20), log tail, tier, memory pressure, helper alive flag, comfy PIDs, repo download state. Polled by the UI every ~1s. |-| `/uploads` | JSON | Recent files in `panel_uploads/` for the picker's "click to reuse" strip. |-| `/models` | JSON | Per-repo install completeness — driven by `required_files.json`. |-| `/settings` | JSON | Returns `{settings, presets, default_preset}`. `settings` is the current `panel_settings.json` content. `presets` is the OUTPUT_PRESETS table (label + blurb + pix_fmt + crf per preset) so the UI doesn't duplicate it. |-| `/loras` | JSON | Returns `{user, curated, loras_dir, civitai_auth}`. `user` is the scanned `mlx_models/loras/*.safetensors` list (each with sidecar metadata if present). `curated` is the curated Lightricks repo registry minus the HDR-toggle entry (which is exposed as a plain checkbox, not in the picker). `civitai_auth` is True iff `CIVITAI_API_KEY` is set in the panel's env. |-| `/civitai/search` | JSON | Proxies CivitAI's `/api/v1/models` filtered to `types=LORA&baseModels=LTXV 2.3`. Query params: `query`, `nsfw` (bool), `cursor` (CivitAI uses cursor pagination), `limit`. Returns `{items, next_cursor, has_more}` with each item carrying `{id, version_id, name, creator, description, downloads, nsfw, preview_url, filename, size_kb, download_url, trigger_words, base_model, civitai_url}`. |-| `/file?path=…` | bytes (mp4/png/jpg) | Range-aware video serve so `<video>` tags can seek without re-downloading. |-| `/image?path=…` | bytes (image) | Same idea for image previews. |-| `/sidecar?path=…` | JSON | Per-output sidecar (job params, output stats) if it exists. |-| `/assets/<file>` | bytes | Static files from `assets/`. |--### POST endpoints--| Path | Body | Returns | Effect |-|---|---|---|---|-| `/run` `/queue/add` | urlencoded job spec | `{ok, id}` | Creates a job, appends to queue. Field reference below. |-| `/queue/batch` | urlencoded with `prompts` (newline-separated, `---` chunks) | `{ok, added, ids}` | Splits the prompts on `^\s*---\s*$` boundaries, creates one job per chunk. |-| `/queue/remove` | `id=<jid>` | `{removed: bool}` | Drop a queued job by id. |-| `/queue/clear` | (none) | `{cleared: int}` | Clear all queued jobs (running job continues). |-| `/queue/pause` | (none) | `{paused: true}` | Worker stops popping new jobs after the current one. |-| `/queue/resume` | (none) | `{paused: false}` | Worker resumes. |-| `/output/hide` | `path=<p>` | `{hidden: path}` | Adds path to the gallery's hidden set (persisted). |-| `/output/show` | `path=<p>` | `{shown: path}` | Inverse. |-| `/output/show_all` | (none) | `{unhidden_count: int}` | Clear hidden set. |-| `/upload` | multipart `image=<file>` | `{ok, path}` | Saves to `panel_uploads/<ts>_<safename>`. |-| `/helper/restart` | (none) | `{ok}` | SIGTERM the helper subprocess; the next job auto-respawns it. Useful for picking up site-packages changes. |-| `/settings` | `output_preset` / `output_pix_fmt` / `output_crf` / `civitai_api_key` / `hf_token` (any subset) | `{ok, settings, helper_restarted}` | Update + persist. Public-safe `settings` view in the response (no token values, just `has_*` booleans). Helper killed on codec change OR token change so the next job spawns with fresh env. Form parser uses `keep_blank_values=True` so `civitai_api_key=` (the Clear button) is treated as "remove this token" rather than dropped silently. Empty input on the front-end Apply path is "leave as-is" — explicit clearing goes through the dedicated Clear button. |-| `/loras/refresh` | (none) | `{ok, user, loras_dir}` | Rescan `mlx_models/loras/`. Filesystem is the source of truth; no caching layer to invalidate. |-| `/loras/delete` | `path=<abs path>` | `{ok, removed}` | Delete a user-installed LoRA + its sidecar. Path must resolve inside `mlx_models/loras/` to prevent traversal. |-| `/civitai/download` | `download_url=<civitai url>`, `meta=<json>` | `{ok, name, path, sidecar_path, size_bytes, skipped}` | Stream-download a LoRA from CivitAI into `mlx_models/loras/`, write a sidecar JSON. Refuses non-civitai.com hosts. Writes to a `.partial` then atomic-renames so a kill mid-write leaves nothing the next scan would mistake for a complete file. Surfaces 401 with a remediation hint pointing at `CIVITAI_API_KEY`. |-| `/stop` | (none) | `{ok}` | Cancel the current job (kills helper + mux). Worker advances. |-| `/stop_comfy` | (none) | `{ok, killed}` | SIGTERM any ComfyUI process matching `LTX_COMFY_PATTERN`. |-| `/open_pinokio` | (none) | `{ok}` | macOS-only: focus the Pinokio app. |-| `/models/download` | `key=q4|q8|gemma` | `{ok}` | Spawn `hf download` for the named repo. Streams to log. |-| `/models/cancel` | (none) | `{ok, killed}` | SIGTERM the active download. |-| `/prompt/enhance` | `prompt=<p>` | `{ok, original, enhanced}` | Calls helper's Gemma-rewrite path. |--## 12. Job spec — the contract for `/run` and `/queue/add`--Every form field below maps to `params.<field>` on the resulting job.-`make_job()` in `mlx_ltx_panel.py:1140` is the canonical reference.--| Field | Type | Default | Meaning |-|---|---|---|---|-| `mode` | str | `"t2v"` | One of: `t2v`, `i2v`, `i2v_clean_audio`, `extend`, `keyframe`. |-| `prompt` | str | `"A cinematic atmospheric scene"` | Empty falls back to the default. |-| `width` | int | `1280` | Clamped to `>= 32`. T2V only — image flows derive from aspect+quality. |-| `height` | int | `704` | Same as width. |-| `frames` | int | `121` | Number of video frames. The 8k+1 rule (latents are groups of 8). |-| `steps` | int | `8` | Denoising steps. T2V/I2V Q4 distilled requires `>= 8` (schedule is locked). |-| `seed` | int-as-str | `"-1"` | `"-1"` → random. Stored as string so the form roundtrip is lossless. |-| `image` | path | `REFERENCE` | Only used when `mode != "t2v"`. |-| `audio` | path | `AUDIO_DEFAULT` | Only used when `mode == "i2v_clean_audio"` (mux external audio). |-| `video_path` | path | `""` | **Required for `mode=extend`**. Source mp4 to extend. |-| `extend_frames` | int | `5` | Latent-frame count to add. Each latent = 8 video frames. So 5 latents = 40 frames ≈ 1.67s at 24 fps. |-| `extend_direction` | str | `"after"` | `"after"` or `"before"`. |-| `extend_steps` | int | `12` | Denoising steps for the extend pipeline (uses dev transformer). |-| `extend_cfg` | float | `1.0` | CFG scale. 1.0 = no guidance (faster, less RAM). 3.0 = guided (Quality mode). |-| `start_image` | path | `""` | **Required for `mode=keyframe`**. First frame. |-| `end_image` | path | `""` | **Required for `mode=keyframe`**. Last frame. |-| `enhance` | "on"/"off" | `"off"` | Reserved (panel uses the explicit `/prompt/enhance` button instead). |-| `stop_comfy` | "on"/"off" | `"off"` | Kill ComfyUI before this render to free MPS memory. |-| `open_when_done` | "on"/"off" | `"off"` | macOS `open` the output when finished. |-| `quality` | str | `"balanced"` | `"quick"`, `"balanced"`, `"standard"`, or `"high"`. Drives dimensions, step counts, and pipeline. |-| `accel` | str | `"off"` | `"off"` (alias for exact) / `"boost"` / `"turbo"`. Adaptive-cache denoise; boost skips up to 2 cached steps, turbo up to 3. T2V/I2V Standard only — disabled for High, Extend, FFLF. |-| `upscale` | str | `"off"` | `"off"` / `"fit_720p"` / `"x2"`. Post-render export. Balanced defaults to `fit_720p`. |-| `upscale_method` | str | `"lanczos"` | `"lanczos"` (ffmpeg) or `"pipersr"` (Sharp, ANE-accelerated). PiperSR adds ~26 s on 5-sec 1024×576. |-| `no_music` | "on"/"off" | `"off"` | When on, the panel appends a no-music audio constraint to the prompt at submit time. |-| `preset_label` | str | `""` | Optional human-readable label for the queue/history rows. |--## 13. Mode → pipeline → transformer routing--| Mode | Pipeline class | Transformer | Quality flag | Notes |-|---|---|---|---|---|-| `t2v` | `TextToVideoPipeline` | distilled Q4 | quick → smaller dims; balanced → 1024×576 + 720p export; standard → 1280×704; high → `TwoStageHQPipeline` w/ Q8 dev | Most common path. |-| `i2v` | `ImageToVideoPipeline` | distilled Q4 | same tier ladder as T2V | Adds VAE-encoder pass on the input image. |-| `i2v_clean_audio` | `ImageToVideoPipeline` | distilled Q4 | same | Generates LTX video, then panel-side ffmpeg mux replaces the LTX audio with `audio` param. |-| `extend` | `ExtendPipeline` | **Q8 dev** (Y1.036+) | always uses Q8 `transformer-dev.safetensors`; cfg_scale dial via `extend_cfg` | Heavier (~75–80 s/step on 768×416 cfg=1.0). Pre-Y1.024 a duplicate `transformer-dev.safetensors` shipped in the Q4 dir as download bloat — Extend silently loaded from there. Y1.024 download filter pruned the dupe and exposed that Extend is structurally Q8-class. Panel auto-routes to Q8 dir; UI gates on Q8 download (same shape as FFLF). |-| `keyframe` (FFLF) | `KeyframePipeline` | **Q8 dev + distilled LoRA** | requires Q8 download | Disabled on `base` tier. Comfortable tier clamps to 768 max-side (the stage-1 → stage-2 transition OOMs at 1280×704 on 64 GB). |--The helper `get_pipe(kind)` lazily loads the pipeline class corresponding-to the mode. Switching modes between jobs frees the previous pipeline-(`release_pipelines(keep_kind=...)`) — strict one-pipeline-at-a-time-policy keeps memory bounded.--## 14. Quality tier mapping--`quality` is independent of the hardware tier. Hardware tier *clamps*-the maximum dimensions per mode; `quality` picks the pipeline + step-count *within* that clamp.--| `quality` | Pipeline used | Dimensions | Steps | Approx wall on M4 Max 64 GB |-|---|---|---|---|---|-| `quick` | distilled Q4 | 640×480 (4:3 fast sanity) | 8 | ~ 2 m 14 s @ 121 frames |-| `balanced` | distilled Q4 + ffmpeg or PiperSR fit_720p export | 1024×576 → 1280×720 export | 8 | ~ 4 m 36 s (Lanczos) / ~ 5 m 02 s (Sharp) @ 121 frames |-| `standard` | distilled Q4 | 1280×704 (full target, clamped by tier) | 8 | ~ 7 m 40 s @ 121 frames; ~ 5 m 26 s with `accel=turbo` |-| `high` | `TwoStageHQPipeline` (Q8 dev + res_2s sampler + TeaCache) | 1280×704 | 15 stage-1 + 3 stage-2 | ~ 11 m 51 s @ 121 frames |--`high` and `keyframe` require the Q8 model on disk (~37 GB filtered).-`extend` also requires Q8 since Y1.036 (transformer-dev). Panel surfaces-a "Download Q8" CTA + disables the Generate button when the user picks-a Q8-required mode without Q8 installed.--Independent of `quality`, T2V/I2V Standard takes an `accel` flag-(`exact` / `boost` / `turbo`) that drives an adaptive-cache denoise-loop. Boost can skip up to 2 stable middle steps, Turbo up to 3.-Cached steps near-zero wall; full steps stay ~53 s/it on 1280×704.-Per-step decisions are recorded in the sidecar's `accel_metrics` block.--## 15. Hardware tier system--`SYSTEM_TIER` is detected once at boot from `sysctl hw.memsize`. The-override `LTX_TIER_OVERRIDE=base|standard|high|pro` forces a tier-(useful for testing what other users see). `CAPABILITIES` in-`mlx_ltx_panel.py:584` is the canonical table; abbreviated:--| Tier | RAM range | Friendly | t2v/i2v max dim | Q8 | FFLF | Extend |-|---|---|---|---|---|---|---|-| `base` | < 48 GB | Compact | 768 | ❌ | ❌ | ❌ |-| `standard` | 48–79 GB | Comfortable | 1280 | ✅ | ✅ | ✅ |-| `high` | 80–119 GB | Roomy | 1280 | ✅ | ✅ (more frames) | ✅ |-| `pro` | ≥ 120 GB | Studio | 0 (no clamp) | ✅ | ✅ | ✅ |--The thresholds are intentional. M-Studio 64 GB is the canonical-"standard" hardware. Anything below 48 GB cannot fit Q8 weights-(transformer-dev.safetensors is 19.18 GiB plus working tensors).--## 15a. `panel_settings.json` schema--User-controllable preferences. Persisted at `<repo>/state/panel_settings.json`-(Y1.004+); pre-Y1.004 lived at `<repo>/panel_settings.json` and is migrated-on first boot by `_migrate_state_dir()`. State directory is gitignored.--The whole `state/` dir is symlinked into Pinokio's fs.link drive so it-survives Reset → Reinstall (see "Persistent storage / fs.link" in install.js-+ update.js). All persisted-state files (`panel_settings.json`,-`panel_queue.json`, `panel_hidden.json`) live there.--Read at panel startup; mutated via `POST /settings`. The helper subprocess-inherits codec env vars from this at spawn — when settings change, the-panel kills the helper so the next job picks up new values.--```jsonc-{- "version": 2, // bumped from v1 when secrets added- // Metadata so the UI knows which preset to highlight as active.- // "custom" means pix_fmt + crf were set manually outside any preset.- "output_preset": "standard" | "archival" | "web" | "custom",- "output_pix_fmt": "yuv420p" | "yuv444p" | "yuv422p"- | "yuv420p10le" | "yuv422p10le" | "yuv444p10le",- "output_crf": "0".."30", // string (passed via env vars)-- // Secrets. Stored locally only, never sent over the network except- // as Authorization headers / ?token= URL params to the upstream- // services (civitai.com, huggingface.co). NEVER returned by GET- // /settings — the public response surfaces booleans only.- "civitai_api_key": "", // empty = not configured- "hf_token": "" // empty = not configured-}-```--### `GET /settings` security contract--The full settings dict — including `civitai_api_key` and `hf_token` —-is read by the backend only. The HTTP API exposes a *masked* view via-`get_settings_public()`:--```jsonc-{- "version": 2,- "output_preset": "standard",- "output_pix_fmt": "yuv420p",- "output_crf": "18",- "has_civitai_key": false, // bool, never the value- "has_hf_token": false // bool, never the value-}-```--This applies on every response that includes settings — success path,-validation errors, helper-restart confirmations. A user with the panel-proxied through ngrok / tunneled to the LAN never leaks their saved-secrets through a status poll.--### Token resolution priority--`_active_civitai_key()` and `_active_hf_token()` resolve the active-token in this order:--1. The saved value in `panel_settings.json` (set via the Settings- modal). This is the path normal users take.-2. The corresponding env var (`CIVITAI_API_KEY` / `HF_TOKEN`). Power- users with shell-level config keep working unchanged.--Empty string in #1 falls through to #2. The settings UI's "Clear"-button writes an empty string to #1, so a user who configured a key-in the UI and then wants to revert to env-var-based config can do-so cleanly.--For HF specifically, `huggingface_hub.snapshot_download` *also* reads-`~/.cache/huggingface/token` (the file `hf auth login` writes). We-don't override that file — if neither settings nor `HF_TOKEN` env is-set, the library falls back to that cached file. This means power-users who already ran `hf auth login` get gated LoRAs working without-touching the panel's token settings.--Preset table is the single source of truth, defined in `OUTPUT_PRESETS`-in `mlx_ltx_panel.py`:--| key | pix_fmt | crf | Approx 5s @ 1280×704 | Use case |-|---|---|---|---|---|-| `standard` ⭐ default | yuv420p | 18 | ~7 MB | Visually lossless, plays everywhere (X / IG / Discord) |-| `archival` (UI label "Video production") | yuv444p | 0 | ~50 MB | Mathematically lossless, full 4:4:4 chroma. For color grading, compositing, VFX, anywhere downstream re-encode is expected. Internal key kept as `archival` for backwards compat with settings files saved before the rename. |-| `web` | yuv420p | 23 | ~3 MB | Smallest, mobile / embedding |--`+faststart` is always applied (codec patch) regardless of preset, so-gallery thumbnails always render the first frame instantly.--When adding a new preset: edit `OUTPUT_PRESETS` in `mlx_ltx_panel.py`-and the `order` array in the settings modal JS (`openSettingsModal`).-The blurb shown in the UI is read from the server at modal-open time,-so server is the source of truth — no JS strings to keep in sync.--## 15b. LoRA system--LTX 2.3 supports LoRA adapters via the upstream `apply_loras()` function-in `ltx_core_mlx.loader.fuse_loras`. The fusion is **weight-level**: at-pipeline load time, each LoRA's `(lora_A, lora_B)` matrix pair is-multiplied + added to the corresponding base-model weight, modified in-place, then quantized. This is *not* a runtime adapter — switching-LoRAs requires reloading the pipeline.--### Integration hook--The base `TextToVideoPipeline.load()` checks-`getattr(self, "_pending_loras", None)` and, if present, calls-`_fuse_pending_loras(transformer_weights, lora_paths)` before-quantizing the transformer. Set `pipe._pending_loras = [(path, strength), ...]`-between construction and the first job.--The wrapper in `mlx_warm_helper.py:_attach_loras` does this. It also-resolves HuggingFace repo IDs (`Lightricks/LTX-2.3-22b-IC-LoRA-HDR`)-to local files via `snapshot_download`, picking the largest-`.safetensors` in the repo (LoRA repos sometimes ship auxiliary files-like scene embeddings alongside the main weights).--### Pipeline cache invalidation--`get_pipe(kind, loras=...)` keys the cache on `(kind, frozen LoRA set)`.-Changing the LoRA set forces a pipeline rebuild — the cached pipeline's-weights have the previous LoRA set fused in and there's no clean way-to "unfuse." Cost is the standard ~30s pipeline reload.--The frozen LoRA set is order-insensitive (set hash, not list hash) —-fusion is commutative so `[A, B]` and `[B, A]` produce identical-weights and should hit the same cache entry.--### `mlx_models/loras/` directory--User-installed LoRAs live as `<name>.safetensors` files at the install-root's `mlx_models/loras/` directory. An optional sidecar JSON next to-each `.safetensors` carries display metadata:--```jsonc-{- "name": "Crisp Enhance",- "description": "Sharper detail and edge enhancement.",- "trigger_words": [], // optional, prepended to prompt at submit- "recommended_strength": 0.8, // 0.0..1.5 typical, displayed as default in slider- "preview_url": "https://...", // optional, shown in the picker- "base_model": "LTXV 2.3",- "civitai_id": 2530917, // optional, for attribution- "civitai_version_id": 2844417,- "civitai_url": "https://civitai.com/models/2530917",- "downloaded_at": "2026-05-01T..."-}-```--Sidecar absent → panel falls back to the filename as the display name-and zero metadata. Sidecar shape matches the JSON `_civitai_download`-writes, so a CivitAI install lands a usable picker entry without any-hand-editing.--### Curated LoRA registry--`CURATED_LORAS` in `mlx_ltx_panel.py` lists Lightricks' official LoRAs-that we know about. Currently:--| key | repo | UI surface | Note |-|---|---|---|---|-| `hdr` | `Lightricks/LTX-2.3-22b-IC-LoRA-HDR` | "HDR" toggle in form | Hidden from picker (`is_hdr_toggle: True`); exposed as a plain checkbox |-| `motion-track` | `Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control` | Listed in picker | IC-LoRA, needs video conditioning (not yet wired in panel UI) |-| `union-control` | `Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control` | Listed in picker | Same |--All three are **gated** on Hugging Face — users must accept the license-on the model page AND have a valid HF token (`hf auth login`) before-the helper's `snapshot_download` will work. The helper translates the-upstream `GatedRepoError` into a clear actionable error.--### Job spec extension--The `generate` and `extend` actions accept an optional `loras` param:--```jsonc-{- "action": "generate",- "id": "...",- "params": {- "mode": "t2v",- "prompt": "...",- "loras": [- {"path": "/abs/path/to/style.safetensors", "strength": 0.8},- {"path": "Lightricks/LTX-2.3-22b-IC-LoRA-HDR", "strength": 1.0}- ],- ...- }-}-```--Path may be a local file or an HF repo id. Strength is clamped to-±2.0 by the panel before sending; values outside the typical 0.0..1.5-range are usually nonsense and risk numerical issues during fusion.--### HDR toggle semantics--The `hdr` boolean on the job (separate from `loras`) is a UI shortcut.-Worker-side, `run_job_inner` resolves it to-`{"path": "Lightricks/LTX-2.3-22b-IC-LoRA-HDR", "strength": 1.0}`-and appends to the LoRA list before calling the helper. Users see-"HDR" as a feature; the LoRA implementation is hidden.--### CivitAI bridge--CivitAI's REST API is the de-facto LoRA distribution hub for LTX 2.3.-Two endpoints proxy through the panel:--- `/civitai/search` filtered to `baseModels=LTXV 2.3` (the canonical- string CivitAI uses for LTX-2.3 LoRAs as of 2026-05). Cursor-paginated.-- `/civitai/download` streams a `.safetensors` into `mlx_models/loras/`- + writes a sidecar JSON.--Both endpoints accept an optional `CIVITAI_API_KEY` env var. As of 2025,-CivitAI requires a token for most LoRA downloads (search still works-without). The token goes in `?token=<key>` on the URL — many CDN hops-only honor that form, not the Authorization header.--### Adding a new curated LoRA--1. Edit `CURATED_LORAS` in `mlx_ltx_panel.py` with `id`, `name`,- `description`, `repo_id`, `default_strength`, and either- `is_hdr_toggle: True` (= surfaces as a header pill, hidden from- picker) or `False` (= shows in picker as a usable entry).-2. If `is_hdr_toggle`, also add the HTML toggle pill (next to "No- music"), the JS sync block, and the `make_job` resolution that maps- the boolean to the curated repo id.--## 16. `required_files.json` schema--Single source of truth for "what counts as installed." Read by:-- `pinokio.js` → menu rendering (Install vs Resume Install vs Start)-- `mlx_ltx_panel.py` → `/status` and per-job validation-- (indirectly) `install.js` and `download_q8.js` — their `hf download`- paths must match the `local_dir` declared here.--```jsonc-{- "_comment": "...",- "repos": [- {- "key": "q4" | "gemma" | "q8",- "kind": "base" | "optional", // "base" = blocks Start, "optional" = unlock features- "name": "human-readable label",- "blurb": "what this is for",- "repo_id": "<hf-org/repo>", // for `hf download`- "local_dir": "mlx_models/<repo-name>", // relative to install root- "size_gb": 25, // approximate, displayed in UI- "files": [ // every required file inside local_dir- "transformer-distilled.safetensors",- "connector.safetensors",- ...- ]- }- ],- "env": {- "marker_paths": [ // OR-checked; first existing path wins- "ltx-2-mlx/env/pyvenv.cfg", // canonical Pinokio install- "ltx-2-mlx/.venv/pyvenv.cfg", // alternate manual layouts- ...- ]- },- "min_size_bytes": 1024 // file must be >= this to count as "complete"-}-```--When the upstream model layout changes, **edit this file first**. The-menu, panel, and install scripts auto-pick up the change.--## 17. Helper protocol (panel ↔ helper)--The helper subprocess (`mlx_warm_helper.py`) speaks newline-delimited-JSON over stdin/stdout. The panel's `WarmHelper` class wraps it.--### Actions panel sends--```jsonc-// Generation request (T2V, I2V, I2V-clean-audio)-{ "action": "generate",- "id": "<job-id>",- "params": { "mode": "t2v"|"i2v"|"i2v_clean_audio",- "prompt": "...", "output_path": "...",- "height": 704, "width": 1280, "frames": 121,- "seed": 42, "steps": 8, "image": "<path>" } }--// Extend request-{ "action": "extend",- "id": "<job-id>",- "params": { "video_path": "...", "prompt": "...",- "extend_frames": 5, "extend_direction": "after",- "extend_steps": 12, "cfg_scale": 1.0,- "output_path": "...", "seed": 42 } }--// Keyframe (FFLF)-{ "action": "keyframe", "id": "...", "params": {...} }--// Prompt rewrite (Gemma)-{ "action": "enhance", "prompt": "..." }-```--### Events helper emits--```jsonc-{ "event": "ready", "model": "<path>", "low_memory": true } // on startup-{ "event": "log", "line": "free-form panel log line" } // any informational message-{ "event": "done", "id": "...", "output": "<path>",- "elapsed_sec": 33.6, "seed_used": 42 }-{ "event": "error", "id": "...", "error": "<message>",- "trace": "<python traceback>" }-{ "event": "exit", "reason": "python_normal_exit"- | "sigterm(15)"- | "sigterm(2)"- | "idle" } // graceful exit-{ "event": "enhanced", "original": "...", "enhanced": "..." } // /prompt/enhance result-```--**No exit event + stdout pipe closes = non-graceful death** (SIGKILL,-SIGSEGV, SIGABRT, SIGBUS). The panel inspects `proc.poll()` returncode-and translates the negative signal number to a name in the user-facing-error.--### Step breadcrumb log lines--The helper emits these inside `generate`/`extend` actions so the panel-can localize a silent death to a phase:--```-step:get_pipe kind=t2v-step:get_pipe done-step:generate_and_save mode=i2v 1280x704 121f steps=8-step:generate_and_save done-```--When a job dies silently, the panel log's last `step:*` line tells you-which phase (pipeline init vs actual generation).--## 18. Memory budget model--Concrete weight sizes (HuggingFace repo metadata, Q4 unless noted):--| Component | Size on disk | Resident peak (rough) | Notes |-|---|---|---|---|-| `transformer-distilled.safetensors` (Q4) | 10.54 GiB | ~12 GiB | T2V/I2V "DiT" |-| `transformer-dev.safetensors` (Q4) | 11.3 GiB (rough) | ~13 GiB | Used by Extend |-| `transformer-dev.safetensors` (Q8) | 19.18 GiB | ~22 GiB | High-quality + FFLF |-| `connector.safetensors` (Q4) | 5.91 GiB | ~6 GiB | Lives inside `feature_extractor` |-| `Gemma-3-12B-it-4bit` | 7.48 GiB | ~9 GiB | Text encoder, freed before DiT load |-| `vae_decoder.safetensors` | 814 MB | ~1 GiB | Used at the end of generation |-| `vae_encoder.safetensors` | 638 MB | ~800 MB | I2V only (encodes input image) |-| `audio_vae.safetensors` | 107 MB | ~150 MB | |-| `vocoder.safetensors` | 258 MB | ~350 MB | |--Peak during T2V Standard 1280×704 (with patches applied): roughly-DiT + active activations + decoder ≈ 20–22 GiB resident. Without-the patches it's 26+ GiB and OOMs on memory-pressured 64 GB Macs.--Peak during I2V is +800 MB for the encoder pass + ~6 GiB for the-connector that lingered in `feature_extractor` (Patches 3 + 4 fix-the lingering — see §5).--macOS `jetsam` is the OOM killer. It enforces memory pressure based-on a percentage that varies with paging activity, swap usage, and other-heuristics — there is no fixed "kill at X% pressure" rule. Closing-Chrome, Slack, iOS Simulator before a Standard render is the safest-single thing a user can do.--## 19. Debug recipes--### "Helper died silently — where did it die?"--```bash-# Scan the panel log for the last step:* breadcrumb-curl -s http://127.0.0.1:8198/status | python3 -c \- "import json,sys; [print(l) for l in json.load(sys.stdin)['log'] if 'step:' in l][-3:]"-```--If last step is `step:get_pipe` → death during pipeline init (model-load). Likely OOM or weight-corrupt. If last step is-`step:generate_and_save` → death during gen (denoise / VAE decode).-Likely OOM peak or Metal kernel issue.--For non-OOM crashes, check `~/Library/Logs/DiagnosticReports/python3.11_*.crash`-or `*.ips` from around the crash time.--### "Panel says port 8198 already in use"--```bash-lsof -nP -iTCP:8198 -sTCP:LISTEN # who has it-pkill -f mlx_ltx_panel.py # nuke all panels-```--Common cause: a previous manual `python mlx_ltx_panel.py` is still-running. Pinokio's start.js can't take over a port someone else owns.-**Don't spawn the panel manually unless you're going to kill it before-handing back.**--### "I edited mlx_ltx_panel.py but the UI hasn't changed"--The HTML/JS is baked into the Python file as a template string. The-running panel served the old version. Restart it:-- via Pinokio: Stop → Start-- or via shell: `pkill -f mlx_ltx_panel.py` then click Start in Pinokio--Then **hard-refresh the browser** (Cmd+Shift+R) to bypass JS cache.--### "I edited the helper or a patch but nothing changed"--```bash-# 1. Re-apply patches (idempotent, applies to site-packages)-cd ~/pinokio/api/phosphene.git-./ltx-2-mlx/env/bin/python3.11 patch_ltx_codec.py--# 2. Restart the helper subprocess (panel keeps running)-curl -X POST http://127.0.0.1:8198/helper/restart--# 3. Submit a test job-curl -X POST http://127.0.0.1:8198/run \- -d "mode=t2v" -d "prompt=quick test" \- -d "width=512" -d "height=288" -d "frames=49" \- -d "steps=8" -d "quality=draft"--# 4. Wait for done-until [ "$(curl -s http://127.0.0.1:8198/status | python3 -c 'import json,sys;d=json.load(sys.stdin);print(0 if d.get(\"current\") else 1)')" = "1" ]; do sleep 5; done-```--### "How loud / clean is the audio in this clip?"--```bash-FFPROBE="/Volumes/Topaz Video AI 4.0.6/Topaz Video AI.app/Contents/MacOS/ffprobe"-FFMPEG="/Volumes/Topaz Video AI 4.0.6/Topaz Video AI.app/Contents/MacOS/ffmpeg"-"$FFMPEG" -i out.mp4 -af "volumedetect" -f null - 2>&1 | grep -E "max_volume|mean_volume"-```--Healthy on dialogue: `max_volume` between -2 and -15 dB. Below -25 dB-peak is the regression fingerprint — usually means a wrong mlx version.--### "Re-encode for X / web (yuv420p, faststart, 8 Mbps)"--```bash-"$FFMPEG" -y -i in.mp4 \- -c:v h264_videotoolbox -profile:v high -pix_fmt yuv420p \- -b:v 8M -maxrate 12M -bufsize 16M \- -movflags +faststart \- -c:a aac -b:a 192k \- out_x.mp4-```--X (Twitter) rejects yuv444p with a misleading "aspect ratio" error.-Use this recipe for any social upload.--### "Stuck queue / can't cancel"--```bash-curl -X POST http://127.0.0.1:8198/stop # cancel current job-curl -X POST http://127.0.0.1:8198/queue/clear # empty the queue-curl -X POST http://127.0.0.1:8198/helper/restart # nuke the helper-```--### "Force re-download a corrupt model file"--```bash-# Delete the file in question, then click Install in Pinokio (resumes-# only the missing files; intact ones are skipped via `hf` hash check).-rm ~/pinokio/api/phosphene.git/mlx_models/ltx-2.3-mlx-q4/<bad-file>.safetensors-# Then in Pinokio: Phosphene → Install (or Resume Install)-```--### "Force a different hardware tier for testing"--```bash-LTX_TIER_OVERRIDE=base ./run_panel.sh # pretend this Mac is < 48 GB-```--`run_panel.sh` actively unsets this on every run unless explicitly set-on the command line — there's a hard-learned bug history about leaked-overrides bleeding across panel restarts.--### "What does `/status` actually return?"--```bash-curl -s http://127.0.0.1:8198/status | python3 -m json.tool | head -60-```--Top-level keys: `running`, `paused`, `current`, `queue`, `history`,-`log`, `tier`, `memory`, `helper`, `download`, `repos_ready`,-`repos_total`, `base_available`, `q8_available`, `q8_missing`,-`comfy_pids`, `outputs`, `server_now`, `avg_elapsed_sec`.--## 20. Patch authoring template (`patch_ltx_codec.py`)--When adding a new patch:--```python-# ---- Patch N: <one-line summary> ---------------------------------------------# Why: <symptom + root cause + why this fix>. Reference upstream-# file:line and any external context (commit SHA, GitHub issue, paper).-PATCH_<NAME>_OLD = '''<exact text expected to be in the upstream file>'''-PATCH_<NAME>_NEW = '''<replacement text — include a `# PATCHED (LTX23MLX): ...` comment>'''-```--Then in `main()`:--```python-target = _find("<relative/path/to/upstream.py>")-outcome = apply_patch(- target, PATCH_<NAME>_OLD, PATCH_<NAME>_NEW,- marker="<unique substring that proves NEW is on disk>",- label="<NAME>",- upgrade_marker="<optional: substring that distinguishes a newer NEW from an older NEW>",-)-# If this patch is optional (don't fail install on drift):-if outcome in (OUTCOME_DRIFT, OUTCOME_MISSING):- print(" [<NAME>] note: ...", file=sys.stderr)- outcome = OUTCOME_ALREADY-outcomes.append(("<NAME>", outcome))-```--Conventions:-- `marker` is a substring **inside** `PATCH_<NAME>_NEW` that's unlikely- to appear in the unpatched file. Convention: include- `PATCHED (LTX23MLX...)` somewhere in the new text.-- `upgrade_marker` lets you ship a v2 of a patch without forcing users- to nuke their venv. Example: codec patch's `+faststart` was added- later; `upgrade_marker="+faststart"` distinguishes v1 (no faststart)- from v2 and rewrites the line in place.-- Patches that target an upstream code structure that may legitimately- vanish (e.g. an internal helper that gets refactored away) should be- marked **optional** in main() so install warns instead of fails.--## 21. Glossary--- **LTX 2.3** — Lightricks' latent text-to-video diffusion model with- joint audio synthesis. Trained on a large internal dataset.-- **MLX** — Apple's open-source ML array framework, native to Metal.- Like JAX/PyTorch but built for Apple Silicon. Not a torch shim.-- **Q4 / Q8** — Quantization levels of the LTX 2.3 weights. Q4 is- 4-bit quantized (smaller, faster, slightly lower quality), Q8 is- 8-bit (larger, slower, sharper detail). Phosphene defaults to Q4;- Q8 is an optional upgrade.-- **DiT** — Diffusion Transformer. The main denoising network. The- largest single weight tensor.-- **VAE** — Variational Autoencoder. Encodes pixels ↔ latents. The- encoder is used in I2V (encodes the input image); the decoder is- used at the end of every generation.-- **FFLF** — First-Frame / Last-Frame interpolation. Provide two images,- the model generates the motion between them. Requires Q8.-- **BWE** — Bandwidth Extension. Inside the audio vocoder, upsamples- a low-rate signal to full audio rate (108 sequential BigVGAN convs).-- **Vocoder** — Converts mel-spectrogram features to PCM audio. The- final stage of LTX's audio pipeline.-- **TeaCache** — A diffusion-step caching technique that skips- redundant computation when consecutive sigmas are similar. Used in- the High quality two-stage path.-- **jetsam** — macOS' kernel-level memory pressure killer. Sends SIGKILL- to processes when system memory gets tight. The dominant cause of- silent helper death on Macs at full RAM.-- **moov atom** — The MP4 metadata index. With `+faststart` it lives- at the front of the file (allows progressive playback / instant- thumbnails); without, it lives at the end (browser must download- the whole file before showing the first frame).-- **Pinokio** — One-click installer + runtime for AI apps on Mac/Linux/Windows.- Phosphene ships as a Pinokio app. cocktailpeanut authored Pinokio.-- **mrbizarro** — The GitHub handle that authors this repo. Use this,- always, in commits and metadata.--## 22. Open questions & v1.1 candidates--Tracked here so they don't get lost between sessions.--- **mlx 0.31.1 → 0.31.2 audio bisect** — find the exact MLX commit- that attenuates LTX 2.3's vocoder by 22 dB. File upstream issue.-- **A2V (audio-to-video) mode** — `ltx-2-mlx` upstream has- `a2vid_two_stage.py` and `a2vid_two_stage_hq.py` but the panel- doesn't expose them. UI work + helper plumbing required.-- **In-app HF token field** — currently we recommend `hf auth login`- in Terminal. A panel-side settings field that writes- `~/.cache/huggingface/token` would be friendlier for non-CLI users.-- **Pre-flight RAM advisory** — before submitting a heavy job, warn- if `s.memory.swap_gb > 8` or `pressure_pct > 75`. Suggest Draft.-- **Audio mode dropdown** — supersede the binary "No music" toggle- with: With music / Voice + ambient / Sound effects / Silent. Each- option appends/prepends a different prompt constraint.-- **Deeper helper breadcrumbs** — current `step:*` markers cover the- outer call. Adding `image_encode_start`, `text_encode_done`,- `denoise_start`, `decode_start` inside the pipeline (via wrapper- patches) would localize silent deaths down to the phase.-- **Q8 path validation** — Q8 generation has not been end-to-end- tested on a 64 GB Mac since the codec/I2V patches landed. Schedule- a 25 GB Q8 download + High quality T2V test.-- **FFLF validation** — same as above. Requires Q8.--## 23. External references--- LTX-2 paper / model card: https://huggingface.co/Lightricks-- MLX framework: https://github.com/ml-explore/mlx-- `ltx-2-mlx` (the MLX port we wrap): https://github.com/dgrauet/ltx-2-mlx-- Pinokio runtime: https://pinokio.computer-- Q4 weights: `dgrauet/ltx-2.3-mlx-q4` on HF-- Q8 weights: `dgrauet/ltx-2.3-mlx-q8` on HF-- Gemma text encoder: `mlx-community/gemma-3-12b-it-4bit` on HF-------# Generic Pinokio Development Guide--(Everything below this line is the upstream Pinokio scaffolding shipped-with every new project. Useful as a reference when working with the-launcher scripts but the Phosphene-specific rules above take-precedence on conflict.)--# Development Guide for Pinokio Projects--## Non-Negotiable Execution Workflow--To guarantee every contribution follows this guide precisely, obey this checklist **before any edits** and **again before finalizing**. Do not skip or reorder.-1. **AGENTS Snapshot:** Re-open this file and write down (in your working notes or response draft) the exact sections relevant to the requested task. No work begins until this snapshot exists.-2. **Destination Resolution:** Before creating or editing any Pinokio launcher files, resolve `PINOKIO_HOME` to an absolute path and record the intended destination root. If running outside Pinokio's own managed runtime, resolve in this order: `~/.pinokio/config.json` `home`, then `GET http://127.0.0.1:42000/pinokio/home` and use its `path` value, and if loopback is unreachable but `access` exists in `~/.pinokio/config.json`, retry the same request against `<protocol>://<host>:<port>/pinokio/home`, then the `PINOKIO_HOME` environment variable. If `PINOKIO_HOME` is still unresolved, stop and ask the user. Never silently use the current workspace as the launcher destination.-3. **Example Lock-in:** Identify the closest matching script in `~/pinokio/prototype/system/examples`. Record its path and keep it open while editing. Every launcher change must mirror that reference unless the user explicitly instructs otherwise.-4. **Pre-flight Checklist:** Convert the applicable rules from this document and `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md into a task-specific checklist (install/start/reset/update structure, regex patterns, menu defaults, log checks, destination path, etc.). Confirm each item is ticked **before** making changes.-5. **Mid-task Verification:** Any time you touch a Pinokio script, cross-check the corresponding example line to ensure syntax and structure match. Document the reference (example path + line) in your reasoning.-6. **Exit Checklist:** Before responding to the user, revisit the pre-flight checklist and explicitly confirm every item is satisfied. If anything diverges from the example or these rules, fix it first.--If any step cannot be completed, stop immediately and ask the user how to proceed. These six steps are mandatory for every session.--### Critical Pattern Lock: Capturing Web UI URLs--When writing `start.js` (or any script that needs to surface a web URL for a server):--1. **Always copy the capture block from an example such as `system/examples/mochi/start.js`.**-```javascript-on: [{- event: "/(http:\\/\\/[0-9.:]+)/",- done: true-}]-```--2. **Set the local variable using the captured match exactly as below (The regex capture object is passed in as `input.event`, so need to use the index 1 inside the parenthesis):**-```javascript-{- method: "local.set",- params: {- url: "{{input.event[1]}}"- }-}-```--3. Always try to come up with the most generic regex.-4. During the exit checklist, explicitly confirm that the `url` local variable is set via `local.set` API by using the captured regex object as passed in as `input.event` from the previous `shell.run` step.--Deviation from this pattern requires written approval from the user.--- Make sure to keep this entire document and `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md in memory with high priority before making any decision. Pinokio is a system that makes it easy to write launchers through scripting by providing various cross-platform APIs, so whenever possible you should prioritize using Pinokio API over lower level APIs.-- When writing pinokio scripts, ALWAYS check the examples folder (in ~/pinokio/prototype/system/examples folder) to see if there are existing example scripts you can imitate, instead of assuming syntax.-- When implementing pinokio script APIs and you cannot infer the syntax just based on the examples, always search the API documentation `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md to use the correct syntax instead of assuming the syntax.-- When trying to fix something or figure out what's going on, ALWAYS start by checking the `logs` folder before doing anything else, as mentioned in the "Troubleshooting with Logs" section.-- Finally, make sure to ALWAYS follow all the items in the "best practices" section below.--## Determine User Intent-If the initial prompt is simply a URL and nothing else, check the website content and determine the intent, and ask the user to confirm. For example a URL may point to--1. A Tutorial: the intent may be to implement a demo for the tutorial and build a launcher.-2. A Demo: the intent may be a 1-click launcher for the demo-3. Open source project: the intent may be a 1-click launcher for the project-4. Regular website: the intent may be to clone the website and a launcher.-5. There can be other cases, but try to guess.--## Working With Launchers--Apply this section only when the task is to create, modify, debug, review, or document a Pinokio launcher project.--If the request is not about launcher work, do not force an app-launcher vs plugin-launcher decision.--When the task does involve launcher work, first determine whether the request is for an app launcher or a plugin launcher. These are separate project types and must not be mixed.--### Mandatory Destination Resolution-- Before creating, editing, or moving any launcher files, resolve `PINOKIO_HOME` to an absolute path.-- If running outside Pinokio's own managed runtime, resolve `PINOKIO_HOME` in this order:- 1. `~/.pinokio/config.json` -> `home`- 2. `GET http://127.0.0.1:42000/pinokio/home` -> `path`- 3. If loopback is unreachable but `access` exists in `~/.pinokio/config.json`, retry the same request against `<protocol>://<host>:<port>/pinokio/home`- 4. `PINOKIO_HOME` environment variable-- Normalize the resolved value to an absolute path before using it.-- If neither source yields a valid `PINOKIO_HOME`, stop immediately and ask the user how to proceed. Do not guess. Do not silently fall back to the current workspace.-- If the current workspace is outside the resolved `PINOKIO_HOME/api` and `PINOKIO_HOME/plugin` trees, treat the current workspace only as source material, reference material, or evidence. Do not create the launcher in that workspace.-- Before any file creation, record and verify the exact target path:- - app launcher: `PINOKIO_HOME/api/<unique_name>`- - plugin launcher: `PINOKIO_HOME/plugin/<unique_name>`-- If the unique folder name is not obvious, ask the user before creating the target folder.--### 1. App launchers-- App launchers must live under `PINOKIO_HOME/api/<unique_name>`.-- App launchers are usually project-local launchers that manage one app in its own launcher/app folder.-- If you are already inside the target app launcher folder, build in that folder.-- If you are not already inside an app launcher folder, create a new folder under `PINOKIO_HOME/api/<unique_name>`.-- If the folder name is not obvious from the project or the user has not provided one, ask the user to confirm the folder name before creating it.-- Do not place app launchers under `PINOKIO_HOME/plugin`.--### 2. Plugin launchers-- Plugin launchers must live under `PINOKIO_HOME/plugin/<unique_name>`.-- Plugin launchers are reusable shared tools that are installed once and then used across many different folders.-- Standalone plugin launchers should keep `path: "plugin"` in the root `pinokio.js` so Pinokio installs them into `PINOKIO_HOME/plugin`.-- If you are already inside the target plugin launcher folder, build in that folder.-- If you are not already inside a plugin launcher folder, create a new folder under `PINOKIO_HOME/plugin/<unique_name>`.-- If the folder name is not obvious from the project or the user has not provided one, ask the user to confirm the folder name before creating it.-- Do not place plugin launchers under `PINOKIO_HOME/api`.-- When a plugin is meant to operate on the user's current project, its `run` step should target the caller's folder with `{{args.cwd}}` instead of the plugin folder itself.--### 3. Apply structure rules only after choosing the launcher type-- App launchers and plugin launchers are peers. Do not treat a plugin launcher as a special case of an app launcher, or vice versa.-- Decide the launcher type and destination folder first, then apply the project structure and script rules below.--## Project Structure--Pinokio projects normally follow a standardized structure with app logic separated from launcher scripts:--Pinokio projects follow a standardized structure with app logic separated from launcher scripts:--```-project-root/-├── app/ # Self-contained app logic (can be standalone repo)-│ ├── package.json # Node.js projects-│ ├── requirements.txt # Python projects-│ └── ... # Other language-specific files-├── README.md # Documentation-├── install.js # Installation script-├── start.js # Launch script-├── update.js # Update script (for updating the scripts and app logic to the latest)-├── reset.js # Reset dependencies script-├── pinokio.js # UI generator script-└── pinokio.json # Metadata (title, description, icon)-```--- Keep app code in `/app` folder only (never in root)-- Store all launcher files in project root (never in `/app`)-- `/app` folder should be self-contained and publishable---The only exceptions are serverless web apps---purely frontend only web applications that do NOT have a server component and connect to 3rd party API endpoints--in which case the folder structure looks like the following (No need for launcher scripts since the index.html will automatically launch. The only thing needed is the metadata file named pinokio.json):--```-project-root/-├── index.html # The serverless web app entry point-├── ...-├── README.md # Documentation-└── pinokio.json # Metadata (title, description, icon)-```--IMPORTANT: ALWAYS try to follow the best practices in the examples folder (~/pinokio/prototype/system/examples) instead of trying to come up with your own structure. The examples have been optimized for the best user experience.--## Launcher Project Working Directory--- The project working directory for a script is always the same directory as the script location.-- For example, when you run `shell.run` API inside `pinokio/start.js`, the default path for shell execution is `pinokio`.-- If the launcher files are in the project root path, then the default path for shell execution is the project root.-- Therefore, it is important to specify the correct `path` attribute when running `shell.run` API commands.--Example: in the following project structure:--```-project-root/-├── pinokio/ # Pinokio launcher folder-│ ├── start.js # Launch script-│ ├── pinokio.js # UI generator script-│ └── pinokio.json # Metadata (title, description, icon)-└─── backend/- ├── requirements.txt # App dependencies- └── app.py # App code-```--The `pinokio/start.js` should use the correct path `../backend` as the `path` attribute, as follows:--```-{- run: [{- ...- }, {- method: "shell.run",- params: {- message: "python app.py",- venv: "env",- path: "../backend"- }- }, {- ...- }]-}-```--## Development Workflow--### 1. Understanding the Project-- Check `SPEC.md` in project root. If the file exists, use that to learn about the project details (what and how to build)-- If no `SPEC.md` exists, build based on user requirements-### 2. Modifying Existing Launcher Projects-If we are starting with existing launcher script files, work with the existing files instead of coming up with your own.-- **Preserve existing functionality:** Only modify necessary parts-- **Don't touch working scripts:** Unless adding/updating specific commands-- **Follow existing conventions:** Match the style and structure already present-### 3. Try to adopt from examples as much as possible-- If starting from scratch, first determine what type of project you will be building, and then check the examples folder (~/pinokio/prototype/system/examples) to see if you can adopt them instead of coming up everything from scratch.-- Even if there are no relevant examples, check the examples to get inspiration for how you would structure the script files even if you have to write from scratch.-### 4. Writing from scratch as a last resort-If there are relevant examples to adopt from, write the scripts from scratch, but just make sure to follow the requirements in the next section.-### 5. Debugging-When the user reports something is not working, ALWAYS inspect the logs folder to get all the execution logs. For more info on how this works, check the "Troubleshooting with Logs" section below.--## Script Requirements--### 1. 1-click launchable-- The main purpose of Pinokio is to provide an easy interface to invoke commands, which may include launching servers, installing programs, etc. Make sure the final product provides ways to install, launch, reset, and update whatever is needed.--### 2. Write Documentation-- ALWAYS write a documentation. A documentation must be stored as `README.md` in the project root folder, along with the rest of the pinokio launcher script files. A documentation file must contain:- - What the app does- - How to use the app- - API documentation for programmatically accessing the app's main features (Javascript, Python, and Curl)--## Types of launchers-## 1. Launching servers-- When an app requires launching a server, here are the commonly used scripts:- - `install.js`: a script to install the app- - `start.js`: a script to start the app- - `reset.js`: a script to reset all the dependencies installed in the `install.js` step. used if the user wants to restart from scratch- - `update.js`: a script to update the launcher AND the app in case there are new updates. Involves pulling in the relevant git repositories installed through `install.js` (often it's the script repo and some git repositories cloned through the install steps if any)- - `pinokio.js`: the launcher script that ties all of the above scripts together by providing a UI that links to these scripts.- - `pinokio.json`: For metadata--Here's a basic server launcher script example (`start.js`). Unless there's a special reason you need to use another pattern, this is the most recommended pattern. Use this or adopt it as needed, but NEVER try something else unless there's a good reason you should not take this approach:--```javascript-module.exports = {- // By setting daemon: true, the script keeps running even after all items in the `run` array finishes running. Mandatory for launching servers, since otherwise the shells running the server process will get killed after the scripts finish running.- daemon: true,- run: [- {- // The "shell.run" API for running a shell session- method: "shell.run",- params: {- // Edit 'venv' to customize the venv folder path- venv: "env",- // Edit 'env' to customize environment variables (see documentation)- env: { },- // Edit 'path' to customize the path to start the shell from- path: "app",- // Edit 'message' to customize the commands, or to run multiple commands- message: [- "python app.py",- ],- on: [{- // The regular expression pattern to monitor.- // Whenever each "event" pattern occurs in the shell terminal, the shell will return,- // and the script will go onto the next step.- // The regular expression match object will be passed on to the next step as `input.event`- // Useful for capturing the URL at which the server is running (in case the server prints some message about where the server is running)- "event": "/(http:\/\/\\S+)/",-- // Use "done": true to move to the next step while keeping the shell alive.- // Use "kill": true to move to the next step after killing the shell.- "done": true- }]- }- },- {- // This step sets the local variable 'url'.- // This local variable will be used in pinokio.js to display the "Open WebUI" tab when the value is set.- method: "local.set",- params: {- // the input.event is the regular expression match object from the previous step- // In this example, since the pattern was "/(http:\/\/\\S+)/", input.event[1] will include the exact http url match caputred by the parenthesis.- // Therefore setting the local variable 'url'- url: "{{input.event[1]}}"- }- }- ]-}-```--## 2. Launching serverless web apps--- In case of purely static web apps WITHOUT servers or backends (for example an HTML based app that connects to 3rd party servers--either remote or localhost), we do NOT need the launcher scripts.-- In these cases, simply include `index.html` in the project root folder and everything should automatically work. No need for any of the pinokio launcher scripts. (Do-- You still need to include the metadata file so they show up properly on pinokio:- - `pinokio.json`: For metadata--## 3. Launching quick scripts without web UI--- In many cases, we may not even need a web UI, but instead just a simple way to run scripts.-- This may include TUI (Terminal User Interface) apps, a simple launcher-- In these cases, all we need is the launcher file `pinokio.js`, which may link to multiple scripts. In this case, there are no web apps (no serverless apsp, no servers), but instead just the default pinokio launcher UI that calls a bunch of scripts.-- Here are some examples:- - A pinokio script to toggle the desktop theme between dark and light- - Write some code (python or javascript or whatever)- - Write a `toggle.js` pinokio script that executes the code- - Write a `pinokio.js` launcher script to create a sidebar UI that displays the `toggle.js` so the user can simply click the "toggle" button to toggle back and forth between desktop themes- - A pinokio script to fetch some file- - Write some code (python or javascript or whatever)- - Write a `fetch.js` pinokio script that executes the code- - Write a `pinokio.js` launcher script to create a sidebar UI that displays the `fetch.js` so the user can simply click the "fetch" button to fetch some data.-- You still need to include the metadata file so they show up properly on pinokio:- - `pinokio.json`: For metadata--## API--This section lists all the script APIs available on Pinokio. To learn the details of how they are used, you can:-1. Check the examples in the ~/pinokio/prototype/system/examples folder-2. Read the `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md further documentation on the full syntax--### Script API--These APIs can be used to describe each step in a pinokio script:-- shell.run: run shell commands-- input: accept user input-- filepicker: accept file upload-- fs.write: write to file-- fs.read: read from file-- fs.copy: copy files-- fs.download: download files-- fs.link: create a symbolic link (or junction on windows) for folders-- fs.open: open the system file explorer at a given path-- fs.cat: print file contents-- jump: jump to a specific step-- local.set: set local variables for the currently running script-- json.set: update a json file-- json.rm: remove keys from a json file-- json.get: get values from a json file-- log: print to the web terminal-- net: make network requests-- notify: display a notification-- script.download: download a script from a git uri-- script.start: start a script-- script.stop: stop a script-- script.return: return values if the current script was called by a caller script, so the caller script can utilize the return value as `input`-- web.open: open a url in web browser-- hf.download: huggingfac-cli download API-### Template variables-The following variables are accessible inside template expressions (example `{{args.command}` in scripts, resulting in dynamic behaviors of scripts:-- input: An input is a variable that gets passed from one RPC call to the next-- args: args is the parameter object that gets passed into the script (via pinokio.js `params`). Unlike `input` which takes the value passed in from the immediately previous step, `args` is a global value that is the same through out the entire script execution.-- local: local variable object that can be set with `local.set` API-- self: refers to the script file itself (which is JSON or JavaScript). For example if `start.js` that's currently running has `daemon: true` set, `{{self.daemon}}` will evaluate to true.-- uri: The current script uri-- port: The next available port. Very useful when you need to launch an app at a specific port without port conflicts.-- cwd: The current script execution folder path-- platform: The current operating system. May be one of the following: `darwin`, `win32`, `linux`-- arch: The current system architecture. May be one of the following: x32, x64, arm, arm64, s390, s390x, mipsel, ia32, mips, ppc, ppc64-- gpus: array of available GPUs on the machine (example: `['apple']`, `['nvidia']`)-- gpu: the first available GPU (example: `nvidia`)-- current: The current variable points to the index of the currently executing instruction within the run array.-- next: The next variable points to the index of the next instruction to be executed. (null if the current instruction is the final instruction in the run array)-- envs: You can access the environment variables of the currently running process with envs object.-- which: Check whether a command exists and return its absolute path (example: `{{which('winget')}}`). This is the correct way to resolve command paths inside reproducible Pinokio scripts, including custom shell selection such as `shell: "{{which('bash')}}"`. If you are outside a Pinokio-managed shell and only need to inspect Pinokio's environment manually, use `pterm which <command>`, but do NOT copy that user-specific absolute path into launcher scripts.-- exists: Check whether a file or folder exists at the specified relative path (example: `"when": "{{!exists('app')}}"`). Can be used with the `when` attribute to determine a path's existence and trigger custom logic. Use relative paths and it will resolve automatically to the current execution folder.-- running: Check whether a script file is running (example: `"when": "{{!running('start.js')}}"`). Can be used with the `when` attribute to determine a path's existence and trigger custom logic. Use relative paths and it will resolve automatically to the current execution folder.-- os: Pinokio exposes the node.js os module through the os variable.-- path: Pinokio exposes the node.js path module through the os variable (example: `{{path.resolve(...)}}`--## System Capabilities-### Package Management (Use in Order of Preference)-The following package managers come pre-installed with Pinokio, so whenever you need to install a 3rd party binary, remember that these are available. Also, you can assume these are available and include the following package manager commands in Pinokio scripts:-1. **UV** - For Python packages (preferred over pip)-2. **NPM** - For Node.js packages-3. **Conda** - For cross-platform 3rd party binaries-4. **Brew** - Mac-only fallback when other options unavailable-5. **Git** - Full access to git is available.-6. **Bun** - For managing bun packages-**Important:** Include all install commands in the install script for reproducibility.-### HTTPS Proxy Support-- All HTTP servers automatically get HTTPS endpoints-- Convention: `http://localhost:<PORT>` → `https://<PORT>.localhost`-- Full proxy list available at: `http://localhost:2019/config/`-### Pterm Features:-- **Clipboard Access:** Read from or Write to system clipboard via pinokio Pterm CLI (`pterm clipboard` command.)-- **Notifications:** Send desktop alerts via pinokio pterm CLI (`pterm push` command.)-- **Script Testing:** Run launcher scripts via pinokio pterm CLI (`pterm start` command.)-- **File Selection:** Use built-in filepicker for user file/folder input (`pterm filepicker` command.)-- **Command Path Resolution:** Inspect the absolute path of any command as seen by Pinokio via `pterm which <command>`. Use this for debugging or external local tooling, especially when a helper process did not inherit Pinokio's `PATH`, for example `pterm which bash` on Windows. Do NOT hardcode the returned absolute path into launcher scripts; use `which()` or `kernel.which()` in the script itself instead.-- **Git Operations:** Clone repositories, push to GitHub-- **GitHub Integration:** Full GitHub CLI support (`gh` commands)--## Troubleshooting with Logs-Pinokio stores the logs for everything that happened in terminal at the following locations, so you can make use of them to determine what's going on:--### Log Structure-In case there is a `pinokio` folder in the project root folder, you should be able to find the logs folder here:--```-pinokio/-└── logs/ # Direct user interaction logs- ├── api/ # Launcher script logs (install.js, start.js, etc.)- ├── dev/ # AI coding tool logs (organized by tool)- └── shell/ # Direct user interaction logs-```--Otherwise, the `logs` folder should be found at project root:--```-logs/-├── api/ # Launcher script logs (install.js, start.js, etc.)-├── dev/ # AI coding tool logs (organized by tool)-└── shell/ # Direct user interaction logs-```--### Log File Naming-- Unix timestamps for each session-- Special "latest" file contains most recent session logs-- **Default:** Use "latest" files for current issues-- **Historical:** Use timestamped files for pattern analysis and the full history.--## Best practices-### 0. Always reference the logs when debugging-- When the user asks to fix something, ALWAYS check the logs folder first to check what went wrong. Check the "Troubleshooting with Logs" section.-### 1. Shell commands for launching programs-- Launch flags related- - Try as hard as possible to minimize launch flags and parameters when launching an app. For example, instead of `python app.py --port 8610`, try to do `python app.py` unless really necessary. The only exception is when the only way to launch the app is to specify the flags.-- Launch IP related- - Always try to find a way to launch servers at 127.0.0.1 or localhost, often by specifying launch flags or using environment variables. Some apps launch apps at 0.0.0.0 by default but we do not want this.-- Launch Port related- - In case the app itself automatically launches at the next available port by default (for example Gradio does this), do NOT specify port, since it's taken care of by the app itself. Always try to minimize the amount of code.- - If the install instruction says to launch at a specific port, don't use the hardcoded port they suggest since there's a risk of port conflicts. Instead, use Pinokio's `{{port}}` template expression to automatically get the next available port.- - For example, if the instruction says `python app.py --port 7860`, don't use that hardcoded port since there might be another app running at that port. Instead, automatically assign the next available port like this: `python app.py --port {{port}}`- - Note that the `{{port}}` expression always returns the next immediately available port for each step, so if you have multiple steps in a script and use `{{port}}` in multiple steps, the value will be different. So if you want to launch at the next available port and then later reuse that port, you will need to first use `{{port}}` to get the next available port, and save the value in local variable using `local.set`, and then use the `{{local.<variable_name>}}` expression later.-### 2. shell.run API-- When writing `shell.run` API requests, always use relative paths (no absolute paths) for the `path` field. For example, if you need to run a command from `app` folder, the `path` attribute should simply be `app`, instead of its full absolute path.-- If a launcher needs to use a command that Pinokio already provides, prefer resolving it with `{{which('command')}}` inside the script instead of assuming the command name will always be on `PATH`.-- Do NOT automatically avoid `bash`-based install commands on Windows. Pinokio's Windows environment includes `bash` through its bundled toolchain, so commands such as `curl -fsSL ... | bash` are acceptable when they run inside a Pinokio-managed shell and there is no simpler cross-platform alternative.-- If a Windows launcher needs to run the shell itself in bash instead of the default `cmd.exe`, set `shell: "{{which('bash')}}"` on the `shell.run` step.-- If a separate debugging process or external local tool did not inherit Pinokio's environment, you may use `pterm which <command>` to inspect what Pinokio would resolve. Do NOT turn that result into a hardcoded script path; for launcher scripts, always use `which()` or `kernel.which()` so the script stays reproducible across machines.-### 2. Package managers-- When installing python packages, try best to use `uv` instead of `pip` even if the install instruction says to use pip. Instead of `pip install -r requirements.txt`, you can simply use `uv pip install -r requirements.txt` for example. Even if the project's own README says use pip or poetry, first check if there's a way to use uv instead.-- When you need to install some global package, try to use `conda` as much as possible. Even on macs, `brew` should be only used if there are no `conda` options.-### 3. Minimal Always-- If you are starting with existing script files, before modifying, creating, or removing any script files, first look at `pinokio.js` to understand which script files are actually used in the launcher. The only script files used are the ones mentioned in the `pinokio.js` file. The `pinokio.js` file is the file that constructs the UI dynamically.-- Do not create a redundant script file that does something that already exists. Instead modify the existing script file for the feature. For example, do not create an `install.json` file for installation if `install.js` already exists. Instead, modify the `install.js` file.-- Pinokio accepts both JSON and JS script files, so when determining whether a script for a specific purpose already exists, check both JSON and JS files mentioned in the `pinokio.js` file. Do not create script files for rendundant purpose.-- When building launchers for existing projects cloned from a repository, try to stay away from modifying the project folder (the `~/pinokio/api/phosphene.git` folder), even if installations are failing. Instead, try to work around it by creating additional files in the launcher folder, and using those files IN ADDITION to the default project.- - The only exception when you may need to make changes to the project folder is when the user explicitly wants to modify the existing project. Otherwise if the purpose is to simply write a launcher, the app logic folder should never be touched.-- When running shell commands, take full advantage of the Pinokio `shell.run` API, which provides features like `env`, `venv`, `input`, `path`, `sudo`, `on`, etc. which can greatly reduce the amount of script code.- - Python apps: Always use virtual environments via `venv` attribute. This attribute automatically creates a venv or uses if it already exists.-### 4. Try to support Cross-platform as much as possible-- Use cross-platform shell commands only.-- This means, prefer to use commands that work on all platforms instead of the current platform.-- If there are no cross platform commands, use Pinokio's template expressions to conditionally use commands depending on `platform`, `arch`, etc.-- Also try to utilize Pinokio Pterm APIs for various cross-platform system features.-- If it is impossible to implement a cross platform solution (due to the nature of the project itself), set the `platform`, `arch`, and/or `gpu` attributes of the `pinokio.json` file to declare the limitation.-- Pinokio provides various APIs for cross-platform way of calling commonly used system functions, or lets you selectively run commands depending on `platform`, `arch`, etc.-### 5. Do not make assumptions about Pinokio API-- Do NOT make assumptions about which Pinokio APIs exist. Check the documentation.-- Do NOT make assumptions about the Pinokio API syntax. Follow the documentation.-### 6. Scripts must be able to replicate install and launch steps 100%-- The whole point of the scripts is for others to easily download and invoke them via Pinokio interface with one click. Therefore, do not assume the end user's system state, and make everything self-contained.-- When a 3rd party package needs to be installed, or a 3rd party repository needs to be downloaded, include them in the scripts.-### 7 Dynamic UI rendering-- The `pinokio.js` launcher script can change dynamically depending on the current state of the script execution. Which means, depending on what the file returns, it can determine what the sidebar looks like at any given moment of the script cycle.- - `info.exists(relative_path)`: The `info.exists` can be used to check whether a relative path (relative to the script root path) exists. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `info.running(relative_path)`: The `info.running` can be used to check whether a script at a relative path is currently running (relative to the script root path) exists. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `info.local(relative_path)`: The `info.local` can be used to return all the local variables tied to a script that's currently running. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `default`: set the `default` attribute on any menu item for whichever menu needs to be selected by default at a given step. Some example scenarios:- - during the install process, the `install.js` menu item needs to be set as the `default`, so it automatically executes the script- - when launching the `start.js` menu item needs to be set as the `default`, so it automatically executes the script- - after the app has launched, the `default` needs to be set on the web UI URL, so the user is sent to the actual app automatically.- - Check the examples in the ~/pinokio/prototype/system/examples folder to see how these are being used.-### 8. No need for stop scripts-- `pinokio.js` does NOT need a separate `stop` script. Every script that can be started can also be natively stopped through the Pinokio UI, therefore you do not need a separate stop script for start script-### 9. Writing launchers for existing projects-- When writing or modifying pinokio launcher scripts, figure out the install/launch steps by reading the project folder `app`.-- In most cases, the `README.md` file in the `~/pinokio/api/phosphene.git` folder contains the instructions needed to install and run the app, but if not, figure out by scanning the rest of the project files.-- Install scripts should work for each specific operating system, so ignore Docker related instructions. Instead use install/launch instructions for each platform.-### 10. Retrofitting an already-working setup-- Sometimes the user starts outside Pinokio, gets an app working through ad-hoc commands, and only later asks to turn that work into a Pinokio launcher.-- In this case, treat the current working setup and the successful session context as the highest priority source of truth. Do NOT restart from scratch if the app is already working.-- First capture the exact install and launch steps that already succeeded: cloned repositories, package manager commands, environment variables, model downloads, ports, working directories, helper scripts, and any fixes that were required.-- Then convert that knowledge into reproducible Pinokio scripts (`install.js`, `start.js`, `reset.js`, `update.js`, `pinokio.js`, `pinokio.json`) instead of telling the user to manually repeat the ad-hoc process.-- When the successful setup lives in a non-Pinokio folder, use that folder as evidence only. Resolve `PINOKIO_HOME` first, then produce the final launcher in the proper Pinokio location (`PINOKIO_HOME/api/<unique_name>` or `PINOKIO_HOME/plugin/<unique_name>`) unless the user explicitly asks for another layout.-- Replace machine-specific state with reproducible steps. Never hardcode absolute paths, user-specific cache locations, session-only ports, or one-off manual edits if they can be expressed in the launcher.-- Do not simply encode whatever happened to work on the current machine. Generalize the result into the broadest practical cross-platform, cross-machine launcher, and if limitations are unavoidable, declare them explicitly in `pinokio.json` instead of silently baking in local assumptions.-- If the app is already installed but the exact setup steps are partially missing, inspect the current working tree, generated files, dependency manifests, shell history when available, and logs to reconstruct the smallest reliable install and start flow.-- Verify from as clean a state as practical. A launcher is only done when another user could reproduce the working result without relying on undocumented steps from the original ad-hoc session.-### 11. Don't use Docker unless really necessary-- Some projects suggest docker as installation options. But even in these cases, try to find "development" options to launch the app without relying on Docker, as much as possible. We do not need Docker since we can automatically install and launch apps specifically for the user's platform, since we can write scripts that run cross platform.-### 12. pinokio.json-- Do not touch the `version` field since the version is the script schema version and the one pre-set in `pinokio.js` must be used.-- `icon`: It's best if we have a user friendly icon to represent the app, so try to get an image and link it from `pinokio.json`.- - If the git repository for the `~/pinokio/api/phosphene.git` folder points to GitHub (for example https://github.com/<USERNAME>/<REPO_NAME>`, ask the user if they want to download the icon from GitHub, and if approved, get the `avatar_url` by fetching `https://api.github.com/users/<USERNAME>`, and then download the image to the root folder as `icon.png`, and set `icon.png` as the `icon` field of the `pinokio.json`.-### 13. Gitignore-- When a launcher involves cloning 3rd party repositories, downloading files dynamically, or some files to be generated, these need to be included in the .gitignore file. This may include things like:- - Cloning git repositories- - Downloading files- - Dynamically creating files during installation or running, such as Sqlite Databases, or environment variables, or anything specific to the user.-- Make sure these file paths are included in the .gitignore file, and if not, include them in .gitignore.--## AI Libraries (Pytorch, Xformers, Triton, Sageattention, etc.)-If the launcher is for running AI models locally, the install script must declare the AI bundle so Pinokio can install the machine-level prerequisites before the script runs:--```-// install.js-module.exports = {- requires: {- bundle: "ai"- },- run: [- ...- ]-}-```--This is required even when the script also uses `torch.js`. The AI bundle is what triggers installation of common local AI prerequisites such as CUDA on NVIDIA systems and Hugging Face CLI.--If the launcher has a dedicated built-in script named `torch.js`, it can be used as follows:--```-// install.js-module.exports = {- requires: {- bundle: "ai"- },- run: [- // Edit this step with your custom install commands- {- method: "shell.run",- params: {- venv: "venv", // Edit this to customize the venv folder path- path: "app",- message: [- "uv pip install -r requirements.txt"- ],- }- },- // Delete this step if your project does not use torch- {- method: "script.start",- params: {- uri: "torch.js",- params: {- path: "app",- venv: "venv", // Edit this to customize the venv folder path- // xformers: true // uncomment this line if your project requires xformers- // triton: true // uncomment this line if your project requires triton- // sageattention: true // uncomment this line if your project requires sageattention- // flashattention: true // uncomment this line if your project requires flashattention- }- }- },- ]-}-```--The `torch.js` script also includes ways to install pytorch dependent libraries such as xformers, triton, sagetattention. If any of these libraries need to be installed, use the torch.js to install in order to install them cross platform.---## Quick Reference-### Essential Documentation-- **Pinokio Programming:** See `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md → "Programming Pinokio" section-- **Dynamic Menus:** See `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md → "Dynamic menu rendering" section-- **CLI Commands:** See `PTERM.md` at ~/pinokio/prototype/PTERM.md-### Common Patterns-- **Python Virtual Env:** `shell.run` with `venv` attribute-- **Cross-platform Commands:** Always test on multiple platforms-- **Error Handling:** Check logs/api for launcher issues-- **GitHub Operations:** Use `gh` CLI for advanced GitHub features-## Development Principles-1. **Minimize Shell Usage:** Leverage API parameters instead of raw commands-2. **Maintain Separation:** Keep app logic and launchers separate-3. **Follow Conventions:** Match existing project patterns-4. **Test Thoroughly:** Use CLI to verify launcher functionality-5. **Document Changes:** Update relevant metadata and documentationdiff --git a/GEMINI.md b/GEMINI.mdnew file mode 120000index 0000000..681311e--- /dev/null+++ b/GEMINI.md@@ -0,0 +1 @@+CLAUDE.md\ No newline at end of filediff --git a/QWEN.md b/QWEN.mddeleted file mode 100644index af1a878..0000000--- a/QWEN.md+++ /dev/null@@ -1,1799 +0,0 @@-# Phosphene — agent operating manual--This file is the single source of truth for how an AI assistant should-work on this codebase. Read it first, every session, before touching-any file. The generic Pinokio scaffolding starts at "## Generic Pinokio-Development Guide" further down — that's the upstream template. This top-section overrides it where they conflict.-------## 0. Current state — Phosphene 2.0 (released 2026-05-04)--VERSION file is now `2.0.0`. Git tag `v2.0.0` points at commit `fca78bb`-on `main`. Going forward: `2.0.x` for patches, `2.1.0` for features,-`3.0.0` only for another major story like this one. Pre-2.0 commit-messages and inline comments still reference the `Y1.NNN` sequential-counter (Y1.001 → Y1.039). Treat those as historical; do not introduce-new `Y` labels.--### What ships in 2.0 (capabilities checklist)--- **Four modes**: T2V, I2V, FFLF (keyframe interpolation), Extend.-- **Joint audio + video** in a single LTX 2.3 forward pass — sound is- generated WITH the video. Lip-sync, footsteps, ambient noise all- align at the frame level. mlx 0.31.1 pin is what keeps the vocoder- output at healthy levels (-9 to -15 dB peak on dialogue prompts).-- **Quality tiers**: `quick` / `balanced` / `standard` / `high`. The- earlier `draft` key was renamed; do not reintroduce it. `high` =- Q8 two-stage HQ pipeline + TeaCache, requires Q8 download.-- **Speed modes** for Standard renders: `exact` (full sampler) /- `boost` (skip up to 2 stable middle steps via adaptive caching) /- `turbo` (skip up to 3). 23–34 % wall-time savings on stable scenes.- Disabled for High, Extend, and FFLF where step-skipping breaks- quality contracts.-- **Sharp upscale** = PiperSR (CoreML) on the Apple Neural Engine.- Real x2 detail recovery before the ffmpeg fit/encode tail; ~26 s- overhead on a 5 s × 1024×576 native. The earlier "model" upscale- path (LTX latent x2) was retired in Y1.026; it remains hidden- behind `LTX_ENABLE_MODEL_UPSCALE=1` for lab archaeology only.-- **VAE temporal streaming** — long clips no longer freeze at the- final decode. Patch in `patch_ltx_codec.py` defaults to- `LTX_VAE_STREAMING=auto`: streams when latent frames > 121 (≈ 5 s),- full-decodes shorter clips. Helper-side env-var override keeps- installs that haven't re-applied the patch on the right path too.-- **Filtered hf downloads** (Y1.024+): Q4 ≈ 20 GB, Q8 ≈ 37 GB,- Gemma ≈ 7.5 GB, optional spatial upscaler ≈ 1 GB. Older builds- pulled the full upstream repos and bloated to Q4=56 GB / Q8=82 GB- by including duplicate transformer variants and unused upscalers.- `update.js` deletes the dupes on existing installs.-- **Hardware-tier auto-config** (`base` / `standard` / `high` / `pro`)- with realistic per-tier time estimates surfaced on the Quality- pills. Compact-tier users see honest "~12 min" instead of M-Studio- numbers.-- **Now-card progress bar** (Y1.039) — phase-aware, denoise-step-aware.- Computed server-side from history bucket + helper log signals.- Replaces the old `elapsed / global_avg` ratio that mis-paced both- Quick and High renders.-- **Gallery race-free** (Y1.039) — `list_outputs()` skips files that- ffmpeg is still writing (in-flight skip + 2 s mtime cutoff), and- the `/file` URL carries an `&v=<mtime>` cache-bust so the browser- fetches the freshest bytes when a file is replaced.-- **CivitAI LoRA browser** built into the panel. `mlx_models/loras/`- is scanned at boot; sidecar JSON metadata travels with each safetensors.-- **Phosphene 2.0 release badge** — violet `2.0` chip next to the- wordmark in the panel header. Distinct from the amber `DEV` profile- badge so dev/prod tabs are distinguishable at a glance.--### Empirical benchmarks (M4 Max 64 GB, "Comfortable" tier)--All numbers are real wall-clock from sidecar `elapsed_sec` on Y1.039 /-v2.0.0 runs today. Output mp4 carries audio in every case.--| Config | Frames | Wall |-|---|---|---|-| T2V Quick · exact · 640×480 | 121 (5 s) | ~ 2 m 14 s |-| T2V Standard · exact · 1280×704 | 121 (5 s) | ~ 7 m 40 s |-| T2V Standard · boost · 1280×704 | 121 (5 s) | ~ 6 m 20 s |-| T2V Standard · turbo · 1280×704 | 121 (5 s) | ~ 5 m 26 s |-| T2V High Q8 · 1280×704 | 121 (5 s) | ~ 11 m 51 s |-| T2V Balanced · turbo · 720p Sharp | 121 (5 s) | **~ 3 m 30 s** |-| T2V Balanced · turbo · 720p Sharp | 241 (10 s) | **~ 8 m 07 s** |-| I2V Standard · exact · 1280×704 | 121 (5 s) | ~ 7 m 51 s |-| I2V Balanced · turbo · 720p Sharp | 121 (5 s) | ~ 3 m 37 s |-| I2V Balanced · turbo · 720p Sharp | 241 (10 s) | ~ 8 m 26 s |-| FFLF · clamped to 768 px (Comfortable cap) | 121 (5 s) | ~ 5 m 29 s |-| Extend · 768 px (Comfortable cap), +3 s | source 121 → output 184 | ~ 15 m 50 s |--The marquee number for the launch thread is the-**Balanced + Turbo + 720p Sharp** recipe: 5-sec clip in 3 m 30 s, 10-sec-in 8 m 07 s. That's the recommended default for "fast HD with audio".--### Repository layout (now — both clones documented)--`~/pinokio/api/phosphene.git/` is the **production** clone.-Tracks `main`. Pinokio's daily-driver Phosphene panel runs from here on-port 8198.--`~/pinokio/api/phosphene-dev.git/` is the **dev** clone.-Tracks `dev`. Set up via `scripts/setup_dev_panel.sh`. Runs on port 8199-with the amber `DEV` profile badge in the header. Symlinks-`mlx_models/`, `mlx_outputs/`, `panel_uploads/`, and `state/` from a-shared Pinokio fs.link drive so weights aren't duplicated and outputs-land in one gallery regardless of which panel rendered them.--When a session starts: `git checkout dev` in the dev clone, work, push.-Promote to main only when Mr Bizarro gives the explicit green light. See §7.-------## 1. What Phosphene is--A free desktop panel that wraps Lightricks' **LTX 2.3** model running-natively on Apple's **MLX** framework, exposed as a one-click install-through **Pinokio**. Generates video and audio jointly in a single-forward pass on Apple Silicon Macs. Apple-Silicon-only.--Public artifacts:-- GitHub: `github.com/mrbizarro/phosphene` (public, MIT for the panel)-- Pinokio Discover: searchable as "Phosphene"-- Twitter handle: `@AIBizarrothe`--The panel itself is MIT. Model weights have Lightricks' own license.--## 2. Where to work--**The Pinokio install dir IS the canonical workspace.** There is no-separate dev folder. Old "local dev" copy was deleted to consolidate.--```-~/pinokio/api/phosphene.git/ ← edit here-├── .git/ ← real working copy of mrbizarro/phosphene-├── ltx-2-mlx/ ← upstream package, cloned at install-│ ├── env/ ← Python 3.11 venv-│ └── packages/{ltx-core-mlx,ltx-pipelines-mlx}-├── mlx_models/ ← downloaded weights-│ ├── ltx-2.3-mlx-q4/ ← Q4 base (~20 GB, filtered Y1.024+)-│ ├── ltx-2.3-mlx-q8/ ← optional Q8 (~37 GB, filtered Y1.024+)-│ └── gemma-3-12b-it-4bit/ ← text encoder (~7.5 GB)-├── mlx_outputs/ ← generated mp4s-├── panel_uploads/ ← user-uploaded reference images-├── assets/ ← logos, banner, hero clip-├── launch/ ← launch post drafts (X, Reddit, Pinokio, CivitAI)-├── mlx_ltx_panel.py ← MAIN: HTTP server, HTML UI, queue-├── mlx_warm_helper.py ← subprocess holding MLX pipelines warm-├── pinokio.js / install.js / start.js / update.js / reset.js / download_q8.js-├── patch_ltx_codec.py ← applies patches to upstream package-└── required_files.json ← single source of truth for "installed"-```--To work: `cd ~/pinokio/api/phosphene.git/`, edit, `git commit`,-`git push`. Pinokio's Update button does `git pull` here. No separate-copy to keep in sync.--## 3. Architecture (request → frame)--```-Pinokio app (port 42000)- └── proxies to → mlx_ltx_panel.py (port 8198)- │- ├── serves a single-page HTML/JS UI from one Python process- ├── exposes HTTP API: /run, /status, /upload, /helper/restart, ...- ├── owns a job queue (FIFO) + worker thread- └── speaks JSON-over-stdin/stdout to:- │- └── mlx_warm_helper.py (subprocess in same venv)- ├── holds MLX pipeline objects warm across jobs- ├── pipeline classes from ltx-2-mlx package- ├── reaps itself after LTX_IDLE_TIMEOUT (1800s default)- ├── emits events: log / done / error / exit- └── on death → panel sees stdout EOF-```--Both processes run inside `ltx-2-mlx/env` (the Python 3.11 venv). Helper-restarts cheaply via `POST /helper/restart` to pick up code changes-without restarting the panel.--## 4. Locked dependency versions--These are not arbitrary — every pin is a paid lesson:--| Pin | Why |-|---|---|-| `python>=3.11` (force `uv venv --python 3.11`) | Pinokio's `venv:` shortcut defaults to conda's base 3.10. ltx-core-mlx requires `>=3.11`. |-| `mlx==0.31.1` | mlx 0.31.2 introduced a numerical regression that attenuates LTX 2.3 vocoder output by ~22 dB. Verified: same model + same prompt + same seed → -42 dB peak on 0.31.2 vs -9 dB peak on 0.31.1. |-| `mlx-lm==0.31.1` `mlx-metal==0.31.1` | Same release line, kept consistent with mlx. |-| `huggingface-hub>=1.0` | `hf` CLI replaced `huggingface-cli`; older Pinokio bundles ship < 1.0. |-| `ltx-2-mlx` at `main` (HEAD) | dcd639e (0.1.0) was tried as a pin during audio investigation but breaks the Extend `cfg_scale` API and forces a `transformer.safetensors` symlink hack. HEAD + the mlx pin is the right combination. |--When changing a version, document the test that proved the new pin is OK.--## 5. The patch system (`patch_ltx_codec.py`)--We patch the upstream `ltx-2-mlx` package after `pip install`. Patches-are idempotent and fail loud on upstream drift (don't silently ship a-broken pipeline). Each patch has an `OLD` text it expects to find and a-`NEW` text it replaces with, plus a `marker` to detect "already patched."--Currently shipped patches:--1. **Codec (`yuv444p crf 0` + `+faststart`)** — required.- Patches `ltx_core_mlx/model/video_vae/video_vae.py`. Default ffmpeg- args become lossless H.264 (yuv444p crf 0) and the moov atom moves- to the front of the file so gallery thumbnails render the first- frame without downloading the full clip.--2. **I2V free DiT before VAE decode** — optional (warns if upstream changed).- Patches `ImageToVideoPipeline.generate_and_save` in- `ltx_pipelines_mlx/ti2vid_one_stage.py`. Frees DiT + text encoder +- feature extractor before the VAE decode step.--3. **I2V free vae_encoder + feature_extractor BEFORE denoise** — optional.- Patches `generate_from_image` to null `self.vae_encoder` and- `self.feature_extractor` right after `_encode_text_and_load`- returns. The connector inside feature_extractor is ~5.91 GiB on Q4- and was sitting resident through denoise for no reason.--4. **Base `load()` also clears feature_extractor** — optional.- Patches `TextToVideoPipeline.load`. Drops one extra ~5.91 GiB blob- from the peak when the DiT loads.--5. **VAE temporal streaming with auto-skip on short clips** (Y1.035 +- Y1.037). Patches `VideoDecoder.decode_and_stream` in- `ltx_core_mlx/model/video_vae/video_vae.py`. Upstream's method- *advertised* streaming but actually called `self.decode(latent)` on- the full video volume before iterating frames — caused multi-minute- end-of-render stalls / jetsam on long 720p clips. The patch routes- through `tiled_decode()` with a `TemporalTilingConfig` (default- tile = 64 latent frames, overlap = 24). Default mode is `auto`:- tiling is applied only when latent length > `LTX_VAE_STREAMING_AUTO_MAX_FRAMES`- (default 121 video frames ≈ 5 s). Short clips skip tiling and gain- ~30 s on 5-sec Standard renders. `LTX_VAE_STREAMING=0` forces- full-decode fallback; `=1` forces always-stream. Helper-side- `_apply_vae_streaming_decision()` in `mlx_warm_helper.py` sets the- env var per-job based on `num_frames`, so even pre-2.0 patched- installs that haven't re-run `patch_ltx_codec.py` get the right- behavior.--The "upgrade" path in `apply_patch()` lets old patched installs receive-new patch versions without a venv rebuild — used when we added-`+faststart` to the existing codec patch.--## 6. Critical history (so we don't relitigate)--- **mlx 0.31.2 audio regression** (resolved): symptom was "audio sounds- like static / very quiet." Root cause was an MLX numerical change- attenuating the vocoder output by ~22 dB. Fix: `mlx==0.31.1` pin.- Bisecting which mlx commit caused it is a v1.1 task.-- **dcd639e pin detour** (resolved): briefly pinned `ltx-2-mlx` to- commit `dcd639e` (0.1.0) thinking the audio regression was in- dgrauet's commits. It wasn't. Pin caused Extend API mismatch- (`cfg_scale` kwarg unknown) and required a transformer.safetensors- symlink because the 0.1.0 loader hardcoded that filename. Reverted.-- **Pinokio menu API mismatch** (resolved): `info.path` is a function- on newer Pinokio, a string on older. `pinokio.js` now detects both- shapes via `getInstallRoot(info)`.-- **start.js URL hijack** (resolved): the `errno`/`error:` event- patterns were cargo-culted from comfy.git's start.js. They captured- any "Errno" line from the helper into `input.event[0]` and the panel- URL got set to the literal string "Errno." Removed those patterns.-- **silent helper death** (mitigated, not fixed): on memory-pressured- Macs the helper subprocess is SIGKILL'd by macOS jetsam during I2V- pipeline load. Patches 3 and 4 reduced peak; the panel now names- the exit signal (SIGKILL/SIGSEGV/SIGABRT/SIGBUS) and points users at- `~/Library/Logs/DiagnosticReports/python3.11_*.crash` for diagnosis.-- **HTML5 form validation silent block** (resolved): a hidden Extend- input had `value=2 min=0.4 step=0.5`. Chrome's validator only accepts- values on the (min + N*step) sequence, so 2 was off-grid → invalid- → entire form silently failed to submit. Fixed `min=0.5`.--## 7. Working rules — non-negotiable--### Identity-- **Author/committer:** `mrbizarro <[email protected]>`.- Already set in this repo's local git config. Never override.-- **Never use the user's real first name** in commits, code, file- contents, or any artifact that lands on disk or in git. Same for- any third party (reviewers, collaborators) — refer to them by- technical role only ("upstream review", "external reviewer", "user- feedback") if at all.-- **Never add `Co-Authored-By:` trailers.** Phosphene reads as a- serious project shipped by `mrbizarro`. Co-author trailers showing- "Claude" make it look like a vibe-coded toy.--### Branch model — `main` is production, `dev` is staging (Y1.015+)--**TL;DR for agents — non-negotiable:**--> **Push every commit to `dev`. Never push to `main` unless Mr Bizarro-> explicitly says "push to main" / "ship to production" / "merge to-> main".** "It works on my machine" is not permission. "The tests-> pass" is not permission. Mr Bizarro has a Phosphene Dev panel in his-> Pinokio that pulls from `dev`; he tests there, then tells you when-> to promote.--The why:--- **`main`** is what real users get. Pinokio's Discover URL points- here; every commit pushed to `main` is shipped to every user the- next time they click Update. Treat it as production.-- **`dev`** is the staging branch. Push WIP commits here freely. The- local "Phosphene Dev" Pinokio panel (set up via- `scripts/setup_dev_panel.sh`) tracks this branch on port 8199 and- shows an amber DEV badge in the header.-- **Workflow**:- 1. Default to working on `dev`. `git checkout dev` at the start of- any session that touches Phosphene; if you're on `main`, switch.- 2. Commit + push to `origin/dev` whenever a logical change is done.- 3. **Stop there.** Tell Mr Bizarro it's on dev. Wait for explicit- instruction to promote.- 4. When Mr Bizarro says push to main: `git checkout main && git merge dev- --ff-only && git push origin main`. Then `git checkout dev` so- subsequent work continues on dev.- 5. If a fast-forward isn't possible (rare — usually a hotfix landed- on main directly), say so and ask before doing anything destructive.-- The panel auto-detects which profile it is from `git rev-parse- --abbrev-ref HEAD`: branch == `dev` → port 8199 + DEV badge; anything- else → port 8198 + normal header. No config files to edit per panel.-- `update.js` is branch-aware as of Y1.015 (`git pull --ff-only origin- <current branch>`) so the dev panel's Update pulls dev and the- production panel's Update pulls main, from the same shared code.-- The dev panel symlinks `mlx_models/` from the production install- (saves ~36 GB of duplicate weights). State / outputs / uploads stay- separate per panel.-- Hotfixes bypassing dev — only allowed when Mr Bizarro explicitly authorises- ("hotfix straight to main" or similar). Otherwise, even one-line- fixes go via dev → tested in Phosphene Dev panel → merged.--### Git policy — NEVER FORCE-PUSH `origin/main`-- A history-rewrite + force-push on 2026-05-01 (to scrub identity- leaks from old commits) broke `git pull` for every existing user- whose local clone contained orphaned commits — Pinokio's Update- silently stalled for them. GitHub aggressively GC'd the orphaned- objects within hours, so the pre-rewrite history is unrecoverable.-- **Do not run `git filter-branch`, `git rebase` (on shared history),- `git push --force`, `git push --force-with-lease`, or anything else- that rewrites already-published commits on `origin/main`. Ever.**-- If a commit needs amending and it's already pushed, **make a new- commit that reverts/fixes it.** The history is append-only.-- If the user requests a history rewrite (e.g. "scrub my name"), say- no and explain why: existing users get orphaned, their `git pull`- fails, the entire user base needs out-of-band recovery.-- Recovery from the 2026-05-01 incident — primary path is **Pinokio- Reset → Install** (clean, no shell scripts, models re-download is a- one-time cost). The `recover.sh` script is an escape hatch for users- who want to keep their downloaded models — kept in the repo with a- clear "read before running" header, NOT promoted as a `curl | bash`- command. Don't remove `recover.sh` or weaken `update.js`'s- fetch+reset fallback.--### `fs.link` — orthogonal to the history-divergence story-- `fs.link` is in `install.js` only (NOT `update.js`). It maps the- heavy/personal directories — `mlx_models/`, `mlx_outputs/`,- `panel_uploads/`, `state/` — to a Pinokio virtual drive that lives- outside the panel install dir. After a Pinokio Reset → Install, the- drive is rediscovered and the symlinks reattach, so models +- outputs + settings survive a clean reinstall.-- Why install.js only: future users get the drive on first install;- existing pre-Y1.004 users get it whenever they next reinstall (which- is the recommended recovery path anyway). Adding `fs.link` to- update.js would technically migrate existing real folders into the- drive without a Reset, but it conflates routine code updates with a- one-time 36 GB merge step, so we keep them separate.-- `fs.link` is unrelated to the 2026-05-01 history-rewrite incident- — don't conflate them in commit messages or docs.--### Commit message style-- Subject line in imperative mood, 50–72 chars max, no trailing period.- Use `feat:` / `fix:` / `refactor:` / `docs:` / `chore:` / `perf:` /- `test:` prefixes when they help.-- Body explains the **why**: the symptom, the root cause, the chosen- fix. The diff already shows the what.-- No first-person voice. Third person, neutral, technical.-- No conversation transcript. No "user said X", "after we discussed Y",- "per the review on Tuesday." Just describe the change.-- Cite concrete things: file paths, line numbers, package versions,- upstream commits, signal names. Avoid vague nouns ("the issue").-- Reference upstream context where it helps future maintainers (HF- metadata size, upstream commit SHA, GitHub issue number).-- One commit = one logical change. Don't bundle UI polish with bug- fixes with version pins.--### Examples — bad vs good--❌ Bad:-```-fix the audio thing peanut found--cocktailpeanut said the audio sounded like static so we tried a few-things and pinning mlx fixed it. great catch!-```--✅ Good:-```-fix(audio): pin mlx==0.31.1 to recover LTX 2.3 vocoder amplitude--mlx 0.31.2 introduces a numerical change that attenuates the LTX 2.3-vocoder output by ~22 dB. Verified empirically with the same prompt,-seed, and weights:- mlx 0.31.2 → max_volume -42.8 dB (audible only with significant boost)- mlx 0.31.1 → max_volume -9.2 dB--install.js installs the pinned trio (mlx, mlx-lm, mlx-metal) before-the ltx-* packages so transitive resolution does not pull latest.-update.js force-reinstalls them with --no-deps so existing installs-that already pulled 0.31.2 can recover by clicking Update.-```--### Code style-- Comments explain non-obvious decisions and trade-offs. They earn- their keep by explaining context the diff cannot.-- When fixing a bug, leave a comment with the symptom + cause + fix- near the change so the next reader doesn't reintroduce it.-- Patches in `patch_ltx_codec.py` always include a comment block- above each patch explaining what the upstream code does, why we- patch it, and what the patch achieves.--### Working with the running install-- The Pinokio panel may already be running on port 8198 when you- start a session. **Never spawn your own panel from the command- line** — Pinokio expects to own that port and will fail Start- with `Address already in use` if you do.-- To pick up code changes without restarting Pinokio, hit- `POST /helper/restart` — that reloads the helper subprocess- against the patched site-packages.-- For HTML/JS/CSS changes you do need a panel restart. Either- click Stop + Start in Pinokio, or `pkill -f mlx_ltx_panel.py`- and click Start.--### Disk consolidation-- Old `~/Documents/Codex/...` dev folder was deleted.- Don't recreate. The Pinokio install is canonical.-- `~/.cache/huggingface/hub/models--*` was also deleted. Pinokio- install pulls weights into `mlx_models/` directly via `hf- download --local-dir`, not the cache. Don't reintroduce HF cache- symlinks.--## 8. Test workflow--For HTML/UI/JS changes:-1. Edit `mlx_ltx_panel.py`-2. Restart panel (Pinokio Stop + Start, OR `pkill -f mlx_ltx_panel.py`- then click Start in Pinokio)-3. Hard-refresh browser (Cmd+Shift+R)-4. Verify the served HTML contains the change:- `python3 -c "import urllib.request; print('your-marker' in urllib.request.urlopen('http://127.0.0.1:8198/').read().decode())"`--For helper changes (Python pipeline, patches):-1. Edit `mlx_warm_helper.py` or `patch_ltx_codec.py`-2. If touching patches: re-run `./ltx-2-mlx/env/bin/python3.11 patch_ltx_codec.py`-3. `curl -X POST http://127.0.0.1:8198/helper/restart`-4. Submit a test job via `curl -X POST http://127.0.0.1:8198/run -d "mode=t2v" -d "prompt=..." -d "width=512" -d "height=288" -d "frames=49" -d "steps=8" -d "quality=draft"`-5. Poll `/status` until `current` is null and verify the new- `history[0]` entry is `done`.--For audio testing — do not use generic "wizard in forest" prompts.-Use prompts with explicit dialogue or sound events so artifacts are-audible:--```-A woman in her 30s with auburn hair, sitting at a wooden table in a-softly lit kitchen, speaks directly to the camera with a small smile-and clear, confident voice. She says "I really did not think I would-see you again, but here we are." Audio: clear human voice, slight-room reverb, gentle ambient kitchen sounds.-```--Measure with ffmpeg:-```-ffmpeg -i out.mp4 -af "volumedetect" -f null - 2>&1 | grep -E "max_volume|mean_volume"-```-Healthy levels: `max_volume` between -2 dB and -15 dB on dialogue-prompts. Below -25 dB peak suggests a regression.--## 9. Pinokio lifecycle scripts--| Script | Trigger | What it does |-|---|---|---|-| `install.js` | First Install or Resume Install | Clone `ltx-2-mlx`, force Python 3.11 venv, install MLX trio at pinned versions, install ltx-* packages, run `patch_ltx_codec.py`, `hf download` Q4 + Gemma. Idempotent. |-| `start.js` | Start button | Spawn panel inside the venv with `LTX_*` env vars set. Capture URL via regex with capture group, route to `local.set` via `input.event[1]`. |-| `update.js` | Update button | `git pull` panel + ltx-2-mlx, force-reinstall mlx pin, force-reinstall ltx-* packages, re-run patches. |-| `reset.js` | Reset button | Wipe venv + clones. **Preserves** `mlx_outputs/`, `panel_uploads/`, `mlx_models/` (those are user content / huge weight files). |-| `download_q8.js` | Download Q8 button | `hf download dgrauet/ltx-2.3-mlx-q8` to `mlx_models/ltx-2.3-mlx-q8`. |-| `pinokio.js` | Always | Renders the menu. State driven by `required_files.json` (env marker + per-repo file completeness). |--## 10. Files to never touch unprompted--- Anything inside `mlx_models/` — these are downloaded weights, GB-scale-- Anything inside `mlx_outputs/` — user-generated content-- Anything inside `panel_uploads/` — user uploads-- `panel_queue.json`, `panel_hidden.json` — runtime state-- `logs/`, `cache/`, `__pycache__/` — runtime caches-- The `.clinerules`, `.cursorrules`, `.windsurfrules`, `.geminiignore`,- `GEMINI.md`, `QWEN.md`, `AGENTS.md` files — these are generic Pinokio- scaffolding mirroring this `CLAUDE.md`. Edit `CLAUDE.md` first if a- project-wide change is needed; the rest are autogenerated mirrors.--## 11. HTTP API reference (panel)--The panel listens on `127.0.0.1:8198` and serves a single-page HTML/JS-UI plus a JSON API. Pinokio proxies its own port 42000 to 8198. All-endpoints are unauthenticated (loopback only).--### GET endpoints--| Path | Returns | Notes |-|---|---|---|-| `/` | HTML | The single-page UI. Built from one big template string in `mlx_ltx_panel.py`. |-| `/status` | JSON | Snapshot of `STATE` — running flag, current job, queue, history (top 20), log tail, tier, memory pressure, helper alive flag, comfy PIDs, repo download state. Polled by the UI every ~1s. |-| `/uploads` | JSON | Recent files in `panel_uploads/` for the picker's "click to reuse" strip. |-| `/models` | JSON | Per-repo install completeness — driven by `required_files.json`. |-| `/settings` | JSON | Returns `{settings, presets, default_preset}`. `settings` is the current `panel_settings.json` content. `presets` is the OUTPUT_PRESETS table (label + blurb + pix_fmt + crf per preset) so the UI doesn't duplicate it. |-| `/loras` | JSON | Returns `{user, curated, loras_dir, civitai_auth}`. `user` is the scanned `mlx_models/loras/*.safetensors` list (each with sidecar metadata if present). `curated` is the curated Lightricks repo registry minus the HDR-toggle entry (which is exposed as a plain checkbox, not in the picker). `civitai_auth` is True iff `CIVITAI_API_KEY` is set in the panel's env. |-| `/civitai/search` | JSON | Proxies CivitAI's `/api/v1/models` filtered to `types=LORA&baseModels=LTXV 2.3`. Query params: `query`, `nsfw` (bool), `cursor` (CivitAI uses cursor pagination), `limit`. Returns `{items, next_cursor, has_more}` with each item carrying `{id, version_id, name, creator, description, downloads, nsfw, preview_url, filename, size_kb, download_url, trigger_words, base_model, civitai_url}`. |-| `/file?path=…` | bytes (mp4/png/jpg) | Range-aware video serve so `<video>` tags can seek without re-downloading. |-| `/image?path=…` | bytes (image) | Same idea for image previews. |-| `/sidecar?path=…` | JSON | Per-output sidecar (job params, output stats) if it exists. |-| `/assets/<file>` | bytes | Static files from `assets/`. |--### POST endpoints--| Path | Body | Returns | Effect |-|---|---|---|---|-| `/run` `/queue/add` | urlencoded job spec | `{ok, id}` | Creates a job, appends to queue. Field reference below. |-| `/queue/batch` | urlencoded with `prompts` (newline-separated, `---` chunks) | `{ok, added, ids}` | Splits the prompts on `^\s*---\s*$` boundaries, creates one job per chunk. |-| `/queue/remove` | `id=<jid>` | `{removed: bool}` | Drop a queued job by id. |-| `/queue/clear` | (none) | `{cleared: int}` | Clear all queued jobs (running job continues). |-| `/queue/pause` | (none) | `{paused: true}` | Worker stops popping new jobs after the current one. |-| `/queue/resume` | (none) | `{paused: false}` | Worker resumes. |-| `/output/hide` | `path=<p>` | `{hidden: path}` | Adds path to the gallery's hidden set (persisted). |-| `/output/show` | `path=<p>` | `{shown: path}` | Inverse. |-| `/output/show_all` | (none) | `{unhidden_count: int}` | Clear hidden set. |-| `/upload` | multipart `image=<file>` | `{ok, path}` | Saves to `panel_uploads/<ts>_<safename>`. |-| `/helper/restart` | (none) | `{ok}` | SIGTERM the helper subprocess; the next job auto-respawns it. Useful for picking up site-packages changes. |-| `/settings` | `output_preset` / `output_pix_fmt` / `output_crf` / `civitai_api_key` / `hf_token` (any subset) | `{ok, settings, helper_restarted}` | Update + persist. Public-safe `settings` view in the response (no token values, just `has_*` booleans). Helper killed on codec change OR token change so the next job spawns with fresh env. Form parser uses `keep_blank_values=True` so `civitai_api_key=` (the Clear button) is treated as "remove this token" rather than dropped silently. Empty input on the front-end Apply path is "leave as-is" — explicit clearing goes through the dedicated Clear button. |-| `/loras/refresh` | (none) | `{ok, user, loras_dir}` | Rescan `mlx_models/loras/`. Filesystem is the source of truth; no caching layer to invalidate. |-| `/loras/delete` | `path=<abs path>` | `{ok, removed}` | Delete a user-installed LoRA + its sidecar. Path must resolve inside `mlx_models/loras/` to prevent traversal. |-| `/civitai/download` | `download_url=<civitai url>`, `meta=<json>` | `{ok, name, path, sidecar_path, size_bytes, skipped}` | Stream-download a LoRA from CivitAI into `mlx_models/loras/`, write a sidecar JSON. Refuses non-civitai.com hosts. Writes to a `.partial` then atomic-renames so a kill mid-write leaves nothing the next scan would mistake for a complete file. Surfaces 401 with a remediation hint pointing at `CIVITAI_API_KEY`. |-| `/stop` | (none) | `{ok}` | Cancel the current job (kills helper + mux). Worker advances. |-| `/stop_comfy` | (none) | `{ok, killed}` | SIGTERM any ComfyUI process matching `LTX_COMFY_PATTERN`. |-| `/open_pinokio` | (none) | `{ok}` | macOS-only: focus the Pinokio app. |-| `/models/download` | `key=q4|q8|gemma` | `{ok}` | Spawn `hf download` for the named repo. Streams to log. |-| `/models/cancel` | (none) | `{ok, killed}` | SIGTERM the active download. |-| `/prompt/enhance` | `prompt=<p>` | `{ok, original, enhanced}` | Calls helper's Gemma-rewrite path. |--## 12. Job spec — the contract for `/run` and `/queue/add`--Every form field below maps to `params.<field>` on the resulting job.-`make_job()` in `mlx_ltx_panel.py:1140` is the canonical reference.--| Field | Type | Default | Meaning |-|---|---|---|---|-| `mode` | str | `"t2v"` | One of: `t2v`, `i2v`, `i2v_clean_audio`, `extend`, `keyframe`. |-| `prompt` | str | `"A cinematic atmospheric scene"` | Empty falls back to the default. |-| `width` | int | `1280` | Clamped to `>= 32`. T2V only — image flows derive from aspect+quality. |-| `height` | int | `704` | Same as width. |-| `frames` | int | `121` | Number of video frames. The 8k+1 rule (latents are groups of 8). |-| `steps` | int | `8` | Denoising steps. T2V/I2V Q4 distilled requires `>= 8` (schedule is locked). |-| `seed` | int-as-str | `"-1"` | `"-1"` → random. Stored as string so the form roundtrip is lossless. |-| `image` | path | `REFERENCE` | Only used when `mode != "t2v"`. |-| `audio` | path | `AUDIO_DEFAULT` | Only used when `mode == "i2v_clean_audio"` (mux external audio). |-| `video_path` | path | `""` | **Required for `mode=extend`**. Source mp4 to extend. |-| `extend_frames` | int | `5` | Latent-frame count to add. Each latent = 8 video frames. So 5 latents = 40 frames ≈ 1.67s at 24 fps. |-| `extend_direction` | str | `"after"` | `"after"` or `"before"`. |-| `extend_steps` | int | `12` | Denoising steps for the extend pipeline (uses dev transformer). |-| `extend_cfg` | float | `1.0` | CFG scale. 1.0 = no guidance (faster, less RAM). 3.0 = guided (Quality mode). |-| `start_image` | path | `""` | **Required for `mode=keyframe`**. First frame. |-| `end_image` | path | `""` | **Required for `mode=keyframe`**. Last frame. |-| `enhance` | "on"/"off" | `"off"` | Reserved (panel uses the explicit `/prompt/enhance` button instead). |-| `stop_comfy` | "on"/"off" | `"off"` | Kill ComfyUI before this render to free MPS memory. |-| `open_when_done` | "on"/"off" | `"off"` | macOS `open` the output when finished. |-| `quality` | str | `"balanced"` | `"quick"`, `"balanced"`, `"standard"`, or `"high"`. Drives dimensions, step counts, and pipeline. |-| `accel` | str | `"off"` | `"off"` (alias for exact) / `"boost"` / `"turbo"`. Adaptive-cache denoise; boost skips up to 2 cached steps, turbo up to 3. T2V/I2V Standard only — disabled for High, Extend, FFLF. |-| `upscale` | str | `"off"` | `"off"` / `"fit_720p"` / `"x2"`. Post-render export. Balanced defaults to `fit_720p`. |-| `upscale_method` | str | `"lanczos"` | `"lanczos"` (ffmpeg) or `"pipersr"` (Sharp, ANE-accelerated). PiperSR adds ~26 s on 5-sec 1024×576. |-| `no_music` | "on"/"off" | `"off"` | When on, the panel appends a no-music audio constraint to the prompt at submit time. |-| `preset_label` | str | `""` | Optional human-readable label for the queue/history rows. |--## 13. Mode → pipeline → transformer routing--| Mode | Pipeline class | Transformer | Quality flag | Notes |-|---|---|---|---|---|-| `t2v` | `TextToVideoPipeline` | distilled Q4 | quick → smaller dims; balanced → 1024×576 + 720p export; standard → 1280×704; high → `TwoStageHQPipeline` w/ Q8 dev | Most common path. |-| `i2v` | `ImageToVideoPipeline` | distilled Q4 | same tier ladder as T2V | Adds VAE-encoder pass on the input image. |-| `i2v_clean_audio` | `ImageToVideoPipeline` | distilled Q4 | same | Generates LTX video, then panel-side ffmpeg mux replaces the LTX audio with `audio` param. |-| `extend` | `ExtendPipeline` | **Q8 dev** (Y1.036+) | always uses Q8 `transformer-dev.safetensors`; cfg_scale dial via `extend_cfg` | Heavier (~75–80 s/step on 768×416 cfg=1.0). Pre-Y1.024 a duplicate `transformer-dev.safetensors` shipped in the Q4 dir as download bloat — Extend silently loaded from there. Y1.024 download filter pruned the dupe and exposed that Extend is structurally Q8-class. Panel auto-routes to Q8 dir; UI gates on Q8 download (same shape as FFLF). |-| `keyframe` (FFLF) | `KeyframePipeline` | **Q8 dev + distilled LoRA** | requires Q8 download | Disabled on `base` tier. Comfortable tier clamps to 768 max-side (the stage-1 → stage-2 transition OOMs at 1280×704 on 64 GB). |--The helper `get_pipe(kind)` lazily loads the pipeline class corresponding-to the mode. Switching modes between jobs frees the previous pipeline-(`release_pipelines(keep_kind=...)`) — strict one-pipeline-at-a-time-policy keeps memory bounded.--## 14. Quality tier mapping--`quality` is independent of the hardware tier. Hardware tier *clamps*-the maximum dimensions per mode; `quality` picks the pipeline + step-count *within* that clamp.--| `quality` | Pipeline used | Dimensions | Steps | Approx wall on M4 Max 64 GB |-|---|---|---|---|---|-| `quick` | distilled Q4 | 640×480 (4:3 fast sanity) | 8 | ~ 2 m 14 s @ 121 frames |-| `balanced` | distilled Q4 + ffmpeg or PiperSR fit_720p export | 1024×576 → 1280×720 export | 8 | ~ 4 m 36 s (Lanczos) / ~ 5 m 02 s (Sharp) @ 121 frames |-| `standard` | distilled Q4 | 1280×704 (full target, clamped by tier) | 8 | ~ 7 m 40 s @ 121 frames; ~ 5 m 26 s with `accel=turbo` |-| `high` | `TwoStageHQPipeline` (Q8 dev + res_2s sampler + TeaCache) | 1280×704 | 15 stage-1 + 3 stage-2 | ~ 11 m 51 s @ 121 frames |--`high` and `keyframe` require the Q8 model on disk (~37 GB filtered).-`extend` also requires Q8 since Y1.036 (transformer-dev). Panel surfaces-a "Download Q8" CTA + disables the Generate button when the user picks-a Q8-required mode without Q8 installed.--Independent of `quality`, T2V/I2V Standard takes an `accel` flag-(`exact` / `boost` / `turbo`) that drives an adaptive-cache denoise-loop. Boost can skip up to 2 stable middle steps, Turbo up to 3.-Cached steps near-zero wall; full steps stay ~53 s/it on 1280×704.-Per-step decisions are recorded in the sidecar's `accel_metrics` block.--## 15. Hardware tier system--`SYSTEM_TIER` is detected once at boot from `sysctl hw.memsize`. The-override `LTX_TIER_OVERRIDE=base|standard|high|pro` forces a tier-(useful for testing what other users see). `CAPABILITIES` in-`mlx_ltx_panel.py:584` is the canonical table; abbreviated:--| Tier | RAM range | Friendly | t2v/i2v max dim | Q8 | FFLF | Extend |-|---|---|---|---|---|---|---|-| `base` | < 48 GB | Compact | 768 | ❌ | ❌ | ❌ |-| `standard` | 48–79 GB | Comfortable | 1280 | ✅ | ✅ | ✅ |-| `high` | 80–119 GB | Roomy | 1280 | ✅ | ✅ (more frames) | ✅ |-| `pro` | ≥ 120 GB | Studio | 0 (no clamp) | ✅ | ✅ | ✅ |--The thresholds are intentional. M-Studio 64 GB is the canonical-"standard" hardware. Anything below 48 GB cannot fit Q8 weights-(transformer-dev.safetensors is 19.18 GiB plus working tensors).--## 15a. `panel_settings.json` schema--User-controllable preferences. Persisted at `<repo>/state/panel_settings.json`-(Y1.004+); pre-Y1.004 lived at `<repo>/panel_settings.json` and is migrated-on first boot by `_migrate_state_dir()`. State directory is gitignored.--The whole `state/` dir is symlinked into Pinokio's fs.link drive so it-survives Reset → Reinstall (see "Persistent storage / fs.link" in install.js-+ update.js). All persisted-state files (`panel_settings.json`,-`panel_queue.json`, `panel_hidden.json`) live there.--Read at panel startup; mutated via `POST /settings`. The helper subprocess-inherits codec env vars from this at spawn — when settings change, the-panel kills the helper so the next job picks up new values.--```jsonc-{- "version": 2, // bumped from v1 when secrets added- // Metadata so the UI knows which preset to highlight as active.- // "custom" means pix_fmt + crf were set manually outside any preset.- "output_preset": "standard" | "archival" | "web" | "custom",- "output_pix_fmt": "yuv420p" | "yuv444p" | "yuv422p"- | "yuv420p10le" | "yuv422p10le" | "yuv444p10le",- "output_crf": "0".."30", // string (passed via env vars)-- // Secrets. Stored locally only, never sent over the network except- // as Authorization headers / ?token= URL params to the upstream- // services (civitai.com, huggingface.co). NEVER returned by GET- // /settings — the public response surfaces booleans only.- "civitai_api_key": "", // empty = not configured- "hf_token": "" // empty = not configured-}-```--### `GET /settings` security contract--The full settings dict — including `civitai_api_key` and `hf_token` —-is read by the backend only. The HTTP API exposes a *masked* view via-`get_settings_public()`:--```jsonc-{- "version": 2,- "output_preset": "standard",- "output_pix_fmt": "yuv420p",- "output_crf": "18",- "has_civitai_key": false, // bool, never the value- "has_hf_token": false // bool, never the value-}-```--This applies on every response that includes settings — success path,-validation errors, helper-restart confirmations. A user with the panel-proxied through ngrok / tunneled to the LAN never leaks their saved-secrets through a status poll.--### Token resolution priority--`_active_civitai_key()` and `_active_hf_token()` resolve the active-token in this order:--1. The saved value in `panel_settings.json` (set via the Settings- modal). This is the path normal users take.-2. The corresponding env var (`CIVITAI_API_KEY` / `HF_TOKEN`). Power- users with shell-level config keep working unchanged.--Empty string in #1 falls through to #2. The settings UI's "Clear"-button writes an empty string to #1, so a user who configured a key-in the UI and then wants to revert to env-var-based config can do-so cleanly.--For HF specifically, `huggingface_hub.snapshot_download` *also* reads-`~/.cache/huggingface/token` (the file `hf auth login` writes). We-don't override that file — if neither settings nor `HF_TOKEN` env is-set, the library falls back to that cached file. This means power-users who already ran `hf auth login` get gated LoRAs working without-touching the panel's token settings.--Preset table is the single source of truth, defined in `OUTPUT_PRESETS`-in `mlx_ltx_panel.py`:--| key | pix_fmt | crf | Approx 5s @ 1280×704 | Use case |-|---|---|---|---|---|-| `standard` ⭐ default | yuv420p | 18 | ~7 MB | Visually lossless, plays everywhere (X / IG / Discord) |-| `archival` (UI label "Video production") | yuv444p | 0 | ~50 MB | Mathematically lossless, full 4:4:4 chroma. For color grading, compositing, VFX, anywhere downstream re-encode is expected. Internal key kept as `archival` for backwards compat with settings files saved before the rename. |-| `web` | yuv420p | 23 | ~3 MB | Smallest, mobile / embedding |--`+faststart` is always applied (codec patch) regardless of preset, so-gallery thumbnails always render the first frame instantly.--When adding a new preset: edit `OUTPUT_PRESETS` in `mlx_ltx_panel.py`-and the `order` array in the settings modal JS (`openSettingsModal`).-The blurb shown in the UI is read from the server at modal-open time,-so server is the source of truth — no JS strings to keep in sync.--## 15b. LoRA system--LTX 2.3 supports LoRA adapters via the upstream `apply_loras()` function-in `ltx_core_mlx.loader.fuse_loras`. The fusion is **weight-level**: at-pipeline load time, each LoRA's `(lora_A, lora_B)` matrix pair is-multiplied + added to the corresponding base-model weight, modified in-place, then quantized. This is *not* a runtime adapter — switching-LoRAs requires reloading the pipeline.--### Integration hook--The base `TextToVideoPipeline.load()` checks-`getattr(self, "_pending_loras", None)` and, if present, calls-`_fuse_pending_loras(transformer_weights, lora_paths)` before-quantizing the transformer. Set `pipe._pending_loras = [(path, strength), ...]`-between construction and the first job.--The wrapper in `mlx_warm_helper.py:_attach_loras` does this. It also-resolves HuggingFace repo IDs (`Lightricks/LTX-2.3-22b-IC-LoRA-HDR`)-to local files via `snapshot_download`, picking the largest-`.safetensors` in the repo (LoRA repos sometimes ship auxiliary files-like scene embeddings alongside the main weights).--### Pipeline cache invalidation--`get_pipe(kind, loras=...)` keys the cache on `(kind, frozen LoRA set)`.-Changing the LoRA set forces a pipeline rebuild — the cached pipeline's-weights have the previous LoRA set fused in and there's no clean way-to "unfuse." Cost is the standard ~30s pipeline reload.--The frozen LoRA set is order-insensitive (set hash, not list hash) —-fusion is commutative so `[A, B]` and `[B, A]` produce identical-weights and should hit the same cache entry.--### `mlx_models/loras/` directory--User-installed LoRAs live as `<name>.safetensors` files at the install-root's `mlx_models/loras/` directory. An optional sidecar JSON next to-each `.safetensors` carries display metadata:--```jsonc-{- "name": "Crisp Enhance",- "description": "Sharper detail and edge enhancement.",- "trigger_words": [], // optional, prepended to prompt at submit- "recommended_strength": 0.8, // 0.0..1.5 typical, displayed as default in slider- "preview_url": "https://...", // optional, shown in the picker- "base_model": "LTXV 2.3",- "civitai_id": 2530917, // optional, for attribution- "civitai_version_id": 2844417,- "civitai_url": "https://civitai.com/models/2530917",- "downloaded_at": "2026-05-01T..."-}-```--Sidecar absent → panel falls back to the filename as the display name-and zero metadata. Sidecar shape matches the JSON `_civitai_download`-writes, so a CivitAI install lands a usable picker entry without any-hand-editing.--### Curated LoRA registry--`CURATED_LORAS` in `mlx_ltx_panel.py` lists Lightricks' official LoRAs-that we know about. Currently:--| key | repo | UI surface | Note |-|---|---|---|---|-| `hdr` | `Lightricks/LTX-2.3-22b-IC-LoRA-HDR` | "HDR" toggle in form | Hidden from picker (`is_hdr_toggle: True`); exposed as a plain checkbox |-| `motion-track` | `Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control` | Listed in picker | IC-LoRA, needs video conditioning (not yet wired in panel UI) |-| `union-control` | `Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control` | Listed in picker | Same |--All three are **gated** on Hugging Face — users must accept the license-on the model page AND have a valid HF token (`hf auth login`) before-the helper's `snapshot_download` will work. The helper translates the-upstream `GatedRepoError` into a clear actionable error.--### Job spec extension--The `generate` and `extend` actions accept an optional `loras` param:--```jsonc-{- "action": "generate",- "id": "...",- "params": {- "mode": "t2v",- "prompt": "...",- "loras": [- {"path": "/abs/path/to/style.safetensors", "strength": 0.8},- {"path": "Lightricks/LTX-2.3-22b-IC-LoRA-HDR", "strength": 1.0}- ],- ...- }-}-```--Path may be a local file or an HF repo id. Strength is clamped to-±2.0 by the panel before sending; values outside the typical 0.0..1.5-range are usually nonsense and risk numerical issues during fusion.--### HDR toggle semantics--The `hdr` boolean on the job (separate from `loras`) is a UI shortcut.-Worker-side, `run_job_inner` resolves it to-`{"path": "Lightricks/LTX-2.3-22b-IC-LoRA-HDR", "strength": 1.0}`-and appends to the LoRA list before calling the helper. Users see-"HDR" as a feature; the LoRA implementation is hidden.--### CivitAI bridge--CivitAI's REST API is the de-facto LoRA distribution hub for LTX 2.3.-Two endpoints proxy through the panel:--- `/civitai/search` filtered to `baseModels=LTXV 2.3` (the canonical- string CivitAI uses for LTX-2.3 LoRAs as of 2026-05). Cursor-paginated.-- `/civitai/download` streams a `.safetensors` into `mlx_models/loras/`- + writes a sidecar JSON.--Both endpoints accept an optional `CIVITAI_API_KEY` env var. As of 2025,-CivitAI requires a token for most LoRA downloads (search still works-without). The token goes in `?token=<key>` on the URL — many CDN hops-only honor that form, not the Authorization header.--### Adding a new curated LoRA--1. Edit `CURATED_LORAS` in `mlx_ltx_panel.py` with `id`, `name`,- `description`, `repo_id`, `default_strength`, and either- `is_hdr_toggle: True` (= surfaces as a header pill, hidden from- picker) or `False` (= shows in picker as a usable entry).-2. If `is_hdr_toggle`, also add the HTML toggle pill (next to "No- music"), the JS sync block, and the `make_job` resolution that maps- the boolean to the curated repo id.--## 16. `required_files.json` schema--Single source of truth for "what counts as installed." Read by:-- `pinokio.js` → menu rendering (Install vs Resume Install vs Start)-- `mlx_ltx_panel.py` → `/status` and per-job validation-- (indirectly) `install.js` and `download_q8.js` — their `hf download`- paths must match the `local_dir` declared here.--```jsonc-{- "_comment": "...",- "repos": [- {- "key": "q4" | "gemma" | "q8",- "kind": "base" | "optional", // "base" = blocks Start, "optional" = unlock features- "name": "human-readable label",- "blurb": "what this is for",- "repo_id": "<hf-org/repo>", // for `hf download`- "local_dir": "mlx_models/<repo-name>", // relative to install root- "size_gb": 25, // approximate, displayed in UI- "files": [ // every required file inside local_dir- "transformer-distilled.safetensors",- "connector.safetensors",- ...- ]- }- ],- "env": {- "marker_paths": [ // OR-checked; first existing path wins- "ltx-2-mlx/env/pyvenv.cfg", // canonical Pinokio install- "ltx-2-mlx/.venv/pyvenv.cfg", // alternate manual layouts- ...- ]- },- "min_size_bytes": 1024 // file must be >= this to count as "complete"-}-```--When the upstream model layout changes, **edit this file first**. The-menu, panel, and install scripts auto-pick up the change.--## 17. Helper protocol (panel ↔ helper)--The helper subprocess (`mlx_warm_helper.py`) speaks newline-delimited-JSON over stdin/stdout. The panel's `WarmHelper` class wraps it.--### Actions panel sends--```jsonc-// Generation request (T2V, I2V, I2V-clean-audio)-{ "action": "generate",- "id": "<job-id>",- "params": { "mode": "t2v"|"i2v"|"i2v_clean_audio",- "prompt": "...", "output_path": "...",- "height": 704, "width": 1280, "frames": 121,- "seed": 42, "steps": 8, "image": "<path>" } }--// Extend request-{ "action": "extend",- "id": "<job-id>",- "params": { "video_path": "...", "prompt": "...",- "extend_frames": 5, "extend_direction": "after",- "extend_steps": 12, "cfg_scale": 1.0,- "output_path": "...", "seed": 42 } }--// Keyframe (FFLF)-{ "action": "keyframe", "id": "...", "params": {...} }--// Prompt rewrite (Gemma)-{ "action": "enhance", "prompt": "..." }-```--### Events helper emits--```jsonc-{ "event": "ready", "model": "<path>", "low_memory": true } // on startup-{ "event": "log", "line": "free-form panel log line" } // any informational message-{ "event": "done", "id": "...", "output": "<path>",- "elapsed_sec": 33.6, "seed_used": 42 }-{ "event": "error", "id": "...", "error": "<message>",- "trace": "<python traceback>" }-{ "event": "exit", "reason": "python_normal_exit"- | "sigterm(15)"- | "sigterm(2)"- | "idle" } // graceful exit-{ "event": "enhanced", "original": "...", "enhanced": "..." } // /prompt/enhance result-```--**No exit event + stdout pipe closes = non-graceful death** (SIGKILL,-SIGSEGV, SIGABRT, SIGBUS). The panel inspects `proc.poll()` returncode-and translates the negative signal number to a name in the user-facing-error.--### Step breadcrumb log lines--The helper emits these inside `generate`/`extend` actions so the panel-can localize a silent death to a phase:--```-step:get_pipe kind=t2v-step:get_pipe done-step:generate_and_save mode=i2v 1280x704 121f steps=8-step:generate_and_save done-```--When a job dies silently, the panel log's last `step:*` line tells you-which phase (pipeline init vs actual generation).--## 18. Memory budget model--Concrete weight sizes (HuggingFace repo metadata, Q4 unless noted):--| Component | Size on disk | Resident peak (rough) | Notes |-|---|---|---|---|-| `transformer-distilled.safetensors` (Q4) | 10.54 GiB | ~12 GiB | T2V/I2V "DiT" |-| `transformer-dev.safetensors` (Q4) | 11.3 GiB (rough) | ~13 GiB | Used by Extend |-| `transformer-dev.safetensors` (Q8) | 19.18 GiB | ~22 GiB | High-quality + FFLF |-| `connector.safetensors` (Q4) | 5.91 GiB | ~6 GiB | Lives inside `feature_extractor` |-| `Gemma-3-12B-it-4bit` | 7.48 GiB | ~9 GiB | Text encoder, freed before DiT load |-| `vae_decoder.safetensors` | 814 MB | ~1 GiB | Used at the end of generation |-| `vae_encoder.safetensors` | 638 MB | ~800 MB | I2V only (encodes input image) |-| `audio_vae.safetensors` | 107 MB | ~150 MB | |-| `vocoder.safetensors` | 258 MB | ~350 MB | |--Peak during T2V Standard 1280×704 (with patches applied): roughly-DiT + active activations + decoder ≈ 20–22 GiB resident. Without-the patches it's 26+ GiB and OOMs on memory-pressured 64 GB Macs.--Peak during I2V is +800 MB for the encoder pass + ~6 GiB for the-connector that lingered in `feature_extractor` (Patches 3 + 4 fix-the lingering — see §5).--macOS `jetsam` is the OOM killer. It enforces memory pressure based-on a percentage that varies with paging activity, swap usage, and other-heuristics — there is no fixed "kill at X% pressure" rule. Closing-Chrome, Slack, iOS Simulator before a Standard render is the safest-single thing a user can do.--## 19. Debug recipes--### "Helper died silently — where did it die?"--```bash-# Scan the panel log for the last step:* breadcrumb-curl -s http://127.0.0.1:8198/status | python3 -c \- "import json,sys; [print(l) for l in json.load(sys.stdin)['log'] if 'step:' in l][-3:]"-```--If last step is `step:get_pipe` → death during pipeline init (model-load). Likely OOM or weight-corrupt. If last step is-`step:generate_and_save` → death during gen (denoise / VAE decode).-Likely OOM peak or Metal kernel issue.--For non-OOM crashes, check `~/Library/Logs/DiagnosticReports/python3.11_*.crash`-or `*.ips` from around the crash time.--### "Panel says port 8198 already in use"--```bash-lsof -nP -iTCP:8198 -sTCP:LISTEN # who has it-pkill -f mlx_ltx_panel.py # nuke all panels-```--Common cause: a previous manual `python mlx_ltx_panel.py` is still-running. Pinokio's start.js can't take over a port someone else owns.-**Don't spawn the panel manually unless you're going to kill it before-handing back.**--### "I edited mlx_ltx_panel.py but the UI hasn't changed"--The HTML/JS is baked into the Python file as a template string. The-running panel served the old version. Restart it:-- via Pinokio: Stop → Start-- or via shell: `pkill -f mlx_ltx_panel.py` then click Start in Pinokio--Then **hard-refresh the browser** (Cmd+Shift+R) to bypass JS cache.--### "I edited the helper or a patch but nothing changed"--```bash-# 1. Re-apply patches (idempotent, applies to site-packages)-cd ~/pinokio/api/phosphene.git-./ltx-2-mlx/env/bin/python3.11 patch_ltx_codec.py--# 2. Restart the helper subprocess (panel keeps running)-curl -X POST http://127.0.0.1:8198/helper/restart--# 3. Submit a test job-curl -X POST http://127.0.0.1:8198/run \- -d "mode=t2v" -d "prompt=quick test" \- -d "width=512" -d "height=288" -d "frames=49" \- -d "steps=8" -d "quality=draft"--# 4. Wait for done-until [ "$(curl -s http://127.0.0.1:8198/status | python3 -c 'import json,sys;d=json.load(sys.stdin);print(0 if d.get(\"current\") else 1)')" = "1" ]; do sleep 5; done-```--### "How loud / clean is the audio in this clip?"--```bash-FFPROBE="/Volumes/Topaz Video AI 4.0.6/Topaz Video AI.app/Contents/MacOS/ffprobe"-FFMPEG="/Volumes/Topaz Video AI 4.0.6/Topaz Video AI.app/Contents/MacOS/ffmpeg"-"$FFMPEG" -i out.mp4 -af "volumedetect" -f null - 2>&1 | grep -E "max_volume|mean_volume"-```--Healthy on dialogue: `max_volume` between -2 and -15 dB. Below -25 dB-peak is the regression fingerprint — usually means a wrong mlx version.--### "Re-encode for X / web (yuv420p, faststart, 8 Mbps)"--```bash-"$FFMPEG" -y -i in.mp4 \- -c:v h264_videotoolbox -profile:v high -pix_fmt yuv420p \- -b:v 8M -maxrate 12M -bufsize 16M \- -movflags +faststart \- -c:a aac -b:a 192k \- out_x.mp4-```--X (Twitter) rejects yuv444p with a misleading "aspect ratio" error.-Use this recipe for any social upload.--### "Stuck queue / can't cancel"--```bash-curl -X POST http://127.0.0.1:8198/stop # cancel current job-curl -X POST http://127.0.0.1:8198/queue/clear # empty the queue-curl -X POST http://127.0.0.1:8198/helper/restart # nuke the helper-```--### "Force re-download a corrupt model file"--```bash-# Delete the file in question, then click Install in Pinokio (resumes-# only the missing files; intact ones are skipped via `hf` hash check).-rm ~/pinokio/api/phosphene.git/mlx_models/ltx-2.3-mlx-q4/<bad-file>.safetensors-# Then in Pinokio: Phosphene → Install (or Resume Install)-```--### "Force a different hardware tier for testing"--```bash-LTX_TIER_OVERRIDE=base ./run_panel.sh # pretend this Mac is < 48 GB-```--`run_panel.sh` actively unsets this on every run unless explicitly set-on the command line — there's a hard-learned bug history about leaked-overrides bleeding across panel restarts.--### "What does `/status` actually return?"--```bash-curl -s http://127.0.0.1:8198/status | python3 -m json.tool | head -60-```--Top-level keys: `running`, `paused`, `current`, `queue`, `history`,-`log`, `tier`, `memory`, `helper`, `download`, `repos_ready`,-`repos_total`, `base_available`, `q8_available`, `q8_missing`,-`comfy_pids`, `outputs`, `server_now`, `avg_elapsed_sec`.--## 20. Patch authoring template (`patch_ltx_codec.py`)--When adding a new patch:--```python-# ---- Patch N: <one-line summary> ---------------------------------------------# Why: <symptom + root cause + why this fix>. Reference upstream-# file:line and any external context (commit SHA, GitHub issue, paper).-PATCH_<NAME>_OLD = '''<exact text expected to be in the upstream file>'''-PATCH_<NAME>_NEW = '''<replacement text — include a `# PATCHED (LTX23MLX): ...` comment>'''-```--Then in `main()`:--```python-target = _find("<relative/path/to/upstream.py>")-outcome = apply_patch(- target, PATCH_<NAME>_OLD, PATCH_<NAME>_NEW,- marker="<unique substring that proves NEW is on disk>",- label="<NAME>",- upgrade_marker="<optional: substring that distinguishes a newer NEW from an older NEW>",-)-# If this patch is optional (don't fail install on drift):-if outcome in (OUTCOME_DRIFT, OUTCOME_MISSING):- print(" [<NAME>] note: ...", file=sys.stderr)- outcome = OUTCOME_ALREADY-outcomes.append(("<NAME>", outcome))-```--Conventions:-- `marker` is a substring **inside** `PATCH_<NAME>_NEW` that's unlikely- to appear in the unpatched file. Convention: include- `PATCHED (LTX23MLX...)` somewhere in the new text.-- `upgrade_marker` lets you ship a v2 of a patch without forcing users- to nuke their venv. Example: codec patch's `+faststart` was added- later; `upgrade_marker="+faststart"` distinguishes v1 (no faststart)- from v2 and rewrites the line in place.-- Patches that target an upstream code structure that may legitimately- vanish (e.g. an internal helper that gets refactored away) should be- marked **optional** in main() so install warns instead of fails.--## 21. Glossary--- **LTX 2.3** — Lightricks' latent text-to-video diffusion model with- joint audio synthesis. Trained on a large internal dataset.-- **MLX** — Apple's open-source ML array framework, native to Metal.- Like JAX/PyTorch but built for Apple Silicon. Not a torch shim.-- **Q4 / Q8** — Quantization levels of the LTX 2.3 weights. Q4 is- 4-bit quantized (smaller, faster, slightly lower quality), Q8 is- 8-bit (larger, slower, sharper detail). Phosphene defaults to Q4;- Q8 is an optional upgrade.-- **DiT** — Diffusion Transformer. The main denoising network. The- largest single weight tensor.-- **VAE** — Variational Autoencoder. Encodes pixels ↔ latents. The- encoder is used in I2V (encodes the input image); the decoder is- used at the end of every generation.-- **FFLF** — First-Frame / Last-Frame interpolation. Provide two images,- the model generates the motion between them. Requires Q8.-- **BWE** — Bandwidth Extension. Inside the audio vocoder, upsamples- a low-rate signal to full audio rate (108 sequential BigVGAN convs).-- **Vocoder** — Converts mel-spectrogram features to PCM audio. The- final stage of LTX's audio pipeline.-- **TeaCache** — A diffusion-step caching technique that skips- redundant computation when consecutive sigmas are similar. Used in- the High quality two-stage path.-- **jetsam** — macOS' kernel-level memory pressure killer. Sends SIGKILL- to processes when system memory gets tight. The dominant cause of- silent helper death on Macs at full RAM.-- **moov atom** — The MP4 metadata index. With `+faststart` it lives- at the front of the file (allows progressive playback / instant- thumbnails); without, it lives at the end (browser must download- the whole file before showing the first frame).-- **Pinokio** — One-click installer + runtime for AI apps on Mac/Linux/Windows.- Phosphene ships as a Pinokio app. cocktailpeanut authored Pinokio.-- **mrbizarro** — The GitHub handle that authors this repo. Use this,- always, in commits and metadata.--## 22. Open questions & v1.1 candidates--Tracked here so they don't get lost between sessions.--- **mlx 0.31.1 → 0.31.2 audio bisect** — find the exact MLX commit- that attenuates LTX 2.3's vocoder by 22 dB. File upstream issue.-- **A2V (audio-to-video) mode** — `ltx-2-mlx` upstream has- `a2vid_two_stage.py` and `a2vid_two_stage_hq.py` but the panel- doesn't expose them. UI work + helper plumbing required.-- **In-app HF token field** — currently we recommend `hf auth login`- in Terminal. A panel-side settings field that writes- `~/.cache/huggingface/token` would be friendlier for non-CLI users.-- **Pre-flight RAM advisory** — before submitting a heavy job, warn- if `s.memory.swap_gb > 8` or `pressure_pct > 75`. Suggest Draft.-- **Audio mode dropdown** — supersede the binary "No music" toggle- with: With music / Voice + ambient / Sound effects / Silent. Each- option appends/prepends a different prompt constraint.-- **Deeper helper breadcrumbs** — current `step:*` markers cover the- outer call. Adding `image_encode_start`, `text_encode_done`,- `denoise_start`, `decode_start` inside the pipeline (via wrapper- patches) would localize silent deaths down to the phase.-- **Q8 path validation** — Q8 generation has not been end-to-end- tested on a 64 GB Mac since the codec/I2V patches landed. Schedule- a 25 GB Q8 download + High quality T2V test.-- **FFLF validation** — same as above. Requires Q8.--## 23. External references--- LTX-2 paper / model card: https://huggingface.co/Lightricks-- MLX framework: https://github.com/ml-explore/mlx-- `ltx-2-mlx` (the MLX port we wrap): https://github.com/dgrauet/ltx-2-mlx-- Pinokio runtime: https://pinokio.computer-- Q4 weights: `dgrauet/ltx-2.3-mlx-q4` on HF-- Q8 weights: `dgrauet/ltx-2.3-mlx-q8` on HF-- Gemma text encoder: `mlx-community/gemma-3-12b-it-4bit` on HF-------# Generic Pinokio Development Guide--(Everything below this line is the upstream Pinokio scaffolding shipped-with every new project. Useful as a reference when working with the-launcher scripts but the Phosphene-specific rules above take-precedence on conflict.)--# Development Guide for Pinokio Projects--## Non-Negotiable Execution Workflow--To guarantee every contribution follows this guide precisely, obey this checklist **before any edits** and **again before finalizing**. Do not skip or reorder.-1. **AGENTS Snapshot:** Re-open this file and write down (in your working notes or response draft) the exact sections relevant to the requested task. No work begins until this snapshot exists.-2. **Destination Resolution:** Before creating or editing any Pinokio launcher files, resolve `PINOKIO_HOME` to an absolute path and record the intended destination root. If running outside Pinokio's own managed runtime, resolve in this order: `~/.pinokio/config.json` `home`, then `GET http://127.0.0.1:42000/pinokio/home` and use its `path` value, and if loopback is unreachable but `access` exists in `~/.pinokio/config.json`, retry the same request against `<protocol>://<host>:<port>/pinokio/home`, then the `PINOKIO_HOME` environment variable. If `PINOKIO_HOME` is still unresolved, stop and ask the user. Never silently use the current workspace as the launcher destination.-3. **Example Lock-in:** Identify the closest matching script in `~/pinokio/prototype/system/examples`. Record its path and keep it open while editing. Every launcher change must mirror that reference unless the user explicitly instructs otherwise.-4. **Pre-flight Checklist:** Convert the applicable rules from this document and `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md into a task-specific checklist (install/start/reset/update structure, regex patterns, menu defaults, log checks, destination path, etc.). Confirm each item is ticked **before** making changes.-5. **Mid-task Verification:** Any time you touch a Pinokio script, cross-check the corresponding example line to ensure syntax and structure match. Document the reference (example path + line) in your reasoning.-6. **Exit Checklist:** Before responding to the user, revisit the pre-flight checklist and explicitly confirm every item is satisfied. If anything diverges from the example or these rules, fix it first.--If any step cannot be completed, stop immediately and ask the user how to proceed. These six steps are mandatory for every session.--### Critical Pattern Lock: Capturing Web UI URLs--When writing `start.js` (or any script that needs to surface a web URL for a server):--1. **Always copy the capture block from an example such as `system/examples/mochi/start.js`.**-```javascript-on: [{- event: "/(http:\\/\\/[0-9.:]+)/",- done: true-}]-```--2. **Set the local variable using the captured match exactly as below (The regex capture object is passed in as `input.event`, so need to use the index 1 inside the parenthesis):**-```javascript-{- method: "local.set",- params: {- url: "{{input.event[1]}}"- }-}-```--3. Always try to come up with the most generic regex.-4. During the exit checklist, explicitly confirm that the `url` local variable is set via `local.set` API by using the captured regex object as passed in as `input.event` from the previous `shell.run` step.--Deviation from this pattern requires written approval from the user.--- Make sure to keep this entire document and `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md in memory with high priority before making any decision. Pinokio is a system that makes it easy to write launchers through scripting by providing various cross-platform APIs, so whenever possible you should prioritize using Pinokio API over lower level APIs.-- When writing pinokio scripts, ALWAYS check the examples folder (in ~/pinokio/prototype/system/examples folder) to see if there are existing example scripts you can imitate, instead of assuming syntax.-- When implementing pinokio script APIs and you cannot infer the syntax just based on the examples, always search the API documentation `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md to use the correct syntax instead of assuming the syntax.-- When trying to fix something or figure out what's going on, ALWAYS start by checking the `logs` folder before doing anything else, as mentioned in the "Troubleshooting with Logs" section.-- Finally, make sure to ALWAYS follow all the items in the "best practices" section below.--## Determine User Intent-If the initial prompt is simply a URL and nothing else, check the website content and determine the intent, and ask the user to confirm. For example a URL may point to--1. A Tutorial: the intent may be to implement a demo for the tutorial and build a launcher.-2. A Demo: the intent may be a 1-click launcher for the demo-3. Open source project: the intent may be a 1-click launcher for the project-4. Regular website: the intent may be to clone the website and a launcher.-5. There can be other cases, but try to guess.--## Working With Launchers--Apply this section only when the task is to create, modify, debug, review, or document a Pinokio launcher project.--If the request is not about launcher work, do not force an app-launcher vs plugin-launcher decision.--When the task does involve launcher work, first determine whether the request is for an app launcher or a plugin launcher. These are separate project types and must not be mixed.--### Mandatory Destination Resolution-- Before creating, editing, or moving any launcher files, resolve `PINOKIO_HOME` to an absolute path.-- If running outside Pinokio's own managed runtime, resolve `PINOKIO_HOME` in this order:- 1. `~/.pinokio/config.json` -> `home`- 2. `GET http://127.0.0.1:42000/pinokio/home` -> `path`- 3. If loopback is unreachable but `access` exists in `~/.pinokio/config.json`, retry the same request against `<protocol>://<host>:<port>/pinokio/home`- 4. `PINOKIO_HOME` environment variable-- Normalize the resolved value to an absolute path before using it.-- If neither source yields a valid `PINOKIO_HOME`, stop immediately and ask the user how to proceed. Do not guess. Do not silently fall back to the current workspace.-- If the current workspace is outside the resolved `PINOKIO_HOME/api` and `PINOKIO_HOME/plugin` trees, treat the current workspace only as source material, reference material, or evidence. Do not create the launcher in that workspace.-- Before any file creation, record and verify the exact target path:- - app launcher: `PINOKIO_HOME/api/<unique_name>`- - plugin launcher: `PINOKIO_HOME/plugin/<unique_name>`-- If the unique folder name is not obvious, ask the user before creating the target folder.--### 1. App launchers-- App launchers must live under `PINOKIO_HOME/api/<unique_name>`.-- App launchers are usually project-local launchers that manage one app in its own launcher/app folder.-- If you are already inside the target app launcher folder, build in that folder.-- If you are not already inside an app launcher folder, create a new folder under `PINOKIO_HOME/api/<unique_name>`.-- If the folder name is not obvious from the project or the user has not provided one, ask the user to confirm the folder name before creating it.-- Do not place app launchers under `PINOKIO_HOME/plugin`.--### 2. Plugin launchers-- Plugin launchers must live under `PINOKIO_HOME/plugin/<unique_name>`.-- Plugin launchers are reusable shared tools that are installed once and then used across many different folders.-- Standalone plugin launchers should keep `path: "plugin"` in the root `pinokio.js` so Pinokio installs them into `PINOKIO_HOME/plugin`.-- If you are already inside the target plugin launcher folder, build in that folder.-- If you are not already inside a plugin launcher folder, create a new folder under `PINOKIO_HOME/plugin/<unique_name>`.-- If the folder name is not obvious from the project or the user has not provided one, ask the user to confirm the folder name before creating it.-- Do not place plugin launchers under `PINOKIO_HOME/api`.-- When a plugin is meant to operate on the user's current project, its `run` step should target the caller's folder with `{{args.cwd}}` instead of the plugin folder itself.--### 3. Apply structure rules only after choosing the launcher type-- App launchers and plugin launchers are peers. Do not treat a plugin launcher as a special case of an app launcher, or vice versa.-- Decide the launcher type and destination folder first, then apply the project structure and script rules below.--## Project Structure--Pinokio projects normally follow a standardized structure with app logic separated from launcher scripts:--Pinokio projects follow a standardized structure with app logic separated from launcher scripts:--```-project-root/-├── app/ # Self-contained app logic (can be standalone repo)-│ ├── package.json # Node.js projects-│ ├── requirements.txt # Python projects-│ └── ... # Other language-specific files-├── README.md # Documentation-├── install.js # Installation script-├── start.js # Launch script-├── update.js # Update script (for updating the scripts and app logic to the latest)-├── reset.js # Reset dependencies script-├── pinokio.js # UI generator script-└── pinokio.json # Metadata (title, description, icon)-```--- Keep app code in `/app` folder only (never in root)-- Store all launcher files in project root (never in `/app`)-- `/app` folder should be self-contained and publishable---The only exceptions are serverless web apps---purely frontend only web applications that do NOT have a server component and connect to 3rd party API endpoints--in which case the folder structure looks like the following (No need for launcher scripts since the index.html will automatically launch. The only thing needed is the metadata file named pinokio.json):--```-project-root/-├── index.html # The serverless web app entry point-├── ...-├── README.md # Documentation-└── pinokio.json # Metadata (title, description, icon)-```--IMPORTANT: ALWAYS try to follow the best practices in the examples folder (~/pinokio/prototype/system/examples) instead of trying to come up with your own structure. The examples have been optimized for the best user experience.--## Launcher Project Working Directory--- The project working directory for a script is always the same directory as the script location.-- For example, when you run `shell.run` API inside `pinokio/start.js`, the default path for shell execution is `pinokio`.-- If the launcher files are in the project root path, then the default path for shell execution is the project root.-- Therefore, it is important to specify the correct `path` attribute when running `shell.run` API commands.--Example: in the following project structure:--```-project-root/-├── pinokio/ # Pinokio launcher folder-│ ├── start.js # Launch script-│ ├── pinokio.js # UI generator script-│ └── pinokio.json # Metadata (title, description, icon)-└─── backend/- ├── requirements.txt # App dependencies- └── app.py # App code-```--The `pinokio/start.js` should use the correct path `../backend` as the `path` attribute, as follows:--```-{- run: [{- ...- }, {- method: "shell.run",- params: {- message: "python app.py",- venv: "env",- path: "../backend"- }- }, {- ...- }]-}-```--## Development Workflow--### 1. Understanding the Project-- Check `SPEC.md` in project root. If the file exists, use that to learn about the project details (what and how to build)-- If no `SPEC.md` exists, build based on user requirements-### 2. Modifying Existing Launcher Projects-If we are starting with existing launcher script files, work with the existing files instead of coming up with your own.-- **Preserve existing functionality:** Only modify necessary parts-- **Don't touch working scripts:** Unless adding/updating specific commands-- **Follow existing conventions:** Match the style and structure already present-### 3. Try to adopt from examples as much as possible-- If starting from scratch, first determine what type of project you will be building, and then check the examples folder (~/pinokio/prototype/system/examples) to see if you can adopt them instead of coming up everything from scratch.-- Even if there are no relevant examples, check the examples to get inspiration for how you would structure the script files even if you have to write from scratch.-### 4. Writing from scratch as a last resort-If there are relevant examples to adopt from, write the scripts from scratch, but just make sure to follow the requirements in the next section.-### 5. Debugging-When the user reports something is not working, ALWAYS inspect the logs folder to get all the execution logs. For more info on how this works, check the "Troubleshooting with Logs" section below.--## Script Requirements--### 1. 1-click launchable-- The main purpose of Pinokio is to provide an easy interface to invoke commands, which may include launching servers, installing programs, etc. Make sure the final product provides ways to install, launch, reset, and update whatever is needed.--### 2. Write Documentation-- ALWAYS write a documentation. A documentation must be stored as `README.md` in the project root folder, along with the rest of the pinokio launcher script files. A documentation file must contain:- - What the app does- - How to use the app- - API documentation for programmatically accessing the app's main features (Javascript, Python, and Curl)--## Types of launchers-## 1. Launching servers-- When an app requires launching a server, here are the commonly used scripts:- - `install.js`: a script to install the app- - `start.js`: a script to start the app- - `reset.js`: a script to reset all the dependencies installed in the `install.js` step. used if the user wants to restart from scratch- - `update.js`: a script to update the launcher AND the app in case there are new updates. Involves pulling in the relevant git repositories installed through `install.js` (often it's the script repo and some git repositories cloned through the install steps if any)- - `pinokio.js`: the launcher script that ties all of the above scripts together by providing a UI that links to these scripts.- - `pinokio.json`: For metadata--Here's a basic server launcher script example (`start.js`). Unless there's a special reason you need to use another pattern, this is the most recommended pattern. Use this or adopt it as needed, but NEVER try something else unless there's a good reason you should not take this approach:--```javascript-module.exports = {- // By setting daemon: true, the script keeps running even after all items in the `run` array finishes running. Mandatory for launching servers, since otherwise the shells running the server process will get killed after the scripts finish running.- daemon: true,- run: [- {- // The "shell.run" API for running a shell session- method: "shell.run",- params: {- // Edit 'venv' to customize the venv folder path- venv: "env",- // Edit 'env' to customize environment variables (see documentation)- env: { },- // Edit 'path' to customize the path to start the shell from- path: "app",- // Edit 'message' to customize the commands, or to run multiple commands- message: [- "python app.py",- ],- on: [{- // The regular expression pattern to monitor.- // Whenever each "event" pattern occurs in the shell terminal, the shell will return,- // and the script will go onto the next step.- // The regular expression match object will be passed on to the next step as `input.event`- // Useful for capturing the URL at which the server is running (in case the server prints some message about where the server is running)- "event": "/(http:\/\/\\S+)/",-- // Use "done": true to move to the next step while keeping the shell alive.- // Use "kill": true to move to the next step after killing the shell.- "done": true- }]- }- },- {- // This step sets the local variable 'url'.- // This local variable will be used in pinokio.js to display the "Open WebUI" tab when the value is set.- method: "local.set",- params: {- // the input.event is the regular expression match object from the previous step- // In this example, since the pattern was "/(http:\/\/\\S+)/", input.event[1] will include the exact http url match caputred by the parenthesis.- // Therefore setting the local variable 'url'- url: "{{input.event[1]}}"- }- }- ]-}-```--## 2. Launching serverless web apps--- In case of purely static web apps WITHOUT servers or backends (for example an HTML based app that connects to 3rd party servers--either remote or localhost), we do NOT need the launcher scripts.-- In these cases, simply include `index.html` in the project root folder and everything should automatically work. No need for any of the pinokio launcher scripts. (Do-- You still need to include the metadata file so they show up properly on pinokio:- - `pinokio.json`: For metadata--## 3. Launching quick scripts without web UI--- In many cases, we may not even need a web UI, but instead just a simple way to run scripts.-- This may include TUI (Terminal User Interface) apps, a simple launcher-- In these cases, all we need is the launcher file `pinokio.js`, which may link to multiple scripts. In this case, there are no web apps (no serverless apsp, no servers), but instead just the default pinokio launcher UI that calls a bunch of scripts.-- Here are some examples:- - A pinokio script to toggle the desktop theme between dark and light- - Write some code (python or javascript or whatever)- - Write a `toggle.js` pinokio script that executes the code- - Write a `pinokio.js` launcher script to create a sidebar UI that displays the `toggle.js` so the user can simply click the "toggle" button to toggle back and forth between desktop themes- - A pinokio script to fetch some file- - Write some code (python or javascript or whatever)- - Write a `fetch.js` pinokio script that executes the code- - Write a `pinokio.js` launcher script to create a sidebar UI that displays the `fetch.js` so the user can simply click the "fetch" button to fetch some data.-- You still need to include the metadata file so they show up properly on pinokio:- - `pinokio.json`: For metadata--## API--This section lists all the script APIs available on Pinokio. To learn the details of how they are used, you can:-1. Check the examples in the ~/pinokio/prototype/system/examples folder-2. Read the `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md further documentation on the full syntax--### Script API--These APIs can be used to describe each step in a pinokio script:-- shell.run: run shell commands-- input: accept user input-- filepicker: accept file upload-- fs.write: write to file-- fs.read: read from file-- fs.copy: copy files-- fs.download: download files-- fs.link: create a symbolic link (or junction on windows) for folders-- fs.open: open the system file explorer at a given path-- fs.cat: print file contents-- jump: jump to a specific step-- local.set: set local variables for the currently running script-- json.set: update a json file-- json.rm: remove keys from a json file-- json.get: get values from a json file-- log: print to the web terminal-- net: make network requests-- notify: display a notification-- script.download: download a script from a git uri-- script.start: start a script-- script.stop: stop a script-- script.return: return values if the current script was called by a caller script, so the caller script can utilize the return value as `input`-- web.open: open a url in web browser-- hf.download: huggingfac-cli download API-### Template variables-The following variables are accessible inside template expressions (example `{{args.command}` in scripts, resulting in dynamic behaviors of scripts:-- input: An input is a variable that gets passed from one RPC call to the next-- args: args is the parameter object that gets passed into the script (via pinokio.js `params`). Unlike `input` which takes the value passed in from the immediately previous step, `args` is a global value that is the same through out the entire script execution.-- local: local variable object that can be set with `local.set` API-- self: refers to the script file itself (which is JSON or JavaScript). For example if `start.js` that's currently running has `daemon: true` set, `{{self.daemon}}` will evaluate to true.-- uri: The current script uri-- port: The next available port. Very useful when you need to launch an app at a specific port without port conflicts.-- cwd: The current script execution folder path-- platform: The current operating system. May be one of the following: `darwin`, `win32`, `linux`-- arch: The current system architecture. May be one of the following: x32, x64, arm, arm64, s390, s390x, mipsel, ia32, mips, ppc, ppc64-- gpus: array of available GPUs on the machine (example: `['apple']`, `['nvidia']`)-- gpu: the first available GPU (example: `nvidia`)-- current: The current variable points to the index of the currently executing instruction within the run array.-- next: The next variable points to the index of the next instruction to be executed. (null if the current instruction is the final instruction in the run array)-- envs: You can access the environment variables of the currently running process with envs object.-- which: Check whether a command exists and return its absolute path (example: `{{which('winget')}}`). This is the correct way to resolve command paths inside reproducible Pinokio scripts, including custom shell selection such as `shell: "{{which('bash')}}"`. If you are outside a Pinokio-managed shell and only need to inspect Pinokio's environment manually, use `pterm which <command>`, but do NOT copy that user-specific absolute path into launcher scripts.-- exists: Check whether a file or folder exists at the specified relative path (example: `"when": "{{!exists('app')}}"`). Can be used with the `when` attribute to determine a path's existence and trigger custom logic. Use relative paths and it will resolve automatically to the current execution folder.-- running: Check whether a script file is running (example: `"when": "{{!running('start.js')}}"`). Can be used with the `when` attribute to determine a path's existence and trigger custom logic. Use relative paths and it will resolve automatically to the current execution folder.-- os: Pinokio exposes the node.js os module through the os variable.-- path: Pinokio exposes the node.js path module through the os variable (example: `{{path.resolve(...)}}`--## System Capabilities-### Package Management (Use in Order of Preference)-The following package managers come pre-installed with Pinokio, so whenever you need to install a 3rd party binary, remember that these are available. Also, you can assume these are available and include the following package manager commands in Pinokio scripts:-1. **UV** - For Python packages (preferred over pip)-2. **NPM** - For Node.js packages-3. **Conda** - For cross-platform 3rd party binaries-4. **Brew** - Mac-only fallback when other options unavailable-5. **Git** - Full access to git is available.-6. **Bun** - For managing bun packages-**Important:** Include all install commands in the install script for reproducibility.-### HTTPS Proxy Support-- All HTTP servers automatically get HTTPS endpoints-- Convention: `http://localhost:<PORT>` → `https://<PORT>.localhost`-- Full proxy list available at: `http://localhost:2019/config/`-### Pterm Features:-- **Clipboard Access:** Read from or Write to system clipboard via pinokio Pterm CLI (`pterm clipboard` command.)-- **Notifications:** Send desktop alerts via pinokio pterm CLI (`pterm push` command.)-- **Script Testing:** Run launcher scripts via pinokio pterm CLI (`pterm start` command.)-- **File Selection:** Use built-in filepicker for user file/folder input (`pterm filepicker` command.)-- **Command Path Resolution:** Inspect the absolute path of any command as seen by Pinokio via `pterm which <command>`. Use this for debugging or external local tooling, especially when a helper process did not inherit Pinokio's `PATH`, for example `pterm which bash` on Windows. Do NOT hardcode the returned absolute path into launcher scripts; use `which()` or `kernel.which()` in the script itself instead.-- **Git Operations:** Clone repositories, push to GitHub-- **GitHub Integration:** Full GitHub CLI support (`gh` commands)--## Troubleshooting with Logs-Pinokio stores the logs for everything that happened in terminal at the following locations, so you can make use of them to determine what's going on:--### Log Structure-In case there is a `pinokio` folder in the project root folder, you should be able to find the logs folder here:--```-pinokio/-└── logs/ # Direct user interaction logs- ├── api/ # Launcher script logs (install.js, start.js, etc.)- ├── dev/ # AI coding tool logs (organized by tool)- └── shell/ # Direct user interaction logs-```--Otherwise, the `logs` folder should be found at project root:--```-logs/-├── api/ # Launcher script logs (install.js, start.js, etc.)-├── dev/ # AI coding tool logs (organized by tool)-└── shell/ # Direct user interaction logs-```--### Log File Naming-- Unix timestamps for each session-- Special "latest" file contains most recent session logs-- **Default:** Use "latest" files for current issues-- **Historical:** Use timestamped files for pattern analysis and the full history.--## Best practices-### 0. Always reference the logs when debugging-- When the user asks to fix something, ALWAYS check the logs folder first to check what went wrong. Check the "Troubleshooting with Logs" section.-### 1. Shell commands for launching programs-- Launch flags related- - Try as hard as possible to minimize launch flags and parameters when launching an app. For example, instead of `python app.py --port 8610`, try to do `python app.py` unless really necessary. The only exception is when the only way to launch the app is to specify the flags.-- Launch IP related- - Always try to find a way to launch servers at 127.0.0.1 or localhost, often by specifying launch flags or using environment variables. Some apps launch apps at 0.0.0.0 by default but we do not want this.-- Launch Port related- - In case the app itself automatically launches at the next available port by default (for example Gradio does this), do NOT specify port, since it's taken care of by the app itself. Always try to minimize the amount of code.- - If the install instruction says to launch at a specific port, don't use the hardcoded port they suggest since there's a risk of port conflicts. Instead, use Pinokio's `{{port}}` template expression to automatically get the next available port.- - For example, if the instruction says `python app.py --port 7860`, don't use that hardcoded port since there might be another app running at that port. Instead, automatically assign the next available port like this: `python app.py --port {{port}}`- - Note that the `{{port}}` expression always returns the next immediately available port for each step, so if you have multiple steps in a script and use `{{port}}` in multiple steps, the value will be different. So if you want to launch at the next available port and then later reuse that port, you will need to first use `{{port}}` to get the next available port, and save the value in local variable using `local.set`, and then use the `{{local.<variable_name>}}` expression later.-### 2. shell.run API-- When writing `shell.run` API requests, always use relative paths (no absolute paths) for the `path` field. For example, if you need to run a command from `app` folder, the `path` attribute should simply be `app`, instead of its full absolute path.-- If a launcher needs to use a command that Pinokio already provides, prefer resolving it with `{{which('command')}}` inside the script instead of assuming the command name will always be on `PATH`.-- Do NOT automatically avoid `bash`-based install commands on Windows. Pinokio's Windows environment includes `bash` through its bundled toolchain, so commands such as `curl -fsSL ... | bash` are acceptable when they run inside a Pinokio-managed shell and there is no simpler cross-platform alternative.-- If a Windows launcher needs to run the shell itself in bash instead of the default `cmd.exe`, set `shell: "{{which('bash')}}"` on the `shell.run` step.-- If a separate debugging process or external local tool did not inherit Pinokio's environment, you may use `pterm which <command>` to inspect what Pinokio would resolve. Do NOT turn that result into a hardcoded script path; for launcher scripts, always use `which()` or `kernel.which()` so the script stays reproducible across machines.-### 2. Package managers-- When installing python packages, try best to use `uv` instead of `pip` even if the install instruction says to use pip. Instead of `pip install -r requirements.txt`, you can simply use `uv pip install -r requirements.txt` for example. Even if the project's own README says use pip or poetry, first check if there's a way to use uv instead.-- When you need to install some global package, try to use `conda` as much as possible. Even on macs, `brew` should be only used if there are no `conda` options.-### 3. Minimal Always-- If you are starting with existing script files, before modifying, creating, or removing any script files, first look at `pinokio.js` to understand which script files are actually used in the launcher. The only script files used are the ones mentioned in the `pinokio.js` file. The `pinokio.js` file is the file that constructs the UI dynamically.-- Do not create a redundant script file that does something that already exists. Instead modify the existing script file for the feature. For example, do not create an `install.json` file for installation if `install.js` already exists. Instead, modify the `install.js` file.-- Pinokio accepts both JSON and JS script files, so when determining whether a script for a specific purpose already exists, check both JSON and JS files mentioned in the `pinokio.js` file. Do not create script files for rendundant purpose.-- When building launchers for existing projects cloned from a repository, try to stay away from modifying the project folder (the `~/pinokio/api/phosphene.git` folder), even if installations are failing. Instead, try to work around it by creating additional files in the launcher folder, and using those files IN ADDITION to the default project.- - The only exception when you may need to make changes to the project folder is when the user explicitly wants to modify the existing project. Otherwise if the purpose is to simply write a launcher, the app logic folder should never be touched.-- When running shell commands, take full advantage of the Pinokio `shell.run` API, which provides features like `env`, `venv`, `input`, `path`, `sudo`, `on`, etc. which can greatly reduce the amount of script code.- - Python apps: Always use virtual environments via `venv` attribute. This attribute automatically creates a venv or uses if it already exists.-### 4. Try to support Cross-platform as much as possible-- Use cross-platform shell commands only.-- This means, prefer to use commands that work on all platforms instead of the current platform.-- If there are no cross platform commands, use Pinokio's template expressions to conditionally use commands depending on `platform`, `arch`, etc.-- Also try to utilize Pinokio Pterm APIs for various cross-platform system features.-- If it is impossible to implement a cross platform solution (due to the nature of the project itself), set the `platform`, `arch`, and/or `gpu` attributes of the `pinokio.json` file to declare the limitation.-- Pinokio provides various APIs for cross-platform way of calling commonly used system functions, or lets you selectively run commands depending on `platform`, `arch`, etc.-### 5. Do not make assumptions about Pinokio API-- Do NOT make assumptions about which Pinokio APIs exist. Check the documentation.-- Do NOT make assumptions about the Pinokio API syntax. Follow the documentation.-### 6. Scripts must be able to replicate install and launch steps 100%-- The whole point of the scripts is for others to easily download and invoke them via Pinokio interface with one click. Therefore, do not assume the end user's system state, and make everything self-contained.-- When a 3rd party package needs to be installed, or a 3rd party repository needs to be downloaded, include them in the scripts.-### 7 Dynamic UI rendering-- The `pinokio.js` launcher script can change dynamically depending on the current state of the script execution. Which means, depending on what the file returns, it can determine what the sidebar looks like at any given moment of the script cycle.- - `info.exists(relative_path)`: The `info.exists` can be used to check whether a relative path (relative to the script root path) exists. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `info.running(relative_path)`: The `info.running` can be used to check whether a script at a relative path is currently running (relative to the script root path) exists. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `info.local(relative_path)`: The `info.local` can be used to return all the local variables tied to a script that's currently running. The `pinokio.js` file can determine which menu items to return based on this value at any given moment.- - `default`: set the `default` attribute on any menu item for whichever menu needs to be selected by default at a given step. Some example scenarios:- - during the install process, the `install.js` menu item needs to be set as the `default`, so it automatically executes the script- - when launching the `start.js` menu item needs to be set as the `default`, so it automatically executes the script- - after the app has launched, the `default` needs to be set on the web UI URL, so the user is sent to the actual app automatically.- - Check the examples in the ~/pinokio/prototype/system/examples folder to see how these are being used.-### 8. No need for stop scripts-- `pinokio.js` does NOT need a separate `stop` script. Every script that can be started can also be natively stopped through the Pinokio UI, therefore you do not need a separate stop script for start script-### 9. Writing launchers for existing projects-- When writing or modifying pinokio launcher scripts, figure out the install/launch steps by reading the project folder `app`.-- In most cases, the `README.md` file in the `~/pinokio/api/phosphene.git` folder contains the instructions needed to install and run the app, but if not, figure out by scanning the rest of the project files.-- Install scripts should work for each specific operating system, so ignore Docker related instructions. Instead use install/launch instructions for each platform.-### 10. Retrofitting an already-working setup-- Sometimes the user starts outside Pinokio, gets an app working through ad-hoc commands, and only later asks to turn that work into a Pinokio launcher.-- In this case, treat the current working setup and the successful session context as the highest priority source of truth. Do NOT restart from scratch if the app is already working.-- First capture the exact install and launch steps that already succeeded: cloned repositories, package manager commands, environment variables, model downloads, ports, working directories, helper scripts, and any fixes that were required.-- Then convert that knowledge into reproducible Pinokio scripts (`install.js`, `start.js`, `reset.js`, `update.js`, `pinokio.js`, `pinokio.json`) instead of telling the user to manually repeat the ad-hoc process.-- When the successful setup lives in a non-Pinokio folder, use that folder as evidence only. Resolve `PINOKIO_HOME` first, then produce the final launcher in the proper Pinokio location (`PINOKIO_HOME/api/<unique_name>` or `PINOKIO_HOME/plugin/<unique_name>`) unless the user explicitly asks for another layout.-- Replace machine-specific state with reproducible steps. Never hardcode absolute paths, user-specific cache locations, session-only ports, or one-off manual edits if they can be expressed in the launcher.-- Do not simply encode whatever happened to work on the current machine. Generalize the result into the broadest practical cross-platform, cross-machine launcher, and if limitations are unavoidable, declare them explicitly in `pinokio.json` instead of silently baking in local assumptions.-- If the app is already installed but the exact setup steps are partially missing, inspect the current working tree, generated files, dependency manifests, shell history when available, and logs to reconstruct the smallest reliable install and start flow.-- Verify from as clean a state as practical. A launcher is only done when another user could reproduce the working result without relying on undocumented steps from the original ad-hoc session.-### 11. Don't use Docker unless really necessary-- Some projects suggest docker as installation options. But even in these cases, try to find "development" options to launch the app without relying on Docker, as much as possible. We do not need Docker since we can automatically install and launch apps specifically for the user's platform, since we can write scripts that run cross platform.-### 12. pinokio.json-- Do not touch the `version` field since the version is the script schema version and the one pre-set in `pinokio.js` must be used.-- `icon`: It's best if we have a user friendly icon to represent the app, so try to get an image and link it from `pinokio.json`.- - If the git repository for the `~/pinokio/api/phosphene.git` folder points to GitHub (for example https://github.com/<USERNAME>/<REPO_NAME>`, ask the user if they want to download the icon from GitHub, and if approved, get the `avatar_url` by fetching `https://api.github.com/users/<USERNAME>`, and then download the image to the root folder as `icon.png`, and set `icon.png` as the `icon` field of the `pinokio.json`.-### 13. Gitignore-- When a launcher involves cloning 3rd party repositories, downloading files dynamically, or some files to be generated, these need to be included in the .gitignore file. This may include things like:- - Cloning git repositories- - Downloading files- - Dynamically creating files during installation or running, such as Sqlite Databases, or environment variables, or anything specific to the user.-- Make sure these file paths are included in the .gitignore file, and if not, include them in .gitignore.--## AI Libraries (Pytorch, Xformers, Triton, Sageattention, etc.)-If the launcher is for running AI models locally, the install script must declare the AI bundle so Pinokio can install the machine-level prerequisites before the script runs:--```-// install.js-module.exports = {- requires: {- bundle: "ai"- },- run: [- ...- ]-}-```--This is required even when the script also uses `torch.js`. The AI bundle is what triggers installation of common local AI prerequisites such as CUDA on NVIDIA systems and Hugging Face CLI.--If the launcher has a dedicated built-in script named `torch.js`, it can be used as follows:--```-// install.js-module.exports = {- requires: {- bundle: "ai"- },- run: [- // Edit this step with your custom install commands- {- method: "shell.run",- params: {- venv: "venv", // Edit this to customize the venv folder path- path: "app",- message: [- "uv pip install -r requirements.txt"- ],- }- },- // Delete this step if your project does not use torch- {- method: "script.start",- params: {- uri: "torch.js",- params: {- path: "app",- venv: "venv", // Edit this to customize the venv folder path- // xformers: true // uncomment this line if your project requires xformers- // triton: true // uncomment this line if your project requires triton- // sageattention: true // uncomment this line if your project requires sageattention- // flashattention: true // uncomment this line if your project requires flashattention- }- }- },- ]-}-```--The `torch.js` script also includes ways to install pytorch dependent libraries such as xformers, triton, sagetattention. If any of these libraries need to be installed, use the torch.js to install in order to install them cross platform.---## Quick Reference-### Essential Documentation-- **Pinokio Programming:** See `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md → "Programming Pinokio" section-- **Dynamic Menus:** See `PINOKIO.md` at ~/pinokio/prototype/PINOKIO.md → "Dynamic menu rendering" section-- **CLI Commands:** See `PTERM.md` at ~/pinokio/prototype/PTERM.md-### Common Patterns-- **Python Virtual Env:** `shell.run` with `venv` attribute-- **Cross-platform Commands:** Always test on multiple platforms-- **Error Handling:** Check logs/api for launcher issues-- **GitHub Operations:** Use `gh` CLI for advanced GitHub features-## Development Principles-1. **Minimize Shell Usage:** Leverage API parameters instead of raw commands-2. **Maintain Separation:** Keep app logic and launchers separate-3. **Follow Conventions:** Match existing project patterns-4. **Test Thoroughly:** Use CLI to verify launcher functionality-5. **Document Changes:** Update relevant metadata and documentationdiff --git a/QWEN.md b/QWEN.mdnew file mode 120000index 0000000..681311e--- /dev/null+++ b/QWEN.md@@ -0,0 +1 @@+CLAUDE.md\ No newline at end of filediff --git a/README.md b/README.mdindex 2d34aab..2c09b15 100644--- a/README.md+++ b/README.md@@ -13,15 +13,13 @@</p>-> ### Updating from v2.x to v3.0? Click Update TWICE.->-> Pinokio runs your existing v2 `update.js` on the first click — it pulls the new code but doesn't install 3.0's new Python deps (ltx-trainer, mlx-vlm for Gemma auto-caption, mflux 0.17.5, pyyaml/pydantic/tqdm/rich). The panel will boot to errors on first launch. **Click Update again** and Pinokio runs the new 3.0 script, installs everything, and the panel boots clean. Fresh installs are unaffected — this only hits v2.x → v3.0 upgrades.+> **Current release: v3.0.6.** Latest on the v3.0 line — character training, Audio-to-Video, the Image Studio tab, hardware capability tiering, and a run of stabilization fixes since launch.## Overview-Phosphene is a local generative-media panel for Apple Silicon. It runs [LTX-Video 2.3](https://github.com/Lightricks/LTX-Video) (MLX port) for joint audio-and-video synthesis, [Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) and an MLX port of [HiDream-O1-Image-Dev](https://huggingface.co/HiDream-ai/HiDream-O1-Image-Dev) for stills, and ships an in-panel LoRA training pipeline for character identity (face + optional voice from a single dataset). Everything runs on-device. No cloud, no API keys, no telemetry.+Phosphene is a local generative-media panel for Apple Silicon. It runs [LTX-Video 2.3](https://github.com/Lightricks/LTX-Video) (MLX port) for joint audio-and-video synthesis, [Qwen-Image-Edit-2509](https://huggingface.co/Qwen/Qwen-Image-Edit-2509) (with a Lightning 4-step fast tier) for stills, and ships an in-panel LoRA training pipeline for character identity (face + optional voice from a single dataset). Everything runs on-device. No cloud, no API keys, no telemetry.-3.0 introduces in-panel character training (face + voice LoRA from one dataset), the Audio-to-Video workflow, the Image Studio tab with two MLX-native engines, hardware capability tiering, and an agentic prompt enhancer / shot planner driven by the same local Gemma 3 12B used for auto-captioning.+3.0 introduces in-panel character training (face + voice LoRA from one dataset), the Audio-to-Video workflow, the Image Studio tab, hardware capability tiering, and an agentic prompt enhancer driven by the same local Gemma 3 12B used for auto-captioning.A 7-second character clip with synced audio renders in roughly 6 minutes on an M4 Max 64 GB. The delivered file is **1280×720 HD** after the built-in 2× upscale; clips are generated at 1024×576 internally and upscaled before mux. Voice + face LoRAs from a 50-image dataset finish in ~3 hours on the same hardware.@@ -39,7 +37,7 @@ Text-to-video, image-to-video, and audio-to-video, all delivered as MP4 with joi<img width="1920" height="843" alt="image" src="https://github.com/user-attachments/assets/8e2f52de-b34c-44cf-9283-df30c4079607" />-Two MLX-native engines share the same tab and the same GPU memory pool. Qwen-Image-Edit-2511 handles instruction edits ("change the white jacket to red") and multi-subject composition with up to three reference images. HiDream-O1-Image-Dev handles photoreal at HD — the MLX port of HiDream-O1 ships with Phosphene (8B Qwen3-VL backbone, unified pixel-patch transformer, MIT-licensed; weights at `mlx-community/HiDream-O1-Image-Dev-mlx-bf16`). It lives in a sibling clone, loaded on demand. Both engines drop cards into a unified gallery, each with an Animate button that pre-fills the I2V form with the source still.+Qwen-Image-Edit (Lightning 4-step) is the default image engine. It handles instruction edits ("change the white jacket to red") and multi-subject composition with up to three reference images, generating four candidates per shot in seconds with the Lightning LoRA baked in. It's an optional one-click install (the panel falls back to a lighter mflux model if Qwen isn't installed). Results drop cards into a unified gallery, each with an Animate button that pre-fills the I2V form with the source still.### Train Character@@ -64,11 +62,11 @@ Drop `.safetensors` into `mlx_models/loras/` for immediate use, or browse and in### HTTP API-Everything the panel does is reachable over plain HTTP on `127.0.0.1:8199`. Queue video jobs, generate images, train characters, manage LoRAs, poll status, fetch outputs — all from `curl`, a Python script, or an external agent like Claude Code or Codex. The panel UI is just one client; nothing about the feature set is exclusive to it.+Everything the panel does is reachable over plain HTTP on `127.0.0.1:8198`. Queue video jobs, generate images, train characters, manage LoRAs, poll status, fetch outputs — all from `curl`, a Python script, or an external agent like Claude Code or Codex. The panel UI is just one client; nothing about the feature set is exclusive to it.```bash# Queue a text-to-video render with a character LoRA stacked on a style LoRA.-curl -s -X POST http://127.0.0.1:8199/queue/add \+curl -s -X POST http://127.0.0.1:8198/queue/add \--data-urlencode "mode=t2v" \--data-urlencode "prompt=Cinematic close-up of bizarrotrn man in a wood-paneled study, golden hour" \--data-urlencode "width=1024" --data-urlencode "height=576" \@@ -173,7 +171,7 @@ Four workflow tabs at the top of the panel: Video, Images, Audio, Train Characte</tr><tr><td align="center"><sub><b>Video / Character mode</b> · round-avatar picker, voice indicator, manage modal</sub></td>-<td align="center"><sub><b>Images</b> · Qwen Edit, HiDream-O1, multi-ref composition</sub></td>+<td align="center"><sub><b>Images</b> · Qwen-Image-Edit, multi-ref composition</sub></td></tr><tr><td width="50%"><img src="assets/screenshots/phos_03_audio_tab.png" alt="Audio tab — audio drives the generation"></td>@@ -197,7 +195,7 @@ Prompting notes:Quit Pinokio (or the panel terminal), then click Update, then Start. Renders, settings, queue, models, and LoRAs all persist across the upgrade via Pinokio's `fs.link` persistent drive. The first update takes a few minutes.-> **Click Update twice on the first v2 → v3 upgrade.** Pinokio loads the user's existing on-disk `update.js` before pulling the new one, which means the v2.x script runs first and doesn't install the new dependencies (ltx-trainer package, mlx-vlm, the trainer's transitive deps). The second Update click runs the new 3.0 script and installs everything. This only applies to the very first migration from a v2.x install.+> Stragglers note: a few very old v2.x installs once had to click Update twice (Pinokio ran the stale on-disk `update.js` before pulling the new one). The update script now force-reinstalls the changed packages, so a single Update is enough. If a v2.x install still boots to dependency errors, click Update once more.Behavioral changes worth noting in 3.0:@@ -216,7 +214,7 @@ Behavioral changes worth noting in 3.0:- `lora_lab/` is vendored from the [`lora-lab`](https://github.com/mrbizarro/lora-lab) authoring tree. Training works out of the box; set `LTX_LORA_LAB_ROOT` to iterate against an external clone.- `mlx_models/` and `mlx_outputs/` both persist across Pinokio Reset via fs.link.-An MLX port of [HiDream-O1-Image-Dev BF16](https://huggingface.co/mlx-community/HiDream-O1-Image-Dev-mlx-bf16) (8B Qwen3-VL backbone, unified pixel-patch transformer, MIT) is included for the Image tab. HiDream lives in a sibling clone (see Setup).+`image_engine.py` also carries a `hidream` backend (8B Qwen3-VL backbone, unified pixel-patch transformer) for photoreal stills. It is **hidden in the UI since v3.0.3** ([#15](https://github.com/mrbizarro/phosphene/issues/15)) and not installed by default — it lives in a separate lab repo you clone manually into `$HIDREAM_LAB_DIR` (see Manual install, step 7) and is loaded on demand.## License and credits@@ -233,7 +231,7 @@ Phosphene depends on the following projects:- [ModelPiper / PiperSR](https://github.com/ModelPiper/PiperSR) — optional 2× upscale on the Apple Neural Engine- [@cocktailpeanut](https://twitter.com/cocktailpeanut) — Pinokio-What Phosphene adds on top of those: a persistent batch queue, a warm helper subprocess with capability-tier feature gating, lossless H.264 output with JSON sidecars, the in-panel character + voice LoRA training pipeline, the Image tab dispatch layer with adaptive wall-time estimates, the agentic prompt-enhancer / shot planner, and the Pinokio install + update lifecycle scripts.+What Phosphene adds on top of those: a persistent batch queue, a warm helper subprocess with capability-tier feature gating, lossless H.264 output with JSON sidecars, the in-panel character + voice LoRA training pipeline, the Image tab dispatch layer with adaptive wall-time estimates, the local Gemma 3 prompt-enhancer, and the Pinokio install + update lifecycle scripts.## Roadmapdiff --git a/VERSION b/VERSIONindex 4a36342..818bd47 100644--- a/VERSION+++ b/VERSION@@ -1 +1 @@-3.0.0+3.0.6diff --git a/docs/IMAGE_GEN_RESEARCH_2026-05.md b/docs/IMAGE_GEN_RESEARCH_2026-05.mdindex de67680..44cd742 100644--- a/docs/IMAGE_GEN_RESEARCH_2026-05.md+++ b/docs/IMAGE_GEN_RESEARCH_2026-05.md@@ -1,5 +1,11 @@# Phosphene Image-Gen Research — Apple Silicon, May 2026+> ⚠️ **Historical research (2026-05) — SUPERSEDED.** The shipped Image Studio+> default is **Qwen-Image-Edit** with a Lightning 4-step fast tier; **HiDream is+> hidden** pending its lab repo (issue #15); **FLUX was removed 2026-05-13.** This+> doc records the point-in-time research that informed those decisions, not the+> current state — see `docs/STATE.md` for what actually ships.+_Tier-aware research for the agent's anchor-still pipeline (i2v inputs).LTX 2.3's t2v is weak; image-gen quality is the dominant lever for video output._@@ -8,7 +14,7 @@ LTX 2.3's t2v is weak; image-gen quality is the dominant lever for video output.- **"Flux Klein"** = **FLUX.2 [klein] 4B** by Black Forest Labs (Jan 2026). Apache 2.0, mflux-compatible at 4-bit (~4.3 GB), 4-step inference, ~12-18 s/image on Comfortable Mac. **Ship as default.**- **mflux 0.17.x is now multi-family.** Legacy `mflux-generate` is FLUX.1-only. New CLIs: `mflux-generate-flux2`, `mflux-generate-z-image-turbo`, `mflux-generate-fibo`, `mflux-generate-qwen`, `mflux-generate-kontext`. Panel currently calls legacy → silently misses every new family.- **Z-Image-Turbo (Tongyi/Alibaba, Nov 2025)** is the Compact-tier default — 6B model, ~5.9 GB at 4-bit, 8-9 steps, fits 16 GB.-- **HiDream-O1-Image-Dev (May 2026 addition)** — 8B Qwen3-VL pixel-patch transformer, MIT licence, runs out of a separate lab venv (NOT mflux). **Q6 default** (~8 GB, ~36 s / 1024×1024) — 2× faster than Q8 with equivalent fidelity (mlx is bandwidth-bound). Strong prompt fidelity across all genres tested. Available as `kind="hidream"` in `agent/image_engine.py` since commit 45cad69.+- **HiDream-O1-Image-Dev (May 2026 addition)** — 8B Qwen3-VL pixel-patch transformer, MIT licence, runs out of a separate lab venv (NOT mflux). **Q6 default** (~8 GB, ~36 s / 1024×1024) — 2× faster than Q8 with equivalent fidelity (mlx is bandwidth-bound). Strong prompt fidelity across all genres tested. Available as `kind="hidream"` in `image_engine.py` since commit 45cad69.## The single default to ship@@ -88,7 +94,7 @@ All share the same argparse mixins — `--prompt`, `--negative-prompt`, `--widthmflux is great but only ships diffusion-based families. **HiDream-O1-Image-Dev** is a different beast: a single 8B Qwen3-VL backbone that does pixel-level patch generation directly (no separate VAE), distilled from a 50-step teacher into a 28-step student via flow matching. License is MIT (cleaner than FLUX.2-dev's NC).-The lab port lives outside Phosphene at `~/HIDREAM-O1-MLX-LAB-active/` so its mlx-vlm dep tree doesn't fight Phosphene's ltx-2-mlx env. Phosphene shells out to the lab's python via subprocess (mirrors mflux pattern). Module path: `agent/image_engine.py`, `kind="hidream"`.+The lab port lives outside Phosphene at `~/HIDREAM-O1-MLX-LAB-active/` so its mlx-vlm dep tree doesn't fight Phosphene's ltx-2-mlx env. Phosphene shells out to the lab's python via subprocess (mirrors mflux pattern). Module path: `image_engine.py`, `kind="hidream"`.### Why ship it alongside mflux@@ -116,7 +122,7 @@ The lab port lives outside Phosphene at `~/HIDREAM-O1-MLX-LAB-active/` so its ml## Implementation plan (20 ranked items)-1. **(M)** Family-aware `_resolve_mflux_bin` in `agent/image_engine.py:131` + new `MFLUX_FAMILY_BIN` lookup table. Without this, none of the 0.17 families are reachable.+1. **(M)** Family-aware `_resolve_mflux_bin` in `image_engine.py:131` + new `MFLUX_FAMILY_BIN` lookup table. Without this, none of the 0.17 families are reachable.2. **(M)** Per-family default-flag table in `_generate_mflux`. klein-4B uses 4 steps + guidance 1; Z-Image-Turbo uses 9 steps + guidance 0; Krea-dev keeps 25 + guidance 4.5.3. **(L)** Tier auto-detect via `_default_image_engine_for_tier` in `mlx_ltx_panel.py:~2042`. Apply at first boot when `agent_image_config.json` doesn't exist.4. **(M)** Add FLUX.2-klein-4B to the model dropdown — `mlx_ltx_panel.py:10881`. Add `<optgroup>` per family.@@ -131,7 +137,7 @@ The lab port lives outside Phosphene at `~/HIDREAM-O1-MLX-LAB-active/` so its ml13. **(M)** Sibling venv `image-gen/env/` so mflux deps don't conflict with LTX's transformers/mlx pins. `_resolve_mflux_bin` checks there first.14. **(S)** HF gating UX — when `_hf_size_estimate` returns `gated: true`, surface "Open HF model card → accept terms → paste HF_TOKEN" inline.15. **(S)** Update `docs/AGENTIC_FLOWS.md § Image generation backends` — document family dispatch, tier defaults, deprecate legacy `mflux-generate`-only path.-16. **(S)** Update `agent/image_engine.py:1-28` module docstring — current still says "Two backends ship in v1: mock, bfl" with mflux as "future".+16. **(S)** Update `image_engine.py:1-28` module docstring — current still says "Two backends ship in v1: mock, bfl" with mflux as "future".17. **(M)** Tier-aware default for `ImageEngineConfig.mflux_model` (was `"krea-dev"` — wrong for 16 GB Macs).18. **(L)** "Compare models" Settings page — runs test prompt across 3 picks side-by-side, click to set as default.19. **(L)** Add `mflux-upscale-seedvr2` as post-render option for image candidates.diff --git a/docs/SDK_KEYFRAME_INTERPOLATION.md b/docs/SDK_KEYFRAME_INTERPOLATION.mdindex 1d352d2..7bc4a5d 100644--- a/docs/SDK_KEYFRAME_INTERPOLATION.md+++ b/docs/SDK_KEYFRAME_INTERPOLATION.md@@ -1,6 +1,6 @@# Multi-keyframe interpolation — the SDK shot-composition primitive-Status: **Layer 1 shipped 2026-05-06** — helper now accepts arbitrary keyframe lists. Layer 2 (panel HTTP form-parsing) and Layer 3 (UI) not yet. Engine has supported this all along; we just exposed it. See [Agent API contract](#agent-api-contract-layer-1-2026-05-06) for the call shape.+Status: **Layers 1 + 2 shipped** — helper accepts arbitrary keyframe lists (Layer 1, 2026-05-06) and the panel HTTP route now parses a `keyframes_json` list and builds the full `keyframe_images`/`keyframe_indices` arrays (Layer 2). Layer 3 (UI with >2 drop-zones) not yet. Engine has supported this all along; we just exposed it. See [Agent API contract](#agent-api-contract-layer-1-2026-05-06) for the call shape.## TL;DR@@ -65,7 +65,7 @@ class KeyframeInterpolationPipeline(TwoStagePipeline):`keyframe_images` and `keyframe_indices` are both **lists**. The function loops over them, encodes each at the appropriate stage resolution, and threads them through stage 1 (half-res guided denoise) and stage 2 (full-res refinement). Multi-keyframe is the native shape of the API.-Today's restriction is one layer up — in the panel's worker:+The original restriction lived one layer up — in the panel's worker:```mlx_warm_helper.py:@@ -73,9 +73,7 @@ mlx_warm_helper.py:keyframe_indices=[0, num_frames - 1],```-Two keyframes, hardcoded to first and last. The panel's `mode == "keyframe"` only accepts `start_image` and `end_image` form fields. UI offers two image drop-zones.--Net: every other keyframe slot the engine could use is silently discarded.+Two keyframes, hardcoded to first and last. **This restriction has since been lifted at the HTTP layer (Layer 2, shipped):** the panel's `mode == "keyframe"` branch now parses a `keyframes_json` list (≥2 items, strictly-ascending range-checked indices) and threads the full `keyframe_images`/`keyframe_indices` arrays to the helper (`mlx_ltx_panel.py` ~6448-6526); `start_image`/`end_image` are kept as a backward-compat fallback. The remaining gap is the **UI**, which still offers only two image drop-zones (Layer 3).## Why this matters for the SDK@@ -129,13 +127,13 @@ Helper `generate_keyframe` action now accepts arbitrary `keyframe_images` + `keyFull agent contract below.-### Layer 2 — panel HTTP form parsing — NOT YET+### Layer 2 — panel HTTP form parsing — DONE-`mlx_ltx_panel.py` `mode == "keyframe"` branch. Currently parses `start_image` + `end_image` form fields and constructs the helper job spec. Change to parse a **list of keyframes** from the form (e.g. `keyframe_image_0`, `keyframe_image_1`, ... or a JSON-encoded list field). Build the lists for the helper.+`mlx_ltx_panel.py` `mode == "keyframe"` branch (~6448-6526). Parses a JSON-encoded **`keyframes_json`** list field: each entry is `{image_path, frame_index}`, the list must have ≥2 items, `frame_index` values must strictly increase, each image must exist, and every index is range-checked against the clip length `[0, frames-1]`. From that list it builds the full `keyframe_images` + `keyframe_indices` arrays and hands them to the helper's Layer-1 contract; it also syncs `start_image`/`end_image` to the first/last entries so the sidecar + gallery preview stay sensible.-Backward compat: if the form has `start_image` + `end_image` only, behave as today.+Backward compat: if `keyframes_json` is empty/absent, it falls back to the legacy `start_image` + `end_image` two-keyframe shape exactly as before.-This is the layer the agent will use if it goes through the panel's HTTP queue endpoint instead of talking to the helper directly.+This is the layer the agent uses when it goes through the panel's HTTP queue endpoint instead of talking to the helper directly.### Layer 3 — UI (the user-visible surface) — NOT YET@@ -305,7 +303,7 @@ result = submit({### What's NOT in this layer- **Per-keyframe strength.** All keyframes are pinned at strength 1.0 (pure anchor). The model cannot "lightly suggest" a keyframe yet — it's all-or-nothing. Strength control requires an upstream patch in `ltx-pipelines-mlx`.-- **Panel HTTP route.** Layer 2 is needed to reach this from the panel's queue (HTTP form-data). Today the only path is the helper's stdin.+- **Panel HTTP route.** *(Update: Layer 2 has since shipped — the panel's queue now reaches multi-keyframe via the `keyframes_json` form field. At Layer-1 authoring time the only path was the helper's stdin.)*- **Panel UI.** Today's panel still has 2 image drop-zones. Adding more rows is Layer 3.- **Audio across keyframe boundaries between shots.** Each clip is its own audio render — boundaries are hard cuts. See "What this doesn't solve" below.diff --git a/docs/STATE.md b/docs/STATE.mdindex 3a18456..16e2469 100644--- a/docs/STATE.md+++ b/docs/STATE.md@@ -1,25 +1,37 @@# Phosphene — project state, history, open work-Current `dev` head: see `git log -1` for the live SHA. The arc landed on May 17 2026 — Codex C+ 7-pass UI restructure (capability tiers, Q4 surface, Character as 5th mode pill, accel kill, HQ-speed move) + Train tab polish (Gemma 3 auto-caption, voice-on default, letterbox crop) + LoRA picker chrome (rename, download, companion-aware delete) + vendored `lora_lab/` for installer-only users + Codex 12.6% skip-step optimization on Q8 HQ.+**Public release: `v3.0.6`.** The entire v3.0 line shipped — v3.0.0 (Characters / Voice / Image Studio / A2V, May 23) → v3.0.1 (FFLF crash fix) → v3.0.2 (Boost/Turbo accel restored after a 2-month silent regression) → v3.0.3 (HiDream hidden, #15) → v3.0.4 (CivitAI SSL) → v3.0.5 (A2V kwarg signature shim, #5) → v3.0.6 (deep-review hardening). See §4 for the full version history. The May-17 Codex C+ UI restructure (capability tiers, Q4 surface, Character as 5th mode pill, accel kill, HQ-speed move) and the Train-tab / LoRA-chrome work that followed are all baseline now, folded into v3.0.x. There is no `v2.0.6` tag.-Latest `main`: still `v2.0.6` — `dev` carries ~30 commits ahead awaiting Mr Bizarro's ship verdict.+Current `dev` head: see `git log -1` for the live SHA. `dev` tracks `beta/main` (private repo — see §1).Live URL: `https://github.com/mrbizarro/phosphene` · Linear project: `https://linear.app/hairstylemojo/project/phosphene-9c11240704bb`This doc is the **session-start handoff**. A new Claude window entering this project should read this first, then `CLAUDE.md` (architecture), then the relevant Linear issues.+> **Authoritative engineering snapshot:** `/Users/salo/AI/projects/phosphene/notes/DEEP_REVIEW_2026-05.md` (2026-05-31) — the deep stabilization review: risk register + phased stabilization plan. It is the source of truth for current bug state and supersedes the historical log in §7 below.+---## 1. Where the code lives+**Repo split (2026-05-22):** there are now two GitHub repos:++- **Public `mrbizarro/phosphene`** — `main`-only, stable releases. The public `dev` branch was **DELETED**. Anyone with a hand-configured public-`dev` install must reinstall from `main`.+- **Private `mrbizarro/phosphene-beta`** — daily development. Holds `main` (daily dev) plus `archive/*` experimental branches.+Two clones on Mr Bizarro's Mac, both managed by Pinokio:| Path | Branch tracked | Port | Role ||---|---|---|---|-| `~/pinokio/api/phosphene-dev.git/` | `dev` | 8199 | Active development. Most edits land here first. |-| `~/pinokio/api/phosphene.git/` | `main` | 8198 | Production / daily driver. Mr Bizarro's actual usage. |+| `~/pinokio/api/phosphene-dev.git/` | local `dev` → `beta/main` (private) | 8199 | Active development. Most edits land here first. |+| `~/pinokio/api/phosphene.git/` | `main` → public `origin` | 8198 | Production / daily driver. Mr Bizarro's actual usage. |++The local branch is still named `dev`, but it tracks **`beta/main`** (private), NOT the deleted public `dev`. `update.js` auto-detects upstream (`@{upstream}`), so the dev clone pulls from `beta` and the prod clone pulls from public `main`.++GitHub is the source of truth (memory: `feedback_github_source_of_truth.md`). Branch policy is strict:-GitHub is the source of truth (memory: `feedback_github_source_of_truth.md`). Branch policy is strict: **NEVER push to `main` without Mr Bizarro's explicit OK** (memory: `phosphene_dev_workflow.md`). When ready to promote, fast-forward only — `git merge --ff-only dev` from main, never a merge commit.+- Push daily work to **`beta`**, never to a public `dev` (it no longer exists).+- **Promotion to PUBLIC `main` is the gated step — NEVER push public `main` without Mr Bizarro's explicit OK** (memory: `phosphene_dev_workflow.md`).State directories that live OUTSIDE the repo via Pinokio's `fs.link`:@@ -30,21 +42,23 @@ State directories that live OUTSIDE the repo via Pinokio's `fs.link`:A Pinokio Reset wipes the install dir but preserves all four — Mr Bizarro can Reset → Install without losing renders or settings.-## 2. Current capabilities (dev head — pending v2.1 promotion)+## 2. Current capabilities (shipped in v3.0.6)++The May-17 Codex C+ items (Studio/Train tabs, Character 5th mode, capability tiers) are baseline now — they shipped as part of the v3.0 line.**Workflow tabs (top nav)**- Manual — video composer (T2V / Character / I2V / FFLF / Extend)-- Studio (NEW) — image generation (was a mode chip inside Manual until 2026-05-17, commit `37c9d21`)-- Train Character (NEW) — dataset → LoRA training, with Gemma 3 auto-caption + letterbox crop+- Studio — image generation (was a mode chip inside Manual until 2026-05-17, commit `37c9d21`)+- Train Character — dataset → LoRA training, with Gemma 3 auto-caption + letterbox crop**Modes (inside Manual)**- Text — pure text→video-- Character (NEW, 2026-05-17 commit `e420e3a`) — first-class mode for trained character LoRAs. Submits `mode=t2v` server-side; backend dispatches on `character_id`. Auto-stacks face + audio LoRAs, swaps the quality strip to Q8-only.+- Character (2026-05-17 commit `e420e3a`) — first-class mode for trained character LoRAs. Submits `mode=t2v` server-side; backend dispatches on `character_id`. Auto-stacks face + audio LoRAs, swaps the quality strip to Q8-only.- Image — image→video (I2V)- FFLF — first/last frame keyframe interpolation- Extend — append seconds onto an existing clip-**Capability tier system (NEW, 2026-05-17 commit `64dad87`)**+**Capability tier system (2026-05-17 commit `64dad87`)**- `body[data-cap-tier="q4|q8"]` set at request time from `SYSTEM_CAPS.allows_q8`.- `q4` (sub-48GB Macs): FFLF / Extend / Character mode pills hidden; chip strip hidden; Q8-Draft/Q8-Pro chips hidden; "High" chip in default strip hidden; skip-step toggle hidden. Manual collapses cleanly to Text/Image.- `q8` (48GB+): full surface, Q4 still reachable via the default Quality strip for plain T2V/I2V.@@ -149,17 +163,21 @@ A Pinokio Reset wipes the install dir but preserves all four — Mr Bizarro canSettings)- Plan/Interactive mode pill toggle in agent header-**Image Studio (dev head — Klein retired 2026-05-13)**-- Two engine families in the dropdown. Six tiers total:- - **Qwen-Image-Edit-2511** — Fast (Lightning, 4-step Q6 + FBCache,+**Image Studio (Klein/FLUX retired 2026-05-13; HiDream hidden since v3.0.3)**+- **Shipped default: Qwen-Image-Edit-2509** with a Lightning 4-step fast tier. Three tiers:+ - **Qwen-Image-Edit-2509** — Fast (Lightning, 4-step Q6 + FBCache,~1:20 / image, multi-ref), Medium (8-step Q6 + FBCache, ~2:05 /image, multi-ref), Quality (40-step Q8 + CFG 4.0 + FBCache, ~3:50- / image, multi-ref).- - **HiDream-O1-Image-Dev** — Fast (3-step, ~3:45 / 4-img batch,- slight bg softness), Medium (6-step + FBCache, ~6 min / 4-img- batch, character-preserving), Quality (12-step + light FBCache,- ~9 min / 4-img batch, best detail).-- HiDream is a separate one-time clone (`HIDREAM_LAB_DIR` env var or+ / image, multi-ref). (The `2511` string in the codebase is the+ Lightning **LoRA** path `lightx2v/Qwen-Image-Edit-2511-Lightning`,+ not the base model.)+- **HiDream-O1-Image-Dev is HIDDEN** since v3.0.3 (issue #15 — held+ pending its lab repo going public). The code is present and still+ reachable via a saved config or `engine_override`, just not in the+ visible dropdown. When exposed it offers Fast (3-step, ~3:45 / 4-img+ batch), Medium (6-step + FBCache, ~6 min, character-preserving),+ Quality (12-step + light FBCache, ~9 min, best detail), and lives in+ a separate one-time clone (`HIDREAM_LAB_DIR` env var or`~/HIDREAM-O1-MLX-LAB-active`); resolved in `image_engine.py`.- Q6 default quantization for mflux Qwen tiers — Apple-Siliconcommunity sweet spot, ~4-6% quality loss vs full precision (Q4 was@@ -186,18 +204,21 @@ A Pinokio Reset wipes the install dir but preserves all four — Mr Bizarro canlabels=bug, optionally bundles the latest 5 .ips crash files into/tmp/phosphene-bug-TS.zip.- FLUX / Klein-Edit / Klein-Base-Edit were retired 2026-05-13 — the- flux2_edit family stopped being competitive with Qwen-Edit-2511 +- HiDream at the same step counts, and dropping it freed UI vocabulary- for the new Qwen Fast/Medium/Quality + HiDream Fast/Medium/Quality- six-tier grid above.+ flux2_edit family stopped being competitive with Qwen-Edit at the+ same step counts, and dropping it freed UI vocabulary for the Qwen+ Fast/Medium/Quality tiers above. (HiDream was later hidden in v3.0.3,+ so the visible dropdown today is Qwen-only.)-**Frontend extraction (parked on `frontend-extraction` branch)**+**Frontend extraction (parked on `archive/frontend-extraction`, private beta)**- `webapp/` directory: index.html, style/all.css, js/main.js,vendor/marked.min.js + dompurify.min.js (MIT/Apache licenses)- Panel slimmed from 16,223 → 5,866 lines (-10,357)- New `/webapp/*` static route, `/api/page-config` endpoint- Markdown rendering swapped to `marked.parse + DOMPurify.sanitize`-- Validated end-to-end on port 8210; merge to dev when reviewed+- Validated end-to-end on port 8210. Post-split this is an `archive/*`+ experimental branch on private `phosphene-beta`, not a merge-to-`dev`+ candidate. Note: a separate `cuda-port` branch (Phase 0 spike only)+ also lives on private beta — **not shipped, not on the Mac product.**## 3. Marquee benchmarks (M4 Max 64 GB, sidecar-measured)@@ -233,10 +254,10 @@ Pre-2.0 was the `Y1.NNN` sequential counter. v2.0.0 cut over to semver on May 3**v2.0.4** (May 5) — Strip em-dash from install.js sanity check. Was breaking install on some Pinokio shells (KTDS + second user hit identical SyntaxError). Pure ASCII now.**v2.0.5** (May 6) — Drop the `print('venv OK: ...')` decoration from the sanity-import step. KTDS reproduced the SyntaxError on v2.0.4 — turns out something in their environment (Pinokio's command preprocessor or a user-side rewriter) was cutting the literal `OK:` out of the Python string AND appending `OK` after the closing shell quote, so Python received `...importable')OK` and bailed. Removing the print sidesteps the rewriter entirely. The exit code from a successful `import` is the only success signal `shell.run` needs anyway.-**v2.0.6** (May 8 2026) — Image Studio overhaul + agent quality pass + security review. Headline ships:+**"v2.0.6" dev codename** (May 8 2026; shipped as part of v3.0.0) — Image Studio overhaul + agent quality pass + security review. Headline ships:- Image jobs flow through the unified queue worker (Now / Queue / Recent / Logs); the in-Studio gallery is gone, the unified Recent tab covers it.- Mode-aware right-pane viewer + OUTPUTS gallery (All / Videos / Photos chips, auto-flip on mode change, "Animate" button on photo cells pre-fills i2v).-- Q6 default quantization for mflux + non-distilled photoreal presets (`flux2_edit_high`, `qwen_edit_high`, `kontext_high`); klein-4B prompt structure taught to the agent (subject → environment → style → technical hierarchy); Qwen-Edit defaults bumped from 2509 → 2511.+- Q6 default quantization for mflux + non-distilled photoreal presets (`flux2_edit_high`, `qwen_edit_high`, `kontext_high`); klein-4B prompt structure taught to the agent (subject → environment → style → technical hierarchy). *(This session bumped the Qwen-Edit default toward 2511, but the **shipped base model is 2509** — see §2; `2511` is the Lightning LoRA path only.)*- `submit_shots` plural tool — agent batches a multi-shot plan in one dispatch + finishes the turn before auto-pause kills the engine (used to crash mid-batch).- Phase C i2v prompt-writing rules in `prompts.py` (forbid still-prompt reuse, require explicit motion beats, ~1 beat / 2-3 sec); production recipe taught as Balanced + Sharp 720p.- Pre-flight RAM/disk check + Metal/MLX SIGABRT detection with actionable OOM hint (no more silent exit -6).@@ -247,7 +268,17 @@ Pre-2.0 was the `Y1.NNN` sequential counter. v2.0.0 cut over to semver on May 3- Agent now switches engines via `engine_override` arg on `generate_shot_images`.- Security review pass: 0 CRITICAL, 4 HIGH, 6 MEDIUM identified — all 10 shipped this session. (See Known bugs section for details.)-Full git log on `main`. Tags `v2.0.0` through `v2.0.4` published. v2.0.5 + v2.0.6 awaiting promotion on dev.+> Note: "v2.0.6" was the dev codename for the work that ultimately shipped as **v3.0.0**. There is **no `v2.0.6` tag** — the published 2.x tags stop at v2.0.5.++**v3.0.0** (May 23 2026) — Marquee release. Folded in the full Characters workflow (Train Character tab + first-class Character mode), Voice/audio LoRAs, the standalone Image Studio (Qwen-Image-Edit-2509 default + Lightning 4-step), A2V, the Codex C+ capability-tier UI restructure, and the Image Studio overhaul above. The in-panel agentic-flows chat was retired in this release.+**v3.0.1** — FFLF crash fix.+**v3.0.2** — Boost/Turbo accel restored after a 2-month silent regression (git-archaeology'd the per-mode tail + re-anchored the accel patch across all `denoise_loop` import sites).+**v3.0.3** — HiDream hidden from the visible dropdown until its lab repo is public (issue #15). Code stays, reachable via saved config / `engine_override`.+**v3.0.4** — CivitAI SSL fix.+**v3.0.5** — A2V `frame_rate` kwarg signature shim (issue #5).+**v3.0.6** — Deep-review hardening: CivitAI token-leak fix, dead-HDR revival (`HELPER_LOW_MEMORY` → `LOW_MEMORY` NameError), GPU-contention guard (inline image vs in-flight video render), boot-time version-gate against the ltx-2-mlx pin.++**Published tags (verified `git tag`): `v2.0.0`–`v2.0.5` and `v3.0.0`–`v3.0.6`.** Current public release is **v3.0.6**. (The `VERSION` file in-repo may lag the tag — read the tag, not the file, for "what release is this.")## 5. The folder layout@@ -291,7 +322,7 @@ phosphene-dev.git/│ └── SDK_KEYFRAME_INTERPOLATION.md ← multi-keyframe interpolation design + plan├── launch/ ← marketing copy (Pinokio article, X thread, Reddit, etc.)│-├── ltx-2-mlx/ ← upstream MLX port (separate clone of dgrauet/ltx-2-mlx)+├── ltx-2-mlx/ ← upstream MLX port, PINNED v0.14.0 (clone of dgrauet/ltx-2-mlx; SHA b35254a)│ └── env/ ← Python 3.11 venv (uv-managed)├── mlx_models/ ← weights (~63 GB, fs.link symlink)├── mlx_outputs/ ← rendered mp4s + sidecars (fs.link symlink)@@ -301,7 +332,9 @@ phosphene-dev.git/└── logs/ ← Pinokio's own command-execution logs```-`mlx_ltx_panel.py` is the heart of it — almost all panel behavior lives there. `mlx_warm_helper.py` is the long-running inference subprocess. `patch_ltx_codec.py` is a 500-line runtime modifier that fixes upstream code without forking it.+`mlx_ltx_panel.py` is the heart of it — almost all panel behavior lives there. `mlx_warm_helper.py` is the long-running inference subprocess. `patch_ltx_codec.py` is a runtime modifier that fixes upstream code without forking it.++> The vendored `ltx-2-mlx` is pinned at **v0.14.0** (SHA `b35254a`). The deep review (`notes/DEEP_REVIEW_2026-05.md`) flags the **runtime-monkey-patch + version-skew axis as the single top fragility** — the panel/helper patch a moving upstream at runtime, and (pre-Phase-0) nothing asserted the imported `ltx_pipelines_mlx` was actually v0.14.0. The stabilization plan there (Phase 0 loud version gate, Phase 3 retire the runtime-patch class) addresses it.## 6. What worked / didn't this session (May 3–5 2026)@@ -336,309 +369,112 @@ phosphene-dev.git/- **E-ANCHOR** (May 4): I2V from M1 frame to test character anchoring. Result was inconclusive in the session; final clip is at `mlx_outputs/` if needed for review.- **20-sec single-clip viability** (May 4): confirmed at Balanced 1024×576 + Turbo + Sharp. ~21 min wall, audio synced, characters stable.-## 7. Known bugs / fixed bugs--### 2026-05-22 (later) — stats dashboard moves INSIDE the panel (private)--Mr Bizarro: "don't put it in the public repo." Pivoted the architecture-from GitHub Pages + committed JSONL to entirely panel-internal:--- **Dashboard URL is now `http://127.0.0.1:8199/stats`** (panel-served,- 127.0.0.1-only). Bookmark this; the panel must be running to view.-- **Data lives at `state/stats-data.jsonl`** — gitignored, on the user's- Mac only, never on the public repo.-- **`panel_assets/stats.html`** holds the dashboard template (still public- code — only the data is private).-- **Panel background thread `stats_fetch_loop`** runs `scripts/fetch_repo_stats.py`- once at startup (if data file is missing/stale ≥ 6h) and daily thereafter.-- **GitHub Action deleted** — no public cron, no GitHub Pages, no GitHub- push needed at all to keep the dashboard working.-- **Token resolution** (first hit wins): `PHOSPHENE_REPO_STATS_TOKEN` →- `GH_STATS_TOKEN` → `GH_TOKEN` / `GITHUB_TOKEN` → `gh auth token`. Skipped- silently if none works (panel logs one warning + carries on).-- The brief window where the data was on the public repo- (`151d0d2`..`827c5d8`) only contained one snapshot of public aggregate- counts — nothing private leaked. Door closed before it became a habit.--Setup is now ZERO steps for the user (panel + token both already there);-just open `/stats`. See `scripts/STATS_DASHBOARD.md` for the full guide.--### 2026-05-22 — ship-prep pass: GitHub-data dashboard + analytics ROLLED BACK--Mr Bizarro decision: opt-in telemetry won't be well-received in the OSS-world. Replaced with a GitHub-Traffic-API-only dashboard that gives the-"how is the app doing" signal without putting any privacy weight on-users. Three additional commits to dev:--- **`da1d6f5` — analytics fully removed.** -871 lines. `analytics.py`- and `TELEMETRY.md` deleted. `mlx_ltx_panel.py` stripped of the- default-settings entries, validator branches, public-safe view- fields, `/settings` opt-in/opt-out/forget-me lifecycle,- `__main__` install + `panel_boot` emit, serve_forever-finally- shutdown, worker `render_start` / `render_done` / `render_failed`- / `render_cancelled` emits + error_category bucketing, the- Settings → Anonymous analytics HTML section, and the three JS- functions + modal-open hook. README reverted to "no telemetry".- Existing `panel_settings.json` files may still carry the old keys;- the loader ignores them silently — no migration script needed.--- **`151d0d2` — solo-maintainer GitHub stats dashboard.** Three- files: `scripts/fetch_repo_stats.py` (stdlib-only Python fetches- repo + traffic + referrers + paths + releases; appends one JSON- line per UTC day to `docs/stats-data.jsonl`; idempotent),- `docs/stats.html` (vanilla-JS + Chart.js dashboard with cards,- time-series, referrer/path tables), and the seed snapshot.- `.github/workflows/repo-stats.yml` is staged locally — Mr Bizarro- must run `gh auth refresh -s workflow` + push it from a local- clone (the in-session OAuth token doesn't have `workflow` scope).--- **Polish pass (next commit, batched with this STATE update).**- Three parallel research/improvement agents:- - **Dashboard polish**: period selector (7d/30d/90d/all),- week-over-week + yesterday delta per card, sparklines per- card, "Growth" daily + weekly card, stars-timeline section- (feature-detected), 15-day rolling window from- `clones_window`/`views_window` when present, loading- skeleton + error card, raw-JSONL header link, theme- toggle (light/dark, persists), keyboard focus rings,- sticky table headers + internal scroll, K/M number- shortening, clickable referrer domains, lazy chart init- via IntersectionObserver, mobile stacking, `aria-label`- everywhere. 1117 lines total, still single file + one CDN.- - **Fetcher enrichment**: full 15-day `clones_window` +- `views_window` arrays; `stars_timeline` (cumulative, login- DROPPED for privacy, capped at 5000); rate-limit handling- (sleep until reset when remaining < 100); 3-retry backoff- on 5xx/network; `--dry-run` + `--repo OWNER/NAME` flags;- one-line summary at exit for `$GITHUB_STEP_SUMMARY`.- - **Workflow hardening + maintainer README**: SHA-pinned- `actions/[email protected]` + `actions/[email protected]`;- concurrency group; `dry_run` + `repo` inputs on- workflow_dispatch; `$GITHUB_STEP_SUMMARY` + commit-body- summary; failure-issue creation that DE-DUPES across- consecutive outage days; minimized `permissions:`- (contents:write + issues:write only); pip cache enabled.- Plus a new `docs/stats-README.md` explaining the whole- stack for future maintainers (running locally, enabling- Pages, adding metrics, what GitHub data CAN'T tell you,- FAQ, troubleshooting).--Today's seed data lands on the dashboard the moment Pages comes-online: 57 stars, 5 forks, 8 open issues, 1 PR, 3954 clones in the-last 14 days (148 unique today), 1601 views, with Google / GitHub-/ Huggingface / Twitter / Reddit as the top funnels. W/w delta:-clones -1130 (last week was a spike — likely a Reddit/HN bump).--### Fixed 2026-05-21 EVENING (overnight push — read this first)--Late-evening continuation after Mr Bizarro called out the still-broken-hang ("fix it. What do you mean 'deferred to roadmap'?") and the-gallery glitch ("the '21:19' label that looks like a render duration").-Five additional commits to `dev`:--- **The post-decode hang is FIXED** (commit `94bd696`, the second- attempt — the first one in `adc1cd2` did not work, see below).-- First attempt: an in-helper daemon-thread watchdog that would emit- done + os._exit(0) past a 30s grace. It was correct in concept but- doesn't fire in practice — **Metal's command-buffer completion- handlers block every Python thread's GIL access during the- deallocator chain**, so the watchdog thread is starved by the very- thing it was meant to escape. Observed live: helper hung 5+ min- after the watchdog should have rescued at 30s; ps showed all 16- helper threads at 0% CPU.-- Working fix: rescue from THE PANEL (a separate process — GIL inside- the helper is irrelevant from out here). `WarmHelper.- _build_post_decode_panic` returns a `(log_hook, panic_check)` pair- that `_read_until` honors. `log_hook` spots upstream's- "[Decoding ... done in X.Xs]" line and arms a 45s grace clock.- `panic_check` runs every 500ms in the select loop; if grace- expired + output file is on disk > 8KB, it SIGKILLs the helper- (from a daemon thread to not block the tick) and returns a- synthetic done event pointing at the known output_path. Helper- respawns on the next job (~30s spawn cost).-- Validated end-to-end on the same 768×416 +6f Extend that hung- 5+ min the first time and 13+ min the time before. Log shows the- exact rescue sequence Mr Bizarro can audit:-- [22:01:07] [Decoding video + audio + muxing] done in 16.9s- [22:01:07] [panel-watchdog] extend …: decode-done signal- seen, grace clock armed (45s).- [22:01:52] [panel-watchdog] extend …: post-decode hang- detected — file on disk (802615 bytes) but helper- silent for 45s after decode-done. SIGKILLing- helper; render counted as done.- [22:01:52] Extend done in 533.98s → …mp4-- Armed for `generate` (T2V/I2V Balanced) + `extend` only. A2V,- FFLF, T2V Character HQ, image, train all return cleanly and- don't need it.--- **Extend speed defaults cut** ("is that not too slow?"). Bundled- in `94bd696`. Default steps 12 → 8 (cfg=1.0 doesn't need 12) and- Extend TeaCache threshold 0.5 → 0.7 (more aggressive block-skip).- Measured ~6 min wall on the same +6f case that was 11 min the- same morning. Quality holds at cfg=1.0 single-branch.--- **Gallery dn-cache leak** (commit `94bd696`). The "21:19" label- Mr Bizarro pointed at was Extend's pre-downscale cache file- (`<base>_dnWxH.mp4`) appearing in the carousel without a sidecar- — the duration formatter fell back to mtime HH:MM. Two fixes:- `list_outputs` filters stems ending in `_dn\d+x\d+` when no- sidecar exists; `_outputDurationLabel` no longer falls back to- HH:MM (uses `_relTimeFromMtime` instead).--- **Elon v3 retrain kicked off** (~22:02, ETA ~4h28m, finishes- around 02:30). Per the validated v2 recipe Mr Bizarro asked for:- rank 32 · 100 epochs (7900 steps for 79 imgs) · lr 1e-4 ·- **512×512 square** · center crop · `user_provided` captions- (the existing [VISUAL]:-format captions already in the dataset).- Identity-only (train_audio=false) so the existing- elontrn.audio.safetensors stays in place. Prior `elontrn_v2`- bundle backed up to `mlx_models/loras/_backups/20260521_v2_pre_v3/`- before kicking off. Output overwrites `elontrn_v2.safetensors`.- Test render queued for after training completes — sample lands- in `mlx_outputs/` for Mr Bizarro's wake-up review.--### Fixed 2026-05-21 (autonomous modality-matrix session)--Mr Bizarro asked for "continue testing and fixing issues as they arise · check all the other modalities · I will be back in a few hours work continuously". Full modality test pass + analytics scaffold + two real bug fixes + one issue-#12 UX gate. 5 commits to `dev`:--- **A2V died at ~10 s every render with a reference image** (`combined_image_conditionings() missing 1 required keyword-only argument: 'frame_rate'`). Upstream `ltx-pipelines-mlx` v0.14.0 made `frame_rate=` mandatory on `utils._orchestration.combined_image_conditionings`, but `a2vid_two_stage.py` and `lipdub.py` still call it without forwarding the kwarg. Fixed via runtime monkey-patch in `mlx_warm_helper.py::_install_a2v_frame_rate_patch()` — wraps the function with `frame_rate=24.0` default, idempotent, parallel to the existing `_install_video_decoder_patch` pattern. Validated: A2V 1024×576 121f + ref now succeeds in 395 s, output valid h264+AAC. Commit `681f429`.-- **Extend died at the downscale step** (`Unable to choose an output format for '<name>.mp4.partial'; use a standard extension or specify the format manually`). `_ensure_downscaled` writes to `<name>.mp4.partial` for atomic rename, but ffmpeg can't infer mp4 from `.partial`. Added `-f mp4`. Was silent for a while because Extend wasn't being exercised after the .partial pattern was introduced. Commit `736ca0d`.-- **Image Studio let users submit Qwen-Image-Edit jobs when the add-on wasn't installed** (issue #12, sureshkpiitk). Job died deep in the helper with "pip install -U mflux>=0.17" buried in the log. Closes the UX hole Mr Bizarro publicly promised on the issue: `/image/engine_status` now returns `family_installed` per engine; the engine pill turns red with "install Qwen-Image-Edit" + Pinokio-sidebar-button tooltip when missing; Generate button refuses upfront. HiDream engines unaffected. Commit `2694f9f`.-- **Panel boot was silent when helper venv was missing** (issue #5 footnote, claude3d hit this from terminal on M5). Now logs a single stderr warning naming both probed paths (`.venv/bin/python3.11` and `env/bin/python3.11` under `ltx-2-mlx/`) plus the `LTX_HELPER_PYTHON` override. No behavior change otherwise. Commit `fa17c61`.-- **Anonymous opt-in telemetry shipped end-to-end** (commit `d94c72f`). `analytics.py` module + Settings UI toggle + `TELEMETRY.md` field-by-field doc + README mention. Default OFF. Schema covers panel_boot / render_start / render_done / render_failed / render_cancelled / helper_crash / settings_opt_in / settings_opt_out. Strictly sanitized — never sends prompts, paths, hostnames, LoRA filenames. 32-char hex install_id with rotate ("forget me") button. Endpoint via `PHOSPHENE_ANALYTICS_ENDPOINT` env var or default. 4 KB payload cap, 4 s socket timeout, 512-event queue with oldest-drop-on-full, daemon thread, fail-safe when endpoint dead. Privacy-validated end-to-end with a local catcher: zero traffic when toggle OFF (multiple /status hits + boot), expected JSON when ON.--### Modality test results (2026-05-21)--Per the "use real prompts that match the inputs" rule (memory `feedback_phosphene_test_prompts.md`). All tests used a bizarrotrn wood-paneled-room reference frame + matching prompts.--| Modality | Result | Wall | Notes |-|---|---|---|---|-| T2V Character HQ | ✅ | 306 s | `TI2VidTwoStagesHQPipeline`, 1024×576 121f, no hang |-| A2V (after fix) | ✅ | 395 s | 1024×576 121f + ref + voice clip; output h264+AAC 5.01 s |-| FFLF | ✅ | 265 s | Cover-cropped to 768×448 (tier_max_dim); output h264+AAC 5.01 s |-| Extend (after fix) | ⚠️ output OK | denoise 10:23 → decode 16 s → **hangs** | Same post-decode hang as I2V Balanced. Output IS written before hang; killed helper to recover. Documented in ROADMAP. |-| Qwen Edit | ✅ | 1477 s (includes 16-file model download on first run) | 2× 1280×720 PNGs, valid |-| HiDream Medium | ✅ | 239 s | 2× 2560×1440 PNGs, snapped to trained dim |-| I2V Balanced | ❌ (deferred) | — | Known hang, see ROADMAP §post-decode hang |--### Fixed this week-- **Y1.034 silent regression**: VAE temporal-streaming patch tiled even on short clips, adding ~30 s decode tax. Fixed in Y1.037 with auto-mode threshold.-- **Y1.024 Extend regression**: my own download-filter pruned `transformer-dev.safetensors` from Q4 dir, breaking Extend mode for Q4-only installs. Fixed in Y1.036 by routing Extend to Q8.-- **I2V tail stall**: 60+ s frozen final-step on long I2V renders. Fixed in Y1.034 (free DiT before VAE decode).-- **Now-card progress bar mis-paced**: Quick crawled, High blasted past 99%. Fixed in Y1.039 with phase-aware + denoise-step-aware computation.-- **New clips appeared black for 2–3 minutes**: gallery race + browser cache holding incomplete bytes. Fixed in Y1.039 with in-flight skip + mtime cache-bust + `Cache-Control: no-cache`.-- **S2 noir dialogue attribution swap**: "Same thing, honey" delivered by wrong character. Root cause: prompt format diverged from LTX docs. Documented in Linear HAI-152.-- **v2.0.2 install sanity check broken by em-dash**: Pinokio shells mangled the unicode em-dash, triggering Python SyntaxError, falsely failing every install. Fixed in v2.0.4 (ASCII colon).--### Fixed 2026-05-17 (Codex C+ UI restructure + Train Character + LoRA chrome session)--The biggest single-day refactor since v2.0.6. Driven by Codex's C+ recommendation (Q4 panel as separate surface from Q8 panel — see `lora-lab/PHOSPHENE_UX_Q4_VS_Q8_BRIEF.md`). 30+ commits today. Root cause + fix per bug:--- **Player aspect-ratio cropped vertical clips** — `.player-surface` was hardcoded `aspect-ratio: 16/9` + `object-fit: cover`. A 9:16 / 576×1024 clip got head-to-toe cropped. Fixed by reading natural dims on `loadedmetadata` into a `--media-aspect` CSS prop + flipping to height-driven sizing for verticals. Object-fit also switched to `contain` as a safety net. Commit `4987022`.-- **Expand button was inline-positioned (not a modal)** — `.expand-lightbox` had no CSS rules at all. Clicking Expand dropped the `<video>` at native dims inline somewhere on the page. Now styled as a real fullscreen modal (position fixed, dark backdrop, z 200, close button, Esc + backdrop-click already worked). Commit `4987022`.-- **`/output/delete` orphaned the raw mp4** — after an upscale pass, only the clicked file + its sidecar moved to Trash. The hidden native `<base>.mp4` (kept on disk for restore-from-source) stayed. Now collects every companion via sidecar fields + filename heuristic (`UPSCALE_TAGS = (720p, v720p, up2x)`). Commit `0dba2dc`.-- **`/sidecar` 404'd on raw card** — after an upscale, the sidecar lives next to `<base>_720p.mp4`, not `<base>.mp4`. Clicking Load Params on a raw card → 404 → loadParams() bails silently. `/sidecar` now walks the same `UPSCALE_TAGS` family. Commit `331795a`.-- **`/stop` didn't kill training subprocess** — face + audio trainer Popens were inherited into the panel's process group. `/stop` killed HELPER + mux ffmpeg but the trainer ran for hours. Both Popens now use `start_new_session=True`; `STATE["train_pgid"]` tracked; `stop_current_job()` SIGTERMs via killpg with 8s SIGKILL fallback. Commit `b6d1222`.-- **`/queue/batch` silently rendered 5s clips when caller sent `duration=10`** — `make_job` reads `frames` directly from form (default 121 = 5s). The UI JS does duration→frames math client-side, but `/queue/batch` from curl skips that. Now: if `duration` is set + `frames` isn't, derive frames via `_duration_to_8k_frames`. Commit `038a0a1`.-- **Train Character High preset subtitle skew** — UI said `~4 h · rank 32 · 768px` while canonical Python preset is `rank 32 · 5000 steps · 512px · ~2h50m`. Fixed HTML + JS copy. Commit `4255f12`.-- **Dual quality strip rendered on top of each other** — `_applyCharacterQualityStripVisibility()` set `hidden` on the unwanted strip, but `.quality-strip { display: grid }` outranked the user-agent `[hidden] { display: none }` at CSS specificity (0,0,1,0) vs (0,0,0,0). Both strips visible at once. Fixed with `.quality-strip[hidden] { display: none !important }`. Commit `7bd5057`.-- **Train tab voice toggle defaulted OFF** — even when a voice clip was uploaded, users had to manually click the toggle to enable audio training. Now defaults ON; the disabled state until upload still prevents submission without a clip. Commit `ea2cf02`.-- **caption_strategy="user_provided" rejected by trainer** — panel auto-fills missing captions then sets `caption_strategy=user_provided`. lora_lab rejected the value ("unknown caption_strategy") and exited code 1 immediately. Mr Bizarro's Eltrumpo run was the first to hit this (auto-caption produced 37 user .txt files). Fixed in lora-lab (commit `b04eaab` on `feat/train-character-cli`): alias map `'user_provided' → 'class_word'` + `'trigger_simple' → 'class_word'`. Commits in this repo: `8b5a3cf` (the eventual panel-side defense-in-depth).-- **Q4-distilled inference of dev-trained character LoRAs gave generic-Trump output** — character LoRAs were silently routed through the Q4 pipeline when `quality=balanced`, but their training base is the Q8 dev transformer with full sigma schedule + CFG. Fused into distilled = wrong base = identity barely locks. UI now forces Q8 chips when a character is selected; backend rejects `character_id + quality != high`. Commits `1d7983a`, `8b5a3cf`.-- **HQ-speed Fast pill was inactive at boot** — the boot cascade ran `_applyCharacterQualityStripVisibility()` with no character selected → else branch fired `_setSkipStepEnabled(false)` → removed `.active` from the HTML-default Fast pill → first character render missed the Codex 12% optimization. Fixed in two commits: `1056c99` (stopped the row-visibility helper from touching skip-step) and `04d2ffd` (stopped the character cascade else branch from touching skip-step too; HQ-speed pill in Customize is now the single source of truth).--### Fixed in v2.0.6 (May 8 2026)--Image Studio + agent quality pass shipped to `dev` across ~18 commits. Each root cause + fix:--- **klein-4B prompt structure mismatch** — outputs from other tools (Draw Things) at the same model + ref looked visibly better than ours. Two implementation bugs: prompts didn't follow klein-4B's preferred subject → environment → style → technical hierarchy, AND default quantization was Q4. Fixed in `agent/prompts.py` (taught the structure) + Q6 default in `ImageEngineConfig.mflux_quantize`.-- **Q4 stylization artifact** — Apple-Silicon community sweet spot is Q6 (~4-6% quality loss vs Q4's 8-12%). Per-image speed gap is 1-2 s on M4 Max. Q6 is the new default for mflux across the board. Draw Things ships Q5/Q6 by default for the same reason.-- **Image jobs invisible in Now / Queue / Recent / Logs** — `/image/generate` was a synchronous HTTP endpoint that bypassed the queue/worker entirely. Image renders never appeared anywhere in the panel's job-tracking surfaces. Fixed by routing `mode='image'` jobs through `make_job` + a new `run_image_job_inner` that calls `agent_image_engine.generate()` with the same on_log → push() bridge so panel Logs streams live. `/image/generate` kept synchronous for agent callers; `_IMG_STUDIO_LOCK` arbitrates between the two paths.-- **Image Studio's redundant in-pane gallery** — deleted now that the unified Recent tab can render image-mode jobs. Outputs land in the unified gallery alongside videos.-- **Original beach.mp4 "barely moves, just a zoom out" bug** — the agent reused the still prompt verbatim for the i2v video render. Result: zero motion verbs in the i2v prompt, ~4 seconds of an almost-static frame. Fixed by adding a "Phase C — Writing prompts FOR i2v VIDEO renders" section to `agent/prompts.py` that forbids still-prompt reuse, requires explicit motion beats (~1 beat / 2-3 sec), and includes a director-rewrite checklist before each `submit_shot`.-- **The 400×400 still-output mystery** — `flux2_edit` family was referenced in saved configs but never wired into `image_engine.py`. The fam fell through to "flux2" via inference; `mflux-generate-flux2` doesn't take `--image-paths`, so refs were silently dropped → output downsized to ref dim. Fixed by adding `flux2_edit` to `MFLUX_FAMILY_BIN` + `MFLUX_FAMILY_DEFAULTS`, routing refs to `--image-paths` for fam in (`qwen_edit`, `flux2_edit`), `--image-path` (singular) for `kontext`. Also added non-distilled photoreal presets (`flux2_edit_high`, `kontext_high`) — the previous engine_override list was all distilled (illustrative-cartoon look).-- **Issue #2 (Akossimon's Metal abort crash)** — mflux SIGABRTs mid-Metal when 24 GB Qwen-Edit is asked to run with 8 GB free. The Python interpreter can't catch SIGABRT; mflux exits -6 and the panel UI shows nothing. Fixed: pre-flight RAM/disk check refuses jobs that can't fit before mflux launches, AND a SIGABRT detector in `agent/image_engine.py` raises a `RuntimeError` with an actionable OOM hint when mflux reaps -6. Report-bug button gives users a one-click GitHub-issue flow with sysinfo + log tail + .ips bundle.-- **`auto_pause_during_renders` killed mid-batch agent calls** — chat engine got stopped the moment the first job hit the LTX worker, so the agent's next `submit_shot` chat call failed with "Connection error" (in the wild: queued 1 of 5 planned shots). Fixed: new `submit_shots` plural tool batches the entire plan in one dispatch + sets `_finish_after_turn` flag the runtime honors as loop-exit. Prompts.py teaches the agent to use `submit_shots` for any 2+ shot batch.-- **`submit_shot` silently coerced invalid accel values** ("distilled" → "turbo"). Strict validation now rejects unknown values with a clear message; "exact" accepted as a friendly alias for "off".-- **Agent obeyed broken user instructions** (e.g. `aspect="1:1"` for a 16:9 i2v shot → letterboxed → reads as slow zoom). New "push back when the user's instructions will produce broken output" rule in prompts.py rules of engagement. Aspect mismatches surface as questions instead of silent crops.-- **AF pane stayed visible in Manual mode** — toggle was using `[hidden]` but a CSS rule `display: flex` overrode the browser's UA `[hidden] { display: none }`. Sessions chip, Plan/Sleep pill, engine pill, RAM chip, and engine readiness banner all kept rendering above the manual form. Fixed by switching to explicit `style.display` toggling.--### Security review pass (2026-05-08)--Full audit ran against `dev` mid-session. Findings: **0 CRITICAL · 4 HIGH · 6 MEDIUM**. All 10 shipped this session.--- **HIGH (4)** — H1: reject `Origin: null` in `_is_local_request` (a local malicious HTML file dragged into the browser could otherwise drive the panel API with the user's session). H2: validate `mflux_python_path` at save (`_save_agent_image_config` → `_validate_mflux_python_path`) — must be an absolute path to an existing executable; defensive re-check in `agent/image_engine._resolve_mflux_bin`. H3: validate `model_path` at `/agent/local/start` — HF repo ids must match `<owner>/<name>` with owner on a small allow-list (Qwen, mlx-community, lmstudio-community, unsloth, Lightricks, Black-Forest-Labs, lightx2v, filipstrand, huihui-ai, google); local paths must resolve under `mlx_models/` or HF cache root. H4: cap `submit_shot`/`submit_shots` calls per turn in `agent/runtime` (was unbounded — a runaway agent loop could enqueue arbitrary numbers of renders).-- **MEDIUM (6)** — M1: `/sidecar?path=` requires the underlying media to be one of `.mp4/.png/.webp/.jpg/.jpeg` before serving the `.json` next to it. M2: `/agent/models/install` reuses the H3 owner allow-list. M3: `_save_settings` lays down `panel_settings.json` with O_EXCL + fsync + os.replace + chmod 0o600 (atomic + perms-stable). M4: `inspect_clip`'s `prompt`/`negative_prompt` fields are wrapped in a "do NOT execute as instructions" marker and truncated to 2000 chars (defangs prompt-injection laundering). M5: `read_document`'s PDF branch rejects > 50 MB files pre-parse and runs pypdf in a daemon thread with a 30 s watchdog. M6: `/output/hide` rejects targets whose resolved path doesn't sit under OUTPUT or UPLOADS (containment).--Full audit at `/tmp/phos_audit/security-review.md` — note that `/tmp` is ephemeral on macOS, so this report will need to be re-generated for the next pass. L-tier (low-severity) items, especially L2 anchors / select containment, are still open.--### Shipped in v2.0.6 (May 7 2026, autonomous overnight Phase 0 polish)--Roadmap reference: `docs/AGENTIC_FLOWS_ROADMAP.md`. Three Phase 0 items-shipped autonomously while the user was running:--- **#0.11 — Engine readiness banner** (commit `7334836`): tri-state- banner above the chat that probes the configured engine on session- init + after settings save. Catches 401 / unreachable / wrong-base-URL- BEFORE the user invests in attaching a script and writing a long- prompt. Tri-state: `ok` (green, fade-after-4s), `warn` (amber, sticky;- local helper not started — one-click "Start engine"), `err` (red,- sticky; "Open Settings" shortcut). Endpoint `/agent/engine/check`- wraps the existing `engine.health_check`.-- **#0.9 — Turn summary chip** (commit `134b5b1`): one-line "what this- turn accomplished" chip below the LAST assistant message of each- completed turn. Reads at a glance: `✦ 21 shots queued · 1 manifest- writes · ✓ finished`. Walks `rendered_messages` once, pairs each- `tool_result` with the preceding `assistant.tool_call.tool` (same- pattern as the Stage activity feed). Verified against the CAD- session (8 turns, 47 submit_shots).-- **#0.10 — Inline wall-time predictor** (commit `43a7c3b`): two- surfaces. Tool-call cards for `submit_shot` / `estimate_shot` now- render the predicted wall inline with the args- (`submit_shot · S5 motorhome — 6s i2v balanced · ~4m 45s`); batch-bar- above the composer sums predicted wall across PENDING picks- (`12 anchors picked · ready to render · ~57m 24s total`). JS predictor- mirrors `agent/tools.py::_estimate_wall_seconds` byte-for-byte (T^1.5- length scaling, accel discounts, mode/quality table, sharp upgrade);- verified across 6 cases — both implementations match exactly.--Cumulative Phase 0 ships in this rolling polish session (v2.0.5 →-v2.0.6): items #0.5 (`/help` slash + capabilities sheet), #0.6-(a11y floor: `:focus-visible` + `prefers-reduced-motion` +-`role="log"` + `aria-live`), #0.12 (atomic project-notes ring buffer),-#0.11, #0.9, #0.10 — six of twelve Phase 0 rows shipped. Remaining:-#0.1 SSE streaming (L), #0.2 server-side cancel (M), #0.3 recoverable-error cards (M), #0.4 first-run tour (M), #0.7 session restore on-panel restart (M), #0.8 default-engine recommendation banner (S).--### Fixed in v2.0.5 (May 6–7 2026, agentic-flows polish session)-- **Stage NOW RENDERING stuck at 0%**: `current.progress` schema went from flat float to structured object (`{phase, phase_label, pct, elapsed_sec, eta_sec, denoise_step}`). Stage pane was calling `Number(progressObj)` → NaN → 0%. Fixed by reading `.pct` directly with backward-compat fallback to flat float.-- **Offline banner was a 1990s solid-red bar**: floating Phosphene-pill at top-center, pulsing logo icon, three-part text, slide-in animation.-- **Typing indicator stuck on "Drafting plan"**: only re-evaluated when message count grew. Now refreshes every 1.5s with elapsed seconds appended after 6s ("Thinking · 12s") and contextual phrases like "Calling submit_shot", "Queued ce5c, planning next".-- **Auto-scroll fought the user**: every async render did `scrollTop = scrollHeight`, yanking users back when scrolled up. Now uses scroll-pinning — auto-scroll only when within 80px of bottom; "↓ New messages" pill appears otherwise.-- **No abort on long turns**: × button on typing row aborts in-flight fetch via AbortController. Server-side run_turn keeps stepping until it finishes (no kill switch in runtime today); UI is unstuck immediately.-- **Tool cards visually outweighed prose**: 3px accent borders + bold accent-bright + monospace 12px overpowered the assistant's planning text. Now muted 2px, smaller, near-transparent — reads as side metadata.-- **Anchor selection only changeable, never removable**: re-clicking the same anchor un-picks it. Per-grid pick-state badge ("✓ picked" / "click to pick") visible at all times. Backend accepts empty `png_path` as remove signal.-- **No batch trigger**: "Queue them" pill above composer when picks > submitted_shots. One-click injects a structured message authorizing the agent to submit each pick as i2v.-- **Multiple takes lost previous**: same-label calls overwrote candidates. New `append:true` arg writes under `take_NN/` subdir + appends to `takes` array. Chat renders all takes stacked with "Take N / M" labels.-- **Memory crashes from OOM**: macOS suspended Claude.app at 83 GB while LTX rendered + Qwen 35B occupied 22 GB. Memory guard now refuses to auto-spawn engine at >92% pressure or >8 GB swap. Plan-and-sleep mode auto-stops engine after `finish` so RAM goes back to renderer.-- **Reasoning models returned empty content** (CRITICAL — found May 7 1am during overnight session): Qwen 3.6 Abliterated splits output into `message.reasoning` + `message.content`. With `max_tokens=3072`, reasoning consumed budget and content came back empty with `finish_reason="length"` → runtime saw empty assistant message → ended turn. Agent appeared "stuck". Fixed: bumped default to 8192, engine.chat() now reads reasoning, raises actionable error on length truncation, falls back to reasoning when content is empty.-- **Agent identity drift**: avatar was orange "C" reading as generic chat bot. Now circular Phosphene logo (favicon glyph), label "Phosphene". Identity is the brand, not the underlying model.--### Open / partially understood-- **KTDS install case** (Linear HAI-156): user reported `ModuleNotFoundError: No module named 'ltx_pipelines_mlx'` after a "green" install. Symptom is now possibly explained by the v2.0.2/v2.0.3 em-dash bug (the sanity check itself was broken, install completed green for the wrong reason). Asked for log tail + VERSION; pending response. Suggested fix: Reset → Install on v2.0.4.-- **Qwen 3.6 Abliterated reasoning loops** (May 7 overnight session): Qwen 3.6 routinely emits 40k+ char chain-of-thought when planning multi-shot batches in long-context sessions. Even with `max_tokens=12000` and `DEFAULT_TIMEOUT_S=900`, a single turn can exhaust both. The new reasoning-aware engine.chat() raises an actionable error pointing the user to bump max_tokens, but for a recursive thought spiral no token budget is enough. Workarounds for the next session:- - **Recommend Gemma 12B for the agent** when the task is "queue 20+ shots". Already on disk at `mlx_models/gemma-3-12b-it-4bit`, no reasoning blocks, responds directly. 7.5 GB resident vs Qwen's 22 GB.- - **Or Anthropic API (claude-sonnet-4-5)** — paid, fast, no local reasoning block.- - **Smaller batches**: trigger 5 shots at a time in plain text (avoids long planning turns). Confirmed working tonight: agent submitted S1-S5 (News Intro, Wife LLM Monitor, Inpatient Toaster, Wife Coffee Variables, Psychiatrist Uptime) in one focused trigger before hitting the next reasoning-loop.- - **System-prompt injection** to discourage over-thinking: "Do not deliberate; emit the action block immediately." Untested as of 2026-05-07; worth a try.- - Future: detect reasoning-token consumption mid-stream + tool-loop checkpoint after each submit_shot so a long-running planner doesn't lose its place.+## 7. Known bugs++> **Source of truth for bug state: `/Users/salo/AI/projects/phosphene/notes/DEEP_REVIEW_2026-05.md`** (the deep stabilization review — full verified risk register + phased plan). The list below is the short reconciliation; the review has the complete severity-ranked register and the fix directions. The CHANGELOG further down is the historical record of what was fixed when — not a list of live bugs.++### Currently open++Reconciled against the deep review (2026-05-31). Everything that was a recent fire — the I2V/Extend post-decode hang (`94bd696`), FFLF crash (v3.0.1), Boost/Turbo regression (v3.0.2), A2V `frame_rate` kwarg (v3.0.5), the HDR NameError / CivitAI token leak / GPU contention (v3.0.6) — is **fixed**. The HiDream no-deadline reader now has a panel-side watchdog too (v3.0.6). What genuinely remains:++**Root-cause fragility (deep review §2, top of the register)**+- **Runtime-monkey-patch / version-skew axis** — the panel + helper patch a moving upstream (`ltx-2-mlx`) at runtime. v3.0.6 added a boot-time version gate (Phase 0); the structural fix (retire the runtime-patch class, re-pin to a SHA-pinned submodule) is Phase 3 of the plan, not yet done.+- **No test seam** — `mlx_ltx_panel.py` is ~28k lines, all 69 routes dispatched via a flat `if path==` chain with zero unit-testable surface. This is *why* regressions reach users. Phase 2 carves the first seams.++**Confirmed correctness / robustness items still open**+- **Recipe-override guardrails** — advanced LoRA-training overrides (rank/lr/steps) bypass the validated-recipe clamps (`train_character.py:135-144`, panel `:4807-4821`). Needs whitelist + a "non-standard recipe" warning.+- **`/status` polling cost** — every poll, per open tab, does a pgrep subprocess + filesystem scans (`mlx_ltx_panel.py:7845-7919`). Split fast fields from slow install-state probes / cache the slow group.+- **154 silent `except: pass` sites**, including persistence + chmod paths (`mlx_ltx_panel.py`). Tier them (best-effort vs state/security vs control-flow) behind a greppable `_swallow(label)` helper.+- Plus a tail of Med/Low items in the review (HiDream preflight exemption, `/prompt/enhance` shares the render lock, stats JSONL unbounded/growth, `character_runtime.py` dead/divergent, BFL URL host-validation, voice-silence trim). See the review's §2 table for the full set + fix directions.++**Model-capability limits (not bugs — won't be "fixed", design around them)**+- **Multi-shot character continuity is naive-failure** — same prompt + different seed = different person. IC-LoRA (deep review §6) is the proposed lever.+- **Faces below ~80 px in-frame** identity-break; **hands / held objects** and **high-motion physics** are out of distribution. See §6.++**Agent caveat (carried over)**+- **Qwen 3.6 reasoning loops** when planning large multi-shot batches — recursive chain-of-thought can exhaust any token budget. Workarounds: prefer Gemma 12B (`mlx_models/gemma-3-12b-it-4bit`, no reasoning blocks, 7.5 GB) or the Anthropic API for 20+ shot batches; trigger 5 shots at a time in plain text. (Note: the in-panel agentic-flows chat was retired in v3.0.0; this applies to the remaining `/prompt/enhance` + any external-engine use.)+- **KTDS install case** (Linear HAI-156): `ModuleNotFoundError: ltx_pipelines_mlx` after a "green" install. Likely the old v2.0.2/v2.0.3 em-dash sanity-check bug (install went green for the wrong reason); fixed in v2.0.4. The v3.0.6 boot version-gate + `--force-reinstall` install determinism should close this class. Pending the user's log tail to confirm.++---+### CHANGELOG (historical — what was fixed, newest first)++#### v3.0.6 — deep-review hardening+- **HDR action un-deaded** — `generate_hdr` referenced undefined `HELPER_LOW_MEMORY` → NameError killed every HDR job. One-token rename to `LOW_MEMORY` (`mlx_warm_helper.py`).+- **CivitAI token-leak fix** — downloader leaked the API token on redirect + used a weak `endswith("civitai.com")` host check. Exact-host allowlist + redirect handler that strips `Authorization`; prefer header over `?token=`.+- **GPU-contention guard** — inline `/image/generate` and the video worker shared no GPU lock; a concurrent mflux + LTX render could OOM the Mac. Now mutually exclusive.+- **Boot version-gate** — helper reads `ltx_pipelines_mlx` version at boot, compares to the expected v0.14.0 pin, and surfaces a loud panel-log banner on mismatch (Phase 0 of the stabilization plan).+- **HiDream `select()`+deadline reader** — the HiDream subprocess reader had no deadline (a hung render blocked the queue forever); now reuses the mflux deadline+`killpg` loop. (HiDream is dropdown-hidden but reachable via saved config / `engine_override`.)++#### v3.0.5 — A2V kwarg signature shim (issue #5)+- **A2V died ~10 s into every render with a reference image** (`combined_image_conditionings() missing 1 required keyword-only argument: 'frame_rate'`). Upstream v0.14.0 made `frame_rate=` mandatory, but `a2vid_two_stage.py` / `lipdub.py` don't forward it. Fixed via runtime monkey-patch `_install_a2v_frame_rate_patch()` (`frame_rate=24.0` default, idempotent). Commit `681f429`. (The shim is still required at v0.14.8 — the upstream bug is live; deep review flags hardcoding 24.0 vs the real fps as a Med item.)++#### v3.0.4 — CivitAI SSL fix++#### v3.0.3 — HiDream hidden (issue #15)+- HiDream removed from the visible Image Studio dropdown until its lab repo is public. Code stays; reachable via saved config / `engine_override`.++#### v3.0.2 — Boost/Turbo accel restored (2-month silent regression)+- Git-archaeology'd a regression where the Boost/Turbo accel path silently stopped firing. Re-anchored the accel `denoise_loop` replacement across all import sites + restored the per-mode tail. (Commit `2694f9f` shipped the issue-#12 install gate earlier; accel fix is the v3.0.2 headline.)++#### v3.0.1 — FFLF crash fix+- **Extend downscale crash** — `_ensure_downscaled` wrote to `<name>.mp4.partial`; ffmpeg can't infer mp4 from `.partial`. Added `-f mp4`. Commit `736ca0d`.+- **Image Studio submitted Qwen-Image-Edit jobs when the add-on wasn't installed** (issue #12). `/image/engine_status` now returns `family_installed` per engine; the engine pill turns red with an install tooltip; Generate refuses upfront. Commit `2694f9f`.+- **Silent panel boot when helper venv missing** (issue #5 footnote) — now logs a single stderr warning naming both probed paths + the `LTX_HELPER_PYTHON` override. Commit `fa17c61`.++#### v3.0.0 (May 23) — marquee release+The Characters / Voice / Image Studio / A2V release. Folded in the May-17 Codex C+ UI restructure, the Train-tab + LoRA-chrome work, the Image Studio overhaul, and the post-decode-hang fix. The in-panel agentic-flows chat was retired here.++**Stats dashboard — panel-internal (private).** The dashboard is served by the panel at **`http://127.0.0.1:8199/stats`** (127.0.0.1-only; panel must be running). Data lives at **`state/stats-data.jsonl`** — gitignored, on the user's Mac only, never on the public repo. `panel_assets/stats.html` holds the template (public code; only the data is private). Panel background thread `stats_fetch_loop` runs `scripts/fetch_repo_stats.py` once at startup (if data is missing/stale ≥ 6h) and daily thereafter. Token resolution (first hit wins): `PHOSPHENE_REPO_STATS_TOKEN` → `GH_STATS_TOKEN` → `GH_TOKEN` / `GITHUB_TOKEN` → `gh auth token`; skipped silently if none. Zero setup for the user — just open `/stats`. See `scripts/STATS_DASHBOARD.md`. *(An earlier GitHub-Pages + committed-JSONL dashboard and an opt-in analytics module were both rolled back before launch; analytics removed entirely in `da1d6f5`. The brief window where stats data touched the public repo (`151d0d2`..`827c5d8`) held one snapshot of public aggregate counts only — nothing private leaked.)*++**Post-decode hang FIXED** (commit `94bd696`). A first attempt at an in-helper daemon-thread watchdog (`adc1cd2`) did not fire in practice — Metal's command-buffer completion handlers block every Python thread's GIL during the deallocator chain, so the watchdog was starved by the very thing it was meant to escape. Working fix rescues from **the panel** (separate process, GIL irrelevant): `WarmHelper._build_post_decode_panic` returns a `(log_hook, panic_check)` pair; `log_hook` spots upstream's `[Decoding ... done in X.Xs]` and arms a 45s grace clock; `panic_check` runs every 500ms and, if grace expired + output file on disk > 8KB, SIGKILLs the helper and returns a synthetic done event. Helper respawns on the next job (~30s). Armed for `generate` (T2V/I2V Balanced) + `extend` only. Validated on the 768×416 +6f Extend that previously hung 5-13 min. Bundled: Extend default steps 12→8 + TeaCache threshold 0.5→0.7 (~6 min wall); gallery `_dnWxH` dn-cache leak fix (the spurious "21:19" duration label).++**Codex C+ UI restructure (2026-05-17, 30+ commits).** Driven by Codex's C+ recommendation (Q4 vs Q8 as separate surfaces). Per-bug:+- Player aspect-ratio cropped vertical clips (`.player-surface` hardcoded 16:9 + `object-fit:cover`) → read natural dims on `loadedmetadata` into `--media-aspect`, height-driven sizing for verticals, `object-fit:contain`. Commit `4987022`.+- Expand button was inline-positioned, not a modal (`.expand-lightbox` had no CSS) → real fullscreen modal. Commit `4987022`.+- `/output/delete` orphaned the raw mp4 after upscale → collects every companion via sidecar fields + `UPSCALE_TAGS` heuristic. Commit `0dba2dc`.+- `/sidecar` 404'd on a raw card after upscale → now walks the `UPSCALE_TAGS` family. Commit `331795a`.+- `/stop` didn't kill the training subprocess (trainer Popens inherited the panel's process group) → `start_new_session=True`, `STATE["train_pgid"]` tracked, SIGTERM via killpg + 8s SIGKILL fallback. Commit `b6d1222`.+- `/queue/batch` rendered 5s clips when curl sent `duration=10` (client-side duration→frames math skipped) → derive frames via `_duration_to_8k_frames`. Commit `038a0a1`.+- Train Character High preset subtitle skew (`~4 h · 768px` vs canonical `5000 steps · 512px · ~2h50m`). Commit `4255f12`.+- Dual quality strip rendered on top of each other (`.quality-strip{display:grid}` outranked UA `[hidden]`) → `.quality-strip[hidden]{display:none!important}`. Commit `7bd5057`.+- Train voice toggle defaulted OFF even with a clip uploaded → defaults ON. Commit `ea2cf02`.+- `caption_strategy="user_provided"` rejected by the trainer → alias map in lora-lab (`b04eaab`); panel-side defense-in-depth `8b5a3cf`.+- Q4-distilled inference of dev-trained character LoRAs gave generic output (wrong base) → UI forces Q8 chips for characters; backend rejects `character_id + quality != high`. Commits `1d7983a`, `8b5a3cf`.+- HQ-speed Fast pill inactive at boot (boot cascade cleared `.active`) → commits `1056c99`, `04d2ffd`; the HQ-speed pill in Customize is now the single source of truth.++**Image Studio + agent quality pass (the "v2.0.6" codename work, ~18 commits).** Per-bug:+- klein-4B prompt-structure mismatch + Q4 default → taught the subject→environment→style→technical hierarchy in `agent/prompts.py`; Q6 default in `ImageEngineConfig.mflux_quantize`.+- Image jobs invisible in Now/Queue/Recent/Logs (`/image/generate` bypassed the queue) → routed `mode='image'` through `make_job` + `run_image_job_inner`; `_IMG_STUDIO_LOCK` arbitrates the sync agent path.+- Redundant in-pane Image Studio gallery deleted (unified Recent tab covers it).+- i2v "barely moves, just a zoom out" (agent reused the still prompt) → "Phase C — writing prompts FOR i2v" rules in `agent/prompts.py` (forbid still-prompt reuse, require explicit motion beats).+- 400×400 still-output mystery (`flux2_edit` referenced in saved configs but never wired) → added to `MFLUX_FAMILY_BIN`/`MFLUX_FAMILY_DEFAULTS`, refs routed to `--image-paths`; added non-distilled photoreal presets. *(FLUX/Klein later removed 2026-05-13.)*+- Issue #2 (Metal abort crash) — mflux SIGABRTs (exit -6, uncatchable) when 24 GB Qwen-Edit runs with 8 GB free → pre-flight RAM/disk check + SIGABRT detector with an OOM hint + Report-bug button.+- `auto_pause_during_renders` killed mid-batch agent calls → `submit_shots` plural tool batches the whole plan + a `_finish_after_turn` flag.+- `submit_shot` coerced invalid accel values silently → strict validation; "exact" accepted as a friendly alias for "off".+- Agent obeyed broken user instructions (e.g. `aspect="1:1"` on a 16:9 i2v) → "push back when instructions produce broken output" rule.+- AF pane stayed visible in Manual mode (`[hidden]` overridden by `display:flex`) → explicit `style.display` toggling.++**Security review pass — 0 CRITICAL · 4 HIGH · 6 MEDIUM, all 10 shipped.**+- HIGH — reject `Origin: null` in `_is_local_request`; validate `mflux_python_path` at save; validate `model_path` at `/agent/local/start` (HF `<owner>/<name>` against an owner allow-list; local paths must resolve under `mlx_models/` or HF cache); cap `submit_shot`/`submit_shots` calls per turn.+- MEDIUM — `/sidecar?path=` requires a media file before serving the `.json`; `/agent/models/install` reuses the owner allow-list; `_save_settings` writes with O_EXCL + fsync + os.replace + chmod 0600; `inspect_clip` prompt fields wrapped + truncated (prompt-injection defang); `read_document` PDF branch rejects > 50 MB + 30 s watchdog; `/output/hide` containment check.++**Agentic-flows polish (the "v2.0.5" codename work).** Stage stuck at 0% (progress schema changed flat-float → object); offline banner restyled; typing indicator now refreshes every 1.5s with elapsed seconds; auto-scroll switched to scroll-pinning + "↓ New messages" pill; abort on long turns via AbortController; tool cards de-emphasized; anchor un-pick on re-click; "Queue them" batch pill; multi-take `append:true`; OOM memory guard (refuse engine auto-spawn >92% pressure / >8 GB swap); **reasoning-model empty-content fix** (Qwen 3.6 splits `reasoning`/`content`; bumped `max_tokens` 3072→8192, engine.chat() reads reasoning, raises on length truncation); Phosphene-branded assistant avatar. Also three Phase 0 agentic items: engine-readiness banner (`7334836`), turn-summary chip (`134b5b1`), inline wall-time predictor (`43a7c3b`).++#### Semver 2.x line (May 3–6)+- **v2.0.0** (May 3) — marquee release; semver versioning starts.+- **v2.0.1** — Spicy mode toggle gates NSFW LoRA visibility.+- **v2.0.2** — install fails loud when pipeline packages are missing.+- **v2.0.3** — install log self-documents the Python toolchain.+- **v2.0.4** (May 5) — strip em-dash from the install.js sanity check (Pinokio shells mangled the unicode em-dash → false SyntaxError → every install failed). Pure ASCII now.+- **v2.0.5** (May 6) — drop the `print('venv OK: ...')` decoration from the sanity-import step (a user-side rewriter was cutting `OK:` out of the string and appending `OK` after the shell quote). The import exit code is the only success signal needed.++#### Pre-semver Y1.NNN line (Apr 28 – May 3)+- **Y1.001 → Y1.013** — first usable T2V/I2V; audio SHIP-BLOCKER fixed by pinning `mlx==0.31.1` (0.31.2 attenuated the vocoder by 22 dB).+- **Y1.014 → Y1.024** — hardware-tier system, Boost/Turbo speed modes, CivitAI LoRA browser, Q8 two-stage HQ tier, FFLF + Extend, `hf_transfer`, Q4/Q8 download filter (~80 GB saved on existing installs).+- **Y1.025 → Y1.035** (Codex-led) — Sharp upscale (PiperSR), I2V tail-stall fix (`Y1.034`, free DiT before VAE decode), VAE temporal-streaming (`Y1.035`), license/install hardening.+- **Y1.036 → Y1.039** — fixed the `Y1.024` Extend regression (route to Q8), VAE auto-streaming threshold (recovered ~7% on short clips; the `Y1.034` patch had tiled even short clips for a ~30 s tax), Now-card progress-bar rewrite (phase + denoise-step aware), gallery black-frame race fix.+- **S2 noir dialogue attribution swap** — wrong character delivered "Same thing, honey"; root cause was prompt format diverging from LTX docs. Linear HAI-152.## 8. Open work / future directionEverything below is also tracked in Linear (HAI-150 → HAI-158 under the Phosphene project). This section duplicates the most current state for fast scan.@@ -727,7 +563,7 @@ research-grade work on token merging.- **Apple Silicon (M1+) only**. No PyTorch, no CUDA, no MPS shim. Native MLX or it doesn't ship.- **Joint audio + video must remain**. That's the differentiator vs Wan / Hunyuan / Mochi. We don't drop audio for length.- **Existing queue + helper + patch architecture stays intact**. No new microservices.-- **Branch policy**: dev → main only via fast-forward, only with Mr Bizarro's explicit OK.+- **Branch policy** (post-2026-05-22 split, see §1): there is **no public `dev`** branch — it was deleted. Daily work goes to private `beta/main` (the local `dev` clone tracks it). **Promotion to PUBLIC `main` is the gated step — only with Mr Bizarro's explicit OK.**- **Mr Bizarro's voice in writing**: copy-edit, don't rewrite. See memory file `feedback_copy_edit_dont_rewrite.md`. Tweets, posts, README copy — fix typos and grammar, never restructure or stack value-prop language.## 10. Memory pointers (for next-Claude)diff --git a/docs/launch_thread_3.0.md b/docs/launch_thread_3.0.mdindex 1c0bada..d3b53b3 100644--- a/docs/launch_thread_3.0.md+++ b/docs/launch_thread_3.0.md@@ -1,5 +1,11 @@# Phosphene 3.0 launch thread — draft+> 📌 **Snapshot: this was the v3.0.0 launch draft (2026-05-18).** The Image+> Studio engine roster has changed since — HiDream was later hidden (issue #15)+> and FLUX was removed. Tweets 5 and 7 describe engines that no longer ship as+> stated. See `docs/STATE.md` for the current engines. This file is preserved as+> historical launch copy; the tweets are not edited.+Twelve tweets. Each ≤ 280 chars. Bracketed `[MEDIA]` notes mark where Mr Bizarro plugs a clip or screencast. Before/after framing per Mr Bizarro's brief — everything claimed here is verified against the panel as it ships.The thread should run on `@PhospheneAI` (and optionally cross-posted to `@AIBizarrothe`).