Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
docs: add docs/STATE.md — session-start handoff for the project
mrbizarro4121cddeb0ef
diff --git a/docs/STATE.md b/docs/STATE.mdnew file mode 100644index 0000000..8cc6e3f--- /dev/null+++ b/docs/STATE.md@@ -0,0 +1,297 @@+# Phosphene — project state, history, open work++Current version: **v2.0.4** on `dev` and `main` (commit `74c7bd1`, May 5 2026).+Live URL: `https://github.com/mrbizarro/phosphene` · Linear project: `https://linear.app/hairstylemojo/project/phosphene-9c11240704bb`++This doc is the **session-start handoff**. A new Claude window entering this project should read this first, then `CLAUDE.md` (architecture), then the relevant Linear issues.++---++## 1. Where the code lives++Two clones on Salo's Mac, both managed by Pinokio:++| Path | Branch tracked | Port | Role |+|---|---|---|---|+| `/Users/salo/pinokio/api/phosphene-dev.git/` | `dev` | 8199 | Active development. Most edits land here first. |+| `/Users/salo/pinokio/api/phosphene.git/` | `main` | 8198 | Production / daily driver. Salo's actual usage. |++GitHub is the source of truth (memory: `feedback_github_source_of_truth.md`). Branch policy is strict: **NEVER push to `main` without Salo's explicit OK** (memory: `phosphene_dev_workflow.md`). When ready to promote, fast-forward only — `git merge --ff-only dev` from main, never a merge commit.++State directories that live OUTSIDE the repo via Pinokio's `fs.link`:++- `mlx_models/` → ~63 GB of LTX 2.3 weights (Q4, Q8, Gemma encoder, PiperSR upscaler). Shared between dev and prod via symlink chain.+- `mlx_outputs/` → all rendered mp4s + sidecar JSON files.+- `panel_uploads/` → user-uploaded reference images for I2V / FFLF.+- `state/` → `panel_settings.json`, `panel_queue.json`, `panel_hidden.json`. Survives a Pinokio Reset.++A Pinokio Reset wipes the install dir but preserves all four — Salo can Reset → Install without losing renders or settings.++## 2. Current capabilities (v2.0.4)++**Modes**+- T2V — text → video+- I2V — image → video+- FFLF (keyframe) — first/last frame interpolation+- Extend — append seconds onto an existing clip++**Quality dial**+- Quick · Balanced · Standard · High (Q8 two-stage HQ + TeaCache)++**Speed dial**+- Exact · Boost · Turbo (adaptive cached denoise)++**Sharp upscale**+- PiperSR on the Apple Neural Engine, optional install via `install_sharp.js`++**Joint audio + video**+- Synced lip movement, footsteps, ambient bed (mlx 0.31.1 pin holds the audio fix)++**Hardware tier system**+- Compact / Comfortable / Roomy / Studio with per-tier feature gating+- Reference benchmarks throughout this doc are on **M4 Max 64 GB** (Comfortable tier)++**Other**+- CivitAI LoRA browser built-in+- Spicy mode gate (NSFW LoRAs hidden by default, opt-in toggle in Settings)+- Per-job progress bar (phase-aware, denoise-step-aware)+- Gallery with cache-bust URLs, no more black-clip race+- 80+ GB less disk than pre-Y1.024 installs (filtered hf downloads)++## 3. Marquee benchmarks (M4 Max 64 GB, sidecar-measured)++| Recipe | 5 sec | 10 sec | 20 sec |+|---|---|---|---|+| T2V Balanced + Turbo + 720p Sharp | 3:30 | 8:07 | 21:38 |+| T2V Quick + Turbo + 720p Sharp | — | — | 10:32 |+| T2V Standard 1280×704 Exact | 7:40 | — | — |+| T2V Standard Turbo | 5:26 | — | — |+| T2V High Q8 (max quality, no Sharp) | 11:51 | — | — |+| I2V Balanced + Turbo + Sharp | 3:37 | 8:26 | — |+| Extend +3 s on Q8 dev (768 px clamp) | — | 15:50 | — |+| FFLF (clamped 768×416, Comfortable tier) | — | 5:29 | — |++Per-step cost scales **~T^1.5** with frame count (218 s/step at 481f vs ~30 s/step at 121f, same width). Sub-quadratic — confirms LTX uses windowed/factorized attention. **20-second single clips are production-viable**; 30 sec at 1024×576 is plausible, 60 sec needs lower res or research breakthrough.++## 4. Version history (compressed)++Pre-2.0 was the `Y1.NNN` sequential counter. v2.0.0 cut over to semver on May 3 2026.++**Y1.001 → Y1.013** (Apr 28–30) — First usable T2V/I2V renders. Audio SHIP-BLOCKER fixed by pinning `mlx==0.31.1` (0.31.2 attenuated vocoder by 22 dB).++**Y1.014 → Y1.024** — Hardware-tier system, Boost/Turbo speed modes (adaptive denoise caching), CivitAI LoRA browser, Q8 two-stage HQ tier, FFLF + Extend modes, `hf_transfer` downloads, Q4/Q8 download filter (saved ~80 GB on existing installs via `update.js` trim).++**Y1.025 → Y1.035** (Codex-led arc) — Sharp upscale via PiperSR (Apple Neural Engine), I2V tail-stall fix (`Y1.034` free DiT before VAE decode), VAE temporal-streaming for long clips (`Y1.035`), license / install hardening, Spicy mode gate prep.++**Y1.036 → Y1.039** — Fixed `Y1.024` Extend regression (route to Q8 dev transformer), VAE auto-streaming threshold (recovered ~7 % on short clips), Now-card progress bar rewrite (phase + denoise-step aware), gallery black-frame race fix.++**v2.0.0** (May 3) — Marquee release. 2.0 badge in panel header, semver versioning starts.+**v2.0.1** — Spicy mode toggle gates NSFW LoRA visibility.+**v2.0.2** — Install fails loud when pipeline packages are missing (sanity-import step in `install.js`).+**v2.0.3** — Install log self-documents Python toolchain (uv version, system python presence, post-pip site-packages list).+**v2.0.4** (May 5) — Strip em-dash from install.js sanity check. Was breaking install on some Pinokio shells (KTDS + second user hit identical SyntaxError). Pure ASCII now.++Full git log on `main`. Tags `v2.0.0` through `v2.0.4` published.++## 5. The folder layout++```+phosphene-dev.git/+├── pinokio.js / pinokio.json ← Pinokio menu logic + manifest+├── install.js / update.js ← idempotent install / update flows+├── install_sharp.js ← optional PiperSR Sharp installer+├── download_q8.js ← optional Q8 weights download+├── download_upscaler.js ← optional spatial upscaler download+├── start.js ← Pinokio start script (launches the panel)+├── reset.js ← Pinokio reset script+├── recover.sh ← rare-case manual recovery+│+├── mlx_ltx_panel.py ← the panel HTTP server (~9000 lines, single file)+│ ├── /status, /queue/*, /run, /upload, /file, /civitai/*, /loras, /settings ...+│ ├── HTML+CSS+JS for the UI all inlined as page() string+│ └── Worker thread + helper subprocess management+│+├── mlx_warm_helper.py ← persistent helper subprocess (~1300 lines)+│ ├── Loads + holds T2V/I2V/Extend/HQ/Keyframe pipelines from ltx_pipelines_mlx+│ ├── Reads job specs from stdin, emits events to stdout+│ └── action types: generate / generate_keyframe / extend+│+├── patch_ltx_codec.py ← idempotent runtime patches against installed+│ ├── Patch 1: codec → yuv444p crf 0 + faststart (lossless H.264)+│ ├── Patch 2: I2V free DiT before decode (matches T2V cleanup)+│ ├── Patch 3: free vae_encoder pre-denoise (peanut review)+│ ├── Patch 4: free feature_extractor in base load() (peanut review)+│ └── Patch 5: VAE temporal streaming decode (long clips no longer freeze)+│+├── required_files.json ← single source of truth for "installed"+├── VERSION ← read by panel + version-check loop+├── .env.local ← LINEAR_API_KEY (gitignored, chmod 600)+│+├── README.md ← user-facing docs (homepage on GitHub)+├── CLAUDE.md / AGENTS.md / GEMINI.md / QWEN.md+│ ← agent manuals (architecture, conventions, history)+├── docs/ ← long-form internal docs+│ ├── STATE.md ← this file+│ └── SDK_KEYFRAME_INTERPOLATION.md ← multi-keyframe interpolation design + plan+├── launch/ ← marketing copy (Pinokio article, X thread, Reddit, etc.)+│+├── ltx-2-mlx/ ← upstream MLX port (separate clone of dgrauet/ltx-2-mlx)+│ └── env/ ← Python 3.11 venv (uv-managed)+├── mlx_models/ ← weights (~63 GB, fs.link symlink)+├── mlx_outputs/ ← rendered mp4s + sidecars (fs.link symlink)+├── panel_uploads/ ← user reference images (fs.link symlink)+├── state/ ← panel_settings/queue/hidden.json (fs.link symlink)+├── cache/ ← HF_HOME for downloads+└── logs/ ← Pinokio's own command-execution logs+```++`mlx_ltx_panel.py` is the heart of it — almost all panel behavior lives there. `mlx_warm_helper.py` is the long-running inference subprocess. `patch_ltx_codec.py` is a 500-line runtime modifier that fixes upstream code without forking it.++## 6. What worked / didn't this session (May 3–5 2026)++### Cinematic capability findings (from rendering ~30 clips)++**The model's wheelhouse**+- Human cinematic moments. Faces at medium and tighter, body language, atmospheric scenes.+- Static or near-static camera works better than moving camera.+- 2–3 dialogue turns per clip work cleanly when prompt follows LTX's docs literally:+ - Single continuous paragraph (NOT uppercase character cards)+ - Voice descriptor on every speech beat (not just first)+ - Single quotes around dialogue+ - Action density ~1 explicit beat per 2–3 sec of clip+- Joint audio + video really IS jointly diffused — lip-sync is uncannily tight.++**The model's weaknesses (avoid in prompts)**+- **Hands and held objects** — fingers morph, written text squiggles, pen/needle/cup interactions look off.+- **High-motion physics** — skater kickflips, water splash, motorcycle blur are out of distribution.+- **Faces below ~80 px in-frame size** — model fills a face-shape but identity-broken. Wide shots of single characters are unusable in their first/last seconds. ([Salo's discovery May 4](#))+- **Multi-shot continuity is naive-failure** — same prompt + different seed = different person. The mom-kid scene experiment (M1 / M2 / M3 in `mlx_outputs/`) confirmed three different women across three angles despite identical character description.++### What earns 20 seconds+- 6–9 explicit beats described in the prompt. Anything less and the model fills with stasis.+- Static or near-static camera. Camera motion costs visual coherence.+- Specific named actions ("she turns slowly", "she breathes out", "the streetlight flickers off") give anchor points.++### Empirical experiment outcomes++- **M1/M2/M3 mom-kid trio** (1024×576, Balanced + Turbo + Sharp, ~21 min each): demonstrated multi-shot character drift problem. Three different women across three angles.+- **N1–N10 cinematographic moments** (May 4): ten 20-sec clips at varying shot scales. Tested medium / wide / two-shot composition with body-language-only prompts (no hands, no held objects). Output quality varied; faces are stable when in the safe pixel range.+- **E-DRAFT** (May 4): tested low-res draft → high-res commit hypothesis. Same prompt + seed at 640×480 vs 1024×576. Salo: low-res output not usable due to face-distance issue. Premise was flawed because lower res = worse faces.+- **E-ANCHOR** (May 4): I2V from M1 frame to test character anchoring. Result was inconclusive in the session; final clip is at `mlx_outputs/` if needed for review.+- **20-sec single-clip viability** (May 4): confirmed at Balanced 1024×576 + Turbo + Sharp. ~21 min wall, audio synced, characters stable.++## 7. Known bugs / fixed bugs++### Fixed this week+- **Y1.034 silent regression**: VAE temporal-streaming patch tiled even on short clips, adding ~30 s decode tax. Fixed in Y1.037 with auto-mode threshold.+- **Y1.024 Extend regression**: my own download-filter pruned `transformer-dev.safetensors` from Q4 dir, breaking Extend mode for Q4-only installs. Fixed in Y1.036 by routing Extend to Q8.+- **I2V tail stall**: 60+ s frozen final-step on long I2V renders. Fixed in Y1.034 (free DiT before VAE decode).+- **Now-card progress bar mis-paced**: Quick crawled, High blasted past 99%. Fixed in Y1.039 with phase-aware + denoise-step-aware computation.+- **New clips appeared black for 2–3 minutes**: gallery race + browser cache holding incomplete bytes. Fixed in Y1.039 with in-flight skip + mtime cache-bust + `Cache-Control: no-cache`.+- **S2 noir dialogue attribution swap**: "Same thing, honey" delivered by wrong character. Root cause: prompt format diverged from LTX docs. Documented in Linear HAI-152.+- **v2.0.2 install sanity check broken by em-dash**: Pinokio shells mangled the unicode em-dash, triggering Python SyntaxError, falsely failing every install. Fixed in v2.0.4 (ASCII colon).++### Open / partially understood+- **KTDS install case** (Linear HAI-156): user reported `ModuleNotFoundError: No module named 'ltx_pipelines_mlx'` after a "green" install. Symptom is now possibly explained by the v2.0.2/v2.0.3 em-dash bug (the sanity check itself was broken, install completed green for the wrong reason). Asked for log tail + VERSION; pending response. Suggested fix: Reset → Install on v2.0.4.++## 8. Open work / future direction++Everything below is also tracked in Linear (HAI-150 → HAI-158 under the Phosphene project). This section duplicates the most current state for fast scan.++### Multi-keyframe interpolation as SDK shot-composition primitive++**See:** `docs/SDK_KEYFRAME_INTERPOLATION.md` (full design + research review).++**TL;DR**: ComfyGuy9000 demoed first-frame-last-frame method via `Deno2026/comfyui-deno-custom-nodes`. Phosphene's `ltx_pipelines_mlx.KeyframeInterpolationPipeline` already accepts arbitrary `list[Image]` keyframes + `list[int]` indices — but our panel/helper artificially restrict it to 2 keyframes (start + end). Exposing the full multi-keyframe API gives us the agentic-flow compositional primitive: agent picks N stills, model fills the motion, character is anchored at every shot start.++**Implementation order**: helper (3 lines) → panel worker (form parsing) → UI (multi-keyframe drop-zone list).++### Long-video research (Strategy A / B / C)++Goal: 1-minute final video on M4 Max 64 GB, ~40-60 min wall time acceptable.++- Strategy A — push single LTX clip beyond 10 sec. 20-sec proven at 1024×576 + Turbo + Sharp. 30-sec untested; 60-sec needs research.+- Strategy B — Extend chaining. ~16 min per +3 s pass, ~4.5 h total for 1-min. Audio continuous.+- Strategy C — multi-scene assembly via LLM-driven shot-list planner. ~42-49 min total, hides cuts cinematically. **This is what the multi-keyframe SDK enables.**++Codex deep-research brief drafted; awaiting return for literature review on FreeNoise / FIFO-Diffusion / StreamingT2V applicability.++Salo also has Claude.ai / ChatGPT deep-research running (May 5) on inference speed without quality loss.++### Director Mode (agent workflow)++Future feature for 1-minute movies from one user idea:++1. Planner (Gemma 3 12B already on disk, or external LLM) expands one-line concept to shot-list with style guide+2. Submitter POSTs each shot to `/queue/add`+3. Stitcher concatenates the K resulting mp4s with ffmpeg+4. Director UI in the panel — paste idea, see planned shot-list, edit any prompt, hit "Render Movie"++Blocked on multi-keyframe interpolation shipping (the compositional primitive) + long-video research deliverable (per-shot length sweet spot).++### Speed optimization candidates (from May 4 research session)++Ordered by what to try first:++1. **Two-stage workflow: draft + commit** — render 5-sec at full res first, then 20-sec same seed if approved. ~6× faster iteration. Replaces the failed "low-res draft" idea (faces don't survive res drop).+2. **Skip Sharp on batch testing** — ~26-100 s saved per clip during iteration.+3. **Pre-warm helper on panel boot** — saves ~30 s on first job of a session.+4. **Resume cancelled jobs from latent checkpoint** — recovers ~10 min per cancellation in iterative work. Higher engineering cost.+5. **Character anchoring via I2V keyframe** — quality unlock, not speed (but enables SDK).+6. **Two parallel helpers on 64 GB** — 2× throughput on batch renders. Refactor risk.++### Marketing / launch (HAI-157, HAI-158)++- Tweet thread + slides drafted in scrollback (5-6 tweets, copy-paste ready).+- Personal-account post drafted for `@AIBizarrothe`.+- Launch copy bundle in `launch/` folder (Pinokio article, X, Reddit, CivitAI).+- Sample mp4s + frames cached in `/tmp/phos_frames/`, `/tmp/phos_frames2/`, `/tmp/phos_dialogue/`, `/tmp/phos_lab_frames/`, `/tmp/phos_sdk_frames/`.+- Awaiting Salo's launch timing call.++## 9. Hard constraints (don't violate)++- **Apple Silicon (M1+) only**. No PyTorch, no CUDA, no MPS shim. Native MLX or it doesn't ship.+- **Joint audio + video must remain**. That's the differentiator vs Wan / Hunyuan / Mochi. We don't drop audio for length.+- **Existing queue + helper + patch architecture stays intact**. No new microservices.+- **Branch policy**: dev → main only via fast-forward, only with Salo's explicit OK.+- **Salo's voice in writing**: copy-edit, don't rewrite. See memory file `feedback_copy_edit_dont_rewrite.md`. Tweets, posts, README copy — fix typos and grammar, never restructure or stack value-prop language.++## 10. Memory pointers (for next-Claude)++Files in `/Users/salo/.claude/projects/-Users-salo-AI/memory/`:++- `phosphene_dev_workflow.md` — branch + dev-panel discipline+- `phosphene_linear_project.md` — Linear project location + credentials+- `feedback_copy_edit_dont_rewrite.md` — when Salo asks me to look at his writing+- `feedback_writing_style.md` — schematic, short, plain+- `feedback_github_source_of_truth.md` — git fetch first, surface drift+- `feedback_dont_ask_to_save_memory.md` — when Salo states a fact, save it; don't ask+- `claudio_repo.md` — shared infra, hosts `.env` with Linear key+- `ltx_video_setup.md` — older notes from MLX vs Comfy decisions++## 11. Linear board++`https://linear.app/hairstylemojo/project/phosphene-9c11240704bb` — Phosphene project under HAI team (free plan caps at 2 teams).++Issue prefixes are `HAI-NN` because of the team constraint. Active:++- HAI-150 History (Done — reference doc)+- HAI-151 Current state (Done — reference doc)+- HAI-152 Lab batch 1 (In Progress — folded into this STATE.md going forward)+- HAI-153 Lab batch 2 (Backlog — depends on what comes next)+- HAI-154 Long-video research Strategy A/B/C (Backlog)+- HAI-155 Director Mode agent workflow (Backlog — depends on multi-keyframe shipping)+- HAI-156 KTDS install case (In Progress — pending log tail)+- HAI-157 Tweet thread + writeup launch (Backlog — drafts ready)+- HAI-158 Marketing scenes (In Progress)++## 12. How to start a fresh session++1. `cd /Users/salo/pinokio/api/phosphene-dev.git/`+2. `git fetch origin && git status -sb` — surface any drift first+3. Read this file (`docs/STATE.md`) and `CLAUDE.md`+4. Skim Linear `HAI-150` through `HAI-158` for state of each workstream+5. Check the dev panel is alive: `curl -s http://127.0.0.1:8199/status | python3 -m json.tool | head -10`+6. Last 5 commits on dev: `git log --oneline -5 dev`++If you find on-disk state contradicts this doc (paths moved, commits diverged), surface that to Salo before working around it. Updating this doc at session-end is part of the loop.