Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
docs(agent): AGENTIC_FLOWS.md reference + STATE.md update + safer mlx-lm flags
mrbizarro73b8557bc6ae
diff --git a/agent/local_server.py b/agent/local_server.pyindex 0156250..0376c04 100644--- a/agent/local_server.py+++ b/agent/local_server.py@@ -78,14 +78,31 @@ def start(model_path: str, *, venv_python: str, log_sink=None) -> dict:_LAST_ERROR = f"venv python not found: {venv_python}"return _status_locked()- # Conservative concurrency. mlx-lm 0.31.1 has a known KV-cache merge- # bug in `_merge_caches` (BatchRotatingKVCache.merge) that fires- # under multi-prompt batching, manifesting as- # `ValueError: [broadcast_shapes] Shapes (1,8,N+1,256) and- # (1,8,N,256) cannot be broadcast.` mid-generation, hanging the- # request until the client times out. Setting prompt/decode- # concurrency to 1 takes that code path out of play. The agent- # is a single-user feature anyway — there's nothing to batch.+ # Defensive flags for mlx-lm 0.31.1.+ #+ # Bug reproduced 2026-05-06: the multi-message agent loop hits+ # `_merge_caches` → `BatchRotatingKVCache.merge` (mlx_lm/models/+ # cache.py:1364) and raises+ # ValueError: [broadcast_shapes] Shapes (1,8,N,256) and+ # (1,8,1024,256) cannot be broadcast.+ # mid-generation, hanging the request until the client times out+ # (300 s default). The shapes match Gemma's sliding-window+ # rotating cache (window=1024); merging caches that crossed the+ # window with caches that didn't gives mismatched temporal axes.+ #+ # Repro details: every second multi-turn request, with a long+ # system prompt (~3.5 k tokens) and a tool-result-bearing user+ # turn, triggers it reliably. `--prompt-concurrency 1` alone is+ # insufficient — the merge fires across cache *snapshots* even+ # for a single in-flight request.+ #+ # The combination below sidesteps both code paths:+ # - --prompt-cache-size 0 keeps zero distinct caches between+ # requests, so nothing to merge across the request boundary.+ # - --prefill-step-size 8192 prefills the entire prompt as+ # one chunk for any prompt under 8 k tokens (our system ++ # tool-result history fits comfortably), so chunked-prefill+ # caches never need merging within a request either.cmd = [venv_python, "-m", "mlx_lm.server","--model", model_path,@@ -94,6 +111,8 @@ def start(model_path: str, *, venv_python: str, log_sink=None) -> dict:"--log-level", "WARNING","--prompt-concurrency", "1","--decode-concurrency", "1",+ "--prompt-cache-size", "0",+ "--prefill-step-size", "8192",]env = os.environ.copy()# mlx-lm honors HF_HOME for model cache lookup. The bundleddiff --git a/docs/AGENTIC_FLOWS.md b/docs/AGENTIC_FLOWS.mdnew file mode 100644index 0000000..07377f4--- /dev/null+++ b/docs/AGENTIC_FLOWS.md@@ -0,0 +1,206 @@+# Agentic Flows — chat-driven shot planner++Status: **First ship 2026-05-06**, alongside multi-keyframe Layer 2.++The Agentic Flows tab in Phosphene's panel is a chat where you paste a+script (or an idea) and the agent breaks it into shots, estimates the+overnight wall time, and queues every shot through the same FIFO queue+the manual UI uses. You go to sleep; the queue runs; in the morning+you wake to a folder of mp4s and a `manifest.json` listing the cut+order. You stitch the film in your editor.++## Where to find it++Panel header → form-pane → top tab strip: **Manual | Agentic Flows**.+Manual is the existing form. Agentic Flows is the new tab. Selection+persists in localStorage.++## How a session goes++1. **You paste** a brief, an idea, or a full script. The composer is+ one big textarea — Cmd/Ctrl+Enter sends.+2. **The agent plans in plain text first.** A numbered shot list with+ one-line visual descriptions, durations, and a total wall-time+ estimate. Nothing is queued yet.+3. **You approve.** "go", "queue it", "yes" — anything that reads as+ confirmation. Tweak the plan in the chat first if you want.+4. **The agent submits each shot** by calling `submit_shot` on the+ panel's existing `/queue/add` endpoint. Each submission produces a+ tool-result chip in the chat with the job id and ETA.+5. **The agent writes a `manifest.json`** listing the queued shots in+ cut order. Lands in `mlx_outputs/agentflow_<session_id>/`.+6. **The agent calls `finish`** — the loop ends and you're returned+ control. The manual queue keeps rendering while you sleep.++The agent intentionally does **not** auto-stitch the clips. It writes+the manifest; the cut belongs to you.++## Engine options++The agent talks to any OpenAI-compatible chat endpoint. Three setups:++### 1. Phosphene Local (default)++Spawns `mlx_lm.server` against the bundled `gemma-3-12b-it-4bit`+weights. These ship with Phosphene as the LTX text encoder — the same+model file is also a capable instruction-tuned chat model. **Zero+extra download** to start using the agent on a fresh install.++Click *Settings → Start* to spawn the server (~10 s cold start, ~30 s+to first token after weights load).++### 2. Local with a stronger model++Drop a chat-capable MLX 4-bit model directory into `mlx_models/` —+`config.json` + a `tokenizer.json` + at least one `.safetensors`.+Examples that work well:++- `mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit` — recommended+ upgrade for serious agent work. ~17 GB, ~3.3 B active params via+ MoE, fast inference, strong tool-use.+- `mlx-community/Devstral-Small-2505-4bit` — Mistral's purpose-built+ agent model. ~14 GB.+- `mlx-community/Qwen2.5-7B-Instruct-4bit` — small fast fallback.++The Settings drawer's *Local model* dropdown auto-discovers any dir+under `mlx_models/` that looks chat-capable.++### 3. Custom (cloud or LAN)++Pick *Custom* in the engine kind picker, then fill:+- *Base URL*: e.g. `https://api.openai.com/v1`,+ `https://api.anthropic.com/v1` (their OpenAI-compat shim), or your+ LM Studio box at `http://192.168.x.x:1234/v1`.+- *Model*: the model id the endpoint expects (`claude-sonnet-4-6`,+ `gpt-5`, etc.).+- *API key*: stored in `state/agent_config.json` and only sent as the+ `Authorization` header. `GET /agent/config` masks it (`has_api_key`+ bool only).++## What the agent knows++The system prompt the agent sees on every turn encodes Phosphene's+operator manual: modes, empirical wall times from `CLAUDE.md` §0,+LTX 2.3 failure modes from `STATE.md` §6, prompting rules, and tool+docstrings. Highlights it acts on:++- LTX is good at: talking heads, medium shots, interviews, slow+ push-ins, mockumentary / news / clinical settings, 2–3 dialogue+ turns per clip, joint audio + video.+- LTX is bad at: hands and held objects, high-motion physics, faces+ below ~80 px in-frame, multi-shot character drift across cuts.+- Hardware tier clamps (Comfortable: 1280 px t2v, 768 px FFLF/Extend).+- Quality / accel / mode → wall-time table for honest ETAs.+- Cross-shot continuity workflow: render shot 1 → `extract_frame`+ the last frame → use as `ref_image_path` for shot 2's i2v anchor.++Update `docs/STATE.md` or `CLAUDE.md` and the system prompt picks the+new knowledge up on the next turn — single source of truth.++## Tool reference++Tools the agent can call. Docstrings are pulled into the system prompt+verbatim (see `agent/runtime.render_tools_doc()`); kept here for human+reference too.++| Tool | What it does |+|---|---|+| `estimate_shot` | Wall-time estimate for a planned shot from the empirical M4 Max table. T^1.5 length scaling. No side effects. |+| `submit_shot` | Append a render job to the panel's FIFO queue. Wraps `make_job` + `STATE['queue'].append()`. Returns job id, queue position, ETA. |+| `get_queue_status` | Snapshot: running, queue, recent history, total wall ETA. |+| `wait_for_shot` | Block until a job hits done/error/cancelled. Used when the agent needs a finished output to chain into the next shot. |+| `extract_frame` | Pull first/last/middle frame of a finished clip as PNG. Used for cross-shot anchoring (last frame of N → keyframe-0 of N+1). |+| `upload_image` | Resolve a chat-attached image's absolute path before passing it as `ref_image_path` or in `keyframes`. |+| `write_session_manifest` | JSON file listing all submitted shots in cut order. Lands in `mlx_outputs/agentflow_<session_id>/`. Doesn't auto-stitch. |+| `finish` | Explicit loop terminator. The runtime stops calling the model after this. |++The fenced-block protocol: the model emits a single+` ```action {...} ``` ` block per turn. The runtime parses, dispatches,+and feeds the result back as a `<tool_result>...</tool_result>` user+message. Universal across every Chat Completions server we've tested.++## Wire flow (debugging)++```+Browser (Agentic Flows tab)+ │+ │ POST /agent/sessions/<id>/message+ │ { "message": "<user prose>" }+ ▼+mlx_ltx_panel.py (HTTP handler)+ │+ │ agent_runtime.run_turn(session, msg, panel_ops, …)+ ▼+agent/runtime.py+ ├── builds system prompt (prompts.py + capabilities snapshot)+ ├── engine.chat(messages, EngineConfig) ← OpenAI-compat HTTP+ │ │+ │ ▼+ │ mlx_lm.server (local, port 8200) OR cloud endpoint+ │+ ├── tools.parse_action_block(reply)+ ├── tools.dispatch(name, args, panel_ops, session_state)+ │ └── panel_ops.submit_job(form)+ │ └── make_job(form) + STATE['queue'].append(...)+ │+ └── loop until no action block OR finish OR max_steps+```++## Known limitations + workarounds++- **Bundled Gemma 3 12B IT is good at planning a shot list** but a+ larger model (Qwen3 Coder 30B-A3B) handles long scripts and tool+ chaining noticeably better. If your script has more than ~10 shots+ and needs to chain anchored frames, install a stronger model in+ `mlx_models/` and pick it via Settings.++- **mlx-lm 0.31.1 has a KV-cache merge bug** (`BatchRotatingKVCache.+ merge` raises on Gemma-style sliding-window caches). Phosphene+ spawns `mlx_lm.server` with `--prompt-cache-size 0+ --prefill-step-size 8192 --prompt-concurrency 1+ --decode-concurrency 1` to sidestep both code paths. Documented in+ `agent/local_server.py:start()`.++- **Cross-shot character drift** is LTX's structural limit; the agent+ can mitigate via i2v / multi-keyframe anchoring but cannot eliminate.+ If your script demands exact continuity (the same actor across many+ shots), expect some drift and plan for tighter coverage.++- **No streaming for now.** The chat sends one turn, blocks until the+ whole tool loop completes, then re-renders. A multi-shot plan can+ take 2–5 minutes of model time on the bundled Gemma; the *Thinking…*+ indicator is the only feedback during that window. Streaming via+ Server-Sent Events is a v2 enhancement.++## File layout++```+agent/+├── __init__.py+├── engine.py ← OpenAI-compat client + role-alternation normalizer+├── local_server.py ← mlx_lm.server lifecycle + model discovery+├── prompts.py ← system prompt builder (sources from STATE.md / CLAUDE.md)+├── tools.py ← tool definitions + dispatcher + PanelOps protocol+└── runtime.py ← Session, run_turn loop, persistence++state/+├── agent_config.json ← persisted EngineConfig (api_key never returned via HTTP)+└── agent_sessions/+ └── <session_id>.json ← one chat thread, atomic-replaced after every turn++mlx_outputs/+└── agentflow_<session_id>/+ └── manifest.json ← cut-order index for an overnight render+```++## Hard constraints honored++- **Apple Silicon only** — no PyTorch, no CUDA. mlx-lm runs the agent's+ LLM the same way the helper runs LTX.+- **No new microservices** — the agent runs inside the panel process.+ mlx_lm.server is a subprocess, exactly like `mlx_warm_helper.py`.+- **Joint audio + video stays** — the agent submits standard t2v / i2v+ / keyframe / extend jobs through the same path as the manual UI; no+ bypass of the audio-bearing pipeline.+- **Branch policy** — every Agentic Flows commit lands on `dev` first.+ Promotion to `main` waits for explicit OK.diff --git a/docs/STATE.md b/docs/STATE.mdindex d66e353..a23b05d 100644--- a/docs/STATE.md+++ b/docs/STATE.md@@ -1,6 +1,7 @@# Phosphene — project state, history, open work-Current version: **v2.0.4** on `dev` and `main` (commit `74c7bd1`, May 5 2026).+Current version: **v2.0.4** on `main` (commit `74c7bd1`, May 5 2026).+Latest on `dev`: **Agentic Flows + multi-keyframe Layer 2** (May 6 2026, awaiting promotion).Live URL: `https://github.com/mrbizarro/phosphene` · Linear project: `https://linear.app/hairstylemojo/project/phosphene-9c11240704bb`This doc is the **session-start handoff**. A new Claude window entering this project should read this first, then `CLAUDE.md` (architecture), then the relevant Linear issues.@@ -205,11 +206,11 @@ Everything below is also tracked in Linear (HAI-150 → HAI-158 under the Phosph**TL;DR**: ComfyGuy9000 demoed first-frame-last-frame method via `Deno2026/comfyui-deno-custom-nodes`. Phosphene's `ltx_pipelines_mlx.KeyframeInterpolationPipeline` already accepts arbitrary `list[Image]` keyframes + `list[int]` indices — but our panel/helper artificially restrict it to 2 keyframes (start + end). Exposing the full multi-keyframe API gives us the agentic-flow compositional primitive: agent picks N stills, model fills the motion, character is anchored at every shot start.**Status (2026-05-06)**:-- **Layer 1 — DONE.** Helper `generate_keyframe` action accepts arbitrary `keyframe_images` + `keyframe_indices` lists, with strict validation. Backward-compatible with the old `start_image`/`end_image` shape so the panel keeps working. Full agent contract in `docs/SDK_KEYFRAME_INTERPOLATION.md` § Agent API contract.-- **Layer 2 (panel HTTP form-parsing) — NOT YET.** Needed if agents want to POST through the panel queue instead of talking to the helper directly via stdin.-- **Layer 3 (panel UI multi-row keyframe list) — NOT YET.**+- **Layer 1 — DONE.** Helper `generate_keyframe` action accepts arbitrary `keyframe_images` + `keyframe_indices` lists, with strict validation. Backward-compatible with the old `start_image`/`end_image` shape so the panel keeps working.+- **Layer 2 — DONE (commit 1afa1be).** `mlx_ltx_panel.py:make_job` reads a `keyframes_json` form field (JSON-encoded list of `{image_path, frame_index}` plus a `keyframes_total_frames` companion). The keyframe branch in `run_job_inner` decodes, validates strictly-increasing indices within `[0, frames-1]`, and forwards `keyframe_images` + `keyframe_indices` arrays to the helper. Backward compat preserved: empty `keyframes_json` falls back to `start_image`/`end_image`.+- **Layer 3 (panel UI multi-row keyframe list) — NOT YET.** The manual UI still has 2 drop-zones. Agents already use the full primitive via `submit_shot(keyframes=[{image_path, frame_index}, ...])`.-**Today's agent path**: spawn a helper subprocess and write JSON jobs to its stdin. See the doc for a working Python example.+**Today's agent path**: through the panel — `agent.tools.submit_shot` composes the form including `keyframes_json` and POSTs to `/queue/add`. The legacy stdin-direct path still works for non-panel callers.### Long-video research (Strategy A / B / C)@@ -223,16 +224,13 @@ Codex deep-research brief drafted; awaiting return for literature review on FreeSalo also has Claude.ai / ChatGPT deep-research running (May 5) on inference speed without quality loss.-### Director Mode (agent workflow)+### Director Mode (agent workflow) — SHIPPED as Agentic Flows-Future feature for 1-minute movies from one user idea:+What ships: a chat-driven shot planner tab in the panel. User pastes a script or idea, agent breaks it into shots, queues every shot through the existing FIFO queue, writes a `manifest.json`, and finishes. Designed for overnight batch rendering. Auto-stitch is intentionally NOT included — manifest is the deliverable; cuts belong to the user.-1. Planner (Gemma 3 12B already on disk, or external LLM) expands one-line concept to shot-list with style guide-2. Submitter POSTs each shot to `/queue/add`-3. Stitcher concatenates the K resulting mp4s with ffmpeg-4. Director UI in the panel — paste idea, see planned shot-list, edit any prompt, hit "Render Movie"+See preceding "Agentic Flows" section + `docs/AGENTIC_FLOWS.md` for the full reference.-Compositional primitive **partially unblocked (2026-05-06)**: Layer 1 of multi-keyframe is live, so an agent can already submit jobs via helper stdin. Layer 2 (panel HTTP route) is the remaining piece for queue-based submission. Long-video research (per-shot length sweet spot) still pending Codex deep-research return.+Long-video research (per-shot length sweet spot, FreeNoise / FIFO-Diffusion / StreamingT2V applicability) still pending Codex deep-research return.### Speed optimization candidates (from May 4 research session)@@ -304,7 +302,7 @@ Issue prefixes are `HAI-NN` because of the team constraint. Active:- HAI-152 Lab batch 1 (In Progress — folded into this STATE.md going forward)- HAI-153 Lab batch 2 (Backlog — depends on what comes next)- HAI-154 Long-video research Strategy A/B/C (Backlog)-- HAI-155 Director Mode agent workflow (Backlog — depends on multi-keyframe shipping)+- HAI-155 Director Mode agent workflow → SHIPPED as Agentic Flows (2026-05-06, dev branch)- HAI-156 KTDS install case (In Progress — pending log tail)- HAI-157 Tweet thread + writeup launch (Backlog — drafts ready)- HAI-158 Marketing scenes (In Progress)