Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

docs(agent): AGENTIC_FLOWS.md reference + STATE.md update + safer mlx-lm flags

mrbizarro73b8557bc6ae
diff --git a/agent/local_server.py b/agent/local_server.py
index 0156250..0376c04 100644
--- a/agent/local_server.py
+++ b/agent/local_server.py
@@ -78,14 +78,31 @@ def start(model_path: str, *, venv_python: str, log_sink=None) -> dict:
_LAST_ERROR = f"venv python not found: {venv_python}"
return _status_locked()
- # Conservative concurrency. mlx-lm 0.31.1 has a known KV-cache merge
- # bug in `_merge_caches` (BatchRotatingKVCache.merge) that fires
- # under multi-prompt batching, manifesting as
- # `ValueError: [broadcast_shapes] Shapes (1,8,N+1,256) and
- # (1,8,N,256) cannot be broadcast.` mid-generation, hanging the
- # request until the client times out. Setting prompt/decode
- # concurrency to 1 takes that code path out of play. The agent
- # is a single-user feature anyway — there's nothing to batch.
+ # Defensive flags for mlx-lm 0.31.1.
+ #
+ # Bug reproduced 2026-05-06: the multi-message agent loop hits
+ # `_merge_caches` → `BatchRotatingKVCache.merge` (mlx_lm/models/
+ # cache.py:1364) and raises
+ # ValueError: [broadcast_shapes] Shapes (1,8,N,256) and
+ # (1,8,1024,256) cannot be broadcast.
+ # mid-generation, hanging the request until the client times out
+ # (300 s default). The shapes match Gemma's sliding-window
+ # rotating cache (window=1024); merging caches that crossed the
+ # window with caches that didn't gives mismatched temporal axes.
+ #
+ # Repro details: every second multi-turn request, with a long
+ # system prompt (~3.5 k tokens) and a tool-result-bearing user
+ # turn, triggers it reliably. `--prompt-concurrency 1` alone is
+ # insufficient — the merge fires across cache *snapshots* even
+ # for a single in-flight request.
+ #
+ # The combination below sidesteps both code paths:
+ # - --prompt-cache-size 0 keeps zero distinct caches between
+ # requests, so nothing to merge across the request boundary.
+ # - --prefill-step-size 8192 prefills the entire prompt as
+ # one chunk for any prompt under 8 k tokens (our system +
+ # tool-result history fits comfortably), so chunked-prefill
+ # caches never need merging within a request either.
cmd = [
venv_python, "-m", "mlx_lm.server",
"--model", model_path,
@@ -94,6 +111,8 @@ def start(model_path: str, *, venv_python: str, log_sink=None) -> dict:
"--log-level", "WARNING",
"--prompt-concurrency", "1",
"--decode-concurrency", "1",
+ "--prompt-cache-size", "0",
+ "--prefill-step-size", "8192",
]
env = os.environ.copy()
# mlx-lm honors HF_HOME for model cache lookup. The bundled
diff --git a/docs/AGENTIC_FLOWS.md b/docs/AGENTIC_FLOWS.md
new file mode 100644
index 0000000..07377f4
--- /dev/null
+++ b/docs/AGENTIC_FLOWS.md
@@ -0,0 +1,206 @@
+# Agentic Flows — chat-driven shot planner
+
+Status: **First ship 2026-05-06**, alongside multi-keyframe Layer 2.
+
+The Agentic Flows tab in Phosphene's panel is a chat where you paste a
+script (or an idea) and the agent breaks it into shots, estimates the
+overnight wall time, and queues every shot through the same FIFO queue
+the manual UI uses. You go to sleep; the queue runs; in the morning
+you wake to a folder of mp4s and a `manifest.json` listing the cut
+order. You stitch the film in your editor.
+
+## Where to find it
+
+Panel header → form-pane → top tab strip: **Manual | Agentic Flows**.
+Manual is the existing form. Agentic Flows is the new tab. Selection
+persists in localStorage.
+
+## How a session goes
+
+1. **You paste** a brief, an idea, or a full script. The composer is
+ one big textarea — Cmd/Ctrl+Enter sends.
+2. **The agent plans in plain text first.** A numbered shot list with
+ one-line visual descriptions, durations, and a total wall-time
+ estimate. Nothing is queued yet.
+3. **You approve.** "go", "queue it", "yes" — anything that reads as
+ confirmation. Tweak the plan in the chat first if you want.
+4. **The agent submits each shot** by calling `submit_shot` on the
+ panel's existing `/queue/add` endpoint. Each submission produces a
+ tool-result chip in the chat with the job id and ETA.
+5. **The agent writes a `manifest.json`** listing the queued shots in
+ cut order. Lands in `mlx_outputs/agentflow_<session_id>/`.
+6. **The agent calls `finish`** — the loop ends and you're returned
+ control. The manual queue keeps rendering while you sleep.
+
+The agent intentionally does **not** auto-stitch the clips. It writes
+the manifest; the cut belongs to you.
+
+## Engine options
+
+The agent talks to any OpenAI-compatible chat endpoint. Three setups:
+
+### 1. Phosphene Local (default)
+
+Spawns `mlx_lm.server` against the bundled `gemma-3-12b-it-4bit`
+weights. These ship with Phosphene as the LTX text encoder — the same
+model file is also a capable instruction-tuned chat model. **Zero
+extra download** to start using the agent on a fresh install.
+
+Click *Settings → Start* to spawn the server (~10 s cold start, ~30 s
+to first token after weights load).
+
+### 2. Local with a stronger model
+
+Drop a chat-capable MLX 4-bit model directory into `mlx_models/` —
+`config.json` + a `tokenizer.json` + at least one `.safetensors`.
+Examples that work well:
+
+- `mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit` — recommended
+ upgrade for serious agent work. ~17 GB, ~3.3 B active params via
+ MoE, fast inference, strong tool-use.
+- `mlx-community/Devstral-Small-2505-4bit` — Mistral's purpose-built
+ agent model. ~14 GB.
+- `mlx-community/Qwen2.5-7B-Instruct-4bit` — small fast fallback.
+
+The Settings drawer's *Local model* dropdown auto-discovers any dir
+under `mlx_models/` that looks chat-capable.
+
+### 3. Custom (cloud or LAN)
+
+Pick *Custom* in the engine kind picker, then fill:
+- *Base URL*: e.g. `https://api.openai.com/v1`,
+ `https://api.anthropic.com/v1` (their OpenAI-compat shim), or your
+ LM Studio box at `http://192.168.x.x:1234/v1`.
+- *Model*: the model id the endpoint expects (`claude-sonnet-4-6`,
+ `gpt-5`, etc.).
+- *API key*: stored in `state/agent_config.json` and only sent as the
+ `Authorization` header. `GET /agent/config` masks it (`has_api_key`
+ bool only).
+
+## What the agent knows
+
+The system prompt the agent sees on every turn encodes Phosphene's
+operator manual: modes, empirical wall times from `CLAUDE.md` §0,
+LTX 2.3 failure modes from `STATE.md` §6, prompting rules, and tool
+docstrings. Highlights it acts on:
+
+- LTX is good at: talking heads, medium shots, interviews, slow
+ push-ins, mockumentary / news / clinical settings, 2–3 dialogue
+ turns per clip, joint audio + video.
+- LTX is bad at: hands and held objects, high-motion physics, faces
+ below ~80 px in-frame, multi-shot character drift across cuts.
+- Hardware tier clamps (Comfortable: 1280 px t2v, 768 px FFLF/Extend).
+- Quality / accel / mode → wall-time table for honest ETAs.
+- Cross-shot continuity workflow: render shot 1 → `extract_frame`
+ the last frame → use as `ref_image_path` for shot 2's i2v anchor.
+
+Update `docs/STATE.md` or `CLAUDE.md` and the system prompt picks the
+new knowledge up on the next turn — single source of truth.
+
+## Tool reference
+
+Tools the agent can call. Docstrings are pulled into the system prompt
+verbatim (see `agent/runtime.render_tools_doc()`); kept here for human
+reference too.
+
+| Tool | What it does |
+|---|---|
+| `estimate_shot` | Wall-time estimate for a planned shot from the empirical M4 Max table. T^1.5 length scaling. No side effects. |
+| `submit_shot` | Append a render job to the panel's FIFO queue. Wraps `make_job` + `STATE['queue'].append()`. Returns job id, queue position, ETA. |
+| `get_queue_status` | Snapshot: running, queue, recent history, total wall ETA. |
+| `wait_for_shot` | Block until a job hits done/error/cancelled. Used when the agent needs a finished output to chain into the next shot. |
+| `extract_frame` | Pull first/last/middle frame of a finished clip as PNG. Used for cross-shot anchoring (last frame of N → keyframe-0 of N+1). |
+| `upload_image` | Resolve a chat-attached image's absolute path before passing it as `ref_image_path` or in `keyframes`. |
+| `write_session_manifest` | JSON file listing all submitted shots in cut order. Lands in `mlx_outputs/agentflow_<session_id>/`. Doesn't auto-stitch. |
+| `finish` | Explicit loop terminator. The runtime stops calling the model after this. |
+
+The fenced-block protocol: the model emits a single
+` ```action {...} ``` ` block per turn. The runtime parses, dispatches,
+and feeds the result back as a `<tool_result>...</tool_result>` user
+message. Universal across every Chat Completions server we've tested.
+
+## Wire flow (debugging)
+
+```
+Browser (Agentic Flows tab)
+ │
+ │ POST /agent/sessions/<id>/message
+ │ { "message": "<user prose>" }
+ ▼
+mlx_ltx_panel.py (HTTP handler)
+ │
+ │ agent_runtime.run_turn(session, msg, panel_ops, …)
+ ▼
+agent/runtime.py
+ ├── builds system prompt (prompts.py + capabilities snapshot)
+ ├── engine.chat(messages, EngineConfig) ← OpenAI-compat HTTP
+ │ │
+ │ ▼
+ │ mlx_lm.server (local, port 8200) OR cloud endpoint
+ │
+ ├── tools.parse_action_block(reply)
+ ├── tools.dispatch(name, args, panel_ops, session_state)
+ │ └── panel_ops.submit_job(form)
+ │ └── make_job(form) + STATE['queue'].append(...)
+ │
+ └── loop until no action block OR finish OR max_steps
+```
+
+## Known limitations + workarounds
+
+- **Bundled Gemma 3 12B IT is good at planning a shot list** but a
+ larger model (Qwen3 Coder 30B-A3B) handles long scripts and tool
+ chaining noticeably better. If your script has more than ~10 shots
+ and needs to chain anchored frames, install a stronger model in
+ `mlx_models/` and pick it via Settings.
+
+- **mlx-lm 0.31.1 has a KV-cache merge bug** (`BatchRotatingKVCache.
+ merge` raises on Gemma-style sliding-window caches). Phosphene
+ spawns `mlx_lm.server` with `--prompt-cache-size 0
+ --prefill-step-size 8192 --prompt-concurrency 1
+ --decode-concurrency 1` to sidestep both code paths. Documented in
+ `agent/local_server.py:start()`.
+
+- **Cross-shot character drift** is LTX's structural limit; the agent
+ can mitigate via i2v / multi-keyframe anchoring but cannot eliminate.
+ If your script demands exact continuity (the same actor across many
+ shots), expect some drift and plan for tighter coverage.
+
+- **No streaming for now.** The chat sends one turn, blocks until the
+ whole tool loop completes, then re-renders. A multi-shot plan can
+ take 2–5 minutes of model time on the bundled Gemma; the *Thinking…*
+ indicator is the only feedback during that window. Streaming via
+ Server-Sent Events is a v2 enhancement.
+
+## File layout
+
+```
+agent/
+├── __init__.py
+├── engine.py ← OpenAI-compat client + role-alternation normalizer
+├── local_server.py ← mlx_lm.server lifecycle + model discovery
+├── prompts.py ← system prompt builder (sources from STATE.md / CLAUDE.md)
+├── tools.py ← tool definitions + dispatcher + PanelOps protocol
+└── runtime.py ← Session, run_turn loop, persistence
+
+state/
+├── agent_config.json ← persisted EngineConfig (api_key never returned via HTTP)
+└── agent_sessions/
+ └── <session_id>.json ← one chat thread, atomic-replaced after every turn
+
+mlx_outputs/
+└── agentflow_<session_id>/
+ └── manifest.json ← cut-order index for an overnight render
+```
+
+## Hard constraints honored
+
+- **Apple Silicon only** — no PyTorch, no CUDA. mlx-lm runs the agent's
+ LLM the same way the helper runs LTX.
+- **No new microservices** — the agent runs inside the panel process.
+ mlx_lm.server is a subprocess, exactly like `mlx_warm_helper.py`.
+- **Joint audio + video stays** — the agent submits standard t2v / i2v
+ / keyframe / extend jobs through the same path as the manual UI; no
+ bypass of the audio-bearing pipeline.
+- **Branch policy** — every Agentic Flows commit lands on `dev` first.
+ Promotion to `main` waits for explicit OK.
diff --git a/docs/STATE.md b/docs/STATE.md
index d66e353..a23b05d 100644
--- a/docs/STATE.md
+++ b/docs/STATE.md
@@ -1,6 +1,7 @@
# Phosphene — project state, history, open work
-Current version: **v2.0.4** on `dev` and `main` (commit `74c7bd1`, May 5 2026).
+Current version: **v2.0.4** on `main` (commit `74c7bd1`, May 5 2026).
+Latest on `dev`: **Agentic Flows + multi-keyframe Layer 2** (May 6 2026, awaiting promotion).
Live URL: `https://github.com/mrbizarro/phosphene` · Linear project: `https://linear.app/hairstylemojo/project/phosphene-9c11240704bb`
This doc is the **session-start handoff**. A new Claude window entering this project should read this first, then `CLAUDE.md` (architecture), then the relevant Linear issues.
@@ -205,11 +206,11 @@ Everything below is also tracked in Linear (HAI-150 → HAI-158 under the Phosph
**TL;DR**: ComfyGuy9000 demoed first-frame-last-frame method via `Deno2026/comfyui-deno-custom-nodes`. Phosphene's `ltx_pipelines_mlx.KeyframeInterpolationPipeline` already accepts arbitrary `list[Image]` keyframes + `list[int]` indices — but our panel/helper artificially restrict it to 2 keyframes (start + end). Exposing the full multi-keyframe API gives us the agentic-flow compositional primitive: agent picks N stills, model fills the motion, character is anchored at every shot start.
**Status (2026-05-06)**:
-- **Layer 1 — DONE.** Helper `generate_keyframe` action accepts arbitrary `keyframe_images` + `keyframe_indices` lists, with strict validation. Backward-compatible with the old `start_image`/`end_image` shape so the panel keeps working. Full agent contract in `docs/SDK_KEYFRAME_INTERPOLATION.md` § Agent API contract.
-- **Layer 2 (panel HTTP form-parsing) — NOT YET.** Needed if agents want to POST through the panel queue instead of talking to the helper directly via stdin.
-- **Layer 3 (panel UI multi-row keyframe list) — NOT YET.**
+- **Layer 1 — DONE.** Helper `generate_keyframe` action accepts arbitrary `keyframe_images` + `keyframe_indices` lists, with strict validation. Backward-compatible with the old `start_image`/`end_image` shape so the panel keeps working.
+- **Layer 2 — DONE (commit 1afa1be).** `mlx_ltx_panel.py:make_job` reads a `keyframes_json` form field (JSON-encoded list of `{image_path, frame_index}` plus a `keyframes_total_frames` companion). The keyframe branch in `run_job_inner` decodes, validates strictly-increasing indices within `[0, frames-1]`, and forwards `keyframe_images` + `keyframe_indices` arrays to the helper. Backward compat preserved: empty `keyframes_json` falls back to `start_image`/`end_image`.
+- **Layer 3 (panel UI multi-row keyframe list) — NOT YET.** The manual UI still has 2 drop-zones. Agents already use the full primitive via `submit_shot(keyframes=[{image_path, frame_index}, ...])`.
-**Today's agent path**: spawn a helper subprocess and write JSON jobs to its stdin. See the doc for a working Python example.
+**Today's agent path**: through the panel — `agent.tools.submit_shot` composes the form including `keyframes_json` and POSTs to `/queue/add`. The legacy stdin-direct path still works for non-panel callers.
### Long-video research (Strategy A / B / C)
@@ -223,16 +224,13 @@ Codex deep-research brief drafted; awaiting return for literature review on Free
Salo also has Claude.ai / ChatGPT deep-research running (May 5) on inference speed without quality loss.
-### Director Mode (agent workflow)
+### Director Mode (agent workflow) — SHIPPED as Agentic Flows
-Future feature for 1-minute movies from one user idea:
+What ships: a chat-driven shot planner tab in the panel. User pastes a script or idea, agent breaks it into shots, queues every shot through the existing FIFO queue, writes a `manifest.json`, and finishes. Designed for overnight batch rendering. Auto-stitch is intentionally NOT included — manifest is the deliverable; cuts belong to the user.
-1. Planner (Gemma 3 12B already on disk, or external LLM) expands one-line concept to shot-list with style guide
-2. Submitter POSTs each shot to `/queue/add`
-3. Stitcher concatenates the K resulting mp4s with ffmpeg
-4. Director UI in the panel — paste idea, see planned shot-list, edit any prompt, hit "Render Movie"
+See preceding "Agentic Flows" section + `docs/AGENTIC_FLOWS.md` for the full reference.
-Compositional primitive **partially unblocked (2026-05-06)**: Layer 1 of multi-keyframe is live, so an agent can already submit jobs via helper stdin. Layer 2 (panel HTTP route) is the remaining piece for queue-based submission. Long-video research (per-shot length sweet spot) still pending Codex deep-research return.
+Long-video research (per-shot length sweet spot, FreeNoise / FIFO-Diffusion / StreamingT2V applicability) still pending Codex deep-research return.
### Speed optimization candidates (from May 4 research session)
@@ -304,7 +302,7 @@ Issue prefixes are `HAI-NN` because of the team constraint. Active:
- HAI-152 Lab batch 1 (In Progress — folded into this STATE.md going forward)
- HAI-153 Lab batch 2 (Backlog — depends on what comes next)
- HAI-154 Long-video research Strategy A/B/C (Backlog)
-- HAI-155 Director Mode agent workflow (Backlog — depends on multi-keyframe shipping)
+- HAI-155 Director Mode agent workflow → SHIPPED as Agentic Flows (2026-05-06, dev branch)
- HAI-156 KTDS install case (In Progress — pending log tail)
- HAI-157 Tweet thread + writeup launch (Backlog — drafts ready)
- HAI-158 Marketing scenes (In Progress)