Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
multi-keyframe interpolation Layer 1 — helper accepts arbitrary keyframe lists
mrbizarro0cfee8c98975
diff --git a/docs/SDK_KEYFRAME_INTERPOLATION.md b/docs/SDK_KEYFRAME_INTERPOLATION.mdindex c95bc7c..1d352d2 100644--- a/docs/SDK_KEYFRAME_INTERPOLATION.md+++ b/docs/SDK_KEYFRAME_INTERPOLATION.md@@ -1,6 +1,6 @@# Multi-keyframe interpolation — the SDK shot-composition primitive-Status: **research + design doc.** Not yet implemented. Documents a known gap in Phosphene where the underlying engine supports more than the UI exposes.+Status: **Layer 1 shipped 2026-05-06** — helper now accepts arbitrary keyframe lists. Layer 2 (panel HTTP form-parsing) and Layer 3 (UI) not yet. Engine has supported this all along; we just exposed it. See [Agent API contract](#agent-api-contract-layer-1-2026-05-06) for the call shape.## TL;DR@@ -123,37 +123,21 @@ This is the SDK's compositional primitive in one sentence: **the editor controlsThree layers, in order:-### Layer 1 — helper (smallest change, biggest unlock)+### Layer 1 — DONE (2026-05-06)-`mlx_warm_helper.py` action `generate_keyframe`. Currently:+Helper `generate_keyframe` action now accepts arbitrary `keyframe_images` + `keyframe_indices` lists with strict input validation. The hardcoded 2-keyframe path remains as a backward-compat fallback so the panel keeps working unchanged. See `mlx_warm_helper.py` lines ~1251-1360.-```python-keyframe_images=[p["start_image"], p["end_image"]],-keyframe_indices=[0, num_frames - 1],-```--Change to read `keyframe_images` and `keyframe_indices` directly from the job spec (lists). Default to the existing 2-keyframe behavior if the spec only provides `start_image` + `end_image` (backward compat). New job spec accepts:--```json-{- "action": "generate_keyframe",- "params": {- "keyframe_images": ["<path1>", "<path2>", "<path3>"],- "keyframe_indices": [0, 60, 120],- ...- }-}-```--Three lines of code, no model change, no upstream change.+Full agent contract below.-### Layer 2 — panel (form parsing + worker dispatch)+### Layer 2 — panel HTTP form parsing — NOT YET`mlx_ltx_panel.py` `mode == "keyframe"` branch. Currently parses `start_image` + `end_image` form fields and constructs the helper job spec. Change to parse a **list of keyframes** from the form (e.g. `keyframe_image_0`, `keyframe_image_1`, ... or a JSON-encoded list field). Build the lists for the helper.Backward compat: if the form has `start_image` + `end_image` only, behave as today.-### Layer 3 — UI (the user-visible surface)+This is the layer the agent will use if it goes through the panel's HTTP queue endpoint instead of talking to the helper directly.++### Layer 3 — UI (the user-visible surface) — NOT YETThe current Keyframe section in the panel HTML has two drop-zones. Replace with a list:@@ -175,6 +159,156 @@ Once multi-keyframe ships, the Director Mode (HAI-155) gains the compositional pThe agent never has to "describe a scene in words and pray." Composition is concrete, predictable, controllable.+## Agent API contract (Layer 1 — 2026-05-06)++The helper subprocess (`mlx_warm_helper.py`) is a long-lived JSON-line server. Each line on stdin is one job; each line on stdout is one event. To use multi-keyframe interpolation, write a single JSON line to the helper's stdin with `action: "generate_keyframe"`.++### Request shape++```json+{+ "id": "<your-unique-job-id-string>",+ "action": "generate_keyframe",+ "params": {+ "prompt": "A young woman walks across a sunlit kitchen, opens the fridge, takes out an apple.",+ "keyframe_images": [+ "/abs/path/to/keyframe_0.png",+ "/abs/path/to/keyframe_1.png",+ "/abs/path/to/keyframe_2.png"+ ],+ "keyframe_indices": [0, 60, 120],+ "frames": 121,+ "width": 768,+ "height": 416,+ "output_path": "/abs/path/to/output.mp4",+ "seed": -1,+ "stage1_steps": 15,+ "stage2_steps": 3,+ "cfg_scale": 3.0,+ "negative_prompt": null,+ "model_dir": null+ }+}+```++### Field rules++| Field | Type | Required | Notes |+|---|---|---|---|+| `id` | str | yes | Echoed back in `done` / `error` events so the agent can match. Use any unique string (UUID, monotonic counter, etc). |+| `action` | str | yes | Must be exactly `"generate_keyframe"`. |+| `params.prompt` | str | yes | Scene description. The model uses this for the motion between keyframes; the keyframes themselves dominate composition. |+| `params.keyframe_images` | list[str] | yes (multi-keyframe) | Absolute file paths. PNG/JPEG. Length must equal `keyframe_indices`. **N ≥ 2.** Files must exist at job-submission time. |+| `params.keyframe_indices` | list[int] | yes (multi-keyframe) | Pixel-frame indices, 0-based, **strictly ascending**, all in `[0, frames-1]`. Must include at least 2 entries. Same length as `keyframe_images`. |+| `params.start_image` + `params.end_image` | str | yes (FFLF backward-compat) | Used only if `keyframe_images`/`keyframe_indices` are absent. Equivalent to `keyframe_indices=[0, frames-1]`. |+| `params.frames` | int | yes | Total pixel frames. Latent grid is 8 frames per slot, so use 8k+1 (49, 121, 241). 24 fps is hardcoded ⇒ 121 frames = ~5.0 sec. |+| `params.width`, `params.height` | int | yes | Output dimensions. Comfortable tier (M-Max 64 GB) is clamped to **768×416** for keyframe jobs (OOM history at full 1280×704). |+| `params.output_path` | str | yes | Absolute path where the resulting `.mp4` will be written (with audio). |+| `params.seed` | int | optional | `-1` (default) = random, otherwise fixed. Echoed back in the `done` event as `seed_used` so the agent can reproduce. |+| `params.stage1_steps` | int | optional | Default `15`. Stage-1 (half-res guided denoise) sigma count. Quality knob. |+| `params.stage2_steps` | int | optional | Default `3`. Stage-2 (full-res refine) sigma count. |+| `params.cfg_scale` | float | optional | Default `3.0`. Classifier-free-guidance strength. |+| `params.negative_prompt` | str \| null | optional | If provided, used as native CFG negative. Null = use default. |+| `params.model_dir` | str \| null | optional | Override the transformer directory. Null = default Q8 dev path. Don't change this unless you know why. |++### Response events++The helper emits JSON lines on stdout:++```json+{"event": "log", "line": "Keyframe multi-3kf — indices=[0, 60, 120], pipeline cover-crops all to 768x416"}+{"event": "progress", "id": "...", "phase": "stage1", "step": 5, "total": 15, "pct": 33}+{"event": "progress", "id": "...", "phase": "stage2", "step": 1, "total": 3, "pct": 90}+{"event": "done", "id": "...", "output": "/abs/path/out.mp4", "elapsed_sec": 312.4, "seed_used": 1234567}+```++On any failure (validation error, OOM, model error):++```json+{"event": "error", "id": "...", "error": "<message>", "trace": "<python-traceback>"}+```++The helper continues running after errors; submit the next job whenever ready.++### Validation errors (fail-fast at submission, before any GPU work)++The helper rejects bad jobs immediately so agent bugs surface early:++- `keyframe_images and keyframe_indices must both be provided as lists`+- `keyframe_images and keyframe_indices must be lists`+- `keyframe_images (N) and keyframe_indices (M) must have the same length`+- `at least 2 keyframes required`+- `keyframe image not found: <path>`+- `keyframe_indices must be integers, got: <repr>`+- `keyframe_index <i> out of range [0, <frames-1>]`+- `keyframe_indices must be strictly ascending, got <list>`++### Concurrency model++The helper handles **one job at a time**. The agent's queue should serialize submissions or wait for `done` / `error` before submitting the next job. Multiple jobs interleaved on stdin will queue but the helper won't process the second until the first finishes.++If you need parallelism, spawn multiple helpers (each takes ~30 GB VRAM + ~30s warm-up; M4 Max 64 GB fits 1 reliably, 2 is tight).++### Wall-time budget++For a 121-frame (5-sec) clip at 768×416 with `stage1_steps=15`, `stage2_steps=3`:++- Cold helper (first job after start): ~30 s warm-up + ~5 min generate = ~5:30+- Warm helper (subsequent jobs): ~5 min generate++Going from 2 keyframes to 8 keyframes adds a few seconds of front-end VAE encoding, not significant against the ~5-min denoise.++### Concrete example — Director Mode three-shot scene++```python+import json, subprocess, uuid, sys++helper = subprocess.Popen(+ ["./ltx-2-mlx/env/bin/python3.11", "-u", "mlx_warm_helper.py"],+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,+ text=True, bufsize=1,+)++def submit(job):+ helper.stdin.write(json.dumps(job) + "\n")+ helper.stdin.flush()+ while True:+ line = helper.stdout.readline()+ if not line: raise RuntimeError("helper died")+ evt = json.loads(line)+ print(evt, file=sys.stderr)+ if evt.get("event") == "done" and evt.get("id") == job["id"]: return evt+ if evt.get("event") == "error" and evt.get("id") == job["id"]: raise RuntimeError(evt["error"])++# Shot 1 — establishing wide. 4 keyframes for explicit choreography.+result = submit({+ "id": str(uuid.uuid4()),+ "action": "generate_keyframe",+ "params": {+ "prompt": "A woman walks across a kitchen, dappled morning light through window blinds.",+ "keyframe_images": [+ "/tmp/agent/kitchen_01_start.png",+ "/tmp/agent/kitchen_02_mid_left.png",+ "/tmp/agent/kitchen_03_mid_right.png",+ "/tmp/agent/kitchen_04_at_fridge.png",+ ],+ "keyframe_indices": [0, 40, 80, 120],+ "frames": 121,+ "width": 768, "height": 416,+ "output_path": "/tmp/agent/shot_01.mp4",+ "seed": 42,+ }+})+# result["output"], result["elapsed_sec"], result["seed_used"]+```++### What's NOT in this layer++- **Per-keyframe strength.** All keyframes are pinned at strength 1.0 (pure anchor). The model cannot "lightly suggest" a keyframe yet — it's all-or-nothing. Strength control requires an upstream patch in `ltx-pipelines-mlx`.+- **Panel HTTP route.** Layer 2 is needed to reach this from the panel's queue (HTTP form-data). Today the only path is the helper's stdin.+- **Panel UI.** Today's panel still has 2 image drop-zones. Adding more rows is Layer 3.+- **Audio across keyframe boundaries between shots.** Each clip is its own audio render — boundaries are hard cuts. See "What this doesn't solve" below.+## Memory budgetA keyframe job today, on Comfortable tier (M-Max 64 GB), runs at clamped 768×416 (Y9 in Linear notes the OOM history at full 1280×704 — kept the clamp). N keyframes don't change peak memory significantly — each VAE-encode is one image at a time, and the sampler latent is the same shape regardless of how many slots are pinned.diff --git a/docs/STATE.md b/docs/STATE.mdindex f6c82b3..d66e353 100644--- a/docs/STATE.md+++ b/docs/STATE.md@@ -204,7 +204,12 @@ Everything below is also tracked in Linear (HAI-150 → HAI-158 under the Phosph**TL;DR**: ComfyGuy9000 demoed first-frame-last-frame method via `Deno2026/comfyui-deno-custom-nodes`. Phosphene's `ltx_pipelines_mlx.KeyframeInterpolationPipeline` already accepts arbitrary `list[Image]` keyframes + `list[int]` indices — but our panel/helper artificially restrict it to 2 keyframes (start + end). Exposing the full multi-keyframe API gives us the agentic-flow compositional primitive: agent picks N stills, model fills the motion, character is anchored at every shot start.-**Implementation order**: helper (3 lines) → panel worker (form parsing) → UI (multi-keyframe drop-zone list).+**Status (2026-05-06)**:+- **Layer 1 — DONE.** Helper `generate_keyframe` action accepts arbitrary `keyframe_images` + `keyframe_indices` lists, with strict validation. Backward-compatible with the old `start_image`/`end_image` shape so the panel keeps working. Full agent contract in `docs/SDK_KEYFRAME_INTERPOLATION.md` § Agent API contract.+- **Layer 2 (panel HTTP form-parsing) — NOT YET.** Needed if agents want to POST through the panel queue instead of talking to the helper directly via stdin.+- **Layer 3 (panel UI multi-row keyframe list) — NOT YET.**++**Today's agent path**: spawn a helper subprocess and write JSON jobs to its stdin. See the doc for a working Python example.### Long-video research (Strategy A / B / C)@@ -227,7 +232,7 @@ Future feature for 1-minute movies from one user idea:3. Stitcher concatenates the K resulting mp4s with ffmpeg4. Director UI in the panel — paste idea, see planned shot-list, edit any prompt, hit "Render Movie"-Blocked on multi-keyframe interpolation shipping (the compositional primitive) + long-video research deliverable (per-shot length sweet spot).+Compositional primitive **partially unblocked (2026-05-06)**: Layer 1 of multi-keyframe is live, so an agent can already submit jobs via helper stdin. Layer 2 (panel HTTP route) is the remaining piece for queue-based submission. Long-video research (per-shot length sweet spot) still pending Codex deep-research return.### Speed optimization candidates (from May 4 research session)diff --git a/mlx_warm_helper.py b/mlx_warm_helper.pyindex 505aacd..d0ad764 100644--- a/mlx_warm_helper.py+++ b/mlx_warm_helper.py@@ -1249,8 +1249,23 @@ for line in sys.__stdin__:continueif action == "generate_keyframe":- # FFLF — two images anchored at frame 0 and frame N-1, model interpolates.- # Like HQ this uses the Q8 dev transformer + two-stage refine.+ # Keyframe interpolation — anchor images at chosen frame indices,+ # model fills the motion between them. Uses the Q8 dev transformer+ # + two-stage refine (same as HQ).+ #+ # Two input shapes are accepted:+ #+ # (A) Multi-keyframe (preferred — used by agents):+ # "keyframe_images": list[str] absolute paths, length N >= 2+ # "keyframe_indices": list[int] pixel-frame indices, length N,+ # strictly ascending, all in [0, frames-1]+ #+ # (B) FFLF backward-compat (used by the panel today):+ # "start_image": str+ # "end_image": str+ # Equivalent to multi-keyframe with indices [0, frames-1].+ #+ # If (A) fields are present they win; (B) is only checked when (A) is absent.job_id = msg.get("id", "?")p = msg.get("params", {}) or {}model_dir = p.get("model_dir") or MODEL_ID@@ -1261,19 +1276,66 @@ for line in sys.__stdin__:try:t0 = time.time()configure_acceleration("off")- for k in ("start_image", "end_image"):- img = p.get(k)- if not img or not os.path.exists(img):- raise RuntimeError(f"{k} not found: {img}")- pipe = get_kf_pipe(model_dir)num_frames = int(p["frames"])++ # ---- Resolve keyframes (multi-keyframe path or FFLF fallback) ----+ kf_images_in = p.get("keyframe_images")+ kf_indices_in = p.get("keyframe_indices")+ if kf_images_in is not None or kf_indices_in is not None:+ # Multi-keyframe — validate strictly so agent bugs surface early.+ if kf_images_in is None or kf_indices_in is None:+ raise RuntimeError(+ "keyframe_images and keyframe_indices must both be provided as lists"+ )+ if not isinstance(kf_images_in, list) or not isinstance(kf_indices_in, list):+ raise RuntimeError("keyframe_images and keyframe_indices must be lists")+ if len(kf_images_in) != len(kf_indices_in):+ raise RuntimeError(+ f"keyframe_images ({len(kf_images_in)}) and "+ f"keyframe_indices ({len(kf_indices_in)}) must have the same length"+ )+ if len(kf_images_in) < 2:+ raise RuntimeError("at least 2 keyframes required")+ for path in kf_images_in:+ if not isinstance(path, str) or not os.path.exists(path):+ raise RuntimeError(f"keyframe image not found: {path}")+ idxs: list[int] = []+ for i in kf_indices_in:+ try:+ idx = int(i)+ except (TypeError, ValueError):+ raise RuntimeError(f"keyframe_indices must be integers, got: {i!r}")+ if idx < 0 or idx >= num_frames:+ raise RuntimeError(+ f"keyframe_index {idx} out of range [0, {num_frames - 1}]"+ )+ idxs.append(idx)+ for a, b in zip(idxs, idxs[1:]):+ if b <= a:+ raise RuntimeError(+ f"keyframe_indices must be strictly ascending, got {idxs}"+ )+ kf_images = list(kf_images_in)+ kf_indices = idxs+ kf_mode_label = f"multi-{len(kf_images)}kf"+ else:+ # FFLF backward-compat — start + end at the boundaries.+ for k in ("start_image", "end_image"):+ img = p.get(k)+ if not img or not os.path.exists(img):+ raise RuntimeError(f"{k} not found: {img}")+ kf_images = [p["start_image"], p["end_image"]]+ kf_indices = [0, num_frames - 1]+ kf_mode_label = "FFLF"++ pipe = get_kf_pipe(model_dir)# Y1.037: short-clip VAE-streaming opt-out (Keyframe path)._apply_vae_streaming_decision(num_frames)kwargs = dict(prompt=p["prompt"],output_path=p["output_path"],- keyframe_images=[p["start_image"], p["end_image"]],- keyframe_indices=[0, num_frames - 1],+ keyframe_images=kf_images,+ keyframe_indices=kf_indices,height=int(p["height"]),width=int(p["width"]),num_frames=num_frames,@@ -1283,7 +1345,13 @@ for line in sys.__stdin__:stage2_steps=int(p.get("stage2_steps", 3)),cfg_scale=float(p.get("cfg_scale", 3.0)),)- emit({"event": "log", "line": f"Keyframe FFLF — frames=[0, {num_frames-1}], pipeline cover-crops both to {kwargs['width']}x{kwargs['height']}"})+ emit({+ "event": "log",+ "line": (+ f"Keyframe {kf_mode_label} — indices={kf_indices}, "+ f"pipeline cover-crops all to {kwargs['width']}x{kwargs['height']}"+ ),+ })with _override_default_negative_prompt(p.get("negative_prompt")) as neg_active:if neg_active:emit({"event": "log", "line": "Avoid terms active via native CFG negative prompt."})