Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

multi-keyframe interpolation Layer 1 — helper accepts arbitrary keyframe lists

mrbizarro0cfee8c98975
diff --git a/docs/SDK_KEYFRAME_INTERPOLATION.md b/docs/SDK_KEYFRAME_INTERPOLATION.md
index c95bc7c..1d352d2 100644
--- a/docs/SDK_KEYFRAME_INTERPOLATION.md
+++ b/docs/SDK_KEYFRAME_INTERPOLATION.md
@@ -1,6 +1,6 @@
# Multi-keyframe interpolation — the SDK shot-composition primitive
-Status: **research + design doc.** Not yet implemented. Documents a known gap in Phosphene where the underlying engine supports more than the UI exposes.
+Status: **Layer 1 shipped 2026-05-06** — helper now accepts arbitrary keyframe lists. Layer 2 (panel HTTP form-parsing) and Layer 3 (UI) not yet. Engine has supported this all along; we just exposed it. See [Agent API contract](#agent-api-contract-layer-1-2026-05-06) for the call shape.
## TL;DR
@@ -123,37 +123,21 @@ This is the SDK's compositional primitive in one sentence: **the editor controls
Three layers, in order:
-### Layer 1 — helper (smallest change, biggest unlock)
+### Layer 1 — DONE (2026-05-06)
-`mlx_warm_helper.py` action `generate_keyframe`. Currently:
+Helper `generate_keyframe` action now accepts arbitrary `keyframe_images` + `keyframe_indices` lists with strict input validation. The hardcoded 2-keyframe path remains as a backward-compat fallback so the panel keeps working unchanged. See `mlx_warm_helper.py` lines ~1251-1360.
-```python
-keyframe_images=[p["start_image"], p["end_image"]],
-keyframe_indices=[0, num_frames - 1],
-```
-
-Change to read `keyframe_images` and `keyframe_indices` directly from the job spec (lists). Default to the existing 2-keyframe behavior if the spec only provides `start_image` + `end_image` (backward compat). New job spec accepts:
-
-```json
-{
- "action": "generate_keyframe",
- "params": {
- "keyframe_images": ["<path1>", "<path2>", "<path3>"],
- "keyframe_indices": [0, 60, 120],
- ...
- }
-}
-```
-
-Three lines of code, no model change, no upstream change.
+Full agent contract below.
-### Layer 2 — panel (form parsing + worker dispatch)
+### Layer 2 — panel HTTP form parsing — NOT YET
`mlx_ltx_panel.py` `mode == "keyframe"` branch. Currently parses `start_image` + `end_image` form fields and constructs the helper job spec. Change to parse a **list of keyframes** from the form (e.g. `keyframe_image_0`, `keyframe_image_1`, ... or a JSON-encoded list field). Build the lists for the helper.
Backward compat: if the form has `start_image` + `end_image` only, behave as today.
-### Layer 3 — UI (the user-visible surface)
+This is the layer the agent will use if it goes through the panel's HTTP queue endpoint instead of talking to the helper directly.
+
+### Layer 3 — UI (the user-visible surface) — NOT YET
The current Keyframe section in the panel HTML has two drop-zones. Replace with a list:
@@ -175,6 +159,156 @@ Once multi-keyframe ships, the Director Mode (HAI-155) gains the compositional p
The agent never has to "describe a scene in words and pray." Composition is concrete, predictable, controllable.
+## Agent API contract (Layer 1 — 2026-05-06)
+
+The helper subprocess (`mlx_warm_helper.py`) is a long-lived JSON-line server. Each line on stdin is one job; each line on stdout is one event. To use multi-keyframe interpolation, write a single JSON line to the helper's stdin with `action: "generate_keyframe"`.
+
+### Request shape
+
+```json
+{
+ "id": "<your-unique-job-id-string>",
+ "action": "generate_keyframe",
+ "params": {
+ "prompt": "A young woman walks across a sunlit kitchen, opens the fridge, takes out an apple.",
+ "keyframe_images": [
+ "/abs/path/to/keyframe_0.png",
+ "/abs/path/to/keyframe_1.png",
+ "/abs/path/to/keyframe_2.png"
+ ],
+ "keyframe_indices": [0, 60, 120],
+ "frames": 121,
+ "width": 768,
+ "height": 416,
+ "output_path": "/abs/path/to/output.mp4",
+ "seed": -1,
+ "stage1_steps": 15,
+ "stage2_steps": 3,
+ "cfg_scale": 3.0,
+ "negative_prompt": null,
+ "model_dir": null
+ }
+}
+```
+
+### Field rules
+
+| Field | Type | Required | Notes |
+|---|---|---|---|
+| `id` | str | yes | Echoed back in `done` / `error` events so the agent can match. Use any unique string (UUID, monotonic counter, etc). |
+| `action` | str | yes | Must be exactly `"generate_keyframe"`. |
+| `params.prompt` | str | yes | Scene description. The model uses this for the motion between keyframes; the keyframes themselves dominate composition. |
+| `params.keyframe_images` | list[str] | yes (multi-keyframe) | Absolute file paths. PNG/JPEG. Length must equal `keyframe_indices`. **N ≥ 2.** Files must exist at job-submission time. |
+| `params.keyframe_indices` | list[int] | yes (multi-keyframe) | Pixel-frame indices, 0-based, **strictly ascending**, all in `[0, frames-1]`. Must include at least 2 entries. Same length as `keyframe_images`. |
+| `params.start_image` + `params.end_image` | str | yes (FFLF backward-compat) | Used only if `keyframe_images`/`keyframe_indices` are absent. Equivalent to `keyframe_indices=[0, frames-1]`. |
+| `params.frames` | int | yes | Total pixel frames. Latent grid is 8 frames per slot, so use 8k+1 (49, 121, 241). 24 fps is hardcoded ⇒ 121 frames = ~5.0 sec. |
+| `params.width`, `params.height` | int | yes | Output dimensions. Comfortable tier (M-Max 64 GB) is clamped to **768×416** for keyframe jobs (OOM history at full 1280×704). |
+| `params.output_path` | str | yes | Absolute path where the resulting `.mp4` will be written (with audio). |
+| `params.seed` | int | optional | `-1` (default) = random, otherwise fixed. Echoed back in the `done` event as `seed_used` so the agent can reproduce. |
+| `params.stage1_steps` | int | optional | Default `15`. Stage-1 (half-res guided denoise) sigma count. Quality knob. |
+| `params.stage2_steps` | int | optional | Default `3`. Stage-2 (full-res refine) sigma count. |
+| `params.cfg_scale` | float | optional | Default `3.0`. Classifier-free-guidance strength. |
+| `params.negative_prompt` | str \| null | optional | If provided, used as native CFG negative. Null = use default. |
+| `params.model_dir` | str \| null | optional | Override the transformer directory. Null = default Q8 dev path. Don't change this unless you know why. |
+
+### Response events
+
+The helper emits JSON lines on stdout:
+
+```json
+{"event": "log", "line": "Keyframe multi-3kf — indices=[0, 60, 120], pipeline cover-crops all to 768x416"}
+{"event": "progress", "id": "...", "phase": "stage1", "step": 5, "total": 15, "pct": 33}
+{"event": "progress", "id": "...", "phase": "stage2", "step": 1, "total": 3, "pct": 90}
+{"event": "done", "id": "...", "output": "/abs/path/out.mp4", "elapsed_sec": 312.4, "seed_used": 1234567}
+```
+
+On any failure (validation error, OOM, model error):
+
+```json
+{"event": "error", "id": "...", "error": "<message>", "trace": "<python-traceback>"}
+```
+
+The helper continues running after errors; submit the next job whenever ready.
+
+### Validation errors (fail-fast at submission, before any GPU work)
+
+The helper rejects bad jobs immediately so agent bugs surface early:
+
+- `keyframe_images and keyframe_indices must both be provided as lists`
+- `keyframe_images and keyframe_indices must be lists`
+- `keyframe_images (N) and keyframe_indices (M) must have the same length`
+- `at least 2 keyframes required`
+- `keyframe image not found: <path>`
+- `keyframe_indices must be integers, got: <repr>`
+- `keyframe_index <i> out of range [0, <frames-1>]`
+- `keyframe_indices must be strictly ascending, got <list>`
+
+### Concurrency model
+
+The helper handles **one job at a time**. The agent's queue should serialize submissions or wait for `done` / `error` before submitting the next job. Multiple jobs interleaved on stdin will queue but the helper won't process the second until the first finishes.
+
+If you need parallelism, spawn multiple helpers (each takes ~30 GB VRAM + ~30s warm-up; M4 Max 64 GB fits 1 reliably, 2 is tight).
+
+### Wall-time budget
+
+For a 121-frame (5-sec) clip at 768×416 with `stage1_steps=15`, `stage2_steps=3`:
+
+- Cold helper (first job after start): ~30 s warm-up + ~5 min generate = ~5:30
+- Warm helper (subsequent jobs): ~5 min generate
+
+Going from 2 keyframes to 8 keyframes adds a few seconds of front-end VAE encoding, not significant against the ~5-min denoise.
+
+### Concrete example — Director Mode three-shot scene
+
+```python
+import json, subprocess, uuid, sys
+
+helper = subprocess.Popen(
+ ["./ltx-2-mlx/env/bin/python3.11", "-u", "mlx_warm_helper.py"],
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, bufsize=1,
+)
+
+def submit(job):
+ helper.stdin.write(json.dumps(job) + "\n")
+ helper.stdin.flush()
+ while True:
+ line = helper.stdout.readline()
+ if not line: raise RuntimeError("helper died")
+ evt = json.loads(line)
+ print(evt, file=sys.stderr)
+ if evt.get("event") == "done" and evt.get("id") == job["id"]: return evt
+ if evt.get("event") == "error" and evt.get("id") == job["id"]: raise RuntimeError(evt["error"])
+
+# Shot 1 — establishing wide. 4 keyframes for explicit choreography.
+result = submit({
+ "id": str(uuid.uuid4()),
+ "action": "generate_keyframe",
+ "params": {
+ "prompt": "A woman walks across a kitchen, dappled morning light through window blinds.",
+ "keyframe_images": [
+ "/tmp/agent/kitchen_01_start.png",
+ "/tmp/agent/kitchen_02_mid_left.png",
+ "/tmp/agent/kitchen_03_mid_right.png",
+ "/tmp/agent/kitchen_04_at_fridge.png",
+ ],
+ "keyframe_indices": [0, 40, 80, 120],
+ "frames": 121,
+ "width": 768, "height": 416,
+ "output_path": "/tmp/agent/shot_01.mp4",
+ "seed": 42,
+ }
+})
+# result["output"], result["elapsed_sec"], result["seed_used"]
+```
+
+### What's NOT in this layer
+
+- **Per-keyframe strength.** All keyframes are pinned at strength 1.0 (pure anchor). The model cannot "lightly suggest" a keyframe yet — it's all-or-nothing. Strength control requires an upstream patch in `ltx-pipelines-mlx`.
+- **Panel HTTP route.** Layer 2 is needed to reach this from the panel's queue (HTTP form-data). Today the only path is the helper's stdin.
+- **Panel UI.** Today's panel still has 2 image drop-zones. Adding more rows is Layer 3.
+- **Audio across keyframe boundaries between shots.** Each clip is its own audio render — boundaries are hard cuts. See "What this doesn't solve" below.
+
## Memory budget
A keyframe job today, on Comfortable tier (M-Max 64 GB), runs at clamped 768×416 (Y9 in Linear notes the OOM history at full 1280×704 — kept the clamp). N keyframes don't change peak memory significantly — each VAE-encode is one image at a time, and the sampler latent is the same shape regardless of how many slots are pinned.
diff --git a/docs/STATE.md b/docs/STATE.md
index f6c82b3..d66e353 100644
--- a/docs/STATE.md
+++ b/docs/STATE.md
@@ -204,7 +204,12 @@ Everything below is also tracked in Linear (HAI-150 → HAI-158 under the Phosph
**TL;DR**: ComfyGuy9000 demoed first-frame-last-frame method via `Deno2026/comfyui-deno-custom-nodes`. Phosphene's `ltx_pipelines_mlx.KeyframeInterpolationPipeline` already accepts arbitrary `list[Image]` keyframes + `list[int]` indices — but our panel/helper artificially restrict it to 2 keyframes (start + end). Exposing the full multi-keyframe API gives us the agentic-flow compositional primitive: agent picks N stills, model fills the motion, character is anchored at every shot start.
-**Implementation order**: helper (3 lines) → panel worker (form parsing) → UI (multi-keyframe drop-zone list).
+**Status (2026-05-06)**:
+- **Layer 1 — DONE.** Helper `generate_keyframe` action accepts arbitrary `keyframe_images` + `keyframe_indices` lists, with strict validation. Backward-compatible with the old `start_image`/`end_image` shape so the panel keeps working. Full agent contract in `docs/SDK_KEYFRAME_INTERPOLATION.md` § Agent API contract.
+- **Layer 2 (panel HTTP form-parsing) — NOT YET.** Needed if agents want to POST through the panel queue instead of talking to the helper directly via stdin.
+- **Layer 3 (panel UI multi-row keyframe list) — NOT YET.**
+
+**Today's agent path**: spawn a helper subprocess and write JSON jobs to its stdin. See the doc for a working Python example.
### Long-video research (Strategy A / B / C)
@@ -227,7 +232,7 @@ Future feature for 1-minute movies from one user idea:
3. Stitcher concatenates the K resulting mp4s with ffmpeg
4. Director UI in the panel — paste idea, see planned shot-list, edit any prompt, hit "Render Movie"
-Blocked on multi-keyframe interpolation shipping (the compositional primitive) + long-video research deliverable (per-shot length sweet spot).
+Compositional primitive **partially unblocked (2026-05-06)**: Layer 1 of multi-keyframe is live, so an agent can already submit jobs via helper stdin. Layer 2 (panel HTTP route) is the remaining piece for queue-based submission. Long-video research (per-shot length sweet spot) still pending Codex deep-research return.
### Speed optimization candidates (from May 4 research session)
diff --git a/mlx_warm_helper.py b/mlx_warm_helper.py
index 505aacd..d0ad764 100644
--- a/mlx_warm_helper.py
+++ b/mlx_warm_helper.py
@@ -1249,8 +1249,23 @@ for line in sys.__stdin__:
continue
if action == "generate_keyframe":
- # FFLF — two images anchored at frame 0 and frame N-1, model interpolates.
- # Like HQ this uses the Q8 dev transformer + two-stage refine.
+ # Keyframe interpolation — anchor images at chosen frame indices,
+ # model fills the motion between them. Uses the Q8 dev transformer
+ # + two-stage refine (same as HQ).
+ #
+ # Two input shapes are accepted:
+ #
+ # (A) Multi-keyframe (preferred — used by agents):
+ # "keyframe_images": list[str] absolute paths, length N >= 2
+ # "keyframe_indices": list[int] pixel-frame indices, length N,
+ # strictly ascending, all in [0, frames-1]
+ #
+ # (B) FFLF backward-compat (used by the panel today):
+ # "start_image": str
+ # "end_image": str
+ # Equivalent to multi-keyframe with indices [0, frames-1].
+ #
+ # If (A) fields are present they win; (B) is only checked when (A) is absent.
job_id = msg.get("id", "?")
p = msg.get("params", {}) or {}
model_dir = p.get("model_dir") or MODEL_ID
@@ -1261,19 +1276,66 @@ for line in sys.__stdin__:
try:
t0 = time.time()
configure_acceleration("off")
- for k in ("start_image", "end_image"):
- img = p.get(k)
- if not img or not os.path.exists(img):
- raise RuntimeError(f"{k} not found: {img}")
- pipe = get_kf_pipe(model_dir)
num_frames = int(p["frames"])
+
+ # ---- Resolve keyframes (multi-keyframe path or FFLF fallback) ----
+ kf_images_in = p.get("keyframe_images")
+ kf_indices_in = p.get("keyframe_indices")
+ if kf_images_in is not None or kf_indices_in is not None:
+ # Multi-keyframe — validate strictly so agent bugs surface early.
+ if kf_images_in is None or kf_indices_in is None:
+ raise RuntimeError(
+ "keyframe_images and keyframe_indices must both be provided as lists"
+ )
+ if not isinstance(kf_images_in, list) or not isinstance(kf_indices_in, list):
+ raise RuntimeError("keyframe_images and keyframe_indices must be lists")
+ if len(kf_images_in) != len(kf_indices_in):
+ raise RuntimeError(
+ f"keyframe_images ({len(kf_images_in)}) and "
+ f"keyframe_indices ({len(kf_indices_in)}) must have the same length"
+ )
+ if len(kf_images_in) < 2:
+ raise RuntimeError("at least 2 keyframes required")
+ for path in kf_images_in:
+ if not isinstance(path, str) or not os.path.exists(path):
+ raise RuntimeError(f"keyframe image not found: {path}")
+ idxs: list[int] = []
+ for i in kf_indices_in:
+ try:
+ idx = int(i)
+ except (TypeError, ValueError):
+ raise RuntimeError(f"keyframe_indices must be integers, got: {i!r}")
+ if idx < 0 or idx >= num_frames:
+ raise RuntimeError(
+ f"keyframe_index {idx} out of range [0, {num_frames - 1}]"
+ )
+ idxs.append(idx)
+ for a, b in zip(idxs, idxs[1:]):
+ if b <= a:
+ raise RuntimeError(
+ f"keyframe_indices must be strictly ascending, got {idxs}"
+ )
+ kf_images = list(kf_images_in)
+ kf_indices = idxs
+ kf_mode_label = f"multi-{len(kf_images)}kf"
+ else:
+ # FFLF backward-compat — start + end at the boundaries.
+ for k in ("start_image", "end_image"):
+ img = p.get(k)
+ if not img or not os.path.exists(img):
+ raise RuntimeError(f"{k} not found: {img}")
+ kf_images = [p["start_image"], p["end_image"]]
+ kf_indices = [0, num_frames - 1]
+ kf_mode_label = "FFLF"
+
+ pipe = get_kf_pipe(model_dir)
# Y1.037: short-clip VAE-streaming opt-out (Keyframe path).
_apply_vae_streaming_decision(num_frames)
kwargs = dict(
prompt=p["prompt"],
output_path=p["output_path"],
- keyframe_images=[p["start_image"], p["end_image"]],
- keyframe_indices=[0, num_frames - 1],
+ keyframe_images=kf_images,
+ keyframe_indices=kf_indices,
height=int(p["height"]),
width=int(p["width"]),
num_frames=num_frames,
@@ -1283,7 +1345,13 @@ for line in sys.__stdin__:
stage2_steps=int(p.get("stage2_steps", 3)),
cfg_scale=float(p.get("cfg_scale", 3.0)),
)
- emit({"event": "log", "line": f"Keyframe FFLF — frames=[0, {num_frames-1}], pipeline cover-crops both to {kwargs['width']}x{kwargs['height']}"})
+ emit({
+ "event": "log",
+ "line": (
+ f"Keyframe {kf_mode_label} — indices={kf_indices}, "
+ f"pipeline cover-crops all to {kwargs['width']}x{kwargs['height']}"
+ ),
+ })
with _override_default_negative_prompt(p.get("negative_prompt")) as neg_active:
if neg_active:
emit({"event": "log", "line": "Avoid terms active via native CFG negative prompt."})