Skip to content

Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.

feat(agent): refine existing clips — "remake this with X" workflow

mrbizarroac4d3bf8239f
diff --git a/agent/prompts.py b/agent/prompts.py
index e56305d..5b92237 100644
--- a/agent/prompts.py
+++ b/agent/prompts.py
@@ -375,6 +375,44 @@ allocation gives the line + breath + look-out for a clean editor cut.
{tools_doc}
+# Variations — referencing + remaking existing clips
+
+When the user references an existing clip and asks for a variation
+("remake S5 with more pause", "redo the doctor reveal louder", "give
+me another take of this"), use `inspect_clip` first.
+
+The user references clips two ways:
+- **Job id** — they paste `j-19dfe67c6e6-001` or click a clip in the
+ Stage pane / gallery (the chat composer prefills with `Refine
+ j-19dfe67c6e6-001:` followed by their request).
+- **By name** — they say "the wife testimony", "the diaper one". You
+ match against your session's `submitted_shots` labels first; if no
+ match, ask which clip.
+
+Workflow:
+
+1. Call `inspect_clip(job_id=...)`. You get back the original
+ `prompt`, `mode`, `quality`, `width`, `height`, `frames`,
+ `duration_seconds`, `seed_used`, `label`, etc.
+2. Apply the user's modification to the prompt textually. Don't
+ rewrite from scratch — keep the master style, the dialogue
+ (sacred), and the framing. Layer the change on top: "louder
+ delivery", "longer pause before the line", "warmer light", etc.
+3. Decide what duration to use. If the user says "make it longer",
+ add 2-4 s. If they ask for more pause / silence, allocate 2-4 s
+ more. Otherwise keep `duration_seconds`.
+4. Submit with `submit_shot`. Use a label like the original PLUS
+ a suffix: original "S5 Wife Legacy" → "S5 Wife Legacy v2".
+ Quality / mode / dimensions = same as the source unless the user
+ explicitly asks to change them. Seed = -1 (random) unless they
+ ask for the same look (then pass `seed_used` from inspect_clip).
+
+Don't auto-resubmit at higher quality unless the user asks. Quality
+bumps cost real time (Standard ≈ 2× Balanced; High ≈ 3×).
+
+Don't queue 4 variants in parallel unless the user says "give me
+options" — default to ONE take per user request.
+
# When you're stuck
If you don't have enough information to plan, ASK. Plain text, no
diff --git a/agent/tools.py b/agent/tools.py
index d8c6744..415c3e6 100644
--- a/agent/tools.py
+++ b/agent/tools.py
@@ -723,6 +723,95 @@ def _generate_shot_images(args: dict, ops: PanelOps, session: dict) -> dict:
}
+@tool("inspect_clip")
+def _inspect_clip(args: dict, ops: PanelOps, session: dict) -> dict:
+ """Look up an existing rendered clip's full parameters from its sidecar.
+
+ Used when the user references an earlier clip and asks for a
+ variation — "remake S5 with more pause", "redo the doctor reveal at
+ higher quality", "give me another take of this". Reads the .mp4.json
+ sidecar that the panel writes next to every render and returns the
+ original prompt, mode, quality, dimensions, frame count, seed, and
+ any LoRAs / loras_json. Use the returned values as the BASELINE for
+ your variation, then submit_shot with the user's modifications layered
+ on top — same shot label suffixed (e.g. "S5 Wife Legacy v2") so the
+ new render reads as a take, not a duplicate.
+
+ Args (one of):
+ job_id: str — agent's submitted shot id (e.g. "j-19dfe67c6e6-001"),
+ OR a clip path like "/path/to/...mp4". The implementation
+ accepts either.
+ clip_path: str — alternative to job_id. Absolute mp4 path; the
+ sidecar at <path>.json is read.
+
+ Returns: {
+ job_id, output_path, sidecar_path, status, elapsed_sec,
+ prompt, negative_prompt, mode, quality, accel,
+ width, height, frames, duration_seconds, seed_used,
+ label, loras, hdr, upscale_method, upscale,
+ }
+
+ On miss: {"error": "no clip found for ..."}.
+ """
+ import json as _json
+
+ job_id = (args.get("job_id") or "").strip()
+ clip_path = (args.get("clip_path") or "").strip()
+ if not job_id and not clip_path:
+ raise _ToolValidationError("provide job_id or clip_path")
+
+ # Resolve to a clip mp4 path.
+ if job_id and not clip_path:
+ j = ops.find_job(job_id)
+ if not j:
+ return {"error": f"no clip found for job_id {job_id}"}
+ clip_path = (j.get("output_path") or "").strip()
+ if not clip_path:
+ return {"error": f"job {job_id} has no output_path yet (status={j.get('status')})"}
+
+ p = Path(clip_path)
+ if not p.is_absolute():
+ p = (ops.outputs_dir / p).resolve()
+ if not p.is_file():
+ return {"error": f"clip not found on disk: {p}"}
+
+ # Sidecar lives at <path>.json next to the mp4.
+ sidecar = p.with_suffix(p.suffix + ".json")
+ if not sidecar.is_file():
+ return {"error": f"no sidecar at {sidecar} — clip was rendered before sidecars existed?"}
+ try:
+ data = _json.loads(sidecar.read_text(encoding="utf-8"))
+ except (OSError, _json.JSONDecodeError) as e:
+ return {"error": f"sidecar unreadable: {e}"}
+
+ params = data.get("params") or {}
+ frames = int(params.get("frames", 0)) or None
+ duration = round((frames - 1) / 24.0, 2) if frames and frames > 1 else None
+
+ return {
+ "job_id": data.get("queue_id") or job_id or "",
+ "output_path": str(p),
+ "sidecar_path": str(sidecar),
+ "status": "done",
+ "elapsed_sec": data.get("elapsed_sec"),
+ "prompt": params.get("prompt"),
+ "negative_prompt": params.get("negative_prompt") or "",
+ "mode": params.get("mode") or "t2v",
+ "quality": params.get("quality") or "balanced",
+ "accel": params.get("accel") or "off",
+ "width": params.get("width"),
+ "height": params.get("height"),
+ "frames": frames,
+ "duration_seconds": duration,
+ "seed_used": params.get("seed_used") or params.get("seed"),
+ "label": params.get("label") or params.get("preset_label") or "",
+ "loras": params.get("loras") or [],
+ "hdr": bool(params.get("hdr")),
+ "upscale_method": params.get("upscale_method") or "",
+ "upscale": params.get("upscale") or "",
+ }
+
+
@tool("get_selected_anchors")
def _get_selected_anchors(args: dict, ops: PanelOps, session: dict) -> dict:
"""Return the user's anchor selections so far.
diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.py
index 71cef20..9b0aa0d 100644
--- a/mlx_ltx_panel.py
+++ b/mlx_ltx_panel.py
@@ -7463,6 +7463,30 @@ HTML = r"""<!doctype html>
.stage-output-cell.failed .badge {
background: rgba(207,34,46,0.85);
}
+ /* Refine button — small circular icon top-right of every output cell.
+ Distinct from the main click area (which opens the lightbox to
+ play the clip). Click → drops a reference into the chat composer
+ so the user can ask the agent for a variation. */
+ .stage-output-cell .refine-btn {
+ position: absolute; top: 4px; right: 4px;
+ width: 22px; height: 22px;
+ border-radius: 50%;
+ background: rgba(0,0,0,0.6);
+ color: white;
+ border: 1px solid rgba(255,255,255,0.15);
+ font-size: 11px;
+ display: flex; align-items: center; justify-content: center;
+ cursor: pointer;
+ opacity: 0;
+ transition: opacity 0.15s, background 0.15s;
+ z-index: 1;
+ width: 22px;
+ }
+ .stage-output-cell:hover .refine-btn { opacity: 1; }
+ .stage-output-cell .refine-btn:hover {
+ background: var(--accent);
+ border-color: var(--accent);
+ }
.stage-empty {
text-align: center; color: var(--muted);
font-size: 12px; padding: 16px;
@@ -7697,6 +7721,64 @@ HTML = r"""<!doctype html>
.stage-lightbox .close-btn:hover {
background: rgba(255,255,255,0.14);
}
+ .stage-lightbox .refine-btn-large {
+ position: absolute;
+ top: 16px; right: 110px;
+ background: var(--accent);
+ color: white;
+ border: 1px solid var(--accent);
+ border-radius: 8px;
+ padding: 8px 16px;
+ font-size: 12px; font-weight: 600;
+ cursor: pointer; width: auto;
+ box-shadow: 0 4px 14px rgba(47,129,247,0.4);
+ }
+ .stage-lightbox .refine-btn-large:hover {
+ background: var(--accent-bright);
+ }
+
+ /* ---- Composer reference chip ----
+ When the user clicks Refine on a clip, a chip appears just above
+ the textarea with the clip's label + a × to clear it. The chip's
+ data is sent verbatim as a prefix to the user's next message so
+ the agent picks up the reference and calls inspect_clip first. */
+ .agent-ref-chip {
+ display: none;
+ align-items: center; gap: 8px;
+ padding: 6px 10px 6px 12px;
+ margin: 0 0 6px 0;
+ background: var(--accent-dim);
+ border: 1px solid var(--accent);
+ border-radius: 999px;
+ font-size: 12px;
+ color: var(--accent-bright);
+ max-width: fit-content;
+ animation: agent-fade-in 0.2s ease;
+ }
+ .agent-ref-chip.visible { display: inline-flex; }
+ .agent-ref-chip .ref-icon {
+ font-size: 11px;
+ }
+ .agent-ref-chip .ref-label {
+ font-family: ui-monospace, SFMono-Regular, Menlo, monospace;
+ font-size: 11px;
+ max-width: 320px;
+ overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
+ }
+ .agent-ref-chip .clear {
+ background: transparent;
+ color: var(--accent-bright);
+ border: none;
+ width: 18px; height: 18px;
+ border-radius: 50%;
+ cursor: pointer;
+ font-size: 14px;
+ line-height: 1;
+ display: flex; align-items: center; justify-content: center;
+ }
+ .agent-ref-chip .clear:hover {
+ background: rgba(255,255,255,0.1);
+ }
/* =========================================================
FULLSCREEN / FOCUS MODE — pro-app polish
@@ -8402,6 +8484,17 @@ HTML = r"""<!doctype html>
<div class="agent-chat" id="agentChat"></div>
<div class="agent-composer">
+ <!-- Reference chip: appears when the user clicks Refine on an
+ existing clip. Carries the clip's job_id; on Send, we
+ prepend "Refine <job_id>: " to the user's message so the
+ agent calls inspect_clip and treats the request as a
+ variation. Clear with the × button. -->
+ <div class="agent-ref-chip" id="agentRefChip" style="margin-left:auto;margin-right:auto;max-width:720px">
+ <span class="ref-icon">↻</span>
+ <span style="color:var(--muted);font-size:11px">refine</span>
+ <span class="ref-label" id="agentRefChipLabel">…</span>
+ <button type="button" class="clear" onclick="agentClearRefine()" title="Cancel refine">×</button>
+ </div>
<div class="agent-composer-wrap">
<textarea id="agentInput"
placeholder="Paste a script, or describe a piece. The agent will plan, estimate the wall time, and queue overnight."
@@ -8919,9 +9012,12 @@ HTML = r"""<!doctype html>
</div>
</aside>
- <!-- Lightbox for stage outputs — plays a finished mp4 full-size on click. -->
+ <!-- Lightbox for stage outputs — plays a finished mp4 full-size on click.
+ Refine button references the playing clip in the composer so the
+ user can ask the agent for a variation while watching. -->
<div class="stage-lightbox" id="agentStageLightbox" onclick="if(event.target===this)agentStageLightboxClose()">
<button class="close-btn" onclick="agentStageLightboxClose()">Close · Esc</button>
+ <button class="refine-btn-large" id="agentStageLightboxRefine" onclick="agentStageLightboxRefine()">↻ Refine this clip</button>
<video id="agentStageLightboxVideo" controls preload="metadata"></video>
</div>
</main>
@@ -13032,11 +13128,23 @@ async function agentSend() {
if (!sess) return;
}
+ // If a Refine reference is set, prepend "Refine <jobid> (<label>): " to
+ // the user's message so the agent picks it up as a variation request.
+ // Clear the chip after so the next message is a normal one.
+ let outgoing = text;
+ if (window.AGENT_REFINE) {
+ const r = window.AGENT_REFINE;
+ const ref = r.jobId || r.clipPath;
+ const lbl = r.label ? ` (${r.label})` : '';
+ outgoing = `Refine ${ref}${lbl}: ${text}`;
+ agentClearRefine();
+ }
+
const chat = document.getElementById('agentChat');
// Clear empty-state if present, then append user bubble + typing
const empty = chat.querySelector('.agent-empty');
if (empty) empty.remove();
- chat.appendChild(renderMessage({kind: 'user', content: text}));
+ chat.appendChild(renderMessage({kind: 'user', content: outgoing}));
chat.appendChild(renderTypingRow('Drafting plan'));
chat.scrollTop = chat.scrollHeight;
@@ -13081,7 +13189,7 @@ async function agentSend() {
{
method: 'POST',
headers: {'Content-Type': 'application/json'},
- body: JSON.stringify({message: text}),
+ body: JSON.stringify({message: outgoing}),
}
);
const j = await r.json();
@@ -13793,7 +13901,18 @@ function agentStageRender(status, sess) {
v.preload = 'metadata';
v.muted = true;
cell.appendChild(v);
- cell.addEventListener('click', () => agentStageLightboxOpen(o.output_path, o.label));
+ cell.addEventListener('click', () => agentStageLightboxOpen(o.output_path, o.label, o.id));
+ // Refine button (overlay top-right): "give me a variation of this clip"
+ const refine = document.createElement('button');
+ refine.type = 'button';
+ refine.className = 'refine-btn';
+ refine.title = 'Refine this clip — start a variation in the chat';
+ refine.textContent = '↻';
+ refine.addEventListener('click', (e) => {
+ e.stopPropagation();
+ agentSetRefine({jobId: o.id, label: o.label, clipPath: o.output_path});
+ });
+ cell.appendChild(refine);
} else {
cell.style.display = 'flex';
cell.style.alignItems = 'center';
@@ -13871,12 +13990,17 @@ function agentFmtDur(seconds) {
return h + 'h ' + (mr < 10 ? '0' : '') + mr + 'm';
}
-function agentStageLightboxOpen(path, label) {
+// Track which clip is currently in the lightbox so the Refine button
+// has something to reference when clicked.
+window.AGENT_STAGE.lightboxCurrent = null;
+
+function agentStageLightboxOpen(path, label, jobId) {
const lb = document.getElementById('agentStageLightbox');
const v = document.getElementById('agentStageLightboxVideo');
if (!lb || !v) return;
v.src = '/file?path=' + encodeURIComponent(path);
v.title = label || '';
+ window.AGENT_STAGE.lightboxCurrent = {jobId: jobId || null, label: label || '', clipPath: path};
lb.classList.add('open');
v.play().catch(() => {});
}
@@ -13887,9 +14011,49 @@ function agentStageLightboxClose() {
if (!lb || !v) return;
v.pause();
v.src = '';
+ window.AGENT_STAGE.lightboxCurrent = null;
lb.classList.remove('open');
}
+function agentStageLightboxRefine() {
+ const cur = window.AGENT_STAGE.lightboxCurrent;
+ if (!cur) return;
+ agentStageLightboxClose();
+ agentSetRefine(cur);
+}
+
+// ---- Composer reference chip — "Refine this clip" -----------------------
+// When the user clicks ↻ on a stage output (or the Refine button in the
+// lightbox), we set a refine reference. The chip shows above the textarea;
+// on next Send, the user's message is prepended with "Refine <job_id>: "
+// so the agent calls inspect_clip and treats the rest as the requested
+// modification. Clear with × on the chip.
+window.AGENT_REFINE = null; // {jobId, label, clipPath}
+
+function agentSetRefine(ref) {
+ window.AGENT_REFINE = ref;
+ const chip = document.getElementById('agentRefChip');
+ const lbl = document.getElementById('agentRefChipLabel');
+ if (!chip || !lbl) return;
+ lbl.textContent = ref.label || ref.jobId || ref.clipPath || 'clip';
+ lbl.title = ref.jobId ? `${ref.label || ''} · ${ref.jobId}` : ref.clipPath;
+ chip.classList.add('visible');
+ // Bring focus to the composer so the user can type their refinement.
+ const ta = document.getElementById('agentInput');
+ if (ta) {
+ ta.placeholder = 'How should this clip be different? (e.g. "more pause", "warmer light", "longer take")';
+ setTimeout(() => ta.focus(), 50);
+ }
+}
+
+function agentClearRefine() {
+ window.AGENT_REFINE = null;
+ const chip = document.getElementById('agentRefChip');
+ if (chip) chip.classList.remove('visible');
+ const ta = document.getElementById('agentInput');
+ if (ta) ta.placeholder = 'Paste a script, or describe a piece. The agent will plan, estimate the wall time, and queue overnight.';
+}
+
// Esc closes the stage lightbox first (before falling through to the
// settings modal / fullscreen exit handlers).
document.addEventListener('keydown', (e) => {