Free, open-source AI video generator for Mac. Run LTX-2.5 and MiniMax H3 (Hailuo) locally on Apple Silicon: text-to-video and image-to-video with sound, lip-sync, character LoRA training, songs from lyrics (YuE2) and image editing (Qwen-Image-Edit). One-click install via Pinokio. Offline: no cloud, no API key, no NVIDIA GPU.
Python ai-filmmakingai-videoai-video-generatorapple-siliconhailuoimage-to-videolip-synclocal-ailora-trainingltx-2ltx-2-5ltx-videomacosminimax-h3mlxmusic-generationoffline-aipinokiotext-to-videovideo-generation
feat(agent): refine existing clips — "remake this with X" workflow
mrbizarroac4d3bf8239f
diff --git a/agent/prompts.py b/agent/prompts.pyindex e56305d..5b92237 100644--- a/agent/prompts.py+++ b/agent/prompts.py@@ -375,6 +375,44 @@ allocation gives the line + breath + look-out for a clean editor cut.{tools_doc}+# Variations — referencing + remaking existing clips++When the user references an existing clip and asks for a variation+("remake S5 with more pause", "redo the doctor reveal louder", "give+me another take of this"), use `inspect_clip` first.++The user references clips two ways:+- **Job id** — they paste `j-19dfe67c6e6-001` or click a clip in the+ Stage pane / gallery (the chat composer prefills with `Refine+ j-19dfe67c6e6-001:` followed by their request).+- **By name** — they say "the wife testimony", "the diaper one". You+ match against your session's `submitted_shots` labels first; if no+ match, ask which clip.++Workflow:++1. Call `inspect_clip(job_id=...)`. You get back the original+ `prompt`, `mode`, `quality`, `width`, `height`, `frames`,+ `duration_seconds`, `seed_used`, `label`, etc.+2. Apply the user's modification to the prompt textually. Don't+ rewrite from scratch — keep the master style, the dialogue+ (sacred), and the framing. Layer the change on top: "louder+ delivery", "longer pause before the line", "warmer light", etc.+3. Decide what duration to use. If the user says "make it longer",+ add 2-4 s. If they ask for more pause / silence, allocate 2-4 s+ more. Otherwise keep `duration_seconds`.+4. Submit with `submit_shot`. Use a label like the original PLUS+ a suffix: original "S5 Wife Legacy" → "S5 Wife Legacy v2".+ Quality / mode / dimensions = same as the source unless the user+ explicitly asks to change them. Seed = -1 (random) unless they+ ask for the same look (then pass `seed_used` from inspect_clip).++Don't auto-resubmit at higher quality unless the user asks. Quality+bumps cost real time (Standard ≈ 2× Balanced; High ≈ 3×).++Don't queue 4 variants in parallel unless the user says "give me+options" — default to ONE take per user request.+# When you're stuckIf you don't have enough information to plan, ASK. Plain text, nodiff --git a/agent/tools.py b/agent/tools.pyindex d8c6744..415c3e6 100644--- a/agent/tools.py+++ b/agent/tools.py@@ -723,6 +723,95 @@ def _generate_shot_images(args: dict, ops: PanelOps, session: dict) -> dict:}+@tool("inspect_clip")+def _inspect_clip(args: dict, ops: PanelOps, session: dict) -> dict:+ """Look up an existing rendered clip's full parameters from its sidecar.++ Used when the user references an earlier clip and asks for a+ variation — "remake S5 with more pause", "redo the doctor reveal at+ higher quality", "give me another take of this". Reads the .mp4.json+ sidecar that the panel writes next to every render and returns the+ original prompt, mode, quality, dimensions, frame count, seed, and+ any LoRAs / loras_json. Use the returned values as the BASELINE for+ your variation, then submit_shot with the user's modifications layered+ on top — same shot label suffixed (e.g. "S5 Wife Legacy v2") so the+ new render reads as a take, not a duplicate.++ Args (one of):+ job_id: str — agent's submitted shot id (e.g. "j-19dfe67c6e6-001"),+ OR a clip path like "/path/to/...mp4". The implementation+ accepts either.+ clip_path: str — alternative to job_id. Absolute mp4 path; the+ sidecar at <path>.json is read.++ Returns: {+ job_id, output_path, sidecar_path, status, elapsed_sec,+ prompt, negative_prompt, mode, quality, accel,+ width, height, frames, duration_seconds, seed_used,+ label, loras, hdr, upscale_method, upscale,+ }++ On miss: {"error": "no clip found for ..."}.+ """+ import json as _json++ job_id = (args.get("job_id") or "").strip()+ clip_path = (args.get("clip_path") or "").strip()+ if not job_id and not clip_path:+ raise _ToolValidationError("provide job_id or clip_path")++ # Resolve to a clip mp4 path.+ if job_id and not clip_path:+ j = ops.find_job(job_id)+ if not j:+ return {"error": f"no clip found for job_id {job_id}"}+ clip_path = (j.get("output_path") or "").strip()+ if not clip_path:+ return {"error": f"job {job_id} has no output_path yet (status={j.get('status')})"}++ p = Path(clip_path)+ if not p.is_absolute():+ p = (ops.outputs_dir / p).resolve()+ if not p.is_file():+ return {"error": f"clip not found on disk: {p}"}++ # Sidecar lives at <path>.json next to the mp4.+ sidecar = p.with_suffix(p.suffix + ".json")+ if not sidecar.is_file():+ return {"error": f"no sidecar at {sidecar} — clip was rendered before sidecars existed?"}+ try:+ data = _json.loads(sidecar.read_text(encoding="utf-8"))+ except (OSError, _json.JSONDecodeError) as e:+ return {"error": f"sidecar unreadable: {e}"}++ params = data.get("params") or {}+ frames = int(params.get("frames", 0)) or None+ duration = round((frames - 1) / 24.0, 2) if frames and frames > 1 else None++ return {+ "job_id": data.get("queue_id") or job_id or "",+ "output_path": str(p),+ "sidecar_path": str(sidecar),+ "status": "done",+ "elapsed_sec": data.get("elapsed_sec"),+ "prompt": params.get("prompt"),+ "negative_prompt": params.get("negative_prompt") or "",+ "mode": params.get("mode") or "t2v",+ "quality": params.get("quality") or "balanced",+ "accel": params.get("accel") or "off",+ "width": params.get("width"),+ "height": params.get("height"),+ "frames": frames,+ "duration_seconds": duration,+ "seed_used": params.get("seed_used") or params.get("seed"),+ "label": params.get("label") or params.get("preset_label") or "",+ "loras": params.get("loras") or [],+ "hdr": bool(params.get("hdr")),+ "upscale_method": params.get("upscale_method") or "",+ "upscale": params.get("upscale") or "",+ }++@tool("get_selected_anchors")def _get_selected_anchors(args: dict, ops: PanelOps, session: dict) -> dict:"""Return the user's anchor selections so far.diff --git a/mlx_ltx_panel.py b/mlx_ltx_panel.pyindex 71cef20..9b0aa0d 100644--- a/mlx_ltx_panel.py+++ b/mlx_ltx_panel.py@@ -7463,6 +7463,30 @@ HTML = r"""<!doctype html>.stage-output-cell.failed .badge {background: rgba(207,34,46,0.85);}+ /* Refine button — small circular icon top-right of every output cell.+ Distinct from the main click area (which opens the lightbox to+ play the clip). Click → drops a reference into the chat composer+ so the user can ask the agent for a variation. */+ .stage-output-cell .refine-btn {+ position: absolute; top: 4px; right: 4px;+ width: 22px; height: 22px;+ border-radius: 50%;+ background: rgba(0,0,0,0.6);+ color: white;+ border: 1px solid rgba(255,255,255,0.15);+ font-size: 11px;+ display: flex; align-items: center; justify-content: center;+ cursor: pointer;+ opacity: 0;+ transition: opacity 0.15s, background 0.15s;+ z-index: 1;+ width: 22px;+ }+ .stage-output-cell:hover .refine-btn { opacity: 1; }+ .stage-output-cell .refine-btn:hover {+ background: var(--accent);+ border-color: var(--accent);+ }.stage-empty {text-align: center; color: var(--muted);font-size: 12px; padding: 16px;@@ -7697,6 +7721,64 @@ HTML = r"""<!doctype html>.stage-lightbox .close-btn:hover {background: rgba(255,255,255,0.14);}+ .stage-lightbox .refine-btn-large {+ position: absolute;+ top: 16px; right: 110px;+ background: var(--accent);+ color: white;+ border: 1px solid var(--accent);+ border-radius: 8px;+ padding: 8px 16px;+ font-size: 12px; font-weight: 600;+ cursor: pointer; width: auto;+ box-shadow: 0 4px 14px rgba(47,129,247,0.4);+ }+ .stage-lightbox .refine-btn-large:hover {+ background: var(--accent-bright);+ }++ /* ---- Composer reference chip ----+ When the user clicks Refine on a clip, a chip appears just above+ the textarea with the clip's label + a × to clear it. The chip's+ data is sent verbatim as a prefix to the user's next message so+ the agent picks up the reference and calls inspect_clip first. */+ .agent-ref-chip {+ display: none;+ align-items: center; gap: 8px;+ padding: 6px 10px 6px 12px;+ margin: 0 0 6px 0;+ background: var(--accent-dim);+ border: 1px solid var(--accent);+ border-radius: 999px;+ font-size: 12px;+ color: var(--accent-bright);+ max-width: fit-content;+ animation: agent-fade-in 0.2s ease;+ }+ .agent-ref-chip.visible { display: inline-flex; }+ .agent-ref-chip .ref-icon {+ font-size: 11px;+ }+ .agent-ref-chip .ref-label {+ font-family: ui-monospace, SFMono-Regular, Menlo, monospace;+ font-size: 11px;+ max-width: 320px;+ overflow: hidden; text-overflow: ellipsis; white-space: nowrap;+ }+ .agent-ref-chip .clear {+ background: transparent;+ color: var(--accent-bright);+ border: none;+ width: 18px; height: 18px;+ border-radius: 50%;+ cursor: pointer;+ font-size: 14px;+ line-height: 1;+ display: flex; align-items: center; justify-content: center;+ }+ .agent-ref-chip .clear:hover {+ background: rgba(255,255,255,0.1);+ }/* =========================================================FULLSCREEN / FOCUS MODE — pro-app polish@@ -8402,6 +8484,17 @@ HTML = r"""<!doctype html><div class="agent-chat" id="agentChat"></div><div class="agent-composer">+ <!-- Reference chip: appears when the user clicks Refine on an+ existing clip. Carries the clip's job_id; on Send, we+ prepend "Refine <job_id>: " to the user's message so the+ agent calls inspect_clip and treats the request as a+ variation. Clear with the × button. -->+ <div class="agent-ref-chip" id="agentRefChip" style="margin-left:auto;margin-right:auto;max-width:720px">+ <span class="ref-icon">↻</span>+ <span style="color:var(--muted);font-size:11px">refine</span>+ <span class="ref-label" id="agentRefChipLabel">…</span>+ <button type="button" class="clear" onclick="agentClearRefine()" title="Cancel refine">×</button>+ </div><div class="agent-composer-wrap"><textarea id="agentInput"placeholder="Paste a script, or describe a piece. The agent will plan, estimate the wall time, and queue overnight."@@ -8919,9 +9012,12 @@ HTML = r"""<!doctype html></div></aside>- <!-- Lightbox for stage outputs — plays a finished mp4 full-size on click. -->+ <!-- Lightbox for stage outputs — plays a finished mp4 full-size on click.+ Refine button references the playing clip in the composer so the+ user can ask the agent for a variation while watching. --><div class="stage-lightbox" id="agentStageLightbox" onclick="if(event.target===this)agentStageLightboxClose()"><button class="close-btn" onclick="agentStageLightboxClose()">Close · Esc</button>+ <button class="refine-btn-large" id="agentStageLightboxRefine" onclick="agentStageLightboxRefine()">↻ Refine this clip</button><video id="agentStageLightboxVideo" controls preload="metadata"></video></div></main>@@ -13032,11 +13128,23 @@ async function agentSend() {if (!sess) return;}+ // If a Refine reference is set, prepend "Refine <jobid> (<label>): " to+ // the user's message so the agent picks it up as a variation request.+ // Clear the chip after so the next message is a normal one.+ let outgoing = text;+ if (window.AGENT_REFINE) {+ const r = window.AGENT_REFINE;+ const ref = r.jobId || r.clipPath;+ const lbl = r.label ? ` (${r.label})` : '';+ outgoing = `Refine ${ref}${lbl}: ${text}`;+ agentClearRefine();+ }+const chat = document.getElementById('agentChat');// Clear empty-state if present, then append user bubble + typingconst empty = chat.querySelector('.agent-empty');if (empty) empty.remove();- chat.appendChild(renderMessage({kind: 'user', content: text}));+ chat.appendChild(renderMessage({kind: 'user', content: outgoing}));chat.appendChild(renderTypingRow('Drafting plan'));chat.scrollTop = chat.scrollHeight;@@ -13081,7 +13189,7 @@ async function agentSend() {{method: 'POST',headers: {'Content-Type': 'application/json'},- body: JSON.stringify({message: text}),+ body: JSON.stringify({message: outgoing}),});const j = await r.json();@@ -13793,7 +13901,18 @@ function agentStageRender(status, sess) {v.preload = 'metadata';v.muted = true;cell.appendChild(v);- cell.addEventListener('click', () => agentStageLightboxOpen(o.output_path, o.label));+ cell.addEventListener('click', () => agentStageLightboxOpen(o.output_path, o.label, o.id));+ // Refine button (overlay top-right): "give me a variation of this clip"+ const refine = document.createElement('button');+ refine.type = 'button';+ refine.className = 'refine-btn';+ refine.title = 'Refine this clip — start a variation in the chat';+ refine.textContent = '↻';+ refine.addEventListener('click', (e) => {+ e.stopPropagation();+ agentSetRefine({jobId: o.id, label: o.label, clipPath: o.output_path});+ });+ cell.appendChild(refine);} else {cell.style.display = 'flex';cell.style.alignItems = 'center';@@ -13871,12 +13990,17 @@ function agentFmtDur(seconds) {return h + 'h ' + (mr < 10 ? '0' : '') + mr + 'm';}-function agentStageLightboxOpen(path, label) {+// Track which clip is currently in the lightbox so the Refine button+// has something to reference when clicked.+window.AGENT_STAGE.lightboxCurrent = null;++function agentStageLightboxOpen(path, label, jobId) {const lb = document.getElementById('agentStageLightbox');const v = document.getElementById('agentStageLightboxVideo');if (!lb || !v) return;v.src = '/file?path=' + encodeURIComponent(path);v.title = label || '';+ window.AGENT_STAGE.lightboxCurrent = {jobId: jobId || null, label: label || '', clipPath: path};lb.classList.add('open');v.play().catch(() => {});}@@ -13887,9 +14011,49 @@ function agentStageLightboxClose() {if (!lb || !v) return;v.pause();v.src = '';+ window.AGENT_STAGE.lightboxCurrent = null;lb.classList.remove('open');}+function agentStageLightboxRefine() {+ const cur = window.AGENT_STAGE.lightboxCurrent;+ if (!cur) return;+ agentStageLightboxClose();+ agentSetRefine(cur);+}++// ---- Composer reference chip — "Refine this clip" -----------------------+// When the user clicks ↻ on a stage output (or the Refine button in the+// lightbox), we set a refine reference. The chip shows above the textarea;+// on next Send, the user's message is prepended with "Refine <job_id>: "+// so the agent calls inspect_clip and treats the rest as the requested+// modification. Clear with × on the chip.+window.AGENT_REFINE = null; // {jobId, label, clipPath}++function agentSetRefine(ref) {+ window.AGENT_REFINE = ref;+ const chip = document.getElementById('agentRefChip');+ const lbl = document.getElementById('agentRefChipLabel');+ if (!chip || !lbl) return;+ lbl.textContent = ref.label || ref.jobId || ref.clipPath || 'clip';+ lbl.title = ref.jobId ? `${ref.label || ''} · ${ref.jobId}` : ref.clipPath;+ chip.classList.add('visible');+ // Bring focus to the composer so the user can type their refinement.+ const ta = document.getElementById('agentInput');+ if (ta) {+ ta.placeholder = 'How should this clip be different? (e.g. "more pause", "warmer light", "longer take")';+ setTimeout(() => ta.focus(), 50);+ }+}++function agentClearRefine() {+ window.AGENT_REFINE = null;+ const chip = document.getElementById('agentRefChip');+ if (chip) chip.classList.remove('visible');+ const ta = document.getElementById('agentInput');+ if (ta) ta.placeholder = 'Paste a script, or describe a piece. The agent will plan, estimate the wall time, and queue overnight.';+}+// Esc closes the stage lightbox first (before falling through to the// settings modal / fullscreen exit handlers).document.addEventListener('keydown', (e) => {