The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.
Rust ffmpegfree-video-editormodel-context-protocolnon-linear-editoroffline-firstopen-source-video-editorself-hostedvideo-editingvideo-editing-automationvideo-editing-softwarevideo-editing-toolsvideo-editorvoice-cloningyoutube-automationyoutube-shorts
feat(preview): stream the engine's true composite while stacked clips play
Jareer07c18c33e243
diff --git a/desktop/docs/decisions/0009-the-monitor-streams-the-true-frame.md b/desktop/docs/decisions/0009-the-monitor-streams-the-true-frame.mdnew file mode 100644index 00000000..842a69e8--- /dev/null+++ b/desktop/docs/decisions/0009-the-monitor-streams-the-true-frame.md@@ -0,0 +1,50 @@+# 0009 - The monitor streams the true frame where the approximation is wrong++## Decision+While the transport runs, the monitor pulls the engine's real composite -+the exporter's own plan, pool and compositor via `preview_frame` - in a loop:+request the frame at the transport's current position, present it, request+the next the moment it lands. The loop engages only while **two or more+visual layers** sit under the playhead; single-layer stretches keep the+webview element's smooth playback, exactly as before. Streamed frames are+capped at 640 px on the long edge; the paused dwell still re-fetches at 960+the moment the transport stops.++This is the presentation half that 0007 left open ("presenting pooled frames+continuously against the transport clock"), built on the seam that already+existed rather than a new one.++## Why this shape+- **Pull, not push.** The UI asks for one frame at the *current* position and+ does not ask again until it arrives. Backpressure is structural: a slow+ machine gets a lower rate, never a growing queue of stale frames, and the+ host's pool mutex serialises requests in order - which is also what makes+ consecutive requests warm roll-forwards instead of seeks.+- **Only where the element is wrong.** The `<video>` element is exact for one+ layer and *cannot composite* a stack. Streaming everywhere would trade the+ common case's 60 fps smoothness for uniform choppiness; streaming only on+ stacks buys truth precisely where the approximation lies.+- **The IPC budget holds.** 640-wide RGBA is ~1 MB a frame as an ArrayBuffer+ response; at the 15-30 fps the pool sustains for warm readers that is well+ inside what Tauri's IPC moves locally, and it degrades by rate, not by+ failure.++## What this is not+- **Not a vsync-locked presentation.** Frames display one fetch-latency+ behind the transport and at whatever rate decoding sustains. It is honest+ compositing, not final playback smoothness.+- **Not the native surface.** Decision 0002's endgame - a GPU surface+ positioned to the frame box, engine-presented - still stands. When it+ arrives it replaces the *transport* of pixels (IPC → surface); the plan,+ pool and compositor this loop exercises are exactly what it will present+ from, which is why building this first is not throwaway.++## What it costs+- Streamed frames are visibly softer than the element (640 upscaled) and+ arrive a beat late. Acceptable on stacks, which previously showed+ something *wrong* rather than something soft.+- A failed frame drops back to the approximation silently (logged host-side).++## What would change our mind+The native surface landing - at which point this loop is deleted, not+adapted, the same clean demolition 0004 got.diff --git a/desktop/src/App.tsx b/desktop/src/App.tsxindex 86cff743..10e132ec 100644--- a/desktop/src/App.tsx+++ b/desktop/src/App.tsx@@ -1,5 +1,5 @@import { useCallback, useEffect, useMemo, useRef, useState } from "react";-import { invoke, isTauri } from "@tauri-apps/api/core";+import { isTauri } from "@tauri-apps/api/core";import { open as openDialog } from "@tauri-apps/plugin-dialog";import { getCurrentWebview } from "@tauri-apps/api/webview";import { getCurrentWindow } from "@tauri-apps/api/window";@@ -45,7 +45,9 @@ import {type TimelineMeta,} from "./lib/editor";import {+ audioSetClips,editorApply,+ onAudioError,editorClose,editorOpen,editorRedo,@@ -388,6 +390,26 @@ function Editor({return () => window.clearTimeout(timer);}, [toast]);+ // Playback failures surface as the toast instead of dying in the dev+ // console - a mute app must never be a mystery. Identical messages within+ // a burst (one per clip of a broken import, say) collapse into one.+ useEffect(() => {+ if (!isTauri()) return;+ let unlisten: (() => void) | null = null;+ let lastMessage = "";+ let lastAt = 0;+ void onAudioError((message) => {+ const now = Date.now();+ if (message === lastMessage && now - lastAt < 10_000) return;+ lastMessage = message;+ lastAt = now;+ setToast({ id: now, message, failed: true });+ }).then((stop) => {+ unlisten = stop;+ });+ return () => unlisten?.();+ }, []);+// Bring back every font the project names, once, when it opens. A font// whose file has moved is marked rather than dropped.useEffect(() => {@@ -563,7 +585,7 @@ function Editor({useEffect(() => {if (!isTauri()) return;const timer = window.setTimeout(() => {- void invoke("audio_set_clips", { project: session.path, clips: audibleClips }).catch(+ void audioSetClips(session.path, audibleClips).catch((cause: unknown) => console.error("WolfCut: could not update the mix", cause),);}, 150);@@ -1204,9 +1226,12 @@ function Editor({} | null>(null);const stillToken = useRef(0);useEffect(() => {+ // While playing, the streaming loop below owns the still; clearing it+ // here on every playhead tick would fight the stream frame by frame.+ if (playing) return;const token = ++stillToken.current;setEngineStill(null);- if (playing || !loaded || exportClips.length === 0) return;+ if (!loaded || exportClips.length === 0) return;const timer = window.setTimeout(() => {// Preview resolution: the frame's aspect, capped for IPC weight. Even@@ -1233,6 +1258,73 @@ function Editor({return () => window.clearTimeout(timer);}, [playing, loaded, playhead, exportClips, frame, session.rateNum, session.rateDen]);+ /**+ * Streaming truth: while playing, present the engine's real composite+ * against the transport clock wherever the element preview is wrong.+ *+ * The DOM approximation is exact for a single visible layer and cannot+ * composite a stack, so the loop engages only while two or more visual+ * layers sit under the playhead - single-clip stretches keep the smooth+ * element playback. Pull, not push: request the frame at the transport's+ * current position, present it, request the next the moment it lands. The+ * loop runs at whatever rate the machine sustains and can never queue+ * stale frames; the pool's warm readers make consecutive requests rolls,+ * not seeks. This is the presentation half of engine decision 0002 /+ * desktop decision 0007 - see desktop decision 0009 for the shape and its+ * exit (a native surface replaces the transport, not this seam).+ */+ const exportClipsRef = useRef(exportClips);+ exportClipsRef.current = exportClips;+ useEffect(() => {+ if (!playing || !loaded) return;+ let live = true;+ // Invalidate any paused-dwell fetch still in flight from before play.+ stillToken.current += 1;+ // Rate beats resolution while the picture moves; the paused dwell+ // re-fetches at the full 960 cap the moment the transport stops.+ const cap = 640;+ const wait = (ms: number) => new Promise((resolve) => window.setTimeout(resolve, ms));+ const visualLayers = (time: number) =>+ clipsAt(latest.current.project, time).filter(+ (clip) => clip.kind === "video" || clip.kind === "image",+ ).length;++ const run = async () => {+ while (live) {+ const time = latest.current.playhead;+ if (visualLayers(time) < 2) {+ setEngineStill(null);+ await wait(120);+ continue;+ }+ const view = latest.current.frame;+ const scale = Math.min(1, cap / Math.max(view.width, view.height));+ const width = Math.max(2, Math.round((view.width * scale) / 2) * 2);+ const height = Math.max(2, Math.round((view.height * scale) / 2) * 2);+ try {+ const bytes = await previewFrame({+ time,+ width,+ height,+ rateNum: session.rateNum,+ rateDen: session.rateDen,+ clips: exportClipsRef.current,+ });+ if (live) setEngineStill({ bytes, width, height });+ } catch {+ // A failed frame keeps the approximation on screen; back off so a+ // persistently failing source cannot spin the pool.+ if (live) setEngineStill(null);+ await wait(200);+ }+ }+ };+ void run();+ return () => {+ live = false;+ };+ }, [playing, loaded, session.rateNum, session.rateDen]);+const selectedClip =selectedClipIds.length === 1 ? findClip(project, selectedClipIds[0]) : null;const inspectorMedia = selectedClip