Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

feat(preview): stream the engine's true composite while stacked clips play

Jareer07c18c33e243
diff --git a/desktop/docs/decisions/0009-the-monitor-streams-the-true-frame.md b/desktop/docs/decisions/0009-the-monitor-streams-the-true-frame.md
new file mode 100644
index 00000000..842a69e8
--- /dev/null
+++ b/desktop/docs/decisions/0009-the-monitor-streams-the-true-frame.md
@@ -0,0 +1,50 @@
+# 0009 - The monitor streams the true frame where the approximation is wrong
+
+## Decision
+While the transport runs, the monitor pulls the engine's real composite -
+the exporter's own plan, pool and compositor via `preview_frame` - in a loop:
+request the frame at the transport's current position, present it, request
+the next the moment it lands. The loop engages only while **two or more
+visual layers** sit under the playhead; single-layer stretches keep the
+webview element's smooth playback, exactly as before. Streamed frames are
+capped at 640 px on the long edge; the paused dwell still re-fetches at 960
+the moment the transport stops.
+
+This is the presentation half that 0007 left open ("presenting pooled frames
+continuously against the transport clock"), built on the seam that already
+existed rather than a new one.
+
+## Why this shape
+- **Pull, not push.** The UI asks for one frame at the *current* position and
+ does not ask again until it arrives. Backpressure is structural: a slow
+ machine gets a lower rate, never a growing queue of stale frames, and the
+ host's pool mutex serialises requests in order - which is also what makes
+ consecutive requests warm roll-forwards instead of seeks.
+- **Only where the element is wrong.** The `<video>` element is exact for one
+ layer and *cannot composite* a stack. Streaming everywhere would trade the
+ common case's 60 fps smoothness for uniform choppiness; streaming only on
+ stacks buys truth precisely where the approximation lies.
+- **The IPC budget holds.** 640-wide RGBA is ~1 MB a frame as an ArrayBuffer
+ response; at the 15-30 fps the pool sustains for warm readers that is well
+ inside what Tauri's IPC moves locally, and it degrades by rate, not by
+ failure.
+
+## What this is not
+- **Not a vsync-locked presentation.** Frames display one fetch-latency
+ behind the transport and at whatever rate decoding sustains. It is honest
+ compositing, not final playback smoothness.
+- **Not the native surface.** Decision 0002's endgame - a GPU surface
+ positioned to the frame box, engine-presented - still stands. When it
+ arrives it replaces the *transport* of pixels (IPC → surface); the plan,
+ pool and compositor this loop exercises are exactly what it will present
+ from, which is why building this first is not throwaway.
+
+## What it costs
+- Streamed frames are visibly softer than the element (640 upscaled) and
+ arrive a beat late. Acceptable on stacks, which previously showed
+ something *wrong* rather than something soft.
+- A failed frame drops back to the approximation silently (logged host-side).
+
+## What would change our mind
+The native surface landing - at which point this loop is deleted, not
+adapted, the same clean demolition 0004 got.
diff --git a/desktop/src/App.tsx b/desktop/src/App.tsx
index 86cff743..10e132ec 100644
--- a/desktop/src/App.tsx
+++ b/desktop/src/App.tsx
@@ -1,5 +1,5 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
-import { invoke, isTauri } from "@tauri-apps/api/core";
+import { isTauri } from "@tauri-apps/api/core";
import { open as openDialog } from "@tauri-apps/plugin-dialog";
import { getCurrentWebview } from "@tauri-apps/api/webview";
import { getCurrentWindow } from "@tauri-apps/api/window";
@@ -45,7 +45,9 @@ import {
type TimelineMeta,
} from "./lib/editor";
import {
+ audioSetClips,
editorApply,
+ onAudioError,
editorClose,
editorOpen,
editorRedo,
@@ -388,6 +390,26 @@ function Editor({
return () => window.clearTimeout(timer);
}, [toast]);
+ // Playback failures surface as the toast instead of dying in the dev
+ // console - a mute app must never be a mystery. Identical messages within
+ // a burst (one per clip of a broken import, say) collapse into one.
+ useEffect(() => {
+ if (!isTauri()) return;
+ let unlisten: (() => void) | null = null;
+ let lastMessage = "";
+ let lastAt = 0;
+ void onAudioError((message) => {
+ const now = Date.now();
+ if (message === lastMessage && now - lastAt < 10_000) return;
+ lastMessage = message;
+ lastAt = now;
+ setToast({ id: now, message, failed: true });
+ }).then((stop) => {
+ unlisten = stop;
+ });
+ return () => unlisten?.();
+ }, []);
+
// Bring back every font the project names, once, when it opens. A font
// whose file has moved is marked rather than dropped.
useEffect(() => {
@@ -563,7 +585,7 @@ function Editor({
useEffect(() => {
if (!isTauri()) return;
const timer = window.setTimeout(() => {
- void invoke("audio_set_clips", { project: session.path, clips: audibleClips }).catch(
+ void audioSetClips(session.path, audibleClips).catch(
(cause: unknown) => console.error("WolfCut: could not update the mix", cause),
);
}, 150);
@@ -1204,9 +1226,12 @@ function Editor({
} | null>(null);
const stillToken = useRef(0);
useEffect(() => {
+ // While playing, the streaming loop below owns the still; clearing it
+ // here on every playhead tick would fight the stream frame by frame.
+ if (playing) return;
const token = ++stillToken.current;
setEngineStill(null);
- if (playing || !loaded || exportClips.length === 0) return;
+ if (!loaded || exportClips.length === 0) return;
const timer = window.setTimeout(() => {
// Preview resolution: the frame's aspect, capped for IPC weight. Even
@@ -1233,6 +1258,73 @@ function Editor({
return () => window.clearTimeout(timer);
}, [playing, loaded, playhead, exportClips, frame, session.rateNum, session.rateDen]);
+ /**
+ * Streaming truth: while playing, present the engine's real composite
+ * against the transport clock wherever the element preview is wrong.
+ *
+ * The DOM approximation is exact for a single visible layer and cannot
+ * composite a stack, so the loop engages only while two or more visual
+ * layers sit under the playhead - single-clip stretches keep the smooth
+ * element playback. Pull, not push: request the frame at the transport's
+ * current position, present it, request the next the moment it lands. The
+ * loop runs at whatever rate the machine sustains and can never queue
+ * stale frames; the pool's warm readers make consecutive requests rolls,
+ * not seeks. This is the presentation half of engine decision 0002 /
+ * desktop decision 0007 - see desktop decision 0009 for the shape and its
+ * exit (a native surface replaces the transport, not this seam).
+ */
+ const exportClipsRef = useRef(exportClips);
+ exportClipsRef.current = exportClips;
+ useEffect(() => {
+ if (!playing || !loaded) return;
+ let live = true;
+ // Invalidate any paused-dwell fetch still in flight from before play.
+ stillToken.current += 1;
+ // Rate beats resolution while the picture moves; the paused dwell
+ // re-fetches at the full 960 cap the moment the transport stops.
+ const cap = 640;
+ const wait = (ms: number) => new Promise((resolve) => window.setTimeout(resolve, ms));
+ const visualLayers = (time: number) =>
+ clipsAt(latest.current.project, time).filter(
+ (clip) => clip.kind === "video" || clip.kind === "image",
+ ).length;
+
+ const run = async () => {
+ while (live) {
+ const time = latest.current.playhead;
+ if (visualLayers(time) < 2) {
+ setEngineStill(null);
+ await wait(120);
+ continue;
+ }
+ const view = latest.current.frame;
+ const scale = Math.min(1, cap / Math.max(view.width, view.height));
+ const width = Math.max(2, Math.round((view.width * scale) / 2) * 2);
+ const height = Math.max(2, Math.round((view.height * scale) / 2) * 2);
+ try {
+ const bytes = await previewFrame({
+ time,
+ width,
+ height,
+ rateNum: session.rateNum,
+ rateDen: session.rateDen,
+ clips: exportClipsRef.current,
+ });
+ if (live) setEngineStill({ bytes, width, height });
+ } catch {
+ // A failed frame keeps the approximation on screen; back off so a
+ // persistently failing source cannot spin the pool.
+ if (live) setEngineStill(null);
+ await wait(200);
+ }
+ }
+ };
+ void run();
+ return () => {
+ live = false;
+ };
+ }, [playing, loaded, session.rateNum, session.rateDen]);
+
const selectedClip =
selectedClipIds.length === 1 ? findClip(project, selectedClipIds[0]) : null;
const inspectorMedia = selectedClip