Open source CapCut alternative.
TypeScript
feat(site): a play with nothing to show stands still
David Lef51603c5ccab
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.mdindex 6c43c7c7..8c530455 100644--- a/docs/guides/cut/README.md+++ b/docs/guides/cut/README.md@@ -50,6 +50,8 @@ Holding on to a walk is what makes that cheap, and it matters most exactly whenThe other rule is about walks that stop. A walk reports the same ending whether it read the file's last frame or gave up on it, and a reader whose bytes never came, or whose decoder was closed under it, ends the same way. Believed, an ending a second into a long clip pins the picture on the frame before it and nothing asks for another, so the sound plays the rest of the cut over one still. So an ending short of the film is treated as a walk that failed, and another goes after the film it did not reach. Three things keep that from running away: a walk that gets no further than a previous one has found where the film really stops and is believed, a walk that died does not aim ahead again, since dying at that anchor has disproved the aim, and a source with two dead walks behind it waits a beat before sending the next. Once an ending is believed, moments past it go to the single-frame fetch — a finished walk cannot be crept along, and a reader that keeps asking it gets no frame and no repaint to ask again on, which is a drag position that never paints and a parked playhead that never goes quiet. A cloud project reads its media over the network in resident chunks, so a slow link puts a reader in these positions for real.+A play whose clip has decoded nothing at all has nothing to play, and carrying the clock through that spends a stretch of the cut nobody saw or heard — the playhead arrives somewhere further on when the file finally opens, and the sound plays over a picture that is not moving. So the clock stops where it is, the preview says it is waiting, and the play carries on from the same moment once a frame is there. A moment of it is a decoder a frame or two behind rather than a stall, so a stall has to hold before it counts; a gap in the timeline is empty on purpose and plays through.+Some machines cannot decode the footage as fast as it plays, and no reader can invent frames that the decoder has not produced. Left alone that play never recovers: the sound goes on, the picture falls further behind with every second, and a minute in it is half a minute stale. So a preview that has stayed behind its sound for a few seconds decodes smaller from then on, and the picture catches up at a lower resolution. It is a trade taken once per play and never taken back — every open decoder is abandoned when the size changes and each source pays a fresh keyframe seek, so undoing it would cost more than the pixels are worth. Two things are excluded from it. The opening seconds of a play, where every machine trails while nothing is decoded yet and files are being opened, and a reader whose walk is waiting on bytes: decoding smaller does nothing for a picture that is late because the link has not delivered it, and the preview would end up softer for the rest of the session having gained nothing. The floor is 360, below which there is no picture worth keeping in sync.Decoders live in a bounded pool, and leaving the pool is suspension before it is closure: an idle source gives up its decoder but keeps the parsed file and the canvases its frames land on, so the next visit wakes it for the cost of one keyframe seek. The canvases are the part that must not churn — a montage of short clips replayed for a few minutes, allocating a fresh set on every reopen, floods the GPU process until frames stop arriving at all. They also must not pile up, so the warm shelf of stood-down sources is capped by the memory it holds rather than by a count of them: a small stage keeps a montage's whole cast, a large one keeps a few, and what does not fit closes for real.diff --git a/site/src/cut/components/Preview.tsx b/site/src/cut/components/Preview.tsxindex 74e90bdf..60528b38 100644--- a/site/src/cut/components/Preview.tsx+++ b/site/src/cut/components/Preview.tsx@@ -1,7 +1,7 @@"use client";import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState, type RefObject } from "react";-import { Check, Copy, ZoomIn, ZoomOut } from "lucide-react";+import { Check, Copy, Loader2, ZoomIn, ZoomOut } from "lucide-react";import { usePlayback } from "@/cut/hooks/usePlayback";import { startDrag } from "@/cut/lib/drag";import {@@ -150,6 +150,20 @@ function pictureInPane(return visX >= Math.min(ONSCREEN_MIN_PX, stage.w) && visY >= Math.min(ONSCREEN_MIN_PX, stage.h);}+/** A play held at a standstill, waiting on the picture. The playhead is not+ * moving, so this says what the frozen frame is doing. */+function BufferingBadge() {+ const buffering = useEditor((st) => st.buffering);+ if (!buffering) return null;+ return (+ <div className="pointer-events-none absolute inset-0 grid place-items-center">+ <span className="rounded-full bg-black/55 p-3 text-white shadow-lg">+ <Loader2 className="size-6 animate-spin" />+ </span>+ </div>+ );+}+export function Preview() {const canvasRef = useRef<HTMLCanvasElement>(null);const wrapRef = useRef<HTMLDivElement>(null);@@ -659,6 +673,7 @@ export function Preview() {className="block size-full"/></StagePictureFx>+ <BufferingBadge /><StagePress.Provider value={stagePress}>{slices.map((slice) =>slice.kind === "elements" ? (diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.tsindex 9966f84b..672dbab8 100644--- a/site/src/cut/hooks/usePlayback.ts+++ b/site/src/cut/hooks/usePlayback.ts@@ -54,6 +54,11 @@ import { registerSourceSampler } from "@/cut/lib/previewCanvas";const WARM_HORIZON_S = 2.5;/** Inside this, an upcoming clip is walked as well as opened. */const WARM_STREAM_S = 0.75;+/** How long the master's picture has to be missing before a play is held at a+ * standstill. Short enough that a real stall is caught before the playhead has+ * gone anywhere, long enough that a decoder a frame behind at a join is not a+ * stutter and a cut in the sound. */+const STALL_HOLD_MS = 200;/*** Decoders alive at once.*@@ -158,6 +163,8 @@ class Engine {/** The playhead value the engine itself wrote last, so its own echo is* tellable from an outside move — a seek while playing. */private written: number | null = null;+ /** Wall time the master's picture went missing, 0 while it is there. */+ private stalledAt = 0;/** How many 180-steps the preview has traded away, for the engine's life. */private dropped = 0;private steppedThisPlay = false;@@ -629,6 +636,8 @@ class Engine {s.overlays.length === 0) {this.mixer.stop();+ this.stalledAt = 0;+ if (s.buffering) useEditor.setState({ buffering: false });this.comp.drawLayer(MISSING_FRAME, undefined, true, 1, 0);if (s.playing) {useEditor.setState({ playing: false });@@ -647,6 +656,27 @@ class Engine {// open can't freeze the playhead or carry playback past a stop mark.const masterFrame = master ? this.frameFor(master, t, playing, true) : MISSING_FRAME;const pendingMaster = masterFrame.kind === "pending";+ // A play whose master clip has decoded nothing has nothing to play: the+ // clock would carry the playhead across a stretch of the cut nobody saw or+ // heard, and land somewhere further on when the file finally opened. So it+ // stands still and says so, and the play carries on from where it stopped.+ // A moment of it is a decoder a frame or two behind rather than a stall,+ // and stopping the clock for that would stutter the picture and cut the+ // sound at every join, so a stall has to hold before it counts. A gap in+ // the timeline is empty on purpose and plays through.+ if (playing && master && pendingMaster) {+ this.stalledAt ||= performance.now();+ if (performance.now() - this.stalledAt >= STALL_HOLD_MS) {+ this.mixer.hold(t);+ this.writeHead(t);+ if (!s.buffering) useEditor.setState({ buffering: true });+ this.dirty = true;+ return;+ }+ } else {+ this.stalledAt = 0;+ if (s.buffering) useEditor.setState({ buffering: false });+ }const fadeGain = this.projectFadeGain(t, total);if (pendingMaster) {this.dirty = true;diff --git a/site/src/cut/lib/previewMixer.test.ts b/site/src/cut/lib/previewMixer.test.tsindex 8b2c8cf3..1d72e7b6 100644--- a/site/src/cut/lib/previewMixer.test.ts+++ b/site/src/cut/lib/previewMixer.test.ts@@ -282,7 +282,13 @@ function reset(file: { endEarlyAt?: number; trackEnd?: number } = {}): void {* it is drawing. Answers what was heard. */async function play(during?: (t: number, ctx: FakeContext) => void,- file: { endEarlyAt?: number; trackEnd?: number } = {}+ file: {+ endEarlyAt?: number;+ trackEnd?: number;+ /** Frames the cut has nothing to show on: the clock is held rather than+ * carried, which is what the engine does with a stalled picture. */+ holdWhile?: (t: number) => boolean;+ } = {}): Promise<Played[]> {reset(file);const mixer = new PreviewMixer();@@ -293,7 +299,8 @@ async function play(step();t = Math.min(mixer.now(), CLIP_S);during?.(t, (mixer as unknown as { ctx: FakeContext }).ctx);- mixer.update(t, [voice]);+ if (file.holdWhile?.(t)) mixer.hold(t);+ else mixer.update(t, [voice]);await settle();}} finally {@@ -406,6 +413,29 @@ describe("the preview's sound over a link that only just keeps up", () => {expect(walks.opened).toBe(2);});+ test("stands still while the cut has nothing to show", async () => {+ // A play whose picture has decoded nothing has nothing to play. Carrying+ // the clock through it spends a stretch of the cut nobody saw or heard and+ // lands somewhere further on when the file opens; the sound plays on over+ // a picture that is not moving. The clock stops instead, and the play+ // carries on from where it stopped.+ const held: number[] = [];+ const sound = await play(undefined, {+ holdWhile: (t) => {+ const stalled = t >= 20 && wall < 27;+ if (stalled) held.push(t);+ return stalled;+ },+ });+ // Seven seconds of wall went by and the playhead stayed where it was.+ expect(held.length).toBeGreaterThan(100);+ expect(Math.max(...held) - Math.min(...held)).toBeLessThan(0.3);+ // Nothing plays on over a picture standing still.+ expect(heard(sound, 22, 26)).toBe(0);+ // And the cut picks up where it stopped rather than further on.+ expect(heard(sound, 27, 40)).toBeGreaterThan(0.9);+ });+test("has the file open before the play begins", async () => {// The failure this is here for: the picture starts on the frame and the// sound comes in seconds after it, worst right after a seek. The sounddiff --git a/site/src/cut/lib/previewMixer.ts b/site/src/cut/lib/previewMixer.tsindex 416b4cc5..9f29f611 100644--- a/site/src/cut/lib/previewMixer.ts+++ b/site/src/cut/lib/previewMixer.ts@@ -170,6 +170,11 @@ export class PreviewMixer {* them touches gains alone. */private fxKey = "";private voices = new Map<string, LiveVoice>();+ /** Whether the clock is being kept still by `hold`, so entering a hold+ * silences the mix once rather than on every frame of it. */+ private holding = false;+ /** Where the clock is being kept while `holding`. */+ private heldAt = 0;/** Timeline time, context time and wall time of the same instant. Everything* scheduled is placed against this. */private anchor: { timeline: number; ctx: number; wall: number } | null = null;@@ -240,6 +245,8 @@ export class PreviewMixer {*/now(): number {if (!this.anchor) return 0;+ // A hold is the clock standing still on purpose; nothing about it is drift.+ if (this.holding) return this.heldAt;const wall = this.anchor.timeline + (performance.now() - this.anchor.wall) / 1000;// A browser holds an `AudioContext` suspended until the page has been// interacted with, and a suspended context's clock does not move. Reading@@ -365,11 +372,38 @@ export class PreviewMixer {this.anchor = { timeline: t, ctx: ctx.currentTime, wall: performance.now() };}+ /**+ * Keep the clock at `t` while the cut has nothing to show.+ *+ * The context's clock runs whatever happens to it, so the anchor moves under+ * it: the timeline stands still while real time goes on, and the play picks+ * up where it stopped. Sound stops with it — a mix playing on over a picture+ * that is not moving is the sound of a cut coming apart, and what is already+ * scheduled would land at the wrong moment when the picture came back.+ */+ hold(t: number): void {+ if (!this.ctx || !this.anchor) return;+ if (!this.holding) {+ this.holding = true;+ this.heldAt = t;+ for (const live of this.voices.values()) {+ this.stopWindows(live);+ this.reaim(live, Math.max(live.start, t));+ }+ }+ // Both clocks are pulled back to where the hold began, every frame of it:+ // the context's runs on, and the wall's is what carries a cut whose context+ // is not running.+ this.anchor.ctx = this.ctx.currentTime - (this.heldAt - this.anchor.timeline);+ this.anchor.wall = performance.now() - (this.heldAt - this.anchor.timeline) * 1000;+ }+/** Stop the clock and silence everything, keeping each voice's reader: the* playhead is parked over the same clips it was playing, and the next play* is usually from where this one stopped. */stop(): void {this.anchor = null;+ this.holding = false;for (const live of this.voices.values()) this.stopWindows(live);}@@ -460,6 +494,7 @@ export class PreviewMixer {*/update(t: number, voices: Voice[]): void {if (!this.anchor || !this.ctx) return;+ this.holding = false;// A context that stops running stops the sound and leaves the picture// playing on wall time — an output device taken away, a page the browser// put to sleep. Nothing else asks it back, since `start` is the only otherdiff --git a/site/src/cut/lib/store.ts b/site/src/cut/lib/store.tsindex e5969c7a..2c330985 100644--- a/site/src/cut/lib/store.ts+++ b/site/src/cut/lib/store.ts@@ -279,6 +279,10 @@ export interface EditorState {track: "pose" | "mask";} | null;playing: boolean;+ /** Set while a play is held at a standstill because the cut has nothing to+ * show yet — a file still opening, a link that has not delivered. The+ * playhead stops where it is and the preview says so. */+ buffering: boolean;/** While playing a scoped effect preview, the time playback auto-pauses at;* null otherwise. Manual seek/play/pause clears it. */previewStopAt: number | null;@@ -1438,6 +1442,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {selection: null,multiSelection: [],playing: false,+ buffering: false,previewStopAt: null,subtitles: emptySubtitles(),subtitleLane: 0,@@ -1486,6 +1491,7 @@ export const useEditor = create<EditorState>((baseSet, get, api) => {multiSelection: [],selectedKey: null,playing: false,+ buffering: false,previewStopAt: null,pxPerSec: 60,timelineH: TIMELINE_H_DEFAULT,