Skip to content

Open source CapCut alternative.

TypeScript

fix(site): preview keeps its canvases instead of churning the GPU process

David Le9d2e67cafad8
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index 3760129c..e5053106 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -40,13 +40,15 @@ The client probes the engine's health endpoint (dedicated port first, dev server
The preview decodes the cut itself. Every clip on screen owns a decoder reading its file directly — the container is parsed in the page and frames come off WebCodecs — and a small ring of already-decoded frames sits beside it. Drawing a frame is a question answered from that ring, never a wait: playing walks each file forward and keeps the ring ahead of the playhead, scrubbing asks for one moment and the newest ask wins, and if the exact frame has not landed the nearest one already decoded goes up and is replaced when the real one arrives. A held frame is worth more than a stall.
-Two things follow from that. A cut is a read from a different ring rather than a handover between players, so crossing a join costs nothing and needs no clock to change hands; and the picture answers a moved playhead inside a frame, because the answer was already in memory. Sound is scheduled on one audio context, a window at a time, and its clock is what playback runs on — the picture follows the sound rather than the two chasing each other. When a browser holds that context suspended, as it does until the page has been interacted with, wall time carries the cut and the audio clock is picked up once it agrees.
+Decoders live in a bounded pool, and leaving the pool is suspension before it is closure: an idle source gives up its decoder but keeps the parsed file and the canvases its frames land on, so the next visit wakes it for the cost of one keyframe seek. The canvases are the part that must not churn — a montage of short clips replayed for a few minutes, allocating a fresh set on every reopen, floods the GPU process until frames stop arriving at all. They also must not pile up, so the warm shelf of stood-down sources is capped by the memory it holds rather than by a count of them: a small stage keeps a montage's whole cast, a large one keeps a few, and what does not fit closes for real.
+
+Two things follow from that. A cut is a read from a different ring rather than a handover between players, so crossing a join costs nothing and needs no clock to change hands; and the picture answers a moved playhead inside a frame, because the answer was already in memory. Sound is scheduled on one audio context, a window at a time, and its clock is what playback runs on — the picture follows the sound rather than the two chasing each other. When a browser holds that context suspended, as it does until the page has been interacted with, wall time carries the cut and the audio clock is picked up once it agrees — and when the clock jumps instead of agreeing, a context resumed late or an output device switched mid-play, every scheduled window restarts where the clock now stands rather than playing out the moment it was placed for.
What is on screen at a given instant is still a pure function of the document and a time, and it is still the same function an export asks. The preview reaches an instant by playing toward it and an export by jumping straight there; because both draw through one compositor from one plan, the file cannot quietly disagree with what was watched.
The playhead lives apart from the document. It moves sixty times a second and the document does not, so a moved playhead wakes the surfaces that draw it and nothing else — and a paused editor with nothing changing schedules no frames at all.
-Whether any of this is true of a given build is measured rather than assumed: `npm run eval:cut-perf` drives a real editor in a real browser against a fixture montage and reports how long the picture takes to answer a moved playhead and whether any frame crossing a cut was the wrong one. Its committed report is the baseline a change is diffed against.
+Whether any of this is true of a given build is measured rather than assumed: `npm run eval:cut-perf` drives a real editor in a real browser against fixture montages and reports how long the picture takes to answer a moved playhead, whether any frame crossing a cut was the wrong one, and whether a fast-cut montage replayed for minutes still holds the picture to the clock the way its first pass did. Its committed report is the baseline a change is diffed against.
## Color grading
diff --git a/evals/cut-perf.latest-report.json b/evals/cut-perf.latest-report.json
index 815e28d0..25e38bff 100644
--- a/evals/cut-perf.latest-report.json
+++ b/evals/cut-perf.latest-report.json
@@ -6,7 +6,12 @@
"jumpP50Ms": 20,
"jumpP95Ms": 32,
"boundaryDrops": 0,
- "longTaskMs": 16
+ "longTaskMs": 16,
+ "filmstripSlackS": 0.2,
+ "lagP95S": 0.05,
+ "lagMaxS": 0.25,
+ "decayRatio": 1.5,
+ "warmMb": 400
},
"fixture": {
"clips": 6,
@@ -19,10 +24,10 @@
"pass": true,
"notes": [],
"scrub": {
- "p50": 7.3,
- "p95": 15.9,
- "mean": 7.92,
- "max": 18.8,
+ "p50": 7.5,
+ "p95": 15.8,
+ "mean": 8.59,
+ "max": 21.5,
"n": 41
},
"longTasks": {
@@ -32,7 +37,7 @@
"max": 0,
"n": 0
},
- "liveSamples": 34
+ "liveSamples": 60
},
{
"name": "drag-across-a-cut",
@@ -40,10 +45,10 @@
"pass": true,
"notes": [],
"scrub": {
- "p50": 7.1,
+ "p50": 7.5,
"p95": 16.1,
- "mean": 8.45,
- "max": 20.6,
+ "mean": 8.8,
+ "max": 19.6,
"n": 41
},
"longTasks": {
@@ -53,7 +58,7 @@
"max": 0,
"n": 0
},
- "liveSamples": 28
+ "liveSamples": 52
},
{
"name": "drag-across-a-transition",
@@ -61,9 +66,9 @@
"pass": true,
"notes": [],
"scrub": {
- "p50": 7.3,
- "p95": 15.7,
- "mean": 8.09,
+ "p50": 7.5,
+ "p95": 15.8,
+ "mean": 8.59,
"max": 22.9,
"n": 41
},
@@ -74,7 +79,7 @@
"max": 0,
"n": 0
},
- "liveSamples": 28
+ "liveSamples": 63
},
{
"name": "jump-across-the-cut",
@@ -82,10 +87,10 @@
"pass": true,
"notes": [],
"scrub": {
- "p50": 15.7,
- "p95": 23.9,
- "mean": 15.29,
- "max": 23.9,
+ "p50": 15.9,
+ "p95": 24,
+ "mean": 15.6,
+ "max": 25.6,
"n": 41
},
"longTasks": {
@@ -95,7 +100,7 @@
"max": 0,
"n": 0
},
- "liveSamples": 60
+ "liveSamples": 88
},
{
"name": "play-hard-cuts",
@@ -103,7 +108,7 @@
"pass": true,
"notes": [],
"drops": 0,
- "presented": 2077,
+ "presented": 2076,
"boundaryDrops": 0,
"longTasks": {
"p50": 0,
@@ -120,7 +125,7 @@
"pass": true,
"notes": [],
"drops": 0,
- "presented": 2086,
+ "presented": 2076,
"boundaryDrops": 0,
"longTasks": {
"p50": 0,
@@ -131,12 +136,98 @@
},
"liveSamples": 60
},
+ {
+ "name": "play-fast-cuts",
+ "bucket": "playback",
+ "pass": true,
+ "notes": [],
+ "drops": 0,
+ "presented": 1619,
+ "lag": {
+ "p50": 0.02,
+ "p95": 0.03,
+ "mean": 0.02,
+ "max": 0.03,
+ "n": 1619
+ },
+ "decay": {
+ "first": 0,
+ "last": 0
+ },
+ "longTasks": {
+ "p50": 0,
+ "p95": 0,
+ "mean": 0,
+ "max": 0,
+ "n": 0
+ },
+ "liveSamples": 120,
+ "liveSources": 12,
+ "keptSources": 20,
+ "warmMb": 354
+ },
+ {
+ "name": "play-fast-cuts-long",
+ "bucket": "playback",
+ "pass": true,
+ "notes": [],
+ "drops": 0,
+ "presented": 6481,
+ "lag": {
+ "p50": 0.02,
+ "p95": 0.03,
+ "mean": 0.02,
+ "max": 0.03,
+ "n": 6481
+ },
+ "decay": {
+ "first": 0,
+ "last": 0
+ },
+ "longTasks": {
+ "p50": 0,
+ "p95": 0,
+ "mean": 0,
+ "max": 0,
+ "n": 0
+ },
+ "liveSamples": 120,
+ "liveSources": 12,
+ "keptSources": 20,
+ "warmMb": 354
+ },
{
"name": "idle-costs-nothing",
"bucket": "playback",
"pass": true,
"notes": [],
"idleTicks": 0
+ },
+ {
+ "name": "filmstrip-under-the-playhead",
+ "bucket": "filmstrip",
+ "pass": true,
+ "notes": [],
+ "tileErr": {
+ "p50": 0.04,
+ "p95": 0.07,
+ "mean": 0.03,
+ "max": 0.13,
+ "n": 35
+ }
+ },
+ {
+ "name": "filmstrip-at-scene-cuts",
+ "bucket": "filmstrip",
+ "pass": true,
+ "notes": [],
+ "tileErr": {
+ "p50": 0.03,
+ "p95": 0.08,
+ "mean": 0.04,
+ "max": 0.11,
+ "n": 32
+ }
}
]
}
diff --git a/site/scripts/eval-cut-perf.ts b/site/scripts/eval-cut-perf.ts
index 209f1649..79cbccef 100644
--- a/site/scripts/eval-cut-perf.ts
+++ b/site/scripts/eval-cut-perf.ts
@@ -21,6 +21,12 @@
* at; the tile is compared against this same browser's
* decode of the file, which cancels colorspace drift between
* the encoder and the canvas.
+ * holding sync — a montage of sub-second clips sliced from one long file
+ * plays through repeatedly, and the trace is checked for the
+ * picture falling behind the clock it shares with the sound
+ * and for late frames rising from the first pass to the
+ * last. This is the session-decay case: a preview that is
+ * smooth for a minute and choppy after five fails here.
*
* Fixtures build deterministically into dist/cut-perf/ (gitignored) from the
* bundled stock clips, so the montage's cut times are known exactly.
@@ -86,6 +92,19 @@ const GATE = {
// a tile's edge lands on the neighboring scene whenever a cut sits inside
// the tile — the standard is the midpoint, so that drift is a failure.
filmstripSlackS: 0.2,
+ // Sound is scheduled against the clock the picture is drawn from, so how far
+ // the picture sits behind the source moment it was asked for is how far it
+ // sits behind the sound. A frame either side of the ask is the picture
+ // keeping up; a quarter second is a viewer noticing.
+ lagP95S: 0.05,
+ lagMaxS: 0.25,
+ // A long play must end the way it began: a session that decays is the bug
+ // this case exists to catch.
+ decayRatio: 1.5,
+ // Canvas backing the pool's warm shelf is holding. The shelf trades decoder
+ // churn for memory, and this is the side of that trade a change could
+ // quietly blow.
+ warmMb: 400,
};
// ── Types shared with perfTrace.ts ──────────────────────────────────────────
@@ -93,6 +112,7 @@ interface PresentRecord {
t: number;
at: number;
srcTs: number | null;
+ wantSrc: number | null;
clipId: string | null;
exact: boolean;
stale: boolean;
@@ -108,6 +128,9 @@ interface Trace {
longTasks: { at: number; ms: number }[];
ticks: number;
liveSamples: number[];
+ liveSources: number[];
+ keptSources: number[];
+ warmMb: number[];
startedAt: number;
}
@@ -214,6 +237,88 @@ async function buildRamp(): Promise<void> {
]);
}
+// ── The fast-cut fixture ────────────────────────────────────────────────────
+//
+// The shape people actually edit in: one long file, sliced into a montage of
+// sub-second clips in shuffled source order, with sound. Every clip is its own
+// decode mapping, so a play crosses a cut about once a second and each cut
+// lands on a source moment the last clip's decoder was nowhere near.
+
+/** The fast-cut source: long enough that its slices are spread across the file. */
+const MONTAGE_SRC_S = 30;
+/** One slice, and how many the montage holds. */
+const SHOT_S = 0.9;
+const SHOTS = 20;
+
+/** A long clip with sound and a sparse keyframe cadence, which is what a phone
+ * or a screen recording gives you. */
+async function buildMontageSource(): Promise<void> {
+ const dst = path.join(OUT, "montage-src.mp4");
+ if (existsSync(dst)) return;
+ const src = path.join(STOCK, "animal-dog-sprint.mp4");
+ await run("ffmpeg", [
+ "-y", "-loglevel", "error",
+ "-stream_loop", "-1", "-i", src,
+ "-f", "lavfi", "-i", "sine=frequency=440:sample_rate=48000",
+ "-t", String(MONTAGE_SRC_S),
+ "-map", "0:v:0", "-map", "1:a:0",
+ "-c:v", "libx264", "-preset", "veryfast", "-g", "60", "-keyint_min", "60",
+ "-r", "30", "-vf", "scale=1280:-2", "-pix_fmt", "yuv420p",
+ "-c:a", "aac", "-b:a", "128k",
+ "-shortest",
+ dst,
+ ]);
+}
+
+/** Where each shot reads from, spread over the source and shuffled by a fixed
+ * step so no two neighbours share a decoder or a keyframe. */
+const SHOT_INS = Array.from({ length: SHOTS }, (_, i) => {
+ const slot = (i * 7) % SHOTS;
+ return +((slot / SHOTS) * (MONTAGE_SRC_S - SHOT_S)).toFixed(3);
+});
+
+/** Seed the montage: one asset, many short clips, each its own mapping. */
+const seedMontage = async (page: Page): Promise<Fixture> =>
+ page.evaluate(
+ async ({ ins, shot, srcDuration }) => {
+ const dev = (window as unknown as {
+ __cutDev: { useEditor: { getState(): Record<string, unknown>; setState(p: unknown): void } };
+ }).__cutDev;
+ dev.useEditor.setState({
+ assets: [
+ {
+ id: "montage",
+ name: "montage-src.mp4",
+ fileName: "montage-src.mp4",
+ type: "video",
+ url: "/__cutperf/montage-src.mp4",
+ duration: srcDuration,
+ width: 1280,
+ height: 720,
+ },
+ ],
+ clips: ins.map((inPoint, i) => ({
+ id: `m-${i}`,
+ assetId: "montage",
+ track: 0,
+ start: +(i * shot).toFixed(3),
+ in: inPoint,
+ out: +(inPoint + shot).toFixed(3),
+ })),
+ audioClips: [],
+ overlays: [],
+ transitions: [],
+ loaded: true,
+ });
+ await new Promise((r) => setTimeout(r, 200));
+ return {
+ cuts: ins.slice(1).map((_, i) => +((i + 1) * shot).toFixed(3)),
+ duration: +(ins.length * shot).toFixed(3),
+ };
+ },
+ { ins: SHOT_INS, shot: SHOT_S, srcDuration: MONTAGE_SRC_S }
+ );
+
/** Where the scene-cut fixture changes scene: every three seconds. */
const CUT_SEG_S = 3;
const FIXTURE_CUTS = Array.from({ length: RAMP_S / CUT_SEG_S - 1 }, (_, i) => (i + 1) * CUT_SEG_S);
@@ -367,13 +472,15 @@ async function rewind(page: Page): Promise<void> {
() =>
(window as unknown as { __cutDev: { playheadAt(): number } }).__cutDev.playheadAt() < 0.01,
undefined,
- { timeout: 5000 }
+ { timeout: 15_000, polling: 100 }
);
await page.waitForTimeout(150);
}
-/** Wait for the picture to catch up to the last thing asked for. */
-const settle = (page: Page, ms = 2500) =>
+/** Wait for the perf hook, then a beat for the picture to catch up. Polled on
+ * an interval: a fresh seed kicks off filmstrip captures for every clip, and
+ * rAF-polled waits starve under that burst. */
+const settle = (page: Page, ms = 15_000) =>
page
.waitForFunction(
() => {
@@ -383,7 +490,7 @@ const settle = (page: Page, ms = 2500) =>
return !!perf;
},
undefined,
- { timeout: ms }
+ { timeout: ms, polling: 100 }
)
.then(() => page.waitForTimeout(120));
@@ -403,6 +510,14 @@ interface CaseResult {
idleTicks?: number;
longTasks?: Agg;
liveSamples?: number;
+ liveSources?: number;
+ /** Sources the pool kept, and the canvas backing on its warm shelf. */
+ keptSources?: number;
+ warmMb?: number;
+ /** Seconds the picture sat behind the moment it was asked for. */
+ lag?: Agg;
+ /** Late-frame share over the first and last third of a long play. */
+ decay?: { first: number; last: number };
/** Seconds between a strip tile's own moment and the moment its picture
* was captured at. */
tileErr?: Agg;
@@ -551,6 +666,156 @@ const playbackCase = (name: string, transitions: boolean): EvalCase => ({
},
});
+/**
+ * A montage played straight through, `passes` times over, with the sound on.
+ *
+ * What it measures is drift: how far the picture sits behind the moment the
+ * clock asked for, and whether the last pass is worse than the first. A single
+ * pass answers "does a fast cut play"; several answer "does it still play after
+ * a while", which is the complaint this case was written for.
+ */
+const montageCase = (name: string, passes: number): EvalCase => ({
+ name,
+ bucket: "playback",
+ transitions: false,
+ seed: seedMontage,
+ run: async (page, fx) => {
+ await rewind(page);
+ // One pass before the trace, so what follows measures playing rather than
+ // opening twenty decoders for the first time.
+ await setPlaying(page, true);
+ await page.waitForTimeout(WARM_PLAY_S * 1000);
+ await rewind(page);
+ await settle(page);
+ await startTrace(page);
+ for (let i = 0; i < passes; i++) {
+ await seek(page, 0);
+ await setPlaying(page, true);
+ await page.waitForTimeout((fx.duration + 0.6) * 1000);
+ if (has("--detail")) {
+ const state = await page.evaluate(() => {
+ const w = window as unknown as {
+ __cutDevEngine?: { poolState(): Record<string, unknown>[] };
+ __cutDev: { useEditor: { getState(): { clips: unknown[]; playing: boolean } } };
+ };
+ const s = w.__cutDev.useEditor.getState();
+ return { pool: w.__cutDevEngine?.poolState() ?? [], clips: s.clips.length, playing: s.playing };
+ });
+ const res = await page.evaluate(
+ () => (window as unknown as { __resourceStats?: unknown }).__resourceStats ?? null
+ );
+ console.log(
+ `[pool] pass ${i + 1}: clips=${state.clips} playing=${state.playing} res=${JSON.stringify(res)}`
+ );
+ for (const s of state.pool) console.log(`[pool] ${JSON.stringify(s)}`);
+ const log = (await page.evaluate(() => {
+ const w = window as unknown as { __cutEngineLog?: { at: number; msg: string }[] };
+ const out = w.__cutEngineLog ?? [];
+ w.__cutEngineLog = [];
+ return out;
+ })) as { at: number; msg: string }[];
+ for (const e of log) console.log(`[elog] ${(e.at / 1000).toFixed(1)}s ${e.msg}`);
+ }
+ await setPlaying(page, false);
+ }
+ const trace = await stopTrace(page);
+ const notes: string[] = [];
+ if (!trace) return { name, bucket: "playback", pass: false, notes: ["no trace"] };
+
+ // The moments a clip's decoder is being opened for the first time in a pass
+ // belong to the open, so the measure starts a beat inside every clip and
+ // ends a beat before the cut.
+ const played = trace.presents.filter((p) => {
+ const into = p.t % SHOT_S;
+ return p.t > 0.4 && p.t < fx.duration - 0.3 && into > 0.15 && into < SHOT_S - 0.05;
+ });
+ if (played.length < 60) {
+ return { name, bucket: "playback", pass: false, notes: [`only ${played.length} frames`] };
+ }
+ const lag = agg(
+ played
+ .filter((p) => p.srcTs !== null && p.wantSrc !== null)
+ .map((p) => Math.max(0, +(p.wantSrc! - p.srcTs!).toFixed(4)))
+ );
+ const late = played.filter((p) => !p.exact || p.stale);
+ const share = (xs: PresentRecord[]) =>
+ xs.length ? +(xs.filter((p) => !p.exact || p.stale).length / xs.length).toFixed(3) : 0;
+ const third = Math.floor(played.length / 3);
+ const decay = { first: share(played.slice(0, third)), last: share(played.slice(-third)) };
+ if (has("--detail")) {
+ const span = (trace.presents.at(-1)!.at - trace.presents[0].at) / 1000;
+ console.log(
+ `[detail] presents=${trace.presents.length} over ${span.toFixed(1)}s (${(
+ trace.presents.length / span
+ ).toFixed(1)}/s) stale=${played.filter((p) => p.stale).length} notExact=${
+ played.filter((p) => !p.exact && !p.stale).length
+ } sources p50=${agg(trace.liveSources).p50} max=${agg(trace.liveSources).max} held p50=${
+ agg(trace.liveSamples).p50
+ } max=${agg(trace.liveSamples).max}`
+ );
+ for (let i = 0; i < passes; i++) {
+ const from = (i * played.length) / passes;
+ const slice = played.slice(Math.floor(from), Math.floor(from + played.length / passes));
+ const gaps = slice.slice(1).map((p, j) => p.at - slice[j].at);
+ console.log(
+ `[detail] pass ${i + 1}: late=${share(slice)} lagP95=${
+ agg(slice.filter((p) => p.srcTs !== null).map((p) => Math.max(0, p.wantSrc! - p.srcTs!)))
+ .p95
+ }s frameGapP95=${agg(gaps).p95}ms n=${slice.length}`
+ );
+ // Which clips went dark, and for how long at a stretch.
+ const byClip = new Map<string, { n: number; stale: number; worst: number }>();
+ let streak = 0;
+ for (let j = 0; j < slice.length; j++) {
+ const p = slice[j];
+ const key = p.clipId ?? "none";
+ const row = byClip.get(key) ?? { n: 0, stale: 0, worst: 0 };
+ row.n++;
+ if (p.stale) {
+ row.stale++;
+ streak = streak + (j > 0 ? p.at - slice[j - 1].at : 0);
+ row.worst = Math.max(row.worst, streak);
+ } else {
+ streak = 0;
+ }
+ byClip.set(key, row);
+ }
+ const dark = [...byClip.entries()]
+ .filter(([, r]) => r.stale > 0)
+ .sort((a, b) => b[1].stale / b[1].n - a[1].stale / a[1].n)
+ .map(([id, r]) => `${id}:${Math.round((100 * r.stale) / r.n)}%/${Math.round(r.worst)}ms`);
+ if (dark.length) console.log(`[detail] dark ${dark.join(" ")}`);
+ }
+ }
+
+ if (lag.p95 > GATE.lagP95S) notes.push(`picture ${lag.p95}s behind the clock at p95`);
+ if (lag.max > GATE.lagMaxS) notes.push(`picture ${lag.max}s behind the clock at worst`);
+ if (passes > 1 && decay.last > Math.max(0.02, decay.first * GATE.decayRatio)) {
+ notes.push(`late frames rose ${decay.first} → ${decay.last} over the play`);
+ }
+ const warmMb = Math.max(0, ...trace.warmMb);
+ if (warmMb > GATE.warmMb) notes.push(`warm shelf holding ${warmMb}MB of canvas`);
+ // Long tasks are reported, not gated: this case judges sync, decay and
+ // memory, and the stray dev-build hiccup that leaves every frame exact is
+ // not a fail.
+ return {
+ name,
+ bucket: "playback",
+ pass: notes.length === 0,
+ notes,
+ drops: late.length,
+ presented: played.length,
+ lag,
+ decay,
+ longTasks: agg(trace.longTasks.map((l) => l.ms)),
+ liveSamples: Math.max(0, ...trace.liveSamples),
+ liveSources: Math.max(0, ...trace.liveSources),
+ keptSources: Math.max(0, ...trace.keptSources),
+ warmMb,
+ };
+ },
+});
+
/** A paused editor with nothing changing must schedule no frames at all. */
const idleCase: EvalCase = {
name: "idle-costs-nothing",
@@ -893,6 +1158,8 @@ const CASES: EvalCase[] = [
}),
playbackCase("play-hard-cuts", false),
playbackCase("play-with-transitions", true),
+ montageCase("play-fast-cuts", 1),
+ montageCase("play-fast-cuts-long", 4),
idleCase,
filmstripCase,
filmstripCutsCase,
@@ -944,6 +1211,78 @@ async function launch(): Promise<{ browser: Browser; page: Page }> {
}),
});
});
+ // With `--detail`, count what the page opens against what it lets go of:
+ // a picture that locks up mid-session is usually a resource nobody closed.
+ if (has("--detail")) {
+ await context.addInitScript(() => {
+ const w = window as unknown as Record<string, unknown>;
+ const stats = { made: 0, closed: 0, errors: 0, lastError: "", canvases: 0, pixels: 0 };
+ w.__resourceStats = stats;
+ const RealDecoder = (w as { VideoDecoder?: unknown }).VideoDecoder as
+ | (new (init: { error?: (e: unknown) => void }) => { close(): void })
+ | undefined;
+ if (RealDecoder) {
+ w.VideoDecoder = class extends RealDecoder {
+ constructor(init: { error?: (e: unknown) => void }) {
+ super({
+ ...init,
+ error: (e: unknown) => {
+ stats.errors++;
+ stats.lastError = String(e).slice(0, 200);
+ init.error?.(e);
+ },
+ });
+ stats.made++;
+ }
+ close() {
+ stats.closed++;
+ return super.close();
+ }
+ };
+ Object.assign(w.VideoDecoder as object, RealDecoder);
+ }
+ const f = Object.assign(stats, { fetches: 0, fetchLive: 0, fetchWorst: 0 });
+ const realFetch = window.fetch.bind(window);
+ window.fetch = (async (...fetchArgs: Parameters<typeof fetch>) => {
+ f.fetches++;
+ f.fetchLive++;
+ f.fetchWorst = Math.max(f.fetchWorst, f.fetchLive);
+ try {
+ return await realFetch(...fetchArgs);
+ } finally {
+ f.fetchLive--;
+ }
+ }) as typeof fetch;
+ const create = document.createElement.bind(document);
+ document.createElement = ((tag: string, ...rest: unknown[]) => {
+ const el = create(tag as "canvas", ...(rest as []));
+ if (tag === "canvas") {
+ stats.canvases++;
+ queueMicrotask(() => {
+ const c = el as HTMLCanvasElement;
+ stats.pixels += c.width * c.height;
+ });
+ }
+ return el;
+ }) as typeof document.createElement;
+ });
+ }
+ // The welcome sequence opens over the editor for an account that has yet to
+ // run it, covers the stage and holds playback. Answering the account read as
+ // finished keeps the eval on the editor, whatever the dev account's row says.
+ await context.route("**/api/account/onboarding*", async (route) => {
+ await route.fulfill({
+ status: 200,
+ contentType: "application/json",
+ body: JSON.stringify({
+ version: 99,
+ completedAt: new Date(0).toISOString(),
+ skipped: false,
+ referralSources: [],
+ referralOther: null,
+ }),
+ });
+ });
// Serve the fixture clips from disk, so no upload or network variance sits
// between the eval and the decoder. Range requests get real 206 slices: a
// <video> element treats a source without them as unseekable and clamps
@@ -1034,12 +1373,15 @@ const detailOf = (r: CaseResult) =>
? `errP50=${r.tileErr.p50}s errMax=${r.tileErr.max}s n=${r.tileErr.n}`
: r.idleTicks !== undefined
? `ticks=${r.idleTicks}`
- : `late=${r.drops}/${r.presented} atCut=${r.boundaryDrops}`;
+ : r.lag
+ ? `late=${r.drops}/${r.presented} lagP50=${r.lag.p50}s lagP95=${r.lag.p95}s lagMax=${r.lag.max}s decoders=${r.liveSources} kept=${r.keptSources} warm=${r.warmMb}MB decay=${r.decay?.first}→${r.decay?.last}`
+ : `late=${r.drops}/${r.presented} atCut=${r.boundaryDrops}`;
async function main(): Promise<void> {
const files = await buildFixtures();
await buildRamp();
await buildCutsClip();
+ await buildMontageSource();
console.log(`[fixtures] ${files.length} clips + ramp + cuts in ${OUT}`);
const cases = CASES.filter(
diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.ts
index 52b0483a..519bba84 100644
--- a/site/src/cut/hooks/usePlayback.ts
+++ b/site/src/cut/hooks/usePlayback.ts
@@ -16,7 +16,13 @@ import { FrameCompositor, MISSING_FRAME, PENDING_FRAME, type Frame } from "@/cut
import { duckGainAt, overlayPlan, trackZeroPlan } from "@/cut/lib/framePlan";
import { type ClipFrameSource, FrameSourcePool, mappingKey } from "@/cut/lib/frameSource";
import { PreviewMixer, type Voice } from "@/cut/lib/previewMixer";
-import { markLiveSamples, markPresent, markTick, tracing } from "@/cut/lib/perfTrace";
+import {
+ markLiveSamples,
+ markLiveSources,
+ markPresent,
+ markTick,
+ tracing,
+} from "@/cut/lib/perfTrace";
import { registerSourceSampler } from "@/cut/lib/previewCanvas";
/**
@@ -53,7 +59,21 @@ const sourceTimeOf = (clip: VideoClip, t: number) =>
const keyOf = (clip: VideoClip, asset: MediaAsset) =>
mappingKey(asset.id, clipSpeed(clip), clip.in, clip.start);
+let engineSerial = 0;
+
+/** Dev-only: engine lifecycle events, for the perf eval's timeline. Bounded so
+ * an ordinary dev session never accumulates it. */
+export function engineLog(msg: string): void {
+ if (process.env.NODE_ENV === "production" || typeof window === "undefined") return;
+ const w = window as unknown as { __cutEngineLog?: { at: number; msg: string }[] };
+ const log = (w.__cutEngineLog ??= []);
+ log.push({ at: Math.round(performance.now()), msg });
+ if (log.length > 500) log.splice(0, log.length - 500);
+}
+
class Engine {
+ /** Which engine this is, for the perf eval's pool dump. */
+ readonly serial = ++engineSerial;
private comp: FrameCompositor;
private pool = new FrameSourcePool(DECODER_BUDGET, () => this.wake());
private mixer = new PreviewMixer();
@@ -88,10 +108,12 @@ class Engine {
// document edit — so the attributes themselves are the wake signal.
this.sizeWatch = new MutationObserver(() => this.wake());
this.sizeWatch.observe(canvas, { attributes: true, attributeFilter: ["width", "height"] });
+ engineLog(`engine ${this.serial} constructed`);
this.wake();
}
dispose() {
+ engineLog(`engine ${this.serial} disposed`);
this.disposed = true;
if (this.raf) cancelAnimationFrame(this.raf);
this.unsubscribe();
@@ -142,6 +164,14 @@ class Engine {
this.raf = requestAnimationFrame(this.tick);
}
+ /** Every open decoder and what it is doing, for the perf eval's pool dump. */
+ poolState(): Record<string, unknown>[] {
+ return [
+ { serial: this.serial, disposed: this.disposed, raf: this.raf, dirty: this.dirty },
+ ...this.pool.debugState(),
+ ];
+ }
+
/** The clip's raw, ungraded decoder frame for analysis (the color panel's
* Auto), or null when nothing has been decoded for it. */
sourceFor(clipId: string): CanvasImageSource | null {
@@ -367,10 +397,20 @@ class Engine {
}
}
+ /** When the last tick ran, for spotting a starved loop in the dev log. */
+ private lastTickAt = 0;
+
private tick() {
this.raf = 0;
if (this.disposed) return;
markTick();
+ if (process.env.NODE_ENV !== "production") {
+ const now = performance.now();
+ if (this.lastTickAt && now - this.lastTickAt > 500) {
+ engineLog(`engine ${this.serial} tick gap ${Math.round(now - this.lastTickAt)}ms`);
+ }
+ this.lastTickAt = now;
+ }
const playing = useEditor.getState().playing;
// Playing redraws every frame; paused, only what changed since the last
// paint. Either way the loop stops as soon as there is nothing to do.
@@ -486,6 +526,7 @@ class Engine {
if (tracing()) {
let srcTs: number | null = null;
+ let wantSrc: number | null = null;
let exact = true;
if (master) {
const src = this.pool.get(
@@ -494,17 +535,20 @@ class Engine {
this.decodeHeight()
);
const st = sourceTimeOf(master.clip, t);
+ wantSrc = st;
srcTs = src.frameAt(st, master.clip.in, master.clip.out)?.timestamp ?? null;
exact = src.hasExact(st);
}
markPresent({
t,
srcTs,
+ wantSrc,
clipId: master?.clip.id ?? null,
exact,
stale: masterFrame.kind !== "ready",
});
markLiveSamples(this.pool.held);
+ markLiveSources(this.pool.active, this.pool.size, this.pool.warmPixels);
}
}
}
diff --git a/site/src/cut/lib/frameSource.test.ts b/site/src/cut/lib/frameSource.test.ts
index 94f2d170..e3406b54 100644
--- a/site/src/cut/lib/frameSource.test.ts
+++ b/site/src/cut/lib/frameSource.test.ts
@@ -5,13 +5,15 @@ import type { MediaAsset } from "./types";
/** A decoded frame at 30fps, as the sink would hand it over. */
const f = (timestamp: number, duration = 1 / 30): Timed => ({ timestamp, duration });
-const asset = (id: string): MediaAsset => ({
+const asset = (id: string, width = 1280, height = 720): MediaAsset => ({
id,
name: `${id}.mp4`,
fileName: `${id}.mp4`,
type: "video",
url: `https://example.test/${id}.mp4`,
duration: 10,
+ width,
+ height,
});
describe("FrameRing", () => {
@@ -166,7 +168,7 @@ describe("FrameSourcePool", () => {
expect(pool.size).toBe(1);
});
- test("evicts the least recently asked-for, never this frame's", () => {
+ test("suspends the least recently asked-for, never this frame's", () => {
const pool = new FrameSourcePool(2);
pool.beginFrame();
const old = pool.get("old", asset("a"), 480);
@@ -177,11 +179,55 @@ describe("FrameSourcePool", () => {
const live = pool.get("live", asset("c"), 480);
expect(pool.size).toBe(3);
pool.evict();
- // Down to the budget, and the one this frame is drawn from survives.
- expect(pool.size).toBe(2);
+ // Past the budget the idle ones stand down, and standing down keeps the
+ // source — its parsed file and its sink's canvases — so the next visit
+ // finds the same instance instead of paying for a fresh open.
+ expect(pool.size).toBe(3);
expect(pool.get("live", asset("c"), 480)).toBe(live);
expect(pool.get("mid", asset("b"), 480)).toBe(mid);
- expect(pool.get("old", asset("a"), 480)).not.toBe(old);
+ expect(pool.get("old", asset("a"), 480)).toBe(old);
+ });
+
+ test("standing down costs a decoder, not a source", () => {
+ const pool = new FrameSourcePool(1);
+ pool.beginFrame();
+ const a = pool.get("a", asset("a"), 360);
+ pool.beginFrame();
+ pool.get("b", asset("b"), 360);
+ for (let i = 0; i < 200; i++) pool.beginFrame();
+ // Both are idle and the budget is one, so the older gives up its decoder.
+ pool.evict();
+ expect(pool.size).toBe(2);
+ expect(pool.active).toBe(1);
+ // Suspension is a state: evicting again changes nothing, and reading the
+ // source is what wakes it.
+ pool.evict();
+ expect(pool.active).toBe(1);
+ a.want(0, false);
+ expect(pool.active).toBe(2);
+ });
+
+ test("closes what the canvas budget cannot keep", () => {
+ // Big frames: a handful of sinks is the whole budget, so a montage of them
+ // is bounded by memory rather than by a count.
+ const pool = new FrameSourcePool(2);
+ for (let i = 0; i < 24; i++) {
+ pool.beginFrame();
+ pool.get(`s${i}`, asset(`x${i}`, 3840, 2160), 2160);
+ }
+ for (let i = 0; i < 200; i++) pool.beginFrame();
+ pool.evict();
+ expect(pool.size).toBeLessThan(6);
+ // Small frames of the same count fit, and all of them stay.
+ const small = new FrameSourcePool(2);
+ for (let i = 0; i < 24; i++) {
+ small.beginFrame();
+ small.get(`s${i}`, asset(`x${i}`, 640, 360), 360);
+ }
+ for (let i = 0; i < 200; i++) small.beginFrame();
+ small.evict();
+ expect(small.size).toBe(24);
+ expect(small.active).toBe(2);
});
test("holds a source through the grace, so a busy cut never thrashes", () => {
diff --git a/site/src/cut/lib/frameSource.ts b/site/src/cut/lib/frameSource.ts
index 95c9f166..fc73d1f7 100644
--- a/site/src/cut/lib/frameSource.ts
+++ b/site/src/cut/lib/frameSource.ts
@@ -28,6 +28,16 @@ import { frameSink, keyframeTimeAt, openMedia, videoTrackOf, type FrameCanvasSin
import { createRasterCanvas } from "./raster";
import type { MediaAsset } from "./types";
+/** Dev-only: pool lifecycle events, into the same log the engine writes.
+ * Bounded so an ordinary dev session never accumulates it. */
+function poolLog(msg: string): void {
+ if (process.env.NODE_ENV === "production" || typeof window === "undefined") return;
+ const w = window as unknown as { __cutEngineLog?: { at: number; msg: string }[] };
+ const log = (w.__cutEngineLog ??= []);
+ log.push({ at: Math.round(performance.now()), msg });
+ if (log.length > 500) log.splice(0, log.length - 500);
+}
+
/**
* Decoded frames a source keeps around where it is being read.
*
@@ -73,10 +83,36 @@ const BACK_FRAMES = 16;
* seconds apart — fills the stretch the drag is entering and pays for that
* stretch alone. */
const BACK_SPAN_S = 2.5;
-/** Frames a source must go unwanted before the pool will close it. About a
+/** Frames a source must go unwanted before the pool will suspend it. About a
* second of playback — long enough that crossing a cut and coming back finds
* the decoder still open. */
const EVICT_GRACE = 90;
+/**
+ * Canvas backing the pool may hold in stood-down sources, past the ones the
+ * decoder budget is keeping live.
+ *
+ * A suspended source has let go of its decoder but keeps the parsed file and
+ * the sink whose canvases its frames land on, and the canvases are what this
+ * budget is about. They must not churn: allocating them floods the GPU
+ * process, and a montage of short clips replayed for a few minutes — every
+ * pass reopening every clip's source, each open minting a sink's worth of
+ * canvases — grinds the whole tab down until frames stop arriving at all. They
+ * also must not pile up: a sink's pool is `POOL` full-size canvases, so
+ * counting sources rather than pixels would keep as many of them at 4K as at
+ * 360p and put gigabytes behind a large stage.
+ *
+ * So the warm shelf is capped by the memory itself, at four bytes a pixel.
+ * What that buys is decided by the decode size: a small stage keeps a
+ * montage's whole cast warm, a large one keeps a few — the right trade in both
+ * directions, since the big frames are the expensive ones to hold and the
+ * cheap ones to lose. The live sources are governed by the decoder budget
+ * alone: closing one the picture is about to need would trade the churn back
+ * for a black frame.
+ */
+const WARM_PIXELS = 96e6;
+/** Stood-down sources kept regardless of the pixel budget, so a project of
+ * tiny frames cannot leave hundreds of parsed files open. */
+const WARM_KEEP_MAX = 32;
/** A failed open tries again this much later, growing per attempt over the
* first `RETRIES` tries, so a network blip heals within seconds. */
const RETRY_MS = 1000;
@@ -293,6 +329,30 @@ export class ClipFrameSource {
* pool's eviction order. */
touched = 0;
+ /** When the pull awaiting the walk's next frame started, for the debug dump. */
+ private nextStartedAt = 0;
+
+ /** Stood down: no decoder, file and sink still held. See `suspend`. */
+ private asleep = false;
+
+ get suspended(): boolean {
+ return this.asleep;
+ }
+
+ /**
+ * Canvas backing this source can be holding: the sink's pool at the decode
+ * size, which is the frame height asked for and the file's own aspect. Read
+ * before the sink has opened — the pool budget has to decide what a source
+ * would cost, and a decode not yet started still costs it a moment later.
+ */
+ get keptPixels(): number {
+ if (this.asset.type === "image") return this.still ? this.height * this.height * 2 : 0;
+ const aspect =
+ this.asset.width && this.asset.height ? this.asset.width / this.asset.height : 16 / 9;
+ const width = Math.min(Math.round(this.height * aspect), this.asset.width ?? Infinity);
+ return POOL * width * this.height;
+ }
+
/** The URL this source is reading. The pool compares it against the store's
* current one, so a re-minted signed URL replaces the source under it. */
get url(): string {
@@ -318,6 +378,32 @@ export class ClipFrameSource {
this.back = null;
}
+ /** What this source is doing, for the perf eval's pool dump. */
+ debugState(): Record<string, unknown> {
+ return {
+ url: this.asset.url.slice(-24),
+ ring: this.ring.size,
+ oldest: this.ring.size ? +this.ring.oldest.toFixed(2) : null,
+ newest: this.ring.size ? +this.ring.newest.toFixed(2) : null,
+ still: !!this.still,
+ opened: !!this.sink,
+ opening: !!this.opening,
+ unreadable: this.unreadable,
+ attempts: this.attempts,
+ walking: !!this.stream,
+ draining: !!this.drainRun,
+ streamFrom: +this.streamFrom.toFixed(2),
+ streamTail: +this.streamTail.toFixed(2),
+ streamDone: this.streamDone,
+ reading: this.reading,
+ wanted: this.wanted,
+ touched: this.touched,
+ suspended: this.asleep,
+ keptMb: +(this.keptPixels * 4e-6).toFixed(1),
+ nextPendingMs: this.nextStartedAt ? Math.round(performance.now() - this.nextStartedAt) : 0,
+ };
+ }
+
get ready(): boolean {
return this.still !== null || this.ring.size > 0;
}
@@ -353,6 +439,9 @@ export class ClipFrameSource {
*/
want(t: number, playing: boolean): void {
if (this.closed || this.unreadable) return;
+ // Being read is what wakes a stood-down source; the walk below is what
+ // gives it a decoder again.
+ this.asleep = false;
if (this.asset.type === "image") {
void this.openStill();
return;
@@ -404,6 +493,30 @@ export class ClipFrameSource {
}
}
+ /**
+ * Stand down without letting go of the file.
+ *
+ * The walk ends — which is what releases the decoder — while the parsed
+ * input and the sink stay, so waking this source later costs one keyframe
+ * seek on canvases it already owns. The ring is dropped with the walk: a
+ * revival starts a fresh walk on the same sink, whose pool then recycles the
+ * canvases the ring's old frames live on, and a held reference would show
+ * whatever the new walk drew over it.
+ *
+ * Standing down is a state rather than an act, so the pool can name the same
+ * source every frame and only the first one does anything.
+ */
+ suspend(): void {
+ if (this.asleep) return;
+ this.asleep = true;
+ this.stopStream();
+ this.streamDone = false;
+ this.ring.clear();
+ this.back = null;
+ this.wanted = null;
+ this.lastAsk = null;
+ }
+
close(): void {
this.closed = true;
this.stopStream();
@@ -505,6 +618,7 @@ export class ClipFrameSource {
private stopStream(): void {
const s = this.stream;
this.stream = null;
+ this.nextStartedAt = 0;
void s?.return(undefined).catch(() => {});
}
@@ -683,7 +797,9 @@ export class ClipFrameSource {
// gestures merely age out of the cache as the walk pushes.
if (this.streamTail >= t + DECODE_AHEAD_S) break;
if (this.ring.between(Math.max(t, this.streamFrom), this.streamTail) >= RING - 1) break;
+ this.nextStartedAt = performance.now();
const { value, done } = await stream.next();
+ this.nextStartedAt = 0;
// A restart or a close while awaiting: this walk is no longer the one.
if (this.stream !== stream) break;
if (done || !value) {
@@ -811,6 +927,7 @@ export class FrameSourcePool {
src = undefined;
}
if (!src) {
+ poolLog(`pool open ${id}`);
src = new ClipFrameSource(asset, height, this.onFrame);
this.sources.set(id, src);
}
@@ -829,8 +946,29 @@ export class FrameSourcePool {
return this.sources.size;
}
+ /** Sources holding a decoder — what the budget is a budget of. The rest are
+ * stood down: a file and its canvases, waiting to be read again. */
+ get active(): number {
+ let n = 0;
+ for (const s of this.sources.values()) if (!s.suspended) n++;
+ return n;
+ }
+
+ /** Canvas backing the warm shelf is holding — the stood-down sources, which
+ * is the part `WARM_PIXELS` governs. For the perf trace. */
+ get warmPixels(): number {
+ let n = 0;
+ for (const s of this.sources.values()) if (s.suspended) n += s.keptPixels;
+ return n;
+ }
+
+ /** Every open source and what it is doing, for the perf eval's pool dump. */
+ debugState(): Record<string, unknown>[] {
+ return [...this.sources.entries()].map(([id, s]) => ({ id, tick: this.tick, ...s.debugState() }));
+ }
+
/**
- * Close the decoders nothing has asked for lately.
+ * Stand down the decoders nothing has asked for lately.
*
* "Lately" is doing real work here. Evicting everything untouched by the
* current frame reads as tidy and behaves terribly: a cut with more clips
@@ -839,6 +977,10 @@ export class FrameSourcePool {
* expensive than the memory it was saving. A source is only a candidate once
* nothing has wanted it for a while, which is long enough that a clip being
* crossed back and forth over is never the one closed.
+ *
+ * Standing down is suspension, not closure: the decoder goes, the sink and
+ * its canvases stay for the next visit — see `KEEP_PIXELS`, which is where
+ * sources actually close.
*/
evict(): void {
// A backward-skim cache on a source nothing reads any more is copies held
@@ -846,19 +988,41 @@ export class FrameSourcePool {
// its cache is not.
for (const s of this.sources.values())
if (this.tick - s.touched >= EVICT_GRACE) s.dropBack();
- if (this.sources.size <= this.budget) return;
const idle = [...this.sources.entries()]
.filter(([, s]) => this.tick - s.touched >= EVICT_GRACE)
.sort((a, b) => a[1].touched - b[1].touched);
- let over = this.sources.size - this.budget;
+
+ // Past the decoder budget, the least recently read decoders stand down.
+ // The budget counts the sources actually holding one, so a stood-down
+ // source is not what pushes the next one out.
+ let live = this.active;
for (const [id, src] of idle) {
- if (over-- <= 0) break;
+ if (live <= this.budget) break;
+ if (src.suspended) continue;
+ poolLog(`pool suspend ${id}`);
+ src.suspend();
+ live--;
+ }
+
+ // Then the memory. The warm shelf is filled most-recently-read first until
+ // the canvas budget runs out, and what does not fit closes for real. Only
+ // stood-down sources are on the shelf, so nothing holding a decoder — and
+ // nothing the picture is being drawn from — is ever closed here.
+ let pixels = 0;
+ let warm = 0;
+ for (const [id, src] of [...this.sources.entries()].sort((a, b) => b[1].touched - a[1].touched)) {
+ if (!src.suspended) continue;
+ pixels += src.keptPixels;
+ warm++;
+ if (pixels <= WARM_PIXELS && warm <= WARM_KEEP_MAX) continue;
+ poolLog(`pool close ${id}`);
src.close();
this.sources.delete(id);
}
}
closeAll(): void {
+ poolLog(`pool closeAll (${this.sources.size})`);
for (const s of this.sources.values()) s.close();
this.sources.clear();
}
diff --git a/site/src/cut/lib/perfTrace.ts b/site/src/cut/lib/perfTrace.ts
index e62dc3f7..ff4f5cb6 100644
--- a/site/src/cut/lib/perfTrace.ts
+++ b/site/src/cut/lib/perfTrace.ts
@@ -23,6 +23,14 @@ export interface PresentRecord {
/** Source timestamp of the master layer's picture, for spotting a frame that
* was shown twice or skipped. */
srcTs: number | null;
+ /**
+ * The source time the master layer was asked for.
+ *
+ * With `srcTs` this is the whole sync story: sound is scheduled against the
+ * same clock `t` comes from, so `wantSrc - srcTs` is how far the picture sits
+ * behind the sound at that instant.
+ */
+ wantSrc: number | null;
/** The clip the master picture came from, so a boundary is findable. */
clipId: string | null;
/**
@@ -62,6 +70,14 @@ export interface Trace {
ticks: number;
/** Decoded frames the sources are holding open, sampled per present. */
liveSamples: number[];
+ /** Decoders open, sampled per present. Stood-down sources hold none, so this
+ * is what the pool's budget is measured against. */
+ liveSources: number[];
+ /** Sources the pool holds at all, decoder or not, sampled per present. */
+ keptSources: number[];
+ /** Megabytes of canvas backing on the warm shelf — the stood-down sources,
+ * which is the part the pool's memory budget governs. */
+ warmMb: number[];
startedAt: number;
}
@@ -86,6 +102,9 @@ export function startTrace(): void {
longTasks: [],
ticks: 0,
liveSamples: [],
+ liveSources: [],
+ keptSources: [],
+ warmMb: [],
startedAt: performance.now(),
};
pendingSeek = null;
@@ -150,3 +169,12 @@ export function markLiveSamples(n: number): void {
if (!trace) return;
trace.liveSamples.push(n);
}
+
+/** What the decoder pool is holding right now: sources with a decoder, sources
+ * kept at all, and the canvas backing behind them. */
+export function markLiveSources(active: number, kept: number, warmPixels: number): void {
+ if (!trace) return;
+ trace.liveSources.push(active);
+ trace.keptSources.push(kept);
+ trace.warmMb.push(Math.round(warmPixels * 4e-6));
+}
diff --git a/site/src/cut/lib/previewMixer.ts b/site/src/cut/lib/previewMixer.ts
index de798032..652723c0 100644
--- a/site/src/cut/lib/previewMixer.ts
+++ b/site/src/cut/lib/previewMixer.ts
@@ -124,11 +124,34 @@ export class PreviewMixer {
const audio = this.anchor.timeline + (this.ctx.currentTime - this.anchor.ctx);
if (Math.abs(audio - wall) > 0.25) {
this.anchor.ctx = this.ctx.currentTime - (wall - this.anchor.timeline);
+ // Everything already scheduled was placed against the old anchor and is
+ // now wrong by the size of the jump — a context that sat suspended until
+ // the user's next gesture, an output device switch. The clock alone
+ // moving would leave those windows playing the wrong moment for up to
+ // their full length, so they stop here and the next update refills from
+ // where the clock now stands.
+ this.restart(wall);
return wall;
}
return audio;
}
+ /** Stop every scheduled window and mark each voice to refill at `timeline`. */
+ private restart(timeline: number): void {
+ for (const live of this.voices.values()) {
+ for (const w of live.windows) {
+ try {
+ w.node.stop();
+ } catch {
+ // Already finished; stopping a spent node throws and means nothing.
+ }
+ w.node.disconnect();
+ }
+ live.windows = [];
+ live.scheduled = Math.max(live.start, timeline);
+ }
+ }
+
/**
* Begin playing at timeline time `t`.
*
@@ -295,20 +318,24 @@ export class PreviewMixer {
this.decoded.set(key, buffer);
}
// The mixer may have been stopped, or the voice dropped, while decoding.
- const still = this.voices.get(id);
- if (!still || still !== live || !this.anchor || !this.ctx) return;
- const node = this.ctx.createBufferSource();
- node.buffer = buffer;
- node.connect(live.gain);
- const at = this.anchor.ctx + (from - this.anchor.timeline);
- const now = this.ctx.currentTime;
+ if (this.voices.get(id) !== live || !this.anchor || !this.ctx) return;
+ // Reading the clock can restart the voices (`restart`), so it happens
+ // before the placement below is computed — and a restarted voice, moved
+ // to a different position while this window decoded, abandons it.
+ const tNow = this.now();
+ if (live.scheduled !== from) return;
const until = from + buffer.duration;
- if (until <= this.now()) {
+ if (until <= tNow) {
// The decode outlived the window it was for — a long read, or a tab
// that was in the background. Skip it and let the next fill catch up.
- live.scheduled = Math.max(live.scheduled, this.now());
+ live.scheduled = Math.max(live.scheduled, tNow);
return;
}
+ const node = this.ctx.createBufferSource();
+ node.buffer = buffer;
+ node.connect(live.gain);
+ const at = this.anchor.ctx + (from - this.anchor.timeline);
+ const now = this.ctx.currentTime;
if (at >= now) {
node.start(at);
} else {