Open source CapCut alternative.
TypeScript
fix(site): a voice re-aims its reader instead of opening the file again [rebuild]
David Le8eda0cb64ce8
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.mdindex 15553421..788eba6b 100644--- a/docs/guides/cut/README.md+++ b/docs/guides/cut/README.md@@ -56,7 +56,7 @@ Decoders live in a bounded pool, and leaving the pool is suspension before it isTwo things follow from that. A cut is a read from a different ring rather than a handover between players, so crossing a join costs nothing and needs no clock to change hands; and the picture answers a moved playhead inside a frame, because the answer was already in memory. Sound is scheduled on one audio context and its clock is what playback runs on — the picture follows the sound rather than the two chasing each other. When a browser holds that context suspended, as it does until the page has been interacted with, wall time carries the cut and the audio clock is picked up once it agrees — and when the clock jumps instead of agreeing, a context resumed late or an output device switched mid-play, every scheduled window restarts where the clock now stands rather than playing out the moment it was placed for. A context that stops running mid-play is asked back every frame, because the picture would otherwise carry on against wall time over a cut gone silent.-Each voice reads its file the way the picture's readers do: a walk held open, pulled forward about a second at a time. Reading in order is what keeps sound and picture in the same part of the file. A voice asking for a long span at once is asking for the whole interleaved stretch that span is scattered through — twenty seconds of sound is a hundred megabytes of a phone recording — so over a link that merely keeps up with playback its every read arrives after the moment it was for, and a cut that sounded for its first stretch plays out in silence. The rule about endings is the picture's rule, for the same reason: a walk reports the same ending whether it read the track's last sample or gave up on it, and an ending believed a few seconds into a long clip plays the rest of the cut in silence while the picture reads happily past it. So an ending short of the clip is treated as a walk that failed and another goes after the sound it did not reach, and a walk that gets no further than the one before it has found where the track really stops — which is often a fraction before the picture does. Nothing else about a read is final either. A read that fails, comes back empty, or stops answering leaves the voice waiting a moment and asking again, quickly at first and then on a slow cadence for as long as the failure holds, and it asks the link keeper for a fresh URL on the way — the same footing the picture's readers are on, which is what stops a blip from silencing a clip whose picture healed seconds later.+Each voice reads its file the way the picture's readers do: a walk held open, pulled forward about a second at a time. Reading in order is what keeps sound and picture in the same part of the file. A voice asking for a long span at once is asking for the whole interleaved stretch that span is scattered through — twenty seconds of sound is a hundred megabytes of a phone recording — so over a link that merely keeps up with playback its every read arrives after the moment it was for, and a cut that sounded for its first stretch plays out in silence. The rule about endings is the picture's rule, for the same reason: a walk reports the same ending whether it read the track's last sample or gave up on it, and an ending believed a few seconds into a long clip plays the rest of the cut in silence while the picture reads happily past it. So an ending short of the clip is treated as a walk that failed and another goes after the sound it did not reach, and a walk that gets no further than the one before it has found where the track really stops — which is often a fraction before the picture does. Nothing else about a read is final either. A read that fails, comes back empty, or stops answering leaves the voice waiting a moment and asking again, quickly at first and then on a slow cadence for as long as the failure holds, and it asks the link keeper for a fresh URL on the way — the same footing the picture's readers are on, which is what stops a blip from silencing a clip whose picture healed seconds later. Opening a walk is nearly all a walk costs — a container read out of a hundred megabytes over a link — so a voice holding sound for a moment that has gone by re-aims the reader it already has, the way the picture's readers are re-aimed. Giving the file up there would start the open again a second behind where it just was, and again after that: the voice pays the open for the rest of the play and never once catches the clock, which is the same silence arrived at from the other direction. The reader moves for a new address too — a library clip copied into the project behind the editor moves off the URL it arrived on mid-play — and the sound already read plays out across the change.What is on screen at a given instant is still a pure function of the document and a time, and it is still the same function an export asks. The preview reaches an instant by playing toward it and an export by jumping straight there; because both draw through one compositor from one plan, the file cannot quietly disagree with what was watched.diff --git a/site/src/cut/lib/mediaRead.ts b/site/src/cut/lib/mediaRead.tsindex 5b9e4b49..96fbf794 100644--- a/site/src/cut/lib/mediaRead.ts+++ b/site/src/cut/lib/mediaRead.ts@@ -487,6 +487,12 @@ export function assembleAudio(* gives up the file however far it got. */export interface AudioWalk {next(): Promise<WrappedAudioBuffer | null>;+ /** Re-aim the walk at `from` without giving up the file. Opening one is+ * nearly all a walk costs — the container read out of a hundred megabytes+ * over a link — and a caller that has to re-aim has every reason not to pay+ * it twice: the clock ran past what this walk was reading, a seek landed+ * somewhere else in the same file. The next read starts there. */+ seek(from: number): void;close(): void;}@@ -510,12 +516,25 @@ export interface AudioWalk {export function openAudioWalk(src: string | Blob, from = 0, to?: number): AudioWalk {const input = openMedia(src);let closed = false;- let opening: Promise<AsyncGenerator<WrappedAudioBuffer, void, unknown> | null> | null = null;- const open = () =>- (opening ??= (async () => {- const track = await audioTrackOf(input);- return track ? new AudioBufferSink(track).buffers(from, to) : null;- })());+ let finding: Promise<InputAudioTrack | null> | null = null;+ let reader: AsyncGenerator<WrappedAudioBuffer, void, unknown> | null = null;+ /** Where the next read starts, until a reader is standing at it. */+ let aim: number | null = from;+ const stop = (gen: AsyncGenerator<WrappedAudioBuffer, void, unknown> | null) => {+ // Returning a generator is where the decoder it holds is released.+ if (gen) void gen.return().catch(() => {});+ };+ const open = async () => {+ const track = await (finding ??= audioTrackOf(input));+ if (!track || closed) return null;+ if (aim !== null) {+ const old = reader;+ reader = new AudioBufferSink(track).buffers(aim, to);+ aim = null;+ stop(old);+ }+ return reader;+ };return {async next() {if (closed) return null;@@ -524,14 +543,17 @@ export function openAudioWalk(src: string | Blob, from = 0, to?: number): AudioWconst step = await gen.next();return step.done ? null : step.value;},+ seek(next) {+ aim = next;+ },close() {if (closed) return;closed = true;- // The generator is told it is over, which is where the decoder it holds- // is released, and the file goes at once behind it: disposing is what- // cancels a read still in the air, and a walk being closed because it- // stopped answering is waiting on exactly one of those.- void opening?.then((gen) => gen?.return()).catch(() => {});+ // The file goes at once behind the reader: disposing is what cancels a+ // read still in the air, and a walk being closed because it stopped+ // answering is waiting on exactly one of those.+ stop(reader);+ reader = null;input.dispose();},};diff --git a/site/src/cut/lib/previewMixer.test.ts b/site/src/cut/lib/previewMixer.test.tsindex 7911fc4f..bbcf7621 100644--- a/site/src/cut/lib/previewMixer.test.ts+++ b/site/src/cut/lib/previewMixer.test.ts@@ -46,13 +46,22 @@ class FakeGain {class FakeSource {buffer: { duration: number } | null = null;+ private span: Played | null = null;connect() {}disconnect() {}start(when: number, offset = 0) {const length = (this.buffer?.duration ?? 0) - offset;- if (length > 0) played.push({ from: when, to: when + length });+ if (length > 0) {+ this.span = { from: when, to: when + length };+ played.push(this.span);+ }+ }+ /** Stopping a scheduled source silences what it had not played yet, so what+ * was heard ends here — the difference between a voice let alone and a voice+ * taken down mid-sound. */+ stop() {+ if (this.span) this.span.to = Math.max(this.span.from, Math.min(this.span.to, ctxTime));}- stop() {}}class FakeContext {@@ -90,8 +99,19 @@ let endEarlyAt = Infinity;/** Where the track truly stops, which can be short of the clip. */let trackEnd = CLIP_S;+/** What the first read off a newly opened walk costs before it answers: a+ * container parsed from nothing over the link. A voice that reopens mid-play+ * pays it; one that keeps its reader does not. */+let openCostS = 0;+const waiting: { at: number; wake: () => void }[] = [];+/** Resolves `seconds` of wall time from now. */+function after(seconds: number): Promise<void> {+ if (seconds <= 0) return Promise.resolve();+ return new Promise((wake) => waiting.push({ at: wall + seconds, wake }));+}+/** Resolves once the link has delivered the file as far as `sourceTime`. */function delivered(sourceTime: number): Promise<void> {const at = Math.max(0, (sourceTime - HEAD_START_S) / LINK_RATE);@@ -108,12 +128,19 @@ const openAudioWalk = (_url: string, from: number, to?: number) => {// given up, and the walk sent after it reads the file properly.const givesUpAt = endEarlyAt;endEarlyAt = Infinity;+ const cost = openCostS;+ let opened = false;let pos = from;let closed = false;let done = false;return {async next() {if (closed || done) return null;+ if (!opened) {+ opened = true;+ await after(cost);+ if (closed) return null;+ }if (wall < hangUntil) return new Promise<null>(() => {});await delivered(pos + PART_S);if (closed) return null;@@ -136,6 +163,10 @@ const openAudioWalk = (_url: string, from: number, to?: number) => {pos += duration;return part;},+ seek(next: number) {+ pos = next;+ done = false;+ },close() {closed = true;},@@ -230,8 +261,10 @@ async function play(reMints.asked = 0;failUntil = 0;hangUntil = 0;+ openCostS = 0;endEarlyAt = file.endEarlyAt ?? Infinity;trackEnd = file.trackEnd ?? CLIP_S;+ voice.url = "clip.mp4";const mixer = new PreviewMixer();mixer.start(0);try {@@ -313,6 +346,45 @@ describe("the preview's sound over a link that only just keeps up", () => {expect(walks.opened).toBeLessThan(4);});+ test("catches the clock up after an open that took seconds", async () => {+ // The failure this is here for, and the shape every report of it took: the+ // sound plays, stops somewhere in the first minute, and never comes back+ // while the picture rolls on to the end. One read fails, the walk that+ // replaces it takes a few seconds to parse the file, and the sound it+ // finally brings back is for a moment that has gone by — so the voice+ // gives the walk up, opens another, and is late again by exactly as much.+ // It pays the open for the rest of the play and never catches the clock.+ const sound = await play((t) => {+ if (t < 20) return;+ openCostS = 5;+ if (t < 21) failUntil = 21;+ });+ expect(heard(sound, 0, 18)).toBeGreaterThan(0.9);+ expect(heard(sound, 32, CLIP_S)).toBeGreaterThan(0.95);+ // One open answered for; the walk is re-aimed after that, not replaced.+ expect(walks.opened).toBeLessThan(4);+ });++ test("plays on when the same sound moves to a new address", async () => {+ // The failure this is here for: a clip dragged in from the library plays+ // from the library's own URL while its bytes are copied into the project+ // behind the editor, and the asset moves to the stored file the moment+ // they land — mid-play, without a note of the sound changing. Taking the+ // voice down there stops what was already scheduled and puts the sound+ // out for as long as a cold open of a long file takes.+ const sound = await play((t) => {+ if (t < 20) return;+ openCostS = 5;+ voice.url = "landed.mp4";+ });+ // What the voice had already read plays out across the swap. It is the+ // reader that moves, and it moves while that sound is playing.+ expect(heard(sound, 18, 22)).toBeGreaterThan(0.95);+ expect(heard(sound, 30, CLIP_S)).toBeGreaterThan(0.95);+ // The new address is read by a walk of its own; the old one is let go.+ expect(walks.opened).toBe(2);+ });+test("picks its context back up when the browser puts it down", async () => {const sound = await play((t, ctx) => {if (t >= 20 && t < 20.1) ctx.state = "suspended";diff --git a/site/src/cut/lib/previewMixer.ts b/site/src/cut/lib/previewMixer.tsindex d19c2d2b..d1f35289 100644--- a/site/src/cut/lib/previewMixer.ts+++ b/site/src/cut/lib/previewMixer.ts@@ -463,21 +463,35 @@ export class PreviewMixer {for (const v of voices) {seen.add(v.id);// An edit to a playing clip arrives as the same id with new geometry — a- // trim, a move, a speed change, a re-minted URL. Gain is the only part of- // a voice that tunes live; everything else is baked into what was already- // scheduled, so the voice reopens and the sound follows the edit the way- // the picture does.+ // trim, a move, a speed change. Gain is the only part of a voice that+ // tunes live; everything else is baked into what was already scheduled,+ // so the voice reopens and the sound follows the edit the way the+ // picture does.const prev = this.voices.get(v.id);if (prev &&- (prev.url !== v.url ||- prev.start !== v.start ||+ (prev.start !== v.start ||prev.in !== v.in ||prev.out !== v.out ||prev.speed !== v.speed)) {this.release(prev);this.voices.delete(v.id);+ } else if (prev && prev.url !== v.url) {+ // The same sound at a new address: a signed link re-minted, an import+ // that landed in project storage and moved off the URL it arrived on.+ // The bytes already scheduled are the same sound, so they play on and+ // only the next read moves — dropping them would take the sound out+ // for as long as a cold open of the new address takes, which on a+ // long file is most of a minute and sounds like the audio quitting.+ this.forget(prev.url);+ prev.url = v.url;+ this.moved(prev);+ // A new address answers for itself, so nothing the old one did holds+ // this voice back from asking straight away.+ prev.attempts = 0;+ prev.retryAt = 0;+ prev.reached = -Infinity;}const live = this.voices.get(v.id) ?? this.open(v, t);if (!live) continue;@@ -669,8 +683,7 @@ export class PreviewMixer {// tab in the background. Where it left the walk is behind the clock// now, so the voice re-aims at the clock and gives up the stretch in// between; that stretch was going by either way.- this.moved(live);- live.scheduled = Math.max(live.scheduled, tNow);+ this.reaim(live, tNow);return;}const node = this.ctx.createBufferSource();@@ -721,6 +734,12 @@ export class PreviewMixer {const part = await walk.next();if (!current()) return null;if (!part) break;+ // The stall guard is about a source that stopped answering, and a walk+ // that is handing sound back has not: every part it delivers is the read+ // saying it is still there. Without this a cold open of a long file over+ // a slow link is disowned on the clock, re-opened from nothing, and+ // disowned again, and the voice never gets a group out.+ live.fillAt = performance.now();parts.push(part);end = part.timestamp + part.duration;}@@ -747,6 +766,26 @@ export class PreviewMixer {return buffer;}+ /**+ * Move a voice up to the clock, keeping the file it is reading.+ *+ * A group that lands after the moment it was for is the ordinary shape of a+ * slow open: the file takes a few seconds to parse, and by the time the+ * first sound comes back the playhead has gone past it. Answering that by+ * giving up the walk starts the whole open again a second behind where it+ * just was, and again after that — the voice pays the open cost for the rest+ * of the play and never once catches the clock, which is a cut that plays+ * out in silence while the picture rolls on. The reader moves instead: same+ * file, same decoder, aimed where the sound is wanted.+ */+ private reaim(live: LiveVoice, t: number): void {+ live.gen++;+ live.filling = false;+ live.ended = false;+ live.scheduled = Math.max(live.scheduled, t);+ live.walk?.seek(live.in + (live.scheduled - live.start) * live.speed);+ }+/** A read that failed, came back empty, or stopped answering. The voice* gives up where it was, waits, and asks again — and asks for as long as the* play lasts. */