Skip to content

Open source CapCut alternative.

TypeScript

fix(site): the sound has its file open before the play begins [rebuild]

David Le3d5625f1f376
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index 788eba6b..6c43c7c7 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -56,7 +56,7 @@ Decoders live in a bounded pool, and leaving the pool is suspension before it is
Two things follow from that. A cut is a read from a different ring rather than a handover between players, so crossing a join costs nothing and needs no clock to change hands; and the picture answers a moved playhead inside a frame, because the answer was already in memory. Sound is scheduled on one audio context and its clock is what playback runs on — the picture follows the sound rather than the two chasing each other. When a browser holds that context suspended, as it does until the page has been interacted with, wall time carries the cut and the audio clock is picked up once it agrees — and when the clock jumps instead of agreeing, a context resumed late or an output device switched mid-play, every scheduled window restarts where the clock now stands rather than playing out the moment it was placed for. A context that stops running mid-play is asked back every frame, because the picture would otherwise carry on against wall time over a cut gone silent.
-Each voice reads its file the way the picture's readers do: a walk held open, pulled forward about a second at a time. Reading in order is what keeps sound and picture in the same part of the file. A voice asking for a long span at once is asking for the whole interleaved stretch that span is scattered through — twenty seconds of sound is a hundred megabytes of a phone recording — so over a link that merely keeps up with playback its every read arrives after the moment it was for, and a cut that sounded for its first stretch plays out in silence. The rule about endings is the picture's rule, for the same reason: a walk reports the same ending whether it read the track's last sample or gave up on it, and an ending believed a few seconds into a long clip plays the rest of the cut in silence while the picture reads happily past it. So an ending short of the clip is treated as a walk that failed and another goes after the sound it did not reach, and a walk that gets no further than the one before it has found where the track really stops — which is often a fraction before the picture does. Nothing else about a read is final either. A read that fails, comes back empty, or stops answering leaves the voice waiting a moment and asking again, quickly at first and then on a slow cadence for as long as the failure holds, and it asks the link keeper for a fresh URL on the way — the same footing the picture's readers are on, which is what stops a blip from silencing a clip whose picture healed seconds later. Opening a walk is nearly all a walk costs — a container read out of a hundred megabytes over a link — so a voice holding sound for a moment that has gone by re-aims the reader it already has, the way the picture's readers are re-aimed. Giving the file up there would start the open again a second behind where it just was, and again after that: the voice pays the open for the rest of the play and never once catches the clock, which is the same silence arrived at from the other direction. The reader moves for a new address too — a library clip copied into the project behind the editor moves off the URL it arrived on mid-play — and the sound already read plays out across the change.
+Each voice reads its file the way the picture's readers do: a walk held open, pulled forward about a second at a time. Reading in order is what keeps sound and picture in the same part of the file. A voice asking for a long span at once is asking for the whole interleaved stretch that span is scattered through — twenty seconds of sound is a hundred megabytes of a phone recording — so over a link that merely keeps up with playback its every read arrives after the moment it was for, and a cut that sounded for its first stretch plays out in silence. The rule about endings is the picture's rule, for the same reason: a walk reports the same ending whether it read the track's last sample or gave up on it, and an ending believed a few seconds into a long clip plays the rest of the cut in silence while the picture reads happily past it. So an ending short of the clip is treated as a walk that failed and another goes after the sound it did not reach, and a walk that gets no further than the one before it has found where the track really stops — which is often a fraction before the picture does. Nothing else about a read is final either. A read that fails, comes back empty, or stops answering leaves the voice waiting a moment and asking again, quickly at first and then on a slow cadence for as long as the failure holds, and it asks the link keeper for a fresh URL on the way — the same footing the picture's readers are on, which is what stops a blip from silencing a clip whose picture healed seconds later. Opening a walk is nearly all a walk costs — a container read out of a hundred megabytes over a link — so a voice holding sound for a moment that has gone by re-aims the reader it already has, the way the picture's readers are re-aimed. Giving the file up there would start the open again a second behind where it just was, and again after that: the voice pays the open for the rest of the play and never once catches the clock, which is the same silence arrived at from the other direction. The reader moves for a new address too — a library clip copied into the project behind the editor moves off the URL it arrived on mid-play — and the sound already read plays out across the change. A reader is opened before it is needed, on the same footing as the picture's: a clip under a parked playhead has had a decoder open on it all along, which is why the picture starts on the frame, and the sound now opens its file and takes one read where the playhead stands rather than doing all of it once the clock is already running. That is what a play straight after a seek used to wait seconds for.
What is on screen at a given instant is still a pure function of the document and a time, and it is still the same function an export asks. The preview reaches an instant by playing toward it and an export by jumping straight there; because both draw through one compositor from one plan, the file cannot quietly disagree with what was watched.
diff --git a/site/src/cut/hooks/usePlayback.ts b/site/src/cut/hooks/usePlayback.ts
index 17e4400d..9966f84b 100644
--- a/site/src/cut/hooks/usePlayback.ts
+++ b/site/src/cut/hooks/usePlayback.ts
@@ -690,6 +690,11 @@ class Engine {
} else {
this.writeHead(t);
}
+ } else {
+ // The sound's readers on the same footing as the picture's: the clip
+ // under a parked playhead has its file open before the play begins,
+ // rather than opening it once the clock is already running.
+ this.mixer.warm(t, this.voicesAt(t, spans, master));
}
if (tracing()) {
diff --git a/site/src/cut/lib/mediaRead.ts b/site/src/cut/lib/mediaRead.ts
index 96fbf794..ea1f93d6 100644
--- a/site/src/cut/lib/mediaRead.ts
+++ b/site/src/cut/lib/mediaRead.ts
@@ -487,6 +487,9 @@ export function assembleAudio(
* gives up the file however far it got. */
export interface AudioWalk {
next(): Promise<WrappedAudioBuffer | null>;
+ /** The source time the next read comes from. A caller that has moved can ask
+ * whether this walk is standing where it wants to read. */
+ readonly position: number;
/** Re-aim the walk at `from` without giving up the file. Opening one is
* nearly all a walk costs — the container read out of a hundred megabytes
* over a link — and a caller that has to re-aim has every reason not to pay
@@ -518,8 +521,9 @@ export function openAudioWalk(src: string | Blob, from = 0, to?: number): AudioW
let closed = false;
let finding: Promise<InputAudioTrack | null> | null = null;
let reader: AsyncGenerator<WrappedAudioBuffer, void, unknown> | null = null;
- /** Where the next read starts, until a reader is standing at it. */
- let aim: number | null = from;
+ /** Where the next read comes from, and whether a reader stands at it. */
+ let at = from;
+ let aimed = false;
const stop = (gen: AsyncGenerator<WrappedAudioBuffer, void, unknown> | null) => {
// Returning a generator is where the decoder it holds is released.
if (gen) void gen.return().catch(() => {});
@@ -527,24 +531,30 @@ export function openAudioWalk(src: string | Blob, from = 0, to?: number): AudioW
const open = async () => {
const track = await (finding ??= audioTrackOf(input));
if (!track || closed) return null;
- if (aim !== null) {
+ if (!aimed) {
const old = reader;
- reader = new AudioBufferSink(track).buffers(aim, to);
- aim = null;
+ reader = new AudioBufferSink(track).buffers(at, to);
+ aimed = true;
stop(old);
}
return reader;
};
return {
+ get position() {
+ return at;
+ },
async next() {
if (closed) return null;
const gen = await open();
if (!gen || closed) return null;
const step = await gen.next();
- return step.done ? null : step.value;
+ if (step.done) return null;
+ at = step.value.timestamp + step.value.duration;
+ return step.value;
},
seek(next) {
- aim = next;
+ at = next;
+ aimed = false;
},
close() {
if (closed) return;
diff --git a/site/src/cut/lib/previewMixer.test.ts b/site/src/cut/lib/previewMixer.test.ts
index e9edb81d..8b2c8cf3 100644
--- a/site/src/cut/lib/previewMixer.test.ts
+++ b/site/src/cut/lib/previewMixer.test.ts
@@ -22,6 +22,8 @@ const CLIP_S = 70;
* picture reads is comfortable. */
const LINK_RATE = 1;
const HEAD_START_S = 2.5;
+/** How much of the file is already here when a case starts. */
+let headStart = HEAD_START_S;
// ── the clock everything shares ─────────────────────────────────────────────
let wall = 0;
@@ -119,7 +121,7 @@ function after(seconds: number): Promise<void> {
/** Resolves once the link has delivered the file as far as `sourceTime`. */
function delivered(sourceTime: number): Promise<void> {
- const at = Math.max(0, (sourceTime - HEAD_START_S) / LINK_RATE);
+ const at = Math.max(0, (sourceTime - headStart) / LINK_RATE);
if (wall >= at) return Promise.resolve();
return new Promise((wake) => waiting.push({ at, wake }));
}
@@ -168,6 +170,9 @@ const openAudioWalk = (_url: string, from: number, to?: number) => {
pos += duration;
return part;
},
+ get position() {
+ return pos;
+ },
seek(next: number) {
pos = next;
done = false;
@@ -253,12 +258,8 @@ const voice = {
gain: 1,
};
-/** Play the clip through, running `during` on each frame at the timeline time
- * it is drawing. Answers what was heard. */
-async function play(
- during?: (t: number, ctx: FakeContext) => void,
- file: { endEarlyAt?: number; trackEnd?: number } = {}
-): Promise<Played[]> {
+/** Put every control back where a case starts. */
+function reset(file: { endEarlyAt?: number; trackEnd?: number } = {}): void {
wall = 0;
ctxTime = 0;
resumes = 0;
@@ -270,10 +271,20 @@ async function play(
hangUntil = 0;
openCostS = 0;
resumeBlockedUntil = 0;
+ headStart = HEAD_START_S;
liveCtx = null;
endEarlyAt = file.endEarlyAt ?? Infinity;
trackEnd = file.trackEnd ?? CLIP_S;
voice.url = "clip.mp4";
+}
+
+/** Play the clip through, running `during` on each frame at the timeline time
+ * it is drawing. Answers what was heard. */
+async function play(
+ during?: (t: number, ctx: FakeContext) => void,
+ file: { endEarlyAt?: number; trackEnd?: number } = {}
+): Promise<Played[]> {
+ reset(file);
const mixer = new PreviewMixer();
mixer.start(0);
liveCtx = (mixer as unknown as { ctx: FakeContext }).ctx;
@@ -395,6 +406,43 @@ describe("the preview's sound over a link that only just keeps up", () => {
expect(walks.opened).toBe(2);
});
+ test("has the file open before the play begins", async () => {
+ // The failure this is here for: the picture starts on the frame and the
+ // sound comes in seconds after it, worst right after a seek. The sound
+ // opened its file once the clock was already running — a container parsed,
+ // a decoder configured, a seek into the middle of a long file — while the
+ // picture had had a decoder open on that clip the whole time it was parked
+ // there. The playhead standing over a clip is the notice the sound needs.
+ reset();
+ openCostS = 5;
+ headStart = CLIP_S; // the link is not the subject here; the open is
+ const mixer = new PreviewMixer();
+ // A page that has played once has a context; warming waits for one.
+ mixer.start(0);
+ liveCtx = (mixer as unknown as { ctx: FakeContext }).ctx;
+ mixer.stop();
+ // Parked at 0:40, six seconds of frames going by.
+ for (let i = 0; i < 160; i++) {
+ step();
+ mixer.warm(40, [voice]);
+ await settle();
+ }
+ played = [];
+ const pressed = ctxTime;
+ mixer.start(40);
+ for (let i = 0; i < 40; i++) {
+ step();
+ mixer.update(Math.min(mixer.now(), CLIP_S), [voice]);
+ await settle();
+ }
+ mixer.dispose();
+ expect(played.length).toBeGreaterThan(0);
+ // Sound within a couple of frames of the press, not a cold open later.
+ expect(Math.min(...played.map((p) => p.from)) - pressed).toBeLessThan(0.5);
+ // The file it read was the one it already had open.
+ expect(walks.opened).toBe(1);
+ });
+
test("keeps its readers when the clock jumps", async () => {
// A context that sat suspended — an output device switched, a tab the
// browser put down — comes back with its clock seconds behind the wall,
diff --git a/site/src/cut/lib/previewMixer.ts b/site/src/cut/lib/previewMixer.ts
index a4d78397..416b4cc5 100644
--- a/site/src/cut/lib/previewMixer.ts
+++ b/site/src/cut/lib/previewMixer.ts
@@ -361,14 +361,16 @@ export class PreviewMixer {
start(t: number): void {
const ctx = this.context();
void ctx.resume();
- this.stopVoices();
+ this.restart(t);
this.anchor = { timeline: t, ctx: ctx.currentTime, wall: performance.now() };
}
- /** Stop the clock and silence everything. */
+ /** Stop the clock and silence everything, keeping each voice's reader: the
+ * playhead is parked over the same clips it was playing, and the next play
+ * is usually from where this one stopped. */
stop(): void {
this.anchor = null;
- this.stopVoices();
+ for (const live of this.voices.values()) this.stopWindows(live);
}
/** The whole-project fade, over the finished mix. */
@@ -522,6 +524,78 @@ export class PreviewMixer {
}
}
+ /**
+ * Get the sound ready for a play that has not started.
+ *
+ * Opening a reader is most of a second on a long file and several on a slow
+ * link — a container parsed, a decoder configured, a seek into the middle of
+ * it — and the sound used to do all of that after the play began. The picture
+ * never did: the clip under a parked playhead has had a decoder open on it
+ * all along, which is why the picture starts on the frame and the sound came
+ * in seconds later, worst right after a seek.
+ *
+ * The playhead standing over a clip is all the notice the sound needs. Each
+ * audible voice opens its file and takes one read at the moment it is parked
+ * on, so the play finds an open file aimed where it is about to read. Nothing
+ * is scheduled and nothing is heard; the context is left alone until a play
+ * asks for it, which is also why this waits for one — a page that has not
+ * played yet has no context to warm into.
+ */
+ warm(t: number, voices: Voice[]): void {
+ if (this.running || !this.ctx) return;
+ const seen = new Set<string>();
+ for (const v of voices) {
+ seen.add(v.id);
+ const prev = this.voices.get(v.id);
+ if (
+ prev &&
+ (prev.url !== v.url ||
+ prev.start !== v.start ||
+ prev.in !== v.in ||
+ prev.out !== v.out ||
+ prev.speed !== v.speed)
+ ) {
+ this.release(prev);
+ this.voices.delete(v.id);
+ }
+ const live = this.voices.get(v.id) ?? this.open(v, t);
+ if (!live) continue;
+ live.gain.gain.value = 0;
+ const from = Math.max(v.start, t);
+ if (live.scheduled !== from) this.reaim(live, from);
+ const source = live.in + (from - live.start) * live.speed;
+ // Already reading, at the end of its source, or standing close enough
+ // that the play's first read is the one it was going to make anyway.
+ if (live.filling || source >= live.out - 1e-3) continue;
+ if (live.walk && live.walk.position >= source && live.walk.position - source <= 1) continue;
+ void this.prime(live, source);
+ }
+ for (const [id, live] of this.voices) {
+ if (seen.has(id)) continue;
+ this.release(live);
+ this.voices.delete(id);
+ }
+ }
+
+ /** One read taken to open the file and put the decoder on the moment the
+ * playhead is parked at. What it reads is thrown away — the walk standing
+ * there is the whole point — and a failure is left for the play to meet, so
+ * a parked playhead never reports anything about a link. */
+ private async prime(live: LiveVoice, source: number): Promise<void> {
+ const gen = live.gen;
+ live.filling = true;
+ live.fillAt = performance.now();
+ try {
+ const walk = (live.walk ??= openAudioWalk(live.url, source, live.out));
+ if (Math.abs(walk.position - source) > 0.02) walk.seek(source);
+ await walk.next();
+ } catch {
+ if (live.gen === gen) this.moved(live);
+ } finally {
+ if (live.gen === gen) live.filling = false;
+ }
+ }
+
/** Drop the decode cache for a source that has been replaced. */
private forget(url: string): void {
for (const key of [...this.decoded.keys()]) {
@@ -531,6 +605,7 @@ export class PreviewMixer {
dispose(): void {
this.stop();
+ this.stopVoices();
this.teardownFx();
this.fxKey = "";
this.decoded.clear();
@@ -695,6 +770,10 @@ export class PreviewMixer {
current: () => boolean
): Promise<AudioBuffer | null> {
const walk = (live.walk ??= openAudioWalk(live.url, sourceFrom, live.out));
+ // A walk outlives the read that opened it, so where it stands is its own
+ // to say. Anything that moved it — a re-aim, a priming read taken before
+ // the play began — is answered here rather than trusted.
+ if (Math.abs(walk.position - sourceFrom) > 0.02) walk.seek(sourceFrom);
const parts: WrappedAudioBuffer[] = [];
let end = sourceFrom;
// Bounded by the sound it holds and by the reads that go into it: a track