Skip to content

Open source CapCut alternative.

TypeScript

fix(site): the sound joins a standing start beside the picture

David Ledad625cf4de7
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index a7de6b41..086ed3da 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -58,7 +58,7 @@ Decoders live in a bounded pool, and leaving the pool is suspension before it is
Two things follow from that. A cut is a read from a different ring rather than a handover between players, so crossing a join costs nothing and needs no clock to change hands; and the picture answers a moved playhead inside a frame, because the answer was already in memory. Sound is scheduled on one audio context and its clock is what playback runs on — the picture follows the sound rather than the two chasing each other. When a browser holds that context suspended, as it does until the page has been interacted with, wall time carries the cut and the audio clock is picked up once it agrees — and when the clock jumps instead of agreeing, a context resumed late or an output device switched mid-play, every scheduled window restarts where the clock now stands rather than playing out the moment it was placed for. A context that stops running mid-play is asked back every frame, because the picture would otherwise carry on against wall time over a cut gone silent.
-Each voice reads its file the way the picture's readers do: a walk held open, pulled forward about a second at a time. Reading in order is what keeps sound and picture in the same part of the file. A voice asking for a long span at once is asking for the whole interleaved stretch that span is scattered through — twenty seconds of sound is a hundred megabytes of a phone recording — so over a link that merely keeps up with playback its every read arrives after the moment it was for, and a cut that sounded for its first stretch plays out in silence. The rule about endings is the picture's rule, for the same reason: a walk reports the same ending whether it read the track's last sample or gave up on it, and an ending believed a few seconds into a long clip plays the rest of the cut in silence while the picture reads happily past it. So an ending short of the clip is treated as a walk that failed and another goes after the sound it did not reach, and a walk that gets no further than the one before it has found where the track really stops — which is often a fraction before the picture does. Nothing else about a read is final either. A read that fails, comes back empty, or stops answering leaves the voice waiting a moment and asking again, quickly at first and then on a slow cadence for as long as the failure holds, and it asks the link keeper for a fresh URL on the way — the same footing the picture's readers are on, which is what stops a blip from silencing a clip whose picture healed seconds later. Opening a walk is nearly all a walk costs — a container read out of a hundred megabytes over a link — so a voice holding sound for a moment that has gone by re-aims the reader it already has, the way the picture's readers are re-aimed. Giving the file up there would start the open again a second behind where it just was, and again after that: the voice pays the open for the rest of the play and never once catches the clock, which is the same silence arrived at from the other direction. The reader moves for a new address too — a library clip copied into the project behind the editor moves off the URL it arrived on mid-play — and the sound already read plays out across the change. A reader is opened before it is needed, on the same footing as the picture's: a clip under a parked playhead has had a decoder open on it all along, which is why the picture starts on the frame, and the sound now opens its file and takes one read where the playhead stands rather than doing all of it once the clock is already running. That is what a play straight after a seek used to wait seconds for.
+Each voice reads its file the way the picture's readers do: a walk held open, pulled forward about a second at a time. Reading in order is what keeps sound and picture in the same part of the file. A voice asking for a long span at once is asking for the whole interleaved stretch that span is scattered through — twenty seconds of sound is a hundred megabytes of a phone recording — so over a link that merely keeps up with playback its every read arrives after the moment it was for, and a cut that sounded for its first stretch plays out in silence. The rule about endings is the picture's rule, for the same reason: a walk reports the same ending whether it read the track's last sample or gave up on it, and an ending believed a few seconds into a long clip plays the rest of the cut in silence while the picture reads happily past it. So an ending short of the clip is treated as a walk that failed and another goes after the sound it did not reach, and a walk that gets no further than the one before it has found where the track really stops — which is often a fraction before the picture does. Nothing else about a read is final either. A read that fails, comes back empty, or stops answering leaves the voice waiting a moment and asking again, quickly at first and then on a slow cadence for as long as the failure holds, and it asks the link keeper for a fresh URL on the way — the same footing the picture's readers are on, which is what stops a blip from silencing a clip whose picture healed seconds later. Opening a walk is nearly all a walk costs — a container read out of a hundred megabytes over a link — so a voice holding sound for a moment that has gone by re-aims the reader it already has, the way the picture's readers are re-aimed. Giving the file up there would start the open again a second behind where it just was, and again after that: the voice pays the open for the rest of the play and never once catches the clock, which is the same silence arrived at from the other direction. The reader moves for a new address too — a library clip copied into the project behind the editor moves off the URL it arrived on mid-play — and the sound already read plays out across the change. A reader is opened before it is needed, on the same footing as the picture's: a clip under a parked playhead has had a decoder open on it all along, which is why the picture starts on the frame, and the sound now opens its file and takes one read where the playhead stands rather than doing all of it once the clock is already running. That is what a play straight after a seek used to wait seconds for. The first window a standing start schedules — a play begun, a seek landed, a picture-hold released — is a short one, so the sound joins the picture's first frame and the windows behind it extend the lead by whole groups; and a play held for a clip still opening primes its sound during the hold, so the picture and the sound open together and the release finds both ready.
What is on screen at a given instant is still a pure function of the document and a time, and it is still the same function an export asks. The preview reaches an instant by playing toward it and an export by jumping straight there; because both draw through one compositor from one plan, the file cannot quietly disagree with what was watched.
diff --git a/site/src/cut/lib/previewMixer.test.ts b/site/src/cut/lib/previewMixer.test.ts
index 1d72e7b6..04faa7ee 100644
--- a/site/src/cut/lib/previewMixer.test.ts
+++ b/site/src/cut/lib/previewMixer.test.ts
@@ -406,8 +406,10 @@ describe("the preview's sound over a link that only just keeps up", () => {
voice.url = "landed.mp4";
});
// What the voice had already read plays out across the swap. It is the
- // reader that moves, and it moves while that sound is playing.
- expect(heard(sound, 18, 22)).toBeGreaterThan(0.95);
+ // reader that moves, and it moves while that sound is playing. The lead in
+ // hand at the move is at least the lookahead less the group a fill was
+ // mid-extension on, so the second past it is the guaranteed part.
+ expect(heard(sound, 18, 21)).toBeGreaterThan(0.95);
expect(heard(sound, 30, CLIP_S)).toBeGreaterThan(0.95);
// The new address is read by a walk of its own; the old one is let go.
expect(walks.opened).toBe(2);
@@ -436,6 +438,114 @@ describe("the preview's sound over a link that only just keeps up", () => {
expect(heard(sound, 27, 40)).toBeGreaterThan(0.9);
});
+ test("a play's first sound is not a whole group away", async () => {
+ // The failure this is here for: the picture opens on a frame it already
+ // holds while the sound assembles a whole group before scheduling its
+ // first window, so a play or a landed seek starts with the picture moving
+ // and the sound a group-read behind it. From a standing start the first
+ // window is short, and the sound joins the first frame.
+ reset();
+ headStart = 0.6; // the head of the file is here; a whole group is not
+ const mixer = new PreviewMixer();
+ mixer.start(0);
+ liveCtx = (mixer as unknown as { ctx: FakeContext }).ctx;
+ for (let i = 0; i < 30; i++) {
+ step();
+ mixer.update(Math.min(mixer.now(), CLIP_S), [voice]);
+ await settle();
+ }
+ mixer.dispose();
+ expect(played.length).toBeGreaterThan(0);
+ expect(Math.min(...played.map((p) => p.from))).toBeLessThan(0.4);
+ expect(walks.opened).toBe(1);
+ });
+
+ test("a seek keeps its reader and lands with its sound", async () => {
+ // The failure this is here for: picking a position mid-play re-anchors the
+ // clock, and a voice that gave up its walk there paid a fresh open — a
+ // container parsed out of a long file — before the first window after the
+ // seek, so the picture jumped and the sound came in seconds later.
+ reset();
+ headStart = CLIP_S; // the link is not the subject; the reader is
+ const mixer = new PreviewMixer();
+ mixer.start(0);
+ liveCtx = (mixer as unknown as { ctx: FakeContext }).ctx;
+ for (let i = 0; i < 20; i++) {
+ step();
+ mixer.update(Math.min(mixer.now(), CLIP_S), [voice]);
+ await settle();
+ }
+ // Opening a file costs seconds from here on; a voice that keeps its
+ // reader never pays it.
+ openCostS = 5;
+ played = [];
+ mixer.start(10);
+ const pressed = ctxTime;
+ for (let i = 0; i < 20; i++) {
+ step();
+ mixer.update(Math.min(mixer.now(), CLIP_S), [voice]);
+ await settle();
+ }
+ mixer.dispose();
+ expect(played.length).toBeGreaterThan(0);
+ expect(Math.min(...played.map((p) => p.from)) - pressed).toBeLessThan(0.5);
+ // The file it read after the seek was the one it already had open.
+ expect(walks.opened).toBe(1);
+ });
+
+ test("a hold pays the sound's open before it releases", async () => {
+ // A play held for a picture still opening used to leave the sound waiting
+ // too: nothing read during the hold, and the release paid the file's open
+ // before the first window. The hold primes each voice at the held moment,
+ // so the open runs beside the picture's and the release finds both ready.
+ reset();
+ headStart = CLIP_S;
+ const mixer = new PreviewMixer();
+ mixer.start(0);
+ liveCtx = (mixer as unknown as { ctx: FakeContext }).ctx;
+ for (let i = 0; i < 20; i++) {
+ step();
+ mixer.update(Math.min(mixer.now(), CLIP_S), [voice]);
+ await settle();
+ }
+ // What an outage leaves behind: the reader given up, the voice backed
+ // off on the long cadence.
+ const live = (
+ mixer as unknown as {
+ voices: Map<string, { walk: { close(): void } | null; retryAt: number }>;
+ }
+ ).voices.get("clip")!;
+ live.walk?.close();
+ live.walk = null;
+ live.retryAt = performance.now() + 10_000;
+ const opens = walks.opened;
+ // The picture stalls; opening the sound's file costs seconds now.
+ openCostS = 3;
+ played = [];
+ const holdUntil = wall + 4;
+ let first = true;
+ while (wall < holdUntil) {
+ step();
+ mixer.hold(Math.min(mixer.now(), CLIP_S));
+ await settle();
+ // Entering the hold is what primes: the walk opens there, during it.
+ if (first) {
+ first = false;
+ expect(walks.opened).toBe(opens + 1);
+ }
+ }
+ expect(played.length).toBe(0); // a hold schedules nothing
+ const released = ctxTime;
+ for (let i = 0; i < 20; i++) {
+ step();
+ mixer.update(Math.min(mixer.now(), CLIP_S), [voice]);
+ await settle();
+ }
+ mixer.dispose();
+ expect(played.length).toBeGreaterThan(0);
+ expect(Math.min(...played.map((p) => p.from)) - released).toBeLessThan(0.5);
+ });
+
test("has the file open before the play begins", async () => {
// The failure this is here for: the picture starts on the frame and the
// sound comes in seconds after it, worst right after a seek. The sound
diff --git a/site/src/cut/lib/previewMixer.ts b/site/src/cut/lib/previewMixer.ts
index 4756a5db..ab12186a 100644
--- a/site/src/cut/lib/previewMixer.ts
+++ b/site/src/cut/lib/previewMixer.ts
@@ -45,6 +45,14 @@ import { timeStretch } from "./timeStretch";
* short enough that the bytes it waits on are the ones the picture is reading
* anyway. */
const GROUP_S = 1;
+/** The first window scheduled from a standing start — a play begun, a seek
+ * landed, a hold released. The picture opens on a frame already decoded, and a
+ * sound that waits for a whole group joins it a group-read late; a short first
+ * window puts the sound beside the first frame, and the fills behind it extend
+ * the lead by whole groups. A link that cannot feed the second window in time
+ * starts it where the clock stands, the way any late window starts — a beat of
+ * sound and a catch-up beats a silence the length of the group. */
+const FIRST_GROUP_S = 0.3;
/** Most reads one group is assembled from. A packet of compressed audio holds
* a fraction of a second, so a group is tens of reads, and this is the ceiling
* over them. */
@@ -388,7 +396,14 @@ export class PreviewMixer {
this.heldAt = t;
for (const live of this.voices.values()) {
this.stopWindows(live);
- this.reaim(live, Math.max(live.start, t));
+ const from = Math.max(live.start, t);
+ this.reaim(live, from);
+ // The held moment's sound comes down while the picture opens: one
+ // read aims the walk and pulls the bytes it sits in — the primer a
+ // parked playhead uses — so the release finds an open file already
+ // read where the first window starts.
+ const source = live.in + (from - live.start) * live.speed;
+ if (!live.filling && source < live.out - 1e-3) void this.prime(live, source);
}
}
// Both clocks are pulled back to where the hold began, every frame of it:
@@ -624,6 +639,13 @@ export class PreviewMixer {
const walk = (live.walk ??= openAudioWalk(live.url, source, live.out));
if (Math.abs(walk.position - source) > 0.02) walk.seek(source);
await walk.next();
+ // The read answering is the source answering: a voice backed off by an
+ // outage is willing again the moment its file reads, so the play that
+ // follows is never left waiting out a cadence the outage booked.
+ if (live.gen === gen) {
+ live.attempts = 0;
+ live.retryAt = 0;
+ }
} catch {
if (live.gen === gen) this.moved(live);
} finally {
@@ -723,10 +745,14 @@ export class PreviewMixer {
}
let buffer: AudioBuffer | null;
try {
+ // Nothing scheduled means the sound is standing still — the play just
+ // began, a seek landed, a hold let go — and the picture is not
+ // waiting. The first window is short so the sound joins it now.
+ const group = live.windows.length ? GROUP_S : FIRST_GROUP_S;
buffer =
Math.abs(live.speed - 1) > 1e-3
? await this.stretchedWindow(live, sourceFrom)
- : await this.walkGroup(live, sourceFrom, current);
+ : await this.walkGroup(live, sourceFrom, current, group);
} catch {
// The read is the only part of this that says anything about the
// source, so it is the only part a failure is spent on.
@@ -802,7 +828,8 @@ export class PreviewMixer {
private async walkGroup(
live: LiveVoice,
sourceFrom: number,
- current: () => boolean
+ current: () => boolean,
+ group: number
): Promise<AudioBuffer | null> {
const walk = (live.walk ??= openAudioWalk(live.url, sourceFrom, live.out));
// A walk outlives the read that opened it, so where it stands is its own
@@ -814,7 +841,7 @@ export class PreviewMixer {
// Bounded by the sound it holds and by the reads that go into it: a track
// answering with buffers that carry no time would otherwise be pulled for
// as long as the frame lasted.
- for (let pulls = 0; end - sourceFrom < GROUP_S && pulls < GROUP_PULLS; pulls++) {
+ for (let pulls = 0; end - sourceFrom < group && pulls < GROUP_PULLS; pulls++) {
const part = await walk.next();
if (!current()) return null;
if (!part) break;