Open source CapCut alternative.
TypeScript
docs(cut): what keeps the picture with the sound
David Led0a6a28f6262
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.mdindex 6da2f6e2..5954739d 100644--- a/docs/guides/cut/README.md+++ b/docs/guides/cut/README.md@@ -46,6 +46,12 @@ The client probes the engine's health endpoint (dedicated port first, dev serverThe preview decodes the cut itself. Every clip on screen owns a decoder reading its file directly — the container is parsed in the page and frames come off WebCodecs — and a small ring of already-decoded frames sits beside it. Drawing a frame is a question answered from that ring, never a wait: playing walks each file forward and keeps the ring ahead of the playhead, scrubbing asks for one moment and the newest ask wins, and if the exact frame has not landed the nearest one already decoded goes up and is replaced when the real one arrives. A held frame is worth more than a stall.+Holding on to a walk is what makes that cheap, and it matters most exactly when it is hardest. A walk that has not yet reached its first frame is doing the only work that can produce a picture — finding the keyframe before the moment asked for, fetching the bytes it sits in, and decoding forward to it — so nothing replaces it while it is coming, however far the playhead has gone past it in the meantime. A walk that has landed frames and then been outrun is re-aimed, and it is aimed ahead of the playhead by what reaching a frame has been costing: anchored where the playhead stands it would land its first frame at a moment already gone by and be outrun again immediately, which is the creep-freeze-jump people report as choppiness. Aiming ahead gives up the span in between, and that span was never going to arrive in time anyway — the sound is going there regardless, and a picture that arrives with it beats one that arrives behind it.++The other rule is about walks that stop. A walk reports the same ending whether it read the file's last frame or gave up on it, and a reader whose bytes never came, or whose decoder was closed under it, ends the same way. Believed, an ending a second into a long clip pins the picture on the frame before it and nothing asks for another, so the sound plays the rest of the cut over one still. So an ending short of the film is treated as a walk that failed, and another goes after the film it did not reach. Three things keep that from running away: a walk that gets no further than a previous one has found where the film really stops and is believed, a walk that died does not aim ahead again, since dying at that anchor has disproved the aim, and a source with two dead walks behind it waits a beat before sending the next. Once an ending is believed, moments past it go to the single-frame fetch — a finished walk cannot be crept along, and a reader that keeps asking it gets no frame and no repaint to ask again on, which is a drag position that never paints and a parked playhead that never goes quiet. A cloud project reads its media over the network in resident chunks, so a slow link puts a reader in these positions for real.++Some machines cannot decode the footage as fast as it plays, and no reader can invent frames that the decoder has not produced. Left alone that play never recovers: the sound goes on, the picture falls further behind with every second, and a minute in it is half a minute stale. So a preview that has stayed behind its sound for a few seconds decodes smaller from then on, and the picture catches up at a lower resolution. It is a trade taken once per play and never taken back — every open decoder is abandoned when the size changes and each source pays a fresh keyframe seek, so undoing it would cost more than the pixels are worth. Two things are excluded from it. The opening seconds of a play, where every machine trails while nothing is decoded yet and files are being opened, and a reader whose walk is waiting on bytes: decoding smaller does nothing for a picture that is late because the link has not delivered it, and the preview would end up softer for the rest of the session having gained nothing. The floor is 360, below which there is no picture worth keeping in sync.+Decoders live in a bounded pool, and leaving the pool is suspension before it is closure: an idle source gives up its decoder but keeps the parsed file and the canvases its frames land on, so the next visit wakes it for the cost of one keyframe seek. The canvases are the part that must not churn — a montage of short clips replayed for a few minutes, allocating a fresh set on every reopen, floods the GPU process until frames stop arriving at all. They also must not pile up, so the warm shelf of stood-down sources is capped by the memory it holds rather than by a count of them: a small stage keeps a montage's whole cast, a large one keeps a few, and what does not fit closes for real.Two things follow from that. A cut is a read from a different ring rather than a handover between players, so crossing a join costs nothing and needs no clock to change hands; and the picture answers a moved playhead inside a frame, because the answer was already in memory. Sound is scheduled on one audio context, a window at a time, and its clock is what playback runs on — the picture follows the sound rather than the two chasing each other. When a browser holds that context suspended, as it does until the page has been interacted with, wall time carries the cut and the audio clock is picked up once it agrees — and when the clock jumps instead of agreeing, a context resumed late or an output device switched mid-play, every scheduled window restarts where the clock now stands rather than playing out the moment it was placed for.@@ -58,9 +64,9 @@ The playhead lives apart from the document. It moves sixty times a second and thWhether any of this is true of a given build is measured rather than assumed: `npm run eval:cut-perf` drives a real editor in a real browser against fixture montages and reports how long the picture takes to answer a moved playhead, whether any frame crossing a cut was the wrong one, and whether a fast-cut montage replayed for minutes still holds the picture to the clock the way its first pass did. Its committed report is the baseline a change is diffed against.-The machine it measures on is part of the measurement. A developer desktop has decode slots to spare and never falls behind, so the preview's caps never bind there and a change that would ruin a laptop reads as a clean sheet. `--machine` puts the page on a smaller one: it takes away hardware decode slots, so decoders past the limit deliver their frames late the way software decoding does, and it slows the main thread to a fraction of this one. `npm run eval:cut-perf-lowend` is the profile the reports come from. What the gates ask there is what a viewer would ask — did the frames arrive, did the picture hitch a quarter second behind the sound, and was the last pass as good as the first — and each machine has its own budget for those, because a number that is honest on a desktop is fantasy on a fifteen-watt laptop.+The machine it measures on is part of the measurement. A developer desktop has decode slots to spare and never falls behind, so the preview's caps never bind there and a change that would ruin a laptop reads as a clean sheet. `--machine` puts the page on a smaller one: it takes away hardware decode slots, so decoders past the limit deliver their frames late the way software decoding does, it slows the main thread to a fraction of this one, and it paces the media over a link the way a cloud project reads it. That last one is its own axis: a reader short of bytes and a reader short of decode both show a picture drifting behind the sound, each takes its own fix, and `--net` and `--rtt` open the link up on a failing case to say which it was. The fixtures cover the shape that binding actually happens in — one long clip on the timeline, read the way a cloud project reads it, played cold and played again after the editor has been closed and reopened. `npm run eval:cut-perf-lowend` is the profile the reports come from. What the gates ask there is what a viewer would ask — did the frames arrive, did the picture hitch a quarter second behind the sound, and was the last pass as good as the first — and each machine has its own budget for those, because a number that is honest on a desktop is fantasy on a fifteen-watt laptop.-A profile it cannot model is the one someone is actually editing on, so the preview can report on itself. Settings carries a diagnostics switch, off until an account turns it on; with it on, the engine counts what its played frames did — how many arrived late, how far the picture sat behind its own sound, what the decoders and the warm shelf were holding, how long the main thread was blocked — and sends one summary per half minute of playback, alongside what the machine brought: cores, memory, the size it is decoding at, and which codecs it decodes in hardware. That last one separates the two stutters that look identical from outside: a machine spending its time after the decoder, and one whose decoder refused the footage and put every stream on the CPU. The summaries are numbers; no frame, no sound, and nothing naming the project leaves the browser.+A profile it cannot model is the one someone is actually editing on, so the preview can report on itself. Settings carries a diagnostics switch, off until an account turns it on; with it on, the engine counts what its played frames did — how many arrived late, how far the picture sat behind its own sound, what the decoders and the warm shelf were holding, how long the main thread was blocked — and sends one summary per half minute of playback, alongside what the machine brought: cores, memory, the size it is decoding at, and which codecs it decodes in hardware. That last one separates two stutters that look identical from outside: a machine spending its time after the decoder, and one whose decoder refused the footage and put every stream on the CPU. A summary also carries how many walks the window started and the worst wait for a single frame off one, which separates the third: a reader whose bytes had not arrived waits seconds on one frame and nothing on the next, where a decoder that is merely slow answers every one of them in tens of milliseconds. The summaries are numbers; no frame, no sound, and nothing naming the project leaves the browser.## Color grading