Skip to content

Open source CapCut alternative.

TypeScript

fix(cut): keep media reads going when decoding fails [rebuild]

David Le29b4b0ef7a4f
diff --git a/docs/guides/cut/local-compute.md b/docs/guides/cut/local-compute.md
index 1fc4e545..88f106f3 100644
--- a/docs/guides/cut/local-compute.md
+++ b/docs/guides/cut/local-compute.md
@@ -55,7 +55,7 @@ Imports and the assistant's conversion tool prepare playable H.264/AAC media thr
## ProRes masters and preview proxies
-ProRes is the one format kept as shot. The master stays in the project and every export reads it; beside it lands a 10-bit HEVC preview proxy carried into Rec.709 and tagged so every decoder draws its values untouched, while the asset's color record still describes the master. The proxy is built where the project lives: the engine on a Mac, the worker in the cloud, and in a browser the page itself when it can encode 10-bit frames, else the Mac. Until it lands the preview decodes the master through a WASM decoder, the one fallback, and the Color panel says so. A library video gets the same proxy on its shelf when a card shows no picture; cards and the viewer play it, and projects still copy the master.
+ProRes is kept as shot. Exports read the master; the preview reads a 10-bit HEVC proxy in Rec.709, and the color record describes the master. The engine builds it on a Mac, the worker in the cloud, a browser page that encodes 10-bit, else the Mac; until then the preview decodes in WASM and the Color panel says so. A browser that cannot decode H.264 or AAC gets a compat proxy in that slot, VP9 or Opus, from the engine or the worker; its preview, strip and waveform read it, and its exports render from the master in ffmpeg. A library card with no picture gets a proxy on its shelf.
## Rules
diff --git a/site/patches/README.md b/site/patches/README.md
index 8a88b337..a9aa380a 100644
--- a/site/patches/README.md
+++ b/site/patches/README.md
@@ -8,17 +8,15 @@ comes with a look at whether the patch is still needed.
## mediabunny: decoded frames in flight
mediabunny's decode pump throttles on WebCodecs' `decodeQueueSize`, which
-drains as soon as a packet reaches the platform decoder, and it starts every
-stream with a 40-packet head start. Frames the decoder is still holding count
-as nothing, so a stream whose reader pauses — the Cut preview with its ring
-full — ends up with about 40 decoded full-size frames waiting in mediabunny's
-sample queue. At 4K that is around a gigabyte per stream, held in the decoder
-process (VTDecoderXPCService on a Mac, the GPU process elsewhere).
+drains as soon as a packet reaches the platform decoder, with a 40-packet head
+start. A stream whose reader pauses — the preview with its ring full — holds
+about 40 decoded frames, around a gigabyte per stream at 4K.
The patch gives `VideoDecoderWrapper` an `inFlight` count (packets submitted
minus frames output, reset on flush), reports it as the decoder's queue size,
-and lowers the head start to 16. The same six edits land in the ES module and
-both bundles.
+and lowers the head start to 16. A read that ends in an error wakes the parked
+pump, so the failed decoder closes and frees its frames. The edits land in the
+ES module and all three bundles.
Delete this patch when upstream's pump counts frames in flight itself: check
`computeMaxQueueSize` and the pump loops around `getDecodeQueueSize()` in
diff --git a/site/patches/mediabunny+1.61.0.patch b/site/patches/mediabunny+1.61.0.patch
index d36f5962..62c1ab90 100644
--- a/site/patches/mediabunny+1.61.0.patch
+++ b/site/patches/mediabunny+1.61.0.patch
@@ -1,5 +1,5 @@
diff --git a/node_modules/mediabunny/dist/bundles/mediabunny.cjs b/node_modules/mediabunny/dist/bundles/mediabunny.cjs
-index f984c84..9c9d93a 100644
+index f984c84..aa1fb83 100644
--- a/node_modules/mediabunny/dist/bundles/mediabunny.cjs
+++ b/node_modules/mediabunny/dist/bundles/mediabunny.cjs
@@ -12381,6 +12381,21 @@ var Mediabunny = (() => {
@@ -24,7 +24,46 @@ index f984c84..9c9d93a 100644
}
}
return;
-@@ -24644,11 +24659,13 @@ var Mediabunny = (() => {
+@@ -24437,6 +24452,9 @@ var Mediabunny = (() => {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ } else if (terminated) {
+ return { value: void 0, done: true };
+@@ -24444,6 +24462,9 @@ var Mediabunny = (() => {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ } else if (sampleQueue.length > 0) {
+ const value = sampleQueue.shift();
+@@ -24608,12 +24629,18 @@ var Mediabunny = (() => {
+ if (track.input._disposed) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ } else if (terminated) {
+ return { value: void 0, done: true };
+ } else if (hasOutOfBandError) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ } else if (sampleQueue.length > 0) {
+ const value = sampleQueue.shift();
+@@ -24644,11 +24671,13 @@ var Mediabunny = (() => {
}
};
var computeMaxQueueSize = (decodedSampleQueueSize) => {
@@ -40,7 +79,7 @@ index f984c84..9c9d93a 100644
this.codec = codec;
this.decoderConfig = decoderConfig;
this.rotation = rotation;
-@@ -24658,6 +24675,13 @@ var Mediabunny = (() => {
+@@ -24658,6 +24687,13 @@ var Mediabunny = (() => {
this.customDecoder = null;
this.customDecoderCallSerializer = new CallSerializer();
this.customDecoderQueueSize = 0;
@@ -54,7 +93,7 @@ index f984c84..9c9d93a 100644
this.inputTimestamps = [];
// Timestamps input into the decoder, sorted.
this.sampleQueue = [];
-@@ -24689,6 +24713,17 @@ var Mediabunny = (() => {
+@@ -24689,6 +24725,17 @@ var Mediabunny = (() => {
if (!(sample instanceof VideoSample)) {
throw new TypeError("The argument passed to onSample must be a VideoSample.");
}
@@ -72,7 +111,7 @@ index f984c84..9c9d93a 100644
this.finalizeAndEmitSample(sample);
};
this.customDecoder.onError = (error) => {
-@@ -24744,6 +24779,7 @@ var Mediabunny = (() => {
+@@ -24744,6 +24791,7 @@ var Mediabunny = (() => {
const stack = new Error("Decoding error").stack;
this.decoder = new VideoDecoder({
output: (frame) => {
@@ -80,7 +119,7 @@ index f984c84..9c9d93a 100644
try {
colorHandler(frame);
} catch (error) {
-@@ -24760,13 +24796,10 @@ var Mediabunny = (() => {
+@@ -24760,13 +24808,10 @@ var Mediabunny = (() => {
}
getDecodeQueueSize() {
if (this.customDecoder) {
@@ -96,7 +135,7 @@ index f984c84..9c9d93a 100644
}
}
decode(packet) {
-@@ -24824,6 +24857,13 @@ var Mediabunny = (() => {
+@@ -24824,6 +24869,13 @@ var Mediabunny = (() => {
}
}
}
@@ -110,7 +149,7 @@ index f984c84..9c9d93a 100644
this.decoder.decode(packet.toEncodedVideoChunk());
this.decodeAlphaData(packet);
}
-@@ -24940,6 +24980,49 @@ var Mediabunny = (() => {
+@@ -24940,6 +24992,49 @@ var Mediabunny = (() => {
sample.setFlip(this.flip);
this.onSample(sample);
}
@@ -160,7 +199,7 @@ index f984c84..9c9d93a 100644
async mergeAlpha(color, alpha) {
const resolver = promiseWithResolvers();
this.mergeAlphaPromises.push(resolver.promise);
-@@ -24968,6 +25051,11 @@ var Mediabunny = (() => {
+@@ -24968,6 +25063,11 @@ var Mediabunny = (() => {
async flush() {
if (this.customDecoder) {
await this.customDecoderCallSerializer.call(() => this.customDecoder.flush());
@@ -172,7 +211,7 @@ index f984c84..9c9d93a 100644
} else {
assert(this.decoder);
await Promise.all([
-@@ -24992,6 +25080,7 @@ var Mediabunny = (() => {
+@@ -24992,6 +25092,7 @@ var Mediabunny = (() => {
}
this.sampleQueue.length = 0;
}
@@ -180,7 +219,7 @@ index f984c84..9c9d93a 100644
this.currentPacketIndex = 0;
this.raslSkipped = false;
}
-@@ -25248,6 +25337,18 @@ var Mediabunny = (() => {
+@@ -25248,6 +25349,18 @@ var Mediabunny = (() => {
if (decoderOptions.optimizeForLatency !== void 0 && typeof decoderOptions.optimizeForLatency !== "boolean") {
throw new TypeError("decoderOptions.optimizeForLatency, when provided, must be a boolean.");
}
@@ -199,7 +238,7 @@ index f984c84..9c9d93a 100644
};
var VideoSampleSink = class extends BaseMediaSampleSink {
/** Creates a new {@link VideoSampleSink} for the given {@link InputVideoTrack}. */
-@@ -25276,12 +25377,19 @@ var Mediabunny = (() => {
+@@ -25276,12 +25389,19 @@ var Mediabunny = (() => {
let decoderConfig = await this._track.getDecoderConfig();
const timeResolution = await this._track.getTimeResolution();
assert(codec && decoderConfig);
@@ -222,7 +261,7 @@ index f984c84..9c9d93a 100644
/** @internal */
_createPacketSink() {
diff --git a/node_modules/mediabunny/dist/bundles/mediabunny.mjs b/node_modules/mediabunny/dist/bundles/mediabunny.mjs
-index e675b1f..5a114ed 100644
+index e675b1f..693fc15 100644
--- a/node_modules/mediabunny/dist/bundles/mediabunny.mjs
+++ b/node_modules/mediabunny/dist/bundles/mediabunny.mjs
@@ -12224,6 +12224,21 @@ var Mp3Demuxer = class extends Demuxer {
@@ -247,7 +286,46 @@ index e675b1f..5a114ed 100644
}
}
return;
-@@ -24487,11 +24502,13 @@ var BaseMediaSampleSink = class {
+@@ -24280,6 +24295,9 @@ var BaseMediaSampleSink = class {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ } else if (terminated) {
+ return { value: void 0, done: true };
+@@ -24287,6 +24305,9 @@ var BaseMediaSampleSink = class {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ } else if (sampleQueue.length > 0) {
+ const value = sampleQueue.shift();
+@@ -24451,12 +24472,18 @@ var BaseMediaSampleSink = class {
+ if (track.input._disposed) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ } else if (terminated) {
+ return { value: void 0, done: true };
+ } else if (hasOutOfBandError) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ } else if (sampleQueue.length > 0) {
+ const value = sampleQueue.shift();
+@@ -24487,11 +24514,13 @@ var BaseMediaSampleSink = class {
}
};
var computeMaxQueueSize = (decodedSampleQueueSize) => {
@@ -263,7 +341,7 @@ index e675b1f..5a114ed 100644
this.codec = codec;
this.decoderConfig = decoderConfig;
this.rotation = rotation;
-@@ -24501,6 +24518,13 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24501,6 +24530,13 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
this.customDecoder = null;
this.customDecoderCallSerializer = new CallSerializer();
this.customDecoderQueueSize = 0;
@@ -277,7 +355,7 @@ index e675b1f..5a114ed 100644
this.inputTimestamps = [];
// Timestamps input into the decoder, sorted.
this.sampleQueue = [];
-@@ -24532,6 +24556,17 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24532,6 +24568,17 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
if (!(sample instanceof VideoSample)) {
throw new TypeError("The argument passed to onSample must be a VideoSample.");
}
@@ -295,7 +373,7 @@ index e675b1f..5a114ed 100644
this.finalizeAndEmitSample(sample);
};
this.customDecoder.onError = (error) => {
-@@ -24587,6 +24622,7 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24587,6 +24634,7 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
const stack = new Error("Decoding error").stack;
this.decoder = new VideoDecoder({
output: (frame) => {
@@ -303,7 +381,7 @@ index e675b1f..5a114ed 100644
try {
colorHandler(frame);
} catch (error) {
-@@ -24603,13 +24639,10 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24603,13 +24651,10 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
}
getDecodeQueueSize() {
if (this.customDecoder) {
@@ -319,7 +397,7 @@ index e675b1f..5a114ed 100644
}
}
decode(packet) {
-@@ -24667,6 +24700,13 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24667,6 +24712,13 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
}
}
}
@@ -333,7 +411,7 @@ index e675b1f..5a114ed 100644
this.decoder.decode(packet.toEncodedVideoChunk());
this.decodeAlphaData(packet);
}
-@@ -24783,6 +24823,49 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24783,6 +24835,49 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
sample.setFlip(this.flip);
this.onSample(sample);
}
@@ -383,7 +461,7 @@ index e675b1f..5a114ed 100644
async mergeAlpha(color, alpha) {
const resolver = promiseWithResolvers();
this.mergeAlphaPromises.push(resolver.promise);
-@@ -24811,6 +24894,11 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24811,6 +24906,11 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
async flush() {
if (this.customDecoder) {
await this.customDecoderCallSerializer.call(() => this.customDecoder.flush());
@@ -395,7 +473,7 @@ index e675b1f..5a114ed 100644
} else {
assert(this.decoder);
await Promise.all([
-@@ -24835,6 +24923,7 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
+@@ -24835,6 +24935,7 @@ var VideoDecoderWrapper = class extends DecoderWrapper {
}
this.sampleQueue.length = 0;
}
@@ -403,7 +481,7 @@ index e675b1f..5a114ed 100644
this.currentPacketIndex = 0;
this.raslSkipped = false;
}
-@@ -25091,6 +25180,18 @@ var validateVideoSinkDecoderOptions = (decoderOptions) => {
+@@ -25091,6 +25192,18 @@ var validateVideoSinkDecoderOptions = (decoderOptions) => {
if (decoderOptions.optimizeForLatency !== void 0 && typeof decoderOptions.optimizeForLatency !== "boolean") {
throw new TypeError("decoderOptions.optimizeForLatency, when provided, must be a boolean.");
}
@@ -422,7 +500,7 @@ index e675b1f..5a114ed 100644
};
var VideoSampleSink = class extends BaseMediaSampleSink {
/** Creates a new {@link VideoSampleSink} for the given {@link InputVideoTrack}. */
-@@ -25119,12 +25220,19 @@ var VideoSampleSink = class extends BaseMediaSampleSink {
+@@ -25119,12 +25232,19 @@ var VideoSampleSink = class extends BaseMediaSampleSink {
let decoderConfig = await this._track.getDecoderConfig();
const timeResolution = await this._track.getTimeResolution();
assert(codec && decoderConfig);
@@ -445,7 +523,7 @@ index e675b1f..5a114ed 100644
/** @internal */
_createPacketSink() {
diff --git a/node_modules/mediabunny/dist/bundles/mediabunny.node.cjs b/node_modules/mediabunny/dist/bundles/mediabunny.node.cjs
-index c828e19..d0af8d1 100644
+index c828e19..a1c28cc 100644
--- a/node_modules/mediabunny/dist/bundles/mediabunny.node.cjs
+++ b/node_modules/mediabunny/dist/bundles/mediabunny.node.cjs
@@ -12378,6 +12378,21 @@ var Mediabunny = (() => {
@@ -470,7 +548,46 @@ index c828e19..d0af8d1 100644
}
}
return;
-@@ -24650,8 +24665,10 @@ var Mediabunny = (() => {
+@@ -24440,6 +24455,9 @@ var Mediabunny = (() => {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ } else if (terminated) {
+ return { value: void 0, done: true };
+@@ -24447,6 +24465,9 @@ var Mediabunny = (() => {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ } else if (sampleQueue.length > 0) {
+ const value = sampleQueue.shift();
+@@ -24611,12 +24632,18 @@ var Mediabunny = (() => {
+ if (track.input._disposed) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ } else if (terminated) {
+ return { value: void 0, done: true };
+ } else if (hasOutOfBandError) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ } else if (sampleQueue.length > 0) {
+ const value = sampleQueue.shift();
+@@ -24650,8 +24677,10 @@ var Mediabunny = (() => {
return decodedSampleQueueSize === 0 ? 40 : 8;
};
var VideoDecoderWrapper = class extends DecoderWrapper {
@@ -482,7 +599,7 @@ index c828e19..d0af8d1 100644
this.codec = codec;
this.decoderConfig = decoderConfig;
this.rotation = rotation;
-@@ -24661,6 +24678,12 @@ var Mediabunny = (() => {
+@@ -24661,6 +24690,12 @@ var Mediabunny = (() => {
this.customDecoder = null;
this.customDecoderCallSerializer = new CallSerializer();
this.customDecoderQueueSize = 0;
@@ -495,7 +612,7 @@ index c828e19..d0af8d1 100644
this.inputTimestamps = [];
// Timestamps input into the decoder, sorted.
this.sampleQueue = [];
-@@ -24692,6 +24715,17 @@ var Mediabunny = (() => {
+@@ -24692,6 +24727,17 @@ var Mediabunny = (() => {
if (!(sample instanceof VideoSample)) {
throw new TypeError("The argument passed to onSample must be a VideoSample.");
}
@@ -513,7 +630,7 @@ index c828e19..d0af8d1 100644
this.finalizeAndEmitSample(sample);
};
this.customDecoder.onError = (error) => {
-@@ -24763,7 +24797,7 @@ var Mediabunny = (() => {
+@@ -24763,7 +24809,7 @@ var Mediabunny = (() => {
}
getDecodeQueueSize() {
if (this.customDecoder) {
@@ -522,7 +639,7 @@ index c828e19..d0af8d1 100644
} else {
assert(this.decoder);
return Math.max(
-@@ -24827,6 +24861,12 @@ var Mediabunny = (() => {
+@@ -24827,6 +24873,12 @@ var Mediabunny = (() => {
}
}
}
@@ -535,7 +652,7 @@ index c828e19..d0af8d1 100644
this.decoder.decode(packet.toEncodedVideoChunk());
this.decodeAlphaData(packet);
}
-@@ -24943,6 +24983,49 @@ var Mediabunny = (() => {
+@@ -24943,6 +24995,49 @@ var Mediabunny = (() => {
sample.setFlip(this.flip);
this.onSample(sample);
}
@@ -585,7 +702,7 @@ index c828e19..d0af8d1 100644
async mergeAlpha(color, alpha) {
const resolver = promiseWithResolvers();
this.mergeAlphaPromises.push(resolver.promise);
-@@ -24971,6 +25054,11 @@ var Mediabunny = (() => {
+@@ -24971,6 +25066,11 @@ var Mediabunny = (() => {
async flush() {
if (this.customDecoder) {
await this.customDecoderCallSerializer.call(() => this.customDecoder.flush());
@@ -597,7 +714,7 @@ index c828e19..d0af8d1 100644
} else {
assert(this.decoder);
await Promise.all([
-@@ -25251,6 +25339,18 @@ var Mediabunny = (() => {
+@@ -25251,6 +25351,18 @@ var Mediabunny = (() => {
if (decoderOptions.optimizeForLatency !== void 0 && typeof decoderOptions.optimizeForLatency !== "boolean") {
throw new TypeError("decoderOptions.optimizeForLatency, when provided, must be a boolean.");
}
@@ -616,7 +733,7 @@ index c828e19..d0af8d1 100644
};
var VideoSampleSink = class extends BaseMediaSampleSink {
/** Creates a new {@link VideoSampleSink} for the given {@link InputVideoTrack}. */
-@@ -25279,12 +25379,19 @@ var Mediabunny = (() => {
+@@ -25279,12 +25391,19 @@ var Mediabunny = (() => {
let decoderConfig = await this._track.getDecoderConfig();
const timeResolution = await this._track.getTimeResolution();
assert(codec && decoderConfig);
@@ -691,7 +808,7 @@ index 056c51c..2463224 100644
/**
* A sink that retrieves decoded video samples (video frames) from a video track.
diff --git a/node_modules/mediabunny/dist/modules/src/media-sink.js b/node_modules/mediabunny/dist/modules/src/media-sink.js
-index 9465e3b..4af78f2 100644
+index 9465e3b..b814d42 100644
--- a/node_modules/mediabunny/dist/modules/src/media-sink.js
+++ b/node_modules/mediabunny/dist/modules/src/media-sink.js
@@ -13,7 +13,7 @@ import { InputAudioTrack, InputTrack, InputVideoTrack } from './input-track.js';
@@ -703,7 +820,47 @@ index 9465e3b..4af78f2 100644
const validatePacketRetrievalOptions = (options) => {
if (!options || typeof options !== 'object') {
throw new TypeError('options must be an object.');
-@@ -662,14 +662,17 @@ export class BaseMediaSampleSink {
+@@ -425,6 +425,9 @@ export class BaseMediaSampleSink {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ }
+ else if (terminated) {
+@@ -434,6 +437,9 @@ export class BaseMediaSampleSink {
+ terminated = true;
+ ended = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ }
+ else if (sampleQueue.length > 0) {
+@@ -621,6 +627,9 @@ export class BaseMediaSampleSink {
+ // nothing can ever close.
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw new InputDisposedError();
+ }
+ else if (terminated) {
+@@ -629,6 +638,9 @@ export class BaseMediaSampleSink {
+ else if (hasOutOfBandError) {
+ terminated = true;
+ closeSamples();
++ // Wake a pump parked on a full queue (Donkey patch): it sees the end, unwinds, and closes the
++ // decoder, which closes the frames it still holds (WebKit's reorder queue).
++ onQueueDequeue();
+ throw outOfBandError;
+ }
+ else if (sampleQueue.length > 0) {
+@@ -662,14 +674,17 @@ export class BaseMediaSampleSink {
}
}
const computeMaxQueueSize = (decodedSampleQueueSize) => {
@@ -726,7 +883,7 @@ index 9465e3b..4af78f2 100644
this.codec = codec;
this.decoderConfig = decoderConfig;
this.rotation = rotation;
-@@ -679,6 +682,13 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -679,6 +694,13 @@ class VideoDecoderWrapper extends DecoderWrapper {
this.customDecoder = null;
this.customDecoderCallSerializer = new CallSerializer();
this.customDecoderQueueSize = 0;
@@ -740,7 +897,7 @@ index 9465e3b..4af78f2 100644
this.inputTimestamps = []; // Timestamps input into the decoder, sorted.
this.sampleQueue = []; // Safari-specific thing, check usage.
this.currentPacketIndex = 0;
-@@ -710,6 +720,20 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -710,6 +732,20 @@ class VideoDecoderWrapper extends DecoderWrapper {
if (!(sample instanceof VideoSample)) {
throw new TypeError('The argument passed to onSample must be a VideoSample.');
}
@@ -761,7 +918,7 @@ index 9465e3b..4af78f2 100644
this.finalizeAndEmitSample(sample);
};
// @ts-expect-error It's technically readonly
-@@ -780,6 +804,7 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -780,6 +816,7 @@ class VideoDecoderWrapper extends DecoderWrapper {
const stack = new Error('Decoding error').stack;
this.decoder = new VideoDecoder({
output: (frame) => {
@@ -769,7 +926,7 @@ index 9465e3b..4af78f2 100644
try {
colorHandler(frame);
}
-@@ -797,11 +822,11 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -797,11 +834,11 @@ class VideoDecoderWrapper extends DecoderWrapper {
}
getDecodeQueueSize() {
if (this.customDecoder) {
@@ -783,7 +940,7 @@ index 9465e3b..4af78f2 100644
}
}
decode(packet) {
-@@ -873,6 +898,13 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -873,6 +910,13 @@ class VideoDecoderWrapper extends DecoderWrapper {
}
}
}
@@ -797,7 +954,7 @@ index 9465e3b..4af78f2 100644
this.decoder.decode(packet.toEncodedVideoChunk());
this.decodeAlphaData(packet);
}
-@@ -1020,6 +1052,49 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -1020,6 +1064,49 @@ class VideoDecoderWrapper extends DecoderWrapper {
sample.setFlip(this.flip);
this.onSample(sample);
}
@@ -847,7 +1004,7 @@ index 9465e3b..4af78f2 100644
async mergeAlpha(color, alpha) {
const resolver = promiseWithResolvers();
this.mergeAlphaPromises.push(resolver.promise);
-@@ -1056,6 +1131,11 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -1056,6 +1143,11 @@ class VideoDecoderWrapper extends DecoderWrapper {
async flush() {
if (this.customDecoder) {
await this.customDecoderCallSerializer.call(() => this.customDecoder.flush());
@@ -859,7 +1016,7 @@ index 9465e3b..4af78f2 100644
}
else {
assert(this.decoder);
-@@ -1081,6 +1161,7 @@ class VideoDecoderWrapper extends DecoderWrapper {
+@@ -1081,6 +1173,7 @@ class VideoDecoderWrapper extends DecoderWrapper {
}
this.sampleQueue.length = 0;
}
@@ -867,7 +1024,7 @@ index 9465e3b..4af78f2 100644
this.currentPacketIndex = 0;
this.raslSkipped = false;
}
-@@ -1358,6 +1439,18 @@ const validateVideoSinkDecoderOptions = (decoderOptions) => {
+@@ -1358,6 +1451,18 @@ const validateVideoSinkDecoderOptions = (decoderOptions) => {
if (decoderOptions.optimizeForLatency !== undefined && typeof decoderOptions.optimizeForLatency !== 'boolean') {
throw new TypeError('decoderOptions.optimizeForLatency, when provided, must be a boolean.');
}
@@ -886,7 +1043,7 @@ index 9465e3b..4af78f2 100644
};
/**
* A sink that retrieves decoded video samples (video frames) from a video track.
-@@ -1390,12 +1483,19 @@ export class VideoSampleSink extends BaseMediaSampleSink {
+@@ -1390,12 +1495,19 @@ export class VideoSampleSink extends BaseMediaSampleSink {
let decoderConfig = await this._track.getDecoderConfig();
const timeResolution = await this._track.getTimeResolution();
assert(codec && decoderConfig);
diff --git a/site/src/cut/components/ColorPanel.tsx b/site/src/cut/components/ColorPanel.tsx
index adf1c45c..2a29609f 100644
--- a/site/src/cut/components/ColorPanel.tsx
+++ b/site/src/cut/components/ColorPanel.tsx
@@ -40,7 +40,7 @@ import { sampleClipRawFrame, sourceProfileOf, toBaseRendering } from "@/cut/lib/
import { useEditor } from "@/cut/lib/store";
import { usePanelState, useRememberedScroll } from "@/cut/lib/panelState";
import type { MediaAsset, VideoClip } from "@/cut/lib/types";
-import { needsProxy, useProxyJobs } from "@/cut/lib/mediaProxy";
+import { useProxyJobs } from "@/cut/lib/mediaProxy";
import {
cachedLut,
libraryLutId,
@@ -209,7 +209,7 @@ function SourceColorRow({ clip }: { clip: VideoClip }) {
function ProxyLine({ asset }: { asset: MediaAsset }) {
const job = useProxyJobs((s) => s.jobs[asset.id]);
// A job only exists for a file that needed a proxy where it was asked.
- if (!job || !needsProxy(asset, true)) return null;
+ if (!job) return null;
const master = asset.color?.codec?.startsWith("ap") ? "the ProRes master" : "the original file";
const text =
job.kind === "making"
diff --git a/site/src/cut/components/RequireSession.tsx b/site/src/cut/components/RequireSession.tsx
index a7f474f4..d06662a4 100644
--- a/site/src/cut/components/RequireSession.tsx
+++ b/site/src/cut/components/RequireSession.tsx
@@ -11,6 +11,7 @@ import { bindCutMediaCache } from "@/cut/lib/chunkCache";
import { useEffect, type ReactNode } from "react";
import { useEnvironment } from "@/cut/lib/environment";
import { EmbeddedSignInFallback } from "@/cut/components/EmbeddedSignInFallback";
+import { onOperationFailure } from "@/cut/lib/operationFailure";
import { authHrefFor } from "@/app/_components/landing/useAppEntryHref";
import {
@@ -31,7 +32,7 @@ import { useAccountConfig } from "@/queries/accountConfig";
// It binds the account id; SessionGate, one level in, is what holds the app
// until the id lands, because every engine URL carries it.
export function RequireSession({ children }: { children: ReactNode }) {
- const { data: session, isPending } = authClient.useSession();
+ const { data: session, isPending, refetch } = authClient.useSession();
const signedOut = !isPending && !session;
const userId = session?.user.id;
@@ -40,6 +41,21 @@ export function RequireSession({ children }: { children: ReactNode }) {
// a browser that dropped it renews it through the card's tool bridge.
const { chatgpt: embedded } = useEnvironment();
+ // A request refused for want of a session — the framed editor's cookie aged
+ // out mid-edit, or a sign-out in another tab — asks for the session again,
+ // so the gate below offers the reconnect or the sign-in. Refusals arrive in
+ // bursts, one per request in flight; one ask answers them all.
+ useEffect(() => {
+ let asking = false;
+ return onOperationFailure((failure) => {
+ if (failure.code !== "authentication_required" || asking) return;
+ asking = true;
+ void refetch().finally(() => {
+ asking = false;
+ });
+ });
+ }, [refetch]);
+
useAppLoaded("cut", session?.user);
// The account's configuration — public settings after overrides and
diff --git a/site/src/cut/components/SidePanel.tools.ts b/site/src/cut/components/SidePanel.tools.ts
index 2f7d8bb8..c8500521 100644
--- a/site/src/cut/components/SidePanel.tools.ts
+++ b/site/src/cut/components/SidePanel.tools.ts
@@ -8,6 +8,7 @@
*/
import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+import { CONVERT_MODES } from "@/cut/lib/convertMode";
import { SIDE_PANEL_TABS } from "@/cut/lib/types";
export const SIDE_PANEL_TOOLS = [
@@ -35,7 +36,7 @@ export const SIDE_PANEL_TOOLS = [
{
name: "convert_media",
description:
- "Convert project media to MP4 — H.264 picture, AAC sound — replacing each asset's file in place, so clips already cut from it keep playing. This is the fix for footage that previews black or refuses to play: a phone's HEVC .mov, a screen recorder's ProRes, a camera file with raw PCM sound. It is also the whole answer to \"turn these into mp4\": the user drops the files on this chat, you convert them, and each converted file is on its own card right here for them to download or drag in — nothing to go looking for. A file already in Media stays in Media. Streams that already fit the container are copied packet for packet, so a .mov of H.264 converts in seconds and loses nothing, and a file that is already an MP4 of those streams comes back untouched. `max_height` shrinks the picture at the same time (1080 for a 4K source that only needs to be a timeline clip). Conversion runs automatically through the project backend. A browser missing a required codec uses hosted preparation and saves the result back into its own project.",
+ "Convert project media to MP4 — H.264 picture, AAC sound — replacing each asset's file in place, so clips already cut from it keep playing. This is the fix for footage that previews black or refuses to play: a phone's HEVC .mov, a screen recorder's ProRes, a camera file with raw PCM sound. It is also the whole answer to \"turn these into mp4\": the user drops the files on this chat, you convert them, and each converted file is on its own card right here for them to download or drag in — nothing to go looking for. A file already in Media stays in Media. Streams that already fit the container are copied packet for packet, so a .mov of H.264 converts in seconds and loses nothing, and a file that is already an MP4 of those streams comes back untouched. `mode` \"reencode\" rewrites the picture even when it fits: the fix for H.264 that stays black after a plain conversion. `max_height` shrinks the picture at the same time (1080 for a 4K source that only needs to be a timeline clip). Conversion runs automatically through the project backend. A browser missing a required codec uses hosted preparation and saves the result back into its own project.",
inputSchema: obj({
asset_ids: {
type: "array",
@@ -43,6 +44,11 @@ export const SIDE_PANEL_TOOLS = [
description: "Project asset ids from `media`, converted in order",
},
max_height: num("Cap the output height in pixels (e.g. 1080); omit to keep the source size"),
+ mode: {
+ type: "string",
+ enum: [...CONVERT_MODES],
+ description: "fit (default) copies streams that already fit; reencode rewrites the picture from what the converter decodes",
+ },
}, ["asset_ids"]),
},
{
diff --git a/site/src/cut/components/Timeline.tsx b/site/src/cut/components/Timeline.tsx
index d3712f34..00d1f870 100644
--- a/site/src/cut/components/Timeline.tsx
+++ b/site/src/cut/components/Timeline.tsx
@@ -54,7 +54,7 @@ import { keyboardToEditor, shortcutDecline } from "@/cut/lib/shortcutGate";
import { additiveClick } from "@/cut/lib/hostKeys";
import { CLIP_GAP, landOnRow, laneDragFor, laneDragParts, laneDragSlots, resolveRow, startLaneMove, startLaneTrim, type LaneDrag, type RowBox } from "@/cut/lib/laneTracks";
import { reportSwallowed } from "@/cut/lib/report";
-import { downloadMedia, dropEdgeFrames, ensurePeaks, importImage, importStockAudio, importStockVideo, peekEdgeFrame, requestEdgeFrame, revealMedia, stripFailedFor, subscribeStripStatus } from "@/cut/lib/media";
+import { downloadMedia, dropEdgeFrames, ensurePeaks, importImage, importStockAudio, importStockVideo, peekEdgeFrame, requestEdgeFrame, missingMediaFor, revealMedia, stripFailedFor, subscribeStripStatus } from "@/cut/lib/media";
import { planFilmstrip, type FilmTile } from "@/cut/lib/filmstrip";
import { waveGain } from "@/cut/lib/waveform";
import { canSplitItem } from "@/cut/lib/itemKinds";
@@ -1739,7 +1739,7 @@ export function Timeline() {
}
placeAssetAt(asset.id, asset.type, next(asset, sticker), audioRow, place, elementRow);
} catch (err) {
- reportSwallowed(`[cut] drop failed for ${item.name}`, err);
+ reportSwallowed("[cut] drop failed", err, item.name);
}
}
};
@@ -3914,6 +3914,7 @@ function ClipView({
<div className="pointer-events-none absolute inset-x-0 top-0" style={{ height: FILM_H }}>
<Filmstrip frames={filmstrip} grade={clip.grade} failed={stripFailed} hidden={offscreen} block={asset?.block} />
</div>
+ <MissingChip assetId={asset.id} />
{clip.grade?.lut && (
// The LUT badge: the file mark, small, where the clip's chrome sits.
// eslint-disable-next-line @next/next/no-img-element
@@ -5005,6 +5006,19 @@ function LoadingChip() {
);
}
+/** Corner chip for a clip whose cloud file is not there (media.ts
+ * missingMediaFor): an upload that never finished, or a file deleted from
+ * under the project. The clip keeps its place and edits; nothing plays. */
+function MissingChip({ assetId }: { assetId: string }) {
+ const missing = useSyncExternalStore(subscribeStripStatus, () => missingMediaFor(assetId), () => false);
+ if (!missing) return null;
+ return (
+ <span className="tl-missing-chip pointer-events-none absolute right-1 bottom-1 z-4 rounded-[5px] bg-black/65 px-1.5 py-px text-[10px] whitespace-nowrap text-white">
+ Missing media
+ </span>
+ );
+}
+
/** Hover chip that toggles a clip's own audio. Stays visible while the clip is
* muted so unmuting is one click. */
function MuteChip({
@@ -5427,6 +5441,7 @@ function AudioView({
</span>
)}
{loading && <LoadingChip />}
+ <MissingChip assetId={asset.id} />
{!loading && <ClipMenu asset={asset} clip={clip} />}
{!loading && (
<MuteChip
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index 7b3e1343..83393865 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -198,6 +198,7 @@ import {
scanSourceSpeech,
watchGeometry,
} from "./media";
+import { convertModeIn } from "./convertMode";
import { convertAssetToMp4 } from "./mediaConvert";
import { isLottieAsset } from "./lottieAssets";
import { AIR, MIN_AIR, placeEdge, REACH, settleJoint, type Pace, type PlacedEdge } from "./cutRefine";
@@ -5260,6 +5261,8 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
const ids = Array.isArray(input.asset_ids) ? input.asset_ids.map((v) => String(v)) : [];
if (ids.length === 0) throw new ToolError("Pass the asset ids to convert.");
const maxHeight = isNum(input.max_height) ? Math.round(input.max_height) : undefined;
+ const mode = convertModeIn(input.mode);
+ if (!mode) throw new ToolError("mode is fit or reencode.");
// Named `assets` so each converted file renders as its own card in the
// thread (ToolOutputAssets keys off assetId) — the user drops files on
// the chat and takes the MP4s back from the same place.
@@ -5273,7 +5276,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
continue;
}
try {
- const out = await convertAssetToMp4(projectId, asset, { maxHeight });
+ const out = await convertAssetToMp4(projectId, asset, { maxHeight, mode });
converted.push({
assetId: out.assetId,
// The card now reads `name`; `from` is what the user dropped.
diff --git a/site/src/cut/lib/backend/browser/library.ts b/site/src/cut/lib/backend/browser/library.ts
index 13d5502a..a17f81fd 100644
--- a/site/src/cut/lib/backend/browser/library.ts
+++ b/site/src/cut/lib/backend/browser/library.ts
@@ -299,14 +299,18 @@ async function removeAsset(id: string): Promise<Response> {
*/
async function makeProxy(req: Request, id: string): Promise<Response> {
try {
- const { maxHeight, crf } = (await req.json()) as { maxHeight?: number; crf?: number };
+ const { maxHeight, crf, compat: asked } = (await req.json()) as { maxHeight?: number; crf?: number; compat?: unknown };
if (!Number.isFinite(maxHeight) || !Number.isFinite(crf)) return err("maxHeight and crf are required.", 400);
+ const { compatCoverIn, compatCovers, compatProxyNameFor } = await import("../../proxyNames");
+ const compat = compatCoverIn(asked);
const asset = (await readIndex()).assets.find((a) => a.id === id);
if (!asset) return err("Library asset not found.", 404);
if (asset.type !== "video") return err("Only a video takes a playable copy.", 400);
const dir = await store.libraryMediaDir(true);
if (!dir) return err("Browser storage is unavailable.", 500);
- if (asset.proxyFile && (await store.readFileAt(dir, asset.proxyFile))) return json(asset);
+ const held = asset.proxyFile;
+ const answers = !!held && (!compat || compatCovers(held, compat));
+ if (held && answers && (await store.readFileAt(dir, held))) return json(asset);
const master = await store.readFileAt(dir, asset.fileName);
if (!master) return err("Media file not found.", 404);
@@ -315,7 +319,16 @@ async function makeProxy(req: Request, id: string): Promise<Response> {
const { probeMediaColor } = await import("../../mediaRead");
const { code } = await probeMediaColor(master);
const from = { matrix: code.matrix, fullRange: code.fullRange };
- let fileName = (await encodeProxyInPage(master, from, dir, asset.fileName, maxHeight!, () => {}))?.fileName;
+ let fileName: string | undefined;
+ if (compat) {
+ // A picture this browser cannot decode has nothing to encode from in
+ // the page; the worker builds the compat copy by way of the cloud shelf.
+ const { compatInTheCloud } = await import("../../cloudLibraryUpload");
+ const blob = await compatInTheCloud(new File([master], asset.fileName), compat);
+ fileName = await store.saveLibraryMedia(blob, compatProxyNameFor(asset.fileName, compat, false));
+ } else {
+ fileName = (await encodeProxyInPage(master, from, dir, asset.fileName, maxHeight!, () => {}))?.fileName;
+ }
if (!fileName) {
const blob = await proxyOnTheMac(async () => master, asset.fileName, maxHeight!, crf!);
if (!blob) return err("This browser can't make a playable copy of this video. Open the Donkey app to make one.", 422);
@@ -331,6 +344,8 @@ async function makeProxy(req: Request, id: string): Promise<Response> {
return live;
});
await register(landed);
+ // The copy this browser could not play leaves with its name.
+ if (held && held !== landed) await dropFiles([held]);
return json(updated);
} catch (e) {
await dropFiles([landed]);
diff --git a/site/src/cut/lib/baseFrame.test.ts b/site/src/cut/lib/baseFrame.test.ts
new file mode 100644
index 00000000..494b738c
--- /dev/null
+++ b/site/src/cut/lib/baseFrame.test.ts
@@ -0,0 +1,63 @@
+import { afterAll, describe, expect, test } from "bun:test";
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { baseFramesAt } from "./baseFrame";
+import { registerBlobFile } from "./backend/browser/registry";
+import { setFrameSinkFactory, type FrameSinkOptions } from "./mediaRead";
+
+// A plain H.264 + AAC clip: the starter project's own footage, the file the
+// reported decoder failures came from.
+const CLIP = join(dirname(fileURLToPath(import.meta.url)), "../../../public/cut-starter/media/ai-watercolor-and-loose-black-ink-sketch-il-1.mp4");
+const url = registerBlobFile("test/base-frames.mp4", new File([readFileSync(CLIP)], "clip.mp4", { type: "video/mp4" }));
+
+const frameAt = (t: number) => ({ canvas: {}, timestamp: t, duration: 1 / 24 });
+
+/** A decoder whose hardware session dies after `good` frames, the way a
+ * machine's decoder says only "Decoding error."; software always decodes. */
+function dyingHardware(good: number, error: () => Error, opened: FrameSinkOptions[]) {
+ setFrameSinkFactory((_track, _size, opts) => {
+ opened.push(opts ?? {});
+ return {
+ getCanvas: async (t: number) => frameAt(t),
+ canvases: async function* () {},
+ canvasesAtTimestamps: async function* (times: Iterable<number> | AsyncIterable<number>) {
+ let n = 0;
+ for await (const t of times) {
+ if (!opts?.software && n++ >= good) throw error();
+ yield frameAt(t);
+ }
+ },
+ } as never;
+ });
+}
+
+afterAll(() => setFrameSinkFactory(null));
+
+describe("baseFramesAt", () => {
+ test("a decoder failure goes on in software from the frame it failed on", async () => {
+ const opened: FrameSinkOptions[] = [];
+ dyingHardware(1, () => new DOMException("Decoding error.", "EncodingError"), opened);
+
+ const got: number[] = [];
+ for await (const frame of baseFramesAt(url, [0, 1, 2])) got.push(frame!.timestamp);
+
+ expect(got).toEqual([0, 1, 2]);
+ expect(opened.map((o) => !!o.software)).toEqual([false, true]);
+ });
+
+ test("a failure that is not the decoder's ends the read", async () => {
+ const opened: FrameSinkOptions[] = [];
+ dyingHardware(0, () => new Error("network gone"), opened);
+
+ const read = async () => {
+ const frames = baseFramesAt(url, [0, 1]);
+ while (!(await frames.next()).done) {
+ // drain
+ }
+ };
+
+ await expect(read()).rejects.toThrow("network gone");
+ expect(opened).toHaveLength(1);
+ });
+});
diff --git a/site/src/cut/lib/baseFrame.ts b/site/src/cut/lib/baseFrame.ts
index e66fabbe..8cf21711 100644
--- a/site/src/cut/lib/baseFrame.ts
+++ b/site/src/cut/lib/baseFrame.ts
@@ -23,7 +23,7 @@ import {
type ParsedLut,
type SourceProfile,
} from "@donkeycut/effects-kit";
-import { frameSink, openMedia, videoTrackOf, type FrameCanvasSink, type FrameSize, type FrameSinkOptions } from "./mediaRead";
+import { coveredSrc, frameSink, openMedia, videoTrackOf, type FrameCanvasSink, type FrameSize, type FrameSinkOptions } from "./mediaRead";
import { sampleClipColorRead, sampleClipFrameData } from "./previewCanvas";
import { colorRead, fileAt, previewFile, type ColorRead } from "./sourceColor";
import { useEditor } from "./store";
@@ -147,7 +147,10 @@ export function baseFrameSink(
}
/** One base-rendering frame of the file at `url`. */
-export async function baseFrameAt(url: string, time: number, size?: FrameSize): Promise<WrappedCanvas | null> {
+export async function baseFrameAt(master: string, time: number, size?: FrameSize): Promise<WrappedCanvas | null> {
+ // A stream this page cannot decode reads from its compat proxy, and the
+ // proxy's own recipe draws it.
+ const url = coveredSrc("video", master);
const input = openMedia(url);
try {
const track = await videoTrackOf(input);
@@ -160,7 +163,10 @@ export async function baseFrameAt(url: string, time: number, size?: FrameSize):
/** The code values of one frame of the file at `url`, read the way the
* preview reads them, with the recipe that draws them. */
-export async function rawFrameAt(url: string, time: number, size: FrameSize): Promise<RawFrame | null> {
+export async function rawFrameAt(master: string, time: number, size: FrameSize): Promise<RawFrame | null> {
+ // A stream this page cannot decode reads from its compat proxy, and the
+ // proxy's own recipe draws it.
+ const url = coveredSrc("video", master);
const input = openMedia(url);
try {
const track = await videoTrackOf(input);
@@ -180,14 +186,36 @@ export async function rawFrameAt(url: string, time: number, size: FrameSize): Pr
}
}
+// The DOMException name WebCodecs gives a decoder that failed on its input.
+const DECODER_FAILURE = "EncodingError";
+
+const decoderFailed = (err: unknown) => err instanceof DOMException && err.name === DECODER_FAILURE;
+
/** Base-rendering frames at ascending `times`, in one decode pass; null for a
- * time the track has no frame for. */
-export async function* baseFramesAt(url: string, times: number[], size?: FrameSize): AsyncGenerator<WrappedCanvas | null> {
+ * time the track has no frame for.
+ *
+ * A machine's hardware decoder can refuse a stream every other decoder plays
+ * — Linux VA-API, an Android MediaCodec, a phone out of sessions — and says
+ * only "Decoding error.". The pass then goes on in software from the frame it
+ * failed on; a software decoder that fails too ends the read. */
+export async function* baseFramesAt(master: string, times: number[], size?: FrameSize): AsyncGenerator<WrappedCanvas | null> {
+ // A stream this page cannot decode reads from its compat proxy, and the
+ // proxy's own recipe draws it.
+ const url = coveredSrc("video", master);
const input = openMedia(url);
try {
const track = await videoTrackOf(input);
if (!track) throw new Error("This file has no readable video.");
- yield* baseFrameSink(track, size, colorReadAt(url)).canvasesAtTimestamps(times);
+ let next = 0;
+ try {
+ for await (const frame of baseFrameSink(track, size, colorReadAt(url)).canvasesAtTimestamps(times)) {
+ next++;
+ yield frame;
+ }
+ } catch (err) {
+ if (!decoderFailed(err)) throw err;
+ yield* baseFrameSink(track, size, colorReadAt(url), { software: true }).canvasesAtTimestamps(times.slice(next));
+ }
} finally {
input.dispose();
}
diff --git a/site/src/cut/lib/chunkCache.ts b/site/src/cut/lib/chunkCache.ts
index 30b30c33..020defe5 100644
--- a/site/src/cut/lib/chunkCache.ts
+++ b/site/src/cut/lib/chunkCache.ts
@@ -53,6 +53,7 @@ import {
} from "./backend/browser/opfs";
import { CUT_MEDIA_ORIGIN } from "./hosts";
import { allowance, holdMemory } from "./memoryBudget";
+import { observeOperationResponse } from "./operationFailure";
import { SETTINGS, type Settings } from "@/lib/config/registry";
export const CHUNK_SIZE = 2 * 1024 * 1024;
@@ -370,13 +371,41 @@ async function doOpen(id: ChunkIdentity, url: string): Promise<ObjState> {
if (!state.size) throw new Error("Cloud media answered with no length.");
state.probe0 = await collect(res, 0, Math.min(CHUNK_SIZE, state.size) - 1, true);
} else {
- throw new Error(`Cloud media read failed (${res.status}).`);
+ throw new CloudMediaError(res.status);
}
void persistChunk(state, 0, state.probe0);
void commitMeta(state);
return state;
}
+/** A cloud media read the server refused, with the status it answered. */
+export class CloudMediaError extends Error {
+ constructor(readonly status: number) {
+ super(`Cloud media read failed (${status}).`);
+ this.name = "CloudMediaError";
+ }
+}
+
+// The answers that mean the bytes are not there: the route has no record of
+// the file, or the media origin holds no object under its key.
+const GONE_STATUSES = new Set([404, 410]);
+
+/** Whether `err`, or a cause behind it, is a cloud read of a file that is not
+ * there — an upload that never finished, or a file deleted from under the doc.
+ * Asking again finds the same nothing. */
+export function cloudMediaGone(err: unknown): boolean {
+ return GONE_STATUSES.has(cloudMediaStatus(err) ?? 0);
+}
+
+/** The status a refused cloud read answered, found through the cause chain,
+ * or null when `err` is some other failure. */
+export function cloudMediaStatus(err: unknown): number | null {
+ for (let at = err, depth = 0; at instanceof Error && depth < 4; depth++, at = at.cause) {
+ if (at instanceof CloudMediaError) return at.status;
+ }
+ return null;
+}
+
/** `chunks/<keyHash>/<versionTag>/`, with every other version of the same key
* swept as it is opened. An object rewritten in place leaves its old chunks
* behind exactly once, and the next open of the new bytes collects them. */
@@ -645,7 +674,7 @@ function startRun(state: ObjState, first: number, last: number): Promise<Uint8Ar
const startB = first * CHUNK_SIZE;
const endB = Math.min(state.size, (last + 1) * CHUNK_SIZE) - 1;
const res = await fetchRange(state.url, startB, endB, state.ac.signal);
- if (!res.ok) throw new Error(`Cloud media read failed (${res.status}).`);
+ if (!res.ok) throw new CloudMediaError(res.status);
// Each chunk is its own array, so holding one can't pin the whole run,
// and it settles the moment its own bytes are here.
await streamRun(res, startB, endB, res.status !== 206, (i, bytes) => {
@@ -974,7 +1003,12 @@ function openThroughRoute(route: string): Promise<ObjState> {
*/
async function probeRoute(route: string): Promise<ObjState> {
const res = await fetchRange(route, 0, 0);
- if (!res.ok) throw new Error(`Cloud media read failed (${res.status}).`);
+ if (!res.ok) {
+ // A 401 here is the session ending under the page — the framed editor's
+ // partitioned cookie aging out mid-edit — which the session gate answers.
+ await observeOperationResponse(res);
+ throw new CloudMediaError(res.status);
+ }
await res.body?.cancel().catch(() => {});
const id = chunkIdentity(res.url);
if (!id) throw new Error("Cloud media route did not land on the media origin.");
diff --git a/site/src/cut/lib/cloudLibraryUpload.ts b/site/src/cut/lib/cloudLibraryUpload.ts
index bca80894..a56c62f5 100644
--- a/site/src/cut/lib/cloudLibraryUpload.ts
+++ b/site/src/cut/lib/cloudLibraryUpload.ts
@@ -4,6 +4,8 @@ import { apiJson } from "@/cut/lib/backend";
import { cloudBackend } from "@/cut/lib/backend/cloud";
import { pollCloudJob } from "@/cut/lib/cloudJob";
import { presignedUpload } from "@/cut/lib/media";
+import type { ConvertMode } from "@/cut/lib/convertMode";
+import type { CompatCover } from "@/cut/lib/proxyNames";
import type { LibraryAsset, LibrarySource } from "@/cut/lib/library";
import type { LibraryUploadStage, LibraryUploadState } from "@/cut/lib/libraryUpload";
@@ -12,6 +14,7 @@ export type CloudUploadOptions = {
source?: LibrarySource;
folderId?: string | null;
maxHeight?: number;
+ mode?: ConvertMode;
state?: LibraryUploadState;
onStage?: (stage: LibraryUploadStage) => void;
};
@@ -32,6 +35,7 @@ export async function uploadCloudLibraryMedia(file: File, options: CloudUploadOp
...(options.folderId ? { folderId: options.folderId } : {}),
...(options.source ? { source: options.source } : {}),
...(options.maxHeight ? { maxHeight: options.maxHeight } : {}),
+ ...(options.mode === "reencode" ? { mode: options.mode } : {}),
}),
});
const body = await apiJson<LibraryAsset | { jobId: string }>(res);
@@ -45,6 +49,32 @@ export async function uploadCloudLibraryMedia(file: File, options: CloudUploadOp
return { ...asset, residency: "cloud" };
}
+/** The compat proxy (proxyNames.ts) of media the page holds, made by the
+ * cloud worker: the file stages on the cloud shelf, the proxy comes back,
+ * and both leave the shelf. */
+export async function compatInTheCloud(file: File, cover: CompatCover, onProgress?: (p: number) => void): Promise<Blob> {
+ const failed = "Could not make the preview proxy.";
+ const asset = await uploadCloudLibraryMedia(file, { name: file.name });
+ try {
+ const res = await cloudBackend.fetch(`/api/cut/library/${encodeURIComponent(asset.id)}/proxy`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ compat: cover }),
+ });
+ const body = await apiJson<LibraryAsset | { jobId: string }>(res);
+ if (!res.ok) throw new Error(body.error ?? failed);
+ const landed = "jobId" in body
+ ? await pollCloudJob<LibraryAsset>(body.jobId, cloudBackend, failed, { timedOut: "The preview proxy took too long.", onProgress })
+ : body;
+ if (!landed.proxyFile) throw new Error(failed);
+ const media = await cloudBackend.fetch(`/api/cut/library/media/${encodeURIComponent(landed.proxyFile)}`);
+ if (!media.ok) throw new Error(failed);
+ return await media.blob();
+ } finally {
+ await cloudBackend.fetch(`/api/cut/library/${encodeURIComponent(asset.id)}`, { method: "DELETE" }).catch(() => {});
+ }
+}
+
/** A browser without a decoder borrows cloud preparation and takes the result home. */
export async function withCloudPreparedMedia<T>(
file: File,
diff --git a/site/src/cut/lib/codecSupport.test.ts b/site/src/cut/lib/codecSupport.test.ts
new file mode 100644
index 00000000..e09dd5ca
--- /dev/null
+++ b/site/src/cut/lib/codecSupport.test.ts
@@ -0,0 +1,47 @@
+import { describe, expect, test } from "bun:test";
+import { readCodecs, type CodecProbes } from "./codecSupport";
+
+/** A storage that keeps what it is given, like a browser's localStorage. */
+function memoryStorage(): Pick<Storage, "getItem" | "setItem"> {
+ const held = new Map<string, string>();
+ return { getItem: (k) => held.get(k) ?? null, setItem: (k, v) => void held.set(k, v) };
+}
+
+/** Probes that answer `answer` and count how often they ran. */
+function probes(answer: { h264: boolean; aac: boolean }, storage = memoryStorage(), agent = "Opera/136") {
+ const ran = { h264: 0, aac: 0 };
+ const io: CodecProbes = {
+ h264: async () => (ran.h264++, answer.h264),
+ aac: async () => (ran.aac++, answer.aac),
+ storage,
+ agent,
+ };
+ return { io, ran, storage };
+}
+
+describe("browser codecs", () => {
+ test("probes once and remembers the answer for this browser", async () => {
+ const first = probes({ h264: true, aac: false });
+ expect(await readCodecs(first.io)).toEqual({ h264: true, aac: false });
+
+ // A later session in the same browser reads the stored answer.
+ const later = probes({ h264: true, aac: true }, first.storage);
+ expect(await readCodecs(later.io)).toEqual({ h264: true, aac: false });
+ expect(later.ran).toEqual({ h264: 0, aac: 0 });
+ });
+
+ test("a browser update probes again", async () => {
+ const before = probes({ h264: true, aac: false }, memoryStorage(), "Opera/136");
+ await readCodecs(before.io);
+
+ const updated = probes({ h264: true, aac: true }, before.storage, "Opera/137");
+ expect(await readCodecs(updated.io)).toEqual({ h264: true, aac: true });
+ expect(updated.ran).toEqual({ h264: 1, aac: 1 });
+ });
+
+ test("no storage still answers", async () => {
+ const { io } = probes({ h264: false, aac: true });
+ io.storage = null;
+ expect(await readCodecs(io)).toEqual({ h264: false, aac: true });
+ });
+});
diff --git a/site/src/cut/lib/codecSupport.ts b/site/src/cut/lib/codecSupport.ts
new file mode 100644
index 00000000..912b8cf9
--- /dev/null
+++ b/site/src/cut/lib/codecSupport.ts
@@ -0,0 +1,145 @@
+"use client";
+
+/**
+ * What this browser decodes of the two codecs every converter writes: H.264
+ * picture and AAC sound.
+ *
+ * Nearly every browser decodes both. Firefox on a Linux machine without the
+ * system's codecs has no H.264, and some browsers built on Chromium (Opera,
+ * Yandex) say they decode AAC and then fail on every packet — so the AAC
+ * answer comes from decoding a real tenth of a second of it, and the H.264
+ * one from the decoder's own config check. The answer is kept per browser
+ * version, so a session in a browser that decodes both pays nothing past its
+ * first. A browser missing one gets a compat proxy per clip (mediaProxy.ts).
+ */
+
+import { audioChunks, frameSinkIsCustom, withMedia } from "./mediaRead";
+import type { CompatCover } from "./proxyNames";
+import type { MediaAsset } from "./types";
+
+export interface BrowserCodecs {
+ h264: boolean;
+ aac: boolean;
+}
+
+/** The checks and the place their answer is kept, swapped by the test. */
+export interface CodecProbes {
+ h264: () => Promise<boolean>;
+ aac: () => Promise<boolean>;
+ storage: Pick<Storage, "getItem" | "setItem"> | null;
+ /** Which browser build answered: a new version asks again. */
+ agent: string;
+}
+
+const STORAGE_KEY = "cut-codecs";
+
+/** The config the H.264 check asks about: High profile at 720p, what phones
+ * and the converters write. */
+const H264_CONFIG = { codec: "avc1.64001f", codedWidth: 1280, codedHeight: 720 };
+
+/** A tenth of a second of 48 kHz stereo AAC-LC in an MP4. */
+const AAC_SAMPLE = "AAAAHGZ0eXBNNEEgAAACAE00QSBpc29taXNvMgAAAxNtb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAAZAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAACPXRyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAAZAAAAAAAAAAAAAAAAQEAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAAGQAAAQAAAEAAAAAAbVtZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAALuAAAAWwFXEAAAAAAAtaGRscgAAAAAAAAAAc291bgAAAAAAAAAAAAAAAFNvdW5kSGFuZGxlcgAAAAFgbWluZgAAABBzbWhkAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAEkc3RibAAAAGpzdHNkAAAAAAAAAAEAAABabXA0YQAAAAAAAAABAAAAAAAAAAAAAgAQAAAAALuAAAAAAAA2ZXNkcwAAAAADgICAJQABAASAgIAXQBUAAAAAAH75AAB++QWAgIAFEZBW5QAGgICAAQIAAAAgc3R0cwAAAAAAAAACAAAABQAABAAAAAABAAACwAAAABxzdHNjAAAAAAAAAAEAAAABAAAABgAAAAEAAAAsc3RzegAAAAAAAAAAAAAABgAAAFYAAAB8AAAASAAAAEcAAABIAAAARAAAABRzdGNvAAAAAAAAAAEAAAM/AAAAGnNncGQBAAAAcm9sbAAAAAIAAAAB//8AAAAcc2JncAAAAAByb2xsAAAAAQAAAAYAAAABAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY2Mi4xMi4xMDEAAAAIZnJlZQAAAfVtZGF03gIATGF2YzYyLjI4LjEwMQBCSqpYMT1q51483epqXIuwEEQUmWqLXyaGCaabLxqqq+U02Xjx48fllmii48ePyy5YouNKNVq51483epqXIuwAAAAAA4AhRWz///zFkpZGWSllI3+vXHj/X9/Or/+P3869v/p+POhtTs265jMGqk6rO5pb0Up0yVlaLDsfy2X/lsEUaAht6ZBIKJ53XWmI5qsRCGJnyiZ8omEqHLCfgPOLOBITMDRo0a3+vXHj/X9/Or/+P3869v/p+POgAAAAAAHAIXWUyVmT+P6fr1xx54nU001egf+G0vufHxmbd9/cH+Pgyff3A+PgD394R8fGYe/vCDzI2ZP4/p+vXHHnidTTTV6AAAAAAAOAIRWUAUxG57/4//i/8v9/94Xql555KEUUS16DGiiiiii7oooiKKJCKon0IqQgqOqAGznv/j/+L/y/3/3heqXnnkoAAAAAABwhFZTyWZe/9f9PNru9S5JcQET4Bw/n6vEI2zA+xg+IjbMD4GD4iMsYHwUT4iOkYHwUWxEc8NmXv/X/Tza7vUuSXEAAAAAAAcAhFZUKWUzc4XLly5JckH8ryrbf1UUFbaaCkpI6NiKmhUJI6NipmhUKjo2KmaFQ0Do2KmzQjZTNzhcuXLklyQAAAAAABw==";
+
+async function h264Decodes(): Promise<boolean> {
+ if (typeof VideoDecoder === "undefined") return false;
+ try {
+ return !!(await VideoDecoder.isConfigSupported(H264_CONFIG)).supported;
+ } catch {
+ return false;
+ }
+}
+
+async function aacDecodes(): Promise<boolean> {
+ const bytes = Uint8Array.from(atob(AAC_SAMPLE), (c) => c.charCodeAt(0));
+ try {
+ for await (const chunk of audioChunks(new Blob([bytes], { type: "audio/mp4" }))) {
+ void chunk;
+ return true;
+ }
+ return false;
+ } catch {
+ return false;
+ }
+}
+
+/** Read the stored answer for this browser build, or run the checks once and
+ * keep theirs. */
+export async function readCodecs(io: CodecProbes): Promise<BrowserCodecs> {
+ try {
+ const held = JSON.parse(io.storage?.getItem(STORAGE_KEY) ?? "null") as (BrowserCodecs & { agent?: string }) | null;
+ if (held?.agent === io.agent) return { h264: !!held.h264, aac: !!held.aac };
+ } catch {
+ // An unreadable entry is asked again.
+ }
+ const [h264, aac] = await Promise.all([io.h264(), io.aac()]);
+ try {
+ io.storage?.setItem(STORAGE_KEY, JSON.stringify({ agent: io.agent, h264, aac }));
+ } catch {
+ // Storage is a convenience; the answer stands for this session.
+ }
+ return { h264, aac };
+}
+
+const BOTH: BrowserCodecs = { h264: true, aac: true };
+let answer: Promise<BrowserCodecs> | null = null;
+
+/** What this browser decodes. A headless runtime reads media through ffmpeg
+ * and decodes both. */
+export function browserCodecs(): Promise<BrowserCodecs> {
+ if (typeof document === "undefined" || frameSinkIsCustom()) return Promise.resolve(BOTH);
+ let storage: CodecProbes["storage"] = null;
+ try {
+ storage = localStorage;
+ } catch {
+ // A page with storage blocked asks once per session.
+ }
+ return (answer ??= readCodecs({ h264: h264Decodes, aac: aacDecodes, storage, agent: navigator.userAgent }));
+}
+
+/** The codecs of a master's two streams, read off its header once per file,
+ * by its address without the query a signed link re-mints. */
+const masterCodecs = new Map<string, Promise<{ video: string | null; audio: string | null }>>();
+
+function codecsOf(asset: Pick<MediaAsset, "url">) {
+ const key = asset.url.split("?")[0];
+ let read = masterCodecs.get(key);
+ if (!read) {
+ read = withMedia(asset.url, async (input) => {
+ const [video, audio] = await Promise.all([input.getPrimaryVideoTrack(), input.getPrimaryAudioTrack()]);
+ return { video: video?.codec ?? null, audio: audio?.codec ?? null };
+ });
+ // A failed read is asked again by the next caller.
+ read.catch(() => masterCodecs.delete(key));
+ masterCodecs.set(key, read);
+ }
+ return read;
+}
+
+/** The stream of the asset's master this browser cannot decode, as the compat
+ * cover that carries it (proxyNames.ts): "video" for H.264 picture, which
+ * covers the sound too, "audio" for AAC sound. Null when this browser decodes
+ * the master — every browser that decodes both codecs answers without reading
+ * the file. ProRes has its own proxy and WASM decoder, and takes none. */
+export async function compatNeedOf(
+ asset: Pick<MediaAsset, "type" | "url" | "block" | "color">
+): Promise<CompatCover | null> {
+ if ((asset.type !== "video" && asset.type !== "audio") || asset.block) {
+ return null;
+ }
+ if (asset.color?.codec?.startsWith("ap")) {
+ return null;
+ }
+ const codecs = await browserCodecs();
+ if (codecs.h264 && codecs.aac) {
+ return null;
+ }
+
+ const master = await codecsOf(asset).catch(() => null);
+ if (!master) {
+ return null;
+ }
+ if (asset.type === "video" && !codecs.h264 && master.video === "avc") {
+ return "video";
+ }
+ return !codecs.aac && master.audio === "aac" ? "audio" : null;
+}
diff --git a/site/src/cut/lib/decodePump.test.ts b/site/src/cut/lib/decodePump.test.ts
new file mode 100644
index 00000000..e3d273b8
--- /dev/null
+++ b/site/src/cut/lib/decodePump.test.ts
@@ -0,0 +1,97 @@
+import { afterAll, describe, expect, test } from "bun:test";
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import {
+ BlobSource,
+ CustomVideoDecoder,
+ type EncodedPacket,
+ Input,
+ MP4,
+ registerDecoder,
+ VideoSample,
+ VideoSampleSink,
+} from "mediabunny";
+
+// mediabunny's decode pump, as patched (patches/mediabunny+1.61.0.patch):
+// what happens to a decoder that fails while the reader has stopped pulling.
+
+const CLIP = join(dirname(fileURLToPath(import.meta.url)), "../../../public/cut-starter/media/ai-watercolor-and-loose-black-ink-sketch-il-1.mp4");
+const bytes = readFileSync(CLIP);
+
+// Registered decoders are process-wide and permanent, so this one answers
+// only while a case here runs.
+let active = false;
+let closed = 0;
+
+/** A decoder that hands back a frame per packet and fails out of band a
+ * moment after its fourth: by then the pump has filled the queue and is
+ * parked waiting for the reader. */
+class FailingDecoder extends CustomVideoDecoder {
+ private n = 0;
+
+ static override supports() {
+ return active;
+ }
+
+ init() {}
+
+ decode(packet: EncodedPacket) {
+ this.onSample(
+ new VideoSample(new Uint8Array(16), {
+ format: "RGBX",
+ codedWidth: 2,
+ codedHeight: 2,
+ timestamp: packet.timestamp,
+ duration: packet.duration,
+ })
+ );
+ if (++this.n === 4) {
+ setTimeout(() => this.onError(new DOMException("Decoder failure", "EncodingError")), 20);
+ }
+ }
+
+ flush() {}
+
+ close() {
+ closed++;
+ }
+}
+registerDecoder(FailingDecoder);
+
+afterAll(() => {
+ active = false;
+});
+
+const settle = () => new Promise((r) => setTimeout(r, 60));
+
+describe("decode pump", () => {
+ const reads = {
+ "a run of frames": (sink: VideoSampleSink) => sink.samples(),
+ "frames at timestamps": (sink: VideoSampleSink) => sink.samplesAtTimestamps(Array.from({ length: 48 }, (_, i) => i / 24)),
+ };
+
+ for (const [name, read] of Object.entries(reads)) {
+ test(`a decoder that fails mid-read is closed: ${name}`, async () => {
+ active = true;
+ closed = 0;
+ const input = new Input({ source: new BlobSource(new Blob([bytes])), formats: [MP4] });
+ try {
+ const track = await input.getPrimaryVideoTrack();
+ const samples = read(new VideoSampleSink(track!));
+
+ // One frame read, then the reader stops while the pump fills up.
+ (await samples.next()).value?.close();
+ await settle();
+
+ await expect(samples.next()).rejects.toThrow("Decoder failure");
+ await settle();
+
+ expect(closed).toBe(1);
+ } finally {
+ active = false;
+ input.dispose();
+ }
+ });
+ }
+});
diff --git a/site/src/cut/lib/derivedWrites.test.ts b/site/src/cut/lib/derivedWrites.test.ts
new file mode 100644
index 00000000..03f702b8
--- /dev/null
+++ b/site/src/cut/lib/derivedWrites.test.ts
@@ -0,0 +1,72 @@
+import { afterAll, expect, test } from "bun:test";
+
+import { landDerived, liveAsset } from "./derivedWrites";
+import { useEditor } from "./store";
+import type { MediaAsset } from "./types";
+
+// A project opening with this many clips lands a cached strip and a cached
+// waveform for each: past fifty separate store writes, React refuses the next.
+const CLIPS = 30;
+
+const video = (i: number): MediaAsset => ({
+ id: `v${i}`,
+ fileName: `v${i}.mp4`,
+ name: `v${i}`,
+ type: "video",
+ duration: 8,
+ width: 1280,
+ height: 720,
+ url: `https://media.test/v${i}.mp4`,
+});
+
+// The page's idle callbacks, run when the test says the page went idle.
+const idle: (() => void)[] = [];
+const g = globalThis as { requestIdleCallback?: unknown };
+const realIdle = g.requestIdleCallback;
+g.requestIdleCallback = (fn: () => void) => idle.push(fn);
+
+afterAll(() => {
+ g.requestIdleCallback = realIdle;
+});
+
+const nextTask = () => new Promise((r) => setTimeout(r, 0));
+
+test("strips and waveforms landing across tasks reach the store in one task", async () => {
+ useEditor.setState({ assets: Array.from({ length: CLIPS }, (_, i) => video(i)), readOnly: false });
+ let writes = 0;
+ const stop = useEditor.subscribe(() => writes++);
+ try {
+ // Each cached read resolves in a task of its own, the way IndexedDB
+ // answers them during a project open.
+ for (let i = 0; i < CLIPS; i++) {
+ landDerived(`v${i}`, { thumbs: [`t${i}`], thumbStep: 1 });
+ await nextTask();
+ landDerived(`v${i}`, { peaks: [i] });
+ await nextTask();
+ }
+
+ // Nothing lands while the page is busy, but readers see what is queued.
+ expect(writes).toBe(0);
+ expect(liveAsset("v3")?.peaks).toEqual([3]);
+
+ // One idle callback lands the lot.
+ expect(idle.length).toBe(1);
+ idle.splice(0).forEach((fn) => fn());
+ expect(writes).toBeGreaterThan(0);
+ const assets = useEditor.getState().assets;
+ expect(assets.every((a, i) => a.thumbs?.[0] === `t${i}` && a.peaks?.[0] === i)).toBe(true);
+ } finally {
+ stop();
+ }
+});
+
+test("a later patch to the same field wins", async () => {
+ useEditor.setState({ assets: [{ ...video(0), peaks: [9] }], readOnly: false });
+ landDerived("v0", { peaks: [1, 2] });
+ landDerived("v0", { peaks: undefined });
+
+ expect(idle.length).toBe(1);
+ idle.splice(0).forEach((fn) => fn());
+
+ expect(useEditor.getState().assets[0].peaks).toBeUndefined();
+});
diff --git a/site/src/cut/lib/derivedWrites.ts b/site/src/cut/lib/derivedWrites.ts
new file mode 100644
index 00000000..a911d8fd
--- /dev/null
+++ b/site/src/cut/lib/derivedWrites.ts
@@ -0,0 +1,67 @@
+import { useEditor } from "./store";
+import type { MediaAsset } from "./types";
+
+/**
+ * Filmstrips, waveforms and scene cuts land on the store in batches, once the
+ * page goes idle.
+ *
+ * Every store write forces React into a blocking render of its own, and that
+ * render throws away whatever lower-priority render React had under way. A
+ * project opening with a few dozen clips answered a cached strip and a cached
+ * waveform for each, every one in a task of its own, so React committed one
+ * forced render after another with its own work still pending — and past fifty
+ * in a row it refused the next write as an update loop (React error #185).
+ * Queued here, those patches wait for an idle moment, which comes only once
+ * React has finished its own work, and then land together as one render.
+ */
+
+/** Longest a queued patch waits for the page to go idle. A page that never
+ * idles — a long play, a busy import — still gets its strips, a batch at a
+ * time. */
+const IDLE_TIMEOUT_MS = 500;
+
+const queued = new Map<string, Partial<MediaAsset>>();
+let scheduled = false;
+
+/** Queue a filmstrip, waveform or scene-cut patch for an asset. A later
+ * patch to the same field replaces the earlier one. */
+export function landDerived(id: string, patch: Partial<MediaAsset>): void {
+ queued.set(id, { ...queued.get(id), ...patch });
+ if (scheduled) {
+ return;
+ }
+
+ scheduled = true;
+ whenIdle(flush);
+}
+
+/** The asset as it stands once its queued patches land: what a reader that
+ * just queued a strip should see. */
+export function liveAsset(id: string): MediaAsset | undefined {
+ const asset = useEditor.getState().assets.find((a) => a.id === id);
+ const patch = queued.get(id);
+ return asset && patch ? { ...asset, ...patch } : asset;
+}
+
+// Land every queued patch in this one task: React renders the writes of one
+// task together.
+function flush(): void {
+ scheduled = false;
+ const batch = [...queued];
+ queued.clear();
+ const { updateAsset } = useEditor.getState();
+ for (const [id, patch] of batch) {
+ updateAsset(id, patch);
+ }
+}
+
+// A runtime with no idle callbacks — the headless engine and worker — renders
+// nothing, so the next task serves.
+function whenIdle(fn: () => void): void {
+ if (typeof requestIdleCallback === "function") {
+ requestIdleCallback(fn, { timeout: IDLE_TIMEOUT_MS });
+ return;
+ }
+
+ setTimeout(fn, 0);
+}
diff --git a/site/src/cut/lib/exportRender.ts b/site/src/cut/lib/exportRender.ts
index 243016bf..b03d88c9 100644
--- a/site/src/cut/lib/exportRender.ts
+++ b/site/src/cut/lib/exportRender.ts
@@ -42,6 +42,7 @@ import { colorRead, type ColorRead, type ReadFile } from "./sourceColor";
import type { OutputSpace } from "@donkeycut/effects-kit";
import { heldSpans, overlayPlan, trackZeroAt, trackZeroPlan } from "./framePlan";
import { heldFrames, holdOverlays, type HeldFrames } from "./frameGrid";
+import { compatNeedOf } from "./codecSupport";
import { frameSink, keyframeTimeAt, openMedia, videoTrackOf, withAssetColors } from "./mediaRead";
import type { InputVideoTrack, WrappedCanvas } from "mediabunny";
import { allowance, canvasBytes, holdMemory } from "./memoryBudget";
@@ -1878,6 +1879,11 @@ export async function canRenderInBrowser(
) {
return false;
}
+ // A master this browser cannot decode — H.264 where none is installed, AAC
+ // its decoder fails on — renders where ffmpeg reads the original.
+ for (const asset of doc.assets) {
+ if (await compatNeedOf(asset)) return false;
+ }
try {
const trim = sourceExportPlan(doc, settings);
if (trim && (settings.codec === "h264" || settings.codec === "hevc")) {
diff --git a/site/src/cut/lib/framelessRewrite.test.ts b/site/src/cut/lib/framelessRewrite.test.ts
new file mode 100644
index 00000000..06e0b54f
--- /dev/null
+++ b/site/src/cut/lib/framelessRewrite.test.ts
@@ -0,0 +1,103 @@
+import { afterAll, describe, expect, spyOn, test } from "bun:test";
+
+import { stubModule } from "@/lib/testing/stubModule";
+import { useEditor } from "./store";
+import type { MediaAsset } from "./types";
+
+// A stream that parses and decodes here — it passes decodable() — and still
+// gives the filmstrip no frames: a file that opens on no keyframe. The
+// converter's rewrite of it reads like any file.
+const FRAMELESS = "stored:p1/clip.mp4";
+const REWRITTEN = "clip-1.mp4";
+const REWRITTEN_URL = `stored:p1/${REWRITTEN}`;
+
+await stubModule<typeof import("./mediaRead")>("./mediaRead", import.meta.url, {
+ withMedia: (async (_src: unknown, fn: (input: unknown) => Promise<unknown>) => fn({})) as never,
+ videoTrackOf: (async () => ({})) as never,
+ readMediaFileSize: async () => 1234,
+});
+await stubModule<typeof import("./baseFrame")>("./baseFrame", import.meta.url, {
+ baseFramesAt: async function* (url: string, times: number[]) {
+ if (url === FRAMELESS) return;
+ for (let i = 0; i < times.length; i++) yield { canvas: {} } as never;
+ },
+});
+await stubModule<typeof import("./raster")>("./raster", import.meta.url, {
+ rasterCanvasToDataUrl: async () => "data:image/jpeg;base64,tile",
+});
+await stubModule<typeof import("./mediaProxy")>("./mediaProxy", import.meta.url, {
+ ensureProxy: async () => {},
+ awaitsCompat: async () => false,
+});
+await stubModule<typeof import("./mediaSync")>("./mediaSync", import.meta.url, {
+ storedMediaUrl: async (projectId: string, fileName: string) => `stored:${projectId}/${fileName}`,
+});
+
+const { enrichAsset, stripFailedFor } = await import("./media");
+const { setHostMediaStore } = await import("./mediaConvert");
+
+// The project's converter, as a headless host installs it: every ask is
+// recorded, and the rewrite lands beside the original.
+const asks: { fileName: string; mode?: string }[] = [];
+setHostMediaStore({
+ async convert(_projectId, source, opts) {
+ asks.push({ fileName: source.fileName, mode: opts.mode });
+ return { fileName: REWRITTEN, reencoded: true, duration: 20, width: 64, height: 36 };
+ },
+ async drop() {},
+});
+
+// The stub frames are no pictures, so the scene probe behind the strip fails
+// and reports; this case is about the strip.
+const warn = spyOn(console, "warn").mockImplementation(() => {});
+const error = spyOn(console, "error").mockImplementation(() => {});
+afterAll(() => {
+ setHostMediaStore(null);
+ warn.mockRestore();
+ error.mockRestore();
+});
+
+const clip = (extra: Partial<MediaAsset> = {}): MediaAsset => ({
+ id: "v1",
+ fileName: "clip.mp4",
+ name: "clip.mp4",
+ type: "video",
+ duration: 20,
+ url: FRAMELESS,
+ peaks: [],
+ color: { matrix: "bt709", fullRange: false, bitDepth: 8, detected: "rec709", codec: "avc1" },
+ ...extra,
+});
+
+const live = () => useEditor.getState().assets.find((a) => a.id === "v1");
+
+/** Let the rewrite that runs behind the enrich land. */
+async function settle(done: () => boolean) {
+ for (let i = 0; i < 200 && !done(); i++) await new Promise((r) => setTimeout(r, 5));
+}
+
+describe("a stream that decodes and gives the filmstrip no frames", () => {
+ test("is re-encoded once a session, keeps its id, and builds its strip from the new file", async () => {
+ useEditor.setState({ projectId: "p1", readOnly: false, playing: false, assets: [clip()] });
+
+ await enrichAsset(clip());
+ await settle(() => (live()?.thumbs?.length ?? 0) > 0);
+
+ // One forced re-encode of the file the strip could not read.
+ expect(asks).toEqual([{ fileName: "clip.mp4", mode: "reencode" }]);
+ // The same asset, now on the rewrite, with a strip read off it.
+ const after = live()!;
+ expect(after.id).toBe("v1");
+ expect(after.fileName).toBe(REWRITTEN);
+ expect(after.url).toBe(REWRITTEN_URL);
+ expect(after.thumbs?.length).toBeGreaterThan(0);
+ expect(stripFailedFor("v1")).toBe(false);
+
+ // The same asset reading no frames again this session is not sent back:
+ // a file no converter can fix fails as unreadable once.
+ useEditor.setState({ assets: [clip()] });
+ await enrichAsset(clip());
+ expect(asks).toHaveLength(1);
+ expect(stripFailedFor("v1")).toBe(true);
+ });
+});
diff --git a/site/src/cut/lib/importQueue.ts b/site/src/cut/lib/importQueue.ts
index c153fa56..7b38e206 100644
--- a/site/src/cut/lib/importQueue.ts
+++ b/site/src/cut/lib/importQueue.ts
@@ -286,12 +286,13 @@ async function run(job: Job) {
upload: undefined,
...(asset.type === "image" ? { thumbs: [url] } : {}),
});
- // A ProRes master's preview proxy, or a heavy cloud video's, is built
- // once the bytes are at rest in the project's own storage, which is now.
+ // A ProRes master's preview proxy, a heavy cloud video's, or the compat
+ // proxy of a stream this browser cannot decode is built once the bytes
+ // are at rest in the project's own storage, which is now.
const landed = useEditor.getState().assets.find((a) => a.id === asset.id);
- if (landed) {
+ if (landed && landed.type !== "image") {
void import("./mediaProxy")
- .then((m) => (m.needsProxy(landed, job.backend.kind === "cloud") ? m.ensureProxy(job.projectId, landed, job.backend) : undefined))
+ .then((m) => m.ensureProxy(job.projectId, landed, job.backend))
.catch(() => {});
}
// Decoders and voices carry across the URL change on what they already
diff --git a/site/src/cut/lib/libraryProxy.ts b/site/src/cut/lib/libraryProxy.ts
index f5db9706..506dceb7 100644
--- a/site/src/cut/lib/libraryProxy.ts
+++ b/site/src/cut/lib/libraryProxy.ts
@@ -17,7 +17,9 @@ import { create } from "zustand";
import { engineFeatures } from "./api";
import { apiJson } from "./backend";
import { pollCloudJob } from "./cloudJob";
+import { browserCodecs } from "./codecSupport";
import { cutColor } from "./colorSettings";
+import { compatCovers } from "./proxyNames";
import type { LibraryAsset, LibraryData } from "./library";
import { useLightbox, lightboxItemFromLibrary } from "./lightbox";
import { backendFor } from "./residency";
@@ -50,6 +52,9 @@ const FAILED = "Could not make a playable copy of this video.";
* and the shelf's next listing carries the copy. */
const STILL_QUEUED = "The playable copy is still being made.";
+/** Asks checking this browser's codecs, by key, so a second ask waits out the first. */
+const asking = new Set<string>();
+
/**
* Have the asset's shelf build its playable copy, then write the copy onto
* the cached listing and onto the viewer if it is showing this asset. Runs
@@ -61,7 +66,14 @@ export async function ensureLibraryProxy(
patch: (fn: (d: LibraryData) => LibraryData) => void
): Promise<void> {
const key = libraryProxyKey(a);
- if (a.type !== "video" || a.proxyFile || useLibraryProxies.getState().jobs[key]) return;
+ if (a.type !== "video" || useLibraryProxies.getState().jobs[key] || asking.has(key)) return;
+ // A browser with no H.264 decoder plays the compat copy (proxyNames.ts).
+ // The Mac's shelf answers browsers that decode it; a browser on that Mac
+ // always does.
+ asking.add(key);
+ const codecs = await browserCodecs().finally(() => asking.delete(key));
+ const compat = a.residency !== "local" && !codecs.h264 ? "video" : null;
+ if (a.proxyFile && (!compat || compatCovers(a.proxyFile, compat))) return;
setJob(key, { kind: "making", progress: 0 });
try {
// A Mac app older than the route has nothing to build the copy with.
@@ -73,7 +85,7 @@ export async function ensureLibraryProxy(
const res = await backend.fetch(`/api/cut/library/${encodeURIComponent(a.id)}/proxy`, {
method: "POST",
headers: { "Content-Type": "application/json" },
- body: JSON.stringify({ maxHeight: proxyMaxHeight, crf: proxyCrf }),
+ body: JSON.stringify({ maxHeight: proxyMaxHeight, crf: proxyCrf, ...(compat ? { compat } : {}) }),
});
const body = await apiJson<LibraryAsset | { jobId: string }>(res);
if (!res.ok) throw new Error(body.error ?? FAILED);
diff --git a/site/src/cut/lib/libraryUpload.ts b/site/src/cut/lib/libraryUpload.ts
index 1282b55f..a3dd1f26 100644
--- a/site/src/cut/lib/libraryUpload.ts
+++ b/site/src/cut/lib/libraryUpload.ts
@@ -1,3 +1,4 @@
+import { CONVERT_MODES } from "./convertMode";
import { z } from "zod";
export const CLOUD_LIBRARY_IMPORT_DESCRIPTION =
@@ -7,6 +8,8 @@ export const libraryUploadSchema = z.object({
key: z.string().min(1),
name: z.string().min(1).max(512),
maxHeight: z.number().int().min(2).max(8192).optional(),
+ /** "reencode" rewrites a video stream that already fits (server/convert.ts). */
+ mode: z.enum(CONVERT_MODES).optional(),
folderId: z.string().min(1).nullable().optional(),
source: z.object({
url: z.string().url(),
diff --git a/site/src/cut/lib/media.ts b/site/src/cut/lib/media.ts
index 4703f854..cc46fd13 100644
--- a/site/src/cut/lib/media.ts
+++ b/site/src/cut/lib/media.ts
@@ -16,7 +16,7 @@ import { downloadFromUrl } from "./download";
import { FONT_RE, LUT_RE } from "./libraryFileType";
import { markUploadStored, startUpload, uploadInFlight } from "./importQueue";
import { convertProjectFile, dropProjectFile, mp4DisplayName } from "./mediaConvert";
-import { ensureProxy, needsProxy } from "./mediaProxy";
+import { awaitsCompat, ensureProxy } from "./mediaProxy";
import {
audioChunks,
audioPeaks,
@@ -30,6 +30,7 @@ import {
readMediaFileSize,
UnreadableMediaError,
videoTrackOf,
+ withMedia,
type IdleGate,
type MediaHandle,
type MediaProbe,
@@ -56,6 +57,8 @@ import {
} from "./raster";
import { pickThumbTimes, type ThumbProbe } from "./filmstrip";
import { reportSwallowed } from "./report";
+import { cloudMediaGone, cloudMediaStatus } from "./chunkCache";
+import { landDerived, liveAsset } from "./derivedWrites";
import { baseFrameAt, baseFramesAt, baseFrameSink, colorReadAt } from "./baseFrame";
import { colorRecord } from "./colorProbe";
import { previewFile, sameColorRead } from "./sourceColor";
@@ -797,6 +800,9 @@ export async function resumePendingUploads(
localUrl: url,
send: sendStoredUpload(backend, projectId, p.fileName),
});
+ // The open's sweep ran first and read the cloud, where these bytes have
+ // not landed yet. The strip and the waveform read the local copy instead.
+ void enrichAsset({ ...asset, url, upload });
}
}
@@ -1969,6 +1975,54 @@ function setStripFailed(assetId: string, failed: boolean) {
for (const fn of stripListeners) fn();
}
+// Assets whose cloud file is not there, by id — session state, like a failed
+// strip. One read finds the gap and the clip says so; nothing reads the file
+// again until the page reloads or an upload of it starts. The bytes of an
+// upload that never finished live only in the browser that started it, so
+// every other device, and a share of the project, finds nothing.
+const missingMedia = new Set<string>();
+
+/** Whether this asset's file was found missing from the cloud this session. */
+export const missingMediaFor = (assetId: string) => missingMedia.has(assetId);
+
+function setMediaMissing(assetId: string, missing: boolean) {
+ if (missing === missingMedia.has(assetId)) return;
+ if (missing) missingMedia.add(assetId);
+ else missingMedia.delete(assetId);
+ for (const fn of stripListeners) fn();
+}
+
+/** Whether to skip reading this asset's file: it was found missing, and no
+ * upload of it has started since. An upload clears the mark. */
+function knownMissing(asset: MediaAsset): boolean {
+ if (asset.upload) setMediaMissing(asset.id, false);
+ return missingMedia.has(asset.id);
+}
+
+// A cloud route answers this when the request carries no session.
+const UNAUTHORIZED = 401;
+
+/**
+ * Settle a failed read of an asset's file. A file that is not there marks the
+ * asset missing, unless an upload of it is running and the bytes are on their
+ * way. A read refused for want of a session belongs to the session gate, which
+ * hears the same refusal (chunkCache.ts). Neither is a fault to report; every
+ * other failure is reported against `what`, the step that failed.
+ */
+function readFailed(asset: MediaAsset, what: string, err: unknown, subject = asset.fileName) {
+ const live = useEditor.getState().assets.find((a) => a.id === asset.id) ?? asset;
+ if (cloudMediaGone(err)) {
+ if (!live.upload) setMediaMissing(asset.id, true);
+ console.warn(`${what} for ${subject}`, err);
+ return;
+ }
+ if (cloudMediaStatus(err) === UNAUTHORIZED) {
+ console.warn(`${what} for ${subject}`, err);
+ return;
+ }
+ reportSwallowed(what, err, subject);
+}
+
// A failed enrich books itself more tries. Filmstrip and waveform reads drop
// out together with the preview's when media reads fail, and the sweep that
// would try again runs only on project open — so a blip at open used to leave
@@ -2119,6 +2173,37 @@ async function* idleFramesAt(
}
}
+/** Assets whose file was rewritten this session after reading no frames:
+ * one try each, so a file no converter can fix never loops. */
+const rewritten = new Set<string>();
+
+/**
+ * A strip that read no frames from a stream this page decodes — one that
+ * opens on no keyframe, a broken reference chain — hands the file to the
+ * project's converter for a forced re-encode to clean H.264, once. The asset
+ * keeps its id and moves to the new file, which enriches again. False when
+ * the stream does not decode here at all, or was rewritten already.
+ */
+async function rewriteFrameless(asset: MediaAsset, src: string): Promise<boolean> {
+ const s = useEditor.getState();
+ const projectId = s.projectId;
+ if (!projectId || s.readOnly || asset.upload || rewritten.has(asset.id)) return false;
+ const decodes = await withMedia(src, async (input) => !!(await videoTrackOf(input))).catch(() => false);
+ if (!decodes) return false;
+ rewritten.add(asset.id);
+ void import("./mediaConvert")
+ .then(async (m) => {
+ const made = await m.convertAssetToMp4(projectId, liveAsset(asset.id) ?? asset, { mode: "reencode" });
+ // A converter that kept the file has nothing new to read.
+ if (made.unchanged) setStripFailed(asset.id, true);
+ })
+ .catch((err: unknown) => {
+ setStripFailed(asset.id, true);
+ readFailed(asset, "[cut] frameless rewrite failed", err);
+ });
+ return true;
+}
+
/** Generate filmstrip thumbnails / waveform peaks and merge them into the
* store. Safe to call repeatedly; skips assets that are already enriched.
* `src` overrides where the frames are read from — an import still uploading
@@ -2127,6 +2212,7 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
// A block paints itself at whatever size asks, so there is no file to probe
// and no still to stand in as its strip.
if (asset.block) return;
+ if (knownMissing(asset)) return;
if (enrichInFlight.has(asset.id)) return;
enrichInFlight.add(asset.id);
const held = importedBytes.get(asset.id);
@@ -2150,20 +2236,24 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
// both read `color`, and a file without it would draw as Rec.709.
// The asset's live record decides: the import's own probe may have
// written it while this sweep waited, and that walk is shared.
- asset = useEditor.getState().assets.find((a) => a.id === asset.id) ?? asset;
+ asset = liveAsset(asset.id) ?? asset;
if (!asset.color && !asset.upload) {
const color = await probeMediaColor(held ?? src).catch(() => null);
if (useEditor.getState().projectId !== openProject) return;
if (color) {
await landProbedFacts(asset.id, { color: assetColor(color) });
- asset = useEditor.getState().assets.find((a) => a.id === asset.id) ?? asset;
+ asset = liveAsset(asset.id) ?? asset;
}
}
// A ProRes master previews through the WASM decoder until its proxy
- // lands, and a heavy video streamed from the cloud stalls at its cuts;
- // the proxy is built behind the editor, once the bytes are at rest in
- // the project's own storage.
- if (openProject && !asset.upload && needsProxy(asset, getBackend().kind === "cloud")) void ensureProxy(openProject, asset);
+ // lands, a heavy video streamed from the cloud stalls at its cuts, and
+ // a stream this browser cannot decode reads from a compat proxy; the
+ // proxy is built behind the editor, once the bytes are at rest in the
+ // project's own storage.
+ if (openProject && !asset.upload) void ensureProxy(openProject, asset);
+ // A picture this browser cannot decode has nothing to sweep until its
+ // compat proxy lands; the landing enriches the asset again.
+ if (await awaitsCompat(asset, "video")) return;
// Footage stored without a length — a shelf row that was never measured
// — reads its own before anything is swept from it: a zero-length sweep
// has no tiles to place and no buckets to fill, and the clips cut from
@@ -2175,7 +2265,7 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
width: probe.width,
height: probe.height,
});
- asset = useEditor.getState().assets.find((a) => a.id === asset.id) ?? asset;
+ asset = liveAsset(asset.id) ?? asset;
if (!(asset.duration > 0)) throw new UnreadableMediaError("The video has no length.");
}
// The clip box draws the sound under the picture, so a video enriches
@@ -2185,12 +2275,12 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
// long a source's frame grabs take. `[]` is an answer too — a silent
// file — and stops the strip from re-decoding every open.
const peaksRun =
- asset.peaks === undefined
+ asset.peaks === undefined && !(await awaitsCompat(asset, "audio"))
? loadPeaks(asset, held ?? src, (partial) =>
- useEditor.getState().updateAsset(asset.id, { peaks: partial })
+ landDerived(asset.id, { peaks: partial })
)
.then((peaks) => {
- if (peaks) useEditor.getState().updateAsset(asset.id, { peaks });
+ if (peaks) landDerived(asset.id, { peaks });
})
.catch((err: unknown) => {
// The strip paints on its own; a peaks failure (an expired
@@ -2198,8 +2288,8 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
// What the walk had published comes off the asset, so the
// next open reads the file again in place of keeping a
// strip that stops partway.
- useEditor.getState().updateAsset(asset.id, { peaks: undefined });
- reportSwallowed(`[cut] peaks failed for ${asset.fileName}`, err);
+ landDerived(asset.id, { peaks: undefined });
+ readFailed(asset, "[cut] peaks failed", err);
})
: undefined;
if (!stripComplete(asset)) {
@@ -2207,7 +2297,7 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
const key = stripCacheKey(useEditor.getState().projectId, asset.id);
const cached = await readCachedStrip(key, asset.duration);
if (cached) {
- useEditor.getState().updateAsset(asset.id, {
+ landDerived(asset.id, {
thumbs: cached.thumbs,
thumbStep: cached.thumbStep,
sceneCuts: cached.cuts,
@@ -2216,38 +2306,45 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
// Midpoint grabs, published tile by tile: the strip fills in as the
// sweep runs, and the scene pass refines the grab points behind it.
const { thumbs, thumbStep } = await makeThumbs(src, asset.duration, [], (partial, step) =>
- useEditor.getState().updateAsset(asset.id, { thumbs: partial, thumbStep: step })
+ landDerived(asset.id, { thumbs: partial, thumbStep: step })
);
+ // A stream this page decodes that gave no frames has its file
+ // rewritten by a converter, which enriches the asset again.
+ if (!thumbs.length && (await rewriteFrameless(asset, src))) return;
// An empty strip is a failure, not a result: persisting it would
// leave the asset looking permanently mid-load.
if (!thumbs.length) throw new UnreadableMediaError("No frames could be read for the filmstrip.");
- useEditor.getState().updateAsset(asset.id, { thumbs, thumbStep });
+ landDerived(asset.id, { thumbs, thumbStep });
writeCache(key, { thumbs, thumbStep, duration: asset.duration, at: Date.now() });
}
}
// The cuts a strip was built without — a fresh sweep, or a cache written
// before the scene pass finished — are probed once the tiles are up.
- const live = useEditor.getState().assets.find((a) => a.id === asset.id);
+ const live = liveAsset(asset.id);
if (live && stripComplete(live) && live.sceneCuts === undefined) {
void refineSceneCuts(asset.id, src, asset.duration).catch((err) => {
// Scene cuts only sharpen the strip; the tiles already painted.
- reportSwallowed(`[cut] scene probe failed for ${asset.fileName}`, err);
+ readFailed(asset, "[cut] scene probe failed", err);
});
}
await peaksRun;
enrichRetries.delete(asset.id);
} else if (asset.type === "audio" && !asset.peaks?.length) {
const openProject = useEditor.getState().projectId;
+ // Sound this browser cannot decode reads from a compat proxy, built
+ // once the bytes are at rest; its landing enriches the asset again.
+ if (openProject && !asset.upload) void ensureProxy(openProject, asset);
+ if (await awaitsCompat(asset, "audio")) return;
await idlePlayback();
if (useEditor.getState().projectId !== openProject) return;
- const publish = (peaks: number[]) => useEditor.getState().updateAsset(asset.id, { peaks });
+ const publish = (peaks: number[]) => landDerived(asset.id, { peaks });
try {
const peaks = await loadPeaks(asset, held ?? src, publish);
if (peaks) publish(peaks);
} catch (err) {
// A walk that published partway comes off the asset before the retry
// below, so the retry reads the file again.
- useEditor.getState().updateAsset(asset.id, { peaks: undefined });
+ landDerived(asset.id, { peaks: undefined });
throw err;
}
enrichRetries.delete(asset.id);
@@ -2258,11 +2355,9 @@ export async function enrichAsset(asset: MediaAsset, src = asset.url) {
// reported, which carries it into PostHog error tracking with the file's
// shape beside the cause (lib/report.ts).
if (asset.type === "video") setStripFailed(asset.id, true);
- if (asset.type !== "image") scheduleEnrichRetry(asset.id);
- reportSwallowed(
- `[cut] enrich failed for ${asset.fileName} (${asset.type}, ${Math.round(asset.duration)}s)`,
- err
- );
+ readFailed(asset, "[cut] enrich failed", err, `${asset.fileName} (${asset.type}, ${Math.round(asset.duration)}s)`);
+ // A missing file stays missing; trying again would ask for the same nothing.
+ if (asset.type !== "image" && !missingMedia.has(asset.id)) scheduleEnrichRetry(asset.id);
} finally {
enrichInFlight.delete(asset.id);
}
@@ -2325,15 +2420,16 @@ export async function detectAssetBeats(asset: MediaAsset): Promise<AssetBeats> {
* onto the soundtrack track (video assets don't get peaks at import). */
export async function ensurePeaks(asset: MediaAsset) {
if (asset.peaks?.length) return;
- const publish = (peaks: number[]) => useEditor.getState().updateAsset(asset.id, { peaks });
+ if (knownMissing(asset)) return;
+ const publish = (peaks: number[]) => landDerived(asset.id, { peaks });
try {
const peaks = await loadPeaks(asset, asset.url, publish);
if (peaks?.length) publish(peaks);
} catch (err) {
// Waveforms are decorative; editing works without them. A walk that
// published partway comes off the asset, so the next read starts clean.
- useEditor.getState().updateAsset(asset.id, { peaks: undefined });
- reportSwallowed(`[cut] peaks failed for ${asset.fileName}`, err);
+ landDerived(asset.id, { peaks: undefined });
+ readFailed(asset, "[cut] peaks failed", err);
}
}
@@ -2662,10 +2758,10 @@ async function refineSceneCuts(assetId: string, url: string, duration: number) {
if (sceneProbeInFlight.has(assetId)) return;
sceneProbeInFlight.add(assetId);
try {
- const live = () => useEditor.getState().assets.find((a) => a.id === assetId);
+ const live = () => liveAsset(assetId);
const { probes, cuts } = await sceneProbe(url, duration);
if (!live()) return;
- useEditor.getState().updateAsset(assetId, { sceneCuts: cuts });
+ landDerived(assetId, { sceneCuts: cuts });
const { count, thumbStep } = thumbGrid(duration);
// The cuts land in the cache however the tiles turn out. A sweep that
// moves no grab point still answered the expensive question, and a cache
@@ -2706,7 +2802,7 @@ async function refineSceneCuts(assetId: string, url: string, duration: number) {
if (!current?.thumbs?.length) return;
const thumbs = [...current.thumbs];
for (const [i, tile] of regrabbed) if (i < thumbs.length) thumbs[i] = tile;
- useEditor.getState().updateAsset(assetId, { thumbs });
+ landDerived(assetId, { thumbs });
persist(thumbs);
} finally {
sceneProbeInFlight.delete(assetId);
diff --git a/site/src/cut/lib/mediaConvert.ts b/site/src/cut/lib/mediaConvert.ts
index f4a4296c..e44a0a03 100644
--- a/site/src/cut/lib/mediaConvert.ts
+++ b/site/src/cut/lib/mediaConvert.ts
@@ -27,6 +27,7 @@ import {
videoTrackOf,
UnreadableMediaError,
} from "./mediaRead";
+import type { ConvertMode } from "./convertMode";
import { useEditor } from "./store";
import type { MediaAsset } from "./types";
@@ -41,6 +42,8 @@ export interface ConvertSource {
export interface ConvertOptions {
maxHeight?: number;
+ /** "reencode" rewrites a video stream that already fits (convertMode.ts). */
+ mode?: ConvertMode;
/** Pinned by work that outlives navigation; defaults to the active one. */
backend?: CutBackend;
}
@@ -118,7 +121,7 @@ export async function convertProjectFile(
// projects, the worker beside R2 for cloud ones. A headless process working
// a cloud project is that machine, so it converts in-process.
return backend.kind === "local" || (backend.kind === "cloud" && inPage())
- ? convertOnBackend(projectId, source, backend, opts.maxHeight)
+ ? convertOnBackend(projectId, source, backend, opts)
: convertNearThePage(projectId, source, { ...opts, backend });
}
@@ -207,12 +210,12 @@ async function convertOnBackend(
projectId: string,
source: ConvertSource,
backend: CutBackend,
- maxHeight?: number
+ { maxHeight, mode }: ConvertOptions
): Promise<FileConversion> {
const res = await backend.fetch(`/api/cut/projects/${projectId}/convert`, {
method: "POST",
headers: { "Content-Type": "application/json" },
- body: JSON.stringify({ file: source.fileName, ...(maxHeight ? { maxHeight } : {}) }),
+ body: JSON.stringify({ file: source.fileName, ...(maxHeight ? { maxHeight } : {}), ...(mode === "reencode" ? { mode } : {}) }),
});
let body: ServerConvert;
if (backend.kind === "cloud") {
@@ -238,7 +241,9 @@ async function convertNearThePage(
source: ConvertSource,
opts: ConvertOptions
): Promise<FileConversion> {
- const decodable = await probeMediaFile(source.url).then(() => true, (error: unknown) => {
+ // A forced re-encode is for a stream the page reads no frames from, so the
+ // page has nothing to encode it from; a headless runtime decodes it itself.
+ const decodable = (opts.mode !== "reencode" || !inPage()) && await probeMediaFile(source.url).then(() => true, (error: unknown) => {
if (error instanceof UnreadableMediaError && error.undecodable) return false;
throw error;
});
@@ -258,7 +263,7 @@ async function convertNearThePage(
return withCloudPreparedMedia(
new File([response], source.fileName, { type: response.type }),
(prepared) => land(projectId, source, prepared, opts),
- { name: source.name, maxHeight: opts.maxHeight },
+ { name: source.name, maxHeight: opts.maxHeight, mode: opts.mode },
);
}
@@ -285,9 +290,11 @@ async function convertInProcess(
const height = video ? await video.getDisplayHeight() : 0;
const cap = opts.maxHeight && opts.maxHeight > 0 ? Math.round(opts.maxHeight) : 0;
const shrink = !!(cap && height > cap);
+ const forced = opts.mode === "reencode" && !!video;
// Already the file a conversion would produce — a second pass would copy
// it to a new name and change nothing.
if (
+ !forced &&
MP4_EXT.test(source.fileName) &&
!shrink &&
(!video || video.codec === "avc") &&
@@ -304,10 +311,11 @@ async function convertInProcess(
// A stream already in the target codec is copied packet for packet; what
// is left over is what gets decoded and re-encoded.
const reencoded =
- (!!video && (video.codec !== "avc" || shrink)) || (!!audio && audio.codec !== "aac");
+ (!!video && (video.codec !== "avc" || shrink || forced)) || (!!audio && audio.codec !== "aac");
const videoOptions: ConversionVideoOptions = {
codec: "avc",
...(shrink ? { height: cap } : {}),
+ ...(forced ? { forceTranscode: true } : {}),
};
const audioOptions: ConversionAudioOptions = { codec: "aac" };
@@ -347,6 +355,7 @@ async function convertOnTheMac(
const form = new FormData();
form.append("file", bytes, source.fileName);
if (opts.maxHeight) form.append("maxHeight", String(Math.round(opts.maxHeight)));
+ if (opts.mode === "reencode") form.append("mode", opts.mode);
const res = await engineFetch("/api/cut/convert", { method: "POST", body: form });
if (!res.ok) {
const body = (await res.json().catch(() => ({}))) as { error?: string };
diff --git a/site/src/cut/lib/mediaMissing.test.ts b/site/src/cut/lib/mediaMissing.test.ts
new file mode 100644
index 00000000..fd9cf0e3
--- /dev/null
+++ b/site/src/cut/lib/mediaMissing.test.ts
@@ -0,0 +1,132 @@
+import { afterAll, beforeEach, describe, expect, spyOn, test } from "bun:test";
+
+import { stubModule } from "@/lib/testing/stubModule";
+import { CloudMediaError } from "./chunkCache";
+import { bindFailureSink } from "./report";
+import { useEditor } from "./store";
+import type { MediaAsset } from "./types";
+
+// Every peaks read answers with whatever the case sets, and counts itself.
+let failure: unknown = null;
+let reads = 0;
+await stubModule<typeof import("./mediaRead")>("./mediaRead", import.meta.url, {
+ audioPeaks: async () => {
+ reads++;
+ if (failure) throw failure;
+ return [0.5];
+ },
+});
+
+// The browser store holds a ledger pin, and the bytes, for whatever file the
+// case names.
+let pinned = "";
+await stubModule<typeof import("./mediaSync")>("./mediaSync", import.meta.url, {
+ pendingUploadsFor: async () => (pinned ? [{ projectId: "p1", fileName: pinned }] : []) as never,
+ localMediaUrl: async () => "blob:local",
+ localMediaFile: async () => null,
+});
+const { enrichAsset, ensurePeaks, missingMediaFor, resumePendingUploads } = await import("./media");
+
+const reported: Error[] = [];
+bindFailureSink((error) => reported.push(error));
+
+const warn = spyOn(console, "warn").mockImplementation(() => {});
+const decoder = globalThis as { AudioDecoder?: unknown };
+const hadDecoder = "AudioDecoder" in decoder;
+decoder.AudioDecoder ??= class {};
+afterAll(() => {
+ warn.mockRestore();
+ if (!hadDecoder) delete decoder.AudioDecoder;
+});
+
+let n = 0;
+const audio = (extra: Partial<MediaAsset> = {}): MediaAsset => ({
+ id: `a${++n}`,
+ fileName: `take-${n}.m4a`,
+ name: `take-${n}.m4a`,
+ type: "audio",
+ duration: 10,
+ url: `/api/cut-cloud/projects/p1/media/take-${n}.m4a`,
+ ...extra,
+});
+
+beforeEach(() => {
+ pinned = "";
+ reads = 0;
+ reported.length = 0;
+ useEditor.setState({ projectId: "p1", playing: false });
+});
+
+describe("a cloud file that is gone", () => {
+ test("marks the asset missing and reports nothing", async () => {
+ failure = new CloudMediaError(404);
+ const asset = audio();
+
+ await ensurePeaks(asset);
+
+ expect(missingMediaFor(asset.id)).toBe(true);
+ expect(reported).toEqual([]);
+ });
+
+ test("is read once per session", async () => {
+ failure = new CloudMediaError(404);
+ const asset = audio();
+
+ await enrichAsset(asset);
+ await enrichAsset(asset);
+ await ensurePeaks(asset);
+
+ expect(reads).toBe(1);
+ });
+
+ test("still uploading is not missing", async () => {
+ failure = new CloudMediaError(404);
+ const asset = audio({ upload: { progress: 0.4, stored: true } });
+
+ await ensurePeaks(asset);
+
+ expect(missingMediaFor(asset.id)).toBe(false);
+ });
+});
+
+describe("an upload this browser still holds", () => {
+ test("resuming it reads the local bytes and clears the mark", async () => {
+ // The open's sweep reads the cloud before the ledger re-marks the upload,
+ // and finds nothing there yet.
+ failure = new CloudMediaError(404);
+ const asset = audio();
+ useEditor.setState({ assets: [asset] });
+ await enrichAsset(asset);
+ expect(missingMediaFor(asset.id)).toBe(true);
+
+ failure = null;
+ pinned = asset.fileName;
+ await resumePendingUploads("p1", { kind: "cloud" } as never);
+ await new Promise((r) => setTimeout(r, 0));
+
+ expect(missingMediaFor(asset.id)).toBe(false);
+ expect(useEditor.getState().assets.find((a) => a.id === asset.id)?.peaks).toEqual([0.5]);
+ });
+});
+
+describe("other read failures", () => {
+ test("a read refused for want of a session is not reported", async () => {
+ failure = new CloudMediaError(401);
+ const asset = audio();
+
+ await ensurePeaks(asset);
+
+ expect(missingMediaFor(asset.id)).toBe(false);
+ expect(reported).toEqual([]);
+ });
+
+ test("an undecodable file is reported against its stage", async () => {
+ failure = new DOMException("Decoding error.", "EncodingError");
+ const asset = audio();
+
+ await ensurePeaks(asset);
+
+ expect(missingMediaFor(asset.id)).toBe(false);
+ expect(reported.map((e) => e.message)).toEqual([`[cut] peaks failed for ${asset.fileName}`]);
+ });
+});
diff --git a/site/src/cut/lib/mediaProxy.test.ts b/site/src/cut/lib/mediaProxy.test.ts
index 504e6e42..1efd6dce 100644
--- a/site/src/cut/lib/mediaProxy.test.ts
+++ b/site/src/cut/lib/mediaProxy.test.ts
@@ -23,7 +23,26 @@ await stubModule<typeof import("./mediaRead")>("./mediaRead", import.meta.url, {
fn({ canRead: async () => storedReads })) as never,
hasUndecodableVideo: async () => proxyCodec === "hevc" && !hevc,
});
+/** The stream of the master this browser cannot decode, per asset id. */
+let compatNeeds: Record<string, "audio" | "video"> = {};
+await stubModule<typeof import("./codecSupport")>("./codecSupport", import.meta.url, {
+ compatNeedOf: (async (asset: { id?: string }) => compatNeeds[asset.id ?? ""] ?? null) as never,
+});
+const staged: { name: string; cover: string }[] = [];
+await stubModule<typeof import("./cloudLibraryUpload")>("./cloudLibraryUpload", import.meta.url, {
+ compatInTheCloud: async (file: File, cover: string) => {
+ staged.push({ name: file.name, cover });
+ return new Blob(["webm"], { type: "video/webm" });
+ },
+});
+const enriched: string[] = [];
+const stored: string[] = [];
+await stubModule<typeof import("./media")>("./media", import.meta.url, {
+ enrichAsset: (async (asset: MediaAsset) => void enriched.push(asset.id)) as never,
+ uploadProjectMediaTo: (async (_b: unknown, _p: string, _blob: Blob, name: string) => (stored.push(name), name)) as never,
+});
const { ensureProxy, HEAVY_PROXY_MAX_HEIGHT, needsProxy, proxyUnreadable, useProxyJobs } = await import("./mediaProxy");
+const { coveredSrc } = await import("./mediaRead");
const { useEditor } = await import("./store");
const prores = (id: string, extra: Partial<MediaAsset> = {}): MediaAsset => ({
@@ -197,6 +216,87 @@ describe("ensureProxy", () => {
});
});
+describe("compat proxy", () => {
+ // A browser that cannot decode a master's H.264 or AAC reads that stream
+ // from a compat proxy, built where the project's media is.
+ afterEach(() => {
+ compatNeeds = {};
+ staged.length = 0;
+ enriched.length = 0;
+ stored.length = 0;
+ });
+
+ test("a cloud project asks the worker for the cover and reads the stream from it", async () => {
+ compatNeeds = { a: "video" };
+ const asset = h264("a", { width: 1280, height: 720, sizeBytes: 2_000_000 });
+ useEditor.setState({ projectId: "p", assets: [asset] });
+ const b = backend("cloud", () => Response.json({ fileName: "a.mp4.compat.webm", sizeBytes: 9 }));
+ await ensureProxy("p", asset, b.api);
+ expect(b.calls[0]).toEqual({ path: "/api/cut/projects/p/proxy", body: { file: "a.mp4", compat: "video" } });
+ expect(useEditor.getState().assets[0].proxy).toEqual({ fileName: "a.mp4.compat.webm", sizeBytes: 9 });
+ // The strip waited on the proxy, and reads it now.
+ expect(enriched).toEqual(["a"]);
+ await Promise.resolve();
+ await new Promise((r) => setTimeout(r, 0));
+ expect(coveredSrc("video", asset.url)).toBe("stored:p/a.mp4.compat.webm");
+ expect(coveredSrc("audio", asset.url)).toBe("stored:p/a.mp4.compat.webm");
+ });
+
+ test("a heavy cloud video gets the light cover", async () => {
+ compatNeeds = { a: "audio" };
+ const asset = h264("a");
+ useEditor.setState({ projectId: "p", assets: [asset] });
+ const b = backend("cloud", () => Response.json({ fileName: "a.mp4.compat.light.mp4", sizeBytes: 9 }));
+ await ensureProxy("p", asset, b.api);
+ expect(b.calls[0].body).toEqual({ file: "a.mp4", compat: "audio", maxHeight: HEAVY_PROXY_MAX_HEIGHT });
+ await new Promise((r) => setTimeout(r, 0));
+ // Only the sound reads from the proxy; this browser decodes the picture.
+ expect(coveredSrc("audio", asset.url)).toBe("stored:p/a.mp4.compat.light.mp4");
+ expect(coveredSrc("video", asset.url)).toBe(asset.url);
+ });
+
+ test("a Mac project asks its engine, and an older engine makes none", async () => {
+ compatNeeds = { a: "audio" };
+ features = new Set(["media.proxy", "media.compat"]);
+ const asset = h264("a");
+ useEditor.setState({ projectId: "p", assets: [asset] });
+ const b = backend("local", () => Response.json({ fileName: "a.mp4.compat.mp4", sizeBytes: 4 }));
+ await ensureProxy("p", asset, b.api);
+ expect(b.calls[0].body).toMatchObject({ file: "a.mp4", compat: "audio" });
+
+ features = new Set(["media.proxy"]);
+ useEditor.setState({ projectId: "p", assets: [h264("b")] });
+ compatNeeds = { b: "audio" };
+ const old = backend("local", () => Response.json({}));
+ await ensureProxy("p", h264("b"), old.api);
+ expect(old.calls).toHaveLength(0);
+ expect(useProxyJobs.getState().jobs.b).toEqual({ kind: "none" });
+ });
+
+ test("a browser project borrows the worker through the cloud shelf and stores the cover itself", async () => {
+ compatNeeds = { a: "video" };
+ const asset = h264("a", { url: "blob:a" });
+ useEditor.setState({ projectId: "p", assets: [asset] });
+ const realFetch = globalThis.fetch;
+ globalThis.fetch = (async () => new Response("master")) as unknown as typeof fetch;
+ try {
+ await ensureProxy("p", asset, backend("browser", () => Response.json({})).api);
+ } finally {
+ globalThis.fetch = realFetch;
+ }
+ expect(staged).toEqual([{ name: "a.mp4", cover: "video" }]);
+ expect(stored).toEqual(["a.mp4.compat.webm"]);
+ expect(useEditor.getState().assets[0].proxy?.fileName).toBe("a.mp4.compat.webm");
+ });
+
+ test("a cover already on the asset is kept; a VP9 cover carries the sound too", async () => {
+ compatNeeds = { a: "audio" };
+ const b = backend("cloud", () => Response.json({}));
+ await ensureProxy("p", h264("a", { width: 1280, height: 720, sizeBytes: 2_000_000, proxy: { fileName: "a.mp4.compat.webm", sizeBytes: 1 } }), b.api);
+ expect(b.calls).toHaveLength(0);
+ });
+});
+
describe("proxyUnreadable", () => {
// A doc can name a proxy whose file is not there (a copy made before
// proxies travelled, a file lost from storage). The reader that failed to
diff --git a/site/src/cut/lib/mediaProxy.ts b/site/src/cut/lib/mediaProxy.ts
index ed1932f4..b02f2bb1 100644
--- a/site/src/cut/lib/mediaProxy.ts
+++ b/site/src/cut/lib/mediaProxy.ts
@@ -23,6 +23,11 @@
* and carries the route, else nobody — the one fallback, shown in the panel.
* The result always lands in the project's own storage and is written onto
* the asset through the store.
+ *
+ * A browser that cannot decode a master's H.264 or AAC gets the compat proxy
+ * in the same slot (proxyNames.ts), from the engine, the worker, or for a
+ * browser project the Mac or the worker by way of the cloud shelf. The page's
+ * readers of that stream then open the proxy (mediaRead.ts `coveredSrc`).
*/
import { create } from "zustand";
@@ -31,12 +36,14 @@ import { getBackend, type CutBackend } from "./backend";
import { mediaDir } from "./backend/browser/opfs";
import { resolveRegisteredBlob } from "./backend/browser/registry";
import { pollCloudJob } from "./cloudJob";
+import { compatInTheCloud } from "./cloudLibraryUpload";
+import { compatNeedOf } from "./codecSupport";
import { cutColor } from "./colorSettings";
-import { uploadProjectMediaTo } from "./media";
-import { frameSinkIsCustom, hasUndecodableVideo, probeMediaFile, withMedia } from "./mediaRead";
+import { enrichAsset, uploadProjectMediaTo } from "./media";
+import { frameSinkIsCustom, hasUndecodableVideo, probeMediaFile, setStreamCovers, withMedia } from "./mediaRead";
import { forgetLocalMediaUrl, storedMediaUrl } from "./mediaSync";
import { encodeProxyInPage, proxyNameFor, proxyOnTheMac, type Landed } from "./proxyEncode";
-import { isLightProxyName } from "./proxyNames";
+import { compatCoverOf, compatCovers, compatProxyNameFor, isLightProxyName, type CompatCover } from "./proxyNames";
import { releaseProxyHold, useEditor } from "./store";
import type { MediaAsset, StoredAsset } from "./types";
@@ -114,11 +121,30 @@ export const useProxyJobs = create<ProxyJobs>((set) => ({
const inFlight = new Set<string>();
+/** The proxy an asset wants in this page: the compat proxy of a stream this
+ * browser cannot decode (light for a heavy video the cloud streams), or the
+ * preview proxy of a ProRes or heavy master. */
+type ProxyAsk = { kind: "compat"; cover: CompatCover; light: boolean } | { kind: "preview" };
+
+async function proxyAsk(asset: MediaAsset, backend: CutBackend): Promise<ProxyAsk | null> {
+ const streamed = backend.kind === "cloud";
+ const cover = await compatNeedOf(asset);
+ if (cover) {
+ const light = streamed && isHeavyVideo(asset);
+ const held = asset.proxy?.fileName;
+ if (held && compatCovers(held, cover) && (!light || isLightProxyName(held))) return null;
+ return { kind: "compat", cover, light };
+ }
+ if (!needsProxy(asset, streamed)) return null;
+ if (!isProres(asset) && !(await proxyDecodes())) return null;
+ return { kind: "preview" };
+}
+
/**
- * Make sure the asset has a proxy: build one through the residency that can,
- * and write it onto the asset. Runs once per asset at a time; a second ask
- * while one is on its way does nothing. A failure leaves the asset on the
- * master and says so in the panel.
+ * Make sure the asset has the proxy this page needs: build one through the
+ * residency that can, and write it onto the asset. Runs once per asset at a
+ * time; a second ask while one is on its way does nothing. A failure leaves
+ * the asset on the master and says so in the panel.
*/
export async function ensureProxy(
projectId: string,
@@ -129,31 +155,79 @@ export async function ensureProxy(
// chat tool and reads the master; the page queues the proxy when the
// project is next opened.
if (frameSinkIsCustom()) return;
- if (!needsProxy(asset, backend.kind === "cloud") || asset.upload || inFlight.has(asset.id)) return;
+ watchCovers();
+ if (asset.upload || asset.block || inFlight.has(asset.id)) return;
inFlight.add(asset.id);
- if (!isProres(asset) && !(await proxyDecodes())) {
- inFlight.delete(asset.id);
- return;
- }
- const jobs = useProxyJobs.getState();
- const progress = (p: number) => jobs.set(asset.id, { kind: "making", progress: Math.max(0, Math.min(1, p)) });
- progress(0);
try {
- const landed = await buildProxy(projectId, asset, backend, progress);
- if (!landed) {
- jobs.set(asset.id, { kind: "none" });
- return;
+ const ask = await proxyAsk(asset, backend);
+ if (!ask) return;
+ const jobs = useProxyJobs.getState();
+ const progress = (p: number) => jobs.set(asset.id, { kind: "making", progress: Math.max(0, Math.min(1, p)) });
+ progress(0);
+ try {
+ const landed =
+ ask.kind === "compat"
+ ? await buildCompat(projectId, asset, backend, ask, progress)
+ : await buildProxy(projectId, asset, backend, progress);
+ if (!landed) {
+ jobs.set(asset.id, { kind: "none" });
+ return;
+ }
+ const proxyUrl = await storedMediaUrl(projectId, landed.fileName, backend);
+ useEditor.getState().setAssetProxy(asset.id, { fileName: landed.fileName, sizeBytes: landed.sizeBytes }, proxyUrl);
+ jobs.set(asset.id, null);
+ // The strip and the waveform of a stream this page could not read
+ // waited for the compat proxy; they read it now.
+ const live = useEditor.getState().assets.find((a) => a.id === asset.id);
+ if (ask.kind === "compat" && live) void enrichAsset(live);
+ } catch (e) {
+ jobs.set(asset.id, { kind: "failed", error: e instanceof Error ? e.message : "Could not make the preview proxy." });
}
- const proxyUrl = await storedMediaUrl(projectId, landed.fileName, backend);
- useEditor.getState().setAssetProxy(asset.id, { fileName: landed.fileName, sizeBytes: landed.sizeBytes }, proxyUrl);
- jobs.set(asset.id, null);
- } catch (e) {
- jobs.set(asset.id, { kind: "failed", error: e instanceof Error ? e.message : "Could not make the preview proxy." });
} finally {
inFlight.delete(asset.id);
}
}
+/** Whether the asset waits on a compat proxy for `stream`: this browser
+ * cannot decode it, and the asset holds no proxy that covers it yet. */
+export async function awaitsCompat(asset: MediaAsset, stream: CompatCover): Promise<boolean> {
+ if (frameSinkIsCustom()) return false;
+ const need = await compatNeedOf(asset);
+ if (!need || (stream === "video" && need !== "video")) return false;
+ return !asset.proxy || !compatCovers(asset.proxy.fileName, need);
+}
+
+let coverPass = 0;
+
+/** Point this page's readers of each stream it cannot decode at the asset's
+ * compat proxy (mediaRead.ts). Every other browser reads the masters, so a
+ * proxy another browser built changes nothing here. */
+async function syncCovers(assets: MediaAsset[]): Promise<void> {
+ const pass = ++coverPass;
+ const video = new Map<string, string>();
+ const audio = new Map<string, string>();
+ for (const a of assets) {
+ if (!a.proxy || !a.proxyUrl || !compatCoverOf(a.proxy.fileName)) continue;
+ const need = await compatNeedOf(a);
+ if (!need || !compatCovers(a.proxy.fileName, need)) continue;
+ audio.set(a.url, a.proxyUrl);
+ if (need === "video") video.set(a.url, a.proxyUrl);
+ }
+ if (pass === coverPass) setStreamCovers({ video, audio });
+}
+
+let watchingCovers = false;
+
+/** Keep the covers in step with the doc from the first proxy ask on. */
+function watchCovers(): void {
+ if (watchingCovers) return;
+ watchingCovers = true;
+ void syncCovers(useEditor.getState().assets);
+ useEditor.subscribe((s, prev) => {
+ if (s.assets !== prev.assets) void syncCovers(s.assets);
+ });
+}
+
const healing = new Set<string>();
/**
@@ -212,12 +286,64 @@ async function proxyCodecMissing(asset: MediaAsset, backend: CutBackend): Promis
}
}
-/** Every asset in the open project that still needs a proxy. */
+/** Every asset in the open project that still needs a proxy here. */
export function ensureProjectProxies(): void {
const s = useEditor.getState();
if (!s.projectId || s.readOnly) return;
- const streamed = getBackend().kind === "cloud";
- for (const asset of s.assets) if (needsProxy(asset, streamed)) void ensureProxy(s.projectId, asset);
+ for (const asset of s.assets) if (asset.type !== "image") void ensureProxy(s.projectId, asset);
+}
+
+/** Ask the project's engine or worker for a proxy. The engine builds it at
+ * once; the cloud answers with one already built or a job to poll. */
+async function requestProxy(
+ projectId: string,
+ backend: CutBackend,
+ body: Record<string, unknown>,
+ onProgress: (p: number) => void
+): Promise<Landed> {
+ const res = await backend.fetch(`/api/cut/projects/${projectId}/proxy`, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify(body),
+ });
+ const started = (await res.json().catch(() => ({}))) as Partial<Landed> & { jobId?: string; error?: string };
+ if (!res.ok) throw new Error(started.error ?? "Could not make the preview proxy.");
+ if (started.fileName && typeof started.sizeBytes === "number") {
+ return { fileName: started.fileName, sizeBytes: started.sizeBytes };
+ }
+ if (backend.kind !== "cloud" || !started.jobId) throw new Error(started.error ?? "Could not make the preview proxy.");
+ const result = await pollCloudJob<Partial<Landed>>(started.jobId, backend, "Could not make the preview proxy.", {
+ timedOut: "The preview proxy took too long.",
+ onProgress,
+ });
+ if (!result.fileName || typeof result.sizeBytes !== "number") throw new Error("Could not make the preview proxy.");
+ return { fileName: result.fileName, sizeBytes: result.sizeBytes };
+}
+
+/** Build the compat proxy through the residency that can. A browser project
+ * hands the master to the Mac for a sound cover, and otherwise to the worker
+ * by way of the cloud shelf: the Mac's ffmpeg carries no VP9 encoder. */
+async function buildCompat(
+ projectId: string,
+ asset: MediaAsset,
+ backend: CutBackend,
+ ask: Extract<ProxyAsk, { kind: "compat" }>,
+ onProgress: (p: number) => void
+): Promise<Landed | null> {
+ const { proxyMaxHeight, proxyCrf } = cutColor();
+ if (backend.kind === "local") {
+ if (!(await engineFeatures()).has("media.compat")) return null;
+ return requestProxy(projectId, backend, { file: asset.fileName, maxHeight: proxyMaxHeight, crf: proxyCrf, compat: ask.cover }, onProgress);
+ }
+ if (backend.kind === "cloud") {
+ const light = ask.light ? { maxHeight: Math.min(proxyMaxHeight, HEAVY_PROXY_MAX_HEIGHT) } : {};
+ return requestProxy(projectId, backend, { file: asset.fileName, compat: ask.cover, ...light }, onProgress);
+ }
+ const master = async () => resolveRegisteredBlob(asset.url) ?? (await (await fetch(asset.url)).blob());
+ const onTheMac = ask.cover === "audio" ? await proxyOnTheMac(master, asset.fileName, proxyMaxHeight, proxyCrf, ask.cover) : null;
+ const blob = onTheMac ?? (await compatInTheCloud(new File([await master()], asset.fileName), ask.cover, onProgress));
+ const fileName = await uploadProjectMediaTo(backend, projectId, blob, compatProxyNameFor(asset.fileName, ask.cover, false));
+ return { fileName, sizeBytes: blob.size };
}
async function buildProxy(
@@ -229,39 +355,13 @@ async function buildProxy(
const { proxyMaxHeight, proxyCrf } = cutColor();
if (backend.kind === "local") {
if (!(await engineFeatures()).has("media.proxy")) return null;
- const res = await backend.fetch(`/api/cut/projects/${projectId}/proxy`, {
- method: "POST",
- headers: { "Content-Type": "application/json" },
- body: JSON.stringify({ file: asset.fileName, maxHeight: proxyMaxHeight, crf: proxyCrf }),
- });
- const body = (await res.json().catch(() => ({}))) as Partial<Landed> & { error?: string };
- if (!res.ok || !body.fileName || typeof body.sizeBytes !== "number") {
- throw new Error(body.error ?? "Could not make the preview proxy.");
- }
- return { fileName: body.fileName, sizeBytes: body.sizeBytes };
+ return requestProxy(projectId, backend, { file: asset.fileName, maxHeight: proxyMaxHeight, crf: proxyCrf }, onProgress);
}
if (backend.kind === "cloud") {
// A heavy video's proxy is there to stream light; a ProRes master's
// keeps the account's size.
const light = isProres(asset) ? {} : { maxHeight: Math.min(proxyMaxHeight, HEAVY_PROXY_MAX_HEIGHT) };
- const res = await backend.fetch(`/api/cut/projects/${projectId}/proxy`, {
- method: "POST",
- headers: { "Content-Type": "application/json" },
- body: JSON.stringify({ file: asset.fileName, ...light }),
- });
- const started = (await res.json().catch(() => ({}))) as Partial<Landed> & { jobId?: string; error?: string };
- if (!res.ok) throw new Error(started.error ?? "Could not make the preview proxy.");
- // A proxy already built answers with the file itself.
- if (started.fileName && typeof started.sizeBytes === "number") {
- return { fileName: started.fileName, sizeBytes: started.sizeBytes };
- }
- if (!started.jobId) throw new Error(started.error ?? "Could not make the preview proxy.");
- const result = await pollCloudJob<Partial<Landed>>(started.jobId, backend, "Could not make the preview proxy.", {
- timedOut: "The preview proxy took too long.",
- onProgress,
- });
- if (!result.fileName || typeof result.sizeBytes !== "number") throw new Error("Could not make the preview proxy.");
- return { fileName: result.fileName, sizeBytes: result.sizeBytes };
+ return requestProxy(projectId, backend, { file: asset.fileName, ...light }, onProgress);
}
// A browser project: the page first, writing straight into the project's
// storage, then the Mac beside it.
diff --git a/site/src/cut/lib/mediaRead.test.ts b/site/src/cut/lib/mediaRead.test.ts
index c5479d27..6e5dd318 100644
--- a/site/src/cut/lib/mediaRead.test.ts
+++ b/site/src/cut/lib/mediaRead.test.ts
@@ -1,6 +1,14 @@
import { describe, expect, test } from "bun:test";
+import { execFileSync } from "node:child_process";
+import { stubModule } from "@/lib/testing/stubModule";
import { CHUNK_SIZE } from "./chunkCache";
-import { spsHooks, urlParallelism } from "./mediaRead";
+import { hasUndecodableVideo, probeMediaFile, readMediaFileSize, spsHooks, UnreadableMediaError, urlParallelism, videoTrackOf } from "./mediaRead";
+
+/** What the stubbed browser decodes; each test sets it. */
+let decodes = { h264: true, aac: true };
+await stubModule<typeof import("./codecSupport")>("./codecSupport", import.meta.url, {
+ browserCodecs: async () => decodes,
+});
/** The cache one reader is given by the memory budget on an ordinary machine. */
const READER_CACHE = 8 * 2 ** 20;
@@ -50,3 +58,95 @@ describe("SPS color hooks", () => {
}
});
});
+
+describe("decodability", () => {
+ test("a track whose header mediabunny can't parse reads as undecodable", async () => {
+ // A corrupt SPS throws from inside canDecode ("Invalid exponential-Golomb
+ // code."): the track exists, and no decoder here can open it.
+ const track = { canDecode: async () => { throw new Error("Invalid exponential-Golomb code."); }, codec: "avc" };
+ const input = { getPrimaryVideoTrack: async () => track } as never;
+ // A page: the decodability gate only runs where something plays.
+ const doc = Object.getOwnPropertyDescriptor(globalThis, "document");
+ Object.defineProperty(globalThis, "document", { value: {}, configurable: true });
+ try {
+ expect(await videoTrackOf(input)).toBeNull();
+ expect(await hasUndecodableVideo(input)).toBe(true);
+ } finally {
+ if (doc) Object.defineProperty(globalThis, "document", doc);
+ else delete (globalThis as { document?: unknown }).document;
+ }
+ });
+});
+
+describe("H.264 in a browser without it", () => {
+ // Firefox on Linux without the system's codecs decodes no H.264 at all. The
+ // footage imports as it is, from its header, and a compat proxy carries its
+ // picture (mediaProxy.ts); a converter would only write H.264 again.
+ const h264 = () => {
+ const bytes = execFileSync("ffmpeg", [
+ "-hide_banner", "-loglevel", "error", "-f", "lavfi", "-i", "color=s=64x48:r=10:d=0.5",
+ "-c:v", "libx264", "-f", "mp4", "-movflags", "frag_keyframe+empty_moov", "pipe:1",
+ ]);
+ return new Blob([new Uint8Array(bytes)], { type: "video/mp4" });
+ };
+
+ const asPage = async <T>(run: () => Promise<T>): Promise<T> => {
+ const doc = Object.getOwnPropertyDescriptor(globalThis, "document");
+ Object.defineProperty(globalThis, "document", { value: {}, configurable: true });
+ try {
+ return await run();
+ } finally {
+ if (doc) Object.defineProperty(globalThis, "document", doc);
+ else delete (globalThis as { document?: unknown }).document;
+ }
+ };
+
+ test("imports from its header with its picture's size", async () => {
+ decodes = { h264: false, aac: true };
+ const probe = await asPage(() => probeMediaFile(h264()));
+ expect(probe).toMatchObject({ hasVideo: true, width: 64, height: 48 });
+ });
+
+ test("a browser that has H.264 and still cannot open the track converts it", async () => {
+ decodes = { h264: true, aac: true };
+ const err = await asPage(() => probeMediaFile(h264())).catch((e: unknown) => e);
+ expect(err instanceof UnreadableMediaError && err.undecodable).toBe(true);
+ });
+});
+
+describe("unparseable containers", () => {
+ test("a container this page can't parse goes to a converter", async () => {
+ // ffmpeg on this Mac or in the cloud reads far more containers than the
+ // page's demuxers, so the import hands the file on.
+ const err = await probeMediaFile(new Blob([new Uint8Array(4096).fill(7)])).catch((e: unknown) => e);
+
+ expect(err instanceof UnreadableMediaError && err.undecodable).toBe(true);
+ });
+});
+
+describe("url reads", () => {
+ // HTTP statuses a media server answers for a file that is gone, or a range
+ // past the end of one that is empty.
+ const NOT_FOUND = 404;
+ const RANGE_NOT_SATISFIABLE = 416;
+
+ for (const status of [NOT_FOUND, RANGE_NOT_SATISFIABLE]) {
+ test(`a ${status} comes back named, once`, async () => {
+ let asked = 0;
+ const real = globalThis.fetch;
+ globalThis.fetch = (async () => {
+ asked++;
+ return new Response(null, { status });
+ }) as unknown as typeof fetch;
+ try {
+ const err = await readMediaFileSize("https://files.example.com/a.mp4").catch((e: unknown) => e);
+
+ expect((err as Error).name).toBe("MediaFetchError");
+ expect((err as Error).message).toContain(String(status));
+ expect(asked).toBe(1);
+ } finally {
+ globalThis.fetch = real;
+ }
+ });
+ }
+});
diff --git a/site/src/cut/lib/mediaRead.ts b/site/src/cut/lib/mediaRead.ts
index e20b3270..f4fe9925 100644
--- a/site/src/cut/lib/mediaRead.ts
+++ b/site/src/cut/lib/mediaRead.ts
@@ -47,6 +47,7 @@ import { CHUNK_SIZE, chunkSourceOptions } from "./chunkCache";
import { probeColor, probeMissingColors, rangeOfBlob, withFoundColors, type ColorProbe, type ReadRange } from "./colorProbe";
import { pictureAssetIds, type ItemLists } from "./itemKinds";
import { allowance, holdMemory } from "./memoryBudget";
+import "./mediabunnyLog";
import { SpsColorRewriter, descriptionLengthSize, spsCodecOf, tagsOfColorSpace } from "./spsColor";
/** What a file turns out to be, read from its container. */
@@ -73,7 +74,7 @@ export interface FrameSize {
}
export class UnreadableMediaError extends Error {
- /** The container parsed, but a media stream needs a decoder this runtime lacks. */
+ /** The file needs a demuxer or a decoder this runtime lacks; a converter elsewhere may read it. */
readonly undecodable: boolean;
constructor(message = "Cut can't read this media file.", undecodable = false) {
@@ -129,9 +130,12 @@ const LOCAL_RETRY_DELAYS_S = [0.5, 1.5];
const urlRetryDelay = (
previousAttempts: number,
- _error: unknown,
+ error: unknown,
src: string | URL | Request
): number | null => {
+ // A server's refusal is its answer: a gone file or a range past the end of
+ // an empty one reads the same on the next try.
+ if (error instanceof MediaFetchError && error.status !== undefined) return null;
// Headless has nothing to hand a giving-up read back to: the engine and the
// render worker open a file once, and a read that quits fails the job. They
// keep waiting, on a widening backoff, until the job's own cancellation
@@ -150,9 +154,13 @@ const urlRetryDelay = (
* error tracking matches on (instrumentation-client.ts) and what keeps a
* caller's own report of it worded as weather (report.ts). */
class MediaFetchError extends Error {
- constructor(message: string, cause?: unknown) {
+ /** The server's answer, for a read it refused. */
+ readonly status?: number;
+
+ constructor(message: string, cause?: unknown, status?: number) {
super(message, { cause });
this.name = "MediaFetchError";
+ this.status = status;
}
}
@@ -176,11 +184,20 @@ function trackedFetch(url: string): typeof fetch {
if (engine && !engineConnected()) {
return Promise.reject(new MediaFetchError("The Donkey app isn't running."));
}
- return fetch(input, init).catch((err: unknown) => {
- if (aborted(err)) throw err;
- if (engine) void confirmEngine();
- throw new MediaFetchError("Cut couldn't read this media over the network.", err);
- });
+ return fetch(input, init).then(
+ (res) => {
+ // A refusal is named here as well: the reader's readahead rejects with
+ // nothing awaiting it, and its own wording names neither the read nor
+ // this module.
+ if (!res.ok) throw new MediaFetchError(`Cut couldn't read this media (${res.status}).`, undefined, res.status);
+ return res;
+ },
+ (err: unknown) => {
+ if (aborted(err)) throw err;
+ if (engine) void confirmEngine();
+ throw new MediaFetchError("Cut couldn't read this media over the network.", err);
+ }
+ );
};
}
@@ -332,12 +349,13 @@ const headless = () => typeof document === "undefined";
/** A file the container parser doesn't recognize reads back as this module's
* own error. mediabunny's own text names no file and reads like a bug report,
* and it reaches the user wherever a read runs — a caption run, an export, a
- * chat tool. */
+ * chat tool. It is marked undecodable: ffmpeg reads far more containers, so an
+ * import hands the file to a converter. */
async function tracked<T>(read: Promise<T>): Promise<T> {
try {
return await read;
} catch (err) {
- if (err instanceof UnsupportedInputFormatError) throw new UnreadableMediaError();
+ if (err instanceof UnsupportedInputFormatError) throw new UnreadableMediaError(undefined, true);
throw err;
}
}
@@ -357,7 +375,10 @@ export async function videoTrackOf(input: Input): Promise<InputVideoTrack | null
* landed, the in-tab export, which always reads the master. */
async function decodable(track: InputVideoTrack): Promise<boolean> {
if (headless()) return true;
- if (await track.canDecode()) return true;
+ // canDecode parses the stream's own header to build the decoder config, and
+ // a corrupt one throws from the parser ("Invalid exponential-Golomb code.").
+ // No decoder here opens that track, which is what false says.
+ if (await track.canDecode().catch(() => false)) return true;
if (track.codec !== "prores") return false;
await (await import("./proresDecoder")).ensureProresDecoder();
return track.canDecode();
@@ -375,10 +396,24 @@ export async function hasUndecodableVideo(input: Input): Promise<boolean> {
return !!track && !(await decodable(track));
}
+/** The primary video track when it is H.264 in a browser with no H.264
+ * decoder at all (Firefox on Linux without the system's codecs). Its picture
+ * plays from a compat proxy (mediaProxy.ts), so the master imports as it is;
+ * a converter would only write H.264 again. A header the parser cannot read
+ * throws from canDecode, and that file goes to the converters. */
+async function coverableVideo(input: Input): Promise<InputVideoTrack | null> {
+ if (headless()) return null;
+ const track = await tracked(input.getPrimaryVideoTrack());
+ if (!track || track.codec !== "avc") return null;
+ if (await track.canDecode().catch(() => true)) return null;
+ const { browserCodecs } = await import("./codecSupport");
+ return (await browserCodecs()).h264 ? null : track;
+}
+
/** The primary audio track, or null — same decodability rule as video. */
export async function audioTrackOf(input: Input): Promise<InputAudioTrack | null> {
return tracked(input.getPrimaryAudioTrack({
- filter: (track) => headless() ? track.codec !== null : track.canDecode(),
+ filter: (track) => headless() ? track.codec !== null : track.canDecode().catch(() => false),
}));
}
@@ -388,8 +423,9 @@ export async function audioTrackOf(input: Input): Promise<InputAudioTrack | null
* asset that plays black. */
export async function probeMediaFile(src: string | Blob): Promise<MediaProbe> {
return withMedia(src, async (input) => {
- if (!(await input.canRead())) throw new UnreadableMediaError();
- const [video, audio] = await Promise.all([videoTrackOf(input), audioTrackOf(input)]);
+ if (!(await input.canRead())) throw new UnreadableMediaError(undefined, true);
+ const [decoded, audio] = await Promise.all([videoTrackOf(input), audioTrackOf(input)]);
+ const video = decoded ?? (await coverableVideo(input));
// Footage this browser can't decode is refused outright. Treating it as
// audio would be a quieter failure, not a smaller one: the clip would look
// imported and simply have no picture.
@@ -464,7 +500,6 @@ function rangeReader(src: string | Blob): ReadRange {
mode: "cors",
headers: { Range: `bytes=${offset}-${offset + length - 1}` },
});
- if (!res.ok) throw new MediaFetchError(`Cut couldn't read the media header (${res.status}).`);
const bytes = new Uint8Array(await res.arrayBuffer());
return res.status === 206 ? bytes : bytes.subarray(offset, offset + length);
};
@@ -627,6 +662,23 @@ function capToTrack(track: InputVideoTrack, size?: FrameSize): FrameSize | undef
};
}
+/** Each stream a compat proxy covers in this page (mediaProxy.ts), by its
+ * master's address: the readers of that stream open the proxy. A render's
+ * clip reader opens the master itself, so an export reads the original. */
+const streamCovers = { video: new Map<string, string>(), audio: new Map<string, string>() };
+
+/** Replace the covers this page reads through: master address → proxy
+ * address, per stream. */
+export function setStreamCovers(covers: { video: Map<string, string>; audio: Map<string, string> }): void {
+ streamCovers.video = covers.video;
+ streamCovers.audio = covers.audio;
+}
+
+/** Where this page reads `stream` of `src` from. */
+export function coveredSrc<T extends string | Blob>(stream: "video" | "audio", src: T): T | string {
+ return typeof src === "string" ? (streamCovers[stream].get(src) ?? src) : src;
+}
+
/**
* Frames at the given times, in one decode pass.
*
@@ -640,7 +692,7 @@ export async function* framesAt(
size?: FrameSize,
opts?: FrameSinkOptions
): AsyncGenerator<WrappedCanvas | null> {
- const input = openMedia(src);
+ const input = openMedia(coveredSrc("video", src));
try {
const track = await videoTrackOf(input);
if (!track) throw new UnreadableMediaError("This file has no readable video.");
@@ -664,7 +716,7 @@ export async function frameAt(
time: number,
size?: FrameSize
): Promise<WrappedCanvas | null> {
- return withMedia(src, async (input) => {
+ return withMedia(coveredSrc("video", src), async (input) => {
const track = await videoTrackOf(input);
if (!track) throw new UnreadableMediaError("This file has no readable video.");
return frameSink(track, size).getCanvas(Math.max(0, time));
@@ -683,7 +735,7 @@ export async function* audioChunks(
from?: number,
to?: number
): AsyncGenerator<WrappedAudioBuffer> {
- const input = openMedia(src);
+ const input = openMedia(coveredSrc("audio", src));
try {
const track = await audioTrackOf(input);
if (!track) return;
@@ -795,7 +847,8 @@ export interface AudioWalk {
* shape the frame readers hold, and it is what lets one open file, one
* container parse and one decoder serve a whole clip.
*/
-export function openAudioWalk(src: string | Blob, from = 0, to?: number): AudioWalk {
+export function openAudioWalk(master: string | Blob, from = 0, to?: number): AudioWalk {
+ const src = coveredSrc("audio", master);
// A voice on a URL reads the file its picture's source already holds open.
const handle =
typeof src === "string"
@@ -895,7 +948,7 @@ export async function audioPeaks(
let from = 0;
for (;;) {
await gate?.idle();
- const input = openMedia(src);
+ const input = openMedia(coveredSrc("audio", src));
let read = 0;
let done = true;
try {
diff --git a/site/src/cut/lib/projectCopy.ts b/site/src/cut/lib/projectCopy.ts
index a9bc9c22..6778de99 100644
--- a/site/src/cut/lib/projectCopy.ts
+++ b/site/src/cut/lib/projectCopy.ts
@@ -11,6 +11,7 @@ import { ensureLinkedOnCloud, linkedIdsIn } from "./linkedLibrary";
import { uploadProjectMediaTo } from "./media";
import { localMediaFile } from "./mediaSync";
import type { ProjectDoc, StoredAsset } from "./types";
+import { compatCoverOf } from "./proxyNames";
/** Move a project's conversations onto the copy.
*
@@ -127,11 +128,13 @@ export async function copyProjectAcross(
// each onto its asset when the project is next opened.
for (const a of assets) {
if (!a.proxy) continue;
+ // A compat proxy is asked for as the cover it was.
+ const compat = compatCoverOf(a.proxy.fileName);
void dst
.fetch(`/api/cut/projects/${summary.id}/proxy`, {
method: "POST",
headers: { "Content-Type": "application/json" },
- body: JSON.stringify({ file: names.get(a.fileName) ?? a.fileName }),
+ body: JSON.stringify({ file: names.get(a.fileName) ?? a.fileName, ...(compat ? { compat } : {}) }),
})
.catch(() => {});
}
diff --git a/site/src/cut/lib/proxyEncode.ts b/site/src/cut/lib/proxyEncode.ts
index e3749129..c388419f 100644
--- a/site/src/cut/lib/proxyEncode.ts
+++ b/site/src/cut/lib/proxyEncode.ts
@@ -25,6 +25,7 @@ import {
import { apiFetch as engineFetch, engineFeatures } from "./api";
import { hasLocalCompute } from "./backend";
import { freeName } from "./backend/browser/opfs";
+import type { CompatCover } from "./proxyNames";
import { audioTrackOf, openMedia, probeMediaFile, videoTrackOf } from "./mediaRead";
import { encoderLead } from "./encoderLead";
import { ensureProresDecoder } from "./proresDecoder";
@@ -231,20 +232,24 @@ async function convertSample(
});
}
-/** Hand the master's bytes to the Mac and take the proxy back; the engine
- * keeps nothing. Null when the app is not there or is older than the route. */
+/** Hand the master's bytes to the Mac and take the proxy back, or the compat
+ * proxy `compat` names (proxyNames.ts); the engine keeps nothing. Null when
+ * the app is not there or is older than the route. */
export async function proxyOnTheMac(
master: () => Promise<Blob>,
masterName: string,
maxHeight: number,
- crf: number
+ crf: number,
+ compat?: CompatCover
): Promise<Blob | null> {
- if (!hasLocalCompute() || !(await engineFeatures()).has("media.proxy")) return null;
+ if (!hasLocalCompute()) return null;
+ if (!(await engineFeatures()).has(compat ? "media.compat" : "media.proxy")) return null;
const bytes = await master();
const form = new FormData();
form.append("file", bytes, masterName);
form.append("maxHeight", String(maxHeight));
form.append("crf", String(crf));
+ if (compat) form.append("compat", compat);
const res = await engineFetch("/api/cut/proxy", { method: "POST", body: form });
if (!res.ok) {
const body = (await res.json().catch(() => ({}))) as { error?: string };
diff --git a/site/src/cut/server/cloud/mediaResume.test.ts b/site/src/cut/server/cloud/mediaResume.test.ts
new file mode 100644
index 00000000..b11edbe0
--- /dev/null
+++ b/site/src/cut/server/cloud/mediaResume.test.ts
@@ -0,0 +1,59 @@
+import { describe, expect, test } from "bun:test";
+
+import { stubModule } from "@/lib/testing/stubModule";
+
+// The project's media rows, as the cases leave them, and every row the handler
+// creates.
+let rows: { fileName: string; bytes: bigint; uploadState: string; r2Key: string }[] = [];
+const created: { fileName: string; uploadState: string }[] = [];
+
+await stubModule("@/lib/prisma", import.meta.url, {
+ prisma: {
+ cutMediaObject: {
+ findFirst: async ({ where }: { where: { fileName: string } }) => rows.find((r) => r.fileName === where.fileName) ?? null,
+ create: async ({ data }: { data: { fileName: string; uploadState: string } }) => {
+ created.push(data);
+ return data;
+ },
+ },
+ },
+} as never);
+await stubModule("./projects", import.meta.url, {
+ getProject: async () => ({ id: "p1" }),
+ takenMediaNames: async () => new Set(rows.map((r) => r.fileName)),
+} as never);
+await stubModule("./r2", import.meta.url, { presignPut: async () => "https://put" } as never);
+await stubModule("./usage", import.meta.url, { quotaCheck: async () => null } as never);
+const { mediaCloud } = await import("./media");
+
+/** Ask for a fresh URL for a claim the browser still holds the bytes of. */
+const resume = async (fileName: string, bytes: number) => {
+ const req = new Request("https://donkeycut.com/api/cut-cloud/projects/p1/media/presign", {
+ method: "POST",
+ body: JSON.stringify({ fileName, bytes, mime: "video/mp4", resume: true }),
+ });
+ return (await (await mediaCloud.presign("u1", "p1", req)).json()) as { fileName: string; url?: string };
+};
+
+describe("resuming an upload", () => {
+ test("re-claims the same name when the sweep dropped the stale claim", async () => {
+ rows = [];
+ created.length = 0;
+
+ const answer = await resume("take.mp4", 100);
+
+ expect(answer.fileName).toBe("take.mp4");
+ expect(answer.url).toBe("https://put");
+ expect(created).toMatchObject([{ fileName: "take.mp4", uploadState: "pending" }]);
+ });
+
+ test("re-signs a claim that is still there", async () => {
+ rows = [{ fileName: "take.mp4", bytes: BigInt(100), uploadState: "pending", r2Key: "cut/u1/projects/p1/media/take.mp4" }];
+ created.length = 0;
+
+ const answer = await resume("take.mp4", 100);
+
+ expect(answer.fileName).toBe("take.mp4");
+ expect(created).toEqual([]);
+ });
+});