Skip to content

Open source CapCut alternative.

TypeScript

feat(site): an export the browser can't encode renders on the worker [rebuild]

David Le0eb22e967461
diff --git a/docs/guides/cut/local-compute.md b/docs/guides/cut/local-compute.md
index 3fb11d8c..26f2e14a 100644
--- a/docs/guides/cut/local-compute.md
+++ b/docs/guides/cut/local-compute.md
@@ -43,7 +43,7 @@ A cloud project exports in the browser, and so does a browser project — the ta
That works because the editor composites the cut live to draw the preview. Rendering is the same drawing done on a clock the tab steps, so a browser and a container produce the same picture from the same document, and the browser needs nothing pulled out of storage that it is not already playing.
-Two facts decide whether a tab can carry a render, both probed per export: origin-private scratch storage to stream the file to, and a WebCodecs video encoder at the requested dimensions. A cut of any length renders in the tab, since the pipeline streams to disk and duration costs only time. A cloud project that fails either probe, or fails mid-render, goes to the worker silently. A browser project has no second machine, so the failure is the user's to resolve: the export says what went wrong and points at Move to Cloud.
+Two facts decide whether a tab can carry a render, both probed per export: origin-private scratch storage to stream the file to, and WebCodecs encoders for the codecs the user chose at the requested dimensions — the video one, and AAC when the cut has sound. The probe asks for exactly those codecs; nothing stands in for a missing one, because a file with a substitute codec inside is a file the user's player refuses. ProRes has no browser encoder, so a master always renders on the worker. A cut of any length renders in the tab, since the pipeline streams to disk and duration costs only time. A project that fails either probe, or fails mid-render, goes to the worker silently. For a cloud project the worker already holds the media. A browser project's media lives only in that tab, so it goes up with the job beside the overlay stills, the worker renders from those uploads, and the finished file comes back down into the project's own exports folder; the cloud keeps nothing afterwards. Whatever the user picked, something renders it — the only thing that stops a browser project's export is an account that cannot use the cloud, and the export says so.
The worker also takes the renders the editor fires on its own — hover proxies, share cards, streaming ladders. Moving export to the Mac would be a third path and there is no longer much to win from it.
@@ -75,7 +75,7 @@ In place is what makes the plain version of this work. Someone with a folder of
| Work | Runs | Why |
| --- | --- | --- |
| Transcription, dictation | the Mac when it is there, hosted otherwise | the app ships the speech tool |
-| Export | the Mac for its own projects, the browser for cloud and browser ones, the worker when a cloud project's tab can't encode | see Export above |
+| Export | the Mac for its own projects, the browser for cloud and browser ones, the worker when the tab can't encode the choice | see Export above |
| Thumbnails, waveforms, media probing | the browser, always | it decodes the media itself |
| Converting media to MP4 | the machine holding the bytes: the Mac, the worker, or the tab — and the Mac for a browser project the tab can't decode | see Converting footage above |
| Image, video, and voice generation | hosted, always | no local counterpart |
diff --git a/site/src/cut/lib/exportStore.ts b/site/src/cut/lib/exportStore.ts
index a3d33b14..31ce8ff4 100644
--- a/site/src/cut/lib/exportStore.ts
+++ b/site/src/cut/lib/exportStore.ts
@@ -10,7 +10,9 @@ import { localBackend } from "./backend/local";
import {
cancelExportJob,
createExportJob,
+ ExportRefusedError,
runBrowserExport,
+ runBrowserExportInCloud,
type ExportDoc,
type ExportSettings,
} from "./exportClient";
@@ -110,13 +112,16 @@ export const useExports = create<ExportsState>((set, get) => ({
// A cloud project renders in the tab: no upload of the cut to a container,
// no queue behind other accounts, and the file matches the preview because
// the same compositor drew both. A browser that can't carry the render —
- // no encoder at these dimensions, no scratch storage — sends it to the
- // worker, which has a whole machine. A browser-resident project renders in
- // the tab too; the tab is its machine.
+ // no encoder for the codec, no scratch storage — sends it to the worker,
+ // which has a whole machine. A browser-resident project renders in the tab
+ // too, and borrows the same worker when the tab can't: its media goes up
+ // with the job and the file comes back into its own storage. Whatever the
+ // user chose, something renders it.
const inBrowser =
(backend.kind === "cloud" || backend.kind === "browser") &&
(await canRenderInBrowser(doc, settings));
- const abort = inBrowser ? new AbortController() : undefined;
+ const tabOwned = backend.kind !== "local";
+ const abort = tabOwned ? new AbortController() : undefined;
set((s) => ({
local: [
...s.local,
@@ -124,8 +129,8 @@ export const useExports = create<ExportsState>((set, get) => ({
id: localId,
projectId,
projectName,
- status: inBrowser ? "rendering" : "preparing",
- ...(inBrowser ? { progress: 0 } : {}),
+ status: tabOwned ? "rendering" : "preparing",
+ ...(tabOwned ? { progress: 0 } : {}),
createdAt: Date.now(),
residency: backend.kind,
abort,
@@ -135,44 +140,69 @@ export const useExports = create<ExportsState>((set, get) => ({
// The loop may have been idling minutes deep; this export needs the fast
// cadence from its first frame.
wake();
- let claimedId: string | null = null;
+ // Every job row this export claims — a tab render's reservation, and for
+ // a borrowed render the cloud row too — stays hidden behind the local row.
+ let claimed: string[] = [];
const release = () => {
- if (claimedId) set((s) => ({ rendering: s.rendering.filter((id) => id !== claimedId) }));
+ const gone = new Set(claimed);
+ claimed = [];
+ set((s) => ({ rendering: s.rendering.filter((id) => !gone.has(id)) }));
};
const fail = (err: unknown) => {
const msg = err instanceof Error ? err.message : String(err);
- // A browser-resident project has exactly one other renderer, so a failed
- // render says where to take it.
- const hint = " Move the project to Cloud to export.";
- const full = backend.kind === "browser" && !msg.includes(hint) ? msg + hint : msg;
set((s) => ({
local: s.local.map((r) =>
- r.id === localId ? { ...r, status: "error" as const, error: full, abort: undefined } : r
+ r.id === localId ? { ...r, status: "error" as const, error: msg, abort: undefined } : r
),
}));
};
- try {
- if (backend.kind === "browser" && !inBrowser) {
- throw new Error("This browser can't render this export.");
+ const tabOpts = {
+ signal: abort?.signal,
+ projectName,
+ // The reservation is a real job row, so the feed would show it beside
+ // the local row that carries the progress. Hide it until this tab is
+ // done with it.
+ onClaimed: (jobId: string) => {
+ claimed.push(jobId);
+ set((s) => ({ rendering: [...new Set([...s.rendering, jobId])] }));
+ },
+ onProgress: (progress: number) =>
+ set((s) => ({
+ local: s.local.map((r) => (r.id === localId ? { ...r, progress } : r)),
+ })),
+ };
+ // The worker renders what the tab could not: a cloud project's own media
+ // is already there, a browser project's goes up with the job.
+ const renderOnWorker = async () => {
+ release();
+ if (backend.kind === "browser") {
+ await runBrowserExportInCloud(projectId, doc, settings, tabOpts);
+ return;
}
+ set((s) => ({
+ local: s.local.map((r) =>
+ r.id === localId ? { ...r, status: "preparing" as const, progress: undefined, abort: undefined } : r
+ ),
+ }));
+ await createExportJob(projectId, doc, settings);
+ };
+ try {
if (inBrowser) {
- await runBrowserExport(projectId, doc, settings, {
- signal: abort!.signal,
- projectName,
- // The reservation is a real job row, so the feed would show it beside
- // the local row that carries the progress. Hide it until this tab is
- // done with it.
- onClaimed: (jobId) => {
- claimedId = jobId;
- set((s) => ({ rendering: [...new Set([...s.rendering, jobId])] }));
- },
- onProgress: (progress) =>
- set((s) => ({
- local: s.local.map((r) => (r.id === localId ? { ...r, progress } : r)),
- })),
- });
+ try {
+ await runBrowserExport(projectId, doc, settings, tabOpts);
+ } catch (err) {
+ // The probe said yes and the browser then failed anyway — an encoder
+ // that refuses mid-stream, scratch storage that gives out. The
+ // export goes to the worker and the user sees it carry on, never an
+ // error for a render they could not have done anything about. A
+ // stopped render is theirs, and a gate's refusal is final: the
+ // worker sits behind the same gate.
+ if (err instanceof DOMException && err.name === "AbortError") throw err;
+ if (err instanceof ExportRefusedError) throw err;
+ await renderOnWorker();
+ }
} else {
- await createExportJob(projectId, doc, settings);
+ await renderOnWorker();
}
// Pull the finished (or queued) job into the feed *before* retiring the
// placeholder — dropping the local row first left a round-trip with
@@ -182,8 +212,9 @@ export const useExports = create<ExportsState>((set, get) => ({
// must not mark a started export as a start error, so it never reaches
// the catch below.
await get().refresh().catch(() => {});
+ const shown = new Set(claimed);
set((s) => ({
- rendering: s.rendering.filter((id) => id !== claimedId),
+ rendering: s.rendering.filter((id) => !shown.has(id)),
local: s.local.filter((r) => r.id !== localId),
}));
} catch (err) {
diff --git a/site/src/cut/server/cloud/jobs.ts b/site/src/cut/server/cloud/jobs.ts
index 3ffd9d07..44244bed 100644
--- a/site/src/cut/server/cloud/jobs.ts
+++ b/site/src/cut/server/cloud/jobs.ts
@@ -7,7 +7,9 @@ import { cutLimitsFor, EXPORT_QUOTA_MARGIN, renderJobCheck } from "./limits";
import { wakeRenderWorker } from "./wake";
import { getProject } from "./projects";
import { MEDIA_REDIRECT_HEADERS, mediaObjectUrl } from "./mediaCdn";
-import { del, head, overlayKey, presignPut, projectExportKey } from "./r2";
+import { del, head, overlayKey, overlayPrefix, presignPut, projectExportKey } from "./r2";
+import { contentTypeFor } from "../serveFile";
+import { containerOfName, deliveryContainer, specMediaFiles, type ExportContainer } from "../../lib/exportDelivery";
import { addUsage, quotaCheck, reservedBytes, usageBytes } from "./usage";
import { caught, err, redirect } from "./util";
@@ -33,9 +35,14 @@ const MAX_RENDER_CLIPS = 2000;
// carries one PNG per caption cue and one per frame of every animated
// element, so the count and the batch run to a long, busy cut.
const MAX_OVERLAY_FILES = 20_000;
-const MAX_OVERLAY_BYTES = 256 * 1024 ** 2;
+// A render's inputs: overlay stills, and for a browser-resident project the
+// cut's own media, which rides with the job because the worker has no other
+// way to reach a file that lives in a tab. A source clip can be as large as
+// any upload.
+const MAX_OVERLAY_BYTES = 4 * 1024 ** 3;
const MAX_OVERLAY_BATCH_BYTES = 8 * 1024 ** 3;
+
// The largest download an import may land, holding for unlimited tiers too.
const IMPORT_MAX_BYTES = 2 * 1024 ** 3;
@@ -116,17 +123,23 @@ async function findJob(userId: string, id: string): Promise<JobRow | null> {
return prisma.cutRenderJob.findFirst({ where: { id, userId } });
}
-/** Engine-style export name: `base.mp4` with a " 2", " 3"… suffix when taken
+/** Engine-style export name: the base with the container's extension and a
+ * " 2", " 3"… suffix when taken
* by an existing export object or a job still in flight. The client derives the
* base from the project name; the dedupe happens here, where the rows are. */
async function exportName(
userId: string,
projectId: string,
baseName: string,
+ container: ExportContainer | undefined,
tx: Prisma.TransactionClient | typeof prisma = prisma
) {
+ // The spec's container names the extension, the same way the engine's
+ // `containerExtension` does; a name alone (a tab's own claim) says it by its
+ // suffix.
+ const { ext } = container ? deliveryContainer(container) : containerOfName(baseName);
const base =
- baseName.replace(/\.mp4$/i, "").replace(/[/\\:*?"<>|]/g, "").trim().slice(0, 60) || "export";
+ baseName.replace(/\.(mp4|mov)$/i, "").replace(/[/\\:*?"<>|]/g, "").trim().slice(0, 60) || "export";
const [files, jobs] = await Promise.all([
tx.cutMediaObject.findMany({
where: { userId, projectId, kind: "export" },
@@ -145,7 +158,7 @@ async function exportName(
...jobs.map((j) => j.outName).filter((n): n is string => !!n),
]);
for (let n = 1; ; n++) {
- const candidate = n === 1 ? `${base}.mp4` : `${base} ${n}.mp4`;
+ const candidate = n === 1 ? `${base}${ext}` : `${base} ${n}${ext}`;
if (!taken.has(candidate)) return candidate;
}
}
@@ -177,18 +190,18 @@ export const jobsCloud = {
if (over) return over;
let total = 0;
for (const f of files) {
- if (!Number.isInteger(f.bytes) || f.bytes! < 1 || f.bytes! > MAX_OVERLAY_BYTES)
- return err("Every overlay needs its size.", 400);
+ if (!Number.isInteger(f.bytes) || f.bytes! < 1) return err("Every overlay needs its size.", 400);
+ if (f.bytes! > MAX_OVERLAY_BYTES) return err(`${f.name} is too large to render in the cloud.`, 413);
total += f.bytes!;
}
- if (total > MAX_OVERLAY_BATCH_BYTES) return err("Overlays are too large.", 413);
+ if (total > MAX_OVERLAY_BATCH_BYTES) return err("The cut's files are too large to render in the cloud.", 413);
const batchId = crypto.randomUUID().slice(0, 12);
const out = await Promise.all(
files.map(async (f) => {
const name = String(f.name ?? "").replace(/[/\\]/g, "");
if (!name) throw new Error("Every overlay needs a name.");
const key = overlayKey(userId, batchId, name);
- const type = name.toLowerCase().endsWith(".mp4") ? "video/mp4" : "image/png";
+ const type = contentTypeFor(name);
// The size is in the signature: a larger body is a 403 at the bucket.
return { name, key, type, url: await presignPut(key, type, f.bytes) };
})
@@ -204,17 +217,29 @@ export const jobsCloud = {
async exportCreate(userId: string, req: Request) {
try {
const body = (await req.json()) as {
- spec?: { target?: string; projectId?: string };
+ spec?: {
+ target?: string;
+ projectId?: string;
+ container?: ExportContainer;
+ clips?: { file?: string }[];
+ overlayVideos?: { file?: string }[];
+ audio?: { file?: string }[];
+ };
overlays?: { name: string; key: string }[];
projectId?: string;
outName?: string;
burnedSubtitles?: boolean;
+ /** "overlays": the cut's media was uploaded beside the stills, so the
+ * project it names is not one the cloud stores — a browser-resident
+ * project borrowing the worker. */
+ mediaFrom?: "overlays";
};
if (!body.spec || typeof body.spec !== "object") return err("spec is required.", 400);
const projectId = body.projectId ?? body.spec.projectId;
if (!projectId) return err("projectId is required.", 400);
+ const borrowed = body.mediaFrom === "overlays";
const project = await getProject(userId, projectId);
- if (!project) return err("Project not found.", 400);
+ if (!project && !borrowed) return err("Project not found.", 400);
// Overlay keys come from the client; only this user's own overlay
// uploads are acceptable render inputs — anything else would let a
// crafted key pull another account's R2 objects into the render.
@@ -224,6 +249,14 @@ export const jobsCloud = {
return err("Invalid overlay key.", 400);
}
}
+ if (borrowed) {
+ // Every file the cut plays has to have come up with the job; the
+ // worker has nowhere else to look for it.
+ const uploaded = new Set((body.overlays ?? []).map((o) => o.name));
+ const missing = specMediaFiles(body.spec).find((f) => !uploaded.has(f));
+ if (missing) return err(`${missing} was not uploaded with the export.`, 400);
+ if (!body.outName?.trim()) return err("outName is required.", 400);
+ }
// Hover proxies, share cards and share ladders are renders the editor
// fires on its own; only renders the user asked for count against the
// cap.
@@ -250,6 +283,7 @@ export const jobsCloud = {
const jobSpec = {
spec: body.spec,
overlays: body.overlays ?? [],
+ ...(borrowed ? { mediaFrom: "overlays" } : {}),
...(target === "hls" ? { burnedSubtitles: body.burnedSubtitles === true } : {}),
} as unknown as Prisma.InputJsonValue;
@@ -292,7 +326,7 @@ export const jobsCloud = {
}
const outName =
target === "export"
- ? await exportName(userId, projectId, body.outName?.trim() || project.name)
+ ? await exportName(userId, projectId, body.outName?.trim() || project?.name || "export", body.spec.container)
: target === "hls"
? "master.m3u8"
: `${target}.mp4`;
@@ -325,7 +359,7 @@ export const jobsCloud = {
// The worker owns the size list (it builds the spec), and falls back to
// the source-matched original for anything it does not know.
const preset = typeof body.preset === "string" ? body.preset : "original";
- const outName = await exportName(userId, projectId, project.name);
+ const outName = await exportName(userId, projectId, project.name, undefined);
const row = await prisma.cutRenderJob.create({
data: {
userId,
@@ -363,12 +397,14 @@ export const jobsCloud = {
});
// A settled row is being dismissed from the dock: it leaves the feed and
// stays on the books, so the day's render count still sees it. A finished
- // file survives as the project's export media object.
+ // file survives as the project's export media object — except a borrowed
+ // render's, which was the job's scratch and goes with the row.
if (canceled.count === 0) {
- await prisma.cutRenderJob.updateMany({
- where: { id: jobId, userId, state: { in: ["done", "error", "canceled"] } },
- data: { state: "dismissed" },
- });
+ const row = await findJob(userId, jobId);
+ if (row && ["done", "error", "canceled"].includes(row.state)) {
+ await prisma.cutRenderJob.update({ where: { id: row.id }, data: { state: "dismissed" } });
+ if (row.outputKey?.startsWith(overlayPrefix(userId))) await del([row.outputKey]);
+ }
}
return Response.json({ ok: true });
},
@@ -400,7 +436,7 @@ export const jobsCloud = {
// completion would then fail on the unique key with the render already
// spent. The row is the reservation.
const row = await prisma.$transaction(async (tx) => {
- const outName = await exportName(userId, projectId, body.outName ?? "export.mp4", tx);
+ const outName = await exportName(userId, projectId, body.outName ?? "export.mp4", undefined, tx);
return tx.cutRenderJob.create({
data: {
userId,
@@ -416,11 +452,13 @@ export const jobsCloud = {
});
});
const key = projectExportKey(userId, projectId, row.outName!);
+ const type = containerOfName(row.outName!).mime;
return Response.json({
jobId: row.id,
key,
outName: row.outName,
- url: await presignPut(key, "video/mp4"),
+ type,
+ url: await presignPut(key, type),
});
} catch (e) {
return caught(e, "Could not start the export.");
@@ -461,7 +499,7 @@ export const jobsCloud = {
projectId,
r2Key: key,
fileName: outName,
- mime: "video/mp4",
+ mime: containerOfName(outName).mime,
bytes: BigInt(bytes),
kind: "export",
uploadState: "complete",
diff --git a/site/src/cut/worker/Dockerfile b/site/src/cut/worker/Dockerfile
index 59185bb3..23b307ff 100644
--- a/site/src/cut/worker/Dockerfile
+++ b/site/src/cut/worker/Dockerfile
@@ -28,6 +28,10 @@ RUN ffmpeg -hide_banner -h full 2>&1 | grep -q -- '-display_rotation' \
# encode time, so it fails here instead.
RUN ffmpeg -hide_banner -encoders 2>&1 | grep -q '\blibx264\b' \
|| { echo "ffmpeg has no libx264: the share HLS ladder cannot encode." >&2; exit 1; }
+# HEVC exports a browser cannot encode itself land here; libx265 is their
+# only encoder on this image.
+RUN ffmpeg -hide_banner -encoders 2>&1 | grep -q '\blibx265\b' \
+ || { echo "ffmpeg has no libx265: HEVC exports cannot encode." >&2; exit 1; }
WORKDIR /app
# The browser primitives a job draws and decodes with: the server canvas, the
diff --git a/site/src/cut/worker/exportJob.ts b/site/src/cut/worker/exportJob.ts
index ed976ed5..219105b0 100644
--- a/site/src/cut/worker/exportJob.ts
+++ b/site/src/cut/worker/exportJob.ts
@@ -5,7 +5,9 @@ import { buildDocExportSpec, isDocExportPreset } from "../lib/headless/docExport
import { runExport, type ExportSpec, type RenderHandle } from "../server/exportPipeline";
import { storeCardArtifacts } from "./cardJob";
import { prisma, registerObject, type ClaimedJob } from "./db";
-import { downloadToFile, exportKey, mediaKey, previewKey, uploadFile } from "./r2";
+import { downloadToFile, exportKey, mediaKey, mimeFor, previewKey, uploadFile } from "./r2";
+import { overlayKey } from "../server/cloud/r2";
+import { specMediaFiles } from "../lib/exportDelivery";
import { runnerSession } from "./session";
/** The stored spec of an export/preview CutRenderJob: the engine export spec
@@ -19,6 +21,11 @@ export interface ExportJobSpec {
* and builds the spec itself. */
fromDoc?: { preset?: string };
overlays?: { name: string; key: string }[];
+ /** "overlays": the cut's media came up with the job — a browser-resident
+ * project borrowing this worker. Its output is the job's scratch too: it
+ * lands in the same namespace, is never a media object, and dies when the
+ * tab dismisses the job or the sweep reaches it. */
+ mediaFrom?: "overlays";
/** HLS only: whether the client burned captions into this render, decided by
* what the share grants. Recorded on the ladder so a share that later stops
* granting Subtitles is refused the render rather than served cue text it
@@ -79,21 +86,21 @@ export async function stageJobMedia(
]);
const spec = body.spec;
- const mediaFiles = new Set<string>();
- for (const c of spec.clips) if (c.file) mediaFiles.add(c.file);
- for (const o of spec.overlayVideos ?? []) if (o.file) mediaFiles.add(o.file);
- for (const a of spec.audio ?? []) if (a.file) mediaFiles.add(a.file);
+ const mediaFiles = new Set(specMediaFiles(spec));
const overlayPrefix = `cut/${job.userId}/overlays/`;
for (const o of body.overlays ?? []) {
if (!o.key.startsWith(overlayPrefix)) throw new Error("Invalid overlay key.");
}
+ // A media file that came up with the job — a browser-resident project's
+ // clip — is read from its upload; the rest are the cloud project's own.
+ const uploaded = new Map((body.overlays ?? []).map((o) => [o.name, o.key]));
await Promise.all([
...[...mediaFiles].map((f) =>
- downloadToFile(mediaKey(job.userId, projectId, f), path.join(mediaDir, path.basename(f)))
- ),
- ...(body.overlays ?? []).map((o) =>
- downloadToFile(o.key, path.join(handle.tmpDir, path.basename(o.name)))
+ downloadToFile(uploaded.get(f) ?? mediaKey(job.userId, projectId, f), path.join(mediaDir, path.basename(f)))
),
+ ...(body.overlays ?? [])
+ .filter((o) => !mediaFiles.has(o.name))
+ .map((o) => downloadToFile(o.key, path.join(handle.tmpDir, path.basename(o.name)))),
]);
return mediaDir;
}
@@ -161,16 +168,22 @@ export async function runExportJob(
});
if (live?.state !== "running") throw new Error("Canceled.");
+ const mime = mimeFor(outName);
+ if (body.mediaFrom === "overlays") {
+ const key = overlayKey(job.userId, job.id, outName);
+ await uploadFile(key, handle.outPath, mime);
+ return { outputKey: key, outName };
+ }
const key = preview
? previewKey(job.userId, projectId)
: exportKey(job.userId, projectId, outName);
- const bytes = await uploadFile(key, handle.outPath, "video/mp4");
+ const bytes = await uploadFile(key, handle.outPath, mime);
await registerObject({
userId: job.userId,
projectId,
r2Key: key,
fileName: outName,
- mime: "video/mp4",
+ mime,
bytes,
kind: preview ? "preview" : "export",
});