Skip to content

Open source CapCut alternative.

TypeScript

perf(site): wake only the render replicas the queue needs [rebuild]

David Le361bdc805297
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index a2025cc2..d6afaa09 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -120,7 +120,7 @@ The cache runs the other way too: opening a cloud project pulls in the assets th
Storage a subscription paid for outlives the subscription for a while: when Pro ends and the account still holds more than the free tier, it keeps everything for thirty days, warned in the editor and on the projects home. After that the daily sweep brings it back under the cap — least-recently-updated projects first, then the oldest library assets, stopping the moment it fits — so what survives is the recent work, and an account already under the cap is never touched.
-A cloud project can be shared read-only, Google-Docs style: the editor's Share dialog invites specific emails (viewers sign in) or opens the link to anyone (no sign-in), and switches opt extra surfaces in — chat, media, AI generations, subtitles, details — on top of the always-shared preview and timeline. The share row's id is the link token, so revoking deletes the link. Viewers ride a third backend driver against `/api/cut-shared/<token>/*`, a deliberately public API surface (the one exception to session auth on the cloud side): the server resolves access per request, filters the doc to the shared surfaces before it leaves, and refuses media outside that filter, so hiding is enforced by the backend rather than the client. A public share's reads are cached at the edge for a lifetime that tracks how long the owner has been idle — a doc saved seconds ago is served near-live, one untouched for an hour is served from cache for minutes — and the same number is handed back as the viewer's poll interval, so a link a crowd opens costs a handful of origin requests rather than one per viewer per tick. A restricted share's reads depend on the viewer's session, so they are never cached. A shared project's media is served from Cloudflare's edge rather than presigned against R2's S3 endpoint, which nothing caches: a Worker holding a private bucket binding checks a short token the hosted API mints, then answers from cache keyed on the object alone. Dropping the token from the cache key is what makes the token affordable to keep short — every viewer and every re-mint share one cached copy — and a short token is what makes unsharing mean something, since a presigned URL is a bearer credential nothing can withdraw before it expires. Owner-side editing still presigns R2 directly, where the URLs are minted on fixed windows so a browser can reuse bytes across reloads. A shared link unfurls with the project's own card: the render worker derives a first-frame still and a five-second animation from the opening of the cut, the page's metadata points at them through a URL carrying the doc version, and a project with no card yet gets a drawn one. Viewers poll the doc version on an interval the server picks — shared views trail live edits, they don't stream them — and a "Copy project" action duplicates everything the share exposes — the filtered doc, its media, and the chat threads when chat is shared — server-side into the viewer's own account and quota. Copies queue rather than run in the request: each request writes a job row and publishes it to a Cloudflare Queue whose serial consumer executes one copy at a time, so a burst of copy requests can't stampede storage; the client polls the job for the outcome. An owner duplicating their own cloud project from the dashboard rides the same queue. A cloud project's export renders in the browser. The tab already draws the whole cut for the preview, so it can draw it again frame by frame into an MP4 and upload the finished file — no round trip that pulls the media back out of storage into a container, no waiting behind someone else's render, and a file that matches the preview because the same compositor drew both. A browser that can't carry the render — no scratch storage, no encoder at the requested frame size — sends it to the worker, and so does everything the editor renders on its own behalf: hover-preview proxies, share cards, streaming ladders, and URL imports. That worker is a pool of containers (`site/src/cut/worker/`) that claim job rows from Postgres, reuse the engine's exact ffmpeg pipeline against R2, and report progress back through the rows the client polls. Each replica takes one job at a time, so a render has the whole machine to itself, and the queue is claimed by what someone is watching first: an export or a URL import goes ahead of the background artwork. The artwork waits for a lull too — a project's hover proxy and share card rebuild when the editor closes or the tab goes to the background, so a session of editing spends the pool on the renders the user asked for. Transcription and mic dictation prefer this Mac: with the app running they use on-device speech even for a cloud project, and fall back to a hosted, credit-metered route — a Gemini speech model reading word timings off browser-rendered audio — when it isn't. That split between where a project's data lives and where its work runs is its own guide: [Local compute](local-compute.md). The assistant offers its Gemini models (which already run entirely in the page); the Claude/Codex CLI providers stay app-only because they are the user's local logins.
+A cloud project can be shared read-only, Google-Docs style: the editor's Share dialog invites specific emails (viewers sign in) or opens the link to anyone (no sign-in), and switches opt extra surfaces in — chat, media, AI generations, subtitles, details — on top of the always-shared preview and timeline. The share row's id is the link token, so revoking deletes the link. Viewers ride a third backend driver against `/api/cut-shared/<token>/*`, a deliberately public API surface (the one exception to session auth on the cloud side): the server resolves access per request, filters the doc to the shared surfaces before it leaves, and refuses media outside that filter, so hiding is enforced by the backend rather than the client. A public share's reads are cached at the edge for a lifetime that tracks how long the owner has been idle — a doc saved seconds ago is served near-live, one untouched for an hour is served from cache for minutes — and the same number is handed back as the viewer's poll interval, so a link a crowd opens costs a handful of origin requests rather than one per viewer per tick. A restricted share's reads depend on the viewer's session, so they are never cached. A shared project's media is served from Cloudflare's edge rather than presigned against R2's S3 endpoint, which nothing caches: a Worker holding a private bucket binding checks a short token the hosted API mints, then answers from cache keyed on the object alone. Dropping the token from the cache key is what makes the token affordable to keep short — every viewer and every re-mint share one cached copy — and a short token is what makes unsharing mean something, since a presigned URL is a bearer credential nothing can withdraw before it expires. Owner-side editing still presigns R2 directly, where the URLs are minted on fixed windows so a browser can reuse bytes across reloads. A shared link unfurls with the project's own card: the render worker derives a first-frame still and a five-second animation from the opening of the cut, the page's metadata points at them through a URL carrying the doc version, and a project with no card yet gets a drawn one. Viewers poll the doc version on an interval the server picks — shared views trail live edits, they don't stream them — and a "Copy project" action duplicates everything the share exposes — the filtered doc, its media, and the chat threads when chat is shared — server-side into the viewer's own account and quota. Copies queue rather than run in the request: each request writes a job row and publishes it to a Cloudflare Queue whose serial consumer executes one copy at a time, so a burst of copy requests can't stampede storage; the client polls the job for the outcome. An owner duplicating their own cloud project from the dashboard rides the same queue. A cloud project's export renders in the browser. The tab already draws the whole cut for the preview, so it can draw it again frame by frame into an MP4 and upload the finished file — no round trip that pulls the media back out of storage into a container, no waiting behind someone else's render, and a file that matches the preview because the same compositor drew both. A browser that can't carry the render — no scratch storage, no encoder at the requested frame size — sends it to the worker, and so does everything the editor renders on its own behalf: hover-preview proxies, share cards, streaming ladders, and URL imports. That worker is a pool of containers (`site/src/cut/worker/`) that claim job rows from Postgres, reuse the engine's exact ffmpeg pipeline against R2, and report progress back through the rows the client polls. The hosted API wakes as many replicas as there are jobs waiting, and a replica with nothing left to claim exits within a minute, so the pool bills for renders and little else. Each replica takes one job at a time, so a render has the whole machine to itself, and the queue is claimed by what someone is watching first: an export or a URL import goes ahead of the background artwork. The artwork waits for a lull too — a project's hover proxy and share card rebuild when the editor closes or the tab goes to the background, so a session of editing spends the pool on the renders the user asked for. Transcription and mic dictation prefer this Mac: with the app running they use on-device speech even for a cloud project, and fall back to a hosted, credit-metered route — a Gemini speech model reading word timings off browser-rendered audio — when it isn't. That split between where a project's data lives and where its work runs is its own guide: [Local compute](local-compute.md). The assistant offers its Gemini models (which already run entirely in the page); the Claude/Codex CLI providers stay app-only because they are the user's local logins.
A share link opened on a phone plays rather than edits. The editor composites a project live — a video decoder per clip, painted onto a canvas — which is the right design on a desktop and an impossible one on a phone, where the decoders alone exhaust what a tab is given. So below the width the editor needs, the share serves a player instead: the cut as a single adaptive stream with the platform's own controls, and whichever surfaces the share opted in — chat, media, subtitles, details — as sheets that rise over it. The desktop share is untouched. The choice is made in code rather than in styling, because hiding the editor with a class would still have mounted it and started every decode.
diff --git a/site/src/cut/server/cloud/wake.ts b/site/src/cut/server/cloud/wake.ts
index 6740e991..eed1d8c6 100644
--- a/site/src/cut/server/cloud/wake.ts
+++ b/site/src/cut/server/cloud/wake.ts
@@ -4,6 +4,7 @@
// route that serves it; the shared secret is the only environment input, and
// holding it is the single switch that turns waking on. Unset (local dev), the
// dev worker is run by hand and polls on its own.
+import { prisma } from "@/lib/prisma";
import { CUT_WORKER_WAKE_URL } from "@/cut/lib/hosts";
const wakeSecret = () => process.env.CUT_RENDER_WAKE_SECRET;
@@ -25,14 +26,27 @@ export function wakeRenderWorker(): void {
const now = Date.now();
if (now - lastWakeAt < WAKE_WINDOW_MS) return;
lastWakeAt = now;
- void fetch(CUT_WORKER_WAKE_URL, {
+ void wake(secret).catch((e: unknown) => {
+ console.error("[cut] render worker wake failed:", e instanceof Error ? e.message : e);
+ });
+}
+
+// The wake carries the queue's depth so the Worker starts as many replicas as
+// there is work for. A container bills its full provisioned memory for every
+// second it is up, idle included, so a replica started with nothing to claim is
+// the pool's one avoidable cost. Running jobs count too: each one holds a
+// replica, and the Worker starts past them so the queued work still lands on
+// an idle box. A burst that arrives inside the wake window is picked up by the
+// next status poll's wake, which counts again.
+async function wake(secret: string): Promise<void> {
+ const [queued, running] = await Promise.all([
+ prisma.cutRenderJob.count({ where: { state: "queued" } }),
+ prisma.cutRenderJob.count({ where: { state: "running" } }),
+ ]);
+ const res = await fetch(CUT_WORKER_WAKE_URL, {
method: "POST",
- headers: { authorization: `Bearer ${secret}` },
- })
- .then((res) => {
- if (!res.ok) console.error(`[cut] render worker wake refused: HTTP ${res.status}`);
- })
- .catch((e: unknown) => {
- console.error("[cut] render worker wake failed:", e instanceof Error ? e.message : e);
- });
+ headers: { authorization: `Bearer ${secret}`, "content-type": "application/json" },
+ body: JSON.stringify({ queued, running }),
+ });
+ if (!res.ok) console.error(`[cut] render worker wake refused: HTTP ${res.status}`);
}
diff --git a/site/src/cut/worker/cf/wakeSize.ts b/site/src/cut/worker/cf/wakeSize.ts
new file mode 100644
index 00000000..933e901c
--- /dev/null
+++ b/site/src/cut/worker/cf/wakeSize.ts
@@ -0,0 +1,23 @@
+// How many replicas a wake starts. Kept apart from worker.ts so it can be
+// tested outside the Workers runtime.
+
+// The pool's size; the same number as REPLICAS in worker.ts and max_instances
+// in ../wrangler.jsonc.
+export const MAX_REPLICAS = 4;
+
+// What the hosted API knows about the queue when it wakes the pool
+// (server/cloud/wake.ts). A wake with no body still starts one replica: the
+// caller has work, it just did not say how much.
+export type WakeBody = { queued?: unknown; running?: unknown };
+
+const count = (v: unknown): number =>
+ typeof v === "number" && Number.isFinite(v) && v > 0 ? Math.floor(v) : 0;
+
+/** How many replicas a wake starts. Queued jobs each need an idle replica;
+ * running jobs each hold one, and which one is unknown, so the count reaches
+ * past them: of the lowest `queued + running` replicas at most `running` are
+ * busy, which leaves at least `queued` idle ones to claim the work. */
+export function replicasToStart(body: WakeBody | null): number {
+ const needed = count(body?.queued) + count(body?.running);
+ return Math.min(MAX_REPLICAS, Math.max(1, needed));
+}
diff --git a/site/src/cut/worker/cf/worker.test.ts b/site/src/cut/worker/cf/worker.test.ts
new file mode 100644
index 00000000..1bbdb979
--- /dev/null
+++ b/site/src/cut/worker/cf/worker.test.ts
@@ -0,0 +1,34 @@
+import { describe, expect, test } from "bun:test";
+import { replicasToStart } from "./wakeSize";
+
+// A replica bills its provisioned memory for every second it is up, so the
+// number a wake starts is the number the queue can use: one per waiting job,
+// reaching past the replicas that running jobs hold, never past the pool.
+describe("replicasToStart", () => {
+ test("a wake with no body starts one replica", () => {
+ expect(replicasToStart(null)).toBe(1);
+ expect(replicasToStart({})).toBe(1);
+ });
+
+ test("one replica per queued job", () => {
+ expect(replicasToStart({ queued: 1, running: 0 })).toBe(1);
+ expect(replicasToStart({ queued: 3, running: 0 })).toBe(3);
+ });
+
+ test("running jobs hold replicas, so the count reaches past them", () => {
+ expect(replicasToStart({ queued: 1, running: 2 })).toBe(3);
+ });
+
+ test("never more than the pool", () => {
+ expect(replicasToStart({ queued: 9, running: 4 })).toBe(4);
+ });
+
+ test("an empty queue still starts one replica", () => {
+ expect(replicasToStart({ queued: 0, running: 0 })).toBe(1);
+ });
+
+ test("junk counts read as zero", () => {
+ expect(replicasToStart({ queued: "4", running: -2 })).toBe(1);
+ expect(replicasToStart({ queued: Number.NaN, running: 2.7 })).toBe(2);
+ });
+});
diff --git a/site/src/cut/worker/cf/worker.ts b/site/src/cut/worker/cf/worker.ts
index c3e98b19..060e0237 100644
--- a/site/src/cut/worker/cf/worker.ts
+++ b/site/src/cut/worker/cf/worker.ts
@@ -8,6 +8,7 @@
import { Container, getContainer } from "@cloudflare/containers";
import { CUT_MEDIA_HOST, CUT_WORKER_HOST } from "../../lib/hosts";
import { serveMedia, type MediaEnv } from "./media";
+import { MAX_REPLICAS, replicasToStart, type WakeBody } from "./wakeSize";
type WorkerEnv = MediaEnv & {
CUT_RENDER_WORKER: unknown;
@@ -32,12 +33,13 @@ type QueueMessage = {
};
type QueueBatch = { queue: string; messages: QueueMessage[] };
-// How many container replicas the render pool runs. Each one takes a single
+// The most container replicas the render pool runs. Each one takes a single
// job at a time, so this is also the number of renders that can be in flight
-// at once. Keep it in step with max_instances in ../wrangler.jsonc; a replica
-// with nothing to claim drains out on its own (IDLE_EXIT_MS in ../main.ts),
-// so waking the whole pool costs idle poll time and nothing else.
-const REPLICAS = 4;
+// at once. Keep it in step with max_instances in ../wrangler.jsonc. A wake
+// starts only as many as the queue needs (wakeSize.ts): a replica bills
+// its whole 12 GiB for every second it is up, idle polling included, so one
+// started with nothing to claim is a minute of paid memory for no render.
+const REPLICAS = MAX_REPLICAS;
const replicaName = (i: number) => `cut-render-${i}`;
@@ -83,12 +85,14 @@ export default {
if (!env.CUT_WAKE_SECRET || auth !== `Bearer ${env.CUT_WAKE_SECRET}`) {
return new Response("Unauthorized", { status: 401 });
}
- // Wake the whole pool: the queue is drained by whichever replicas are up,
- // and each claim is atomic, so starting them all is how a burst of jobs
- // renders in parallel instead of one at a time. start() is a no-op on a
- // replica already running and a boot on one that slept.
+ // Start as many replicas as the queue needs, lowest index first. Each
+ // claim is atomic, so a burst of jobs still renders in parallel, one per
+ // replica; start() is a no-op on a replica already running and a boot
+ // on one that slept.
+ const body = (await request.json().catch(() => null)) as WakeBody | null;
+ const wanted = replicasToStart(body);
const starts = await Promise.allSettled(
- Array.from({ length: REPLICAS }, (_, i) =>
+ Array.from({ length: wanted }, (_, i) =>
getContainer(env.CUT_RENDER_WORKER as never, replicaName(i)).start()
)
);
diff --git a/site/src/cut/worker/wrangler.jsonc b/site/src/cut/worker/wrangler.jsonc
index 49f24e55..cebc5039 100644
--- a/site/src/cut/worker/wrangler.jsonc
+++ b/site/src/cut/worker/wrangler.jsonc
@@ -19,13 +19,15 @@
// Renders scale out, not up: each replica runs ONE job on the whole box
// (MAX_RUNNING in ../main.ts), so an export never splits cores with
// another render or queues behind someone else's. Keep this in step with
- // REPLICAS in cf/worker.ts — that constant is what /wake starts.
+ // REPLICAS in cf/worker.ts — that constant caps what /wake starts.
"max_instances": 4,
// The platform maximum (4 vCPU / 12 GiB / 20 GB). ffmpeg encodes are
// CPU-bound and x264 scales nearly linearly across threads, and container
- // CPU bills on active use rather than provisioned, so the wider box costs
- // about the same per render and finishes in half the time. Cloudflare
- // validation: >= 3 GiB memory per vCPU, disk <= 2x memory.
+ // CPU bills on active use, so the wider box costs about the same per
+ // render and finishes in half the time. Memory bills on the provisioned
+ // 12 GiB for every second a replica is up, which is why a wake starts
+ // only the replicas the queue needs. Cloudflare validation: >= 3 GiB
+ // memory per vCPU, disk <= 2x memory.
"instance_type": { "vcpu": 4, "memory_mib": 12288, "disk_mb": 20000 }
}
],