Skip to content

Open source CapCut alternative.

TypeScript

feat(site): a long clip's quality matte bakes in parts [rebuild]

David Le9adc4f405152
diff --git a/docs/guides/cut/README.md b/docs/guides/cut/README.md
index a6a0090b..3cc3b41e 100644
--- a/docs/guides/cut/README.md
+++ b/docs/guides/cut/README.md
@@ -32,7 +32,7 @@ The client probes the engine's health endpoint (dedicated port first, dev server
**Framing is per clip.** Every video clip meets its box — the whole frame, or the region it was given — through one geometry: fitted with margins or covering with a crop, zoomed in further, panned to choose what the crop keeps, turned, faded, cornered, masked, and shadowed. One function computes where the picture lands, the preview draws that rectangle, and the export graph scales and crops to the same numbers, so a reframed clip looks the same on every surface, freeze frames and headless renders included. On the stage the selected clip wears two outlines: a blue box for its layout, with grips to move, resize, and rotate it, and a grey dashed one standing where the source really lands, so the part a crop discards stays visible — its corners zoom the picture and its interior pans it. The shadow is cast by the shape the clip actually shows, mask included, which is what makes a circle-masked camera bubble throw a circular shadow.
-**A cutout keys the clip to a matte.** Every video clip can carry a cutout — the Inspector's Cutout tab: an alpha matte decides which of its pixels stay, ink can trace the kept silhouette, and a color or image can fill in behind it, with lower tracks showing through wherever nothing fills. Every mode resolves to that one matte in the clip's own pixels. A chroma key computes it live, from reference math in the effects kit that the GPU shader, the CPU pass, and the export all share. The AI modes bake it into a small grayscale video stored as a project asset: auto mattes the person with the on-device segmenter in seconds and for free, then a hosted tracker (credits, like all AI work) re-bakes it at quality and swaps the asset in; a custom selection goes straight to the tracker, seeded by a few words naming the subject or by the strokes painted on the stage — the instant feedback while painting comes from a small on-device tap-to-select model, and the stored words and seeds make every bake reproducible. A fingerprint on the matte records what it was baked from — the source and the selection, trims left out — so a retrim inside the baked range keeps the matte and its bill, while a changed selection re-bakes; a background sweep settles bakes against the doc through undo, redo, and deletion, and holds them while a brush session is still painting. For the ffmpeg export each cutout clip ships as a client-rendered pair of videos — the keyed picture's color and its alpha — baked by the same compositor that draws the preview, so the engine and the worker composite exactly what the tab showed. Headless surfaces consume stored mattes; a turn that runs with no browser writes the cutout onto the clip and the next editor session's sweep starts the bake it owes.
+**A cutout keys the clip to a matte.** Every video clip can carry a cutout — the Inspector's Cutout tab: an alpha matte decides which of its pixels stay, ink can trace the kept silhouette, and a color or image can fill in behind it, with lower tracks showing through wherever nothing fills. Every mode resolves to that one matte in the clip's own pixels. A chroma key computes it live, from reference math in the effects kit that the GPU shader, the CPU pass, and the export all share. The AI modes bake it into a small grayscale video stored as a project asset: auto mattes the person with the on-device segmenter in seconds and for free, then a hosted tracker (credits, like all AI work) re-bakes it at quality and swaps the asset in; a custom selection goes straight to the tracker, seeded by a few words naming the subject or by the strokes painted on the stage — the instant feedback while painting comes from a small on-device tap-to-select model, and the stored words and seeds make every bake reproducible. A long clip tracks in parts: the span splits into pieces of a few minutes, each submitted to the tracker as its own billable job with the selection handed across the seam, and a persisted ticket pins the submitted parts and the exact span they were cut from — a reload resumes the run on the paid pieces, and a part whose track dies is re-bought alone on retry. A fingerprint on the matte records what it was baked from — the source and the selection, trims left out — so a retrim inside the baked range keeps the matte and its bill, while a changed selection re-bakes; a background sweep settles bakes against the doc through undo, redo, and deletion, and holds them while a brush session is still painting. For the ffmpeg export each cutout clip ships as a client-rendered pair of videos — the keyed picture's color and its alpha — baked by the same compositor that draws the preview, so the engine and the worker composite exactly what the tab showed. Headless surfaces consume stored mattes; a turn that runs with no browser writes the cutout onto the clip and the next editor session's sweep starts the bake it owes.
**Words as the picture.** A lyric video, a kinetic-typography passage, a quote sequence: the assistant places a whole run of lines in one call, carrying a named look. A look is the design of the whole run rather than of one line — the frame color, the cards it cycles, and the ensemble the run walks across its length: the faces it rotates through, the entrances, where successive lines land in the frame, and what each line does while it holds. That is what stops a hundred lines coming out identical and dead center, which is the failure a text video actually has. The looks live in one registry the tools and the skills both read, so adding a look teaches the assistant a new style with no other change. Words ride one of two carriers — a caption track, which holds per-word timings, or elements, which give each line its own card, color and motion. Both carriers play a line word by word, and what a word does is data like everything else: three poses — before its moment, at it, and after — and the seconds between them. An emphasis travels along a line that is fully up, because the pose after a word's moment is the pose before it; a build assembles the line as it is spoken, because the pose after is the pose it arrived at. That one shape covers a karaoke highlight, a word swelling on the beat, a sentence appearing a word at a time, and a faint line going solid word by word, and it is what makes adding another an entry in the catalog. One evaluator resolves each word to a color, an opacity and a pose, and the preview, the burn-in and the frame samplers all read that, so a word effect looks the same wherever it plays. Text the user already has (lyrics, a script) is timed by aligning it to a transcript of the audio: the recognizer supplies the clock and the user's wording is kept exactly, which is what makes it survive singing.
diff --git a/site/packages/effects-kit/src/removal.ts b/site/packages/effects-kit/src/removal.ts
index 0805d854..2e724ef3 100644
--- a/site/packages/effects-kit/src/removal.ts
+++ b/site/packages/effects-kit/src/removal.ts
@@ -212,6 +212,11 @@ export function chromaAlphaOf(d: number, p: ChromaParams): number {
export const MATTE_FPS = 15;
export const MATTE_SHORT = 480;
export const MATTE_BITRATE = 1_000_000;
+/** The longest span one hosted tracking part covers. Each uploaded segment
+ * rides a fixed byte budget, so past a few minutes its bitrate — and the
+ * tracker's masks with it — collapse; a longer clip's bake splits its span
+ * into parts of at most this length and tracks each on its own budget. */
+export const MATTE_MAX_S = 180;
/** Turn a decoded matte frame's luma into alpha coverage, in place: alpha
* becomes the red channel's luma and the ink turns white, so the result
diff --git a/site/src/cut/components/RemovalPanel.tools.ts b/site/src/cut/components/RemovalPanel.tools.ts
index 2e5e3dad..8eb822f8 100644
--- a/site/src/cut/components/RemovalPanel.tools.ts
+++ b/site/src/cut/components/RemovalPanel.tools.ts
@@ -22,8 +22,7 @@ import { num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
export const REMOVAL_TOOLS = [
{
name: "set_removal",
- description:
- 'Cut a video clip\'s background away — the Inspector\'s Cutout tab, on any track. Modes: "auto" mattes the person in the shot with AI; "custom" mattes whatever `subject` describes in a few words ("the dog") and tracks every match through the clip — picking one exact instance by hand is the panel\'s brush flow, so send the user there for that; "chroma" keys out a backdrop color (green screen) with intensity/softness/spill; "off" removes the whole cutout, stroke and background fill included. The AI mattes bake in the background — wait_for_renders covers the bake. Where the background was, lower tracks or the project background show through; set_removal_background fills that area within the clip itself. Read the background-removal skill for the full recipes.',
+ description: `Cut a video clip's background away — the Inspector's Cutout tab, on any track. Modes: "auto" mattes the person in the shot with AI; "custom" mattes whatever \`subject\` describes in a few words ("the dog") and tracks every match through the clip — picking one exact instance by hand is the panel's brush flow, so send the user there for that; "chroma" keys out a backdrop color (green screen) with intensity/softness/spill; "off" removes the whole cutout, stroke and background fill included. The AI mattes prepare in the background — wait_for_renders covers that work — and a long clip tracks in parts automatically, so length needs no special handling. Where the background was, lower tracks or the project background show through; set_removal_background fills that area within the clip itself. Read the background-removal skill for the full recipes.`,
inputSchema: obj(
{
clipId: str("Video clip id"),
diff --git a/site/src/cut/lib/removal/bakeJobs.ts b/site/src/cut/lib/removal/bakeJobs.ts
index 2bf8630f..54d6dcc8 100644
--- a/site/src/cut/lib/removal/bakeJobs.ts
+++ b/site/src/cut/lib/removal/bakeJobs.ts
@@ -20,7 +20,7 @@ import { importFileToProject } from "../media";
import { useEditor } from "../store";
import { removalFingerprint, type ClipRemoval, type VideoClip } from "../types";
import { useBrushUi } from "./brushUi";
-import { hostedBakeMatte, MatteTrackFailed, type HostedBakeTicket } from "./hostedBake";
+import { hostedBakeMatte, type HostedBakeTicket } from "./hostedBake";
import { localBakeMatte, MatteBakeCancelled } from "./localBake";
type MatteQuality = "local" | "hq";
@@ -64,10 +64,11 @@ const running = new Map<string, { fp: string; quality: MatteQuality; ctl: AbortC
// The hosted bake bills at submit, so its poll ticket persists per browser: a
// reload re-attaches to the paid track through it and pays nothing new. One
-// record per clip. A ticket outlives cancellation — clearing a cutout and
-// undoing it comes back to the same fingerprint, and the undo resumes the
-// paid track. Landing or terminal failure drops it; the sweep retires the
-// rest: a changed selection, a clip gone from the doc, or plain age.
+// record per clip. A ticket outlives cancellation and failure — clearing a
+// cutout and undoing it comes back to the same fingerprint, and the undo
+// resumes the paid track; a dead track forgets only its own part inside the
+// run. Landing drops the record; the sweep retires the rest: a changed
+// selection, a clip gone from the doc, or plain age.
const TICKETS_KEY = "cut-matte-bake-tickets";
const TICKET_TTL_MS = 48 * 60 * 60 * 1000;
type PersistedTicket = {
@@ -275,9 +276,9 @@ export function ensureMatteBake(clipId: string, opts: { whileBrushing?: boolean
setJob(clipId, null);
return;
}
- // A dead track's ticket has nothing to resume — drop it so Retry submits
- // fresh. A transient failure keeps it, and the retry re-attaches.
- if (e instanceof MatteTrackFailed) dropTicket(clipId);
+ // The ticket stands through failure: the run already forgot any part
+ // whose track settled dead, so Retry resumes the paid parts and re-buys
+ // only what died.
failed.set(clipId, { fp, quality: want });
setJob(clipId, {
quality: want,
diff --git a/site/src/cut/lib/removal/hostedBake.ts b/site/src/cut/lib/removal/hostedBake.ts
index 784eb9d8..f1f3bfac 100644
--- a/site/src/cut/lib/removal/hostedBake.ts
+++ b/site/src/cut/lib/removal/hostedBake.ts
@@ -8,33 +8,47 @@
* the tracker from the on-device person matte; custom mode seeds it from the
* brush's stored prompts and paint. Driven by the page or the cloud worker —
* the local engine only ever consumes the stored matte.
+ *
+ * A long clip tracks in parts: the span splits into pieces of at most
+ * `MATTE_MAX_S`, each rendered, uploaded, and tracked on its own upload
+ * budget, and every piece encodes into the one matte. A painted selection
+ * crosses the seam by handoff — the last mask frame of one part seeds point
+ * prompts for the next. Each part's submit is its own billable moment, and
+ * the persisted ticket carries every submitted part and the exact span it
+ * was built for, so a reload resumes the run without paying for finished
+ * pieces again; a part whose track settles dead is forgotten alone, and a
+ * retry re-buys just that piece.
*/
import {
coverageToGray,
MATTE_BITRATE,
MATTE_FPS,
+ MATTE_MAX_S,
MATTE_SHORT,
refineMatteAgainstFrame,
} from "@donkeycut/effects-kit";
-import { openCanvasVideo, scaledEvenSize } from "../canvasVideo";
+import { openCanvasVideo, scaledEvenSize, type CanvasVideoEncoder } from "../canvasVideo";
import { NO_CREDITS_MESSAGE } from "../credits";
import { ClipReader } from "../exportRender";
import { hostedPost } from "../hosted";
import { storeHostedBlob } from "../hostedBlobs";
-import { createRasterCanvas, decodeRasterImageUrl } from "../raster";
+import { createRasterCanvas, decodeRasterImageUrl, type RasterSurface } from "../raster";
import type { MediaAsset, RemovalSeeds, VideoClip } from "../types";
-import { localBakeMatte, MatteBakeCancelled } from "./localBake";
+import { liveReader } from "../liveReader";
+import { MatteBakeCancelled } from "./localBake";
/** Segment sent to the tracker: capped resolution and rate — the mask comes
* back at the same shape, and tracking cost scales with both. The bitrate
- * also shrinks with length so the whole segment stays under the upload
- * route's 32MB blob cap: a long clip trades bitrate for admission. */
+ * also shrinks with length so each part stays under the upload route's 32MB
+ * blob cap. */
const SEGMENT_SHORT = 720;
const SEGMENT_BITRATE = 2_500_000;
const SEGMENT_BYTE_BUDGET = 30 * 1024 * 1024;
const POLL_MS = 5_000;
const DEADLINE_MS = 10 * 60_000;
+/** The most handoff points sampled from a part's final mask. */
+const CARRY_POINTS_MAX = 6;
/** The hosted tier answered 402: credits unlock it, the local matte stands.
* Carries the app-wide balance message so every surface renders it the same
@@ -45,10 +59,11 @@ export class MatteNeedsCredits extends Error {
}
}
-/** The tracker settled this generation as failed — the charge is spent and
- * the track is dead, so the caller drops its persisted ticket and a retry
- * submits fresh. Transient errors (a poll that couldn't reach the route)
- * throw plain errors instead, and the ticket stands for a resume. */
+/** The tracker settled a generation as failed — the charge is spent and the
+ * track is dead. The run forgets the dead part in the ticket it reports, so
+ * a retry re-buys that piece alone and resumes the rest. Transient errors
+ * (a poll that couldn't reach the route) throw plain errors instead, and
+ * the whole ticket stands for a resume. */
export class MatteTrackFailed extends Error {}
type Point = { x: number; y: number; frame: number; label: 0 | 1; object: string };
@@ -70,44 +85,45 @@ function checkAborted(signal: AbortSignal | undefined) {
if (signal?.aborted) throw new MatteBakeCancelled();
}
-/** Encode the clip's trimmed source range as a compact video-only segment. */
+/** Encode one span of the clip's source as a compact video-only segment.
+ * The segment's own clock starts at zero — the tracker sees a standalone
+ * video — and `from` records where it sits in source time. The reader is the
+ * run's shared one; the caller owns its lifetime. */
async function renderSegment(
- asset: MediaAsset,
- clip: VideoClip,
- signal: AbortSignal | undefined
+ reader: ClipReader,
+ still: boolean,
+ from: number,
+ to: number,
+ frames: number,
+ signal: AbortSignal | undefined,
+ onProgress?: (fraction: number) => void
): Promise<{ blob: Blob; from: number; w: number; h: number; frames: number }> {
- const still = asset.type === "image";
- const from = still ? 0 : Math.max(0, clip.in);
- const to = still ? from + 1 / MATTE_FPS : Math.min(clip.out, asset.duration || clip.out);
- const frames = still ? 1 : Math.max(1, Math.ceil(Math.max(1 / MATTE_FPS, to - from) * MATTE_FPS));
- const reader = new ClipReader(asset, () => asset.url);
+ const first = await reader.frameAt(from);
+ if (first.kind !== "ready") throw new Error("The clip's picture could not be read.");
+ const { w, h } = scaledEvenSize(first.width, first.height, SEGMENT_SHORT);
+ const seconds = frames / MATTE_FPS;
+ const enc = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: MATTE_FPS,
+ frames,
+ bitrate: Math.min(SEGMENT_BITRATE, Math.floor((SEGMENT_BYTE_BUDGET * 8) / seconds)),
+ });
try {
- const first = await reader.frameAt(from);
- if (first.kind !== "ready") throw new Error("The clip's picture could not be read.");
- const { w, h } = scaledEvenSize(first.width, first.height, SEGMENT_SHORT);
- const seconds = frames / MATTE_FPS;
- const enc = await openCanvasVideo({
- width: w,
- height: h,
- fps: MATTE_FPS,
- frames,
- bitrate: Math.min(SEGMENT_BITRATE, Math.floor((SEGMENT_BYTE_BUDGET * 8) / seconds)),
- });
for (let i = 0; i < frames; i++) {
- if (signal?.aborted) {
- await enc.cancel();
- throw new MatteBakeCancelled();
- }
- const at = Math.min(to - 1 / (MATTE_FPS * 2), from + i / MATTE_FPS);
+ checkAborted(signal);
+ const at = still ? 0 : Math.min(to - 1 / (MATTE_FPS * 2), from + i / MATTE_FPS);
const frame = await reader.frameAt(at);
enc.ctx.fillStyle = "#000000";
enc.ctx.fillRect(0, 0, w, h);
if (frame.kind === "ready") enc.ctx.drawImage(frame.image, 0, 0, w, h);
await enc.add(i / MATTE_FPS, 1 / MATTE_FPS);
+ onProgress?.((i + 1) / frames);
}
return { blob: await enc.finish(), from, w, h, frames };
- } finally {
- reader.dispose();
+ } catch (e) {
+ await enc.cancel();
+ throw e;
}
}
@@ -133,9 +149,9 @@ function centroidOf(px: Uint8ClampedArray, w: number, h: number, channel: 0 | 3)
}
/** Custom mode's tracker seeds, from the brush's stored prompts and paint.
- * A seed the trims moved past points at a frame the segment doesn't hold —
- * clamped it would prompt on a different moment of the shot, so it is
- * dropped instead. */
+ * A seed outside this part's span points at a frame the segment doesn't
+ * hold — clamped it would prompt on a different moment of the shot, so it
+ * is dropped instead (a seed in another part lands with that part). */
async function seedPoints(
seeds: RemovalSeeds,
seg: { w: number; h: number; frames: number },
@@ -185,74 +201,80 @@ async function seedPoints(
return out;
}
-/** What a reload needs to re-attach to a paid in-flight track: the
- * generation's poll payload, as the submit returned it. */
-export type HostedBakeTicket = Generation;
-
-/**
- * Run the hosted quality bake for a clip and return the refined matte, ready
- * to store as the project's matte asset. Throws `MatteNeedsCredits` on a 402,
- * `MatteTrackFailed` when the tracker settled the paid run as dead, and
- * `MatteBakeCancelled` on abort. The submit bills, so the caller persists the
- * ticket handed to `onSubmitted` and passes it back as `resume` after a
- * reload — the bake then re-attaches to the running track and pays nothing
- * new.
- */
-export async function hostedBakeMatte(
- asset: MediaAsset,
- clip: VideoClip,
- opts: {
- onProgress?: (fraction: number) => void;
- signal?: AbortSignal;
- resume?: HostedBakeTicket;
- onSubmitted?: (ticket: HostedBakeTicket) => void;
- } = {}
-): Promise<{ blob: Blob; in: number }> {
- const r = clip.removal;
- if (!r) throw new Error("The clip has no removal to bake.");
- const still = asset.type === "image";
- const from = still ? 0 : Math.max(0, clip.in);
- let gen: Generation;
- if (opts.resume) {
- gen = opts.resume;
- opts.onProgress?.(0.2);
- } else {
- opts.onProgress?.(0.02);
- const segment = await renderSegment(asset, clip, opts.signal);
- opts.onProgress?.(0.15);
- // The words and the strokes ride together: auto mode and a described
- // subject send a text prompt, painted seeds send point prompts, and a
- // custom removal with both sends both — the segmenter's points refine what
- // the words detected. The route picks the configured segmenter.
- const painted =
- r.mode === "custom" && !!(r.seeds?.prompts.length || r.seeds?.paint?.length);
- const concept = r.mode === "custom" ? (r.subject ?? "").trim() : "person";
- const points = painted ? await seedPoints(r.seeds!, segment, segment.from) : [];
- if (!concept && points.length === 0) throw new Error("No selection to track.");
- const parameters = {
- ...(concept ? { prompt: concept } : {}),
- ...(points.length > 0 ? { points } : {}),
- };
- checkAborted(opts.signal);
+/** Handoff seeds for the next part: positive points sampled from the mask's
+ * white regions, as frame fractions — a coarse grid keeps them spread over
+ * every kept area. They all seed the one carried selection, so the tracker
+ * follows it as a single object however many cells it lights. */
+function carryPoints(px: Uint8ClampedArray, w: number, h: number): { x: number; y: number }[] {
+ const out: { x: number; y: number }[] = [];
+ const cols = 3;
+ const rows = 2;
+ for (let cy = 0; cy < rows; cy++) {
+ for (let cx = 0; cx < cols; cx++) {
+ let sx = 0;
+ let sy = 0;
+ let count = 0;
+ const x0 = Math.floor((cx * w) / cols);
+ const x1 = Math.floor(((cx + 1) * w) / cols);
+ const y0 = Math.floor((cy * h) / rows);
+ const y1 = Math.floor(((cy + 1) * h) / rows);
+ for (let y = y0; y < y1; y++) {
+ for (let x = x0; x < x1; x++) {
+ if (px[(y * w + x) * 4] > 160) {
+ sx += x;
+ sy += y;
+ count++;
+ }
+ }
+ }
+ // The cell's centroid can land on background between two kept areas of
+ // the same cell; dense sampling is not worth the prompt noise, so a
+ // cell contributes only when it is meaningfully lit.
+ if (count > 24 && out.length < CARRY_POINTS_MAX) {
+ out.push({ x: sx / count / w, y: sy / count / h });
+ }
+ }
+ }
+ return out;
+}
- const blobRef = await storeHostedBlob(segment.blob, "donkey-cut");
- if (!blobRef) throw new Error("The segment could not reach storage.");
- opts.onProgress?.(0.2);
+/** What a reload needs to re-attach to a paid in-flight run: every part's
+ * poll payload, in order (a dead track's slot holds null), how many parts
+ * the whole run has, and the source span the parts were cut from. */
+export interface HostedBakeTicket {
+ parts: (Generation | null)[];
+ total: number;
+ from: number;
+ to: number;
+}
- const submit = await hostedPost("/api/inference/assets", {
- kind: "matte",
- prompt: "Track the selected subject through the clip.",
- inputs: { video: { blobRef, blobField: "url", blobUrl: true, mimeType: "video/mp4" } },
- parameters,
- });
- if (submit.status === 402) throw new MatteNeedsCredits();
- if (!submit.ok) throw new Error("The quality cutout could not start.");
- gen = (await submit.json()) as Generation;
- // The submit is the billable moment: from here the ticket outlives the
- // page, so a reload resumes this track instead of paying for another.
- opts.onSubmitted?.(gen);
- }
+/** Read a persisted ticket. The parts were tracked for one exact source
+ * span, so a ticket whose span or split no longer matches the clip's (a
+ * retrim moved the trims under the same fingerprint) is unusable and
+ * dropped; so is a record from before the span was stamped. */
+function ticketParts(
+ v: HostedBakeTicket | undefined,
+ total: number,
+ from: number,
+ to: number
+): (Generation | null)[] {
+ if (!v || !("parts" in v) || !Array.isArray(v.parts)) return [];
+ const holds =
+ v.total === total &&
+ typeof v.from === "number" &&
+ typeof v.to === "number" &&
+ Math.abs(v.from - from) < 0.001 &&
+ Math.abs(v.to - to) < 0.001;
+ return holds ? v.parts : [];
+}
+/** Poll one part's track to completion and hand back its mask URL. */
+async function pollTrack(
+ start: Generation,
+ opts: { signal?: AbortSignal },
+ onFrac: (f: number) => void
+): Promise<string> {
+ let gen = start;
const deadline = Date.now() + DEADLINE_MS;
while (gen.status === "in_progress" || gen.status === "pending") {
checkAborted(opts.signal);
@@ -271,7 +293,7 @@ export async function hostedBakeMatte(
});
if (!poll.ok) throw new Error("The quality cutout failed.");
gen = (await poll.json()) as Generation;
- opts.onProgress?.(Math.min(0.7, 0.2 + (1 - (deadline - Date.now()) / DEADLINE_MS) * 2));
+ onFrac(Math.min(1, 1 - (deadline - Date.now()) / DEADLINE_MS));
}
if (gen.status !== "completed") {
const message =
@@ -282,24 +304,34 @@ export async function hostedBakeMatte(
}
const maskUrl = gen.outputs.find((o) => o.url)?.url;
if (!maskUrl) throw new MatteTrackFailed("The tracker returned no mask.");
- checkAborted(opts.signal);
- opts.onProgress?.(0.72);
+ return maskUrl;
+}
- return refineAndEncode(asset, clip, maskUrl, from, opts);
+/** The one encoder the whole matte writes through, opened on the first
+ * part's mask, plus the scratch the refine pass reuses. */
+interface MatteSink {
+ enc: CanvasVideoEncoder;
+ w: number;
+ h: number;
+ guide: RasterSurface;
+ gctx: CanvasRenderingContext2D;
+ lit: boolean;
}
-/** Refine the tracker's mask against the source pixels and encode the final
- * grayscale matte. */
-async function refineAndEncode(
- asset: MediaAsset,
- clip: VideoClip,
+/**
+ * Refine one part's tracked mask against the source pixels and append its
+ * frames to the shared matte. Returns the part's final mask pixels for the
+ * next part's handoff.
+ */
+async function refinePart(
+ sink: { current: MatteSink | null },
+ srcReader: ClipReader,
+ still: boolean,
maskUrl: string,
- from: number,
+ span: { from: number; to: number; frames: number; offset: number },
+ totalFrames: number,
opts: { onProgress?: (fraction: number) => void; signal?: AbortSignal }
-): Promise<{ blob: Blob; in: number }> {
- const still = asset.type === "image";
- const to = still ? from + 1 / MATTE_FPS : Math.min(clip.out, asset.duration || clip.out);
- const frames = still ? 1 : Math.max(1, Math.ceil(Math.max(1 / MATTE_FPS, to - from) * MATTE_FPS));
+): Promise<ImageData | null> {
// The browser reads the mask from a local blob: the whole file downloads
// once, and the decoder never streams the provider's CDN. Headless runs
// have no object URLs and read the URL directly.
@@ -318,74 +350,224 @@ async function refineAndEncode(
name: "hosted-mask",
} as MediaAsset;
const maskReader = new ClipReader(maskAsset, () => readUrl);
- const srcReader = new ClipReader(asset, () => asset.url);
try {
const firstMask = await maskReader.frameAt(0);
if (firstMask.kind !== "ready") throw new Error("The tracked mask could not be read.");
- const { w, h } = scaledEvenSize(firstMask.width, firstMask.height, MATTE_SHORT);
- const enc = await openCanvasVideo({
- width: w,
- height: h,
- fps: MATTE_FPS,
- frames,
- bitrate: MATTE_BITRATE,
- readback: true,
- });
- const ctx = enc.ctx;
- const guide = createRasterCanvas(w, h);
- const gctx = guide.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D;
- // A track that detected nothing comes back as an all-black mask video; an
- // empty matte would make the clip vanish, so it fails the bake instead.
- let lit = false;
+ let s = sink.current;
+ if (!s) {
+ const { w, h } = scaledEvenSize(firstMask.width, firstMask.height, MATTE_SHORT);
+ const enc = await openCanvasVideo({
+ width: w,
+ height: h,
+ fps: MATTE_FPS,
+ frames: totalFrames,
+ bitrate: MATTE_BITRATE,
+ readback: true,
+ });
+ const guide = createRasterCanvas(w, h);
+ const gctx = guide.getContext("2d", { willReadFrequently: true }) as CanvasRenderingContext2D;
+ s = { enc, w, h, guide, gctx, lit: false };
+ sink.current = s;
+ }
+ const ctx = s.enc.ctx;
+ let last: ImageData | null = null;
- for (let i = 0; i < frames; i++) {
- if (opts.signal?.aborted) {
- await enc.cancel();
- throw new MatteBakeCancelled();
- }
+ for (let i = 0; i < span.frames; i++) {
+ checkAborted(opts.signal);
const t = i / MATTE_FPS;
- const mask = await maskReader.frameAt(Math.min(t, (frames - 0.5) / MATTE_FPS));
- const src = await srcReader.frameAt(still ? 0 : Math.min(to - 1 / (MATTE_FPS * 2), from + t));
+ const mask = await maskReader.frameAt(Math.min(t, (span.frames - 0.5) / MATTE_FPS));
+ const src = await srcReader.frameAt(
+ still ? 0 : Math.min(span.to - 1 / (MATTE_FPS * 2), span.from + t)
+ );
ctx.fillStyle = "#000000";
- ctx.fillRect(0, 0, w, h);
+ ctx.fillRect(0, 0, s.w, s.h);
if (mask.kind === "ready") {
ctx.imageSmoothingEnabled = true;
- ctx.drawImage(mask.image, 0, 0, w, h);
- const mattePx = ctx.getImageData(0, 0, w, h);
+ ctx.drawImage(mask.image, 0, 0, s.w, s.h);
+ const mattePx = ctx.getImageData(0, 0, s.w, s.h);
coverageToGray(mattePx.data);
- if (!lit) {
+ // A track that detected nothing comes back as an all-black mask
+ // video; an empty matte would make the clip vanish, so the whole
+ // bake fails instead once every part has drawn.
+ if (!s.lit) {
const px = mattePx.data;
for (let p = 0; p < px.length; p += 4) {
if (px[p] > 32) {
- lit = true;
+ s.lit = true;
break;
}
}
}
+ // The handoff reads the part's last frame that decoded; a mask
+ // video's very final frame is allowed to come back pending without
+ // costing the seam its seeds.
+ last = mattePx;
if (src.kind === "ready") {
- gctx.drawImage(src.image, 0, 0, w, h);
- const framePx = gctx.getImageData(0, 0, w, h);
- refineMatteAgainstFrame(mattePx.data, framePx.data, w, h, 6);
+ s.gctx.drawImage(src.image, 0, 0, s.w, s.h);
+ const framePx = s.gctx.getImageData(0, 0, s.w, s.h);
+ refineMatteAgainstFrame(mattePx.data, framePx.data, s.w, s.h, 6);
}
ctx.putImageData(mattePx, 0, 0);
}
- await enc.add(t, 1 / MATTE_FPS);
- opts.onProgress?.(0.72 + ((i + 1) / frames) * 0.27);
+ await s.enc.add((span.offset + i) / MATTE_FPS, 1 / MATTE_FPS);
+ opts.onProgress?.((i + 1) / span.frames);
+ }
+ return last;
+ } finally {
+ maskReader.dispose();
+ if (localUrl) URL.revokeObjectURL(localUrl);
+ }
+}
+
+/**
+ * Run the hosted quality bake for a clip and return the refined matte, ready
+ * to store as the project's matte asset. Throws `MatteNeedsCredits` on a 402,
+ * `MatteTrackFailed` when the tracker settled a paid run as dead, and
+ * `MatteBakeCancelled` on abort. Each part's submit bills, so the caller
+ * persists the ticket handed to `onSubmitted` and passes it back as `resume`
+ * after a reload — the run then re-attaches to every submitted part and pays
+ * only for the ones it has not started.
+ */
+export async function hostedBakeMatte(
+ asset: MediaAsset,
+ clip: VideoClip,
+ opts: {
+ onProgress?: (fraction: number) => void;
+ signal?: AbortSignal;
+ resume?: HostedBakeTicket;
+ onSubmitted?: (ticket: HostedBakeTicket) => void;
+ } = {}
+): Promise<{ blob: Blob; in: number }> {
+ const r = clip.removal;
+ if (!r) throw new Error("The clip has no removal to bake.");
+ const still = asset.type === "image";
+ const from = still ? 0 : Math.max(0, clip.in);
+ const to = still ? from + 1 / MATTE_FPS : Math.min(clip.out, asset.duration || clip.out);
+ const totalFrames = still
+ ? 1
+ : Math.max(1, Math.ceil(Math.max(1 / MATTE_FPS, to - from) * MATTE_FPS));
+ const maxFrames = Math.max(1, Math.round(MATTE_MAX_S * MATTE_FPS));
+ const nParts = Math.max(1, Math.ceil(totalFrames / maxFrames));
+ const perPart = Math.ceil(totalFrames / nParts);
+
+ const painted = r.mode === "custom" && !!(r.seeds?.prompts.length || r.seeds?.paint?.length);
+ const concept = r.mode === "custom" ? (r.subject ?? "").trim() : "person";
+
+ const gens = [...ticketParts(opts.resume, nParts, from, to)];
+ const ticket = (upto: number): HostedBakeTicket => ({
+ parts: gens.slice(0, upto).map((g) => g ?? null),
+ total: nParts,
+ from,
+ to,
+ });
+ const sink: { current: MatteSink | null } = { current: null };
+ const srcReader = liveReader(asset);
+ let carry: { x: number; y: number }[] = [];
+
+ try {
+ for (let p = 0; p < nParts; p++) {
+ const startFrame = p * perPart;
+ const frames = Math.min(totalFrames, (p + 1) * perPart) - startFrame;
+ const partFrom = from + startFrame / MATTE_FPS;
+ const partTo = Math.min(to, from + (startFrame + frames) / MATTE_FPS);
+ const frac = (f: number) => opts.onProgress?.((p + f) / nParts);
+
+ let gen = gens[p];
+ if (!gen) {
+ frac(0.02);
+ const segment = await renderSegment(srcReader, still, partFrom, partTo, frames, opts.signal, (f) =>
+ frac(0.02 + f * 0.13)
+ );
+ // The words and the strokes ride together: auto mode and a described
+ // subject send a text prompt, painted seeds send point prompts, and a
+ // custom removal with both sends both — the segmenter's points refine
+ // what the words detected. A painted part with no strokes of its own
+ // takes the handoff points from the previous part's final mask, so the
+ // picked selection crosses the seam. The route picks the segmenter.
+ let points = painted ? await seedPoints(r.seeds!, segment, segment.from) : [];
+ if (painted && points.length === 0 && carry.length > 0) {
+ points = carry.map((c) => ({
+ x: c.x * segment.w,
+ y: c.y * segment.h,
+ frame: 0,
+ label: 1 as const,
+ object: "carry",
+ }));
+ }
+ if (!concept && points.length === 0) {
+ throw p === 0
+ ? new Error("No selection to track.")
+ : new MatteTrackFailed("The selection was lost at a part boundary.");
+ }
+ const parameters = {
+ ...(concept ? { prompt: concept } : {}),
+ ...(points.length > 0 ? { points } : {}),
+ };
+ checkAborted(opts.signal);
+
+ const blobRef = await storeHostedBlob(segment.blob, "donkey-cut");
+ if (!blobRef) throw new Error("The segment could not reach storage.");
+ frac(0.2);
+
+ const submit = await hostedPost("/api/inference/assets", {
+ kind: "matte",
+ prompt: "Track the selected subject through the clip.",
+ inputs: { video: { blobRef, blobField: "url", blobUrl: true, mimeType: "video/mp4" } },
+ parameters,
+ });
+ if (submit.status === 402) throw new MatteNeedsCredits();
+ if (!submit.ok) throw new Error("The quality cutout could not start.");
+ gen = (await submit.json()) as Generation;
+ gens[p] = gen;
+ // The submit is the billable moment: from here the ticket outlives the
+ // page, so a reload resumes this run's submitted parts instead of
+ // paying for them again.
+ opts.onSubmitted?.(ticket(p + 1));
+ } else {
+ frac(0.2);
+ }
+
+ let maskUrl: string;
+ try {
+ maskUrl = await pollTrack(gen, opts, (f) => frac(0.2 + f * 0.5));
+ } catch (e) {
+ // A dead track spends only its own part: the ticket forgets this
+ // generation and keeps every other submitted one, so a retry re-buys
+ // just this piece.
+ if (e instanceof MatteTrackFailed && gens[p]) {
+ gens[p] = null;
+ opts.onSubmitted?.(ticket(Math.max(p + 1, gens.length)));
+ }
+ throw e;
+ }
+ checkAborted(opts.signal);
+ const last = await refinePart(
+ sink,
+ srcReader,
+ still,
+ maskUrl,
+ { from: partFrom, to: partTo, frames, offset: startFrame },
+ totalFrames,
+ { ...opts, onProgress: (f) => frac(0.72 + f * 0.28) }
+ );
+ carry = last ? carryPoints(last.data, last.width, last.height) : [];
}
- if (!lit) {
- await enc.cancel();
+
+ const s = sink.current;
+ if (!s) throw new MatteTrackFailed("The tracker returned no mask.");
+ if (!s.lit) {
throw new MatteTrackFailed(
"The tracker found nothing to keep — describe the subject or paint it."
);
}
- return { blob: await enc.finish(), in: from };
+ return { blob: await s.enc.finish(), in: from };
+ } catch (e) {
+ // Whatever escaped, the matte encoder must not outlive the run — the
+ // browser's hardware encoder pool is finite, and a leaked one starves
+ // every bake after it. Cancel is idempotent.
+ await sink.current?.enc.cancel();
+ throw e;
} finally {
- maskReader.dispose();
srcReader.dispose();
- if (localUrl) URL.revokeObjectURL(localUrl);
}
}
-
-// The local quick bake rides beside the hosted one so callers pick a rung
-// without importing two modules.
-export { localBakeMatte, MatteBakeCancelled };
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index c1d7866f..60207912 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -128,9 +128,9 @@ Grade like a colorist: read the image, correct it, then style it — in that ord
**Craft.** Skin lives around orange at modest saturation — check faces after strong saturation, hue, or wheel moves, and reach for protect_skin or a gentle orange-band correction rather than abandoning the look. Keep adjacent clips consistent: one preset + per-clip correction reads as one film; different presets per clip read as a slideshow. Strong looks sit better with contrast established first. Grades are per-clip — "grade the whole video" means every clip, same recipe.`,
"background-removal": `# Background removal (Cutout)
-Every video clip — any track, stills too — can carry a cutout, the Inspector's Cutout tab: the clip's picture keys to an alpha matte, the subject stays, and where the background was, lower tracks or the project background show through. Preview, timeline thumbnails, and export render the same matte.
-- set_removal mode "auto": the AI person matte. A free on-device pass lands in seconds, and a hosted quality pass (spends credits like generation) upgrades hair and edge detail in the background. The tool call starts the bake itself; in the panel, picking a mode only previews the live matte, and the user's Start (auto) or Apply (custom) press starts the bake — hand-instructions include that step. Each cutout clip reports under \`removal\` in editor_state videoTrack/overlayVideo — mode, matte readiness ("ready (quick)" / "ready (quality)" / "none"), and a live bake's progress — and wait_for_renders settles on bakes, so claims about a finished cutout come from there. Short of credits, the quick matte stands and the panel offers the upgrade.
-- set_removal mode "custom" with \`subject\`: an arbitrary object by description — "the dog", "the red car" — matted and tracked through the clip (spends credits, less than a brushed track). Every match of the words is kept, so "cut out the dog" is one call. Picking one exact instance among lookalikes is hands-on: the user picks Custom in the Cutout tab, paints it on the preview with the quick and manual brushes, and presses Apply — the hosted tracker follows those exact strokes; with a subject set too, the strokes refine which instance the words picked.
+Every video clip — any track, stills too — can carry a cutout, the Inspector's Cutout tab: the clip's picture keys to an alpha matte, the subject stays, and where the background was, lower tracks or the project background show through. Preview, timeline thumbnails, and export render the same matte. A long clip tracks in parts automatically — cutout length needs no special handling.
+- set_removal mode "auto": the AI person matte. A free on-device pass lands in seconds, and a hosted quality pass (spends credits like generation) upgrades hair and edge detail in the background. The tool call starts the matte work itself; in the panel, picking a mode only previews the live matte, and the user's Start (auto) or Apply (custom) press starts it — hand-instructions include that step. Each cutout clip reports under \`removal\` in editor_state videoTrack/overlayVideo — mode, matte readiness ("ready (quick)" / "ready (quality)" / "none"), and a running matte's progress — and wait_for_renders settles on that work, so claims about a finished cutout come from there. Short of credits, the quick matte stands and the panel offers the upgrade.
+- set_removal mode "custom" with \`subject\`: an arbitrary object by description — "the dog", "the red car" — matted and tracked through the clip (spends credits, less than a brushed track). Every match of the words is kept, so "cut out the dog" is one call. Picking one exact instance among lookalikes is hands-on: the user picks Custom in the Cutout tab, paints it on the preview with the Brush and Erase tools (the wand toggles Smart select, which grabs the whole object or flat region under the stroke; off by default, strokes paint exact pixels), and presses Apply — the hosted tracker follows those exact strokes; with a subject set too, the strokes refine which instance the words picked.
- set_removal mode "chroma": the green-screen key. The key color plus intensity (tolerance), softness (edge rolloff), and spill (pulls key-colored fringe off the kept pixels); omitting color samples the footage's borders. Footage shot or generated over a solid backdrop keys cleanly and costs nothing.
- set_removal_stroke draws ink around the kept silhouette — glow, hand-drawn, straight cut, solid, offset (a thrown silhouette copy placed with offset_x/offset_y), dotted — with color and width. The die-cut sticker look is a white solid or cut stroke.
- set_removal_background fills in behind the subject inside the clip's own picture: a flat color or a project image. For new scenery behind a person, an image fill here or scenery on a lower track both read; the lower track keeps the scenery independently editable.